2023-10-14T16:22:11.989 INFO:root:teuthology version: 0.0.1.dev197+g8cdab07 2023-10-14T16:22:11.991 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857 --description fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} --name mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi --owner mchangir -- /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/orig.config.yaml 2023-10-14T16:22:12.037 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T16:22:12.125 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857 branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} email: mchangir@redhat.com first_in_suite: false job_id: '7427857' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug finisher: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding log-whitelist: - OSD full dropping all updates - OSD near full - pausewr flag - failsafe engaged, dropping updates - failsafe disengaged, no longer dropping - is full \(reached quota - POOL_FULL - POOL_BACKFILLFULL sha1: c7471068426370a3754c2250e7729280233d552b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: c7471068426370a3754c2250e7729280233d552b kclient: syntax: v1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments sha1: c7471068426370a3754c2250e7729280233d552b owner: mchangir priority: 25 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 5837 sha1: c7471068426370a3754c2250e7729280233d552b sleep_before_teardown: 0 suite: fs suite_branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c7471068426370a3754c2250e7729280233d552b/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c7471068426370a3754c2250e7729280233d552b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi029.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIenUNMjS8jLbCKDUoA7LlCVAfZYMnpf8GelG1yGs0Dgy+1xDbuTILZXW1iFdqKmiHAbnH8lIKg+zsbMUw3XwYQ= smithi186.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU6xheaByxOWomGjN7eaKx5atbspCF5BwcdL4eoU8q6q3asLPy0dBw6fYz2uLTcrEYRjiqWnzO8he55rz4DVHM= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - kclient: null - cephfs_test_runner: modules: - tasks.cephfs.test_snap_schedules teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 8cdab074dcca9a68965bc5a50e9c30b691949723 timestamp: 2023-10-14_16:12:46 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-14T16:22:12.125 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c7471068426370a3754c2250e7729280233d552b/qa; will attempt to use it 2023-10-14T16:22:12.126 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c7471068426370a3754c2250e7729280233d552b/qa/tasks 2023-10-14T16:22:12.126 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-14T16:22:12.128 INFO:teuthology.task.internal:Checking packages... 2023-10-14T16:22:12.151 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c7471068426370a3754c2250e7729280233d552b' 2023-10-14T16:22:12.151 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-14T16:22:12.152 INFO:teuthology.packaging:ref: None 2023-10-14T16:22:12.152 INFO:teuthology.packaging:tag: None 2023-10-14T16:22:12.152 INFO:teuthology.packaging:branch: wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-14T16:22:12.153 INFO:teuthology.packaging:sha1: c7471068426370a3754c2250e7729280233d552b 2023-10-14T16:22:12.153 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments 2023-10-14T16:22:12.310 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-6675.gc7471068 2023-10-14T16:22:12.310 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-14T16:22:12.317 INFO:teuthology.task.internal:no buildpackages task found 2023-10-14T16:22:12.318 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-14T16:22:12.323 INFO:teuthology.task.internal:Saving configuration 2023-10-14T16:22:12.342 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-14T16:22:12.347 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T16:22:12.364 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 16:16:54.151268', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIenUNMjS8jLbCKDUoA7LlCVAfZYMnpf8GelG1yGs0Dgy+1xDbuTILZXW1iFdqKmiHAbnH8lIKg+zsbMUw3XwYQ='} 2023-10-14T16:22:12.382 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 16:16:54.149741', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU6xheaByxOWomGjN7eaKx5atbspCF5BwcdL4eoU8q6q3asLPy0dBw6fYz2uLTcrEYRjiqWnzO8he55rz4DVHM='} 2023-10-14T16:22:12.383 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-14T16:22:12.398 INFO:teuthology.task.internal:roles: ubuntu@smithi029.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2023-10-14T16:22:12.398 INFO:teuthology.task.internal:roles: ubuntu@smithi186.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2023-10-14T16:22:12.399 INFO:teuthology.run_tasks:Running task console_log... 2023-10-14T16:22:12.470 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7eff0ad67a60>, signals=[15]) 2023-10-14T16:22:12.471 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-14T16:22:12.476 INFO:teuthology.task.internal:Opening connections... 2023-10-14T16:22:12.477 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:22:12.479 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:22:12.577 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:22:12.578 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:22:12.661 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-14T16:22:12.667 DEBUG:teuthology.orchestra.run.smithi029:> uname -m 2023-10-14T16:22:12.685 INFO:teuthology.orchestra.run.smithi029.stdout:x86_64 2023-10-14T16:22:12.686 DEBUG:teuthology.orchestra.run.smithi029:> cat /etc/os-release 2023-10-14T16:22:12.743 INFO:teuthology.orchestra.run.smithi029.stdout:NAME="CentOS Stream" 2023-10-14T16:22:12.743 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION="9" 2023-10-14T16:22:12.743 INFO:teuthology.orchestra.run.smithi029.stdout:ID="centos" 2023-10-14T16:22:12.744 INFO:teuthology.orchestra.run.smithi029.stdout:ID_LIKE="rhel fedora" 2023-10-14T16:22:12.744 INFO:teuthology.orchestra.run.smithi029.stdout:VERSION_ID="9" 2023-10-14T16:22:12.744 INFO:teuthology.orchestra.run.smithi029.stdout:PLATFORM_ID="platform:el9" 2023-10-14T16:22:12.744 INFO:teuthology.orchestra.run.smithi029.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-14T16:22:12.744 INFO:teuthology.orchestra.run.smithi029.stdout:ANSI_COLOR="0;31" 2023-10-14T16:22:12.745 INFO:teuthology.orchestra.run.smithi029.stdout:LOGO="fedora-logo-icon" 2023-10-14T16:22:12.745 INFO:teuthology.orchestra.run.smithi029.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-14T16:22:12.745 INFO:teuthology.orchestra.run.smithi029.stdout:HOME_URL="https://centos.org/" 2023-10-14T16:22:12.745 INFO:teuthology.orchestra.run.smithi029.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-14T16:22:12.745 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-14T16:22:12.746 INFO:teuthology.orchestra.run.smithi029.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-14T16:22:12.747 INFO:teuthology.lock.ops:Updating smithi029.front.sepia.ceph.com on lock server 2023-10-14T16:22:12.770 DEBUG:teuthology.orchestra.run.smithi186:> uname -m 2023-10-14T16:22:12.788 INFO:teuthology.orchestra.run.smithi186.stdout:x86_64 2023-10-14T16:22:12.788 DEBUG:teuthology.orchestra.run.smithi186:> cat /etc/os-release 2023-10-14T16:22:12.846 INFO:teuthology.orchestra.run.smithi186.stdout:NAME="CentOS Stream" 2023-10-14T16:22:12.846 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION="9" 2023-10-14T16:22:12.847 INFO:teuthology.orchestra.run.smithi186.stdout:ID="centos" 2023-10-14T16:22:12.847 INFO:teuthology.orchestra.run.smithi186.stdout:ID_LIKE="rhel fedora" 2023-10-14T16:22:12.847 INFO:teuthology.orchestra.run.smithi186.stdout:VERSION_ID="9" 2023-10-14T16:22:12.847 INFO:teuthology.orchestra.run.smithi186.stdout:PLATFORM_ID="platform:el9" 2023-10-14T16:22:12.847 INFO:teuthology.orchestra.run.smithi186.stdout:PRETTY_NAME="CentOS Stream 9" 2023-10-14T16:22:12.848 INFO:teuthology.orchestra.run.smithi186.stdout:ANSI_COLOR="0;31" 2023-10-14T16:22:12.848 INFO:teuthology.orchestra.run.smithi186.stdout:LOGO="fedora-logo-icon" 2023-10-14T16:22:12.848 INFO:teuthology.orchestra.run.smithi186.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2023-10-14T16:22:12.849 INFO:teuthology.orchestra.run.smithi186.stdout:HOME_URL="https://centos.org/" 2023-10-14T16:22:12.849 INFO:teuthology.orchestra.run.smithi186.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-14T16:22:12.849 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2023-10-14T16:22:12.850 INFO:teuthology.orchestra.run.smithi186.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-14T16:22:12.850 INFO:teuthology.lock.ops:Updating smithi186.front.sepia.ceph.com on lock server 2023-10-14T16:22:12.870 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-14T16:22:12.957 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-14T16:22:12.963 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-14T16:22:12.964 DEBUG:teuthology.orchestra.run.smithi029:> test '!' -e /home/ubuntu/cephtest 2023-10-14T16:22:12.967 DEBUG:teuthology.orchestra.run.smithi186:> test '!' -e /home/ubuntu/cephtest 2023-10-14T16:22:12.984 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-14T16:22:12.989 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-14T16:22:12.990 DEBUG:teuthology.orchestra.run.smithi029:> test -z $(ls -A /var/lib/ceph) 2023-10-14T16:22:13.025 DEBUG:teuthology.orchestra.run.smithi186:> test -z $(ls -A /var/lib/ceph) 2023-10-14T16:22:13.050 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-14T16:22:13.106 INFO:teuthology.run_tasks:Running task kernel... 2023-10-14T16:22:13.121 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2023-10-14T16:22:13.122 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}, 'client.2': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}, 'client.3': {'branch': 'testing'}} 2023-10-14T16:22:13.122 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}, 'client.1': {'branch': 'testing'}}, timeout 300 2023-10-14T16:22:13.122 DEBUG:teuthology.orchestra.run.smithi029:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-14T16:22:13.123 DEBUG:teuthology.orchestra.run.smithi186:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-14T16:22:13.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:22:13.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-14T16:22:13.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:22:13.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2023-10-14T16:22:13.389 DEBUG:teuthology.task.kernel:sha1 for client.0 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:22:13.390 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-14T16:22:13.390 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-14T16:22:13.395 DEBUG:teuthology.task.kernel:sha1 for client.1 is 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:22:13.395 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "47672119149b103ec7cec2cc65c063ccbb001ac7"... 2023-10-14T16:22:13.395 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2023-10-14T16:22:13.408 INFO:teuthology.orchestra.run.smithi029.stdout:5.14.0-361.el9.x86_64 2023-10-14T16:22:13.408 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:22:13.408 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-14T16:22:13.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum upgrade -y linux-firmware 2023-10-14T16:22:13.412 INFO:teuthology.orchestra.run.smithi186.stdout:5.14.0-361.el9.x86_64 2023-10-14T16:22:13.412 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:22:13.413 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2023-10-14T16:22:13.413 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum upgrade -y linux-firmware 2023-10-14T16:22:15.541 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 15 MB/s | 6.3 MB 00:00 2023-10-14T16:22:16.546 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 12 MB/s | 6.3 MB 00:00 2023-10-14T16:22:17.108 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 3.6 MB/s | 2.0 MB 00:00 2023-10-14T16:22:17.940 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2023-10-14T16:22:18.176 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 7.9 MB 00:00 2023-10-14T16:22:21.458 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 7.9 MB/s | 18 MB 00:02 2023-10-14T16:22:21.573 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 2.4 MB/s | 7.9 MB 00:03 2023-10-14T16:22:24.525 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 9.5 MB/s | 18 MB 00:01 2023-10-14T16:22:27.150 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 2.4 MB/s | 5.7 MB 00:02 2023-10-14T16:22:28.900 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 14 kB 00:00 2023-10-14T16:22:29.918 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.1 MB/s | 5.7 MB 00:01 2023-10-14T16:22:30.168 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 19 MB 00:01 2023-10-14T16:22:31.346 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 46 kB/s | 14 kB 00:00 2023-10-14T16:22:33.308 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-14T16:22:34.643 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2023-10-14T16:22:36.837 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2023-10-14T16:22:36.844 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2023-10-14T16:22:36.845 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2023-10-14T16:22:36.845 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2023-10-14T16:22:36.845 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2023-10-14T16:22:36.845 INFO:teuthology.orchestra.run.smithi186.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-14T16:22:36.846 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:22:36.846 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2023-10-14T16:22:36.846 INFO:teuthology.orchestra.run.smithi186.stdout:================================================================================ 2023-10-14T16:22:36.846 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 1 Package 2023-10-14T16:22:36.846 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:22:36.847 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 326 M 2023-10-14T16:22:36.847 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2023-10-14T16:22:37.950 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-10-14T16:22:40.212 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-14T16:22:40.227 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-14T16:22:40.227 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-10-14T16:22:40.228 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-14T16:22:40.228 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-14T16:22:40.228 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware noarch 20230926-140.el9 CentOS-BaseOS 326 M 2023-10-14T16:22:40.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:22:40.228 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-14T16:22:40.229 INFO:teuthology.orchestra.run.smithi029.stdout:================================================================================ 2023-10-14T16:22:40.229 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 1 Package 2023-10-14T16:22:40.229 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:22:40.229 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 326 M 2023-10-14T16:22:40.229 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-14T16:22:46.677 INFO:teuthology.orchestra.run.smithi186.stdout:linux-firmware-20230926-140.el9.noarch.rpm 33 MB/s | 326 MB 00:09 2023-10-14T16:22:46.678 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2023-10-14T16:22:46.679 INFO:teuthology.orchestra.run.smithi186.stdout:Total 33 MB/s | 326 MB 00:09 2023-10-14T16:22:46.679 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2023-10-14T16:22:46.759 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2023-10-14T16:22:46.759 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2023-10-14T16:22:48.844 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2023-10-14T16:22:48.845 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2023-10-14T16:22:51.038 INFO:teuthology.orchestra.run.smithi029.stdout:linux-firmware-20230926-140.el9.noarch.rpm 30 MB/s | 326 MB 00:10 2023-10-14T16:22:51.039 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-14T16:22:51.039 INFO:teuthology.orchestra.run.smithi029.stdout:Total 30 MB/s | 326 MB 00:10 2023-10-14T16:22:51.039 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-14T16:22:51.142 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-14T16:22:51.143 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-14T16:22:53.310 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-14T16:22:53.310 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-14T16:22:54.548 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2023-10-14T16:22:55.002 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-14T16:22:55.090 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:22:55.898 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:22:55.898 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-14T16:22:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:22:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:22:56.309 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2023-10-14T16:22:56.309 INFO:teuthology.orchestra.run.smithi186.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-14T16:22:56.310 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:22:56.310 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2023-10-14T16:22:56.626 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-14T16:22:56.627 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:22:56.770 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-14T16:22:56.770 DEBUG:teuthology.orchestra.run.smithi186:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-14T16:22:56.773 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi186.front.sepia.ceph.com to complete... 2023-10-14T16:23:01.282 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-14T16:23:01.573 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-14T16:23:01.681 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:23:02.951 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:23:02.952 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230926-140.el9.noarch 1/2 2023-10-14T16:23:03.380 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2023-10-14T16:23:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:23:03.381 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-14T16:23:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: linux-firmware-20230926-140.el9.noarch 2023-10-14T16:23:03.381 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:23:03.381 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-14T16:23:03.754 INFO:teuthology.task.kernel:Downloading kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-14T16:23:03.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=47672119149b103ec7cec2cc65c063ccbb001ac7 2023-10-14T16:23:04.007 INFO:teuthology.task.kernel:fetching, builder baseurl is https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ 2023-10-14T16:23:04.008 DEBUG:teuthology.orchestra.run.smithi029:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/ --input-file=- 2023-10-14T16:23:04.013 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi029.front.sepia.ceph.com to complete... 2023-10-14T16:23:08.581 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14 16:23:08 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-14T16:23:08.581 INFO:teuthology.orchestra.run.smithi186.stderr:FINISHED --2023-10-14 16:23:08-- 2023-10-14T16:23:08.581 INFO:teuthology.orchestra.run.smithi186.stderr:Total wall clock time: 12s 2023-10-14T16:23:08.582 INFO:teuthology.orchestra.run.smithi186.stderr:Downloaded: 1 files, 306M in 11s (26.6 MB/s) 2023-10-14T16:23:08.583 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.1... 2023-10-14T16:23:08.583 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-14T16:23:10.437 INFO:teuthology.orchestra.run.smithi186.stdout:Verifying... ######################################## 2023-10-14T16:23:10.525 INFO:teuthology.orchestra.run.smithi186.stdout:Preparing... ######################################## 2023-10-14T16:23:10.605 INFO:teuthology.orchestra.run.smithi186.stdout:Updating / installing... 2023-10-14T16:23:17.735 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-14T16:23:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14 16:23:23 URL:https://2.chacra.ceph.com/r/kernel/testing/47672119149b103ec7cec2cc65c063ccbb001ac7/centos/9/flavors/default/x86_64/kernel-6.6.0_rc5_g47672119149b-1.x86_64.rpm [320683515/320683515] -> "/tmp/kernel.x86_64.rpm" [1] 2023-10-14T16:23:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:FINISHED --2023-10-14 16:23:23-- 2023-10-14T16:23:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:Total wall clock time: 19s 2023-10-14T16:23:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:Downloaded: 1 files, 306M in 19s (16.2 MB/s) 2023-10-14T16:23:23.518 INFO:teuthology.task.kernel:Installing kernel 47672119149b103ec7cec2cc65c063ccbb001ac7 on client.0... 2023-10-14T16:23:23.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2023-10-14T16:23:25.461 INFO:teuthology.orchestra.run.smithi029.stdout:Verifying... ######################################## 2023-10-14T16:23:25.538 INFO:teuthology.orchestra.run.smithi029.stdout:Preparing... ######################################## 2023-10-14T16:23:25.622 INFO:teuthology.orchestra.run.smithi029.stdout:Updating / installing... 2023-10-14T16:23:33.003 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-6.6.0_rc5_g47672119149b-1 ######################################## 2023-10-14T16:23:33.075 INFO:teuthology.orchestra.run.smithi186.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-14T16:23:54.306 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi186.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-14T16:23:54.307 DEBUG:teuthology.orchestra.run.smithi186:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-14T16:23:54.386 INFO:teuthology.orchestra.run.smithi186.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.386 INFO:teuthology.orchestra.run.smithi186.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.387 INFO:teuthology.orchestra.run.smithi186.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.387 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.387 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-14T16:23:54.387 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-14T16:23:54.388 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-14T16:23:54.388 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-14T16:23:54.388 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-14T16:23:54.388 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-14T16:23:54.389 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-14T16:23:54.389 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-14T16:23:54.389 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-14T16:23:54.390 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-14T16:23:54.390 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-14T16:23:54.390 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-14T16:23:54.390 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-14T16:23:54.391 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-14T16:23:54.391 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-14T16:23:54.391 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-14T16:23:54.391 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-14T16:23:54.392 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-14T16:23:54.392 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-14T16:23:54.392 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-14T16:23:54.392 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-14T16:23:54.392 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-14T16:23:54.393 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-14T16:23:54.393 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-14T16:23:54.393 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-14T16:23:54.393 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-14T16:23:54.394 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-14T16:23:54.394 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-14T16:23:54.394 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-14T16:23:54.394 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-14T16:23:54.395 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-14T16:23:54.395 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-14T16:23:54.395 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-14T16:23:54.395 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-14T16:23:54.396 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-14T16:23:54.396 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-14T16:23:54.396 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-14T16:23:54.396 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-14T16:23:54.396 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-14T16:23:54.397 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-14T16:23:54.397 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-14T16:23:54.397 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-14T16:23:54.397 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-14T16:23:54.397 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-14T16:23:54.398 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-14T16:23:54.398 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-14T16:23:54.398 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-14T16:23:54.398 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-14T16:23:54.399 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-14T16:23:54.399 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-14T16:23:54.399 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-14T16:23:54.399 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-14T16:23:54.399 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-14T16:23:54.400 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-14T16:23:54.400 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-14T16:23:54.400 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-14T16:23:54.400 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-14T16:23:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-14T16:23:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-14T16:23:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-14T16:23:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-14T16:23:54.401 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-14T16:23:54.402 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-14T16:23:54.402 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-14T16:23:54.402 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-14T16:23:54.402 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-14T16:23:54.403 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-14T16:23:54.403 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-14T16:23:54.403 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-14T16:23:54.403 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-14T16:23:54.403 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-14T16:23:54.404 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-14T16:23:54.404 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-14T16:23:54.404 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-14T16:23:54.404 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-14T16:23:54.405 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-14T16:23:54.405 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-14T16:23:54.405 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-14T16:23:54.405 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-14T16:23:54.405 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-14T16:23:54.406 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-14T16:23:54.406 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-14T16:23:54.406 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-14T16:23:54.406 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-14T16:23:54.407 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-14T16:23:54.407 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-14T16:23:54.407 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-14T16:23:54.407 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-14T16:23:54.407 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-14T16:23:54.408 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-14T16:23:54.408 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-14T16:23:54.408 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-14T16:23:54.408 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-14T16:23:54.408 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-14T16:23:54.409 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-14T16:23:54.409 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-14T16:23:54.409 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-14T16:23:54.409 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-14T16:23:54.409 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-14T16:23:54.410 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-14T16:23:54.410 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-14T16:23:54.410 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-14T16:23:54.410 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-14T16:23:54.411 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-14T16:23:54.411 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-14T16:23:54.411 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-14T16:23:54.411 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-14T16:23:54.412 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-14T16:23:54.412 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-14T16:23:54.412 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-14T16:23:54.412 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-14T16:23:54.412 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-14T16:23:54.413 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-14T16:23:54.413 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-14T16:23:54.413 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-14T16:23:54.413 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-14T16:23:54.413 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-14T16:23:54.414 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-14T16:23:54.414 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-14T16:23:54.414 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-14T16:23:54.414 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-14T16:23:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-14T16:23:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-14T16:23:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-14T16:23:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-14T16:23:54.415 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-14T16:23:54.416 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-14T16:23:54.416 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-14T16:23:54.416 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-14T16:23:54.416 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-14T16:23:54.417 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-14T16:23:54.417 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-14T16:23:54.417 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-14T16:23:54.417 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-14T16:23:54.417 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-14T16:23:54.418 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-14T16:23:54.418 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-14T16:23:54.418 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-14T16:23:54.418 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-14T16:23:54.418 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-14T16:23:54.419 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-14T16:23:54.419 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-14T16:23:54.419 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-14T16:23:54.419 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-14T16:23:54.420 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-14T16:23:54.420 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-14T16:23:54.420 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-14T16:23:54.420 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-14T16:23:54.420 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-14T16:23:54.421 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-14T16:23:54.421 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-14T16:23:54.421 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-14T16:23:54.421 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-14T16:23:54.422 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-14T16:23:54.422 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-14T16:23:54.422 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-14T16:23:54.422 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-14T16:23:54.423 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-14T16:23:54.423 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-14T16:23:54.423 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-14T16:23:54.423 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-14T16:23:54.423 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-14T16:23:54.424 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-14T16:23:54.424 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-14T16:23:54.424 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-14T16:23:54.424 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-14T16:23:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-14T16:23:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-14T16:23:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-14T16:23:54.425 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-14T16:23:54.426 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-14T16:23:54.426 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-14T16:23:54.426 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-14T16:23:54.426 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-14T16:23:54.426 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-14T16:23:54.427 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-14T16:23:54.427 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-14T16:23:54.427 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-14T16:23:54.427 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-14T16:23:54.428 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-14T16:23:54.428 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-14T16:23:54.428 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-14T16:23:54.428 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-14T16:23:54.429 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-14T16:23:54.429 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-14T16:23:54.429 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-14T16:23:54.429 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-14T16:23:54.429 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-14T16:23:54.430 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-14T16:23:54.430 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-14T16:23:54.430 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-14T16:23:54.430 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-14T16:23:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-14T16:23:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-14T16:23:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-14T16:23:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-14T16:23:54.431 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-14T16:23:54.432 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-14T16:23:54.432 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-14T16:23:54.432 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-14T16:23:54.432 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-14T16:23:54.432 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-14T16:23:54.433 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-14T16:23:54.433 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-14T16:23:54.433 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-14T16:23:54.433 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-14T16:23:54.434 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-14T16:23:54.434 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-14T16:23:54.434 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-14T16:23:54.434 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-14T16:23:54.434 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-14T16:23:54.435 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-14T16:23:54.435 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-14T16:23:54.435 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-14T16:23:54.435 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-14T16:23:54.436 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-14T16:23:54.436 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-14T16:23:54.436 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-14T16:23:54.436 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-14T16:23:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-14T16:23:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-14T16:23:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-14T16:23:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-14T16:23:54.437 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-14T16:23:54.438 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-14T16:23:54.438 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-14T16:23:54.438 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-14T16:23:54.438 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-14T16:23:54.439 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-14T16:23:54.439 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-14T16:23:54.439 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-14T16:23:54.439 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-14T16:23:54.439 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-14T16:23:54.440 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-14T16:23:54.440 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-14T16:23:54.440 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-14T16:23:54.440 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-14T16:23:54.441 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-14T16:23:54.441 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-14T16:23:54.441 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-14T16:23:54.441 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-14T16:23:54.441 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-14T16:23:54.442 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-14T16:23:54.442 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-14T16:23:54.442 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-14T16:23:54.442 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-14T16:23:54.445 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-14T16:23:54.445 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-14T16:23:54.445 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-14T16:23:54.446 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-14T16:23:54.446 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-14T16:23:54.446 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-14T16:23:54.446 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-14T16:23:54.446 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-14T16:23:54.447 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-14T16:23:54.447 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-14T16:23:54.447 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-14T16:23:54.447 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-14T16:23:54.448 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-14T16:23:54.448 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-14T16:23:54.448 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-14T16:23:54.448 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-14T16:23:54.449 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-14T16:23:54.449 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-14T16:23:54.449 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-14T16:23:54.449 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-14T16:23:54.449 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-14T16:23:54.450 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-14T16:23:54.450 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-14T16:23:54.450 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-14T16:23:54.450 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-14T16:23:54.450 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-14T16:23:54.451 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-14T16:23:54.451 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-14T16:23:54.451 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-14T16:23:54.451 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-14T16:23:54.452 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-14T16:23:54.452 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-14T16:23:54.452 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-14T16:23:54.452 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-14T16:23:54.452 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-14T16:23:54.453 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-14T16:23:54.453 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-14T16:23:54.453 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-14T16:23:54.453 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-14T16:23:54.453 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-14T16:23:54.454 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-14T16:23:54.454 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-14T16:23:54.454 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-14T16:23:54.454 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-14T16:23:54.455 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-14T16:23:54.455 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-14T16:23:54.455 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-14T16:23:54.455 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-14T16:23:54.455 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-14T16:23:54.456 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-14T16:23:54.456 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-14T16:23:54.456 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-14T16:23:54.456 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-14T16:23:54.457 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-14T16:23:54.457 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-14T16:23:54.457 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-14T16:23:54.457 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-14T16:23:54.457 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-14T16:23:54.458 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-14T16:23:54.458 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-14T16:23:54.458 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-14T16:23:54.458 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-14T16:23:54.458 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-14T16:23:54.459 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-14T16:23:54.459 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-14T16:23:54.459 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-14T16:23:54.459 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-14T16:23:54.460 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-14T16:23:54.460 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-14T16:23:54.460 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-14T16:23:54.460 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-14T16:23:54.460 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-14T16:23:54.461 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-14T16:23:54.461 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-14T16:23:54.461 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-14T16:23:54.461 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-14T16:23:54.461 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-14T16:23:54.462 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-14T16:23:54.462 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-14T16:23:54.462 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-14T16:23:54.462 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-14T16:23:54.463 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-14T16:23:54.463 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-14T16:23:54.463 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-14T16:23:54.463 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-14T16:23:54.463 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-14T16:23:54.464 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-14T16:23:54.464 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-14T16:23:54.464 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-14T16:23:54.464 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-14T16:23:54.464 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-14T16:23:54.465 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-14T16:23:54.465 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-14T16:23:54.465 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-14T16:23:54.465 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-14T16:23:54.466 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-14T16:23:54.466 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-14T16:23:54.466 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-14T16:23:54.466 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-14T16:23:54.466 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-14T16:23:54.467 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-14T16:23:54.467 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-14T16:23:54.467 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-14T16:23:54.467 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-14T16:23:54.467 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-14T16:23:54.468 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-14T16:23:54.468 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-14T16:23:54.468 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-14T16:23:54.468 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-14T16:23:54.469 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-14T16:23:54.469 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-14T16:23:54.469 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-14T16:23:54.469 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-14T16:23:54.469 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-14T16:23:54.470 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-14T16:23:54.470 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-14T16:23:54.470 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-14T16:23:54.470 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-14T16:23:54.470 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-14T16:23:54.471 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-14T16:23:54.471 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-14T16:23:54.471 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-14T16:23:54.471 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-14T16:23:54.471 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-14T16:23:54.472 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-14T16:23:54.472 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-14T16:23:54.472 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-14T16:23:54.472 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-14T16:23:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-14T16:23:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-14T16:23:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-14T16:23:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-14T16:23:54.473 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-14T16:23:54.474 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-14T16:23:54.474 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-14T16:23:54.474 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-14T16:23:54.474 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-14T16:23:54.475 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-14T16:23:54.475 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-14T16:23:54.475 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-14T16:23:54.475 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-14T16:23:54.475 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-14T16:23:54.476 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-14T16:23:54.476 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-14T16:23:54.476 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-14T16:23:54.476 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-14T16:23:54.477 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-14T16:23:54.477 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-14T16:23:54.477 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-14T16:23:54.477 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-14T16:23:54.478 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-14T16:23:54.478 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-14T16:23:54.478 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-14T16:23:54.478 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-14T16:23:54.478 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-14T16:23:54.479 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-14T16:23:54.479 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-14T16:23:54.479 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-14T16:23:54.479 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-14T16:23:54.480 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-14T16:23:54.480 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-14T16:23:54.480 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-14T16:23:54.480 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-14T16:23:54.480 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-14T16:23:54.481 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-14T16:23:54.481 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-14T16:23:54.481 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-14T16:23:54.481 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-14T16:23:54.481 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-14T16:23:54.482 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-14T16:23:54.482 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-14T16:23:54.482 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-14T16:23:54.482 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-14T16:23:54.483 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-14T16:23:54.483 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-14T16:23:54.483 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-14T16:23:54.483 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-14T16:23:54.484 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-14T16:23:54.484 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-14T16:23:54.484 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-14T16:23:54.484 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-14T16:23:54.485 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-14T16:23:54.485 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-14T16:23:54.485 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-14T16:23:54.485 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-14T16:23:54.485 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-14T16:23:54.486 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-14T16:23:54.486 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-14T16:23:54.486 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-14T16:23:54.486 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-14T16:23:54.487 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-14T16:23:54.487 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-14T16:23:54.487 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-14T16:23:54.487 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-14T16:23:54.488 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-14T16:23:54.488 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-14T16:23:54.488 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-14T16:23:54.488 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-14T16:23:54.488 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-14T16:23:54.489 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-14T16:23:54.489 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-14T16:23:54.489 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-14T16:23:54.489 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-14T16:23:54.489 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-14T16:23:54.490 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-14T16:23:54.490 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-14T16:23:54.490 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-14T16:23:54.491 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-14T16:23:54.491 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-14T16:23:54.491 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-14T16:23:54.491 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-14T16:23:54.491 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-14T16:23:54.492 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-14T16:23:54.492 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-14T16:23:54.492 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-14T16:23:54.492 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-14T16:23:54.492 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-14T16:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-14T16:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-14T16:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-14T16:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-14T16:23:54.493 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-14T16:23:54.494 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-14T16:23:54.494 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-14T16:23:54.494 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-14T16:23:54.494 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-14T16:23:54.495 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-14T16:23:54.495 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-14T16:23:54.495 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-14T16:23:54.495 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-14T16:23:54.495 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-14T16:23:54.496 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-14T16:23:54.496 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-14T16:23:54.496 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-14T16:23:54.496 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-14T16:23:54.497 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-14T16:23:54.497 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-14T16:23:54.497 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-14T16:23:54.497 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-14T16:23:54.497 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-14T16:23:54.498 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-14T16:23:54.498 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-14T16:23:54.498 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-14T16:23:54.498 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-14T16:23:54.499 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-14T16:23:54.499 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-14T16:23:54.499 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-14T16:23:54.499 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-14T16:23:54.499 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-14T16:23:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-14T16:23:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-14T16:23:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-14T16:23:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-14T16:23:54.500 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-14T16:23:54.501 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-14T16:23:54.501 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-14T16:23:54.501 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-14T16:23:54.501 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-14T16:23:54.501 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-14T16:23:54.502 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-14T16:23:54.502 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-14T16:23:54.502 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-14T16:23:54.502 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-14T16:23:54.503 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-14T16:23:54.503 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-14T16:23:54.503 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-14T16:23:54.503 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-14T16:23:54.503 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-14T16:23:54.504 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-14T16:23:54.504 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-14T16:23:54.504 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-14T16:23:54.504 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-14T16:23:54.505 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-14T16:23:54.505 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-14T16:23:54.505 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-14T16:23:54.505 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-14T16:23:54.505 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-14T16:23:54.506 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-14T16:23:54.506 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-14T16:23:54.506 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-14T16:23:54.506 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-14T16:23:54.507 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-14T16:23:54.507 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-14T16:23:54.507 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-14T16:23:54.507 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-14T16:23:54.507 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-14T16:23:54.508 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-14T16:23:54.508 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-14T16:23:54.508 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-14T16:23:54.508 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-14T16:23:54.509 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-14T16:23:54.509 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-14T16:23:54.509 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-14T16:23:54.509 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-14T16:23:54.509 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-14T16:23:54.510 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-14T16:23:54.510 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-14T16:23:54.510 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-14T16:23:54.510 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-14T16:23:54.511 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-14T16:23:54.511 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-14T16:23:54.511 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-14T16:23:54.511 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-14T16:23:54.511 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-14T16:23:54.512 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-14T16:23:54.512 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-14T16:23:54.512 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-14T16:23:54.512 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-14T16:23:54.513 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-14T16:23:54.513 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-14T16:23:54.513 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-14T16:23:54.513 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-14T16:23:54.513 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-14T16:23:54.514 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-14T16:23:54.514 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-14T16:23:54.514 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-14T16:23:54.514 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-14T16:23:54.515 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-14T16:23:54.515 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-14T16:23:54.515 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-14T16:23:54.515 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-14T16:23:54.515 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-14T16:23:54.516 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-14T16:23:54.516 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-14T16:23:54.516 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-14T16:23:54.516 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-14T16:23:54.516 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-14T16:23:54.517 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-14T16:23:54.517 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-14T16:23:54.517 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-14T16:23:54.517 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-14T16:23:54.518 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-14T16:23:54.518 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-14T16:23:54.518 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-14T16:23:54.518 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-14T16:23:54.518 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-14T16:23:54.519 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-14T16:23:54.519 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-14T16:23:54.519 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-14T16:23:54.519 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-14T16:23:54.520 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-14T16:23:54.520 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-14T16:23:54.520 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-14T16:23:54.520 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-14T16:23:54.520 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-14T16:23:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-14T16:23:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-14T16:23:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-14T16:23:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-14T16:23:54.521 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-14T16:23:54.522 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-14T16:23:54.522 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-14T16:23:54.522 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-14T16:23:54.522 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-14T16:23:54.523 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-14T16:23:54.523 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-14T16:23:54.523 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-14T16:23:54.523 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-14T16:23:54.523 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-14T16:23:54.524 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-14T16:23:54.524 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-14T16:23:54.524 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-14T16:23:54.524 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-14T16:23:54.525 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-14T16:23:54.525 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-14T16:23:54.525 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-14T16:23:54.525 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-14T16:23:54.525 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-14T16:23:54.526 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-14T16:23:54.526 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-14T16:23:54.526 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-14T16:23:54.526 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-14T16:23:54.526 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-14T16:23:54.527 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-14T16:23:54.527 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-14T16:23:54.527 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-14T16:23:54.527 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-14T16:23:54.528 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-14T16:23:54.528 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-14T16:23:54.528 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-14T16:23:54.528 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-14T16:23:54.528 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-14T16:23:54.529 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-14T16:23:54.529 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-14T16:23:54.529 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-14T16:23:54.529 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-14T16:23:54.529 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-14T16:23:54.530 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-14T16:23:54.530 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-14T16:23:54.530 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-14T16:23:54.530 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-14T16:23:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-14T16:23:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-14T16:23:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-14T16:23:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-14T16:23:54.531 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-14T16:23:54.532 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-14T16:23:54.532 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-14T16:23:54.532 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-14T16:23:54.532 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-14T16:23:54.532 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-14T16:23:54.533 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-14T16:23:54.533 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-14T16:23:54.533 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-14T16:23:54.533 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-14T16:23:54.533 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-14T16:23:54.534 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-14T16:23:54.534 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-14T16:23:54.534 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-14T16:23:54.534 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-14T16:23:54.534 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-14T16:23:54.535 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-14T16:23:54.535 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-14T16:23:54.535 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-14T16:23:54.535 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-14T16:23:54.536 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-14T16:23:54.536 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-14T16:23:54.536 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-14T16:23:54.536 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-14T16:23:54.536 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-14T16:23:54.537 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-14T16:23:54.537 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-14T16:23:54.537 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-14T16:23:54.537 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-14T16:23:54.537 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-14T16:23:54.538 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-14T16:23:54.538 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-14T16:23:54.538 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-14T16:23:54.538 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-14T16:23:54.538 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-14T16:23:54.539 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-14T16:23:54.539 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-14T16:23:54.539 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-14T16:23:54.539 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-14T16:23:54.539 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-14T16:23:54.540 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-14T16:23:54.540 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-14T16:23:54.540 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-14T16:23:54.540 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-14T16:23:54.541 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-14T16:23:54.541 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-14T16:23:54.541 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-14T16:23:54.541 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-14T16:23:54.541 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-14T16:23:54.542 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-14T16:23:54.542 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-14T16:23:54.542 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-14T16:23:54.542 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-14T16:23:54.542 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-14T16:23:54.543 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-14T16:23:54.543 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-14T16:23:54.543 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-14T16:23:54.543 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-14T16:23:54.544 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-14T16:23:54.544 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-14T16:23:54.544 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-14T16:23:54.544 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-14T16:23:54.544 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-14T16:23:54.545 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-14T16:23:54.545 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-14T16:23:54.545 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-14T16:23:54.545 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-14T16:23:54.545 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-14T16:23:54.546 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-14T16:23:54.546 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-14T16:23:54.546 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-14T16:23:54.546 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-14T16:23:54.546 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-14T16:23:54.547 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-14T16:23:54.547 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-14T16:23:54.547 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-14T16:23:54.547 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-14T16:23:54.547 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-14T16:23:54.548 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-14T16:23:54.548 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-14T16:23:54.548 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-14T16:23:54.548 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-14T16:23:54.549 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-14T16:23:54.549 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-14T16:23:54.549 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-14T16:23:54.549 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-14T16:23:54.549 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-14T16:23:54.550 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-14T16:23:54.550 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-14T16:23:54.550 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-14T16:23:54.550 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-14T16:23:54.550 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-14T16:23:54.551 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-14T16:23:54.551 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-14T16:23:54.551 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-14T16:23:54.551 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-14T16:23:54.551 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-14T16:23:54.552 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-14T16:23:54.552 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-14T16:23:54.552 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-14T16:23:54.552 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-14T16:23:54.552 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-14T16:23:54.553 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-14T16:23:54.553 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-14T16:23:54.553 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-14T16:23:54.553 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-14T16:23:54.554 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-14T16:23:54.554 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-14T16:23:54.554 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-14T16:23:54.554 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-14T16:23:54.554 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-14T16:23:54.555 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-14T16:23:54.555 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-14T16:23:54.555 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-14T16:23:54.555 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-14T16:23:54.555 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-14T16:23:54.556 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-14T16:23:54.556 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-14T16:23:54.556 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-14T16:23:54.556 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-14T16:23:54.557 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-14T16:23:54.557 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-14T16:23:54.557 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-14T16:23:54.557 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-14T16:23:54.557 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-14T16:23:54.558 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-14T16:23:54.558 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-14T16:23:54.558 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-14T16:23:54.558 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-14T16:23:54.558 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-14T16:23:54.559 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-14T16:23:54.559 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-14T16:23:54.559 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-14T16:23:54.559 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-14T16:23:54.559 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-14T16:23:54.560 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-14T16:23:54.560 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-14T16:23:54.560 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-14T16:23:54.560 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-14T16:23:54.561 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-14T16:23:54.561 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-14T16:23:54.561 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-14T16:23:54.561 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-14T16:23:54.561 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-14T16:23:54.562 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-14T16:23:54.562 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-14T16:23:54.562 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-14T16:23:54.562 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-14T16:23:54.562 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-14T16:23:54.563 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-14T16:23:54.563 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-14T16:23:54.563 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-14T16:23:54.563 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-14T16:23:54.563 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-14T16:23:54.564 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-14T16:23:54.564 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-14T16:23:54.564 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-14T16:23:54.564 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-14T16:23:54.565 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-14T16:23:54.565 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-14T16:23:54.565 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-14T16:23:54.565 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-14T16:23:54.566 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-14T16:23:54.566 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-14T16:23:54.566 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-14T16:23:54.566 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-14T16:23:54.566 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-14T16:23:54.567 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-14T16:23:54.567 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-14T16:23:54.567 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-14T16:23:54.567 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-14T16:23:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-14T16:23:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-14T16:23:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-14T16:23:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-14T16:23:54.568 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-14T16:23:54.569 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-14T16:23:54.569 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-14T16:23:54.569 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-14T16:23:54.569 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-14T16:23:54.569 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-14T16:23:54.570 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-14T16:23:54.570 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-14T16:23:54.570 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-14T16:23:54.570 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-14T16:23:54.570 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-14T16:23:54.571 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-14T16:23:54.571 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-14T16:23:54.571 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-14T16:23:54.571 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-14T16:23:54.571 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-14T16:23:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-14T16:23:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-14T16:23:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-14T16:23:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-14T16:23:54.572 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-14T16:23:54.573 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-14T16:23:54.573 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-14T16:23:54.573 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-14T16:23:54.573 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-14T16:23:54.574 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-14T16:23:54.574 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-14T16:23:54.574 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-14T16:23:54.574 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-14T16:23:54.575 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-14T16:23:54.575 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-14T16:23:54.575 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-14T16:23:54.575 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-14T16:23:54.575 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-14T16:23:54.576 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-14T16:23:54.576 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-14T16:23:54.576 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-14T16:23:54.576 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-14T16:23:54.576 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-14T16:23:54.577 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-14T16:23:54.577 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-14T16:23:54.577 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-14T16:23:54.577 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-14T16:23:54.577 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-14T16:23:54.578 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-14T16:23:54.578 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-14T16:23:54.578 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-14T16:23:54.578 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-14T16:23:54.579 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-14T16:23:54.579 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-14T16:23:54.579 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-14T16:23:54.579 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-14T16:23:54.579 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-14T16:23:54.580 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-14T16:23:54.580 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-14T16:23:54.580 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-14T16:23:54.580 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-14T16:23:54.580 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-14T16:23:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-14T16:23:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-14T16:23:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-14T16:23:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-14T16:23:54.581 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-14T16:23:54.582 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-14T16:23:54.582 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-14T16:23:54.582 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-14T16:23:54.582 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-14T16:23:54.582 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-14T16:23:54.583 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-14T16:23:54.583 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-14T16:23:54.583 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-14T16:23:54.583 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-14T16:23:54.584 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-14T16:23:54.584 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-14T16:23:54.584 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-14T16:23:54.584 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-14T16:23:54.584 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-14T16:23:54.585 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-14T16:23:54.585 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-14T16:23:54.585 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-14T16:23:54.585 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-14T16:23:54.586 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-14T16:23:54.586 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-14T16:23:54.586 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-14T16:23:54.586 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-14T16:23:54.586 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-14T16:23:54.587 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-14T16:23:54.587 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-14T16:23:54.587 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-14T16:23:54.587 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-14T16:23:54.588 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-14T16:23:54.588 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-14T16:23:54.588 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-14T16:23:54.588 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-14T16:23:54.588 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-14T16:23:54.589 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-14T16:23:54.589 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-14T16:23:54.589 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-14T16:23:54.589 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-14T16:23:54.589 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-14T16:23:54.590 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-14T16:23:54.590 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-14T16:23:54.590 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-14T16:23:54.590 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-14T16:23:54.591 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-14T16:23:54.591 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-14T16:23:54.591 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-14T16:23:54.591 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-14T16:23:54.591 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-14T16:23:54.592 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-14T16:23:54.592 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-14T16:23:54.592 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-14T16:23:54.592 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-14T16:23:54.593 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-14T16:23:54.593 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-14T16:23:54.593 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-14T16:23:54.593 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-14T16:23:54.593 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-14T16:23:54.594 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-14T16:23:54.594 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-14T16:23:54.594 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-14T16:23:54.594 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-14T16:23:54.594 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-14T16:23:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-14T16:23:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-14T16:23:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-14T16:23:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-14T16:23:54.595 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-14T16:23:54.596 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-14T16:23:54.596 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-14T16:23:54.596 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-14T16:23:54.596 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-14T16:23:54.597 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-14T16:23:54.597 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-14T16:23:54.597 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-14T16:23:54.597 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-14T16:23:54.597 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-14T16:23:54.598 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-14T16:23:54.598 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-14T16:23:54.598 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-14T16:23:54.598 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-14T16:23:54.598 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-14T16:23:54.599 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-14T16:23:54.599 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-14T16:23:54.599 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-14T16:23:54.599 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-14T16:23:54.599 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-14T16:23:54.600 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-14T16:23:54.600 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-14T16:23:54.600 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-14T16:23:54.600 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-14T16:23:54.600 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-14T16:23:54.601 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-14T16:23:54.601 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-14T16:23:54.601 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-14T16:23:54.601 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-14T16:23:54.601 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-14T16:23:54.602 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-14T16:23:54.602 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-14T16:23:54.602 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-14T16:23:54.602 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-14T16:23:54.603 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-14T16:23:54.603 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-14T16:23:54.603 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-14T16:23:54.603 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-14T16:23:54.604 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-14T16:23:54.604 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-14T16:23:54.604 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-14T16:23:54.604 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-14T16:23:54.604 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-14T16:23:54.605 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-14T16:23:54.605 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-14T16:23:54.605 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-14T16:23:54.605 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-14T16:23:54.606 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-14T16:23:54.606 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-14T16:23:54.606 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-14T16:23:54.606 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-14T16:23:54.606 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-14T16:23:54.607 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-14T16:23:54.607 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-14T16:23:54.607 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-14T16:23:54.607 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-14T16:23:54.608 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-14T16:23:54.608 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-14T16:23:54.608 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-14T16:23:54.608 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-14T16:23:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-14T16:23:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-14T16:23:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-14T16:23:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-14T16:23:54.609 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-14T16:23:54.610 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-14T16:23:54.610 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-14T16:23:54.610 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-14T16:23:54.610 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-14T16:23:54.611 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-14T16:23:54.611 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-14T16:23:54.611 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-14T16:23:54.611 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-14T16:23:54.611 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-14T16:23:54.612 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-14T16:23:54.612 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-14T16:23:54.612 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-14T16:23:54.612 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-14T16:23:54.612 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-14T16:23:54.613 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-14T16:23:54.613 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-14T16:23:54.613 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-14T16:23:54.613 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-14T16:23:54.613 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-14T16:23:54.614 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-14T16:23:54.614 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-14T16:23:54.614 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-14T16:23:54.614 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-14T16:23:54.615 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-14T16:23:54.615 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-14T16:23:54.615 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-14T16:23:54.615 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-14T16:23:54.615 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-14T16:23:54.616 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-14T16:23:54.616 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-14T16:23:54.616 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-14T16:23:54.616 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-14T16:23:54.616 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-14T16:23:54.617 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-14T16:23:54.617 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-14T16:23:54.617 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-14T16:23:54.617 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-14T16:23:54.617 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-14T16:23:54.618 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-14T16:23:54.618 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-14T16:23:54.618 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-14T16:23:54.618 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-14T16:23:54.618 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-14T16:23:54.619 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-14T16:23:54.619 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-14T16:23:54.619 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-14T16:23:54.619 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-14T16:23:54.619 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-14T16:23:54.620 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-14T16:23:54.621 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-14T16:23:54.621 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-14T16:23:54.621 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-14T16:23:54.622 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-14T16:23:54.622 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-14T16:23:54.622 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-14T16:23:54.622 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-14T16:23:54.623 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-14T16:23:54.623 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-14T16:23:54.623 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-14T16:23:54.624 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-14T16:23:54.624 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-14T16:23:54.624 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-14T16:23:54.624 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-14T16:23:54.624 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-14T16:23:54.625 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-14T16:23:54.625 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-14T16:23:54.625 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-14T16:23:54.625 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-14T16:23:54.625 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-14T16:23:54.626 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-14T16:23:54.626 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-14T16:23:54.626 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-14T16:23:54.626 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-14T16:23:54.627 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-14T16:23:54.627 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-14T16:23:54.627 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-14T16:23:54.627 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-14T16:23:54.627 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-14T16:23:54.628 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-14T16:23:54.628 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-14T16:23:54.628 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-14T16:23:54.628 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-14T16:23:54.629 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-14T16:23:54.629 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-14T16:23:54.629 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-14T16:23:54.629 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-14T16:23:54.629 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-14T16:23:54.630 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-14T16:23:54.630 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-14T16:23:54.630 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-14T16:23:54.630 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-14T16:23:54.630 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-14T16:23:54.631 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-14T16:23:54.631 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-14T16:23:54.631 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-14T16:23:54.631 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-14T16:23:54.632 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-14T16:23:54.632 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-14T16:23:54.632 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-14T16:23:54.632 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-14T16:23:54.632 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-14T16:23:54.633 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-14T16:23:54.633 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-14T16:23:54.633 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-14T16:23:54.633 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-14T16:23:54.633 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-14T16:23:54.634 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-14T16:23:54.634 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-14T16:23:54.634 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-14T16:23:54.634 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-14T16:23:54.634 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-14T16:23:54.635 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-14T16:23:54.635 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-14T16:23:54.635 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-14T16:23:54.635 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-14T16:23:54.636 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-14T16:23:54.636 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-14T16:23:54.636 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-14T16:23:54.636 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-14T16:23:54.636 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-14T16:23:54.637 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-14T16:23:54.637 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-14T16:23:54.637 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-14T16:23:54.637 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-14T16:23:54.638 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-14T16:23:54.638 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-14T16:23:54.638 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-14T16:23:54.638 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-14T16:23:54.638 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-14T16:23:54.639 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-14T16:23:54.639 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-14T16:23:54.639 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-14T16:23:54.639 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-14T16:23:54.640 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-14T16:23:54.640 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-14T16:23:54.640 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-14T16:23:54.640 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-14T16:23:54.641 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-14T16:23:54.641 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-14T16:23:54.641 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-14T16:23:54.641 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-14T16:23:54.641 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-14T16:23:54.642 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-14T16:23:54.642 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-14T16:23:54.642 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-14T16:23:54.642 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-14T16:23:54.642 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-14T16:23:54.643 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-14T16:23:54.643 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-14T16:23:54.643 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-14T16:23:54.643 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-14T16:23:54.644 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-14T16:23:54.644 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-14T16:23:54.644 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-14T16:23:54.644 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-14T16:23:54.644 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-14T16:23:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-14T16:23:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-14T16:23:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-14T16:23:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-14T16:23:54.645 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-14T16:23:54.646 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-14T16:23:54.646 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-14T16:23:54.646 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-14T16:23:54.646 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-14T16:23:54.646 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-14T16:23:54.647 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-14T16:23:54.647 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-14T16:23:54.647 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-14T16:23:54.647 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-14T16:23:54.648 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-14T16:23:54.648 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-14T16:23:54.648 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-14T16:23:54.648 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-14T16:23:54.648 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-14T16:23:54.649 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-14T16:23:54.649 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-14T16:23:54.649 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-14T16:23:54.649 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-14T16:23:54.649 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-14T16:23:54.650 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-14T16:23:54.650 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-14T16:23:54.650 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-14T16:23:54.650 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-14T16:23:54.651 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-14T16:23:54.651 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-14T16:23:54.651 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-14T16:23:54.651 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-14T16:23:54.651 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-14T16:23:54.652 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-14T16:23:54.652 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-14T16:23:54.652 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-14T16:23:54.652 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-14T16:23:54.652 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-14T16:23:54.653 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-14T16:23:54.653 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-14T16:23:54.653 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-14T16:23:54.653 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-14T16:23:54.654 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-14T16:23:54.654 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-14T16:23:54.654 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-14T16:23:54.654 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-14T16:23:54.654 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-14T16:23:54.655 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-14T16:23:54.655 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-14T16:23:54.655 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-14T16:23:54.655 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-14T16:23:54.656 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-14T16:23:54.656 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-14T16:23:54.656 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-14T16:23:54.656 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-14T16:23:54.657 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-14T16:23:54.657 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-14T16:23:54.657 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-14T16:23:54.657 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-14T16:23:54.657 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-14T16:23:54.658 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-14T16:23:54.658 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-14T16:23:54.658 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-14T16:23:54.658 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-14T16:23:54.659 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-14T16:23:54.659 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-14T16:23:54.659 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-14T16:23:54.659 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-14T16:23:54.659 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-14T16:23:54.660 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-14T16:23:54.660 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-14T16:23:54.660 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-14T16:23:54.660 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-14T16:23:54.661 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-14T16:23:54.661 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-14T16:23:54.661 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-14T16:23:54.661 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-14T16:23:54.661 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-14T16:23:54.662 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-14T16:23:54.662 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-14T16:23:54.662 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-14T16:23:54.662 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-14T16:23:54.663 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-14T16:23:54.663 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-14T16:23:54.663 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-14T16:23:54.663 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-14T16:23:54.663 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-14T16:23:54.664 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-14T16:23:54.664 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-14T16:23:54.664 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-14T16:23:54.664 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-14T16:23:54.664 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-14T16:23:54.665 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-14T16:23:54.665 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-14T16:23:54.665 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-14T16:23:54.665 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-14T16:23:54.666 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-14T16:23:54.666 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-14T16:23:54.666 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-14T16:23:54.666 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-14T16:23:54.666 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-14T16:23:54.667 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-14T16:23:54.667 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-14T16:23:54.667 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-14T16:23:54.667 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-14T16:23:54.667 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-14T16:23:54.668 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-14T16:23:54.668 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-14T16:23:54.668 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-14T16:23:54.668 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-14T16:23:54.668 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-14T16:23:54.669 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-14T16:23:54.669 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-14T16:23:54.669 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-14T16:23:54.669 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-14T16:23:54.669 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-14T16:23:54.670 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-14T16:23:54.670 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-14T16:23:54.670 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-14T16:23:54.670 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-14T16:23:54.671 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-14T16:23:54.671 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-14T16:23:54.671 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-14T16:23:54.671 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-14T16:23:54.671 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-14T16:23:54.672 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-14T16:23:54.672 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-14T16:23:54.672 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-14T16:23:54.672 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-14T16:23:54.672 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-14T16:23:54.673 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-14T16:23:54.673 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-14T16:23:54.673 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-14T16:23:54.673 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-14T16:23:54.673 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-14T16:23:54.674 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-14T16:23:54.674 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-14T16:23:54.674 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-14T16:23:54.674 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-14T16:23:54.674 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-14T16:23:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-14T16:23:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-14T16:23:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-14T16:23:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-14T16:23:54.675 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-14T16:23:54.676 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-14T16:23:54.676 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-14T16:23:54.676 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-14T16:23:54.676 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-14T16:23:54.676 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-14T16:23:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-14T16:23:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-14T16:23:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-14T16:23:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-14T16:23:54.677 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-14T16:23:54.678 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-14T16:23:54.678 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-14T16:23:54.678 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-14T16:23:54.678 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-14T16:23:54.678 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-14T16:23:54.679 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-14T16:23:54.679 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-14T16:23:54.679 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-14T16:23:54.679 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-14T16:23:54.680 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-14T16:23:54.680 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-14T16:23:54.680 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-14T16:23:54.680 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-14T16:23:54.680 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-14T16:23:54.681 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-14T16:23:54.681 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-14T16:23:54.681 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-14T16:23:54.681 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-14T16:23:54.681 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-14T16:23:54.682 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-14T16:23:54.682 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-14T16:23:54.682 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-14T16:23:54.682 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-14T16:23:54.683 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-14T16:23:54.683 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-14T16:23:54.683 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-14T16:23:54.683 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-14T16:23:54.683 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-14T16:23:54.684 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-14T16:23:54.684 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-14T16:23:54.684 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-14T16:23:54.684 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-14T16:23:54.685 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-14T16:23:54.685 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-14T16:23:54.685 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-14T16:23:54.685 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-14T16:23:54.685 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-14T16:23:54.686 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-14T16:23:54.686 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-14T16:23:54.686 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-14T16:23:54.686 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-14T16:23:54.687 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-14T16:23:54.687 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-14T16:23:54.687 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-14T16:23:54.687 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-14T16:23:54.687 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-14T16:23:54.688 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-14T16:23:54.688 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-14T16:23:54.688 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-14T16:23:54.688 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-14T16:23:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-14T16:23:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-14T16:23:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-14T16:23:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-14T16:23:54.689 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-14T16:23:54.690 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-14T16:23:54.690 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-14T16:23:54.690 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-14T16:23:54.690 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-14T16:23:54.690 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-14T16:23:54.691 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-14T16:23:54.691 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-14T16:23:54.691 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-14T16:23:54.691 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-14T16:23:54.692 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-14T16:23:54.692 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-14T16:23:54.692 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-14T16:23:54.692 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-14T16:23:54.692 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-14T16:23:54.693 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-14T16:23:54.693 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-14T16:23:54.693 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-14T16:23:54.693 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-14T16:23:54.693 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-14T16:23:54.694 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-14T16:23:54.694 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-14T16:23:54.694 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-14T16:23:54.694 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-14T16:23:54.694 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-14T16:23:54.695 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-14T16:23:54.695 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-14T16:23:54.695 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-14T16:23:54.695 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-14T16:23:54.696 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-14T16:23:54.696 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-14T16:23:54.696 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-14T16:23:54.696 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-14T16:23:54.696 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-14T16:23:54.697 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-14T16:23:54.697 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-14T16:23:54.697 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-14T16:23:54.697 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-14T16:23:54.697 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-14T16:23:54.698 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-14T16:23:54.698 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-14T16:23:54.698 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-14T16:23:54.698 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-14T16:23:54.699 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-14T16:23:54.699 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-14T16:23:54.699 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-14T16:23:54.699 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-14T16:23:54.700 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-14T16:23:54.700 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-14T16:23:54.700 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-14T16:23:54.700 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-14T16:23:54.701 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-14T16:23:54.701 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-14T16:23:54.701 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-14T16:23:54.701 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-14T16:23:54.701 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-14T16:23:54.702 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-14T16:23:54.702 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-14T16:23:54.702 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-14T16:23:54.702 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-14T16:23:54.702 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-14T16:23:54.703 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-14T16:23:54.703 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-14T16:23:54.703 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-14T16:23:54.703 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-14T16:23:54.703 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-14T16:23:54.704 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-14T16:23:54.704 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-14T16:23:54.704 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-14T16:23:54.704 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-14T16:23:54.705 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-14T16:23:54.705 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-14T16:23:54.705 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-14T16:23:54.705 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-14T16:23:54.705 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-14T16:23:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-14T16:23:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-14T16:23:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-14T16:23:54.706 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-14T16:23:54.707 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-14T16:23:54.707 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-14T16:23:54.707 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-14T16:23:54.707 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-14T16:23:54.707 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-14T16:23:54.708 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-14T16:23:54.708 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-14T16:23:54.708 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-14T16:23:54.708 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-14T16:23:54.709 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-14T16:23:54.709 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-14T16:23:54.709 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-14T16:23:54.709 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-14T16:23:54.709 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-14T16:23:54.710 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-14T16:23:54.710 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-14T16:23:54.710 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-14T16:23:54.710 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-14T16:23:54.711 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-14T16:23:54.711 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-14T16:23:54.711 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-14T16:23:54.711 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-14T16:23:54.712 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-14T16:23:54.712 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-14T16:23:54.712 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-14T16:23:54.712 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-14T16:23:54.712 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-14T16:23:54.713 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-14T16:23:54.713 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-14T16:23:54.713 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-14T16:23:54.713 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-14T16:23:54.714 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-14T16:23:54.714 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-14T16:23:54.714 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-14T16:23:54.714 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-14T16:23:54.715 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-14T16:23:54.715 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-14T16:23:54.715 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-14T16:23:54.715 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-14T16:23:54.716 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-14T16:23:54.716 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-14T16:23:54.716 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-14T16:23:54.716 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-14T16:23:54.716 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-14T16:23:54.717 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-14T16:23:54.717 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-14T16:23:54.717 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-14T16:23:54.717 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-14T16:23:54.718 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-14T16:23:54.718 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-14T16:23:54.718 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-14T16:23:54.718 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-14T16:23:54.718 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-14T16:23:54.719 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-14T16:23:54.719 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-14T16:23:54.719 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-14T16:23:54.719 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-14T16:23:54.720 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-14T16:23:54.720 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-14T16:23:54.720 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-14T16:23:54.720 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-14T16:23:54.720 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-14T16:23:54.721 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-14T16:23:54.721 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-14T16:23:54.721 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-14T16:23:54.721 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-14T16:23:54.721 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-14T16:23:54.722 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-14T16:23:54.722 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-14T16:23:54.722 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-14T16:23:54.722 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-14T16:23:54.723 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-14T16:23:54.723 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-14T16:23:54.723 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-14T16:23:54.723 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-14T16:23:54.723 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-14T16:23:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-14T16:23:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-14T16:23:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-14T16:23:54.724 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-14T16:23:54.725 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-14T16:23:54.725 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-14T16:23:54.725 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-14T16:23:54.725 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-14T16:23:54.725 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-14T16:23:54.726 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-14T16:23:54.726 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-14T16:23:54.726 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-14T16:23:54.726 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-14T16:23:54.727 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-14T16:23:54.727 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-14T16:23:54.727 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-14T16:23:54.727 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-14T16:23:54.727 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-14T16:23:54.728 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-14T16:23:54.728 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-14T16:23:54.728 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-14T16:23:54.728 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-14T16:23:54.728 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-14T16:23:54.729 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-14T16:23:54.729 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-14T16:23:54.729 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-14T16:23:54.729 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-14T16:23:54.729 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-14T16:23:54.730 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-14T16:23:54.730 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-14T16:23:54.730 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-14T16:23:54.730 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-14T16:23:54.731 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-14T16:23:54.731 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-14T16:23:54.731 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-14T16:23:54.731 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-14T16:23:54.731 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-14T16:23:54.732 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-14T16:23:54.732 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-14T16:23:54.732 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-14T16:23:54.732 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-14T16:23:54.732 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-14T16:23:54.733 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-14T16:23:54.733 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-14T16:23:54.733 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-14T16:23:54.733 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-14T16:23:54.733 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-14T16:23:54.734 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-14T16:23:54.734 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-14T16:23:54.734 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-14T16:23:54.734 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-14T16:23:54.735 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-14T16:23:54.735 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-14T16:23:54.735 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-14T16:23:54.735 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-14T16:23:54.735 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-14T16:23:54.736 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-14T16:23:54.736 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-14T16:23:54.736 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-14T16:23:54.736 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-14T16:23:54.736 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-14T16:23:54.737 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-14T16:23:54.737 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-14T16:23:54.737 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-14T16:23:54.737 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-14T16:23:54.738 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-14T16:23:54.738 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-14T16:23:54.738 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-14T16:23:54.738 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-14T16:23:54.738 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-14T16:23:54.739 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-14T16:23:54.739 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-14T16:23:54.739 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-14T16:23:54.739 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-14T16:23:54.739 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-14T16:23:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-14T16:23:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-14T16:23:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-14T16:23:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-14T16:23:54.740 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-14T16:23:54.741 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-14T16:23:54.741 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-14T16:23:54.741 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-14T16:23:54.741 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-14T16:23:54.741 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-14T16:23:54.742 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-14T16:23:54.742 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-14T16:23:54.742 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-14T16:23:54.742 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-14T16:23:54.743 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-14T16:23:54.743 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-14T16:23:54.743 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-14T16:23:54.743 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-14T16:23:54.743 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-14T16:23:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-14T16:23:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-14T16:23:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-14T16:23:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-14T16:23:54.744 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-14T16:23:54.745 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-14T16:23:54.745 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-14T16:23:54.745 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-14T16:23:54.745 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-14T16:23:54.746 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-14T16:23:54.746 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-14T16:23:54.746 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-14T16:23:54.746 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-14T16:23:54.746 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-14T16:23:54.747 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-14T16:23:54.747 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-14T16:23:54.747 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-14T16:23:54.747 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-14T16:23:54.747 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-14T16:23:54.748 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-14T16:23:54.748 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-14T16:23:54.748 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-14T16:23:54.748 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-14T16:23:54.748 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-14T16:23:54.749 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-14T16:23:54.749 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-14T16:23:54.749 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-14T16:23:54.749 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-14T16:23:54.750 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-14T16:23:54.750 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-14T16:23:54.750 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-14T16:23:54.750 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-14T16:23:54.750 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-14T16:23:54.751 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-14T16:23:54.751 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-14T16:23:54.751 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-14T16:23:54.751 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-14T16:23:54.751 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-14T16:23:54.752 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-14T16:23:54.752 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-14T16:23:54.752 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-14T16:23:54.752 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-14T16:23:54.752 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-14T16:23:54.753 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-14T16:23:54.753 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-14T16:23:54.753 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-14T16:23:54.753 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-14T16:23:54.753 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-14T16:23:54.754 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-14T16:23:54.754 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-14T16:23:54.754 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-14T16:23:54.754 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-14T16:23:54.754 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-14T16:23:54.755 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-14T16:23:54.755 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-14T16:23:54.755 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-14T16:23:54.755 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-14T16:23:54.755 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-14T16:23:54.756 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-14T16:23:54.756 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-14T16:23:54.756 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-14T16:23:54.756 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-14T16:23:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-14T16:23:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-14T16:23:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-14T16:23:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-14T16:23:54.757 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-14T16:23:54.758 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-14T16:23:54.758 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-14T16:23:54.758 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-14T16:23:54.758 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-14T16:23:54.758 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-14T16:23:54.759 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-14T16:23:54.759 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-14T16:23:54.759 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-14T16:23:54.759 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-14T16:23:54.759 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-14T16:23:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-14T16:23:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-14T16:23:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-14T16:23:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-14T16:23:54.760 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-14T16:23:54.761 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-14T16:23:54.761 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-14T16:23:54.761 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-14T16:23:54.761 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-14T16:23:54.761 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-14T16:23:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-14T16:23:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-14T16:23:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-14T16:23:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-14T16:23:54.762 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-14T16:23:54.763 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-14T16:23:54.763 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-14T16:23:54.763 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-14T16:23:54.763 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-14T16:23:54.764 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-14T16:23:54.764 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-14T16:23:54.764 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-14T16:23:54.764 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-14T16:23:54.764 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-14T16:23:54.765 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-14T16:23:54.765 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-14T16:23:54.765 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-14T16:23:54.765 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-14T16:23:54.765 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-14T16:23:54.766 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-14T16:23:54.766 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-14T16:23:54.766 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-14T16:23:54.766 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-14T16:23:54.767 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-14T16:23:54.767 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-14T16:23:54.767 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-14T16:23:54.767 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-14T16:23:54.767 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-14T16:23:54.768 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-14T16:23:54.769 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-14T16:23:54.769 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-14T16:23:54.769 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-14T16:23:54.769 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-14T16:23:54.769 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-14T16:23:54.770 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-14T16:23:54.770 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-14T16:23:54.770 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-14T16:23:54.770 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-14T16:23:54.771 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-14T16:23:54.771 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-14T16:23:54.771 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-14T16:23:54.771 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-14T16:23:54.771 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-14T16:23:54.772 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-14T16:23:54.772 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-14T16:23:54.772 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-14T16:23:54.772 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-14T16:23:54.772 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-14T16:23:54.773 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-14T16:23:54.774 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-14T16:23:54.774 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-14T16:23:54.774 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-14T16:23:54.774 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-14T16:23:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-14T16:23:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-14T16:23:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-14T16:23:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-14T16:23:54.775 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-14T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-14T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-14T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-14T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-14T16:23:54.776 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-14T16:23:54.777 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-14T16:23:54.777 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-14T16:23:54.777 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-14T16:23:54.777 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-14T16:23:54.778 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-14T16:23:54.778 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-14T16:23:54.778 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-14T16:23:54.778 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-14T16:23:54.779 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-14T16:23:54.779 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-14T16:23:54.779 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-14T16:23:54.779 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-14T16:23:54.779 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-14T16:23:54.780 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-14T16:23:54.780 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-14T16:23:54.780 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-14T16:23:54.780 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-14T16:23:54.780 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-14T16:23:54.781 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-14T16:23:54.782 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-14T16:23:54.782 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-14T16:23:54.782 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-14T16:23:54.782 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-14T16:23:54.782 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-14T16:23:54.783 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-14T16:23:54.783 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-14T16:23:54.783 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-14T16:23:54.783 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-14T16:23:54.784 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-14T16:23:54.784 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-14T16:23:54.784 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-14T16:23:54.784 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-14T16:23:54.784 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-14T16:23:54.785 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-14T16:23:54.785 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-14T16:23:54.785 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-14T16:23:54.785 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-14T16:23:54.785 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-14T16:23:54.786 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-14T16:23:54.786 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-14T16:23:54.786 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-14T16:23:54.786 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-14T16:23:54.786 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-14T16:23:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-14T16:23:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-14T16:23:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-14T16:23:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-14T16:23:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-14T16:23:54.788 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-14T16:23:54.788 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-14T16:23:54.788 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-14T16:23:54.788 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-14T16:23:54.789 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-14T16:23:54.789 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-14T16:23:54.789 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-14T16:23:54.789 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-14T16:23:54.790 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-14T16:23:54.790 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-14T16:23:54.790 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-14T16:23:54.790 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-14T16:23:54.791 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-14T16:23:54.791 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-14T16:23:54.791 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-14T16:23:54.791 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-14T16:23:54.791 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-14T16:23:54.792 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-14T16:23:54.792 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-14T16:23:54.792 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-14T16:23:54.792 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-14T16:23:54.792 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-14T16:23:54.793 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-14T16:23:54.793 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-14T16:23:54.793 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-14T16:23:54.793 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-14T16:23:54.793 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-14T16:23:54.794 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-14T16:23:54.794 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-14T16:23:54.794 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-14T16:23:54.794 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-14T16:23:54.795 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-14T16:23:54.795 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-14T16:23:54.795 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-14T16:23:54.795 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-14T16:23:54.795 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-14T16:23:54.796 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-14T16:23:54.796 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-14T16:23:54.796 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-14T16:23:54.796 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-14T16:23:54.797 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-14T16:23:54.797 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-14T16:23:54.797 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-14T16:23:54.797 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-14T16:23:54.797 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-14T16:23:54.798 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-14T16:23:54.798 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-14T16:23:54.798 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-14T16:23:54.798 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-14T16:23:54.798 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-14T16:23:54.799 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-14T16:23:54.799 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-14T16:23:54.799 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-14T16:23:54.799 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-14T16:23:54.799 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-14T16:23:54.800 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-14T16:23:54.800 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-14T16:23:54.800 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-14T16:23:54.800 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-14T16:23:54.801 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-14T16:23:54.801 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-14T16:23:54.801 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-14T16:23:54.801 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-14T16:23:54.801 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-14T16:23:54.802 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-14T16:23:54.802 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-14T16:23:54.802 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-14T16:23:54.802 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-14T16:23:54.803 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-14T16:23:54.803 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-14T16:23:54.803 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-14T16:23:54.803 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-14T16:23:54.803 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-14T16:23:54.804 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-14T16:23:54.804 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-14T16:23:54.804 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-14T16:23:54.804 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-14T16:23:54.804 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-14T16:23:54.805 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-14T16:23:54.805 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-14T16:23:54.805 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-14T16:23:54.805 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-14T16:23:54.806 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-14T16:23:54.806 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-14T16:23:54.806 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-14T16:23:54.806 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-14T16:23:54.806 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-14T16:23:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-14T16:23:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-14T16:23:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-14T16:23:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-14T16:23:54.807 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-14T16:23:54.808 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-14T16:23:54.808 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-14T16:23:54.808 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-14T16:23:54.808 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-14T16:23:54.808 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-14T16:23:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-14T16:23:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-14T16:23:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-14T16:23:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-14T16:23:54.809 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-14T16:23:54.810 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-14T16:23:54.810 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-14T16:23:54.810 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-14T16:23:54.810 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-14T16:23:54.811 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-14T16:23:54.811 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-14T16:23:54.811 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-14T16:23:54.811 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-14T16:23:54.811 INFO:teuthology.orchestra.run.smithi186.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-14T16:23:54.812 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.813 DEBUG:teuthology.orchestra.run.smithi186:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-14T16:23:54.838 INFO:teuthology.orchestra.run.smithi186.stdout:postinstall scriptlet (using /bin/sh): 2023-10-14T16:23:54.839 INFO:teuthology.orchestra.run.smithi186.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-14T16:23:54.839 INFO:teuthology.orchestra.run.smithi186.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:23:54.839 INFO:teuthology.orchestra.run.smithi186.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:23:54.839 INFO:teuthology.orchestra.run.smithi186.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.839 INFO:teuthology.orchestra.run.smithi186.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:23:54.840 INFO:teuthology.orchestra.run.smithi186.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:23:54.840 INFO:teuthology.orchestra.run.smithi186.stdout:fi 2023-10-14T16:23:54.840 INFO:teuthology.orchestra.run.smithi186.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-14T16:23:54.840 INFO:teuthology.orchestra.run.smithi186.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-14T16:23:54.840 INFO:teuthology.orchestra.run.smithi186.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-14T16:23:54.841 INFO:teuthology.orchestra.run.smithi186.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-14T16:23:54.841 INFO:teuthology.orchestra.run.smithi186.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.841 INFO:teuthology.orchestra.run.smithi186.stdout:fi 2023-10-14T16:23:54.841 INFO:teuthology.orchestra.run.smithi186.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-14T16:23:54.841 INFO:teuthology.orchestra.run.smithi186.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-14T16:23:54.842 INFO:teuthology.orchestra.run.smithi186.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.842 INFO:teuthology.orchestra.run.smithi186.stdout:fi 2023-10-14T16:23:54.843 DEBUG:teuthology.orchestra.run.smithi186:> sudo rpm -qi grub2-tools 2023-10-14T16:23:54.921 INFO:teuthology.orchestra.run.smithi186.stdout:Name : grub2-tools 2023-10-14T16:23:54.922 INFO:teuthology.orchestra.run.smithi186.stdout:Epoch : 1 2023-10-14T16:23:54.922 INFO:teuthology.orchestra.run.smithi186.stdout:Version : 2.06 2023-10-14T16:23:54.922 INFO:teuthology.orchestra.run.smithi186.stdout:Release : 68.el9 2023-10-14T16:23:54.922 INFO:teuthology.orchestra.run.smithi186.stdout:Architecture: x86_64 2023-10-14T16:23:54.923 INFO:teuthology.orchestra.run.smithi186.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-14T16:23:54.923 INFO:teuthology.orchestra.run.smithi186.stdout:Group : Unspecified 2023-10-14T16:23:54.923 INFO:teuthology.orchestra.run.smithi186.stdout:Size : 8264254 2023-10-14T16:23:54.923 INFO:teuthology.orchestra.run.smithi186.stdout:License : GPLv3+ 2023-10-14T16:23:54.923 INFO:teuthology.orchestra.run.smithi186.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-14T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-14T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-14T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-14T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Packager : builder@centos.org 2023-10-14T16:23:54.924 INFO:teuthology.orchestra.run.smithi186.stdout:Vendor : CentOS 2023-10-14T16:23:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-14T16:23:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:Summary : Support tools for GRUB. 2023-10-14T16:23:54.925 INFO:teuthology.orchestra.run.smithi186.stdout:Description : 2023-10-14T16:23:54.926 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:23:54.926 INFO:teuthology.orchestra.run.smithi186.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-14T16:23:54.926 INFO:teuthology.orchestra.run.smithi186.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-14T16:23:54.926 INFO:teuthology.orchestra.run.smithi186.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-14T16:23:54.926 INFO:teuthology.orchestra.run.smithi186.stdout:hardware devices. 2023-10-14T16:23:54.927 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:23:54.927 INFO:teuthology.orchestra.run.smithi186.stdout:This subpackage provides tools for support of all platforms. 2023-10-14T16:23:54.928 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-14T16:23:54.928 INFO:teuthology.task.kernel:Updating grub on smithi186 to boot 6.6.0-rc5-g47672119149b 2023-10-14T16:23:54.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-14T16:23:55.613 INFO:teuthology.orchestra.run.smithi186.stderr:Generating grub configuration file ... 2023-10-14T16:23:56.970 INFO:teuthology.orchestra.run.smithi186.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-14T16:23:57.000 INFO:teuthology.orchestra.run.smithi186.stderr:done 2023-10-14T16:23:57.002 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2023-10-14T16:23:57.017 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.p6lPgoOs72 2023-10-14T16:23:57.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p6lPgoOs72 2023-10-14T16:23:57.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.p6lPgoOs72 2023-10-14T16:23:57.228 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.p6lPgoOs72 is 6KB 2023-10-14T16:23:57.276 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.p6lPgoOs72 2023-10-14T16:23:57.291 DEBUG:teuthology.orchestra.run.smithi186:> sudo /bin/ls /boot/loader/entries || true 2023-10-14T16:23:57.364 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-14T16:23:57.364 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-14T16:23:57.364 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-14T16:23:57.365 INFO:teuthology.orchestra.run.smithi186.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-14T16:23:57.366 DEBUG:teuthology.orchestra.run.smithi186:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-14T16:23:57.556 DEBUG:teuthology.orchestra.run.smithi186:> sudo shutdown -r now 2023-10-14T16:24:02.784 INFO:teuthology.orchestra.run.smithi029.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2023-10-14T16:24:26.308 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi029.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2023-10-14T16:24:26.309 DEBUG:teuthology.orchestra.run.smithi029:> rpm -qlp /tmp/kernel.x86_64.rpm 2023-10-14T16:24:26.404 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/System.map-6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.404 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/config-6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.404 INFO:teuthology.orchestra.run.smithi029.stdout:/boot/vmlinuz-6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.405 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.405 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel 2023-10-14T16:24:26.405 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch 2023-10-14T16:24:26.405 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86 2023-10-14T16:24:26.405 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto 2023-10-14T16:24:26.406 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/blowfish-x86_64.ko 2023-10-14T16:24:26.406 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2023-10-14T16:24:26.406 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2023-10-14T16:24:26.406 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/camellia-x86_64.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32-pclmul.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crc32c-intel.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2023-10-14T16:24:26.407 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2023-10-14T16:24:26.408 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2023-10-14T16:24:26.408 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2023-10-14T16:24:26.408 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-avx2.ko 2023-10-14T16:24:26.408 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2023-10-14T16:24:26.408 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha1-ssse3.ko 2023-10-14T16:24:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha256-ssse3.ko 2023-10-14T16:24:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/sha512-ssse3.ko 2023-10-14T16:24:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2023-10-14T16:24:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2023-10-14T16:24:26.409 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/crypto/twofish-x86_64.ko 2023-10-14T16:24:26.410 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel 2023-10-14T16:24:26.410 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu 2023-10-14T16:24:26.410 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce 2023-10-14T16:24:26.410 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2023-10-14T16:24:26.410 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm 2023-10-14T16:24:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-amd.ko 2023-10-14T16:24:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm-intel.ko 2023-10-14T16:24:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/arch/x86/kvm/kvm.ko 2023-10-14T16:24:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto 2023-10-14T16:24:26.411 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ansi_cprng.ko 2023-10-14T16:24:26.412 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/anubis.ko 2023-10-14T16:24:26.412 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/arc4.ko 2023-10-14T16:24:26.412 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx 2023-10-14T16:24:26.412 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_memcpy.ko 2023-10-14T16:24:26.412 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_pq.ko 2023-10-14T16:24:26.413 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_raid6_recov.ko 2023-10-14T16:24:26.413 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_tx.ko 2023-10-14T16:24:26.413 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/async_xor.ko 2023-10-14T16:24:26.413 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/async_tx/raid6test.ko 2023-10-14T16:24:26.413 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authenc.ko 2023-10-14T16:24:26.414 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/authencesn.ko 2023-10-14T16:24:26.414 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blake2b_generic.ko 2023-10-14T16:24:26.414 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_common.ko 2023-10-14T16:24:26.414 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/blowfish_generic.ko 2023-10-14T16:24:26.414 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/camellia_generic.ko 2023-10-14T16:24:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast5_generic.ko 2023-10-14T16:24:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast6_generic.ko 2023-10-14T16:24:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cast_common.ko 2023-10-14T16:24:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/ccm.ko 2023-10-14T16:24:26.415 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cmac.ko 2023-10-14T16:24:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crc32_generic.ko 2023-10-14T16:24:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_engine.ko 2023-10-14T16:24:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/crypto_user.ko 2023-10-14T16:24:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/cts.ko 2023-10-14T16:24:26.416 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/des_generic.ko 2023-10-14T16:24:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/dh_generic.ko 2023-10-14T16:24:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/echainiv.ko 2023-10-14T16:24:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/essiv.ko 2023-10-14T16:24:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/fcrypt.ko 2023-10-14T16:24:26.417 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/khazad.ko 2023-10-14T16:24:26.418 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4.ko 2023-10-14T16:24:26.418 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/lz4hc.ko 2023-10-14T16:24:26.418 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/md4.ko 2023-10-14T16:24:26.418 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/michael_mic.ko 2023-10-14T16:24:26.418 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcbc.ko 2023-10-14T16:24:26.419 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/pcrypt.ko 2023-10-14T16:24:26.419 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/rmd160.ko 2023-10-14T16:24:26.419 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/seed.ko 2023-10-14T16:24:26.419 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/serpent_generic.ko 2023-10-14T16:24:26.419 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tcrypt.ko 2023-10-14T16:24:26.420 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/tea.ko 2023-10-14T16:24:26.420 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_common.ko 2023-10-14T16:24:26.420 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/twofish_generic.ko 2023-10-14T16:24:26.420 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/vmac.ko 2023-10-14T16:24:26.420 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/wp512.ko 2023-10-14T16:24:26.421 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xcbc.ko 2023-10-14T16:24:26.421 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xor.ko 2023-10-14T16:24:26.421 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/crypto/xxhash_generic.ko 2023-10-14T16:24:26.421 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers 2023-10-14T16:24:26.421 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi 2023-10-14T16:24:26.422 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_ipmi.ko 2023-10-14T16:24:26.422 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/acpi_pad.ko 2023-10-14T16:24:26.422 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/custom_method.ko 2023-10-14T16:24:26.422 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/ec_sys.ko 2023-10-14T16:24:26.422 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/platform_profile.ko 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbs.ko 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/sbshc.ko 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/acpi/video.ko 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/acard-ahci.ko 2023-10-14T16:24:26.423 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ahci_platform.ko 2023-10-14T16:24:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/ata_generic.ko 2023-10-14T16:24:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/libahci_platform.ko 2023-10-14T16:24:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_acpi.ko 2023-10-14T16:24:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ali.ko 2023-10-14T16:24:26.424 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_amd.ko 2023-10-14T16:24:26.425 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_artop.ko 2023-10-14T16:24:26.425 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atiixp.ko 2023-10-14T16:24:26.425 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_atp867x.ko 2023-10-14T16:24:26.425 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd640.ko 2023-10-14T16:24:26.425 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cmd64x.ko 2023-10-14T16:24:26.426 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_cypress.ko 2023-10-14T16:24:26.426 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_efar.ko 2023-10-14T16:24:26.426 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt366.ko 2023-10-14T16:24:26.426 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt37x.ko 2023-10-14T16:24:26.426 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x2n.ko 2023-10-14T16:24:26.427 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_hpt3x3.ko 2023-10-14T16:24:26.427 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it8213.ko 2023-10-14T16:24:26.427 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_it821x.ko 2023-10-14T16:24:26.427 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_jmicron.ko 2023-10-14T16:24:26.428 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_marvell.ko 2023-10-14T16:24:26.428 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_mpiix.ko 2023-10-14T16:24:26.428 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_netcell.ko 2023-10-14T16:24:26.428 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ninja32.ko 2023-10-14T16:24:26.428 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87410.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_ns87415.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_oldpiix.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_opti.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_optidma.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pcmcia.ko 2023-10-14T16:24:26.429 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc2027x.ko 2023-10-14T16:24:26.430 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_pdc202xx_old.ko 2023-10-14T16:24:26.430 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_piccolo.ko 2023-10-14T16:24:26.430 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_rdc.ko 2023-10-14T16:24:26.430 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sch.ko 2023-10-14T16:24:26.430 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_serverworks.ko 2023-10-14T16:24:26.431 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sil680.ko 2023-10-14T16:24:26.431 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sis.ko 2023-10-14T16:24:26.431 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_sl82c105.ko 2023-10-14T16:24:26.431 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_triflex.ko 2023-10-14T16:24:26.431 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pata_via.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/pdc_adma.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_inic162x.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_mv.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_nv.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_promise.ko 2023-10-14T16:24:26.432 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_qstor.ko 2023-10-14T16:24:26.433 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil.ko 2023-10-14T16:24:26.433 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sil24.ko 2023-10-14T16:24:26.433 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sis.ko 2023-10-14T16:24:26.433 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_svw.ko 2023-10-14T16:24:26.433 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_sx4.ko 2023-10-14T16:24:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_uli.ko 2023-10-14T16:24:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_via.ko 2023-10-14T16:24:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ata/sata_vsc.ko 2023-10-14T16:24:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay 2023-10-14T16:24:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/auxdisplay/ks0108.ko 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-i2c.ko 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/base/regmap/regmap-mmio.ko 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma 2023-10-14T16:24:26.435 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/bcma/bcma.ko 2023-10-14T16:24:26.436 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block 2023-10-14T16:24:26.436 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe 2023-10-14T16:24:26.436 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/aoe/aoe.ko 2023-10-14T16:24:26.436 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/brd.ko 2023-10-14T16:24:26.436 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd 2023-10-14T16:24:26.437 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/drbd/drbd.ko 2023-10-14T16:24:26.437 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/floppy.ko 2023-10-14T16:24:26.437 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/loop.ko 2023-10-14T16:24:26.437 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx 2023-10-14T16:24:26.437 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/mtip32xx/mtip32xx.ko 2023-10-14T16:24:26.438 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/nbd.ko 2023-10-14T16:24:26.438 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk 2023-10-14T16:24:26.438 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/null_blk/null_blk.ko 2023-10-14T16:24:26.438 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/pktcdvd.ko 2023-10-14T16:24:26.438 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/rbd.ko 2023-10-14T16:24:26.439 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/virtio_blk.ko 2023-10-14T16:24:26.439 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback 2023-10-14T16:24:26.439 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkback/xen-blkback.ko 2023-10-14T16:24:26.439 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/xen-blkfront.ko 2023-10-14T16:24:26.439 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram 2023-10-14T16:24:26.440 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/block/zram/zram.ko 2023-10-14T16:24:26.440 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char 2023-10-14T16:24:26.440 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hangcheck-timer.ko 2023-10-14T16:24:26.440 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random 2023-10-14T16:24:26.440 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/amd-rng.ko 2023-10-14T16:24:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/intel-rng.ko 2023-10-14T16:24:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/timeriomem-rng.ko 2023-10-14T16:24:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/via-rng.ko 2023-10-14T16:24:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/hw_random/virtio-rng.ko 2023-10-14T16:24:26.441 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi 2023-10-14T16:24:26.442 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_devintf.ko 2023-10-14T16:24:26.442 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2023-10-14T16:24:26.442 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2023-10-14T16:24:26.442 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_si.ko 2023-10-14T16:24:26.442 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_ssif.ko 2023-10-14T16:24:26.443 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2023-10-14T16:24:26.443 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/lp.ko 2023-10-14T16:24:26.443 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave 2023-10-14T16:24:26.443 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/mwave/mwave.ko 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/ppdev.ko 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tlclk.ko 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm.ko 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_atmel.ko 2023-10-14T16:24:26.444 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_infineon.ko 2023-10-14T16:24:26.445 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_nsc.ko 2023-10-14T16:24:26.445 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis.ko 2023-10-14T16:24:26.445 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/tpm/tpm_tis_core.ko 2023-10-14T16:24:26.445 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/uv_mmtimer.ko 2023-10-14T16:24:26.445 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/char/virtio_console.ko 2023-10-14T16:24:26.446 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq 2023-10-14T16:24:26.446 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/acpi-cpufreq.ko 2023-10-14T16:24:26.446 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2023-10-14T16:24:26.446 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/p4-clockmod.ko 2023-10-14T16:24:26.446 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/pcc-cpufreq.ko 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/powernow-k8.ko 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/cpufreq/speedstep-lib.ko 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp-crypto.ko 2023-10-14T16:24:26.447 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/ccp/ccp.ko 2023-10-14T16:24:26.448 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel 2023-10-14T16:24:26.448 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat 2023-10-14T16:24:26.448 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common 2023-10-14T16:24:26.448 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2023-10-14T16:24:26.448 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2023-10-14T16:24:26.449 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2023-10-14T16:24:26.449 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-aes.ko 2023-10-14T16:24:26.449 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/padlock-sha.ko 2023-10-14T16:24:26.449 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio 2023-10-14T16:24:26.449 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/crypto/virtio/virtio_crypto.ko 2023-10-14T16:24:26.450 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca 2023-10-14T16:24:26.450 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dca/dca.ko 2023-10-14T16:24:26.450 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq 2023-10-14T16:24:26.450 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/devfreq/governor_simpleondemand.ko 2023-10-14T16:24:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma 2023-10-14T16:24:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw 2023-10-14T16:24:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/dw/dw_dmac.ko 2023-10-14T16:24:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat 2023-10-14T16:24:26.451 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/dma/ioat/ioatdma.ko 2023-10-14T16:24:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac 2023-10-14T16:24:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/amd64_edac.ko 2023-10-14T16:24:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/e752x_edac.ko 2023-10-14T16:24:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/edac_mce_amd.ko 2023-10-14T16:24:26.452 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3000_edac.ko 2023-10-14T16:24:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i3200_edac.ko 2023-10-14T16:24:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5100_edac.ko 2023-10-14T16:24:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i5400_edac.ko 2023-10-14T16:24:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7300_edac.ko 2023-10-14T16:24:26.453 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i7core_edac.ko 2023-10-14T16:24:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/i82975x_edac.ko 2023-10-14T16:24:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/ie31200_edac.ko 2023-10-14T16:24:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/sb_edac.ko 2023-10-14T16:24:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/edac/x38_edac.ko 2023-10-14T16:24:26.454 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware 2023-10-14T16:24:26.455 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/edd.ko 2023-10-14T16:24:26.455 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/firmware/iscsi_ibft.ko 2023-10-14T16:24:26.455 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio 2023-10-14T16:24:26.455 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-ich.ko 2023-10-14T16:24:26.455 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/gpio/gpio-viperboard.ko 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-appleir.ko 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-aureal.ko 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-axff.ko 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-dr.ko 2023-10-14T16:24:26.456 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elecom.ko 2023-10-14T16:24:26.457 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-elo.ko 2023-10-14T16:24:26.457 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-emsff.ko 2023-10-14T16:24:26.457 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gaff.ko 2023-10-14T16:24:26.457 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gt683r.ko 2023-10-14T16:24:26.458 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-gyration.ko 2023-10-14T16:24:26.458 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-kbd.ko 2023-10-14T16:24:26.458 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtek-mouse.ko 2023-10-14T16:24:26.458 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-holtekff.ko 2023-10-14T16:24:26.458 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-hyperv.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-icade.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-keytouch.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-kye.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lcpower.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-led.ko 2023-10-14T16:24:26.459 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-lenovo.ko 2023-10-14T16:24:26.460 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-dj.ko 2023-10-14T16:24:26.460 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-logitech-hidpp.ko 2023-10-14T16:24:26.461 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-multitouch.ko 2023-10-14T16:24:26.461 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-ortek.ko 2023-10-14T16:24:26.461 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-penmount.ko 2023-10-14T16:24:26.462 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-petalynx.ko 2023-10-14T16:24:26.462 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-picolcd.ko 2023-10-14T16:24:26.462 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-pl.ko 2023-10-14T16:24:26.462 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-plantronics.ko 2023-10-14T16:24:26.463 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-primax.ko 2023-10-14T16:24:26.463 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-rmi.ko 2023-10-14T16:24:26.463 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-arvo.ko 2023-10-14T16:24:26.463 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-common.ko 2023-10-14T16:24:26.464 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-isku.ko 2023-10-14T16:24:26.464 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kone.ko 2023-10-14T16:24:26.464 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-koneplus.ko 2023-10-14T16:24:26.465 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-konepure.ko 2023-10-14T16:24:26.465 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-kovaplus.ko 2023-10-14T16:24:26.466 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-lua.ko 2023-10-14T16:24:26.466 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-pyra.ko 2023-10-14T16:24:26.466 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-ryos.ko 2023-10-14T16:24:26.467 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat-savu.ko 2023-10-14T16:24:26.467 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-roccat.ko 2023-10-14T16:24:26.467 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-saitek.ko 2023-10-14T16:24:26.468 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-samsung.ko 2023-10-14T16:24:26.468 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sensor-hub.ko 2023-10-14T16:24:26.468 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sjoy.ko 2023-10-14T16:24:26.469 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sony.ko 2023-10-14T16:24:26.469 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-speedlink.ko 2023-10-14T16:24:26.469 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-steelseries.ko 2023-10-14T16:24:26.469 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-sunplus.ko 2023-10-14T16:24:26.469 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-thrustmaster.ko 2023-10-14T16:24:26.470 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tivo.ko 2023-10-14T16:24:26.470 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-tmff.ko 2023-10-14T16:24:26.470 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-topseed.ko 2023-10-14T16:24:26.470 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-twinhan.ko 2023-10-14T16:24:26.471 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-uclogic.ko 2023-10-14T16:24:26.471 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-waltop.ko 2023-10-14T16:24:26.471 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-wiimote.ko 2023-10-14T16:24:26.471 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-xinmo.ko 2023-10-14T16:24:26.472 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zpff.ko 2023-10-14T16:24:26.472 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/hid-zydacron.ko 2023-10-14T16:24:26.472 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/uhid.ko 2023-10-14T16:24:26.474 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hid/wacom.ko 2023-10-14T16:24:26.474 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv 2023-10-14T16:24:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_balloon.ko 2023-10-14T16:24:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_utils.ko 2023-10-14T16:24:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hv/hv_vmbus.ko 2023-10-14T16:24:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon 2023-10-14T16:24:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru.ko 2023-10-14T16:24:26.476 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/abituguru3.ko 2023-10-14T16:24:26.476 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/acpi_power_meter.ko 2023-10-14T16:24:26.476 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7414.ko 2023-10-14T16:24:26.476 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ad7418.ko 2023-10-14T16:24:26.476 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adc128d818.ko 2023-10-14T16:24:26.477 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1025.ko 2023-10-14T16:24:26.477 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1026.ko 2023-10-14T16:24:26.477 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1029.ko 2023-10-14T16:24:26.477 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm1031.ko 2023-10-14T16:24:26.477 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adm9240.ko 2023-10-14T16:24:26.478 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ads7828.ko 2023-10-14T16:24:26.478 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7410.ko 2023-10-14T16:24:26.478 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7411.ko 2023-10-14T16:24:26.478 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7462.ko 2023-10-14T16:24:26.479 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7470.ko 2023-10-14T16:24:26.479 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7475.ko 2023-10-14T16:24:26.479 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/adt7x10.ko 2023-10-14T16:24:26.479 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/amc6821.ko 2023-10-14T16:24:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/applesmc.ko 2023-10-14T16:24:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asb100.ko 2023-10-14T16:24:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asc7621.ko 2023-10-14T16:24:26.480 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/asus_atk0110.ko 2023-10-14T16:24:26.481 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/atxp1.ko 2023-10-14T16:24:26.481 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/coretemp.ko 2023-10-14T16:24:26.481 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dell-smm-hwmon.ko 2023-10-14T16:24:26.481 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/dme1737.ko 2023-10-14T16:24:26.481 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds1621.ko 2023-10-14T16:24:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ds620.ko 2023-10-14T16:24:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc1403.ko 2023-10-14T16:24:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/emc6w201.ko 2023-10-14T16:24:26.482 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71805f.ko 2023-10-14T16:24:26.483 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f71882fg.ko 2023-10-14T16:24:26.483 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/f75375s.ko 2023-10-14T16:24:26.483 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fam15h_power.ko 2023-10-14T16:24:26.483 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/fschmd.ko 2023-10-14T16:24:26.483 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g760a.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/g762.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl518sm.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/gl520sm.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/hwmon-vid.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5500_temp.ko 2023-10-14T16:24:26.484 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/i5k_amb.ko 2023-10-14T16:24:26.485 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmaem.ko 2023-10-14T16:24:26.485 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ibmpex.ko 2023-10-14T16:24:26.485 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina209.ko 2023-10-14T16:24:26.485 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ina2xx.ko 2023-10-14T16:24:26.485 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/it87.ko 2023-10-14T16:24:26.486 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k10temp.ko 2023-10-14T16:24:26.486 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/k8temp.ko 2023-10-14T16:24:26.486 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lineage-pem.ko 2023-10-14T16:24:26.486 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm63.ko 2023-10-14T16:24:26.486 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm73.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm75.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm77.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm78.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm80.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm83.ko 2023-10-14T16:24:26.487 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm85.ko 2023-10-14T16:24:26.488 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm87.ko 2023-10-14T16:24:26.488 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm90.ko 2023-10-14T16:24:26.488 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm92.ko 2023-10-14T16:24:26.488 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm93.ko 2023-10-14T16:24:26.488 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95234.ko 2023-10-14T16:24:26.489 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95241.ko 2023-10-14T16:24:26.489 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/lm95245.ko 2023-10-14T16:24:26.489 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc2945.ko 2023-10-14T16:24:26.489 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4151.ko 2023-10-14T16:24:26.490 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4215.ko 2023-10-14T16:24:26.490 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4222.ko 2023-10-14T16:24:26.490 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4245.ko 2023-10-14T16:24:26.490 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4260.ko 2023-10-14T16:24:26.491 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/ltc4261.ko 2023-10-14T16:24:26.491 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max16065.ko 2023-10-14T16:24:26.491 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1619.ko 2023-10-14T16:24:26.491 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max1668.ko 2023-10-14T16:24:26.492 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max197.ko 2023-10-14T16:24:26.492 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6639.ko 2023-10-14T16:24:26.492 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6650.ko 2023-10-14T16:24:26.492 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/max6697.ko 2023-10-14T16:24:26.492 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/mcp3021.ko 2023-10-14T16:24:26.493 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6683.ko 2023-10-14T16:24:26.493 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775-core.ko 2023-10-14T16:24:26.493 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct6775.ko 2023-10-14T16:24:26.493 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/nct7802.ko 2023-10-14T16:24:26.493 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87360.ko 2023-10-14T16:24:26.494 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pc87427.ko 2023-10-14T16:24:26.494 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pcf8591.ko 2023-10-14T16:24:26.494 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus 2023-10-14T16:24:26.494 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/adm1275.ko 2023-10-14T16:24:26.494 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/lm25066.ko 2023-10-14T16:24:26.495 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ltc2978.ko 2023-10-14T16:24:26.495 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max16064.ko 2023-10-14T16:24:26.495 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max34440.ko 2023-10-14T16:24:26.495 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/max8688.ko 2023-10-14T16:24:26.495 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus.ko 2023-10-14T16:24:26.496 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2023-10-14T16:24:26.496 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/tps40422.ko 2023-10-14T16:24:26.496 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9000.ko 2023-10-14T16:24:26.496 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/ucd9200.ko 2023-10-14T16:24:26.496 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/pmbus/zl6100.ko 2023-10-14T16:24:26.497 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/powr1220.ko 2023-10-14T16:24:26.497 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5627.ko 2023-10-14T16:24:26.497 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch5636.ko 2023-10-14T16:24:26.497 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sch56xx-common.ko 2023-10-14T16:24:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht15.ko 2023-10-14T16:24:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sht21.ko 2023-10-14T16:24:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/shtc1.ko 2023-10-14T16:24:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/sis5595.ko 2023-10-14T16:24:26.498 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47b397.ko 2023-10-14T16:24:26.499 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m1.ko 2023-10-14T16:24:26.499 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/smsc47m192.ko 2023-10-14T16:24:26.499 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/thmc50.ko 2023-10-14T16:24:26.499 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp102.ko 2023-10-14T16:24:26.499 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp103.ko 2023-10-14T16:24:26.500 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp401.ko 2023-10-14T16:24:26.500 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/tmp421.ko 2023-10-14T16:24:26.500 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via-cputemp.ko 2023-10-14T16:24:26.500 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/via686a.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt1211.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/vt8231.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627ehf.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83627hf.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83781d.ko 2023-10-14T16:24:26.501 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83791d.ko 2023-10-14T16:24:26.502 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83792d.ko 2023-10-14T16:24:26.502 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83793.ko 2023-10-14T16:24:26.502 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83795.ko 2023-10-14T16:24:26.502 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l785ts.ko 2023-10-14T16:24:26.502 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/hwmon/w83l786ng.ko 2023-10-14T16:24:26.503 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c 2023-10-14T16:24:26.503 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos 2023-10-14T16:24:26.503 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2023-10-14T16:24:26.503 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2023-10-14T16:24:26.503 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses 2023-10-14T16:24:26.504 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2023-10-14T16:24:26.504 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd756.ko 2023-10-14T16:24:26.504 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-amd8111.ko 2023-10-14T16:24:26.504 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2023-10-14T16:24:26.505 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-core.ko 2023-10-14T16:24:26.505 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2023-10-14T16:24:26.505 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2023-10-14T16:24:26.505 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2023-10-14T16:24:26.506 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-i801.ko 2023-10-14T16:24:26.506 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-isch.ko 2023-10-14T16:24:26.507 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-ismt.ko 2023-10-14T16:24:26.507 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2023-10-14T16:24:26.507 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-nforce2.ko 2023-10-14T16:24:26.508 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-parport.ko 2023-10-14T16:24:26.508 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2023-10-14T16:24:26.508 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-piix4.ko 2023-10-14T16:24:26.509 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-scmi.ko 2023-10-14T16:24:26.509 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-simtec.ko 2023-10-14T16:24:26.509 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-sis96x.ko 2023-10-14T16:24:26.509 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2023-10-14T16:24:26.510 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-via.ko 2023-10-14T16:24:26.510 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viapro.ko 2023-10-14T16:24:26.510 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/busses/i2c-viperboard.ko 2023-10-14T16:24:26.511 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-dev.ko 2023-10-14T16:24:26.511 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-mux.ko 2023-10-14T16:24:26.511 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-slave-eeprom.ko 2023-10-14T16:24:26.511 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-smbus.ko 2023-10-14T16:24:26.511 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/i2c/i2c-stub.ko 2023-10-14T16:24:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband 2023-10-14T16:24:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core 2023-10-14T16:24:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_cm.ko 2023-10-14T16:24:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_core.ko 2023-10-14T16:24:26.512 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_umad.ko 2023-10-14T16:24:26.513 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/ib_uverbs.ko 2023-10-14T16:24:26.513 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/iw_cm.ko 2023-10-14T16:24:26.513 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_cm.ko 2023-10-14T16:24:26.513 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/core/rdma_ucm.ko 2023-10-14T16:24:26.513 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw 2023-10-14T16:24:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4 2023-10-14T16:24:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2023-10-14T16:24:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4 2023-10-14T16:24:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2023-10-14T16:24:26.514 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5 2023-10-14T16:24:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2023-10-14T16:24:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca 2023-10-14T16:24:26.515 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2023-10-14T16:24:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp 2023-10-14T16:24:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib 2023-10-14T16:24:26.516 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2023-10-14T16:24:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser 2023-10-14T16:24:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2023-10-14T16:24:26.517 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert 2023-10-14T16:24:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2023-10-14T16:24:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp 2023-10-14T16:24:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2023-10-14T16:24:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt 2023-10-14T16:24:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2023-10-14T16:24:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input 2023-10-14T16:24:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport 2023-10-14T16:24:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/emu10k1-gp.ko 2023-10-14T16:24:26.519 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/fm801-gp.ko 2023-10-14T16:24:26.520 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/gameport/gameport.ko 2023-10-14T16:24:26.520 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/joydev.ko 2023-10-14T16:24:26.520 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard 2023-10-14T16:24:26.520 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/keyboard/gpio_keys.ko 2023-10-14T16:24:26.520 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse 2023-10-14T16:24:26.521 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/appletouch.ko 2023-10-14T16:24:26.521 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/bcm5974.ko 2023-10-14T16:24:26.521 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/cyapatp.ko 2023-10-14T16:24:26.521 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/elan_i2c.ko 2023-10-14T16:24:26.522 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/sermouse.ko 2023-10-14T16:24:26.522 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_i2c.ko 2023-10-14T16:24:26.522 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/synaptics_usb.ko 2023-10-14T16:24:26.522 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/mouse/vsxxxaa.ko 2023-10-14T16:24:26.523 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4 2023-10-14T16:24:26.523 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/rmi4/rmi_core.ko 2023-10-14T16:24:26.523 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio 2023-10-14T16:24:26.523 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/altera_ps2.ko 2023-10-14T16:24:26.524 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/arc_ps2.ko 2023-10-14T16:24:26.524 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/hyperv-keyboard.ko 2023-10-14T16:24:26.524 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/serio/serio_raw.ko 2023-10-14T16:24:26.524 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/input/sparse-keymap.ko 2023-10-14T16:24:26.524 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/iommu/amd/iommu_v2.ko 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-blinkm.ko 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lm3530.ko 2023-10-14T16:24:26.525 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-lp3944.ko 2023-10-14T16:24:26.526 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/leds-ss4200.ko 2023-10-14T16:24:26.526 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger 2023-10-14T16:24:26.526 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-audio.ko 2023-10-14T16:24:26.526 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2023-10-14T16:24:26.526 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-camera.ko 2023-10-14T16:24:26.527 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2023-10-14T16:24:26.527 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2023-10-14T16:24:26.527 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2023-10-14T16:24:26.528 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-timer.ko 2023-10-14T16:24:26.528 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/leds/trigger/ledtrig-transient.ko 2023-10-14T16:24:26.528 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md 2023-10-14T16:24:26.528 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache 2023-10-14T16:24:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/bcache/bcache.ko 2023-10-14T16:24:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-bio-prison.ko 2023-10-14T16:24:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache-smq.ko 2023-10-14T16:24:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-cache.ko 2023-10-14T16:24:26.529 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-crypt.ko 2023-10-14T16:24:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-delay.ko 2023-10-14T16:24:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-flakey.ko 2023-10-14T16:24:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-log-userspace.ko 2023-10-14T16:24:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-multipath.ko 2023-10-14T16:24:26.530 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-queue-length.ko 2023-10-14T16:24:26.531 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-raid.ko 2023-10-14T16:24:26.531 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-round-robin.ko 2023-10-14T16:24:26.531 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-service-time.ko 2023-10-14T16:24:26.531 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-switch.ko 2023-10-14T16:24:26.532 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-thin-pool.ko 2023-10-14T16:24:26.532 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/dm-verity.ko 2023-10-14T16:24:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/faulty.ko 2023-10-14T16:24:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/linear.ko 2023-10-14T16:24:26.533 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/multipath.ko 2023-10-14T16:24:26.534 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data 2023-10-14T16:24:26.534 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2023-10-14T16:24:26.534 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid0.ko 2023-10-14T16:24:26.535 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid1.ko 2023-10-14T16:24:26.535 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid10.ko 2023-10-14T16:24:26.535 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/md/raid456.ko 2023-10-14T16:24:26.535 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message 2023-10-14T16:24:26.535 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion 2023-10-14T16:24:26.536 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptbase.ko 2023-10-14T16:24:26.536 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptctl.ko 2023-10-14T16:24:26.536 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptfc.ko 2023-10-14T16:24:26.536 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptlan.ko 2023-10-14T16:24:26.537 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptsas.ko 2023-10-14T16:24:26.537 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptscsih.ko 2023-10-14T16:24:26.537 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/message/fusion/mptspi.ko 2023-10-14T16:24:26.537 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd 2023-10-14T16:24:26.537 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_ich.ko 2023-10-14T16:24:26.538 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/lpc_sch.ko 2023-10-14T16:24:26.538 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/mfd-core.ko 2023-10-14T16:24:26.538 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/sm501.ko 2023-10-14T16:24:26.538 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/viperboard.ko 2023-10-14T16:24:26.538 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/vx855.ko 2023-10-14T16:24:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/mfd/wl1273-core.ko 2023-10-14T16:24:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc 2023-10-14T16:24:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl 2023-10-14T16:24:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2023-10-14T16:24:26.539 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds9802als.ko 2023-10-14T16:24:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/apds990x.ko 2023-10-14T16:24:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/bh1770glc.ko 2023-10-14T16:24:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710 2023-10-14T16:24:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/cb710/cb710.ko 2023-10-14T16:24:26.540 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom 2023-10-14T16:24:26.541 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/at24.ko 2023-10-14T16:24:26.541 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom.ko 2023-10-14T16:24:26.541 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2023-10-14T16:24:26.541 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/eeprom/max6875.ko 2023-10-14T16:24:26.541 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/enclosure.ko 2023-10-14T16:24:26.542 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/hpilo.ko 2023-10-14T16:24:26.542 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29003.ko 2023-10-14T16:24:26.542 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/isl29020.ko 2023-10-14T16:24:26.542 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d 2023-10-14T16:24:26.543 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2023-10-14T16:24:26.543 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2023-10-14T16:24:26.543 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei 2023-10-14T16:24:26.543 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-me.ko 2023-10-14T16:24:26.544 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei-txe.ko 2023-10-14T16:24:26.544 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/mei/mei.ko 2023-10-14T16:24:26.544 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru 2023-10-14T16:24:26.544 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-gru/gru.ko 2023-10-14T16:24:26.545 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp 2023-10-14T16:24:26.545 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xp.ko 2023-10-14T16:24:26.545 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpc.ko 2023-10-14T16:24:26.545 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/sgi-xp/xpnet.ko 2023-10-14T16:24:26.546 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_7xx1.ko 2023-10-14T16:24:26.546 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tifm_core.ko 2023-10-14T16:24:26.546 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/tsl2550.ko 2023-10-14T16:24:26.546 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_balloon.ko 2023-10-14T16:24:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci 2023-10-14T16:24:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2023-10-14T16:24:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net 2023-10-14T16:24:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding 2023-10-14T16:24:26.547 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/bonding/bonding.ko 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/b53/b53_common.ko 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/bcm-sf2.ko 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6060.ko 2023-10-14T16:24:26.548 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx 2023-10-14T16:24:26.549 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2023-10-14T16:24:26.549 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/dummy.ko 2023-10-14T16:24:26.549 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/eql.ko 2023-10-14T16:24:26.549 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet 2023-10-14T16:24:26.549 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com 2023-10-14T16:24:26.550 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2023-10-14T16:24:26.550 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2023-10-14T16:24:26.550 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/3c59x.ko 2023-10-14T16:24:26.550 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/3com/typhoon.ko 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/8390.ko 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2023-10-14T16:24:26.551 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/adaptec/starfire.ko 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/agere/et131x.ko 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/alteon/acenic.ko 2023-10-14T16:24:26.552 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera 2023-10-14T16:24:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/altera/altera_tse.ko 2023-10-14T16:24:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd 2023-10-14T16:24:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/amd8111e.ko 2023-10-14T16:24:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2023-10-14T16:24:26.553 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/amd/pcnet32.ko 2023-10-14T16:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros 2023-10-14T16:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx 2023-10-14T16:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2023-10-14T16:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c 2023-10-14T16:24:26.554 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2023-10-14T16:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e 2023-10-14T16:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2023-10-14T16:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx 2023-10-14T16:24:26.555 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2023-10-14T16:24:26.556 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2023-10-14T16:24:26.556 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom 2023-10-14T16:24:26.556 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/b44.ko 2023-10-14T16:24:26.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2023-10-14T16:24:26.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x 2023-10-14T16:24:26.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2023-10-14T16:24:26.557 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/cnic.ko 2023-10-14T16:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet 2023-10-14T16:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2023-10-14T16:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/broadcom/tg3.ko 2023-10-14T16:24:26.558 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade 2023-10-14T16:24:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna 2023-10-14T16:24:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2023-10-14T16:24:26.559 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio 2023-10-14T16:24:26.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb 2023-10-14T16:24:26.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2023-10-14T16:24:26.560 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3 2023-10-14T16:24:26.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2023-10-14T16:24:26.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4 2023-10-14T16:24:26.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2023-10-14T16:24:26.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2023-10-14T16:24:26.561 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2023-10-14T16:24:26.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb 2023-10-14T16:24:26.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2023-10-14T16:24:26.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco 2023-10-14T16:24:26.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic 2023-10-14T16:24:26.562 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2023-10-14T16:24:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec 2023-10-14T16:24:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip 2023-10-14T16:24:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2023-10-14T16:24:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2023-10-14T16:24:26.563 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2023-10-14T16:24:26.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2023-10-14T16:24:26.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2023-10-14T16:24:26.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2023-10-14T16:24:26.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink 2023-10-14T16:24:26.564 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/dl2k.ko 2023-10-14T16:24:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dlink/sundance.ko 2023-10-14T16:24:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/dnet.ko 2023-10-14T16:24:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex 2023-10-14T16:24:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet 2023-10-14T16:24:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2023-10-14T16:24:26.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ethoc.ko 2023-10-14T16:24:26.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/fealnx.ko 2023-10-14T16:24:26.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel 2023-10-14T16:24:26.566 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e100.ko 2023-10-14T16:24:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000 2023-10-14T16:24:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2023-10-14T16:24:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e 2023-10-14T16:24:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2023-10-14T16:24:26.567 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k 2023-10-14T16:24:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2023-10-14T16:24:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e 2023-10-14T16:24:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2023-10-14T16:24:26.568 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf 2023-10-14T16:24:26.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2023-10-14T16:24:26.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb 2023-10-14T16:24:26.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igb/igb.ko 2023-10-14T16:24:26.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf 2023-10-14T16:24:26.569 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2023-10-14T16:24:26.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe 2023-10-14T16:24:26.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2023-10-14T16:24:26.570 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf 2023-10-14T16:24:26.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2023-10-14T16:24:26.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/jme.ko 2023-10-14T16:24:26.571 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell 2023-10-14T16:24:26.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2023-10-14T16:24:26.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/skge.ko 2023-10-14T16:24:26.572 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/marvell/sky2.ko 2023-10-14T16:24:26.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox 2023-10-14T16:24:26.573 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4 2023-10-14T16:24:26.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2023-10-14T16:24:26.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2023-10-14T16:24:26.574 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5 2023-10-14T16:24:26.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core 2023-10-14T16:24:26.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2023-10-14T16:24:26.575 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel 2023-10-14T16:24:26.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2023-10-14T16:24:26.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom 2023-10-14T16:24:26.576 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge 2023-10-14T16:24:26.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2023-10-14T16:24:26.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi 2023-10-14T16:24:26.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2023-10-14T16:24:26.577 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2023-10-14T16:24:26.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion 2023-10-14T16:24:26.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/neterion/s2io.ko 2023-10-14T16:24:26.578 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia 2023-10-14T16:24:26.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2023-10-14T16:24:26.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines 2023-10-14T16:24:26.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2023-10-14T16:24:26.579 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2023-10-14T16:24:26.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic 2023-10-14T16:24:26.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen 2023-10-14T16:24:26.580 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2023-10-14T16:24:26.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2023-10-14T16:24:26.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic 2023-10-14T16:24:26.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2023-10-14T16:24:26.581 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc 2023-10-14T16:24:26.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rdc/r6040.ko 2023-10-14T16:24:26.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek 2023-10-14T16:24:26.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139cp.ko 2023-10-14T16:24:26.582 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/8139too.ko 2023-10-14T16:24:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/atp.ko 2023-10-14T16:24:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/realtek/r8169.ko 2023-10-14T16:24:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker 2023-10-14T16:24:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/rocker/rocker.ko 2023-10-14T16:24:26.583 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc 2023-10-14T16:24:26.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sfc/sfc.ko 2023-10-14T16:24:26.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan 2023-10-14T16:24:26.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/silan/sc92031.ko 2023-10-14T16:24:26.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis 2023-10-14T16:24:26.584 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis190.ko 2023-10-14T16:24:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sis/sis900.ko 2023-10-14T16:24:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc 2023-10-14T16:24:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/epic100.ko 2023-10-14T16:24:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2023-10-14T16:24:26.585 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2023-10-14T16:24:26.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2023-10-14T16:24:26.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro 2023-10-14T16:24:26.586 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac 2023-10-14T16:24:26.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2023-10-14T16:24:26.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2023-10-14T16:24:26.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun 2023-10-14T16:24:26.587 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/cassini.ko 2023-10-14T16:24:26.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/niu.ko 2023-10-14T16:24:26.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sungem.ko 2023-10-14T16:24:26.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/sun/sunhme.ko 2023-10-14T16:24:26.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti 2023-10-14T16:24:26.588 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2023-10-14T16:24:26.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti 2023-10-14T16:24:26.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/ti/tlan.ko 2023-10-14T16:24:26.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via 2023-10-14T16:24:26.589 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-rhine.ko 2023-10-14T16:24:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/via/via-velocity.ko 2023-10-14T16:24:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet 2023-10-14T16:24:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5100.ko 2023-10-14T16:24:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/wiznet/w5300.ko 2023-10-14T16:24:26.590 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom 2023-10-14T16:24:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2023-10-14T16:24:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/geneve.ko 2023-10-14T16:24:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv 2023-10-14T16:24:26.591 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/hyperv/hv_netvsc.ko 2023-10-14T16:24:26.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154 2023-10-14T16:24:26.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ieee802154/fakelb.ko 2023-10-14T16:24:26.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ifb.ko 2023-10-14T16:24:26.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan 2023-10-14T16:24:26.592 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ipvlan/ipvlan.ko 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvlan.ko 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/macvtap.ko 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio.ko 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2023-10-14T16:24:26.593 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-bitbang.ko 2023-10-14T16:24:26.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mdio/mdio-regmap.ko 2023-10-14T16:24:26.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/mii.ko 2023-10-14T16:24:26.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/net_failover.ko 2023-10-14T16:24:26.594 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/netconsole.ko 2023-10-14T16:24:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/nlmon.ko 2023-10-14T16:24:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs 2023-10-14T16:24:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs-lynx.ko 2023-10-14T16:24:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/pcs/pcs_xpcs.ko 2023-10-14T16:24:26.595 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy 2023-10-14T16:24:26.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/amd.ko 2023-10-14T16:24:26.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-lib.ko 2023-10-14T16:24:26.596 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm-phy-ptp.ko 2023-10-14T16:24:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm7xxx.ko 2023-10-14T16:24:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/bcm87xx.ko 2023-10-14T16:24:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/broadcom.ko 2023-10-14T16:24:26.597 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/cicada.ko 2023-10-14T16:24:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/davicom.ko 2023-10-14T16:24:26.598 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/dp83640.ko 2023-10-14T16:24:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/et1011c.ko 2023-10-14T16:24:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/icplus.ko 2023-10-14T16:24:26.599 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/lxt.ko 2023-10-14T16:24:26.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/marvell.ko 2023-10-14T16:24:26.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/micrel.ko 2023-10-14T16:24:26.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/national.ko 2023-10-14T16:24:26.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/phylink.ko 2023-10-14T16:24:26.600 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/qsemi.ko 2023-10-14T16:24:26.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/realtek.ko 2023-10-14T16:24:26.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/smsc.ko 2023-10-14T16:24:26.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/ste10Xp.ko 2023-10-14T16:24:26.601 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/phy/vitesse.ko 2023-10-14T16:24:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp 2023-10-14T16:24:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/bsd_comp.ko 2023-10-14T16:24:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_async.ko 2023-10-14T16:24:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_deflate.ko 2023-10-14T16:24:26.602 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_generic.ko 2023-10-14T16:24:26.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_mppe.ko 2023-10-14T16:24:26.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/ppp_synctty.ko 2023-10-14T16:24:26.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppoe.ko 2023-10-14T16:24:26.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pppox.ko 2023-10-14T16:24:26.603 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/ppp/pptp.ko 2023-10-14T16:24:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip 2023-10-14T16:24:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slhc.ko 2023-10-14T16:24:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/slip/slip.ko 2023-10-14T16:24:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/sungem_phy.ko 2023-10-14T16:24:26.604 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tap.ko 2023-10-14T16:24:26.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team 2023-10-14T16:24:26.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team.ko 2023-10-14T16:24:26.605 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_activebackup.ko 2023-10-14T16:24:26.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_broadcast.ko 2023-10-14T16:24:26.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_loadbalance.ko 2023-10-14T16:24:26.606 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_random.ko 2023-10-14T16:24:26.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/team/team_mode_roundrobin.ko 2023-10-14T16:24:26.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/tun.ko 2023-10-14T16:24:26.607 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/veth.ko 2023-10-14T16:24:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/virtio_net.ko 2023-10-14T16:24:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3 2023-10-14T16:24:26.608 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vmxnet3/vmxnet3.ko 2023-10-14T16:24:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan 2023-10-14T16:24:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/vxlan/vxlan.ko 2023-10-14T16:24:26.609 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback 2023-10-14T16:24:26.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netback/xen-netback.ko 2023-10-14T16:24:26.610 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/net/xen-netfront.ko 2023-10-14T16:24:26.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb 2023-10-14T16:24:26.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ntb/ntb.ko 2023-10-14T16:24:26.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme 2023-10-14T16:24:26.611 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host 2023-10-14T16:24:26.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme-core.ko 2023-10-14T16:24:26.612 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/nvme/host/nvme.ko 2023-10-14T16:24:26.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport 2023-10-14T16:24:26.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport.ko 2023-10-14T16:24:26.613 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_cs.ko 2023-10-14T16:24:26.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_pc.ko 2023-10-14T16:24:26.614 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/parport/parport_serial.ko 2023-10-14T16:24:26.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci 2023-10-14T16:24:26.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug 2023-10-14T16:24:26.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2023-10-14T16:24:26.615 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie 2023-10-14T16:24:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/pcie/aer_inject.ko 2023-10-14T16:24:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pci/xen-pcifront.ko 2023-10-14T16:24:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia 2023-10-14T16:24:26.616 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/i82092.ko 2023-10-14T16:24:26.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/pd6729.ko 2023-10-14T16:24:26.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pcmcia/yenta_socket.ko 2023-10-14T16:24:26.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl 2023-10-14T16:24:26.617 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel 2023-10-14T16:24:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2023-10-14T16:24:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform 2023-10-14T16:24:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86 2023-10-14T16:24:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acer-wmi.ko 2023-10-14T16:24:26.618 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/acerhdf.ko 2023-10-14T16:24:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/apple-gmux.ko 2023-10-14T16:24:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-laptop.ko 2023-10-14T16:24:26.619 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-nb-wmi.ko 2023-10-14T16:24:26.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/asus-wmi.ko 2023-10-14T16:24:26.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/classmate-laptop.ko 2023-10-14T16:24:26.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-laptop.ko 2023-10-14T16:24:26.620 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/eeepc-wmi.ko 2023-10-14T16:24:26.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-laptop.ko 2023-10-14T16:24:26.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/fujitsu-tablet.ko 2023-10-14T16:24:26.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/hdaps.ko 2023-10-14T16:24:26.621 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel 2023-10-14T16:24:26.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel/intel-rst.ko 2023-10-14T16:24:26.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/intel_ips.ko 2023-10-14T16:24:26.622 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/msi-wmi.ko 2023-10-14T16:24:26.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/mxm-wmi.ko 2023-10-14T16:24:26.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/panasonic-laptop.ko 2023-10-14T16:24:26.623 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-laptop.ko 2023-10-14T16:24:26.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/samsung-q10.ko 2023-10-14T16:24:26.624 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/topstar-laptop.ko 2023-10-14T16:24:26.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2023-10-14T16:24:26.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/toshiba_haps.ko 2023-10-14T16:24:26.625 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi-bmof.ko 2023-10-14T16:24:26.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/platform/x86/wmi.ko 2023-10-14T16:24:26.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap 2023-10-14T16:24:26.626 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_common.ko 2023-10-14T16:24:26.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/powercap/intel_rapl_msr.ko 2023-10-14T16:24:26.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps 2023-10-14T16:24:26.627 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients 2023-10-14T16:24:26.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-gpio.ko 2023-10-14T16:24:26.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps-ldisc.ko 2023-10-14T16:24:26.628 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/clients/pps_parport.ko 2023-10-14T16:24:26.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/pps/pps_core.ko 2023-10-14T16:24:26.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp 2023-10-14T16:24:26.629 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp.ko 2023-10-14T16:24:26.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ptp/ptp_kvm.ko 2023-10-14T16:24:26.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc 2023-10-14T16:24:26.630 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-bq32k.ko 2023-10-14T16:24:26.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1286.ko 2023-10-14T16:24:26.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1307.ko 2023-10-14T16:24:26.631 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1374.ko 2023-10-14T16:24:26.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1511.ko 2023-10-14T16:24:26.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1553.ko 2023-10-14T16:24:26.632 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1672.ko 2023-10-14T16:24:26.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds1742.ko 2023-10-14T16:24:26.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds2404.ko 2023-10-14T16:24:26.633 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-ds3232.ko 2023-10-14T16:24:26.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-em3027.ko 2023-10-14T16:24:26.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-fm3130.ko 2023-10-14T16:24:26.634 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl12022.ko 2023-10-14T16:24:26.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-isl1208.ko 2023-10-14T16:24:26.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m41t80.ko 2023-10-14T16:24:26.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t35.ko 2023-10-14T16:24:26.635 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-m48t59.ko 2023-10-14T16:24:26.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-max6900.ko 2023-10-14T16:24:26.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-msm6242.ko 2023-10-14T16:24:26.636 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf2127.ko 2023-10-14T16:24:26.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf85063.ko 2023-10-14T16:24:26.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8523.ko 2023-10-14T16:24:26.637 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8563.ko 2023-10-14T16:24:26.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-pcf8583.ko 2023-10-14T16:24:26.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rp5c01.ko 2023-10-14T16:24:26.638 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rs5c372.ko 2023-10-14T16:24:26.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rv3029c2.ko 2023-10-14T16:24:26.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8025.ko 2023-10-14T16:24:26.639 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-rx8581.ko 2023-10-14T16:24:26.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-stk17ta8.ko 2023-10-14T16:24:26.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/rtc/rtc-x1205.ko 2023-10-14T16:24:26.640 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi 2023-10-14T16:24:26.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-9xxx.ko 2023-10-14T16:24:26.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-sas.ko 2023-10-14T16:24:26.641 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/3w-xxxx.ko 2023-10-14T16:24:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/BusLogic.ko 2023-10-14T16:24:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/a100u2w.ko 2023-10-14T16:24:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid 2023-10-14T16:24:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aacraid/aacraid.ko 2023-10-14T16:24:26.642 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/advansys.ko 2023-10-14T16:24:26.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx 2023-10-14T16:24:26.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2023-10-14T16:24:26.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2023-10-14T16:24:26.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/am53c974.ko 2023-10-14T16:24:26.643 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr 2023-10-14T16:24:26.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/arcmsr/arcmsr.ko 2023-10-14T16:24:26.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/atp870u.ko 2023-10-14T16:24:26.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi 2023-10-14T16:24:26.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2023-10-14T16:24:26.644 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa 2023-10-14T16:24:26.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bfa/bfa.ko 2023-10-14T16:24:26.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc 2023-10-14T16:24:26.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2023-10-14T16:24:26.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i 2023-10-14T16:24:26.645 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/bnx2i/bnx2i.ko 2023-10-14T16:24:26.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ch.ko 2023-10-14T16:24:26.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor 2023-10-14T16:24:26.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/csiostor/csiostor.ko 2023-10-14T16:24:26.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi 2023-10-14T16:24:26.646 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i 2023-10-14T16:24:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2023-10-14T16:24:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i 2023-10-14T16:24:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2023-10-14T16:24:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2023-10-14T16:24:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dc395x.ko 2023-10-14T16:24:26.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler 2023-10-14T16:24:26.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2023-10-14T16:24:26.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2023-10-14T16:24:26.648 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2023-10-14T16:24:26.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2023-10-14T16:24:26.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/dmx3191d.ko 2023-10-14T16:24:26.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r 2023-10-14T16:24:26.649 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esas2r/esas2r.ko 2023-10-14T16:24:26.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/esp_scsi.ko 2023-10-14T16:24:26.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe 2023-10-14T16:24:26.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/fcoe.ko 2023-10-14T16:24:26.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fcoe/libfcoe.ko 2023-10-14T16:24:26.650 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic 2023-10-14T16:24:26.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/fnic/fnic.ko 2023-10-14T16:24:26.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hpsa.ko 2023-10-14T16:24:26.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hptiop.ko 2023-10-14T16:24:26.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/hv_storvsc.ko 2023-10-14T16:24:26.651 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/initio.ko 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ipr.ko 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ips.ko 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/isci/isci.ko 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2023-10-14T16:24:26.652 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/iscsi_tcp.ko 2023-10-14T16:24:26.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc 2023-10-14T16:24:26.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libfc/libfc.ko 2023-10-14T16:24:26.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi.ko 2023-10-14T16:24:26.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libiscsi_tcp.ko 2023-10-14T16:24:26.653 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas 2023-10-14T16:24:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/libsas/libsas.ko 2023-10-14T16:24:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc 2023-10-14T16:24:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/lpfc/lpfc.ko 2023-10-14T16:24:26.654 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid 2023-10-14T16:24:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid.ko 2023-10-14T16:24:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2023-10-14T16:24:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2023-10-14T16:24:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2023-10-14T16:24:26.655 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvsas/mvsas.ko 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/mvumi.ko 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001 2023-10-14T16:24:26.656 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pm8001/pm80xx.ko 2023-10-14T16:24:26.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/pmcraid.ko 2023-10-14T16:24:26.657 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla1280.ko 2023-10-14T16:24:26.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx 2023-10-14T16:24:26.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2023-10-14T16:24:26.658 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2023-10-14T16:24:26.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx 2023-10-14T16:24:26.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2023-10-14T16:24:26.659 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/raid_class.ko 2023-10-14T16:24:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_debug.ko 2023-10-14T16:24:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_fc.ko 2023-10-14T16:24:26.660 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_iscsi.ko 2023-10-14T16:24:26.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_sas.ko 2023-10-14T16:24:26.661 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_spi.ko 2023-10-14T16:24:26.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/scsi_transport_srp.ko 2023-10-14T16:24:26.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/ses.ko 2023-10-14T16:24:26.662 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/st.ko 2023-10-14T16:24:26.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/stex.ko 2023-10-14T16:24:26.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2 2023-10-14T16:24:26.663 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2023-10-14T16:24:26.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/virtio_scsi.ko 2023-10-14T16:24:26.664 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/vmw_pvscsi.ko 2023-10-14T16:24:26.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/scsi/wd719x.ko 2023-10-14T16:24:26.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb 2023-10-14T16:24:26.665 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ssb/ssb.ko 2023-10-14T16:24:26.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target 2023-10-14T16:24:26.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi 2023-10-14T16:24:26.666 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2023-10-14T16:24:26.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback 2023-10-14T16:24:26.667 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/loopback/tcm_loop.ko 2023-10-14T16:24:26.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_file.ko 2023-10-14T16:24:26.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_iblock.ko 2023-10-14T16:24:26.668 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_mod.ko 2023-10-14T16:24:26.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_pscsi.ko 2023-10-14T16:24:26.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/target_core_user.ko 2023-10-14T16:24:26.669 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc 2023-10-14T16:24:26.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/target/tcm_fc/tcm_fc.ko 2023-10-14T16:24:26.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal 2023-10-14T16:24:26.670 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel 2023-10-14T16:24:26.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal 2023-10-14T16:24:26.671 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2023-10-14T16:24:26.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2023-10-14T16:24:26.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2023-10-14T16:24:26.672 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2023-10-14T16:24:26.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2023-10-14T16:24:26.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2023-10-14T16:24:26.673 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2023-10-14T16:24:26.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2023-10-14T16:24:26.674 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2023-10-14T16:24:26.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2023-10-14T16:24:26.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2023-10-14T16:24:26.675 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2023-10-14T16:24:26.676 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2023-10-14T16:24:26.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2023-10-14T16:24:26.677 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2023-10-14T16:24:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty 2023-10-14T16:24:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless 2023-10-14T16:24:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/ipwireless/ipwireless.ko 2023-10-14T16:24:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_gsm.ko 2023-10-14T16:24:26.678 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/n_hdlc.ko 2023-10-14T16:24:26.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/nozomi.ko 2023-10-14T16:24:26.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial 2023-10-14T16:24:26.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250 2023-10-14T16:24:26.679 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/8250/serial_cs.ko 2023-10-14T16:24:26.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/arc_uart.ko 2023-10-14T16:24:26.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm 2023-10-14T16:24:26.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/tty/serial/jsm/jsm.ko 2023-10-14T16:24:26.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs 2023-10-14T16:24:26.680 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core 2023-10-14T16:24:26.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/core/ufshcd-core.ko 2023-10-14T16:24:26.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host 2023-10-14T16:24:26.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/ufs/host/ufshcd-pci.ko 2023-10-14T16:24:26.681 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio 2023-10-14T16:24:26.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio.ko 2023-10-14T16:24:26.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_aec.ko 2023-10-14T16:24:26.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_cif.ko 2023-10-14T16:24:26.682 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_pci_generic.ko 2023-10-14T16:24:26.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/uio/uio_sercos3.ko 2023-10-14T16:24:26.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb 2023-10-14T16:24:26.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class 2023-10-14T16:24:26.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-acm.ko 2023-10-14T16:24:26.683 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/cdc-wdm.ko 2023-10-14T16:24:26.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usblp.ko 2023-10-14T16:24:26.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/class/usbtmc.ko 2023-10-14T16:24:26.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host 2023-10-14T16:24:26.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/host/sl811-hcd.ko 2023-10-14T16:24:26.684 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image 2023-10-14T16:24:26.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/mdc800.ko 2023-10-14T16:24:26.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/image/microtek.ko 2023-10-14T16:24:26.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc 2023-10-14T16:24:26.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/adutux.ko 2023-10-14T16:24:26.685 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/appledisplay.ko 2023-10-14T16:24:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi26.ko 2023-10-14T16:24:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/emi62.ko 2023-10-14T16:24:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ezusb.ko 2023-10-14T16:24:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/idmouse.ko 2023-10-14T16:24:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/iowarrior.ko 2023-10-14T16:24:26.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/isight_firmware.ko 2023-10-14T16:24:26.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/ldusb.ko 2023-10-14T16:24:26.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/legousbtower.ko 2023-10-14T16:24:26.687 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/trancevibrator.ko 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usb3503.ko 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usblcd.ko 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/usbsevseg.ko 2023-10-14T16:24:26.688 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/uss720.ko 2023-10-14T16:24:26.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/misc/yurex.ko 2023-10-14T16:24:26.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy 2023-10-14T16:24:26.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/phy/phy-generic.ko 2023-10-14T16:24:26.689 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial 2023-10-14T16:24:26.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/aircable.ko 2023-10-14T16:24:26.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ark3116.ko 2023-10-14T16:24:26.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/belkin_sa.ko 2023-10-14T16:24:26.690 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ch341.ko 2023-10-14T16:24:26.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cp210x.ko 2023-10-14T16:24:26.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cyberjack.ko 2023-10-14T16:24:26.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/cypress_m8.ko 2023-10-14T16:24:26.691 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/digi_acceleport.ko 2023-10-14T16:24:26.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/empeg.ko 2023-10-14T16:24:26.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ftdi_sio.ko 2023-10-14T16:24:26.692 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/garmin_gps.ko 2023-10-14T16:24:26.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_edgeport.ko 2023-10-14T16:24:26.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/io_ti.ko 2023-10-14T16:24:26.693 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipaq.ko 2023-10-14T16:24:26.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ipw.ko 2023-10-14T16:24:26.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ir-usb.ko 2023-10-14T16:24:26.694 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/iuu_phoenix.ko 2023-10-14T16:24:26.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan.ko 2023-10-14T16:24:26.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/keyspan_pda.ko 2023-10-14T16:24:26.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kl5kusb105.ko 2023-10-14T16:24:26.695 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/kobil_sct.ko 2023-10-14T16:24:26.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mct_u232.ko 2023-10-14T16:24:26.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7720.ko 2023-10-14T16:24:26.696 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/mos7840.ko 2023-10-14T16:24:26.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/navman.ko 2023-10-14T16:24:26.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/omninet.ko 2023-10-14T16:24:26.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/opticon.ko 2023-10-14T16:24:26.697 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/option.ko 2023-10-14T16:24:26.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/oti6858.ko 2023-10-14T16:24:26.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/pl2303.ko 2023-10-14T16:24:26.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcaux.ko 2023-10-14T16:24:26.698 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/qcserial.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/quatech2.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/safe_serial.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/sierra.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/spcp8x5.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ssu100.ko 2023-10-14T16:24:26.699 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/symbolserial.ko 2023-10-14T16:24:26.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2023-10-14T16:24:26.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb-serial-simple.ko 2023-10-14T16:24:26.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_debug.ko 2023-10-14T16:24:26.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/usb_wwan.ko 2023-10-14T16:24:26.700 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/visor.ko 2023-10-14T16:24:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/whiteheat.ko 2023-10-14T16:24:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/serial/xsens_mt.ko 2023-10-14T16:24:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage 2023-10-14T16:24:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/uas.ko 2023-10-14T16:24:26.701 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-alauda.ko 2023-10-14T16:24:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-cypress.ko 2023-10-14T16:24:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-datafab.ko 2023-10-14T16:24:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-eneub6250.ko 2023-10-14T16:24:26.702 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-freecom.ko 2023-10-14T16:24:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-isd200.ko 2023-10-14T16:24:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-jumpshot.ko 2023-10-14T16:24:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-karma.ko 2023-10-14T16:24:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-onetouch.ko 2023-10-14T16:24:26.703 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-realtek.ko 2023-10-14T16:24:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr09.ko 2023-10-14T16:24:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-sddr55.ko 2023-10-14T16:24:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/ums-usbat.ko 2023-10-14T16:24:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/storage/usb-storage.ko 2023-10-14T16:24:26.704 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip 2023-10-14T16:24:26.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-core.ko 2023-10-14T16:24:26.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/usbip-host.ko 2023-10-14T16:24:26.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/usb/usbip/vhci-hcd.ko 2023-10-14T16:24:26.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio 2023-10-14T16:24:26.705 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci 2023-10-14T16:24:26.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci-core.ko 2023-10-14T16:24:26.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/pci/vfio-pci.ko 2023-10-14T16:24:26.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio.ko 2023-10-14T16:24:26.706 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vfio/vfio_iommu_type1.ko 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost.ko 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_iotlb.ko 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_net.ko 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/vhost/vhost_scsi.ko 2023-10-14T16:24:26.707 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video 2023-10-14T16:24:26.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight 2023-10-14T16:24:26.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/apple_bl.ko 2023-10-14T16:24:26.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/lcd.ko 2023-10-14T16:24:26.708 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/video/backlight/platform_lcd.ko 2023-10-14T16:24:26.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio 2023-10-14T16:24:26.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio.ko 2023-10-14T16:24:26.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_balloon.ko 2023-10-14T16:24:26.709 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_mmio.ko 2023-10-14T16:24:26.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci.ko 2023-10-14T16:24:26.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2023-10-14T16:24:26.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2023-10-14T16:24:26.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/virtio/virtio_ring.ko 2023-10-14T16:24:26.710 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1 2023-10-14T16:24:26.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters 2023-10-14T16:24:26.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2482.ko 2023-10-14T16:24:26.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/masters/ds2490.ko 2023-10-14T16:24:26.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves 2023-10-14T16:24:26.711 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2406.ko 2023-10-14T16:24:26.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2408.ko 2023-10-14T16:24:26.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2413.ko 2023-10-14T16:24:26.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2423.ko 2023-10-14T16:24:26.712 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2431.ko 2023-10-14T16:24:26.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2433.ko 2023-10-14T16:24:26.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2780.ko 2023-10-14T16:24:26.713 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds2781.ko 2023-10-14T16:24:26.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_ds28e04.ko 2023-10-14T16:24:26.714 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_smem.ko 2023-10-14T16:24:26.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/slaves/w1_therm.ko 2023-10-14T16:24:26.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/w1/wire.ko 2023-10-14T16:24:26.715 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog 2023-10-14T16:24:26.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim1535_wdt.ko 2023-10-14T16:24:26.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/alim7101_wdt.ko 2023-10-14T16:24:26.716 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/f71808e_wdt.ko 2023-10-14T16:24:26.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/hpwdt.ko 2023-10-14T16:24:26.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/i6300esb.ko 2023-10-14T16:24:26.717 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_vendor_support.ko 2023-10-14T16:24:26.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/iTCO_wdt.ko 2023-10-14T16:24:26.718 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ib700wdt.ko 2023-10-14T16:24:26.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ibmasr.ko 2023-10-14T16:24:26.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/ie6xx_wdt.ko 2023-10-14T16:24:26.719 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it8712f_wdt.ko 2023-10-14T16:24:26.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/it87_wdt.ko 2023-10-14T16:24:26.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/machzwd.ko 2023-10-14T16:24:26.720 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/nv_tco.ko 2023-10-14T16:24:26.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_pci.ko 2023-10-14T16:24:26.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/pcwd_usb.ko 2023-10-14T16:24:26.721 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2023-10-14T16:24:26.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sch311x_wdt.ko 2023-10-14T16:24:26.722 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/softdog.ko 2023-10-14T16:24:26.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/sp5100_tco.ko 2023-10-14T16:24:26.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/via_wdt.ko 2023-10-14T16:24:26.723 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83627hf_wdt.ko 2023-10-14T16:24:26.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83877f_wdt.ko 2023-10-14T16:24:26.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/w83977f_wdt.ko 2023-10-14T16:24:26.724 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/wdt_pci.ko 2023-10-14T16:24:26.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/watchdog/xen_wdt.ko 2023-10-14T16:24:26.725 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen 2023-10-14T16:24:26.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-acpi-processor.ko 2023-10-14T16:24:26.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-evtchn.ko 2023-10-14T16:24:26.726 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntalloc.ko 2023-10-14T16:24:26.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-gntdev.ko 2023-10-14T16:24:26.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback 2023-10-14T16:24:26.727 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2023-10-14T16:24:26.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xen-privcmd.ko 2023-10-14T16:24:26.728 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs 2023-10-14T16:24:26.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/drivers/xen/xenfs/xenfs.ko 2023-10-14T16:24:26.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs 2023-10-14T16:24:26.729 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p 2023-10-14T16:24:26.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/9p/9p.ko 2023-10-14T16:24:26.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs 2023-10-14T16:24:26.730 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/affs/affs.ko 2023-10-14T16:24:26.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs 2023-10-14T16:24:26.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/befs/befs.ko 2023-10-14T16:24:26.731 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/binfmt_misc.ko 2023-10-14T16:24:26.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs 2023-10-14T16:24:26.732 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/btrfs/btrfs.ko 2023-10-14T16:24:26.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles 2023-10-14T16:24:26.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cachefiles/cachefiles.ko 2023-10-14T16:24:26.733 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph 2023-10-14T16:24:26.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ceph/ceph.ko 2023-10-14T16:24:26.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda 2023-10-14T16:24:26.734 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/coda/coda.ko 2023-10-14T16:24:26.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs 2023-10-14T16:24:26.735 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/cramfs/cramfs.ko 2023-10-14T16:24:26.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm 2023-10-14T16:24:26.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/dlm/dlm.ko 2023-10-14T16:24:26.736 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs 2023-10-14T16:24:26.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ecryptfs/ecryptfs.ko 2023-10-14T16:24:26.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs 2023-10-14T16:24:26.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/f2fs/f2fs.ko 2023-10-14T16:24:26.737 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat 2023-10-14T16:24:26.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/fat.ko 2023-10-14T16:24:26.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/msdos.ko 2023-10-14T16:24:26.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fat/vfat.ko 2023-10-14T16:24:26.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache 2023-10-14T16:24:26.738 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fscache/fscache.ko 2023-10-14T16:24:26.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse 2023-10-14T16:24:26.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/cuse.ko 2023-10-14T16:24:26.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/fuse/fuse.ko 2023-10-14T16:24:26.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2 2023-10-14T16:24:26.739 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/gfs2/gfs2.ko 2023-10-14T16:24:26.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs 2023-10-14T16:24:26.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfs/hfs.ko 2023-10-14T16:24:26.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus 2023-10-14T16:24:26.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/hfsplus/hfsplus.ko 2023-10-14T16:24:26.740 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/isofs/isofs.ko 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/jfs/jfs.ko 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/lockd/lockd.ko 2023-10-14T16:24:26.741 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix 2023-10-14T16:24:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/minix/minix.ko 2023-10-14T16:24:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs 2023-10-14T16:24:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/netfs/netfs.ko 2023-10-14T16:24:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs 2023-10-14T16:24:26.742 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout 2023-10-14T16:24:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2023-10-14T16:24:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout 2023-10-14T16:24:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2023-10-14T16:24:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout 2023-10-14T16:24:26.743 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfs.ko 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv3.ko 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs/nfsv4.ko 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/grace.ko 2023-10-14T16:24:26.744 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfs_common/nfs_acl.ko 2023-10-14T16:24:26.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd 2023-10-14T16:24:26.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nfsd/nfsd.ko 2023-10-14T16:24:26.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2 2023-10-14T16:24:26.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nilfs2/nilfs2.ko 2023-10-14T16:24:26.745 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls 2023-10-14T16:24:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-celtic.ko 2023-10-14T16:24:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-centeuro.ko 2023-10-14T16:24:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-croatian.ko 2023-10-14T16:24:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-cyrillic.ko 2023-10-14T16:24:26.746 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-gaelic.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-greek.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-iceland.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-inuit.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-roman.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-romanian.ko 2023-10-14T16:24:26.747 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/mac-turkish.ko 2023-10-14T16:24:26.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1250.ko 2023-10-14T16:24:26.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1251.ko 2023-10-14T16:24:26.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp1255.ko 2023-10-14T16:24:26.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp737.ko 2023-10-14T16:24:26.748 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp775.ko 2023-10-14T16:24:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp850.ko 2023-10-14T16:24:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp852.ko 2023-10-14T16:24:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp855.ko 2023-10-14T16:24:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp857.ko 2023-10-14T16:24:26.749 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp860.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp861.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp862.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp863.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp864.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp865.ko 2023-10-14T16:24:26.750 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp866.ko 2023-10-14T16:24:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp869.ko 2023-10-14T16:24:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp874.ko 2023-10-14T16:24:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp932.ko 2023-10-14T16:24:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp936.ko 2023-10-14T16:24:26.751 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp949.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_cp950.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_euc-jp.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-1.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-13.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-14.ko 2023-10-14T16:24:26.752 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-15.ko 2023-10-14T16:24:26.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-2.ko 2023-10-14T16:24:26.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-3.ko 2023-10-14T16:24:26.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-4.ko 2023-10-14T16:24:26.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-5.ko 2023-10-14T16:24:26.753 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-6.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-7.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_iso8859-9.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-r.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-ru.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_koi8-u.ko 2023-10-14T16:24:26.754 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_ucs2_utils.ko 2023-10-14T16:24:26.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/nls/nls_utf8.ko 2023-10-14T16:24:26.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2 2023-10-14T16:24:26.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster 2023-10-14T16:24:26.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2023-10-14T16:24:26.755 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm 2023-10-14T16:24:26.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2023-10-14T16:24:26.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs 2023-10-14T16:24:26.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2023-10-14T16:24:26.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2.ko 2023-10-14T16:24:26.756 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stack_user.ko 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ocfs2/ocfs2_stackglue.ko 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/overlayfs/overlay.ko 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore 2023-10-14T16:24:26.757 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/pstore/ramoops.ko 2023-10-14T16:24:26.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs 2023-10-14T16:24:26.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/reiserfs/reiserfs.ko 2023-10-14T16:24:26.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs 2023-10-14T16:24:26.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/romfs/romfs.ko 2023-10-14T16:24:26.758 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb 2023-10-14T16:24:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client 2023-10-14T16:24:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/client/cifs.ko 2023-10-14T16:24:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common 2023-10-14T16:24:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_arc4.ko 2023-10-14T16:24:26.759 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/smb/common/cifs_md4.ko 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/squashfs/squashfs.ko 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/sysv/sysv.ko 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf 2023-10-14T16:24:26.760 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/udf/udf.ko 2023-10-14T16:24:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs 2023-10-14T16:24:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/ufs/ufs.ko 2023-10-14T16:24:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs 2023-10-14T16:24:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/fs/xfs/xfs.ko 2023-10-14T16:24:26.761 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel 2023-10-14T16:24:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu 2023-10-14T16:24:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/rcu/rcutorture.ko 2023-10-14T16:24:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/torture.ko 2023-10-14T16:24:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace 2023-10-14T16:24:26.762 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/kernel/trace/ring_buffer_benchmark.ko 2023-10-14T16:24:26.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib 2023-10-14T16:24:26.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/asn1_encoder.ko 2023-10-14T16:24:26.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc-itu-t.ko 2023-10-14T16:24:26.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crc8.ko 2023-10-14T16:24:26.763 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto 2023-10-14T16:24:26.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libarc4.ko 2023-10-14T16:24:26.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/crypto/libdes.ko 2023-10-14T16:24:26.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/libcrc32c.ko 2023-10-14T16:24:26.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lru_cache.ko 2023-10-14T16:24:26.764 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4_compress.ko 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/lz4/lz4hc_compress.ko 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/math/cordic.ko 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6 2023-10-14T16:24:26.765 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/raid6/raid6_pq.ko 2023-10-14T16:24:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon 2023-10-14T16:24:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/reed_solomon/reed_solomon.ko 2023-10-14T16:24:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_bm.ko 2023-10-14T16:24:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_fsm.ko 2023-10-14T16:24:26.766 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/ts_kmp.ko 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/lib/zstd/zstd_compress.ko 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/mm/hwpoison-inject.ko 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net 2023-10-14T16:24:26.767 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan 2023-10-14T16:24:26.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/6lowpan.ko 2023-10-14T16:24:26.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_dest.ko 2023-10-14T16:24:26.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_fragment.ko 2023-10-14T16:24:26.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_hop.ko 2023-10-14T16:24:26.768 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_ipv6.ko 2023-10-14T16:24:26.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_mobility.ko 2023-10-14T16:24:26.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_routing.ko 2023-10-14T16:24:26.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/6lowpan/nhc_udp.ko 2023-10-14T16:24:26.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802 2023-10-14T16:24:26.769 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/garp.ko 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/mrp.ko 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/p8022.ko 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/psnap.ko 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/802/stp.ko 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q 2023-10-14T16:24:26.770 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/8021q/8021q.ko 2023-10-14T16:24:26.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p 2023-10-14T16:24:26.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet.ko 2023-10-14T16:24:26.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_fd.ko 2023-10-14T16:24:26.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_rdma.ko 2023-10-14T16:24:26.771 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/9p/9pnet_virtio.ko 2023-10-14T16:24:26.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv 2023-10-14T16:24:26.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/batman-adv/batman-adv.ko 2023-10-14T16:24:26.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge 2023-10-14T16:24:26.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/br_netfilter.ko 2023-10-14T16:24:26.772 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/bridge.ko 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_802_3.ko 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_among.ko 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arp.ko 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_arpreply.ko 2023-10-14T16:24:26.773 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_dnat.ko 2023-10-14T16:24:26.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip.ko 2023-10-14T16:24:26.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_ip6.ko 2023-10-14T16:24:26.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_limit.ko 2023-10-14T16:24:26.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_log.ko 2023-10-14T16:24:26.774 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark.ko 2023-10-14T16:24:26.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_mark_m.ko 2023-10-14T16:24:26.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_nflog.ko 2023-10-14T16:24:26.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_pkttype.ko 2023-10-14T16:24:26.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_redirect.ko 2023-10-14T16:24:26.775 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_snat.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_stp.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebt_vlan.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_broute.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_filter.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtable_nat.ko 2023-10-14T16:24:26.776 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/ebtables.ko 2023-10-14T16:24:26.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2023-10-14T16:24:26.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2023-10-14T16:24:26.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2023-10-14T16:24:26.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph 2023-10-14T16:24:26.777 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ceph/libceph.ko 2023-10-14T16:24:26.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core 2023-10-14T16:24:26.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/failover.ko 2023-10-14T16:24:26.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/core/pktgen.ko 2023-10-14T16:24:26.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp 2023-10-14T16:24:26.778 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp.ko 2023-10-14T16:24:26.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_diag.ko 2023-10-14T16:24:26.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv4.ko 2023-10-14T16:24:26.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dccp/dccp_ipv6.ko 2023-10-14T16:24:26.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver 2023-10-14T16:24:26.779 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dns_resolver/dns_resolver.ko 2023-10-14T16:24:26.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa 2023-10-14T16:24:26.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/dsa_core.ko 2023-10-14T16:24:26.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_brcm.ko 2023-10-14T16:24:26.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_dsa.ko 2023-10-14T16:24:26.780 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_none.ko 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/dsa/tag_trailer.ko 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154.ko 2023-10-14T16:24:26.781 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ieee802154/ieee802154_socket.ko 2023-10-14T16:24:26.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4 2023-10-14T16:24:26.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ah4.ko 2023-10-14T16:24:26.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/esp4.ko 2023-10-14T16:24:26.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/fou.ko 2023-10-14T16:24:26.782 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/gre.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/inet_diag.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_gre.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_tunnel.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ip_vti.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipcomp.ko 2023-10-14T16:24:26.783 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/ipip.ko 2023-10-14T16:24:26.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter 2023-10-14T16:24:26.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arp_tables.ko 2023-10-14T16:24:26.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arpt_mangle.ko 2023-10-14T16:24:26.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/arptable_filter.ko 2023-10-14T16:24:26.784 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ECN.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_ah.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_mangle.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_nat.ko 2023-10-14T16:24:26.785 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_raw.ko 2023-10-14T16:24:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/iptable_security.ko 2023-10-14T16:24:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2023-10-14T16:24:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2023-10-14T16:24:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2023-10-14T16:24:26.786 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2023-10-14T16:24:26.787 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2023-10-14T16:24:26.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_bic.ko 2023-10-14T16:24:26.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_dctcp.ko 2023-10-14T16:24:26.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_diag.ko 2023-10-14T16:24:26.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_highspeed.ko 2023-10-14T16:24:26.788 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_htcp.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_hybla.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_illinois.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_lp.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_scalable.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_vegas.ko 2023-10-14T16:24:26.789 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_veno.ko 2023-10-14T16:24:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_westwood.ko 2023-10-14T16:24:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tcp_yeah.ko 2023-10-14T16:24:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/tunnel4.ko 2023-10-14T16:24:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_diag.ko 2023-10-14T16:24:26.790 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/udp_tunnel.ko 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv4/xfrm4_tunnel.ko 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ah6.ko 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/esp6.ko 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/fou6.ko 2023-10-14T16:24:26.791 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_tunnel.ko 2023-10-14T16:24:26.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_udp_tunnel.ko 2023-10-14T16:24:26.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ip6_vti.ko 2023-10-14T16:24:26.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/ipcomp6.ko 2023-10-14T16:24:26.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter 2023-10-14T16:24:26.792 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6_tables.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ah.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_frag.ko 2023-10-14T16:24:26.793 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2023-10-14T16:24:26.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2023-10-14T16:24:26.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_mh.ko 2023-10-14T16:24:26.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2023-10-14T16:24:26.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6t_rt.ko 2023-10-14T16:24:26.794 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_filter.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_nat.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_raw.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/ip6table_security.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2023-10-14T16:24:26.795 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2023-10-14T16:24:26.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2023-10-14T16:24:26.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2023-10-14T16:24:26.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2023-10-14T16:24:26.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2023-10-14T16:24:26.796 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/sit.ko 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/tunnel6.ko 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/ipv6/xfrm6_tunnel.ko 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key 2023-10-14T16:24:26.797 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/key/af_key.ko 2023-10-14T16:24:26.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp 2023-10-14T16:24:26.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_core.ko 2023-10-14T16:24:26.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_debugfs.ko 2023-10-14T16:24:26.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_eth.ko 2023-10-14T16:24:26.798 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip.ko 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ip6.ko 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_netlink.ko 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/l2tp/l2tp_ppp.ko 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/llc/llc.ko 2023-10-14T16:24:26.799 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154 2023-10-14T16:24:26.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mac802154/mac802154.ko 2023-10-14T16:24:26.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls 2023-10-14T16:24:26.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/mpls/mpls_gso.ko 2023-10-14T16:24:26.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter 2023-10-14T16:24:26.800 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set.ko 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2023-10-14T16:24:26.801 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2023-10-14T16:24:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2023-10-14T16:24:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2023-10-14T16:24:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2023-10-14T16:24:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2023-10-14T16:24:26.802 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2023-10-14T16:24:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2023-10-14T16:24:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2023-10-14T16:24:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2023-10-14T16:24:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2023-10-14T16:24:26.803 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipset/ip_set_list_set.ko 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs.ko 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2023-10-14T16:24:26.804 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2023-10-14T16:24:26.805 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2023-10-14T16:24:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2023-10-14T16:24:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2023-10-14T16:24:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2023-10-14T16:24:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conncount.ko 2023-10-14T16:24:26.806 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack.ko 2023-10-14T16:24:26.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_amanda.ko 2023-10-14T16:24:26.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_broadcast.ko 2023-10-14T16:24:26.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_ftp.ko 2023-10-14T16:24:26.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_h323.ko 2023-10-14T16:24:26.807 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_irc.ko 2023-10-14T16:24:26.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2023-10-14T16:24:26.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_netlink.ko 2023-10-14T16:24:26.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_pptp.ko 2023-10-14T16:24:26.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sane.ko 2023-10-14T16:24:26.808 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_sip.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_snmp.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_conntrack_tftp.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_dup_netdev.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_flow_table_inet.ko 2023-10-14T16:24:26.809 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_log_syslog.ko 2023-10-14T16:24:26.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat.ko 2023-10-14T16:24:26.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_amanda.ko 2023-10-14T16:24:26.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_ftp.ko 2023-10-14T16:24:26.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_irc.ko 2023-10-14T16:24:26.810 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_sip.ko 2023-10-14T16:24:26.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_nat_tftp.ko 2023-10-14T16:24:26.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_synproxy_core.ko 2023-10-14T16:24:26.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nf_tables.ko 2023-10-14T16:24:26.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink.ko 2023-10-14T16:24:26.811 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_acct.ko 2023-10-14T16:24:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_log.ko 2023-10-14T16:24:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_osf.ko 2023-10-14T16:24:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nfnetlink_queue.ko 2023-10-14T16:24:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_chain_nat.ko 2023-10-14T16:24:26.812 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_compat.ko 2023-10-14T16:24:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_ct.ko 2023-10-14T16:24:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_dup_netdev.ko 2023-10-14T16:24:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fib.ko 2023-10-14T16:24:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_fwd_netdev.ko 2023-10-14T16:24:26.813 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_hash.ko 2023-10-14T16:24:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_limit.ko 2023-10-14T16:24:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_log.ko 2023-10-14T16:24:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_masq.ko 2023-10-14T16:24:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_nat.ko 2023-10-14T16:24:26.814 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_numgen.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_queue.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_quota.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_redir.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_reject_inet.ko 2023-10-14T16:24:26.815 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_socket.ko 2023-10-14T16:24:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/nft_tunnel.ko 2023-10-14T16:24:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_AUDIT.ko 2023-10-14T16:24:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CHECKSUM.ko 2023-10-14T16:24:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CLASSIFY.ko 2023-10-14T16:24:26.816 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CONNSECMARK.ko 2023-10-14T16:24:26.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_CT.ko 2023-10-14T16:24:26.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_DSCP.ko 2023-10-14T16:24:26.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HL.ko 2023-10-14T16:24:26.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_HMARK.ko 2023-10-14T16:24:26.817 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_IDLETIMER.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LED.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_LOG.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_MASQUERADE.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NETMAP.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFLOG.ko 2023-10-14T16:24:26.818 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_NFQUEUE.ko 2023-10-14T16:24:26.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_RATEEST.ko 2023-10-14T16:24:26.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_REDIRECT.ko 2023-10-14T16:24:26.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_SECMARK.ko 2023-10-14T16:24:26.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPMSS.ko 2023-10-14T16:24:26.819 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TEE.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TPROXY.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_TRACE.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_addrtype.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_bpf.ko 2023-10-14T16:24:26.820 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cgroup.ko 2023-10-14T16:24:26.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cluster.ko 2023-10-14T16:24:26.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_comment.ko 2023-10-14T16:24:26.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connbytes.ko 2023-10-14T16:24:26.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlabel.ko 2023-10-14T16:24:26.821 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connlimit.ko 2023-10-14T16:24:26.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_connmark.ko 2023-10-14T16:24:26.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_conntrack.ko 2023-10-14T16:24:26.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_cpu.ko 2023-10-14T16:24:26.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dccp.ko 2023-10-14T16:24:26.822 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_devgroup.ko 2023-10-14T16:24:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_dscp.ko 2023-10-14T16:24:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ecn.ko 2023-10-14T16:24:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_esp.ko 2023-10-14T16:24:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hashlimit.ko 2023-10-14T16:24:26.823 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_helper.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_hl.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipcomp.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_iprange.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_ipvs.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_l2tp.ko 2023-10-14T16:24:26.824 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_length.ko 2023-10-14T16:24:26.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_limit.ko 2023-10-14T16:24:26.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mac.ko 2023-10-14T16:24:26.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_mark.ko 2023-10-14T16:24:26.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_multiport.ko 2023-10-14T16:24:26.825 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nat.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_nfacct.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_osf.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_owner.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_physdev.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_pkttype.ko 2023-10-14T16:24:26.826 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_policy.ko 2023-10-14T16:24:26.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_quota.ko 2023-10-14T16:24:26.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_rateest.ko 2023-10-14T16:24:26.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_realm.ko 2023-10-14T16:24:26.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_recent.ko 2023-10-14T16:24:26.827 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_sctp.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_set.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_socket.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_state.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_statistic.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_string.ko 2023-10-14T16:24:26.828 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_tcpmss.ko 2023-10-14T16:24:26.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_time.ko 2023-10-14T16:24:26.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netfilter/xt_u32.ko 2023-10-14T16:24:26.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink 2023-10-14T16:24:26.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/netlink/netlink_diag.ko 2023-10-14T16:24:26.829 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/nsh/nsh.ko 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/openvswitch.ko 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-geneve.ko 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-gre.ko 2023-10-14T16:24:26.830 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/openvswitch/vport-vxlan.ko 2023-10-14T16:24:26.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet 2023-10-14T16:24:26.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/packet/af_packet_diag.ko 2023-10-14T16:24:26.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds 2023-10-14T16:24:26.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds.ko 2023-10-14T16:24:26.831 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_rdma.ko 2023-10-14T16:24:26.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/rds/rds_tcp.ko 2023-10-14T16:24:26.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched 2023-10-14T16:24:26.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_csum.ko 2023-10-14T16:24:26.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_gact.ko 2023-10-14T16:24:26.832 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_ipt.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_mirred.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_nat.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_pedit.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_police.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_simple.ko 2023-10-14T16:24:26.833 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_skbedit.ko 2023-10-14T16:24:26.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/act_vlan.ko 2023-10-14T16:24:26.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_basic.ko 2023-10-14T16:24:26.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_bpf.ko 2023-10-14T16:24:26.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_flow.ko 2023-10-14T16:24:26.834 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_fw.ko 2023-10-14T16:24:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_route.ko 2023-10-14T16:24:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/cls_u32.ko 2023-10-14T16:24:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_cmp.ko 2023-10-14T16:24:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_ipset.ko 2023-10-14T16:24:26.835 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_meta.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_nbyte.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_text.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/em_u32.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_choke.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_codel.ko 2023-10-14T16:24:26.836 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_drr.ko 2023-10-14T16:24:26.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_fq.ko 2023-10-14T16:24:26.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_gred.ko 2023-10-14T16:24:26.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hfsc.ko 2023-10-14T16:24:26.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_hhf.ko 2023-10-14T16:24:26.837 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_htb.ko 2023-10-14T16:24:26.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_ingress.ko 2023-10-14T16:24:26.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio.ko 2023-10-14T16:24:26.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_mqprio_lib.ko 2023-10-14T16:24:26.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_multiq.ko 2023-10-14T16:24:26.838 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_netem.ko 2023-10-14T16:24:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_pie.ko 2023-10-14T16:24:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_plug.ko 2023-10-14T16:24:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_prio.ko 2023-10-14T16:24:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_qfq.ko 2023-10-14T16:24:26.839 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_red.ko 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfb.ko 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_sfq.ko 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_tbf.ko 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sched/sch_teql.ko 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp 2023-10-14T16:24:26.840 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp.ko 2023-10-14T16:24:26.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sctp/sctp_diag.ko 2023-10-14T16:24:26.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc 2023-10-14T16:24:26.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss 2023-10-14T16:24:26.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2023-10-14T16:24:26.841 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/sunrpc.ko 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/diag.ko 2023-10-14T16:24:26.842 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/tipc/tipc.ko 2023-10-14T16:24:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix 2023-10-14T16:24:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/unix/unix_diag.ko 2023-10-14T16:24:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock 2023-10-14T16:24:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2023-10-14T16:24:26.843 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock.ko 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_diag.ko 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/vmw_vsock/vsock_loopback.ko 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/net/xfrm/xfrm_ipcomp.ko 2023-10-14T16:24:26.844 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security 2023-10-14T16:24:26.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys 2023-10-14T16:24:26.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys 2023-10-14T16:24:26.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2023-10-14T16:24:26.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys 2023-10-14T16:24:26.845 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/security/keys/trusted-keys/trusted.ko 2023-10-14T16:24:26.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt 2023-10-14T16:24:26.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib 2023-10-14T16:24:26.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/kernel/virt/lib/irqbypass.ko 2023-10-14T16:24:26.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias 2023-10-14T16:24:26.846 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.alias.bin 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.alias.bin 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.bin 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.builtin.modinfo 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep 2023-10-14T16:24:26.847 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.dep.bin 2023-10-14T16:24:26.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.devname 2023-10-14T16:24:26.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.order 2023-10-14T16:24:26.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.softdep 2023-10-14T16:24:26.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols 2023-10-14T16:24:26.848 INFO:teuthology.orchestra.run.smithi029.stdout:/lib/modules/6.6.0-rc5-g47672119149b/modules.symbols.bin 2023-10-14T16:24:26.849 DEBUG:teuthology.task.kernel:get_image_version: 6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.849 DEBUG:teuthology.orchestra.run.smithi029:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2023-10-14T16:24:26.879 INFO:teuthology.orchestra.run.smithi029.stdout:postinstall scriptlet (using /bin/sh): 2023-10-14T16:24:26.879 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/installkernel -a -r /boot/vmlinuz-6.6.0-rc5-g47672119149b -a -r /boot/System.map-6.6.0-rc5-g47672119149b ]; then 2023-10-14T16:24:26.879 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:24:26.880 INFO:teuthology.orchestra.run.smithi029.stdout:cp /boot/System.map-6.6.0-rc5-g47672119149b /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:24:26.880 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/vmlinuz-6.6.0-rc5-g47672119149b /boot/System.map-6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.880 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/installkernel 6.6.0-rc5-g47672119149b /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:24:26.880 INFO:teuthology.orchestra.run.smithi029.stdout:rm -f /boot/.vmlinuz-6.6.0-rc5-g47672119149b-rpm /boot/.System.map-6.6.0-rc5-g47672119149b-rpm 2023-10-14T16:24:26.880 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-14T16:24:26.881 INFO:teuthology.orchestra.run.smithi029.stdout:preuninstall scriptlet (using /bin/sh): 2023-10-14T16:24:26.881 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2023-10-14T16:24:26.881 INFO:teuthology.orchestra.run.smithi029.stdout:new-kernel-pkg --remove 6.6.0-rc5-g47672119149b --rminitrd --initrdfile=/boot/initramfs-6.6.0-rc5-g47672119149b.img 2023-10-14T16:24:26.881 INFO:teuthology.orchestra.run.smithi029.stdout:elif [ -x /usr/bin/kernel-install ]; then 2023-10-14T16:24:26.881 INFO:teuthology.orchestra.run.smithi029.stdout:kernel-install remove 6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-14T16:24:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:postuninstall scriptlet (using /bin/sh): 2023-10-14T16:24:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:if [ -x /sbin/update-bootloader ]; then 2023-10-14T16:24:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:/sbin/update-bootloader --remove 6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.882 INFO:teuthology.orchestra.run.smithi029.stdout:fi 2023-10-14T16:24:26.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo rpm -qi grub2-tools 2023-10-14T16:24:26.961 INFO:teuthology.orchestra.run.smithi029.stdout:Name : grub2-tools 2023-10-14T16:24:26.961 INFO:teuthology.orchestra.run.smithi029.stdout:Epoch : 1 2023-10-14T16:24:26.961 INFO:teuthology.orchestra.run.smithi029.stdout:Version : 2.06 2023-10-14T16:24:26.962 INFO:teuthology.orchestra.run.smithi029.stdout:Release : 68.el9 2023-10-14T16:24:26.962 INFO:teuthology.orchestra.run.smithi029.stdout:Architecture: x86_64 2023-10-14T16:24:26.962 INFO:teuthology.orchestra.run.smithi029.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2023-10-14T16:24:26.962 INFO:teuthology.orchestra.run.smithi029.stdout:Group : Unspecified 2023-10-14T16:24:26.962 INFO:teuthology.orchestra.run.smithi029.stdout:Size : 8264254 2023-10-14T16:24:26.963 INFO:teuthology.orchestra.run.smithi029.stdout:License : GPLv3+ 2023-10-14T16:24:26.963 INFO:teuthology.orchestra.run.smithi029.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2023-10-14T16:24:26.963 INFO:teuthology.orchestra.run.smithi029.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2023-10-14T16:24:26.963 INFO:teuthology.orchestra.run.smithi029.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2023-10-14T16:24:26.963 INFO:teuthology.orchestra.run.smithi029.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2023-10-14T16:24:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:Packager : builder@centos.org 2023-10-14T16:24:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:Vendor : CentOS 2023-10-14T16:24:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-14T16:24:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:Summary : Support tools for GRUB. 2023-10-14T16:24:26.964 INFO:teuthology.orchestra.run.smithi029.stdout:Description : 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout:hardware devices. 2023-10-14T16:24:26.965 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:24:26.966 INFO:teuthology.orchestra.run.smithi029.stdout:This subpackage provides tools for support of all platforms. 2023-10-14T16:24:26.967 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-14T16:24:26.967 INFO:teuthology.task.kernel:Updating grub on smithi029 to boot 6.6.0-rc5-g47672119149b 2023-10-14T16:24:26.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-14T16:24:27.559 INFO:teuthology.task.kernel:Checking client client.1 for new kernel version... 2023-10-14T16:24:27.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:24:27.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:24:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:Generating grub configuration file ... 2023-10-14T16:24:29.431 INFO:teuthology.orchestra.run.smithi029.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2023-10-14T16:24:29.456 INFO:teuthology.orchestra.run.smithi029.stderr:done 2023-10-14T16:24:29.458 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-14T16:24:29.474 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.6cV7BSnw5l 2023-10-14T16:24:29.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6cV7BSnw5l 2023-10-14T16:24:29.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.6cV7BSnw5l 2023-10-14T16:24:29.703 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.6cV7BSnw5l is 6KB 2023-10-14T16:24:29.752 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.6cV7BSnw5l 2023-10-14T16:24:29.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo /bin/ls /boot/loader/entries || true 2023-10-14T16:24:29.835 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2023-10-14T16:24:29.835 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2023-10-14T16:24:29.836 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2023-10-14T16:24:29.836 INFO:teuthology.orchestra.run.smithi029.stdout:15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b.conf 2023-10-14T16:24:29.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.6.0-rc5-g47672119149b 2023-10-14T16:24:29.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo shutdown -r now 2023-10-14T16:24:45.950 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-10-14T16:24:54.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:24:54.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:24:58.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-10-14T16:25:00.001 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2023-10-14T16:25:00.001 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:25:00.002 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:25:10.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:25:10.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:25:18.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.29 2023-10-14T16:25:27.466 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:25:27.467 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:25:28.442 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-10-14T16:25:43.454 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:25:43.455 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:25:46.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.186 2023-10-14T16:26:04.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:26:04.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:26:04.965 DEBUG:teuthology.orchestra.run.smithi186:> true 2023-10-14T16:26:05.644 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:26:05.644 INFO:teuthology.task.kernel:Checking kernel version of client.1, want "6.6.0-rc5-g47672119149b"... 2023-10-14T16:26:05.645 DEBUG:teuthology.orchestra.run.smithi186:> uname -r 2023-10-14T16:26:05.683 INFO:teuthology.orchestra.run.smithi186.stdout:6.6.0-rc5-g47672119149b 2023-10-14T16:26:05.684 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-14T16:26:05.684 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-14T16:26:05.684 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-14T16:26:06.686 DEBUG:teuthology.parallel:result is None 2023-10-14T16:26:27.469 DEBUG:teuthology.orchestra.remote:timed out 2023-10-14T16:26:39.471 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:26:39.472 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:26:40.304 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-14T16:26:41.028 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:26:41.028 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.6.0-rc5-g47672119149b"... 2023-10-14T16:26:41.028 DEBUG:teuthology.orchestra.run.smithi029:> uname -r 2023-10-14T16:26:41.084 INFO:teuthology.orchestra.run.smithi029.stdout:6.6.0-rc5-g47672119149b 2023-10-14T16:26:41.084 DEBUG:teuthology.task.kernel:current kernel version is 6.6.0-rc5-g47672119149b vs 6.6.0-rc5-g47672119149b 2023-10-14T16:26:41.084 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-14T16:26:41.085 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-14T16:26:42.085 DEBUG:teuthology.parallel:result is None 2023-10-14T16:26:42.086 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-14T16:26:42.095 INFO:teuthology.task.internal:Creating test directory... 2023-10-14T16:26:42.096 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-14T16:26:42.105 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-14T16:26:42.167 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-14T16:26:42.174 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-14T16:26:42.180 INFO:teuthology.task.internal:Creating archive directory... 2023-10-14T16:26:42.181 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-14T16:26:42.186 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-14T16:26:42.257 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-14T16:26:42.263 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-14T16:26:42.264 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-14T16:26:42.270 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-14T16:26:42.432 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T16:26:42.439 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T16:26:42.505 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T16:26:42.518 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-14T16:26:42.523 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-14T16:26:42.530 INFO:teuthology.task.internal:Configuring sudo... 2023-10-14T16:26:42.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-14T16:26:42.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-14T16:26:42.639 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-14T16:26:42.648 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-14T16:26:42.649 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-14T16:26:42.658 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-14T16:26:42.714 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T16:26:42.806 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T16:26:42.920 DEBUG:teuthology.orchestra.run.smithi029:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T16:26:43.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T16:26:43.111 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:26:43.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-14T16:26:43.216 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T16:26:43.307 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-14T16:26:43.420 DEBUG:teuthology.orchestra.run.smithi186:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T16:26:43.508 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-14T16:26:43.608 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:26:43.609 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-14T16:26:43.704 DEBUG:teuthology.orchestra.run.smithi029:> sudo service rsyslog restart 2023-10-14T16:26:43.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo service rsyslog restart 2023-10-14T16:26:43.843 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T16:26:43.881 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T16:26:44.192 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-14T16:26:44.200 INFO:teuthology.task.internal:Starting timer... 2023-10-14T16:26:44.200 INFO:teuthology.run_tasks:Running task pcp... 2023-10-14T16:26:44.212 INFO:teuthology.run_tasks:Running task selinux... 2023-10-14T16:26:44.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo service auditd rotate 2023-10-14T16:26:44.417 INFO:teuthology.orchestra.run.smithi029.stdout:Rotating logs: 2023-10-14T16:26:44.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo service auditd rotate 2023-10-14T16:26:44.552 INFO:teuthology.orchestra.run.smithi186.stdout:Rotating logs: 2023-10-14T16:26:44.556 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-14T16:26:44.557 DEBUG:teuthology.orchestra.run.smithi029:> /usr/sbin/getenforce 2023-10-14T16:26:44.623 INFO:teuthology.orchestra.run.smithi029.stdout:Permissive 2023-10-14T16:26:44.623 DEBUG:teuthology.orchestra.run.smithi186:> /usr/sbin/getenforce 2023-10-14T16:26:44.685 INFO:teuthology.orchestra.run.smithi186.stdout:Permissive 2023-10-14T16:26:44.685 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi029.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi186.front.sepia.ceph.com': 'permissive'} 2023-10-14T16:26:44.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T16:26:44.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:26:44.788 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T16:26:44.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:26:44.888 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-14T16:26:44.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo /usr/sbin/setenforce permissive 2023-10-14T16:26:45.004 DEBUG:teuthology.orchestra.run.smithi186:> sudo /usr/sbin/setenforce permissive 2023-10-14T16:26:45.133 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-14T16:26:45.143 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-14T16:26:45.148 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-14T16:26:45.276 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-14T16:26:45.297 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-14T16:26:45.299 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi029.front.sepia.ceph.com,smithi186.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-14T16:37:06.821 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi029.front.sepia.ceph.com'), Remote(name='ubuntu@smithi186.front.sepia.ceph.com')] 2023-10-14T16:37:06.822 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:37:06.823 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi029.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:37:06.965 DEBUG:teuthology.orchestra.run.smithi029:> true 2023-10-14T16:37:07.117 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi029.front.sepia.ceph.com' 2023-10-14T16:37:07.118 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:37:07.119 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi186.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-14T16:37:07.257 DEBUG:teuthology.orchestra.run.smithi186:> true 2023-10-14T16:37:07.421 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi186.front.sepia.ceph.com' 2023-10-14T16:37:07.422 INFO:teuthology.run_tasks:Running task clock... 2023-10-14T16:37:07.435 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-14T16:37:07.435 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T16:37:07.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T16:37:07.445 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-14T16:37:07.445 DEBUG:teuthology.orchestra.run.smithi186:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T16:37:07.554 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-14T16:37:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-14T16:37:07.608 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-14T16:37:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-14T16:37:07.692 INFO:teuthology.orchestra.run.smithi186.stderr:sudo: ntpd: command not found 2023-10-14T16:37:07.712 INFO:teuthology.orchestra.run.smithi029.stderr:sudo: ntpd: command not found 2023-10-14T16:37:07.732 INFO:teuthology.orchestra.run.smithi186.stdout:506 Cannot talk to daemon 2023-10-14T16:37:07.756 INFO:teuthology.orchestra.run.smithi029.stdout:506 Cannot talk to daemon 2023-10-14T16:37:07.786 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-14T16:37:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-14T16:37:07.841 INFO:teuthology.orchestra.run.smithi186.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-14T16:37:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-14T16:37:07.963 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2023-10-14T16:37:07.973 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T16:37:07.973 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2023-10-14T16:37:07.974 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:07.974 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:07.974 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:07.974 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:08.003 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-10-14T16:37:08.012 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T16:37:08.012 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-14T16:37:08.013 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:08.013 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:08.013 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:08.013 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T16:37:08.014 INFO:teuthology.run_tasks:Running task install... 2023-10-14T16:37:08.022 DEBUG:teuthology.task.install:project ceph 2023-10-14T16:37:08.023 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}} 2023-10-14T16:37:08.023 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'} 2023-10-14T16:37:08.023 INFO:teuthology.task.install:Using flavor: default 2023-10-14T16:37:08.029 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-10-14T16:37:08.029 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']} 2023-10-14T16:37:08.030 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-14T16:37:08.030 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-14T16:37:08.032 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'tag': None, 'wait_for_package': False} 2023-10-14T16:37:08.033 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-14T16:37:08.179 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/ 2023-10-14T16:37:08.179 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-14T16:37:08.184 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/ 2023-10-14T16:37:08.184 INFO:teuthology.task.install.rpm:Package version is 18.0.0-6675.gc7471068 2023-10-14T16:37:08.292 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-14T16:37:08.292 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:37:08.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-14T16:37:08.349 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments/c7471068426370a3754c2250e7729280233d552b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-10-14T16:37:08.349 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:37:08.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-10-14T16:37:08.399 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-14T16:37:08.400 DEBUG:teuthology.orchestra.run.smithi186:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-14T16:37:08.456 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2023-10-14T16:37:08.457 DEBUG:teuthology.orchestra.run.smithi029:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c7471068426370a3754c2250e7729280233d552b/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-10-14T16:37:08.541 DEBUG:teuthology.orchestra.run.smithi186:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-14T16:37:08.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-10-14T16:37:08.677 DEBUG:teuthology.orchestra.run.smithi186:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-14T16:37:08.740 DEBUG:teuthology.orchestra.run.smithi029:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-10-14T16:37:08.769 INFO:teuthology.orchestra.run.smithi186.stdout:check_obsoletes = 1 2023-10-14T16:37:08.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum clean all 2023-10-14T16:37:08.830 INFO:teuthology.orchestra.run.smithi029.stdout:check_obsoletes = 1 2023-10-14T16:37:08.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum clean all 2023-10-14T16:37:09.128 INFO:teuthology.orchestra.run.smithi186.stdout:70 files removed 2023-10-14T16:37:09.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-14T16:37:09.195 INFO:teuthology.orchestra.run.smithi029.stdout:70 files removed 2023-10-14T16:37:09.235 DEBUG:teuthology.orchestra.run.smithi029:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2023-10-14T16:37:10.222 INFO:teuthology.orchestra.run.smithi186.stdout:ceph packages for x86_64 136 kB/s | 78 kB 00:00 2023-10-14T16:37:10.250 INFO:teuthology.orchestra.run.smithi029.stdout:ceph packages for x86_64 158 kB/s | 78 kB 00:00 2023-10-14T16:37:10.714 INFO:teuthology.orchestra.run.smithi186.stdout:ceph noarch packages 24 kB/s | 9.5 kB 00:00 2023-10-14T16:37:10.728 INFO:teuthology.orchestra.run.smithi029.stdout:ceph noarch packages 25 kB/s | 9.5 kB 00:00 2023-10-14T16:37:11.246 INFO:teuthology.orchestra.run.smithi029.stdout:ceph source packages 4.2 kB/s | 1.9 kB 00:00 2023-10-14T16:37:11.397 INFO:teuthology.orchestra.run.smithi186.stdout:ceph source packages 3.1 kB/s | 1.9 kB 00:00 2023-10-14T16:37:12.026 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - AppStream 8.9 MB/s | 6.3 MB 00:00 2023-10-14T16:37:12.085 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - AppStream 10 MB/s | 6.3 MB 00:00 2023-10-14T16:37:13.644 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2023-10-14T16:37:13.719 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS-9 - BaseOS 4.6 MB/s | 2.0 MB 00:00 2023-10-14T16:37:14.675 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 7.9 MB 00:00 2023-10-14T16:37:15.265 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - BaseOS 6.8 MB/s | 7.9 MB 00:01 2023-10-14T16:37:17.283 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - AppStream 20 MB/s | 18 MB 00:00 2023-10-14T16:37:18.195 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - AppStream 7.6 MB/s | 18 MB 00:02 2023-10-14T16:37:21.717 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 5.7 MB 00:00 2023-10-14T16:37:23.368 INFO:teuthology.orchestra.run.smithi186.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 14 kB 00:00 2023-10-14T16:37:23.422 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - CRB 3.8 MB/s | 5.7 MB 00:01 2023-10-14T16:37:24.717 INFO:teuthology.orchestra.run.smithi186.stdout:Extra Packages for Enterprise Linux 16 MB/s | 19 MB 00:01 2023-10-14T16:37:25.140 INFO:teuthology.orchestra.run.smithi029.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 14 kB 00:00 2023-10-14T16:37:27.195 INFO:teuthology.orchestra.run.smithi029.stdout:Extra Packages for Enterprise Linux 10 MB/s | 19 MB 00:01 2023-10-14T16:37:29.878 INFO:teuthology.orchestra.run.smithi186.stdout:lab-extras 19 kB/s | 1.7 kB 00:00 2023-10-14T16:37:32.424 INFO:teuthology.orchestra.run.smithi186.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-14T16:37:32.425 INFO:teuthology.orchestra.run.smithi186.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-14T16:37:32.445 INFO:teuthology.orchestra.run.smithi186.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-14T16:37:32.446 INFO:teuthology.orchestra.run.smithi186.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-14T16:37:32.448 INFO:teuthology.orchestra.run.smithi186.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-14T16:37:32.449 INFO:teuthology.orchestra.run.smithi186.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-14T16:37:32.452 INFO:teuthology.orchestra.run.smithi186.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-14T16:37:32.455 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-14T16:37:32.457 INFO:teuthology.orchestra.run.smithi186.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-14T16:37:32.536 INFO:teuthology.orchestra.run.smithi186.stdout:Dependencies resolved. 2023-10-14T16:37:32.560 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2023-10-14T16:37:32.560 INFO:teuthology.orchestra.run.smithi186.stdout: Package Arch Version Repository Size 2023-10-14T16:37:32.560 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2023-10-14T16:37:32.560 INFO:teuthology.orchestra.run.smithi186.stdout:Installing: 2023-10-14T16:37:32.561 INFO:teuthology.orchestra.run.smithi186.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-14T16:37:32.561 INFO:teuthology.orchestra.run.smithi186.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el9 ceph 6.5 k 2023-10-14T16:37:32.561 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el9 ceph 5.2 M 2023-10-14T16:37:32.561 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 855 k 2023-10-14T16:37:32.561 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el9 ceph 141 k 2023-10-14T16:37:32.562 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el9 ceph 1.5 M 2023-10-14T16:37:32.562 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 132 k 2023-10-14T16:37:32.562 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 1.8 M 2023-10-14T16:37:32.562 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 7.4 M 2023-10-14T16:37:32.563 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 47 k 2023-10-14T16:37:32.563 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 8.7 M 2023-10-14T16:37:32.563 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el9 ceph 41 M 2023-10-14T16:37:32.563 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 262 k 2023-10-14T16:37:32.563 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 269 k 2023-10-14T16:37:32.564 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 218 k 2023-10-14T16:37:32.564 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 18 k 2023-10-14T16:37:32.564 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-14T16:37:32.565 INFO:teuthology.orchestra.run.smithi186.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-14T16:37:32.565 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-14T16:37:32.565 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 31 k 2023-10-14T16:37:32.565 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 715 k 2023-10-14T16:37:32.566 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-14T16:37:32.566 INFO:teuthology.orchestra.run.smithi186.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-14T16:37:32.566 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-14T16:37:32.566 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-14T16:37:32.567 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-14T16:37:32.567 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-14T16:37:32.567 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-14T16:37:32.568 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el9 ceph 162 k 2023-10-14T16:37:32.568 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el9 ceph 320 k 2023-10-14T16:37:32.568 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 297 k 2023-10-14T16:37:32.568 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 99 k 2023-10-14T16:37:32.568 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 85 k 2023-10-14T16:37:32.569 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-14T16:37:32.569 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 168 k 2023-10-14T16:37:32.569 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-14T16:37:32.569 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrading: 2023-10-14T16:37:32.570 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-14T16:37:32.570 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-14T16:37:32.570 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-14T16:37:32.570 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-14T16:37:32.570 INFO:teuthology.orchestra.run.smithi186.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-14T16:37:32.571 INFO:teuthology.orchestra.run.smithi186.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-14T16:37:32.571 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-14T16:37:32.571 INFO:teuthology.orchestra.run.smithi186.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.3 M 2023-10-14T16:37:32.571 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-14T16:37:32.572 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-14T16:37:32.572 INFO:teuthology.orchestra.run.smithi186.stdout:Installing dependencies: 2023-10-14T16:37:32.572 INFO:teuthology.orchestra.run.smithi186.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-14T16:37:32.572 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-14T16:37:32.572 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 19 M 2023-10-14T16:37:32.573 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 23 k 2023-10-14T16:37:32.573 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el9 ceph 2.1 M 2023-10-14T16:37:32.573 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 243 k 2023-10-14T16:37:32.573 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.4 M 2023-10-14T16:37:32.573 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 18 M 2023-10-14T16:37:32.574 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 15 k 2023-10-14T16:37:32.574 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el9 ceph 24 k 2023-10-14T16:37:32.574 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-14T16:37:32.574 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-14T16:37:32.574 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-14T16:37:32.575 INFO:teuthology.orchestra.run.smithi186.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-14T16:37:32.575 INFO:teuthology.orchestra.run.smithi186.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-14T16:37:32.575 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-14T16:37:32.575 INFO:teuthology.orchestra.run.smithi186.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-14T16:37:32.576 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-14T16:37:32.576 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-14T16:37:32.576 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el9 ceph 163 k 2023-10-14T16:37:32.576 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-14T16:37:32.577 INFO:teuthology.orchestra.run.smithi186.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-14T16:37:32.577 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-14T16:37:32.577 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-14T16:37:32.577 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-14T16:37:32.578 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 473 k 2023-10-14T16:37:32.578 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-14T16:37:32.578 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.8 M 2023-10-14T16:37:32.578 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-14T16:37:32.579 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-14T16:37:32.580 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-14T16:37:32.580 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-14T16:37:32.580 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-14T16:37:32.580 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-14T16:37:32.580 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-14T16:37:32.581 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 45 k 2023-10-14T16:37:32.581 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-14T16:37:32.581 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-14T16:37:32.581 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-14T16:37:32.581 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-14T16:37:32.582 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-14T16:37:32.582 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-14T16:37:32.582 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-14T16:37:32.582 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-14T16:37:32.582 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-14T16:37:32.583 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-14T16:37:32.583 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-14T16:37:32.583 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-14T16:37:32.583 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-14T16:37:32.584 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-14T16:37:32.584 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-14T16:37:32.584 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-14T16:37:32.584 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-14T16:37:32.584 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-14T16:37:32.585 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-14T16:37:32.585 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-14T16:37:32.585 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-14T16:37:32.585 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-14T16:37:32.586 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-14T16:37:32.586 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-14T16:37:32.586 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-14T16:37:32.586 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-14T16:37:32.586 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-14T16:37:32.587 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-14T16:37:32.587 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-14T16:37:32.587 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-14T16:37:32.587 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-14T16:37:32.587 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-14T16:37:32.588 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-14T16:37:32.588 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-14T16:37:32.588 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-14T16:37:32.589 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-14T16:37:32.589 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-14T16:37:32.589 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-14T16:37:32.589 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-14T16:37:32.589 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-14T16:37:32.590 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-14T16:37:32.590 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-14T16:37:32.590 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-14T16:37:32.590 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-14T16:37:32.590 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-14T16:37:32.591 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-14T16:37:32.591 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-14T16:37:32.591 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-14T16:37:32.591 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-14T16:37:32.591 INFO:teuthology.orchestra.run.smithi186.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-14T16:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-14T16:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-14T16:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-14T16:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-14T16:37:32.592 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-14T16:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout:Installing weak dependencies: 2023-10-14T16:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-14T16:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction Summary 2023-10-14T16:37:32.593 INFO:teuthology.orchestra.run.smithi186.stdout:============================================================================================ 2023-10-14T16:37:32.594 INFO:teuthology.orchestra.run.smithi186.stdout:Install 133 Packages 2023-10-14T16:37:32.594 INFO:teuthology.orchestra.run.smithi186.stdout:Upgrade 10 Packages 2023-10-14T16:37:32.594 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:37:32.595 INFO:teuthology.orchestra.run.smithi186.stdout:Total download size: 200 M 2023-10-14T16:37:32.595 INFO:teuthology.orchestra.run.smithi186.stdout:Downloading Packages: 2023-10-14T16:37:32.613 INFO:teuthology.orchestra.run.smithi029.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2023-10-14T16:37:33.288 INFO:teuthology.orchestra.run.smithi186.stdout:(1/143): ceph-18.0.0-6675.gc7471068.el9.x86_64. 22 kB/s | 6.5 kB 00:00 2023-10-14T16:37:33.680 INFO:teuthology.orchestra.run.smithi186.stdout:(2/143): ceph-base-18.0.0-6675.gc7471068.el9.x8 7.5 MB/s | 5.2 MB 00:00 2023-10-14T16:37:33.713 INFO:teuthology.orchestra.run.smithi186.stdout:(3/143): ceph-fuse-18.0.0-6675.gc7471068.el9.x8 2.0 MB/s | 855 kB 00:00 2023-10-14T16:37:33.755 INFO:teuthology.orchestra.run.smithi186.stdout:(4/143): ceph-immutable-object-cache-18.0.0-667 1.9 MB/s | 141 kB 00:00 2023-10-14T16:37:33.931 INFO:teuthology.orchestra.run.smithi186.stdout:(5/143): ceph-mds-18.0.0-6675.gc7471068.el9.x86 9.6 MB/s | 2.1 MB 00:00 2023-10-14T16:37:34.056 INFO:teuthology.orchestra.run.smithi186.stdout:(6/143): ceph-mgr-18.0.0-6675.gc7471068.el9.x86 4.9 MB/s | 1.5 MB 00:00 2023-10-14T16:37:34.348 INFO:teuthology.orchestra.run.smithi186.stdout:(7/143): ceph-mon-18.0.0-6675.gc7471068.el9.x86 11 MB/s | 4.4 MB 00:00 2023-10-14T16:37:35.265 INFO:teuthology.orchestra.run.smithi186.stdout:(8/143): ceph-common-18.0.0-6675.gc7471068.el9. 8.3 MB/s | 19 MB 00:02 2023-10-14T16:37:35.293 INFO:teuthology.orchestra.run.smithi029.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-14T16:37:35.295 INFO:teuthology.orchestra.run.smithi029.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2023-10-14T16:37:35.317 INFO:teuthology.orchestra.run.smithi029.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2023-10-14T16:37:35.318 INFO:teuthology.orchestra.run.smithi029.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2023-10-14T16:37:35.320 INFO:teuthology.orchestra.run.smithi029.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2023-10-14T16:37:35.321 INFO:teuthology.orchestra.run.smithi029.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2023-10-14T16:37:35.324 INFO:teuthology.orchestra.run.smithi029.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2023-10-14T16:37:35.328 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2023-10-14T16:37:35.330 INFO:teuthology.orchestra.run.smithi029.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2023-10-14T16:37:35.382 INFO:teuthology.orchestra.run.smithi186.stdout:(9/143): ceph-radosgw-18.0.0-6675.gc7471068.el9 8.4 MB/s | 8.7 MB 00:01 2023-10-14T16:37:35.413 INFO:teuthology.orchestra.run.smithi029.stdout:Dependencies resolved. 2023-10-14T16:37:35.432 INFO:teuthology.orchestra.run.smithi186.stdout:(10/143): ceph-selinux-18.0.0-6675.gc7471068.el 144 kB/s | 24 kB 00:00 2023-10-14T16:37:35.440 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-14T16:37:35.440 INFO:teuthology.orchestra.run.smithi029.stdout: Package Arch Version Repository Size 2023-10-14T16:37:35.440 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-14T16:37:35.441 INFO:teuthology.orchestra.run.smithi029.stdout:Installing: 2023-10-14T16:37:35.441 INFO:teuthology.orchestra.run.smithi029.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2023-10-14T16:37:35.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph x86_64 2:18.0.0-6675.gc7471068.el9 ceph 6.5 k 2023-10-14T16:37:35.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base x86_64 2:18.0.0-6675.gc7471068.el9 ceph 5.2 M 2023-10-14T16:37:35.441 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 855 k 2023-10-14T16:37:35.442 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache x86_64 2:18.0.0-6675.gc7471068.el9 ceph 141 k 2023-10-14T16:37:35.442 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr x86_64 2:18.0.0-6675.gc7471068.el9 ceph 1.5 M 2023-10-14T16:37:35.442 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 132 k 2023-10-14T16:37:35.442 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 1.8 M 2023-10-14T16:37:35.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 7.4 M 2023-10-14T16:37:35.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 47 k 2023-10-14T16:37:35.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 8.7 M 2023-10-14T16:37:35.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test x86_64 2:18.0.0-6675.gc7471068.el9 ceph 41 M 2023-10-14T16:37:35.443 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 262 k 2023-10-14T16:37:35.444 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 269 k 2023-10-14T16:37:35.444 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 218 k 2023-10-14T16:37:35.444 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 18 k 2023-10-14T16:37:35.444 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel x86_64 0.189-3.el9 appstream 22 k 2023-10-14T16:37:35.444 INFO:teuthology.orchestra.run.smithi029.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2023-10-14T16:37:35.445 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2023-10-14T16:37:35.445 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 31 k 2023-10-14T16:37:35.446 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 715 k 2023-10-14T16:37:35.446 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-14T16:37:35.446 INFO:teuthology.orchestra.run.smithi029.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2023-10-14T16:37:35.446 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel x86_64 2.37.4-15.el9 appstream 26 k 2023-10-14T16:37:35.447 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel x86_64 1:3.0.7-24.el9 appstream 4.1 M 2023-10-14T16:37:35.447 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2023-10-14T16:37:35.447 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2023-10-14T16:37:35.447 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2023-10-14T16:37:35.448 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs x86_64 2:18.0.0-6675.gc7471068.el9 ceph 162 k 2023-10-14T16:37:35.448 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados x86_64 2:18.0.0-6675.gc7471068.el9 ceph 320 k 2023-10-14T16:37:35.448 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 297 k 2023-10-14T16:37:35.448 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw x86_64 2:18.0.0-6675.gc7471068.el9 ceph 99 k 2023-10-14T16:37:35.448 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 85 k 2023-10-14T16:37:35.449 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-14T16:37:35.449 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 168 k 2023-10-14T16:37:35.449 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel x86_64 5.19.0-4.el9 appstream 86 k 2023-10-14T16:37:35.449 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrading: 2023-10-14T16:37:35.449 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager x86_64 1:1.45.4-1.el9 CentOS-BaseOS 2.2 M 2023-10-14T16:37:35.450 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm x86_64 1:1.45.4-1.el9 CentOS-BaseOS 1.8 M 2023-10-14T16:37:35.450 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team x86_64 1:1.45.4-1.el9 CentOS-BaseOS 41 k 2023-10-14T16:37:35.450 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui x86_64 1:1.45.4-1.el9 CentOS-BaseOS 244 k 2023-10-14T16:37:35.450 INFO:teuthology.orchestra.run.smithi029.stdout: acl x86_64 2.3.1-4.el9 CentOS-BaseOS 69 k 2023-10-14T16:37:35.451 INFO:teuthology.orchestra.run.smithi029.stdout: libacl x86_64 2.3.1-4.el9 CentOS-BaseOS 22 k 2023-10-14T16:37:35.451 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel x86_64 2.3.1-4.el9 CentOS-AppStream 68 k 2023-10-14T16:37:35.451 INFO:teuthology.orchestra.run.smithi029.stdout: librados2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.3 M 2023-10-14T16:37:35.451 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 3.0 M 2023-10-14T16:37:35.451 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2023-10-14T16:37:35.452 INFO:teuthology.orchestra.run.smithi029.stdout:Installing dependencies: 2023-10-14T16:37:35.452 INFO:teuthology.orchestra.run.smithi029.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2023-10-14T16:37:35.452 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2023-10-14T16:37:35.452 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 19 M 2023-10-14T16:37:35.452 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 23 k 2023-10-14T16:37:35.453 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds x86_64 2:18.0.0-6675.gc7471068.el9 ceph 2.1 M 2023-10-14T16:37:35.453 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 243 k 2023-10-14T16:37:35.453 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.4 M 2023-10-14T16:37:35.453 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd x86_64 2:18.0.0-6675.gc7471068.el9 ceph 18 M 2023-10-14T16:37:35.453 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts noarch 2:18.0.0-6675.gc7471068.el9 ceph-noarch 15 k 2023-10-14T16:37:35.454 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux x86_64 2:18.0.0-6675.gc7471068.el9 ceph 24 k 2023-10-14T16:37:35.454 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2023-10-14T16:37:35.454 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2023-10-14T16:37:35.454 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2023-10-14T16:37:35.455 INFO:teuthology.orchestra.run.smithi029.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2023-10-14T16:37:35.455 INFO:teuthology.orchestra.run.smithi029.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2023-10-14T16:37:35.455 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2023-10-14T16:37:35.455 INFO:teuthology.orchestra.run.smithi029.stdout: jq x86_64 1.6-15.el9 appstream 187 k 2023-10-14T16:37:35.455 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow x86_64 9.0.0-9.el9 epel 4.4 M 2023-10-14T16:37:35.456 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc noarch 9.0.0-9.el9 epel 24 k 2023-10-14T16:37:35.456 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite x86_64 2:18.0.0-6675.gc7471068.el9 ceph 163 k 2023-10-14T16:37:35.456 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran x86_64 11.4.1-2.3.el9 CentOS-BaseOS 803 k 2023-10-14T16:37:35.456 INFO:teuthology.orchestra.run.smithi029.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2023-10-14T16:37:35.456 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2023-10-14T16:37:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath x86_64 11.4.1-2.3.el9 CentOS-BaseOS 195 k 2023-10-14T16:37:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2023-10-14T16:37:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 473 k 2023-10-14T16:37:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2023-10-14T16:37:35.457 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2 x86_64 2:18.0.0-6675.gc7471068.el9 ceph 4.8 M 2023-10-14T16:37:35.458 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2023-10-14T16:37:35.458 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2023-10-14T16:37:35.458 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2023-10-14T16:37:35.458 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2023-10-14T16:37:35.459 INFO:teuthology.orchestra.run.smithi029.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2023-10-14T16:37:35.459 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2023-10-14T16:37:35.459 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs x86_64 9.0.0-9.el9 epel 837 k 2023-10-14T16:37:35.459 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2023-10-14T16:37:35.459 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh noarch 2.13.2-3.el9 epel 548 k 2023-10-14T16:37:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2023-10-14T16:37:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2023-10-14T16:37:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2023-10-14T16:37:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse x86_64 2:18.0.0-6675.gc7471068.el9 ceph 45 k 2023-10-14T16:37:35.460 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common x86_64 2:18.0.0-6675.gc7471068.el9 ceph 127 k 2023-10-14T16:37:35.461 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2023-10-14T16:37:35.461 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2023-10-14T16:37:35.461 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2023-10-14T16:37:35.461 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2023-10-14T16:37:35.461 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2023-10-14T16:37:35.462 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel x86_64 3.9.17-2.el9 CentOS-AppStream 205 k 2023-10-14T16:37:35.462 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth noarch 1:2.23.2-1.el9 epel 196 k 2023-10-14T16:37:35.462 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2023-10-14T16:37:35.462 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2023-10-14T16:37:35.463 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2023-10-14T16:37:35.463 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2023-10-14T16:37:35.463 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2023-10-14T16:37:35.463 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2 noarch 2.11.3-4.el9 CentOS-AppStream 228 k 2023-10-14T16:37:35.463 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2023-10-14T16:37:35.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2023-10-14T16:37:35.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2023-10-14T16:37:35.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2023-10-14T16:37:35.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2023-10-14T16:37:35.464 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2023-10-14T16:37:35.465 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2023-10-14T16:37:35.465 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2023-10-14T16:37:35.465 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2023-10-14T16:37:35.465 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2023-10-14T16:37:35.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2023-10-14T16:37:35.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2023-10-14T16:37:35.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2023-10-14T16:37:35.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2023-10-14T16:37:35.466 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2023-10-14T16:37:35.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2023-10-14T16:37:35.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2023-10-14T16:37:35.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2023-10-14T16:37:35.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2023-10-14T16:37:35.467 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2023-10-14T16:37:35.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2023-10-14T16:37:35.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2023-10-14T16:37:35.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests noarch 2.25.1-7.el9 CentOS-BaseOS 113 k 2023-10-14T16:37:35.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2023-10-14T16:37:35.468 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2023-10-14T16:37:35.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2023-10-14T16:37:35.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2023-10-14T16:37:35.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2023-10-14T16:37:35.469 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2023-10-14T16:37:35.470 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2023-10-14T16:37:35.470 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3 noarch 1.26.5-3.el9 CentOS-BaseOS 188 k 2023-10-14T16:37:35.470 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2023-10-14T16:37:35.470 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2023-10-14T16:37:35.470 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2023-10-14T16:37:35.471 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2023-10-14T16:37:35.471 INFO:teuthology.orchestra.run.smithi029.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2023-10-14T16:37:35.471 INFO:teuthology.orchestra.run.smithi029.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2023-10-14T16:37:35.471 INFO:teuthology.orchestra.run.smithi029.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2023-10-14T16:37:35.472 INFO:teuthology.orchestra.run.smithi029.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2023-10-14T16:37:35.472 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2023-10-14T16:37:35.472 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel x86_64 1.2.11-40.el9 appstream 45 k 2023-10-14T16:37:35.472 INFO:teuthology.orchestra.run.smithi029.stdout:Installing weak dependencies: 2023-10-14T16:37:35.472 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2023-10-14T16:37:35.473 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:37:35.473 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction Summary 2023-10-14T16:37:35.473 INFO:teuthology.orchestra.run.smithi029.stdout:============================================================================================ 2023-10-14T16:37:35.473 INFO:teuthology.orchestra.run.smithi029.stdout:Install 133 Packages 2023-10-14T16:37:35.473 INFO:teuthology.orchestra.run.smithi029.stdout:Upgrade 10 Packages 2023-10-14T16:37:35.474 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:37:35.474 INFO:teuthology.orchestra.run.smithi029.stdout:Total download size: 200 M 2023-10-14T16:37:35.475 INFO:teuthology.orchestra.run.smithi029.stdout:Downloading Packages: 2023-10-14T16:37:35.540 INFO:teuthology.orchestra.run.smithi186.stdout:(11/143): cephfs-mirror-18.0.0-6675.gc7471068.e 2.0 MB/s | 218 kB 00:00 2023-10-14T16:37:35.658 INFO:teuthology.orchestra.run.smithi186.stdout:(12/143): libcephfs-devel-18.0.0-6675.gc7471068 269 kB/s | 31 kB 00:00 2023-10-14T16:37:35.825 INFO:teuthology.orchestra.run.smithi186.stdout:(13/143): libcephfs2-18.0.0-6675.gc7471068.el9. 4.2 MB/s | 715 kB 00:00 2023-10-14T16:37:35.917 INFO:teuthology.orchestra.run.smithi186.stdout:(14/143): libcephsqlite-18.0.0-6675.gc7471068.e 1.8 MB/s | 163 kB 00:00 2023-10-14T16:37:36.000 INFO:teuthology.orchestra.run.smithi186.stdout:(15/143): librados-devel-18.0.0-6675.gc7471068. 1.5 MB/s | 127 kB 00:00 2023-10-14T16:37:36.109 INFO:teuthology.orchestra.run.smithi186.stdout:(16/143): libradosstriper1-18.0.0-6675.gc747106 4.3 MB/s | 473 kB 00:00 2023-10-14T16:37:36.238 INFO:teuthology.orchestra.run.smithi029.stdout:(1/143): ceph-18.0.0-6675.gc7471068.el9.x86_64. 18 kB/s | 6.5 kB 00:00 2023-10-14T16:37:36.534 INFO:teuthology.orchestra.run.smithi186.stdout:(17/143): ceph-osd-18.0.0-6675.gc7471068.el9.x8 7.1 MB/s | 18 MB 00:02 2023-10-14T16:37:36.668 INFO:teuthology.orchestra.run.smithi186.stdout:(18/143): python3-ceph-argparse-18.0.0-6675.gc7 341 kB/s | 45 kB 00:00 2023-10-14T16:37:36.751 INFO:teuthology.orchestra.run.smithi186.stdout:(19/143): librgw2-18.0.0-6675.gc7471068.el9.x86 7.5 MB/s | 4.8 MB 00:00 2023-10-14T16:37:36.785 INFO:teuthology.orchestra.run.smithi186.stdout:(20/143): python3-ceph-common-18.0.0-6675.gc747 1.1 MB/s | 127 kB 00:00 2023-10-14T16:37:36.818 INFO:teuthology.orchestra.run.smithi186.stdout:(21/143): python3-cephfs-18.0.0-6675.gc7471068. 2.4 MB/s | 162 kB 00:00 2023-10-14T16:37:36.856 INFO:teuthology.orchestra.run.smithi029.stdout:(2/143): ceph-fuse-18.0.0-6675.gc7471068.el9.x8 1.4 MB/s | 855 kB 00:00 2023-10-14T16:37:36.894 INFO:teuthology.orchestra.run.smithi186.stdout:(22/143): python3-rados-18.0.0-6675.gc7471068.e 2.9 MB/s | 320 kB 00:00 2023-10-14T16:37:36.919 INFO:teuthology.orchestra.run.smithi186.stdout:(23/143): python3-rbd-18.0.0-6675.gc7471068.el9 2.9 MB/s | 297 kB 00:00 2023-10-14T16:37:36.948 INFO:teuthology.orchestra.run.smithi029.stdout:(3/143): ceph-immutable-object-cache-18.0.0-667 1.5 MB/s | 141 kB 00:00 2023-10-14T16:37:36.994 INFO:teuthology.orchestra.run.smithi186.stdout:(24/143): python3-rgw-18.0.0-6675.gc7471068.el9 1.3 MB/s | 99 kB 00:00 2023-10-14T16:37:37.045 INFO:teuthology.orchestra.run.smithi186.stdout:(25/143): rbd-fuse-18.0.0-6675.gc7471068.el9.x8 687 kB/s | 85 kB 00:00 2023-10-14T16:37:37.170 INFO:teuthology.orchestra.run.smithi186.stdout:(26/143): rbd-nbd-18.0.0-6675.gc7471068.el9.x86 1.3 MB/s | 168 kB 00:00 2023-10-14T16:37:37.320 INFO:teuthology.orchestra.run.smithi186.stdout:(27/143): ceph-grafana-dashboards-18.0.0-6675.g 155 kB/s | 23 kB 00:00 2023-10-14T16:37:37.446 INFO:teuthology.orchestra.run.smithi186.stdout:(28/143): ceph-mgr-cephadm-18.0.0-6675.gc747106 1.0 MB/s | 132 kB 00:00 2023-10-14T16:37:37.602 INFO:teuthology.orchestra.run.smithi186.stdout:(29/143): rbd-mirror-18.0.0-6675.gc7471068.el9. 5.0 MB/s | 3.0 MB 00:00 2023-10-14T16:37:37.786 INFO:teuthology.orchestra.run.smithi186.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6675.gc7471 5.2 MB/s | 1.8 MB 00:00 2023-10-14T16:37:37.903 INFO:teuthology.orchestra.run.smithi186.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6675.gc7 2.0 MB/s | 243 kB 00:00 2023-10-14T16:37:37.925 INFO:teuthology.orchestra.run.smithi029.stdout:(4/143): ceph-mds-18.0.0-6675.gc7471068.el9.x86 2.1 MB/s | 2.1 MB 00:00 2023-10-14T16:37:38.004 INFO:teuthology.orchestra.run.smithi186.stdout:(32/143): ceph-mgr-rook-18.0.0-6675.gc7471068.e 479 kB/s | 47 kB 00:00 2023-10-14T16:37:38.121 INFO:teuthology.orchestra.run.smithi186.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6675.gc 126 kB/s | 15 kB 00:00 2023-10-14T16:37:38.251 INFO:teuthology.orchestra.run.smithi029.stdout:(5/143): ceph-base-18.0.0-6675.gc7471068.el9.x8 2.2 MB/s | 5.2 MB 00:02 2023-10-14T16:37:38.418 INFO:teuthology.orchestra.run.smithi029.stdout:(6/143): ceph-mgr-18.0.0-6675.gc7471068.el9.x86 3.0 MB/s | 1.5 MB 00:00 2023-10-14T16:37:38.580 INFO:teuthology.orchestra.run.smithi186.stdout:(34/143): ceph-test-18.0.0-6675.gc7471068.el9.x 13 MB/s | 41 MB 00:03 2023-10-14T16:37:38.638 INFO:teuthology.orchestra.run.smithi186.stdout:(35/143): ceph-volume-18.0.0-6675.gc7471068.el9 508 kB/s | 262 kB 00:00 2023-10-14T16:37:38.671 INFO:teuthology.orchestra.run.smithi186.stdout:(36/143): cephadm-18.0.0-6675.gc7471068.el9.noa 2.9 MB/s | 269 kB 00:00 2023-10-14T16:37:38.714 INFO:teuthology.orchestra.run.smithi186.stdout:(37/143): cephfs-top-18.0.0-6675.gc7471068.el9. 245 kB/s | 18 kB 00:00 2023-10-14T16:37:38.927 INFO:teuthology.orchestra.run.smithi029.stdout:(7/143): ceph-mon-18.0.0-6675.gc7471068.el9.x86 6.5 MB/s | 4.4 MB 00:00 2023-10-14T16:37:38.955 INFO:teuthology.orchestra.run.smithi186.stdout:(38/143): ceph-mgr-diskprediction-local-18.0.0- 5.5 MB/s | 7.4 MB 00:01 2023-10-14T16:37:39.039 INFO:teuthology.orchestra.run.smithi186.stdout:(39/143): boost-program-options-1.75.0-8.el9.x8 414 kB/s | 106 kB 00:00 2023-10-14T16:37:39.123 INFO:teuthology.orchestra.run.smithi186.stdout:(40/143): bison-3.7.4-5.el9.x86_64.rpm 2.0 MB/s | 920 kB 00:00 2023-10-14T16:37:39.156 INFO:teuthology.orchestra.run.smithi186.stdout:(41/143): flex-2.6.4-9.el9.x86_64.rpm 1.5 MB/s | 307 kB 00:00 2023-10-14T16:37:39.198 INFO:teuthology.orchestra.run.smithi186.stdout:(42/143): flexiblas-3.0.4-8.el9.x86_64.rpm 203 kB/s | 32 kB 00:00 2023-10-14T16:37:39.357 INFO:teuthology.orchestra.run.smithi186.stdout:(43/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 13 MB/s | 3.0 MB 00:00 2023-10-14T16:37:39.420 INFO:teuthology.orchestra.run.smithi186.stdout:(44/143): libaio-devel-0.3.111-13.el9.x86_64.rp 47 kB/s | 10 kB 00:00 2023-10-14T16:37:39.474 INFO:teuthology.orchestra.run.smithi186.stdout:(45/143): flexiblas-openblas-openmp-3.0.4-8.el9 64 kB/s | 17 kB 00:00 2023-10-14T16:37:39.507 INFO:teuthology.orchestra.run.smithi186.stdout:(46/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.8 MB/s | 159 kB 00:00 2023-10-14T16:37:39.549 INFO:teuthology.orchestra.run.smithi186.stdout:(47/143): librdkafka-1.6.1-102.el9.x86_64.rpm 5.0 MB/s | 661 kB 00:00 2023-10-14T16:37:39.591 INFO:teuthology.orchestra.run.smithi186.stdout:(48/143): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 578 kB 00:00 2023-10-14T16:37:39.642 INFO:teuthology.orchestra.run.smithi186.stdout:(49/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 538 kB/s | 46 kB 00:00 2023-10-14T16:37:39.667 INFO:teuthology.orchestra.run.smithi186.stdout:(50/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.5 MB/s | 282 kB 00:00 2023-10-14T16:37:39.700 INFO:teuthology.orchestra.run.smithi186.stdout:(51/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.0 MB/s | 217 kB 00:00 2023-10-14T16:37:39.767 INFO:teuthology.orchestra.run.smithi186.stdout:(52/143): openblas-0.3.21-2.el9.x86_64.rpm 351 kB/s | 34 kB 00:00 2023-10-14T16:37:39.893 INFO:teuthology.orchestra.run.smithi186.stdout:(53/143): openblas-openmp-0.3.21-2.el9.x86_64.r 21 MB/s | 4.7 MB 00:00 2023-10-14T16:37:39.951 INFO:teuthology.orchestra.run.smithi186.stdout:(54/143): postgresql-13.11-1.el9.x86_64.rpm 6.4 MB/s | 1.5 MB 00:00 2023-10-14T16:37:39.993 INFO:teuthology.orchestra.run.smithi186.stdout:(55/143): postgresql-contrib-13.11-1.el9.x86_64 3.8 MB/s | 811 kB 00:00 2023-10-14T16:37:40.018 INFO:teuthology.orchestra.run.smithi186.stdout:(56/143): postgresql-private-libs-13.11-1.el9.x 1.1 MB/s | 134 kB 00:00 2023-10-14T16:37:40.019 INFO:teuthology.orchestra.run.smithi029.stdout:(8/143): ceph-radosgw-18.0.0-6675.gc7471068.el9 7.9 MB/s | 8.7 MB 00:01 2023-10-14T16:37:40.069 INFO:teuthology.orchestra.run.smithi029.stdout:(9/143): ceph-selinux-18.0.0-6675.gc7471068.el9 489 kB/s | 24 kB 00:00 2023-10-14T16:37:40.277 INFO:teuthology.orchestra.run.smithi186.stdout:(57/143): postgresql-server-13.11-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2023-10-14T16:37:40.319 INFO:teuthology.orchestra.run.smithi186.stdout:(58/143): python3-devel-3.9.17-2.el9.x86_64.rpm 693 kB/s | 205 kB 00:00 2023-10-14T16:37:40.430 INFO:teuthology.orchestra.run.smithi186.stdout:(59/143): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 5.8 MB 00:00 2023-10-14T16:37:40.453 INFO:teuthology.orchestra.run.smithi186.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 1.4 MB/s | 228 kB 00:00 2023-10-14T16:37:40.557 INFO:teuthology.orchestra.run.smithi186.stdout:(61/143): python3-jsonpatch-1.21-16.el9.noarch. 104 kB/s | 24 kB 00:00 2023-10-14T16:37:40.620 INFO:teuthology.orchestra.run.smithi186.stdout:(62/143): python3-jsonpointer-2.0-4.el9.noarch. 109 kB/s | 18 kB 00:00 2023-10-14T16:37:40.645 INFO:teuthology.orchestra.run.smithi186.stdout:(63/143): python3-mako-1.1.4-6.el9.noarch.rpm 756 kB/s | 144 kB 00:00 2023-10-14T16:37:40.687 INFO:teuthology.orchestra.run.smithi186.stdout:(64/143): python3-markupsafe-1.1.1-12.el9.x86_6 248 kB/s | 32 kB 00:00 2023-10-14T16:37:40.721 INFO:teuthology.orchestra.run.smithi186.stdout:(65/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 217 kB 00:00 2023-10-14T16:37:40.846 INFO:teuthology.orchestra.run.smithi186.stdout:(66/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 4.8 MB 00:00 2023-10-14T16:37:40.871 INFO:teuthology.orchestra.run.smithi186.stdout:(67/143): python3-oauthlib-3.1.1-5.el9.noarch.r 965 kB/s | 174 kB 00:00 2023-10-14T16:37:40.913 INFO:teuthology.orchestra.run.smithi186.stdout:(68/143): python3-prettytable-0.7.2-27.el9.noar 231 kB/s | 41 kB 00:00 2023-10-14T16:37:40.955 INFO:teuthology.orchestra.run.smithi186.stdout:(69/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.2 MB/s | 132 kB 00:00 2023-10-14T16:37:41.014 INFO:teuthology.orchestra.run.smithi186.stdout:(70/143): python3-requests-oauthlib-1.3.0-12.el 435 kB/s | 43 kB 00:00 2023-10-14T16:37:41.047 INFO:teuthology.orchestra.run.smithi186.stdout:(71/143): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 211 kB 00:00 2023-10-14T16:37:41.089 INFO:teuthology.orchestra.run.smithi186.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 534 kB/s | 39 kB 00:00 2023-10-14T16:37:41.162 INFO:teuthology.orchestra.run.smithi029.stdout:(10/143): ceph-common-18.0.0-6675.gc7471068.el9 3.6 MB/s | 19 MB 00:05 2023-10-14T16:37:41.248 INFO:teuthology.orchestra.run.smithi186.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 1.5 MB/s | 300 kB 00:00 2023-10-14T16:37:41.290 INFO:teuthology.orchestra.run.smithi186.stdout:(74/143): uuid-1.6.2-55.el9.x86_64.rpm 303 kB/s | 56 kB 00:00 2023-10-14T16:37:41.490 INFO:teuthology.orchestra.run.smithi186.stdout:(75/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2023-10-14T16:37:41.520 INFO:teuthology.orchestra.run.smithi029.stdout:(11/143): ceph-osd-18.0.0-6675.gc7471068.el9.x8 5.7 MB/s | 18 MB 00:03 2023-10-14T16:37:41.532 INFO:teuthology.orchestra.run.smithi186.stdout:(76/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 221 kB/s | 63 kB 00:00 2023-10-14T16:37:41.557 INFO:teuthology.orchestra.run.smithi186.stdout:(77/143): fuse-2.9.9-15.el9.x86_64.rpm 299 kB/s | 79 kB 00:00 2023-10-14T16:37:41.562 INFO:teuthology.orchestra.run.smithi029.stdout:(12/143): cephfs-mirror-18.0.0-6675.gc7471068.e 546 kB/s | 218 kB 00:00 2023-10-14T16:37:41.591 INFO:teuthology.orchestra.run.smithi186.stdout:(78/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 3.4 MB/s | 195 kB 00:00 2023-10-14T16:37:41.596 INFO:teuthology.orchestra.run.smithi029.stdout:(13/143): libcephfs-devel-18.0.0-6675.gc7471068 418 kB/s | 31 kB 00:00 2023-10-14T16:37:41.633 INFO:teuthology.orchestra.run.smithi186.stdout:(79/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 8.0 MB/s | 803 kB 00:00 2023-10-14T16:37:41.658 INFO:teuthology.orchestra.run.smithi186.stdout:(80/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.5 MB/s | 241 kB 00:00 2023-10-14T16:37:41.672 INFO:teuthology.orchestra.run.smithi029.stdout:(14/143): libcephsqlite-18.0.0-6675.gc7471068.e 2.1 MB/s | 163 kB 00:00 2023-10-14T16:37:41.683 INFO:teuthology.orchestra.run.smithi186.stdout:(81/143): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2023-10-14T16:37:41.708 INFO:teuthology.orchestra.run.smithi186.stdout:(82/143): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2023-10-14T16:37:41.733 INFO:teuthology.orchestra.run.smithi186.stdout:(83/143): python3-requests-2.25.1-7.el9.noarch. 1.5 MB/s | 113 kB 00:00 2023-10-14T16:37:41.739 INFO:teuthology.orchestra.run.smithi029.stdout:(15/143): librados-devel-18.0.0-6675.gc7471068. 1.9 MB/s | 127 kB 00:00 2023-10-14T16:37:41.790 INFO:teuthology.orchestra.run.smithi029.stdout:(16/143): libcephfs2-18.0.0-6675.gc7471068.el9. 3.1 MB/s | 715 kB 00:00 2023-10-14T16:37:41.797 INFO:teuthology.orchestra.run.smithi186.stdout:(84/143): python3-urllib3-1.26.5-3.el9.noarch.r 1.7 MB/s | 188 kB 00:00 2023-10-14T16:37:41.882 INFO:teuthology.orchestra.run.smithi029.stdout:(17/143): libradosstriper1-18.0.0-6675.gc747106 3.3 MB/s | 473 kB 00:00 2023-10-14T16:37:41.943 INFO:teuthology.orchestra.run.smithi029.stdout:(18/143): python3-ceph-argparse-18.0.0-6675.gc7 762 kB/s | 45 kB 00:00 2023-10-14T16:37:42.001 INFO:teuthology.orchestra.run.smithi029.stdout:(19/143): python3-ceph-common-18.0.0-6675.gc747 2.2 MB/s | 127 kB 00:00 2023-10-14T16:37:42.069 INFO:teuthology.orchestra.run.smithi029.stdout:(20/143): python3-cephfs-18.0.0-6675.gc7471068. 2.4 MB/s | 162 kB 00:00 2023-10-14T16:37:42.084 INFO:teuthology.orchestra.run.smithi186.stdout:(85/143): elfutils-libelf-devel-0.189-3.el9.x86 78 kB/s | 22 kB 00:00 2023-10-14T16:37:42.161 INFO:teuthology.orchestra.run.smithi029.stdout:(21/143): python3-rados-18.0.0-6675.gc7471068.e 3.4 MB/s | 320 kB 00:00 2023-10-14T16:37:42.237 INFO:teuthology.orchestra.run.smithi029.stdout:(22/143): python3-rbd-18.0.0-6675.gc7471068.el9 3.9 MB/s | 297 kB 00:00 2023-10-14T16:37:42.296 INFO:teuthology.orchestra.run.smithi029.stdout:(23/143): python3-rgw-18.0.0-6675.gc7471068.el9 1.7 MB/s | 99 kB 00:00 2023-10-14T16:37:42.354 INFO:teuthology.orchestra.run.smithi029.stdout:(24/143): rbd-fuse-18.0.0-6675.gc7471068.el9.x8 1.4 MB/s | 85 kB 00:00 2023-10-14T16:37:42.453 INFO:teuthology.orchestra.run.smithi186.stdout:(86/143): jq-1.6-15.el9.x86_64.rpm 508 kB/s | 187 kB 00:00 2023-10-14T16:37:42.576 INFO:teuthology.orchestra.run.smithi186.stdout:(87/143): automake-1.16.2-8.el9.noarch.rpm 821 kB/s | 689 kB 00:00 2023-10-14T16:37:42.637 INFO:teuthology.orchestra.run.smithi186.stdout:(88/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 247 kB/s | 45 kB 00:00 2023-10-14T16:37:42.687 INFO:teuthology.orchestra.run.smithi186.stdout:(89/143): python3-cryptography-36.0.1-4.el9.x86 1.3 MB/s | 1.2 MB 00:00 2023-10-14T16:37:42.738 INFO:teuthology.orchestra.run.smithi186.stdout:(90/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 159 kB/s | 26 kB 00:00 2023-10-14T16:37:42.788 INFO:teuthology.orchestra.run.smithi029.stdout:(25/143): librgw2-18.0.0-6675.gc7471068.el9.x86 4.8 MB/s | 4.8 MB 00:00 2023-10-14T16:37:42.854 INFO:teuthology.orchestra.run.smithi186.stdout:(91/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 744 kB/s | 86 kB 00:00 2023-10-14T16:37:42.939 INFO:teuthology.orchestra.run.smithi029.stdout:(26/143): rbd-mirror-18.0.0-6675.gc7471068.el9. 5.2 MB/s | 3.0 MB 00:00 2023-10-14T16:37:42.973 INFO:teuthology.orchestra.run.smithi029.stdout:(27/143): rbd-nbd-18.0.0-6675.gc7471068.el9.x86 921 kB/s | 168 kB 00:00 2023-10-14T16:37:42.988 INFO:teuthology.orchestra.run.smithi186.stdout:(92/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 340 kB/s | 45 kB 00:00 2023-10-14T16:37:43.007 INFO:teuthology.orchestra.run.smithi029.stdout:(28/143): ceph-grafana-dashboards-18.0.0-6675.g 349 kB/s | 23 kB 00:00 2023-10-14T16:37:43.038 INFO:teuthology.orchestra.run.smithi186.stdout:(93/143): python3-pytz-2021.1-5.el9.noarch.rpm 146 kB/s | 51 kB 00:00 2023-10-14T16:37:43.040 INFO:teuthology.orchestra.run.smithi029.stdout:(29/143): ceph-mgr-cephadm-18.0.0-6675.gc747106 1.9 MB/s | 132 kB 00:00 2023-10-14T16:37:43.114 INFO:teuthology.orchestra.run.smithi186.stdout:(94/143): fmt-8.1.1-5.el9.x86_64.rpm 891 kB/s | 111 kB 00:00 2023-10-14T16:37:43.264 INFO:teuthology.orchestra.run.smithi186.stdout:(95/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 6.6 MB/s | 4.1 MB 00:00 2023-10-14T16:37:43.291 INFO:teuthology.orchestra.run.smithi029.stdout:(30/143): ceph-mgr-dashboard-18.0.0-6675.gc7471 6.2 MB/s | 1.8 MB 00:00 2023-10-14T16:37:43.297 INFO:teuthology.orchestra.run.smithi186.stdout:(96/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 1.2 MB/s | 309 kB 00:00 2023-10-14T16:37:43.375 INFO:teuthology.orchestra.run.smithi029.stdout:(31/143): ceph-mgr-modules-core-18.0.0-6675.gc7 2.9 MB/s | 243 kB 00:00 2023-10-14T16:37:43.381 INFO:teuthology.orchestra.run.smithi186.stdout:(97/143): liboath-2.6.7-2.el9.x86_64.rpm 588 kB/s | 49 kB 00:00 2023-10-14T16:37:43.406 INFO:teuthology.orchestra.run.smithi186.stdout:(98/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 172 kB/s | 24 kB 00:00 2023-10-14T16:37:43.427 INFO:teuthology.orchestra.run.smithi029.stdout:(32/143): ceph-mgr-rook-18.0.0-6675.gc7471068.e 938 kB/s | 47 kB 00:00 2023-10-14T16:37:43.431 INFO:teuthology.orchestra.run.smithi186.stdout:(99/143): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2023-10-14T16:37:43.478 INFO:teuthology.orchestra.run.smithi029.stdout:(33/143): ceph-prometheus-alerts-18.0.0-6675.gc 297 kB/s | 15 kB 00:00 2023-10-14T16:37:43.565 INFO:teuthology.orchestra.run.smithi186.stdout:(100/143): python3-asyncssh-2.13.2-3.el9.noarch 4.0 MB/s | 548 kB 00:00 2023-10-14T16:37:43.570 INFO:teuthology.orchestra.run.smithi029.stdout:(34/143): ceph-volume-18.0.0-6675.gc7471068.el9 2.8 MB/s | 262 kB 00:00 2023-10-14T16:37:43.607 INFO:teuthology.orchestra.run.smithi186.stdout:(101/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 1.0 MB/s | 43 kB 00:00 2023-10-14T16:37:43.649 INFO:teuthology.orchestra.run.smithi186.stdout:(102/143): python3-cachetools-4.2.4-1.el9.noarc 787 kB/s | 32 kB 00:00 2023-10-14T16:37:43.654 INFO:teuthology.orchestra.run.smithi029.stdout:(35/143): cephadm-18.0.0-6675.gc7471068.el9.noa 3.2 MB/s | 269 kB 00:00 2023-10-14T16:37:43.699 INFO:teuthology.orchestra.run.smithi186.stdout:(103/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 2.8 MB/s | 837 kB 00:00 2023-10-14T16:37:43.713 INFO:teuthology.orchestra.run.smithi029.stdout:(36/143): cephfs-top-18.0.0-6675.gc7471068.el9. 315 kB/s | 18 kB 00:00 2023-10-14T16:37:43.724 INFO:teuthology.orchestra.run.smithi186.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 201 kB/s | 15 kB 00:00 2023-10-14T16:37:43.800 INFO:teuthology.orchestra.run.smithi186.stdout:(105/143): python3-cheroot-8.6.0-4.el9.noarch.r 2.3 MB/s | 172 kB 00:00 2023-10-14T16:37:43.858 INFO:teuthology.orchestra.run.smithi186.stdout:(106/143): python3-cherrypy-18.6.1-2.el9.noarch 2.6 MB/s | 358 kB 00:00 2023-10-14T16:37:43.892 INFO:teuthology.orchestra.run.smithi186.stdout:(107/143): python3-google-auth-2.23.2-1.el9.noa 2.1 MB/s | 196 kB 00:00 2023-10-14T16:37:43.914 INFO:teuthology.orchestra.run.smithi029.stdout:(37/143): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2023-10-14T16:37:43.917 INFO:teuthology.orchestra.run.smithi186.stdout:(108/143): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2023-10-14T16:37:43.943 INFO:teuthology.orchestra.run.smithi186.stdout:(109/143): python3-jaraco-classes-3.2.1-5.el9.n 358 kB/s | 18 kB 00:00 2023-10-14T16:37:43.956 INFO:teuthology.orchestra.run.smithi029.stdout:(38/143): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2023-10-14T16:37:43.968 INFO:teuthology.orchestra.run.smithi186.stdout:(110/143): python3-jaraco-collections-3.0.0-8.e 468 kB/s | 23 kB 00:00 2023-10-14T16:37:44.006 INFO:teuthology.orchestra.run.smithi029.stdout:(39/143): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 307 kB 00:00 2023-10-14T16:37:44.010 INFO:teuthology.orchestra.run.smithi186.stdout:(111/143): python3-jaraco-functools-3.5.0-2.el9 294 kB/s | 19 kB 00:00 2023-10-14T16:37:44.035 INFO:teuthology.orchestra.run.smithi186.stdout:(112/143): python3-jaraco-text-3.2.0-6.el9.noar 297 kB/s | 20 kB 00:00 2023-10-14T16:37:44.048 INFO:teuthology.orchestra.run.smithi029.stdout:(40/143): flexiblas-3.0.4-8.el9.x86_64.rpm 773 kB/s | 32 kB 00:00 2023-10-14T16:37:44.219 INFO:teuthology.orchestra.run.smithi186.stdout:(113/143): python3-jwt+crypto-2.4.0-1.el9.noarc 43 kB/s | 9.0 kB 00:00 2023-10-14T16:37:44.232 INFO:teuthology.orchestra.run.smithi029.stdout:(41/143): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2023-10-14T16:37:44.244 INFO:teuthology.orchestra.run.smithi186.stdout:(114/143): python3-jwt-2.4.0-1.el9.noarch.rpm 197 kB/s | 41 kB 00:00 2023-10-14T16:37:44.275 INFO:teuthology.orchestra.run.smithi029.stdout:(42/143): flexiblas-openblas-openmp-3.0.4-8.el9 419 kB/s | 17 kB 00:00 2023-10-14T16:37:44.317 INFO:teuthology.orchestra.run.smithi029.stdout:(43/143): libaio-devel-0.3.111-13.el9.x86_64.rp 246 kB/s | 10 kB 00:00 2023-10-14T16:37:44.336 INFO:teuthology.orchestra.run.smithi186.stdout:(115/143): python3-logutils-0.3.5-21.el9.noarch 510 kB/s | 46 kB 00:00 2023-10-14T16:37:44.359 INFO:teuthology.orchestra.run.smithi029.stdout:(44/143): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.8 MB/s | 159 kB 00:00 2023-10-14T16:37:44.386 INFO:teuthology.orchestra.run.smithi186.stdout:(116/143): python3-more-itertools-8.12.0-2.el9. 1.6 MB/s | 79 kB 00:00 2023-10-14T16:37:44.418 INFO:teuthology.orchestra.run.smithi029.stdout:(45/143): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2023-10-14T16:37:44.437 INFO:teuthology.orchestra.run.smithi186.stdout:(117/143): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2023-10-14T16:37:44.554 INFO:teuthology.orchestra.run.smithi186.stdout:(118/143): libarrow-9.0.0-9.el9.x86_64.rpm 3.1 MB/s | 4.4 MB 00:01 2023-10-14T16:37:44.688 INFO:teuthology.orchestra.run.smithi186.stdout:(119/143): python3-kubernetes-26.1.0-2.el9.noar 2.2 MB/s | 1.0 MB 00:00 2023-10-14T16:37:44.713 INFO:teuthology.orchestra.run.smithi186.stdout:(120/143): python3-portend-3.1.0-2.el9.noarch.r 104 kB/s | 16 kB 00:00 2023-10-14T16:37:44.746 INFO:teuthology.orchestra.run.smithi186.stdout:(121/143): python3-pecan-1.4.2-3.el9.noarch.rpm 882 kB/s | 272 kB 00:00 2023-10-14T16:37:44.772 INFO:teuthology.orchestra.run.smithi186.stdout:(122/143): python3-repoze-lru-0.7-16.el9.noarch 532 kB/s | 31 kB 00:00 2023-10-14T16:37:44.797 INFO:teuthology.orchestra.run.smithi186.stdout:(123/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 831 kB/s | 90 kB 00:00 2023-10-14T16:37:44.812 INFO:teuthology.orchestra.run.smithi029.stdout:(46/143): ceph-mgr-diskprediction-local-18.0.0- 4.2 MB/s | 7.4 MB 00:01 2023-10-14T16:37:44.822 INFO:teuthology.orchestra.run.smithi186.stdout:(124/143): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2023-10-14T16:37:44.848 INFO:teuthology.orchestra.run.smithi186.stdout:(125/143): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2023-10-14T16:37:44.858 INFO:teuthology.orchestra.run.smithi029.stdout:(47/143): libtool-2.4.6-45.el9.x86_64.rpm 1.3 MB/s | 578 kB 00:00 2023-10-14T16:37:44.890 INFO:teuthology.orchestra.run.smithi186.stdout:(126/143): python3-tempora-5.0.0-2.el9.noarch.r 392 kB/s | 36 kB 00:00 2023-10-14T16:37:44.892 INFO:teuthology.orchestra.run.smithi029.stdout:(48/143): libzstd-devel-1.5.1-2.el9.x86_64.rpm 633 kB/s | 46 kB 00:00 2023-10-14T16:37:44.936 INFO:teuthology.orchestra.run.smithi029.stdout:(49/143): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.6 MB/s | 282 kB 00:00 2023-10-14T16:37:44.937 INFO:teuthology.orchestra.run.smithi186.stdout:(127/143): python3-typing-extensions-4.4.0-2.el 447 kB/s | 51 kB 00:00 2023-10-14T16:37:44.970 INFO:teuthology.orchestra.run.smithi029.stdout:(50/143): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.9 MB/s | 217 kB 00:00 2023-10-14T16:37:44.973 INFO:teuthology.orchestra.run.smithi186.stdout:(128/143): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2023-10-14T16:37:45.004 INFO:teuthology.orchestra.run.smithi029.stdout:(51/143): openblas-0.3.21-2.el9.x86_64.rpm 529 kB/s | 34 kB 00:00 2023-10-14T16:37:45.007 INFO:teuthology.orchestra.run.smithi186.stdout:(129/143): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2023-10-14T16:37:45.024 INFO:teuthology.orchestra.run.smithi186.stdout:(130/143): python3-zc-lockfile-2.0-10.el9.noarc 404 kB/s | 20 kB 00:00 2023-10-14T16:37:45.083 INFO:teuthology.orchestra.run.smithi186.stdout:(131/143): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2023-10-14T16:37:45.116 INFO:teuthology.orchestra.run.smithi186.stdout:(132/143): python3-werkzeug-2.0.3-3.el9.1.noarc 2.3 MB/s | 427 kB 00:00 2023-10-14T16:37:45.171 INFO:teuthology.orchestra.run.smithi029.stdout:(52/143): openblas-openmp-0.3.21-2.el9.x86_64.r 24 MB/s | 4.7 MB 00:00 2023-10-14T16:37:45.230 INFO:teuthology.orchestra.run.smithi029.stdout:(53/143): postgresql-13.11-1.el9.x86_64.rpm 7.3 MB/s | 1.5 MB 00:00 2023-10-14T16:37:45.281 INFO:teuthology.orchestra.run.smithi029.stdout:(54/143): postgresql-contrib-13.11-1.el9.x86_64 7.4 MB/s | 811 kB 00:00 2023-10-14T16:37:45.314 INFO:teuthology.orchestra.run.smithi029.stdout:(55/143): postgresql-private-libs-13.11-1.el9.x 1.7 MB/s | 134 kB 00:00 2023-10-14T16:37:45.433 INFO:teuthology.orchestra.run.smithi186.stdout:(133/143): librbd1-18.0.0-6675.gc7471068.el9.x8 9.6 MB/s | 3.0 MB 00:00 2023-10-14T16:37:45.484 INFO:teuthology.orchestra.run.smithi186.stdout:(134/143): thrift-0.15.0-2.el9.x86_64.rpm 3.5 MB/s | 1.6 MB 00:00 2023-10-14T16:37:45.508 INFO:teuthology.orchestra.run.smithi186.stdout:(135/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 911 kB/s | 68 kB 00:00 2023-10-14T16:37:45.565 INFO:teuthology.orchestra.run.smithi029.stdout:(56/143): postgresql-server-13.11-1.el9.x86_64. 20 MB/s | 5.7 MB 00:00 2023-10-14T16:37:45.642 INFO:teuthology.orchestra.run.smithi186.stdout:(136/143): librados2-18.0.0-6675.gc7471068.el9. 5.9 MB/s | 3.3 MB 00:00 2023-10-14T16:37:45.757 INFO:teuthology.orchestra.run.smithi029.stdout:(57/143): python3-babel-2.9.1-2.el9.noarch.rpm 14 MB/s | 5.8 MB 00:00 2023-10-14T16:37:45.791 INFO:teuthology.orchestra.run.smithi029.stdout:(58/143): python3-devel-3.9.17-2.el9.x86_64.rpm 976 kB/s | 205 kB 00:00 2023-10-14T16:37:45.833 INFO:teuthology.orchestra.run.smithi029.stdout:(59/143): python3-jsonpatch-1.21-16.el9.noarch. 604 kB/s | 24 kB 00:00 2023-10-14T16:37:45.876 INFO:teuthology.orchestra.run.smithi029.stdout:(60/143): python3-jinja2-2.11.3-4.el9.noarch.rp 2.7 MB/s | 228 kB 00:00 2023-10-14T16:37:45.901 INFO:teuthology.orchestra.run.smithi029.stdout:(61/143): python3-jsonpointer-2.0-4.el9.noarch. 280 kB/s | 18 kB 00:00 2023-10-14T16:37:45.935 INFO:teuthology.orchestra.run.smithi029.stdout:(62/143): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2023-10-14T16:37:45.962 INFO:teuthology.orchestra.run.smithi029.stdout:(63/143): python3-markupsafe-1.1.1-12.el9.x86_6 540 kB/s | 32 kB 00:00 2023-10-14T16:37:46.130 INFO:teuthology.orchestra.run.smithi029.stdout:(64/143): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 4.8 MB 00:00 2023-10-14T16:37:46.159 INFO:teuthology.orchestra.run.smithi186.stdout:(137/143): NetworkManager-1.45.4-1.el9.x86_64.r 3.3 MB/s | 2.2 MB 00:00 2023-10-14T16:37:46.163 INFO:teuthology.orchestra.run.smithi029.stdout:(65/143): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.2 MB/s | 217 kB 00:00 2023-10-14T16:37:46.206 INFO:teuthology.orchestra.run.smithi029.stdout:(66/143): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 41 kB 00:00 2023-10-14T16:37:46.239 INFO:teuthology.orchestra.run.smithi029.stdout:(67/143): python3-oauthlib-3.1.1-5.el9.noarch.r 2.3 MB/s | 174 kB 00:00 2023-10-14T16:37:46.268 INFO:teuthology.orchestra.run.smithi186.stdout:(138/143): NetworkManager-libnm-1.45.4-1.el9.x8 2.3 MB/s | 1.8 MB 00:00 2023-10-14T16:37:46.273 INFO:teuthology.orchestra.run.smithi029.stdout:(68/143): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.0 MB/s | 132 kB 00:00 2023-10-14T16:37:46.293 INFO:teuthology.orchestra.run.smithi186.stdout:(139/143): NetworkManager-team-1.45.4-1.el9.x86 64 kB/s | 41 kB 00:00 2023-10-14T16:37:46.307 INFO:teuthology.orchestra.run.smithi029.stdout:(69/143): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2023-10-14T16:37:46.318 INFO:teuthology.orchestra.run.smithi186.stdout:(140/143): NetworkManager-tui-1.45.4-1.el9.x86_ 1.6 MB/s | 244 kB 00:00 2023-10-14T16:37:46.341 INFO:teuthology.orchestra.run.smithi029.stdout:(70/143): python3-requests-oauthlib-1.3.0-12.el 662 kB/s | 43 kB 00:00 2023-10-14T16:37:46.343 INFO:teuthology.orchestra.run.smithi186.stdout:(141/143): acl-2.3.1-4.el9.x86_64.rpm 955 kB/s | 69 kB 00:00 2023-10-14T16:37:46.369 INFO:teuthology.orchestra.run.smithi186.stdout:(142/143): libacl-2.3.1-4.el9.x86_64.rpm 454 kB/s | 22 kB 00:00 2023-10-14T16:37:46.410 INFO:teuthology.orchestra.run.smithi186.stdout:(143/143): xfsdump-3.1.12-4.el9.x86_64.rpm 3.5 MB/s | 321 kB 00:00 2023-10-14T16:37:46.420 INFO:teuthology.orchestra.run.smithi186.stdout:-------------------------------------------------------------------------------- 2023-10-14T16:37:46.421 INFO:teuthology.orchestra.run.smithi186.stdout:Total 14 MB/s | 200 MB 00:13 2023-10-14T16:37:46.775 INFO:teuthology.orchestra.run.smithi029.stdout:(71/143): python3-scipy-1.6.2-8.el9.x86_64.rpm 34 MB/s | 16 MB 00:00 2023-10-14T16:37:46.800 INFO:teuthology.orchestra.run.smithi029.stdout:(72/143): python3-toml-0.10.2-6.el9.noarch.rpm 87 kB/s | 39 kB 00:00 2023-10-14T16:37:46.843 INFO:teuthology.orchestra.run.smithi029.stdout:(73/143): socat-1.7.4.1-5.el9.x86_64.rpm 4.4 MB/s | 300 kB 00:00 2023-10-14T16:37:46.850 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction check 2023-10-14T16:37:46.868 INFO:teuthology.orchestra.run.smithi029.stdout:(74/143): uuid-1.6.2-55.el9.x86_64.rpm 884 kB/s | 56 kB 00:00 2023-10-14T16:37:46.894 INFO:teuthology.orchestra.run.smithi029.stdout:(75/143): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2023-10-14T16:37:46.928 INFO:teuthology.orchestra.run.smithi029.stdout:(76/143): fuse-2.9.9-15.el9.x86_64.rpm 1.3 MB/s | 79 kB 00:00 2023-10-14T16:37:46.978 INFO:teuthology.orchestra.run.smithi029.stdout:(77/143): libgfortran-11.4.1-2.3.el9.x86_64.rpm 9.5 MB/s | 803 kB 00:00 2023-10-14T16:37:47.014 INFO:teuthology.orchestra.run.smithi029.stdout:(78/143): libquadmath-11.4.1-2.3.el9.x86_64.rpm 2.5 MB/s | 195 kB 00:00 2023-10-14T16:37:47.051 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction check succeeded. 2023-10-14T16:37:47.052 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction test 2023-10-14T16:37:47.056 INFO:teuthology.orchestra.run.smithi029.stdout:(79/143): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2023-10-14T16:37:47.090 INFO:teuthology.orchestra.run.smithi029.stdout:(80/143): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2023-10-14T16:37:47.124 INFO:teuthology.orchestra.run.smithi029.stdout:(81/143): python3-pycparser-2.20-6.el9.noarch.r 1.8 MB/s | 124 kB 00:00 2023-10-14T16:37:47.158 INFO:teuthology.orchestra.run.smithi029.stdout:(82/143): python3-requests-2.25.1-7.el9.noarch. 1.7 MB/s | 113 kB 00:00 2023-10-14T16:37:47.192 INFO:teuthology.orchestra.run.smithi029.stdout:(83/143): python3-urllib3-1.26.5-3.el9.noarch.r 2.8 MB/s | 188 kB 00:00 2023-10-14T16:37:47.951 INFO:teuthology.orchestra.run.smithi029.stdout:(84/143): ceph-test-18.0.0-6675.gc7471068.el9.x 5.2 MB/s | 41 MB 00:07 2023-10-14T16:37:48.210 INFO:teuthology.orchestra.run.smithi029.stdout:(85/143): python3-cryptography-36.0.1-4.el9.x86 1.2 MB/s | 1.2 MB 00:01 2023-10-14T16:37:48.421 INFO:teuthology.orchestra.run.smithi029.stdout:(86/143): elfutils-libelf-devel-0.189-3.el9.x86 48 kB/s | 22 kB 00:00 2023-10-14T16:37:48.736 INFO:teuthology.orchestra.run.smithi029.stdout:(87/143): librabbitmq-0.11.0-7.el9.x86_64.rpm 143 kB/s | 45 kB 00:00 2023-10-14T16:37:48.770 INFO:teuthology.orchestra.run.smithi029.stdout:(88/143): jq-1.6-15.el9.x86_64.rpm 336 kB/s | 187 kB 00:00 2023-10-14T16:37:48.865 INFO:teuthology.orchestra.run.smithi029.stdout:(89/143): automake-1.16.2-8.el9.noarch.rpm 413 kB/s | 689 kB 00:01 2023-10-14T16:37:48.887 INFO:teuthology.orchestra.run.smithi029.stdout:(90/143): libuuid-devel-2.37.4-15.el9.x86_64.rp 170 kB/s | 26 kB 00:00 2023-10-14T16:37:49.130 INFO:teuthology.orchestra.run.smithi029.stdout:(91/143): python3-pytz-2021.1-5.el9.noarch.rpm 192 kB/s | 51 kB 00:00 2023-10-14T16:37:49.273 INFO:teuthology.orchestra.run.smithi186.stdout:Transaction test succeeded. 2023-10-14T16:37:49.274 INFO:teuthology.orchestra.run.smithi186.stdout:Running transaction 2023-10-14T16:37:49.298 INFO:teuthology.orchestra.run.smithi029.stdout:(92/143): openssl-devel-3.0.7-24.el9.x86_64.rpm 7.9 MB/s | 4.1 MB 00:00 2023-10-14T16:37:49.324 INFO:teuthology.orchestra.run.smithi029.stdout:(93/143): xfsprogs-devel-5.19.0-4.el9.x86_64.rp 200 kB/s | 86 kB 00:00 2023-10-14T16:37:49.358 INFO:teuthology.orchestra.run.smithi029.stdout:(94/143): zlib-devel-1.2.11-40.el9.x86_64.rpm 199 kB/s | 45 kB 00:00 2023-10-14T16:37:49.783 INFO:teuthology.orchestra.run.smithi029.stdout:(95/143): fmt-8.1.1-5.el9.x86_64.rpm 242 kB/s | 111 kB 00:00 2023-10-14T16:37:49.867 INFO:teuthology.orchestra.run.smithi029.stdout:(96/143): gperftools-libs-2.9.1-2.el9.x86_64.rp 571 kB/s | 309 kB 00:00 2023-10-14T16:37:49.901 INFO:teuthology.orchestra.run.smithi029.stdout:(97/143): libarrow-doc-9.0.0-9.el9.noarch.rpm 209 kB/s | 24 kB 00:00 2023-10-14T16:37:49.977 INFO:teuthology.orchestra.run.smithi029.stdout:(98/143): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2023-10-14T16:37:50.019 INFO:teuthology.orchestra.run.smithi029.stdout:(99/143): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2023-10-14T16:37:50.194 INFO:teuthology.orchestra.run.smithi029.stdout:(100/143): parquet-libs-9.0.0-9.el9.x86_64.rpm 3.8 MB/s | 837 kB 00:00 2023-10-14T16:37:50.295 INFO:teuthology.orchestra.run.smithi029.stdout:(101/143): python3-bcrypt-3.2.2-1.el9.x86_64.rp 438 kB/s | 43 kB 00:00 2023-10-14T16:37:50.337 INFO:teuthology.orchestra.run.smithi029.stdout:(102/143): python3-asyncssh-2.13.2-3.el9.noarch 1.7 MB/s | 548 kB 00:00 2023-10-14T16:37:50.487 INFO:teuthology.orchestra.run.smithi029.stdout:(103/143): libarrow-9.0.0-9.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2023-10-14T16:37:50.563 INFO:teuthology.orchestra.run.smithi029.stdout:(104/143): python3-certifi-2021.10.8-2.el9.noar 66 kB/s | 15 kB 00:00 2023-10-14T16:37:50.589 INFO:teuthology.orchestra.run.smithi029.stdout:(105/143): python3-cachetools-4.2.4-1.el9.noarc 110 kB/s | 32 kB 00:00 2023-10-14T16:37:50.622 INFO:teuthology.orchestra.run.smithi029.stdout:(106/143): python3-cheroot-8.6.0-4.el9.noarch.r 1.3 MB/s | 172 kB 00:00 2023-10-14T16:37:50.683 INFO:teuthology.orchestra.run.smithi029.stdout:(107/143): python3-cherrypy-18.6.1-2.el9.noarch 2.9 MB/s | 358 kB 00:00 2023-10-14T16:37:50.717 INFO:teuthology.orchestra.run.smithi029.stdout:(108/143): python3-google-auth-2.23.2-1.el9.noa 1.5 MB/s | 196 kB 00:00 2023-10-14T16:37:50.742 INFO:teuthology.orchestra.run.smithi029.stdout:(109/143): python3-jaraco-8.2.1-3.el9.noarch.rp 90 kB/s | 11 kB 00:00 2023-10-14T16:37:50.776 INFO:teuthology.orchestra.run.smithi029.stdout:(110/143): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2023-10-14T16:37:50.802 INFO:teuthology.orchestra.run.smithi029.stdout:(111/143): python3-jaraco-collections-3.0.0-8.e 276 kB/s | 23 kB 00:00 2023-10-14T16:37:50.836 INFO:teuthology.orchestra.run.smithi029.stdout:(112/143): python3-jaraco-functools-3.5.0-2.el9 210 kB/s | 19 kB 00:00 2023-10-14T16:37:50.869 INFO:teuthology.orchestra.run.smithi029.stdout:(113/143): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2023-10-14T16:37:50.912 INFO:teuthology.orchestra.run.smithi029.stdout:(114/143): python3-jwt+crypto-2.4.0-1.el9.noarc 83 kB/s | 9.0 kB 00:00 2023-10-14T16:37:50.945 INFO:teuthology.orchestra.run.smithi029.stdout:(115/143): python3-jwt-2.4.0-1.el9.noarch.rpm 376 kB/s | 41 kB 00:00 2023-10-14T16:37:51.046 INFO:teuthology.orchestra.run.smithi029.stdout:(116/143): python3-kubernetes-26.1.0-2.el9.noar 5.8 MB/s | 1.0 MB 00:00 2023-10-14T16:37:51.080 INFO:teuthology.orchestra.run.smithi029.stdout:(117/143): python3-logutils-0.3.5-21.el9.noarch 277 kB/s | 46 kB 00:00 2023-10-14T16:37:51.105 INFO:teuthology.orchestra.run.smithi029.stdout:(118/143): python3-more-itertools-8.12.0-2.el9. 496 kB/s | 79 kB 00:00 2023-10-14T16:37:51.139 INFO:teuthology.orchestra.run.smithi029.stdout:(119/143): python3-natsort-7.1.1-5.el9.noarch.r 626 kB/s | 58 kB 00:00 2023-10-14T16:37:51.206 INFO:teuthology.orchestra.run.smithi029.stdout:(120/143): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2023-10-14T16:37:51.240 INFO:teuthology.orchestra.run.smithi029.stdout:(121/143): python3-portend-3.1.0-2.el9.noarch.r 122 kB/s | 16 kB 00:00 2023-10-14T16:37:51.266 INFO:teuthology.orchestra.run.smithi029.stdout:(122/143): python3-pyOpenSSL-21.0.0-1.el9.noarc 715 kB/s | 90 kB 00:00 2023-10-14T16:37:51.301 INFO:teuthology.orchestra.run.smithi029.stdout:(123/143): python3-repoze-lru-0.7-16.el9.noarch 328 kB/s | 31 kB 00:00 2023-10-14T16:37:51.360 INFO:teuthology.orchestra.run.smithi029.stdout:(124/143): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2023-10-14T16:37:51.386 INFO:teuthology.orchestra.run.smithi029.stdout:(125/143): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2023-10-14T16:37:51.411 INFO:teuthology.orchestra.run.smithi029.stdout:(126/143): python3-rsa-4.9-2.el9.noarch.rpm 409 kB/s | 59 kB 00:00 2023-10-14T16:37:51.453 INFO:teuthology.orchestra.run.smithi029.stdout:(127/143): python3-typing-extensions-4.4.0-2.el 554 kB/s | 51 kB 00:00 2023-10-14T16:37:51.521 INFO:teuthology.orchestra.run.smithi029.stdout:(128/143): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2023-10-14T16:37:51.555 INFO:teuthology.orchestra.run.smithi029.stdout:(129/143): python3-websocket-client-1.2.3-2.el9 629 kB/s | 90 kB 00:00 2023-10-14T16:37:51.597 INFO:teuthology.orchestra.run.smithi029.stdout:(130/143): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2023-10-14T16:37:51.622 INFO:teuthology.orchestra.run.smithi029.stdout:(131/143): python3-zc-lockfile-2.0-10.el9.noarc 199 kB/s | 20 kB 00:00 2023-10-14T16:37:51.698 INFO:teuthology.orchestra.run.smithi029.stdout:(132/143): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2023-10-14T16:37:51.857 INFO:teuthology.orchestra.run.smithi029.stdout:(133/143): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2023-10-14T16:37:51.899 INFO:teuthology.orchestra.run.smithi029.stdout:(134/143): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.6 MB/s | 68 kB 00:00 2023-10-14T16:37:52.008 INFO:teuthology.orchestra.run.smithi029.stdout:(135/143): NetworkManager-1.45.4-1.el9.x86_64.r 21 MB/s | 2.2 MB 00:00 2023-10-14T16:37:52.109 INFO:teuthology.orchestra.run.smithi029.stdout:(136/143): NetworkManager-libnm-1.45.4-1.el9.x8 18 MB/s | 1.8 MB 00:00 2023-10-14T16:37:52.151 INFO:teuthology.orchestra.run.smithi029.stdout:(137/143): NetworkManager-team-1.45.4-1.el9.x86 1.0 MB/s | 41 kB 00:00 2023-10-14T16:37:52.195 INFO:teuthology.orchestra.run.smithi029.stdout:(138/143): NetworkManager-tui-1.45.4-1.el9.x86_ 5.6 MB/s | 244 kB 00:00 2023-10-14T16:37:52.303 INFO:teuthology.orchestra.run.smithi029.stdout:(139/143): librados2-18.0.0-6675.gc7471068.el9. 4.9 MB/s | 3.3 MB 00:00 2023-10-14T16:37:52.329 INFO:teuthology.orchestra.run.smithi029.stdout:(140/143): acl-2.3.1-4.el9.x86_64.rpm 517 kB/s | 69 kB 00:00 2023-10-14T16:37:52.355 INFO:teuthology.orchestra.run.smithi029.stdout:(141/143): libacl-2.3.1-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2023-10-14T16:37:52.389 INFO:teuthology.orchestra.run.smithi029.stdout:(142/143): xfsdump-3.1.12-4.el9.x86_64.rpm 5.4 MB/s | 321 kB 00:00 2023-10-14T16:37:52.488 INFO:teuthology.orchestra.run.smithi029.stdout:(143/143): librbd1-18.0.0-6675.gc7471068.el9.x8 3.8 MB/s | 3.0 MB 00:00 2023-10-14T16:37:52.498 INFO:teuthology.orchestra.run.smithi029.stdout:-------------------------------------------------------------------------------- 2023-10-14T16:37:52.498 INFO:teuthology.orchestra.run.smithi029.stdout:Total 12 MB/s | 200 MB 00:17 2023-10-14T16:37:52.949 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction check 2023-10-14T16:37:53.166 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction check succeeded. 2023-10-14T16:37:53.166 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction test 2023-10-14T16:37:55.529 INFO:teuthology.orchestra.run.smithi186.stdout: Preparing : 1/1 2023-10-14T16:37:55.609 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-14T16:37:55.639 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-14T16:37:55.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-14T16:37:55.713 INFO:teuthology.orchestra.run.smithi029.stdout:Transaction test succeeded. 2023-10-14T16:37:55.714 INFO:teuthology.orchestra.run.smithi029.stdout:Running transaction 2023-10-14T16:37:55.793 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-14T16:37:56.194 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-14T16:37:56.248 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:37:56.393 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:37:56.458 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:37:56.575 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:37:56.641 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-14T16:37:56.688 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-14T16:37:56.950 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-14T16:37:57.014 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-14T16:37:57.039 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:37:57.194 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:37:57.247 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-14T16:37:57.341 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-14T16:37:57.719 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-14T16:37:57.904 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-14T16:37:57.984 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-14T16:37:58.200 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:37:58.249 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:37:58.696 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:37:58.746 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-14T16:37:58.799 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-14T16:37:58.939 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-14T16:37:59.005 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-14T16:37:59.049 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-14T16:37:59.108 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-14T16:37:59.211 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-14T16:37:59.260 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-14T16:37:59.294 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-14T16:37:59.400 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-14T16:37:59.440 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 27/153 2023-10-14T16:37:59.489 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 28/153 2023-10-14T16:37:59.594 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 29/153 2023-10-14T16:37:59.779 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-14T16:37:59.844 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-14T16:38:00.551 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-14T16:38:00.649 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-14T16:38:00.694 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-14T16:38:00.742 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 35/153 2023-10-14T16:38:00.806 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-14T16:38:00.962 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-14T16:38:01.290 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-14T16:38:01.388 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-14T16:38:01.501 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-14T16:38:01.552 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-14T16:38:01.593 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-14T16:38:01.639 INFO:teuthology.orchestra.run.smithi029.stdout: Preparing : 1/1 2023-10-14T16:38:01.665 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-14T16:38:01.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-14T16:38:01.737 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : thrift-0.15.0-2.el9.x86_64 1/153 2023-10-14T16:38:01.771 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 2/153 2023-10-14T16:38:01.793 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-14T16:38:01.800 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 3/153 2023-10-14T16:38:01.986 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-14T16:38:02.042 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-14T16:38:02.107 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-14T16:38:02.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/153 2023-10-14T16:38:02.225 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-14T16:38:02.275 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-14T16:38:02.322 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-14T16:38:02.356 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-14T16:38:02.395 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-14T16:38:02.438 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-14T16:38:02.508 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-14T16:38:02.649 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 5/153 2023-10-14T16:38:02.663 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-14T16:38:02.722 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-14T16:38:02.766 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:38:02.784 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-14T16:38:02.862 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-14T16:38:02.969 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:38:03.010 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:38:03.130 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-14T16:38:03.154 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:38:03.193 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-14T16:38:03.234 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-14T16:38:03.283 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 9/153 2023-10-14T16:38:03.370 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-14T16:38:03.429 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-14T16:38:03.453 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-14T16:38:03.492 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-14T16:38:03.535 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-14T16:38:03.558 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 10/153 2023-10-14T16:38:03.578 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-14T16:38:03.612 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-13.11-1.el9.x86_64 11/153 2023-10-14T16:38:03.631 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:38:03.634 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-14T16:38:03.740 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-14T16:38:03.789 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:38:03.806 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-14T16:38:03.824 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-14T16:38:03.848 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-14T16:38:03.918 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 13/153 2023-10-14T16:38:03.974 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-14T16:38:04.052 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-14T16:38:04.087 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-14T16:38:04.152 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-14T16:38:04.198 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-14T16:38:04.345 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-14T16:38:04.395 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-14T16:38:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 14/153 2023-10-14T16:38:04.561 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-14T16:38:04.580 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-14T16:38:04.653 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-14T16:38:04.675 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 15/153 2023-10-14T16:38:04.991 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:38:05.043 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:38:05.601 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.45.4-1.el9.x86_64 16/153 2023-10-14T16:38:05.642 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 17/153 2023-10-14T16:38:05.663 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-14T16:38:05.703 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : liboath-2.6.7-2.el9.x86_64 18/153 2023-10-14T16:38:05.713 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-14T16:38:05.746 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-14T16:38:05.836 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 19/153 2023-10-14T16:38:05.873 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-14T16:38:05.912 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-urllib3-1.26.5-3.el9.noarch 20/153 2023-10-14T16:38:05.921 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-14T16:38:05.952 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-2.25.1-7.el9.noarch 21/153 2023-10-14T16:38:06.037 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libquadmath-11.4.1-2.3.el9.x86_64 22/153 2023-10-14T16:38:06.146 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-14T16:38:06.157 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libgfortran-11.4.1-2.3.el9.x86_64 23/153 2023-10-14T16:38:06.177 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-14T16:38:06.212 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/153 2023-10-14T16:38:06.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/153 2023-10-14T16:38:06.271 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-14T16:38:06.377 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/153 2023-10-14T16:38:06.420 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 27/153 2023-10-14T16:38:06.520 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 28/153 2023-10-14T16:38:06.654 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 29/153 2023-10-14T16:38:06.842 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 30/153 2023-10-14T16:38:06.910 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 31/153 2023-10-14T16:38:07.088 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-14T16:38:07.648 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rsa-4.9-2.el9.noarch 32/153 2023-10-14T16:38:07.726 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-14T16:38:07.798 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 33/153 2023-10-14T16:38:07.840 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jinja2-2.11.3-4.el9.noarch 34/153 2023-10-14T16:38:07.880 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 35/153 2023-10-14T16:38:07.919 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 36/153 2023-10-14T16:38:08.088 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : re2-1:20211101-3.el9.x86_64 37/153 2023-10-14T16:38:08.131 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-9.0.0-9.el9.x86_64 38/153 2023-10-14T16:38:08.214 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 39/153 2023-10-14T16:38:08.324 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 40/153 2023-10-14T16:38:08.370 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 41/153 2023-10-14T16:38:08.412 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 42/153 2023-10-14T16:38:08.484 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 43/153 2023-10-14T16:38:08.553 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 44/153 2023-10-14T16:38:08.616 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 45/153 2023-10-14T16:38:08.816 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 46/153 2023-10-14T16:38:08.873 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 47/153 2023-10-14T16:38:08.957 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 48/153 2023-10-14T16:38:09.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 49/153 2023-10-14T16:38:09.163 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-google-auth-1:2.23.2-1.el9.noarch 50/153 2023-10-14T16:38:09.227 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 51/153 2023-10-14T16:38:09.261 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 52/153 2023-10-14T16:38:09.303 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libarrow-doc-9.0.0-9.el9.noarch 53/153 2023-10-14T16:38:09.353 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fmt-8.1.1-5.el9.x86_64 54/153 2023-10-14T16:38:09.406 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 55/153 2023-10-14T16:38:09.524 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-14T16:38:09.577 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libuuid-devel-2.37.4-15.el9.x86_64 56/153 2023-10-14T16:38:09.624 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-14T16:38:09.634 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : automake-1.16.2-8.el9.noarch 57/153 2023-10-14T16:38:09.662 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-14T16:38:09.692 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-14T16:38:09.720 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-ply-3.11-14.el9.noarch 58/153 2023-10-14T16:38:09.732 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-14T16:38:09.823 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 59/153 2023-10-14T16:38:10.106 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 60/153 2023-10-14T16:38:10.118 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-14T16:38:10.172 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-14T16:38:10.189 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 61/153 2023-10-14T16:38:10.294 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-14T16:38:10.328 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 98/153 2023-10-14T16:38:10.377 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 62/153 2023-10-14T16:38:10.384 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-14T16:38:10.444 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-asyncssh-2.13.2-3.el9.noarch 63/153 2023-10-14T16:38:10.477 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 64/153 2023-10-14T16:38:10.519 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 65/153 2023-10-14T16:38:10.571 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 66/153 2023-10-14T16:38:10.641 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : fuse-2.9.9-15.el9.x86_64 67/153 2023-10-14T16:38:10.711 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/153 2023-10-14T16:38:10.798 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/153 2023-10-14T16:38:10.853 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 70/153 2023-10-14T16:38:10.869 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-14T16:38:10.896 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/153 2023-10-14T16:38:10.900 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-14T16:38:11.000 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-14T16:38:11.027 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 102/153 2023-10-14T16:38:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/153 2023-10-14T16:38:11.076 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 73/153 2023-10-14T16:38:11.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 74/153 2023-10-14T16:38:11.188 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 75/153 2023-10-14T16:38:11.241 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 76/153 2023-10-14T16:38:11.399 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 77/153 2023-10-14T16:38:11.450 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 78/153 2023-10-14T16:38:11.614 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/153 2023-10-14T16:38:11.700 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 80/153 2023-10-14T16:38:12.748 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 81/153 2023-10-14T16:38:12.811 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 82/153 2023-10-14T16:38:12.854 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 83/153 2023-10-14T16:38:12.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 84/153 2023-10-14T16:38:13.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 85/153 2023-10-14T16:38:13.292 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-0.3.21-2.el9.x86_64 86/153 2023-10-14T16:38:13.330 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 87/153 2023-10-14T16:38:13.440 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 88/153 2023-10-14T16:38:13.526 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:13.565 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:14.192 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:14.244 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-14T16:38:14.305 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 89/153 2023-10-14T16:38:14.442 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-14T16:38:14.442 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-14T16:38:14.443 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-14T16:38:14.443 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:14.465 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 90/153 2023-10-14T16:38:14.511 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-14T16:38:16.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 91/153 2023-10-14T16:38:16.283 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 92/153 2023-10-14T16:38:16.316 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 93/153 2023-10-14T16:38:16.355 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : jq-1.6-15.el9.x86_64 94/153 2023-10-14T16:38:16.398 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 95/153 2023-10-14T16:38:16.794 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 96/153 2023-10-14T16:38:16.827 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-14T16:38:16.950 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 97/153 2023-10-14T16:38:16.984 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 98/153 2023-10-14T16:38:17.050 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 99/153 2023-10-14T16:38:17.582 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : parquet-libs-9.0.0-9.el9.x86_64 100/153 2023-10-14T16:38:17.626 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-14T16:38:17.736 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 101/153 2023-10-14T16:38:17.762 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 102/153 2023-10-14T16:38:21.002 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:21.040 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:21.714 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 103/153 2023-10-14T16:38:21.764 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-14T16:38:21.941 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 104/153 2023-10-14T16:38:21.941 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2023-10-14T16:38:21.941 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2023-10-14T16:38:21.942 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:21.984 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-14T16:38:27.105 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-14T16:38:27.105 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /sys 2023-10-14T16:38:27.105 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /proc 2023-10-14T16:38:27.105 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /mnt 2023-10-14T16:38:27.106 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /var/tmp 2023-10-14T16:38:27.106 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /home 2023-10-14T16:38:27.106 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /root 2023-10-14T16:38:27.106 INFO:teuthology.orchestra.run.smithi186.stdout:skipping the directory /tmp 2023-10-14T16:38:27.106 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:27.308 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-14T16:38:27.351 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-14T16:38:27.351 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:27.351 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-14T16:38:27.351 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:29.054 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-14T16:38:29.156 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-14T16:38:29.156 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:29.156 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-14T16:38:29.156 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-14T16:38:29.157 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-14T16:38:29.157 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:29.386 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-14T16:38:29.468 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-14T16:38:29.468 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:29.469 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-14T16:38:29.469 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-14T16:38:29.469 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-14T16:38:29.469 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:29.995 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-14T16:38:30.091 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-14T16:38:30.091 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:30.092 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-14T16:38:30.093 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-14T16:38:30.093 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-14T16:38:30.094 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:30.763 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-14T16:38:30.799 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-14T16:38:30.846 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 111/153 2023-10-14T16:38:30.956 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 112/153 2023-10-14T16:38:30.985 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-14T16:38:31.309 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-14T16:38:31.344 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-14T16:38:32.475 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-14T16:38:32.511 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-14T16:38:32.661 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-14T16:38:32.828 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 116/153 2023-10-14T16:38:32.863 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-14T16:38:32.932 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-14T16:38:32.932 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:32.933 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-14T16:38:32.933 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-14T16:38:32.933 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-14T16:38:32.933 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:33.006 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-14T16:38:33.045 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-14T16:38:33.931 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 119/153 2023-10-14T16:38:33.965 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-14T16:38:34.037 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-14T16:38:34.037 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:34.037 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-14T16:38:34.037 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-14T16:38:34.038 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-14T16:38:34.038 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:34.091 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-14T16:38:34.187 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-14T16:38:34.188 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:34.188 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-14T16:38:34.188 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:34.267 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-14T16:38:34.368 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-14T16:38:34.368 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:34.368 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-14T16:38:34.369 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-14T16:38:34.369 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-14T16:38:34.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:34.734 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-14T16:38:34.806 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-14T16:38:34.806 INFO:teuthology.orchestra.run.smithi186.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:34.807 INFO:teuthology.orchestra.run.smithi186.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-14T16:38:34.807 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-14T16:38:34.807 INFO:teuthology.orchestra.run.smithi186.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-14T16:38:34.807 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:37.960 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 105/153 2023-10-14T16:38:37.960 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /sys 2023-10-14T16:38:37.961 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /proc 2023-10-14T16:38:37.961 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /mnt 2023-10-14T16:38:37.961 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /var/tmp 2023-10-14T16:38:37.961 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /home 2023-10-14T16:38:37.961 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /root 2023-10-14T16:38:37.962 INFO:teuthology.orchestra.run.smithi029.stdout:skipping the directory /tmp 2023-10-14T16:38:37.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:38.264 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-14T16:38:38.318 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 106/153 2023-10-14T16:38:38.318 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:38.318 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2023-10-14T16:38:38.318 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:38.982 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 124/153 2023-10-14T16:38:39.046 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 125/153 2023-10-14T16:38:39.079 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 126/153 2023-10-14T16:38:39.355 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-14T16:38:39.479 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-14T16:38:39.548 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 129/153 2023-10-14T16:38:39.596 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-14T16:38:39.627 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-14T16:38:39.682 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 132/153 2023-10-14T16:38:39.720 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-14T16:38:39.815 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-14T16:38:39.887 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-14T16:38:39.920 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-14T16:38:40.102 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-14T16:38:40.192 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 107/153 2023-10-14T16:38:40.192 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:40.193 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2023-10-14T16:38:40.193 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-14T16:38:40.193 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2023-10-14T16:38:40.194 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:40.428 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-14T16:38:40.531 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 108/153 2023-10-14T16:38:40.531 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:40.532 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2023-10-14T16:38:40.532 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-14T16:38:40.532 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2023-10-14T16:38:40.532 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:40.920 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:40.969 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:41.025 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:41.054 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-14T16:38:41.096 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 138/153 2023-10-14T16:38:41.143 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 109/153 2023-10-14T16:38:41.144 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:41.144 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2023-10-14T16:38:41.145 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-14T16:38:41.145 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2023-10-14T16:38:41.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:41.816 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-14T16:38:41.876 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 110/153 2023-10-14T16:38:41.983 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 111/153 2023-10-14T16:38:42.110 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 112/153 2023-10-14T16:38:42.152 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-14T16:38:42.441 INFO:teuthology.orchestra.run.smithi186.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-14T16:38:42.498 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 113/153 2023-10-14T16:38:42.533 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-14T16:38:42.537 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-14T16:38:42.576 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-14T16:38:42.694 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-14T16:38:42.729 INFO:teuthology.orchestra.run.smithi186.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-14T16:38:42.757 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-14T16:38:42.757 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-14T16:38:42.902 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-14T16:38:42.934 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-14T16:38:42.966 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-14T16:38:42.999 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-14T16:38:43.039 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:43.039 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:43.359 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:43.359 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-14T16:38:43.505 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-14T16:38:43.535 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-14T16:38:43.536 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-14T16:38:43.708 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-14T16:38:43.725 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 114/153 2023-10-14T16:38:43.754 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-14T16:38:43.841 INFO:teuthology.orchestra.run.smithi186.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:38:43.902 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 115/153 2023-10-14T16:38:44.080 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 116/153 2023-10-14T16:38:44.101 INFO:teuthology.orchestra.run.smithi186.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:38:44.101 INFO:teuthology.orchestra.run.smithi186.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-14T16:38:44.101 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:44.108 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-14T16:38:44.186 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 117/153 2023-10-14T16:38:44.187 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:44.187 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2023-10-14T16:38:44.187 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-14T16:38:44.187 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2023-10-14T16:38:44.188 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:44.244 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-14T16:38:44.285 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 118/153 2023-10-14T16:38:45.217 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 119/153 2023-10-14T16:38:45.255 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-14T16:38:45.332 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 120/153 2023-10-14T16:38:45.332 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:45.332 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2023-10-14T16:38:45.332 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-14T16:38:45.333 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2023-10-14T16:38:45.333 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:45.390 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-14T16:38:45.470 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.0.0-6675.gc7471 121/153 2023-10-14T16:38:45.470 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:45.470 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2023-10-14T16:38:45.471 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:45.533 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-14T16:38:45.615 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 122/153 2023-10-14T16:38:45.616 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:45.616 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2023-10-14T16:38:45.616 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-14T16:38:45.616 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2023-10-14T16:38:45.617 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:45.992 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-14T16:38:46.070 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 123/153 2023-10-14T16:38:46.071 INFO:teuthology.orchestra.run.smithi029.stdout:Glob pattern passed to enable, but globs are not supported for this. 2023-10-14T16:38:46.071 INFO:teuthology.orchestra.run.smithi029.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2023-10-14T16:38:46.071 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-14T16:38:46.071 INFO:teuthology.orchestra.run.smithi029.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2023-10-14T16:38:46.072 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:38:48.722 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 1/153 2023-10-14T16:38:48.723 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2/153 2023-10-14T16:38:48.723 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 3/153 2023-10-14T16:38:48.723 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 4/153 2023-10-14T16:38:48.723 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/153 2023-10-14T16:38:48.723 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 9/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 10/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 11/153 2023-10-14T16:38:48.724 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:38:48.725 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 13/153 2023-10-14T16:38:48.725 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 14/153 2023-10-14T16:38:48.725 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 15/153 2023-10-14T16:38:48.725 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 16/153 2023-10-14T16:38:48.725 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 17/153 2023-10-14T16:38:48.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 18/153 2023-10-14T16:38:48.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 19/153 2023-10-14T16:38:48.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/153 2023-10-14T16:38:48.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 21/153 2023-10-14T16:38:48.726 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 22/153 2023-10-14T16:38:48.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 23/153 2023-10-14T16:38:48.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 24/153 2023-10-14T16:38:48.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 25/153 2023-10-14T16:38:48.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 26/153 2023-10-14T16:38:48.727 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 27/153 2023-10-14T16:38:48.728 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 28/153 2023-10-14T16:38:48.738 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/153 2023-10-14T16:38:48.738 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 30/153 2023-10-14T16:38:48.739 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 31/153 2023-10-14T16:38:48.739 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/153 2023-10-14T16:38:48.739 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/153 2023-10-14T16:38:48.739 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 34/153 2023-10-14T16:38:48.740 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/153 2023-10-14T16:38:48.740 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 36/153 2023-10-14T16:38:48.740 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 37/153 2023-10-14T16:38:48.740 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 38/153 2023-10-14T16:38:48.740 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-14T16:38:48.741 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-14T16:38:48.741 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-14T16:38:48.741 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-14T16:38:48.741 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-14T16:38:48.741 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-14T16:38:48.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-14T16:38:48.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-14T16:38:48.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-14T16:38:48.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-14T16:38:48.742 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-14T16:38:48.743 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-14T16:38:48.743 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-14T16:38:48.743 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-14T16:38:48.743 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-14T16:38:48.743 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-14T16:38:48.744 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-14T16:38:48.745 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-14T16:38:48.745 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-14T16:38:48.745 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-14T16:38:48.745 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-14T16:38:48.745 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-14T16:38:48.746 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-14T16:38:48.746 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-14T16:38:48.746 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-14T16:38:48.746 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-14T16:38:48.746 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-14T16:38:48.747 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-14T16:38:48.747 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-14T16:38:48.747 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-14T16:38:48.747 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-14T16:38:48.748 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-14T16:38:48.749 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-14T16:38:48.749 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-14T16:38:48.749 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-14T16:38:48.749 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-14T16:38:48.749 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-14T16:38:48.750 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-14T16:38:48.750 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-14T16:38:48.750 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-14T16:38:48.750 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-14T16:38:48.750 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-14T16:38:48.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-14T16:38:48.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-14T16:38:48.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-14T16:38:48.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-14T16:38:48.751 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-14T16:38:48.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-14T16:38:48.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-14T16:38:48.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-14T16:38:48.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-14T16:38:48.752 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-14T16:38:48.753 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-14T16:38:48.753 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-14T16:38:48.753 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-14T16:38:48.753 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-14T16:38:48.753 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-14T16:38:48.754 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-14T16:38:48.754 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-14T16:38:48.754 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-14T16:38:48.754 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-14T16:38:48.754 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-14T16:38:48.755 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-14T16:38:48.755 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-14T16:38:48.755 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-14T16:38:48.755 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-14T16:38:48.755 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-14T16:38:48.756 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-14T16:38:48.757 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-14T16:38:48.757 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-14T16:38:48.757 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-14T16:38:48.757 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-14T16:38:48.757 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-14T16:38:48.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-14T16:38:48.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-14T16:38:48.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-14T16:38:48.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-14T16:38:48.758 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-14T16:38:48.761 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-14T16:38:48.761 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-14T16:38:48.761 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 134/153 2023-10-14T16:38:48.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-14T16:38:48.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 136/153 2023-10-14T16:38:48.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-14T16:38:48.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-14T16:38:48.762 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-14T16:38:48.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-14T16:38:48.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-14T16:38:48.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-14T16:38:48.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-14T16:38:48.763 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-14T16:38:48.764 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-14T16:38:48.765 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-14T16:38:48.765 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-14T16:38:50.471 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 124/153 2023-10-14T16:38:50.561 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 125/153 2023-10-14T16:38:50.603 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 126/153 2023-10-14T16:38:50.901 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : elfutils-libelf-devel-0.189-3.el9.x86_64 127/153 2023-10-14T16:38:50.963 INFO:teuthology.orchestra.run.smithi186.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:38:50.963 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:50.963 INFO:teuthology.orchestra.run.smithi186.stdout:Upgraded: 2023-10-14T16:38:50.963 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: acl-2.3.1-4.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-14T16:38:50.964 INFO:teuthology.orchestra.run.smithi186.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-14T16:38:50.965 INFO:teuthology.orchestra.run.smithi186.stdout: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.965 INFO:teuthology.orchestra.run.smithi186.stdout: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.965 INFO:teuthology.orchestra.run.smithi186.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-14T16:38:50.965 INFO:teuthology.orchestra.run.smithi186.stdout:Installed: 2023-10-14T16:38:50.965 INFO:teuthology.orchestra.run.smithi186.stdout: automake-1.16.2-8.el9.noarch 2023-10-14T16:38:50.966 INFO:teuthology.orchestra.run.smithi186.stdout: bison-3.7.4-5.el9.x86_64 2023-10-14T16:38:50.966 INFO:teuthology.orchestra.run.smithi186.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-14T16:38:50.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.966 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.967 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.968 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.969 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.969 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.969 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.969 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.969 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.970 INFO:teuthology.orchestra.run.smithi186.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.970 INFO:teuthology.orchestra.run.smithi186.stdout: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.970 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.970 INFO:teuthology.orchestra.run.smithi186.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: flex-2.6.4-9.el9.x86_64 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-14T16:38:50.971 INFO:teuthology.orchestra.run.smithi186.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-14T16:38:50.972 INFO:teuthology.orchestra.run.smithi186.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-14T16:38:50.972 INFO:teuthology.orchestra.run.smithi186.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-14T16:38:50.972 INFO:teuthology.orchestra.run.smithi186.stdout: jq-1.6-15.el9.x86_64 2023-10-14T16:38:50.972 INFO:teuthology.orchestra.run.smithi186.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-14T16:38:50.973 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-14T16:38:50.973 INFO:teuthology.orchestra.run.smithi186.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-14T16:38:50.973 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.973 INFO:teuthology.orchestra.run.smithi186.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.973 INFO:teuthology.orchestra.run.smithi186.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.974 INFO:teuthology.orchestra.run.smithi186.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-14T16:38:50.974 INFO:teuthology.orchestra.run.smithi186.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-14T16:38:50.974 INFO:teuthology.orchestra.run.smithi186.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-14T16:38:50.974 INFO:teuthology.orchestra.run.smithi186.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-14T16:38:50.974 INFO:teuthology.orchestra.run.smithi186.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: librados-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-14T16:38:50.975 INFO:teuthology.orchestra.run.smithi186.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-14T16:38:50.976 INFO:teuthology.orchestra.run.smithi186.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-14T16:38:50.976 INFO:teuthology.orchestra.run.smithi186.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-14T16:38:50.976 INFO:teuthology.orchestra.run.smithi186.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-14T16:38:50.976 INFO:teuthology.orchestra.run.smithi186.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-14T16:38:50.976 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-14T16:38:50.977 INFO:teuthology.orchestra.run.smithi186.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-14T16:38:50.977 INFO:teuthology.orchestra.run.smithi186.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-14T16:38:50.977 INFO:teuthology.orchestra.run.smithi186.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-14T16:38:50.977 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-13.11-1.el9.x86_64 2023-10-14T16:38:50.977 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-14T16:38:50.978 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-14T16:38:50.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-14T16:38:50.979 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-14T16:38:50.980 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-14T16:38:50.980 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-14T16:38:50.980 INFO:teuthology.orchestra.run.smithi186.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-14T16:38:50.980 INFO:teuthology.orchestra.run.smithi186.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-14T16:38:50.980 INFO:teuthology.orchestra.run.smithi186.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-14T16:38:50.981 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-14T16:38:50.982 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-14T16:38:50.982 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-14T16:38:50.982 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-14T16:38:50.982 INFO:teuthology.orchestra.run.smithi186.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-14T16:38:50.982 INFO:teuthology.orchestra.run.smithi186.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-14T16:38:50.983 INFO:teuthology.orchestra.run.smithi186.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-14T16:38:50.983 INFO:teuthology.orchestra.run.smithi186.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-14T16:38:50.983 INFO:teuthology.orchestra.run.smithi186.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-14T16:38:50.983 INFO:teuthology.orchestra.run.smithi186.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-14T16:38:50.983 INFO:teuthology.orchestra.run.smithi186.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-14T16:38:50.984 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-14T16:38:50.984 INFO:teuthology.orchestra.run.smithi186.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-14T16:38:50.984 INFO:teuthology.orchestra.run.smithi186.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-14T16:38:50.984 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-14T16:38:50.984 INFO:teuthology.orchestra.run.smithi186.stdout: python3-ply-3.11-14.el9.noarch 2023-10-14T16:38:50.985 INFO:teuthology.orchestra.run.smithi186.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-14T16:38:50.985 INFO:teuthology.orchestra.run.smithi186.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-14T16:38:50.985 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-14T16:38:50.985 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-14T16:38:50.985 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-14T16:38:50.986 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-14T16:38:50.986 INFO:teuthology.orchestra.run.smithi186.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-14T16:38:50.986 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.986 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.986 INFO:teuthology.orchestra.run.smithi186.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-14T16:38:50.987 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-14T16:38:50.987 INFO:teuthology.orchestra.run.smithi186.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-14T16:38:50.987 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.987 INFO:teuthology.orchestra.run.smithi186.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-14T16:38:50.987 INFO:teuthology.orchestra.run.smithi186.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-14T16:38:50.988 INFO:teuthology.orchestra.run.smithi186.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-14T16:38:50.989 INFO:teuthology.orchestra.run.smithi186.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-14T16:38:50.989 INFO:teuthology.orchestra.run.smithi186.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-14T16:38:50.989 INFO:teuthology.orchestra.run.smithi186.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-14T16:38:50.989 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.989 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: re2-1:20211101-3.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-14T16:38:50.990 INFO:teuthology.orchestra.run.smithi186.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-14T16:38:50.991 INFO:teuthology.orchestra.run.smithi186.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-14T16:38:50.991 INFO:teuthology.orchestra.run.smithi186.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-14T16:38:50.991 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:38:50.991 INFO:teuthology.orchestra.run.smithi186.stdout:Complete! 2023-10-14T16:38:51.045 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 128/153 2023-10-14T16:38:51.121 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 129/153 2023-10-14T16:38:51.174 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libtool-2.4.6-45.el9.x86_64 130/153 2023-10-14T16:38:51.211 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : xfsprogs-devel-5.19.0-4.el9.x86_64 131/153 2023-10-14T16:38:51.240 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 132/153 2023-10-14T16:38:51.287 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-team-1:1.45.4-1.el9.x86_64 133/153 2023-10-14T16:38:51.404 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : NetworkManager-tui-1:1.45.4-1.el9.x86_64 134/153 2023-10-14T16:38:51.458 DEBUG:teuthology.parallel:result is None 2023-10-14T16:38:51.470 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 135/153 2023-10-14T16:38:51.516 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 136/153 2023-10-14T16:38:52.529 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:52.591 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:52.664 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 137/153 2023-10-14T16:38:52.749 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 138/153 2023-10-14T16:38:54.243 INFO:teuthology.orchestra.run.smithi029.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 139/153 2023-10-14T16:38:54.363 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : openssl-devel-1:3.0.7-24.el9.x86_64 140/153 2023-10-14T16:38:54.423 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 141/153 2023-10-14T16:38:54.566 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : flex-2.6.4-9.el9.x86_64 142/153 2023-10-14T16:38:54.611 INFO:teuthology.orchestra.run.smithi029.stdout: Installing : bison-3.7.4-5.el9.x86_64 143/153 2023-10-14T16:38:54.650 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 144/153 2023-10-14T16:38:54.650 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-14T16:38:54.779 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 145/153 2023-10-14T16:38:54.810 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 146/153 2023-10-14T16:38:54.851 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 147/153 2023-10-14T16:38:54.884 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 148/153 2023-10-14T16:38:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:54.924 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:55.260 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 149/153 2023-10-14T16:38:55.260 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-14T16:38:55.451 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 150/153 2023-10-14T16:38:55.491 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-14T16:38:55.491 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-14T16:38:55.639 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 152/153 2023-10-14T16:38:55.776 INFO:teuthology.orchestra.run.smithi029.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:38:56.047 INFO:teuthology.orchestra.run.smithi029.stdout: Running scriptlet: xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:38:56.048 INFO:teuthology.orchestra.run.smithi029.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2023-10-14T16:38:56.048 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:04.436 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-2:18.0.0-6675.gc7471068.el9.x86_64 1/153 2023-10-14T16:39:04.436 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2/153 2023-10-14T16:39:04.436 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 3/153 2023-10-14T16:39:04.437 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 4/153 2023-10-14T16:39:04.437 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-immutable-object-cache-2:18.0.0-6675.gc7471 5/153 2023-10-14T16:39:04.437 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 6/153 2023-10-14T16:39:04.437 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 7/153 2023-10-14T16:39:04.437 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 8/153 2023-10-14T16:39:04.438 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 9/153 2023-10-14T16:39:04.438 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 10/153 2023-10-14T16:39:04.438 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 11/153 2023-10-14T16:39:04.439 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 12/153 2023-10-14T16:39:04.439 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 13/153 2023-10-14T16:39:04.439 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_ 14/153 2023-10-14T16:39:04.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 15/153 2023-10-14T16:39:04.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 16/153 2023-10-14T16:39:04.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados-devel-2:18.0.0-6675.gc7471068.el9.x86_6 17/153 2023-10-14T16:39:04.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86 18/153 2023-10-14T16:39:04.440 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 19/153 2023-10-14T16:39:04.441 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-argparse-2:18.0.0-6675.gc7471068.el 20/153 2023-10-14T16:39:04.441 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ceph-common-2:18.0.0-6675.gc7471068.el9. 21/153 2023-10-14T16:39:04.441 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_6 22/153 2023-10-14T16:39:04.441 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 23/153 2023-10-14T16:39:04.441 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 24/153 2023-10-14T16:39:04.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 25/153 2023-10-14T16:39:04.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 26/153 2023-10-14T16:39:04.442 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 27/153 2023-10-14T16:39:04.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 28/153 2023-10-14T16:39:04.443 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-grafana-dashboards-2:18.0.0-6675.gc7471068. 29/153 2023-10-14T16:39:04.455 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noa 30/153 2023-10-14T16:39:04.455 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.n 31/153 2023-10-14T16:39:04.455 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.0.0-6675.gc74 32/153 2023-10-14T16:39:04.455 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el 33/153 2023-10-14T16:39:04.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 34/153 2023-10-14T16:39:04.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.e 35/153 2023-10-14T16:39:04.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 36/153 2023-10-14T16:39:04.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephadm-2:18.0.0-6675.gc7471068.el9.noarch 37/153 2023-10-14T16:39:04.456 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 38/153 2023-10-14T16:39:04.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : bison-3.7.4-5.el9.x86_64 39/153 2023-10-14T16:39:04.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 40/153 2023-10-14T16:39:04.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flex-2.6.4-9.el9.x86_64 41/153 2023-10-14T16:39:04.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 42/153 2023-10-14T16:39:04.457 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 43/153 2023-10-14T16:39:04.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 44/153 2023-10-14T16:39:04.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 45/153 2023-10-14T16:39:04.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 46/153 2023-10-14T16:39:04.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 47/153 2023-10-14T16:39:04.458 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 48/153 2023-10-14T16:39:04.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 49/153 2023-10-14T16:39:04.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 50/153 2023-10-14T16:39:04.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 51/153 2023-10-14T16:39:04.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 52/153 2023-10-14T16:39:04.459 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 53/153 2023-10-14T16:39:04.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-13.11-1.el9.x86_64 54/153 2023-10-14T16:39:04.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 55/153 2023-10-14T16:39:04.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 56/153 2023-10-14T16:39:04.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 57/153 2023-10-14T16:39:04.460 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 58/153 2023-10-14T16:39:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 59/153 2023-10-14T16:39:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jinja2-2.11.3-4.el9.noarch 60/153 2023-10-14T16:39:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 61/153 2023-10-14T16:39:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 62/153 2023-10-14T16:39:04.461 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 63/153 2023-10-14T16:39:04.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 64/153 2023-10-14T16:39:04.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 65/153 2023-10-14T16:39:04.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 66/153 2023-10-14T16:39:04.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 67/153 2023-10-14T16:39:04.462 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 68/153 2023-10-14T16:39:04.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 69/153 2023-10-14T16:39:04.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 70/153 2023-10-14T16:39:04.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 71/153 2023-10-14T16:39:04.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 72/153 2023-10-14T16:39:04.463 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 73/153 2023-10-14T16:39:04.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 74/153 2023-10-14T16:39:04.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 75/153 2023-10-14T16:39:04.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 76/153 2023-10-14T16:39:04.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 77/153 2023-10-14T16:39:04.464 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libgfortran-11.4.1-2.3.el9.x86_64 78/153 2023-10-14T16:39:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libquadmath-11.4.1-2.3.el9.x86_64 79/153 2023-10-14T16:39:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 80/153 2023-10-14T16:39:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-ply-3.11-14.el9.noarch 81/153 2023-10-14T16:39:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 82/153 2023-10-14T16:39:04.465 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-requests-2.25.1-7.el9.noarch 83/153 2023-10-14T16:39:04.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-urllib3-1.26.5-3.el9.noarch 84/153 2023-10-14T16:39:04.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 85/153 2023-10-14T16:39:04.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : automake-1.16.2-8.el9.noarch 86/153 2023-10-14T16:39:04.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : elfutils-libelf-devel-0.189-3.el9.x86_64 87/153 2023-10-14T16:39:04.466 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : jq-1.6-15.el9.x86_64 88/153 2023-10-14T16:39:04.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 89/153 2023-10-14T16:39:04.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libuuid-devel-2.37.4-15.el9.x86_64 90/153 2023-10-14T16:39:04.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : openssl-devel-1:3.0.7-24.el9.x86_64 91/153 2023-10-14T16:39:04.467 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 92/153 2023-10-14T16:39:04.468 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsprogs-devel-5.19.0-4.el9.x86_64 93/153 2023-10-14T16:39:04.468 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 94/153 2023-10-14T16:39:04.468 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 95/153 2023-10-14T16:39:04.468 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 96/153 2023-10-14T16:39:04.468 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-9.0.0-9.el9.x86_64 97/153 2023-10-14T16:39:04.469 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libarrow-doc-9.0.0-9.el9.noarch 98/153 2023-10-14T16:39:04.469 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 99/153 2023-10-14T16:39:04.469 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 100/153 2023-10-14T16:39:04.469 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : parquet-libs-9.0.0-9.el9.x86_64 101/153 2023-10-14T16:39:04.469 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-asyncssh-2.13.2-3.el9.noarch 102/153 2023-10-14T16:39:04.470 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 103/153 2023-10-14T16:39:04.470 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 104/153 2023-10-14T16:39:04.470 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 105/153 2023-10-14T16:39:04.470 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 106/153 2023-10-14T16:39:04.470 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 107/153 2023-10-14T16:39:04.471 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-google-auth-1:2.23.2-1.el9.noarch 108/153 2023-10-14T16:39:04.471 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 109/153 2023-10-14T16:39:04.471 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 110/153 2023-10-14T16:39:04.471 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 111/153 2023-10-14T16:39:04.471 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 112/153 2023-10-14T16:39:04.472 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 113/153 2023-10-14T16:39:04.472 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 114/153 2023-10-14T16:39:04.472 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 115/153 2023-10-14T16:39:04.472 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 116/153 2023-10-14T16:39:04.472 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 117/153 2023-10-14T16:39:04.473 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 118/153 2023-10-14T16:39:04.473 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 119/153 2023-10-14T16:39:04.473 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 120/153 2023-10-14T16:39:04.473 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 121/153 2023-10-14T16:39:04.473 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 122/153 2023-10-14T16:39:04.474 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 123/153 2023-10-14T16:39:04.474 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 124/153 2023-10-14T16:39:04.474 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 125/153 2023-10-14T16:39:04.474 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 126/153 2023-10-14T16:39:04.474 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 127/153 2023-10-14T16:39:04.475 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 128/153 2023-10-14T16:39:04.475 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 129/153 2023-10-14T16:39:04.475 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 130/153 2023-10-14T16:39:04.475 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 131/153 2023-10-14T16:39:04.475 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : re2-1:20211101-3.el9.x86_64 132/153 2023-10-14T16:39:04.476 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 133/153 2023-10-14T16:39:04.476 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:18.0.0-6675.gc7471068.el9.x86_64 134/153 2023-10-14T16:39:04.476 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 135/153 2023-10-14T16:39:04.476 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 136/153 2023-10-14T16:39:04.476 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 137/153 2023-10-14T16:39:04.477 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 138/153 2023-10-14T16:39:04.477 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 139/153 2023-10-14T16:39:04.477 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.45.4-1.el9.x86_64 140/153 2023-10-14T16:39:04.477 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 141/153 2023-10-14T16:39:04.477 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.45.4-1.el9.x86_64 142/153 2023-10-14T16:39:04.478 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 143/153 2023-10-14T16:39:04.478 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.45.4-1.el9.x86_64 144/153 2023-10-14T16:39:04.478 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 145/153 2023-10-14T16:39:04.478 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.45.4-1.el9.x86_64 146/153 2023-10-14T16:39:04.478 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 147/153 2023-10-14T16:39:04.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-4.el9.x86_64 148/153 2023-10-14T16:39:04.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : acl-2.3.1-3.el9.x86_64 149/153 2023-10-14T16:39:04.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 150/153 2023-10-14T16:39:04.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 151/153 2023-10-14T16:39:04.479 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 152/153 2023-10-14T16:39:06.553 INFO:teuthology.orchestra.run.smithi029.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 153/153 2023-10-14T16:39:06.554 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:06.554 INFO:teuthology.orchestra.run.smithi029.stdout:Upgraded: 2023-10-14T16:39:06.554 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-1:1.45.4-1.el9.x86_64 2023-10-14T16:39:06.555 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-libnm-1:1.45.4-1.el9.x86_64 2023-10-14T16:39:06.555 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-team-1:1.45.4-1.el9.x86_64 2023-10-14T16:39:06.555 INFO:teuthology.orchestra.run.smithi029.stdout: NetworkManager-tui-1:1.45.4-1.el9.x86_64 2023-10-14T16:39:06.555 INFO:teuthology.orchestra.run.smithi029.stdout: acl-2.3.1-4.el9.x86_64 2023-10-14T16:39:06.555 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-2.3.1-4.el9.x86_64 2023-10-14T16:39:06.556 INFO:teuthology.orchestra.run.smithi029.stdout: libacl-devel-2.3.1-4.el9.x86_64 2023-10-14T16:39:06.556 INFO:teuthology.orchestra.run.smithi029.stdout: librados2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.556 INFO:teuthology.orchestra.run.smithi029.stdout: librbd1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.556 INFO:teuthology.orchestra.run.smithi029.stdout: xfsdump-3.1.12-4.el9.x86_64 2023-10-14T16:39:06.556 INFO:teuthology.orchestra.run.smithi029.stdout:Installed: 2023-10-14T16:39:06.557 INFO:teuthology.orchestra.run.smithi029.stdout: automake-1.16.2-8.el9.noarch 2023-10-14T16:39:06.557 INFO:teuthology.orchestra.run.smithi029.stdout: bison-3.7.4-5.el9.x86_64 2023-10-14T16:39:06.557 INFO:teuthology.orchestra.run.smithi029.stdout: boost-program-options-1.75.0-8.el9.x86_64 2023-10-14T16:39:06.557 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.557 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-base-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.558 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.558 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.558 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-grafana-dashboards-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.558 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-immutable-object-cache-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.558 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mds-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.559 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.559 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.559 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-dashboard-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.559 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-diskprediction-local-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.559 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-modules-core-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.560 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mgr-rook-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.560 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-mon-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.560 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-osd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.560 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-prometheus-alerts-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.560 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-radosgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-selinux-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-test-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.561 INFO:teuthology.orchestra.run.smithi029.stdout: ceph-volume-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.561 INFO:teuthology.orchestra.run.smithi029.stdout: cephadm-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.561 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.562 INFO:teuthology.orchestra.run.smithi029.stdout: cephfs-top-2:18.0.0-6675.gc7471068.el9.noarch 2023-10-14T16:39:06.562 INFO:teuthology.orchestra.run.smithi029.stdout: elfutils-libelf-devel-0.189-3.el9.x86_64 2023-10-14T16:39:06.562 INFO:teuthology.orchestra.run.smithi029.stdout: flex-2.6.4-9.el9.x86_64 2023-10-14T16:39:06.562 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-3.0.4-8.el9.x86_64 2023-10-14T16:39:06.562 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2023-10-14T16:39:06.563 INFO:teuthology.orchestra.run.smithi029.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2023-10-14T16:39:06.563 INFO:teuthology.orchestra.run.smithi029.stdout: fmt-8.1.1-5.el9.x86_64 2023-10-14T16:39:06.563 INFO:teuthology.orchestra.run.smithi029.stdout: fuse-2.9.9-15.el9.x86_64 2023-10-14T16:39:06.563 INFO:teuthology.orchestra.run.smithi029.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2023-10-14T16:39:06.563 INFO:teuthology.orchestra.run.smithi029.stdout: jq-1.6-15.el9.x86_64 2023-10-14T16:39:06.564 INFO:teuthology.orchestra.run.smithi029.stdout: libaio-devel-0.3.111-13.el9.x86_64 2023-10-14T16:39:06.564 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-9.0.0-9.el9.x86_64 2023-10-14T16:39:06.564 INFO:teuthology.orchestra.run.smithi029.stdout: libarrow-doc-9.0.0-9.el9.noarch 2023-10-14T16:39:06.564 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.564 INFO:teuthology.orchestra.run.smithi029.stdout: libcephfs2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.565 INFO:teuthology.orchestra.run.smithi029.stdout: libcephsqlite-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.565 INFO:teuthology.orchestra.run.smithi029.stdout: libgfortran-11.4.1-2.3.el9.x86_64 2023-10-14T16:39:06.565 INFO:teuthology.orchestra.run.smithi029.stdout: liboath-2.6.7-2.el9.x86_64 2023-10-14T16:39:06.565 INFO:teuthology.orchestra.run.smithi029.stdout: libpmemobj-1.12.1-1.el9.x86_64 2023-10-14T16:39:06.566 INFO:teuthology.orchestra.run.smithi029.stdout: libquadmath-11.4.1-2.3.el9.x86_64 2023-10-14T16:39:06.566 INFO:teuthology.orchestra.run.smithi029.stdout: librabbitmq-0.11.0-7.el9.x86_64 2023-10-14T16:39:06.566 INFO:teuthology.orchestra.run.smithi029.stdout: librados-devel-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.566 INFO:teuthology.orchestra.run.smithi029.stdout: libradosstriper1-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.566 INFO:teuthology.orchestra.run.smithi029.stdout: librdkafka-1.6.1-102.el9.x86_64 2023-10-14T16:39:06.567 INFO:teuthology.orchestra.run.smithi029.stdout: librgw2-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.567 INFO:teuthology.orchestra.run.smithi029.stdout: libtool-2.4.6-45.el9.x86_64 2023-10-14T16:39:06.567 INFO:teuthology.orchestra.run.smithi029.stdout: libunwind-1.6.2-1.el9.x86_64 2023-10-14T16:39:06.567 INFO:teuthology.orchestra.run.smithi029.stdout: libuuid-devel-2.37.4-15.el9.x86_64 2023-10-14T16:39:06.567 INFO:teuthology.orchestra.run.smithi029.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: lttng-ust-2.12.0-6.el9.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-0.3.21-2.el9.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: openssl-devel-1:3.0.7-24.el9.x86_64 2023-10-14T16:39:06.568 INFO:teuthology.orchestra.run.smithi029.stdout: parquet-libs-9.0.0-9.el9.x86_64 2023-10-14T16:39:06.569 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-13.11-1.el9.x86_64 2023-10-14T16:39:06.569 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-contrib-13.11-1.el9.x86_64 2023-10-14T16:39:06.569 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2023-10-14T16:39:06.569 INFO:teuthology.orchestra.run.smithi029.stdout: postgresql-server-13.11-1.el9.x86_64 2023-10-14T16:39:06.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-asyncssh-2.13.2-3.el9.noarch 2023-10-14T16:39:06.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-babel-2.9.1-2.el9.noarch 2023-10-14T16:39:06.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2023-10-14T16:39:06.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cachetools-4.2.4-1.el9.noarch 2023-10-14T16:39:06.570 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-argparse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ceph-common-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cephfs-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-certifi-2021.10.8-2.el9.noarch 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cffi-1.14.5-5.el9.x86_64 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cheroot-8.6.0-4.el9.noarch 2023-10-14T16:39:06.571 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2023-10-14T16:39:06.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2023-10-14T16:39:06.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-devel-3.9.17-2.el9.x86_64 2023-10-14T16:39:06.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-google-auth-1:2.23.2-1.el9.noarch 2023-10-14T16:39:06.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-8.2.1-3.el9.noarch 2023-10-14T16:39:06.572 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2023-10-14T16:39:06.573 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2023-10-14T16:39:06.573 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2023-10-14T16:39:06.573 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2023-10-14T16:39:06.573 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jinja2-2.11.3-4.el9.noarch 2023-10-14T16:39:06.573 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpatch-1.21-16.el9.noarch 2023-10-14T16:39:06.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jsonpointer-2.0-4.el9.noarch 2023-10-14T16:39:06.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt-2.4.0-1.el9.noarch 2023-10-14T16:39:06.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2023-10-14T16:39:06.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2023-10-14T16:39:06.574 INFO:teuthology.orchestra.run.smithi029.stdout: python3-logutils-0.3.5-21.el9.noarch 2023-10-14T16:39:06.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-mako-1.1.4-6.el9.noarch 2023-10-14T16:39:06.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2023-10-14T16:39:06.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2023-10-14T16:39:06.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-natsort-7.1.1-5.el9.noarch 2023-10-14T16:39:06.575 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2023-10-14T16:39:06.576 INFO:teuthology.orchestra.run.smithi029.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2023-10-14T16:39:06.576 INFO:teuthology.orchestra.run.smithi029.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2023-10-14T16:39:06.576 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pecan-1.4.2-3.el9.noarch 2023-10-14T16:39:06.576 INFO:teuthology.orchestra.run.smithi029.stdout: python3-ply-3.11-14.el9.noarch 2023-10-14T16:39:06.576 INFO:teuthology.orchestra.run.smithi029.stdout: python3-portend-3.1.0-2.el9.noarch 2023-10-14T16:39:06.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-prettytable-0.7.2-27.el9.noarch 2023-10-14T16:39:06.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2023-10-14T16:39:06.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2023-10-14T16:39:06.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2023-10-14T16:39:06.577 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pycparser-2.20-6.el9.noarch 2023-10-14T16:39:06.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-pytz-2021.1-5.el9.noarch 2023-10-14T16:39:06.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rados-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-repoze-lru-0.7-16.el9.noarch 2023-10-14T16:39:06.578 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-2.25.1-7.el9.noarch 2023-10-14T16:39:06.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2023-10-14T16:39:06.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rgw-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-routes-2.5.1-5.el9.noarch 2023-10-14T16:39:06.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-rsa-4.9-2.el9.noarch 2023-10-14T16:39:06.579 INFO:teuthology.orchestra.run.smithi029.stdout: python3-scipy-1.6.2-8.el9.x86_64 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-tempora-5.0.0-2.el9.noarch 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-toml-0.10.2-6.el9.noarch 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-urllib3-1.26.5-3.el9.noarch 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-webob-1.8.7-6.el9.noarch 2023-10-14T16:39:06.580 INFO:teuthology.orchestra.run.smithi029.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2023-10-14T16:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2023-10-14T16:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2023-10-14T16:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-fuse-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-mirror-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.581 INFO:teuthology.orchestra.run.smithi029.stdout: rbd-nbd-2:18.0.0-6675.gc7471068.el9.x86_64 2023-10-14T16:39:06.582 INFO:teuthology.orchestra.run.smithi029.stdout: re2-1:20211101-3.el9.x86_64 2023-10-14T16:39:06.582 INFO:teuthology.orchestra.run.smithi029.stdout: socat-1.7.4.1-5.el9.x86_64 2023-10-14T16:39:06.582 INFO:teuthology.orchestra.run.smithi029.stdout: thrift-0.15.0-2.el9.x86_64 2023-10-14T16:39:06.582 INFO:teuthology.orchestra.run.smithi029.stdout: uuid-1.6.2-55.el9.x86_64 2023-10-14T16:39:06.582 INFO:teuthology.orchestra.run.smithi029.stdout: xfsprogs-devel-5.19.0-4.el9.x86_64 2023-10-14T16:39:06.583 INFO:teuthology.orchestra.run.smithi029.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2023-10-14T16:39:06.583 INFO:teuthology.orchestra.run.smithi029.stdout: zlib-devel-1.2.11-40.el9.x86_64 2023-10-14T16:39:06.583 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:06.584 INFO:teuthology.orchestra.run.smithi029.stdout:Complete! 2023-10-14T16:39:07.260 DEBUG:teuthology.parallel:result is None 2023-10-14T16:39:07.261 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-14T16:39:07.490 DEBUG:teuthology.orchestra.run.smithi029:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-14T16:39:07.547 INFO:teuthology.orchestra.run.smithi029.stdout:18.0.0-6675.gc7471068.el9 2023-10-14T16:39:07.548 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el9 2023-10-14T16:39:07.548 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-14T16:39:07.549 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c7471068426370a3754c2250e7729280233d552b 2023-10-14T16:39:07.759 DEBUG:teuthology.orchestra.run.smithi186:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-10-14T16:39:07.830 INFO:teuthology.orchestra.run.smithi186.stdout:18.0.0-6675.gc7471068.el9 2023-10-14T16:39:07.830 INFO:teuthology.packaging:The installed version of ceph is 18.0.0-6675.gc7471068.el9 2023-10-14T16:39:07.831 INFO:teuthology.task.install:The correct ceph version 18.0.0-6675.gc7471068 is installed. 2023-10-14T16:39:07.831 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-10-14T16:39:07.832 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:07.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T16:39:07.919 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:07.919 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-10-14T16:39:08.022 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-10-14T16:39:08.022 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:08.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T16:39:08.100 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T16:39:08.216 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:08.216 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/daemon-helper 2023-10-14T16:39:08.323 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-10-14T16:39:08.460 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-10-14T16:39:08.461 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:08.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T16:39:08.538 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T16:39:08.653 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:08.654 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/adjust-ulimits 2023-10-14T16:39:08.744 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-10-14T16:39:08.866 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-10-14T16:39:08.866 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:08.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T16:39:08.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T16:39:09.051 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:09.052 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/usr/bin/stdin-killer 2023-10-14T16:39:09.142 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-10-14T16:39:09.216 INFO:teuthology.run_tasks:Running task ceph... 2023-10-14T16:39:09.281 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-10-14T16:39:09.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 777 /var/log/ceph 2023-10-14T16:39:09.288 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 777 /var/log/ceph 2023-10-14T16:39:09.369 INFO:tasks.ceph:Disabling ceph logrotate... 2023-10-14T16:39:09.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T16:39:09.374 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/logrotate.d/ceph 2023-10-14T16:39:09.452 INFO:tasks.ceph:Creating extra log directories... 2023-10-14T16:39:09.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T16:39:09.458 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-10-14T16:39:09.551 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2023-10-14T16:39:09.552 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2023-10-14T16:39:09.552 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2023-10-14T16:39:09.552 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:09.553 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-14T16:39:09.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:09.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:09.957 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:09.958 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2023-10-14T16:39:10.015 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:10.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:10.397 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-10-14T16:39:10.398 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-10-14T16:39:10.399 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857', 'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule}', 'email': 'mchangir@redhat.com', 'first_in_suite': False, 'job_id': '7427857', 'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}, 'ktype': 'upstream', 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, 'kclient': {'syntax': 'v1'}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, 'owner': 'mchangir', 'priority': 25, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 5837, 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c7471068426370a3754c2250e7729280233d552b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c7471068426370a3754c2250e7729280233d552b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi029.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIenUNMjS8jLbCKDUoA7LlCVAfZYMnpf8GelG1yGs0Dgy+1xDbuTILZXW1iFdqKmiHAbnH8lIKg+zsbMUw3XwYQ=', 'smithi186.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU6xheaByxOWomGjN7eaKx5atbspCF5BwcdL4eoU8q6q3asLPy0dBw6fYz2uLTcrEYRjiqWnzO8he55rz4DVHM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c7471068426370a3754c2250e7729280233d552b'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug finisher': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'log-whitelist': ['OSD full dropping all updates', 'OSD near full', 'pausewr flag', 'failsafe engaged, dropping updates', 'failsafe disengaged, no longer dropping', 'is full \\(reached quota', 'POOL_FULL', 'POOL_BACKFILLFULL'], 'sha1': 'c7471068426370a3754c2250e7729280233d552b', 'cluster': 'ceph'}}, {'kclient': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_snap_schedules']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ["local function is_kupstream() return yaml.ktype == 'upstream' end", "local function is_kdistro() return yaml.ktype == 'distro' end", 'local function is_fuse() return false end', 'local function syntax_version() return yaml.overrides.kclient.syntax end']}, 'teuthology_branch': 'main', 'teuthology_sha1': '8cdab074dcca9a68965bc5a50e9c30b691949723', 'timestamp': '2023-10-14_16:12:46', 'tube': 'smithi', 'user': 'mchangir', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-10-14T16:39:10.400 DEBUG:teuthology.orchestra.run.smithi029:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T16:39:10.409 DEBUG:teuthology.orchestra.run.smithi186:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-10-14T16:39:10.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T16:39:10.474 DEBUG:teuthology.orchestra.run.smithi186:> sudo install -d -m0777 -- /var/run/ceph 2023-10-14T16:39:10.572 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:10.573 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T16:39:10.634 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T16:39:10.634 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_1 2023-10-14T16:39:10.688 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T16:39:10.689 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 638 Links: 1 2023-10-14T16:39:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:10.689 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-14 16:38:56.565033796 +0000 2023-10-14T16:39:10.690 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-14 16:36:23.209884648 +0000 2023-10-14T16:39:10.690 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-14 16:36:23.209884648 +0000 2023-10-14T16:39:10.690 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-14 16:36:23.209884648 +0000 2023-10-14T16:39:10.691 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T16:39:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-14T16:39:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-14T16:39:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000388258 s, 1.3 MB/s 2023-10-14T16:39:10.795 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T16:39:10.852 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_2 2023-10-14T16:39:10.900 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T16:39:10.900 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:10.901 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 656 Links: 1 2023-10-14T16:39:10.901 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:10.901 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:10.901 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-14 16:38:56.566033780 +0000 2023-10-14T16:39:10.901 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-14 16:36:24.244868101 +0000 2023-10-14T16:39:10.902 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-14 16:36:24.245868085 +0000 2023-10-14T16:39:10.902 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-14 16:36:24.244868101 +0000 2023-10-14T16:39:10.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T16:39:11.044 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-14T16:39:11.045 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-14T16:39:11.045 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000246213 s, 2.1 MB/s 2023-10-14T16:39:11.049 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T16:39:11.114 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_3 2023-10-14T16:39:11.171 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T16:39:11.172 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:11.172 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 675 Links: 1 2023-10-14T16:39:11.173 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:11.173 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:11.173 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-14 16:38:56.566033780 +0000 2023-10-14T16:39:11.174 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-14 16:36:25.300851217 +0000 2023-10-14T16:39:11.174 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-14 16:36:25.300851217 +0000 2023-10-14T16:39:11.175 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-14 16:36:25.300851217 +0000 2023-10-14T16:39:11.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T16:39:11.275 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-14T16:39:11.275 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-14T16:39:11.276 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000253193 s, 2.0 MB/s 2023-10-14T16:39:11.280 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T16:39:11.342 DEBUG:teuthology.orchestra.run.smithi029:> stat /dev/vg_nvme/lv_4 2023-10-14T16:39:11.396 INFO:teuthology.orchestra.run.smithi029.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T16:39:11.396 INFO:teuthology.orchestra.run.smithi029.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Device: 5h/5d Inode: 694 Links: 1 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Access: 2023-10-14 16:38:56.567033765 +0000 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Modify: 2023-10-14 16:36:26.362834237 +0000 2023-10-14T16:39:11.397 INFO:teuthology.orchestra.run.smithi029.stdout:Change: 2023-10-14 16:36:26.362834237 +0000 2023-10-14T16:39:11.398 INFO:teuthology.orchestra.run.smithi029.stdout: Birth: 2023-10-14 16:36:26.362834237 +0000 2023-10-14T16:39:11.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T16:39:11.489 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records in 2023-10-14T16:39:11.490 INFO:teuthology.orchestra.run.smithi029.stderr:1+0 records out 2023-10-14T16:39:11.490 INFO:teuthology.orchestra.run.smithi029.stderr:512 bytes copied, 0.000267392 s, 1.9 MB/s 2023-10-14T16:39:11.493 DEBUG:teuthology.orchestra.run.smithi029:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T16:39:11.552 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:11.553 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:11.553 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/scratch_devs of=/dev/stdout 2023-10-14T16:39:11.612 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-14T16:39:11.613 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_1 2023-10-14T16:39:11.665 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-14T16:39:11.665 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:11.665 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 633 Links: 1 2023-10-14T16:39:11.665 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:11.666 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:11.666 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2023-10-14 16:38:44.582154829 +0000 2023-10-14T16:39:11.666 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2023-10-14 16:35:53.514742703 +0000 2023-10-14T16:39:11.666 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2023-10-14 16:35:53.514742703 +0000 2023-10-14T16:39:11.666 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2023-10-14 16:35:53.514742703 +0000 2023-10-14T16:39:11.667 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-14T16:39:11.759 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2023-10-14T16:39:11.759 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2023-10-14T16:39:11.759 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000336972 s, 1.5 MB/s 2023-10-14T16:39:11.762 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-14T16:39:11.823 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_2 2023-10-14T16:39:11.875 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-14T16:39:11.876 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:11.876 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 652 Links: 1 2023-10-14T16:39:11.876 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:11.876 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:11.877 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2023-10-14 16:38:44.582154829 +0000 2023-10-14T16:39:11.877 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2023-10-14 16:35:54.578724089 +0000 2023-10-14T16:39:11.877 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2023-10-14 16:35:54.578724089 +0000 2023-10-14T16:39:11.877 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2023-10-14 16:35:54.578724089 +0000 2023-10-14T16:39:11.877 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-14T16:39:11.966 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2023-10-14T16:39:11.966 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2023-10-14T16:39:11.967 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000309745 s, 1.7 MB/s 2023-10-14T16:39:11.970 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-14T16:39:12.027 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_3 2023-10-14T16:39:12.072 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-14T16:39:12.073 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:12.073 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 672 Links: 1 2023-10-14T16:39:12.073 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:12.073 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:12.073 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2023-10-14 16:38:44.583154812 +0000 2023-10-14T16:39:12.074 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2023-10-14 16:35:55.606706104 +0000 2023-10-14T16:39:12.074 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2023-10-14 16:35:55.606706104 +0000 2023-10-14T16:39:12.074 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2023-10-14 16:35:55.606706104 +0000 2023-10-14T16:39:12.074 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-14T16:39:12.209 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2023-10-14T16:39:12.209 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2023-10-14T16:39:12.209 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000338339 s, 1.5 MB/s 2023-10-14T16:39:12.212 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-14T16:39:12.271 DEBUG:teuthology.orchestra.run.smithi186:> stat /dev/vg_nvme/lv_4 2023-10-14T16:39:12.320 INFO:teuthology.orchestra.run.smithi186.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-14T16:39:12.320 INFO:teuthology.orchestra.run.smithi186.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-14T16:39:12.321 INFO:teuthology.orchestra.run.smithi186.stdout:Device: 5h/5d Inode: 690 Links: 1 2023-10-14T16:39:12.321 INFO:teuthology.orchestra.run.smithi186.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-14T16:39:12.321 INFO:teuthology.orchestra.run.smithi186.stdout:Context: system_u:object_r:device_t:s0 2023-10-14T16:39:12.321 INFO:teuthology.orchestra.run.smithi186.stdout:Access: 2023-10-14 16:38:44.583154812 +0000 2023-10-14T16:39:12.322 INFO:teuthology.orchestra.run.smithi186.stdout:Modify: 2023-10-14 16:35:56.632688155 +0000 2023-10-14T16:39:12.322 INFO:teuthology.orchestra.run.smithi186.stdout:Change: 2023-10-14 16:35:56.632688155 +0000 2023-10-14T16:39:12.322 INFO:teuthology.orchestra.run.smithi186.stdout: Birth: 2023-10-14 16:35:56.632688155 +0000 2023-10-14T16:39:12.322 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-14T16:39:12.404 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records in 2023-10-14T16:39:12.405 INFO:teuthology.orchestra.run.smithi186.stderr:1+0 records out 2023-10-14T16:39:12.405 INFO:teuthology.orchestra.run.smithi186.stderr:512 bytes copied, 0.000365768 s, 1.4 MB/s 2023-10-14T16:39:12.407 DEBUG:teuthology.orchestra.run.smithi186:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-14T16:39:12.464 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:12.465 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-14T16:39:12.465 INFO:tasks.ceph:Generating config... 2023-10-14T16:39:12.466 INFO:tasks.ceph:[client] client mount timeout = 600 2023-10-14T16:39:12.467 INFO:tasks.ceph:[client] debug client = 20 2023-10-14T16:39:12.467 INFO:tasks.ceph:[client] debug ms = 1 2023-10-14T16:39:12.467 INFO:tasks.ceph:[client] pid file = 2023-10-14T16:39:12.467 INFO:tasks.ceph:[client] rados mon op timeout = 900 2023-10-14T16:39:12.468 INFO:tasks.ceph:[client] rados osd op timeout = 900 2023-10-14T16:39:12.468 INFO:tasks.ceph:[global] ms die on skipped message = False 2023-10-14T16:39:12.468 INFO:tasks.ceph:[mds] debug mds = 20 2023-10-14T16:39:12.469 INFO:tasks.ceph:[mds] debug mds balancer = 20 2023-10-14T16:39:12.469 INFO:tasks.ceph:[mds] debug ms = 1 2023-10-14T16:39:12.469 INFO:tasks.ceph:[mds] mds debug frag = True 2023-10-14T16:39:12.469 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2023-10-14T16:39:12.470 INFO:tasks.ceph:[mds] mds op complaint time = 180 2023-10-14T16:39:12.470 INFO:tasks.ceph:[mds] mds verify scatter = True 2023-10-14T16:39:12.470 INFO:tasks.ceph:[mds] osd op complaint time = 180 2023-10-14T16:39:12.470 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2023-10-14T16:39:12.471 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2023-10-14T16:39:12.471 INFO:tasks.ceph:[mgr] debug client = 20 2023-10-14T16:39:12.471 INFO:tasks.ceph:[mgr] debug finisher = 20 2023-10-14T16:39:12.471 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-10-14T16:39:12.472 INFO:tasks.ceph:[mgr] debug ms = 1 2023-10-14T16:39:12.472 INFO:tasks.ceph:[mon] debug mon = 20 2023-10-14T16:39:12.472 INFO:tasks.ceph:[mon] debug ms = 1 2023-10-14T16:39:12.472 INFO:tasks.ceph:[mon] debug paxos = 20 2023-10-14T16:39:12.473 INFO:tasks.ceph:[mon] mon op complaint time = 120 2023-10-14T16:39:12.473 INFO:tasks.ceph:[osd] bdev async discard = True 2023-10-14T16:39:12.473 INFO:tasks.ceph:[osd] bdev enable discard = True 2023-10-14T16:39:12.474 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2023-10-14T16:39:12.474 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-10-14T16:39:12.474 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-10-14T16:39:12.474 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2023-10-14T16:39:12.475 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2023-10-14T16:39:12.475 INFO:tasks.ceph:[osd] debug ms = 1 2023-10-14T16:39:12.475 INFO:tasks.ceph:[osd] debug osd = 20 2023-10-14T16:39:12.475 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2023-10-14T16:39:12.476 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-10-14T16:39:12.476 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-10-14T16:39:12.476 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-10-14T16:39:12.476 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-10-14T16:39:12.477 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-10-14T16:39:12.477 INFO:tasks.ceph:[osd] osd op complaint time = 180 2023-10-14T16:39:12.477 INFO:tasks.ceph:Setting up mon.a... 2023-10-14T16:39:12.478 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-10-14T16:39:12.628 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.keyring 2023-10-14T16:39:12.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-10-14T16:39:12.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T16:39:12.877 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.29'), ('mon.b', '172.21.15.186'), ('mon.c', '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]')] 2023-10-14T16:39:12.877 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.29,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]', 'ms die on skipped message': False}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20, 'debug finisher': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2023-10-14T16:39:12.878 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:12.879 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T16:39:12.929 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.29 --add b 172.21.15.186 --addv c '[v2:172.21.15.186:3301,v1:172.21.15.186:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:13.095 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:13.095 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: generated fsid f6aee177-7aff-4117-99f6-f2cafd5adc35 2023-10-14T16:39:13.095 INFO:teuthology.orchestra.run.smithi029.stdout:setting min_mon_release = pacific 2023-10-14T16:39:13.095 INFO:teuthology.orchestra.run.smithi029.stdout:epoch 0 2023-10-14T16:39:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:fsid f6aee177-7aff-4117-99f6-f2cafd5adc35 2023-10-14T16:39:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:last_changed 2023-10-14T16:39:13.095101+0000 2023-10-14T16:39:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:created 2023-10-14T16:39:13.095101+0000 2023-10-14T16:39:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:min_mon_release 16 (pacific) 2023-10-14T16:39:13.096 INFO:teuthology.orchestra.run.smithi029.stdout:election_strategy: 1 2023-10-14T16:39:13.097 INFO:teuthology.orchestra.run.smithi029.stdout:0: [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] mon.a 2023-10-14T16:39:13.097 INFO:teuthology.orchestra.run.smithi029.stdout:1: [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] mon.b 2023-10-14T16:39:13.097 INFO:teuthology.orchestra.run.smithi029.stdout:2: [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] mon.c 2023-10-14T16:39:13.097 INFO:teuthology.orchestra.run.smithi029.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-10-14T16:39:13.098 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-10-14T16:39:13.152 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f6aee177-7aff-4117-99f6-f2cafd5adc35... 2023-10-14T16:39:13.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T16:39:13.159 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-10-14T16:39:13.334 INFO:teuthology.orchestra.run.smithi029.stdout:[global] 2023-10-14T16:39:13.335 INFO:teuthology.orchestra.run.smithi029.stdout: chdir = "" 2023-10-14T16:39:13.335 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T16:39:13.335 INFO:teuthology.orchestra.run.smithi029.stdout: auth supported = cephx 2023-10-14T16:39:13.335 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.336 INFO:teuthology.orchestra.run.smithi029.stdout: filestore xattr use omap = true 2023-10-14T16:39:13.336 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.336 INFO:teuthology.orchestra.run.smithi029.stdout: mon clock drift allowed = 1.000 2023-10-14T16:39:13.336 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.336 INFO:teuthology.orchestra.run.smithi029.stdout: osd crush chooseleaf type = 0 2023-10-14T16:39:13.337 INFO:teuthology.orchestra.run.smithi029.stdout: auth debug = true 2023-10-14T16:39:13.337 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.337 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on old message = true 2023-10-14T16:39:13.337 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on bug = true 2023-10-14T16:39:13.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.338 INFO:teuthology.orchestra.run.smithi029.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T16:39:13.338 INFO:teuthology.orchestra.run.smithi029.stdout: mon pg warn max object skew = 0 2023-10-14T16:39:13.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.338 INFO:teuthology.orchestra.run.smithi029.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T16:39:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T16:39:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default size = 2 2023-10-14T16:39:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.339 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow primary affinity = true 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd allow pg remap = true 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on legacy crush tunables = false 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on crush straw calc version zero = false 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on no sortbitwise = false 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on osd down out interval zero = false 2023-10-14T16:39:13.340 INFO:teuthology.orchestra.run.smithi029.stdout: mon warn on too few osds = false 2023-10-14T16:39:13.341 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T16:39:13.341 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T16:39:13.341 INFO:teuthology.orchestra.run.smithi029.stdout: mon_allow_pool_size_one = true 2023-10-14T16:39:13.341 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.341 INFO:teuthology.orchestra.run.smithi029.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T16:39:13.342 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.342 INFO:teuthology.orchestra.run.smithi029.stdout: osd default data pool replay window = 5 2023-10-14T16:39:13.342 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.342 INFO:teuthology.orchestra.run.smithi029.stdout: mon allow pool delete = true 2023-10-14T16:39:13.343 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.343 INFO:teuthology.orchestra.run.smithi029.stdout: mon cluster log file level = debug 2023-10-14T16:39:13.343 INFO:teuthology.orchestra.run.smithi029.stdout: debug asserts on shutdown = true 2023-10-14T16:39:13.343 INFO:teuthology.orchestra.run.smithi029.stdout: mon health detail to clog = false 2023-10-14T16:39:13.343 INFO:teuthology.orchestra.run.smithi029.stdout: mon host = "172.21.15.29,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout: ms die on skipped message = False 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout: fsid = f6aee177-7aff-4117-99f6-f2cafd5adc35 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout:[osd] 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout: osd journal size = 100 2023-10-14T16:39:13.344 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.345 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub load threshold = 5.0 2023-10-14T16:39:13.345 INFO:teuthology.orchestra.run.smithi029.stdout: osd scrub max interval = 600 2023-10-14T16:39:13.345 INFO:teuthology.orchestra.run.smithi029.stdout: osd mclock profile = high_recovery_ops 2023-10-14T16:39:13.345 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.345 INFO:teuthology.orchestra.run.smithi029.stdout: osd recover clone overlap = true 2023-10-14T16:39:13.346 INFO:teuthology.orchestra.run.smithi029.stdout: osd recovery max chunk = 1048576 2023-10-14T16:39:13.346 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.346 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug shutdown = true 2023-10-14T16:39:13.346 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug op order = true 2023-10-14T16:39:13.346 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug verify stray on activate = true 2023-10-14T16:39:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: osd open classes on start = true 2023-10-14T16:39:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug pg log writeout = true 2023-10-14T16:39:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.347 INFO:teuthology.orchestra.run.smithi029.stdout: osd deep scrub update digest min age = 30 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: osd map max advance = 10 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: journal zero on create = true 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.348 INFO:teuthology.orchestra.run.smithi029.stdout: filestore ondisk finisher threads = 3 2023-10-14T16:39:13.349 INFO:teuthology.orchestra.run.smithi029.stdout: filestore apply finisher threads = 3 2023-10-14T16:39:13.349 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.349 INFO:teuthology.orchestra.run.smithi029.stdout: bdev debug aio = true 2023-10-14T16:39:13.349 INFO:teuthology.orchestra.run.smithi029.stdout: osd debug misdirected ops = true 2023-10-14T16:39:13.349 INFO:teuthology.orchestra.run.smithi029.stdout: bdev async discard = True 2023-10-14T16:39:13.350 INFO:teuthology.orchestra.run.smithi029.stdout: bdev enable discard = True 2023-10-14T16:39:13.350 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore allocator = bitmap 2023-10-14T16:39:13.350 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore block size = 96636764160 2023-10-14T16:39:13.350 INFO:teuthology.orchestra.run.smithi029.stdout: bluestore fsck on mount = True 2023-10-14T16:39:13.350 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluefs = 1/20 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: debug bluestore = 1/20 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: debug osd = 20 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: debug rocksdb = 4/10 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T16:39:13.351 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd full ratio = 0.9 2023-10-14T16:39:13.352 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd nearfull ratio = 0.8 2023-10-14T16:39:13.352 INFO:teuthology.orchestra.run.smithi029.stdout: osd failsafe full ratio = 0.95 2023-10-14T16:39:13.352 INFO:teuthology.orchestra.run.smithi029.stdout: osd objectstore = bluestore 2023-10-14T16:39:13.352 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-14T16:39:13.352 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr] 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout: debug mgr = 20 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-14T16:39:13.353 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-14T16:39:13.354 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-14T16:39:13.354 INFO:teuthology.orchestra.run.smithi029.stdout: mgr/telemetry/nag = false 2023-10-14T16:39:13.354 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-14T16:39:13.354 INFO:teuthology.orchestra.run.smithi029.stdout: debug finisher = 20 2023-10-14T16:39:13.354 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.355 INFO:teuthology.orchestra.run.smithi029.stdout:[mon] 2023-10-14T16:39:13.355 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-14T16:39:13.355 INFO:teuthology.orchestra.run.smithi029.stdout: debug mon = 20 2023-10-14T16:39:13.355 INFO:teuthology.orchestra.run.smithi029.stdout: debug paxos = 20 2023-10-14T16:39:13.355 INFO:teuthology.orchestra.run.smithi029.stdout: debug auth = 20 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon data avail warn = 5 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon mgr mkfs grace = 240 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min pgs per osd = 4 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd reporter subtree level = osd 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon osd prime pg temp = true 2023-10-14T16:39:13.356 INFO:teuthology.orchestra.run.smithi029.stdout: mon reweight min bytes per osd = 10 2023-10-14T16:39:13.357 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.357 INFO:teuthology.orchestra.run.smithi029.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T16:39:13.357 INFO:teuthology.orchestra.run.smithi029.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T16:39:13.357 INFO:teuthology.orchestra.run.smithi029.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T16:39:13.357 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.358 INFO:teuthology.orchestra.run.smithi029.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T16:39:13.358 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T16:39:13.358 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T16:39:13.358 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.358 INFO:teuthology.orchestra.run.smithi029.stdout: # 1m isn't quite enough 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout: mon_down_mkfs_grace = 2m 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout: mon_warn_on_filestore_osds = false 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout: mon op complaint time = 120 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:39:13.359 INFO:teuthology.orchestra.run.smithi029.stdout:[client] 2023-10-14T16:39:13.360 INFO:teuthology.orchestra.run.smithi029.stdout: rgw cache enabled = true 2023-10-14T16:39:13.360 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable ops log = true 2023-10-14T16:39:13.360 INFO:teuthology.orchestra.run.smithi029.stdout: rgw enable usage log = true 2023-10-14T16:39:13.360 INFO:teuthology.orchestra.run.smithi029.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T16:39:13.360 INFO:teuthology.orchestra.run.smithi029.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: client mount timeout = 600 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: debug client = 20 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: pid file = "" 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-14T16:39:13.361 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-14T16:39:13.362 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.a] 2023-10-14T16:39:13.362 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.b] 2023-10-14T16:39:13.362 INFO:teuthology.orchestra.run.smithi029.stdout:[mon.c] 2023-10-14T16:39:13.362 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-14T16:39:13.362 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-14T16:39:13.363 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-14T16:39:13.363 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-14T16:39:13.363 INFO:teuthology.orchestra.run.smithi029.stdout:[mds] 2023-10-14T16:39:13.363 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds = 20 2023-10-14T16:39:13.363 INFO:teuthology.orchestra.run.smithi029.stdout: debug mds balancer = 20 2023-10-14T16:39:13.364 INFO:teuthology.orchestra.run.smithi029.stdout: debug ms = 1 2023-10-14T16:39:13.364 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug frag = True 2023-10-14T16:39:13.364 INFO:teuthology.orchestra.run.smithi029.stdout: mds debug scatterstat = True 2023-10-14T16:39:13.364 INFO:teuthology.orchestra.run.smithi029.stdout: mds op complaint time = 180 2023-10-14T16:39:13.364 INFO:teuthology.orchestra.run.smithi029.stdout: mds verify scatter = True 2023-10-14T16:39:13.365 INFO:teuthology.orchestra.run.smithi029.stdout: osd op complaint time = 180 2023-10-14T16:39:13.365 INFO:teuthology.orchestra.run.smithi029.stdout: rados mon op timeout = 900 2023-10-14T16:39:13.365 INFO:teuthology.orchestra.run.smithi029.stdout: rados osd op timeout = 900 2023-10-14T16:39:13.365 INFO:teuthology.orchestra.run.smithi186.stdout:[global] 2023-10-14T16:39:13.366 INFO:teuthology.orchestra.run.smithi186.stdout: chdir = "" 2023-10-14T16:39:13.366 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-10-14T16:39:13.366 INFO:teuthology.orchestra.run.smithi186.stdout: auth supported = cephx 2023-10-14T16:39:13.366 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.366 INFO:teuthology.orchestra.run.smithi186.stdout: filestore xattr use omap = true 2023-10-14T16:39:13.367 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.367 INFO:teuthology.orchestra.run.smithi186.stdout: mon clock drift allowed = 1.000 2023-10-14T16:39:13.367 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.367 INFO:teuthology.orchestra.run.smithi186.stdout: osd crush chooseleaf type = 0 2023-10-14T16:39:13.367 INFO:teuthology.orchestra.run.smithi186.stdout: auth debug = true 2023-10-14T16:39:13.368 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.368 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on old message = true 2023-10-14T16:39:13.368 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on bug = true 2023-10-14T16:39:13.368 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.368 INFO:teuthology.orchestra.run.smithi186.stdout: mon max pg per osd = 10000 # >= luminous 2023-10-14T16:39:13.369 INFO:teuthology.orchestra.run.smithi186.stdout: mon pg warn max object skew = 0 2023-10-14T16:39:13.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.369 INFO:teuthology.orchestra.run.smithi186.stdout: # disable pg_autoscaler by default for new pools 2023-10-14T16:39:13.369 INFO:teuthology.orchestra.run.smithi186.stdout: osd_pool_default_pg_autoscale_mode = off 2023-10-14T16:39:13.369 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.370 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default size = 2 2023-10-14T16:39:13.370 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.370 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow primary affinity = true 2023-10-14T16:39:13.370 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd allow pg remap = true 2023-10-14T16:39:13.370 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on legacy crush tunables = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on crush straw calc version zero = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on no sortbitwise = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on osd down out interval zero = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon warn on too few osds = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-10-14T16:39:13.371 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_pool_no_redundancy = false 2023-10-14T16:39:13.372 INFO:teuthology.orchestra.run.smithi186.stdout: mon_allow_pool_size_one = true 2023-10-14T16:39:13.372 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.372 INFO:teuthology.orchestra.run.smithi186.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-10-14T16:39:13.372 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.372 INFO:teuthology.orchestra.run.smithi186.stdout: osd default data pool replay window = 5 2023-10-14T16:39:13.373 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.373 INFO:teuthology.orchestra.run.smithi186.stdout: mon allow pool delete = true 2023-10-14T16:39:13.373 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.373 INFO:teuthology.orchestra.run.smithi186.stdout: mon cluster log file level = debug 2023-10-14T16:39:13.373 INFO:teuthology.orchestra.run.smithi186.stdout: debug asserts on shutdown = true 2023-10-14T16:39:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: mon health detail to clog = false 2023-10-14T16:39:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: mon host = "172.21.15.29,172.21.15.186,[v2:172.21.15.186:3301,v1:172.21.15.186:6790]" 2023-10-14T16:39:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: ms die on skipped message = False 2023-10-14T16:39:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: fsid = f6aee177-7aff-4117-99f6-f2cafd5adc35 2023-10-14T16:39:13.374 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout:[osd] 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout: osd journal size = 100 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub load threshold = 5.0 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout: osd scrub max interval = 600 2023-10-14T16:39:13.375 INFO:teuthology.orchestra.run.smithi186.stdout: osd mclock profile = high_recovery_ops 2023-10-14T16:39:13.376 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.376 INFO:teuthology.orchestra.run.smithi186.stdout: osd recover clone overlap = true 2023-10-14T16:39:13.376 INFO:teuthology.orchestra.run.smithi186.stdout: osd recovery max chunk = 1048576 2023-10-14T16:39:13.376 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.376 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug shutdown = true 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug op order = true 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug verify stray on activate = true 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: osd open classes on start = true 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug pg log writeout = true 2023-10-14T16:39:13.377 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.378 INFO:teuthology.orchestra.run.smithi186.stdout: osd deep scrub update digest min age = 30 2023-10-14T16:39:13.378 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.378 INFO:teuthology.orchestra.run.smithi186.stdout: osd map max advance = 10 2023-10-14T16:39:13.378 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: journal zero on create = true 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: filestore ondisk finisher threads = 3 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: filestore apply finisher threads = 3 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.379 INFO:teuthology.orchestra.run.smithi186.stdout: bdev debug aio = true 2023-10-14T16:39:13.380 INFO:teuthology.orchestra.run.smithi186.stdout: osd debug misdirected ops = true 2023-10-14T16:39:13.380 INFO:teuthology.orchestra.run.smithi186.stdout: bdev async discard = True 2023-10-14T16:39:13.380 INFO:teuthology.orchestra.run.smithi186.stdout: bdev enable discard = True 2023-10-14T16:39:13.380 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore allocator = bitmap 2023-10-14T16:39:13.380 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore block size = 96636764160 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: bluestore fsck on mount = True 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluefs = 1/20 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: debug bluestore = 1/20 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: debug osd = 20 2023-10-14T16:39:13.381 INFO:teuthology.orchestra.run.smithi186.stdout: debug rocksdb = 4/10 2023-10-14T16:39:13.382 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd backfillfull_ratio = 0.85 2023-10-14T16:39:13.382 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd full ratio = 0.9 2023-10-14T16:39:13.382 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd nearfull ratio = 0.8 2023-10-14T16:39:13.382 INFO:teuthology.orchestra.run.smithi186.stdout: osd failsafe full ratio = 0.95 2023-10-14T16:39:13.382 INFO:teuthology.orchestra.run.smithi186.stdout: osd objectstore = bluestore 2023-10-14T16:39:13.383 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2023-10-14T16:39:13.383 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.383 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr] 2023-10-14T16:39:13.383 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2023-10-14T16:39:13.383 INFO:teuthology.orchestra.run.smithi186.stdout: debug mgr = 20 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: mgr/telemetry/nag = false 2023-10-14T16:39:13.384 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2023-10-14T16:39:13.385 INFO:teuthology.orchestra.run.smithi186.stdout: debug finisher = 20 2023-10-14T16:39:13.385 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.385 INFO:teuthology.orchestra.run.smithi186.stdout:[mon] 2023-10-14T16:39:13.385 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2023-10-14T16:39:13.385 INFO:teuthology.orchestra.run.smithi186.stdout: debug mon = 20 2023-10-14T16:39:13.386 INFO:teuthology.orchestra.run.smithi186.stdout: debug paxos = 20 2023-10-14T16:39:13.386 INFO:teuthology.orchestra.run.smithi186.stdout: debug auth = 20 2023-10-14T16:39:13.386 INFO:teuthology.orchestra.run.smithi186.stdout: mon data avail warn = 5 2023-10-14T16:39:13.386 INFO:teuthology.orchestra.run.smithi186.stdout: mon mgr mkfs grace = 240 2023-10-14T16:39:13.386 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min pgs per osd = 4 2023-10-14T16:39:13.387 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd reporter subtree level = osd 2023-10-14T16:39:13.387 INFO:teuthology.orchestra.run.smithi186.stdout: mon osd prime pg temp = true 2023-10-14T16:39:13.387 INFO:teuthology.orchestra.run.smithi186.stdout: mon reweight min bytes per osd = 10 2023-10-14T16:39:13.387 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.387 INFO:teuthology.orchestra.run.smithi186.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-10-14T16:39:13.388 INFO:teuthology.orchestra.run.smithi186.stdout: auth mon ticket ttl = 660 # 11m 2023-10-14T16:39:13.388 INFO:teuthology.orchestra.run.smithi186.stdout: auth service ticket ttl = 240 # 4m 2023-10-14T16:39:13.388 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.388 INFO:teuthology.orchestra.run.smithi186.stdout: # don't complain about insecure global_id in the test suite 2023-10-14T16:39:13.388 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-10-14T16:39:13.389 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-10-14T16:39:13.389 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.389 INFO:teuthology.orchestra.run.smithi186.stdout: # 1m isn't quite enough 2023-10-14T16:39:13.389 INFO:teuthology.orchestra.run.smithi186.stdout: mon_down_mkfs_grace = 2m 2023-10-14T16:39:13.389 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.390 INFO:teuthology.orchestra.run.smithi186.stdout: mon_warn_on_filestore_osds = false 2023-10-14T16:39:13.390 INFO:teuthology.orchestra.run.smithi186.stdout: mon op complaint time = 120 2023-10-14T16:39:13.390 INFO:teuthology.orchestra.run.smithi186.stdout: 2023-10-14T16:39:13.390 INFO:teuthology.orchestra.run.smithi186.stdout:[client] 2023-10-14T16:39:13.390 INFO:teuthology.orchestra.run.smithi186.stdout: rgw cache enabled = true 2023-10-14T16:39:13.391 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable ops log = true 2023-10-14T16:39:13.391 INFO:teuthology.orchestra.run.smithi186.stdout: rgw enable usage log = true 2023-10-14T16:39:13.391 INFO:teuthology.orchestra.run.smithi186.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-10-14T16:39:13.391 INFO:teuthology.orchestra.run.smithi186.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-10-14T16:39:13.391 INFO:teuthology.orchestra.run.smithi186.stdout: client mount timeout = 600 2023-10-14T16:39:13.392 INFO:teuthology.orchestra.run.smithi186.stdout: debug client = 20 2023-10-14T16:39:13.392 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2023-10-14T16:39:13.392 INFO:teuthology.orchestra.run.smithi186.stdout: pid file = "" 2023-10-14T16:39:13.392 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2023-10-14T16:39:13.392 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2023-10-14T16:39:13.393 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.a] 2023-10-14T16:39:13.393 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.b] 2023-10-14T16:39:13.393 INFO:teuthology.orchestra.run.smithi186.stdout:[mon.c] 2023-10-14T16:39:13.393 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout:[mds] 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds = 20 2023-10-14T16:39:13.394 INFO:teuthology.orchestra.run.smithi186.stdout: debug mds balancer = 20 2023-10-14T16:39:13.395 INFO:teuthology.orchestra.run.smithi186.stdout: debug ms = 1 2023-10-14T16:39:13.395 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug frag = True 2023-10-14T16:39:13.395 INFO:teuthology.orchestra.run.smithi186.stdout: mds debug scatterstat = True 2023-10-14T16:39:13.395 INFO:teuthology.orchestra.run.smithi186.stdout: mds op complaint time = 180 2023-10-14T16:39:13.395 INFO:teuthology.orchestra.run.smithi186.stdout: mds verify scatter = True 2023-10-14T16:39:13.396 INFO:teuthology.orchestra.run.smithi186.stdout: osd op complaint time = 180 2023-10-14T16:39:13.396 INFO:teuthology.orchestra.run.smithi186.stdout: rados mon op timeout = 900 2023-10-14T16:39:13.396 INFO:teuthology.orchestra.run.smithi186.stdout: rados osd op timeout = 900 2023-10-14T16:39:13.397 INFO:tasks.ceph:Creating admin key on mon.a... 2023-10-14T16:39:13.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-10-14T16:39:13.557 INFO:tasks.ceph:Copying monmap to all nodes... 2023-10-14T16:39:13.558 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:13.558 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-10-14T16:39:13.612 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:13.612 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-10-14T16:39:13.668 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:39:13.668 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:13.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T16:39:13.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T16:39:13.829 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:13.830 DEBUG:teuthology.orchestra.run.smithi029:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:13.933 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:39:13.933 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:13.933 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd of=/etc/ceph/ceph.keyring 2023-10-14T16:39:13.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-10-14T16:39:14.084 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:14.084 DEBUG:teuthology.orchestra.run.smithi186:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:14.182 INFO:tasks.ceph:Setting up mon nodes... 2023-10-14T16:39:14.182 INFO:tasks.ceph:Setting up mgr nodes... 2023-10-14T16:39:14.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2023-10-14T16:39:14.381 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2023-10-14T16:39:14.392 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T16:39:14.603 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-10-14T16:39:14.614 INFO:tasks.ceph:Setting up mds nodes... 2023-10-14T16:39:14.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2023-10-14T16:39:14.827 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2023-10-14T16:39:14.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2023-10-14T16:39:14.929 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2023-10-14T16:39:15.119 INFO:teuthology.orchestra.run.smithi029.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2023-10-14T16:39:15.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2023-10-14T16:39:15.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2023-10-14T16:39:15.410 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2023-10-14T16:39:15.419 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2023-10-14T16:39:15.514 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2023-10-14T16:39:15.704 INFO:teuthology.orchestra.run.smithi186.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2023-10-14T16:39:15.714 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2023-10-14T16:39:15.815 INFO:tasks.ceph_client:Setting up client nodes... 2023-10-14T16:39:15.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-14T16:39:15.959 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-10-14T16:39:16.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-14T16:39:16.156 INFO:teuthology.orchestra.run.smithi029.stdout:creating /etc/ceph/ceph.client.2.keyring 2023-10-14T16:39:16.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-14T16:39:16.343 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.1.keyring 2023-10-14T16:39:16.399 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2023-10-14T16:39:16.529 INFO:teuthology.orchestra.run.smithi186.stdout:creating /etc/ceph/ceph.client.3.keyring 2023-10-14T16:39:16.591 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-10-14T16:39:16.591 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi029.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi186.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2023-10-14T16:39:16.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-10-14T16:39:16.701 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:16.701 INFO:tasks.ceph:role: osd.0 2023-10-14T16:39:16.702 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:39:16.702 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T16:39:16.800 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:16.801 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:16.801 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:16.801 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:16.801 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:16.801 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:16.802 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:16.802 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:16.802 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:16.802 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:16.834 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-14T16:39:16.840 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-14T16:39:16.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-10-14T16:39:16.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-10-14T16:39:17.047 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-10-14T16:39:17.142 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:17.142 INFO:tasks.ceph:role: osd.1 2023-10-14T16:39:17.143 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:39:17.143 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T16:39:17.245 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:17.245 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:17.245 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:17.245 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:17.245 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:17.246 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:17.246 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:17.246 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:17.246 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:17.246 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:17.278 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-14T16:39:17.285 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-14T16:39:17.285 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-10-14T16:39:17.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-10-14T16:39:17.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-10-14T16:39:17.604 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:17.604 INFO:tasks.ceph:role: osd.2 2023-10-14T16:39:17.604 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:39:17.605 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T16:39:17.706 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:17.707 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:17.707 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:17.707 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:17.707 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:17.708 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:17.708 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:17.708 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:17.708 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:17.708 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:17.741 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-14T16:39:17.747 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-14T16:39:17.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-10-14T16:39:17.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-10-14T16:39:17.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-10-14T16:39:18.074 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:18.075 INFO:tasks.ceph:role: osd.3 2023-10-14T16:39:18.075 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:39:18.075 DEBUG:teuthology.orchestra.run.smithi029:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-14T16:39:18.176 INFO:teuthology.orchestra.run.smithi029.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:18.177 INFO:teuthology.orchestra.run.smithi029.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:18.177 INFO:teuthology.orchestra.run.smithi029.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:18.178 INFO:teuthology.orchestra.run.smithi029.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:18.178 INFO:teuthology.orchestra.run.smithi029.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:18.178 INFO:teuthology.orchestra.run.smithi029.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:18.178 INFO:teuthology.orchestra.run.smithi029.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:18.215 INFO:teuthology.orchestra.run.smithi029.stdout:Discarding blocks...Done. 2023-10-14T16:39:18.221 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi029.front.sepia.ceph.com -o noatime 2023-10-14T16:39:18.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2023-10-14T16:39:18.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-10-14T16:39:18.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:18.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.584+0000 7f743534a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-10-14T16:39:18.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.585+0000 7f743534a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-10-14T16:39:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.586+0000 7f743534a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T16:39:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.586+0000 7f743534a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T16:39:18.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.586+0000 7f743534a500 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-10-14T16:39:18.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:18.589+0000 7f743534a500 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-10-14T16:39:21.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-10-14T16:39:21.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.016+0000 7f469dc69500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-10-14T16:39:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.016+0000 7f469dc69500 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-10-14T16:39:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.016+0000 7f469dc69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T16:39:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.016+0000 7f469dc69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T16:39:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.016+0000 7f469dc69500 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-10-14T16:39:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:22.018+0000 7f469dc69500 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-10-14T16:39:24.949 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-10-14T16:39:25.057 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.207+0000 7fa83c416500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-10-14T16:39:25.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.207+0000 7fa83c416500 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-10-14T16:39:25.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.207+0000 7fa83c416500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T16:39:25.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.208+0000 7fa83c416500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T16:39:25.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.208+0000 7fa83c416500 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-10-14T16:39:25.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:25.210+0000 7fa83c416500 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-10-14T16:39:28.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-10-14T16:39:28.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.379+0000 7f86790c9500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-10-14T16:39:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.379+0000 7f86790c9500 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-10-14T16:39:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.379+0000 7f86790c9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T16:39:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.379+0000 7f86790c9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T16:39:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.379+0000 7f86790c9500 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-10-14T16:39:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:28.381+0000 7f86790c9500 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-10-14T16:39:31.568 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-10-14T16:39:31.670 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-10-14T16:39:31.774 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:31.774 INFO:tasks.ceph:role: osd.4 2023-10-14T16:39:31.775 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:39:31.775 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-10-14T16:39:31.876 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:31.876 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:31.877 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:31.877 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:31.877 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:31.877 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:31.877 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:31.878 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:31.878 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:31.878 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:32.402 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2023-10-14T16:39:32.408 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2023-10-14T16:39:32.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2023-10-14T16:39:32.589 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-10-14T16:39:32.701 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-10-14T16:39:32.783 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:32.783 INFO:tasks.ceph:role: osd.5 2023-10-14T16:39:32.784 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:39:32.784 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-10-14T16:39:32.872 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:32.872 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:32.872 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:32.872 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:32.872 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:32.873 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:32.873 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:32.873 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:32.873 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:32.874 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:33.338 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2023-10-14T16:39:33.343 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2023-10-14T16:39:33.344 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2023-10-14T16:39:33.468 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-10-14T16:39:33.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2023-10-14T16:39:33.636 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:33.637 INFO:tasks.ceph:role: osd.6 2023-10-14T16:39:33.637 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:39:33.638 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-10-14T16:39:33.725 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:33.726 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:33.726 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:33.726 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:33.726 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:33.727 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:33.727 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:33.727 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:33.727 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:33.727 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:34.248 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2023-10-14T16:39:34.253 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2023-10-14T16:39:34.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2023-10-14T16:39:34.433 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2023-10-14T16:39:34.534 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2023-10-14T16:39:34.627 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2023-10-14T16:39:34.628 INFO:tasks.ceph:role: osd.7 2023-10-14T16:39:34.628 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:39:34.629 DEBUG:teuthology.orchestra.run.smithi186:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2023-10-14T16:39:34.731 INFO:teuthology.orchestra.run.smithi186.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2023-10-14T16:39:34.731 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 attr=2, projid32bit=1 2023-10-14T16:39:34.732 INFO:teuthology.orchestra.run.smithi186.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-10-14T16:39:34.732 INFO:teuthology.orchestra.run.smithi186.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2023-10-14T16:39:34.732 INFO:teuthology.orchestra.run.smithi186.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-10-14T16:39:34.732 INFO:teuthology.orchestra.run.smithi186.stdout: = sunit=0 swidth=0 blks 2023-10-14T16:39:34.732 INFO:teuthology.orchestra.run.smithi186.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-10-14T16:39:34.733 INFO:teuthology.orchestra.run.smithi186.stdout:log =internal log bsize=4096 blocks=16384, version=2 2023-10-14T16:39:34.733 INFO:teuthology.orchestra.run.smithi186.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-10-14T16:39:34.733 INFO:teuthology.orchestra.run.smithi186.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-10-14T16:39:35.180 INFO:teuthology.orchestra.run.smithi186.stdout:Discarding blocks...Done. 2023-10-14T16:39:35.186 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi186.front.sepia.ceph.com -o noatime 2023-10-14T16:39:35.187 DEBUG:teuthology.orchestra.run.smithi186:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2023-10-14T16:39:35.322 DEBUG:teuthology.orchestra.run.smithi186:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2023-10-14T16:39:35.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:35.572 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.571+0000 7f7e1ca12500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-10-14T16:39:35.574 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.573+0000 7f7e1ca12500 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-10-14T16:39:35.575 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.573+0000 7f7e1ca12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T16:39:35.575 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.573+0000 7f7e1ca12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T16:39:35.575 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.573+0000 7f7e1ca12500 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-10-14T16:39:35.579 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:35.578+0000 7f7e1ca12500 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-10-14T16:39:38.751 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-10-14T16:39:38.849 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:38.994 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.993+0000 7f95b220d500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-10-14T16:39:38.995 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.994+0000 7f95b220d500 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-10-14T16:39:38.996 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.994+0000 7f95b220d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T16:39:38.996 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.994+0000 7f95b220d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T16:39:38.996 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.994+0000 7f95b220d500 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-10-14T16:39:38.997 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:38.996+0000 7f95b220d500 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-10-14T16:39:40.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:40.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:39:42.162 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-10-14T16:39:42.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:42.403 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.403+0000 7f7ef9c0e500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2023-10-14T16:39:42.404 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.403+0000 7f7ef9c0e500 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2023-10-14T16:39:42.404 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.403+0000 7f7ef9c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T16:39:42.404 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.403+0000 7f7ef9c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T16:39:42.404 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.403+0000 7f7ef9c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2023-10-14T16:39:42.405 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:42.406+0000 7f7ef9c0e500 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2023-10-14T16:39:45.328 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2023-10-14T16:39:45.425 DEBUG:teuthology.orchestra.run.smithi186:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:45.567 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.567+0000 7f41b361a500 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2023-10-14T16:39:45.568 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.568+0000 7f41b361a500 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2023-10-14T16:39:45.568 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.568+0000 7f41b361a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T16:39:45.569 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.568+0000 7f41b361a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T16:39:45.569 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.568+0000 7f41b361a500 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2023-10-14T16:39:45.570 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:39:45.570+0000 7f41b361a500 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2023-10-14T16:39:48.731 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2023-10-14T16:39:48.828 INFO:tasks.ceph:Reading keys from all nodes... 2023-10-14T16:39:48.828 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:48.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2023-10-14T16:39:48.930 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:48.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2023-10-14T16:39:49.025 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:49.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2023-10-14T16:39:49.120 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:49.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-10-14T16:39:49.222 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:49.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-10-14T16:39:49.327 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:49.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-10-14T16:39:49.432 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:49.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-10-14T16:39:49.540 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:49.540 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-10-14T16:39:49.647 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:49.647 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2023-10-14T16:39:49.737 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:49.738 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2023-10-14T16:39:49.879 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:49.879 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-10-14T16:39:49.975 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:49.976 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-10-14T16:39:50.073 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:50.073 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2023-10-14T16:39:50.170 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:50.170 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2023-10-14T16:39:50.267 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:50.267 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-10-14T16:39:50.330 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:39:50.330 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2023-10-14T16:39:50.386 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:50.387 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2023-10-14T16:39:50.445 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:39:50.446 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2023-10-14T16:39:50.498 INFO:tasks.ceph:Adding keys to all mons... 2023-10-14T16:39:50.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T16:39:50.507 DEBUG:teuthology.orchestra.run.smithi186:> sudo tee -a /etc/ceph/ceph.keyring 2023-10-14T16:39:50.602 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.y] 2023-10-14T16:39:50.602 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q== 2023-10-14T16:39:50.602 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.a] 2023-10-14T16:39:50.603 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA== 2023-10-14T16:39:50.603 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.c] 2023-10-14T16:39:50.603 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ== 2023-10-14T16:39:50.603 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.0] 2023-10-14T16:39:50.603 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw== 2023-10-14T16:39:50.604 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.1] 2023-10-14T16:39:50.604 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg== 2023-10-14T16:39:50.604 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.2] 2023-10-14T16:39:50.604 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ== 2023-10-14T16:39:50.604 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.3] 2023-10-14T16:39:50.605 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA== 2023-10-14T16:39:50.605 INFO:teuthology.orchestra.run.smithi029.stdout:[mgr.x] 2023-10-14T16:39:50.605 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g== 2023-10-14T16:39:50.605 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.b] 2023-10-14T16:39:50.605 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw== 2023-10-14T16:39:50.606 INFO:teuthology.orchestra.run.smithi029.stdout:[mds.d] 2023-10-14T16:39:50.606 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw== 2023-10-14T16:39:50.606 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.4] 2023-10-14T16:39:50.606 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w== 2023-10-14T16:39:50.606 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.5] 2023-10-14T16:39:50.607 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg== 2023-10-14T16:39:50.607 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.6] 2023-10-14T16:39:50.607 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg== 2023-10-14T16:39:50.607 INFO:teuthology.orchestra.run.smithi029.stdout:[osd.7] 2023-10-14T16:39:50.607 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ== 2023-10-14T16:39:50.608 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:39:50.608 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:39:50.608 INFO:teuthology.orchestra.run.smithi029.stdout:[client.2] 2023-10-14T16:39:50.608 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg== 2023-10-14T16:39:50.609 INFO:teuthology.orchestra.run.smithi029.stdout:[client.1] 2023-10-14T16:39:50.609 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A== 2023-10-14T16:39:50.609 INFO:teuthology.orchestra.run.smithi029.stdout:[client.3] 2023-10-14T16:39:50.609 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w== 2023-10-14T16:39:50.611 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.y] 2023-10-14T16:39:50.611 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q== 2023-10-14T16:39:50.611 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.a] 2023-10-14T16:39:50.611 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA== 2023-10-14T16:39:50.612 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.c] 2023-10-14T16:39:50.612 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ== 2023-10-14T16:39:50.612 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.0] 2023-10-14T16:39:50.612 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw== 2023-10-14T16:39:50.612 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.1] 2023-10-14T16:39:50.613 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg== 2023-10-14T16:39:50.613 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.2] 2023-10-14T16:39:50.613 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ== 2023-10-14T16:39:50.613 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.3] 2023-10-14T16:39:50.614 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA== 2023-10-14T16:39:50.614 INFO:teuthology.orchestra.run.smithi186.stdout:[mgr.x] 2023-10-14T16:39:50.614 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g== 2023-10-14T16:39:50.614 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.b] 2023-10-14T16:39:50.614 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw== 2023-10-14T16:39:50.615 INFO:teuthology.orchestra.run.smithi186.stdout:[mds.d] 2023-10-14T16:39:50.615 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw== 2023-10-14T16:39:50.615 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.4] 2023-10-14T16:39:50.615 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w== 2023-10-14T16:39:50.615 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.5] 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg== 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.6] 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg== 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout:[osd.7] 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ== 2023-10-14T16:39:50.616 INFO:teuthology.orchestra.run.smithi186.stdout:[client.0] 2023-10-14T16:39:50.617 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:39:50.617 INFO:teuthology.orchestra.run.smithi186.stdout:[client.2] 2023-10-14T16:39:50.617 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg== 2023-10-14T16:39:50.617 INFO:teuthology.orchestra.run.smithi186.stdout:[client.1] 2023-10-14T16:39:50.617 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A== 2023-10-14T16:39:50.618 INFO:teuthology.orchestra.run.smithi186.stdout:[client.3] 2023-10-14T16:39:50.618 INFO:teuthology.orchestra.run.smithi186.stdout: key = AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w== 2023-10-14T16:39:50.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T16:39:50.627 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T16:39:50.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:50.783 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:50.931 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:50.937 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:51.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.096 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.247 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.554 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:51.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T16:39:51.713 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-10-14T16:39:51.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:51.866 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:52.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:52.024 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2023-10-14T16:39:52.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.179 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.487 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.641 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-10-14T16:39:52.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:52.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:52.947 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:52.954 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:53.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:53.105 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:53.255 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:53.263 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-10-14T16:39:53.414 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-10-14T16:39:53.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-10-14T16:39:53.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T16:39:53.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-10-14T16:39:53.758 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-10-14T16:39:53.858 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T16:39:54.017 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-10-14T16:39:54.107 DEBUG:teuthology.orchestra.run.smithi186:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-10-14T16:39:54.202 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-10-14T16:39:54.355 DEBUG:teuthology.orchestra.run.smithi186:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-10-14T16:39:54.438 DEBUG:teuthology.orchestra.run.smithi029:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:54.447 DEBUG:teuthology.orchestra.run.smithi186:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-10-14T16:39:54.503 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-10-14T16:39:54.504 INFO:tasks.ceph.mon.a:Restarting daemon 2023-10-14T16:39:54.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-10-14T16:39:54.511 INFO:tasks.ceph.mon.a:Started 2023-10-14T16:39:54.511 INFO:tasks.ceph.mon.b:Restarting daemon 2023-10-14T16:39:54.511 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-10-14T16:39:54.517 INFO:tasks.ceph.mon.b:Started 2023-10-14T16:39:54.517 INFO:tasks.ceph.mon.c:Restarting daemon 2023-10-14T16:39:54.518 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-10-14T16:39:54.523 INFO:tasks.ceph.mon.c:Started 2023-10-14T16:39:54.524 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-10-14T16:39:54.524 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-14T16:39:54.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-14T16:39:54.530 INFO:tasks.ceph.mgr.y:Started 2023-10-14T16:39:54.531 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-10-14T16:39:54.531 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-10-14T16:39:54.536 INFO:tasks.ceph.mgr.x:Started 2023-10-14T16:39:54.536 INFO:tasks.ceph:Setting crush tunables to default 2023-10-14T16:39:54.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph osd crush tunables default 2023-10-14T16:39:54.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.717+0000 7ff024ddc640 1 Processor -- start 2023-10-14T16:39:54.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.717+0000 7ff024ddc640 1 -- start start 2023-10-14T16:39:54.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.718+0000 7ff024ddc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:39:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.718+0000 7ff024ddc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0200b3940 con 0x7ff0200b3e70 2023-10-14T16:39:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.718+0000 7ff024ddc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0200b45e0 con 0x7ff0200b3070 2023-10-14T16:39:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.718+0000 7ff024ddc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200c7880 con 0x7ff0200cfc10 2023-10-14T16:39:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.718+0000 7ff01ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff0200b3e70 0x7ff0200d1040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56542/0 (socket says 172.21.15.29:56542) 2023-10-14T16:39:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff01ed9d640 1 -- 172.21.15.29:0/1012529152 learned_addr learned my addr 172.21.15.29:0/1012529152 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:39:54.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:54.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:54.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:54.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:54.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:39:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.921+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:54.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.921+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.921+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.921+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:54.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:54.921+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:39:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:55.323+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:55.323+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:55.323+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:55.323+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:55.323+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:39:56.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:56.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:56.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:56.125+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:56.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:56.125+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:56.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:56.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:56.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:56.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:39:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 msgr2=0x7ff0200d0b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:39:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.186:3300/0 conn(0x7ff0200cfc10 0x7ff0200d0b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:39:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 >> v1:172.21.15.186:6789/0 conn(0x7ff0200b3070 legacy=0x7ff0200cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:39:57.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 >> v1:172.21.15.29:6789/0 conn(0x7ff0200b3e70 legacy=0x7ff0200d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:39:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:39:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:39:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0200c7880 con 0x7ff0000061f0 2023-10-14T16:39:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff000008d60 con 0x7ff000003710 2023-10-14T16:39:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-14T16:39:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.719+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:39:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.720+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:57.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.720+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.720+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:57.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.720+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:39:57.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.920+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.921+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-14T16:39:57.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.921+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.921+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.921+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:39:57.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.921+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.922+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.922+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.922+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:57.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:57.922+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:39:58.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.322+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.323+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-14T16:39:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.323+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.323+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.323+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:39:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.323+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.324+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.324+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.324+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:58.324+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:39:59.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:59.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.125+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-10-14T16:39:59.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.125+0000 7ff017fff640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:59.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:59.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.125+0000 7ff017fff640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:39:59.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.126+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:39:59.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.126+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-10-14T16:39:59.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.126+0000 7ff01e59c640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-10-14T16:39:59.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.126+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2023-10-14T16:39:59.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:39:59.126+0000 7ff01e59c640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:40:00.690 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:00.689+0000 7f0b23c2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:40:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 msgr2=0x7ff000005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:40:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000003710 0x7ff000005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 msgr2=0x7ff0000086b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:40:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> v2:172.21.15.29:3300/0 conn(0x7ff0000061f0 0x7ff0000086b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01ed9d640 1 -- 172.21.15.29:0/1012529152 reap_dead start 2023-10-14T16:40:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff000008270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0200b3b90 con 0x7ff000003b50 2023-10-14T16:40:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0000088b0 con 0x7ff0200b3070 2023-10-14T16:40:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.719+0000 7ff01cd99640 1 -- 172.21.15.29:0/1012529152 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0200c7880 con 0x7ff000007dc0 2023-10-14T16:40:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.720+0000 7ff01ed9d640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff000008270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.720+0000 7ff01dd9b640 1 -- 172.21.15.29:0/1012529152 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629571612 0 0) 0x7ff0000088b0 con 0x7ff0200b3070 2023-10-14T16:40:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.720+0000 7ff01dd9b640 1 -- 172.21.15.29:0/1012529152 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff00c003520 con 0x7ff0200b3070 2023-10-14T16:40:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.720+0000 7ff01ed9d640 1 -- 172.21.15.29:0/1012529152 >> v1:172.21.15.186:6789/0 conn(0x7ff0200b3070 legacy=0x7ff00000b170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.720+0000 7ff01ed9d640 1 -- 172.21.15.29:0/1012529152 >> v1:172.21.15.29:6789/0 conn(0x7ff000003b50 legacy=0x7ff0000066a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.721+0000 7ff01ed9d640 1 -- 172.21.15.29:0/1012529152 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0000108c0 con 0x7ff000007dc0 2023-10-14T16:40:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.723+0000 7ff01ed9d640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff000008270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff0200b4250 tx=0x7ff01802ba80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c6bdccda92bfed3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.724+0000 7ff01dd9b640 1 -- 172.21.15.29:0/1012529152 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01802d840 con 0x7ff000007dc0 2023-10-14T16:40:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.724+0000 7ff01dd9b640 1 -- 172.21.15.29:0/1012529152 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01802d9a0 con 0x7ff000007dc0 2023-10-14T16:40:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.724+0000 7ff01dd9b640 1 -- 172.21.15.29:0/1012529152 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff01802dd10 con 0x7ff000007dc0 2023-10-14T16:40:00.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.725+0000 7ff024ddc640 1 -- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 msgr2=0x7ff000008270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:00.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.725+0000 7ff024ddc640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff000008270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7ff0200b4250 tx=0x7ff01802ba80 comp rx=0 tx=0).stop 2023-10-14T16:40:00.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.726+0000 7ff024ddc640 1 -- 172.21.15.29:0/1012529152 shutdown_connections 2023-10-14T16:40:00.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.726+0000 7ff024ddc640 1 --2- 172.21.15.29:0/1012529152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff000008270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.726+0000 7ff024ddc640 1 -- 172.21.15.29:0/1012529152 >> 172.21.15.29:0/1012529152 conn(0x7ff0200d2d20 msgr2=0x7ff0200d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.726+0000 7ff024ddc640 1 -- 172.21.15.29:0/1012529152 shutdown_connections 2023-10-14T16:40:00.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.726+0000 7ff024ddc640 1 -- 172.21.15.29:0/1012529152 wait complete. 2023-10-14T16:40:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.729+0000 7ff024ddc640 1 Processor -- start 2023-10-14T16:40:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.730+0000 7ff024ddc640 1 -- start start 2023-10-14T16:40:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.731+0000 7ff024ddc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:00.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.731+0000 7ff024ddc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff0200c7240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.731+0000 7ff024ddc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0200b3070 0x7ff0201e6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff024ddc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f2670 con 0x7ff000003b50 2023-10-14T16:40:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff024ddc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f2950 con 0x7ff0200b3070 2023-10-14T16:40:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff024ddc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0201f2c30 con 0x7ff000007dc0 2023-10-14T16:40:00.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:00.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38266/0 (socket says 172.21.15.29:38266) 2023-10-14T16:40:00.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 -- 172.21.15.29:0/4109688746 learned_addr learned my addr 172.21.15.29:0/4109688746 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:00.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff017fff640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff0200c7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:00.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 -- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 msgr2=0x7ff0200c7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff0200c7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 -- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0200b3070 msgr2=0x7ff0201e6ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0200b3070 0x7ff0201e6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.732+0000 7ff01e59c640 1 -- 172.21.15.29:0/4109688746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0200b3940 con 0x7ff000003b50 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff01e59c640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff010007bd0 tx=0x7ff010001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:00.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010016660 con 0x7ff000003b50 2023-10-14T16:40:00.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0100167c0 con 0x7ff000003b50 2023-10-14T16:40:00.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff010016b30 con 0x7ff000003b50 2023-10-14T16:40:00.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0201f2f10 con 0x7ff000003b50 2023-10-14T16:40:00.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.733+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0201f31f0 con 0x7ff000003b50 2023-10-14T16:40:00.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.734+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7ff010016c90 con 0x7ff000003b50 2023-10-14T16:40:00.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.734+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff0100153b0 con 0x7ff000003b50 2023-10-14T16:40:00.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.734+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff02004e160 con 0x7ff000003b50 2023-10-14T16:40:00.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.738+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7ff01001a020 con 0x7ff000003b50 2023-10-14T16:40:00.761 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:00.760+0000 7f0b23c2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:40:00.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.810+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff0200ce440 con 0x7ff000003b50 2023-10-14T16:40:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.812+0000 7ff016ffd640 1 -- 172.21.15.29:0/4109688746 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7ff01001f540 con 0x7ff000003b50 2023-10-14T16:40:00.813 INFO:teuthology.orchestra.run.smithi029.stderr:adjusted tunables profile to default 2023-10-14T16:40:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 msgr2=0x7ff0200b6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff010007bd0 tx=0x7ff010001650 comp rx=0 tx=0).stop 2023-10-14T16:40:00.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 shutdown_connections 2023-10-14T16:40:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0200b3070 0x7ff0201e6ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff000007dc0 0x7ff0200c7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 --2- 172.21.15.29:0/4109688746 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff000003b50 0x7ff0200b6960 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.814+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 >> 172.21.15.29:0/4109688746 conn(0x7ff0200d2d20 msgr2=0x7ff0200c7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:00.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.815+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 shutdown_connections 2023-10-14T16:40:00.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:00.815+0000 7ff024ddc640 1 -- 172.21.15.29:0/4109688746 wait complete. 2023-10-14T16:40:00.834 INFO:tasks.ceph:check_enable_crimson: False 2023-10-14T16:40:00.834 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-10-14T16:40:00.834 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:00.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-10-14T16:40:00.929 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:00.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-10-14T16:40:01.058 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:01.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-10-14T16:40:01.191 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:01.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-10-14T16:40:01.263 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:01.262+0000 7f9c1b8da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:40:01.278 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:01.279 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-10-14T16:40:01.326 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:01.325+0000 7f9c1b8da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:40:01.370 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:01.371 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-10-14T16:40:01.453 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.451+0000 7f0b23c2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:40:01.498 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:01.498 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2023-10-14T16:40:01.522 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.520+0000 7f0b23c2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:40:01.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.578+0000 7f0b23c2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:40:01.626 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:01.626 DEBUG:teuthology.orchestra.run.smithi186:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2023-10-14T16:40:01.649 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.648+0000 7f0b23c2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:40:01.704 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 7a6bfcaa-8402-4ea5-913f-e9eab85c856a 0 2023-10-14T16:40:01.706 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.704+0000 7f0b23c2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:40:01.832 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.830+0000 7f0b23c2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:40:01.917 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.915+0000 7f0b23c2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:40:01.941 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.940+0000 7fb1f9b73640 1 Processor -- start 2023-10-14T16:40:01.942 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.940+0000 7fb1f9b73640 1 -- start start 2023-10-14T16:40:01.943 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb1f415ad70 0x7fb1f4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.944 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1f40c9ca0 0x7fb1f406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.944 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40ca3b0 0x7fb1f40ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.944 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40cad80 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.944 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1f40cb030 con 0x7fb1f415ad70 2023-10-14T16:40:01.945 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f9b73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c91c0 con 0x7fb1f40ca3b0 2023-10-14T16:40:01.945 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f37fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb1f415ad70 0x7fb1f4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.945 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f37fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb1f415ad70 0x7fb1f4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43112/0 (socket says 172.21.15.186:43112) 2023-10-14T16:40:01.945 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f37fe640 1 -- 172.21.15.186:0/3742404596 learned_addr learned my addr 172.21.15.186:0/3742404596 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:01.945 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.942+0000 7fb1f3fff640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40ca3b0 0x7fb1f40ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1ebfff640 1 --2- 172.21.15.186:0/3742404596 >> v2:172.21.15.29:3300/0 conn(0x7fb1f40c9ca0 0x7fb1f406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1f37fe640 1 -- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40ca3b0 msgr2=0x7fb1f40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40ca3b0 0x7fb1f40ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1f37fe640 1 -- 172.21.15.186:0/3742404596 >> v2:172.21.15.29:3300/0 conn(0x7fb1f40c9ca0 msgr2=0x7fb1f406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/3742404596 >> v2:172.21.15.29:3300/0 conn(0x7fb1f40c9ca0 0x7fb1f406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.946 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.943+0000 7fb1f37fe640 1 -- 172.21.15.186:0/3742404596 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f40c94a0 con 0x7fb1f415ad70 2023-10-14T16:40:01.948 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.944+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f4069da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40cf780 tx=0x7fb1e402b900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=db45fc39b250082d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:01.948 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.944+0000 7fb1f2ffd640 1 -- 172.21.15.186:0/3742404596 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4031860 con 0x7fb1f415ad70 2023-10-14T16:40:01.948 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.944+0000 7fb1f2ffd640 1 -- 172.21.15.186:0/3742404596 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e40319c0 con 0x7fb1f415ad70 2023-10-14T16:40:01.948 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.944+0000 7fb1f2ffd640 1 -- 172.21.15.186:0/3742404596 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4031d50 con 0x7fb1f415ad70 2023-10-14T16:40:01.949 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 -- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 msgr2=0x7fb1f4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:01.949 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f4069da0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb1f40cf780 tx=0x7fb1e402b900 comp rx=0 tx=0).stop 2023-10-14T16:40:01.949 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 -- 172.21.15.186:0/3742404596 shutdown_connections 2023-10-14T16:40:01.949 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40ca3b0 0x7fb1f40ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.949 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/3742404596 >> v2:172.21.15.29:3300/0 conn(0x7fb1f40c9ca0 0x7fb1f406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.950 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/3742404596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f4069da0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.950 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 -- 172.21.15.186:0/3742404596 >> 172.21.15.186:0/3742404596 conn(0x7fb1f40ce9b0 msgr2=0x7fb1f40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:01.950 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 -- 172.21.15.186:0/3742404596 shutdown_connections 2023-10-14T16:40:01.950 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.945+0000 7fb1f9b73640 1 -- 172.21.15.186:0/3742404596 wait complete. 2023-10-14T16:40:01.950 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.947+0000 7fb1f9b73640 1 Processor -- start 2023-10-14T16:40:01.951 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.947+0000 7fb1f9b73640 1 -- start start 2023-10-14T16:40:01.952 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.953 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1f40ca3b0 0x7fb1f40aa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.953 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f40aa8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:01.953 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f41585b0 con 0x7fb1f40ca3b0 2023-10-14T16:40:01.953 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c69e0 con 0x7fb1f415ad70 2023-10-14T16:40:01.953 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f9b73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1f40c6cc0 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.954 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.954 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41898/0 (socket says 172.21.15.186:41898) 2023-10-14T16:40:01.954 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 -- 172.21.15.186:0/2696199879 learned_addr learned my addr 172.21.15.186:0/2696199879 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:01.954 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f3fff640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f40aa8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.954 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 -- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 msgr2=0x7fb1f40aa8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:01.955 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1ebfff640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1f40ca3b0 0x7fb1f40aa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:01.955 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f40aa8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.955 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 -- 172.21.15.186:0/2696199879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1f40ca3b0 msgr2=0x7fb1f40aa3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:01.955 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1f40ca3b0 0x7fb1f40aa3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:01.955 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.948+0000 7fb1f37fe640 1 -- 172.21.15.186:0/2696199879 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1e40094d0 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.956 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f37fe640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1e402b940 tx=0x7fb1e40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:01.956 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e403ac60 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.956 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1e403ade0 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.956 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1e4044630 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.956 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1f40c6fa0 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.957 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1f40ac420 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.957 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fb1e4044a90 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.957 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.949+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb1e404da40 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.957 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.950+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1f404e160 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.958 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:01.953+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fb1e4035020 con 0x7fb1f40c9ca0 2023-10-14T16:40:01.968 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:01.967+0000 7f9c1b8da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:40:01.973 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:01.971+0000 7f0b23c2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:40:02.031 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.030+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "7a6bfcaa-8402-4ea5-913f-e9eab85c856a", "id": 0} v 0) v1 -- 0x7fb1f406fcc0 con 0x7fb1f40c9ca0 2023-10-14T16:40:02.032 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.031+0000 7f9c1b8da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:40:02.036 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.035+0000 7f0b23c2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:40:02.037 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.036+0000 7fb1f0ff9640 1 -- 172.21.15.186:0/2696199879 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a6bfcaa-8402-4ea5-913f-e9eab85c856a", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb1e404d450 con 0x7fb1f40c9ca0 2023-10-14T16:40:02.037 INFO:teuthology.orchestra.run.smithi186.stdout:0 2023-10-14T16:40:02.038 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 msgr2=0x7fb1f4158070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.039 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fb1e402b940 tx=0x7fb1e40046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.039 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 shutdown_connections 2023-10-14T16:40:02.039 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1f415ad70 0x7fb1f40aa8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.039 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1f40ca3b0 0x7fb1f40aa3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.039 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 --2- 172.21.15.186:0/2696199879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1f40c9ca0 0x7fb1f4158070 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.040 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.037+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 >> 172.21.15.186:0/2696199879 conn(0x7fb1f40ce9b0 msgr2=0x7fb1f406b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.040 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.038+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 shutdown_connections 2023-10-14T16:40:02.040 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.038+0000 7fb1f9b73640 1 -- 172.21.15.186:0/2696199879 wait complete. 2023-10-14T16:40:02.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 38ce4325-d8a0-4af1-aa17-9e7cf06d521f 1 2023-10-14T16:40:02.088 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.087+0000 7f9c1b8da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:40:02.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.093+0000 7f0b23c2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:40:02.164 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.163+0000 7f9c1b8da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:40:02.217 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.216+0000 7f9c1b8da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:40:02.247 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.246+0000 7f2ad9941640 1 Processor -- start 2023-10-14T16:40:02.248 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.246+0000 7f2ad9941640 1 -- start start 2023-10-14T16:40:02.249 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad9941640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ad4159ab0 0x7f2ad40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.249 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad9941640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad4070360 con 0x7f2ad4159e90 2023-10-14T16:40:02.249 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad9941640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad40cd1f0 con 0x7f2ad406c140 2023-10-14T16:40:02.249 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad9941640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad415d890 con 0x7f2ad4159ab0 2023-10-14T16:40:02.250 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2ad406c140 0x7f2ad415dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40650/0 (socket says 172.21.15.186:40650) 2023-10-14T16:40:02.250 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad37fe640 1 -- 172.21.15.186:0/3621474569 learned_addr learned my addr 172.21.15.186:0/3621474569 (peer_addr_for_me v1:172.21.15.186:0/0) 2023-10-14T16:40:02.254 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad2ffd640 1 --2- 172.21.15.186:0/3621474569 >> v2:172.21.15.29:3300/0 conn(0x7f2ad4159ab0 0x7f2ad40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.254 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.248+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258451815 0 0) 0x7f2ad40cd1f0 con 0x7f2ad406c140 2023-10-14T16:40:02.254 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2abc005730 con 0x7f2ad406c140 2023-10-14T16:40:02.255 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4106764232 0 0) 0x7f2abc005730 con 0x7f2ad406c140 2023-10-14T16:40:02.255 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 >> v2:172.21.15.29:3300/0 conn(0x7f2ad4159ab0 msgr2=0x7f2ad40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.255 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 --2- 172.21.15.186:0/3621474569 >> v2:172.21.15.29:3300/0 conn(0x7f2ad4159ab0 0x7f2ad40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.255 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 >> v1:172.21.15.29:6789/0 conn(0x7f2ad4159e90 legacy=0x7f2ad406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.255 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad40cd4d0 con 0x7f2ad406c140 2023-10-14T16:40:02.256 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2acc0019f0 con 0x7f2ad406c140 2023-10-14T16:40:02.256 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 >> v1:172.21.15.186:6789/0 conn(0x7f2ad406c140 legacy=0x7f2ad415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.256 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2abc005bd0 0x7f2abc005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.256 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 0x7f2abc006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.256 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0074b0 0x7f2abc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.257 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad415d890 con 0x7f2abc005bd0 2023-10-14T16:40:02.257 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc00e3b0 con 0x7f2abc0074b0 2023-10-14T16:40:02.257 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2abc00e570 con 0x7f2abc0069d0 2023-10-14T16:40:02.257 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad37fe640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0074b0 0x7f2abc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.257 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.249+0000 7f2ad27fc640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 0x7f2abc006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.258 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad2ffd640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2abc005bd0 0x7f2abc005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.258 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad27fc640 1 -- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0074b0 msgr2=0x7f2abc00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.258 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad27fc640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0074b0 0x7f2abc00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.258 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad27fc640 1 -- 172.21.15.186:0/3621474569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2abc005bd0 msgr2=0x7f2abc005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.258 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad27fc640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2abc005bd0 0x7f2abc005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.259 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad27fc640 1 -- 172.21.15.186:0/3621474569 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2abc005730 con 0x7f2abc0069d0 2023-10-14T16:40:02.259 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.250+0000 7f2ad37fe640 1 -- 172.21.15.186:0/3621474569 reap_dead start 2023-10-14T16:40:02.259 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.251+0000 7f2ad27fc640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 0x7f2abc006e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8001750 tx=0x7f2abc007a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d2c10332b0712cc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.259 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.251+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac80052d0 con 0x7f2abc0069d0 2023-10-14T16:40:02.259 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.251+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac802e050 con 0x7f2abc0069d0 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.251+0000 7f2ad1ffb640 1 -- 172.21.15.186:0/3621474569 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac8003830 con 0x7f2abc0069d0 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 -- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 msgr2=0x7f2abc006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 0x7f2abc006e50 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2ac8001750 tx=0x7f2abc007a00 comp rx=0 tx=0).stop 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 -- 172.21.15.186:0/3621474569 shutdown_connections 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3621474569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2abc0069d0 0x7f2abc006e50 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.260 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 -- 172.21.15.186:0/3621474569 >> 172.21.15.186:0/3621474569 conn(0x7f2ad40ce9b0 msgr2=0x7f2ad40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.261 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 -- 172.21.15.186:0/3621474569 shutdown_connections 2023-10-14T16:40:02.261 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.252+0000 7f2ad9941640 1 -- 172.21.15.186:0/3621474569 wait complete. 2023-10-14T16:40:02.263 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.255+0000 7f2ad9941640 1 Processor -- start 2023-10-14T16:40:02.264 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.255+0000 7f2ad9941640 1 -- start start 2023-10-14T16:40:02.264 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.256+0000 7f2ad9941640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.264 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.256+0000 7f2ad9941640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 0x7f2ad40b2c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.264 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad9941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad40c3a70 0x7f2ad40c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.264 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad9941640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad40c4430 con 0x7f2ad40c3a70 2023-10-14T16:40:02.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad9941640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad40b2360 con 0x7f2abc0069d0 2023-10-14T16:40:02.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad9941640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ad40c5330 con 0x7f2ad40b27f0 2023-10-14T16:40:02.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad27fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 0x7f2ad40b2c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43128/0 (socket says 172.21.15.186:43128) 2023-10-14T16:40:02.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad27fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 0x7f2ad40b2c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41918/0 (socket says 172.21.15.186:41918) 2023-10-14T16:40:02.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 -- 172.21.15.186:0/3093834470 learned_addr learned my addr 172.21.15.186:0/3093834470 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:02.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad37fe640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad40c3a70 0x7f2ad40c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 -- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 msgr2=0x7f2ad40b2c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 0x7f2ad40b2c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 -- 172.21.15.186:0/3093834470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad40c3a70 msgr2=0x7f2ad40c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad40c3a70 0x7f2ad40c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.257+0000 7f2ad2ffd640 1 -- 172.21.15.186:0/3093834470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad41f32b0 con 0x7f2abc0069d0 2023-10-14T16:40:02.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2ad2ffd640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4011510 tx=0x7f2ac40119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4015870 con 0x7f2abc0069d0 2023-10-14T16:40:02.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ac40159d0 con 0x7f2abc0069d0 2023-10-14T16:40:02.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ac4015d40 con 0x7f2abc0069d0 2023-10-14T16:40:02.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad40b1a30 con 0x7f2abc0069d0 2023-10-14T16:40:02.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad40b1ff0 con 0x7f2abc0069d0 2023-10-14T16:40:02.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f2ac401d400 con 0x7f2abc0069d0 2023-10-14T16:40:02.269 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.258+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (secure 0 0 0) 0x7f2ac4013890 con 0x7f2abc0069d0 2023-10-14T16:40:02.269 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.259+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad40b21b0 con 0x7f2abc0069d0 2023-10-14T16:40:02.269 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.262+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f2ac403cb40 con 0x7f2abc0069d0 2023-10-14T16:40:02.340 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.339+0000 7f9c1b8da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:40:02.340 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.339+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "38ce4325-d8a0-4af1-aa17-9e7cf06d521f", "id": 1} v 0) v1 -- 0x7f2ad404e160 con 0x7f2abc0069d0 2023-10-14T16:40:02.346 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.345+0000 7f2abbfff640 1 -- 172.21.15.186:0/3093834470 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "38ce4325-d8a0-4af1-aa17-9e7cf06d521f", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f2ac402b070 con 0x7f2abc0069d0 2023-10-14T16:40:02.346 INFO:teuthology.orchestra.run.smithi186.stdout:1 2023-10-14T16:40:02.348 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.346+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 msgr2=0x7f2ad40c4df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.348 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.346+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f2ac4011510 tx=0x7f2ac40119d0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.348 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 shutdown_connections 2023-10-14T16:40:02.348 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ad40c3a70 0x7f2ad40c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.349 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ad40b27f0 0x7f2ad40b2c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.349 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 --2- 172.21.15.186:0/3093834470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2abc0069d0 0x7f2ad40c4df0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.349 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 >> 172.21.15.186:0/3093834470 conn(0x7f2ad40ce9b0 msgr2=0x7f2ad40c1c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.349 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 shutdown_connections 2023-10-14T16:40:02.349 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.347+0000 7f2ad9941640 1 -- 172.21.15.186:0/3093834470 wait complete. 2023-10-14T16:40:02.364 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new fdb2602a-e449-419e-9eb0-3aa215a44dd6 2 2023-10-14T16:40:02.424 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.423+0000 7f9c1b8da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:40:02.481 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.480+0000 7f9c1b8da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:40:02.545 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.544+0000 7f9c1b8da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:40:02.569 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.567+0000 7f0b23c2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:40:02.602 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:02.601+0000 7f9c1b8da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:40:02.606 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.605+0000 7f1acad5a640 1 Processor -- start 2023-10-14T16:40:02.607 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.605+0000 7f1acad5a640 1 -- start start 2023-10-14T16:40:02.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1acad5a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ac40cf710 0x7f1ac40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1acad5a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ac40b6cd0 0x7f1ac40b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1acad5a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ac40d3980 con 0x7f1ac40cfaf0 2023-10-14T16:40:02.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1acad5a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40d3c60 con 0x7f1ac40b6cd0 2023-10-14T16:40:02.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1acad5a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ac40ce630 con 0x7f1ac40cf710 2023-10-14T16:40:02.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1ac8af6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ac40cf710 0x7f1ac40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1ac8af6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ac40cf710 0x7f1ac40b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43132/0 (socket says 172.21.15.186:43132) 2023-10-14T16:40:02.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.607+0000 7f1ac8af6640 1 -- 172.21.15.186:0/1124266265 learned_addr learned my addr 172.21.15.186:0/1124266265 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:02.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.608+0000 7f1ac92f7640 1 --2- 172.21.15.186:0/1124266265 >> v2:172.21.15.29:3300/0 conn(0x7f1ac40b6cd0 0x7f1ac40b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.611 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.608+0000 7f1ac8af6640 1 -- 172.21.15.186:0/1124266265 >> v2:172.21.15.29:3300/0 conn(0x7f1ac40b6cd0 msgr2=0x7f1ac40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.611 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.608+0000 7f1ac8af6640 1 --2- 172.21.15.186:0/1124266265 >> v2:172.21.15.29:3300/0 conn(0x7f1ac40b6cd0 0x7f1ac40b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.611 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.608+0000 7f1ac8af6640 1 -- 172.21.15.186:0/1124266265 >> v1:172.21.15.29:6789/0 conn(0x7f1ac40cfaf0 legacy=0x7f1ac40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.611 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.608+0000 7f1ac8af6640 1 -- 172.21.15.186:0/1124266265 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac40ce910 con 0x7f1ac40cf710 2023-10-14T16:40:02.612 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.609+0000 7f1ac8af6640 1 --2- 172.21.15.186:0/1124266265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac40b2d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40d2c20 tx=0x7f1ab802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ddcea466dbe8ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.609+0000 7f1ac37fe640 1 -- 172.21.15.186:0/1124266265 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8004230 con 0x7f1ac40cf710 2023-10-14T16:40:02.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.609+0000 7f1ac37fe640 1 -- 172.21.15.186:0/1124266265 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab8004390 con 0x7f1ac40cf710 2023-10-14T16:40:02.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.609+0000 7f1ac37fe640 1 -- 172.21.15.186:0/1124266265 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab8003810 con 0x7f1ac40cf710 2023-10-14T16:40:02.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 -- 172.21.15.186:0/1124266265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 msgr2=0x7f1ac40b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 --2- 172.21.15.186:0/1124266265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac40b2d10 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1ac40d2c20 tx=0x7f1ab802b860 comp rx=0 tx=0).stop 2023-10-14T16:40:02.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 -- 172.21.15.186:0/1124266265 shutdown_connections 2023-10-14T16:40:02.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 --2- 172.21.15.186:0/1124266265 >> v2:172.21.15.29:3300/0 conn(0x7f1ac40b6cd0 0x7f1ac40b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 --2- 172.21.15.186:0/1124266265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac40b2d10 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 -- 172.21.15.186:0/1124266265 >> 172.21.15.186:0/1124266265 conn(0x7f1ac40d2cc0 msgr2=0x7f1ac40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 -- 172.21.15.186:0/1124266265 shutdown_connections 2023-10-14T16:40:02.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.610+0000 7f1acad5a640 1 -- 172.21.15.186:0/1124266265 wait complete. 2023-10-14T16:40:02.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.612+0000 7f1acad5a640 1 Processor -- start 2023-10-14T16:40:02.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.612+0000 7f1acad5a640 1 -- start start 2023-10-14T16:40:02.618 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 0x7f1ac40c9d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.618 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.618 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac40cfaf0 0x7f1ac40b0530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.618 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4154fe0 con 0x7f1ac40cfaf0 2023-10-14T16:40:02.619 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac4154dd0 con 0x7f1ac40cf710 2023-10-14T16:40:02.619 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1acad5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ac40ca420 con 0x7f1ac40b6cd0 2023-10-14T16:40:02.619 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1ac3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.619 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1ac8af6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 0x7f1ac40c9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.619 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1ac3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43146/0 (socket says 172.21.15.186:43146) 2023-10-14T16:40:02.620 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.613+0000 7f1ac3fff640 1 -- 172.21.15.186:0/234278343 learned_addr learned my addr 172.21.15.186:0/234278343 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:02.620 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac92f7640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac40cfaf0 0x7f1ac40b0530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.620 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 -- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 msgr2=0x7f1ac40c9d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.620 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 0x7f1ac40c9d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.621 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 -- 172.21.15.186:0/234278343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac40cfaf0 msgr2=0x7f1ac40b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.621 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac40cfaf0 0x7f1ac40b0530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.621 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 -- 172.21.15.186:0/234278343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab8009460 con 0x7f1ac40cf710 2023-10-14T16:40:02.621 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac8af6640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 0x7f1ac40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:02.621 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac3fff640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c430 tx=0x7f1ab000c8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.622 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011790 con 0x7f1ac40cf710 2023-10-14T16:40:02.622 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab00118f0 con 0x7f1ac40cf710 2023-10-14T16:40:02.622 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac40ca700 con 0x7f1ac40cf710 2023-10-14T16:40:02.622 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.614+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab0011ca0 con 0x7f1ac40cf710 2023-10-14T16:40:02.622 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.615+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ac40cacc0 con 0x7f1ac40cf710 2023-10-14T16:40:02.623 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.615+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1ab001c760 con 0x7f1ac40cf710 2023-10-14T16:40:02.623 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.615+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (secure 0 0 0) 0x7f1ab0011e00 con 0x7f1ac40cf710 2023-10-14T16:40:02.623 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.616+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ac4155120 con 0x7f1ac40cf710 2023-10-14T16:40:02.624 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.618+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f1ac4155120 con 0x7f1ac40cf710 2023-10-14T16:40:02.697 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.696+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fdb2602a-e449-419e-9eb0-3aa215a44dd6", "id": 2} v 0) v1 -- 0x7f1ac40cae50 con 0x7f1ac40cf710 2023-10-14T16:40:02.706 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.705+0000 7f1ac17fa640 1 -- 172.21.15.186:0/234278343 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fdb2602a-e449-419e-9eb0-3aa215a44dd6", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1ab001ca10 con 0x7f1ac40cf710 2023-10-14T16:40:02.707 INFO:teuthology.orchestra.run.smithi186.stdout:2 2023-10-14T16:40:02.708 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.706+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 msgr2=0x7f1ac4154890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.708 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.706+0000 7f1acad5a640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1ab000c430 tx=0x7f1ab000c8f0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.708 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 shutdown_connections 2023-10-14T16:40:02.708 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ac40cfaf0 0x7f1ac40b0530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.708 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ac40cf710 0x7f1ac4154890 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.709 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 --2- 172.21.15.186:0/234278343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ac40b6cd0 0x7f1ac40c9d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.709 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 >> 172.21.15.186:0/234278343 conn(0x7f1ac40d2cc0 msgr2=0x7f1ac40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.709 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 shutdown_connections 2023-10-14T16:40:02.709 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.707+0000 7f1acad5a640 1 -- 172.21.15.186:0/234278343 wait complete. 2023-10-14T16:40:02.723 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new e78f6d2c-1521-4d00-9d3b-8988f5aa90ac 3 2023-10-14T16:40:02.738 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.737+0000 7f0b23c2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:40:02.882 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.880+0000 7f0b23c2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:40:02.937 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:02.936+0000 7f0b23c2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:40:02.957 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.956+0000 7fe9dca2a640 1 Processor -- start 2023-10-14T16:40:02.958 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.957+0000 7fe9dca2a640 1 -- start start 2023-10-14T16:40:02.959 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9dca2a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9d815cd70 0x7fe9d80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.959 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9dca2a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 0x7fe9d80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.959 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9dca2a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9d815d150 con 0x7fe9d80b8480 2023-10-14T16:40:02.960 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9dca2a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80d1180 con 0x7fe9d80cded0 2023-10-14T16:40:02.960 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9dca2a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9d80d1460 con 0x7fe9d815cd70 2023-10-14T16:40:02.960 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9d6d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9d815cd70 0x7fe9d80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.960 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9d6d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9d815cd70 0x7fe9d80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43154/0 (socket says 172.21.15.186:43154) 2023-10-14T16:40:02.960 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9d6d9d640 1 -- 172.21.15.186:0/2048400672 learned_addr learned my addr 172.21.15.186:0/2048400672 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:02.962 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.958+0000 7fe9d759e640 1 --2- 172.21.15.186:0/2048400672 >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 0x7fe9d80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.962 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.959+0000 7fe9d6d9d640 1 -- 172.21.15.186:0/2048400672 >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 msgr2=0x7fe9d80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:02.962 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.959+0000 7fe9d6d9d640 1 --2- 172.21.15.186:0/2048400672 >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 0x7fe9d80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.963 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.959+0000 7fe9d6d9d640 1 -- 172.21.15.186:0/2048400672 >> v1:172.21.15.186:6789/0 conn(0x7fe9d80b8480 legacy=0x7fe9d80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.963 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.959+0000 7fe9d6d9d640 1 -- 172.21.15.186:0/2048400672 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d80d1740 con 0x7fe9d815cd70 2023-10-14T16:40:02.963 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.959+0000 7fe9d759e640 1 --2- 172.21.15.186:0/2048400672 >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 0x7fe9d80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:02.963 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.960+0000 7fe9d6d9d640 1 --2- 172.21.15.186:0/2048400672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d815cd70 0x7fe9d80d2220 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80b56a0 tx=0x7fe9c802b9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d49b1b48d44bc7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.963 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.960+0000 7fe9d5d9b640 1 -- 172.21.15.186:0/2048400672 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8030870 con 0x7fe9d815cd70 2023-10-14T16:40:02.964 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.960+0000 7fe9d5d9b640 1 -- 172.21.15.186:0/2048400672 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9c80309d0 con 0x7fe9d815cd70 2023-10-14T16:40:02.964 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.960+0000 7fe9d5d9b640 1 -- 172.21.15.186:0/2048400672 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9c8030d60 con 0x7fe9d815cd70 2023-10-14T16:40:02.964 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 -- 172.21.15.186:0/2048400672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d815cd70 msgr2=0x7fe9d80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.964 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/2048400672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d815cd70 0x7fe9d80d2220 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe9d80b56a0 tx=0x7fe9c802b9b0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.964 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 -- 172.21.15.186:0/2048400672 shutdown_connections 2023-10-14T16:40:02.965 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/2048400672 >> v2:172.21.15.29:3300/0 conn(0x7fe9d80cded0 0x7fe9d80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.965 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/2048400672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d815cd70 0x7fe9d80d2220 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.965 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 -- 172.21.15.186:0/2048400672 >> 172.21.15.186:0/2048400672 conn(0x7fe9d80cf570 msgr2=0x7fe9d80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:02.965 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 -- 172.21.15.186:0/2048400672 shutdown_connections 2023-10-14T16:40:02.965 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.961+0000 7fe9dca2a640 1 -- 172.21.15.186:0/2048400672 wait complete. 2023-10-14T16:40:02.967 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.962+0000 7fe9dca2a640 1 Processor -- start 2023-10-14T16:40:02.968 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.963+0000 7fe9dca2a640 1 -- start start 2023-10-14T16:40:02.968 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.963+0000 7fe9dca2a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d80b8480 0x7fe9d8156550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.968 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9dca2a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.968 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9dca2a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:02.968 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9dca2a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8156a90 con 0x7fe9d80b8480 2023-10-14T16:40:02.969 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9dca2a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8158990 con 0x7fe9d80cded0 2023-10-14T16:40:02.969 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9dca2a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9d8158c10 con 0x7fe9d815cd70 2023-10-14T16:40:02.969 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.969 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.969 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41926/0 (socket says 172.21.15.186:41926) 2023-10-14T16:40:02.970 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43164/0 (socket says 172.21.15.186:43164) 2023-10-14T16:40:02.970 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d759e640 1 -- 172.21.15.186:0/3091139580 learned_addr learned my addr 172.21.15.186:0/3091139580 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:02.970 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d6d9d640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d80b8480 0x7fe9d8156550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:02.970 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 -- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 msgr2=0x7fe9d8148020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.970 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.971 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 -- 172.21.15.186:0/3091139580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d80b8480 msgr2=0x7fe9d8156550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:02.971 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d80b8480 0x7fe9d8156550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:02.971 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d659c640 1 -- 172.21.15.186:0/3091139580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9c80094d0 con 0x7fe9d80cded0 2023-10-14T16:40:02.971 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.964+0000 7fe9d759e640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:02.971 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9d659c640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c390 tx=0x7fe9cc00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc0052d0 con 0x7fe9d80cded0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9cc005430 con 0x7fe9d80cded0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9d8158ef0 con 0x7fe9d80cded0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9cc003830 con 0x7fe9d80cded0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9d81594b0 con 0x7fe9d80cded0 2023-10-14T16:40:02.972 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fe9cc003c90 con 0x7fe9d80cded0 2023-10-14T16:40:02.973 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.965+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (secure 0 0 0) 0x7fe9cc0128b0 con 0x7fe9d80cded0 2023-10-14T16:40:02.973 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.966+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9d804e9b0 con 0x7fe9d80cded0 2023-10-14T16:40:02.973 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:02.970+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fe9cc017070 con 0x7fe9d80cded0 2023-10-14T16:40:03.049 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.047+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "e78f6d2c-1521-4d00-9d3b-8988f5aa90ac", "id": 3} v 0) v1 -- 0x7fe9d80b44a0 con 0x7fe9d80cded0 2023-10-14T16:40:03.051 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:03.050+0000 7f0b23c2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:40:03.055 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.054+0000 7fe9bf7fe640 1 -- 172.21.15.186:0/3091139580 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e78f6d2c-1521-4d00-9d3b-8988f5aa90ac", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe9cc012480 con 0x7fe9d80cded0 2023-10-14T16:40:03.055 INFO:teuthology.orchestra.run.smithi186.stdout:3 2023-10-14T16:40:03.056 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 msgr2=0x7fe9d8144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.056 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fe9cc00c390 tx=0x7fe9cc00c850 comp rx=0 tx=0).stop 2023-10-14T16:40:03.056 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 shutdown_connections 2023-10-14T16:40:03.057 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9d815cd70 0x7fe9d8148020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.057 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9d80cded0 0x7fe9d8144e10 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.057 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 --2- 172.21.15.186:0/3091139580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9d80b8480 0x7fe9d8156550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.057 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 >> 172.21.15.186:0/3091139580 conn(0x7fe9d80cf570 msgr2=0x7fe9d80b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.057 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 shutdown_connections 2023-10-14T16:40:03.058 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.055+0000 7fe9dca2a640 1 -- 172.21.15.186:0/3091139580 wait complete. 2023-10-14T16:40:03.072 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new c1c0fbc8-08a3-46f0-8377-2acd51c3e048 4 2023-10-14T16:40:03.091 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.090+0000 7f9c1b8da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:40:03.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:03.168+0000 7f0b23c2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:40:03.259 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.258+0000 7f9c1b8da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:40:03.265 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.264+0000 7f1b0131a640 1 Processor -- start 2023-10-14T16:40:03.266 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.265+0000 7f1b0131a640 1 -- start start 2023-10-14T16:40:03.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1b0131a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 0x7f1afc0abd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.267 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1b0131a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1afc0710e0 0x7f1afc071540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1b0131a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1afc071a80 con 0x7f1afc06d5a0 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1b0131a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1afc159fb0 con 0x7f1afc0710e0 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1b0131a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc0ae890 con 0x7f1afc0b4530 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1afbfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f1afc06d5a0 0x7f1afc0ae180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40672/0 (socket says 172.21.15.186:40672) 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1afbfff640 1 -- 172.21.15.186:0/2230655233 learned_addr learned my addr 172.21.15.186:0/2230655233 (peer_addr_for_me v1:172.21.15.186:0/0) 2023-10-14T16:40:03.268 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.266+0000 7f1afb7fe640 1 --2- 172.21.15.186:0/2230655233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 0x7f1afc0abd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.270 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1b00b19640 1 --2- 172.21.15.186:0/2230655233 >> v2:172.21.15.29:3300/0 conn(0x7f1afc0710e0 0x7f1afc071540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.270 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1afb7fe640 1 -- 172.21.15.186:0/2230655233 >> v2:172.21.15.29:3300/0 conn(0x7f1afc0710e0 msgr2=0x7f1afc071540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:03.270 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1afb7fe640 1 --2- 172.21.15.186:0/2230655233 >> v2:172.21.15.29:3300/0 conn(0x7f1afc0710e0 0x7f1afc071540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.271 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1afb7fe640 1 -- 172.21.15.186:0/2230655233 >> v1:172.21.15.186:6789/0 conn(0x7f1afc06d5a0 legacy=0x7f1afc0ae180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.271 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1afb7fe640 1 -- 172.21.15.186:0/2230655233 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1afc0b4d80 con 0x7f1afc0b4530 2023-10-14T16:40:03.271 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.267+0000 7f1afaffd640 1 -- 172.21.15.186:0/2230655233 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873070639 0 0) 0x7f1afc071a80 con 0x7f1afc06d5a0 2023-10-14T16:40:03.271 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.268+0000 7f1afb7fe640 1 --2- 172.21.15.186:0/2230655233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 0x7f1afc0abd40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61cecc4e1854a51e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.271 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.268+0000 7f1afaffd640 1 -- 172.21.15.186:0/2230655233 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031830 con 0x7f1afc0b4530 2023-10-14T16:40:03.272 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.268+0000 7f1afaffd640 1 -- 172.21.15.186:0/2230655233 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1af0031990 con 0x7f1afc0b4530 2023-10-14T16:40:03.272 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.268+0000 7f1afaffd640 1 -- 172.21.15.186:0/2230655233 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1af0031d20 con 0x7f1afc0b4530 2023-10-14T16:40:03.272 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 -- 172.21.15.186:0/2230655233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 msgr2=0x7f1afc0abd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.272 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 --2- 172.21.15.186:0/2230655233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 0x7f1afc0abd40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1af00097a0 tx=0x7f1af002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.272 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 -- 172.21.15.186:0/2230655233 shutdown_connections 2023-10-14T16:40:03.273 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 --2- 172.21.15.186:0/2230655233 >> v2:172.21.15.29:3300/0 conn(0x7f1afc0710e0 0x7f1afc071540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.273 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 --2- 172.21.15.186:0/2230655233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0b4530 0x7f1afc0abd40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.273 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 -- 172.21.15.186:0/2230655233 >> 172.21.15.186:0/2230655233 conn(0x7f1afc0b0a80 msgr2=0x7f1afc0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.273 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 -- 172.21.15.186:0/2230655233 shutdown_connections 2023-10-14T16:40:03.273 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.269+0000 7f1b0131a640 1 -- 172.21.15.186:0/2230655233 wait complete. 2023-10-14T16:40:03.276 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.271+0000 7f1b0131a640 1 Processor -- start 2023-10-14T16:40:03.276 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.271+0000 7f1b0131a640 1 -- start start 2023-10-14T16:40:03.276 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 0x7f1afc15bef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.276 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.277 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1afc0b4530 0x7f1afc15c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.277 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc15af50 con 0x7f1afc0b4530 2023-10-14T16:40:03.277 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc15b130 con 0x7f1afc06d5a0 2023-10-14T16:40:03.277 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b0131a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1afc15b410 con 0x7f1afc0710e0 2023-10-14T16:40:03.277 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 0x7f1afc15bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.278 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.278 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41942/0 (socket says 172.21.15.186:41942) 2023-10-14T16:40:03.278 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afb7fe640 1 -- 172.21.15.186:0/1808762673 learned_addr learned my addr 172.21.15.186:0/1808762673 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:03.278 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1b00b19640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1afc0b4530 0x7f1afc15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.278 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afbfff640 1 -- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 msgr2=0x7f1afc15c430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.279 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afbfff640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.279 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afbfff640 1 -- 172.21.15.186:0/1808762673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1afc0b4530 msgr2=0x7f1afc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.279 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.272+0000 7f1afbfff640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1afc0b4530 0x7f1afc15c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.279 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1afbfff640 1 -- 172.21.15.186:0/1808762673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1af0009400 con 0x7f1afc06d5a0 2023-10-14T16:40:03.279 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1afb7fe640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1afbfff640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 0x7f1afc15bef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c450 tx=0x7f1aec00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec0074e0 con 0x7f1afc06d5a0 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aec007640 con 0x7f1afc06d5a0 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1afc15b6f0 con 0x7f1afc06d5a0 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1aec004a20 con 0x7f1afc06d5a0 2023-10-14T16:40:03.280 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1afc15ceb0 con 0x7f1afc06d5a0 2023-10-14T16:40:03.281 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.273+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f1aec012400 con 0x7f1afc06d5a0 2023-10-14T16:40:03.281 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.274+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (secure 0 0 0) 0x7f1aec012bc0 con 0x7f1afc06d5a0 2023-10-14T16:40:03.281 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.274+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1afc15bc10 con 0x7f1afc06d5a0 2023-10-14T16:40:03.282 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.277+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f1afc15bc10 con 0x7f1afc06d5a0 2023-10-14T16:40:03.336 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:03.334+0000 7f0b23c2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:40:03.355 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.354+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "c1c0fbc8-08a3-46f0-8377-2acd51c3e048", "id": 4} v 0) v1 -- 0x7f1afc04e9b0 con 0x7f1afc06d5a0 2023-10-14T16:40:03.361 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.361+0000 7f1af8ff9640 1 -- 172.21.15.186:0/1808762673 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c1c0fbc8-08a3-46f0-8377-2acd51c3e048", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1aec018070 con 0x7f1afc06d5a0 2023-10-14T16:40:03.362 INFO:teuthology.orchestra.run.smithi186.stdout:4 2023-10-14T16:40:03.363 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 msgr2=0x7f1afc15bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.363 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 0x7f1afc15bef0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f1aec00c450 tx=0x7f1aec00c910 comp rx=0 tx=0).stop 2023-10-14T16:40:03.363 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 shutdown_connections 2023-10-14T16:40:03.364 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1afc0b4530 0x7f1afc15c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.364 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1afc0710e0 0x7f1afc15c430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.364 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 --2- 172.21.15.186:0/1808762673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1afc06d5a0 0x7f1afc15bef0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.364 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 >> 172.21.15.186:0/1808762673 conn(0x7f1afc0b0a80 msgr2=0x7f1afc0b3bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.364 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 shutdown_connections 2023-10-14T16:40:03.365 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.362+0000 7f1b0131a640 1 -- 172.21.15.186:0/1808762673 wait complete. 2023-10-14T16:40:03.379 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 8ba443e1-4c4b-42a6-a349-c1098dcdecb0 5 2023-10-14T16:40:03.389 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:03.387+0000 7f0b23c2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:40:03.399 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.398+0000 7f9c1b8da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:40:03.456 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.455+0000 7f9c1b8da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:40:03.554 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:03.552+0000 7f0b23c2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:40:03.571 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.570+0000 7f9c1b8da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:40:03.602 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.601+0000 7f3c929c9640 1 Processor -- start 2023-10-14T16:40:03.602 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.601+0000 7f3c929c9640 1 -- start start 2023-10-14T16:40:03.604 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.602+0000 7f3c929c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3c8c15d410 0x7f3c8c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.604 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.602+0000 7f3c929c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c8c0cd260 0x7f3c8c0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.604 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.602+0000 7f3c929c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c8c0b5d90 con 0x7f3c8c15d030 2023-10-14T16:40:03.605 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.602+0000 7f3c929c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0d2b50 con 0x7f3c8c0cd260 2023-10-14T16:40:03.605 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.602+0000 7f3c929c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c8c0cdbf0 con 0x7f3c8c15d410 2023-10-14T16:40:03.605 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3c8c15d410 0x7f3c8c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.605 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3c8c15d410 0x7f3c8c0d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43192/0 (socket says 172.21.15.186:43192) 2023-10-14T16:40:03.605 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1876740836 learned_addr learned my addr 172.21.15.186:0/1876740836 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:03.606 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c90f66640 1 --2- 172.21.15.186:0/1876740836 >> v2:172.21.15.29:3300/0 conn(0x7f3c8c0cd260 0x7f3c8c0cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.606 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1876740836 >> v2:172.21.15.29:3300/0 conn(0x7f3c8c0cd260 msgr2=0x7f3c8c0cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:03.606 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 --2- 172.21.15.186:0/1876740836 >> v2:172.21.15.29:3300/0 conn(0x7f3c8c0cd260 0x7f3c8c0cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.606 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1876740836 >> v1:172.21.15.29:6789/0 conn(0x7f3c8c15d030 legacy=0x7f3c8c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:03.607 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.603+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1876740836 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c8c160fc0 con 0x7f3c8c15d410 2023-10-14T16:40:03.608 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.604+0000 7f3c8b7fe640 1 --2- 172.21.15.186:0/1876740836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d410 0x7f3c8c0d1630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c160cd0 tx=0x7f3c8002b9a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b8975358cb70255 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.608 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.604+0000 7f3c8affd640 1 -- 172.21.15.186:0/1876740836 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80030860 con 0x7f3c8c15d410 2023-10-14T16:40:03.608 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c8affd640 1 -- 172.21.15.186:0/1876740836 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c800309c0 con 0x7f3c8c15d410 2023-10-14T16:40:03.608 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c8affd640 1 -- 172.21.15.186:0/1876740836 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80030d50 con 0x7f3c8c15d410 2023-10-14T16:40:03.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 -- 172.21.15.186:0/1876740836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d410 msgr2=0x7f3c8c0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1876740836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d410 0x7f3c8c0d1630 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c160cd0 tx=0x7f3c8002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 -- 172.21.15.186:0/1876740836 shutdown_connections 2023-10-14T16:40:03.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1876740836 >> v2:172.21.15.29:3300/0 conn(0x7f3c8c0cd260 0x7f3c8c0cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.609 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1876740836 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d410 0x7f3c8c0d1630 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 -- 172.21.15.186:0/1876740836 >> 172.21.15.186:0/1876740836 conn(0x7f3c8c0cf720 msgr2=0x7f3c8c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 -- 172.21.15.186:0/1876740836 shutdown_connections 2023-10-14T16:40:03.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.605+0000 7f3c929c9640 1 -- 172.21.15.186:0/1876740836 wait complete. 2023-10-14T16:40:03.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.607+0000 7f3c929c9640 1 Processor -- start 2023-10-14T16:40:03.610 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.607+0000 7f3c929c9640 1 -- start start 2023-10-14T16:40:03.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c0cd260 0x7f3c8c14e100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c8c15d410 0x7f3c8c1f67e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c14ca10 con 0x7f3c8c0cd260 2023-10-14T16:40:03.613 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c14bd90 con 0x7f3c8c15d030 2023-10-14T16:40:03.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c929c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8c14c070 con 0x7f3c8c15d410 2023-10-14T16:40:03.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c8b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c8b7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.186:43194/0 (socket says 172.21.15.186:43194) 2023-10-14T16:40:03.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1563961199 learned_addr learned my addr 172.21.15.186:0/1563961199 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:03.614 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.608+0000 7f3c90f66640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c8c15d410 0x7f3c8c1f67e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8bfff640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c0cd260 0x7f3c8c14e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c8c15d410 msgr2=0x7f3c8c1f67e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c8c15d410 0x7f3c8c1f67e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1563961199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c0cd260 msgr2=0x7f3c8c14e100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.615 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c0cd260 0x7f3c8c14e100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.616 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 -- 172.21.15.186:0/1563961199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c800094d0 con 0x7f3c8c15d030 2023-10-14T16:40:03.616 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c8b7fe640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c8002b9e0 tx=0x7f3c80004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.616 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80039a90 con 0x7f3c8c15d030 2023-10-14T16:40:03.616 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c80039bf0 con 0x7f3c8c15d030 2023-10-14T16:40:03.616 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c80043410 con 0x7f3c8c15d030 2023-10-14T16:40:03.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.609+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c8c14c350 con 0x7f3c8c15d030 2023-10-14T16:40:03.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.610+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c8c150e70 con 0x7f3c8c15d030 2023-10-14T16:40:03.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.610+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f3c80043870 con 0x7f3c8c15d030 2023-10-14T16:40:03.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.610+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (secure 0 0 0) 0x7f3c8004d440 con 0x7f3c8c15d030 2023-10-14T16:40:03.617 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.611+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8c0d1ef0 con 0x7f3c8c15d030 2023-10-14T16:40:03.618 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.613+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f3c80048070 con 0x7f3c8c15d030 2023-10-14T16:40:03.690 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.688+0000 7f9c1b8da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:40:03.691 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.690+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd new", "uuid": "8ba443e1-4c4b-42a6-a349-c1098dcdecb0", "id": 5} v 0) v1 -- 0x7f3c8c0d11c0 con 0x7f3c8c15d030 2023-10-14T16:40:03.698 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.697+0000 7f3c88ff9640 1 -- 172.21.15.186:0/1563961199 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8ba443e1-4c4b-42a6-a349-c1098dcdecb0", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3c80034070 con 0x7f3c8c15d030 2023-10-14T16:40:03.699 INFO:teuthology.orchestra.run.smithi186.stdout:5 2023-10-14T16:40:03.700 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.698+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 msgr2=0x7f3c8c1f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.700 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.698+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3c8002b9e0 tx=0x7f3c80004730 comp rx=0 tx=0).stop 2023-10-14T16:40:03.700 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 shutdown_connections 2023-10-14T16:40:03.700 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c8c15d410 0x7f3c8c1f67e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.701 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c8c15d030 0x7f3c8c1f62a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.701 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 --2- 172.21.15.186:0/1563961199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c8c0cd260 0x7f3c8c14e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.701 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 >> 172.21.15.186:0/1563961199 conn(0x7f3c8c0cf720 msgr2=0x7f3c8c065ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.701 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 shutdown_connections 2023-10-14T16:40:03.701 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.699+0000 7f3c929c9640 1 -- 172.21.15.186:0/1563961199 wait complete. 2023-10-14T16:40:03.717 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new d143f29f-dde3-4f91-b56a-e45c65002e1a 6 2023-10-14T16:40:03.860 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.858+0000 7f9c1b8da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:40:03.908 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.906+0000 7fd99270f640 1 Processor -- start 2023-10-14T16:40:03.908 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.907+0000 7fd99270f640 1 -- start start 2023-10-14T16:40:03.909 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd99270f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.910 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd99270f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c0cb030 con 0x7fd98c06a4b0 2023-10-14T16:40:03.910 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd99270f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd98c0c91c0 con 0x7fd98c15ad70 2023-10-14T16:40:03.910 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd99270f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0c94a0 con 0x7fd98c0caac0 2023-10-14T16:40:03.910 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd98bfff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd98c15ad70 0x7fd98c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40682/0 (socket says 172.21.15.186:40682) 2023-10-14T16:40:03.910 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd98bfff640 1 -- 172.21.15.186:0/149725260 learned_addr learned my addr 172.21.15.186:0/149725260 (peer_addr_for_me v1:172.21.15.186:0/0) 2023-10-14T16:40:03.911 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.908+0000 7fd990cac640 1 --2- 172.21.15.186:0/149725260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.911 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.909+0000 7fd98b7fe640 1 -- 172.21.15.186:0/149725260 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81517041 0 0) 0x7fd98c0c91c0 con 0x7fd98c15ad70 2023-10-14T16:40:03.912 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.909+0000 7fd98b7fe640 1 -- 172.21.15.186:0/149725260 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd978003520 con 0x7fd98c15ad70 2023-10-14T16:40:03.912 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.909+0000 7fd990cac640 1 -- 172.21.15.186:0/149725260 >> v1:172.21.15.186:6789/0 conn(0x7fd98c15ad70 legacy=0x7fd98c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.912 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.909+0000 7fd990cac640 1 -- 172.21.15.186:0/149725260 >> v1:172.21.15.29:6789/0 conn(0x7fd98c06a4b0 legacy=0x7fd98c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.912 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.909+0000 7fd990cac640 1 -- 172.21.15.186:0/149725260 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd98c0c9780 con 0x7fd98c0caac0 2023-10-14T16:40:03.913 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.910+0000 7fd990cac640 1 --2- 172.21.15.186:0/149725260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c0ad4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd984009dc0 tx=0x7fd98402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea02b6e988661e2b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.913 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.910+0000 7fd98b7fe640 1 -- 172.21.15.186:0/149725260 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd98402ccd0 con 0x7fd98c0caac0 2023-10-14T16:40:03.913 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.910+0000 7fd98b7fe640 1 -- 172.21.15.186:0/149725260 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd98402ce30 con 0x7fd98c0caac0 2023-10-14T16:40:03.913 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.910+0000 7fd98b7fe640 1 -- 172.21.15.186:0/149725260 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd984034660 con 0x7fd98c0caac0 2023-10-14T16:40:03.914 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 -- 172.21.15.186:0/149725260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 msgr2=0x7fd98c0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.915 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 --2- 172.21.15.186:0/149725260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c0ad4d0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fd984009dc0 tx=0x7fd98402bf20 comp rx=0 tx=0).stop 2023-10-14T16:40:03.915 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 -- 172.21.15.186:0/149725260 shutdown_connections 2023-10-14T16:40:03.915 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 --2- 172.21.15.186:0/149725260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c0ad4d0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.915 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 -- 172.21.15.186:0/149725260 >> 172.21.15.186:0/149725260 conn(0x7fd98c0ce9b0 msgr2=0x7fd98c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:03.915 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 -- 172.21.15.186:0/149725260 shutdown_connections 2023-10-14T16:40:03.916 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.911+0000 7fd99270f640 1 -- 172.21.15.186:0/149725260 wait complete. 2023-10-14T16:40:03.916 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:03.911+0000 7f9c1b8da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:40:03.918 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.913+0000 7fd99270f640 1 Processor -- start 2023-10-14T16:40:03.918 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.913+0000 7fd99270f640 1 -- start start 2023-10-14T16:40:03.919 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd98c06a4b0 0x7fd98c1560c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.919 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.919 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 0x7fd98c0c4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:03.919 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c0af400 con 0x7fd98c06a4b0 2023-10-14T16:40:03.919 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c156d40 con 0x7fd98c15ad70 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd99270f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd98c1568a0 con 0x7fd98c0caac0 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd983fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd990cac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 0x7fd98c0c4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.914+0000 7fd983fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41956/0 (socket says 172.21.15.186:41956) 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 -- 172.21.15.186:0/1647081100 learned_addr learned my addr 172.21.15.186:0/1647081100 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:03.920 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd98bfff640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd98c06a4b0 0x7fd98c1560c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:03.921 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 -- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 msgr2=0x7fd98c0c4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.921 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 0x7fd98c0c4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.921 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 -- 172.21.15.186:0/1647081100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd98c06a4b0 msgr2=0x7fd98c1560c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:03.921 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd98c06a4b0 0x7fd98c1560c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:03.921 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 -- 172.21.15.186:0/1647081100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd984009a70 con 0x7fd98c0caac0 2023-10-14T16:40:03.922 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd990cac640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 0x7fd98c0c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:03.922 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.915+0000 7fd983fff640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c450 tx=0x7fd97400c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:03.922 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9740052d0 con 0x7fd98c0caac0 2023-10-14T16:40:03.922 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd974005430 con 0x7fd98c0caac0 2023-10-14T16:40:03.922 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd974012870 con 0x7fd98c0caac0 2023-10-14T16:40:03.923 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd98c0af650 con 0x7fd98c0caac0 2023-10-14T16:40:03.923 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd98c15b990 con 0x7fd98c0caac0 2023-10-14T16:40:03.923 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7fd974012cd0 con 0x7fd98c0caac0 2023-10-14T16:40:03.923 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.916+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (secure 0 0 0) 0x7fd974011380 con 0x7fd98c0caac0 2023-10-14T16:40:03.923 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.917+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd98c1566e0 con 0x7fd98c0caac0 2023-10-14T16:40:03.924 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.920+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7fd9740035f0 con 0x7fd98c0caac0 2023-10-14T16:40:03.998 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:03.997+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "d143f29f-dde3-4f91-b56a-e45c65002e1a", "id": 6} v 0) v1 -- 0x7fd98c04e160 con 0x7fd98c0caac0 2023-10-14T16:40:04.005 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.004+0000 7fd9897fa640 1 -- 172.21.15.186:0/1647081100 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d143f29f-dde3-4f91-b56a-e45c65002e1a", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fd974003790 con 0x7fd98c0caac0 2023-10-14T16:40:04.005 INFO:teuthology.orchestra.run.smithi186.stdout:6 2023-10-14T16:40:04.006 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 msgr2=0x7fd98c15b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.006 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fd97400c450 tx=0x7fd97400c910 comp rx=0 tx=0).stop 2023-10-14T16:40:04.007 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 shutdown_connections 2023-10-14T16:40:04.007 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd98c15ad70 0x7fd98c0c4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.007 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd98c0caac0 0x7fd98c15b450 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.007 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 --2- 172.21.15.186:0/1647081100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd98c06a4b0 0x7fd98c1560c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.007 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 >> 172.21.15.186:0/1647081100 conn(0x7fd98c0ce9b0 msgr2=0x7fd98c0b1780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:04.008 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 shutdown_connections 2023-10-14T16:40:04.008 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.005+0000 7fd99270f640 1 -- 172.21.15.186:0/1647081100 wait complete. 2023-10-14T16:40:04.023 DEBUG:teuthology.orchestra.run.smithi186:> sudo ceph --cluster ceph osd new 36e18e36-52bb-4245-9dc0-c5f7a85f4cb2 7 2023-10-14T16:40:04.085 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:04.084+0000 7f9c1b8da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:40:04.223 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.222+0000 7f4f13472640 1 Processor -- start 2023-10-14T16:40:04.223 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.222+0000 7f4f13472640 1 -- start start 2023-10-14T16:40:04.225 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.223+0000 7f4f13472640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f0c0cded0 0x7f4f0c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.225 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.223+0000 7f4f13472640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0c15d150 con 0x7f4f0c0b8480 2023-10-14T16:40:04.225 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.223+0000 7f4f13472640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0c0d1180 con 0x7f4f0c15cd70 2023-10-14T16:40:04.225 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.223+0000 7f4f13472640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f0c0d1460 con 0x7f4f0c0cded0 2023-10-14T16:40:04.225 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f12470640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4f0c15cd70 0x7f4f0c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.186:40684/0 (socket says 172.21.15.186:40684) 2023-10-14T16:40:04.226 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f12470640 1 -- 172.21.15.186:0/1318102358 learned_addr learned my addr 172.21.15.186:0/1318102358 (peer_addr_for_me v1:172.21.15.186:0/0) 2023-10-14T16:40:04.228 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f12c71640 1 --2- 172.21.15.186:0/1318102358 >> v2:172.21.15.29:3300/0 conn(0x7f4f0c0cded0 0x7f4f0c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.228 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113085218 0 0) 0x7f4f0c0d1180 con 0x7f4f0c15cd70 2023-10-14T16:40:04.229 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef8003520 con 0x7f4f0c15cd70 2023-10-14T16:40:04.229 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255641503 0 0) 0x7f4ef8003520 con 0x7f4f0c15cd70 2023-10-14T16:40:04.229 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 >> v2:172.21.15.29:3300/0 conn(0x7f4f0c0cded0 msgr2=0x7f4f0c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.229 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 --2- 172.21.15.186:0/1318102358 >> v2:172.21.15.29:3300/0 conn(0x7f4f0c0cded0 0x7f4f0c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.229 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.224+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 >> v1:172.21.15.29:6789/0 conn(0x7f4f0c0b8480 legacy=0x7f4f0c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.230 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 --> v1:172.21.15.186:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c0d1740 con 0x7f4f0c15cd70 2023-10-14T16:40:04.230 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v1:172.21.15.186:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4f08001670 con 0x7f4f0c15cd70 2023-10-14T16:40:04.230 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 >> v1:172.21.15.186:6789/0 conn(0x7f4f0c15cd70 legacy=0x7f4f0c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.230 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 0x7f4ef8003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.230 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80047c0 0x7f4ef8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.231 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8005180 0x7f4ef800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.231 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c0d1460 con 0x7f4ef8005180 2023-10-14T16:40:04.231 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef800c000 con 0x7f4ef8003970 2023-10-14T16:40:04.231 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef800c1c0 con 0x7f4ef80047c0 2023-10-14T16:40:04.231 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 0x7f4ef8003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.233 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f11c6f640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80047c0 0x7f4ef8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.233 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 -- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80047c0 msgr2=0x7f4ef8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.233 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ef80047c0 0x7f4ef8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.233 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 -- 172.21.15.186:0/1318102358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8005180 msgr2=0x7f4ef800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.234 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12c71640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8005180 0x7f4ef800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.234 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8005180 0x7f4ef800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.234 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12c71640 1 -- 172.21.15.186:0/1318102358 reap_dead start 2023-10-14T16:40:04.234 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.225+0000 7f4f12470640 1 -- 172.21.15.186:0/1318102358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef8003520 con 0x7f4ef8003970 2023-10-14T16:40:04.234 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f12470640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 0x7f4ef8003d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800c210 tx=0x7f4ef80057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c33f80e26f524017 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:04.235 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f08005ad0 con 0x7f4ef8003970 2023-10-14T16:40:04.235 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f08005c50 con 0x7f4ef8003970 2023-10-14T16:40:04.235 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f1146e640 1 -- 172.21.15.186:0/1318102358 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f0802c870 con 0x7f4ef8003970 2023-10-14T16:40:04.235 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f13472640 1 -- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 msgr2=0x7f4ef8003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.235 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.227+0000 7f4f13472640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 0x7f4ef8003d70 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f0800c210 tx=0x7f4ef80057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.239 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.228+0000 7f4f13472640 1 -- 172.21.15.186:0/1318102358 shutdown_connections 2023-10-14T16:40:04.239 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.228+0000 7f4f13472640 1 --2- 172.21.15.186:0/1318102358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef8003970 0x7f4ef8003d70 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.239 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.228+0000 7f4f13472640 1 -- 172.21.15.186:0/1318102358 >> 172.21.15.186:0/1318102358 conn(0x7f4f0c0cf570 msgr2=0x7f4f0c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:04.239 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.228+0000 7f4f13472640 1 -- 172.21.15.186:0/1318102358 shutdown_connections 2023-10-14T16:40:04.240 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.228+0000 7f4f13472640 1 -- 172.21.15.186:0/1318102358 wait complete. 2023-10-14T16:40:04.240 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.229+0000 7f4f13472640 1 Processor -- start 2023-10-14T16:40:04.240 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.230+0000 7f4f13472640 1 -- start start 2023-10-14T16:40:04.240 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.230+0000 7f4f13472640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8003970 0x7f4f0c1f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.240 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.230+0000 7f4f13472640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 0x7f4f0c0a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.241 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f13472640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.241 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f13472640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c14c8a0 con 0x7f4ef8003970 2023-10-14T16:40:04.241 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f13472640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c0a9eb0 con 0x7f4f0c1f1e50 2023-10-14T16:40:04.241 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f13472640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0c1f22d0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.241 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.242 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f11c6f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 0x7f4f0c0a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.242 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.186:41984/0 (socket says 172.21.15.186:41984) 2023-10-14T16:40:04.242 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 -- 172.21.15.186:0/2939886001 learned_addr learned my addr 172.21.15.186:0/2939886001 (peer_addr_for_me v2:172.21.15.186:0/0) 2023-10-14T16:40:04.242 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12470640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8003970 0x7f4f0c1f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.243 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 -- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 msgr2=0x7f4f0c0a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.243 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 0x7f4f0c0a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.243 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 -- 172.21.15.186:0/2939886001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8003970 msgr2=0x7f4f0c1f3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.243 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8003970 0x7f4f0c1f3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.243 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f12c71640 1 -- 172.21.15.186:0/2939886001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0c14c5c0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.244 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.231+0000 7f4f11c6f640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 0x7f4f0c0a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:04.244 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f12c71640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8004740 tx=0x7f4f0400f520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:04.244 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f04009b30 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.244 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f04016420 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.244 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f040167d0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f0c15e280 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f0c15e840 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f4f04016930 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.232+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (secure 0 0 0) 0x7f4f04016be0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.233+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0c15e0e0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.245 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.236+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f4f0c15e0e0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.314 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.312+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd new", "uuid": "36e18e36-52bb-4245-9dc0-c5f7a85f4cb2", "id": 7} v 0) v1 -- 0x7f4f0c04e9b0 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.323 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.322+0000 7f4f02ffd640 1 -- 172.21.15.186:0/2939886001 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "36e18e36-52bb-4245-9dc0-c5f7a85f4cb2", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4f0401f410 con 0x7f4f0c0a9a30 2023-10-14T16:40:04.324 INFO:teuthology.orchestra.run.smithi186.stdout:7 2023-10-14T16:40:04.325 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 msgr2=0x7f4f0c0b0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.325 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4ef8004740 tx=0x7f4f0400f520 comp rx=0 tx=0).stop 2023-10-14T16:40:04.325 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 shutdown_connections 2023-10-14T16:40:04.325 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0c0a9a30 0x7f4f0c0b0880 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.325 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f0c1f1e50 0x7f4f0c0a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.326 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 --2- 172.21.15.186:0/2939886001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef8003970 0x7f4f0c1f3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.326 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 >> 172.21.15.186:0/2939886001 conn(0x7f4f0c0cf570 msgr2=0x7f4f0c0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:04.326 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 shutdown_connections 2023-10-14T16:40:04.326 INFO:teuthology.orchestra.run.smithi186.stderr:2023-10-14T16:40:04.323+0000 7f4f13472640 1 -- 172.21.15.186:0/2939886001 wait complete. 2023-10-14T16:40:04.340 INFO:tasks.ceph.osd.0:Restarting daemon 2023-10-14T16:40:04.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-10-14T16:40:04.354 INFO:tasks.ceph.osd.0:Started 2023-10-14T16:40:04.355 INFO:tasks.ceph.osd.1:Restarting daemon 2023-10-14T16:40:04.355 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-10-14T16:40:04.368 INFO:tasks.ceph.osd.1:Started 2023-10-14T16:40:04.369 INFO:tasks.ceph.osd.2:Restarting daemon 2023-10-14T16:40:04.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-10-14T16:40:04.379 INFO:tasks.ceph.osd.2:Started 2023-10-14T16:40:04.380 INFO:tasks.ceph.osd.3:Restarting daemon 2023-10-14T16:40:04.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-10-14T16:40:04.386 INFO:tasks.ceph.osd.3:Started 2023-10-14T16:40:04.387 INFO:tasks.ceph.osd.4:Restarting daemon 2023-10-14T16:40:04.387 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-10-14T16:40:04.396 INFO:tasks.ceph.osd.4:Started 2023-10-14T16:40:04.396 INFO:tasks.ceph.osd.5:Restarting daemon 2023-10-14T16:40:04.396 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-10-14T16:40:04.404 INFO:tasks.ceph.osd.5:Started 2023-10-14T16:40:04.405 INFO:tasks.ceph.osd.6:Restarting daemon 2023-10-14T16:40:04.405 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2023-10-14T16:40:04.413 INFO:tasks.ceph.osd.6:Started 2023-10-14T16:40:04.414 INFO:tasks.ceph.osd.7:Restarting daemon 2023-10-14T16:40:04.414 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2023-10-14T16:40:04.420 INFO:tasks.ceph.osd.7:Started 2023-10-14T16:40:04.421 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:04.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.646+0000 7f9116901640 1 Processor -- start 2023-10-14T16:40:04.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.646+0000 7f9116901640 1 -- start start 2023-10-14T16:40:04.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9110159b20 0x7f911006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9110159f00 0x7f91100b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f911006c120 0x7f91100b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911006f7f0 con 0x7f9110159b20 2023-10-14T16:40:04.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f911006c5e0 con 0x7f911006c120 2023-10-14T16:40:04.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9116901640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f911006c8c0 con 0x7f9110159f00 2023-10-14T16:40:04.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f910ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9110159b20 0x7f911006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f910ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9110159b20 0x7f911006bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57818/0 (socket says 172.21.15.29:57818) 2023-10-14T16:40:04.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f910ffff640 1 -- 172.21.15.29:0/3187930426 learned_addr learned my addr 172.21.15.29:0/3187930426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:04.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f9114e9e640 1 --2- 172.21.15.29:0/3187930426 >> v2:172.21.15.186:3300/0 conn(0x7f911006c120 0x7f91100b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.647+0000 7f910f7fe640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9110159f00 0x7f91100b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.648+0000 7f910ffff640 1 -- 172.21.15.29:0/3187930426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9110159f00 msgr2=0x7f91100b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.648+0000 7f910ffff640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9110159f00 0x7f91100b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.648+0000 7f910ffff640 1 -- 172.21.15.29:0/3187930426 >> v2:172.21.15.186:3300/0 conn(0x7f911006c120 msgr2=0x7f91100b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:04.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.648+0000 7f910ffff640 1 --2- 172.21.15.29:0/3187930426 >> v2:172.21.15.186:3300/0 conn(0x7f911006c120 0x7f91100b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.648+0000 7f910ffff640 1 -- 172.21.15.29:0/3187930426 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911006cba0 con 0x7f9110159b20 2023-10-14T16:40:04.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.649+0000 7f910ffff640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f911006bbe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91100c9270 tx=0x7f910002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=617953a642e1cc36 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:04.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.650+0000 7f910effd640 1 -- 172.21.15.29:0/3187930426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031870 con 0x7f9110159b20 2023-10-14T16:40:04.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.650+0000 7f910effd640 1 -- 172.21.15.29:0/3187930426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91000319d0 con 0x7f9110159b20 2023-10-14T16:40:04.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.650+0000 7f910effd640 1 -- 172.21.15.29:0/3187930426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9100031d60 con 0x7f9110159b20 2023-10-14T16:40:04.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.650+0000 7f9116901640 1 -- 172.21.15.29:0/3187930426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 msgr2=0x7f911006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.650+0000 7f9116901640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f911006bbe0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f91100c9270 tx=0x7f910002b910 comp rx=0 tx=0).stop 2023-10-14T16:40:04.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 -- 172.21.15.29:0/3187930426 shutdown_connections 2023-10-14T16:40:04.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 --2- 172.21.15.29:0/3187930426 >> v2:172.21.15.186:3300/0 conn(0x7f911006c120 0x7f91100b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9110159f00 0x7f91100b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 --2- 172.21.15.29:0/3187930426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f911006bbe0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 -- 172.21.15.29:0/3187930426 >> 172.21.15.29:0/3187930426 conn(0x7f91100cb410 msgr2=0x7f91100cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 -- 172.21.15.29:0/3187930426 shutdown_connections 2023-10-14T16:40:04.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.651+0000 7f9116901640 1 -- 172.21.15.29:0/3187930426 wait complete. 2023-10-14T16:40:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.652+0000 7f9116901640 1 Processor -- start 2023-10-14T16:40:04.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.652+0000 7f9116901640 1 -- start start 2023-10-14T16:40:04.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f911006c120 0x7f91100b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9110159f00 0x7f9110149cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:04.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91100b38a0 con 0x7f9110159b20 2023-10-14T16:40:04.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91100b2c90 con 0x7f9110159f00 2023-10-14T16:40:04.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.653+0000 7f9116901640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91100b2f70 con 0x7f911006c120 2023-10-14T16:40:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57824/0 (socket says 172.21.15.29:57824) 2023-10-14T16:40:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 -- 172.21.15.29:0/2709652247 learned_addr learned my addr 172.21.15.29:0/2709652247 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f9114e9e640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9110159f00 0x7f9110149cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910ffff640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f911006c120 0x7f91100b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 -- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f911006c120 msgr2=0x7f91100b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f911006c120 0x7f91100b2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 -- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9110159f00 msgr2=0x7f9110149cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9110159f00 0x7f9110149cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 -- 172.21.15.29:0/2709652247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91000094d0 con 0x7f9110159b20 2023-10-14T16:40:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910f7fe640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f910400e980 tx=0x7f910400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.654+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9104009670 con 0x7f9110159b20 2023-10-14T16:40:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f910400c980 con 0x7f9110159b20 2023-10-14T16:40:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f910400cd10 con 0x7f9110159b20 2023-10-14T16:40:04.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91100b3250 con 0x7f9110159b20 2023-10-14T16:40:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91100b2860 con 0x7f9110159b20 2023-10-14T16:40:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 1) v1 ==== 660+0+0 (secure 0 0 0) 0x7f9104005c90 con 0x7f9110159b20 2023-10-14T16:40:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.655+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9104006840 con 0x7f9110159b20 2023-10-14T16:40:04.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.656+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91100b3d30 con 0x7f9110159b20 2023-10-14T16:40:04.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.661+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9104021060 con 0x7f9110159b20 2023-10-14T16:40:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.725+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91100b3a10 con 0x7f9110159b20 2023-10-14T16:40:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.726+0000 7f910cff9640 1 -- 172.21.15.29:0/2709652247 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9104029020 con 0x7f9110159b20 2023-10-14T16:40:04.728 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:04.729 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":10,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:04.315491+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 msgr2=0x7f91101588b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f910400e980 tx=0x7f910400ee40 comp rx=0 tx=0).stop 2023-10-14T16:40:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 shutdown_connections 2023-10-14T16:40:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9110159f00 0x7f9110149cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9110159b20 0x7f91101588b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 --2- 172.21.15.29:0/2709652247 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f911006c120 0x7f91100b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 >> 172.21.15.29:0/2709652247 conn(0x7f91100cb410 msgr2=0x7f91100aed60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 shutdown_connections 2023-10-14T16:40:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:04.728+0000 7f9116901640 1 -- 172.21.15.29:0/2709652247 wait complete. 2023-10-14T16:40:04.746 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-10-14T16:40:04.746 INFO:tasks.ceph:Waiting for OSDs to come up 2023-10-14T16:40:04.761 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:04.760+0000 7f0b23c2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:40:05.082 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:05.080+0000 7f0b23c2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:40:05.090 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-14T16:40:05.089+0000 7f453a016500 -1 Falling back to public interface 2023-10-14T16:40:05.094 INFO:tasks.ceph.osd.4.smithi186.stderr:2023-10-14T16:40:05.092+0000 7fe7e334a500 -1 Falling back to public interface 2023-10-14T16:40:05.099 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:05.097+0000 7f330176c500 -1 Falling back to public interface 2023-10-14T16:40:05.101 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-14T16:40:05.100+0000 7f014478d500 -1 Falling back to public interface 2023-10-14T16:40:05.103 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-14T16:40:05.102+0000 7f0143734640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6810/0: (98) Address already in use 2023-10-14T16:40:05.105 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-14T16:40:05.104+0000 7f653a09c500 -1 Falling back to public interface 2023-10-14T16:40:05.109 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:05.108+0000 7f3300713640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6816/0: (98) Address already in use 2023-10-14T16:40:05.113 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:05.111+0000 7f3300713640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6822/0: (98) Address already in use 2023-10-14T16:40:05.113 INFO:tasks.ceph.osd.7.smithi186.stderr:2023-10-14T16:40:05.112+0000 7f3ad9a12500 -1 Falling back to public interface 2023-10-14T16:40:05.127 INFO:tasks.ceph.osd.5.smithi186.stderr:2023-10-14T16:40:05.126+0000 7f29a99fe500 -1 Falling back to public interface 2023-10-14T16:40:05.137 INFO:tasks.ceph.osd.6.smithi186.stderr:2023-10-14T16:40:05.136+0000 7fe8f9770500 -1 Falling back to public interface 2023-10-14T16:40:05.152 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:05.150+0000 7f0b23c2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:40:05.163 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:05.162+0000 7f9c1b8da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:40:05.270 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:05.269+0000 7f0b23c2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:40:05.481 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:05.480+0000 7f9c1b8da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:40:05.540 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:05.539+0000 7f9c1b8da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:40:05.551 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:05.653 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:40:05.652+0000 7f9c1b8da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:40:05.736 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.735+0000 7f98254b4640 1 Processor -- start 2023-10-14T16:40:05.737 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.735+0000 7f98254b4640 1 -- start start 2023-10-14T16:40:05.738 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f98254b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f9820167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:05.738 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f98254b4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200d15a0 con 0x7f982015cbe0 2023-10-14T16:40:05.738 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f98254b4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98200d1880 con 0x7f982015cfc0 2023-10-14T16:40:05.739 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f98254b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200d1b60 con 0x7f98200b4c80 2023-10-14T16:40:05.739 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f981ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f982015cbe0 0x7f98200d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40200/0 (socket says 172.21.15.29:40200) 2023-10-14T16:40:05.739 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.736+0000 7f981ffff640 1 -- 172.21.15.29:0/1533333893 learned_addr learned my addr 172.21.15.29:0/1533333893 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:05.739 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f9824cb3640 1 --2- 172.21.15.29:0/1533333893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f9820167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:05.740 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366789400 0 0) 0x7f98200d15a0 con 0x7f982015cbe0 2023-10-14T16:40:05.741 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9808003520 con 0x7f982015cbe0 2023-10-14T16:40:05.741 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734810219 0 0) 0x7f98200d1880 con 0x7f982015cfc0 2023-10-14T16:40:05.741 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98200d15a0 con 0x7f982015cfc0 2023-10-14T16:40:05.741 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f9824cb3640 1 -- 172.21.15.29:0/1533333893 >> v1:172.21.15.186:6789/0 conn(0x7f982015cfc0 legacy=0x7f98200b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.741 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f9824cb3640 1 -- 172.21.15.29:0/1533333893 >> v1:172.21.15.29:6789/0 conn(0x7f982015cbe0 legacy=0x7f98200d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.742 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.737+0000 7f9824cb3640 1 -- 172.21.15.29:0/1533333893 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98200d1e40 con 0x7f98200b4c80 2023-10-14T16:40:05.742 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.738+0000 7f9824cb3640 1 --2- 172.21.15.29:0/1533333893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f9820167910 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f98180097d0 tx=0x7f981802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=785913155e854868 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:05.742 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818031890 con 0x7f98200b4c80 2023-10-14T16:40:05.742 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98180319f0 con 0x7f98200b4c80 2023-10-14T16:40:05.742 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f981effd640 1 -- 172.21.15.29:0/1533333893 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9818031d60 con 0x7f98200b4c80 2023-10-14T16:40:05.743 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f98254b4640 1 -- 172.21.15.29:0/1533333893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 msgr2=0x7f9820167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.744 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f98254b4640 1 --2- 172.21.15.29:0/1533333893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f9820167910 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f98180097d0 tx=0x7f981802b930 comp rx=0 tx=0).stop 2023-10-14T16:40:05.744 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f98254b4640 1 -- 172.21.15.29:0/1533333893 shutdown_connections 2023-10-14T16:40:05.744 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f98254b4640 1 --2- 172.21.15.29:0/1533333893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f9820167910 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.744 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.739+0000 7f98254b4640 1 -- 172.21.15.29:0/1533333893 >> 172.21.15.29:0/1533333893 conn(0x7f98200cf570 msgr2=0x7f98200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:05.744 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.740+0000 7f98254b4640 1 -- 172.21.15.29:0/1533333893 shutdown_connections 2023-10-14T16:40:05.745 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.740+0000 7f98254b4640 1 -- 172.21.15.29:0/1533333893 wait complete. 2023-10-14T16:40:05.745 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.741+0000 7f98254b4640 1 Processor -- start 2023-10-14T16:40:05.745 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.741+0000 7f98254b4640 1 -- start start 2023-10-14T16:40:05.747 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f98200ccc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:05.747 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:05.747 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f982015cfc0 0x7f98200b2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:05.748 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820158ea0 con 0x7f982015cbe0 2023-10-14T16:40:05.748 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9820158290 con 0x7f982015cfc0 2023-10-14T16:40:05.748 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.742+0000 7f98254b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9820158570 con 0x7f98200b4c80 2023-10-14T16:40:05.748 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:05.748 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57902/0 (socket says 172.21.15.29:57902) 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 -- 172.21.15.29:0/3748036187 learned_addr learned my addr 172.21.15.29:0/3748036187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f9824cb3640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f982015cfc0 0x7f98200b2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981ffff640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f98200ccc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 -- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 msgr2=0x7f98200ccc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f98200ccc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.749 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 -- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f982015cfc0 msgr2=0x7f98200b2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.750 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f982015cfc0 0x7f98200b2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.750 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 -- 172.21.15.29:0/3748036187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9818009400 con 0x7f982015cbe0 2023-10-14T16:40:05.750 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981ffff640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f98200ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:05.750 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.743+0000 7f981f7fe640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f981400c7f0 tx=0x7f981400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:05.750 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.744+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814003da0 con 0x7f982015cbe0 2023-10-14T16:40:05.751 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.744+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9814005070 con 0x7f982015cbe0 2023-10-14T16:40:05.751 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.744+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9814005400 con 0x7f982015cbe0 2023-10-14T16:40:05.751 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.744+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9820158850 con 0x7f982015cbe0 2023-10-14T16:40:05.751 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.744+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98200cc880 con 0x7f982015cbe0 2023-10-14T16:40:05.751 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.745+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9820159050 con 0x7f982015cbe0 2023-10-14T16:40:05.752 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.745+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 2) v1 ==== 44566+0+0 (secure 0 0 0) 0x7f9814005560 con 0x7f982015cbe0 2023-10-14T16:40:05.752 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.746+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (secure 0 0 0) 0x7f9814048f30 con 0x7f982015cbe0 2023-10-14T16:40:05.752 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.748+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72483 (secure 0 0 0) 0x7f9820159050 con 0x7f982015cbe0 2023-10-14T16:40:05.817 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.815+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9820159050 con 0x7f982015cbe0 2023-10-14T16:40:05.817 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.816+0000 7f981cff9640 1 -- 172.21.15.29:0/3748036187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f9820159050 con 0x7f982015cbe0 2023-10-14T16:40:05.818 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-14T16:40:05.818 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":10,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:04.315491+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:05.820 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.817+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 msgr2=0x7f9820159330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:05.820 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.817+0000 7f98254b4640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f981400c7f0 tx=0x7f981400ccb0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.820 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 shutdown_connections 2023-10-14T16:40:05.820 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f982015cfc0 0x7f98200b2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.820 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982015cbe0 0x7f9820159330 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 --2- 172.21.15.29:0/3748036187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200b4c80 0x7f98200ccc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:05.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 >> 172.21.15.29:0/3748036187 conn(0x7f98200cf570 msgr2=0x7f982015c060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:05.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 shutdown_connections 2023-10-14T16:40:05.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:05.818+0000 7f98254b4640 1 -- 172.21.15.29:0/3748036187 wait complete. 2023-10-14T16:40:05.833 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-14T16:40:06.313 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:06.311+0000 7f0ac48d7640 -1 mgr.server handle_report got status from non-daemon mon.b 2023-10-14T16:40:06.314 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:40:06.312+0000 7f0ac48d7640 -1 mgr.server handle_report got status from non-daemon mon.c 2023-10-14T16:40:08.606 INFO:tasks.ceph.osd.4.smithi186.stderr:2023-10-14T16:40:08.604+0000 7fe7e334a500 -1 osd.4 0 log_to_monitors true 2023-10-14T16:40:08.660 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:08.659+0000 7f330176c500 -1 osd.2 0 log_to_monitors true 2023-10-14T16:40:08.667 INFO:tasks.ceph.osd.5.smithi186.stderr:2023-10-14T16:40:08.666+0000 7f29a99fe500 -1 osd.5 0 log_to_monitors true 2023-10-14T16:40:08.667 INFO:tasks.ceph.osd.7.smithi186.stderr:2023-10-14T16:40:08.666+0000 7f3ad9a12500 -1 osd.7 0 log_to_monitors true 2023-10-14T16:40:08.714 INFO:tasks.ceph.osd.6.smithi186.stderr:2023-10-14T16:40:08.713+0000 7fe8f9770500 -1 osd.6 0 log_to_monitors true 2023-10-14T16:40:08.877 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-14T16:40:08.875+0000 7f453a016500 -1 osd.0 0 log_to_monitors true 2023-10-14T16:40:08.916 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-14T16:40:08.914+0000 7f014478d500 -1 osd.3 0 log_to_monitors true 2023-10-14T16:40:08.926 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-14T16:40:08.924+0000 7f653a09c500 -1 osd.1 0 log_to_monitors true 2023-10-14T16:40:10.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:40:10.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:40:12.124 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-14T16:40:12.122+0000 7f653703f640 -1 osd.1 0 waiting for initial osdmap 2023-10-14T16:40:12.138 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:12.137+0000 7f32fced5640 -1 osd.2 0 waiting for initial osdmap 2023-10-14T16:40:12.146 INFO:tasks.ceph.osd.1.smithi029.stderr:2023-10-14T16:40:12.144+0000 7f652de38640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.149 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-14T16:40:12.147+0000 7f4535159640 -1 osd.0 0 waiting for initial osdmap 2023-10-14T16:40:12.153 INFO:tasks.ceph.osd.2.smithi029.stderr:2023-10-14T16:40:12.152+0000 7f32f5508640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.171 INFO:tasks.ceph.osd.0.smithi029.stderr:2023-10-14T16:40:12.171+0000 7f452d78c640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.190 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-14T16:40:12.189+0000 7f0141730640 -1 osd.3 0 waiting for initial osdmap 2023-10-14T16:40:12.204 INFO:tasks.ceph.osd.3.smithi029.stderr:2023-10-14T16:40:12.202+0000 7f0138529640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.639 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:12.671 INFO:tasks.ceph.osd.7.smithi186.stderr:2023-10-14T16:40:12.670+0000 7f3ad4b1d640 -1 osd.7 0 waiting for initial osdmap 2023-10-14T16:40:12.689 INFO:tasks.ceph.osd.7.smithi186.stderr:2023-10-14T16:40:12.688+0000 7f3acd150640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.713 INFO:tasks.ceph.osd.5.smithi186.stderr:2023-10-14T16:40:12.712+0000 7f29a5167640 -1 osd.5 0 waiting for initial osdmap 2023-10-14T16:40:12.718 INFO:tasks.ceph.osd.4.smithi186.stderr:2023-10-14T16:40:12.717+0000 7fe7deab5640 -1 osd.4 0 waiting for initial osdmap 2023-10-14T16:40:12.730 INFO:tasks.ceph.osd.5.smithi186.stderr:2023-10-14T16:40:12.729+0000 7f299d79a640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.735 INFO:tasks.ceph.osd.4.smithi186.stderr:2023-10-14T16:40:12.734+0000 7fe7d70e8640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.755 INFO:tasks.ceph.osd.6.smithi186.stderr:2023-10-14T16:40:12.754+0000 7fe8f6713640 -1 osd.6 0 waiting for initial osdmap 2023-10-14T16:40:12.773 INFO:tasks.ceph.osd.6.smithi186.stderr:2023-10-14T16:40:12.772+0000 7fe8ed50c640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-14T16:40:12.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.819+0000 7f1ccd6a2640 1 Processor -- start 2023-10-14T16:40:12.821 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.819+0000 7f1ccd6a2640 1 -- start start 2023-10-14T16:40:12.823 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1ccd6a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc80cfc10 0x7f1cc80d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.823 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1ccd6a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80b3e70 0x7f1cc80d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.823 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1ccd6a2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cc80b3940 con 0x7f1cc80b3070 2023-10-14T16:40:12.824 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1ccd6a2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc80b45e0 con 0x7f1cc80cfc10 2023-10-14T16:40:12.824 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1ccd6a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80c7880 con 0x7f1cc80b3e70 2023-10-14T16:40:12.824 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1cc67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc80cfc10 0x7f1cc80d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.824 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1cc67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc80cfc10 0x7f1cc80d0b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50992/0 (socket says 172.21.15.29:50992) 2023-10-14T16:40:12.824 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1cc67fc640 1 -- 172.21.15.29:0/4065755368 learned_addr learned my addr 172.21.15.29:0/4065755368 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:12.825 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.821+0000 7f1cc77fe640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80b3e70 0x7f1cc80d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.826 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.822+0000 7f1cc67fc640 1 -- 172.21.15.29:0/4065755368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80b3e70 msgr2=0x7f1cc80d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:12.826 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.822+0000 7f1cc67fc640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80b3e70 0x7f1cc80d1040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:12.826 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.822+0000 7f1cc67fc640 1 -- 172.21.15.29:0/4065755368 >> v1:172.21.15.186:6789/0 conn(0x7f1cc80b3070 legacy=0x7f1cc80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:12.827 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.822+0000 7f1cc67fc640 1 -- 172.21.15.29:0/4065755368 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc80b3b90 con 0x7f1cc80cfc10 2023-10-14T16:40:12.827 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.823+0000 7f1cc67fc640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80cfc10 0x7f1cc80d0b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=acbc19092f3195f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:12.827 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.823+0000 7f1cc5ffb640 1 -- 172.21.15.29:0/4065755368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0030840 con 0x7f1cc80cfc10 2023-10-14T16:40:12.827 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.823+0000 7f1cc5ffb640 1 -- 172.21.15.29:0/4065755368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb00309a0 con 0x7f1cc80cfc10 2023-10-14T16:40:12.827 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.823+0000 7f1cc5ffb640 1 -- 172.21.15.29:0/4065755368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0030d30 con 0x7f1cc80cfc10 2023-10-14T16:40:12.828 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/4065755368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80cfc10 msgr2=0x7f1cc80d0b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:12.828 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80cfc10 0x7f1cc80d0b00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1cb0009870 tx=0x7f1cb002b980 comp rx=0 tx=0).stop 2023-10-14T16:40:12.828 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/4065755368 shutdown_connections 2023-10-14T16:40:12.828 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80b3e70 0x7f1cc80d1040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:12.828 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/4065755368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80cfc10 0x7f1cc80d0b00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:12.829 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/4065755368 >> 172.21.15.29:0/4065755368 conn(0x7f1cc80d2d20 msgr2=0x7f1cc80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:12.829 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/4065755368 shutdown_connections 2023-10-14T16:40:12.829 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.824+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/4065755368 wait complete. 2023-10-14T16:40:12.831 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.826+0000 7f1ccd6a2640 1 Processor -- start 2023-10-14T16:40:12.831 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.826+0000 7f1ccd6a2640 1 -- start start 2023-10-14T16:40:12.831 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80b3070 0x7f1cc815e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.831 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.832 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80cfc10 0x7f1cc809f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.832 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc815e560 con 0x7f1cc80b3e70 2023-10-14T16:40:12.832 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc809d300 con 0x7f1cc80b3070 2023-10-14T16:40:12.832 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1ccd6a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc809d5e0 con 0x7f1cc80cfc10 2023-10-14T16:40:12.832 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.833 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50998/0 (socket says 172.21.15.29:50998) 2023-10-14T16:40:12.833 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 -- 172.21.15.29:0/1629904258 learned_addr learned my addr 172.21.15.29:0/1629904258 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:12.833 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc6ffd640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80b3070 0x7f1cc815e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.833 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc77fe640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80cfc10 0x7f1cc809f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.834 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 -- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80cfc10 msgr2=0x7f1cc809f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:12.834 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80cfc10 0x7f1cc809f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:12.834 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 -- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80b3070 msgr2=0x7f1cc815e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:12.834 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80b3070 0x7f1cc815e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:12.834 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.827+0000 7f1cc67fc640 1 -- 172.21.15.29:0/1629904258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb00094d0 con 0x7f1cc80b3e70 2023-10-14T16:40:12.835 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1cc67fc640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cb002b9c0 tx=0x7f1cb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:12.835 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb003e070 con 0x7f1cc80b3e70 2023-10-14T16:40:12.835 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb0039a10 con 0x7f1cc80b3e70 2023-10-14T16:40:12.835 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb0039de0 con 0x7f1cc80b3e70 2023-10-14T16:40:12.835 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc809d8c0 con 0x7f1cc80b3e70 2023-10-14T16:40:12.836 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.828+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc809de80 con 0x7f1cc80b3e70 2023-10-14T16:40:12.836 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.829+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc80cd120 con 0x7f1cc80b3e70 2023-10-14T16:40:12.836 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.830+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 4) v1 ==== 89173+0+0 (secure 0 0 0) 0x7f1cb004b400 con 0x7f1cc80b3e70 2023-10-14T16:40:12.837 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.831+0000 7f1cbf7fe640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 0x7f1ca806d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:12.837 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.831+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 2231+0+0 (secure 0 0 0) 0x7f1cb00aac60 con 0x7f1cc80b3e70 2023-10-14T16:40:12.837 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.831+0000 7f1cc6ffd640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 0x7f1ca806d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:12.837 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.832+0000 7f1cc6ffd640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 0x7f1ca806d2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8001330 tx=0x7f1cb8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:12.838 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:12.836+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cb0076b90 con 0x7f1cc80b3e70 2023-10-14T16:40:13.028 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.026+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cc804e160 con 0x7f1cc80b3e70 2023-10-14T16:40:13.030 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.027+0000 7f1cbf7fe640 1 -- 172.21.15.29:0/1629904258 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) v1 ==== 74+0+7057 (secure 0 0 0) 0x7f1cb0034070 con 0x7f1cc80b3e70 2023-10-14T16:40:13.030 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-14T16:40:13.030 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":12,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:10.317203+0000","last_up_change":"0.000000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:13.033 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.030+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 msgr2=0x7f1ca806d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:13.033 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.030+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 0x7f1ca806d2a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f1cb8001330 tx=0x7f1cb8009040 comp rx=0 tx=0).stop 2023-10-14T16:40:13.034 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 msgr2=0x7f1cc809ca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:13.034 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f1cb002b9c0 tx=0x7f1cb00046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:13.034 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 shutdown_connections 2023-10-14T16:40:13.034 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1ca806ade0 0x7f1ca806d2a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:13.034 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc80cfc10 0x7f1cc809f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:13.035 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc80b3e70 0x7f1cc809ca00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:13.035 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 --2- 172.21.15.29:0/1629904258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc80b3070 0x7f1cc815e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:13.035 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 >> 172.21.15.29:0/1629904258 conn(0x7f1cc80d2d20 msgr2=0x7f1cc80b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:13.035 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 shutdown_connections 2023-10-14T16:40:13.036 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:13.031+0000 7f1ccd6a2640 1 -- 172.21.15.29:0/1629904258 wait complete. 2023-10-14T16:40:13.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2023-10-14T16:40:19.853 DEBUG:teuthology.orchestra.run.smithi029:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:20.053 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.052+0000 7f6a9b59e640 1 Processor -- start 2023-10-14T16:40:20.054 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.052+0000 7f6a9b59e640 1 -- start start 2023-10-14T16:40:20.060 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6a9c070b60 0x7f6a9c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.060 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9c15aef0 con 0x7f6a9c159950 2023-10-14T16:40:20.061 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9c15d910 con 0x7f6a9c06c100 2023-10-14T16:40:20.061 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9c15e490 con 0x7f6a9c070b60 2023-10-14T16:40:20.061 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a9c159950 0x7f6a9c070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57806/0 (socket says 172.21.15.29:57806) 2023-10-14T16:40:20.061 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.054+0000 7f6a9a59c640 1 -- 172.21.15.29:0/2588109758 learned_addr learned my addr 172.21.15.29:0/2588109758 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:20.061 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528878894 0 0) 0x7f6a8c001110 con 0x7f6a9c159950 2023-10-14T16:40:20.062 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a93fff640 1 --2- 172.21.15.29:0/2588109758 >> v2:172.21.15.186:3300/0 conn(0x7f6a9c070b60 0x7f6a9c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.062 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a88003520 con 0x7f6a9c159950 2023-10-14T16:40:20.062 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1977414970 0 0) 0x7f6a88003520 con 0x7f6a9c159950 2023-10-14T16:40:20.062 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 >> v2:172.21.15.186:3300/0 conn(0x7f6a9c070b60 msgr2=0x7f6a9c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.062 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 --2- 172.21.15.29:0/2588109758 >> v2:172.21.15.186:3300/0 conn(0x7f6a9c070b60 0x7f6a9c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.063 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 >> v1:172.21.15.186:6789/0 conn(0x7f6a9c06c100 legacy=0x7f6a9c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.063 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c15dbf0 con 0x7f6a9c159950 2023-10-14T16:40:20.063 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6a8c001800 con 0x7f6a9c159950 2023-10-14T16:40:20.063 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 >> v1:172.21.15.29:6789/0 conn(0x7f6a9c159950 legacy=0x7f6a9c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.063 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a880039c0 0x7f6a88003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.064 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.064 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a880051d0 0x7f6a8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.064 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c15e490 con 0x7f6a880047c0 2023-10-14T16:40:20.064 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8800c100 con 0x7f6a880039c0 2023-10-14T16:40:20.064 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.055+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a8800c2c0 con 0x7f6a880051d0 2023-10-14T16:40:20.065 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.065 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a9a59c640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a880039c0 0x7f6a88003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.065 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 -- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a880051d0 msgr2=0x7f6a8800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.065 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a880051d0 0x7f6a8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.065 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 -- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a880039c0 msgr2=0x7f6a88003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.066 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a880039c0 0x7f6a88003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.066 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a93fff640 1 -- 172.21.15.29:0/2588109758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a88003520 con 0x7f6a880047c0 2023-10-14T16:40:20.066 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.056+0000 7f6a9ad9d640 1 -- 172.21.15.29:0/2588109758 reap_dead start 2023-10-14T16:40:20.066 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.057+0000 7f6a93fff640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a88004c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009ac0 tx=0x7f6a880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f411c881a6c2066c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.066 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.057+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8402c6d0 con 0x7f6a880047c0 2023-10-14T16:40:20.067 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8402c830 con 0x7f6a880047c0 2023-10-14T16:40:20.067 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a99d9b640 1 -- 172.21.15.29:0/2588109758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8402cbc0 con 0x7f6a880047c0 2023-10-14T16:40:20.067 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 -- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 msgr2=0x7f6a88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.067 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a88004c40 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6a84009ac0 tx=0x7f6a880057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.067 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 -- 172.21.15.29:0/2588109758 shutdown_connections 2023-10-14T16:40:20.068 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/2588109758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a88004c40 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.068 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 -- 172.21.15.29:0/2588109758 >> 172.21.15.29:0/2588109758 conn(0x7f6a9c0ce6a0 msgr2=0x7f6a9c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:20.068 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 -- 172.21.15.29:0/2588109758 shutdown_connections 2023-10-14T16:40:20.068 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.058+0000 7f6a9b59e640 1 -- 172.21.15.29:0/2588109758 wait complete. 2023-10-14T16:40:20.070 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.060+0000 7f6a9b59e640 1 Processor -- start 2023-10-14T16:40:20.071 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.060+0000 7f6a9b59e640 1 -- start start 2023-10-14T16:40:20.071 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.071 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 0x7f6a9c15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.071 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a9c0ae490 0x7f6a9c0ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.072 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c0aee50 con 0x7f6a880047c0 2023-10-14T16:40:20.072 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c15c200 con 0x7f6a9c158b70 2023-10-14T16:40:20.072 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.061+0000 7f6a9b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9c15bc40 con 0x7f6a9c0ae490 2023-10-14T16:40:20.072 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.072 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51048/0 (socket says 172.21.15.29:51048) 2023-10-14T16:40:20.073 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 -- 172.21.15.29:0/3322754615 learned_addr learned my addr 172.21.15.29:0/3322754615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:20.073 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 -- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a9c0ae490 msgr2=0x7f6a9c0ae910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.073 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9ad9d640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a9c0ae490 0x7f6a9c0ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.073 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a93fff640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 0x7f6a9c15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.073 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a9c0ae490 0x7f6a9c0ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.074 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 -- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 msgr2=0x7f6a9c15b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.074 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 0x7f6a9c15b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.074 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 -- 172.21.15.29:0/3322754615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9c15bf20 con 0x7f6a880047c0 2023-10-14T16:40:20.074 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a93fff640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 0x7f6a9c15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:20.074 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.062+0000 7f6a9a59c640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00d7a0 tx=0x7f6a8c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.075 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.063+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c0074e0 con 0x7f6a880047c0 2023-10-14T16:40:20.075 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.063+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a8c007640 con 0x7f6a880047c0 2023-10-14T16:40:20.075 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.063+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a9c0acbc0 con 0x7f6a880047c0 2023-10-14T16:40:20.075 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.063+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a8c004a00 con 0x7f6a880047c0 2023-10-14T16:40:20.075 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.063+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a9c0ad900 con 0x7f6a880047c0 2023-10-14T16:40:20.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.064+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a9c0aca20 con 0x7f6a880047c0 2023-10-14T16:40:20.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.065+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6a8c004b60 con 0x7f6a880047c0 2023-10-14T16:40:20.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.066+0000 7f6a92ffd640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 0x7f6a7c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.066+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6a8c088b50 con 0x7f6a880047c0 2023-10-14T16:40:20.076 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.066+0000 7f6a93fff640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 0x7f6a7c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.077 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.067+0000 7f6a93fff640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 0x7f6a7c06d360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036000 tx=0x7f6a84030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.077 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.075+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a9c0aca20 con 0x7f6a880047c0 2023-10-14T16:40:20.272 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.271+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a9c0ad0e0 con 0x7f6a880047c0 2023-10-14T16:40:20.273 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.272+0000 7f6a92ffd640 1 -- 172.21.15.29:0/3322754615 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f6a9c0ad0e0 con 0x7f6a880047c0 2023-10-14T16:40:20.274 INFO:teuthology.misc.health.smithi029.stdout: 2023-10-14T16:40:20.274 INFO:teuthology.misc.health.smithi029.stdout:{"epoch":16,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:16.156611+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:20.276 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 msgr2=0x7f6a7c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.276 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 0x7f6a7c06d360 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6a84036000 tx=0x7f6a84030000 comp rx=0 tx=0).stop 2023-10-14T16:40:20.276 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 msgr2=0x7f6a9c0ad3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.277 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a8c00d7a0 tx=0x7f6a8c00dc60 comp rx=0 tx=0).stop 2023-10-14T16:40:20.277 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 shutdown_connections 2023-10-14T16:40:20.277 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6a7c06aea0 0x7f6a7c06d360 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.277 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a9c0ae490 0x7f6a9c0ae910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.277 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a9c158b70 0x7f6a9c15b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.278 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 --2- 172.21.15.29:0/3322754615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a880047c0 0x7f6a9c0ad3c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.278 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 >> 172.21.15.29:0/3322754615 conn(0x7f6a9c0ce6a0 msgr2=0x7f6a9c0c9c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:20.278 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 shutdown_connections 2023-10-14T16:40:20.278 INFO:teuthology.misc.health.smithi029.stderr:2023-10-14T16:40:20.275+0000 7f6a9b59e640 1 -- 172.21.15.29:0/3322754615 wait complete. 2023-10-14T16:40:20.293 DEBUG:teuthology.misc:8 of 8 OSDs are up 2023-10-14T16:40:20.293 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-10-14T16:40:20.293 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:40:20.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:40:20.303 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:40:20.303 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:40:20.304 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:40:20.313 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:40:20.313 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:40:20.314 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:40:20.323 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:40:20.324 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:40:20.324 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:40:20.332 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:40:20.333 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2023-10-14T16:40:20.333 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:20.465 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:40:20.469 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:40:20.488 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:40:20.490 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:40:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.523+0000 7f273c5a9640 1 Processor -- start 2023-10-14T16:40:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.523+0000 7f273c5a9640 1 -- start start 2023-10-14T16:40:20.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273c5a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f27340b0b20 0x7f27341615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273c5a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f273415d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273c5a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f273415e3e0 con 0x7f27340cd0c0 2023-10-14T16:40:20.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273c5a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27340cd4a0 con 0x7f27340b0b20 2023-10-14T16:40:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273c5a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f273415e100 con 0x7f273415d3f0 2023-10-14T16:40:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273a345640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f27340cd0c0 0x7f27340b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57816/0 (socket says 172.21.15.29:57816) 2023-10-14T16:40:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273a345640 1 -- 172.21.15.29:0/2371214254 learned_addr learned my addr 172.21.15.29:0/2371214254 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f273ab46640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f273415d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.524+0000 7f2739b44640 1 --2- 172.21.15.29:0/2371214254 >> v2:172.21.15.186:3300/0 conn(0x7f27340b0b20 0x7f27341615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528836859 0 0) 0x7f273415e3e0 con 0x7f27340cd0c0 2023-10-14T16:40:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2720003520 con 0x7f27340cd0c0 2023-10-14T16:40:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3517409374 0 0) 0x7f2720003520 con 0x7f27340cd0c0 2023-10-14T16:40:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 msgr2=0x7f273415d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f273415d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 >> v2:172.21.15.186:3300/0 conn(0x7f27340b0b20 msgr2=0x7f27341615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 --2- 172.21.15.29:0/2371214254 >> v2:172.21.15.186:3300/0 conn(0x7f27340b0b20 0x7f27341615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2734160b10 con 0x7f27340cd0c0 2023-10-14T16:40:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2730001670 con 0x7f27340cd0c0 2023-10-14T16:40:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f273ab46640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f273415d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 >> v1:172.21.15.29:6789/0 conn(0x7f27340cd0c0 legacy=0x7f27340b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2720003970 0x7f2720003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f27200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720004d00 0x7f2720005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340cd4a0 con 0x7f2720004d00 2023-10-14T16:40:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273415e100 con 0x7f2720003970 2023-10-14T16:40:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f272000b980 con 0x7f273415d3f0 2023-10-14T16:40:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f2739b44640 1 --2- 172.21.15.29:0/2371214254 >> v2:172.21.15.186:3300/0 conn(0x7f27340b0b20 0x7f27341615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f273ab46640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720004d00 0x7f2720005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.525+0000 7f273a345640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2720003970 0x7f2720003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f2739b44640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f27200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273a345640 1 -- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 msgr2=0x7f27200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273a345640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273415d3f0 0x7f27200047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273a345640 1 -- 172.21.15.29:0/2371214254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720004d00 msgr2=0x7f2720005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273a345640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720004d00 0x7f2720005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273a345640 1 -- 172.21.15.29:0/2371214254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f272c009a70 con 0x7f2720003970 2023-10-14T16:40:20.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.526+0000 7f273ab46640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720004d00 0x7f2720005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.527+0000 7f273a345640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2720003970 0x7f2720003d90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f273000c240 tx=0x7f27200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d65525c186f25ac6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.527+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273002cb00 con 0x7f2720003970 2023-10-14T16:40:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.527+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f273002cc60 con 0x7f2720003970 2023-10-14T16:40:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.527+0000 7f2739343640 1 -- 172.21.15.29:0/2371214254 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730005af0 con 0x7f2720003970 2023-10-14T16:40:20.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 -- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2720003970 msgr2=0x7f2720003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 --2- 172.21.15.29:0/2371214254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2720003970 0x7f2720003d90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f273000c240 tx=0x7f27200057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273ab46640 1 -- 172.21.15.29:0/2371214254 reap_dead start 2023-10-14T16:40:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 -- 172.21.15.29:0/2371214254 shutdown_connections 2023-10-14T16:40:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 -- 172.21.15.29:0/2371214254 >> 172.21.15.29:0/2371214254 conn(0x7f27340cb410 msgr2=0x7f27340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 -- 172.21.15.29:0/2371214254 shutdown_connections 2023-10-14T16:40:20.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.528+0000 7f273c5a9640 1 -- 172.21.15.29:0/2371214254 wait complete. 2023-10-14T16:40:20.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.530+0000 7f273c5a9640 1 Processor -- start 2023-10-14T16:40:20.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.530+0000 7f273c5a9640 1 -- start start 2023-10-14T16:40:20.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2734156e90 0x7f273406f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 0x7f27340b1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f273406fe10 con 0x7f2720003970 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27340c8860 con 0x7f2734156e90 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273c5a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27340b2920 con 0x7f273406f990 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273a345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273a345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51104/0 (socket says 172.21.15.29:51104) 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273a345640 1 -- 172.21.15.29:0/638747970 learned_addr learned my addr 172.21.15.29:0/638747970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:20.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f2739b44640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2734156e90 0x7f273406f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.531+0000 7f273ab46640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 0x7f27340b1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 -- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 msgr2=0x7f27340b1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 0x7f27340b1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 -- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2734156e90 msgr2=0x7f273406f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2734156e90 0x7f273406f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 -- 172.21.15.29:0/638747970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27340b2c00 con 0x7f2720003970 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273ab46640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 0x7f27340b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f273a345640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f273000c240 tx=0x7f2730001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2730039070 con 0x7f2720003970 2023-10-14T16:40:20.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2730005c80 con 0x7f2720003970 2023-10-14T16:40:20.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.532+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f273003c040 con 0x7f2720003970 2023-10-14T16:40:20.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.533+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27340b2ee0 con 0x7f2720003970 2023-10-14T16:40:20.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.533+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27340b34a0 con 0x7f2720003970 2023-10-14T16:40:20.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.534+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27341f28d0 con 0x7f2720003970 2023-10-14T16:40:20.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.535+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2730032070 con 0x7f2720003970 2023-10-14T16:40:20.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.536+0000 7f272affd640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 0x7f271806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.536+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f27300aa800 con 0x7f2720003970 2023-10-14T16:40:20.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.537+0000 7f2739b44640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 0x7f271806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.537+0000 7f2739b44640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 0x7f271806d330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f273406c1b0 tx=0x7f2724008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.541+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2730075c10 con 0x7f2720003970 2023-10-14T16:40:20.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.739+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f273404e160 con 0x7f2720003970 2023-10-14T16:40:20.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.741+0000 7f272affd640 1 -- 172.21.15.29:0/638747970 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f2730046020 con 0x7f2720003970 2023-10-14T16:40:20.742 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:20.742 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:16.156611+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 msgr2=0x7f271806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 0x7f271806d330 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f273406c1b0 tx=0x7f2724008040 comp rx=0 tx=0).stop 2023-10-14T16:40:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 msgr2=0x7f27340c8320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f273000c240 tx=0x7f2730001a00 comp rx=0 tx=0).stop 2023-10-14T16:40:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 shutdown_connections 2023-10-14T16:40:20.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f271806ae70 0x7f271806d330 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f273406f990 0x7f27340b1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2734156e90 0x7f273406f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 --2- 172.21.15.29:0/638747970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2720003970 0x7f27340c8320 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.743+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 >> 172.21.15.29:0/638747970 conn(0x7f27340cb410 msgr2=0x7f273406cf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:20.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.744+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 shutdown_connections 2023-10-14T16:40:20.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.744+0000 7f273c5a9640 1 -- 172.21.15.29:0/638747970 wait complete. 2023-10-14T16:40:20.761 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:40:20.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:40:20.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.982+0000 7fbe11101640 1 Processor -- start 2023-10-14T16:40:20.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.983+0000 7fbe11101640 1 -- start start 2023-10-14T16:40:20.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe11101640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe0c0ad350 0x7fbe0c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe11101640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c06a2e0 0x7fbe0c0c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe11101640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe0c06a730 con 0x7fbe0c15d150 2023-10-14T16:40:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe11101640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe0c15d530 con 0x7fbe0c0ad350 2023-10-14T16:40:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe11101640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0ac890 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe03fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe0c0ad350 0x7fbe0c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.984+0000 7fbe03fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe0c0ad350 0x7fbe0c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60662/0 (socket says 172.21.15.29:60662) 2023-10-14T16:40:20.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe03fff640 1 -- 172.21.15.29:0/1019464174 learned_addr learned my addr 172.21.15.29:0/1019464174 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:20.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe0b59e640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c06a2e0 0x7fbe0c0c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe03fff640 1 -- 172.21.15.29:0/1019464174 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c06a2e0 msgr2=0x7fbe0c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c06a2e0 0x7fbe0c0c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe03fff640 1 -- 172.21.15.29:0/1019464174 >> v1:172.21.15.186:6789/0 conn(0x7fbe0c15d150 legacy=0x7fbe0c160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.985+0000 7fbe03fff640 1 -- 172.21.15.29:0/1019464174 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe0c0ad7a0 con 0x7fbe0c0ad350 2023-10-14T16:40:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.986+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c069da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0cf990 tx=0x7fbdf402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bf511d57cf7c68b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.986+0000 7fbe0a59c640 1 -- 172.21.15.29:0/1019464174 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4031830 con 0x7fbe0c0ad350 2023-10-14T16:40:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.986+0000 7fbe0a59c640 1 -- 172.21.15.29:0/1019464174 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdf4031990 con 0x7fbe0c0ad350 2023-10-14T16:40:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.986+0000 7fbe0a59c640 1 -- 172.21.15.29:0/1019464174 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdf4031d20 con 0x7fbe0c0ad350 2023-10-14T16:40:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 -- 172.21.15.29:0/1019464174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c0ad350 msgr2=0x7fbe0c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c069da0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c0cf990 tx=0x7fbdf402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 -- 172.21.15.29:0/1019464174 shutdown_connections 2023-10-14T16:40:20.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c06a2e0 0x7fbe0c0c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 --2- 172.21.15.29:0/1019464174 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c069da0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 -- 172.21.15.29:0/1019464174 >> 172.21.15.29:0/1019464174 conn(0x7fbe0c0ce9b0 msgr2=0x7fbe0c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:20.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 -- 172.21.15.29:0/1019464174 shutdown_connections 2023-10-14T16:40:20.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.987+0000 7fbe11101640 1 -- 172.21.15.29:0/1019464174 wait complete. 2023-10-14T16:40:20.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.989+0000 7fbe11101640 1 Processor -- start 2023-10-14T16:40:20.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.989+0000 7fbe11101640 1 -- start start 2023-10-14T16:40:20.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c15d150 0x7fbe0c1f2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0b2d80 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c0b0730 con 0x7fbe0c0ad350 2023-10-14T16:40:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe11101640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0c1f35f0 con 0x7fbe0c15d150 2023-10-14T16:40:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60666/0 (socket says 172.21.15.29:60666) 2023-10-14T16:40:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 -- 172.21.15.29:0/1945530194 learned_addr learned my addr 172.21.15.29:0/1945530194 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:20.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0b59e640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c15d150 0x7fbe0c1f2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 -- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c15d150 msgr2=0x7fbe0c1f2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c15d150 0x7fbe0c1f2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 -- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 msgr2=0x7fbe0c1f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:20.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c1f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:20.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe0ad9d640 1 -- 172.21.15.29:0/1945530194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf40094d0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.990+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c1f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:20.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe0ad9d640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00c430 tx=0x7fbdfc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:20.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc011790 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdfc0118f0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbdfc011ce0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe0c1f38d0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.991+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe0c0b29a0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.992+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0c0b2ec0 con 0x7fbe0c06a2e0 2023-10-14T16:40:20.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.993+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbdfc01c460 con 0x7fbe0c06a2e0 2023-10-14T16:40:21.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.994+0000 7fbe037fe640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 0x7fbdec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.994+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fbdfc089210 con 0x7fbe0c06a2e0 2023-10-14T16:40:21.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.994+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 0x7fbdec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.995+0000 7fbe03fff640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 0x7fbdec06d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4036000 tx=0x7fbdf403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:20.999+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbdfc054580 con 0x7fbe0c06a2e0 2023-10-14T16:40:21.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.187+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe0c04e160 con 0x7fbe0c06a2e0 2023-10-14T16:40:21.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.188+0000 7fbe037fe640 1 -- 172.21.15.29:0/1945530194 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbdfc057ba0 con 0x7fbe0c06a2e0 2023-10-14T16:40:21.199 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:40:21.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 msgr2=0x7fbdec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 0x7fbdec06d330 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4036000 tx=0x7fbdf403c040 comp rx=0 tx=0).stop 2023-10-14T16:40:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 msgr2=0x7fbe0c0ae4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbdfc00c430 tx=0x7fbdfc00c8f0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 shutdown_connections 2023-10-14T16:40:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdec06ae70 0x7fbdec06d330 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe0c15d150 0x7fbe0c1f2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe0c0ad350 0x7fbe0c1f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 --2- 172.21.15.29:0/1945530194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe0c06a2e0 0x7fbe0c0ae4b0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 >> 172.21.15.29:0/1945530194 conn(0x7fbe0c0ce9b0 msgr2=0x7fbe0c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:21.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 shutdown_connections 2023-10-14T16:40:21.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.190+0000 7fbe11101640 1 -- 172.21.15.29:0/1945530194 wait complete. 2023-10-14T16:40:21.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:40:21.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.425+0000 7f4317fff640 1 Processor -- start 2023-10-14T16:40:21.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.426+0000 7f4317fff640 1 -- start start 2023-10-14T16:40:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.427+0000 7f4317fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4318159950 0x7f431815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.427+0000 7f4317fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c810 0x7f43180708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f4317fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431815aef0 con 0x7f4318070450 2023-10-14T16:40:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f4317fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4318070e20 con 0x7f431806c810 2023-10-14T16:40:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f4317fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431815e490 con 0x7f4318159950 2023-10-14T16:40:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c810 0x7f43180708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f431806c810 0x7f43180708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60672/0 (socket says 172.21.15.29:60672) 2023-10-14T16:40:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 -- 172.21.15.29:0/2318816289 learned_addr learned my addr 172.21.15.29:0/2318816289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f430ffff640 1 --1- 172.21.15.29:0/2318816289 >> v1:172.21.15.29:6789/0 conn(0x7f4318070450 0x7f431806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41986/0 (socket says 172.21.15.29:41986) 2023-10-14T16:40:21.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f4316ffd640 1 --2- 172.21.15.29:0/2318816289 >> v2:172.21.15.186:3300/0 conn(0x7f4318159950 0x7f431815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 -- 172.21.15.29:0/2318816289 >> v2:172.21.15.186:3300/0 conn(0x7f4318159950 msgr2=0x7f431815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:21.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 --2- 172.21.15.29:0/2318816289 >> v2:172.21.15.186:3300/0 conn(0x7f4318159950 0x7f431815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 -- 172.21.15.29:0/2318816289 >> v1:172.21.15.29:6789/0 conn(0x7f4318070450 legacy=0x7f431806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.428+0000 7f43177fe640 1 -- 172.21.15.29:0/2318816289 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431815d910 con 0x7f431806c810 2023-10-14T16:40:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f43177fe640 1 --2- 172.21.15.29:0/2318816289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f43180708e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43180ca950 tx=0x7f431002e0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21f26325d8cd7f75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f43167fc640 1 -- 172.21.15.29:0/2318816289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f431002ea90 con 0x7f431806c810 2023-10-14T16:40:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f43167fc640 1 -- 172.21.15.29:0/2318816289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431002ebf0 con 0x7f431806c810 2023-10-14T16:40:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f43167fc640 1 -- 172.21.15.29:0/2318816289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310009600 con 0x7f431806c810 2023-10-14T16:40:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f4317fff640 1 -- 172.21.15.29:0/2318816289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 msgr2=0x7f43180708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f4317fff640 1 --2- 172.21.15.29:0/2318816289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f43180708e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43180ca950 tx=0x7f431002e0f0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f4317fff640 1 -- 172.21.15.29:0/2318816289 shutdown_connections 2023-10-14T16:40:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.430+0000 7f4317fff640 1 --2- 172.21.15.29:0/2318816289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f43180708e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.431+0000 7f4317fff640 1 --2- 172.21.15.29:0/2318816289 >> v2:172.21.15.186:3300/0 conn(0x7f4318159950 0x7f431815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.431+0000 7f4317fff640 1 -- 172.21.15.29:0/2318816289 >> 172.21.15.29:0/2318816289 conn(0x7f43180ce6a0 msgr2=0x7f43180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.431+0000 7f4317fff640 1 -- 172.21.15.29:0/2318816289 shutdown_connections 2023-10-14T16:40:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.431+0000 7f4317fff640 1 -- 172.21.15.29:0/2318816289 wait complete. 2023-10-14T16:40:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.432+0000 7f4317fff640 1 Processor -- start 2023-10-14T16:40:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.433+0000 7f4317fff640 1 -- start start 2023-10-14T16:40:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.433+0000 7f4317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.433+0000 7f4317fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4318070450 0x7f43180c7740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.433+0000 7f4317fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4318159950 0x7f431815c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.433+0000 7f4317fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180c2800 con 0x7f431806c810 2023-10-14T16:40:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4317fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180c8190 con 0x7f4318070450 2023-10-14T16:40:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4317fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180c8470 con 0x7f4318159950 2023-10-14T16:40:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60682/0 (socket says 172.21.15.29:60682) 2023-10-14T16:40:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 -- 172.21.15.29:0/3398310786 learned_addr learned my addr 172.21.15.29:0/3398310786 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 -- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4318159950 msgr2=0x7f431815c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4318159950 0x7f431815c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 -- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4318070450 msgr2=0x7f43180c7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4318070450 0x7f43180c7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 -- 172.21.15.29:0/3398310786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431000bd40 con 0x7f431806c810 2023-10-14T16:40:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.434+0000 7f4316ffd640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f430800d800 tx=0x7f430800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.435+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800b6c0 con 0x7f431806c810 2023-10-14T16:40:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.435+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430800b820 con 0x7f431806c810 2023-10-14T16:40:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.435+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f430800bbb0 con 0x7f431806c810 2023-10-14T16:40:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.435+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43180c21b0 con 0x7f431806c810 2023-10-14T16:40:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.435+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431815d150 con 0x7f431806c810 2023-10-14T16:40:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.436+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431804e9b0 con 0x7f431806c810 2023-10-14T16:40:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.437+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f430800bd10 con 0x7f431806c810 2023-10-14T16:40:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.439+0000 7f430f7fe640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 0x7f42f806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.439+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f43080898c0 con 0x7f431806c810 2023-10-14T16:40:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.439+0000 7f430ffff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 0x7f42f806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.440+0000 7f430ffff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 0x7f42f806d330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43180ad6a0 tx=0x7f43000079c0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.443+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4308054c20 con 0x7f431806c810 2023-10-14T16:40:21.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.682+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43180b4620 con 0x7f431806c810 2023-10-14T16:40:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.683+0000 7f430f7fe640 1 -- 172.21.15.29:0/3398310786 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+3700 (secure 0 0 0) 0x7f4308058240 con 0x7f431806c810 2023-10-14T16:40:21.685 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:21.685 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4272,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2023-10-14T16:40:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 3 2023-10-14T16:40:21.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 msgr2=0x7f42f806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 0x7f42f806d330 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f43180ad6a0 tx=0x7f43000079c0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 msgr2=0x7f4318155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f430800d800 tx=0x7f430800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 shutdown_connections 2023-10-14T16:40:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f42f806ae70 0x7f42f806d330 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4318159950 0x7f431815c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4318070450 0x7f43180c7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 --2- 172.21.15.29:0/3398310786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f431806c810 0x7f4318155e40 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 >> 172.21.15.29:0/3398310786 conn(0x7f43180ce6a0 msgr2=0x7f43180cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.686+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 shutdown_connections 2023-10-14T16:40:21.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.687+0000 7f4317fff640 1 -- 172.21.15.29:0/3398310786 wait complete. 2023-10-14T16:40:21.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:21.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.937+0000 7f6e9c947640 1 Processor -- start 2023-10-14T16:40:21.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.937+0000 7f6e9c947640 1 -- start start 2023-10-14T16:40:21.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e9c947640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e980ca0a0 0x7f6e9806c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e9c947640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9815d910 con 0x7f6e98159950 2023-10-14T16:40:21.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e9c947640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e9815dbf0 con 0x7f6e98070b60 2023-10-14T16:40:21.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e9c947640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e9815e490 con 0x7f6e980ca0a0 2023-10-14T16:40:21.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e980ca0a0 0x7f6e9806c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e97fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e980ca0a0 0x7f6e9806c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60694/0 (socket says 172.21.15.29:60694) 2023-10-14T16:40:21.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e97fff640 1 -- 172.21.15.29:0/3232246791 learned_addr learned my addr 172.21.15.29:0/3232246791 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:21.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.939+0000 7f6e97fff640 1 -- 172.21.15.29:0/3232246791 >> v1:172.21.15.186:6789/0 conn(0x7f6e98070b60 legacy=0x7f6e9806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:21.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.940+0000 7f6e97fff640 1 -- 172.21.15.29:0/3232246791 >> v1:172.21.15.29:6789/0 conn(0x7f6e98159950 legacy=0x7f6e98070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.940+0000 7f6e97fff640 1 -- 172.21.15.29:0/3232246791 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e9815ded0 con 0x7f6e980ca0a0 2023-10-14T16:40:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.941+0000 7f6e97fff640 1 --2- 172.21.15.29:0/3232246791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806c810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009870 tx=0x7f6e9002db20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1ea0f34b37d485 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.941+0000 7f6e967fc640 1 -- 172.21.15.29:0/3232246791 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9002f8e0 con 0x7f6e980ca0a0 2023-10-14T16:40:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.941+0000 7f6e967fc640 1 -- 172.21.15.29:0/3232246791 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e9002fa40 con 0x7f6e980ca0a0 2023-10-14T16:40:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.941+0000 7f6e967fc640 1 -- 172.21.15.29:0/3232246791 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e9002fdd0 con 0x7f6e980ca0a0 2023-10-14T16:40:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 -- 172.21.15.29:0/3232246791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e980ca0a0 msgr2=0x7f6e9806c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 --2- 172.21.15.29:0/3232246791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806c810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009870 tx=0x7f6e9002db20 comp rx=0 tx=0).stop 2023-10-14T16:40:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 -- 172.21.15.29:0/3232246791 shutdown_connections 2023-10-14T16:40:21.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 --2- 172.21.15.29:0/3232246791 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806c810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 -- 172.21.15.29:0/3232246791 >> 172.21.15.29:0/3232246791 conn(0x7f6e980ce6a0 msgr2=0x7f6e980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 -- 172.21.15.29:0/3232246791 shutdown_connections 2023-10-14T16:40:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.942+0000 7f6e9c947640 1 -- 172.21.15.29:0/3232246791 wait complete. 2023-10-14T16:40:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.943+0000 7f6e9c947640 1 Processor -- start 2023-10-14T16:40:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.944+0000 7f6e9c947640 1 -- start start 2023-10-14T16:40:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.944+0000 7f6e9c947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e98070b60 0x7f6e980c5d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e9c947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e9c947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e9c947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e981574d0 con 0x7f6e98159950 2023-10-14T16:40:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e9c947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e98158010 con 0x7f6e980ca0a0 2023-10-14T16:40:21.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e9c947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e981576e0 con 0x7f6e98070b60 2023-10-14T16:40:21.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60700/0 (socket says 172.21.15.29:60700) 2023-10-14T16:40:21.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 -- 172.21.15.29:0/739486343 learned_addr learned my addr 172.21.15.29:0/739486343 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:21.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e96ffd640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e977fe640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e98070b60 0x7f6e980c5d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 -- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e98070b60 msgr2=0x7f6e980c5d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e98070b60 0x7f6e980c5d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 -- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 msgr2=0x7f6e9806f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e97fff640 1 -- 172.21.15.29:0/739486343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e900094d0 con 0x7f6e98159950 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.945+0000 7f6e96ffd640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e97fff640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009870 tx=0x7f6e900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90037d70 con 0x7f6e98159950 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e90042470 con 0x7f6e98159950 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e90042820 con 0x7f6e98159950 2023-10-14T16:40:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e981579c0 con 0x7f6e98159950 2023-10-14T16:40:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.946+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e981584a0 con 0x7f6e98159950 2023-10-14T16:40:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.947+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98158180 con 0x7f6e98159950 2023-10-14T16:40:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.948+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6e90042980 con 0x7f6e98159950 2023-10-14T16:40:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.949+0000 7f6e7ffff640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 0x7f6e8406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.949+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f6e90033070 con 0x7f6e98159950 2023-10-14T16:40:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.950+0000 7f6e977fe640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 0x7f6e8406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.950+0000 7f6e977fe640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 0x7f6e8406d330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e88007fd0 tx=0x7f6e88009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:21.954+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e90077960 con 0x7f6e98159950 2023-10-14T16:40:22.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.154+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e9804e9b0 con 0x7f6e98159950 2023-10-14T16:40:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.156+0000 7f6e7ffff640 1 -- 172.21.15.29:0/739486343 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+12838 (secure 0 0 0) 0x7f6e90046090 con 0x7f6e98159950 2023-10-14T16:40:22.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:22.157 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":16,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:16.156611+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 msgr2=0x7f6e8406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 0x7f6e8406d330 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6e88007fd0 tx=0x7f6e88009040 comp rx=0 tx=0).stop 2023-10-14T16:40:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 msgr2=0x7f6e9806f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6e90009870 tx=0x7f6e900046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 shutdown_connections 2023-10-14T16:40:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e8406ae70 0x7f6e8406d330 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e98159950 0x7f6e9806f730 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e980ca0a0 0x7f6e9806f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 --2- 172.21.15.29:0/739486343 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e98070b60 0x7f6e980c5d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.158+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 >> 172.21.15.29:0/739486343 conn(0x7f6e980ce6a0 msgr2=0x7f6e98070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:22.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.159+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 shutdown_connections 2023-10-14T16:40:22.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.159+0000 7f6e9c947640 1 -- 172.21.15.29:0/739486343 wait complete. 2023-10-14T16:40:22.180 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:40:22.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:40:22.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.404+0000 7fda12f87640 1 Processor -- start 2023-10-14T16:40:22.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.404+0000 7fda12f87640 1 -- start start 2023-10-14T16:40:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12f87640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fda0c070380 0x7fda0c0707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12f87640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda0c0ca680 0x7fda0c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12f87640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda0c15af90 con 0x7fda0c15abb0 2023-10-14T16:40:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12f87640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c070d30 con 0x7fda0c0ca680 2023-10-14T16:40:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12f87640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda0c0b3010 con 0x7fda0c070380 2023-10-14T16:40:22.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12786640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda0c0ca680 0x7fda0c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda11784640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fda0c070380 0x7fda0c0707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12786640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda0c0ca680 0x7fda0c0caaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60704/0 (socket says 172.21.15.29:60704) 2023-10-14T16:40:22.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.406+0000 7fda12786640 1 -- 172.21.15.29:0/4222297981 learned_addr learned my addr 172.21.15.29:0/4222297981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:22.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.407+0000 7fda12786640 1 -- 172.21.15.29:0/4222297981 >> v2:172.21.15.186:3300/0 conn(0x7fda0c070380 msgr2=0x7fda0c0707f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.407+0000 7fda12786640 1 --2- 172.21.15.29:0/4222297981 >> v2:172.21.15.186:3300/0 conn(0x7fda0c070380 0x7fda0c0707f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.407+0000 7fda12786640 1 -- 172.21.15.29:0/4222297981 >> v1:172.21.15.186:6789/0 conn(0x7fda0c15abb0 legacy=0x7fda0c0b25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.407+0000 7fda12786640 1 -- 172.21.15.29:0/4222297981 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda0c0b32f0 con 0x7fda0c0ca680 2023-10-14T16:40:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.408+0000 7fda12786640 1 --2- 172.21.15.29:0/4222297981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c0ca680 0x7fda0c0caaf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda0c15e590 tx=0x7fda0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ce0d5320616e7c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.408+0000 7fda10f83640 1 -- 172.21.15.29:0/4222297981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04030860 con 0x7fda0c0ca680 2023-10-14T16:40:22.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda10f83640 1 -- 172.21.15.29:0/4222297981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda040309c0 con 0x7fda0c0ca680 2023-10-14T16:40:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda10f83640 1 -- 172.21.15.29:0/4222297981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04030d50 con 0x7fda0c0ca680 2023-10-14T16:40:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 -- 172.21.15.29:0/4222297981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c0ca680 msgr2=0x7fda0c0caaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 --2- 172.21.15.29:0/4222297981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c0ca680 0x7fda0c0caaf0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fda0c15e590 tx=0x7fda0402b9a0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 -- 172.21.15.29:0/4222297981 shutdown_connections 2023-10-14T16:40:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 --2- 172.21.15.29:0/4222297981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c0ca680 0x7fda0c0caaf0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 --2- 172.21.15.29:0/4222297981 >> v2:172.21.15.186:3300/0 conn(0x7fda0c070380 0x7fda0c0707f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 -- 172.21.15.29:0/4222297981 >> 172.21.15.29:0/4222297981 conn(0x7fda0c0cf790 msgr2=0x7fda0c0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 -- 172.21.15.29:0/4222297981 shutdown_connections 2023-10-14T16:40:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.409+0000 7fda12f87640 1 -- 172.21.15.29:0/4222297981 wait complete. 2023-10-14T16:40:22.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.411+0000 7fda12f87640 1 Processor -- start 2023-10-14T16:40:22.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.411+0000 7fda12f87640 1 -- start start 2023-10-14T16:40:22.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 0x7fda0c1f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda0c0ca680 0x7fda0c1f5870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c1f6150 con 0x7fda0c15abb0 2023-10-14T16:40:22.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c1f6430 con 0x7fda0c0ca680 2023-10-14T16:40:22.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.412+0000 7fda12f87640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda0c0ac930 con 0x7fda0c070380 2023-10-14T16:40:22.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60712/0 (socket says 172.21.15.29:60712) 2023-10-14T16:40:22.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 -- 172.21.15.29:0/1883002153 learned_addr learned my addr 172.21.15.29:0/1883002153 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:22.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda11f85640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 0x7fda0c1f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 -- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 msgr2=0x7fda0c1f1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda11784640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda0c0ca680 0x7fda0c1f5870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 0x7fda0c1f1dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 -- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda0c0ca680 msgr2=0x7fda0c1f5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda0c0ca680 0x7fda0c1f5870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 -- 172.21.15.29:0/1883002153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda040094d0 con 0x7fda0c15abb0 2023-10-14T16:40:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda11f85640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 0x7fda0c1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:22.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.413+0000 7fda12786640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda0402b9e0 tx=0x7fda04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.414+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04039c90 con 0x7fda0c15abb0 2023-10-14T16:40:22.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.414+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda04039e10 con 0x7fda0c15abb0 2023-10-14T16:40:22.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.414+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda04043640 con 0x7fda0c15abb0 2023-10-14T16:40:22.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.414+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda0c0acc10 con 0x7fda0c15abb0 2023-10-14T16:40:22.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.414+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda0c1f5db0 con 0x7fda0c15abb0 2023-10-14T16:40:22.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.416+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda0c04e9b0 con 0x7fda0c15abb0 2023-10-14T16:40:22.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.416+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fda040437a0 con 0x7fda0c15abb0 2023-10-14T16:40:22.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.417+0000 7fda0a7fc640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 0x7fd9f806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.417+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fda0403e070 con 0x7fda0c15abb0 2023-10-14T16:40:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.418+0000 7fda11f85640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 0x7fd9f806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.418+0000 7fda11f85640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 0x7fd9f806d360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc0012e0 tx=0x7fd9fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.422+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda04077730 con 0x7fda0c15abb0 2023-10-14T16:40:22.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.619+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fda0c0715b0 con 0x7fda0c15abb0 2023-10-14T16:40:22.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.620+0000 7fda0a7fc640 1 -- 172.21.15.29:0/1883002153 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) v1 ==== 93+0+10 (secure 0 0 0) 0x7fda0407ad50 con 0x7fda0c15abb0 2023-10-14T16:40:22.621 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:40:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.622+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 msgr2=0x7fd9f806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.622+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 0x7fd9f806d360 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd9fc0012e0 tx=0x7fd9fc009040 comp rx=0 tx=0).stop 2023-10-14T16:40:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.622+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 msgr2=0x7fda0c146700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.622+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fda0402b9e0 tx=0x7fda04004730 comp rx=0 tx=0).stop 2023-10-14T16:40:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 shutdown_connections 2023-10-14T16:40:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd9f806aea0 0x7fd9f806d360 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda0c15abb0 0x7fda0c146700 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda0c0ca680 0x7fda0c1f5870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 --2- 172.21.15.29:0/1883002153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda0c070380 0x7fda0c1f1dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 >> 172.21.15.29:0/1883002153 conn(0x7fda0c0cf790 msgr2=0x7fda0c0c8130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:22.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 shutdown_connections 2023-10-14T16:40:22.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.623+0000 7fda12f87640 1 -- 172.21.15.29:0/1883002153 wait complete. 2023-10-14T16:40:22.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:40:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.865+0000 7f70b23c5640 1 Processor -- start 2023-10-14T16:40:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.865+0000 7f70b23c5640 1 -- start start 2023-10-14T16:40:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b23c5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac159b20 0x7f70ac06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b23c5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70ac0b0b20 0x7f70ac06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b23c5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70ac06f7f0 con 0x7f70ac159f00 2023-10-14T16:40:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b23c5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70ac06c5e0 con 0x7f70ac0b0b20 2023-10-14T16:40:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b23c5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac06c8c0 con 0x7f70ac159b20 2023-10-14T16:40:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b0962640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70ac0b0b20 0x7f70ac06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b0962640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70ac0b0b20 0x7f70ac06fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60724/0 (socket says 172.21.15.29:60724) 2023-10-14T16:40:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.867+0000 7f70b0962640 1 -- 172.21.15.29:0/2199519086 learned_addr learned my addr 172.21.15.29:0/2199519086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.868+0000 7f70abfff640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac159b20 0x7f70ac06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.868+0000 7f70b0962640 1 -- 172.21.15.29:0/2199519086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac159b20 msgr2=0x7f70ac06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.868+0000 7f70b0962640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac159b20 0x7f70ac06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.868+0000 7f70b0962640 1 -- 172.21.15.29:0/2199519086 >> v1:172.21.15.186:6789/0 conn(0x7f70ac159f00 legacy=0x7f70ac0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.868+0000 7f70b0962640 1 -- 172.21.15.29:0/2199519086 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70ac06cba0 con 0x7f70ac0b0b20 2023-10-14T16:40:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.869+0000 7f70b0962640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac0b0b20 0x7f70ac06fc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009e50 tx=0x7f70a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dce907681a2a10cc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.869+0000 7f70aaffd640 1 -- 172.21.15.29:0/2199519086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a402ca70 con 0x7f70ac0b0b20 2023-10-14T16:40:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.869+0000 7f70aaffd640 1 -- 172.21.15.29:0/2199519086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a402cbd0 con 0x7f70ac0b0b20 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.869+0000 7f70aaffd640 1 -- 172.21.15.29:0/2199519086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a40343f0 con 0x7f70ac0b0b20 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 -- 172.21.15.29:0/2199519086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac0b0b20 msgr2=0x7f70ac06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac0b0b20 0x7f70ac06fc80 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f70a4009e50 tx=0x7f70a402bed0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 -- 172.21.15.29:0/2199519086 shutdown_connections 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac0b0b20 0x7f70ac06fc80 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 --2- 172.21.15.29:0/2199519086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac159b20 0x7f70ac06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 -- 172.21.15.29:0/2199519086 >> 172.21.15.29:0/2199519086 conn(0x7f70ac0cb410 msgr2=0x7f70ac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 -- 172.21.15.29:0/2199519086 shutdown_connections 2023-10-14T16:40:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.870+0000 7f70b23c5640 1 -- 172.21.15.29:0/2199519086 wait complete. 2023-10-14T16:40:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.873+0000 7f70b23c5640 1 Processor -- start 2023-10-14T16:40:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.873+0000 7f70b23c5640 1 -- start start 2023-10-14T16:40:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac0b0b20 0x7f70ac155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac159f00 0x7f70ac15c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0b28d0 con 0x7f70ac159b20 2023-10-14T16:40:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0b2bb0 con 0x7f70ac159f00 2023-10-14T16:40:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70b23c5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70ac0b2e90 con 0x7f70ac0b0b20 2023-10-14T16:40:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.874+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60728/0 (socket says 172.21.15.29:60728) 2023-10-14T16:40:22.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 -- 172.21.15.29:0/729063089 learned_addr learned my addr 172.21.15.29:0/729063089 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:22.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70abfff640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac0b0b20 0x7f70ac155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70b0962640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac159f00 0x7f70ac15c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 -- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac0b0b20 msgr2=0x7f70ac155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac0b0b20 0x7f70ac155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 -- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac159f00 msgr2=0x7f70ac15c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac159f00 0x7f70ac15c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 -- 172.21.15.29:0/729063089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70a4009b30 con 0x7f70ac159b20 2023-10-14T16:40:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.875+0000 7f70ab7fe640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a000c400 tx=0x7f70a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.876+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a0011790 con 0x7f70ac159b20 2023-10-14T16:40:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.876+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70a0011930 con 0x7f70ac159b20 2023-10-14T16:40:22.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.876+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70ac0b3170 con 0x7f70ac159b20 2023-10-14T16:40:22.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.876+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70a0011ce0 con 0x7f70ac159b20 2023-10-14T16:40:22.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.876+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70ac155bb0 con 0x7f70ac159b20 2023-10-14T16:40:22.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.877+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70ac04e160 con 0x7f70ac159b20 2023-10-14T16:40:22.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.878+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f70a001c460 con 0x7f70ac159b20 2023-10-14T16:40:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.879+0000 7f70a8ff9640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 0x7f709806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.879+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f70a0089230 con 0x7f70ac159b20 2023-10-14T16:40:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.879+0000 7f70abfff640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 0x7f709806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.880+0000 7f70abfff640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 0x7f709806d330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f709c006fd0 tx=0x7f709c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:22.883+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70a0054630 con 0x7f70ac159b20 2023-10-14T16:40:23.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.082+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70ac0b10b0 con 0x7f70ac159b20 2023-10-14T16:40:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.083+0000 7f70a8ff9640 1 -- 172.21.15.29:0/729063089 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) v1 ==== 84+0+65 (secure 0 0 0) 0x7f70a0057c50 con 0x7f70ac159b20 2023-10-14T16:40:23.084 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:23.085 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:40:23.085 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:40:23.085 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:40:23.085 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:40:23.085 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:40:23.086 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:40:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.085+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 msgr2=0x7f709806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.085+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 0x7f709806d330 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f709c006fd0 tx=0x7f709c008040 comp rx=0 tx=0).stop 2023-10-14T16:40:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 msgr2=0x7f70ac0c4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f70a000c400 tx=0x7f70a000c8c0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 shutdown_connections 2023-10-14T16:40:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f709806ae70 0x7f709806d330 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70ac159f00 0x7f70ac15c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70ac159b20 0x7f70ac0c4db0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 --2- 172.21.15.29:0/729063089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70ac0b0b20 0x7f70ac155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 >> 172.21.15.29:0/729063089 conn(0x7f70ac0cb410 msgr2=0x7f70ac0ccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 shutdown_connections 2023-10-14T16:40:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.086+0000 7f70b23c5640 1 -- 172.21.15.29:0/729063089 wait complete. 2023-10-14T16:40:23.105 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:40:23.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:40:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.338+0000 7fe60ffff640 1 Processor -- start 2023-10-14T16:40:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.338+0000 7fe60ffff640 1 -- start start 2023-10-14T16:40:23.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.339+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe61015ad80 0x7fe61006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6100ca310 0x7fe61006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6100c8e70 con 0x7fe6100cae40 2023-10-14T16:40:23.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6100b4170 con 0x7fe6100ca310 2023-10-14T16:40:23.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100adde0 con 0x7fe61015ad80 2023-10-14T16:40:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6100ca310 0x7fe61006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6100ca310 0x7fe61006d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:38308/0 (socket says 172.21.15.29:38308) 2023-10-14T16:40:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60f7fe640 1 -- 172.21.15.29:0/2988309996 learned_addr learned my addr 172.21.15.29:0/2988309996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.340+0000 7fe60effd640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe61015ad80 0x7fe61006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.341+0000 7fe60f7fe640 1 -- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe61015ad80 msgr2=0x7fe61006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.341+0000 7fe60f7fe640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe61015ad80 0x7fe61006cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.341+0000 7fe60f7fe640 1 -- 172.21.15.29:0/2988309996 >> v1:172.21.15.186:6789/0 conn(0x7fe6100cae40 legacy=0x7fe61006b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.341+0000 7fe60f7fe640 1 -- 172.21.15.29:0/2988309996 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6100b4410 con 0x7fe6100ca310 2023-10-14T16:40:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.342+0000 7fe60f7fe640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100ca310 0x7fe61006d0b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe608009e50 tx=0x7fe60802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=61f2972f737e6d4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60dffb640 1 -- 172.21.15.29:0/2988309996 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60802cb00 con 0x7fe6100ca310 2023-10-14T16:40:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60dffb640 1 -- 172.21.15.29:0/2988309996 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60802cc60 con 0x7fe6100ca310 2023-10-14T16:40:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60dffb640 1 -- 172.21.15.29:0/2988309996 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6080343f0 con 0x7fe6100ca310 2023-10-14T16:40:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 -- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100ca310 msgr2=0x7fe61006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100ca310 0x7fe61006d0b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe608009e50 tx=0x7fe60802bf60 comp rx=0 tx=0).stop 2023-10-14T16:40:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 -- 172.21.15.29:0/2988309996 shutdown_connections 2023-10-14T16:40:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100ca310 0x7fe61006d0b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 --2- 172.21.15.29:0/2988309996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe61015ad80 0x7fe61006cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.343+0000 7fe60ffff640 1 -- 172.21.15.29:0/2988309996 >> 172.21.15.29:0/2988309996 conn(0x7fe6100ce820 msgr2=0x7fe6100cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.344+0000 7fe60ffff640 1 -- 172.21.15.29:0/2988309996 shutdown_connections 2023-10-14T16:40:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.344+0000 7fe60ffff640 1 -- 172.21.15.29:0/2988309996 wait complete. 2023-10-14T16:40:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.345+0000 7fe60ffff640 1 Processor -- start 2023-10-14T16:40:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.346+0000 7fe60ffff640 1 -- start start 2023-10-14T16:40:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6100ca310 0x7fe61014fd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 0x7fe6100aeb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe610150280 con 0x7fe61015ad80 2023-10-14T16:40:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100c6810 con 0x7fe6100cae40 2023-10-14T16:40:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6100c6af0 con 0x7fe6100ca310 2023-10-14T16:40:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60742/0 (socket says 172.21.15.29:60742) 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60f7fe640 1 -- 172.21.15.29:0/1611898485 learned_addr learned my addr 172.21.15.29:0/1611898485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60e7fc640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 0x7fe6100aeb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.347+0000 7fe60effd640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6100ca310 0x7fe61014fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 -- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6100ca310 msgr2=0x7fe61014fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6100ca310 0x7fe61014fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 -- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 msgr2=0x7fe6100aeb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 0x7fe6100aeb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 -- 172.21.15.29:0/1611898485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe608009b30 con 0x7fe61015ad80 2023-10-14T16:40:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60e7fc640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 0x7fe6100aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60f7fe640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe608036040 tx=0x7fe608004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe608009510 con 0x7fe61015ad80 2023-10-14T16:40:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe60803e450 con 0x7fe61015ad80 2023-10-14T16:40:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe60803e820 con 0x7fe61015ad80 2023-10-14T16:40:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6100c6dd0 con 0x7fe61015ad80 2023-10-14T16:40:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.348+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6100c7390 con 0x7fe61015ad80 2023-10-14T16:40:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.350+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6100c75b0 con 0x7fe61015ad80 2023-10-14T16:40:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.351+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe60803e980 con 0x7fe61015ad80 2023-10-14T16:40:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.352+0000 7fe5f7fff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 0x7fe5fc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.352+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fe6080abf60 con 0x7fe61015ad80 2023-10-14T16:40:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.352+0000 7fe60effd640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 0x7fe5fc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.353+0000 7fe60effd640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 0x7fe5fc06d360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe600009a30 tx=0x7fe600008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.357+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe608077350 con 0x7fe61015ad80 2023-10-14T16:40:23.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.555+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe61004e9b0 con 0x7fe61015ad80 2023-10-14T16:40:23.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.721+0000 7fe5f7fff640 1 -- 172.21.15.29:0/1611898485 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe60807a970 con 0x7fe61015ad80 2023-10-14T16:40:23.723 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:40:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.725+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 msgr2=0x7fe5fc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.725+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 0x7fe5fc06d360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe600009a30 tx=0x7fe600008040 comp rx=0 tx=0).stop 2023-10-14T16:40:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.725+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 msgr2=0x7fe6100af070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.725+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe608036040 tx=0x7fe608004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 shutdown_connections 2023-10-14T16:40:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe5fc06aea0 0x7fe5fc06d360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe61015ad80 0x7fe6100af070 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6100cae40 0x7fe6100aeb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 --2- 172.21.15.29:0/1611898485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6100ca310 0x7fe61014fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 >> 172.21.15.29:0/1611898485 conn(0x7fe6100ce820 msgr2=0x7fe6100afb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 shutdown_connections 2023-10-14T16:40:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.726+0000 7fe60ffff640 1 -- 172.21.15.29:0/1611898485 wait complete. 2023-10-14T16:40:23.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:40:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.967+0000 7f8fd14e7640 1 Processor -- start 2023-10-14T16:40:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.968+0000 7f8fd14e7640 1 -- start start 2023-10-14T16:40:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fd14e7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fcc15aaa0 0x7f8fcc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fd14e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15ae80 0x7f8fcc0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fd14e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fcc0707e0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fd14e7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fcc06b7a0 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fd14e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15ef20 con 0x7f8fcc15ae80 2023-10-14T16:40:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fcaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fcc15aaa0 0x7f8fcc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fcb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8fcc0ccaa0 0x7f8fcc070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42006/0 (socket says 172.21.15.29:42006) 2023-10-14T16:40:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fcb7fe640 1 -- 172.21.15.29:0/1290713157 learned_addr learned my addr 172.21.15.29:0/1290713157 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.970+0000 7f8fc3fff640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15ae80 0x7f8fcc0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.971+0000 7f8fcaffd640 1 -- 172.21.15.29:0/1290713157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15ae80 msgr2=0x7f8fcc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.971+0000 7f8fcaffd640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15ae80 0x7f8fcc0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.971+0000 7f8fcaffd640 1 -- 172.21.15.29:0/1290713157 >> v1:172.21.15.29:6789/0 conn(0x7f8fcc0ccaa0 legacy=0x7f8fcc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.971+0000 7f8fcaffd640 1 -- 172.21.15.29:0/1290713157 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc15f200 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.971+0000 7f8fca7fc640 1 -- 172.21.15.29:0/1290713157 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3203615148 0 0) 0x7f8fcc0707e0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.972+0000 7f8fcaffd640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc15aaa0 0x7f8fcc15e820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0097a0 tx=0x7f8fbc02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=442caa140ff2b24f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fca7fc640 1 -- 172.21.15.29:0/1290713157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc02e070 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fca7fc640 1 -- 172.21.15.29:0/1290713157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc0315a0 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fca7fc640 1 -- 172.21.15.29:0/1290713157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc031930 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 -- 172.21.15.29:0/1290713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc15aaa0 msgr2=0x7f8fcc15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc15aaa0 0x7f8fcc15e820 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc0097a0 tx=0x7f8fbc02b840 comp rx=0 tx=0).stop 2023-10-14T16:40:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 -- 172.21.15.29:0/1290713157 shutdown_connections 2023-10-14T16:40:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15ae80 0x7f8fcc0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/1290713157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc15aaa0 0x7f8fcc15e820 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.973+0000 7f8fd14e7640 1 -- 172.21.15.29:0/1290713157 >> 172.21.15.29:0/1290713157 conn(0x7f8fcc0cb410 msgr2=0x7f8fcc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:23.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.974+0000 7f8fd14e7640 1 -- 172.21.15.29:0/1290713157 shutdown_connections 2023-10-14T16:40:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.974+0000 7f8fd14e7640 1 -- 172.21.15.29:0/1290713157 wait complete. 2023-10-14T16:40:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.975+0000 7f8fd14e7640 1 Processor -- start 2023-10-14T16:40:23.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.976+0000 7f8fd14e7640 1 -- start start 2023-10-14T16:40:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.976+0000 7f8fd14e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fd14e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15aaa0 0x7f8fcc15cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fd14e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fcc15ae80 0x7f8fcc15b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fd14e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc159c10 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fd14e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15bd00 con 0x7f8fcc15ae80 2023-10-14T16:40:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fd14e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fcc15bfe0 con 0x7f8fcc15aaa0 2023-10-14T16:40:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60756/0 (socket says 172.21.15.29:60756) 2023-10-14T16:40:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 -- 172.21.15.29:0/168205728 learned_addr learned my addr 172.21.15.29:0/168205728 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 -- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15aaa0 msgr2=0x7f8fcc15cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15aaa0 0x7f8fcc15cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 -- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fcc15ae80 msgr2=0x7f8fcc15b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fcc15ae80 0x7f8fcc15b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.977+0000 7f8fcaffd640 1 -- 172.21.15.29:0/168205728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc009400 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fcaffd640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc034040 tx=0x7f8fbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc032840 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fbc0329a0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fcc15c2c0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fbc032d50 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.978+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fcc15c880 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.979+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fcc15ca70 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.980+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8fbc04e430 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.981+0000 7f8fc37fe640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 0x7f8fac06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.981+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f8fbc0ab4c0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.982+0000 7f8fc3fff640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 0x7f8fac06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.982+0000 7f8fc3fff640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 0x7f8fac06d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4006fd0 tx=0x7f8fb4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:23.985+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fbc0769e0 con 0x7f8fcc0ccaa0 2023-10-14T16:40:24.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.183+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f8fcc04e160 con 0x7f8fcc0ccaa0 2023-10-14T16:40:24.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.733+0000 7f8fc37fe640 1 -- 172.21.15.29:0/168205728 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) v1 ==== 170+0+0 (secure 0 0 0) 0x7f8fbc07a000 con 0x7f8fcc0ccaa0 2023-10-14T16:40:24.735 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:40:24.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 msgr2=0x7f8fac06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 0x7f8fac06d360 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4006fd0 tx=0x7f8fb4008040 comp rx=0 tx=0).stop 2023-10-14T16:40:24.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 msgr2=0x7f8fcc1596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f8fbc034040 tx=0x7f8fbc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 shutdown_connections 2023-10-14T16:40:24.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.737+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8fac06aea0 0x7f8fac06d360 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fcc15ae80 0x7f8fcc15b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fcc15aaa0 0x7f8fcc15cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 --2- 172.21.15.29:0/168205728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fcc0ccaa0 0x7f8fcc1596d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 >> 172.21.15.29:0/168205728 conn(0x7f8fcc0cb410 msgr2=0x7f8fcc0b25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:24.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 shutdown_connections 2023-10-14T16:40:24.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.738+0000 7f8fd14e7640 1 -- 172.21.15.29:0/168205728 wait complete. 2023-10-14T16:40:24.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:40:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.965+0000 7fecab54e640 1 Processor -- start 2023-10-14T16:40:24.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.966+0000 7fecab54e640 1 -- start start 2023-10-14T16:40:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7fecab54e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feca406a4b0 0x7feca40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7fecab54e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feca40c9ca0 0x7feca40ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7fecab54e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feca40cb030 con 0x7feca415ad70 2023-10-14T16:40:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7fecab54e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca40cae30 con 0x7feca406a4b0 2023-10-14T16:40:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7fecab54e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feca40c91c0 con 0x7feca40c9ca0 2023-10-14T16:40:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7feca92ea640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feca415ad70 0x7feca4069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42012/0 (socket says 172.21.15.29:42012) 2023-10-14T16:40:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7feca92ea640 1 -- 172.21.15.29:0/1231927260 learned_addr learned my addr 172.21.15.29:0/1231927260 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.967+0000 7feca8ae9640 1 --2- 172.21.15.29:0/1231927260 >> v2:172.21.15.29:3300/0 conn(0x7feca406a4b0 0x7feca40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7feca9aeb640 1 --2- 172.21.15.29:0/1231927260 >> v2:172.21.15.186:3300/0 conn(0x7feca40c9ca0 0x7feca40ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7fec9bfff640 1 -- 172.21.15.29:0/1231927260 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375628753 0 0) 0x7feca40cb030 con 0x7feca415ad70 2023-10-14T16:40:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7fec9bfff640 1 -- 172.21.15.29:0/1231927260 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec90003520 con 0x7feca415ad70 2023-10-14T16:40:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7feca8ae9640 1 -- 172.21.15.29:0/1231927260 >> v2:172.21.15.186:3300/0 conn(0x7feca40c9ca0 msgr2=0x7feca40ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7feca8ae9640 1 --2- 172.21.15.29:0/1231927260 >> v2:172.21.15.186:3300/0 conn(0x7feca40c9ca0 0x7feca40ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7feca8ae9640 1 -- 172.21.15.29:0/1231927260 >> v1:172.21.15.29:6789/0 conn(0x7feca415ad70 legacy=0x7feca4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.968+0000 7feca8ae9640 1 -- 172.21.15.29:0/1231927260 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca40c94a0 con 0x7feca406a4b0 2023-10-14T16:40:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.969+0000 7feca8ae9640 1 --2- 172.21.15.29:0/1231927260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca40ca3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feca406f270 tx=0x7fec9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20b902d721ef7c03 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fec9bfff640 1 -- 172.21.15.29:0/1231927260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94031830 con 0x7feca406a4b0 2023-10-14T16:40:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fec9bfff640 1 -- 172.21.15.29:0/1231927260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec94031990 con 0x7feca406a4b0 2023-10-14T16:40:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fec9bfff640 1 -- 172.21.15.29:0/1231927260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec94031d00 con 0x7feca406a4b0 2023-10-14T16:40:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 -- 172.21.15.29:0/1231927260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 msgr2=0x7feca40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 --2- 172.21.15.29:0/1231927260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca40ca3b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7feca406f270 tx=0x7fec9402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 -- 172.21.15.29:0/1231927260 shutdown_connections 2023-10-14T16:40:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 --2- 172.21.15.29:0/1231927260 >> v2:172.21.15.186:3300/0 conn(0x7feca40c9ca0 0x7feca40ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 --2- 172.21.15.29:0/1231927260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca40ca3b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 -- 172.21.15.29:0/1231927260 >> 172.21.15.29:0/1231927260 conn(0x7feca40ce9b0 msgr2=0x7feca40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 -- 172.21.15.29:0/1231927260 shutdown_connections 2023-10-14T16:40:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.970+0000 7fecab54e640 1 -- 172.21.15.29:0/1231927260 wait complete. 2023-10-14T16:40:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.972+0000 7fecab54e640 1 Processor -- start 2023-10-14T16:40:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.972+0000 7fecab54e640 1 -- start start 2023-10-14T16:40:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feca40c9ca0 0x7feca40c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 0x7feca40c4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40c3840 con 0x7feca406a4b0 2023-10-14T16:40:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca40c4fd0 con 0x7feca415ad70 2023-10-14T16:40:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7fecab54e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feca4151740 con 0x7feca40c9ca0 2023-10-14T16:40:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7feca92ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7feca92ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60768/0 (socket says 172.21.15.29:60768) 2023-10-14T16:40:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.973+0000 7feca92ea640 1 -- 172.21.15.29:0/3558576361 learned_addr learned my addr 172.21.15.29:0/3558576361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca9aeb640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 0x7feca40c4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca8ae9640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feca40c9ca0 0x7feca40c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 -- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feca40c9ca0 msgr2=0x7feca40c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feca40c9ca0 0x7feca40c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 -- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 msgr2=0x7feca40c4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 0x7feca40c4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 -- 172.21.15.29:0/3558576361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94009400 con 0x7feca406a4b0 2023-10-14T16:40:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca9aeb640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 0x7feca40c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7feca92ea640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feca000caa0 tx=0x7feca000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca000ac00 con 0x7feca406a4b0 2023-10-14T16:40:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feca000ad60 con 0x7feca406a4b0 2023-10-14T16:40:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.974+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca0012620 con 0x7feca406a4b0 2023-10-14T16:40:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.975+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca40c5bb0 con 0x7feca406a4b0 2023-10-14T16:40:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.975+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca40c6170 con 0x7feca406a4b0 2023-10-14T16:40:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.976+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca404e160 con 0x7feca406a4b0 2023-10-14T16:40:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.977+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feca0007070 con 0x7feca406a4b0 2023-10-14T16:40:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.978+0000 7fec99ffb640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 0x7fec8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.978+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5609+0+0 (secure 0 0 0) 0x7feca0089560 con 0x7feca406a4b0 2023-10-14T16:40:24.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.978+0000 7feca8ae9640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 0x7fec8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.979+0000 7feca8ae9640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 0x7fec8806d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fec94036000 tx=0x7fec94008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:24.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:24.983+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feca0054660 con 0x7feca406a4b0 2023-10-14T16:40:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.221+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feca4071660 con 0x7feca406a4b0 2023-10-14T16:40:25.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.737+0000 7fec99ffb640 1 -- 172.21.15.29:0/3558576361 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) v1 ==== 172+0+0 (secure 0 0 0) 0x7feca00170c0 con 0x7feca406a4b0 2023-10-14T16:40:25.739 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 2 and data pool 3 2023-10-14T16:40:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.741+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 msgr2=0x7fec8806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:25.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.741+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 0x7fec8806d330 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fec94036000 tx=0x7fec94008030 comp rx=0 tx=0).stop 2023-10-14T16:40:25.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.741+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 msgr2=0x7feca4151200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:25.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.741+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7feca000caa0 tx=0x7feca000cf60 comp rx=0 tx=0).stop 2023-10-14T16:40:25.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 shutdown_connections 2023-10-14T16:40:25.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fec8806ae70 0x7fec8806d330 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feca415ad70 0x7feca40c4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feca40c9ca0 0x7feca40c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 --2- 172.21.15.29:0/3558576361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feca406a4b0 0x7feca4151200 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 >> 172.21.15.29:0/3558576361 conn(0x7feca40ce9b0 msgr2=0x7feca40b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:25.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 shutdown_connections 2023-10-14T16:40:25.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.742+0000 7fecab54e640 1 -- 172.21.15.29:0/3558576361 wait complete. 2023-10-14T16:40:25.783 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:25.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.971+0000 7f3474e7e640 1 Processor -- start 2023-10-14T16:40:25.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.971+0000 7f3474e7e640 1 -- start start 2023-10-14T16:40:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f347015cbe0 0x7f34700d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f347015cfc0 0x7f34700b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34700d1090 0x7f34700b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34700d15a0 con 0x7f34700d1090 2023-10-14T16:40:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34700d1880 con 0x7f347015cfc0 2023-10-14T16:40:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.972+0000 7f3474e7e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34700d1b60 con 0x7f347015cbe0 2023-10-14T16:40:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34700d1090 0x7f34700b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34700d1090 0x7f34700b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60782/0 (socket says 172.21.15.29:60782) 2023-10-14T16:40:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 -- 172.21.15.29:0/2475974074 learned_addr learned my addr 172.21.15.29:0/2475974074 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:25.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f3466ffd640 1 --2- 172.21.15.29:0/2475974074 >> v2:172.21.15.186:3300/0 conn(0x7f347015cfc0 0x7f34700b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346f7fe640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f347015cbe0 0x7f34700d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 -- 172.21.15.29:0/2475974074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f347015cbe0 msgr2=0x7f34700d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f347015cbe0 0x7f34700d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 -- 172.21.15.29:0/2475974074 >> v2:172.21.15.186:3300/0 conn(0x7f347015cfc0 msgr2=0x7f34700b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 --2- 172.21.15.29:0/2475974074 >> v2:172.21.15.186:3300/0 conn(0x7f347015cfc0 0x7f34700b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.973+0000 7f346ffff640 1 -- 172.21.15.29:0/2475974074 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34700d1e40 con 0x7f34700d1090 2023-10-14T16:40:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.975+0000 7f346ffff640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700d1090 0x7f34700b4ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34700724e0 tx=0x7f346802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3e7fc7cb746bf75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.975+0000 7f346effd640 1 -- 172.21.15.29:0/2475974074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346802cb00 con 0x7f34700d1090 2023-10-14T16:40:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.975+0000 7f346effd640 1 -- 172.21.15.29:0/2475974074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346802cc60 con 0x7f34700d1090 2023-10-14T16:40:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.975+0000 7f346effd640 1 -- 172.21.15.29:0/2475974074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34680343f0 con 0x7f34700d1090 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 -- 172.21.15.29:0/2475974074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700d1090 msgr2=0x7f34700b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700d1090 0x7f34700b4ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f34700724e0 tx=0x7f346802bf60 comp rx=0 tx=0).stop 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 -- 172.21.15.29:0/2475974074 shutdown_connections 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34700d1090 0x7f34700b4ab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 --2- 172.21.15.29:0/2475974074 >> v2:172.21.15.186:3300/0 conn(0x7f347015cfc0 0x7f34700b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 --2- 172.21.15.29:0/2475974074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f347015cbe0 0x7f34700d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 -- 172.21.15.29:0/2475974074 >> 172.21.15.29:0/2475974074 conn(0x7f34700cf570 msgr2=0x7f34700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 -- 172.21.15.29:0/2475974074 shutdown_connections 2023-10-14T16:40:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.976+0000 7f3474e7e640 1 -- 172.21.15.29:0/2475974074 wait complete. 2023-10-14T16:40:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.978+0000 7f3474e7e640 1 Processor -- start 2023-10-14T16:40:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.978+0000 7f3474e7e640 1 -- start start 2023-10-14T16:40:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34700d1090 0x7f34700cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347015cbe0 0x7f34700b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700cc1c0 con 0x7f347015cfc0 2023-10-14T16:40:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34700cb890 con 0x7f347015cbe0 2023-10-14T16:40:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f3474e7e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34700cbb70 con 0x7f34700d1090 2023-10-14T16:40:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60786/0 (socket says 172.21.15.29:60786) 2023-10-14T16:40:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 -- 172.21.15.29:0/3574493147 learned_addr learned my addr 172.21.15.29:0/3574493147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 -- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34700d1090 msgr2=0x7f34700cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34700d1090 0x7f34700cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 -- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347015cbe0 msgr2=0x7f34700b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347015cbe0 0x7f34700b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.979+0000 7f346ffff640 1 -- 172.21.15.29:0/3574493147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3468009b30 con 0x7f347015cfc0 2023-10-14T16:40:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f346ffff640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3468036040 tx=0x7f3468004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3468009510 con 0x7f347015cfc0 2023-10-14T16:40:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f346803e450 con 0x7f347015cfc0 2023-10-14T16:40:25.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f346803e800 con 0x7f347015cfc0 2023-10-14T16:40:25.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34700cbe50 con 0x7f347015cfc0 2023-10-14T16:40:25.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.980+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34700b4dc0 con 0x7f347015cfc0 2023-10-14T16:40:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.982+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f347004e9b0 con 0x7f347015cfc0 2023-10-14T16:40:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.982+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f346803e960 con 0x7f347015cfc0 2023-10-14T16:40:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.984+0000 7f346cff9640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 0x7f345006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.984+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34680ac3e0 con 0x7f347015cfc0 2023-10-14T16:40:25.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.984+0000 7f346f7fe640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 0x7f345006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:25.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.984+0000 7f346f7fe640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 0x7f345006d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34700ccbc0 tx=0x7f3460008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:25.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:25.988+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34680773f0 con 0x7f347015cfc0 2023-10-14T16:40:26.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.181+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34700cc300 con 0x7f347015cfc0 2023-10-14T16:40:26.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.183+0000 7f346cff9640 1 -- 172.21.15.29:0/3574493147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16506 (secure 0 0 0) 0x7f346807aa10 con 0x7f347015cfc0 2023-10-14T16:40:26.184 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:26.185 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:25.222748+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:26.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.185+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 msgr2=0x7f345006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.185+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 0x7f345006d330 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f34700ccbc0 tx=0x7f3460008040 comp rx=0 tx=0).stop 2023-10-14T16:40:26.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 msgr2=0x7f347014a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3468036040 tx=0x7f3468004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:26.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 shutdown_connections 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f345006ae70 0x7f345006d330 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f347015cfc0 0x7f347014a620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f347015cbe0 0x7f34700b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 --2- 172.21.15.29:0/3574493147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34700d1090 0x7f34700cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 >> 172.21.15.29:0/3574493147 conn(0x7f34700cf570 msgr2=0x7f34700d3410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 shutdown_connections 2023-10-14T16:40:26.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.186+0000 7f3474e7e640 1 -- 172.21.15.29:0/3574493147 wait complete. 2023-10-14T16:40:26.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.424+0000 7f792d16c640 1 Processor -- start 2023-10-14T16:40:26.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.424+0000 7f792d16c640 1 -- start start 2023-10-14T16:40:26.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f792d16c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280ad970 0x7f79280cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f792d16c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792815b9d0 con 0x7f79280ccfb0 2023-10-14T16:40:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f792d16c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f792815bcb0 con 0x7f792806c040 2023-10-14T16:40:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f792d16c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f792815bf90 con 0x7f79280ad970 2023-10-14T16:40:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f791f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280ad970 0x7f79280cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f791f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79280ad970 0x7f79280cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60788/0 (socket says 172.21.15.29:60788) 2023-10-14T16:40:26.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f791f7fe640 1 -- 172.21.15.29:0/573120010 learned_addr learned my addr 172.21.15.29:0/573120010 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f791f7fe640 1 -- 172.21.15.29:0/573120010 >> v1:172.21.15.186:6789/0 conn(0x7f792806c040 legacy=0x7f792806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f791f7fe640 1 -- 172.21.15.29:0/573120010 >> v1:172.21.15.29:6789/0 conn(0x7f79280ccfb0 legacy=0x7f79280cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.427+0000 7f791f7fe640 1 -- 172.21.15.29:0/573120010 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79280cc270 con 0x7f79280ad970 2023-10-14T16:40:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.426+0000 7f79277fe640 1 -- 172.21.15.29:0/573120010 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476910646 0 0) 0x7f792815b9d0 con 0x7f79280ccfb0 2023-10-14T16:40:26.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.428+0000 7f791f7fe640 1 --2- 172.21.15.29:0/573120010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280cca70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7910009e80 tx=0x7f791002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14b7fe59a457a946 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.428+0000 7f79277fe640 1 -- 172.21.15.29:0/573120010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791002caf0 con 0x7f79280ad970 2023-10-14T16:40:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.428+0000 7f79277fe640 1 -- 172.21.15.29:0/573120010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791002cc50 con 0x7f79280ad970 2023-10-14T16:40:26.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.428+0000 7f79277fe640 1 -- 172.21.15.29:0/573120010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79100343f0 con 0x7f79280ad970 2023-10-14T16:40:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 -- 172.21.15.29:0/573120010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 msgr2=0x7f79280cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 --2- 172.21.15.29:0/573120010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280cca70 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7910009e80 tx=0x7f791002bf50 comp rx=0 tx=0).stop 2023-10-14T16:40:26.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 -- 172.21.15.29:0/573120010 shutdown_connections 2023-10-14T16:40:26.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 --2- 172.21.15.29:0/573120010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280cca70 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 -- 172.21.15.29:0/573120010 >> 172.21.15.29:0/573120010 conn(0x7f79280ce6a0 msgr2=0x7f79280ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:26.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 -- 172.21.15.29:0/573120010 shutdown_connections 2023-10-14T16:40:26.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.429+0000 7f792d16c640 1 -- 172.21.15.29:0/573120010 wait complete. 2023-10-14T16:40:26.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.430+0000 7f792d16c640 1 Processor -- start 2023-10-14T16:40:26.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.431+0000 7f792d16c640 1 -- start start 2023-10-14T16:40:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f792806c040 0x7f79281581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79280ccfb0 0x7f79280c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b41b0 con 0x7f79280ad970 2023-10-14T16:40:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b3630 con 0x7f79280ccfb0 2023-10-14T16:40:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f792d16c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79280b3910 con 0x7f792806c040 2023-10-14T16:40:26.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60798/0 (socket says 172.21.15.29:60798) 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 -- 172.21.15.29:0/4240407207 learned_addr learned my addr 172.21.15.29:0/4240407207 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 -- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f792806c040 msgr2=0x7f79281581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f792806c040 0x7f79281581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 -- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79280ccfb0 msgr2=0x7f79280c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79280ccfb0 0x7f79280c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.432+0000 7f791f7fe640 1 -- 172.21.15.29:0/4240407207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7910009b30 con 0x7f79280ad970 2023-10-14T16:40:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f791f7fe640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7910009e80 tx=0x7f7910004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791003e450 con 0x7f79280ad970 2023-10-14T16:40:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f791003e5b0 con 0x7f79280ad970 2023-10-14T16:40:26.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f791003e960 con 0x7f79280ad970 2023-10-14T16:40:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79280b3bf0 con 0x7f79280ad970 2023-10-14T16:40:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.433+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79280afda0 con 0x7f79280ad970 2023-10-14T16:40:26.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.434+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f792806b330 con 0x7f79280ad970 2023-10-14T16:40:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.435+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f791003eac0 con 0x7f79280ad970 2023-10-14T16:40:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.436+0000 7f79257fa640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 0x7f790806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.436+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f79100ac4e0 con 0x7f79280ad970 2023-10-14T16:40:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.436+0000 7f7927fff640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 0x7f790806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.437+0000 7f7927fff640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 0x7f790806d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7918006fd0 tx=0x7f7918006bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.441+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7910077590 con 0x7f79280ad970 2023-10-14T16:40:26.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.642+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f792804e9b0 con 0x7f79280ad970 2023-10-14T16:40:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.645+0000 7f79257fa640 1 -- 172.21.15.29:0/4240407207 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+16506 (secure 0 0 0) 0x7f791007abb0 con 0x7f79280ad970 2023-10-14T16:40:26.646 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:26.647 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":19,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:25.222748+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 msgr2=0x7f790806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 0x7f790806d330 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7918006fd0 tx=0x7f7918006bc0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 msgr2=0x7f79280af860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7910009e80 tx=0x7f7910004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 shutdown_connections 2023-10-14T16:40:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f790806ae70 0x7f790806d330 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79280ccfb0 0x7f79280c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79280ad970 0x7f79280af860 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 --2- 172.21.15.29:0/4240407207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f792806c040 0x7f79281581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 >> 172.21.15.29:0/4240407207 conn(0x7f79280ce6a0 msgr2=0x7f792806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 shutdown_connections 2023-10-14T16:40:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.649+0000 7f792d16c640 1 -- 172.21.15.29:0/4240407207 wait complete. 2023-10-14T16:40:26.668 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:40:26.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.898+0000 7fc39f66f640 1 Processor -- start 2023-10-14T16:40:26.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.899+0000 7fc39f66f640 1 -- start start 2023-10-14T16:40:26.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.900+0000 7fc39f66f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc3980caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.900+0000 7fc39f66f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc39806a4b0 0x7fc3981650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39f66f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980cb140 con 0x7fc3980c9000 2023-10-14T16:40:26.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39f66f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc398161a50 con 0x7fc39806a4b0 2023-10-14T16:40:26.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39f66f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39806b030 con 0x7fc3980addb0 2023-10-14T16:40:26.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39cc0a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3980c9000 0x7fc398069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42030/0 (socket says 172.21.15.29:42030) 2023-10-14T16:40:26.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39cc0a640 1 -- 172.21.15.29:0/4119310131 learned_addr learned my addr 172.21.15.29:0/4119310131 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:26.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39dc0c640 1 --2- 172.21.15.29:0/4119310131 >> v2:172.21.15.186:3300/0 conn(0x7fc39806a4b0 0x7fc3981650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc39d40b640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc3980caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 663411215 0 0) 0x7fc3980cb140 con 0x7fc3980c9000 2023-10-14T16:40:26.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.901+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc384003520 con 0x7fc3980c9000 2023-10-14T16:40:26.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945713540 0 0) 0x7fc384003520 con 0x7fc3980c9000 2023-10-14T16:40:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 msgr2=0x7fc3980caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc3980caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 >> v2:172.21.15.186:3300/0 conn(0x7fc39806a4b0 msgr2=0x7fc3981650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 --2- 172.21.15.29:0/4119310131 >> v2:172.21.15.186:3300/0 conn(0x7fc39806a4b0 0x7fc3981650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39806b310 con 0x7fc3980c9000 2023-10-14T16:40:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc388001670 con 0x7fc3980c9000 2023-10-14T16:40:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 >> v1:172.21.15.29:6789/0 conn(0x7fc3980c9000 legacy=0x7fc398069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc384003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc384004d00 0x7fc384005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc398161a50 con 0x7fc384003ef0 2023-10-14T16:40:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc39806b030 con 0x7fc384004d00 2023-10-14T16:40:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc38400b980 con 0x7fc3980addb0 2023-10-14T16:40:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.902+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39dc0c640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc384004d00 0x7fc384005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39cc0a640 1 -- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 msgr2=0x7fc384003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39d40b640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc384003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3980addb0 0x7fc384003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39cc0a640 1 -- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc384004d00 msgr2=0x7fc384005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:26.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc384004d00 0x7fc384005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39cc0a640 1 -- 172.21.15.29:0/4119310131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc384003520 con 0x7fc384003ef0 2023-10-14T16:40:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.903+0000 7fc39dc0c640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc384004d00 0x7fc384005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.904+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3840047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc38800c290 tx=0x7fc3840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59a068d57e04f0bf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.904+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc388005ad0 con 0x7fc384003ef0 2023-10-14T16:40:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.904+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388005c30 con 0x7fc384003ef0 2023-10-14T16:40:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.904+0000 7fc38ffff640 1 -- 172.21.15.29:0/4119310131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc38802c8c0 con 0x7fc384003ef0 2023-10-14T16:40:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 -- 172.21.15.29:0/4119310131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 msgr2=0x7fc3840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 --2- 172.21.15.29:0/4119310131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3840047c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc38800c290 tx=0x7fc3840057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39dc0c640 1 -- 172.21.15.29:0/4119310131 reap_dead start 2023-10-14T16:40:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 -- 172.21.15.29:0/4119310131 shutdown_connections 2023-10-14T16:40:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 -- 172.21.15.29:0/4119310131 >> 172.21.15.29:0/4119310131 conn(0x7fc3980ce9b0 msgr2=0x7fc3980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 -- 172.21.15.29:0/4119310131 shutdown_connections 2023-10-14T16:40:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.905+0000 7fc39f66f640 1 -- 172.21.15.29:0/4119310131 wait complete. 2023-10-14T16:40:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.907+0000 7fc39f66f640 1 Processor -- start 2023-10-14T16:40:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.907+0000 7fc39f66f640 1 -- start start 2023-10-14T16:40:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 0x7fc3980af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 0x7fc3980afa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980b2ca0 con 0x7fc384003ef0 2023-10-14T16:40:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980b2090 con 0x7fc3980b1b10 2023-10-14T16:40:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39f66f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3980b2370 con 0x7fc39815f0e0 2023-10-14T16:40:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39d40b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39d40b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60814/0 (socket says 172.21.15.29:60814) 2023-10-14T16:40:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.908+0000 7fc39d40b640 1 -- 172.21.15.29:0/308430296 learned_addr learned my addr 172.21.15.29:0/308430296 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:26.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39dc0c640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 0x7fc3980afa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 0x7fc3980af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 -- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 msgr2=0x7fc3980af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 0x7fc3980af510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 -- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 msgr2=0x7fc3980afa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 0x7fc3980afa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 -- 172.21.15.29:0/308430296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3980b2650 con 0x7fc384003ef0 2023-10-14T16:40:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39dc0c640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 0x7fc3980afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 0x7fc3980af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39d40b640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc394010860 tx=0x7fc394010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394009610 con 0x7fc384003ef0 2023-10-14T16:40:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39400eb60 con 0x7fc384003ef0 2023-10-14T16:40:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3980b2930 con 0x7fc384003ef0 2023-10-14T16:40:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39401c400 con 0x7fc384003ef0 2023-10-14T16:40:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.909+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3981f9eb0 con 0x7fc384003ef0 2023-10-14T16:40:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.911+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc39804e160 con 0x7fc384003ef0 2023-10-14T16:40:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.912+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc39401c560 con 0x7fc384003ef0 2023-10-14T16:40:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.913+0000 7fc38dffb640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 0x7fc37c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.913+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc394089b70 con 0x7fc384003ef0 2023-10-14T16:40:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.913+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 0x7fc37c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.913+0000 7fc39cc0a640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 0x7fc37c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc388034000 tx=0x7fc388007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:26.917+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc394054c30 con 0x7fc384003ef0 2023-10-14T16:40:27.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.153+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fc39815f550 con 0x7fc384003ef0 2023-10-14T16:40:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.781+0000 7fc38dffb640 1 -- 172.21.15.29:0/308430296 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7fc394058250 con 0x7fc384003ef0 2023-10-14T16:40:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 msgr2=0x7fc37c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 0x7fc37c06d330 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fc388034000 tx=0x7fc388007000 comp rx=0 tx=0).stop 2023-10-14T16:40:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 msgr2=0x7fc3980b15d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc394010860 tx=0x7fc394010d20 comp rx=0 tx=0).stop 2023-10-14T16:40:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 shutdown_connections 2023-10-14T16:40:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc37c06ae70 0x7fc37c06d330 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3980b1b10 0x7fc3980afa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc39815f0e0 0x7fc3980af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 --2- 172.21.15.29:0/308430296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc384003ef0 0x7fc3980b15d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 >> 172.21.15.29:0/308430296 conn(0x7fc3980ce9b0 msgr2=0x7fc3980b0cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 shutdown_connections 2023-10-14T16:40:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:27.785+0000 7fc39f66f640 1 -- 172.21.15.29:0/308430296 wait complete. 2023-10-14T16:40:27.807 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:40:27.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:40:28.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.033+0000 7f03034e0640 1 Processor -- start 2023-10-14T16:40:28.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.034+0000 7f03034e0640 1 -- start start 2023-10-14T16:40:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f03034e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02fc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f03034e0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02fc0cd260 0x7f02fc0cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f03034e0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02fc0b5d90 con 0x7f02fc15d030 2023-10-14T16:40:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f03034e0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02fc0d2b50 con 0x7f02fc0cd260 2023-10-14T16:40:28.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f03034e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc160fc0 con 0x7f02fc15d410 2023-10-14T16:40:28.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f030127c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f02fc15d030 0x7f02fc161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42044/0 (socket says 172.21.15.29:42044) 2023-10-14T16:40:28.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.035+0000 7f030127c640 1 -- 172.21.15.29:0/2103796650 learned_addr learned my addr 172.21.15.29:0/2103796650 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:28.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f0300a7b640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02fc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f0301a7d640 1 --2- 172.21.15.29:0/2103796650 >> v2:172.21.15.186:3300/0 conn(0x7f02fc0cd260 0x7f02fc0cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982428353 0 0) 0x7f02fc0b5d90 con 0x7f02fc15d030 2023-10-14T16:40:28.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f02e8003520 con 0x7f02fc15d030 2023-10-14T16:40:28.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3186225726 0 0) 0x7f02e8003520 con 0x7f02fc15d030 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 msgr2=0x7f02fc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02fc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 >> v2:172.21.15.186:3300/0 conn(0x7f02fc0cd260 msgr2=0x7f02fc0cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 --2- 172.21.15.29:0/2103796650 >> v2:172.21.15.186:3300/0 conn(0x7f02fc0cd260 0x7f02fc0cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc0d25d0 con 0x7f02fc15d030 2023-10-14T16:40:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f02f8001670 con 0x7f02fc15d030 2023-10-14T16:40:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 >> v1:172.21.15.29:6789/0 conn(0x7f02fc15d030 legacy=0x7f02fc161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02e8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8003ef0 0x7f02e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02e8004d00 0x7f02e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc160fc0 con 0x7f02e8004d00 2023-10-14T16:40:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e800b980 con 0x7f02e8003ef0 2023-10-14T16:40:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.036+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02e800baf0 con 0x7f02fc15d410 2023-10-14T16:40:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02e8004d00 0x7f02e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f030127c640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02e8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0300a7b640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8003ef0 0x7f02e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 -- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 msgr2=0x7f02e8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc15d410 0x7f02e8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 -- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8003ef0 msgr2=0x7f02e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8003ef0 0x7f02e80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.037+0000 7f0301a7d640 1 -- 172.21.15.29:0/2103796650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e8003520 con 0x7f02e8004d00 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.038+0000 7f0301a7d640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02e8004d00 0x7f02e8005180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009fc0 tx=0x7f02e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ca99aa1b3290ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.038+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f402cc60 con 0x7f02e8004d00 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.038+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02f400f040 con 0x7f02e8004d00 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.038+0000 7f02f3fff640 1 -- 172.21.15.29:0/2103796650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02f4034440 con 0x7f02e8004d00 2023-10-14T16:40:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 -- 172.21.15.29:0/2103796650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02e8004d00 msgr2=0x7f02e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 --2- 172.21.15.29:0/2103796650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02e8004d00 0x7f02e8005180 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009fc0 tx=0x7f02e80057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f0301a7d640 1 -- 172.21.15.29:0/2103796650 reap_dead start 2023-10-14T16:40:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 -- 172.21.15.29:0/2103796650 shutdown_connections 2023-10-14T16:40:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 -- 172.21.15.29:0/2103796650 >> 172.21.15.29:0/2103796650 conn(0x7f02fc0cf720 msgr2=0x7f02fc0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 -- 172.21.15.29:0/2103796650 shutdown_connections 2023-10-14T16:40:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.039+0000 7f03034e0640 1 -- 172.21.15.29:0/2103796650 wait complete. 2023-10-14T16:40:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.041+0000 7f03034e0640 1 Processor -- start 2023-10-14T16:40:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.041+0000 7f03034e0640 1 -- start start 2023-10-14T16:40:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8004d00 0x7f02fc0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc150e00 0x7f02fc151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc0c8e50 con 0x7f02fc15d7f0 2023-10-14T16:40:28.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc15ddb0 con 0x7f02e8004d00 2023-10-14T16:40:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f03034e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc15e090 con 0x7f02fc150e00 2023-10-14T16:40:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f0300a7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f0300a7b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60838/0 (socket says 172.21.15.29:60838) 2023-10-14T16:40:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f0300a7b640 1 -- 172.21.15.29:0/621193111 learned_addr learned my addr 172.21.15.29:0/621193111 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.042+0000 7f030127c640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8004d00 0x7f02fc0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0301a7d640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc150e00 0x7f02fc151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 -- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc150e00 msgr2=0x7f02fc151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc150e00 0x7f02fc151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 -- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8004d00 msgr2=0x7f02fc0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8004d00 0x7f02fc0c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 -- 172.21.15.29:0/621193111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02fc15e370 con 0x7f02fc15d7f0 2023-10-14T16:40:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f0300a7b640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f02ec011540 tx=0x7f02ec011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec015930 con 0x7f02fc15d7f0 2023-10-14T16:40:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02ec015a90 con 0x7f02fc15d7f0 2023-10-14T16:40:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02ec015e60 con 0x7f02fc15d7f0 2023-10-14T16:40:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.043+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc15e650 con 0x7f02fc15d7f0 2023-10-14T16:40:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.044+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc1517c0 con 0x7f02fc15d7f0 2023-10-14T16:40:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.045+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc0d01a0 con 0x7f02fc15d7f0 2023-10-14T16:40:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.046+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f02ec00f460 con 0x7f02fc15d7f0 2023-10-14T16:40:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.046+0000 7f02f1ffb640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 0x7f02e006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.047+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f02ec089de0 con 0x7f02fc15d7f0 2023-10-14T16:40:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.047+0000 7f030127c640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 0x7f02e006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.047+0000 7f030127c640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 0x7f02e006d360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02f8005b30 tx=0x7f02f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.051+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02ec054e90 con 0x7f02fc15d7f0 2023-10-14T16:40:28.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.286+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f02fc04e160 con 0x7f02fc15d7f0 2023-10-14T16:40:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.796+0000 7f02f1ffb640 1 -- 172.21.15.29:0/621193111 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f02ec0584b0 con 0x7f02fc15d7f0 2023-10-14T16:40:28.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 msgr2=0x7f02e006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 0x7f02e006d360 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f02f8005b30 tx=0x7f02f8009040 comp rx=0 tx=0).stop 2023-10-14T16:40:28.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 msgr2=0x7f02fc0b2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f02ec011540 tx=0x7f02ec011a00 comp rx=0 tx=0).stop 2023-10-14T16:40:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 shutdown_connections 2023-10-14T16:40:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f02e006aea0 0x7f02e006d360 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02fc150e00 0x7f02fc151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02fc15d7f0 0x7f02fc0b2870 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 --2- 172.21.15.29:0/621193111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02e8004d00 0x7f02fc0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 >> 172.21.15.29:0/621193111 conn(0x7f02fc0cf720 msgr2=0x7f02fc15c400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 shutdown_connections 2023-10-14T16:40:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:28.801+0000 7f03034e0640 1 -- 172.21.15.29:0/621193111 wait complete. 2023-10-14T16:40:28.826 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:40:28.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:40:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.050+0000 7f29aa4ca640 1 Processor -- start 2023-10-14T16:40:29.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.051+0000 7f29aa4ca640 1 -- start start 2023-10-14T16:40:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29a4159950 0x7f29a415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f29a4070450 0x7f29a40708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a406c100 0x7f29a406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a4070e20 con 0x7f29a4159950 2023-10-14T16:40:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29a415e490 con 0x7f29a4070450 2023-10-14T16:40:29.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.052+0000 7f29aa4ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a415aef0 con 0x7f29a406c100 2023-10-14T16:40:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29a4159950 0x7f29a415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29a4159950 0x7f29a415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60854/0 (socket says 172.21.15.29:60854) 2023-10-14T16:40:29.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 -- 172.21.15.29:0/2508941442 learned_addr learned my addr 172.21.15.29:0/2508941442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a8cc7640 1 --2- 172.21.15.29:0/2508941442 >> v2:172.21.15.186:3300/0 conn(0x7f29a4070450 0x7f29a40708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a9cc9640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a406c100 0x7f29a406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 -- 172.21.15.29:0/2508941442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a406c100 msgr2=0x7f29a406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a406c100 0x7f29a406c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 -- 172.21.15.29:0/2508941442 >> v2:172.21.15.186:3300/0 conn(0x7f29a4070450 msgr2=0x7f29a40708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2508941442 >> v2:172.21.15.186:3300/0 conn(0x7f29a4070450 0x7f29a40708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.053+0000 7f29a94c8640 1 -- 172.21.15.29:0/2508941442 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a415d910 con 0x7f29a4159950 2023-10-14T16:40:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.054+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a415a9b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29a40cab70 tx=0x7f299402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f13f6b39b03734b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.054+0000 7f29a3fff640 1 -- 172.21.15.29:0/2508941442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29940317c0 con 0x7f29a4159950 2023-10-14T16:40:29.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29a3fff640 1 -- 172.21.15.29:0/2508941442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2994031920 con 0x7f29a4159950 2023-10-14T16:40:29.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29a3fff640 1 -- 172.21.15.29:0/2508941442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2994031cb0 con 0x7f29a4159950 2023-10-14T16:40:29.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2508941442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 msgr2=0x7f29a415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a415a9b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f29a40cab70 tx=0x7f299402b860 comp rx=0 tx=0).stop 2023-10-14T16:40:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2508941442 shutdown_connections 2023-10-14T16:40:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a406c100 0x7f29a406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2508941442 >> v2:172.21.15.186:3300/0 conn(0x7f29a4070450 0x7f29a40708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2508941442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a415a9b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2508941442 >> 172.21.15.29:0/2508941442 conn(0x7f29a40ce6a0 msgr2=0x7f29a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2508941442 shutdown_connections 2023-10-14T16:40:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.055+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2508941442 wait complete. 2023-10-14T16:40:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.057+0000 7f29aa4ca640 1 Processor -- start 2023-10-14T16:40:29.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.057+0000 7f29aa4ca640 1 -- start start 2023-10-14T16:40:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 0x7f29a414a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 0x7f29a40c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a414ac80 con 0x7f29a4159950 2023-10-14T16:40:29.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29a40b2680 con 0x7f29a406c100 2023-10-14T16:40:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.058+0000 7f29aa4ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29a40c60a0 con 0x7f29a4070450 2023-10-14T16:40:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60860/0 (socket says 172.21.15.29:60860) 2023-10-14T16:40:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 -- 172.21.15.29:0/2302975981 learned_addr learned my addr 172.21.15.29:0/2302975981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:29.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 0x7f29a414a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a8cc7640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 0x7f29a40c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 -- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 msgr2=0x7f29a40c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 0x7f29a40c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 -- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 msgr2=0x7f29a414a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 0x7f29a414a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 -- 172.21.15.29:0/2302975981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29940094d0 con 0x7f29a4159950 2023-10-14T16:40:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a8cc7640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 0x7f29a40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 0x7f29a414a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.059+0000 7f29a9cc9640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f299c00cc30 tx=0x7f299c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.060+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c00ce40 con 0x7f29a4159950 2023-10-14T16:40:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.060+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f299c003990 con 0x7f29a4159950 2023-10-14T16:40:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.060+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299c003d20 con 0x7f29a4159950 2023-10-14T16:40:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.060+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29a40c6380 con 0x7f29a4159950 2023-10-14T16:40:29.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.060+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29a40b2310 con 0x7f29a4159950 2023-10-14T16:40:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.061+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29a40b27f0 con 0x7f29a4159950 2023-10-14T16:40:29.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.062+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f299c003e80 con 0x7f29a4159950 2023-10-14T16:40:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.063+0000 7f29a1ffb640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 0x7f299006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.063+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f299c089aa0 con 0x7f29a4159950 2023-10-14T16:40:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.063+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 0x7f299006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.064+0000 7f29a94c8640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 0x7f299006d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2994036000 tx=0x7f299403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.067+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f299c054ab0 con 0x7f29a4159950 2023-10-14T16:40:29.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.310+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29a404e9b0 con 0x7f29a4159950 2023-10-14T16:40:29.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.311+0000 7f29a1ffb640 1 -- 172.21.15.29:0/2302975981 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f299c01d070 con 0x7f29a4159950 2023-10-14T16:40:29.312 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:29.312 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:40:29.313 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:40:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.313+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 msgr2=0x7f299006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 0x7f299006d330 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f2994036000 tx=0x7f299403d040 comp rx=0 tx=0).stop 2023-10-14T16:40:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 msgr2=0x7f29a40c8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f299c00cc30 tx=0x7f299c00a5c0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 shutdown_connections 2023-10-14T16:40:29.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f299006ae70 0x7f299006d330 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29a4159950 0x7f29a40c8f20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29a4070450 0x7f29a40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 --2- 172.21.15.29:0/2302975981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29a406c100 0x7f29a414a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 >> 172.21.15.29:0/2302975981 conn(0x7f29a40ce6a0 msgr2=0x7f29a415a690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:29.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 shutdown_connections 2023-10-14T16:40:29.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.314+0000 7f29aa4ca640 1 -- 172.21.15.29:0/2302975981 wait complete. 2023-10-14T16:40:29.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.559+0000 7ff1a654e640 1 Processor -- start 2023-10-14T16:40:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.559+0000 7ff1a654e640 1 -- start start 2023-10-14T16:40:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.561+0000 7ff1a654e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1a015aaa0 0x7ff1a015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.561+0000 7ff1a654e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.561+0000 7ff1a654e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1a015ed60 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.561+0000 7ff1a654e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1a006b810 con 0x7ff1a015aaa0 2023-10-14T16:40:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.561+0000 7ff1a654e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a00707e0 con 0x7ff1a015ae80 2023-10-14T16:40:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1a015aaa0 0x7ff1a015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:40152/0 (socket says 172.21.15.29:40152) 2023-10-14T16:40:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 -- 172.21.15.29:0/2463420427 learned_addr learned my addr 172.21.15.29:0/2463420427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 -- 172.21.15.29:0/2463420427 >> v2:172.21.15.186:3300/0 conn(0x7ff1a015aaa0 msgr2=0x7ff1a015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 --2- 172.21.15.29:0/2463420427 >> v2:172.21.15.186:3300/0 conn(0x7ff1a015aaa0 0x7ff1a015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 -- 172.21.15.29:0/2463420427 >> v1:172.21.15.186:6789/0 conn(0x7ff1a00ccaa0 legacy=0x7ff1a0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.562+0000 7ff19f7fe640 1 -- 172.21.15.29:0/2463420427 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1a015ef90 con 0x7ff1a015ae80 2023-10-14T16:40:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.564+0000 7ff19f7fe640 1 --2- 172.21.15.29:0/2463420427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff194009870 tx=0x7ff19402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87bbcdddd6a4314c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.564+0000 7ff19effd640 1 -- 172.21.15.29:0/2463420427 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194030840 con 0x7ff1a015ae80 2023-10-14T16:40:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.564+0000 7ff19effd640 1 -- 172.21.15.29:0/2463420427 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1940309a0 con 0x7ff1a015ae80 2023-10-14T16:40:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.564+0000 7ff19effd640 1 -- 172.21.15.29:0/2463420427 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194030d30 con 0x7ff1a015ae80 2023-10-14T16:40:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 -- 172.21.15.29:0/2463420427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 msgr2=0x7ff1a00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2463420427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ff194009870 tx=0x7ff19402b980 comp rx=0 tx=0).stop 2023-10-14T16:40:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 -- 172.21.15.29:0/2463420427 shutdown_connections 2023-10-14T16:40:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2463420427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015ae80 0x7ff1a00cc560 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2463420427 >> v2:172.21.15.186:3300/0 conn(0x7ff1a015aaa0 0x7ff1a015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 -- 172.21.15.29:0/2463420427 >> 172.21.15.29:0/2463420427 conn(0x7ff1a00cb410 msgr2=0x7ff1a00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 -- 172.21.15.29:0/2463420427 shutdown_connections 2023-10-14T16:40:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.565+0000 7ff1a654e640 1 -- 172.21.15.29:0/2463420427 wait complete. 2023-10-14T16:40:29.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.567+0000 7ff1a654e640 1 Processor -- start 2023-10-14T16:40:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.567+0000 7ff1a654e640 1 -- start start 2023-10-14T16:40:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015aaa0 0x7ff1a0156870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 0x7ff1a00b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0159a40 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a01589a0 con 0x7ff1a015ae80 2023-10-14T16:40:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.568+0000 7ff1a654e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a0158c80 con 0x7ff1a015aaa0 2023-10-14T16:40:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60862/0 (socket says 172.21.15.29:60862) 2023-10-14T16:40:29.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 -- 172.21.15.29:0/2482328050 learned_addr learned my addr 172.21.15.29:0/2482328050 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19f7fe640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015aaa0 0x7ff1a0156870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff1a4aeb640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 0x7ff1a00b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 -- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015aaa0 msgr2=0x7ff1a0156870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015aaa0 0x7ff1a0156870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 -- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 msgr2=0x7ff1a00b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 0x7ff1a00b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff19ffff640 1 -- 172.21.15.29:0/2482328050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1940094d0 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.569+0000 7ff1a4aeb640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 0x7ff1a00b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:29.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff19ffff640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff19000efc0 tx=0x7ff19000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190014480 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19000f040 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190014800 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a0158f60 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.570+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a0159520 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.571+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a004e160 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.572+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff190014960 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.573+0000 7ff19cff9640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:29.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.573+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff19008a2e0 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.573+0000 7ff19f7fe640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.574+0000 7ff19f7fe640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 0x7ff18c06d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff194036000 tx=0x7ff194008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:29.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.577+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1900552f0 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.775+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1a00cdc60 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.778+0000 7ff19cff9640 1 -- 172.21.15.29:0/2482328050 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7ff190058910 con 0x7ff1a00ccaa0 2023-10-14T16:40:29.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:29.780 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 msgr2=0x7ff18c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 0x7ff18c06d330 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff194036000 tx=0x7ff194008030 comp rx=0 tx=0).stop 2023-10-14T16:40:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 msgr2=0x7ff1a00c4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff19000efc0 tx=0x7ff19000c490 comp rx=0 tx=0).stop 2023-10-14T16:40:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 shutdown_connections 2023-10-14T16:40:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff18c06ae70 0x7ff18c06d330 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1a015ae80 0x7ff1a00b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1a015aaa0 0x7ff1a0156870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 --2- 172.21.15.29:0/2482328050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1a00ccaa0 0x7ff1a00c4cc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 >> 172.21.15.29:0/2482328050 conn(0x7ff1a00cb410 msgr2=0x7ff1a00b2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 shutdown_connections 2023-10-14T16:40:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:29.781+0000 7ff1a654e640 1 -- 172.21.15.29:0/2482328050 wait complete. 2023-10-14T16:40:29.802 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:40:29.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:40:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.032+0000 7f2ea7b16640 1 Processor -- start 2023-10-14T16:40:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.033+0000 7f2ea7b16640 1 -- start start 2023-10-14T16:40:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.034+0000 7f2ea7b16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 0x7f2ea00ce820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.034+0000 7f2ea7b16640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ea00afee0 0x7f2ea00b3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.034+0000 7f2ea7b16640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea00cee00 con 0x7f2ea015b010 2023-10-14T16:40:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.034+0000 7f2ea7b16640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea00aeba0 con 0x7f2ea00afee0 2023-10-14T16:40:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.034+0000 7f2ea7b16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea00ae270 con 0x7f2ea00cf730 2023-10-14T16:40:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.035+0000 7f2ea60b3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ea00afee0 0x7f2ea00b3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.035+0000 7f2ea50b1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 0x7f2ea00ce820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.035+0000 7f2ea60b3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ea00afee0 0x7f2ea00b3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:38480/0 (socket says 172.21.15.29:38480) 2023-10-14T16:40:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.035+0000 7f2ea60b3640 1 -- 172.21.15.29:0/3731967249 learned_addr learned my addr 172.21.15.29:0/3731967249 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.036+0000 7f2ea60b3640 1 -- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 msgr2=0x7f2ea00ce820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.036+0000 7f2ea60b3640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 0x7f2ea00ce820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.036+0000 7f2ea60b3640 1 -- 172.21.15.29:0/3731967249 >> v1:172.21.15.186:6789/0 conn(0x7f2ea015b010 legacy=0x7f2ea00cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.036+0000 7f2ea60b3640 1 -- 172.21.15.29:0/3731967249 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea00ae550 con 0x7f2ea00afee0 2023-10-14T16:40:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.036+0000 7f2ea50b1640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 0x7f2ea00ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.037+0000 7f2ea60b3640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea00afee0 0x7f2ea00b3920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009870 tx=0x7f2e9802ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f402fb170f5d2614 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.038+0000 7f2ea48b0640 1 -- 172.21.15.29:0/3731967249 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9802f830 con 0x7f2ea00afee0 2023-10-14T16:40:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.038+0000 7f2ea48b0640 1 -- 172.21.15.29:0/3731967249 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9802f990 con 0x7f2ea00afee0 2023-10-14T16:40:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.038+0000 7f2ea48b0640 1 -- 172.21.15.29:0/3731967249 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9802fd20 con 0x7f2ea00afee0 2023-10-14T16:40:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 -- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea00afee0 msgr2=0x7f2ea00b3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea00afee0 0x7f2ea00b3920 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009870 tx=0x7f2e9802ba10 comp rx=0 tx=0).stop 2023-10-14T16:40:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 -- 172.21.15.29:0/3731967249 shutdown_connections 2023-10-14T16:40:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00cf730 0x7f2ea00ce820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/3731967249 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea00afee0 0x7f2ea00b3920 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 -- 172.21.15.29:0/3731967249 >> 172.21.15.29:0/3731967249 conn(0x7f2ea00d0ad0 msgr2=0x7f2ea00d0ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 -- 172.21.15.29:0/3731967249 shutdown_connections 2023-10-14T16:40:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.039+0000 7f2ea7b16640 1 -- 172.21.15.29:0/3731967249 wait complete. 2023-10-14T16:40:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.041+0000 7f2ea7b16640 1 Processor -- start 2023-10-14T16:40:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.042+0000 7f2ea7b16640 1 -- start start 2023-10-14T16:40:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00afee0 0x7f2ea0158900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea015b010 0x7f2ea015c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea00c74d0 con 0x7f2ea00cf730 2023-10-14T16:40:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea00c6f10 con 0x7f2ea015b010 2023-10-14T16:40:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea7b16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea00c71f0 con 0x7f2ea00afee0 2023-10-14T16:40:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea50b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea50b1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60870/0 (socket says 172.21.15.29:60870) 2023-10-14T16:40:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea50b1640 1 -- 172.21.15.29:0/146726532 learned_addr learned my addr 172.21.15.29:0/146726532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea60b3640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea015b010 0x7f2ea015c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.043+0000 7f2ea58b2640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00afee0 0x7f2ea0158900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 -- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00afee0 msgr2=0x7f2ea0158900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00afee0 0x7f2ea0158900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 -- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea015b010 msgr2=0x7f2ea015c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea015b010 0x7f2ea015c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 -- 172.21.15.29:0/146726532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e90009400 con 0x7f2ea00cf730 2023-10-14T16:40:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea50b1640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000ebe0 tx=0x7f2e9000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e90009760 con 0x7f2ea00cf730 2023-10-14T16:40:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e9000cc30 con 0x7f2ea00cf730 2023-10-14T16:40:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9000eee0 con 0x7f2ea00cf730 2023-10-14T16:40:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.044+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e980094d0 con 0x7f2ea00cf730 2023-10-14T16:40:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.045+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea0145d80 con 0x7f2ea00cf730 2023-10-14T16:40:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.046+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea0145900 con 0x7f2ea00cf730 2023-10-14T16:40:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.047+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2e90013470 con 0x7f2ea00cf730 2023-10-14T16:40:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.047+0000 7f2e967fc640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 0x7f2e8406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.048+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f2e900898a0 con 0x7f2ea00cf730 2023-10-14T16:40:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.048+0000 7f2ea58b2640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 0x7f2e8406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.049+0000 7f2ea58b2640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 0x7f2e8406d360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ea01596c0 tx=0x7f2e9c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.052+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ea0145900 con 0x7f2ea00cf730 2023-10-14T16:40:30.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.249+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2ea00ae830 con 0x7f2e8406aea0 2023-10-14T16:40:30.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.274+0000 7f2e967fc640 1 -- 172.21.15.29:0/146726532 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264471 (secure 0 0 0) 0x7f2e9c0039c0 con 0x7f2e8406aea0 2023-10-14T16:40:30.279 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:30.280 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:40:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.280+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 msgr2=0x7f2e8406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.280+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 0x7f2e8406d360 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2ea01596c0 tx=0x7f2e9c009040 comp rx=0 tx=0).stop 2023-10-14T16:40:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 msgr2=0x7f2ea01f1e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000ebe0 tx=0x7f2e9000c520 comp rx=0 tx=0).stop 2023-10-14T16:40:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 shutdown_connections 2023-10-14T16:40:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2e8406aea0 0x7f2e8406d360 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ea015b010 0x7f2ea015c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ea00cf730 0x7f2ea01f1e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 --2- 172.21.15.29:0/146726532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ea00afee0 0x7f2ea0158900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 >> 172.21.15.29:0/146726532 conn(0x7f2ea00d0ad0 msgr2=0x7f2ea00cb5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 shutdown_connections 2023-10-14T16:40:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.281+0000 7f2ea7b16640 1 -- 172.21.15.29:0/146726532 wait complete. 2023-10-14T16:40:30.305 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-14T16:40:29.299909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235588,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739132,"statfs":{"total":773094113280,"available":772852871168,"internally_reserved":0,"allocated":2830336,"data_stored":1721240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.574711"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790397+0000","last_change":"2023-10-14T16:40:25.778039+0000","last_active":"2023-10-14T16:40:25.790397+0000","last_peered":"2023-10-14T16:40:25.790397+0000","last_clean":"2023-10-14T16:40:25.790397+0000","last_became_active":"2023-10-14T16:40:25.777745+0000","last_became_peered":"2023-10-14T16:40:25.777745+0000","last_unstale":"2023-10-14T16:40:25.790397+0000","last_undegraded":"2023-10-14T16:40:25.790397+0000","last_fullsized":"2023-10-14T16:40:25.790397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786508+0000","last_change":"2023-10-14T16:40:25.782038+0000","last_active":"2023-10-14T16:40:25.786508+0000","last_peered":"2023-10-14T16:40:25.786508+0000","last_clean":"2023-10-14T16:40:25.786508+0000","last_became_active":"2023-10-14T16:40:25.781722+0000","last_became_peered":"2023-10-14T16:40:25.781722+0000","last_unstale":"2023-10-14T16:40:25.786508+0000","last_undegraded":"2023-10-14T16:40:25.786508+0000","last_fullsized":"2023-10-14T16:40:25.786508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.766375+0000","last_change":"2023-10-14T16:40:25.757114+0000","last_active":"2023-10-14T16:40:25.766375+0000","last_peered":"2023-10-14T16:40:25.766375+0000","last_clean":"2023-10-14T16:40:25.766375+0000","last_became_active":"2023-10-14T16:40:25.756793+0000","last_became_peered":"2023-10-14T16:40:25.756793+0000","last_unstale":"2023-10-14T16:40:25.766375+0000","last_undegraded":"2023-10-14T16:40:25.766375+0000","last_fullsized":"2023-10-14T16:40:25.766375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793293+0000","last_change":"2023-10-14T16:40:25.788571+0000","last_active":"2023-10-14T16:40:25.793293+0000","last_peered":"2023-10-14T16:40:25.793293+0000","last_clean":"2023-10-14T16:40:25.793293+0000","last_became_active":"2023-10-14T16:40:25.788388+0000","last_became_peered":"2023-10-14T16:40:25.788388+0000","last_unstale":"2023-10-14T16:40:25.793293+0000","last_undegraded":"2023-10-14T16:40:25.793293+0000","last_fullsized":"2023-10-14T16:40:25.793293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:01.231512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.791613+0000","last_change":"2023-10-14T16:40:25.781929+0000","last_active":"2023-10-14T16:40:25.791613+0000","last_peered":"2023-10-14T16:40:25.791613+0000","last_clean":"2023-10-14T16:40:25.791613+0000","last_became_active":"2023-10-14T16:40:25.774025+0000","last_became_peered":"2023-10-14T16:40:25.774025+0000","last_unstale":"2023-10-14T16:40:25.791613+0000","last_undegraded":"2023-10-14T16:40:25.791613+0000","last_fullsized":"2023-10-14T16:40:25.791613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789735+0000","last_change":"2023-10-14T16:40:25.785668+0000","last_active":"2023-10-14T16:40:25.789735+0000","last_peered":"2023-10-14T16:40:25.789735+0000","last_clean":"2023-10-14T16:40:25.789735+0000","last_became_active":"2023-10-14T16:40:25.785300+0000","last_became_peered":"2023-10-14T16:40:25.785300+0000","last_unstale":"2023-10-14T16:40:25.789735+0000","last_undegraded":"2023-10-14T16:40:25.789735+0000","last_fullsized":"2023-10-14T16:40:25.789735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793812+0000","last_change":"2023-10-14T16:40:25.786146+0000","last_active":"2023-10-14T16:40:25.793812+0000","last_peered":"2023-10-14T16:40:25.793812+0000","last_clean":"2023-10-14T16:40:25.793812+0000","last_became_active":"2023-10-14T16:40:25.785748+0000","last_became_peered":"2023-10-14T16:40:25.785748+0000","last_unstale":"2023-10-14T16:40:25.793812+0000","last_undegraded":"2023-10-14T16:40:25.793812+0000","last_fullsized":"2023-10-14T16:40:25.793812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774509+0000","last_change":"2023-10-14T16:40:25.783832+0000","last_active":"2023-10-14T16:40:26.774509+0000","last_peered":"2023-10-14T16:40:26.774509+0000","last_clean":"2023-10-14T16:40:26.774509+0000","last_became_active":"2023-10-14T16:40:25.781635+0000","last_became_peered":"2023-10-14T16:40:25.781635+0000","last_unstale":"2023-10-14T16:40:26.774509+0000","last_undegraded":"2023-10-14T16:40:26.774509+0000","last_fullsized":"2023-10-14T16:40:26.774509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797707+0000","last_change":"2023-10-14T16:40:25.791696+0000","last_active":"2023-10-14T16:40:25.797707+0000","last_peered":"2023-10-14T16:40:25.797707+0000","last_clean":"2023-10-14T16:40:25.797707+0000","last_became_active":"2023-10-14T16:40:25.791413+0000","last_became_peered":"2023-10-14T16:40:25.791413+0000","last_unstale":"2023-10-14T16:40:25.797707+0000","last_undegraded":"2023-10-14T16:40:25.797707+0000","last_fullsized":"2023-10-14T16:40:25.797707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781741+0000","last_change":"2023-10-14T16:40:25.786842+0000","last_active":"2023-10-14T16:40:26.781741+0000","last_peered":"2023-10-14T16:40:26.781741+0000","last_clean":"2023-10-14T16:40:26.781741+0000","last_became_active":"2023-10-14T16:40:25.786528+0000","last_became_peered":"2023-10-14T16:40:25.786528+0000","last_unstale":"2023-10-14T16:40:26.781741+0000","last_undegraded":"2023-10-14T16:40:26.781741+0000","last_fullsized":"2023-10-14T16:40:26.781741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780857+0000","last_change":"2023-10-14T16:40:25.788130+0000","last_active":"2023-10-14T16:40:26.780857+0000","last_peered":"2023-10-14T16:40:26.780857+0000","last_clean":"2023-10-14T16:40:26.780857+0000","last_became_active":"2023-10-14T16:40:25.787596+0000","last_became_peered":"2023-10-14T16:40:25.787596+0000","last_unstale":"2023-10-14T16:40:26.780857+0000","last_undegraded":"2023-10-14T16:40:26.780857+0000","last_fullsized":"2023-10-14T16:40:26.780857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778866+0000","last_change":"2023-10-14T16:40:25.765437+0000","last_active":"2023-10-14T16:40:26.778866+0000","last_peered":"2023-10-14T16:40:26.778866+0000","last_clean":"2023-10-14T16:40:26.778866+0000","last_became_active":"2023-10-14T16:40:25.765116+0000","last_became_peered":"2023-10-14T16:40:25.765116+0000","last_unstale":"2023-10-14T16:40:26.778866+0000","last_undegraded":"2023-10-14T16:40:26.778866+0000","last_fullsized":"2023-10-14T16:40:26.778866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774963+0000","last_change":"2023-10-14T16:40:25.784431+0000","last_active":"2023-10-14T16:40:26.774963+0000","last_peered":"2023-10-14T16:40:26.774963+0000","last_clean":"2023-10-14T16:40:26.774963+0000","last_became_active":"2023-10-14T16:40:25.775585+0000","last_became_peered":"2023-10-14T16:40:25.775585+0000","last_unstale":"2023-10-14T16:40:26.774963+0000","last_undegraded":"2023-10-14T16:40:26.774963+0000","last_fullsized":"2023-10-14T16:40:26.774963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783410+0000","last_change":"2023-10-14T16:40:25.761749+0000","last_active":"2023-10-14T16:40:25.783410+0000","last_peered":"2023-10-14T16:40:25.783410+0000","last_clean":"2023-10-14T16:40:25.783410+0000","last_became_active":"2023-10-14T16:40:25.761444+0000","last_became_peered":"2023-10-14T16:40:25.761444+0000","last_unstale":"2023-10-14T16:40:25.783410+0000","last_undegraded":"2023-10-14T16:40:25.783410+0000","last_fullsized":"2023-10-14T16:40:25.783410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785615+0000","last_change":"2023-10-14T16:40:25.776450+0000","last_active":"2023-10-14T16:40:25.785615+0000","last_peered":"2023-10-14T16:40:25.785615+0000","last_clean":"2023-10-14T16:40:25.785615+0000","last_became_active":"2023-10-14T16:40:25.776153+0000","last_became_peered":"2023-10-14T16:40:25.776153+0000","last_unstale":"2023-10-14T16:40:25.785615+0000","last_undegraded":"2023-10-14T16:40:25.785615+0000","last_fullsized":"2023-10-14T16:40:25.785615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:01.231512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789732+0000","last_change":"2023-10-14T16:40:25.786126+0000","last_active":"2023-10-14T16:40:25.789732+0000","last_peered":"2023-10-14T16:40:25.789732+0000","last_clean":"2023-10-14T16:40:25.789732+0000","last_became_active":"2023-10-14T16:40:25.785701+0000","last_became_peered":"2023-10-14T16:40:25.785701+0000","last_unstale":"2023-10-14T16:40:25.789732+0000","last_undegraded":"2023-10-14T16:40:25.789732+0000","last_fullsized":"2023-10-14T16:40:25.789732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793424+0000","last_change":"2023-10-14T16:40:25.792223+0000","last_active":"2023-10-14T16:40:25.793424+0000","last_peered":"2023-10-14T16:40:25.793424+0000","last_clean":"2023-10-14T16:40:25.793424+0000","last_became_active":"2023-10-14T16:40:25.791860+0000","last_became_peered":"2023-10-14T16:40:25.791860+0000","last_unstale":"2023-10-14T16:40:25.793424+0000","last_undegraded":"2023-10-14T16:40:25.793424+0000","last_fullsized":"2023-10-14T16:40:25.793424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776199+0000","last_change":"2023-10-14T16:40:25.766839+0000","last_active":"2023-10-14T16:40:27.776199+0000","last_peered":"2023-10-14T16:40:27.776199+0000","last_clean":"2023-10-14T16:40:27.776199+0000","last_became_active":"2023-10-14T16:40:25.766493+0000","last_became_peered":"2023-10-14T16:40:25.766493+0000","last_unstale":"2023-10-14T16:40:27.776199+0000","last_undegraded":"2023-10-14T16:40:27.776199+0000","last_fullsized":"2023-10-14T16:40:27.776199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797728+0000","last_change":"2023-10-14T16:40:25.792353+0000","last_active":"2023-10-14T16:40:25.797728+0000","last_peered":"2023-10-14T16:40:25.797728+0000","last_clean":"2023-10-14T16:40:25.797728+0000","last_became_active":"2023-10-14T16:40:25.792040+0000","last_became_peered":"2023-10-14T16:40:25.792040+0000","last_unstale":"2023-10-14T16:40:25.797728+0000","last_undegraded":"2023-10-14T16:40:25.797728+0000","last_fullsized":"2023-10-14T16:40:25.797728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781176+0000","last_change":"2023-10-14T16:40:25.770102+0000","last_active":"2023-10-14T16:40:26.781176+0000","last_peered":"2023-10-14T16:40:26.781176+0000","last_clean":"2023-10-14T16:40:26.781176+0000","last_became_active":"2023-10-14T16:40:25.769804+0000","last_became_peered":"2023-10-14T16:40:25.769804+0000","last_unstale":"2023-10-14T16:40:26.781176+0000","last_undegraded":"2023-10-14T16:40:26.781176+0000","last_fullsized":"2023-10-14T16:40:26.781176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784916+0000","last_change":"2023-10-14T16:40:25.779852+0000","last_active":"2023-10-14T16:40:25.784916+0000","last_peered":"2023-10-14T16:40:25.784916+0000","last_clean":"2023-10-14T16:40:25.784916+0000","last_became_active":"2023-10-14T16:40:25.779362+0000","last_became_peered":"2023-10-14T16:40:25.779362+0000","last_unstale":"2023-10-14T16:40:25.784916+0000","last_undegraded":"2023-10-14T16:40:25.784916+0000","last_fullsized":"2023-10-14T16:40:25.784916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:25:00.025546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786861+0000","last_change":"2023-10-14T16:40:25.781280+0000","last_active":"2023-10-14T16:40:25.786861+0000","last_peered":"2023-10-14T16:40:25.786861+0000","last_clean":"2023-10-14T16:40:25.786861+0000","last_became_active":"2023-10-14T16:40:25.780595+0000","last_became_peered":"2023-10-14T16:40:25.780595+0000","last_unstale":"2023-10-14T16:40:25.786861+0000","last_undegraded":"2023-10-14T16:40:25.786861+0000","last_fullsized":"2023-10-14T16:40:25.786861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:39.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774501+0000","last_change":"2023-10-14T16:40:25.777771+0000","last_active":"2023-10-14T16:40:26.774501+0000","last_peered":"2023-10-14T16:40:26.774501+0000","last_clean":"2023-10-14T16:40:26.774501+0000","last_became_active":"2023-10-14T16:40:25.777508+0000","last_became_peered":"2023-10-14T16:40:25.777508+0000","last_unstale":"2023-10-14T16:40:26.774501+0000","last_undegraded":"2023-10-14T16:40:26.774501+0000","last_fullsized":"2023-10-14T16:40:26.774501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783460+0000","last_change":"2023-10-14T16:40:25.772029+0000","last_active":"2023-10-14T16:40:25.783460+0000","last_peered":"2023-10-14T16:40:25.783460+0000","last_clean":"2023-10-14T16:40:25.783460+0000","last_became_active":"2023-10-14T16:40:25.771700+0000","last_became_peered":"2023-10-14T16:40:25.771700+0000","last_unstale":"2023-10-14T16:40:25.783460+0000","last_undegraded":"2023-10-14T16:40:25.783460+0000","last_fullsized":"2023-10-14T16:40:25.783460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775784+0000","last_change":"2023-10-14T16:40:25.777077+0000","last_active":"2023-10-14T16:40:26.775784+0000","last_peered":"2023-10-14T16:40:26.775784+0000","last_clean":"2023-10-14T16:40:26.775784+0000","last_became_active":"2023-10-14T16:40:25.776755+0000","last_became_peered":"2023-10-14T16:40:25.776755+0000","last_unstale":"2023-10-14T16:40:26.775784+0000","last_undegraded":"2023-10-14T16:40:26.775784+0000","last_fullsized":"2023-10-14T16:40:26.775784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781826+0000","last_change":"2023-10-14T16:40:25.777054+0000","last_active":"2023-10-14T16:40:26.781826+0000","last_peered":"2023-10-14T16:40:26.781826+0000","last_clean":"2023-10-14T16:40:26.781826+0000","last_became_active":"2023-10-14T16:40:25.773588+0000","last_became_peered":"2023-10-14T16:40:25.773588+0000","last_unstale":"2023-10-14T16:40:26.781826+0000","last_undegraded":"2023-10-14T16:40:26.781826+0000","last_fullsized":"2023-10-14T16:40:26.781826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790984+0000","last_change":"2023-10-14T16:40:25.787594+0000","last_active":"2023-10-14T16:40:25.790984+0000","last_peered":"2023-10-14T16:40:25.790984+0000","last_clean":"2023-10-14T16:40:25.790984+0000","last_became_active":"2023-10-14T16:40:25.786833+0000","last_became_peered":"2023-10-14T16:40:25.786833+0000","last_unstale":"2023-10-14T16:40:25.790984+0000","last_undegraded":"2023-10-14T16:40:25.790984+0000","last_fullsized":"2023-10-14T16:40:25.790984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775301+0000","last_change":"2023-10-14T16:40:25.782397+0000","last_active":"2023-10-14T16:40:26.775301+0000","last_peered":"2023-10-14T16:40:26.775301+0000","last_clean":"2023-10-14T16:40:26.775301+0000","last_became_active":"2023-10-14T16:40:25.782109+0000","last_became_peered":"2023-10-14T16:40:25.782109+0000","last_unstale":"2023-10-14T16:40:26.775301+0000","last_undegraded":"2023-10-14T16:40:26.775301+0000","last_fullsized":"2023-10-14T16:40:26.775301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788167+0000","last_change":"2023-10-14T16:40:25.783124+0000","last_active":"2023-10-14T16:40:25.788167+0000","last_peered":"2023-10-14T16:40:25.788167+0000","last_clean":"2023-10-14T16:40:25.788167+0000","last_became_active":"2023-10-14T16:40:25.782805+0000","last_became_peered":"2023-10-14T16:40:25.782805+0000","last_unstale":"2023-10-14T16:40:25.788167+0000","last_undegraded":"2023-10-14T16:40:25.788167+0000","last_fullsized":"2023-10-14T16:40:25.788167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786881+0000","last_change":"2023-10-14T16:40:25.784068+0000","last_active":"2023-10-14T16:40:25.786881+0000","last_peered":"2023-10-14T16:40:25.786881+0000","last_clean":"2023-10-14T16:40:25.786881+0000","last_became_active":"2023-10-14T16:40:25.783671+0000","last_became_peered":"2023-10-14T16:40:25.783671+0000","last_unstale":"2023-10-14T16:40:25.786881+0000","last_undegraded":"2023-10-14T16:40:25.786881+0000","last_fullsized":"2023-10-14T16:40:25.786881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775610+0000","last_change":"2023-10-14T16:40:25.777635+0000","last_active":"2023-10-14T16:40:27.775610+0000","last_peered":"2023-10-14T16:40:27.775610+0000","last_clean":"2023-10-14T16:40:27.775610+0000","last_became_active":"2023-10-14T16:40:25.777400+0000","last_became_peered":"2023-10-14T16:40:25.777400+0000","last_unstale":"2023-10-14T16:40:27.775610+0000","last_undegraded":"2023-10-14T16:40:27.775610+0000","last_fullsized":"2023-10-14T16:40:27.775610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779941+0000","last_change":"2023-10-14T16:40:25.758613+0000","last_active":"2023-10-14T16:40:26.779941+0000","last_peered":"2023-10-14T16:40:26.779941+0000","last_clean":"2023-10-14T16:40:26.779941+0000","last_became_active":"2023-10-14T16:40:25.758173+0000","last_became_peered":"2023-10-14T16:40:25.758173+0000","last_unstale":"2023-10-14T16:40:26.779941+0000","last_undegraded":"2023-10-14T16:40:26.779941+0000","last_fullsized":"2023-10-14T16:40:26.779941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778458+0000","last_change":"2023-10-14T16:40:25.759820+0000","last_active":"2023-10-14T16:40:26.778458+0000","last_peered":"2023-10-14T16:40:26.778458+0000","last_clean":"2023-10-14T16:40:26.778458+0000","last_became_active":"2023-10-14T16:40:25.759498+0000","last_became_peered":"2023-10-14T16:40:25.759498+0000","last_unstale":"2023-10-14T16:40:26.778458+0000","last_undegraded":"2023-10-14T16:40:26.778458+0000","last_fullsized":"2023-10-14T16:40:26.778458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775078+0000","last_change":"2023-10-14T16:40:24.768741+0000","last_active":"2023-10-14T16:40:26.775078+0000","last_peered":"2023-10-14T16:40:26.775078+0000","last_clean":"2023-10-14T16:40:26.775078+0000","last_became_active":"2023-10-14T16:40:24.768379+0000","last_became_peered":"2023-10-14T16:40:24.768379+0000","last_unstale":"2023-10-14T16:40:26.775078+0000","last_undegraded":"2023-10-14T16:40:26.775078+0000","last_fullsized":"2023-10-14T16:40:26.775078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784898+0000","last_change":"2023-10-14T16:40:25.779646+0000","last_active":"2023-10-14T16:40:25.784898+0000","last_peered":"2023-10-14T16:40:25.784898+0000","last_clean":"2023-10-14T16:40:25.784898+0000","last_became_active":"2023-10-14T16:40:25.779001+0000","last_became_peered":"2023-10-14T16:40:25.779001+0000","last_unstale":"2023-10-14T16:40:25.784898+0000","last_undegraded":"2023-10-14T16:40:25.784898+0000","last_fullsized":"2023-10-14T16:40:25.784898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780736+0000","last_change":"2023-10-14T16:40:24.763894+0000","last_active":"2023-10-14T16:40:26.780736+0000","last_peered":"2023-10-14T16:40:26.780736+0000","last_clean":"2023-10-14T16:40:26.780736+0000","last_became_active":"2023-10-14T16:40:24.762343+0000","last_became_peered":"2023-10-14T16:40:24.762343+0000","last_unstale":"2023-10-14T16:40:26.780736+0000","last_undegraded":"2023-10-14T16:40:26.780736+0000","last_fullsized":"2023-10-14T16:40:26.780736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774426+0000","last_change":"2023-10-14T16:40:24.775055+0000","last_active":"2023-10-14T16:40:26.774426+0000","last_peered":"2023-10-14T16:40:26.774426+0000","last_clean":"2023-10-14T16:40:26.774426+0000","last_became_active":"2023-10-14T16:40:24.774256+0000","last_became_peered":"2023-10-14T16:40:24.774256+0000","last_unstale":"2023-10-14T16:40:26.774426+0000","last_undegraded":"2023-10-14T16:40:26.774426+0000","last_fullsized":"2023-10-14T16:40:26.774426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774375+0000","last_change":"2023-10-14T16:40:25.781085+0000","last_active":"2023-10-14T16:40:26.774375+0000","last_peered":"2023-10-14T16:40:26.774375+0000","last_clean":"2023-10-14T16:40:26.774375+0000","last_became_active":"2023-10-14T16:40:25.780746+0000","last_became_peered":"2023-10-14T16:40:25.780746+0000","last_unstale":"2023-10-14T16:40:26.774375+0000","last_undegraded":"2023-10-14T16:40:26.774375+0000","last_fullsized":"2023-10-14T16:40:26.774375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.768784+0000","last_change":"2023-10-14T16:40:24.763402+0000","last_active":"2023-10-14T16:40:25.768784+0000","last_peered":"2023-10-14T16:40:25.768784+0000","last_clean":"2023-10-14T16:40:25.768784+0000","last_became_active":"2023-10-14T16:40:24.763093+0000","last_became_peered":"2023-10-14T16:40:24.763093+0000","last_unstale":"2023-10-14T16:40:25.768784+0000","last_undegraded":"2023-10-14T16:40:25.768784+0000","last_fullsized":"2023-10-14T16:40:25.768784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786384+0000","last_change":"2023-10-14T16:40:25.780063+0000","last_active":"2023-10-14T16:40:25.786384+0000","last_peered":"2023-10-14T16:40:25.786384+0000","last_clean":"2023-10-14T16:40:25.786384+0000","last_became_active":"2023-10-14T16:40:25.776862+0000","last_became_peered":"2023-10-14T16:40:25.776862+0000","last_unstale":"2023-10-14T16:40:25.786384+0000","last_undegraded":"2023-10-14T16:40:25.786384+0000","last_fullsized":"2023-10-14T16:40:25.786384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781765+0000","last_change":"2023-10-14T16:40:24.768554+0000","last_active":"2023-10-14T16:40:26.781765+0000","last_peered":"2023-10-14T16:40:26.781765+0000","last_clean":"2023-10-14T16:40:26.781765+0000","last_became_active":"2023-10-14T16:40:24.768232+0000","last_became_peered":"2023-10-14T16:40:24.768232+0000","last_unstale":"2023-10-14T16:40:26.781765+0000","last_undegraded":"2023-10-14T16:40:26.781765+0000","last_fullsized":"2023-10-14T16:40:26.781765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:59.941812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785704+0000","last_change":"2023-10-14T16:40:25.778676+0000","last_active":"2023-10-14T16:40:25.785704+0000","last_peered":"2023-10-14T16:40:25.785704+0000","last_clean":"2023-10-14T16:40:25.785704+0000","last_became_active":"2023-10-14T16:40:25.778295+0000","last_became_peered":"2023-10-14T16:40:25.778295+0000","last_unstale":"2023-10-14T16:40:25.785704+0000","last_undegraded":"2023-10-14T16:40:25.785704+0000","last_fullsized":"2023-10-14T16:40:25.785704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.770673+0000","last_change":"2023-10-14T16:40:24.764655+0000","last_active":"2023-10-14T16:40:25.770673+0000","last_peered":"2023-10-14T16:40:25.770673+0000","last_clean":"2023-10-14T16:40:25.770673+0000","last_became_active":"2023-10-14T16:40:24.764327+0000","last_became_peered":"2023-10-14T16:40:24.764327+0000","last_unstale":"2023-10-14T16:40:25.770673+0000","last_undegraded":"2023-10-14T16:40:25.770673+0000","last_fullsized":"2023-10-14T16:40:25.770673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.733975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788129+0000","last_change":"2023-10-14T16:40:25.783442+0000","last_active":"2023-10-14T16:40:25.788129+0000","last_peered":"2023-10-14T16:40:25.788129+0000","last_clean":"2023-10-14T16:40:25.788129+0000","last_became_active":"2023-10-14T16:40:25.783169+0000","last_became_peered":"2023-10-14T16:40:25.783169+0000","last_unstale":"2023-10-14T16:40:25.788129+0000","last_undegraded":"2023-10-14T16:40:25.788129+0000","last_fullsized":"2023-10-14T16:40:25.788129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:04.110533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.787415+0000","last_change":"2023-10-14T16:40:24.767382+0000","last_active":"2023-10-14T16:40:25.787415+0000","last_peered":"2023-10-14T16:40:25.787415+0000","last_clean":"2023-10-14T16:40:25.787415+0000","last_became_active":"2023-10-14T16:40:24.767094+0000","last_became_peered":"2023-10-14T16:40:24.767094+0000","last_unstale":"2023-10-14T16:40:25.787415+0000","last_undegraded":"2023-10-14T16:40:25.787415+0000","last_fullsized":"2023-10-14T16:40:25.787415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778953+0000","last_change":"2023-10-14T16:40:25.776539+0000","last_active":"2023-10-14T16:40:26.778953+0000","last_peered":"2023-10-14T16:40:26.778953+0000","last_clean":"2023-10-14T16:40:26.778953+0000","last_became_active":"2023-10-14T16:40:25.776126+0000","last_became_peered":"2023-10-14T16:40:25.776126+0000","last_unstale":"2023-10-14T16:40:26.778953+0000","last_undegraded":"2023-10-14T16:40:26.778953+0000","last_fullsized":"2023-10-14T16:40:26.778953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781420+0000","last_change":"2023-10-14T16:40:24.768941+0000","last_active":"2023-10-14T16:40:26.781420+0000","last_peered":"2023-10-14T16:40:26.781420+0000","last_clean":"2023-10-14T16:40:26.781420+0000","last_became_active":"2023-10-14T16:40:24.767075+0000","last_became_peered":"2023-10-14T16:40:24.767075+0000","last_unstale":"2023-10-14T16:40:26.781420+0000","last_undegraded":"2023-10-14T16:40:26.781420+0000","last_fullsized":"2023-10-14T16:40:26.781420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794708+0000","last_change":"2023-10-14T16:40:25.791099+0000","last_active":"2023-10-14T16:40:25.794708+0000","last_peered":"2023-10-14T16:40:25.794708+0000","last_clean":"2023-10-14T16:40:25.794708+0000","last_became_active":"2023-10-14T16:40:25.790819+0000","last_became_peered":"2023-10-14T16:40:25.790819+0000","last_unstale":"2023-10-14T16:40:25.794708+0000","last_undegraded":"2023-10-14T16:40:25.794708+0000","last_fullsized":"2023-10-14T16:40:25.794708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793667+0000","last_change":"2023-10-14T16:40:24.760669+0000","last_active":"2023-10-14T16:40:25.793667+0000","last_peered":"2023-10-14T16:40:25.793667+0000","last_clean":"2023-10-14T16:40:25.793667+0000","last_became_active":"2023-10-14T16:40:24.760395+0000","last_became_peered":"2023-10-14T16:40:24.760395+0000","last_unstale":"2023-10-14T16:40:25.793667+0000","last_undegraded":"2023-10-14T16:40:25.793667+0000","last_fullsized":"2023-10-14T16:40:25.793667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784659+0000","last_change":"2023-10-14T16:40:25.779561+0000","last_active":"2023-10-14T16:40:25.784659+0000","last_peered":"2023-10-14T16:40:25.784659+0000","last_clean":"2023-10-14T16:40:25.784659+0000","last_became_active":"2023-10-14T16:40:25.779246+0000","last_became_peered":"2023-10-14T16:40:25.779246+0000","last_unstale":"2023-10-14T16:40:25.784659+0000","last_undegraded":"2023-10-14T16:40:25.784659+0000","last_fullsized":"2023-10-14T16:40:25.784659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769539+0000","last_change":"2023-10-14T16:40:24.764378+0000","last_active":"2023-10-14T16:40:25.769539+0000","last_peered":"2023-10-14T16:40:25.769539+0000","last_clean":"2023-10-14T16:40:25.769539+0000","last_became_active":"2023-10-14T16:40:24.764094+0000","last_became_peered":"2023-10-14T16:40:24.764094+0000","last_unstale":"2023-10-14T16:40:25.769539+0000","last_undegraded":"2023-10-14T16:40:25.769539+0000","last_fullsized":"2023-10-14T16:40:25.769539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:55.307930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793123+0000","last_change":"2023-10-14T16:40:25.782667+0000","last_active":"2023-10-14T16:40:25.793123+0000","last_peered":"2023-10-14T16:40:25.793123+0000","last_clean":"2023-10-14T16:40:25.793123+0000","last_became_active":"2023-10-14T16:40:25.782311+0000","last_became_peered":"2023-10-14T16:40:25.782311+0000","last_unstale":"2023-10-14T16:40:25.793123+0000","last_undegraded":"2023-10-14T16:40:25.793123+0000","last_fullsized":"2023-10-14T16:40:25.793123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780460+0000","last_change":"2023-10-14T16:40:24.765776+0000","last_active":"2023-10-14T16:40:26.780460+0000","last_peered":"2023-10-14T16:40:26.780460+0000","last_clean":"2023-10-14T16:40:26.780460+0000","last_became_active":"2023-10-14T16:40:24.761753+0000","last_became_peered":"2023-10-14T16:40:24.761753+0000","last_unstale":"2023-10-14T16:40:26.780460+0000","last_undegraded":"2023-10-14T16:40:26.780460+0000","last_fullsized":"2023-10-14T16:40:26.780460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780507+0000","last_change":"2023-10-14T16:40:25.761768+0000","last_active":"2023-10-14T16:40:26.780507+0000","last_peered":"2023-10-14T16:40:26.780507+0000","last_clean":"2023-10-14T16:40:26.780507+0000","last_became_active":"2023-10-14T16:40:25.761374+0000","last_became_peered":"2023-10-14T16:40:25.761374+0000","last_unstale":"2023-10-14T16:40:26.780507+0000","last_undegraded":"2023-10-14T16:40:26.780507+0000","last_fullsized":"2023-10-14T16:40:26.780507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775466+0000","last_change":"2023-10-14T16:40:24.758887+0000","last_active":"2023-10-14T16:40:27.775466+0000","last_peered":"2023-10-14T16:40:27.775466+0000","last_clean":"2023-10-14T16:40:27.775466+0000","last_became_active":"2023-10-14T16:40:24.758086+0000","last_became_peered":"2023-10-14T16:40:24.758086+0000","last_unstale":"2023-10-14T16:40:27.775466+0000","last_undegraded":"2023-10-14T16:40:27.775466+0000","last_fullsized":"2023-10-14T16:40:27.775466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:03.099034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797811+0000","last_change":"2023-10-14T16:40:25.793027+0000","last_active":"2023-10-14T16:40:25.797811+0000","last_peered":"2023-10-14T16:40:25.797811+0000","last_clean":"2023-10-14T16:40:25.797811+0000","last_became_active":"2023-10-14T16:40:25.791542+0000","last_became_peered":"2023-10-14T16:40:25.791542+0000","last_unstale":"2023-10-14T16:40:25.797811+0000","last_undegraded":"2023-10-14T16:40:25.797811+0000","last_fullsized":"2023-10-14T16:40:25.797811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779724+0000","last_change":"2023-10-14T16:40:24.763877+0000","last_active":"2023-10-14T16:40:26.779724+0000","last_peered":"2023-10-14T16:40:26.779724+0000","last_clean":"2023-10-14T16:40:26.779724+0000","last_became_active":"2023-10-14T16:40:24.763637+0000","last_became_peered":"2023-10-14T16:40:24.763637+0000","last_unstale":"2023-10-14T16:40:26.779724+0000","last_undegraded":"2023-10-14T16:40:26.779724+0000","last_fullsized":"2023-10-14T16:40:26.779724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797721+0000","last_change":"2023-10-14T16:40:25.792192+0000","last_active":"2023-10-14T16:40:25.797721+0000","last_peered":"2023-10-14T16:40:25.797721+0000","last_clean":"2023-10-14T16:40:25.797721+0000","last_became_active":"2023-10-14T16:40:25.791748+0000","last_became_peered":"2023-10-14T16:40:25.791748+0000","last_unstale":"2023-10-14T16:40:25.797721+0000","last_undegraded":"2023-10-14T16:40:25.797721+0000","last_fullsized":"2023-10-14T16:40:25.797721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776156+0000","last_change":"2023-10-14T16:40:24.766861+0000","last_active":"2023-10-14T16:40:27.776156+0000","last_peered":"2023-10-14T16:40:27.776156+0000","last_clean":"2023-10-14T16:40:27.776156+0000","last_became_active":"2023-10-14T16:40:24.766319+0000","last_became_peered":"2023-10-14T16:40:24.766319+0000","last_unstale":"2023-10-14T16:40:27.776156+0000","last_undegraded":"2023-10-14T16:40:27.776156+0000","last_fullsized":"2023-10-14T16:40:27.776156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794264+0000","last_change":"2023-10-14T16:40:25.782879+0000","last_active":"2023-10-14T16:40:25.794264+0000","last_peered":"2023-10-14T16:40:25.794264+0000","last_clean":"2023-10-14T16:40:25.794264+0000","last_became_active":"2023-10-14T16:40:25.782368+0000","last_became_peered":"2023-10-14T16:40:25.782368+0000","last_unstale":"2023-10-14T16:40:25.794264+0000","last_undegraded":"2023-10-14T16:40:25.794264+0000","last_fullsized":"2023-10-14T16:40:25.794264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789069+0000","last_change":"2023-10-14T16:40:24.763429+0000","last_active":"2023-10-14T16:40:25.789069+0000","last_peered":"2023-10-14T16:40:25.789069+0000","last_clean":"2023-10-14T16:40:25.789069+0000","last_became_active":"2023-10-14T16:40:24.761118+0000","last_became_peered":"2023-10-14T16:40:24.761118+0000","last_unstale":"2023-10-14T16:40:25.789069+0000","last_undegraded":"2023-10-14T16:40:25.789069+0000","last_fullsized":"2023-10-14T16:40:25.789069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789505+0000","last_change":"2023-10-14T16:40:25.784774+0000","last_active":"2023-10-14T16:40:25.789505+0000","last_peered":"2023-10-14T16:40:25.789505+0000","last_clean":"2023-10-14T16:40:25.789505+0000","last_became_active":"2023-10-14T16:40:25.784416+0000","last_became_peered":"2023-10-14T16:40:25.784416+0000","last_unstale":"2023-10-14T16:40:25.789505+0000","last_undegraded":"2023-10-14T16:40:25.789505+0000","last_fullsized":"2023-10-14T16:40:25.789505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775852+0000","last_change":"2023-10-14T16:40:24.767764+0000","last_active":"2023-10-14T16:40:27.775852+0000","last_peered":"2023-10-14T16:40:27.775852+0000","last_clean":"2023-10-14T16:40:27.775852+0000","last_became_active":"2023-10-14T16:40:24.767467+0000","last_became_peered":"2023-10-14T16:40:24.767467+0000","last_unstale":"2023-10-14T16:40:27.775852+0000","last_undegraded":"2023-10-14T16:40:27.775852+0000","last_fullsized":"2023-10-14T16:40:27.775852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797777+0000","last_change":"2023-10-14T16:40:25.792940+0000","last_active":"2023-10-14T16:40:25.797777+0000","last_peered":"2023-10-14T16:40:25.797777+0000","last_clean":"2023-10-14T16:40:25.797777+0000","last_became_active":"2023-10-14T16:40:25.790465+0000","last_became_peered":"2023-10-14T16:40:25.790465+0000","last_unstale":"2023-10-14T16:40:25.797777+0000","last_undegraded":"2023-10-14T16:40:25.797777+0000","last_fullsized":"2023-10-14T16:40:25.797777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-14T16:40:24.771054+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:40:24.771054+0000","last_peered":"2023-10-14T16:40:24.771054+0000","last_clean":"2023-10-14T16:40:24.771054+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:40:24.771054+0000","last_undegraded":"2023-10-14T16:40:24.771054+0000","last_fullsized":"2023-10-14T16:40:24.771054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769523+0000","last_change":"2023-10-14T16:40:24.765633+0000","last_active":"2023-10-14T16:40:25.769523+0000","last_peered":"2023-10-14T16:40:25.769523+0000","last_clean":"2023-10-14T16:40:25.769523+0000","last_became_active":"2023-10-14T16:40:24.760310+0000","last_became_peered":"2023-10-14T16:40:24.760310+0000","last_unstale":"2023-10-14T16:40:25.769523+0000","last_undegraded":"2023-10-14T16:40:25.769523+0000","last_fullsized":"2023-10-14T16:40:25.769523+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:42.771145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775084+0000","last_change":"2023-10-14T16:40:25.783431+0000","last_active":"2023-10-14T16:40:26.775084+0000","last_peered":"2023-10-14T16:40:26.775084+0000","last_clean":"2023-10-14T16:40:26.775084+0000","last_became_active":"2023-10-14T16:40:25.782668+0000","last_became_peered":"2023-10-14T16:40:25.782668+0000","last_unstale":"2023-10-14T16:40:26.775084+0000","last_undegraded":"2023-10-14T16:40:26.775084+0000","last_fullsized":"2023-10-14T16:40:26.775084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.758384+0000","last_change":"2023-10-14T16:40:24.774157+0000","last_active":"2023-10-14T16:40:25.758384+0000","last_peered":"2023-10-14T16:40:25.758384+0000","last_clean":"2023-10-14T16:40:25.758384+0000","last_became_active":"2023-10-14T16:40:24.773773+0000","last_became_peered":"2023-10-14T16:40:24.773773+0000","last_unstale":"2023-10-14T16:40:25.758384+0000","last_undegraded":"2023-10-14T16:40:25.758384+0000","last_fullsized":"2023-10-14T16:40:25.758384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785039+0000","last_change":"2023-10-14T16:40:25.772755+0000","last_active":"2023-10-14T16:40:25.785039+0000","last_peered":"2023-10-14T16:40:25.785039+0000","last_clean":"2023-10-14T16:40:25.785039+0000","last_became_active":"2023-10-14T16:40:25.772431+0000","last_became_peered":"2023-10-14T16:40:25.772431+0000","last_unstale":"2023-10-14T16:40:25.785039+0000","last_undegraded":"2023-10-14T16:40:25.785039+0000","last_fullsized":"2023-10-14T16:40:25.785039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775795+0000","last_change":"2023-10-14T16:40:24.770677+0000","last_active":"2023-10-14T16:40:27.775795+0000","last_peered":"2023-10-14T16:40:27.775795+0000","last_clean":"2023-10-14T16:40:27.775795+0000","last_became_active":"2023-10-14T16:40:24.770259+0000","last_became_peered":"2023-10-14T16:40:24.770259+0000","last_unstale":"2023-10-14T16:40:27.775795+0000","last_undegraded":"2023-10-14T16:40:27.775795+0000","last_fullsized":"2023-10-14T16:40:27.775795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785363+0000","last_change":"2023-10-14T16:40:25.768484+0000","last_active":"2023-10-14T16:40:25.785363+0000","last_peered":"2023-10-14T16:40:25.785363+0000","last_clean":"2023-10-14T16:40:25.785363+0000","last_became_active":"2023-10-14T16:40:25.767865+0000","last_became_peered":"2023-10-14T16:40:25.767865+0000","last_unstale":"2023-10-14T16:40:25.785363+0000","last_undegraded":"2023-10-14T16:40:25.785363+0000","last_fullsized":"2023-10-14T16:40:25.785363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769671+0000","last_change":"2023-10-14T16:40:24.762562+0000","last_active":"2023-10-14T16:40:25.769671+0000","last_peered":"2023-10-14T16:40:25.769671+0000","last_clean":"2023-10-14T16:40:25.769671+0000","last_became_active":"2023-10-14T16:40:24.761399+0000","last_became_peered":"2023-10-14T16:40:24.761399+0000","last_unstale":"2023-10-14T16:40:25.769671+0000","last_undegraded":"2023-10-14T16:40:25.769671+0000","last_fullsized":"2023-10-14T16:40:25.769671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:40.569040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778450+0000","last_change":"2023-10-14T16:40:25.767520+0000","last_active":"2023-10-14T16:40:26.778450+0000","last_peered":"2023-10-14T16:40:26.778450+0000","last_clean":"2023-10-14T16:40:26.778450+0000","last_became_active":"2023-10-14T16:40:25.765947+0000","last_became_peered":"2023-10-14T16:40:25.765947+0000","last_unstale":"2023-10-14T16:40:26.778450+0000","last_undegraded":"2023-10-14T16:40:26.778450+0000","last_fullsized":"2023-10-14T16:40:26.778450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.751137+0000","last_change":"2023-10-14T16:40:24.758028+0000","last_active":"2023-10-14T16:40:25.751137+0000","last_peered":"2023-10-14T16:40:25.751137+0000","last_clean":"2023-10-14T16:40:25.751137+0000","last_became_active":"2023-10-14T16:40:24.757727+0000","last_became_peered":"2023-10-14T16:40:24.757727+0000","last_unstale":"2023-10-14T16:40:25.751137+0000","last_undegraded":"2023-10-14T16:40:25.751137+0000","last_fullsized":"2023-10-14T16:40:25.751137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793310+0000","last_change":"2023-10-14T16:40:25.789247+0000","last_active":"2023-10-14T16:40:25.793310+0000","last_peered":"2023-10-14T16:40:25.793310+0000","last_clean":"2023-10-14T16:40:25.793310+0000","last_became_active":"2023-10-14T16:40:25.788936+0000","last_became_peered":"2023-10-14T16:40:25.788936+0000","last_unstale":"2023-10-14T16:40:25.793310+0000","last_undegraded":"2023-10-14T16:40:25.793310+0000","last_fullsized":"2023-10-14T16:40:25.793310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.764174+0000","last_change":"2023-10-14T16:40:24.776223+0000","last_active":"2023-10-14T16:40:25.764174+0000","last_peered":"2023-10-14T16:40:25.764174+0000","last_clean":"2023-10-14T16:40:25.764174+0000","last_became_active":"2023-10-14T16:40:24.775651+0000","last_became_peered":"2023-10-14T16:40:24.775651+0000","last_unstale":"2023-10-14T16:40:25.764174+0000","last_undegraded":"2023-10-14T16:40:25.764174+0000","last_fullsized":"2023-10-14T16:40:25.764174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783772+0000","last_change":"2023-10-14T16:40:25.768945+0000","last_active":"2023-10-14T16:40:25.783772+0000","last_peered":"2023-10-14T16:40:25.783772+0000","last_clean":"2023-10-14T16:40:25.783772+0000","last_became_active":"2023-10-14T16:40:25.768628+0000","last_became_peered":"2023-10-14T16:40:25.768628+0000","last_unstale":"2023-10-14T16:40:25.783772+0000","last_undegraded":"2023-10-14T16:40:25.783772+0000","last_fullsized":"2023-10-14T16:40:25.783772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.756781+0000","last_change":"2023-10-14T16:40:24.768276+0000","last_active":"2023-10-14T16:40:25.756781+0000","last_peered":"2023-10-14T16:40:25.756781+0000","last_clean":"2023-10-14T16:40:25.756781+0000","last_became_active":"2023-10-14T16:40:24.766828+0000","last_became_peered":"2023-10-14T16:40:24.766828+0000","last_unstale":"2023-10-14T16:40:25.756781+0000","last_undegraded":"2023-10-14T16:40:25.756781+0000","last_fullsized":"2023-10-14T16:40:25.756781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775754+0000","last_change":"2023-10-14T16:40:25.779777+0000","last_active":"2023-10-14T16:40:26.775754+0000","last_peered":"2023-10-14T16:40:26.775754+0000","last_clean":"2023-10-14T16:40:26.775754+0000","last_became_active":"2023-10-14T16:40:25.779380+0000","last_became_peered":"2023-10-14T16:40:25.779380+0000","last_unstale":"2023-10-14T16:40:26.775754+0000","last_undegraded":"2023-10-14T16:40:26.775754+0000","last_fullsized":"2023-10-14T16:40:26.775754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781895+0000","last_change":"2023-10-14T16:40:24.773797+0000","last_active":"2023-10-14T16:40:26.781895+0000","last_peered":"2023-10-14T16:40:26.781895+0000","last_clean":"2023-10-14T16:40:26.781895+0000","last_became_active":"2023-10-14T16:40:24.773498+0000","last_became_peered":"2023-10-14T16:40:24.773498+0000","last_unstale":"2023-10-14T16:40:26.781895+0000","last_undegraded":"2023-10-14T16:40:26.781895+0000","last_fullsized":"2023-10-14T16:40:26.781895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783586+0000","last_change":"2023-10-14T16:40:25.766374+0000","last_active":"2023-10-14T16:40:25.783586+0000","last_peered":"2023-10-14T16:40:25.783586+0000","last_clean":"2023-10-14T16:40:25.783586+0000","last_became_active":"2023-10-14T16:40:25.764271+0000","last_became_peered":"2023-10-14T16:40:25.764271+0000","last_unstale":"2023-10-14T16:40:25.783586+0000","last_undegraded":"2023-10-14T16:40:25.783586+0000","last_fullsized":"2023-10-14T16:40:25.783586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.759520+0000","last_change":"2023-10-14T16:40:24.775192+0000","last_active":"2023-10-14T16:40:25.759520+0000","last_peered":"2023-10-14T16:40:25.759520+0000","last_clean":"2023-10-14T16:40:25.759520+0000","last_became_active":"2023-10-14T16:40:24.774186+0000","last_became_peered":"2023-10-14T16:40:24.774186+0000","last_unstale":"2023-10-14T16:40:25.759520+0000","last_undegraded":"2023-10-14T16:40:25.759520+0000","last_fullsized":"2023-10-14T16:40:25.759520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778279+0000","last_change":"2023-10-14T16:40:25.769524+0000","last_active":"2023-10-14T16:40:26.778279+0000","last_peered":"2023-10-14T16:40:26.778279+0000","last_clean":"2023-10-14T16:40:26.778279+0000","last_became_active":"2023-10-14T16:40:25.769237+0000","last_became_peered":"2023-10-14T16:40:25.769237+0000","last_unstale":"2023-10-14T16:40:26.778279+0000","last_undegraded":"2023-10-14T16:40:26.778279+0000","last_fullsized":"2023-10-14T16:40:26.778279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.753218+0000","last_change":"2023-10-14T16:40:24.765632+0000","last_active":"2023-10-14T16:40:25.753218+0000","last_peered":"2023-10-14T16:40:25.753218+0000","last_clean":"2023-10-14T16:40:25.753218+0000","last_became_active":"2023-10-14T16:40:24.765393+0000","last_became_peered":"2023-10-14T16:40:24.765393+0000","last_unstale":"2023-10-14T16:40:25.753218+0000","last_undegraded":"2023-10-14T16:40:25.753218+0000","last_fullsized":"2023-10-14T16:40:25.753218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:24.098376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774557+0000","last_change":"2023-10-14T16:40:25.766437+0000","last_active":"2023-10-14T16:40:26.774557+0000","last_peered":"2023-10-14T16:40:26.774557+0000","last_clean":"2023-10-14T16:40:26.774557+0000","last_became_active":"2023-10-14T16:40:25.765798+0000","last_became_peered":"2023-10-14T16:40:25.765798+0000","last_unstale":"2023-10-14T16:40:26.774557+0000","last_undegraded":"2023-10-14T16:40:26.774557+0000","last_fullsized":"2023-10-14T16:40:26.774557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.750604+0000","last_change":"2023-10-14T16:40:24.765207+0000","last_active":"2023-10-14T16:40:25.750604+0000","last_peered":"2023-10-14T16:40:25.750604+0000","last_clean":"2023-10-14T16:40:25.750604+0000","last_became_active":"2023-10-14T16:40:24.764610+0000","last_became_peered":"2023-10-14T16:40:24.764610+0000","last_unstale":"2023-10-14T16:40:25.750604+0000","last_undegraded":"2023-10-14T16:40:25.750604+0000","last_fullsized":"2023-10-14T16:40:25.750604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775031+0000","last_change":"2023-10-14T16:40:25.788202+0000","last_active":"2023-10-14T16:40:26.775031+0000","last_peered":"2023-10-14T16:40:26.775031+0000","last_clean":"2023-10-14T16:40:26.775031+0000","last_became_active":"2023-10-14T16:40:25.785927+0000","last_became_peered":"2023-10-14T16:40:25.785927+0000","last_unstale":"2023-10-14T16:40:26.775031+0000","last_undegraded":"2023-10-14T16:40:26.775031+0000","last_fullsized":"2023-10-14T16:40:26.775031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797842+0000","last_change":"2023-10-14T16:40:24.775618+0000","last_active":"2023-10-14T16:40:25.797842+0000","last_peered":"2023-10-14T16:40:25.797842+0000","last_clean":"2023-10-14T16:40:25.797842+0000","last_became_active":"2023-10-14T16:40:24.775346+0000","last_became_peered":"2023-10-14T16:40:24.775346+0000","last_unstale":"2023-10-14T16:40:25.797842+0000","last_undegraded":"2023-10-14T16:40:25.797842+0000","last_fullsized":"2023-10-14T16:40:25.797842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780727+0000","last_change":"2023-10-14T16:40:25.778910+0000","last_active":"2023-10-14T16:40:26.780727+0000","last_peered":"2023-10-14T16:40:26.780727+0000","last_clean":"2023-10-14T16:40:26.780727+0000","last_became_active":"2023-10-14T16:40:25.777772+0000","last_became_peered":"2023-10-14T16:40:25.777772+0000","last_unstale":"2023-10-14T16:40:26.780727+0000","last_undegraded":"2023-10-14T16:40:26.780727+0000","last_fullsized":"2023-10-14T16:40:26.780727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785645+0000","last_change":"2023-10-14T16:40:25.779144+0000","last_active":"2023-10-14T16:40:25.785645+0000","last_peered":"2023-10-14T16:40:25.785645+0000","last_clean":"2023-10-14T16:40:25.785645+0000","last_became_active":"2023-10-14T16:40:25.778298+0000","last_became_peered":"2023-10-14T16:40:25.778298+0000","last_unstale":"2023-10-14T16:40:25.785645+0000","last_undegraded":"2023-10-14T16:40:25.785645+0000","last_fullsized":"2023-10-14T16:40:25.785645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797945+0000","last_change":"2023-10-14T16:40:24.776824+0000","last_active":"2023-10-14T16:40:25.797945+0000","last_peered":"2023-10-14T16:40:25.797945+0000","last_clean":"2023-10-14T16:40:25.797945+0000","last_became_active":"2023-10-14T16:40:24.776368+0000","last_became_peered":"2023-10-14T16:40:24.776368+0000","last_unstale":"2023-10-14T16:40:25.797945+0000","last_undegraded":"2023-10-14T16:40:25.797945+0000","last_fullsized":"2023-10-14T16:40:25.797945+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785146+0000","last_change":"2023-10-14T16:40:25.777666+0000","last_active":"2023-10-14T16:40:25.785146+0000","last_peered":"2023-10-14T16:40:25.785146+0000","last_clean":"2023-10-14T16:40:25.785146+0000","last_became_active":"2023-10-14T16:40:25.776881+0000","last_became_peered":"2023-10-14T16:40:25.776881+0000","last_unstale":"2023-10-14T16:40:25.785146+0000","last_undegraded":"2023-10-14T16:40:25.785146+0000","last_fullsized":"2023-10-14T16:40:25.785146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790530+0000","last_change":"2023-10-14T16:40:24.777751+0000","last_active":"2023-10-14T16:40:25.790530+0000","last_peered":"2023-10-14T16:40:25.790530+0000","last_clean":"2023-10-14T16:40:25.790530+0000","last_became_active":"2023-10-14T16:40:24.777483+0000","last_became_peered":"2023-10-14T16:40:24.777483+0000","last_unstale":"2023-10-14T16:40:25.790530+0000","last_undegraded":"2023-10-14T16:40:25.790530+0000","last_fullsized":"2023-10-14T16:40:25.790530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788111+0000","last_change":"2023-10-14T16:40:25.783100+0000","last_active":"2023-10-14T16:40:25.788111+0000","last_peered":"2023-10-14T16:40:25.788111+0000","last_clean":"2023-10-14T16:40:25.788111+0000","last_became_active":"2023-10-14T16:40:25.782769+0000","last_became_peered":"2023-10-14T16:40:25.782769+0000","last_unstale":"2023-10-14T16:40:25.788111+0000","last_undegraded":"2023-10-14T16:40:25.788111+0000","last_fullsized":"2023-10-14T16:40:25.788111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:37:49.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775509+0000","last_change":"2023-10-14T16:40:24.768856+0000","last_active":"2023-10-14T16:40:26.775509+0000","last_peered":"2023-10-14T16:40:26.775509+0000","last_clean":"2023-10-14T16:40:26.775509+0000","last_became_active":"2023-10-14T16:40:24.768497+0000","last_became_peered":"2023-10-14T16:40:24.768497+0000","last_unstale":"2023-10-14T16:40:26.775509+0000","last_undegraded":"2023-10-14T16:40:26.775509+0000","last_fullsized":"2023-10-14T16:40:26.775509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:33.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775555+0000","last_change":"2023-10-14T16:40:25.784770+0000","last_active":"2023-10-14T16:40:26.775555+0000","last_peered":"2023-10-14T16:40:26.775555+0000","last_clean":"2023-10-14T16:40:26.775555+0000","last_became_active":"2023-10-14T16:40:25.782997+0000","last_became_peered":"2023-10-14T16:40:25.782997+0000","last_unstale":"2023-10-14T16:40:26.775555+0000","last_undegraded":"2023-10-14T16:40:26.775555+0000","last_fullsized":"2023-10-14T16:40:26.775555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794922+0000","last_change":"2023-10-14T16:40:24.772362+0000","last_active":"2023-10-14T16:40:25.794922+0000","last_peered":"2023-10-14T16:40:25.794922+0000","last_clean":"2023-10-14T16:40:25.794922+0000","last_became_active":"2023-10-14T16:40:24.772021+0000","last_became_peered":"2023-10-14T16:40:24.772021+0000","last_unstale":"2023-10-14T16:40:25.794922+0000","last_undegraded":"2023-10-14T16:40:25.794922+0000","last_fullsized":"2023-10-14T16:40:25.794922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":64688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6639999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":64758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":798720,"data_stored":655604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:40:30.311 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:40:30.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.577+0000 7f33a9587640 1 Processor -- start 2023-10-14T16:40:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.578+0000 7f33a9587640 1 -- start start 2023-10-14T16:40:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.579+0000 7f33a9587640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f33a415c9f0 0x7f33a40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.579+0000 7f33a9587640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a4160a90 con 0x7f33a415dde0 2023-10-14T16:40:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.579+0000 7f33a9587640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a40cd900 con 0x7f33a40c9230 2023-10-14T16:40:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.579+0000 7f33a9587640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33a40cdec0 con 0x7f33a415c9f0 2023-10-14T16:40:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a8d86640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f33a415dde0 0x7f33a40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42048/0 (socket says 172.21.15.29:42048) 2023-10-14T16:40:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a8d86640 1 -- 172.21.15.29:0/3494027812 learned_addr learned my addr 172.21.15.29:0/3494027812 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a3fff640 1 --2- 172.21.15.29:0/3494027812 >> v2:172.21.15.186:3300/0 conn(0x7f33a415c9f0 0x7f33a40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802479797 0 0) 0x7f33a4160a90 con 0x7f33a415dde0 2023-10-14T16:40:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3390003520 con 0x7f33a415dde0 2023-10-14T16:40:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3756143724 0 0) 0x7f3390003520 con 0x7f33a415dde0 2023-10-14T16:40:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 >> v2:172.21.15.186:3300/0 conn(0x7f33a415c9f0 msgr2=0x7f33a40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 --2- 172.21.15.29:0/3494027812 >> v2:172.21.15.186:3300/0 conn(0x7f33a415c9f0 0x7f33a40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 >> v1:172.21.15.186:6789/0 conn(0x7f33a40c9230 legacy=0x7f33a40709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a40cdbe0 con 0x7f33a415dde0 2023-10-14T16:40:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a3fff640 1 --2- 172.21.15.29:0/3494027812 >> v2:172.21.15.186:3300/0 conn(0x7f33a415c9f0 0x7f33a40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.580+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f339c0019f0 con 0x7f33a415dde0 2023-10-14T16:40:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 >> v1:172.21.15.29:6789/0 conn(0x7f33a415dde0 legacy=0x7f33a40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3390003970 0x7f3390003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f3390004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3390005180 0x7f339000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000bff0 con 0x7f33900047c0 2023-10-14T16:40:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f339000c160 con 0x7f3390005180 2023-10-14T16:40:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f339000c320 con 0x7f3390003970 2023-10-14T16:40:30.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f3390004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a3fff640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3390003970 0x7f3390003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a8d86640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3390005180 0x7f339000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 -- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3390003970 msgr2=0x7f3390003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3390003970 0x7f3390003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 -- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3390005180 msgr2=0x7f339000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3390005180 0x7f339000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f339bfff640 1 -- 172.21.15.29:0/3494027812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3390003520 con 0x7f33900047c0 2023-10-14T16:40:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a8d86640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3390005180 0x7f339000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.581+0000 7f33a8d86640 1 -- 172.21.15.29:0/3494027812 reap_dead start 2023-10-14T16:40:30.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f339bfff640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f3390004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f338c001750 tx=0x7f33900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb5d4842e4faa4a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c0052d0 con 0x7f33900047c0 2023-10-14T16:40:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f338c02e050 con 0x7f33900047c0 2023-10-14T16:40:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a37fe640 1 -- 172.21.15.29:0/3494027812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f338c003830 con 0x7f33900047c0 2023-10-14T16:40:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a9587640 1 -- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 msgr2=0x7f3390004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a9587640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f3390004c40 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f338c001750 tx=0x7f33900057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a9587640 1 -- 172.21.15.29:0/3494027812 shutdown_connections 2023-10-14T16:40:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.583+0000 7f33a9587640 1 --2- 172.21.15.29:0/3494027812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f3390004c40 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.584+0000 7f33a9587640 1 -- 172.21.15.29:0/3494027812 >> 172.21.15.29:0/3494027812 conn(0x7f33a40ce6a0 msgr2=0x7f33a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.584+0000 7f33a9587640 1 -- 172.21.15.29:0/3494027812 shutdown_connections 2023-10-14T16:40:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.584+0000 7f33a9587640 1 -- 172.21.15.29:0/3494027812 wait complete. 2023-10-14T16:40:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.585+0000 7f33a9587640 1 Processor -- start 2023-10-14T16:40:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 -- start start 2023-10-14T16:40:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33a406a300 0x7f33a41405c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a4143060 0x7f33a40c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a41464e0 con 0x7f33900047c0 2023-10-14T16:40:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33a406aae0 con 0x7f33a4143060 2023-10-14T16:40:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.586+0000 7f33a9587640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33a406a800 con 0x7f33a406a300 2023-10-14T16:40:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60896/0 (socket says 172.21.15.29:60896) 2023-10-14T16:40:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 -- 172.21.15.29:0/2910763188 learned_addr learned my addr 172.21.15.29:0/2910763188 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a8d86640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a4143060 0x7f33a40c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 -- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33a406a300 msgr2=0x7f33a41405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33a406a300 0x7f33a41405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 -- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a4143060 msgr2=0x7f33a40c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a4143060 0x7f33a40c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.587+0000 7f33a3fff640 1 -- 172.21.15.29:0/2910763188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a4157fb0 con 0x7f33900047c0 2023-10-14T16:40:30.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a3fff640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3394011800 tx=0x7f3394011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339400f9c0 con 0x7f33900047c0 2023-10-14T16:40:30.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f339400fb20 con 0x7f33900047c0 2023-10-14T16:40:30.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f339400fe90 con 0x7f33900047c0 2023-10-14T16:40:30.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a4158290 con 0x7f33900047c0 2023-10-14T16:40:30.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.588+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a4143510 con 0x7f33900047c0 2023-10-14T16:40:30.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.589+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33a4157e10 con 0x7f33900047c0 2023-10-14T16:40:30.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.590+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3394015460 con 0x7f33900047c0 2023-10-14T16:40:30.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.591+0000 7f33a17fa640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 0x7f338406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:30.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.591+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3394089b80 con 0x7f33900047c0 2023-10-14T16:40:30.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.591+0000 7f339bfff640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 0x7f338406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:30.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.592+0000 7f339bfff640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 0x7f338406d360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f338c033000 tx=0x7f338c02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:30.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.596+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33a4157e10 con 0x7f33900047c0 2023-10-14T16:40:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.787+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f33a40704c0 con 0x7f338406aea0 2023-10-14T16:40:30.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.808+0000 7f33a17fa640 1 -- 172.21.15.29:0/2910763188 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264471 (secure 0 0 0) 0x7f338c0052d0 con 0x7f338406aea0 2023-10-14T16:40:30.812 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:30.814 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:40:30.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 msgr2=0x7f338406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 0x7f338406d360 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f338c033000 tx=0x7f338c02f000 comp rx=0 tx=0).stop 2023-10-14T16:40:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 msgr2=0x7f33a40c10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3394011800 tx=0x7f3394011cc0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 shutdown_connections 2023-10-14T16:40:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f338406aea0 0x7f338406d360 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33a4143060 0x7f33a40c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33a406a300 0x7f33a41405c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 --2- 172.21.15.29:0/2910763188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33900047c0 0x7f33a40c10c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:30.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.813+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 >> 172.21.15.29:0/2910763188 conn(0x7f33a40ce6a0 msgr2=0x7f33a40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.814+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 shutdown_connections 2023-10-14T16:40:30.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:30.814+0000 7f33a9587640 1 -- 172.21.15.29:0/2910763188 wait complete. 2023-10-14T16:40:30.836 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2023-10-14T16:40:29.299909+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":186,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235588,"kb_used_data":2764,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739132,"statfs":{"total":773094113280,"available":772852871168,"internally_reserved":0,"allocated":2830336,"data_stored":1721240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.574711"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790397+0000","last_change":"2023-10-14T16:40:25.778039+0000","last_active":"2023-10-14T16:40:25.790397+0000","last_peered":"2023-10-14T16:40:25.790397+0000","last_clean":"2023-10-14T16:40:25.790397+0000","last_became_active":"2023-10-14T16:40:25.777745+0000","last_became_peered":"2023-10-14T16:40:25.777745+0000","last_unstale":"2023-10-14T16:40:25.790397+0000","last_undegraded":"2023-10-14T16:40:25.790397+0000","last_fullsized":"2023-10-14T16:40:25.790397+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786508+0000","last_change":"2023-10-14T16:40:25.782038+0000","last_active":"2023-10-14T16:40:25.786508+0000","last_peered":"2023-10-14T16:40:25.786508+0000","last_clean":"2023-10-14T16:40:25.786508+0000","last_became_active":"2023-10-14T16:40:25.781722+0000","last_became_peered":"2023-10-14T16:40:25.781722+0000","last_unstale":"2023-10-14T16:40:25.786508+0000","last_undegraded":"2023-10-14T16:40:25.786508+0000","last_fullsized":"2023-10-14T16:40:25.786508+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.766375+0000","last_change":"2023-10-14T16:40:25.757114+0000","last_active":"2023-10-14T16:40:25.766375+0000","last_peered":"2023-10-14T16:40:25.766375+0000","last_clean":"2023-10-14T16:40:25.766375+0000","last_became_active":"2023-10-14T16:40:25.756793+0000","last_became_peered":"2023-10-14T16:40:25.756793+0000","last_unstale":"2023-10-14T16:40:25.766375+0000","last_undegraded":"2023-10-14T16:40:25.766375+0000","last_fullsized":"2023-10-14T16:40:25.766375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793293+0000","last_change":"2023-10-14T16:40:25.788571+0000","last_active":"2023-10-14T16:40:25.793293+0000","last_peered":"2023-10-14T16:40:25.793293+0000","last_clean":"2023-10-14T16:40:25.793293+0000","last_became_active":"2023-10-14T16:40:25.788388+0000","last_became_peered":"2023-10-14T16:40:25.788388+0000","last_unstale":"2023-10-14T16:40:25.793293+0000","last_undegraded":"2023-10-14T16:40:25.793293+0000","last_fullsized":"2023-10-14T16:40:25.793293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:01.231512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.791613+0000","last_change":"2023-10-14T16:40:25.781929+0000","last_active":"2023-10-14T16:40:25.791613+0000","last_peered":"2023-10-14T16:40:25.791613+0000","last_clean":"2023-10-14T16:40:25.791613+0000","last_became_active":"2023-10-14T16:40:25.774025+0000","last_became_peered":"2023-10-14T16:40:25.774025+0000","last_unstale":"2023-10-14T16:40:25.791613+0000","last_undegraded":"2023-10-14T16:40:25.791613+0000","last_fullsized":"2023-10-14T16:40:25.791613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789735+0000","last_change":"2023-10-14T16:40:25.785668+0000","last_active":"2023-10-14T16:40:25.789735+0000","last_peered":"2023-10-14T16:40:25.789735+0000","last_clean":"2023-10-14T16:40:25.789735+0000","last_became_active":"2023-10-14T16:40:25.785300+0000","last_became_peered":"2023-10-14T16:40:25.785300+0000","last_unstale":"2023-10-14T16:40:25.789735+0000","last_undegraded":"2023-10-14T16:40:25.789735+0000","last_fullsized":"2023-10-14T16:40:25.789735+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793812+0000","last_change":"2023-10-14T16:40:25.786146+0000","last_active":"2023-10-14T16:40:25.793812+0000","last_peered":"2023-10-14T16:40:25.793812+0000","last_clean":"2023-10-14T16:40:25.793812+0000","last_became_active":"2023-10-14T16:40:25.785748+0000","last_became_peered":"2023-10-14T16:40:25.785748+0000","last_unstale":"2023-10-14T16:40:25.793812+0000","last_undegraded":"2023-10-14T16:40:25.793812+0000","last_fullsized":"2023-10-14T16:40:25.793812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774509+0000","last_change":"2023-10-14T16:40:25.783832+0000","last_active":"2023-10-14T16:40:26.774509+0000","last_peered":"2023-10-14T16:40:26.774509+0000","last_clean":"2023-10-14T16:40:26.774509+0000","last_became_active":"2023-10-14T16:40:25.781635+0000","last_became_peered":"2023-10-14T16:40:25.781635+0000","last_unstale":"2023-10-14T16:40:26.774509+0000","last_undegraded":"2023-10-14T16:40:26.774509+0000","last_fullsized":"2023-10-14T16:40:26.774509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797707+0000","last_change":"2023-10-14T16:40:25.791696+0000","last_active":"2023-10-14T16:40:25.797707+0000","last_peered":"2023-10-14T16:40:25.797707+0000","last_clean":"2023-10-14T16:40:25.797707+0000","last_became_active":"2023-10-14T16:40:25.791413+0000","last_became_peered":"2023-10-14T16:40:25.791413+0000","last_unstale":"2023-10-14T16:40:25.797707+0000","last_undegraded":"2023-10-14T16:40:25.797707+0000","last_fullsized":"2023-10-14T16:40:25.797707+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781741+0000","last_change":"2023-10-14T16:40:25.786842+0000","last_active":"2023-10-14T16:40:26.781741+0000","last_peered":"2023-10-14T16:40:26.781741+0000","last_clean":"2023-10-14T16:40:26.781741+0000","last_became_active":"2023-10-14T16:40:25.786528+0000","last_became_peered":"2023-10-14T16:40:25.786528+0000","last_unstale":"2023-10-14T16:40:26.781741+0000","last_undegraded":"2023-10-14T16:40:26.781741+0000","last_fullsized":"2023-10-14T16:40:26.781741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780857+0000","last_change":"2023-10-14T16:40:25.788130+0000","last_active":"2023-10-14T16:40:26.780857+0000","last_peered":"2023-10-14T16:40:26.780857+0000","last_clean":"2023-10-14T16:40:26.780857+0000","last_became_active":"2023-10-14T16:40:25.787596+0000","last_became_peered":"2023-10-14T16:40:25.787596+0000","last_unstale":"2023-10-14T16:40:26.780857+0000","last_undegraded":"2023-10-14T16:40:26.780857+0000","last_fullsized":"2023-10-14T16:40:26.780857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778866+0000","last_change":"2023-10-14T16:40:25.765437+0000","last_active":"2023-10-14T16:40:26.778866+0000","last_peered":"2023-10-14T16:40:26.778866+0000","last_clean":"2023-10-14T16:40:26.778866+0000","last_became_active":"2023-10-14T16:40:25.765116+0000","last_became_peered":"2023-10-14T16:40:25.765116+0000","last_unstale":"2023-10-14T16:40:26.778866+0000","last_undegraded":"2023-10-14T16:40:26.778866+0000","last_fullsized":"2023-10-14T16:40:26.778866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774963+0000","last_change":"2023-10-14T16:40:25.784431+0000","last_active":"2023-10-14T16:40:26.774963+0000","last_peered":"2023-10-14T16:40:26.774963+0000","last_clean":"2023-10-14T16:40:26.774963+0000","last_became_active":"2023-10-14T16:40:25.775585+0000","last_became_peered":"2023-10-14T16:40:25.775585+0000","last_unstale":"2023-10-14T16:40:26.774963+0000","last_undegraded":"2023-10-14T16:40:26.774963+0000","last_fullsized":"2023-10-14T16:40:26.774963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783410+0000","last_change":"2023-10-14T16:40:25.761749+0000","last_active":"2023-10-14T16:40:25.783410+0000","last_peered":"2023-10-14T16:40:25.783410+0000","last_clean":"2023-10-14T16:40:25.783410+0000","last_became_active":"2023-10-14T16:40:25.761444+0000","last_became_peered":"2023-10-14T16:40:25.761444+0000","last_unstale":"2023-10-14T16:40:25.783410+0000","last_undegraded":"2023-10-14T16:40:25.783410+0000","last_fullsized":"2023-10-14T16:40:25.783410+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785615+0000","last_change":"2023-10-14T16:40:25.776450+0000","last_active":"2023-10-14T16:40:25.785615+0000","last_peered":"2023-10-14T16:40:25.785615+0000","last_clean":"2023-10-14T16:40:25.785615+0000","last_became_active":"2023-10-14T16:40:25.776153+0000","last_became_peered":"2023-10-14T16:40:25.776153+0000","last_unstale":"2023-10-14T16:40:25.785615+0000","last_undegraded":"2023-10-14T16:40:25.785615+0000","last_fullsized":"2023-10-14T16:40:25.785615+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:01.231512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789732+0000","last_change":"2023-10-14T16:40:25.786126+0000","last_active":"2023-10-14T16:40:25.789732+0000","last_peered":"2023-10-14T16:40:25.789732+0000","last_clean":"2023-10-14T16:40:25.789732+0000","last_became_active":"2023-10-14T16:40:25.785701+0000","last_became_peered":"2023-10-14T16:40:25.785701+0000","last_unstale":"2023-10-14T16:40:25.789732+0000","last_undegraded":"2023-10-14T16:40:25.789732+0000","last_fullsized":"2023-10-14T16:40:25.789732+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793424+0000","last_change":"2023-10-14T16:40:25.792223+0000","last_active":"2023-10-14T16:40:25.793424+0000","last_peered":"2023-10-14T16:40:25.793424+0000","last_clean":"2023-10-14T16:40:25.793424+0000","last_became_active":"2023-10-14T16:40:25.791860+0000","last_became_peered":"2023-10-14T16:40:25.791860+0000","last_unstale":"2023-10-14T16:40:25.793424+0000","last_undegraded":"2023-10-14T16:40:25.793424+0000","last_fullsized":"2023-10-14T16:40:25.793424+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776199+0000","last_change":"2023-10-14T16:40:25.766839+0000","last_active":"2023-10-14T16:40:27.776199+0000","last_peered":"2023-10-14T16:40:27.776199+0000","last_clean":"2023-10-14T16:40:27.776199+0000","last_became_active":"2023-10-14T16:40:25.766493+0000","last_became_peered":"2023-10-14T16:40:25.766493+0000","last_unstale":"2023-10-14T16:40:27.776199+0000","last_undegraded":"2023-10-14T16:40:27.776199+0000","last_fullsized":"2023-10-14T16:40:27.776199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797728+0000","last_change":"2023-10-14T16:40:25.792353+0000","last_active":"2023-10-14T16:40:25.797728+0000","last_peered":"2023-10-14T16:40:25.797728+0000","last_clean":"2023-10-14T16:40:25.797728+0000","last_became_active":"2023-10-14T16:40:25.792040+0000","last_became_peered":"2023-10-14T16:40:25.792040+0000","last_unstale":"2023-10-14T16:40:25.797728+0000","last_undegraded":"2023-10-14T16:40:25.797728+0000","last_fullsized":"2023-10-14T16:40:25.797728+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781176+0000","last_change":"2023-10-14T16:40:25.770102+0000","last_active":"2023-10-14T16:40:26.781176+0000","last_peered":"2023-10-14T16:40:26.781176+0000","last_clean":"2023-10-14T16:40:26.781176+0000","last_became_active":"2023-10-14T16:40:25.769804+0000","last_became_peered":"2023-10-14T16:40:25.769804+0000","last_unstale":"2023-10-14T16:40:26.781176+0000","last_undegraded":"2023-10-14T16:40:26.781176+0000","last_fullsized":"2023-10-14T16:40:26.781176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784916+0000","last_change":"2023-10-14T16:40:25.779852+0000","last_active":"2023-10-14T16:40:25.784916+0000","last_peered":"2023-10-14T16:40:25.784916+0000","last_clean":"2023-10-14T16:40:25.784916+0000","last_became_active":"2023-10-14T16:40:25.779362+0000","last_became_peered":"2023-10-14T16:40:25.779362+0000","last_unstale":"2023-10-14T16:40:25.784916+0000","last_undegraded":"2023-10-14T16:40:25.784916+0000","last_fullsized":"2023-10-14T16:40:25.784916+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:25:00.025546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786861+0000","last_change":"2023-10-14T16:40:25.781280+0000","last_active":"2023-10-14T16:40:25.786861+0000","last_peered":"2023-10-14T16:40:25.786861+0000","last_clean":"2023-10-14T16:40:25.786861+0000","last_became_active":"2023-10-14T16:40:25.780595+0000","last_became_peered":"2023-10-14T16:40:25.780595+0000","last_unstale":"2023-10-14T16:40:25.786861+0000","last_undegraded":"2023-10-14T16:40:25.786861+0000","last_fullsized":"2023-10-14T16:40:25.786861+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:39.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774501+0000","last_change":"2023-10-14T16:40:25.777771+0000","last_active":"2023-10-14T16:40:26.774501+0000","last_peered":"2023-10-14T16:40:26.774501+0000","last_clean":"2023-10-14T16:40:26.774501+0000","last_became_active":"2023-10-14T16:40:25.777508+0000","last_became_peered":"2023-10-14T16:40:25.777508+0000","last_unstale":"2023-10-14T16:40:26.774501+0000","last_undegraded":"2023-10-14T16:40:26.774501+0000","last_fullsized":"2023-10-14T16:40:26.774501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783460+0000","last_change":"2023-10-14T16:40:25.772029+0000","last_active":"2023-10-14T16:40:25.783460+0000","last_peered":"2023-10-14T16:40:25.783460+0000","last_clean":"2023-10-14T16:40:25.783460+0000","last_became_active":"2023-10-14T16:40:25.771700+0000","last_became_peered":"2023-10-14T16:40:25.771700+0000","last_unstale":"2023-10-14T16:40:25.783460+0000","last_undegraded":"2023-10-14T16:40:25.783460+0000","last_fullsized":"2023-10-14T16:40:25.783460+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775784+0000","last_change":"2023-10-14T16:40:25.777077+0000","last_active":"2023-10-14T16:40:26.775784+0000","last_peered":"2023-10-14T16:40:26.775784+0000","last_clean":"2023-10-14T16:40:26.775784+0000","last_became_active":"2023-10-14T16:40:25.776755+0000","last_became_peered":"2023-10-14T16:40:25.776755+0000","last_unstale":"2023-10-14T16:40:26.775784+0000","last_undegraded":"2023-10-14T16:40:26.775784+0000","last_fullsized":"2023-10-14T16:40:26.775784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781826+0000","last_change":"2023-10-14T16:40:25.777054+0000","last_active":"2023-10-14T16:40:26.781826+0000","last_peered":"2023-10-14T16:40:26.781826+0000","last_clean":"2023-10-14T16:40:26.781826+0000","last_became_active":"2023-10-14T16:40:25.773588+0000","last_became_peered":"2023-10-14T16:40:25.773588+0000","last_unstale":"2023-10-14T16:40:26.781826+0000","last_undegraded":"2023-10-14T16:40:26.781826+0000","last_fullsized":"2023-10-14T16:40:26.781826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790984+0000","last_change":"2023-10-14T16:40:25.787594+0000","last_active":"2023-10-14T16:40:25.790984+0000","last_peered":"2023-10-14T16:40:25.790984+0000","last_clean":"2023-10-14T16:40:25.790984+0000","last_became_active":"2023-10-14T16:40:25.786833+0000","last_became_peered":"2023-10-14T16:40:25.786833+0000","last_unstale":"2023-10-14T16:40:25.790984+0000","last_undegraded":"2023-10-14T16:40:25.790984+0000","last_fullsized":"2023-10-14T16:40:25.790984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775301+0000","last_change":"2023-10-14T16:40:25.782397+0000","last_active":"2023-10-14T16:40:26.775301+0000","last_peered":"2023-10-14T16:40:26.775301+0000","last_clean":"2023-10-14T16:40:26.775301+0000","last_became_active":"2023-10-14T16:40:25.782109+0000","last_became_peered":"2023-10-14T16:40:25.782109+0000","last_unstale":"2023-10-14T16:40:26.775301+0000","last_undegraded":"2023-10-14T16:40:26.775301+0000","last_fullsized":"2023-10-14T16:40:26.775301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788167+0000","last_change":"2023-10-14T16:40:25.783124+0000","last_active":"2023-10-14T16:40:25.788167+0000","last_peered":"2023-10-14T16:40:25.788167+0000","last_clean":"2023-10-14T16:40:25.788167+0000","last_became_active":"2023-10-14T16:40:25.782805+0000","last_became_peered":"2023-10-14T16:40:25.782805+0000","last_unstale":"2023-10-14T16:40:25.788167+0000","last_undegraded":"2023-10-14T16:40:25.788167+0000","last_fullsized":"2023-10-14T16:40:25.788167+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786881+0000","last_change":"2023-10-14T16:40:25.784068+0000","last_active":"2023-10-14T16:40:25.786881+0000","last_peered":"2023-10-14T16:40:25.786881+0000","last_clean":"2023-10-14T16:40:25.786881+0000","last_became_active":"2023-10-14T16:40:25.783671+0000","last_became_peered":"2023-10-14T16:40:25.783671+0000","last_unstale":"2023-10-14T16:40:25.786881+0000","last_undegraded":"2023-10-14T16:40:25.786881+0000","last_fullsized":"2023-10-14T16:40:25.786881+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775610+0000","last_change":"2023-10-14T16:40:25.777635+0000","last_active":"2023-10-14T16:40:27.775610+0000","last_peered":"2023-10-14T16:40:27.775610+0000","last_clean":"2023-10-14T16:40:27.775610+0000","last_became_active":"2023-10-14T16:40:25.777400+0000","last_became_peered":"2023-10-14T16:40:25.777400+0000","last_unstale":"2023-10-14T16:40:27.775610+0000","last_undegraded":"2023-10-14T16:40:27.775610+0000","last_fullsized":"2023-10-14T16:40:27.775610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779941+0000","last_change":"2023-10-14T16:40:25.758613+0000","last_active":"2023-10-14T16:40:26.779941+0000","last_peered":"2023-10-14T16:40:26.779941+0000","last_clean":"2023-10-14T16:40:26.779941+0000","last_became_active":"2023-10-14T16:40:25.758173+0000","last_became_peered":"2023-10-14T16:40:25.758173+0000","last_unstale":"2023-10-14T16:40:26.779941+0000","last_undegraded":"2023-10-14T16:40:26.779941+0000","last_fullsized":"2023-10-14T16:40:26.779941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778458+0000","last_change":"2023-10-14T16:40:25.759820+0000","last_active":"2023-10-14T16:40:26.778458+0000","last_peered":"2023-10-14T16:40:26.778458+0000","last_clean":"2023-10-14T16:40:26.778458+0000","last_became_active":"2023-10-14T16:40:25.759498+0000","last_became_peered":"2023-10-14T16:40:25.759498+0000","last_unstale":"2023-10-14T16:40:26.778458+0000","last_undegraded":"2023-10-14T16:40:26.778458+0000","last_fullsized":"2023-10-14T16:40:26.778458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775078+0000","last_change":"2023-10-14T16:40:24.768741+0000","last_active":"2023-10-14T16:40:26.775078+0000","last_peered":"2023-10-14T16:40:26.775078+0000","last_clean":"2023-10-14T16:40:26.775078+0000","last_became_active":"2023-10-14T16:40:24.768379+0000","last_became_peered":"2023-10-14T16:40:24.768379+0000","last_unstale":"2023-10-14T16:40:26.775078+0000","last_undegraded":"2023-10-14T16:40:26.775078+0000","last_fullsized":"2023-10-14T16:40:26.775078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784898+0000","last_change":"2023-10-14T16:40:25.779646+0000","last_active":"2023-10-14T16:40:25.784898+0000","last_peered":"2023-10-14T16:40:25.784898+0000","last_clean":"2023-10-14T16:40:25.784898+0000","last_became_active":"2023-10-14T16:40:25.779001+0000","last_became_peered":"2023-10-14T16:40:25.779001+0000","last_unstale":"2023-10-14T16:40:25.784898+0000","last_undegraded":"2023-10-14T16:40:25.784898+0000","last_fullsized":"2023-10-14T16:40:25.784898+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780736+0000","last_change":"2023-10-14T16:40:24.763894+0000","last_active":"2023-10-14T16:40:26.780736+0000","last_peered":"2023-10-14T16:40:26.780736+0000","last_clean":"2023-10-14T16:40:26.780736+0000","last_became_active":"2023-10-14T16:40:24.762343+0000","last_became_peered":"2023-10-14T16:40:24.762343+0000","last_unstale":"2023-10-14T16:40:26.780736+0000","last_undegraded":"2023-10-14T16:40:26.780736+0000","last_fullsized":"2023-10-14T16:40:26.780736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774426+0000","last_change":"2023-10-14T16:40:24.775055+0000","last_active":"2023-10-14T16:40:26.774426+0000","last_peered":"2023-10-14T16:40:26.774426+0000","last_clean":"2023-10-14T16:40:26.774426+0000","last_became_active":"2023-10-14T16:40:24.774256+0000","last_became_peered":"2023-10-14T16:40:24.774256+0000","last_unstale":"2023-10-14T16:40:26.774426+0000","last_undegraded":"2023-10-14T16:40:26.774426+0000","last_fullsized":"2023-10-14T16:40:26.774426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774375+0000","last_change":"2023-10-14T16:40:25.781085+0000","last_active":"2023-10-14T16:40:26.774375+0000","last_peered":"2023-10-14T16:40:26.774375+0000","last_clean":"2023-10-14T16:40:26.774375+0000","last_became_active":"2023-10-14T16:40:25.780746+0000","last_became_peered":"2023-10-14T16:40:25.780746+0000","last_unstale":"2023-10-14T16:40:26.774375+0000","last_undegraded":"2023-10-14T16:40:26.774375+0000","last_fullsized":"2023-10-14T16:40:26.774375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.768784+0000","last_change":"2023-10-14T16:40:24.763402+0000","last_active":"2023-10-14T16:40:25.768784+0000","last_peered":"2023-10-14T16:40:25.768784+0000","last_clean":"2023-10-14T16:40:25.768784+0000","last_became_active":"2023-10-14T16:40:24.763093+0000","last_became_peered":"2023-10-14T16:40:24.763093+0000","last_unstale":"2023-10-14T16:40:25.768784+0000","last_undegraded":"2023-10-14T16:40:25.768784+0000","last_fullsized":"2023-10-14T16:40:25.768784+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.786384+0000","last_change":"2023-10-14T16:40:25.780063+0000","last_active":"2023-10-14T16:40:25.786384+0000","last_peered":"2023-10-14T16:40:25.786384+0000","last_clean":"2023-10-14T16:40:25.786384+0000","last_became_active":"2023-10-14T16:40:25.776862+0000","last_became_peered":"2023-10-14T16:40:25.776862+0000","last_unstale":"2023-10-14T16:40:25.786384+0000","last_undegraded":"2023-10-14T16:40:25.786384+0000","last_fullsized":"2023-10-14T16:40:25.786384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781765+0000","last_change":"2023-10-14T16:40:24.768554+0000","last_active":"2023-10-14T16:40:26.781765+0000","last_peered":"2023-10-14T16:40:26.781765+0000","last_clean":"2023-10-14T16:40:26.781765+0000","last_became_active":"2023-10-14T16:40:24.768232+0000","last_became_peered":"2023-10-14T16:40:24.768232+0000","last_unstale":"2023-10-14T16:40:26.781765+0000","last_undegraded":"2023-10-14T16:40:26.781765+0000","last_fullsized":"2023-10-14T16:40:26.781765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:59.941812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785704+0000","last_change":"2023-10-14T16:40:25.778676+0000","last_active":"2023-10-14T16:40:25.785704+0000","last_peered":"2023-10-14T16:40:25.785704+0000","last_clean":"2023-10-14T16:40:25.785704+0000","last_became_active":"2023-10-14T16:40:25.778295+0000","last_became_peered":"2023-10-14T16:40:25.778295+0000","last_unstale":"2023-10-14T16:40:25.785704+0000","last_undegraded":"2023-10-14T16:40:25.785704+0000","last_fullsized":"2023-10-14T16:40:25.785704+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.770673+0000","last_change":"2023-10-14T16:40:24.764655+0000","last_active":"2023-10-14T16:40:25.770673+0000","last_peered":"2023-10-14T16:40:25.770673+0000","last_clean":"2023-10-14T16:40:25.770673+0000","last_became_active":"2023-10-14T16:40:24.764327+0000","last_became_peered":"2023-10-14T16:40:24.764327+0000","last_unstale":"2023-10-14T16:40:25.770673+0000","last_undegraded":"2023-10-14T16:40:25.770673+0000","last_fullsized":"2023-10-14T16:40:25.770673+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.733975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788129+0000","last_change":"2023-10-14T16:40:25.783442+0000","last_active":"2023-10-14T16:40:25.788129+0000","last_peered":"2023-10-14T16:40:25.788129+0000","last_clean":"2023-10-14T16:40:25.788129+0000","last_became_active":"2023-10-14T16:40:25.783169+0000","last_became_peered":"2023-10-14T16:40:25.783169+0000","last_unstale":"2023-10-14T16:40:25.788129+0000","last_undegraded":"2023-10-14T16:40:25.788129+0000","last_fullsized":"2023-10-14T16:40:25.788129+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:04.110533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.787415+0000","last_change":"2023-10-14T16:40:24.767382+0000","last_active":"2023-10-14T16:40:25.787415+0000","last_peered":"2023-10-14T16:40:25.787415+0000","last_clean":"2023-10-14T16:40:25.787415+0000","last_became_active":"2023-10-14T16:40:24.767094+0000","last_became_peered":"2023-10-14T16:40:24.767094+0000","last_unstale":"2023-10-14T16:40:25.787415+0000","last_undegraded":"2023-10-14T16:40:25.787415+0000","last_fullsized":"2023-10-14T16:40:25.787415+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778953+0000","last_change":"2023-10-14T16:40:25.776539+0000","last_active":"2023-10-14T16:40:26.778953+0000","last_peered":"2023-10-14T16:40:26.778953+0000","last_clean":"2023-10-14T16:40:26.778953+0000","last_became_active":"2023-10-14T16:40:25.776126+0000","last_became_peered":"2023-10-14T16:40:25.776126+0000","last_unstale":"2023-10-14T16:40:26.778953+0000","last_undegraded":"2023-10-14T16:40:26.778953+0000","last_fullsized":"2023-10-14T16:40:26.778953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781420+0000","last_change":"2023-10-14T16:40:24.768941+0000","last_active":"2023-10-14T16:40:26.781420+0000","last_peered":"2023-10-14T16:40:26.781420+0000","last_clean":"2023-10-14T16:40:26.781420+0000","last_became_active":"2023-10-14T16:40:24.767075+0000","last_became_peered":"2023-10-14T16:40:24.767075+0000","last_unstale":"2023-10-14T16:40:26.781420+0000","last_undegraded":"2023-10-14T16:40:26.781420+0000","last_fullsized":"2023-10-14T16:40:26.781420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794708+0000","last_change":"2023-10-14T16:40:25.791099+0000","last_active":"2023-10-14T16:40:25.794708+0000","last_peered":"2023-10-14T16:40:25.794708+0000","last_clean":"2023-10-14T16:40:25.794708+0000","last_became_active":"2023-10-14T16:40:25.790819+0000","last_became_peered":"2023-10-14T16:40:25.790819+0000","last_unstale":"2023-10-14T16:40:25.794708+0000","last_undegraded":"2023-10-14T16:40:25.794708+0000","last_fullsized":"2023-10-14T16:40:25.794708+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793667+0000","last_change":"2023-10-14T16:40:24.760669+0000","last_active":"2023-10-14T16:40:25.793667+0000","last_peered":"2023-10-14T16:40:25.793667+0000","last_clean":"2023-10-14T16:40:25.793667+0000","last_became_active":"2023-10-14T16:40:24.760395+0000","last_became_peered":"2023-10-14T16:40:24.760395+0000","last_unstale":"2023-10-14T16:40:25.793667+0000","last_undegraded":"2023-10-14T16:40:25.793667+0000","last_fullsized":"2023-10-14T16:40:25.793667+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.784659+0000","last_change":"2023-10-14T16:40:25.779561+0000","last_active":"2023-10-14T16:40:25.784659+0000","last_peered":"2023-10-14T16:40:25.784659+0000","last_clean":"2023-10-14T16:40:25.784659+0000","last_became_active":"2023-10-14T16:40:25.779246+0000","last_became_peered":"2023-10-14T16:40:25.779246+0000","last_unstale":"2023-10-14T16:40:25.784659+0000","last_undegraded":"2023-10-14T16:40:25.784659+0000","last_fullsized":"2023-10-14T16:40:25.784659+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:52.494454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769539+0000","last_change":"2023-10-14T16:40:24.764378+0000","last_active":"2023-10-14T16:40:25.769539+0000","last_peered":"2023-10-14T16:40:25.769539+0000","last_clean":"2023-10-14T16:40:25.769539+0000","last_became_active":"2023-10-14T16:40:24.764094+0000","last_became_peered":"2023-10-14T16:40:24.764094+0000","last_unstale":"2023-10-14T16:40:25.769539+0000","last_undegraded":"2023-10-14T16:40:25.769539+0000","last_fullsized":"2023-10-14T16:40:25.769539+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:55.307930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793123+0000","last_change":"2023-10-14T16:40:25.782667+0000","last_active":"2023-10-14T16:40:25.793123+0000","last_peered":"2023-10-14T16:40:25.793123+0000","last_clean":"2023-10-14T16:40:25.793123+0000","last_became_active":"2023-10-14T16:40:25.782311+0000","last_became_peered":"2023-10-14T16:40:25.782311+0000","last_unstale":"2023-10-14T16:40:25.793123+0000","last_undegraded":"2023-10-14T16:40:25.793123+0000","last_fullsized":"2023-10-14T16:40:25.793123+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780460+0000","last_change":"2023-10-14T16:40:24.765776+0000","last_active":"2023-10-14T16:40:26.780460+0000","last_peered":"2023-10-14T16:40:26.780460+0000","last_clean":"2023-10-14T16:40:26.780460+0000","last_became_active":"2023-10-14T16:40:24.761753+0000","last_became_peered":"2023-10-14T16:40:24.761753+0000","last_unstale":"2023-10-14T16:40:26.780460+0000","last_undegraded":"2023-10-14T16:40:26.780460+0000","last_fullsized":"2023-10-14T16:40:26.780460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780507+0000","last_change":"2023-10-14T16:40:25.761768+0000","last_active":"2023-10-14T16:40:26.780507+0000","last_peered":"2023-10-14T16:40:26.780507+0000","last_clean":"2023-10-14T16:40:26.780507+0000","last_became_active":"2023-10-14T16:40:25.761374+0000","last_became_peered":"2023-10-14T16:40:25.761374+0000","last_unstale":"2023-10-14T16:40:26.780507+0000","last_undegraded":"2023-10-14T16:40:26.780507+0000","last_fullsized":"2023-10-14T16:40:26.780507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775466+0000","last_change":"2023-10-14T16:40:24.758887+0000","last_active":"2023-10-14T16:40:27.775466+0000","last_peered":"2023-10-14T16:40:27.775466+0000","last_clean":"2023-10-14T16:40:27.775466+0000","last_became_active":"2023-10-14T16:40:24.758086+0000","last_became_peered":"2023-10-14T16:40:24.758086+0000","last_unstale":"2023-10-14T16:40:27.775466+0000","last_undegraded":"2023-10-14T16:40:27.775466+0000","last_fullsized":"2023-10-14T16:40:27.775466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:03.099034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797811+0000","last_change":"2023-10-14T16:40:25.793027+0000","last_active":"2023-10-14T16:40:25.797811+0000","last_peered":"2023-10-14T16:40:25.797811+0000","last_clean":"2023-10-14T16:40:25.797811+0000","last_became_active":"2023-10-14T16:40:25.791542+0000","last_became_peered":"2023-10-14T16:40:25.791542+0000","last_unstale":"2023-10-14T16:40:25.797811+0000","last_undegraded":"2023-10-14T16:40:25.797811+0000","last_fullsized":"2023-10-14T16:40:25.797811+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779724+0000","last_change":"2023-10-14T16:40:24.763877+0000","last_active":"2023-10-14T16:40:26.779724+0000","last_peered":"2023-10-14T16:40:26.779724+0000","last_clean":"2023-10-14T16:40:26.779724+0000","last_became_active":"2023-10-14T16:40:24.763637+0000","last_became_peered":"2023-10-14T16:40:24.763637+0000","last_unstale":"2023-10-14T16:40:26.779724+0000","last_undegraded":"2023-10-14T16:40:26.779724+0000","last_fullsized":"2023-10-14T16:40:26.779724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797721+0000","last_change":"2023-10-14T16:40:25.792192+0000","last_active":"2023-10-14T16:40:25.797721+0000","last_peered":"2023-10-14T16:40:25.797721+0000","last_clean":"2023-10-14T16:40:25.797721+0000","last_became_active":"2023-10-14T16:40:25.791748+0000","last_became_peered":"2023-10-14T16:40:25.791748+0000","last_unstale":"2023-10-14T16:40:25.797721+0000","last_undegraded":"2023-10-14T16:40:25.797721+0000","last_fullsized":"2023-10-14T16:40:25.797721+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776156+0000","last_change":"2023-10-14T16:40:24.766861+0000","last_active":"2023-10-14T16:40:27.776156+0000","last_peered":"2023-10-14T16:40:27.776156+0000","last_clean":"2023-10-14T16:40:27.776156+0000","last_became_active":"2023-10-14T16:40:24.766319+0000","last_became_peered":"2023-10-14T16:40:24.766319+0000","last_unstale":"2023-10-14T16:40:27.776156+0000","last_undegraded":"2023-10-14T16:40:27.776156+0000","last_fullsized":"2023-10-14T16:40:27.776156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794264+0000","last_change":"2023-10-14T16:40:25.782879+0000","last_active":"2023-10-14T16:40:25.794264+0000","last_peered":"2023-10-14T16:40:25.794264+0000","last_clean":"2023-10-14T16:40:25.794264+0000","last_became_active":"2023-10-14T16:40:25.782368+0000","last_became_peered":"2023-10-14T16:40:25.782368+0000","last_unstale":"2023-10-14T16:40:25.794264+0000","last_undegraded":"2023-10-14T16:40:25.794264+0000","last_fullsized":"2023-10-14T16:40:25.794264+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789069+0000","last_change":"2023-10-14T16:40:24.763429+0000","last_active":"2023-10-14T16:40:25.789069+0000","last_peered":"2023-10-14T16:40:25.789069+0000","last_clean":"2023-10-14T16:40:25.789069+0000","last_became_active":"2023-10-14T16:40:24.761118+0000","last_became_peered":"2023-10-14T16:40:24.761118+0000","last_unstale":"2023-10-14T16:40:25.789069+0000","last_undegraded":"2023-10-14T16:40:25.789069+0000","last_fullsized":"2023-10-14T16:40:25.789069+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.789505+0000","last_change":"2023-10-14T16:40:25.784774+0000","last_active":"2023-10-14T16:40:25.789505+0000","last_peered":"2023-10-14T16:40:25.789505+0000","last_clean":"2023-10-14T16:40:25.789505+0000","last_became_active":"2023-10-14T16:40:25.784416+0000","last_became_peered":"2023-10-14T16:40:25.784416+0000","last_unstale":"2023-10-14T16:40:25.789505+0000","last_undegraded":"2023-10-14T16:40:25.789505+0000","last_fullsized":"2023-10-14T16:40:25.789505+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775852+0000","last_change":"2023-10-14T16:40:24.767764+0000","last_active":"2023-10-14T16:40:27.775852+0000","last_peered":"2023-10-14T16:40:27.775852+0000","last_clean":"2023-10-14T16:40:27.775852+0000","last_became_active":"2023-10-14T16:40:24.767467+0000","last_became_peered":"2023-10-14T16:40:24.767467+0000","last_unstale":"2023-10-14T16:40:27.775852+0000","last_undegraded":"2023-10-14T16:40:27.775852+0000","last_fullsized":"2023-10-14T16:40:27.775852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797777+0000","last_change":"2023-10-14T16:40:25.792940+0000","last_active":"2023-10-14T16:40:25.797777+0000","last_peered":"2023-10-14T16:40:25.797777+0000","last_clean":"2023-10-14T16:40:25.797777+0000","last_became_active":"2023-10-14T16:40:25.790465+0000","last_became_peered":"2023-10-14T16:40:25.790465+0000","last_unstale":"2023-10-14T16:40:25.797777+0000","last_undegraded":"2023-10-14T16:40:25.797777+0000","last_fullsized":"2023-10-14T16:40:25.797777+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":18,"state":"active+clean","last_fresh":"2023-10-14T16:40:24.771054+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:40:24.771054+0000","last_peered":"2023-10-14T16:40:24.771054+0000","last_clean":"2023-10-14T16:40:24.771054+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:40:24.771054+0000","last_undegraded":"2023-10-14T16:40:24.771054+0000","last_fullsized":"2023-10-14T16:40:24.771054+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769523+0000","last_change":"2023-10-14T16:40:24.765633+0000","last_active":"2023-10-14T16:40:25.769523+0000","last_peered":"2023-10-14T16:40:25.769523+0000","last_clean":"2023-10-14T16:40:25.769523+0000","last_became_active":"2023-10-14T16:40:24.760310+0000","last_became_peered":"2023-10-14T16:40:24.760310+0000","last_unstale":"2023-10-14T16:40:25.769523+0000","last_undegraded":"2023-10-14T16:40:25.769523+0000","last_fullsized":"2023-10-14T16:40:25.769523+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:42.771145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775084+0000","last_change":"2023-10-14T16:40:25.783431+0000","last_active":"2023-10-14T16:40:26.775084+0000","last_peered":"2023-10-14T16:40:26.775084+0000","last_clean":"2023-10-14T16:40:26.775084+0000","last_became_active":"2023-10-14T16:40:25.782668+0000","last_became_peered":"2023-10-14T16:40:25.782668+0000","last_unstale":"2023-10-14T16:40:26.775084+0000","last_undegraded":"2023-10-14T16:40:26.775084+0000","last_fullsized":"2023-10-14T16:40:26.775084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.758384+0000","last_change":"2023-10-14T16:40:24.774157+0000","last_active":"2023-10-14T16:40:25.758384+0000","last_peered":"2023-10-14T16:40:25.758384+0000","last_clean":"2023-10-14T16:40:25.758384+0000","last_became_active":"2023-10-14T16:40:24.773773+0000","last_became_peered":"2023-10-14T16:40:24.773773+0000","last_unstale":"2023-10-14T16:40:25.758384+0000","last_undegraded":"2023-10-14T16:40:25.758384+0000","last_fullsized":"2023-10-14T16:40:25.758384+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785039+0000","last_change":"2023-10-14T16:40:25.772755+0000","last_active":"2023-10-14T16:40:25.785039+0000","last_peered":"2023-10-14T16:40:25.785039+0000","last_clean":"2023-10-14T16:40:25.785039+0000","last_became_active":"2023-10-14T16:40:25.772431+0000","last_became_peered":"2023-10-14T16:40:25.772431+0000","last_unstale":"2023-10-14T16:40:25.785039+0000","last_undegraded":"2023-10-14T16:40:25.785039+0000","last_fullsized":"2023-10-14T16:40:25.785039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775795+0000","last_change":"2023-10-14T16:40:24.770677+0000","last_active":"2023-10-14T16:40:27.775795+0000","last_peered":"2023-10-14T16:40:27.775795+0000","last_clean":"2023-10-14T16:40:27.775795+0000","last_became_active":"2023-10-14T16:40:24.770259+0000","last_became_peered":"2023-10-14T16:40:24.770259+0000","last_unstale":"2023-10-14T16:40:27.775795+0000","last_undegraded":"2023-10-14T16:40:27.775795+0000","last_fullsized":"2023-10-14T16:40:27.775795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785363+0000","last_change":"2023-10-14T16:40:25.768484+0000","last_active":"2023-10-14T16:40:25.785363+0000","last_peered":"2023-10-14T16:40:25.785363+0000","last_clean":"2023-10-14T16:40:25.785363+0000","last_became_active":"2023-10-14T16:40:25.767865+0000","last_became_peered":"2023-10-14T16:40:25.767865+0000","last_unstale":"2023-10-14T16:40:25.785363+0000","last_undegraded":"2023-10-14T16:40:25.785363+0000","last_fullsized":"2023-10-14T16:40:25.785363+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:00:03.420096+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.769671+0000","last_change":"2023-10-14T16:40:24.762562+0000","last_active":"2023-10-14T16:40:25.769671+0000","last_peered":"2023-10-14T16:40:25.769671+0000","last_clean":"2023-10-14T16:40:25.769671+0000","last_became_active":"2023-10-14T16:40:24.761399+0000","last_became_peered":"2023-10-14T16:40:24.761399+0000","last_unstale":"2023-10-14T16:40:25.769671+0000","last_undegraded":"2023-10-14T16:40:25.769671+0000","last_fullsized":"2023-10-14T16:40:25.769671+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:40.569040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778450+0000","last_change":"2023-10-14T16:40:25.767520+0000","last_active":"2023-10-14T16:40:26.778450+0000","last_peered":"2023-10-14T16:40:26.778450+0000","last_clean":"2023-10-14T16:40:26.778450+0000","last_became_active":"2023-10-14T16:40:25.765947+0000","last_became_peered":"2023-10-14T16:40:25.765947+0000","last_unstale":"2023-10-14T16:40:26.778450+0000","last_undegraded":"2023-10-14T16:40:26.778450+0000","last_fullsized":"2023-10-14T16:40:26.778450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.751137+0000","last_change":"2023-10-14T16:40:24.758028+0000","last_active":"2023-10-14T16:40:25.751137+0000","last_peered":"2023-10-14T16:40:25.751137+0000","last_clean":"2023-10-14T16:40:25.751137+0000","last_became_active":"2023-10-14T16:40:24.757727+0000","last_became_peered":"2023-10-14T16:40:24.757727+0000","last_unstale":"2023-10-14T16:40:25.751137+0000","last_undegraded":"2023-10-14T16:40:25.751137+0000","last_fullsized":"2023-10-14T16:40:25.751137+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.793310+0000","last_change":"2023-10-14T16:40:25.789247+0000","last_active":"2023-10-14T16:40:25.793310+0000","last_peered":"2023-10-14T16:40:25.793310+0000","last_clean":"2023-10-14T16:40:25.793310+0000","last_became_active":"2023-10-14T16:40:25.788936+0000","last_became_peered":"2023-10-14T16:40:25.788936+0000","last_unstale":"2023-10-14T16:40:25.793310+0000","last_undegraded":"2023-10-14T16:40:25.793310+0000","last_fullsized":"2023-10-14T16:40:25.793310+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.764174+0000","last_change":"2023-10-14T16:40:24.776223+0000","last_active":"2023-10-14T16:40:25.764174+0000","last_peered":"2023-10-14T16:40:25.764174+0000","last_clean":"2023-10-14T16:40:25.764174+0000","last_became_active":"2023-10-14T16:40:24.775651+0000","last_became_peered":"2023-10-14T16:40:24.775651+0000","last_unstale":"2023-10-14T16:40:25.764174+0000","last_undegraded":"2023-10-14T16:40:25.764174+0000","last_fullsized":"2023-10-14T16:40:25.764174+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783772+0000","last_change":"2023-10-14T16:40:25.768945+0000","last_active":"2023-10-14T16:40:25.783772+0000","last_peered":"2023-10-14T16:40:25.783772+0000","last_clean":"2023-10-14T16:40:25.783772+0000","last_became_active":"2023-10-14T16:40:25.768628+0000","last_became_peered":"2023-10-14T16:40:25.768628+0000","last_unstale":"2023-10-14T16:40:25.783772+0000","last_undegraded":"2023-10-14T16:40:25.783772+0000","last_fullsized":"2023-10-14T16:40:25.783772+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.756781+0000","last_change":"2023-10-14T16:40:24.768276+0000","last_active":"2023-10-14T16:40:25.756781+0000","last_peered":"2023-10-14T16:40:25.756781+0000","last_clean":"2023-10-14T16:40:25.756781+0000","last_became_active":"2023-10-14T16:40:24.766828+0000","last_became_peered":"2023-10-14T16:40:24.766828+0000","last_unstale":"2023-10-14T16:40:25.756781+0000","last_undegraded":"2023-10-14T16:40:25.756781+0000","last_fullsized":"2023-10-14T16:40:25.756781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775754+0000","last_change":"2023-10-14T16:40:25.779777+0000","last_active":"2023-10-14T16:40:26.775754+0000","last_peered":"2023-10-14T16:40:26.775754+0000","last_clean":"2023-10-14T16:40:26.775754+0000","last_became_active":"2023-10-14T16:40:25.779380+0000","last_became_peered":"2023-10-14T16:40:25.779380+0000","last_unstale":"2023-10-14T16:40:26.775754+0000","last_undegraded":"2023-10-14T16:40:26.775754+0000","last_fullsized":"2023-10-14T16:40:26.775754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781895+0000","last_change":"2023-10-14T16:40:24.773797+0000","last_active":"2023-10-14T16:40:26.781895+0000","last_peered":"2023-10-14T16:40:26.781895+0000","last_clean":"2023-10-14T16:40:26.781895+0000","last_became_active":"2023-10-14T16:40:24.773498+0000","last_became_peered":"2023-10-14T16:40:24.773498+0000","last_unstale":"2023-10-14T16:40:26.781895+0000","last_undegraded":"2023-10-14T16:40:26.781895+0000","last_fullsized":"2023-10-14T16:40:26.781895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.783586+0000","last_change":"2023-10-14T16:40:25.766374+0000","last_active":"2023-10-14T16:40:25.783586+0000","last_peered":"2023-10-14T16:40:25.783586+0000","last_clean":"2023-10-14T16:40:25.783586+0000","last_became_active":"2023-10-14T16:40:25.764271+0000","last_became_peered":"2023-10-14T16:40:25.764271+0000","last_unstale":"2023-10-14T16:40:25.783586+0000","last_undegraded":"2023-10-14T16:40:25.783586+0000","last_fullsized":"2023-10-14T16:40:25.783586+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.759520+0000","last_change":"2023-10-14T16:40:24.775192+0000","last_active":"2023-10-14T16:40:25.759520+0000","last_peered":"2023-10-14T16:40:25.759520+0000","last_clean":"2023-10-14T16:40:25.759520+0000","last_became_active":"2023-10-14T16:40:24.774186+0000","last_became_peered":"2023-10-14T16:40:24.774186+0000","last_unstale":"2023-10-14T16:40:25.759520+0000","last_undegraded":"2023-10-14T16:40:25.759520+0000","last_fullsized":"2023-10-14T16:40:25.759520+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778279+0000","last_change":"2023-10-14T16:40:25.769524+0000","last_active":"2023-10-14T16:40:26.778279+0000","last_peered":"2023-10-14T16:40:26.778279+0000","last_clean":"2023-10-14T16:40:26.778279+0000","last_became_active":"2023-10-14T16:40:25.769237+0000","last_became_peered":"2023-10-14T16:40:25.769237+0000","last_unstale":"2023-10-14T16:40:26.778279+0000","last_undegraded":"2023-10-14T16:40:26.778279+0000","last_fullsized":"2023-10-14T16:40:26.778279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.753218+0000","last_change":"2023-10-14T16:40:24.765632+0000","last_active":"2023-10-14T16:40:25.753218+0000","last_peered":"2023-10-14T16:40:25.753218+0000","last_clean":"2023-10-14T16:40:25.753218+0000","last_became_active":"2023-10-14T16:40:24.765393+0000","last_became_peered":"2023-10-14T16:40:24.765393+0000","last_unstale":"2023-10-14T16:40:25.753218+0000","last_undegraded":"2023-10-14T16:40:25.753218+0000","last_fullsized":"2023-10-14T16:40:25.753218+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:24.098376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774557+0000","last_change":"2023-10-14T16:40:25.766437+0000","last_active":"2023-10-14T16:40:26.774557+0000","last_peered":"2023-10-14T16:40:26.774557+0000","last_clean":"2023-10-14T16:40:26.774557+0000","last_became_active":"2023-10-14T16:40:25.765798+0000","last_became_peered":"2023-10-14T16:40:25.765798+0000","last_unstale":"2023-10-14T16:40:26.774557+0000","last_undegraded":"2023-10-14T16:40:26.774557+0000","last_fullsized":"2023-10-14T16:40:26.774557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":11,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.750604+0000","last_change":"2023-10-14T16:40:24.765207+0000","last_active":"2023-10-14T16:40:25.750604+0000","last_peered":"2023-10-14T16:40:25.750604+0000","last_clean":"2023-10-14T16:40:25.750604+0000","last_became_active":"2023-10-14T16:40:24.764610+0000","last_became_peered":"2023-10-14T16:40:24.764610+0000","last_unstale":"2023-10-14T16:40:25.750604+0000","last_undegraded":"2023-10-14T16:40:25.750604+0000","last_fullsized":"2023-10-14T16:40:25.750604+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775031+0000","last_change":"2023-10-14T16:40:25.788202+0000","last_active":"2023-10-14T16:40:26.775031+0000","last_peered":"2023-10-14T16:40:26.775031+0000","last_clean":"2023-10-14T16:40:26.775031+0000","last_became_active":"2023-10-14T16:40:25.785927+0000","last_became_peered":"2023-10-14T16:40:25.785927+0000","last_unstale":"2023-10-14T16:40:26.775031+0000","last_undegraded":"2023-10-14T16:40:26.775031+0000","last_fullsized":"2023-10-14T16:40:26.775031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797842+0000","last_change":"2023-10-14T16:40:24.775618+0000","last_active":"2023-10-14T16:40:25.797842+0000","last_peered":"2023-10-14T16:40:25.797842+0000","last_clean":"2023-10-14T16:40:25.797842+0000","last_became_active":"2023-10-14T16:40:24.775346+0000","last_became_peered":"2023-10-14T16:40:24.775346+0000","last_unstale":"2023-10-14T16:40:25.797842+0000","last_undegraded":"2023-10-14T16:40:25.797842+0000","last_fullsized":"2023-10-14T16:40:25.797842+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780727+0000","last_change":"2023-10-14T16:40:25.778910+0000","last_active":"2023-10-14T16:40:26.780727+0000","last_peered":"2023-10-14T16:40:26.780727+0000","last_clean":"2023-10-14T16:40:26.780727+0000","last_became_active":"2023-10-14T16:40:25.777772+0000","last_became_peered":"2023-10-14T16:40:25.777772+0000","last_unstale":"2023-10-14T16:40:26.780727+0000","last_undegraded":"2023-10-14T16:40:26.780727+0000","last_fullsized":"2023-10-14T16:40:26.780727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785645+0000","last_change":"2023-10-14T16:40:25.779144+0000","last_active":"2023-10-14T16:40:25.785645+0000","last_peered":"2023-10-14T16:40:25.785645+0000","last_clean":"2023-10-14T16:40:25.785645+0000","last_became_active":"2023-10-14T16:40:25.778298+0000","last_became_peered":"2023-10-14T16:40:25.778298+0000","last_unstale":"2023-10-14T16:40:25.785645+0000","last_undegraded":"2023-10-14T16:40:25.785645+0000","last_fullsized":"2023-10-14T16:40:25.785645+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.797945+0000","last_change":"2023-10-14T16:40:24.776824+0000","last_active":"2023-10-14T16:40:25.797945+0000","last_peered":"2023-10-14T16:40:25.797945+0000","last_clean":"2023-10-14T16:40:25.797945+0000","last_became_active":"2023-10-14T16:40:24.776368+0000","last_became_peered":"2023-10-14T16:40:24.776368+0000","last_unstale":"2023-10-14T16:40:25.797945+0000","last_undegraded":"2023-10-14T16:40:25.797945+0000","last_fullsized":"2023-10-14T16:40:25.797945+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.785146+0000","last_change":"2023-10-14T16:40:25.777666+0000","last_active":"2023-10-14T16:40:25.785146+0000","last_peered":"2023-10-14T16:40:25.785146+0000","last_clean":"2023-10-14T16:40:25.785146+0000","last_became_active":"2023-10-14T16:40:25.776881+0000","last_became_peered":"2023-10-14T16:40:25.776881+0000","last_unstale":"2023-10-14T16:40:25.785146+0000","last_undegraded":"2023-10-14T16:40:25.785146+0000","last_fullsized":"2023-10-14T16:40:25.785146+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.790530+0000","last_change":"2023-10-14T16:40:24.777751+0000","last_active":"2023-10-14T16:40:25.790530+0000","last_peered":"2023-10-14T16:40:25.790530+0000","last_clean":"2023-10-14T16:40:25.790530+0000","last_became_active":"2023-10-14T16:40:24.777483+0000","last_became_peered":"2023-10-14T16:40:24.777483+0000","last_unstale":"2023-10-14T16:40:25.790530+0000","last_undegraded":"2023-10-14T16:40:25.790530+0000","last_fullsized":"2023-10-14T16:40:25.790530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.788111+0000","last_change":"2023-10-14T16:40:25.783100+0000","last_active":"2023-10-14T16:40:25.788111+0000","last_peered":"2023-10-14T16:40:25.788111+0000","last_clean":"2023-10-14T16:40:25.788111+0000","last_became_active":"2023-10-14T16:40:25.782769+0000","last_became_peered":"2023-10-14T16:40:25.782769+0000","last_unstale":"2023-10-14T16:40:25.788111+0000","last_undegraded":"2023-10-14T16:40:25.788111+0000","last_fullsized":"2023-10-14T16:40:25.788111+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:37:49.253081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775509+0000","last_change":"2023-10-14T16:40:24.768856+0000","last_active":"2023-10-14T16:40:26.775509+0000","last_peered":"2023-10-14T16:40:26.775509+0000","last_clean":"2023-10-14T16:40:26.775509+0000","last_became_active":"2023-10-14T16:40:24.768497+0000","last_became_peered":"2023-10-14T16:40:24.768497+0000","last_unstale":"2023-10-14T16:40:26.775509+0000","last_undegraded":"2023-10-14T16:40:26.775509+0000","last_fullsized":"2023-10-14T16:40:26.775509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:33.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775555+0000","last_change":"2023-10-14T16:40:25.784770+0000","last_active":"2023-10-14T16:40:26.775555+0000","last_peered":"2023-10-14T16:40:26.775555+0000","last_clean":"2023-10-14T16:40:26.775555+0000","last_became_active":"2023-10-14T16:40:25.782997+0000","last_became_peered":"2023-10-14T16:40:25.782997+0000","last_unstale":"2023-10-14T16:40:26.775555+0000","last_undegraded":"2023-10-14T16:40:26.775555+0000","last_fullsized":"2023-10-14T16:40:26.775555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.794922+0000","last_change":"2023-10-14T16:40:24.772362+0000","last_active":"2023-10-14T16:40:25.794922+0000","last_peered":"2023-10-14T16:40:25.794922+0000","last_clean":"2023-10-14T16:40:25.794922+0000","last_became_active":"2023-10-14T16:40:24.772021+0000","last_became_peered":"2023-10-14T16:40:24.772021+0000","last_unstale":"2023-10-14T16:40:25.794922+0000","last_undegraded":"2023-10-14T16:40:25.794922+0000","last_fullsized":"2023-10-14T16:40:25.794922+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27792,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344048,"statfs":{"total":96636764160,"available":96608305152,"internally_reserved":0,"allocated":790528,"data_stored":655548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1160000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.169}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93899999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4319999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.387}]}]},{"osd":6,"up_from":13,"seq":55834574852,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.242}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1220000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.954}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.901}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.0110000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":196608,"data_stored":64688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.268}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.167}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2250000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4159999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0369999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3520000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4910000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574853,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1339999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3049999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6639999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5900000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78300000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0109999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.385}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.772}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.274}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.5960000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4410000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574853,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31368,"kb_used_data":188,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340472,"statfs":{"total":96636764160,"available":96604643328,"internally_reserved":0,"allocated":192512,"data_stored":64598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99099999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79400000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.546}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.609}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4059999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.492}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.766}]}]},{"osd":1,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":200704,"data_stored":64758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80300000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3280000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.288}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.264}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]}]},{"osd":0,"up_from":13,"seq":55834574852,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31968,"kb_used_data":780,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339872,"statfs":{"total":96636764160,"available":96604028928,"internally_reserved":0,"allocated":798720,"data_stored":655604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1759999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.377}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3169999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:40:30.842 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:40:30.843 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-14T16:40:30.843 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-10-14T16:40:30.843 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-14T16:40:30.844 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-10-14T16:40:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.109+0000 7f457cf4e640 1 Processor -- start 2023-10-14T16:40:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.110+0000 7f457cf4e640 1 -- start start 2023-10-14T16:40:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.111+0000 7f457cf4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f457815db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.111+0000 7f457cf4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f457815e3e0 con 0x7f45780b0b20 2023-10-14T16:40:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.111+0000 7f457cf4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45780cd4a0 con 0x7f45780cd0c0 2023-10-14T16:40:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.111+0000 7f457cf4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f457815e100 con 0x7f45781615f0 2023-10-14T16:40:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.112+0000 7f4575d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f45780b0b20 0x7f457815d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47766/0 (socket says 172.21.15.29:47766) 2023-10-14T16:40:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.112+0000 7f4575d9b640 1 -- 172.21.15.29:0/2327924138 learned_addr learned my addr 172.21.15.29:0/2327924138 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.112+0000 7f4576d9d640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f457815db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.112+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766034108 0 0) 0x7f457815e3e0 con 0x7f45780b0b20 2023-10-14T16:40:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.112+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4560003520 con 0x7f45780b0b20 2023-10-14T16:40:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2860750972 0 0) 0x7f4560003520 con 0x7f45780b0b20 2023-10-14T16:40:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 msgr2=0x7f457815db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f457815db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 >> v1:172.21.15.186:6789/0 conn(0x7f45780cd0c0 legacy=0x7f45780b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4578160b10 con 0x7f45780b0b20 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f456c001670 con 0x7f45780b0b20 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 >> v1:172.21.15.29:6789/0 conn(0x7f45780b0b20 legacy=0x7f457815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f4560003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600047c0 0x7f4560004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f45600051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f457815e100 con 0x7f45600039a0 2023-10-14T16:40:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456000bab0 con 0x7f45600047c0 2023-10-14T16:40:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f456000bc70 con 0x7f45781615f0 2023-10-14T16:40:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f457659c640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f4560003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f4575d9b640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600047c0 0x7f4560004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.113+0000 7f4576d9d640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f45600051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.114+0000 7f457659c640 1 -- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 msgr2=0x7f45600051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:31.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.114+0000 7f457659c640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45781615f0 0x7f45600051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.114+0000 7f457659c640 1 -- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600047c0 msgr2=0x7f4560004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.114+0000 7f457659c640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45600047c0 0x7f4560004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.114+0000 7f457659c640 1 -- 172.21.15.29:0/2327924138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560003520 con 0x7f45600039a0 2023-10-14T16:40:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.115+0000 7f457659c640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f4560003dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4568001e50 tx=0x7f45600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab7355c3943c9211 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.115+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568003b70 con 0x7f45600039a0 2023-10-14T16:40:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.115+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568003cd0 con 0x7f45600039a0 2023-10-14T16:40:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.115+0000 7f457559a640 1 -- 172.21.15.29:0/2327924138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568005310 con 0x7f45600039a0 2023-10-14T16:40:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 -- 172.21.15.29:0/2327924138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 msgr2=0x7f4560003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 --2- 172.21.15.29:0/2327924138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f4560003dc0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f4568001e50 tx=0x7f45600057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f4576d9d640 1 -- 172.21.15.29:0/2327924138 reap_dead start 2023-10-14T16:40:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 -- 172.21.15.29:0/2327924138 shutdown_connections 2023-10-14T16:40:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 -- 172.21.15.29:0/2327924138 >> 172.21.15.29:0/2327924138 conn(0x7f45780cb410 msgr2=0x7f45780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 -- 172.21.15.29:0/2327924138 shutdown_connections 2023-10-14T16:40:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.116+0000 7f457cf4e640 1 -- 172.21.15.29:0/2327924138 wait complete. 2023-10-14T16:40:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.118+0000 7f457cf4e640 1 Processor -- start 2023-10-14T16:40:31.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.118+0000 7f457cf4e640 1 -- start start 2023-10-14T16:40:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45780c4b80 0x7f45780b32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45780b3e00 0x7f4578156290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45780b2ed0 con 0x7f45600039a0 2023-10-14T16:40:31.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45780b3810 con 0x7f45780c4b80 2023-10-14T16:40:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457cf4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45780c52d0 con 0x7f45780b3e00 2023-10-14T16:40:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36318/0 (socket says 172.21.15.29:36318) 2023-10-14T16:40:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.119+0000 7f457659c640 1 -- 172.21.15.29:0/3925330281 learned_addr learned my addr 172.21.15.29:0/3925330281 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:31.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f4575d9b640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45780c4b80 0x7f45780b32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f4576d9d640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45780b3e00 0x7f4578156290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 -- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45780b3e00 msgr2=0x7f4578156290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45780b3e00 0x7f4578156290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 -- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45780c4b80 msgr2=0x7f45780b32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45780c4b80 0x7f45780b32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 -- 172.21.15.29:0/3925330281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45780b2530 con 0x7f45600039a0 2023-10-14T16:40:31.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f457659c640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f456802bf90 tx=0x7f45680014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.120+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f456803e070 con 0x7f45600039a0 2023-10-14T16:40:31.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.121+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4568006b50 con 0x7f45600039a0 2023-10-14T16:40:31.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.121+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4568007030 con 0x7f45600039a0 2023-10-14T16:40:31.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.121+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45780b2810 con 0x7f45600039a0 2023-10-14T16:40:31.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.121+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45780c4fe0 con 0x7f45600039a0 2023-10-14T16:40:31.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.122+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f457804e160 con 0x7f45600039a0 2023-10-14T16:40:31.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.123+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4568007190 con 0x7f45600039a0 2023-10-14T16:40:31.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.124+0000 7f4566ffd640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 0x7f455806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.124+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f45680ab1d0 con 0x7f45600039a0 2023-10-14T16:40:31.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.124+0000 7f4575d9b640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 0x7f455806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.125+0000 7f4575d9b640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 0x7f455806d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f456c001c90 tx=0x7f456c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.128+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4568076270 con 0x7f45600039a0 2023-10-14T16:40:31.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.374+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f45780b4910 con 0x7f45600039a0 2023-10-14T16:40:31.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.380+0000 7f4566ffd640 1 -- 172.21.15.29:0/3925330281 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+171030 (secure 0 0 0) 0x7f4568079890 con 0x7f45600039a0 2023-10-14T16:40:31.384 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:31.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 msgr2=0x7f455806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 0x7f455806d330 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f456c001c90 tx=0x7f456c009040 comp rx=0 tx=0).stop 2023-10-14T16:40:31.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 msgr2=0x7f45780c6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f456802bf90 tx=0x7f45680014a0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 shutdown_connections 2023-10-14T16:40:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f455806ae70 0x7f455806d330 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f45780b3e00 0x7f4578156290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45780c4b80 0x7f45780b32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 --2- 172.21.15.29:0/3925330281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45600039a0 0x7f45780c6ba0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 >> 172.21.15.29:0/3925330281 conn(0x7f45780cb410 msgr2=0x7f45780c9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 shutdown_connections 2023-10-14T16:40:31.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.385+0000 7f457cf4e640 1 -- 172.21.15.29:0/3925330281 wait complete. 2023-10-14T16:40:31.406 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":5,"active_gid":4101,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":3113675785},{"type":"v1","addr":"172.21.15.29:6833","nonce":3113675785}]},"active_addr":"172.21.15.29:6833/3113675785","active_change":"2023-10-14T16:40:05.274063+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4103,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2383868706}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3481939635}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1900972994}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":4149980298}]}]} 2023-10-14T16:40:31.411 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-10-14T16:40:31.411 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-10-14T16:40:31.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:31.491 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T16:40:31.490+0000 7f70ddf0c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:40:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.637+0000 7f61ab59e640 1 Processor -- start 2023-10-14T16:40:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.637+0000 7f61ab59e640 1 -- start start 2023-10-14T16:40:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61ab59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61ac15dde0 0x7f61ac0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61ab59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac160a90 con 0x7f61ac15c9f0 2023-10-14T16:40:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61ab59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ac0cd900 con 0x7f61ac070f20 2023-10-14T16:40:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61ab59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ac071350 con 0x7f61ac15dde0 2023-10-14T16:40:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61aa59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61ac15c9f0 0x7f61ac0c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47782/0 (socket says 172.21.15.29:47782) 2023-10-14T16:40:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61aa59c640 1 -- 172.21.15.29:0/3139456228 learned_addr learned my addr 172.21.15.29:0/3139456228 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.639+0000 7f61a1d9b640 1 --2- 172.21.15.29:0/3139456228 >> v2:172.21.15.29:3300/0 conn(0x7f61ac15dde0 0x7f61ac0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.640+0000 7f61a1d9b640 1 -- 172.21.15.29:0/3139456228 >> v1:172.21.15.186:6789/0 conn(0x7f61ac070f20 legacy=0x7f61ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.640+0000 7f61a1d9b640 1 -- 172.21.15.29:0/3139456228 >> v1:172.21.15.29:6789/0 conn(0x7f61ac15c9f0 legacy=0x7f61ac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.640+0000 7f61a1d9b640 1 -- 172.21.15.29:0/3139456228 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ac0cdbe0 con 0x7f61ac15dde0 2023-10-14T16:40:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.640+0000 7f61a9d9b640 1 -- 172.21.15.29:0/3139456228 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2958642199 0 0) 0x7f61ac160a90 con 0x7f61ac15c9f0 2023-10-14T16:40:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.641+0000 7f61a1d9b640 1 --2- 172.21.15.29:0/3139456228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15dde0 0x7f61ac0709e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61940097a0 tx=0x7f619402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5e90d795746e000 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.641+0000 7f61a9d9b640 1 -- 172.21.15.29:0/3139456228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031830 con 0x7f61ac15dde0 2023-10-14T16:40:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.641+0000 7f61a9d9b640 1 -- 172.21.15.29:0/3139456228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194031990 con 0x7f61ac15dde0 2023-10-14T16:40:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.641+0000 7f61a9d9b640 1 -- 172.21.15.29:0/3139456228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194031d20 con 0x7f61ac15dde0 2023-10-14T16:40:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 -- 172.21.15.29:0/3139456228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15dde0 msgr2=0x7f61ac0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 --2- 172.21.15.29:0/3139456228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15dde0 0x7f61ac0709e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f61940097a0 tx=0x7f619402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 -- 172.21.15.29:0/3139456228 shutdown_connections 2023-10-14T16:40:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 --2- 172.21.15.29:0/3139456228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15dde0 0x7f61ac0709e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 -- 172.21.15.29:0/3139456228 >> 172.21.15.29:0/3139456228 conn(0x7f61ac0ce6a0 msgr2=0x7f61ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 -- 172.21.15.29:0/3139456228 shutdown_connections 2023-10-14T16:40:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.642+0000 7f61ab59e640 1 -- 172.21.15.29:0/3139456228 wait complete. 2023-10-14T16:40:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.644+0000 7f61ab59e640 1 Processor -- start 2023-10-14T16:40:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.644+0000 7f61ab59e640 1 -- start start 2023-10-14T16:40:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 0x7f61ac0c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 0x7f61ac0c99a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac1425c0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac143100 con 0x7f61ac070f20 2023-10-14T16:40:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61ab59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ac1427d0 con 0x7f61ac15dde0 2023-10-14T16:40:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61a1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61a1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36328/0 (socket says 172.21.15.29:36328) 2023-10-14T16:40:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.645+0000 7f61a1d9b640 1 -- 172.21.15.29:0/780578432 learned_addr learned my addr 172.21.15.29:0/780578432 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61aa59c640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 0x7f61ac0c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61aad9d640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 0x7f61ac0c99a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 -- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 msgr2=0x7f61ac0c99a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 0x7f61ac0c99a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 -- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 msgr2=0x7f61ac0c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 0x7f61ac0c38b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 -- 172.21.15.29:0/780578432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6194009400 con 0x7f61ac15c9f0 2023-10-14T16:40:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61aad9d640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 0x7f61ac0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61aa59c640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 0x7f61ac0c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a1d9b640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f61940098c0 tx=0x7f61940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619403ba70 con 0x7f61ac15c9f0 2023-10-14T16:40:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.646+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619403bbd0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.647+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61940453f0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.647+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ac142ab0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.647+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ac0c3df0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.648+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ac04e9b0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.649+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f619404e450 con 0x7f61ac15c9f0 2023-10-14T16:40:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.650+0000 7f61a37fe640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 0x7f618c06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.650+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f61940ab920 con 0x7f61ac15c9f0 2023-10-14T16:40:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.650+0000 7f61aa59c640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 0x7f618c06d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.651+0000 7f61aa59c640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 0x7f618c06d380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f619c000f60 tx=0x7f619c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.654+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61940769b0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.857+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61ac0ae930 con 0x7f61ac15c9f0 2023-10-14T16:40:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.859+0000 7f61a37fe640 1 -- 172.21.15.29:0/780578432 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f6194079fd0 con 0x7f61ac15c9f0 2023-10-14T16:40:31.861 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:31.861 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.862+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 msgr2=0x7f618c06d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.862+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 0x7f618c06d380 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f619c000f60 tx=0x7f619c009040 comp rx=0 tx=0).stop 2023-10-14T16:40:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.862+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 msgr2=0x7f61ac0c9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.862+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f61940098c0 tx=0x7f61940046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 shutdown_connections 2023-10-14T16:40:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f618c06aec0 0x7f618c06d380 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61ac15dde0 0x7f61ac0c99a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61ac15c9f0 0x7f61ac0c9460 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 --2- 172.21.15.29:0/780578432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61ac070f20 0x7f61ac0c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 >> 172.21.15.29:0/780578432 conn(0x7f61ac0ce6a0 msgr2=0x7f61ac0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:31.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 shutdown_connections 2023-10-14T16:40:31.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:31.863+0000 7f61ab59e640 1 -- 172.21.15.29:0/780578432 wait complete. 2023-10-14T16:40:31.885 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-10-14T16:40:31.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.106+0000 7f655891d640 1 Processor -- start 2023-10-14T16:40:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.106+0000 7f655891d640 1 -- start start 2023-10-14T16:40:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.107+0000 7f655891d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f655415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f655891d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f655415ed60 con 0x7f65540ce390 2023-10-14T16:40:32.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f655891d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65540707e0 con 0x7f655415ae80 2023-10-14T16:40:32.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f655891d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f655406b810 con 0x7f655415aaa0 2023-10-14T16:40:32.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f6552d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65540ce390 0x7f6554070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47796/0 (socket says 172.21.15.29:47796) 2023-10-14T16:40:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f6552d9d640 1 -- 172.21.15.29:0/2251218192 learned_addr learned my addr 172.21.15.29:0/2251218192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:32.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f655259c640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f655415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.108+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552847324 0 0) 0x7f655415ed60 con 0x7f65540ce390 2023-10-14T16:40:32.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6540003520 con 0x7f65540ce390 2023-10-14T16:40:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 953457741 0 0) 0x7f6540003520 con 0x7f65540ce390 2023-10-14T16:40:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 msgr2=0x7f655415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f655415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 >> v1:172.21.15.186:6789/0 conn(0x7f655415ae80 legacy=0x7f65540cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f655415ef90 con 0x7f65540ce390 2023-10-14T16:40:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f654c001670 con 0x7f65540ce390 2023-10-14T16:40:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 >> v1:172.21.15.29:6789/0 conn(0x7f65540ce390 legacy=0x7f6554070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65400039a0 0x7f6540003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f65400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6540004d00 0x7f6540005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655406b810 con 0x7f6540004d00 2023-10-14T16:40:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f654000b980 con 0x7f65400039a0 2023-10-14T16:40:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.109+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f654000bb40 con 0x7f655415aaa0 2023-10-14T16:40:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6540004d00 0x7f6540005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f655259c640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65400039a0 0x7f6540003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6551d9b640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f65400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 -- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 msgr2=0x7f65400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f655415aaa0 0x7f65400047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 -- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65400039a0 msgr2=0x7f6540003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65400039a0 0x7f6540003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.110+0000 7f6552d9d640 1 -- 172.21.15.29:0/2251218192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6540003520 con 0x7f6540004d00 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.111+0000 7f6552d9d640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6540004d00 0x7f6540005180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c210 tx=0x7f65400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8363aae17b431c00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.111+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c005ad0 con 0x7f6540004d00 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.111+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c005c50 con 0x7f6540004d00 2023-10-14T16:40:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.111+0000 7f655159a640 1 -- 172.21.15.29:0/2251218192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c02c8d0 con 0x7f6540004d00 2023-10-14T16:40:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 -- 172.21.15.29:0/2251218192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6540004d00 msgr2=0x7f6540005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 --2- 172.21.15.29:0/2251218192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6540004d00 0x7f6540005180 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c210 tx=0x7f65400057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f6552d9d640 1 -- 172.21.15.29:0/2251218192 reap_dead start 2023-10-14T16:40:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 -- 172.21.15.29:0/2251218192 shutdown_connections 2023-10-14T16:40:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 -- 172.21.15.29:0/2251218192 >> 172.21.15.29:0/2251218192 conn(0x7f65540cb410 msgr2=0x7f65540cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 -- 172.21.15.29:0/2251218192 shutdown_connections 2023-10-14T16:40:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.112+0000 7f655891d640 1 -- 172.21.15.29:0/2251218192 wait complete. 2023-10-14T16:40:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.114+0000 7f655891d640 1 Processor -- start 2023-10-14T16:40:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.114+0000 7f655891d640 1 -- start start 2023-10-14T16:40:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 0x7f6554154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6554155160 0x7f65540c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65540c5090 con 0x7f65540c4c10 2023-10-14T16:40:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65540c3a50 con 0x7f6540004d00 2023-10-14T16:40:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f655891d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65540c2ed0 con 0x7f6554155160 2023-10-14T16:40:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.115+0000 7f6552d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36354/0 (socket says 172.21.15.29:36354) 2023-10-14T16:40:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 -- 172.21.15.29:0/1688500958 learned_addr learned my addr 172.21.15.29:0/1688500958 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f655259c640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 0x7f6554154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6551d9b640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6554155160 0x7f65540c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 -- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6554155160 msgr2=0x7f65540c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6554155160 0x7f65540c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 -- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 msgr2=0x7f6554154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 0x7f6554154a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 -- 172.21.15.29:0/1688500958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6554154f80 con 0x7f65540c4c10 2023-10-14T16:40:32.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f655259c640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 0x7f6554154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:32.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.116+0000 7f6552d9d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c330 tx=0x7f654c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.117+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c03c040 con 0x7f65540c4c10 2023-10-14T16:40:32.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.117+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f654c02cce0 con 0x7f65540c4c10 2023-10-14T16:40:32.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.117+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654c007290 con 0x7f65540c4c10 2023-10-14T16:40:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.117+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65540c3120 con 0x7f65540c4c10 2023-10-14T16:40:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.117+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65540c36e0 con 0x7f65540c4c10 2023-10-14T16:40:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.118+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65540c3c00 con 0x7f65540c4c10 2023-10-14T16:40:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.119+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f654c032070 con 0x7f65540c4c10 2023-10-14T16:40:32.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.120+0000 7f654affd640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 0x7f653406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.120+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f654c0aa5b0 con 0x7f65540c4c10 2023-10-14T16:40:32.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.120+0000 7f655259c640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 0x7f653406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.121+0000 7f655259c640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 0x7f653406d330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f65541584c0 tx=0x7f654400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.124+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65540c3c00 con 0x7f65540c4c10 2023-10-14T16:40:32.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.321+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65540c3c00 con 0x7f65540c4c10 2023-10-14T16:40:32.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.323+0000 7f654affd640 1 -- 172.21.15.29:0/1688500958 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f65540c3c00 con 0x7f65540c4c10 2023-10-14T16:40:32.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:32.324 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.325+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 msgr2=0x7f653406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.325+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 0x7f653406d330 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f65541584c0 tx=0x7f654400b040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.325+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 msgr2=0x7f6554155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.325+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f654c00c330 tx=0x7f654c02e070 comp rx=0 tx=0).stop 2023-10-14T16:40:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 shutdown_connections 2023-10-14T16:40:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f653406ae70 0x7f653406d330 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65540c4c10 0x7f6554155990 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6554155160 0x7f65540c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 --2- 172.21.15.29:0/1688500958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6540004d00 0x7f6554154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 >> 172.21.15.29:0/1688500958 conn(0x7f65540cb410 msgr2=0x7f655415a070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 shutdown_connections 2023-10-14T16:40:32.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.326+0000 7f655891d640 1 -- 172.21.15.29:0/1688500958 wait complete. 2023-10-14T16:40:32.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-10-14T16:40:32.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-10-14T16:40:32.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-10-14T16:40:32.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-10-14T16:40:32.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-10-14T16:40:32.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-10-14T16:40:32.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2023-10-14T16:40:32.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2023-10-14T16:40:32.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.665+0000 7f2616a1f640 1 Processor -- start 2023-10-14T16:40:32.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.666+0000 7f2616a1f640 1 -- start start 2023-10-14T16:40:32.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.668+0000 7f2616a1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 0x7f260803e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.668+0000 7f2616a1f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 0x7f2608054160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.668+0000 7f2616a1f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f260803c8d0 con 0x7f26080382c0 2023-10-14T16:40:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.668+0000 7f2616a1f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26080546a0 con 0x7f26080e64b0 2023-10-14T16:40:32.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.668+0000 7f2616a1f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2608057200 con 0x7f2608053cf0 2023-10-14T16:40:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.669+0000 7f260f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f26080382c0 0x7f260803c1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47802/0 (socket says 172.21.15.29:47802) 2023-10-14T16:40:32.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.669+0000 7f260f7fe640 1 -- 172.21.15.29:0/1313027249 learned_addr learned my addr 172.21.15.29:0/1313027249 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.669+0000 7f2614fbc640 1 --2- 172.21.15.29:0/1313027249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 0x7f2608054160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f260effd640 1 -- 172.21.15.29:0/1313027249 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641343117 0 0) 0x7f25fc001110 con 0x7f26080382c0 2023-10-14T16:40:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f260effd640 1 -- 172.21.15.29:0/1313027249 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2600003520 con 0x7f26080382c0 2023-10-14T16:40:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f260ffff640 1 --2- 172.21.15.29:0/1313027249 >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 0x7f260803e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f2614fbc640 1 -- 172.21.15.29:0/1313027249 >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 msgr2=0x7f260803e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f2614fbc640 1 --2- 172.21.15.29:0/1313027249 >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 0x7f260803e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f2614fbc640 1 -- 172.21.15.29:0/1313027249 >> v1:172.21.15.29:6789/0 conn(0x7f26080382c0 legacy=0x7f260803c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.670+0000 7f2614fbc640 1 -- 172.21.15.29:0/1313027249 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26080574a0 con 0x7f2608053cf0 2023-10-14T16:40:32.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.671+0000 7f260ffff640 1 --2- 172.21.15.29:0/1313027249 >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 0x7f260803e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:32.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.671+0000 7fcc5359e640 1 Processor -- start 2023-10-14T16:40:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.671+0000 7f8d6144b640 1 Processor -- start 2023-10-14T16:40:32.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.671+0000 7f8d6144b640 1 -- start start 2023-10-14T16:40:32.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.671+0000 7fcc5359e640 1 -- start start 2023-10-14T16:40:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.672+0000 7f2614fbc640 1 --2- 172.21.15.29:0/1313027249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 0x7f2608054160 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f261004f940 tx=0x7f26100a5c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8242a481c19ca79 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f260effd640 1 -- 172.21.15.29:0/1313027249 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100a6aa0 con 0x7f2608053cf0 2023-10-14T16:40:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f260effd640 1 -- 172.21.15.29:0/1313027249 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26100a6c40 con 0x7f2608053cf0 2023-10-14T16:40:32.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7fcc5359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc4c0e78b0 0x7fcc4c04f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fa8459bd640 1 Processor -- start 2023-10-14T16:40:32.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc5359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0536f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc5359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4c050340 con 0x7fcc4c0eb2c0 2023-10-14T16:40:32.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc5359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4c006480 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc5359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c007920 con 0x7fcc4c0eb700 2023-10-14T16:40:32.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fa8459bd640 1 -- start start 2023-10-14T16:40:32.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f260effd640 1 -- 172.21.15.29:0/1313027249 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26100a6aa0 con 0x7f2608053cf0 2023-10-14T16:40:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7f2616a1f640 1 -- 172.21.15.29:0/1313027249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 msgr2=0x7f2608054160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7f2616a1f640 1 --2- 172.21.15.29:0/1313027249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 0x7f2608054160 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f261004f940 tx=0x7f26100a5c30 comp rx=0 tx=0).stop 2023-10-14T16:40:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc5259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc4c0e78b0 0x7fcc4c04f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc51d9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcc4c0eb2c0 0x7fcc4c052fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:38012/0 (socket says 172.21.15.29:38012) 2023-10-14T16:40:32.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7fcc51d9b640 1 -- 172.21.15.29:0/642736476 learned_addr learned my addr 172.21.15.29:0/642736476 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:32.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7fcc52d9d640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0536f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f2616a1f640 1 -- 172.21.15.29:0/1313027249 shutdown_connections 2023-10-14T16:40:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f2616a1f640 1 --2- 172.21.15.29:0/1313027249 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2608053cf0 0x7f2608054160 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f2616a1f640 1 --2- 172.21.15.29:0/1313027249 >> v2:172.21.15.29:3300/0 conn(0x7f26080e64b0 0x7f260803e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f2616a1f640 1 -- 172.21.15.29:0/1313027249 >> 172.21.15.29:0/1313027249 conn(0x7f2608055a30 msgr2=0x7f2608055e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7fcc5159a640 1 -- 172.21.15.29:0/642736476 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 691802718 0 0) 0x7fcc4c050340 con 0x7fcc4c0eb2c0 2023-10-14T16:40:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fcc5159a640 1 -- 172.21.15.29:0/642736476 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc3c003520 con 0x7fcc4c0eb2c0 2023-10-14T16:40:32.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fa8459bd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa838055730 0x7fa8380df500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f8d6144b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d5c1594d0 0x7f8d5c15da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f8d6144b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d5c0c91c0 0x7f8d5c0c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f8d6144b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c15df60 con 0x7f8d5c070c00 2023-10-14T16:40:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f8d6144b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c06fc10 con 0x7f8d5c1594d0 2023-10-14T16:40:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.673+0000 7f8d6144b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c9b30 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7f8d5affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d5c1594d0 0x7f8d5c15da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fa8459bd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 0x7fa838007580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fa8459bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 0x7fa8380e2f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7f8d5affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8d5c1594d0 0x7f8d5c15da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36378/0 (socket says 172.21.15.29:36378) 2023-10-14T16:40:32.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.674+0000 7f8d5affd640 1 -- 172.21.15.29:0/1582233451 learned_addr learned my addr 172.21.15.29:0/1582233451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f8d5affd640 1 -- 172.21.15.29:0/1582233451 >> v2:172.21.15.186:3300/0 conn(0x7f8d5c0c91c0 msgr2=0x7f8d5c0c95f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:40:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f8d5affd640 1 --2- 172.21.15.29:0/1582233451 >> v2:172.21.15.186:3300/0 conn(0x7f8d5c0c91c0 0x7f8d5c0c95f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f8d5affd640 1 -- 172.21.15.29:0/1582233451 >> v1:172.21.15.186:6789/0 conn(0x7f8d5c070c00 legacy=0x7f8d5c0700a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:40:32.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f8d5affd640 1 -- 172.21.15.29:0/1582233451 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c15a8c0 con 0x7f8d5c1594d0 2023-10-14T16:40:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fa8459bd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380064e0 con 0x7fa838055730 2023-10-14T16:40:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.675+0000 7f2616a1f640 1 -- 172.21.15.29:0/1313027249 shutdown_connections 2023-10-14T16:40:32.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7f8d5affd640 1 --2- 172.21.15.29:0/1582233451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c1594d0 0x7f8d5c15da20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c06ac50 tx=0x7f8d5402c0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=858a0ce83e84c795 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fa83ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 0x7fa838007580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8451bc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 0x7fa8380e2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8449bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa838055730 0x7fa8380df500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8451bc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 0x7fa8380e2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:34934/0 (socket says 172.21.15.29:34934) 2023-10-14T16:40:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8451bc640 1 -- 172.21.15.29:0/2328996675 learned_addr learned my addr 172.21.15.29:0/2328996675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8380067c0 con 0x7fa838054260 2023-10-14T16:40:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa838006aa0 con 0x7fa838005fe0 2023-10-14T16:40:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7fa83ffff640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 0x7fa838007580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47770/0 (socket says 172.21.15.29:47770) 2023-10-14T16:40:32.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7f2616a1f640 1 -- 172.21.15.29:0/1313027249 wait complete. 2023-10-14T16:40:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fcc5259c640 1 -- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 msgr2=0x7fcc4c0536f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fcc5259c640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0536f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fcc5259c640 1 -- 172.21.15.29:0/642736476 >> v1:172.21.15.186:6789/0 conn(0x7fcc4c0eb2c0 legacy=0x7fcc4c052fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.676+0000 7fcc5259c640 1 -- 172.21.15.29:0/642736476 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4c04fd80 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.677+0000 7f8d59ffb640 1 -- 172.21.15.29:0/1582233451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5402ca80 con 0x7f8d5c1594d0 2023-10-14T16:40:32.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fcc5259c640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c04f260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097a0 tx=0x7fcc4402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=467903d3cbf1a485 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7f8d59ffb640 1 -- 172.21.15.29:0/1582233451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5402cbe0 con 0x7f8d5c1594d0 2023-10-14T16:40:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fa8449bb640 1 -- 172.21.15.29:0/2328996675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 msgr2=0x7fa838007580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fa8449bb640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 0x7fa838007580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7f8d59ffb640 1 -- 172.21.15.29:0/1582233451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d540343f0 con 0x7f8d5c1594d0 2023-10-14T16:40:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fa8449bb640 1 -- 172.21.15.29:0/2328996675 >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 msgr2=0x7fa8380e2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fcc5159a640 1 -- 172.21.15.29:0/642736476 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc440316f0 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7fcc5159a640 1 -- 172.21.15.29:0/642736476 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc44031850 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7fcc5159a640 1 -- 172.21.15.29:0/642736476 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44031be0 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 -- 172.21.15.29:0/1582233451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c1594d0 msgr2=0x7f8d5c15da20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 --2- 172.21.15.29:0/1582233451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c1594d0 0x7f8d5c15da20 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8d5c06ac50 tx=0x7f8d5402c0e0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 -- 172.21.15.29:0/1582233451 shutdown_connections 2023-10-14T16:40:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 --2- 172.21.15.29:0/1582233451 >> v2:172.21.15.186:3300/0 conn(0x7f8d5c0c91c0 0x7f8d5c0c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 --2- 172.21.15.29:0/1582233451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c1594d0 0x7f8d5c15da20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 -- 172.21.15.29:0/1582233451 >> 172.21.15.29:0/1582233451 conn(0x7f8d5c0ce9b0 msgr2=0x7f8d5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fa8449bb640 1 --2- 172.21.15.29:0/2328996675 >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 0x7fa8380e2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7f2616a1f640 1 Processor -- start 2023-10-14T16:40:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7f2616a1f640 1 -- start start 2023-10-14T16:40:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 -- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0e78b0 msgr2=0x7fcc4c04f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c04f260 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcc440097a0 tx=0x7fcc4402b790 comp rx=0 tx=0).stop 2023-10-14T16:40:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 -- 172.21.15.29:0/642736476 shutdown_connections 2023-10-14T16:40:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0536f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 --2- 172.21.15.29:0/642736476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c04f260 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fcc5359e640 1 -- 172.21.15.29:0/642736476 >> 172.21.15.29:0/642736476 conn(0x7fcc4c054c10 msgr2=0x7fcc4c054ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7fcc5359e640 1 -- 172.21.15.29:0/642736476 shutdown_connections 2023-10-14T16:40:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7fcc5359e640 1 -- 172.21.15.29:0/642736476 wait complete. 2023-10-14T16:40:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7f8d6144b640 1 -- 172.21.15.29:0/1582233451 shutdown_connections 2023-10-14T16:40:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.678+0000 7fa8449bb640 1 -- 172.21.15.29:0/2328996675 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa838006d80 con 0x7fa838055730 2023-10-14T16:40:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.679+0000 7fa8449bb640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838055730 0x7fa8380df500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa84004fa60 tx=0x7fa8400a59b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=761402d65f3e56de server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fa83f7fe640 1 -- 172.21.15.29:0/2328996675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400aa720 con 0x7fa838055730 2023-10-14T16:40:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fa83f7fe640 1 -- 172.21.15.29:0/2328996675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8400aa8c0 con 0x7fa838055730 2023-10-14T16:40:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7fa83f7fe640 1 -- 172.21.15.29:0/2328996675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8400aad10 con 0x7fa838055730 2023-10-14T16:40:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838055730 msgr2=0x7fa8380df500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7fa8459bd640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838055730 0x7fa8380df500 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa84004fa60 tx=0x7fa8400a59b0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 shutdown_connections 2023-10-14T16:40:32.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 --2- 172.21.15.29:0/2328996675 >> v2:172.21.15.186:3300/0 conn(0x7fa838054260 0x7fa8380e2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838005fe0 0x7fa838007580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 --2- 172.21.15.29:0/2328996675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838055730 0x7fa8380df500 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 >> 172.21.15.29:0/2328996675 conn(0x7fa838054c10 msgr2=0x7fa838055020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 shutdown_connections 2023-10-14T16:40:32.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7fa8459bd640 1 -- 172.21.15.29:0/2328996675 wait complete. 2023-10-14T16:40:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7f3ddab1d640 1 Processor -- start 2023-10-14T16:40:32.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.681+0000 7f3ddab1d640 1 -- start start 2023-10-14T16:40:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.680+0000 7f8d6144b640 1 -- 172.21.15.29:0/1582233451 wait complete. 2023-10-14T16:40:32.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.682+0000 7f8d6144b640 1 Processor -- start 2023-10-14T16:40:32.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f8d6144b640 1 -- start start 2023-10-14T16:40:32.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 0x7f26080509c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2608053cf0 0x7f260804ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26080e64b0 0x7f260804f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f260804f700 con 0x7f2608053cf0 2023-10-14T16:40:32.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26080e2d20 con 0x7f26080e64b0 2023-10-14T16:40:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.683+0000 7f2616a1f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26080e35c0 con 0x7f26080382c0 2023-10-14T16:40:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.684+0000 7fcc5359e640 1 Processor -- start 2023-10-14T16:40:32.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.684+0000 7f260ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 0x7f26080509c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.684+0000 7fa8459bd640 1 Processor -- start 2023-10-14T16:40:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 0x7f26080509c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47776/0 (socket says 172.21.15.29:47776) 2023-10-14T16:40:32.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260ffff640 1 -- 172.21.15.29:0/2046613367 learned_addr learned my addr 172.21.15.29:0/2046613367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2608053cf0 0x7f260804ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f3ddab1d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3dd40addb0 0x7f3dd40caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7fcc5359e640 1 -- start start 2023-10-14T16:40:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f3ddab1d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3dd406a4b0 0x7f3dd41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f3ddab1d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd40cb140 con 0x7f3dd40c9000 2023-10-14T16:40:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f8d6144b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c070c00 0x7f8d5c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f3ddab1d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd4161a50 con 0x7f3dd406a4b0 2023-10-14T16:40:32.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f3ddab1d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd406b030 con 0x7f3dd40addb0 2023-10-14T16:40:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f3dd90ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3dd406a4b0 0x7f3dd41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7fa8459bd640 1 -- start start 2023-10-14T16:40:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c070c00 0x7f8d5c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f3dd90ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3dd406a4b0 0x7f3dd41650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36402/0 (socket says 172.21.15.29:36402) 2023-10-14T16:40:32.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 -- 172.21.15.29:0/2046613367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 msgr2=0x7f26080509c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 0x7f26080509c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 -- 172.21.15.29:0/2046613367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26080e64b0 msgr2=0x7f260804f1c0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T16:40:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26080e64b0 0x7f260804f1c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f8d5affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c070c00 0x7f8d5c0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36410/0 (socket says 172.21.15.29:36410) 2023-10-14T16:40:32.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f3dd90ba640 1 -- 172.21.15.29:0/1843884744 learned_addr learned my addr 172.21.15.29:0/1843884744 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.685+0000 7f260f7fe640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f261004e6d0 con 0x7f2608053cf0 2023-10-14T16:40:32.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f260f7fe640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2608053cf0 0x7f260804ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00c780 tx=0x7f25fc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f8d6144b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d5c0c91c0 0x7f8d5c14f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7f260ffff640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26080382c0 0x7f26080509c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:32.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.686+0000 7fcc5359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c0dbf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc0058b0 con 0x7f2608053cf0 2023-10-14T16:40:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25fc005a10 con 0x7f2608053cf0 2023-10-14T16:40:32.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25fc005d80 con 0x7f2608053cf0 2023-10-14T16:40:32.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f3dd88b9640 1 --2- 172.21.15.29:0/1843884744 >> v2:172.21.15.186:3300/0 conn(0x7f3dd40addb0 0x7f3dd40caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0eb2c0 0x7fcc4c04aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa838005fe0 0x7fa8380ceed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 0x7fa8380cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838055730 0x7fa83804d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380dffb0 con 0x7fa838054260 2023-10-14T16:40:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380e0290 con 0x7fa838005fe0 2023-10-14T16:40:32.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fa8459bd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8380e0570 con 0x7fa838055730 2023-10-14T16:40:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0d2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0ea8e0 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0e9fb0 con 0x7fcc4c0eb2c0 2023-10-14T16:40:32.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4c0ea290 con 0x7fcc4c0eb700 2023-10-14T16:40:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c0dbf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c0dbf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36424/0 (socket says 172.21.15.29:36424) 2023-10-14T16:40:32.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7fcc5259c640 1 -- 172.21.15.29:0/1075563561 learned_addr learned my addr 172.21.15.29:0/1075563561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7f8b387b0640 1 Processor -- start 2023-10-14T16:40:32.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc5259c640 1 -- 172.21.15.29:0/1075563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 msgr2=0x7fcc4c0d2180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc5259c640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc4c0eb700 0x7fcc4c0d2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc5259c640 1 -- 172.21.15.29:0/1075563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0eb2c0 msgr2=0x7fcc4c04aeb0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T16:40:32.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc51d9b640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0eb2c0 0x7fcc4c04aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fa83ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 0x7fa8380cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fa83ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 0x7fa8380cc220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36432/0 (socket says 172.21.15.29:36432) 2023-10-14T16:40:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fa83ffff640 1 -- 172.21.15.29:0/3635426843 learned_addr learned my addr 172.21.15.29:0/3635426843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc5259c640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc4c0eb2c0 0x7fcc4c04aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fcc5259c640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44009400 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7f8b387b0640 1 -- start start 2023-10-14T16:40:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.688+0000 7fa8451bc640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838055730 0x7fa83804d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 -- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838055730 msgr2=0x7fa83804d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa838055730 0x7fa83804d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 -- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa838005fe0 msgr2=0x7fa8380ceed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f3dd90ba640 1 -- 172.21.15.29:0/1843884744 >> v2:172.21.15.186:3300/0 conn(0x7f3dd40addb0 msgr2=0x7f3dd40caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1843884744 >> v2:172.21.15.186:3300/0 conn(0x7f3dd40addb0 0x7f3dd40caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f3dd90ba640 1 -- 172.21.15.29:0/1843884744 >> v1:172.21.15.186:6789/0 conn(0x7f3dd40c9000 legacy=0x7f3dd4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f3dd90ba640 1 -- 172.21.15.29:0/1843884744 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd406b310 con 0x7f3dd406a4b0 2023-10-14T16:40:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa8449bb640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa838005fe0 0x7fa8380ceed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa838005fe0 0x7fa8380ceed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa84004f050 con 0x7fa838054260 2023-10-14T16:40:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26080d4260 con 0x7f2608053cf0 2023-10-14T16:40:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26080e3250 con 0x7f2608053cf0 2023-10-14T16:40:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fcc5259c640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c0dbf70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc44037040 tx=0x7fcc440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44034890 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc4c0ea570 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.687+0000 7f8d5affd640 1 -- 172.21.15.29:0/41669073 learned_addr learned my addr 172.21.15.29:0/41669073 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8d6144b640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d5c1594d0 0x7f8d5c0cce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c7220 con 0x7f8d5c070c00 2023-10-14T16:40:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c6ed0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d5c0c6380 con 0x7f8d5c1594d0 2023-10-14T16:40:32.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8d5a7fc640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d5c0c91c0 0x7f8d5c14f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7fa83ffff640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 0x7fa8380cc220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa834007ef0 tx=0x7fa83400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1843884744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd406a4b0 0x7f3dd41650d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b15a0 tx=0x7f3dcc02c140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a071e2f628122ee0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8b387b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b3015d410 0x7f8b300d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8b387b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b300b5d90 con 0x7f8b3015d030 2023-10-14T16:40:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8b387b0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b30160fc0 con 0x7f8b300cd260 2023-10-14T16:40:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.689+0000 7f8b387b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b300d2b50 con 0x7f8b3015d410 2023-10-14T16:40:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2608057a60 con 0x7f2608053cf0 2023-10-14T16:40:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8b3654c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b3015d030 0x7f8b30161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47818/0 (socket says 172.21.15.29:47818) 2023-10-14T16:40:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8b3654c640 1 -- 172.21.15.29:0/875880779 learned_addr learned my addr 172.21.15.29:0/875880779 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8b35d4b640 1 --2- 172.21.15.29:0/875880779 >> v2:172.21.15.29:3300/0 conn(0x7f8b3015d410 0x7f8b300d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8d5a7fc640 1 -- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d5c1594d0 msgr2=0x7f8d5c0cce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8d5a7fc640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d5c1594d0 0x7f8d5c0cce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8d5a7fc640 1 -- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c070c00 msgr2=0x7f8d5c0c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8d5a7fc640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d5c070c00 0x7f8d5c0c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7f8d5a7fc640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d54009b30 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f3dd37fe640 1 -- 172.21.15.29:0/1843884744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc02cae0 con 0x7f3dd406a4b0 2023-10-14T16:40:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f7164de2640 1 Processor -- start 2023-10-14T16:40:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f8b35d4b640 1 -- 172.21.15.29:0/875880779 >> v1:172.21.15.186:6789/0 conn(0x7f8b300cd260 legacy=0x7f8b300b4720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T16:40:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.690+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc4c0dc670 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834014620 con 0x7fa838054260 2023-10-14T16:40:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f3dd37fe640 1 -- 172.21.15.29:0/1843884744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dcc02cc40 con 0x7f3dd406a4b0 2023-10-14T16:40:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f3dd37fe640 1 -- 172.21.15.29:0/1843884744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dcc0343f0 con 0x7f3dd406a4b0 2023-10-14T16:40:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f8b35d4b640 1 -- 172.21.15.29:0/875880779 >> v1:172.21.15.29:6789/0 conn(0x7f8b3015d030 legacy=0x7f8b30161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f8b35d4b640 1 -- 172.21.15.29:0/875880779 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b300d25d0 con 0x7f8b3015d410 2023-10-14T16:40:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa834014780 con 0x7fa838054260 2023-10-14T16:40:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa834014b30 con 0x7fa838054260 2023-10-14T16:40:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8380e0850 con 0x7fa838054260 2023-10-14T16:40:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8380cc760 con 0x7fa838054260 2023-10-14T16:40:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.692+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc440349f0 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.692+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcc4c0ea120 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.692+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1843884744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd406a4b0 msgr2=0x7f3dd41650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.692+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1843884744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd406a4b0 0x7f3dd41650d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b15a0 tx=0x7f3dcc02c140 comp rx=0 tx=0).stop 2023-10-14T16:40:32.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f8d5a7fc640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d5c0c91c0 0x7f8d5c14f630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c890 tx=0x7f8d5000cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.692+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc44034d60 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa820000fc0 con 0x7fa838054260 2023-10-14T16:40:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d5000da40 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000dba0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d50015440 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1843884744 shutdown_connections 2023-10-14T16:40:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.691+0000 7f7164de2640 1 -- start start 2023-10-14T16:40:32.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1843884744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd406a4b0 0x7f3dd41650d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f7164de2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f71600addb0 0x7f71600caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1843884744 >> v2:172.21.15.186:3300/0 conn(0x7f3dd40addb0 0x7f3dd40caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f7164de2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f71600c9000 0x7f7160069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f7164de2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f716006a6f0 con 0x7f716006a2e0 2023-10-14T16:40:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f7164de2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7160161a50 con 0x7f71600addb0 2023-10-14T16:40:32.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1843884744 >> 172.21.15.29:0/1843884744 conn(0x7f3dd40ce9b0 msgr2=0x7f3dd40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f7164de2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71600cb140 con 0x7f71600c9000 2023-10-14T16:40:32.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c0c6660 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c0c5fc0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.694+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1843884744 shutdown_connections 2023-10-14T16:40:32.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.694+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1843884744 wait complete. 2023-10-14T16:40:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f715ed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f716006a2e0 0x7f71600cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:38044/0 (socket says 172.21.15.29:38044) 2023-10-14T16:40:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f715ed9d640 1 -- 172.21.15.29:0/360007374 learned_addr learned my addr 172.21.15.29:0/360007374 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:32.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d5c06f5c0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc4403b040 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f7155d9b640 1 --2- 172.21.15.29:0/360007374 >> v2:172.21.15.186:3300/0 conn(0x7f71600c9000 0x7f7160069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f715e59c640 1 --2- 172.21.15.29:0/360007374 >> v2:172.21.15.29:3300/0 conn(0x7f71600addb0 0x7f71600caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f8b35d4b640 1 --2- 172.21.15.29:0/875880779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b3015d410 0x7f8b300d1630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b200097a0 tx=0x7f8b2002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99d942c4a636c8b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.693+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f25fc007070 con 0x7f2608053cf0 2023-10-14T16:40:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.695+0000 7f260cff9640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f25f406aef0 0x7f25f406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f25fc088ff0 con 0x7f2608053cf0 2023-10-14T16:40:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f260cff9640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f25f4073110 0x7f25f4075530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f25f4075be0 con 0x7f25f4073110 2023-10-14T16:40:32.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f25fc0893e0 con 0x7f2608053cf0 2023-10-14T16:40:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f715dd9b640 1 -- 172.21.15.29:0/360007374 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426859562 0 0) 0x7f716006a6f0 con 0x7f716006a2e0 2023-10-14T16:40:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f715dd9b640 1 -- 172.21.15.29:0/360007374 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c003520 con 0x7f716006a2e0 2023-10-14T16:40:32.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa834014c90 con 0x7fa838054260 2023-10-14T16:40:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f8b3554a640 1 -- 172.21.15.29:0/875880779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20030610 con 0x7f8b3015d410 2023-10-14T16:40:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f8b3554a640 1 -- 172.21.15.29:0/875880779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b20030770 con 0x7f8b3015d410 2023-10-14T16:40:32.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f8b3554a640 1 -- 172.21.15.29:0/875880779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b20030b00 con 0x7f8b3015d410 2023-10-14T16:40:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f7155d9b640 1 -- 172.21.15.29:0/360007374 >> v2:172.21.15.29:3300/0 conn(0x7f71600addb0 msgr2=0x7f71600caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f7155d9b640 1 --2- 172.21.15.29:0/360007374 >> v2:172.21.15.29:3300/0 conn(0x7f71600addb0 0x7f71600caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7fcc42ffd640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcc3406ae70 0x7fcc3406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc440ab0a0 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7fcc42ffd640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7fcc34073010 0x7fcc34075430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcc34075ae0 con 0x7fcc34073010 2023-10-14T16:40:32.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc440ad680 con 0x7fcc4c0e78b0 2023-10-14T16:40:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 -- 172.21.15.29:0/875880779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b3015d410 msgr2=0x7f8b300d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 --2- 172.21.15.29:0/875880779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b3015d410 0x7f8b300d1630 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b200097a0 tx=0x7f8b2002b960 comp rx=0 tx=0).stop 2023-10-14T16:40:32.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fcc51d9b640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcc3406ae70 0x7fcc3406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 -- 172.21.15.29:0/875880779 shutdown_connections 2023-10-14T16:40:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 --2- 172.21.15.29:0/875880779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b3015d410 0x7f8b300d1630 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 -- 172.21.15.29:0/875880779 >> 172.21.15.29:0/875880779 conn(0x7f8b300cf720 msgr2=0x7f8b300cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.696+0000 7f7155d9b640 1 -- 172.21.15.29:0/360007374 >> v1:172.21.15.186:6789/0 conn(0x7f716006a2e0 legacy=0x7f71600cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f7155d9b640 1 -- 172.21.15.29:0/360007374 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f716006b030 con 0x7f71600c9000 2023-10-14T16:40:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fa83d7fa640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa82406af70 0x7fa82406d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f3ddab1d640 1 Processor -- start 2023-10-14T16:40:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f3ddab1d640 1 -- start start 2023-10-14T16:40:32.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fcc52d9d640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7fcc34073010 0x7fcc34075430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8d500155a0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f8b387b0640 1 -- 172.21.15.29:0/875880779 shutdown_connections 2023-10-14T16:40:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.697+0000 7f260ffff640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f25f406aef0 0x7f25f406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f2614fbc640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f25f4073110 0x7f25f4075530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa83408ada0 con 0x7fa838054260 2023-10-14T16:40:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fa83d7fa640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7fa824073190 0x7fa8240755b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa824075c60 con 0x7fa824073190 2023-10-14T16:40:32.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa83408b150 con 0x7fa838054260 2023-10-14T16:40:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f3ddab1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dd406a4b0 0x7f3dd415fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f3ddab1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 0x7f3dd40c0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3ddab1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dd40c9000 0x7f3dd40be2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3ddab1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b1740 con 0x7f3dd40addb0 2023-10-14T16:40:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3ddab1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b1a20 con 0x7f3dd40c9000 2023-10-14T16:40:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3ddab1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b1d00 con 0x7f3dd406a4b0 2023-10-14T16:40:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.698+0000 7f7155d9b640 1 --2- 172.21.15.29:0/360007374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71600c9000 0x7f7160069da0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f71600cdbe0 tx=0x7f714802b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4aac087a9e12761a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f715dd9b640 1 -- 172.21.15.29:0/360007374 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148031640 con 0x7f71600c9000 2023-10-14T16:40:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f715dd9b640 1 -- 172.21.15.29:0/360007374 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71480317a0 con 0x7f71600c9000 2023-10-14T16:40:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f715dd9b640 1 -- 172.21.15.29:0/360007374 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148031640 con 0x7f71600c9000 2023-10-14T16:40:32.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 0x7f3dd40c0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3dd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 0x7f3dd40c0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36468/0 (socket says 172.21.15.29:36468) 2023-10-14T16:40:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3dd3fff640 1 -- 172.21.15.29:0/1747644519 learned_addr learned my addr 172.21.15.29:0/1747644519 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f3dd88b9640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dd406a4b0 0x7f3dd415fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f260ffff640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f25f406aef0 0x7f25f406d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f260400ab50 tx=0x7f2604008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f2614fbc640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f25f4073110 0x7f25f4075530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dd40c9000 0x7f3dd40be2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f7164de2640 1 -- 172.21.15.29:0/360007374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71600c9000 msgr2=0x7f7160069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f7164de2640 1 --2- 172.21.15.29:0/360007374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71600c9000 0x7f7160069da0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f71600cdbe0 tx=0x7f714802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd3fff640 1 -- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dd406a4b0 msgr2=0x7f3dd415fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd3fff640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dd406a4b0 0x7f3dd415fda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd3fff640 1 -- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dd40c9000 msgr2=0x7f3dd40be2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd3fff640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dd40c9000 0x7f3dd40be2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd3fff640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc009b30 con 0x7f3dd40addb0 2023-10-14T16:40:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dd40c9000 0x7f3dd40be2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7f3dd88b9640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dd406a4b0 0x7f3dd415fda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:32.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8d437fe640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8d4806af40 0x7f8d4806d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f7164de2640 1 -- 172.21.15.29:0/360007374 shutdown_connections 2023-10-14T16:40:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f7164de2640 1 --2- 172.21.15.29:0/360007374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71600c9000 0x7f7160069da0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f7164de2640 1 --2- 172.21.15.29:0/360007374 >> v2:172.21.15.29:3300/0 conn(0x7f71600addb0 0x7f71600caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f7164de2640 1 -- 172.21.15.29:0/360007374 >> 172.21.15.29:0/360007374 conn(0x7f71600ce9b0 msgr2=0x7f71600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8d50089fd0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8d5affd640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8d4806af40 0x7f8d4806d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f3dd3fff640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 0x7f3dd40c0f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000b3e0 tx=0x7f3dc000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f7164de2640 1 -- 172.21.15.29:0/360007374 shutdown_connections 2023-10-14T16:40:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8d437fe640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f8d48073160 0x7f8d48075580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d48075c30 con 0x7f8d48073160 2023-10-14T16:40:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d5008c5f0 con 0x7f8d5c0c91c0 2023-10-14T16:40:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7f8b387b0640 1 -- 172.21.15.29:0/875880779 wait complete. 2023-10-14T16:40:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f8b387b0640 1 Processor -- start 2023-10-14T16:40:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.701+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011830 con 0x7f3dd40addb0 2023-10-14T16:40:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc00119b0 con 0x7f3dd40addb0 2023-10-14T16:40:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dc0011d60 con 0x7f3dd40addb0 2023-10-14T16:40:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dd40b1fe0 con 0x7f3dd40addb0 2023-10-14T16:40:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd41602e0 con 0x7f3dd40addb0 2023-10-14T16:40:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== osd.0 v2:172.21.15.29:6800/3107383011 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f25f4075be0 con 0x7f25f4073110 2023-10-14T16:40:32.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f68d814e640 1 Processor -- start 2023-10-14T16:40:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7fa8451bc640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7fa824073190 0x7fa8240755b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.699+0000 7fa8449bb640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa82406af70 0x7fa82406d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.700+0000 7fa8449bb640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa82406af70 0x7fa82406d430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa838055d10 tx=0x7fa84006c030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f68d814e640 1 -- start start 2023-10-14T16:40:32.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.703+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3dd406b8d0 con 0x7f3dd40addb0 2023-10-14T16:40:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f7164de2640 1 -- 172.21.15.29:0/360007374 wait complete. 2023-10-14T16:40:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.703+0000 7fcc51d9b640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcc3406ae70 0x7fcc3406d330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c04dd50 tx=0x7fcc48009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.703+0000 7fa8451bc640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7fa824073190 0x7fa8240755b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f8b387b0640 1 -- start start 2023-10-14T16:40:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f8d5b7fe640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f8d48073160 0x7f8d48075580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.702+0000 7f8d5affd640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8d4806af40 0x7f8d4806d400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d5402c120 tx=0x7f8d54030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d814e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68d00b3420 0x7f68d00b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d814e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f68d00cfaf0 0x7f68d00b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d814e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d00d3980 con 0x7f68d00cf710 2023-10-14T16:40:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d814e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d00d3c60 con 0x7f68d00b3420 2023-10-14T16:40:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d814e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d00ce630 con 0x7f68d00cfaf0 2023-10-14T16:40:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7f68d56e9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68d00b3420 0x7f68d00b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d56e9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f68d00b3420 0x7f68d00b5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36484/0 (socket says 172.21.15.29:36484) 2023-10-14T16:40:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d56e9640 1 -- 172.21.15.29:0/542339913 learned_addr learned my addr 172.21.15.29:0/542339913 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d66eb640 1 --2- 172.21.15.29:0/542339913 >> v2:172.21.15.186:3300/0 conn(0x7f68d00cfaf0 0x7f68d00b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d56e9640 1 -- 172.21.15.29:0/542339913 >> v2:172.21.15.186:3300/0 conn(0x7f68d00cfaf0 msgr2=0x7f68d00b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d56e9640 1 --2- 172.21.15.29:0/542339913 >> v2:172.21.15.186:3300/0 conn(0x7f68d00cfaf0 0x7f68d00b6200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== osd.3 v2:172.21.15.29:6811/149520408 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fa824075c60 con 0x7fa824073190 2023-10-14T16:40:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3dc001a400 con 0x7f3dd40addb0 2023-10-14T16:40:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f68d56e9640 1 -- 172.21.15.29:0/542339913 >> v1:172.21.15.29:6789/0 conn(0x7f68d00cf710 legacy=0x7f68d00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f68d56e9640 1 -- 172.21.15.29:0/542339913 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00ce910 con 0x7f68d00b3420 2023-10-14T16:40:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.704+0000 7fcc52d9d640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7fcc34073010 0x7fcc34075430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 0x7f8b301601a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b3015d030 0x7f8b3015a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b3015d410 0x7f8b300c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b3015b900 con 0x7f8b300cd260 2023-10-14T16:40:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300c6c00 con 0x7f8b3015d410 2023-10-14T16:40:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.705+0000 7f8b387b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b300c6ee0 con 0x7f8b3015d030 2023-10-14T16:40:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 0x7f8b301601a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 0x7f8b301601a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36500/0 (socket says 172.21.15.29:36500) 2023-10-14T16:40:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 -- 172.21.15.29:0/902552587 learned_addr learned my addr 172.21.15.29:0/902552587 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b36d4d640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b3015d410 0x7f8b300c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b35d4b640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b3015d030 0x7f8b3015a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== osd.1 v2:172.21.15.29:6814/3723048683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fcc34075ae0 con 0x7fcc34073010 2023-10-14T16:40:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8d5b7fe640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f8d48073160 0x7f8d48075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 -- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b3015d030 msgr2=0x7f8b3015a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b3015d030 0x7f8b3015a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 -- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b3015d410 msgr2=0x7f8b300c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.706+0000 7f8b3654c640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b3015d410 0x7f8b300c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.707+0000 7f8b3654c640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b20009400 con 0x7f8b300cd260 2023-10-14T16:40:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.707+0000 7f68d56e9640 1 --2- 172.21.15.29:0/542339913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00b3420 0x7f68d00b5cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68d00d07a0 tx=0x7f68c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7769ae6b98fc527 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.707+0000 7f8b3654c640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 0x7f8b301601a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cba0 tx=0x7f8b2800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800ad40 con 0x7f8b300cd260 2023-10-14T16:40:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f7164de2640 1 Processor -- start 2023-10-14T16:40:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b28012400 con 0x7f8b300cd260 2023-10-14T16:40:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b300c71c0 con 0x7f8b300cd260 2023-10-14T16:40:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b300c7780 con 0x7f8b300cd260 2023-10-14T16:40:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b2800ad40 con 0x7f8b300cd260 2023-10-14T16:40:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.708+0000 7f7164de2640 1 -- start start 2023-10-14T16:40:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f716006a2e0 0x7f716014c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 0x7f71600b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71600c9000 0x7f71600b1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71600b2f90 con 0x7f71600addb0 2023-10-14T16:40:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71600b38e0 con 0x7f716006a2e0 2023-10-14T16:40:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.709+0000 7f7164de2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71600c7720 con 0x7f71600c9000 2023-10-14T16:40:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f68d4ee8640 1 -- 172.21.15.29:0/542339913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0031870 con 0x7f68d00b3420 2023-10-14T16:40:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f68d4ee8640 1 -- 172.21.15.29:0/542339913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c00319d0 con 0x7f68d00b3420 2023-10-14T16:40:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f68d4ee8640 1 -- 172.21.15.29:0/542339913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c0031d60 con 0x7f68d00b3420 2023-10-14T16:40:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== osd.2 v2:172.21.15.29:6808/3586046734 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f8d48075c30 con 0x7f8d48073160 2023-10-14T16:40:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f3dd17fa640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3db806aec0 0x7f3db806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3dc0089c90 con 0x7f3dd40addb0 2023-10-14T16:40:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f3dd17fa640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f3db80730e0 0x7f3db8075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3db8075bb0 con 0x7f3db80730e0 2023-10-14T16:40:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.710+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3dc008a130 con 0x7f3dd40addb0 2023-10-14T16:40:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f7155d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 0x7f71600b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 -- 172.21.15.29:0/542339913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00b3420 msgr2=0x7f68d00b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 --2- 172.21.15.29:0/542339913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00b3420 0x7f68d00b5cc0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f68d00d07a0 tx=0x7f68c002b910 comp rx=0 tx=0).stop 2023-10-14T16:40:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f7155d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 0x7f71600b15a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36508/0 (socket says 172.21.15.29:36508) 2023-10-14T16:40:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f7155d9b640 1 -- 172.21.15.29:0/1099939600 learned_addr learned my addr 172.21.15.29:0/1099939600 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 -- 172.21.15.29:0/542339913 shutdown_connections 2023-10-14T16:40:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 --2- 172.21.15.29:0/542339913 >> v2:172.21.15.186:3300/0 conn(0x7f68d00cfaf0 0x7f68d00b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 --2- 172.21.15.29:0/542339913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00b3420 0x7f68d00b5cc0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f68d814e640 1 -- 172.21.15.29:0/542339913 >> 172.21.15.29:0/542339913 conn(0x7f68d00d2cc0 msgr2=0x7f68d00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8b300d2870 con 0x7f8b300cd260 2023-10-14T16:40:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.711+0000 7f3dd88b9640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3db806aec0 0x7f3db806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.712+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f3db80730e0 0x7f3db8075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.712+0000 7f68d814e640 1 -- 172.21.15.29:0/542339913 shutdown_connections 2023-10-14T16:40:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.712+0000 7f68d814e640 1 -- 172.21.15.29:0/542339913 wait complete. 2023-10-14T16:40:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.712+0000 7f715ed9d640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71600c9000 0x7f71600b1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f7155d9b640 1 -- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71600c9000 msgr2=0x7f71600b1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f7155d9b640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71600c9000 0x7f71600b1ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f7155d9b640 1 -- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f716006a2e0 msgr2=0x7f716014c470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T16:40:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f7155d9b640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f716006a2e0 0x7f716014c470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f7155d9b640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7148009400 con 0x7f71600addb0 2023-10-14T16:40:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f3dd88b9640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3db806aec0 0x7f3db806d380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b4930 tx=0x7f3dc8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.713+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8b28012570 con 0x7f8b300cd260 2023-10-14T16:40:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.714+0000 7f68d814e640 1 Processor -- start 2023-10-14T16:40:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.714+0000 7f3dd90ba640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f3db80730e0 0x7f3db8075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.714+0000 7f7155d9b640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 0x7f71600b15a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f714802bf00 tx=0x7f71480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.714+0000 7f68d814e640 1 -- start start 2023-10-14T16:40:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.714+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148006e10 con 0x7f71600addb0 2023-10-14T16:40:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f8b26ffd640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8b1406aed0 0x7f8b1406d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f714803f070 con 0x7f71600addb0 2023-10-14T16:40:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71600c7a00 con 0x7f71600addb0 2023-10-14T16:40:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71600c7fc0 con 0x7f71600addb0 2023-10-14T16:40:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7148003a40 con 0x7f71600addb0 2023-10-14T16:40:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68d00b3420 0x7f68d015e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00cf710 0x7f68d01e6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.715+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8b28057f40 con 0x7f8b300cd260 2023-10-14T16:40:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f8b35d4b640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8b1406aed0 0x7f8b1406d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d00cfaf0 0x7f68d015a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f8b26ffd640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f8b14073d50 0x7f8b140761f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8b140768c0 con 0x7f8b14073d50 2023-10-14T16:40:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b2800aea0 con 0x7f8b300cd260 2023-10-14T16:40:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d015d5a0 con 0x7f68d00cf710 2023-10-14T16:40:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d015d880 con 0x7f68d00cfaf0 2023-10-14T16:40:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d814e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d015db60 con 0x7f68d00b3420 2023-10-14T16:40:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.716+0000 7f68d5eea640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68d00b3420 0x7f68d015e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d5eea640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68d00b3420 0x7f68d015e640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47856/0 (socket says 172.21.15.29:47856) 2023-10-14T16:40:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d5eea640 1 -- 172.21.15.29:0/1846178435 learned_addr learned my addr 172.21.15.29:0/1846178435 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00cf710 0x7f68d01e6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f8b35d4b640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8b1406aed0 0x7f8b1406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b20036000 tx=0x7f8b2003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== osd.4 v2:172.21.15.186:6800/189625010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f3db8075bb0 con 0x7f3db80730e0 2023-10-14T16:40:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d66eb640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d00cfaf0 0x7f68d015a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 -- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68d00b3420 msgr2=0x7f68d015e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f68d00b3420 0x7f68d015e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 -- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d00cfaf0 msgr2=0x7f68d015a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f68d00cfaf0 0x7f68d015a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.717+0000 7f68d56e9640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c00094d0 con 0x7f68d00cf710 2023-10-14T16:40:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.718+0000 7f68d56e9640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00cf710 0x7f68d01e6360 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f68c0033000 tx=0x7f68c00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.719+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c003ad70 con 0x7f68d00cf710 2023-10-14T16:40:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.719+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d015de40 con 0x7f68d00cf710 2023-10-14T16:40:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.719+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d01e68a0 con 0x7f68d00cf710 2023-10-14T16:40:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.719+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c00443f0 con 0x7f68d00cf710 2023-10-14T16:40:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.719+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68c00447a0 con 0x7f68d00cf710 2023-10-14T16:40:32.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.720+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f68b8000fc0 con 0x7f68d00cf710 2023-10-14T16:40:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.718+0000 7f8b36d4d640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f8b14073d50 0x7f8b140761f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.718+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7144000fc0 con 0x7f71600addb0 2023-10-14T16:40:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.721+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f714802f050 con 0x7f71600addb0 2023-10-14T16:40:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.722+0000 7f8b36d4d640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f8b14073d50 0x7f8b140761f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.722+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f68c004d450 con 0x7f68d00cf710 2023-10-14T16:40:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f71577fe640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f714006ae70 0x7f714006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f68c67fc640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f68b406af40 0x7f68b406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f7148041070 con 0x7f71600addb0 2023-10-14T16:40:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f715e59c640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f714006ae70 0x7f714006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f68c002d040 con 0x7f68d00cf710 2023-10-14T16:40:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f68c67fc640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7f68b40730e0 0x7f68b4075500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f71577fe640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7140073090 0x7f71400754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:32.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68b4075bb0 con 0x7f68b40730e0 2023-10-14T16:40:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f68d5eea640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f68b406af40 0x7f68b406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7140075b60 con 0x7f7140073090 2023-10-14T16:40:32.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.723+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f71480b02a0 con 0x7f71600addb0 2023-10-14T16:40:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.724+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== osd.5 v2:172.21.15.186:6816/1666789598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f8b140768c0 con 0x7f8b14073d50 2023-10-14T16:40:32.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.724+0000 7f715ed9d640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7140073090 0x7f71400754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.724+0000 7f68d66eb640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7f68b40730e0 0x7f68b4075500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:32.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.725+0000 7f715ed9d640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7140073090 0x7f71400754b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.726+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68c0076ba0 con 0x7f68d00cf710 2023-10-14T16:40:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.726+0000 7f68d66eb640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7f68b40730e0 0x7f68b4075500 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.727+0000 7f68d5eea640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f68b406af40 0x7f68b406d400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68cc001e90 tx=0x7f68cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.728+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== osd.6 v2:172.21.15.186:6824/3237784830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f7140075b60 con 0x7f7140073090 2023-10-14T16:40:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.728+0000 7f715e59c640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f714006ae70 0x7f714006d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f715000e190 tx=0x7f715000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:32.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.731+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== osd.7 v2:172.21.15.186:6808/4274731638 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f68b4075bb0 con 0x7f68b40730e0 2023-10-14T16:40:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.739+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 --> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d5c071170 con 0x7f8d48073160 2023-10-14T16:40:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.741+0000 7f8d437fe640 1 -- 172.21.15.29:0/41669073 <== osd.2 v2:172.21.15.29:6808/3586046734 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8d4c009280 con 0x7f8d48073160 2023-10-14T16:40:32.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.741+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 --> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3dd406b5f0 con 0x7f3db80730e0 2023-10-14T16:40:32.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 --> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2608057780 con 0x7f25f4073110 2023-10-14T16:40:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f8d48073160 msgr2=0x7f8d48075580 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f8d48073160 0x7f8d48075580 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8d4806af40 msgr2=0x7f8d4806d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8d4806af40 0x7f8d4806d400 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8d5402c120 tx=0x7f8d54030000 comp rx=0 tx=0).stop 2023-10-14T16:40:32.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d5c0c91c0 msgr2=0x7f8d5c14f630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d6144b640 1 --2- 172.21.15.29:0/41669073 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d5c0c91c0 0x7f8d5c14f630 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f8d5000c890 tx=0x7f8d5000cd50 comp rx=0 tx=0).stop 2023-10-14T16:40:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f3dd17fa640 1 -- 172.21.15.29:0/1747644519 <== osd.4 v2:172.21.15.186:6800/189625010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3dcc009290 con 0x7f3db80730e0 2023-10-14T16:40:32.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.742+0000 7f8d5b7fe640 1 -- 172.21.15.29:0/41669073 reap_dead start 2023-10-14T16:40:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f260cff9640 1 -- 172.21.15.29:0/2046613367 <== osd.0 v2:172.21.15.29:6800/3107383011 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f26100a68a0 con 0x7f25f4073110 2023-10-14T16:40:32.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 shutdown_connections 2023-10-14T16:40:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 >> 172.21.15.29:0/41669073 conn(0x7f8d5c0ce9b0 msgr2=0x7f8d5c0700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f3db80730e0 msgr2=0x7f3db8075500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f3db80730e0 0x7f3db8075500 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 shutdown_connections 2023-10-14T16:40:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f25f4073110 msgr2=0x7f25f4075530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f2616a1f640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f25f4073110 0x7f25f4075530 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3db806aec0 msgr2=0x7f3db806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3db806aec0 0x7f3db806d380 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f3dd40b4930 tx=0x7f3dc8008040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 msgr2=0x7f3dd40c0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 --2- 172.21.15.29:0/1747644519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3dd40addb0 0x7f3dd40c0f80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3dc000b3e0 tx=0x7f3dc000b8a0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3dd90ba640 1 -- 172.21.15.29:0/1747644519 reap_dead start 2023-10-14T16:40:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 shutdown_connections 2023-10-14T16:40:32.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 >> 172.21.15.29:0/1747644519 conn(0x7f3dd40ce9b0 msgr2=0x7f3dd40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f8d6144b640 1 -- 172.21.15.29:0/41669073 wait complete. 2023-10-14T16:40:32.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.743+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f25f406aef0 msgr2=0x7f25f406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2616a1f640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f25f406aef0 0x7f25f406d3b0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f260400ab50 tx=0x7f2604008040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2608053cf0 msgr2=0x7f260804ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2616a1f640 1 --2- 172.21.15.29:0/2046613367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2608053cf0 0x7f260804ec80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f25fc00c780 tx=0x7f25fc00cc40 comp rx=0 tx=0).stop 2023-10-14T16:40:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2614fbc640 1 -- 172.21.15.29:0/2046613367 reap_dead start 2023-10-14T16:40:32.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 shutdown_connections 2023-10-14T16:40:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 >> 172.21.15.29:0/2046613367 conn(0x7f2608055a30 msgr2=0x7f2608058160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 shutdown_connections 2023-10-14T16:40:32.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7f3ddab1d640 1 -- 172.21.15.29:0/1747644519 wait complete. 2023-10-14T16:40:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.744+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 --> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa820002cb0 con 0x7fa824073190 2023-10-14T16:40:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.745+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 shutdown_connections 2023-10-14T16:40:32.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.746+0000 7f2616a1f640 1 -- 172.21.15.29:0/2046613367 wait complete. 2023-10-14T16:40:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.747+0000 7fa83d7fa640 1 -- 172.21.15.29:0/3635426843 <== osd.3 v2:172.21.15.29:6811/149520408 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa820002cb0 con 0x7fa824073190 2023-10-14T16:40:32.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.750+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7fa824073190 msgr2=0x7fa8240755b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.750+0000 7fa8459bd640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7fa824073190 0x7fa8240755b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.751+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 --> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcc4c006110 con 0x7fcc34073010 2023-10-14T16:40:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa82406af70 msgr2=0x7fa82406d430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa82406af70 0x7fa82406d430 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa838055d10 tx=0x7fa84006c030 comp rx=0 tx=0).stop 2023-10-14T16:40:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 msgr2=0x7fa8380cc220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 --2- 172.21.15.29:0/3635426843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa838054260 0x7fa8380cc220 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa834007ef0 tx=0x7fa83400e4b0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8451bc640 1 -- 172.21.15.29:0/3635426843 reap_dead start 2023-10-14T16:40:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fcc42ffd640 1 -- 172.21.15.29:0/1075563561 <== osd.1 v2:172.21.15.29:6814/3723048683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcc5404e750 con 0x7fcc34073010 2023-10-14T16:40:32.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 shutdown_connections 2023-10-14T16:40:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.752+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 >> 172.21.15.29:0/3635426843 conn(0x7fa838054c10 msgr2=0x7fa8380d62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.753+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7fcc34073010 msgr2=0x7fcc34075430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.753+0000 7fcc5359e640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7fcc34073010 0x7fcc34075430 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 shutdown_connections 2023-10-14T16:40:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcc3406ae70 msgr2=0x7fcc3406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcc3406ae70 0x7fcc3406d330 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fcc4c04dd50 tx=0x7fcc48009040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 msgr2=0x7fcc4c0dbf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 --2- 172.21.15.29:0/1075563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc4c0e78b0 0x7fcc4c0dbf70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcc44037040 tx=0x7fcc440046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc52d9d640 1 -- 172.21.15.29:0/1075563561 reap_dead start 2023-10-14T16:40:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 shutdown_connections 2023-10-14T16:40:32.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 >> 172.21.15.29:0/1075563561 conn(0x7fcc4c054c10 msgr2=0x7fcc4c04e3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.755+0000 7fa8459bd640 1 -- 172.21.15.29:0/3635426843 wait complete. 2023-10-14T16:40:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.757+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 shutdown_connections 2023-10-14T16:40:32.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.757+0000 7fcc5359e640 1 -- 172.21.15.29:0/1075563561 wait complete. 2023-10-14T16:40:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.758+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 --> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8b300d2290 con 0x7f8b14073d50 2023-10-14T16:40:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.760+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 --> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7144002cb0 con 0x7f7140073090 2023-10-14T16:40:32.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.761+0000 7f8b26ffd640 1 -- 172.21.15.29:0/902552587 <== osd.5 v2:172.21.15.186:6816/1666789598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f8b2c00b980 con 0x7f8b14073d50 2023-10-14T16:40:32.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.762+0000 7f71577fe640 1 -- 172.21.15.29:0/1099939600 <== osd.6 v2:172.21.15.186:6824/3237784830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7144002cb0 con 0x7f7140073090 2023-10-14T16:40:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.762+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f8b14073d50 msgr2=0x7f8b140761f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.762+0000 7f8b387b0640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f8b14073d50 0x7f8b140761f0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8b1406aed0 msgr2=0x7f8b1406d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f8b387b0640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8b1406aed0 0x7f8b1406d390 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8b20036000 tx=0x7f8b2003b040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 msgr2=0x7f8b301601a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f8b387b0640 1 --2- 172.21.15.29:0/902552587 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b300cd260 0x7f8b301601a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8b2800cba0 tx=0x7f8b2800a430 comp rx=0 tx=0).stop 2023-10-14T16:40:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7140073090 msgr2=0x7f71400754b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7140073090 0x7f71400754b0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.762+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 --> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f68b8002cb0 con 0x7f68b40730e0 2023-10-14T16:40:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f8b36d4d640 1 -- 172.21.15.29:0/902552587 reap_dead start 2023-10-14T16:40:32.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.764+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 shutdown_connections 2023-10-14T16:40:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.764+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 >> 172.21.15.29:0/902552587 conn(0x7f8b300cf720 msgr2=0x7f8b300b70e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f714006ae70 msgr2=0x7f714006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f714006ae70 0x7f714006d330 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f715000e190 tx=0x7f715000b040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 msgr2=0x7f71600b15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.763+0000 7f7164de2640 1 --2- 172.21.15.29:0/1099939600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71600addb0 0x7f71600b15a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f714802bf00 tx=0x7f71480046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.764+0000 7f715ed9d640 1 -- 172.21.15.29:0/1099939600 reap_dead start 2023-10-14T16:40:32.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.764+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 shutdown_connections 2023-10-14T16:40:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.764+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 >> 172.21.15.29:0/1099939600 conn(0x7f71600ce9b0 msgr2=0x7f71600ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.765+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 shutdown_connections 2023-10-14T16:40:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.765+0000 7f7164de2640 1 -- 172.21.15.29:0/1099939600 wait complete. 2023-10-14T16:40:32.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.765+0000 7f68c67fc640 1 -- 172.21.15.29:0/1846178435 <== osd.7 v2:172.21.15.186:6808/4274731638 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f68b8002cb0 con 0x7f68b40730e0 2023-10-14T16:40:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.765+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 shutdown_connections 2023-10-14T16:40:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.765+0000 7f8b387b0640 1 -- 172.21.15.29:0/902552587 wait complete. 2023-10-14T16:40:32.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.770+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7f68b40730e0 msgr2=0x7f68b4075500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.877 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.877 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-14T16:40:32.879 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T16:40:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.770+0000 7f68d814e640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7f68b40730e0 0x7f68b4075500 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.771+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f68b406af40 msgr2=0x7f68b406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.771+0000 7f68d814e640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f68b406af40 0x7f68b406d400 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f68cc001e90 tx=0x7f68cc009040 comp rx=0 tx=0).stop 2023-10-14T16:40:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.771+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00cf710 msgr2=0x7f68d01e6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.771+0000 7f68d814e640 1 --2- 172.21.15.29:0/1846178435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f68d00cf710 0x7f68d01e6360 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f68c0033000 tx=0x7f68c00046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.771+0000 7f68d66eb640 1 -- 172.21.15.29:0/1846178435 reap_dead start 2023-10-14T16:40:32.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.772+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 shutdown_connections 2023-10-14T16:40:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.772+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 >> 172.21.15.29:0/1846178435 conn(0x7f68d00d2cc0 msgr2=0x7f68d01600b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.772+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 shutdown_connections 2023-10-14T16:40:32.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:32.772+0000 7f68d814e640 1 -- 172.21.15.29:0/1846178435 wait complete. 2023-10-14T16:40:32.883 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:32.883 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T16:40:32.884 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.885 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T16:40:32.885 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.886 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-14T16:40:32.887 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.887 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-14T16:40:32.888 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:32.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-14T16:40:32.889 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:32.890 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-14T16:40:33.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.225+0000 7ff88446f640 1 Processor -- start 2023-10-14T16:40:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.225+0000 7fde9edb0640 1 Processor -- start 2023-10-14T16:40:33.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.225+0000 7fde9edb0640 1 -- start start 2023-10-14T16:40:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7fde9edb0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde980b4580 0x7fde980b49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7fde9edb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980b60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7fde9edb0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980b4f20 con 0x7fde9815cf10 2023-10-14T16:40:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7fde9edb0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde980d2250 con 0x7fde980b4580 2023-10-14T16:40:33.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7fde9edb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde980d1640 con 0x7fde980b5c30 2023-10-14T16:40:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.225+0000 7ff88446f640 1 -- start start 2023-10-14T16:40:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7ff88446f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7ff88446f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff87c0cd350 0x7ff87c0cd750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7ff88446f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff87c0cc2b0 con 0x7ff87c158eb0 2023-10-14T16:40:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7ff88446f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c15d250 con 0x7ff87c0cd350 2023-10-14T16:40:33.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.227+0000 7ff88446f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff87c0cc590 con 0x7ff87c15c880 2023-10-14T16:40:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.228+0000 7fde9cb4c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fde9815cf10 0x7fde980d0b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:38048/0 (socket says 172.21.15.29:38048) 2023-10-14T16:40:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.228+0000 7fde9cb4c640 1 -- 172.21.15.29:0/2946607413 learned_addr learned my addr 172.21.15.29:0/2946607413 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2946607413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980b60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde97fff640 1 --2- 172.21.15.29:0/2946607413 >> v2:172.21.15.186:3300/0 conn(0x7fde980b4580 0x7fde980b49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.228+0000 7ff881a0a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7ff881a0a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35062/0 (socket says 172.21.15.29:35062) 2023-10-14T16:40:33.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7ff881a0a640 1 -- 172.21.15.29:0/1237879553 learned_addr learned my addr 172.21.15.29:0/1237879553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7ff882a0c640 1 --2- 172.21.15.29:0/1237879553 >> v2:172.21.15.29:3300/0 conn(0x7ff87c0cd350 0x7ff87c0cd750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde9d34d640 1 -- 172.21.15.29:0/2946607413 >> v2:172.21.15.186:3300/0 conn(0x7fde980b4580 msgr2=0x7fde980b49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2946607413 >> v2:172.21.15.186:3300/0 conn(0x7fde980b4580 0x7fde980b49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde9d34d640 1 -- 172.21.15.29:0/2946607413 >> v1:172.21.15.186:6789/0 conn(0x7fde9815cf10 legacy=0x7fde980d0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.229+0000 7fde9d34d640 1 -- 172.21.15.29:0/2946607413 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980d1920 con 0x7fde980b5c30 2023-10-14T16:40:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7ff882a0c640 1 -- 172.21.15.29:0/1237879553 >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 msgr2=0x7ff87c15cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7ff882a0c640 1 --2- 172.21.15.29:0/1237879553 >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7ff881209640 1 -- 172.21.15.29:0/1237879553 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283258717 0 0) 0x7ff87c0cc2b0 con 0x7ff87c158eb0 2023-10-14T16:40:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7ff882a0c640 1 -- 172.21.15.29:0/1237879553 >> v1:172.21.15.29:6789/0 conn(0x7ff87c158eb0 legacy=0x7ff87c06a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2946607413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980b60c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde880097a0 tx=0x7fde8802b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=223fbf8a0571a252 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.231+0000 7ff882a0c640 1 -- 172.21.15.29:0/1237879553 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff87c0cc870 con 0x7ff87c0cd350 2023-10-14T16:40:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.232+0000 7ff881a0a640 1 --2- 172.21.15.29:0/1237879553 >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.234+0000 7ff882a0c640 1 --2- 172.21.15.29:0/1237879553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c0cd350 0x7ff87c0cd750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0c5500 tx=0x7ff87402c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb14b03f8a30be1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.232+0000 7fde977fe640 1 -- 172.21.15.29:0/2946607413 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88030710 con 0x7fde980b5c30 2023-10-14T16:40:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.232+0000 7fde977fe640 1 -- 172.21.15.29:0/2946607413 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8802e050 con 0x7fde980b5c30 2023-10-14T16:40:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.232+0000 7fde977fe640 1 -- 172.21.15.29:0/2946607413 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88030710 con 0x7fde980b5c30 2023-10-14T16:40:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 -- 172.21.15.29:0/2946607413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 msgr2=0x7fde980b60c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2946607413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980b60c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde880097a0 tx=0x7fde8802b730 comp rx=0 tx=0).stop 2023-10-14T16:40:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 -- 172.21.15.29:0/2946607413 shutdown_connections 2023-10-14T16:40:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2946607413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980b60c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2946607413 >> v2:172.21.15.186:3300/0 conn(0x7fde980b4580 0x7fde980b49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.233+0000 7fde9edb0640 1 -- 172.21.15.29:0/2946607413 >> 172.21.15.29:0/2946607413 conn(0x7fde980cf580 msgr2=0x7fde980cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.234+0000 7fde9edb0640 1 -- 172.21.15.29:0/2946607413 shutdown_connections 2023-10-14T16:40:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.234+0000 7fde9edb0640 1 -- 172.21.15.29:0/2946607413 wait complete. 2023-10-14T16:40:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.235+0000 7ff881209640 1 -- 172.21.15.29:0/1237879553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87402ccc0 con 0x7ff87c0cd350 2023-10-14T16:40:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.235+0000 7ff881209640 1 -- 172.21.15.29:0/1237879553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87402ce20 con 0x7ff87c0cd350 2023-10-14T16:40:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.235+0000 7ff881209640 1 -- 172.21.15.29:0/1237879553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87402ccc0 con 0x7ff87c0cd350 2023-10-14T16:40:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.237+0000 7ff88446f640 1 -- 172.21.15.29:0/1237879553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c0cd350 msgr2=0x7ff87c0cd750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.237+0000 7ff88446f640 1 --2- 172.21.15.29:0/1237879553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c0cd350 0x7ff87c0cd750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff87c0c5500 tx=0x7ff87402c120 comp rx=0 tx=0).stop 2023-10-14T16:40:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.236+0000 7fde9edb0640 1 Processor -- start 2023-10-14T16:40:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.237+0000 7fde9edb0640 1 -- start start 2023-10-14T16:40:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.238+0000 7ff88446f640 1 -- 172.21.15.29:0/1237879553 shutdown_connections 2023-10-14T16:40:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.238+0000 7ff88446f640 1 --2- 172.21.15.29:0/1237879553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c0cd350 0x7ff87c0cd750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.238+0000 7ff88446f640 1 --2- 172.21.15.29:0/1237879553 >> v2:172.21.15.186:3300/0 conn(0x7ff87c15c880 0x7ff87c15cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.238+0000 7ff88446f640 1 -- 172.21.15.29:0/1237879553 >> 172.21.15.29:0/1237879553 conn(0x7ff87c0ce6e0 msgr2=0x7ff87c0ceaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.239+0000 7ff88446f640 1 -- 172.21.15.29:0/1237879553 shutdown_connections 2023-10-14T16:40:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.239+0000 7ff88446f640 1 -- 172.21.15.29:0/1237879553 wait complete. 2023-10-14T16:40:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.239+0000 7fde9edb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 0x7fde98150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.239+0000 7fde9edb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.239+0000 7fde9edb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 0x7fde9814ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde9edb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde98159360 con 0x7fde980b4580 2023-10-14T16:40:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde9edb0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde981f6690 con 0x7fde9815cf10 2023-10-14T16:40:33.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde9edb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde981f6970 con 0x7fde980b5c30 2023-10-14T16:40:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde97fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde9d34d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 0x7fde9814ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde97fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47870/0 (socket says 172.21.15.29:47870) 2023-10-14T16:40:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.240+0000 7fde97fff640 1 -- 172.21.15.29:0/2514548091 learned_addr learned my addr 172.21.15.29:0/2514548091 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9cb4c640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 0x7fde98150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9d34d640 1 -- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 msgr2=0x7fde980cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9d34d640 1 -- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 msgr2=0x7fde98150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 0x7fde98150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9d34d640 1 -- 172.21.15.29:0/2514548091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde88009400 con 0x7fde9815cf10 2023-10-14T16:40:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde97fff640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.241+0000 7fde9cb4c640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 0x7fde98150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.242+0000 7fde9d34d640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 0x7fde9814ef80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde88032000 tx=0x7fde880046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.242+0000 7ff88446f640 1 Processor -- start 2023-10-14T16:40:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.242+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88030710 con 0x7fde9815cf10 2023-10-14T16:40:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.242+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde88030870 con 0x7fde9815cf10 2023-10-14T16:40:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.243+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde88030c20 con 0x7fde9815cf10 2023-10-14T16:40:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.243+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde981f6c50 con 0x7fde9815cf10 2023-10-14T16:40:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.243+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde981506d0 con 0x7fde9815cf10 2023-10-14T16:40:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.243+0000 7ff88446f640 1 -- start start 2023-10-14T16:40:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff87c0cd350 0x7ff87c1557a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 0x7ff87c0c4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 0x7ff87c0c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c8bf0 con 0x7ff87c158eb0 2023-10-14T16:40:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c4950 con 0x7ff87c15c880 2023-10-14T16:40:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.245+0000 7ff88446f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff87c0c5a90 con 0x7ff87c0cd350 2023-10-14T16:40:33.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde981594d0 con 0x7fde9815cf10 2023-10-14T16:40:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7ff882a0c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 0x7ff87c0c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7ff882a0c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 0x7ff87c0c4170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35088/0 (socket says 172.21.15.29:35088) 2023-10-14T16:40:33.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7ff882a0c640 1 -- 172.21.15.29:0/2231655861 learned_addr learned my addr 172.21.15.29:0/2231655861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7ff881a0a640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 0x7ff87c0c4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde88030d80 con 0x7fde9815cf10 2023-10-14T16:40:33.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7fde957fa640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 0x7fde8406d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fde880ab520 con 0x7fde9815cf10 2023-10-14T16:40:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7fde9cb4c640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 0x7fde8406d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.250+0000 7fde9cb4c640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 0x7fde8406d400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000f60 tx=0x7fde8c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.247+0000 7ff88220b640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff87c0cd350 0x7ff87c1557a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7ff881a0a640 1 -- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff87c0cd350 msgr2=0x7ff87c1557a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7ff881a0a640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff87c0cd350 0x7ff87c1557a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7ff881a0a640 1 -- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 msgr2=0x7ff87c0c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7ff881a0a640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 0x7ff87c0c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.248+0000 7ff881a0a640 1 -- 172.21.15.29:0/2231655861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff878008550 con 0x7ff87c158eb0 2023-10-14T16:40:33.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7ff881a0a640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 0x7ff87c0c4cb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff878006ca0 tx=0x7ff87800e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878010040 con 0x7ff87c158eb0 2023-10-14T16:40:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87800ed00 con 0x7ff87c158eb0 2023-10-14T16:40:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878004230 con 0x7ff87c158eb0 2023-10-14T16:40:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.249+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff874009a70 con 0x7ff87c158eb0 2023-10-14T16:40:33.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.250+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff87c0c5d30 con 0x7ff87c158eb0 2023-10-14T16:40:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.252+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff864005250 con 0x7ff87c158eb0 2023-10-14T16:40:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.252+0000 7fed85a00640 1 Processor -- start 2023-10-14T16:40:33.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.253+0000 7f802487c640 1 Processor -- start 2023-10-14T16:40:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.253+0000 7fed85a00640 1 -- start start 2023-10-14T16:40:33.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff878004390 con 0x7ff87c158eb0 2023-10-14T16:40:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.253+0000 7f802487c640 1 -- start start 2023-10-14T16:40:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7f802487c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f802006a4b0 0x7f80200ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7f802487c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80200ce1c0 0x7f80200cc660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7f802487c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f802006cf50 con 0x7f8020158f60 2023-10-14T16:40:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7f802487c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802015d970 con 0x7f80200ce1c0 2023-10-14T16:40:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.255+0000 7f802487c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f802015ccf0 con 0x7f802006a4b0 2023-10-14T16:40:33.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7fed85a00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed80161800 0x7fed800c96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7fed85a00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed80070e80 0x7fed800ca080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7f801e59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8020158f60 0x7f8020069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:38052/0 (socket says 172.21.15.29:38052) 2023-10-14T16:40:33.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7f801e59c640 1 -- 172.21.15.29:0/2489194209 learned_addr learned my addr 172.21.15.29:0/2489194209 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7fed85a00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed800c9cc0 con 0x7fed800712c0 2023-10-14T16:40:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7fed85a00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed800cc380 con 0x7fed80161800 2023-10-14T16:40:33.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7fed7e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed80070e80 0x7fed800ca080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7fed85a00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed80161be0 con 0x7fed80070e80 2023-10-14T16:40:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7fed7effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed80161800 0x7fed800c96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7f1b9f061640 1 Processor -- start 2023-10-14T16:40:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7f1b9f061640 1 -- start start 2023-10-14T16:40:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7f801dd9b640 1 --2- 172.21.15.29:0/2489194209 >> v2:172.21.15.186:3300/0 conn(0x7f802006a4b0 0x7f80200ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7fed7effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed80161800 0x7fed800c96e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36540/0 (socket says 172.21.15.29:36540) 2023-10-14T16:40:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f801ed9d640 1 --2- 172.21.15.29:0/2489194209 >> v2:172.21.15.29:3300/0 conn(0x7f80200ce1c0 0x7f80200cc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7fed7effd640 1 -- 172.21.15.29:0/2576644771 learned_addr learned my addr 172.21.15.29:0/2576644771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.256+0000 7ff872ffd640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 0x7ff86007e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.257+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff878089cf0 con 0x7ff87c158eb0 2023-10-14T16:40:33.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b98159110 0x7f1b9815cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 0x7f1b98069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 0x7f1b9806b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b9806a770 con 0x7f1b98159110 2023-10-14T16:40:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b981594f0 con 0x7f1b980ade50 2023-10-14T16:40:33.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9f061640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9815d130 con 0x7f1b9806a2e0 2023-10-14T16:40:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7ff88220b640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 0x7ff86007e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b8ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 0x7f1b98069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b8ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 0x7f1b98069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35106/0 (socket says 172.21.15.29:35106) 2023-10-14T16:40:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b8ffff640 1 -- 172.21.15.29:0/188031511 learned_addr learned my addr 172.21.15.29:0/188031511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.258+0000 7f1b9d5fe640 1 --2- 172.21.15.29:0/188031511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 0x7f1b9806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7fed7effd640 1 -- 172.21.15.29:0/2576644771 >> v2:172.21.15.186:3300/0 conn(0x7fed80070e80 msgr2=0x7fed800ca080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7fed7effd640 1 --2- 172.21.15.29:0/2576644771 >> v2:172.21.15.186:3300/0 conn(0x7fed80070e80 0x7fed800ca080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7fed7effd640 1 -- 172.21.15.29:0/2576644771 >> v1:172.21.15.29:6789/0 conn(0x7fed800712c0 legacy=0x7fed801652e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7fed7effd640 1 -- 172.21.15.29:0/2576644771 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed8006f340 con 0x7fed80161800 2023-10-14T16:40:33.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7ff88220b640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 0x7ff86007e340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff87c1563f0 tx=0x7ff86c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7f1b9cdfd640 1 --2- 172.21.15.29:0/188031511 >> v2:172.21.15.29:3300/0 conn(0x7f1b98159110 0x7f1b9815cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7f1b9d5fe640 1 -- 172.21.15.29:0/188031511 >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 msgr2=0x7f1b98069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7f1b9d5fe640 1 --2- 172.21.15.29:0/188031511 >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 0x7f1b98069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7f1b9d5fe640 1 -- 172.21.15.29:0/188031511 >> v2:172.21.15.29:3300/0 conn(0x7f1b98159110 msgr2=0x7f1b9815cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7f1b9d5fe640 1 --2- 172.21.15.29:0/188031511 >> v2:172.21.15.29:3300/0 conn(0x7f1b98159110 0x7f1b9815cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.260+0000 7f1b9d5fe640 1 -- 172.21.15.29:0/188031511 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9815d400 con 0x7f1b9806a2e0 2023-10-14T16:40:33.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7f801dd9b640 1 -- 172.21.15.29:0/2489194209 >> v2:172.21.15.29:3300/0 conn(0x7f80200ce1c0 msgr2=0x7f80200cc660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7f801dd9b640 1 --2- 172.21.15.29:0/2489194209 >> v2:172.21.15.29:3300/0 conn(0x7f80200ce1c0 0x7f80200cc660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7f801dd9b640 1 -- 172.21.15.29:0/2489194209 >> v1:172.21.15.186:6789/0 conn(0x7f8020158f60 legacy=0x7f8020069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.259+0000 7f801dd9b640 1 -- 172.21.15.29:0/2489194209 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f802015cfd0 con 0x7f802006a4b0 2023-10-14T16:40:33.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.261+0000 7fed7effd640 1 --2- 172.21.15.29:0/2576644771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c96e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fed80073df0 tx=0x7fed7002bb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecd8bb4ede8356eb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.261+0000 7f801dd9b640 1 --2- 172.21.15.29:0/2489194209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200ade50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8020066a10 tx=0x7f800802b7e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65cdaa53ab9da74e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.261+0000 7f1b9d5fe640 1 --2- 172.21.15.29:0/188031511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 0x7f1b9806b030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1b98066a50 tx=0x7f1b90030220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9ddf6e0cf814be2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7f1b8f7fe640 1 -- 172.21.15.29:0/188031511 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000d0d0 con 0x7f1b9806a2e0 2023-10-14T16:40:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7f0ef0a61640 1 Processor -- start 2023-10-14T16:40:33.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7f1b8f7fe640 1 -- 172.21.15.29:0/188031511 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9000d230 con 0x7f1b9806a2e0 2023-10-14T16:40:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7f1b8f7fe640 1 -- 172.21.15.29:0/188031511 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b9000d5c0 con 0x7f1b9806a2e0 2023-10-14T16:40:33.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7fed7dffb640 1 -- 172.21.15.29:0/2576644771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7002c710 con 0x7fed80161800 2023-10-14T16:40:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7fed7dffb640 1 -- 172.21.15.29:0/2576644771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7002c870 con 0x7fed80161800 2023-10-14T16:40:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7fed7dffb640 1 -- 172.21.15.29:0/2576644771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7002cc00 con 0x7fed80161800 2023-10-14T16:40:33.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.262+0000 7f0ef0a61640 1 -- start start 2023-10-14T16:40:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 -- 172.21.15.29:0/188031511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 msgr2=0x7f1b9806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff878054e30 con 0x7ff87c158eb0 2023-10-14T16:40:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 --2- 172.21.15.29:0/188031511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 0x7f1b9806b030 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f1b98066a50 tx=0x7f1b90030220 comp rx=0 tx=0).stop 2023-10-14T16:40:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 -- 172.21.15.29:0/188031511 shutdown_connections 2023-10-14T16:40:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 --2- 172.21.15.29:0/188031511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b9806a2e0 0x7f1b9806b030 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 --2- 172.21.15.29:0/188031511 >> v2:172.21.15.186:3300/0 conn(0x7f1b980ade50 0x7f1b98069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 --2- 172.21.15.29:0/188031511 >> v2:172.21.15.29:3300/0 conn(0x7f1b98159110 0x7f1b9815cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7f1b9f061640 1 -- 172.21.15.29:0/188031511 >> 172.21.15.29:0/188031511 conn(0x7f1b980ce9b0 msgr2=0x7f1b980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 -- 172.21.15.29:0/2576644771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 msgr2=0x7fed800c96e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 --2- 172.21.15.29:0/2576644771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c96e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fed80073df0 tx=0x7fed7002bb70 comp rx=0 tx=0).stop 2023-10-14T16:40:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 -- 172.21.15.29:0/2576644771 shutdown_connections 2023-10-14T16:40:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 --2- 172.21.15.29:0/2576644771 >> v2:172.21.15.186:3300/0 conn(0x7fed80070e80 0x7fed800ca080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 --2- 172.21.15.29:0/2576644771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c96e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.263+0000 7fed85a00640 1 -- 172.21.15.29:0/2576644771 >> 172.21.15.29:0/2576644771 conn(0x7fed800ce680 msgr2=0x7fed800cea90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.264+0000 7fed85a00640 1 -- 172.21.15.29:0/2576644771 shutdown_connections 2023-10-14T16:40:33.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.264+0000 7fed85a00640 1 -- 172.21.15.29:0/2576644771 wait complete. 2023-10-14T16:40:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.264+0000 7f1b9f061640 1 -- 172.21.15.29:0/188031511 shutdown_connections 2023-10-14T16:40:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.264+0000 7f1b9f061640 1 -- 172.21.15.29:0/188031511 wait complete. 2023-10-14T16:40:33.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.264+0000 7f0ef0a61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0eec0addb0 0x7f0eec0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f1b9f061640 1 Processor -- start 2023-10-14T16:40:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f1b9f061640 1 -- start start 2023-10-14T16:40:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0ef0a61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec0cb140 con 0x7f0eec0c9000 2023-10-14T16:40:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f801d59a640 1 -- 172.21.15.29:0/2489194209 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008031740 con 0x7f802006a4b0 2023-10-14T16:40:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f801d59a640 1 -- 172.21.15.29:0/2489194209 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80080318a0 con 0x7f802006a4b0 2023-10-14T16:40:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f801d59a640 1 -- 172.21.15.29:0/2489194209 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8008031c70 con 0x7f802006a4b0 2023-10-14T16:40:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0ef0a61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0eec06b030 con 0x7f0eec06a4b0 2023-10-14T16:40:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0ef0a61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0eec161a50 con 0x7f0eec0addb0 2023-10-14T16:40:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0ee9d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0eec0c9000 0x7f0eec069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47856/0 (socket says 172.21.15.29:47856) 2023-10-14T16:40:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0ee9d9b640 1 -- 172.21.15.29:0/2024576947 learned_addr learned my addr 172.21.15.29:0/2024576947 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.265+0000 7f0eea59c640 1 --2- 172.21.15.29:0/2024576947 >> v2:172.21.15.29:3300/0 conn(0x7f0eec0addb0 0x7f0eec0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829255241 0 0) 0x7f0eec0cb140 con 0x7f0eec0c9000 2023-10-14T16:40:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ed4003520 con 0x7f0eec0c9000 2023-10-14T16:40:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446167888 0 0) 0x7f0ed4003520 con 0x7f0eec0c9000 2023-10-14T16:40:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 >> v2:172.21.15.29:3300/0 conn(0x7f0eec0addb0 msgr2=0x7f0eec0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 --2- 172.21.15.29:0/2024576947 >> v2:172.21.15.29:3300/0 conn(0x7f0eec0addb0 0x7f0eec0caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 >> v1:172.21.15.186:6789/0 conn(0x7f0eec06a4b0 legacy=0x7f0eec1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec06b310 con 0x7f0eec0c9000 2023-10-14T16:40:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0ee0001670 con 0x7f0eec0c9000 2023-10-14T16:40:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 >> v1:172.21.15.29:6789/0 conn(0x7f0eec0c9000 legacy=0x7f0eec069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed4003970 0x7f0ed4003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 0x7f0ed4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40052a0 0x7f0ed400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec161a50 con 0x7f0ed40052a0 2023-10-14T16:40:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed400c1a0 con 0x7f0ed4003970 2023-10-14T16:40:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ed400c360 con 0x7f0ed40047c0 2023-10-14T16:40:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7fae97f97640 1 Processor -- start 2023-10-14T16:40:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 -- 172.21.15.29:0/2489194209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 msgr2=0x7f80200ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 --2- 172.21.15.29:0/2489194209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200ade50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8020066a10 tx=0x7f800802b7e0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7fae97f97640 1 -- start start 2023-10-14T16:40:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.266+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fde880765b0 con 0x7fde9815cf10 2023-10-14T16:40:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 -- 172.21.15.29:0/2489194209 shutdown_connections 2023-10-14T16:40:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 --2- 172.21.15.29:0/2489194209 >> v2:172.21.15.29:3300/0 conn(0x7f80200ce1c0 0x7f80200cc660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 --2- 172.21.15.29:0/2489194209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200ade50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 -- 172.21.15.29:0/2489194209 >> 172.21.15.29:0/2489194209 conn(0x7f80200ce9b0 msgr2=0x7f80200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 -- 172.21.15.29:0/2489194209 shutdown_connections 2023-10-14T16:40:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f0eea59c640 1 --2- 172.21.15.29:0/2024576947 >> v2:172.21.15.29:3300/0 conn(0x7f0eec0addb0 0x7f0eec0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7fed85a00640 1 Processor -- start 2023-10-14T16:40:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f802487c640 1 -- 172.21.15.29:0/2489194209 wait complete. 2023-10-14T16:40:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f0ee9d9b640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 0x7f0ed4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7fed85a00640 1 -- start start 2023-10-14T16:40:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 0x7f1b9814a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b98159110 0x7f1b9814adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b9814b2f0 con 0x7f1b9806a2e0 2023-10-14T16:40:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b980c9960 con 0x7f1b980ade50 2023-10-14T16:40:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f1b9f061640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b98157490 con 0x7f1b98159110 2023-10-14T16:40:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.267+0000 7f0eea59c640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed4003970 0x7f0ed4003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f1b9cdfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f1b8ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 0x7f1b9814a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f1b9cdfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36586/0 (socket says 172.21.15.29:36586) 2023-10-14T16:40:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f1b9cdfd640 1 -- 172.21.15.29:0/4009353642 learned_addr learned my addr 172.21.15.29:0/4009353642 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0ee9d9b640 1 -- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed4003970 msgr2=0x7f0ed4003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0ee9d9b640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0ed4003970 0x7f0ed4003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0ee9d9b640 1 -- 172.21.15.29:0/2024576947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40052a0 msgr2=0x7f0ed400b980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-10-14T16:40:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0eead9d640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40052a0 0x7f0ed400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0ee9d9b640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40052a0 0x7f0ed400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.268+0000 7f0ee9d9b640 1 -- 172.21.15.29:0/2024576947 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec06b310 con 0x7f0ed40047c0 2023-10-14T16:40:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7f0eead9d640 1 -- 172.21.15.29:0/2024576947 reap_dead start 2023-10-14T16:40:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7f1b9d5fe640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b98159110 0x7f1b9814adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7f1b8ffff640 1 -- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b98159110 msgr2=0x7f1b9814adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7f1b8ffff640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b98159110 0x7f1b9814adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7f1b8ffff640 1 -- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 msgr2=0x7f1b980af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7fae97f97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae90159ab0 0x7fae900ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7fae97f97640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.269+0000 7fae97f97640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae90070360 con 0x7fae90159e90 2023-10-14T16:40:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7fae97f97640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9006cb10 con 0x7fae90159ab0 2023-10-14T16:40:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7fae97f97640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015d890 con 0x7fae9006c140 2023-10-14T16:40:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7f802487c640 1 Processor -- start 2023-10-14T16:40:33.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7fae96534640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7fae96534640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47914/0 (socket says 172.21.15.29:47914) 2023-10-14T16:40:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7f0ee9d9b640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 0x7f0ed4004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c260 tx=0x7f0ed40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2569636fbe597667 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 0x7fed800c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.270+0000 7fae96534640 1 -- 172.21.15.29:0/1600539739 learned_addr learned my addr 172.21.15.29:0/1600539739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 0x7fed800c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fae96534640 1 -- 172.21.15.29:0/1600539739 >> v2:172.21.15.29:3300/0 conn(0x7fae90159ab0 msgr2=0x7fae900ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fae96534640 1 --2- 172.21.15.29:0/1600539739 >> v2:172.21.15.29:3300/0 conn(0x7fae90159ab0 0x7fae900ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed800c7080 con 0x7fed80161800 2023-10-14T16:40:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed800c7360 con 0x7fed80070e80 2023-10-14T16:40:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fae96534640 1 -- 172.21.15.29:0/1600539739 >> v1:172.21.15.29:6789/0 conn(0x7fae90159e90 legacy=0x7fae9006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fae96534640 1 -- 172.21.15.29:0/1600539739 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae900cd1f0 con 0x7fae9006c140 2023-10-14T16:40:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed85a00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed800c7640 con 0x7fed800712c0 2023-10-14T16:40:33.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed7effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 0x7fed800c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed7effd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 0x7fed800c8120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35128/0 (socket says 172.21.15.29:35128) 2023-10-14T16:40:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed7e7fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 0x7fed800c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7fed7effd640 1 -- 172.21.15.29:0/84874520 learned_addr learned my addr 172.21.15.29:0/84874520 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee002cb00 con 0x7f0ed40047c0 2023-10-14T16:40:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7fed7f7fe640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.271+0000 7f802487c640 1 -- start start 2023-10-14T16:40:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200c70c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 0x7f80200c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802014ebf0 con 0x7f8020158f60 2023-10-14T16:40:33.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f802014eed0 con 0x7f802006a4b0 2023-10-14T16:40:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f802487c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802014f1b0 con 0x7f80200ce1c0 2023-10-14T16:40:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f801ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f801ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36602/0 (socket says 172.21.15.29:36602) 2023-10-14T16:40:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f801ed9d640 1 -- 172.21.15.29:0/2213282357 learned_addr learned my addr 172.21.15.29:0/2213282357 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f1b8ffff640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f1b8ffff640 1 -- 172.21.15.29:0/4009353642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9000f050 con 0x7f1b980ade50 2023-10-14T16:40:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f1b9cdfd640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b8ffff640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 0x7f1b9814a870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b710 tx=0x7f1b880091c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee002cc60 con 0x7f0ed40047c0 2023-10-14T16:40:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.272+0000 7f0ee959a640 1 -- 172.21.15.29:0/2024576947 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee0005b10 con 0x7f0ed40047c0 2023-10-14T16:40:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f0ef0a61640 1 -- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 msgr2=0x7f0ed4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 0x7f0ed4004c40 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0ee000c260 tx=0x7f0ed40057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f801e59c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200c70c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f801dd9b640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 0x7f80200c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b88010040 con 0x7f1b980ade50 2023-10-14T16:40:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b88009dd0 con 0x7f1b980ade50 2023-10-14T16:40:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b880165f0 con 0x7f1b980ade50 2023-10-14T16:40:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b980c8d90 con 0x7f1b980ade50 2023-10-14T16:40:33.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.273+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b980c9310 con 0x7f1b980ade50 2023-10-14T16:40:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.274+0000 7fed7e7fc640 1 -- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 msgr2=0x7fed800c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.274+0000 7fed7e7fc640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 0x7fed800c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.274+0000 7fed7e7fc640 1 -- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 msgr2=0x7fed800c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.274+0000 7fed7e7fc640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.274+0000 7fed7e7fc640 1 -- 172.21.15.29:0/84874520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed700094d0 con 0x7fed800712c0 2023-10-14T16:40:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f801ed9d640 1 -- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 msgr2=0x7f80200c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f0ef0a61640 1 -- 172.21.15.29:0/2024576947 shutdown_connections 2023-10-14T16:40:33.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/2024576947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0ed40047c0 0x7f0ed4004c40 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f0ef0a61640 1 -- 172.21.15.29:0/2024576947 >> 172.21.15.29:0/2024576947 conn(0x7f0eec0ce9b0 msgr2=0x7f0eec0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f801ed9d640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 0x7f80200c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7fae96534640 1 --2- 172.21.15.29:0/1600539739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00bb10 tx=0x7fae8c02dfa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=36746e44d215ea1a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b980b1580 con 0x7f1b980ade50 2023-10-14T16:40:33.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7fed7e7fc640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 0x7fed800c2fb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed7400d5f0 tx=0x7fed7400dab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f0ef0a61640 1 -- 172.21.15.29:0/2024576947 shutdown_connections 2023-10-14T16:40:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f801ed9d640 1 -- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 msgr2=0x7f80200c70c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f801ed9d640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200c70c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed740125f0 con 0x7fed800712c0 2023-10-14T16:40:33.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7fae94d31640 1 -- 172.21.15.29:0/1600539739 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c02ed50 con 0x7fae9006c140 2023-10-14T16:40:33.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.276+0000 7fae94d31640 1 -- 172.21.15.29:0/1600539739 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c036440 con 0x7fae9006c140 2023-10-14T16:40:33.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.276+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed800c7920 con 0x7fed800712c0 2023-10-14T16:40:33.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.276+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed800c34f0 con 0x7fed800712c0 2023-10-14T16:40:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.276+0000 7f801ed9d640 1 -- 172.21.15.29:0/2213282357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8008009400 con 0x7f8020158f60 2023-10-14T16:40:33.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.276+0000 7f801dd9b640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 0x7f80200c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 -- 172.21.15.29:0/1600539739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 msgr2=0x7fae9006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae94d31640 1 -- 172.21.15.29:0/1600539739 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c0367d0 con 0x7fae9006c140 2023-10-14T16:40:33.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 --2- 172.21.15.29:0/1600539739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fae8c00bb10 tx=0x7fae8c02dfa0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 -- 172.21.15.29:0/1600539739 shutdown_connections 2023-10-14T16:40:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 --2- 172.21.15.29:0/1600539739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9006c140 0x7fae9006c5d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 --2- 172.21.15.29:0/1600539739 >> v2:172.21.15.29:3300/0 conn(0x7fae90159ab0 0x7fae900ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 -- 172.21.15.29:0/1600539739 >> 172.21.15.29:0/1600539739 conn(0x7fae900ce9b0 msgr2=0x7fae900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7f801e59c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 -- 172.21.15.29:0/1600539739 shutdown_connections 2023-10-14T16:40:33.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fae97f97640 1 -- 172.21.15.29:0/1600539739 wait complete. 2023-10-14T16:40:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.277+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8004e9b0 con 0x7fed800712c0 2023-10-14T16:40:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.278+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed74012750 con 0x7fed800712c0 2023-10-14T16:40:33.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.278+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74012b00 con 0x7fed800712c0 2023-10-14T16:40:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.278+0000 7f801ed9d640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8010007e40 tx=0x7f801000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.275+0000 7f0ef0a61640 1 -- 172.21.15.29:0/2024576947 wait complete. 2023-10-14T16:40:33.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f0ef0a61640 1 Processor -- start 2023-10-14T16:40:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8010014620 con 0x7f8020158f60 2023-10-14T16:40:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f801000ee90 con 0x7f8020158f60 2023-10-14T16:40:33.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80100149e0 con 0x7f8020158f60 2023-10-14T16:40:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.280+0000 7fae97f97640 1 Processor -- start 2023-10-14T16:40:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f802014f490 con 0x7f8020158f60 2023-10-14T16:40:33.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.279+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f802014de50 con 0x7f8020158f60 2023-10-14T16:40:33.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.280+0000 7fae97f97640 1 -- start start 2023-10-14T16:40:33.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.281+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1b88016750 con 0x7f1b980ade50 2023-10-14T16:40:33.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.281+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80201363e0 con 0x7f8020158f60 2023-10-14T16:40:33.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.282+0000 7fae97f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.282+0000 7f0ef0a61640 1 -- start start 2023-10-14T16:40:33.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.282+0000 7f1b8d7fa640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 0x7f1b8406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.282+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1b8808a5e0 con 0x7f1b980ade50 2023-10-14T16:40:33.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f1b9cdfd640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 0x7f1b8406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fd701165640 1 Processor -- start 2023-10-14T16:40:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fae97f97640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 0x7fae900af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40047c0 0x7f0eec0a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0eec1f4a90 0x7f0eec0b2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fae95d33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fd701165640 1 -- start start 2023-10-14T16:40:33.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fae95d33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36610/0 (socket says 172.21.15.29:36610) 2023-10-14T16:40:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7fae97f97640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0b3920 con 0x7f0ed40047c0 2023-10-14T16:40:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0b2d10 con 0x7f0eec1f4f10 2023-10-14T16:40:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.283+0000 7f0ef0a61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0eec0b2ff0 con 0x7f0eec1f4a90 2023-10-14T16:40:33.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7f1b9cdfd640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 0x7f1b8406d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b94006fd0 tx=0x7f1b94008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7fae95532640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 0x7fae900af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7f0eead9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7f0eead9d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35160/0 (socket says 172.21.15.29:35160) 2023-10-14T16:40:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7f0eead9d640 1 -- 172.21.15.29:0/26826115 learned_addr learned my addr 172.21.15.29:0/26826115 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.282+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fed74012c60 con 0x7fed800712c0 2023-10-14T16:40:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7fed67fff640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 0x7fed6c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fed7408a590 con 0x7fed800712c0 2023-10-14T16:40:33.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.284+0000 7f0ee9d9b640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0eec1f4a90 0x7f0eec0b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fed7effd640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 0x7fed6c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7f0eead9d640 1 -- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0eec1f4a90 msgr2=0x7f0eec0b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7f0eead9d640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0eec1f4a90 0x7f0eec0b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7f0eead9d640 1 -- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40047c0 msgr2=0x7f0eec0a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7f0eead9d640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40047c0 0x7f0eec0a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7f0eead9d640 1 -- 172.21.15.29:0/26826115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0eec0b32d0 con 0x7f0eec1f4f10 2023-10-14T16:40:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae96534640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae95532640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 0x7fae900af1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47940/0 (socket says 172.21.15.29:47940) 2023-10-14T16:40:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fd701165640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6fc0cf710 0x7fd6fc0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fd701165640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 0x7fd6fc0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fd701165640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6fc0d3980 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fd701165640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0d3c60 con 0x7fd6fc0cf710 2023-10-14T16:40:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fd701165640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6fc0ce630 con 0x7fd6fc0b6cd0 2023-10-14T16:40:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae96534640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35168/0 (socket says 172.21.15.29:35168) 2023-10-14T16:40:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae97f97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae900c6170 con 0x7fae9006c140 2023-10-14T16:40:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae97f97640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015c650 con 0x7fae90159e90 2023-10-14T16:40:33.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae97f97640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9015c8b0 con 0x7fae90159ab0 2023-10-14T16:40:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fd6fb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 0x7fd6fc0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fd6fad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6fc0cf710 0x7fd6fc0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fd6fb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 0x7fd6fc0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35172/0 (socket says 172.21.15.29:35172) 2023-10-14T16:40:33.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fd6fb59e640 1 -- 172.21.15.29:0/3524352739 learned_addr learned my addr 172.21.15.29:0/3524352739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7f0eead9d640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0eec06af00 tx=0x7f0ee400d890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.285+0000 7fae95d33640 1 -- 172.21.15.29:0/3949157055 learned_addr learned my addr 172.21.15.29:0/3949157055 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae95d33640 1 -- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 msgr2=0x7fae900af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae95d33640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 0x7fae900af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae95d33640 1 -- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 msgr2=0x7fae9015c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae95d33640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae95d33640 1 -- 172.21.15.29:0/3949157055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c00b770 con 0x7fae9006c140 2023-10-14T16:40:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.286+0000 7fae96534640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2023-10-14T16:40:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.287+0000 7fae95d33640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae8800ba50 tx=0x7fae8800bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.287+0000 7fd6fad9d640 1 -- 172.21.15.29:0/3524352739 >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 msgr2=0x7fd6fc0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.287+0000 7fd6fad9d640 1 --2- 172.21.15.29:0/3524352739 >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 0x7fd6fc0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.287+0000 7fd6fad9d640 1 -- 172.21.15.29:0/3524352739 >> v1:172.21.15.186:6789/0 conn(0x7fd6fc0cfaf0 legacy=0x7fd6fc0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.287+0000 7fd6fad9d640 1 -- 172.21.15.29:0/3524352739 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6fc0ce910 con 0x7fd6fc0cf710 2023-10-14T16:40:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.289+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8010014b40 con 0x7f8020158f60 2023-10-14T16:40:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.289+0000 7fed7effd640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 0x7fed6c06d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed7002bbb0 tx=0x7fed70030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.289+0000 7fd6fad9d640 1 --2- 172.21.15.29:0/3524352739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0cf710 0x7fd6fc0b2d10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0b5890 tx=0x7fd6f002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3077a2b5bad89dfa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.290+0000 7f8016ffd640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 0x7f800006d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.290+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f801008a450 con 0x7f8020158f60 2023-10-14T16:40:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee40128c0 con 0x7f0eec1f4f10 2023-10-14T16:40:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee4012a40 con 0x7f0eec1f4f10 2023-10-14T16:40:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee4012df0 con 0x7f0eec1f4f10 2023-10-14T16:40:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0eec0b35b0 con 0x7f0eec1f4f10 2023-10-14T16:40:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f801e59c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 0x7f800006d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0eec0c5010 con 0x7f0eec1f4f10 2023-10-14T16:40:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8010055570 con 0x7f8020158f60 2023-10-14T16:40:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.291+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b88055720 con 0x7f1b980ade50 2023-10-14T16:40:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8800cbe0 con 0x7fae9006c140 2023-10-14T16:40:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8800cd60 con 0x7fae9006c140 2023-10-14T16:40:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae88014680 con 0x7fae9006c140 2023-10-14T16:40:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0eec04e160 con 0x7f0eec1f4f10 2023-10-14T16:40:33.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fd6f9d9b640 1 -- 172.21.15.29:0/3524352739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f002e070 con 0x7fd6fc0cf710 2023-10-14T16:40:33.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fd6f9d9b640 1 -- 172.21.15.29:0/3524352739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f00315a0 con 0x7fd6fc0cf710 2023-10-14T16:40:33.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9015cb90 con 0x7fae9006c140 2023-10-14T16:40:33.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae900c5e00 con 0x7fae9006c140 2023-10-14T16:40:33.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.292+0000 7fd6f9d9b640 1 -- 172.21.15.29:0/3524352739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6f0031930 con 0x7fd6fc0cf710 2023-10-14T16:40:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 -- 172.21.15.29:0/3524352739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0cf710 msgr2=0x7fd6fc0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 --2- 172.21.15.29:0/3524352739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0cf710 0x7fd6fc0b2d10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fd6fc0b5890 tx=0x7fd6f002b840 comp rx=0 tx=0).stop 2023-10-14T16:40:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 -- 172.21.15.29:0/3524352739 shutdown_connections 2023-10-14T16:40:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 --2- 172.21.15.29:0/3524352739 >> v2:172.21.15.186:3300/0 conn(0x7fd6fc0b6cd0 0x7fd6fc0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 --2- 172.21.15.29:0/3524352739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0cf710 0x7fd6fc0b2d10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 -- 172.21.15.29:0/3524352739 >> 172.21.15.29:0/3524352739 conn(0x7fd6fc0d2cc0 msgr2=0x7fd6fc0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 -- 172.21.15.29:0/3524352739 shutdown_connections 2023-10-14T16:40:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.293+0000 7fd701165640 1 -- 172.21.15.29:0/3524352739 wait complete. 2023-10-14T16:40:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.295+0000 7f801e59c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 0x7f800006d400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8018005880 tx=0x7f8018009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.295+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9004e160 con 0x7fae9006c140 2023-10-14T16:40:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.297+0000 7fd701165640 1 Processor -- start 2023-10-14T16:40:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.297+0000 7fd701165640 1 -- start start 2023-10-14T16:40:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.298+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0ee401d460 con 0x7f0eec1f4f10 2023-10-14T16:40:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6fc0b6cd0 0x7fd6fc14b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0b1490 0x7fd6fc0b1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc1ee600 con 0x7fd6fc0b1490 2023-10-14T16:40:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc14c040 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fd701165640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6fc0ca0d0 con 0x7fd6fc0b6cd0 2023-10-14T16:40:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.297+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae8801d3b0 con 0x7fae9006c140 2023-10-14T16:40:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.298+0000 7fae867fc640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 0x7fae7406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.300+0000 7f0edaffd640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 0x7f0ecc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.299+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae88089670 con 0x7fae9006c140 2023-10-14T16:40:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.300+0000 7fd6fa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.300+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ee4089ee0 con 0x7f0eec1f4f10 2023-10-14T16:40:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.300+0000 7fd6fa59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35176/0 (socket says 172.21.15.29:35176) 2023-10-14T16:40:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.300+0000 7fd6fa59c640 1 -- 172.21.15.29:0/2211553865 learned_addr learned my addr 172.21.15.29:0/2211553865 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fd6fa59c640 1 -- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6fc0b6cd0 msgr2=0x7fd6fc14b770 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:40:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fd6fa59c640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6fc0b6cd0 0x7fd6fc14b770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fd6fa59c640 1 -- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0b1490 msgr2=0x7fd6fc0b1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fd6fa59c640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0b1490 0x7fd6fc0b1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fd6fa59c640 1 -- 172.21.15.29:0/2211553865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f00093a0 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.301+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed74055620 con 0x7fed800712c0 2023-10-14T16:40:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.302+0000 7fae95532640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 0x7fae7406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.302+0000 7fd6fa59c640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c450 tx=0x7fd6ec00c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.302+0000 7f0eea59c640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 0x7f0ecc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.303+0000 7fae95532640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 0x7fae7406d380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae80007fd0 tx=0x7fae80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.303+0000 7f0eea59c640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 0x7f0ecc06d360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0edc006df0 tx=0x7f0edc006d80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.309+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec0052d0 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.309+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6ec005430 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.309+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6ec012850 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.310+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ee4055000 con 0x7f0eec1f4f10 2023-10-14T16:40:33.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.310+0000 7fd701165640 1 -- 172.21.15.29:0/2211553865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6fc0ca3b0 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.310+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae88054700 con 0x7fae9006c140 2023-10-14T16:40:33.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.310+0000 7fd701165640 1 -- 172.21.15.29:0/2211553865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6fc0afd10 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.312+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6fc0af890 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.314+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd6ec0129b0 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.315+0000 7fd6e37fe640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 0x7fd6e806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.316+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd6ec0585a0 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.316+0000 7fd6fad9d640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 0x7fd6e806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:33.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.317+0000 7fd6fad9d640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 0x7fd6e806d330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0036000 tx=0x7fd6f0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:33.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.332+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6ec054970 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.694+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fde981f6ab0 con 0x7fde9815cf10 2023-10-14T16:40:33.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.695+0000 7fde957fa640 1 -- 172.21.15.29:0/2514548091 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fde981f6ab0 con 0x7fde9815cf10 2023-10-14T16:40:33.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.696+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7ff8640057d0 con 0x7ff87c158eb0 2023-10-14T16:40:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.696+0000 7ff872ffd640 1 -- 172.21.15.29:0/2231655861 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff878025020 con 0x7ff87c158eb0 2023-10-14T16:40:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.698 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.699+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 msgr2=0x7fde8406d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.699+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 0x7fde8406d400 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fde8c000f60 tx=0x7fde8c009040 comp rx=0 tx=0).stop 2023-10-14T16:40:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.699+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 msgr2=0x7fde9814ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.699+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 0x7fde9814ef80 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fde88032000 tx=0x7fde880046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 msgr2=0x7ff86007e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 shutdown_connections 2023-10-14T16:40:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde8406af40 0x7fde8406d400 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde9815cf10 0x7fde9814ef80 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde980b5c30 0x7fde980cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 --2- 172.21.15.29:0/2514548091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde980b4580 0x7fde98150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.700+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 >> 172.21.15.29:0/2514548091 conn(0x7fde980cf580 msgr2=0x7fde980d0970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.701+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 0x7ff86007e340 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff87c1563f0 tx=0x7ff86c009040 comp rx=0 tx=0).stop 2023-10-14T16:40:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.701+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 shutdown_connections 2023-10-14T16:40:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.701+0000 7fde9edb0640 1 -- 172.21.15.29:0/2514548091 wait complete. 2023-10-14T16:40:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.701+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 msgr2=0x7ff87c0c4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.701+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 0x7ff87c0c4cb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7ff878006ca0 tx=0x7ff87800e0c0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 shutdown_connections 2023-10-14T16:40:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff86007be80 0x7ff86007e340 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff87c15c880 0x7ff87c0c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff87c158eb0 0x7ff87c0c4cb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 --2- 172.21.15.29:0/2231655861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff87c0cd350 0x7ff87c1557a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.702+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 >> 172.21.15.29:0/2231655861 conn(0x7ff87c0ce6e0 msgr2=0x7ff87c0ceac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.703+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 shutdown_connections 2023-10-14T16:40:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.703+0000 7ff88446f640 1 -- 172.21.15.29:0/2231655861 wait complete. 2023-10-14T16:40:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.704+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f80201363e0 con 0x7f8020158f60 2023-10-14T16:40:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.705+0000 7f8016ffd640 1 -- 172.21.15.29:0/2213282357 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f80201363e0 con 0x7f8020158f60 2023-10-14T16:40:33.708 INFO:teuthology.orchestra.run.smithi029.stdout:55834574852 2023-10-14T16:40:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.709+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 msgr2=0x7f800006d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.709+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 0x7f800006d400 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8018005880 tx=0x7f8018009040 comp rx=0 tx=0).stop 2023-10-14T16:40:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.709+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 msgr2=0x7f80200c8190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.709+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f8010007e40 tx=0x7f801000e4b0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 shutdown_connections 2023-10-14T16:40:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f800006af40 0x7f800006d400 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8020158f60 0x7f80200c8190 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80200ce1c0 0x7f80200c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 --2- 172.21.15.29:0/2213282357 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f802006a4b0 0x7f80200c70c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.710+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 >> 172.21.15.29:0/2213282357 conn(0x7f80200ce9b0 msgr2=0x7f802006f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.711+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 shutdown_connections 2023-10-14T16:40:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.711+0000 7f802487c640 1 -- 172.21.15.29:0/2213282357 wait complete. 2023-10-14T16:40:33.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.724+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f1b980b1580 con 0x7f1b980ade50 2023-10-14T16:40:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.725+0000 7f1b8d7fa640 1 -- 172.21.15.29:0/4009353642 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1b980b1580 con 0x7f1b980ade50 2023-10-14T16:40:33.729 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.732 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.2 2023-10-14T16:40:33.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 msgr2=0x7f1b8406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 0x7f1b8406d330 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b94006fd0 tx=0x7f1b94008040 comp rx=0 tx=0).stop 2023-10-14T16:40:33.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 msgr2=0x7f1b9814a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 0x7f1b9814a870 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1b8800b710 tx=0x7f1b880091c0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 shutdown_connections 2023-10-14T16:40:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1b8406ae70 0x7f1b8406d330 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b98159110 0x7f1b9814adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b980ade50 0x7f1b9814a870 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 --2- 172.21.15.29:0/4009353642 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b9806a2e0 0x7f1b980af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 >> 172.21.15.29:0/4009353642 conn(0x7f1b980ce9b0 msgr2=0x7f1b9806fcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 shutdown_connections 2023-10-14T16:40:33.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.732+0000 7f1b9f061640 1 -- 172.21.15.29:0/4009353642 wait complete. 2023-10-14T16:40:33.736 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.4 2023-10-14T16:40:33.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.736+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fae900ccb60 con 0x7fae9006c140 2023-10-14T16:40:33.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.737+0000 7fae867fc640 1 -- 172.21.15.29:0/3949157055 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fae88057d20 con 0x7fae9006c140 2023-10-14T16:40:33.738 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.737+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f0eec0b3a90 con 0x7f0eec1f4f10 2023-10-14T16:40:33.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.738+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fed800c7780 con 0x7fed800712c0 2023-10-14T16:40:33.740 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.0 2023-10-14T16:40:33.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.738+0000 7f0edaffd640 1 -- 172.21.15.29:0/26826115 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0ee4058620 con 0x7f0eec1f4f10 2023-10-14T16:40:33.741 INFO:teuthology.orchestra.run.smithi029.stdout:55834574852 2023-10-14T16:40:33.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.739+0000 7fed67fff640 1 -- 172.21.15.29:0/84874520 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed800c7780 con 0x7fed800712c0 2023-10-14T16:40:33.741 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 msgr2=0x7fae7406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 0x7fae7406d380 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fae80007fd0 tx=0x7fae80009040 comp rx=0 tx=0).stop 2023-10-14T16:40:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 msgr2=0x7fae900c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fae8800ba50 tx=0x7fae8800bf10 comp rx=0 tx=0).stop 2023-10-14T16:40:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 shutdown_connections 2023-10-14T16:40:33.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7406aec0 0x7fae7406d380 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae90159e90 0x7fae9015c110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae90159ab0 0x7fae900af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 --2- 172.21.15.29:0/3949157055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9006c140 0x7fae900c6600 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 >> 172.21.15.29:0/3949157055 conn(0x7fae900ce9b0 msgr2=0x7fae900b0b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 shutdown_connections 2023-10-14T16:40:33.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.740+0000 7fae97f97640 1 -- 172.21.15.29:0/3949157055 wait complete. 2023-10-14T16:40:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 msgr2=0x7f0ecc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 0x7f0ecc06d360 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0edc006df0 tx=0x7f0edc006d80 comp rx=0 tx=0).stop 2023-10-14T16:40:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 msgr2=0x7f0eec0b3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f0eec06af00 tx=0x7f0ee400d890 comp rx=0 tx=0).stop 2023-10-14T16:40:33.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 shutdown_connections 2023-10-14T16:40:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0ecc06aea0 0x7f0ecc06d360 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0eec1f4f10 0x7f0eec0b3db0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0eec1f4a90 0x7f0eec0b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 --2- 172.21.15.29:0/26826115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0ed40047c0 0x7f0eec0a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.742+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 >> 172.21.15.29:0/26826115 conn(0x7f0eec0ce9b0 msgr2=0x7f0eec0af4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.743+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 shutdown_connections 2023-10-14T16:40:33.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.743+0000 7f0ef0a61640 1 -- 172.21.15.29:0/26826115 wait complete. 2023-10-14T16:40:33.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 msgr2=0x7fed6c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 0x7fed6c06d330 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fed7002bbb0 tx=0x7fed70030000 comp rx=0 tx=0).stop 2023-10-14T16:40:33.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 msgr2=0x7fed800c2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 0x7fed800c2fb0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fed7400d5f0 tx=0x7fed7400dab0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 shutdown_connections 2023-10-14T16:40:33.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fed6c06ae70 0x7fed6c06d330 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed80161800 0x7fed800c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed800712c0 0x7fed800c2fb0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 --2- 172.21.15.29:0/84874520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed80070e80 0x7fed800c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 >> 172.21.15.29:0/84874520 conn(0x7fed800ce680 msgr2=0x7fed800c8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 shutdown_connections 2023-10-14T16:40:33.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.745+0000 7fed85a00640 1 -- 172.21.15.29:0/84874520 wait complete. 2023-10-14T16:40:33.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.752+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd6fc0af890 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.753+0000 7fd6e37fe640 1 -- 172.21.15.29:0/2211553865 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd6fc0af890 con 0x7fd6fc0cfaf0 2023-10-14T16:40:33.754 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 msgr2=0x7fd6e806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 0x7fd6e806d330 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fd6f0036000 tx=0x7fd6f0008030 comp rx=0 tx=0).stop 2023-10-14T16:40:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 msgr2=0x7fd6fc1ee0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:33.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c450 tx=0x7fd6ec00c910 comp rx=0 tx=0).stop 2023-10-14T16:40:33.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 shutdown_connections 2023-10-14T16:40:33.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd6e806ae70 0x7fd6e806d330 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6fc0b1490 0x7fd6fc0b1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6fc0cfaf0 0x7fd6fc1ee0c0 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd6ec00c450 tx=0x7fd6ec00c910 comp rx=0 tx=0).stop 2023-10-14T16:40:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.755+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/2211553865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6fc0b6cd0 0x7fd6fc14b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.756+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 >> 172.21.15.29:0/2211553865 conn(0x7fd6fc0d2cc0 msgr2=0x7fd6fc160230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:33.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.756+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 shutdown_connections 2023-10-14T16:40:33.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:33.756+0000 7fd6e17fa640 1 -- 172.21.15.29:0/2211553865 wait complete. 2023-10-14T16:40:33.759 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.3 2023-10-14T16:40:33.761 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.5 2023-10-14T16:40:33.766 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.1 2023-10-14T16:40:33.768 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574852 for osd.6 2023-10-14T16:40:33.775 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574853 for osd.7 2023-10-14T16:40:34.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-10-14T16:40:34.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-10-14T16:40:34.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-10-14T16:40:34.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-10-14T16:40:34.762 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-10-14T16:40:34.767 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-10-14T16:40:34.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2023-10-14T16:40:34.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2023-10-14T16:40:35.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.019+0000 7f13865bf640 1 Processor -- start 2023-10-14T16:40:35.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.019+0000 7f13865bf640 1 -- start start 2023-10-14T16:40:35.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.022+0000 7f13865bf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.022+0000 7f13865bf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 0x7f1378054260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.022+0000 7f13865bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 0x7f13780500c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.023+0000 7f13855bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.023+0000 7f13855bd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35186/0 (socket says 172.21.15.29:35186) 2023-10-14T16:40:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.023+0000 7f13855bd640 1 -- 172.21.15.29:0/2663515615 learned_addr learned my addr 172.21.15.29:0/2663515615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.024+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13780df8e0 con 0x7f1378005fe0 2023-10-14T16:40:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.024+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1378050600 con 0x7f13780df500 2023-10-14T16:40:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.024+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13780064e0 con 0x7f137804fc70 2023-10-14T16:40:35.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.024+0000 7f1385dbe640 1 --2- 172.21.15.29:0/2663515615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 0x7f13780500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 0x7f1378054260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1385dbe640 1 -- 172.21.15.29:0/2663515615 >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 msgr2=0x7f1378007580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1385dbe640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1385dbe640 1 -- 172.21.15.29:0/2663515615 >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 msgr2=0x7f1378054260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1385dbe640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 0x7f1378054260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1385dbe640 1 -- 172.21.15.29:0/2663515615 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13780067c0 con 0x7f137804fc70 2023-10-14T16:40:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.025+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 0x7f1378054260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.027+0000 7f13855bd640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.028+0000 7f1385dbe640 1 --2- 172.21.15.29:0/2663515615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 0x7f13780500c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f138004fa60 tx=0x7f13800a5d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2cd11fe8f7c8259f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.028+0000 7f1377fff640 1 -- 172.21.15.29:0/2663515615 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800a69a0 con 0x7f137804fc70 2023-10-14T16:40:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.028+0000 7f1377fff640 1 -- 172.21.15.29:0/2663515615 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13800a6b40 con 0x7f137804fc70 2023-10-14T16:40:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.028+0000 7f1377fff640 1 -- 172.21.15.29:0/2663515615 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13800ae440 con 0x7f137804fc70 2023-10-14T16:40:35.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.029+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 msgr2=0x7f13780500c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.029+0000 7f13865bf640 1 --2- 172.21.15.29:0/2663515615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 0x7f13780500c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f138004fa60 tx=0x7f13800a5d40 comp rx=0 tx=0).stop 2023-10-14T16:40:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 shutdown_connections 2023-10-14T16:40:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 --2- 172.21.15.29:0/2663515615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f137804fc70 0x7f13780500c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.29:3300/0 conn(0x7f1378005fe0 0x7f1378054260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 --2- 172.21.15.29:0/2663515615 >> v2:172.21.15.186:3300/0 conn(0x7f13780df500 0x7f1378007580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 >> 172.21.15.29:0/2663515615 conn(0x7f1378054c10 msgr2=0x7f1378055020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 shutdown_connections 2023-10-14T16:40:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.031+0000 7f13865bf640 1 -- 172.21.15.29:0/2663515615 wait complete. 2023-10-14T16:40:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.034+0000 7f13865bf640 1 Processor -- start 2023-10-14T16:40:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.035+0000 7f13865bf640 1 -- start start 2023-10-14T16:40:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.037+0000 7f13865bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.037+0000 7f13865bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 0x7f1378184d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.038+0000 7f13855bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.038+0000 7f13855bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35196/0 (socket says 172.21.15.29:35196) 2023-10-14T16:40:35.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.038+0000 7f13855bd640 1 -- 172.21.15.29:0/2382627201 learned_addr learned my addr 172.21.15.29:0/2382627201 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.039+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13780df500 0x7f137816ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.039+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13780db2e0 con 0x7f137804fc70 2023-10-14T16:40:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.039+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1378048ef0 con 0x7f1378005fe0 2023-10-14T16:40:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.040+0000 7f1385dbe640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13780df500 0x7f137816ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.040+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137804f460 con 0x7f13780df500 2023-10-14T16:40:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.040+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 0x7f1378184d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.041+0000 7f13855bd640 1 -- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13780df500 msgr2=0x7f137816ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.041+0000 7f13855bd640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13780df500 0x7f137816ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.041+0000 7f13855bd640 1 -- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 msgr2=0x7f1378184d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.042+0000 7f13855bd640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 0x7f1378184d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.042+0000 7f13855bd640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f138004f050 con 0x7f1378005fe0 2023-10-14T16:40:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.042+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 0x7f1378184d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.043+0000 7f13855bd640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1370006f40 tx=0x7f137000dcb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.044+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137000f040 con 0x7f1378005fe0 2023-10-14T16:40:35.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.044+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f137000ba30 con 0x7f1378005fe0 2023-10-14T16:40:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.045+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f137816de10 con 0x7f1378005fe0 2023-10-14T16:40:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.045+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1378048b80 con 0x7f1378005fe0 2023-10-14T16:40:35.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.045+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f137000bda0 con 0x7f1378005fe0 2023-10-14T16:40:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.047+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360005250 con 0x7f1378005fe0 2023-10-14T16:40:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.050+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f137001c400 con 0x7f1378005fe0 2023-10-14T16:40:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.052+0000 7f1375ffb640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 0x7f136406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.053+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13700898c0 con 0x7f1378005fe0 2023-10-14T16:40:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.054+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 0x7f136406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.055+0000 7f1384dbc640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 0x7f136406d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.069+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1370054970 con 0x7f1378005fe0 2023-10-14T16:40:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.073+0000 7fcc1b59e640 1 Processor -- start 2023-10-14T16:40:35.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.073+0000 7fcc1b59e640 1 -- start start 2023-10-14T16:40:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.075+0000 7fcc1b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc140e3190 0x7fcc1403e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.075+0000 7fcc1b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc140544a0 0x7fcc140573e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.075+0000 7fcc1b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc1403ee60 con 0x7fcc14038180 2023-10-14T16:40:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.075+0000 7fcc1b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc1403f100 con 0x7fcc140e3190 2023-10-14T16:40:35.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.075+0000 7fcc1b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc14007c40 con 0x7fcc140544a0 2023-10-14T16:40:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.076+0000 7fcc1a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc140e3190 0x7fcc1403e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.076+0000 7fcc1a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcc140e3190 0x7fcc1403e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36668/0 (socket says 172.21.15.29:36668) 2023-10-14T16:40:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.076+0000 7fcc1a59c640 1 -- 172.21.15.29:0/302755443 learned_addr learned my addr 172.21.15.29:0/302755443 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.077+0000 7fcc1ad9d640 1 --2- 172.21.15.29:0/302755443 >> v2:172.21.15.186:3300/0 conn(0x7fcc140544a0 0x7fcc140573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.077+0000 7fcc1a59c640 1 -- 172.21.15.29:0/302755443 >> v2:172.21.15.186:3300/0 conn(0x7fcc140544a0 msgr2=0x7fcc140573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.077+0000 7fcc1a59c640 1 --2- 172.21.15.29:0/302755443 >> v2:172.21.15.186:3300/0 conn(0x7fcc140544a0 0x7fcc140573e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.077+0000 7fcc1a59c640 1 -- 172.21.15.29:0/302755443 >> v1:172.21.15.186:6789/0 conn(0x7fcc14038180 legacy=0x7fcc14053d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.077+0000 7fcc1a59c640 1 -- 172.21.15.29:0/302755443 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc14006e80 con 0x7fcc140e3190 2023-10-14T16:40:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.079+0000 7fcc1a59c640 1 --2- 172.21.15.29:0/302755443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140e3190 0x7fcc1403e920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc14059d70 tx=0x7fcc1002b9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1099b652f4bb7fac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.080+0000 7fcc1959a640 1 -- 172.21.15.29:0/302755443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10030880 con 0x7fcc140e3190 2023-10-14T16:40:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.080+0000 7fcc1959a640 1 -- 172.21.15.29:0/302755443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc100309e0 con 0x7fcc140e3190 2023-10-14T16:40:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.080+0000 7fcc1959a640 1 -- 172.21.15.29:0/302755443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10030880 con 0x7fcc140e3190 2023-10-14T16:40:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 -- 172.21.15.29:0/302755443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140e3190 msgr2=0x7fcc1403e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 --2- 172.21.15.29:0/302755443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140e3190 0x7fcc1403e920 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcc14059d70 tx=0x7fcc1002b9c0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 -- 172.21.15.29:0/302755443 shutdown_connections 2023-10-14T16:40:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 --2- 172.21.15.29:0/302755443 >> v2:172.21.15.186:3300/0 conn(0x7fcc140544a0 0x7fcc140573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 --2- 172.21.15.29:0/302755443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140e3190 0x7fcc1403e920 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.081+0000 7fcc1b59e640 1 -- 172.21.15.29:0/302755443 >> 172.21.15.29:0/302755443 conn(0x7fcc140590c0 msgr2=0x7fcc140594d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.082+0000 7fcc1b59e640 1 -- 172.21.15.29:0/302755443 shutdown_connections 2023-10-14T16:40:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.082+0000 7fcc1b59e640 1 -- 172.21.15.29:0/302755443 wait complete. 2023-10-14T16:40:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.084+0000 7fd7d147c640 1 Processor -- start 2023-10-14T16:40:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.085+0000 7fd7d147c640 1 -- start start 2023-10-14T16:40:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.087+0000 7fd7d147c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.087+0000 7fd7d147c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 0x7fd7cc0b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.087+0000 7fcc1b59e640 1 Processor -- start 2023-10-14T16:40:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.087+0000 7fcc1b59e640 1 -- start start 2023-10-14T16:40:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.087+0000 7fd7caffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.088+0000 7fd7caffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:47968/0 (socket says 172.21.15.29:47968) 2023-10-14T16:40:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.088+0000 7fd7caffd640 1 -- 172.21.15.29:0/927598336 learned_addr learned my addr 172.21.15.29:0/927598336 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.088+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7cc0cb200 con 0x7fd7cc0b1d40 2023-10-14T16:40:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.088+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7cc0b2190 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.088+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc06ce70 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.089+0000 7fd7ca7fc640 1 --2- 172.21.15.29:0/927598336 >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 0x7fd7cc0b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.089+0000 7fcc1b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.089+0000 7fcc1b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 0x7fcc1400d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.089+0000 7fcc1b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc140e3190 0x7fcc14052890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.090+0000 7fcc1a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.090+0000 7fcc19d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 0x7fcc1400d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.089+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295971095 0 0) 0x7fd7c4001490 con 0x7fd7cc0b1d40 2023-10-14T16:40:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.090+0000 7fcc1a59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35210/0 (socket says 172.21.15.29:35210) 2023-10-14T16:40:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.090+0000 7fcc19d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 0x7fcc1400d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36682/0 (socket says 172.21.15.29:36682) 2023-10-14T16:40:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.090+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7b4003520 con 0x7fd7cc0b1d40 2023-10-14T16:40:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fd7caffd640 1 -- 172.21.15.29:0/927598336 >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 msgr2=0x7fd7cc0b1800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4059768080 0 0) 0x7fd7b4003520 con 0x7fd7cc0b1d40 2023-10-14T16:40:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fcc1a59c640 1 -- 172.21.15.29:0/1118158411 learned_addr learned my addr 172.21.15.29:0/1118158411 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fcc1b59e640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1400b590 con 0x7fcc140544a0 2023-10-14T16:40:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fcc1b59e640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1400b870 con 0x7fcc14038180 2023-10-14T16:40:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fcc1b59e640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc1400bb50 con 0x7fcc140e3190 2023-10-14T16:40:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fcc1a59c640 1 -- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc140e3190 msgr2=0x7fcc14052890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fcc1a59c640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc140e3190 0x7fcc14052890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fcc1a59c640 1 -- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 msgr2=0x7fcc1400d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fcc1a59c640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 0x7fcc1400d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fcc1a59c640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc100094d0 con 0x7fcc14038180 2023-10-14T16:40:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fd7caffd640 1 --2- 172.21.15.29:0/927598336 >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 0x7fd7cc0b1800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fd7caffd640 1 -- 172.21.15.29:0/927598336 >> v1:172.21.15.29:6789/0 conn(0x7fd7cc0b1d40 legacy=0x7fd7cc0ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.091+0000 7fd7caffd640 1 -- 172.21.15.29:0/927598336 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7cc0caf20 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.092+0000 7fd7ca7fc640 1 --2- 172.21.15.29:0/927598336 >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 0x7fd7cc0b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:35.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.093+0000 7fcc1a59c640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc100090b0 tx=0x7fcc100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.093+0000 7fd7caffd640 1 --2- 172.21.15.29:0/927598336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009870 tx=0x7fd7bc02b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3accf70be2324 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.094+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10003aa0 con 0x7fcc14038180 2023-10-14T16:40:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.094+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc10003c00 con 0x7fcc14038180 2023-10-14T16:40:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.094+0000 7fcc1b59e640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc1400be30 con 0x7fcc14038180 2023-10-14T16:40:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.094+0000 7fcc1b59e640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc140dc080 con 0x7fcc14038180 2023-10-14T16:40:35.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.095+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7c4001b80 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.095+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc031540 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.095+0000 7fd7c9ffb640 1 -- 172.21.15.29:0/927598336 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0318d0 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc10039450 con 0x7fcc14038180 2023-10-14T16:40:35.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf4005250 con 0x7fcc14038180 2023-10-14T16:40:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 msgr2=0x7fd7cc06bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fd7d147c640 1 --2- 172.21.15.29:0/927598336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009870 tx=0x7fd7bc02b860 comp rx=0 tx=0).stop 2023-10-14T16:40:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 shutdown_connections 2023-10-14T16:40:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fd7d147c640 1 --2- 172.21.15.29:0/927598336 >> v2:172.21.15.29:3300/0 conn(0x7fd7cc06c1f0 0x7fd7cc0b1800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.096+0000 7fd7d147c640 1 --2- 172.21.15.29:0/927598336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc06bcb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.097+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 >> 172.21.15.29:0/927598336 conn(0x7fd7cc0ce9b0 msgr2=0x7fd7cc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.097+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 shutdown_connections 2023-10-14T16:40:35.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.097+0000 7fd7d147c640 1 -- 172.21.15.29:0/927598336 wait complete. 2023-10-14T16:40:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.100+0000 7fd7d147c640 1 Processor -- start 2023-10-14T16:40:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.102+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcc100395b0 con 0x7fcc14038180 2023-10-14T16:40:35.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fcc0affd640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 0x7fcbfc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.102+0000 7fd7d147c640 1 -- start start 2023-10-14T16:40:35.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.103+0000 7fd7d147c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fd7d147c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 0x7fd7cc0c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fd7d147c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc1f9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fd7d147c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc14b270 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fd7d147c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc158660 con 0x7fd7cc0b1d40 2023-10-14T16:40:35.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fd7d147c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7cc0c7510 con 0x7fd7cc15e2d0 2023-10-14T16:40:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.104+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcc100ac710 con 0x7fcc14038180 2023-10-14T16:40:35.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fd7caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fd7ca7fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 0x7fd7cc0c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fd7caffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36684/0 (socket says 172.21.15.29:36684) 2023-10-14T16:40:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fd7caffd640 1 -- 172.21.15.29:0/2162998703 learned_addr learned my addr 172.21.15.29:0/2162998703 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fd7cb7fe640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc1f9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.105+0000 7fcc19d9b640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 0x7fcbfc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.106+0000 7fd7caffd640 1 -- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 msgr2=0x7fd7cc1f9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.106+0000 7fd7caffd640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc1f9bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fd7caffd640 1 -- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 msgr2=0x7fd7cc0c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fd7caffd640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 0x7fd7cc0c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fd7caffd640 1 -- 172.21.15.29:0/2162998703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc0094d0 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fcc19d9b640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 0x7fcbfc06d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc1404f0e0 tx=0x7fcc0c007e30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fd7cb7fe640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc1f9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.107+0000 7fd7ca7fc640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 0x7fd7cc0c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.108+0000 7fd7caffd640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0090b0 tx=0x7fd7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.109+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc033a00 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.109+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc033b60 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.109+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc03d410 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.109+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7cc0c77f0 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.109+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7cc0c7db0 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.112+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc10077850 con 0x7fcc14038180 2023-10-14T16:40:35.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.112+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7cc04e160 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.114+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd7bc03a050 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.115+0000 7fd7b37fe640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 0x7fd7b806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.118+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd7bc07a030 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.118+0000 7fd7ca7fc640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 0x7fd7b806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.119+0000 7fd7ca7fc640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 0x7fd7b806d380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0ae730 tx=0x7fd7c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.127+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7bc076b00 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.140+0000 7fadde032640 1 Processor -- start 2023-10-14T16:40:35.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.140+0000 7fadde032640 1 -- start start 2023-10-14T16:40:35.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7fadde032640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fadd80cf1c0 0x7fadd806c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7fadde032640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadd80cd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7fadde032640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fadd80cdbc0 con 0x7fadd806a080 2023-10-14T16:40:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7fadde032640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fadd80cc8c0 con 0x7fadd80cf1c0 2023-10-14T16:40:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7fadde032640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ccba0 con 0x7fadd806f440 2023-10-14T16:40:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.143+0000 7faddd831640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fadd806a080 0x7fadd806a510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47884/0 (socket says 172.21.15.29:47884) 2023-10-14T16:40:35.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.143+0000 7faddd831640 1 -- 172.21.15.29:0/3664717275 learned_addr learned my addr 172.21.15.29:0/3664717275 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:35.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.143+0000 7f1c94997640 1 Processor -- start 2023-10-14T16:40:35.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.143+0000 7f1c94997640 1 -- start start 2023-10-14T16:40:35.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7f358b147640 1 Processor -- start 2023-10-14T16:40:35.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.142+0000 7f358b147640 1 -- start start 2023-10-14T16:40:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7faddd030640 1 --2- 172.21.15.29:0/3664717275 >> v2:172.21.15.186:3300/0 conn(0x7fadd80cf1c0 0x7fadd806c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7faddc82f640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadd80cd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119025260 0 0) 0x7fadd80cdbc0 con 0x7fadd806a080 2023-10-14T16:40:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadc4003520 con 0x7fadd806a080 2023-10-14T16:40:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 434592151 0 0) 0x7fadc4003520 con 0x7fadd806a080 2023-10-14T16:40:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 msgr2=0x7fadd80cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadd80cd680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 >> v2:172.21.15.186:3300/0 conn(0x7fadd80cf1c0 msgr2=0x7fadd806c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7fadcbfff640 1 --2- 172.21.15.29:0/3664717275 >> v2:172.21.15.186:3300/0 conn(0x7fadd80cf1c0 0x7fadd806c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80cce80 con 0x7fadd806a080 2023-10-14T16:40:35.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7f358b147640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35840cc9e0 0x7f3584069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.144+0000 7f358b147640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358408a570 con 0x7f358416e4d0 2023-10-14T16:40:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7f358b147640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f358416e8b0 con 0x7f35840cc580 2023-10-14T16:40:35.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7f358b147640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f358408ad80 con 0x7f35840cc9e0 2023-10-14T16:40:35.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fadd4001650 con 0x7fadd806a080 2023-10-14T16:40:35.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 >> v1:172.21.15.29:6789/0 conn(0x7fadd806a080 legacy=0x7fadd806a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7f3588ee3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f358416e4d0 0x7f35840caaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47890/0 (socket says 172.21.15.29:47890) 2023-10-14T16:40:35.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7f3588ee3640 1 -- 172.21.15.29:0/3796745808 learned_addr learned my addr 172.21.15.29:0/3796745808 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:35.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadc40039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4003f30 0x7fadc40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80cc8c0 con 0x7fadc4004d00 2023-10-14T16:40:35.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80ccba0 con 0x7fadc4003f30 2023-10-14T16:40:35.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.146+0000 7f35896e4640 1 --2- 172.21.15.29:0/3796745808 >> v2:172.21.15.29:3300/0 conn(0x7f35840cc9e0 0x7f3584069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadc400ba30 con 0x7fadd806f440 2023-10-14T16:40:35.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.145+0000 7faddc82f640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4003f30 0x7fadc40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.146+0000 7faddd030640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadc40039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.146+0000 7faddd831640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddd030640 1 -- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4003f30 msgr2=0x7fadc40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3394543752 0 0) 0x7f358408a570 con 0x7f358416e4d0 2023-10-14T16:40:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c003520 con 0x7f358416e4d0 2023-10-14T16:40:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601412950 0 0) 0x7f358416e8b0 con 0x7f35840cc580 2023-10-14T16:40:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f356c003840 con 0x7f35840cc580 2023-10-14T16:40:35.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddd030640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4003f30 0x7fadc40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c90158f60 0x7f1c900ade50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1c9006a2e0 0x7f1c9006b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f35896e4640 1 -- 172.21.15.29:0/3796745808 >> v1:172.21.15.186:6789/0 conn(0x7f35840cc580 legacy=0x7f358406b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddd030640 1 -- 172.21.15.29:0/3664717275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc4004d00 msgr2=0x7fadc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddd030640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.148+0000 7f35896e4640 1 -- 172.21.15.29:0/3796745808 >> v1:172.21.15.29:6789/0 conn(0x7f358416e4d0 legacy=0x7f35840caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.148+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2320421342 0 0) 0x7f356c003520 con 0x7f358416e4d0 2023-10-14T16:40:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.148+0000 7f35896e4640 1 -- 172.21.15.29:0/3796745808 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35840cce10 con 0x7f35840cc9e0 2023-10-14T16:40:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddd030640 1 -- 172.21.15.29:0/3664717275 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80cce80 con 0x7fadd806f440 2023-10-14T16:40:35.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7faddc82f640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadc4003f30 0x7fadc40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:35.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9015d970 con 0x7f1c90158f60 2023-10-14T16:40:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9006cf50 con 0x7f1c9006a2e0 2023-10-14T16:40:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.147+0000 7f1c94997640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9006a730 con 0x7f1c900ce1c0 2023-10-14T16:40:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.148+0000 7f1c8e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c90158f60 0x7f1c900ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:48020/0 (socket says 172.21.15.29:48020) 2023-10-14T16:40:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 -- 172.21.15.29:0/1038121501 learned_addr learned my addr 172.21.15.29:0/1038121501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c8ed9d640 1 --2- 172.21.15.29:0/1038121501 >> v2:172.21.15.186:3300/0 conn(0x7f1c9006a2e0 0x7f1c9006b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7faddd030640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadc40039b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009ac0 tx=0x7fadc40057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9bb07b044ff9480b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7faddd831640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadc4004d00 0x7fadc4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd002c710 con 0x7fadd806f440 2023-10-14T16:40:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd002c870 con 0x7fadd806f440 2023-10-14T16:40:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7fadde032640 1 -- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 msgr2=0x7fadc40039b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7f35896e4640 1 --2- 172.21.15.29:0/3796745808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc9e0 0x7f3584069da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f357c009d90 tx=0x7f357c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466d8a17e95581c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3574001650 con 0x7f35840cc9e0 2023-10-14T16:40:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c02c9d0 con 0x7f35840cc9e0 2023-10-14T16:40:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.150+0000 7f35837fe640 1 -- 172.21.15.29:0/3796745808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c02cd60 con 0x7f35840cc9e0 2023-10-14T16:40:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7fadcbfff640 1 -- 172.21.15.29:0/3664717275 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd002c710 con 0x7fadd806f440 2023-10-14T16:40:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7fadde032640 1 --2- 172.21.15.29:0/3664717275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd806f440 0x7fadc40039b0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009ac0 tx=0x7fadc40057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7faddd831640 1 -- 172.21.15.29:0/3664717275 reap_dead start 2023-10-14T16:40:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 -- 172.21.15.29:0/1038121501 >> v2:172.21.15.186:3300/0 conn(0x7f1c9006a2e0 msgr2=0x7f1c9006b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/1038121501 >> v2:172.21.15.186:3300/0 conn(0x7f1c9006a2e0 0x7f1c9006b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 -- 172.21.15.29:0/1038121501 >> v2:172.21.15.29:3300/0 conn(0x7f1c90158f60 msgr2=0x7f1c900ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/1038121501 >> v2:172.21.15.29:3300/0 conn(0x7f1c90158f60 0x7f1c900ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.149+0000 7f1c85d9b640 1 -- 172.21.15.29:0/1038121501 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9015ccf0 con 0x7f1c900ce1c0 2023-10-14T16:40:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/1038121501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c9015ca90 tx=0x7f1c7802ffc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43229fb89fbee04b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7f358b147640 1 -- 172.21.15.29:0/3796745808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc9e0 msgr2=0x7f3584069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.151+0000 7f358b147640 1 --2- 172.21.15.29:0/3796745808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc9e0 0x7f3584069da0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f357c009d90 tx=0x7f357c02bea0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7fadde032640 1 -- 172.21.15.29:0/3664717275 shutdown_connections 2023-10-14T16:40:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7fadde032640 1 -- 172.21.15.29:0/3664717275 >> 172.21.15.29:0/3664717275 conn(0x7fadd80ce6a0 msgr2=0x7fadd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f1c8dd9b640 1 -- 172.21.15.29:0/1038121501 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c78030d70 con 0x7f1c900ce1c0 2023-10-14T16:40:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f1c8dd9b640 1 -- 172.21.15.29:0/1038121501 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c78034070 con 0x7f1c900ce1c0 2023-10-14T16:40:35.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f1c8dd9b640 1 -- 172.21.15.29:0/1038121501 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c7803a620 con 0x7f1c900ce1c0 2023-10-14T16:40:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f358b147640 1 -- 172.21.15.29:0/3796745808 shutdown_connections 2023-10-14T16:40:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f358b147640 1 --2- 172.21.15.29:0/3796745808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc9e0 0x7f3584069da0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f358b147640 1 -- 172.21.15.29:0/3796745808 >> 172.21.15.29:0/3796745808 conn(0x7f35840ce9b0 msgr2=0x7f35840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7fadde032640 1 -- 172.21.15.29:0/3664717275 shutdown_connections 2023-10-14T16:40:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7fadde032640 1 -- 172.21.15.29:0/3664717275 wait complete. 2023-10-14T16:40:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f358b147640 1 -- 172.21.15.29:0/3796745808 shutdown_connections 2023-10-14T16:40:35.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.153+0000 7f1c94997640 1 -- 172.21.15.29:0/1038121501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 msgr2=0x7f1c90069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.153+0000 7f1c94997640 1 --2- 172.21.15.29:0/1038121501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1c9015ca90 tx=0x7f1c7802ffc0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 -- 172.21.15.29:0/1038121501 shutdown_connections 2023-10-14T16:40:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 --2- 172.21.15.29:0/1038121501 >> v2:172.21.15.186:3300/0 conn(0x7f1c9006a2e0 0x7f1c9006b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 --2- 172.21.15.29:0/1038121501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c900ce1c0 0x7f1c90069da0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 --2- 172.21.15.29:0/1038121501 >> v2:172.21.15.29:3300/0 conn(0x7f1c90158f60 0x7f1c900ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 -- 172.21.15.29:0/1038121501 >> 172.21.15.29:0/1038121501 conn(0x7f1c900ce9b0 msgr2=0x7f1c900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 -- 172.21.15.29:0/1038121501 shutdown_connections 2023-10-14T16:40:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.152+0000 7f358b147640 1 -- 172.21.15.29:0/3796745808 wait complete. 2023-10-14T16:40:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.155+0000 7fadde032640 1 Processor -- start 2023-10-14T16:40:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.155+0000 7f358b147640 1 Processor -- start 2023-10-14T16:40:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.155+0000 7fadde032640 1 -- start start 2023-10-14T16:40:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.155+0000 7f358b147640 1 -- start start 2023-10-14T16:40:35.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.156+0000 7fd1ab10a640 1 Processor -- start 2023-10-14T16:40:35.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.157+0000 7fadde032640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 0x7fadd8151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.154+0000 7f1c94997640 1 -- 172.21.15.29:0/1038121501 wait complete. 2023-10-14T16:40:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.157+0000 7f1c94997640 1 Processor -- start 2023-10-14T16:40:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.157+0000 7fadde032640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 0x7fadd8152bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.157+0000 7fadde032640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd8153110 0x7fadd80a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f1c94997640 1 -- start start 2023-10-14T16:40:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f358b147640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fadde032640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80c6990 con 0x7fadd80c5770 2023-10-14T16:40:35.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fadde032640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd8153570 con 0x7fadd806f440 2023-10-14T16:40:35.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fadde032640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fadd80c5d10 con 0x7fadd8153110 2023-10-14T16:40:35.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f358b147640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35840cc9e0 0x7f35840c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f3588ee3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f358b147640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 0x7f35840c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.158+0000 7f3588ee3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36740/0 (socket says 172.21.15.29:36740) 2023-10-14T16:40:35.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f3588ee3640 1 -- 172.21.15.29:0/600577732 learned_addr learned my addr 172.21.15.29:0/600577732 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f358b147640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3584214a20 con 0x7f35840cc580 2023-10-14T16:40:35.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f358b147640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3584213e50 con 0x7f358416e4d0 2023-10-14T16:40:35.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f358b147640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358417d960 con 0x7f35840cc9e0 2023-10-14T16:40:35.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.157+0000 7fd1ab10a640 1 -- start start 2023-10-14T16:40:35.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fd1ab10a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a40ca340 0x7fd1a406a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fd1ab10a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40cc5e0 con 0x7fd1a406f440 2023-10-14T16:40:35.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fd1ab10a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1a40cc8c0 con 0x7fd1a40c9850 2023-10-14T16:40:35.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7fd1ab10a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1a40ccba0 con 0x7fd1a40ca340 2023-10-14T16:40:35.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f3583fff640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35840cc9e0 0x7f35840c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7faddd831640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd8153110 0x7fadd80a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7faddd030640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 0x7fadd8151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7faddd030640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 0x7fadd8151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35274/0 (socket says 172.21.15.29:35274) 2023-10-14T16:40:35.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7faddd030640 1 -- 172.21.15.29:0/135856740 learned_addr learned my addr 172.21.15.29:0/135856740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7faddc82f640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 0x7fadd8152bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.159+0000 7f35896e4640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 0x7f35840c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7fd1aa909640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a40ca340 0x7fd1a406a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7fd1aa909640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1a40ca340 0x7fd1a406a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36746/0 (socket says 172.21.15.29:36746) 2023-10-14T16:40:35.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f1c94997640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f35896e4640 1 -- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35840cc9e0 msgr2=0x7f35840c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f35896e4640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35840cc9e0 0x7f35840c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f35896e4640 1 -- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 msgr2=0x7f358417d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f35896e4640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f35896e4640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357c009a70 con 0x7f358416e4d0 2023-10-14T16:40:35.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7f3588ee3640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:35.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.160+0000 7fd1aa909640 1 -- 172.21.15.29:0/3944133609 learned_addr learned my addr 172.21.15.29:0/3944133609 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7faddc82f640 1 -- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd8153110 msgr2=0x7fadd80a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7faddc82f640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd8153110 0x7fadd80a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7faddc82f640 1 -- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 msgr2=0x7fadd8151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7faddc82f640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 0x7fadd8151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7faddc82f640 1 -- 172.21.15.29:0/135856740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80c5ff0 con 0x7fadd80c5770 2023-10-14T16:40:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f1c94997640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 0x7f1c90070fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f1c94997640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c90158f60 0x7f1c900c8560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f1c8e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f1c8e59c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35302/0 (socket says 172.21.15.29:35302) 2023-10-14T16:40:35.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f35896e4640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 0x7f35840c5fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f357c033000 tx=0x7f357c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.161+0000 7f1c8e59c640 1 -- 172.21.15.29:0/3569594561 learned_addr learned my addr 172.21.15.29:0/3569594561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c90159340 con 0x7f1c900ce1c0 2023-10-14T16:40:35.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7faddc82f640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 0x7fadd8152bd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadcc011e30 tx=0x7fadcc00f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c901e8710 con 0x7f1c9006a2e0 2023-10-14T16:40:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f1c8ed9d640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c90158f60 0x7f1c900c8560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c035d60 con 0x7f358416e4d0 2023-10-14T16:40:35.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f358b147640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35842140f0 con 0x7f358416e4d0 2023-10-14T16:40:35.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c90159590 con 0x7f1c90158f60 2023-10-14T16:40:35.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 0x7f1c90070fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f358b147640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35842146b0 con 0x7f358416e4d0 2023-10-14T16:40:35.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f357c03e410 con 0x7f358416e4d0 2023-10-14T16:40:35.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f357c035d60 con 0x7f358416e4d0 2023-10-14T16:40:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7fd1a9106640 1 -- 172.21.15.29:0/3944133609 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733917517 0 0) 0x7fd1a40cc8c0 con 0x7fd1a40c9850 2023-10-14T16:40:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7fd1a9106640 1 -- 172.21.15.29:0/3944133609 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd190003440 con 0x7fd1a40c9850 2023-10-14T16:40:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7fd1aa909640 1 -- 172.21.15.29:0/3944133609 >> v1:172.21.15.186:6789/0 conn(0x7fd1a40c9850 legacy=0x7fd1a40cd680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7fd1aa909640 1 -- 172.21.15.29:0/3944133609 >> v1:172.21.15.29:6789/0 conn(0x7fd1a406f440 legacy=0x7fd1a40c9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.162+0000 7fd1aa909640 1 -- 172.21.15.29:0/3944133609 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a40cce80 con 0x7fd1a40ca340 2023-10-14T16:40:35.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.163+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc00e040 con 0x7fadd80c5770 2023-10-14T16:40:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.163+0000 7f1c8e59c640 1 -- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c90158f60 msgr2=0x7f1c900c8560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7f1c8e59c640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c90158f60 0x7f1c900c8560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7f1c8e59c640 1 -- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 msgr2=0x7f1c90070fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7f1c8e59c640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 0x7f1c90070fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7fadde032640 1 -- 172.21.15.29:0/135856740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd80c62d0 con 0x7fadd80c5770 2023-10-14T16:40:35.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7f1c8e59c640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7800b760 con 0x7f1c9006a2e0 2023-10-14T16:40:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadcc00f040 con 0x7fadd80c5770 2023-10-14T16:40:35.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3584214b90 con 0x7f358416e4d0 2023-10-14T16:40:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.164+0000 7fd1aa909640 1 --2- 172.21.15.29:0/3944133609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40ca340 0x7fd1a406a080 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009a50 tx=0x7fd19c02c2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=603d758a44eb42da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7fadde032640 1 -- 172.21.15.29:0/135856740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd81523a0 con 0x7fadd80c5770 2023-10-14T16:40:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 0x7f1c90070fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7f1c8e59c640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1c80007ef0 tx=0x7f1c8000e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadcc003d90 con 0x7fadd80c5770 2023-10-14T16:40:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80014620 con 0x7f1c9006a2e0 2023-10-14T16:40:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.166+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80014780 con 0x7f1c9006a2e0 2023-10-14T16:40:35.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.167+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c90159870 con 0x7f1c9006a2e0 2023-10-14T16:40:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.167+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c90159e30 con 0x7f1c9006a2e0 2023-10-14T16:40:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.165+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f357c03b050 con 0x7f358416e4d0 2023-10-14T16:40:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.167+0000 7f35817fa640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 0x7f357006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.167+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f357c0ab4e0 con 0x7f358416e4d0 2023-10-14T16:40:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.167+0000 7f3588ee3640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 0x7f357006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.168+0000 7f3588ee3640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 0x7f357006d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3574001eb0 tx=0x7f3574001940 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.168+0000 7fd1a9106640 1 -- 172.21.15.29:0/3944133609 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c02ce80 con 0x7fd1a40ca340 2023-10-14T16:40:35.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.168+0000 7fd1a9106640 1 -- 172.21.15.29:0/3944133609 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c0343f0 con 0x7fd1a40ca340 2023-10-14T16:40:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.168+0000 7fd1a9106640 1 -- 172.21.15.29:0/3944133609 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c034760 con 0x7fd1a40ca340 2023-10-14T16:40:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80014af0 con 0x7f1c9006a2e0 2023-10-14T16:40:35.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb0005250 con 0x7fadd80c5770 2023-10-14T16:40:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fd1ab10a640 1 -- 172.21.15.29:0/3944133609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40ca340 msgr2=0x7fd1a406a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/3944133609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40ca340 0x7fd1a406a080 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009a50 tx=0x7fd19c02c2e0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fd1ab10a640 1 -- 172.21.15.29:0/3944133609 shutdown_connections 2023-10-14T16:40:35.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/3944133609 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40ca340 0x7fd1a406a080 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.169+0000 7fd1ab10a640 1 -- 172.21.15.29:0/3944133609 >> 172.21.15.29:0/3944133609 conn(0x7fd1a40ce6a0 msgr2=0x7fd1a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.170+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fadcc004020 con 0x7fadd80c5770 2023-10-14T16:40:35.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.170+0000 7fd1ab10a640 1 -- 172.21.15.29:0/3944133609 shutdown_connections 2023-10-14T16:40:35.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.170+0000 7fd1ab10a640 1 -- 172.21.15.29:0/3944133609 wait complete. 2023-10-14T16:40:35.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.170+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c74005250 con 0x7f1c9006a2e0 2023-10-14T16:40:35.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.171+0000 7fd224a1f640 1 Processor -- start 2023-10-14T16:40:35.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.171+0000 7fadc9ffb640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 0x7fadbc06d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.172+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fadcc08a4d0 con 0x7fadd80c5770 2023-10-14T16:40:35.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.172+0000 7faddd030640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 0x7fadbc06d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.173+0000 7fd1ab10a640 1 Processor -- start 2023-10-14T16:40:35.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.173+0000 7faddd030640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 0x7fadbc06d400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000c00 tx=0x7fadd0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.171+0000 7fd224a1f640 1 -- start start 2023-10-14T16:40:35.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.173+0000 7fd1ab10a640 1 -- start start 2023-10-14T16:40:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.173+0000 7fd224a1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2200addb0 0x7fd2200caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.174+0000 7fd224a1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2200c9000 0x7fd220069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.174+0000 7fd224a1f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd22006a6f0 con 0x7fd22006a2e0 2023-10-14T16:40:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.174+0000 7fd224a1f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd220161a50 con 0x7fd2200c9000 2023-10-14T16:40:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.174+0000 7fd224a1f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2200cb140 con 0x7fd2200addb0 2023-10-14T16:40:35.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.174+0000 7fd21e59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2200addb0 0x7fd2200caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21dd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2200c9000 0x7fd220069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21ed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd22006a2e0 0x7fd2200cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:38124/0 (socket says 172.21.15.29:38124) 2023-10-14T16:40:35.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21ed9d640 1 -- 172.21.15.29:0/3042109471 learned_addr learned my addr 172.21.15.29:0/3042109471 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1a406f440 0x7fd1a40c4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40c9850 0x7fd1a4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40c2010 con 0x7fd1a40c9850 2023-10-14T16:40:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a4158cd0 con 0x7fd1a40ca340 2023-10-14T16:40:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1ab10a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1a40c6ed0 con 0x7fd1a406f440 2023-10-14T16:40:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1aa909640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1aa909640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35318/0 (socket says 172.21.15.29:35318) 2023-10-14T16:40:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1aa909640 1 -- 172.21.15.29:0/829536254 learned_addr learned my addr 172.21.15.29:0/829536254 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd1aa108640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1a406f440 0x7fd1a40c4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21dd9b640 1 -- 172.21.15.29:0/3042109471 >> v2:172.21.15.186:3300/0 conn(0x7fd2200addb0 msgr2=0x7fd2200caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21dd9b640 1 --2- 172.21.15.29:0/3042109471 >> v2:172.21.15.186:3300/0 conn(0x7fd2200addb0 0x7fd2200caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21dd9b640 1 -- 172.21.15.29:0/3042109471 >> v1:172.21.15.186:6789/0 conn(0x7fd22006a2e0 legacy=0x7fd2200cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.175+0000 7fd21dd9b640 1 -- 172.21.15.29:0/3042109471 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd22006b030 con 0x7fd2200c9000 2023-10-14T16:40:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.176+0000 7fd1aa909640 1 -- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1a406f440 msgr2=0x7fd1a40c4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.176+0000 7fd1aa909640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1a406f440 0x7fd1a40c4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.176+0000 7fd1aa909640 1 -- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40c9850 msgr2=0x7fd1a4158790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.176+0000 7fd1aa909640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40c9850 0x7fd1a4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.176+0000 7fd1aa909640 1 -- 172.21.15.29:0/829536254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19c0096b0 con 0x7fd1a40ca340 2023-10-14T16:40:35.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.177+0000 7fd1aa909640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009a50 tx=0x7fd19c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.177+0000 7fd21dd9b640 1 --2- 172.21.15.29:0/3042109471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2200c9000 0x7fd220069da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2200b15a0 tx=0x7fd21002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1a653bf56f2abce server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.178+0000 7fd21d59a640 1 -- 172.21.15.29:0/3042109471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210031640 con 0x7fd2200c9000 2023-10-14T16:40:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.178+0000 7fd21d59a640 1 -- 172.21.15.29:0/3042109471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2100317a0 con 0x7fd2200c9000 2023-10-14T16:40:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.178+0000 7fd21d59a640 1 -- 172.21.15.29:0/3042109471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd210031b70 con 0x7fd2200c9000 2023-10-14T16:40:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fadcc0556b0 con 0x7fadd80c5770 2023-10-14T16:40:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c03e660 con 0x7fd1a40ca340 2023-10-14T16:40:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd19c03e7e0 con 0x7fd1a40ca340 2023-10-14T16:40:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd19c03eb90 con 0x7fd1a40ca340 2023-10-14T16:40:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1a40c71b0 con 0x7fd1a40ca340 2023-10-14T16:40:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1a41ee350 con 0x7fd1a40ca340 2023-10-14T16:40:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd224a1f640 1 -- 172.21.15.29:0/3042109471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2200c9000 msgr2=0x7fd220069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7fd224a1f640 1 --2- 172.21.15.29:0/3042109471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2200c9000 0x7fd220069da0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd2200b15a0 tx=0x7fd21002b8f0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.179+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f357c0766c0 con 0x7f358416e4d0 2023-10-14T16:40:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 -- 172.21.15.29:0/3042109471 shutdown_connections 2023-10-14T16:40:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 --2- 172.21.15.29:0/3042109471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2200c9000 0x7fd220069da0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 --2- 172.21.15.29:0/3042109471 >> v2:172.21.15.186:3300/0 conn(0x7fd2200addb0 0x7fd2200caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 -- 172.21.15.29:0/3042109471 >> 172.21.15.29:0/3042109471 conn(0x7fd2200ce9b0 msgr2=0x7fd2200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 -- 172.21.15.29:0/3042109471 shutdown_connections 2023-10-14T16:40:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.180+0000 7fd224a1f640 1 -- 172.21.15.29:0/3042109471 wait complete. 2023-10-14T16:40:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.183+0000 7fd224a1f640 1 Processor -- start 2023-10-14T16:40:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.183+0000 7fd224a1f640 1 -- start start 2023-10-14T16:40:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.183+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1c80014c50 con 0x7f1c9006a2e0 2023-10-14T16:40:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd18c005250 con 0x7fd1a40ca340 2023-10-14T16:40:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2200addb0 0x7fd2201641f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2200c9000 0x7fd2200cc540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd220156390 con 0x7fd22006a2e0 2023-10-14T16:40:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2200c5df0 con 0x7fd2200c9000 2023-10-14T16:40:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.184+0000 7fd224a1f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2200c60d0 con 0x7fd2200addb0 2023-10-14T16:40:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.185+0000 7f1c877fe640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 0x7f1c7006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.185+0000 7fd21e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.185+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1c8008ae90 con 0x7f1c9006a2e0 2023-10-14T16:40:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.186+0000 7fd21e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36774/0 (socket says 172.21.15.29:36774) 2023-10-14T16:40:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.186+0000 7fd21e59c640 1 -- 172.21.15.29:0/1499544534 learned_addr learned my addr 172.21.15.29:0/1499544534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.186+0000 7fd21ed9d640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2200c9000 0x7fd2200cc540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.186+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 0x7f1c7006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.186+0000 7fd21dd9b640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2200addb0 0x7fd2201641f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7fd21e59c640 1 -- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2200addb0 msgr2=0x7fd2201641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7fd21e59c640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2200addb0 0x7fd2201641f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7fd21e59c640 1 -- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2200c9000 msgr2=0x7fd2200cc540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7fd21e59c640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2200c9000 0x7fd2200cc540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7fd21e59c640 1 -- 172.21.15.29:0/1499544534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd210009400 con 0x7fd22006a2e0 2023-10-14T16:40:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.187+0000 7f1c85d9b640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 0x7f1c7006d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c7803b000 tx=0x7f1c7800f040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.188+0000 7fd21e59c640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd20800e9b0 tx=0x7fd20800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.188+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208009670 con 0x7fd22006a2e0 2023-10-14T16:40:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.188+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd20800c9a0 con 0x7fd22006a2e0 2023-10-14T16:40:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.188+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd208009670 con 0x7fd22006a2e0 2023-10-14T16:40:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.188+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2200c63b0 con 0x7fd22006a2e0 2023-10-14T16:40:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.189+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2200c6970 con 0x7fd22006a2e0 2023-10-14T16:40:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.191+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd22004e160 con 0x7fd22006a2e0 2023-10-14T16:40:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.192+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd20800cb10 con 0x7fd22006a2e0 2023-10-14T16:40:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.193+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c80055f20 con 0x7f1c9006a2e0 2023-10-14T16:40:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.194+0000 7fd216ffd640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 0x7fd20006d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.195+0000 7fd21dd9b640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 0x7fd20006d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.196+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd19c03ecf0 con 0x7fd1a40ca340 2023-10-14T16:40:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.196+0000 7fd21dd9b640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 0x7fd20006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd210036000 tx=0x7fd21003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.197+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd208058220 con 0x7fd22006a2e0 2023-10-14T16:40:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.197+0000 7fd19affd640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 0x7fd18806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.197+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd19c0ace90 con 0x7fd1a40ca340 2023-10-14T16:40:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.198+0000 7fd1aa108640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 0x7fd18806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.200+0000 7fd1aa108640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 0x7fd18806d380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00011e0 tx=0x7fd1a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.206+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd208054cf0 con 0x7fd22006a2e0 2023-10-14T16:40:35.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.207+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd19c077fe0 con 0x7fd1a40ca340 2023-10-14T16:40:35.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.477+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f13600057d0 con 0x7f1378005fe0 2023-10-14T16:40:35.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.479+0000 7f1375ffb640 1 -- 172.21.15.29:0/2382627201 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1370057f90 con 0x7f1378005fe0 2023-10-14T16:40:35.479 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.482+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 msgr2=0x7f136406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.482+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 0x7f136406d330 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c008040 comp rx=0 tx=0).stop 2023-10-14T16:40:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.482+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 msgr2=0x7f137804ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.482+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1370006f40 tx=0x7f137000dcb0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 shutdown_connections 2023-10-14T16:40:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f136406ae70 0x7f136406d330 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13780df500 0x7f137816ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f137804fc70 0x7f1378184d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 --2- 172.21.15.29:0/2382627201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1378005fe0 0x7f137804ef20 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.483+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 >> 172.21.15.29:0/2382627201 conn(0x7f1378054c10 msgr2=0x7f13780382d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.484+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 shutdown_connections 2023-10-14T16:40:35.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.484+0000 7f13865bf640 1 -- 172.21.15.29:0/2382627201 wait complete. 2023-10-14T16:40:35.512 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2023-10-14T16:40:35.512 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.546+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fcbf40057d0 con 0x7fcc14038180 2023-10-14T16:40:35.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.547+0000 7fcc0affd640 1 -- 172.21.15.29:0/1118158411 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fcc10077850 con 0x7fcc14038180 2023-10-14T16:40:35.548 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 msgr2=0x7fcbfc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 0x7fcbfc06d330 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fcc1404f0e0 tx=0x7fcc0c007e30 comp rx=0 tx=0).stop 2023-10-14T16:40:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 msgr2=0x7fcc140dbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fcc100090b0 tx=0x7fcc100046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 shutdown_connections 2023-10-14T16:40:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcbfc06ae70 0x7fcbfc06d330 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc140e3190 0x7fcc14052890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc140544a0 0x7fcc1400d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 --2- 172.21.15.29:0/1118158411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc14038180 0x7fcc140dbb40 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.550+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 >> 172.21.15.29:0/1118158411 conn(0x7fcc140590c0 msgr2=0x7fcc140539b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.551+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 shutdown_connections 2023-10-14T16:40:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.551+0000 7fcc08ff9640 1 -- 172.21.15.29:0/1118158411 wait complete. 2023-10-14T16:40:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.559+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f1c740057d0 con 0x7f1c9006a2e0 2023-10-14T16:40:35.560 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.559+0000 7f1c877fe640 1 -- 172.21.15.29:0/3569594561 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f1c80059540 con 0x7f1c9006a2e0 2023-10-14T16:40:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 msgr2=0x7f1c7006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 0x7f1c7006d330 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1c7803b000 tx=0x7f1c7800f040 comp rx=0 tx=0).stop 2023-10-14T16:40:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 msgr2=0x7f1c90070a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1c80007ef0 tx=0x7f1c8000e4b0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 shutdown_connections 2023-10-14T16:40:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1c7006ae70 0x7f1c7006d330 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c90158f60 0x7f1c900c8560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c900ce1c0 0x7f1c90070fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 --2- 172.21.15.29:0/3569594561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c9006a2e0 0x7f1c90070a90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 >> 172.21.15.29:0/3569594561 conn(0x7f1c900ce9b0 msgr2=0x7f1c900cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.562+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 shutdown_connections 2023-10-14T16:40:35.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.563+0000 7f1c94997640 1 -- 172.21.15.29:0/3569594561 wait complete. 2023-10-14T16:40:35.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.565+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd7cc0ca660 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.566+0000 7fd7b37fe640 1 -- 172.21.15.29:0/2162998703 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd7bc02f090 con 0x7fd7cc06c1f0 2023-10-14T16:40:35.567 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.569+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 msgr2=0x7fd7b806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.569+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 0x7fd7b806d380 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd7cc0ae730 tx=0x7fd7c0008040 comp rx=0 tx=0).stop 2023-10-14T16:40:35.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.569+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 msgr2=0x7fd7cc0c85b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.569+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc0090b0 tx=0x7fd7bc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 shutdown_connections 2023-10-14T16:40:35.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd7b806aec0 0x7fd7b806d380 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7cc15e2d0 0x7fd7cc1f9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7cc0b1d40 0x7fd7cc0c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 --2- 172.21.15.29:0/2162998703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7cc06c1f0 0x7fd7cc0c85b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 >> 172.21.15.29:0/2162998703 conn(0x7fd7cc0ce9b0 msgr2=0x7fd7cc0b2330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 shutdown_connections 2023-10-14T16:40:35.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.570+0000 7fd7d147c640 1 -- 172.21.15.29:0/2162998703 wait complete. 2023-10-14T16:40:35.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2023-10-14T16:40:35.576 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.581 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2023-10-14T16:40:35.581 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.589 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.0 2023-10-14T16:40:35.590 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.596+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f358404e160 con 0x7f358416e4d0 2023-10-14T16:40:35.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.598+0000 7f35817fa640 1 -- 172.21.15.29:0/600577732 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f357c079ce0 con 0x7f358416e4d0 2023-10-14T16:40:35.598 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 msgr2=0x7f357006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 0x7f357006d3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3574001eb0 tx=0x7f3574001940 comp rx=0 tx=0).stop 2023-10-14T16:40:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 msgr2=0x7f35840c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 0x7f35840c5fc0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f357c033000 tx=0x7f357c004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 shutdown_connections 2023-10-14T16:40:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f357006af00 0x7f357006d3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358416e4d0 0x7f35840c5fc0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35840cc9e0 0x7f35840c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 --2- 172.21.15.29:0/600577732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35840cc580 0x7f358417d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 >> 172.21.15.29:0/600577732 conn(0x7f35840ce9b0 msgr2=0x7f35840c4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 shutdown_connections 2023-10-14T16:40:35.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.601+0000 7f356affd640 1 -- 172.21.15.29:0/600577732 wait complete. 2023-10-14T16:40:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.617+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fadb0005500 con 0x7fadd80c5770 2023-10-14T16:40:35.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.618+0000 7fadc9ffb640 1 -- 172.21.15.29:0/135856740 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fadcc058cd0 con 0x7fadd80c5770 2023-10-14T16:40:35.619 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 msgr2=0x7fadbc06d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 0x7fadbc06d400 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000c00 tx=0x7fadd0030000 comp rx=0 tx=0).stop 2023-10-14T16:40:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 msgr2=0x7fadd8152bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 0x7fadd8152bd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fadcc011e30 tx=0x7fadcc00f580 comp rx=0 tx=0).stop 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 shutdown_connections 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fadbc06af40 0x7fadbc06d400 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fadd8153110 0x7fadd80a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fadd80c5770 0x7fadd8152bd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 --2- 172.21.15.29:0/135856740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fadd806f440 0x7fadd8151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 >> 172.21.15.29:0/135856740 conn(0x7fadd80ce6a0 msgr2=0x7fadd80c41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 shutdown_connections 2023-10-14T16:40:35.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.622+0000 7fadc37fe640 1 -- 172.21.15.29:0/135856740 wait complete. 2023-10-14T16:40:35.626 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2023-10-14T16:40:35.627 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.631+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fd22006c5b0 con 0x7fd22006a2e0 2023-10-14T16:40:35.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.632+0000 7fd216ffd640 1 -- 172.21.15.29:0/1499544534 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd208021070 con 0x7fd22006a2e0 2023-10-14T16:40:35.633 INFO:teuthology.orchestra.run.smithi029.stdout:55834574854 2023-10-14T16:40:35.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 msgr2=0x7fd20006d390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 0x7fd20006d390 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd210036000 tx=0x7fd21003c040 comp rx=0 tx=0).stop 2023-10-14T16:40:35.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 msgr2=0x7fd220155e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd20800e9b0 tx=0x7fd20800ee70 comp rx=0 tx=0).stop 2023-10-14T16:40:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 shutdown_connections 2023-10-14T16:40:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd20006aed0 0x7fd20006d390 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2200c9000 0x7fd2200cc540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2200addb0 0x7fd2201641f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 --2- 172.21.15.29:0/1499544534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd22006a2e0 0x7fd220155e50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 >> 172.21.15.29:0/1499544534 conn(0x7fd2200ce9b0 msgr2=0x7fd2200ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 shutdown_connections 2023-10-14T16:40:35.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.634+0000 7fd224a1f640 1 -- 172.21.15.29:0/1499544534 wait complete. 2023-10-14T16:40:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.643+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7fd18c0057d0 con 0x7fd1a40ca340 2023-10-14T16:40:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.644+0000 7fd19affd640 1 -- 172.21.15.29:0/829536254 <== mon.1 v2:172.21.15.186:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd19c07b600 con 0x7fd1a40ca340 2023-10-14T16:40:35.645 INFO:teuthology.orchestra.run.smithi029.stdout:55834574853 2023-10-14T16:40:35.646 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2023-10-14T16:40:35.647 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 msgr2=0x7fd18806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 0x7fd18806d380 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd1a00011e0 tx=0x7fd1a0009040 comp rx=0 tx=0).stop 2023-10-14T16:40:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 msgr2=0x7fd1a40c3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009a50 tx=0x7fd19c004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 shutdown_connections 2023-10-14T16:40:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd18806aec0 0x7fd18806d380 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1a40ca340 0x7fd1a40c3d60 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1a40c9850 0x7fd1a4158790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 --2- 172.21.15.29:0/829536254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1a406f440 0x7fd1a40c4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.646+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 >> 172.21.15.29:0/829536254 conn(0x7fd1a40ce6a0 msgr2=0x7fd1a40af930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.647+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 shutdown_connections 2023-10-14T16:40:35.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.647+0000 7fd1ab10a640 1 -- 172.21.15.29:0/829536254 wait complete. 2023-10-14T16:40:35.651 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.7 2023-10-14T16:40:35.651 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.663 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.6 2023-10-14T16:40:35.663 DEBUG:teuthology.parallel:result is None 2023-10-14T16:40:35.663 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T16:40:35.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:40:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.923+0000 7ff742440640 1 Processor -- start 2023-10-14T16:40:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.923+0000 7ff742440640 1 -- start start 2023-10-14T16:40:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.924+0000 7ff742440640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 0x7ff73c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff742440640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff73c0c9230 0x7ff73c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff742440640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 0x7ff73c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff742440640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c0cdec0 con 0x7ff73c0c9230 2023-10-14T16:40:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff742440640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff73c160a90 con 0x7ff73c15c9f0 2023-10-14T16:40:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff742440640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0cd900 con 0x7ff73c070f20 2023-10-14T16:40:35.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff73c0c9230 0x7ff73c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff73c0c9230 0x7ff73c0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36786/0 (socket says 172.21.15.29:36786) 2023-10-14T16:40:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 -- 172.21.15.29:0/2056950603 learned_addr learned my addr 172.21.15.29:0/2056950603 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff74143e640 1 --2- 172.21.15.29:0/2056950603 >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 0x7ff73c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff741c3f640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 0x7ff73c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 -- 172.21.15.29:0/2056950603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 msgr2=0x7ff73c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 0x7ff73c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 -- 172.21.15.29:0/2056950603 >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 msgr2=0x7ff73c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 --2- 172.21.15.29:0/2056950603 >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 0x7ff73c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff740c3d640 1 -- 172.21.15.29:0/2056950603 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73c0cdbe0 con 0x7ff73c0c9230 2023-10-14T16:40:35.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff741c3f640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 0x7ff73c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:35.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.925+0000 7ff74143e640 1 --2- 172.21.15.29:0/2056950603 >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 0x7ff73c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.927+0000 7ff740c3d640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c0c9230 0x7ff73c0709e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0ad4e0 tx=0x7ff73002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4892628fe2c17eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.927+0000 7ff72ffff640 1 -- 172.21.15.29:0/2056950603 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730031880 con 0x7ff73c0c9230 2023-10-14T16:40:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.927+0000 7ff72ffff640 1 -- 172.21.15.29:0/2056950603 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7300319e0 con 0x7ff73c0c9230 2023-10-14T16:40:35.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.927+0000 7ff72ffff640 1 -- 172.21.15.29:0/2056950603 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730031d70 con 0x7ff73c0c9230 2023-10-14T16:40:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 -- 172.21.15.29:0/2056950603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c0c9230 msgr2=0x7ff73c0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c0c9230 0x7ff73c0709e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff73c0ad4e0 tx=0x7ff73002b920 comp rx=0 tx=0).stop 2023-10-14T16:40:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 -- 172.21.15.29:0/2056950603 shutdown_connections 2023-10-14T16:40:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c070f20 0x7ff73c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 --2- 172.21.15.29:0/2056950603 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c0c9230 0x7ff73c0709e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 --2- 172.21.15.29:0/2056950603 >> v2:172.21.15.186:3300/0 conn(0x7ff73c15c9f0 0x7ff73c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 -- 172.21.15.29:0/2056950603 >> 172.21.15.29:0/2056950603 conn(0x7ff73c0ce6a0 msgr2=0x7ff73c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 -- 172.21.15.29:0/2056950603 shutdown_connections 2023-10-14T16:40:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.928+0000 7ff742440640 1 -- 172.21.15.29:0/2056950603 wait complete. 2023-10-14T16:40:35.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.929+0000 7ff742440640 1 Processor -- start 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.930+0000 7ff742440640 1 -- start start 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.930+0000 7ff742440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff742440640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c0c9230 0x7ff73c06c490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff742440640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff73c15c9f0 0x7ff73c0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff742440640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c06cce0 con 0x7ff73c070f20 2023-10-14T16:40:35.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff742440640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0c3850 con 0x7ff73c15c9f0 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff742440640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff73c0c3ad0 con 0x7ff73c0c9230 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36798/0 (socket says 172.21.15.29:36798) 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 -- 172.21.15.29:0/3452879339 learned_addr learned my addr 172.21.15.29:0/3452879339 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 -- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c0c9230 msgr2=0x7ff73c06c490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c0c9230 0x7ff73c06c490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 -- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff73c15c9f0 msgr2=0x7ff73c0c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff73c15c9f0 0x7ff73c0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:35.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 -- 172.21.15.29:0/3452879339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7300094d0 con 0x7ff73c070f20 2023-10-14T16:40:35.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.931+0000 7ff74143e640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff73800c7f0 tx=0x7ff73800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.932+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738007820 con 0x7ff73c070f20 2023-10-14T16:40:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.932+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff738007980 con 0x7ff73c070f20 2023-10-14T16:40:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.932+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff738007cf0 con 0x7ff73c070f20 2023-10-14T16:40:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.932+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73c1571e0 con 0x7ff73c070f20 2023-10-14T16:40:35.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.932+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73c0c4050 con 0x7ff73c070f20 2023-10-14T16:40:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.933+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff73c04e9b0 con 0x7ff73c070f20 2023-10-14T16:40:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.934+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff7380113f0 con 0x7ff73c070f20 2023-10-14T16:40:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.935+0000 7ff72dffb640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 0x7ff72006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.935+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff7380898b0 con 0x7ff73c070f20 2023-10-14T16:40:35.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.935+0000 7ff740c3d640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 0x7ff72006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.936+0000 7ff740c3d640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 0x7ff72006d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff730036000 tx=0x7ff730030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:35.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:35.939+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff738054950 con 0x7ff73c070f20 2023-10-14T16:40:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.123+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff73c0704c0 con 0x7ff72006ae70 2023-10-14T16:40:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.146+0000 7ff72dffb640 1 -- 172.21.15.29:0/3452879339 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264439 (secure 0 0 0) 0x7ff730031c50 con 0x7ff72006ae70 2023-10-14T16:40:36.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:40:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 msgr2=0x7ff72006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 0x7ff72006d330 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7ff730036000 tx=0x7ff730030000 comp rx=0 tx=0).stop 2023-10-14T16:40:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 msgr2=0x7ff73c156ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff73800c7f0 tx=0x7ff73800ccb0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 shutdown_connections 2023-10-14T16:40:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff72006ae70 0x7ff72006d330 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff73c15c9f0 0x7ff73c0c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff73c0c9230 0x7ff73c06c490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 --2- 172.21.15.29:0/3452879339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff73c070f20 0x7ff73c156ca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 >> 172.21.15.29:0/3452879339 conn(0x7ff73c0ce6a0 msgr2=0x7ff73c15ba20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:36.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 shutdown_connections 2023-10-14T16:40:36.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.152+0000 7ff742440640 1 -- 172.21.15.29:0/3452879339 wait complete. 2023-10-14T16:40:36.172 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-14T16:40:35.301882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235628,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739092,"statfs":{"total":773094113280,"available":772852830208,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.560837"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083981+0000","last_change":"2023-10-14T16:40:25.778039+0000","last_active":"2023-10-14T16:40:29.083981+0000","last_peered":"2023-10-14T16:40:29.083981+0000","last_clean":"2023-10-14T16:40:29.083981+0000","last_became_active":"2023-10-14T16:40:25.777745+0000","last_became_peered":"2023-10-14T16:40:25.777745+0000","last_unstale":"2023-10-14T16:40:29.083981+0000","last_undegraded":"2023-10-14T16:40:29.083981+0000","last_fullsized":"2023-10-14T16:40:29.083981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:21.372595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.121147+0000","last_change":"2023-10-14T16:40:25.782038+0000","last_active":"2023-10-14T16:40:30.121147+0000","last_peered":"2023-10-14T16:40:30.121147+0000","last_clean":"2023-10-14T16:40:30.121147+0000","last_became_active":"2023-10-14T16:40:25.781722+0000","last_became_peered":"2023-10-14T16:40:25.781722+0000","last_unstale":"2023-10-14T16:40:30.121147+0000","last_undegraded":"2023-10-14T16:40:30.121147+0000","last_fullsized":"2023-10-14T16:40:30.121147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122914+0000","last_change":"2023-10-14T16:40:25.757114+0000","last_active":"2023-10-14T16:40:30.122914+0000","last_peered":"2023-10-14T16:40:30.122914+0000","last_clean":"2023-10-14T16:40:30.122914+0000","last_became_active":"2023-10-14T16:40:25.756793+0000","last_became_peered":"2023-10-14T16:40:25.756793+0000","last_unstale":"2023-10-14T16:40:30.122914+0000","last_undegraded":"2023-10-14T16:40:30.122914+0000","last_fullsized":"2023-10-14T16:40:30.122914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484919+0000","last_change":"2023-10-14T16:40:25.788571+0000","last_active":"2023-10-14T16:40:29.484919+0000","last_peered":"2023-10-14T16:40:29.484919+0000","last_clean":"2023-10-14T16:40:29.484919+0000","last_became_active":"2023-10-14T16:40:25.788388+0000","last_became_peered":"2023-10-14T16:40:25.788388+0000","last_unstale":"2023-10-14T16:40:29.484919+0000","last_undegraded":"2023-10-14T16:40:29.484919+0000","last_fullsized":"2023-10-14T16:40:29.484919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083267+0000","last_change":"2023-10-14T16:40:25.781929+0000","last_active":"2023-10-14T16:40:29.083267+0000","last_peered":"2023-10-14T16:40:29.083267+0000","last_clean":"2023-10-14T16:40:29.083267+0000","last_became_active":"2023-10-14T16:40:25.774025+0000","last_became_peered":"2023-10-14T16:40:25.774025+0000","last_unstale":"2023-10-14T16:40:29.083267+0000","last_undegraded":"2023-10-14T16:40:29.083267+0000","last_fullsized":"2023-10-14T16:40:29.083267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484458+0000","last_change":"2023-10-14T16:40:25.785668+0000","last_active":"2023-10-14T16:40:29.484458+0000","last_peered":"2023-10-14T16:40:29.484458+0000","last_clean":"2023-10-14T16:40:29.484458+0000","last_became_active":"2023-10-14T16:40:25.785300+0000","last_became_peered":"2023-10-14T16:40:25.785300+0000","last_unstale":"2023-10-14T16:40:29.484458+0000","last_undegraded":"2023-10-14T16:40:29.484458+0000","last_fullsized":"2023-10-14T16:40:29.484458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083019+0000","last_change":"2023-10-14T16:40:25.786146+0000","last_active":"2023-10-14T16:40:29.083019+0000","last_peered":"2023-10-14T16:40:29.083019+0000","last_clean":"2023-10-14T16:40:29.083019+0000","last_became_active":"2023-10-14T16:40:25.785748+0000","last_became_peered":"2023-10-14T16:40:25.785748+0000","last_unstale":"2023-10-14T16:40:29.083019+0000","last_undegraded":"2023-10-14T16:40:29.083019+0000","last_fullsized":"2023-10-14T16:40:29.083019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774509+0000","last_change":"2023-10-14T16:40:25.783832+0000","last_active":"2023-10-14T16:40:26.774509+0000","last_peered":"2023-10-14T16:40:26.774509+0000","last_clean":"2023-10-14T16:40:26.774509+0000","last_became_active":"2023-10-14T16:40:25.781635+0000","last_became_peered":"2023-10-14T16:40:25.781635+0000","last_unstale":"2023-10-14T16:40:26.774509+0000","last_undegraded":"2023-10-14T16:40:26.774509+0000","last_fullsized":"2023-10-14T16:40:26.774509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084293+0000","last_change":"2023-10-14T16:40:25.791696+0000","last_active":"2023-10-14T16:40:29.084293+0000","last_peered":"2023-10-14T16:40:29.084293+0000","last_clean":"2023-10-14T16:40:29.084293+0000","last_became_active":"2023-10-14T16:40:25.791413+0000","last_became_peered":"2023-10-14T16:40:25.791413+0000","last_unstale":"2023-10-14T16:40:29.084293+0000","last_undegraded":"2023-10-14T16:40:29.084293+0000","last_fullsized":"2023-10-14T16:40:29.084293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:31.604878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781741+0000","last_change":"2023-10-14T16:40:25.786842+0000","last_active":"2023-10-14T16:40:26.781741+0000","last_peered":"2023-10-14T16:40:26.781741+0000","last_clean":"2023-10-14T16:40:26.781741+0000","last_became_active":"2023-10-14T16:40:25.786528+0000","last_became_peered":"2023-10-14T16:40:25.786528+0000","last_unstale":"2023-10-14T16:40:26.781741+0000","last_undegraded":"2023-10-14T16:40:26.781741+0000","last_fullsized":"2023-10-14T16:40:26.781741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780857+0000","last_change":"2023-10-14T16:40:25.788130+0000","last_active":"2023-10-14T16:40:26.780857+0000","last_peered":"2023-10-14T16:40:26.780857+0000","last_clean":"2023-10-14T16:40:26.780857+0000","last_became_active":"2023-10-14T16:40:25.787596+0000","last_became_peered":"2023-10-14T16:40:25.787596+0000","last_unstale":"2023-10-14T16:40:26.780857+0000","last_undegraded":"2023-10-14T16:40:26.780857+0000","last_fullsized":"2023-10-14T16:40:26.780857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778866+0000","last_change":"2023-10-14T16:40:25.765437+0000","last_active":"2023-10-14T16:40:26.778866+0000","last_peered":"2023-10-14T16:40:26.778866+0000","last_clean":"2023-10-14T16:40:26.778866+0000","last_became_active":"2023-10-14T16:40:25.765116+0000","last_became_peered":"2023-10-14T16:40:25.765116+0000","last_unstale":"2023-10-14T16:40:26.778866+0000","last_undegraded":"2023-10-14T16:40:26.778866+0000","last_fullsized":"2023-10-14T16:40:26.778866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774963+0000","last_change":"2023-10-14T16:40:25.784431+0000","last_active":"2023-10-14T16:40:26.774963+0000","last_peered":"2023-10-14T16:40:26.774963+0000","last_clean":"2023-10-14T16:40:26.774963+0000","last_became_active":"2023-10-14T16:40:25.775585+0000","last_became_peered":"2023-10-14T16:40:25.775585+0000","last_unstale":"2023-10-14T16:40:26.774963+0000","last_undegraded":"2023-10-14T16:40:26.774963+0000","last_fullsized":"2023-10-14T16:40:26.774963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122194+0000","last_change":"2023-10-14T16:40:25.761749+0000","last_active":"2023-10-14T16:40:30.122194+0000","last_peered":"2023-10-14T16:40:30.122194+0000","last_clean":"2023-10-14T16:40:30.122194+0000","last_became_active":"2023-10-14T16:40:25.761444+0000","last_became_peered":"2023-10-14T16:40:25.761444+0000","last_unstale":"2023-10-14T16:40:30.122194+0000","last_undegraded":"2023-10-14T16:40:30.122194+0000","last_fullsized":"2023-10-14T16:40:30.122194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120661+0000","last_change":"2023-10-14T16:40:25.776450+0000","last_active":"2023-10-14T16:40:30.120661+0000","last_peered":"2023-10-14T16:40:30.120661+0000","last_clean":"2023-10-14T16:40:30.120661+0000","last_became_active":"2023-10-14T16:40:25.776153+0000","last_became_peered":"2023-10-14T16:40:25.776153+0000","last_unstale":"2023-10-14T16:40:30.120661+0000","last_undegraded":"2023-10-14T16:40:30.120661+0000","last_fullsized":"2023-10-14T16:40:30.120661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484358+0000","last_change":"2023-10-14T16:40:25.786126+0000","last_active":"2023-10-14T16:40:29.484358+0000","last_peered":"2023-10-14T16:40:29.484358+0000","last_clean":"2023-10-14T16:40:29.484358+0000","last_became_active":"2023-10-14T16:40:25.785701+0000","last_became_peered":"2023-10-14T16:40:25.785701+0000","last_unstale":"2023-10-14T16:40:29.484358+0000","last_undegraded":"2023-10-14T16:40:29.484358+0000","last_fullsized":"2023-10-14T16:40:29.484358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.486352+0000","last_change":"2023-10-14T16:40:25.792223+0000","last_active":"2023-10-14T16:40:29.486352+0000","last_peered":"2023-10-14T16:40:29.486352+0000","last_clean":"2023-10-14T16:40:29.486352+0000","last_became_active":"2023-10-14T16:40:25.791860+0000","last_became_peered":"2023-10-14T16:40:25.791860+0000","last_unstale":"2023-10-14T16:40:29.486352+0000","last_undegraded":"2023-10-14T16:40:29.486352+0000","last_fullsized":"2023-10-14T16:40:29.486352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.760301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776199+0000","last_change":"2023-10-14T16:40:25.766839+0000","last_active":"2023-10-14T16:40:27.776199+0000","last_peered":"2023-10-14T16:40:27.776199+0000","last_clean":"2023-10-14T16:40:27.776199+0000","last_became_active":"2023-10-14T16:40:25.766493+0000","last_became_peered":"2023-10-14T16:40:25.766493+0000","last_unstale":"2023-10-14T16:40:27.776199+0000","last_undegraded":"2023-10-14T16:40:27.776199+0000","last_fullsized":"2023-10-14T16:40:27.776199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084022+0000","last_change":"2023-10-14T16:40:25.792353+0000","last_active":"2023-10-14T16:40:29.084022+0000","last_peered":"2023-10-14T16:40:29.084022+0000","last_clean":"2023-10-14T16:40:29.084022+0000","last_became_active":"2023-10-14T16:40:25.792040+0000","last_became_peered":"2023-10-14T16:40:25.792040+0000","last_unstale":"2023-10-14T16:40:29.084022+0000","last_undegraded":"2023-10-14T16:40:29.084022+0000","last_fullsized":"2023-10-14T16:40:29.084022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:25.312672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781176+0000","last_change":"2023-10-14T16:40:25.770102+0000","last_active":"2023-10-14T16:40:26.781176+0000","last_peered":"2023-10-14T16:40:26.781176+0000","last_clean":"2023-10-14T16:40:26.781176+0000","last_became_active":"2023-10-14T16:40:25.769804+0000","last_became_peered":"2023-10-14T16:40:25.769804+0000","last_unstale":"2023-10-14T16:40:26.781176+0000","last_undegraded":"2023-10-14T16:40:26.781176+0000","last_fullsized":"2023-10-14T16:40:26.781176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123282+0000","last_change":"2023-10-14T16:40:25.779852+0000","last_active":"2023-10-14T16:40:30.123282+0000","last_peered":"2023-10-14T16:40:30.123282+0000","last_clean":"2023-10-14T16:40:30.123282+0000","last_became_active":"2023-10-14T16:40:25.779362+0000","last_became_peered":"2023-10-14T16:40:25.779362+0000","last_unstale":"2023-10-14T16:40:30.123282+0000","last_undegraded":"2023-10-14T16:40:30.123282+0000","last_fullsized":"2023-10-14T16:40:30.123282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484764+0000","last_change":"2023-10-14T16:40:25.781280+0000","last_active":"2023-10-14T16:40:29.484764+0000","last_peered":"2023-10-14T16:40:29.484764+0000","last_clean":"2023-10-14T16:40:29.484764+0000","last_became_active":"2023-10-14T16:40:25.780595+0000","last_became_peered":"2023-10-14T16:40:25.780595+0000","last_unstale":"2023-10-14T16:40:29.484764+0000","last_undegraded":"2023-10-14T16:40:29.484764+0000","last_fullsized":"2023-10-14T16:40:29.484764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774501+0000","last_change":"2023-10-14T16:40:25.777771+0000","last_active":"2023-10-14T16:40:26.774501+0000","last_peered":"2023-10-14T16:40:26.774501+0000","last_clean":"2023-10-14T16:40:26.774501+0000","last_became_active":"2023-10-14T16:40:25.777508+0000","last_became_peered":"2023-10-14T16:40:25.777508+0000","last_unstale":"2023-10-14T16:40:26.774501+0000","last_undegraded":"2023-10-14T16:40:26.774501+0000","last_fullsized":"2023-10-14T16:40:26.774501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484634+0000","last_change":"2023-10-14T16:40:25.772029+0000","last_active":"2023-10-14T16:40:29.484634+0000","last_peered":"2023-10-14T16:40:29.484634+0000","last_clean":"2023-10-14T16:40:29.484634+0000","last_became_active":"2023-10-14T16:40:25.771700+0000","last_became_peered":"2023-10-14T16:40:25.771700+0000","last_unstale":"2023-10-14T16:40:29.484634+0000","last_undegraded":"2023-10-14T16:40:29.484634+0000","last_fullsized":"2023-10-14T16:40:29.484634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775784+0000","last_change":"2023-10-14T16:40:25.777077+0000","last_active":"2023-10-14T16:40:26.775784+0000","last_peered":"2023-10-14T16:40:26.775784+0000","last_clean":"2023-10-14T16:40:26.775784+0000","last_became_active":"2023-10-14T16:40:25.776755+0000","last_became_peered":"2023-10-14T16:40:25.776755+0000","last_unstale":"2023-10-14T16:40:26.775784+0000","last_undegraded":"2023-10-14T16:40:26.775784+0000","last_fullsized":"2023-10-14T16:40:26.775784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781826+0000","last_change":"2023-10-14T16:40:25.777054+0000","last_active":"2023-10-14T16:40:26.781826+0000","last_peered":"2023-10-14T16:40:26.781826+0000","last_clean":"2023-10-14T16:40:26.781826+0000","last_became_active":"2023-10-14T16:40:25.773588+0000","last_became_peered":"2023-10-14T16:40:25.773588+0000","last_unstale":"2023-10-14T16:40:26.781826+0000","last_undegraded":"2023-10-14T16:40:26.781826+0000","last_fullsized":"2023-10-14T16:40:26.781826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120557+0000","last_change":"2023-10-14T16:40:25.787594+0000","last_active":"2023-10-14T16:40:30.120557+0000","last_peered":"2023-10-14T16:40:30.120557+0000","last_clean":"2023-10-14T16:40:30.120557+0000","last_became_active":"2023-10-14T16:40:25.786833+0000","last_became_peered":"2023-10-14T16:40:25.786833+0000","last_unstale":"2023-10-14T16:40:30.120557+0000","last_undegraded":"2023-10-14T16:40:30.120557+0000","last_fullsized":"2023-10-14T16:40:30.120557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775301+0000","last_change":"2023-10-14T16:40:25.782397+0000","last_active":"2023-10-14T16:40:26.775301+0000","last_peered":"2023-10-14T16:40:26.775301+0000","last_clean":"2023-10-14T16:40:26.775301+0000","last_became_active":"2023-10-14T16:40:25.782109+0000","last_became_peered":"2023-10-14T16:40:25.782109+0000","last_unstale":"2023-10-14T16:40:26.775301+0000","last_undegraded":"2023-10-14T16:40:26.775301+0000","last_fullsized":"2023-10-14T16:40:26.775301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123515+0000","last_change":"2023-10-14T16:40:25.783124+0000","last_active":"2023-10-14T16:40:30.123515+0000","last_peered":"2023-10-14T16:40:30.123515+0000","last_clean":"2023-10-14T16:40:30.123515+0000","last_became_active":"2023-10-14T16:40:25.782805+0000","last_became_peered":"2023-10-14T16:40:25.782805+0000","last_unstale":"2023-10-14T16:40:30.123515+0000","last_undegraded":"2023-10-14T16:40:30.123515+0000","last_fullsized":"2023-10-14T16:40:30.123515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484995+0000","last_change":"2023-10-14T16:40:25.784068+0000","last_active":"2023-10-14T16:40:29.484995+0000","last_peered":"2023-10-14T16:40:29.484995+0000","last_clean":"2023-10-14T16:40:29.484995+0000","last_became_active":"2023-10-14T16:40:25.783671+0000","last_became_peered":"2023-10-14T16:40:25.783671+0000","last_unstale":"2023-10-14T16:40:29.484995+0000","last_undegraded":"2023-10-14T16:40:29.484995+0000","last_fullsized":"2023-10-14T16:40:29.484995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:56.842342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775610+0000","last_change":"2023-10-14T16:40:25.777635+0000","last_active":"2023-10-14T16:40:27.775610+0000","last_peered":"2023-10-14T16:40:27.775610+0000","last_clean":"2023-10-14T16:40:27.775610+0000","last_became_active":"2023-10-14T16:40:25.777400+0000","last_became_peered":"2023-10-14T16:40:25.777400+0000","last_unstale":"2023-10-14T16:40:27.775610+0000","last_undegraded":"2023-10-14T16:40:27.775610+0000","last_fullsized":"2023-10-14T16:40:27.775610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779941+0000","last_change":"2023-10-14T16:40:25.758613+0000","last_active":"2023-10-14T16:40:26.779941+0000","last_peered":"2023-10-14T16:40:26.779941+0000","last_clean":"2023-10-14T16:40:26.779941+0000","last_became_active":"2023-10-14T16:40:25.758173+0000","last_became_peered":"2023-10-14T16:40:25.758173+0000","last_unstale":"2023-10-14T16:40:26.779941+0000","last_undegraded":"2023-10-14T16:40:26.779941+0000","last_fullsized":"2023-10-14T16:40:26.779941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778458+0000","last_change":"2023-10-14T16:40:25.759820+0000","last_active":"2023-10-14T16:40:26.778458+0000","last_peered":"2023-10-14T16:40:26.778458+0000","last_clean":"2023-10-14T16:40:26.778458+0000","last_became_active":"2023-10-14T16:40:25.759498+0000","last_became_peered":"2023-10-14T16:40:25.759498+0000","last_unstale":"2023-10-14T16:40:26.778458+0000","last_undegraded":"2023-10-14T16:40:26.778458+0000","last_fullsized":"2023-10-14T16:40:26.778458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775078+0000","last_change":"2023-10-14T16:40:24.768741+0000","last_active":"2023-10-14T16:40:26.775078+0000","last_peered":"2023-10-14T16:40:26.775078+0000","last_clean":"2023-10-14T16:40:26.775078+0000","last_became_active":"2023-10-14T16:40:24.768379+0000","last_became_peered":"2023-10-14T16:40:24.768379+0000","last_unstale":"2023-10-14T16:40:26.775078+0000","last_undegraded":"2023-10-14T16:40:26.775078+0000","last_fullsized":"2023-10-14T16:40:26.775078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123684+0000","last_change":"2023-10-14T16:40:25.779646+0000","last_active":"2023-10-14T16:40:30.123684+0000","last_peered":"2023-10-14T16:40:30.123684+0000","last_clean":"2023-10-14T16:40:30.123684+0000","last_became_active":"2023-10-14T16:40:25.779001+0000","last_became_peered":"2023-10-14T16:40:25.779001+0000","last_unstale":"2023-10-14T16:40:30.123684+0000","last_undegraded":"2023-10-14T16:40:30.123684+0000","last_fullsized":"2023-10-14T16:40:30.123684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780736+0000","last_change":"2023-10-14T16:40:24.763894+0000","last_active":"2023-10-14T16:40:26.780736+0000","last_peered":"2023-10-14T16:40:26.780736+0000","last_clean":"2023-10-14T16:40:26.780736+0000","last_became_active":"2023-10-14T16:40:24.762343+0000","last_became_peered":"2023-10-14T16:40:24.762343+0000","last_unstale":"2023-10-14T16:40:26.780736+0000","last_undegraded":"2023-10-14T16:40:26.780736+0000","last_fullsized":"2023-10-14T16:40:26.780736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774426+0000","last_change":"2023-10-14T16:40:24.775055+0000","last_active":"2023-10-14T16:40:26.774426+0000","last_peered":"2023-10-14T16:40:26.774426+0000","last_clean":"2023-10-14T16:40:26.774426+0000","last_became_active":"2023-10-14T16:40:24.774256+0000","last_became_peered":"2023-10-14T16:40:24.774256+0000","last_unstale":"2023-10-14T16:40:26.774426+0000","last_undegraded":"2023-10-14T16:40:26.774426+0000","last_fullsized":"2023-10-14T16:40:26.774426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774375+0000","last_change":"2023-10-14T16:40:25.781085+0000","last_active":"2023-10-14T16:40:26.774375+0000","last_peered":"2023-10-14T16:40:26.774375+0000","last_clean":"2023-10-14T16:40:26.774375+0000","last_became_active":"2023-10-14T16:40:25.780746+0000","last_became_peered":"2023-10-14T16:40:25.780746+0000","last_unstale":"2023-10-14T16:40:26.774375+0000","last_undegraded":"2023-10-14T16:40:26.774375+0000","last_fullsized":"2023-10-14T16:40:26.774375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351413+0000","last_change":"2023-10-14T16:40:24.763402+0000","last_active":"2023-10-14T16:40:28.351413+0000","last_peered":"2023-10-14T16:40:28.351413+0000","last_clean":"2023-10-14T16:40:28.351413+0000","last_became_active":"2023-10-14T16:40:24.763093+0000","last_became_peered":"2023-10-14T16:40:24.763093+0000","last_unstale":"2023-10-14T16:40:28.351413+0000","last_undegraded":"2023-10-14T16:40:28.351413+0000","last_fullsized":"2023-10-14T16:40:28.351413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120612+0000","last_change":"2023-10-14T16:40:25.780063+0000","last_active":"2023-10-14T16:40:30.120612+0000","last_peered":"2023-10-14T16:40:30.120612+0000","last_clean":"2023-10-14T16:40:30.120612+0000","last_became_active":"2023-10-14T16:40:25.776862+0000","last_became_peered":"2023-10-14T16:40:25.776862+0000","last_unstale":"2023-10-14T16:40:30.120612+0000","last_undegraded":"2023-10-14T16:40:30.120612+0000","last_fullsized":"2023-10-14T16:40:30.120612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781765+0000","last_change":"2023-10-14T16:40:24.768554+0000","last_active":"2023-10-14T16:40:26.781765+0000","last_peered":"2023-10-14T16:40:26.781765+0000","last_clean":"2023-10-14T16:40:26.781765+0000","last_became_active":"2023-10-14T16:40:24.768232+0000","last_became_peered":"2023-10-14T16:40:24.768232+0000","last_unstale":"2023-10-14T16:40:26.781765+0000","last_undegraded":"2023-10-14T16:40:26.781765+0000","last_fullsized":"2023-10-14T16:40:26.781765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:59.941812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485534+0000","last_change":"2023-10-14T16:40:25.778676+0000","last_active":"2023-10-14T16:40:29.485534+0000","last_peered":"2023-10-14T16:40:29.485534+0000","last_clean":"2023-10-14T16:40:29.485534+0000","last_became_active":"2023-10-14T16:40:25.778295+0000","last_became_peered":"2023-10-14T16:40:25.778295+0000","last_unstale":"2023-10-14T16:40:29.485534+0000","last_undegraded":"2023-10-14T16:40:29.485534+0000","last_fullsized":"2023-10-14T16:40:29.485534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:25.312672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351788+0000","last_change":"2023-10-14T16:40:24.764655+0000","last_active":"2023-10-14T16:40:28.351788+0000","last_peered":"2023-10-14T16:40:28.351788+0000","last_clean":"2023-10-14T16:40:28.351788+0000","last_became_active":"2023-10-14T16:40:24.764327+0000","last_became_peered":"2023-10-14T16:40:24.764327+0000","last_unstale":"2023-10-14T16:40:28.351788+0000","last_undegraded":"2023-10-14T16:40:28.351788+0000","last_fullsized":"2023-10-14T16:40:28.351788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.733975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123336+0000","last_change":"2023-10-14T16:40:25.783442+0000","last_active":"2023-10-14T16:40:30.123336+0000","last_peered":"2023-10-14T16:40:30.123336+0000","last_clean":"2023-10-14T16:40:30.123336+0000","last_became_active":"2023-10-14T16:40:25.783169+0000","last_became_peered":"2023-10-14T16:40:25.783169+0000","last_unstale":"2023-10-14T16:40:30.123336+0000","last_undegraded":"2023-10-14T16:40:30.123336+0000","last_fullsized":"2023-10-14T16:40:30.123336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359494+0000","last_change":"2023-10-14T16:40:24.767382+0000","last_active":"2023-10-14T16:40:28.359494+0000","last_peered":"2023-10-14T16:40:28.359494+0000","last_clean":"2023-10-14T16:40:28.359494+0000","last_became_active":"2023-10-14T16:40:24.767094+0000","last_became_peered":"2023-10-14T16:40:24.767094+0000","last_unstale":"2023-10-14T16:40:28.359494+0000","last_undegraded":"2023-10-14T16:40:28.359494+0000","last_fullsized":"2023-10-14T16:40:28.359494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778953+0000","last_change":"2023-10-14T16:40:25.776539+0000","last_active":"2023-10-14T16:40:26.778953+0000","last_peered":"2023-10-14T16:40:26.778953+0000","last_clean":"2023-10-14T16:40:26.778953+0000","last_became_active":"2023-10-14T16:40:25.776126+0000","last_became_peered":"2023-10-14T16:40:25.776126+0000","last_unstale":"2023-10-14T16:40:26.778953+0000","last_undegraded":"2023-10-14T16:40:26.778953+0000","last_fullsized":"2023-10-14T16:40:26.778953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781420+0000","last_change":"2023-10-14T16:40:24.768941+0000","last_active":"2023-10-14T16:40:26.781420+0000","last_peered":"2023-10-14T16:40:26.781420+0000","last_clean":"2023-10-14T16:40:26.781420+0000","last_became_active":"2023-10-14T16:40:24.767075+0000","last_became_peered":"2023-10-14T16:40:24.767075+0000","last_unstale":"2023-10-14T16:40:26.781420+0000","last_undegraded":"2023-10-14T16:40:26.781420+0000","last_fullsized":"2023-10-14T16:40:26.781420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083537+0000","last_change":"2023-10-14T16:40:25.791099+0000","last_active":"2023-10-14T16:40:29.083537+0000","last_peered":"2023-10-14T16:40:29.083537+0000","last_clean":"2023-10-14T16:40:29.083537+0000","last_became_active":"2023-10-14T16:40:25.790819+0000","last_became_peered":"2023-10-14T16:40:25.790819+0000","last_unstale":"2023-10-14T16:40:29.083537+0000","last_undegraded":"2023-10-14T16:40:29.083537+0000","last_fullsized":"2023-10-14T16:40:29.083537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:56.842342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.356872+0000","last_change":"2023-10-14T16:40:24.760669+0000","last_active":"2023-10-14T16:40:28.356872+0000","last_peered":"2023-10-14T16:40:28.356872+0000","last_clean":"2023-10-14T16:40:28.356872+0000","last_became_active":"2023-10-14T16:40:24.760395+0000","last_became_peered":"2023-10-14T16:40:24.760395+0000","last_unstale":"2023-10-14T16:40:28.356872+0000","last_undegraded":"2023-10-14T16:40:28.356872+0000","last_fullsized":"2023-10-14T16:40:28.356872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122343+0000","last_change":"2023-10-14T16:40:25.779561+0000","last_active":"2023-10-14T16:40:30.122343+0000","last_peered":"2023-10-14T16:40:30.122343+0000","last_clean":"2023-10-14T16:40:30.122343+0000","last_became_active":"2023-10-14T16:40:25.779246+0000","last_became_peered":"2023-10-14T16:40:25.779246+0000","last_unstale":"2023-10-14T16:40:30.122343+0000","last_undegraded":"2023-10-14T16:40:30.122343+0000","last_fullsized":"2023-10-14T16:40:30.122343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351501+0000","last_change":"2023-10-14T16:40:24.764378+0000","last_active":"2023-10-14T16:40:28.351501+0000","last_peered":"2023-10-14T16:40:28.351501+0000","last_clean":"2023-10-14T16:40:28.351501+0000","last_became_active":"2023-10-14T16:40:24.764094+0000","last_became_peered":"2023-10-14T16:40:24.764094+0000","last_unstale":"2023-10-14T16:40:28.351501+0000","last_undegraded":"2023-10-14T16:40:28.351501+0000","last_fullsized":"2023-10-14T16:40:28.351501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:55.307930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083734+0000","last_change":"2023-10-14T16:40:25.782667+0000","last_active":"2023-10-14T16:40:29.083734+0000","last_peered":"2023-10-14T16:40:29.083734+0000","last_clean":"2023-10-14T16:40:29.083734+0000","last_became_active":"2023-10-14T16:40:25.782311+0000","last_became_peered":"2023-10-14T16:40:25.782311+0000","last_unstale":"2023-10-14T16:40:29.083734+0000","last_undegraded":"2023-10-14T16:40:29.083734+0000","last_fullsized":"2023-10-14T16:40:29.083734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:35.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780460+0000","last_change":"2023-10-14T16:40:24.765776+0000","last_active":"2023-10-14T16:40:26.780460+0000","last_peered":"2023-10-14T16:40:26.780460+0000","last_clean":"2023-10-14T16:40:26.780460+0000","last_became_active":"2023-10-14T16:40:24.761753+0000","last_became_peered":"2023-10-14T16:40:24.761753+0000","last_unstale":"2023-10-14T16:40:26.780460+0000","last_undegraded":"2023-10-14T16:40:26.780460+0000","last_fullsized":"2023-10-14T16:40:26.780460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780507+0000","last_change":"2023-10-14T16:40:25.761768+0000","last_active":"2023-10-14T16:40:26.780507+0000","last_peered":"2023-10-14T16:40:26.780507+0000","last_clean":"2023-10-14T16:40:26.780507+0000","last_became_active":"2023-10-14T16:40:25.761374+0000","last_became_peered":"2023-10-14T16:40:25.761374+0000","last_unstale":"2023-10-14T16:40:26.780507+0000","last_undegraded":"2023-10-14T16:40:26.780507+0000","last_fullsized":"2023-10-14T16:40:26.780507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775466+0000","last_change":"2023-10-14T16:40:24.758887+0000","last_active":"2023-10-14T16:40:27.775466+0000","last_peered":"2023-10-14T16:40:27.775466+0000","last_clean":"2023-10-14T16:40:27.775466+0000","last_became_active":"2023-10-14T16:40:24.758086+0000","last_became_peered":"2023-10-14T16:40:24.758086+0000","last_unstale":"2023-10-14T16:40:27.775466+0000","last_undegraded":"2023-10-14T16:40:27.775466+0000","last_fullsized":"2023-10-14T16:40:27.775466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:03.099034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083022+0000","last_change":"2023-10-14T16:40:25.793027+0000","last_active":"2023-10-14T16:40:29.083022+0000","last_peered":"2023-10-14T16:40:29.083022+0000","last_clean":"2023-10-14T16:40:29.083022+0000","last_became_active":"2023-10-14T16:40:25.791542+0000","last_became_peered":"2023-10-14T16:40:25.791542+0000","last_unstale":"2023-10-14T16:40:29.083022+0000","last_undegraded":"2023-10-14T16:40:29.083022+0000","last_fullsized":"2023-10-14T16:40:29.083022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779724+0000","last_change":"2023-10-14T16:40:24.763877+0000","last_active":"2023-10-14T16:40:26.779724+0000","last_peered":"2023-10-14T16:40:26.779724+0000","last_clean":"2023-10-14T16:40:26.779724+0000","last_became_active":"2023-10-14T16:40:24.763637+0000","last_became_peered":"2023-10-14T16:40:24.763637+0000","last_unstale":"2023-10-14T16:40:26.779724+0000","last_undegraded":"2023-10-14T16:40:26.779724+0000","last_fullsized":"2023-10-14T16:40:26.779724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083503+0000","last_change":"2023-10-14T16:40:25.792192+0000","last_active":"2023-10-14T16:40:29.083503+0000","last_peered":"2023-10-14T16:40:29.083503+0000","last_clean":"2023-10-14T16:40:29.083503+0000","last_became_active":"2023-10-14T16:40:25.791748+0000","last_became_peered":"2023-10-14T16:40:25.791748+0000","last_unstale":"2023-10-14T16:40:29.083503+0000","last_undegraded":"2023-10-14T16:40:29.083503+0000","last_fullsized":"2023-10-14T16:40:29.083503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776156+0000","last_change":"2023-10-14T16:40:24.766861+0000","last_active":"2023-10-14T16:40:27.776156+0000","last_peered":"2023-10-14T16:40:27.776156+0000","last_clean":"2023-10-14T16:40:27.776156+0000","last_became_active":"2023-10-14T16:40:24.766319+0000","last_became_peered":"2023-10-14T16:40:24.766319+0000","last_unstale":"2023-10-14T16:40:27.776156+0000","last_undegraded":"2023-10-14T16:40:27.776156+0000","last_fullsized":"2023-10-14T16:40:27.776156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084450+0000","last_change":"2023-10-14T16:40:25.782879+0000","last_active":"2023-10-14T16:40:29.084450+0000","last_peered":"2023-10-14T16:40:29.084450+0000","last_clean":"2023-10-14T16:40:29.084450+0000","last_became_active":"2023-10-14T16:40:25.782368+0000","last_became_peered":"2023-10-14T16:40:25.782368+0000","last_unstale":"2023-10-14T16:40:29.084450+0000","last_undegraded":"2023-10-14T16:40:29.084450+0000","last_fullsized":"2023-10-14T16:40:29.084450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.760301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359525+0000","last_change":"2023-10-14T16:40:24.763429+0000","last_active":"2023-10-14T16:40:28.359525+0000","last_peered":"2023-10-14T16:40:28.359525+0000","last_clean":"2023-10-14T16:40:28.359525+0000","last_became_active":"2023-10-14T16:40:24.761118+0000","last_became_peered":"2023-10-14T16:40:24.761118+0000","last_unstale":"2023-10-14T16:40:28.359525+0000","last_undegraded":"2023-10-14T16:40:28.359525+0000","last_fullsized":"2023-10-14T16:40:28.359525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485360+0000","last_change":"2023-10-14T16:40:25.784774+0000","last_active":"2023-10-14T16:40:29.485360+0000","last_peered":"2023-10-14T16:40:29.485360+0000","last_clean":"2023-10-14T16:40:29.485360+0000","last_became_active":"2023-10-14T16:40:25.784416+0000","last_became_peered":"2023-10-14T16:40:25.784416+0000","last_unstale":"2023-10-14T16:40:29.485360+0000","last_undegraded":"2023-10-14T16:40:29.485360+0000","last_fullsized":"2023-10-14T16:40:29.485360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:35.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775852+0000","last_change":"2023-10-14T16:40:24.767764+0000","last_active":"2023-10-14T16:40:27.775852+0000","last_peered":"2023-10-14T16:40:27.775852+0000","last_clean":"2023-10-14T16:40:27.775852+0000","last_became_active":"2023-10-14T16:40:24.767467+0000","last_became_peered":"2023-10-14T16:40:24.767467+0000","last_unstale":"2023-10-14T16:40:27.775852+0000","last_undegraded":"2023-10-14T16:40:27.775852+0000","last_fullsized":"2023-10-14T16:40:27.775852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083636+0000","last_change":"2023-10-14T16:40:25.792940+0000","last_active":"2023-10-14T16:40:29.083636+0000","last_peered":"2023-10-14T16:40:29.083636+0000","last_clean":"2023-10-14T16:40:29.083636+0000","last_became_active":"2023-10-14T16:40:25.790465+0000","last_became_peered":"2023-10-14T16:40:25.790465+0000","last_unstale":"2023-10-14T16:40:29.083636+0000","last_undegraded":"2023-10-14T16:40:29.083636+0000","last_fullsized":"2023-10-14T16:40:29.083636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:50.009933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.765033+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:40:25.765033+0000","last_peered":"2023-10-14T16:40:25.765033+0000","last_clean":"2023-10-14T16:40:25.765033+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:40:25.765033+0000","last_undegraded":"2023-10-14T16:40:25.765033+0000","last_fullsized":"2023-10-14T16:40:25.765033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351635+0000","last_change":"2023-10-14T16:40:24.765633+0000","last_active":"2023-10-14T16:40:28.351635+0000","last_peered":"2023-10-14T16:40:28.351635+0000","last_clean":"2023-10-14T16:40:28.351635+0000","last_became_active":"2023-10-14T16:40:24.760310+0000","last_became_peered":"2023-10-14T16:40:24.760310+0000","last_unstale":"2023-10-14T16:40:28.351635+0000","last_undegraded":"2023-10-14T16:40:28.351635+0000","last_fullsized":"2023-10-14T16:40:28.351635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:42.771145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775084+0000","last_change":"2023-10-14T16:40:25.783431+0000","last_active":"2023-10-14T16:40:26.775084+0000","last_peered":"2023-10-14T16:40:26.775084+0000","last_clean":"2023-10-14T16:40:26.775084+0000","last_became_active":"2023-10-14T16:40:25.782668+0000","last_became_peered":"2023-10-14T16:40:25.782668+0000","last_unstale":"2023-10-14T16:40:26.775084+0000","last_undegraded":"2023-10-14T16:40:26.775084+0000","last_fullsized":"2023-10-14T16:40:26.775084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340850+0000","last_change":"2023-10-14T16:40:24.774157+0000","last_active":"2023-10-14T16:40:28.340850+0000","last_peered":"2023-10-14T16:40:28.340850+0000","last_clean":"2023-10-14T16:40:28.340850+0000","last_became_active":"2023-10-14T16:40:24.773773+0000","last_became_peered":"2023-10-14T16:40:24.773773+0000","last_unstale":"2023-10-14T16:40:28.340850+0000","last_undegraded":"2023-10-14T16:40:28.340850+0000","last_fullsized":"2023-10-14T16:40:28.340850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122220+0000","last_change":"2023-10-14T16:40:25.772755+0000","last_active":"2023-10-14T16:40:30.122220+0000","last_peered":"2023-10-14T16:40:30.122220+0000","last_clean":"2023-10-14T16:40:30.122220+0000","last_became_active":"2023-10-14T16:40:25.772431+0000","last_became_peered":"2023-10-14T16:40:25.772431+0000","last_unstale":"2023-10-14T16:40:30.122220+0000","last_undegraded":"2023-10-14T16:40:30.122220+0000","last_fullsized":"2023-10-14T16:40:30.122220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:39.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775795+0000","last_change":"2023-10-14T16:40:24.770677+0000","last_active":"2023-10-14T16:40:27.775795+0000","last_peered":"2023-10-14T16:40:27.775795+0000","last_clean":"2023-10-14T16:40:27.775795+0000","last_became_active":"2023-10-14T16:40:24.770259+0000","last_became_peered":"2023-10-14T16:40:24.770259+0000","last_unstale":"2023-10-14T16:40:27.775795+0000","last_undegraded":"2023-10-14T16:40:27.775795+0000","last_fullsized":"2023-10-14T16:40:27.775795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120889+0000","last_change":"2023-10-14T16:40:25.768484+0000","last_active":"2023-10-14T16:40:30.120889+0000","last_peered":"2023-10-14T16:40:30.120889+0000","last_clean":"2023-10-14T16:40:30.120889+0000","last_became_active":"2023-10-14T16:40:25.767865+0000","last_became_peered":"2023-10-14T16:40:25.767865+0000","last_unstale":"2023-10-14T16:40:30.120889+0000","last_undegraded":"2023-10-14T16:40:30.120889+0000","last_fullsized":"2023-10-14T16:40:30.120889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351522+0000","last_change":"2023-10-14T16:40:24.762562+0000","last_active":"2023-10-14T16:40:28.351522+0000","last_peered":"2023-10-14T16:40:28.351522+0000","last_clean":"2023-10-14T16:40:28.351522+0000","last_became_active":"2023-10-14T16:40:24.761399+0000","last_became_peered":"2023-10-14T16:40:24.761399+0000","last_unstale":"2023-10-14T16:40:28.351522+0000","last_undegraded":"2023-10-14T16:40:28.351522+0000","last_fullsized":"2023-10-14T16:40:28.351522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:40.569040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778450+0000","last_change":"2023-10-14T16:40:25.767520+0000","last_active":"2023-10-14T16:40:26.778450+0000","last_peered":"2023-10-14T16:40:26.778450+0000","last_clean":"2023-10-14T16:40:26.778450+0000","last_became_active":"2023-10-14T16:40:25.765947+0000","last_became_peered":"2023-10-14T16:40:25.765947+0000","last_unstale":"2023-10-14T16:40:26.778450+0000","last_undegraded":"2023-10-14T16:40:26.778450+0000","last_fullsized":"2023-10-14T16:40:26.778450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359389+0000","last_change":"2023-10-14T16:40:24.758028+0000","last_active":"2023-10-14T16:40:28.359389+0000","last_peered":"2023-10-14T16:40:28.359389+0000","last_clean":"2023-10-14T16:40:28.359389+0000","last_became_active":"2023-10-14T16:40:24.757727+0000","last_became_peered":"2023-10-14T16:40:24.757727+0000","last_unstale":"2023-10-14T16:40:28.359389+0000","last_undegraded":"2023-10-14T16:40:28.359389+0000","last_fullsized":"2023-10-14T16:40:28.359389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485567+0000","last_change":"2023-10-14T16:40:25.789247+0000","last_active":"2023-10-14T16:40:29.485567+0000","last_peered":"2023-10-14T16:40:29.485567+0000","last_clean":"2023-10-14T16:40:29.485567+0000","last_became_active":"2023-10-14T16:40:25.788936+0000","last_became_peered":"2023-10-14T16:40:25.788936+0000","last_unstale":"2023-10-14T16:40:29.485567+0000","last_undegraded":"2023-10-14T16:40:29.485567+0000","last_fullsized":"2023-10-14T16:40:29.485567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:31.604878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341168+0000","last_change":"2023-10-14T16:40:24.776223+0000","last_active":"2023-10-14T16:40:28.341168+0000","last_peered":"2023-10-14T16:40:28.341168+0000","last_clean":"2023-10-14T16:40:28.341168+0000","last_became_active":"2023-10-14T16:40:24.775651+0000","last_became_peered":"2023-10-14T16:40:24.775651+0000","last_unstale":"2023-10-14T16:40:28.341168+0000","last_undegraded":"2023-10-14T16:40:28.341168+0000","last_fullsized":"2023-10-14T16:40:28.341168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120628+0000","last_change":"2023-10-14T16:40:25.768945+0000","last_active":"2023-10-14T16:40:30.120628+0000","last_peered":"2023-10-14T16:40:30.120628+0000","last_clean":"2023-10-14T16:40:30.120628+0000","last_became_active":"2023-10-14T16:40:25.768628+0000","last_became_peered":"2023-10-14T16:40:25.768628+0000","last_unstale":"2023-10-14T16:40:30.120628+0000","last_undegraded":"2023-10-14T16:40:30.120628+0000","last_fullsized":"2023-10-14T16:40:30.120628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.121273+0000","last_change":"2023-10-14T16:40:24.768276+0000","last_active":"2023-10-14T16:40:30.121273+0000","last_peered":"2023-10-14T16:40:30.121273+0000","last_clean":"2023-10-14T16:40:30.121273+0000","last_became_active":"2023-10-14T16:40:24.766828+0000","last_became_peered":"2023-10-14T16:40:24.766828+0000","last_unstale":"2023-10-14T16:40:30.121273+0000","last_undegraded":"2023-10-14T16:40:30.121273+0000","last_fullsized":"2023-10-14T16:40:30.121273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775754+0000","last_change":"2023-10-14T16:40:25.779777+0000","last_active":"2023-10-14T16:40:26.775754+0000","last_peered":"2023-10-14T16:40:26.775754+0000","last_clean":"2023-10-14T16:40:26.775754+0000","last_became_active":"2023-10-14T16:40:25.779380+0000","last_became_peered":"2023-10-14T16:40:25.779380+0000","last_unstale":"2023-10-14T16:40:26.775754+0000","last_undegraded":"2023-10-14T16:40:26.775754+0000","last_fullsized":"2023-10-14T16:40:26.775754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781895+0000","last_change":"2023-10-14T16:40:24.773797+0000","last_active":"2023-10-14T16:40:26.781895+0000","last_peered":"2023-10-14T16:40:26.781895+0000","last_clean":"2023-10-14T16:40:26.781895+0000","last_became_active":"2023-10-14T16:40:24.773498+0000","last_became_peered":"2023-10-14T16:40:24.773498+0000","last_unstale":"2023-10-14T16:40:26.781895+0000","last_undegraded":"2023-10-14T16:40:26.781895+0000","last_fullsized":"2023-10-14T16:40:26.781895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122877+0000","last_change":"2023-10-14T16:40:25.766374+0000","last_active":"2023-10-14T16:40:30.122877+0000","last_peered":"2023-10-14T16:40:30.122877+0000","last_clean":"2023-10-14T16:40:30.122877+0000","last_became_active":"2023-10-14T16:40:25.764271+0000","last_became_peered":"2023-10-14T16:40:25.764271+0000","last_unstale":"2023-10-14T16:40:30.122877+0000","last_undegraded":"2023-10-14T16:40:30.122877+0000","last_fullsized":"2023-10-14T16:40:30.122877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340815+0000","last_change":"2023-10-14T16:40:24.775192+0000","last_active":"2023-10-14T16:40:28.340815+0000","last_peered":"2023-10-14T16:40:28.340815+0000","last_clean":"2023-10-14T16:40:28.340815+0000","last_became_active":"2023-10-14T16:40:24.774186+0000","last_became_peered":"2023-10-14T16:40:24.774186+0000","last_unstale":"2023-10-14T16:40:28.340815+0000","last_undegraded":"2023-10-14T16:40:28.340815+0000","last_fullsized":"2023-10-14T16:40:28.340815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778279+0000","last_change":"2023-10-14T16:40:25.769524+0000","last_active":"2023-10-14T16:40:26.778279+0000","last_peered":"2023-10-14T16:40:26.778279+0000","last_clean":"2023-10-14T16:40:26.778279+0000","last_became_active":"2023-10-14T16:40:25.769237+0000","last_became_peered":"2023-10-14T16:40:25.769237+0000","last_unstale":"2023-10-14T16:40:26.778279+0000","last_undegraded":"2023-10-14T16:40:26.778279+0000","last_fullsized":"2023-10-14T16:40:26.778279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359223+0000","last_change":"2023-10-14T16:40:24.765632+0000","last_active":"2023-10-14T16:40:28.359223+0000","last_peered":"2023-10-14T16:40:28.359223+0000","last_clean":"2023-10-14T16:40:28.359223+0000","last_became_active":"2023-10-14T16:40:24.765393+0000","last_became_peered":"2023-10-14T16:40:24.765393+0000","last_unstale":"2023-10-14T16:40:28.359223+0000","last_undegraded":"2023-10-14T16:40:28.359223+0000","last_fullsized":"2023-10-14T16:40:28.359223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:24.098376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774557+0000","last_change":"2023-10-14T16:40:25.766437+0000","last_active":"2023-10-14T16:40:26.774557+0000","last_peered":"2023-10-14T16:40:26.774557+0000","last_clean":"2023-10-14T16:40:26.774557+0000","last_became_active":"2023-10-14T16:40:25.765798+0000","last_became_peered":"2023-10-14T16:40:25.765798+0000","last_unstale":"2023-10-14T16:40:26.774557+0000","last_undegraded":"2023-10-14T16:40:26.774557+0000","last_fullsized":"2023-10-14T16:40:26.774557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359248+0000","last_change":"2023-10-14T16:40:24.765207+0000","last_active":"2023-10-14T16:40:28.359248+0000","last_peered":"2023-10-14T16:40:28.359248+0000","last_clean":"2023-10-14T16:40:28.359248+0000","last_became_active":"2023-10-14T16:40:24.764610+0000","last_became_peered":"2023-10-14T16:40:24.764610+0000","last_unstale":"2023-10-14T16:40:28.359248+0000","last_undegraded":"2023-10-14T16:40:28.359248+0000","last_fullsized":"2023-10-14T16:40:28.359248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775031+0000","last_change":"2023-10-14T16:40:25.788202+0000","last_active":"2023-10-14T16:40:26.775031+0000","last_peered":"2023-10-14T16:40:26.775031+0000","last_clean":"2023-10-14T16:40:26.775031+0000","last_became_active":"2023-10-14T16:40:25.785927+0000","last_became_peered":"2023-10-14T16:40:25.785927+0000","last_unstale":"2023-10-14T16:40:26.775031+0000","last_undegraded":"2023-10-14T16:40:26.775031+0000","last_fullsized":"2023-10-14T16:40:26.775031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340988+0000","last_change":"2023-10-14T16:40:24.775618+0000","last_active":"2023-10-14T16:40:28.340988+0000","last_peered":"2023-10-14T16:40:28.340988+0000","last_clean":"2023-10-14T16:40:28.340988+0000","last_became_active":"2023-10-14T16:40:24.775346+0000","last_became_peered":"2023-10-14T16:40:24.775346+0000","last_unstale":"2023-10-14T16:40:28.340988+0000","last_undegraded":"2023-10-14T16:40:28.340988+0000","last_fullsized":"2023-10-14T16:40:28.340988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780727+0000","last_change":"2023-10-14T16:40:25.778910+0000","last_active":"2023-10-14T16:40:26.780727+0000","last_peered":"2023-10-14T16:40:26.780727+0000","last_clean":"2023-10-14T16:40:26.780727+0000","last_became_active":"2023-10-14T16:40:25.777772+0000","last_became_peered":"2023-10-14T16:40:25.777772+0000","last_unstale":"2023-10-14T16:40:26.780727+0000","last_undegraded":"2023-10-14T16:40:26.780727+0000","last_fullsized":"2023-10-14T16:40:26.780727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485428+0000","last_change":"2023-10-14T16:40:25.779144+0000","last_active":"2023-10-14T16:40:29.485428+0000","last_peered":"2023-10-14T16:40:29.485428+0000","last_clean":"2023-10-14T16:40:29.485428+0000","last_became_active":"2023-10-14T16:40:25.778298+0000","last_became_peered":"2023-10-14T16:40:25.778298+0000","last_unstale":"2023-10-14T16:40:29.485428+0000","last_undegraded":"2023-10-14T16:40:29.485428+0000","last_fullsized":"2023-10-14T16:40:29.485428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:21.372595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341014+0000","last_change":"2023-10-14T16:40:24.776824+0000","last_active":"2023-10-14T16:40:28.341014+0000","last_peered":"2023-10-14T16:40:28.341014+0000","last_clean":"2023-10-14T16:40:28.341014+0000","last_became_active":"2023-10-14T16:40:24.776368+0000","last_became_peered":"2023-10-14T16:40:24.776368+0000","last_unstale":"2023-10-14T16:40:28.341014+0000","last_undegraded":"2023-10-14T16:40:28.341014+0000","last_fullsized":"2023-10-14T16:40:28.341014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485000+0000","last_change":"2023-10-14T16:40:25.777666+0000","last_active":"2023-10-14T16:40:29.485000+0000","last_peered":"2023-10-14T16:40:29.485000+0000","last_clean":"2023-10-14T16:40:29.485000+0000","last_became_active":"2023-10-14T16:40:25.776881+0000","last_became_peered":"2023-10-14T16:40:25.776881+0000","last_unstale":"2023-10-14T16:40:29.485000+0000","last_undegraded":"2023-10-14T16:40:29.485000+0000","last_fullsized":"2023-10-14T16:40:29.485000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:50.009933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341121+0000","last_change":"2023-10-14T16:40:24.777751+0000","last_active":"2023-10-14T16:40:28.341121+0000","last_peered":"2023-10-14T16:40:28.341121+0000","last_clean":"2023-10-14T16:40:28.341121+0000","last_became_active":"2023-10-14T16:40:24.777483+0000","last_became_peered":"2023-10-14T16:40:24.777483+0000","last_unstale":"2023-10-14T16:40:28.341121+0000","last_undegraded":"2023-10-14T16:40:28.341121+0000","last_fullsized":"2023-10-14T16:40:28.341121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123205+0000","last_change":"2023-10-14T16:40:25.783100+0000","last_active":"2023-10-14T16:40:30.123205+0000","last_peered":"2023-10-14T16:40:30.123205+0000","last_clean":"2023-10-14T16:40:30.123205+0000","last_became_active":"2023-10-14T16:40:25.782769+0000","last_became_peered":"2023-10-14T16:40:25.782769+0000","last_unstale":"2023-10-14T16:40:30.123205+0000","last_undegraded":"2023-10-14T16:40:30.123205+0000","last_fullsized":"2023-10-14T16:40:30.123205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775509+0000","last_change":"2023-10-14T16:40:24.768856+0000","last_active":"2023-10-14T16:40:26.775509+0000","last_peered":"2023-10-14T16:40:26.775509+0000","last_clean":"2023-10-14T16:40:26.775509+0000","last_became_active":"2023-10-14T16:40:24.768497+0000","last_became_peered":"2023-10-14T16:40:24.768497+0000","last_unstale":"2023-10-14T16:40:26.775509+0000","last_undegraded":"2023-10-14T16:40:26.775509+0000","last_fullsized":"2023-10-14T16:40:26.775509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:33.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775555+0000","last_change":"2023-10-14T16:40:25.784770+0000","last_active":"2023-10-14T16:40:26.775555+0000","last_peered":"2023-10-14T16:40:26.775555+0000","last_clean":"2023-10-14T16:40:26.775555+0000","last_became_active":"2023-10-14T16:40:25.782997+0000","last_became_peered":"2023-10-14T16:40:25.782997+0000","last_unstale":"2023-10-14T16:40:26.775555+0000","last_undegraded":"2023-10-14T16:40:26.775555+0000","last_fullsized":"2023-10-14T16:40:26.775555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.356987+0000","last_change":"2023-10-14T16:40:24.772362+0000","last_active":"2023-10-14T16:40:28.356987+0000","last_peered":"2023-10-14T16:40:28.356987+0000","last_clean":"2023-10-14T16:40:28.356987+0000","last_became_active":"2023-10-14T16:40:24.772021+0000","last_became_peered":"2023-10-14T16:40:24.772021+0000","last_unstale":"2023-10-14T16:40:28.356987+0000","last_undegraded":"2023-10-14T16:40:28.356987+0000","last_fullsized":"2023-10-14T16:40:28.356987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3280000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6539999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5789999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6639999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6360000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.452}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:40:36.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:40:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.441+0000 7fa3a166d640 1 Processor -- start 2023-10-14T16:40:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.441+0000 7fa3a166d640 1 -- start start 2023-10-14T16:40:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa39c06c040 0x7fa39c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 0x7fa39c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 0x7fa39c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c06c960 con 0x7fa39c06c040 2023-10-14T16:40:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa39c15b9d0 con 0x7fa39c0ad970 2023-10-14T16:40:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.442+0000 7fa3a166d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c15bcb0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa39c06c040 0x7fa39c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa39c06c040 0x7fa39c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36810/0 (socket says 172.21.15.29:36810) 2023-10-14T16:40:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 -- 172.21.15.29:0/2282302123 learned_addr learned my addr 172.21.15.29:0/2282302123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa3937fe640 1 --2- 172.21.15.29:0/2282302123 >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 0x7fa39c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 -- 172.21.15.29:0/2282302123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 msgr2=0x7fa39c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa3a0e6c640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 0x7fa39c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 0x7fa39c15f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 -- 172.21.15.29:0/2282302123 >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 msgr2=0x7fa39c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 --2- 172.21.15.29:0/2282302123 >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 0x7fa39c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa39bfff640 1 -- 172.21.15.29:0/2282302123 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c15bf90 con 0x7fa39c06c040 2023-10-14T16:40:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa3937fe640 1 --2- 172.21.15.29:0/2282302123 >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 0x7fa39c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.443+0000 7fa3a0e6c640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 0x7fa39c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.444+0000 7fa39bfff640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c06c040 0x7fa39c06c420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0cb060 tx=0x7fa38c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=909cf92e20e4f12d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa39b7fe640 1 -- 172.21.15.29:0/2282302123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c031880 con 0x7fa39c06c040 2023-10-14T16:40:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa39b7fe640 1 -- 172.21.15.29:0/2282302123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa38c0319e0 con 0x7fa39c06c040 2023-10-14T16:40:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa39b7fe640 1 -- 172.21.15.29:0/2282302123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa38c031d70 con 0x7fa39c06c040 2023-10-14T16:40:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 -- 172.21.15.29:0/2282302123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c06c040 msgr2=0x7fa39c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c06c040 0x7fa39c06c420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa39c0cb060 tx=0x7fa38c02b920 comp rx=0 tx=0).stop 2023-10-14T16:40:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 -- 172.21.15.29:0/2282302123 shutdown_connections 2023-10-14T16:40:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c15ebc0 0x7fa39c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 --2- 172.21.15.29:0/2282302123 >> v2:172.21.15.186:3300/0 conn(0x7fa39c0ad970 0x7fa39c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 --2- 172.21.15.29:0/2282302123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c06c040 0x7fa39c06c420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 -- 172.21.15.29:0/2282302123 >> 172.21.15.29:0/2282302123 conn(0x7fa39c0ce6a0 msgr2=0x7fa39c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 -- 172.21.15.29:0/2282302123 shutdown_connections 2023-10-14T16:40:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.445+0000 7fa3a166d640 1 -- 172.21.15.29:0/2282302123 wait complete. 2023-10-14T16:40:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.447+0000 7fa3a166d640 1 Processor -- start 2023-10-14T16:40:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.447+0000 7fa3a166d640 1 -- start start 2023-10-14T16:40:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa39c06c040 0x7fa39c0c7f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 0x7fa39c0c4b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0c5090 con 0x7fa39c15ebc0 2023-10-14T16:40:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c1f65e0 con 0x7fa39c06c040 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a166d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0c8580 con 0x7fa39c0ad970 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a0e6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a0e6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36822/0 (socket says 172.21.15.29:36822) 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.448+0000 7fa3a0e6c640 1 -- 172.21.15.29:0/1040697880 learned_addr learned my addr 172.21.15.29:0/1040697880 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa39bfff640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa39c06c040 0x7fa39c0c7f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3937fe640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 0x7fa39c0c4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 -- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 msgr2=0x7fa39c0c4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 0x7fa39c0c4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 -- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa39c06c040 msgr2=0x7fa39c0c7f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa39c06c040 0x7fa39c0c7f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 -- 172.21.15.29:0/1040697880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa38c0094d0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3937fe640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 0x7fa39c0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a0e6c640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa39400cef0 tx=0x7fa39400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39400d840 con 0x7fa39c15ebc0 2023-10-14T16:40:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa39400d9c0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c148400 con 0x7fa39c15ebc0 2023-10-14T16:40:36.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.449+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa39400dd70 con 0x7fa39c15ebc0 2023-10-14T16:40:36.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.450+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c1489c0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.451+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c06b330 con 0x7fa39c15ebc0 2023-10-14T16:40:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.452+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa39401e3b0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.453+0000 7fa3997fa640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 0x7fa37c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.453+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa39408a0b0 con 0x7fa39c15ebc0 2023-10-14T16:40:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.453+0000 7fa39bfff640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 0x7fa37c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.453+0000 7fa39bfff640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 0x7fa37c06d330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa38c036000 tx=0x7fa38c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.457+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa394055220 con 0x7fa39c15ebc0 2023-10-14T16:40:36.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.643+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa39c0cc550 con 0x7fa37c06ae70 2023-10-14T16:40:36.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.663+0000 7fa3997fa640 1 -- 172.21.15.29:0/1040697880 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+264439 (secure 0 0 0) 0x7fa38c0317a0 con 0x7fa37c06ae70 2023-10-14T16:40:36.666 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:36.669 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:40:36.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 msgr2=0x7fa37c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 0x7fa37c06d330 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa38c036000 tx=0x7fa38c03c040 comp rx=0 tx=0).stop 2023-10-14T16:40:36.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 msgr2=0x7fa39c1f60a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa39400cef0 tx=0x7fa39400a710 comp rx=0 tx=0).stop 2023-10-14T16:40:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 shutdown_connections 2023-10-14T16:40:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa37c06ae70 0x7fa37c06d330 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa39c15ebc0 0x7fa39c1f60a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa39c0ad970 0x7fa39c0c4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 --2- 172.21.15.29:0/1040697880 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa39c06c040 0x7fa39c0c7f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 >> 172.21.15.29:0/1040697880 conn(0x7fa39c0ce6a0 msgr2=0x7fa39c0cd6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 shutdown_connections 2023-10-14T16:40:36.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.670+0000 7fa3a166d640 1 -- 172.21.15.29:0/1040697880 wait complete. 2023-10-14T16:40:36.688 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":24,"stamp":"2023-10-14T16:40:35.301882+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":235628,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754739092,"statfs":{"total":773094113280,"available":772852830208,"internally_reserved":0,"allocated":2891776,"data_stored":1752740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.560837"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083981+0000","last_change":"2023-10-14T16:40:25.778039+0000","last_active":"2023-10-14T16:40:29.083981+0000","last_peered":"2023-10-14T16:40:29.083981+0000","last_clean":"2023-10-14T16:40:29.083981+0000","last_became_active":"2023-10-14T16:40:25.777745+0000","last_became_peered":"2023-10-14T16:40:25.777745+0000","last_unstale":"2023-10-14T16:40:29.083981+0000","last_undegraded":"2023-10-14T16:40:29.083981+0000","last_fullsized":"2023-10-14T16:40:29.083981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:21.372595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.121147+0000","last_change":"2023-10-14T16:40:25.782038+0000","last_active":"2023-10-14T16:40:30.121147+0000","last_peered":"2023-10-14T16:40:30.121147+0000","last_clean":"2023-10-14T16:40:30.121147+0000","last_became_active":"2023-10-14T16:40:25.781722+0000","last_became_peered":"2023-10-14T16:40:25.781722+0000","last_unstale":"2023-10-14T16:40:30.121147+0000","last_undegraded":"2023-10-14T16:40:30.121147+0000","last_fullsized":"2023-10-14T16:40:30.121147+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122914+0000","last_change":"2023-10-14T16:40:25.757114+0000","last_active":"2023-10-14T16:40:30.122914+0000","last_peered":"2023-10-14T16:40:30.122914+0000","last_clean":"2023-10-14T16:40:30.122914+0000","last_became_active":"2023-10-14T16:40:25.756793+0000","last_became_peered":"2023-10-14T16:40:25.756793+0000","last_unstale":"2023-10-14T16:40:30.122914+0000","last_undegraded":"2023-10-14T16:40:30.122914+0000","last_fullsized":"2023-10-14T16:40:30.122914+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484919+0000","last_change":"2023-10-14T16:40:25.788571+0000","last_active":"2023-10-14T16:40:29.484919+0000","last_peered":"2023-10-14T16:40:29.484919+0000","last_clean":"2023-10-14T16:40:29.484919+0000","last_became_active":"2023-10-14T16:40:25.788388+0000","last_became_peered":"2023-10-14T16:40:25.788388+0000","last_unstale":"2023-10-14T16:40:29.484919+0000","last_undegraded":"2023-10-14T16:40:29.484919+0000","last_fullsized":"2023-10-14T16:40:29.484919+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083267+0000","last_change":"2023-10-14T16:40:25.781929+0000","last_active":"2023-10-14T16:40:29.083267+0000","last_peered":"2023-10-14T16:40:29.083267+0000","last_clean":"2023-10-14T16:40:29.083267+0000","last_became_active":"2023-10-14T16:40:25.774025+0000","last_became_peered":"2023-10-14T16:40:25.774025+0000","last_unstale":"2023-10-14T16:40:29.083267+0000","last_undegraded":"2023-10-14T16:40:29.083267+0000","last_fullsized":"2023-10-14T16:40:29.083267+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484458+0000","last_change":"2023-10-14T16:40:25.785668+0000","last_active":"2023-10-14T16:40:29.484458+0000","last_peered":"2023-10-14T16:40:29.484458+0000","last_clean":"2023-10-14T16:40:29.484458+0000","last_became_active":"2023-10-14T16:40:25.785300+0000","last_became_peered":"2023-10-14T16:40:25.785300+0000","last_unstale":"2023-10-14T16:40:29.484458+0000","last_undegraded":"2023-10-14T16:40:29.484458+0000","last_fullsized":"2023-10-14T16:40:29.484458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083019+0000","last_change":"2023-10-14T16:40:25.786146+0000","last_active":"2023-10-14T16:40:29.083019+0000","last_peered":"2023-10-14T16:40:29.083019+0000","last_clean":"2023-10-14T16:40:29.083019+0000","last_became_active":"2023-10-14T16:40:25.785748+0000","last_became_peered":"2023-10-14T16:40:25.785748+0000","last_unstale":"2023-10-14T16:40:29.083019+0000","last_undegraded":"2023-10-14T16:40:29.083019+0000","last_fullsized":"2023-10-14T16:40:29.083019+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774509+0000","last_change":"2023-10-14T16:40:25.783832+0000","last_active":"2023-10-14T16:40:26.774509+0000","last_peered":"2023-10-14T16:40:26.774509+0000","last_clean":"2023-10-14T16:40:26.774509+0000","last_became_active":"2023-10-14T16:40:25.781635+0000","last_became_peered":"2023-10-14T16:40:25.781635+0000","last_unstale":"2023-10-14T16:40:26.774509+0000","last_undegraded":"2023-10-14T16:40:26.774509+0000","last_fullsized":"2023-10-14T16:40:26.774509+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084293+0000","last_change":"2023-10-14T16:40:25.791696+0000","last_active":"2023-10-14T16:40:29.084293+0000","last_peered":"2023-10-14T16:40:29.084293+0000","last_clean":"2023-10-14T16:40:29.084293+0000","last_became_active":"2023-10-14T16:40:25.791413+0000","last_became_peered":"2023-10-14T16:40:25.791413+0000","last_unstale":"2023-10-14T16:40:29.084293+0000","last_undegraded":"2023-10-14T16:40:29.084293+0000","last_fullsized":"2023-10-14T16:40:29.084293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:31.604878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781741+0000","last_change":"2023-10-14T16:40:25.786842+0000","last_active":"2023-10-14T16:40:26.781741+0000","last_peered":"2023-10-14T16:40:26.781741+0000","last_clean":"2023-10-14T16:40:26.781741+0000","last_became_active":"2023-10-14T16:40:25.786528+0000","last_became_peered":"2023-10-14T16:40:25.786528+0000","last_unstale":"2023-10-14T16:40:26.781741+0000","last_undegraded":"2023-10-14T16:40:26.781741+0000","last_fullsized":"2023-10-14T16:40:26.781741+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780857+0000","last_change":"2023-10-14T16:40:25.788130+0000","last_active":"2023-10-14T16:40:26.780857+0000","last_peered":"2023-10-14T16:40:26.780857+0000","last_clean":"2023-10-14T16:40:26.780857+0000","last_became_active":"2023-10-14T16:40:25.787596+0000","last_became_peered":"2023-10-14T16:40:25.787596+0000","last_unstale":"2023-10-14T16:40:26.780857+0000","last_undegraded":"2023-10-14T16:40:26.780857+0000","last_fullsized":"2023-10-14T16:40:26.780857+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778866+0000","last_change":"2023-10-14T16:40:25.765437+0000","last_active":"2023-10-14T16:40:26.778866+0000","last_peered":"2023-10-14T16:40:26.778866+0000","last_clean":"2023-10-14T16:40:26.778866+0000","last_became_active":"2023-10-14T16:40:25.765116+0000","last_became_peered":"2023-10-14T16:40:25.765116+0000","last_unstale":"2023-10-14T16:40:26.778866+0000","last_undegraded":"2023-10-14T16:40:26.778866+0000","last_fullsized":"2023-10-14T16:40:26.778866+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:50:00.953311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774963+0000","last_change":"2023-10-14T16:40:25.784431+0000","last_active":"2023-10-14T16:40:26.774963+0000","last_peered":"2023-10-14T16:40:26.774963+0000","last_clean":"2023-10-14T16:40:26.774963+0000","last_became_active":"2023-10-14T16:40:25.775585+0000","last_became_peered":"2023-10-14T16:40:25.775585+0000","last_unstale":"2023-10-14T16:40:26.774963+0000","last_undegraded":"2023-10-14T16:40:26.774963+0000","last_fullsized":"2023-10-14T16:40:26.774963+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122194+0000","last_change":"2023-10-14T16:40:25.761749+0000","last_active":"2023-10-14T16:40:30.122194+0000","last_peered":"2023-10-14T16:40:30.122194+0000","last_clean":"2023-10-14T16:40:30.122194+0000","last_became_active":"2023-10-14T16:40:25.761444+0000","last_became_peered":"2023-10-14T16:40:25.761444+0000","last_unstale":"2023-10-14T16:40:30.122194+0000","last_undegraded":"2023-10-14T16:40:30.122194+0000","last_fullsized":"2023-10-14T16:40:30.122194+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:25.109874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120661+0000","last_change":"2023-10-14T16:40:25.776450+0000","last_active":"2023-10-14T16:40:30.120661+0000","last_peered":"2023-10-14T16:40:30.120661+0000","last_clean":"2023-10-14T16:40:30.120661+0000","last_became_active":"2023-10-14T16:40:25.776153+0000","last_became_peered":"2023-10-14T16:40:25.776153+0000","last_unstale":"2023-10-14T16:40:30.120661+0000","last_undegraded":"2023-10-14T16:40:30.120661+0000","last_fullsized":"2023-10-14T16:40:30.120661+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484358+0000","last_change":"2023-10-14T16:40:25.786126+0000","last_active":"2023-10-14T16:40:29.484358+0000","last_peered":"2023-10-14T16:40:29.484358+0000","last_clean":"2023-10-14T16:40:29.484358+0000","last_became_active":"2023-10-14T16:40:25.785701+0000","last_became_peered":"2023-10-14T16:40:25.785701+0000","last_unstale":"2023-10-14T16:40:29.484358+0000","last_undegraded":"2023-10-14T16:40:29.484358+0000","last_fullsized":"2023-10-14T16:40:29.484358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.486352+0000","last_change":"2023-10-14T16:40:25.792223+0000","last_active":"2023-10-14T16:40:29.486352+0000","last_peered":"2023-10-14T16:40:29.486352+0000","last_clean":"2023-10-14T16:40:29.486352+0000","last_became_active":"2023-10-14T16:40:25.791860+0000","last_became_peered":"2023-10-14T16:40:25.791860+0000","last_unstale":"2023-10-14T16:40:29.486352+0000","last_undegraded":"2023-10-14T16:40:29.486352+0000","last_fullsized":"2023-10-14T16:40:29.486352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.760301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776199+0000","last_change":"2023-10-14T16:40:25.766839+0000","last_active":"2023-10-14T16:40:27.776199+0000","last_peered":"2023-10-14T16:40:27.776199+0000","last_clean":"2023-10-14T16:40:27.776199+0000","last_became_active":"2023-10-14T16:40:25.766493+0000","last_became_peered":"2023-10-14T16:40:25.766493+0000","last_unstale":"2023-10-14T16:40:27.776199+0000","last_undegraded":"2023-10-14T16:40:27.776199+0000","last_fullsized":"2023-10-14T16:40:27.776199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:42.220060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084022+0000","last_change":"2023-10-14T16:40:25.792353+0000","last_active":"2023-10-14T16:40:29.084022+0000","last_peered":"2023-10-14T16:40:29.084022+0000","last_clean":"2023-10-14T16:40:29.084022+0000","last_became_active":"2023-10-14T16:40:25.792040+0000","last_became_peered":"2023-10-14T16:40:25.792040+0000","last_unstale":"2023-10-14T16:40:29.084022+0000","last_undegraded":"2023-10-14T16:40:29.084022+0000","last_fullsized":"2023-10-14T16:40:29.084022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:25.312672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781176+0000","last_change":"2023-10-14T16:40:25.770102+0000","last_active":"2023-10-14T16:40:26.781176+0000","last_peered":"2023-10-14T16:40:26.781176+0000","last_clean":"2023-10-14T16:40:26.781176+0000","last_became_active":"2023-10-14T16:40:25.769804+0000","last_became_peered":"2023-10-14T16:40:25.769804+0000","last_unstale":"2023-10-14T16:40:26.781176+0000","last_undegraded":"2023-10-14T16:40:26.781176+0000","last_fullsized":"2023-10-14T16:40:26.781176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123282+0000","last_change":"2023-10-14T16:40:25.779852+0000","last_active":"2023-10-14T16:40:30.123282+0000","last_peered":"2023-10-14T16:40:30.123282+0000","last_clean":"2023-10-14T16:40:30.123282+0000","last_became_active":"2023-10-14T16:40:25.779362+0000","last_became_peered":"2023-10-14T16:40:25.779362+0000","last_unstale":"2023-10-14T16:40:30.123282+0000","last_undegraded":"2023-10-14T16:40:30.123282+0000","last_fullsized":"2023-10-14T16:40:30.123282+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484764+0000","last_change":"2023-10-14T16:40:25.781280+0000","last_active":"2023-10-14T16:40:29.484764+0000","last_peered":"2023-10-14T16:40:29.484764+0000","last_clean":"2023-10-14T16:40:29.484764+0000","last_became_active":"2023-10-14T16:40:25.780595+0000","last_became_peered":"2023-10-14T16:40:25.780595+0000","last_unstale":"2023-10-14T16:40:29.484764+0000","last_undegraded":"2023-10-14T16:40:29.484764+0000","last_fullsized":"2023-10-14T16:40:29.484764+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774501+0000","last_change":"2023-10-14T16:40:25.777771+0000","last_active":"2023-10-14T16:40:26.774501+0000","last_peered":"2023-10-14T16:40:26.774501+0000","last_clean":"2023-10-14T16:40:26.774501+0000","last_became_active":"2023-10-14T16:40:25.777508+0000","last_became_peered":"2023-10-14T16:40:25.777508+0000","last_unstale":"2023-10-14T16:40:26.774501+0000","last_undegraded":"2023-10-14T16:40:26.774501+0000","last_fullsized":"2023-10-14T16:40:26.774501+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484634+0000","last_change":"2023-10-14T16:40:25.772029+0000","last_active":"2023-10-14T16:40:29.484634+0000","last_peered":"2023-10-14T16:40:29.484634+0000","last_clean":"2023-10-14T16:40:29.484634+0000","last_became_active":"2023-10-14T16:40:25.771700+0000","last_became_peered":"2023-10-14T16:40:25.771700+0000","last_unstale":"2023-10-14T16:40:29.484634+0000","last_undegraded":"2023-10-14T16:40:29.484634+0000","last_fullsized":"2023-10-14T16:40:29.484634+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775784+0000","last_change":"2023-10-14T16:40:25.777077+0000","last_active":"2023-10-14T16:40:26.775784+0000","last_peered":"2023-10-14T16:40:26.775784+0000","last_clean":"2023-10-14T16:40:26.775784+0000","last_became_active":"2023-10-14T16:40:25.776755+0000","last_became_peered":"2023-10-14T16:40:25.776755+0000","last_unstale":"2023-10-14T16:40:26.775784+0000","last_undegraded":"2023-10-14T16:40:26.775784+0000","last_fullsized":"2023-10-14T16:40:26.775784+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781826+0000","last_change":"2023-10-14T16:40:25.777054+0000","last_active":"2023-10-14T16:40:26.781826+0000","last_peered":"2023-10-14T16:40:26.781826+0000","last_clean":"2023-10-14T16:40:26.781826+0000","last_became_active":"2023-10-14T16:40:25.773588+0000","last_became_peered":"2023-10-14T16:40:25.773588+0000","last_unstale":"2023-10-14T16:40:26.781826+0000","last_undegraded":"2023-10-14T16:40:26.781826+0000","last_fullsized":"2023-10-14T16:40:26.781826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120557+0000","last_change":"2023-10-14T16:40:25.787594+0000","last_active":"2023-10-14T16:40:30.120557+0000","last_peered":"2023-10-14T16:40:30.120557+0000","last_clean":"2023-10-14T16:40:30.120557+0000","last_became_active":"2023-10-14T16:40:25.786833+0000","last_became_peered":"2023-10-14T16:40:25.786833+0000","last_unstale":"2023-10-14T16:40:30.120557+0000","last_undegraded":"2023-10-14T16:40:30.120557+0000","last_fullsized":"2023-10-14T16:40:30.120557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775301+0000","last_change":"2023-10-14T16:40:25.782397+0000","last_active":"2023-10-14T16:40:26.775301+0000","last_peered":"2023-10-14T16:40:26.775301+0000","last_clean":"2023-10-14T16:40:26.775301+0000","last_became_active":"2023-10-14T16:40:25.782109+0000","last_became_peered":"2023-10-14T16:40:25.782109+0000","last_unstale":"2023-10-14T16:40:26.775301+0000","last_undegraded":"2023-10-14T16:40:26.775301+0000","last_fullsized":"2023-10-14T16:40:26.775301+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123515+0000","last_change":"2023-10-14T16:40:25.783124+0000","last_active":"2023-10-14T16:40:30.123515+0000","last_peered":"2023-10-14T16:40:30.123515+0000","last_clean":"2023-10-14T16:40:30.123515+0000","last_became_active":"2023-10-14T16:40:25.782805+0000","last_became_peered":"2023-10-14T16:40:25.782805+0000","last_unstale":"2023-10-14T16:40:30.123515+0000","last_undegraded":"2023-10-14T16:40:30.123515+0000","last_fullsized":"2023-10-14T16:40:30.123515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.484995+0000","last_change":"2023-10-14T16:40:25.784068+0000","last_active":"2023-10-14T16:40:29.484995+0000","last_peered":"2023-10-14T16:40:29.484995+0000","last_clean":"2023-10-14T16:40:29.484995+0000","last_became_active":"2023-10-14T16:40:25.783671+0000","last_became_peered":"2023-10-14T16:40:25.783671+0000","last_unstale":"2023-10-14T16:40:29.484995+0000","last_undegraded":"2023-10-14T16:40:29.484995+0000","last_fullsized":"2023-10-14T16:40:29.484995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:56.842342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775610+0000","last_change":"2023-10-14T16:40:25.777635+0000","last_active":"2023-10-14T16:40:27.775610+0000","last_peered":"2023-10-14T16:40:27.775610+0000","last_clean":"2023-10-14T16:40:27.775610+0000","last_became_active":"2023-10-14T16:40:25.777400+0000","last_became_peered":"2023-10-14T16:40:25.777400+0000","last_unstale":"2023-10-14T16:40:27.775610+0000","last_undegraded":"2023-10-14T16:40:27.775610+0000","last_fullsized":"2023-10-14T16:40:27.775610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779941+0000","last_change":"2023-10-14T16:40:25.758613+0000","last_active":"2023-10-14T16:40:26.779941+0000","last_peered":"2023-10-14T16:40:26.779941+0000","last_clean":"2023-10-14T16:40:26.779941+0000","last_became_active":"2023-10-14T16:40:25.758173+0000","last_became_peered":"2023-10-14T16:40:25.758173+0000","last_unstale":"2023-10-14T16:40:26.779941+0000","last_undegraded":"2023-10-14T16:40:26.779941+0000","last_fullsized":"2023-10-14T16:40:26.779941+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778458+0000","last_change":"2023-10-14T16:40:25.759820+0000","last_active":"2023-10-14T16:40:26.778458+0000","last_peered":"2023-10-14T16:40:26.778458+0000","last_clean":"2023-10-14T16:40:26.778458+0000","last_became_active":"2023-10-14T16:40:25.759498+0000","last_became_peered":"2023-10-14T16:40:25.759498+0000","last_unstale":"2023-10-14T16:40:26.778458+0000","last_undegraded":"2023-10-14T16:40:26.778458+0000","last_fullsized":"2023-10-14T16:40:26.778458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:47.238788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775078+0000","last_change":"2023-10-14T16:40:24.768741+0000","last_active":"2023-10-14T16:40:26.775078+0000","last_peered":"2023-10-14T16:40:26.775078+0000","last_clean":"2023-10-14T16:40:26.775078+0000","last_became_active":"2023-10-14T16:40:24.768379+0000","last_became_peered":"2023-10-14T16:40:24.768379+0000","last_unstale":"2023-10-14T16:40:26.775078+0000","last_undegraded":"2023-10-14T16:40:26.775078+0000","last_fullsized":"2023-10-14T16:40:26.775078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123684+0000","last_change":"2023-10-14T16:40:25.779646+0000","last_active":"2023-10-14T16:40:30.123684+0000","last_peered":"2023-10-14T16:40:30.123684+0000","last_clean":"2023-10-14T16:40:30.123684+0000","last_became_active":"2023-10-14T16:40:25.779001+0000","last_became_peered":"2023-10-14T16:40:25.779001+0000","last_unstale":"2023-10-14T16:40:30.123684+0000","last_undegraded":"2023-10-14T16:40:30.123684+0000","last_fullsized":"2023-10-14T16:40:30.123684+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780736+0000","last_change":"2023-10-14T16:40:24.763894+0000","last_active":"2023-10-14T16:40:26.780736+0000","last_peered":"2023-10-14T16:40:26.780736+0000","last_clean":"2023-10-14T16:40:26.780736+0000","last_became_active":"2023-10-14T16:40:24.762343+0000","last_became_peered":"2023-10-14T16:40:24.762343+0000","last_unstale":"2023-10-14T16:40:26.780736+0000","last_undegraded":"2023-10-14T16:40:26.780736+0000","last_fullsized":"2023-10-14T16:40:26.780736+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774426+0000","last_change":"2023-10-14T16:40:24.775055+0000","last_active":"2023-10-14T16:40:26.774426+0000","last_peered":"2023-10-14T16:40:26.774426+0000","last_clean":"2023-10-14T16:40:26.774426+0000","last_became_active":"2023-10-14T16:40:24.774256+0000","last_became_peered":"2023-10-14T16:40:24.774256+0000","last_unstale":"2023-10-14T16:40:26.774426+0000","last_undegraded":"2023-10-14T16:40:26.774426+0000","last_fullsized":"2023-10-14T16:40:26.774426+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774375+0000","last_change":"2023-10-14T16:40:25.781085+0000","last_active":"2023-10-14T16:40:26.774375+0000","last_peered":"2023-10-14T16:40:26.774375+0000","last_clean":"2023-10-14T16:40:26.774375+0000","last_became_active":"2023-10-14T16:40:25.780746+0000","last_became_peered":"2023-10-14T16:40:25.780746+0000","last_unstale":"2023-10-14T16:40:26.774375+0000","last_undegraded":"2023-10-14T16:40:26.774375+0000","last_fullsized":"2023-10-14T16:40:26.774375+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351413+0000","last_change":"2023-10-14T16:40:24.763402+0000","last_active":"2023-10-14T16:40:28.351413+0000","last_peered":"2023-10-14T16:40:28.351413+0000","last_clean":"2023-10-14T16:40:28.351413+0000","last_became_active":"2023-10-14T16:40:24.763093+0000","last_became_peered":"2023-10-14T16:40:24.763093+0000","last_unstale":"2023-10-14T16:40:28.351413+0000","last_undegraded":"2023-10-14T16:40:28.351413+0000","last_fullsized":"2023-10-14T16:40:28.351413+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120612+0000","last_change":"2023-10-14T16:40:25.780063+0000","last_active":"2023-10-14T16:40:30.120612+0000","last_peered":"2023-10-14T16:40:30.120612+0000","last_clean":"2023-10-14T16:40:30.120612+0000","last_became_active":"2023-10-14T16:40:25.776862+0000","last_became_peered":"2023-10-14T16:40:25.776862+0000","last_unstale":"2023-10-14T16:40:30.120612+0000","last_undegraded":"2023-10-14T16:40:30.120612+0000","last_fullsized":"2023-10-14T16:40:30.120612+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781765+0000","last_change":"2023-10-14T16:40:24.768554+0000","last_active":"2023-10-14T16:40:26.781765+0000","last_peered":"2023-10-14T16:40:26.781765+0000","last_clean":"2023-10-14T16:40:26.781765+0000","last_became_active":"2023-10-14T16:40:24.768232+0000","last_became_peered":"2023-10-14T16:40:24.768232+0000","last_unstale":"2023-10-14T16:40:26.781765+0000","last_undegraded":"2023-10-14T16:40:26.781765+0000","last_fullsized":"2023-10-14T16:40:26.781765+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:59.941812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485534+0000","last_change":"2023-10-14T16:40:25.778676+0000","last_active":"2023-10-14T16:40:29.485534+0000","last_peered":"2023-10-14T16:40:29.485534+0000","last_clean":"2023-10-14T16:40:29.485534+0000","last_became_active":"2023-10-14T16:40:25.778295+0000","last_became_peered":"2023-10-14T16:40:25.778295+0000","last_unstale":"2023-10-14T16:40:29.485534+0000","last_undegraded":"2023-10-14T16:40:29.485534+0000","last_fullsized":"2023-10-14T16:40:29.485534+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:25.312672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351788+0000","last_change":"2023-10-14T16:40:24.764655+0000","last_active":"2023-10-14T16:40:28.351788+0000","last_peered":"2023-10-14T16:40:28.351788+0000","last_clean":"2023-10-14T16:40:28.351788+0000","last_became_active":"2023-10-14T16:40:24.764327+0000","last_became_peered":"2023-10-14T16:40:24.764327+0000","last_unstale":"2023-10-14T16:40:28.351788+0000","last_undegraded":"2023-10-14T16:40:28.351788+0000","last_fullsized":"2023-10-14T16:40:28.351788+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.733975+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123336+0000","last_change":"2023-10-14T16:40:25.783442+0000","last_active":"2023-10-14T16:40:30.123336+0000","last_peered":"2023-10-14T16:40:30.123336+0000","last_clean":"2023-10-14T16:40:30.123336+0000","last_became_active":"2023-10-14T16:40:25.783169+0000","last_became_peered":"2023-10-14T16:40:25.783169+0000","last_unstale":"2023-10-14T16:40:30.123336+0000","last_undegraded":"2023-10-14T16:40:30.123336+0000","last_fullsized":"2023-10-14T16:40:30.123336+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":15,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359494+0000","last_change":"2023-10-14T16:40:24.767382+0000","last_active":"2023-10-14T16:40:28.359494+0000","last_peered":"2023-10-14T16:40:28.359494+0000","last_clean":"2023-10-14T16:40:28.359494+0000","last_became_active":"2023-10-14T16:40:24.767094+0000","last_became_peered":"2023-10-14T16:40:24.767094+0000","last_unstale":"2023-10-14T16:40:28.359494+0000","last_undegraded":"2023-10-14T16:40:28.359494+0000","last_fullsized":"2023-10-14T16:40:28.359494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778953+0000","last_change":"2023-10-14T16:40:25.776539+0000","last_active":"2023-10-14T16:40:26.778953+0000","last_peered":"2023-10-14T16:40:26.778953+0000","last_clean":"2023-10-14T16:40:26.778953+0000","last_became_active":"2023-10-14T16:40:25.776126+0000","last_became_peered":"2023-10-14T16:40:25.776126+0000","last_unstale":"2023-10-14T16:40:26.778953+0000","last_undegraded":"2023-10-14T16:40:26.778953+0000","last_fullsized":"2023-10-14T16:40:26.778953+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:27.121567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781420+0000","last_change":"2023-10-14T16:40:24.768941+0000","last_active":"2023-10-14T16:40:26.781420+0000","last_peered":"2023-10-14T16:40:26.781420+0000","last_clean":"2023-10-14T16:40:26.781420+0000","last_became_active":"2023-10-14T16:40:24.767075+0000","last_became_peered":"2023-10-14T16:40:24.767075+0000","last_unstale":"2023-10-14T16:40:26.781420+0000","last_undegraded":"2023-10-14T16:40:26.781420+0000","last_fullsized":"2023-10-14T16:40:26.781420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083537+0000","last_change":"2023-10-14T16:40:25.791099+0000","last_active":"2023-10-14T16:40:29.083537+0000","last_peered":"2023-10-14T16:40:29.083537+0000","last_clean":"2023-10-14T16:40:29.083537+0000","last_became_active":"2023-10-14T16:40:25.790819+0000","last_became_peered":"2023-10-14T16:40:25.790819+0000","last_unstale":"2023-10-14T16:40:29.083537+0000","last_undegraded":"2023-10-14T16:40:29.083537+0000","last_fullsized":"2023-10-14T16:40:29.083537+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:56.842342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.356872+0000","last_change":"2023-10-14T16:40:24.760669+0000","last_active":"2023-10-14T16:40:28.356872+0000","last_peered":"2023-10-14T16:40:28.356872+0000","last_clean":"2023-10-14T16:40:28.356872+0000","last_became_active":"2023-10-14T16:40:24.760395+0000","last_became_peered":"2023-10-14T16:40:24.760395+0000","last_unstale":"2023-10-14T16:40:28.356872+0000","last_undegraded":"2023-10-14T16:40:28.356872+0000","last_fullsized":"2023-10-14T16:40:28.356872+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:06:26.110068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122343+0000","last_change":"2023-10-14T16:40:25.779561+0000","last_active":"2023-10-14T16:40:30.122343+0000","last_peered":"2023-10-14T16:40:30.122343+0000","last_clean":"2023-10-14T16:40:30.122343+0000","last_became_active":"2023-10-14T16:40:25.779246+0000","last_became_peered":"2023-10-14T16:40:25.779246+0000","last_unstale":"2023-10-14T16:40:30.122343+0000","last_undegraded":"2023-10-14T16:40:30.122343+0000","last_fullsized":"2023-10-14T16:40:30.122343+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:34.704624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351501+0000","last_change":"2023-10-14T16:40:24.764378+0000","last_active":"2023-10-14T16:40:28.351501+0000","last_peered":"2023-10-14T16:40:28.351501+0000","last_clean":"2023-10-14T16:40:28.351501+0000","last_became_active":"2023-10-14T16:40:24.764094+0000","last_became_peered":"2023-10-14T16:40:24.764094+0000","last_unstale":"2023-10-14T16:40:28.351501+0000","last_undegraded":"2023-10-14T16:40:28.351501+0000","last_fullsized":"2023-10-14T16:40:28.351501+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:55.307930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083734+0000","last_change":"2023-10-14T16:40:25.782667+0000","last_active":"2023-10-14T16:40:29.083734+0000","last_peered":"2023-10-14T16:40:29.083734+0000","last_clean":"2023-10-14T16:40:29.083734+0000","last_became_active":"2023-10-14T16:40:25.782311+0000","last_became_peered":"2023-10-14T16:40:25.782311+0000","last_unstale":"2023-10-14T16:40:29.083734+0000","last_undegraded":"2023-10-14T16:40:29.083734+0000","last_fullsized":"2023-10-14T16:40:29.083734+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:35.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780460+0000","last_change":"2023-10-14T16:40:24.765776+0000","last_active":"2023-10-14T16:40:26.780460+0000","last_peered":"2023-10-14T16:40:26.780460+0000","last_clean":"2023-10-14T16:40:26.780460+0000","last_became_active":"2023-10-14T16:40:24.761753+0000","last_became_peered":"2023-10-14T16:40:24.761753+0000","last_unstale":"2023-10-14T16:40:26.780460+0000","last_undegraded":"2023-10-14T16:40:26.780460+0000","last_fullsized":"2023-10-14T16:40:26.780460+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780507+0000","last_change":"2023-10-14T16:40:25.761768+0000","last_active":"2023-10-14T16:40:26.780507+0000","last_peered":"2023-10-14T16:40:26.780507+0000","last_clean":"2023-10-14T16:40:26.780507+0000","last_became_active":"2023-10-14T16:40:25.761374+0000","last_became_peered":"2023-10-14T16:40:25.761374+0000","last_unstale":"2023-10-14T16:40:26.780507+0000","last_undegraded":"2023-10-14T16:40:26.780507+0000","last_fullsized":"2023-10-14T16:40:26.780507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:43.782644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775466+0000","last_change":"2023-10-14T16:40:24.758887+0000","last_active":"2023-10-14T16:40:27.775466+0000","last_peered":"2023-10-14T16:40:27.775466+0000","last_clean":"2023-10-14T16:40:27.775466+0000","last_became_active":"2023-10-14T16:40:24.758086+0000","last_became_peered":"2023-10-14T16:40:24.758086+0000","last_unstale":"2023-10-14T16:40:27.775466+0000","last_undegraded":"2023-10-14T16:40:27.775466+0000","last_fullsized":"2023-10-14T16:40:27.775466+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:03.099034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083022+0000","last_change":"2023-10-14T16:40:25.793027+0000","last_active":"2023-10-14T16:40:29.083022+0000","last_peered":"2023-10-14T16:40:29.083022+0000","last_clean":"2023-10-14T16:40:29.083022+0000","last_became_active":"2023-10-14T16:40:25.791542+0000","last_became_peered":"2023-10-14T16:40:25.791542+0000","last_unstale":"2023-10-14T16:40:29.083022+0000","last_undegraded":"2023-10-14T16:40:29.083022+0000","last_fullsized":"2023-10-14T16:40:29.083022+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:12:59.334588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.779724+0000","last_change":"2023-10-14T16:40:24.763877+0000","last_active":"2023-10-14T16:40:26.779724+0000","last_peered":"2023-10-14T16:40:26.779724+0000","last_clean":"2023-10-14T16:40:26.779724+0000","last_became_active":"2023-10-14T16:40:24.763637+0000","last_became_peered":"2023-10-14T16:40:24.763637+0000","last_unstale":"2023-10-14T16:40:26.779724+0000","last_undegraded":"2023-10-14T16:40:26.779724+0000","last_fullsized":"2023-10-14T16:40:26.779724+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083503+0000","last_change":"2023-10-14T16:40:25.792192+0000","last_active":"2023-10-14T16:40:29.083503+0000","last_peered":"2023-10-14T16:40:29.083503+0000","last_clean":"2023-10-14T16:40:29.083503+0000","last_became_active":"2023-10-14T16:40:25.791748+0000","last_became_peered":"2023-10-14T16:40:25.791748+0000","last_unstale":"2023-10-14T16:40:29.083503+0000","last_undegraded":"2023-10-14T16:40:29.083503+0000","last_fullsized":"2023-10-14T16:40:29.083503+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:28.478063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.776156+0000","last_change":"2023-10-14T16:40:24.766861+0000","last_active":"2023-10-14T16:40:27.776156+0000","last_peered":"2023-10-14T16:40:27.776156+0000","last_clean":"2023-10-14T16:40:27.776156+0000","last_became_active":"2023-10-14T16:40:24.766319+0000","last_became_peered":"2023-10-14T16:40:24.766319+0000","last_unstale":"2023-10-14T16:40:27.776156+0000","last_undegraded":"2023-10-14T16:40:27.776156+0000","last_fullsized":"2023-10-14T16:40:27.776156+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.084450+0000","last_change":"2023-10-14T16:40:25.782879+0000","last_active":"2023-10-14T16:40:29.084450+0000","last_peered":"2023-10-14T16:40:29.084450+0000","last_clean":"2023-10-14T16:40:29.084450+0000","last_became_active":"2023-10-14T16:40:25.782368+0000","last_became_peered":"2023-10-14T16:40:25.782368+0000","last_unstale":"2023-10-14T16:40:29.084450+0000","last_undegraded":"2023-10-14T16:40:29.084450+0000","last_fullsized":"2023-10-14T16:40:29.084450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:15:00.760301+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":13,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359525+0000","last_change":"2023-10-14T16:40:24.763429+0000","last_active":"2023-10-14T16:40:28.359525+0000","last_peered":"2023-10-14T16:40:28.359525+0000","last_clean":"2023-10-14T16:40:28.359525+0000","last_became_active":"2023-10-14T16:40:24.761118+0000","last_became_peered":"2023-10-14T16:40:24.761118+0000","last_unstale":"2023-10-14T16:40:28.359525+0000","last_undegraded":"2023-10-14T16:40:28.359525+0000","last_fullsized":"2023-10-14T16:40:28.359525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485360+0000","last_change":"2023-10-14T16:40:25.784774+0000","last_active":"2023-10-14T16:40:29.485360+0000","last_peered":"2023-10-14T16:40:29.485360+0000","last_clean":"2023-10-14T16:40:29.485360+0000","last_became_active":"2023-10-14T16:40:25.784416+0000","last_became_peered":"2023-10-14T16:40:25.784416+0000","last_unstale":"2023-10-14T16:40:29.485360+0000","last_undegraded":"2023-10-14T16:40:29.485360+0000","last_fullsized":"2023-10-14T16:40:29.485360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:35.102976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775852+0000","last_change":"2023-10-14T16:40:24.767764+0000","last_active":"2023-10-14T16:40:27.775852+0000","last_peered":"2023-10-14T16:40:27.775852+0000","last_clean":"2023-10-14T16:40:27.775852+0000","last_became_active":"2023-10-14T16:40:24.767467+0000","last_became_peered":"2023-10-14T16:40:24.767467+0000","last_unstale":"2023-10-14T16:40:27.775852+0000","last_undegraded":"2023-10-14T16:40:27.775852+0000","last_fullsized":"2023-10-14T16:40:27.775852+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.083636+0000","last_change":"2023-10-14T16:40:25.792940+0000","last_active":"2023-10-14T16:40:29.083636+0000","last_peered":"2023-10-14T16:40:29.083636+0000","last_clean":"2023-10-14T16:40:29.083636+0000","last_became_active":"2023-10-14T16:40:25.790465+0000","last_became_peered":"2023-10-14T16:40:25.790465+0000","last_unstale":"2023-10-14T16:40:29.083636+0000","last_undegraded":"2023-10-14T16:40:29.083636+0000","last_fullsized":"2023-10-14T16:40:29.083636+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:50.009933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":165,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:25.765033+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:40:25.765033+0000","last_peered":"2023-10-14T16:40:25.765033+0000","last_clean":"2023-10-14T16:40:25.765033+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:40:25.765033+0000","last_undegraded":"2023-10-14T16:40:25.765033+0000","last_fullsized":"2023-10-14T16:40:25.765033+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351635+0000","last_change":"2023-10-14T16:40:24.765633+0000","last_active":"2023-10-14T16:40:28.351635+0000","last_peered":"2023-10-14T16:40:28.351635+0000","last_clean":"2023-10-14T16:40:28.351635+0000","last_became_active":"2023-10-14T16:40:24.760310+0000","last_became_peered":"2023-10-14T16:40:24.760310+0000","last_unstale":"2023-10-14T16:40:28.351635+0000","last_undegraded":"2023-10-14T16:40:28.351635+0000","last_fullsized":"2023-10-14T16:40:28.351635+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:42.771145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775084+0000","last_change":"2023-10-14T16:40:25.783431+0000","last_active":"2023-10-14T16:40:26.775084+0000","last_peered":"2023-10-14T16:40:26.775084+0000","last_clean":"2023-10-14T16:40:26.775084+0000","last_became_active":"2023-10-14T16:40:25.782668+0000","last_became_peered":"2023-10-14T16:40:25.782668+0000","last_unstale":"2023-10-14T16:40:26.775084+0000","last_undegraded":"2023-10-14T16:40:26.775084+0000","last_fullsized":"2023-10-14T16:40:26.775084+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:13.705915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340850+0000","last_change":"2023-10-14T16:40:24.774157+0000","last_active":"2023-10-14T16:40:28.340850+0000","last_peered":"2023-10-14T16:40:28.340850+0000","last_clean":"2023-10-14T16:40:28.340850+0000","last_became_active":"2023-10-14T16:40:24.773773+0000","last_became_peered":"2023-10-14T16:40:24.773773+0000","last_unstale":"2023-10-14T16:40:28.340850+0000","last_undegraded":"2023-10-14T16:40:28.340850+0000","last_fullsized":"2023-10-14T16:40:28.340850+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122220+0000","last_change":"2023-10-14T16:40:25.772755+0000","last_active":"2023-10-14T16:40:30.122220+0000","last_peered":"2023-10-14T16:40:30.122220+0000","last_clean":"2023-10-14T16:40:30.122220+0000","last_became_active":"2023-10-14T16:40:25.772431+0000","last_became_peered":"2023-10-14T16:40:25.772431+0000","last_unstale":"2023-10-14T16:40:30.122220+0000","last_undegraded":"2023-10-14T16:40:30.122220+0000","last_fullsized":"2023-10-14T16:40:30.122220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:39.227344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:27.775795+0000","last_change":"2023-10-14T16:40:24.770677+0000","last_active":"2023-10-14T16:40:27.775795+0000","last_peered":"2023-10-14T16:40:27.775795+0000","last_clean":"2023-10-14T16:40:27.775795+0000","last_became_active":"2023-10-14T16:40:24.770259+0000","last_became_peered":"2023-10-14T16:40:24.770259+0000","last_unstale":"2023-10-14T16:40:27.775795+0000","last_undegraded":"2023-10-14T16:40:27.775795+0000","last_fullsized":"2023-10-14T16:40:27.775795+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120889+0000","last_change":"2023-10-14T16:40:25.768484+0000","last_active":"2023-10-14T16:40:30.120889+0000","last_peered":"2023-10-14T16:40:30.120889+0000","last_clean":"2023-10-14T16:40:30.120889+0000","last_became_active":"2023-10-14T16:40:25.767865+0000","last_became_peered":"2023-10-14T16:40:25.767865+0000","last_unstale":"2023-10-14T16:40:30.120889+0000","last_undegraded":"2023-10-14T16:40:30.120889+0000","last_fullsized":"2023-10-14T16:40:30.120889+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:30:01.997983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.351522+0000","last_change":"2023-10-14T16:40:24.762562+0000","last_active":"2023-10-14T16:40:28.351522+0000","last_peered":"2023-10-14T16:40:28.351522+0000","last_clean":"2023-10-14T16:40:28.351522+0000","last_became_active":"2023-10-14T16:40:24.761399+0000","last_became_peered":"2023-10-14T16:40:24.761399+0000","last_unstale":"2023-10-14T16:40:28.351522+0000","last_undegraded":"2023-10-14T16:40:28.351522+0000","last_fullsized":"2023-10-14T16:40:28.351522+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:40.569040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778450+0000","last_change":"2023-10-14T16:40:25.767520+0000","last_active":"2023-10-14T16:40:26.778450+0000","last_peered":"2023-10-14T16:40:26.778450+0000","last_clean":"2023-10-14T16:40:26.778450+0000","last_became_active":"2023-10-14T16:40:25.765947+0000","last_became_peered":"2023-10-14T16:40:25.765947+0000","last_unstale":"2023-10-14T16:40:26.778450+0000","last_undegraded":"2023-10-14T16:40:26.778450+0000","last_fullsized":"2023-10-14T16:40:26.778450+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:02.971690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359389+0000","last_change":"2023-10-14T16:40:24.758028+0000","last_active":"2023-10-14T16:40:28.359389+0000","last_peered":"2023-10-14T16:40:28.359389+0000","last_clean":"2023-10-14T16:40:28.359389+0000","last_became_active":"2023-10-14T16:40:24.757727+0000","last_became_peered":"2023-10-14T16:40:24.757727+0000","last_unstale":"2023-10-14T16:40:28.359389+0000","last_undegraded":"2023-10-14T16:40:28.359389+0000","last_fullsized":"2023-10-14T16:40:28.359389+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485567+0000","last_change":"2023-10-14T16:40:25.789247+0000","last_active":"2023-10-14T16:40:29.485567+0000","last_peered":"2023-10-14T16:40:29.485567+0000","last_clean":"2023-10-14T16:40:29.485567+0000","last_became_active":"2023-10-14T16:40:25.788936+0000","last_became_peered":"2023-10-14T16:40:25.788936+0000","last_unstale":"2023-10-14T16:40:29.485567+0000","last_undegraded":"2023-10-14T16:40:29.485567+0000","last_fullsized":"2023-10-14T16:40:29.485567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:31.604878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341168+0000","last_change":"2023-10-14T16:40:24.776223+0000","last_active":"2023-10-14T16:40:28.341168+0000","last_peered":"2023-10-14T16:40:28.341168+0000","last_clean":"2023-10-14T16:40:28.341168+0000","last_became_active":"2023-10-14T16:40:24.775651+0000","last_became_peered":"2023-10-14T16:40:24.775651+0000","last_unstale":"2023-10-14T16:40:28.341168+0000","last_undegraded":"2023-10-14T16:40:28.341168+0000","last_fullsized":"2023-10-14T16:40:28.341168+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:16.308104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.120628+0000","last_change":"2023-10-14T16:40:25.768945+0000","last_active":"2023-10-14T16:40:30.120628+0000","last_peered":"2023-10-14T16:40:30.120628+0000","last_clean":"2023-10-14T16:40:30.120628+0000","last_became_active":"2023-10-14T16:40:25.768628+0000","last_became_peered":"2023-10-14T16:40:25.768628+0000","last_unstale":"2023-10-14T16:40:30.120628+0000","last_undegraded":"2023-10-14T16:40:30.120628+0000","last_fullsized":"2023-10-14T16:40:30.120628+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:41.580539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.121273+0000","last_change":"2023-10-14T16:40:24.768276+0000","last_active":"2023-10-14T16:40:30.121273+0000","last_peered":"2023-10-14T16:40:30.121273+0000","last_clean":"2023-10-14T16:40:30.121273+0000","last_became_active":"2023-10-14T16:40:24.766828+0000","last_became_peered":"2023-10-14T16:40:24.766828+0000","last_unstale":"2023-10-14T16:40:30.121273+0000","last_undegraded":"2023-10-14T16:40:30.121273+0000","last_fullsized":"2023-10-14T16:40:30.121273+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775754+0000","last_change":"2023-10-14T16:40:25.779777+0000","last_active":"2023-10-14T16:40:26.775754+0000","last_peered":"2023-10-14T16:40:26.775754+0000","last_clean":"2023-10-14T16:40:26.775754+0000","last_became_active":"2023-10-14T16:40:25.779380+0000","last_became_peered":"2023-10-14T16:40:25.779380+0000","last_unstale":"2023-10-14T16:40:26.775754+0000","last_undegraded":"2023-10-14T16:40:26.775754+0000","last_fullsized":"2023-10-14T16:40:26.775754+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:27:09.924411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.781895+0000","last_change":"2023-10-14T16:40:24.773797+0000","last_active":"2023-10-14T16:40:26.781895+0000","last_peered":"2023-10-14T16:40:26.781895+0000","last_clean":"2023-10-14T16:40:26.781895+0000","last_became_active":"2023-10-14T16:40:24.773498+0000","last_became_peered":"2023-10-14T16:40:24.773498+0000","last_unstale":"2023-10-14T16:40:26.781895+0000","last_undegraded":"2023-10-14T16:40:26.781895+0000","last_fullsized":"2023-10-14T16:40:26.781895+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:41.208561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.122877+0000","last_change":"2023-10-14T16:40:25.766374+0000","last_active":"2023-10-14T16:40:30.122877+0000","last_peered":"2023-10-14T16:40:30.122877+0000","last_clean":"2023-10-14T16:40:30.122877+0000","last_became_active":"2023-10-14T16:40:25.764271+0000","last_became_peered":"2023-10-14T16:40:25.764271+0000","last_unstale":"2023-10-14T16:40:30.122877+0000","last_undegraded":"2023-10-14T16:40:30.122877+0000","last_fullsized":"2023-10-14T16:40:30.122877+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:17.319603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340815+0000","last_change":"2023-10-14T16:40:24.775192+0000","last_active":"2023-10-14T16:40:28.340815+0000","last_peered":"2023-10-14T16:40:28.340815+0000","last_clean":"2023-10-14T16:40:28.340815+0000","last_became_active":"2023-10-14T16:40:24.774186+0000","last_became_peered":"2023-10-14T16:40:24.774186+0000","last_unstale":"2023-10-14T16:40:28.340815+0000","last_undegraded":"2023-10-14T16:40:28.340815+0000","last_fullsized":"2023-10-14T16:40:28.340815+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:40:01.960191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.778279+0000","last_change":"2023-10-14T16:40:25.769524+0000","last_active":"2023-10-14T16:40:26.778279+0000","last_peered":"2023-10-14T16:40:26.778279+0000","last_clean":"2023-10-14T16:40:26.778279+0000","last_became_active":"2023-10-14T16:40:25.769237+0000","last_became_peered":"2023-10-14T16:40:25.769237+0000","last_unstale":"2023-10-14T16:40:26.778279+0000","last_undegraded":"2023-10-14T16:40:26.778279+0000","last_fullsized":"2023-10-14T16:40:26.778279+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:06.059757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359223+0000","last_change":"2023-10-14T16:40:24.765632+0000","last_active":"2023-10-14T16:40:28.359223+0000","last_peered":"2023-10-14T16:40:28.359223+0000","last_clean":"2023-10-14T16:40:28.359223+0000","last_became_active":"2023-10-14T16:40:24.765393+0000","last_became_peered":"2023-10-14T16:40:24.765393+0000","last_unstale":"2023-10-14T16:40:28.359223+0000","last_undegraded":"2023-10-14T16:40:28.359223+0000","last_fullsized":"2023-10-14T16:40:28.359223+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:24.098376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.774557+0000","last_change":"2023-10-14T16:40:25.766437+0000","last_active":"2023-10-14T16:40:26.774557+0000","last_peered":"2023-10-14T16:40:26.774557+0000","last_clean":"2023-10-14T16:40:26.774557+0000","last_became_active":"2023-10-14T16:40:25.765798+0000","last_became_peered":"2023-10-14T16:40:25.765798+0000","last_unstale":"2023-10-14T16:40:26.774557+0000","last_undegraded":"2023-10-14T16:40:26.774557+0000","last_fullsized":"2023-10-14T16:40:26.774557+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.745474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":12,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.359248+0000","last_change":"2023-10-14T16:40:24.765207+0000","last_active":"2023-10-14T16:40:28.359248+0000","last_peered":"2023-10-14T16:40:28.359248+0000","last_clean":"2023-10-14T16:40:28.359248+0000","last_became_active":"2023-10-14T16:40:24.764610+0000","last_became_peered":"2023-10-14T16:40:24.764610+0000","last_unstale":"2023-10-14T16:40:28.359248+0000","last_undegraded":"2023-10-14T16:40:28.359248+0000","last_fullsized":"2023-10-14T16:40:28.359248+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:05.048258+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775031+0000","last_change":"2023-10-14T16:40:25.788202+0000","last_active":"2023-10-14T16:40:26.775031+0000","last_peered":"2023-10-14T16:40:26.775031+0000","last_clean":"2023-10-14T16:40:26.775031+0000","last_became_active":"2023-10-14T16:40:25.785927+0000","last_became_peered":"2023-10-14T16:40:25.785927+0000","last_unstale":"2023-10-14T16:40:26.775031+0000","last_undegraded":"2023-10-14T16:40:26.775031+0000","last_fullsized":"2023-10-14T16:40:26.775031+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:46.697224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.340988+0000","last_change":"2023-10-14T16:40:24.775618+0000","last_active":"2023-10-14T16:40:28.340988+0000","last_peered":"2023-10-14T16:40:28.340988+0000","last_clean":"2023-10-14T16:40:28.340988+0000","last_became_active":"2023-10-14T16:40:24.775346+0000","last_became_peered":"2023-10-14T16:40:24.775346+0000","last_unstale":"2023-10-14T16:40:28.340988+0000","last_undegraded":"2023-10-14T16:40:28.340988+0000","last_fullsized":"2023-10-14T16:40:28.340988+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:00.220013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.780727+0000","last_change":"2023-10-14T16:40:25.778910+0000","last_active":"2023-10-14T16:40:26.780727+0000","last_peered":"2023-10-14T16:40:26.780727+0000","last_clean":"2023-10-14T16:40:26.780727+0000","last_became_active":"2023-10-14T16:40:25.777772+0000","last_became_peered":"2023-10-14T16:40:25.777772+0000","last_unstale":"2023-10-14T16:40:26.780727+0000","last_undegraded":"2023-10-14T16:40:26.780727+0000","last_fullsized":"2023-10-14T16:40:26.780727+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485428+0000","last_change":"2023-10-14T16:40:25.779144+0000","last_active":"2023-10-14T16:40:29.485428+0000","last_peered":"2023-10-14T16:40:29.485428+0000","last_clean":"2023-10-14T16:40:29.485428+0000","last_became_active":"2023-10-14T16:40:25.778298+0000","last_became_peered":"2023-10-14T16:40:25.778298+0000","last_unstale":"2023-10-14T16:40:29.485428+0000","last_undegraded":"2023-10-14T16:40:29.485428+0000","last_fullsized":"2023-10-14T16:40:29.485428+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:21.372595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341014+0000","last_change":"2023-10-14T16:40:24.776824+0000","last_active":"2023-10-14T16:40:28.341014+0000","last_peered":"2023-10-14T16:40:28.341014+0000","last_clean":"2023-10-14T16:40:28.341014+0000","last_became_active":"2023-10-14T16:40:24.776368+0000","last_became_peered":"2023-10-14T16:40:24.776368+0000","last_unstale":"2023-10-14T16:40:28.341014+0000","last_undegraded":"2023-10-14T16:40:28.341014+0000","last_fullsized":"2023-10-14T16:40:28.341014+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:46.227289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:29.485000+0000","last_change":"2023-10-14T16:40:25.777666+0000","last_active":"2023-10-14T16:40:29.485000+0000","last_peered":"2023-10-14T16:40:29.485000+0000","last_clean":"2023-10-14T16:40:29.485000+0000","last_became_active":"2023-10-14T16:40:25.776881+0000","last_became_peered":"2023-10-14T16:40:25.776881+0000","last_unstale":"2023-10-14T16:40:29.485000+0000","last_undegraded":"2023-10-14T16:40:29.485000+0000","last_fullsized":"2023-10-14T16:40:29.485000+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:50.009933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":14,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.341121+0000","last_change":"2023-10-14T16:40:24.777751+0000","last_active":"2023-10-14T16:40:28.341121+0000","last_peered":"2023-10-14T16:40:28.341121+0000","last_clean":"2023-10-14T16:40:28.341121+0000","last_became_active":"2023-10-14T16:40:24.777483+0000","last_became_peered":"2023-10-14T16:40:24.777483+0000","last_unstale":"2023-10-14T16:40:28.341121+0000","last_undegraded":"2023-10-14T16:40:28.341121+0000","last_fullsized":"2023-10-14T16:40:28.341121+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:38.215845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:30.123205+0000","last_change":"2023-10-14T16:40:25.783100+0000","last_active":"2023-10-14T16:40:30.123205+0000","last_peered":"2023-10-14T16:40:30.123205+0000","last_clean":"2023-10-14T16:40:30.123205+0000","last_became_active":"2023-10-14T16:40:25.782769+0000","last_became_peered":"2023-10-14T16:40:25.782769+0000","last_unstale":"2023-10-14T16:40:30.123205+0000","last_undegraded":"2023-10-14T16:40:30.123205+0000","last_fullsized":"2023-10-14T16:40:30.123205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:56.319428+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":13,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775509+0000","last_change":"2023-10-14T16:40:24.768856+0000","last_active":"2023-10-14T16:40:26.775509+0000","last_peered":"2023-10-14T16:40:26.775509+0000","last_clean":"2023-10-14T16:40:26.775509+0000","last_became_active":"2023-10-14T16:40:24.768497+0000","last_became_peered":"2023-10-14T16:40:24.768497+0000","last_unstale":"2023-10-14T16:40:26.775509+0000","last_undegraded":"2023-10-14T16:40:26.775509+0000","last_fullsized":"2023-10-14T16:40:26.775509+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:33.693125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":11,"reported_epoch":20,"state":"active+clean","last_fresh":"2023-10-14T16:40:26.775555+0000","last_change":"2023-10-14T16:40:25.784770+0000","last_active":"2023-10-14T16:40:26.775555+0000","last_peered":"2023-10-14T16:40:26.775555+0000","last_clean":"2023-10-14T16:40:26.775555+0000","last_became_active":"2023-10-14T16:40:25.782997+0000","last_became_peered":"2023-10-14T16:40:25.782997+0000","last_unstale":"2023-10-14T16:40:26.775555+0000","last_undegraded":"2023-10-14T16:40:26.775555+0000","last_fullsized":"2023-10-14T16:40:26.775555+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:24.722807+0000","last_clean_scrub_stamp":"2023-10-14T16:40:24.722807+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:22.951817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":16,"reported_epoch":19,"state":"active+clean","last_fresh":"2023-10-14T16:40:28.356987+0000","last_change":"2023-10-14T16:40:24.772362+0000","last_active":"2023-10-14T16:40:28.356987+0000","last_peered":"2023-10-14T16:40:28.356987+0000","last_clean":"2023-10-14T16:40:28.356987+0000","last_became_active":"2023-10-14T16:40:24.772021+0000","last_became_peered":"2023-10-14T16:40:24.772021+0000","last_unstale":"2023-10-14T16:40:28.356987+0000","last_undegraded":"2023-10-14T16:40:28.356987+0000","last_fullsized":"2023-10-14T16:40:28.356987+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:23.711309+0000","last_clean_scrub_stamp":"2023-10-14T16:40:23.711309+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:20.213211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":661848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.7130000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.52}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.194}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88600000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99199999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.103}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2430000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.339}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3280000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.6539999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.8319999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.2529999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.202}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":70988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":3.2000000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.1600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76700000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.044}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.081}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6080000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.4930000000000001}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":217088,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.3810000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.038}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.248}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98899999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1459999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3620000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":217088,"data_stored":72510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55800000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83899999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1850000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96099999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98699999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31384,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340456,"statfs":{"total":96636764160,"available":96604626944,"internally_reserved":0,"allocated":204800,"data_stored":70898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42299999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65200000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93700000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83499999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":212992,"data_stored":71058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.96799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3120000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.5339999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.5789999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.6639999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":4.6360000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":2.452}]}]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31972,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339868,"statfs":{"total":96636764160,"available":96604024832,"internally_reserved":0,"allocated":811008,"data_stored":661904,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64800000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93600000000000005}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.147}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89400000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0860000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.3100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.425}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:40:36.693 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T16:40:36.693 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-14T16:40:36.694 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-10-14T16:40:36.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-10-14T16:40:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.954+0000 7f6ef0940640 1 Processor -- start 2023-10-14T16:40:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.954+0000 7f6ef0940640 1 -- start start 2023-10-14T16:40:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.955+0000 7f6ef0940640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6eec15cbe0 0x7f6eec0d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.955+0000 7f6ef0940640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6eec0d2430 0x7f6eec0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.955+0000 7f6ef0940640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0d0b70 con 0x7f6eec0cd980 2023-10-14T16:40:36.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.955+0000 7f6ef0940640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec0d28a0 con 0x7f6eec0d2430 2023-10-14T16:40:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.955+0000 7f6ef0940640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec0d0e50 con 0x7f6eec15cbe0 2023-10-14T16:40:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6eec0d2430 0x7f6eec0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eeb59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6eec0cd980 0x7f6eec0b4530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47906/0 (socket says 172.21.15.29:47906) 2023-10-14T16:40:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6eec0d2430 0x7f6eec0cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36834/0 (socket says 172.21.15.29:36834) 2023-10-14T16:40:36.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eeb59e640 1 -- 172.21.15.29:0/383446400 learned_addr learned my addr 172.21.15.29:0/383446400 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eead9d640 1 --2- 172.21.15.29:0/383446400 >> v2:172.21.15.186:3300/0 conn(0x7f6eec15cbe0 0x7f6eec0d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 -- 172.21.15.29:0/383446400 >> v2:172.21.15.186:3300/0 conn(0x7f6eec15cbe0 msgr2=0x7f6eec0d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 --2- 172.21.15.29:0/383446400 >> v2:172.21.15.186:3300/0 conn(0x7f6eec15cbe0 0x7f6eec0d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 -- 172.21.15.29:0/383446400 >> v1:172.21.15.29:6789/0 conn(0x7f6eec0cd980 legacy=0x7f6eec0b4530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6eea59c640 1 -- 172.21.15.29:0/383446400 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec0d1130 con 0x7f6eec0d2430 2023-10-14T16:40:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.956+0000 7f6ee9d9b640 1 -- 172.21.15.29:0/383446400 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 681027477 0 0) 0x7f6eec0d0b70 con 0x7f6eec0cd980 2023-10-14T16:40:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.957+0000 7f6eea59c640 1 --2- 172.21.15.29:0/383446400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0d2430 0x7f6eec0cd440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6eec067320 tx=0x7f6ed402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fb68947931e52d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ee9d9b640 1 -- 172.21.15.29:0/383446400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed402cd00 con 0x7f6eec0d2430 2023-10-14T16:40:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ee9d9b640 1 -- 172.21.15.29:0/383446400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed402ce60 con 0x7f6eec0d2430 2023-10-14T16:40:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ee9d9b640 1 -- 172.21.15.29:0/383446400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ed4034660 con 0x7f6eec0d2430 2023-10-14T16:40:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 -- 172.21.15.29:0/383446400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0d2430 msgr2=0x7f6eec0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 --2- 172.21.15.29:0/383446400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0d2430 0x7f6eec0cd440 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6eec067320 tx=0x7f6ed402bf50 comp rx=0 tx=0).stop 2023-10-14T16:40:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 -- 172.21.15.29:0/383446400 shutdown_connections 2023-10-14T16:40:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 --2- 172.21.15.29:0/383446400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0d2430 0x7f6eec0cd440 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 --2- 172.21.15.29:0/383446400 >> v2:172.21.15.186:3300/0 conn(0x7f6eec15cbe0 0x7f6eec0d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 -- 172.21.15.29:0/383446400 >> 172.21.15.29:0/383446400 conn(0x7f6eec0d2ab0 msgr2=0x7f6eec0d2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 -- 172.21.15.29:0/383446400 shutdown_connections 2023-10-14T16:40:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.958+0000 7f6ef0940640 1 -- 172.21.15.29:0/383446400 wait complete. 2023-10-14T16:40:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.960+0000 7f6ef0940640 1 Processor -- start 2023-10-14T16:40:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.960+0000 7f6ef0940640 1 -- start start 2023-10-14T16:40:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6eec0d2430 0x7f6eec0b44a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6eec15cbe0 0x7f6eec0ca500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b1230 con 0x7f6eec0cd980 2023-10-14T16:40:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b36e0 con 0x7f6eec15cbe0 2023-10-14T16:40:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6ef0940640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec0b39c0 con 0x7f6eec0d2430 2023-10-14T16:40:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.961+0000 7f6eead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36846/0 (socket says 172.21.15.29:36846) 2023-10-14T16:40:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 -- 172.21.15.29:0/3948146506 learned_addr learned my addr 172.21.15.29:0/3948146506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 -- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6eec0d2430 msgr2=0x7f6eec0b44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6eec0d2430 0x7f6eec0b44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 -- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6eec15cbe0 msgr2=0x7f6eec0ca500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6eec15cbe0 0x7f6eec0ca500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 -- 172.21.15.29:0/3948146506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4009a70 con 0x7f6eec0cd980 2023-10-14T16:40:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6eead9d640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00d800 tx=0x7f6edc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.962+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00b6c0 con 0x7f6eec0cd980 2023-10-14T16:40:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.963+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc00b820 con 0x7f6eec0cd980 2023-10-14T16:40:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.963+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec0b3ca0 con 0x7f6eec0cd980 2023-10-14T16:40:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.963+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6edc00bbb0 con 0x7f6eec0cd980 2023-10-14T16:40:36.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.963+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec0caf80 con 0x7f6eec0cd980 2023-10-14T16:40:36.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.964+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec0b41c0 con 0x7f6eec0cd980 2023-10-14T16:40:36.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.965+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6edc00bd10 con 0x7f6eec0cd980 2023-10-14T16:40:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.966+0000 7f6ee37fe640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 0x7f6ecc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.966+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6edc089ca0 con 0x7f6eec0cd980 2023-10-14T16:40:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.966+0000 7f6eea59c640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 0x7f6ecc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:36.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.967+0000 7f6eea59c640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 0x7f6ecc06d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036000 tx=0x7f6ed402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:36.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:36.971+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6eec0b41c0 con 0x7f6eec0cd980 2023-10-14T16:40:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.197+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6eec0b41c0 con 0x7f6eec0cd980 2023-10-14T16:40:37.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.198+0000 7f6ee37fe640 1 -- 172.21.15.29:0/3948146506 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f6eec0b41c0 con 0x7f6eec0cd980 2023-10-14T16:40:37.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:37.199 INFO:teuthology.orchestra.run.smithi029.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-14T16:40:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 msgr2=0x7f6ecc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 0x7f6ecc06d330 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f6ed4036000 tx=0x7f6ed402e000 comp rx=0 tx=0).stop 2023-10-14T16:40:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 msgr2=0x7f6eec0b0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6edc00d800 tx=0x7f6edc00dcc0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 shutdown_connections 2023-10-14T16:40:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6ecc06ae70 0x7f6ecc06d330 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6eec15cbe0 0x7f6eec0ca500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6eec0d2430 0x7f6eec0b44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 --2- 172.21.15.29:0/3948146506 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6eec0cd980 0x7f6eec0b0cf0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 >> 172.21.15.29:0/3948146506 conn(0x7f6eec0d2ab0 msgr2=0x7f6eec0b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 shutdown_connections 2023-10-14T16:40:37.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.200+0000 7f6ef0940640 1 -- 172.21.15.29:0/3948146506 wait complete. 2023-10-14T16:40:37.219 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2023-10-14T16:40:37.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.432+0000 7f8a50c21640 1 Processor -- start 2023-10-14T16:40:37.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.432+0000 7f8a50c21640 1 -- start start 2023-10-14T16:40:37.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.433+0000 7f8a50c21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a4c167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.433+0000 7f8a50c21640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0d15a0 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.433+0000 7f8a50c21640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a4c0d1880 con 0x7f8a4c15cbe0 2023-10-14T16:40:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.433+0000 7f8a50c21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0d1b60 con 0x7f8a4c0b4c80 2023-10-14T16:40:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8a4c15cfc0 0x7f8a4c0b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47922/0 (socket says 172.21.15.29:47922) 2023-10-14T16:40:37.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4affd640 1 -- 172.21.15.29:0/3687776620 learned_addr learned my addr 172.21.15.29:0/3687776620 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4bfff640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a4c167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3077569307 0 0) 0x7f8a4c0d15a0 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a38003520 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.434+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3531531027 0 0) 0x7f8a38003520 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 msgr2=0x7f8a4c167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a4c167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 >> v1:172.21.15.186:6789/0 conn(0x7f8a4c15cbe0 legacy=0x7f8a4c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0d1e40 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8a340019f0 con 0x7f8a4c15cfc0 2023-10-14T16:40:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 >> v1:172.21.15.29:6789/0 conn(0x7f8a4c15cfc0 legacy=0x7f8a4c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a380039a0 0x7f8a38003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a38004d00 0x7f8a38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c0d1b60 con 0x7f8a380039a0 2023-10-14T16:40:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3800bac0 con 0x7f8a38004d00 2023-10-14T16:40:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a3800bc80 con 0x7f8a4c0b4c80 2023-10-14T16:40:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a380039a0 0x7f8a38003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 -- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 msgr2=0x7f8a380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4affd640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4bfff640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a38004d00 0x7f8a38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 -- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a38004d00 msgr2=0x7f8a38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:37.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a38004d00 0x7f8a38005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4b7fe640 1 -- 172.21.15.29:0/3687776620 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a38003520 con 0x7f8a380039a0 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.435+0000 7f8a4affd640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c0b4c80 0x7f8a380047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.436+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a380039a0 0x7f8a38003dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c001720 tx=0x7f8a380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cd7027b3c6cfdf8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0052d0 con 0x7f8a380039a0 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3c02e050 con 0x7f8a380039a0 2023-10-14T16:40:37.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a4a7fc640 1 -- 172.21.15.29:0/3687776620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a3c0035f0 con 0x7f8a380039a0 2023-10-14T16:40:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 -- 172.21.15.29:0/3687776620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a380039a0 msgr2=0x7f8a38003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 --2- 172.21.15.29:0/3687776620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a380039a0 0x7f8a38003dc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c001720 tx=0x7f8a380057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a4bfff640 1 -- 172.21.15.29:0/3687776620 reap_dead start 2023-10-14T16:40:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 -- 172.21.15.29:0/3687776620 shutdown_connections 2023-10-14T16:40:37.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 -- 172.21.15.29:0/3687776620 >> 172.21.15.29:0/3687776620 conn(0x7f8a4c0cf570 msgr2=0x7f8a4c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:37.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 -- 172.21.15.29:0/3687776620 shutdown_connections 2023-10-14T16:40:37.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.437+0000 7f8a50c21640 1 -- 172.21.15.29:0/3687776620 wait complete. 2023-10-14T16:40:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.439+0000 7f8a50c21640 1 Processor -- start 2023-10-14T16:40:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.439+0000 7f8a50c21640 1 -- start start 2023-10-14T16:40:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a380039a0 0x7f8a4c151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c1f5900 0x7f8a4c1f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c1f62c0 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c15ee30 con 0x7f8a380039a0 2023-10-14T16:40:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.440+0000 7f8a50c21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c152200 con 0x7f8a4c1f5900 2023-10-14T16:40:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36870/0 (socket says 172.21.15.29:36870) 2023-10-14T16:40:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 -- 172.21.15.29:0/152786486 learned_addr learned my addr 172.21.15.29:0/152786486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a380039a0 0x7f8a4c151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4bfff640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c1f5900 0x7f8a4c1f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 -- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c1f5900 msgr2=0x7f8a4c1f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c1f5900 0x7f8a4c1f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 -- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a380039a0 msgr2=0x7f8a4c151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a380039a0 0x7f8a4c151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.441+0000 7f8a4affd640 1 -- 172.21.15.29:0/152786486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a4c0ca130 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a4affd640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a340020f0 tx=0x7f8a3400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a34004d60 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a3400be60 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a340072d0 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c15f080 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.442+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c15f640 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.443+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c04e9b0 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.444+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8a34007430 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.445+0000 7f8a43fff640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 0x7f8a2c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.445+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8a34089660 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.445+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 0x7f8a2c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.446+0000 7f8a4b7fe640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 0x7f8a2c06d360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c033000 tx=0x7f8a3c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.449+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a34054700 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.645+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a4c155780 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.648+0000 7f8a43fff640 1 -- 172.21.15.29:0/152786486 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f8a34057d20 con 0x7f8a4c0c9bc0 2023-10-14T16:40:37.649 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:37.649 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 msgr2=0x7f8a2c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 0x7f8a2c06d360 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8a3c033000 tx=0x7f8a3c009030 comp rx=0 tx=0).stop 2023-10-14T16:40:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 msgr2=0x7f8a4c155240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8a340020f0 tx=0x7f8a3400b4a0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 shutdown_connections 2023-10-14T16:40:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8a2c06aea0 0x7f8a2c06d360 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a4c1f5900 0x7f8a4c1f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a4c0c9bc0 0x7f8a4c155240 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 --2- 172.21.15.29:0/152786486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a380039a0 0x7f8a4c151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 >> 172.21.15.29:0/152786486 conn(0x7f8a4c0cf570 msgr2=0x7f8a4c0b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 shutdown_connections 2023-10-14T16:40:37.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.651+0000 7f8a50c21640 1 -- 172.21.15.29:0/152786486 wait complete. 2023-10-14T16:40:37.672 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:40:37.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:40:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.903+0000 7f0740d67640 1 Processor -- start 2023-10-14T16:40:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.903+0000 7f0740d67640 1 -- start start 2023-10-14T16:40:37.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.905+0000 7f0740d67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c159ad0 0x7f073c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.905+0000 7f0740d67640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f073c15d650 0x7f073c06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.905+0000 7f0740d67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0cd260 con 0x7f073c06b3e0 2023-10-14T16:40:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.905+0000 7f0740d67640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c071470 con 0x7f073c15d650 2023-10-14T16:40:37.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.905+0000 7f0740d67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c15de30 con 0x7f073c159ad0 2023-10-14T16:40:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f073c15d650 0x7f073c06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f073c15d650 0x7f073c06aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:35416/0 (socket says 172.21.15.29:35416) 2023-10-14T16:40:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 -- 172.21.15.29:0/2211573773 learned_addr learned my addr 172.21.15.29:0/2211573773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f073b7fe640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c159ad0 0x7f073c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 -- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c159ad0 msgr2=0x7f073c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c159ad0 0x7f073c070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 -- 172.21.15.29:0/2211573773 >> v1:172.21.15.186:6789/0 conn(0x7f073c06b3e0 legacy=0x7f073c0c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.906+0000 7f0733fff640 1 -- 172.21.15.29:0/2211573773 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c15e110 con 0x7f073c15d650 2023-10-14T16:40:37.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.908+0000 7f0733fff640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c15d650 0x7f073c06aea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f072402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=655fc01cccfdcdf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.908+0000 7f073affd640 1 -- 172.21.15.29:0/2211573773 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724030840 con 0x7f073c15d650 2023-10-14T16:40:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.908+0000 7f073affd640 1 -- 172.21.15.29:0/2211573773 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07240309a0 con 0x7f073c15d650 2023-10-14T16:40:37.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.908+0000 7f073affd640 1 -- 172.21.15.29:0/2211573773 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724030d30 con 0x7f073c15d650 2023-10-14T16:40:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 -- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c15d650 msgr2=0x7f073c06aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c15d650 0x7f073c06aea0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f072402b980 comp rx=0 tx=0).stop 2023-10-14T16:40:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 -- 172.21.15.29:0/2211573773 shutdown_connections 2023-10-14T16:40:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c15d650 0x7f073c06aea0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 --2- 172.21.15.29:0/2211573773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c159ad0 0x7f073c070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 -- 172.21.15.29:0/2211573773 >> 172.21.15.29:0/2211573773 conn(0x7f073c0b0170 msgr2=0x7f073c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 -- 172.21.15.29:0/2211573773 shutdown_connections 2023-10-14T16:40:37.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.909+0000 7f0740d67640 1 -- 172.21.15.29:0/2211573773 wait complete. 2023-10-14T16:40:37.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.911+0000 7f0740d67640 1 Processor -- start 2023-10-14T16:40:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.911+0000 7f0740d67640 1 -- start start 2023-10-14T16:40:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c06b3e0 0x7f073c14e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c15d650 0x7f073c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0b11a0 con 0x7f073c159ad0 2023-10-14T16:40:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0b1a40 con 0x7f073c06b3e0 2023-10-14T16:40:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0740d67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0b2bc0 con 0x7f073c15d650 2023-10-14T16:40:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0733fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0733fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36874/0 (socket says 172.21.15.29:36874) 2023-10-14T16:40:37.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.912+0000 7f0733fff640 1 -- 172.21.15.29:0/83610799 learned_addr learned my addr 172.21.15.29:0/83610799 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f073b7fe640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c06b3e0 0x7f073c14e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f073bfff640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c15d650 0x7f073c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 -- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c15d650 msgr2=0x7f073c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c15d650 0x7f073c0acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 -- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c06b3e0 msgr2=0x7f073c14e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c06b3e0 0x7f073c14e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 -- 172.21.15.29:0/83610799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07240094d0 con 0x7f073c159ad0 2023-10-14T16:40:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0733fff640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f0724004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724039c90 con 0x7f073c159ad0 2023-10-14T16:40:37.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0724039e10 con 0x7f073c159ad0 2023-10-14T16:40:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.913+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0724043640 con 0x7f073c159ad0 2023-10-14T16:40:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.914+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c0b13f0 con 0x7f073c159ad0 2023-10-14T16:40:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.914+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c14e8c0 con 0x7f073c159ad0 2023-10-14T16:40:37.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.915+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c0b4240 con 0x7f073c159ad0 2023-10-14T16:40:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.916+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f07240437a0 con 0x7f073c159ad0 2023-10-14T16:40:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.916+0000 7f0738ff9640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 0x7f071c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:37.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.917+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f072403e070 con 0x7f073c159ad0 2023-10-14T16:40:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.917+0000 7f073b7fe640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 0x7f071c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.918+0000 7f073b7fe640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 0x7f071c06d360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f073c153a60 tx=0x7f072c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:37.921+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07240776e0 con 0x7f073c159ad0 2023-10-14T16:40:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.115+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f073c0b1bb0 con 0x7f073c159ad0 2023-10-14T16:40:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.116+0000 7f0738ff9640 1 -- 172.21.15.29:0/83610799 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f072407ad00 con 0x7f073c159ad0 2023-10-14T16:40:38.117 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:40:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 msgr2=0x7f071c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 0x7f071c06d360 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f073c153a60 tx=0x7f072c008040 comp rx=0 tx=0).stop 2023-10-14T16:40:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 msgr2=0x7f073c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f0724009870 tx=0x7f0724004730 comp rx=0 tx=0).stop 2023-10-14T16:40:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 shutdown_connections 2023-10-14T16:40:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f071c06aea0 0x7f071c06d360 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f073c15d650 0x7f073c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.119+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f073c159ad0 0x7f073c0ac890 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.120+0000 7f0740d67640 1 --2- 172.21.15.29:0/83610799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073c06b3e0 0x7f073c14e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.120+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 >> 172.21.15.29:0/83610799 conn(0x7f073c0b0170 msgr2=0x7f073c0b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:38.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.120+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 shutdown_connections 2023-10-14T16:40:38.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.120+0000 7f0740d67640 1 -- 172.21.15.29:0/83610799 wait complete. 2023-10-14T16:40:38.141 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:40:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.372+0000 7fd2c26bb640 1 Processor -- start 2023-10-14T16:40:38.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.372+0000 7fd2c26bb640 1 -- start start 2023-10-14T16:40:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2c26bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2bc15d330 0x7fd2bc1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2c26bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2c26bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2bc0d0890 con 0x7fd2bc06cc80 2023-10-14T16:40:38.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2c26bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc06b1e0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2c26bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2bc0d05b0 con 0x7fd2bc15d330 2023-10-14T16:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36880/0 (socket says 172.21.15.29:36880) 2023-10-14T16:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 -- 172.21.15.29:0/913111935 learned_addr learned my addr 172.21.15.29:0/913111935 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:38.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bbfff640 1 --2- 172.21.15.29:0/913111935 >> v2:172.21.15.186:3300/0 conn(0x7fd2bc15d330 0x7fd2bc1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 -- 172.21.15.29:0/913111935 >> v2:172.21.15.186:3300/0 conn(0x7fd2bc15d330 msgr2=0x7fd2bc1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/913111935 >> v2:172.21.15.186:3300/0 conn(0x7fd2bc15d330 0x7fd2bc1618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.374+0000 7fd2bb7fe640 1 -- 172.21.15.29:0/913111935 >> v1:172.21.15.29:6789/0 conn(0x7fd2bc06cc80 legacy=0x7fd2bc0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.375+0000 7fd2bb7fe640 1 -- 172.21.15.29:0/913111935 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2bc160dd0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.376+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/913111935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0744f0 tx=0x7fd2a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a748d46c9fe9580 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.376+0000 7fd2baffd640 1 -- 172.21.15.29:0/913111935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4031870 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.376+0000 7fd2baffd640 1 -- 172.21.15.29:0/913111935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a40319d0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.376+0000 7fd2baffd640 1 -- 172.21.15.29:0/913111935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a4031d60 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 -- 172.21.15.29:0/913111935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 msgr2=0x7fd2bc065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/913111935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd2bc0744f0 tx=0x7fd2a402b910 comp rx=0 tx=0).stop 2023-10-14T16:40:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 -- 172.21.15.29:0/913111935 shutdown_connections 2023-10-14T16:40:38.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/913111935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc065ab0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/913111935 >> v2:172.21.15.186:3300/0 conn(0x7fd2bc15d330 0x7fd2bc1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 -- 172.21.15.29:0/913111935 >> 172.21.15.29:0/913111935 conn(0x7fd2bc0d0ab0 msgr2=0x7fd2bc0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 -- 172.21.15.29:0/913111935 shutdown_connections 2023-10-14T16:40:38.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.377+0000 7fd2c26bb640 1 -- 172.21.15.29:0/913111935 wait complete. 2023-10-14T16:40:38.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.379+0000 7fd2c26bb640 1 Processor -- start 2023-10-14T16:40:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.379+0000 7fd2c26bb640 1 -- start start 2023-10-14T16:40:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 0x7fd2bc1600b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2bc15d330 0x7fd2bc0ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc155130 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0c4550 con 0x7fd2bc06cc80 2023-10-14T16:40:38.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2c26bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2bc0c4ad0 con 0x7fd2bc15d330 2023-10-14T16:40:38.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36882/0 (socket says 172.21.15.29:36882) 2023-10-14T16:40:38.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 -- 172.21.15.29:0/2721572021 learned_addr learned my addr 172.21.15.29:0/2721572021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 -- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2bc15d330 msgr2=0x7fd2bc0ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2bc15d330 0x7fd2bc0ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 -- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 msgr2=0x7fd2bc1600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 0x7fd2bc1600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 0x7fd2bc1600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bbfff640 1 -- 172.21.15.29:0/2721572021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a40094d0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.380+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 0x7fd2bc1600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2bbfff640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac007ef0 tx=0x7fd2ac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac014620 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2ac014780 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2ac014b30 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2bc154ba0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.381+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2bc15ebe0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.382+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2bc0cd3a0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.383+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd2ac014c90 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.384+0000 7fd2b8ff9640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 0x7fd29c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.384+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd2ac08a0a0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.384+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 0x7fd29c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.385+0000 7fd2bb7fe640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 0x7fd29c06d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4036000 tx=0x7fd2a403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.389+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2ac0550b0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.584+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd2bc1618b0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.585+0000 7fd2b8ff9640 1 -- 172.21.15.29:0/2721572021 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd2ac0586d0 con 0x7fd2bc06c7d0 2023-10-14T16:40:38.586 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:40:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 msgr2=0x7fd29c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 0x7fd29c06d330 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd2a4036000 tx=0x7fd2a403c040 comp rx=0 tx=0).stop 2023-10-14T16:40:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 msgr2=0x7fd2bc154660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd2ac007ef0 tx=0x7fd2ac00e4b0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 shutdown_connections 2023-10-14T16:40:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd29c06ae70 0x7fd29c06d330 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2bc15d330 0x7fd2bc0ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2bc06cc80 0x7fd2bc1600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 --2- 172.21.15.29:0/2721572021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2bc06c7d0 0x7fd2bc154660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 >> 172.21.15.29:0/2721572021 conn(0x7fd2bc0d0ab0 msgr2=0x7fd2bc0cf4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.588+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 shutdown_connections 2023-10-14T16:40:38.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.589+0000 7fd2c26bb640 1 -- 172.21.15.29:0/2721572021 wait complete. 2023-10-14T16:40:38.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:40:38.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.838+0000 7f328ab5a640 1 Processor -- start 2023-10-14T16:40:38.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.839+0000 7f328ab5a640 1 -- start start 2023-10-14T16:40:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f328ab5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3284069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f328ab5a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f32840caac0 0x7f328406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f328ab5a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32840cb030 con 0x7f32840c9ca0 2023-10-14T16:40:38.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f328ab5a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32840c91c0 con 0x7f32840caac0 2023-10-14T16:40:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f328ab5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c94a0 con 0x7f328415ad70 2023-10-14T16:40:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f3283fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32840c9ca0 0x7f32840ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47934/0 (socket says 172.21.15.29:47934) 2023-10-14T16:40:38.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.840+0000 7f3283fff640 1 -- 172.21.15.29:0/1947264588 learned_addr learned my addr 172.21.15.29:0/1947264588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:38.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32890f7640 1 --2- 172.21.15.29:0/1947264588 >> v2:172.21.15.186:3300/0 conn(0x7f32840caac0 0x7f328406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32888f6640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3284069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 203544258 0 0) 0x7f32840cb030 con 0x7f32840c9ca0 2023-10-14T16:40:38.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3274003520 con 0x7f32840c9ca0 2023-10-14T16:40:38.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457685472 0 0) 0x7f3274003520 con 0x7f32840c9ca0 2023-10-14T16:40:38.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 msgr2=0x7f3284069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:38.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3284069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 >> v2:172.21.15.186:3300/0 conn(0x7f32840caac0 msgr2=0x7f328406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:38.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 --2- 172.21.15.29:0/1947264588 >> v2:172.21.15.186:3300/0 conn(0x7f32840caac0 0x7f328406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840c9780 con 0x7f32840c9ca0 2023-10-14T16:40:38.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3270001670 con 0x7f32840c9ca0 2023-10-14T16:40:38.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 >> v1:172.21.15.29:6789/0 conn(0x7f32840c9ca0 legacy=0x7f32840ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.841+0000 7f32837fe640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274003970 0x7f3274003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32837fe640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f3274004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32837fe640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3274005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c91c0 con 0x7f32740047c0 2023-10-14T16:40:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c94a0 con 0x7f3274003970 2023-10-14T16:40:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f327400b980 con 0x7f328415ad70 2023-10-14T16:40:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f3274004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32888f6640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274003970 0x7f3274003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 -- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 msgr2=0x7f3274005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:38.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32890f7640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3274005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f328415ad70 0x7f3274005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 -- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274003970 msgr2=0x7f3274003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:38.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274003970 0x7f3274003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f3283fff640 1 -- 172.21.15.29:0/1947264588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3274003520 con 0x7f32740047c0 2023-10-14T16:40:38.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.842+0000 7f32888f6640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3274003970 0x7f3274003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:38.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.843+0000 7f3283fff640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f3274004c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f327000c290 tx=0x7f32740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50cc52e0e97a7184 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3270005ad0 con 0x7f32740047c0 2023-10-14T16:40:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3270005c30 con 0x7f32740047c0 2023-10-14T16:40:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f32837fe640 1 -- 172.21.15.29:0/1947264588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327002c8d0 con 0x7f32740047c0 2023-10-14T16:40:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f328ab5a640 1 -- 172.21.15.29:0/1947264588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 msgr2=0x7f3274004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f328ab5a640 1 --2- 172.21.15.29:0/1947264588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f3274004c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f327000c290 tx=0x7f32740057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f32890f7640 1 -- 172.21.15.29:0/1947264588 reap_dead start 2023-10-14T16:40:38.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f328ab5a640 1 -- 172.21.15.29:0/1947264588 shutdown_connections 2023-10-14T16:40:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f328ab5a640 1 -- 172.21.15.29:0/1947264588 >> 172.21.15.29:0/1947264588 conn(0x7f32840ce9b0 msgr2=0x7f32840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.844+0000 7f328ab5a640 1 -- 172.21.15.29:0/1947264588 shutdown_connections 2023-10-14T16:40:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.845+0000 7f328ab5a640 1 -- 172.21.15.29:0/1947264588 wait complete. 2023-10-14T16:40:38.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.846+0000 7f328ab5a640 1 Processor -- start 2023-10-14T16:40:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.847+0000 7f328ab5a640 1 -- start start 2023-10-14T16:40:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.847+0000 7f328ab5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f328ab5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32840c5ce0 0x7f32840c8890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f328ab5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32840c7340 0x7f32840c77c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f328ab5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c5160 con 0x7f32740047c0 2023-10-14T16:40:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f328ab5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328414bc20 con 0x7f32840c7340 2023-10-14T16:40:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f328ab5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32840c62a0 con 0x7f32840c5ce0 2023-10-14T16:40:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36892/0 (socket says 172.21.15.29:36892) 2023-10-14T16:40:38.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 -- 172.21.15.29:0/3943304234 learned_addr learned my addr 172.21.15.29:0/3943304234 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 -- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32840c5ce0 msgr2=0x7f32840c8890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32840c5ce0 0x7f32840c8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 -- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32840c7340 msgr2=0x7f32840c77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32840c7340 0x7f32840c77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:38.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.848+0000 7f32888f6640 1 -- 172.21.15.29:0/3943304234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32840c6580 con 0x7f32740047c0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f32888f6640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32780108b0 tx=0x7f3278010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327800eae0 con 0x7f32740047c0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f327800ec40 con 0x7f32740047c0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f327801c400 con 0x7f32740047c0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32840c6860 con 0x7f32740047c0 2023-10-14T16:40:38.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.849+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32840c6e20 con 0x7f32740047c0 2023-10-14T16:40:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.850+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328404e160 con 0x7f32740047c0 2023-10-14T16:40:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.851+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f327801c560 con 0x7f32740047c0 2023-10-14T16:40:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.852+0000 7f32817fa640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 0x7f326806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.852+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3278089a90 con 0x7f32740047c0 2023-10-14T16:40:38.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.853+0000 7f3283fff640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 0x7f326806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.853+0000 7f3283fff640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 0x7f326806d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3270034000 tx=0x7f327002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:38.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:38.857+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3278054c80 con 0x7f32740047c0 2023-10-14T16:40:39.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.056+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f328415de80 con 0x7f32740047c0 2023-10-14T16:40:39.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.057+0000 7f32817fa640 1 -- 172.21.15.29:0/3943304234 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f32780582a0 con 0x7f32740047c0 2023-10-14T16:40:39.058 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:40:39.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 msgr2=0x7f326806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 0x7f326806d330 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3270034000 tx=0x7f327002e000 comp rx=0 tx=0).stop 2023-10-14T16:40:39.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 msgr2=0x7f32840c4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f32780108b0 tx=0x7f3278010d70 comp rx=0 tx=0).stop 2023-10-14T16:40:39.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 shutdown_connections 2023-10-14T16:40:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f326806ae70 0x7f326806d330 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32840c7340 0x7f32840c77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32840c5ce0 0x7f32840c8890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 --2- 172.21.15.29:0/3943304234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32740047c0 0x7f32840c4c20 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 >> 172.21.15.29:0/3943304234 conn(0x7f32840ce9b0 msgr2=0x7f32840b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:39.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 shutdown_connections 2023-10-14T16:40:39.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.060+0000 7f328ab5a640 1 -- 172.21.15.29:0/3943304234 wait complete. 2023-10-14T16:40:39.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:40:39.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.303+0000 7efc51ed1640 1 Processor -- start 2023-10-14T16:40:39.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.304+0000 7efc51ed1640 1 -- start start 2023-10-14T16:40:39.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc51ed1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efc4c06c040 0x7efc4c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc51ed1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc4c0cda80 0x7efc4c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc51ed1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc4c06c960 con 0x7efc4c0ad970 2023-10-14T16:40:39.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc51ed1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c15b9d0 con 0x7efc4c0cda80 2023-10-14T16:40:39.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc51ed1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc4c15bcb0 con 0x7efc4c06c040 2023-10-14T16:40:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.305+0000 7efc4bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc4c0ad970 0x7efc4c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47942/0 (socket says 172.21.15.29:47942) 2023-10-14T16:40:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc4bfff640 1 -- 172.21.15.29:0/3633185684 learned_addr learned my addr 172.21.15.29:0/3633185684 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc516d0640 1 --2- 172.21.15.29:0/3633185684 >> v2:172.21.15.29:3300/0 conn(0x7efc4c0cda80 0x7efc4c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc50ecf640 1 --2- 172.21.15.29:0/3633185684 >> v2:172.21.15.186:3300/0 conn(0x7efc4c06c040 0x7efc4c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc4b7fe640 1 -- 172.21.15.29:0/3633185684 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022243064 0 0) 0x7efc4c06c960 con 0x7efc4c0ad970 2023-10-14T16:40:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc4b7fe640 1 -- 172.21.15.29:0/3633185684 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc3c003520 con 0x7efc4c0ad970 2023-10-14T16:40:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc516d0640 1 -- 172.21.15.29:0/3633185684 >> v2:172.21.15.186:3300/0 conn(0x7efc4c06c040 msgr2=0x7efc4c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc516d0640 1 --2- 172.21.15.29:0/3633185684 >> v2:172.21.15.186:3300/0 conn(0x7efc4c06c040 0x7efc4c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc516d0640 1 -- 172.21.15.29:0/3633185684 >> v1:172.21.15.29:6789/0 conn(0x7efc4c0ad970 legacy=0x7efc4c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.306+0000 7efc516d0640 1 -- 172.21.15.29:0/3633185684 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc4c15bf90 con 0x7efc4c0cda80 2023-10-14T16:40:39.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.307+0000 7efc516d0640 1 --2- 172.21.15.29:0/3633185684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c15ed90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0cf350 tx=0x7efc4402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb02f63492e0bba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc4b7fe640 1 -- 172.21.15.29:0/3633185684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4402cc80 con 0x7efc4c0cda80 2023-10-14T16:40:39.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc4b7fe640 1 -- 172.21.15.29:0/3633185684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4402cde0 con 0x7efc4c0cda80 2023-10-14T16:40:39.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc4b7fe640 1 -- 172.21.15.29:0/3633185684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc44034660 con 0x7efc4c0cda80 2023-10-14T16:40:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 -- 172.21.15.29:0/3633185684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 msgr2=0x7efc4c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 --2- 172.21.15.29:0/3633185684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c15ed90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7efc4c0cf350 tx=0x7efc4402bed0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 -- 172.21.15.29:0/3633185684 shutdown_connections 2023-10-14T16:40:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 --2- 172.21.15.29:0/3633185684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c15ed90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 --2- 172.21.15.29:0/3633185684 >> v2:172.21.15.186:3300/0 conn(0x7efc4c06c040 0x7efc4c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 -- 172.21.15.29:0/3633185684 >> 172.21.15.29:0/3633185684 conn(0x7efc4c0ce6a0 msgr2=0x7efc4c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 -- 172.21.15.29:0/3633185684 shutdown_connections 2023-10-14T16:40:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.308+0000 7efc51ed1640 1 -- 172.21.15.29:0/3633185684 wait complete. 2023-10-14T16:40:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.310+0000 7efc51ed1640 1 Processor -- start 2023-10-14T16:40:39.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.310+0000 7efc51ed1640 1 -- start start 2023-10-14T16:40:39.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc4c06c040 0x7efc4c0b0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 0x7efc4c1f0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c0c5bf0 con 0x7efc4c0cda80 2023-10-14T16:40:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1f0bf0 con 0x7efc4c0ad970 2023-10-14T16:40:39.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc51ed1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc4c1f0ed0 con 0x7efc4c06c040 2023-10-14T16:40:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc516d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc516d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36902/0 (socket says 172.21.15.29:36902) 2023-10-14T16:40:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.311+0000 7efc516d0640 1 -- 172.21.15.29:0/742699976 learned_addr learned my addr 172.21.15.29:0/742699976 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc50ecf640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc4c06c040 0x7efc4c0b0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc4bfff640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 0x7efc4c1f0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 -- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc4c06c040 msgr2=0x7efc4c0b0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc4c06c040 0x7efc4c0b0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 -- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 msgr2=0x7efc4c1f0300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 0x7efc4c1f0300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 -- 172.21.15.29:0/742699976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc44009a70 con 0x7efc4c0cda80 2023-10-14T16:40:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc4bfff640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 0x7efc4c1f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.312+0000 7efc516d0640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc44009e80 tx=0x7efc44004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.313+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4403e610 con 0x7efc4c0cda80 2023-10-14T16:40:39.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.313+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4403e770 con 0x7efc4c0cda80 2023-10-14T16:40:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.313+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4403eb60 con 0x7efc4c0cda80 2023-10-14T16:40:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.313+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc4c1f11b0 con 0x7efc4c0cda80 2023-10-14T16:40:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.313+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc4c1f0840 con 0x7efc4c0cda80 2023-10-14T16:40:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.314+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc4c06b310 con 0x7efc4c0cda80 2023-10-14T16:40:39.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.315+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7efc4403b050 con 0x7efc4c0cda80 2023-10-14T16:40:39.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.316+0000 7efc497fa640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 0x7efc3006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.316+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7efc440abea0 con 0x7efc4c0cda80 2023-10-14T16:40:39.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.316+0000 7efc50ecf640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 0x7efc3006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.317+0000 7efc50ecf640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 0x7efc3006d330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efc40006fd0 tx=0x7efc40008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.321+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc44076f40 con 0x7efc4c0cda80 2023-10-14T16:40:39.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.551+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc4c04e9b0 con 0x7efc4c0cda80 2023-10-14T16:40:39.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.552+0000 7efc497fa640 1 -- 172.21.15.29:0/742699976 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7efc4404e090 con 0x7efc4c0cda80 2023-10-14T16:40:39.553 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:39.554 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:40:39.554 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:40:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 msgr2=0x7efc3006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 0x7efc3006d330 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7efc40006fd0 tx=0x7efc40008040 comp rx=0 tx=0).stop 2023-10-14T16:40:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 msgr2=0x7efc4c0c7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efc44009e80 tx=0x7efc44004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 shutdown_connections 2023-10-14T16:40:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7efc3006ae70 0x7efc3006d330 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc4c0cda80 0x7efc4c0c7af0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc4c0ad970 0x7efc4c1f0300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 --2- 172.21.15.29:0/742699976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc4c06c040 0x7efc4c0b0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 >> 172.21.15.29:0/742699976 conn(0x7efc4c0ce6a0 msgr2=0x7efc4c15e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 shutdown_connections 2023-10-14T16:40:39.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.555+0000 7efc51ed1640 1 -- 172.21.15.29:0/742699976 wait complete. 2023-10-14T16:40:39.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:39.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.804+0000 7fd4125c7640 1 Processor -- start 2023-10-14T16:40:39.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.804+0000 7fd4125c7640 1 -- start start 2023-10-14T16:40:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd4125c7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd40c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd4125c7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c06f7f0 con 0x7fd40c15d880 2023-10-14T16:40:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd4125c7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd40c06c5e0 con 0x7fd40c159b20 2023-10-14T16:40:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd4125c7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c06c8c0 con 0x7fd40c159f00 2023-10-14T16:40:39.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd410b64640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd40c15d880 0x7fd40c06fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47950/0 (socket says 172.21.15.29:47950) 2023-10-14T16:40:39.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd410b64640 1 -- 172.21.15.29:0/3380751890 learned_addr learned my addr 172.21.15.29:0/3380751890 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:39.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.806+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd40c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544788472 0 0) 0x7fd40c06f7f0 con 0x7fd40c15d880 2023-10-14T16:40:39.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f4003520 con 0x7fd40c15d880 2023-10-14T16:40:39.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4081009152 0 0) 0x7fd3f4003520 con 0x7fd40c15d880 2023-10-14T16:40:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 msgr2=0x7fd40c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd40c0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 >> v1:172.21.15.186:6789/0 conn(0x7fd40c159b20 legacy=0x7fd40c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c06cba0 con 0x7fd40c15d880 2023-10-14T16:40:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd4040019f0 con 0x7fd40c15d880 2023-10-14T16:40:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 >> v1:172.21.15.29:6789/0 conn(0x7fd40c15d880 legacy=0x7fd40c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd3f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd3f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3f4004d00 0x7fd3f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c06c8c0 con 0x7fd3f4003f20 2023-10-14T16:40:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f400ba60 con 0x7fd3f4004d00 2023-10-14T16:40:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3f400bc20 con 0x7fd40c159f00 2023-10-14T16:40:39.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.807+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd3f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd410b64640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3f4004d00 0x7fd3f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40bfff640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd3f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40b7fe640 1 -- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 msgr2=0x7fd3f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c159f00 0x7fd3f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40b7fe640 1 -- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3f4004d00 msgr2=0x7fd3f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3f4004d00 0x7fd3f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.808+0000 7fd40b7fe640 1 -- 172.21.15.29:0/3380751890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f4003520 con 0x7fd3f4003f20 2023-10-14T16:40:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.809+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd3f40047c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd400009650 tx=0x7fd3f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc2e551d9af5c2e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.809+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40002e890 con 0x7fd3f4003f20 2023-10-14T16:40:39.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.809+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd40002e9f0 con 0x7fd3f4003f20 2023-10-14T16:40:39.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.809+0000 7fd40affd640 1 -- 172.21.15.29:0/3380751890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd40002ed80 con 0x7fd3f4003f20 2023-10-14T16:40:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 -- 172.21.15.29:0/3380751890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 msgr2=0x7fd3f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 --2- 172.21.15.29:0/3380751890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd3f40047c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fd400009650 tx=0x7fd3f40057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd410b64640 1 -- 172.21.15.29:0/3380751890 reap_dead start 2023-10-14T16:40:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 -- 172.21.15.29:0/3380751890 shutdown_connections 2023-10-14T16:40:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 -- 172.21.15.29:0/3380751890 >> 172.21.15.29:0/3380751890 conn(0x7fd40c0cb410 msgr2=0x7fd40c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 -- 172.21.15.29:0/3380751890 shutdown_connections 2023-10-14T16:40:39.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.810+0000 7fd4125c7640 1 -- 172.21.15.29:0/3380751890 wait complete. 2023-10-14T16:40:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.812+0000 7fd4125c7640 1 Processor -- start 2023-10-14T16:40:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.812+0000 7fd4125c7640 1 -- start start 2023-10-14T16:40:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd40c1f2b20 0x7fd40c0cf1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 0x7fd40c15c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c0cf6f0 con 0x7fd3f4003f20 2023-10-14T16:40:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c1f3020 con 0x7fd40c1f2b20 2023-10-14T16:40:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd4125c7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd40c1f3300 con 0x7fd40c152080 2023-10-14T16:40:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.813+0000 7fd40bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36924/0 (socket says 172.21.15.29:36924) 2023-10-14T16:40:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 -- 172.21.15.29:0/1751660060 learned_addr learned my addr 172.21.15.29:0/1751660060 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd410b64640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 0x7fd40c15c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd40c1f2b20 0x7fd40c0cf1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 -- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 msgr2=0x7fd40c15c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 0x7fd40c15c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 -- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd40c1f2b20 msgr2=0x7fd40c0cf1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd40c1f2b20 0x7fd40c0cf1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 -- 172.21.15.29:0/1751660060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd40c1f35e0 con 0x7fd3f4003f20 2023-10-14T16:40:39.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd410b64640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 0x7fd40c15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd40bfff640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00e550 tx=0x7fd3fc00ea10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc0052e0 con 0x7fd3f4003f20 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3fc005440 con 0x7fd3f4003f20 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.814+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3fc003810 con 0x7fd3f4003f20 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.815+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd40c1f38c0 con 0x7fd3f4003f20 2023-10-14T16:40:39.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.815+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd40c15ca20 con 0x7fd3f4003f20 2023-10-14T16:40:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.816+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd40c0ae110 con 0x7fd3f4003f20 2023-10-14T16:40:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.817+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd3fc003970 con 0x7fd3f4003f20 2023-10-14T16:40:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.818+0000 7fd408ff9640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 0x7fd3f806d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.818+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fd3fc089280 con 0x7fd3f4003f20 2023-10-14T16:40:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.818+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 0x7fd3f806d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.819+0000 7fd40b7fe640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 0x7fd3f806d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd400038000 tx=0x7fd400032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:39.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:39.823+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3fc0543a0 con 0x7fd3f4003f20 2023-10-14T16:40:40.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.023+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd40c04e160 con 0x7fd3f4003f20 2023-10-14T16:40:40.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.025+0000 7fd408ff9640 1 -- 172.21.15.29:0/1751660060 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fd3fc0579c0 con 0x7fd3f4003f20 2023-10-14T16:40:40.027 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:40.027 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 msgr2=0x7fd3f806d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 0x7fd3f806d3b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd400038000 tx=0x7fd400032000 comp rx=0 tx=0).stop 2023-10-14T16:40:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 msgr2=0x7fd40c1f3ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fd3fc00e550 tx=0x7fd3fc00ea10 comp rx=0 tx=0).stop 2023-10-14T16:40:40.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 shutdown_connections 2023-10-14T16:40:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd3f806aef0 0x7fd3f806d3b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd40c152080 0x7fd40c15c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd40c1f2b20 0x7fd40c0cf1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 --2- 172.21.15.29:0/1751660060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3f4003f20 0x7fd40c1f3ce0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 >> 172.21.15.29:0/1751660060 conn(0x7fd40c0cb410 msgr2=0x7fd40c0b0960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 shutdown_connections 2023-10-14T16:40:40.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.028+0000 7fd4125c7640 1 -- 172.21.15.29:0/1751660060 wait complete. 2023-10-14T16:40:40.049 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:40:40.050 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:40:40.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.280+0000 7f5097b05640 1 Processor -- start 2023-10-14T16:40:40.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.280+0000 7f5097b05640 1 -- start start 2023-10-14T16:40:40.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.281+0000 7f5097b05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.281+0000 7f5097b05640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f509006c7d0 0x7f5090065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f5097b05640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50900d0890 con 0x7f509006cc80 2023-10-14T16:40:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f5097b05640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f509006b1e0 con 0x7f509006c7d0 2023-10-14T16:40:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f5097b05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50900d05b0 con 0x7f509015d330 2023-10-14T16:40:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f50958a1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f50958a1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:48246/0 (socket says 172.21.15.29:48246) 2023-10-14T16:40:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f50958a1640 1 -- 172.21.15.29:0/3128466189 learned_addr learned my addr 172.21.15.29:0/3128466189 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.282+0000 7f50950a0640 1 --2- 172.21.15.29:0/3128466189 >> v2:172.21.15.186:3300/0 conn(0x7f509006c7d0 0x7f5090065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.283+0000 7f50958a1640 1 -- 172.21.15.29:0/3128466189 >> v2:172.21.15.186:3300/0 conn(0x7f509006c7d0 msgr2=0x7f5090065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.283+0000 7f50958a1640 1 --2- 172.21.15.29:0/3128466189 >> v2:172.21.15.186:3300/0 conn(0x7f509006c7d0 0x7f5090065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.283+0000 7f50958a1640 1 -- 172.21.15.29:0/3128466189 >> v1:172.21.15.186:6789/0 conn(0x7f509006cc80 legacy=0x7f50900ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.283+0000 7f50958a1640 1 -- 172.21.15.29:0/3128466189 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5090160dd0 con 0x7f509015d330 2023-10-14T16:40:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.284+0000 7f50958a1640 1 --2- 172.21.15.29:0/3128466189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f508c009870 tx=0x7f508c02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1e44e14dff666cc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.285+0000 7f509489f640 1 -- 172.21.15.29:0/3128466189 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c031640 con 0x7f509015d330 2023-10-14T16:40:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.285+0000 7f509489f640 1 -- 172.21.15.29:0/3128466189 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508c0317a0 con 0x7f509015d330 2023-10-14T16:40:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.285+0000 7f509489f640 1 -- 172.21.15.29:0/3128466189 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508c031b30 con 0x7f509015d330 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.285+0000 7f5097b05640 1 -- 172.21.15.29:0/3128466189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 msgr2=0x7f50901618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.285+0000 7f5097b05640 1 --2- 172.21.15.29:0/3128466189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f508c009870 tx=0x7f508c02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 -- 172.21.15.29:0/3128466189 shutdown_connections 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 --2- 172.21.15.29:0/3128466189 >> v2:172.21.15.186:3300/0 conn(0x7f509006c7d0 0x7f5090065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 --2- 172.21.15.29:0/3128466189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901618b0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 -- 172.21.15.29:0/3128466189 >> 172.21.15.29:0/3128466189 conn(0x7f50900d0ab0 msgr2=0x7f50900d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:40.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 -- 172.21.15.29:0/3128466189 shutdown_connections 2023-10-14T16:40:40.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.286+0000 7f5097b05640 1 -- 172.21.15.29:0/3128466189 wait complete. 2023-10-14T16:40:40.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.288+0000 7f5097b05640 1 Processor -- start 2023-10-14T16:40:40.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.288+0000 7f5097b05640 1 -- start start 2023-10-14T16:40:40.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f509006c7d0 0x7f5090158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901552f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50900aff40 con 0x7f509006cc80 2023-10-14T16:40:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5090155830 con 0x7f509006c7d0 2023-10-14T16:40:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f5097b05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50900ca380 con 0x7f509015d330 2023-10-14T16:40:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f50950a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f50950a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36938/0 (socket says 172.21.15.29:36938) 2023-10-14T16:40:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f50950a0640 1 -- 172.21.15.29:0/3539802212 learned_addr learned my addr 172.21.15.29:0/3539802212 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f50960a2640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901552f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.289+0000 7f50958a1640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f509006c7d0 0x7f5090158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 -- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 msgr2=0x7f50901552f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901552f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 -- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f509006c7d0 msgr2=0x7f5090158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f509006c7d0 0x7f5090158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 -- 172.21.15.29:0/3539802212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f508c0094d0 con 0x7f509006cc80 2023-10-14T16:40:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50950a0640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f508000e870 tx=0x7f508000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50800094f0 con 0x7f509006cc80 2023-10-14T16:40:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f508000c940 con 0x7f509006cc80 2023-10-14T16:40:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508000ccf0 con 0x7f509006cc80 2023-10-14T16:40:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.290+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50900c9dc0 con 0x7f509006cc80 2023-10-14T16:40:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.291+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50900afb60 con 0x7f509006cc80 2023-10-14T16:40:40.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.292+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f509004e160 con 0x7f509006cc80 2023-10-14T16:40:40.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.293+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5080013470 con 0x7f509006cc80 2023-10-14T16:40:40.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.293+0000 7f50867fc640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 0x7f507406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.294+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f50800897c0 con 0x7f509006cc80 2023-10-14T16:40:40.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.294+0000 7f50958a1640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 0x7f507406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.294+0000 7f50958a1640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 0x7f507406d330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f508c036000 tx=0x7f508c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.298+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50800547e0 con 0x7f509006cc80 2023-10-14T16:40:40.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.493+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f509006f3a0 con 0x7f509006cc80 2023-10-14T16:40:40.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.494+0000 7f50867fc640 1 -- 172.21.15.29:0/3539802212 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5080057e00 con 0x7f509006cc80 2023-10-14T16:40:40.495 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:40:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 msgr2=0x7f507406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 0x7f507406d330 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f508c036000 tx=0x7f508c03c040 comp rx=0 tx=0).stop 2023-10-14T16:40:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 msgr2=0x7f5090154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f508000e870 tx=0x7f508000ed30 comp rx=0 tx=0).stop 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 shutdown_connections 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f507406ae70 0x7f507406d330 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f509015d330 0x7f50901552f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f509006cc80 0x7f5090154db0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 --2- 172.21.15.29:0/3539802212 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f509006c7d0 0x7f5090158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 >> 172.21.15.29:0/3539802212 conn(0x7f50900d0ab0 msgr2=0x7f509015e190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 shutdown_connections 2023-10-14T16:40:40.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.497+0000 7f5097b05640 1 -- 172.21.15.29:0/3539802212 wait complete. 2023-10-14T16:40:40.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:40:40.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.745+0000 7f5ec6f7a640 1 Processor -- start 2023-10-14T16:40:40.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.745+0000 7f5ec6f7a640 1 -- start start 2023-10-14T16:40:40.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5ec015c9f0 0x7f5ec00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec00c9230 0x7f5ec00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec00cdec0 con 0x7f5ec00c9230 2023-10-14T16:40:40.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ec0160a90 con 0x7f5ec015c9f0 2023-10-14T16:40:40.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.746+0000 7f5ec6f7a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec00cd900 con 0x7f5ec0070f20 2023-10-14T16:40:40.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec00c9230 0x7f5ec00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ec00c9230 0x7f5ec00709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36954/0 (socket says 172.21.15.29:36954) 2023-10-14T16:40:40.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 -- 172.21.15.29:0/3685120781 learned_addr learned my addr 172.21.15.29:0/3685120781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:40.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5f78640 1 --2- 172.21.15.29:0/3685120781 >> v2:172.21.15.186:3300/0 conn(0x7f5ec015c9f0 0x7f5ec00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec6779640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 -- 172.21.15.29:0/3685120781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 msgr2=0x7f5ec00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:40.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 -- 172.21.15.29:0/3685120781 >> v2:172.21.15.186:3300/0 conn(0x7f5ec015c9f0 msgr2=0x7f5ec00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:40.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 --2- 172.21.15.29:0/3685120781 >> v2:172.21.15.186:3300/0 conn(0x7f5ec015c9f0 0x7f5ec00c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.747+0000 7f5ec5777640 1 -- 172.21.15.29:0/3685120781 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec00cdbe0 con 0x7f5ec00c9230 2023-10-14T16:40:40.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.749+0000 7f5ec5777640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec00c9230 0x7f5ec00709e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00ad4e0 tx=0x7f5eb002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df38bb540e45d963 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.749+0000 7f5ec4f76640 1 -- 172.21.15.29:0/3685120781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0031870 con 0x7f5ec00c9230 2023-10-14T16:40:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.749+0000 7f5ec4f76640 1 -- 172.21.15.29:0/3685120781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb00319d0 con 0x7f5ec00c9230 2023-10-14T16:40:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.749+0000 7f5ec4f76640 1 -- 172.21.15.29:0/3685120781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0031d60 con 0x7f5ec00c9230 2023-10-14T16:40:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/3685120781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec00c9230 msgr2=0x7f5ec00709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec00c9230 0x7f5ec00709e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5ec00ad4e0 tx=0x7f5eb002b910 comp rx=0 tx=0).stop 2023-10-14T16:40:40.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/3685120781 shutdown_connections 2023-10-14T16:40:40.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/3685120781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec00c9230 0x7f5ec00709e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/3685120781 >> v2:172.21.15.186:3300/0 conn(0x7f5ec015c9f0 0x7f5ec00c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/3685120781 >> 172.21.15.29:0/3685120781 conn(0x7f5ec00ce6a0 msgr2=0x7f5ec00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:40.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/3685120781 shutdown_connections 2023-10-14T16:40:40.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.750+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/3685120781 wait complete. 2023-10-14T16:40:40.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.752+0000 7f5ec6f7a640 1 Processor -- start 2023-10-14T16:40:40.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.752+0000 7f5ec6f7a640 1 -- start start 2023-10-14T16:40:40.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 0x7f5ec01f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec006a0a0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01f3420 con 0x7f5ec00c9230 2023-10-14T16:40:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6f7a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec006be40 con 0x7f5ec0070f20 2023-10-14T16:40:40.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6779640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6779640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36960/0 (socket says 172.21.15.29:36960) 2023-10-14T16:40:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6779640 1 -- 172.21.15.29:0/22065192 learned_addr learned my addr 172.21.15.29:0/22065192 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec5777640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 0x7f5ec01f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec5f78640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.753+0000 7f5ec6779640 1 -- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 msgr2=0x7f5ec00c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6779640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6779640 1 -- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 msgr2=0x7f5ec01f2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6779640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 0x7f5ec01f2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6779640 1 -- 172.21.15.29:0/22065192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5eb00094d0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec5777640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 0x7f5ec01f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec5f78640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6779640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8007e40 tx=0x7f5eb800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb8014620 con 0x7f5ec015c9f0 2023-10-14T16:40:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5eb800ee90 con 0x7f5ec015c9f0 2023-10-14T16:40:40.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb80149e0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec006c120 con 0x7f5ec015c9f0 2023-10-14T16:40:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.754+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec006c6e0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.756+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec004e9b0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.756+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5eb8014b40 con 0x7f5ec015c9f0 2023-10-14T16:40:40.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.758+0000 7f5eb67fc640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 0x7f5ea406d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:40.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.758+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5eb8089f80 con 0x7f5ec015c9f0 2023-10-14T16:40:40.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.758+0000 7f5ec5f78640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 0x7f5ea406d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:40.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.759+0000 7f5ec5f78640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 0x7f5ea406d380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc006fd0 tx=0x7f5ebc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:40.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.762+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5eb80550a0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:40:40.822 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:40:40.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.963+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ec00adac0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.963+0000 7f5eb67fc640 1 -- 172.21.15.29:0/22065192 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5eb80586c0 con 0x7f5ec015c9f0 2023-10-14T16:40:40.965 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:40:40.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.966+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 msgr2=0x7f5ea406d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 0x7f5ea406d380 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f5ebc006fd0 tx=0x7f5ebc008040 comp rx=0 tx=0).stop 2023-10-14T16:40:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 msgr2=0x7f5ec01f5a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5eb8007e40 tx=0x7f5eb800e4b0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 shutdown_connections 2023-10-14T16:40:40.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5ea406aec0 0x7f5ea406d380 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ec015c9f0 0x7f5ec01f5a50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ec00c9230 0x7f5ec01f2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 --2- 172.21.15.29:0/22065192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ec0070f20 0x7f5ec00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:40.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 >> 172.21.15.29:0/22065192 conn(0x7f5ec00ce6a0 msgr2=0x7f5ec00c9920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:40.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 shutdown_connections 2023-10-14T16:40:40.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:40.967+0000 7f5ec6f7a640 1 -- 172.21.15.29:0/22065192 wait complete. 2023-10-14T16:40:40.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:40:41.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.261+0000 7f745731d640 1 Processor -- start 2023-10-14T16:40:41.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.261+0000 7f745731d640 1 -- start start 2023-10-14T16:40:41.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f745731d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f745015d150 0x7f74500ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f745731d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f7450160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f745731d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74500ac890 con 0x7f7450069da0 2023-10-14T16:40:41.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f745731d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f745015d530 con 0x7f745015d150 2023-10-14T16:40:41.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f745731d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7450161650 con 0x7f7450160b90 2023-10-14T16:40:41.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f74558ba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7450069da0 0x7f745006a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36966/0 (socket says 172.21.15.29:36966) 2023-10-14T16:40:41.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.263+0000 7f74558ba640 1 -- 172.21.15.29:0/3518425877 learned_addr learned my addr 172.21.15.29:0/3518425877 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:41.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f74548b8640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f7450160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f74550b9640 1 --2- 172.21.15.29:0/3518425877 >> v2:172.21.15.186:3300/0 conn(0x7f745015d150 0x7f74500ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156200602 0 0) 0x7f74500ac890 con 0x7f7450069da0 2023-10-14T16:40:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f743c003520 con 0x7f7450069da0 2023-10-14T16:40:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1857249071 0 0) 0x7f743c003520 con 0x7f7450069da0 2023-10-14T16:40:41.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 msgr2=0x7f7450160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f7450160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 >> v2:172.21.15.186:3300/0 conn(0x7f745015d150 msgr2=0x7f74500ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 --2- 172.21.15.29:0/3518425877 >> v2:172.21.15.186:3300/0 conn(0x7f745015d150 0x7f74500ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7450161930 con 0x7f7450069da0 2023-10-14T16:40:41.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f74480019f0 con 0x7f7450069da0 2023-10-14T16:40:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 >> v1:172.21.15.29:6789/0 conn(0x7f7450069da0 legacy=0x7f745006a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f743c003970 0x7f743c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f743c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f743c004d00 0x7f743c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f745015d530 con 0x7f743c004d00 2023-10-14T16:40:41.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7450161650 con 0x7f743c003970 2023-10-14T16:40:41.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.264+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f743c00b980 con 0x7f7450160b90 2023-10-14T16:40:41.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f743c004d00 0x7f743c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74550b9640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f743c003970 0x7f743c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74548b8640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f743c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 -- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 msgr2=0x7f743c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7450160b90 0x7f743c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 -- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f743c003970 msgr2=0x7f743c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f743c003970 0x7f743c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74558ba640 1 -- 172.21.15.29:0/3518425877 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743c003520 con 0x7f743c004d00 2023-10-14T16:40:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.265+0000 7f74550b9640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f743c003970 0x7f743c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.266+0000 7f74558ba640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f743c004d00 0x7f743c005180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f744800a860 tx=0x7f743c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d74f17e5b61e9ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448034630 con 0x7f743c004d00 2023-10-14T16:40:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f744802f070 con 0x7f743c004d00 2023-10-14T16:40:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f7447fff640 1 -- 172.21.15.29:0/3518425877 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480349f0 con 0x7f743c004d00 2023-10-14T16:40:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 -- 172.21.15.29:0/3518425877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f743c004d00 msgr2=0x7f743c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 --2- 172.21.15.29:0/3518425877 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f743c004d00 0x7f743c005180 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f744800a860 tx=0x7f743c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f74558ba640 1 -- 172.21.15.29:0/3518425877 reap_dead start 2023-10-14T16:40:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 -- 172.21.15.29:0/3518425877 shutdown_connections 2023-10-14T16:40:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 -- 172.21.15.29:0/3518425877 >> 172.21.15.29:0/3518425877 conn(0x7f74500ce9b0 msgr2=0x7f74500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 -- 172.21.15.29:0/3518425877 shutdown_connections 2023-10-14T16:40:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.267+0000 7f745731d640 1 -- 172.21.15.29:0/3518425877 wait complete. 2023-10-14T16:40:41.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.269+0000 7f745731d640 1 Processor -- start 2023-10-14T16:40:41.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.269+0000 7f745731d640 1 -- start start 2023-10-14T16:40:41.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f743c004d00 0x7f7450155cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74501573a0 0x7f74501f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7450156f10 con 0x7f74500b3530 2023-10-14T16:40:41.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74501f39e0 con 0x7f74501573a0 2023-10-14T16:40:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.270+0000 7f745731d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7450156300 con 0x7f743c004d00 2023-10-14T16:40:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48814/0 (socket says 172.21.15.29:48814) 2023-10-14T16:40:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 -- 172.21.15.29:0/784013586 learned_addr learned my addr 172.21.15.29:0/784013586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:41.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74550b9640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f743c004d00 0x7f7450155cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74548b8640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74501573a0 0x7f74501f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 -- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f743c004d00 msgr2=0x7f7450155cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f743c004d00 0x7f7450155cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 -- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74501573a0 msgr2=0x7f74501f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74501573a0 0x7f74501f34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.271+0000 7f74558ba640 1 -- 172.21.15.29:0/784013586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74501565e0 con 0x7f74500b3530 2023-10-14T16:40:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f74558ba640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f744800a860 tx=0x7f7448001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7448001fd0 con 0x7f74500b3530 2023-10-14T16:40:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7448034640 con 0x7f74500b3530 2023-10-14T16:40:41.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74480349d0 con 0x7f74500b3530 2023-10-14T16:40:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74501568c0 con 0x7f74500b3530 2023-10-14T16:40:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.272+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74500b39b0 con 0x7f74500b3530 2023-10-14T16:40:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.273+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7450157080 con 0x7f74500b3530 2023-10-14T16:40:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.274+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7448031030 con 0x7f74500b3530 2023-10-14T16:40:41.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.275+0000 7f7445ffb640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 0x7f743406d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.275+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f74480ab6f0 con 0x7f74500b3530 2023-10-14T16:40:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.276+0000 7f74550b9640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 0x7f743406d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.276+0000 7f74550b9640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 0x7f743406d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7450155c10 tx=0x7f744c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.280+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f744807a2a0 con 0x7f74500b3530 2023-10-14T16:40:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.476+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7450157050 con 0x7f74500b3530 2023-10-14T16:40:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.477+0000 7f7445ffb640 1 -- 172.21.15.29:0/784013586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7448079e30 con 0x7f74500b3530 2023-10-14T16:40:41.478 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:40:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.479+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 msgr2=0x7f743406d3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.479+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 0x7f743406d3b0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7450155c10 tx=0x7f744c008040 comp rx=0 tx=0).stop 2023-10-14T16:40:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.479+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 msgr2=0x7f74500c6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.479+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f744800a860 tx=0x7f7448001d80 comp rx=0 tx=0).stop 2023-10-14T16:40:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 shutdown_connections 2023-10-14T16:40:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f743406aef0 0x7f743406d3b0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74500b3530 0x7f74500c6400 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74501573a0 0x7f74501f34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 --2- 172.21.15.29:0/784013586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f743c004d00 0x7f7450155cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 >> 172.21.15.29:0/784013586 conn(0x7f74500ce9b0 msgr2=0x7f74500ad5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:41.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 shutdown_connections 2023-10-14T16:40:41.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.480+0000 7f745731d640 1 -- 172.21.15.29:0/784013586 wait complete. 2023-10-14T16:40:41.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:40:41.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.726+0000 7fc131e61640 1 Processor -- start 2023-10-14T16:40:41.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.727+0000 7fc131e61640 1 -- start start 2023-10-14T16:40:41.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c15aaa0 0x7fc12c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15ae80 0x7fc12c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc12c0ccaa0 0x7fc12c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c06b810 con 0x7fc12c15aaa0 2023-10-14T16:40:41.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c15ed60 con 0x7fc12c0ccaa0 2023-10-14T16:40:41.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc131e61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0707e0 con 0x7fc12c15ae80 2023-10-14T16:40:41.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc12b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c15aaa0 0x7fc12c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc12b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c15aaa0 0x7fc12c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48830/0 (socket says 172.21.15.29:48830) 2023-10-14T16:40:41.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2161032270 learned_addr learned my addr 172.21.15.29:0/2161032270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:41.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.728+0000 7fc12affd640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15ae80 0x7fc12c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12bfff640 1 --2- 172.21.15.29:0/2161032270 >> v2:172.21.15.186:3300/0 conn(0x7fc12c0ccaa0 0x7fc12c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2161032270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15ae80 msgr2=0x7fc12c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15ae80 0x7fc12c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2161032270 >> v2:172.21.15.186:3300/0 conn(0x7fc12c0ccaa0 msgr2=0x7fc12c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:41.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2161032270 >> v2:172.21.15.186:3300/0 conn(0x7fc12c0ccaa0 0x7fc12c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.729+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2161032270 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c15ef90 con 0x7fc12c15aaa0 2023-10-14T16:40:41.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.730+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15aaa0 0x7fc12c15e820 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0ae3c0 tx=0x7fc11c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d40f80b93b6ebaa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.730+0000 7fc12a7fc640 1 -- 172.21.15.29:0/2161032270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c02ca80 con 0x7fc12c15aaa0 2023-10-14T16:40:41.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.730+0000 7fc12a7fc640 1 -- 172.21.15.29:0/2161032270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c02cbe0 con 0x7fc12c15aaa0 2023-10-14T16:40:41.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.730+0000 7fc12a7fc640 1 -- 172.21.15.29:0/2161032270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c0343f0 con 0x7fc12c15aaa0 2023-10-14T16:40:41.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 -- 172.21.15.29:0/2161032270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15aaa0 msgr2=0x7fc12c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15aaa0 0x7fc12c15e820 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fc12c0ae3c0 tx=0x7fc11c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 -- 172.21.15.29:0/2161032270 shutdown_connections 2023-10-14T16:40:41.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 --2- 172.21.15.29:0/2161032270 >> v2:172.21.15.186:3300/0 conn(0x7fc12c0ccaa0 0x7fc12c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15ae80 0x7fc12c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 --2- 172.21.15.29:0/2161032270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15aaa0 0x7fc12c15e820 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 -- 172.21.15.29:0/2161032270 >> 172.21.15.29:0/2161032270 conn(0x7fc12c0cb410 msgr2=0x7fc12c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:41.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 -- 172.21.15.29:0/2161032270 shutdown_connections 2023-10-14T16:40:41.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.731+0000 7fc131e61640 1 -- 172.21.15.29:0/2161032270 wait complete. 2023-10-14T16:40:41.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.733+0000 7fc131e61640 1 Processor -- start 2023-10-14T16:40:41.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.733+0000 7fc131e61640 1 -- start start 2023-10-14T16:40:41.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0ccaa0 0x7fc12c158cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15aaa0 0x7fc12c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c15b9e0 con 0x7fc12c15ae80 2023-10-14T16:40:41.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c156470 con 0x7fc12c0ccaa0 2023-10-14T16:40:41.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc131e61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c15b640 con 0x7fc12c15aaa0 2023-10-14T16:40:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc12bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc12bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48846/0 (socket says 172.21.15.29:48846) 2023-10-14T16:40:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc12bfff640 1 -- 172.21.15.29:0/243915721 learned_addr learned my addr 172.21.15.29:0/243915721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0ccaa0 0x7fc12c158cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.734+0000 7fc12affd640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15aaa0 0x7fc12c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 -- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15aaa0 msgr2=0x7fc12c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15aaa0 0x7fc12c155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 -- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0ccaa0 msgr2=0x7fc12c158cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0ccaa0 0x7fc12c158cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 -- 172.21.15.29:0/243915721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c009b30 con 0x7fc12c15ae80 2023-10-14T16:40:41.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc12bfff640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c4c0 tx=0x7fc12400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124004220 con 0x7fc12c15ae80 2023-10-14T16:40:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc124004380 con 0x7fc12c15ae80 2023-10-14T16:40:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc124003810 con 0x7fc12c15ae80 2023-10-14T16:40:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c15e240 con 0x7fc12c15ae80 2023-10-14T16:40:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.735+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c1595b0 con 0x7fc12c15ae80 2023-10-14T16:40:41.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.737+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c15bb20 con 0x7fc12c15ae80 2023-10-14T16:40:41.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.738+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fc124003970 con 0x7fc12c15ae80 2023-10-14T16:40:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.739+0000 7fc123fff640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 0x7fc10c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.739+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fc1240892d0 con 0x7fc12c15ae80 2023-10-14T16:40:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.739+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 0x7fc10c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.740+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 0x7fc10c06d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc11c035000 tx=0x7fc11c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.743+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc12c15bb20 con 0x7fc12c15ae80 2023-10-14T16:40:41.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.990+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc12c15bb20 con 0x7fc12c15ae80 2023-10-14T16:40:41.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.991+0000 7fc123fff640 1 -- 172.21.15.29:0/243915721 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fc12c15bb20 con 0x7fc12c15ae80 2023-10-14T16:40:41.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:41.992 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:40:41.993 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:40:41.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.993+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 msgr2=0x7fc10c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.993+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 0x7fc10c06d330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc11c035000 tx=0x7fc11c030000 comp rx=0 tx=0).stop 2023-10-14T16:40:41.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.993+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 msgr2=0x7fc12c15dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:41.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.993+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc12400c4c0 tx=0x7fc12400c980 comp rx=0 tx=0).stop 2023-10-14T16:40:41.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 shutdown_connections 2023-10-14T16:40:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fc10c06ae70 0x7fc10c06d330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c15ae80 0x7fc12c15dd00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c15aaa0 0x7fc12c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 --2- 172.21.15.29:0/243915721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0ccaa0 0x7fc12c158cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 >> 172.21.15.29:0/243915721 conn(0x7fc12c0cb410 msgr2=0x7fc12c06ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 shutdown_connections 2023-10-14T16:40:41.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:41.994+0000 7fc131e61640 1 -- 172.21.15.29:0/243915721 wait complete. 2023-10-14T16:40:42.015 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:42.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.238+0000 7fa91359e640 1 Processor -- start 2023-10-14T16:40:42.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.238+0000 7fa91359e640 1 -- start start 2023-10-14T16:40:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.239+0000 7fa91359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa91415cfc0 0x7fa9140b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa91359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 0x7fa9140b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa91359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140d15a0 con 0x7fa91415cbe0 2023-10-14T16:40:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa91359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9140d1880 con 0x7fa91415cfc0 2023-10-14T16:40:42.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa91359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9140d1b60 con 0x7fa9140d2640 2023-10-14T16:40:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa91415cfc0 0x7fa9140b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa91415cfc0 0x7fa9140b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48862/0 (socket says 172.21.15.29:48862) 2023-10-14T16:40:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 -- 172.21.15.29:0/234535387 learned_addr learned my addr 172.21.15.29:0/234535387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:42.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa912d9d640 1 --2- 172.21.15.29:0/234535387 >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 0x7fa9140b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 -- 172.21.15.29:0/234535387 >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 msgr2=0x7fa9140b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 --2- 172.21.15.29:0/234535387 >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 0x7fa9140b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 -- 172.21.15.29:0/234535387 >> v1:172.21.15.186:6789/0 conn(0x7fa91415cbe0 legacy=0x7fa9140d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:42.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa911d9b640 1 -- 172.21.15.29:0/234535387 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140d1e40 con 0x7fa91415cfc0 2023-10-14T16:40:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.240+0000 7fa912d9d640 1 --2- 172.21.15.29:0/234535387 >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 0x7fa9140b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.242+0000 7fa911d9b640 1 --2- 172.21.15.29:0/234535387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa9140b4570 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b5b70 tx=0x7fa8fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3be0d2c35b16085 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.242+0000 7fa91159a640 1 -- 172.21.15.29:0/234535387 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc030860 con 0x7fa91415cfc0 2023-10-14T16:40:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.242+0000 7fa91159a640 1 -- 172.21.15.29:0/234535387 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8fc0309c0 con 0x7fa91415cfc0 2023-10-14T16:40:42.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.242+0000 7fa91159a640 1 -- 172.21.15.29:0/234535387 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8fc030d50 con 0x7fa91415cfc0 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 -- 172.21.15.29:0/234535387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 msgr2=0x7fa9140b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 --2- 172.21.15.29:0/234535387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa9140b4570 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa9140b5b70 tx=0x7fa8fc02b9a0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 -- 172.21.15.29:0/234535387 shutdown_connections 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 --2- 172.21.15.29:0/234535387 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa9140b4570 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 --2- 172.21.15.29:0/234535387 >> v2:172.21.15.186:3300/0 conn(0x7fa9140d2640 0x7fa9140b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 -- 172.21.15.29:0/234535387 >> 172.21.15.29:0/234535387 conn(0x7fa9140cf570 msgr2=0x7fa9140cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 -- 172.21.15.29:0/234535387 shutdown_connections 2023-10-14T16:40:42.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.243+0000 7fa91359e640 1 -- 172.21.15.29:0/234535387 wait complete. 2023-10-14T16:40:42.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.245+0000 7fa91359e640 1 Processor -- start 2023-10-14T16:40:42.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.245+0000 7fa91359e640 1 -- start start 2023-10-14T16:40:42.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 0x7fa9140b0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa91415cbe0 0x7fa9140c7670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c8cd0 con 0x7fa91415cfc0 2023-10-14T16:40:42.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c80c0 con 0x7fa91415cbe0 2023-10-14T16:40:42.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.246+0000 7fa91359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140c83a0 con 0x7fa9140d2640 2023-10-14T16:40:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48878/0 (socket says 172.21.15.29:48878) 2023-10-14T16:40:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 -- 172.21.15.29:0/2323142014 learned_addr learned my addr 172.21.15.29:0/2323142014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa911d9b640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa91415cbe0 0x7fa9140c7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa91259c640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 0x7fa9140b0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 -- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 msgr2=0x7fa9140b0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 0x7fa9140b0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 -- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa91415cbe0 msgr2=0x7fa9140c7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa91415cbe0 0x7fa9140c7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa912d9d640 1 -- 172.21.15.29:0/2323142014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8fc0094d0 con 0x7fa91415cfc0 2023-10-14T16:40:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.247+0000 7fa91259c640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 0x7fa9140b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa912d9d640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c5e0 tx=0x7fa90c00caa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c011800 con 0x7fa91415cfc0 2023-10-14T16:40:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c0119a0 con 0x7fa91415cfc0 2023-10-14T16:40:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9140c8680 con 0x7fa91415cfc0 2023-10-14T16:40:42.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa90c011d50 con 0x7fa91415cfc0 2023-10-14T16:40:42.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.248+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9140b1260 con 0x7fa91415cfc0 2023-10-14T16:40:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.249+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9140c8e40 con 0x7fa91415cfc0 2023-10-14T16:40:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.250+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa90c022350 con 0x7fa91415cfc0 2023-10-14T16:40:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.251+0000 7fa90affd640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 0x7fa8f406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.251+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa90c089760 con 0x7fa91415cfc0 2023-10-14T16:40:42.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.251+0000 7fa91259c640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 0x7fa8f406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.252+0000 7fa91259c640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 0x7fa8f406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa904004520 tx=0x7fa904009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.256+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa90c054770 con 0x7fa91415cfc0 2023-10-14T16:40:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.458+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa91404e9b0 con 0x7fa91415cfc0 2023-10-14T16:40:42.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.460+0000 7fa90affd640 1 -- 172.21.15.29:0/2323142014 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fa90c057d90 con 0x7fa91415cfc0 2023-10-14T16:40:42.461 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:42.461 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 msgr2=0x7fa8f406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 0x7fa8f406d330 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa904004520 tx=0x7fa904009040 comp rx=0 tx=0).stop 2023-10-14T16:40:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 msgr2=0x7fa91415f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa90c00c5e0 tx=0x7fa90c00caa0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 shutdown_connections 2023-10-14T16:40:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa8f406ae70 0x7fa8f406d330 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa91415cfc0 0x7fa91415f5a0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa91415cbe0 0x7fa9140c7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 --2- 172.21.15.29:0/2323142014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9140d2640 0x7fa9140b0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 >> 172.21.15.29:0/2323142014 conn(0x7fa9140cf570 msgr2=0x7fa9140b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 shutdown_connections 2023-10-14T16:40:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.463+0000 7fa91359e640 1 -- 172.21.15.29:0/2323142014 wait complete. 2023-10-14T16:40:42.483 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:40:42.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.706+0000 7f100d705640 1 Processor -- start 2023-10-14T16:40:42.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.706+0000 7f100d705640 1 -- start start 2023-10-14T16:40:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f100d705640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 0x7f100815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f100d705640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10080ce390 0x7f1008070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f100d705640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f100815ed60 con 0x7f100815aaa0 2023-10-14T16:40:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f100d705640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f100806b810 con 0x7f10080ce390 2023-10-14T16:40:42.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f100d705640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080707e0 con 0x7f100815ae80 2023-10-14T16:40:42.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f1006ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f100815aaa0 0x7f10080cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36970/0 (socket says 172.21.15.29:36970) 2023-10-14T16:40:42.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f1006ffd640 1 -- 172.21.15.29:0/2804779171 learned_addr learned my addr 172.21.15.29:0/2804779171 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.707+0000 7f10067fc640 1 --2- 172.21.15.29:0/2804779171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 0x7f100815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f10077fe640 1 --2- 172.21.15.29:0/2804779171 >> v2:172.21.15.186:3300/0 conn(0x7f10080ce390 0x7f1008070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f1005ffb640 1 -- 172.21.15.29:0/2804779171 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1102926556 0 0) 0x7f100815ed60 con 0x7f100815aaa0 2023-10-14T16:40:42.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f1005ffb640 1 -- 172.21.15.29:0/2804779171 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ff4003520 con 0x7f100815aaa0 2023-10-14T16:40:42.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f10067fc640 1 -- 172.21.15.29:0/2804779171 >> v2:172.21.15.186:3300/0 conn(0x7f10080ce390 msgr2=0x7f1008070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:42.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f10067fc640 1 --2- 172.21.15.29:0/2804779171 >> v2:172.21.15.186:3300/0 conn(0x7f10080ce390 0x7f1008070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f10067fc640 1 -- 172.21.15.29:0/2804779171 >> v1:172.21.15.29:6789/0 conn(0x7f100815aaa0 legacy=0x7f10080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.708+0000 7f10067fc640 1 -- 172.21.15.29:0/2804779171 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f100815ef90 con 0x7f100815ae80 2023-10-14T16:40:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.709+0000 7f10067fc640 1 --2- 172.21.15.29:0/2804779171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 0x7f100815e820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00097a0 tx=0x7f0ff002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=846fce147ec862d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f1005ffb640 1 -- 172.21.15.29:0/2804779171 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0031830 con 0x7f100815ae80 2023-10-14T16:40:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f1005ffb640 1 -- 172.21.15.29:0/2804779171 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff0031990 con 0x7f100815ae80 2023-10-14T16:40:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f1005ffb640 1 -- 172.21.15.29:0/2804779171 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0031d00 con 0x7f100815ae80 2023-10-14T16:40:42.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 -- 172.21.15.29:0/2804779171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 msgr2=0x7f100815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 --2- 172.21.15.29:0/2804779171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 0x7f100815e820 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0ff00097a0 tx=0x7f0ff002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 -- 172.21.15.29:0/2804779171 shutdown_connections 2023-10-14T16:40:42.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 --2- 172.21.15.29:0/2804779171 >> v2:172.21.15.186:3300/0 conn(0x7f10080ce390 0x7f1008070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 --2- 172.21.15.29:0/2804779171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f100815ae80 0x7f100815e820 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 -- 172.21.15.29:0/2804779171 >> 172.21.15.29:0/2804779171 conn(0x7f10080cb410 msgr2=0x7f10080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:42.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 -- 172.21.15.29:0/2804779171 shutdown_connections 2023-10-14T16:40:42.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.710+0000 7f100d705640 1 -- 172.21.15.29:0/2804779171 wait complete. 2023-10-14T16:40:42.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.712+0000 7f100d705640 1 Processor -- start 2023-10-14T16:40:42.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.712+0000 7f100d705640 1 -- start start 2023-10-14T16:40:42.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10080ce390 0x7f10081f6650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f100815ae80 0x7f10080c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10081f3640 con 0x7f100815aaa0 2023-10-14T16:40:42.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10080ae620 con 0x7f100815ae80 2023-10-14T16:40:42.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.713+0000 7f100d705640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10080c3e40 con 0x7f10080ce390 2023-10-14T16:40:42.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48884/0 (socket says 172.21.15.29:48884) 2023-10-14T16:40:42.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 -- 172.21.15.29:0/1933027217 learned_addr learned my addr 172.21.15.29:0/1933027217 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:42.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f1006ffd640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10080ce390 0x7f10081f6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10077fe640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f100815ae80 0x7f10080c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 -- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10080ce390 msgr2=0x7f10081f6650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10080ce390 0x7f10081f6650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 -- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f100815ae80 msgr2=0x7f10080c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f100815ae80 0x7f10080c24a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.714+0000 7f10067fc640 1 -- 172.21.15.29:0/1933027217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff0009400 con 0x7f100815aaa0 2023-10-14T16:40:42.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f10067fc640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0000c00 tx=0x7f0ff00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0004250 con 0x7f100815aaa0 2023-10-14T16:40:42.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ff00043b0 con 0x7f100815aaa0 2023-10-14T16:40:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ff0004520 con 0x7f100815aaa0 2023-10-14T16:40:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1008158770 con 0x7f100815aaa0 2023-10-14T16:40:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.715+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10081f3080 con 0x7f100815aaa0 2023-10-14T16:40:42.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.716+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10080ae760 con 0x7f100815aaa0 2023-10-14T16:40:42.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.717+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0ff0003830 con 0x7f100815aaa0 2023-10-14T16:40:42.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.718+0000 7f0fff7fe640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 0x7f0fe806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.718+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f0ff00aaff0 con 0x7f100815aaa0 2023-10-14T16:40:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.718+0000 7f1006ffd640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 0x7f0fe806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.719+0000 7f1006ffd640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 0x7f0fe806d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005c00 tx=0x7f0ff8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:42.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.722+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10080ae760 con 0x7f100815aaa0 2023-10-14T16:40:42.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.955+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10080ae760 con 0x7f100815aaa0 2023-10-14T16:40:42.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.957+0000 7f0fff7fe640 1 -- 172.21.15.29:0/1933027217 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f10080ae760 con 0x7f100815aaa0 2023-10-14T16:40:42.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:42.958 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:40:42.959 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:40:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 msgr2=0x7f0fe806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 0x7f0fe806d330 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f0ff8005c00 tx=0x7f0ff8009040 comp rx=0 tx=0).stop 2023-10-14T16:40:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 msgr2=0x7f10081f2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:42.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0ff0000c00 tx=0x7f0ff00046c0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 shutdown_connections 2023-10-14T16:40:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0fe806ae70 0x7f0fe806d330 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f100815ae80 0x7f10080c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f100815aaa0 0x7f10081f2650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 --2- 172.21.15.29:0/1933027217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10080ce390 0x7f10081f6650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 >> 172.21.15.29:0/1933027217 conn(0x7f10080cb410 msgr2=0x7f10080ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 shutdown_connections 2023-10-14T16:40:42.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:42.960+0000 7f100d705640 1 -- 172.21.15.29:0/1933027217 wait complete. 2023-10-14T16:40:43.381 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 8, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:40:25.223449+0000', 'modified': '2023-10-14T16:40:28.288537+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4272}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4272': {'gid': 4272, 'name': 'c', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/3245669368', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 3245669368}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 3245669368}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:40:43.382 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:40:43.382 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T16:40:43.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.617+0000 7f40ce46f640 1 Processor -- start 2023-10-14T16:40:43.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.617+0000 7f40ce46f640 1 -- start start 2023-10-14T16:40:43.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.618+0000 7f40ce46f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f40c815d150 0x7f40c80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.618+0000 7f40ce46f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40c806a4b0 0x7f40c8160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.618+0000 7f40ce46f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40c80ac890 con 0x7f40c8160b90 2023-10-14T16:40:43.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.618+0000 7f40ce46f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c815d530 con 0x7f40c806a4b0 2023-10-14T16:40:43.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.618+0000 7f40ce46f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40c8161650 con 0x7f40c815d150 2023-10-14T16:40:43.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.619+0000 7f40cca0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40c806a4b0 0x7f40c8160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.619+0000 7f40cca0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40c806a4b0 0x7f40c8160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48892/0 (socket says 172.21.15.29:48892) 2023-10-14T16:40:43.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.619+0000 7f40cca0c640 1 -- 172.21.15.29:0/3655886862 learned_addr learned my addr 172.21.15.29:0/3655886862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.619+0000 7f40c7fff640 1 --2- 172.21.15.29:0/3655886862 >> v2:172.21.15.186:3300/0 conn(0x7f40c815d150 0x7f40c80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.620+0000 7f40cca0c640 1 -- 172.21.15.29:0/3655886862 >> v2:172.21.15.186:3300/0 conn(0x7f40c815d150 msgr2=0x7f40c80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.620+0000 7f40cca0c640 1 --2- 172.21.15.29:0/3655886862 >> v2:172.21.15.186:3300/0 conn(0x7f40c815d150 0x7f40c80ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.620+0000 7f40cca0c640 1 -- 172.21.15.29:0/3655886862 >> v1:172.21.15.186:6789/0 conn(0x7f40c8160b90 legacy=0x7f40c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.620+0000 7f40cca0c640 1 -- 172.21.15.29:0/3655886862 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c8161930 con 0x7f40c806a4b0 2023-10-14T16:40:43.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.621+0000 7f40cca0c640 1 --2- 172.21.15.29:0/3655886862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c806a4b0 0x7f40c8160fe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b15a0 tx=0x7f40c002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eeee1d28e9e5c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.621+0000 7f40c77fe640 1 -- 172.21.15.29:0/3655886862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c002ca80 con 0x7f40c806a4b0 2023-10-14T16:40:43.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.621+0000 7f40c77fe640 1 -- 172.21.15.29:0/3655886862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c002cbe0 con 0x7f40c806a4b0 2023-10-14T16:40:43.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.621+0000 7f40c77fe640 1 -- 172.21.15.29:0/3655886862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c00343f0 con 0x7f40c806a4b0 2023-10-14T16:40:43.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 -- 172.21.15.29:0/3655886862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c806a4b0 msgr2=0x7f40c8160fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 --2- 172.21.15.29:0/3655886862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c806a4b0 0x7f40c8160fe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f40c80b15a0 tx=0x7f40c002bee0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 -- 172.21.15.29:0/3655886862 shutdown_connections 2023-10-14T16:40:43.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 --2- 172.21.15.29:0/3655886862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c806a4b0 0x7f40c8160fe0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 --2- 172.21.15.29:0/3655886862 >> v2:172.21.15.186:3300/0 conn(0x7f40c815d150 0x7f40c80ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 -- 172.21.15.29:0/3655886862 >> 172.21.15.29:0/3655886862 conn(0x7f40c80ce9b0 msgr2=0x7f40c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 -- 172.21.15.29:0/3655886862 shutdown_connections 2023-10-14T16:40:43.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.622+0000 7f40ce46f640 1 -- 172.21.15.29:0/3655886862 wait complete. 2023-10-14T16:40:43.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.624+0000 7f40ce46f640 1 Processor -- start 2023-10-14T16:40:43.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.624+0000 7f40ce46f640 1 -- start start 2023-10-14T16:40:43.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.625+0000 7f40ce46f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c806a4b0 0x7f40c81439b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.625+0000 7f40ce46f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 0x7f40c80b0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40ce46f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40ce46f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80b11f0 con 0x7f40c8160b90 2023-10-14T16:40:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40ce46f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c815e880 con 0x7f40c806a4b0 2023-10-14T16:40:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40ce46f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40c80b0f10 con 0x7f40c815d150 2023-10-14T16:40:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48894/0 (socket says 172.21.15.29:48894) 2023-10-14T16:40:43.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 -- 172.21.15.29:0/2106384513 learned_addr learned my addr 172.21.15.29:0/2106384513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40c7fff640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c806a4b0 0x7f40c81439b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40bffff640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 0x7f40c80b0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 -- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 msgr2=0x7f40c80b0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 0x7f40c80b0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 -- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c806a4b0 msgr2=0x7f40c81439b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c806a4b0 0x7f40c81439b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40cca0c640 1 -- 172.21.15.29:0/2106384513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40c0009b30 con 0x7f40c8160b90 2023-10-14T16:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.626+0000 7f40bffff640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 0x7f40c80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40cca0c640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f40c0036040 tx=0x7f40c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40c57fa640 1 -- 172.21.15.29:0/2106384513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c0009510 con 0x7f40c8160b90 2023-10-14T16:40:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40c57fa640 1 -- 172.21.15.29:0/2106384513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40c003e450 con 0x7f40c8160b90 2023-10-14T16:40:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40c57fa640 1 -- 172.21.15.29:0/2106384513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40c003e800 con 0x7f40c8160b90 2023-10-14T16:40:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c815eb20 con 0x7f40c8160b90 2023-10-14T16:40:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.627+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c80ae8a0 con 0x7f40c8160b90 2023-10-14T16:40:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.630+0000 7f40c57fa640 1 -- 172.21.15.29:0/2106384513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f40c002cd50 con 0x7f40c8160b90 2023-10-14T16:40:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.631+0000 7f40c57fa640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 0x7f40a806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.631+0000 7f40c57fa640 1 -- 172.21.15.29:0/2106384513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40c00abde0 con 0x7f40c8160b90 2023-10-14T16:40:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.631+0000 7f40c7fff640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 0x7f40a806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.632+0000 7f40c7fff640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 0x7f40a806d360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.632+0000 7f40d1000740 1 Processor -- start 2023-10-14T16:40:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.632+0000 7f40d1000740 1 -- start start 2023-10-14T16:40:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ec6360 0x556af2ec8d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2e993c0 0x556af2ec9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2db97a0 con 0x556af2ec5f80 2023-10-14T16:40:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2e0b740 con 0x556af2ec6360 2023-10-14T16:40:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556af2e08e30 con 0x556af2e993c0 2023-10-14T16:40:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48896/0 (socket says 172.21.15.29:48896) 2023-10-14T16:40:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40c7fff640 1 -- 172.21.15.29:0/1997099743 learned_addr learned my addr 172.21.15.29:0/1997099743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40bffff640 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ec6360 0x556af2ec8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40cca0c640 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2e993c0 0x556af2ec9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40c7fff640 1 -- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2e993c0 msgr2=0x556af2ec9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.633+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2e993c0 0x556af2ec9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40c7fff640 1 -- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ec6360 msgr2=0x556af2ec8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ec6360 0x556af2ec8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40c7fff640 1 -- 172.21.15.29:0/1997099743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556af2dc8910 con 0x556af2ec5f80 2023-10-14T16:40:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40b800fa50 tx=0x7f40b8012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d7c0e18e1153ae6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40bdffb640 1 -- 172.21.15.29:0/1997099743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8014be0 con 0x556af2ec5f80 2023-10-14T16:40:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40bdffb640 1 -- 172.21.15.29:0/1997099743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8019070 con 0x556af2ec5f80 2023-10-14T16:40:43.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.634+0000 7f40bdffb640 1 -- 172.21.15.29:0/1997099743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b8020440 con 0x556af2ec5f80 2023-10-14T16:40:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 -- 172.21.15.29:0/1997099743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 msgr2=0x556af2e98e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f40b800fa50 tx=0x7f40b8012000 comp rx=0 tx=0).stop 2023-10-14T16:40:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 -- 172.21.15.29:0/1997099743 shutdown_connections 2023-10-14T16:40:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2e993c0 0x556af2ec9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ec6360 0x556af2ec8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 --2- 172.21.15.29:0/1997099743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec5f80 0x556af2e98e80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 -- 172.21.15.29:0/1997099743 >> 172.21.15.29:0/1997099743 conn(0x556af2e9b190 msgr2=0x556af2ee3d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 -- 172.21.15.29:0/1997099743 shutdown_connections 2023-10-14T16:40:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.635+0000 7f40d1000740 1 -- 172.21.15.29:0/1997099743 wait complete. 2023-10-14T16:40:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.636+0000 7f40d1000740 1 -- start start 2023-10-14T16:40:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.636+0000 7f40d1000740 1 Processor -- start 2023-10-14T16:40:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2eca510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eb4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2e04720 con 0x556af2ec6360 2023-10-14T16:40:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2da2970 con 0x556af2ecbc30 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556af2da2610 con 0x556af2ec5f80 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40d1000740 10 client.0 _ll_register_callbacks cb 0x556af2dc49d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48908/0 (socket says 172.21.15.29:48908) 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.637+0000 7f40bffff640 1 -- 172.21.15.29:0/1021978458 learned_addr learned my addr 172.21.15.29:0/1021978458 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2eca510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40cca0c640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eb4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 -- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 msgr2=0x556af2eca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2eca510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 -- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 msgr2=0x556af2eb4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eb4550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 -- 172.21.15.29:0/1021978458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b800e040 con 0x556af2ec6360 2023-10-14T16:40:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40bffff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001d80 tx=0x7f40b0001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26989e7ec13936f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:40:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0005110 con 0x556af2ec6360 2023-10-14T16:40:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40b0005270 con 0x556af2ec6360 2023-10-14T16:40:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.638+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556af2e2b160 con 0x556af2ec6360 2023-10-14T16:40:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b0005430 con 0x556af2ec6360 2023-10-14T16:40:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 10 client.4659 fetch_fsmap learned FSMap version 8 2023-10-14T16:40:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x556af2e02da0 con 0x556af2ec6360 2023-10-14T16:40:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556af2e2b160 con 0x556af2ec6360 2023-10-14T16:40:43.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f40b0012070 con 0x556af2ec6360 2023-10-14T16:40:43.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 10 client.4659 fetch_fsmap finished waiting for FSMap version 8 2023-10-14T16:40:43.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 10 client.4659 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:40:43.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 20 client.4659 populate_metadata read hostname 'smithi029' 2023-10-14T16:40:43.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ef3a50 0x556af2ef3e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 4 client.4659 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:40:43.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 --> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556af2dd6cc0 con 0x556af2ef3a50 2023-10-14T16:40:43.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.639+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ef3a50 0x556af2ef3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.640+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ef3a50 0x556af2ef3e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6638942e8030520a server_cookie=e5fb2588fed1b2a4 in_seq=0 out_seq=0 2023-10-14T16:40:43.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.640+0000 7f40a6ffd640 10 client.4659 ms_handle_connect on v2:172.21.15.29:6836/3245669368 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1021978458 <== mds.0 v2:172.21.15.29:6836/3245669368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556af2dd6cc0 con 0x556af2ef3a50 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40a6ffd640 10 client.4659 handle_command_reply: tid=0 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 client.4659 shutdown 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 msgr2=0x556af2eb7020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001d80 tx=0x7f40b0001e70 comp rx=0 tx=0).stop 2023-10-14T16:40:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 shutdown_connections 2023-10-14T16:40:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eb4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2eb7020 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ef3a50 0x556af2ef3e30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 --2- 172.21.15.29:0/1021978458 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2eca510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 >> 172.21.15.29:0/1021978458 conn(0x556af2e993c0 msgr2=0x556af2e9ce00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40a6ffd640 0 client.4659 ms_handle_reset on v2:172.21.15.29:6836/3245669368 2023-10-14T16:40:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 shutdown_connections 2023-10-14T16:40:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.641+0000 7f40d1000740 1 -- 172.21.15.29:0/1021978458 wait complete. 2023-10-14T16:40:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.642+0000 7f40d1000740 20 client.4659 trim_cache size 0 max 16384 2023-10-14T16:40:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.658+0000 7f40d1000740 1 Processor -- start 2023-10-14T16:40:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.658+0000 7f40d1000740 1 -- start start 2023-10-14T16:40:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2ec9d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2b68f20 con 0x556af2ec6360 2023-10-14T16:40:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2ecb4c0 con 0x556af2ecbc30 2023-10-14T16:40:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556af2eda8c0 con 0x556af2ec5f80 2023-10-14T16:40:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.659+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48912/0 (socket says 172.21.15.29:48912) 2023-10-14T16:40:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 -- 172.21.15.29:0/2599752391 learned_addr learned my addr 172.21.15.29:0/2599752391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40c7fff640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2ec9d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40cca0c640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 -- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 msgr2=0x556af2ec9d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2ec9d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 -- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 msgr2=0x556af2eddb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 -- 172.21.15.29:0/2599752391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556af2da1b80 con 0x556af2ec6360 2023-10-14T16:40:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40c7fff640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2ec9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40bffff640 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b000d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6040d88a818f2283 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.660+0000 7f40be7fc640 1 -- 172.21.15.29:0/2599752391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0005070 con 0x556af2ec6360 2023-10-14T16:40:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40be7fc640 1 -- 172.21.15.29:0/2599752391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b000d070 con 0x556af2ec6360 2023-10-14T16:40:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40be7fc640 1 -- 172.21.15.29:0/2599752391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0005410 con 0x556af2ec6360 2023-10-14T16:40:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 -- 172.21.15.29:0/2599752391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 msgr2=0x556af2edd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f40b0000c00 tx=0x7f40b000d000 comp rx=0 tx=0).stop 2023-10-14T16:40:43.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 -- 172.21.15.29:0/2599752391 shutdown_connections 2023-10-14T16:40:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 --2- 172.21.15.29:0/2599752391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2ec9d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 -- 172.21.15.29:0/2599752391 >> 172.21.15.29:0/2599752391 conn(0x556af2e993c0 msgr2=0x556af2e9f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 -- 172.21.15.29:0/2599752391 shutdown_connections 2023-10-14T16:40:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.661+0000 7f40d1000740 1 -- 172.21.15.29:0/2599752391 wait complete. 2023-10-14T16:40:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.662+0000 7f40d1000740 1 -- start start 2023-10-14T16:40:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.662+0000 7f40d1000740 1 Processor -- start 2023-10-14T16:40:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2f4bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2ec94d0 con 0x556af2ec6360 2023-10-14T16:40:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556af2eb62e0 con 0x556af2ecbc30 2023-10-14T16:40:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556af2ec8fa0 con 0x556af2ec5f80 2023-10-14T16:40:43.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40d1000740 10 client.0 _ll_register_callbacks cb 0x7f40b0007d80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:40:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48918/0 (socket says 172.21.15.29:48918) 2023-10-14T16:40:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.663+0000 7f40bffff640 1 -- 172.21.15.29:0/1525679491 learned_addr learned my addr 172.21.15.29:0/1525679491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2f4bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40cca0c640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 -- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 msgr2=0x556af2f4bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2f4bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 -- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 msgr2=0x556af2eddb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 -- 172.21.15.29:0/1525679491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556af2e02da0 con 0x556af2ec6360 2023-10-14T16:40:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40bffff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001240 tx=0x7f40b0001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f7297127a6fedd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.664+0000 7f40a6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:40:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40b0007a20 con 0x556af2ec6360 2023-10-14T16:40:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f40b0033ab0 con 0x556af2ec6360 2023-10-14T16:40:43.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f40b8019070 con 0x556af2ec6360 2023-10-14T16:40:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=8) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40b0033c70 con 0x556af2ec6360 2023-10-14T16:40:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 10 client.4665 fetch_fsmap learned FSMap version 8 2023-10-14T16:40:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=8,monmap=2+}) v3 -- 0x556af2dc7170 con 0x556af2ec6360 2023-10-14T16:40:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40b8019070 con 0x556af2ec6360 2023-10-14T16:40:43.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 8) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f40b003b020 con 0x556af2ec6360 2023-10-14T16:40:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 10 client.4665 fetch_fsmap finished waiting for FSMap version 8 2023-10-14T16:40:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 10 client.4665 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:40:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 20 client.4665 populate_metadata read hostname 'smithi029' 2023-10-14T16:40:43.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ec3a90 0x556af2ec3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 4 client.4665 mds_command: new command op to 4272 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:40:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.665+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 --> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556af2dfd700 con 0x556af2ec3a90 2023-10-14T16:40:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ec3a90 0x556af2ec3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40c7fff640 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ec3a90 0x556af2ec3e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5caa79b60590b65 server_cookie=65c273926b5b1079 in_seq=0 out_seq=0 2023-10-14T16:40:43.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40a6ffd640 10 client.4665 ms_handle_connect on v2:172.21.15.29:6836/3245669368 2023-10-14T16:40:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40a6ffd640 1 -- 172.21.15.29:0/1525679491 <== mds.0 v2:172.21.15.29:6836/3245669368 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+330 (crc 0 0 0) 0x556af2dfd700 con 0x556af2ec3a90 2023-10-14T16:40:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40a6ffd640 10 client.4665 handle_command_reply: tid=0 2023-10-14T16:40:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.666+0000 7f40d1000740 1 client.4665 shutdown 2023-10-14T16:40:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 msgr2=0x556af2edd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f40b0001240 tx=0x7f40b0001080 comp rx=0 tx=0).stop 2023-10-14T16:40:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 shutdown_connections 2023-10-14T16:40:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556af2ecbc30 0x556af2eddb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556af2ec6360 0x556af2edd5f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.29:6836/3245669368,v1:172.21.15.29:6837/3245669368] conn(0x556af2ec3a90 0x556af2ec3e70 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 --2- 172.21.15.29:0/1525679491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556af2ec5f80 0x556af2f4bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 >> 172.21.15.29:0/1525679491 conn(0x556af2e993c0 msgr2=0x556af2ec9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40a6ffd640 0 client.4665 ms_handle_reset on v2:172.21.15.29:6836/3245669368 2023-10-14T16:40:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 shutdown_connections 2023-10-14T16:40:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 1 -- 172.21.15.29:0/1525679491 wait complete. 2023-10-14T16:40:43.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.667+0000 7f40d1000740 20 client.4665 trim_cache size 0 max 16384 2023-10-14T16:40:43.682 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:40:43.682 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:40:43.682 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:40:43.682 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 4272, 2023-10-14T16:40:43.683 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:40:43.683 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:40:43.683 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:40:43.683 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 17.928049739999999, 2023-10-14T16:40:43.684 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 8, 2023-10-14T16:40:43.684 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 20, 2023-10-14T16:40:43.684 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 20, 2023-10-14T16:40:43.684 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 23.191817149999999 2023-10-14T16:40:43.684 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:40:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 msgr2=0x7f40a806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 0x7f40a806d360 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f40b8006fd0 tx=0x7f40b8008040 comp rx=0 tx=0).stop 2023-10-14T16:40:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 msgr2=0x7f40c80ae360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f40c0036040 tx=0x7f40c0004a40 comp rx=0 tx=0).stop 2023-10-14T16:40:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 shutdown_connections 2023-10-14T16:40:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f40a806aea0 0x7f40a806d360 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40c8160b90 0x7f40c80ae360 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40c815d150 0x7f40c80b0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 --2- 172.21.15.29:0/2106384513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40c806a4b0 0x7f40c81439b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:43.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 >> 172.21.15.29:0/2106384513 conn(0x7f40c80ce9b0 msgr2=0x7f40c80c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:43.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 shutdown_connections 2023-10-14T16:40:43.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:43.668+0000 7f40ce46f640 1 -- 172.21.15.29:0/2106384513 wait complete. 2023-10-14T16:40:43.689 INFO:teuthology.run_tasks:Running task kclient... 2023-10-14T16:40:43.713 INFO:tasks.kclient:Mounting kernel clients... 2023-10-14T16:40:43.714 INFO:tasks.kclient:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2023-10-14T16:40:43.714 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:43.715 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:40:43.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:43.777 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-14T16:40:43.877 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-14T16:40:43.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:40:43.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:43.881 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:40:43.942 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:43.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-14T16:40:44.047 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-14T16:40:44.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:40:44.054 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.054 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T16:40:44.116 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2023-10-14T16:40:44.221 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2023-10-14T16:40:44.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:40:44.225 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.225 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T16:40:44.278 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.278 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2023-10-14T16:40:44.372 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2023-10-14T16:40:44.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T16:40:44.377 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2023-10-14T16:40:44.377 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T16:40:44.378 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:40:44.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:40:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.611+0000 7fa595ff0640 1 Processor -- start 2023-10-14T16:40:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.611+0000 7fa595ff0640 1 -- start start 2023-10-14T16:40:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.612+0000 7fa595ff0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa590159f00 0x7fa5900b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.612+0000 7fa595ff0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa59015d880 0x7fa5900b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.612+0000 7fa595ff0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59006f7f0 con 0x7fa590159b20 2023-10-14T16:40:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa595ff0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59006c5e0 con 0x7fa59015d880 2023-10-14T16:40:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa595ff0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa59006c8c0 con 0x7fa590159f00 2023-10-14T16:40:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa590159b20 0x7fa59006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36972/0 (socket says 172.21.15.29:36972) 2023-10-14T16:40:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58f7fe640 1 -- 172.21.15.29:0/2860973080 learned_addr learned my addr 172.21.15.29:0/2860973080 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58ffff640 1 --2- 172.21.15.29:0/2860973080 >> v2:172.21.15.29:3300/0 conn(0x7fa59015d880 0x7fa5900b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58e7fc640 1 -- 172.21.15.29:0/2860973080 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297448721 0 0) 0x7fa59006f7f0 con 0x7fa590159b20 2023-10-14T16:40:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58e7fc640 1 -- 172.21.15.29:0/2860973080 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa578003520 con 0x7fa590159b20 2023-10-14T16:40:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58effd640 1 --2- 172.21.15.29:0/2860973080 >> v2:172.21.15.186:3300/0 conn(0x7fa590159f00 0x7fa5900b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58ffff640 1 -- 172.21.15.29:0/2860973080 >> v2:172.21.15.186:3300/0 conn(0x7fa590159f00 msgr2=0x7fa5900b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58ffff640 1 --2- 172.21.15.29:0/2860973080 >> v2:172.21.15.186:3300/0 conn(0x7fa590159f00 0x7fa5900b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58ffff640 1 -- 172.21.15.29:0/2860973080 >> v1:172.21.15.29:6789/0 conn(0x7fa590159b20 legacy=0x7fa59006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.613+0000 7fa58ffff640 1 -- 172.21.15.29:0/2860973080 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59006cba0 con 0x7fa59015d880 2023-10-14T16:40:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa58ffff640 1 --2- 172.21.15.29:0/2860973080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa59015d880 0x7fa5900b0950 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa59015d770 tx=0x7fa58802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1dcbdb79ee7254c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa58e7fc640 1 -- 172.21.15.29:0/2860973080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588031830 con 0x7fa59015d880 2023-10-14T16:40:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa58e7fc640 1 -- 172.21.15.29:0/2860973080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa588031990 con 0x7fa59015d880 2023-10-14T16:40:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa58e7fc640 1 -- 172.21.15.29:0/2860973080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa588031d20 con 0x7fa59015d880 2023-10-14T16:40:44.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa595ff0640 1 -- 172.21.15.29:0/2860973080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa59015d880 msgr2=0x7fa5900b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.615+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2860973080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa59015d880 0x7fa5900b0950 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa59015d770 tx=0x7fa58802b8d0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 -- 172.21.15.29:0/2860973080 shutdown_connections 2023-10-14T16:40:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2860973080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa59015d880 0x7fa5900b0950 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2860973080 >> v2:172.21.15.186:3300/0 conn(0x7fa590159f00 0x7fa5900b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 -- 172.21.15.29:0/2860973080 >> 172.21.15.29:0/2860973080 conn(0x7fa5900cb410 msgr2=0x7fa5900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 -- 172.21.15.29:0/2860973080 shutdown_connections 2023-10-14T16:40:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.616+0000 7fa595ff0640 1 -- 172.21.15.29:0/2860973080 wait complete. 2023-10-14T16:40:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.617+0000 7fa595ff0640 1 Processor -- start 2023-10-14T16:40:44.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.618+0000 7fa595ff0640 1 -- start start 2023-10-14T16:40:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.618+0000 7fa595ff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa595ff0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa590159f00 0x7fa5901494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa595ff0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa59015d880 0x7fa590150840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa595ff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59014d390 con 0x7fa590159b20 2023-10-14T16:40:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa595ff0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59014d670 con 0x7fa590159f00 2023-10-14T16:40:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa595ff0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa59014d950 con 0x7fa59015d880 2023-10-14T16:40:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48936/0 (socket says 172.21.15.29:48936) 2023-10-14T16:40:44.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 -- 172.21.15.29:0/2944381844 learned_addr learned my addr 172.21.15.29:0/2944381844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 -- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa59015d880 msgr2=0x7fa590150840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa59015d880 0x7fa590150840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 -- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa590159f00 msgr2=0x7fa5901494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa590159f00 0x7fa5901494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 -- 172.21.15.29:0/2944381844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa588009400 con 0x7fa590159b20 2023-10-14T16:40:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.619+0000 7fa58f7fe640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa58000cbc0 tx=0x7fa58000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.620+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58000ad40 con 0x7fa590159b20 2023-10-14T16:40:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.620+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa580012400 con 0x7fa590159b20 2023-10-14T16:40:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.620+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59014dc30 con 0x7fa590159b20 2023-10-14T16:40:44.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.620+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58000aea0 con 0x7fa590159b20 2023-10-14T16:40:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.620+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa590149a30 con 0x7fa590159b20 2023-10-14T16:40:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.621+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590071720 con 0x7fa590159b20 2023-10-14T16:40:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.622+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa580007070 con 0x7fa590159b20 2023-10-14T16:40:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.623+0000 7fa577fff640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 0x7fa57c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:44.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.623+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa580089700 con 0x7fa590159b20 2023-10-14T16:40:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.623+0000 7fa58effd640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 0x7fa57c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.624+0000 7fa58effd640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 0x7fa57c06d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa584006fd0 tx=0x7fa584008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:44.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.627+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa580054710 con 0x7fa590159b20 2023-10-14T16:40:44.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.868+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fa59004e160 con 0x7fa590159b20 2023-10-14T16:40:44.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.869+0000 7fa577fff640 1 -- 172.21.15.29:0/2944381844 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fa580017060 con 0x7fa590159b20 2023-10-14T16:40:44.869 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:40:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.871+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 msgr2=0x7fa57c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.871+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 0x7fa57c06d330 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fa584006fd0 tx=0x7fa584008040 comp rx=0 tx=0).stop 2023-10-14T16:40:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 msgr2=0x7fa59014cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:44.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fa58000cbc0 tx=0x7fa58000a430 comp rx=0 tx=0).stop 2023-10-14T16:40:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 shutdown_connections 2023-10-14T16:40:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa57c06ae70 0x7fa57c06d330 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa59015d880 0x7fa590150840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa590159f00 0x7fa5901494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 --2- 172.21.15.29:0/2944381844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa590159b20 0x7fa59014cda0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 >> 172.21.15.29:0/2944381844 conn(0x7fa5900cb410 msgr2=0x7fa59006fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 shutdown_connections 2023-10-14T16:40:44.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:44.872+0000 7fa595ff0640 1 -- 172.21.15.29:0/2944381844 wait complete. 2023-10-14T16:40:44.891 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T16:40:44.891 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:40:44.891 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:40:44.892 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:40:44.892 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:40:44.892 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T16:40:44.893 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T16:40:44.894 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:40:44.894 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:40:44.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.895 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:40:44.952 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:44.952 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:44.952 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:44.953 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:44.953 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:44.953 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:44.953 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:40:44.954 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:44.954 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:40:44.955 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:40:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:40:44.955 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:40:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42331sec preferred_lft 42331sec 2023-10-14T16:40:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:40:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:44.956 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:44.956 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:44.957 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:40:44.958 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-14T16:40:44.958 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:44.959 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:44.959 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:44.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add name ceph-brx type bridge 2023-10-14T16:40:44.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr flush dev ceph-brx 2023-10-14T16:40:44.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set ceph-brx up 2023-10-14T16:40:44.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-14T16:40:44.960 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:45.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:45.602 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:40:45.689 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:40:45.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:45.693 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:40:45.751 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:40:45.751 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:40:45.752 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:40:45.752 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:40:45.752 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:40:45.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:45.753 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:45.753 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:45.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:40:45.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:40:45.754 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:40:45.755 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:47.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:47.108 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:40:47.173 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:47.173 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-14T16:40:47.232 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:47.233 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:47.233 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:47.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:40:47.233 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2023-10-14T16:40:47.234 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:47.386 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2023-10-14T16:40:47.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:47.387 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:47.387 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:47.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2023-10-14T16:40:47.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-14T16:40:47.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2023-10-14T16:40:47.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2023-10-14T16:40:47.389 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2023-10-14T16:40:47.389 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:47.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:47.730 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:47.730 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:47.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:40:47.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.0 master ceph-brx 2023-10-14T16:40:47.731 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:47.907 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:47.908 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:40:47.959 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:40:47.960 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:47.961 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:40:48.008 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:40:48.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:48.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:40:48.211 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:40:48.212 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:48.923 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:40:48.924 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:40:48.924 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:48.924 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:48.924 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:48.960 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:40:48.960 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:40:48.960 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:48.960 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:48.961 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:48.961 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:40:48.961 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:40:48.961 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:40:48.965 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:40:48.965 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:40:48.965 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:40:48.965 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:40:48.965 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:40:48.969 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:40:48.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:48.970 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:40:49.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:49.076 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:49.145 INFO:teuthology.orchestra.run.smithi029.stdout:client4677 2023-10-14T16:40:49.146 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:49.146 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:40:49.204 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:49.205 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:49.275 INFO:teuthology.orchestra.run.smithi029.stdout:client4677 2023-10-14T16:40:49.276 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:49.276 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:49.350 INFO:teuthology.orchestra.run.smithi029.stdout:client4677 2023-10-14T16:40:49.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:49.351 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f6aee177-7aff-4117-99f6-f2cafd5adc35.client4677/status') 2023-10-14T16:40:49.454 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.4677 (3)192.168.0.1:0/54716681 2023-10-14T16:40:49.454 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2023-10-14T16:40:49.454 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2023-10-14T16:40:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2023-10-14T16:40:49.455 INFO:teuthology.orchestra.run.smithi029.stderr:67 bytes copied, 6.9363e-05 s, 966 kB/s 2023-10-14T16:40:49.459 INFO:tasks.kclient:client.2 config is {'syntax': 'v1'} 2023-10-14T16:40:49.459 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T16:40:49.459 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-14T16:40:49.460 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:40:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.682+0000 7fb63115a640 1 Processor -- start 2023-10-14T16:40:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.682+0000 7fb63115a640 1 -- start start 2023-10-14T16:40:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.684+0000 7fb63115a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb62c159b20 0x7fb62c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.684+0000 7fb63115a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.684+0000 7fb63115a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c06f7f0 con 0x7fb62c159f00 2023-10-14T16:40:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.684+0000 7fb63115a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c06c5e0 con 0x7fb62c159b20 2023-10-14T16:40:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.684+0000 7fb63115a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c06c8c0 con 0x7fb62c0b0b20 2023-10-14T16:40:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb62c159b20 0x7fb62c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb62c159b20 0x7fb62c06bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:48100/0 (socket says 172.21.15.29:48100) 2023-10-14T16:40:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 -- 172.21.15.29:0/376378817 learned_addr learned my addr 172.21.15.29:0/376378817 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:49.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 -- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 msgr2=0x7fb62c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 --2- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 -- 172.21.15.29:0/376378817 >> v1:172.21.15.186:6789/0 conn(0x7fb62c159f00 legacy=0x7fb62c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.685+0000 7fb62ad9d640 1 -- 172.21.15.29:0/376378817 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c06cba0 con 0x7fb62c159b20 2023-10-14T16:40:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.687+0000 7fb62ad9d640 1 --2- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159b20 0x7fb62c06bbe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009e50 tx=0x7fb61c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519492399bc70b66 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.687+0000 7fb62a59c640 1 -- 172.21.15.29:0/376378817 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c02cb00 con 0x7fb62c159b20 2023-10-14T16:40:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.687+0000 7fb62a59c640 1 -- 172.21.15.29:0/376378817 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb61c02cc60 con 0x7fb62c159b20 2023-10-14T16:40:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.687+0000 7fb62a59c640 1 -- 172.21.15.29:0/376378817 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb61c0343f0 con 0x7fb62c159b20 2023-10-14T16:40:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 -- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159b20 msgr2=0x7fb62c06bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 --2- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159b20 0x7fb62c06bbe0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb61c009e50 tx=0x7fb61c02bf60 comp rx=0 tx=0).stop 2023-10-14T16:40:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 -- 172.21.15.29:0/376378817 shutdown_connections 2023-10-14T16:40:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 --2- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c06fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 --2- 172.21.15.29:0/376378817 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159b20 0x7fb62c06bbe0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 -- 172.21.15.29:0/376378817 >> 172.21.15.29:0/376378817 conn(0x7fb62c0cb410 msgr2=0x7fb62c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 -- 172.21.15.29:0/376378817 shutdown_connections 2023-10-14T16:40:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.688+0000 7fb63115a640 1 -- 172.21.15.29:0/376378817 wait complete. 2023-10-14T16:40:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.690+0000 7fb63115a640 1 Processor -- start 2023-10-14T16:40:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.690+0000 7fb63115a640 1 -- start start 2023-10-14T16:40:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c1f5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 0x7fb62c0c7160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c15b360 con 0x7fb62c159b20 2023-10-14T16:40:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0aced0 con 0x7fb62c159f00 2023-10-14T16:40:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.691+0000 7fb63115a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0ad170 con 0x7fb62c0b0b20 2023-10-14T16:40:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:48942/0 (socket says 172.21.15.29:48942) 2023-10-14T16:40:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 -- 172.21.15.29:0/1568988559 learned_addr learned my addr 172.21.15.29:0/1568988559 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62b59e640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 0x7fb62c0c7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62ad9d640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c1f5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 -- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 msgr2=0x7fb62c1f5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c1f5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 -- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 msgr2=0x7fb62c0c7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 0x7fb62c0c7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 -- 172.21.15.29:0/1568988559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c009b30 con 0x7fb62c159b20 2023-10-14T16:40:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62b59e640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 0x7fb62c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:40:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.692+0000 7fb62259c640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c440 tx=0x7fb614001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.693+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6140052d0 con 0x7fb62c159b20 2023-10-14T16:40:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.693+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb614005430 con 0x7fb62c159b20 2023-10-14T16:40:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.693+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb614007260 con 0x7fb62c159b20 2023-10-14T16:40:49.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.693+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c0ad450 con 0x7fb62c159b20 2023-10-14T16:40:49.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.693+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c1f5ea0 con 0x7fb62c159b20 2023-10-14T16:40:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.694+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c1f6030 con 0x7fb62c159b20 2023-10-14T16:40:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.695+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb6140073c0 con 0x7fb62c159b20 2023-10-14T16:40:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.696+0000 7fb623fff640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 0x7fb60c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.696+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb614088a20 con 0x7fb62c159b20 2023-10-14T16:40:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.696+0000 7fb62ad9d640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 0x7fb60c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.697+0000 7fb62ad9d640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 0x7fb60c06d330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb61c035000 tx=0x7fb61c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.702+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb62c1f6030 con 0x7fb62c159b20 2023-10-14T16:40:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.943+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb62c1f6030 con 0x7fb62c159b20 2023-10-14T16:40:49.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.943+0000 7fb623fff640 1 -- 172.21.15.29:0/1568988559 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb62c1f6030 con 0x7fb62c159b20 2023-10-14T16:40:49.945 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:40:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 msgr2=0x7fb60c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 0x7fb60c06d330 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fb61c035000 tx=0x7fb61c030000 comp rx=0 tx=0).stop 2023-10-14T16:40:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 msgr2=0x7fb62c1f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:49.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb61400c440 tx=0x7fb614001e70 comp rx=0 tx=0).stop 2023-10-14T16:40:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 shutdown_connections 2023-10-14T16:40:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb60c06ae70 0x7fb60c06d330 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c159f00 0x7fb62c0c7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c159b20 0x7fb62c1f3750 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 --2- 172.21.15.29:0/1568988559 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0b0b20 0x7fb62c1f5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 >> 172.21.15.29:0/1568988559 conn(0x7fb62c0cb410 msgr2=0x7fb62c0cf1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.947+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 shutdown_connections 2023-10-14T16:40:49.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:49.948+0000 7fb63115a640 1 -- 172.21.15.29:0/1568988559 wait complete. 2023-10-14T16:40:49.969 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T16:40:49.969 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:40:49.970 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:40:49.970 INFO:tasks.cephfs.mount:self.client.name = client.2 2023-10-14T16:40:49.970 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2023-10-14T16:40:49.970 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T16:40:49.971 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T16:40:49.971 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:40:49.971 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2023-10-14T16:40:49.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:49.972 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:40:50.023 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:50.023 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:50.024 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:50.024 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.024 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:50.024 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.024 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:50.025 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:50.025 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:40:50.025 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:50.025 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:50.026 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:40:50.026 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:40:50.026 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:50.027 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:40:50.027 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:40:50.027 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42326sec preferred_lft 42326sec 2023-10-14T16:40:50.028 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:40:50.028 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.028 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:50.028 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:50.028 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:40:50.029 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:50.029 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:50.029 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:40:50.029 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.029 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:40:50.030 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.030 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:40:50.030 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:40:50.031 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:40:50.031 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.032 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:40:50.123 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:40:50.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.127 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:40:50.184 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:40:50.184 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:40:50.184 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:40:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:40:50.185 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:40:50.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.185 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:50.186 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:50.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:40:50.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:40:50.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:40:50.187 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:50.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.379 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:40:50.435 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:40:50.436 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.437 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list-id 2023-10-14T16:40:50.493 INFO:teuthology.orchestra.run.smithi029.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2023-10-14T16:40:50.494 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.495 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:50.495 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:50.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:40:50.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2023-10-14T16:40:50.496 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:50.650 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-14T16:40:50.744 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:50.744 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:50.744 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:50.744 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.745 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:50.745 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.745 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:50.745 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 26:7c:37:5c:78:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-14T16:40:50.745 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-14T16:40:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::247c:37ff:fe5c:78ce/64 scope link 2023-10-14T16:40:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.749 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2023-10-14T16:40:50.850 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:50.850 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:50.851 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:50.851 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.851 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:50.851 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.851 INFO:teuthology.orchestra.run.smithi029.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:50.852 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 26:7c:37:5c:78:ce brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-14T16:40:50.852 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-14T16:40:50.852 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.852 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::247c:37ff:fe5c:78ce/64 scope link 2023-10-14T16:40:50.852 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:50.854 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2023-10-14T16:40:50.855 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:50.855 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:50.855 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:50.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2023-10-14T16:40:50.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-14T16:40:50.856 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2023-10-14T16:40:50.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2023-10-14T16:40:50.857 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2023-10-14T16:40:50.857 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:51.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.142 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:40:51.142 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:40:51.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.1 up 2023-10-14T16:40:51.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set dev brx.1 master ceph-brx 2023-10-14T16:40:51.143 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:40:51.329 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:51.330 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2023-10-14T16:40:51.384 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2023-10-14T16:40:51.385 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:51.385 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2023-10-14T16:40:51.437 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:40:51.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.2 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-14T16:40:51.590 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-14T16:40:51.591 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-14T16:40:51.591 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:51.614 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-14T16:40:51.615 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-14T16:40:51.615 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:51.615 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:51.615 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:51.642 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=2,mds_namespace=cephfs 2023-10-14T16:40:51.642 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs". 2023-10-14T16:40:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:40:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:40:51.643 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:40:51.644 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=2,mds_namespace=cephfs,key=2,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:40:51.644 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.2 does not contain SELinux labels. 2023-10-14T16:40:51.644 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:40:51.644 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:40:51.645 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:40:51.645 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:40:51.648 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:40:51.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2023-10-14T16:40:51.738 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.739 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:51.800 INFO:teuthology.orchestra.run.smithi029.stdout:client4549 2023-10-14T16:40:51.800 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:40:51.801 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:40:51.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.857 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:51.917 INFO:teuthology.orchestra.run.smithi029.stdout:client4549 2023-10-14T16:40:51.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.918 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:51.980 INFO:teuthology.orchestra.run.smithi029.stdout:client4549 2023-10-14T16:40:51.980 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:51.981 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.2 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f6aee177-7aff-4117-99f6-f2cafd5adc35.client4549/status') 2023-10-14T16:40:52.080 INFO:teuthology.orchestra.run.smithi029.stdout:instance: client.4549 (3)172.21.15.29:0/4257193973 2023-10-14T16:40:52.081 INFO:teuthology.orchestra.run.smithi029.stdout:blocklisted: false 2023-10-14T16:40:52.081 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records in 2023-10-14T16:40:52.081 INFO:teuthology.orchestra.run.smithi029.stderr:0+1 records out 2023-10-14T16:40:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:70 bytes copied, 9.6212e-05 s, 728 kB/s 2023-10-14T16:40:52.085 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2023-10-14T16:40:52.085 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T16:40:52.085 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-14T16:40:52.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:40:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.308+0000 7f98b71ce640 1 Processor -- start 2023-10-14T16:40:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.308+0000 7f98b71ce640 1 -- start start 2023-10-14T16:40:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b71ce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98b006c3f0 0x7f98b006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b71ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b71ce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b00cd630 con 0x7f98b015e540 2023-10-14T16:40:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b71ce640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b00ce1b0 con 0x7f98b006c3f0 2023-10-14T16:40:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b71ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b006cd10 con 0x7f98b015e0c0 2023-10-14T16:40:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b4f6a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98b006c3f0 0x7f98b006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b4f6a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98b006c3f0 0x7f98b006c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39474/0 (socket says 172.21.15.29:39474) 2023-10-14T16:40:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.310+0000 7f98b4f6a640 1 -- 172.21.15.29:0/378943812 learned_addr learned my addr 172.21.15.29:0/378943812 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:52.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.311+0000 7f98affff640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.311+0000 7f98b4f6a640 1 -- 172.21.15.29:0/378943812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 msgr2=0x7f98b00b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:52.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.311+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.311+0000 7f98b4f6a640 1 -- 172.21.15.29:0/378943812 >> v1:172.21.15.186:6789/0 conn(0x7f98b015e540 legacy=0x7f98b00b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:52.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.311+0000 7f98b4f6a640 1 -- 172.21.15.29:0/378943812 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b00d2170 con 0x7f98b006c3f0 2023-10-14T16:40:52.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.312+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b006c3f0 0x7f98b006c7d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009e50 tx=0x7f98a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79928c48a227ad0d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:52.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.312+0000 7f98af7fe640 1 -- 172.21.15.29:0/378943812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a002ca70 con 0x7f98b006c3f0 2023-10-14T16:40:52.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.312+0000 7f98af7fe640 1 -- 172.21.15.29:0/378943812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a002cbd0 con 0x7f98b006c3f0 2023-10-14T16:40:52.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.312+0000 7f98af7fe640 1 -- 172.21.15.29:0/378943812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a00343f0 con 0x7f98b006c3f0 2023-10-14T16:40:52.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 -- 172.21.15.29:0/378943812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b006c3f0 msgr2=0x7f98b006c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:52.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b006c3f0 0x7f98b006c7d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f98a0009e50 tx=0x7f98a002bed0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 -- 172.21.15.29:0/378943812 shutdown_connections 2023-10-14T16:40:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 --2- 172.21.15.29:0/378943812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b006c3f0 0x7f98b006c7d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 -- 172.21.15.29:0/378943812 >> 172.21.15.29:0/378943812 conn(0x7f98b00d2cb0 msgr2=0x7f98b00d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 -- 172.21.15.29:0/378943812 shutdown_connections 2023-10-14T16:40:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.313+0000 7f98b71ce640 1 -- 172.21.15.29:0/378943812 wait complete. 2023-10-14T16:40:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.315+0000 7f98b71ce640 1 Processor -- start 2023-10-14T16:40:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.315+0000 7f98b71ce640 1 -- start start 2023-10-14T16:40:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 0x7f98b01f3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b00b6df0 con 0x7f98b015e540 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98b01f4600 con 0x7f98b006c3f0 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b71ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98b01f48e0 con 0x7f98b015e0c0 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b576b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b576b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39478/0 (socket says 172.21.15.29:39478) 2023-10-14T16:40:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.316+0000 7f98b576b640 1 -- 172.21.15.29:0/1345274308 learned_addr learned my addr 172.21.15.29:0/1345274308 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 0x7f98b01f3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98affff640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 -- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 msgr2=0x7f98b00b68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 -- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 msgr2=0x7f98b01f3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 0x7f98b01f3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 -- 172.21.15.29:0/1345274308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98a0009b30 con 0x7f98b015e540 2023-10-14T16:40:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 0x7f98b01f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b576b640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98a800c410 tx=0x7f98a800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8011790 con 0x7f98b015e540 2023-10-14T16:40:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98a80118f0 con 0x7f98b015e540 2023-10-14T16:40:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98a8011ce0 con 0x7f98b015e540 2023-10-14T16:40:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.317+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98b01f4bc0 con 0x7f98b015e540 2023-10-14T16:40:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.318+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98b01f5180 con 0x7f98b015e540 2023-10-14T16:40:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.319+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98b01f53a0 con 0x7f98b015e540 2023-10-14T16:40:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.320+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f98a801c460 con 0x7f98b015e540 2023-10-14T16:40:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.321+0000 7f98ad7fa640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 0x7f989c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.321+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f98a80894c0 con 0x7f98b015e540 2023-10-14T16:40:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.321+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 0x7f989c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.322+0000 7f98b4f6a640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 0x7f989c06d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98a0035000 tx=0x7f98a0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.326+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98a8054630 con 0x7f98b015e540 2023-10-14T16:40:52.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.563+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f98b004e160 con 0x7f98b015e540 2023-10-14T16:40:52.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.564+0000 7f98ad7fa640 1 -- 172.21.15.29:0/1345274308 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f98a8057c50 con 0x7f98b015e540 2023-10-14T16:40:52.565 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:40:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 msgr2=0x7f989c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 0x7f989c06d360 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f98a0035000 tx=0x7f98a0030000 comp rx=0 tx=0).stop 2023-10-14T16:40:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 msgr2=0x7f98b015c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f98a800c410 tx=0x7f98a800c810 comp rx=0 tx=0).stop 2023-10-14T16:40:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 shutdown_connections 2023-10-14T16:40:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f989c06aea0 0x7f989c06d360 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98b015e540 0x7f98b015c710 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98b015e0c0 0x7f98b00b68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 --2- 172.21.15.29:0/1345274308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98b006c3f0 0x7f98b01f3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:52.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 >> 172.21.15.29:0/1345274308 conn(0x7f98b00d2cb0 msgr2=0x7f98b00d3090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:52.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 shutdown_connections 2023-10-14T16:40:52.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:52.567+0000 7f98b71ce640 1 -- 172.21.15.29:0/1345274308 wait complete. 2023-10-14T16:40:52.587 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T16:40:52.587 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:40:52.587 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2023-10-14T16:40:52.588 INFO:tasks.cephfs.mount:self.client.name = client.1 2023-10-14T16:40:52.588 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2023-10-14T16:40:52.588 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T16:40:52.588 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T16:40:52.589 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:40:52.589 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2023-10-14T16:40:52.589 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:52.590 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2023-10-14T16:40:52.649 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:52.650 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:52.650 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:52.650 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:52.651 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:52.651 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:52.651 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:52.651 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:52.651 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2023-10-14T16:40:52.652 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:52.652 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:52.652 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2023-10-14T16:40:52.652 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:52.652 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:52.653 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:40:52.653 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:52.653 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-14T16:40:52.653 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42295sec preferred_lft 42295sec 2023-10-14T16:40:52.653 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2023-10-14T16:40:52.654 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:52.655 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2023-10-14T16:40:52.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:52.656 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:52.656 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:52.656 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add name ceph-brx type bridge 2023-10-14T16:40:52.657 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr flush dev ceph-brx 2023-10-14T16:40:52.657 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set ceph-brx up 2023-10-14T16:40:52.657 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2023-10-14T16:40:52.658 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:53.114 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:53.114 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:40:53.192 INFO:teuthology.orchestra.run.smithi186.stdout:1 2023-10-14T16:40:53.196 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:53.197 DEBUG:teuthology.orchestra.run.smithi186:> route 2023-10-14T16:40:53.251 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2023-10-14T16:40:53.251 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:40:53.251 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-14T16:40:53.252 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-14T16:40:53.252 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:40:53.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:53.253 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:53.254 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:53.254 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-14T16:40:53.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-14T16:40:53.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-14T16:40:53.256 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:54.028 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.029 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T16:40:54.084 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.084 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2023-10-14T16:40:54.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.140 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:54.140 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:54.140 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-14T16:40:54.141 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2023-10-14T16:40:54.141 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:54.290 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2023-10-14T16:40:54.291 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.291 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:54.291 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:54.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2023-10-14T16:40:54.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2023-10-14T16:40:54.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2023-10-14T16:40:54.293 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2023-10-14T16:40:54.293 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2023-10-14T16:40:54.293 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:54.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.577 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:54.578 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:54.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.0 up 2023-10-14T16:40:54.578 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.0 master ceph-brx 2023-10-14T16:40:54.579 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:54.738 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:54.739 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2023-10-14T16:40:54.787 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2023-10-14T16:40:54.788 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:54.788 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2023-10-14T16:40:54.836 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:40:54.837 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:54.837 DEBUG:teuthology.orchestra.run.smithi186:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-14T16:40:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-14T16:40:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-14T16:40:54.975 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:55.487 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-14T16:40:55.487 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-14T16:40:55.487 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:55.488 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:55.488 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:55.520 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2023-10-14T16:40:55.521 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2023-10-14T16:40:55.521 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:55.521 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:55.521 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:55.522 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:40:55.522 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:40:55.522 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:40:55.529 INFO:teuthology.orchestra.run.smithi186.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2023-10-14T16:40:55.529 INFO:teuthology.orchestra.run.smithi186.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:40:55.529 INFO:teuthology.orchestra.run.smithi186.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:40:55.529 INFO:teuthology.orchestra.run.smithi186.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:40:55.529 INFO:teuthology.orchestra.run.smithi186.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:40:55.534 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:40:55.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:55.537 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2023-10-14T16:40:55.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:55.632 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:55.691 INFO:teuthology.orchestra.run.smithi186.stdout:client4552 2023-10-14T16:40:55.692 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:55.692 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:40:55.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:55.744 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:55.802 INFO:teuthology.orchestra.run.smithi186.stdout:client4552 2023-10-14T16:40:55.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:55.803 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:55.861 INFO:teuthology.orchestra.run.smithi186.stdout:client4552 2023-10-14T16:40:55.862 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:55.862 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f6aee177-7aff-4117-99f6-f2cafd5adc35.client4552/status') 2023-10-14T16:40:55.955 INFO:teuthology.orchestra.run.smithi186.stdout:instance: client.4552 (3)192.168.0.1:0/2532179754 2023-10-14T16:40:55.956 INFO:teuthology.orchestra.run.smithi186.stdout:blocklisted: false 2023-10-14T16:40:55.956 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2023-10-14T16:40:55.956 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2023-10-14T16:40:55.957 INFO:teuthology.orchestra.run.smithi186.stderr:69 bytes copied, 9.4991e-05 s, 726 kB/s 2023-10-14T16:40:55.959 INFO:tasks.kclient:client.3 config is {'syntax': 'v1'} 2023-10-14T16:40:55.959 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2023-10-14T16:40:55.960 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-14T16:40:55.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:40:56.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.203+0000 7f89659ef640 1 Processor -- start 2023-10-14T16:40:56.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.203+0000 7f89659ef640 1 -- start start 2023-10-14T16:40:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89659ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f896015aad0 0x7f89600cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89659ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89600712c0 0x7f8960070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89659ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600b4620 con 0x7f8960070d70 2023-10-14T16:40:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89659ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89600b4340 con 0x7f896015aad0 2023-10-14T16:40:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89659ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896006f520 con 0x7f89600712c0 2023-10-14T16:40:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f89649ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f896015aad0 0x7f89600cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f8957fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8960070d70 0x7f896006bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46018/0 (socket says 172.21.15.29:46018) 2023-10-14T16:40:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.205+0000 7f8957fff640 1 -- 172.21.15.29:0/3859728913 learned_addr learned my addr 172.21.15.29:0/3859728913 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89651ee640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89600712c0 0x7f8960070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89577fe640 1 -- 172.21.15.29:0/3859728913 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696278246 0 0) 0x7f89600b4620 con 0x7f8960070d70 2023-10-14T16:40:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89577fe640 1 -- 172.21.15.29:0/3859728913 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8948003520 con 0x7f8960070d70 2023-10-14T16:40:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89649ed640 1 -- 172.21.15.29:0/3859728913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89600712c0 msgr2=0x7f8960070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89649ed640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89600712c0 0x7f8960070000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89649ed640 1 -- 172.21.15.29:0/3859728913 >> v1:172.21.15.29:6789/0 conn(0x7f8960070d70 legacy=0x7f896006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.206+0000 7f89649ed640 1 -- 172.21.15.29:0/3859728913 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f896006f800 con 0x7f896015aad0 2023-10-14T16:40:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.207+0000 7f89649ed640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f896015aad0 0x7f89600cc270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f895c0097a0 tx=0x7f895c02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=206d795164ba28f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.207+0000 7f89577fe640 1 -- 172.21.15.29:0/3859728913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c030800 con 0x7f896015aad0 2023-10-14T16:40:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.207+0000 7f89577fe640 1 -- 172.21.15.29:0/3859728913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f895c030960 con 0x7f896015aad0 2023-10-14T16:40:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.207+0000 7f89577fe640 1 -- 172.21.15.29:0/3859728913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f895c030cf0 con 0x7f896015aad0 2023-10-14T16:40:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 -- 172.21.15.29:0/3859728913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f896015aad0 msgr2=0x7f89600cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f896015aad0 0x7f89600cc270 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f895c0097a0 tx=0x7f895c02b940 comp rx=0 tx=0).stop 2023-10-14T16:40:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 -- 172.21.15.29:0/3859728913 shutdown_connections 2023-10-14T16:40:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89600712c0 0x7f8960070000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 --2- 172.21.15.29:0/3859728913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f896015aad0 0x7f89600cc270 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 -- 172.21.15.29:0/3859728913 >> 172.21.15.29:0/3859728913 conn(0x7f89600b0a90 msgr2=0x7f89600ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:56.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 -- 172.21.15.29:0/3859728913 shutdown_connections 2023-10-14T16:40:56.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.208+0000 7f89659ef640 1 -- 172.21.15.29:0/3859728913 wait complete. 2023-10-14T16:40:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.210+0000 7f89659ef640 1 Processor -- start 2023-10-14T16:40:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.210+0000 7f89659ef640 1 -- start start 2023-10-14T16:40:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 0x7f8960151d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896015aad0 0x7f89600b2920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896015c8a0 con 0x7f89600712c0 2023-10-14T16:40:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f896015cb80 con 0x7f8960070d70 2023-10-14T16:40:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f89659ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f896015ce60 con 0x7f896015aad0 2023-10-14T16:40:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f8957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f8957fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39506/0 (socket says 172.21.15.29:39506) 2023-10-14T16:40:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.211+0000 7f8957fff640 1 -- 172.21.15.29:0/2307961667 learned_addr learned my addr 172.21.15.29:0/2307961667 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f89651ee640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896015aad0 0x7f89600b2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f89649ed640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 0x7f8960151d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 -- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896015aad0 msgr2=0x7f89600b2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896015aad0 0x7f89600b2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 -- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 msgr2=0x7f8960151d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 0x7f8960151d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 -- 172.21.15.29:0/2307961667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f895c009400 con 0x7f89600712c0 2023-10-14T16:40:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f89649ed640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 0x7f8960151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f8957fff640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f895000c500 tx=0x7f895000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.212+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950005ce0 con 0x7f89600712c0 2023-10-14T16:40:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.213+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8950005e40 con 0x7f89600712c0 2023-10-14T16:40:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.213+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8950007330 con 0x7f89600712c0 2023-10-14T16:40:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.213+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f896015d140 con 0x7f89600712c0 2023-10-14T16:40:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.213+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8960152240 con 0x7f89600712c0 2023-10-14T16:40:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.214+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896015d660 con 0x7f89600712c0 2023-10-14T16:40:56.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.215+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8950007550 con 0x7f89600712c0 2023-10-14T16:40:56.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.216+0000 7f89557fa640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 0x7f894c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.216+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8950088cc0 con 0x7f89600712c0 2023-10-14T16:40:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.217+0000 7f89649ed640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 0x7f894c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.217+0000 7f89649ed640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 0x7f894c06d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f895c036000 tx=0x7f895c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:56.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.221+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f896015d660 con 0x7f89600712c0 2023-10-14T16:40:56.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.454+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f896015d660 con 0x7f89600712c0 2023-10-14T16:40:56.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.455+0000 7f89557fa640 1 -- 172.21.15.29:0/2307961667 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v8) v1 ==== 53+0+73 (secure 0 0 0) 0x7f896015d660 con 0x7f89600712c0 2023-10-14T16:40:56.456 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:40:56.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 msgr2=0x7f894c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 0x7f894c06d330 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f895c036000 tx=0x7f895c03b040 comp rx=0 tx=0).stop 2023-10-14T16:40:56.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 msgr2=0x7f89600b23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:56.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f895000c500 tx=0x7f895000c9c0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 shutdown_connections 2023-10-14T16:40:56.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f894c06ae70 0x7f894c06d330 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f896015aad0 0x7f89600b2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89600712c0 0x7f89600b23e0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.457+0000 7f89659ef640 1 --2- 172.21.15.29:0/2307961667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8960070d70 0x7f8960151d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.458+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 >> 172.21.15.29:0/2307961667 conn(0x7f89600b0a90 msgr2=0x7f896006bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:56.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.458+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 shutdown_connections 2023-10-14T16:40:56.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:56.458+0000 7f89659ef640 1 -- 172.21.15.29:0/2307961667 wait complete. 2023-10-14T16:40:56.478 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2023-10-14T16:40:56.478 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:40:56.478 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi186.front.sepia.ceph.com 2023-10-14T16:40:56.479 INFO:tasks.cephfs.mount:self.client.name = client.3 2023-10-14T16:40:56.479 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2023-10-14T16:40:56.479 INFO:tasks.cephfs.mount:self.cephfs_name = None 2023-10-14T16:40:56.479 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2023-10-14T16:40:56.480 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:40:56.480 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2023-10-14T16:40:56.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.481 DEBUG:teuthology.orchestra.run.smithi186:> ip addr 2023-10-14T16:40:56.539 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:56.540 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:56.540 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:56.540 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.540 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:56.541 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.541 INFO:teuthology.orchestra.run.smithi186.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:56.541 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2c brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:56.541 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f0 2023-10-14T16:40:56.541 INFO:teuthology.orchestra.run.smithi186.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:56.542 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:d9:38:2d brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:56.542 INFO:teuthology.orchestra.run.smithi186.stdout: altname enp7s0f1 2023-10-14T16:40:56.542 INFO:teuthology.orchestra.run.smithi186.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:40:56.542 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f4 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:56.542 INFO:teuthology.orchestra.run.smithi186.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:40:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 0c:c4:7a:88:6d:f5 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: inet 172.21.15.186/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2023-10-14T16:40:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft 42291sec preferred_lft 42291sec 2023-10-14T16:40:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::ec4:7aff:fe88:6df5/64 scope link 2023-10-14T16:40:56.543 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.544 INFO:teuthology.orchestra.run.smithi186.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:56.544 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 52:96:2b:98:2d:0b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:40:56.544 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:40:56.544 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.544 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::a093:49ff:fe76:a2d7/64 scope link 2023-10-14T16:40:56.545 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.545 INFO:teuthology.orchestra.run.smithi186.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:40:56.545 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 52:96:2b:98:2d:0b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-14T16:40:56.545 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::5096:2bff:fe98:2d0b/64 scope link tentative 2023-10-14T16:40:56.545 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:56.546 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.547 DEBUG:teuthology.orchestra.run.smithi186:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:40:56.642 INFO:teuthology.orchestra.run.smithi186.stdout:1 2023-10-14T16:40:56.648 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.648 DEBUG:teuthology.orchestra.run.smithi186:> route 2023-10-14T16:40:56.703 INFO:teuthology.orchestra.run.smithi186.stdout:Kernel IP routing table 2023-10-14T16:40:56.704 INFO:teuthology.orchestra.run.smithi186.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:40:56.704 INFO:teuthology.orchestra.run.smithi186.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2023-10-14T16:40:56.704 INFO:teuthology.orchestra.run.smithi186.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2023-10-14T16:40:56.704 INFO:teuthology.orchestra.run.smithi186.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:40:56.705 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.705 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:56.705 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:56.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2023-10-14T16:40:56.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2023-10-14T16:40:56.706 DEBUG:teuthology.orchestra.run.smithi186:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2023-10-14T16:40:56.707 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:56.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.895 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T16:40:56.944 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-14T16:40:56.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.945 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list-id 2023-10-14T16:40:56.991 INFO:teuthology.orchestra.run.smithi186.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2023-10-14T16:40:56.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:56.992 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:56.993 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:56.993 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-14T16:40:56.993 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2023-10-14T16:40:56.993 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:57.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:57.182 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-14T16:40:57.275 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:57.276 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 3e:55:03:72:98:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::3c55:3ff:fe72:989e/64 scope link 2023-10-14T16:40:57.277 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:57.280 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2023-10-14T16:40:57.374 INFO:teuthology.orchestra.run.smithi186.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:40:57.374 INFO:teuthology.orchestra.run.smithi186.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:40:57.375 INFO:teuthology.orchestra.run.smithi186.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:40:57.375 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.375 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 ::1/128 scope host 2023-10-14T16:40:57.375 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.375 INFO:teuthology.orchestra.run.smithi186.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:40:57.376 INFO:teuthology.orchestra.run.smithi186.stdout: link/ether 3e:55:03:72:98:9e brd ff:ff:ff:ff:ff:ff link-netnsid 0 2023-10-14T16:40:57.376 INFO:teuthology.orchestra.run.smithi186.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2023-10-14T16:40:57.376 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.376 INFO:teuthology.orchestra.run.smithi186.stdout: inet6 fe80::3c55:3ff:fe72:989e/64 scope link 2023-10-14T16:40:57.376 INFO:teuthology.orchestra.run.smithi186.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:40:57.378 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2023-10-14T16:40:57.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:57.379 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:57.379 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:57.379 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2023-10-14T16:40:57.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2023-10-14T16:40:57.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2023-10-14T16:40:57.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2023-10-14T16:40:57.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2023-10-14T16:40:57.381 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:57.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:57.634 DEBUG:teuthology.orchestra.run.smithi186:> (cd / && exec bash -c ' 2023-10-14T16:40:57.634 DEBUG:teuthology.orchestra.run.smithi186:> set -e 2023-10-14T16:40:57.634 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set brx.1 up 2023-10-14T16:40:57.635 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link set dev brx.1 master ceph-brx 2023-10-14T16:40:57.635 DEBUG:teuthology.orchestra.run.smithi186:> ') 2023-10-14T16:40:57.804 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:57.805 DEBUG:teuthology.orchestra.run.smithi186:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2023-10-14T16:40:57.852 INFO:teuthology.orchestra.run.smithi186.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2023-10-14T16:40:57.852 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:40:57.853 DEBUG:teuthology.orchestra.run.smithi186:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2023-10-14T16:40:57.952 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:40:57.953 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:57.953 DEBUG:teuthology.orchestra.run.smithi186:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.3 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-14T16:40:58.095 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-14T16:40:58.096 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-14T16:40:58.096 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:58.117 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-14T16:40:58.117 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-14T16:40:58.117 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:58.117 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:58.118 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:58.146 INFO:teuthology.orchestra.run.smithi186.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=3,mds_namespace=cephfs 2023-10-14T16:40:58.146 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs". 2023-10-14T16:40:58.146 INFO:teuthology.orchestra.run.smithi186.stdout:invalid new device string format 2023-10-14T16:40:58.147 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:40:58.147 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:40:58.147 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:40:58.147 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:40:58.147 INFO:teuthology.orchestra.run.smithi186.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=3,mds_namespace=cephfs,key=3,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:40:58.148 INFO:teuthology.orchestra.run.smithi186.stdout:mount: /home/ubuntu/cephtest/mnt.3 does not contain SELinux labels. 2023-10-14T16:40:58.149 INFO:teuthology.orchestra.run.smithi186.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:40:58.149 INFO:teuthology.orchestra.run.smithi186.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:40:58.149 INFO:teuthology.orchestra.run.smithi186.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:40:58.149 INFO:teuthology.orchestra.run.smithi186.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:40:58.153 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:40:58.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:58.154 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2023-10-14T16:40:58.245 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:58.245 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:58.301 INFO:teuthology.orchestra.run.smithi186.stdout:client4698 2023-10-14T16:40:58.302 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:40:58.302 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:40:58.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:58.354 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:58.414 INFO:teuthology.orchestra.run.smithi186.stdout:client4698 2023-10-14T16:40:58.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:58.415 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:40:58.473 INFO:teuthology.orchestra.run.smithi186.stdout:client4698 2023-10-14T16:40:58.474 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:40:58.474 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest/mnt.3 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/f6aee177-7aff-4117-99f6-f2cafd5adc35.client4698/status') 2023-10-14T16:40:58.563 INFO:teuthology.orchestra.run.smithi186.stdout:instance: client.4698 (3)172.21.15.186:0/517577938 2023-10-14T16:40:58.564 INFO:teuthology.orchestra.run.smithi186.stdout:blocklisted: false 2023-10-14T16:40:58.564 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records in 2023-10-14T16:40:58.564 INFO:teuthology.orchestra.run.smithi186.stderr:0+1 records out 2023-10-14T16:40:58.565 INFO:teuthology.orchestra.run.smithi186.stderr:70 bytes copied, 6.5726e-05 s, 1.1 MB/s 2023-10-14T16:40:58.567 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2023-10-14T16:40:58.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:40:58.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.823+0000 7f58eb59e640 1 Processor -- start 2023-10-14T16:40:58.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.823+0000 7f58eb59e640 1 -- start start 2023-10-14T16:40:58.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58eb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58ec15b320 0x7f58ec0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:58.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58eb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec15b9d0 con 0x7f58ec06c040 2023-10-14T16:40:58.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58eb59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58ec15bcb0 con 0x7f58ec0ad970 2023-10-14T16:40:58.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58eb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58ec15b7d0 con 0x7f58ec15b320 2023-10-14T16:40:58.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58ea59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f58ec06c040 0x7f58ec06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46022/0 (socket says 172.21.15.29:46022) 2023-10-14T16:40:58.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58ea59c640 1 -- 172.21.15.29:0/4272392250 learned_addr learned my addr 172.21.15.29:0/4272392250 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:40:58.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.825+0000 7f58ead9d640 1 --2- 172.21.15.29:0/4272392250 >> v2:172.21.15.29:3300/0 conn(0x7f58ec15b320 0x7f58ec0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.826+0000 7f58e9d9b640 1 -- 172.21.15.29:0/4272392250 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133641811 0 0) 0x7f58ec15b9d0 con 0x7f58ec06c040 2023-10-14T16:40:58.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.826+0000 7f58e9d9b640 1 -- 172.21.15.29:0/4272392250 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d8003520 con 0x7f58ec06c040 2023-10-14T16:40:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.826+0000 7f58ead9d640 1 -- 172.21.15.29:0/4272392250 >> v1:172.21.15.186:6789/0 conn(0x7f58ec0ad970 legacy=0x7f58ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.826+0000 7f58ead9d640 1 -- 172.21.15.29:0/4272392250 >> v1:172.21.15.29:6789/0 conn(0x7f58ec06c040 legacy=0x7f58ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.826+0000 7f58ead9d640 1 -- 172.21.15.29:0/4272392250 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec15bf90 con 0x7f58ec15b320 2023-10-14T16:40:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.827+0000 7f58ead9d640 1 --2- 172.21.15.29:0/4272392250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec15b320 0x7f58ec0cd180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009dc0 tx=0x7f58e402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0731323c4116488 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:58.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.827+0000 7f58e9d9b640 1 -- 172.21.15.29:0/4272392250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e402cc40 con 0x7f58ec15b320 2023-10-14T16:40:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.827+0000 7f58e9d9b640 1 -- 172.21.15.29:0/4272392250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58e402cda0 con 0x7f58ec15b320 2023-10-14T16:40:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.827+0000 7f58e9d9b640 1 -- 172.21.15.29:0/4272392250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58e4034660 con 0x7f58ec15b320 2023-10-14T16:40:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 -- 172.21.15.29:0/4272392250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec15b320 msgr2=0x7f58ec0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:58.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 --2- 172.21.15.29:0/4272392250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec15b320 0x7f58ec0cd180 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f58e4009dc0 tx=0x7f58e402be90 comp rx=0 tx=0).stop 2023-10-14T16:40:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 -- 172.21.15.29:0/4272392250 shutdown_connections 2023-10-14T16:40:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 --2- 172.21.15.29:0/4272392250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec15b320 0x7f58ec0cd180 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 -- 172.21.15.29:0/4272392250 >> 172.21.15.29:0/4272392250 conn(0x7f58ec0ce6a0 msgr2=0x7f58ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 -- 172.21.15.29:0/4272392250 shutdown_connections 2023-10-14T16:40:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.828+0000 7f58eb59e640 1 -- 172.21.15.29:0/4272392250 wait complete. 2023-10-14T16:40:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.830+0000 7f58eb59e640 1 Processor -- start 2023-10-14T16:40:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.830+0000 7f58eb59e640 1 -- start start 2023-10-14T16:40:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58ec06c040 0x7f58ec0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58ec15b320 0x7f58ec0b0560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec1f2860 con 0x7f58ec0ad970 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec155c00 con 0x7f58ec06c040 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.831+0000 7f58eb59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58ec1524b0 con 0x7f58ec15b320 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39520/0 (socket says 172.21.15.29:39520) 2023-10-14T16:40:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 -- 172.21.15.29:0/1202527224 learned_addr learned my addr 172.21.15.29:0/1202527224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 -- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58ec15b320 msgr2=0x7f58ec0b0560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58ec15b320 0x7f58ec0b0560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 -- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58ec06c040 msgr2=0x7f58ec0c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:58.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58ec06c040 0x7f58ec0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 -- 172.21.15.29:0/1202527224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4009a70 con 0x7f58ec0ad970 2023-10-14T16:40:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.832+0000 7f58e3fff640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c400 tx=0x7f58d400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.833+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4011790 con 0x7f58ec0ad970 2023-10-14T16:40:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.833+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58d4011930 con 0x7f58ec0ad970 2023-10-14T16:40:58.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.833+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58ec0c7970 con 0x7f58ec0ad970 2023-10-14T16:40:58.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.833+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58d4011ce0 con 0x7f58ec0ad970 2023-10-14T16:40:58.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.833+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58ec1f2d20 con 0x7f58ec0ad970 2023-10-14T16:40:58.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.834+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58ec06b330 con 0x7f58ec0ad970 2023-10-14T16:40:58.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.835+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f58d401c460 con 0x7f58ec0ad970 2023-10-14T16:40:58.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.836+0000 7f58e2ffd640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 0x7f58cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:58.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.836+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f58d4089650 con 0x7f58ec0ad970 2023-10-14T16:40:58.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.836+0000 7f58ea59c640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 0x7f58cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:58.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.837+0000 7f58ea59c640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 0x7f58cc06d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58dc001570 tx=0x7f58dc007930 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:58.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:58.840+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58d4054660 con 0x7f58ec0ad970 2023-10-14T16:40:59.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.038+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58ec04e9b0 con 0x7f58ec0ad970 2023-10-14T16:40:59.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.041+0000 7f58e2ffd640 1 -- 172.21.15.29:0/1202527224 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f58d4057c80 con 0x7f58ec0ad970 2023-10-14T16:40:59.042 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:40:59.043 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:40:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 msgr2=0x7f58cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 0x7f58cc06d330 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f58dc001570 tx=0x7f58dc007930 comp rx=0 tx=0).stop 2023-10-14T16:40:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 msgr2=0x7f58ec0c72a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f58d400c400 tx=0x7f58d400c8c0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 shutdown_connections 2023-10-14T16:40:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f58cc06ae70 0x7f58cc06d330 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58ec15b320 0x7f58ec0b0560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58ec0ad970 0x7f58ec0c72a0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 --2- 172.21.15.29:0/1202527224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58ec06c040 0x7f58ec0c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.044+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 >> 172.21.15.29:0/1202527224 conn(0x7f58ec0ce6a0 msgr2=0x7f58ec159da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.045+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 shutdown_connections 2023-10-14T16:40:59.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.045+0000 7f58eb59e640 1 -- 172.21.15.29:0/1202527224 wait complete. 2023-10-14T16:40:59.065 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:40:59.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:40:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.299+0000 7f946ab21640 1 Processor -- start 2023-10-14T16:40:59.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.299+0000 7f946ab21640 1 -- start start 2023-10-14T16:40:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9464159ad0 0x7f9464070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946415d650 0x7f946406aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f946406b3e0 0x7f94640c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9464071470 con 0x7f946406b3e0 2023-10-14T16:40:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94640cd260 con 0x7f9464159ad0 2023-10-14T16:40:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946ab21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f946415de30 con 0x7f946415d650 2023-10-14T16:40:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946a320640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f946406b3e0 0x7f94640c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946a320640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f946406b3e0 0x7f94640c9de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39524/0 (socket says 172.21.15.29:39524) 2023-10-14T16:40:59.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.301+0000 7f946a320640 1 -- 172.21.15.29:0/1435185986 learned_addr learned my addr 172.21.15.29:0/1435185986 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f9469b1f640 1 --2- 172.21.15.29:0/1435185986 >> v2:172.21.15.186:3300/0 conn(0x7f9464159ad0 0x7f9464070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946931e640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946415d650 0x7f946406aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946a320640 1 -- 172.21.15.29:0/1435185986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946415d650 msgr2=0x7f946406aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946a320640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946415d650 0x7f946406aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946a320640 1 -- 172.21.15.29:0/1435185986 >> v2:172.21.15.186:3300/0 conn(0x7f9464159ad0 msgr2=0x7f9464070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:59.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946a320640 1 --2- 172.21.15.29:0/1435185986 >> v2:172.21.15.186:3300/0 conn(0x7f9464159ad0 0x7f9464070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.302+0000 7f946a320640 1 -- 172.21.15.29:0/1435185986 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946415e110 con 0x7f946406b3e0 2023-10-14T16:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.303+0000 7f946a320640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946406b3e0 0x7f94640c9de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f946406d010 tx=0x7f945c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a5ee8a7307e5a37 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.303+0000 7f9468b1d640 1 -- 172.21.15.29:0/1435185986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c02ca70 con 0x7f946406b3e0 2023-10-14T16:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.303+0000 7f9468b1d640 1 -- 172.21.15.29:0/1435185986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f945c02cbd0 con 0x7f946406b3e0 2023-10-14T16:40:59.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.303+0000 7f9468b1d640 1 -- 172.21.15.29:0/1435185986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f945c0343f0 con 0x7f946406b3e0 2023-10-14T16:40:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 -- 172.21.15.29:0/1435185986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946406b3e0 msgr2=0x7f94640c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946406b3e0 0x7f94640c9de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f946406d010 tx=0x7f945c02bed0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 -- 172.21.15.29:0/1435185986 shutdown_connections 2023-10-14T16:40:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f946406b3e0 0x7f94640c9de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 --2- 172.21.15.29:0/1435185986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946415d650 0x7f946406aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 --2- 172.21.15.29:0/1435185986 >> v2:172.21.15.186:3300/0 conn(0x7f9464159ad0 0x7f9464070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 -- 172.21.15.29:0/1435185986 >> 172.21.15.29:0/1435185986 conn(0x7f94640b0170 msgr2=0x7f94640b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 -- 172.21.15.29:0/1435185986 shutdown_connections 2023-10-14T16:40:59.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.304+0000 7f946ab21640 1 -- 172.21.15.29:0/1435185986 wait complete. 2023-10-14T16:40:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.306+0000 7f946ab21640 1 Processor -- start 2023-10-14T16:40:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.306+0000 7f946ab21640 1 -- start start 2023-10-14T16:40:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946406b3e0 0x7f9464146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946415d650 0x7f946414cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f946414cb30 con 0x7f9464159ad0 2023-10-14T16:40:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9464166140 con 0x7f946415d650 2023-10-14T16:40:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946ab21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94640c42b0 con 0x7f946406b3e0 2023-10-14T16:40:59.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946931e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946931e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39528/0 (socket says 172.21.15.29:39528) 2023-10-14T16:40:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.307+0000 7f946931e640 1 -- 172.21.15.29:0/2586664478 learned_addr learned my addr 172.21.15.29:0/2586664478 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946a320640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946415d650 0x7f946414cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 -- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946406b3e0 msgr2=0x7f9464146cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946406b3e0 0x7f9464146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 -- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946415d650 msgr2=0x7f946414cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946415d650 0x7f946414cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 -- 172.21.15.29:0/2586664478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f945c009b30 con 0x7f9464159ad0 2023-10-14T16:40:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946931e640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f946000d5f0 tx=0x7f946000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94600125f0 con 0x7f9464159ad0 2023-10-14T16:40:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9460012750 con 0x7f9464159ad0 2023-10-14T16:40:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94640c4590 con 0x7f9464159ad0 2023-10-14T16:40:59.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.308+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460012b00 con 0x7f9464159ad0 2023-10-14T16:40:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.309+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9464159220 con 0x7f9464159ad0 2023-10-14T16:40:59.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.310+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f946406dae0 con 0x7f9464159ad0 2023-10-14T16:40:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.311+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9460012c60 con 0x7f9464159ad0 2023-10-14T16:40:59.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.312+0000 7f94567fc640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 0x7f944806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.312+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9460089b50 con 0x7f9464159ad0 2023-10-14T16:40:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.312+0000 7f9469b1f640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 0x7f944806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.313+0000 7f9469b1f640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 0x7f944806d330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9458009870 tx=0x7f9458008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.316+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9460054b60 con 0x7f9464159ad0 2023-10-14T16:40:59.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.522+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f946414cc70 con 0x7f9464159ad0 2023-10-14T16:40:59.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.523+0000 7f94567fc640 1 -- 172.21.15.29:0/2586664478 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f946414cc70 con 0x7f9464159ad0 2023-10-14T16:40:59.524 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:40:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 msgr2=0x7f944806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 0x7f944806d330 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f9458009870 tx=0x7f9458008040 comp rx=0 tx=0).stop 2023-10-14T16:40:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 msgr2=0x7f9464166530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f946000d5f0 tx=0x7f946000dab0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 shutdown_connections 2023-10-14T16:40:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f944806ae70 0x7f944806d330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f946415d650 0x7f946414cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9464159ad0 0x7f9464166530 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 --2- 172.21.15.29:0/2586664478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f946406b3e0 0x7f9464146cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 >> 172.21.15.29:0/2586664478 conn(0x7f94640b0170 msgr2=0x7f9464150860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 shutdown_connections 2023-10-14T16:40:59.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.526+0000 7f946ab21640 1 -- 172.21.15.29:0/2586664478 wait complete. 2023-10-14T16:40:59.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:40:59.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.777+0000 7ff438ed1640 1 Processor -- start 2023-10-14T16:40:59.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.777+0000 7ff438ed1640 1 -- start start 2023-10-14T16:40:59.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.779+0000 7ff438ed1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 0x7ff4340c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.779+0000 7ff438ed1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff43415dde0 0x7ff4340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.779+0000 7ff438ed1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff434160a90 con 0x7ff4340c9230 2023-10-14T16:40:59.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.779+0000 7ff438ed1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4340cdec0 con 0x7ff43415dde0 2023-10-14T16:40:59.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.779+0000 7ff438ed1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4340cd900 con 0x7ff43415c9f0 2023-10-14T16:40:59.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.780+0000 7ff433fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff43415dde0 0x7ff4340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.780+0000 7ff4337fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 0x7ff4340c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.780+0000 7ff433fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff43415dde0 0x7ff4340ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:48638/0 (socket says 172.21.15.29:48638) 2023-10-14T16:40:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.780+0000 7ff433fff640 1 -- 172.21.15.29:0/1570603510 learned_addr learned my addr 172.21.15.29:0/1570603510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:59.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.781+0000 7ff433fff640 1 -- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 msgr2=0x7ff4340c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:40:59.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.781+0000 7ff433fff640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 0x7ff4340c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.781+0000 7ff433fff640 1 -- 172.21.15.29:0/1570603510 >> v1:172.21.15.186:6789/0 conn(0x7ff4340c9230 legacy=0x7ff4340709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.781+0000 7ff433fff640 1 -- 172.21.15.29:0/1570603510 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4340cdbe0 con 0x7ff43415dde0 2023-10-14T16:40:59.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.781+0000 7ff4337fe640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 0x7ff4340c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:40:59.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.782+0000 7ff433fff640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415dde0 0x7ff4340ac890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a58c9fa7505b5246 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.783+0000 7ff432ffd640 1 -- 172.21.15.29:0/1570603510 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c02cb00 con 0x7ff43415dde0 2023-10-14T16:40:59.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.783+0000 7ff432ffd640 1 -- 172.21.15.29:0/1570603510 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c02cc60 con 0x7ff43415dde0 2023-10-14T16:40:59.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.783+0000 7ff432ffd640 1 -- 172.21.15.29:0/1570603510 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c0343f0 con 0x7ff43415dde0 2023-10-14T16:40:59.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.783+0000 7ff438ed1640 1 -- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415dde0 msgr2=0x7ff4340ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.783+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415dde0 0x7ff4340ac890 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c02bf60 comp rx=0 tx=0).stop 2023-10-14T16:40:59.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 -- 172.21.15.29:0/1570603510 shutdown_connections 2023-10-14T16:40:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415dde0 0x7ff4340ac890 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1570603510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff43415c9f0 0x7ff4340c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 -- 172.21.15.29:0/1570603510 >> 172.21.15.29:0/1570603510 conn(0x7ff4340ce6a0 msgr2=0x7ff4340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:40:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 -- 172.21.15.29:0/1570603510 shutdown_connections 2023-10-14T16:40:59.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.784+0000 7ff438ed1640 1 -- 172.21.15.29:0/1570603510 wait complete. 2023-10-14T16:40:59.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.786+0000 7ff438ed1640 1 Processor -- start 2023-10-14T16:40:59.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.786+0000 7ff438ed1640 1 -- start start 2023-10-14T16:40:59.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4340c9230 0x7ff4340c1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 0x7ff43406adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43406c450 con 0x7ff43415dde0 2023-10-14T16:40:59.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4340c1750 con 0x7ff43415c9f0 2023-10-14T16:40:59.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.787+0000 7ff438ed1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43406b840 con 0x7ff4340c9230 2023-10-14T16:40:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39544/0 (socket says 172.21.15.29:39544) 2023-10-14T16:40:59.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 -- 172.21.15.29:0/1509658344 learned_addr learned my addr 172.21.15.29:0/1509658344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:40:59.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff42bfff640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 0x7ff43406adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff4337fe640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4340c9230 0x7ff4340c1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 -- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4340c9230 msgr2=0x7ff4340c1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4340c9230 0x7ff4340c1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 -- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 msgr2=0x7ff43406adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:40:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 0x7ff43406adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:40:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff433fff640 1 -- 172.21.15.29:0/1509658344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff424009400 con 0x7ff43415dde0 2023-10-14T16:40:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.788+0000 7ff42bfff640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 0x7ff43406adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:40:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff433fff640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c009510 con 0x7ff43415dde0 2023-10-14T16:40:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff42c03e450 con 0x7ff43415dde0 2023-10-14T16:40:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c009b30 con 0x7ff43415dde0 2023-10-14T16:40:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42c03e800 con 0x7ff43415dde0 2023-10-14T16:40:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.789+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff43406be00 con 0x7ff43415dde0 2023-10-14T16:40:59.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.791+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff43406c600 con 0x7ff43415dde0 2023-10-14T16:40:59.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.792+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff42c03e960 con 0x7ff43415dde0 2023-10-14T16:40:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.793+0000 7ff430ff9640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 0x7ff41406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:40:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.793+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff42c0ac230 con 0x7ff43415dde0 2023-10-14T16:40:59.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.793+0000 7ff4337fe640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 0x7ff41406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:40:59.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.794+0000 7ff4337fe640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 0x7ff41406d360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff424009b80 tx=0x7ff424008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:40:59.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.797+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff42c077370 con 0x7ff43415dde0 2023-10-14T16:40:59.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.998+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff43404e9b0 con 0x7ff43415dde0 2023-10-14T16:40:59.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:40:59.999+0000 7ff430ff9640 1 -- 172.21.15.29:0/1509658344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff42c07a990 con 0x7ff43415dde0 2023-10-14T16:41:00.000 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 msgr2=0x7ff41406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 0x7ff41406d360 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff424009b80 tx=0x7ff424008040 comp rx=0 tx=0).stop 2023-10-14T16:41:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 msgr2=0x7ff434151510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ff42c009e50 tx=0x7ff42c004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 shutdown_connections 2023-10-14T16:41:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff41406aea0 0x7ff41406d360 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43415dde0 0x7ff434151510 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.001+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43415c9f0 0x7ff43406adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.002+0000 7ff438ed1640 1 --2- 172.21.15.29:0/1509658344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4340c9230 0x7ff4340c1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.002+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 >> 172.21.15.29:0/1509658344 conn(0x7ff4340ce6a0 msgr2=0x7ff4340708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:00.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.002+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 shutdown_connections 2023-10-14T16:41:00.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.002+0000 7ff438ed1640 1 -- 172.21.15.29:0/1509658344 wait complete. 2023-10-14T16:41:00.023 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:00.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.254+0000 7f3512301640 1 Processor -- start 2023-10-14T16:41:00.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.254+0000 7f3512301640 1 -- start start 2023-10-14T16:41:00.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.255+0000 7f3512301640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f350c15ad70 0x7f350c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f3512301640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 0x7f350c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f3512301640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f350c0cb030 con 0x7f350c0c9ca0 2023-10-14T16:41:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f3512301640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350c0c91c0 con 0x7f350c15ad70 2023-10-14T16:41:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f3512301640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0c94a0 con 0x7f350c0caac0 2023-10-14T16:41:00.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f350c15ad70 0x7f350c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f350c15ad70 0x7f350c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39550/0 (socket says 172.21.15.29:39550) 2023-10-14T16:41:00.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 -- 172.21.15.29:0/1561515401 learned_addr learned my addr 172.21.15.29:0/1561515401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:00.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f351089e640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 0x7f350c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 -- 172.21.15.29:0/1561515401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 msgr2=0x7f350c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:00.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 0x7f350c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 -- 172.21.15.29:0/1561515401 >> v1:172.21.15.29:6789/0 conn(0x7f350c0c9ca0 legacy=0x7f350c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f350bfff640 1 -- 172.21.15.29:0/1561515401 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350c0c9780 con 0x7f350c15ad70 2023-10-14T16:41:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.256+0000 7f351089e640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 0x7f350c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.258+0000 7f350bfff640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c15ad70 0x7f350c069da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009870 tx=0x7f34fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623003f0f3e6e1fa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.258+0000 7f350affd640 1 -- 172.21.15.29:0/1561515401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc030840 con 0x7f350c15ad70 2023-10-14T16:41:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.258+0000 7f350affd640 1 -- 172.21.15.29:0/1561515401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc0309a0 con 0x7f350c15ad70 2023-10-14T16:41:00.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.258+0000 7f350affd640 1 -- 172.21.15.29:0/1561515401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc030d30 con 0x7f350c15ad70 2023-10-14T16:41:00.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 -- 172.21.15.29:0/1561515401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c15ad70 msgr2=0x7f350c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c15ad70 0x7f350c069da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f34fc009870 tx=0x7f34fc02b980 comp rx=0 tx=0).stop 2023-10-14T16:41:00.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 -- 172.21.15.29:0/1561515401 shutdown_connections 2023-10-14T16:41:00.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c0caac0 0x7f350c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 --2- 172.21.15.29:0/1561515401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c15ad70 0x7f350c069da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 -- 172.21.15.29:0/1561515401 >> 172.21.15.29:0/1561515401 conn(0x7f350c0ce9b0 msgr2=0x7f350c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 -- 172.21.15.29:0/1561515401 shutdown_connections 2023-10-14T16:41:00.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.259+0000 7f3512301640 1 -- 172.21.15.29:0/1561515401 wait complete. 2023-10-14T16:41:00.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.261+0000 7f3512301640 1 Processor -- start 2023-10-14T16:41:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.261+0000 7f3512301640 1 -- start start 2023-10-14T16:41:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c0caac0 0x7f350c1582a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c15ad70 0x7f350c0b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c0c5050 con 0x7f350c0c9ca0 2023-10-14T16:41:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f350c15b5f0 con 0x7f350c0caac0 2023-10-14T16:41:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f3512301640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350c15b8d0 con 0x7f350c15ad70 2023-10-14T16:41:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39562/0 (socket says 172.21.15.29:39562) 2023-10-14T16:41:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 -- 172.21.15.29:0/1651466279 learned_addr learned my addr 172.21.15.29:0/1651466279 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 -- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c15ad70 msgr2=0x7f350c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c15ad70 0x7f350c0b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 -- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c0caac0 msgr2=0x7f350c1582a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c0caac0 0x7f350c1582a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.262+0000 7f350bfff640 1 -- 172.21.15.29:0/1651466279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34fc0094d0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f350bfff640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34fc000c00 tx=0x7f34fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc030810 con 0x7f350c0c9ca0 2023-10-14T16:41:00.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34fc039d90 con 0x7f350c0c9ca0 2023-10-14T16:41:00.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350c15bbb0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34fc043670 con 0x7f350c0c9ca0 2023-10-14T16:41:00.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.263+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350c1587e0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.264+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350c15c0d0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.265+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f34fc04c3b0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.266+0000 7f3508ff9640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 0x7f34ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.267+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f34fc0abc30 con 0x7f350c0c9ca0 2023-10-14T16:41:00.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.267+0000 7f350b7fe640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 0x7f34ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.267+0000 7f350b7fe640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 0x7f34ec06d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f34f40012e0 tx=0x7f34f4007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.271+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f350c15c0d0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.466+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f350c15c0d0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.468+0000 7f3508ff9640 1 -- 172.21.15.29:0/1651466279 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f350c15c0d0 con 0x7f350c0c9ca0 2023-10-14T16:41:00.470 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.470+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 msgr2=0x7f34ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.470+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 0x7f34ec06d330 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f34f40012e0 tx=0x7f34f4007d30 comp rx=0 tx=0).stop 2023-10-14T16:41:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.470+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 msgr2=0x7f350c0c4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.470+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f34fc000c00 tx=0x7f34fc004730 comp rx=0 tx=0).stop 2023-10-14T16:41:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 shutdown_connections 2023-10-14T16:41:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f34ec06ae70 0x7f34ec06d330 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f350c15ad70 0x7f350c0b2620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f350c0caac0 0x7f350c1582a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 --2- 172.21.15.29:0/1651466279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f350c0c9ca0 0x7f350c0c4b10 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 >> 172.21.15.29:0/1651466279 conn(0x7f350c0ce9b0 msgr2=0x7f350c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 shutdown_connections 2023-10-14T16:41:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.471+0000 7f3512301640 1 -- 172.21.15.29:0/1651466279 wait complete. 2023-10-14T16:41:00.490 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:00.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.713+0000 7fa50759e640 1 Processor -- start 2023-10-14T16:41:00.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.713+0000 7fa50759e640 1 -- start start 2023-10-14T16:41:00.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.715+0000 7fa50759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa50815cd70 0x7fa5080d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.715+0000 7fa50759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5080b8480 0x7fa5080cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.715+0000 7fa50759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa50815d150 con 0x7fa5080d0be0 2023-10-14T16:41:00.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.715+0000 7fa50759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080d1180 con 0x7fa50815cd70 2023-10-14T16:41:00.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.715+0000 7fa50759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5080d1460 con 0x7fa5080b8480 2023-10-14T16:41:00.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa50815cd70 0x7fa5080d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa50815cd70 0x7fa5080d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39574/0 (socket says 172.21.15.29:39574) 2023-10-14T16:41:00.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 -- 172.21.15.29:0/2943130605 learned_addr learned my addr 172.21.15.29:0/2943130605 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:00.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa4fffff640 1 --2- 172.21.15.29:0/2943130605 >> v2:172.21.15.186:3300/0 conn(0x7fa5080b8480 0x7fa5080cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 -- 172.21.15.29:0/2943130605 >> v2:172.21.15.186:3300/0 conn(0x7fa5080b8480 msgr2=0x7fa5080cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 --2- 172.21.15.29:0/2943130605 >> v2:172.21.15.186:3300/0 conn(0x7fa5080b8480 0x7fa5080cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 -- 172.21.15.29:0/2943130605 >> v1:172.21.15.186:6789/0 conn(0x7fa5080d0be0 legacy=0x7fa5080b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.716+0000 7fa50659c640 1 -- 172.21.15.29:0/2943130605 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5080d1740 con 0x7fa50815cd70 2023-10-14T16:41:00.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.717+0000 7fa50659c640 1 --2- 172.21.15.29:0/2943130605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50815cd70 0x7fa5080d2220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa508066a00 tx=0x7fa4f802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b5e0d424a20da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa505d9b640 1 -- 172.21.15.29:0/2943130605 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f80317e0 con 0x7fa50815cd70 2023-10-14T16:41:00.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa505d9b640 1 -- 172.21.15.29:0/2943130605 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f8031940 con 0x7fa50815cd70 2023-10-14T16:41:00.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa505d9b640 1 -- 172.21.15.29:0/2943130605 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f8031cd0 con 0x7fa50815cd70 2023-10-14T16:41:00.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 -- 172.21.15.29:0/2943130605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50815cd70 msgr2=0x7fa5080d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 --2- 172.21.15.29:0/2943130605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50815cd70 0x7fa5080d2220 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa508066a00 tx=0x7fa4f802b880 comp rx=0 tx=0).stop 2023-10-14T16:41:00.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 -- 172.21.15.29:0/2943130605 shutdown_connections 2023-10-14T16:41:00.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 --2- 172.21.15.29:0/2943130605 >> v2:172.21.15.186:3300/0 conn(0x7fa5080b8480 0x7fa5080cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 --2- 172.21.15.29:0/2943130605 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa50815cd70 0x7fa5080d2220 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 -- 172.21.15.29:0/2943130605 >> 172.21.15.29:0/2943130605 conn(0x7fa5080cf570 msgr2=0x7fa5080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:00.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 -- 172.21.15.29:0/2943130605 shutdown_connections 2023-10-14T16:41:00.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.718+0000 7fa50759e640 1 -- 172.21.15.29:0/2943130605 wait complete. 2023-10-14T16:41:00.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.720+0000 7fa50759e640 1 Processor -- start 2023-10-14T16:41:00.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.721+0000 7fa50759e640 1 -- start start 2023-10-14T16:41:00.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.721+0000 7fa50759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5080b8480 0x7fa508145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa50815cd70 0x7fa5081591c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa508158d30 con 0x7fa5080d0be0 2023-10-14T16:41:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5080c8e20 con 0x7fa5080b8480 2023-10-14T16:41:00.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5081f2b00 con 0x7fa50815cd70 2023-10-14T16:41:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39588/0 (socket says 172.21.15.29:39588) 2023-10-14T16:41:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 -- 172.21.15.29:0/2213340069 learned_addr learned my addr 172.21.15.29:0/2213340069 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:00.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa50659c640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5080b8480 0x7fa508145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 -- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa50815cd70 msgr2=0x7fa5081591c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa50815cd70 0x7fa5081591c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 -- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5080b8480 msgr2=0x7fa508145c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5080b8480 0x7fa508145c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.722+0000 7fa4fffff640 1 -- 172.21.15.29:0/2213340069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f80094d0 con 0x7fa5080d0be0 2023-10-14T16:41:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa4fffff640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000d5f0 tx=0x7fa4f000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f00125f0 con 0x7fa5080d0be0 2023-10-14T16:41:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4f0012750 con 0x7fa5080d0be0 2023-10-14T16:41:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5081f2de0 con 0x7fa5080d0be0 2023-10-14T16:41:00.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4f0012b00 con 0x7fa5080d0be0 2023-10-14T16:41:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.723+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5081f5830 con 0x7fa5080d0be0 2023-10-14T16:41:00.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.724+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa508158ea0 con 0x7fa5080d0be0 2023-10-14T16:41:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.725+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa4f0012c60 con 0x7fa5080d0be0 2023-10-14T16:41:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.726+0000 7fa4feffd640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 0x7fa4e806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.726+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa4f0089b80 con 0x7fa5080d0be0 2023-10-14T16:41:00.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.726+0000 7fa50659c640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 0x7fa4e806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.727+0000 7fa50659c640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 0x7fa4e806d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8036000 tx=0x7fa4f803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:00.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.730+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4f0054b90 con 0x7fa5080d0be0 2023-10-14T16:41:00.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.928+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa50804e9b0 con 0x7fa5080d0be0 2023-10-14T16:41:00.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.931+0000 7fa4feffd640 1 -- 172.21.15.29:0/2213340069 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fa4f0089e60 con 0x7fa5080d0be0 2023-10-14T16:41:00.932 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:00.932 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:00.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.933+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 msgr2=0x7fa4e806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.933+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 0x7fa4e806d330 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa4f8036000 tx=0x7fa4f803d040 comp rx=0 tx=0).stop 2023-10-14T16:41:00.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.933+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 msgr2=0x7fa5080b5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:00.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.933+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa4f000d5f0 tx=0x7fa4f000dab0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 shutdown_connections 2023-10-14T16:41:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa4e806ae70 0x7fa4e806d330 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa50815cd70 0x7fa5081591c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5080d0be0 0x7fa5080b5c50 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 --2- 172.21.15.29:0/2213340069 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5080b8480 0x7fa508145c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 >> 172.21.15.29:0/2213340069 conn(0x7fa5080cf570 msgr2=0x7fa5080b77d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 shutdown_connections 2023-10-14T16:41:00.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:00.934+0000 7fa50759e640 1 -- 172.21.15.29:0/2213340069 wait complete. 2023-10-14T16:41:00.954 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:00.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:01.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.185+0000 7fefc032c640 1 Processor -- start 2023-10-14T16:41:01.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.185+0000 7fefc032c640 1 -- start start 2023-10-14T16:41:01.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.187+0000 7fefc032c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefb80caac0 0x7fefb80ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.187+0000 7fefc032c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80cb030 con 0x7fefb806a4b0 2023-10-14T16:41:01.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.187+0000 7fefc032c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80c91c0 con 0x7fefb815ad70 2023-10-14T16:41:01.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.187+0000 7fefc032c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefb80c94a0 con 0x7fefb80caac0 2023-10-14T16:41:01.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbd8c7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fefb806a4b0 0x7fefb80ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60986/0 (socket says 172.21.15.29:60986) 2023-10-14T16:41:01.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbd8c7640 1 -- 172.21.15.29:0/2590502227 learned_addr learned my addr 172.21.15.29:0/2590502227 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:01.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbe8c9640 1 --2- 172.21.15.29:0/2590502227 >> v2:172.21.15.29:3300/0 conn(0x7fefb80caac0 0x7fefb80ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3391701728 0 0) 0x7fefb80cb030 con 0x7fefb806a4b0 2023-10-14T16:41:01.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa4003520 con 0x7fefb806a4b0 2023-10-14T16:41:01.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbe8c9640 1 -- 172.21.15.29:0/2590502227 >> v1:172.21.15.186:6789/0 conn(0x7fefb815ad70 legacy=0x7fefb8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2137130825 0 0) 0x7fefb80c91c0 con 0x7fefb815ad70 2023-10-14T16:41:01.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.188+0000 7fefbe8c9640 1 -- 172.21.15.29:0/2590502227 >> v1:172.21.15.29:6789/0 conn(0x7fefb806a4b0 legacy=0x7fefb80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.189+0000 7fefbe8c9640 1 -- 172.21.15.29:0/2590502227 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80c9780 con 0x7fefb80caac0 2023-10-14T16:41:01.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.190+0000 7fefbe8c9640 1 --2- 172.21.15.29:0/2590502227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb80caac0 0x7fefb80ad4d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009d90 tx=0x7fefb002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e171d7b36c4cac20 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.190+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb002ccf0 con 0x7fefb80caac0 2023-10-14T16:41:01.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.190+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb002ce50 con 0x7fefb80caac0 2023-10-14T16:41:01.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.190+0000 7fefbd0c6640 1 -- 172.21.15.29:0/2590502227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb0034660 con 0x7fefb80caac0 2023-10-14T16:41:01.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 -- 172.21.15.29:0/2590502227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb80caac0 msgr2=0x7fefb80ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 --2- 172.21.15.29:0/2590502227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb80caac0 0x7fefb80ad4d0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fefb0009d90 tx=0x7fefb002bf40 comp rx=0 tx=0).stop 2023-10-14T16:41:01.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 -- 172.21.15.29:0/2590502227 shutdown_connections 2023-10-14T16:41:01.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 --2- 172.21.15.29:0/2590502227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb80caac0 0x7fefb80ad4d0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 -- 172.21.15.29:0/2590502227 >> 172.21.15.29:0/2590502227 conn(0x7fefb80ce9b0 msgr2=0x7fefb80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:01.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 -- 172.21.15.29:0/2590502227 shutdown_connections 2023-10-14T16:41:01.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.191+0000 7fefc032c640 1 -- 172.21.15.29:0/2590502227 wait complete. 2023-10-14T16:41:01.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.193+0000 7fefc032c640 1 Processor -- start 2023-10-14T16:41:01.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.193+0000 7fefc032c640 1 -- start start 2023-10-14T16:41:01.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 0x7fefb814ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefb815ad70 0x7fefb80ade80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c5f10 con 0x7fefb806a4b0 2023-10-14T16:41:01.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c61f0 con 0x7fefb80caac0 2023-10-14T16:41:01.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefc032c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefb80c64d0 con 0x7fefb815ad70 2023-10-14T16:41:01.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefbe0c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefbe0c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56542/0 (socket says 172.21.15.29:56542) 2023-10-14T16:41:01.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.194+0000 7fefbe0c8640 1 -- 172.21.15.29:0/1170821564 learned_addr learned my addr 172.21.15.29:0/1170821564 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:01.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbd8c7640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 0x7fefb814ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe8c9640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefb815ad70 0x7fefb80ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 -- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefb815ad70 msgr2=0x7fefb80ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefb815ad70 0x7fefb80ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 -- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 msgr2=0x7fefb814ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 0x7fefb814ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 -- 172.21.15.29:0/1170821564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb0009a70 con 0x7fefb806a4b0 2023-10-14T16:41:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbd8c7640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 0x7fefb814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.195+0000 7fefbe0c8640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fefb400c410 tx=0x7fefb400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.196+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb40035f0 con 0x7fefb806a4b0 2023-10-14T16:41:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.196+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefb4003770 con 0x7fefb806a4b0 2023-10-14T16:41:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.196+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefb4003b20 con 0x7fefb806a4b0 2023-10-14T16:41:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.196+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb80c67b0 con 0x7fefb806a4b0 2023-10-14T16:41:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.196+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb80c7540 con 0x7fefb806a4b0 2023-10-14T16:41:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.197+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80c6cd0 con 0x7fefb806a4b0 2023-10-14T16:41:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.198+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fefb4003c80 con 0x7fefb806a4b0 2023-10-14T16:41:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.199+0000 7fefaeffd640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 0x7fef9c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.200+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fefb4088c50 con 0x7fefb806a4b0 2023-10-14T16:41:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.200+0000 7fefbd8c7640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 0x7fef9c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.201+0000 7fefbd8c7640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 0x7fef9c06d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefa8007fd0 tx=0x7fefa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.204+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefb80c6cd0 con 0x7fefb806a4b0 2023-10-14T16:41:01.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.402+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fefb80c6cd0 con 0x7fefb806a4b0 2023-10-14T16:41:01.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.403+0000 7fefaeffd640 1 -- 172.21.15.29:0/1170821564 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefb80c6cd0 con 0x7fefb806a4b0 2023-10-14T16:41:01.404 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:01.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 msgr2=0x7fef9c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 0x7fef9c06d330 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fefa8007fd0 tx=0x7fefa8009040 comp rx=0 tx=0).stop 2023-10-14T16:41:01.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 msgr2=0x7fefb80c7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fefb400c410 tx=0x7fefb400c8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 shutdown_connections 2023-10-14T16:41:01.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fef9c06ae70 0x7fef9c06d330 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefb815ad70 0x7fefb80ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefb80caac0 0x7fefb814ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 --2- 172.21.15.29:0/1170821564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefb806a4b0 0x7fefb80c7aa0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 >> 172.21.15.29:0/1170821564 conn(0x7fefb80ce9b0 msgr2=0x7fefb8069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:01.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 shutdown_connections 2023-10-14T16:41:01.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.406+0000 7fefc032c640 1 -- 172.21.15.29:0/1170821564 wait complete. 2023-10-14T16:41:01.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.657+0000 7f69fb18c640 1 Processor -- start 2023-10-14T16:41:01.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.657+0000 7f69fb18c640 1 -- start start 2023-10-14T16:41:01.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fb18c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69f406c040 0x7f69f406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fb18c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69f40cda80 0x7f69f415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fb18c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69f406c960 con 0x7f69f40ad970 2023-10-14T16:41:01.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fb18c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f415b9d0 con 0x7f69f406c040 2023-10-14T16:41:01.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fb18c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69f415bcb0 con 0x7f69f40cda80 2023-10-14T16:41:01.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69fa18a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69f406c040 0x7f69f406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69f9989640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f69f40ad970 0x7f69f415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60992/0 (socket says 172.21.15.29:60992) 2023-10-14T16:41:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.659+0000 7f69f9989640 1 -- 172.21.15.29:0/2266666881 learned_addr learned my addr 172.21.15.29:0/2266666881 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69fa98b640 1 --2- 172.21.15.29:0/2266666881 >> v2:172.21.15.186:3300/0 conn(0x7f69f40cda80 0x7f69f415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69fa18a640 1 -- 172.21.15.29:0/2266666881 >> v2:172.21.15.186:3300/0 conn(0x7f69f40cda80 msgr2=0x7f69f415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:01.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69fa18a640 1 --2- 172.21.15.29:0/2266666881 >> v2:172.21.15.186:3300/0 conn(0x7f69f40cda80 0x7f69f415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69fa18a640 1 -- 172.21.15.29:0/2266666881 >> v1:172.21.15.29:6789/0 conn(0x7f69f40ad970 legacy=0x7f69f415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69fa18a640 1 -- 172.21.15.29:0/2266666881 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f415bf90 con 0x7f69f406c040 2023-10-14T16:41:01.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.660+0000 7f69f9188640 1 -- 172.21.15.29:0/2266666881 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1619387868 0 0) 0x7f69f406c960 con 0x7f69f40ad970 2023-10-14T16:41:01.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.662+0000 7f69fa18a640 1 --2- 172.21.15.29:0/2266666881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f406c040 0x7f69f406c420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69f40cf300 tx=0x7f69f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1deb5703938e9380 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.662+0000 7f69f9188640 1 -- 172.21.15.29:0/2266666881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f002cc80 con 0x7f69f406c040 2023-10-14T16:41:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.662+0000 7f69f9188640 1 -- 172.21.15.29:0/2266666881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69f002cde0 con 0x7f69f406c040 2023-10-14T16:41:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.662+0000 7f69f9188640 1 -- 172.21.15.29:0/2266666881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69f0034660 con 0x7f69f406c040 2023-10-14T16:41:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 -- 172.21.15.29:0/2266666881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f406c040 msgr2=0x7f69f406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 --2- 172.21.15.29:0/2266666881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f406c040 0x7f69f406c420 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f69f40cf300 tx=0x7f69f002bed0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 -- 172.21.15.29:0/2266666881 shutdown_connections 2023-10-14T16:41:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 --2- 172.21.15.29:0/2266666881 >> v2:172.21.15.186:3300/0 conn(0x7f69f40cda80 0x7f69f415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 --2- 172.21.15.29:0/2266666881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f406c040 0x7f69f406c420 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 -- 172.21.15.29:0/2266666881 >> 172.21.15.29:0/2266666881 conn(0x7f69f40ce6a0 msgr2=0x7f69f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:01.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 -- 172.21.15.29:0/2266666881 shutdown_connections 2023-10-14T16:41:01.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.663+0000 7f69fb18c640 1 -- 172.21.15.29:0/2266666881 wait complete. 2023-10-14T16:41:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.665+0000 7f69fb18c640 1 Processor -- start 2023-10-14T16:41:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.665+0000 7f69fb18c640 1 -- start start 2023-10-14T16:41:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.666+0000 7f69fb18c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69f406c040 0x7f69f414fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.666+0000 7f69fb18c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fb18c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69f40cda80 0x7f69f40b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fb18c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40c5770 con 0x7f69f40ad970 2023-10-14T16:41:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fb18c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69f4153ab0 con 0x7f69f406c040 2023-10-14T16:41:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fb18c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69f40c6560 con 0x7f69f40cda80 2023-10-14T16:41:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56554/0 (socket says 172.21.15.29:56554) 2023-10-14T16:41:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 -- 172.21.15.29:0/1417339368 learned_addr learned my addr 172.21.15.29:0/1417339368 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fa18a640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69f406c040 0x7f69f414fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69fa98b640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69f40cda80 0x7f69f40b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 -- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69f40cda80 msgr2=0x7f69f40b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69f40cda80 0x7f69f40b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.667+0000 7f69f9989640 1 -- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69f406c040 msgr2=0x7f69f414fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69f9989640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69f406c040 0x7f69f414fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69f9989640 1 -- 172.21.15.29:0/1417339368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69f0009a70 con 0x7f69f40ad970 2023-10-14T16:41:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69f9989640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69e400c4b0 tx=0x7f69e400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e40074e0 con 0x7f69f40ad970 2023-10-14T16:41:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69e4007660 con 0x7f69f40ad970 2023-10-14T16:41:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69e4004a20 con 0x7f69f40ad970 2023-10-14T16:41:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69f40c5950 con 0x7f69f40ad970 2023-10-14T16:41:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.668+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69f40c61f0 con 0x7f69f40ad970 2023-10-14T16:41:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.670+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69f406b330 con 0x7f69f40ad970 2023-10-14T16:41:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.671+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f69e4004b80 con 0x7f69f40ad970 2023-10-14T16:41:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.672+0000 7f69eaffd640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 0x7f69d806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.673+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f69e40888a0 con 0x7f69f40ad970 2023-10-14T16:41:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.673+0000 7f69fa18a640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 0x7f69d806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.674+0000 7f69fa18a640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 0x7f69d806d360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036000 tx=0x7f69f0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.677+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69e40538e0 con 0x7f69f40ad970 2023-10-14T16:41:01.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.876+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f69f40c5d40 con 0x7f69f40ad970 2023-10-14T16:41:01.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.877+0000 7f69eaffd640 1 -- 172.21.15.29:0/1417339368 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f69e4018020 con 0x7f69f40ad970 2023-10-14T16:41:01.878 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:01.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 msgr2=0x7f69d806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 0x7f69d806d360 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f69f0036000 tx=0x7f69f0030000 comp rx=0 tx=0).stop 2023-10-14T16:41:01.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 msgr2=0x7f69f40c7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:01.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f69e400c4b0 tx=0x7f69e400c970 comp rx=0 tx=0).stop 2023-10-14T16:41:01.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 shutdown_connections 2023-10-14T16:41:01.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f69d806aea0 0x7f69d806d360 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69f40cda80 0x7f69f40b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69f40ad970 0x7f69f40c7370 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 --2- 172.21.15.29:0/1417339368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69f406c040 0x7f69f414fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 >> 172.21.15.29:0/1417339368 conn(0x7f69f40ce6a0 msgr2=0x7f69f406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 shutdown_connections 2023-10-14T16:41:01.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:01.880+0000 7f69fb18c640 1 -- 172.21.15.29:0/1417339368 wait complete. 2023-10-14T16:41:01.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:02.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.132+0000 7f26634d7640 1 Processor -- start 2023-10-14T16:41:02.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.133+0000 7f26634d7640 1 -- start start 2023-10-14T16:41:02.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.134+0000 7f26634d7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f265c0caa80 0x7f265c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f26634d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c06a2e0 0x7f265c1650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f26634d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f265c06a730 con 0x7f265c0addb0 2023-10-14T16:41:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f26634d7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f265c161a50 con 0x7f265c0caa80 2023-10-14T16:41:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f26634d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0caef0 con 0x7f265c06a2e0 2023-10-14T16:41:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2661273640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f265c0addb0 0x7f265c0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60994/0 (socket says 172.21.15.29:60994) 2023-10-14T16:41:02.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2661273640 1 -- 172.21.15.29:0/903322341 learned_addr learned my addr 172.21.15.29:0/903322341 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:02.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 176202053 0 0) 0x7f2658001110 con 0x7f265c0addb0 2023-10-14T16:41:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2648003520 con 0x7f265c0addb0 2023-10-14T16:41:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2660a72640 1 --2- 172.21.15.29:0/903322341 >> v2:172.21.15.186:3300/0 conn(0x7f265c0caa80 0x7f265c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.135+0000 7f2661a74640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c06a2e0 0x7f265c1650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3516415248 0 0) 0x7f2648003520 con 0x7f265c0addb0 2023-10-14T16:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c06a2e0 msgr2=0x7f265c1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c06a2e0 0x7f265c1650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 >> v2:172.21.15.186:3300/0 conn(0x7f265c0caa80 msgr2=0x7f265c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 --2- 172.21.15.29:0/903322341 >> v2:172.21.15.186:3300/0 conn(0x7f265c0caa80 0x7f265c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0cb140 con 0x7f265c0addb0 2023-10-14T16:41:02.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2658001800 con 0x7f265c0addb0 2023-10-14T16:41:02.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 >> v1:172.21.15.29:6789/0 conn(0x7f265c0addb0 legacy=0x7f265c0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2648003970 0x7f2648003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26480047c0 0x7f2648004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648005180 0x7f264800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c161a50 con 0x7f2648003970 2023-10-14T16:41:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0caef0 con 0x7f26480047c0 2023-10-14T16:41:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f264800bf70 con 0x7f2648005180 2023-10-14T16:41:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2661a74640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f265c06a2e0 0x7f265c1650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:02.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.136+0000 7f2661273640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2648003970 0x7f2648003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2660a72640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26480047c0 0x7f2648004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661a74640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648005180 0x7f264800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661273640 1 -- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648005180 msgr2=0x7f264800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661273640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648005180 0x7f264800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661273640 1 -- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26480047c0 msgr2=0x7f2648004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:02.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661273640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26480047c0 0x7f2648004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661273640 1 -- 172.21.15.29:0/903322341 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2648003520 con 0x7f2648003970 2023-10-14T16:41:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.137+0000 7f2661a74640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648005180 0x7f264800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.138+0000 7f2661273640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2648003970 0x7f2648003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f265800c110 tx=0x7f26480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdb1c442ccd3d423 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.138+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2658005ad0 con 0x7f2648003970 2023-10-14T16:41:02.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.138+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2658005c30 con 0x7f2648003970 2023-10-14T16:41:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.138+0000 7f2653fff640 1 -- 172.21.15.29:0/903322341 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f265802c810 con 0x7f2648003970 2023-10-14T16:41:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 -- 172.21.15.29:0/903322341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2648003970 msgr2=0x7f2648003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 --2- 172.21.15.29:0/903322341 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2648003970 0x7f2648003d90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f265800c110 tx=0x7f26480057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f2661a74640 1 -- 172.21.15.29:0/903322341 reap_dead start 2023-10-14T16:41:02.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 -- 172.21.15.29:0/903322341 shutdown_connections 2023-10-14T16:41:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 -- 172.21.15.29:0/903322341 >> 172.21.15.29:0/903322341 conn(0x7f265c0ce9b0 msgr2=0x7f265c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 -- 172.21.15.29:0/903322341 shutdown_connections 2023-10-14T16:41:02.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.139+0000 7f26634d7640 1 -- 172.21.15.29:0/903322341 wait complete. 2023-10-14T16:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.141+0000 7f26634d7640 1 Processor -- start 2023-10-14T16:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.141+0000 7f26634d7640 1 -- start start 2023-10-14T16:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648003970 0x7f265c156da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c0c6130 0x7f265c0c65b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0c7980 con 0x7f265c0c8590 2023-10-14T16:41:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0c6e00 con 0x7f265c0c6130 2023-10-14T16:41:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f26634d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f265c0c70e0 con 0x7f2648003970 2023-10-14T16:41:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f2660a72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f2660a72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56578/0 (socket says 172.21.15.29:56578) 2023-10-14T16:41:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.142+0000 7f2660a72640 1 -- 172.21.15.29:0/4163545954 learned_addr learned my addr 172.21.15.29:0/4163545954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2661273640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648003970 0x7f265c156da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2661a74640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c0c6130 0x7f265c0c65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 -- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648003970 msgr2=0x7f265c156da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648003970 0x7f265c156da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 -- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c0c6130 msgr2=0x7f265c0c65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c0c6130 0x7f265c0c65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 -- 172.21.15.29:0/4163545954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f265c0c73c0 con 0x7f265c0c8590 2023-10-14T16:41:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2660a72640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f264c011a70 tx=0x7f264c011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c009680 con 0x7f265c0c8590 2023-10-14T16:41:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f264c00fb50 con 0x7f265c0c8590 2023-10-14T16:41:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f264c00fee0 con 0x7f265c0c8590 2023-10-14T16:41:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.143+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f265c0c76a0 con 0x7f265c0c8590 2023-10-14T16:41:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.144+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f265c0c6af0 con 0x7f265c0c8590 2023-10-14T16:41:02.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.145+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f265c04e160 con 0x7f265c0c8590 2023-10-14T16:41:02.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.146+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f264c014440 con 0x7f265c0c8590 2023-10-14T16:41:02.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.147+0000 7f2651ffb640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 0x7f264006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.147+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f264c089f10 con 0x7f265c0c8590 2023-10-14T16:41:02.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.147+0000 7f2661273640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 0x7f264006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.148+0000 7f2661273640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 0x7f264006d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2658034000 tx=0x7f2658007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.151+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f264c054f20 con 0x7f265c0c8590 2023-10-14T16:41:02.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.349+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f265c0b2ca0 con 0x7f265c0c8590 2023-10-14T16:41:02.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.349+0000 7f2651ffb640 1 -- 172.21.15.29:0/4163545954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f264c022020 con 0x7f265c0c8590 2023-10-14T16:41:02.350 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:02.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 msgr2=0x7f264006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 0x7f264006d330 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2658034000 tx=0x7f2658007000 comp rx=0 tx=0).stop 2023-10-14T16:41:02.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 msgr2=0x7f265c0c8a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f264c011a70 tx=0x7f264c011f30 comp rx=0 tx=0).stop 2023-10-14T16:41:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 shutdown_connections 2023-10-14T16:41:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f264006ae70 0x7f264006d330 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f265c0c6130 0x7f265c0c65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f265c0c8590 0x7f265c0c8a10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 --2- 172.21.15.29:0/4163545954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2648003970 0x7f265c156da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 >> 172.21.15.29:0/4163545954 conn(0x7f265c0ce9b0 msgr2=0x7f265c0afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:02.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 shutdown_connections 2023-10-14T16:41:02.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.352+0000 7f26634d7640 1 -- 172.21.15.29:0/4163545954 wait complete. 2023-10-14T16:41:02.373 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:02.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.602+0000 7f134e72e640 1 Processor -- start 2023-10-14T16:41:02.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.602+0000 7f134e72e640 1 -- start start 2023-10-14T16:41:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134e72e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 0x7f134806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134e72e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13480cda80 0x7f134815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134e72e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134806c960 con 0x7f13480ad970 2023-10-14T16:41:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134e72e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134815b9d0 con 0x7f13480cda80 2023-10-14T16:41:02.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134e72e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134815bcb0 con 0x7f134806c040 2023-10-14T16:41:02.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134df2d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13480cda80 0x7f134815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134cf2b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f13480ad970 0x7f134815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32768/0 (socket says 172.21.15.29:32768) 2023-10-14T16:41:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.604+0000 7f134cf2b640 1 -- 172.21.15.29:0/754891752 learned_addr learned my addr 172.21.15.29:0/754891752 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134d72c640 1 --2- 172.21.15.29:0/754891752 >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 0x7f134806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134df2d640 1 -- 172.21.15.29:0/754891752 >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 msgr2=0x7f134806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134df2d640 1 --2- 172.21.15.29:0/754891752 >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 0x7f134806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134df2d640 1 -- 172.21.15.29:0/754891752 >> v1:172.21.15.29:6789/0 conn(0x7f13480ad970 legacy=0x7f134815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134df2d640 1 -- 172.21.15.29:0/754891752 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134815bf90 con 0x7f13480cda80 2023-10-14T16:41:02.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f133ffff640 1 -- 172.21.15.29:0/754891752 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557032962 0 0) 0x7f134806c960 con 0x7f13480ad970 2023-10-14T16:41:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.605+0000 7f134d72c640 1 --2- 172.21.15.29:0/754891752 >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 0x7f134806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:02.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.606+0000 7f134df2d640 1 --2- 172.21.15.29:0/754891752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f134815ed90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f13480cf350 tx=0x7f134002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa78e37ec35f92c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.606+0000 7f133ffff640 1 -- 172.21.15.29:0/754891752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134002cc70 con 0x7f13480cda80 2023-10-14T16:41:02.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.606+0000 7f133ffff640 1 -- 172.21.15.29:0/754891752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134002cdd0 con 0x7f13480cda80 2023-10-14T16:41:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.606+0000 7f133ffff640 1 -- 172.21.15.29:0/754891752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1340034660 con 0x7f13480cda80 2023-10-14T16:41:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 -- 172.21.15.29:0/754891752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 msgr2=0x7f134815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 --2- 172.21.15.29:0/754891752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f134815ed90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f13480cf350 tx=0x7f134002bec0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 -- 172.21.15.29:0/754891752 shutdown_connections 2023-10-14T16:41:02.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 --2- 172.21.15.29:0/754891752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f134815ed90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 --2- 172.21.15.29:0/754891752 >> v2:172.21.15.186:3300/0 conn(0x7f134806c040 0x7f134806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 -- 172.21.15.29:0/754891752 >> 172.21.15.29:0/754891752 conn(0x7f13480ce6a0 msgr2=0x7f13480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 -- 172.21.15.29:0/754891752 shutdown_connections 2023-10-14T16:41:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.607+0000 7f134e72e640 1 -- 172.21.15.29:0/754891752 wait complete. 2023-10-14T16:41:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.609+0000 7f134e72e640 1 Processor -- start 2023-10-14T16:41:02.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.609+0000 7f134e72e640 1 -- start start 2023-10-14T16:41:02.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f134806c040 0x7f13480c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 0x7f134814e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f134814e9c0 con 0x7f13480cda80 2023-10-14T16:41:02.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13480ca240 con 0x7f13480ad970 2023-10-14T16:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134e72e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13481f5a10 con 0x7f134806c040 2023-10-14T16:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134df2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134df2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56586/0 (socket says 172.21.15.29:56586) 2023-10-14T16:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.610+0000 7f134df2d640 1 -- 172.21.15.29:0/1582110897 learned_addr learned my addr 172.21.15.29:0/1582110897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134cf2b640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 0x7f134814e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134d72c640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f134806c040 0x7f13480c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 -- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f134806c040 msgr2=0x7f13480c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f134806c040 0x7f13480c8220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 -- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 msgr2=0x7f134814e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 0x7f134814e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 -- 172.21.15.29:0/1582110897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1340009a70 con 0x7f13480cda80 2023-10-14T16:41:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134cf2b640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 0x7f134814e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134df2d640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1340009e80 tx=0x7f1340004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134003e610 con 0x7f13480cda80 2023-10-14T16:41:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f134003e790 con 0x7f13480cda80 2023-10-14T16:41:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f134003eb40 con 0x7f13480cda80 2023-10-14T16:41:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.611+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13481f5cf0 con 0x7f13480cda80 2023-10-14T16:41:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.612+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13481f62b0 con 0x7f13480cda80 2023-10-14T16:41:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.613+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134806b330 con 0x7f13480cda80 2023-10-14T16:41:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.614+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f134003b050 con 0x7f13480cda80 2023-10-14T16:41:02.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.615+0000 7f133dffb640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 0x7f132c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:02.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.615+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f13400ac080 con 0x7f13480cda80 2023-10-14T16:41:02.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.615+0000 7f134d72c640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 0x7f132c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:02.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.616+0000 7f134d72c640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 0x7f132c06d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1344008850 tx=0x7f1344008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:02.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.619+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13400defe0 con 0x7f13480cda80 2023-10-14T16:41:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.817+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f134804e9b0 con 0x7f13480cda80 2023-10-14T16:41:02.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.819+0000 7f133dffb640 1 -- 172.21.15.29:0/1582110897 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7f1340077090 con 0x7f13480cda80 2023-10-14T16:41:02.821 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:02.821 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 msgr2=0x7f132c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 0x7f132c06d330 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f1344008850 tx=0x7f1344008040 comp rx=0 tx=0).stop 2023-10-14T16:41:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 msgr2=0x7f13480ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1340009e80 tx=0x7f1340004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 shutdown_connections 2023-10-14T16:41:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f132c06ae70 0x7f132c06d330 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13480cda80 0x7f13480ac890 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13480ad970 0x7f134814e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 --2- 172.21.15.29:0/1582110897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f134806c040 0x7f13480c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.822+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 >> 172.21.15.29:0/1582110897 conn(0x7f13480ce6a0 msgr2=0x7f134815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.823+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 shutdown_connections 2023-10-14T16:41:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:02.823+0000 7f134e72e640 1 -- 172.21.15.29:0/1582110897 wait complete. 2023-10-14T16:41:02.844 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:02.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.073+0000 7f6e6359e640 1 Processor -- start 2023-10-14T16:41:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.073+0000 7f6e6359e640 1 -- start start 2023-10-14T16:41:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e6359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6e640adb30 0x7f6e640cd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e6359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e640b0c90 0x7f6e640c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e6359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e6406f3a0 con 0x7f6e6406c630 2023-10-14T16:41:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e6359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640c8e70 con 0x7f6e640b0c90 2023-10-14T16:41:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e6359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e640c9150 con 0x7f6e640adb30 2023-10-14T16:41:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.075+0000 7f6e62d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e640b0c90 0x7f6e640c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6e640b0c90 0x7f6e640c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56602/0 (socket says 172.21.15.29:56602) 2023-10-14T16:41:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 -- 172.21.15.29:0/3275881535 learned_addr learned my addr 172.21.15.29:0/3275881535 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e5bfff640 1 --2- 172.21.15.29:0/3275881535 >> v2:172.21.15.186:3300/0 conn(0x7f6e640adb30 0x7f6e640cd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 -- 172.21.15.29:0/3275881535 >> v2:172.21.15.186:3300/0 conn(0x7f6e640adb30 msgr2=0x7f6e640cd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 --2- 172.21.15.29:0/3275881535 >> v2:172.21.15.186:3300/0 conn(0x7f6e640adb30 0x7f6e640cd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 -- 172.21.15.29:0/3275881535 >> v1:172.21.15.186:6789/0 conn(0x7f6e6406c630 legacy=0x7f6e640cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.076+0000 7f6e62d9d640 1 -- 172.21.15.29:0/3275881535 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e640c9430 con 0x7f6e640b0c90 2023-10-14T16:41:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.077+0000 7f6e62d9d640 1 --2- 172.21.15.29:0/3275881535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e640b0c90 0x7f6e640c9950 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6e64073cb0 tx=0x7f6e5c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3c35bd64e7ac99f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.078+0000 7f6e61d9b640 1 -- 172.21.15.29:0/3275881535 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c02c9f0 con 0x7f6e640b0c90 2023-10-14T16:41:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.078+0000 7f6e61d9b640 1 -- 172.21.15.29:0/3275881535 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e5c02cb50 con 0x7f6e640b0c90 2023-10-14T16:41:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.078+0000 7f6e61d9b640 1 -- 172.21.15.29:0/3275881535 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0343f0 con 0x7f6e640b0c90 2023-10-14T16:41:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.078+0000 7f6e6359e640 1 -- 172.21.15.29:0/3275881535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e640b0c90 msgr2=0x7f6e640c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.078+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3275881535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e640b0c90 0x7f6e640c9950 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f6e64073cb0 tx=0x7f6e5c02be50 comp rx=0 tx=0).stop 2023-10-14T16:41:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 -- 172.21.15.29:0/3275881535 shutdown_connections 2023-10-14T16:41:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3275881535 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e640b0c90 0x7f6e640c9950 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3275881535 >> v2:172.21.15.186:3300/0 conn(0x7f6e640adb30 0x7f6e640cd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 -- 172.21.15.29:0/3275881535 >> 172.21.15.29:0/3275881535 conn(0x7f6e640b0170 msgr2=0x7f6e640b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 -- 172.21.15.29:0/3275881535 shutdown_connections 2023-10-14T16:41:03.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.079+0000 7f6e6359e640 1 -- 172.21.15.29:0/3275881535 wait complete. 2023-10-14T16:41:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.081+0000 7f6e6359e640 1 Processor -- start 2023-10-14T16:41:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.081+0000 7f6e6359e640 1 -- start start 2023-10-14T16:41:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e640adb30 0x7f6e640c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 0x7f6e64163fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640c66a0 con 0x7f6e6406c630 2023-10-14T16:41:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640b1340 con 0x7f6e640b0c90 2023-10-14T16:41:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e640b1620 con 0x7f6e640adb30 2023-10-14T16:41:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56616/0 (socket says 172.21.15.29:56616) 2023-10-14T16:41:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e6259c640 1 -- 172.21.15.29:0/3563042922 learned_addr learned my addr 172.21.15.29:0/3563042922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e5bfff640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e640adb30 0x7f6e640c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.082+0000 7f6e62d9d640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 0x7f6e64163fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 -- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e640adb30 msgr2=0x7f6e640c6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e640adb30 0x7f6e640c6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 -- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 msgr2=0x7f6e64163fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 0x7f6e64163fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 -- 172.21.15.29:0/3563042922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c009b30 con 0x7f6e6406c630 2023-10-14T16:41:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e62d9d640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 0x7f6e64163fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6259c640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c430 tx=0x7f6e5400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54011790 con 0x7f6e6406c630 2023-10-14T16:41:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e540118f0 con 0x7f6e6406c630 2023-10-14T16:41:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54011ce0 con 0x7f6e6406c630 2023-10-14T16:41:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e640b1900 con 0x7f6e6406c630 2023-10-14T16:41:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.083+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e640b1ec0 con 0x7f6e6406c630 2023-10-14T16:41:03.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.085+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e640c77a0 con 0x7f6e6406c630 2023-10-14T16:41:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.086+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6e5401c460 con 0x7f6e6406c630 2023-10-14T16:41:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.087+0000 7f6e5affd640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 0x7f6e4406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.087+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6e54089560 con 0x7f6e6406c630 2023-10-14T16:41:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.087+0000 7f6e5bfff640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 0x7f6e4406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.088+0000 7f6e5bfff640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 0x7f6e4406d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6e640af630 tx=0x7f6e4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.091+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e54054610 con 0x7f6e6406c630 2023-10-14T16:41:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.286+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6e6404e9b0 con 0x7f6e6406c630 2023-10-14T16:41:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.287+0000 7f6e5affd640 1 -- 172.21.15.29:0/3563042922 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6e54057c30 con 0x7f6e6406c630 2023-10-14T16:41:03.288 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 msgr2=0x7f6e4406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 0x7f6e4406d330 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6e640af630 tx=0x7f6e4c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 msgr2=0x7f6e640c7260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400c430 tx=0x7f6e5400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 shutdown_connections 2023-10-14T16:41:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6e4406ae70 0x7f6e4406d330 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e640b0c90 0x7f6e64163fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e640adb30 0x7f6e640c6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 --2- 172.21.15.29:0/3563042922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e6406c630 0x7f6e640c7260 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.290+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 >> 172.21.15.29:0/3563042922 conn(0x7f6e640b0170 msgr2=0x7f6e640af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.291+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 shutdown_connections 2023-10-14T16:41:03.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.291+0000 7f6e6359e640 1 -- 172.21.15.29:0/3563042922 wait complete. 2023-10-14T16:41:03.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.540+0000 7fae7dff0640 1 Processor -- start 2023-10-14T16:41:03.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.540+0000 7fae7dff0640 1 -- start start 2023-10-14T16:41:03.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.541+0000 7fae7dff0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fae7815d150 0x7fae780ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.541+0000 7fae7dff0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 0x7fae78160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae7dff0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae780ac890 con 0x7fae78069da0 2023-10-14T16:41:03.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae7dff0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae7815d530 con 0x7fae7815d150 2023-10-14T16:41:03.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae7dff0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae78161650 con 0x7fae78160b90 2023-10-14T16:41:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae77fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae78069da0 0x7fae7806a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32782/0 (socket says 172.21.15.29:32782) 2023-10-14T16:41:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae77fff640 1 -- 172.21.15.29:0/2410416858 learned_addr learned my addr 172.21.15.29:0/2410416858 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae777fe640 1 --2- 172.21.15.29:0/2410416858 >> v2:172.21.15.186:3300/0 conn(0x7fae7815d150 0x7fae780ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 0x7fae78160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.542+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1274367740 0 0) 0x7fae780ac890 con 0x7fae78069da0 2023-10-14T16:41:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae60003520 con 0x7fae78069da0 2023-10-14T16:41:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2976905727 0 0) 0x7fae60003520 con 0x7fae78069da0 2023-10-14T16:41:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 msgr2=0x7fae78160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 0x7fae78160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 >> v2:172.21.15.186:3300/0 conn(0x7fae7815d150 msgr2=0x7fae780ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 --2- 172.21.15.29:0/2410416858 >> v2:172.21.15.186:3300/0 conn(0x7fae7815d150 0x7fae780ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae78161930 con 0x7fae78069da0 2023-10-14T16:41:03.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fae700019f0 con 0x7fae78069da0 2023-10-14T16:41:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 >> v1:172.21.15.29:6789/0 conn(0x7fae78069da0 legacy=0x7fae7806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae60003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae600047c0 0x7fae60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 0x7fae60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae7815d530 con 0x7fae60003970 2023-10-14T16:41:03.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae78161650 con 0x7fae600047c0 2023-10-14T16:41:03.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae6000b980 con 0x7fae78160b90 2023-10-14T16:41:03.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.543+0000 7fae777fe640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae60003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae600047c0 0x7fae60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae777fe640 1 -- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 msgr2=0x7fae60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae777fe640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae78160b90 0x7fae60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae777fe640 1 -- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae600047c0 msgr2=0x7fae60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:03.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae777fe640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae600047c0 0x7fae60004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae777fe640 1 -- 172.21.15.29:0/2410416858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae60003520 con 0x7fae60003970 2023-10-14T16:41:03.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.544+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae600047c0 0x7fae60004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.545+0000 7fae777fe640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae60003d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ac0 tx=0x7fae600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625e80374aeee20b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6802c6f0 con 0x7fae60003970 2023-10-14T16:41:03.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6802c850 con 0x7fae60003970 2023-10-14T16:41:03.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae767fc640 1 -- 172.21.15.29:0/2410416858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6802cbe0 con 0x7fae60003970 2023-10-14T16:41:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae7dff0640 1 -- 172.21.15.29:0/2410416858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 msgr2=0x7fae60003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2410416858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae60003d90 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fae68009ac0 tx=0x7fae600057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae77fff640 1 -- 172.21.15.29:0/2410416858 reap_dead start 2023-10-14T16:41:03.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae7dff0640 1 -- 172.21.15.29:0/2410416858 shutdown_connections 2023-10-14T16:41:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae7dff0640 1 -- 172.21.15.29:0/2410416858 >> 172.21.15.29:0/2410416858 conn(0x7fae780ce9b0 msgr2=0x7fae780cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.546+0000 7fae7dff0640 1 -- 172.21.15.29:0/2410416858 shutdown_connections 2023-10-14T16:41:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.547+0000 7fae7dff0640 1 -- 172.21.15.29:0/2410416858 wait complete. 2023-10-14T16:41:03.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.548+0000 7fae7dff0640 1 Processor -- start 2023-10-14T16:41:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 -- start start 2023-10-14T16:41:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 0x7fae780af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae780af9f0 0x7fae781e6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae780b07e0 con 0x7fae60003970 2023-10-14T16:41:03.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae78156490 con 0x7fae780af030 2023-10-14T16:41:03.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.549+0000 7fae7dff0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae780aff40 con 0x7fae780af9f0 2023-10-14T16:41:03.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56634/0 (socket says 172.21.15.29:56634) 2023-10-14T16:41:03.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 -- 172.21.15.29:0/2891830878 learned_addr learned my addr 172.21.15.29:0/2891830878 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:03.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 0x7fae780af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 -- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae780af9f0 msgr2=0x7fae781e6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae780af9f0 0x7fae781e6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 -- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 msgr2=0x7fae780af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 0x7fae780af4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae777fe640 1 -- 172.21.15.29:0/2891830878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae780b0220 con 0x7fae60003970 2023-10-14T16:41:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.550+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 0x7fae780af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:03.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae777fe640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae6802c290 tx=0x7fae68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68003810 con 0x7fae60003970 2023-10-14T16:41:03.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68003970 con 0x7fae60003970 2023-10-14T16:41:03.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68003ce0 con 0x7fae60003970 2023-10-14T16:41:03.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae780b0500 con 0x7fae60003970 2023-10-14T16:41:03.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.551+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae781e72d0 con 0x7fae60003970 2023-10-14T16:41:03.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.552+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7804e160 con 0x7fae60003970 2023-10-14T16:41:03.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.553+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae68040450 con 0x7fae60003970 2023-10-14T16:41:03.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.555+0000 7fae5ffff640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 0x7fae6406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:03.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.555+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae680ab500 con 0x7fae60003970 2023-10-14T16:41:03.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.555+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 0x7fae6406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:03.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.556+0000 7fae76ffd640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 0x7fae6406d360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae780c8a30 tx=0x7fae6c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:03.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.559+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae68076510 con 0x7fae60003970 2023-10-14T16:41:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.755+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae781e69b0 con 0x7fae60003970 2023-10-14T16:41:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.756+0000 7fae5ffff640 1 -- 172.21.15.29:0/2891830878 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae68079b30 con 0x7fae60003970 2023-10-14T16:41:03.757 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 msgr2=0x7fae6406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 0x7fae6406d360 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae780c8a30 tx=0x7fae6c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 msgr2=0x7fae78155470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:03.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fae6802c290 tx=0x7fae68004730 comp rx=0 tx=0).stop 2023-10-14T16:41:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 shutdown_connections 2023-10-14T16:41:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae6406aea0 0x7fae6406d360 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae780af9f0 0x7fae781e6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae780af030 0x7fae780af4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 --2- 172.21.15.29:0/2891830878 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae60003970 0x7fae78155470 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.758+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 >> 172.21.15.29:0/2891830878 conn(0x7fae780ce9b0 msgr2=0x7fae78070a50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:03.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.759+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 shutdown_connections 2023-10-14T16:41:03.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:03.759+0000 7fae7dff0640 1 -- 172.21.15.29:0/2891830878 wait complete. 2023-10-14T16:41:03.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:04.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.005+0000 7f3ca2165640 1 Processor -- start 2023-10-14T16:41:04.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.006+0000 7f3ca2165640 1 -- start start 2023-10-14T16:41:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3ca2165640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3c9c15ad70 0x7f3c9c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3ca2165640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c0cb030 con 0x7f3c9c0caac0 2023-10-14T16:41:04.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3ca2165640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c9c0c91c0 con 0x7f3c9c0c9ca0 2023-10-14T16:41:04.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3ca2165640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c9c0c94a0 con 0x7f3c9c15ad70 2023-10-14T16:41:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3c9bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c9c0caac0 0x7f3c9c0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32794/0 (socket says 172.21.15.29:32794) 2023-10-14T16:41:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2081384060 learned_addr learned my addr 172.21.15.29:0/2081384060 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2081384060 >> v2:172.21.15.186:3300/0 conn(0x7f3c9c15ad70 0x7f3c9c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2617267068 0 0) 0x7f3c9c0cb030 con 0x7f3c9c0caac0 2023-10-14T16:41:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.007+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c88003520 con 0x7f3c9c0caac0 2023-10-14T16:41:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2730629395 0 0) 0x7f3c88003520 con 0x7f3c9c0caac0 2023-10-14T16:41:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 >> v2:172.21.15.186:3300/0 conn(0x7f3c9c15ad70 msgr2=0x7f3c9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 --2- 172.21.15.29:0/2081384060 >> v2:172.21.15.186:3300/0 conn(0x7f3c9c15ad70 0x7f3c9c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 >> v1:172.21.15.186:6789/0 conn(0x7f3c9c0c9ca0 legacy=0x7f3c9c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0c9780 con 0x7f3c9c0caac0 2023-10-14T16:41:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3c94001670 con 0x7f3c9c0caac0 2023-10-14T16:41:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 >> v1:172.21.15.29:6789/0 conn(0x7f3c9c0caac0 legacy=0x7f3c9c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c880039c0 0x7f3c88003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 0x7f3c88004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c88005180 0x7f3c8800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0c94a0 con 0x7f3c880047c0 2023-10-14T16:41:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800c080 con 0x7f3c880039c0 2023-10-14T16:41:04.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8800c240 con 0x7f3c88005180 2023-10-14T16:41:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.008+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 0x7f3c88004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c880039c0 0x7f3c88003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9bfff640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c88005180 0x7f3c8800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9affd640 1 -- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c88005180 msgr2=0x7f3c8800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c88005180 0x7f3c8800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9affd640 1 -- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c880039c0 msgr2=0x7f3c88003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c880039c0 0x7f3c88003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9affd640 1 -- 172.21.15.29:0/2081384060 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c88003520 con 0x7f3c880047c0 2023-10-14T16:41:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9bfff640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c88005180 0x7f3c8800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:04.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.009+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2081384060 reap_dead start 2023-10-14T16:41:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.010+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 0x7f3c88004c40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3c84001750 tx=0x7f3c880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1c4f5098c55645f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.010+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c840052d0 con 0x7f3c880047c0 2023-10-14T16:41:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.010+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c8402e050 con 0x7f3c880047c0 2023-10-14T16:41:04.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.010+0000 7f3c9a7fc640 1 -- 172.21.15.29:0/2081384060 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c84003830 con 0x7f3c880047c0 2023-10-14T16:41:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 -- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 msgr2=0x7f3c88004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 0x7f3c88004c40 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f3c84001750 tx=0x7f3c880057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 -- 172.21.15.29:0/2081384060 shutdown_connections 2023-10-14T16:41:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2081384060 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c880047c0 0x7f3c88004c40 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 -- 172.21.15.29:0/2081384060 >> 172.21.15.29:0/2081384060 conn(0x7f3c9c0ce9b0 msgr2=0x7f3c9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 -- 172.21.15.29:0/2081384060 shutdown_connections 2023-10-14T16:41:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.011+0000 7f3ca2165640 1 -- 172.21.15.29:0/2081384060 wait complete. 2023-10-14T16:41:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.013+0000 7f3ca2165640 1 Processor -- start 2023-10-14T16:41:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.013+0000 7f3ca2165640 1 -- start start 2023-10-14T16:41:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 0x7f3c9c0a8730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 0x7f3c9c1f3060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0c7140 con 0x7f3c9c14e560 2023-10-14T16:41:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0c7420 con 0x7f3c9c0c68e0 2023-10-14T16:41:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3ca2165640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c9c0c7700 con 0x7f3c880047c0 2023-10-14T16:41:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.014+0000 7f3c9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56650/0 (socket says 172.21.15.29:56650) 2023-10-14T16:41:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2341630 learned_addr learned my addr 172.21.15.29:0/2341630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 0x7f3c9c1f3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 msgr2=0x7f3c9c0a8730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 0x7f3c9c0a8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 0x7f3c9c0a8730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 msgr2=0x7f3c9c1f3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 0x7f3c9c1f3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 -- 172.21.15.29:0/2341630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c9c0c79e0 con 0x7f3c9c14e560 2023-10-14T16:41:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9affd640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 0x7f3c9c1f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 0x7f3c9c0a8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.015+0000 7f3c9bfff640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c94001200 tx=0x7f3c9400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.016+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94007440 con 0x7f3c9c14e560 2023-10-14T16:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.016+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c940075c0 con 0x7f3c9c14e560 2023-10-14T16:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.016+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c9c1f35a0 con 0x7f3c9c14e560 2023-10-14T16:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.016+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c94004a20 con 0x7f3c9c14e560 2023-10-14T16:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.016+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c9c14ef20 con 0x7f3c9c14e560 2023-10-14T16:41:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.017+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c9c04e160 con 0x7f3c9c14e560 2023-10-14T16:41:04.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.018+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3c94004b80 con 0x7f3c9c14e560 2023-10-14T16:41:04.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.019+0000 7f3c93fff640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 0x7f3c7c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.019+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f3c94088fe0 con 0x7f3c9c14e560 2023-10-14T16:41:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.019+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 0x7f3c7c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.020+0000 7f3c9b7fe640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 0x7f3c7c06d360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c007440 tx=0x7f3c8c00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.023+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c940540e0 con 0x7f3c9c14e560 2023-10-14T16:41:04.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.222+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3c9c06f3b0 con 0x7f3c9c14e560 2023-10-14T16:41:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.223+0000 7f3c93fff640 1 -- 172.21.15.29:0/2341630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3c94057700 con 0x7f3c9c14e560 2023-10-14T16:41:04.225 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 msgr2=0x7f3c7c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 0x7f3c7c06d360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f3c8c007440 tx=0x7f3c8c00b040 comp rx=0 tx=0).stop 2023-10-14T16:41:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 msgr2=0x7f3c9c14e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f3c94001200 tx=0x7f3c9400d950 comp rx=0 tx=0).stop 2023-10-14T16:41:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 shutdown_connections 2023-10-14T16:41:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3c7c06aea0 0x7f3c7c06d360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c9c14e560 0x7f3c9c14e9e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c9c0c68e0 0x7f3c9c1f3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 --2- 172.21.15.29:0/2341630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c880047c0 0x7f3c9c0a8730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.226+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 >> 172.21.15.29:0/2341630 conn(0x7f3c9c0ce9b0 msgr2=0x7f3c9c0c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.227+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 shutdown_connections 2023-10-14T16:41:04.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.227+0000 7f3ca2165640 1 -- 172.21.15.29:0/2341630 wait complete. 2023-10-14T16:41:04.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:04.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.464+0000 7fa5deabb640 1 Processor -- start 2023-10-14T16:41:04.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.464+0000 7fa5deabb640 1 -- start start 2023-10-14T16:41:04.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5deabb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806c040 0x7fa5d806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5deabb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80ad970 0x7fa5d815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5deabb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d806c960 con 0x7fa5d815ebc0 2023-10-14T16:41:04.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5deabb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d815b9d0 con 0x7fa5d806c040 2023-10-14T16:41:04.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5deabb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d815bcb0 con 0x7fa5d80ad970 2023-10-14T16:41:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5ddab9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806c040 0x7fa5d806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5ddab9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806c040 0x7fa5d806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56660/0 (socket says 172.21.15.29:56660) 2023-10-14T16:41:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5ddab9640 1 -- 172.21.15.29:0/820778108 learned_addr learned my addr 172.21.15.29:0/820778108 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:04.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5dd2b8640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80ad970 0x7fa5d815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.466+0000 7fa5ddab9640 1 -- 172.21.15.29:0/820778108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80ad970 msgr2=0x7fa5d815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.467+0000 7fa5ddab9640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80ad970 0x7fa5d815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.467+0000 7fa5ddab9640 1 -- 172.21.15.29:0/820778108 >> v1:172.21.15.186:6789/0 conn(0x7fa5d815ebc0 legacy=0x7fa5d80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.467+0000 7fa5ddab9640 1 -- 172.21.15.29:0/820778108 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d815bf90 con 0x7fa5d806c040 2023-10-14T16:41:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.468+0000 7fa5ddab9640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806c040 0x7fa5d806c420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40098a0 tx=0x7fa5d402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87e6837e3dedd35d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.468+0000 7fa5dcab7640 1 -- 172.21.15.29:0/820778108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4031640 con 0x7fa5d806c040 2023-10-14T16:41:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.468+0000 7fa5dcab7640 1 -- 172.21.15.29:0/820778108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5d40317a0 con 0x7fa5d806c040 2023-10-14T16:41:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.468+0000 7fa5dcab7640 1 -- 172.21.15.29:0/820778108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5d4031b30 con 0x7fa5d806c040 2023-10-14T16:41:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 -- 172.21.15.29:0/820778108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806c040 msgr2=0x7fa5d806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806c040 0x7fa5d806c420 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa5d40098a0 tx=0x7fa5d402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 -- 172.21.15.29:0/820778108 shutdown_connections 2023-10-14T16:41:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80ad970 0x7fa5d815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 --2- 172.21.15.29:0/820778108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806c040 0x7fa5d806c420 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 -- 172.21.15.29:0/820778108 >> 172.21.15.29:0/820778108 conn(0x7fa5d80ce6a0 msgr2=0x7fa5d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 -- 172.21.15.29:0/820778108 shutdown_connections 2023-10-14T16:41:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.469+0000 7fa5deabb640 1 -- 172.21.15.29:0/820778108 wait complete. 2023-10-14T16:41:04.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.471+0000 7fa5deabb640 1 Processor -- start 2023-10-14T16:41:04.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.471+0000 7fa5deabb640 1 -- start start 2023-10-14T16:41:04.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d806c040 0x7fa5d80c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d815ebc0 0x7fa5d8158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c66d0 con 0x7fa5d80ad970 2023-10-14T16:41:04.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c63f0 con 0x7fa5d806c040 2023-10-14T16:41:04.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5deabb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c8e00 con 0x7fa5d815ebc0 2023-10-14T16:41:04.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56676/0 (socket says 172.21.15.29:56676) 2023-10-14T16:41:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 -- 172.21.15.29:0/2651625057 learned_addr learned my addr 172.21.15.29:0/2651625057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5ddab9640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d806c040 0x7fa5d80c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 -- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d815ebc0 msgr2=0x7fa5d8158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d815ebc0 0x7fa5d8158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 -- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d806c040 msgr2=0x7fa5d80c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d806c040 0x7fa5d80c98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.472+0000 7fa5dd2b8640 1 -- 172.21.15.29:0/2651625057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d40094d0 con 0x7fa5d80ad970 2023-10-14T16:41:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5dd2b8640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c7f0 tx=0x7fa5c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8004220 con 0x7fa5d80ad970 2023-10-14T16:41:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c8004380 con 0x7fa5d80ad970 2023-10-14T16:41:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c8007840 con 0x7fa5d80ad970 2023-10-14T16:41:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d80c90e0 con 0x7fa5d80ad970 2023-10-14T16:41:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.473+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d814be20 con 0x7fa5d80ad970 2023-10-14T16:41:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.475+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d806b330 con 0x7fa5d80ad970 2023-10-14T16:41:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.476+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa5c8007bb0 con 0x7fa5d80ad970 2023-10-14T16:41:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.476+0000 7fa5ce7fc640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.477+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa5c8089480 con 0x7fa5d80ad970 2023-10-14T16:41:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.477+0000 7fa5ddab9640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.477+0000 7fa5ddab9640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4036000 tx=0x7fa5d403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.481+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5c80545c0 con 0x7fa5d80ad970 2023-10-14T16:41:04.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.686+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5d80c9600 con 0x7fa5d80ad970 2023-10-14T16:41:04.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.689+0000 7fa5ce7fc640 1 -- 172.21.15.29:0/2651625057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) v1 ==== 74+0+16493 (secure 0 0 0) 0x7fa5d80c9600 con 0x7fa5d80ad970 2023-10-14T16:41:04.691 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:04.692 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":20,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:40:26.760346+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:04.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 msgr2=0x7fa5bc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fa5d4036000 tx=0x7fa5d403c040 comp rx=0 tx=0).stop 2023-10-14T16:41:04.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 msgr2=0x7fa5d814b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa5c800c7f0 tx=0x7fa5c800ccb0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 shutdown_connections 2023-10-14T16:41:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d815ebc0 0x7fa5d8158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d80ad970 0x7fa5d814b8e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 --2- 172.21.15.29:0/2651625057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d806c040 0x7fa5d80c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 >> 172.21.15.29:0/2651625057 conn(0x7fa5d80ce6a0 msgr2=0x7fa5d815f010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 shutdown_connections 2023-10-14T16:41:04.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.692+0000 7fa5deabb640 1 -- 172.21.15.29:0/2651625057 wait complete. 2023-10-14T16:41:04.714 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:04.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.941+0000 7fcdf8a1a640 1 Processor -- start 2023-10-14T16:41:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.941+0000 7fcdf8a1a640 1 -- start start 2023-10-14T16:41:04.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.942+0000 7fcdf8a1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf40d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.942+0000 7fcdf8a1a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdf40cfc40 0x7fcdf40b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.942+0000 7fcdf8a1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf40d0970 con 0x7fcdf40b1ca0 2023-10-14T16:41:04.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.942+0000 7fcdf8a1a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf40d0ea0 con 0x7fcdf40cfc40 2023-10-14T16:41:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.942+0000 7fcdf8a1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf40d0bc0 con 0x7fcdf40b2080 2023-10-14T16:41:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdf40cfc40 0x7fcdf40b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcdf40b1ca0 0x7fcdf40ce420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32808/0 (socket says 172.21.15.29:32808) 2023-10-14T16:41:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf37fe640 1 -- 172.21.15.29:0/1156703870 learned_addr learned my addr 172.21.15.29:0/1156703870 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf2ffd640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf40d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf3fff640 1 -- 172.21.15.29:0/1156703870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 msgr2=0x7fcdf40d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf3fff640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf40d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf3fff640 1 -- 172.21.15.29:0/1156703870 >> v1:172.21.15.29:6789/0 conn(0x7fcdf40b1ca0 legacy=0x7fcdf40ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf3fff640 1 -- 172.21.15.29:0/1156703870 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf40cd940 con 0x7fcdf40cfc40 2023-10-14T16:41:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.943+0000 7fcdf2ffd640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf40d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:04.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.944+0000 7fcdf3fff640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf40b24c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0097a0 tx=0x7fcdec02b940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6e943e8dd396da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf27fc640 1 -- 172.21.15.29:0/1156703870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec030800 con 0x7fcdf40cfc40 2023-10-14T16:41:04.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf27fc640 1 -- 172.21.15.29:0/1156703870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec030960 con 0x7fcdf40cfc40 2023-10-14T16:41:04.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf27fc640 1 -- 172.21.15.29:0/1156703870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec030cf0 con 0x7fcdf40cfc40 2023-10-14T16:41:04.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/1156703870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 msgr2=0x7fcdf40b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf40b24c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fcdec0097a0 tx=0x7fcdec02b940 comp rx=0 tx=0).stop 2023-10-14T16:41:04.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/1156703870 shutdown_connections 2023-10-14T16:41:04.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf40b24c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/1156703870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf40d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/1156703870 >> 172.21.15.29:0/1156703870 conn(0x7fcdf40d2b90 msgr2=0x7fcdf40d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:04.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/1156703870 shutdown_connections 2023-10-14T16:41:04.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.945+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/1156703870 wait complete. 2023-10-14T16:41:04.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.947+0000 7fcdf8a1a640 1 Processor -- start 2023-10-14T16:41:04.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.947+0000 7fcdf8a1a640 1 -- start start 2023-10-14T16:41:04.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf40b1ca0 0x7fcdf40cae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf4158540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf415a1a0 con 0x7fcdf40cfc40 2023-10-14T16:41:04.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf4153ec0 con 0x7fcdf40b1ca0 2023-10-14T16:41:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf8a1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf415c340 con 0x7fcdf40b2080 2023-10-14T16:41:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56692/0 (socket says 172.21.15.29:56692) 2023-10-14T16:41:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 -- 172.21.15.29:0/89146346 learned_addr learned my addr 172.21.15.29:0/89146346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf37fe640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf40b1ca0 0x7fcdf40cae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 -- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 msgr2=0x7fcdf4158540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf4158540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 -- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf40b1ca0 msgr2=0x7fcdf40cae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf40b1ca0 0x7fcdf40cae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.949+0000 7fcdf3fff640 1 -- 172.21.15.29:0/89146346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdec009400 con 0x7fcdf40cfc40 2023-10-14T16:41:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdf3fff640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcdec02b980 tx=0x7fcdec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec03e070 con 0x7fcdf40cfc40 2023-10-14T16:41:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdec039b00 con 0x7fcdf40cfc40 2023-10-14T16:41:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdec039e90 con 0x7fcdf40cfc40 2023-10-14T16:41:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf4154160 con 0x7fcdf40cfc40 2023-10-14T16:41:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.950+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf40cb340 con 0x7fcdf40cfc40 2023-10-14T16:41:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.952+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf41542f0 con 0x7fcdf40cfc40 2023-10-14T16:41:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.952+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fcdec0423f0 con 0x7fcdf40cfc40 2023-10-14T16:41:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.953+0000 7fcdebfff640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 0x7fcdd406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:04.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.953+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fcdec0ab640 con 0x7fcdf40cfc40 2023-10-14T16:41:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.954+0000 7fcdf37fe640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 0x7fcdd406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.954+0000 7fcdf37fe640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 0x7fcdd406d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005c00 tx=0x7fcde4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:04.959+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcdec0766f0 con 0x7fcdf40cfc40 2023-10-14T16:41:05.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.155+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcdf404e9b0 con 0x7fcdf40cfc40 2023-10-14T16:41:05.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.156+0000 7fcdebfff640 1 -- 172.21.15.29:0/89146346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcdec079d10 con 0x7fcdf40cfc40 2023-10-14T16:41:05.158 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 msgr2=0x7fcdd406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 0x7fcdd406d330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fcde4005c00 tx=0x7fcde4009040 comp rx=0 tx=0).stop 2023-10-14T16:41:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 msgr2=0x7fcdf415d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcdec02b980 tx=0x7fcdec0046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 shutdown_connections 2023-10-14T16:41:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fcdd406ae70 0x7fcdd406d330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf40cfc40 0x7fcdf415d550 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf40b2080 0x7fcdf4158540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 --2- 172.21.15.29:0/89146346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf40b1ca0 0x7fcdf40cae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.158+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 >> 172.21.15.29:0/89146346 conn(0x7fcdf40d2b90 msgr2=0x7fcdf40b1260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.159+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 shutdown_connections 2023-10-14T16:41:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.159+0000 7fcdf8a1a640 1 -- 172.21.15.29:0/89146346 wait complete. 2023-10-14T16:41:05.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.408+0000 7f9a326f2640 1 Processor -- start 2023-10-14T16:41:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.408+0000 7f9a326f2640 1 -- start start 2023-10-14T16:41:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a2c15ad70 0x7f9a2c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a2c0c9ca0 0x7f9a2c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 0x7f9a2c0ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c0cad80 con 0x7f9a2c15ad70 2023-10-14T16:41:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a2c0cb030 con 0x7f9a2c0c9ca0 2023-10-14T16:41:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.409+0000 7f9a326f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c0c91c0 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a2c15ad70 0x7f9a2c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a2c15ad70 0x7f9a2c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56700/0 (socket says 172.21.15.29:56700) 2023-10-14T16:41:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 -- 172.21.15.29:0/2504552967 learned_addr learned my addr 172.21.15.29:0/2504552967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2b7fe640 1 --2- 172.21.15.29:0/2504552967 >> v2:172.21.15.186:3300/0 conn(0x7f9a2c0c9ca0 0x7f9a2c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a30c8f640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 0x7f9a2c0ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 -- 172.21.15.29:0/2504552967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 msgr2=0x7f9a2c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:05.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 0x7f9a2c0ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 -- 172.21.15.29:0/2504552967 >> v2:172.21.15.186:3300/0 conn(0x7f9a2c0c9ca0 msgr2=0x7f9a2c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:05.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2504552967 >> v2:172.21.15.186:3300/0 conn(0x7f9a2c0c9ca0 0x7f9a2c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a2bfff640 1 -- 172.21.15.29:0/2504552967 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a2c0c94a0 con 0x7f9a2c15ad70 2023-10-14T16:41:05.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.410+0000 7f9a30c8f640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 0x7f9a2c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:05.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.411+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c15ad70 0x7f9a2c069da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0ce870 tx=0x7f9a1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c89cf82a5071854 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.412+0000 7f9a2affd640 1 -- 172.21.15.29:0/2504552967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c02ca80 con 0x7f9a2c15ad70 2023-10-14T16:41:05.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.412+0000 7f9a2affd640 1 -- 172.21.15.29:0/2504552967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a1c02cbe0 con 0x7f9a2c15ad70 2023-10-14T16:41:05.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.412+0000 7f9a2affd640 1 -- 172.21.15.29:0/2504552967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a1c0343f0 con 0x7f9a2c15ad70 2023-10-14T16:41:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.412+0000 7f9a326f2640 1 -- 172.21.15.29:0/2504552967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c15ad70 msgr2=0x7f9a2c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.412+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c15ad70 0x7f9a2c069da0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9a2c0ce870 tx=0x7f9a1c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 -- 172.21.15.29:0/2504552967 shutdown_connections 2023-10-14T16:41:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c0ca3b0 0x7f9a2c0ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2504552967 >> v2:172.21.15.186:3300/0 conn(0x7f9a2c0c9ca0 0x7f9a2c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2504552967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c15ad70 0x7f9a2c069da0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 -- 172.21.15.29:0/2504552967 >> 172.21.15.29:0/2504552967 conn(0x7f9a2c0ce9b0 msgr2=0x7f9a2c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:05.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 -- 172.21.15.29:0/2504552967 shutdown_connections 2023-10-14T16:41:05.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.413+0000 7f9a326f2640 1 -- 172.21.15.29:0/2504552967 wait complete. 2023-10-14T16:41:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.414+0000 7f9a326f2640 1 Processor -- start 2023-10-14T16:41:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.415+0000 7f9a326f2640 1 -- start start 2023-10-14T16:41:05.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c0c9ca0 0x7f9a2c14e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 0x7f9a2c15be10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c14e730 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c1502b0 con 0x7f9a2c0c9ca0 2023-10-14T16:41:05.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a326f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a2c14f6a0 con 0x7f9a2c15ad70 2023-10-14T16:41:05.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56716/0 (socket says 172.21.15.29:56716) 2023-10-14T16:41:05.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a2b7fe640 1 -- 172.21.15.29:0/2829836826 learned_addr learned my addr 172.21.15.29:0/2829836826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:05.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c0c9ca0 0x7f9a2c14e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a2b7fe640 1 -- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 msgr2=0x7f9a2c15be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.416+0000 7f9a30c8f640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 0x7f9a2c15be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a2b7fe640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 0x7f9a2c15be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a2b7fe640 1 -- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c0c9ca0 msgr2=0x7f9a2c14e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a2b7fe640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c0c9ca0 0x7f9a2c14e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a2b7fe640 1 -- 172.21.15.29:0/2829836826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1c009b30 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a30c8f640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 0x7f9a2c15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a2b7fe640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400d800 tx=0x7f9a1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a140096a0 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a14009800 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a14009b90 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a2c14f980 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.417+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a2c14ff40 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.419+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a2c04e160 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.420+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9a14009cf0 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.421+0000 7f9a28ff9640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 0x7f9a0c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.421+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f9a14089c40 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.421+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 0x7f9a0c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.422+0000 7f9a2bfff640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 0x7f9a0c06d360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c035000 tx=0x7f9a1c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.425+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a14054d80 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.621+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9a2c0c9750 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.621+0000 7f9a28ff9640 1 -- 172.21.15.29:0/2829836826 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9a1400b440 con 0x7f9a2c0ca3b0 2023-10-14T16:41:05.624 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.624+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 msgr2=0x7f9a0c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.624+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 0x7f9a0c06d360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9a1c035000 tx=0x7f9a1c030000 comp rx=0 tx=0).stop 2023-10-14T16:41:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.624+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 msgr2=0x7f9a2c14ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.624+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9a1400d800 tx=0x7f9a1400dcc0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 shutdown_connections 2023-10-14T16:41:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9a0c06aea0 0x7f9a0c06d360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a2c15ad70 0x7f9a2c15be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a2c0ca3b0 0x7f9a2c14ebe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 --2- 172.21.15.29:0/2829836826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a2c0c9ca0 0x7f9a2c14e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 >> 172.21.15.29:0/2829836826 conn(0x7f9a2c0ce9b0 msgr2=0x7f9a2c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 shutdown_connections 2023-10-14T16:41:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.625+0000 7f9a326f2640 1 -- 172.21.15.29:0/2829836826 wait complete. 2023-10-14T16:41:05.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:05.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.877+0000 7f4daa460640 1 Processor -- start 2023-10-14T16:41:05.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.878+0000 7f4daa460640 1 -- start start 2023-10-14T16:41:05.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.879+0000 7f4daa460640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4da40c9060 0x7f4da406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.879+0000 7f4daa460640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da406c9e0 0x7f4da406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4daa460640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4da40ca560 con 0x7f4da40c9440 2023-10-14T16:41:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4daa460640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da40c9950 con 0x7f4da406c9e0 2023-10-14T16:41:05.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4daa460640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4da40c9c30 con 0x7f4da40c9060 2023-10-14T16:41:05.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da406c9e0 0x7f4da406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4da406c9e0 0x7f4da406f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56724/0 (socket says 172.21.15.29:56724) 2023-10-14T16:41:05.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 -- 172.21.15.29:0/1013575899 learned_addr learned my addr 172.21.15.29:0/1013575899 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da945e640 1 --2- 172.21.15.29:0/1013575899 >> v2:172.21.15.186:3300/0 conn(0x7f4da40c9060 0x7f4da406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 -- 172.21.15.29:0/1013575899 >> v2:172.21.15.186:3300/0 conn(0x7f4da40c9060 msgr2=0x7f4da406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/1013575899 >> v2:172.21.15.186:3300/0 conn(0x7f4da40c9060 0x7f4da406f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 -- 172.21.15.29:0/1013575899 >> v1:172.21.15.186:6789/0 conn(0x7f4da40c9440 legacy=0x7f4da4164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.880+0000 7f4da8c5d640 1 -- 172.21.15.29:0/1013575899 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da40c9f10 con 0x7f4da406c9e0 2023-10-14T16:41:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.881+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/1013575899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da406f980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4da406d490 tx=0x7f4d9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455e0e6d2e934042 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.882+0000 7f4da3fff640 1 -- 172.21.15.29:0/1013575899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98031870 con 0x7f4da406c9e0 2023-10-14T16:41:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.882+0000 7f4da3fff640 1 -- 172.21.15.29:0/1013575899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d980319d0 con 0x7f4da406c9e0 2023-10-14T16:41:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.882+0000 7f4da3fff640 1 -- 172.21.15.29:0/1013575899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d98031d60 con 0x7f4da406c9e0 2023-10-14T16:41:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.882+0000 7f4daa460640 1 -- 172.21.15.29:0/1013575899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 msgr2=0x7f4da406f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.882+0000 7f4daa460640 1 --2- 172.21.15.29:0/1013575899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da406f980 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f4da406d490 tx=0x7f4d9802b910 comp rx=0 tx=0).stop 2023-10-14T16:41:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 -- 172.21.15.29:0/1013575899 shutdown_connections 2023-10-14T16:41:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 --2- 172.21.15.29:0/1013575899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da406f980 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 --2- 172.21.15.29:0/1013575899 >> v2:172.21.15.186:3300/0 conn(0x7f4da40c9060 0x7f4da406f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 -- 172.21.15.29:0/1013575899 >> 172.21.15.29:0/1013575899 conn(0x7f4da40ce6a0 msgr2=0x7f4da40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:05.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 -- 172.21.15.29:0/1013575899 shutdown_connections 2023-10-14T16:41:05.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.883+0000 7f4daa460640 1 -- 172.21.15.29:0/1013575899 wait complete. 2023-10-14T16:41:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.885+0000 7f4daa460640 1 Processor -- start 2023-10-14T16:41:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.885+0000 7f4daa460640 1 -- start start 2023-10-14T16:41:05.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 0x7f4da40c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 0x7f4da40cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da41f6630 con 0x7f4da406c9e0 2023-10-14T16:41:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40c79c0 con 0x7f4da40c9440 2023-10-14T16:41:05.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4daa460640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4da40c60e0 con 0x7f4da40c9060 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4da945e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4da945e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56736/0 (socket says 172.21.15.29:56736) 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.886+0000 7f4da945e640 1 -- 172.21.15.29:0/476322222 learned_addr learned my addr 172.21.15.29:0/476322222 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 -- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 msgr2=0x7f4da40c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da9c5f640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 0x7f4da40cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 0x7f4da40c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 0x7f4da40c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 -- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 msgr2=0x7f4da40cea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 0x7f4da40cea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:05.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 -- 172.21.15.29:0/476322222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980094d0 con 0x7f4da406c9e0 2023-10-14T16:41:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da9c5f640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 0x7f4da40cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 0x7f4da40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da945e640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400ce10 tx=0x7f4d9400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94003b90 con 0x7f4da406c9e0 2023-10-14T16:41:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d94003cf0 con 0x7f4da406c9e0 2023-10-14T16:41:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d94003ea0 con 0x7f4da406c9e0 2023-10-14T16:41:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.887+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4da40c63c0 con 0x7f4da406c9e0 2023-10-14T16:41:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.888+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4da40c6980 con 0x7f4da406c9e0 2023-10-14T16:41:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.889+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4da40c6b70 con 0x7f4da406c9e0 2023-10-14T16:41:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.890+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f4d94004240 con 0x7f4da406c9e0 2023-10-14T16:41:05.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.891+0000 7f4da1ffb640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 0x7f4d9006d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.891+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f4d94012070 con 0x7f4da406c9e0 2023-10-14T16:41:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.891+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 0x7f4d9006d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.892+0000 7f4da8c5d640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 0x7f4d9006d360 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4d98036000 tx=0x7f4d9803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:05.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:05.895+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4d940550b0 con 0x7f4da406c9e0 2023-10-14T16:41:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.104+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4da404e9b0 con 0x7f4da406c9e0 2023-10-14T16:41:06.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.105+0000 7f4da1ffb640 1 -- 172.21.15.29:0/476322222 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4d940586d0 con 0x7f4da406c9e0 2023-10-14T16:41:06.107 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 msgr2=0x7f4d9006d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 0x7f4d9006d360 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f4d98036000 tx=0x7f4d9803c040 comp rx=0 tx=0).stop 2023-10-14T16:41:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 msgr2=0x7f4da40c7480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f4d9400ce10 tx=0x7f4d9400a660 comp rx=0 tx=0).stop 2023-10-14T16:41:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 shutdown_connections 2023-10-14T16:41:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f4d9006aea0 0x7f4d9006d360 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4da40c9440 0x7f4da40cea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4da40c9060 0x7f4da40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 --2- 172.21.15.29:0/476322222 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4da406c9e0 0x7f4da40c7480 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 >> 172.21.15.29:0/476322222 conn(0x7f4da40ce6a0 msgr2=0x7f4da40c2090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 shutdown_connections 2023-10-14T16:41:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.108+0000 7f4daa460640 1 -- 172.21.15.29:0/476322222 wait complete. 2023-10-14T16:41:06.141 INFO:tasks.cephfs_test_runner:Starting test: test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test concurrent snap creates in same file-system without db issues 2023-10-14T16:41:06.142 INFO:tasks.cephfs_test_runner:test_concurrent_snap_creates (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:41:06.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:06.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.367+0000 7f699978a640 1 Processor -- start 2023-10-14T16:41:06.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.368+0000 7f699978a640 1 -- start start 2023-10-14T16:41:06.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.369+0000 7f699978a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f699415ae80 0x7f699415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.369+0000 7f699978a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69940ce390 0x7f6994070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.369+0000 7f699978a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f699415ed60 con 0x7f699415aaa0 2023-10-14T16:41:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f699978a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f699406b810 con 0x7f69940ce390 2023-10-14T16:41:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f699978a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69940707e0 con 0x7f699415ae80 2023-10-14T16:41:06.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f6992ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f699415aaa0 0x7f69940cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32816/0 (socket says 172.21.15.29:32816) 2023-10-14T16:41:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f6992ffd640 1 -- 172.21.15.29:0/3747258901 learned_addr learned my addr 172.21.15.29:0/3747258901 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:06.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69937fe640 1 --2- 172.21.15.29:0/3747258901 >> v2:172.21.15.29:3300/0 conn(0x7f69940ce390 0x7f6994070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f698bfff640 1 --2- 172.21.15.29:0/3747258901 >> v2:172.21.15.186:3300/0 conn(0x7f699415ae80 0x7f699415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69937fe640 1 -- 172.21.15.29:0/3747258901 >> v2:172.21.15.186:3300/0 conn(0x7f699415ae80 msgr2=0x7f699415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:06.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69937fe640 1 --2- 172.21.15.29:0/3747258901 >> v2:172.21.15.186:3300/0 conn(0x7f699415ae80 0x7f699415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69927fc640 1 -- 172.21.15.29:0/3747258901 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296382668 0 0) 0x7f699415ed60 con 0x7f699415aaa0 2023-10-14T16:41:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69937fe640 1 -- 172.21.15.29:0/3747258901 >> v1:172.21.15.29:6789/0 conn(0x7f699415aaa0 legacy=0x7f69940cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.370+0000 7f69937fe640 1 -- 172.21.15.29:0/3747258901 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699415ef90 con 0x7f69940ce390 2023-10-14T16:41:06.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f69937fe640 1 --2- 172.21.15.29:0/3747258901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940ce390 0x7f6994070f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f69940c59c0 tx=0x7f698c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6655d59e01214ae3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f69927fc640 1 -- 172.21.15.29:0/3747258901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c02cc70 con 0x7f69940ce390 2023-10-14T16:41:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f69927fc640 1 -- 172.21.15.29:0/3747258901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c02cdd0 con 0x7f69940ce390 2023-10-14T16:41:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f69927fc640 1 -- 172.21.15.29:0/3747258901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c034660 con 0x7f69940ce390 2023-10-14T16:41:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f699978a640 1 -- 172.21.15.29:0/3747258901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940ce390 msgr2=0x7f6994070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.372+0000 7f699978a640 1 --2- 172.21.15.29:0/3747258901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940ce390 0x7f6994070f50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f69940c59c0 tx=0x7f698c02bec0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 -- 172.21.15.29:0/3747258901 shutdown_connections 2023-10-14T16:41:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 --2- 172.21.15.29:0/3747258901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940ce390 0x7f6994070f50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 --2- 172.21.15.29:0/3747258901 >> v2:172.21.15.186:3300/0 conn(0x7f699415ae80 0x7f699415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 -- 172.21.15.29:0/3747258901 >> 172.21.15.29:0/3747258901 conn(0x7f69940cb410 msgr2=0x7f69940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 -- 172.21.15.29:0/3747258901 shutdown_connections 2023-10-14T16:41:06.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.373+0000 7f699978a640 1 -- 172.21.15.29:0/3747258901 wait complete. 2023-10-14T16:41:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.375+0000 7f699978a640 1 Processor -- start 2023-10-14T16:41:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.375+0000 7f699978a640 1 -- start start 2023-10-14T16:41:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940ce390 0x7f69940c4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 0x7f6994153ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c4ad0 con 0x7f699415ae80 2023-10-14T16:41:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c3e50 con 0x7f69940ce390 2023-10-14T16:41:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f699978a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c4130 con 0x7f699415aaa0 2023-10-14T16:41:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56746/0 (socket says 172.21.15.29:56746) 2023-10-14T16:41:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 -- 172.21.15.29:0/1207793687 learned_addr learned my addr 172.21.15.29:0/1207793687 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f698bfff640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 0x7f6994153ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 -- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 msgr2=0x7f6994153ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 0x7f6994153ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 -- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940ce390 msgr2=0x7f69940c4ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940ce390 0x7f69940c4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f69937fe640 1 -- 172.21.15.29:0/1207793687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c009a70 con 0x7f699415ae80 2023-10-14T16:41:06.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.376+0000 7f698bfff640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 0x7f6994153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f69937fe640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f698c02c4d0 tx=0x7f698c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c03e610 con 0x7f699415ae80 2023-10-14T16:41:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c03e790 con 0x7f699415ae80 2023-10-14T16:41:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c03eb40 con 0x7f699415ae80 2023-10-14T16:41:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69940c4410 con 0x7f699415ae80 2023-10-14T16:41:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.377+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69940c3a20 con 0x7f699415ae80 2023-10-14T16:41:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.378+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69940c3bb0 con 0x7f699415ae80 2023-10-14T16:41:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.379+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f698c03b050 con 0x7f699415ae80 2023-10-14T16:41:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.380+0000 7f698b7fe640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 0x7f697406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.380+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f698c0ab8e0 con 0x7f699415ae80 2023-10-14T16:41:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.381+0000 7f6992ffd640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 0x7f697406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.381+0000 7f6992ffd640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 0x7f697406d360 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6984007fd0 tx=0x7f6984007920 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.385+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f698c076910 con 0x7f699415ae80 2023-10-14T16:41:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.583+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f69940c4c10 con 0x7f699415ae80 2023-10-14T16:41:06.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.586+0000 7f698b7fe640 1 -- 172.21.15.29:0/1207793687 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16742 (secure 0 0 0) 0x7f69940c4c10 con 0x7f699415ae80 2023-10-14T16:41:06.588 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:06.588 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":21,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:05.978543+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 msgr2=0x7f697406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 0x7f697406d360 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6984007fd0 tx=0x7f6984007920 comp rx=0 tx=0).stop 2023-10-14T16:41:06.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 msgr2=0x7f69940c34e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f698c02c4d0 tx=0x7f698c004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 shutdown_connections 2023-10-14T16:41:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406aea0 0x7f697406d360 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f699415ae80 0x7f69940c34e0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f699415aaa0 0x7f6994153ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 --2- 172.21.15.29:0/1207793687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940ce390 0x7f69940c4ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 >> 172.21.15.29:0/1207793687 conn(0x7f69940cb410 msgr2=0x7f699415a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 shutdown_connections 2023-10-14T16:41:06.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.589+0000 7f699978a640 1 -- 172.21.15.29:0/1207793687 wait complete. 2023-10-14T16:41:06.611 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:06.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.835+0000 7fbe214d1640 1 Processor -- start 2023-10-14T16:41:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.835+0000 7fbe214d1640 1 -- start start 2023-10-14T16:41:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe214d1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbe1c159ad0 0x7fbe1c070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe214d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe1c0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe214d1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe1c0cd260 con 0x7fbe1c15d650 2023-10-14T16:41:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe214d1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe1c071470 con 0x7fbe1c159ad0 2023-10-14T16:41:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe214d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0ca7b0 con 0x7fbe1c0c9de0 2023-10-14T16:41:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe13fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe1c15d650 0x7fbe1c06aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32830/0 (socket says 172.21.15.29:32830) 2023-10-14T16:41:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.836+0000 7fbe13fff640 1 -- 172.21.15.29:0/1648581111 learned_addr learned my addr 172.21.15.29:0/1648581111 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe20cd0640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe1c0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1648581111 >> v2:172.21.15.186:3300/0 conn(0x7fbe1c159ad0 0x7fbe1c070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289189977 0 0) 0x7fbe1c0cd260 con 0x7fbe1c15d650 2023-10-14T16:41:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe08003520 con 0x7fbe1c15d650 2023-10-14T16:41:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2729865932 0 0) 0x7fbe08003520 con 0x7fbe1c15d650 2023-10-14T16:41:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 msgr2=0x7fbe1c0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe1c0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 >> v2:172.21.15.186:3300/0 conn(0x7fbe1c159ad0 msgr2=0x7fbe1c070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/1648581111 >> v2:172.21.15.186:3300/0 conn(0x7fbe1c159ad0 0x7fbe1c070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c15de30 con 0x7fbe1c15d650 2023-10-14T16:41:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fbe04001670 con 0x7fbe1c15d650 2023-10-14T16:41:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 >> v1:172.21.15.29:6789/0 conn(0x7fbe1c15d650 legacy=0x7fbe1c06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe08003970 0x7fbe08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.837+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1b7fe640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe08004d00 0x7fbe08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c071470 con 0x7fbe08003970 2023-10-14T16:41:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0ca7b0 con 0x7fbe08004d00 2023-10-14T16:41:06.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe0800b980 con 0x7fbe1c0c9de0 2023-10-14T16:41:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe08003970 0x7fbe08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe13fff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 -- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 msgr2=0x7fbe080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 -- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe08004d00 msgr2=0x7fbe08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe08004d00 0x7fbe08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe1bfff640 1 -- 172.21.15.29:0/1648581111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe08003520 con 0x7fbe08003970 2023-10-14T16:41:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.838+0000 7fbe13fff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe1c0c9de0 0x7fbe080047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.839+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe08003970 0x7fbe08003d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009ac0 tx=0x7fbe080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e370c0ec71537ec server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c02c710 con 0x7fbe08003970 2023-10-14T16:41:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe0c02c870 con 0x7fbe08003970 2023-10-14T16:41:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe1b7fe640 1 -- 172.21.15.29:0/1648581111 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe0c02cc00 con 0x7fbe08003970 2023-10-14T16:41:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 -- 172.21.15.29:0/1648581111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe08003970 msgr2=0x7fbe08003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1648581111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe08003970 0x7fbe08003d90 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c009ac0 tx=0x7fbe080057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe20cd0640 1 -- 172.21.15.29:0/1648581111 reap_dead start 2023-10-14T16:41:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 -- 172.21.15.29:0/1648581111 shutdown_connections 2023-10-14T16:41:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 -- 172.21.15.29:0/1648581111 >> 172.21.15.29:0/1648581111 conn(0x7fbe1c0b0170 msgr2=0x7fbe1c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 -- 172.21.15.29:0/1648581111 shutdown_connections 2023-10-14T16:41:06.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.840+0000 7fbe214d1640 1 -- 172.21.15.29:0/1648581111 wait complete. 2023-10-14T16:41:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.842+0000 7fbe214d1640 1 Processor -- start 2023-10-14T16:41:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.842+0000 7fbe214d1640 1 -- start start 2023-10-14T16:41:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe08003970 0x7fbe1c0afa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe1c15b400 0x7fbe1c15b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c0affc0 con 0x7fbe1c145880 2023-10-14T16:41:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c151900 con 0x7fbe1c15b400 2023-10-14T16:41:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.843+0000 7fbe214d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c152170 con 0x7fbe08003970 2023-10-14T16:41:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56768/0 (socket says 172.21.15.29:56768) 2023-10-14T16:41:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 -- 172.21.15.29:0/1543893964 learned_addr learned my addr 172.21.15.29:0/1543893964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 -- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe08003970 msgr2=0x7fbe1c0afa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe08003970 0x7fbe1c0afa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 -- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe1c15b400 msgr2=0x7fbe1c15b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe1c15b400 0x7fbe1c15b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 -- 172.21.15.29:0/1543893964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c15a960 con 0x7fbe1c145880 2023-10-14T16:41:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.844+0000 7fbe13fff640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400d7e0 tx=0x7fbe0400dca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.845+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe040074e0 con 0x7fbe1c145880 2023-10-14T16:41:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.845+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe04007640 con 0x7fbe1c145880 2023-10-14T16:41:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.845+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe04004a00 con 0x7fbe1c145880 2023-10-14T16:41:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.845+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c15ac00 con 0x7fbe1c145880 2023-10-14T16:41:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.845+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c152410 con 0x7fbe1c145880 2023-10-14T16:41:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.846+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c15b120 con 0x7fbe1c145880 2023-10-14T16:41:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.847+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbe04004b60 con 0x7fbe1c145880 2023-10-14T16:41:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.848+0000 7fbe197fa640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 0x7fbdfc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.849+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbe04088fe0 con 0x7fbe1c145880 2023-10-14T16:41:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.849+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 0x7fbdfc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.849+0000 7fbe1bfff640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 0x7fbdfc06d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c036000 tx=0x7fbe0c02e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:06.852+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe1c15b120 con 0x7fbe1c145880 2023-10-14T16:41:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.062+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbe1c15b120 con 0x7fbe1c145880 2023-10-14T16:41:07.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.063+0000 7fbe197fa640 1 -- 172.21.15.29:0/1543893964 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbe1c15b120 con 0x7fbe1c145880 2023-10-14T16:41:07.065 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.065+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 msgr2=0x7fbdfc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.065+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 0x7fbdfc06d330 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fbe0c036000 tx=0x7fbe0c02e000 comp rx=0 tx=0).stop 2023-10-14T16:41:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.065+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 msgr2=0x7fbe1c0c40e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.065+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbe0400d7e0 tx=0x7fbe0400dca0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 shutdown_connections 2023-10-14T16:41:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbdfc06ae70 0x7fbdfc06d330 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe1c15b400 0x7fbe1c15b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe1c145880 0x7fbe1c0c40e0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 --2- 172.21.15.29:0/1543893964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe08003970 0x7fbe1c0afa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 >> 172.21.15.29:0/1543893964 conn(0x7fbe1c0b0170 msgr2=0x7fbe1c15ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 shutdown_connections 2023-10-14T16:41:07.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.066+0000 7fbe214d1640 1 -- 172.21.15.29:0/1543893964 wait complete. 2023-10-14T16:41:07.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.302+0000 7f7785021640 1 Processor -- start 2023-10-14T16:41:07.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.302+0000 7f7785021640 1 -- start start 2023-10-14T16:41:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f7785021640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f778006a4b0 0x7f77800ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f7785021640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77800cb030 con 0x7f778015ad70 2023-10-14T16:41:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f7785021640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77800c91c0 con 0x7f77800c9ca0 2023-10-14T16:41:07.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f7785021640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77800c94a0 con 0x7f778006a4b0 2023-10-14T16:41:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f777ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f778015ad70 0x7f7780069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32834/0 (socket says 172.21.15.29:32834) 2023-10-14T16:41:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f777ed9d640 1 -- 172.21.15.29:0/1253269069 learned_addr learned my addr 172.21.15.29:0/1253269069 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:07.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.304+0000 7f777e59c640 1 --2- 172.21.15.29:0/1253269069 >> v2:172.21.15.29:3300/0 conn(0x7f778006a4b0 0x7f77800ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.305+0000 7f777dd9b640 1 -- 172.21.15.29:0/1253269069 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 23578955 0 0) 0x7f77800cb030 con 0x7f778015ad70 2023-10-14T16:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.305+0000 7f777dd9b640 1 -- 172.21.15.29:0/1253269069 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7768003520 con 0x7f778015ad70 2023-10-14T16:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.305+0000 7f777e59c640 1 -- 172.21.15.29:0/1253269069 >> v1:172.21.15.186:6789/0 conn(0x7f77800c9ca0 legacy=0x7f77800ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:07.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.305+0000 7f777e59c640 1 -- 172.21.15.29:0/1253269069 >> v1:172.21.15.29:6789/0 conn(0x7f778015ad70 legacy=0x7f7780069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.305+0000 7f777e59c640 1 -- 172.21.15.29:0/1253269069 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800c9780 con 0x7f778006a4b0 2023-10-14T16:41:07.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.306+0000 7f777e59c640 1 --2- 172.21.15.29:0/1253269069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f778006a4b0 0x7f77800ca3b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f77740097a0 tx=0x7f777402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a73efb8e12f119ca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.306+0000 7f777dd9b640 1 -- 172.21.15.29:0/1253269069 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774031830 con 0x7f778006a4b0 2023-10-14T16:41:07.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.306+0000 7f777dd9b640 1 -- 172.21.15.29:0/1253269069 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7774031990 con 0x7f778006a4b0 2023-10-14T16:41:07.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.306+0000 7f777dd9b640 1 -- 172.21.15.29:0/1253269069 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7774031d20 con 0x7f778006a4b0 2023-10-14T16:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 -- 172.21.15.29:0/1253269069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f778006a4b0 msgr2=0x7f77800ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 --2- 172.21.15.29:0/1253269069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f778006a4b0 0x7f77800ca3b0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f77740097a0 tx=0x7f777402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 -- 172.21.15.29:0/1253269069 shutdown_connections 2023-10-14T16:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 --2- 172.21.15.29:0/1253269069 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f778006a4b0 0x7f77800ca3b0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 -- 172.21.15.29:0/1253269069 >> 172.21.15.29:0/1253269069 conn(0x7f77800ce9b0 msgr2=0x7f77800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:07.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 -- 172.21.15.29:0/1253269069 shutdown_connections 2023-10-14T16:41:07.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.307+0000 7f7785021640 1 -- 172.21.15.29:0/1253269069 wait complete. 2023-10-14T16:41:07.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.309+0000 7f7785021640 1 Processor -- start 2023-10-14T16:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.309+0000 7f7785021640 1 -- start start 2023-10-14T16:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f778006a4b0 0x7f77801f3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f778015ad70 0x7f7780158ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77801f4410 con 0x7f77800c9ca0 2023-10-14T16:41:07.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7780159d40 con 0x7f778006a4b0 2023-10-14T16:41:07.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f7785021640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77801f0fe0 con 0x7f778015ad70 2023-10-14T16:41:07.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f777e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f777e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56790/0 (socket says 172.21.15.29:56790) 2023-10-14T16:41:07.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f777e59c640 1 -- 172.21.15.29:0/215478128 learned_addr learned my addr 172.21.15.29:0/215478128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:07.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f777ed9d640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f778006a4b0 0x7f77801f3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.310+0000 7f777e59c640 1 -- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f778015ad70 msgr2=0x7f7780158ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f777e59c640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f778015ad70 0x7f7780158ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f777e59c640 1 -- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f778006a4b0 msgr2=0x7f77801f3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f777e59c640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f778006a4b0 0x7f77801f3ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f777e59c640 1 -- 172.21.15.29:0/215478128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7774009400 con 0x7f77800c9ca0 2023-10-14T16:41:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f777e59c640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f777402bee0 tx=0x7f77740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777403a6f0 con 0x7f77800c9ca0 2023-10-14T16:41:07.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777403a870 con 0x7f77800c9ca0 2023-10-14T16:41:07.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777403ac20 con 0x7f77800c9ca0 2023-10-14T16:41:07.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.311+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77800a93d0 con 0x7f77800c9ca0 2023-10-14T16:41:07.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.312+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77801f1530 con 0x7f77800c9ca0 2023-10-14T16:41:07.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.313+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f778004e160 con 0x7f77800c9ca0 2023-10-14T16:41:07.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.314+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f777404c3b0 con 0x7f77800c9ca0 2023-10-14T16:41:07.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.315+0000 7f77677fe640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 0x7f776c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.315+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f77740ab460 con 0x7f77800c9ca0 2023-10-14T16:41:07.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.315+0000 7f777ed9d640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 0x7f776c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.315+0000 7f777ed9d640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 0x7f776c06d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f77700011e0 tx=0x7f7770009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.319+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f77740763f0 con 0x7f77800c9ca0 2023-10-14T16:41:07.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.514+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f77800c8f00 con 0x7f77800c9ca0 2023-10-14T16:41:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.515+0000 7f77677fe640 1 -- 172.21.15.29:0/215478128 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7774079a10 con 0x7f77800c9ca0 2023-10-14T16:41:07.517 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:07.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 msgr2=0x7f776c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 0x7f776c06d330 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f77700011e0 tx=0x7f7770009040 comp rx=0 tx=0).stop 2023-10-14T16:41:07.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 msgr2=0x7f77800c4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f777402bee0 tx=0x7f77740046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 shutdown_connections 2023-10-14T16:41:07.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f776c06ae70 0x7f776c06d330 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f778015ad70 0x7f7780158ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f77800c9ca0 0x7f77800c4900 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 --2- 172.21.15.29:0/215478128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f778006a4b0 0x7f77801f3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 >> 172.21.15.29:0/215478128 conn(0x7f77800ce9b0 msgr2=0x7f77800afd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:07.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 shutdown_connections 2023-10-14T16:41:07.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.518+0000 7f7785021640 1 -- 172.21.15.29:0/215478128 wait complete. 2023-10-14T16:41:07.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:07.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.762+0000 7f8e8c947640 1 Processor -- start 2023-10-14T16:41:07.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.762+0000 7f8e8c947640 1 -- start start 2023-10-14T16:41:07.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.763+0000 7f8e8c947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e880cc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.763+0000 7f8e8c947640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e880b4740 0x7f8e880b4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.763+0000 7f8e8c947640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e880b3730 con 0x7f8e8815aa60 2023-10-14T16:41:07.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.763+0000 7f8e8c947640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e880b3980 con 0x7f8e880b4740 2023-10-14T16:41:07.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.763+0000 7f8e8c947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880b3c60 con 0x7f8e880cc2e0 2023-10-14T16:41:07.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e877fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8e8815aa60 0x7f8e880af820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32844/0 (socket says 172.21.15.29:32844) 2023-10-14T16:41:07.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e877fe640 1 -- 172.21.15.29:0/294426229 learned_addr learned my addr 172.21.15.29:0/294426229 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e87fff640 1 --2- 172.21.15.29:0/294426229 >> v2:172.21.15.186:3300/0 conn(0x7f8e880b4740 0x7f8e880b4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103117922 0 0) 0x7f8e880b3730 con 0x7f8e8815aa60 2023-10-14T16:41:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e74003520 con 0x7f8e8815aa60 2023-10-14T16:41:07.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.764+0000 7f8e86ffd640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e880cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3220577259 0 0) 0x7f8e74003520 con 0x7f8e8815aa60 2023-10-14T16:41:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 msgr2=0x7f8e880cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e880cc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 >> v2:172.21.15.186:3300/0 conn(0x7f8e880b4740 msgr2=0x7f8e880b4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:07.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 --2- 172.21.15.29:0/294426229 >> v2:172.21.15.186:3300/0 conn(0x7f8e880b4740 0x7f8e880b4bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e880b3f40 con 0x7f8e8815aa60 2023-10-14T16:41:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8e780019f0 con 0x7f8e8815aa60 2023-10-14T16:41:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e86ffd640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e880cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 >> v1:172.21.15.29:6789/0 conn(0x7f8e8815aa60 legacy=0x7f8e880af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e74003970 0x7f8e74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74004d00 0x7f8e74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880b3980 con 0x7f8e74003970 2023-10-14T16:41:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e880b3c60 con 0x7f8e74004d00 2023-10-14T16:41:07.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7400b980 con 0x7f8e880cc2e0 2023-10-14T16:41:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.765+0000 7f8e877fe640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e74003970 0x7f8e74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e87fff640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74004d00 0x7f8e74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e877fe640 1 -- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 msgr2=0x7f8e740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e877fe640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880cc2e0 0x7f8e740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e877fe640 1 -- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74004d00 msgr2=0x7f8e74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e877fe640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74004d00 0x7f8e74005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e877fe640 1 -- 172.21.15.29:0/294426229 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e74003520 con 0x7f8e74003970 2023-10-14T16:41:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.766+0000 7f8e87fff640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74004d00 0x7f8e74005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e877fe640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e74003970 0x7f8e74003d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800a790 tx=0x7f8e740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81889c18595fb759 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e78034630 con 0x7f8e74003970 2023-10-14T16:41:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7802f070 con 0x7f8e74003970 2023-10-14T16:41:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e867fc640 1 -- 172.21.15.29:0/294426229 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e780349f0 con 0x7f8e74003970 2023-10-14T16:41:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e8c947640 1 -- 172.21.15.29:0/294426229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e74003970 msgr2=0x7f8e74003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.767+0000 7f8e8c947640 1 --2- 172.21.15.29:0/294426229 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e74003970 0x7f8e74003d90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f8e7800a790 tx=0x7f8e740057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.768+0000 7f8e87fff640 1 -- 172.21.15.29:0/294426229 reap_dead start 2023-10-14T16:41:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.768+0000 7f8e8c947640 1 -- 172.21.15.29:0/294426229 shutdown_connections 2023-10-14T16:41:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.768+0000 7f8e8c947640 1 -- 172.21.15.29:0/294426229 >> 172.21.15.29:0/294426229 conn(0x7f8e88065ed0 msgr2=0x7f8e880662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.768+0000 7f8e8c947640 1 -- 172.21.15.29:0/294426229 shutdown_connections 2023-10-14T16:41:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.768+0000 7f8e8c947640 1 -- 172.21.15.29:0/294426229 wait complete. 2023-10-14T16:41:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.770+0000 7f8e8c947640 1 Processor -- start 2023-10-14T16:41:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.770+0000 7f8e8c947640 1 -- start start 2023-10-14T16:41:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74003970 0x7f8e880c6eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880aec00 0x7f8e880af080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881f2060 con 0x7f8e8815cf60 2023-10-14T16:41:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881f22d0 con 0x7f8e74003970 2023-10-14T16:41:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e8c947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e881f25b0 con 0x7f8e880aec00 2023-10-14T16:41:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56804/0 (socket says 172.21.15.29:56804) 2023-10-14T16:41:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 -- 172.21.15.29:0/102518996 learned_addr learned my addr 172.21.15.29:0/102518996 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 -- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880aec00 msgr2=0x7f8e880af080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880aec00 0x7f8e880af080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 -- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74003970 msgr2=0x7f8e880c6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74003970 0x7f8e880c6eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.771+0000 7f8e86ffd640 1 -- 172.21.15.29:0/102518996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e881f2890 con 0x7f8e8815cf60 2023-10-14T16:41:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e86ffd640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000e7e0 tx=0x7f8e7000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000c650 con 0x7f8e8815cf60 2023-10-14T16:41:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e7000c7b0 con 0x7f8e8815cf60 2023-10-14T16:41:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000cb20 con 0x7f8e8815cf60 2023-10-14T16:41:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e8815d900 con 0x7f8e8815cf60 2023-10-14T16:41:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.772+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e881f63c0 con 0x7f8e8815cf60 2023-10-14T16:41:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.773+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e8804e9b0 con 0x7f8e8815cf60 2023-10-14T16:41:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.774+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8e7000cc80 con 0x7f8e8815cf60 2023-10-14T16:41:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.775+0000 7f8e7ffff640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 0x7f8e6806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.775+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8e70089420 con 0x7f8e8815cf60 2023-10-14T16:41:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.776+0000 7f8e877fe640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 0x7f8e6806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.776+0000 7f8e877fe640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 0x7f8e6806d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e78034000 tx=0x7f8e78007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.780+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e700543b0 con 0x7f8e8815cf60 2023-10-14T16:41:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.966+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8e8815a160 con 0x7f8e8815cf60 2023-10-14T16:41:07.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.967+0000 7f8e7ffff640 1 -- 172.21.15.29:0/102518996 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e700579d0 con 0x7f8e8815cf60 2023-10-14T16:41:07.969 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 msgr2=0x7f8e6806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 0x7f8e6806d330 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f8e78034000 tx=0x7f8e78007000 comp rx=0 tx=0).stop 2023-10-14T16:41:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 msgr2=0x7f8e8815d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000e7e0 tx=0x7f8e7000eca0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 shutdown_connections 2023-10-14T16:41:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8e6806ae70 0x7f8e6806d330 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e880aec00 0x7f8e880af080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e8815cf60 0x7f8e8815d3c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 --2- 172.21.15.29:0/102518996 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e74003970 0x7f8e880c6eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.969+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 >> 172.21.15.29:0/102518996 conn(0x7f8e88065ed0 msgr2=0x7f8e880cf770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.970+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 shutdown_connections 2023-10-14T16:41:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:07.970+0000 7f8e8c947640 1 -- 172.21.15.29:0/102518996 wait complete. 2023-10-14T16:41:07.990 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-14T16:41:08.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.205+0000 7f60bf58e640 1 Processor -- start 2023-10-14T16:41:08.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.205+0000 7f60bf58e640 1 -- start start 2023-10-14T16:41:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.206+0000 7f60bf58e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f60b815aad0 0x7f60b80cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.206+0000 7f60bf58e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60b806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.206+0000 7f60bf58e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b80b4620 con 0x7f60b80712c0 2023-10-14T16:41:08.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.206+0000 7f60bf58e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b80b4340 con 0x7f60b815aad0 2023-10-14T16:41:08.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.206+0000 7f60bf58e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b806f520 con 0x7f60b8070d70 2023-10-14T16:41:08.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60bed8d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f60b80712c0 0x7f60b8070000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32860/0 (socket says 172.21.15.29:32860) 2023-10-14T16:41:08.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60bed8d640 1 -- 172.21.15.29:0/2197648553 learned_addr learned my addr 172.21.15.29:0/2197648553 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:08.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60be58c640 1 --2- 172.21.15.29:0/2197648553 >> v2:172.21.15.186:3300/0 conn(0x7f60b815aad0 0x7f60b80cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60bdd8b640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60b806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549465250 0 0) 0x7f60b80b4620 con 0x7f60b80712c0 2023-10-14T16:41:08.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.207+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60a4003520 con 0x7f60b80712c0 2023-10-14T16:41:08.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2476363362 0 0) 0x7f60a4003520 con 0x7f60b80712c0 2023-10-14T16:41:08.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 msgr2=0x7f60b806bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60b806bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 >> v2:172.21.15.186:3300/0 conn(0x7f60b815aad0 msgr2=0x7f60b80cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 --2- 172.21.15.29:0/2197648553 >> v2:172.21.15.186:3300/0 conn(0x7f60b815aad0 0x7f60b80cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b806f800 con 0x7f60b80712c0 2023-10-14T16:41:08.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f60b00019f0 con 0x7f60b80712c0 2023-10-14T16:41:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 >> v1:172.21.15.29:6789/0 conn(0x7f60b80712c0 legacy=0x7f60b8070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60a4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4003f40 0x7f60a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4004d00 0x7f60a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b80b4340 con 0x7f60a4004d00 2023-10-14T16:41:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b806f520 con 0x7f60a4003f40 2023-10-14T16:41:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a400ba00 con 0x7f60b8070d70 2023-10-14T16:41:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.208+0000 7f60bed8d640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4004d00 0x7f60a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bdd8b640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4003f40 0x7f60a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bed8d640 1 -- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 msgr2=0x7f60a4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60be58c640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60a4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bed8d640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60a4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bed8d640 1 -- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4003f40 msgr2=0x7f60a40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bed8d640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4003f40 0x7f60a40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60bed8d640 1 -- 172.21.15.29:0/2197648553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a4003520 con 0x7f60a4004d00 2023-10-14T16:41:08.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.209+0000 7f60be58c640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60b8070d70 0x7f60a4003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.210+0000 7f60bed8d640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4004d00 0x7f60a4005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f60b000a790 tx=0x7f60a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7efc9f5dfad1391 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.210+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0034630 con 0x7f60a4004d00 2023-10-14T16:41:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.210+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60b002f070 con 0x7f60a4004d00 2023-10-14T16:41:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.210+0000 7f60bd58a640 1 -- 172.21.15.29:0/2197648553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b0034a30 con 0x7f60a4004d00 2023-10-14T16:41:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 -- 172.21.15.29:0/2197648553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4004d00 msgr2=0x7f60a4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2197648553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a4004d00 0x7f60a4005180 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f60b000a790 tx=0x7f60a40057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bed8d640 1 -- 172.21.15.29:0/2197648553 reap_dead start 2023-10-14T16:41:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 -- 172.21.15.29:0/2197648553 shutdown_connections 2023-10-14T16:41:08.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 -- 172.21.15.29:0/2197648553 >> 172.21.15.29:0/2197648553 conn(0x7f60b80b0a90 msgr2=0x7f60b80ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 -- 172.21.15.29:0/2197648553 shutdown_connections 2023-10-14T16:41:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.211+0000 7f60bf58e640 1 -- 172.21.15.29:0/2197648553 wait complete. 2023-10-14T16:41:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.213+0000 7f60bf58e640 1 Processor -- start 2023-10-14T16:41:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.213+0000 7f60bf58e640 1 -- start start 2023-10-14T16:41:08.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4004d00 0x7f60b80b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b80c21b0 0x7f60b80af6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b8152980 con 0x7f60b8152500 2023-10-14T16:41:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b815aeb0 con 0x7f60b80c21b0 2023-10-14T16:41:08.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.214+0000 7f60bf58e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b81562b0 con 0x7f60a4004d00 2023-10-14T16:41:08.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56820/0 (socket says 172.21.15.29:56820) 2023-10-14T16:41:08.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 -- 172.21.15.29:0/2968979997 learned_addr learned my addr 172.21.15.29:0/2968979997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:08.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bed8d640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b80c21b0 0x7f60b80af6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 -- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4004d00 msgr2=0x7f60b80b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60be58c640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4004d00 0x7f60b80b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4004d00 0x7f60b80b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 -- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b80c21b0 msgr2=0x7f60b80af6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b80c21b0 0x7f60b80af6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:08.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 -- 172.21.15.29:0/2968979997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b815b150 con 0x7f60b8152500 2023-10-14T16:41:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.215+0000 7f60bdd8b640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f60a8011570 tx=0x7f60a8011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.216+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8015930 con 0x7f60b8152500 2023-10-14T16:41:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.216+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a8015a90 con 0x7f60b8152500 2023-10-14T16:41:08.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.216+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60a8015e20 con 0x7f60b8152500 2023-10-14T16:41:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.216+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b81f0200 con 0x7f60b8152500 2023-10-14T16:41:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.216+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b815cd50 con 0x7f60b8152500 2023-10-14T16:41:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.217+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b815b2e0 con 0x7f60b8152500 2023-10-14T16:41:08.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.218+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f60a800f460 con 0x7f60b8152500 2023-10-14T16:41:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.219+0000 7f60aeffd640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 0x7f609c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:08.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.219+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f60a8089e70 con 0x7f60b8152500 2023-10-14T16:41:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.219+0000 7f60be58c640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 0x7f609c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.220+0000 7f60be58c640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 0x7f609c06d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f60b8070cf0 tx=0x7f60b4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:08.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.223+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60a8054e00 con 0x7f60b8152500 2023-10-14T16:41:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:08.492+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f60b80cc730 con 0x7f60b8152500 2023-10-14T16:41:09.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.013+0000 7f60aeffd640 1 -- 172.21.15.29:0/2968979997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v62) v1 ==== 161+0+0 (secure 0 0 0) 0x7f60a8058420 con 0x7f60b8152500 2023-10-14T16:41:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 msgr2=0x7f609c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 0x7f609c06d360 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f60b8070cf0 tx=0x7f60b4008040 comp rx=0 tx=0).stop 2023-10-14T16:41:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 msgr2=0x7f60b815c390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f60a8011570 tx=0x7f60a8011a30 comp rx=0 tx=0).stop 2023-10-14T16:41:09.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 shutdown_connections 2023-10-14T16:41:09.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f609c06aea0 0x7f609c06d360 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60b80c21b0 0x7f60b80af6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60b8152500 0x7f60b815c390 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 --2- 172.21.15.29:0/2968979997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4004d00 0x7f60b80b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.016+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 >> 172.21.15.29:0/2968979997 conn(0x7f60b80b0a90 msgr2=0x7f60b8159970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:09.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.017+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 shutdown_connections 2023-10-14T16:41:09.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.017+0000 7f60bf58e640 1 -- 172.21.15.29:0/2968979997 wait complete. 2023-10-14T16:41:09.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:41:09.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.273+0000 7ff1c6e14640 1 Processor -- start 2023-10-14T16:41:09.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.273+0000 7ff1c6e14640 1 -- start start 2023-10-14T16:41:09.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.274+0000 7ff1c6e14640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1c00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.274+0000 7ff1c6e14640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1c00ccfb0 0x7ff1c00cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.274+0000 7ff1c6e14640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1c015b9d0 con 0x7ff1c006c040 2023-10-14T16:41:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.274+0000 7ff1c6e14640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1c015bcb0 con 0x7ff1c00ccfb0 2023-10-14T16:41:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.274+0000 7ff1c6e14640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015bf90 con 0x7ff1c00ad970 2023-10-14T16:41:09.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c5e12640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff1c006c040 0x7ff1c006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32864/0 (socket says 172.21.15.29:32864) 2023-10-14T16:41:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c5e12640 1 -- 172.21.15.29:0/3922883337 learned_addr learned my addr 172.21.15.29:0/3922883337 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1448099280 0 0) 0x7ff1c015b9d0 con 0x7ff1c006c040 2023-10-14T16:41:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ac003520 con 0x7ff1c006c040 2023-10-14T16:41:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c6613640 1 --2- 172.21.15.29:0/3922883337 >> v2:172.21.15.186:3300/0 conn(0x7ff1c00ccfb0 0x7ff1c00cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.275+0000 7ff1c5611640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1c00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 814408443 0 0) 0x7ff1ac003520 con 0x7ff1c006c040 2023-10-14T16:41:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 msgr2=0x7ff1c00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1c00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 >> v2:172.21.15.186:3300/0 conn(0x7ff1c00ccfb0 msgr2=0x7ff1c00cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 --2- 172.21.15.29:0/3922883337 >> v2:172.21.15.186:3300/0 conn(0x7ff1c00ccfb0 0x7ff1c00cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c00cc270 con 0x7ff1c006c040 2023-10-14T16:41:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff1bc001670 con 0x7ff1c006c040 2023-10-14T16:41:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 >> v1:172.21.15.29:6789/0 conn(0x7ff1c006c040 legacy=0x7ff1c006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1ac003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac003ff0 0x7ff1ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac004d00 0x7ff1ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015bcb0 con 0x7ff1ac004d00 2023-10-14T16:41:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015bf90 con 0x7ff1ac003ff0 2023-10-14T16:41:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ac00bae0 con 0x7ff1c00ad970 2023-10-14T16:41:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c6613640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac004d00 0x7ff1ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.276+0000 7ff1c5e12640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1ac003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c5611640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac003ff0 0x7ff1ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c6613640 1 -- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 msgr2=0x7ff1ac003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c6613640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c00ad970 0x7ff1ac003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c6613640 1 -- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac003ff0 msgr2=0x7ff1ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:09.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c6613640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac003ff0 0x7ff1ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c6613640 1 -- 172.21.15.29:0/3922883337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac003520 con 0x7ff1ac004d00 2023-10-14T16:41:09.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.277+0000 7ff1c5611640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ac003ff0 0x7ff1ac0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:09.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.278+0000 7ff1c6613640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac004d00 0x7ff1ac005180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8007360 tx=0x7ff1ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9ed7cb22578e38a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:09.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.278+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b802ccf0 con 0x7ff1ac004d00 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.278+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b802ce50 con 0x7ff1ac004d00 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.278+0000 7ff1c4e10640 1 -- 172.21.15.29:0/3922883337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b8034660 con 0x7ff1ac004d00 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 -- 172.21.15.29:0/3922883337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac004d00 msgr2=0x7ff1ac005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/3922883337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ac004d00 0x7ff1ac005180 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff1b8007360 tx=0x7ff1ac0057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6613640 1 -- 172.21.15.29:0/3922883337 reap_dead start 2023-10-14T16:41:09.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 -- 172.21.15.29:0/3922883337 shutdown_connections 2023-10-14T16:41:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 -- 172.21.15.29:0/3922883337 >> 172.21.15.29:0/3922883337 conn(0x7ff1c00ce6a0 msgr2=0x7ff1c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 -- 172.21.15.29:0/3922883337 shutdown_connections 2023-10-14T16:41:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.279+0000 7ff1c6e14640 1 -- 172.21.15.29:0/3922883337 wait complete. 2023-10-14T16:41:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.281+0000 7ff1c6e14640 1 Processor -- start 2023-10-14T16:41:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.281+0000 7ff1c6e14640 1 -- start start 2023-10-14T16:41:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00c4de0 0x7ff1c01f1810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c01533b0 0x7ff1c01451c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c01f21d0 con 0x7ff1c01f1d50 2023-10-14T16:41:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c015c1d0 con 0x7ff1c00c4de0 2023-10-14T16:41:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c6e14640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1c0153b80 con 0x7ff1c01533b0 2023-10-14T16:41:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c5611640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c5611640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56848/0 (socket says 172.21.15.29:56848) 2023-10-14T16:41:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.282+0000 7ff1c5611640 1 -- 172.21.15.29:0/973523661 learned_addr learned my addr 172.21.15.29:0/973523661 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5e12640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00c4de0 0x7ff1c01f1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c6613640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c01533b0 0x7ff1c01451c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 -- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c01533b0 msgr2=0x7ff1c01451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c01533b0 0x7ff1c01451c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 -- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00c4de0 msgr2=0x7ff1c01f1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00c4de0 0x7ff1c01f1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 -- 172.21.15.29:0/973523661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c0153e60 con 0x7ff1c01f1d50 2023-10-14T16:41:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.283+0000 7ff1c5611640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e7e0 tx=0x7ff1b000eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.284+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000c670 con 0x7ff1c01f1d50 2023-10-14T16:41:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.284+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1b000c7d0 con 0x7ff1c01f1d50 2023-10-14T16:41:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.284+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1b000cb80 con 0x7ff1c01f1d50 2023-10-14T16:41:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.284+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c0154140 con 0x7ff1c01f1d50 2023-10-14T16:41:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.284+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1c0153860 con 0x7ff1c01f1d50 2023-10-14T16:41:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.285+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1c004e9b0 con 0x7ff1c01f1d50 2023-10-14T16:41:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.286+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff1b000cce0 con 0x7ff1c01f1d50 2023-10-14T16:41:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.287+0000 7ff1b67fc640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 0x7ff1a406d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.287+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff1b0089450 con 0x7ff1c01f1d50 2023-10-14T16:41:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.287+0000 7ff1c5e12640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 0x7ff1a406d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.288+0000 7ff1c5e12640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 0x7ff1a406d350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc001eb0 tx=0x7ff1bc001170 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.291+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1b0054470 con 0x7ff1c01f1d50 2023-10-14T16:41:09.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.486+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff1c006b2b0 con 0x7ff1c01f1d50 2023-10-14T16:41:09.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.491+0000 7ff1b67fc640 1 -- 172.21.15.29:0/973523661 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff1b0057a90 con 0x7ff1c01f1d50 2023-10-14T16:41:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 msgr2=0x7ff1a406d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 0x7ff1a406d350 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7ff1bc001eb0 tx=0x7ff1bc001170 comp rx=0 tx=0).stop 2023-10-14T16:41:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 msgr2=0x7ff1c0152e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7ff1b000e7e0 tx=0x7ff1b000eca0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 shutdown_connections 2023-10-14T16:41:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1a406ae90 0x7ff1a406d350 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1c01533b0 0x7ff1c01451c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1c01f1d50 0x7ff1c0152e70 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 --2- 172.21.15.29:0/973523661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1c00c4de0 0x7ff1c01f1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 >> 172.21.15.29:0/973523661 conn(0x7ff1c00ce6a0 msgr2=0x7ff1c006a480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 shutdown_connections 2023-10-14T16:41:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:09.494+0000 7ff1c6e14640 1 -- 172.21.15.29:0/973523661 wait complete. 2023-10-14T16:41:09.513 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:41:09.514 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:41:09.637 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:41:09.758 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:41:09.759 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:09.759 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:09.822 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:09.823 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:09.879 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:09.880 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:09.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.163+0000 7faf30bac640 1 Processor -- start 2023-10-14T16:41:10.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.163+0000 7faf30bac640 1 -- start start 2023-10-14T16:41:10.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.165+0000 7faf30bac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf2c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.165+0000 7faf30bac640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2c15b9d0 con 0x7faf2c06c040 2023-10-14T16:41:10.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.165+0000 7faf30bac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf2c15bcb0 con 0x7faf2c0ccfb0 2023-10-14T16:41:10.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.165+0000 7faf30bac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c15bf90 con 0x7faf2c0ad970 2023-10-14T16:41:10.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf2c06c040 0x7faf2c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32876/0 (socket says 172.21.15.29:32876) 2023-10-14T16:41:10.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2b7fe640 1 -- 172.21.15.29:0/2253317037 learned_addr learned my addr 172.21.15.29:0/2253317037 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2affd640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf2c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3298302577 0 0) 0x7faf2c15b9d0 con 0x7faf2c06c040 2023-10-14T16:41:10.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf14003520 con 0x7faf2c06c040 2023-10-14T16:41:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.166+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839160769 0 0) 0x7faf14003520 con 0x7faf2c06c040 2023-10-14T16:41:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 msgr2=0x7faf2c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf2c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 >> v1:172.21.15.186:6789/0 conn(0x7faf2c0ccfb0 legacy=0x7faf2c0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0cc270 con 0x7faf2c06c040 2023-10-14T16:41:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7faf1c001670 con 0x7faf2c06c040 2023-10-14T16:41:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 >> v1:172.21.15.29:6789/0 conn(0x7faf2c06c040 legacy=0x7faf2c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf140039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf14003f70 0x7faf140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf14005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c15bf90 con 0x7faf14004d00 2023-10-14T16:41:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf1400bac0 con 0x7faf14003f70 2023-10-14T16:41:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf1400bc80 con 0x7faf2c0ad970 2023-10-14T16:41:10.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2bfff640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf14005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2affd640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf14003f70 0x7faf140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.167+0000 7faf2b7fe640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf140039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.168+0000 7faf2bfff640 1 -- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 msgr2=0x7faf140039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.168+0000 7faf2bfff640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c0ad970 0x7faf140039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.168+0000 7faf2bfff640 1 -- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf14003f70 msgr2=0x7faf140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.168+0000 7faf2bfff640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf14003f70 0x7faf140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.168+0000 7faf2bfff640 1 -- 172.21.15.29:0/2253317037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf14003520 con 0x7faf14004d00 2023-10-14T16:41:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.169+0000 7faf2bfff640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf14005180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf24001e50 tx=0x7faf140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bb1971d21dc8a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.169+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24003b70 con 0x7faf14004d00 2023-10-14T16:41:10.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.169+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf24003cd0 con 0x7faf14004d00 2023-10-14T16:41:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.169+0000 7faf2a7fc640 1 -- 172.21.15.29:0/2253317037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24005310 con 0x7faf14004d00 2023-10-14T16:41:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 -- 172.21.15.29:0/2253317037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 msgr2=0x7faf14005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 --2- 172.21.15.29:0/2253317037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf14005180 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7faf24001e50 tx=0x7faf140057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf2bfff640 1 -- 172.21.15.29:0/2253317037 reap_dead start 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 -- 172.21.15.29:0/2253317037 shutdown_connections 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 -- 172.21.15.29:0/2253317037 >> 172.21.15.29:0/2253317037 conn(0x7faf2c0ce6a0 msgr2=0x7faf2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 -- 172.21.15.29:0/2253317037 shutdown_connections 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.170+0000 7faf30bac640 1 -- 172.21.15.29:0/2253317037 wait complete. 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.172+0000 7faf30bac640 1 Processor -- start 2023-10-14T16:41:10.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.172+0000 7faf30bac640 1 -- start start 2023-10-14T16:41:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 0x7faf2c15c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c15c710 0x7faf2c15cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c0c8660 con 0x7faf14004d00 2023-10-14T16:41:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1593e0 con 0x7faf2c14e430 2023-10-14T16:41:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf30bac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c15d0e0 con 0x7faf2c15c710 2023-10-14T16:41:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56872/0 (socket says 172.21.15.29:56872) 2023-10-14T16:41:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.173+0000 7faf2b7fe640 1 -- 172.21.15.29:0/3136840865 learned_addr learned my addr 172.21.15.29:0/3136840865 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2affd640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 0x7faf2c15c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2bfff640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c15c710 0x7faf2c15cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 -- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c15c710 msgr2=0x7faf2c15cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c15c710 0x7faf2c15cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 -- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 msgr2=0x7faf2c15c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 0x7faf2c15c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 -- 172.21.15.29:0/3136840865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c0cdcd0 con 0x7faf14004d00 2023-10-14T16:41:10.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2affd640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 0x7faf2c15c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:10.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf2b7fe640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00e7e0 tx=0x7faf1c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c00c8d0 con 0x7faf14004d00 2023-10-14T16:41:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf1c00ca30 con 0x7faf14004d00 2023-10-14T16:41:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c00cde0 con 0x7faf14004d00 2023-10-14T16:41:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.174+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c15db80 con 0x7faf14004d00 2023-10-14T16:41:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.175+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c0c8380 con 0x7faf14004d00 2023-10-14T16:41:10.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.176+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf2c06b330 con 0x7faf14004d00 2023-10-14T16:41:10.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.177+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7faf1c01d460 con 0x7faf14004d00 2023-10-14T16:41:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.177+0000 7faf13fff640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 0x7faf1806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.178+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faf1c089740 con 0x7faf14004d00 2023-10-14T16:41:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.178+0000 7faf2affd640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 0x7faf1806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.179+0000 7faf2affd640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 0x7faf1806d330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf20009a00 tx=0x7faf20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.183+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf1c054760 con 0x7faf14004d00 2023-10-14T16:41:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.380+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faf2c04e9b0 con 0x7faf14004d00 2023-10-14T16:41:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.384+0000 7faf13fff640 1 -- 172.21.15.29:0/3136840865 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7faf1c0015a0 con 0x7faf14004d00 2023-10-14T16:41:10.386 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:10.386 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.386+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 msgr2=0x7faf1806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.386+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 0x7faf1806d330 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7faf20009a00 tx=0x7faf20008040 comp rx=0 tx=0).stop 2023-10-14T16:41:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.386+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 msgr2=0x7faf2c0c1b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.386+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00e7e0 tx=0x7faf1c00eca0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 shutdown_connections 2023-10-14T16:41:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7faf1806ae70 0x7faf1806d330 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf2c15c710 0x7faf2c15cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf2c14e430 0x7faf2c15c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 --2- 172.21.15.29:0/3136840865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf14004d00 0x7faf2c0c1b20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 >> 172.21.15.29:0/3136840865 conn(0x7faf2c0ce6a0 msgr2=0x7faf2c0c9a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 shutdown_connections 2023-10-14T16:41:10.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.387+0000 7faf30bac640 1 -- 172.21.15.29:0/3136840865 wait complete. 2023-10-14T16:41:10.407 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:10.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.639+0000 7fa5df256640 1 Processor -- start 2023-10-14T16:41:10.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.639+0000 7fa5df256640 1 -- start start 2023-10-14T16:41:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5df256640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5df256640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5df256640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5d80cb030 con 0x7fa5d815ad70 2023-10-14T16:41:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5df256640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5d80cae30 con 0x7fa5d806a4b0 2023-10-14T16:41:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5df256640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c91c0 con 0x7fa5d80c9ca0 2023-10-14T16:41:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5d7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56884/0 (socket says 172.21.15.29:56884) 2023-10-14T16:41:10.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.641+0000 7fa5d7fff640 1 -- 172.21.15.29:0/538149965 learned_addr learned my addr 172.21.15.29:0/538149965 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:10.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.642+0000 7fa5dd7f3640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.642+0000 7fa5d7fff640 1 -- 172.21.15.29:0/538149965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 msgr2=0x7fa5d80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:10.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.642+0000 7fa5d7fff640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.642+0000 7fa5d7fff640 1 -- 172.21.15.29:0/538149965 >> v1:172.21.15.186:6789/0 conn(0x7fa5d815ad70 legacy=0x7fa5d8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.642+0000 7fa5d7fff640 1 -- 172.21.15.29:0/538149965 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5d80c94a0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.643+0000 7fa5d7fff640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009870 tx=0x7fa5c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=339b1ec2cb95659a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.643+0000 7fa5d77fe640 1 -- 172.21.15.29:0/538149965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4030840 con 0x7fa5d806a4b0 2023-10-14T16:41:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5d77fe640 1 -- 172.21.15.29:0/538149965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5c40309a0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5d77fe640 1 -- 172.21.15.29:0/538149965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5c4030d30 con 0x7fa5d806a4b0 2023-10-14T16:41:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 -- 172.21.15.29:0/538149965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 msgr2=0x7fa5d80ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4009870 tx=0x7fa5c402b980 comp rx=0 tx=0).stop 2023-10-14T16:41:10.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 -- 172.21.15.29:0/538149965 shutdown_connections 2023-10-14T16:41:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 --2- 172.21.15.29:0/538149965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80ca3b0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 -- 172.21.15.29:0/538149965 >> 172.21.15.29:0/538149965 conn(0x7fa5d80ce9b0 msgr2=0x7fa5d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 -- 172.21.15.29:0/538149965 shutdown_connections 2023-10-14T16:41:10.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.644+0000 7fa5df256640 1 -- 172.21.15.29:0/538149965 wait complete. 2023-10-14T16:41:10.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.646+0000 7fa5df256640 1 Processor -- start 2023-10-14T16:41:10.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.646+0000 7fa5df256640 1 -- start start 2023-10-14T16:41:10.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80b22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d815ad70 0x7fa5d815ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c5d80 con 0x7fa5d806a4b0 2023-10-14T16:41:10.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c5490 con 0x7fa5d815ad70 2023-10-14T16:41:10.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.647+0000 7fa5df256640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5d80c35c0 con 0x7fa5d80c9ca0 2023-10-14T16:41:10.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56896/0 (socket says 172.21.15.29:56896) 2023-10-14T16:41:10.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 -- 172.21.15.29:0/1916893205 learned_addr learned my addr 172.21.15.29:0/1916893205 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:10.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dd7f3640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d815ad70 0x7fa5d815ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 -- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 msgr2=0x7fa5d80b22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80b22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 -- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d815ad70 msgr2=0x7fa5d815ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d815ad70 0x7fa5d815ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 -- 172.21.15.29:0/1916893205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5c40094d0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.648+0000 7fa5dcff2640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00c400 tx=0x7fa5cc00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.649+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc0052d0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.649+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5cc005430 con 0x7fa5d806a4b0 2023-10-14T16:41:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.649+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5d80c38a0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.649+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5cc003830 con 0x7fa5d806a4b0 2023-10-14T16:41:10.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.649+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5d80c3e60 con 0x7fa5d806a4b0 2023-10-14T16:41:10.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.650+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5d804e160 con 0x7fa5d806a4b0 2023-10-14T16:41:10.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.651+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa5cc003990 con 0x7fa5d806a4b0 2023-10-14T16:41:10.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.652+0000 7fa5d57fa640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:10.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.652+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa5cc088b80 con 0x7fa5d806a4b0 2023-10-14T16:41:10.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.652+0000 7fa5d7fff640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:10.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.653+0000 7fa5d7fff640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4036000 tx=0x7fa5c403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:10.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.657+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5cc053c40 con 0x7fa5d806a4b0 2023-10-14T16:41:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.853+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5d80ce2b0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.854+0000 7fa5d57fa640 1 -- 172.21.15.29:0/1916893205 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5cc0170a0 con 0x7fa5d806a4b0 2023-10-14T16:41:10.856 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:10.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 msgr2=0x7fa5bc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa5c4036000 tx=0x7fa5c403b040 comp rx=0 tx=0).stop 2023-10-14T16:41:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 msgr2=0x7fa5d80c4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:10.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fa5cc00c400 tx=0x7fa5cc00c8c0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 shutdown_connections 2023-10-14T16:41:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa5bc06ae70 0x7fa5bc06d330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5d815ad70 0x7fa5d815ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5d80c9ca0 0x7fa5d80b22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 --2- 172.21.15.29:0/1916893205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5d806a4b0 0x7fa5d80c4f50 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 >> 172.21.15.29:0/1916893205 conn(0x7fa5d80ce9b0 msgr2=0x7fa5d80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 shutdown_connections 2023-10-14T16:41:10.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:10.857+0000 7fa5df256640 1 -- 172.21.15.29:0/1916893205 wait complete. 2023-10-14T16:41:10.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:10.966 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:41:10.975 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:41:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.105+0000 7fe474cc9640 1 Processor -- start 2023-10-14T16:41:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.105+0000 7fe474cc9640 1 -- start start 2023-10-14T16:41:11.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe474cc9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4700cfaf0 0x7fe4700b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe474cc9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4700d3980 con 0x7fe4700b3420 2023-10-14T16:41:11.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe474cc9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4700ce630 con 0x7fe4700cf710 2023-10-14T16:41:11.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe474cc9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4700d3c60 con 0x7fe4700cfaf0 2023-10-14T16:41:11.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe46dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe4700b3420 0x7fe4700b5cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48116/0 (socket says 172.21.15.29:48116) 2023-10-14T16:41:11.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe46dd9b640 1 -- 172.21.15.29:0/793591413 learned_addr learned my addr 172.21.15.29:0/793591413 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:11.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.107+0000 7fe46ed9d640 1 --2- 172.21.15.29:0/793591413 >> v2:172.21.15.29:3300/0 conn(0x7fe4700cfaf0 0x7fe4700b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.108+0000 7fe46d59a640 1 -- 172.21.15.29:0/793591413 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052519391 0 0) 0x7fe4700d3980 con 0x7fe4700b3420 2023-10-14T16:41:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.108+0000 7fe46d59a640 1 -- 172.21.15.29:0/793591413 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe458003520 con 0x7fe4700b3420 2023-10-14T16:41:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.108+0000 7fe46ed9d640 1 -- 172.21.15.29:0/793591413 >> v1:172.21.15.186:6789/0 conn(0x7fe4700cf710 legacy=0x7fe4700b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.108+0000 7fe46ed9d640 1 -- 172.21.15.29:0/793591413 >> v1:172.21.15.29:6789/0 conn(0x7fe4700b3420 legacy=0x7fe4700b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.108+0000 7fe46ed9d640 1 -- 172.21.15.29:0/793591413 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4700ce910 con 0x7fe4700cfaf0 2023-10-14T16:41:11.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.109+0000 7fe46ed9d640 1 --2- 172.21.15.29:0/793591413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cfaf0 0x7fe4700b63d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe468009d90 tx=0x7fe46802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f278ea99770427b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.109+0000 7fe46d59a640 1 -- 172.21.15.29:0/793591413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46802cc60 con 0x7fe4700cfaf0 2023-10-14T16:41:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.109+0000 7fe46d59a640 1 -- 172.21.15.29:0/793591413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46802cdc0 con 0x7fe4700cfaf0 2023-10-14T16:41:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.109+0000 7fe46d59a640 1 -- 172.21.15.29:0/793591413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe468034660 con 0x7fe4700cfaf0 2023-10-14T16:41:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 -- 172.21.15.29:0/793591413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cfaf0 msgr2=0x7fe4700b63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 --2- 172.21.15.29:0/793591413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cfaf0 0x7fe4700b63d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe468009d90 tx=0x7fe46802beb0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 -- 172.21.15.29:0/793591413 shutdown_connections 2023-10-14T16:41:11.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 --2- 172.21.15.29:0/793591413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cfaf0 0x7fe4700b63d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 -- 172.21.15.29:0/793591413 >> 172.21.15.29:0/793591413 conn(0x7fe4700d2cc0 msgr2=0x7fe4700d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 -- 172.21.15.29:0/793591413 shutdown_connections 2023-10-14T16:41:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.110+0000 7fe474cc9640 1 -- 172.21.15.29:0/793591413 wait complete. 2023-10-14T16:41:11.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.112+0000 7fe474cc9640 1 Processor -- start 2023-10-14T16:41:11.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.112+0000 7fe474cc9640 1 -- start start 2023-10-14T16:41:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4700b3420 0x7fe4701500c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4700cfaf0 0x7fe4700b5b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700a0020 con 0x7fe4700cf710 2023-10-14T16:41:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700cc5b0 con 0x7fe4700b3420 2023-10-14T16:41:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.113+0000 7fe474cc9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4700a0270 con 0x7fe4700cfaf0 2023-10-14T16:41:11.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36788/0 (socket says 172.21.15.29:36788) 2023-10-14T16:41:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 -- 172.21.15.29:0/1573251181 learned_addr learned my addr 172.21.15.29:0/1573251181 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46e59c640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4700b3420 0x7fe4701500c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 -- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4700cfaf0 msgr2=0x7fe4700b5b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4700cfaf0 0x7fe4700b5b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 -- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4700b3420 msgr2=0x7fe4701500c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4700b3420 0x7fe4701500c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 -- 172.21.15.29:0/1573251181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009a70 con 0x7fe4700cf710 2023-10-14T16:41:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.114+0000 7fe46dd9b640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c500 tx=0x7fe46400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.115+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464005ce0 con 0x7fe4700cf710 2023-10-14T16:41:11.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.115+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe464005e40 con 0x7fe4700cf710 2023-10-14T16:41:11.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.115+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe464007330 con 0x7fe4700cf710 2023-10-14T16:41:11.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.115+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4700a0550 con 0x7fe4700cf710 2023-10-14T16:41:11.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.115+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe470150600 con 0x7fe4700cf710 2023-10-14T16:41:11.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.116+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4700a0a70 con 0x7fe4700cf710 2023-10-14T16:41:11.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.117+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe464007550 con 0x7fe4700cf710 2023-10-14T16:41:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.118+0000 7fe45effd640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 0x7fe45006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.118+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe464088d40 con 0x7fe4700cf710 2023-10-14T16:41:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.118+0000 7fe46e59c640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 0x7fe45006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.119+0000 7fe46e59c640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 0x7fe45006d330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe460000fb0 tx=0x7fe460009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.122+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4700a0a70 con 0x7fe4700cf710 2023-10-14T16:41:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.321+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4700a0a70 con 0x7fe4700cf710 2023-10-14T16:41:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.322+0000 7fe45effd640 1 -- 172.21.15.29:0/1573251181 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4700a0a70 con 0x7fe4700cf710 2023-10-14T16:41:11.325 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 msgr2=0x7fe45006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 0x7fe45006d330 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fe460000fb0 tx=0x7fe460009040 comp rx=0 tx=0).stop 2023-10-14T16:41:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 msgr2=0x7fe47009c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe46400c500 tx=0x7fe46400c9c0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 shutdown_connections 2023-10-14T16:41:11.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe45006ae70 0x7fe45006d330 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4700cfaf0 0x7fe4700b5b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4700cf710 0x7fe47009c9a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 --2- 172.21.15.29:0/1573251181 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4700b3420 0x7fe4701500c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 >> 172.21.15.29:0/1573251181 conn(0x7fe4700d2cc0 msgr2=0x7fe47006bb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.325+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 shutdown_connections 2023-10-14T16:41:11.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.326+0000 7fe474cc9640 1 -- 172.21.15.29:0/1573251181 wait complete. 2023-10-14T16:41:11.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.609+0000 7f9e0ea46640 1 Processor -- start 2023-10-14T16:41:11.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.609+0000 7f9e0ea46640 1 -- start start 2023-10-14T16:41:11.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0ea46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159e90 0x7f9e080ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0ea46640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c140 0x7f9e0806c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0ea46640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0806cad0 con 0x7f9e08159ab0 2023-10-14T16:41:11.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0ea46640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0815d890 con 0x7f9e08159e90 2023-10-14T16:41:11.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0ea46640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e08070360 con 0x7f9e0806c140 2023-10-14T16:41:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9dfffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159e90 0x7f9e080ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9dfffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159e90 0x7f9e080ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36794/0 (socket says 172.21.15.29:36794) 2023-10-14T16:41:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9dfffff640 1 -- 172.21.15.29:0/2963705620 learned_addr learned my addr 172.21.15.29:0/2963705620 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:11.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.611+0000 7f9e0cfe3640 1 --2- 172.21.15.29:0/2963705620 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c140 0x7f9e0806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.612+0000 7f9dfffff640 1 -- 172.21.15.29:0/2963705620 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c140 msgr2=0x7f9e0806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:11.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.612+0000 7f9dfffff640 1 --2- 172.21.15.29:0/2963705620 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c140 0x7f9e0806c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.612+0000 7f9dfffff640 1 -- 172.21.15.29:0/2963705620 >> v1:172.21.15.186:6789/0 conn(0x7f9e08159ab0 legacy=0x7f9e0806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.612+0000 7f9dfffff640 1 -- 172.21.15.29:0/2963705620 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e080cd1f0 con 0x7f9e08159e90 2023-10-14T16:41:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.613+0000 7f9dfffff640 1 --2- 172.21.15.29:0/2963705620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159e90 0x7f9e080ce370 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9e080adcd0 tx=0x7f9df002da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db96419a5d0477e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.613+0000 7f9e077fe640 1 -- 172.21.15.29:0/2963705620 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df002f850 con 0x7f9e08159e90 2023-10-14T16:41:11.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.613+0000 7f9e077fe640 1 -- 172.21.15.29:0/2963705620 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df002f9b0 con 0x7f9e08159e90 2023-10-14T16:41:11.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.613+0000 7f9e077fe640 1 -- 172.21.15.29:0/2963705620 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df002fd40 con 0x7f9e08159e90 2023-10-14T16:41:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 -- 172.21.15.29:0/2963705620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159e90 msgr2=0x7f9e080ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/2963705620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159e90 0x7f9e080ce370 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f9e080adcd0 tx=0x7f9df002da90 comp rx=0 tx=0).stop 2023-10-14T16:41:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 -- 172.21.15.29:0/2963705620 shutdown_connections 2023-10-14T16:41:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/2963705620 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159e90 0x7f9e080ce370 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/2963705620 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c140 0x7f9e0806c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 -- 172.21.15.29:0/2963705620 >> 172.21.15.29:0/2963705620 conn(0x7f9e080ce9b0 msgr2=0x7f9e080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 -- 172.21.15.29:0/2963705620 shutdown_connections 2023-10-14T16:41:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.614+0000 7f9e0ea46640 1 -- 172.21.15.29:0/2963705620 wait complete. 2023-10-14T16:41:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.616+0000 7f9e0ea46640 1 Processor -- start 2023-10-14T16:41:11.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.616+0000 7f9e0ea46640 1 -- start start 2023-10-14T16:41:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08159ab0 0x7f9e080c4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e08159e90 0x7f9e080c6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0814f400 con 0x7f9e0806c140 2023-10-14T16:41:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e08158b30 con 0x7f9e08159e90 2023-10-14T16:41:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e0ea46640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080c54d0 con 0x7f9e08159ab0 2023-10-14T16:41:11.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.617+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36800/0 (socket says 172.21.15.29:36800) 2023-10-14T16:41:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 -- 172.21.15.29:0/1196687163 learned_addr learned my addr 172.21.15.29:0/1196687163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9dfffff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08159ab0 0x7f9e080c4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e0cfe3640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e08159e90 0x7f9e080c6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 -- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08159ab0 msgr2=0x7f9e080c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08159ab0 0x7f9e080c4f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 -- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e08159e90 msgr2=0x7f9e080c6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e08159e90 0x7f9e080c6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 -- 172.21.15.29:0/1196687163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df00094d0 con 0x7f9e0806c140 2023-10-14T16:41:11.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.618+0000 7f9e07fff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c450 tx=0x7f9df800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.619+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df80052d0 con 0x7f9e0806c140 2023-10-14T16:41:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.619+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df8005430 con 0x7f9e0806c140 2023-10-14T16:41:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.619+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8012870 con 0x7f9e0806c140 2023-10-14T16:41:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.619+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e080c5f10 con 0x7f9e0806c140 2023-10-14T16:41:11.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.619+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e080c64d0 con 0x7f9e0806c140 2023-10-14T16:41:11.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.620+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0804e160 con 0x7f9e0806c140 2023-10-14T16:41:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.621+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9df80129d0 con 0x7f9e0806c140 2023-10-14T16:41:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.622+0000 7f9e057fa640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 0x7f9de806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.623+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9df8089060 con 0x7f9e0806c140 2023-10-14T16:41:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.623+0000 7f9dfffff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 0x7f9de806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:11.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.624+0000 7f9dfffff640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 0x7f9de806d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9df0038000 tx=0x7f9df0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:11.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.627+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9df80bbfe0 con 0x7f9e0806c140 2023-10-14T16:41:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.827+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e0806cad0 con 0x7f9e0806c140 2023-10-14T16:41:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.827+0000 7f9e057fa640 1 -- 172.21.15.29:0/1196687163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e0806cad0 con 0x7f9e0806c140 2023-10-14T16:41:11.829 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 msgr2=0x7f9de806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 0x7f9de806d330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9df0038000 tx=0x7f9df0008030 comp rx=0 tx=0).stop 2023-10-14T16:41:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 msgr2=0x7f9e0814eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c450 tx=0x7f9df800c910 comp rx=0 tx=0).stop 2023-10-14T16:41:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 shutdown_connections 2023-10-14T16:41:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9de806ae70 0x7f9de806d330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e08159e90 0x7f9e080c6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08159ab0 0x7f9e080c4f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 --2- 172.21.15.29:0/1196687163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e0806c140 0x7f9e0814eec0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:11.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 >> 172.21.15.29:0/1196687163 conn(0x7f9e080ce9b0 msgr2=0x7f9e080afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:11.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 shutdown_connections 2023-10-14T16:41:11.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:11.830+0000 7f9e0ea46640 1 -- 172.21.15.29:0/1196687163 wait complete. 2023-10-14T16:41:11.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:41:12.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.055+0000 7f83931f0640 1 Processor -- start 2023-10-14T16:41:12.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.055+0000 7f83931f0640 1 -- start start 2023-10-14T16:41:12.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.056+0000 7f83931f0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 0x7f838c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.056+0000 7f83931f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f838c160b90 0x7f838c160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.056+0000 7f83931f0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f838c0ac890 con 0x7f838c069da0 2023-10-14T16:41:12.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.056+0000 7f83931f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f838c15d530 con 0x7f838c160b90 2023-10-14T16:41:12.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.056+0000 7f83931f0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c161650 con 0x7f838c15d150 2023-10-14T16:41:12.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f838c160b90 0x7f838c160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f838c160b90 0x7f838c160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36812/0 (socket says 172.21.15.29:36812) 2023-10-14T16:41:12.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 -- 172.21.15.29:0/4221307575 learned_addr learned my addr 172.21.15.29:0/4221307575 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:12.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8390f8c640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 0x7f838c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 -- 172.21.15.29:0/4221307575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 msgr2=0x7f838c0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 0x7f838c0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 -- 172.21.15.29:0/4221307575 >> v1:172.21.15.186:6789/0 conn(0x7f838c069da0 legacy=0x7f838c06a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8383fff640 1 -- 172.21.15.29:0/4221307575 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f838c161930 con 0x7f838c160b90 2023-10-14T16:41:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.057+0000 7f8390f8c640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 0x7f838c0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:12.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.058+0000 7f8383fff640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c160b90 0x7f838c160fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0370c4abb3702b0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.059+0000 7f83837fe640 1 -- 172.21.15.29:0/4221307575 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c031640 con 0x7f838c160b90 2023-10-14T16:41:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.059+0000 7f83837fe640 1 -- 172.21.15.29:0/4221307575 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c0317a0 con 0x7f838c160b90 2023-10-14T16:41:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.059+0000 7f83837fe640 1 -- 172.21.15.29:0/4221307575 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c031b30 con 0x7f838c160b90 2023-10-14T16:41:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.059+0000 7f83931f0640 1 -- 172.21.15.29:0/4221307575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c160b90 msgr2=0x7f838c160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.059+0000 7f83931f0640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c160b90 0x7f838c160fb0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 -- 172.21.15.29:0/4221307575 shutdown_connections 2023-10-14T16:41:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c160b90 0x7f838c160fb0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 --2- 172.21.15.29:0/4221307575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c15d150 0x7f838c0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 -- 172.21.15.29:0/4221307575 >> 172.21.15.29:0/4221307575 conn(0x7f838c0ce9b0 msgr2=0x7f838c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 -- 172.21.15.29:0/4221307575 shutdown_connections 2023-10-14T16:41:12.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.060+0000 7f83931f0640 1 -- 172.21.15.29:0/4221307575 wait complete. 2023-10-14T16:41:12.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.061+0000 7f83931f0640 1 Processor -- start 2023-10-14T16:41:12.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.062+0000 7f83931f0640 1 -- start start 2023-10-14T16:41:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c069da0 0x7f838c0c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f838c160b90 0x7f838c1f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0c45f0 con 0x7f838c15d150 2023-10-14T16:41:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0c3fe0 con 0x7f838c160b90 2023-10-14T16:41:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f83931f0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f838c0b0da0 con 0x7f838c069da0 2023-10-14T16:41:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f8383fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f8383fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36826/0 (socket says 172.21.15.29:36826) 2023-10-14T16:41:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f8383fff640 1 -- 172.21.15.29:0/2630550490 learned_addr learned my addr 172.21.15.29:0/2630550490 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f8390f8c640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c069da0 0x7f838c0c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.063+0000 7f839178d640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f838c160b90 0x7f838c1f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 -- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c069da0 msgr2=0x7f838c0c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c069da0 0x7f838c0c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 -- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f838c160b90 msgr2=0x7f838c1f63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f838c160b90 0x7f838c1f63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 -- 172.21.15.29:0/2630550490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837c0094d0 con 0x7f838c15d150 2023-10-14T16:41:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f8383fff640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c03ac60 con 0x7f838c15d150 2023-10-14T16:41:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f837c03ade0 con 0x7f838c15d150 2023-10-14T16:41:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f837c044630 con 0x7f838c15d150 2023-10-14T16:41:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.064+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f838c0c4280 con 0x7f838c15d150 2023-10-14T16:41:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.065+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f838c1507f0 con 0x7f838c15d150 2023-10-14T16:41:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.066+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f838c0c4410 con 0x7f838c15d150 2023-10-14T16:41:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.067+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f837c02f050 con 0x7f838c15d150 2023-10-14T16:41:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.067+0000 7f83817fa640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 0x7f837806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.068+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f837c0ab850 con 0x7f838c15d150 2023-10-14T16:41:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.068+0000 7f8390f8c640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 0x7f837806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.068+0000 7f8390f8c640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 0x7f837806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f838800a990 tx=0x7f8388008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.076+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f837c0767e0 con 0x7f838c15d150 2023-10-14T16:41:12.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.293+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f838c04e160 con 0x7f838c15d150 2023-10-14T16:41:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.294+0000 7f83817fa640 1 -- 172.21.15.29:0/2630550490 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f837c079e00 con 0x7f838c15d150 2023-10-14T16:41:12.296 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-14T16:41:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 msgr2=0x7f837806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 0x7f837806d330 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f838800a990 tx=0x7f8388008040 comp rx=0 tx=0).stop 2023-10-14T16:41:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 msgr2=0x7f838c1f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f837c009870 tx=0x7f837c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 shutdown_connections 2023-10-14T16:41:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f837806ae70 0x7f837806d330 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f838c160b90 0x7f838c1f63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f838c15d150 0x7f838c1f5e70 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 --2- 172.21.15.29:0/2630550490 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f838c069da0 0x7f838c0c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 >> 172.21.15.29:0/2630550490 conn(0x7f838c0ce9b0 msgr2=0x7f838c0c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:12.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 shutdown_connections 2023-10-14T16:41:12.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.297+0000 7f83931f0640 1 -- 172.21.15.29:0/2630550490 wait complete. 2023-10-14T16:41:12.319 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:41:12.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:12.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 -f 2023-10-14T16:41:12.400 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:41:12.401 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:12.401 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:41:12.452 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:12.453 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:12.507 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:12.508 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:12.554 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:12.555 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:12.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.861+0000 7f29ffabb640 1 Processor -- start 2023-10-14T16:41:12.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.861+0000 7f29ffabb640 1 -- start start 2023-10-14T16:41:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29ffabb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29f806c3f0 0x7f29f806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29ffabb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f80cd630 con 0x7f29f815e0c0 2023-10-14T16:41:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29ffabb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29f80ce1b0 con 0x7f29f815e540 2023-10-14T16:41:12.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29ffabb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29f806cd10 con 0x7f29f806c3f0 2023-10-14T16:41:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29fd857640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29f806c3f0 0x7f29f806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29fd857640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29f806c3f0 0x7f29f806c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36840/0 (socket says 172.21.15.29:36840) 2023-10-14T16:41:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29fd857640 1 -- 172.21.15.29:0/1386323092 learned_addr learned my addr 172.21.15.29:0/1386323092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:12.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.863+0000 7f29fd056640 1 --1- 172.21.15.29:0/1386323092 >> v1:172.21.15.29:6789/0 conn(0x7f29f815e0c0 0x7f29f80b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48124/0 (socket says 172.21.15.29:48124) 2023-10-14T16:41:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.864+0000 7f29fd857640 1 -- 172.21.15.29:0/1386323092 >> v1:172.21.15.186:6789/0 conn(0x7f29f815e540 legacy=0x7f29f80b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.864+0000 7f29fd857640 1 -- 172.21.15.29:0/1386323092 >> v1:172.21.15.29:6789/0 conn(0x7f29f815e0c0 legacy=0x7f29f80b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.864+0000 7f29fd857640 1 -- 172.21.15.29:0/1386323092 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f80d2170 con 0x7f29f806c3f0 2023-10-14T16:41:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29fd857640 1 --2- 172.21.15.29:0/1386323092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f806c3f0 0x7f29f806c7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009870 tx=0x7f29f402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d3a7991df1d39d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29fc855640 1 -- 172.21.15.29:0/1386323092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f402f850 con 0x7f29f806c3f0 2023-10-14T16:41:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29fc855640 1 -- 172.21.15.29:0/1386323092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29f402f9b0 con 0x7f29f806c3f0 2023-10-14T16:41:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29fc855640 1 -- 172.21.15.29:0/1386323092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f402fd40 con 0x7f29f806c3f0 2023-10-14T16:41:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29ffabb640 1 -- 172.21.15.29:0/1386323092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f806c3f0 msgr2=0x7f29f806c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.865+0000 7f29ffabb640 1 --2- 172.21.15.29:0/1386323092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f806c3f0 0x7f29f806c7d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f29f4009870 tx=0x7f29f402ba30 comp rx=0 tx=0).stop 2023-10-14T16:41:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.866+0000 7f29ffabb640 1 -- 172.21.15.29:0/1386323092 shutdown_connections 2023-10-14T16:41:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.866+0000 7f29ffabb640 1 --2- 172.21.15.29:0/1386323092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f806c3f0 0x7f29f806c7d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.866+0000 7f29ffabb640 1 -- 172.21.15.29:0/1386323092 >> 172.21.15.29:0/1386323092 conn(0x7f29f80d2cb0 msgr2=0x7f29f80d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.866+0000 7f29ffabb640 1 -- 172.21.15.29:0/1386323092 shutdown_connections 2023-10-14T16:41:12.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.866+0000 7f29ffabb640 1 -- 172.21.15.29:0/1386323092 wait complete. 2023-10-14T16:41:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.867+0000 7f29ffabb640 1 Processor -- start 2023-10-14T16:41:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.867+0000 7f29ffabb640 1 -- start start 2023-10-14T16:41:12.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29f806c3f0 0x7f29f814d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f815e540 0x7f29f815a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80cc0e0 con 0x7f29f815e0c0 2023-10-14T16:41:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80cc3c0 con 0x7f29f815e540 2023-10-14T16:41:12.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.868+0000 7f29ffabb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29f80cc6a0 con 0x7f29f806c3f0 2023-10-14T16:41:12.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36844/0 (socket says 172.21.15.29:36844) 2023-10-14T16:41:12.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 -- 172.21.15.29:0/184391637 learned_addr learned my addr 172.21.15.29:0/184391637 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:12.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 -- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29f806c3f0 msgr2=0x7f29f814d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29f806c3f0 0x7f29f814d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 -- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f815e540 msgr2=0x7f29f815a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f815e540 0x7f29f815a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 -- 172.21.15.29:0/184391637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29f40094d0 con 0x7f29f815e0c0 2023-10-14T16:41:12.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29fd056640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f29e800c3e0 tx=0x7f29e800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.869+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80074e0 con 0x7f29f815e0c0 2023-10-14T16:41:12.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.870+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29e8007640 con 0x7f29f815e0c0 2023-10-14T16:41:12.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.870+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29f80cc980 con 0x7f29f815e0c0 2023-10-14T16:41:12.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.870+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29e80049e0 con 0x7f29f815e0c0 2023-10-14T16:41:12.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.870+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29f8156fc0 con 0x7f29f815e0c0 2023-10-14T16:41:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.871+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29f80ccea0 con 0x7f29f815e0c0 2023-10-14T16:41:12.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.872+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f29e8004b40 con 0x7f29f815e0c0 2023-10-14T16:41:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.874+0000 7f29ee7fc640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 0x7f29dc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.874+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f29e8088ff0 con 0x7f29f815e0c0 2023-10-14T16:41:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.874+0000 7f29fd857640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 0x7f29dc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:12.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.875+0000 7f29fd857640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 0x7f29dc06d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f29f4036000 tx=0x7f29f4039040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:12.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:12.878+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29f80ccea0 con 0x7f29f815e0c0 2023-10-14T16:41:13.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.077+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29f80ccea0 con 0x7f29f815e0c0 2023-10-14T16:41:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.080+0000 7f29ee7fc640 1 -- 172.21.15.29:0/184391637 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7f29f80ccea0 con 0x7f29f815e0c0 2023-10-14T16:41:13.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:13.083 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:13.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 msgr2=0x7f29dc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 0x7f29dc06d330 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f29f4036000 tx=0x7f29f4039040 comp rx=0 tx=0).stop 2023-10-14T16:41:13.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 msgr2=0x7f29f80cb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f29e800c3e0 tx=0x7f29e800c8a0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 shutdown_connections 2023-10-14T16:41:13.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f29dc06ae70 0x7f29dc06d330 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29f815e540 0x7f29f815a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29f815e0c0 0x7f29f80cb720 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 --2- 172.21.15.29:0/184391637 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29f806c3f0 0x7f29f814d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 >> 172.21.15.29:0/184391637 conn(0x7f29f80d2cb0 msgr2=0x7f29f80d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:13.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 shutdown_connections 2023-10-14T16:41:13.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.083+0000 7f29ffabb640 1 -- 172.21.15.29:0/184391637 wait complete. 2023-10-14T16:41:13.103 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:13.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:13.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.327+0000 7fa6eb59e640 1 Processor -- start 2023-10-14T16:41:13.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.328+0000 7fa6eb59e640 1 -- start start 2023-10-14T16:41:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.329+0000 7fa6eb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6ec0b3ec0 0x7fa6ec0cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.329+0000 7fa6eb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.329+0000 7fa6eb59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6ec06be30 con 0x7fa6ec0b0ea0 2023-10-14T16:41:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.329+0000 7fa6eb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec06ca00 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.329+0000 7fa6eb59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6ec0ada10 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36852/0 (socket says 172.21.15.29:36852) 2023-10-14T16:41:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 -- 172.21.15.29:0/2583623349 learned_addr learned my addr 172.21.15.29:0/2583623349 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 -- 172.21.15.29:0/2583623349 >> v2:172.21.15.186:3300/0 conn(0x7fa6ec0b3ec0 msgr2=0x7fa6ec0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 --2- 172.21.15.29:0/2583623349 >> v2:172.21.15.186:3300/0 conn(0x7fa6ec0b3ec0 0x7fa6ec0cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 -- 172.21.15.29:0/2583623349 >> v1:172.21.15.186:6789/0 conn(0x7fa6ec0b0ea0 legacy=0x7fa6ec15a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.330+0000 7fa6ead9d640 1 -- 172.21.15.29:0/2583623349 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6ec06c060 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.331+0000 7fa6ead9d640 1 --2- 172.21.15.29:0/2583623349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0742a0 tx=0x7fa6e402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d918badee8c990a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.331+0000 7fa6e9d9b640 1 -- 172.21.15.29:0/2583623349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e402c9f0 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.331+0000 7fa6e9d9b640 1 -- 172.21.15.29:0/2583623349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6e402cb50 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.331+0000 7fa6e9d9b640 1 -- 172.21.15.29:0/2583623349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6e40343f0 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 -- 172.21.15.29:0/2583623349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b43c0 msgr2=0x7fa6ec0cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/2583623349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0742a0 tx=0x7fa6e402be50 comp rx=0 tx=0).stop 2023-10-14T16:41:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 -- 172.21.15.29:0/2583623349 shutdown_connections 2023-10-14T16:41:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/2583623349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec0cd7a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/2583623349 >> v2:172.21.15.186:3300/0 conn(0x7fa6ec0b3ec0 0x7fa6ec0cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 -- 172.21.15.29:0/2583623349 >> 172.21.15.29:0/2583623349 conn(0x7fa6ec0b0170 msgr2=0x7fa6ec0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 -- 172.21.15.29:0/2583623349 shutdown_connections 2023-10-14T16:41:13.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.332+0000 7fa6eb59e640 1 -- 172.21.15.29:0/2583623349 wait complete. 2023-10-14T16:41:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.334+0000 7fa6eb59e640 1 Processor -- start 2023-10-14T16:41:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.334+0000 7fa6eb59e640 1 -- start start 2023-10-14T16:41:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6ec0b0ea0 0x7fa6ec15c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec1f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c3bd0 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c2d20 con 0x7fa6ec0b43c0 2023-10-14T16:41:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6eb59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6ec0c2fc0 con 0x7fa6ec0b0ea0 2023-10-14T16:41:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.335+0000 7fa6e1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36860/0 (socket says 172.21.15.29:36860) 2023-10-14T16:41:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 -- 172.21.15.29:0/1160811844 learned_addr learned my addr 172.21.15.29:0/1160811844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6ea59c640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6ec0b0ea0 0x7fa6ec15c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6ead9d640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec1f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 -- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6ec0b0ea0 msgr2=0x7fa6ec15c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6ec0b0ea0 0x7fa6ec15c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 -- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ec0b43c0 msgr2=0x7fa6ec1f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec1f26e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.336+0000 7fa6e1d9b640 1 -- 172.21.15.29:0/1160811844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e4009b30 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6e1d9b640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b3e0 tx=0x7fa6d400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011830 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6d40119b0 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6d4011d60 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6ec0c32a0 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.337+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6ec0c3860 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.338+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6ec04e9b0 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.339+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa6d401a400 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.340+0000 7fa6e37fe640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 0x7fa6cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.340+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa6d4089810 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.340+0000 7fa6ea59c640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 0x7fa6cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.341+0000 7fa6ea59c640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 0x7fa6cc06d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0c4290 tx=0x7fa6dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.344+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6d4054840 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.530+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa6ec0c4a80 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.531+0000 7fa6e37fe640 1 -- 172.21.15.29:0/1160811844 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa6d4015020 con 0x7fa6ec0b3ec0 2023-10-14T16:41:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:13.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.533+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 msgr2=0x7fa6cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.533+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 0x7fa6cc06d330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec0c4290 tx=0x7fa6dc008040 comp rx=0 tx=0).stop 2023-10-14T16:41:13.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.533+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 msgr2=0x7fa6ec0c4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.533+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fa6d400b3e0 tx=0x7fa6d400b8a0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 shutdown_connections 2023-10-14T16:41:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa6cc06ae70 0x7fa6cc06d330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6ec0b43c0 0x7fa6ec1f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6ec0b3ec0 0x7fa6ec0c4ca0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 --2- 172.21.15.29:0/1160811844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6ec0b0ea0 0x7fa6ec15c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 >> 172.21.15.29:0/1160811844 conn(0x7fa6ec0b0170 msgr2=0x7fa6ec0b0550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:13.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 shutdown_connections 2023-10-14T16:41:13.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.534+0000 7fa6eb59e640 1 -- 172.21.15.29:0/1160811844 wait complete. 2023-10-14T16:41:13.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.765+0000 7fdeddde3640 1 Processor -- start 2023-10-14T16:41:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.766+0000 7fdeddde3640 1 -- start start 2023-10-14T16:41:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.767+0000 7fdeddde3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fded80ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.767+0000 7fdeddde3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded815b1f0 con 0x7fded80cf330 2023-10-14T16:41:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.767+0000 7fdeddde3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fded806ce20 con 0x7fded806d570 2023-10-14T16:41:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.767+0000 7fdeddde3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded806f650 con 0x7fded815ae10 2023-10-14T16:41:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fded80cf330 0x7fded806c710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48134/0 (socket says 172.21.15.29:48134) 2023-10-14T16:41:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded7fff640 1 -- 172.21.15.29:0/745852277 learned_addr learned my addr 172.21.15.29:0/745852277 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded77fe640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fded80ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196397036 0 0) 0x7fded0001490 con 0x7fded80cf330 2023-10-14T16:41:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdec4003520 con 0x7fded80cf330 2023-10-14T16:41:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 396298802 0 0) 0x7fdec4003520 con 0x7fded80cf330 2023-10-14T16:41:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 msgr2=0x7fded80ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.768+0000 7fded6ffd640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fded80ca590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 >> v1:172.21.15.186:6789/0 conn(0x7fded806d570 legacy=0x7fded806c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded806bb70 con 0x7fded80cf330 2023-10-14T16:41:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fded0001b80 con 0x7fded80cf330 2023-10-14T16:41:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 >> v1:172.21.15.29:6789/0 conn(0x7fded80cf330 legacy=0x7fded806c710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fdec40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdec4003ee0 0x7fdec40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400ba30 con 0x7fdec4004d00 2023-10-14T16:41:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400bb70 con 0x7fdec4003ee0 2023-10-14T16:41:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdec400bd30 con 0x7fded815ae10 2023-10-14T16:41:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded7fff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.769+0000 7fded77fe640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fdec40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fdecffff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdec4003ee0 0x7fdec40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded7fff640 1 -- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 msgr2=0x7fdec40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded7fff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fdec40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded7fff640 1 -- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdec4003ee0 msgr2=0x7fdec40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:13.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded7fff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdec4003ee0 0x7fdec40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded7fff640 1 -- 172.21.15.29:0/745852277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec4003520 con 0x7fdec4004d00 2023-10-14T16:41:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fded77fe640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded815ae10 0x7fdec40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.770+0000 7fdecffff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdec4003ee0 0x7fdec40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:13.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.771+0000 7fded7fff640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fded000a780 tx=0x7fdec40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8956d24720e984a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.771+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded00072d0 con 0x7fdec4004d00 2023-10-14T16:41:13.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.771+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded002f070 con 0x7fdec4004d00 2023-10-14T16:41:13.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.771+0000 7fded6ffd640 1 -- 172.21.15.29:0/745852277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fded0007670 con 0x7fdec4004d00 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 -- 172.21.15.29:0/745852277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 msgr2=0x7fdec4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 --2- 172.21.15.29:0/745852277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fdec4005180 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fded000a780 tx=0x7fdec40057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fded7fff640 1 -- 172.21.15.29:0/745852277 reap_dead start 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 -- 172.21.15.29:0/745852277 shutdown_connections 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 -- 172.21.15.29:0/745852277 >> 172.21.15.29:0/745852277 conn(0x7fded80ce840 msgr2=0x7fded80cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 -- 172.21.15.29:0/745852277 shutdown_connections 2023-10-14T16:41:13.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.772+0000 7fdeddde3640 1 -- 172.21.15.29:0/745852277 wait complete. 2023-10-14T16:41:13.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.774+0000 7fdeddde3640 1 Processor -- start 2023-10-14T16:41:13.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.774+0000 7fdeddde3640 1 -- start start 2023-10-14T16:41:13.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 0x7fded8156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded8156980 0x7fded80ca260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c3470 con 0x7fdec4004d00 2023-10-14T16:41:13.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c3d10 con 0x7fded8156980 2023-10-14T16:41:13.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fdeddde3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded80c46e0 con 0x7fded80c5910 2023-10-14T16:41:13.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fded77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fded77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36872/0 (socket says 172.21.15.29:36872) 2023-10-14T16:41:13.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.775+0000 7fded77fe640 1 -- 172.21.15.29:0/1682009955 learned_addr learned my addr 172.21.15.29:0/1682009955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:13.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fdecffff640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 0x7fded8156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded7fff640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded8156980 0x7fded80ca260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 -- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 msgr2=0x7fded8156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 0x7fded8156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 -- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded8156980 msgr2=0x7fded80ca260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded8156980 0x7fded80ca260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 -- 172.21.15.29:0/1682009955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded8156750 con 0x7fdec4004d00 2023-10-14T16:41:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fdecffff640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 0x7fded8156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded77fe640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdec80116d0 tx=0x7fdec8011b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.776+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec800f9c0 con 0x7fdec4004d00 2023-10-14T16:41:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.777+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec800fb20 con 0x7fdec4004d00 2023-10-14T16:41:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.777+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdec800fed0 con 0x7fdec4004d00 2023-10-14T16:41:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.777+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded80c5e80 con 0x7fdec4004d00 2023-10-14T16:41:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.777+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded80c39a0 con 0x7fdec4004d00 2023-10-14T16:41:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.778+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fded804e160 con 0x7fdec4004d00 2023-10-14T16:41:13.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.779+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdec8015460 con 0x7fdec4004d00 2023-10-14T16:41:13.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.780+0000 7fded4ff9640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 0x7fdeb806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:13.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.780+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdec8089fe0 con 0x7fdec4004d00 2023-10-14T16:41:13.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.780+0000 7fdecffff640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 0x7fdeb806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:13.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.781+0000 7fdecffff640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 0x7fdeb806d360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdec0001270 tx=0x7fdec0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:13.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.785+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdec8054fb0 con 0x7fdec4004d00 2023-10-14T16:41:13.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.981+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fded8070400 con 0x7fdec4004d00 2023-10-14T16:41:13.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.981+0000 7fded4ff9640 1 -- 172.21.15.29:0/1682009955 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fdec80585d0 con 0x7fdec4004d00 2023-10-14T16:41:13.984 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:13.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 msgr2=0x7fdeb806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 0x7fdeb806d360 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fdec0001270 tx=0x7fdec0009040 comp rx=0 tx=0).stop 2023-10-14T16:41:13.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 msgr2=0x7fded80c41a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:13.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdec80116d0 tx=0x7fdec8011b90 comp rx=0 tx=0).stop 2023-10-14T16:41:13.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 shutdown_connections 2023-10-14T16:41:13.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdeb806aea0 0x7fdeb806d360 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fded8156980 0x7fded80ca260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fded80c5910 0x7fded8156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 --2- 172.21.15.29:0/1682009955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdec4004d00 0x7fded80c41a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:13.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 >> 172.21.15.29:0/1682009955 conn(0x7fded80ce840 msgr2=0x7fded80cd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:13.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 shutdown_connections 2023-10-14T16:41:13.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:13.984+0000 7fdeddde3640 1 -- 172.21.15.29:0/1682009955 wait complete. 2023-10-14T16:41:14.004 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:14.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.230+0000 7f5452ebb640 1 Processor -- start 2023-10-14T16:41:14.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.230+0000 7f5452ebb640 1 -- start start 2023-10-14T16:41:14.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0addb0 0x7f544c0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f544c0c9000 0x7f544c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f544c06a2e0 0x7f544c0cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544c161a50 con 0x7f544c0c9000 2023-10-14T16:41:14.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f544c06a750 con 0x7f544c06a2e0 2023-10-14T16:41:14.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.231+0000 7f5452ebb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544c0cb140 con 0x7f544c0addb0 2023-10-14T16:41:14.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f544c0c9000 0x7f544c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f544c0c9000 0x7f544c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36876/0 (socket says 172.21.15.29:36876) 2023-10-14T16:41:14.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 -- 172.21.15.29:0/3320517092 learned_addr learned my addr 172.21.15.29:0/3320517092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 -- 172.21.15.29:0/3320517092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0addb0 msgr2=0x7f544c0caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 --2- 172.21.15.29:0/3320517092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0addb0 0x7f544c0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 -- 172.21.15.29:0/3320517092 >> v2:172.21.15.186:3300/0 conn(0x7f544c06a2e0 msgr2=0x7f544c0cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 --2- 172.21.15.29:0/3320517092 >> v2:172.21.15.186:3300/0 conn(0x7f544c06a2e0 0x7f544c0cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.232+0000 7f5443fff640 1 -- 172.21.15.29:0/3320517092 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f544c06b030 con 0x7f544c0c9000 2023-10-14T16:41:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.233+0000 7f5443fff640 1 --2- 172.21.15.29:0/3320517092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0c9000 0x7f544c069da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f544c0cf850 tx=0x7f543c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47f6cc7e18396be server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.234+0000 7f54437fe640 1 -- 172.21.15.29:0/3320517092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c031860 con 0x7f544c0c9000 2023-10-14T16:41:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.234+0000 7f54437fe640 1 -- 172.21.15.29:0/3320517092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c0319c0 con 0x7f544c0c9000 2023-10-14T16:41:14.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.234+0000 7f54437fe640 1 -- 172.21.15.29:0/3320517092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c031d50 con 0x7f544c0c9000 2023-10-14T16:41:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 -- 172.21.15.29:0/3320517092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0c9000 msgr2=0x7f544c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3320517092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0c9000 0x7f544c069da0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f544c0cf850 tx=0x7f543c02b900 comp rx=0 tx=0).stop 2023-10-14T16:41:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 -- 172.21.15.29:0/3320517092 shutdown_connections 2023-10-14T16:41:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3320517092 >> v2:172.21.15.186:3300/0 conn(0x7f544c06a2e0 0x7f544c0cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3320517092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0c9000 0x7f544c069da0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3320517092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c0addb0 0x7f544c0caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 -- 172.21.15.29:0/3320517092 >> 172.21.15.29:0/3320517092 conn(0x7f544c0ce9b0 msgr2=0x7f544c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:14.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 -- 172.21.15.29:0/3320517092 shutdown_connections 2023-10-14T16:41:14.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.235+0000 7f5452ebb640 1 -- 172.21.15.29:0/3320517092 wait complete. 2023-10-14T16:41:14.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.237+0000 7f5452ebb640 1 Processor -- start 2023-10-14T16:41:14.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.237+0000 7f5452ebb640 1 -- start start 2023-10-14T16:41:14.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 0x7f544c0c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 0x7f544c15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c15d750 con 0x7f544c0addb0 2023-10-14T16:41:14.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f544c15c8a0 con 0x7f544c0c9000 2023-10-14T16:41:14.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5452ebb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f544c15cb40 con 0x7f544c06a2e0 2023-10-14T16:41:14.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5443fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5443fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36878/0 (socket says 172.21.15.29:36878) 2023-10-14T16:41:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.238+0000 7f5443fff640 1 -- 172.21.15.29:0/3385079081 learned_addr learned my addr 172.21.15.29:0/3385079081 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5450c57640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 0x7f544c0c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 -- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 msgr2=0x7f544c0c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5451458640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 0x7f544c15c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 0x7f544c0c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 -- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 msgr2=0x7f544c15c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 0x7f544c15c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 -- 172.21.15.29:0/3385079081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f543c0094d0 con 0x7f544c0addb0 2023-10-14T16:41:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5451458640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 0x7f544c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5450c57640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 0x7f544c0c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5443fff640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f543c0090b0 tx=0x7f543c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c031810 con 0x7f544c0addb0 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f543c0443f0 con 0x7f544c0addb0 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f543c0447a0 con 0x7f544c0addb0 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.239+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f544c15ce20 con 0x7f544c0addb0 2023-10-14T16:41:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.240+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f544c15d3e0 con 0x7f544c0addb0 2023-10-14T16:41:14.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.241+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f544c04e160 con 0x7f544c0addb0 2023-10-14T16:41:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.242+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f543c044900 con 0x7f544c0addb0 2023-10-14T16:41:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.242+0000 7f54417fa640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 0x7f543806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.243+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f543c0abfb0 con 0x7f544c0addb0 2023-10-14T16:41:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.243+0000 7f5450c57640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 0x7f543806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.244+0000 7f5450c57640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 0x7f543806d360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5448006fd0 tx=0x7f5448008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.247+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f543c076f40 con 0x7f544c0addb0 2023-10-14T16:41:14.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.443+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f544c06c5b0 con 0x7f544c0addb0 2023-10-14T16:41:14.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.444+0000 7f54417fa640 1 -- 172.21.15.29:0/3385079081 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f543c07a560 con 0x7f544c0addb0 2023-10-14T16:41:14.446 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 msgr2=0x7f543806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 0x7f543806d360 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5448006fd0 tx=0x7f5448008040 comp rx=0 tx=0).stop 2023-10-14T16:41:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 msgr2=0x7f544c1f6690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f543c0090b0 tx=0x7f543c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 shutdown_connections 2023-10-14T16:41:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f543806aea0 0x7f543806d360 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f544c0c9000 0x7f544c15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f544c0addb0 0x7f544c1f6690 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 --2- 172.21.15.29:0/3385079081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f544c06a2e0 0x7f544c0c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 >> 172.21.15.29:0/3385079081 conn(0x7f544c0ce9b0 msgr2=0x7f544c0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:14.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 shutdown_connections 2023-10-14T16:41:14.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.447+0000 7f5452ebb640 1 -- 172.21.15.29:0/3385079081 wait complete. 2023-10-14T16:41:14.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:41:14.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.689+0000 7fa571c34640 1 Processor -- start 2023-10-14T16:41:14.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.689+0000 7fa571c34640 1 -- start start 2023-10-14T16:41:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa571c34640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa571c34640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c0c9ca0 0x7fa56c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa571c34640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56c0cb030 con 0x7fa56c15ad70 2023-10-14T16:41:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa571c34640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56c0cae30 con 0x7fa56c06a4b0 2023-10-14T16:41:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa571c34640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c91c0 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36890/0 (socket says 172.21.15.29:36890) 2023-10-14T16:41:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 -- 172.21.15.29:0/3942068949 learned_addr learned my addr 172.21.15.29:0/3942068949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa56bfff640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c0c9ca0 0x7fa56c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 -- 172.21.15.29:0/3942068949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c0c9ca0 msgr2=0x7fa56c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c0c9ca0 0x7fa56c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 -- 172.21.15.29:0/3942068949 >> v1:172.21.15.186:6789/0 conn(0x7fa56c15ad70 legacy=0x7fa56c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:14.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.691+0000 7fa563fff640 1 -- 172.21.15.29:0/3942068949 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56c0c94a0 con 0x7fa56c06a4b0 2023-10-14T16:41:14.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.692+0000 7fa563fff640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa55402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10695a29a7979efb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.692+0000 7fa56affd640 1 -- 172.21.15.29:0/3942068949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554031640 con 0x7fa56c06a4b0 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.692+0000 7fa56affd640 1 -- 172.21.15.29:0/3942068949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5540317a0 con 0x7fa56c06a4b0 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.692+0000 7fa56affd640 1 -- 172.21.15.29:0/3942068949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554031b30 con 0x7fa56c06a4b0 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 -- 172.21.15.29:0/3942068949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c06a4b0 msgr2=0x7fa56c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa55402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 -- 172.21.15.29:0/3942068949 shutdown_connections 2023-10-14T16:41:14.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c0c9ca0 0x7fa56c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 --2- 172.21.15.29:0/3942068949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c06a4b0 0x7fa56c0ca3b0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 -- 172.21.15.29:0/3942068949 >> 172.21.15.29:0/3942068949 conn(0x7fa56c0ce9b0 msgr2=0x7fa56c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 -- 172.21.15.29:0/3942068949 shutdown_connections 2023-10-14T16:41:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.693+0000 7fa571c34640 1 -- 172.21.15.29:0/3942068949 wait complete. 2023-10-14T16:41:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.695+0000 7fa571c34640 1 Processor -- start 2023-10-14T16:41:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.695+0000 7fa571c34640 1 -- start start 2023-10-14T16:41:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 0x7fa56c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa56c15ad70 0x7fa56c0c5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c49b0 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0c27c0 con 0x7fa56c15ad70 2023-10-14T16:41:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.696+0000 7fa571c34640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56c0af6a0 con 0x7fa56c06a4b0 2023-10-14T16:41:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36896/0 (socket says 172.21.15.29:36896) 2023-10-14T16:41:14.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 -- 172.21.15.29:0/2747358371 learned_addr learned my addr 172.21.15.29:0/2747358371 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:14.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa56b7fe640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 0x7fa56c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa56bfff640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa56c15ad70 0x7fa56c0c5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 -- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 msgr2=0x7fa56c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 0x7fa56c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 -- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa56c15ad70 msgr2=0x7fa56c0c5310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa56c15ad70 0x7fa56c0c5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa563fff640 1 -- 172.21.15.29:0/2747358371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540094d0 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.697+0000 7fa56b7fe640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 0x7fa56c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa563fff640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa5540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa55403ac60 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55403adc0 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa56c0c4010 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa554044630 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.698+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa56c0c45d0 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.699+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa56c0c4760 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.700+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa55402f050 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.701+0000 7fa568ff9640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 0x7fa54c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.701+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa5540ab570 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.702+0000 7fa56b7fe640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 0x7fa54c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.703+0000 7fa56b7fe640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 0x7fa54c06d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00a830 tx=0x7fa55c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.706+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa554076590 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.915+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fa56c04e160 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.915+0000 7fa568ff9640 1 -- 172.21.15.29:0/2747358371 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7fa554031e60 con 0x7fa56c0c9ca0 2023-10-14T16:41:14.918 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-14T16:41:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 msgr2=0x7fa54c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 0x7fa54c06d330 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00a830 tx=0x7fa55c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 msgr2=0x7fa56c0c4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fa554009870 tx=0x7fa5540046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 shutdown_connections 2023-10-14T16:41:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa54c06ae70 0x7fa54c06d330 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa56c15ad70 0x7fa56c0c5310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa56c0c9ca0 0x7fa56c0c4dd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 --2- 172.21.15.29:0/2747358371 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa56c06a4b0 0x7fa56c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:14.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.918+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 >> 172.21.15.29:0/2747358371 conn(0x7fa56c0ce9b0 msgr2=0x7fa56c0cd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.919+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 shutdown_connections 2023-10-14T16:41:14.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:14.919+0000 7fa571c34640 1 -- 172.21.15.29:0/2747358371 wait complete. 2023-10-14T16:41:14.940 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1... 2023-10-14T16:41:14.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:14.941 DEBUG:teuthology.orchestra.run.smithi186:> sudo umount /home/ubuntu/cephtest/mnt.1 -f 2023-10-14T16:41:15.049 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:41:15.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:15.050 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2023-10-14T16:41:15.115 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:15.115 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:15.181 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:15.182 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:15.239 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:15.240 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:15.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:15.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.526+0000 7f51527f4640 1 Processor -- start 2023-10-14T16:41:15.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.526+0000 7f51527f4640 1 -- start start 2023-10-14T16:41:15.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.527+0000 7f51527f4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 0x7f514c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.527+0000 7f51527f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.527+0000 7f51527f4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f514c06c960 con 0x7f514c0ad970 2023-10-14T16:41:15.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.527+0000 7f51527f4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f514c15b9d0 con 0x7f514c06c040 2023-10-14T16:41:15.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.527+0000 7f51527f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c15bcb0 con 0x7f514c0cda80 2023-10-14T16:41:15.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.528+0000 7f5151ff3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.528+0000 7f5151ff3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:39616/0 (socket says 172.21.15.29:39616) 2023-10-14T16:41:15.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.528+0000 7f5151ff3640 1 -- 172.21.15.29:0/4025074166 learned_addr learned my addr 172.21.15.29:0/4025074166 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:15.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.529+0000 7f51517f2640 1 --2- 172.21.15.29:0/4025074166 >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 0x7f514c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.530+0000 7f5151ff3640 1 -- 172.21.15.29:0/4025074166 >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 msgr2=0x7f514c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:15.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.530+0000 7f5151ff3640 1 --2- 172.21.15.29:0/4025074166 >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 0x7f514c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.530+0000 7f5151ff3640 1 -- 172.21.15.29:0/4025074166 >> v1:172.21.15.186:6789/0 conn(0x7f514c0ad970 legacy=0x7f514c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.530+0000 7f5151ff3640 1 -- 172.21.15.29:0/4025074166 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514c15bf90 con 0x7f514c0cda80 2023-10-14T16:41:15.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.530+0000 7f51517f2640 1 --2- 172.21.15.29:0/4025074166 >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 0x7f514c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:15.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.531+0000 7f5151ff3640 1 --2- 172.21.15.29:0/4025074166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f514402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58de8b0391785b44 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:15.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f5143fff640 1 -- 172.21.15.29:0/4025074166 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514402cb00 con 0x7f514c0cda80 2023-10-14T16:41:15.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f5143fff640 1 -- 172.21.15.29:0/4025074166 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514402cc60 con 0x7f514c0cda80 2023-10-14T16:41:15.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f5143fff640 1 -- 172.21.15.29:0/4025074166 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51440343f0 con 0x7f514c0cda80 2023-10-14T16:41:15.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 -- 172.21.15.29:0/4025074166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 msgr2=0x7f514c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 --2- 172.21.15.29:0/4025074166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f514402bf60 comp rx=0 tx=0).stop 2023-10-14T16:41:15.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 -- 172.21.15.29:0/4025074166 shutdown_connections 2023-10-14T16:41:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 --2- 172.21.15.29:0/4025074166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0cda80 0x7f514c15ed90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 --2- 172.21.15.29:0/4025074166 >> v2:172.21.15.186:3300/0 conn(0x7f514c06c040 0x7f514c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 -- 172.21.15.29:0/4025074166 >> 172.21.15.29:0/4025074166 conn(0x7f514c0ce6a0 msgr2=0x7f514c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 -- 172.21.15.29:0/4025074166 shutdown_connections 2023-10-14T16:41:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.533+0000 7f51527f4640 1 -- 172.21.15.29:0/4025074166 wait complete. 2023-10-14T16:41:15.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.534+0000 7f51527f4640 1 Processor -- start 2023-10-14T16:41:15.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.535+0000 7f51527f4640 1 -- start start 2023-10-14T16:41:15.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.535+0000 7f51527f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 0x7f514c0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51527f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0ad970 0x7f514c0c48b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51527f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51527f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b3cc0 con 0x7f514c0cda80 2023-10-14T16:41:15.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51527f4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0b39e0 con 0x7f514c06c040 2023-10-14T16:41:15.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51527f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f514c0c6290 con 0x7f514c0ad970 2023-10-14T16:41:15.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36910/0 (socket says 172.21.15.29:36910) 2023-10-14T16:41:15.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 -- 172.21.15.29:0/2130455805 learned_addr learned my addr 172.21.15.29:0/2130455805 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:15.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51517f2640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 0x7f514c0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5150ff1640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0ad970 0x7f514c0c48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 -- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0ad970 msgr2=0x7f514c0c48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0ad970 0x7f514c0c48b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 -- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 msgr2=0x7f514c0ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 0x7f514c0ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f5151ff3640 1 -- 172.21.15.29:0/2130455805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5144009b30 con 0x7f514c0cda80 2023-10-14T16:41:15.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.536+0000 7f51517f2640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 0x7f514c0ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:15.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f5151ff3640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f5144004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:15.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144040670 con 0x7f514c0cda80 2023-10-14T16:41:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51440407d0 con 0x7f514c0cda80 2023-10-14T16:41:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f514c0c6570 con 0x7f514c0cda80 2023-10-14T16:41:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5144040b80 con 0x7f514c0cda80 2023-10-14T16:41:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.537+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f514c0c4df0 con 0x7f514c0cda80 2023-10-14T16:41:15.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.538+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f514c06b330 con 0x7f514c0cda80 2023-10-14T16:41:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.539+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5144040ce0 con 0x7f514c0cda80 2023-10-14T16:41:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.540+0000 7f5141ffb640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 0x7f513006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.540+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f51440ac780 con 0x7f514c0cda80 2023-10-14T16:41:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.541+0000 7f51517f2640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 0x7f513006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:15.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.541+0000 7f51517f2640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 0x7f513006d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5148006fd0 tx=0x7f5148008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:15.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.546+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5144077710 con 0x7f514c0cda80 2023-10-14T16:41:15.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.743+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f514c0c60f0 con 0x7f514c0cda80 2023-10-14T16:41:15.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.746+0000 7f5141ffb640 1 -- 172.21.15.29:0/2130455805 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7f514c0c60f0 con 0x7f514c0cda80 2023-10-14T16:41:15.747 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:15.747 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 msgr2=0x7f513006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 0x7f513006d330 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5148006fd0 tx=0x7f5148008040 comp rx=0 tx=0).stop 2023-10-14T16:41:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 msgr2=0x7f514c155180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:15.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f5144009e50 tx=0x7f5144004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 shutdown_connections 2023-10-14T16:41:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f513006ae70 0x7f513006d330 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f514c0cda80 0x7f514c155180 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f514c0ad970 0x7f514c0c48b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 --2- 172.21.15.29:0/2130455805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f514c06c040 0x7f514c0ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 >> 172.21.15.29:0/2130455805 conn(0x7f514c0ce6a0 msgr2=0x7f514c15e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 shutdown_connections 2023-10-14T16:41:15.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:15.749+0000 7f51527f4640 1 -- 172.21.15.29:0/2130455805 wait complete. 2023-10-14T16:41:15.768 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:15.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.000+0000 7f9e5dbe9640 1 Processor -- start 2023-10-14T16:41:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.000+0000 7f9e5dbe9640 1 -- start start 2023-10-14T16:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.001+0000 7f9e5dbe9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.001+0000 7f9e5dbe9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e580b43c0 0x7f9e580cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.001+0000 7f9e5dbe9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5806be30 con 0x7f9e580b0ea0 2023-10-14T16:41:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.001+0000 7f9e5dbe9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5806ca00 con 0x7f9e580b43c0 2023-10-14T16:41:16.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.001+0000 7f9e5dbe9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580ada10 con 0x7f9e580b3ec0 2023-10-14T16:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e580b43c0 0x7f9e580cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e580b43c0 0x7f9e580cd7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36916/0 (socket says 172.21.15.29:36916) 2023-10-14T16:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/1048218345 learned_addr learned my addr 172.21.15.29:0/1048218345 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5cbe7640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/1048218345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 msgr2=0x7f9e580cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/1048218345 >> v1:172.21.15.186:6789/0 conn(0x7f9e580b0ea0 legacy=0x7f9e5815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.002+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/1048218345 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5806c060 con 0x7f9e580b43c0 2023-10-14T16:41:16.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.004+0000 7f9e5d3e8640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580cd7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009e50 tx=0x7f9e5002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86602de60f4c4bdd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.004+0000 7f9e577fe640 1 -- 172.21.15.29:0/1048218345 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5002ca80 con 0x7f9e580b43c0 2023-10-14T16:41:16.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.004+0000 7f9e577fe640 1 -- 172.21.15.29:0/1048218345 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5002cbe0 con 0x7f9e580b43c0 2023-10-14T16:41:16.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.004+0000 7f9e577fe640 1 -- 172.21.15.29:0/1048218345 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e500343f0 con 0x7f9e580b43c0 2023-10-14T16:41:16.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/1048218345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 msgr2=0x7f9e580cd7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580cd7a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009e50 tx=0x7f9e5002bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/1048218345 shutdown_connections 2023-10-14T16:41:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580cd7a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/1048218345 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/1048218345 >> 172.21.15.29:0/1048218345 conn(0x7f9e580b0170 msgr2=0x7f9e580b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/1048218345 shutdown_connections 2023-10-14T16:41:16.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.005+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/1048218345 wait complete. 2023-10-14T16:41:16.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.007+0000 7f9e5dbe9640 1 Processor -- start 2023-10-14T16:41:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.007+0000 7f9e5dbe9640 1 -- start start 2023-10-14T16:41:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 0x7f9e580a8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580c3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580c7a90 con 0x7f9e580b43c0 2023-10-14T16:41:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580c3c90 con 0x7f9e580b0ea0 2023-10-14T16:41:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5dbe9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e580c3f70 con 0x7f9e580b3ec0 2023-10-14T16:41:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36930/0 (socket says 172.21.15.29:36930) 2023-10-14T16:41:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/3181712440 learned_addr learned my addr 172.21.15.29:0/3181712440 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5cbe7640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 0x7f9e580a8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 msgr2=0x7f9e580c3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580c3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 msgr2=0x7f9e580a8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 0x7f9e580a8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5d3e8640 1 -- 172.21.15.29:0/3181712440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e50009b30 con 0x7f9e580b43c0 2023-10-14T16:41:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.008+0000 7f9e5cbe7640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 0x7f9e580a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:16.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e5d3e8640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009e50 tx=0x7f9e50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e50009510 con 0x7f9e580b43c0 2023-10-14T16:41:16.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e5003e450 con 0x7f9e580b43c0 2023-10-14T16:41:16.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e5003e800 con 0x7f9e580b43c0 2023-10-14T16:41:16.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e580c4250 con 0x7f9e580b43c0 2023-10-14T16:41:16.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.009+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e580c4810 con 0x7f9e580b43c0 2023-10-14T16:41:16.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.010+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e580c4a00 con 0x7f9e580b43c0 2023-10-14T16:41:16.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.012+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9e5002cd50 con 0x7f9e580b43c0 2023-10-14T16:41:16.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.013+0000 7f9e557fa640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 0x7f9e3c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.013+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9e500ac470 con 0x7f9e580b43c0 2023-10-14T16:41:16.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.014+0000 7f9e5cbe7640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 0x7f9e3c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.014+0000 7f9e5cbe7640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 0x7f9e3c06d330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c006fd0 tx=0x7f9e4c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.017+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e50077610 con 0x7f9e580b43c0 2023-10-14T16:41:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.217+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9e5804e9b0 con 0x7f9e580b43c0 2023-10-14T16:41:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.218+0000 7f9e557fa640 1 -- 172.21.15.29:0/3181712440 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9e500430a0 con 0x7f9e580b43c0 2023-10-14T16:41:16.218 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.220+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 msgr2=0x7f9e3c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.220+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 0x7f9e3c06d330 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c006fd0 tx=0x7f9e4c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.220+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 msgr2=0x7f9e580c3680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.220+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f9e50009e50 tx=0x7f9e50004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 shutdown_connections 2023-10-14T16:41:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9e3c06ae70 0x7f9e3c06d330 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e580b43c0 0x7f9e580c3680 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e580b3ec0 0x7f9e580c3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 --2- 172.21.15.29:0/3181712440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e580b0ea0 0x7f9e580a8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 >> 172.21.15.29:0/3181712440 conn(0x7f9e580b0170 msgr2=0x7f9e580af580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 shutdown_connections 2023-10-14T16:41:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.221+0000 7f9e5dbe9640 1 -- 172.21.15.29:0/3181712440 wait complete. 2023-10-14T16:41:16.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:16.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.471+0000 7fae2759e640 1 Processor -- start 2023-10-14T16:41:16.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.472+0000 7fae2759e640 1 -- start start 2023-10-14T16:41:16.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.473+0000 7fae2759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae280ad7a0 0x7fae28070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.473+0000 7fae2759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae280adc00 0x7fae28070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.473+0000 7fae2759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae280ae1a0 con 0x7fae28160c80 2023-10-14T16:41:16.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.473+0000 7fae2759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae280ae480 con 0x7fae280adc00 2023-10-14T16:41:16.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.473+0000 7fae2759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae280ae760 con 0x7fae280ad7a0 2023-10-14T16:41:16.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae26d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae280adc00 0x7fae28070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae2659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae28160c80 0x7fae28161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48146/0 (socket says 172.21.15.29:48146) 2023-10-14T16:41:16.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae2659c640 1 -- 172.21.15.29:0/680060504 learned_addr learned my addr 172.21.15.29:0/680060504 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae25d9b640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae280ad7a0 0x7fae28070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae26d9d640 1 -- 172.21.15.29:0/680060504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae280ad7a0 msgr2=0x7fae28070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae26d9d640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae280ad7a0 0x7fae28070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae26d9d640 1 -- 172.21.15.29:0/680060504 >> v1:172.21.15.29:6789/0 conn(0x7fae28160c80 legacy=0x7fae28161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae26d9d640 1 -- 172.21.15.29:0/680060504 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae280aea40 con 0x7fae280adc00 2023-10-14T16:41:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.474+0000 7fae2559a640 1 -- 172.21.15.29:0/680060504 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228147573 0 0) 0x7fae280ae1a0 con 0x7fae28160c80 2023-10-14T16:41:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.475+0000 7fae26d9d640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae28070780 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fae200097a0 tx=0x7fae2002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e768af0775ea283 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2559a640 1 -- 172.21.15.29:0/680060504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae2002d040 con 0x7fae280adc00 2023-10-14T16:41:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2559a640 1 -- 172.21.15.29:0/680060504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae20031810 con 0x7fae280adc00 2023-10-14T16:41:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2559a640 1 -- 172.21.15.29:0/680060504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae20031ba0 con 0x7fae280adc00 2023-10-14T16:41:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 -- 172.21.15.29:0/680060504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 msgr2=0x7fae28070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae28070780 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fae200097a0 tx=0x7fae2002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 -- 172.21.15.29:0/680060504 shutdown_connections 2023-10-14T16:41:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae28070780 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 --2- 172.21.15.29:0/680060504 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae280ad7a0 0x7fae28070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.476+0000 7fae2759e640 1 -- 172.21.15.29:0/680060504 >> 172.21.15.29:0/680060504 conn(0x7fae280cc230 msgr2=0x7fae280cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:16.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.477+0000 7fae2759e640 1 -- 172.21.15.29:0/680060504 shutdown_connections 2023-10-14T16:41:16.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.477+0000 7fae2759e640 1 -- 172.21.15.29:0/680060504 wait complete. 2023-10-14T16:41:16.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.478+0000 7fae2759e640 1 Processor -- start 2023-10-14T16:41:16.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.478+0000 7fae2759e640 1 -- start start 2023-10-14T16:41:16.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae280ad7a0 0x7fae281f1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae28160c80 0x7fae280c4b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae281e7970 con 0x7fae280adc00 2023-10-14T16:41:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae281e70d0 con 0x7fae280ad7a0 2023-10-14T16:41:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.479+0000 7fae2759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae281e73b0 con 0x7fae28160c80 2023-10-14T16:41:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36952/0 (socket says 172.21.15.29:36952) 2023-10-14T16:41:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 -- 172.21.15.29:0/3706183206 learned_addr learned my addr 172.21.15.29:0/3706183206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae2659c640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae280ad7a0 0x7fae281f1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae26d9d640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae28160c80 0x7fae280c4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 -- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae28160c80 msgr2=0x7fae280c4b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae28160c80 0x7fae280c4b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 -- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae280ad7a0 msgr2=0x7fae281f1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae280ad7a0 0x7fae281f1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.480+0000 7fae25d9b640 1 -- 172.21.15.29:0/3706183206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae20009400 con 0x7fae280adc00 2023-10-14T16:41:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae25d9b640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c7f0 tx=0x7fae1000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10004220 con 0x7fae280adc00 2023-10-14T16:41:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae100043a0 con 0x7fae280adc00 2023-10-14T16:41:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae281e7690 con 0x7fae280adc00 2023-10-14T16:41:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae10007860 con 0x7fae280adc00 2023-10-14T16:41:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.481+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae281f1e40 con 0x7fae280adc00 2023-10-14T16:41:16.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.482+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae281f2470 con 0x7fae280adc00 2023-10-14T16:41:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.483+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae10007bd0 con 0x7fae280adc00 2023-10-14T16:41:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.484+0000 7fae1effd640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 0x7fae0806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.484+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae10089760 con 0x7fae280adc00 2023-10-14T16:41:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.484+0000 7fae2659c640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 0x7fae0806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.485+0000 7fae2659c640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 0x7fae0806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae18007ed0 tx=0x7fae18007e60 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.491+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae281f2470 con 0x7fae280adc00 2023-10-14T16:41:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.688+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fae281f2470 con 0x7fae280adc00 2023-10-14T16:41:16.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.689+0000 7fae1effd640 1 -- 172.21.15.29:0/3706183206 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7fae281f2470 con 0x7fae280adc00 2023-10-14T16:41:16.690 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 msgr2=0x7fae0806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 0x7fae0806d330 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fae18007ed0 tx=0x7fae18007e60 comp rx=0 tx=0).stop 2023-10-14T16:41:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 msgr2=0x7fae281e6730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fae1000c7f0 tx=0x7fae1000ccb0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 shutdown_connections 2023-10-14T16:41:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae0806ae70 0x7fae0806d330 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae28160c80 0x7fae280c4b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae280adc00 0x7fae281e6730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 --2- 172.21.15.29:0/3706183206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae280ad7a0 0x7fae281f1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.691+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 >> 172.21.15.29:0/3706183206 conn(0x7fae280cc230 msgr2=0x7fae2815d440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.692+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 shutdown_connections 2023-10-14T16:41:16.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.692+0000 7fae2759e640 1 -- 172.21.15.29:0/3706183206 wait complete. 2023-10-14T16:41:16.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:16.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.937+0000 7f9ac811d640 1 Processor -- start 2023-10-14T16:41:16.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.937+0000 7f9ac811d640 1 -- start start 2023-10-14T16:41:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac811d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 0x7f9ac00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac811d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c140 0x7f9ac006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac811d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac0070360 con 0x7f9ac0159e90 2023-10-14T16:41:16.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac811d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac006cb10 con 0x7f9ac006c140 2023-10-14T16:41:16.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac811d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac015d890 con 0x7f9ac0159ab0 2023-10-14T16:41:16.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac56b8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9ac0159e90 0x7f9ac006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48154/0 (socket says 172.21.15.29:48154) 2023-10-14T16:41:16.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.939+0000 7f9ac56b8640 1 -- 172.21.15.29:0/3999742201 learned_addr learned my addr 172.21.15.29:0/3999742201 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac5eb9640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 0x7f9ac00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac66ba640 1 --2- 172.21.15.29:0/3999742201 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c140 0x7f9ac006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823671765 0 0) 0x7f9ac0070360 con 0x7f9ac0159e90 2023-10-14T16:41:16.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aac005730 con 0x7f9ac0159e90 2023-10-14T16:41:16.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 480369327 0 0) 0x7f9aac005730 con 0x7f9ac0159e90 2023-10-14T16:41:16.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 msgr2=0x7f9ac00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 0x7f9ac00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c140 msgr2=0x7f9ac006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 --2- 172.21.15.29:0/3999742201 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c140 0x7f9ac006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00cd1f0 con 0x7f9ac0159e90 2023-10-14T16:41:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9ab0001670 con 0x7f9ac0159e90 2023-10-14T16:41:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.940+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 >> v1:172.21.15.29:6789/0 conn(0x7f9ac0159e90 legacy=0x7f9ac006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 0x7f9aac005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9aac0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac006f10 0x7f9aac007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac006cb10 con 0x7f9aac006150 2023-10-14T16:41:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac015d890 con 0x7f9aac006f10 2023-10-14T16:41:16.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac00dcc0 con 0x7f9ac0159ab0 2023-10-14T16:41:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9aac0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 -- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 msgr2=0x7f9aac005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0159ab0 0x7f9aac005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 -- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac006f10 msgr2=0x7f9aac007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac66ba640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac006f10 0x7f9aac007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aac006f10 0x7f9aac007390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.941+0000 7f9ac56b8640 1 -- 172.21.15.29:0/3999742201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac005730 con 0x7f9aac006150 2023-10-14T16:41:16.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.942+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9aac0069d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c0e0 tx=0x7f9aac007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406d7cd2a688e0a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab0005ad0 con 0x7f9aac006150 2023-10-14T16:41:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab0005c30 con 0x7f9aac006150 2023-10-14T16:41:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac4eb7640 1 -- 172.21.15.29:0/3999742201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab002c7d0 con 0x7f9aac006150 2023-10-14T16:41:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 -- 172.21.15.29:0/3999742201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 msgr2=0x7f9aac0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 --2- 172.21.15.29:0/3999742201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9aac0069d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f9ab000c0e0 tx=0x7f9aac007a00 comp rx=0 tx=0).stop 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac66ba640 1 -- 172.21.15.29:0/3999742201 reap_dead start 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 -- 172.21.15.29:0/3999742201 shutdown_connections 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 -- 172.21.15.29:0/3999742201 >> 172.21.15.29:0/3999742201 conn(0x7f9ac00ce9b0 msgr2=0x7f9ac00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 -- 172.21.15.29:0/3999742201 shutdown_connections 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.943+0000 7f9ac811d640 1 -- 172.21.15.29:0/3999742201 wait complete. 2023-10-14T16:41:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.945+0000 7f9ac811d640 1 Processor -- start 2023-10-14T16:41:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.945+0000 7f9ac811d640 1 -- start start 2023-10-14T16:41:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01565e0 0x7f9ac00ae3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0156a40 0x7f9ac00ae920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c2790 con 0x7f9aac006150 2023-10-14T16:41:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c2a50 con 0x7f9ac01565e0 2023-10-14T16:41:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.946+0000 7f9ac811d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c2d30 con 0x7f9ac0156a40 2023-10-14T16:41:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36976/0 (socket says 172.21.15.29:36976) 2023-10-14T16:41:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 -- 172.21.15.29:0/417554793 learned_addr learned my addr 172.21.15.29:0/417554793 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 -- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0156a40 msgr2=0x7f9ac00ae920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0156a40 0x7f9ac00ae920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 -- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01565e0 msgr2=0x7f9ac00ae3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01565e0 0x7f9ac00ae3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 -- 172.21.15.29:0/417554793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00c3010 con 0x7f9aac006150 2023-10-14T16:41:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.947+0000 7f9ac5eb9640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00e7c0 tx=0x7f9abc00ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.948+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00eed0 con 0x7f9aac006150 2023-10-14T16:41:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.948+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9abc00c8c0 con 0x7f9aac006150 2023-10-14T16:41:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.948+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9abc00cc50 con 0x7f9aac006150 2023-10-14T16:41:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.948+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac00c8850 con 0x7f9aac006150 2023-10-14T16:41:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.948+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac015aa50 con 0x7f9aac006150 2023-10-14T16:41:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.949+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac004e160 con 0x7f9aac006150 2023-10-14T16:41:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.950+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f9abc013470 con 0x7f9aac006150 2023-10-14T16:41:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.951+0000 7f9ab67fc640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 0x7f9aa406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.951+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9abc0894e0 con 0x7f9aac006150 2023-10-14T16:41:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.951+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 0x7f9aa406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.952+0000 7f9ac56b8640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 0x7f9aa406d360 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0034000 tx=0x7f9ab0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:16.955+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9abc054470 con 0x7f9aac006150 2023-10-14T16:41:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.153+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ac015aa50 con 0x7f9aac006150 2023-10-14T16:41:17.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.153+0000 7f9ab67fc640 1 -- 172.21.15.29:0/417554793 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9ac015aa50 con 0x7f9aac006150 2023-10-14T16:41:17.154 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 msgr2=0x7f9aa406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 0x7f9aa406d360 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0034000 tx=0x7f9ab0007000 comp rx=0 tx=0).stop 2023-10-14T16:41:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 msgr2=0x7f9ac015a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f9abc00e7c0 tx=0x7f9abc00ec80 comp rx=0 tx=0).stop 2023-10-14T16:41:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 shutdown_connections 2023-10-14T16:41:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f9aa406aea0 0x7f9aa406d360 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac0156a40 0x7f9ac00ae920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01565e0 0x7f9ac00ae3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 --2- 172.21.15.29:0/417554793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aac006150 0x7f9ac015a330 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 >> 172.21.15.29:0/417554793 conn(0x7f9ac00ce9b0 msgr2=0x7f9ac00afc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 shutdown_connections 2023-10-14T16:41:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.156+0000 7f9ac811d640 1 -- 172.21.15.29:0/417554793 wait complete. 2023-10-14T16:41:17.175 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:41:17.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.401+0000 7f4705587640 1 Processor -- start 2023-10-14T16:41:17.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.401+0000 7f4705587640 1 -- start start 2023-10-14T16:41:17.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.402+0000 7f4705587640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f470015c9f0 0x7f47000c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.402+0000 7f4705587640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f47000c9230 0x7f47000709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.402+0000 7f4705587640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4700160a90 con 0x7f4700070f20 2023-10-14T16:41:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.402+0000 7f4705587640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47000cdec0 con 0x7f470015c9f0 2023-10-14T16:41:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.402+0000 7f4705587640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47000cd900 con 0x7f47000c9230 2023-10-14T16:41:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f470015c9f0 0x7f47000c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f470015c9f0 0x7f47000c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36978/0 (socket says 172.21.15.29:36978) 2023-10-14T16:41:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 -- 172.21.15.29:0/4187076922 learned_addr learned my addr 172.21.15.29:0/4187076922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 -- 172.21.15.29:0/4187076922 >> v2:172.21.15.186:3300/0 conn(0x7f47000c9230 msgr2=0x7f47000709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 --2- 172.21.15.29:0/4187076922 >> v2:172.21.15.186:3300/0 conn(0x7f47000c9230 0x7f47000709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 -- 172.21.15.29:0/4187076922 >> v1:172.21.15.186:6789/0 conn(0x7f4700070f20 legacy=0x7f47000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.403+0000 7f46fffff640 1 -- 172.21.15.29:0/4187076922 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47000cdbe0 con 0x7f470015c9f0 2023-10-14T16:41:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.404+0000 7f46fffff640 1 --2- 172.21.15.29:0/4187076922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015c9f0 0x7f47000c8cf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47000b1260 tx=0x7f46f002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f78a03c9add94671 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.405+0000 7f46feffd640 1 -- 172.21.15.29:0/4187076922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f002ca80 con 0x7f470015c9f0 2023-10-14T16:41:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.405+0000 7f46feffd640 1 -- 172.21.15.29:0/4187076922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f002cbe0 con 0x7f470015c9f0 2023-10-14T16:41:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.405+0000 7f46feffd640 1 -- 172.21.15.29:0/4187076922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f00343f0 con 0x7f470015c9f0 2023-10-14T16:41:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.405+0000 7f4705587640 1 -- 172.21.15.29:0/4187076922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015c9f0 msgr2=0x7f47000c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.405+0000 7f4705587640 1 --2- 172.21.15.29:0/4187076922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015c9f0 0x7f47000c8cf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f47000b1260 tx=0x7f46f002bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 -- 172.21.15.29:0/4187076922 shutdown_connections 2023-10-14T16:41:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 --2- 172.21.15.29:0/4187076922 >> v2:172.21.15.186:3300/0 conn(0x7f47000c9230 0x7f47000709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 --2- 172.21.15.29:0/4187076922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f470015c9f0 0x7f47000c8cf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 -- 172.21.15.29:0/4187076922 >> 172.21.15.29:0/4187076922 conn(0x7f47000ce6a0 msgr2=0x7f47000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 -- 172.21.15.29:0/4187076922 shutdown_connections 2023-10-14T16:41:17.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.406+0000 7f4705587640 1 -- 172.21.15.29:0/4187076922 wait complete. 2023-10-14T16:41:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.408+0000 7f4705587640 1 Processor -- start 2023-10-14T16:41:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.408+0000 7f4705587640 1 -- start start 2023-10-14T16:41:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.409+0000 7f4705587640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4700070f20 0x7f47000cce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.409+0000 7f4705587640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.409+0000 7f4705587640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f470015c9f0 0x7f47000c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f4705587640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47000c6b60 con 0x7f47000c9230 2023-10-14T16:41:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f4705587640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47000c5cb0 con 0x7f470015c9f0 2023-10-14T16:41:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f4705587640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47000c5ee0 con 0x7f4700070f20 2023-10-14T16:41:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36984/0 (socket says 172.21.15.29:36984) 2023-10-14T16:41:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 -- 172.21.15.29:0/2470363448 learned_addr learned my addr 172.21.15.29:0/2470363448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46fffff640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4700070f20 0x7f47000cce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 -- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4700070f20 msgr2=0x7f47000cce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4700070f20 0x7f47000cce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 -- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f470015c9f0 msgr2=0x7f47000c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f470015c9f0 0x7f47000c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.410+0000 7f46ff7fe640 1 -- 172.21.15.29:0/2470363448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f0009b30 con 0x7f47000c9230 2023-10-14T16:41:17.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f46ff7fe640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b4a0 tx=0x7f46f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4010610 con 0x7f47000c9230 2023-10-14T16:41:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f4010770 con 0x7f47000c9230 2023-10-14T16:41:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47000c61c0 con 0x7f47000c9230 2023-10-14T16:41:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f4010b20 con 0x7f47000c9230 2023-10-14T16:41:17.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.411+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47000c6780 con 0x7f47000c9230 2023-10-14T16:41:17.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.412+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47000c6ca0 con 0x7f47000c9230 2023-10-14T16:41:17.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.413+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f46f4010c80 con 0x7f47000c9230 2023-10-14T16:41:17.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.414+0000 7f46fcff9640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 0x7f46ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:17.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.414+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f46f4089320 con 0x7f47000c9230 2023-10-14T16:41:17.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.415+0000 7f46fffff640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 0x7f46ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:17.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.415+0000 7f46fffff640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 0x7f46ec06d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46f0035000 tx=0x7f46f0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:17.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.418+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47000c6ca0 con 0x7f47000c9230 2023-10-14T16:41:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.629+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f47000c6ca0 con 0x7f47000c9230 2023-10-14T16:41:17.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.630+0000 7f46fcff9640 1 -- 172.21.15.29:0/2470363448 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f47000c6ca0 con 0x7f47000c9230 2023-10-14T16:41:17.631 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-14T16:41:17.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 msgr2=0x7f46ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 0x7f46ec06d330 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f46f0035000 tx=0x7f46f0030000 comp rx=0 tx=0).stop 2023-10-14T16:41:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 msgr2=0x7f4700149bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f46f400b4a0 tx=0x7f46f400b960 comp rx=0 tx=0).stop 2023-10-14T16:41:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 shutdown_connections 2023-10-14T16:41:17.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f46ec06ae70 0x7f46ec06d330 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f470015c9f0 0x7f47000c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47000c9230 0x7f4700149bc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 --2- 172.21.15.29:0/2470363448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4700070f20 0x7f47000cce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 >> 172.21.15.29:0/2470363448 conn(0x7f47000ce6a0 msgr2=0x7f47000ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 shutdown_connections 2023-10-14T16:41:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:17.632+0000 7f4705587640 1 -- 172.21.15.29:0/2470363448 wait complete. 2023-10-14T16:41:17.652 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.2... 2023-10-14T16:41:17.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:17.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.2 -f 2023-10-14T16:41:17.756 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:41:17.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:17.757 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2023-10-14T16:41:17.815 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:17.816 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:17.875 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:17.875 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:17.926 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:41:17.926 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:17.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.205+0000 7f7224e21640 1 Processor -- start 2023-10-14T16:41:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.205+0000 7f7224e21640 1 -- start start 2023-10-14T16:41:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f7224e21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 0x7f72200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f7224e21640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72200ccfb0 0x7f72200cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f7224e21640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f722015b9d0 con 0x7f722006c040 2023-10-14T16:41:18.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f7224e21640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722015bcb0 con 0x7f72200ccfb0 2023-10-14T16:41:18.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f7224e21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722015bf90 con 0x7f72200ad970 2023-10-14T16:41:18.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f721ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72200ccfb0 0x7f72200cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f721ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72200ccfb0 0x7f72200cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36994/0 (socket says 172.21.15.29:36994) 2023-10-14T16:41:18.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.207+0000 7f721ffff640 1 -- 172.21.15.29:0/2390709141 learned_addr learned my addr 172.21.15.29:0/2390709141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f7217fff640 1 --2- 172.21.15.29:0/2390709141 >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 0x7f72200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f721ffff640 1 -- 172.21.15.29:0/2390709141 >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 msgr2=0x7f72200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f721ffff640 1 --2- 172.21.15.29:0/2390709141 >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 0x7f72200cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f721ffff640 1 -- 172.21.15.29:0/2390709141 >> v1:172.21.15.186:6789/0 conn(0x7f722006c040 legacy=0x7f722006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f721ffff640 1 -- 172.21.15.29:0/2390709141 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72200cc270 con 0x7f72200ccfb0 2023-10-14T16:41:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.208+0000 7f7217fff640 1 --2- 172.21.15.29:0/2390709141 >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 0x7f72200cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.209+0000 7f721ffff640 1 --2- 172.21.15.29:0/2390709141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200cd460 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7218009e50 tx=0x7f721802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6429930dba461963 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.209+0000 7f721effd640 1 -- 172.21.15.29:0/2390709141 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f721802ca80 con 0x7f72200ccfb0 2023-10-14T16:41:18.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.209+0000 7f721effd640 1 -- 172.21.15.29:0/2390709141 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f721802cbe0 con 0x7f72200ccfb0 2023-10-14T16:41:18.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.209+0000 7f721effd640 1 -- 172.21.15.29:0/2390709141 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72180343f0 con 0x7f72200ccfb0 2023-10-14T16:41:18.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.210+0000 7f7224e21640 1 -- 172.21.15.29:0/2390709141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 msgr2=0x7f72200cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.210+0000 7f7224e21640 1 --2- 172.21.15.29:0/2390709141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200cd460 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7218009e50 tx=0x7f721802bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.210+0000 7f7224e21640 1 -- 172.21.15.29:0/2390709141 shutdown_connections 2023-10-14T16:41:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.210+0000 7f7224e21640 1 --2- 172.21.15.29:0/2390709141 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200cd460 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.210+0000 7f7224e21640 1 --2- 172.21.15.29:0/2390709141 >> v2:172.21.15.186:3300/0 conn(0x7f72200ad970 0x7f72200cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.211+0000 7f7224e21640 1 -- 172.21.15.29:0/2390709141 >> 172.21.15.29:0/2390709141 conn(0x7f72200ce6a0 msgr2=0x7f72200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:18.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.211+0000 7f7224e21640 1 -- 172.21.15.29:0/2390709141 shutdown_connections 2023-10-14T16:41:18.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.211+0000 7f7224e21640 1 -- 172.21.15.29:0/2390709141 wait complete. 2023-10-14T16:41:18.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.213+0000 7f7224e21640 1 Processor -- start 2023-10-14T16:41:18.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.213+0000 7f7224e21640 1 -- start start 2023-10-14T16:41:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f722006c040 0x7f722014ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72200ad970 0x7f72200b2430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72200b3a20 con 0x7f72200ccfb0 2023-10-14T16:41:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f722014b090 con 0x7f72200ad970 2023-10-14T16:41:18.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f7224e21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72200b2da0 con 0x7f722006c040 2023-10-14T16:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f721ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f721ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37008/0 (socket says 172.21.15.29:37008) 2023-10-14T16:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f721ffff640 1 -- 172.21.15.29:0/3634943052 learned_addr learned my addr 172.21.15.29:0/3634943052 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.215+0000 7f721f7fe640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f722006c040 0x7f722014ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f7217fff640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72200ad970 0x7f72200b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 -- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f722006c040 msgr2=0x7f722014ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f722006c040 0x7f722014ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 -- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72200ad970 msgr2=0x7f72200b2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72200ad970 0x7f72200b2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 -- 172.21.15.29:0/3634943052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7218009b30 con 0x7f72200ccfb0 2023-10-14T16:41:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721ffff640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7218036040 tx=0x7f7218004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218040670 con 0x7f72200ccfb0 2023-10-14T16:41:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72180407f0 con 0x7f72200ccfb0 2023-10-14T16:41:18.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7218040ba0 con 0x7f72200ccfb0 2023-10-14T16:41:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.216+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72200b3080 con 0x7f72200ccfb0 2023-10-14T16:41:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.217+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72200b3640 con 0x7f72200ccfb0 2023-10-14T16:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.218+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722006b330 con 0x7f72200ccfb0 2023-10-14T16:41:18.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.219+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f721802cd50 con 0x7f72200ccfb0 2023-10-14T16:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.219+0000 7f721cff9640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 0x7f720006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.220+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f72180ac550 con 0x7f72200ccfb0 2023-10-14T16:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.220+0000 7f721f7fe640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 0x7f720006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.221+0000 7f721f7fe640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 0x7f720006d330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7210007fd0 tx=0x7f7210009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.224+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7218077570 con 0x7f72200ccfb0 2023-10-14T16:41:18.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.426+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72200b3b60 con 0x7f72200ccfb0 2023-10-14T16:41:18.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.428+0000 7f721cff9640 1 -- 172.21.15.29:0/3634943052 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7f72200b3b60 con 0x7f72200ccfb0 2023-10-14T16:41:18.430 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:18.430 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 msgr2=0x7f720006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 0x7f720006d330 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7210007fd0 tx=0x7f7210009040 comp rx=0 tx=0).stop 2023-10-14T16:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 msgr2=0x7f72200c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7218036040 tx=0x7f7218004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:18.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 shutdown_connections 2023-10-14T16:41:18.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f720006ae70 0x7f720006d330 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72200ccfb0 0x7f72200c7940 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72200ad970 0x7f72200b2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 --2- 172.21.15.29:0/3634943052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f722006c040 0x7f722014ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 >> 172.21.15.29:0/3634943052 conn(0x7f72200ce6a0 msgr2=0x7f722006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:18.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.431+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 shutdown_connections 2023-10-14T16:41:18.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.432+0000 7f7224e21640 1 -- 172.21.15.29:0/3634943052 wait complete. 2023-10-14T16:41:18.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:18.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:18.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.680+0000 7f2a51c90640 1 Processor -- start 2023-10-14T16:41:18.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.680+0000 7f2a51c90640 1 -- start start 2023-10-14T16:41:18.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a51c90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c0cb000 0x7f2a4c0cb460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a51c90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a4c070380 0x7f2a4c0707f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a51c90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a4c15d0e0 con 0x7f2a4c15cd00 2023-10-14T16:41:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a51c90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a4c070d30 con 0x7f2a4c070380 2023-10-14T16:41:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a51c90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0cbd80 con 0x7f2a4c0cb000 2023-10-14T16:41:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a50c8e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a4c15cd00 0x7f2a4c0cdfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48156/0 (socket says 172.21.15.29:48156) 2023-10-14T16:41:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a50c8e640 1 -- 172.21.15.29:0/1225174898 learned_addr learned my addr 172.21.15.29:0/1225174898 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:18.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.681+0000 7f2a5148f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a4c070380 0x7f2a4c0707f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a4bfff640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c0cb000 0x7f2a4c0cb460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/1225174898 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932104223 0 0) 0x7f2a4c15d0e0 con 0x7f2a4c15cd00 2023-10-14T16:41:18.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/1225174898 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a34003520 con 0x7f2a4c15cd00 2023-10-14T16:41:18.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a5148f640 1 -- 172.21.15.29:0/1225174898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c0cb000 msgr2=0x7f2a4c0cb460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:18.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a5148f640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c0cb000 0x7f2a4c0cb460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a5148f640 1 -- 172.21.15.29:0/1225174898 >> v1:172.21.15.29:6789/0 conn(0x7f2a4c15cd00 legacy=0x7f2a4c0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.682+0000 7f2a5148f640 1 -- 172.21.15.29:0/1225174898 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a4c0cbaa0 con 0x7f2a4c070380 2023-10-14T16:41:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.683+0000 7f2a5148f640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c0707f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2a440097a0 tx=0x7f2a4402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85d7de14270cc29d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/1225174898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44031830 con 0x7f2a4c070380 2023-10-14T16:41:18.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/1225174898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a44031990 con 0x7f2a4c070380 2023-10-14T16:41:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a4b7fe640 1 -- 172.21.15.29:0/1225174898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a44031d20 con 0x7f2a4c070380 2023-10-14T16:41:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 -- 172.21.15.29:0/1225174898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 msgr2=0x7f2a4c0707f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c0707f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f2a440097a0 tx=0x7f2a4402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 -- 172.21.15.29:0/1225174898 shutdown_connections 2023-10-14T16:41:18.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c0707f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 --2- 172.21.15.29:0/1225174898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c0cb000 0x7f2a4c0cb460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 -- 172.21.15.29:0/1225174898 >> 172.21.15.29:0/1225174898 conn(0x7f2a4c0cf790 msgr2=0x7f2a4c0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:18.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 -- 172.21.15.29:0/1225174898 shutdown_connections 2023-10-14T16:41:18.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.684+0000 7f2a51c90640 1 -- 172.21.15.29:0/1225174898 wait complete. 2023-10-14T16:41:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.686+0000 7f2a51c90640 1 Processor -- start 2023-10-14T16:41:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.686+0000 7f2a51c90640 1 -- start start 2023-10-14T16:41:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a4c0cb000 0x7f2a4c0c37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c15cd00 0x7f2a4c0c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c157070 con 0x7f2a4c070380 2023-10-14T16:41:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c3220 con 0x7f2a4c0cb000 2023-10-14T16:41:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.687+0000 7f2a51c90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a4c0c2f20 con 0x7f2a4c15cd00 2023-10-14T16:41:18.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37020/0 (socket says 172.21.15.29:37020) 2023-10-14T16:41:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 -- 172.21.15.29:0/2131385580 learned_addr learned my addr 172.21.15.29:0/2131385580 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 -- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c15cd00 msgr2=0x7f2a4c0c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c15cd00 0x7f2a4c0c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 -- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a4c0cb000 msgr2=0x7f2a4c0c37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a4bfff640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a4c0cb000 0x7f2a4c0c37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a4c0cb000 0x7f2a4c0c37f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.688+0000 7f2a50c8e640 1 -- 172.21.15.29:0/2131385580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a44009400 con 0x7f2a4c070380 2023-10-14T16:41:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a50c8e640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00cad0 tx=0x7f2a3c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c00ac30 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3c00ad90 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a3c012620 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a4c1600e0 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.689+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a4c0c6260 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.690+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a4c04e9b0 con 0x7f2a4c070380 2023-10-14T16:41:18.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.691+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2a3c007070 con 0x7f2a4c070380 2023-10-14T16:41:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.692+0000 7f2a497fa640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 0x7f2a3806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.692+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2a3c089670 con 0x7f2a4c070380 2023-10-14T16:41:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.692+0000 7f2a4bfff640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 0x7f2a3806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.693+0000 7f2a4bfff640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 0x7f2a3806d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b0340 tx=0x7f2a40008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.696+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a3c054690 con 0x7f2a4c070380 2023-10-14T16:41:18.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.892+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2a4c0b5710 con 0x7f2a4c070380 2023-10-14T16:41:18.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.893+0000 7f2a497fa640 1 -- 172.21.15.29:0/2131385580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2a3c0170a0 con 0x7f2a4c070380 2023-10-14T16:41:18.894 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:18.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.895+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 msgr2=0x7f2a3806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.895+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 0x7f2a3806d330 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c0b0340 tx=0x7f2a40008040 comp rx=0 tx=0).stop 2023-10-14T16:41:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 msgr2=0x7f2a4c1567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c00cad0 tx=0x7f2a3c00cf90 comp rx=0 tx=0).stop 2023-10-14T16:41:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 shutdown_connections 2023-10-14T16:41:18.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a3806ae70 0x7f2a3806d330 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a4c15cd00 0x7f2a4c0c3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a4c0cb000 0x7f2a4c0c37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 --2- 172.21.15.29:0/2131385580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a4c070380 0x7f2a4c1567c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:18.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 >> 172.21.15.29:0/2131385580 conn(0x7f2a4c0cf790 msgr2=0x7f2a4c0b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 shutdown_connections 2023-10-14T16:41:18.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:18.896+0000 7f2a51c90640 1 -- 172.21.15.29:0/2131385580 wait complete. 2023-10-14T16:41:18.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:19.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.143+0000 7f2722f7c640 1 Processor -- start 2023-10-14T16:41:19.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.143+0000 7f2722f7c640 1 -- start start 2023-10-14T16:41:19.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.144+0000 7f2722f7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 0x7f271c06ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.144+0000 7f2722f7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f271c1428c0 0x7f271c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2722f7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f271c0cc6b0 con 0x7f271c0c9690 2023-10-14T16:41:19.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2722f7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f271c0c9b20 con 0x7f271c1428c0 2023-10-14T16:41:19.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2722f7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c1599b0 con 0x7f271c06b660 2023-10-14T16:41:19.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2721519640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f271c0c9690 0x7f271c15d080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48162/0 (socket says 172.21.15.29:48162) 2023-10-14T16:41:19.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2721519640 1 -- 172.21.15.29:0/1055232659 learned_addr learned my addr 172.21.15.29:0/1055232659 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f271bfff640 1 --2- 172.21.15.29:0/1055232659 >> v2:172.21.15.29:3300/0 conn(0x7f271c1428c0 0x7f271c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f2720d18640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 0x7f271c06ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.146+0000 7f271bfff640 1 -- 172.21.15.29:0/1055232659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 msgr2=0x7f271c06ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:19.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.145+0000 7f271b7fe640 1 -- 172.21.15.29:0/1055232659 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211767176 0 0) 0x7f271c0cc6b0 con 0x7f271c0c9690 2023-10-14T16:41:19.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.146+0000 7f271bfff640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 0x7f271c06ba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.146+0000 7f271bfff640 1 -- 172.21.15.29:0/1055232659 >> v1:172.21.15.29:6789/0 conn(0x7f271c0c9690 legacy=0x7f271c15d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.146+0000 7f271bfff640 1 -- 172.21.15.29:0/1055232659 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f271c06c290 con 0x7f271c1428c0 2023-10-14T16:41:19.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.146+0000 7f2720d18640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 0x7f271c06ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:19.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.147+0000 7f271bfff640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c1428c0 0x7f271c0c9150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f27100097a0 tx=0x7f271002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=834910d156bc219f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.147+0000 7f271b7fe640 1 -- 172.21.15.29:0/1055232659 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f271002d040 con 0x7f271c1428c0 2023-10-14T16:41:19.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.147+0000 7f271b7fe640 1 -- 172.21.15.29:0/1055232659 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2710031810 con 0x7f271c1428c0 2023-10-14T16:41:19.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.147+0000 7f271b7fe640 1 -- 172.21.15.29:0/1055232659 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2710031ba0 con 0x7f271c1428c0 2023-10-14T16:41:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 -- 172.21.15.29:0/1055232659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c1428c0 msgr2=0x7f271c0c9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c1428c0 0x7f271c0c9150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f27100097a0 tx=0x7f271002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 -- 172.21.15.29:0/1055232659 shutdown_connections 2023-10-14T16:41:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c1428c0 0x7f271c0c9150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1055232659 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c06b660 0x7f271c06ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 -- 172.21.15.29:0/1055232659 >> 172.21.15.29:0/1055232659 conn(0x7f271c0ce9b0 msgr2=0x7f271c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:19.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 -- 172.21.15.29:0/1055232659 shutdown_connections 2023-10-14T16:41:19.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.148+0000 7f2722f7c640 1 -- 172.21.15.29:0/1055232659 wait complete. 2023-10-14T16:41:19.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.149+0000 7f2722f7c640 1 Processor -- start 2023-10-14T16:41:19.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.150+0000 7f2722f7c640 1 -- start start 2023-10-14T16:41:19.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f271c0c9690 0x7f271c1e7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c1428c0 0x7f271c1e8400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c158d30 con 0x7f271c06b660 2023-10-14T16:41:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f271c1e5b40 con 0x7f271c0c9690 2023-10-14T16:41:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2722f7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f271c1e4fc0 con 0x7f271c1428c0 2023-10-14T16:41:19.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37032/0 (socket says 172.21.15.29:37032) 2023-10-14T16:41:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 -- 172.21.15.29:0/1585623352 learned_addr learned my addr 172.21.15.29:0/1585623352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 -- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c1428c0 msgr2=0x7f271c1e8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c1428c0 0x7f271c1e8400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 -- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f271c0c9690 msgr2=0x7f271c1e7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f271c0c9690 0x7f271c1e7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.151+0000 7f2720d18640 1 -- 172.21.15.29:0/1585623352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2710009400 con 0x7f271c06b660 2023-10-14T16:41:19.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f2720d18640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f270c00ce00 tx=0x7f270c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c00f890 con 0x7f271c06b660 2023-10-14T16:41:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f270c00f9f0 con 0x7f271c06b660 2023-10-14T16:41:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f270c00fdc0 con 0x7f271c06b660 2023-10-14T16:41:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f271c1e52a0 con 0x7f271c06b660 2023-10-14T16:41:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.152+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f271c1e5860 con 0x7f271c06b660 2023-10-14T16:41:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.153+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f271c04e160 con 0x7f271c06b660 2023-10-14T16:41:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.154+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f270c00b040 con 0x7f271c06b660 2023-10-14T16:41:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.155+0000 7f27197fa640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 0x7f270806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.155+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f270c08a0c0 con 0x7f271c06b660 2023-10-14T16:41:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.155+0000 7f271bfff640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 0x7f270806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.156+0000 7f271bfff640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 0x7f270806d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2710036000 tx=0x7f2710030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.160+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f270c055050 con 0x7f271c06b660 2023-10-14T16:41:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.355+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f271c0ad7e0 con 0x7f271c06b660 2023-10-14T16:41:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.356+0000 7f27197fa640 1 -- 172.21.15.29:0/1585623352 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f270c058670 con 0x7f271c06b660 2023-10-14T16:41:19.358 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 msgr2=0x7f270806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 0x7f270806d330 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2710036000 tx=0x7f2710030000 comp rx=0 tx=0).stop 2023-10-14T16:41:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 msgr2=0x7f271c0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f270c00ce00 tx=0x7f270c00a650 comp rx=0 tx=0).stop 2023-10-14T16:41:19.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 shutdown_connections 2023-10-14T16:41:19.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f270806ae70 0x7f270806d330 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.359+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f271c1428c0 0x7f271c1e8400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.360+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f271c0c9690 0x7f271c1e7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.360+0000 7f2722f7c640 1 --2- 172.21.15.29:0/1585623352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f271c06b660 0x7f271c0c4530 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.360+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 >> 172.21.15.29:0/1585623352 conn(0x7f271c0ce9b0 msgr2=0x7f271c15bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.360+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 shutdown_connections 2023-10-14T16:41:19.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.360+0000 7f2722f7c640 1 -- 172.21.15.29:0/1585623352 wait complete. 2023-10-14T16:41:19.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:19.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.613+0000 7fe6eee6d640 1 Processor -- start 2023-10-14T16:41:19.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.613+0000 7fe6eee6d640 1 -- start start 2023-10-14T16:41:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.614+0000 7fe6eee6d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6e80b8480 0x7fe6e80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.614+0000 7fe6eee6d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6e80d0be0 0x7fe6e80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.614+0000 7fe6eee6d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e815d150 con 0x7fe6e815cd70 2023-10-14T16:41:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.614+0000 7fe6eee6d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80d1180 con 0x7fe6e80d0be0 2023-10-14T16:41:19.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.614+0000 7fe6eee6d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80d1460 con 0x7fe6e80b8480 2023-10-14T16:41:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.615+0000 7fe6ee66c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6e80d0be0 0x7fe6e80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.615+0000 7fe6ee66c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6e80d0be0 0x7fe6e80cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37040/0 (socket says 172.21.15.29:37040) 2023-10-14T16:41:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.615+0000 7fe6ee66c640 1 -- 172.21.15.29:0/1798905844 learned_addr learned my addr 172.21.15.29:0/1798905844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:19.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.615+0000 7fe6ed66a640 1 --2- 172.21.15.29:0/1798905844 >> v2:172.21.15.186:3300/0 conn(0x7fe6e80b8480 0x7fe6e80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.616+0000 7fe6ee66c640 1 -- 172.21.15.29:0/1798905844 >> v2:172.21.15.186:3300/0 conn(0x7fe6e80b8480 msgr2=0x7fe6e80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.616+0000 7fe6ee66c640 1 --2- 172.21.15.29:0/1798905844 >> v2:172.21.15.186:3300/0 conn(0x7fe6e80b8480 0x7fe6e80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.616+0000 7fe6ee66c640 1 -- 172.21.15.29:0/1798905844 >> v1:172.21.15.186:6789/0 conn(0x7fe6e815cd70 legacy=0x7fe6e80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.616+0000 7fe6ee66c640 1 -- 172.21.15.29:0/1798905844 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80d1740 con 0x7fe6e80d0be0 2023-10-14T16:41:19.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.617+0000 7fe6ee66c640 1 --2- 172.21.15.29:0/1798905844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e80cdd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8066a00 tx=0x7fe6e002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ada28cf25f2ba26 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.617+0000 7fe6ece69640 1 -- 172.21.15.29:0/1798905844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e002ca80 con 0x7fe6e80d0be0 2023-10-14T16:41:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.617+0000 7fe6ece69640 1 -- 172.21.15.29:0/1798905844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e002cbe0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.617+0000 7fe6ece69640 1 -- 172.21.15.29:0/1798905844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6e00343f0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 -- 172.21.15.29:0/1798905844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 msgr2=0x7fe6e80cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/1798905844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e80cdd00 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fe6e8066a00 tx=0x7fe6e002bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 -- 172.21.15.29:0/1798905844 shutdown_connections 2023-10-14T16:41:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/1798905844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e80cdd00 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/1798905844 >> v2:172.21.15.186:3300/0 conn(0x7fe6e80b8480 0x7fe6e80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 -- 172.21.15.29:0/1798905844 >> 172.21.15.29:0/1798905844 conn(0x7fe6e80cf570 msgr2=0x7fe6e80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:19.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 -- 172.21.15.29:0/1798905844 shutdown_connections 2023-10-14T16:41:19.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.618+0000 7fe6eee6d640 1 -- 172.21.15.29:0/1798905844 wait complete. 2023-10-14T16:41:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.620+0000 7fe6eee6d640 1 Processor -- start 2023-10-14T16:41:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.620+0000 7fe6eee6d640 1 -- start start 2023-10-14T16:41:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 0x7fe6e8158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 0x7fe6e80cc250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e8159610 con 0x7fe6e80d0be0 2023-10-14T16:41:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e815e390 con 0x7fe6e815cd70 2023-10-14T16:41:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6eee6d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e815e670 con 0x7fe6e80b8480 2023-10-14T16:41:19.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6ed66a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6ed66a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37048/0 (socket says 172.21.15.29:37048) 2023-10-14T16:41:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.621+0000 7fe6ed66a640 1 -- 172.21.15.29:0/640154050 learned_addr learned my addr 172.21.15.29:0/640154050 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ede6b640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 0x7fe6e8158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ee66c640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 0x7fe6e80cc250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 -- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 msgr2=0x7fe6e8158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 0x7fe6e8158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 -- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 msgr2=0x7fe6e80cc250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 0x7fe6e80cc250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 -- 172.21.15.29:0/640154050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e0009b30 con 0x7fe6e80d0be0 2023-10-14T16:41:19.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ede6b640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 0x7fe6e8158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ee66c640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 0x7fe6e80cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6ed66a640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800d5f0 tx=0x7fe6d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d80125f0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8012750 con 0x7fe6e80d0be0 2023-10-14T16:41:19.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6d8012b00 con 0x7fe6e80d0be0 2023-10-14T16:41:19.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.622+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e815e950 con 0x7fe6e80d0be0 2023-10-14T16:41:19.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.623+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e8159200 con 0x7fe6e80d0be0 2023-10-14T16:41:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.624+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e804e9b0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.625+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe6d8012c60 con 0x7fe6e80d0be0 2023-10-14T16:41:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.626+0000 7fe6de7fc640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 0x7fe6cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.626+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe6d8089c00 con 0x7fe6e80d0be0 2023-10-14T16:41:19.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.627+0000 7fe6ede6b640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 0x7fe6cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:19.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.627+0000 7fe6ede6b640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 0x7fe6cc06d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6e81f00c0 tx=0x7fe6e4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:19.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.631+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6d8054b90 con 0x7fe6e80d0be0 2023-10-14T16:41:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.824+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe6e80b6fa0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.824+0000 7fe6de7fc640 1 -- 172.21.15.29:0/640154050 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe6d80581b0 con 0x7fe6e80d0be0 2023-10-14T16:41:19.825 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 msgr2=0x7fe6cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 0x7fe6cc06d330 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe6e81f00c0 tx=0x7fe6e4009040 comp rx=0 tx=0).stop 2023-10-14T16:41:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 msgr2=0x7fe6e8156600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe6d800d5f0 tx=0x7fe6d800dab0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 shutdown_connections 2023-10-14T16:41:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe6cc06ae70 0x7fe6cc06d330 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6e815cd70 0x7fe6e80cc250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6e80d0be0 0x7fe6e8156600 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 --2- 172.21.15.29:0/640154050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6e80b8480 0x7fe6e8158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 >> 172.21.15.29:0/640154050 conn(0x7fe6e80cf570 msgr2=0x7fe6e80b74c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:19.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 shutdown_connections 2023-10-14T16:41:19.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:19.827+0000 7fe6eee6d640 1 -- 172.21.15.29:0/640154050 wait complete. 2023-10-14T16:41:19.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:41:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.061+0000 7f549cb73640 1 Processor -- start 2023-10-14T16:41:20.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.061+0000 7f549cb73640 1 -- start start 2023-10-14T16:41:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f549cb73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54980ad970 0x7f54980cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f549cb73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 0x7f54980cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f549cb73640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f549815b9d0 con 0x7f549806c040 2023-10-14T16:41:20.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f549cb73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f549815bcb0 con 0x7f54980ad970 2023-10-14T16:41:20.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f549cb73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f549815bf90 con 0x7f54980ccfb0 2023-10-14T16:41:20.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54980ad970 0x7f54980cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54980ad970 0x7f54980cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37062/0 (socket says 172.21.15.29:37062) 2023-10-14T16:41:20.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 -- 172.21.15.29:0/3741385922 learned_addr learned my addr 172.21.15.29:0/3741385922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:20.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5497fff640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 0x7f54980cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 -- 172.21.15.29:0/3741385922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 msgr2=0x7f54980cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:20.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 0x7f54980cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.063+0000 7f5496ffd640 1 -- 172.21.15.29:0/3741385922 >> v1:172.21.15.186:6789/0 conn(0x7f549806c040 legacy=0x7f549806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.064+0000 7f5496ffd640 1 -- 172.21.15.29:0/3741385922 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54980cc270 con 0x7f54980ad970 2023-10-14T16:41:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.064+0000 7f5497fff640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 0x7f54980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.065+0000 7f5496ffd640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ad970 0x7f54980cca70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f548c009870 tx=0x7f548c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58b33c325b9e9fae server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.065+0000 7f54967fc640 1 -- 172.21.15.29:0/3741385922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c031640 con 0x7f54980ad970 2023-10-14T16:41:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.065+0000 7f54967fc640 1 -- 172.21.15.29:0/3741385922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f548c0317a0 con 0x7f54980ad970 2023-10-14T16:41:20.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.065+0000 7f54967fc640 1 -- 172.21.15.29:0/3741385922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f548c031b30 con 0x7f54980ad970 2023-10-14T16:41:20.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 -- 172.21.15.29:0/3741385922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ad970 msgr2=0x7f54980cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ad970 0x7f54980cca70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f548c009870 tx=0x7f548c02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 -- 172.21.15.29:0/3741385922 shutdown_connections 2023-10-14T16:41:20.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54980ccfb0 0x7f54980cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 --2- 172.21.15.29:0/3741385922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ad970 0x7f54980cca70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 -- 172.21.15.29:0/3741385922 >> 172.21.15.29:0/3741385922 conn(0x7f54980ce6a0 msgr2=0x7f54980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:20.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 -- 172.21.15.29:0/3741385922 shutdown_connections 2023-10-14T16:41:20.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.066+0000 7f549cb73640 1 -- 172.21.15.29:0/3741385922 wait complete. 2023-10-14T16:41:20.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.067+0000 7f549cb73640 1 Processor -- start 2023-10-14T16:41:20.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.068+0000 7f549cb73640 1 -- start start 2023-10-14T16:41:20.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f549806c040 0x7f54981f1d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 0x7f549806f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54981f2240 con 0x7f54980ccfb0 2023-10-14T16:41:20.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54980c6f00 con 0x7f54980ad970 2023-10-14T16:41:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f549cb73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54980c71e0 con 0x7f549806c040 2023-10-14T16:41:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37064/0 (socket says 172.21.15.29:37064) 2023-10-14T16:41:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 -- 172.21.15.29:0/2059622159 learned_addr learned my addr 172.21.15.29:0/2059622159 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:20.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5496ffd640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 0x7f549806f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 -- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f549806c040 msgr2=0x7f54981f1d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f549806c040 0x7f54981f1d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 -- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 msgr2=0x7f549806f700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 0x7f549806f700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5497fff640 1 -- 172.21.15.29:0/2059622159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c0094d0 con 0x7f54980ccfb0 2023-10-14T16:41:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.069+0000 7f5496ffd640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 0x7f549806f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f5497fff640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f549000cd50 tx=0x7f549000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000f680 con 0x7f54980ccfb0 2023-10-14T16:41:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f549000f800 con 0x7f54980ccfb0 2023-10-14T16:41:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54980c74c0 con 0x7f54980ccfb0 2023-10-14T16:41:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f549000fbb0 con 0x7f54980ccfb0 2023-10-14T16:41:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.070+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5498152a90 con 0x7f54980ccfb0 2023-10-14T16:41:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.071+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549806b330 con 0x7f54980ccfb0 2023-10-14T16:41:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.072+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f549000b040 con 0x7f54980ccfb0 2023-10-14T16:41:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.073+0000 7f547ffff640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 0x7f548406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.073+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f549008a0a0 con 0x7f54980ccfb0 2023-10-14T16:41:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.073+0000 7f54977fe640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 0x7f548406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.074+0000 7f54977fe640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 0x7f548406d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5488003cc0 tx=0x7f54880079b0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.077+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54900550c0 con 0x7f54980ccfb0 2023-10-14T16:41:20.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.288+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f5498152cb0 con 0x7f54980ccfb0 2023-10-14T16:41:20.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.289+0000 7f547ffff640 1 -- 172.21.15.29:0/2059622159 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v22) v1 ==== 80+0+0 (secure 0 0 0) 0x7f54900586e0 con 0x7f54980ccfb0 2023-10-14T16:41:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:listed 0 entries 2023-10-14T16:41:20.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.291+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 msgr2=0x7f548406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.291+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 0x7f548406d330 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f5488003cc0 tx=0x7f54880079b0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.291+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 msgr2=0x7f549806fc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.291+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f549000cd50 tx=0x7f549000a650 comp rx=0 tx=0).stop 2023-10-14T16:41:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 shutdown_connections 2023-10-14T16:41:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f548406ae70 0x7f548406d330 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54980ccfb0 0x7f549806fc40 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54980ad970 0x7f549806f700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 --2- 172.21.15.29:0/2059622159 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f549806c040 0x7f54981f1d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 >> 172.21.15.29:0/2059622159 conn(0x7f54980ce6a0 msgr2=0x7f549806c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:20.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 shutdown_connections 2023-10-14T16:41:20.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.292+0000 7f549cb73640 1 -- 172.21.15.29:0/2059622159 wait complete. 2023-10-14T16:41:20.311 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.3... 2023-10-14T16:41:20.312 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:20.312 DEBUG:teuthology.orchestra.run.smithi186:> sudo umount /home/ubuntu/cephtest/mnt.3 -f 2023-10-14T16:41:20.421 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T16:41:20.422 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:20.422 DEBUG:teuthology.orchestra.run.smithi186:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2023-10-14T16:41:20.472 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:41:20.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.706+0000 7fdf1a002640 1 Processor -- start 2023-10-14T16:41:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.706+0000 7fdf1a002640 1 -- start start 2023-10-14T16:41:20.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf1a002640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdf140b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf1a002640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1406f7f0 con 0x7fdf1415d880 2023-10-14T16:41:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf1a002640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf1406c5e0 con 0x7fdf14159b20 2023-10-14T16:41:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf1a002640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1406c8c0 con 0x7fdf14159f00 2023-10-14T16:41:20.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf13fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdf1415d880 0x7fdf1406fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48170/0 (socket says 172.21.15.29:48170) 2023-10-14T16:41:20.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf13fff640 1 -- 172.21.15.29:0/1359480326 learned_addr learned my addr 172.21.15.29:0/1359480326 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:20.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.708+0000 7fdf12ffd640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdf140b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319416916 0 0) 0x7fdf1406f7f0 con 0x7fdf1415d880 2023-10-14T16:41:20.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdefc005730 con 0x7fdf1415d880 2023-10-14T16:41:20.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3836114948 0 0) 0x7fdefc005730 con 0x7fdf1415d880 2023-10-14T16:41:20.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 msgr2=0x7fdf140b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdf140b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 >> v1:172.21.15.186:6789/0 conn(0x7fdf14159b20 legacy=0x7fdf1406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1406cba0 con 0x7fdf1415d880 2023-10-14T16:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdf0c0019f0 con 0x7fdf1415d880 2023-10-14T16:41:20.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 >> v1:172.21.15.29:6789/0 conn(0x7fdf1415d880 legacy=0x7fdf1406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdefc005bb0 0x7fdefc005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdefc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdefc006f10 0x7fdefc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1406c8c0 con 0x7fdefc006f10 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc00dc40 con 0x7fdefc005bb0 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.709+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdefc00de00 con 0x7fdf14159f00 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdefc006f10 0x7fdefc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf137fe640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdefc005bb0 0x7fdefc005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf12ffd640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdefc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 -- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 msgr2=0x7fdefc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf14159f00 0x7fdefc0069d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 -- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdefc005bb0 msgr2=0x7fdefc005fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdefc005bb0 0x7fdefc005fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.710+0000 7fdf13fff640 1 -- 172.21.15.29:0/1359480326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdefc005730 con 0x7fdefc006f10 2023-10-14T16:41:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.711+0000 7fdf13fff640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdefc006f10 0x7fdefc007390 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00a800 tx=0x7fdefc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c566426263967d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.711+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0c0072d0 con 0x7fdefc006f10 2023-10-14T16:41:20.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.711+0000 7fdf127fc640 1 -- 172.21.15.29:0/1359480326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0c02f070 con 0x7fdefc006f10 2023-10-14T16:41:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 -- 172.21.15.29:0/1359480326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdefc006f10 msgr2=0x7fdefc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 --2- 172.21.15.29:0/1359480326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdefc006f10 0x7fdefc007390 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fdf0c00a800 tx=0x7fdefc007a00 comp rx=0 tx=0).stop 2023-10-14T16:41:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf13fff640 1 -- 172.21.15.29:0/1359480326 reap_dead start 2023-10-14T16:41:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 -- 172.21.15.29:0/1359480326 shutdown_connections 2023-10-14T16:41:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 -- 172.21.15.29:0/1359480326 >> 172.21.15.29:0/1359480326 conn(0x7fdf140cb410 msgr2=0x7fdf140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 -- 172.21.15.29:0/1359480326 shutdown_connections 2023-10-14T16:41:20.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.712+0000 7fdf1a002640 1 -- 172.21.15.29:0/1359480326 wait complete. 2023-10-14T16:41:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.714+0000 7fdf1a002640 1 Processor -- start 2023-10-14T16:41:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.714+0000 7fdf1a002640 1 -- start start 2023-10-14T16:41:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdefc006f10 0x7fdf1415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 0x7fdf14147d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf141475f0 con 0x7fdf140c4800 2023-10-14T16:41:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140c3510 con 0x7fdf14147890 2023-10-14T16:41:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf1a002640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf140c4370 con 0x7fdefc006f10 2023-10-14T16:41:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf12ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37074/0 (socket says 172.21.15.29:37074) 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf12ffd640 1 -- 172.21.15.29:0/453992814 learned_addr learned my addr 172.21.15.29:0/453992814 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf137fe640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdefc006f10 0x7fdf1415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf13fff640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 0x7fdf14147d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.715+0000 7fdf12ffd640 1 -- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdefc006f10 msgr2=0x7fdf1415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf12ffd640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdefc006f10 0x7fdf1415cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf12ffd640 1 -- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 msgr2=0x7fdf14147d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf12ffd640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 0x7fdf14147d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf12ffd640 1 -- 172.21.15.29:0/453992814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf140c3760 con 0x7fdf140c4800 2023-10-14T16:41:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf13fff640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 0x7fdf14147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf12ffd640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdf08010890 tx=0x7fdf08010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf080095b0 con 0x7fdf140c4800 2023-10-14T16:41:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf0800eb80 con 0x7fdf140c4800 2023-10-14T16:41:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf0801c400 con 0x7fdf140c4800 2023-10-14T16:41:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf140c3a40 con 0x7fdf140c4800 2023-10-14T16:41:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.716+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf140c4000 con 0x7fdf140c4800 2023-10-14T16:41:20.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.718+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1404e160 con 0x7fdf140c4800 2023-10-14T16:41:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.718+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdf080143f0 con 0x7fdf140c4800 2023-10-14T16:41:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.719+0000 7fdefbfff640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 0x7fdf0006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.719+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdf08089950 con 0x7fdf140c4800 2023-10-14T16:41:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.720+0000 7fdf137fe640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 0x7fdf0006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.720+0000 7fdf137fe640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 0x7fdf0006d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf04007fd0 tx=0x7fdf04009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:20.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.724+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf080548e0 con 0x7fdf140c4800 2023-10-14T16:41:20.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.961+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf140c44b0 con 0x7fdf140c4800 2023-10-14T16:41:20.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.962+0000 7fdefbfff640 1 -- 172.21.15.29:0/453992814 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fdf08057f00 con 0x7fdf140c4800 2023-10-14T16:41:20.963 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:20.963 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:41:20.963 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:41:20.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 msgr2=0x7fdf0006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 0x7fdf0006d330 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fdf04007fd0 tx=0x7fdf04009040 comp rx=0 tx=0).stop 2023-10-14T16:41:20.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 msgr2=0x7fdf140c4c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:20.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fdf08010890 tx=0x7fdf08010d50 comp rx=0 tx=0).stop 2023-10-14T16:41:20.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 shutdown_connections 2023-10-14T16:41:20.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf0006ae70 0x7fdf0006d330 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf14147890 0x7fdf14147d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf140c4800 0x7fdf140c4c80 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 --2- 172.21.15.29:0/453992814 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdefc006f10 0x7fdf1415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.964+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 >> 172.21.15.29:0/453992814 conn(0x7fdf140cb410 msgr2=0x7fdf140c7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.965+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 shutdown_connections 2023-10-14T16:41:20.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:20.965+0000 7fdf1a002640 1 -- 172.21.15.29:0/453992814 wait complete. 2023-10-14T16:41:20.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.210+0000 7f000fda9640 1 Processor -- start 2023-10-14T16:41:21.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.210+0000 7f000fda9640 1 -- start start 2023-10-14T16:41:21.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000fda9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7f000815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000fda9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f000815ed60 con 0x7f00080ce390 2023-10-14T16:41:21.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000fda9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00080707e0 con 0x7f000815aaa0 2023-10-14T16:41:21.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000fda9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000806b810 con 0x7f000815ae80 2023-10-14T16:41:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000e346640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f00080ce390 0x7f0008070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40292/0 (socket says 172.21.15.29:40292) 2023-10-14T16:41:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.212+0000 7f000e346640 1 -- 172.21.15.29:0/439999794 learned_addr learned my addr 172.21.15.29:0/439999794 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000d344640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7f000815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356938474 0 0) 0x7f000815ed60 con 0x7f00080ce390 2023-10-14T16:41:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efff4003520 con 0x7f00080ce390 2023-10-14T16:41:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4012365885 0 0) 0x7efff4003520 con 0x7f00080ce390 2023-10-14T16:41:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 msgr2=0x7f000815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7f000815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 >> v1:172.21.15.186:6789/0 conn(0x7f000815aaa0 legacy=0x7f00080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f000815ef90 con 0x7f00080ce390 2023-10-14T16:41:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f00000019f0 con 0x7f00080ce390 2023-10-14T16:41:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 >> v1:172.21.15.29:6789/0 conn(0x7f00080ce390 legacy=0x7f0008070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7efff40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efff4003f70 0x7efff40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.213+0000 7f000cb43640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7efff4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000806b810 con 0x7efff4004d00 2023-10-14T16:41:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efff400bac0 con 0x7efff4003f70 2023-10-14T16:41:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efff400bc80 con 0x7f000815ae80 2023-10-14T16:41:21.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7efff4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000d344640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efff4003f70 0x7efff40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000db45640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7efff40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 -- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 msgr2=0x7efff40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000815ae80 0x7efff40039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 -- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efff4003f70 msgr2=0x7efff40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:21.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efff4003f70 0x7efff40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.214+0000 7f000e346640 1 -- 172.21.15.29:0/439999794 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff4003520 con 0x7efff4004d00 2023-10-14T16:41:21.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.215+0000 7f000e346640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7efff4005180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f000000a760 tx=0x7efff40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7168bc2560c6b40 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000034630 con 0x7efff4004d00 2023-10-14T16:41:21.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000002f070 con 0x7efff4004d00 2023-10-14T16:41:21.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000cb43640 1 -- 172.21.15.29:0/439999794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00000349f0 con 0x7efff4004d00 2023-10-14T16:41:21.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000fda9640 1 -- 172.21.15.29:0/439999794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 msgr2=0x7efff4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000fda9640 1 --2- 172.21.15.29:0/439999794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7efff4005180 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f000000a760 tx=0x7efff40057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000e346640 1 -- 172.21.15.29:0/439999794 reap_dead start 2023-10-14T16:41:21.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000fda9640 1 -- 172.21.15.29:0/439999794 shutdown_connections 2023-10-14T16:41:21.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.216+0000 7f000fda9640 1 -- 172.21.15.29:0/439999794 >> 172.21.15.29:0/439999794 conn(0x7f00080cb410 msgr2=0x7f00080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:21.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.217+0000 7f000fda9640 1 -- 172.21.15.29:0/439999794 shutdown_connections 2023-10-14T16:41:21.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.217+0000 7f000fda9640 1 -- 172.21.15.29:0/439999794 wait complete. 2023-10-14T16:41:21.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.218+0000 7f000fda9640 1 Processor -- start 2023-10-14T16:41:21.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 -- start start 2023-10-14T16:41:21.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000806bb20 0x7f000814d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0008148ac0 0x7f000814d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00081453d0 con 0x7efff4004d00 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00080cb040 con 0x7f0008148ac0 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.219+0000 7f000fda9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000814f3b0 con 0x7f000806bb20 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60410/0 (socket says 172.21.15.29:60410) 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 -- 172.21.15.29:0/1288291771 learned_addr learned my addr 172.21.15.29:0/1288291771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:21.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 -- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000806bb20 msgr2=0x7f000814d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000806bb20 0x7f000814d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 -- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0008148ac0 msgr2=0x7f000814d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0008148ac0 0x7f000814d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 -- 172.21.15.29:0/1288291771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080c2910 con 0x7efff4004d00 2023-10-14T16:41:21.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.220+0000 7f000db45640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f000400d570 tx=0x7f000400da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.221+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004005480 con 0x7efff4004d00 2023-10-14T16:41:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.221+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0004003610 con 0x7efff4004d00 2023-10-14T16:41:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.221+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0004003980 con 0x7efff4004d00 2023-10-14T16:41:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.221+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f000814f0d0 con 0x7efff4004d00 2023-10-14T16:41:21.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.221+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000814deb0 con 0x7efff4004d00 2023-10-14T16:41:21.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.222+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f000814d280 con 0x7efff4004d00 2023-10-14T16:41:21.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.223+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0004003ae0 con 0x7efff4004d00 2023-10-14T16:41:21.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.224+0000 7efffe7fc640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 0x7effec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.224+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0004089510 con 0x7efff4004d00 2023-10-14T16:41:21.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.224+0000 7f000d344640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 0x7effec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.225+0000 7f000d344640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 0x7effec06d330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f000814f060 tx=0x7efff8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.228+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0004054530 con 0x7efff4004d00 2023-10-14T16:41:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.426+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00080b0960 con 0x7efff4004d00 2023-10-14T16:41:21.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.430+0000 7efffe7fc640 1 -- 172.21.15.29:0/1288291771 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7f00080b0960 con 0x7efff4004d00 2023-10-14T16:41:21.431 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:21.431 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 msgr2=0x7effec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 0x7effec06d330 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f000814f060 tx=0x7efff8008040 comp rx=0 tx=0).stop 2023-10-14T16:41:21.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 msgr2=0x7f0008144e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f000400d570 tx=0x7f000400da30 comp rx=0 tx=0).stop 2023-10-14T16:41:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 shutdown_connections 2023-10-14T16:41:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7effec06ae70 0x7effec06d330 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0008148ac0 0x7f000814d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f000806bb20 0x7f000814d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 --2- 172.21.15.29:0/1288291771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efff4004d00 0x7f0008144e90 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.432+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 >> 172.21.15.29:0/1288291771 conn(0x7f00080cb410 msgr2=0x7f000806cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.433+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 shutdown_connections 2023-10-14T16:41:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.433+0000 7f000fda9640 1 -- 172.21.15.29:0/1288291771 wait complete. 2023-10-14T16:41:21.452 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:40:23.612990+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:40:24.238212+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:21.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:21.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.684+0000 7f7cacf2c640 1 Processor -- start 2023-10-14T16:41:21.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.684+0000 7f7cacf2c640 1 -- start start 2023-10-14T16:41:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.685+0000 7f7cacf2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ca815d150 0x7f7ca80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.685+0000 7f7cacf2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7ca806a4b0 0x7f7ca8160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.685+0000 7f7cacf2c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ca80ac890 con 0x7f7ca8160b90 2023-10-14T16:41:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7cacf2c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca815d530 con 0x7f7ca815d150 2023-10-14T16:41:21.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7cacf2c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ca8161650 con 0x7f7ca806a4b0 2023-10-14T16:41:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7ca659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ca815d150 0x7f7ca80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7ca659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ca815d150 0x7f7ca80ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60426/0 (socket says 172.21.15.29:60426) 2023-10-14T16:41:21.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7ca659c640 1 -- 172.21.15.29:0/1373808076 learned_addr learned my addr 172.21.15.29:0/1373808076 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:21.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7ca6d9d640 1 --2- 172.21.15.29:0/1373808076 >> v2:172.21.15.186:3300/0 conn(0x7f7ca806a4b0 0x7f7ca8160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.686+0000 7f7ca659c640 1 -- 172.21.15.29:0/1373808076 >> v2:172.21.15.186:3300/0 conn(0x7f7ca806a4b0 msgr2=0x7f7ca8160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:21.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.687+0000 7f7ca659c640 1 --2- 172.21.15.29:0/1373808076 >> v2:172.21.15.186:3300/0 conn(0x7f7ca806a4b0 0x7f7ca8160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.687+0000 7f7ca659c640 1 -- 172.21.15.29:0/1373808076 >> v1:172.21.15.186:6789/0 conn(0x7f7ca8160b90 legacy=0x7f7ca8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.687+0000 7f7ca659c640 1 -- 172.21.15.29:0/1373808076 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ca8161930 con 0x7f7ca815d150 2023-10-14T16:41:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.688+0000 7f7ca659c640 1 --2- 172.21.15.29:0/1373808076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca815d150 0x7f7ca80ad350 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80adde0 tx=0x7f7c9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5db95a97f0f15eac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.688+0000 7f7ca5d9b640 1 -- 172.21.15.29:0/1373808076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c980317e0 con 0x7f7ca815d150 2023-10-14T16:41:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.688+0000 7f7ca5d9b640 1 -- 172.21.15.29:0/1373808076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c98031940 con 0x7f7ca815d150 2023-10-14T16:41:21.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.688+0000 7f7ca5d9b640 1 -- 172.21.15.29:0/1373808076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c98031cd0 con 0x7f7ca815d150 2023-10-14T16:41:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 -- 172.21.15.29:0/1373808076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca815d150 msgr2=0x7f7ca80ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/1373808076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca815d150 0x7f7ca80ad350 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f7ca80adde0 tx=0x7f7c9802b880 comp rx=0 tx=0).stop 2023-10-14T16:41:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 -- 172.21.15.29:0/1373808076 shutdown_connections 2023-10-14T16:41:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/1373808076 >> v2:172.21.15.186:3300/0 conn(0x7f7ca806a4b0 0x7f7ca8160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/1373808076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca815d150 0x7f7ca80ad350 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 -- 172.21.15.29:0/1373808076 >> 172.21.15.29:0/1373808076 conn(0x7f7ca80ce9b0 msgr2=0x7f7ca80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:21.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 -- 172.21.15.29:0/1373808076 shutdown_connections 2023-10-14T16:41:21.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.689+0000 7f7cacf2c640 1 -- 172.21.15.29:0/1373808076 wait complete. 2023-10-14T16:41:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.692+0000 7f7cacf2c640 1 Processor -- start 2023-10-14T16:41:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.692+0000 7f7cacf2c640 1 -- start start 2023-10-14T16:41:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca806a4b0 0x7f7ca81f37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ca815d150 0x7f7ca80af220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca81f3350 con 0x7f7ca8160b90 2023-10-14T16:41:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca81f6d60 con 0x7f7ca806a4b0 2023-10-14T16:41:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.693+0000 7f7cacf2c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ca81f7040 con 0x7f7ca815d150 2023-10-14T16:41:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60428/0 (socket says 172.21.15.29:60428) 2023-10-14T16:41:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 -- 172.21.15.29:0/3459341550 learned_addr learned my addr 172.21.15.29:0/3459341550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca659c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca806a4b0 0x7f7ca81f37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 -- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ca815d150 msgr2=0x7f7ca80af220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ca815d150 0x7f7ca80af220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 -- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca806a4b0 msgr2=0x7f7ca81f37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca806a4b0 0x7f7ca81f37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 -- 172.21.15.29:0/3459341550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c980094d0 con 0x7f7ca8160b90 2023-10-14T16:41:21.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.694+0000 7f7ca6d9d640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000cc30 tx=0x7f7ca000a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.695+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca000ce40 con 0x7f7ca8160b90 2023-10-14T16:41:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.695+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ca0003990 con 0x7f7ca8160b90 2023-10-14T16:41:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.695+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ca0003d20 con 0x7f7ca8160b90 2023-10-14T16:41:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.695+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ca81f2fe0 con 0x7f7ca8160b90 2023-10-14T16:41:21.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.695+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ca81f2cb0 con 0x7f7ca8160b90 2023-10-14T16:41:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.696+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ca81f34c0 con 0x7f7ca8160b90 2023-10-14T16:41:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.697+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f7ca0003e80 con 0x7f7ca8160b90 2023-10-14T16:41:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.698+0000 7f7c9effd640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 0x7f7c8806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.699+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7ca0089ac0 con 0x7f7ca8160b90 2023-10-14T16:41:21.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.699+0000 7f7ca659c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 0x7f7c8806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:21.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.700+0000 7f7ca659c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 0x7f7c8806d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c98036000 tx=0x7f7c9803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:21.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.703+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ca0054a50 con 0x7f7ca8160b90 2023-10-14T16:41:21.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.900+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f7ca80ace00 con 0x7f7ca8160b90 2023-10-14T16:41:21.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.901+0000 7f7c9effd640 1 -- 172.21.15.29:0/3459341550 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) v1 ==== 93+0+10 (secure 0 0 0) 0x7f7ca001d070 con 0x7f7ca8160b90 2023-10-14T16:41:21.903 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 msgr2=0x7f7c8806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 0x7f7c8806d330 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f7c98036000 tx=0x7f7c9803d040 comp rx=0 tx=0).stop 2023-10-14T16:41:21.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 msgr2=0x7f7ca81f62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:21.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f7ca000cc30 tx=0x7f7ca000a5c0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 shutdown_connections 2023-10-14T16:41:21.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f7c8806ae70 0x7f7c8806d330 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ca8160b90 0x7f7ca81f62a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ca815d150 0x7f7ca80af220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 --2- 172.21.15.29:0/3459341550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ca806a4b0 0x7f7ca81f37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:21.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 >> 172.21.15.29:0/3459341550 conn(0x7f7ca80ce9b0 msgr2=0x7f7ca80c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:21.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 shutdown_connections 2023-10-14T16:41:21.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:21.904+0000 7f7cacf2c640 1 -- 172.21.15.29:0/3459341550 wait complete. 2023-10-14T16:41:21.921 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:41:22.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.145+0000 7f0e6244b640 1 Processor -- start 2023-10-14T16:41:22.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.146+0000 7f0e6244b640 1 -- start start 2023-10-14T16:41:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.147+0000 7f0e6244b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 0x7f0e5c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.147+0000 7f0e6244b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c070360 con 0x7f0e5c159ab0 2023-10-14T16:41:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.147+0000 7f0e6244b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e5c0cd1f0 con 0x7f0e5c06c140 2023-10-14T16:41:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.147+0000 7f0e6244b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c15d890 con 0x7f0e5c159e90 2023-10-14T16:41:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0e5c159ab0 0x7f0e5c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40300/0 (socket says 172.21.15.29:40300) 2023-10-14T16:41:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5bfff640 1 -- 172.21.15.29:0/2440717178 learned_addr learned my addr 172.21.15.29:0/2440717178 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 0x7f0e5c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1442549211 0 0) 0x7f0e5c070360 con 0x7f0e5c159ab0 2023-10-14T16:41:22.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e44003520 con 0x7f0e5c159ab0 2023-10-14T16:41:22.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1273879139 0 0) 0x7f0e44003520 con 0x7f0e5c159ab0 2023-10-14T16:41:22.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 msgr2=0x7f0e5c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:22.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.148+0000 7f0e5affd640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 0x7f0e5c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 >> v1:172.21.15.186:6789/0 conn(0x7f0e5c06c140 legacy=0x7f0e5c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c0cd4d0 con 0x7f0e5c159ab0 2023-10-14T16:41:22.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0e4c001670 con 0x7f0e5c159ab0 2023-10-14T16:41:22.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 >> v1:172.21.15.29:6789/0 conn(0x7f0e5c159ab0 legacy=0x7f0e5c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 0x7f0e440039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e44004d00 0x7f0e44005160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c15d890 con 0x7f0e44003f70 2023-10-14T16:41:22.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4400bc10 con 0x7f0e44004d00 2023-10-14T16:41:22.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e4400dfe0 con 0x7f0e5c159e90 2023-10-14T16:41:22.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5b7fe640 1 -- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 msgr2=0x7f0e440039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:22.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c159e90 0x7f0e440039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e609e8640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e44004d00 0x7f0e44005160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.149+0000 7f0e5b7fe640 1 -- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e44004d00 msgr2=0x7f0e44005160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.150+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e44004d00 0x7f0e44005160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.150+0000 7f0e5b7fe640 1 -- 172.21.15.29:0/2440717178 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e44003520 con 0x7f0e44003f70 2023-10-14T16:41:22.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.150+0000 7f0e609e8640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e44004d00 0x7f0e44005160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:22.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e440047c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009ac0 tx=0x7f0e440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=545042d57168310 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5002c710 con 0x7f0e44003f70 2023-10-14T16:41:22.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e5002c870 con 0x7f0e44003f70 2023-10-14T16:41:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e5affd640 1 -- 172.21.15.29:0/2440717178 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e5002cc00 con 0x7f0e44003f70 2023-10-14T16:41:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e6244b640 1 -- 172.21.15.29:0/2440717178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 msgr2=0x7f0e440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2440717178 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e440047c0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f0e50009ac0 tx=0x7f0e440057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e609e8640 1 -- 172.21.15.29:0/2440717178 reap_dead start 2023-10-14T16:41:22.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e6244b640 1 -- 172.21.15.29:0/2440717178 shutdown_connections 2023-10-14T16:41:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.151+0000 7f0e6244b640 1 -- 172.21.15.29:0/2440717178 >> 172.21.15.29:0/2440717178 conn(0x7f0e5c0ce9b0 msgr2=0x7f0e5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.152+0000 7f0e6244b640 1 -- 172.21.15.29:0/2440717178 shutdown_connections 2023-10-14T16:41:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.152+0000 7f0e6244b640 1 -- 172.21.15.29:0/2440717178 wait complete. 2023-10-14T16:41:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.153+0000 7f0e6244b640 1 Processor -- start 2023-10-14T16:41:22.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.154+0000 7f0e6244b640 1 -- start start 2023-10-14T16:41:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.154+0000 7f0e6244b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.154+0000 7f0e6244b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c155180 0x7f0e5c155600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.154+0000 7f0e6244b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5c1f5fa0 0x7f0e5c1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e6244b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c2ea0 con 0x7f0e44003f70 2023-10-14T16:41:22.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e6244b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c1f6960 con 0x7f0e5c1f5fa0 2023-10-14T16:41:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e6244b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e5c0c1a90 con 0x7f0e5c155180 2023-10-14T16:41:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60440/0 (socket says 172.21.15.29:60440) 2023-10-14T16:41:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 -- 172.21.15.29:0/2343264913 learned_addr learned my addr 172.21.15.29:0/2343264913 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:22.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e609e8640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5c1f5fa0 0x7f0e5c1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 -- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c155180 msgr2=0x7f0e5c155600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c155180 0x7f0e5c155600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 -- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5c1f5fa0 msgr2=0x7f0e5c1f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5c1f5fa0 0x7f0e5c1f6420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.155+0000 7f0e5bfff640 1 -- 172.21.15.29:0/2343264913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e5c155c20 con 0x7f0e44003f70 2023-10-14T16:41:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e5bfff640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c700 tx=0x7f0e4c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0072d0 con 0x7f0e44003f70 2023-10-14T16:41:22.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e4c007430 con 0x7f0e44003f70 2023-10-14T16:41:22.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e4c0047e0 con 0x7f0e44003f70 2023-10-14T16:41:22.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e5c155f00 con 0x7f0e44003f70 2023-10-14T16:41:22.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.156+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e5c1564c0 con 0x7f0e44003f70 2023-10-14T16:41:22.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.157+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e5c06bd50 con 0x7f0e44003f70 2023-10-14T16:41:22.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.158+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0e4c004940 con 0x7f0e44003f70 2023-10-14T16:41:22.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.159+0000 7f0e58ff9640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 0x7f0e4806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.159+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0e4c088ad0 con 0x7f0e44003f70 2023-10-14T16:41:22.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.159+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 0x7f0e4806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.160+0000 7f0e5b7fe640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 0x7f0e4806d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e50036000 tx=0x7f0e5002e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.164+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e4c053b00 con 0x7f0e44003f70 2023-10-14T16:41:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.358+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e5c04e160 con 0x7f0e44003f70 2023-10-14T16:41:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.358+0000 7f0e58ff9640 1 -- 172.21.15.29:0/2343264913 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0e4c018020 con 0x7f0e44003f70 2023-10-14T16:41:22.360 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:41:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 msgr2=0x7f0e4806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 0x7f0e4806d330 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0e50036000 tx=0x7f0e5002e000 comp rx=0 tx=0).stop 2023-10-14T16:41:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 msgr2=0x7f0e5c0c2960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f0e4c00c700 tx=0x7f0e4c00cbc0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 shutdown_connections 2023-10-14T16:41:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0e4806ae70 0x7f0e4806d330 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e5c1f5fa0 0x7f0e5c1f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e5c155180 0x7f0e5c155600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 --2- 172.21.15.29:0/2343264913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e44003f70 0x7f0e5c0c2960 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 >> 172.21.15.29:0/2343264913 conn(0x7f0e5c0ce9b0 msgr2=0x7f0e5c1e2960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 shutdown_connections 2023-10-14T16:41:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.361+0000 7f0e6244b640 1 -- 172.21.15.29:0/2343264913 wait complete. 2023-10-14T16:41:22.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:41:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.557+0000 7f6ce2661640 1 Processor -- start 2023-10-14T16:41:22.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.557+0000 7f6ce2661640 1 -- start start 2023-10-14T16:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6ce2661640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6cdc15ad70 0x7f6cdc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6ce2661640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0caac0 0x7f6cdc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6ce2661640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cdc0cb030 con 0x7f6cdc0c9ca0 2023-10-14T16:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6ce2661640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cdc0c91c0 con 0x7f6cdc15ad70 2023-10-14T16:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6ce2661640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc0c94a0 con 0x7f6cdc0caac0 2023-10-14T16:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6cdbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6cdc15ad70 0x7f6cdc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6cdbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6cdc15ad70 0x7f6cdc069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60456/0 (socket says 172.21.15.29:60456) 2023-10-14T16:41:22.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6cdbfff640 1 -- 172.21.15.29:0/581784497 learned_addr learned my addr 172.21.15.29:0/581784497 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:22.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.558+0000 7f6cd3fff640 1 --1- 172.21.15.29:0/581784497 >> v1:172.21.15.29:6789/0 conn(0x7f6cdc0c9ca0 0x7f6cdc0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40306/0 (socket says 172.21.15.29:40306) 2023-10-14T16:41:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6cdbfff640 1 -- 172.21.15.29:0/581784497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0caac0 msgr2=0x7f6cdc06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:22.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6ce0bfe640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0caac0 0x7f6cdc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0caac0 0x7f6cdc06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6cdbfff640 1 -- 172.21.15.29:0/581784497 >> v1:172.21.15.29:6789/0 conn(0x7f6cdc0c9ca0 legacy=0x7f6cdc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6cdbfff640 1 -- 172.21.15.29:0/581784497 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cdc0c9780 con 0x7f6cdc15ad70 2023-10-14T16:41:22.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.559+0000 7f6cdb7fe640 1 -- 172.21.15.29:0/581784497 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2832429813 0 0) 0x7f6cdc0cb030 con 0x7f6cdc0c9ca0 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.560+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc069da0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009870 tx=0x7f6ccc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd15edeae118d613 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.560+0000 7f6cdb7fe640 1 -- 172.21.15.29:0/581784497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc030840 con 0x7f6cdc15ad70 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.560+0000 7f6cdb7fe640 1 -- 172.21.15.29:0/581784497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0309a0 con 0x7f6cdc15ad70 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.560+0000 7f6cdb7fe640 1 -- 172.21.15.29:0/581784497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ccc030d30 con 0x7f6cdc15ad70 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 -- 172.21.15.29:0/581784497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc15ad70 msgr2=0x7f6cdc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc069da0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc009870 tx=0x7f6ccc02b980 comp rx=0 tx=0).stop 2023-10-14T16:41:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 -- 172.21.15.29:0/581784497 shutdown_connections 2023-10-14T16:41:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0caac0 0x7f6cdc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 --2- 172.21.15.29:0/581784497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc069da0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 -- 172.21.15.29:0/581784497 >> 172.21.15.29:0/581784497 conn(0x7f6cdc0ce9b0 msgr2=0x7f6cdc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:22.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 -- 172.21.15.29:0/581784497 shutdown_connections 2023-10-14T16:41:22.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.561+0000 7f6ce2661640 1 -- 172.21.15.29:0/581784497 wait complete. 2023-10-14T16:41:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.563+0000 7f6ce2661640 1 Processor -- start 2023-10-14T16:41:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.563+0000 7f6ce2661640 1 -- start start 2023-10-14T16:41:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 0x7f6cdc158fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc1f5730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc15b450 con 0x7f6cdc0caac0 2023-10-14T16:41:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc15b9d0 con 0x7f6cdc15ad70 2023-10-14T16:41:22.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6ce2661640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc1f6180 con 0x7f6cdc0c9ca0 2023-10-14T16:41:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6cd3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60470/0 (socket says 172.21.15.29:60470) 2023-10-14T16:41:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.564+0000 7f6cd3fff640 1 -- 172.21.15.29:0/84428388 learned_addr learned my addr 172.21.15.29:0/84428388 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 0x7f6cdc158fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 -- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 msgr2=0x7f6cdc158fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 0x7f6cdc158fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6ce0bfe640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc1f5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 -- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 msgr2=0x7f6cdc1f5730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc1f5730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 -- 172.21.15.29:0/84428388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ccc0094d0 con 0x7f6cdc0caac0 2023-10-14T16:41:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 0x7f6cdc158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6ce0bfe640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc1f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd3fff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c4d0 tx=0x7f6cc400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc40074e0 con 0x7f6cdc0caac0 2023-10-14T16:41:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc4007660 con 0x7f6cdc0caac0 2023-10-14T16:41:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc1f6460 con 0x7f6cdc0caac0 2023-10-14T16:41:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.565+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cc4004a20 con 0x7f6cdc0caac0 2023-10-14T16:41:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.566+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc0c5470 con 0x7f6cdc0caac0 2023-10-14T16:41:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.567+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc04e160 con 0x7f6cdc0caac0 2023-10-14T16:41:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.568+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f6cc4004b80 con 0x7f6cdc0caac0 2023-10-14T16:41:22.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.568+0000 7f6cd97fa640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.569+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6cc4088ca0 con 0x7f6cdc0caac0 2023-10-14T16:41:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.569+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.569+0000 7f6cdbfff640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 0x7f6cbc06d330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc036000 tx=0x7f6ccc03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:22.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.576+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6cc4053c30 con 0x7f6cdc0caac0 2023-10-14T16:41:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.765+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6cdc1f6650 con 0x7f6cdc0caac0 2023-10-14T16:41:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.765+0000 7f6cd97fa640 1 -- 172.21.15.29:0/84428388 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6cdc1f6650 con 0x7f6cdc0caac0 2023-10-14T16:41:22.767 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:41:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 msgr2=0x7f6cbc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 0x7f6cbc06d330 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f6ccc036000 tx=0x7f6ccc03b040 comp rx=0 tx=0).stop 2023-10-14T16:41:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 msgr2=0x7f6cdc0c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f6cc400c4d0 tx=0x7f6cc400c990 comp rx=0 tx=0).stop 2023-10-14T16:41:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 shutdown_connections 2023-10-14T16:41:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f6cbc06ae70 0x7f6cbc06d330 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cdc15ad70 0x7f6cdc1f5730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cdc0caac0 0x7f6cdc0c5a80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 --2- 172.21.15.29:0/84428388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cdc0c9ca0 0x7f6cdc158fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 >> 172.21.15.29:0/84428388 conn(0x7f6cdc0ce9b0 msgr2=0x7f6cdc070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.768+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 shutdown_connections 2023-10-14T16:41:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:22.769+0000 7f6ce2661640 1 -- 172.21.15.29:0/84428388 wait complete. 2023-10-14T16:41:22.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:23.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.002+0000 7f3bb9275640 1 Processor -- start 2023-10-14T16:41:23.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.002+0000 7f3bb9275640 1 -- start start 2023-10-14T16:41:23.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb9275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bb415aaa0 0x7f3bb415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb9275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb415ed60 con 0x7f3bb40ce390 2023-10-14T16:41:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb9275640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb40707e0 con 0x7f3bb415ae80 2023-10-14T16:41:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb9275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb406b810 con 0x7f3bb415aaa0 2023-10-14T16:41:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bb415aaa0 0x7f3bb415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3bb415aaa0 0x7f3bb415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60478/0 (socket says 172.21.15.29:60478) 2023-10-14T16:41:23.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.004+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/2306878699 learned_addr learned my addr 172.21.15.29:0/2306878699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.005+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/2306878699 >> v1:172.21.15.186:6789/0 conn(0x7f3bb415ae80 legacy=0x7f3bb40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.005+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/2306878699 >> v1:172.21.15.29:6789/0 conn(0x7f3bb40ce390 legacy=0x7f3bb4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.005+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/2306878699 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb415ef90 con 0x7f3bb415aaa0 2023-10-14T16:41:23.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.006+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/2306878699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415aaa0 0x7f3bb415e820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009870 tx=0x7f3ba402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b30c77d6a2f02c5a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.007+0000 7f3bb1ffb640 1 -- 172.21.15.29:0/2306878699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba402f850 con 0x7f3bb415aaa0 2023-10-14T16:41:23.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.007+0000 7f3bb1ffb640 1 -- 172.21.15.29:0/2306878699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba402f9b0 con 0x7f3bb415aaa0 2023-10-14T16:41:23.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.007+0000 7f3bb1ffb640 1 -- 172.21.15.29:0/2306878699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba402fd40 con 0x7f3bb415aaa0 2023-10-14T16:41:23.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 -- 172.21.15.29:0/2306878699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415aaa0 msgr2=0x7f3bb415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2306878699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415aaa0 0x7f3bb415e820 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4009870 tx=0x7f3ba402ba30 comp rx=0 tx=0).stop 2023-10-14T16:41:23.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 -- 172.21.15.29:0/2306878699 shutdown_connections 2023-10-14T16:41:23.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2306878699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415aaa0 0x7f3bb415e820 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 -- 172.21.15.29:0/2306878699 >> 172.21.15.29:0/2306878699 conn(0x7f3bb40cb410 msgr2=0x7f3bb40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 -- 172.21.15.29:0/2306878699 shutdown_connections 2023-10-14T16:41:23.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.008+0000 7f3bb9275640 1 -- 172.21.15.29:0/2306878699 wait complete. 2023-10-14T16:41:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.010+0000 7f3bb9275640 1 Processor -- start 2023-10-14T16:41:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.010+0000 7f3bb9275640 1 -- start start 2023-10-14T16:41:23.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ce390 0x7f3bb4159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb415aaa0 0x7f3bb414feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb41503f0 con 0x7f3bb415ae80 2023-10-14T16:41:23.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40c3340 con 0x7f3bb40ce390 2023-10-14T16:41:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb9275640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40c3620 con 0x7f3bb415aaa0 2023-10-14T16:41:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60482/0 (socket says 172.21.15.29:60482) 2023-10-14T16:41:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb37fe640 1 -- 172.21.15.29:0/2290722646 learned_addr learned my addr 172.21.15.29:0/2290722646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.011+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ce390 0x7f3bb4159930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb27fc640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb415aaa0 0x7f3bb414feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 -- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb415aaa0 msgr2=0x7f3bb414feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb415aaa0 0x7f3bb414feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 -- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ce390 msgr2=0x7f3bb4159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ce390 0x7f3bb4159930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 -- 172.21.15.29:0/2290722646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba40094d0 con 0x7f3bb415ae80 2023-10-14T16:41:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00c7a0 tx=0x7f3bac00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac003b70 con 0x7f3bb415ae80 2023-10-14T16:41:23.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bac003cf0 con 0x7f3bb415ae80 2023-10-14T16:41:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb40c3900 con 0x7f3bb415ae80 2023-10-14T16:41:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.012+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bac003ea0 con 0x7f3bb415ae80 2023-10-14T16:41:23.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.013+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb40c3ec0 con 0x7f3bb415ae80 2023-10-14T16:41:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.014+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb40c40e0 con 0x7f3bb415ae80 2023-10-14T16:41:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.015+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3bac00aa90 con 0x7f3bb415ae80 2023-10-14T16:41:23.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.016+0000 7f3b9b7fe640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 0x7f3ba006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.016+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3bac089430 con 0x7f3bb415ae80 2023-10-14T16:41:23.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.016+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 0x7f3ba006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.017+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 0x7f3ba006d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4036000 tx=0x7f3ba4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.021+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3bac054460 con 0x7f3bb415ae80 2023-10-14T16:41:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.254+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb404e160 con 0x7f3bb415ae80 2023-10-14T16:41:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.255+0000 7f3b9b7fe640 1 -- 172.21.15.29:0/2290722646 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7f3bac057a80 con 0x7f3bb415ae80 2023-10-14T16:41:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:23.256 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:41:23.257 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:41:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.257+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 msgr2=0x7f3ba006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.257+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 0x7f3ba006d330 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3ba4036000 tx=0x7f3ba4008030 comp rx=0 tx=0).stop 2023-10-14T16:41:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.257+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 msgr2=0x7f3bb40b14b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.257+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f3bac00c7a0 tx=0x7f3bac00cc60 comp rx=0 tx=0).stop 2023-10-14T16:41:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 shutdown_connections 2023-10-14T16:41:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3ba006ae70 0x7f3ba006d330 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bb415ae80 0x7f3bb40b14b0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb415aaa0 0x7f3bb414feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 --2- 172.21.15.29:0/2290722646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ce390 0x7f3bb4159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 >> 172.21.15.29:0/2290722646 conn(0x7f3bb40cb410 msgr2=0x7f3bb40cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 shutdown_connections 2023-10-14T16:41:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.258+0000 7f3bb9275640 1 -- 172.21.15.29:0/2290722646 wait complete. 2023-10-14T16:41:23.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:23.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.494+0000 7fe3cda15640 1 Processor -- start 2023-10-14T16:41:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.495+0000 7fe3cda15640 1 -- start start 2023-10-14T16:41:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.496+0000 7fe3cda15640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 0x7fe3c806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.496+0000 7fe3cda15640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c8070fd0 0x7fe3c80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.496+0000 7fe3cda15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c8071450 con 0x7fe3c80adf30 2023-10-14T16:41:23.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.496+0000 7fe3cda15640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c80cc600 con 0x7fe3c8070fd0 2023-10-14T16:41:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.496+0000 7fe3cda15640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80cc8e0 con 0x7fe3c80adb50 2023-10-14T16:41:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c8070fd0 0x7fe3c80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3c8070fd0 0x7fe3c80b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60492/0 (socket says 172.21.15.29:60492) 2023-10-14T16:41:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3c77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe3c80adf30 0x7fe3c8164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40318/0 (socket says 172.21.15.29:40318) 2023-10-14T16:41:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 -- 172.21.15.29:0/1605460521 learned_addr learned my addr 172.21.15.29:0/1605460521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3c6ffd640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 0x7fe3c806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 -- 172.21.15.29:0/1605460521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 msgr2=0x7fe3c806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 0x7fe3c806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 -- 172.21.15.29:0/1605460521 >> v1:172.21.15.29:6789/0 conn(0x7fe3c80adf30 legacy=0x7fe3c8164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3c67fc640 1 -- 172.21.15.29:0/1605460521 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2485536193 0 0) 0x7fe3c8071450 con 0x7fe3c80adf30 2023-10-14T16:41:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3bffff640 1 -- 172.21.15.29:0/1605460521 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c80ccbc0 con 0x7fe3c8070fd0 2023-10-14T16:41:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.497+0000 7fe3c6ffd640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 0x7fe3c806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.498+0000 7fe3bffff640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b0410 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009870 tx=0x7fe3b002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3279513eaba52ea3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3c67fc640 1 -- 172.21.15.29:0/1605460521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0030840 con 0x7fe3c8070fd0 2023-10-14T16:41:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3c67fc640 1 -- 172.21.15.29:0/1605460521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b00309a0 con 0x7fe3c8070fd0 2023-10-14T16:41:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3c67fc640 1 -- 172.21.15.29:0/1605460521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0030d30 con 0x7fe3c8070fd0 2023-10-14T16:41:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 -- 172.21.15.29:0/1605460521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c8070fd0 msgr2=0x7fe3c80b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b0410 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0009870 tx=0x7fe3b002b980 comp rx=0 tx=0).stop 2023-10-14T16:41:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 -- 172.21.15.29:0/1605460521 shutdown_connections 2023-10-14T16:41:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b0410 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 --2- 172.21.15.29:0/1605460521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adb50 0x7fe3c806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.499+0000 7fe3cda15640 1 -- 172.21.15.29:0/1605460521 >> 172.21.15.29:0/1605460521 conn(0x7fe3c80cb410 msgr2=0x7fe3c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.500+0000 7fe3cda15640 1 -- 172.21.15.29:0/1605460521 shutdown_connections 2023-10-14T16:41:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.500+0000 7fe3cda15640 1 -- 172.21.15.29:0/1605460521 wait complete. 2023-10-14T16:41:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.501+0000 7fe3cda15640 1 Processor -- start 2023-10-14T16:41:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.501+0000 7fe3cda15640 1 -- start start 2023-10-14T16:41:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 0x7fe3c80c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c81fade0 con 0x7fe3c80adb50 2023-10-14T16:41:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c815e7f0 con 0x7fe3c8070fd0 2023-10-14T16:41:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.502+0000 7fe3cda15640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c80b4630 con 0x7fe3c80adf30 2023-10-14T16:41:23.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60504/0 (socket says 172.21.15.29:60504) 2023-10-14T16:41:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 -- 172.21.15.29:0/3904342664 learned_addr learned my addr 172.21.15.29:0/3904342664 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 -- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 msgr2=0x7fe3c80c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3c6ffd640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3c77fe640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 0x7fe3c80c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 0x7fe3c80c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 -- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c8070fd0 msgr2=0x7fe3c80b3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3bffff640 1 -- 172.21.15.29:0/3904342664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00094d0 con 0x7fe3c80adb50 2023-10-14T16:41:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.503+0000 7fe3c77fe640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 0x7fe3c80c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3bffff640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0000c00 tx=0x7fe3b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0030810 con 0x7fe3c80adb50 2023-10-14T16:41:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b0039d90 con 0x7fe3c80adb50 2023-10-14T16:41:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3b0043670 con 0x7fe3c80adb50 2023-10-14T16:41:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c80b15b0 con 0x7fe3c80adb50 2023-10-14T16:41:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.504+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c80b1b70 con 0x7fe3c80adb50 2023-10-14T16:41:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.505+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c80c2150 con 0x7fe3c80adb50 2023-10-14T16:41:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.506+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe3b004c3b0 con 0x7fe3c80adb50 2023-10-14T16:41:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.507+0000 7fe3c4ff9640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 0x7fe3a806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.507+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe3b00abe90 con 0x7fe3c80adb50 2023-10-14T16:41:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.508+0000 7fe3c6ffd640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 0x7fe3a806d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.508+0000 7fe3c6ffd640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 0x7fe3a806d380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8008850 tx=0x7fe3b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.512+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3b0076ea0 con 0x7fe3c80adb50 2023-10-14T16:41:23.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.706+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c804e160 con 0x7fe3c80adb50 2023-10-14T16:41:23.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.709+0000 7fe3c4ff9640 1 -- 172.21.15.29:0/3904342664 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7fe3b0034090 con 0x7fe3c80adb50 2023-10-14T16:41:23.710 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:23.711 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 msgr2=0x7fe3a806d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 0x7fe3a806d380 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fe3b8008850 tx=0x7fe3b8008040 comp rx=0 tx=0).stop 2023-10-14T16:41:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 msgr2=0x7fe3c81fa8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe3b0000c00 tx=0x7fe3b0004730 comp rx=0 tx=0).stop 2023-10-14T16:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 shutdown_connections 2023-10-14T16:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe3a806aec0 0x7fe3a806d380 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3c80adf30 0x7fe3c80c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3c80adb50 0x7fe3c81fa8a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 --2- 172.21.15.29:0/3904342664 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3c8070fd0 0x7fe3c80b3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 >> 172.21.15.29:0/3904342664 conn(0x7fe3c80cb410 msgr2=0x7fe3c8160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:23.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 shutdown_connections 2023-10-14T16:41:23.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.712+0000 7fe3cda15640 1 -- 172.21.15.29:0/3904342664 wait complete. 2023-10-14T16:41:23.732 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:41:23.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:23.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.953+0000 7fde69f6c640 1 Processor -- start 2023-10-14T16:41:23.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.953+0000 7fde69f6c640 1 -- start start 2023-10-14T16:41:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde69f6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde640adb50 0x7fde6406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde69f6c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fde64070fd0 0x7fde640b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde69f6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde64071450 con 0x7fde640adf30 2023-10-14T16:41:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde69f6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640cc600 con 0x7fde640adb50 2023-10-14T16:41:23.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde69f6c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde640cc8e0 con 0x7fde64070fd0 2023-10-14T16:41:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.955+0000 7fde637fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde640adb50 0x7fde6406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fde640adb50 0x7fde6406f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60520/0 (socket says 172.21.15.29:60520) 2023-10-14T16:41:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 -- 172.21.15.29:0/3658058121 learned_addr learned my addr 172.21.15.29:0/3658058121 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde5bfff640 1 --2- 172.21.15.29:0/3658058121 >> v2:172.21.15.186:3300/0 conn(0x7fde64070fd0 0x7fde640b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 -- 172.21.15.29:0/3658058121 >> v2:172.21.15.186:3300/0 conn(0x7fde64070fd0 msgr2=0x7fde640b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 --2- 172.21.15.29:0/3658058121 >> v2:172.21.15.186:3300/0 conn(0x7fde64070fd0 0x7fde640b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 -- 172.21.15.29:0/3658058121 >> v1:172.21.15.29:6789/0 conn(0x7fde640adf30 legacy=0x7fde64164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.956+0000 7fde637fe640 1 -- 172.21.15.29:0/3658058121 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde640ccbc0 con 0x7fde640adb50 2023-10-14T16:41:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.957+0000 7fde637fe640 1 --2- 172.21.15.29:0/3658058121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adb50 0x7fde6406f550 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fde640cc2b0 tx=0x7fde5402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c7708ce43ebad0c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.957+0000 7fde62ffd640 1 -- 172.21.15.29:0/3658058121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54030870 con 0x7fde640adb50 2023-10-14T16:41:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.957+0000 7fde62ffd640 1 -- 172.21.15.29:0/3658058121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde540309d0 con 0x7fde640adb50 2023-10-14T16:41:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.957+0000 7fde62ffd640 1 -- 172.21.15.29:0/3658058121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde54030d60 con 0x7fde640adb50 2023-10-14T16:41:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 -- 172.21.15.29:0/3658058121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adb50 msgr2=0x7fde6406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 --2- 172.21.15.29:0/3658058121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adb50 0x7fde6406f550 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fde640cc2b0 tx=0x7fde5402b9b0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 -- 172.21.15.29:0/3658058121 shutdown_connections 2023-10-14T16:41:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 --2- 172.21.15.29:0/3658058121 >> v2:172.21.15.186:3300/0 conn(0x7fde64070fd0 0x7fde640b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 --2- 172.21.15.29:0/3658058121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adb50 0x7fde6406f550 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 -- 172.21.15.29:0/3658058121 >> 172.21.15.29:0/3658058121 conn(0x7fde640cb410 msgr2=0x7fde640cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 -- 172.21.15.29:0/3658058121 shutdown_connections 2023-10-14T16:41:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.958+0000 7fde69f6c640 1 -- 172.21.15.29:0/3658058121 wait complete. 2023-10-14T16:41:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.960+0000 7fde69f6c640 1 Processor -- start 2023-10-14T16:41:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.960+0000 7fde69f6c640 1 -- start start 2023-10-14T16:41:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde64070fd0 0x7fde640c1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde640adb50 0x7fde640c37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640c3d20 con 0x7fde640adf30 2023-10-14T16:41:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde640c3f50 con 0x7fde640adb50 2023-10-14T16:41:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde69f6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde640c4230 con 0x7fde64070fd0 2023-10-14T16:41:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.961+0000 7fde63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60532/0 (socket says 172.21.15.29:60532) 2023-10-14T16:41:23.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 -- 172.21.15.29:0/2623617752 learned_addr learned my addr 172.21.15.29:0/2623617752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:23.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde5bfff640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde640adb50 0x7fde640c37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde637fe640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde64070fd0 0x7fde640c1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 -- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde64070fd0 msgr2=0x7fde640c1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde64070fd0 0x7fde640c1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 -- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde640adb50 msgr2=0x7fde640c37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde640adb50 0x7fde640c37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 -- 172.21.15.29:0/2623617752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde540094d0 con 0x7fde640adf30 2023-10-14T16:41:23.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.962+0000 7fde63fff640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00c7a0 tx=0x7fde5c00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.963+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c003b70 con 0x7fde640adf30 2023-10-14T16:41:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.963+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5c003cf0 con 0x7fde640adf30 2023-10-14T16:41:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.963+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde5c003ea0 con 0x7fde640adf30 2023-10-14T16:41:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.963+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde640c2370 con 0x7fde640adf30 2023-10-14T16:41:23.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.963+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde640c2930 con 0x7fde640adf30 2023-10-14T16:41:23.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.964+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde640c2ac0 con 0x7fde640adf30 2023-10-14T16:41:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.965+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fde5c00aa90 con 0x7fde640adf30 2023-10-14T16:41:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.966+0000 7fde60ff9640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 0x7fde4406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.966+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fde5c089590 con 0x7fde640adf30 2023-10-14T16:41:23.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.966+0000 7fde637fe640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 0x7fde4406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.967+0000 7fde637fe640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 0x7fde4406d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fde54036000 tx=0x7fde5403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:23.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:23.970+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fde5c054520 con 0x7fde640adf30 2023-10-14T16:41:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.214+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde6406fa10 con 0x7fde640adf30 2023-10-14T16:41:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.215+0000 7fde60ff9640 1 -- 172.21.15.29:0/2623617752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7fde5c057b40 con 0x7fde640adf30 2023-10-14T16:41:24.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:24.216 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:41:24.217 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:41:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 msgr2=0x7fde4406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 0x7fde4406d330 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fde54036000 tx=0x7fde5403b040 comp rx=0 tx=0).stop 2023-10-14T16:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 msgr2=0x7fde64164ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fde5c00c7a0 tx=0x7fde5c00cc60 comp rx=0 tx=0).stop 2023-10-14T16:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 shutdown_connections 2023-10-14T16:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fde4406ae70 0x7fde4406d330 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fde640adf30 0x7fde64164ef0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fde640adb50 0x7fde640c37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 --2- 172.21.15.29:0/2623617752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fde64070fd0 0x7fde640c1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.217+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 >> 172.21.15.29:0/2623617752 conn(0x7fde640cb410 msgr2=0x7fde640ae560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:24.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.218+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 shutdown_connections 2023-10-14T16:41:24.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.218+0000 7fde69f6c640 1 -- 172.21.15.29:0/2623617752 wait complete. 2023-10-14T16:41:24.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.457+0000 7feeb3275640 1 Processor -- start 2023-10-14T16:41:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.457+0000 7feeb3275640 1 -- start start 2023-10-14T16:41:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.458+0000 7feeb3275640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7feeac0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb3275640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac160a90 con 0x7feeac070f20 2023-10-14T16:41:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb3275640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feeac0cd900 con 0x7feeac15c9f0 2023-10-14T16:41:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb3275640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0cdec0 con 0x7feeac15dde0 2023-10-14T16:41:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb2a74640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feeac070f20 0x7feeac0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40342/0 (socket says 172.21.15.29:40342) 2023-10-14T16:41:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb2a74640 1 -- 172.21.15.29:0/1766015673 learned_addr learned my addr 172.21.15.29:0/1766015673 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb1a72640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7feeac0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889383902 0 0) 0x7feeac160a90 con 0x7feeac070f20 2023-10-14T16:41:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.459+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee98003520 con 0x7feeac070f20 2023-10-14T16:41:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1312680053 0 0) 0x7fee98003520 con 0x7feeac070f20 2023-10-14T16:41:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 msgr2=0x7feeac0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7feeac0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 >> v1:172.21.15.186:6789/0 conn(0x7feeac15c9f0 legacy=0x7feeac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0cdbe0 con 0x7feeac070f20 2023-10-14T16:41:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7feea4001670 con 0x7feeac070f20 2023-10-14T16:41:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 >> v1:172.21.15.29:6789/0 conn(0x7feeac070f20 legacy=0x7feeac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7fee980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7fee98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800ba30 con 0x7fee98004d00 2023-10-14T16:41:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800bba0 con 0x7fee980039a0 2023-10-14T16:41:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.460+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee9800bd60 con 0x7feeac15dde0 2023-10-14T16:41:24.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7fee98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2273640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb1a72640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7fee980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 -- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 msgr2=0x7fee980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:24.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac15dde0 0x7fee980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 -- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee980039a0 msgr2=0x7fee98003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee980039a0 0x7fee98003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.461+0000 7feeb2a74640 1 -- 172.21.15.29:0/1766015673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee98003520 con 0x7fee98004d00 2023-10-14T16:41:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.462+0000 7feeb2a74640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7fee98005180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feea4001eb0 tx=0x7fee980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b84a7f6364dc9c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.462+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea4005ad0 con 0x7fee98004d00 2023-10-14T16:41:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.462+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea4005c30 con 0x7fee98004d00 2023-10-14T16:41:24.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.462+0000 7feeb1271640 1 -- 172.21.15.29:0/1766015673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea402c7c0 con 0x7fee98004d00 2023-10-14T16:41:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 -- 172.21.15.29:0/1766015673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 msgr2=0x7fee98005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 --2- 172.21.15.29:0/1766015673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7fee98005180 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7feea4001eb0 tx=0x7fee980057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb2a74640 1 -- 172.21.15.29:0/1766015673 reap_dead start 2023-10-14T16:41:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 -- 172.21.15.29:0/1766015673 shutdown_connections 2023-10-14T16:41:24.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 -- 172.21.15.29:0/1766015673 >> 172.21.15.29:0/1766015673 conn(0x7feeac0ce6a0 msgr2=0x7feeac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:24.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 -- 172.21.15.29:0/1766015673 shutdown_connections 2023-10-14T16:41:24.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.463+0000 7feeb3275640 1 -- 172.21.15.29:0/1766015673 wait complete. 2023-10-14T16:41:24.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.465+0000 7feeb3275640 1 Processor -- start 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.465+0000 7feeb3275640 1 -- start start 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac157330 0x7feeac15f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feeac15f8b0 0x7feeac069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c5e40 con 0x7fee98004d00 2023-10-14T16:41:24.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feeac15fd30 con 0x7feeac15f8b0 2023-10-14T16:41:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb3275640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feeac0c88c0 con 0x7feeac157330 2023-10-14T16:41:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.466+0000 7feeb2273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60540/0 (socket says 172.21.15.29:60540) 2023-10-14T16:41:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 -- 172.21.15.29:0/3439049346 learned_addr learned my addr 172.21.15.29:0/3439049346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:24.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb1a72640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac157330 0x7feeac15f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 -- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac157330 msgr2=0x7feeac15f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac157330 0x7feeac15f370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 -- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feeac15f8b0 msgr2=0x7feeac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feeac15f8b0 0x7feeac069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 -- 172.21.15.29:0/3439049346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feeac0c6020 con 0x7fee98004d00 2023-10-14T16:41:24.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.467+0000 7feeb2273640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feea800e8c0 tx=0x7feea800ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.468+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea8003b80 con 0x7fee98004d00 2023-10-14T16:41:24.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.468+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feea8003ce0 con 0x7fee98004d00 2023-10-14T16:41:24.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.468+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feea80052f0 con 0x7fee98004d00 2023-10-14T16:41:24.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.468+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feeac0c6300 con 0x7fee98004d00 2023-10-14T16:41:24.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.468+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feeac0c85d0 con 0x7fee98004d00 2023-10-14T16:41:24.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.469+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeac04e9b0 con 0x7fee98004d00 2023-10-14T16:41:24.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.470+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7feea8005450 con 0x7fee98004d00 2023-10-14T16:41:24.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.471+0000 7feea2ffd640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 0x7fee9006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.471+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feea80897b0 con 0x7fee98004d00 2023-10-14T16:41:24.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.472+0000 7feeb1a72640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 0x7fee9006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.472+0000 7feeb1a72640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 0x7fee9006d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feeac0ca410 tx=0x7fee9c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.476+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feea80547d0 con 0x7fee98004d00 2023-10-14T16:41:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.712+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feeac0adac0 con 0x7fee98004d00 2023-10-14T16:41:24.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.713+0000 7feea2ffd640 1 -- 172.21.15.29:0/3439049346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+4913 (secure 0 0 0) 0x7feea801a310 con 0x7fee98004d00 2023-10-14T16:41:24.714 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:24.715 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4214,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1190964963","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1190964963},{"type":"v1","addr":"172.21.15.186:6834","nonce":1190964963}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":4269,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3814623691","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3814623691},{"type":"v1","addr":"172.21.15.29:6835","nonce":3814623691}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":4270,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1187510407","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1187510407},{"type":"v1","addr":"172.21.15.186:6835","nonce":1187510407}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":8,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:40:25.223449+0000","modified":"2023-10-14T16:40:28.288537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":4272},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4272":{"gid":4272,"name":"c","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3245669368","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3245669368},{"type":"v1","addr":"172.21.15.29:6837","nonce":3245669368}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2023-10-14T16:41:24.715 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 8 2023-10-14T16:41:24.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.715+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 msgr2=0x7fee9006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.715+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 0x7fee9006d330 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feeac0ca410 tx=0x7fee9c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:24.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 msgr2=0x7feeac0c8090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feea800e8c0 tx=0x7feea800ed80 comp rx=0 tx=0).stop 2023-10-14T16:41:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 shutdown_connections 2023-10-14T16:41:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fee9006ae70 0x7fee9006d330 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feeac15f8b0 0x7feeac069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feeac157330 0x7feeac15f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 --2- 172.21.15.29:0/3439049346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee98004d00 0x7feeac0c8090 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 >> 172.21.15.29:0/3439049346 conn(0x7feeac0ce6a0 msgr2=0x7feeac15a7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 shutdown_connections 2023-10-14T16:41:24.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.716+0000 7feeb3275640 1 -- 172.21.15.29:0/3439049346 wait complete. 2023-10-14T16:41:24.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.965+0000 7fd5d4d68640 1 Processor -- start 2023-10-14T16:41:24.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.965+0000 7fd5d4d68640 1 -- start start 2023-10-14T16:41:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.966+0000 7fd5d4d68640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd5d0159b20 0x7fd5d006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5d4d68640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5d0159f00 0x7fd5d00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5d4d68640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5d006f7f0 con 0x7fd5d006c120 2023-10-14T16:41:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5d4d68640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d006c5e0 con 0x7fd5d0159f00 2023-10-14T16:41:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5d4d68640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5d006c8c0 con 0x7fd5d0159b20 2023-10-14T16:41:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5d0159f00 0x7fd5d00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5d0159f00 0x7fd5d00b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60554/0 (socket says 172.21.15.29:60554) 2023-10-14T16:41:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5c7fff640 1 -- 172.21.15.29:0/2201164226 learned_addr learned my addr 172.21.15.29:0/2201164226 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:24.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.967+0000 7fd5ce59c640 1 --2- 172.21.15.29:0/2201164226 >> v2:172.21.15.186:3300/0 conn(0x7fd5d0159b20 0x7fd5d006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.968+0000 7fd5c7fff640 1 -- 172.21.15.29:0/2201164226 >> v2:172.21.15.186:3300/0 conn(0x7fd5d0159b20 msgr2=0x7fd5d006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.968+0000 7fd5c7fff640 1 --2- 172.21.15.29:0/2201164226 >> v2:172.21.15.186:3300/0 conn(0x7fd5d0159b20 0x7fd5d006bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.968+0000 7fd5c7fff640 1 -- 172.21.15.29:0/2201164226 >> v1:172.21.15.186:6789/0 conn(0x7fd5d006c120 legacy=0x7fd5d00cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.968+0000 7fd5c7fff640 1 -- 172.21.15.29:0/2201164226 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5d006cba0 con 0x7fd5d0159f00 2023-10-14T16:41:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.969+0000 7fd5c7fff640 1 --2- 172.21.15.29:0/2201164226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159f00 0x7fd5d00b0410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00c9340 tx=0x7fd5b802e0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a38e94433d2de72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.969+0000 7fd5cdd9b640 1 -- 172.21.15.29:0/2201164226 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b802ee90 con 0x7fd5d0159f00 2023-10-14T16:41:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.969+0000 7fd5cdd9b640 1 -- 172.21.15.29:0/2201164226 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8036440 con 0x7fd5d0159f00 2023-10-14T16:41:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.969+0000 7fd5cdd9b640 1 -- 172.21.15.29:0/2201164226 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80367d0 con 0x7fd5d0159f00 2023-10-14T16:41:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 -- 172.21.15.29:0/2201164226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159f00 msgr2=0x7fd5d00b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/2201164226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159f00 0x7fd5d00b0410 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd5d00c9340 tx=0x7fd5b802e0e0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 -- 172.21.15.29:0/2201164226 shutdown_connections 2023-10-14T16:41:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/2201164226 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159f00 0x7fd5d00b0410 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/2201164226 >> v2:172.21.15.186:3300/0 conn(0x7fd5d0159b20 0x7fd5d006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 -- 172.21.15.29:0/2201164226 >> 172.21.15.29:0/2201164226 conn(0x7fd5d00cb410 msgr2=0x7fd5d00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:24.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 -- 172.21.15.29:0/2201164226 shutdown_connections 2023-10-14T16:41:24.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.970+0000 7fd5d4d68640 1 -- 172.21.15.29:0/2201164226 wait complete. 2023-10-14T16:41:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.972+0000 7fd5d4d68640 1 Processor -- start 2023-10-14T16:41:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.972+0000 7fd5d4d68640 1 -- start start 2023-10-14T16:41:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 0x7fd5d00c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5d0159f00 0x7fd5d00afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00b3910 con 0x7fd5d0159b20 2023-10-14T16:41:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00c4790 con 0x7fd5d006c120 2023-10-14T16:41:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5d4d68640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5d00ad5a0 con 0x7fd5d0159f00 2023-10-14T16:41:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60564/0 (socket says 172.21.15.29:60564) 2023-10-14T16:41:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5c7fff640 1 -- 172.21.15.29:0/3462680867 learned_addr learned my addr 172.21.15.29:0/3462680867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.973+0000 7fd5ce59c640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 0x7fd5d00c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 -- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5d0159f00 msgr2=0x7fd5d00afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5d0159f00 0x7fd5d00afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 -- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 msgr2=0x7fd5d00c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 0x7fd5d00c5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 -- 172.21.15.29:0/3462680867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b800b770 con 0x7fd5d0159b20 2023-10-14T16:41:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5ce59c640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 0x7fd5d00c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:24.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c7fff640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802e120 tx=0x7fd5b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b80348d0 con 0x7fd5d0159b20 2023-10-14T16:41:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5b8034a30 con 0x7fd5d0159b20 2023-10-14T16:41:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5b8034de0 con 0x7fd5d0159b20 2023-10-14T16:41:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5d00c5fa0 con 0x7fd5d0159b20 2023-10-14T16:41:24.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.974+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5d00ad840 con 0x7fd5d0159b20 2023-10-14T16:41:24.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.976+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5d004e160 con 0x7fd5d0159b20 2023-10-14T16:41:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.977+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fd5b80403e0 con 0x7fd5d0159b20 2023-10-14T16:41:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.978+0000 7fd5c6ffd640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 0x7fd5b006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.978+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd5b80ab9c0 con 0x7fd5d0159b20 2023-10-14T16:41:24.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.978+0000 7fd5ce59c640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 0x7fd5b006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.979+0000 7fd5ce59c640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 0x7fd5b006d330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0006fd0 tx=0x7fd5c0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:24.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:24.983+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5b80769f0 con 0x7fd5d0159b20 2023-10-14T16:41:25.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.184+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5d00b0720 con 0x7fd5d0159b20 2023-10-14T16:41:25.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.186+0000 7fd5c6ffd640 1 -- 172.21.15.29:0/3462680867 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+16742 (secure 0 0 0) 0x7fd5b803b090 con 0x7fd5d0159b20 2023-10-14T16:41:25.188 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:25.188 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":22,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:06.987506+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:40:23.612990+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2023-10-14T16:40:24.238212+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.c","mappings":[{"from":5,"to":2}]},{"pgid":"3.15","mappings":[{"from":7,"to":1}]},{"pgid":"3.25","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 msgr2=0x7fd5b006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 0x7fd5b006d330 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd5c0006fd0 tx=0x7fd5c0008040 comp rx=0 tx=0).stop 2023-10-14T16:41:25.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 msgr2=0x7fd5d01542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fd5b802e120 tx=0x7fd5b80046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 shutdown_connections 2023-10-14T16:41:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fd5b006ae70 0x7fd5b006d330 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5d0159f00 0x7fd5d00afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5d0159b20 0x7fd5d01542a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 --2- 172.21.15.29:0/3462680867 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5d006c120 0x7fd5d00c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 >> 172.21.15.29:0/3462680867 conn(0x7fd5d00cb410 msgr2=0x7fd5d00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 shutdown_connections 2023-10-14T16:41:25.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.189+0000 7fd5d4d68640 1 -- 172.21.15.29:0/3462680867 wait complete. 2023-10-14T16:41:25.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:41:25.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.437+0000 7f3bdebb3640 1 Processor -- start 2023-10-14T16:41:25.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.437+0000 7f3bdebb3640 1 -- start start 2023-10-14T16:41:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.438+0000 7f3bdebb3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3bd806c040 0x7f3bd806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.438+0000 7f3bdebb3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 0x7f3bd815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.438+0000 7f3bdebb3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bd806c960 con 0x7f3bd80ad970 2023-10-14T16:41:25.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.438+0000 7f3bdebb3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bd815b9d0 con 0x7f3bd806c040 2023-10-14T16:41:25.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.438+0000 7f3bdebb3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd815bcb0 con 0x7f3bd80cda80 2023-10-14T16:41:25.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bdd3b0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3bd80ad970 0x7f3bd815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40350/0 (socket says 172.21.15.29:40350) 2023-10-14T16:41:25.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bdd3b0640 1 -- 172.21.15.29:0/912219646 learned_addr learned my addr 172.21.15.29:0/912219646 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:25.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/912219646 >> v2:172.21.15.186:3300/0 conn(0x7f3bd806c040 0x7f3bd806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bde3b2640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 0x7f3bd815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2830854837 0 0) 0x7f3bd806c960 con 0x7f3bd80ad970 2023-10-14T16:41:25.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.439+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bc4003520 con 0x7f3bd80ad970 2023-10-14T16:41:25.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2906183553 0 0) 0x7f3bc4003520 con 0x7f3bd80ad970 2023-10-14T16:41:25.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bddbb1640 1 -- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 msgr2=0x7f3bd815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:25.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 0x7f3bd815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bddbb1640 1 -- 172.21.15.29:0/912219646 >> v1:172.21.15.29:6789/0 conn(0x7f3bd80ad970 legacy=0x7f3bd815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bddbb1640 1 -- 172.21.15.29:0/912219646 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd815bf90 con 0x7f3bd806c040 2023-10-14T16:41:25.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.440+0000 7f3bde3b2640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 0x7f3bd815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:41:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.441+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd806c040 0x7f3bd806c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009d90 tx=0x7f3bd402bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=734114fb9e8f885 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8001670 con 0x7f3bd806c040 2023-10-14T16:41:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bd402c8a0 con 0x7f3bd806c040 2023-10-14T16:41:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdcbaf640 1 -- 172.21.15.29:0/912219646 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bd402cc10 con 0x7f3bd806c040 2023-10-14T16:41:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 -- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd806c040 msgr2=0x7f3bd806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd806c040 0x7f3bd806c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4009d90 tx=0x7f3bd402bd70 comp rx=0 tx=0).stop 2023-10-14T16:41:25.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 -- 172.21.15.29:0/912219646 shutdown_connections 2023-10-14T16:41:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd80cda80 0x7f3bd815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/912219646 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd806c040 0x7f3bd806c420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 -- 172.21.15.29:0/912219646 >> 172.21.15.29:0/912219646 conn(0x7f3bd80ce6a0 msgr2=0x7f3bd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.442+0000 7f3bdebb3640 1 -- 172.21.15.29:0/912219646 shutdown_connections 2023-10-14T16:41:25.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.443+0000 7f3bdebb3640 1 -- 172.21.15.29:0/912219646 wait complete. 2023-10-14T16:41:25.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.444+0000 7f3bdebb3640 1 Processor -- start 2023-10-14T16:41:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.444+0000 7f3bdebb3640 1 -- start start 2023-10-14T16:41:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 0x7f3bd80ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd80cda80 0x7f3bd80b29d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd80c8530 con 0x7f3bd80ad970 2023-10-14T16:41:25.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd80b0c90 con 0x7f3bd80cda80 2023-10-14T16:41:25.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.445+0000 7f3bdebb3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd80ae550 con 0x7f3bd806c040 2023-10-14T16:41:25.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60570/0 (socket says 172.21.15.29:60570) 2023-10-14T16:41:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 -- 172.21.15.29:0/3662748048 learned_addr learned my addr 172.21.15.29:0/3662748048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bde3b2640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd80cda80 0x7f3bd80b29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 0x7f3bd80ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 -- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 msgr2=0x7f3bd80ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 0x7f3bd80ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 -- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd80cda80 msgr2=0x7f3bd80b29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd80cda80 0x7f3bd80b29d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bdd3b0640 1 -- 172.21.15.29:0/3662748048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd0009400 con 0x7f3bd80ad970 2023-10-14T16:41:25.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.446+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 0x7f3bd80ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bdd3b0640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80017a0 tx=0x7f3bc800cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc8005ad0 con 0x7f3bd80ad970 2023-10-14T16:41:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bc8005c30 con 0x7f3bd80ad970 2023-10-14T16:41:25.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd4009a70 con 0x7f3bd80ad970 2023-10-14T16:41:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc80072b0 con 0x7f3bd80ad970 2023-10-14T16:41:25.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.447+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd80c5810 con 0x7f3bd80ad970 2023-10-14T16:41:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.448+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd806b330 con 0x7f3bd80ad970 2023-10-14T16:41:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.449+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3bc8007410 con 0x7f3bd80ad970 2023-10-14T16:41:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.450+0000 7f3bce7fc640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 0x7f3bbc06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.450+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3bc80893f0 con 0x7f3bd80ad970 2023-10-14T16:41:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.451+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 0x7f3bbc06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.451+0000 7f3bddbb1640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 0x7f3bbc06d360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.455+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3bc8054420 con 0x7f3bd80ad970 2023-10-14T16:41:25.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.694+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3bd804e9b0 con 0x7f3bd80ad970 2023-10-14T16:41:25.696 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:41:25.695+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:41:25.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.700+0000 7f3bce7fc640 1 -- 172.21.15.29:0/3662748048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v9) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3bc8057a40 con 0x7f3bd80ad970 2023-10-14T16:41:25.701 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:41:25.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 msgr2=0x7f3bbc06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 0x7f3bbc06d360 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f3bd4036000 tx=0x7f3bd4030000 comp rx=0 tx=0).stop 2023-10-14T16:41:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 msgr2=0x7f3bd80b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f3bc80017a0 tx=0x7f3bc800cbe0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 shutdown_connections 2023-10-14T16:41:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f3bbc06aea0 0x7f3bbc06d360 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bd80cda80 0x7f3bd80b29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3bd80ad970 0x7f3bd80b0750 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 --2- 172.21.15.29:0/3662748048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bd806c040 0x7f3bd80ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.703+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 >> 172.21.15.29:0/3662748048 conn(0x7f3bd80ce6a0 msgr2=0x7f3bd815d870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.704+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 shutdown_connections 2023-10-14T16:41:25.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.704+0000 7f3bdebb3640 1 -- 172.21.15.29:0/3662748048 wait complete. 2023-10-14T16:41:25.726 INFO:tasks.ceph.mds.c.smithi029.stderr: -7458> 2023-10-14T16:40:31.490+0000 7f70ddf0c640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:41:25.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:41:25.797 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:41:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.935+0000 7f2f8c90b640 1 Processor -- start 2023-10-14T16:41:25.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.936+0000 7f2f8c90b640 1 -- start start 2023-10-14T16:41:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f8c90b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f880ad970 0x7f2f880cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f8c90b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 0x7f2f880cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f8c90b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f8815b9d0 con 0x7f2f8806c040 2023-10-14T16:41:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f8c90b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f8815bcb0 con 0x7f2f880ad970 2023-10-14T16:41:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f8c90b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f8815bf90 con 0x7f2f880ccfb0 2023-10-14T16:41:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f880ad970 0x7f2f880cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f880ad970 0x7f2f880cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60580/0 (socket says 172.21.15.29:60580) 2023-10-14T16:41:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.937+0000 7f2f7e59c640 1 -- 172.21.15.29:0/2911888170 learned_addr learned my addr 172.21.15.29:0/2911888170 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f8759e640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 0x7f2f880cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f7e59c640 1 -- 172.21.15.29:0/2911888170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 msgr2=0x7f2f880cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f7e59c640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 0x7f2f880cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f7e59c640 1 -- 172.21.15.29:0/2911888170 >> v1:172.21.15.186:6789/0 conn(0x7f2f8806c040 legacy=0x7f2f8806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f7e59c640 1 -- 172.21.15.29:0/2911888170 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f880cc270 con 0x7f2f880ad970 2023-10-14T16:41:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.938+0000 7f2f8759e640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 0x7f2f880cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.939+0000 7f2f7e59c640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ad970 0x7f2f880cca70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbf65011602fee25 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.939+0000 7f2f8659c640 1 -- 172.21.15.29:0/2911888170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70031640 con 0x7f2f880ad970 2023-10-14T16:41:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.939+0000 7f2f8659c640 1 -- 172.21.15.29:0/2911888170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f700317a0 con 0x7f2f880ad970 2023-10-14T16:41:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.939+0000 7f2f8659c640 1 -- 172.21.15.29:0/2911888170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f70031b30 con 0x7f2f880ad970 2023-10-14T16:41:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 -- 172.21.15.29:0/2911888170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ad970 msgr2=0x7f2f880cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ad970 0x7f2f880cca70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2f70009870 tx=0x7f2f7002b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 -- 172.21.15.29:0/2911888170 shutdown_connections 2023-10-14T16:41:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ccfb0 0x7f2f880cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/2911888170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ad970 0x7f2f880cca70 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 -- 172.21.15.29:0/2911888170 >> 172.21.15.29:0/2911888170 conn(0x7f2f880ce6a0 msgr2=0x7f2f880ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 -- 172.21.15.29:0/2911888170 shutdown_connections 2023-10-14T16:41:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.940+0000 7f2f8c90b640 1 -- 172.21.15.29:0/2911888170 wait complete. 2023-10-14T16:41:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.942+0000 7f2f8c90b640 1 Processor -- start 2023-10-14T16:41:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.942+0000 7f2f8c90b640 1 -- start start 2023-10-14T16:41:25.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8806c040 0x7f2f88153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ad970 0x7f2f880c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880c6bd0 con 0x7f2f880ccfb0 2023-10-14T16:41:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880c6eb0 con 0x7f2f8806c040 2023-10-14T16:41:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8c90b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f880c7190 con 0x7f2f880ad970 2023-10-14T16:41:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60590/0 (socket says 172.21.15.29:60590) 2023-10-14T16:41:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.943+0000 7f2f8759e640 1 -- 172.21.15.29:0/3338640981 learned_addr learned my addr 172.21.15.29:0/3338640981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f7e59c640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ad970 0x7f2f880c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f86d9d640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8806c040 0x7f2f88153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 -- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ad970 msgr2=0x7f2f880c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ad970 0x7f2f880c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 -- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8806c040 msgr2=0x7f2f88153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8806c040 0x7f2f88153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 -- 172.21.15.29:0/3338640981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f700094d0 con 0x7f2f880ccfb0 2023-10-14T16:41:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f8759e640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000cd50 tx=0x7f2f8000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.944+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000f680 con 0x7f2f880ccfb0 2023-10-14T16:41:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.945+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f8000f800 con 0x7f2f880ccfb0 2023-10-14T16:41:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.945+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f880c7470 con 0x7f2f880ccfb0 2023-10-14T16:41:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.945+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f8000fbb0 con 0x7f2f880ccfb0 2023-10-14T16:41:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.945+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f880ca070 con 0x7f2f880ccfb0 2023-10-14T16:41:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.946+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f8806b330 con 0x7f2f880ccfb0 2023-10-14T16:41:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.947+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2f8000b040 con 0x7f2f880ccfb0 2023-10-14T16:41:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.948+0000 7f2f7ffff640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 0x7f2f6806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.948+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f2f8008a190 con 0x7f2f880ccfb0 2023-10-14T16:41:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.948+0000 7f2f86d9d640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 0x7f2f6806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.949+0000 7f2f86d9d640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 0x7f2f6806d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f78004ea0 tx=0x7f2f78009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:25.952+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f800550d0 con 0x7f2f880ccfb0 2023-10-14T16:41:26.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.196+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2f8804e9b0 con 0x7f2f880ccfb0 2023-10-14T16:41:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.703+0000 7f2f7ffff640 1 -- 172.21.15.29:0/3338640981 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7f2f800586f0 con 0x7f2f880ccfb0 2023-10-14T16:41:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 msgr2=0x7f2f6806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 0x7f2f6806d330 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2f78004ea0 tx=0x7f2f78009040 comp rx=0 tx=0).stop 2023-10-14T16:41:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 msgr2=0x7f2f880ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f2f8000cd50 tx=0x7f2f8000a650 comp rx=0 tx=0).stop 2023-10-14T16:41:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 shutdown_connections 2023-10-14T16:41:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2f6806ae70 0x7f2f6806d330 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f880ccfb0 0x7f2f880ac890 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f880ad970 0x7f2f880c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 --2- 172.21.15.29:0/3338640981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f8806c040 0x7f2f88153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 >> 172.21.15.29:0/3338640981 conn(0x7f2f880ce6a0 msgr2=0x7f2f8815dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 shutdown_connections 2023-10-14T16:41:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.708+0000 7f2f8c90b640 1 -- 172.21.15.29:0/3338640981 wait complete. 2023-10-14T16:41:26.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:41:26.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.958+0000 7f0f7d76c640 1 Processor -- start 2023-10-14T16:41:26.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.959+0000 7f0f7d76c640 1 -- start start 2023-10-14T16:41:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f7d76c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c8fa0 0x7f0f78165aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f7d76c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0f780b0360 0x7f0f780b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f7d76c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f78165610 con 0x7f0f780c1d30 2023-10-14T16:41:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f7d76c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f780b0d10 con 0x7f0f780b0360 2023-10-14T16:41:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f7d76c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780ce370 con 0x7f0f780c8fa0 2023-10-14T16:41:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f76ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0f780c1d30 0x7f0f780cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40364/0 (socket says 172.21.15.29:40364) 2023-10-14T16:41:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f76ffd640 1 -- 172.21.15.29:0/3648832563 learned_addr learned my addr 172.21.15.29:0/3648832563 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.961+0000 7f0f777fe640 1 --2- 172.21.15.29:0/3648832563 >> v2:172.21.15.29:3300/0 conn(0x7f0f780b0360 0x7f0f780b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f6ffff640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c8fa0 0x7f0f78165aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f767fc640 1 -- 172.21.15.29:0/3648832563 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623289980 0 0) 0x7f0f78165610 con 0x7f0f780c1d30 2023-10-14T16:41:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f777fe640 1 -- 172.21.15.29:0/3648832563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c8fa0 msgr2=0x7f0f78165aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f777fe640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c8fa0 0x7f0f78165aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f777fe640 1 -- 172.21.15.29:0/3648832563 >> v1:172.21.15.29:6789/0 conn(0x7f0f780c1d30 legacy=0x7f0f780cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.962+0000 7f0f777fe640 1 -- 172.21.15.29:0/3648832563 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f780ce650 con 0x7f0f780b0360 2023-10-14T16:41:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.963+0000 7f0f777fe640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780b0360 0x7f0f780b07d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009d90 tx=0x7f0f7002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98018946b9b36afd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:26.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.963+0000 7f0f767fc640 1 -- 172.21.15.29:0/3648832563 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f7002cc50 con 0x7f0f780b0360 2023-10-14T16:41:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.963+0000 7f0f767fc640 1 -- 172.21.15.29:0/3648832563 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f7002cdb0 con 0x7f0f780b0360 2023-10-14T16:41:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.963+0000 7f0f767fc640 1 -- 172.21.15.29:0/3648832563 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70034660 con 0x7f0f780b0360 2023-10-14T16:41:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 -- 172.21.15.29:0/3648832563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780b0360 msgr2=0x7f0f780b07d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780b0360 0x7f0f780b07d0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009d90 tx=0x7f0f7002bea0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 -- 172.21.15.29:0/3648832563 shutdown_connections 2023-10-14T16:41:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780b0360 0x7f0f780b07d0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/3648832563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c8fa0 0x7f0f78165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 -- 172.21.15.29:0/3648832563 >> 172.21.15.29:0/3648832563 conn(0x7f0f780cb360 msgr2=0x7f0f780cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 -- 172.21.15.29:0/3648832563 shutdown_connections 2023-10-14T16:41:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.964+0000 7f0f7d76c640 1 -- 172.21.15.29:0/3648832563 wait complete. 2023-10-14T16:41:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.966+0000 7f0f7d76c640 1 Processor -- start 2023-10-14T16:41:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.966+0000 7f0f7d76c640 1 -- start start 2023-10-14T16:41:26.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f780b0360 0x7f0f78157450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c1d30 0x7f0f781637b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78163cf0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f78163f80 con 0x7f0f780b0360 2023-10-14T16:41:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f7d76c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f780c2410 con 0x7f0f780c1d30 2023-10-14T16:41:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.967+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60610/0 (socket says 172.21.15.29:60610) 2023-10-14T16:41:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 -- 172.21.15.29:0/2864915020 learned_addr learned my addr 172.21.15.29:0/2864915020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 -- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c1d30 msgr2=0x7f0f781637b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f6ffff640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c1d30 0x7f0f781637b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c1d30 0x7f0f781637b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 -- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f780b0360 msgr2=0x7f0f78157450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f780b0360 0x7f0f78157450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 -- 172.21.15.29:0/2864915020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70009a70 con 0x7f0f780c8fa0 2023-10-14T16:41:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.968+0000 7f0f777fe640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009d90 tx=0x7f0f70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.969+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70040650 con 0x7f0f780c8fa0 2023-10-14T16:41:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.969+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f700407d0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.969+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f70040b80 con 0x7f0f780c8fa0 2023-10-14T16:41:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.969+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f780c26f0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.969+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f780c2cb0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.970+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7804e160 con 0x7f0f780c8fa0 2023-10-14T16:41:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.971+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0f70040ce0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.972+0000 7f0f6f7fe640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 0x7f0f5806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.972+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 5911+0+0 (secure 0 0 0) 0x7f0f700ac5a0 con 0x7f0f780c8fa0 2023-10-14T16:41:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.972+0000 7f0f76ffd640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 0x7f0f5806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.973+0000 7f0f76ffd640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 0x7f0f5806d330 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f68007fd0 tx=0x7f0f68007960 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:26.977+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f700774e0 con 0x7f0f780c8fa0 2023-10-14T16:41:27.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:27.176+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0f780c14e0 con 0x7f0f780c8fa0 2023-10-14T16:41:28.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.142+0000 7f0f6f7fe640 1 -- 172.21.15.29:0/2864915020 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) v1 ==== 181+0+0 (secure 0 0 0) 0x7f0f7007ab00 con 0x7f0f780c8fa0 2023-10-14T16:41:28.143 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:41:28.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.145+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 msgr2=0x7f0f5806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.145+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 0x7f0f5806d330 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f0f68007fd0 tx=0x7f0f68007960 comp rx=0 tx=0).stop 2023-10-14T16:41:28.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.145+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 msgr2=0x7f0f780c31d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.145+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009d90 tx=0x7f0f70004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:28.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 shutdown_connections 2023-10-14T16:41:28.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f0f5806ae70 0x7f0f5806d330 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f780c8fa0 0x7f0f780c31d0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f780c1d30 0x7f0f781637b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 --2- 172.21.15.29:0/2864915020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f780b0360 0x7f0f78157450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.146+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 >> 172.21.15.29:0/2864915020 conn(0x7f0f780cb360 msgr2=0x7f0f780cc490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:28.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.147+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 shutdown_connections 2023-10-14T16:41:28.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.147+0000 7f0f7d76c640 1 -- 172.21.15.29:0/2864915020 wait complete. 2023-10-14T16:41:28.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:41:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.404+0000 7fdc748b4640 1 Processor -- start 2023-10-14T16:41:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.404+0000 7fdc748b4640 1 -- start start 2023-10-14T16:41:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.405+0000 7fdc748b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdc7015ac10 0x7fdc7006c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.405+0000 7fdc748b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc700b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc748b4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc700b0d00 con 0x7fdc700ca510 2023-10-14T16:41:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc748b4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc7015f800 con 0x7fdc7015ac10 2023-10-14T16:41:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc748b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7015b310 con 0x7fdc700b0360 2023-10-14T16:41:28.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc6ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdc700ca510 0x7fdc700ca940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40378/0 (socket says 172.21.15.29:40378) 2023-10-14T16:41:28.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc6ed9d640 1 -- 172.21.15.29:0/69486631 learned_addr learned my addr 172.21.15.29:0/69486631 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/69486631 >> v2:172.21.15.186:3300/0 conn(0x7fdc7015ac10 0x7fdc7006c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.406+0000 7fdc67fff640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc700b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149624438 0 0) 0x7fdc700b0d00 con 0x7fdc700ca510 2023-10-14T16:41:28.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc5c003520 con 0x7fdc700ca510 2023-10-14T16:41:28.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1701238893 0 0) 0x7fdc5c003520 con 0x7fdc700ca510 2023-10-14T16:41:28.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 msgr2=0x7fdc700b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:28.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc700b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 >> v2:172.21.15.186:3300/0 conn(0x7fdc7015ac10 msgr2=0x7fdc7006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:28.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 --2- 172.21.15.29:0/69486631 >> v2:172.21.15.186:3300/0 conn(0x7fdc7015ac10 0x7fdc7006c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7006f640 con 0x7fdc700ca510 2023-10-14T16:41:28.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdc680019f0 con 0x7fdc700ca510 2023-10-14T16:41:28.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 >> v1:172.21.15.29:6789/0 conn(0x7fdc700ca510 legacy=0x7fdc700ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0039e0 0x7fdc5c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc5c0047c0 0x7fdc5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7015f800 con 0x7fdc5c0047c0 2023-10-14T16:41:28.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7015b310 con 0x7fdc5c0039e0 2023-10-14T16:41:28.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.407+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5c00ba60 con 0x7fdc700b0360 2023-10-14T16:41:28.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc5c0047c0 0x7fdc5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0039e0 0x7fdc5c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 -- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 msgr2=0x7fdc5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc6ed9d640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc700b0360 0x7fdc5c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 -- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0039e0 msgr2=0x7fdc5c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:28.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0039e0 0x7fdc5c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc67fff640 1 -- 172.21.15.29:0/69486631 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c003520 con 0x7fdc5c0047c0 2023-10-14T16:41:28.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.408+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0039e0 0x7fdc5c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:28.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.409+0000 7fdc67fff640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc5c0047c0 0x7fdc5c004c40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009a90 tx=0x7fdc5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdc412ea305f0c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:28.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5802c660 con 0x7fdc5c0047c0 2023-10-14T16:41:28.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc5802c7c0 con 0x7fdc5c0047c0 2023-10-14T16:41:28.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc6dd9b640 1 -- 172.21.15.29:0/69486631 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc5802cb50 con 0x7fdc5c0047c0 2023-10-14T16:41:28.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 -- 172.21.15.29:0/69486631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc5c0047c0 msgr2=0x7fdc5c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 --2- 172.21.15.29:0/69486631 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc5c0047c0 0x7fdc5c004c40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fdc58009a90 tx=0x7fdc5c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc6ed9d640 1 -- 172.21.15.29:0/69486631 reap_dead start 2023-10-14T16:41:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 -- 172.21.15.29:0/69486631 shutdown_connections 2023-10-14T16:41:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 -- 172.21.15.29:0/69486631 >> 172.21.15.29:0/69486631 conn(0x7fdc700cb360 msgr2=0x7fdc700cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:28.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 -- 172.21.15.29:0/69486631 shutdown_connections 2023-10-14T16:41:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.410+0000 7fdc748b4640 1 -- 172.21.15.29:0/69486631 wait complete. 2023-10-14T16:41:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.412+0000 7fdc748b4640 1 Processor -- start 2023-10-14T16:41:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.412+0000 7fdc748b4640 1 -- start start 2023-10-14T16:41:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0047c0 0x7fdc701f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc7015e610 0x7fdc7015ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc7015efd0 con 0x7fdc700cc4b0 2023-10-14T16:41:28.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc700cdb80 con 0x7fdc5c0047c0 2023-10-14T16:41:28.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.413+0000 7fdc748b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc701f23b0 con 0x7fdc7015e610 2023-10-14T16:41:28.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60638/0 (socket says 172.21.15.29:60638) 2023-10-14T16:41:28.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 -- 172.21.15.29:0/3541091363 learned_addr learned my addr 172.21.15.29:0/3541091363 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:28.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0047c0 0x7fdc701f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 -- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc7015e610 msgr2=0x7fdc7015ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc7015e610 0x7fdc7015ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 -- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0047c0 msgr2=0x7fdc701f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0047c0 0x7fdc701f1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 -- 172.21.15.29:0/3541091363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc700ccf70 con 0x7fdc700cc4b0 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.414+0000 7fdc67fff640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc5802c200 tx=0x7fdc58004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.415+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58003810 con 0x7fdc700cc4b0 2023-10-14T16:41:28.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.415+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc58003970 con 0x7fdc700cc4b0 2023-10-14T16:41:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.415+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc58003d00 con 0x7fdc700cc4b0 2023-10-14T16:41:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.415+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc700cd250 con 0x7fdc700cc4b0 2023-10-14T16:41:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.415+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc700cd810 con 0x7fdc700cc4b0 2023-10-14T16:41:28.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.416+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc700cd9d0 con 0x7fdc700cc4b0 2023-10-14T16:41:28.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.417+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdc58034440 con 0x7fdc700cc4b0 2023-10-14T16:41:28.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.418+0000 7fdc66ffd640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 0x7fdc5006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:28.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.418+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 5473+0+0 (secure 0 0 0) 0x7fdc580ab400 con 0x7fdc700cc4b0 2023-10-14T16:41:28.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.418+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 0x7fdc5006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:28.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.419+0000 7fdc6e59c640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 0x7fdc5006d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009a30 tx=0x7fdc60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:28.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.422+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdc58076750 con 0x7fdc700cc4b0 2023-10-14T16:41:28.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:28.633+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdc7004e160 con 0x7fdc700cc4b0 2023-10-14T16:41:29.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.156+0000 7fdc66ffd640 1 -- 172.21.15.29:0/3541091363 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) v1 ==== 169+0+0 (secure 0 0 0) 0x7fdc58003e60 con 0x7fdc700cc4b0 2023-10-14T16:41:29.157 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:41:29.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.161+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 msgr2=0x7fdc5006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:29.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.161+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 0x7fdc5006d330 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fdc60009a30 tx=0x7fdc60008040 comp rx=0 tx=0).stop 2023-10-14T16:41:29.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.161+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 msgr2=0x7fdc700cc930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:29.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.161+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fdc5802c200 tx=0x7fdc58004730 comp rx=0 tx=0).stop 2023-10-14T16:41:29.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 shutdown_connections 2023-10-14T16:41:29.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdc5006ae70 0x7fdc5006d330 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc7015e610 0x7fdc7015ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc700cc4b0 0x7fdc700cc930 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 --2- 172.21.15.29:0/3541091363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc5c0047c0 0x7fdc701f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 >> 172.21.15.29:0/3541091363 conn(0x7fdc700cb360 msgr2=0x7fdc70159ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:29.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 shutdown_connections 2023-10-14T16:41:29.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.162+0000 7fdc748b4640 1 -- 172.21.15.29:0/3541091363 wait complete. 2023-10-14T16:41:29.189 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:41:29.189 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:41:29.190 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:41:29.190 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:41:29.190 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:41:29.190 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:41:29.191 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:41:29.191 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:41:29.191 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:41:29.192 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:41:29.192 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:41:29.192 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:41:29.192 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:41:29.193 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:41:29.193 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:41:29.193 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:41:29.212 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:41:29.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:41:29.214 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:41:29.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:41:29.215 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:41:29.216 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:41:29.217 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:41:29.217 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:41:29.261 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:41:29.262 DEBUG:teuthology.parallel:result is None 2023-10-14T16:41:29.265 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:41:29.266 DEBUG:teuthology.parallel:result is None 2023-10-14T16:41:29.268 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:41:29.269 DEBUG:teuthology.parallel:result is None 2023-10-14T16:41:29.270 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:41:29.271 DEBUG:teuthology.parallel:result is None 2023-10-14T16:41:29.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:41:29.427 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:41:29.428 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:41:29.431 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:41:29.433 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:41:29.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.472+0000 7fa3783d5640 1 Processor -- start 2023-10-14T16:41:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.472+0000 7fa3783d5640 1 -- start start 2023-10-14T16:41:29.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa3783d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 0x7fa3700ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa3783d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa37006c140 0x7fa37006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa3783d5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa370070360 con 0x7fa370159e90 2023-10-14T16:41:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa3783d5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa37006cb10 con 0x7fa37006c140 2023-10-14T16:41:29.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa3783d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa37015d890 con 0x7fa370159ab0 2023-10-14T16:41:29.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa376972640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa37006c140 0x7fa37006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa376972640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa37006c140 0x7fa37006c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60684/0 (socket says 172.21.15.29:60684) 2023-10-14T16:41:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.473+0000 7fa376972640 1 -- 172.21.15.29:0/3230205359 learned_addr learned my addr 172.21.15.29:0/3230205359 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376171640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 0x7fa3700ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376972640 1 -- 172.21.15.29:0/3230205359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 msgr2=0x7fa3700ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:29.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376972640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 0x7fa3700ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376972640 1 -- 172.21.15.29:0/3230205359 >> v1:172.21.15.186:6789/0 conn(0x7fa370159e90 legacy=0x7fa37006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:29.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376972640 1 -- 172.21.15.29:0/3230205359 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3700cd1f0 con 0x7fa37006c140 2023-10-14T16:41:29.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.474+0000 7fa376171640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 0x7fa3700ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:29.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.475+0000 7fa376972640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa37006c140 0x7fa37006c5d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa368009e80 tx=0x7fa36802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52cb28fbe4b4285e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.475+0000 7fa37516f640 1 -- 172.21.15.29:0/3230205359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36802c930 con 0x7fa37006c140 2023-10-14T16:41:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.475+0000 7fa37516f640 1 -- 172.21.15.29:0/3230205359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa36802ca90 con 0x7fa37006c140 2023-10-14T16:41:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.475+0000 7fa37516f640 1 -- 172.21.15.29:0/3230205359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa36802ce20 con 0x7fa37006c140 2023-10-14T16:41:29.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 -- 172.21.15.29:0/3230205359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa37006c140 msgr2=0x7fa37006c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa37006c140 0x7fa37006c5d0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fa368009e80 tx=0x7fa36802bd90 comp rx=0 tx=0).stop 2023-10-14T16:41:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 -- 172.21.15.29:0/3230205359 shutdown_connections 2023-10-14T16:41:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa37006c140 0x7fa37006c5d0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 --2- 172.21.15.29:0/3230205359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa370159ab0 0x7fa3700ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 -- 172.21.15.29:0/3230205359 >> 172.21.15.29:0/3230205359 conn(0x7fa3700ce9b0 msgr2=0x7fa3700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 -- 172.21.15.29:0/3230205359 shutdown_connections 2023-10-14T16:41:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.476+0000 7fa3783d5640 1 -- 172.21.15.29:0/3230205359 wait complete. 2023-10-14T16:41:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.478+0000 7fa3783d5640 1 Processor -- start 2023-10-14T16:41:29.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.478+0000 7fa3783d5640 1 -- start start 2023-10-14T16:41:29.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa37006c140 0x7fa370155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa370159e90 0x7fa3701e5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3701f2110 con 0x7fa370159ab0 2023-10-14T16:41:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3701f22e0 con 0x7fa370159e90 2023-10-14T16:41:29.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa3783d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3700c3410 con 0x7fa37006c140 2023-10-14T16:41:29.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa375970640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa375970640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60700/0 (socket says 172.21.15.29:60700) 2023-10-14T16:41:29.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.479+0000 7fa375970640 1 -- 172.21.15.29:0/2840862812 learned_addr learned my addr 172.21.15.29:0/2840862812 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:29.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa376972640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa370159e90 0x7fa3701e5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa376171640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa37006c140 0x7fa370155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 -- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa37006c140 msgr2=0x7fa370155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:29.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa37006c140 0x7fa370155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 -- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa370159e90 msgr2=0x7fa3701e5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:29.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa370159e90 0x7fa3701e5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:29.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 -- 172.21.15.29:0/2840862812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa368009b30 con 0x7fa370159ab0 2023-10-14T16:41:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.480+0000 7fa375970640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa36000c430 tx=0x7fa36000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.481+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa360011790 con 0x7fa370159ab0 2023-10-14T16:41:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.481+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3600118f0 con 0x7fa370159ab0 2023-10-14T16:41:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.481+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa360011ce0 con 0x7fa370159ab0 2023-10-14T16:41:29.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.481+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3700c36f0 con 0x7fa370159ab0 2023-10-14T16:41:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.481+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3700c3cb0 con 0x7fa370159ab0 2023-10-14T16:41:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.482+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3701f2420 con 0x7fa370159ab0 2023-10-14T16:41:29.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.483+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fa36001c460 con 0x7fa370159ab0 2023-10-14T16:41:29.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.484+0000 7fa366ffd640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 0x7fa35406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:29.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.484+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 4943+0+0 (secure 0 0 0) 0x7fa3600891b0 con 0x7fa370159ab0 2023-10-14T16:41:29.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.484+0000 7fa376171640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 0x7fa35406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:29.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.485+0000 7fa376171640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 0x7fa35406d330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00aac0 tx=0x7fa36c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:29.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.488+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3701f2420 con 0x7fa370159ab0 2023-10-14T16:41:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:29.702+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fa3701f2420 con 0x7fa370159ab0 2023-10-14T16:41:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.770+0000 7fa366ffd640 1 -- 172.21.15.29:0/2840862812 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v30) v1 ==== 97+0+0 (secure 0 0 0) 0x7fa3701f2420 con 0x7fa370159ab0 2023-10-14T16:41:30.771 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:41:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 msgr2=0x7fa35406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 0x7fa35406d330 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa36c00aac0 tx=0x7fa36c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 msgr2=0x7fa3701e4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fa36000c430 tx=0x7fa36000c8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 shutdown_connections 2023-10-14T16:41:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa35406ae70 0x7fa35406d330 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa370159e90 0x7fa3701e5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.774+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa370159ab0 0x7fa3701e4c30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.775+0000 7fa3783d5640 1 --2- 172.21.15.29:0/2840862812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa37006c140 0x7fa370155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.775+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 >> 172.21.15.29:0/2840862812 conn(0x7fa3700ce9b0 msgr2=0x7fa3700b05a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.775+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 shutdown_connections 2023-10-14T16:41:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:30.775+0000 7fa3783d5640 1 -- 172.21.15.29:0/2840862812 wait complete. 2023-10-14T16:41:30.795 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:41:31.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.015+0000 7fb58353c640 1 Processor -- start 2023-10-14T16:41:31.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.015+0000 7fb58353c640 1 -- start start 2023-10-14T16:41:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58353c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb57c0ad970 0x7fb57c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58353c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb57c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58353c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57c15b9d0 con 0x7fb57c06c040 2023-10-14T16:41:31.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58353c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb57c15bcb0 con 0x7fb57c0ad970 2023-10-14T16:41:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58353c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c15bf90 con 0x7fb57c0ccfb0 2023-10-14T16:41:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58253a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb57c06c040 0x7fb57c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36938/0 (socket says 172.21.15.29:36938) 2023-10-14T16:41:31.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb58253a640 1 -- 172.21.15.29:0/1070445696 learned_addr learned my addr 172.21.15.29:0/1070445696 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3645686416 0 0) 0x7fb57c15b9d0 con 0x7fb57c06c040 2023-10-14T16:41:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb581d39640 1 --2- 172.21.15.29:0/1070445696 >> v2:172.21.15.186:3300/0 conn(0x7fb57c0ad970 0x7fb57c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.017+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb568003520 con 0x7fb57c06c040 2023-10-14T16:41:31.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb582d3b640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb57c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 99518857 0 0) 0x7fb568003520 con 0x7fb57c06c040 2023-10-14T16:41:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 msgr2=0x7fb57c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb57c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 >> v2:172.21.15.186:3300/0 conn(0x7fb57c0ad970 msgr2=0x7fb57c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 --2- 172.21.15.29:0/1070445696 >> v2:172.21.15.186:3300/0 conn(0x7fb57c0ad970 0x7fb57c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0cc270 con 0x7fb57c06c040 2023-10-14T16:41:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb578001670 con 0x7fb57c06c040 2023-10-14T16:41:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 >> v1:172.21.15.29:6789/0 conn(0x7fb57c06c040 legacy=0x7fb57c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb568003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb568003f40 0x7fb5680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568004d00 0x7fb568005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c15bcb0 con 0x7fb568003f40 2023-10-14T16:41:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c15bf90 con 0x7fb568004d00 2023-10-14T16:41:31.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb56800ba30 con 0x7fb57c0ccfb0 2023-10-14T16:41:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.018+0000 7fb581d39640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb568003f40 0x7fb5680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb582d3b640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568004d00 0x7fb568005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb58253a640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb568003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb581d39640 1 -- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 msgr2=0x7fb568003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb581d39640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0ccfb0 0x7fb568003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb581d39640 1 -- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568004d00 msgr2=0x7fb568005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb581d39640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568004d00 0x7fb568005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb581d39640 1 -- 172.21.15.29:0/1070445696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb568003520 con 0x7fb568003f40 2023-10-14T16:41:31.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.019+0000 7fb582d3b640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568004d00 0x7fb568005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb581d39640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb568003f40 0x7fb5680047c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009a90 tx=0x7fb5680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c674ef2b3439a71 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c02c6b0 con 0x7fb568003f40 2023-10-14T16:41:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c02c810 con 0x7fb568003f40 2023-10-14T16:41:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb581538640 1 -- 172.21.15.29:0/1070445696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c02cbc0 con 0x7fb568003f40 2023-10-14T16:41:31.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb58353c640 1 -- 172.21.15.29:0/1070445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb568003f40 msgr2=0x7fb5680047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.020+0000 7fb58353c640 1 --2- 172.21.15.29:0/1070445696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb568003f40 0x7fb5680047c0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fb56c009a90 tx=0x7fb5680057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.021+0000 7fb582d3b640 1 -- 172.21.15.29:0/1070445696 reap_dead start 2023-10-14T16:41:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.021+0000 7fb58353c640 1 -- 172.21.15.29:0/1070445696 shutdown_connections 2023-10-14T16:41:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.021+0000 7fb58353c640 1 -- 172.21.15.29:0/1070445696 >> 172.21.15.29:0/1070445696 conn(0x7fb57c0ce6a0 msgr2=0x7fb57c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:31.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.021+0000 7fb58353c640 1 -- 172.21.15.29:0/1070445696 shutdown_connections 2023-10-14T16:41:31.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.021+0000 7fb58353c640 1 -- 172.21.15.29:0/1070445696 wait complete. 2023-10-14T16:41:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.022+0000 7fb58353c640 1 Processor -- start 2023-10-14T16:41:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.023+0000 7fb58353c640 1 -- start start 2023-10-14T16:41:31.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.023+0000 7fb58353c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568003f40 0x7fb57c14baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb58353c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb58353c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0aefe0 0x7fb57c159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb58353c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0b0710 con 0x7fb57c0af790 2023-10-14T16:41:31.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb58353c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0b0150 con 0x7fb568003f40 2023-10-14T16:41:31.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb58353c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb57c0b0430 con 0x7fb57c0aefe0 2023-10-14T16:41:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55708/0 (socket says 172.21.15.29:55708) 2023-10-14T16:41:31.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 -- 172.21.15.29:0/1517643984 learned_addr learned my addr 172.21.15.29:0/1517643984 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 -- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0aefe0 msgr2=0x7fb57c159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0aefe0 0x7fb57c159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 -- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568003f40 msgr2=0x7fb57c14baa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568003f40 0x7fb57c14baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.024+0000 7fb581d39640 1 -- 172.21.15.29:0/1517643984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57c0ae2a0 con 0x7fb57c0af790 2023-10-14T16:41:31.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb581d39640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0099e0 tx=0x7fb56c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c034a80 con 0x7fb57c0af790 2023-10-14T16:41:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c034c00 con 0x7fb57c0af790 2023-10-14T16:41:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57c0ae580 con 0x7fb57c0af790 2023-10-14T16:41:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb56c03e440 con 0x7fb57c0af790 2023-10-14T16:41:31.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.025+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57c1590a0 con 0x7fb57c0af790 2023-10-14T16:41:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.026+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb57c06b330 con 0x7fb57c0af790 2023-10-14T16:41:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.027+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb56c03e5a0 con 0x7fb57c0af790 2023-10-14T16:41:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.028+0000 7fb572ffd640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 0x7fb56006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.029+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fb56c0ab120 con 0x7fb57c0af790 2023-10-14T16:41:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.029+0000 7fb58253a640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 0x7fb56006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.029+0000 7fb58253a640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 0x7fb56006d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb578001eb0 tx=0x7fb578009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.033+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb56c0765a0 con 0x7fb57c0af790 2023-10-14T16:41:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.288+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb57c0ae100 con 0x7fb57c0af790 2023-10-14T16:41:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.290+0000 7fb572ffd640 1 -- 172.21.15.29:0/1517643984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v2) v1 ==== 80+0+6645 (secure 0 0 0) 0x7fb57c0ae100 con 0x7fb57c0af790 2023-10-14T16:41:31.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:31.291 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:41:31.291 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:41:31.291 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.292 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:41:31.292 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:41:31.292 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.292 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.292 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:41:31.293 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:41:31.293 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.293 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.293 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.293 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.294 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:41:31.294 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:41:31.294 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.294 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.294 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:41:31.295 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:41:31.295 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.295 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.295 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.296 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:41:31.296 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:41:31.296 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.296 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:41:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:41:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:41:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:41:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:41:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:41:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.300 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:41:31.300 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:41:31.300 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.301 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.301 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.301 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.301 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.302 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.302 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:41:31.302 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:41:31.302 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.302 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.303 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.303 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:41:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:41:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.304 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.305 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.305 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.305 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.305 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:41:31.306 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:41:31.306 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.306 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.306 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.306 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.307 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.307 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.307 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.307 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:41:31.307 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:41:31.308 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.308 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.308 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.308 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.309 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.309 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.309 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.309 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:41:31.309 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:41:31.310 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.310 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.310 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.310 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.310 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.311 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.311 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:41:31.311 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:41:31.312 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.312 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.312 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.313 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:41:31.313 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:41:31.313 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:41:31.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:41:31.314 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.314 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.315 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:41:31.315 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:41:31.315 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:41:31.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:41:31.316 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:41:31.316 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.317 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.317 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:41:31.317 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:41:31.317 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.318 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.318 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:41:31.318 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:41:31.318 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:41:31.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:41:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:41:31.319 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.320 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.320 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:41:31.320 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:41:31.320 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:41:31.320 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.321 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.322 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:41:31.322 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:41:31.322 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.322 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:41:31.323 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:41:31.323 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:41:31.323 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:41:31.323 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.324 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:41:31.324 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:41:31.324 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:41:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:41:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:41:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.325 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mds", 2023-10-14T16:41:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCpl0E7JOBAAUjSReSGd+gUEM/nWa9iKRg==", 2023-10-14T16:41:31.326 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.327 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mds" 2023-10-14T16:41:31.327 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.327 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.327 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.327 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-mgr", 2023-10-14T16:41:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCplhmrJOBAAanvaT7gQwgbjnOmWa3rxXQ==", 2023-10-14T16:41:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-mgr" 2023-10-14T16:41:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.329 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.329 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-osd", 2023-10-14T16:41:31.329 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCplHITJOBAAGMkA9qpEhX1hDc5aMf1YTA==", 2023-10-14T16:41:31.330 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.330 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-osd" 2023-10-14T16:41:31.330 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.330 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.331 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.331 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd", 2023-10-14T16:41:31.331 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCplUp7JOBAA3cwoE2Gag9geFrvryJg9+A==", 2023-10-14T16:41:31.331 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.331 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd" 2023-10-14T16:41:31.332 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.332 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.332 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.332 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rbd-mirror", 2023-10-14T16:41:31.332 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCplw7fJOBAAKRLLRHB45WV06IPqBroL2Q==", 2023-10-14T16:41:31.333 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.333 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2023-10-14T16:41:31.333 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.333 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.333 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.334 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.bootstrap-rgw", 2023-10-14T16:41:31.334 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBfxCplENLJOBAAFk8LReaZOp3eAkdavYrUOA==", 2023-10-14T16:41:31.334 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.334 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile bootstrap-rgw" 2023-10-14T16:41:31.334 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.335 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:41:31.336 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:41:31.336 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.336 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.336 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:41:31.336 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:31.337 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:41:31.337 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:41:31.337 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:41:31.337 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:41:31.337 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:41:31.338 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:41:31.338 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.338 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:31.338 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:41:31.338 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:41:31.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.292+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 msgr2=0x7fb56006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.292+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 0x7fb56006d330 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fb578001eb0 tx=0x7fb578009040 comp rx=0 tx=0).stop 2023-10-14T16:41:31.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.292+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 msgr2=0x7fb57c0aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.292+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fb56c0099e0 tx=0x7fb56c004730 comp rx=0 tx=0).stop 2023-10-14T16:41:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 shutdown_connections 2023-10-14T16:41:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb56006ae70 0x7fb56006d330 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb57c0aefe0 0x7fb57c159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb57c0af790 0x7fb57c0aeaa0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 --2- 172.21.15.29:0/1517643984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb568003f40 0x7fb57c14baa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 >> 172.21.15.29:0/1517643984 conn(0x7fb57c0ce6a0 msgr2=0x7fb57c0b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:31.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 shutdown_connections 2023-10-14T16:41:31.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.293+0000 7fb58353c640 1 -- 172.21.15.29:0/1517643984 wait complete. 2023-10-14T16:41:31.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2023-10-14T16:41:31.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.567+0000 7f373759e640 1 Processor -- start 2023-10-14T16:41:31.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.567+0000 7f373759e640 1 -- start start 2023-10-14T16:41:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 0x7f37380ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3738070240 0x7f37380706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3738070be0 con 0x7f373815aaa0 2023-10-14T16:41:31.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37380b2a60 con 0x7f3738070240 2023-10-14T16:41:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37380b2d00 con 0x7f37380b2600 2023-10-14T16:41:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f373815aaa0 0x7f37380b3da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36946/0 (socket says 172.21.15.29:36946) 2023-10-14T16:41:31.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.568+0000 7f373659c640 1 -- 172.21.15.29:0/449790130 learned_addr learned my addr 172.21.15.29:0/449790130 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f3736d9d640 1 --2- 172.21.15.29:0/449790130 >> v2:172.21.15.186:3300/0 conn(0x7f3738070240 0x7f37380706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301647015 0 0) 0x7f3738070be0 con 0x7f373815aaa0 2023-10-14T16:41:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3724003520 con 0x7f373815aaa0 2023-10-14T16:41:31.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f3735d9b640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 0x7f37380ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3032687855 0 0) 0x7f3724003520 con 0x7f373815aaa0 2023-10-14T16:41:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 msgr2=0x7f37380ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 0x7f37380ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 >> v2:172.21.15.186:3300/0 conn(0x7f3738070240 msgr2=0x7f37380706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 --2- 172.21.15.29:0/449790130 >> v2:172.21.15.186:3300/0 conn(0x7f3738070240 0x7f37380706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380b2fe0 con 0x7f373815aaa0 2023-10-14T16:41:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f37280019f0 con 0x7f373815aaa0 2023-10-14T16:41:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 >> v1:172.21.15.29:6789/0 conn(0x7f373815aaa0 legacy=0x7f37380b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724003970 0x7f3724003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37240047c0 0x7f3724004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 0x7f3724005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380b2a60 con 0x7f37240047c0 2023-10-14T16:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380b2d00 con 0x7f3724003970 2023-10-14T16:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.569+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f372400fef0 con 0x7f37380b2600 2023-10-14T16:41:31.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3736d9d640 1 --2- 172.21.15.29:0/449790130 >> v2:172.21.15.186:3300/0 conn(0x7f3738070240 0x7f37380706a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37240047c0 0x7f3724004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 -- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 msgr2=0x7f3724005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f373659c640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724003970 0x7f3724003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37380b2600 0x7f3724005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 -- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724003970 msgr2=0x7f3724003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:31.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3724003970 0x7f3724003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.570+0000 7f3735d9b640 1 -- 172.21.15.29:0/449790130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3724003520 con 0x7f37240047c0 2023-10-14T16:41:31.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.571+0000 7f3735d9b640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37240047c0 0x7f3724004c40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f372000ba00 tx=0x7f37240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba403820b3746d4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720032620 con 0x7f37240047c0 2023-10-14T16:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3720032780 con 0x7f37240047c0 2023-10-14T16:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373559a640 1 -- 172.21.15.29:0/449790130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720032b10 con 0x7f37240047c0 2023-10-14T16:41:31.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373759e640 1 -- 172.21.15.29:0/449790130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37240047c0 msgr2=0x7f3724004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373759e640 1 --2- 172.21.15.29:0/449790130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37240047c0 0x7f3724004c40 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f372000ba00 tx=0x7f37240057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f3736d9d640 1 -- 172.21.15.29:0/449790130 reap_dead start 2023-10-14T16:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373759e640 1 -- 172.21.15.29:0/449790130 shutdown_connections 2023-10-14T16:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373759e640 1 -- 172.21.15.29:0/449790130 >> 172.21.15.29:0/449790130 conn(0x7f37380cc230 msgr2=0x7f37380cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:31.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.572+0000 7f373759e640 1 -- 172.21.15.29:0/449790130 shutdown_connections 2023-10-14T16:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.573+0000 7f373759e640 1 -- 172.21.15.29:0/449790130 wait complete. 2023-10-14T16:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.574+0000 7f373759e640 1 Processor -- start 2023-10-14T16:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.574+0000 7f373759e640 1 -- start start 2023-10-14T16:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 0x7f3738154270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806b560 0x7f373806b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380c7710 con 0x7f3738154a80 2023-10-14T16:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37380c79f0 con 0x7f37240047c0 2023-10-14T16:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.575+0000 7f373759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37380c7cd0 con 0x7f373806b560 2023-10-14T16:41:31.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55718/0 (socket says 172.21.15.29:55718) 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 -- 172.21.15.29:0/2836330415 learned_addr learned my addr 172.21.15.29:0/2836330415 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f373659c640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 0x7f3738154270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3736d9d640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806b560 0x7f373806b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 -- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806b560 msgr2=0x7f373806b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806b560 0x7f373806b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 -- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 msgr2=0x7f3738154270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 0x7f3738154270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f3735d9b640 1 -- 172.21.15.29:0/2836330415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37380c7fb0 con 0x7f3738154a80 2023-10-14T16:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.576+0000 7f373659c640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 0x7f3738154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:31.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f3735d9b640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f372000bb20 tx=0x7f3720004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3720004020 con 0x7f3738154a80 2023-10-14T16:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3720004180 con 0x7f3738154a80 2023-10-14T16:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37200044f0 con 0x7f3738154a80 2023-10-14T16:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37380c8290 con 0x7f3738154a80 2023-10-14T16:41:31.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.577+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37380c9af0 con 0x7f3738154a80 2023-10-14T16:41:31.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.578+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37380c8420 con 0x7f3738154a80 2023-10-14T16:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.579+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f3720003610 con 0x7f3738154a80 2023-10-14T16:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.581+0000 7f372effd640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 0x7f371806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.581+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f37200ab5e0 con 0x7f3738154a80 2023-10-14T16:41:31.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.581+0000 7f373659c640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 0x7f371806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:31.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.582+0000 7f373659c640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 0x7f371806d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3728001ba0 tx=0x7f3728009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:31.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.585+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f372007a3d0 con 0x7f3738154a80 2023-10-14T16:41:31.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.850+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f37380c8420 con 0x7f3738154a80 2023-10-14T16:41:31.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.855+0000 7f372effd640 1 -- 172.21.15.29:0/2836330415 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v3) v1 ==== 90+0+0 (secure 0 0 0) 0x7f3720006c30 con 0x7f3738154a80 2023-10-14T16:41:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 msgr2=0x7f371806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 0x7f371806d330 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3728001ba0 tx=0x7f3728009040 comp rx=0 tx=0).stop 2023-10-14T16:41:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 msgr2=0x7f37380c95b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f372000bb20 tx=0x7f3720004730 comp rx=0 tx=0).stop 2023-10-14T16:41:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 shutdown_connections 2023-10-14T16:41:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f371806ae70 0x7f371806d330 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f373806b560 0x7f373806b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3738154a80 0x7f37380c95b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 --2- 172.21.15.29:0/2836330415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37240047c0 0x7f3738154270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 >> 172.21.15.29:0/2836330415 conn(0x7f37380cc230 msgr2=0x7f37380ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.858+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 shutdown_connections 2023-10-14T16:41:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:31.859+0000 7f373759e640 1 -- 172.21.15.29:0/2836330415 wait complete. 2023-10-14T16:41:31.877 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2023-10-14T16:41:32.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.105+0000 7ff4a6a9f640 1 Processor -- start 2023-10-14T16:41:32.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.105+0000 7ff4a6a9f640 1 -- start start 2023-10-14T16:41:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.106+0000 7ff4a6a9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4a015cd70 0x7ff4a00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.106+0000 7ff4a6a9f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 0x7ff4a00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a6a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a6a9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a015d150 con 0x7ff4a015cd70 2023-10-14T16:41:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a6a9f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4a00d1180 con 0x7ff4a00b8480 2023-10-14T16:41:32.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a6a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a00d1460 con 0x7ff4a00d0be0 2023-10-14T16:41:32.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a5a9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4a015cd70 0x7ff4a00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a5a9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff4a015cd70 0x7ff4a00d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55726/0 (socket says 172.21.15.29:55726) 2023-10-14T16:41:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3190109685 learned_addr learned my addr 172.21.15.29:0/3190109685 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.107+0000 7ff4a529c640 1 --2- 172.21.15.29:0/3190109685 >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 0x7ff4a00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a629e640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3190109685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 msgr2=0x7ff4a00cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3190109685 >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 msgr2=0x7ff4a00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3190109685 >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 0x7ff4a00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3190109685 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4a00d1740 con 0x7ff4a015cd70 2023-10-14T16:41:32.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.108+0000 7ff4a529c640 1 --2- 172.21.15.29:0/3190109685 >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 0x7ff4a00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.109+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a015cd70 0x7ff4a00d2220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0066a00 tx=0x7ff49002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57859193a61c3128 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a4a9b640 1 -- 172.21.15.29:0/3190109685 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49002ca70 con 0x7ff4a015cd70 2023-10-14T16:41:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a4a9b640 1 -- 172.21.15.29:0/3190109685 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49002cbd0 con 0x7ff4a015cd70 2023-10-14T16:41:32.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a4a9b640 1 -- 172.21.15.29:0/3190109685 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4900343f0 con 0x7ff4a015cd70 2023-10-14T16:41:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3190109685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a015cd70 msgr2=0x7ff4a00d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a015cd70 0x7ff4a00d2220 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7ff4a0066a00 tx=0x7ff49002bed0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3190109685 shutdown_connections 2023-10-14T16:41:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3190109685 >> v2:172.21.15.186:3300/0 conn(0x7ff4a00b8480 0x7ff4a00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3190109685 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a015cd70 0x7ff4a00d2220 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3190109685 >> 172.21.15.29:0/3190109685 conn(0x7ff4a00cf570 msgr2=0x7ff4a00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3190109685 shutdown_connections 2023-10-14T16:41:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.110+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3190109685 wait complete. 2023-10-14T16:41:32.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.112+0000 7ff4a6a9f640 1 Processor -- start 2023-10-14T16:41:32.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.112+0000 7ff4a6a9f640 1 -- start start 2023-10-14T16:41:32.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00b7060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a015cd70 0x7ff4a00b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01559d0 con 0x7ff4a00b8480 2023-10-14T16:41:32.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a0154dc0 con 0x7ff4a015cd70 2023-10-14T16:41:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.113+0000 7ff4a6a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4a01550a0 con 0x7ff4a00d0be0 2023-10-14T16:41:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55738/0 (socket says 172.21.15.29:55738) 2023-10-14T16:41:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3947695632 learned_addr learned my addr 172.21.15.29:0/3947695632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:32.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 msgr2=0x7ff4a00b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a015cd70 msgr2=0x7ff4a00b75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a015cd70 0x7ff4a00b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 -- 172.21.15.29:0/3947695632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff490009b30 con 0x7ff4a00b8480 2023-10-14T16:41:32.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.114+0000 7ff4a5a9d640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff490000f80 tx=0x7ff490004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.115+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490009510 con 0x7ff4a00b8480 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.115+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49003e450 con 0x7ff4a00b8480 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.115+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49003e800 con 0x7ff4a00b8480 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.115+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4a0155380 con 0x7ff4a00b8480 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.115+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4a00c8b20 con 0x7ff4a00b8480 2023-10-14T16:41:32.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.116+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4a004e9b0 con 0x7ff4a00b8480 2023-10-14T16:41:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.117+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff49003e960 con 0x7ff4a00b8480 2023-10-14T16:41:32.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.118+0000 7ff49e7fc640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 0x7ff48c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.118+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff4900ac120 con 0x7ff4a00b8480 2023-10-14T16:41:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.118+0000 7ff4a529c640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 0x7ff48c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.119+0000 7ff4a529c640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 0x7ff48c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff494006fd0 tx=0x7ff494008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.123+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff490077510 con 0x7ff4a00b8480 2023-10-14T16:41:32.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.393+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7ff4a00b6860 con 0x7ff4a00b8480 2023-10-14T16:41:32.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.397+0000 7ff49e7fc640 1 -- 172.21.15.29:0/3947695632 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v4) v1 ==== 90+0+0 (secure 0 0 0) 0x7ff49007ab30 con 0x7ff4a00b8480 2023-10-14T16:41:32.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 msgr2=0x7ff48c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 0x7ff48c06d330 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7ff494006fd0 tx=0x7ff494008040 comp rx=0 tx=0).stop 2023-10-14T16:41:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 msgr2=0x7ff4a00c85e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7ff490000f80 tx=0x7ff490004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 shutdown_connections 2023-10-14T16:41:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff48c06ae70 0x7ff48c06d330 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff4a015cd70 0x7ff4a00b75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4a00d0be0 0x7ff4a00b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 --2- 172.21.15.29:0/3947695632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff4a00b8480 0x7ff4a00c85e0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 >> 172.21.15.29:0/3947695632 conn(0x7ff4a00cf570 msgr2=0x7ff4a00d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 shutdown_connections 2023-10-14T16:41:32.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.400+0000 7ff4a6a9f640 1 -- 172.21.15.29:0/3947695632 wait complete. 2023-10-14T16:41:32.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2023-10-14T16:41:32.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.639+0000 7f0764b27640 1 Processor -- start 2023-10-14T16:41:32.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.640+0000 7f0764b27640 1 -- start start 2023-10-14T16:41:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.641+0000 7f0764b27640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f076006c040 0x7f076006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.641+0000 7f0764b27640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f07600ad970 0x7f076015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.641+0000 7f0764b27640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f076006c960 con 0x7f076015ebc0 2023-10-14T16:41:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.641+0000 7f0764b27640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076015b9d0 con 0x7f076006c040 2023-10-14T16:41:32.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.641+0000 7f0764b27640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076015bcb0 con 0x7f07600ad970 2023-10-14T16:41:32.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f076006c040 0x7f076006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f076006c040 0x7f076006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55740/0 (socket says 172.21.15.29:55740) 2023-10-14T16:41:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 -- 172.21.15.29:0/1340856807 learned_addr learned my addr 172.21.15.29:0/1340856807 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075effd640 1 --2- 172.21.15.29:0/1340856807 >> v2:172.21.15.186:3300/0 conn(0x7f07600ad970 0x7f076015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 -- 172.21.15.29:0/1340856807 >> v2:172.21.15.186:3300/0 conn(0x7f07600ad970 msgr2=0x7f076015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:32.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 --2- 172.21.15.29:0/1340856807 >> v2:172.21.15.186:3300/0 conn(0x7f07600ad970 0x7f076015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 -- 172.21.15.29:0/1340856807 >> v1:172.21.15.186:6789/0 conn(0x7f076015ebc0 legacy=0x7f07600cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.642+0000 7f075f7fe640 1 -- 172.21.15.29:0/1340856807 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076015bf90 con 0x7f076006c040 2023-10-14T16:41:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.643+0000 7f075f7fe640 1 --2- 172.21.15.29:0/1340856807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f076006c420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07600cb060 tx=0x7f075002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fd3144ed67969c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f075e7fc640 1 -- 172.21.15.29:0/1340856807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750031870 con 0x7f076006c040 2023-10-14T16:41:32.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f075e7fc640 1 -- 172.21.15.29:0/1340856807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07500319d0 con 0x7f076006c040 2023-10-14T16:41:32.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f075e7fc640 1 -- 172.21.15.29:0/1340856807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0750031d60 con 0x7f076006c040 2023-10-14T16:41:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 -- 172.21.15.29:0/1340856807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 msgr2=0x7f076006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 --2- 172.21.15.29:0/1340856807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f076006c420 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f07600cb060 tx=0x7f075002b910 comp rx=0 tx=0).stop 2023-10-14T16:41:32.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 -- 172.21.15.29:0/1340856807 shutdown_connections 2023-10-14T16:41:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 --2- 172.21.15.29:0/1340856807 >> v2:172.21.15.186:3300/0 conn(0x7f07600ad970 0x7f076015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 --2- 172.21.15.29:0/1340856807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f076006c420 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.644+0000 7f0764b27640 1 -- 172.21.15.29:0/1340856807 >> 172.21.15.29:0/1340856807 conn(0x7f07600ce6a0 msgr2=0x7f07600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.645+0000 7f0764b27640 1 -- 172.21.15.29:0/1340856807 shutdown_connections 2023-10-14T16:41:32.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.645+0000 7f0764b27640 1 -- 172.21.15.29:0/1340856807 wait complete. 2023-10-14T16:41:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.646+0000 7f0764b27640 1 Processor -- start 2023-10-14T16:41:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.646+0000 7f0764b27640 1 -- start start 2023-10-14T16:41:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600ad970 0x7f07600b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 0x7f07600b2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600b39b0 con 0x7f076006c040 2023-10-14T16:41:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0760155f60 con 0x7f076015ebc0 2023-10-14T16:41:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f0764b27640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07600b2d30 con 0x7f07600ad970 2023-10-14T16:41:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.647+0000 7f075f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55744/0 (socket says 172.21.15.29:55744) 2023-10-14T16:41:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 -- 172.21.15.29:0/2870237861 learned_addr learned my addr 172.21.15.29:0/2870237861 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075ffff640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 0x7f07600b2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 -- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600ad970 msgr2=0x7f07600b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075effd640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600ad970 0x7f07600b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600ad970 0x7f07600b3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 -- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 msgr2=0x7f07600b2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 0x7f07600b2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 -- 172.21.15.29:0/2870237861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07500094d0 con 0x7f076006c040 2023-10-14T16:41:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075ffff640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 0x7f07600b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.648+0000 7f075f7fe640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f075002b950 tx=0x7f07500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.649+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075003ad30 con 0x7f076006c040 2023-10-14T16:41:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.649+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07500443f0 con 0x7f076006c040 2023-10-14T16:41:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.649+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07500447a0 con 0x7f076006c040 2023-10-14T16:41:32.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.649+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07600b3010 con 0x7f076006c040 2023-10-14T16:41:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.649+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07600b35d0 con 0x7f076006c040 2023-10-14T16:41:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.650+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076006b330 con 0x7f076006c040 2023-10-14T16:41:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.651+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f0750044900 con 0x7f076006c040 2023-10-14T16:41:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.652+0000 7f0747fff640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 0x7f074c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:32.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.652+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f075002d040 con 0x7f076006c040 2023-10-14T16:41:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.652+0000 7f075effd640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 0x7f074c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.653+0000 7f075effd640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 0x7f074c06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0754006fd0 tx=0x7f0754008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:32.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.656+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0750076cf0 con 0x7f076006c040 2023-10-14T16:41:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.911+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7f07600b3c20 con 0x7f076006c040 2023-10-14T16:41:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.916+0000 7f0747fff640 1 -- 172.21.15.29:0/2870237861 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v5) v1 ==== 90+0+0 (secure 0 0 0) 0x7f0750035020 con 0x7f076006c040 2023-10-14T16:41:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 msgr2=0x7f074c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 0x7f074c06d330 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f0754006fd0 tx=0x7f0754008040 comp rx=0 tx=0).stop 2023-10-14T16:41:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 msgr2=0x7f07600af490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f075002b950 tx=0x7f07500046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 shutdown_connections 2023-10-14T16:41:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f074c06ae70 0x7f074c06d330 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f076015ebc0 0x7f07600b2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600ad970 0x7f07600b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 --2- 172.21.15.29:0/2870237861 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f076006c040 0x7f07600af490 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 >> 172.21.15.29:0/2870237861 conn(0x7f07600ce6a0 msgr2=0x7f07600c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.918+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 shutdown_connections 2023-10-14T16:41:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:32.919+0000 7f0764b27640 1 -- 172.21.15.29:0/2870237861 wait complete. 2023-10-14T16:41:32.936 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2023-10-14T16:41:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.159+0000 7f5fa2251640 1 Processor -- start 2023-10-14T16:41:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.159+0000 7f5fa2251640 1 -- start start 2023-10-14T16:41:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5fa2251640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c15ad70 0x7f5f9c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5fa2251640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f9c0c9ca0 0x7f5f9c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5fa2251640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9c0cb030 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5fa2251640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9c0c91c0 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5fa2251640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c0c94a0 con 0x7f5f9c15ad70 2023-10-14T16:41:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5f9affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5f9c0c9ca0 0x7f5f9c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5f9bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f9c0ca3b0 0x7f5f9c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36952/0 (socket says 172.21.15.29:36952) 2023-10-14T16:41:33.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.161+0000 7f5f9bfff640 1 -- 172.21.15.29:0/1061072318 learned_addr learned my addr 172.21.15.29:0/1061072318 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9b7fe640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c15ad70 0x7f5f9c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9affd640 1 -- 172.21.15.29:0/1061072318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c15ad70 msgr2=0x7f5f9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9affd640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c15ad70 0x7f5f9c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9affd640 1 -- 172.21.15.29:0/1061072318 >> v1:172.21.15.29:6789/0 conn(0x7f5f9c0ca3b0 legacy=0x7f5f9c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9a7fc640 1 -- 172.21.15.29:0/1061072318 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313030141 0 0) 0x7f5f9c0cb030 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.162+0000 7f5f9affd640 1 -- 172.21.15.29:0/1061072318 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c0c9780 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.163+0000 7f5f9affd640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c9ca0 0x7f5f9c06a2e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5f840097a0 tx=0x7f5f8402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7dd7608816a98a49 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.164+0000 7f5f9a7fc640 1 -- 172.21.15.29:0/1061072318 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84031830 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.164+0000 7f5f9a7fc640 1 -- 172.21.15.29:0/1061072318 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84031990 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.164+0000 7f5f9a7fc640 1 -- 172.21.15.29:0/1061072318 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f84031d20 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.164+0000 7f5fa2251640 1 -- 172.21.15.29:0/1061072318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c9ca0 msgr2=0x7f5f9c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.164+0000 7f5fa2251640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c9ca0 0x7f5f9c06a2e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5f840097a0 tx=0x7f5f8402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 -- 172.21.15.29:0/1061072318 shutdown_connections 2023-10-14T16:41:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0c9ca0 0x7f5f9c06a2e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 --2- 172.21.15.29:0/1061072318 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c15ad70 0x7f5f9c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 -- 172.21.15.29:0/1061072318 >> 172.21.15.29:0/1061072318 conn(0x7f5f9c0ce9b0 msgr2=0x7f5f9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 -- 172.21.15.29:0/1061072318 shutdown_connections 2023-10-14T16:41:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.165+0000 7f5fa2251640 1 -- 172.21.15.29:0/1061072318 wait complete. 2023-10-14T16:41:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.166+0000 7f5fa2251640 1 Processor -- start 2023-10-14T16:41:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.166+0000 7f5fa2251640 1 -- start start 2023-10-14T16:41:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c0c9ca0 0x7f5f9c0c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f9c15ad70 0x7f5f9c15d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c1f2540 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c15ddf0 con 0x7f5f9c15ad70 2023-10-14T16:41:33.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.167+0000 7f5fa2251640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c15e300 con 0x7f5f9c0c9ca0 2023-10-14T16:41:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55758/0 (socket says 172.21.15.29:55758) 2023-10-14T16:41:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 -- 172.21.15.29:0/2439855481 learned_addr learned my addr 172.21.15.29:0/2439855481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:33.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 -- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c0c9ca0 msgr2=0x7f5f9c0c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c0c9ca0 0x7f5f9c0c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 -- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f9c15ad70 msgr2=0x7f5f9c15d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f9c15ad70 0x7f5f9c15d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 -- 172.21.15.29:0/2439855481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f84009400 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.168+0000 7f5f9affd640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5f8402bee0 tx=0x7f5f840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.169+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8403a6f0 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.169+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f8403a870 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.169+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f8403ac20 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.169+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c0c2530 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.169+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c0c27d0 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.170+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c066de0 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.171+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5f84030380 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.172+0000 7f5f93fff640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 0x7f5f7c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.172+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f5f840aad30 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.172+0000 7f5f9b7fe640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 0x7f5f7c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.173+0000 7f5f9b7fe640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 0x7f5f7c06d330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0070b0 tx=0x7f5f8c007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.176+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f840b0020 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.442+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f5f9c0b3ec0 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.446+0000 7f5f93fff640 1 -- 172.21.15.29:0/2439855481 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v6) v1 ==== 90+0+0 (secure 0 0 0) 0x7f5f84076120 con 0x7f5f9c0ca3b0 2023-10-14T16:41:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 msgr2=0x7f5f7c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 0x7f5f7c06d330 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5f8c0070b0 tx=0x7f5f8c007040 comp rx=0 tx=0).stop 2023-10-14T16:41:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 msgr2=0x7f5f9c0b18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f5f8402bee0 tx=0x7f5f840046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 shutdown_connections 2023-10-14T16:41:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f7c06ae70 0x7f5f7c06d330 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f9c15ad70 0x7f5f9c15d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f9c0ca3b0 0x7f5f9c0b18a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 --2- 172.21.15.29:0/2439855481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f9c0c9ca0 0x7f5f9c0c4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 >> 172.21.15.29:0/2439855481 conn(0x7f5f9c0ce9b0 msgr2=0x7f5f9c06a510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 shutdown_connections 2023-10-14T16:41:33.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.450+0000 7f5fa2251640 1 -- 172.21.15.29:0/2439855481 wait complete. 2023-10-14T16:41:33.471 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2023-10-14T16:41:33.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.695+0000 7f2ec759e640 1 Processor -- start 2023-10-14T16:41:33.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.695+0000 7f2ec759e640 1 -- start start 2023-10-14T16:41:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.697+0000 7f2ec759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.697+0000 7f2ec759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ec80ccfb0 0x7f2ec80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.697+0000 7f2ec759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec815b9d0 con 0x7f2ec806c040 2023-10-14T16:41:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.697+0000 7f2ec759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec815bcb0 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.697+0000 7f2ec759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec815bf90 con 0x7f2ec80ad970 2023-10-14T16:41:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec5d9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec659c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f2ec806c040 0x7f2ec806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:59780/0 (socket says 172.21.15.29:59780) 2023-10-14T16:41:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec659c640 1 -- 172.21.15.29:0/24718848 learned_addr learned my addr 172.21.15.29:0/24718848 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec6d9d640 1 --2- 172.21.15.29:0/24718848 >> v2:172.21.15.186:3300/0 conn(0x7f2ec80ccfb0 0x7f2ec80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec5d9b640 1 -- 172.21.15.29:0/24718848 >> v2:172.21.15.186:3300/0 conn(0x7f2ec80ccfb0 msgr2=0x7f2ec80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec5d9b640 1 --2- 172.21.15.29:0/24718848 >> v2:172.21.15.186:3300/0 conn(0x7f2ec80ccfb0 0x7f2ec80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec5d9b640 1 -- 172.21.15.29:0/24718848 >> v1:172.21.15.186:6789/0 conn(0x7f2ec806c040 legacy=0x7f2ec806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec5d9b640 1 -- 172.21.15.29:0/24718848 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec80cc270 con 0x7f2ec80ad970 2023-10-14T16:41:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.698+0000 7f2ec559a640 1 -- 172.21.15.29:0/24718848 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724522799 0 0) 0x7f2ec815b9d0 con 0x7f2ec806c040 2023-10-14T16:41:33.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.700+0000 7f2ec5d9b640 1 --2- 172.21.15.29:0/24718848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80cca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0097d0 tx=0x7f2ebc02b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b31bdac30712b72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec559a640 1 -- 172.21.15.29:0/24718848 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc031800 con 0x7f2ec80ad970 2023-10-14T16:41:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec559a640 1 -- 172.21.15.29:0/24718848 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ebc031960 con 0x7f2ec80ad970 2023-10-14T16:41:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec559a640 1 -- 172.21.15.29:0/24718848 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ebc031cf0 con 0x7f2ec80ad970 2023-10-14T16:41:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 -- 172.21.15.29:0/24718848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 msgr2=0x7f2ec80cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 --2- 172.21.15.29:0/24718848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80cca70 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f2ebc0097d0 tx=0x7f2ebc02b8a0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 -- 172.21.15.29:0/24718848 shutdown_connections 2023-10-14T16:41:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 --2- 172.21.15.29:0/24718848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80cca70 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 --2- 172.21.15.29:0/24718848 >> v2:172.21.15.186:3300/0 conn(0x7f2ec80ccfb0 0x7f2ec80cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 -- 172.21.15.29:0/24718848 >> 172.21.15.29:0/24718848 conn(0x7f2ec80ce6a0 msgr2=0x7f2ec80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.702+0000 7f2ec759e640 1 -- 172.21.15.29:0/24718848 shutdown_connections 2023-10-14T16:41:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.703+0000 7f2ec759e640 1 -- 172.21.15.29:0/24718848 wait complete. 2023-10-14T16:41:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.704+0000 7f2ec759e640 1 Processor -- start 2023-10-14T16:41:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.704+0000 7f2ec759e640 1 -- start start 2023-10-14T16:41:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.705+0000 7f2ec759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 0x7f2ec81570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.705+0000 7f2ec759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80c8140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.705+0000 7f2ec759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80af460 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80b10b0 con 0x7f2ec806c040 2023-10-14T16:41:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec8159f30 con 0x7f2ec80ad970 2023-10-14T16:41:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55770/0 (socket says 172.21.15.29:55770) 2023-10-14T16:41:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 -- 172.21.15.29:0/2854657413 learned_addr learned my addr 172.21.15.29:0/2854657413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec5d9b640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec659c640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 0x7f2ec81570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 -- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 msgr2=0x7f2ec80c8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80c8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 -- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 msgr2=0x7f2ec81570a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 0x7f2ec81570a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec6d9d640 1 -- 172.21.15.29:0/2854657413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ebc009400 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec659c640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 0x7f2ec81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.706+0000 7f2ec5d9b640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2ec6d9d640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009d30 tx=0x7f2ec000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0009ba0 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec000f040 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec8159600 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00146c0 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.707+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec8159bc0 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.708+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec806b330 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.710+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2ec0014820 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.710+0000 7f2eb6ffd640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 0x7f2ea806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.711+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2ec0089d50 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.711+0000 7f2ec659c640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 0x7f2ea806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.711+0000 7f2ec659c640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 0x7f2ea806d360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8000f60 tx=0x7f2eb8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.715+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ec008d030 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.982+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f2ec80acfa0 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.986+0000 7f2eb6ffd640 1 -- 172.21.15.29:0/2854657413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2ec0055150 con 0x7f2ec80ccfb0 2023-10-14T16:41:33.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.989+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 msgr2=0x7f2ea806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.989+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 0x7f2ea806d360 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2eb8000f60 tx=0x7f2eb8009040 comp rx=0 tx=0).stop 2023-10-14T16:41:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.989+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 msgr2=0x7f2ec80ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.989+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009d30 tx=0x7f2ec000c490 comp rx=0 tx=0).stop 2023-10-14T16:41:33.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 shutdown_connections 2023-10-14T16:41:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2ea806aea0 0x7f2ea806d360 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec80ccfb0 0x7f2ec80ae100 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec80ad970 0x7f2ec80c8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 --2- 172.21.15.29:0/2854657413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec806c040 0x7f2ec81570a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 >> 172.21.15.29:0/2854657413 conn(0x7f2ec80ce6a0 msgr2=0x7f2ec80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 shutdown_connections 2023-10-14T16:41:33.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:33.990+0000 7f2ec759e640 1 -- 172.21.15.29:0/2854657413 wait complete. 2023-10-14T16:41:34.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2023-10-14T16:41:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.228+0000 7f2a727b0640 1 Processor -- start 2023-10-14T16:41:34.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.228+0000 7f2a727b0640 1 -- start start 2023-10-14T16:41:34.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a727b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a6c15d410 0x7f2a6c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a727b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c0cd260 0x7f2a6c0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a727b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a6c0b5d90 con 0x7f2a6c15d030 2023-10-14T16:41:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a727b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a6c0d2b50 con 0x7f2a6c15d410 2023-10-14T16:41:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a727b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c0cdbf0 con 0x7f2a6c0cd260 2023-10-14T16:41:34.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a6b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a6c15d410 0x7f2a6c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a6b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a6c15d410 0x7f2a6c0d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55774/0 (socket says 172.21.15.29:55774) 2023-10-14T16:41:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a6b7fe640 1 -- 172.21.15.29:0/3132252486 learned_addr learned my addr 172.21.15.29:0/3132252486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.230+0000 7f2a6bfff640 1 --1- 172.21.15.29:0/3132252486 >> v1:172.21.15.29:6789/0 conn(0x7f2a6c15d030 0x7f2a6c161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36958/0 (socket says 172.21.15.29:36958) 2023-10-14T16:41:34.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a70d4d640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c0cd260 0x7f2a6c0cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a6b7fe640 1 -- 172.21.15.29:0/3132252486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c0cd260 msgr2=0x7f2a6c0cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a6b7fe640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c0cd260 0x7f2a6c0cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a6b7fe640 1 -- 172.21.15.29:0/3132252486 >> v1:172.21.15.29:6789/0 conn(0x7f2a6c15d030 legacy=0x7f2a6c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a6b7fe640 1 -- 172.21.15.29:0/3132252486 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c160fc0 con 0x7f2a6c15d410 2023-10-14T16:41:34.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.231+0000 7f2a6affd640 1 -- 172.21.15.29:0/3132252486 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2153840718 0 0) 0x7f2a6c0b5d90 con 0x7f2a6c15d030 2023-10-14T16:41:34.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.232+0000 7f2a6b7fe640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0d1630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009870 tx=0x7f2a6002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=537ac962f68d3bd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.232+0000 7f2a6affd640 1 -- 172.21.15.29:0/3132252486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6002f830 con 0x7f2a6c15d410 2023-10-14T16:41:34.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.232+0000 7f2a6affd640 1 -- 172.21.15.29:0/3132252486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a6002f990 con 0x7f2a6c15d410 2023-10-14T16:41:34.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.232+0000 7f2a6affd640 1 -- 172.21.15.29:0/3132252486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a6002fd20 con 0x7f2a6c15d410 2023-10-14T16:41:34.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 -- 172.21.15.29:0/3132252486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 msgr2=0x7f2a6c0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0d1630 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f2a60009870 tx=0x7f2a6002ba10 comp rx=0 tx=0).stop 2023-10-14T16:41:34.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 -- 172.21.15.29:0/3132252486 shutdown_connections 2023-10-14T16:41:34.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c0cd260 0x7f2a6c0cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3132252486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0d1630 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 -- 172.21.15.29:0/3132252486 >> 172.21.15.29:0/3132252486 conn(0x7f2a6c0cf720 msgr2=0x7f2a6c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 -- 172.21.15.29:0/3132252486 shutdown_connections 2023-10-14T16:41:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.233+0000 7f2a727b0640 1 -- 172.21.15.29:0/3132252486 wait complete. 2023-10-14T16:41:34.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.235+0000 7f2a727b0640 1 Processor -- start 2023-10-14T16:41:34.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.235+0000 7f2a727b0640 1 -- start start 2023-10-14T16:41:34.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a6c0cd260 0x7f2a6c0cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c15d030 0x7f2a6c15a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1e78d0 con 0x7f2a6c15d410 2023-10-14T16:41:34.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c15e310 con 0x7f2a6c0cd260 2023-10-14T16:41:34.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a727b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c0cc960 con 0x7f2a6c15d030 2023-10-14T16:41:34.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a70d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a70d4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55786/0 (socket says 172.21.15.29:55786) 2023-10-14T16:41:34.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.236+0000 7f2a70d4d640 1 -- 172.21.15.29:0/3845152477 learned_addr learned my addr 172.21.15.29:0/3845152477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:34.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a6bfff640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a6c0cd260 0x7f2a6c0cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a6b7fe640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c15d030 0x7f2a6c15a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 -- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c15d030 msgr2=0x7f2a6c15a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c15d030 0x7f2a6c15a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 -- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a6c0cd260 msgr2=0x7f2a6c0cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a6c0cd260 0x7f2a6c0cc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 -- 172.21.15.29:0/3845152477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a600094d0 con 0x7f2a6c15d410 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a70d4d640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400cd70 tx=0x7f2a6400a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64003b90 con 0x7f2a6c15d410 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a64003cf0 con 0x7f2a6c15d410 2023-10-14T16:41:34.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c15d9e0 con 0x7f2a6c15d410 2023-10-14T16:41:34.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.237+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a64004240 con 0x7f2a6c15d410 2023-10-14T16:41:34.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.238+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c15dfa0 con 0x7f2a6c15d410 2023-10-14T16:41:34.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.239+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6c15e480 con 0x7f2a6c15d410 2023-10-14T16:41:34.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.240+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f2a640043a0 con 0x7f2a6c15d410 2023-10-14T16:41:34.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.241+0000 7f2a68ff9640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 0x7f2a5806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.241+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7f2a64012070 con 0x7f2a6c15d410 2023-10-14T16:41:34.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.241+0000 7f2a6bfff640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 0x7f2a5806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.242+0000 7f2a6bfff640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 0x7f2a5806d360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000f60 tx=0x7f2a5c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.245+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a64055080 con 0x7f2a6c15d410 2023-10-14T16:41:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.508+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f2a6c0c6630 con 0x7f2a6c15d410 2023-10-14T16:41:34.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.512+0000 7f2a68ff9640 1 -- 172.21.15.29:0/3845152477 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v8) v1 ==== 90+0+0 (secure 0 0 0) 0x7f2a640586a0 con 0x7f2a6c15d410 2023-10-14T16:41:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 msgr2=0x7f2a5806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 0x7f2a5806d360 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c000f60 tx=0x7f2a5c009040 comp rx=0 tx=0).stop 2023-10-14T16:41:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 msgr2=0x7f2a6c0b2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f2a6400cd70 tx=0x7f2a6400a660 comp rx=0 tx=0).stop 2023-10-14T16:41:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 shutdown_connections 2023-10-14T16:41:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f2a5806aea0 0x7f2a5806d360 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a6c15d410 0x7f2a6c0b2e10 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6c15d030 0x7f2a6c15a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 --2- 172.21.15.29:0/3845152477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a6c0cd260 0x7f2a6c0cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.515+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 >> 172.21.15.29:0/3845152477 conn(0x7f2a6c0cf720 msgr2=0x7f2a6c15c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.516+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 shutdown_connections 2023-10-14T16:41:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.516+0000 7f2a727b0640 1 -- 172.21.15.29:0/3845152477 wait complete. 2023-10-14T16:41:34.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.747+0000 7ff217f47640 1 Processor -- start 2023-10-14T16:41:34.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.747+0000 7ff217f47640 1 -- start start 2023-10-14T16:41:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff217f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 0x7ff2101615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff217f47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff21015d3f0 0x7ff21015d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff217f47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21015e3e0 con 0x7ff2100cd0c0 2023-10-14T16:41:34.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff217f47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2100cd4a0 con 0x7ff21015d3f0 2023-10-14T16:41:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff217f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff21015e100 con 0x7ff2100b0b20 2023-10-14T16:41:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff215ce3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff2100cd0c0 0x7ff2100b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36962/0 (socket says 172.21.15.29:36962) 2023-10-14T16:41:34.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.749+0000 7ff215ce3640 1 -- 172.21.15.29:0/3520915131 learned_addr learned my addr 172.21.15.29:0/3520915131 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff2164e4640 1 --2- 172.21.15.29:0/3520915131 >> v2:172.21.15.186:3300/0 conn(0x7ff21015d3f0 0x7ff21015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff2154e2640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 0x7ff2101615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639639828 0 0) 0x7ff21015e3e0 con 0x7ff2100cd0c0 2023-10-14T16:41:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc003520 con 0x7ff2100cd0c0 2023-10-14T16:41:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015658542 0 0) 0x7ff1fc003520 con 0x7ff2100cd0c0 2023-10-14T16:41:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 msgr2=0x7ff2101615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:34.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 0x7ff2101615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 >> v2:172.21.15.186:3300/0 conn(0x7ff21015d3f0 msgr2=0x7ff21015d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 --2- 172.21.15.29:0/3520915131 >> v2:172.21.15.186:3300/0 conn(0x7ff21015d3f0 0x7ff21015d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff210160b10 con 0x7ff2100cd0c0 2023-10-14T16:41:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff20c0019f0 con 0x7ff2100cd0c0 2023-10-14T16:41:34.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 >> v1:172.21.15.29:6789/0 conn(0x7ff2100cd0c0 legacy=0x7ff2100b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1fc003970 0x7ff1fc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 0x7ff1fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc004d00 0x7ff1fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100cd4a0 con 0x7ff1fc003970 2023-10-14T16:41:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21015e100 con 0x7ff1fc004d00 2023-10-14T16:41:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.750+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1fc00b9b0 con 0x7ff2100b0b20 2023-10-14T16:41:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1fc003970 0x7ff1fc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 -- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 msgr2=0x7ff1fc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2100b0b20 0x7ff1fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 -- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc004d00 msgr2=0x7ff1fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc004d00 0x7ff1fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.751+0000 7ff215ce3640 1 -- 172.21.15.29:0/3520915131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc003520 con 0x7ff1fc003970 2023-10-14T16:41:34.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.752+0000 7ff215ce3640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1fc003970 0x7ff1fc003d90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00a8d0 tx=0x7ff1fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c501d9b764f4ee77 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.752+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c034630 con 0x7ff1fc003970 2023-10-14T16:41:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20c02f070 con 0x7ff1fc003970 2023-10-14T16:41:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff214ce1640 1 -- 172.21.15.29:0/3520915131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c0349d0 con 0x7ff1fc003970 2023-10-14T16:41:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 -- 172.21.15.29:0/3520915131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1fc003970 msgr2=0x7ff1fc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 --2- 172.21.15.29:0/3520915131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1fc003970 0x7ff1fc003d90 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00a8d0 tx=0x7ff1fc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff2164e4640 1 -- 172.21.15.29:0/3520915131 reap_dead start 2023-10-14T16:41:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 -- 172.21.15.29:0/3520915131 shutdown_connections 2023-10-14T16:41:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 -- 172.21.15.29:0/3520915131 >> 172.21.15.29:0/3520915131 conn(0x7ff2100cb410 msgr2=0x7ff2100cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 -- 172.21.15.29:0/3520915131 shutdown_connections 2023-10-14T16:41:34.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.753+0000 7ff217f47640 1 -- 172.21.15.29:0/3520915131 wait complete. 2023-10-14T16:41:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.755+0000 7ff217f47640 1 Processor -- start 2023-10-14T16:41:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.755+0000 7ff217f47640 1 -- start start 2023-10-14T16:41:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 0x7ff2100af4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff210154e70 0x7ff21015e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff210160330 con 0x7ff2100afa10 2023-10-14T16:41:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2101552f0 con 0x7ff1fc003970 2023-10-14T16:41:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff217f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2100b16f0 con 0x7ff210154e70 2023-10-14T16:41:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff2154e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff2154e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55808/0 (socket says 172.21.15.29:55808) 2023-10-14T16:41:34.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.756+0000 7ff2154e2640 1 -- 172.21.15.29:0/1436589420 learned_addr learned my addr 172.21.15.29:0/1436589420 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff215ce3640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 0x7ff2100af4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2164e4640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff210154e70 0x7ff21015e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 -- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff210154e70 msgr2=0x7ff21015e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff210154e70 0x7ff21015e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 -- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 msgr2=0x7ff2100af4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 0x7ff2100af4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 -- 172.21.15.29:0/1436589420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2100b19d0 con 0x7ff2100afa10 2023-10-14T16:41:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff215ce3640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 0x7ff2100af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2154e2640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e7c0 tx=0x7ff20000ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000eed0 con 0x7ff2100afa10 2023-10-14T16:41:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.757+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff20000c940 con 0x7ff2100afa10 2023-10-14T16:41:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.758+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20000ccd0 con 0x7ff2100afa10 2023-10-14T16:41:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.758+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2100b1cb0 con 0x7ff2100afa10 2023-10-14T16:41:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.758+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2100b2270 con 0x7ff2100afa10 2023-10-14T16:41:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.759+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2100b2430 con 0x7ff2100afa10 2023-10-14T16:41:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.760+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff200013470 con 0x7ff2100afa10 2023-10-14T16:41:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.760+0000 7ff2067fc640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 0x7ff1f406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.761+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff2000893b0 con 0x7ff2100afa10 2023-10-14T16:41:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.761+0000 7ff215ce3640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 0x7ff1f406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.761+0000 7ff215ce3640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 0x7ff1f406d360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff20c034000 tx=0x7ff20c009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.765+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff200054710 con 0x7ff2100afa10 2023-10-14T16:41:34.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.964+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff21004e160 con 0x7ff2100afa10 2023-10-14T16:41:34.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.966+0000 7ff2067fc640 1 -- 172.21.15.29:0/1436589420 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v30) v1 ==== 74+0+12845 (secure 0 0 0) 0x7ff200057d30 con 0x7ff2100afa10 2023-10-14T16:41:34.967 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:34.967 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":30,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:30.766942+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 msgr2=0x7ff1f406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 0x7ff1f406d360 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff20c034000 tx=0x7ff20c009030 comp rx=0 tx=0).stop 2023-10-14T16:41:34.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 msgr2=0x7ff210154930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ff20000e7c0 tx=0x7ff20000ec80 comp rx=0 tx=0).stop 2023-10-14T16:41:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 shutdown_connections 2023-10-14T16:41:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff1f406aea0 0x7ff1f406d360 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff210154e70 0x7ff21015e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2100afa10 0x7ff210154930 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 --2- 172.21.15.29:0/1436589420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1fc003970 0x7ff2100af4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 >> 172.21.15.29:0/1436589420 conn(0x7ff2100cb410 msgr2=0x7ff21006f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 shutdown_connections 2023-10-14T16:41:34.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:34.969+0000 7ff217f47640 1 -- 172.21.15.29:0/1436589420 wait complete. 2023-10-14T16:41:34.990 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:41:34.991 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:41:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.224+0000 7ff465b61640 1 Processor -- start 2023-10-14T16:41:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.225+0000 7ff465b61640 1 -- start start 2023-10-14T16:41:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.226+0000 7ff465b61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46015d030 0x7ff4600d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.226+0000 7ff465b61640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 0x7ff460161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.226+0000 7ff465b61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600b5d90 con 0x7ff460161990 2023-10-14T16:41:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.226+0000 7ff465b61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4600d2b50 con 0x7ff46015d030 2023-10-14T16:41:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.226+0000 7ff465b61640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff460160fc0 con 0x7ff46015d410 2023-10-14T16:41:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46015d030 0x7ff4600d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff46015d030 0x7ff4600d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55812/0 (socket says 172.21.15.29:55812) 2023-10-14T16:41:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 -- 172.21.15.29:0/1696216421 learned_addr learned my addr 172.21.15.29:0/1696216421 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 -- 172.21.15.29:0/1696216421 >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 msgr2=0x7ff460161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45effd640 1 --2- 172.21.15.29:0/1696216421 >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 0x7ff460161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 --2- 172.21.15.29:0/1696216421 >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 0x7ff460161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 -- 172.21.15.29:0/1696216421 >> v1:172.21.15.186:6789/0 conn(0x7ff460161990 legacy=0x7ff4600cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:35.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45f7fe640 1 -- 172.21.15.29:0/1696216421 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4600d25d0 con 0x7ff46015d030 2023-10-14T16:41:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.227+0000 7ff45effd640 1 --2- 172.21.15.29:0/1696216421 >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 0x7ff460161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:35.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.228+0000 7ff45f7fe640 1 --2- 172.21.15.29:0/1696216421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff4600d1630 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff460160cd0 tx=0x7ff45002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af18a3a63904ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.229+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1696216421 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45002cb20 con 0x7ff46015d030 2023-10-14T16:41:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.229+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1696216421 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45002cc80 con 0x7ff46015d030 2023-10-14T16:41:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.229+0000 7ff45e7fc640 1 -- 172.21.15.29:0/1696216421 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4500343f0 con 0x7ff46015d030 2023-10-14T16:41:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.229+0000 7ff465b61640 1 -- 172.21.15.29:0/1696216421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 msgr2=0x7ff4600d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.229+0000 7ff465b61640 1 --2- 172.21.15.29:0/1696216421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff4600d1630 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ff460160cd0 tx=0x7ff45002bf80 comp rx=0 tx=0).stop 2023-10-14T16:41:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 -- 172.21.15.29:0/1696216421 shutdown_connections 2023-10-14T16:41:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 --2- 172.21.15.29:0/1696216421 >> v2:172.21.15.186:3300/0 conn(0x7ff46015d410 0x7ff460161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 --2- 172.21.15.29:0/1696216421 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff4600d1630 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 -- 172.21.15.29:0/1696216421 >> 172.21.15.29:0/1696216421 conn(0x7ff4600cf720 msgr2=0x7ff4600cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 -- 172.21.15.29:0/1696216421 shutdown_connections 2023-10-14T16:41:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.230+0000 7ff465b61640 1 -- 172.21.15.29:0/1696216421 wait complete. 2023-10-14T16:41:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.231+0000 7ff465b61640 1 Processor -- start 2023-10-14T16:41:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.232+0000 7ff465b61640 1 -- start start 2023-10-14T16:41:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 0x7ff460159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 0x7ff460159970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff46015b750 con 0x7ff46015d030 2023-10-14T16:41:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4600cc9a0 con 0x7ff460161990 2023-10-14T16:41:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff465b61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4600cb900 con 0x7ff46015d410 2023-10-14T16:41:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55818/0 (socket says 172.21.15.29:55818) 2023-10-14T16:41:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 -- 172.21.15.29:0/4226529779 learned_addr learned my addr 172.21.15.29:0/4226529779 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45effd640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 0x7ff460159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45ffff640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 0x7ff460159970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 -- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 msgr2=0x7ff460159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 0x7ff460159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 -- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 msgr2=0x7ff460159970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 0x7ff460159970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45f7fe640 1 -- 172.21.15.29:0/4226529779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff450009b30 con 0x7ff46015d030 2023-10-14T16:41:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45effd640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 0x7ff460159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.233+0000 7ff45ffff640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 0x7ff460159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff45f7fe640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff450036040 tx=0x7ff450004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff450009510 con 0x7ff46015d030 2023-10-14T16:41:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff45003e450 con 0x7ff46015d030 2023-10-14T16:41:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4600cbbe0 con 0x7ff46015d030 2023-10-14T16:41:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff45003e800 con 0x7ff46015d030 2023-10-14T16:41:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.234+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4600cc1a0 con 0x7ff46015d030 2023-10-14T16:41:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.236+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46004e160 con 0x7ff46015d030 2023-10-14T16:41:35.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.237+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff45003e960 con 0x7ff46015d030 2023-10-14T16:41:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.238+0000 7ff447fff640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 0x7ff44c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.238+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff4500abf30 con 0x7ff46015d030 2023-10-14T16:41:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.238+0000 7ff45effd640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 0x7ff44c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.239+0000 7ff45effd640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 0x7ff44c06d360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff454006fd0 tx=0x7ff454008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.242+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff450077330 con 0x7ff46015d030 2023-10-14T16:41:35.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.439+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff4600c9aa0 con 0x7ff46015d030 2023-10-14T16:41:35.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.440+0000 7ff447fff640 1 -- 172.21.15.29:0/4226529779 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v30) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff45007a950 con 0x7ff46015d030 2023-10-14T16:41:35.441 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:41:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 msgr2=0x7ff44c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 0x7ff44c06d360 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff454006fd0 tx=0x7ff454008040 comp rx=0 tx=0).stop 2023-10-14T16:41:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 msgr2=0x7ff460154840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff450036040 tx=0x7ff450004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 shutdown_connections 2023-10-14T16:41:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff44c06aea0 0x7ff44c06d360 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff460161990 0x7ff460159970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff46015d410 0x7ff460159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 --2- 172.21.15.29:0/4226529779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff46015d030 0x7ff460154840 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 >> 172.21.15.29:0/4226529779 conn(0x7ff4600cf720 msgr2=0x7ff4600b5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 shutdown_connections 2023-10-14T16:41:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.443+0000 7ff465b61640 1 -- 172.21.15.29:0/4226529779 wait complete. 2023-10-14T16:41:35.461 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:41:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.688+0000 7ff510941640 1 Processor -- start 2023-10-14T16:41:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.688+0000 7ff510941640 1 -- start start 2023-10-14T16:41:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff510941640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff50c0cd0c0 0x7ff50c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff510941640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c15e3e0 con 0x7ff50c15db00 2023-10-14T16:41:35.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff510941640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff50c0cd4a0 con 0x7ff50c0b0410 2023-10-14T16:41:35.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff510941640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff50c15e100 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff50a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff50c0cd0c0 0x7ff50c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff50a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff50c0cd0c0 0x7ff50c1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55826/0 (socket says 172.21.15.29:55826) 2023-10-14T16:41:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff50ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff50c15db00 0x7ff50c06f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36974/0 (socket says 172.21.15.29:36974) 2023-10-14T16:41:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.690+0000 7ff50a59c640 1 -- 172.21.15.29:0/3818197985 learned_addr learned my addr 172.21.15.29:0/3818197985 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:35.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.691+0000 7ff50a59c640 1 -- 172.21.15.29:0/3818197985 >> v1:172.21.15.186:6789/0 conn(0x7ff50c0b0410 legacy=0x7ff50c15d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.691+0000 7ff50a59c640 1 -- 172.21.15.29:0/3818197985 >> v1:172.21.15.29:6789/0 conn(0x7ff50c15db00 legacy=0x7ff50c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.691+0000 7ff50a59c640 1 -- 172.21.15.29:0/3818197985 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff50c160b10 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.692+0000 7ff50a59c640 1 --2- 172.21.15.29:0/3818197985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c1615f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11d6d9a5197aaca5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.692+0000 7ff509d9b640 1 -- 172.21.15.29:0/3818197985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc031870 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.692+0000 7ff509d9b640 1 -- 172.21.15.29:0/3818197985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4fc0319d0 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.692+0000 7ff509d9b640 1 -- 172.21.15.29:0/3818197985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4fc031d60 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 -- 172.21.15.29:0/3818197985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 msgr2=0x7ff50c1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 --2- 172.21.15.29:0/3818197985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c1615f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc009870 tx=0x7ff4fc02b910 comp rx=0 tx=0).stop 2023-10-14T16:41:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 -- 172.21.15.29:0/3818197985 shutdown_connections 2023-10-14T16:41:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 --2- 172.21.15.29:0/3818197985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c1615f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 -- 172.21.15.29:0/3818197985 >> 172.21.15.29:0/3818197985 conn(0x7ff50c0cb410 msgr2=0x7ff50c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 -- 172.21.15.29:0/3818197985 shutdown_connections 2023-10-14T16:41:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.693+0000 7ff510941640 1 -- 172.21.15.29:0/3818197985 wait complete. 2023-10-14T16:41:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.695+0000 7ff510941640 1 Processor -- start 2023-10-14T16:41:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.695+0000 7ff510941640 1 -- start start 2023-10-14T16:41:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 0x7ff50c0c4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 0x7ff50c0b1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0c3390 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0c3670 con 0x7ff50c0b0410 2023-10-14T16:41:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff510941640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff50c0c3950 con 0x7ff50c15db00 2023-10-14T16:41:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff503fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55838/0 (socket says 172.21.15.29:55838) 2023-10-14T16:41:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff503fff640 1 -- 172.21.15.29:0/2323313465 learned_addr learned my addr 172.21.15.29:0/2323313465 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff50ad9d640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 0x7ff50c0b1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff50a59c640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 0x7ff50c0c4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.696+0000 7ff503fff640 1 -- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 msgr2=0x7ff50c0b1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff503fff640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 0x7ff50c0b1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff503fff640 1 -- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 msgr2=0x7ff50c0c4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff503fff640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 0x7ff50c0c4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff503fff640 1 -- 172.21.15.29:0/2323313465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4fc0094d0 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff50ad9d640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 0x7ff50c0b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff50a59c640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 0x7ff50c0c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff503fff640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c430 tx=0x7ff4f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4011790 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4f40118f0 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff50c0c3c30 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4f4011ce0 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.697+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff50c157380 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.698+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff50c04e160 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.699+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff4f401c460 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.700+0000 7ff502ffd640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 0x7ff4ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.700+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7ff4f4089320 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.701+0000 7ff50a59c640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 0x7ff4ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.701+0000 7ff50a59c640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 0x7ff4ec06d330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:35.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.705+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff4f4054680 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.905+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7ff50c15bd30 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.906+0000 7ff502ffd640 1 -- 172.21.15.29:0/2323313465 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v30) v1 ==== 84+0+65 (secure 0 0 0) 0x7ff4f4057ca0 con 0x7ff50c0cd0c0 2023-10-14T16:41:35.907 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:35.907 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:41:35.907 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:35.907 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:41:35.908 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:41:35.908 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:35.908 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:41:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.908+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 msgr2=0x7ff4ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.908+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 0x7ff4ec06d330 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff4fc036000 tx=0x7ff4fc03c040 comp rx=0 tx=0).stop 2023-10-14T16:41:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.908+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 msgr2=0x7ff50c156e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.908+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff4f400c430 tx=0x7ff4f400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.908+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 shutdown_connections 2023-10-14T16:41:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff4ec06ae70 0x7ff4ec06d330 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff50c15db00 0x7ff50c0b1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff50c0cd0c0 0x7ff50c156e40 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 --2- 172.21.15.29:0/2323313465 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff50c0b0410 0x7ff50c0c4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 >> 172.21.15.29:0/2323313465 conn(0x7ff50c0cb410 msgr2=0x7ff50c15d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 shutdown_connections 2023-10-14T16:41:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:35.909+0000 7ff510941640 1 -- 172.21.15.29:0/2323313465 wait complete. 2023-10-14T16:41:35.929 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:41:35.930 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:41:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.161+0000 7fca3d7d7640 1 Processor -- start 2023-10-14T16:41:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.161+0000 7fca3d7d7640 1 -- start start 2023-10-14T16:41:36.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.162+0000 7fca3d7d7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca3815ad80 0x7fca3806cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3d7d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca3806d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3d7d7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fca380ad7f0 0x7fca3806b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3d7d7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380c8e70 con 0x7fca3815ad80 2023-10-14T16:41:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3d7d7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca380b4170 con 0x7fca380ad7f0 2023-10-14T16:41:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3d7d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca380adde0 con 0x7fca380cae40 2023-10-14T16:41:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca37fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca3815ad80 0x7fca3806cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca37fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca3815ad80 0x7fca3806cb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55852/0 (socket says 172.21.15.29:55852) 2023-10-14T16:41:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca37fff640 1 -- 172.21.15.29:0/496237431 learned_addr learned my addr 172.21.15.29:0/496237431 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca377fe640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca3806d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.163+0000 7fca3cfd6640 1 --2- 172.21.15.29:0/496237431 >> v2:172.21.15.186:3300/0 conn(0x7fca380ad7f0 0x7fca3806b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.164+0000 7fca37fff640 1 -- 172.21.15.29:0/496237431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 msgr2=0x7fca3806d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.164+0000 7fca37fff640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca3806d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.164+0000 7fca37fff640 1 -- 172.21.15.29:0/496237431 >> v2:172.21.15.186:3300/0 conn(0x7fca380ad7f0 msgr2=0x7fca3806b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.164+0000 7fca37fff640 1 --2- 172.21.15.29:0/496237431 >> v2:172.21.15.186:3300/0 conn(0x7fca380ad7f0 0x7fca3806b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.164+0000 7fca37fff640 1 -- 172.21.15.29:0/496237431 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca380b4410 con 0x7fca3815ad80 2023-10-14T16:41:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.165+0000 7fca37fff640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815ad80 0x7fca3806cb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fca3806f330 tx=0x7fca2802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0085c65a9b66e30 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.165+0000 7fca36ffd640 1 -- 172.21.15.29:0/496237431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2802ca80 con 0x7fca3815ad80 2023-10-14T16:41:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca36ffd640 1 -- 172.21.15.29:0/496237431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2802cbe0 con 0x7fca3815ad80 2023-10-14T16:41:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca36ffd640 1 -- 172.21.15.29:0/496237431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca280343f0 con 0x7fca3815ad80 2023-10-14T16:41:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 -- 172.21.15.29:0/496237431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815ad80 msgr2=0x7fca3806cb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815ad80 0x7fca3806cb70 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fca3806f330 tx=0x7fca2802bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 -- 172.21.15.29:0/496237431 shutdown_connections 2023-10-14T16:41:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/496237431 >> v2:172.21.15.186:3300/0 conn(0x7fca380ad7f0 0x7fca3806b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca3806d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/496237431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca3815ad80 0x7fca3806cb70 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.166+0000 7fca3d7d7640 1 -- 172.21.15.29:0/496237431 >> 172.21.15.29:0/496237431 conn(0x7fca380ce820 msgr2=0x7fca380cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.167+0000 7fca3d7d7640 1 -- 172.21.15.29:0/496237431 shutdown_connections 2023-10-14T16:41:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.167+0000 7fca3d7d7640 1 -- 172.21.15.29:0/496237431 wait complete. 2023-10-14T16:41:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.168+0000 7fca3d7d7640 1 Processor -- start 2023-10-14T16:41:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.169+0000 7fca3d7d7640 1 -- start start 2023-10-14T16:41:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.169+0000 7fca3d7d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3d7d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca380c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3d7d7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 0x7fca380cd200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3d7d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38153080 con 0x7fca380ad7f0 2023-10-14T16:41:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3d7d7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca38151750 con 0x7fca3815ad80 2023-10-14T16:41:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3d7d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca38151fe0 con 0x7fca380cae40 2023-10-14T16:41:36.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55862/0 (socket says 172.21.15.29:55862) 2023-10-14T16:41:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 -- 172.21.15.29:0/387374378 learned_addr learned my addr 172.21.15.29:0/387374378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca377fe640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca380c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3cfd6640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 0x7fca380cd200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 -- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 msgr2=0x7fca380c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca380c7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 -- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 msgr2=0x7fca380cd200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 0x7fca380cd200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca37fff640 1 -- 172.21.15.29:0/387374378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca28009b30 con 0x7fca380ad7f0 2023-10-14T16:41:36.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.170+0000 7fca3cfd6640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 0x7fca380cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca37fff640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca28000f80 tx=0x7fca28004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca28009510 con 0x7fca380ad7f0 2023-10-14T16:41:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca2803e450 con 0x7fca380ad7f0 2023-10-14T16:41:36.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca2803e800 con 0x7fca380ad7f0 2023-10-14T16:41:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca381522c0 con 0x7fca380ad7f0 2023-10-14T16:41:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.171+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca38152880 con 0x7fca380ad7f0 2023-10-14T16:41:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.172+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca3804e9b0 con 0x7fca380ad7f0 2023-10-14T16:41:36.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.173+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fca2802cd50 con 0x7fca380ad7f0 2023-10-14T16:41:36.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.174+0000 7fca34ff9640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 0x7fca2406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:36.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.174+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 4857+0+0 (secure 0 0 0) 0x7fca280abed0 con 0x7fca380ad7f0 2023-10-14T16:41:36.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.175+0000 7fca377fe640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 0x7fca2406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:36.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.175+0000 7fca377fe640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 0x7fca2406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006fd0 tx=0x7fca2c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:36.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.179+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca280772c0 con 0x7fca380ad7f0 2023-10-14T16:41:36.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:36.374+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fca381591f0 con 0x7fca380ad7f0 2023-10-14T16:41:37.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.218+0000 7fca34ff9640 1 -- 172.21.15.29:0/387374378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7fca2807a8e0 con 0x7fca380ad7f0 2023-10-14T16:41:37.219 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:41:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.221+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 msgr2=0x7fca2406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.221+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 0x7fca2406d330 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fca2c006fd0 tx=0x7fca2c008040 comp rx=0 tx=0).stop 2023-10-14T16:41:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 msgr2=0x7fca38151210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:37.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fca28000f80 tx=0x7fca28004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 shutdown_connections 2023-10-14T16:41:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fca2406ae70 0x7fca2406d330 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca3815ad80 0x7fca380cd200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca380cae40 0x7fca380c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 --2- 172.21.15.29:0/387374378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca380ad7f0 0x7fca38151210 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 >> 172.21.15.29:0/387374378 conn(0x7fca380ce820 msgr2=0x7fca3815a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:37.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 shutdown_connections 2023-10-14T16:41:37.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.222+0000 7fca3d7d7640 1 -- 172.21.15.29:0/387374378 wait complete. 2023-10-14T16:41:37.252 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:41:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.481+0000 7f1610adf640 1 Processor -- start 2023-10-14T16:41:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.481+0000 7f1610adf640 1 -- start start 2023-10-14T16:41:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f1610adf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f160c15dc90 0x7f160c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f1610adf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f160c071300 0x7f160c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f1610adf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160c070760 con 0x7f160c0c9230 2023-10-14T16:41:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f1610adf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c0cdac0 con 0x7f160c071300 2023-10-14T16:41:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f1610adf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160c070110 con 0x7f160c15dc90 2023-10-14T16:41:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f160bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f160c071300 0x7f160c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f160bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f160c071300 0x7f160c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55872/0 (socket says 172.21.15.29:55872) 2023-10-14T16:41:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f160bfff640 1 -- 172.21.15.29:0/630234891 learned_addr learned my addr 172.21.15.29:0/630234891 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.483+0000 7f160b7fe640 1 --2- 172.21.15.29:0/630234891 >> v2:172.21.15.186:3300/0 conn(0x7f160c15dc90 0x7f160c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.484+0000 7f160bfff640 1 -- 172.21.15.29:0/630234891 >> v2:172.21.15.186:3300/0 conn(0x7f160c15dc90 msgr2=0x7f160c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.484+0000 7f160bfff640 1 --2- 172.21.15.29:0/630234891 >> v2:172.21.15.186:3300/0 conn(0x7f160c15dc90 0x7f160c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.484+0000 7f160bfff640 1 -- 172.21.15.29:0/630234891 >> v1:172.21.15.186:6789/0 conn(0x7f160c0c9230 legacy=0x7f160c070bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:37.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.484+0000 7f160bfff640 1 -- 172.21.15.29:0/630234891 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160c0703f0 con 0x7f160c071300 2023-10-14T16:41:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.485+0000 7f160bfff640 1 --2- 172.21.15.29:0/630234891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c071300 0x7f160c0ac890 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f160c0719b0 tx=0x7f160402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ef14781695757eb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.485+0000 7f160affd640 1 -- 172.21.15.29:0/630234891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f160402ca80 con 0x7f160c071300 2023-10-14T16:41:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.485+0000 7f160affd640 1 -- 172.21.15.29:0/630234891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f160402cbe0 con 0x7f160c071300 2023-10-14T16:41:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.485+0000 7f160affd640 1 -- 172.21.15.29:0/630234891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16040343f0 con 0x7f160c071300 2023-10-14T16:41:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 -- 172.21.15.29:0/630234891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c071300 msgr2=0x7f160c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 --2- 172.21.15.29:0/630234891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c071300 0x7f160c0ac890 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f160c0719b0 tx=0x7f160402bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 -- 172.21.15.29:0/630234891 shutdown_connections 2023-10-14T16:41:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 --2- 172.21.15.29:0/630234891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c071300 0x7f160c0ac890 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 --2- 172.21.15.29:0/630234891 >> v2:172.21.15.186:3300/0 conn(0x7f160c15dc90 0x7f160c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 -- 172.21.15.29:0/630234891 >> 172.21.15.29:0/630234891 conn(0x7f160c0ce6a0 msgr2=0x7f160c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 -- 172.21.15.29:0/630234891 shutdown_connections 2023-10-14T16:41:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.486+0000 7f1610adf640 1 -- 172.21.15.29:0/630234891 wait complete. 2023-10-14T16:41:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.488+0000 7f1610adf640 1 Processor -- start 2023-10-14T16:41:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.488+0000 7f1610adf640 1 -- start start 2023-10-14T16:41:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f160c071300 0x7f160c150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f160c15dc90 0x7f160c15e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c06b010 con 0x7f160c0c9230 2023-10-14T16:41:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f160c06a760 con 0x7f160c071300 2023-10-14T16:41:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1610adf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160c0b3cd0 con 0x7f160c15dc90 2023-10-14T16:41:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1603fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1603fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55888/0 (socket says 172.21.15.29:55888) 2023-10-14T16:41:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.489+0000 7f1603fff640 1 -- 172.21.15.29:0/106810388 learned_addr learned my addr 172.21.15.29:0/106810388 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f160b7fe640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f160c071300 0x7f160c150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f160bfff640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f160c15dc90 0x7f160c15e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f1603fff640 1 -- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f160c15dc90 msgr2=0x7f160c15e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f1603fff640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f160c15dc90 0x7f160c15e480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f1603fff640 1 -- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f160c071300 msgr2=0x7f160c150830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f1603fff640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f160c071300 0x7f160c150830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.490+0000 7f1603fff640 1 -- 172.21.15.29:0/106810388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1604009b30 con 0x7f160c0c9230 2023-10-14T16:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1603fff640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f15f400c430 tx=0x7f15f400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011790 con 0x7f160c0c9230 2023-10-14T16:41:37.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f40118f0 con 0x7f160c0c9230 2023-10-14T16:41:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f4011ce0 con 0x7f160c0c9230 2023-10-14T16:41:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f160c0b3fb0 con 0x7f160c0c9230 2023-10-14T16:41:37.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.491+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f160c06aca0 con 0x7f160c0c9230 2023-10-14T16:41:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.492+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f160c06b180 con 0x7f160c0c9230 2023-10-14T16:41:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.494+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f15f401c460 con 0x7f160c0c9230 2023-10-14T16:41:37.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.494+0000 7f1608ff9640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 0x7f15ec06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.494+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5227+0+0 (secure 0 0 0) 0x7f15f4089480 con 0x7f160c0c9230 2023-10-14T16:41:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.495+0000 7f160b7fe640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 0x7f15ec06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:37.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.496+0000 7f160b7fe640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 0x7f15ec06d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f15fc006fd0 tx=0x7f15fc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:37.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.503+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f15f4054670 con 0x7f160c0c9230 2023-10-14T16:41:37.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:37.705+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f160c04e9b0 con 0x7f160c0c9230 2023-10-14T16:41:38.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.232+0000 7f1608ff9640 1 -- 172.21.15.29:0/106810388 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v32) v1 ==== 170+0+0 (secure 0 0 0) 0x7f15f4057c90 con 0x7f160c0c9230 2023-10-14T16:41:38.234 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:41:38.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.236+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 msgr2=0x7f15ec06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.236+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 0x7f15ec06d330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f15fc006fd0 tx=0x7f15fc008040 comp rx=0 tx=0).stop 2023-10-14T16:41:38.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.236+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 msgr2=0x7f160c06b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.236+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f15f400c430 tx=0x7f15f400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 shutdown_connections 2023-10-14T16:41:38.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15ec06ae70 0x7f15ec06d330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f160c15dc90 0x7f160c15e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f160c0c9230 0x7f160c06b4a0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 --2- 172.21.15.29:0/106810388 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f160c071300 0x7f160c150830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 >> 172.21.15.29:0/106810388 conn(0x7f160c0ce6a0 msgr2=0x7f160c0c9610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:38.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 shutdown_connections 2023-10-14T16:41:38.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.237+0000 7f1610adf640 1 -- 172.21.15.29:0/106810388 wait complete. 2023-10-14T16:41:38.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:41:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.463+0000 7f048947c640 1 Processor -- start 2023-10-14T16:41:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.463+0000 7f048947c640 1 -- start start 2023-10-14T16:41:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f048947c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f048415ad70 0x7f0484069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f048947c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 0x7f048406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f048947c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04840cb030 con 0x7f04840c9ca0 2023-10-14T16:41:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f048947c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840c91c0 con 0x7f048415ad70 2023-10-14T16:41:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f048947c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04840c94a0 con 0x7f04840caac0 2023-10-14T16:41:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.465+0000 7f0482ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f048415ad70 0x7f0484069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f048415ad70 0x7f0484069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55890/0 (socket says 172.21.15.29:55890) 2023-10-14T16:41:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 -- 172.21.15.29:0/252401888 learned_addr learned my addr 172.21.15.29:0/252401888 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f047bfff640 1 --1- 172.21.15.29:0/252401888 >> v1:172.21.15.29:6789/0 conn(0x7f04840c9ca0 0x7f04840ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36978/0 (socket says 172.21.15.29:36978) 2023-10-14T16:41:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f04837fe640 1 --2- 172.21.15.29:0/252401888 >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 0x7f048406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 -- 172.21.15.29:0/252401888 >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 msgr2=0x7f048406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 --2- 172.21.15.29:0/252401888 >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 0x7f048406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 -- 172.21.15.29:0/252401888 >> v1:172.21.15.29:6789/0 conn(0x7f04840c9ca0 legacy=0x7f04840ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f0482ffd640 1 -- 172.21.15.29:0/252401888 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04840c9780 con 0x7f048415ad70 2023-10-14T16:41:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.466+0000 7f04837fe640 1 --2- 172.21.15.29:0/252401888 >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 0x7f048406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.467+0000 7f0482ffd640 1 --2- 172.21.15.29:0/252401888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f048415ad70 0x7f0484069da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f048406d090 tx=0x7f047402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f432f294a8f3666 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f04827fc640 1 -- 172.21.15.29:0/252401888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474030860 con 0x7f048415ad70 2023-10-14T16:41:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f04827fc640 1 -- 172.21.15.29:0/252401888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04740309c0 con 0x7f048415ad70 2023-10-14T16:41:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f04827fc640 1 -- 172.21.15.29:0/252401888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0474030d50 con 0x7f048415ad70 2023-10-14T16:41:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 -- 172.21.15.29:0/252401888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f048415ad70 msgr2=0x7f0484069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 --2- 172.21.15.29:0/252401888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f048415ad70 0x7f0484069da0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f048406d090 tx=0x7f047402b9a0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 -- 172.21.15.29:0/252401888 shutdown_connections 2023-10-14T16:41:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 --2- 172.21.15.29:0/252401888 >> v2:172.21.15.186:3300/0 conn(0x7f04840caac0 0x7f048406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 --2- 172.21.15.29:0/252401888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f048415ad70 0x7f0484069da0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 -- 172.21.15.29:0/252401888 >> 172.21.15.29:0/252401888 conn(0x7f04840ce9b0 msgr2=0x7f04840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 -- 172.21.15.29:0/252401888 shutdown_connections 2023-10-14T16:41:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.468+0000 7f048947c640 1 -- 172.21.15.29:0/252401888 wait complete. 2023-10-14T16:41:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.470+0000 7f048947c640 1 Processor -- start 2023-10-14T16:41:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.470+0000 7f048947c640 1 -- start start 2023-10-14T16:41:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04840c9ca0 0x7f04841f2250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 0x7f048415b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048415d910 con 0x7f04840caac0 2023-10-14T16:41:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f048414e4f0 con 0x7f04840c9ca0 2023-10-14T16:41:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.471+0000 7f048947c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04841f3180 con 0x7f048415ad70 2023-10-14T16:41:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55894/0 (socket says 172.21.15.29:55894) 2023-10-14T16:41:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 -- 172.21.15.29:0/276240319 learned_addr learned my addr 172.21.15.29:0/276240319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f04837fe640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 0x7f048415b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f0482ffd640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04840c9ca0 0x7f04841f2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 -- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 msgr2=0x7f048415b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 0x7f048415b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 -- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04840c9ca0 msgr2=0x7f04841f2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:38.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04840c9ca0 0x7f04841f2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:38.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 -- 172.21.15.29:0/276240319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04740094d0 con 0x7f04840caac0 2023-10-14T16:41:38.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f04837fe640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 0x7f048415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:38.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.472+0000 7f047bfff640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c4d0 tx=0x7f046c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.473+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c007480 con 0x7f04840caac0 2023-10-14T16:41:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.473+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046c0075e0 con 0x7f04840caac0 2023-10-14T16:41:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.473+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046c0049e0 con 0x7f04840caac0 2023-10-14T16:41:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.473+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f048415bad0 con 0x7f04840caac0 2023-10-14T16:41:38.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.473+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04841f2bc0 con 0x7f04840caac0 2023-10-14T16:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.474+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f048404e160 con 0x7f04840caac0 2023-10-14T16:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.475+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f046c004b40 con 0x7f04840caac0 2023-10-14T16:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.476+0000 7f047b7fe640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 0x7f046406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.476+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 5609+0+0 (secure 0 0 0) 0x7f046c0886d0 con 0x7f04840caac0 2023-10-14T16:41:38.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.477+0000 7f0482ffd640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 0x7f046406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.477+0000 7f0482ffd640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 0x7f046406d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0474036000 tx=0x7f047403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:38.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.481+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f046c0537f0 con 0x7f04840caac0 2023-10-14T16:41:38.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:38.719+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f04840ca450 con 0x7f04840caac0 2023-10-14T16:41:39.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.261+0000 7f047b7fe640 1 -- 172.21.15.29:0/276240319 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v12) v1 ==== 172+0+0 (secure 0 0 0) 0x7f046c056e10 con 0x7f04840caac0 2023-10-14T16:41:39.268 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 4 and data pool 5 2023-10-14T16:41:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 msgr2=0x7f046406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 0x7f046406d330 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f0474036000 tx=0x7f047403b040 comp rx=0 tx=0).stop 2023-10-14T16:41:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 msgr2=0x7f048414dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c4d0 tx=0x7f046c00c990 comp rx=0 tx=0).stop 2023-10-14T16:41:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 shutdown_connections 2023-10-14T16:41:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f046406ae70 0x7f046406d330 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f048415ad70 0x7f048415b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04840caac0 0x7f048414dfb0 secure :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f046c00c4d0 tx=0x7f046c00c990 comp rx=0 tx=0).stop 2023-10-14T16:41:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 --2- 172.21.15.29:0/276240319 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04840c9ca0 0x7f04841f2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.278+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 >> 172.21.15.29:0/276240319 conn(0x7f04840ce9b0 msgr2=0x7f04840c5230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.280+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 shutdown_connections 2023-10-14T16:41:39.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.280+0000 7f048947c640 1 -- 172.21.15.29:0/276240319 wait complete. 2023-10-14T16:41:39.310 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:39.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.496+0000 7f5520a61640 1 Processor -- start 2023-10-14T16:41:39.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.496+0000 7f5520a61640 1 -- start start 2023-10-14T16:41:39.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.497+0000 7f5520a61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159b20 0x7f551c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5520a61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f551c159f00 0x7f551c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5520a61640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f551c06c120 0x7f551c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5520a61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c06f7f0 con 0x7f551c159f00 2023-10-14T16:41:39.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5520a61640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f551c06c5e0 con 0x7f551c06c120 2023-10-14T16:41:39.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5520a61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c06c8c0 con 0x7f551c159b20 2023-10-14T16:41:39.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f551c159f00 0x7f551c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f551c159f00 0x7f551c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55900/0 (socket says 172.21.15.29:55900) 2023-10-14T16:41:39.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 -- 172.21.15.29:0/298920656 learned_addr learned my addr 172.21.15.29:0/298920656 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:39.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f551ad9d640 1 --2- 172.21.15.29:0/298920656 >> v2:172.21.15.186:3300/0 conn(0x7f551c06c120 0x7f551c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f551a59c640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159b20 0x7f551c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 -- 172.21.15.29:0/298920656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159b20 msgr2=0x7f551c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159b20 0x7f551c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 -- 172.21.15.29:0/298920656 >> v2:172.21.15.186:3300/0 conn(0x7f551c06c120 msgr2=0x7f551c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.498+0000 7f5519d9b640 1 --2- 172.21.15.29:0/298920656 >> v2:172.21.15.186:3300/0 conn(0x7f551c06c120 0x7f551c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.499+0000 7f5519d9b640 1 -- 172.21.15.29:0/298920656 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f551c06cba0 con 0x7f551c159f00 2023-10-14T16:41:39.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.500+0000 7f5519d9b640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c159f00 0x7f551c0b0410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f551c0c9270 tx=0x7f551002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86c5d323303c46e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.500+0000 7f551959a640 1 -- 172.21.15.29:0/298920656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5510031860 con 0x7f551c159f00 2023-10-14T16:41:39.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.500+0000 7f551959a640 1 -- 172.21.15.29:0/298920656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55100319c0 con 0x7f551c159f00 2023-10-14T16:41:39.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.500+0000 7f551959a640 1 -- 172.21.15.29:0/298920656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5510031d50 con 0x7f551c159f00 2023-10-14T16:41:39.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 -- 172.21.15.29:0/298920656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c159f00 msgr2=0x7f551c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c159f00 0x7f551c0b0410 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f551c0c9270 tx=0x7f551002b900 comp rx=0 tx=0).stop 2023-10-14T16:41:39.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 -- 172.21.15.29:0/298920656 shutdown_connections 2023-10-14T16:41:39.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 --2- 172.21.15.29:0/298920656 >> v2:172.21.15.186:3300/0 conn(0x7f551c06c120 0x7f551c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c159f00 0x7f551c0b0410 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 --2- 172.21.15.29:0/298920656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159b20 0x7f551c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 -- 172.21.15.29:0/298920656 >> 172.21.15.29:0/298920656 conn(0x7f551c0cb410 msgr2=0x7f551c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:39.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 -- 172.21.15.29:0/298920656 shutdown_connections 2023-10-14T16:41:39.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.501+0000 7f5520a61640 1 -- 172.21.15.29:0/298920656 wait complete. 2023-10-14T16:41:39.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.503+0000 7f5520a61640 1 Processor -- start 2023-10-14T16:41:39.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.503+0000 7f5520a61640 1 -- start start 2023-10-14T16:41:39.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f551c159b20 0x7f551c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159f00 0x7f551c1f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0ae850 con 0x7f551c06c120 2023-10-14T16:41:39.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f551c155d60 con 0x7f551c159b20 2023-10-14T16:41:39.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5520a61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f551c0aea40 con 0x7f551c159f00 2023-10-14T16:41:39.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f551a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f551a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55904/0 (socket says 172.21.15.29:55904) 2023-10-14T16:41:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f551a59c640 1 -- 172.21.15.29:0/19662756 learned_addr learned my addr 172.21.15.29:0/19662756 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f5519d9b640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f551c159b20 0x7f551c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.504+0000 7f551ad9d640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159f00 0x7f551c1f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 -- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159f00 msgr2=0x7f551c1f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159f00 0x7f551c1f36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 -- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f551c159b20 msgr2=0x7f551c155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f551c159b20 0x7f551c155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 -- 172.21.15.29:0/19662756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55100094d0 con 0x7f551c06c120 2023-10-14T16:41:39.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f551a59c640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f550c00d830 tx=0x7f550c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c00b6c0 con 0x7f551c06c120 2023-10-14T16:41:39.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f550c00b820 con 0x7f551c06c120 2023-10-14T16:41:39.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f550c00bbb0 con 0x7f551c06c120 2023-10-14T16:41:39.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.505+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f551c0aed20 con 0x7f551c06c120 2023-10-14T16:41:39.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.506+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f551c0af2e0 con 0x7f551c06c120 2023-10-14T16:41:39.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.507+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f551c04e160 con 0x7f551c06c120 2023-10-14T16:41:39.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.508+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f550c01e460 con 0x7f551c06c120 2023-10-14T16:41:39.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.509+0000 7f550affd640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 0x7f54fc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.509+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f550c089d10 con 0x7f551c06c120 2023-10-14T16:41:39.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.509+0000 7f5519d9b640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 0x7f54fc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.510+0000 7f5519d9b640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 0x7f54fc06d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5510036000 tx=0x7f551003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.513+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f550c08f020 con 0x7f551c06c120 2023-10-14T16:41:39.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.707+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f551c0c9610 con 0x7f551c06c120 2023-10-14T16:41:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.710+0000 7f550affd640 1 -- 172.21.15.29:0/19662756 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16570 (secure 0 0 0) 0x7f550c054db0 con 0x7f551c06c120 2023-10-14T16:41:39.711 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:39.711 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:38.720673+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.712+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 msgr2=0x7f54fc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.712+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 0x7f54fc06d330 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5510036000 tx=0x7f551003c040 comp rx=0 tx=0).stop 2023-10-14T16:41:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.712+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 msgr2=0x7f551c0ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.712+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f550c00d830 tx=0x7f550c00dcf0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 shutdown_connections 2023-10-14T16:41:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f54fc06ae70 0x7f54fc06d330 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f551c159f00 0x7f551c1f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f551c159b20 0x7f551c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 --2- 172.21.15.29:0/19662756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f551c06c120 0x7f551c0ae310 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 >> 172.21.15.29:0/19662756 conn(0x7f551c0cb410 msgr2=0x7f551c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 shutdown_connections 2023-10-14T16:41:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.713+0000 7f5520a61640 1 -- 172.21.15.29:0/19662756 wait complete. 2023-10-14T16:41:39.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:39.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.922+0000 7fae9b26d640 1 Processor -- start 2023-10-14T16:41:39.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.922+0000 7fae9b26d640 1 -- start start 2023-10-14T16:41:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9b26d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae9406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9b26d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fae940ad970 0x7fae9415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9b26d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9406c960 con 0x7fae9415ebc0 2023-10-14T16:41:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9b26d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9415b9d0 con 0x7fae940ad970 2023-10-14T16:41:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9b26d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415bcb0 con 0x7fae9406c040 2023-10-14T16:41:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9aa6c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae9415ebc0 0x7fae940cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36992/0 (socket says 172.21.15.29:36992) 2023-10-14T16:41:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9aa6c640 1 -- 172.21.15.29:0/73822748 learned_addr learned my addr 172.21.15.29:0/73822748 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.924+0000 7fae9a26b640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae9406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99a6a640 1 --2- 172.21.15.29:0/73822748 >> v2:172.21.15.186:3300/0 conn(0x7fae940ad970 0x7fae9415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147058370 0 0) 0x7fae9406c960 con 0x7fae9415ebc0 2023-10-14T16:41:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae80003520 con 0x7fae9415ebc0 2023-10-14T16:41:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3435695309 0 0) 0x7fae80003520 con 0x7fae9415ebc0 2023-10-14T16:41:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 msgr2=0x7fae9406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae9406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 >> v2:172.21.15.186:3300/0 conn(0x7fae940ad970 msgr2=0x7fae9415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 --2- 172.21.15.29:0/73822748 >> v2:172.21.15.186:3300/0 conn(0x7fae940ad970 0x7fae9415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9415bf90 con 0x7fae9415ebc0 2023-10-14T16:41:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fae8c0019f0 con 0x7fae9415ebc0 2023-10-14T16:41:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 >> v1:172.21.15.29:6789/0 conn(0x7fae9415ebc0 legacy=0x7fae940cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae80003970 0x7fae80003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae80004d00 0x7fae80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415b9d0 con 0x7fae80003970 2023-10-14T16:41:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9415bcb0 con 0x7fae80004d00 2023-10-14T16:41:39.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.925+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae8000b980 con 0x7fae9406c040 2023-10-14T16:41:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae80003970 0x7fae80003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae99a6a640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 -- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 msgr2=0x7fae800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9406c040 0x7fae800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 -- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae80004d00 msgr2=0x7fae80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae80004d00 0x7fae80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.926+0000 7fae9a26b640 1 -- 172.21.15.29:0/73822748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae80003520 con 0x7fae80003970 2023-10-14T16:41:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.927+0000 7fae9a26b640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae80003970 0x7fae80003d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ac0 tx=0x7fae800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43b6bbcc14ecf906 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.927+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9002c6b0 con 0x7fae80003970 2023-10-14T16:41:39.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.927+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9002c810 con 0x7fae80003970 2023-10-14T16:41:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.927+0000 7fae99269640 1 -- 172.21.15.29:0/73822748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9002cba0 con 0x7fae80003970 2023-10-14T16:41:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 -- 172.21.15.29:0/73822748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae80003970 msgr2=0x7fae80003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 --2- 172.21.15.29:0/73822748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae80003970 0x7fae80003d90 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fae90009ac0 tx=0x7fae800057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9aa6c640 1 -- 172.21.15.29:0/73822748 reap_dead start 2023-10-14T16:41:39.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 -- 172.21.15.29:0/73822748 shutdown_connections 2023-10-14T16:41:39.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 -- 172.21.15.29:0/73822748 >> 172.21.15.29:0/73822748 conn(0x7fae940ce6a0 msgr2=0x7fae940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:39.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 -- 172.21.15.29:0/73822748 shutdown_connections 2023-10-14T16:41:39.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.928+0000 7fae9b26d640 1 -- 172.21.15.29:0/73822748 wait complete. 2023-10-14T16:41:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.930+0000 7fae9b26d640 1 Processor -- start 2023-10-14T16:41:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.930+0000 7fae9b26d640 1 -- start start 2023-10-14T16:41:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 0x7fae940c7e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 0x7fae940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940c8370 con 0x7fae94149da0 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae940ae0c0 con 0x7fae940af440 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9b26d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae940ae3a0 con 0x7fae80003970 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9aa6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9aa6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55910/0 (socket says 172.21.15.29:55910) 2023-10-14T16:41:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9aa6c640 1 -- 172.21.15.29:0/1094864767 learned_addr learned my addr 172.21.15.29:0/1094864767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae99a6a640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 0x7fae940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.931+0000 7fae9a26b640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 0x7fae940c7e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 -- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 msgr2=0x7fae940c7e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 0x7fae940c7e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 -- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 msgr2=0x7fae940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 0x7fae940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 -- 172.21.15.29:0/1094864767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae940ae680 con 0x7fae94149da0 2023-10-14T16:41:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae99a6a640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 0x7fae940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:39.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9a26b640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 0x7fae940c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9aa6c640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fae9406d2e0 tx=0x7fae8c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00cea0 con 0x7fae94149da0 2023-10-14T16:41:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c00a610 con 0x7fae94149da0 2023-10-14T16:41:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae940ae960 con 0x7fae94149da0 2023-10-14T16:41:39.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c00a9e0 con 0x7fae94149da0 2023-10-14T16:41:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.932+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae940aef20 con 0x7fae94149da0 2023-10-14T16:41:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.933+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae940af140 con 0x7fae94149da0 2023-10-14T16:41:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.935+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fae8c00ab40 con 0x7fae94149da0 2023-10-14T16:41:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.936+0000 7fae8affd640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 0x7fae7806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:39.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.936+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fae8c088f20 con 0x7fae94149da0 2023-10-14T16:41:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.936+0000 7fae9a26b640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 0x7fae7806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.936+0000 7fae9a26b640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 0x7fae7806d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae90036000 tx=0x7fae90030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:39.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:39.940+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae8c057a50 con 0x7fae94149da0 2023-10-14T16:41:40.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.127+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae940cc550 con 0x7fae94149da0 2023-10-14T16:41:40.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.129+0000 7fae8affd640 1 -- 172.21.15.29:0/1094864767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16570 (secure 0 0 0) 0x7fae8c01a020 con 0x7fae94149da0 2023-10-14T16:41:40.130 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:40.130 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":33,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:38.720673+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:40.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.131+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 msgr2=0x7fae7806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.131+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 0x7fae7806d330 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fae90036000 tx=0x7fae90030000 comp rx=0 tx=0).stop 2023-10-14T16:41:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.131+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 msgr2=0x7fae940acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.131+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fae9406d2e0 tx=0x7fae8c00cc50 comp rx=0 tx=0).stop 2023-10-14T16:41:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 shutdown_connections 2023-10-14T16:41:40.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fae7806ae70 0x7fae7806d330 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae94149da0 0x7fae940acdd0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae940af440 0x7fae940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 --2- 172.21.15.29:0/1094864767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae80003970 0x7fae940c7e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 >> 172.21.15.29:0/1094864767 conn(0x7fae940ce6a0 msgr2=0x7fae9415d8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 shutdown_connections 2023-10-14T16:41:40.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.132+0000 7fae9b26d640 1 -- 172.21.15.29:0/1094864767 wait complete. 2023-10-14T16:41:40.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:41:40.184 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-14T16:41:40.183+0000 7f2f01da8640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:41:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.363+0000 7fe1b5885640 1 Processor -- start 2023-10-14T16:41:40.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.364+0000 7fe1b5885640 1 -- start start 2023-10-14T16:41:40.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b5885640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1b006c040 0x7fe1b006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b5885640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b015f4a0 con 0x7fe1b00ad970 2023-10-14T16:41:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b5885640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1b006c960 con 0x7fe1b00cda80 2023-10-14T16:41:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b5885640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1b015b9d0 con 0x7fe1b006c040 2023-10-14T16:41:40.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b4883640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1b006c040 0x7fe1b006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b4883640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe1b006c040 0x7fe1b006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55920/0 (socket says 172.21.15.29:55920) 2023-10-14T16:41:40.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.365+0000 7fe1b4883640 1 -- 172.21.15.29:0/3696693177 learned_addr learned my addr 172.21.15.29:0/3696693177 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.366+0000 7fe1b4883640 1 -- 172.21.15.29:0/3696693177 >> v1:172.21.15.186:6789/0 conn(0x7fe1b00cda80 legacy=0x7fe1b015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.366+0000 7fe1b4883640 1 -- 172.21.15.29:0/3696693177 >> v1:172.21.15.29:6789/0 conn(0x7fe1b00ad970 legacy=0x7fe1b015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.366+0000 7fe1af7fe640 1 -- 172.21.15.29:0/3696693177 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033762519 0 0) 0x7fe1b015f4a0 con 0x7fe1b00ad970 2023-10-14T16:41:40.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.366+0000 7fe1b4883640 1 -- 172.21.15.29:0/3696693177 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1b015bcb0 con 0x7fe1b006c040 2023-10-14T16:41:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.367+0000 7fe1b4883640 1 --2- 172.21.15.29:0/3696693177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b006c040 0x7fe1b006c420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fb7a62d2520ba6f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.367+0000 7fe1af7fe640 1 -- 172.21.15.29:0/3696693177 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0030850 con 0x7fe1b006c040 2023-10-14T16:41:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.367+0000 7fe1af7fe640 1 -- 172.21.15.29:0/3696693177 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a00309b0 con 0x7fe1b006c040 2023-10-14T16:41:40.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.367+0000 7fe1af7fe640 1 -- 172.21.15.29:0/3696693177 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a0030d40 con 0x7fe1b006c040 2023-10-14T16:41:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 -- 172.21.15.29:0/3696693177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b006c040 msgr2=0x7fe1b006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 --2- 172.21.15.29:0/3696693177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b006c040 0x7fe1b006c420 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0009870 tx=0x7fe1a002b990 comp rx=0 tx=0).stop 2023-10-14T16:41:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 -- 172.21.15.29:0/3696693177 shutdown_connections 2023-10-14T16:41:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 --2- 172.21.15.29:0/3696693177 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b006c040 0x7fe1b006c420 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 -- 172.21.15.29:0/3696693177 >> 172.21.15.29:0/3696693177 conn(0x7fe1b00ce6a0 msgr2=0x7fe1b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 -- 172.21.15.29:0/3696693177 shutdown_connections 2023-10-14T16:41:40.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.368+0000 7fe1b5885640 1 -- 172.21.15.29:0/3696693177 wait complete. 2023-10-14T16:41:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.370+0000 7fe1b5885640 1 Processor -- start 2023-10-14T16:41:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.370+0000 7fe1b5885640 1 -- start start 2023-10-14T16:41:40.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1b006c040 0x7fe1b014dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1b00ad970 0x7fe1b00b2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b3c00 con 0x7fe1b00cda80 2023-10-14T16:41:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b3640 con 0x7fe1b006c040 2023-10-14T16:41:40.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5885640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1b00b3920 con 0x7fe1b00ad970 2023-10-14T16:41:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55922/0 (socket says 172.21.15.29:55922) 2023-10-14T16:41:40.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 -- 172.21.15.29:0/729363077 learned_addr learned my addr 172.21.15.29:0/729363077 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 -- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1b00ad970 msgr2=0x7fe1b00b2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1b00ad970 0x7fe1b00b2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b4883640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1b006c040 0x7fe1b014dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 -- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1b006c040 msgr2=0x7fe1b014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:40.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1b006c040 0x7fe1b014dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.371+0000 7fe1b5084640 1 -- 172.21.15.29:0/729363077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00094d0 con 0x7fe1b00cda80 2023-10-14T16:41:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1b5084640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800c410 tx=0x7fe1a800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8011790 con 0x7fe1b00cda80 2023-10-14T16:41:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1a80118f0 con 0x7fe1b00cda80 2023-10-14T16:41:40.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1b00c67a0 con 0x7fe1b00cda80 2023-10-14T16:41:40.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1a8011ce0 con 0x7fe1b00cda80 2023-10-14T16:41:40.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.372+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1b00b3230 con 0x7fe1b00cda80 2023-10-14T16:41:40.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.373+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1b006b330 con 0x7fe1b00cda80 2023-10-14T16:41:40.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.374+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe1a801c460 con 0x7fe1b00cda80 2023-10-14T16:41:40.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.376+0000 7fe1ad7fa640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 0x7fe19c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.376+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe1a8089300 con 0x7fe1b00cda80 2023-10-14T16:41:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.376+0000 7fe1b4883640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 0x7fe19c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:40.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.377+0000 7fe1b4883640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 0x7fe19c06d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0036000 tx=0x7fe1a003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:40.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.380+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe1a80543a0 con 0x7fe1b00cda80 2023-10-14T16:41:40.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:40.618+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe1b00c6cc0 con 0x7fe1b00cda80 2023-10-14T16:41:41.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:41:41.117 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:41:41.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.270+0000 7fe1ad7fa640 1 -- 172.21.15.29:0/729363077 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe1b00c6cc0 con 0x7fe1b00cda80 2023-10-14T16:41:41.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 msgr2=0x7fe19c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 0x7fe19c06d330 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fe1a0036000 tx=0x7fe1a003b040 comp rx=0 tx=0).stop 2023-10-14T16:41:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 msgr2=0x7fe1b00c29c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fe1a800c410 tx=0x7fe1a800c8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 shutdown_connections 2023-10-14T16:41:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe19c06ae70 0x7fe19c06d330 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe1b00cda80 0x7fe1b00c29c0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe1b00ad970 0x7fe1b00b2cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 --2- 172.21.15.29:0/729363077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe1b006c040 0x7fe1b014dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 >> 172.21.15.29:0/729363077 conn(0x7fe1b00ce6a0 msgr2=0x7fe1b00c9290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:41.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 shutdown_connections 2023-10-14T16:41:41.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.274+0000 7fe1b5885640 1 -- 172.21.15.29:0/729363077 wait complete. 2023-10-14T16:41:41.297 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:41:41.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:41:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.575+0000 7fbbdbeec640 1 Processor -- start 2023-10-14T16:41:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.575+0000 7fbbdbeec640 1 -- start start 2023-10-14T16:41:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbdbeec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbdbeec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbbd4159e90 0x7fbbd406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbdbeec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd40cd1f0 con 0x7fbbd406c140 2023-10-14T16:41:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbdbeec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbd415d890 con 0x7fbbd4159e90 2023-10-14T16:41:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbdbeec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40cd4d0 con 0x7fbbd4159ab0 2023-10-14T16:41:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbd9487640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbbd4159e90 0x7fbbd406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbda489640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbbd406c140 0x7fbbd415dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45164/0 (socket says 172.21.15.29:45164) 2023-10-14T16:41:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.577+0000 7fbbda489640 1 -- 172.21.15.29:0/2484801165 learned_addr learned my addr 172.21.15.29:0/2484801165 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd9c88640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd9487640 1 -- 172.21.15.29:0/2484801165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 msgr2=0x7fbbd40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd9487640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd9487640 1 -- 172.21.15.29:0/2484801165 >> v1:172.21.15.29:6789/0 conn(0x7fbbd406c140 legacy=0x7fbbd415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd9487640 1 -- 172.21.15.29:0/2484801165 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd40cd7b0 con 0x7fbbd4159e90 2023-10-14T16:41:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.578+0000 7fbbd8c86640 1 -- 172.21.15.29:0/2484801165 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183504319 0 0) 0x7fbbd40cd1f0 con 0x7fbbd406c140 2023-10-14T16:41:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.579+0000 7fbbd9487640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd406fc50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40097d0 tx=0x7fbbc402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a729d937bc42e933 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbd8c86640 1 -- 172.21.15.29:0/2484801165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc40316f0 con 0x7fbbd4159e90 2023-10-14T16:41:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbd8c86640 1 -- 172.21.15.29:0/2484801165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc4031850 con 0x7fbbd4159e90 2023-10-14T16:41:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbd8c86640 1 -- 172.21.15.29:0/2484801165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc4031be0 con 0x7fbbd4159e90 2023-10-14T16:41:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 -- 172.21.15.29:0/2484801165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 msgr2=0x7fbbd406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd406fc50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7fbbc40097d0 tx=0x7fbbc402b790 comp rx=0 tx=0).stop 2023-10-14T16:41:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 -- 172.21.15.29:0/2484801165 shutdown_connections 2023-10-14T16:41:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd406fc50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/2484801165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 -- 172.21.15.29:0/2484801165 >> 172.21.15.29:0/2484801165 conn(0x7fbbd40ce9b0 msgr2=0x7fbbd40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:41.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 -- 172.21.15.29:0/2484801165 shutdown_connections 2023-10-14T16:41:41.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.580+0000 7fbbdbeec640 1 -- 172.21.15.29:0/2484801165 wait complete. 2023-10-14T16:41:41.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.582+0000 7fbbdbeec640 1 Processor -- start 2023-10-14T16:41:41.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.582+0000 7fbbdbeec640 1 -- start start 2023-10-14T16:41:41.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbd406c140 0x7fbbd40c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd4148080 con 0x7fbbd4159e90 2023-10-14T16:41:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40c81d0 con 0x7fbbd406c140 2023-10-14T16:41:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.583+0000 7fbbdbeec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd40c84b0 con 0x7fbbd4159ab0 2023-10-14T16:41:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39310/0 (socket says 172.21.15.29:39310) 2023-10-14T16:41:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 -- 172.21.15.29:0/3395561783 learned_addr learned my addr 172.21.15.29:0/3395561783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbd9c88640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbd406c140 0x7fbbd40c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbd9487640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 -- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 msgr2=0x7fbbd40c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40c74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 -- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbd406c140 msgr2=0x7fbbd40c7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbd406c140 0x7fbbd40c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 -- 172.21.15.29:0/3395561783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc4009400 con 0x7fbbd4159e90 2023-10-14T16:41:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.584+0000 7fbbda489640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00c700 tx=0x7fbbcc00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:41.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.585+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc00ce10 con 0x7fbbd4159e90 2023-10-14T16:41:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.585+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbcc00a640 con 0x7fbbd4159e90 2023-10-14T16:41:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.585+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbcc00aa70 con 0x7fbbd4159e90 2023-10-14T16:41:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.585+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd40c6ce0 con 0x7fbbd4159e90 2023-10-14T16:41:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.585+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd40aefb0 con 0x7fbbd4159e90 2023-10-14T16:41:41.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.586+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd40c7200 con 0x7fbbd4159e90 2023-10-14T16:41:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.587+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fbbcc00abd0 con 0x7fbbd4159e90 2023-10-14T16:41:41.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.588+0000 7fbbca7fc640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 0x7fbbb806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.588+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fbbcc088b50 con 0x7fbbd4159e90 2023-10-14T16:41:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.588+0000 7fbbd9c88640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 0x7fbbb806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.589+0000 7fbbd9c88640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 0x7fbbb806d360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006fd0 tx=0x7fbbd0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:41.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.592+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbd40c7200 con 0x7fbbd4159e90 2023-10-14T16:41:41.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:41.826+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fbbd40c7200 con 0x7fbbd4159e90 2023-10-14T16:41:42.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.281+0000 7fbbca7fc640 1 -- 172.21.15.29:0/3395561783 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) v1 ==== 122+0+0 (secure 0 0 0) 0x7fbbd40c7200 con 0x7fbbd4159e90 2023-10-14T16:41:42.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.284+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 msgr2=0x7fbbb806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.284+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 0x7fbbb806d360 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbbd0006fd0 tx=0x7fbbd0008040 comp rx=0 tx=0).stop 2023-10-14T16:41:42.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.284+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 msgr2=0x7fbbd4147b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.284+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fbbcc00c700 tx=0x7fbbcc00cbc0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 shutdown_connections 2023-10-14T16:41:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fbbb806aea0 0x7fbbb806d360 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbd4159e90 0x7fbbd4147b40 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbd4159ab0 0x7fbbd40c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 --2- 172.21.15.29:0/3395561783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbd406c140 0x7fbbd40c7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 >> 172.21.15.29:0/3395561783 conn(0x7fbbd40ce9b0 msgr2=0x7fbbd40ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 shutdown_connections 2023-10-14T16:41:42.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.285+0000 7fbbdbeec640 1 -- 172.21.15.29:0/3395561783 wait complete. 2023-10-14T16:41:42.307 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:41:42.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:42.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.513+0000 7f1ac01b0640 1 Processor -- start 2023-10-14T16:41:42.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.513+0000 7f1ac01b0640 1 -- start start 2023-10-14T16:41:42.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.515+0000 7f1ac01b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1ab815a970 0x7f1ab806ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.515+0000 7f1ac01b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab8070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.515+0000 7f1ac01b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ab8070660 con 0x7f1ab8071360 2023-10-14T16:41:42.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.515+0000 7f1ac01b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ab815dc60 con 0x7f1ab815a970 2023-10-14T16:41:42.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.515+0000 7f1ac01b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab815df40 con 0x7f1ab815d430 2023-10-14T16:41:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.516+0000 7f1abe74d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1ab8071360 0x7f1ab80cd880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45166/0 (socket says 172.21.15.29:45166) 2023-10-14T16:41:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.516+0000 7f1abe74d640 1 -- 172.21.15.29:0/3331591081 learned_addr learned my addr 172.21.15.29:0/3331591081 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.516+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/3331591081 >> v2:172.21.15.186:3300/0 conn(0x7f1ab815a970 0x7f1ab806ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.516+0000 7f1abd74b640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab8070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.516+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797425321 0 0) 0x7f1ab8070660 con 0x7f1ab8071360 2023-10-14T16:41:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1aa4003520 con 0x7f1ab8071360 2023-10-14T16:41:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abdf4c640 1 -- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 msgr2=0x7f1ab8070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab8070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3919678381 0 0) 0x7f1aa4003520 con 0x7f1ab8071360 2023-10-14T16:41:42.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abdf4c640 1 -- 172.21.15.29:0/3331591081 >> v1:172.21.15.29:6789/0 conn(0x7f1ab8071360 legacy=0x7f1ab80cd880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.517+0000 7f1abdf4c640 1 -- 172.21.15.29:0/3331591081 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab815e220 con 0x7f1ab815a970 2023-10-14T16:41:42.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.519+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab815a970 0x7f1ab806ecd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40097a0 tx=0x7f1ab402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd70ed7c0c2f3198 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:42.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.519+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab00019f0 con 0x7f1ab815a970 2023-10-14T16:41:42.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.519+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab402e050 con 0x7f1ab815a970 2023-10-14T16:41:42.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.519+0000 7f1abcf4a640 1 -- 172.21.15.29:0/3331591081 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab4030730 con 0x7f1ab815a970 2023-10-14T16:41:42.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 -- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab815a970 msgr2=0x7f1ab806ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab815a970 0x7f1ab806ecd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40097a0 tx=0x7f1ab402b790 comp rx=0 tx=0).stop 2023-10-14T16:41:42.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 -- 172.21.15.29:0/3331591081 shutdown_connections 2023-10-14T16:41:42.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab8070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/3331591081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab815a970 0x7f1ab806ecd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 -- 172.21.15.29:0/3331591081 >> 172.21.15.29:0/3331591081 conn(0x7f1ab80cfaa0 msgr2=0x7f1ab80cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:42.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 -- 172.21.15.29:0/3331591081 shutdown_connections 2023-10-14T16:41:42.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.520+0000 7f1ac01b0640 1 -- 172.21.15.29:0/3331591081 wait complete. 2023-10-14T16:41:42.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.523+0000 7f1ac01b0640 1 Processor -- start 2023-10-14T16:41:42.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.523+0000 7f1ac01b0640 1 -- start start 2023-10-14T16:41:42.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab80c7610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 0x7f1ab8156f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab8153820 con 0x7f1ab815a970 2023-10-14T16:41:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80c31c0 con 0x7f1ab80c7b50 2023-10-14T16:41:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1ac01b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ab80b04d0 con 0x7f1ab815d430 2023-10-14T16:41:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39314/0 (socket says 172.21.15.29:39314) 2023-10-14T16:41:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 -- 172.21.15.29:0/1963418701 learned_addr learned my addr 172.21.15.29:0/1963418701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 -- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 msgr2=0x7f1ab80c7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab80c7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 -- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 msgr2=0x7f1ab8156f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abe74d640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 0x7f1ab8156f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 0x7f1ab8156f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abdf4c640 1 -- 172.21.15.29:0/1963418701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ab4009400 con 0x7f1ab815a970 2023-10-14T16:41:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.525+0000 7f1abe74d640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 0x7f1ab8156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1abdf4c640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40098c0 tx=0x7f1ab40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab403b810 con 0x7f1ab815a970 2023-10-14T16:41:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ab403b970 con 0x7f1ab815a970 2023-10-14T16:41:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ab403bd20 con 0x7f1ab815a970 2023-10-14T16:41:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ab80af8c0 con 0x7f1ab815a970 2023-10-14T16:41:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.526+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ab80afe80 con 0x7f1ab815a970 2023-10-14T16:41:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.528+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ab804e160 con 0x7f1ab815a970 2023-10-14T16:41:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.529+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1ab4039040 con 0x7f1ab815a970 2023-10-14T16:41:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.529+0000 7f1aae7fc640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 0x7f1a9c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.530+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1ab40aae70 con 0x7f1ab815a970 2023-10-14T16:41:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.530+0000 7f1abd74b640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 0x7f1a9c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.531+0000 7f1abd74b640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 0x7f1a9c06d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80c9570 tx=0x7f1aa8009500 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.534+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ab4075df0 con 0x7f1ab815a970 2023-10-14T16:41:42.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.767+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ab815b9d0 con 0x7f1ab815a970 2023-10-14T16:41:42.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.768+0000 7f1aae7fc640 1 -- 172.21.15.29:0/1963418701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f1ab4037090 con 0x7f1ab815a970 2023-10-14T16:41:42.769 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:42.769 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:41:42.770 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:41:42.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 msgr2=0x7f1a9c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 0x7f1a9c06d330 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ab80c9570 tx=0x7f1aa8009500 comp rx=0 tx=0).stop 2023-10-14T16:41:42.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 msgr2=0x7f1ab80c5c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40098c0 tx=0x7f1ab40046c0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 shutdown_connections 2023-10-14T16:41:42.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a9c06ae70 0x7f1a9c06d330 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ab80c7b50 0x7f1ab8156f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ab815d430 0x7f1ab80c7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 --2- 172.21.15.29:0/1963418701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ab815a970 0x7f1ab80c5c60 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.770+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 >> 172.21.15.29:0/1963418701 conn(0x7f1ab80cfaa0 msgr2=0x7f1ab815bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:42.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.771+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 shutdown_connections 2023-10-14T16:41:42.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.771+0000 7f1ac01b0640 1 -- 172.21.15.29:0/1963418701 wait complete. 2023-10-14T16:41:42.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:41:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.991+0000 7f66356e0640 1 Processor -- start 2023-10-14T16:41:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.991+0000 7f66356e0640 1 -- start start 2023-10-14T16:41:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.992+0000 7f66356e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 0x7f663015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.992+0000 7f66356e0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f66300ce390 0x7f6630070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:42.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.992+0000 7f66356e0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f663015ed60 con 0x7f663015aaa0 2023-10-14T16:41:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.992+0000 7f66356e0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f663006b810 con 0x7f66300ce390 2023-10-14T16:41:42.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.992+0000 7f66356e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66300707e0 con 0x7f663015ae80 2023-10-14T16:41:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.993+0000 7f662e7fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 0x7f663015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.993+0000 7f662f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f66300ce390 0x7f6630070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.993+0000 7f662f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f66300ce390 0x7f6630070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:47016/0 (socket says 172.21.15.29:47016) 2023-10-14T16:41:42.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.993+0000 7f662f7fe640 1 -- 172.21.15.29:0/2176584095 learned_addr learned my addr 172.21.15.29:0/2176584095 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.993+0000 7f662e7fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 0x7f663015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:57924/0 (socket says 172.21.15.29:57924) 2023-10-14T16:41:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.994+0000 7f662f7fe640 1 -- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 msgr2=0x7f663015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.994+0000 7f662f7fe640 1 --2- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 0x7f663015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.994+0000 7f662f7fe640 1 -- 172.21.15.29:0/2176584095 >> v1:172.21.15.186:6789/0 conn(0x7f663015aaa0 legacy=0x7f66300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.994+0000 7f662f7fe640 1 -- 172.21.15.29:0/2176584095 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f663015ef90 con 0x7f66300ce390 2023-10-14T16:41:42.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.996+0000 7f662f7fe640 1 --2- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66300ce390 0x7f6630070f50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6628009870 tx=0x7f662802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10d6847623caff08 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.996+0000 7f662dffb640 1 -- 172.21.15.29:0/2176584095 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628030840 con 0x7f66300ce390 2023-10-14T16:41:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.996+0000 7f662dffb640 1 -- 172.21.15.29:0/2176584095 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66280309a0 con 0x7f66300ce390 2023-10-14T16:41:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.996+0000 7f662dffb640 1 -- 172.21.15.29:0/2176584095 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628030d30 con 0x7f66300ce390 2023-10-14T16:41:42.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 -- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66300ce390 msgr2=0x7f6630070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 --2- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66300ce390 0x7f6630070f50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f6628009870 tx=0x7f662802b980 comp rx=0 tx=0).stop 2023-10-14T16:41:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 -- 172.21.15.29:0/2176584095 shutdown_connections 2023-10-14T16:41:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 --2- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f663015ae80 0x7f663015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 --2- 172.21.15.29:0/2176584095 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66300ce390 0x7f6630070f50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:42.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 -- 172.21.15.29:0/2176584095 >> 172.21.15.29:0/2176584095 conn(0x7f66300cb410 msgr2=0x7f66300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 -- 172.21.15.29:0/2176584095 shutdown_connections 2023-10-14T16:41:43.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.997+0000 7f66356e0640 1 -- 172.21.15.29:0/2176584095 wait complete. 2023-10-14T16:41:43.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.999+0000 7f66356e0640 1 Processor -- start 2023-10-14T16:41:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:42.999+0000 7f66356e0640 1 -- start start 2023-10-14T16:41:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66300ce390 0x7f663015de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f663015aaa0 0x7f66301f5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663015e3a0 con 0x7f663015ae80 2023-10-14T16:41:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66301468e0 con 0x7f663015aaa0 2023-10-14T16:41:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f66356e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6630146bc0 con 0x7f66300ce390 2023-10-14T16:41:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f662f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f662f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39318/0 (socket says 172.21.15.29:39318) 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f662f7fe640 1 -- 172.21.15.29:0/3932056709 learned_addr learned my addr 172.21.15.29:0/3932056709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f662effd640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66300ce390 0x7f663015de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.000+0000 7f662e7fc640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f663015aaa0 0x7f66301f5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 -- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66300ce390 msgr2=0x7f663015de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66300ce390 0x7f663015de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 -- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f663015aaa0 msgr2=0x7f66301f5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f663015aaa0 0x7f66301f5130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 -- 172.21.15.29:0/3932056709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66280094d0 con 0x7f663015ae80 2023-10-14T16:41:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f662f7fe640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6628009870 tx=0x7f6628004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628039a90 con 0x7f663015ae80 2023-10-14T16:41:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6628039c10 con 0x7f663015ae80 2023-10-14T16:41:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.001+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6628043410 con 0x7f663015ae80 2023-10-14T16:41:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.002+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6630146ea0 con 0x7f663015ae80 2023-10-14T16:41:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.002+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6630147460 con 0x7f663015ae80 2023-10-14T16:41:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.003+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f663004e160 con 0x7f663015ae80 2023-10-14T16:41:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.005+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f662802f390 con 0x7f663015ae80 2023-10-14T16:41:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.005+0000 7f66277fe640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 0x7f661006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.006+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f66280ab860 con 0x7f663015ae80 2023-10-14T16:41:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.006+0000 7f662effd640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 0x7f661006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.007+0000 7f662effd640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 0x7f661006d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f66200012e0 tx=0x7f6620009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.010+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66280767e0 con 0x7f663015ae80 2023-10-14T16:41:43.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.207+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66300cdc60 con 0x7f663015ae80 2023-10-14T16:41:43.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.210+0000 7f66277fe640 1 -- 172.21.15.29:0/3932056709 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v34) v1 ==== 74+0+16557 (secure 0 0 0) 0x7f6628034090 con 0x7f663015ae80 2023-10-14T16:41:43.211 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:43.211 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:41:40.259072+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"34","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:41:43.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.212+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 msgr2=0x7f661006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.212+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 0x7f661006d330 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f66200012e0 tx=0x7f6620009040 comp rx=0 tx=0).stop 2023-10-14T16:41:43.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.212+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 msgr2=0x7f663015b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.212+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f6628009870 tx=0x7f6628004730 comp rx=0 tx=0).stop 2023-10-14T16:41:43.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 shutdown_connections 2023-10-14T16:41:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f661006ae70 0x7f661006d330 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f663015ae80 0x7f663015b470 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f663015aaa0 0x7f66301f5130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 --2- 172.21.15.29:0/3932056709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66300ce390 0x7f663015de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 >> 172.21.15.29:0/3932056709 conn(0x7f66300cb410 msgr2=0x7f663006f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:43.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 shutdown_connections 2023-10-14T16:41:43.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.213+0000 7f66356e0640 1 -- 172.21.15.29:0/3932056709 wait complete. 2023-10-14T16:41:43.233 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:41:43.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:41:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.445+0000 7fb3b1021640 1 Processor -- start 2023-10-14T16:41:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.445+0000 7fb3b1021640 1 -- start start 2023-10-14T16:41:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.446+0000 7fb3b1021640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.446+0000 7fb3b1021640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac15d910 con 0x7fb3ac070b60 2023-10-14T16:41:43.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.446+0000 7fb3b1021640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac15dbf0 con 0x7fb3ac159950 2023-10-14T16:41:43.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.446+0000 7fb3b1021640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac15e490 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39330/0 (socket says 172.21.15.29:39330) 2023-10-14T16:41:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 -- 172.21.15.29:0/2680309858 learned_addr learned my addr 172.21.15.29:0/2680309858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 -- 172.21.15.29:0/2680309858 >> v1:172.21.15.186:6789/0 conn(0x7fb3ac159950 legacy=0x7fb3ac070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 -- 172.21.15.29:0/2680309858 >> v1:172.21.15.29:6789/0 conn(0x7fb3ac070b60 legacy=0x7fb3ac06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.447+0000 7fb3b0820640 1 -- 172.21.15.29:0/2680309858 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac15ded0 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.448+0000 7fb3b0820640 1 --2- 172.21.15.29:0/2680309858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400bb10 tx=0x7fb3a402e0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b95266f0be49d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3aaffd640 1 -- 172.21.15.29:0/2680309858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a402ee80 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3aaffd640 1 -- 172.21.15.29:0/2680309858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a4036440 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3aaffd640 1 -- 172.21.15.29:0/2680309858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a40367b0 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 -- 172.21.15.29:0/2680309858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac0ca0a0 msgr2=0x7fb3ac06c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 --2- 172.21.15.29:0/2680309858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fb3a400bb10 tx=0x7fb3a402e0d0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 -- 172.21.15.29:0/2680309858 shutdown_connections 2023-10-14T16:41:43.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 --2- 172.21.15.29:0/2680309858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac06c810 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 -- 172.21.15.29:0/2680309858 >> 172.21.15.29:0/2680309858 conn(0x7fb3ac0ce6a0 msgr2=0x7fb3ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 -- 172.21.15.29:0/2680309858 shutdown_connections 2023-10-14T16:41:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.449+0000 7fb3b1021640 1 -- 172.21.15.29:0/2680309858 wait complete. 2023-10-14T16:41:43.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.451+0000 7fb3b1021640 1 Processor -- start 2023-10-14T16:41:43.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.451+0000 7fb3b1021640 1 -- start start 2023-10-14T16:41:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac1517d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac159950 0x7fb3ac151d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac159030 con 0x7fb3ac070b60 2023-10-14T16:41:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b2f20 con 0x7fb3ac0ca0a0 2023-10-14T16:41:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.452+0000 7fb3b1021640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b3200 con 0x7fb3ac159950 2023-10-14T16:41:43.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39338/0 (socket says 172.21.15.29:39338) 2023-10-14T16:41:43.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 -- 172.21.15.29:0/1277374740 learned_addr learned my addr 172.21.15.29:0/1277374740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 -- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac159950 msgr2=0x7fb3ac151d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac159950 0x7fb3ac151d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 -- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca0a0 msgr2=0x7fb3ac1517d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac1517d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 -- 172.21.15.29:0/1277374740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a400b770 con 0x7fb3ac070b60 2023-10-14T16:41:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.453+0000 7fb3abfff640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00b3e0 tx=0x7fb39c00b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.454+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c011830 con 0x7fb3ac070b60 2023-10-14T16:41:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.454+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb39c0119b0 con 0x7fb3ac070b60 2023-10-14T16:41:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.454+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb39c011d60 con 0x7fb3ac070b60 2023-10-14T16:41:43.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.454+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac0b34e0 con 0x7fb3ac070b60 2023-10-14T16:41:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.454+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac0b3aa0 con 0x7fb3ac070b60 2023-10-14T16:41:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.455+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac0b3cc0 con 0x7fb3ac070b60 2023-10-14T16:41:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.456+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb39c01a400 con 0x7fb3ac070b60 2023-10-14T16:41:43.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.457+0000 7fb3a8ff9640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 0x7fb39806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.457+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb39c089820 con 0x7fb3ac070b60 2023-10-14T16:41:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.457+0000 7fb3ab7fe640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 0x7fb39806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.458+0000 7fb3ab7fe640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 0x7fb39806d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00012e0 tx=0x7fb3a0007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.461+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb39c054830 con 0x7fb3ac070b60 2023-10-14T16:41:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.647+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb3ac159d30 con 0x7fb39806ae70 2023-10-14T16:41:43.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.674+0000 7fb3a8ff9640 1 -- 172.21.15.29:0/1277374740 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281613 (secure 0 0 0) 0x7fb3a00019c0 con 0x7fb39806ae70 2023-10-14T16:41:43.680 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:43.682 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:41:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.678+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 msgr2=0x7fb39806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.678+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 0x7fb39806d330 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb3a00012e0 tx=0x7fb3a0007d30 comp rx=0 tx=0).stop 2023-10-14T16:41:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.678+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 msgr2=0x7fb3ac0acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.678+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00b3e0 tx=0x7fb39c00b8a0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 shutdown_connections 2023-10-14T16:41:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb39806ae70 0x7fb39806d330 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac159950 0x7fb3ac151d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca0a0 0x7fb3ac1517d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 --2- 172.21.15.29:0/1277374740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070b60 0x7fb3ac0acb00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 >> 172.21.15.29:0/1277374740 conn(0x7fb3ac0ce6a0 msgr2=0x7fb3ac06b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:43.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 shutdown_connections 2023-10-14T16:41:43.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.679+0000 7fb3b1021640 1 -- 172.21.15.29:0/1277374740 wait complete. 2023-10-14T16:41:43.699 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2023-10-14T16:41:43.322343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236696,"kb_used_data":3892,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738024,"statfs":{"total":773094113280,"available":772851736576,"internally_reserved":0,"allocated":3985408,"data_stored":2250211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001496"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251935+0000","last_change":"2023-10-14T16:41:38.251935+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251935+0000","last_undegraded":"2023-10-14T16:41:38.251935+0000","last_fullsized":"2023-10-14T16:41:38.251935+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251427+0000","last_change":"2023-10-14T16:41:38.251427+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251427+0000","last_undegraded":"2023-10-14T16:41:38.251427+0000","last_fullsized":"2023-10-14T16:41:38.251427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252219+0000","last_change":"2023-10-14T16:41:38.252219+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252219+0000","last_undegraded":"2023-10-14T16:41:38.252219+0000","last_fullsized":"2023-10-14T16:41:38.252219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258644+0000","last_change":"2023-10-14T16:41:38.258644+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258644+0000","last_undegraded":"2023-10-14T16:41:38.258644+0000","last_fullsized":"2023-10-14T16:41:38.258644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248994+0000","last_change":"2023-10-14T16:41:38.248994+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248994+0000","last_undegraded":"2023-10-14T16:41:38.248994+0000","last_fullsized":"2023-10-14T16:41:38.248994+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247097+0000","last_change":"2023-10-14T16:41:38.247097+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247097+0000","last_undegraded":"2023-10-14T16:41:38.247097+0000","last_fullsized":"2023-10-14T16:41:38.247097+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263052+0000","last_change":"2023-10-14T16:41:38.263052+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263052+0000","last_undegraded":"2023-10-14T16:41:38.263052+0000","last_fullsized":"2023-10-14T16:41:38.263052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260191+0000","last_change":"2023-10-14T16:41:38.260191+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260191+0000","last_undegraded":"2023-10-14T16:41:38.260191+0000","last_fullsized":"2023-10-14T16:41:38.260191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249866+0000","last_change":"2023-10-14T16:41:38.249866+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249866+0000","last_undegraded":"2023-10-14T16:41:38.249866+0000","last_fullsized":"2023-10-14T16:41:38.249866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253885+0000","last_change":"2023-10-14T16:41:38.253885+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253885+0000","last_undegraded":"2023-10-14T16:41:38.253885+0000","last_fullsized":"2023-10-14T16:41:38.253885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.261410+0000","last_change":"2023-10-14T16:41:38.261410+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.261410+0000","last_undegraded":"2023-10-14T16:41:38.261410+0000","last_fullsized":"2023-10-14T16:41:38.261410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250453+0000","last_change":"2023-10-14T16:41:38.250453+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250453+0000","last_undegraded":"2023-10-14T16:41:38.250453+0000","last_fullsized":"2023-10-14T16:41:38.250453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248204+0000","last_change":"2023-10-14T16:41:38.248204+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248204+0000","last_undegraded":"2023-10-14T16:41:38.248204+0000","last_fullsized":"2023-10-14T16:41:38.248204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246426+0000","last_change":"2023-10-14T16:41:38.246426+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246426+0000","last_undegraded":"2023-10-14T16:41:38.246426+0000","last_fullsized":"2023-10-14T16:41:38.246426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248085+0000","last_change":"2023-10-14T16:41:38.248085+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248085+0000","last_undegraded":"2023-10-14T16:41:38.248085+0000","last_fullsized":"2023-10-14T16:41:38.248085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257273+0000","last_change":"2023-10-14T16:41:38.257273+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257273+0000","last_undegraded":"2023-10-14T16:41:38.257273+0000","last_fullsized":"2023-10-14T16:41:38.257273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.262216+0000","last_change":"2023-10-14T16:41:38.262216+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.262216+0000","last_undegraded":"2023-10-14T16:41:38.262216+0000","last_fullsized":"2023-10-14T16:41:38.262216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250965+0000","last_change":"2023-10-14T16:41:38.250965+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250965+0000","last_undegraded":"2023-10-14T16:41:38.250965+0000","last_fullsized":"2023-10-14T16:41:38.250965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245112+0000","last_change":"2023-10-14T16:41:38.245112+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245112+0000","last_undegraded":"2023-10-14T16:41:38.245112+0000","last_fullsized":"2023-10-14T16:41:38.245112+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249462+0000","last_change":"2023-10-14T16:41:38.249462+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249462+0000","last_undegraded":"2023-10-14T16:41:38.249462+0000","last_fullsized":"2023-10-14T16:41:38.249462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252360+0000","last_change":"2023-10-14T16:41:38.252360+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252360+0000","last_undegraded":"2023-10-14T16:41:38.252360+0000","last_fullsized":"2023-10-14T16:41:38.252360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253610+0000","last_change":"2023-10-14T16:41:38.253610+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253610+0000","last_undegraded":"2023-10-14T16:41:38.253610+0000","last_fullsized":"2023-10-14T16:41:38.253610+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243316+0000","last_change":"2023-10-14T16:41:38.243316+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243316+0000","last_undegraded":"2023-10-14T16:41:38.243316+0000","last_fullsized":"2023-10-14T16:41:38.243316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246964+0000","last_change":"2023-10-14T16:41:38.246964+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246964+0000","last_undegraded":"2023-10-14T16:41:38.246964+0000","last_fullsized":"2023-10-14T16:41:38.246964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256479+0000","last_change":"2023-10-14T16:41:38.256479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256479+0000","last_undegraded":"2023-10-14T16:41:38.256479+0000","last_fullsized":"2023-10-14T16:41:38.256479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245009+0000","last_change":"2023-10-14T16:41:38.245009+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245009+0000","last_undegraded":"2023-10-14T16:41:38.245009+0000","last_fullsized":"2023-10-14T16:41:38.245009+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258281+0000","last_change":"2023-10-14T16:41:38.258281+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258281+0000","last_undegraded":"2023-10-14T16:41:38.258281+0000","last_fullsized":"2023-10-14T16:41:38.258281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249479+0000","last_change":"2023-10-14T16:41:38.249479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249479+0000","last_undegraded":"2023-10-14T16:41:38.249479+0000","last_fullsized":"2023-10-14T16:41:38.249479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247132+0000","last_change":"2023-10-14T16:41:38.247132+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247132+0000","last_undegraded":"2023-10-14T16:41:38.247132+0000","last_fullsized":"2023-10-14T16:41:38.247132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260202+0000","last_change":"2023-10-14T16:41:38.260202+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260202+0000","last_undegraded":"2023-10-14T16:41:38.260202+0000","last_fullsized":"2023-10-14T16:41:38.260202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246717+0000","last_change":"2023-10-14T16:41:38.246717+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246717+0000","last_undegraded":"2023-10-14T16:41:38.246717+0000","last_fullsized":"2023-10-14T16:41:38.246717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259641+0000","last_change":"2023-10-14T16:41:38.259641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259641+0000","last_undegraded":"2023-10-14T16:41:38.259641+0000","last_fullsized":"2023-10-14T16:41:38.259641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244200+0000","last_change":"2023-10-14T16:41:38.244200+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244200+0000","last_undegraded":"2023-10-14T16:41:38.244200+0000","last_fullsized":"2023-10-14T16:41:38.244200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267873+0000","last_change":"2023-10-14T16:41:38.263891+0000","last_active":"2023-10-14T16:41:38.267873+0000","last_peered":"2023-10-14T16:41:38.267873+0000","last_clean":"2023-10-14T16:41:38.267873+0000","last_became_active":"2023-10-14T16:41:38.263622+0000","last_became_peered":"2023-10-14T16:41:38.263622+0000","last_unstale":"2023-10-14T16:41:38.267873+0000","last_undegraded":"2023-10-14T16:41:38.267873+0000","last_fullsized":"2023-10-14T16:41:38.267873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242058+0000","last_change":"2023-10-14T16:41:38.242058+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242058+0000","last_undegraded":"2023-10-14T16:41:38.242058+0000","last_fullsized":"2023-10-14T16:41:38.242058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272429+0000","last_change":"2023-10-14T16:41:38.266472+0000","last_active":"2023-10-14T16:41:38.272429+0000","last_peered":"2023-10-14T16:41:38.272429+0000","last_clean":"2023-10-14T16:41:38.272429+0000","last_became_active":"2023-10-14T16:41:38.266130+0000","last_became_peered":"2023-10-14T16:41:38.266130+0000","last_unstale":"2023-10-14T16:41:38.272429+0000","last_undegraded":"2023-10-14T16:41:38.272429+0000","last_fullsized":"2023-10-14T16:41:38.272429+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248641+0000","last_change":"2023-10-14T16:41:38.248641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248641+0000","last_undegraded":"2023-10-14T16:41:38.248641+0000","last_fullsized":"2023-10-14T16:41:38.248641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274644+0000","last_change":"2023-10-14T16:41:38.270437+0000","last_active":"2023-10-14T16:41:38.274644+0000","last_peered":"2023-10-14T16:41:38.274644+0000","last_clean":"2023-10-14T16:41:38.274644+0000","last_became_active":"2023-10-14T16:41:38.270157+0000","last_became_peered":"2023-10-14T16:41:38.270157+0000","last_unstale":"2023-10-14T16:41:38.274644+0000","last_undegraded":"2023-10-14T16:41:38.274644+0000","last_fullsized":"2023-10-14T16:41:38.274644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251155+0000","last_change":"2023-10-14T16:41:38.251155+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251155+0000","last_undegraded":"2023-10-14T16:41:38.251155+0000","last_fullsized":"2023-10-14T16:41:38.251155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272894+0000","last_change":"2023-10-14T16:41:38.271123+0000","last_active":"2023-10-14T16:41:38.272894+0000","last_peered":"2023-10-14T16:41:38.272894+0000","last_clean":"2023-10-14T16:41:38.272894+0000","last_became_active":"2023-10-14T16:41:38.270752+0000","last_became_peered":"2023-10-14T16:41:38.270752+0000","last_unstale":"2023-10-14T16:41:38.272894+0000","last_undegraded":"2023-10-14T16:41:38.272894+0000","last_fullsized":"2023-10-14T16:41:38.272894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259712+0000","last_change":"2023-10-14T16:41:38.259712+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259712+0000","last_undegraded":"2023-10-14T16:41:38.259712+0000","last_fullsized":"2023-10-14T16:41:38.259712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268235+0000","last_change":"2023-10-14T16:41:38.263100+0000","last_active":"2023-10-14T16:41:38.268235+0000","last_peered":"2023-10-14T16:41:38.268235+0000","last_clean":"2023-10-14T16:41:38.268235+0000","last_became_active":"2023-10-14T16:41:38.262254+0000","last_became_peered":"2023-10-14T16:41:38.262254+0000","last_unstale":"2023-10-14T16:41:38.268235+0000","last_undegraded":"2023-10-14T16:41:38.268235+0000","last_fullsized":"2023-10-14T16:41:38.268235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:11.858605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247837+0000","last_change":"2023-10-14T16:41:38.247837+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247837+0000","last_undegraded":"2023-10-14T16:41:38.247837+0000","last_fullsized":"2023-10-14T16:41:38.247837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267006+0000","last_change":"2023-10-14T16:41:38.259912+0000","last_active":"2023-10-14T16:41:38.267006+0000","last_peered":"2023-10-14T16:41:38.267006+0000","last_clean":"2023-10-14T16:41:38.267006+0000","last_became_active":"2023-10-14T16:41:38.258669+0000","last_became_peered":"2023-10-14T16:41:38.258669+0000","last_unstale":"2023-10-14T16:41:38.267006+0000","last_undegraded":"2023-10-14T16:41:38.267006+0000","last_fullsized":"2023-10-14T16:41:38.267006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:16:13.242402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244395+0000","last_change":"2023-10-14T16:41:38.244395+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244395+0000","last_undegraded":"2023-10-14T16:41:38.244395+0000","last_fullsized":"2023-10-14T16:41:38.244395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.258466+0000","last_change":"2023-10-14T16:41:38.257048+0000","last_active":"2023-10-14T16:41:38.258466+0000","last_peered":"2023-10-14T16:41:38.258466+0000","last_clean":"2023-10-14T16:41:38.258466+0000","last_became_active":"2023-10-14T16:41:38.256705+0000","last_became_peered":"2023-10-14T16:41:38.256705+0000","last_unstale":"2023-10-14T16:41:38.258466+0000","last_undegraded":"2023-10-14T16:41:38.258466+0000","last_fullsized":"2023-10-14T16:41:38.258466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257970+0000","last_change":"2023-10-14T16:41:38.257970+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257970+0000","last_undegraded":"2023-10-14T16:41:38.257970+0000","last_fullsized":"2023-10-14T16:41:38.257970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272295+0000","last_change":"2023-10-14T16:41:38.267598+0000","last_active":"2023-10-14T16:41:38.272295+0000","last_peered":"2023-10-14T16:41:38.272295+0000","last_clean":"2023-10-14T16:41:38.272295+0000","last_became_active":"2023-10-14T16:41:38.266568+0000","last_became_peered":"2023-10-14T16:41:38.266568+0000","last_unstale":"2023-10-14T16:41:38.272295+0000","last_undegraded":"2023-10-14T16:41:38.272295+0000","last_fullsized":"2023-10-14T16:41:38.272295+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:34.374586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246503+0000","last_change":"2023-10-14T16:41:38.246503+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246503+0000","last_undegraded":"2023-10-14T16:41:38.246503+0000","last_fullsized":"2023-10-14T16:41:38.246503+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273736+0000","last_change":"2023-10-14T16:41:38.269698+0000","last_active":"2023-10-14T16:41:38.273736+0000","last_peered":"2023-10-14T16:41:38.273736+0000","last_clean":"2023-10-14T16:41:38.273736+0000","last_became_active":"2023-10-14T16:41:38.269391+0000","last_became_peered":"2023-10-14T16:41:38.269391+0000","last_unstale":"2023-10-14T16:41:38.273736+0000","last_undegraded":"2023-10-14T16:41:38.273736+0000","last_fullsized":"2023-10-14T16:41:38.273736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:17.441986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248995+0000","last_change":"2023-10-14T16:41:38.248995+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248995+0000","last_undegraded":"2023-10-14T16:41:38.248995+0000","last_fullsized":"2023-10-14T16:41:38.248995+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.261889+0000","last_change":"2023-10-14T16:41:38.259142+0000","last_active":"2023-10-14T16:41:38.261889+0000","last_peered":"2023-10-14T16:41:38.261889+0000","last_clean":"2023-10-14T16:41:38.261889+0000","last_became_active":"2023-10-14T16:41:38.258853+0000","last_became_peered":"2023-10-14T16:41:38.258853+0000","last_unstale":"2023-10-14T16:41:38.261889+0000","last_undegraded":"2023-10-14T16:41:38.261889+0000","last_fullsized":"2023-10-14T16:41:38.261889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244949+0000","last_change":"2023-10-14T16:41:38.244949+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244949+0000","last_undegraded":"2023-10-14T16:41:38.244949+0000","last_fullsized":"2023-10-14T16:41:38.244949+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272571+0000","last_change":"2023-10-14T16:41:38.269707+0000","last_active":"2023-10-14T16:41:38.272571+0000","last_peered":"2023-10-14T16:41:38.272571+0000","last_clean":"2023-10-14T16:41:38.272571+0000","last_became_active":"2023-10-14T16:41:38.269101+0000","last_became_peered":"2023-10-14T16:41:38.269101+0000","last_unstale":"2023-10-14T16:41:38.272571+0000","last_undegraded":"2023-10-14T16:41:38.272571+0000","last_fullsized":"2023-10-14T16:41:38.272571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250366+0000","last_change":"2023-10-14T16:41:38.250366+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250366+0000","last_undegraded":"2023-10-14T16:41:38.250366+0000","last_fullsized":"2023-10-14T16:41:38.250366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273696+0000","last_change":"2023-10-14T16:41:38.268083+0000","last_active":"2023-10-14T16:41:38.273696+0000","last_peered":"2023-10-14T16:41:38.273696+0000","last_clean":"2023-10-14T16:41:38.273696+0000","last_became_active":"2023-10-14T16:41:38.267791+0000","last_became_peered":"2023-10-14T16:41:38.267791+0000","last_unstale":"2023-10-14T16:41:38.273696+0000","last_undegraded":"2023-10-14T16:41:38.273696+0000","last_fullsized":"2023-10-14T16:41:38.273696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251138+0000","last_change":"2023-10-14T16:41:38.251138+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251138+0000","last_undegraded":"2023-10-14T16:41:38.251138+0000","last_fullsized":"2023-10-14T16:41:38.251138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266868+0000","last_change":"2023-10-14T16:41:38.259070+0000","last_active":"2023-10-14T16:41:38.266868+0000","last_peered":"2023-10-14T16:41:38.266868+0000","last_clean":"2023-10-14T16:41:38.266868+0000","last_became_active":"2023-10-14T16:41:38.258658+0000","last_became_peered":"2023-10-14T16:41:38.258658+0000","last_unstale":"2023-10-14T16:41:38.266868+0000","last_undegraded":"2023-10-14T16:41:38.266868+0000","last_fullsized":"2023-10-14T16:41:38.266868+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244826+0000","last_change":"2023-10-14T16:41:38.244826+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244826+0000","last_undegraded":"2023-10-14T16:41:38.244826+0000","last_fullsized":"2023-10-14T16:41:38.244826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269845+0000","last_change":"2023-10-14T16:41:38.265796+0000","last_active":"2023-10-14T16:41:38.269845+0000","last_peered":"2023-10-14T16:41:38.269845+0000","last_clean":"2023-10-14T16:41:38.269845+0000","last_became_active":"2023-10-14T16:41:38.265491+0000","last_became_peered":"2023-10-14T16:41:38.265491+0000","last_unstale":"2023-10-14T16:41:38.269845+0000","last_undegraded":"2023-10-14T16:41:38.269845+0000","last_fullsized":"2023-10-14T16:41:38.269845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:25.436192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259056+0000","last_change":"2023-10-14T16:41:38.259056+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259056+0000","last_undegraded":"2023-10-14T16:41:38.259056+0000","last_fullsized":"2023-10-14T16:41:38.259056+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274094+0000","last_change":"2023-10-14T16:41:38.271323+0000","last_active":"2023-10-14T16:41:38.274094+0000","last_peered":"2023-10-14T16:41:38.274094+0000","last_clean":"2023-10-14T16:41:38.274094+0000","last_became_active":"2023-10-14T16:41:38.271014+0000","last_became_peered":"2023-10-14T16:41:38.271014+0000","last_unstale":"2023-10-14T16:41:38.274094+0000","last_undegraded":"2023-10-14T16:41:38.274094+0000","last_fullsized":"2023-10-14T16:41:38.274094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242243+0000","last_change":"2023-10-14T16:41:38.242243+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242243+0000","last_undegraded":"2023-10-14T16:41:38.242243+0000","last_fullsized":"2023-10-14T16:41:38.242243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274622+0000","last_change":"2023-10-14T16:41:38.272215+0000","last_active":"2023-10-14T16:41:38.274622+0000","last_peered":"2023-10-14T16:41:38.274622+0000","last_clean":"2023-10-14T16:41:38.274622+0000","last_became_active":"2023-10-14T16:41:38.271943+0000","last_became_peered":"2023-10-14T16:41:38.271943+0000","last_unstale":"2023-10-14T16:41:38.274622+0000","last_undegraded":"2023-10-14T16:41:38.274622+0000","last_fullsized":"2023-10-14T16:41:38.274622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253161+0000","last_change":"2023-10-14T16:41:38.253161+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253161+0000","last_undegraded":"2023-10-14T16:41:38.253161+0000","last_fullsized":"2023-10-14T16:41:38.253161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270665+0000","last_change":"2023-10-14T16:41:38.267446+0000","last_active":"2023-10-14T16:41:38.270665+0000","last_peered":"2023-10-14T16:41:38.270665+0000","last_clean":"2023-10-14T16:41:38.270665+0000","last_became_active":"2023-10-14T16:41:38.267122+0000","last_became_peered":"2023-10-14T16:41:38.267122+0000","last_unstale":"2023-10-14T16:41:38.270665+0000","last_undegraded":"2023-10-14T16:41:38.270665+0000","last_fullsized":"2023-10-14T16:41:38.270665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246383+0000","last_change":"2023-10-14T16:41:38.246383+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246383+0000","last_undegraded":"2023-10-14T16:41:38.246383+0000","last_fullsized":"2023-10-14T16:41:38.246383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268444+0000","last_change":"2023-10-14T16:41:38.266145+0000","last_active":"2023-10-14T16:41:38.268444+0000","last_peered":"2023-10-14T16:41:38.268444+0000","last_clean":"2023-10-14T16:41:38.268444+0000","last_became_active":"2023-10-14T16:41:38.264561+0000","last_became_peered":"2023-10-14T16:41:38.264561+0000","last_unstale":"2023-10-14T16:41:38.268444+0000","last_undegraded":"2023-10-14T16:41:38.268444+0000","last_fullsized":"2023-10-14T16:41:38.268444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:45:54.740964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256466+0000","last_change":"2023-10-14T16:41:38.256466+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256466+0000","last_undegraded":"2023-10-14T16:41:38.256466+0000","last_fullsized":"2023-10-14T16:41:38.256466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268496+0000","last_change":"2023-10-14T16:41:38.266046+0000","last_active":"2023-10-14T16:41:38.268496+0000","last_peered":"2023-10-14T16:41:38.268496+0000","last_clean":"2023-10-14T16:41:38.268496+0000","last_became_active":"2023-10-14T16:41:38.263725+0000","last_became_peered":"2023-10-14T16:41:38.263725+0000","last_unstale":"2023-10-14T16:41:38.268496+0000","last_undegraded":"2023-10-14T16:41:38.268496+0000","last_fullsized":"2023-10-14T16:41:38.268496+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:44.086979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252808+0000","last_change":"2023-10-14T16:41:38.252808+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252808+0000","last_undegraded":"2023-10-14T16:41:38.252808+0000","last_fullsized":"2023-10-14T16:41:38.252808+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268294+0000","last_change":"2023-10-14T16:41:38.263662+0000","last_active":"2023-10-14T16:41:38.268294+0000","last_peered":"2023-10-14T16:41:38.268294+0000","last_clean":"2023-10-14T16:41:38.268294+0000","last_became_active":"2023-10-14T16:41:38.262441+0000","last_became_peered":"2023-10-14T16:41:38.262441+0000","last_unstale":"2023-10-14T16:41:38.268294+0000","last_undegraded":"2023-10-14T16:41:38.268294+0000","last_fullsized":"2023-10-14T16:41:38.268294+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:54.579331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245609+0000","last_change":"2023-10-14T16:41:38.245609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245609+0000","last_undegraded":"2023-10-14T16:41:38.245609+0000","last_fullsized":"2023-10-14T16:41:38.245609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273039+0000","last_change":"2023-10-14T16:41:38.270622+0000","last_active":"2023-10-14T16:41:38.273039+0000","last_peered":"2023-10-14T16:41:38.273039+0000","last_clean":"2023-10-14T16:41:38.273039+0000","last_became_active":"2023-10-14T16:41:38.269459+0000","last_became_peered":"2023-10-14T16:41:38.269459+0000","last_unstale":"2023-10-14T16:41:38.273039+0000","last_undegraded":"2023-10-14T16:41:38.273039+0000","last_fullsized":"2023-10-14T16:41:38.273039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247414+0000","last_change":"2023-10-14T16:41:38.247414+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247414+0000","last_undegraded":"2023-10-14T16:41:38.247414+0000","last_fullsized":"2023-10-14T16:41:38.247414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274818+0000","last_change":"2023-10-14T16:41:38.270414+0000","last_active":"2023-10-14T16:41:38.274818+0000","last_peered":"2023-10-14T16:41:38.274818+0000","last_clean":"2023-10-14T16:41:38.274818+0000","last_became_active":"2023-10-14T16:41:38.269445+0000","last_became_peered":"2023-10-14T16:41:38.269445+0000","last_unstale":"2023-10-14T16:41:38.274818+0000","last_undegraded":"2023-10-14T16:41:38.274818+0000","last_fullsized":"2023-10-14T16:41:38.274818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247587+0000","last_change":"2023-10-14T16:41:38.247587+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247587+0000","last_undegraded":"2023-10-14T16:41:38.247587+0000","last_fullsized":"2023-10-14T16:41:38.247587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.275360+0000","last_change":"2023-10-14T16:41:38.271734+0000","last_active":"2023-10-14T16:41:38.275360+0000","last_peered":"2023-10-14T16:41:38.275360+0000","last_clean":"2023-10-14T16:41:38.275360+0000","last_became_active":"2023-10-14T16:41:38.271386+0000","last_became_peered":"2023-10-14T16:41:38.271386+0000","last_unstale":"2023-10-14T16:41:38.275360+0000","last_undegraded":"2023-10-14T16:41:38.275360+0000","last_fullsized":"2023-10-14T16:41:38.275360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:46:16.492082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263609+0000","last_change":"2023-10-14T16:41:38.263609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263609+0000","last_undegraded":"2023-10-14T16:41:38.263609+0000","last_fullsized":"2023-10-14T16:41:38.263609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270337+0000","last_change":"2023-10-14T16:41:38.266162+0000","last_active":"2023-10-14T16:41:38.270337+0000","last_peered":"2023-10-14T16:41:38.270337+0000","last_clean":"2023-10-14T16:41:38.270337+0000","last_became_active":"2023-10-14T16:41:38.265806+0000","last_became_peered":"2023-10-14T16:41:38.265806+0000","last_unstale":"2023-10-14T16:41:38.270337+0000","last_undegraded":"2023-10-14T16:41:38.270337+0000","last_fullsized":"2023-10-14T16:41:38.270337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252719+0000","last_change":"2023-10-14T16:41:38.252719+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252719+0000","last_undegraded":"2023-10-14T16:41:38.252719+0000","last_fullsized":"2023-10-14T16:41:38.252719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273662+0000","last_change":"2023-10-14T16:41:38.271489+0000","last_active":"2023-10-14T16:41:38.273662+0000","last_peered":"2023-10-14T16:41:38.273662+0000","last_clean":"2023-10-14T16:41:38.273662+0000","last_became_active":"2023-10-14T16:41:38.271096+0000","last_became_peered":"2023-10-14T16:41:38.271096+0000","last_unstale":"2023-10-14T16:41:38.273662+0000","last_undegraded":"2023-10-14T16:41:38.273662+0000","last_fullsized":"2023-10-14T16:41:38.273662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244568+0000","last_change":"2023-10-14T16:41:38.244568+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244568+0000","last_undegraded":"2023-10-14T16:41:38.244568+0000","last_fullsized":"2023-10-14T16:41:38.244568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272453+0000","last_change":"2023-10-14T16:41:38.267492+0000","last_active":"2023-10-14T16:41:38.272453+0000","last_peered":"2023-10-14T16:41:38.272453+0000","last_clean":"2023-10-14T16:41:38.272453+0000","last_became_active":"2023-10-14T16:41:38.267208+0000","last_became_peered":"2023-10-14T16:41:38.267208+0000","last_unstale":"2023-10-14T16:41:38.272453+0000","last_undegraded":"2023-10-14T16:41:38.272453+0000","last_fullsized":"2023-10-14T16:41:38.272453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:07.318539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243653+0000","last_change":"2023-10-14T16:41:38.243653+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243653+0000","last_undegraded":"2023-10-14T16:41:38.243653+0000","last_fullsized":"2023-10-14T16:41:38.243653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272435+0000","last_change":"2023-10-14T16:41:38.269649+0000","last_active":"2023-10-14T16:41:38.272435+0000","last_peered":"2023-10-14T16:41:38.272435+0000","last_clean":"2023-10-14T16:41:38.272435+0000","last_became_active":"2023-10-14T16:41:38.269092+0000","last_became_peered":"2023-10-14T16:41:38.269092+0000","last_unstale":"2023-10-14T16:41:38.272435+0000","last_undegraded":"2023-10-14T16:41:38.272435+0000","last_fullsized":"2023-10-14T16:41:38.272435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.241898+0000","last_change":"2023-10-14T16:41:38.241898+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.241898+0000","last_undegraded":"2023-10-14T16:41:38.241898+0000","last_fullsized":"2023-10-14T16:41:38.241898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.262512+0000","last_change":"2023-10-14T16:41:38.260866+0000","last_active":"2023-10-14T16:41:38.262512+0000","last_peered":"2023-10-14T16:41:38.262512+0000","last_clean":"2023-10-14T16:41:38.262512+0000","last_became_active":"2023-10-14T16:41:38.260498+0000","last_became_peered":"2023-10-14T16:41:38.260498+0000","last_unstale":"2023-10-14T16:41:38.262512+0000","last_undegraded":"2023-10-14T16:41:38.262512+0000","last_fullsized":"2023-10-14T16:41:38.262512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246257+0000","last_change":"2023-10-14T16:41:38.246257+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246257+0000","last_undegraded":"2023-10-14T16:41:38.246257+0000","last_fullsized":"2023-10-14T16:41:38.246257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269808+0000","last_change":"2023-10-14T16:41:38.265778+0000","last_active":"2023-10-14T16:41:38.269808+0000","last_peered":"2023-10-14T16:41:38.269808+0000","last_clean":"2023-10-14T16:41:38.269808+0000","last_became_active":"2023-10-14T16:41:38.265477+0000","last_became_peered":"2023-10-14T16:41:38.265477+0000","last_unstale":"2023-10-14T16:41:38.269808+0000","last_undegraded":"2023-10-14T16:41:38.269808+0000","last_fullsized":"2023-10-14T16:41:38.269808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246290+0000","last_change":"2023-10-14T16:41:38.246290+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246290+0000","last_undegraded":"2023-10-14T16:41:38.246290+0000","last_fullsized":"2023-10-14T16:41:38.246290+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268405+0000","last_change":"2023-10-14T16:41:38.265852+0000","last_active":"2023-10-14T16:41:38.268405+0000","last_peered":"2023-10-14T16:41:38.268405+0000","last_clean":"2023-10-14T16:41:38.268405+0000","last_became_active":"2023-10-14T16:41:38.265513+0000","last_became_peered":"2023-10-14T16:41:38.265513+0000","last_unstale":"2023-10-14T16:41:38.268405+0000","last_undegraded":"2023-10-14T16:41:38.268405+0000","last_fullsized":"2023-10-14T16:41:38.268405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:44:33.854697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247745+0000","last_change":"2023-10-14T16:41:38.247745+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247745+0000","last_undegraded":"2023-10-14T16:41:38.247745+0000","last_fullsized":"2023-10-14T16:41:38.247745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266793+0000","last_change":"2023-10-14T16:41:38.265704+0000","last_active":"2023-10-14T16:41:38.266793+0000","last_peered":"2023-10-14T16:41:38.266793+0000","last_clean":"2023-10-14T16:41:38.266793+0000","last_became_active":"2023-10-14T16:41:38.264927+0000","last_became_peered":"2023-10-14T16:41:38.264927+0000","last_unstale":"2023-10-14T16:41:38.266793+0000","last_undegraded":"2023-10-14T16:41:38.266793+0000","last_fullsized":"2023-10-14T16:41:38.266793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256689+0000","last_change":"2023-10-14T16:41:38.256689+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256689+0000","last_undegraded":"2023-10-14T16:41:38.256689+0000","last_fullsized":"2023-10-14T16:41:38.256689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268528+0000","last_change":"2023-10-14T16:41:38.267370+0000","last_active":"2023-10-14T16:41:38.268528+0000","last_peered":"2023-10-14T16:41:38.268528+0000","last_clean":"2023-10-14T16:41:38.268528+0000","last_became_active":"2023-10-14T16:41:38.267087+0000","last_became_peered":"2023-10-14T16:41:38.267087+0000","last_unstale":"2023-10-14T16:41:38.268528+0000","last_undegraded":"2023-10-14T16:41:38.268528+0000","last_fullsized":"2023-10-14T16:41:38.268528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T16:41:37.236449+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:41:37.236449+0000","last_peered":"2023-10-14T16:41:37.236449+0000","last_clean":"2023-10-14T16:41:37.236449+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:41:37.236449+0000","last_undegraded":"2023-10-14T16:41:37.236449+0000","last_fullsized":"2023-10-14T16:41:37.236449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.034},{"interface":"front","average":{"1min":1.173,"5min":1.173,"15min":1.173},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.272}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.125}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.3069999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.179},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.6020000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":1.575},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.427}]},{"osd":6,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":1.77}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":1.627},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.0720000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":1.429},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.51}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":1.29},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.6659999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":3.0110000000000001,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":1.754},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":3.206,"5min":3.206,"15min":3.206},"last":1.476}]},{"osd":7,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.843}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":139168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":0.98199999999999998}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":1.716}]},{"osd":2,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.3639999999999999,"5min":4.3639999999999999,"15min":4.3639999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":1.1120000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.823},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":1.4590000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.212},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.893}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.9470000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"last":1.6799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":1.093},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":4.4059999999999997},"last":1.149}]},{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.4510000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.876},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.98399999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.194}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.48},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.3200000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.29},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.3680000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.81200000000000006}]},{"osd":1,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.024},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":0.95399999999999996}]},{"osd":4,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":1.101},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":1.0549999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.2490000000000001,"5min":3.2490000000000001,"15min":3.2490000000000001},"last":1.179}]},{"osd":6,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.121}]},{"osd":7,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.236}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.5},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.2410000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.3080000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.583},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.006},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":1.518}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.1200000000000001,"5min":3.1200000000000001,"15min":3.1200000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.1060000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.054}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.79600000000000004}]},{"osd":2,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.3759999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.603},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":0.98299999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.429},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.526}]},{"osd":6,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":4.6360000000000001,"5min":4.6360000000000001,"15min":4.6360000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.4970000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.667},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.4650000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.05}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.84999999999999998}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.042},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.88200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:41:43.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:41:43.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.968+0000 7f8c1ab9e640 1 Processor -- start 2023-10-14T16:41:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.969+0000 7f8c1ab9e640 1 -- start start 2023-10-14T16:41:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.970+0000 7f8c1ab9e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c1406c040 0x7f8c1406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.970+0000 7f8c1ab9e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1415f4a0 con 0x7f8c140cda80 2023-10-14T16:41:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.970+0000 7f8c1ab9e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c1406c960 con 0x7f8c140ad970 2023-10-14T16:41:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.970+0000 7f8c1ab9e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c1415b9d0 con 0x7f8c1406c040 2023-10-14T16:41:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.970+0000 7f8c19b9c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c1406c040 0x7f8c1406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.971+0000 7f8c19b9c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c1406c040 0x7f8c1406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39350/0 (socket says 172.21.15.29:39350) 2023-10-14T16:41:43.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.971+0000 7f8c19b9c640 1 -- 172.21.15.29:0/1284302866 learned_addr learned my addr 172.21.15.29:0/1284302866 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.971+0000 7f8c19b9c640 1 -- 172.21.15.29:0/1284302866 >> v1:172.21.15.186:6789/0 conn(0x7f8c140ad970 legacy=0x7f8c1415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.971+0000 7f8c19b9c640 1 -- 172.21.15.29:0/1284302866 >> v1:172.21.15.29:6789/0 conn(0x7f8c140cda80 legacy=0x7f8c1415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.971+0000 7f8c19b9c640 1 -- 172.21.15.29:0/1284302866 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c1415bcb0 con 0x7f8c1406c040 2023-10-14T16:41:43.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.972+0000 7f8c19b9c640 1 --2- 172.21.15.29:0/1284302866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1406c040 0x7f8c1406c420 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009870 tx=0x7f8c1002b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45a742a2397376df server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.972+0000 7f8c18b9a640 1 -- 172.21.15.29:0/1284302866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10030850 con 0x7f8c1406c040 2023-10-14T16:41:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.972+0000 7f8c18b9a640 1 -- 172.21.15.29:0/1284302866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c100309b0 con 0x7f8c1406c040 2023-10-14T16:41:43.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c18b9a640 1 -- 172.21.15.29:0/1284302866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10030d40 con 0x7f8c1406c040 2023-10-14T16:41:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1284302866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1406c040 msgr2=0x7f8c1406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1284302866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1406c040 0x7f8c1406c420 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009870 tx=0x7f8c1002b990 comp rx=0 tx=0).stop 2023-10-14T16:41:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1284302866 shutdown_connections 2023-10-14T16:41:43.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1284302866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1406c040 0x7f8c1406c420 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1284302866 >> 172.21.15.29:0/1284302866 conn(0x7f8c140ce6a0 msgr2=0x7f8c140ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1284302866 shutdown_connections 2023-10-14T16:41:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.973+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1284302866 wait complete. 2023-10-14T16:41:43.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.975+0000 7f8c1ab9e640 1 Processor -- start 2023-10-14T16:41:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.975+0000 7f8c1ab9e640 1 -- start start 2023-10-14T16:41:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 0x7f8c140c2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c140cda80 0x7f8c140b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140c7950 con 0x7f8c140ad970 2023-10-14T16:41:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140c7670 con 0x7f8c1406c040 2023-10-14T16:41:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.976+0000 7f8c1ab9e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c140c2300 con 0x7f8c140cda80 2023-10-14T16:41:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39362/0 (socket says 172.21.15.29:39362) 2023-10-14T16:41:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 -- 172.21.15.29:0/1273712544 learned_addr learned my addr 172.21.15.29:0/1273712544 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c19b9c640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 0x7f8c140c2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 -- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c140cda80 msgr2=0x7f8c140b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1a39d640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c140cda80 0x7f8c140b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c140cda80 0x7f8c140b2f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 -- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 msgr2=0x7f8c140c2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 0x7f8c140c2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 -- 172.21.15.29:0/1273712544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c100094d0 con 0x7f8c140ad970 2023-10-14T16:41:43.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c19b9c640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 0x7f8c140c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.977+0000 7f8c1939b640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c400 tx=0x7f8c0400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.978+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c040035f0 con 0x7f8c140ad970 2023-10-14T16:41:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.978+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c04003750 con 0x7f8c140ad970 2023-10-14T16:41:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.978+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c04003b40 con 0x7f8c140ad970 2023-10-14T16:41:43.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.978+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c140c25e0 con 0x7f8c140ad970 2023-10-14T16:41:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.978+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c140c71e0 con 0x7f8c140ad970 2023-10-14T16:41:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.979+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1406b330 con 0x7f8c140ad970 2023-10-14T16:41:43.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.980+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f8c04003ca0 con 0x7f8c140ad970 2023-10-14T16:41:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.981+0000 7f8c0a7fc640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 0x7f8bf806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:43.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.981+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f8c04088760 con 0x7f8c140ad970 2023-10-14T16:41:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.981+0000 7f8c19b9c640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 0x7f8bf806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.982+0000 7f8c19b9c640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 0x7f8bf806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8c10036000 tx=0x7f8c1003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:43.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:43.985+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c04053950 con 0x7f8c140ad970 2023-10-14T16:41:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.177+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8c140cc270 con 0x7f8bf806ae70 2023-10-14T16:41:44.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.192+0000 7f8c0a7fc640 1 -- 172.21.15.29:0/1273712544 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281613 (secure 0 0 0) 0x7f8c10030c20 con 0x7f8bf806ae70 2023-10-14T16:41:44.197 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:44.199 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:41:44.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 msgr2=0x7f8bf806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 0x7f8bf806d330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8c10036000 tx=0x7f8c1003b040 comp rx=0 tx=0).stop 2023-10-14T16:41:44.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 msgr2=0x7f8c140b2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f8c0400c400 tx=0x7f8c0400c8c0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 shutdown_connections 2023-10-14T16:41:44.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f8bf806ae70 0x7f8bf806d330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c140cda80 0x7f8c140b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c140ad970 0x7f8c140b2a10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 --2- 172.21.15.29:0/1273712544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c1406c040 0x7f8c140c2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 >> 172.21.15.29:0/1273712544 conn(0x7f8c140ce6a0 msgr2=0x7f8c140cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:44.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 shutdown_connections 2023-10-14T16:41:44.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.198+0000 7f8c1ab9e640 1 -- 172.21.15.29:0/1273712544 wait complete. 2023-10-14T16:41:44.220 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2023-10-14T16:41:43.322343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236696,"kb_used_data":3892,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738024,"statfs":{"total":773094113280,"available":772851736576,"internally_reserved":0,"allocated":3985408,"data_stored":2250211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001496"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251935+0000","last_change":"2023-10-14T16:41:38.251935+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251935+0000","last_undegraded":"2023-10-14T16:41:38.251935+0000","last_fullsized":"2023-10-14T16:41:38.251935+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251427+0000","last_change":"2023-10-14T16:41:38.251427+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251427+0000","last_undegraded":"2023-10-14T16:41:38.251427+0000","last_fullsized":"2023-10-14T16:41:38.251427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252219+0000","last_change":"2023-10-14T16:41:38.252219+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252219+0000","last_undegraded":"2023-10-14T16:41:38.252219+0000","last_fullsized":"2023-10-14T16:41:38.252219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258644+0000","last_change":"2023-10-14T16:41:38.258644+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258644+0000","last_undegraded":"2023-10-14T16:41:38.258644+0000","last_fullsized":"2023-10-14T16:41:38.258644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248994+0000","last_change":"2023-10-14T16:41:38.248994+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248994+0000","last_undegraded":"2023-10-14T16:41:38.248994+0000","last_fullsized":"2023-10-14T16:41:38.248994+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247097+0000","last_change":"2023-10-14T16:41:38.247097+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247097+0000","last_undegraded":"2023-10-14T16:41:38.247097+0000","last_fullsized":"2023-10-14T16:41:38.247097+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263052+0000","last_change":"2023-10-14T16:41:38.263052+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263052+0000","last_undegraded":"2023-10-14T16:41:38.263052+0000","last_fullsized":"2023-10-14T16:41:38.263052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260191+0000","last_change":"2023-10-14T16:41:38.260191+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260191+0000","last_undegraded":"2023-10-14T16:41:38.260191+0000","last_fullsized":"2023-10-14T16:41:38.260191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249866+0000","last_change":"2023-10-14T16:41:38.249866+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249866+0000","last_undegraded":"2023-10-14T16:41:38.249866+0000","last_fullsized":"2023-10-14T16:41:38.249866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253885+0000","last_change":"2023-10-14T16:41:38.253885+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253885+0000","last_undegraded":"2023-10-14T16:41:38.253885+0000","last_fullsized":"2023-10-14T16:41:38.253885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.261410+0000","last_change":"2023-10-14T16:41:38.261410+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.261410+0000","last_undegraded":"2023-10-14T16:41:38.261410+0000","last_fullsized":"2023-10-14T16:41:38.261410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250453+0000","last_change":"2023-10-14T16:41:38.250453+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250453+0000","last_undegraded":"2023-10-14T16:41:38.250453+0000","last_fullsized":"2023-10-14T16:41:38.250453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248204+0000","last_change":"2023-10-14T16:41:38.248204+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248204+0000","last_undegraded":"2023-10-14T16:41:38.248204+0000","last_fullsized":"2023-10-14T16:41:38.248204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246426+0000","last_change":"2023-10-14T16:41:38.246426+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246426+0000","last_undegraded":"2023-10-14T16:41:38.246426+0000","last_fullsized":"2023-10-14T16:41:38.246426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248085+0000","last_change":"2023-10-14T16:41:38.248085+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248085+0000","last_undegraded":"2023-10-14T16:41:38.248085+0000","last_fullsized":"2023-10-14T16:41:38.248085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257273+0000","last_change":"2023-10-14T16:41:38.257273+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257273+0000","last_undegraded":"2023-10-14T16:41:38.257273+0000","last_fullsized":"2023-10-14T16:41:38.257273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.262216+0000","last_change":"2023-10-14T16:41:38.262216+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.262216+0000","last_undegraded":"2023-10-14T16:41:38.262216+0000","last_fullsized":"2023-10-14T16:41:38.262216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250965+0000","last_change":"2023-10-14T16:41:38.250965+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250965+0000","last_undegraded":"2023-10-14T16:41:38.250965+0000","last_fullsized":"2023-10-14T16:41:38.250965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245112+0000","last_change":"2023-10-14T16:41:38.245112+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245112+0000","last_undegraded":"2023-10-14T16:41:38.245112+0000","last_fullsized":"2023-10-14T16:41:38.245112+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249462+0000","last_change":"2023-10-14T16:41:38.249462+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249462+0000","last_undegraded":"2023-10-14T16:41:38.249462+0000","last_fullsized":"2023-10-14T16:41:38.249462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252360+0000","last_change":"2023-10-14T16:41:38.252360+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252360+0000","last_undegraded":"2023-10-14T16:41:38.252360+0000","last_fullsized":"2023-10-14T16:41:38.252360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253610+0000","last_change":"2023-10-14T16:41:38.253610+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253610+0000","last_undegraded":"2023-10-14T16:41:38.253610+0000","last_fullsized":"2023-10-14T16:41:38.253610+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243316+0000","last_change":"2023-10-14T16:41:38.243316+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243316+0000","last_undegraded":"2023-10-14T16:41:38.243316+0000","last_fullsized":"2023-10-14T16:41:38.243316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246964+0000","last_change":"2023-10-14T16:41:38.246964+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246964+0000","last_undegraded":"2023-10-14T16:41:38.246964+0000","last_fullsized":"2023-10-14T16:41:38.246964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256479+0000","last_change":"2023-10-14T16:41:38.256479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256479+0000","last_undegraded":"2023-10-14T16:41:38.256479+0000","last_fullsized":"2023-10-14T16:41:38.256479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245009+0000","last_change":"2023-10-14T16:41:38.245009+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245009+0000","last_undegraded":"2023-10-14T16:41:38.245009+0000","last_fullsized":"2023-10-14T16:41:38.245009+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258281+0000","last_change":"2023-10-14T16:41:38.258281+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258281+0000","last_undegraded":"2023-10-14T16:41:38.258281+0000","last_fullsized":"2023-10-14T16:41:38.258281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249479+0000","last_change":"2023-10-14T16:41:38.249479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249479+0000","last_undegraded":"2023-10-14T16:41:38.249479+0000","last_fullsized":"2023-10-14T16:41:38.249479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247132+0000","last_change":"2023-10-14T16:41:38.247132+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247132+0000","last_undegraded":"2023-10-14T16:41:38.247132+0000","last_fullsized":"2023-10-14T16:41:38.247132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260202+0000","last_change":"2023-10-14T16:41:38.260202+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260202+0000","last_undegraded":"2023-10-14T16:41:38.260202+0000","last_fullsized":"2023-10-14T16:41:38.260202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246717+0000","last_change":"2023-10-14T16:41:38.246717+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246717+0000","last_undegraded":"2023-10-14T16:41:38.246717+0000","last_fullsized":"2023-10-14T16:41:38.246717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259641+0000","last_change":"2023-10-14T16:41:38.259641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259641+0000","last_undegraded":"2023-10-14T16:41:38.259641+0000","last_fullsized":"2023-10-14T16:41:38.259641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244200+0000","last_change":"2023-10-14T16:41:38.244200+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244200+0000","last_undegraded":"2023-10-14T16:41:38.244200+0000","last_fullsized":"2023-10-14T16:41:38.244200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267873+0000","last_change":"2023-10-14T16:41:38.263891+0000","last_active":"2023-10-14T16:41:38.267873+0000","last_peered":"2023-10-14T16:41:38.267873+0000","last_clean":"2023-10-14T16:41:38.267873+0000","last_became_active":"2023-10-14T16:41:38.263622+0000","last_became_peered":"2023-10-14T16:41:38.263622+0000","last_unstale":"2023-10-14T16:41:38.267873+0000","last_undegraded":"2023-10-14T16:41:38.267873+0000","last_fullsized":"2023-10-14T16:41:38.267873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242058+0000","last_change":"2023-10-14T16:41:38.242058+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242058+0000","last_undegraded":"2023-10-14T16:41:38.242058+0000","last_fullsized":"2023-10-14T16:41:38.242058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272429+0000","last_change":"2023-10-14T16:41:38.266472+0000","last_active":"2023-10-14T16:41:38.272429+0000","last_peered":"2023-10-14T16:41:38.272429+0000","last_clean":"2023-10-14T16:41:38.272429+0000","last_became_active":"2023-10-14T16:41:38.266130+0000","last_became_peered":"2023-10-14T16:41:38.266130+0000","last_unstale":"2023-10-14T16:41:38.272429+0000","last_undegraded":"2023-10-14T16:41:38.272429+0000","last_fullsized":"2023-10-14T16:41:38.272429+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248641+0000","last_change":"2023-10-14T16:41:38.248641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248641+0000","last_undegraded":"2023-10-14T16:41:38.248641+0000","last_fullsized":"2023-10-14T16:41:38.248641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274644+0000","last_change":"2023-10-14T16:41:38.270437+0000","last_active":"2023-10-14T16:41:38.274644+0000","last_peered":"2023-10-14T16:41:38.274644+0000","last_clean":"2023-10-14T16:41:38.274644+0000","last_became_active":"2023-10-14T16:41:38.270157+0000","last_became_peered":"2023-10-14T16:41:38.270157+0000","last_unstale":"2023-10-14T16:41:38.274644+0000","last_undegraded":"2023-10-14T16:41:38.274644+0000","last_fullsized":"2023-10-14T16:41:38.274644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251155+0000","last_change":"2023-10-14T16:41:38.251155+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251155+0000","last_undegraded":"2023-10-14T16:41:38.251155+0000","last_fullsized":"2023-10-14T16:41:38.251155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272894+0000","last_change":"2023-10-14T16:41:38.271123+0000","last_active":"2023-10-14T16:41:38.272894+0000","last_peered":"2023-10-14T16:41:38.272894+0000","last_clean":"2023-10-14T16:41:38.272894+0000","last_became_active":"2023-10-14T16:41:38.270752+0000","last_became_peered":"2023-10-14T16:41:38.270752+0000","last_unstale":"2023-10-14T16:41:38.272894+0000","last_undegraded":"2023-10-14T16:41:38.272894+0000","last_fullsized":"2023-10-14T16:41:38.272894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259712+0000","last_change":"2023-10-14T16:41:38.259712+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259712+0000","last_undegraded":"2023-10-14T16:41:38.259712+0000","last_fullsized":"2023-10-14T16:41:38.259712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268235+0000","last_change":"2023-10-14T16:41:38.263100+0000","last_active":"2023-10-14T16:41:38.268235+0000","last_peered":"2023-10-14T16:41:38.268235+0000","last_clean":"2023-10-14T16:41:38.268235+0000","last_became_active":"2023-10-14T16:41:38.262254+0000","last_became_peered":"2023-10-14T16:41:38.262254+0000","last_unstale":"2023-10-14T16:41:38.268235+0000","last_undegraded":"2023-10-14T16:41:38.268235+0000","last_fullsized":"2023-10-14T16:41:38.268235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:11.858605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247837+0000","last_change":"2023-10-14T16:41:38.247837+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247837+0000","last_undegraded":"2023-10-14T16:41:38.247837+0000","last_fullsized":"2023-10-14T16:41:38.247837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267006+0000","last_change":"2023-10-14T16:41:38.259912+0000","last_active":"2023-10-14T16:41:38.267006+0000","last_peered":"2023-10-14T16:41:38.267006+0000","last_clean":"2023-10-14T16:41:38.267006+0000","last_became_active":"2023-10-14T16:41:38.258669+0000","last_became_peered":"2023-10-14T16:41:38.258669+0000","last_unstale":"2023-10-14T16:41:38.267006+0000","last_undegraded":"2023-10-14T16:41:38.267006+0000","last_fullsized":"2023-10-14T16:41:38.267006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:16:13.242402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244395+0000","last_change":"2023-10-14T16:41:38.244395+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244395+0000","last_undegraded":"2023-10-14T16:41:38.244395+0000","last_fullsized":"2023-10-14T16:41:38.244395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.258466+0000","last_change":"2023-10-14T16:41:38.257048+0000","last_active":"2023-10-14T16:41:38.258466+0000","last_peered":"2023-10-14T16:41:38.258466+0000","last_clean":"2023-10-14T16:41:38.258466+0000","last_became_active":"2023-10-14T16:41:38.256705+0000","last_became_peered":"2023-10-14T16:41:38.256705+0000","last_unstale":"2023-10-14T16:41:38.258466+0000","last_undegraded":"2023-10-14T16:41:38.258466+0000","last_fullsized":"2023-10-14T16:41:38.258466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257970+0000","last_change":"2023-10-14T16:41:38.257970+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257970+0000","last_undegraded":"2023-10-14T16:41:38.257970+0000","last_fullsized":"2023-10-14T16:41:38.257970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272295+0000","last_change":"2023-10-14T16:41:38.267598+0000","last_active":"2023-10-14T16:41:38.272295+0000","last_peered":"2023-10-14T16:41:38.272295+0000","last_clean":"2023-10-14T16:41:38.272295+0000","last_became_active":"2023-10-14T16:41:38.266568+0000","last_became_peered":"2023-10-14T16:41:38.266568+0000","last_unstale":"2023-10-14T16:41:38.272295+0000","last_undegraded":"2023-10-14T16:41:38.272295+0000","last_fullsized":"2023-10-14T16:41:38.272295+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:34.374586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246503+0000","last_change":"2023-10-14T16:41:38.246503+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246503+0000","last_undegraded":"2023-10-14T16:41:38.246503+0000","last_fullsized":"2023-10-14T16:41:38.246503+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273736+0000","last_change":"2023-10-14T16:41:38.269698+0000","last_active":"2023-10-14T16:41:38.273736+0000","last_peered":"2023-10-14T16:41:38.273736+0000","last_clean":"2023-10-14T16:41:38.273736+0000","last_became_active":"2023-10-14T16:41:38.269391+0000","last_became_peered":"2023-10-14T16:41:38.269391+0000","last_unstale":"2023-10-14T16:41:38.273736+0000","last_undegraded":"2023-10-14T16:41:38.273736+0000","last_fullsized":"2023-10-14T16:41:38.273736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:17.441986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248995+0000","last_change":"2023-10-14T16:41:38.248995+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248995+0000","last_undegraded":"2023-10-14T16:41:38.248995+0000","last_fullsized":"2023-10-14T16:41:38.248995+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.261889+0000","last_change":"2023-10-14T16:41:38.259142+0000","last_active":"2023-10-14T16:41:38.261889+0000","last_peered":"2023-10-14T16:41:38.261889+0000","last_clean":"2023-10-14T16:41:38.261889+0000","last_became_active":"2023-10-14T16:41:38.258853+0000","last_became_peered":"2023-10-14T16:41:38.258853+0000","last_unstale":"2023-10-14T16:41:38.261889+0000","last_undegraded":"2023-10-14T16:41:38.261889+0000","last_fullsized":"2023-10-14T16:41:38.261889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244949+0000","last_change":"2023-10-14T16:41:38.244949+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244949+0000","last_undegraded":"2023-10-14T16:41:38.244949+0000","last_fullsized":"2023-10-14T16:41:38.244949+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272571+0000","last_change":"2023-10-14T16:41:38.269707+0000","last_active":"2023-10-14T16:41:38.272571+0000","last_peered":"2023-10-14T16:41:38.272571+0000","last_clean":"2023-10-14T16:41:38.272571+0000","last_became_active":"2023-10-14T16:41:38.269101+0000","last_became_peered":"2023-10-14T16:41:38.269101+0000","last_unstale":"2023-10-14T16:41:38.272571+0000","last_undegraded":"2023-10-14T16:41:38.272571+0000","last_fullsized":"2023-10-14T16:41:38.272571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250366+0000","last_change":"2023-10-14T16:41:38.250366+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250366+0000","last_undegraded":"2023-10-14T16:41:38.250366+0000","last_fullsized":"2023-10-14T16:41:38.250366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273696+0000","last_change":"2023-10-14T16:41:38.268083+0000","last_active":"2023-10-14T16:41:38.273696+0000","last_peered":"2023-10-14T16:41:38.273696+0000","last_clean":"2023-10-14T16:41:38.273696+0000","last_became_active":"2023-10-14T16:41:38.267791+0000","last_became_peered":"2023-10-14T16:41:38.267791+0000","last_unstale":"2023-10-14T16:41:38.273696+0000","last_undegraded":"2023-10-14T16:41:38.273696+0000","last_fullsized":"2023-10-14T16:41:38.273696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251138+0000","last_change":"2023-10-14T16:41:38.251138+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251138+0000","last_undegraded":"2023-10-14T16:41:38.251138+0000","last_fullsized":"2023-10-14T16:41:38.251138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266868+0000","last_change":"2023-10-14T16:41:38.259070+0000","last_active":"2023-10-14T16:41:38.266868+0000","last_peered":"2023-10-14T16:41:38.266868+0000","last_clean":"2023-10-14T16:41:38.266868+0000","last_became_active":"2023-10-14T16:41:38.258658+0000","last_became_peered":"2023-10-14T16:41:38.258658+0000","last_unstale":"2023-10-14T16:41:38.266868+0000","last_undegraded":"2023-10-14T16:41:38.266868+0000","last_fullsized":"2023-10-14T16:41:38.266868+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244826+0000","last_change":"2023-10-14T16:41:38.244826+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244826+0000","last_undegraded":"2023-10-14T16:41:38.244826+0000","last_fullsized":"2023-10-14T16:41:38.244826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269845+0000","last_change":"2023-10-14T16:41:38.265796+0000","last_active":"2023-10-14T16:41:38.269845+0000","last_peered":"2023-10-14T16:41:38.269845+0000","last_clean":"2023-10-14T16:41:38.269845+0000","last_became_active":"2023-10-14T16:41:38.265491+0000","last_became_peered":"2023-10-14T16:41:38.265491+0000","last_unstale":"2023-10-14T16:41:38.269845+0000","last_undegraded":"2023-10-14T16:41:38.269845+0000","last_fullsized":"2023-10-14T16:41:38.269845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:25.436192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259056+0000","last_change":"2023-10-14T16:41:38.259056+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259056+0000","last_undegraded":"2023-10-14T16:41:38.259056+0000","last_fullsized":"2023-10-14T16:41:38.259056+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274094+0000","last_change":"2023-10-14T16:41:38.271323+0000","last_active":"2023-10-14T16:41:38.274094+0000","last_peered":"2023-10-14T16:41:38.274094+0000","last_clean":"2023-10-14T16:41:38.274094+0000","last_became_active":"2023-10-14T16:41:38.271014+0000","last_became_peered":"2023-10-14T16:41:38.271014+0000","last_unstale":"2023-10-14T16:41:38.274094+0000","last_undegraded":"2023-10-14T16:41:38.274094+0000","last_fullsized":"2023-10-14T16:41:38.274094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242243+0000","last_change":"2023-10-14T16:41:38.242243+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242243+0000","last_undegraded":"2023-10-14T16:41:38.242243+0000","last_fullsized":"2023-10-14T16:41:38.242243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274622+0000","last_change":"2023-10-14T16:41:38.272215+0000","last_active":"2023-10-14T16:41:38.274622+0000","last_peered":"2023-10-14T16:41:38.274622+0000","last_clean":"2023-10-14T16:41:38.274622+0000","last_became_active":"2023-10-14T16:41:38.271943+0000","last_became_peered":"2023-10-14T16:41:38.271943+0000","last_unstale":"2023-10-14T16:41:38.274622+0000","last_undegraded":"2023-10-14T16:41:38.274622+0000","last_fullsized":"2023-10-14T16:41:38.274622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253161+0000","last_change":"2023-10-14T16:41:38.253161+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253161+0000","last_undegraded":"2023-10-14T16:41:38.253161+0000","last_fullsized":"2023-10-14T16:41:38.253161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270665+0000","last_change":"2023-10-14T16:41:38.267446+0000","last_active":"2023-10-14T16:41:38.270665+0000","last_peered":"2023-10-14T16:41:38.270665+0000","last_clean":"2023-10-14T16:41:38.270665+0000","last_became_active":"2023-10-14T16:41:38.267122+0000","last_became_peered":"2023-10-14T16:41:38.267122+0000","last_unstale":"2023-10-14T16:41:38.270665+0000","last_undegraded":"2023-10-14T16:41:38.270665+0000","last_fullsized":"2023-10-14T16:41:38.270665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246383+0000","last_change":"2023-10-14T16:41:38.246383+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246383+0000","last_undegraded":"2023-10-14T16:41:38.246383+0000","last_fullsized":"2023-10-14T16:41:38.246383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268444+0000","last_change":"2023-10-14T16:41:38.266145+0000","last_active":"2023-10-14T16:41:38.268444+0000","last_peered":"2023-10-14T16:41:38.268444+0000","last_clean":"2023-10-14T16:41:38.268444+0000","last_became_active":"2023-10-14T16:41:38.264561+0000","last_became_peered":"2023-10-14T16:41:38.264561+0000","last_unstale":"2023-10-14T16:41:38.268444+0000","last_undegraded":"2023-10-14T16:41:38.268444+0000","last_fullsized":"2023-10-14T16:41:38.268444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:45:54.740964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256466+0000","last_change":"2023-10-14T16:41:38.256466+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256466+0000","last_undegraded":"2023-10-14T16:41:38.256466+0000","last_fullsized":"2023-10-14T16:41:38.256466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268496+0000","last_change":"2023-10-14T16:41:38.266046+0000","last_active":"2023-10-14T16:41:38.268496+0000","last_peered":"2023-10-14T16:41:38.268496+0000","last_clean":"2023-10-14T16:41:38.268496+0000","last_became_active":"2023-10-14T16:41:38.263725+0000","last_became_peered":"2023-10-14T16:41:38.263725+0000","last_unstale":"2023-10-14T16:41:38.268496+0000","last_undegraded":"2023-10-14T16:41:38.268496+0000","last_fullsized":"2023-10-14T16:41:38.268496+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:44.086979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252808+0000","last_change":"2023-10-14T16:41:38.252808+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252808+0000","last_undegraded":"2023-10-14T16:41:38.252808+0000","last_fullsized":"2023-10-14T16:41:38.252808+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268294+0000","last_change":"2023-10-14T16:41:38.263662+0000","last_active":"2023-10-14T16:41:38.268294+0000","last_peered":"2023-10-14T16:41:38.268294+0000","last_clean":"2023-10-14T16:41:38.268294+0000","last_became_active":"2023-10-14T16:41:38.262441+0000","last_became_peered":"2023-10-14T16:41:38.262441+0000","last_unstale":"2023-10-14T16:41:38.268294+0000","last_undegraded":"2023-10-14T16:41:38.268294+0000","last_fullsized":"2023-10-14T16:41:38.268294+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:54.579331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245609+0000","last_change":"2023-10-14T16:41:38.245609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245609+0000","last_undegraded":"2023-10-14T16:41:38.245609+0000","last_fullsized":"2023-10-14T16:41:38.245609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273039+0000","last_change":"2023-10-14T16:41:38.270622+0000","last_active":"2023-10-14T16:41:38.273039+0000","last_peered":"2023-10-14T16:41:38.273039+0000","last_clean":"2023-10-14T16:41:38.273039+0000","last_became_active":"2023-10-14T16:41:38.269459+0000","last_became_peered":"2023-10-14T16:41:38.269459+0000","last_unstale":"2023-10-14T16:41:38.273039+0000","last_undegraded":"2023-10-14T16:41:38.273039+0000","last_fullsized":"2023-10-14T16:41:38.273039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247414+0000","last_change":"2023-10-14T16:41:38.247414+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247414+0000","last_undegraded":"2023-10-14T16:41:38.247414+0000","last_fullsized":"2023-10-14T16:41:38.247414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274818+0000","last_change":"2023-10-14T16:41:38.270414+0000","last_active":"2023-10-14T16:41:38.274818+0000","last_peered":"2023-10-14T16:41:38.274818+0000","last_clean":"2023-10-14T16:41:38.274818+0000","last_became_active":"2023-10-14T16:41:38.269445+0000","last_became_peered":"2023-10-14T16:41:38.269445+0000","last_unstale":"2023-10-14T16:41:38.274818+0000","last_undegraded":"2023-10-14T16:41:38.274818+0000","last_fullsized":"2023-10-14T16:41:38.274818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247587+0000","last_change":"2023-10-14T16:41:38.247587+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247587+0000","last_undegraded":"2023-10-14T16:41:38.247587+0000","last_fullsized":"2023-10-14T16:41:38.247587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.275360+0000","last_change":"2023-10-14T16:41:38.271734+0000","last_active":"2023-10-14T16:41:38.275360+0000","last_peered":"2023-10-14T16:41:38.275360+0000","last_clean":"2023-10-14T16:41:38.275360+0000","last_became_active":"2023-10-14T16:41:38.271386+0000","last_became_peered":"2023-10-14T16:41:38.271386+0000","last_unstale":"2023-10-14T16:41:38.275360+0000","last_undegraded":"2023-10-14T16:41:38.275360+0000","last_fullsized":"2023-10-14T16:41:38.275360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:46:16.492082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263609+0000","last_change":"2023-10-14T16:41:38.263609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263609+0000","last_undegraded":"2023-10-14T16:41:38.263609+0000","last_fullsized":"2023-10-14T16:41:38.263609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270337+0000","last_change":"2023-10-14T16:41:38.266162+0000","last_active":"2023-10-14T16:41:38.270337+0000","last_peered":"2023-10-14T16:41:38.270337+0000","last_clean":"2023-10-14T16:41:38.270337+0000","last_became_active":"2023-10-14T16:41:38.265806+0000","last_became_peered":"2023-10-14T16:41:38.265806+0000","last_unstale":"2023-10-14T16:41:38.270337+0000","last_undegraded":"2023-10-14T16:41:38.270337+0000","last_fullsized":"2023-10-14T16:41:38.270337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252719+0000","last_change":"2023-10-14T16:41:38.252719+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252719+0000","last_undegraded":"2023-10-14T16:41:38.252719+0000","last_fullsized":"2023-10-14T16:41:38.252719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273662+0000","last_change":"2023-10-14T16:41:38.271489+0000","last_active":"2023-10-14T16:41:38.273662+0000","last_peered":"2023-10-14T16:41:38.273662+0000","last_clean":"2023-10-14T16:41:38.273662+0000","last_became_active":"2023-10-14T16:41:38.271096+0000","last_became_peered":"2023-10-14T16:41:38.271096+0000","last_unstale":"2023-10-14T16:41:38.273662+0000","last_undegraded":"2023-10-14T16:41:38.273662+0000","last_fullsized":"2023-10-14T16:41:38.273662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244568+0000","last_change":"2023-10-14T16:41:38.244568+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244568+0000","last_undegraded":"2023-10-14T16:41:38.244568+0000","last_fullsized":"2023-10-14T16:41:38.244568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272453+0000","last_change":"2023-10-14T16:41:38.267492+0000","last_active":"2023-10-14T16:41:38.272453+0000","last_peered":"2023-10-14T16:41:38.272453+0000","last_clean":"2023-10-14T16:41:38.272453+0000","last_became_active":"2023-10-14T16:41:38.267208+0000","last_became_peered":"2023-10-14T16:41:38.267208+0000","last_unstale":"2023-10-14T16:41:38.272453+0000","last_undegraded":"2023-10-14T16:41:38.272453+0000","last_fullsized":"2023-10-14T16:41:38.272453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:07.318539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243653+0000","last_change":"2023-10-14T16:41:38.243653+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243653+0000","last_undegraded":"2023-10-14T16:41:38.243653+0000","last_fullsized":"2023-10-14T16:41:38.243653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272435+0000","last_change":"2023-10-14T16:41:38.269649+0000","last_active":"2023-10-14T16:41:38.272435+0000","last_peered":"2023-10-14T16:41:38.272435+0000","last_clean":"2023-10-14T16:41:38.272435+0000","last_became_active":"2023-10-14T16:41:38.269092+0000","last_became_peered":"2023-10-14T16:41:38.269092+0000","last_unstale":"2023-10-14T16:41:38.272435+0000","last_undegraded":"2023-10-14T16:41:38.272435+0000","last_fullsized":"2023-10-14T16:41:38.272435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.241898+0000","last_change":"2023-10-14T16:41:38.241898+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.241898+0000","last_undegraded":"2023-10-14T16:41:38.241898+0000","last_fullsized":"2023-10-14T16:41:38.241898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.262512+0000","last_change":"2023-10-14T16:41:38.260866+0000","last_active":"2023-10-14T16:41:38.262512+0000","last_peered":"2023-10-14T16:41:38.262512+0000","last_clean":"2023-10-14T16:41:38.262512+0000","last_became_active":"2023-10-14T16:41:38.260498+0000","last_became_peered":"2023-10-14T16:41:38.260498+0000","last_unstale":"2023-10-14T16:41:38.262512+0000","last_undegraded":"2023-10-14T16:41:38.262512+0000","last_fullsized":"2023-10-14T16:41:38.262512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246257+0000","last_change":"2023-10-14T16:41:38.246257+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246257+0000","last_undegraded":"2023-10-14T16:41:38.246257+0000","last_fullsized":"2023-10-14T16:41:38.246257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269808+0000","last_change":"2023-10-14T16:41:38.265778+0000","last_active":"2023-10-14T16:41:38.269808+0000","last_peered":"2023-10-14T16:41:38.269808+0000","last_clean":"2023-10-14T16:41:38.269808+0000","last_became_active":"2023-10-14T16:41:38.265477+0000","last_became_peered":"2023-10-14T16:41:38.265477+0000","last_unstale":"2023-10-14T16:41:38.269808+0000","last_undegraded":"2023-10-14T16:41:38.269808+0000","last_fullsized":"2023-10-14T16:41:38.269808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246290+0000","last_change":"2023-10-14T16:41:38.246290+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246290+0000","last_undegraded":"2023-10-14T16:41:38.246290+0000","last_fullsized":"2023-10-14T16:41:38.246290+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268405+0000","last_change":"2023-10-14T16:41:38.265852+0000","last_active":"2023-10-14T16:41:38.268405+0000","last_peered":"2023-10-14T16:41:38.268405+0000","last_clean":"2023-10-14T16:41:38.268405+0000","last_became_active":"2023-10-14T16:41:38.265513+0000","last_became_peered":"2023-10-14T16:41:38.265513+0000","last_unstale":"2023-10-14T16:41:38.268405+0000","last_undegraded":"2023-10-14T16:41:38.268405+0000","last_fullsized":"2023-10-14T16:41:38.268405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:44:33.854697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247745+0000","last_change":"2023-10-14T16:41:38.247745+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247745+0000","last_undegraded":"2023-10-14T16:41:38.247745+0000","last_fullsized":"2023-10-14T16:41:38.247745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266793+0000","last_change":"2023-10-14T16:41:38.265704+0000","last_active":"2023-10-14T16:41:38.266793+0000","last_peered":"2023-10-14T16:41:38.266793+0000","last_clean":"2023-10-14T16:41:38.266793+0000","last_became_active":"2023-10-14T16:41:38.264927+0000","last_became_peered":"2023-10-14T16:41:38.264927+0000","last_unstale":"2023-10-14T16:41:38.266793+0000","last_undegraded":"2023-10-14T16:41:38.266793+0000","last_fullsized":"2023-10-14T16:41:38.266793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256689+0000","last_change":"2023-10-14T16:41:38.256689+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256689+0000","last_undegraded":"2023-10-14T16:41:38.256689+0000","last_fullsized":"2023-10-14T16:41:38.256689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268528+0000","last_change":"2023-10-14T16:41:38.267370+0000","last_active":"2023-10-14T16:41:38.268528+0000","last_peered":"2023-10-14T16:41:38.268528+0000","last_clean":"2023-10-14T16:41:38.268528+0000","last_became_active":"2023-10-14T16:41:38.267087+0000","last_became_peered":"2023-10-14T16:41:38.267087+0000","last_unstale":"2023-10-14T16:41:38.268528+0000","last_undegraded":"2023-10-14T16:41:38.268528+0000","last_fullsized":"2023-10-14T16:41:38.268528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T16:41:37.236449+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:41:37.236449+0000","last_peered":"2023-10-14T16:41:37.236449+0000","last_clean":"2023-10-14T16:41:37.236449+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:41:37.236449+0000","last_undegraded":"2023-10-14T16:41:37.236449+0000","last_fullsized":"2023-10-14T16:41:37.236449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.034},{"interface":"front","average":{"1min":1.173,"5min":1.173,"15min":1.173},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.272}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.125}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.3069999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.179},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.6020000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":1.575},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.427}]},{"osd":6,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":1.77}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":1.627},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.0720000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":1.429},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.51}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":1.29},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.6659999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":3.0110000000000001,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":1.754},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":3.206,"5min":3.206,"15min":3.206},"last":1.476}]},{"osd":7,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.843}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":139168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":0.98199999999999998}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":1.716}]},{"osd":2,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.3639999999999999,"5min":4.3639999999999999,"15min":4.3639999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":1.1120000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.823},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":1.4590000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.212},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.893}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.9470000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"last":1.6799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":1.093},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":4.4059999999999997},"last":1.149}]},{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.4510000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.876},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.98399999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.194}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.48},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.3200000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.29},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.3680000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.81200000000000006}]},{"osd":1,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.024},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":0.95399999999999996}]},{"osd":4,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":1.101},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":1.0549999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.2490000000000001,"5min":3.2490000000000001,"15min":3.2490000000000001},"last":1.179}]},{"osd":6,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.121}]},{"osd":7,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.236}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.5},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.2410000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.3080000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.583},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.006},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":1.518}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.1200000000000001,"5min":3.1200000000000001,"15min":3.1200000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.1060000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.054}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.79600000000000004}]},{"osd":2,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.3759999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.603},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":0.98299999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.429},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.526}]},{"osd":6,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":4.6360000000000001,"5min":4.6360000000000001,"15min":4.6360000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.4970000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.667},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.4650000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.05}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.84999999999999998}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.042},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.88200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:41:44.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:41:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.492+0000 7f1a7138a640 1 Processor -- start 2023-10-14T16:41:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.492+0000 7f1a7138a640 1 -- start start 2023-10-14T16:41:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.493+0000 7f1a7138a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a6c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.493+0000 7f1a7138a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06f7f0 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.493+0000 7f1a7138a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a6c06c5e0 con 0x7f1a6c159f00 2023-10-14T16:41:44.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.493+0000 7f1a7138a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c06c8c0 con 0x7f1a6c159b20 2023-10-14T16:41:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.494+0000 7f1a6b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a6c0b0b20 0x7f1a6c06fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45198/0 (socket says 172.21.15.29:45198) 2023-10-14T16:41:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.494+0000 7f1a6b7fe640 1 -- 172.21.15.29:0/2636005089 learned_addr learned my addr 172.21.15.29:0/2636005089 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.494+0000 7f1a6affd640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a6c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.494+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288521270 0 0) 0x7f1a6c06f7f0 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.494+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a58003520 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522558544 0 0) 0x7f1a6c06c5e0 con 0x7f1a6c159f00 2023-10-14T16:41:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a6c06f7f0 con 0x7f1a6c159f00 2023-10-14T16:41:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3016953736 0 0) 0x7f1a58003520 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 msgr2=0x7f1a6c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a6c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 >> v1:172.21.15.186:6789/0 conn(0x7f1a6c159f00 legacy=0x7f1a6c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c06cba0 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1a64001670 con 0x7f1a6c0b0b20 2023-10-14T16:41:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 >> v1:172.21.15.29:6789/0 conn(0x7f1a6c0b0b20 legacy=0x7f1a6c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a58003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a580047c0 0x7f1a58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c06c8c0 con 0x7f1a58003b30 2023-10-14T16:41:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5800ba30 con 0x7f1a580047c0 2023-10-14T16:41:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a5800bbc0 con 0x7f1a6c159b20 2023-10-14T16:41:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a6affd640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a58003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.495+0000 7f1a6b7fe640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6a7fc640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a580047c0 0x7f1a58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6affd640 1 -- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 msgr2=0x7f1a58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6affd640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c159b20 0x7f1a58005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6affd640 1 -- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a580047c0 msgr2=0x7f1a58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6affd640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a580047c0 0x7f1a58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.496+0000 7f1a6affd640 1 -- 172.21.15.29:0/2636005089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a58003520 con 0x7f1a58003b30 2023-10-14T16:41:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.497+0000 7f1a6affd640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a58003f30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009fc0 tx=0x7f1a580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c50c15e22732dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.497+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c02cba0 con 0x7f1a58003b30 2023-10-14T16:41:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c00f040 con 0x7f1a58003b30 2023-10-14T16:41:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a69ffb640 1 -- 172.21.15.29:0/2636005089 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c034440 con 0x7f1a58003b30 2023-10-14T16:41:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 -- 172.21.15.29:0/2636005089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 msgr2=0x7f1a58003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 --2- 172.21.15.29:0/2636005089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a58003f30 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c009fc0 tx=0x7f1a580057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a6b7fe640 1 -- 172.21.15.29:0/2636005089 reap_dead start 2023-10-14T16:41:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 -- 172.21.15.29:0/2636005089 shutdown_connections 2023-10-14T16:41:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 -- 172.21.15.29:0/2636005089 >> 172.21.15.29:0/2636005089 conn(0x7f1a6c0cb410 msgr2=0x7f1a6c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 -- 172.21.15.29:0/2636005089 shutdown_connections 2023-10-14T16:41:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.498+0000 7f1a7138a640 1 -- 172.21.15.29:0/2636005089 wait complete. 2023-10-14T16:41:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.500+0000 7f1a7138a640 1 Processor -- start 2023-10-14T16:41:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.500+0000 7f1a7138a640 1 -- start start 2023-10-14T16:41:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a6c15c530 0x7f1a6c15c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 0x7f1a6c0b41b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c15cef0 con 0x7f1a58003b30 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0c1f30 con 0x7f1a6c15c530 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.501+0000 7f1a7138a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a6c0c2e00 con 0x7f1a6c0b3d30 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39382/0 (socket says 172.21.15.29:39382) 2023-10-14T16:41:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 -- 172.21.15.29:0/627827552 learned_addr learned my addr 172.21.15.29:0/627827552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 -- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 msgr2=0x7f1a6c0b41b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6b7fe640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 0x7f1a6c0b41b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6a7fc640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a6c15c530 0x7f1a6c15c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 0x7f1a6c0b41b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 -- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a6c15c530 msgr2=0x7f1a6c15c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a6c15c530 0x7f1a6c15c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6affd640 1 -- 172.21.15.29:0/627827552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c0b46f0 con 0x7f1a58003b30 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.502+0000 7f1a6b7fe640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 0x7f1a6c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a6affd640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c02c610 tx=0x7f1a5c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c034bd0 con 0x7f1a58003b30 2023-10-14T16:41:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a5c034d30 con 0x7f1a58003b30 2023-10-14T16:41:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c034ea0 con 0x7f1a58003b30 2023-10-14T16:41:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a6c0c2180 con 0x7f1a58003b30 2023-10-14T16:41:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.503+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a6c0c2740 con 0x7f1a58003b30 2023-10-14T16:41:44.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.505+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a6c04e160 con 0x7f1a58003b30 2023-10-14T16:41:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.506+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f1a5c040450 con 0x7f1a58003b30 2023-10-14T16:41:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.507+0000 7f1a637fe640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 0x7f1a4c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.507+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f1a5c0abd90 con 0x7f1a58003b30 2023-10-14T16:41:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.507+0000 7f1a6a7fc640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 0x7f1a4c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:44.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.508+0000 7f1a6a7fc640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 0x7f1a4c06d360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a54007fd0 tx=0x7f1a54009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:44.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.511+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a5c076ff0 con 0x7f1a58003b30 2023-10-14T16:41:44.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.703+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1a6c06ce80 con 0x7f1a4c06aea0 2023-10-14T16:41:44.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.724+0000 7f1a637fe640 1 -- 172.21.15.29:0/627827552 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+281613 (secure 0 0 0) 0x7f1a5400d050 con 0x7f1a4c06aea0 2023-10-14T16:41:44.729 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:44.731 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:41:44.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.731+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 msgr2=0x7f1a4c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.731+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 0x7f1a4c06d360 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1a54007fd0 tx=0x7f1a54009040 comp rx=0 tx=0).stop 2023-10-14T16:41:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.731+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 msgr2=0x7f1a6c14fd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.731+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c02c610 tx=0x7f1a5c009750 comp rx=0 tx=0).stop 2023-10-14T16:41:44.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 shutdown_connections 2023-10-14T16:41:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f1a4c06aea0 0x7f1a4c06d360 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a6c0b3d30 0x7f1a6c0b41b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a6c15c530 0x7f1a6c15c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 --2- 172.21.15.29:0/627827552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a58003b30 0x7f1a6c14fd10 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:44.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 >> 172.21.15.29:0/627827552 conn(0x7f1a6c0cb410 msgr2=0x7f1a6c070690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:44.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 shutdown_connections 2023-10-14T16:41:44.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:44.732+0000 7f1a7138a640 1 -- 172.21.15.29:0/627827552 wait complete. 2023-10-14T16:41:44.752 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":72,"stamp":"2023-10-14T16:41:43.322343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":8,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":236696,"kb_used_data":3892,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754738024,"statfs":{"total":773094113280,"available":772851736576,"internally_reserved":0,"allocated":3985408,"data_stored":2250211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":35,"apply_latency_ms":35,"commit_latency_ns":35000000,"apply_latency_ns":35000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001496"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251935+0000","last_change":"2023-10-14T16:41:38.251935+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251935+0000","last_undegraded":"2023-10-14T16:41:38.251935+0000","last_fullsized":"2023-10-14T16:41:38.251935+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251427+0000","last_change":"2023-10-14T16:41:38.251427+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251427+0000","last_undegraded":"2023-10-14T16:41:38.251427+0000","last_fullsized":"2023-10-14T16:41:38.251427+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252219+0000","last_change":"2023-10-14T16:41:38.252219+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252219+0000","last_undegraded":"2023-10-14T16:41:38.252219+0000","last_fullsized":"2023-10-14T16:41:38.252219+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258644+0000","last_change":"2023-10-14T16:41:38.258644+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258644+0000","last_undegraded":"2023-10-14T16:41:38.258644+0000","last_fullsized":"2023-10-14T16:41:38.258644+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248994+0000","last_change":"2023-10-14T16:41:38.248994+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248994+0000","last_undegraded":"2023-10-14T16:41:38.248994+0000","last_fullsized":"2023-10-14T16:41:38.248994+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247097+0000","last_change":"2023-10-14T16:41:38.247097+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247097+0000","last_undegraded":"2023-10-14T16:41:38.247097+0000","last_fullsized":"2023-10-14T16:41:38.247097+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263052+0000","last_change":"2023-10-14T16:41:38.263052+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263052+0000","last_undegraded":"2023-10-14T16:41:38.263052+0000","last_fullsized":"2023-10-14T16:41:38.263052+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260191+0000","last_change":"2023-10-14T16:41:38.260191+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260191+0000","last_undegraded":"2023-10-14T16:41:38.260191+0000","last_fullsized":"2023-10-14T16:41:38.260191+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249866+0000","last_change":"2023-10-14T16:41:38.249866+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249866+0000","last_undegraded":"2023-10-14T16:41:38.249866+0000","last_fullsized":"2023-10-14T16:41:38.249866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253885+0000","last_change":"2023-10-14T16:41:38.253885+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253885+0000","last_undegraded":"2023-10-14T16:41:38.253885+0000","last_fullsized":"2023-10-14T16:41:38.253885+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.261410+0000","last_change":"2023-10-14T16:41:38.261410+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.261410+0000","last_undegraded":"2023-10-14T16:41:38.261410+0000","last_fullsized":"2023-10-14T16:41:38.261410+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250453+0000","last_change":"2023-10-14T16:41:38.250453+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250453+0000","last_undegraded":"2023-10-14T16:41:38.250453+0000","last_fullsized":"2023-10-14T16:41:38.250453+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248204+0000","last_change":"2023-10-14T16:41:38.248204+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248204+0000","last_undegraded":"2023-10-14T16:41:38.248204+0000","last_fullsized":"2023-10-14T16:41:38.248204+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246426+0000","last_change":"2023-10-14T16:41:38.246426+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246426+0000","last_undegraded":"2023-10-14T16:41:38.246426+0000","last_fullsized":"2023-10-14T16:41:38.246426+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248085+0000","last_change":"2023-10-14T16:41:38.248085+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248085+0000","last_undegraded":"2023-10-14T16:41:38.248085+0000","last_fullsized":"2023-10-14T16:41:38.248085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257273+0000","last_change":"2023-10-14T16:41:38.257273+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257273+0000","last_undegraded":"2023-10-14T16:41:38.257273+0000","last_fullsized":"2023-10-14T16:41:38.257273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.262216+0000","last_change":"2023-10-14T16:41:38.262216+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.262216+0000","last_undegraded":"2023-10-14T16:41:38.262216+0000","last_fullsized":"2023-10-14T16:41:38.262216+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250965+0000","last_change":"2023-10-14T16:41:38.250965+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250965+0000","last_undegraded":"2023-10-14T16:41:38.250965+0000","last_fullsized":"2023-10-14T16:41:38.250965+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245112+0000","last_change":"2023-10-14T16:41:38.245112+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245112+0000","last_undegraded":"2023-10-14T16:41:38.245112+0000","last_fullsized":"2023-10-14T16:41:38.245112+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249462+0000","last_change":"2023-10-14T16:41:38.249462+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249462+0000","last_undegraded":"2023-10-14T16:41:38.249462+0000","last_fullsized":"2023-10-14T16:41:38.249462+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252360+0000","last_change":"2023-10-14T16:41:38.252360+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252360+0000","last_undegraded":"2023-10-14T16:41:38.252360+0000","last_fullsized":"2023-10-14T16:41:38.252360+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253610+0000","last_change":"2023-10-14T16:41:38.253610+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253610+0000","last_undegraded":"2023-10-14T16:41:38.253610+0000","last_fullsized":"2023-10-14T16:41:38.253610+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243316+0000","last_change":"2023-10-14T16:41:38.243316+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243316+0000","last_undegraded":"2023-10-14T16:41:38.243316+0000","last_fullsized":"2023-10-14T16:41:38.243316+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246964+0000","last_change":"2023-10-14T16:41:38.246964+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246964+0000","last_undegraded":"2023-10-14T16:41:38.246964+0000","last_fullsized":"2023-10-14T16:41:38.246964+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256479+0000","last_change":"2023-10-14T16:41:38.256479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256479+0000","last_undegraded":"2023-10-14T16:41:38.256479+0000","last_fullsized":"2023-10-14T16:41:38.256479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245009+0000","last_change":"2023-10-14T16:41:38.245009+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245009+0000","last_undegraded":"2023-10-14T16:41:38.245009+0000","last_fullsized":"2023-10-14T16:41:38.245009+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.258281+0000","last_change":"2023-10-14T16:41:38.258281+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.258281+0000","last_undegraded":"2023-10-14T16:41:38.258281+0000","last_fullsized":"2023-10-14T16:41:38.258281+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.249479+0000","last_change":"2023-10-14T16:41:38.249479+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.249479+0000","last_undegraded":"2023-10-14T16:41:38.249479+0000","last_fullsized":"2023-10-14T16:41:38.249479+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247132+0000","last_change":"2023-10-14T16:41:38.247132+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247132+0000","last_undegraded":"2023-10-14T16:41:38.247132+0000","last_fullsized":"2023-10-14T16:41:38.247132+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.260202+0000","last_change":"2023-10-14T16:41:38.260202+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.260202+0000","last_undegraded":"2023-10-14T16:41:38.260202+0000","last_fullsized":"2023-10-14T16:41:38.260202+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246717+0000","last_change":"2023-10-14T16:41:38.246717+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246717+0000","last_undegraded":"2023-10-14T16:41:38.246717+0000","last_fullsized":"2023-10-14T16:41:38.246717+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259641+0000","last_change":"2023-10-14T16:41:38.259641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259641+0000","last_undegraded":"2023-10-14T16:41:38.259641+0000","last_fullsized":"2023-10-14T16:41:38.259641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244200+0000","last_change":"2023-10-14T16:41:38.244200+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244200+0000","last_undegraded":"2023-10-14T16:41:38.244200+0000","last_fullsized":"2023-10-14T16:41:38.244200+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267873+0000","last_change":"2023-10-14T16:41:38.263891+0000","last_active":"2023-10-14T16:41:38.267873+0000","last_peered":"2023-10-14T16:41:38.267873+0000","last_clean":"2023-10-14T16:41:38.267873+0000","last_became_active":"2023-10-14T16:41:38.263622+0000","last_became_peered":"2023-10-14T16:41:38.263622+0000","last_unstale":"2023-10-14T16:41:38.267873+0000","last_undegraded":"2023-10-14T16:41:38.267873+0000","last_fullsized":"2023-10-14T16:41:38.267873+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242058+0000","last_change":"2023-10-14T16:41:38.242058+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242058+0000","last_undegraded":"2023-10-14T16:41:38.242058+0000","last_fullsized":"2023-10-14T16:41:38.242058+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272429+0000","last_change":"2023-10-14T16:41:38.266472+0000","last_active":"2023-10-14T16:41:38.272429+0000","last_peered":"2023-10-14T16:41:38.272429+0000","last_clean":"2023-10-14T16:41:38.272429+0000","last_became_active":"2023-10-14T16:41:38.266130+0000","last_became_peered":"2023-10-14T16:41:38.266130+0000","last_unstale":"2023-10-14T16:41:38.272429+0000","last_undegraded":"2023-10-14T16:41:38.272429+0000","last_fullsized":"2023-10-14T16:41:38.272429+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248641+0000","last_change":"2023-10-14T16:41:38.248641+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248641+0000","last_undegraded":"2023-10-14T16:41:38.248641+0000","last_fullsized":"2023-10-14T16:41:38.248641+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274644+0000","last_change":"2023-10-14T16:41:38.270437+0000","last_active":"2023-10-14T16:41:38.274644+0000","last_peered":"2023-10-14T16:41:38.274644+0000","last_clean":"2023-10-14T16:41:38.274644+0000","last_became_active":"2023-10-14T16:41:38.270157+0000","last_became_peered":"2023-10-14T16:41:38.270157+0000","last_unstale":"2023-10-14T16:41:38.274644+0000","last_undegraded":"2023-10-14T16:41:38.274644+0000","last_fullsized":"2023-10-14T16:41:38.274644+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251155+0000","last_change":"2023-10-14T16:41:38.251155+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251155+0000","last_undegraded":"2023-10-14T16:41:38.251155+0000","last_fullsized":"2023-10-14T16:41:38.251155+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272894+0000","last_change":"2023-10-14T16:41:38.271123+0000","last_active":"2023-10-14T16:41:38.272894+0000","last_peered":"2023-10-14T16:41:38.272894+0000","last_clean":"2023-10-14T16:41:38.272894+0000","last_became_active":"2023-10-14T16:41:38.270752+0000","last_became_peered":"2023-10-14T16:41:38.270752+0000","last_unstale":"2023-10-14T16:41:38.272894+0000","last_undegraded":"2023-10-14T16:41:38.272894+0000","last_fullsized":"2023-10-14T16:41:38.272894+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259712+0000","last_change":"2023-10-14T16:41:38.259712+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259712+0000","last_undegraded":"2023-10-14T16:41:38.259712+0000","last_fullsized":"2023-10-14T16:41:38.259712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268235+0000","last_change":"2023-10-14T16:41:38.263100+0000","last_active":"2023-10-14T16:41:38.268235+0000","last_peered":"2023-10-14T16:41:38.268235+0000","last_clean":"2023-10-14T16:41:38.268235+0000","last_became_active":"2023-10-14T16:41:38.262254+0000","last_became_peered":"2023-10-14T16:41:38.262254+0000","last_unstale":"2023-10-14T16:41:38.268235+0000","last_undegraded":"2023-10-14T16:41:38.268235+0000","last_fullsized":"2023-10-14T16:41:38.268235+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:11.858605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247837+0000","last_change":"2023-10-14T16:41:38.247837+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247837+0000","last_undegraded":"2023-10-14T16:41:38.247837+0000","last_fullsized":"2023-10-14T16:41:38.247837+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.267006+0000","last_change":"2023-10-14T16:41:38.259912+0000","last_active":"2023-10-14T16:41:38.267006+0000","last_peered":"2023-10-14T16:41:38.267006+0000","last_clean":"2023-10-14T16:41:38.267006+0000","last_became_active":"2023-10-14T16:41:38.258669+0000","last_became_peered":"2023-10-14T16:41:38.258669+0000","last_unstale":"2023-10-14T16:41:38.267006+0000","last_undegraded":"2023-10-14T16:41:38.267006+0000","last_fullsized":"2023-10-14T16:41:38.267006+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:16:13.242402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244395+0000","last_change":"2023-10-14T16:41:38.244395+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244395+0000","last_undegraded":"2023-10-14T16:41:38.244395+0000","last_fullsized":"2023-10-14T16:41:38.244395+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.258466+0000","last_change":"2023-10-14T16:41:38.257048+0000","last_active":"2023-10-14T16:41:38.258466+0000","last_peered":"2023-10-14T16:41:38.258466+0000","last_clean":"2023-10-14T16:41:38.258466+0000","last_became_active":"2023-10-14T16:41:38.256705+0000","last_became_peered":"2023-10-14T16:41:38.256705+0000","last_unstale":"2023-10-14T16:41:38.258466+0000","last_undegraded":"2023-10-14T16:41:38.258466+0000","last_fullsized":"2023-10-14T16:41:38.258466+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.257970+0000","last_change":"2023-10-14T16:41:38.257970+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.257970+0000","last_undegraded":"2023-10-14T16:41:38.257970+0000","last_fullsized":"2023-10-14T16:41:38.257970+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272295+0000","last_change":"2023-10-14T16:41:38.267598+0000","last_active":"2023-10-14T16:41:38.272295+0000","last_peered":"2023-10-14T16:41:38.272295+0000","last_clean":"2023-10-14T16:41:38.272295+0000","last_became_active":"2023-10-14T16:41:38.266568+0000","last_became_peered":"2023-10-14T16:41:38.266568+0000","last_unstale":"2023-10-14T16:41:38.272295+0000","last_undegraded":"2023-10-14T16:41:38.272295+0000","last_fullsized":"2023-10-14T16:41:38.272295+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:34.374586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246503+0000","last_change":"2023-10-14T16:41:38.246503+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246503+0000","last_undegraded":"2023-10-14T16:41:38.246503+0000","last_fullsized":"2023-10-14T16:41:38.246503+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273736+0000","last_change":"2023-10-14T16:41:38.269698+0000","last_active":"2023-10-14T16:41:38.273736+0000","last_peered":"2023-10-14T16:41:38.273736+0000","last_clean":"2023-10-14T16:41:38.273736+0000","last_became_active":"2023-10-14T16:41:38.269391+0000","last_became_peered":"2023-10-14T16:41:38.269391+0000","last_unstale":"2023-10-14T16:41:38.273736+0000","last_undegraded":"2023-10-14T16:41:38.273736+0000","last_fullsized":"2023-10-14T16:41:38.273736+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:17.441986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.248995+0000","last_change":"2023-10-14T16:41:38.248995+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.248995+0000","last_undegraded":"2023-10-14T16:41:38.248995+0000","last_fullsized":"2023-10-14T16:41:38.248995+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.261889+0000","last_change":"2023-10-14T16:41:38.259142+0000","last_active":"2023-10-14T16:41:38.261889+0000","last_peered":"2023-10-14T16:41:38.261889+0000","last_clean":"2023-10-14T16:41:38.261889+0000","last_became_active":"2023-10-14T16:41:38.258853+0000","last_became_peered":"2023-10-14T16:41:38.258853+0000","last_unstale":"2023-10-14T16:41:38.261889+0000","last_undegraded":"2023-10-14T16:41:38.261889+0000","last_fullsized":"2023-10-14T16:41:38.261889+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:57.526751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244949+0000","last_change":"2023-10-14T16:41:38.244949+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244949+0000","last_undegraded":"2023-10-14T16:41:38.244949+0000","last_fullsized":"2023-10-14T16:41:38.244949+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272571+0000","last_change":"2023-10-14T16:41:38.269707+0000","last_active":"2023-10-14T16:41:38.272571+0000","last_peered":"2023-10-14T16:41:38.272571+0000","last_clean":"2023-10-14T16:41:38.272571+0000","last_became_active":"2023-10-14T16:41:38.269101+0000","last_became_peered":"2023-10-14T16:41:38.269101+0000","last_unstale":"2023-10-14T16:41:38.272571+0000","last_undegraded":"2023-10-14T16:41:38.272571+0000","last_fullsized":"2023-10-14T16:41:38.272571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.250366+0000","last_change":"2023-10-14T16:41:38.250366+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.250366+0000","last_undegraded":"2023-10-14T16:41:38.250366+0000","last_fullsized":"2023-10-14T16:41:38.250366+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273696+0000","last_change":"2023-10-14T16:41:38.268083+0000","last_active":"2023-10-14T16:41:38.273696+0000","last_peered":"2023-10-14T16:41:38.273696+0000","last_clean":"2023-10-14T16:41:38.273696+0000","last_became_active":"2023-10-14T16:41:38.267791+0000","last_became_peered":"2023-10-14T16:41:38.267791+0000","last_unstale":"2023-10-14T16:41:38.273696+0000","last_undegraded":"2023-10-14T16:41:38.273696+0000","last_fullsized":"2023-10-14T16:41:38.273696+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.251138+0000","last_change":"2023-10-14T16:41:38.251138+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.251138+0000","last_undegraded":"2023-10-14T16:41:38.251138+0000","last_fullsized":"2023-10-14T16:41:38.251138+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266868+0000","last_change":"2023-10-14T16:41:38.259070+0000","last_active":"2023-10-14T16:41:38.266868+0000","last_peered":"2023-10-14T16:41:38.266868+0000","last_clean":"2023-10-14T16:41:38.266868+0000","last_became_active":"2023-10-14T16:41:38.258658+0000","last_became_peered":"2023-10-14T16:41:38.258658+0000","last_unstale":"2023-10-14T16:41:38.266868+0000","last_undegraded":"2023-10-14T16:41:38.266868+0000","last_fullsized":"2023-10-14T16:41:38.266868+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:37.794773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244826+0000","last_change":"2023-10-14T16:41:38.244826+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244826+0000","last_undegraded":"2023-10-14T16:41:38.244826+0000","last_fullsized":"2023-10-14T16:41:38.244826+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269845+0000","last_change":"2023-10-14T16:41:38.265796+0000","last_active":"2023-10-14T16:41:38.269845+0000","last_peered":"2023-10-14T16:41:38.269845+0000","last_clean":"2023-10-14T16:41:38.269845+0000","last_became_active":"2023-10-14T16:41:38.265491+0000","last_became_peered":"2023-10-14T16:41:38.265491+0000","last_unstale":"2023-10-14T16:41:38.269845+0000","last_undegraded":"2023-10-14T16:41:38.269845+0000","last_fullsized":"2023-10-14T16:41:38.269845+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:25.436192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.259056+0000","last_change":"2023-10-14T16:41:38.259056+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.259056+0000","last_undegraded":"2023-10-14T16:41:38.259056+0000","last_fullsized":"2023-10-14T16:41:38.259056+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274094+0000","last_change":"2023-10-14T16:41:38.271323+0000","last_active":"2023-10-14T16:41:38.274094+0000","last_peered":"2023-10-14T16:41:38.274094+0000","last_clean":"2023-10-14T16:41:38.274094+0000","last_became_active":"2023-10-14T16:41:38.271014+0000","last_became_peered":"2023-10-14T16:41:38.271014+0000","last_unstale":"2023-10-14T16:41:38.274094+0000","last_undegraded":"2023-10-14T16:41:38.274094+0000","last_fullsized":"2023-10-14T16:41:38.274094+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.242243+0000","last_change":"2023-10-14T16:41:38.242243+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.242243+0000","last_undegraded":"2023-10-14T16:41:38.242243+0000","last_fullsized":"2023-10-14T16:41:38.242243+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274622+0000","last_change":"2023-10-14T16:41:38.272215+0000","last_active":"2023-10-14T16:41:38.274622+0000","last_peered":"2023-10-14T16:41:38.274622+0000","last_clean":"2023-10-14T16:41:38.274622+0000","last_became_active":"2023-10-14T16:41:38.271943+0000","last_became_peered":"2023-10-14T16:41:38.271943+0000","last_unstale":"2023-10-14T16:41:38.274622+0000","last_undegraded":"2023-10-14T16:41:38.274622+0000","last_fullsized":"2023-10-14T16:41:38.274622+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.253161+0000","last_change":"2023-10-14T16:41:38.253161+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.253161+0000","last_undegraded":"2023-10-14T16:41:38.253161+0000","last_fullsized":"2023-10-14T16:41:38.253161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270665+0000","last_change":"2023-10-14T16:41:38.267446+0000","last_active":"2023-10-14T16:41:38.270665+0000","last_peered":"2023-10-14T16:41:38.270665+0000","last_clean":"2023-10-14T16:41:38.270665+0000","last_became_active":"2023-10-14T16:41:38.267122+0000","last_became_peered":"2023-10-14T16:41:38.267122+0000","last_unstale":"2023-10-14T16:41:38.270665+0000","last_undegraded":"2023-10-14T16:41:38.270665+0000","last_fullsized":"2023-10-14T16:41:38.270665+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246383+0000","last_change":"2023-10-14T16:41:38.246383+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246383+0000","last_undegraded":"2023-10-14T16:41:38.246383+0000","last_fullsized":"2023-10-14T16:41:38.246383+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268444+0000","last_change":"2023-10-14T16:41:38.266145+0000","last_active":"2023-10-14T16:41:38.268444+0000","last_peered":"2023-10-14T16:41:38.268444+0000","last_clean":"2023-10-14T16:41:38.268444+0000","last_became_active":"2023-10-14T16:41:38.264561+0000","last_became_peered":"2023-10-14T16:41:38.264561+0000","last_unstale":"2023-10-14T16:41:38.268444+0000","last_undegraded":"2023-10-14T16:41:38.268444+0000","last_fullsized":"2023-10-14T16:41:38.268444+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:45:54.740964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256466+0000","last_change":"2023-10-14T16:41:38.256466+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256466+0000","last_undegraded":"2023-10-14T16:41:38.256466+0000","last_fullsized":"2023-10-14T16:41:38.256466+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268496+0000","last_change":"2023-10-14T16:41:38.266046+0000","last_active":"2023-10-14T16:41:38.268496+0000","last_peered":"2023-10-14T16:41:38.268496+0000","last_clean":"2023-10-14T16:41:38.268496+0000","last_became_active":"2023-10-14T16:41:38.263725+0000","last_became_peered":"2023-10-14T16:41:38.263725+0000","last_unstale":"2023-10-14T16:41:38.268496+0000","last_undegraded":"2023-10-14T16:41:38.268496+0000","last_fullsized":"2023-10-14T16:41:38.268496+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:44.086979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252808+0000","last_change":"2023-10-14T16:41:38.252808+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252808+0000","last_undegraded":"2023-10-14T16:41:38.252808+0000","last_fullsized":"2023-10-14T16:41:38.252808+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268294+0000","last_change":"2023-10-14T16:41:38.263662+0000","last_active":"2023-10-14T16:41:38.268294+0000","last_peered":"2023-10-14T16:41:38.268294+0000","last_clean":"2023-10-14T16:41:38.268294+0000","last_became_active":"2023-10-14T16:41:38.262441+0000","last_became_peered":"2023-10-14T16:41:38.262441+0000","last_unstale":"2023-10-14T16:41:38.268294+0000","last_undegraded":"2023-10-14T16:41:38.268294+0000","last_fullsized":"2023-10-14T16:41:38.268294+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:54.579331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.245609+0000","last_change":"2023-10-14T16:41:38.245609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.245609+0000","last_undegraded":"2023-10-14T16:41:38.245609+0000","last_fullsized":"2023-10-14T16:41:38.245609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273039+0000","last_change":"2023-10-14T16:41:38.270622+0000","last_active":"2023-10-14T16:41:38.273039+0000","last_peered":"2023-10-14T16:41:38.273039+0000","last_clean":"2023-10-14T16:41:38.273039+0000","last_became_active":"2023-10-14T16:41:38.269459+0000","last_became_peered":"2023-10-14T16:41:38.269459+0000","last_unstale":"2023-10-14T16:41:38.273039+0000","last_undegraded":"2023-10-14T16:41:38.273039+0000","last_fullsized":"2023-10-14T16:41:38.273039+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247414+0000","last_change":"2023-10-14T16:41:38.247414+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247414+0000","last_undegraded":"2023-10-14T16:41:38.247414+0000","last_fullsized":"2023-10-14T16:41:38.247414+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.274818+0000","last_change":"2023-10-14T16:41:38.270414+0000","last_active":"2023-10-14T16:41:38.274818+0000","last_peered":"2023-10-14T16:41:38.274818+0000","last_clean":"2023-10-14T16:41:38.274818+0000","last_became_active":"2023-10-14T16:41:38.269445+0000","last_became_peered":"2023-10-14T16:41:38.269445+0000","last_unstale":"2023-10-14T16:41:38.274818+0000","last_undegraded":"2023-10-14T16:41:38.274818+0000","last_fullsized":"2023-10-14T16:41:38.274818+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247587+0000","last_change":"2023-10-14T16:41:38.247587+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247587+0000","last_undegraded":"2023-10-14T16:41:38.247587+0000","last_fullsized":"2023-10-14T16:41:38.247587+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.275360+0000","last_change":"2023-10-14T16:41:38.271734+0000","last_active":"2023-10-14T16:41:38.275360+0000","last_peered":"2023-10-14T16:41:38.275360+0000","last_clean":"2023-10-14T16:41:38.275360+0000","last_became_active":"2023-10-14T16:41:38.271386+0000","last_became_peered":"2023-10-14T16:41:38.271386+0000","last_unstale":"2023-10-14T16:41:38.275360+0000","last_undegraded":"2023-10-14T16:41:38.275360+0000","last_fullsized":"2023-10-14T16:41:38.275360+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:46:16.492082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.263609+0000","last_change":"2023-10-14T16:41:38.263609+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.263609+0000","last_undegraded":"2023-10-14T16:41:38.263609+0000","last_fullsized":"2023-10-14T16:41:38.263609+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.270337+0000","last_change":"2023-10-14T16:41:38.266162+0000","last_active":"2023-10-14T16:41:38.270337+0000","last_peered":"2023-10-14T16:41:38.270337+0000","last_clean":"2023-10-14T16:41:38.270337+0000","last_became_active":"2023-10-14T16:41:38.265806+0000","last_became_peered":"2023-10-14T16:41:38.265806+0000","last_unstale":"2023-10-14T16:41:38.270337+0000","last_undegraded":"2023-10-14T16:41:38.270337+0000","last_fullsized":"2023-10-14T16:41:38.270337+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.252719+0000","last_change":"2023-10-14T16:41:38.252719+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.252719+0000","last_undegraded":"2023-10-14T16:41:38.252719+0000","last_fullsized":"2023-10-14T16:41:38.252719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.273662+0000","last_change":"2023-10-14T16:41:38.271489+0000","last_active":"2023-10-14T16:41:38.273662+0000","last_peered":"2023-10-14T16:41:38.273662+0000","last_clean":"2023-10-14T16:41:38.273662+0000","last_became_active":"2023-10-14T16:41:38.271096+0000","last_became_peered":"2023-10-14T16:41:38.271096+0000","last_unstale":"2023-10-14T16:41:38.273662+0000","last_undegraded":"2023-10-14T16:41:38.273662+0000","last_fullsized":"2023-10-14T16:41:38.273662+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.244568+0000","last_change":"2023-10-14T16:41:38.244568+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.244568+0000","last_undegraded":"2023-10-14T16:41:38.244568+0000","last_fullsized":"2023-10-14T16:41:38.244568+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272453+0000","last_change":"2023-10-14T16:41:38.267492+0000","last_active":"2023-10-14T16:41:38.272453+0000","last_peered":"2023-10-14T16:41:38.272453+0000","last_clean":"2023-10-14T16:41:38.272453+0000","last_became_active":"2023-10-14T16:41:38.267208+0000","last_became_peered":"2023-10-14T16:41:38.267208+0000","last_unstale":"2023-10-14T16:41:38.272453+0000","last_undegraded":"2023-10-14T16:41:38.272453+0000","last_fullsized":"2023-10-14T16:41:38.272453+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:07.318539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.243653+0000","last_change":"2023-10-14T16:41:38.243653+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.243653+0000","last_undegraded":"2023-10-14T16:41:38.243653+0000","last_fullsized":"2023-10-14T16:41:38.243653+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.272435+0000","last_change":"2023-10-14T16:41:38.269649+0000","last_active":"2023-10-14T16:41:38.272435+0000","last_peered":"2023-10-14T16:41:38.272435+0000","last_clean":"2023-10-14T16:41:38.272435+0000","last_became_active":"2023-10-14T16:41:38.269092+0000","last_became_peered":"2023-10-14T16:41:38.269092+0000","last_unstale":"2023-10-14T16:41:38.272435+0000","last_undegraded":"2023-10-14T16:41:38.272435+0000","last_fullsized":"2023-10-14T16:41:38.272435+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.241898+0000","last_change":"2023-10-14T16:41:38.241898+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.241898+0000","last_undegraded":"2023-10-14T16:41:38.241898+0000","last_fullsized":"2023-10-14T16:41:38.241898+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.262512+0000","last_change":"2023-10-14T16:41:38.260866+0000","last_active":"2023-10-14T16:41:38.262512+0000","last_peered":"2023-10-14T16:41:38.262512+0000","last_clean":"2023-10-14T16:41:38.262512+0000","last_became_active":"2023-10-14T16:41:38.260498+0000","last_became_peered":"2023-10-14T16:41:38.260498+0000","last_unstale":"2023-10-14T16:41:38.262512+0000","last_undegraded":"2023-10-14T16:41:38.262512+0000","last_fullsized":"2023-10-14T16:41:38.262512+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246257+0000","last_change":"2023-10-14T16:41:38.246257+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246257+0000","last_undegraded":"2023-10-14T16:41:38.246257+0000","last_fullsized":"2023-10-14T16:41:38.246257+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.269808+0000","last_change":"2023-10-14T16:41:38.265778+0000","last_active":"2023-10-14T16:41:38.269808+0000","last_peered":"2023-10-14T16:41:38.269808+0000","last_clean":"2023-10-14T16:41:38.269808+0000","last_became_active":"2023-10-14T16:41:38.265477+0000","last_became_peered":"2023-10-14T16:41:38.265477+0000","last_unstale":"2023-10-14T16:41:38.269808+0000","last_undegraded":"2023-10-14T16:41:38.269808+0000","last_fullsized":"2023-10-14T16:41:38.269808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:50.123566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.246290+0000","last_change":"2023-10-14T16:41:38.246290+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.246290+0000","last_undegraded":"2023-10-14T16:41:38.246290+0000","last_fullsized":"2023-10-14T16:41:38.246290+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268405+0000","last_change":"2023-10-14T16:41:38.265852+0000","last_active":"2023-10-14T16:41:38.268405+0000","last_peered":"2023-10-14T16:41:38.268405+0000","last_clean":"2023-10-14T16:41:38.268405+0000","last_became_active":"2023-10-14T16:41:38.265513+0000","last_became_peered":"2023-10-14T16:41:38.265513+0000","last_unstale":"2023-10-14T16:41:38.268405+0000","last_undegraded":"2023-10-14T16:41:38.268405+0000","last_fullsized":"2023-10-14T16:41:38.268405+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:44:33.854697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.247745+0000","last_change":"2023-10-14T16:41:38.247745+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.247745+0000","last_undegraded":"2023-10-14T16:41:38.247745+0000","last_fullsized":"2023-10-14T16:41:38.247745+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.266793+0000","last_change":"2023-10-14T16:41:38.265704+0000","last_active":"2023-10-14T16:41:38.266793+0000","last_peered":"2023-10-14T16:41:38.266793+0000","last_clean":"2023-10-14T16:41:38.266793+0000","last_became_active":"2023-10-14T16:41:38.264927+0000","last_became_peered":"2023-10-14T16:41:38.264927+0000","last_unstale":"2023-10-14T16:41:38.266793+0000","last_undegraded":"2023-10-14T16:41:38.266793+0000","last_fullsized":"2023-10-14T16:41:38.266793+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":1,"reported_epoch":32,"state":"creating+peering","last_fresh":"2023-10-14T16:41:38.256689+0000","last_change":"2023-10-14T16:41:38.256689+0000","last_active":"2023-10-14T16:41:38.219749+0000","last_peered":"2023-10-14T16:41:38.219749+0000","last_clean":"2023-10-14T16:41:38.219749+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2023-10-14T16:41:38.256689+0000","last_undegraded":"2023-10-14T16:41:38.256689+0000","last_fullsized":"2023-10-14T16:41:38.256689+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":9,"reported_epoch":32,"state":"active+clean","last_fresh":"2023-10-14T16:41:38.268528+0000","last_change":"2023-10-14T16:41:38.267370+0000","last_active":"2023-10-14T16:41:38.268528+0000","last_peered":"2023-10-14T16:41:38.268528+0000","last_clean":"2023-10-14T16:41:38.268528+0000","last_became_active":"2023-10-14T16:41:38.267087+0000","last_became_peered":"2023-10-14T16:41:38.267087+0000","last_unstale":"2023-10-14T16:41:38.268528+0000","last_undegraded":"2023-10-14T16:41:38.268528+0000","last_fullsized":"2023-10-14T16:41:38.268528+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":31,"state":"active+clean","last_fresh":"2023-10-14T16:41:37.236449+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:41:37.236449+0000","last_peered":"2023-10-14T16:41:37.236449+0000","last_clean":"2023-10-14T16:41:37.236449+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:41:37.236449+0000","last_undegraded":"2023-10-14T16:41:37.236449+0000","last_fullsized":"2023-10-14T16:41:37.236449+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343900,"statfs":{"total":96636764160,"available":96608153600,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":1.034},{"interface":"front","average":{"1min":1.173,"5min":1.173,"15min":1.173},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":1.272}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.0680000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.125}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.3069999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.179},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.6020000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":1.575},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.427}]},{"osd":6,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":1.77}]}]},{"osd":6,"up_from":13,"seq":55834574867,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":1.1890000000000001},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":1.627},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.0720000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":1.429},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.51}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":1.29},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":1.5580000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.6659999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":3.0110000000000001,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":1.754},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":3.206,"5min":3.206,"15min":3.206},"last":1.476}]},{"osd":7,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":1.8120000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":1.843}]}]},{"osd":5,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27376,"kb_used_data":352,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344464,"statfs":{"total":96636764160,"available":96608731136,"internally_reserved":0,"allocated":360448,"data_stored":139168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":0.98199999999999998}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":1.716}]},{"osd":2,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.3639999999999999,"5min":4.3639999999999999,"15min":4.3639999999999999},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":1.1120000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.823},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":1.4590000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.212},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.893}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.5009999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.9470000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"last":1.6799999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27360,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344480,"statfs":{"total":96636764160,"available":96608747520,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":1.093},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":4.4059999999999997},"last":1.149}]},{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":1.2589999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.4510000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.876},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":0.98399999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.0349999999999999},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.194}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.48},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.3200000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.29},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.3919999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.3680000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574868,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":0.59099999999999997},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":0.81200000000000006}]},{"osd":1,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.69099999999999995},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.65600000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.024},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":0.95399999999999996}]},{"osd":4,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":1.101},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":1.0549999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.1519999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.2490000000000001,"5min":3.2490000000000001,"15min":3.2490000000000001},"last":1.179}]},{"osd":6,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.121}]},{"osd":7,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.236}]}]},{"osd":2,"up_from":13,"seq":55834574868,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":0.45900000000000002}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.5},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.2410000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.3080000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.583},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.79400000000000004}]},{"osd":5,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.006},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":1.518}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.1200000000000001,"5min":3.1200000000000001,"15min":3.1200000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.1060000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":0.84899999999999998},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.054}]}]},{"osd":1,"up_from":13,"seq":55834574868,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31520,"kb_used_data":340,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340320,"statfs":{"total":96636764160,"available":96604487680,"internally_reserved":0,"allocated":348160,"data_stored":132901,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.79600000000000004}]},{"osd":2,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.86599999999999999},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":1.3759999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":1.603},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":0.98299999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":1.2749999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.429},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":1.526}]},{"osd":6,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":4.6360000000000001,"5min":4.6360000000000001,"15min":4.6360000000000001},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":1.4970000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.667},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":1.4650000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574867,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32100,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339740,"statfs":{"total":96636764160,"available":96603893760,"internally_reserved":0,"allocated":942080,"data_stored":723269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.46999999999999997}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.49199999999999999},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.51700000000000002}]},{"osd":4,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":0.63600000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.83299999999999996}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":0.76700000000000002},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":1.05}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":0.69599999999999995},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":0.84999999999999998}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.042},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":0.88200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:41:44.758 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2023-10-14T16:41:44.758 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.251935+0000', 'last_change': '2023-10-14T16:41:38.251935+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.251935+0000', 'last_undegraded': '2023-10-14T16:41:38.251935+0000', 'last_fullsized': '2023-10-14T16:41:38.251935+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T16:41:44.759 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2023-10-14T16:41:44.759 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.251427+0000', 'last_change': '2023-10-14T16:41:38.251427+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.251427+0000', 'last_undegraded': '2023-10-14T16:41:38.251427+0000', 'last_fullsized': '2023-10-14T16:41:38.251427+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.760 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2023-10-14T16:41:44.760 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.252219+0000', 'last_change': '2023-10-14T16:41:38.252219+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.252219+0000', 'last_undegraded': '2023-10-14T16:41:38.252219+0000', 'last_fullsized': '2023-10-14T16:41:38.252219+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.760 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2023-10-14T16:41:44.761 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.258644+0000', 'last_change': '2023-10-14T16:41:38.258644+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.258644+0000', 'last_undegraded': '2023-10-14T16:41:38.258644+0000', 'last_fullsized': '2023-10-14T16:41:38.258644+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.761 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2023-10-14T16:41:44.762 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.248994+0000', 'last_change': '2023-10-14T16:41:38.248994+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.248994+0000', 'last_undegraded': '2023-10-14T16:41:38.248994+0000', 'last_fullsized': '2023-10-14T16:41:38.248994+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 3], 'acting': [2, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.762 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2023-10-14T16:41:44.762 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247097+0000', 'last_change': '2023-10-14T16:41:38.247097+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247097+0000', 'last_undegraded': '2023-10-14T16:41:38.247097+0000', 'last_fullsized': '2023-10-14T16:41:38.247097+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 5], 'acting': [3, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.763 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2023-10-14T16:41:44.763 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.263052+0000', 'last_change': '2023-10-14T16:41:38.263052+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.263052+0000', 'last_undegraded': '2023-10-14T16:41:38.263052+0000', 'last_fullsized': '2023-10-14T16:41:38.263052+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.764 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2023-10-14T16:41:44.764 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.260191+0000', 'last_change': '2023-10-14T16:41:38.260191+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.260191+0000', 'last_undegraded': '2023-10-14T16:41:38.260191+0000', 'last_fullsized': '2023-10-14T16:41:38.260191+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.764 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2023-10-14T16:41:44.765 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.249866+0000', 'last_change': '2023-10-14T16:41:38.249866+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.249866+0000', 'last_undegraded': '2023-10-14T16:41:38.249866+0000', 'last_fullsized': '2023-10-14T16:41:38.249866+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.765 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2023-10-14T16:41:44.766 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.253885+0000', 'last_change': '2023-10-14T16:41:38.253885+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.253885+0000', 'last_undegraded': '2023-10-14T16:41:38.253885+0000', 'last_fullsized': '2023-10-14T16:41:38.253885+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.766 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2023-10-14T16:41:44.766 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.261410+0000', 'last_change': '2023-10-14T16:41:38.261410+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.261410+0000', 'last_undegraded': '2023-10-14T16:41:38.261410+0000', 'last_fullsized': '2023-10-14T16:41:38.261410+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 1], 'acting': [7, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.767 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2023-10-14T16:41:44.767 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.250453+0000', 'last_change': '2023-10-14T16:41:38.250453+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.250453+0000', 'last_undegraded': '2023-10-14T16:41:38.250453+0000', 'last_fullsized': '2023-10-14T16:41:38.250453+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.768 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2023-10-14T16:41:44.768 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.248204+0000', 'last_change': '2023-10-14T16:41:38.248204+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.248204+0000', 'last_undegraded': '2023-10-14T16:41:38.248204+0000', 'last_fullsized': '2023-10-14T16:41:38.248204+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.768 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2023-10-14T16:41:44.769 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246426+0000', 'last_change': '2023-10-14T16:41:38.246426+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246426+0000', 'last_undegraded': '2023-10-14T16:41:38.246426+0000', 'last_fullsized': '2023-10-14T16:41:38.246426+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.769 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2023-10-14T16:41:44.770 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.248085+0000', 'last_change': '2023-10-14T16:41:38.248085+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.248085+0000', 'last_undegraded': '2023-10-14T16:41:38.248085+0000', 'last_fullsized': '2023-10-14T16:41:38.248085+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 6], 'acting': [5, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.770 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2023-10-14T16:41:44.770 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.257273+0000', 'last_change': '2023-10-14T16:41:38.257273+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.257273+0000', 'last_undegraded': '2023-10-14T16:41:38.257273+0000', 'last_fullsized': '2023-10-14T16:41:38.257273+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 3], 'acting': [7, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.771 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2023-10-14T16:41:44.771 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.262216+0000', 'last_change': '2023-10-14T16:41:38.262216+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.262216+0000', 'last_undegraded': '2023-10-14T16:41:38.262216+0000', 'last_fullsized': '2023-10-14T16:41:38.262216+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.772 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2023-10-14T16:41:44.772 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.250965+0000', 'last_change': '2023-10-14T16:41:38.250965+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.250965+0000', 'last_undegraded': '2023-10-14T16:41:38.250965+0000', 'last_fullsized': '2023-10-14T16:41:38.250965+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.772 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2023-10-14T16:41:44.773 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.245112+0000', 'last_change': '2023-10-14T16:41:38.245112+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.245112+0000', 'last_undegraded': '2023-10-14T16:41:38.245112+0000', 'last_fullsized': '2023-10-14T16:41:38.245112+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.773 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2023-10-14T16:41:44.774 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.249462+0000', 'last_change': '2023-10-14T16:41:38.249462+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.249462+0000', 'last_undegraded': '2023-10-14T16:41:38.249462+0000', 'last_fullsized': '2023-10-14T16:41:38.249462+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.774 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2023-10-14T16:41:44.775 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.252360+0000', 'last_change': '2023-10-14T16:41:38.252360+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.252360+0000', 'last_undegraded': '2023-10-14T16:41:38.252360+0000', 'last_fullsized': '2023-10-14T16:41:38.252360+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 0], 'acting': [4, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.775 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2023-10-14T16:41:44.775 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.253610+0000', 'last_change': '2023-10-14T16:41:38.253610+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.253610+0000', 'last_undegraded': '2023-10-14T16:41:38.253610+0000', 'last_fullsized': '2023-10-14T16:41:38.253610+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.776 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2023-10-14T16:41:44.776 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.243316+0000', 'last_change': '2023-10-14T16:41:38.243316+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.243316+0000', 'last_undegraded': '2023-10-14T16:41:38.243316+0000', 'last_fullsized': '2023-10-14T16:41:38.243316+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T16:41:44.777 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2023-10-14T16:41:44.777 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246964+0000', 'last_change': '2023-10-14T16:41:38.246964+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246964+0000', 'last_undegraded': '2023-10-14T16:41:38.246964+0000', 'last_fullsized': '2023-10-14T16:41:38.246964+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 2], 'acting': [5, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.777 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2023-10-14T16:41:44.778 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.256479+0000', 'last_change': '2023-10-14T16:41:38.256479+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.256479+0000', 'last_undegraded': '2023-10-14T16:41:38.256479+0000', 'last_fullsized': '2023-10-14T16:41:38.256479+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 7], 'acting': [1, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.778 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2023-10-14T16:41:44.779 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.245009+0000', 'last_change': '2023-10-14T16:41:38.245009+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.245009+0000', 'last_undegraded': '2023-10-14T16:41:38.245009+0000', 'last_fullsized': '2023-10-14T16:41:38.245009+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.779 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2023-10-14T16:41:44.779 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.258281+0000', 'last_change': '2023-10-14T16:41:38.258281+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.258281+0000', 'last_undegraded': '2023-10-14T16:41:38.258281+0000', 'last_fullsized': '2023-10-14T16:41:38.258281+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.780 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2023-10-14T16:41:44.780 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.249479+0000', 'last_change': '2023-10-14T16:41:38.249479+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.249479+0000', 'last_undegraded': '2023-10-14T16:41:38.249479+0000', 'last_fullsized': '2023-10-14T16:41:38.249479+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.780 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2023-10-14T16:41:44.781 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247132+0000', 'last_change': '2023-10-14T16:41:38.247132+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247132+0000', 'last_undegraded': '2023-10-14T16:41:38.247132+0000', 'last_fullsized': '2023-10-14T16:41:38.247132+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.781 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2023-10-14T16:41:44.782 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.260202+0000', 'last_change': '2023-10-14T16:41:38.260202+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.260202+0000', 'last_undegraded': '2023-10-14T16:41:38.260202+0000', 'last_fullsized': '2023-10-14T16:41:38.260202+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.782 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2023-10-14T16:41:44.783 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246717+0000', 'last_change': '2023-10-14T16:41:38.246717+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246717+0000', 'last_undegraded': '2023-10-14T16:41:38.246717+0000', 'last_fullsized': '2023-10-14T16:41:38.246717+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 2], 'acting': [6, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T16:41:44.783 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2023-10-14T16:41:44.783 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.259641+0000', 'last_change': '2023-10-14T16:41:38.259641+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.259641+0000', 'last_undegraded': '2023-10-14T16:41:38.259641+0000', 'last_fullsized': '2023-10-14T16:41:38.259641+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.784 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2023-10-14T16:41:44.784 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.244200+0000', 'last_change': '2023-10-14T16:41:38.244200+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.244200+0000', 'last_undegraded': '2023-10-14T16:41:38.244200+0000', 'last_fullsized': '2023-10-14T16:41:38.244200+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T16:41:44.785 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2023-10-14T16:41:44.785 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.242058+0000', 'last_change': '2023-10-14T16:41:38.242058+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.242058+0000', 'last_undegraded': '2023-10-14T16:41:38.242058+0000', 'last_fullsized': '2023-10-14T16:41:38.242058+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T16:41:44.785 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2023-10-14T16:41:44.786 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.248641+0000', 'last_change': '2023-10-14T16:41:38.248641+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.248641+0000', 'last_undegraded': '2023-10-14T16:41:38.248641+0000', 'last_fullsized': '2023-10-14T16:41:38.248641+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.786 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2023-10-14T16:41:44.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.251155+0000', 'last_change': '2023-10-14T16:41:38.251155+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.251155+0000', 'last_undegraded': '2023-10-14T16:41:38.251155+0000', 'last_fullsized': '2023-10-14T16:41:38.251155+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.787 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2023-10-14T16:41:44.787 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.259712+0000', 'last_change': '2023-10-14T16:41:38.259712+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.259712+0000', 'last_undegraded': '2023-10-14T16:41:38.259712+0000', 'last_fullsized': '2023-10-14T16:41:38.259712+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.788 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2023-10-14T16:41:44.788 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247837+0000', 'last_change': '2023-10-14T16:41:38.247837+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247837+0000', 'last_undegraded': '2023-10-14T16:41:38.247837+0000', 'last_fullsized': '2023-10-14T16:41:38.247837+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 0], 'acting': [5, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2023-10-14T16:41:44.789 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.244395+0000', 'last_change': '2023-10-14T16:41:38.244395+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.244395+0000', 'last_undegraded': '2023-10-14T16:41:38.244395+0000', 'last_fullsized': '2023-10-14T16:41:38.244395+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.789 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2023-10-14T16:41:44.790 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.257970+0000', 'last_change': '2023-10-14T16:41:38.257970+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.257970+0000', 'last_undegraded': '2023-10-14T16:41:38.257970+0000', 'last_fullsized': '2023-10-14T16:41:38.257970+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.790 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2023-10-14T16:41:44.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246503+0000', 'last_change': '2023-10-14T16:41:38.246503+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246503+0000', 'last_undegraded': '2023-10-14T16:41:38.246503+0000', 'last_fullsized': '2023-10-14T16:41:38.246503+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 3], 'acting': [5, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.791 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2023-10-14T16:41:44.791 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.248995+0000', 'last_change': '2023-10-14T16:41:38.248995+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.248995+0000', 'last_undegraded': '2023-10-14T16:41:38.248995+0000', 'last_fullsized': '2023-10-14T16:41:38.248995+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.792 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2023-10-14T16:41:44.792 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.244949+0000', 'last_change': '2023-10-14T16:41:38.244949+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.244949+0000', 'last_undegraded': '2023-10-14T16:41:38.244949+0000', 'last_fullsized': '2023-10-14T16:41:38.244949+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2023-10-14T16:41:44.793 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.250366+0000', 'last_change': '2023-10-14T16:41:38.250366+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.250366+0000', 'last_undegraded': '2023-10-14T16:41:38.250366+0000', 'last_fullsized': '2023-10-14T16:41:38.250366+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.793 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2023-10-14T16:41:44.794 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.251138+0000', 'last_change': '2023-10-14T16:41:38.251138+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.251138+0000', 'last_undegraded': '2023-10-14T16:41:38.251138+0000', 'last_fullsized': '2023-10-14T16:41:38.251138+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.794 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2023-10-14T16:41:44.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.244826+0000', 'last_change': '2023-10-14T16:41:38.244826+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.244826+0000', 'last_undegraded': '2023-10-14T16:41:38.244826+0000', 'last_fullsized': '2023-10-14T16:41:38.244826+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 0], 'acting': [3, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.795 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2023-10-14T16:41:44.795 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.259056+0000', 'last_change': '2023-10-14T16:41:38.259056+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.259056+0000', 'last_undegraded': '2023-10-14T16:41:38.259056+0000', 'last_fullsized': '2023-10-14T16:41:38.259056+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 4], 'acting': [7, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.796 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2023-10-14T16:41:44.796 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.242243+0000', 'last_change': '2023-10-14T16:41:38.242243+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.242243+0000', 'last_undegraded': '2023-10-14T16:41:38.242243+0000', 'last_fullsized': '2023-10-14T16:41:38.242243+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T16:41:44.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2023-10-14T16:41:44.797 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.253161+0000', 'last_change': '2023-10-14T16:41:38.253161+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.253161+0000', 'last_undegraded': '2023-10-14T16:41:38.253161+0000', 'last_fullsized': '2023-10-14T16:41:38.253161+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.797 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2023-10-14T16:41:44.798 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246383+0000', 'last_change': '2023-10-14T16:41:38.246383+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246383+0000', 'last_undegraded': '2023-10-14T16:41:38.246383+0000', 'last_fullsized': '2023-10-14T16:41:38.246383+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 4], 'acting': [5, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.798 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2023-10-14T16:41:44.799 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.256466+0000', 'last_change': '2023-10-14T16:41:38.256466+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.256466+0000', 'last_undegraded': '2023-10-14T16:41:38.256466+0000', 'last_fullsized': '2023-10-14T16:41:38.256466+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 4], 'acting': [1, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2023-10-14T16:41:44.799 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2023-10-14T16:41:44.800 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.252808+0000', 'last_change': '2023-10-14T16:41:38.252808+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.252808+0000', 'last_undegraded': '2023-10-14T16:41:38.252808+0000', 'last_fullsized': '2023-10-14T16:41:38.252808+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 7], 'acting': [2, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.800 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2023-10-14T16:41:44.800 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.245609+0000', 'last_change': '2023-10-14T16:41:38.245609+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.245609+0000', 'last_undegraded': '2023-10-14T16:41:38.245609+0000', 'last_fullsized': '2023-10-14T16:41:38.245609+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 4], 'acting': [2, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.801 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2023-10-14T16:41:44.801 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247414+0000', 'last_change': '2023-10-14T16:41:38.247414+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247414+0000', 'last_undegraded': '2023-10-14T16:41:38.247414+0000', 'last_fullsized': '2023-10-14T16:41:38.247414+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.802 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2023-10-14T16:41:44.802 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247587+0000', 'last_change': '2023-10-14T16:41:38.247587+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247587+0000', 'last_undegraded': '2023-10-14T16:41:38.247587+0000', 'last_fullsized': '2023-10-14T16:41:38.247587+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 0], 'acting': [2, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.802 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2023-10-14T16:41:44.803 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.263609+0000', 'last_change': '2023-10-14T16:41:38.263609+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.263609+0000', 'last_undegraded': '2023-10-14T16:41:38.263609+0000', 'last_fullsized': '2023-10-14T16:41:38.263609+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 6], 'acting': [7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.803 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2023-10-14T16:41:44.804 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.252719+0000', 'last_change': '2023-10-14T16:41:38.252719+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.252719+0000', 'last_undegraded': '2023-10-14T16:41:38.252719+0000', 'last_fullsized': '2023-10-14T16:41:38.252719+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 5], 'acting': [2, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2023-10-14T16:41:44.804 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2023-10-14T16:41:44.804 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.244568+0000', 'last_change': '2023-10-14T16:41:38.244568+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.244568+0000', 'last_undegraded': '2023-10-14T16:41:38.244568+0000', 'last_fullsized': '2023-10-14T16:41:38.244568+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 1], 'acting': [5, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2023-10-14T16:41:44.805 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2023-10-14T16:41:44.805 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.243653+0000', 'last_change': '2023-10-14T16:41:38.243653+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.243653+0000', 'last_undegraded': '2023-10-14T16:41:38.243653+0000', 'last_fullsized': '2023-10-14T16:41:38.243653+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T16:41:44.806 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2023-10-14T16:41:44.806 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.241898+0000', 'last_change': '2023-10-14T16:41:38.241898+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.241898+0000', 'last_undegraded': '2023-10-14T16:41:38.241898+0000', 'last_fullsized': '2023-10-14T16:41:38.241898+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 0], 'acting': [6, 0], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2023-10-14T16:41:44.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2023-10-14T16:41:44.807 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246257+0000', 'last_change': '2023-10-14T16:41:38.246257+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246257+0000', 'last_undegraded': '2023-10-14T16:41:38.246257+0000', 'last_fullsized': '2023-10-14T16:41:38.246257+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 6], 'acting': [0, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2023-10-14T16:41:44.807 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2023-10-14T16:41:44.808 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.246290+0000', 'last_change': '2023-10-14T16:41:38.246290+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.246290+0000', 'last_undegraded': '2023-10-14T16:41:38.246290+0000', 'last_fullsized': '2023-10-14T16:41:38.246290+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2023-10-14T16:41:44.808 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2023-10-14T16:41:44.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.247745+0000', 'last_change': '2023-10-14T16:41:38.247745+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.247745+0000', 'last_undegraded': '2023-10-14T16:41:38.247745+0000', 'last_fullsized': '2023-10-14T16:41:38.247745+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2023-10-14T16:41:44.809 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2023-10-14T16:41:44.809 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 32, 'state': 'creating+peering', 'last_fresh': '2023-10-14T16:41:38.256689+0000', 'last_change': '2023-10-14T16:41:38.256689+0000', 'last_active': '2023-10-14T16:41:38.219749+0000', 'last_peered': '2023-10-14T16:41:38.219749+0000', 'last_clean': '2023-10-14T16:41:38.219749+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2023-10-14T16:41:38.256689+0000', 'last_undegraded': '2023-10-14T16:41:38.256689+0000', 'last_fullsized': '2023-10-14T16:41:38.256689+0000', 'mapping_epoch': 32, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 32, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'last_clean_scrub_stamp': '2023-10-14T16:41:38.219749+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7, 2], 'acting': [7, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2023-10-14T16:41:44.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2023-10-14T16:41:45.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.042+0000 7f15eb59e640 1 Processor -- start 2023-10-14T16:41:45.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.043+0000 7f15eb59e640 1 -- start start 2023-10-14T16:41:45.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f15ec15d300 0x7f15ec06f630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15ec0afbe0 0x7f15ec161570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 0x7f15ec06aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ec160e40 con 0x7f15ec0afbe0 2023-10-14T16:41:45.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15ec15d6e0 con 0x7f15ec15d300 2023-10-14T16:41:45.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.044+0000 7f15eb59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec06c740 con 0x7f15ec161ab0 2023-10-14T16:41:45.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15ec0afbe0 0x7f15ec161570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f15ec0afbe0 0x7f15ec161570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39392/0 (socket says 172.21.15.29:39392) 2023-10-14T16:41:45.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 -- 172.21.15.29:0/2364536592 learned_addr learned my addr 172.21.15.29:0/2364536592 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:45.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15ea59c640 1 --2- 172.21.15.29:0/2364536592 >> v2:172.21.15.186:3300/0 conn(0x7f15ec15d300 0x7f15ec06f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15ead9d640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 0x7f15ec06aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 -- 172.21.15.29:0/2364536592 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 msgr2=0x7f15ec06aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 0x7f15ec06aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 -- 172.21.15.29:0/2364536592 >> v2:172.21.15.186:3300/0 conn(0x7f15ec15d300 msgr2=0x7f15ec06f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 --2- 172.21.15.29:0/2364536592 >> v2:172.21.15.186:3300/0 conn(0x7f15ec15d300 0x7f15ec06f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15e9d9b640 1 -- 172.21.15.29:0/2364536592 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ec06ca20 con 0x7f15ec0afbe0 2023-10-14T16:41:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.045+0000 7f15ead9d640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 0x7f15ec06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.046+0000 7f15e9d9b640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec0afbe0 0x7f15ec161570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b32c0 tx=0x7f15d402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6246c2ab1e13737b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.047+0000 7f15e959a640 1 -- 172.21.15.29:0/2364536592 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d402ca80 con 0x7f15ec0afbe0 2023-10-14T16:41:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.047+0000 7f15e959a640 1 -- 172.21.15.29:0/2364536592 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15d402cbe0 con 0x7f15ec0afbe0 2023-10-14T16:41:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.047+0000 7f15e959a640 1 -- 172.21.15.29:0/2364536592 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d40343f0 con 0x7f15ec0afbe0 2023-10-14T16:41:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.047+0000 7f15eb59e640 1 -- 172.21.15.29:0/2364536592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec0afbe0 msgr2=0x7f15ec161570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.047+0000 7f15eb59e640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec0afbe0 0x7f15ec161570 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f15ec0b32c0 tx=0x7f15d402bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 -- 172.21.15.29:0/2364536592 shutdown_connections 2023-10-14T16:41:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec161ab0 0x7f15ec06aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 --2- 172.21.15.29:0/2364536592 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec0afbe0 0x7f15ec161570 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 --2- 172.21.15.29:0/2364536592 >> v2:172.21.15.186:3300/0 conn(0x7f15ec15d300 0x7f15ec06f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 -- 172.21.15.29:0/2364536592 >> 172.21.15.29:0/2364536592 conn(0x7f15ec0b23d0 msgr2=0x7f15ec0b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:45.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 -- 172.21.15.29:0/2364536592 shutdown_connections 2023-10-14T16:41:45.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.048+0000 7f15eb59e640 1 -- 172.21.15.29:0/2364536592 wait complete. 2023-10-14T16:41:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.050+0000 7f15eb59e640 1 Processor -- start 2023-10-14T16:41:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.050+0000 7f15eb59e640 1 -- start start 2023-10-14T16:41:45.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 0x7f15ec156970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15ec15d300 0x7f15ec160100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec15ece0 con 0x7f15ec160640 2023-10-14T16:41:45.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec1f2ef0 con 0x7f15ec15d300 2023-10-14T16:41:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15eb59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f15ec1f3190 con 0x7f15ec0afbe0 2023-10-14T16:41:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15ead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15ead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39406/0 (socket says 172.21.15.29:39406) 2023-10-14T16:41:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15ead9d640 1 -- 172.21.15.29:0/1850339170 learned_addr learned my addr 172.21.15.29:0/1850339170 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:45.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15ea59c640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 0x7f15ec156970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.051+0000 7f15e9d9b640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15ec15d300 0x7f15ec160100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 -- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 msgr2=0x7f15ec156970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 0x7f15ec156970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 -- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15ec15d300 msgr2=0x7f15ec160100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15ec15d300 0x7f15ec160100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 -- 172.21.15.29:0/1850339170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15d4009b30 con 0x7f15ec160640 2023-10-14T16:41:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ea59c640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 0x7f15ec156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15ead9d640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c590 tx=0x7f15e400ca50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e4004440 con 0x7f15ec160640 2023-10-14T16:41:45.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15e4003610 con 0x7f15ec160640 2023-10-14T16:41:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e40039c0 con 0x7f15ec160640 2023-10-14T16:41:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.052+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15ec1f3470 con 0x7f15ec160640 2023-10-14T16:41:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.053+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15ec1f3a30 con 0x7f15ec160640 2023-10-14T16:41:45.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.054+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15ec1f3c50 con 0x7f15ec160640 2023-10-14T16:41:45.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.055+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f15e4003b20 con 0x7f15ec160640 2023-10-14T16:41:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.056+0000 7f15e2ffd640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 0x7f15cc06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.056+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f15e40897e0 con 0x7f15ec160640 2023-10-14T16:41:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.056+0000 7f15ea59c640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 0x7f15cc06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.057+0000 7f15ea59c640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 0x7f15cc06d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00ba50 tx=0x7f15dc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.060+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f15e4054880 con 0x7f15ec160640 2023-10-14T16:41:45.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.326+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f15ec04e160 con 0x7f15ec160640 2023-10-14T16:41:45.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.327+0000 7f15e2ffd640 1 -- 172.21.15.29:0/1850339170 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1194 (secure 0 0 0) 0x7f15e4057ea0 con 0x7f15ec160640 2023-10-14T16:41:45.328 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:45.328 INFO:teuthology.orchestra.run.smithi029.stdout:{"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":105,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":34,"num_osds":8,"num_up_osds":8,"osd_up_since":1697301613,"num_in_osds":8,"osd_in_since":1697301604,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"creating+peering","count":64},{"state_name":"active+clean","count":33}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":242376704,"bytes_avail":772851736576,"bytes_total":773094113280,"inactive_pgs_ratio":0.65979379415512085},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"a","status":"up:active","gid":5118}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":5,"modified":"2023-10-14T16:41:23.317529+0000","services":{"mds":{"daemons":{"summary":"","c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2023-10-14T16:41:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 msgr2=0x7f15cc06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 0x7f15cc06d330 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f15dc00ba50 tx=0x7f15dc008040 comp rx=0 tx=0).stop 2023-10-14T16:41:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 msgr2=0x7f15ec0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f15e400c590 tx=0x7f15e400ca50 comp rx=0 tx=0).stop 2023-10-14T16:41:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 shutdown_connections 2023-10-14T16:41:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f15cc06ae70 0x7f15cc06d330 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f15ec160640 0x7f15ec0b0b90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f15ec15d300 0x7f15ec160100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 --2- 172.21.15.29:0/1850339170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f15ec0afbe0 0x7f15ec156970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 >> 172.21.15.29:0/1850339170 conn(0x7f15ec0b23d0 msgr2=0x7f15ec0b1ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 shutdown_connections 2023-10-14T16:41:45.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.330+0000 7f15eb59e640 1 -- 172.21.15.29:0/1850339170 wait complete. 2023-10-14T16:41:45.350 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2023-10-14T16:41:45.350 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:41:45.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.570+0000 7fb9b9067640 1 Processor -- start 2023-10-14T16:41:45.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.570+0000 7fb9b9067640 1 -- start start 2023-10-14T16:41:45.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9b406c040 0x7fb9b406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb9b40ad970 0x7fb9b415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 0x7fb9b415f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b406c960 con 0x7fb9b406c040 2023-10-14T16:41:45.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b415b9d0 con 0x7fb9b40ad970 2023-10-14T16:41:45.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.571+0000 7fb9b9067640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b415bcb0 con 0x7fb9b415ebc0 2023-10-14T16:41:45.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9b406c040 0x7fb9b406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9b406c040 0x7fb9b406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39414/0 (socket says 172.21.15.29:39414) 2023-10-14T16:41:45.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 -- 172.21.15.29:0/3487593951 learned_addr learned my addr 172.21.15.29:0/3487593951 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:45.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b8866640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 0x7fb9b415f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 -- 172.21.15.29:0/3487593951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 msgr2=0x7fb9b415f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:45.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 0x7fb9b415f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 -- 172.21.15.29:0/3487593951 >> v2:172.21.15.186:3300/0 conn(0x7fb9b40ad970 msgr2=0x7fb9b415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:45.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/3487593951 >> v2:172.21.15.186:3300/0 conn(0x7fb9b40ad970 0x7fb9b415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b3fff640 1 -- 172.21.15.29:0/3487593951 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b415bf90 con 0x7fb9b406c040 2023-10-14T16:41:45.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.572+0000 7fb9b8866640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 0x7fb9b415f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.573+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c040 0x7fb9b406c420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40cb060 tx=0x7fb9a402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e3bd0bed91d7b78 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b2ffd640 1 -- 172.21.15.29:0/3487593951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a402ca80 con 0x7fb9b406c040 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b2ffd640 1 -- 172.21.15.29:0/3487593951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a402cbe0 con 0x7fb9b406c040 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b2ffd640 1 -- 172.21.15.29:0/3487593951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a40343f0 con 0x7fb9b406c040 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 -- 172.21.15.29:0/3487593951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c040 msgr2=0x7fb9b406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c040 0x7fb9b406c420 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb9b40cb060 tx=0x7fb9a402bee0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 -- 172.21.15.29:0/3487593951 shutdown_connections 2023-10-14T16:41:45.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b415ebc0 0x7fb9b415f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 --2- 172.21.15.29:0/3487593951 >> v2:172.21.15.186:3300/0 conn(0x7fb9b40ad970 0x7fb9b415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 --2- 172.21.15.29:0/3487593951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b406c040 0x7fb9b406c420 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 -- 172.21.15.29:0/3487593951 >> 172.21.15.29:0/3487593951 conn(0x7fb9b40ce6a0 msgr2=0x7fb9b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:45.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 -- 172.21.15.29:0/3487593951 shutdown_connections 2023-10-14T16:41:45.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.574+0000 7fb9b9067640 1 -- 172.21.15.29:0/3487593951 wait complete. 2023-10-14T16:41:45.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.576+0000 7fb9b9067640 1 Processor -- start 2023-10-14T16:41:45.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.576+0000 7fb9b9067640 1 -- start start 2023-10-14T16:41:45.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b406c040 0x7fb9b4153e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 0x7fb9b415ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b4156640 con 0x7fb9b40ad970 2023-10-14T16:41:45.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b41559c0 con 0x7fb9b415ebc0 2023-10-14T16:41:45.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.577+0000 7fb9b9067640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b4155ca0 con 0x7fb9b406c040 2023-10-14T16:41:45.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39428/0 (socket says 172.21.15.29:39428) 2023-10-14T16:41:45.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 -- 172.21.15.29:0/1640651661 learned_addr learned my addr 172.21.15.29:0/1640651661 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:45.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b406c040 0x7fb9b4153e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b8866640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 0x7fb9b415ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 -- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b406c040 msgr2=0x7fb9b4153e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b406c040 0x7fb9b4153e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 -- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 msgr2=0x7fb9b415ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 0x7fb9b415ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 -- 172.21.15.29:0/1640651661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4009b30 con 0x7fb9b40ad970 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b8866640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 0x7fb9b415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:45.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.578+0000 7fb9b37fe640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b410 tx=0x7fb9a800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.579+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8011830 con 0x7fb9b40ad970 2023-10-14T16:41:45.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.579+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8011990 con 0x7fb9b40ad970 2023-10-14T16:41:45.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.579+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8011d40 con 0x7fb9b40ad970 2023-10-14T16:41:45.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.579+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b4155f80 con 0x7fb9b40ad970 2023-10-14T16:41:45.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.579+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b40c78d0 con 0x7fb9b40ad970 2023-10-14T16:41:45.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.580+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b40ae6c0 con 0x7fb9b40ad970 2023-10-14T16:41:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.581+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fb9a801a400 con 0x7fb9b40ad970 2023-10-14T16:41:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.582+0000 7fb9b0ff9640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 0x7fb9a006d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.582+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fb9a80898c0 con 0x7fb9b40ad970 2023-10-14T16:41:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.582+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 0x7fb9a006d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:45.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.583+0000 7fb9b3fff640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 0x7fb9a006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4035000 tx=0x7fb9a4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:45.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.586+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9a8054840 con 0x7fb9b40ad970 2023-10-14T16:41:45.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.783+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb9b40cc550 con 0x7fb9a006ae70 2023-10-14T16:41:45.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.791+0000 7fb9b0ff9640 1 -- 172.21.15.29:0/1640651661 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287024 (secure 0 0 0) 0x7fb9a40343f0 con 0x7fb9a006ae70 2023-10-14T16:41:45.796 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:45.797 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:41:45.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 msgr2=0x7fb9a006d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 0x7fb9a006d330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb9a4035000 tx=0x7fb9a4030000 comp rx=0 tx=0).stop 2023-10-14T16:41:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 msgr2=0x7fb9b4154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb9a800b410 tx=0x7fb9a800b8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 shutdown_connections 2023-10-14T16:41:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fb9a006ae70 0x7fb9a006d330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9b415ebc0 0x7fb9b415ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b40ad970 0x7fb9b4154fb0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 --2- 172.21.15.29:0/1640651661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b406c040 0x7fb9b4153e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:45.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.797+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 >> 172.21.15.29:0/1640651661 conn(0x7fb9b40ce6a0 msgr2=0x7fb9b40b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:45.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.798+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 shutdown_connections 2023-10-14T16:41:45.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:45.798+0000 7fb9b9067640 1 -- 172.21.15.29:0/1640651661 wait complete. 2023-10-14T16:41:45.817 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":73,"stamp":"2023-10-14T16:41:45.322861+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237084,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737636,"statfs":{"total":773094113280,"available":772851339264,"internally_reserved":0,"allocated":4333568,"data_stored":2403500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.086816"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275868+0000","last_change":"2023-10-14T16:41:39.281568+0000","last_active":"2023-10-14T16:41:40.275868+0000","last_peered":"2023-10-14T16:41:40.275868+0000","last_clean":"2023-10-14T16:41:40.275868+0000","last_became_active":"2023-10-14T16:41:39.280093+0000","last_became_peered":"2023-10-14T16:41:39.280093+0000","last_unstale":"2023-10-14T16:41:40.275868+0000","last_undegraded":"2023-10-14T16:41:40.275868+0000","last_fullsized":"2023-10-14T16:41:40.275868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273463+0000","last_change":"2023-10-14T16:41:39.275677+0000","last_active":"2023-10-14T16:41:40.273463+0000","last_peered":"2023-10-14T16:41:40.273463+0000","last_clean":"2023-10-14T16:41:40.273463+0000","last_became_active":"2023-10-14T16:41:39.275406+0000","last_became_peered":"2023-10-14T16:41:39.275406+0000","last_unstale":"2023-10-14T16:41:40.273463+0000","last_undegraded":"2023-10-14T16:41:40.273463+0000","last_fullsized":"2023-10-14T16:41:40.273463+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750971+0000","last_change":"2023-10-14T16:41:39.282923+0000","last_active":"2023-10-14T16:41:40.750971+0000","last_peered":"2023-10-14T16:41:40.750971+0000","last_clean":"2023-10-14T16:41:40.750971+0000","last_became_active":"2023-10-14T16:41:39.282613+0000","last_became_peered":"2023-10-14T16:41:39.282613+0000","last_unstale":"2023-10-14T16:41:40.750971+0000","last_undegraded":"2023-10-14T16:41:40.750971+0000","last_fullsized":"2023-10-14T16:41:40.750971+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749154+0000","last_change":"2023-10-14T16:41:39.275832+0000","last_active":"2023-10-14T16:41:40.749154+0000","last_peered":"2023-10-14T16:41:40.749154+0000","last_clean":"2023-10-14T16:41:40.749154+0000","last_became_active":"2023-10-14T16:41:39.275554+0000","last_became_peered":"2023-10-14T16:41:39.275554+0000","last_unstale":"2023-10-14T16:41:40.749154+0000","last_undegraded":"2023-10-14T16:41:40.749154+0000","last_fullsized":"2023-10-14T16:41:40.749154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:39:57.469905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750712+0000","last_change":"2023-10-14T16:41:39.276882+0000","last_active":"2023-10-14T16:41:40.750712+0000","last_peered":"2023-10-14T16:41:40.750712+0000","last_clean":"2023-10-14T16:41:40.750712+0000","last_became_active":"2023-10-14T16:41:39.274595+0000","last_became_peered":"2023-10-14T16:41:39.274595+0000","last_unstale":"2023-10-14T16:41:40.750712+0000","last_undegraded":"2023-10-14T16:41:40.750712+0000","last_fullsized":"2023-10-14T16:41:40.750712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:08.333380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.272460+0000","last_change":"2023-10-14T16:41:39.289475+0000","last_active":"2023-10-14T16:41:40.272460+0000","last_peered":"2023-10-14T16:41:40.272460+0000","last_clean":"2023-10-14T16:41:40.272460+0000","last_became_active":"2023-10-14T16:41:39.287681+0000","last_became_peered":"2023-10-14T16:41:39.287681+0000","last_unstale":"2023-10-14T16:41:40.272460+0000","last_undegraded":"2023-10-14T16:41:40.272460+0000","last_fullsized":"2023-10-14T16:41:40.272460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276308+0000","last_change":"2023-10-14T16:41:39.276259+0000","last_active":"2023-10-14T16:41:40.276308+0000","last_peered":"2023-10-14T16:41:40.276308+0000","last_clean":"2023-10-14T16:41:40.276308+0000","last_became_active":"2023-10-14T16:41:39.275950+0000","last_became_peered":"2023-10-14T16:41:39.275950+0000","last_unstale":"2023-10-14T16:41:40.276308+0000","last_undegraded":"2023-10-14T16:41:40.276308+0000","last_fullsized":"2023-10-14T16:41:40.276308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275124+0000","last_change":"2023-10-14T16:41:39.282445+0000","last_active":"2023-10-14T16:41:40.275124+0000","last_peered":"2023-10-14T16:41:40.275124+0000","last_clean":"2023-10-14T16:41:40.275124+0000","last_became_active":"2023-10-14T16:41:39.280747+0000","last_became_peered":"2023-10-14T16:41:39.280747+0000","last_unstale":"2023-10-14T16:41:40.275124+0000","last_undegraded":"2023-10-14T16:41:40.275124+0000","last_fullsized":"2023-10-14T16:41:40.275124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:26.358627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.752695+0000","last_change":"2023-10-14T16:41:39.281160+0000","last_active":"2023-10-14T16:41:40.752695+0000","last_peered":"2023-10-14T16:41:40.752695+0000","last_clean":"2023-10-14T16:41:40.752695+0000","last_became_active":"2023-10-14T16:41:39.280860+0000","last_became_peered":"2023-10-14T16:41:39.280860+0000","last_unstale":"2023-10-14T16:41:40.752695+0000","last_undegraded":"2023-10-14T16:41:40.752695+0000","last_fullsized":"2023-10-14T16:41:40.752695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:33.006319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749273+0000","last_change":"2023-10-14T16:41:39.290545+0000","last_active":"2023-10-14T16:41:40.749273+0000","last_peered":"2023-10-14T16:41:40.749273+0000","last_clean":"2023-10-14T16:41:40.749273+0000","last_became_active":"2023-10-14T16:41:39.290234+0000","last_became_peered":"2023-10-14T16:41:39.290234+0000","last_unstale":"2023-10-14T16:41:40.749273+0000","last_undegraded":"2023-10-14T16:41:40.749273+0000","last_fullsized":"2023-10-14T16:41:40.749273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:50.380762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276931+0000","last_change":"2023-10-14T16:41:39.283061+0000","last_active":"2023-10-14T16:41:40.276931+0000","last_peered":"2023-10-14T16:41:40.276931+0000","last_clean":"2023-10-14T16:41:40.276931+0000","last_became_active":"2023-10-14T16:41:39.282811+0000","last_became_peered":"2023-10-14T16:41:39.282811+0000","last_unstale":"2023-10-14T16:41:40.276931+0000","last_undegraded":"2023-10-14T16:41:40.276931+0000","last_fullsized":"2023-10-14T16:41:40.276931+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750368+0000","last_change":"2023-10-14T16:41:39.288795+0000","last_active":"2023-10-14T16:41:40.750368+0000","last_peered":"2023-10-14T16:41:40.750368+0000","last_clean":"2023-10-14T16:41:40.750368+0000","last_became_active":"2023-10-14T16:41:39.287348+0000","last_became_peered":"2023-10-14T16:41:39.287348+0000","last_unstale":"2023-10-14T16:41:40.750368+0000","last_undegraded":"2023-10-14T16:41:40.750368+0000","last_fullsized":"2023-10-14T16:41:40.750368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751066+0000","last_change":"2023-10-14T16:41:39.266227+0000","last_active":"2023-10-14T16:41:40.751066+0000","last_peered":"2023-10-14T16:41:40.751066+0000","last_clean":"2023-10-14T16:41:40.751066+0000","last_became_active":"2023-10-14T16:41:39.265760+0000","last_became_peered":"2023-10-14T16:41:39.265760+0000","last_unstale":"2023-10-14T16:41:40.751066+0000","last_undegraded":"2023-10-14T16:41:40.751066+0000","last_fullsized":"2023-10-14T16:41:40.751066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.391215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.272606+0000","last_change":"2023-10-14T16:41:39.278809+0000","last_active":"2023-10-14T16:41:40.272606+0000","last_peered":"2023-10-14T16:41:40.272606+0000","last_clean":"2023-10-14T16:41:40.272606+0000","last_became_active":"2023-10-14T16:41:39.274047+0000","last_became_peered":"2023-10-14T16:41:39.274047+0000","last_unstale":"2023-10-14T16:41:40.272606+0000","last_undegraded":"2023-10-14T16:41:40.272606+0000","last_fullsized":"2023-10-14T16:41:40.272606+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749818+0000","last_change":"2023-10-14T16:41:39.269986+0000","last_active":"2023-10-14T16:41:40.749818+0000","last_peered":"2023-10-14T16:41:40.749818+0000","last_clean":"2023-10-14T16:41:40.749818+0000","last_became_active":"2023-10-14T16:41:39.269641+0000","last_became_peered":"2023-10-14T16:41:39.269641+0000","last_unstale":"2023-10-14T16:41:40.749818+0000","last_undegraded":"2023-10-14T16:41:40.749818+0000","last_fullsized":"2023-10-14T16:41:40.749818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276014+0000","last_change":"2023-10-14T16:41:39.292170+0000","last_active":"2023-10-14T16:41:40.276014+0000","last_peered":"2023-10-14T16:41:40.276014+0000","last_clean":"2023-10-14T16:41:40.276014+0000","last_became_active":"2023-10-14T16:41:39.291896+0000","last_became_peered":"2023-10-14T16:41:39.291896+0000","last_unstale":"2023-10-14T16:41:40.276014+0000","last_undegraded":"2023-10-14T16:41:40.276014+0000","last_fullsized":"2023-10-14T16:41:40.276014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.277591+0000","last_change":"2023-10-14T16:41:39.287004+0000","last_active":"2023-10-14T16:41:40.277591+0000","last_peered":"2023-10-14T16:41:40.277591+0000","last_clean":"2023-10-14T16:41:40.277591+0000","last_became_active":"2023-10-14T16:41:39.286729+0000","last_became_peered":"2023-10-14T16:41:39.286729+0000","last_unstale":"2023-10-14T16:41:40.277591+0000","last_undegraded":"2023-10-14T16:41:40.277591+0000","last_fullsized":"2023-10-14T16:41:40.277591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273643+0000","last_change":"2023-10-14T16:41:39.285284+0000","last_active":"2023-10-14T16:41:40.273643+0000","last_peered":"2023-10-14T16:41:40.273643+0000","last_clean":"2023-10-14T16:41:40.273643+0000","last_became_active":"2023-10-14T16:41:39.284719+0000","last_became_peered":"2023-10-14T16:41:39.284719+0000","last_unstale":"2023-10-14T16:41:40.273643+0000","last_undegraded":"2023-10-14T16:41:40.273643+0000","last_fullsized":"2023-10-14T16:41:40.273643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:49.310290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273355+0000","last_change":"2023-10-14T16:41:39.287370+0000","last_active":"2023-10-14T16:41:40.273355+0000","last_peered":"2023-10-14T16:41:40.273355+0000","last_clean":"2023-10-14T16:41:40.273355+0000","last_became_active":"2023-10-14T16:41:39.286680+0000","last_became_peered":"2023-10-14T16:41:39.286680+0000","last_unstale":"2023-10-14T16:41:40.273355+0000","last_undegraded":"2023-10-14T16:41:40.273355+0000","last_fullsized":"2023-10-14T16:41:40.273355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751330+0000","last_change":"2023-10-14T16:41:39.275840+0000","last_active":"2023-10-14T16:41:40.751330+0000","last_peered":"2023-10-14T16:41:40.751330+0000","last_clean":"2023-10-14T16:41:40.751330+0000","last_became_active":"2023-10-14T16:41:39.275555+0000","last_became_peered":"2023-10-14T16:41:39.275555+0000","last_unstale":"2023-10-14T16:41:40.751330+0000","last_undegraded":"2023-10-14T16:41:40.751330+0000","last_fullsized":"2023-10-14T16:41:40.751330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:51.138406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751337+0000","last_change":"2023-10-14T16:41:39.273422+0000","last_active":"2023-10-14T16:41:40.751337+0000","last_peered":"2023-10-14T16:41:40.751337+0000","last_clean":"2023-10-14T16:41:40.751337+0000","last_became_active":"2023-10-14T16:41:39.273075+0000","last_became_peered":"2023-10-14T16:41:39.273075+0000","last_unstale":"2023-10-14T16:41:40.751337+0000","last_undegraded":"2023-10-14T16:41:40.751337+0000","last_fullsized":"2023-10-14T16:41:40.751337+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:15.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751409+0000","last_change":"2023-10-14T16:41:39.280965+0000","last_active":"2023-10-14T16:41:40.751409+0000","last_peered":"2023-10-14T16:41:40.751409+0000","last_clean":"2023-10-14T16:41:40.751409+0000","last_became_active":"2023-10-14T16:41:39.276351+0000","last_became_peered":"2023-10-14T16:41:39.276351+0000","last_unstale":"2023-10-14T16:41:40.751409+0000","last_undegraded":"2023-10-14T16:41:40.751409+0000","last_fullsized":"2023-10-14T16:41:40.751409+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:56.968826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407416+0000","last_change":"2023-10-14T16:41:39.275497+0000","last_active":"2023-10-14T16:41:40.407416+0000","last_peered":"2023-10-14T16:41:40.407416+0000","last_clean":"2023-10-14T16:41:40.407416+0000","last_became_active":"2023-10-14T16:41:39.275142+0000","last_became_peered":"2023-10-14T16:41:39.275142+0000","last_unstale":"2023-10-14T16:41:40.407416+0000","last_undegraded":"2023-10-14T16:41:40.407416+0000","last_fullsized":"2023-10-14T16:41:40.407416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:15.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749599+0000","last_change":"2023-10-14T16:41:39.273988+0000","last_active":"2023-10-14T16:41:40.749599+0000","last_peered":"2023-10-14T16:41:40.749599+0000","last_clean":"2023-10-14T16:41:40.749599+0000","last_became_active":"2023-10-14T16:41:39.273710+0000","last_became_peered":"2023-10-14T16:41:39.273710+0000","last_unstale":"2023-10-14T16:41:40.749599+0000","last_undegraded":"2023-10-14T16:41:40.749599+0000","last_fullsized":"2023-10-14T16:41:40.749599+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749720+0000","last_change":"2023-10-14T16:41:39.277140+0000","last_active":"2023-10-14T16:41:40.749720+0000","last_peered":"2023-10-14T16:41:40.749720+0000","last_clean":"2023-10-14T16:41:40.749720+0000","last_became_active":"2023-10-14T16:41:39.276726+0000","last_became_peered":"2023-10-14T16:41:39.276726+0000","last_unstale":"2023-10-14T16:41:40.749720+0000","last_undegraded":"2023-10-14T16:41:40.749720+0000","last_fullsized":"2023-10-14T16:41:40.749720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273623+0000","last_change":"2023-10-14T16:41:39.285065+0000","last_active":"2023-10-14T16:41:40.273623+0000","last_peered":"2023-10-14T16:41:40.273623+0000","last_clean":"2023-10-14T16:41:40.273623+0000","last_became_active":"2023-10-14T16:41:39.284290+0000","last_became_peered":"2023-10-14T16:41:39.284290+0000","last_unstale":"2023-10-14T16:41:40.273623+0000","last_undegraded":"2023-10-14T16:41:40.273623+0000","last_fullsized":"2023-10-14T16:41:40.273623+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:05.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750085+0000","last_change":"2023-10-14T16:41:39.278823+0000","last_active":"2023-10-14T16:41:40.750085+0000","last_peered":"2023-10-14T16:41:40.750085+0000","last_clean":"2023-10-14T16:41:40.750085+0000","last_became_active":"2023-10-14T16:41:39.277088+0000","last_became_peered":"2023-10-14T16:41:39.277088+0000","last_unstale":"2023-10-14T16:41:40.750085+0000","last_undegraded":"2023-10-14T16:41:40.750085+0000","last_fullsized":"2023-10-14T16:41:40.750085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750719+0000","last_change":"2023-10-14T16:41:39.286089+0000","last_active":"2023-10-14T16:41:40.750719+0000","last_peered":"2023-10-14T16:41:40.750719+0000","last_clean":"2023-10-14T16:41:40.750719+0000","last_became_active":"2023-10-14T16:41:39.285814+0000","last_became_peered":"2023-10-14T16:41:39.285814+0000","last_unstale":"2023-10-14T16:41:40.750719+0000","last_undegraded":"2023-10-14T16:41:40.750719+0000","last_fullsized":"2023-10-14T16:41:40.750719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751322+0000","last_change":"2023-10-14T16:41:39.283370+0000","last_active":"2023-10-14T16:41:40.751322+0000","last_peered":"2023-10-14T16:41:40.751322+0000","last_clean":"2023-10-14T16:41:40.751322+0000","last_became_active":"2023-10-14T16:41:39.282975+0000","last_became_peered":"2023-10-14T16:41:39.282975+0000","last_unstale":"2023-10-14T16:41:40.751322+0000","last_undegraded":"2023-10-14T16:41:40.751322+0000","last_fullsized":"2023-10-14T16:41:40.751322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276247+0000","last_change":"2023-10-14T16:41:39.275366+0000","last_active":"2023-10-14T16:41:40.276247+0000","last_peered":"2023-10-14T16:41:40.276247+0000","last_clean":"2023-10-14T16:41:40.276247+0000","last_became_active":"2023-10-14T16:41:39.275045+0000","last_became_peered":"2023-10-14T16:41:39.275045+0000","last_unstale":"2023-10-14T16:41:40.276247+0000","last_undegraded":"2023-10-14T16:41:40.276247+0000","last_fullsized":"2023-10-14T16:41:40.276247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.274820+0000","last_change":"2023-10-14T16:41:39.272725+0000","last_active":"2023-10-14T16:41:40.274820+0000","last_peered":"2023-10-14T16:41:40.274820+0000","last_clean":"2023-10-14T16:41:40.274820+0000","last_became_active":"2023-10-14T16:41:39.272130+0000","last_became_peered":"2023-10-14T16:41:39.272130+0000","last_unstale":"2023-10-14T16:41:40.274820+0000","last_undegraded":"2023-10-14T16:41:40.274820+0000","last_fullsized":"2023-10-14T16:41:40.274820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:44:03.658409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275340+0000","last_change":"2023-10-14T16:41:39.284463+0000","last_active":"2023-10-14T16:41:40.275340+0000","last_peered":"2023-10-14T16:41:40.275340+0000","last_clean":"2023-10-14T16:41:40.275340+0000","last_became_active":"2023-10-14T16:41:39.283467+0000","last_became_peered":"2023-10-14T16:41:39.283467+0000","last_unstale":"2023-10-14T16:41:40.275340+0000","last_undegraded":"2023-10-14T16:41:40.275340+0000","last_fullsized":"2023-10-14T16:41:40.275340+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407077+0000","last_change":"2023-10-14T16:41:39.275061+0000","last_active":"2023-10-14T16:41:40.407077+0000","last_peered":"2023-10-14T16:41:40.407077+0000","last_clean":"2023-10-14T16:41:40.407077+0000","last_became_active":"2023-10-14T16:41:39.274764+0000","last_became_peered":"2023-10-14T16:41:39.274764+0000","last_unstale":"2023-10-14T16:41:40.407077+0000","last_undegraded":"2023-10-14T16:41:40.407077+0000","last_fullsized":"2023-10-14T16:41:40.407077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294710+0000","last_change":"2023-10-14T16:41:38.263891+0000","last_active":"2023-10-14T16:41:39.294710+0000","last_peered":"2023-10-14T16:41:39.294710+0000","last_clean":"2023-10-14T16:41:39.294710+0000","last_became_active":"2023-10-14T16:41:38.263622+0000","last_became_peered":"2023-10-14T16:41:38.263622+0000","last_unstale":"2023-10-14T16:41:39.294710+0000","last_undegraded":"2023-10-14T16:41:39.294710+0000","last_fullsized":"2023-10-14T16:41:39.294710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275866+0000","last_change":"2023-10-14T16:41:39.270667+0000","last_active":"2023-10-14T16:41:40.275866+0000","last_peered":"2023-10-14T16:41:40.275866+0000","last_clean":"2023-10-14T16:41:40.275866+0000","last_became_active":"2023-10-14T16:41:39.270375+0000","last_became_peered":"2023-10-14T16:41:39.270375+0000","last_unstale":"2023-10-14T16:41:40.275866+0000","last_undegraded":"2023-10-14T16:41:40.275866+0000","last_fullsized":"2023-10-14T16:41:40.275866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.290951+0000","last_change":"2023-10-14T16:41:38.266472+0000","last_active":"2023-10-14T16:41:39.290951+0000","last_peered":"2023-10-14T16:41:39.290951+0000","last_clean":"2023-10-14T16:41:39.290951+0000","last_became_active":"2023-10-14T16:41:38.266130+0000","last_became_peered":"2023-10-14T16:41:38.266130+0000","last_unstale":"2023-10-14T16:41:39.290951+0000","last_undegraded":"2023-10-14T16:41:39.290951+0000","last_fullsized":"2023-10-14T16:41:39.290951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751891+0000","last_change":"2023-10-14T16:41:39.282136+0000","last_active":"2023-10-14T16:41:40.751891+0000","last_peered":"2023-10-14T16:41:40.751891+0000","last_clean":"2023-10-14T16:41:40.751891+0000","last_became_active":"2023-10-14T16:41:39.281908+0000","last_became_peered":"2023-10-14T16:41:39.281908+0000","last_unstale":"2023-10-14T16:41:40.751891+0000","last_undegraded":"2023-10-14T16:41:40.751891+0000","last_fullsized":"2023-10-14T16:41:40.751891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:22.816208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.298292+0000","last_change":"2023-10-14T16:41:38.270437+0000","last_active":"2023-10-14T16:41:39.298292+0000","last_peered":"2023-10-14T16:41:39.298292+0000","last_clean":"2023-10-14T16:41:39.298292+0000","last_became_active":"2023-10-14T16:41:38.270157+0000","last_became_peered":"2023-10-14T16:41:38.270157+0000","last_unstale":"2023-10-14T16:41:39.298292+0000","last_undegraded":"2023-10-14T16:41:39.298292+0000","last_fullsized":"2023-10-14T16:41:39.298292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:59.304198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750158+0000","last_change":"2023-10-14T16:41:39.277102+0000","last_active":"2023-10-14T16:41:40.750158+0000","last_peered":"2023-10-14T16:41:40.750158+0000","last_clean":"2023-10-14T16:41:40.750158+0000","last_became_active":"2023-10-14T16:41:39.276670+0000","last_became_peered":"2023-10-14T16:41:39.276670+0000","last_unstale":"2023-10-14T16:41:40.750158+0000","last_undegraded":"2023-10-14T16:41:40.750158+0000","last_fullsized":"2023-10-14T16:41:40.750158+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.292977+0000","last_change":"2023-10-14T16:41:38.271123+0000","last_active":"2023-10-14T16:41:39.292977+0000","last_peered":"2023-10-14T16:41:39.292977+0000","last_clean":"2023-10-14T16:41:39.292977+0000","last_became_active":"2023-10-14T16:41:38.270752+0000","last_became_peered":"2023-10-14T16:41:38.270752+0000","last_unstale":"2023-10-14T16:41:39.292977+0000","last_undegraded":"2023-10-14T16:41:39.292977+0000","last_fullsized":"2023-10-14T16:41:39.292977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:46:16.492082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276762+0000","last_change":"2023-10-14T16:41:39.285605+0000","last_active":"2023-10-14T16:41:40.276762+0000","last_peered":"2023-10-14T16:41:40.276762+0000","last_clean":"2023-10-14T16:41:40.276762+0000","last_became_active":"2023-10-14T16:41:39.285255+0000","last_became_peered":"2023-10-14T16:41:39.285255+0000","last_unstale":"2023-10-14T16:41:40.276762+0000","last_undegraded":"2023-10-14T16:41:40.276762+0000","last_fullsized":"2023-10-14T16:41:40.276762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296659+0000","last_change":"2023-10-14T16:41:38.263100+0000","last_active":"2023-10-14T16:41:39.296659+0000","last_peered":"2023-10-14T16:41:39.296659+0000","last_clean":"2023-10-14T16:41:39.296659+0000","last_became_active":"2023-10-14T16:41:38.262254+0000","last_became_peered":"2023-10-14T16:41:38.262254+0000","last_unstale":"2023-10-14T16:41:39.296659+0000","last_undegraded":"2023-10-14T16:41:39.296659+0000","last_fullsized":"2023-10-14T16:41:39.296659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749849+0000","last_change":"2023-10-14T16:41:39.280390+0000","last_active":"2023-10-14T16:41:40.749849+0000","last_peered":"2023-10-14T16:41:40.749849+0000","last_clean":"2023-10-14T16:41:40.749849+0000","last_became_active":"2023-10-14T16:41:39.280034+0000","last_became_peered":"2023-10-14T16:41:39.280034+0000","last_unstale":"2023-10-14T16:41:40.749849+0000","last_undegraded":"2023-10-14T16:41:40.749849+0000","last_fullsized":"2023-10-14T16:41:40.749849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296823+0000","last_change":"2023-10-14T16:41:38.259912+0000","last_active":"2023-10-14T16:41:39.296823+0000","last_peered":"2023-10-14T16:41:39.296823+0000","last_clean":"2023-10-14T16:41:39.296823+0000","last_became_active":"2023-10-14T16:41:38.258669+0000","last_became_peered":"2023-10-14T16:41:38.258669+0000","last_unstale":"2023-10-14T16:41:39.296823+0000","last_undegraded":"2023-10-14T16:41:39.296823+0000","last_fullsized":"2023-10-14T16:41:39.296823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750622+0000","last_change":"2023-10-14T16:41:39.262663+0000","last_active":"2023-10-14T16:41:40.750622+0000","last_peered":"2023-10-14T16:41:40.750622+0000","last_clean":"2023-10-14T16:41:40.750622+0000","last_became_active":"2023-10-14T16:41:39.262161+0000","last_became_peered":"2023-10-14T16:41:39.262161+0000","last_unstale":"2023-10-14T16:41:40.750622+0000","last_undegraded":"2023-10-14T16:41:40.750622+0000","last_fullsized":"2023-10-14T16:41:40.750622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.260565+0000","last_change":"2023-10-14T16:41:38.257048+0000","last_active":"2023-10-14T16:41:39.260565+0000","last_peered":"2023-10-14T16:41:39.260565+0000","last_clean":"2023-10-14T16:41:39.260565+0000","last_became_active":"2023-10-14T16:41:38.256705+0000","last_became_peered":"2023-10-14T16:41:38.256705+0000","last_unstale":"2023-10-14T16:41:39.260565+0000","last_undegraded":"2023-10-14T16:41:39.260565+0000","last_fullsized":"2023-10-14T16:41:39.260565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:34:07.807752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.748527+0000","last_change":"2023-10-14T16:41:39.276961+0000","last_active":"2023-10-14T16:41:40.748527+0000","last_peered":"2023-10-14T16:41:40.748527+0000","last_clean":"2023-10-14T16:41:40.748527+0000","last_became_active":"2023-10-14T16:41:39.276539+0000","last_became_peered":"2023-10-14T16:41:39.276539+0000","last_unstale":"2023-10-14T16:41:40.748527+0000","last_undegraded":"2023-10-14T16:41:40.748527+0000","last_fullsized":"2023-10-14T16:41:40.748527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:33.006319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.250590+0000","last_change":"2023-10-14T16:41:38.267598+0000","last_active":"2023-10-14T16:41:39.250590+0000","last_peered":"2023-10-14T16:41:39.250590+0000","last_clean":"2023-10-14T16:41:39.250590+0000","last_became_active":"2023-10-14T16:41:38.266568+0000","last_became_peered":"2023-10-14T16:41:38.266568+0000","last_unstale":"2023-10-14T16:41:39.250590+0000","last_undegraded":"2023-10-14T16:41:39.250590+0000","last_fullsized":"2023-10-14T16:41:39.250590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:51.871838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.752019+0000","last_change":"2023-10-14T16:41:39.290107+0000","last_active":"2023-10-14T16:41:40.752019+0000","last_peered":"2023-10-14T16:41:40.752019+0000","last_clean":"2023-10-14T16:41:40.752019+0000","last_became_active":"2023-10-14T16:41:39.287409+0000","last_became_peered":"2023-10-14T16:41:39.287409+0000","last_unstale":"2023-10-14T16:41:40.752019+0000","last_undegraded":"2023-10-14T16:41:40.752019+0000","last_fullsized":"2023-10-14T16:41:40.752019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.270851+0000","last_change":"2023-10-14T16:41:38.269698+0000","last_active":"2023-10-14T16:41:39.270851+0000","last_peered":"2023-10-14T16:41:39.270851+0000","last_clean":"2023-10-14T16:41:39.270851+0000","last_became_active":"2023-10-14T16:41:38.269391+0000","last_became_peered":"2023-10-14T16:41:38.269391+0000","last_unstale":"2023-10-14T16:41:39.270851+0000","last_undegraded":"2023-10-14T16:41:39.270851+0000","last_fullsized":"2023-10-14T16:41:39.270851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:45.278582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273900+0000","last_change":"2023-10-14T16:41:39.293266+0000","last_active":"2023-10-14T16:41:40.273900+0000","last_peered":"2023-10-14T16:41:40.273900+0000","last_clean":"2023-10-14T16:41:40.273900+0000","last_became_active":"2023-10-14T16:41:39.289266+0000","last_became_peered":"2023-10-14T16:41:39.289266+0000","last_unstale":"2023-10-14T16:41:40.273900+0000","last_undegraded":"2023-10-14T16:41:40.273900+0000","last_fullsized":"2023-10-14T16:41:40.273900+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:53:17.963190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255190+0000","last_change":"2023-10-14T16:41:38.259142+0000","last_active":"2023-10-14T16:41:39.255190+0000","last_peered":"2023-10-14T16:41:39.255190+0000","last_clean":"2023-10-14T16:41:39.255190+0000","last_became_active":"2023-10-14T16:41:38.258853+0000","last_became_peered":"2023-10-14T16:41:38.258853+0000","last_unstale":"2023-10-14T16:41:39.255190+0000","last_undegraded":"2023-10-14T16:41:39.255190+0000","last_fullsized":"2023-10-14T16:41:39.255190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273508+0000","last_change":"2023-10-14T16:41:39.281438+0000","last_active":"2023-10-14T16:41:40.273508+0000","last_peered":"2023-10-14T16:41:40.273508+0000","last_clean":"2023-10-14T16:41:40.273508+0000","last_became_active":"2023-10-14T16:41:39.280886+0000","last_became_peered":"2023-10-14T16:41:39.280886+0000","last_unstale":"2023-10-14T16:41:40.273508+0000","last_undegraded":"2023-10-14T16:41:40.273508+0000","last_fullsized":"2023-10-14T16:41:40.273508+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.293349+0000","last_change":"2023-10-14T16:41:38.269707+0000","last_active":"2023-10-14T16:41:39.293349+0000","last_peered":"2023-10-14T16:41:39.293349+0000","last_clean":"2023-10-14T16:41:39.293349+0000","last_became_active":"2023-10-14T16:41:38.269101+0000","last_became_peered":"2023-10-14T16:41:38.269101+0000","last_unstale":"2023-10-14T16:41:39.293349+0000","last_undegraded":"2023-10-14T16:41:39.293349+0000","last_fullsized":"2023-10-14T16:41:39.293349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:25.343787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750734+0000","last_change":"2023-10-14T16:41:39.280373+0000","last_active":"2023-10-14T16:41:40.750734+0000","last_peered":"2023-10-14T16:41:40.750734+0000","last_clean":"2023-10-14T16:41:40.750734+0000","last_became_active":"2023-10-14T16:41:39.280069+0000","last_became_peered":"2023-10-14T16:41:39.280069+0000","last_unstale":"2023-10-14T16:41:40.750734+0000","last_undegraded":"2023-10-14T16:41:40.750734+0000","last_fullsized":"2023-10-14T16:41:40.750734+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.301804+0000","last_change":"2023-10-14T16:41:38.268083+0000","last_active":"2023-10-14T16:41:39.301804+0000","last_peered":"2023-10-14T16:41:39.301804+0000","last_clean":"2023-10-14T16:41:39.301804+0000","last_became_active":"2023-10-14T16:41:38.267791+0000","last_became_peered":"2023-10-14T16:41:38.267791+0000","last_unstale":"2023-10-14T16:41:39.301804+0000","last_undegraded":"2023-10-14T16:41:39.301804+0000","last_fullsized":"2023-10-14T16:41:39.301804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.748547+0000","last_change":"2023-10-14T16:41:39.277743+0000","last_active":"2023-10-14T16:41:40.748547+0000","last_peered":"2023-10-14T16:41:40.748547+0000","last_clean":"2023-10-14T16:41:40.748547+0000","last_became_active":"2023-10-14T16:41:39.277482+0000","last_became_peered":"2023-10-14T16:41:39.277482+0000","last_unstale":"2023-10-14T16:41:40.748547+0000","last_undegraded":"2023-10-14T16:41:40.748547+0000","last_fullsized":"2023-10-14T16:41:40.748547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:34:08.822592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294903+0000","last_change":"2023-10-14T16:41:38.259070+0000","last_active":"2023-10-14T16:41:39.294903+0000","last_peered":"2023-10-14T16:41:39.294903+0000","last_clean":"2023-10-14T16:41:39.294903+0000","last_became_active":"2023-10-14T16:41:38.258658+0000","last_became_peered":"2023-10-14T16:41:38.258658+0000","last_unstale":"2023-10-14T16:41:39.294903+0000","last_undegraded":"2023-10-14T16:41:39.294903+0000","last_fullsized":"2023-10-14T16:41:39.294903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273225+0000","last_change":"2023-10-14T16:41:39.275431+0000","last_active":"2023-10-14T16:41:40.273225+0000","last_peered":"2023-10-14T16:41:40.273225+0000","last_clean":"2023-10-14T16:41:40.273225+0000","last_became_active":"2023-10-14T16:41:39.275138+0000","last_became_peered":"2023-10-14T16:41:39.275138+0000","last_unstale":"2023-10-14T16:41:40.273225+0000","last_undegraded":"2023-10-14T16:41:40.273225+0000","last_fullsized":"2023-10-14T16:41:40.273225+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.264902+0000","last_change":"2023-10-14T16:41:38.265796+0000","last_active":"2023-10-14T16:41:39.264902+0000","last_peered":"2023-10-14T16:41:39.264902+0000","last_clean":"2023-10-14T16:41:39.264902+0000","last_became_active":"2023-10-14T16:41:38.265491+0000","last_became_peered":"2023-10-14T16:41:38.265491+0000","last_unstale":"2023-10-14T16:41:39.264902+0000","last_undegraded":"2023-10-14T16:41:39.264902+0000","last_fullsized":"2023-10-14T16:41:39.264902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:27.930694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275533+0000","last_change":"2023-10-14T16:41:39.283974+0000","last_active":"2023-10-14T16:41:40.275533+0000","last_peered":"2023-10-14T16:41:40.275533+0000","last_clean":"2023-10-14T16:41:40.275533+0000","last_became_active":"2023-10-14T16:41:39.283465+0000","last_became_peered":"2023-10-14T16:41:39.283465+0000","last_unstale":"2023-10-14T16:41:40.275533+0000","last_undegraded":"2023-10-14T16:41:40.275533+0000","last_fullsized":"2023-10-14T16:41:40.275533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.260953+0000","last_change":"2023-10-14T16:41:38.271323+0000","last_active":"2023-10-14T16:41:39.260953+0000","last_peered":"2023-10-14T16:41:39.260953+0000","last_clean":"2023-10-14T16:41:39.260953+0000","last_became_active":"2023-10-14T16:41:38.271014+0000","last_became_peered":"2023-10-14T16:41:38.271014+0000","last_unstale":"2023-10-14T16:41:39.260953+0000","last_undegraded":"2023-10-14T16:41:39.260953+0000","last_fullsized":"2023-10-14T16:41:39.260953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:39:56.455065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275149+0000","last_change":"2023-10-14T16:41:39.272441+0000","last_active":"2023-10-14T16:41:40.275149+0000","last_peered":"2023-10-14T16:41:40.275149+0000","last_clean":"2023-10-14T16:41:40.275149+0000","last_became_active":"2023-10-14T16:41:39.271666+0000","last_became_peered":"2023-10-14T16:41:39.271666+0000","last_unstale":"2023-10-14T16:41:40.275149+0000","last_undegraded":"2023-10-14T16:41:40.275149+0000","last_fullsized":"2023-10-14T16:41:40.275149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.299798+0000","last_change":"2023-10-14T16:41:38.272215+0000","last_active":"2023-10-14T16:41:39.299798+0000","last_peered":"2023-10-14T16:41:39.299798+0000","last_clean":"2023-10-14T16:41:39.299798+0000","last_became_active":"2023-10-14T16:41:38.271943+0000","last_became_peered":"2023-10-14T16:41:38.271943+0000","last_unstale":"2023-10-14T16:41:39.299798+0000","last_undegraded":"2023-10-14T16:41:39.299798+0000","last_fullsized":"2023-10-14T16:41:39.299798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:27.930694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751505+0000","last_change":"2023-10-14T16:41:39.281314+0000","last_active":"2023-10-14T16:41:40.751505+0000","last_peered":"2023-10-14T16:41:40.751505+0000","last_clean":"2023-10-14T16:41:40.751505+0000","last_became_active":"2023-10-14T16:41:39.279602+0000","last_became_peered":"2023-10-14T16:41:39.279602+0000","last_unstale":"2023-10-14T16:41:40.751505+0000","last_undegraded":"2023-10-14T16:41:40.751505+0000","last_fullsized":"2023-10-14T16:41:40.751505+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:29.535419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296101+0000","last_change":"2023-10-14T16:41:38.267446+0000","last_active":"2023-10-14T16:41:39.296101+0000","last_peered":"2023-10-14T16:41:39.296101+0000","last_clean":"2023-10-14T16:41:39.296101+0000","last_became_active":"2023-10-14T16:41:38.267122+0000","last_became_peered":"2023-10-14T16:41:38.267122+0000","last_unstale":"2023-10-14T16:41:39.296101+0000","last_undegraded":"2023-10-14T16:41:39.296101+0000","last_fullsized":"2023-10-14T16:41:39.296101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:28.520579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751980+0000","last_change":"2023-10-14T16:41:39.275277+0000","last_active":"2023-10-14T16:41:40.751980+0000","last_peered":"2023-10-14T16:41:40.751980+0000","last_clean":"2023-10-14T16:41:40.751980+0000","last_became_active":"2023-10-14T16:41:39.273250+0000","last_became_peered":"2023-10-14T16:41:39.273250+0000","last_unstale":"2023-10-14T16:41:40.751980+0000","last_undegraded":"2023-10-14T16:41:40.751980+0000","last_fullsized":"2023-10-14T16:41:40.751980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.295047+0000","last_change":"2023-10-14T16:41:38.266145+0000","last_active":"2023-10-14T16:41:39.295047+0000","last_peered":"2023-10-14T16:41:39.295047+0000","last_clean":"2023-10-14T16:41:39.295047+0000","last_became_active":"2023-10-14T16:41:38.264561+0000","last_became_peered":"2023-10-14T16:41:38.264561+0000","last_unstale":"2023-10-14T16:41:39.295047+0000","last_undegraded":"2023-10-14T16:41:39.295047+0000","last_fullsized":"2023-10-14T16:41:39.295047+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749516+0000","last_change":"2023-10-14T16:41:39.273389+0000","last_active":"2023-10-14T16:41:40.749516+0000","last_peered":"2023-10-14T16:41:40.749516+0000","last_clean":"2023-10-14T16:41:40.749516+0000","last_became_active":"2023-10-14T16:41:39.273083+0000","last_became_peered":"2023-10-14T16:41:39.273083+0000","last_unstale":"2023-10-14T16:41:40.749516+0000","last_undegraded":"2023-10-14T16:41:40.749516+0000","last_fullsized":"2023-10-14T16:41:40.749516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:26.358627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294514+0000","last_change":"2023-10-14T16:41:38.266046+0000","last_active":"2023-10-14T16:41:39.294514+0000","last_peered":"2023-10-14T16:41:39.294514+0000","last_clean":"2023-10-14T16:41:39.294514+0000","last_became_active":"2023-10-14T16:41:38.263725+0000","last_became_peered":"2023-10-14T16:41:38.263725+0000","last_unstale":"2023-10-14T16:41:39.294514+0000","last_undegraded":"2023-10-14T16:41:39.294514+0000","last_fullsized":"2023-10-14T16:41:39.294514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751656+0000","last_change":"2023-10-14T16:41:39.278224+0000","last_active":"2023-10-14T16:41:40.751656+0000","last_peered":"2023-10-14T16:41:40.751656+0000","last_clean":"2023-10-14T16:41:40.751656+0000","last_became_active":"2023-10-14T16:41:39.277986+0000","last_became_peered":"2023-10-14T16:41:39.277986+0000","last_unstale":"2023-10-14T16:41:40.751656+0000","last_undegraded":"2023-10-14T16:41:40.751656+0000","last_fullsized":"2023-10-14T16:41:40.751656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:49.911818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.254835+0000","last_change":"2023-10-14T16:41:38.263662+0000","last_active":"2023-10-14T16:41:39.254835+0000","last_peered":"2023-10-14T16:41:39.254835+0000","last_clean":"2023-10-14T16:41:39.254835+0000","last_became_active":"2023-10-14T16:41:38.262441+0000","last_became_peered":"2023-10-14T16:41:38.262441+0000","last_unstale":"2023-10-14T16:41:39.254835+0000","last_undegraded":"2023-10-14T16:41:39.254835+0000","last_fullsized":"2023-10-14T16:41:39.254835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750772+0000","last_change":"2023-10-14T16:41:39.266628+0000","last_active":"2023-10-14T16:41:40.750772+0000","last_peered":"2023-10-14T16:41:40.750772+0000","last_clean":"2023-10-14T16:41:40.750772+0000","last_became_active":"2023-10-14T16:41:39.266023+0000","last_became_peered":"2023-10-14T16:41:39.266023+0000","last_unstale":"2023-10-14T16:41:40.750772+0000","last_undegraded":"2023-10-14T16:41:40.750772+0000","last_fullsized":"2023-10-14T16:41:40.750772+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:58.541591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.291472+0000","last_change":"2023-10-14T16:41:38.270622+0000","last_active":"2023-10-14T16:41:39.291472+0000","last_peered":"2023-10-14T16:41:39.291472+0000","last_clean":"2023-10-14T16:41:39.291472+0000","last_became_active":"2023-10-14T16:41:38.269459+0000","last_became_peered":"2023-10-14T16:41:38.269459+0000","last_unstale":"2023-10-14T16:41:39.291472+0000","last_undegraded":"2023-10-14T16:41:39.291472+0000","last_fullsized":"2023-10-14T16:41:39.291472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751161+0000","last_change":"2023-10-14T16:41:39.273339+0000","last_active":"2023-10-14T16:41:40.751161+0000","last_peered":"2023-10-14T16:41:40.751161+0000","last_clean":"2023-10-14T16:41:40.751161+0000","last_became_active":"2023-10-14T16:41:39.273045+0000","last_became_peered":"2023-10-14T16:41:39.273045+0000","last_unstale":"2023-10-14T16:41:40.751161+0000","last_undegraded":"2023-10-14T16:41:40.751161+0000","last_fullsized":"2023-10-14T16:41:40.751161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:18.315197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.249521+0000","last_change":"2023-10-14T16:41:38.270414+0000","last_active":"2023-10-14T16:41:39.249521+0000","last_peered":"2023-10-14T16:41:39.249521+0000","last_clean":"2023-10-14T16:41:39.249521+0000","last_became_active":"2023-10-14T16:41:38.269445+0000","last_became_peered":"2023-10-14T16:41:38.269445+0000","last_unstale":"2023-10-14T16:41:39.249521+0000","last_undegraded":"2023-10-14T16:41:39.249521+0000","last_fullsized":"2023-10-14T16:41:39.249521+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750513+0000","last_change":"2023-10-14T16:41:39.273636+0000","last_active":"2023-10-14T16:41:40.750513+0000","last_peered":"2023-10-14T16:41:40.750513+0000","last_clean":"2023-10-14T16:41:40.750513+0000","last_became_active":"2023-10-14T16:41:39.272522+0000","last_became_peered":"2023-10-14T16:41:39.272522+0000","last_unstale":"2023-10-14T16:41:40.750513+0000","last_undegraded":"2023-10-14T16:41:40.750513+0000","last_fullsized":"2023-10-14T16:41:40.750513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:52.886679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255580+0000","last_change":"2023-10-14T16:41:38.271734+0000","last_active":"2023-10-14T16:41:39.255580+0000","last_peered":"2023-10-14T16:41:39.255580+0000","last_clean":"2023-10-14T16:41:39.255580+0000","last_became_active":"2023-10-14T16:41:38.271386+0000","last_became_peered":"2023-10-14T16:41:38.271386+0000","last_unstale":"2023-10-14T16:41:39.255580+0000","last_undegraded":"2023-10-14T16:41:39.255580+0000","last_fullsized":"2023-10-14T16:41:39.255580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275968+0000","last_change":"2023-10-14T16:41:39.277752+0000","last_active":"2023-10-14T16:41:40.275968+0000","last_peered":"2023-10-14T16:41:40.275968+0000","last_clean":"2023-10-14T16:41:40.275968+0000","last_became_active":"2023-10-14T16:41:39.276648+0000","last_became_peered":"2023-10-14T16:41:39.276648+0000","last_unstale":"2023-10-14T16:41:40.275968+0000","last_undegraded":"2023-10-14T16:41:40.275968+0000","last_fullsized":"2023-10-14T16:41:40.275968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.259544+0000","last_change":"2023-10-14T16:41:38.266162+0000","last_active":"2023-10-14T16:41:39.259544+0000","last_peered":"2023-10-14T16:41:39.259544+0000","last_clean":"2023-10-14T16:41:39.259544+0000","last_became_active":"2023-10-14T16:41:38.265806+0000","last_became_peered":"2023-10-14T16:41:38.265806+0000","last_unstale":"2023-10-14T16:41:39.259544+0000","last_undegraded":"2023-10-14T16:41:39.259544+0000","last_fullsized":"2023-10-14T16:41:39.259544+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751528+0000","last_change":"2023-10-14T16:41:39.276093+0000","last_active":"2023-10-14T16:41:40.751528+0000","last_peered":"2023-10-14T16:41:40.751528+0000","last_clean":"2023-10-14T16:41:40.751528+0000","last_became_active":"2023-10-14T16:41:39.275759+0000","last_became_peered":"2023-10-14T16:41:39.275759+0000","last_unstale":"2023-10-14T16:41:40.751528+0000","last_undegraded":"2023-10-14T16:41:40.751528+0000","last_fullsized":"2023-10-14T16:41:40.751528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:26.451032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.293345+0000","last_change":"2023-10-14T16:41:38.271489+0000","last_active":"2023-10-14T16:41:39.293345+0000","last_peered":"2023-10-14T16:41:39.293345+0000","last_clean":"2023-10-14T16:41:39.293345+0000","last_became_active":"2023-10-14T16:41:38.271096+0000","last_became_peered":"2023-10-14T16:41:38.271096+0000","last_unstale":"2023-10-14T16:41:39.293345+0000","last_undegraded":"2023-10-14T16:41:39.293345+0000","last_fullsized":"2023-10-14T16:41:39.293345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751953+0000","last_change":"2023-10-14T16:41:39.278834+0000","last_active":"2023-10-14T16:41:40.751953+0000","last_peered":"2023-10-14T16:41:40.751953+0000","last_clean":"2023-10-14T16:41:40.751953+0000","last_became_active":"2023-10-14T16:41:39.278474+0000","last_became_peered":"2023-10-14T16:41:39.278474+0000","last_unstale":"2023-10-14T16:41:40.751953+0000","last_undegraded":"2023-10-14T16:41:40.751953+0000","last_fullsized":"2023-10-14T16:41:40.751953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255282+0000","last_change":"2023-10-14T16:41:38.267492+0000","last_active":"2023-10-14T16:41:39.255282+0000","last_peered":"2023-10-14T16:41:39.255282+0000","last_clean":"2023-10-14T16:41:39.255282+0000","last_became_active":"2023-10-14T16:41:38.267208+0000","last_became_peered":"2023-10-14T16:41:38.267208+0000","last_unstale":"2023-10-14T16:41:39.255282+0000","last_undegraded":"2023-10-14T16:41:39.255282+0000","last_fullsized":"2023-10-14T16:41:39.255282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.406979+0000","last_change":"2023-10-14T16:41:39.270875+0000","last_active":"2023-10-14T16:41:40.406979+0000","last_peered":"2023-10-14T16:41:40.406979+0000","last_clean":"2023-10-14T16:41:40.406979+0000","last_became_active":"2023-10-14T16:41:39.270562+0000","last_became_peered":"2023-10-14T16:41:39.270562+0000","last_unstale":"2023-10-14T16:41:40.406979+0000","last_undegraded":"2023-10-14T16:41:40.406979+0000","last_fullsized":"2023-10-14T16:41:40.406979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.265266+0000","last_change":"2023-10-14T16:41:38.269649+0000","last_active":"2023-10-14T16:41:39.265266+0000","last_peered":"2023-10-14T16:41:39.265266+0000","last_clean":"2023-10-14T16:41:39.265266+0000","last_became_active":"2023-10-14T16:41:38.269092+0000","last_became_peered":"2023-10-14T16:41:38.269092+0000","last_unstale":"2023-10-14T16:41:39.265266+0000","last_undegraded":"2023-10-14T16:41:39.265266+0000","last_fullsized":"2023-10-14T16:41:39.265266+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:55.953985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.274712+0000","last_change":"2023-10-14T16:41:39.274584+0000","last_active":"2023-10-14T16:41:40.274712+0000","last_peered":"2023-10-14T16:41:40.274712+0000","last_clean":"2023-10-14T16:41:40.274712+0000","last_became_active":"2023-10-14T16:41:39.274261+0000","last_became_peered":"2023-10-14T16:41:39.274261+0000","last_unstale":"2023-10-14T16:41:40.274712+0000","last_undegraded":"2023-10-14T16:41:40.274712+0000","last_fullsized":"2023-10-14T16:41:40.274712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:28.199524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.254884+0000","last_change":"2023-10-14T16:41:38.260866+0000","last_active":"2023-10-14T16:41:39.254884+0000","last_peered":"2023-10-14T16:41:39.254884+0000","last_clean":"2023-10-14T16:41:39.254884+0000","last_became_active":"2023-10-14T16:41:38.260498+0000","last_became_peered":"2023-10-14T16:41:38.260498+0000","last_unstale":"2023-10-14T16:41:39.254884+0000","last_undegraded":"2023-10-14T16:41:39.254884+0000","last_fullsized":"2023-10-14T16:41:39.254884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:17.441986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407161+0000","last_change":"2023-10-14T16:41:39.269959+0000","last_active":"2023-10-14T16:41:40.407161+0000","last_peered":"2023-10-14T16:41:40.407161+0000","last_clean":"2023-10-14T16:41:40.407161+0000","last_became_active":"2023-10-14T16:41:39.267787+0000","last_became_peered":"2023-10-14T16:41:39.267787+0000","last_unstale":"2023-10-14T16:41:40.407161+0000","last_undegraded":"2023-10-14T16:41:40.407161+0000","last_fullsized":"2023-10-14T16:41:40.407161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.265079+0000","last_change":"2023-10-14T16:41:38.265778+0000","last_active":"2023-10-14T16:41:39.265079+0000","last_peered":"2023-10-14T16:41:39.265079+0000","last_clean":"2023-10-14T16:41:39.265079+0000","last_became_active":"2023-10-14T16:41:38.265477+0000","last_became_peered":"2023-10-14T16:41:38.265477+0000","last_unstale":"2023-10-14T16:41:39.265079+0000","last_undegraded":"2023-10-14T16:41:39.265079+0000","last_fullsized":"2023-10-14T16:41:39.265079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:14.919317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.271841+0000","last_change":"2023-10-14T16:41:39.282237+0000","last_active":"2023-10-14T16:41:40.271841+0000","last_peered":"2023-10-14T16:41:40.271841+0000","last_clean":"2023-10-14T16:41:40.271841+0000","last_became_active":"2023-10-14T16:41:39.281738+0000","last_became_peered":"2023-10-14T16:41:39.281738+0000","last_unstale":"2023-10-14T16:41:40.271841+0000","last_undegraded":"2023-10-14T16:41:40.271841+0000","last_fullsized":"2023-10-14T16:41:40.271841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255649+0000","last_change":"2023-10-14T16:41:38.265852+0000","last_active":"2023-10-14T16:41:39.255649+0000","last_peered":"2023-10-14T16:41:39.255649+0000","last_clean":"2023-10-14T16:41:39.255649+0000","last_became_active":"2023-10-14T16:41:38.265513+0000","last_became_peered":"2023-10-14T16:41:38.265513+0000","last_unstale":"2023-10-14T16:41:39.255649+0000","last_undegraded":"2023-10-14T16:41:39.255649+0000","last_fullsized":"2023-10-14T16:41:39.255649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750355+0000","last_change":"2023-10-14T16:41:39.276674+0000","last_active":"2023-10-14T16:41:40.750355+0000","last_peered":"2023-10-14T16:41:40.750355+0000","last_clean":"2023-10-14T16:41:40.750355+0000","last_became_active":"2023-10-14T16:41:39.274101+0000","last_became_peered":"2023-10-14T16:41:39.274101+0000","last_unstale":"2023-10-14T16:41:40.750355+0000","last_undegraded":"2023-10-14T16:41:40.750355+0000","last_fullsized":"2023-10-14T16:41:40.750355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.258177+0000","last_change":"2023-10-14T16:41:38.265704+0000","last_active":"2023-10-14T16:41:39.258177+0000","last_peered":"2023-10-14T16:41:39.258177+0000","last_clean":"2023-10-14T16:41:39.258177+0000","last_became_active":"2023-10-14T16:41:38.264927+0000","last_became_peered":"2023-10-14T16:41:38.264927+0000","last_unstale":"2023-10-14T16:41:39.258177+0000","last_undegraded":"2023-10-14T16:41:39.258177+0000","last_fullsized":"2023-10-14T16:41:39.258177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:14.919317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276871+0000","last_change":"2023-10-14T16:41:39.280368+0000","last_active":"2023-10-14T16:41:40.276871+0000","last_peered":"2023-10-14T16:41:40.276871+0000","last_clean":"2023-10-14T16:41:40.276871+0000","last_became_active":"2023-10-14T16:41:39.280081+0000","last_became_peered":"2023-10-14T16:41:39.280081+0000","last_unstale":"2023-10-14T16:41:40.276871+0000","last_undegraded":"2023-10-14T16:41:40.276871+0000","last_fullsized":"2023-10-14T16:41:40.276871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.249909+0000","last_change":"2023-10-14T16:41:38.267370+0000","last_active":"2023-10-14T16:41:39.249909+0000","last_peered":"2023-10-14T16:41:39.249909+0000","last_clean":"2023-10-14T16:41:39.249909+0000","last_became_active":"2023-10-14T16:41:38.267087+0000","last_became_peered":"2023-10-14T16:41:38.267087+0000","last_unstale":"2023-10-14T16:41:39.249909+0000","last_undegraded":"2023-10-14T16:41:39.249909+0000","last_fullsized":"2023-10-14T16:41:39.249909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.268835+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:41:39.268835+0000","last_peered":"2023-10-14T16:41:39.268835+0000","last_clean":"2023-10-14T16:41:39.268835+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:41:39.268835+0000","last_undegraded":"2023-10-14T16:41:39.268835+0000","last_fullsized":"2023-10-14T16:41:39.268835+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.173,"5min":1.173,"15min":1.173},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.173},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.321}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.054},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.2769999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.4139999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.486},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.7090000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.651}]},{"osd":6,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":1.8080000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.4419999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.1970000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":2.133},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.669}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":2.085},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":2.3199999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.335},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.9339999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":3.0110000000000001,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":3.206,"5min":3.206,"15min":3.206},"last":1.8180000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":2.2290000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":2.0009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":0.75}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":1.028}]},{"osd":2,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.3639999999999999,"5min":4.3639999999999999,"15min":4.3639999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":1.075}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":1.2030000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.2629999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.8169999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"last":1.863}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":4.4059999999999997},"last":0.96399999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.747},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.127}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.294},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.2070000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.026},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.4059999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.516}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.5880000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.8069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":153877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.98599999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.613},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":2.282}]},{"osd":4,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":1.278}]},{"osd":5,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.2490000000000001,"5min":3.2490000000000001,"15min":3.2490000000000001},"last":1.3280000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":1.429},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.7}]},{"osd":7,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.7789999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":389120,"data_stored":152313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.004},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":1.194}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.2410000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":1.4650000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.1200000000000001,"5min":3.1200000000000001,"15min":3.1200000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.4950000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":1.278},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.3029999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":153877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.63900000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":0.72799999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.071},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.88700000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.171},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.84999999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":4.6360000000000001,"5min":4.6360000000000001,"15min":4.6360000000000001},"last":0.872},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":0.96699999999999997}]},{"osd":7,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.94899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":978944,"data_stored":742611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.624}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.87}]},{"osd":4,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.93999999999999995}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.98199999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.337},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.2190000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.415}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:41:48.824 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:41:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.059+0000 7f0588872640 1 Processor -- start 2023-10-14T16:41:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.059+0000 7f0588872640 1 -- start start 2023-10-14T16:41:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.060+0000 7f0588872640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f058415c9f0 0x7f05840c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f0588872640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05840c9230 0x7f05840709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f0588872640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0584160a90 con 0x7f0584070f20 2023-10-14T16:41:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f0588872640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05840cdec0 con 0x7f058415c9f0 2023-10-14T16:41:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f0588872640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840cd900 con 0x7f05840c9230 2023-10-14T16:41:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f058415c9f0 0x7f05840c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f058415c9f0 0x7f05840c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39438/0 (socket says 172.21.15.29:39438) 2023-10-14T16:41:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 -- 172.21.15.29:0/2848563440 learned_addr learned my addr 172.21.15.29:0/2848563440 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f057bfff640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05840c9230 0x7f05840709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 -- 172.21.15.29:0/2848563440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05840c9230 msgr2=0x7f05840709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05840c9230 0x7f05840709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 -- 172.21.15.29:0/2848563440 >> v1:172.21.15.186:6789/0 conn(0x7f0584070f20 legacy=0x7f05840ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.061+0000 7f05837fe640 1 -- 172.21.15.29:0/2848563440 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05840cdbe0 con 0x7f058415c9f0 2023-10-14T16:41:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f05837fe640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058415c9f0 0x7f05840c8cf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0574009870 tx=0x7f057402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d4182785ab4b62d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f0582ffd640 1 -- 172.21.15.29:0/2848563440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574031640 con 0x7f058415c9f0 2023-10-14T16:41:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f0582ffd640 1 -- 172.21.15.29:0/2848563440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05740317a0 con 0x7f058415c9f0 2023-10-14T16:41:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f0582ffd640 1 -- 172.21.15.29:0/2848563440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0574031b30 con 0x7f058415c9f0 2023-10-14T16:41:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f0588872640 1 -- 172.21.15.29:0/2848563440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058415c9f0 msgr2=0x7f05840c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.063+0000 7f0588872640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058415c9f0 0x7f05840c8cf0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f0574009870 tx=0x7f057402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 -- 172.21.15.29:0/2848563440 shutdown_connections 2023-10-14T16:41:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05840c9230 0x7f05840709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 --2- 172.21.15.29:0/2848563440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058415c9f0 0x7f05840c8cf0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 -- 172.21.15.29:0/2848563440 >> 172.21.15.29:0/2848563440 conn(0x7f05840ce6a0 msgr2=0x7f05840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 -- 172.21.15.29:0/2848563440 shutdown_connections 2023-10-14T16:41:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.064+0000 7f0588872640 1 -- 172.21.15.29:0/2848563440 wait complete. 2023-10-14T16:41:49.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.065+0000 7f0588872640 1 Processor -- start 2023-10-14T16:41:49.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 -- start start 2023-10-14T16:41:49.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0584070f20 0x7f0584159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058415c9f0 0x7f05840c2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840c2a20 con 0x7f05840c9230 2023-10-14T16:41:49.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05840c23f0 con 0x7f0584070f20 2023-10-14T16:41:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.066+0000 7f0588872640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05840c4fe0 con 0x7f058415c9f0 2023-10-14T16:41:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39440/0 (socket says 172.21.15.29:39440) 2023-10-14T16:41:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 -- 172.21.15.29:0/136073442 learned_addr learned my addr 172.21.15.29:0/136073442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:49.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f05837fe640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0584070f20 0x7f0584159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 -- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058415c9f0 msgr2=0x7f05840c2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058415c9f0 0x7f05840c2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 -- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0584070f20 msgr2=0x7f0584159a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0584070f20 0x7f0584159a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.067+0000 7f057bfff640 1 -- 172.21.15.29:0/136073442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05740094d0 con 0x7f05840c9230 2023-10-14T16:41:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f057bfff640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c7f0 tx=0x7f056c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c004220 con 0x7f05840c9230 2023-10-14T16:41:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f056c0043a0 con 0x7f05840c9230 2023-10-14T16:41:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05840c52c0 con 0x7f05840c9230 2023-10-14T16:41:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f056c007860 con 0x7f05840c9230 2023-10-14T16:41:49.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.068+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0584159f70 con 0x7f05840c9230 2023-10-14T16:41:49.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.069+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05840c2b60 con 0x7f05840c9230 2023-10-14T16:41:49.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.070+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f056c007bd0 con 0x7f05840c9230 2023-10-14T16:41:49.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.071+0000 7f0580ff9640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 0x7f056406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.071+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f056c089640 con 0x7f05840c9230 2023-10-14T16:41:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.072+0000 7f05837fe640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 0x7f056406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.072+0000 7f05837fe640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 0x7f056406d330 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0574036000 tx=0x7f057403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.076+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05840c2b60 con 0x7f05840c9230 2023-10-14T16:41:49.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.269+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f05840704c0 con 0x7f056406ae70 2023-10-14T16:41:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.282+0000 7f0580ff9640 1 -- 172.21.15.29:0/136073442 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287024 (secure 0 0 0) 0x7f0574031ae0 con 0x7f056406ae70 2023-10-14T16:41:49.284 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:41:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 msgr2=0x7f056406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 0x7f056406d330 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0574036000 tx=0x7f057403c040 comp rx=0 tx=0).stop 2023-10-14T16:41:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 msgr2=0x7f05840c69f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f056c00c7f0 tx=0x7f056c00ccb0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 shutdown_connections 2023-10-14T16:41:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f056406ae70 0x7f056406d330 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058415c9f0 0x7f05840c2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05840c9230 0x7f05840c69f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 --2- 172.21.15.29:0/136073442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0584070f20 0x7f0584159a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 >> 172.21.15.29:0/136073442 conn(0x7f05840ce6a0 msgr2=0x7f05840ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 shutdown_connections 2023-10-14T16:41:49.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.289+0000 7f0588872640 1 -- 172.21.15.29:0/136073442 wait complete. 2023-10-14T16:41:49.313 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":74,"stamp":"2023-10-14T16:41:47.323495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":237084,"kb_used_data":4232,"kb_used_omap":12,"kb_used_meta":232691,"kb_avail":754737636,"statfs":{"total":773094113280,"available":772851339264,"internally_reserved":0,"allocated":4333568,"data_stored":2403500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":238276152},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":49,"apply_latency_ms":49,"commit_latency_ns":49000000,"apply_latency_ns":49000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.081573"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275868+0000","last_change":"2023-10-14T16:41:39.281568+0000","last_active":"2023-10-14T16:41:40.275868+0000","last_peered":"2023-10-14T16:41:40.275868+0000","last_clean":"2023-10-14T16:41:40.275868+0000","last_became_active":"2023-10-14T16:41:39.280093+0000","last_became_peered":"2023-10-14T16:41:39.280093+0000","last_unstale":"2023-10-14T16:41:40.275868+0000","last_undegraded":"2023-10-14T16:41:40.275868+0000","last_fullsized":"2023-10-14T16:41:40.275868+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273463+0000","last_change":"2023-10-14T16:41:39.275677+0000","last_active":"2023-10-14T16:41:40.273463+0000","last_peered":"2023-10-14T16:41:40.273463+0000","last_clean":"2023-10-14T16:41:40.273463+0000","last_became_active":"2023-10-14T16:41:39.275406+0000","last_became_peered":"2023-10-14T16:41:39.275406+0000","last_unstale":"2023-10-14T16:41:40.273463+0000","last_undegraded":"2023-10-14T16:41:40.273463+0000","last_fullsized":"2023-10-14T16:41:40.273463+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750971+0000","last_change":"2023-10-14T16:41:39.282923+0000","last_active":"2023-10-14T16:41:40.750971+0000","last_peered":"2023-10-14T16:41:40.750971+0000","last_clean":"2023-10-14T16:41:40.750971+0000","last_became_active":"2023-10-14T16:41:39.282613+0000","last_became_peered":"2023-10-14T16:41:39.282613+0000","last_unstale":"2023-10-14T16:41:40.750971+0000","last_undegraded":"2023-10-14T16:41:40.750971+0000","last_fullsized":"2023-10-14T16:41:40.750971+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749154+0000","last_change":"2023-10-14T16:41:39.275832+0000","last_active":"2023-10-14T16:41:40.749154+0000","last_peered":"2023-10-14T16:41:40.749154+0000","last_clean":"2023-10-14T16:41:40.749154+0000","last_became_active":"2023-10-14T16:41:39.275554+0000","last_became_peered":"2023-10-14T16:41:39.275554+0000","last_unstale":"2023-10-14T16:41:40.749154+0000","last_undegraded":"2023-10-14T16:41:40.749154+0000","last_fullsized":"2023-10-14T16:41:40.749154+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:39:57.469905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750712+0000","last_change":"2023-10-14T16:41:39.276882+0000","last_active":"2023-10-14T16:41:40.750712+0000","last_peered":"2023-10-14T16:41:40.750712+0000","last_clean":"2023-10-14T16:41:40.750712+0000","last_became_active":"2023-10-14T16:41:39.274595+0000","last_became_peered":"2023-10-14T16:41:39.274595+0000","last_unstale":"2023-10-14T16:41:40.750712+0000","last_undegraded":"2023-10-14T16:41:40.750712+0000","last_fullsized":"2023-10-14T16:41:40.750712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:08.333380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.272460+0000","last_change":"2023-10-14T16:41:39.289475+0000","last_active":"2023-10-14T16:41:40.272460+0000","last_peered":"2023-10-14T16:41:40.272460+0000","last_clean":"2023-10-14T16:41:40.272460+0000","last_became_active":"2023-10-14T16:41:39.287681+0000","last_became_peered":"2023-10-14T16:41:39.287681+0000","last_unstale":"2023-10-14T16:41:40.272460+0000","last_undegraded":"2023-10-14T16:41:40.272460+0000","last_fullsized":"2023-10-14T16:41:40.272460+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276308+0000","last_change":"2023-10-14T16:41:39.276259+0000","last_active":"2023-10-14T16:41:40.276308+0000","last_peered":"2023-10-14T16:41:40.276308+0000","last_clean":"2023-10-14T16:41:40.276308+0000","last_became_active":"2023-10-14T16:41:39.275950+0000","last_became_peered":"2023-10-14T16:41:39.275950+0000","last_unstale":"2023-10-14T16:41:40.276308+0000","last_undegraded":"2023-10-14T16:41:40.276308+0000","last_fullsized":"2023-10-14T16:41:40.276308+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275124+0000","last_change":"2023-10-14T16:41:39.282445+0000","last_active":"2023-10-14T16:41:40.275124+0000","last_peered":"2023-10-14T16:41:40.275124+0000","last_clean":"2023-10-14T16:41:40.275124+0000","last_became_active":"2023-10-14T16:41:39.280747+0000","last_became_peered":"2023-10-14T16:41:39.280747+0000","last_unstale":"2023-10-14T16:41:40.275124+0000","last_undegraded":"2023-10-14T16:41:40.275124+0000","last_fullsized":"2023-10-14T16:41:40.275124+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:26.358627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.752695+0000","last_change":"2023-10-14T16:41:39.281160+0000","last_active":"2023-10-14T16:41:40.752695+0000","last_peered":"2023-10-14T16:41:40.752695+0000","last_clean":"2023-10-14T16:41:40.752695+0000","last_became_active":"2023-10-14T16:41:39.280860+0000","last_became_peered":"2023-10-14T16:41:39.280860+0000","last_unstale":"2023-10-14T16:41:40.752695+0000","last_undegraded":"2023-10-14T16:41:40.752695+0000","last_fullsized":"2023-10-14T16:41:40.752695+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:33.006319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749273+0000","last_change":"2023-10-14T16:41:39.290545+0000","last_active":"2023-10-14T16:41:40.749273+0000","last_peered":"2023-10-14T16:41:40.749273+0000","last_clean":"2023-10-14T16:41:40.749273+0000","last_became_active":"2023-10-14T16:41:39.290234+0000","last_became_peered":"2023-10-14T16:41:39.290234+0000","last_unstale":"2023-10-14T16:41:40.749273+0000","last_undegraded":"2023-10-14T16:41:40.749273+0000","last_fullsized":"2023-10-14T16:41:40.749273+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:50.380762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276931+0000","last_change":"2023-10-14T16:41:39.283061+0000","last_active":"2023-10-14T16:41:40.276931+0000","last_peered":"2023-10-14T16:41:40.276931+0000","last_clean":"2023-10-14T16:41:40.276931+0000","last_became_active":"2023-10-14T16:41:39.282811+0000","last_became_peered":"2023-10-14T16:41:39.282811+0000","last_unstale":"2023-10-14T16:41:40.276931+0000","last_undegraded":"2023-10-14T16:41:40.276931+0000","last_fullsized":"2023-10-14T16:41:40.276931+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750368+0000","last_change":"2023-10-14T16:41:39.288795+0000","last_active":"2023-10-14T16:41:40.750368+0000","last_peered":"2023-10-14T16:41:40.750368+0000","last_clean":"2023-10-14T16:41:40.750368+0000","last_became_active":"2023-10-14T16:41:39.287348+0000","last_became_peered":"2023-10-14T16:41:39.287348+0000","last_unstale":"2023-10-14T16:41:40.750368+0000","last_undegraded":"2023-10-14T16:41:40.750368+0000","last_fullsized":"2023-10-14T16:41:40.750368+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751066+0000","last_change":"2023-10-14T16:41:39.266227+0000","last_active":"2023-10-14T16:41:40.751066+0000","last_peered":"2023-10-14T16:41:40.751066+0000","last_clean":"2023-10-14T16:41:40.751066+0000","last_became_active":"2023-10-14T16:41:39.265760+0000","last_became_peered":"2023-10-14T16:41:39.265760+0000","last_unstale":"2023-10-14T16:41:40.751066+0000","last_undegraded":"2023-10-14T16:41:40.751066+0000","last_fullsized":"2023-10-14T16:41:40.751066+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.391215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.272606+0000","last_change":"2023-10-14T16:41:39.278809+0000","last_active":"2023-10-14T16:41:40.272606+0000","last_peered":"2023-10-14T16:41:40.272606+0000","last_clean":"2023-10-14T16:41:40.272606+0000","last_became_active":"2023-10-14T16:41:39.274047+0000","last_became_peered":"2023-10-14T16:41:39.274047+0000","last_unstale":"2023-10-14T16:41:40.272606+0000","last_undegraded":"2023-10-14T16:41:40.272606+0000","last_fullsized":"2023-10-14T16:41:40.272606+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749818+0000","last_change":"2023-10-14T16:41:39.269986+0000","last_active":"2023-10-14T16:41:40.749818+0000","last_peered":"2023-10-14T16:41:40.749818+0000","last_clean":"2023-10-14T16:41:40.749818+0000","last_became_active":"2023-10-14T16:41:39.269641+0000","last_became_peered":"2023-10-14T16:41:39.269641+0000","last_unstale":"2023-10-14T16:41:40.749818+0000","last_undegraded":"2023-10-14T16:41:40.749818+0000","last_fullsized":"2023-10-14T16:41:40.749818+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276014+0000","last_change":"2023-10-14T16:41:39.292170+0000","last_active":"2023-10-14T16:41:40.276014+0000","last_peered":"2023-10-14T16:41:40.276014+0000","last_clean":"2023-10-14T16:41:40.276014+0000","last_became_active":"2023-10-14T16:41:39.291896+0000","last_became_peered":"2023-10-14T16:41:39.291896+0000","last_unstale":"2023-10-14T16:41:40.276014+0000","last_undegraded":"2023-10-14T16:41:40.276014+0000","last_fullsized":"2023-10-14T16:41:40.276014+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.277591+0000","last_change":"2023-10-14T16:41:39.287004+0000","last_active":"2023-10-14T16:41:40.277591+0000","last_peered":"2023-10-14T16:41:40.277591+0000","last_clean":"2023-10-14T16:41:40.277591+0000","last_became_active":"2023-10-14T16:41:39.286729+0000","last_became_peered":"2023-10-14T16:41:39.286729+0000","last_unstale":"2023-10-14T16:41:40.277591+0000","last_undegraded":"2023-10-14T16:41:40.277591+0000","last_fullsized":"2023-10-14T16:41:40.277591+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273643+0000","last_change":"2023-10-14T16:41:39.285284+0000","last_active":"2023-10-14T16:41:40.273643+0000","last_peered":"2023-10-14T16:41:40.273643+0000","last_clean":"2023-10-14T16:41:40.273643+0000","last_became_active":"2023-10-14T16:41:39.284719+0000","last_became_peered":"2023-10-14T16:41:39.284719+0000","last_unstale":"2023-10-14T16:41:40.273643+0000","last_undegraded":"2023-10-14T16:41:40.273643+0000","last_fullsized":"2023-10-14T16:41:40.273643+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:49.310290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273355+0000","last_change":"2023-10-14T16:41:39.287370+0000","last_active":"2023-10-14T16:41:40.273355+0000","last_peered":"2023-10-14T16:41:40.273355+0000","last_clean":"2023-10-14T16:41:40.273355+0000","last_became_active":"2023-10-14T16:41:39.286680+0000","last_became_peered":"2023-10-14T16:41:39.286680+0000","last_unstale":"2023-10-14T16:41:40.273355+0000","last_undegraded":"2023-10-14T16:41:40.273355+0000","last_fullsized":"2023-10-14T16:41:40.273355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751330+0000","last_change":"2023-10-14T16:41:39.275840+0000","last_active":"2023-10-14T16:41:40.751330+0000","last_peered":"2023-10-14T16:41:40.751330+0000","last_clean":"2023-10-14T16:41:40.751330+0000","last_became_active":"2023-10-14T16:41:39.275555+0000","last_became_peered":"2023-10-14T16:41:39.275555+0000","last_unstale":"2023-10-14T16:41:40.751330+0000","last_undegraded":"2023-10-14T16:41:40.751330+0000","last_fullsized":"2023-10-14T16:41:40.751330+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:51.138406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751337+0000","last_change":"2023-10-14T16:41:39.273422+0000","last_active":"2023-10-14T16:41:40.751337+0000","last_peered":"2023-10-14T16:41:40.751337+0000","last_clean":"2023-10-14T16:41:40.751337+0000","last_became_active":"2023-10-14T16:41:39.273075+0000","last_became_peered":"2023-10-14T16:41:39.273075+0000","last_unstale":"2023-10-14T16:41:40.751337+0000","last_undegraded":"2023-10-14T16:41:40.751337+0000","last_fullsized":"2023-10-14T16:41:40.751337+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:15.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751409+0000","last_change":"2023-10-14T16:41:39.280965+0000","last_active":"2023-10-14T16:41:40.751409+0000","last_peered":"2023-10-14T16:41:40.751409+0000","last_clean":"2023-10-14T16:41:40.751409+0000","last_became_active":"2023-10-14T16:41:39.276351+0000","last_became_peered":"2023-10-14T16:41:39.276351+0000","last_unstale":"2023-10-14T16:41:40.751409+0000","last_undegraded":"2023-10-14T16:41:40.751409+0000","last_fullsized":"2023-10-14T16:41:40.751409+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:56.968826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407416+0000","last_change":"2023-10-14T16:41:39.275497+0000","last_active":"2023-10-14T16:41:40.407416+0000","last_peered":"2023-10-14T16:41:40.407416+0000","last_clean":"2023-10-14T16:41:40.407416+0000","last_became_active":"2023-10-14T16:41:39.275142+0000","last_became_peered":"2023-10-14T16:41:39.275142+0000","last_unstale":"2023-10-14T16:41:40.407416+0000","last_undegraded":"2023-10-14T16:41:40.407416+0000","last_fullsized":"2023-10-14T16:41:40.407416+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:15.934158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749599+0000","last_change":"2023-10-14T16:41:39.273988+0000","last_active":"2023-10-14T16:41:40.749599+0000","last_peered":"2023-10-14T16:41:40.749599+0000","last_clean":"2023-10-14T16:41:40.749599+0000","last_became_active":"2023-10-14T16:41:39.273710+0000","last_became_peered":"2023-10-14T16:41:39.273710+0000","last_unstale":"2023-10-14T16:41:40.749599+0000","last_undegraded":"2023-10-14T16:41:40.749599+0000","last_fullsized":"2023-10-14T16:41:40.749599+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749720+0000","last_change":"2023-10-14T16:41:39.277140+0000","last_active":"2023-10-14T16:41:40.749720+0000","last_peered":"2023-10-14T16:41:40.749720+0000","last_clean":"2023-10-14T16:41:40.749720+0000","last_became_active":"2023-10-14T16:41:39.276726+0000","last_became_peered":"2023-10-14T16:41:39.276726+0000","last_unstale":"2023-10-14T16:41:40.749720+0000","last_undegraded":"2023-10-14T16:41:40.749720+0000","last_fullsized":"2023-10-14T16:41:40.749720+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273623+0000","last_change":"2023-10-14T16:41:39.285065+0000","last_active":"2023-10-14T16:41:40.273623+0000","last_peered":"2023-10-14T16:41:40.273623+0000","last_clean":"2023-10-14T16:41:40.273623+0000","last_became_active":"2023-10-14T16:41:39.284290+0000","last_became_peered":"2023-10-14T16:41:39.284290+0000","last_unstale":"2023-10-14T16:41:40.273623+0000","last_undegraded":"2023-10-14T16:41:40.273623+0000","last_fullsized":"2023-10-14T16:41:40.273623+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:05.886795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750085+0000","last_change":"2023-10-14T16:41:39.278823+0000","last_active":"2023-10-14T16:41:40.750085+0000","last_peered":"2023-10-14T16:41:40.750085+0000","last_clean":"2023-10-14T16:41:40.750085+0000","last_became_active":"2023-10-14T16:41:39.277088+0000","last_became_peered":"2023-10-14T16:41:39.277088+0000","last_unstale":"2023-10-14T16:41:40.750085+0000","last_undegraded":"2023-10-14T16:41:40.750085+0000","last_fullsized":"2023-10-14T16:41:40.750085+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750719+0000","last_change":"2023-10-14T16:41:39.286089+0000","last_active":"2023-10-14T16:41:40.750719+0000","last_peered":"2023-10-14T16:41:40.750719+0000","last_clean":"2023-10-14T16:41:40.750719+0000","last_became_active":"2023-10-14T16:41:39.285814+0000","last_became_peered":"2023-10-14T16:41:39.285814+0000","last_unstale":"2023-10-14T16:41:40.750719+0000","last_undegraded":"2023-10-14T16:41:40.750719+0000","last_fullsized":"2023-10-14T16:41:40.750719+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751322+0000","last_change":"2023-10-14T16:41:39.283370+0000","last_active":"2023-10-14T16:41:40.751322+0000","last_peered":"2023-10-14T16:41:40.751322+0000","last_clean":"2023-10-14T16:41:40.751322+0000","last_became_active":"2023-10-14T16:41:39.282975+0000","last_became_peered":"2023-10-14T16:41:39.282975+0000","last_unstale":"2023-10-14T16:41:40.751322+0000","last_undegraded":"2023-10-14T16:41:40.751322+0000","last_fullsized":"2023-10-14T16:41:40.751322+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276247+0000","last_change":"2023-10-14T16:41:39.275366+0000","last_active":"2023-10-14T16:41:40.276247+0000","last_peered":"2023-10-14T16:41:40.276247+0000","last_clean":"2023-10-14T16:41:40.276247+0000","last_became_active":"2023-10-14T16:41:39.275045+0000","last_became_peered":"2023-10-14T16:41:39.275045+0000","last_unstale":"2023-10-14T16:41:40.276247+0000","last_undegraded":"2023-10-14T16:41:40.276247+0000","last_fullsized":"2023-10-14T16:41:40.276247+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:14.099299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.274820+0000","last_change":"2023-10-14T16:41:39.272725+0000","last_active":"2023-10-14T16:41:40.274820+0000","last_peered":"2023-10-14T16:41:40.274820+0000","last_clean":"2023-10-14T16:41:40.274820+0000","last_became_active":"2023-10-14T16:41:39.272130+0000","last_became_peered":"2023-10-14T16:41:39.272130+0000","last_unstale":"2023-10-14T16:41:40.274820+0000","last_undegraded":"2023-10-14T16:41:40.274820+0000","last_fullsized":"2023-10-14T16:41:40.274820+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:44:03.658409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275340+0000","last_change":"2023-10-14T16:41:39.284463+0000","last_active":"2023-10-14T16:41:40.275340+0000","last_peered":"2023-10-14T16:41:40.275340+0000","last_clean":"2023-10-14T16:41:40.275340+0000","last_became_active":"2023-10-14T16:41:39.283467+0000","last_became_peered":"2023-10-14T16:41:39.283467+0000","last_unstale":"2023-10-14T16:41:40.275340+0000","last_undegraded":"2023-10-14T16:41:40.275340+0000","last_fullsized":"2023-10-14T16:41:40.275340+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:26:54.639582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407077+0000","last_change":"2023-10-14T16:41:39.275061+0000","last_active":"2023-10-14T16:41:40.407077+0000","last_peered":"2023-10-14T16:41:40.407077+0000","last_clean":"2023-10-14T16:41:40.407077+0000","last_became_active":"2023-10-14T16:41:39.274764+0000","last_became_peered":"2023-10-14T16:41:39.274764+0000","last_unstale":"2023-10-14T16:41:40.407077+0000","last_undegraded":"2023-10-14T16:41:40.407077+0000","last_fullsized":"2023-10-14T16:41:40.407077+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"33'4","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294710+0000","last_change":"2023-10-14T16:41:38.263891+0000","last_active":"2023-10-14T16:41:39.294710+0000","last_peered":"2023-10-14T16:41:39.294710+0000","last_clean":"2023-10-14T16:41:39.294710+0000","last_became_active":"2023-10-14T16:41:38.263622+0000","last_became_peered":"2023-10-14T16:41:38.263622+0000","last_unstale":"2023-10-14T16:41:39.294710+0000","last_undegraded":"2023-10-14T16:41:39.294710+0000","last_fullsized":"2023-10-14T16:41:39.294710+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275866+0000","last_change":"2023-10-14T16:41:39.270667+0000","last_active":"2023-10-14T16:41:40.275866+0000","last_peered":"2023-10-14T16:41:40.275866+0000","last_clean":"2023-10-14T16:41:40.275866+0000","last_became_active":"2023-10-14T16:41:39.270375+0000","last_became_peered":"2023-10-14T16:41:39.270375+0000","last_unstale":"2023-10-14T16:41:40.275866+0000","last_undegraded":"2023-10-14T16:41:40.275866+0000","last_fullsized":"2023-10-14T16:41:40.275866+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.290951+0000","last_change":"2023-10-14T16:41:38.266472+0000","last_active":"2023-10-14T16:41:39.290951+0000","last_peered":"2023-10-14T16:41:39.290951+0000","last_clean":"2023-10-14T16:41:39.290951+0000","last_became_active":"2023-10-14T16:41:38.266130+0000","last_became_peered":"2023-10-14T16:41:38.266130+0000","last_unstale":"2023-10-14T16:41:39.290951+0000","last_undegraded":"2023-10-14T16:41:39.290951+0000","last_fullsized":"2023-10-14T16:41:39.290951+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751891+0000","last_change":"2023-10-14T16:41:39.282136+0000","last_active":"2023-10-14T16:41:40.751891+0000","last_peered":"2023-10-14T16:41:40.751891+0000","last_clean":"2023-10-14T16:41:40.751891+0000","last_became_active":"2023-10-14T16:41:39.281908+0000","last_became_peered":"2023-10-14T16:41:39.281908+0000","last_unstale":"2023-10-14T16:41:40.751891+0000","last_undegraded":"2023-10-14T16:41:40.751891+0000","last_fullsized":"2023-10-14T16:41:40.751891+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:22.816208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.298292+0000","last_change":"2023-10-14T16:41:38.270437+0000","last_active":"2023-10-14T16:41:39.298292+0000","last_peered":"2023-10-14T16:41:39.298292+0000","last_clean":"2023-10-14T16:41:39.298292+0000","last_became_active":"2023-10-14T16:41:38.270157+0000","last_became_peered":"2023-10-14T16:41:38.270157+0000","last_unstale":"2023-10-14T16:41:39.298292+0000","last_undegraded":"2023-10-14T16:41:39.298292+0000","last_fullsized":"2023-10-14T16:41:39.298292+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:59.304198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750158+0000","last_change":"2023-10-14T16:41:39.277102+0000","last_active":"2023-10-14T16:41:40.750158+0000","last_peered":"2023-10-14T16:41:40.750158+0000","last_clean":"2023-10-14T16:41:40.750158+0000","last_became_active":"2023-10-14T16:41:39.276670+0000","last_became_peered":"2023-10-14T16:41:39.276670+0000","last_unstale":"2023-10-14T16:41:40.750158+0000","last_undegraded":"2023-10-14T16:41:40.750158+0000","last_fullsized":"2023-10-14T16:41:40.750158+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:46.293423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.292977+0000","last_change":"2023-10-14T16:41:38.271123+0000","last_active":"2023-10-14T16:41:39.292977+0000","last_peered":"2023-10-14T16:41:39.292977+0000","last_clean":"2023-10-14T16:41:39.292977+0000","last_became_active":"2023-10-14T16:41:38.270752+0000","last_became_peered":"2023-10-14T16:41:38.270752+0000","last_unstale":"2023-10-14T16:41:39.292977+0000","last_undegraded":"2023-10-14T16:41:39.292977+0000","last_fullsized":"2023-10-14T16:41:39.292977+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:46:16.492082+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276762+0000","last_change":"2023-10-14T16:41:39.285605+0000","last_active":"2023-10-14T16:41:40.276762+0000","last_peered":"2023-10-14T16:41:40.276762+0000","last_clean":"2023-10-14T16:41:40.276762+0000","last_became_active":"2023-10-14T16:41:39.285255+0000","last_became_peered":"2023-10-14T16:41:39.285255+0000","last_unstale":"2023-10-14T16:41:40.276762+0000","last_undegraded":"2023-10-14T16:41:40.276762+0000","last_fullsized":"2023-10-14T16:41:40.276762+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296659+0000","last_change":"2023-10-14T16:41:38.263100+0000","last_active":"2023-10-14T16:41:39.296659+0000","last_peered":"2023-10-14T16:41:39.296659+0000","last_clean":"2023-10-14T16:41:39.296659+0000","last_became_active":"2023-10-14T16:41:38.262254+0000","last_became_peered":"2023-10-14T16:41:38.262254+0000","last_unstale":"2023-10-14T16:41:39.296659+0000","last_undegraded":"2023-10-14T16:41:39.296659+0000","last_fullsized":"2023-10-14T16:41:39.296659+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749849+0000","last_change":"2023-10-14T16:41:39.280390+0000","last_active":"2023-10-14T16:41:40.749849+0000","last_peered":"2023-10-14T16:41:40.749849+0000","last_clean":"2023-10-14T16:41:40.749849+0000","last_became_active":"2023-10-14T16:41:39.280034+0000","last_became_peered":"2023-10-14T16:41:39.280034+0000","last_unstale":"2023-10-14T16:41:40.749849+0000","last_undegraded":"2023-10-14T16:41:40.749849+0000","last_fullsized":"2023-10-14T16:41:40.749849+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296823+0000","last_change":"2023-10-14T16:41:38.259912+0000","last_active":"2023-10-14T16:41:39.296823+0000","last_peered":"2023-10-14T16:41:39.296823+0000","last_clean":"2023-10-14T16:41:39.296823+0000","last_became_active":"2023-10-14T16:41:38.258669+0000","last_became_peered":"2023-10-14T16:41:38.258669+0000","last_unstale":"2023-10-14T16:41:39.296823+0000","last_undegraded":"2023-10-14T16:41:39.296823+0000","last_fullsized":"2023-10-14T16:41:39.296823+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:06.376375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750622+0000","last_change":"2023-10-14T16:41:39.262663+0000","last_active":"2023-10-14T16:41:40.750622+0000","last_peered":"2023-10-14T16:41:40.750622+0000","last_clean":"2023-10-14T16:41:40.750622+0000","last_became_active":"2023-10-14T16:41:39.262161+0000","last_became_peered":"2023-10-14T16:41:39.262161+0000","last_unstale":"2023-10-14T16:41:40.750622+0000","last_undegraded":"2023-10-14T16:41:40.750622+0000","last_fullsized":"2023-10-14T16:41:40.750622+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:00.319038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.260565+0000","last_change":"2023-10-14T16:41:38.257048+0000","last_active":"2023-10-14T16:41:39.260565+0000","last_peered":"2023-10-14T16:41:39.260565+0000","last_clean":"2023-10-14T16:41:39.260565+0000","last_became_active":"2023-10-14T16:41:38.256705+0000","last_became_peered":"2023-10-14T16:41:38.256705+0000","last_unstale":"2023-10-14T16:41:39.260565+0000","last_undegraded":"2023-10-14T16:41:39.260565+0000","last_fullsized":"2023-10-14T16:41:39.260565+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:34:07.807752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.748527+0000","last_change":"2023-10-14T16:41:39.276961+0000","last_active":"2023-10-14T16:41:40.748527+0000","last_peered":"2023-10-14T16:41:40.748527+0000","last_clean":"2023-10-14T16:41:40.748527+0000","last_became_active":"2023-10-14T16:41:39.276539+0000","last_became_peered":"2023-10-14T16:41:39.276539+0000","last_unstale":"2023-10-14T16:41:40.748527+0000","last_undegraded":"2023-10-14T16:41:40.748527+0000","last_fullsized":"2023-10-14T16:41:40.748527+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:33.006319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.250590+0000","last_change":"2023-10-14T16:41:38.267598+0000","last_active":"2023-10-14T16:41:39.250590+0000","last_peered":"2023-10-14T16:41:39.250590+0000","last_clean":"2023-10-14T16:41:39.250590+0000","last_became_active":"2023-10-14T16:41:38.266568+0000","last_became_peered":"2023-10-14T16:41:38.266568+0000","last_unstale":"2023-10-14T16:41:39.250590+0000","last_undegraded":"2023-10-14T16:41:39.250590+0000","last_fullsized":"2023-10-14T16:41:39.250590+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:51.871838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.752019+0000","last_change":"2023-10-14T16:41:39.290107+0000","last_active":"2023-10-14T16:41:40.752019+0000","last_peered":"2023-10-14T16:41:40.752019+0000","last_clean":"2023-10-14T16:41:40.752019+0000","last_became_active":"2023-10-14T16:41:39.287409+0000","last_became_peered":"2023-10-14T16:41:39.287409+0000","last_unstale":"2023-10-14T16:41:40.752019+0000","last_undegraded":"2023-10-14T16:41:40.752019+0000","last_fullsized":"2023-10-14T16:41:40.752019+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.270851+0000","last_change":"2023-10-14T16:41:38.269698+0000","last_active":"2023-10-14T16:41:39.270851+0000","last_peered":"2023-10-14T16:41:39.270851+0000","last_clean":"2023-10-14T16:41:39.270851+0000","last_became_active":"2023-10-14T16:41:38.269391+0000","last_became_peered":"2023-10-14T16:41:38.269391+0000","last_unstale":"2023-10-14T16:41:39.270851+0000","last_undegraded":"2023-10-14T16:41:39.270851+0000","last_fullsized":"2023-10-14T16:41:39.270851+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:08:45.278582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273900+0000","last_change":"2023-10-14T16:41:39.293266+0000","last_active":"2023-10-14T16:41:40.273900+0000","last_peered":"2023-10-14T16:41:40.273900+0000","last_clean":"2023-10-14T16:41:40.273900+0000","last_became_active":"2023-10-14T16:41:39.289266+0000","last_became_peered":"2023-10-14T16:41:39.289266+0000","last_unstale":"2023-10-14T16:41:40.273900+0000","last_undegraded":"2023-10-14T16:41:40.273900+0000","last_fullsized":"2023-10-14T16:41:40.273900+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:53:17.963190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255190+0000","last_change":"2023-10-14T16:41:38.259142+0000","last_active":"2023-10-14T16:41:39.255190+0000","last_peered":"2023-10-14T16:41:39.255190+0000","last_clean":"2023-10-14T16:41:39.255190+0000","last_became_active":"2023-10-14T16:41:38.258853+0000","last_became_peered":"2023-10-14T16:41:38.258853+0000","last_unstale":"2023-10-14T16:41:39.255190+0000","last_undegraded":"2023-10-14T16:41:39.255190+0000","last_fullsized":"2023-10-14T16:41:39.255190+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:34:00.783913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273508+0000","last_change":"2023-10-14T16:41:39.281438+0000","last_active":"2023-10-14T16:41:40.273508+0000","last_peered":"2023-10-14T16:41:40.273508+0000","last_clean":"2023-10-14T16:41:40.273508+0000","last_became_active":"2023-10-14T16:41:39.280886+0000","last_became_peered":"2023-10-14T16:41:39.280886+0000","last_unstale":"2023-10-14T16:41:40.273508+0000","last_undegraded":"2023-10-14T16:41:40.273508+0000","last_fullsized":"2023-10-14T16:41:40.273508+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:57:26.709261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"33'3","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.293349+0000","last_change":"2023-10-14T16:41:38.269707+0000","last_active":"2023-10-14T16:41:39.293349+0000","last_peered":"2023-10-14T16:41:39.293349+0000","last_clean":"2023-10-14T16:41:39.293349+0000","last_became_active":"2023-10-14T16:41:38.269101+0000","last_became_peered":"2023-10-14T16:41:38.269101+0000","last_unstale":"2023-10-14T16:41:39.293349+0000","last_undegraded":"2023-10-14T16:41:39.293349+0000","last_fullsized":"2023-10-14T16:41:39.293349+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:25.343787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750734+0000","last_change":"2023-10-14T16:41:39.280373+0000","last_active":"2023-10-14T16:41:40.750734+0000","last_peered":"2023-10-14T16:41:40.750734+0000","last_clean":"2023-10-14T16:41:40.750734+0000","last_became_active":"2023-10-14T16:41:39.280069+0000","last_became_peered":"2023-10-14T16:41:39.280069+0000","last_unstale":"2023-10-14T16:41:40.750734+0000","last_undegraded":"2023-10-14T16:41:40.750734+0000","last_fullsized":"2023-10-14T16:41:40.750734+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:18:06.615265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"33'6","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.301804+0000","last_change":"2023-10-14T16:41:38.268083+0000","last_active":"2023-10-14T16:41:39.301804+0000","last_peered":"2023-10-14T16:41:39.301804+0000","last_clean":"2023-10-14T16:41:39.301804+0000","last_became_active":"2023-10-14T16:41:38.267791+0000","last_became_peered":"2023-10-14T16:41:38.267791+0000","last_unstale":"2023-10-14T16:41:39.301804+0000","last_undegraded":"2023-10-14T16:41:39.301804+0000","last_fullsized":"2023-10-14T16:41:39.301804+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.748547+0000","last_change":"2023-10-14T16:41:39.277743+0000","last_active":"2023-10-14T16:41:40.748547+0000","last_peered":"2023-10-14T16:41:40.748547+0000","last_clean":"2023-10-14T16:41:40.748547+0000","last_became_active":"2023-10-14T16:41:39.277482+0000","last_became_peered":"2023-10-14T16:41:39.277482+0000","last_unstale":"2023-10-14T16:41:40.748547+0000","last_undegraded":"2023-10-14T16:41:40.748547+0000","last_fullsized":"2023-10-14T16:41:40.748547+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:34:08.822592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294903+0000","last_change":"2023-10-14T16:41:38.259070+0000","last_active":"2023-10-14T16:41:39.294903+0000","last_peered":"2023-10-14T16:41:39.294903+0000","last_clean":"2023-10-14T16:41:39.294903+0000","last_became_active":"2023-10-14T16:41:38.258658+0000","last_became_peered":"2023-10-14T16:41:38.258658+0000","last_unstale":"2023-10-14T16:41:39.294903+0000","last_undegraded":"2023-10-14T16:41:39.294903+0000","last_fullsized":"2023-10-14T16:41:39.294903+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.273225+0000","last_change":"2023-10-14T16:41:39.275431+0000","last_active":"2023-10-14T16:41:40.273225+0000","last_peered":"2023-10-14T16:41:40.273225+0000","last_clean":"2023-10-14T16:41:40.273225+0000","last_became_active":"2023-10-14T16:41:39.275138+0000","last_became_peered":"2023-10-14T16:41:39.275138+0000","last_unstale":"2023-10-14T16:41:40.273225+0000","last_undegraded":"2023-10-14T16:41:40.273225+0000","last_fullsized":"2023-10-14T16:41:40.273225+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:38.477958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.264902+0000","last_change":"2023-10-14T16:41:38.265796+0000","last_active":"2023-10-14T16:41:39.264902+0000","last_peered":"2023-10-14T16:41:39.264902+0000","last_clean":"2023-10-14T16:41:39.264902+0000","last_became_active":"2023-10-14T16:41:38.265491+0000","last_became_peered":"2023-10-14T16:41:38.265491+0000","last_unstale":"2023-10-14T16:41:39.264902+0000","last_undegraded":"2023-10-14T16:41:39.264902+0000","last_fullsized":"2023-10-14T16:41:39.264902+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:27.930694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275533+0000","last_change":"2023-10-14T16:41:39.283974+0000","last_active":"2023-10-14T16:41:40.275533+0000","last_peered":"2023-10-14T16:41:40.275533+0000","last_clean":"2023-10-14T16:41:40.275533+0000","last_became_active":"2023-10-14T16:41:39.283465+0000","last_became_peered":"2023-10-14T16:41:39.283465+0000","last_unstale":"2023-10-14T16:41:40.275533+0000","last_undegraded":"2023-10-14T16:41:40.275533+0000","last_fullsized":"2023-10-14T16:41:40.275533+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:11:43.985668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.260953+0000","last_change":"2023-10-14T16:41:38.271323+0000","last_active":"2023-10-14T16:41:39.260953+0000","last_peered":"2023-10-14T16:41:39.260953+0000","last_clean":"2023-10-14T16:41:39.260953+0000","last_became_active":"2023-10-14T16:41:38.271014+0000","last_became_peered":"2023-10-14T16:41:38.271014+0000","last_unstale":"2023-10-14T16:41:39.260953+0000","last_undegraded":"2023-10-14T16:41:39.260953+0000","last_fullsized":"2023-10-14T16:41:39.260953+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:39:56.455065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275149+0000","last_change":"2023-10-14T16:41:39.272441+0000","last_active":"2023-10-14T16:41:40.275149+0000","last_peered":"2023-10-14T16:41:40.275149+0000","last_clean":"2023-10-14T16:41:40.275149+0000","last_became_active":"2023-10-14T16:41:39.271666+0000","last_became_peered":"2023-10-14T16:41:39.271666+0000","last_unstale":"2023-10-14T16:41:40.275149+0000","last_undegraded":"2023-10-14T16:41:40.275149+0000","last_fullsized":"2023-10-14T16:41:40.275149+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"33'8","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.299798+0000","last_change":"2023-10-14T16:41:38.272215+0000","last_active":"2023-10-14T16:41:39.299798+0000","last_peered":"2023-10-14T16:41:39.299798+0000","last_clean":"2023-10-14T16:41:39.299798+0000","last_became_active":"2023-10-14T16:41:38.271943+0000","last_became_peered":"2023-10-14T16:41:38.271943+0000","last_unstale":"2023-10-14T16:41:39.299798+0000","last_undegraded":"2023-10-14T16:41:39.299798+0000","last_fullsized":"2023-10-14T16:41:39.299798+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:27.930694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751505+0000","last_change":"2023-10-14T16:41:39.281314+0000","last_active":"2023-10-14T16:41:40.751505+0000","last_peered":"2023-10-14T16:41:40.751505+0000","last_clean":"2023-10-14T16:41:40.751505+0000","last_became_active":"2023-10-14T16:41:39.279602+0000","last_became_peered":"2023-10-14T16:41:39.279602+0000","last_unstale":"2023-10-14T16:41:40.751505+0000","last_undegraded":"2023-10-14T16:41:40.751505+0000","last_fullsized":"2023-10-14T16:41:40.751505+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:29.535419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.296101+0000","last_change":"2023-10-14T16:41:38.267446+0000","last_active":"2023-10-14T16:41:39.296101+0000","last_peered":"2023-10-14T16:41:39.296101+0000","last_clean":"2023-10-14T16:41:39.296101+0000","last_became_active":"2023-10-14T16:41:38.267122+0000","last_became_peered":"2023-10-14T16:41:38.267122+0000","last_unstale":"2023-10-14T16:41:39.296101+0000","last_undegraded":"2023-10-14T16:41:39.296101+0000","last_fullsized":"2023-10-14T16:41:39.296101+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:28.520579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751980+0000","last_change":"2023-10-14T16:41:39.275277+0000","last_active":"2023-10-14T16:41:40.751980+0000","last_peered":"2023-10-14T16:41:40.751980+0000","last_clean":"2023-10-14T16:41:40.751980+0000","last_became_active":"2023-10-14T16:41:39.273250+0000","last_became_peered":"2023-10-14T16:41:39.273250+0000","last_unstale":"2023-10-14T16:41:40.751980+0000","last_undegraded":"2023-10-14T16:41:40.751980+0000","last_fullsized":"2023-10-14T16:41:40.751980+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:33:19.533902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.295047+0000","last_change":"2023-10-14T16:41:38.266145+0000","last_active":"2023-10-14T16:41:39.295047+0000","last_peered":"2023-10-14T16:41:39.295047+0000","last_clean":"2023-10-14T16:41:39.295047+0000","last_became_active":"2023-10-14T16:41:38.264561+0000","last_became_peered":"2023-10-14T16:41:38.264561+0000","last_unstale":"2023-10-14T16:41:39.295047+0000","last_undegraded":"2023-10-14T16:41:39.295047+0000","last_fullsized":"2023-10-14T16:41:39.295047+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:28.458858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.749516+0000","last_change":"2023-10-14T16:41:39.273389+0000","last_active":"2023-10-14T16:41:40.749516+0000","last_peered":"2023-10-14T16:41:40.749516+0000","last_clean":"2023-10-14T16:41:40.749516+0000","last_became_active":"2023-10-14T16:41:39.273083+0000","last_became_peered":"2023-10-14T16:41:39.273083+0000","last_unstale":"2023-10-14T16:41:40.749516+0000","last_undegraded":"2023-10-14T16:41:40.749516+0000","last_fullsized":"2023-10-14T16:41:40.749516+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:26.358627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"33'1","reported_seq":13,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.294514+0000","last_change":"2023-10-14T16:41:38.266046+0000","last_active":"2023-10-14T16:41:39.294514+0000","last_peered":"2023-10-14T16:41:39.294514+0000","last_clean":"2023-10-14T16:41:39.294514+0000","last_became_active":"2023-10-14T16:41:38.263725+0000","last_became_peered":"2023-10-14T16:41:38.263725+0000","last_unstale":"2023-10-14T16:41:39.294514+0000","last_undegraded":"2023-10-14T16:41:39.294514+0000","last_fullsized":"2023-10-14T16:41:39.294514+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:17.300356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751656+0000","last_change":"2023-10-14T16:41:39.278224+0000","last_active":"2023-10-14T16:41:40.751656+0000","last_peered":"2023-10-14T16:41:40.751656+0000","last_clean":"2023-10-14T16:41:40.751656+0000","last_became_active":"2023-10-14T16:41:39.277986+0000","last_became_peered":"2023-10-14T16:41:39.277986+0000","last_unstale":"2023-10-14T16:41:40.751656+0000","last_undegraded":"2023-10-14T16:41:40.751656+0000","last_fullsized":"2023-10-14T16:41:40.751656+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:49.911818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.254835+0000","last_change":"2023-10-14T16:41:38.263662+0000","last_active":"2023-10-14T16:41:39.254835+0000","last_peered":"2023-10-14T16:41:39.254835+0000","last_clean":"2023-10-14T16:41:39.254835+0000","last_became_active":"2023-10-14T16:41:38.262441+0000","last_became_peered":"2023-10-14T16:41:38.262441+0000","last_unstale":"2023-10-14T16:41:39.254835+0000","last_undegraded":"2023-10-14T16:41:39.254835+0000","last_fullsized":"2023-10-14T16:41:39.254835+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:11:48.896977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750772+0000","last_change":"2023-10-14T16:41:39.266628+0000","last_active":"2023-10-14T16:41:40.750772+0000","last_peered":"2023-10-14T16:41:40.750772+0000","last_clean":"2023-10-14T16:41:40.750772+0000","last_became_active":"2023-10-14T16:41:39.266023+0000","last_became_peered":"2023-10-14T16:41:39.266023+0000","last_unstale":"2023-10-14T16:41:40.750772+0000","last_undegraded":"2023-10-14T16:41:40.750772+0000","last_fullsized":"2023-10-14T16:41:40.750772+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:58.541591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.291472+0000","last_change":"2023-10-14T16:41:38.270622+0000","last_active":"2023-10-14T16:41:39.291472+0000","last_peered":"2023-10-14T16:41:39.291472+0000","last_clean":"2023-10-14T16:41:39.291472+0000","last_became_active":"2023-10-14T16:41:38.269459+0000","last_became_peered":"2023-10-14T16:41:38.269459+0000","last_unstale":"2023-10-14T16:41:39.291472+0000","last_undegraded":"2023-10-14T16:41:39.291472+0000","last_fullsized":"2023-10-14T16:41:39.291472+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751161+0000","last_change":"2023-10-14T16:41:39.273339+0000","last_active":"2023-10-14T16:41:40.751161+0000","last_peered":"2023-10-14T16:41:40.751161+0000","last_clean":"2023-10-14T16:41:40.751161+0000","last_became_active":"2023-10-14T16:41:39.273045+0000","last_became_peered":"2023-10-14T16:41:39.273045+0000","last_unstale":"2023-10-14T16:41:40.751161+0000","last_undegraded":"2023-10-14T16:41:40.751161+0000","last_fullsized":"2023-10-14T16:41:40.751161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:18.315197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.249521+0000","last_change":"2023-10-14T16:41:38.270414+0000","last_active":"2023-10-14T16:41:39.249521+0000","last_peered":"2023-10-14T16:41:39.249521+0000","last_clean":"2023-10-14T16:41:39.249521+0000","last_became_active":"2023-10-14T16:41:38.269445+0000","last_became_peered":"2023-10-14T16:41:38.269445+0000","last_unstale":"2023-10-14T16:41:39.249521+0000","last_undegraded":"2023-10-14T16:41:39.249521+0000","last_fullsized":"2023-10-14T16:41:39.249521+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:06.981277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750513+0000","last_change":"2023-10-14T16:41:39.273636+0000","last_active":"2023-10-14T16:41:40.750513+0000","last_peered":"2023-10-14T16:41:40.750513+0000","last_clean":"2023-10-14T16:41:40.750513+0000","last_became_active":"2023-10-14T16:41:39.272522+0000","last_became_peered":"2023-10-14T16:41:39.272522+0000","last_unstale":"2023-10-14T16:41:40.750513+0000","last_undegraded":"2023-10-14T16:41:40.750513+0000","last_fullsized":"2023-10-14T16:41:40.750513+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:52.886679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255580+0000","last_change":"2023-10-14T16:41:38.271734+0000","last_active":"2023-10-14T16:41:39.255580+0000","last_peered":"2023-10-14T16:41:39.255580+0000","last_clean":"2023-10-14T16:41:39.255580+0000","last_became_active":"2023-10-14T16:41:38.271386+0000","last_became_peered":"2023-10-14T16:41:38.271386+0000","last_unstale":"2023-10-14T16:41:39.255580+0000","last_undegraded":"2023-10-14T16:41:39.255580+0000","last_fullsized":"2023-10-14T16:41:39.255580+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.275968+0000","last_change":"2023-10-14T16:41:39.277752+0000","last_active":"2023-10-14T16:41:40.275968+0000","last_peered":"2023-10-14T16:41:40.275968+0000","last_clean":"2023-10-14T16:41:40.275968+0000","last_became_active":"2023-10-14T16:41:39.276648+0000","last_became_peered":"2023-10-14T16:41:39.276648+0000","last_unstale":"2023-10-14T16:41:40.275968+0000","last_undegraded":"2023-10-14T16:41:40.275968+0000","last_fullsized":"2023-10-14T16:41:40.275968+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.259544+0000","last_change":"2023-10-14T16:41:38.266162+0000","last_active":"2023-10-14T16:41:39.259544+0000","last_peered":"2023-10-14T16:41:39.259544+0000","last_clean":"2023-10-14T16:41:39.259544+0000","last_became_active":"2023-10-14T16:41:38.265806+0000","last_became_peered":"2023-10-14T16:41:38.265806+0000","last_unstale":"2023-10-14T16:41:39.259544+0000","last_undegraded":"2023-10-14T16:41:39.259544+0000","last_fullsized":"2023-10-14T16:41:39.259544+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751528+0000","last_change":"2023-10-14T16:41:39.276093+0000","last_active":"2023-10-14T16:41:40.751528+0000","last_peered":"2023-10-14T16:41:40.751528+0000","last_clean":"2023-10-14T16:41:40.751528+0000","last_became_active":"2023-10-14T16:41:39.275759+0000","last_became_peered":"2023-10-14T16:41:39.275759+0000","last_unstale":"2023-10-14T16:41:40.751528+0000","last_undegraded":"2023-10-14T16:41:40.751528+0000","last_fullsized":"2023-10-14T16:41:40.751528+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:26.451032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"33'2","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.293345+0000","last_change":"2023-10-14T16:41:38.271489+0000","last_active":"2023-10-14T16:41:39.293345+0000","last_peered":"2023-10-14T16:41:39.293345+0000","last_clean":"2023-10-14T16:41:39.293345+0000","last_became_active":"2023-10-14T16:41:38.271096+0000","last_became_peered":"2023-10-14T16:41:38.271096+0000","last_unstale":"2023-10-14T16:41:39.293345+0000","last_undegraded":"2023-10-14T16:41:39.293345+0000","last_fullsized":"2023-10-14T16:41:39.293345+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:58:21.801368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.751953+0000","last_change":"2023-10-14T16:41:39.278834+0000","last_active":"2023-10-14T16:41:40.751953+0000","last_peered":"2023-10-14T16:41:40.751953+0000","last_clean":"2023-10-14T16:41:40.751953+0000","last_became_active":"2023-10-14T16:41:39.278474+0000","last_became_peered":"2023-10-14T16:41:39.278474+0000","last_unstale":"2023-10-14T16:41:40.751953+0000","last_undegraded":"2023-10-14T16:41:40.751953+0000","last_fullsized":"2023-10-14T16:41:40.751953+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255282+0000","last_change":"2023-10-14T16:41:38.267492+0000","last_active":"2023-10-14T16:41:39.255282+0000","last_peered":"2023-10-14T16:41:39.255282+0000","last_clean":"2023-10-14T16:41:39.255282+0000","last_became_active":"2023-10-14T16:41:38.267208+0000","last_became_peered":"2023-10-14T16:41:38.267208+0000","last_unstale":"2023-10-14T16:41:39.255282+0000","last_undegraded":"2023-10-14T16:41:39.255282+0000","last_fullsized":"2023-10-14T16:41:39.255282+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:27.184684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.406979+0000","last_change":"2023-10-14T16:41:39.270875+0000","last_active":"2023-10-14T16:41:40.406979+0000","last_peered":"2023-10-14T16:41:40.406979+0000","last_clean":"2023-10-14T16:41:40.406979+0000","last_became_active":"2023-10-14T16:41:39.270562+0000","last_became_peered":"2023-10-14T16:41:39.270562+0000","last_unstale":"2023-10-14T16:41:40.406979+0000","last_undegraded":"2023-10-14T16:41:40.406979+0000","last_fullsized":"2023-10-14T16:41:40.406979+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.265266+0000","last_change":"2023-10-14T16:41:38.269649+0000","last_active":"2023-10-14T16:41:39.265266+0000","last_peered":"2023-10-14T16:41:39.265266+0000","last_clean":"2023-10-14T16:41:39.265266+0000","last_became_active":"2023-10-14T16:41:38.269092+0000","last_became_peered":"2023-10-14T16:41:38.269092+0000","last_unstale":"2023-10-14T16:41:39.265266+0000","last_undegraded":"2023-10-14T16:41:39.265266+0000","last_fullsized":"2023-10-14T16:41:39.265266+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:55.953985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.274712+0000","last_change":"2023-10-14T16:41:39.274584+0000","last_active":"2023-10-14T16:41:40.274712+0000","last_peered":"2023-10-14T16:41:40.274712+0000","last_clean":"2023-10-14T16:41:40.274712+0000","last_became_active":"2023-10-14T16:41:39.274261+0000","last_became_peered":"2023-10-14T16:41:39.274261+0000","last_unstale":"2023-10-14T16:41:40.274712+0000","last_undegraded":"2023-10-14T16:41:40.274712+0000","last_fullsized":"2023-10-14T16:41:40.274712+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:28.199524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.254884+0000","last_change":"2023-10-14T16:41:38.260866+0000","last_active":"2023-10-14T16:41:39.254884+0000","last_peered":"2023-10-14T16:41:39.254884+0000","last_clean":"2023-10-14T16:41:39.254884+0000","last_became_active":"2023-10-14T16:41:38.260498+0000","last_became_peered":"2023-10-14T16:41:38.260498+0000","last_unstale":"2023-10-14T16:41:39.254884+0000","last_undegraded":"2023-10-14T16:41:39.254884+0000","last_fullsized":"2023-10-14T16:41:39.254884+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:17.441986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.407161+0000","last_change":"2023-10-14T16:41:39.269959+0000","last_active":"2023-10-14T16:41:40.407161+0000","last_peered":"2023-10-14T16:41:40.407161+0000","last_clean":"2023-10-14T16:41:40.407161+0000","last_became_active":"2023-10-14T16:41:39.267787+0000","last_became_peered":"2023-10-14T16:41:39.267787+0000","last_unstale":"2023-10-14T16:41:40.407161+0000","last_undegraded":"2023-10-14T16:41:40.407161+0000","last_fullsized":"2023-10-14T16:41:40.407161+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:28.945535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.265079+0000","last_change":"2023-10-14T16:41:38.265778+0000","last_active":"2023-10-14T16:41:39.265079+0000","last_peered":"2023-10-14T16:41:39.265079+0000","last_clean":"2023-10-14T16:41:39.265079+0000","last_became_active":"2023-10-14T16:41:38.265477+0000","last_became_peered":"2023-10-14T16:41:38.265477+0000","last_unstale":"2023-10-14T16:41:39.265079+0000","last_undegraded":"2023-10-14T16:41:39.265079+0000","last_fullsized":"2023-10-14T16:41:39.265079+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:14.919317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.271841+0000","last_change":"2023-10-14T16:41:39.282237+0000","last_active":"2023-10-14T16:41:40.271841+0000","last_peered":"2023-10-14T16:41:40.271841+0000","last_clean":"2023-10-14T16:41:40.271841+0000","last_became_active":"2023-10-14T16:41:39.281738+0000","last_became_peered":"2023-10-14T16:41:39.281738+0000","last_unstale":"2023-10-14T16:41:40.271841+0000","last_undegraded":"2023-10-14T16:41:40.271841+0000","last_fullsized":"2023-10-14T16:41:40.271841+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:07:09.306511+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.255649+0000","last_change":"2023-10-14T16:41:38.265852+0000","last_active":"2023-10-14T16:41:39.255649+0000","last_peered":"2023-10-14T16:41:39.255649+0000","last_clean":"2023-10-14T16:41:39.255649+0000","last_became_active":"2023-10-14T16:41:38.265513+0000","last_became_peered":"2023-10-14T16:41:38.265513+0000","last_unstale":"2023-10-14T16:41:39.255649+0000","last_undegraded":"2023-10-14T16:41:39.255649+0000","last_fullsized":"2023-10-14T16:41:39.255649+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:28.955674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.750355+0000","last_change":"2023-10-14T16:41:39.276674+0000","last_active":"2023-10-14T16:41:40.750355+0000","last_peered":"2023-10-14T16:41:40.750355+0000","last_clean":"2023-10-14T16:41:40.750355+0000","last_became_active":"2023-10-14T16:41:39.274101+0000","last_became_peered":"2023-10-14T16:41:39.274101+0000","last_unstale":"2023-10-14T16:41:40.750355+0000","last_undegraded":"2023-10-14T16:41:40.750355+0000","last_fullsized":"2023-10-14T16:41:40.750355+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:05.554009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.258177+0000","last_change":"2023-10-14T16:41:38.265704+0000","last_active":"2023-10-14T16:41:39.258177+0000","last_peered":"2023-10-14T16:41:39.258177+0000","last_clean":"2023-10-14T16:41:39.258177+0000","last_became_active":"2023-10-14T16:41:38.264927+0000","last_became_peered":"2023-10-14T16:41:38.264927+0000","last_unstale":"2023-10-14T16:41:39.258177+0000","last_undegraded":"2023-10-14T16:41:39.258177+0000","last_fullsized":"2023-10-14T16:41:39.258177+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:14.919317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":11,"reported_epoch":34,"state":"active+clean","last_fresh":"2023-10-14T16:41:40.276871+0000","last_change":"2023-10-14T16:41:39.280368+0000","last_active":"2023-10-14T16:41:40.276871+0000","last_peered":"2023-10-14T16:41:40.276871+0000","last_clean":"2023-10-14T16:41:40.276871+0000","last_became_active":"2023-10-14T16:41:39.280081+0000","last_became_peered":"2023-10-14T16:41:39.280081+0000","last_unstale":"2023-10-14T16:41:40.276871+0000","last_undegraded":"2023-10-14T16:41:40.276871+0000","last_fullsized":"2023-10-14T16:41:40.276871+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":32,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:38.219749+0000","last_clean_scrub_stamp":"2023-10-14T16:41:38.219749+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:50:54.194726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.249909+0000","last_change":"2023-10-14T16:41:38.267370+0000","last_active":"2023-10-14T16:41:39.249909+0000","last_peered":"2023-10-14T16:41:39.249909+0000","last_clean":"2023-10-14T16:41:39.249909+0000","last_became_active":"2023-10-14T16:41:38.267087+0000","last_became_peered":"2023-10-14T16:41:38.267087+0000","last_unstale":"2023-10-14T16:41:39.249909+0000","last_undegraded":"2023-10-14T16:41:39.249909+0000","last_fullsized":"2023-10-14T16:41:39.249909+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:41:37.204909+0000","last_clean_scrub_stamp":"2023-10-14T16:41:37.204909+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:26.172133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":175,"reported_epoch":33,"state":"active+clean","last_fresh":"2023-10-14T16:41:39.268835+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:41:39.268835+0000","last_peered":"2023-10-14T16:41:39.268835+0000","last_clean":"2023-10-14T16:41:39.268835+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:41:39.268835+0000","last_undegraded":"2023-10-14T16:41:39.268835+0000","last_fullsized":"2023-10-14T16:41:39.268835+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27988,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343852,"statfs":{"total":96636764160,"available":96608104448,"internally_reserved":0,"allocated":983040,"data_stored":742701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.042,"5min":1.042,"15min":1.042},"min":{"1min":0.503,"5min":0.503,"15min":0.503},"max":{"1min":1.7130000000000001,"5min":1.7130000000000001,"15min":1.7130000000000001},"last":0.90100000000000002},{"interface":"front","average":{"1min":1.173,"5min":1.173,"15min":1.173},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.206,"5min":1.206,"15min":1.206},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":1.9359999999999999},"last":1.173},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0580000000000001,"15min":1.0580000000000001},"min":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"max":{"1min":1.6739999999999999,"5min":1.6739999999999999,"15min":1.6739999999999999},"last":1.321}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2130000000000001,"15min":1.2130000000000001},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0070000000000001,"5min":2.0070000000000001,"15min":2.0070000000000001},"last":1.054},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.2969999999999999,"15min":1.2969999999999999},"min":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"max":{"1min":2.3500000000000001,"5min":2.3500000000000001,"15min":2.3500000000000001},"last":1.2769999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.196,"5min":1.196,"15min":1.196},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.671,"5min":1.671,"15min":1.671},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.423,"15min":1.423},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":2.5819999999999999,"5min":2.5819999999999999,"15min":2.5819999999999999},"last":1.4139999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3,"15min":1.3},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"max":{"1min":2.2959999999999998,"5min":2.2959999999999998,"15min":2.2959999999999998},"last":1.486},{"interface":"front","average":{"1min":1.262,"5min":1.262,"15min":1.262},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.706,"5min":1.706,"15min":1.706},"last":1.7090000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.474,"15min":1.474},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":3.5270000000000001,"5min":3.5270000000000001,"15min":3.5270000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"min":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"max":{"1min":2.0899999999999999,"5min":2.0899999999999999,"15min":2.0899999999999999},"last":1.651}]},{"osd":6,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4830000000000001,"5min":1.4830000000000001,"15min":1.4830000000000001},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.1579999999999999,"5min":2.1579999999999999,"15min":2.1579999999999999},"last":1.8460000000000001},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.7040000000000002,"5min":2.7040000000000002,"15min":2.7040000000000002},"last":1.8080000000000001}]}]},{"osd":6,"up_from":13,"seq":55834574868,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"max":{"1min":2.339,"5min":2.339,"15min":2.339},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.2509999999999999,"5min":3.2509999999999999,"15min":3.2509999999999999},"last":1.4419999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.3360000000000001,"15min":1.3360000000000001},"min":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"max":{"1min":4.3280000000000003,"5min":4.3280000000000003,"15min":4.3280000000000003},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.98099999999999998,"5min":0.98099999999999998,"15min":0.98099999999999998},"max":{"1min":3.431,"5min":3.431,"15min":3.431},"last":1.1970000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.3100000000000001,"15min":1.3100000000000001},"min":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"max":{"1min":2.6539999999999999,"5min":2.6539999999999999,"15min":2.6539999999999999},"last":2.133},{"interface":"front","average":{"1min":1.478,"5min":1.478,"15min":1.478},"min":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.7200000000000002,"5min":2.7200000000000002,"15min":2.7200000000000002},"last":1.669}]},{"osd":3,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4650000000000001,"5min":1.4650000000000001,"15min":1.4650000000000001},"min":{"1min":0.90100000000000002,"5min":0.90100000000000002,"15min":0.90100000000000002},"max":{"1min":2.8319999999999999,"5min":2.8319999999999999,"15min":2.8319999999999999},"last":2.085},{"interface":"front","average":{"1min":1.446,"5min":1.446,"15min":1.446},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":3.6059999999999999,"5min":3.6059999999999999,"15min":3.6059999999999999},"last":2.3199999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.85499999999999998},"max":{"1min":3.0790000000000002,"5min":3.0790000000000002,"15min":3.0790000000000002},"last":1.335},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"max":{"1min":2.738,"5min":2.738,"15min":2.738},"last":1.9339999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":3.0110000000000001,"5min":3.0110000000000001,"15min":3.0110000000000001},"last":2.1800000000000002},{"interface":"front","average":{"1min":1.7250000000000001,"5min":1.7250000000000001,"15min":1.7250000000000001},"min":{"1min":1.27,"5min":1.27,"15min":1.27},"max":{"1min":3.206,"5min":3.206,"15min":3.206},"last":1.8180000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:17 2023","interfaces":[{"interface":"back","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":1.1759999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"max":{"1min":2.6349999999999998,"5min":2.6349999999999998,"15min":2.6349999999999998},"last":2.2290000000000001},{"interface":"front","average":{"1min":1.681,"5min":1.681,"15min":1.681},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":3.1320000000000001,"5min":3.1320000000000001,"15min":3.1320000000000001},"last":2.0009999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574869,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152915,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.202,"15min":1.202},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":3.2000000000000002,"5min":3.2000000000000002,"15min":3.2000000000000002},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.2569999999999999,"5min":1.2569999999999999,"15min":1.2569999999999999},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"max":{"1min":4.101,"5min":4.101,"15min":4.101},"last":0.75}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4019999999999999,"15min":1.4019999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.1659999999999999,"5min":3.1659999999999999,"15min":3.1659999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.2549999999999999},"last":1.028}]},{"osd":2,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.4490000000000001,"15min":1.4490000000000001},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":4.3639999999999999,"5min":4.3639999999999999,"15min":4.3639999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":1.52,"5min":1.52,"15min":1.52},"min":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"max":{"1min":4.3339999999999996,"5min":4.3339999999999996,"15min":4.3339999999999996},"last":1.075}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.397,"15min":1.397},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":2.431,"5min":2.431,"15min":2.431},"last":1.1499999999999999},{"interface":"front","average":{"1min":1.556,"5min":1.556,"15min":1.556},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.90800000000000003},"max":{"1min":3.6890000000000001,"5min":3.6890000000000001,"15min":3.6890000000000001},"last":1.2030000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.419,"15min":1.419},"min":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"max":{"1min":2.577,"5min":2.577,"15min":2.577},"last":1.3049999999999999},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.6030000000000002,"5min":2.6030000000000002,"15min":2.6030000000000002},"last":1.2629999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.655,"5min":1.655,"15min":1.655},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":3.5739999999999998,"5min":3.5739999999999998,"15min":3.5739999999999998},"last":1.4590000000000001},{"interface":"front","average":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.7629999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":3.2949999999999999,"5min":3.2949999999999999,"15min":3.2949999999999999},"last":1.8169999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.6579999999999999,"5min":1.6579999999999999,"15min":1.6579999999999999},"min":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"max":{"1min":3.0289999999999999,"5min":3.0289999999999999,"15min":3.0289999999999999},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.673,"5min":1.673,"15min":1.673},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":3.7869999999999999,"5min":3.7869999999999999,"15min":3.7869999999999999},"last":1.863}]}]},{"osd":4,"up_from":13,"seq":55834574869,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27408,"kb_used_data":384,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344432,"statfs":{"total":96636764160,"available":96608698368,"internally_reserved":0,"allocated":393216,"data_stored":152347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.264,"5min":1.264,"15min":1.264},"min":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"max":{"1min":4.3810000000000002,"5min":4.3810000000000002,"15min":4.3810000000000002},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.365,"5min":1.365,"15min":1.365},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":4.4059999999999997,"5min":4.4059999999999997,"15min":4.4059999999999997},"last":0.96399999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.258,"5min":1.258,"15min":1.258},"min":{"1min":0.60699999999999998,"5min":0.60699999999999998,"15min":0.60699999999999998},"max":{"1min":1.8400000000000001,"5min":1.8400000000000001,"15min":1.8400000000000001},"last":0.747},{"interface":"front","average":{"1min":1.121,"5min":1.121,"15min":1.121},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.5129999999999999,"5min":1.5129999999999999,"15min":1.5129999999999999},"last":1.127}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.266,"15min":1.266},"min":{"1min":0.93999999999999995,"5min":0.93999999999999995,"15min":0.93999999999999995},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":1.294},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":2.0579999999999998,"5min":2.0579999999999998,"15min":2.0579999999999998},"last":1.2070000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2370000000000001,"5min":1.2370000000000001,"15min":1.2370000000000001},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.026},{"interface":"front","average":{"1min":1.401,"5min":1.401,"15min":1.401},"min":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":1.4059999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.98899999999999999,"5min":0.98899999999999999,"15min":0.98899999999999999},"max":{"1min":2.2320000000000002,"5min":2.2320000000000002,"15min":2.2320000000000002},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.4339999999999999,"15min":1.4339999999999999},"min":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.0620000000000001},"max":{"1min":2.109,"5min":2.109,"15min":2.109},"last":1.516}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5600000000000001,"5min":1.5600000000000001,"15min":1.5600000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.98999999999999999},"max":{"1min":2.0739999999999998,"5min":2.0739999999999998,"15min":2.0739999999999998},"last":1.9039999999999999},{"interface":"front","average":{"1min":1.639,"5min":1.639,"15min":1.639},"min":{"1min":1.1919999999999999,"5min":1.1919999999999999,"15min":1.1919999999999999},"max":{"1min":2.1600000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.5880000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5940000000000001,"5min":1.5940000000000001,"15min":1.5940000000000001},"min":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"max":{"1min":2.2069999999999999,"5min":2.2069999999999999,"15min":2.2069999999999999},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.7070000000000001,"5min":1.7070000000000001,"15min":1.7070000000000001},"min":{"1min":1.29,"5min":1.29,"15min":1.29},"max":{"1min":2.3999999999999999,"5min":2.3999999999999999,"15min":2.3999999999999999},"last":1.8069999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574869,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":153877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":1.27,"5min":1.27,"15min":1.27},"last":1.1100000000000001},{"interface":"front","average":{"1min":1.0129999999999999,"5min":1.0129999999999999,"15min":1.0129999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":1.6120000000000001,"5min":1.6120000000000001,"15min":1.6120000000000001},"last":1.1479999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"min":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"max":{"1min":1.7889999999999999,"5min":1.7889999999999999,"15min":1.7889999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":2.294,"5min":2.294,"15min":2.294},"last":0.98599999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.198,"5min":1.198,"15min":1.198},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":2.6819999999999999,"5min":2.6819999999999999,"15min":2.6819999999999999},"last":1.613},{"interface":"front","average":{"1min":1.131,"5min":1.131,"15min":1.131},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":2.77,"5min":2.77,"15min":2.77},"last":2.282}]},{"osd":4,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.4259999999999999,"15min":1.4259999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.8239999999999998,"5min":2.8239999999999998,"15min":2.8239999999999998},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5289999999999999,"15min":1.5289999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":3.9540000000000002,"5min":3.9540000000000002,"15min":3.9540000000000002},"last":1.278}]},{"osd":5,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.444,"5min":1.444,"15min":1.444},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":2.8900000000000001,"5min":2.8900000000000001,"15min":2.8900000000000001},"last":1.8700000000000001},{"interface":"front","average":{"1min":1.518,"5min":1.518,"15min":1.518},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.2490000000000001,"5min":3.2490000000000001,"15min":3.2490000000000001},"last":1.3280000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.524,"5min":1.524,"15min":1.524},"min":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":0.65100000000000002},"max":{"1min":4.0570000000000004,"5min":4.0570000000000004,"15min":4.0570000000000004},"last":1.429},{"interface":"front","average":{"1min":1.6859999999999999,"5min":1.6859999999999999,"15min":1.6859999999999999},"min":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"max":{"1min":4.1900000000000004,"5min":4.1900000000000004,"15min":4.1900000000000004},"last":1.7}]},{"osd":7,"last update":"Sat Oct 14 16:41:13 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.454,"15min":1.454},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":2.9900000000000002,"5min":2.9900000000000002,"15min":2.9900000000000002},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.5620000000000001,"15min":1.5620000000000001},"min":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.9569999999999999,"5min":2.9569999999999999,"15min":2.9569999999999999},"last":1.7789999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574869,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31568,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340272,"statfs":{"total":96636764160,"available":96604438528,"internally_reserved":0,"allocated":389120,"data_stored":152313,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"last":1.004},{"interface":"front","average":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98599999999999999},"min":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"max":{"1min":3.2149999999999999,"5min":3.2149999999999999,"15min":3.2149999999999999},"last":1.194}]},{"osd":1,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.286,"5min":1.286,"15min":1.286},"last":0.84399999999999997},{"interface":"front","average":{"1min":1.113,"5min":1.113,"15min":1.113},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.78800000000000003}]},{"osd":3,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.097,"5min":1.097,"15min":1.097},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.175,"5min":1.175,"15min":1.175},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.6970000000000001,"5min":1.6970000000000001,"15min":1.6970000000000001},"last":1.2410000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3160000000000001,"15min":1.3160000000000001},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"max":{"1min":2.6259999999999999,"5min":2.6259999999999999,"15min":2.6259999999999999},"last":1.3600000000000001},{"interface":"front","average":{"1min":1.331,"5min":1.331,"15min":1.331},"min":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"max":{"1min":3.0459999999999998,"5min":3.0459999999999998,"15min":3.0459999999999998},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.361,"15min":1.361},"min":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4319999999999999,"15min":1.4319999999999999},"min":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"max":{"1min":2.883,"5min":2.883,"15min":2.883},"last":1.4650000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.83499999999999996},"max":{"1min":3.1200000000000001,"5min":3.1200000000000001,"15min":3.1200000000000001},"last":1.3320000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5109999999999999},"min":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":2.3420000000000001},"last":1.4950000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:14 2023","interfaces":[{"interface":"back","average":{"1min":1.5149999999999999,"5min":1.5149999999999999,"15min":1.5149999999999999},"min":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":3.169,"5min":3.169,"15min":3.169},"last":1.278},{"interface":"front","average":{"1min":1.6000000000000001,"5min":1.6000000000000001,"15min":1.6000000000000001},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.996,"5min":2.996,"15min":2.996},"last":1.3029999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574869,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31580,"kb_used_data":392,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340260,"statfs":{"total":96636764160,"available":96604426240,"internally_reserved":0,"allocated":401408,"data_stored":153877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.82399999999999995},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3080000000000001,"5min":1.3080000000000001,"15min":1.3080000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.92900000000000005,"5min":0.92900000000000005,"15min":0.92900000000000005},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"max":{"1min":1.669,"5min":1.669,"15min":1.669},"last":0.54800000000000004}]},{"osd":2,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.46,"5min":1.46,"15min":1.46},"last":0.69899999999999995},{"interface":"front","average":{"1min":1.0680000000000001,"5min":1.0680000000000001,"15min":1.0680000000000001},"min":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"max":{"1min":2.1499999999999999,"5min":2.1499999999999999,"15min":2.1499999999999999},"last":0.63900000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.2150000000000001},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.3169999999999999,"15min":1.3169999999999999},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":4.6799999999999997,"5min":4.6799999999999997,"15min":4.6799999999999997},"last":0.72799999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.526,"5min":1.526,"15min":1.526},"min":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"max":{"1min":4.5789999999999997,"5min":4.5789999999999997,"15min":4.5789999999999997},"last":1.071},{"interface":"front","average":{"1min":1.278,"5min":1.278,"15min":1.278},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0840000000000001,"5min":2.0840000000000001,"15min":2.0840000000000001},"last":0.88700000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.3500000000000001,"15min":1.3500000000000001},"min":{"1min":0.94699999999999995,"5min":0.94699999999999995,"15min":0.94699999999999995},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.171},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.5489999999999999,"15min":1.5489999999999999},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.4980000000000002,"5min":2.4980000000000002,"15min":2.4980000000000002},"last":0.84999999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.605,"5min":1.605,"15min":1.605},"min":{"1min":0.97299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"max":{"1min":4.6360000000000001,"5min":4.6360000000000001,"15min":4.6360000000000001},"last":0.872},{"interface":"front","average":{"1min":1.4790000000000001,"5min":1.4790000000000001,"15min":1.4790000000000001},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.85999999999999999},"max":{"1min":2.2970000000000002,"5min":2.2970000000000002,"15min":2.2970000000000002},"last":0.96699999999999997}]},{"osd":7,"last update":"Sat Oct 14 16:41:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5960000000000001,"5min":1.5960000000000001,"15min":1.5960000000000001},"min":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"max":{"1min":2.452,"5min":2.452,"15min":2.452},"last":1.1459999999999999},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.5449999999999999,"15min":1.5449999999999999},"min":{"1min":1.0269999999999999,"5min":1.0269999999999999,"15min":1.0269999999999999},"max":{"1min":2.359,"5min":2.359,"15min":2.359},"last":0.94899999999999995}]}]},{"osd":0,"up_from":13,"seq":55834574868,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32144,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339696,"statfs":{"total":96636764160,"available":96603848704,"internally_reserved":0,"allocated":978944,"data_stored":742611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.5189999999999999,"5min":1.5189999999999999,"15min":1.5189999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.96399999999999997},"min":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"max":{"1min":1.7929999999999999,"5min":1.7929999999999999,"15min":1.7929999999999999},"last":0.624}]},{"osd":2,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"min":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"max":{"1min":1.96,"5min":1.96,"15min":1.96},"last":0.47699999999999998},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.629,"5min":0.629,"15min":0.629},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.69499999999999995}]},{"osd":3,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.1549999999999998,"5min":2.1549999999999998,"15min":2.1549999999999998},"last":0.71399999999999997},{"interface":"front","average":{"1min":1.169,"5min":1.169,"15min":1.169},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.0059999999999998,"5min":2.0059999999999998,"15min":2.0059999999999998},"last":0.87}]},{"osd":4,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.147,"15min":1.147},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.762,"5min":1.762,"15min":1.762},"last":1.0429999999999999},{"interface":"front","average":{"1min":1.46,"5min":1.46,"15min":1.46},"min":{"1min":0.96999999999999997,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.3460000000000001},"last":0.93999999999999995}]},{"osd":5,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.323,"5min":1.323,"15min":1.323},"min":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.0539999999999998},"last":1.1299999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.4119999999999999,"15min":1.4119999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.125,"5min":2.125,"15min":2.125},"last":0.98199999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4530000000000001,"5min":1.4530000000000001,"15min":1.4530000000000001},"min":{"1min":1.052,"5min":1.052,"15min":1.052},"max":{"1min":2.2309999999999999,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.337},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"min":{"1min":0.879,"5min":0.879,"15min":0.879},"max":{"1min":1.9299999999999999,"5min":1.9299999999999999,"15min":1.9299999999999999},"last":1.2190000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:41:15 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.464,"15min":1.464},"min":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"max":{"1min":2.5219999999999998,"5min":2.5219999999999998,"15min":2.5219999999999998},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.488,"5min":1.488,"15min":1.488},"min":{"1min":1.0720000000000001,"5min":1.0720000000000001,"15min":1.0720000000000001},"max":{"1min":2.3119999999999998,"5min":2.3119999999999998,"15min":2.3119999999999998},"last":1.415}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:41:49.320 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:41:49.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.596+0000 7ff17ce52640 1 Processor -- start 2023-10-14T16:41:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.596+0000 7ff17ce52640 1 -- start start 2023-10-14T16:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.597+0000 7ff17ce52640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.597+0000 7ff17ce52640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 0x7ff1780c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff17ce52640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1780ce050 con 0x7ff178070820 2023-10-14T16:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff17ce52640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1780cac30 con 0x7ff1780cd210 2023-10-14T16:41:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff17ce52640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff17815a660 con 0x7ff1780ca850 2023-10-14T16:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff177fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 0x7ff1780c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff1777fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:58038/0 (socket says 172.21.15.29:58038) 2023-10-14T16:41:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.598+0000 7ff1777fe640 1 -- 172.21.15.29:0/2204178992 learned_addr learned my addr 172.21.15.29:0/2204178992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.599+0000 7ff1777fe640 1 -- 172.21.15.29:0/2204178992 >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 msgr2=0x7ff1780c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.599+0000 7ff1777fe640 1 --2- 172.21.15.29:0/2204178992 >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 0x7ff1780c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.599+0000 7ff1777fe640 1 -- 172.21.15.29:0/2204178992 >> v1:172.21.15.186:6789/0 conn(0x7ff178070820 legacy=0x7ff1780ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.599+0000 7ff1777fe640 1 -- 172.21.15.29:0/2204178992 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17806f780 con 0x7ff1780ca850 2023-10-14T16:41:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.599+0000 7ff177fff640 1 --2- 172.21.15.29:0/2204178992 >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 0x7ff1780c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.601+0000 7ff1777fe640 1 --2- 172.21.15.29:0/2204178992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff168009870 tx=0x7ff16802b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=245477b339abdb4e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.601+0000 7ff1767fc640 1 -- 172.21.15.29:0/2204178992 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168031640 con 0x7ff1780ca850 2023-10-14T16:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.601+0000 7ff1767fc640 1 -- 172.21.15.29:0/2204178992 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1680317a0 con 0x7ff1780ca850 2023-10-14T16:41:49.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.601+0000 7ff1767fc640 1 -- 172.21.15.29:0/2204178992 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff168031b30 con 0x7ff1780ca850 2023-10-14T16:41:49.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 -- 172.21.15.29:0/2204178992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 msgr2=0x7ff17815a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 --2- 172.21.15.29:0/2204178992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff168009870 tx=0x7ff16802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 -- 172.21.15.29:0/2204178992 shutdown_connections 2023-10-14T16:41:49.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 --2- 172.21.15.29:0/2204178992 >> v2:172.21.15.186:3300/0 conn(0x7ff1780cd210 0x7ff1780c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 --2- 172.21.15.29:0/2204178992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff17815a120 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 -- 172.21.15.29:0/2204178992 >> 172.21.15.29:0/2204178992 conn(0x7ff1780ce6a0 msgr2=0x7ff1780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:49.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 -- 172.21.15.29:0/2204178992 shutdown_connections 2023-10-14T16:41:49.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.602+0000 7ff17ce52640 1 -- 172.21.15.29:0/2204178992 wait complete. 2023-10-14T16:41:49.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.604+0000 7ff17ce52640 1 Processor -- start 2023-10-14T16:41:49.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.605+0000 7ff17ce52640 1 -- start start 2023-10-14T16:41:49.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff178070820 0x7ff178150cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff1780c4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780b0300 con 0x7ff1780cd210 2023-10-14T16:41:49.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff178151230 con 0x7ff178070820 2023-10-14T16:41:49.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff17ce52640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1780c4b60 con 0x7ff1780ca850 2023-10-14T16:41:49.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff177fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff177fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39454/0 (socket says 172.21.15.29:39454) 2023-10-14T16:41:49.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.606+0000 7ff177fff640 1 -- 172.21.15.29:0/1570837326 learned_addr learned my addr 172.21.15.29:0/1570837326 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:49.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff176ffd640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff1780c4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff1777fe640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff178070820 0x7ff178150cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 -- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 msgr2=0x7ff1780c4620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff1780c4620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 -- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff178070820 msgr2=0x7ff178150cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff178070820 0x7ff178150cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 -- 172.21.15.29:0/1570837326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff17000ab80 con 0x7ff1780cd210 2023-10-14T16:41:49.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff177fff640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff170009dd0 tx=0x7ff1700075c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170003b90 con 0x7ff1780cd210 2023-10-14T16:41:49.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff170010040 con 0x7ff1780cd210 2023-10-14T16:41:49.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff170007e60 con 0x7ff1780cd210 2023-10-14T16:41:49.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.607+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1680094d0 con 0x7ff1780cd210 2023-10-14T16:41:49.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.608+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1780af540 con 0x7ff1780cd210 2023-10-14T16:41:49.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.609+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff17804e9b0 con 0x7ff1780cd210 2023-10-14T16:41:49.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.610+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7ff170004030 con 0x7ff1780cd210 2023-10-14T16:41:49.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.611+0000 7ff15ffff640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 0x7ff16406d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:49.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.611+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7ff170089840 con 0x7ff1780cd210 2023-10-14T16:41:49.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.611+0000 7ff1777fe640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 0x7ff16406d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:49.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.612+0000 7ff1777fe640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 0x7ff16406d360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff168036000 tx=0x7ff16803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:49.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.615+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff170054850 con 0x7ff1780cd210 2023-10-14T16:41:49.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.882+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff1780c1cf0 con 0x7ff1780cd210 2023-10-14T16:41:49.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.887+0000 7ff15ffff640 1 -- 172.21.15.29:0/1570837326 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v9) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff170019020 con 0x7ff1780cd210 2023-10-14T16:41:49.888 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:41:49.888 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:41:49.889 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:41:49.889 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:41:49.889 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:41:49.889 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:41:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 msgr2=0x7ff16406d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 0x7ff16406d360 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ff168036000 tx=0x7ff16803c040 comp rx=0 tx=0).stop 2023-10-14T16:41:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 msgr2=0x7ff1780b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff170009dd0 tx=0x7ff1700075c0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 shutdown_connections 2023-10-14T16:41:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7ff16406aea0 0x7ff16406d360 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1780cd210 0x7ff1780b36e0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1780ca850 0x7ff1780c4620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 --2- 172.21.15.29:0/1570837326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff178070820 0x7ff178150cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 >> 172.21.15.29:0/1570837326 conn(0x7ff1780ce6a0 msgr2=0x7ff17815ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:49.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 shutdown_connections 2023-10-14T16:41:49.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:49.890+0000 7ff17ce52640 1 -- 172.21.15.29:0/1570837326 wait complete. 2023-10-14T16:41:49.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:41:50.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.127+0000 7f5bdb3e2640 1 Processor -- start 2023-10-14T16:41:50.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.127+0000 7f5bdb3e2640 1 -- start start 2023-10-14T16:41:50.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.128+0000 7f5bdb3e2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd40ad970 0x7f5bd40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.128+0000 7f5bdb3e2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.128+0000 7f5bdb3e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd415b9d0 con 0x7f5bd406c040 2023-10-14T16:41:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bdb3e2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd415bcb0 con 0x7f5bd40ad970 2023-10-14T16:41:50.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bdb3e2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd415bf90 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bdabe1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bd9bdf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd40ad970 0x7f5bd40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bdabe1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:58050/0 (socket says 172.21.15.29:58050) 2023-10-14T16:41:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bdabe1640 1 -- 172.21.15.29:0/960498747 learned_addr learned my addr 172.21.15.29:0/960498747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.129+0000 7f5bda3e0640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5bd406c040 0x7f5bd406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:55686/0 (socket says 172.21.15.29:55686) 2023-10-14T16:41:50.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.130+0000 7f5bdabe1640 1 -- 172.21.15.29:0/960498747 >> v2:172.21.15.186:3300/0 conn(0x7f5bd40ad970 msgr2=0x7f5bd40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.130+0000 7f5bdabe1640 1 --2- 172.21.15.29:0/960498747 >> v2:172.21.15.186:3300/0 conn(0x7f5bd40ad970 0x7f5bd40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.130+0000 7f5bdabe1640 1 -- 172.21.15.29:0/960498747 >> v1:172.21.15.186:6789/0 conn(0x7f5bd406c040 legacy=0x7f5bd406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.130+0000 7f5bdabe1640 1 -- 172.21.15.29:0/960498747 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd40cc270 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.130+0000 7f5bd93de640 1 -- 172.21.15.29:0/960498747 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1808689478 0 0) 0x7f5bd415b9d0 con 0x7f5bd406c040 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.132+0000 7f5bdabe1640 1 --2- 172.21.15.29:0/960498747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e50 tx=0x7f5bcc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b319cd35797a76de server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.132+0000 7f5bd93de640 1 -- 172.21.15.29:0/960498747 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc02cb00 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.132+0000 7f5bd93de640 1 -- 172.21.15.29:0/960498747 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bcc02cc60 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.132+0000 7f5bd93de640 1 -- 172.21.15.29:0/960498747 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc0343f0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/960498747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 msgr2=0x7f5bd40cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/960498747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e50 tx=0x7f5bcc02bf60 comp rx=0 tx=0).stop 2023-10-14T16:41:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/960498747 shutdown_connections 2023-10-14T16:41:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/960498747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd40ccfb0 0x7f5bd40cd460 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/960498747 >> v2:172.21.15.186:3300/0 conn(0x7f5bd40ad970 0x7f5bd40cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/960498747 >> 172.21.15.29:0/960498747 conn(0x7f5bd40ce6a0 msgr2=0x7f5bd40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:50.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/960498747 shutdown_connections 2023-10-14T16:41:50.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.133+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/960498747 wait complete. 2023-10-14T16:41:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.135+0000 7f5bdb3e2640 1 Processor -- start 2023-10-14T16:41:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.135+0000 7f5bdb3e2640 1 -- start start 2023-10-14T16:41:50.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd406c040 0x7f5bd40ca980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd40ad970 0x7f5bd4156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c98e0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c9bc0 con 0x7f5bd40ad970 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdb3e2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd40c9ea0 con 0x7f5bd406c040 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdabe1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdabe1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39466/0 (socket says 172.21.15.29:39466) 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bdabe1640 1 -- 172.21.15.29:0/1419977984 learned_addr learned my addr 172.21.15.29:0/1419977984 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bd9bdf640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd40ad970 0x7f5bd4156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.136+0000 7f5bda3e0640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd406c040 0x7f5bd40ca980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 -- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd406c040 msgr2=0x7f5bd40ca980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd406c040 0x7f5bd40ca980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 -- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd40ad970 msgr2=0x7f5bd4156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd40ad970 0x7f5bd4156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 -- 172.21.15.29:0/1419977984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc009b30 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdabe1640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e50 tx=0x7f5bcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc040670 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bcc040810 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd40ca180 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.137+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bcc040bc0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.138+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd4156a80 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.139+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd406b330 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.139+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f5bcc040d20 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.140+0000 7f5bcaffd640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 0x7f5bb806d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.140+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5bcc0ac6a0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.141+0000 7f5bda3e0640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 0x7f5bb806d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:50.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.141+0000 7f5bda3e0640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 0x7f5bb806d330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000f60 tx=0x7f5bd0007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:50.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.145+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bcc0777d0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.369+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bd404e9b0 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.370+0000 7f5bcaffd640 1 -- 172.21.15.29:0/1419977984 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f5bcc009510 con 0x7f5bd40ccfb0 2023-10-14T16:41:50.372 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:41:50.372 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:41:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:41:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 msgr2=0x7f5bb806d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 0x7f5bb806d330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0000f60 tx=0x7f5bd0007b50 comp rx=0 tx=0).stop 2023-10-14T16:41:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 msgr2=0x7f5bd40c6780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc009e50 tx=0x7f5bcc004a40 comp rx=0 tx=0).stop 2023-10-14T16:41:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 shutdown_connections 2023-10-14T16:41:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5bb806ae70 0x7f5bb806d330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd40ccfb0 0x7f5bd40c6780 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bd40ad970 0x7f5bd4156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 --2- 172.21.15.29:0/1419977984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd406c040 0x7f5bd40ca980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:50.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.372+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 >> 172.21.15.29:0/1419977984 conn(0x7f5bd40ce6a0 msgr2=0x7f5bd406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.373+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 shutdown_connections 2023-10-14T16:41:50.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:50.373+0000 7f5bdb3e2640 1 -- 172.21.15.29:0/1419977984 wait complete. 2023-10-14T16:41:50.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:41:38.721456+0000', 'modified': '2023-10-14T16:41:41.828160+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5118}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5118': {'gid': 5118, 'name': 'a', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.29:6837/2277792616', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6835', 'nonce': 2277792616}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 2277792616}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:41:50.793 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:41:50.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-14T16:41:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.029+0000 7f699887c640 1 Processor -- start 2023-10-14T16:41:51.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.029+0000 7f699887c640 1 -- start start 2023-10-14T16:41:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.030+0000 7f699887c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69940ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.030+0000 7f699887c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940cb030 con 0x7f699415ad70 2023-10-14T16:41:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.030+0000 7f699887c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69940c91c0 con 0x7f699406a4b0 2023-10-14T16:41:51.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.030+0000 7f699887c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c94a0 con 0x7f69940caac0 2023-10-14T16:41:51.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f699415ad70 0x7f6994069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41668/0 (socket says 172.21.15.29:41668) 2023-10-14T16:41:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699259c640 1 -- 172.21.15.29:0/4078060600 learned_addr learned my addr 172.21.15.29:0/4078060600 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238770201 0 0) 0x7f69940cb030 con 0x7f699415ad70 2023-10-14T16:41:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6980003520 con 0x7f699415ad70 2023-10-14T16:41:51.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69940ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 822358583 0 0) 0x7f6980003520 con 0x7f699415ad70 2023-10-14T16:41:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 msgr2=0x7f69940ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69940ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 >> v1:172.21.15.186:6789/0 conn(0x7f699406a4b0 legacy=0x7f69940ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940c9780 con 0x7f699415ad70 2023-10-14T16:41:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.031+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6984001670 con 0x7f699415ad70 2023-10-14T16:41:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 >> v1:172.21.15.29:6789/0 conn(0x7f699415ad70 legacy=0x7f6994069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69800039a0 0x7f6980003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69800047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6980004d00 0x7f6980005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c94a0 con 0x7f6980004d00 2023-10-14T16:41:51.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698000bb10 con 0x7f69800039a0 2023-10-14T16:41:51.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698000bcd0 con 0x7f69940caac0 2023-10-14T16:41:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6980004d00 0x7f6980005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f699259c640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69800039a0 0x7f6980003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f6991d9b640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69800047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.032+0000 7f6992d9d640 1 -- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 msgr2=0x7f69800047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.033+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69940caac0 0x7f69800047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.033+0000 7f6992d9d640 1 -- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69800039a0 msgr2=0x7f6980003da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.033+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69800039a0 0x7f6980003da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.033+0000 7f6992d9d640 1 -- 172.21.15.29:0/4078060600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6980003520 con 0x7f6980004d00 2023-10-14T16:41:51.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.034+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6980004d00 0x7f6980005180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f698c00ba30 tx=0x7f69800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7112aaa93ce4d58 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.034+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c032640 con 0x7f6980004d00 2023-10-14T16:41:51.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.034+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c0327a0 con 0x7f6980004d00 2023-10-14T16:41:51.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.034+0000 7f699159a640 1 -- 172.21.15.29:0/4078060600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c032b30 con 0x7f6980004d00 2023-10-14T16:41:51.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 -- 172.21.15.29:0/4078060600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6980004d00 msgr2=0x7f6980005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 --2- 172.21.15.29:0/4078060600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6980004d00 0x7f6980005180 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f698c00ba30 tx=0x7f69800057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f6992d9d640 1 -- 172.21.15.29:0/4078060600 reap_dead start 2023-10-14T16:41:51.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 -- 172.21.15.29:0/4078060600 shutdown_connections 2023-10-14T16:41:51.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 -- 172.21.15.29:0/4078060600 >> 172.21.15.29:0/4078060600 conn(0x7f69940ce9b0 msgr2=0x7f69940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 -- 172.21.15.29:0/4078060600 shutdown_connections 2023-10-14T16:41:51.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.035+0000 7f699887c640 1 -- 172.21.15.29:0/4078060600 wait complete. 2023-10-14T16:41:51.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.037+0000 7f699887c640 1 Processor -- start 2023-10-14T16:41:51.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.037+0000 7f699887c640 1 -- start start 2023-10-14T16:41:51.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 0x7f6994158dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 0x7f699406b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69941f3960 con 0x7f69940c86a0 2023-10-14T16:41:51.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69940c7c50 con 0x7f69940c8b20 2023-10-14T16:41:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f699887c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69940ab6b0 con 0x7f6980004d00 2023-10-14T16:41:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33752/0 (socket says 172.21.15.29:33752) 2023-10-14T16:41:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.038+0000 7f6991d9b640 1 -- 172.21.15.29:0/4015589763 learned_addr learned my addr 172.21.15.29:0/4015589763 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 0x7f699406b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f699259c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 0x7f6994158dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 -- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 msgr2=0x7f6994158dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 0x7f6994158dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 -- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 msgr2=0x7f699406b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 0x7f699406b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 -- 172.21.15.29:0/4015589763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69940ab990 con 0x7f69940c86a0 2023-10-14T16:41:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6992d9d640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 0x7f699406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f699259c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 0x7f6994158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f6991d9b640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e870 tx=0x7f697c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f698affd640 1 -- 172.21.15.29:0/4015589763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00c8d0 con 0x7f69940c86a0 2023-10-14T16:41:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f698affd640 1 -- 172.21.15.29:0/4015589763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c00ca30 con 0x7f69940c86a0 2023-10-14T16:41:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.039+0000 7f698affd640 1 -- 172.21.15.29:0/4015589763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00cde0 con 0x7f69940c86a0 2023-10-14T16:41:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.040+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69940c7ea0 con 0x7f69940c86a0 2023-10-14T16:41:51.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.040+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6994159300 con 0x7f69940c86a0 2023-10-14T16:41:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.042+0000 7f698affd640 1 -- 172.21.15.29:0/4015589763 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7f697c01f460 con 0x7f69940c86a0 2023-10-14T16:41:51.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.043+0000 7f698affd640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 0x7f697406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.043+0000 7f698affd640 1 -- 172.21.15.29:0/4015589763 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f697c089e50 con 0x7f69940c86a0 2023-10-14T16:41:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.043+0000 7f699259c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 0x7f697406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.044+0000 7f699b40d740 1 Processor -- start 2023-10-14T16:41:51.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.044+0000 7f699b40d740 1 -- start start 2023-10-14T16:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.044+0000 7f699259c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 0x7f697406d330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6984001050 tx=0x7f6984001190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 0x558de0c266c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c08340 0x558de0be0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0afb7a0 con 0x558de0be0570 2023-10-14T16:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0b4d740 con 0x558de0c28b00 2023-10-14T16:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de0b4ae30 con 0x558de0c08340 2023-10-14T16:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f6992d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f6992d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33768/0 (socket says 172.21.15.29:33768) 2023-10-14T16:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.045+0000 7f6992d9d640 1 -- 172.21.15.29:0/3090634442 learned_addr learned my addr 172.21.15.29:0/3090634442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f699259c640 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 0x558de0c266c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 -- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c08340 msgr2=0x558de0be0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c08340 0x558de0be0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 -- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 msgr2=0x558de0c266c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 0x558de0c266c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 -- 172.21.15.29:0/3090634442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558de0b0a910 con 0x558de0be0570 2023-10-14T16:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f699259c640 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 0x558de0c266c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6992d9d640 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f698c00bb50 tx=0x7f698c030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0b760b78bb7406 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6973fff640 1 -- 172.21.15.29:0/3090634442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c032580 con 0x558de0be0570 2023-10-14T16:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6973fff640 1 -- 172.21.15.29:0/3090634442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c004020 con 0x558de0be0570 2023-10-14T16:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.046+0000 7f6973fff640 1 -- 172.21.15.29:0/3090634442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f698c004390 con 0x558de0be0570 2023-10-14T16:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 -- 172.21.15.29:0/3090634442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 msgr2=0x558de0c0ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f698c00bb50 tx=0x7f698c030000 comp rx=0 tx=0).stop 2023-10-14T16:41:51.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 -- 172.21.15.29:0/3090634442 shutdown_connections 2023-10-14T16:41:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0c0ad10 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c08340 0x558de0be0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 --2- 172.21.15.29:0/3090634442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c28b00 0x558de0c266c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 -- 172.21.15.29:0/3090634442 >> 172.21.15.29:0/3090634442 conn(0x558de0bdd190 msgr2=0x558de0c25d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 -- 172.21.15.29:0/3090634442 shutdown_connections 2023-10-14T16:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.047+0000 7f699b40d740 1 -- 172.21.15.29:0/3090634442 wait complete. 2023-10-14T16:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.048+0000 7f699b40d740 1 -- start start 2023-10-14T16:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.048+0000 7f699b40d740 1 Processor -- start 2023-10-14T16:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c3a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0bf4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0b46720 con 0x558de0c28b00 2023-10-14T16:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0ae4970 con 0x558de0c08340 2023-10-14T16:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de0ae4610 con 0x558de0c03130 2023-10-14T16:41:51.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f699b40d740 10 client.0 _ll_register_callbacks cb 0x558de0b069d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33784/0 (socket says 172.21.15.29:33784) 2023-10-14T16:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.049+0000 7f6991d9b640 1 -- 172.21.15.29:0/2769468263 learned_addr learned my addr 172.21.15.29:0/2769468263 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 -- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 msgr2=0x558de0bf4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0bf4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 -- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 msgr2=0x558de0c3a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c3a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 -- 172.21.15.29:0/2769468263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6980003520 con 0x558de0c28b00 2023-10-14T16:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f6991d9b640 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f697c060980 tx=0x7f697c0593f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13bee178628e2c3d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f69717fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c0580a0 con 0x558de0c28b00 2023-10-14T16:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f697c058200 con 0x558de0c28b00 2023-10-14T16:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.050+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x558de0b6d160 con 0x558de0c28b00 2023-10-14T16:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f697c0880b0 con 0x558de0c28b00 2023-10-14T16:41:51.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 10 client.5310 fetch_fsmap learned FSMap version 15 2023-10-14T16:41:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558de0b44da0 con 0x558de0c28b00 2023-10-14T16:41:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x558de0b6d160 con 0x558de0c28b00 2023-10-14T16:41:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f697c060a20 con 0x558de0c28b00 2023-10-14T16:41:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 10 client.5310 fetch_fsmap finished waiting for FSMap version 15 2023-10-14T16:41:51.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 10 client.5310 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T16:41:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 20 client.5310 populate_metadata read hostname 'smithi029' 2023-10-14T16:41:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c3dd90 0x558de0c08bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 4 client.5310 mds_command: new command op to 5118 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:41:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.051+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 --> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558de0b18cc0 con 0x558de0c3dd90 2023-10-14T16:41:51.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.052+0000 7f699259c640 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c3dd90 0x558de0c08bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.052+0000 7f699259c640 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c3dd90 0x558de0c08bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d6d2ceddf8f33d2 server_cookie=e51e30eaadc5e070 in_seq=0 out_seq=0 2023-10-14T16:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.052+0000 7f69717fa640 10 client.5310 ms_handle_connect on v2:172.21.15.29:6835/2277792616 2023-10-14T16:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f69717fa640 1 -- 172.21.15.29:0/2769468263 <== mds.0 v2:172.21.15.29:6835/2277792616 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x558de0b18cc0 con 0x558de0c3dd90 2023-10-14T16:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f69717fa640 10 client.5310 handle_command_reply: tid=0 2023-10-14T16:41:51.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 client.5310 shutdown 2023-10-14T16:41:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 msgr2=0x558de0c0ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f697c060980 tx=0x7f697c0593f0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 shutdown_connections 2023-10-14T16:41:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c3dd90 0x558de0c08bb0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0bf4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c28b00 0x558de0c0ddc0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 --2- 172.21.15.29:0/2769468263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c3a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f69717fa640 0 client.5310 ms_handle_reset on v2:172.21.15.29:6835/2277792616 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 >> 172.21.15.29:0/2769468263 conn(0x558de0be0570 msgr2=0x558de0c25c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.053+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 shutdown_connections 2023-10-14T16:41:51.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.054+0000 7f699b40d740 1 -- 172.21.15.29:0/2769468263 wait complete. 2023-10-14T16:41:51.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.054+0000 7f699b40d740 20 client.5310 trim_cache size 0 max 16384 2023-10-14T16:41:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.070+0000 7f699b40d740 1 Processor -- start 2023-10-14T16:41:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 -- start start 2023-10-14T16:41:51.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0be0570 0x558de0bdaec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0bda380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de08aaf20 con 0x558de0c03130 2023-10-14T16:41:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0c0d860 con 0x558de0c08340 2023-10-14T16:41:51.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.071+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de0c0a2a0 con 0x558de0be0570 2023-10-14T16:41:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33790/0 (socket says 172.21.15.29:33790) 2023-10-14T16:41:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 -- 172.21.15.29:0/1946088455 learned_addr learned my addr 172.21.15.29:0/1946088455 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6992d9d640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0bda380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 -- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0be0570 msgr2=0x558de0bdaec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f699259c640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0be0570 0x558de0bdaec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0be0570 0x558de0bdaec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 -- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 msgr2=0x558de0bda380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0bda380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6991d9b640 1 -- 172.21.15.29:0/1946088455 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558de0ae3b80 con 0x558de0c03130 2023-10-14T16:41:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.072+0000 7f6992d9d640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0bda380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f6991d9b640 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f697c00c4b0 tx=0x7f697c06d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=424a008a87ac5221 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f6988ff9640 1 -- 172.21.15.29:0/1946088455 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c07c070 con 0x558de0c03130 2023-10-14T16:41:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f6988ff9640 1 -- 172.21.15.29:0/1946088455 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697c060a20 con 0x558de0c03130 2023-10-14T16:41:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f6988ff9640 1 -- 172.21.15.29:0/1946088455 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c060b90 con 0x558de0c03130 2023-10-14T16:41:51.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 -- 172.21.15.29:0/1946088455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 msgr2=0x558de0bd9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f697c00c4b0 tx=0x7f697c06d000 comp rx=0 tx=0).stop 2023-10-14T16:41:51.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 -- 172.21.15.29:0/1946088455 shutdown_connections 2023-10-14T16:41:51.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0bda380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0c03130 0x558de0bd9e40 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 --2- 172.21.15.29:0/1946088455 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0be0570 0x558de0bdaec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.073+0000 7f699b40d740 1 -- 172.21.15.29:0/1946088455 >> 172.21.15.29:0/1946088455 conn(0x558de0c254e0 msgr2=0x558de0c28010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.074+0000 7f699b40d740 1 -- 172.21.15.29:0/1946088455 shutdown_connections 2023-10-14T16:41:51.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.074+0000 7f699b40d740 1 -- 172.21.15.29:0/1946088455 wait complete. 2023-10-14T16:41:51.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.074+0000 7f699b40d740 1 -- start start 2023-10-14T16:41:51.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.074+0000 7f699b40d740 1 Processor -- start 2023-10-14T16:41:51.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0c30d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c31240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0c32600 con 0x558de0be0570 2023-10-14T16:41:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x558de0c091d0 con 0x558de0c08340 2023-10-14T16:41:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x558de0c31b00 con 0x558de0c03130 2023-10-14T16:41:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.075+0000 7f699b40d740 10 client.0 _ll_register_callbacks cb 0x7f697c004fb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:41:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33798/0 (socket says 172.21.15.29:33798) 2023-10-14T16:41:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 -- 172.21.15.29:0/2578474895 learned_addr learned my addr 172.21.15.29:0/2578474895 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 -- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 msgr2=0x558de0c30d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0c30d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 -- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 msgr2=0x558de0c31240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c31240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 -- 172.21.15.29:0/2578474895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f697c076070 con 0x558de0be0570 2023-10-14T16:41:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f699259c640 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x558de0c3d470 tx=0x7f6984001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb213b0a98e5add server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.076+0000 7f69717fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:41:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6984023b90 con 0x558de0be0570 2023-10-14T16:41:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f6984023cf0 con 0x558de0be0570 2023-10-14T16:41:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f698c004020 con 0x558de0be0570 2023-10-14T16:41:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6984014940 con 0x558de0be0570 2023-10-14T16:41:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 10 client.5316 fetch_fsmap learned FSMap version 15 2023-10-14T16:41:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) v3 -- 0x558de0b09170 con 0x558de0be0570 2023-10-14T16:41:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f698c004020 con 0x558de0be0570 2023-10-14T16:41:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 15) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f6984021050 con 0x558de0be0570 2023-10-14T16:41:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 10 client.5316 fetch_fsmap finished waiting for FSMap version 15 2023-10-14T16:41:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 10 client.5316 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T16:41:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 20 client.5316 populate_metadata read hostname 'smithi029' 2023-10-14T16:41:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c28b00 0x558de0c29050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 4 client.5316 mds_command: new command op to 5118 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:41:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.077+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 --> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] -- command(tid 0: {"prefix": "status"}) v1 -- 0x558de0b3f700 con 0x558de0c28b00 2023-10-14T16:41:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f6991d9b640 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c28b00 0x558de0c29050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f6991d9b640 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c28b00 0x558de0c29050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ef9efccb9f54fd40 server_cookie=682826b12769753f in_seq=0 out_seq=0 2023-10-14T16:41:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f69717fa640 10 client.5316 ms_handle_connect on v2:172.21.15.29:6835/2277792616 2023-10-14T16:41:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f69717fa640 1 -- 172.21.15.29:0/2578474895 <== mds.0 v2:172.21.15.29:6835/2277792616 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+319 (crc 0 0 0) 0x558de0b3f700 con 0x558de0c28b00 2023-10-14T16:41:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f69717fa640 10 client.5316 handle_command_reply: tid=0 2023-10-14T16:41:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.078+0000 7f699b40d740 1 client.5316 shutdown 2023-10-14T16:41:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 msgr2=0x558de0bda240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x558de0c3d470 tx=0x7f6984001930 comp rx=0 tx=0).stop 2023-10-14T16:41:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 shutdown_connections 2023-10-14T16:41:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:6835/2277792616,v1:172.21.15.29:6837/2277792616] conn(0x558de0c28b00 0x558de0c29050 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x558de0c08340 0x558de0c31240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x558de0c03130 0x558de0c30d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 --2- 172.21.15.29:0/2578474895 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x558de0be0570 0x558de0bda240 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f69717fa640 0 client.5316 ms_handle_reset on v2:172.21.15.29:6835/2277792616 2023-10-14T16:41:51.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 >> 172.21.15.29:0/2578474895 conn(0x558de0c254e0 msgr2=0x558de0bdae50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 shutdown_connections 2023-10-14T16:41:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 1 -- 172.21.15.29:0/2578474895 wait complete. 2023-10-14T16:41:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699b40d740 20 client.5316 trim_cache size 0 max 16384 2023-10-14T16:41:51.094 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:41:51.095 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:41:51.095 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:41:51.095 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5118, 2023-10-14T16:41:51.095 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:41:51.095 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:41:51.096 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:41:51.096 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 11.814734864, 2023-10-14T16:41:51.096 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 15, 2023-10-14T16:41:51.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 34, 2023-10-14T16:41:51.096 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 34, 2023-10-14T16:41:51.097 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 21.639972643 2023-10-14T16:41:51.097 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:41:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 msgr2=0x7f697406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.079+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 0x7f697406d330 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6984001050 tx=0x7f6984001190 comp rx=0 tx=0).stop 2023-10-14T16:41:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 msgr2=0x7f699406b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e870 tx=0x7f697c00ed30 comp rx=0 tx=0).stop 2023-10-14T16:41:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 shutdown_connections 2023-10-14T16:41:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f697406ae70 0x7f697406d330 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69940c8b20 0x7f699406b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69940c86a0 0x7f699406b030 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 --2- 172.21.15.29:0/4015589763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6980004d00 0x7f6994158dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 >> 172.21.15.29:0/4015589763 conn(0x7f69940ce9b0 msgr2=0x7f69940b2c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 shutdown_connections 2023-10-14T16:41:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.080+0000 7f699887c640 1 -- 172.21.15.29:0/4015589763 wait complete. 2023-10-14T16:41:51.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:41:51.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.327+0000 7fe34359e640 1 Processor -- start 2023-10-14T16:41:51.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.327+0000 7fe34359e640 1 -- start start 2023-10-14T16:41:51.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe34359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3440712c0 0x7fe344070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe34359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440b4620 con 0x7fe344070d70 2023-10-14T16:41:51.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe34359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3440b4340 con 0x7fe34415aad0 2023-10-14T16:41:51.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe34359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe34406f520 con 0x7fe3440712c0 2023-10-14T16:41:51.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe342d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3440712c0 0x7fe344070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe342d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe3440712c0 0x7fe344070000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33810/0 (socket says 172.21.15.29:33810) 2023-10-14T16:41:51.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe342d9d640 1 -- 172.21.15.29:0/682117167 learned_addr learned my addr 172.21.15.29:0/682117167 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe342d9d640 1 -- 172.21.15.29:0/682117167 >> v1:172.21.15.186:6789/0 conn(0x7fe34415aad0 legacy=0x7fe3440cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:51.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.329+0000 7fe342d9d640 1 -- 172.21.15.29:0/682117167 >> v1:172.21.15.29:6789/0 conn(0x7fe344070d70 legacy=0x7fe34406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.330+0000 7fe342d9d640 1 -- 172.21.15.29:0/682117167 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe34406f800 con 0x7fe3440712c0 2023-10-14T16:41:51.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.331+0000 7fe342d9d640 1 --2- 172.21.15.29:0/682117167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344070000 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009e50 tx=0x7fe33c02c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fd309bf50fe8436 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.331+0000 7fe341d9b640 1 -- 172.21.15.29:0/682117167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c02cbb0 con 0x7fe3440712c0 2023-10-14T16:41:51.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.331+0000 7fe341d9b640 1 -- 172.21.15.29:0/682117167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe33c02cd10 con 0x7fe3440712c0 2023-10-14T16:41:51.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.331+0000 7fe341d9b640 1 -- 172.21.15.29:0/682117167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe33c034610 con 0x7fe3440712c0 2023-10-14T16:41:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 -- 172.21.15.29:0/682117167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 msgr2=0x7fe344070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 --2- 172.21.15.29:0/682117167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344070000 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe33c009e50 tx=0x7fe33c02c010 comp rx=0 tx=0).stop 2023-10-14T16:41:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 -- 172.21.15.29:0/682117167 shutdown_connections 2023-10-14T16:41:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 --2- 172.21.15.29:0/682117167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344070000 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 -- 172.21.15.29:0/682117167 >> 172.21.15.29:0/682117167 conn(0x7fe3440b0a90 msgr2=0x7fe3440ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 -- 172.21.15.29:0/682117167 shutdown_connections 2023-10-14T16:41:51.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.332+0000 7fe34359e640 1 -- 172.21.15.29:0/682117167 wait complete. 2023-10-14T16:41:51.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.333+0000 7fe34359e640 1 Processor -- start 2023-10-14T16:41:51.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.334+0000 7fe34359e640 1 -- start start 2023-10-14T16:41:51.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe344070d70 0x7fe34415c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 0x7fe3440c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe34415e040 con 0x7fe3440712c0 2023-10-14T16:41:51.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3441f05a0 con 0x7fe34415aad0 2023-10-14T16:41:51.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe34415ccf0 con 0x7fe344070d70 2023-10-14T16:41:51.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33820/0 (socket says 172.21.15.29:33820) 2023-10-14T16:41:51.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 -- 172.21.15.29:0/1028606700 learned_addr learned my addr 172.21.15.29:0/1028606700 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:51.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe342d9d640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 0x7fe3440c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe34259c640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe344070d70 0x7fe34415c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 -- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe344070d70 msgr2=0x7fe34415c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe344070d70 0x7fe34415c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 -- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 msgr2=0x7fe3440c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 0x7fe3440c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe33bfff640 1 -- 172.21.15.29:0/1028606700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe33c009b30 con 0x7fe3440712c0 2023-10-14T16:41:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.335+0000 7fe342d9d640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 0x7fe3440c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe33bfff640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c480 tx=0x7fe32c00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c0052d0 con 0x7fe3440712c0 2023-10-14T16:41:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe32c005430 con 0x7fe3440712c0 2023-10-14T16:41:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe32c012870 con 0x7fe3440712c0 2023-10-14T16:41:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe34415dd60 con 0x7fe3440712c0 2023-10-14T16:41:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.336+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe34415bfb0 con 0x7fe3440712c0 2023-10-14T16:41:51.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.337+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe34415bb30 con 0x7fe3440712c0 2023-10-14T16:41:51.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.338+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fe32c0129d0 con 0x7fe3440712c0 2023-10-14T16:41:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.339+0000 7fe33affd640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 0x7fe32406d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.339+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe32c0891c0 con 0x7fe3440712c0 2023-10-14T16:41:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.340+0000 7fe34259c640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 0x7fe32406d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.340+0000 7fe34259c640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 0x7fe32406d330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe33400a750 tx=0x7fe334008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:51.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.344+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe34415bb30 con 0x7fe3440712c0 2023-10-14T16:41:51.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.578+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fe34415bb30 con 0x7fe3440712c0 2023-10-14T16:41:51.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.579+0000 7fe33affd640 1 -- 172.21.15.29:0/1028606700 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) v1 ==== 53+0+73 (secure 0 0 0) 0x7fe34415bb30 con 0x7fe3440712c0 2023-10-14T16:41:51.580 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:41:51.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 msgr2=0x7fe32406d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 0x7fe32406d330 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe33400a750 tx=0x7fe334008040 comp rx=0 tx=0).stop 2023-10-14T16:41:51.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 msgr2=0x7fe344158560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe32c00c480 tx=0x7fe32c00c940 comp rx=0 tx=0).stop 2023-10-14T16:41:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 shutdown_connections 2023-10-14T16:41:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe32406ae70 0x7fe32406d330 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe34415aad0 0x7fe3440c75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3440712c0 0x7fe344158560 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 --2- 172.21.15.29:0/1028606700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe344070d70 0x7fe34415c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 >> 172.21.15.29:0/1028606700 conn(0x7fe3440b0a90 msgr2=0x7fe3440b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 shutdown_connections 2023-10-14T16:41:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:51.582+0000 7fe34359e640 1 -- 172.21.15.29:0/1028606700 wait complete. 2023-10-14T16:41:51.602 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:41:51.602 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:41:51.603 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:41:51.603 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:41:51.603 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:41:51.604 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:41:51.604 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:41:51.604 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:41:51.605 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:41:51.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:51.606 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:41:51.662 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:41:51.662 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:41:51.663 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:41:51.663 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.663 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:41:51.664 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.664 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:41:51.665 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:41:51.665 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:41:51.665 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:41:51.666 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:41:51.666 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:41:51.667 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:41:51.667 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:41:51.667 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:41:51.668 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:41:51.668 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 42265sec preferred_lft 42265sec 2023-10-14T16:41:51.668 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:41:51.669 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.669 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:41:51.669 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:41:51.670 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:41:51.670 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:41:51.671 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:41:51.671 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:41:51.671 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.672 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:41:51.672 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.672 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:41:51.673 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:41:51.673 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:41:51.674 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.674 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:41:51.674 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:41:51.675 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:41:51.675 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:41:51.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:51.676 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:41:51.776 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:41:51.781 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:51.781 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:41:51.845 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:41:51.846 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:41:51.846 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:41:51.846 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:41:51.846 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:41:51.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:51.848 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:41:51.848 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:41:51.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:41:51.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:41:51.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:41:51.850 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:41:52.052 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:52.053 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:41:52.109 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:41:52.109 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:41:52.110 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:41:52.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:52.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:41:52.213 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:41:52.214 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:41:52.273 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:41:52.274 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:41:52.274 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:41:52.328 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:41:52.329 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:52.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:41:52.479 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:41:52.479 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:41:52.479 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:41:52.501 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:41:52.502 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:41:52.502 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:41:52.502 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:41:52.502 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:41:52.528 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:41:52.528 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:41:52.528 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:41:52.528 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:41:52.529 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:41:52.529 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:41:52.529 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:41:52.529 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:41:52.530 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:41:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:41:52.530 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:41:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:41:52.531 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:41:52.533 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:41:52.534 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:52.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:41:52.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:52.619 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:41:52.684 INFO:teuthology.orchestra.run.smithi029.stdout:client5328 2023-10-14T16:41:52.685 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:41:52.685 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:41:52.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:41:52.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.969+0000 7fdf8d85d640 1 Processor -- start 2023-10-14T16:41:52.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.969+0000 7fdf8d85d640 1 -- start start 2023-10-14T16:41:52.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf8d85d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf8815ad70 0x7fdf88069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:52.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf8d85d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880cb030 con 0x7fdf880caac0 2023-10-14T16:41:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf8d85d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf880c91c0 con 0x7fdf880c9ca0 2023-10-14T16:41:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf8d85d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf880c94a0 con 0x7fdf8815ad70 2023-10-14T16:41:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf86ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf8815ad70 0x7fdf88069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:52.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf86ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf8815ad70 0x7fdf88069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33834/0 (socket says 172.21.15.29:33834) 2023-10-14T16:41:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.970+0000 7fdf86ffd640 1 -- 172.21.15.29:0/2428407353 learned_addr learned my addr 172.21.15.29:0/2428407353 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.971+0000 7fdf86ffd640 1 -- 172.21.15.29:0/2428407353 >> v1:172.21.15.186:6789/0 conn(0x7fdf880c9ca0 legacy=0x7fdf880ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.971+0000 7fdf86ffd640 1 -- 172.21.15.29:0/2428407353 >> v1:172.21.15.29:6789/0 conn(0x7fdf880caac0 legacy=0x7fdf880ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:52.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.971+0000 7fdf86ffd640 1 -- 172.21.15.29:0/2428407353 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf880c9780 con 0x7fdf8815ad70 2023-10-14T16:41:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.972+0000 7fdf86ffd640 1 --2- 172.21.15.29:0/2428407353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf88069da0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009e50 tx=0x7fdf7802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6d99b42355db811 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.972+0000 7fdf85ffb640 1 -- 172.21.15.29:0/2428407353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7802cb10 con 0x7fdf8815ad70 2023-10-14T16:41:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf85ffb640 1 -- 172.21.15.29:0/2428407353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7802cc70 con 0x7fdf8815ad70 2023-10-14T16:41:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf85ffb640 1 -- 172.21.15.29:0/2428407353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf780343f0 con 0x7fdf8815ad70 2023-10-14T16:41:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 -- 172.21.15.29:0/2428407353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 msgr2=0x7fdf88069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/2428407353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf88069da0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fdf78009e50 tx=0x7fdf7802bf70 comp rx=0 tx=0).stop 2023-10-14T16:41:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 -- 172.21.15.29:0/2428407353 shutdown_connections 2023-10-14T16:41:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/2428407353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf88069da0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 -- 172.21.15.29:0/2428407353 >> 172.21.15.29:0/2428407353 conn(0x7fdf880ce9b0 msgr2=0x7fdf880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 -- 172.21.15.29:0/2428407353 shutdown_connections 2023-10-14T16:41:52.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.973+0000 7fdf8d85d640 1 -- 172.21.15.29:0/2428407353 wait complete. 2023-10-14T16:41:52.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.975+0000 7fdf8d85d640 1 Processor -- start 2023-10-14T16:41:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.975+0000 7fdf8d85d640 1 -- start start 2023-10-14T16:41:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf880c9ca0 0x7fdf8814e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 0x7fdf8814deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8814e3f0 con 0x7fdf8815ad70 2023-10-14T16:41:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf880c6e70 con 0x7fdf880caac0 2023-10-14T16:41:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.976+0000 7fdf8d85d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf880c7150 con 0x7fdf880c9ca0 2023-10-14T16:41:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33844/0 (socket says 172.21.15.29:33844) 2023-10-14T16:41:52.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 -- 172.21.15.29:0/3236409323 learned_addr learned my addr 172.21.15.29:0/3236409323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:52.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf867fc640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 0x7fdf8814deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:52.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 -- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf880c9ca0 msgr2=0x7fdf8814e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:52.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf880c9ca0 0x7fdf8814e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:52.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 -- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 msgr2=0x7fdf8814deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:52.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 0x7fdf8814deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:52.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 -- 172.21.15.29:0/3236409323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf78009b30 con 0x7fdf8815ad70 2023-10-14T16:41:52.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf867fc640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 0x7fdf8814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:52.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.977+0000 7fdf877fe640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001eb0 tx=0x7fdf8000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:52.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.978+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf800052d0 con 0x7fdf8815ad70 2023-10-14T16:41:52.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.978+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf80005430 con 0x7fdf8815ad70 2023-10-14T16:41:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.978+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf880c7430 con 0x7fdf8815ad70 2023-10-14T16:41:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.978+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf80003810 con 0x7fdf8815ad70 2023-10-14T16:41:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.978+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf880c79f0 con 0x7fdf8815ad70 2023-10-14T16:41:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.979+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf8804e160 con 0x7fdf8815ad70 2023-10-14T16:41:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.980+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fdf80003970 con 0x7fdf8815ad70 2023-10-14T16:41:52.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.981+0000 7fdf7ffff640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 0x7fdf6806d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:52.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.981+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fdf80088cc0 con 0x7fdf8815ad70 2023-10-14T16:41:52.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.981+0000 7fdf86ffd640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 0x7fdf6806d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:52.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.982+0000 7fdf86ffd640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 0x7fdf6806d360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdf78035000 tx=0x7fdf7802e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:52.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:52.986+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf80053c40 con 0x7fdf8815ad70 2023-10-14T16:41:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.213+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fdf8806a2e0 con 0x7fdf6806aea0 2023-10-14T16:41:53.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.216+0000 7fdf7ffff640 1 -- 172.21.15.29:0/3236409323 <== mgr.4101 v2:172.21.15.29:6832/3113675785 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fdf78009510 con 0x7fdf6806aea0 2023-10-14T16:41:53.217 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:41:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:41:53.217 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:41:53.218 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:41:53.218 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:41:53.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.218+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 msgr2=0x7fdf6806d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.218+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 0x7fdf6806d360 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fdf78035000 tx=0x7fdf7802e000 comp rx=0 tx=0).stop 2023-10-14T16:41:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.218+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 msgr2=0x7fdf880cc580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.218+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fdf80001eb0 tx=0x7fdf8000c890 comp rx=0 tx=0).stop 2023-10-14T16:41:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 shutdown_connections 2023-10-14T16:41:53.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fdf6806aea0 0x7fdf6806d360 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf8815ad70 0x7fdf880cc580 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf880caac0 0x7fdf8814deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 --2- 172.21.15.29:0/3236409323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf880c9ca0 0x7fdf8814e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 >> 172.21.15.29:0/3236409323 conn(0x7fdf880ce9b0 msgr2=0x7fdf880cdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 shutdown_connections 2023-10-14T16:41:53.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.219+0000 7fdf8d85d640 1 -- 172.21.15.29:0/3236409323 wait complete. 2023-10-14T16:41:53.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:41:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.458+0000 7fab4eb2e640 1 Processor -- start 2023-10-14T16:41:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.458+0000 7fab4eb2e640 1 -- start start 2023-10-14T16:41:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4eb2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4815cd70 0x7fab480d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4eb2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480cded0 0x7fab480b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4eb2e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4815d150 con 0x7fab480b8480 2023-10-14T16:41:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4eb2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab480d1180 con 0x7fab4815cd70 2023-10-14T16:41:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4eb2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480d1460 con 0x7fab480cded0 2023-10-14T16:41:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.459+0000 7fab4db2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4815cd70 0x7fab480d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab4815cd70 0x7fab480d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33860/0 (socket says 172.21.15.29:33860) 2023-10-14T16:41:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 -- 172.21.15.29:0/191175095 learned_addr learned my addr 172.21.15.29:0/191175095 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4d32b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab480b8480 0x7fab480cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41688/0 (socket says 172.21.15.29:41688) 2023-10-14T16:41:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 -- 172.21.15.29:0/191175095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480cded0 msgr2=0x7fab480b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 --2- 172.21.15.29:0/191175095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480cded0 0x7fab480b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 -- 172.21.15.29:0/191175095 >> v1:172.21.15.29:6789/0 conn(0x7fab480b8480 legacy=0x7fab480cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4db2c640 1 -- 172.21.15.29:0/191175095 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab480d1740 con 0x7fab4815cd70 2023-10-14T16:41:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.460+0000 7fab4cb2a640 1 -- 172.21.15.29:0/191175095 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 452657108 0 0) 0x7fab4815d150 con 0x7fab480b8480 2023-10-14T16:41:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.461+0000 7fab4db2c640 1 --2- 172.21.15.29:0/191175095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480d2220 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fab44009870 tx=0x7fab4402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edd8875d8ddfbfa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.461+0000 7fab4cb2a640 1 -- 172.21.15.29:0/191175095 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44030840 con 0x7fab4815cd70 2023-10-14T16:41:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.461+0000 7fab4cb2a640 1 -- 172.21.15.29:0/191175095 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab440309a0 con 0x7fab4815cd70 2023-10-14T16:41:53.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.461+0000 7fab4cb2a640 1 -- 172.21.15.29:0/191175095 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab44030d30 con 0x7fab4815cd70 2023-10-14T16:41:53.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 -- 172.21.15.29:0/191175095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 msgr2=0x7fab480d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/191175095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480d2220 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fab44009870 tx=0x7fab4402b980 comp rx=0 tx=0).stop 2023-10-14T16:41:53.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 -- 172.21.15.29:0/191175095 shutdown_connections 2023-10-14T16:41:53.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/191175095 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480cded0 0x7fab480b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/191175095 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480d2220 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 -- 172.21.15.29:0/191175095 >> 172.21.15.29:0/191175095 conn(0x7fab480cf570 msgr2=0x7fab480cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:53.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 -- 172.21.15.29:0/191175095 shutdown_connections 2023-10-14T16:41:53.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.462+0000 7fab4eb2e640 1 -- 172.21.15.29:0/191175095 wait complete. 2023-10-14T16:41:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.464+0000 7fab4eb2e640 1 Processor -- start 2023-10-14T16:41:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.464+0000 7fab4eb2e640 1 -- start start 2023-10-14T16:41:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480b8480 0x7fab481f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 0x7fab480cc800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab48150a40 con 0x7fab4815cd70 2023-10-14T16:41:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab481f2350 con 0x7fab480cded0 2023-10-14T16:41:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4eb2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab480aa430 con 0x7fab480b8480 2023-10-14T16:41:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4e32d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:53.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4e32d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33866/0 (socket says 172.21.15.29:33866) 2023-10-14T16:41:53.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.465+0000 7fab4e32d640 1 -- 172.21.15.29:0/3215933566 learned_addr learned my addr 172.21.15.29:0/3215933566 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:53.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4d32b640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 0x7fab480cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:53.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4db2c640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480b8480 0x7fab481f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:53.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 -- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480b8480 msgr2=0x7fab481f1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480b8480 0x7fab481f1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 -- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 msgr2=0x7fab480cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 0x7fab480cc800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 -- 172.21.15.29:0/3215933566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab440094d0 con 0x7fab4815cd70 2023-10-14T16:41:53.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4d32b640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 0x7fab480cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:53.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab4e32d640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b9e0 tx=0x7fab4000bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:53.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.466+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab4000caa0 con 0x7fab4815cd70 2023-10-14T16:41:53.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.467+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab4000cc20 con 0x7fab4815cd70 2023-10-14T16:41:53.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.467+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab480a9820 con 0x7fab4815cd70 2023-10-14T16:41:53.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.467+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab40014440 con 0x7fab4815cd70 2023-10-14T16:41:53.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.467+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab480aa0c0 con 0x7fab4815cd70 2023-10-14T16:41:53.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.468+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4804e9b0 con 0x7fab4815cd70 2023-10-14T16:41:53.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.469+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 5) v1 ==== 89229+0+0 (secure 0 0 0) 0x7fab4001d3b0 con 0x7fab4815cd70 2023-10-14T16:41:53.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.470+0000 7fab3e7fc640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 0x7fab2c06d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:53.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.470+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fab400894f0 con 0x7fab4815cd70 2023-10-14T16:41:53.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.470+0000 7fab4db2c640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 0x7fab2c06d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:53.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.471+0000 7fab4db2c640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 0x7fab2c06d330 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab44036000 tx=0x7fab4403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:53.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.475+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab40054500 con 0x7fab4815cd70 2023-10-14T16:41:53.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.724+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fab480aa570 con 0x7fab4815cd70 2023-10-14T16:41:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.780+0000 7fab3e7fc640 1 -- 172.21.15.29:0/3215933566 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v6) v1 ==== 92+0+0 (secure 0 0 0) 0x7fab40057b20 con 0x7fab4815cd70 2023-10-14T16:41:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 msgr2=0x7fab2c06d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 0x7fab2c06d330 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fab44036000 tx=0x7fab4403b040 comp rx=0 tx=0).stop 2023-10-14T16:41:53.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 msgr2=0x7fab480aad80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fab4000b9e0 tx=0x7fab4000bea0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 shutdown_connections 2023-10-14T16:41:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fab2c06ae70 0x7fab2c06d330 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab4815cd70 0x7fab480aad80 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab480cded0 0x7fab480cc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 --2- 172.21.15.29:0/3215933566 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab480b8480 0x7fab481f1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.784+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 >> 172.21.15.29:0/3215933566 conn(0x7fab480cf570 msgr2=0x7fab480b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:53.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.785+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 shutdown_connections 2023-10-14T16:41:53.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:53.787+0000 7fab4eb2e640 1 -- 172.21.15.29:0/3215933566 wait complete. 2023-10-14T16:41:53.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:41:54.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.004+0000 7faa05861640 1 Processor -- start 2023-10-14T16:41:54.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.005+0000 7faa05861640 1 -- start start 2023-10-14T16:41:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7faa05861640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7faa000ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7faa05861640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000cb030 con 0x7faa0006a4b0 2023-10-14T16:41:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7faa05861640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa000c91c0 con 0x7faa0015ad70 2023-10-14T16:41:54.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7faa05861640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa000c94a0 con 0x7faa000caac0 2023-10-14T16:41:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7fa9fe7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa0006a4b0 0x7faa000ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41694/0 (socket says 172.21.15.29:41694) 2023-10-14T16:41:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.006+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/704219969 learned_addr learned my addr 172.21.15.29:0/704219969 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9ff7fe640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7faa000ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785205943 0 0) 0x7faa000cb030 con 0x7faa0006a4b0 2023-10-14T16:41:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9ec003520 con 0x7faa0006a4b0 2023-10-14T16:41:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54444052 0 0) 0x7fa9ec003520 con 0x7faa0006a4b0 2023-10-14T16:41:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 msgr2=0x7faa000ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7faa000ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 >> v1:172.21.15.186:6789/0 conn(0x7faa0015ad70 legacy=0x7faa00069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000c9780 con 0x7faa0006a4b0 2023-10-14T16:41:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa9e8001670 con 0x7faa0006a4b0 2023-10-14T16:41:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 >> v1:172.21.15.29:6789/0 conn(0x7faa0006a4b0 legacy=0x7faa000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9ec0039a0 0x7fa9ec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7fa9ec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000c94a0 con 0x7fa9ec0047c0 2023-10-14T16:41:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec00ba60 con 0x7fa9ec0039a0 2023-10-14T16:41:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.007+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9ec00bc20 con 0x7faa000caac0 2023-10-14T16:41:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9ff7fe640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7fa9ec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9feffd640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9ec0039a0 0x7fa9ec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 msgr2=0x7fa9ec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7fa9ec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9ec0039a0 msgr2=0x7fa9ec003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa9ec0039a0 0x7fa9ec003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/704219969 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9ec003520 con 0x7fa9ec0047c0 2023-10-14T16:41:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.008+0000 7fa9ff7fe640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa000caac0 0x7fa9ec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:41:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.010+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8001eb0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5399120f7eb3155 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.010+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8005ad0 con 0x7fa9ec0047c0 2023-10-14T16:41:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.010+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e8005c30 con 0x7fa9ec0047c0 2023-10-14T16:41:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.010+0000 7fa9fdffb640 1 -- 172.21.15.29:0/704219969 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e802c7a0 con 0x7fa9ec0047c0 2023-10-14T16:41:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 -- 172.21.15.29:0/704219969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9ec0047c0 msgr2=0x7fa9ec004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 --2- 172.21.15.29:0/704219969 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa9ec0047c0 0x7fa9ec004c40 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8001eb0 tx=0x7fa9ec0057f0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7fa9ff7fe640 1 -- 172.21.15.29:0/704219969 reap_dead start 2023-10-14T16:41:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 -- 172.21.15.29:0/704219969 shutdown_connections 2023-10-14T16:41:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 -- 172.21.15.29:0/704219969 >> 172.21.15.29:0/704219969 conn(0x7faa000ce9b0 msgr2=0x7faa000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 -- 172.21.15.29:0/704219969 shutdown_connections 2023-10-14T16:41:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.011+0000 7faa05861640 1 -- 172.21.15.29:0/704219969 wait complete. 2023-10-14T16:41:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.013+0000 7faa05861640 1 Processor -- start 2023-10-14T16:41:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.013+0000 7faa05861640 1 -- start start 2023-10-14T16:41:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9ec0047c0 0x7faa00142d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa000c4d10 0x7faa00143e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa001439c0 con 0x7faa000c25f0 2023-10-14T16:41:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa000c51e0 con 0x7faa000c4d10 2023-10-14T16:41:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7faa05861640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa000c54c0 con 0x7fa9ec0047c0 2023-10-14T16:41:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.014+0000 7fa9fe7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33890/0 (socket says 172.21.15.29:33890) 2023-10-14T16:41:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/1705949633 learned_addr learned my addr 172.21.15.29:0/1705949633 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9feffd640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9ec0047c0 0x7faa00142d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9ff7fe640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa000c4d10 0x7faa00143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9ec0047c0 msgr2=0x7faa00142d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9ec0047c0 0x7faa00142d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa000c4d10 msgr2=0x7faa00143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa000c4d10 0x7faa00143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.015+0000 7fa9fe7fc640 1 -- 172.21.15.29:0/1705949633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa000c57a0 con 0x7faa000c25f0 2023-10-14T16:41:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7fa9fe7fc640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8001eb0 tx=0x7fa9e8001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e803c040 con 0x7faa000c25f0 2023-10-14T16:41:54.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9e802dc40 con 0x7faa000c25f0 2023-10-14T16:41:54.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa00143650 con 0x7faa000c25f0 2023-10-14T16:41:54.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9e8007070 con 0x7faa000c25f0 2023-10-14T16:41:54.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.016+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa00144390 con 0x7faa000c25f0 2023-10-14T16:41:54.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.018+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0004e160 con 0x7faa000c25f0 2023-10-14T16:41:54.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.018+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fa9e8032070 con 0x7faa000c25f0 2023-10-14T16:41:54.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.019+0000 7fa9f7fff640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 0x7fa9e006d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.019+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fa9e80aaef0 con 0x7faa000c25f0 2023-10-14T16:41:54.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.019+0000 7fa9feffd640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 msgr2=0x7fa9e006d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:54.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.019+0000 7fa9feffd640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 0x7fa9e006d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:41:54.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.029+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa9e8076030 con 0x7faa000c25f0 2023-10-14T16:41:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.220+0000 7fa9feffd640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 msgr2=0x7fa9e006d3b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:54.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.220+0000 7fa9feffd640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 0x7fa9e006d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:41:54.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.229+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7faa0006aa30 con 0x7faa000c25f0 2023-10-14T16:41:54.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.234+0000 7fa9f7fff640 1 -- 172.21.15.29:0/1705949633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v11)=0 v11) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa9e8079650 con 0x7faa000c25f0 2023-10-14T16:41:54.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.236+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 msgr2=0x7fa9e006d3b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:41:54.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.236+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 0x7fa9e006d3b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.236+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 msgr2=0x7faa000c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.236+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fa9e8001eb0 tx=0x7fa9e8001930 comp rx=0 tx=0).stop 2023-10-14T16:41:54.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 shutdown_connections 2023-10-14T16:41:54.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fa9e006aef0 0x7fa9e006d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa000c4d10 0x7faa00143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa000c25f0 0x7faa000c47d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 --2- 172.21.15.29:0/1705949633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa9ec0047c0 0x7faa00142d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 >> 172.21.15.29:0/1705949633 conn(0x7faa000ce9b0 msgr2=0x7faa00069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 shutdown_connections 2023-10-14T16:41:54.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.237+0000 7faa05861640 1 -- 172.21.15.29:0/1705949633 wait complete. 2023-10-14T16:41:54.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:41:54.305 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.304+0000 7fdc614da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:41:54.340 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:54.339+0000 7f1895411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:41:54.359 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.358+0000 7fdc614da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:41:54.398 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:54.397+0000 7f1895411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:41:54.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.469+0000 7fe7eefcb640 1 Processor -- start 2023-10-14T16:41:54.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.469+0000 7fe7eefcb640 1 -- start start 2023-10-14T16:41:54.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.470+0000 7fe7eefcb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e806d220 0x7fe7e80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.470+0000 7fe7eefcb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe7e8065ab0 0x7fe7e80cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.470+0000 7fe7eefcb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7e80acd70 con 0x7fe7e806d600 2023-10-14T16:41:54.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7eefcb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e80ac4a0 con 0x7fe7e806d220 2023-10-14T16:41:54.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7eefcb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7e8065f40 con 0x7fe7e8065ab0 2023-10-14T16:41:54.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e806d220 0x7fe7e80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7ee7ca640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe7e806d600 0x7fe7e80ad200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41710/0 (socket says 172.21.15.29:41710) 2023-10-14T16:41:54.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7ee7ca640 1 -- 172.21.15.29:0/351332161 learned_addr learned my addr 172.21.15.29:0/351332161 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:41:54.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7e806d220 0x7fe7e80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33902/0 (socket says 172.21.15.29:33902) 2023-10-14T16:41:54.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7ed7c8640 1 --2- 172.21.15.29:0/351332161 >> v2:172.21.15.186:3300/0 conn(0x7fe7e8065ab0 0x7fe7e80cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 -- 172.21.15.29:0/351332161 >> v2:172.21.15.186:3300/0 conn(0x7fe7e8065ab0 msgr2=0x7fe7e80cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:54.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 --2- 172.21.15.29:0/351332161 >> v2:172.21.15.186:3300/0 conn(0x7fe7e8065ab0 0x7fe7e80cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 -- 172.21.15.29:0/351332161 >> v1:172.21.15.29:6789/0 conn(0x7fe7e806d600 legacy=0x7fe7e80ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7edfc9640 1 -- 172.21.15.29:0/351332161 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e80ac720 con 0x7fe7e806d220 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.471+0000 7fe7ecfc7640 1 -- 172.21.15.29:0/351332161 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244879699 0 0) 0x7fe7e80acd70 con 0x7fe7e806d600 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7edfc9640 1 --2- 172.21.15.29:0/351332161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d220 0x7fe7e80c95e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80cb8f0 tx=0x7fe7e402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=723e7c16ffc5f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7ecfc7640 1 -- 172.21.15.29:0/351332161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4031830 con 0x7fe7e806d220 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7ecfc7640 1 -- 172.21.15.29:0/351332161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7e4031990 con 0x7fe7e806d220 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7ecfc7640 1 -- 172.21.15.29:0/351332161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e4031d20 con 0x7fe7e806d220 2023-10-14T16:41:54.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7eefcb640 1 -- 172.21.15.29:0/351332161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d220 msgr2=0x7fe7e80c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/351332161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d220 0x7fe7e80c95e0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fe7e80cb8f0 tx=0x7fe7e402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.473+0000 7fe7eefcb640 1 -- 172.21.15.29:0/351332161 shutdown_connections 2023-10-14T16:41:54.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.474+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/351332161 >> v2:172.21.15.186:3300/0 conn(0x7fe7e8065ab0 0x7fe7e80cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.474+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/351332161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d220 0x7fe7e80c95e0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.474+0000 7fe7eefcb640 1 -- 172.21.15.29:0/351332161 >> 172.21.15.29:0/351332161 conn(0x7fe7e80b0a80 msgr2=0x7fe7e80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:54.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.474+0000 7fe7eefcb640 1 -- 172.21.15.29:0/351332161 shutdown_connections 2023-10-14T16:41:54.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.474+0000 7fe7eefcb640 1 -- 172.21.15.29:0/351332161 wait complete. 2023-10-14T16:41:54.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.475+0000 7fe7eefcb640 1 Processor -- start 2023-10-14T16:41:54.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.475+0000 7fe7eefcb640 1 -- start start 2023-10-14T16:41:54.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7e806d220 0x7fe7e81f6540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7e806b290 0x7fe7e806b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e806ae70 con 0x7fe7e806d600 2023-10-14T16:41:54.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e8149c50 con 0x7fe7e806d220 2023-10-14T16:41:54.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.476+0000 7fe7eefcb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81f6a80 con 0x7fe7e806b290 2023-10-14T16:41:54.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33914/0 (socket says 172.21.15.29:33914) 2023-10-14T16:41:54.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 -- 172.21.15.29:0/2049294187 learned_addr learned my addr 172.21.15.29:0/2049294187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:54.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ee7ca640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7e806b290 0x7fe7e806b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7edfc9640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7e806d220 0x7fe7e81f6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:54.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7e806b290 msgr2=0x7fe7e806b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7e806b290 0x7fe7e806b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7e806d220 msgr2=0x7fe7e81f6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7e806d220 0x7fe7e81f6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 -- 172.21.15.29:0/2049294187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e4009400 con 0x7fe7e806d600 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.477+0000 7fe7ed7c8640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800d5f0 tx=0x7fe7d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:54.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.478+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80125f0 con 0x7fe7e806d600 2023-10-14T16:41:54.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.478+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7d8012750 con 0x7fe7e806d600 2023-10-14T16:41:54.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.478+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8012b00 con 0x7fe7e806d600 2023-10-14T16:41:54.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.478+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e806a1f0 con 0x7fe7e806d600 2023-10-14T16:41:54.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.478+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e806a7b0 con 0x7fe7e806d600 2023-10-14T16:41:54.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.479+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7e804e9b0 con 0x7fe7e806d600 2023-10-14T16:41:54.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.480+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7fe7d8012c60 con 0x7fe7e806d600 2023-10-14T16:41:54.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.481+0000 7fe7de7fc640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 0x7fe7cc06d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:54.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.481+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7fe7d8089bd0 con 0x7fe7e806d600 2023-10-14T16:41:54.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.481+0000 7fe7edfc9640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 msgr2=0x7fe7cc06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:54.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.481+0000 7fe7edfc9640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 0x7fe7cc06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:41:54.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.486+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7d8054c70 con 0x7fe7e806d600 2023-10-14T16:41:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.681+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fe7e80b4f10 con 0x7fe7e806d600 2023-10-14T16:41:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.682+0000 7fe7edfc9640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 msgr2=0x7fe7cc06d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.682+0000 7fe7edfc9640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 0x7fe7cc06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:41:54.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.685+0000 7fe7de7fc640 1 -- 172.21.15.29:0/2049294187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v12)=0 v12) v1 ==== 133+0+0 (secure 0 0 0) 0x7fe7d8058290 con 0x7fe7e806d600 2023-10-14T16:41:54.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 msgr2=0x7fe7cc06d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:41:54.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 0x7fe7cc06d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 msgr2=0x7fe7e81f2890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800d5f0 tx=0x7fe7d800dab0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 shutdown_connections 2023-10-14T16:41:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7fe7cc06aec0 0x7fe7cc06d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7e806b290 0x7fe7e806b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7e806d600 0x7fe7e81f2890 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 --2- 172.21.15.29:0/2049294187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe7e806d220 0x7fe7e81f6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:54.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 >> 172.21.15.29:0/2049294187 conn(0x7fe7e80b0a80 msgr2=0x7fe7e80afe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:54.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.688+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 shutdown_connections 2023-10-14T16:41:54.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:54.689+0000 7fe7eefcb640 1 -- 172.21.15.29:0/2049294187 wait complete. 2023-10-14T16:41:54.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:54.709 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T16:41:54.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:54.762 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir0) 2023-10-14T16:41:54.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir0 1m 2023-10-14T16:41:54.812 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.811+0000 7fdc614da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:41:54.864 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.863+0000 7fdc614da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:41:54.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:54.880+0000 7f1895411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:41:54.913 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.912+0000 7fdc614da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:41:54.936 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:54.936+0000 7f1895411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:41:54.974 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:54.973+0000 7fdc614da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:41:54.991 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:54.990+0000 7f1895411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:41:55.022 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.021+0000 7fdc614da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:41:55.055 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.054+0000 7f1895411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:41:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.066+0000 7f5fbce61640 1 Processor -- start 2023-10-14T16:41:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.066+0000 7f5fbce61640 1 -- start start 2023-10-14T16:41:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fbce61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5fb80cf710 0x7f5fb80b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fbce61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb80b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fbce61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fb80d3980 con 0x7f5fb80b38e0 2023-10-14T16:41:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fbce61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fb80d3c60 con 0x7f5fb80cf710 2023-10-14T16:41:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fbce61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb80ce630 con 0x7f5fb80cfaf0 2023-10-14T16:41:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fb659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5fb80cf710 0x7f5fb80b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fb659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5fb80cf710 0x7f5fb80b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33918/0 (socket says 172.21.15.29:33918) 2023-10-14T16:41:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.068+0000 7f5fb659c640 1 -- 172.21.15.29:0/3725015815 learned_addr learned my addr 172.21.15.29:0/3725015815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:55.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.069+0000 7f5faffff640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb80b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.069+0000 7f5fb659c640 1 -- 172.21.15.29:0/3725015815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 msgr2=0x7f5fb80b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.069+0000 7f5fb659c640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb80b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:55.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.069+0000 7f5fb659c640 1 -- 172.21.15.29:0/3725015815 >> v1:172.21.15.186:6789/0 conn(0x7f5fb80b38e0 legacy=0x7f5fb80d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.069+0000 7f5fb659c640 1 -- 172.21.15.29:0/3725015815 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb80ce910 con 0x7f5fb80cf710 2023-10-14T16:41:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.070+0000 7f5fb659c640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb80b2d10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009850 tx=0x7f5fa802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54187024d074b68e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.070+0000 7f5fb5d9b640 1 -- 172.21.15.29:0/3725015815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa802e070 con 0x7f5fb80cf710 2023-10-14T16:41:55.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.070+0000 7f5fb5d9b640 1 -- 172.21.15.29:0/3725015815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa8004030 con 0x7f5fb80cf710 2023-10-14T16:41:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.070+0000 7f5fb5d9b640 1 -- 172.21.15.29:0/3725015815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8004420 con 0x7f5fb80cf710 2023-10-14T16:41:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 -- 172.21.15.29:0/3725015815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 msgr2=0x7f5fb80b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb80b2d10 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009850 tx=0x7f5fa802b840 comp rx=0 tx=0).stop 2023-10-14T16:41:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 -- 172.21.15.29:0/3725015815 shutdown_connections 2023-10-14T16:41:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb80b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 --2- 172.21.15.29:0/3725015815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb80b2d10 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 -- 172.21.15.29:0/3725015815 >> 172.21.15.29:0/3725015815 conn(0x7f5fb80d2cc0 msgr2=0x7f5fb80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 -- 172.21.15.29:0/3725015815 shutdown_connections 2023-10-14T16:41:55.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.071+0000 7f5fbce61640 1 -- 172.21.15.29:0/3725015815 wait complete. 2023-10-14T16:41:55.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.073+0000 7f5fbce61640 1 Processor -- start 2023-10-14T16:41:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.073+0000 7f5fbce61640 1 -- start start 2023-10-14T16:41:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fb80b38e0 0x7f5fb806ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb806b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb81f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb81f6040 con 0x7f5fb80cf710 2023-10-14T16:41:55.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb81f3830 con 0x7f5fb80b38e0 2023-10-14T16:41:55.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5fbce61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb81f6290 con 0x7f5fb80cfaf0 2023-10-14T16:41:55.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5faffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb806b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:55.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5faffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb806b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33924/0 (socket says 172.21.15.29:33924) 2023-10-14T16:41:55.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.074+0000 7f5faffff640 1 -- 172.21.15.29:0/64052382 learned_addr learned my addr 172.21.15.29:0/64052382 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:55.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fb80b38e0 0x7f5fb806ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:55.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5fb6d9d640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb81f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:55.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 msgr2=0x7f5fb81f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:55.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fb80cfaf0 0x7f5fb81f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:55.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fb80b38e0 msgr2=0x7f5fb806ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:55.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fb80b38e0 0x7f5fb806ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:55.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fa8009460 con 0x7f5fb80cf710 2023-10-14T16:41:55.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faffff640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb806b0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000d5f0 tx=0x7f5fa000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:55.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.075+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa00125f0 con 0x7f5fb80cf710 2023-10-14T16:41:55.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.076+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fa0012750 con 0x7f5fb80cf710 2023-10-14T16:41:55.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.076+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa0012b00 con 0x7f5fb80cf710 2023-10-14T16:41:55.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.076+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb81f6570 con 0x7f5fb80cf710 2023-10-14T16:41:55.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.076+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb81f6d70 con 0x7f5fb80cf710 2023-10-14T16:41:55.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.077+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fb804e160 con 0x7f5fb80cf710 2023-10-14T16:41:55.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.078+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 6) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f5fa0012c60 con 0x7f5fb80cf710 2023-10-14T16:41:55.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.079+0000 7f5faeffd640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:55.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.079+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 5709+0+0 (secure 0 0 0) 0x7f5fa0089b00 con 0x7f5fb80cf710 2023-10-14T16:41:55.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.079+0000 7f5fb659c640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:55.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.079+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:41:55.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.083+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5fa008be60 con 0x7f5fb80cf710 2023-10-14T16:41:55.109 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.108+0000 7f1895411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:41:55.134 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.133+0000 7fdc614da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:41:55.196 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.195+0000 7fdc614da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:41:55.226 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.226+0000 7f1895411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:41:55.244 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.243+0000 7fdc614da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:41:55.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.280+0000 7f5fb659c640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:55.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.280+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:41:55.298 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.297+0000 7f1895411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:41:55.306 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.305+0000 7fdc614da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:41:55.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.308+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fb80d1d50 con 0x7f5f9806aec0 2023-10-14T16:41:55.350 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.349+0000 7f1895411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:41:55.357 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.355+0000 7fdc614da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:41:55.410 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.409+0000 7f1895411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:41:55.463 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.462+0000 7f1895411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:41:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.681+0000 7f5fb659c640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:55.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:55.681+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:41:55.735 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.734+0000 7fdc614da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:41:55.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:55.864+0000 7f1895411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:41:55.893 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:55.892+0000 7fdc614da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:41:56.001 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.000+0000 7fdc614da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:41:56.033 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.033+0000 7f1895411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:41:56.050 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.049+0000 7fdc614da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:41:56.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.147+0000 7f1895411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:41:56.157 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.156+0000 7fdc614da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:41:56.200 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.199+0000 7f1895411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:41:56.263 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.262+0000 7fdc614da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:41:56.313 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.312+0000 7f1895411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:41:56.421 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.419+0000 7fdc614da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:41:56.425 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.424+0000 7f1895411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:41:56.469 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.468+0000 7fdc614da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:41:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:56.482+0000 7f5fb659c640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:56.482+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:41:56.587 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:56.586+0000 7fdc614da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:41:56.590 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.589+0000 7f1895411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:41:56.642 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.641+0000 7f1895411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:41:56.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:56.766+0000 7f1895411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:41:57.422 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:57.420+0000 7fdc614da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:41:57.639 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:57.638+0000 7f1895411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:41:57.745 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:57.744+0000 7fdc614da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:41:57.803 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:57.802+0000 7fdc614da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:41:57.916 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:41:57.915+0000 7fdc614da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:41:57.950 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:57.949+0000 7f1895411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:41:57.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:57.986+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 7) v1 ==== 89246+0+0 (secure 0 0 0) 0x7f5fa0054ee0 con 0x7f5fb80cf710 2023-10-14T16:41:58.014 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.013+0000 7f1895411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:41:58.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:58.084+0000 7f5fb659c640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3113675785 2023-10-14T16:41:58.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:58.084+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:41:58.132 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.131+0000 7f1895411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:41:58.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:58.151+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 8) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f5fa00702f0 con 0x7f5fb80cf710 2023-10-14T16:41:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:58.151+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 msgr2=0x7f5f9806d380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:41:58.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:58.151+0000 7f5faeffd640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/3113675785,v1:172.21.15.29:6833/3113675785] conn(0x7f5f9806aec0 0x7f5f9806d380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:58.482 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f18113a5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.482 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f18113a5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f18113a5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f18113a5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f18113a5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f1814bec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.483 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f1814bec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.484 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f1814bec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.484 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f1814bec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:58.484 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:41:58.481+0000 7f1814bec640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:41:59.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.164+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f5fa00557b0 con 0x7f5fb80cf710 2023-10-14T16:41:59.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.165+0000 7f5faeffd640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5f98027c00 0x7f5f98074d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.166+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir0", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f9800fb10 con 0x7f5f98027c00 2023-10-14T16:41:59.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.173+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5f98027c00 0x7f5f98074d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.179+0000 7f5fb659c640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5f98027c00 0x7f5f98074d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034000 tx=0x7f5fa8008030 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.388+0000 7f5faeffd640 1 -- 172.21.15.29:0/64052382 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5fa80043c0 con 0x7f5f98027c00 2023-10-14T16:41:59.389 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir0 2023-10-14T16:41:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5f98027c00 msgr2=0x7f5f98074d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5f98027c00 0x7f5f98074d80 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8034000 tx=0x7f5fa8008030 comp rx=0 tx=0).stop 2023-10-14T16:41:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 msgr2=0x7f5fb806b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 --2- 172.21.15.29:0/64052382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fb80cf710 0x7f5fb806b0c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f5fa000d5f0 tx=0x7f5fa000dab0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fb6d9d640 1 -- 172.21.15.29:0/64052382 reap_dead start 2023-10-14T16:41:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 shutdown_connections 2023-10-14T16:41:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.391+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 >> 172.21.15.29:0/64052382 conn(0x7f5fb80d2cc0 msgr2=0x7f5fb80d1330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.392+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 shutdown_connections 2023-10-14T16:41:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.392+0000 7f5fbce61640 1 -- 172.21.15.29:0/64052382 wait complete. 2023-10-14T16:41:59.415 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir0 2023-10-14T16:41:59.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:41:59.416 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir1) 2023-10-14T16:41:59.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir1 1m 2023-10-14T16:41:59.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.747+0000 7f1adca44640 1 Processor -- start 2023-10-14T16:41:59.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.747+0000 7f1adca44640 1 -- start start 2023-10-14T16:41:59.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.748+0000 7f1adca44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ad815d330 0x7f1ad81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1adca44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad80661c0 0x7f1ad80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1adca44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80d0890 con 0x7f1ad806c7d0 2023-10-14T16:41:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1adca44640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad806b1e0 con 0x7f1ad815d330 2023-10-14T16:41:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1adca44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80d05b0 con 0x7f1ad80661c0 2023-10-14T16:41:59.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ad815d330 0x7f1ad81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ad815d330 0x7f1ad81618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34058/0 (socket says 172.21.15.29:34058) 2023-10-14T16:41:59.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 -- 172.21.15.29:0/3594266127 learned_addr learned my addr 172.21.15.29:0/3594266127 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 -- 172.21.15.29:0/3594266127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad80661c0 msgr2=0x7f1ad80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:59.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 --2- 172.21.15.29:0/3594266127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad80661c0 0x7f1ad80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 -- 172.21.15.29:0/3594266127 >> v1:172.21.15.186:6789/0 conn(0x7f1ad806c7d0 legacy=0x7f1ad8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:41:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.749+0000 7f1ad659c640 1 -- 172.21.15.29:0/3594266127 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8160dd0 con 0x7f1ad815d330 2023-10-14T16:41:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.750+0000 7f1ad659c640 1 --2- 172.21.15.29:0/3594266127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad815d330 0x7f1ad81618b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009870 tx=0x7f1ac802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b11822868d037c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1ad5d9b640 1 -- 172.21.15.29:0/3594266127 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac80317c0 con 0x7f1ad815d330 2023-10-14T16:41:59.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1ad5d9b640 1 -- 172.21.15.29:0/3594266127 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8031920 con 0x7f1ad815d330 2023-10-14T16:41:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1ad5d9b640 1 -- 172.21.15.29:0/3594266127 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8031cb0 con 0x7f1ad815d330 2023-10-14T16:41:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1adca44640 1 -- 172.21.15.29:0/3594266127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad815d330 msgr2=0x7f1ad81618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1adca44640 1 --2- 172.21.15.29:0/3594266127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad815d330 0x7f1ad81618b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8009870 tx=0x7f1ac802b860 comp rx=0 tx=0).stop 2023-10-14T16:41:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.751+0000 7f1adca44640 1 -- 172.21.15.29:0/3594266127 shutdown_connections 2023-10-14T16:41:59.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.752+0000 7f1adca44640 1 --2- 172.21.15.29:0/3594266127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad80661c0 0x7f1ad80ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.752+0000 7f1adca44640 1 --2- 172.21.15.29:0/3594266127 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad815d330 0x7f1ad81618b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.752+0000 7f1adca44640 1 -- 172.21.15.29:0/3594266127 >> 172.21.15.29:0/3594266127 conn(0x7f1ad80d0ab0 msgr2=0x7f1ad80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:41:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.752+0000 7f1adca44640 1 -- 172.21.15.29:0/3594266127 shutdown_connections 2023-10-14T16:41:59.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.752+0000 7f1adca44640 1 -- 172.21.15.29:0/3594266127 wait complete. 2023-10-14T16:41:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.753+0000 7f1adca44640 1 Processor -- start 2023-10-14T16:41:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.754+0000 7f1adca44640 1 -- start start 2023-10-14T16:41:59.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.754+0000 7f1adca44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad80661c0 0x7f1ad8151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1adca44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1adca44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 0x7f1ad80c5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1adca44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b4720 con 0x7f1ad806c7d0 2023-10-14T16:41:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1adca44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b3aa0 con 0x7f1ad80661c0 2023-10-14T16:41:59.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1adca44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad80b3d80 con 0x7f1ad815d330 2023-10-14T16:41:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1acffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1acffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34062/0 (socket says 172.21.15.29:34062) 2023-10-14T16:41:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1acffff640 1 -- 172.21.15.29:0/3510858040 learned_addr learned my addr 172.21.15.29:0/3510858040 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:41:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1ad659c640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad80661c0 0x7f1ad8151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1ad6d9d640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 0x7f1ad80c5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1acffff640 1 -- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 msgr2=0x7f1ad80c5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.755+0000 7f1acffff640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 0x7f1ad80c5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1acffff640 1 -- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad80661c0 msgr2=0x7f1ad8151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:41:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1acffff640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad80661c0 0x7f1ad8151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:41:59.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1acffff640 1 -- 172.21.15.29:0/3510858040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ac80094d0 con 0x7f1ad806c7d0 2023-10-14T16:41:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1ad6d9d640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 0x7f1ad80c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:41:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1acffff640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000b3e0 tx=0x7f1ac000b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0011830 con 0x7f1ad806c7d0 2023-10-14T16:41:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac00119b0 con 0x7f1ad806c7d0 2023-10-14T16:41:59.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac0011d60 con 0x7f1ad806c7d0 2023-10-14T16:41:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad80b4060 con 0x7f1ad806c7d0 2023-10-14T16:41:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.756+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad8152480 con 0x7f1ad806c7d0 2023-10-14T16:41:59.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.757+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad80b4860 con 0x7f1ad806c7d0 2023-10-14T16:41:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.758+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1ac001a400 con 0x7f1ad806c7d0 2023-10-14T16:41:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.759+0000 7f1aceffd640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 0x7f1ab806d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:41:59.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.759+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1ac00896e0 con 0x7f1ad806c7d0 2023-10-14T16:41:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.760+0000 7f1ad659c640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 0x7f1ab806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:41:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.760+0000 7f1ad659c640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 0x7f1ab806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac803d040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:41:59.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.764+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1ad80b4860 con 0x7f1ad806c7d0 2023-10-14T16:41:59.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:41:59.981+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ad8161390 con 0x7f1ab806ad40 2023-10-14T16:42:00.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.168+0000 7f1aceffd640 1 -- 172.21.15.29:0/3510858040 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1ac8031b90 con 0x7f1ab806ad40 2023-10-14T16:42:00.170 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir1 2023-10-14T16:42:00.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 msgr2=0x7f1ab806d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 0x7f1ab806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ac8036000 tx=0x7f1ac803d040 comp rx=0 tx=0).stop 2023-10-14T16:42:00.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 msgr2=0x7f1ad80c66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f1ac000b3e0 tx=0x7f1ac000b8a0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 shutdown_connections 2023-10-14T16:42:00.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad815d330 0x7f1ad80c5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad806c7d0 0x7f1ad80c66c0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f1ab806ad40 0x7f1ab806d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 --2- 172.21.15.29:0/3510858040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad80661c0 0x7f1ad8151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.171+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 >> 172.21.15.29:0/3510858040 conn(0x7f1ad80d0ab0 msgr2=0x7f1ad815ffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:00.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.172+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 shutdown_connections 2023-10-14T16:42:00.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.172+0000 7f1adca44640 1 -- 172.21.15.29:0/3510858040 wait complete. 2023-10-14T16:42:00.194 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir1 2023-10-14T16:42:00.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:00.195 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir2) 2023-10-14T16:42:00.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir2 1m 2023-10-14T16:42:00.416 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:42:00.415+0000 7f180438b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:42:00.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:42:00.415+0000 7f180438b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:42:00.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:42:00.415+0000 7f180438b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:42:00.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:42:00.415+0000 7f180438b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:42:00.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:42:00.415+0000 7f180438b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:42:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.457+0000 7f9891e83640 1 Processor -- start 2023-10-14T16:42:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.457+0000 7f9891e83640 1 -- start start 2023-10-14T16:42:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f988c160530 0x7f988c1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c065ab0 0x7f988c160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f988c161470 0x7f988c0ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0ac890 con 0x7f988c161470 2023-10-14T16:42:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f988c0ae800 con 0x7f988c160530 2023-10-14T16:42:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.458+0000 7f9891e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0ae1f0 con 0x7f988c065ab0 2023-10-14T16:42:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.459+0000 7f988bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f988c161470 0x7f988c0ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.459+0000 7f988bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f988c161470 0x7f988c0ccd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34100/0 (socket says 172.21.15.29:34100) 2023-10-14T16:42:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.459+0000 7f988bfff640 1 -- 172.21.15.29:0/1238078607 learned_addr learned my addr 172.21.15.29:0/1238078607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.459+0000 7f988b7fe640 1 --2- 172.21.15.29:0/1238078607 >> v2:172.21.15.186:3300/0 conn(0x7f988c160530 0x7f988c1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.459+0000 7f9882ffd640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c065ab0 0x7f988c160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.460+0000 7f988bfff640 1 -- 172.21.15.29:0/1238078607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c065ab0 msgr2=0x7f988c160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.460+0000 7f988bfff640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c065ab0 0x7f988c160f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.460+0000 7f988bfff640 1 -- 172.21.15.29:0/1238078607 >> v2:172.21.15.186:3300/0 conn(0x7f988c160530 msgr2=0x7f988c1609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.460+0000 7f988bfff640 1 --2- 172.21.15.29:0/1238078607 >> v2:172.21.15.186:3300/0 conn(0x7f988c160530 0x7f988c1609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.460+0000 7f988bfff640 1 -- 172.21.15.29:0/1238078607 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f988c0acae0 con 0x7f988c161470 2023-10-14T16:42:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.461+0000 7f988bfff640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c161470 0x7f988c0ccd80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f988c0cf030 tx=0x7f988402b870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a2cab3c6cbc97f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.461+0000 7f988affd640 1 -- 172.21.15.29:0/1238078607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98840317d0 con 0x7f988c161470 2023-10-14T16:42:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f988affd640 1 -- 172.21.15.29:0/1238078607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9884031930 con 0x7f988c161470 2023-10-14T16:42:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f988affd640 1 -- 172.21.15.29:0/1238078607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9884031cc0 con 0x7f988c161470 2023-10-14T16:42:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 -- 172.21.15.29:0/1238078607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c161470 msgr2=0x7f988c0ccd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c161470 0x7f988c0ccd80 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f988c0cf030 tx=0x7f988402b870 comp rx=0 tx=0).stop 2023-10-14T16:42:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 -- 172.21.15.29:0/1238078607 shutdown_connections 2023-10-14T16:42:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c161470 0x7f988c0ccd80 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 --2- 172.21.15.29:0/1238078607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c065ab0 0x7f988c160f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 --2- 172.21.15.29:0/1238078607 >> v2:172.21.15.186:3300/0 conn(0x7f988c160530 0x7f988c1609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.462+0000 7f9891e83640 1 -- 172.21.15.29:0/1238078607 >> 172.21.15.29:0/1238078607 conn(0x7f988c06bc10 msgr2=0x7f988c06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:00.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.463+0000 7f9891e83640 1 -- 172.21.15.29:0/1238078607 shutdown_connections 2023-10-14T16:42:00.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.463+0000 7f9891e83640 1 -- 172.21.15.29:0/1238078607 wait complete. 2023-10-14T16:42:00.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.465+0000 7f9891e83640 1 Processor -- start 2023-10-14T16:42:00.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.465+0000 7f9891e83640 1 -- start start 2023-10-14T16:42:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c160530 0x7f988c0c4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f988c161470 0x7f988c0afb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0c51a0 con 0x7f988c065ab0 2023-10-14T16:42:00.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0b2450 con 0x7f988c161470 2023-10-14T16:42:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f9891e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f988c0b2730 con 0x7f988c160530 2023-10-14T16:42:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f988b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f988b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34114/0 (socket says 172.21.15.29:34114) 2023-10-14T16:42:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.466+0000 7f988b7fe640 1 -- 172.21.15.29:0/4175921593 learned_addr learned my addr 172.21.15.29:0/4175921593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:00.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988bfff640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f988c161470 0x7f988c0afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f9882ffd640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c160530 0x7f988c0c4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 -- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c160530 msgr2=0x7f988c0c4c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c160530 0x7f988c0c4c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 -- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f988c161470 msgr2=0x7f988c0afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f988c161470 0x7f988c0afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 -- 172.21.15.29:0/4175921593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98840094d0 con 0x7f988c065ab0 2023-10-14T16:42:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.467+0000 7f988b7fe640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f987c00efc0 tx=0x7f987c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.468+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c014480 con 0x7f988c065ab0 2023-10-14T16:42:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.468+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f987c00f040 con 0x7f988c065ab0 2023-10-14T16:42:00.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.468+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f987c014800 con 0x7f988c065ab0 2023-10-14T16:42:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.468+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f988c0b2a10 con 0x7f988c065ab0 2023-10-14T16:42:00.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.468+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f988c0c47e0 con 0x7f988c065ab0 2023-10-14T16:42:00.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.469+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f988c04e160 con 0x7f988c065ab0 2023-10-14T16:42:00.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.470+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 9) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f987c014960 con 0x7f988c065ab0 2023-10-14T16:42:00.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.471+0000 7f9888ff9640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 0x7f986c06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:00.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.471+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f987c08a090 con 0x7f988c065ab0 2023-10-14T16:42:00.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.472+0000 7f9882ffd640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 0x7f986c06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:00.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.473+0000 7f9882ffd640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 0x7f986c06d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f988c0c4380 tx=0x7f9874008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:00.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.476+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f987c0552b0 con 0x7f988c065ab0 2023-10-14T16:42:00.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.704+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir2", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f988c0ae490 con 0x7f986c06ad10 2023-10-14T16:42:00.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.753+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f987c055680 con 0x7f988c065ab0 2023-10-14T16:42:00.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.884+0000 7f9888ff9640 1 -- 172.21.15.29:0/4175921593 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f987400b640 con 0x7f986c06ad10 2023-10-14T16:42:00.885 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir2 2023-10-14T16:42:00.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 msgr2=0x7f986c06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 0x7f986c06d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f988c0c4380 tx=0x7f9874008040 comp rx=0 tx=0).stop 2023-10-14T16:42:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 msgr2=0x7f988c150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f987c00efc0 tx=0x7f987c00c490 comp rx=0 tx=0).stop 2023-10-14T16:42:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 shutdown_connections 2023-10-14T16:42:00.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f986c06ad10 0x7f986c06d1d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f988c161470 0x7f988c0afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f988c160530 0x7f988c0c4c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 --2- 172.21.15.29:0/4175921593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f988c065ab0 0x7f988c150660 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:00.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.887+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 >> 172.21.15.29:0/4175921593 conn(0x7f988c06bc10 msgr2=0x7f988c15e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.888+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 shutdown_connections 2023-10-14T16:42:00.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:00.888+0000 7f9891e83640 1 -- 172.21.15.29:0/4175921593 wait complete. 2023-10-14T16:42:00.910 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir2 2023-10-14T16:42:00.911 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:00.911 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir3) 2023-10-14T16:42:00.972 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir3 1m 2023-10-14T16:42:01.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.200+0000 7fd9ea4ad640 1 Processor -- start 2023-10-14T16:42:01.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.200+0000 7fd9ea4ad640 1 -- start start 2023-10-14T16:42:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.201+0000 7fd9ea4ad640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 0x7fd9e40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.201+0000 7fd9ea4ad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9e40d0be0 0x7fd9e40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.201+0000 7fd9ea4ad640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e415d150 con 0x7fd9e415cd70 2023-10-14T16:42:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.201+0000 7fd9ea4ad640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40d1180 con 0x7fd9e40d0be0 2023-10-14T16:42:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.201+0000 7fd9ea4ad640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e40d1460 con 0x7fd9e40b8480 2023-10-14T16:42:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e94ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9e415cd70 0x7fd9e40d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35108/0 (socket says 172.21.15.29:35108) 2023-10-14T16:42:01.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e94ab640 1 -- 172.21.15.29:0/2145993180 learned_addr learned my addr 172.21.15.29:0/2145993180 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e9cac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9e40d0be0 0x7fd9e40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e8caa640 1 --2- 172.21.15.29:0/2145993180 >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 0x7fd9e40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e9cac640 1 -- 172.21.15.29:0/2145993180 >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 msgr2=0x7fd9e40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e9cac640 1 --2- 172.21.15.29:0/2145993180 >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 0x7fd9e40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e9cac640 1 -- 172.21.15.29:0/2145993180 >> v1:172.21.15.29:6789/0 conn(0x7fd9e415cd70 legacy=0x7fd9e40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9e9cac640 1 -- 172.21.15.29:0/2145993180 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e40d1740 con 0x7fd9e40d0be0 2023-10-14T16:42:01.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.202+0000 7fd9dbfff640 1 -- 172.21.15.29:0/2145993180 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568308247 0 0) 0x7fd9e415d150 con 0x7fd9e415cd70 2023-10-14T16:42:01.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.203+0000 7fd9e8caa640 1 --2- 172.21.15.29:0/2145993180 >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 0x7fd9e40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:01.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9e9cac640 1 --2- 172.21.15.29:0/2145993180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cdd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4066a00 tx=0x7fd9dc02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af78cb1888daffc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9dbfff640 1 -- 172.21.15.29:0/2145993180 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc02cbe0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9dbfff640 1 -- 172.21.15.29:0/2145993180 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc02cd40 con 0x7fd9e40d0be0 2023-10-14T16:42:01.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9dbfff640 1 -- 172.21.15.29:0/2145993180 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc034660 con 0x7fd9e40d0be0 2023-10-14T16:42:01.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/2145993180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 msgr2=0x7fd9e40cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.204+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/2145993180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cdd00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4066a00 tx=0x7fd9dc02be30 comp rx=0 tx=0).stop 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/2145993180 shutdown_connections 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/2145993180 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cdd00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/2145993180 >> v2:172.21.15.186:3300/0 conn(0x7fd9e40b8480 0x7fd9e40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/2145993180 >> 172.21.15.29:0/2145993180 conn(0x7fd9e40cf570 msgr2=0x7fd9e40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:01.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/2145993180 shutdown_connections 2023-10-14T16:42:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.205+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/2145993180 wait complete. 2023-10-14T16:42:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.207+0000 7fd9ea4ad640 1 Processor -- start 2023-10-14T16:42:01.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.207+0000 7fd9ea4ad640 1 -- start start 2023-10-14T16:42:01.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 0x7fd9e40b41e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9e415cd70 0x7fd9e40cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40cb470 con 0x7fd9e40d0be0 2023-10-14T16:42:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e40cca60 con 0x7fd9e40b8480 2023-10-14T16:42:01.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9ea4ad640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e4153690 con 0x7fd9e415cd70 2023-10-14T16:42:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9e8caa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9e8caa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58818/0 (socket says 172.21.15.29:58818) 2023-10-14T16:42:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.208+0000 7fd9e8caa640 1 -- 172.21.15.29:0/1912517570 learned_addr learned my addr 172.21.15.29:0/1912517570 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:01.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e94ab640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 0x7fd9e40b41e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e9cac640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9e415cd70 0x7fd9e40cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 -- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9e415cd70 msgr2=0x7fd9e40cc520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9e415cd70 0x7fd9e40cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 -- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 msgr2=0x7fd9e40b41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 0x7fd9e40b41e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 -- 172.21.15.29:0/1912517570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9dc009a70 con 0x7fd9e40d0be0 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e94ab640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 0x7fd9e40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9e8caa640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c7f0 tx=0x7fd9d400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4007820 con 0x7fd9e40d0be0 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d4007980 con 0x7fd9e40d0be0 2023-10-14T16:42:01.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9e40cabd0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.209+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d4007cf0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.210+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e40cb190 con 0x7fd9e40d0be0 2023-10-14T16:42:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.211+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e40ca950 con 0x7fd9e40d0be0 2023-10-14T16:42:01.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.212+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd9d40113f0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.213+0000 7fd9d9ffb640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 0x7fd9c806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.213+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd9d4089ca0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.213+0000 7fd9e94ab640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 0x7fd9c806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.214+0000 7fd9e94ab640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 0x7fd9c806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0000f60 tx=0x7fd9e0009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.217+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9d4054aa0 con 0x7fd9e40d0be0 2023-10-14T16:42:01.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.429+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir3", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd9e40d1d00 con 0x7fd9c806afe0 2023-10-14T16:42:01.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.610+0000 7fd9d9ffb640 1 -- 172.21.15.29:0/1912517570 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd9e0001d50 con 0x7fd9c806afe0 2023-10-14T16:42:01.612 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir3 2023-10-14T16:42:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 msgr2=0x7fd9c806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 0x7fd9c806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0000f60 tx=0x7fd9e0009040 comp rx=0 tx=0).stop 2023-10-14T16:42:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 msgr2=0x7fd9e40cbfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7fd9d400c7f0 tx=0x7fd9d400ccb0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 shutdown_connections 2023-10-14T16:42:01.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9e415cd70 0x7fd9e40cc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9e40d0be0 0x7fd9e40cbfe0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd9c806afe0 0x7fd9c806d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.613+0000 7fd9ea4ad640 1 --2- 172.21.15.29:0/1912517570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9e40b8480 0x7fd9e40b41e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.614+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 >> 172.21.15.29:0/1912517570 conn(0x7fd9e40cf570 msgr2=0x7fd9e40cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.614+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 shutdown_connections 2023-10-14T16:42:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.614+0000 7fd9ea4ad640 1 -- 172.21.15.29:0/1912517570 wait complete. 2023-10-14T16:42:01.637 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir3 2023-10-14T16:42:01.638 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:01.638 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir4) 2023-10-14T16:42:01.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir4 1m 2023-10-14T16:42:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.928+0000 7fb85d440640 1 Processor -- start 2023-10-14T16:42:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.928+0000 7fb85d440640 1 -- start start 2023-10-14T16:42:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.929+0000 7fb85d440640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb85815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.929+0000 7fb85d440640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb85806c810 0x7fb8580708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.929+0000 7fb85d440640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb858070df0 con 0x7fb858070450 2023-10-14T16:42:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.929+0000 7fb85d440640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb85815e490 con 0x7fb85806c810 2023-10-14T16:42:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.929+0000 7fb85d440640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85815aef0 con 0x7fb858159950 2023-10-14T16:42:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb85806c810 0x7fb8580708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb85806c810 0x7fb8580708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58834/0 (socket says 172.21.15.29:58834) 2023-10-14T16:42:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 -- 172.21.15.29:0/467372806 learned_addr learned my addr 172.21.15.29:0/467372806 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb857fff640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb85815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 -- 172.21.15.29:0/467372806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 msgr2=0x7fb85815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb85815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 -- 172.21.15.29:0/467372806 >> v1:172.21.15.186:6789/0 conn(0x7fb858070450 legacy=0x7fb85806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb85cc3f640 1 -- 172.21.15.29:0/467372806 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85806cc80 con 0x7fb85806c810 2023-10-14T16:42:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.930+0000 7fb857fff640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb85815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.932+0000 7fb85cc3f640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb85806c810 0x7fb8580708b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb850009e50 tx=0x7fb85002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a5b549e7ab44010 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.932+0000 7fb856ffd640 1 -- 172.21.15.29:0/467372806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb85002c9f0 con 0x7fb85806c810 2023-10-14T16:42:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.932+0000 7fb856ffd640 1 -- 172.21.15.29:0/467372806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85002cb50 con 0x7fb85806c810 2023-10-14T16:42:01.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.932+0000 7fb856ffd640 1 -- 172.21.15.29:0/467372806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8500343f0 con 0x7fb85806c810 2023-10-14T16:42:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 -- 172.21.15.29:0/467372806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb85806c810 msgr2=0x7fb8580708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb85806c810 0x7fb8580708b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7fb850009e50 tx=0x7fb85002be50 comp rx=0 tx=0).stop 2023-10-14T16:42:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 -- 172.21.15.29:0/467372806 shutdown_connections 2023-10-14T16:42:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb85806c810 0x7fb8580708b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 --2- 172.21.15.29:0/467372806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb85815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 -- 172.21.15.29:0/467372806 >> 172.21.15.29:0/467372806 conn(0x7fb8580ce6a0 msgr2=0x7fb8580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 -- 172.21.15.29:0/467372806 shutdown_connections 2023-10-14T16:42:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.933+0000 7fb85d440640 1 -- 172.21.15.29:0/467372806 wait complete. 2023-10-14T16:42:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.935+0000 7fb85d440640 1 Processor -- start 2023-10-14T16:42:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.935+0000 7fb85d440640 1 -- start start 2023-10-14T16:42:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 0x7fb858156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb8580acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8581f3050 con 0x7fb858070450 2023-10-14T16:42:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8581f23d0 con 0x7fb85806c810 2023-10-14T16:42:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.936+0000 7fb85d440640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8581f26b0 con 0x7fb858159950 2023-10-14T16:42:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58850/0 (socket says 172.21.15.29:58850) 2023-10-14T16:42:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 -- 172.21.15.29:0/2424927433 learned_addr learned my addr 172.21.15.29:0/2424927433 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb85cc3f640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb8580acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb857fff640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 0x7fb858156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 -- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 msgr2=0x7fb8580acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb8580acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 -- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 msgr2=0x7fb858156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 0x7fb858156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 -- 172.21.15.29:0/2424927433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb850009b30 con 0x7fb858070450 2023-10-14T16:42:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb85cc3f640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb8580acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb857fff640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 0x7fb858156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.937+0000 7fb8577fe640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00b3f0 tx=0x7fb84c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.938+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c011830 con 0x7fb858070450 2023-10-14T16:42:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.938+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb84c011990 con 0x7fb858070450 2023-10-14T16:42:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.938+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb84c011d40 con 0x7fb858070450 2023-10-14T16:42:01.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.938+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8581f2990 con 0x7fb858070450 2023-10-14T16:42:01.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.938+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb858157300 con 0x7fb858070450 2023-10-14T16:42:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.939+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8581f3190 con 0x7fb858070450 2023-10-14T16:42:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.940+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fb84c01a400 con 0x7fb858070450 2023-10-14T16:42:01.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.941+0000 7fb854ff9640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 0x7fb84406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.941+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb84c089700 con 0x7fb858070450 2023-10-14T16:42:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.941+0000 7fb857fff640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 0x7fb84406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.942+0000 7fb857fff640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 0x7fb84406d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb848006fd0 tx=0x7fb848008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:01.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:01.946+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb84c054590 con 0x7fb858070450 2023-10-14T16:42:02.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.164+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir4", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb85815dbf0 con 0x7fb84406afe0 2023-10-14T16:42:02.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.353+0000 7fb854ff9640 1 -- 172.21.15.29:0/2424927433 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fb84800c040 con 0x7fb84406afe0 2023-10-14T16:42:02.355 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir4 2023-10-14T16:42:02.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 msgr2=0x7fb84406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:02.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 0x7fb84406d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb848006fd0 tx=0x7fb848008040 comp rx=0 tx=0).stop 2023-10-14T16:42:02.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 msgr2=0x7fb8580ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7fb84c00b3f0 tx=0x7fb84c00b8b0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 shutdown_connections 2023-10-14T16:42:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb858159950 0x7fb8580acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb858070450 0x7fb8580ac890 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb84406afe0 0x7fb84406d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 --2- 172.21.15.29:0/2424927433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb85806c810 0x7fb858156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 >> 172.21.15.29:0/2424927433 conn(0x7fb8580ce6a0 msgr2=0x7fb85815b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 shutdown_connections 2023-10-14T16:42:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.357+0000 7fb85d440640 1 -- 172.21.15.29:0/2424927433 wait complete. 2023-10-14T16:42:02.380 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir4 2023-10-14T16:42:02.380 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:02.381 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir5) 2023-10-14T16:42:02.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir5 1m 2023-10-14T16:42:02.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.670+0000 7f7ce593e640 1 Processor -- start 2023-10-14T16:42:02.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.671+0000 7f7ce593e640 1 -- start start 2023-10-14T16:42:02.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.672+0000 7f7ce593e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ce0070830 0x7f7ce006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.672+0000 7f7ce593e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070c10 0x7f7ce00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.672+0000 7f7ce593e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ce0161740 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.672+0000 7f7ce593e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ce00cdbd0 con 0x7f7ce0070830 2023-10-14T16:42:02.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.672+0000 7f7ce593e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce006b0f0 con 0x7f7ce0070c10 2023-10-14T16:42:02.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ce0070830 0x7f7ce006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:02.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7ce0070830 0x7f7ce006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58858/0 (socket says 172.21.15.29:58858) 2023-10-14T16:42:02.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 -- 172.21.15.29:0/3283116416 learned_addr learned my addr 172.21.15.29:0/3283116416 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:02.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 -- 172.21.15.29:0/3283116416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070c10 msgr2=0x7f7ce00cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:02.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 --2- 172.21.15.29:0/3283116416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070c10 0x7f7ce00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 -- 172.21.15.29:0/3283116416 >> v1:172.21.15.186:6789/0 conn(0x7f7ce00cf5f0 legacy=0x7f7ce006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:02.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.673+0000 7f7cdeffd640 1 -- 172.21.15.29:0/3283116416 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ce01619b0 con 0x7f7ce0070830 2023-10-14T16:42:02.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.675+0000 7f7cdeffd640 1 --2- 172.21.15.29:0/3283116416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce0070830 0x7f7ce006d3c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009870 tx=0x7f7cc802b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9659cd676bd795 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:02.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.675+0000 7f7cddffb640 1 -- 172.21.15.29:0/3283116416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8031740 con 0x7f7ce0070830 2023-10-14T16:42:02.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.675+0000 7f7cddffb640 1 -- 172.21.15.29:0/3283116416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc80318a0 con 0x7f7ce0070830 2023-10-14T16:42:02.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.675+0000 7f7cddffb640 1 -- 172.21.15.29:0/3283116416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8031c30 con 0x7f7ce0070830 2023-10-14T16:42:02.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 -- 172.21.15.29:0/3283116416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce0070830 msgr2=0x7f7ce006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 --2- 172.21.15.29:0/3283116416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce0070830 0x7f7ce006d3c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009870 tx=0x7f7cc802b7e0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 -- 172.21.15.29:0/3283116416 shutdown_connections 2023-10-14T16:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 --2- 172.21.15.29:0/3283116416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070c10 0x7f7ce00cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 --2- 172.21.15.29:0/3283116416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce0070830 0x7f7ce006d3c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 -- 172.21.15.29:0/3283116416 >> 172.21.15.29:0/3283116416 conn(0x7f7ce00cfac0 msgr2=0x7f7ce00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 -- 172.21.15.29:0/3283116416 shutdown_connections 2023-10-14T16:42:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.676+0000 7f7ce593e640 1 -- 172.21.15.29:0/3283116416 wait complete. 2023-10-14T16:42:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.678+0000 7f7ce593e640 1 Processor -- start 2023-10-14T16:42:02.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.678+0000 7f7ce593e640 1 -- start start 2023-10-14T16:42:02.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070830 0x7f7ce014e260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ce0070c10 0x7f7ce01f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01f6d20 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce01f7000 con 0x7f7ce0070c10 2023-10-14T16:42:02.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7ce593e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ce00c7fa0 con 0x7f7ce0070830 2023-10-14T16:42:02.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7cdf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:02.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7cdf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58860/0 (socket says 172.21.15.29:58860) 2023-10-14T16:42:02.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.679+0000 7f7cdf7fe640 1 -- 172.21.15.29:0/1727161187 learned_addr learned my addr 172.21.15.29:0/1727161187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:02.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cde7fc640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ce0070c10 0x7f7ce01f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:02.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdeffd640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070830 0x7f7ce014e260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 -- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070830 msgr2=0x7f7ce014e260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070830 0x7f7ce014e260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 -- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ce0070c10 msgr2=0x7f7ce01f62d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ce0070c10 0x7f7ce01f62d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:02.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 -- 172.21.15.29:0/1727161187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc80094d0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cdf7fe640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800b970 tx=0x7f7cd800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.680+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd800cb40 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.681+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cd800ccc0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.681+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cd8014680 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.681+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ce00c8280 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.681+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ce014e7a0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.682+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ce00c84c0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.683+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 10) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f7cd801d3b0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.684+0000 7f7cd7fff640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 0x7f7cc006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:02.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.684+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f7cd8089910 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.684+0000 7f7cdeffd640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 0x7f7cc006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.685+0000 7f7cdeffd640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 0x7f7cc006d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8035000 tx=0x7f7cc8008030 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:02.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.689+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7ce00c84c0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.765+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f7cd8054ae0 con 0x7f7ce00cf5f0 2023-10-14T16:42:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:02.913+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir5", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7ce00c7e20 con 0x7f7cc006afe0 2023-10-14T16:42:03.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.098+0000 7f7cd7fff640 1 -- 172.21.15.29:0/1727161187 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f7cc8041070 con 0x7f7cc006afe0 2023-10-14T16:42:03.099 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir5 2023-10-14T16:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 msgr2=0x7f7cc006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 0x7f7cc006d4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8035000 tx=0x7f7cc8008030 comp rx=0 tx=0).stop 2023-10-14T16:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 msgr2=0x7f7ce00afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f7cd800b970 tx=0x7f7cd800be30 comp rx=0 tx=0).stop 2023-10-14T16:42:03.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 shutdown_connections 2023-10-14T16:42:03.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7ce00cf5f0 0x7f7ce00afda0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f7cc006afe0 0x7f7cc006d4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7ce0070c10 0x7f7ce01f62d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 --2- 172.21.15.29:0/1727161187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7ce0070830 0x7f7ce014e260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 >> 172.21.15.29:0/1727161187 conn(0x7f7ce00cfac0 msgr2=0x7f7ce00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:03.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 shutdown_connections 2023-10-14T16:42:03.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.102+0000 7f7ce593e640 1 -- 172.21.15.29:0/1727161187 wait complete. 2023-10-14T16:42:03.129 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir5 2023-10-14T16:42:03.129 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:03.129 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir6) 2023-10-14T16:42:03.186 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir6 1m 2023-10-14T16:42:03.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.411+0000 7f0d00a2e640 1 Processor -- start 2023-10-14T16:42:03.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.411+0000 7f0d00a2e640 1 -- start start 2023-10-14T16:42:03.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.412+0000 7f0d00a2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.412+0000 7f0d00a2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.412+0000 7f0d00a2e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cfc15d150 con 0x7f0cfc0d0be0 2023-10-14T16:42:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.412+0000 7f0d00a2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cfc0d1180 con 0x7f0cfc0b8480 2023-10-14T16:42:03.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.412+0000 7f0d00a2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0d1460 con 0x7f0cfc15cd70 2023-10-14T16:42:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58872/0 (socket says 172.21.15.29:58872) 2023-10-14T16:42:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 -- 172.21.15.29:0/2912049716 learned_addr learned my addr 172.21.15.29:0/2912049716 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:03.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 -- 172.21.15.29:0/2912049716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 msgr2=0x7f0cfc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 --2- 172.21.15.29:0/2912049716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 -- 172.21.15.29:0/2912049716 >> v1:172.21.15.186:6789/0 conn(0x7f0cfc0d0be0 legacy=0x7f0cfc0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.413+0000 7f0cfaffd640 1 -- 172.21.15.29:0/2912049716 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc0d1740 con 0x7f0cfc0b8480 2023-10-14T16:42:03.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.414+0000 7f0cfaffd640 1 --2- 172.21.15.29:0/2912049716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8176313de75f049d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.415+0000 7f0cfa7fc640 1 -- 172.21.15.29:0/2912049716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4031640 con 0x7f0cfc0b8480 2023-10-14T16:42:03.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.415+0000 7f0cfa7fc640 1 -- 172.21.15.29:0/2912049716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ce40317a0 con 0x7f0cfc0b8480 2023-10-14T16:42:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.415+0000 7f0cfa7fc640 1 -- 172.21.15.29:0/2912049716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ce4031b30 con 0x7f0cfc0b8480 2023-10-14T16:42:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.415+0000 7f0d00a2e640 1 -- 172.21.15.29:0/2912049716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 msgr2=0x7f0cfc0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.415+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/2912049716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4009870 tx=0x7f0ce402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 -- 172.21.15.29:0/2912049716 shutdown_connections 2023-10-14T16:42:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/2912049716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc0cd7c0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/2912049716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 -- 172.21.15.29:0/2912049716 >> 172.21.15.29:0/2912049716 conn(0x7f0cfc0cf570 msgr2=0x7f0cfc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 -- 172.21.15.29:0/2912049716 shutdown_connections 2023-10-14T16:42:03.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.416+0000 7f0d00a2e640 1 -- 172.21.15.29:0/2912049716 wait complete. 2023-10-14T16:42:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.418+0000 7f0d00a2e640 1 Processor -- start 2023-10-14T16:42:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.418+0000 7f0d00a2e640 1 -- start start 2023-10-14T16:42:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cfc0d0be0 0x7f0cfc0caee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc14e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0c6f90 con 0x7f0cfc0b8480 2023-10-14T16:42:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc15ec30 con 0x7f0cfc0d0be0 2023-10-14T16:42:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0d00a2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0c65f0 con 0x7f0cfc15cd70 2023-10-14T16:42:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0cfb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0cfb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58888/0 (socket says 172.21.15.29:58888) 2023-10-14T16:42:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0cfb7fe640 1 -- 172.21.15.29:0/3520410839 learned_addr learned my addr 172.21.15.29:0/3520410839 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.419+0000 7f0cfaffd640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cfc0d0be0 0x7f0cfc0caee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfbfff640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc14e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:03.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 -- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 msgr2=0x7f0cfc14e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc14e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 -- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cfc0d0be0 msgr2=0x7f0cfc0caee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cfc0d0be0 0x7f0cfc0caee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 -- 172.21.15.29:0/3520410839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce40094d0 con 0x7f0cfc0b8480 2023-10-14T16:42:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfbfff640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc14e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cfb7fe640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b4a0 tx=0x7f0cec00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec010610 con 0x7f0cfc0b8480 2023-10-14T16:42:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cec010770 con 0x7f0cfc0b8480 2023-10-14T16:42:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cec010b20 con 0x7f0cfc0b8480 2023-10-14T16:42:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.420+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc0c68d0 con 0x7f0cfc0b8480 2023-10-14T16:42:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.421+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc15ff20 con 0x7f0cfc0b8480 2023-10-14T16:42:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.422+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc160110 con 0x7f0cfc0b8480 2023-10-14T16:42:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.423+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f0cec010c80 con 0x7f0cfc0b8480 2023-10-14T16:42:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.424+0000 7f0cf3fff640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 0x7f0cdc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.424+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f0cec0897e0 con 0x7f0cfc0b8480 2023-10-14T16:42:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.424+0000 7f0cfaffd640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 0x7f0cdc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.425+0000 7f0cfaffd640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 0x7f0cdc06d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4036000 tx=0x7f0ce403c040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:03.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.428+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0cec054420 con 0x7f0cfc0b8480 2023-10-14T16:42:03.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.647+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir6", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cfc0d1d00 con 0x7f0cdc06b100 2023-10-14T16:42:03.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.830+0000 7f0cf3fff640 1 -- 172.21.15.29:0/3520410839 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f0ce40315e0 con 0x7f0cdc06b100 2023-10-14T16:42:03.832 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir6 2023-10-14T16:42:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.834+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 msgr2=0x7f0cdc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.834+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 0x7f0cdc06d5c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4036000 tx=0x7f0ce403c040 comp rx=0 tx=0).stop 2023-10-14T16:42:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.834+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 msgr2=0x7f0cfc15f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.834+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f0cec00b4a0 tx=0x7f0cec00b960 comp rx=0 tx=0).stop 2023-10-14T16:42:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 shutdown_connections 2023-10-14T16:42:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0cfc15cd70 0x7f0cfc14e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f0cdc06b100 0x7f0cdc06d5c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0cfc0d0be0 0x7f0cfc0caee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 --2- 172.21.15.29:0/3520410839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0cfc0b8480 0x7f0cfc15f9e0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:03.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 >> 172.21.15.29:0/3520410839 conn(0x7f0cfc0cf570 msgr2=0x7f0cfc0b7b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:03.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 shutdown_connections 2023-10-14T16:42:03.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:03.835+0000 7f0d00a2e640 1 -- 172.21.15.29:0/3520410839 wait complete. 2023-10-14T16:42:03.856 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir6 2023-10-14T16:42:03.857 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:03.857 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir7) 2023-10-14T16:42:03.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir7 1m 2023-10-14T16:42:04.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.144+0000 7f9f2f59e640 1 Processor -- start 2023-10-14T16:42:04.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.145+0000 7f9f2f59e640 1 -- start start 2023-10-14T16:42:04.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.146+0000 7f9f2f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 0x7f9f300cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.146+0000 7f9f2f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3015b9d0 con 0x7f9f300ccfb0 2023-10-14T16:42:04.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.146+0000 7f9f2f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f3015bcb0 con 0x7f9f3006c040 2023-10-14T16:42:04.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.146+0000 7f9f2f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015bf90 con 0x7f9f300ad970 2023-10-14T16:42:04.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9f300ccfb0 0x7f9f300cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35118/0 (socket says 172.21.15.29:35118) 2023-10-14T16:42:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2ed9d640 1 -- 172.21.15.29:0/3362534750 learned_addr learned my addr 172.21.15.29:0/3362534750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f27fff640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 0x7f9f300cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2999033699 0 0) 0x7f9f3015b9d0 con 0x7f9f300ccfb0 2023-10-14T16:42:04.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c003520 con 0x7f9f300ccfb0 2023-10-14T16:42:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4032930484 0 0) 0x7f9f1c003520 con 0x7f9f300ccfb0 2023-10-14T16:42:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 msgr2=0x7f9f300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:04.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 0x7f9f300cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 >> v1:172.21.15.186:6789/0 conn(0x7f9f3006c040 legacy=0x7f9f3006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300cc270 con 0x7f9f300ccfb0 2023-10-14T16:42:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9f280019f0 con 0x7f9f300ccfb0 2023-10-14T16:42:04.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.147+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 >> v1:172.21.15.29:6789/0 conn(0x7f9f300ccfb0 legacy=0x7f9f300cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c0039a0 0x7f9f1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0047c0 0x7f9f1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 0x7f9f1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3015bf90 con 0x7f9f1c0039a0 2023-10-14T16:42:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c00ba80 con 0x7f9f1c0047c0 2023-10-14T16:42:04.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f1c00bc40 con 0x7f9f300ad970 2023-10-14T16:42:04.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c0039a0 0x7f9f1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 -- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 msgr2=0x7f9f1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:04.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f300ad970 0x7f9f1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 -- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0047c0 msgr2=0x7f9f1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:04.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0047c0 0x7f9f1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.148+0000 7f9f2e59c640 1 -- 172.21.15.29:0/3362534750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1c003520 con 0x7f9f1c0039a0 2023-10-14T16:42:04.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.149+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c0039a0 0x7f9f1c003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9f20001750 tx=0x7f9f1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a374dbf98c5af77a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200052d0 con 0x7f9f1c0039a0 2023-10-14T16:42:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f2002e050 con 0x7f9f1c0039a0 2023-10-14T16:42:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2dd9b640 1 -- 172.21.15.29:0/3362534750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f200035f0 con 0x7f9f1c0039a0 2023-10-14T16:42:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3362534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c0039a0 msgr2=0x7f9f1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3362534750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f1c0039a0 0x7f9f1c003dc0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f9f20001750 tx=0x7f9f1c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2ed9d640 1 -- 172.21.15.29:0/3362534750 reap_dead start 2023-10-14T16:42:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3362534750 shutdown_connections 2023-10-14T16:42:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3362534750 >> 172.21.15.29:0/3362534750 conn(0x7f9f300ce6a0 msgr2=0x7f9f300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3362534750 shutdown_connections 2023-10-14T16:42:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.150+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3362534750 wait complete. 2023-10-14T16:42:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.152+0000 7f9f2f59e640 1 Processor -- start 2023-10-14T16:42:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.152+0000 7f9f2f59e640 1 -- start start 2023-10-14T16:42:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.153+0000 7f9f2f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 0x7f9f301539d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.153+0000 7f9f2f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.153+0000 7f9f2f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f301534a0 0x7f9f300c74c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.153+0000 7f9f2f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f30155330 con 0x7f9f300ae100 2023-10-14T16:42:04.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f2f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f301543d0 con 0x7f9f1c0039a0 2023-10-14T16:42:04.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f2f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f301546b0 con 0x7f9f301534a0 2023-10-14T16:42:04.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58898/0 (socket says 172.21.15.29:58898) 2023-10-14T16:42:04.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 -- 172.21.15.29:0/3716324434 learned_addr learned my addr 172.21.15.29:0/3716324434 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:04.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 0x7f9f301539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 -- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f301534a0 msgr2=0x7f9f300c74c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f301534a0 0x7f9f300c74c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 -- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 msgr2=0x7f9f301539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 0x7f9f301539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 -- 172.21.15.29:0/3716324434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f300cdcd0 con 0x7f9f300ae100 2023-10-14T16:42:04.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 0x7f9f301539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:04.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.154+0000 7f9f27fff640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800e7a0 tx=0x7f9f1800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.155+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1800eeb0 con 0x7f9f300ae100 2023-10-14T16:42:04.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.155+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f1800c940 con 0x7f9f300ae100 2023-10-14T16:42:04.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.155+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f30154990 con 0x7f9f300ae100 2023-10-14T16:42:04.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.155+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f1800cd10 con 0x7f9f300ae100 2023-10-14T16:42:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.155+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f30154f50 con 0x7f9f300ae100 2023-10-14T16:42:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.156+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f3006b330 con 0x7f9f300ae100 2023-10-14T16:42:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.157+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9f18013470 con 0x7f9f300ae100 2023-10-14T16:42:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.158+0000 7f9f26ffd640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 0x7f9f1006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.158+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9f18089b70 con 0x7f9f300ae100 2023-10-14T16:42:04.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.158+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 0x7f9f1006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.159+0000 7f9f2e59c640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 0x7f9f1006d5f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f20033000 tx=0x7f9f20009030 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.163+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f180547a0 con 0x7f9f300ae100 2023-10-14T16:42:04.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.383+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir7", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9f300cdb50 con 0x7f9f1006b130 2023-10-14T16:42:04.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.576+0000 7f9f26ffd640 1 -- 172.21.15.29:0/3716324434 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f9f20005430 con 0x7f9f1006b130 2023-10-14T16:42:04.577 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir7 2023-10-14T16:42:04.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 msgr2=0x7f9f1006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 0x7f9f1006d5f0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9f20033000 tx=0x7f9f20009030 comp rx=0 tx=0).stop 2023-10-14T16:42:04.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 msgr2=0x7f9f30152c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f9f1800e7a0 tx=0x7f9f1800ec60 comp rx=0 tx=0).stop 2023-10-14T16:42:04.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 shutdown_connections 2023-10-14T16:42:04.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f301534a0 0x7f9f300c74c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f300ae100 0x7f9f30152c60 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9f1006b130 0x7f9f1006d5f0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.580+0000 7f9f2f59e640 1 --2- 172.21.15.29:0/3716324434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f1c0039a0 0x7f9f301539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.581+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 >> 172.21.15.29:0/3716324434 conn(0x7f9f300ce6a0 msgr2=0x7f9f300b3760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.581+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 shutdown_connections 2023-10-14T16:42:04.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.581+0000 7f9f2f59e640 1 -- 172.21.15.29:0/3716324434 wait complete. 2023-10-14T16:42:04.604 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir7 2023-10-14T16:42:04.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:04.605 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir8) 2023-10-14T16:42:04.660 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir8 1m 2023-10-14T16:42:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.888+0000 7fc570a61640 1 Processor -- start 2023-10-14T16:42:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.888+0000 7fc570a61640 1 -- start start 2023-10-14T16:42:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc570a61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c0cf5f0 0x7fc56c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc570a61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc56c070c10 0x7fc56c0cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc570a61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc56c161740 con 0x7fc56c070830 2023-10-14T16:42:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc570a61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56c0cdbd0 con 0x7fc56c070c10 2023-10-14T16:42:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc570a61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c06b0f0 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc56ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc56c070c10 0x7fc56c0cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc56a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc56c070830 0x7fc56c06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35124/0 (socket says 172.21.15.29:35124) 2023-10-14T16:42:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc56a59c640 1 -- 172.21.15.29:0/4127449025 learned_addr learned my addr 172.21.15.29:0/4127449025 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.890+0000 7fc569d9b640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c0cf5f0 0x7fc56c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.891+0000 7fc56ad9d640 1 -- 172.21.15.29:0/4127449025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c0cf5f0 msgr2=0x7fc56c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.891+0000 7fc56ad9d640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c0cf5f0 0x7fc56c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.891+0000 7fc56ad9d640 1 -- 172.21.15.29:0/4127449025 >> v1:172.21.15.29:6789/0 conn(0x7fc56c070830 legacy=0x7fc56c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.891+0000 7fc56959a640 1 -- 172.21.15.29:0/4127449025 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967183802 0 0) 0x7fc56c161740 con 0x7fc56c070830 2023-10-14T16:42:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.891+0000 7fc56ad9d640 1 -- 172.21.15.29:0/4127449025 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56c1619b0 con 0x7fc56c070c10 2023-10-14T16:42:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.892+0000 7fc56ad9d640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c070c10 0x7fc56c0cd690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc56402b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e37c9e7a897946 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.892+0000 7fc56959a640 1 -- 172.21.15.29:0/4127449025 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc56402f050 con 0x7fc56c070c10 2023-10-14T16:42:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.892+0000 7fc56959a640 1 -- 172.21.15.29:0/4127449025 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc564031700 con 0x7fc56c070c10 2023-10-14T16:42:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.892+0000 7fc56959a640 1 -- 172.21.15.29:0/4127449025 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564031a90 con 0x7fc56c070c10 2023-10-14T16:42:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 -- 172.21.15.29:0/4127449025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c070c10 msgr2=0x7fc56c0cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c070c10 0x7fc56c0cd690 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc56402b7a0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 -- 172.21.15.29:0/4127449025 shutdown_connections 2023-10-14T16:42:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c070c10 0x7fc56c0cd690 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 --2- 172.21.15.29:0/4127449025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c0cf5f0 0x7fc56c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 -- 172.21.15.29:0/4127449025 >> 172.21.15.29:0/4127449025 conn(0x7fc56c0cfac0 msgr2=0x7fc56c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:04.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 -- 172.21.15.29:0/4127449025 shutdown_connections 2023-10-14T16:42:04.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.893+0000 7fc570a61640 1 -- 172.21.15.29:0/4127449025 wait complete. 2023-10-14T16:42:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.895+0000 7fc570a61640 1 Processor -- start 2023-10-14T16:42:04.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.895+0000 7fc570a61640 1 -- start start 2023-10-14T16:42:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 0x7fc56c0b5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 0x7fc56c14d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14a9e0 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14dbb0 con 0x7fc56c070830 2023-10-14T16:42:04.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.896+0000 7fc570a61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc56c14afc0 con 0x7fc56c070c10 2023-10-14T16:42:04.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58922/0 (socket says 172.21.15.29:58922) 2023-10-14T16:42:04.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 -- 172.21.15.29:0/3473109879 learned_addr learned my addr 172.21.15.29:0/3473109879 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56a59c640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 0x7fc56c0b5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc569d9b640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 0x7fc56c14d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 -- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 msgr2=0x7fc56c14d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 0x7fc56c14d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 -- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 msgr2=0x7fc56c0b5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 0x7fc56c0b5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 -- 172.21.15.29:0/3473109879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc564009400 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56a59c640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 0x7fc56c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:42:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc569d9b640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 0x7fc56c14d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:04.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.897+0000 7fc56ad9d640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc5640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.898+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564033a30 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.898+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc564033b90 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.898+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc564004030 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.898+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56c14b2a0 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.898+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56c14b860 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.899+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56c04e160 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.900+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc564004190 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.901+0000 7fc562ffd640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 0x7fc54c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:04.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.901+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc5640abe80 con 0x7fc56c0cf5f0 2023-10-14T16:42:04.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.901+0000 7fc56a59c640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 0x7fc54c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:04.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.902+0000 7fc56a59c640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 0x7fc54c06d5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0012e0 tx=0x7fc55c007f90 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:04.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:04.906+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc564076920 con 0x7fc56c0cf5f0 2023-10-14T16:42:05.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.123+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir8", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc56c0cf060 con 0x7fc54c06b130 2023-10-14T16:42:05.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.306+0000 7fc562ffd640 1 -- 172.21.15.29:0/3473109879 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fc55c007170 con 0x7fc54c06b130 2023-10-14T16:42:05.308 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir8 2023-10-14T16:42:05.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 msgr2=0x7fc54c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 0x7fc54c06d5f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0012e0 tx=0x7fc55c007f90 comp rx=0 tx=0).stop 2023-10-14T16:42:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 msgr2=0x7fc56c14e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7fc5640097a0 tx=0x7fc5640046c0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 shutdown_connections 2023-10-14T16:42:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc56c0cf5f0 0x7fc56c14e110 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc56c070c10 0x7fc56c14d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc54c06b130 0x7fc54c06d5f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 --2- 172.21.15.29:0/3473109879 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc56c070830 0x7fc56c0b5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.310+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 >> 172.21.15.29:0/3473109879 conn(0x7fc56c0cfac0 msgr2=0x7fc56c0c3260 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.311+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 shutdown_connections 2023-10-14T16:42:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.311+0000 7fc570a61640 1 -- 172.21.15.29:0/3473109879 wait complete. 2023-10-14T16:42:05.333 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir8 2023-10-14T16:42:05.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:42:05.334 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/dir9) 2023-10-14T16:42:05.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/dir9 1m 2023-10-14T16:42:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.626+0000 7f5e36fbf640 1 Processor -- start 2023-10-14T16:42:05.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.626+0000 7f5e36fbf640 1 -- start start 2023-10-14T16:42:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e36fbf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e300cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e36fbf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3015b9d0 con 0x7f5e300ad970 2023-10-14T16:42:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e36fbf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3015bcb0 con 0x7f5e3006c040 2023-10-14T16:42:05.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e36fbf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3015b7d0 con 0x7f5e3015b320 2023-10-14T16:42:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e357bc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5e300ad970 0x7f5e300cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35128/0 (socket says 172.21.15.29:35128) 2023-10-14T16:42:05.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.628+0000 7f5e357bc640 1 -- 172.21.15.29:0/3246055186 learned_addr learned my addr 172.21.15.29:0/3246055186 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3460693644 0 0) 0x7f5e3015b9d0 con 0x7f5e300ad970 2023-10-14T16:42:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e1c003520 con 0x7f5e300ad970 2023-10-14T16:42:05.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e367be640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e300cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 874055917 0 0) 0x7f5e1c003520 con 0x7f5e300ad970 2023-10-14T16:42:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 msgr2=0x7f5e300cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e300cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 >> v1:172.21.15.186:6789/0 conn(0x7f5e3006c040 legacy=0x7f5e3006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3015bf90 con 0x7f5e300ad970 2023-10-14T16:42:05.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5e20001670 con 0x7f5e300ad970 2023-10-14T16:42:05.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 >> v1:172.21.15.29:6789/0 conn(0x7f5e300ad970 legacy=0x7f5e300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e1c0039a0 0x7f5e1c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e1c0047c0 0x7f5e1c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e1c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3015b7d0 con 0x7f5e1c0039a0 2023-10-14T16:42:05.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c00bae0 con 0x7f5e1c0047c0 2023-10-14T16:42:05.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.629+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c00bca0 con 0x7f5e3015b320 2023-10-14T16:42:05.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e1c0039a0 0x7f5e1c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e357bc640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e1c0047c0 0x7f5e1c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 -- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 msgr2=0x7f5e1c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:05.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e367be640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e1c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e3015b320 0x7f5e1c0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 -- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e1c0047c0 msgr2=0x7f5e1c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:05.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e1c0047c0 0x7f5e1c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.630+0000 7f5e35fbd640 1 -- 172.21.15.29:0/3246055186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c003520 con 0x7f5e1c0039a0 2023-10-14T16:42:05.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.631+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e1c0039a0 0x7f5e1c003dc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c001750 tx=0x7f5e1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce7f2fda464408f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:05.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0052d0 con 0x7f5e1c0039a0 2023-10-14T16:42:05.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c02e050 con 0x7f5e1c0039a0 2023-10-14T16:42:05.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e34fbb640 1 -- 172.21.15.29:0/3246055186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2c0035f0 con 0x7f5e1c0039a0 2023-10-14T16:42:05.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 -- 172.21.15.29:0/3246055186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e1c0039a0 msgr2=0x7f5e1c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/3246055186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e1c0039a0 0x7f5e1c003dc0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c001750 tx=0x7f5e1c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e367be640 1 -- 172.21.15.29:0/3246055186 reap_dead start 2023-10-14T16:42:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 -- 172.21.15.29:0/3246055186 shutdown_connections 2023-10-14T16:42:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 -- 172.21.15.29:0/3246055186 >> 172.21.15.29:0/3246055186 conn(0x7f5e300ce6a0 msgr2=0x7f5e300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 -- 172.21.15.29:0/3246055186 shutdown_connections 2023-10-14T16:42:05.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.632+0000 7f5e36fbf640 1 -- 172.21.15.29:0/3246055186 wait complete. 2023-10-14T16:42:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.634+0000 7f5e36fbf640 1 Processor -- start 2023-10-14T16:42:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.634+0000 7f5e36fbf640 1 -- start start 2023-10-14T16:42:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e1c0039a0 0x7f5e30154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e300b3190 0x7f5e3015a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e301f25a0 con 0x7f5e300b2d10 2023-10-14T16:42:05.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e30154c90 con 0x7f5e300b3190 2023-10-14T16:42:05.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.635+0000 7f5e36fbf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e301f1990 con 0x7f5e1c0039a0 2023-10-14T16:42:05.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58942/0 (socket says 172.21.15.29:58942) 2023-10-14T16:42:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 -- 172.21.15.29:0/1872341569 learned_addr learned my addr 172.21.15.29:0/1872341569 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e1c0039a0 0x7f5e30154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e367be640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e300b3190 0x7f5e3015a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 -- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e1c0039a0 msgr2=0x7f5e30154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e1c0039a0 0x7f5e30154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 -- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e300b3190 msgr2=0x7f5e3015a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:05.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e300b3190 0x7f5e3015a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:05.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.636+0000 7f5e357bc640 1 -- 172.21.15.29:0/1872341569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e301f1c70 con 0x7f5e300b2d10 2023-10-14T16:42:05.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e357bc640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e730 tx=0x7f5e2000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:05.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000ee40 con 0x7f5e300b2d10 2023-10-14T16:42:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2000c8d0 con 0x7f5e300b2d10 2023-10-14T16:42:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e2000cc80 con 0x7f5e300b2d10 2023-10-14T16:42:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e301f1f50 con 0x7f5e300b2d10 2023-10-14T16:42:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.637+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e30157310 con 0x7f5e300b2d10 2023-10-14T16:42:05.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.638+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3006c8d0 con 0x7f5e300b2d10 2023-10-14T16:42:05.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.639+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f5e2001d460 con 0x7f5e300b2d10 2023-10-14T16:42:05.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.640+0000 7f5e267fc640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 0x7f5e1406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:05.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.640+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5e20089a20 con 0x7f5e300b2d10 2023-10-14T16:42:05.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.640+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 0x7f5e1406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:05.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.641+0000 7f5e35fbd640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 0x7f5e1406d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c033000 tx=0x7f5e2c009030 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:05.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.645+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5e200581c0 con 0x7f5e300b2d10 2023-10-14T16:42:05.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:05.867+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/dir9", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5e300cc550 con 0x7f5e1406b100 2023-10-14T16:42:06.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.060+0000 7f5e267fc640 1 -- 172.21.15.29:0/1872341569 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f5e2c005340 con 0x7f5e1406b100 2023-10-14T16:42:06.061 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/dir9 2023-10-14T16:42:06.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 msgr2=0x7f5e1406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 0x7f5e1406d5c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f5e2c033000 tx=0x7f5e2c009030 comp rx=0 tx=0).stop 2023-10-14T16:42:06.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 msgr2=0x7f5e30159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e730 tx=0x7f5e2000ebf0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 shutdown_connections 2023-10-14T16:42:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5e1406b100 0x7f5e1406d5c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5e300b3190 0x7f5e3015a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5e300b2d10 0x7f5e30159da0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 --2- 172.21.15.29:0/1872341569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5e1c0039a0 0x7f5e30154750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 >> 172.21.15.29:0/1872341569 conn(0x7f5e300ce6a0 msgr2=0x7f5e3015d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 shutdown_connections 2023-10-14T16:42:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.064+0000 7f5e36fbf640 1 -- 172.21.15.29:0/1872341569 wait complete. 2023-10-14T16:42:06.086 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/dir9 2023-10-14T16:42:06.086 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir0 1m 2023-10-14T16:42:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.323+0000 7fb9bef0a640 1 Processor -- start 2023-10-14T16:42:06.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.323+0000 7fb9bef0a640 1 -- start start 2023-10-14T16:42:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bef0a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb9b815dde0 0x7fb9b80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bef0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b8160a90 con 0x7fb9b815c9f0 2023-10-14T16:42:06.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bef0a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9b80cd900 con 0x7fb9b8070f20 2023-10-14T16:42:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bef0a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9b8071350 con 0x7fb9b815dde0 2023-10-14T16:42:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bdf08640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb9b815c9f0 0x7fb9b80c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35136/0 (socket says 172.21.15.29:35136) 2023-10-14T16:42:06.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.325+0000 7fb9bdf08640 1 -- 172.21.15.29:0/2173523570 learned_addr learned my addr 172.21.15.29:0/2173523570 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bd707640 1 --2- 172.21.15.29:0/2173523570 >> v2:172.21.15.186:3300/0 conn(0x7fb9b815dde0 0x7fb9b80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353807093 0 0) 0x7fb9b8160a90 con 0x7fb9b815c9f0 2023-10-14T16:42:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9a4003520 con 0x7fb9b815c9f0 2023-10-14T16:42:06.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3224640208 0 0) 0x7fb9a4003520 con 0x7fb9b815c9f0 2023-10-14T16:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 >> v2:172.21.15.186:3300/0 conn(0x7fb9b815dde0 msgr2=0x7fb9b80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 --2- 172.21.15.29:0/2173523570 >> v2:172.21.15.186:3300/0 conn(0x7fb9b815dde0 0x7fb9b80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 >> v1:172.21.15.186:6789/0 conn(0x7fb9b8070f20 legacy=0x7fb9b80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b80cdbe0 con 0x7fb9b815c9f0 2023-10-14T16:42:06.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb9b4001670 con 0x7fb9b815c9f0 2023-10-14T16:42:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.326+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 >> v1:172.21.15.29:6789/0 conn(0x7fb9b815c9f0 legacy=0x7fb9b80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9a4003970 0x7fb9a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8071350 con 0x7fb9a40047c0 2023-10-14T16:42:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400c020 con 0x7fb9a4005180 2023-10-14T16:42:06.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a400c1e0 con 0x7fb9a4003970 2023-10-14T16:42:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9be709640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 -- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9a4003970 msgr2=0x7fb9a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bdf08640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9a4003970 0x7fb9a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9a4003970 0x7fb9a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 -- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a4005180 msgr2=0x7fb9a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.327+0000 7fb9bd707640 1 -- 172.21.15.29:0/2173523570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9a4003520 con 0x7fb9a40047c0 2023-10-14T16:42:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.328+0000 7fb9bdf08640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9a4003970 0x7fb9a4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.328+0000 7fb9be709640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a4005180 0x7fb9a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:42:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.328+0000 7fb9be709640 1 -- 172.21.15.29:0/2173523570 reap_dead start 2023-10-14T16:42:06.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.329+0000 7fb9bd707640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009ac0 tx=0x7fb9a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=566663175e4f0a84 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.329+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a802c6b0 con 0x7fb9a40047c0 2023-10-14T16:42:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.329+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a802c810 con 0x7fb9a40047c0 2023-10-14T16:42:06.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.329+0000 7fb9bcf06640 1 -- 172.21.15.29:0/2173523570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a802cba0 con 0x7fb9a40047c0 2023-10-14T16:42:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 -- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 msgr2=0x7fb9a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8009ac0 tx=0x7fb9a40057f0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 -- 172.21.15.29:0/2173523570 shutdown_connections 2023-10-14T16:42:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/2173523570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9a40047c0 0x7fb9a4004c40 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 -- 172.21.15.29:0/2173523570 >> 172.21.15.29:0/2173523570 conn(0x7fb9b80ce6a0 msgr2=0x7fb9b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 -- 172.21.15.29:0/2173523570 shutdown_connections 2023-10-14T16:42:06.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.330+0000 7fb9bef0a640 1 -- 172.21.15.29:0/2173523570 wait complete. 2023-10-14T16:42:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.332+0000 7fb9bef0a640 1 Processor -- start 2023-10-14T16:42:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.332+0000 7fb9bef0a640 1 -- start start 2023-10-14T16:42:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a40047c0 0x7fb9b80c4990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b806a2e0 0x7fb9b80c31e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b806a790 con 0x7fb9b80c29d0 2023-10-14T16:42:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8159e90 con 0x7fb9a40047c0 2023-10-14T16:42:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bef0a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9b8159720 con 0x7fb9b806a2e0 2023-10-14T16:42:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.333+0000 7fb9bd707640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58960/0 (socket says 172.21.15.29:58960) 2023-10-14T16:42:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 -- 172.21.15.29:0/1926589654 learned_addr learned my addr 172.21.15.29:0/1926589654 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bdf08640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a40047c0 0x7fb9b80c4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9be709640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b806a2e0 0x7fb9b80c31e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 -- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b806a2e0 msgr2=0x7fb9b80c31e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b806a2e0 0x7fb9b80c31e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 -- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a40047c0 msgr2=0x7fb9b80c4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a40047c0 0x7fb9b80c4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 -- 172.21.15.29:0/1926589654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9b806c120 con 0x7fb9b80c29d0 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.334+0000 7fb9bd707640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8037040 tx=0x7fb9a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.335+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8003810 con 0x7fb9b80c29d0 2023-10-14T16:42:06.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.335+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9a8003970 con 0x7fb9b80c29d0 2023-10-14T16:42:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.335+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb9a8003d00 con 0x7fb9b80c29d0 2023-10-14T16:42:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.335+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9b806c400 con 0x7fb9b80c29d0 2023-10-14T16:42:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.335+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9b806cbe0 con 0x7fb9b80c29d0 2023-10-14T16:42:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.336+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9b804e9b0 con 0x7fb9b80c29d0 2023-10-14T16:42:06.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.337+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb9a8040450 con 0x7fb9b80c29d0 2023-10-14T16:42:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.338+0000 7fb9ae7fc640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 0x7fb99c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.338+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fb9a80abff0 con 0x7fb9b80c29d0 2023-10-14T16:42:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.338+0000 7fb9bdf08640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 0x7fb99c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.339+0000 7fb9bdf08640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 0x7fb99c06d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000fb0 tx=0x7fb9b40014d0 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.342+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb9a8076d70 con 0x7fb9b80c29d0 2023-10-14T16:42:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.565+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fb9b80704c0 con 0x7fb99c06b100 2023-10-14T16:42:06.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.705+0000 7fb9ae7fc640 1 -- 172.21.15.29:0/1926589654 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb9b400d070 con 0x7fb99c06b100 2023-10-14T16:42:06.707 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir0 2023-10-14T16:42:06.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 msgr2=0x7fb99c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 0x7fb99c06d5c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb9b4000fb0 tx=0x7fb9b40014d0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 msgr2=0x7fb9b8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fb9a8037040 tx=0x7fb9a8004730 comp rx=0 tx=0).stop 2023-10-14T16:42:06.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 shutdown_connections 2023-10-14T16:42:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb9b806a2e0 0x7fb9b80c31e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9b80c29d0 0x7fb9b8069da0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fb99c06b100 0x7fb99c06d5c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 --2- 172.21.15.29:0/1926589654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9a40047c0 0x7fb9b80c4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 >> 172.21.15.29:0/1926589654 conn(0x7fb9b80ce6a0 msgr2=0x7fb9b815ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:06.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 shutdown_connections 2023-10-14T16:42:06.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.709+0000 7fb9bef0a640 1 -- 172.21.15.29:0/1926589654 wait complete. 2023-10-14T16:42:06.732 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir0 2023-10-14T16:42:06.733 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir1 1m 2023-10-14T16:42:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.954+0000 7fcb0414c640 1 Processor -- start 2023-10-14T16:42:06.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.954+0000 7fcb0414c640 1 -- start start 2023-10-14T16:42:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb0414c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcafc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb0414c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcafc070360 con 0x7fcafc06c140 2023-10-14T16:42:06.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb0414c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcafc0cd1f0 con 0x7fcafc159ab0 2023-10-14T16:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb0414c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15d890 con 0x7fcafc159e90 2023-10-14T16:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb026e9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcafc06c140 0x7fcafc15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35152/0 (socket says 172.21.15.29:35152) 2023-10-14T16:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.956+0000 7fcb026e9640 1 -- 172.21.15.29:0/4233468355 learned_addr learned my addr 172.21.15.29:0/4233468355 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb016e7640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcafc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629926810 0 0) 0x7fcafc070360 con 0x7fcafc06c140 2023-10-14T16:42:06.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcae8003520 con 0x7fcafc06c140 2023-10-14T16:42:06.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1924034274 0 0) 0x7fcae8003520 con 0x7fcafc06c140 2023-10-14T16:42:06.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 msgr2=0x7fcafc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcafc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 >> v1:172.21.15.186:6789/0 conn(0x7fcafc159ab0 legacy=0x7fcafc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc0cd4d0 con 0x7fcafc06c140 2023-10-14T16:42:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fcaf40019f0 con 0x7fcafc06c140 2023-10-14T16:42:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 >> v1:172.21.15.29:6789/0 conn(0x7fcafc06c140 legacy=0x7fcafc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcae80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcae80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15d890 con 0x7fcae8003f20 2023-10-14T16:42:06.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcae800ddd0 con 0x7fcae8004d00 2023-10-14T16:42:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.957+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae800df90 con 0x7fcafc159e90 2023-10-14T16:42:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcae80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb01ee8640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcae80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 -- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 msgr2=0x7fcae80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc159e90 0x7fcae80039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 -- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcae8004d00 msgr2=0x7fcae8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcae8004d00 0x7fcae8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.958+0000 7fcb016e7640 1 -- 172.21.15.29:0/4233468355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae8003520 con 0x7fcae8003f20 2023-10-14T16:42:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.959+0000 7fcb016e7640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcae80047c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009ac0 tx=0x7fcae80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387283a56a787b8a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.959+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec02c6b0 con 0x7fcae8003f20 2023-10-14T16:42:06.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.959+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaec02c810 con 0x7fcae8003f20 2023-10-14T16:42:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.959+0000 7fcb00ee6640 1 -- 172.21.15.29:0/4233468355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec02cba0 con 0x7fcae8003f20 2023-10-14T16:42:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 -- 172.21.15.29:0/4233468355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 msgr2=0x7fcae80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 --2- 172.21.15.29:0/4233468355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcae80047c0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7fcaec009ac0 tx=0x7fcae80057f0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb026e9640 1 -- 172.21.15.29:0/4233468355 reap_dead start 2023-10-14T16:42:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 -- 172.21.15.29:0/4233468355 shutdown_connections 2023-10-14T16:42:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 -- 172.21.15.29:0/4233468355 >> 172.21.15.29:0/4233468355 conn(0x7fcafc0ce9b0 msgr2=0x7fcafc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 -- 172.21.15.29:0/4233468355 shutdown_connections 2023-10-14T16:42:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.960+0000 7fcb0414c640 1 -- 172.21.15.29:0/4233468355 wait complete. 2023-10-14T16:42:06.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.962+0000 7fcb0414c640 1 Processor -- start 2023-10-14T16:42:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.962+0000 7fcb0414c640 1 -- start start 2023-10-14T16:42:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc15a330 0x7fcafc15be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcafc15a7b0 0x7fcafc15c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15b990 con 0x7fcae8003f20 2023-10-14T16:42:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1f2ba0 con 0x7fcafc15a7b0 2023-10-14T16:42:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb0414c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc15ad80 con 0x7fcafc15a330 2023-10-14T16:42:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb01ee8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb01ee8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58978/0 (socket says 172.21.15.29:58978) 2023-10-14T16:42:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.963+0000 7fcb01ee8640 1 -- 172.21.15.29:0/935915954 learned_addr learned my addr 172.21.15.29:0/935915954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 -- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc15a330 msgr2=0x7fcafc15be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc15a330 0x7fcafc15be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 -- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcafc15a7b0 msgr2=0x7fcafc15c360 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-10-14T16:42:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcafc15a7b0 0x7fcafc15c360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 -- 172.21.15.29:0/935915954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc15b060 con 0x7fcae8003f20 2023-10-14T16:42:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcb01ee8640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800d570 tx=0x7fcaf800da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.964+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8005480 con 0x7fcae8003f20 2023-10-14T16:42:06.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.965+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcaf8003610 con 0x7fcae8003f20 2023-10-14T16:42:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.965+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf8003980 con 0x7fcae8003f20 2023-10-14T16:42:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.965+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc15b340 con 0x7fcae8003f20 2023-10-14T16:42:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.965+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc158870 con 0x7fcae8003f20 2023-10-14T16:42:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.966+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcafc158a00 con 0x7fcae8003f20 2023-10-14T16:42:06.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.967+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fcaf8003ae0 con 0x7fcae8003f20 2023-10-14T16:42:06.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.968+0000 7fcaf27fc640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 0x7fcae006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:06.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.968+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fcaf8089c20 con 0x7fcae8003f20 2023-10-14T16:42:06.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.968+0000 7fcb016e7640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 0x7fcae006d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.969+0000 7fcb016e7640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 0x7fcae006d5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcaec036000 tx=0x7fcaec02e000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:06.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:06.972+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcaf8054760 con 0x7fcae8003f20 2023-10-14T16:42:07.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.193+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcafc0cda90 con 0x7fcae006b130 2023-10-14T16:42:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.340+0000 7fcaf27fc640 1 -- 172.21.15.29:0/935915954 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fcaec0040c0 con 0x7fcae006b130 2023-10-14T16:42:07.342 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir1 2023-10-14T16:42:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 msgr2=0x7fcae006d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 0x7fcae006d5f0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcaec036000 tx=0x7fcaec02e000 comp rx=0 tx=0).stop 2023-10-14T16:42:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 msgr2=0x7fcafc1f2660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fcaf800d570 tx=0x7fcaf800da30 comp rx=0 tx=0).stop 2023-10-14T16:42:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 shutdown_connections 2023-10-14T16:42:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcae006b130 0x7fcae006d5f0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.344+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcafc15a7b0 0x7fcafc15c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.345+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcafc15a330 0x7fcafc15be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.345+0000 7fcb0414c640 1 --2- 172.21.15.29:0/935915954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcae8003f20 0x7fcafc1f2660 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.345+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 >> 172.21.15.29:0/935915954 conn(0x7fcafc0ce9b0 msgr2=0x7fcafc0ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.345+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 shutdown_connections 2023-10-14T16:42:07.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.345+0000 7fcb0414c640 1 -- 172.21.15.29:0/935915954 wait complete. 2023-10-14T16:42:07.369 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir1 2023-10-14T16:42:07.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir2 1m 2023-10-14T16:42:07.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.598+0000 7f9ee9735640 1 Processor -- start 2023-10-14T16:42:07.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.598+0000 7f9ee9735640 1 -- start start 2023-10-14T16:42:07.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.599+0000 7f9ee9735640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.599+0000 7f9ee9735640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ee40d0be0 0x7f9ee40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.599+0000 7f9ee9735640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ee415d150 con 0x7f9ee415cd70 2023-10-14T16:42:07.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.599+0000 7f9ee9735640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ee40d1180 con 0x7f9ee40d0be0 2023-10-14T16:42:07.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.599+0000 7f9ee9735640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee40d1460 con 0x7f9ee40b8480 2023-10-14T16:42:07.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.600+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.600+0000 7f9edb7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:52680/0 (socket says 172.21.15.29:52680) 2023-10-14T16:42:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.600+0000 7f9ee8f34640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ee40d0be0 0x7f9ee40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.600+0000 7f9edb7fe640 1 -- 172.21.15.29:0/822109168 learned_addr learned my addr 172.21.15.29:0/822109168 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.601+0000 7f9edb7fe640 1 -- 172.21.15.29:0/822109168 >> v2:172.21.15.186:3300/0 conn(0x7f9ee40d0be0 msgr2=0x7f9ee40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.601+0000 7f9edb7fe640 1 --2- 172.21.15.29:0/822109168 >> v2:172.21.15.186:3300/0 conn(0x7f9ee40d0be0 0x7f9ee40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.601+0000 7f9edb7fe640 1 -- 172.21.15.29:0/822109168 >> v1:172.21.15.186:6789/0 conn(0x7f9ee415cd70 legacy=0x7f9ee40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.601+0000 7f9edb7fe640 1 -- 172.21.15.29:0/822109168 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee40d1740 con 0x7f9ee40b8480 2023-10-14T16:42:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.602+0000 7f9edb7fe640 1 --2- 172.21.15.29:0/822109168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009870 tx=0x7f9ecc02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18102be7bb24c366 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.603+0000 7f9ee37fe640 1 -- 172.21.15.29:0/822109168 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc030840 con 0x7f9ee40b8480 2023-10-14T16:42:07.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.603+0000 7f9ee37fe640 1 -- 172.21.15.29:0/822109168 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ecc0309a0 con 0x7f9ee40b8480 2023-10-14T16:42:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.603+0000 7f9ee37fe640 1 -- 172.21.15.29:0/822109168 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ecc030d30 con 0x7f9ee40b8480 2023-10-14T16:42:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 -- 172.21.15.29:0/822109168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 msgr2=0x7f9ee40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 --2- 172.21.15.29:0/822109168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc009870 tx=0x7f9ecc02b980 comp rx=0 tx=0).stop 2023-10-14T16:42:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 -- 172.21.15.29:0/822109168 shutdown_connections 2023-10-14T16:42:07.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 --2- 172.21.15.29:0/822109168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee40b8480 0x7f9ee40cd7c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 --2- 172.21.15.29:0/822109168 >> v2:172.21.15.186:3300/0 conn(0x7f9ee40d0be0 0x7f9ee40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 -- 172.21.15.29:0/822109168 >> 172.21.15.29:0/822109168 conn(0x7f9ee40cf570 msgr2=0x7f9ee40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:07.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 -- 172.21.15.29:0/822109168 shutdown_connections 2023-10-14T16:42:07.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.604+0000 7f9ee9735640 1 -- 172.21.15.29:0/822109168 wait complete. 2023-10-14T16:42:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.606+0000 7f9ee9735640 1 Processor -- start 2023-10-14T16:42:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.606+0000 7f9ee9735640 1 -- start start 2023-10-14T16:42:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ee40d0be0 0x7f9ee4151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee415cd70 0x7f9ee415df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee415d4c0 con 0x7f9ee40b8480 2023-10-14T16:42:07.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee415f500 con 0x7f9ee40d0be0 2023-10-14T16:42:07.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee9735640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ee415d780 con 0x7f9ee415cd70 2023-10-14T16:42:07.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58984/0 (socket says 172.21.15.29:58984) 2023-10-14T16:42:07.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.607+0000 7f9ee3fff640 1 -- 172.21.15.29:0/4221271672 learned_addr learned my addr 172.21.15.29:0/4221271672 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:07.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee8f34640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee415cd70 0x7f9ee415df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9edb7fe640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ee40d0be0 0x7f9ee4151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 -- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee415cd70 msgr2=0x7f9ee415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee415cd70 0x7f9ee415df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 -- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ee40d0be0 msgr2=0x7f9ee4151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ee40d0be0 0x7f9ee4151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 -- 172.21.15.29:0/4221271672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ecc0094d0 con 0x7f9ee40b8480 2023-10-14T16:42:07.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.608+0000 7f9ee3fff640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c870 tx=0x7f9ed400cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:07.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.609+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed4003b70 con 0x7f9ee40b8480 2023-10-14T16:42:07.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.609+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ed4003cf0 con 0x7f9ee40b8480 2023-10-14T16:42:07.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.609+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ee415da60 con 0x7f9ee40b8480 2023-10-14T16:42:07.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.609+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ed400aa90 con 0x7f9ee40b8480 2023-10-14T16:42:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.609+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ee4152100 con 0x7f9ee40b8480 2023-10-14T16:42:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.610+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ee404e9b0 con 0x7f9ee40b8480 2023-10-14T16:42:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.611+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f9ed400abf0 con 0x7f9ee40b8480 2023-10-14T16:42:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.612+0000 7f9ee17fa640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 0x7f9ec406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.612+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9ed4089a90 con 0x7f9ee40b8480 2023-10-14T16:42:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.612+0000 7f9edb7fe640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 0x7f9ec406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:07.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.613+0000 7f9edb7fe640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 0x7f9ec406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc036000 tx=0x7f9ecc03b040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.616+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ed40546d0 con 0x7f9ee40b8480 2023-10-14T16:42:07.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.835+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ee40d1d00 con 0x7f9ec406b100 2023-10-14T16:42:07.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.980+0000 7f9ee17fa640 1 -- 172.21.15.29:0/4221271672 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f9ecc030b90 con 0x7f9ec406b100 2023-10-14T16:42:07.982 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir2 2023-10-14T16:42:07.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 msgr2=0x7f9ec406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 0x7f9ec406d5c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9ecc036000 tx=0x7f9ecc03b040 comp rx=0 tx=0).stop 2023-10-14T16:42:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 msgr2=0x7f9ee41f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f9ed400c870 tx=0x7f9ed400cd30 comp rx=0 tx=0).stop 2023-10-14T16:42:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 shutdown_connections 2023-10-14T16:42:07.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ee415cd70 0x7f9ee415df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9ec406b100 0x7f9ec406d5c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ee40d0be0 0x7f9ee4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 --2- 172.21.15.29:0/4221271672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ee40b8480 0x7f9ee41f2a10 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:07.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.983+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 >> 172.21.15.29:0/4221271672 conn(0x7f9ee40cf570 msgr2=0x7f9ee40d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:07.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.984+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 shutdown_connections 2023-10-14T16:42:07.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:07.984+0000 7f9ee9735640 1 -- 172.21.15.29:0/4221271672 wait complete. 2023-10-14T16:42:08.008 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir2 2023-10-14T16:42:08.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir3 1m 2023-10-14T16:42:08.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.229+0000 7f570935a640 1 Processor -- start 2023-10-14T16:42:08.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.229+0000 7f570935a640 1 -- start start 2023-10-14T16:42:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.230+0000 7f570935a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f570935a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 0x7f570406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f570935a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 0x7f57040ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f570935a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040cad80 con 0x7f570415ad70 2023-10-14T16:42:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f570935a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57040cb030 con 0x7f57040ca3b0 2023-10-14T16:42:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f570935a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c91c0 con 0x7f57040c9ca0 2023-10-14T16:42:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f5702ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f5702ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58986/0 (socket says 172.21.15.29:58986) 2023-10-14T16:42:08.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f5702ffd640 1 -- 172.21.15.29:0/72565774 learned_addr learned my addr 172.21.15.29:0/72565774 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f5702ffd640 1 -- 172.21.15.29:0/72565774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 msgr2=0x7f570406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:08.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f57037fe640 1 --2- 172.21.15.29:0/72565774 >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 0x7f57040ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.231+0000 7f57027fc640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 0x7f570406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f5702ffd640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 0x7f570406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f5702ffd640 1 -- 172.21.15.29:0/72565774 >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 msgr2=0x7f57040ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f5702ffd640 1 --2- 172.21.15.29:0/72565774 >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 0x7f57040ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f5702ffd640 1 -- 172.21.15.29:0/72565774 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57040c94a0 con 0x7f570415ad70 2023-10-14T16:42:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f57037fe640 1 --2- 172.21.15.29:0/72565774 >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 0x7f57040ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.232+0000 7f57027fc640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 0x7f570406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.233+0000 7f5702ffd640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f570415ad70 0x7f5704069da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f57040cf820 tx=0x7f56f402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e867f2385f141e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.233+0000 7f5701ffb640 1 -- 172.21.15.29:0/72565774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4031880 con 0x7f570415ad70 2023-10-14T16:42:08.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.233+0000 7f5701ffb640 1 -- 172.21.15.29:0/72565774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f40319e0 con 0x7f570415ad70 2023-10-14T16:42:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.233+0000 7f5701ffb640 1 -- 172.21.15.29:0/72565774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4031d70 con 0x7f570415ad70 2023-10-14T16:42:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 -- 172.21.15.29:0/72565774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f570415ad70 msgr2=0x7f5704069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f570415ad70 0x7f5704069da0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f57040cf820 tx=0x7f56f402b920 comp rx=0 tx=0).stop 2023-10-14T16:42:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 -- 172.21.15.29:0/72565774 shutdown_connections 2023-10-14T16:42:08.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 --2- 172.21.15.29:0/72565774 >> v2:172.21.15.186:3300/0 conn(0x7f57040ca3b0 0x7f57040ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040c9ca0 0x7f570406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 --2- 172.21.15.29:0/72565774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f570415ad70 0x7f5704069da0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 -- 172.21.15.29:0/72565774 >> 172.21.15.29:0/72565774 conn(0x7f57040ce9b0 msgr2=0x7f57040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 -- 172.21.15.29:0/72565774 shutdown_connections 2023-10-14T16:42:08.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.234+0000 7f570935a640 1 -- 172.21.15.29:0/72565774 wait complete. 2023-10-14T16:42:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.236+0000 7f570935a640 1 Processor -- start 2023-10-14T16:42:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.236+0000 7f570935a640 1 -- start start 2023-10-14T16:42:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040ca3b0 0x7f57040b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f570415ad70 0x7f570414be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040c43e0 con 0x7f57040c9ca0 2023-10-14T16:42:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b3740 con 0x7f570415ad70 2023-10-14T16:42:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f570935a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57040b1ce0 con 0x7f57040ca3b0 2023-10-14T16:42:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58998/0 (socket says 172.21.15.29:58998) 2023-10-14T16:42:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 -- 172.21.15.29:0/3146337265 learned_addr learned my addr 172.21.15.29:0/3146337265 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 -- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040ca3b0 msgr2=0x7f57040b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f57027fc640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040ca3b0 0x7f57040b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f57037fe640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f570415ad70 0x7f570414be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040ca3b0 0x7f57040b17a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 -- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f570415ad70 msgr2=0x7f570414be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f570415ad70 0x7f570414be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.237+0000 7f5702ffd640 1 -- 172.21.15.29:0/3146337265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f40094d0 con 0x7f57040c9ca0 2023-10-14T16:42:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f5702ffd640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f4031850 con 0x7f57040c9ca0 2023-10-14T16:42:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56f40443f0 con 0x7f57040c9ca0 2023-10-14T16:42:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f40447a0 con 0x7f57040c9ca0 2023-10-14T16:42:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570414c3b0 con 0x7f57040c9ca0 2023-10-14T16:42:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.238+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57040b3990 con 0x7f57040c9ca0 2023-10-14T16:42:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.239+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f570404e160 con 0x7f57040c9ca0 2023-10-14T16:42:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.240+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f56f40303d0 con 0x7f57040c9ca0 2023-10-14T16:42:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.241+0000 7f56eb7fe640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 0x7f56f006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.241+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f56f40ac5c0 con 0x7f57040c9ca0 2023-10-14T16:42:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.241+0000 7f57027fc640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 0x7f56f006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.242+0000 7f57027fc640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 0x7f56f006d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.246+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56f40770f0 con 0x7f57040c9ca0 2023-10-14T16:42:08.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.466+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f57040c9780 con 0x7f56f006b100 2023-10-14T16:42:08.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.606+0000 7f56eb7fe640 1 -- 172.21.15.29:0/3146337265 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f56f800c040 con 0x7f56f006b100 2023-10-14T16:42:08.609 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir3 2023-10-14T16:42:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.610+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 msgr2=0x7f56f006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 0x7f56f006d5c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f56f8006fd0 tx=0x7f56f8008040 comp rx=0 tx=0).stop 2023-10-14T16:42:08.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 msgr2=0x7f57040b3200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f56f4000c00 tx=0x7f56f40046c0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 shutdown_connections 2023-10-14T16:42:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56f006b100 0x7f56f006d5c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f570415ad70 0x7f570414be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57040ca3b0 0x7f57040b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 --2- 172.21.15.29:0/3146337265 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57040c9ca0 0x7f57040b3200 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 >> 172.21.15.29:0/3146337265 conn(0x7f57040ce9b0 msgr2=0x7f570415c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:08.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 shutdown_connections 2023-10-14T16:42:08.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.611+0000 7f570935a640 1 -- 172.21.15.29:0/3146337265 wait complete. 2023-10-14T16:42:08.634 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir3 2023-10-14T16:42:08.635 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir4 1m 2023-10-14T16:42:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.853+0000 7f169f0bb640 1 Processor -- start 2023-10-14T16:42:08.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.853+0000 7f169f0bb640 1 -- start start 2023-10-14T16:42:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.854+0000 7f169f0bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16980caac0 0x7f16980ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.854+0000 7f169f0bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980cb030 con 0x7f169806a4b0 2023-10-14T16:42:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f169f0bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16980c91c0 con 0x7f169815ad70 2023-10-14T16:42:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f169f0bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16980c94a0 con 0x7f16980caac0 2023-10-14T16:42:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f1697fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f169806a4b0 0x7f16980ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35156/0 (socket says 172.21.15.29:35156) 2023-10-14T16:42:08.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f1697fff640 1 -- 172.21.15.29:0/4256338250 learned_addr learned my addr 172.21.15.29:0/4256338250 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f169d658640 1 --2- 172.21.15.29:0/4256338250 >> v2:172.21.15.186:3300/0 conn(0x7f16980caac0 0x7f16980ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2208234968 0 0) 0x7f16980cb030 con 0x7f169806a4b0 2023-10-14T16:42:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.855+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1680003520 con 0x7f169806a4b0 2023-10-14T16:42:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4230613571 0 0) 0x7f1680003520 con 0x7f169806a4b0 2023-10-14T16:42:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 >> v2:172.21.15.186:3300/0 conn(0x7f16980caac0 msgr2=0x7f16980ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 --2- 172.21.15.29:0/4256338250 >> v2:172.21.15.186:3300/0 conn(0x7f16980caac0 0x7f16980ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 >> v1:172.21.15.186:6789/0 conn(0x7f169815ad70 legacy=0x7f1698069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980c9780 con 0x7f169806a4b0 2023-10-14T16:42:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f168c001670 con 0x7f169806a4b0 2023-10-14T16:42:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 >> v1:172.21.15.29:6789/0 conn(0x7f169806a4b0 legacy=0x7f16980ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1680003970 0x7f1680003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16800047c0 0x7f1680004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f168000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c94a0 con 0x7f1680005180 2023-10-14T16:42:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168000c020 con 0x7f1680003970 2023-10-14T16:42:08.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168000c1e0 con 0x7f16800047c0 2023-10-14T16:42:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.856+0000 7f169d658640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f168000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169ce57640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1680003970 0x7f1680003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169d658640 1 -- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16800047c0 msgr2=0x7f1680004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169d658640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16800047c0 0x7f1680004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169d658640 1 -- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1680003970 msgr2=0x7f1680003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169d658640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1680003970 0x7f1680003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169d658640 1 -- 172.21.15.29:0/4256338250 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1680003520 con 0x7f1680005180 2023-10-14T16:42:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.857+0000 7f169ce57640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1680003970 0x7f1680003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.858+0000 7f169d658640 1 -- 172.21.15.29:0/4256338250 reap_dead start 2023-10-14T16:42:08.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.858+0000 7f169d658640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f168000b980 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1690009ac0 tx=0x7f16800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd68e49a0d55ea0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.858+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690004020 con 0x7f1680005180 2023-10-14T16:42:08.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.858+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690004180 con 0x7f1680005180 2023-10-14T16:42:08.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.858+0000 7f16977fe640 1 -- 172.21.15.29:0/4256338250 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169002c710 con 0x7f1680005180 2023-10-14T16:42:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 -- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 msgr2=0x7f168000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f168000b980 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f1690009ac0 tx=0x7f16800057f0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 -- 172.21.15.29:0/4256338250 shutdown_connections 2023-10-14T16:42:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 --2- 172.21.15.29:0/4256338250 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f168000b980 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 -- 172.21.15.29:0/4256338250 >> 172.21.15.29:0/4256338250 conn(0x7f16980ce9b0 msgr2=0x7f16980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 -- 172.21.15.29:0/4256338250 shutdown_connections 2023-10-14T16:42:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.859+0000 7f169f0bb640 1 -- 172.21.15.29:0/4256338250 wait complete. 2023-10-14T16:42:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.861+0000 7f169f0bb640 1 Processor -- start 2023-10-14T16:42:08.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.861+0000 7f169f0bb640 1 -- start start 2023-10-14T16:42:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16980c5a80 0x7f169815bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 0x7f16980c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f169815c5e0 con 0x7f1680005180 2023-10-14T16:42:08.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16981f22b0 con 0x7f169815c150 2023-10-14T16:42:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169f0bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16980c43e0 con 0x7f16980c5a80 2023-10-14T16:42:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169ce57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169ce57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59026/0 (socket says 172.21.15.29:59026) 2023-10-14T16:42:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.862+0000 7f169ce57640 1 -- 172.21.15.29:0/1588864316 learned_addr learned my addr 172.21.15.29:0/1588864316 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:08.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f1697fff640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16980c5a80 0x7f169815bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169d658640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 0x7f16980c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 -- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16980c5a80 msgr2=0x7f169815bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16980c5a80 0x7f169815bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 -- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 msgr2=0x7f16980c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 0x7f16980c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:08.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 -- 172.21.15.29:0/1588864316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16980c5f80 con 0x7f1680005180 2023-10-14T16:42:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169d658640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 0x7f16980c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.863+0000 7f169ce57640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f168800dae0 tx=0x7f1688002070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.864+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688003da0 con 0x7f1680005180 2023-10-14T16:42:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.864+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1688005070 con 0x7f1680005180 2023-10-14T16:42:08.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.864+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16980c3620 con 0x7f1680005180 2023-10-14T16:42:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.864+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1688005420 con 0x7f1680005180 2023-10-14T16:42:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.864+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16980c3be0 con 0x7f1680005180 2023-10-14T16:42:08.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.865+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169804e160 con 0x7f1680005180 2023-10-14T16:42:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.866+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f1688005580 con 0x7f1680005180 2023-10-14T16:42:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.867+0000 7f16957fa640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 0x7f168406d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:08.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.867+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f1688089ec0 con 0x7f1680005180 2023-10-14T16:42:08.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.867+0000 7f1697fff640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 0x7f168406d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:08.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.868+0000 7f1697fff640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 0x7f168406d5f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f169806ced0 tx=0x7f168c001680 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:08.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:08.874+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16880549f0 con 0x7f1680005180 2023-10-14T16:42:09.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.095+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f16980c9ca0 con 0x7f168406b130 2023-10-14T16:42:09.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.249+0000 7f16957fa640 1 -- 172.21.15.29:0/1588864316 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f168c00e870 con 0x7f168406b130 2023-10-14T16:42:09.252 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir4 2023-10-14T16:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 msgr2=0x7f168406d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 0x7f168406d5f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f169806ced0 tx=0x7f168c001680 comp rx=0 tx=0).stop 2023-10-14T16:42:09.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 msgr2=0x7f16980ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f168800dae0 tx=0x7f1688002070 comp rx=0 tx=0).stop 2023-10-14T16:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 shutdown_connections 2023-10-14T16:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f168406b130 0x7f168406d5f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f169815c150 0x7f16980c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16980c5a80 0x7f169815bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 --2- 172.21.15.29:0/1588864316 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1680005180 0x7f16980ac890 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.253+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 >> 172.21.15.29:0/1588864316 conn(0x7f16980ce9b0 msgr2=0x7f16980b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.254+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 shutdown_connections 2023-10-14T16:42:09.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.254+0000 7f169f0bb640 1 -- 172.21.15.29:0/1588864316 wait complete. 2023-10-14T16:42:09.277 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir4 2023-10-14T16:42:09.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir5 1m 2023-10-14T16:42:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.496+0000 7fc3a9f16640 1 Processor -- start 2023-10-14T16:42:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.497+0000 7fc3a9f16640 1 -- start start 2023-10-14T16:42:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.498+0000 7fc3a9f16640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3a40b8480 0x7fc3a40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.498+0000 7fc3a9f16640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3a40d0be0 0x7fc3a40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.498+0000 7fc3a9f16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a415d150 con 0x7fc3a415cd70 2023-10-14T16:42:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.498+0000 7fc3a9f16640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40d1180 con 0x7fc3a40d0be0 2023-10-14T16:42:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.498+0000 7fc3a9f16640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a40d1460 con 0x7fc3a40b8480 2023-10-14T16:42:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a8f14640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3a415cd70 0x7fc3a40d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35168/0 (socket says 172.21.15.29:35168) 2023-10-14T16:42:09.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a8f14640 1 -- 172.21.15.29:0/355511934 learned_addr learned my addr 172.21.15.29:0/355511934 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:09.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a9715640 1 --2- 172.21.15.29:0/355511934 >> v2:172.21.15.29:3300/0 conn(0x7fc3a40d0be0 0x7fc3a40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:09.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a3fff640 1 --2- 172.21.15.29:0/355511934 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40b8480 0x7fc3a40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:09.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a37fe640 1 -- 172.21.15.29:0/355511934 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2514549210 0 0) 0x7fc3a415d150 con 0x7fc3a415cd70 2023-10-14T16:42:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a37fe640 1 -- 172.21.15.29:0/355511934 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc394003520 con 0x7fc3a415cd70 2023-10-14T16:42:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a9715640 1 -- 172.21.15.29:0/355511934 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40b8480 msgr2=0x7fc3a40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a9715640 1 --2- 172.21.15.29:0/355511934 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40b8480 0x7fc3a40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a9715640 1 -- 172.21.15.29:0/355511934 >> v1:172.21.15.29:6789/0 conn(0x7fc3a415cd70 legacy=0x7fc3a40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.499+0000 7fc3a9715640 1 -- 172.21.15.29:0/355511934 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40d1740 con 0x7fc3a40d0be0 2023-10-14T16:42:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.500+0000 7fc3a9715640 1 --2- 172.21.15.29:0/355511934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40d0be0 0x7fc3a40cdd00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4066a00 tx=0x7fc39c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e2bb99018d789 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a37fe640 1 -- 172.21.15.29:0/355511934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c02cbe0 con 0x7fc3a40d0be0 2023-10-14T16:42:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a37fe640 1 -- 172.21.15.29:0/355511934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c02cd40 con 0x7fc3a40d0be0 2023-10-14T16:42:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a37fe640 1 -- 172.21.15.29:0/355511934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c034660 con 0x7fc3a40d0be0 2023-10-14T16:42:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a9f16640 1 -- 172.21.15.29:0/355511934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40d0be0 msgr2=0x7fc3a40cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/355511934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40d0be0 0x7fc3a40cdd00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4066a00 tx=0x7fc39c02be30 comp rx=0 tx=0).stop 2023-10-14T16:42:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a9f16640 1 -- 172.21.15.29:0/355511934 shutdown_connections 2023-10-14T16:42:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/355511934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40d0be0 0x7fc3a40cdd00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.501+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/355511934 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40b8480 0x7fc3a40cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.502+0000 7fc3a9f16640 1 -- 172.21.15.29:0/355511934 >> 172.21.15.29:0/355511934 conn(0x7fc3a40cf570 msgr2=0x7fc3a40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.502+0000 7fc3a9f16640 1 -- 172.21.15.29:0/355511934 shutdown_connections 2023-10-14T16:42:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.502+0000 7fc3a9f16640 1 -- 172.21.15.29:0/355511934 wait complete. 2023-10-14T16:42:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.503+0000 7fc3a9f16640 1 Processor -- start 2023-10-14T16:42:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.503+0000 7fc3a9f16640 1 -- start start 2023-10-14T16:42:09.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 0x7fc3a4142a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40d0be0 0x7fc3a415f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40ccc30 con 0x7fc3a415cd70 2023-10-14T16:42:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415e390 con 0x7fc3a40b8480 2023-10-14T16:42:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.504+0000 7fc3a9f16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4142f80 con 0x7fc3a40d0be0 2023-10-14T16:42:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59052/0 (socket says 172.21.15.29:59052) 2023-10-14T16:42:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 -- 172.21.15.29:0/2410745489 learned_addr learned my addr 172.21.15.29:0/2410745489 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 -- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40d0be0 msgr2=0x7fc3a415f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40d0be0 0x7fc3a415f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a8f14640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 0x7fc3a4142a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 -- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 msgr2=0x7fc3a4142a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 0x7fc3a4142a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a9715640 1 -- 172.21.15.29:0/2410745489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39c009a70 con 0x7fc3a415cd70 2023-10-14T16:42:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.505+0000 7fc3a8f14640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 0x7fc3a4142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a9715640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc39c02c440 tx=0x7fc39c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c040650 con 0x7fc3a415cd70 2023-10-14T16:42:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c0407b0 con 0x7fc3a415cd70 2023-10-14T16:42:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c040b60 con 0x7fc3a415cd70 2023-10-14T16:42:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a415e5e0 con 0x7fc3a415cd70 2023-10-14T16:42:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.506+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a415eba0 con 0x7fc3a415cd70 2023-10-14T16:42:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.507+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3a40d28f0 con 0x7fc3a415cd70 2023-10-14T16:42:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.508+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fc39c040cc0 con 0x7fc3a415cd70 2023-10-14T16:42:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.509+0000 7fc3a17fa640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 0x7fc38806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.509+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc39c0ac8a0 con 0x7fc3a415cd70 2023-10-14T16:42:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.509+0000 7fc3a8f14640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 0x7fc38806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.510+0000 7fc3a8f14640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 0x7fc38806d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3980011e0 tx=0x7fc398009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.513+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc39c0774e0 con 0x7fc3a415cd70 2023-10-14T16:42:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.732+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3a40d1d00 con 0x7fc38806b100 2023-10-14T16:42:09.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.888+0000 7fc3a17fa640 1 -- 172.21.15.29:0/2410745489 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fc398001d20 con 0x7fc38806b100 2023-10-14T16:42:09.890 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir5 2023-10-14T16:42:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 msgr2=0x7fc38806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 0x7fc38806d5c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc3980011e0 tx=0x7fc398009040 comp rx=0 tx=0).stop 2023-10-14T16:42:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 msgr2=0x7fc3a4158db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fc39c02c440 tx=0x7fc39c004a40 comp rx=0 tx=0).stop 2023-10-14T16:42:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 shutdown_connections 2023-10-14T16:42:09.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a415cd70 0x7fc3a4158db0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40d0be0 0x7fc3a415f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc38806b100 0x7fc38806d5c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 --2- 172.21.15.29:0/2410745489 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40b8480 0x7fc3a4142a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.891+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 >> 172.21.15.29:0/2410745489 conn(0x7fc3a40cf570 msgr2=0x7fc3a40b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.892+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 shutdown_connections 2023-10-14T16:42:09.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:09.892+0000 7fc3a9f16640 1 -- 172.21.15.29:0/2410745489 wait complete. 2023-10-14T16:42:09.913 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir5 2023-10-14T16:42:09.913 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir6 1m 2023-10-14T16:42:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.136+0000 7f044cedf640 1 Processor -- start 2023-10-14T16:42:10.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.136+0000 7f044cedf640 1 -- start start 2023-10-14T16:42:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 0x7f044806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04480ad970 0x7f044815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044815ebc0 0x7f044815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044806c960 con 0x7f04480ad970 2023-10-14T16:42:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f044815b9d0 con 0x7f044806c040 2023-10-14T16:42:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f044cedf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f044815bcb0 con 0x7f044815ebc0 2023-10-14T16:42:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.138+0000 7f043effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04480ad970 0x7f044815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f04480ad970 0x7f044815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59060/0 (socket says 172.21.15.29:59060) 2023-10-14T16:42:10.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 -- 172.21.15.29:0/1498151150 learned_addr learned my addr 172.21.15.29:0/1498151150 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f04477fe640 1 --2- 172.21.15.29:0/1498151150 >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 0x7f044806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f0447fff640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044815ebc0 0x7f044815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 -- 172.21.15.29:0/1498151150 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044815ebc0 msgr2=0x7f044815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044815ebc0 0x7f044815f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 -- 172.21.15.29:0/1498151150 >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 msgr2=0x7f044806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 --2- 172.21.15.29:0/1498151150 >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 0x7f044806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f043effd640 1 -- 172.21.15.29:0/1498151150 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f044815bf90 con 0x7f04480ad970 2023-10-14T16:42:10.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.139+0000 7f04477fe640 1 --2- 172.21.15.29:0/1498151150 >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 0x7f044806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:10.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.140+0000 7f043effd640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f044815e680 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f04480cf350 tx=0x7f043002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=731143931894c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f0446ffd640 1 -- 172.21.15.29:0/1498151150 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031870 con 0x7f04480ad970 2023-10-14T16:42:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f0446ffd640 1 -- 172.21.15.29:0/1498151150 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04300319d0 con 0x7f04480ad970 2023-10-14T16:42:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f0446ffd640 1 -- 172.21.15.29:0/1498151150 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430031d60 con 0x7f04480ad970 2023-10-14T16:42:10.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 -- 172.21.15.29:0/1498151150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 msgr2=0x7f044815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f044815e680 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f04480cf350 tx=0x7f043002b910 comp rx=0 tx=0).stop 2023-10-14T16:42:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 -- 172.21.15.29:0/1498151150 shutdown_connections 2023-10-14T16:42:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044815ebc0 0x7f044815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 --2- 172.21.15.29:0/1498151150 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f044815e680 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 --2- 172.21.15.29:0/1498151150 >> v2:172.21.15.186:3300/0 conn(0x7f044806c040 0x7f044806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.141+0000 7f044cedf640 1 -- 172.21.15.29:0/1498151150 >> 172.21.15.29:0/1498151150 conn(0x7f04480ce6a0 msgr2=0x7f04480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.142+0000 7f044cedf640 1 -- 172.21.15.29:0/1498151150 shutdown_connections 2023-10-14T16:42:10.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.142+0000 7f044cedf640 1 -- 172.21.15.29:0/1498151150 wait complete. 2023-10-14T16:42:10.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.143+0000 7f044cedf640 1 Processor -- start 2023-10-14T16:42:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.144+0000 7f044cedf640 1 -- start start 2023-10-14T16:42:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044806c040 0x7f044814f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 0x7f04480ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f044814fab0 con 0x7f04480ad970 2023-10-14T16:42:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04480b0810 con 0x7f044815ebc0 2023-10-14T16:42:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f044cedf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04480b0af0 con 0x7f044806c040 2023-10-14T16:42:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59068/0 (socket says 172.21.15.29:59068) 2023-10-14T16:42:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 -- 172.21.15.29:0/2369473978 learned_addr learned my addr 172.21.15.29:0/2369473978 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f04477fe640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044806c040 0x7f044814f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f0447fff640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 0x7f04480ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 -- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044806c040 msgr2=0x7f044814f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044806c040 0x7f044814f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 -- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 msgr2=0x7f04480ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 0x7f04480ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f043effd640 1 -- 172.21.15.29:0/2369473978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04300094d0 con 0x7f04480ad970 2023-10-14T16:42:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.145+0000 7f0447fff640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 0x7f04480ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:10.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f043effd640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f04300097c0 tx=0x7f04300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f043003ac60 con 0x7f04480ad970 2023-10-14T16:42:10.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f043003adc0 con 0x7f04480ad970 2023-10-14T16:42:10.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0430044610 con 0x7f04480ad970 2023-10-14T16:42:10.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04480c5770 con 0x7f04480ad970 2023-10-14T16:42:10.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.146+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04480c5d30 con 0x7f04480ad970 2023-10-14T16:42:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.148+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044806b330 con 0x7f04480ad970 2023-10-14T16:42:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.149+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f043004d450 con 0x7f04480ad970 2023-10-14T16:42:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.149+0000 7f0444ff9640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 0x7f042806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.150+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f043002d040 con 0x7f04480ad970 2023-10-14T16:42:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.150+0000 7f04477fe640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 0x7f042806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.150+0000 7f04477fe640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 0x7f042806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04481f1c70 tx=0x7f0438008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.154+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0430076db0 con 0x7f04480ad970 2023-10-14T16:42:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.381+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f04480cc550 con 0x7f042806b100 2023-10-14T16:42:10.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.517+0000 7f0444ff9640 1 -- 172.21.15.29:0/2369473978 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f043800c050 con 0x7f042806b100 2023-10-14T16:42:10.519 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir6 2023-10-14T16:42:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 msgr2=0x7f042806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 0x7f042806d5c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04481f1c70 tx=0x7f0438008040 comp rx=0 tx=0).stop 2023-10-14T16:42:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 msgr2=0x7f04480aff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f04300097c0 tx=0x7f04300046c0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 shutdown_connections 2023-10-14T16:42:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f042806b100 0x7f042806d5c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f044815ebc0 0x7f04480ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04480ad970 0x7f04480aff90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 --2- 172.21.15.29:0/2369473978 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f044806c040 0x7f044814f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 >> 172.21.15.29:0/2369473978 conn(0x7f04480ce6a0 msgr2=0x7f044815daf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 shutdown_connections 2023-10-14T16:42:10.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.521+0000 7f044cedf640 1 -- 172.21.15.29:0/2369473978 wait complete. 2023-10-14T16:42:10.545 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir6 2023-10-14T16:42:10.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir7 1m 2023-10-14T16:42:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.771+0000 7f59b2fa9640 1 Processor -- start 2023-10-14T16:42:10.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.771+0000 7f59b2fa9640 1 -- start start 2023-10-14T16:42:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.772+0000 7f59b2fa9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 0x7f59ac1618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b2fa9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59ac0661c0 0x7f59ac0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b2fa9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59ac0d0890 con 0x7f59ac06c7d0 2023-10-14T16:42:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b2fa9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ac06b1e0 con 0x7f59ac0661c0 2023-10-14T16:42:10.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b2fa9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59ac0d05b0 con 0x7f59ac15d330 2023-10-14T16:42:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b1546640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59ac0661c0 0x7f59ac0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b1546640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f59ac0661c0 0x7f59ac0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59070/0 (socket says 172.21.15.29:59070) 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59abfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f59ac06c7d0 0x7f59ac065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35184/0 (socket says 172.21.15.29:35184) 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b1546640 1 -- 172.21.15.29:0/2721882968 learned_addr learned my addr 172.21.15.29:0/2721882968 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.773+0000 7f59b0d45640 1 --2- 172.21.15.29:0/2721882968 >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 0x7f59ac1618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.774+0000 7f59b1546640 1 -- 172.21.15.29:0/2721882968 >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 msgr2=0x7f59ac1618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.774+0000 7f59b1546640 1 --2- 172.21.15.29:0/2721882968 >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 0x7f59ac1618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.774+0000 7f59b1546640 1 -- 172.21.15.29:0/2721882968 >> v1:172.21.15.29:6789/0 conn(0x7f59ac06c7d0 legacy=0x7f59ac065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.774+0000 7f59b1546640 1 -- 172.21.15.29:0/2721882968 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59ac160dd0 con 0x7f59ac0661c0 2023-10-14T16:42:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.774+0000 7f59b0d45640 1 --2- 172.21.15.29:0/2721882968 >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 0x7f59ac1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:42:10.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.775+0000 7f59b1546640 1 --2- 172.21.15.29:0/2721882968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac0661c0 0x7f59ac0ce9b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0744f0 tx=0x7f59a402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22a5fdc1f4d60545 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.775+0000 7f59ab7fe640 1 -- 172.21.15.29:0/2721882968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4031880 con 0x7f59ac0661c0 2023-10-14T16:42:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.775+0000 7f59ab7fe640 1 -- 172.21.15.29:0/2721882968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a40319e0 con 0x7f59ac0661c0 2023-10-14T16:42:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.775+0000 7f59ab7fe640 1 -- 172.21.15.29:0/2721882968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a4031d70 con 0x7f59ac0661c0 2023-10-14T16:42:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 -- 172.21.15.29:0/2721882968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac0661c0 msgr2=0x7f59ac0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/2721882968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac0661c0 0x7f59ac0ce9b0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f59ac0744f0 tx=0x7f59a402b920 comp rx=0 tx=0).stop 2023-10-14T16:42:10.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 -- 172.21.15.29:0/2721882968 shutdown_connections 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/2721882968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac0661c0 0x7f59ac0ce9b0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/2721882968 >> v2:172.21.15.186:3300/0 conn(0x7f59ac15d330 0x7f59ac1618b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 -- 172.21.15.29:0/2721882968 >> 172.21.15.29:0/2721882968 conn(0x7f59ac0d0ab0 msgr2=0x7f59ac0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 -- 172.21.15.29:0/2721882968 shutdown_connections 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.776+0000 7f59b2fa9640 1 -- 172.21.15.29:0/2721882968 wait complete. 2023-10-14T16:42:10.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.778+0000 7f59b2fa9640 1 Processor -- start 2023-10-14T16:42:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.778+0000 7f59b2fa9640 1 -- start start 2023-10-14T16:42:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59ac0661c0 0x7f59ac0af300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59ac15d330 0x7f59ac0c50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0c7290 con 0x7f59ac06c7d0 2023-10-14T16:42:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0c6940 con 0x7f59ac0661c0 2023-10-14T16:42:10.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59b2fa9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59ac0ac890 con 0x7f59ac15d330 2023-10-14T16:42:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59abfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59080/0 (socket says 172.21.15.29:59080) 2023-10-14T16:42:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.779+0000 7f59abfff640 1 -- 172.21.15.29:0/932735494 learned_addr learned my addr 172.21.15.29:0/932735494 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59b0d45640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59ac0661c0 0x7f59ac0af300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59b1546640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59ac15d330 0x7f59ac0c50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 -- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59ac15d330 msgr2=0x7f59ac0c50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59ac15d330 0x7f59ac0c50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 -- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59ac0661c0 msgr2=0x7f59ac0af300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59ac0661c0 0x7f59ac0af300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 -- 172.21.15.29:0/932735494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a40094d0 con 0x7f59ac06c7d0 2023-10-14T16:42:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59abfff640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c7a0 tx=0x7f59a000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.780+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a000a640 con 0x7f59ac06c7d0 2023-10-14T16:42:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.781+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a000a7a0 con 0x7f59ac06c7d0 2023-10-14T16:42:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.781+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59ac0af840 con 0x7f59ac06c7d0 2023-10-14T16:42:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.781+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59a000ab10 con 0x7f59ac06c7d0 2023-10-14T16:42:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.781+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59ac0acb50 con 0x7f59ac06c7d0 2023-10-14T16:42:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.782+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59ac0c7400 con 0x7f59ac06c7d0 2023-10-14T16:42:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.783+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f59a000ac70 con 0x7f59ac06c7d0 2023-10-14T16:42:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.784+0000 7f59a97fa640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 0x7f599806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.784+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f59a00892b0 con 0x7f59ac06c7d0 2023-10-14T16:42:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.784+0000 7f59b0d45640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 0x7f599806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.785+0000 7f59b0d45640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 0x7f599806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59ac160080 tx=0x7f599c008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:10.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:10.788+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59a0053fa0 con 0x7f59ac06c7d0 2023-10-14T16:42:11.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.009+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f59ac161390 con 0x7f599806b100 2023-10-14T16:42:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.151+0000 7f59a97fa640 1 -- 172.21.15.29:0/932735494 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f599c00a1f0 con 0x7f599806b100 2023-10-14T16:42:11.153 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir7 2023-10-14T16:42:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 msgr2=0x7f599806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 0x7f599806d5c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f59ac160080 tx=0x7f599c008040 comp rx=0 tx=0).stop 2023-10-14T16:42:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 msgr2=0x7f59ac157f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f59a000c7a0 tx=0x7f59a000cc60 comp rx=0 tx=0).stop 2023-10-14T16:42:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 shutdown_connections 2023-10-14T16:42:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59ac15d330 0x7f59ac0c50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59ac06c7d0 0x7f59ac157f50 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f599806b100 0x7f599806d5c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 --2- 172.21.15.29:0/932735494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59ac0661c0 0x7f59ac0af300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 >> 172.21.15.29:0/932735494 conn(0x7f59ac0d0ab0 msgr2=0x7f59ac15bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 shutdown_connections 2023-10-14T16:42:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.155+0000 7f59b2fa9640 1 -- 172.21.15.29:0/932735494 wait complete. 2023-10-14T16:42:11.182 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir7 2023-10-14T16:42:11.182 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir8 1m 2023-10-14T16:42:11.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:42:11.264 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:42:11.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.407+0000 7f69bdfc0640 1 Processor -- start 2023-10-14T16:42:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.407+0000 7f69bdfc0640 1 -- start start 2023-10-14T16:42:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.408+0000 7f69bdfc0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 0x7f69b8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.408+0000 7f69bdfc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 0x7f69b806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.408+0000 7f69bdfc0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69b80cb030 con 0x7f69b80c9ca0 2023-10-14T16:42:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.408+0000 7f69bdfc0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69b80c91c0 con 0x7f69b815ad70 2023-10-14T16:42:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.408+0000 7f69bdfc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c94a0 con 0x7f69b80caac0 2023-10-14T16:42:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.409+0000 7f69b77fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 0x7f69b8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.409+0000 7f69b77fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 0x7f69b8069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:42946/0 (socket says 172.21.15.29:42946) 2023-10-14T16:42:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.409+0000 7f69b77fe640 1 -- 172.21.15.29:0/199340351 learned_addr learned my addr 172.21.15.29:0/199340351 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.409+0000 7f69b7fff640 1 --2- 172.21.15.29:0/199340351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 0x7f69b806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.410+0000 7f69b7fff640 1 -- 172.21.15.29:0/199340351 >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 msgr2=0x7f69b8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.410+0000 7f69b7fff640 1 --2- 172.21.15.29:0/199340351 >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 0x7f69b8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.410+0000 7f69b7fff640 1 -- 172.21.15.29:0/199340351 >> v1:172.21.15.186:6789/0 conn(0x7f69b80c9ca0 legacy=0x7f69b80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.410+0000 7f69b7fff640 1 -- 172.21.15.29:0/199340351 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b80c9780 con 0x7f69b80caac0 2023-10-14T16:42:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.412+0000 7f69b7fff640 1 --2- 172.21.15.29:0/199340351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 0x7f69b806a2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b002beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dddc15a1c0b39ae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.412+0000 7f69b67fc640 1 -- 172.21.15.29:0/199340351 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b002cc60 con 0x7f69b80caac0 2023-10-14T16:42:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.412+0000 7f69b67fc640 1 -- 172.21.15.29:0/199340351 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69b002cdc0 con 0x7f69b80caac0 2023-10-14T16:42:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.412+0000 7f69b67fc640 1 -- 172.21.15.29:0/199340351 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0034660 con 0x7f69b80caac0 2023-10-14T16:42:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 -- 172.21.15.29:0/199340351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 msgr2=0x7f69b806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/199340351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 0x7f69b806a2e0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f69b0009d90 tx=0x7f69b002beb0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 -- 172.21.15.29:0/199340351 shutdown_connections 2023-10-14T16:42:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/199340351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80caac0 0x7f69b806a2e0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/199340351 >> v2:172.21.15.186:3300/0 conn(0x7f69b815ad70 0x7f69b8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 -- 172.21.15.29:0/199340351 >> 172.21.15.29:0/199340351 conn(0x7f69b80ce9b0 msgr2=0x7f69b80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 -- 172.21.15.29:0/199340351 shutdown_connections 2023-10-14T16:42:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.413+0000 7f69bdfc0640 1 -- 172.21.15.29:0/199340351 wait complete. 2023-10-14T16:42:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.415+0000 7f69bdfc0640 1 Processor -- start 2023-10-14T16:42:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.415+0000 7f69bdfc0640 1 -- start start 2023-10-14T16:42:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80c9ca0 0x7f69b80af5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69b815ad70 0x7f69b80c8670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b815c820 con 0x7f69b80caac0 2023-10-14T16:42:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c4420 con 0x7f69b815ad70 2023-10-14T16:42:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69bdfc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69b80c4700 con 0x7f69b80c9ca0 2023-10-14T16:42:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69b6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34976/0 (socket says 172.21.15.29:34976) 2023-10-14T16:42:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.416+0000 7f69b6ffd640 1 -- 172.21.15.29:0/3409716117 learned_addr learned my addr 172.21.15.29:0/3409716117 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b77fe640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80c9ca0 0x7f69b80af5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b7fff640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69b815ad70 0x7f69b80c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 -- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80c9ca0 msgr2=0x7f69b80af5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80c9ca0 0x7f69b80af5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 -- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69b815ad70 msgr2=0x7f69b80c8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69b815ad70 0x7f69b80c8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 -- 172.21.15.29:0/3409716117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69b0009a70 con 0x7f69b80caac0 2023-10-14T16:42:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69b6ffd640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c450 tx=0x7f69ac00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac0052d0 con 0x7f69b80caac0 2023-10-14T16:42:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac005430 con 0x7f69b80caac0 2023-10-14T16:42:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac012870 con 0x7f69b80caac0 2023-10-14T16:42:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.417+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69b80c49e0 con 0x7f69b80caac0 2023-10-14T16:42:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.418+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69b80afb00 con 0x7f69b80caac0 2023-10-14T16:42:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.419+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69b804e160 con 0x7f69b80caac0 2023-10-14T16:42:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.420+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f69ac0129d0 con 0x7f69b80caac0 2023-10-14T16:42:11.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.421+0000 7f699ffff640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 0x7f69a406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:11.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.421+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f69ac089820 con 0x7f69b80caac0 2023-10-14T16:42:11.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.421+0000 7f69b77fe640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 0x7f69a406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:11.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.422+0000 7f69b77fe640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 0x7f69a406d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009ac0 tx=0x7f69a8008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:11.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.425+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69ac054340 con 0x7f69b80caac0 2023-10-14T16:42:11.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.650+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f69b80ad4d0 con 0x7f69a406b100 2023-10-14T16:42:11.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.796+0000 7f699ffff640 1 -- 172.21.15.29:0/3409716117 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f69a8009710 con 0x7f69a406b100 2023-10-14T16:42:11.799 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir8 2023-10-14T16:42:11.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.799+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 msgr2=0x7f69a406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.799+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 0x7f69a406d5c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69a8009ac0 tx=0x7f69a8008040 comp rx=0 tx=0).stop 2023-10-14T16:42:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 msgr2=0x7f69b815d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f69ac00c450 tx=0x7f69ac00c910 comp rx=0 tx=0).stop 2023-10-14T16:42:11.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 shutdown_connections 2023-10-14T16:42:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f69a406b100 0x7f69a406d5c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69b815ad70 0x7f69b80c8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69b80caac0 0x7f69b815d3a0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 --2- 172.21.15.29:0/3409716117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69b80c9ca0 0x7f69b80af5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:11.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 >> 172.21.15.29:0/3409716117 conn(0x7f69b80ce9b0 msgr2=0x7f69b8070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 shutdown_connections 2023-10-14T16:42:11.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:11.800+0000 7f69bdfc0640 1 -- 172.21.15.29:0/3409716117 wait complete. 2023-10-14T16:42:11.822 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir8 2023-10-14T16:42:11.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/dir9 1m 2023-10-14T16:42:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.096+0000 7f110635e640 1 Processor -- start 2023-10-14T16:42:12.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.096+0000 7f110635e640 1 -- start start 2023-10-14T16:42:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110635e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f110006c040 0x7f110006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110635e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000ad970 0x7f110015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110635e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f110006c960 con 0x7f110015ebc0 2023-10-14T16:42:12.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110635e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f110015b9d0 con 0x7f110006c040 2023-10-14T16:42:12.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110635e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f110015bcb0 con 0x7f11000ad970 2023-10-14T16:42:12.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110535c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f110006c040 0x7f110006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:12.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f1105b5d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f110015ebc0 0x7f11000cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36776/0 (socket says 172.21.15.29:36776) 2023-10-14T16:42:12.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f1105b5d640 1 -- 172.21.15.29:0/1106152532 learned_addr learned my addr 172.21.15.29:0/1106152532 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:42:12.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.098+0000 7f110535c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f110006c040 0x7f110006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34990/0 (socket says 172.21.15.29:34990) 2023-10-14T16:42:12.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f1104b5b640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000ad970 0x7f110015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:12.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f110535c640 1 -- 172.21.15.29:0/1106152532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000ad970 msgr2=0x7f110015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:42:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f110535c640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000ad970 0x7f110015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f110535c640 1 -- 172.21.15.29:0/1106152532 >> v1:172.21.15.29:6789/0 conn(0x7f110015ebc0 legacy=0x7f11000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f110535c640 1 -- 172.21.15.29:0/1106152532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f110015bf90 con 0x7f110006c040 2023-10-14T16:42:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.099+0000 7f10f7fff640 1 -- 172.21.15.29:0/1106152532 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1542423169 0 0) 0x7f110006c960 con 0x7f110015ebc0 2023-10-14T16:42:12.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.100+0000 7f110535c640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110006c420 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0097d0 tx=0x7f10fc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47eb6aecc1c53e3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.100+0000 7f10f7fff640 1 -- 172.21.15.29:0/1106152532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc031800 con 0x7f110006c040 2023-10-14T16:42:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.100+0000 7f10f7fff640 1 -- 172.21.15.29:0/1106152532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc031960 con 0x7f110006c040 2023-10-14T16:42:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.100+0000 7f10f7fff640 1 -- 172.21.15.29:0/1106152532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc031cf0 con 0x7f110006c040 2023-10-14T16:42:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 -- 172.21.15.29:0/1106152532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 msgr2=0x7f110006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110006c420 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0097d0 tx=0x7f10fc02b8a0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 -- 172.21.15.29:0/1106152532 shutdown_connections 2023-10-14T16:42:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11000ad970 0x7f110015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 --2- 172.21.15.29:0/1106152532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110006c420 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 -- 172.21.15.29:0/1106152532 >> 172.21.15.29:0/1106152532 conn(0x7f11000ce6a0 msgr2=0x7f11000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 -- 172.21.15.29:0/1106152532 shutdown_connections 2023-10-14T16:42:12.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.101+0000 7f110635e640 1 -- 172.21.15.29:0/1106152532 wait complete. 2023-10-14T16:42:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.103+0000 7f110635e640 1 Processor -- start 2023-10-14T16:42:12.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.103+0000 7f110635e640 1 -- start start 2023-10-14T16:42:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000ad970 0x7f11001f1d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f110015ebc0 0x7f11000b31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11001f2260 con 0x7f110006c040 2023-10-14T16:42:12.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f110014a3c0 con 0x7f11000ad970 2023-10-14T16:42:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110635e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11000b1040 con 0x7f110015ebc0 2023-10-14T16:42:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34996/0 (socket says 172.21.15.29:34996) 2023-10-14T16:42:12.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 -- 172.21.15.29:0/2301878780 learned_addr learned my addr 172.21.15.29:0/2301878780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:42:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 -- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f110015ebc0 msgr2=0x7f11000b31f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f110015ebc0 0x7f11000b31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 -- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000ad970 msgr2=0x7f11001f1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000ad970 0x7f11001f1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.104+0000 7f110535c640 1 -- 172.21.15.29:0/2301878780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10fc009400 con 0x7f110006c040 2023-10-14T16:42:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f110535c640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f10fc034040 tx=0x7f10fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc03ad20 con 0x7f110006c040 2023-10-14T16:42:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10fc044400 con 0x7f110006c040 2023-10-14T16:42:12.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10fc03aea0 con 0x7f110006c040 2023-10-14T16:42:12.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11000b01b0 con 0x7f110006c040 2023-10-14T16:42:12.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.105+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11000b0710 con 0x7f110006c040 2023-10-14T16:42:12.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.106+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f110006b330 con 0x7f110006c040 2023-10-14T16:42:12.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.107+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 11) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f10fc04d440 con 0x7f110006c040 2023-10-14T16:42:12.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.108+0000 7f10f5ffb640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 0x7f10e406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:42:12.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.108+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f10fc02e070 con 0x7f110006c040 2023-10-14T16:42:12.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.109+0000 7f1104b5b640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 0x7f10e406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:42:12.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.109+0000 7f1104b5b640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 0x7f10e406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10f0006fd0 tx=0x7f10f0008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:42:12.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.113+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10fc076c40 con 0x7f110006c040 2023-10-14T16:42:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.333+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f11000cc550 con 0x7f10e406b100 2023-10-14T16:42:12.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.481+0000 7f10f5ffb640 1 -- 172.21.15.29:0/2301878780 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f10f000be20 con 0x7f10e406b100 2023-10-14T16:42:12.484 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/dir9 2023-10-14T16:42:12.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.484+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 msgr2=0x7f10e406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 0x7f10e406d5c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f10f0006fd0 tx=0x7f10f0008040 comp rx=0 tx=0).stop 2023-10-14T16:42:12.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 msgr2=0x7f110014a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:42:12.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f10fc034040 tx=0x7f10fc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 shutdown_connections 2023-10-14T16:42:12.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f110015ebc0 0x7f11000b31f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f10e406b100 0x7f10e406d5c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11000ad970 0x7f11001f1d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 --2- 172.21.15.29:0/2301878780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f110006c040 0x7f110014a660 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:42:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 >> 172.21.15.29:0/2301878780 conn(0x7f11000ce6a0 msgr2=0x7f110006a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:42:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 shutdown_connections 2023-10-14T16:42:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:42:12.485+0000 7f110635e640 1 -- 172.21.15.29:0/2301878780 wait complete. 2023-10-14T16:42:12.511 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/dir9 2023-10-14T16:42:41.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:42:41.406 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:43:11.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:43:11.561 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:43:41.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:43:41.705 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:44:11.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:44:11.856 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:44:41.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:44:42.005 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:45:12.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:45:12.157 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:45:42.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:45:42.323 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:46:12.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:46:12.472 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:46:42.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:46:42.626 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:47:00.451 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:47:00.450+0000 7f17f82f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:47:00.451 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:47:00.450+0000 7f17f82f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:47:00.452 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:47:00.450+0000 7f17f82f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:47:00.452 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:47:00.450+0000 7f17f82f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:47:00.452 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:47:00.450+0000 7f17f82f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:47:12.764 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:47:12.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:47:42.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:47:42.924 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:48:13.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:48:13.076 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:48:43.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:48:43.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:49:13.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:49:13.373 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:49:43.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:49:43.527 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:50:13.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:50:13.666 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:50:43.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:50:43.849 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:51:13.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:51:14.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:51:44.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:51:44.149 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:52:14.284 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:52:14.294 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:52:44.426 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:52:44.436 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:53:14.610 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:53:14.619 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:53:21.513 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir0 1m 2023-10-14T16:53:21.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.750+0000 7f811be6f640 1 Processor -- start 2023-10-14T16:53:21.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.750+0000 7f811be6f640 1 -- start start 2023-10-14T16:53:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.751+0000 7f811be6f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f8114167a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811be6f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81140cd390 con 0x7f811415d130 2023-10-14T16:53:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811be6f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81140cc2f0 con 0x7f811406b8c0 2023-10-14T16:53:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811be6f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81140cc5d0 con 0x7f811406b1a0 2023-10-14T16:53:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811940a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f811415d130 0x7f81140cbaa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39286/0 (socket says 172.21.15.29:39286) 2023-10-14T16:53:21.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811940a640 1 -- 172.21.15.29:0/3859467363 learned_addr learned my addr 172.21.15.29:0/3859467363 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1185339717 0 0) 0x7f81140cd390 con 0x7f811415d130 2023-10-14T16:53:21.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80fc003520 con 0x7f811415d130 2023-10-14T16:53:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.752+0000 7f811a40c640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f8114167a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825565474 0 0) 0x7f80fc003520 con 0x7f811415d130 2023-10-14T16:53:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 msgr2=0x7f8114167a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f8114167a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 >> v1:172.21.15.186:6789/0 conn(0x7f811406b8c0 legacy=0x7f811406bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81140cc8b0 con 0x7f811415d130 2023-10-14T16:53:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8108001670 con 0x7f811415d130 2023-10-14T16:53:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 >> v1:172.21.15.29:6789/0 conn(0x7f811415d130 legacy=0x7f81140cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f80fc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc004020 0x7f80fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80fc004d00 0x7f80fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140cc5d0 con 0x7f80fc004020 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bbc0 con 0x7f80fc004d00 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80fc00bd80 con 0x7f811406b1a0 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.753+0000 7f811940a640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc004020 0x7f80fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811a40c640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80fc004d00 0x7f80fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f8119c0b640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f80fc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811940a640 1 -- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 msgr2=0x7f80fc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811940a640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f80fc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811940a640 1 -- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80fc004d00 msgr2=0x7f80fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:21.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811940a640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80fc004d00 0x7f80fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811940a640 1 -- 172.21.15.29:0/3859467363 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc003520 con 0x7f80fc004020 2023-10-14T16:53:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f811a40c640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80fc004d00 0x7f80fc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.754+0000 7f8119c0b640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f811406b1a0 0x7f80fc0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:21.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.755+0000 7f811940a640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc004020 0x7f80fc0047c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f810800c0e0 tx=0x7f80fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e2398808d548945 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.755+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108005ad0 con 0x7f80fc004020 2023-10-14T16:53:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.755+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8108005c30 con 0x7f80fc004020 2023-10-14T16:53:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.755+0000 7f8118c09640 1 -- 172.21.15.29:0/3859467363 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810802ca30 con 0x7f80fc004020 2023-10-14T16:53:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 -- 172.21.15.29:0/3859467363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc004020 msgr2=0x7f80fc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 --2- 172.21.15.29:0/3859467363 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80fc004020 0x7f80fc0047c0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f810800c0e0 tx=0x7f80fc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811a40c640 1 -- 172.21.15.29:0/3859467363 reap_dead start 2023-10-14T16:53:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 -- 172.21.15.29:0/3859467363 shutdown_connections 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 -- 172.21.15.29:0/3859467363 >> 172.21.15.29:0/3859467363 conn(0x7f81140d0ad0 msgr2=0x7f81140d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 -- 172.21.15.29:0/3859467363 shutdown_connections 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.756+0000 7f811be6f640 1 -- 172.21.15.29:0/3859467363 wait complete. 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.758+0000 7f811be6f640 1 Processor -- start 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.758+0000 7f811be6f640 1 -- start start 2023-10-14T16:53:21.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80fc004020 0x7f811409df50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 0x7f81140c4ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81140c53e0 con 0x7f81140c7150 2023-10-14T16:53:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811409cbd0 con 0x7f81140c75b0 2023-10-14T16:53:21.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811be6f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f811409ceb0 con 0x7f80fc004020 2023-10-14T16:53:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811940a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811940a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53444/0 (socket says 172.21.15.29:53444) 2023-10-14T16:53:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.759+0000 7f811940a640 1 -- 172.21.15.29:0/115994820 learned_addr learned my addr 172.21.15.29:0/115994820 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f8119c0b640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80fc004020 0x7f811409df50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811a40c640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 0x7f81140c4ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 -- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80fc004020 msgr2=0x7f811409df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80fc004020 0x7f811409df50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 -- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 msgr2=0x7f81140c4ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 0x7f81140c4ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:21.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 -- 172.21.15.29:0/115994820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811409d190 con 0x7f81140c7150 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811a40c640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 0x7f81140c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.760+0000 7f811940a640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f810802c390 tx=0x7f8108001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.761+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f810803c040 con 0x7f81140c7150 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.761+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f810802cc00 con 0x7f81140c7150 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.761+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8108007070 con 0x7f81140c7150 2023-10-14T16:53:21.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.761+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811409d470 con 0x7f81140c7150 2023-10-14T16:53:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.761+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811409da30 con 0x7f81140c7150 2023-10-14T16:53:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.762+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811404e160 con 0x7f81140c7150 2023-10-14T16:53:21.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.763+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f8108032070 con 0x7f81140c7150 2023-10-14T16:53:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.764+0000 7f81127fc640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 0x7f810006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.764+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f81080aaf50 con 0x7f81140c7150 2023-10-14T16:53:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.764+0000 7f8119c0b640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 0x7f810006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:21.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.765+0000 7f8119c0b640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 0x7f810006d560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8104007fd0 tx=0x7f8104009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:21.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.770+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8108075c90 con 0x7f81140c7150 2023-10-14T16:53:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:21.990+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f81140cce70 con 0x7f810006b0a0 2023-10-14T16:53:22.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.199+0000 7f81127fc640 1 -- 172.21.15.29:0/115994820 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f810400d050 con 0x7f810006b0a0 2023-10-14T16:53:22.200 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir0 2023-10-14T16:53:22.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 msgr2=0x7f810006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 0x7f810006d560 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8104007fd0 tx=0x7f8104009040 comp rx=0 tx=0).stop 2023-10-14T16:53:22.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 msgr2=0x7f81140c4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f810802c390 tx=0x7f8108001930 comp rx=0 tx=0).stop 2023-10-14T16:53:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 shutdown_connections 2023-10-14T16:53:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f810006b0a0 0x7f810006d560 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81140c75b0 0x7f81140c4ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81140c7150 0x7f81140c4960 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 --2- 172.21.15.29:0/115994820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80fc004020 0x7f811409df50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 >> 172.21.15.29:0/115994820 conn(0x7f81140d0ad0 msgr2=0x7f811406e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:22.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 shutdown_connections 2023-10-14T16:53:22.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.203+0000 7f811be6f640 1 -- 172.21.15.29:0/115994820 wait complete. 2023-10-14T16:53:22.228 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir0 2023-10-14T16:53:22.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir1 1m 2023-10-14T16:53:22.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.456+0000 7fc461f2e640 1 Processor -- start 2023-10-14T16:53:22.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.457+0000 7fc461f2e640 1 -- start start 2023-10-14T16:53:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.458+0000 7fc461f2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc45c0b8480 0x7fc45c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.458+0000 7fc461f2e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc45c0d0be0 0x7fc45c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.458+0000 7fc461f2e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc45c15d150 con 0x7fc45c15cd70 2023-10-14T16:53:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.458+0000 7fc461f2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c0d1180 con 0x7fc45c0b8480 2023-10-14T16:53:22.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.458+0000 7fc461f2e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc45c0d1460 con 0x7fc45c0d0be0 2023-10-14T16:53:22.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc453fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc45c0b8480 0x7fc45c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:22.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc460f2c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc45c15cd70 0x7fc45c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39288/0 (socket says 172.21.15.29:39288) 2023-10-14T16:53:22.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc460f2c640 1 -- 172.21.15.29:0/450571899 learned_addr learned my addr 172.21.15.29:0/450571899 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:22.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc46172d640 1 --2- 172.21.15.29:0/450571899 >> v2:172.21.15.186:3300/0 conn(0x7fc45c0d0be0 0x7fc45c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:22.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc453fff640 1 -- 172.21.15.29:0/450571899 >> v2:172.21.15.186:3300/0 conn(0x7fc45c0d0be0 msgr2=0x7fc45c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:22.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc453fff640 1 --2- 172.21.15.29:0/450571899 >> v2:172.21.15.186:3300/0 conn(0x7fc45c0d0be0 0x7fc45c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc453fff640 1 -- 172.21.15.29:0/450571899 >> v1:172.21.15.29:6789/0 conn(0x7fc45c15cd70 legacy=0x7fc45c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.459+0000 7fc453fff640 1 -- 172.21.15.29:0/450571899 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc45c0d1740 con 0x7fc45c0b8480 2023-10-14T16:53:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.461+0000 7fc453fff640 1 --2- 172.21.15.29:0/450571899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c0cd7c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0b8910 tx=0x7fc44c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a6607775752b8f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.461+0000 7fc4537fe640 1 -- 172.21.15.29:0/450571899 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c02e070 con 0x7fc45c0b8480 2023-10-14T16:53:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.461+0000 7fc4537fe640 1 -- 172.21.15.29:0/450571899 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c0315a0 con 0x7fc45c0b8480 2023-10-14T16:53:22.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.461+0000 7fc4537fe640 1 -- 172.21.15.29:0/450571899 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc44c031930 con 0x7fc45c0b8480 2023-10-14T16:53:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 -- 172.21.15.29:0/450571899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 msgr2=0x7fc45c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 --2- 172.21.15.29:0/450571899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c0cd7c0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7fc45c0b8910 tx=0x7fc44c02b840 comp rx=0 tx=0).stop 2023-10-14T16:53:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 -- 172.21.15.29:0/450571899 shutdown_connections 2023-10-14T16:53:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 --2- 172.21.15.29:0/450571899 >> v2:172.21.15.186:3300/0 conn(0x7fc45c0d0be0 0x7fc45c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 --2- 172.21.15.29:0/450571899 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c0cd7c0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 -- 172.21.15.29:0/450571899 >> 172.21.15.29:0/450571899 conn(0x7fc45c0cf570 msgr2=0x7fc45c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 -- 172.21.15.29:0/450571899 shutdown_connections 2023-10-14T16:53:22.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.462+0000 7fc461f2e640 1 -- 172.21.15.29:0/450571899 wait complete. 2023-10-14T16:53:22.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.464+0000 7fc461f2e640 1 Processor -- start 2023-10-14T16:53:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.464+0000 7fc461f2e640 1 -- start start 2023-10-14T16:53:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc45c0d0be0 0x7fc45c0c8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45c15cd70 0x7fc45c0cbbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15dc70 con 0x7fc45c0b8480 2023-10-14T16:53:22.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c0cb400 con 0x7fc45c15cd70 2023-10-14T16:53:22.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc461f2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc45c15d340 con 0x7fc45c0d0be0 2023-10-14T16:53:22.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc460f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:22.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc460f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53474/0 (socket says 172.21.15.29:53474) 2023-10-14T16:53:22.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.465+0000 7fc460f2c640 1 -- 172.21.15.29:0/247980834 learned_addr learned my addr 172.21.15.29:0/247980834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 -- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc45c0d0be0 msgr2=0x7fc45c0c8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc45c0d0be0 0x7fc45c0c8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 -- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45c15cd70 msgr2=0x7fc45c0cbbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc46172d640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45c15cd70 0x7fc45c0cbbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:22.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45c15cd70 0x7fc45c0cbbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 -- 172.21.15.29:0/247980834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c009400 con 0x7fc45c0b8480 2023-10-14T16:53:22.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc460f2c640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c4a0 tx=0x7fc45800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:22.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458005ce0 con 0x7fc45c0b8480 2023-10-14T16:53:22.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc458005e40 con 0x7fc45c0b8480 2023-10-14T16:53:22.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45c15d620 con 0x7fc45c0b8480 2023-10-14T16:53:22.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.466+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc458007270 con 0x7fc45c0b8480 2023-10-14T16:53:22.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.467+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45c0cc0f0 con 0x7fc45c0b8480 2023-10-14T16:53:22.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.468+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc45c04e9b0 con 0x7fc45c0b8480 2023-10-14T16:53:22.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.469+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fc4580073d0 con 0x7fc45c0b8480 2023-10-14T16:53:22.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.469+0000 7fc4517fa640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 0x7fc44806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:22.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.470+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fc458088f80 con 0x7fc45c0b8480 2023-10-14T16:53:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.470+0000 7fc453fff640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 0x7fc44806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.470+0000 7fc453fff640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 0x7fc44806d560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c030000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:22.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.474+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc458053cc0 con 0x7fc45c0b8480 2023-10-14T16:53:22.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.694+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc45c0d1a20 con 0x7fc44806b0a0 2023-10-14T16:53:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.897+0000 7fc4517fa640 1 -- 172.21.15.29:0/247980834 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fc44c031990 con 0x7fc44806b0a0 2023-10-14T16:53:22.898 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir1 2023-10-14T16:53:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 msgr2=0x7fc44806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 0x7fc44806d560 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fc44c036000 tx=0x7fc44c030000 comp rx=0 tx=0).stop 2023-10-14T16:53:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 msgr2=0x7fc45c1539c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:22.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7fc45800c4a0 tx=0x7fc45800c960 comp rx=0 tx=0).stop 2023-10-14T16:53:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 shutdown_connections 2023-10-14T16:53:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc44806b0a0 0x7fc44806d560 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc45c15cd70 0x7fc45c0cbbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc45c0d0be0 0x7fc45c0c8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 --2- 172.21.15.29:0/247980834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc45c0b8480 0x7fc45c1539c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:22.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 >> 172.21.15.29:0/247980834 conn(0x7fc45c0cf570 msgr2=0x7fc45c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:22.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 shutdown_connections 2023-10-14T16:53:22.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:22.901+0000 7fc461f2e640 1 -- 172.21.15.29:0/247980834 wait complete. 2023-10-14T16:53:22.923 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir1 2023-10-14T16:53:22.924 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir2 1m 2023-10-14T16:53:23.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.148+0000 7f565a651640 1 Processor -- start 2023-10-14T16:53:23.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.148+0000 7f565a651640 1 -- start start 2023-10-14T16:53:23.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.149+0000 7f565a651640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.149+0000 7f565a651640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 0x7f5654069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.149+0000 7f565a651640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56540cb140 con 0x7f565406a2e0 2023-10-14T16:53:23.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f565a651640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5654161a50 con 0x7f56540c9000 2023-10-14T16:53:23.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f565a651640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565406a750 con 0x7f56540addb0 2023-10-14T16:53:23.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f56537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 0x7f5654069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f5653fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f56537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 0x7f5654069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:55312/0 (socket says 172.21.15.29:55312) 2023-10-14T16:53:23.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f5653fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:51148/0 (socket says 172.21.15.29:51148) 2023-10-14T16:53:23.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.150+0000 7f56537fe640 1 -- 172.21.15.29:0/1033748257 learned_addr learned my addr 172.21.15.29:0/1033748257 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:23.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.151+0000 7f5653fff640 1 -- 172.21.15.29:0/1033748257 >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 msgr2=0x7f5654069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:23.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.151+0000 7f5653fff640 1 --2- 172.21.15.29:0/1033748257 >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 0x7f5654069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.151+0000 7f5653fff640 1 -- 172.21.15.29:0/1033748257 >> v1:172.21.15.186:6789/0 conn(0x7f565406a2e0 legacy=0x7f56540cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.151+0000 7f5653fff640 1 -- 172.21.15.29:0/1033748257 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565406b030 con 0x7f56540addb0 2023-10-14T16:53:23.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.153+0000 7f5653fff640 1 --2- 172.21.15.29:0/1033748257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db62f9dc907b80c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.153+0000 7f5652ffd640 1 -- 172.21.15.29:0/1033748257 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f564402cc50 con 0x7f56540addb0 2023-10-14T16:53:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.153+0000 7f5652ffd640 1 -- 172.21.15.29:0/1033748257 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564402cdb0 con 0x7f56540addb0 2023-10-14T16:53:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.153+0000 7f5652ffd640 1 -- 172.21.15.29:0/1033748257 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644034660 con 0x7f56540addb0 2023-10-14T16:53:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 -- 172.21.15.29:0/1033748257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 msgr2=0x7f56540caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 --2- 172.21.15.29:0/1033748257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5644009d90 tx=0x7f564402bea0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 -- 172.21.15.29:0/1033748257 shutdown_connections 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 --2- 172.21.15.29:0/1033748257 >> v2:172.21.15.186:3300/0 conn(0x7f56540c9000 0x7f5654069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 --2- 172.21.15.29:0/1033748257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f56540caa80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 -- 172.21.15.29:0/1033748257 >> 172.21.15.29:0/1033748257 conn(0x7f56540ce9b0 msgr2=0x7f56540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 -- 172.21.15.29:0/1033748257 shutdown_connections 2023-10-14T16:53:23.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.154+0000 7f565a651640 1 -- 172.21.15.29:0/1033748257 wait complete. 2023-10-14T16:53:23.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.156+0000 7f565a651640 1 Processor -- start 2023-10-14T16:53:23.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.156+0000 7f565a651640 1 -- start start 2023-10-14T16:53:23.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f5654160280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c9000 0x7f5654162560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c61a0 con 0x7f565406a2e0 2023-10-14T16:53:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56540c5b20 con 0x7f56540c9000 2023-10-14T16:53:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f565a651640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5654162ea0 con 0x7f56540addb0 2023-10-14T16:53:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.157+0000 7f5653fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53476/0 (socket says 172.21.15.29:53476) 2023-10-14T16:53:23.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 -- 172.21.15.29:0/2717441013 learned_addr learned my addr 172.21.15.29:0/2717441013 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:23.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f56537fe640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f5654160280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5658bee640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c9000 0x7f5654162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 -- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 msgr2=0x7f5654160280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f5654160280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 -- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c9000 msgr2=0x7f5654162560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c9000 0x7f5654162560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 -- 172.21.15.29:0/2717441013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5644009a70 con 0x7f565406a2e0 2023-10-14T16:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.158+0000 7f5653fff640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5644037040 tx=0x7f5644004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.159+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644040650 con 0x7f565406a2e0 2023-10-14T16:53:23.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.159+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56440407b0 con 0x7f565406a2e0 2023-10-14T16:53:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.159+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56540c5dc0 con 0x7f565406a2e0 2023-10-14T16:53:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.159+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5644040b60 con 0x7f565406a2e0 2023-10-14T16:53:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.159+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56541607c0 con 0x7f565406a2e0 2023-10-14T16:53:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.160+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56540c62e0 con 0x7f565406a2e0 2023-10-14T16:53:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.161+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f5644040cc0 con 0x7f565406a2e0 2023-10-14T16:53:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.162+0000 7f5650ff9640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.162+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f56440ac950 con 0x7f565406a2e0 2023-10-14T16:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.162+0000 7f56537fe640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.163+0000 7f56537fe640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56540cefd0 tx=0x7f5648008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.167+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5644077670 con 0x7f565406a2e0 2023-10-14T16:53:23.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.395+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f565406b5f0 con 0x7f564006b070 2023-10-14T16:53:23.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.599+0000 7f5650ff9640 1 -- 172.21.15.29:0/2717441013 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5648009710 con 0x7f564006b070 2023-10-14T16:53:23.600 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir2 2023-10-14T16:53:23.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.602+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 msgr2=0x7f564006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.602+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f56540cefd0 tx=0x7f5648008040 comp rx=0 tx=0).stop 2023-10-14T16:53:23.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.602+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 msgr2=0x7f56540b22c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.602+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f5644037040 tx=0x7f5644004a40 comp rx=0 tx=0).stop 2023-10-14T16:53:23.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 shutdown_connections 2023-10-14T16:53:23.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56540c9000 0x7f5654162560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56540addb0 0x7f5654160280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 --2- 172.21.15.29:0/2717441013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565406a2e0 0x7f56540b22c0 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 >> 172.21.15.29:0/2717441013 conn(0x7f56540ce9b0 msgr2=0x7f5654070bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 shutdown_connections 2023-10-14T16:53:23.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.603+0000 7f565a651640 1 -- 172.21.15.29:0/2717441013 wait complete. 2023-10-14T16:53:23.623 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir2 2023-10-14T16:53:23.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir3 1m 2023-10-14T16:53:23.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.854+0000 7fd1b74d1640 1 Processor -- start 2023-10-14T16:53:23.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.854+0000 7fd1b74d1640 1 -- start start 2023-10-14T16:53:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.855+0000 7fd1b74d1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd1b00ad970 0x7fd1b00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.855+0000 7fd1b74d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1b00ccfb0 0x7fd1b00cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.855+0000 7fd1b74d1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1b015b9d0 con 0x7fd1b006c040 2023-10-14T16:53:23.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.855+0000 7fd1b74d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b015bcb0 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.855+0000 7fd1b74d1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1b015bf90 con 0x7fd1b00ad970 2023-10-14T16:53:23.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b64cf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd1b006c040 0x7fd1b006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39292/0 (socket says 172.21.15.29:39292) 2023-10-14T16:53:23.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b64cf640 1 -- 172.21.15.29:0/1265659945 learned_addr learned my addr 172.21.15.29:0/1265659945 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:23.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b6cd0640 1 --2- 172.21.15.29:0/1265659945 >> v2:172.21.15.29:3300/0 conn(0x7fd1b00ccfb0 0x7fd1b00cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b54cd640 1 -- 172.21.15.29:0/1265659945 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293898523 0 0) 0x7fd1b015b9d0 con 0x7fd1b006c040 2023-10-14T16:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b54cd640 1 -- 172.21.15.29:0/1265659945 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd19c003520 con 0x7fd1b006c040 2023-10-14T16:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b5cce640 1 --2- 172.21.15.29:0/1265659945 >> v2:172.21.15.186:3300/0 conn(0x7fd1b00ad970 0x7fd1b00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b6cd0640 1 -- 172.21.15.29:0/1265659945 >> v2:172.21.15.186:3300/0 conn(0x7fd1b00ad970 msgr2=0x7fd1b00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:23.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.856+0000 7fd1b6cd0640 1 --2- 172.21.15.29:0/1265659945 >> v2:172.21.15.186:3300/0 conn(0x7fd1b00ad970 0x7fd1b00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.857+0000 7fd1b6cd0640 1 -- 172.21.15.29:0/1265659945 >> v1:172.21.15.29:6789/0 conn(0x7fd1b006c040 legacy=0x7fd1b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.857+0000 7fd1b6cd0640 1 -- 172.21.15.29:0/1265659945 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1b00cc270 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.858+0000 7fd1b6cd0640 1 --2- 172.21.15.29:0/1265659945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ccfb0 0x7fd1b00cd460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009d90 tx=0x7fd1a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8cc9449fb0775ba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.858+0000 7fd1b54cd640 1 -- 172.21.15.29:0/1265659945 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a802cc60 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.858+0000 7fd1b54cd640 1 -- 172.21.15.29:0/1265659945 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a802cdc0 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.858+0000 7fd1b54cd640 1 -- 172.21.15.29:0/1265659945 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8034640 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 -- 172.21.15.29:0/1265659945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ccfb0 msgr2=0x7fd1b00cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/1265659945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ccfb0 0x7fd1b00cd460 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd1a8009d90 tx=0x7fd1a802beb0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 -- 172.21.15.29:0/1265659945 shutdown_connections 2023-10-14T16:53:23.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/1265659945 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ccfb0 0x7fd1b00cd460 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/1265659945 >> v2:172.21.15.186:3300/0 conn(0x7fd1b00ad970 0x7fd1b00cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 -- 172.21.15.29:0/1265659945 >> 172.21.15.29:0/1265659945 conn(0x7fd1b00ce6a0 msgr2=0x7fd1b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 -- 172.21.15.29:0/1265659945 shutdown_connections 2023-10-14T16:53:23.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.859+0000 7fd1b74d1640 1 -- 172.21.15.29:0/1265659945 wait complete. 2023-10-14T16:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.861+0000 7fd1b74d1640 1 Processor -- start 2023-10-14T16:53:23.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.861+0000 7fd1b74d1640 1 -- start start 2023-10-14T16:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b006c040 0x7fd1b01f2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b00ccfb0 0x7fd1b00b2f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00b23d0 con 0x7fd1b00ad970 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b01f3380 con 0x7fd1b006c040 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b74d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1b00b17c0 con 0x7fd1b00ccfb0 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.862+0000 7fd1b5cce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53490/0 (socket says 172.21.15.29:53490) 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 -- 172.21.15.29:0/471189037 learned_addr learned my addr 172.21.15.29:0/471189037 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:23.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b6cd0640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b00ccfb0 0x7fd1b00b2f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b64cf640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b006c040 0x7fd1b01f2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 -- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b00ccfb0 msgr2=0x7fd1b00b2f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b00ccfb0 0x7fd1b00b2f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 -- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b006c040 msgr2=0x7fd1b01f2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b006c040 0x7fd1b01f2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.863+0000 7fd1b5cce640 1 -- 172.21.15.29:0/471189037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1a8009a70 con 0x7fd1b00ad970 2023-10-14T16:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1b5cce640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000d7d0 tx=0x7fd1a000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000b6c0 con 0x7fd1b00ad970 2023-10-14T16:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1a000b820 con 0x7fd1b00ad970 2023-10-14T16:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1a000bbb0 con 0x7fd1b00ad970 2023-10-14T16:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1b00b1aa0 con 0x7fd1b00ad970 2023-10-14T16:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.864+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1b00b2060 con 0x7fd1b00ad970 2023-10-14T16:53:23.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.865+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1b006b330 con 0x7fd1b00ad970 2023-10-14T16:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.866+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fd1a000bd10 con 0x7fd1b00ad970 2023-10-14T16:53:23.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.868+0000 7fd1a6ffd640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 0x7fd19406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.868+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fd1a008a130 con 0x7fd1b00ad970 2023-10-14T16:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.868+0000 7fd1b64cf640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 0x7fd19406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:23.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.869+0000 7fd1b64cf640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 0x7fd19406d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac007fd0 tx=0x7fd1ac009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:23.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:23.872+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1a0054dc0 con 0x7fd1b00ad970 2023-10-14T16:53:24.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.097+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fd1b00cc550 con 0x7fd19406b070 2023-10-14T16:53:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.296+0000 7fd1a6ffd640 1 -- 172.21.15.29:0/471189037 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fd1ac001d20 con 0x7fd19406b070 2023-10-14T16:53:24.297 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir3 2023-10-14T16:53:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 msgr2=0x7fd19406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 0x7fd19406d530 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd1ac007fd0 tx=0x7fd1ac009040 comp rx=0 tx=0).stop 2023-10-14T16:53:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 msgr2=0x7fd1b01f1bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd1a000d7d0 tx=0x7fd1a000dc90 comp rx=0 tx=0).stop 2023-10-14T16:53:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 shutdown_connections 2023-10-14T16:53:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1b00ccfb0 0x7fd1b00b2f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1b00ad970 0x7fd1b01f1bf0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fd19406b070 0x7fd19406d530 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 --2- 172.21.15.29:0/471189037 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1b006c040 0x7fd1b01f2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.299+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 >> 172.21.15.29:0/471189037 conn(0x7fd1b00ce6a0 msgr2=0x7fd1b015dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.300+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 shutdown_connections 2023-10-14T16:53:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.300+0000 7fd1b74d1640 1 -- 172.21.15.29:0/471189037 wait complete. 2023-10-14T16:53:24.322 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir3 2023-10-14T16:53:24.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir4 1m 2023-10-14T16:53:24.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.546+0000 7f3d57fff640 1 Processor -- start 2023-10-14T16:53:24.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.547+0000 7f3d57fff640 1 -- start start 2023-10-14T16:53:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.548+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d580ad970 0x7f3d580cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.548+0000 7f3d57fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3d580ccfb0 0x7f3d580cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.548+0000 7f3d57fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d5815b9d0 con 0x7f3d5806c040 2023-10-14T16:53:24.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.548+0000 7f3d57fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d5815bcb0 con 0x7f3d580ccfb0 2023-10-14T16:53:24.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.548+0000 7f3d57fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815bf90 con 0x7f3d580ad970 2023-10-14T16:53:24.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d56ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3d5806c040 0x7f3d5806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39298/0 (socket says 172.21.15.29:39298) 2023-10-14T16:53:24.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d56ffd640 1 -- 172.21.15.29:0/1414109129 learned_addr learned my addr 172.21.15.29:0/1414109129 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:24.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d577fe640 1 --2- 172.21.15.29:0/1414109129 >> v2:172.21.15.186:3300/0 conn(0x7f3d580ccfb0 0x7f3d580cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d567fc640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d580ad970 0x7f3d580cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426468509 0 0) 0x7f3d5815b9d0 con 0x7f3d5806c040 2023-10-14T16:53:24.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.549+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d44003520 con 0x7f3d5806c040 2023-10-14T16:53:24.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2215594539 0 0) 0x7f3d44003520 con 0x7f3d5806c040 2023-10-14T16:53:24.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d580ad970 msgr2=0x7f3d580cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:24.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d580ad970 0x7f3d580cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 >> v2:172.21.15.186:3300/0 conn(0x7f3d580ccfb0 msgr2=0x7f3d580cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:24.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 --2- 172.21.15.29:0/1414109129 >> v2:172.21.15.186:3300/0 conn(0x7f3d580ccfb0 0x7f3d580cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d580cc270 con 0x7f3d5806c040 2023-10-14T16:53:24.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3d480019f0 con 0x7f3d5806c040 2023-10-14T16:53:24.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 >> v1:172.21.15.29:6789/0 conn(0x7f3d5806c040 legacy=0x7f3d5806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44003970 0x7f3d44003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d440047c0 0x7f3d44004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d44005180 0x7f3d4400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815bcb0 con 0x7f3d44005180 2023-10-14T16:53:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5815bf90 con 0x7f3d44003970 2023-10-14T16:53:24.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4400bfa0 con 0x7f3d440047c0 2023-10-14T16:53:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d567fc640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d580ad970 0x7f3d580cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:24.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.550+0000 7f3d577fe640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d44005180 0x7f3d4400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d56ffd640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44003970 0x7f3d44003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d577fe640 1 -- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d440047c0 msgr2=0x7f3d44004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:24.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d577fe640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d440047c0 0x7f3d44004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d577fe640 1 -- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44003970 msgr2=0x7f3d44003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:24.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d577fe640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44003970 0x7f3d44003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.551+0000 7f3d577fe640 1 -- 172.21.15.29:0/1414109129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d44003520 con 0x7f3d44005180 2023-10-14T16:53:24.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.552+0000 7f3d577fe640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d44005180 0x7f3d4400b980 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009ac0 tx=0x7f3d440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcb960dd6a2a274d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:24.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.552+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5002c710 con 0x7f3d44005180 2023-10-14T16:53:24.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.552+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5002c870 con 0x7f3d44005180 2023-10-14T16:53:24.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.552+0000 7f3d55ffb640 1 -- 172.21.15.29:0/1414109129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d5002cc00 con 0x7f3d44005180 2023-10-14T16:53:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 -- 172.21.15.29:0/1414109129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d44005180 msgr2=0x7f3d4400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 --2- 172.21.15.29:0/1414109129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d44005180 0x7f3d4400b980 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009ac0 tx=0x7f3d440057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d577fe640 1 -- 172.21.15.29:0/1414109129 reap_dead start 2023-10-14T16:53:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 -- 172.21.15.29:0/1414109129 shutdown_connections 2023-10-14T16:53:24.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 -- 172.21.15.29:0/1414109129 >> 172.21.15.29:0/1414109129 conn(0x7f3d580ce6a0 msgr2=0x7f3d580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 -- 172.21.15.29:0/1414109129 shutdown_connections 2023-10-14T16:53:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.553+0000 7f3d57fff640 1 -- 172.21.15.29:0/1414109129 wait complete. 2023-10-14T16:53:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.555+0000 7f3d57fff640 1 Processor -- start 2023-10-14T16:53:24.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.555+0000 7f3d57fff640 1 -- start start 2023-10-14T16:53:24.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 0x7f3d581f0320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d58152f60 0x7f3d581533f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58153930 con 0x7f3d5814a9d0 2023-10-14T16:53:24.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d58158ca0 con 0x7f3d44005180 2023-10-14T16:53:24.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d57fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d580c4170 con 0x7f3d58152f60 2023-10-14T16:53:24.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d567fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d567fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53508/0 (socket says 172.21.15.29:53508) 2023-10-14T16:53:24.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.556+0000 7f3d567fc640 1 -- 172.21.15.29:0/3228421420 learned_addr learned my addr 172.21.15.29:0/3228421420 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:24.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d56ffd640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 0x7f3d581f0320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 -- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d58152f60 msgr2=0x7f3d581533f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d58152f60 0x7f3d581533f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 -- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 msgr2=0x7f3d581f0320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 0x7f3d581f0320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 -- 172.21.15.29:0/3228421420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d581f0970 con 0x7f3d5814a9d0 2023-10-14T16:53:24.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d56ffd640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 0x7f3d581f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:24.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d567fc640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e4e0 tx=0x7f3d4000e9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:24.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40004230 con 0x7f3d5814a9d0 2023-10-14T16:53:24.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d400043b0 con 0x7f3d5814a9d0 2023-10-14T16:53:24.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d581f0c50 con 0x7f3d5814a9d0 2023-10-14T16:53:24.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.557+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d40004520 con 0x7f3d5814a9d0 2023-10-14T16:53:24.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.558+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d581f1210 con 0x7f3d5814a9d0 2023-10-14T16:53:24.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.559+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d5806b330 con 0x7f3d5814a9d0 2023-10-14T16:53:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.560+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3d40003850 con 0x7f3d5814a9d0 2023-10-14T16:53:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.561+0000 7f3d4ffff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 0x7f3d3806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.561+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3d40089670 con 0x7f3d5814a9d0 2023-10-14T16:53:24.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.561+0000 7f3d56ffd640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 0x7f3d3806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:24.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.562+0000 7f3d56ffd640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 0x7f3d3806d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d48001f40 tx=0x7f3d48001370 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:24.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.565+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d40054280 con 0x7f3d5814a9d0 2023-10-14T16:53:24.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.788+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d580cc550 con 0x7f3d3806b070 2023-10-14T16:53:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.991+0000 7f3d4ffff640 1 -- 172.21.15.29:0/3228421420 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3d48001980 con 0x7f3d3806b070 2023-10-14T16:53:24.992 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir4 2023-10-14T16:53:24.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 msgr2=0x7f3d3806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 0x7f3d3806d530 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f3d48001f40 tx=0x7f3d48001370 comp rx=0 tx=0).stop 2023-10-14T16:53:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 msgr2=0x7f3d580c3c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000e4e0 tx=0x7f3d4000e9a0 comp rx=0 tx=0).stop 2023-10-14T16:53:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 shutdown_connections 2023-10-14T16:53:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d58152f60 0x7f3d581533f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d5814a9d0 0x7f3d580c3c30 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3d3806b070 0x7f3d3806d530 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 --2- 172.21.15.29:0/3228421420 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d44005180 0x7f3d581f0320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.994+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 >> 172.21.15.29:0/3228421420 conn(0x7f3d580ce6a0 msgr2=0x7f3d5815d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.995+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 shutdown_connections 2023-10-14T16:53:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:24.995+0000 7f3d57fff640 1 -- 172.21.15.29:0/3228421420 wait complete. 2023-10-14T16:53:25.017 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir4 2023-10-14T16:53:25.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir5 1m 2023-10-14T16:53:25.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.243+0000 7f3f6a8ec640 1 Processor -- start 2023-10-14T16:53:25.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.243+0000 7f3f6a8ec640 1 -- start start 2023-10-14T16:53:25.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.244+0000 7f3f6a8ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f640ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f6a8ec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f64070360 con 0x7f3f64159ab0 2023-10-14T16:53:25.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f6a8ec640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f640cd1f0 con 0x7f3f64159e90 2023-10-14T16:53:25.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f6a8ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6415d890 con 0x7f3f6406c850 2023-10-14T16:53:25.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f63fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3f64159ab0 0x7f3f6406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39306/0 (socket says 172.21.15.29:39306) 2023-10-14T16:53:25.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f63fff640 1 -- 172.21.15.29:0/3039951047 learned_addr learned my addr 172.21.15.29:0/3039951047 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f68e89640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f640ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3532765634 0 0) 0x7f3f64070360 con 0x7f3f64159ab0 2023-10-14T16:53:25.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.245+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f4c003520 con 0x7f3f64159ab0 2023-10-14T16:53:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506736032 0 0) 0x7f3f4c003520 con 0x7f3f64159ab0 2023-10-14T16:53:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 msgr2=0x7f3f640ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f640ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 >> v1:172.21.15.186:6789/0 conn(0x7f3f64159e90 legacy=0x7f3f6406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f640cd4d0 con 0x7f3f64159ab0 2023-10-14T16:53:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3f540019f0 con 0x7f3f64159ab0 2023-10-14T16:53:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 >> v1:172.21.15.29:6789/0 conn(0x7f3f64159ab0 legacy=0x7f3f6406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f4c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c003f20 0x7f3f4c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c004d00 0x7f3f4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6415d890 con 0x7f3f4c004d00 2023-10-14T16:53:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c00bbc0 con 0x7f3f4c003f20 2023-10-14T16:53:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c00bd80 con 0x7f3f6406c850 2023-10-14T16:53:25.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.246+0000 7f3f68e89640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c004d00 0x7f3f4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f63fff640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f4c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f637fe640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c003f20 0x7f3f4c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f68e89640 1 -- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 msgr2=0x7f3f4c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f68e89640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f6406c850 0x7f3f4c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f68e89640 1 -- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c003f20 msgr2=0x7f3f4c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f68e89640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c003f20 0x7f3f4c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.247+0000 7f3f68e89640 1 -- 172.21.15.29:0/3039951047 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c003520 con 0x7f3f4c004d00 2023-10-14T16:53:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.248+0000 7f3f68e89640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c004d00 0x7f3f4c005180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3f640ae3e0 tx=0x7f3f4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d20ed1314d5834c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.248+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c02c660 con 0x7f3f4c004d00 2023-10-14T16:53:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f5c02c7c0 con 0x7f3f4c004d00 2023-10-14T16:53:25.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f62ffd640 1 -- 172.21.15.29:0/3039951047 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f5c02cb50 con 0x7f3f4c004d00 2023-10-14T16:53:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/3039951047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c004d00 msgr2=0x7f3f4c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/3039951047 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c004d00 0x7f3f4c005180 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f3f640ae3e0 tx=0x7f3f4c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f68e89640 1 -- 172.21.15.29:0/3039951047 reap_dead start 2023-10-14T16:53:25.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/3039951047 shutdown_connections 2023-10-14T16:53:25.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/3039951047 >> 172.21.15.29:0/3039951047 conn(0x7f3f640ce9b0 msgr2=0x7f3f640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:25.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/3039951047 shutdown_connections 2023-10-14T16:53:25.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.249+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/3039951047 wait complete. 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.251+0000 7f3f6a8ec640 1 Processor -- start 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.251+0000 7f3f6a8ec640 1 -- start start 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 0x7f3f6414d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f6406fcc0 0x7f3f64070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6415d680 con 0x7f3f6415ccc0 2023-10-14T16:53:25.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f640c5a80 con 0x7f3f6406fcc0 2023-10-14T16:53:25.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f6a8ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f64154470 con 0x7f3f4c004d00 2023-10-14T16:53:25.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.252+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53528/0 (socket says 172.21.15.29:53528) 2023-10-14T16:53:25.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 -- 172.21.15.29:0/2830344334 learned_addr learned my addr 172.21.15.29:0/2830344334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:25.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f63fff640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 0x7f3f6414d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 -- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 msgr2=0x7f3f6414d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 0x7f3f6414d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 -- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f6406fcc0 msgr2=0x7f3f64070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f6406fcc0 0x7f3f64070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 -- 172.21.15.29:0/2830344334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f64154750 con 0x7f3f6415ccc0 2023-10-14T16:53:25.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f63fff640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 0x7f3f6414d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:25.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f637fe640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800e500 tx=0x7f3f5800e9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.253+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f580052e0 con 0x7f3f6415ccc0 2023-10-14T16:53:25.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.254+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f58005440 con 0x7f3f6415ccc0 2023-10-14T16:53:25.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.254+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f64154a30 con 0x7f3f6415ccc0 2023-10-14T16:53:25.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.254+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f58003810 con 0x7f3f6415ccc0 2023-10-14T16:53:25.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.254+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6414dd00 con 0x7f3f6415ccc0 2023-10-14T16:53:25.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.255+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6404e160 con 0x7f3f6415ccc0 2023-10-14T16:53:25.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.256+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f3f58003970 con 0x7f3f6415ccc0 2023-10-14T16:53:25.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.257+0000 7f3f60ff9640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 0x7f3f5006d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.257+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f3f58089850 con 0x7f3f6415ccc0 2023-10-14T16:53:25.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.257+0000 7f3f63fff640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 0x7f3f5006d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.258+0000 7f3f63fff640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 0x7f3f5006d520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3f54001460 tx=0x7f3f54001880 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.262+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f58054570 con 0x7f3f6415ccc0 2023-10-14T16:53:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.482+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f640cda90 con 0x7f3f5006b060 2023-10-14T16:53:25.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.687+0000 7f3f60ff9640 1 -- 172.21.15.29:0/2830344334 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3f54005d10 con 0x7f3f5006b060 2023-10-14T16:53:25.688 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir5 2023-10-14T16:53:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.690+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 msgr2=0x7f3f5006d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.690+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 0x7f3f5006d520 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f3f54001460 tx=0x7f3f54001880 comp rx=0 tx=0).stop 2023-10-14T16:53:25.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 msgr2=0x7f3f6415d140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f3f5800e500 tx=0x7f3f5800e9c0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 shutdown_connections 2023-10-14T16:53:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f3f5006b060 0x7f3f5006d520 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f6406fcc0 0x7f3f64070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f6415ccc0 0x7f3f6415d140 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 --2- 172.21.15.29:0/2830344334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f4c004d00 0x7f3f6414d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 >> 172.21.15.29:0/2830344334 conn(0x7f3f640ce9b0 msgr2=0x7f3f64158c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:25.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 shutdown_connections 2023-10-14T16:53:25.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.691+0000 7f3f6a8ec640 1 -- 172.21.15.29:0/2830344334 wait complete. 2023-10-14T16:53:25.712 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir5 2023-10-14T16:53:25.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir6 1m 2023-10-14T16:53:25.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.937+0000 7f707c967640 1 Processor -- start 2023-10-14T16:53:25.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.937+0000 7f707c967640 1 -- start start 2023-10-14T16:53:25.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f707c967640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f707806c040 0x7f707806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f707c967640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70780ad970 0x7f707815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f707c967640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f707806c960 con 0x7f707815ebc0 2023-10-14T16:53:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f707c967640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707815b9d0 con 0x7f70780ad970 2023-10-14T16:53:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f707c967640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f707815bcb0 con 0x7f707806c040 2023-10-14T16:53:25.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f7076ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70780ad970 0x7f707815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f7076ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70780ad970 0x7f707815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53544/0 (socket says 172.21.15.29:53544) 2023-10-14T16:53:25.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f7076ffd640 1 -- 172.21.15.29:0/3951267639 learned_addr learned my addr 172.21.15.29:0/3951267639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.939+0000 7f70777fe640 1 --2- 172.21.15.29:0/3951267639 >> v2:172.21.15.186:3300/0 conn(0x7f707806c040 0x7f707806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.940+0000 7f7076ffd640 1 -- 172.21.15.29:0/3951267639 >> v2:172.21.15.186:3300/0 conn(0x7f707806c040 msgr2=0x7f707806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:25.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.940+0000 7f7076ffd640 1 --2- 172.21.15.29:0/3951267639 >> v2:172.21.15.186:3300/0 conn(0x7f707806c040 0x7f707806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.940+0000 7f7076ffd640 1 -- 172.21.15.29:0/3951267639 >> v1:172.21.15.186:6789/0 conn(0x7f707815ebc0 legacy=0x7f70780cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.940+0000 7f7076ffd640 1 -- 172.21.15.29:0/3951267639 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707815bf90 con 0x7f70780ad970 2023-10-14T16:53:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.941+0000 7f7076ffd640 1 --2- 172.21.15.29:0/3951267639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f707815e680 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f70780cf350 tx=0x7f706c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=846744041daa0883 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f70767fc640 1 -- 172.21.15.29:0/3951267639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c030860 con 0x7f70780ad970 2023-10-14T16:53:25.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f70767fc640 1 -- 172.21.15.29:0/3951267639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c0309c0 con 0x7f70780ad970 2023-10-14T16:53:25.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f70767fc640 1 -- 172.21.15.29:0/3951267639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c030d50 con 0x7f70780ad970 2023-10-14T16:53:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 -- 172.21.15.29:0/3951267639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 msgr2=0x7f707815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 --2- 172.21.15.29:0/3951267639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f707815e680 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f70780cf350 tx=0x7f706c02b9a0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 -- 172.21.15.29:0/3951267639 shutdown_connections 2023-10-14T16:53:25.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 --2- 172.21.15.29:0/3951267639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f707815e680 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 --2- 172.21.15.29:0/3951267639 >> v2:172.21.15.186:3300/0 conn(0x7f707806c040 0x7f707806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.942+0000 7f707c967640 1 -- 172.21.15.29:0/3951267639 >> 172.21.15.29:0/3951267639 conn(0x7f70780ce6a0 msgr2=0x7f70780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:25.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.943+0000 7f707c967640 1 -- 172.21.15.29:0/3951267639 shutdown_connections 2023-10-14T16:53:25.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.943+0000 7f707c967640 1 -- 172.21.15.29:0/3951267639 wait complete. 2023-10-14T16:53:25.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.944+0000 7f707c967640 1 Processor -- start 2023-10-14T16:53:25.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.945+0000 7f707c967640 1 -- start start 2023-10-14T16:53:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f707806c040 0x7f70781e64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f707815ebc0 0x7f70780c48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70781e73a0 con 0x7f70780ad970 2023-10-14T16:53:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7078145820 con 0x7f707815ebc0 2023-10-14T16:53:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f707c967640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70780b1000 con 0x7f707806c040 2023-10-14T16:53:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53556/0 (socket says 172.21.15.29:53556) 2023-10-14T16:53:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 -- 172.21.15.29:0/2818837713 learned_addr learned my addr 172.21.15.29:0/2818837713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f70777fe640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f707806c040 0x7f70781e64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7077fff640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f707815ebc0 0x7f70780c48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 -- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f707806c040 msgr2=0x7f70781e64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f707806c040 0x7f70781e64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 -- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f707815ebc0 msgr2=0x7f70780c48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f707815ebc0 0x7f70780c48e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.946+0000 7f7076ffd640 1 -- 172.21.15.29:0/2818837713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706c0094d0 con 0x7f70780ad970 2023-10-14T16:53:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f7076ffd640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f706c02b9e0 tx=0x7f706c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c039a90 con 0x7f70780ad970 2023-10-14T16:53:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f706c039bf0 con 0x7f70780ad970 2023-10-14T16:53:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70781e6b00 con 0x7f70780ad970 2023-10-14T16:53:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706c043410 con 0x7f70780ad970 2023-10-14T16:53:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.947+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70781e70c0 con 0x7f70780ad970 2023-10-14T16:53:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.949+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707806b330 con 0x7f70780ad970 2023-10-14T16:53:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.949+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f706c04c3b0 con 0x7f70780ad970 2023-10-14T16:53:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.951+0000 7f705ffff640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 0x7f706406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.951+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f706c0ac2c0 con 0x7f70780ad970 2023-10-14T16:53:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.951+0000 7f70777fe640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 0x7f706406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.952+0000 7f70777fe640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 0x7f706406d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:25.956+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f706c076e40 con 0x7f70780ad970 2023-10-14T16:53:26.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.178+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7078149940 con 0x7f706406b070 2023-10-14T16:53:26.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.386+0000 7f705ffff640 1 -- 172.21.15.29:0/2818837713 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f7078149940 con 0x7f706406b070 2023-10-14T16:53:26.387 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir6 2023-10-14T16:53:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 msgr2=0x7f706406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 0x7f706406d530 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7068006fd0 tx=0x7f7068008040 comp rx=0 tx=0).stop 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 msgr2=0x7f70780b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f706c02b9e0 tx=0x7f706c004730 comp rx=0 tx=0).stop 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 shutdown_connections 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f706406b070 0x7f706406d530 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f707815ebc0 0x7f70780c48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70780ad970 0x7f70780b0ac0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 --2- 172.21.15.29:0/2818837713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f707806c040 0x7f70781e64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.389+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 >> 172.21.15.29:0/2818837713 conn(0x7f70780ce6a0 msgr2=0x7f707815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:26.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.390+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 shutdown_connections 2023-10-14T16:53:26.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.390+0000 7f707c967640 1 -- 172.21.15.29:0/2818837713 wait complete. 2023-10-14T16:53:26.413 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir6 2023-10-14T16:53:26.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir7 1m 2023-10-14T16:53:26.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.639+0000 7f54e876c640 1 Processor -- start 2023-10-14T16:53:26.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.639+0000 7f54e876c640 1 -- start start 2023-10-14T16:53:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e876c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54e006b660 0x7f54e006ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:26.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e876c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e00cc6b0 con 0x7f54e01428c0 2023-10-14T16:53:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e876c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54e01599b0 con 0x7f54e00c9860 2023-10-14T16:53:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e876c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54e006c290 con 0x7f54e006b660 2023-10-14T16:53:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e5d07640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f54e01428c0 0x7f54e00c9150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39322/0 (socket says 172.21.15.29:39322) 2023-10-14T16:53:26.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e5d07640 1 -- 172.21.15.29:0/2622692057 learned_addr learned my addr 172.21.15.29:0/2622692057 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.641+0000 7f54e6508640 1 --2- 172.21.15.29:0/2622692057 >> v2:172.21.15.29:3300/0 conn(0x7f54e006b660 0x7f54e006ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.642+0000 7f54e5506640 1 -- 172.21.15.29:0/2622692057 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 273976564 0 0) 0x7f54e00cc6b0 con 0x7f54e01428c0 2023-10-14T16:53:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.642+0000 7f54e5506640 1 -- 172.21.15.29:0/2622692057 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54cc005730 con 0x7f54e01428c0 2023-10-14T16:53:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.642+0000 7f54e6508640 1 -- 172.21.15.29:0/2622692057 >> v1:172.21.15.186:6789/0 conn(0x7f54e00c9860 legacy=0x7f54e00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.642+0000 7f54e6508640 1 -- 172.21.15.29:0/2622692057 >> v1:172.21.15.29:6789/0 conn(0x7f54e01428c0 legacy=0x7f54e00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.642+0000 7f54e6508640 1 -- 172.21.15.29:0/2622692057 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e006c570 con 0x7f54e006b660 2023-10-14T16:53:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.643+0000 7f54e6508640 1 --2- 172.21.15.29:0/2622692057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e006b660 0x7f54e006ba40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0097a0 tx=0x7f54dc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99d7af27d6fc82e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.643+0000 7f54e5506640 1 -- 172.21.15.29:0/2622692057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc031830 con 0x7f54e006b660 2023-10-14T16:53:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.643+0000 7f54e5506640 1 -- 172.21.15.29:0/2622692057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc031990 con 0x7f54e006b660 2023-10-14T16:53:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.643+0000 7f54e5506640 1 -- 172.21.15.29:0/2622692057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc031d00 con 0x7f54e006b660 2023-10-14T16:53:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 -- 172.21.15.29:0/2622692057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e006b660 msgr2=0x7f54e006ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 --2- 172.21.15.29:0/2622692057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e006b660 0x7f54e006ba40 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f54dc0097a0 tx=0x7f54dc02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 -- 172.21.15.29:0/2622692057 shutdown_connections 2023-10-14T16:53:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 --2- 172.21.15.29:0/2622692057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e006b660 0x7f54e006ba40 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 -- 172.21.15.29:0/2622692057 >> 172.21.15.29:0/2622692057 conn(0x7f54e00ce9b0 msgr2=0x7f54e00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 -- 172.21.15.29:0/2622692057 shutdown_connections 2023-10-14T16:53:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.644+0000 7f54e876c640 1 -- 172.21.15.29:0/2622692057 wait complete. 2023-10-14T16:53:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.646+0000 7f54e876c640 1 Processor -- start 2023-10-14T16:53:26.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.646+0000 7f54e876c640 1 -- start start 2023-10-14T16:53:26.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54e006b660 0x7f54e014f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 0x7f54e014c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e01365f0 con 0x7f54e01428c0 2023-10-14T16:53:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e01f23e0 con 0x7f54e00c9860 2023-10-14T16:53:26.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e876c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54e014cc80 con 0x7f54e006b660 2023-10-14T16:53:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.647+0000 7f54e6d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53576/0 (socket says 172.21.15.29:53576) 2023-10-14T16:53:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 -- 172.21.15.29:0/1413375596 learned_addr learned my addr 172.21.15.29:0/1413375596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6508640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54e006b660 0x7f54e014f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:26.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e5d07640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 0x7f54e014c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 -- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54e006b660 msgr2=0x7f54e014f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54e006b660 0x7f54e014f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 -- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 msgr2=0x7f54e014c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 0x7f54e014c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:26.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 -- 172.21.15.29:0/1413375596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc009400 con 0x7f54e01428c0 2023-10-14T16:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e5d07640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 0x7f54e014c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.648+0000 7f54e6d09640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f54d800c700 tx=0x7f54d800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.649+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d800aa50 con 0x7f54e01428c0 2023-10-14T16:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.649+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d800abb0 con 0x7f54e01428c0 2023-10-14T16:53:26.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.649+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e014cf20 con 0x7f54e01428c0 2023-10-14T16:53:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.649+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d800ce10 con 0x7f54e01428c0 2023-10-14T16:53:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.649+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e014d2b0 con 0x7f54e01428c0 2023-10-14T16:53:26.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.650+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e004e160 con 0x7f54e01428c0 2023-10-14T16:53:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.651+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f54d8012420 con 0x7f54e01428c0 2023-10-14T16:53:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.652+0000 7f54d6ffd640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 0x7f54c406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.652+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f54d808a080 con 0x7f54e01428c0 2023-10-14T16:53:26.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.652+0000 7f54e6508640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 0x7f54c406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:26.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.653+0000 7f54e6508640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 0x7f54c406d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54dc036000 tx=0x7f54dc03c040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:26.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.657+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54d80bcfe0 con 0x7f54e01428c0 2023-10-14T16:53:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:26.877+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f54e006cb30 con 0x7f54c406b070 2023-10-14T16:53:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.085+0000 7f54d6ffd640 1 -- 172.21.15.29:0/1413375596 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f54dc031b00 con 0x7f54c406b070 2023-10-14T16:53:27.087 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir7 2023-10-14T16:53:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 msgr2=0x7f54c406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 0x7f54c406d530 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f54dc036000 tx=0x7f54dc03c040 comp rx=0 tx=0).stop 2023-10-14T16:53:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 msgr2=0x7f54e014dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7f54d800c700 tx=0x7f54d800cbc0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 shutdown_connections 2023-10-14T16:53:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54e01428c0 0x7f54e014dfc0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f54c406b070 0x7f54c406d530 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54e00c9860 0x7f54e014c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 --2- 172.21.15.29:0/1413375596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54e006b660 0x7f54e014f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 >> 172.21.15.29:0/1413375596 conn(0x7f54e00ce9b0 msgr2=0x7f54e0070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 shutdown_connections 2023-10-14T16:53:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.089+0000 7f54e876c640 1 -- 172.21.15.29:0/1413375596 wait complete. 2023-10-14T16:53:27.110 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir7 2023-10-14T16:53:27.110 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir8 1m 2023-10-14T16:53:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.334+0000 7fbcad085640 1 Processor -- start 2023-10-14T16:53:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.334+0000 7fbcad085640 1 -- start start 2023-10-14T16:53:27.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcad085640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca80c9060 0x7fbca806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcad085640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbca806c9e0 0x7fbca806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcad085640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbca80ca560 con 0x7fbca80c9440 2023-10-14T16:53:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcad085640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbca80c9950 con 0x7fbca806c9e0 2023-10-14T16:53:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcad085640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80c9c30 con 0x7fbca80c9060 2023-10-14T16:53:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcac884640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbca80c9440 0x7fbca8164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39328/0 (socket says 172.21.15.29:39328) 2023-10-14T16:53:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbcac884640 1 -- 172.21.15.29:0/726265234 learned_addr learned my addr 172.21.15.29:0/726265234 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:27.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbc9f7fe640 1 --2- 172.21.15.29:0/726265234 >> v2:172.21.15.29:3300/0 conn(0x7fbca806c9e0 0x7fbca806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.336+0000 7fbca7fff640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca80c9060 0x7fbca806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.337+0000 7fbc9f7fe640 1 -- 172.21.15.29:0/726265234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca80c9060 msgr2=0x7fbca806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.337+0000 7fbc9f7fe640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca80c9060 0x7fbca806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.337+0000 7fbc9f7fe640 1 -- 172.21.15.29:0/726265234 >> v1:172.21.15.29:6789/0 conn(0x7fbca80c9440 legacy=0x7fbca8164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.337+0000 7fbc9f7fe640 1 -- 172.21.15.29:0/726265234 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbca80c9f10 con 0x7fbca806c9e0 2023-10-14T16:53:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.338+0000 7fbc9f7fe640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca806c9e0 0x7fbca806f980 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbc900097a0 tx=0x7fbc9002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24f7175c20617dcf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.338+0000 7fbca77fe640 1 -- 172.21.15.29:0/726265234 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9002d040 con 0x7fbca806c9e0 2023-10-14T16:53:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.338+0000 7fbca77fe640 1 -- 172.21.15.29:0/726265234 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc90031810 con 0x7fbca806c9e0 2023-10-14T16:53:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.338+0000 7fbca77fe640 1 -- 172.21.15.29:0/726265234 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc90031ba0 con 0x7fbca806c9e0 2023-10-14T16:53:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 -- 172.21.15.29:0/726265234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca806c9e0 msgr2=0x7fbca806f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca806c9e0 0x7fbca806f980 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbc900097a0 tx=0x7fbc9002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 -- 172.21.15.29:0/726265234 shutdown_connections 2023-10-14T16:53:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca806c9e0 0x7fbca806f980 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 --2- 172.21.15.29:0/726265234 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca80c9060 0x7fbca806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 -- 172.21.15.29:0/726265234 >> 172.21.15.29:0/726265234 conn(0x7fbca80ce6a0 msgr2=0x7fbca80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:27.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 -- 172.21.15.29:0/726265234 shutdown_connections 2023-10-14T16:53:27.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.339+0000 7fbcad085640 1 -- 172.21.15.29:0/726265234 wait complete. 2023-10-14T16:53:27.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.341+0000 7fbcad085640 1 Processor -- start 2023-10-14T16:53:27.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.341+0000 7fbcad085640 1 -- start start 2023-10-14T16:53:27.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca806c9e0 0x7fbca80c3fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca80c9060 0x7fbca80c4520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca81508f0 con 0x7fbca80c9440 2023-10-14T16:53:27.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbca80c8580 con 0x7fbca80c9060 2023-10-14T16:53:27.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.342+0000 7fbcad085640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbca81505a0 con 0x7fbca806c9e0 2023-10-14T16:53:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53588/0 (socket says 172.21.15.29:53588) 2023-10-14T16:53:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 -- 172.21.15.29:0/1466401591 learned_addr learned my addr 172.21.15.29:0/1466401591 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:27.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbc9f7fe640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca80c9060 0x7fbca80c4520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbca7fff640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca806c9e0 0x7fbca80c3fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 -- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca806c9e0 msgr2=0x7fbca80c3fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca806c9e0 0x7fbca80c3fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 -- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca80c9060 msgr2=0x7fbca80c4520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca80c9060 0x7fbca80c4520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.343+0000 7fbcac884640 1 -- 172.21.15.29:0/1466401591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90009400 con 0x7fbca80c9440 2023-10-14T16:53:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbcac884640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbca000cad0 tx=0x7fbca000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca000ac30 con 0x7fbca80c9440 2023-10-14T16:53:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbca000ad90 con 0x7fbca80c9440 2023-10-14T16:53:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbca0012600 con 0x7fbca80c9440 2023-10-14T16:53:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbca80c4da0 con 0x7fbca80c9440 2023-10-14T16:53:27.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.344+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbca80c5360 con 0x7fbca80c9440 2023-10-14T16:53:27.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.345+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbca8150aa0 con 0x7fbca80c9440 2023-10-14T16:53:27.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.347+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fbca0007070 con 0x7fbca80c9440 2023-10-14T16:53:27.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.347+0000 7fbca57fa640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 0x7fbc8806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:27.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.348+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fbca00898b0 con 0x7fbca80c9440 2023-10-14T16:53:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.348+0000 7fbca7fff640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 0x7fbc8806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.349+0000 7fbca7fff640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 0x7fbc8806d560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006fd0 tx=0x7fbc98008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:27.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.352+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbca8150aa0 con 0x7fbca80c9440 2023-10-14T16:53:27.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.575+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbca80c1f10 con 0x7fbc8806b0a0 2023-10-14T16:53:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.782+0000 7fbca57fa640 1 -- 172.21.15.29:0/1466401591 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fbc98009520 con 0x7fbc8806b0a0 2023-10-14T16:53:27.783 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir8 2023-10-14T16:53:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.785+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 msgr2=0x7fbc8806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.785+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 0x7fbc8806d560 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006fd0 tx=0x7fbc98008040 comp rx=0 tx=0).stop 2023-10-14T16:53:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.785+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 msgr2=0x7fbca80c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.785+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbca000cad0 tx=0x7fbca000cf90 comp rx=0 tx=0).stop 2023-10-14T16:53:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 shutdown_connections 2023-10-14T16:53:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbca80c9440 0x7fbca80c67c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbc8806b0a0 0x7fbc8806d560 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbca80c9060 0x7fbca80c4520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 --2- 172.21.15.29:0/1466401591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbca806c9e0 0x7fbca80c3fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 >> 172.21.15.29:0/1466401591 conn(0x7fbca80ce6a0 msgr2=0x7fbca80b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 shutdown_connections 2023-10-14T16:53:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:27.786+0000 7fbcad085640 1 -- 172.21.15.29:0/1466401591 wait complete. 2023-10-14T16:53:27.808 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir8 2023-10-14T16:53:27.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/dir9 1m 2023-10-14T16:53:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.035+0000 7ff5aef33640 1 Processor -- start 2023-10-14T16:53:28.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.036+0000 7ff5aef33640 1 -- start start 2023-10-14T16:53:28.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.037+0000 7ff5aef33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 0x7ff5a8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.037+0000 7ff5aef33640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5a80c9ca0 0x7ff5a806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.037+0000 7ff5aef33640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5a80cb030 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.037+0000 7ff5aef33640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5a80c91c0 con 0x7ff5a80c9ca0 2023-10-14T16:53:28.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.037+0000 7ff5aef33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c94a0 con 0x7ff5a815ad70 2023-10-14T16:53:28.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5ad4d0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff5a80ca3b0 0x7ff5a80ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39340/0 (socket says 172.21.15.29:39340) 2023-10-14T16:53:28.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5ad4d0640 1 -- 172.21.15.29:0/3455138062 learned_addr learned my addr 172.21.15.29:0/3455138062 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/3455138062 >> v2:172.21.15.186:3300/0 conn(0x7ff5a80c9ca0 0x7ff5a806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747413826 0 0) 0x7ff5a80cb030 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff594003520 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5acccf640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 0x7ff5a8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 477581968 0 0) 0x7ff594003520 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.038+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 msgr2=0x7ff5a8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 0x7ff5a8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 >> v2:172.21.15.186:3300/0 conn(0x7ff5a80c9ca0 msgr2=0x7ff5a806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/3455138062 >> v2:172.21.15.186:3300/0 conn(0x7ff5a80c9ca0 0x7ff5a806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80c9780 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff5a0001670 con 0x7ff5a80ca3b0 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 >> v1:172.21.15.29:6789/0 conn(0x7ff5a80ca3b0 legacy=0x7ff5a80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff594003970 0x7ff594003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 0x7ff5940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff594005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c91c0 con 0x7ff594004d00 2023-10-14T16:53:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c94a0 con 0x7ff594003970 2023-10-14T16:53:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff59400bab0 con 0x7ff5a815ad70 2023-10-14T16:53:28.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/3455138062 >> v2:172.21.15.186:3300/0 conn(0x7ff5a80c9ca0 0x7ff5a806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff594005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 -- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 msgr2=0x7ff5940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a815ad70 0x7ff5940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 -- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff594003970 msgr2=0x7ff594003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:28.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff594003970 0x7ff594003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.039+0000 7ff5ad4d0640 1 -- 172.21.15.29:0/3455138062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff594003520 con 0x7ff594004d00 2023-10-14T16:53:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5ad4d0640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff594005180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c110 tx=0x7ff5940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a11ae62b55913bdf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a0005ad0 con 0x7ff594004d00 2023-10-14T16:53:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5a0005c50 con 0x7ff594004d00 2023-10-14T16:53:28.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5a77fe640 1 -- 172.21.15.29:0/3455138062 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5a002c830 con 0x7ff594004d00 2023-10-14T16:53:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5aef33640 1 -- 172.21.15.29:0/3455138062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 msgr2=0x7ff594005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5aef33640 1 --2- 172.21.15.29:0/3455138062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff594005180 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7ff5a000c110 tx=0x7ff5940057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.041+0000 7ff5ad4d0640 1 -- 172.21.15.29:0/3455138062 reap_dead start 2023-10-14T16:53:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.042+0000 7ff5aef33640 1 -- 172.21.15.29:0/3455138062 shutdown_connections 2023-10-14T16:53:28.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.042+0000 7ff5aef33640 1 -- 172.21.15.29:0/3455138062 >> 172.21.15.29:0/3455138062 conn(0x7ff5a80ce9b0 msgr2=0x7ff5a80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.042+0000 7ff5aef33640 1 -- 172.21.15.29:0/3455138062 shutdown_connections 2023-10-14T16:53:28.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.042+0000 7ff5aef33640 1 -- 172.21.15.29:0/3455138062 wait complete. 2023-10-14T16:53:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.043+0000 7ff5aef33640 1 Processor -- start 2023-10-14T16:53:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.044+0000 7ff5aef33640 1 -- start start 2023-10-14T16:53:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5a80ad2a0 0x7ff5a80c6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a80ad720 0x7ff5a80c7520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a8159980 con 0x7ff594004d00 2023-10-14T16:53:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80c2a80 con 0x7ff5a80ad2a0 2023-10-14T16:53:28.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5aef33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a80ae790 con 0x7ff5a80ad720 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53606/0 (socket says 172.21.15.29:53606) 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 -- 172.21.15.29:0/497685482 learned_addr learned my addr 172.21.15.29:0/497685482 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5a80ad2a0 0x7ff5a80c6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 -- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a80ad720 msgr2=0x7ff5a80c7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a80ad720 0x7ff5a80c7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 -- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5a80ad2a0 msgr2=0x7ff5a80c6fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5a80ad2a0 0x7ff5a80c6fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.045+0000 7ff5acccf640 1 -- 172.21.15.29:0/497685482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a80adc10 con 0x7ff594004d00 2023-10-14T16:53:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5acccf640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff598007dc0 tx=0x7ff59800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598003990 con 0x7ff594004d00 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff598003b10 con 0x7ff594004d00 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598004030 con 0x7ff594004d00 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a80adef0 con 0x7ff594004d00 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.046+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a80ae4b0 con 0x7ff594004d00 2023-10-14T16:53:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.048+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a804e160 con 0x7ff594004d00 2023-10-14T16:53:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.049+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff598004190 con 0x7ff594004d00 2023-10-14T16:53:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.050+0000 7ff5a57fa640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 0x7ff58806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.050+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff598089ec0 con 0x7ff594004d00 2023-10-14T16:53:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.050+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 0x7ff58806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.051+0000 7ff5a7fff640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 0x7ff58806d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5a81f5840 tx=0x7ff590008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.054+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff598054b50 con 0x7ff594004d00 2023-10-14T16:53:28.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.277+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff5a8069da0 con 0x7ff58806b070 2023-10-14T16:53:28.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.489+0000 7ff5a57fa640 1 -- 172.21.15.29:0/497685482 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7ff59000dea0 con 0x7ff58806b070 2023-10-14T16:53:28.490 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/dir9 2023-10-14T16:53:28.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 msgr2=0x7ff58806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 0x7ff58806d530 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff5a81f5840 tx=0x7ff590008040 comp rx=0 tx=0).stop 2023-10-14T16:53:28.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 msgr2=0x7ff5a8159440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7ff598007dc0 tx=0x7ff59800e4b0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 shutdown_connections 2023-10-14T16:53:28.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5a80ad720 0x7ff5a80c7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff58806b070 0x7ff58806d530 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5a80ad2a0 0x7ff5a80c6fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 --2- 172.21.15.29:0/497685482 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff594004d00 0x7ff5a8159440 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.493+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 >> 172.21.15.29:0/497685482 conn(0x7ff5a80ce9b0 msgr2=0x7ff5a815d980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.494+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 shutdown_connections 2023-10-14T16:53:28.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.494+0000 7ff5aef33640 1 -- 172.21.15.29:0/497685482 wait complete. 2023-10-14T16:53:28.519 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/dir9 2023-10-14T16:53:28.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:28.520 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-14T16:53:28.589 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_42_00_UTC 2023-10-14T16:53:28.589 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:28.589 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:28.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:28.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:28.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:28.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:28.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:28.591 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:28.591 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:28.591 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:28.591 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:28.592 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_42_00_UTC', 'scheduled-2023-10-14-16_43_01_UTC', 'scheduled-2023-10-14-16_44_01_UTC', 'scheduled-2023-10-14-16_45_01_UTC', 'scheduled-2023-10-14-16_46_01_UTC', 'scheduled-2023-10-14-16_47_01_UTC', 'scheduled-2023-10-14-16_48_01_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_00_UTC', 'scheduled-2023-10-14-16_51_01_UTC', 'scheduled-2023-10-14-16_52_01_UTC', 'scheduled-2023-10-14-16_53_00_UTC'] 2023-10-14T16:53:28.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir0 2023-10-14T16:53:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.830+0000 7fe4c0814640 1 Processor -- start 2023-10-14T16:53:28.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.830+0000 7fe4c0814640 1 -- start start 2023-10-14T16:53:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.831+0000 7fe4c0814640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4bc0ad970 0x7fe4bc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.831+0000 7fe4c0814640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.831+0000 7fe4c0814640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4bc15b9d0 con 0x7fe4bc06c040 2023-10-14T16:53:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.831+0000 7fe4c0814640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4bc15bcb0 con 0x7fe4bc0ad970 2023-10-14T16:53:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.831+0000 7fe4c0814640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc15bf90 con 0x7fe4bc0ccfb0 2023-10-14T16:53:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.832+0000 7fe4ba59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4bc0ad970 0x7fe4bc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.832+0000 7fe4bb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.832+0000 7fe4ba59c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4bc0ad970 0x7fe4bc0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:55478/0 (socket says 172.21.15.29:55478) 2023-10-14T16:53:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.832+0000 7fe4ba59c640 1 -- 172.21.15.29:0/3180996316 learned_addr learned my addr 172.21.15.29:0/3180996316 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:28.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4b9d9b640 1 -- 172.21.15.29:0/3180996316 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215597248 0 0) 0x7fe4bc15b9d0 con 0x7fe4bc06c040 2023-10-14T16:53:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4b9d9b640 1 -- 172.21.15.29:0/3180996316 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4a4003520 con 0x7fe4bc06c040 2023-10-14T16:53:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4ba59c640 1 -- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 msgr2=0x7fe4bc0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4ba59c640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4ba59c640 1 -- 172.21.15.29:0/3180996316 >> v1:172.21.15.186:6789/0 conn(0x7fe4bc06c040 legacy=0x7fe4bc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4ba59c640 1 -- 172.21.15.29:0/3180996316 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc0cc270 con 0x7fe4bc0ad970 2023-10-14T16:53:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.833+0000 7fe4bb59e640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:53:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.834+0000 7fe4ba59c640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0cca70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009870 tx=0x7fe4b002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd4ca1a7aee3260d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.835+0000 7fe4b9d9b640 1 -- 172.21.15.29:0/3180996316 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0030840 con 0x7fe4bc0ad970 2023-10-14T16:53:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.835+0000 7fe4b9d9b640 1 -- 172.21.15.29:0/3180996316 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b00309a0 con 0x7fe4bc0ad970 2023-10-14T16:53:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.835+0000 7fe4b9d9b640 1 -- 172.21.15.29:0/3180996316 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0030d30 con 0x7fe4bc0ad970 2023-10-14T16:53:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 -- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ad970 msgr2=0x7fe4bc0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0cca70 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0009870 tx=0x7fe4b002b980 comp rx=0 tx=0).stop 2023-10-14T16:53:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 -- 172.21.15.29:0/3180996316 shutdown_connections 2023-10-14T16:53:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 --2- 172.21.15.29:0/3180996316 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0cca70 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 -- 172.21.15.29:0/3180996316 >> 172.21.15.29:0/3180996316 conn(0x7fe4bc0ce6a0 msgr2=0x7fe4bc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 -- 172.21.15.29:0/3180996316 shutdown_connections 2023-10-14T16:53:28.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.836+0000 7fe4c0814640 1 -- 172.21.15.29:0/3180996316 wait complete. 2023-10-14T16:53:28.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.838+0000 7fe4c0814640 1 Processor -- start 2023-10-14T16:53:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.838+0000 7fe4c0814640 1 -- start start 2023-10-14T16:53:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc06c040 0x7fe4bc1f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc1545e0 con 0x7fe4bc0ad970 2023-10-14T16:53:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc0c5770 con 0x7fe4bc0ccfb0 2023-10-14T16:53:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4c0814640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc0c5a50 con 0x7fe4bc06c040 2023-10-14T16:53:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4ba59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4ba59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53618/0 (socket says 172.21.15.29:53618) 2023-10-14T16:53:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4ba59c640 1 -- 172.21.15.29:0/2068362783 learned_addr learned my addr 172.21.15.29:0/2068362783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.839+0000 7fe4bad9d640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc06c040 0x7fe4bc1f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4bb59e640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 -- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc06c040 msgr2=0x7fe4bc1f32e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc06c040 0x7fe4bc1f32e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 -- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ccfb0 msgr2=0x7fe4bc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 -- 172.21.15.29:0/2068362783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b4009a70 con 0x7fe4bc0ad970 2023-10-14T16:53:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4ba59c640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00097c0 tx=0x7fe4b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0047070 con 0x7fe4bc0ad970 2023-10-14T16:53:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0039a90 con 0x7fe4bc0ad970 2023-10-14T16:53:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.840+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b00094d0 con 0x7fe4bc0ad970 2023-10-14T16:53:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.841+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0039e20 con 0x7fe4bc0ad970 2023-10-14T16:53:28.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.841+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc0c6010 con 0x7fe4bc0ad970 2023-10-14T16:53:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.842+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4bc06b330 con 0x7fe4bc0ad970 2023-10-14T16:53:28.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.843+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe4b004c410 con 0x7fe4bc0ad970 2023-10-14T16:53:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.844+0000 7fe4a37fe640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 0x7fe4a806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.844+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fe4b003e070 con 0x7fe4bc0ad970 2023-10-14T16:53:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.844+0000 7fe4bad9d640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 0x7fe4a806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:28.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.845+0000 7fe4bad9d640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 0x7fe4a806d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac001650 tx=0x7fe4ac009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:28.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:28.848+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4b00af030 con 0x7fe4bc0ad970 2023-10-14T16:53:29.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.068+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir0", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe4bc0c5bc0 con 0x7fe4a806b070 2023-10-14T16:53:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.083+0000 7fe4a37fe640 1 -- 172.21.15.29:0/2068362783 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fe4bc0c5bc0 con 0x7fe4a806b070 2023-10-14T16:53:29.085 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:29.085 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:41:59", "first": "2023-10-14T16:42:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-14T16:53:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 msgr2=0x7fe4a806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 0x7fe4a806d530 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac001650 tx=0x7fe4ac009040 comp rx=0 tx=0).stop 2023-10-14T16:53:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 msgr2=0x7fe4bc0c9700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fe4b00097c0 tx=0x7fe4b0004730 comp rx=0 tx=0).stop 2023-10-14T16:53:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 shutdown_connections 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe4a806b070 0x7fe4a806d530 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4bc0ccfb0 0x7fe4bc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4bc0ad970 0x7fe4bc0c9700 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 --2- 172.21.15.29:0/2068362783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4bc06c040 0x7fe4bc1f32e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 >> 172.21.15.29:0/2068362783 conn(0x7fe4bc0ce6a0 msgr2=0x7fe4bc06c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:29.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 shutdown_connections 2023-10-14T16:53:29.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.086+0000 7fe4c0814640 1 -- 172.21.15.29:0/2068362783 wait complete. 2023-10-14T16:53:29.107 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir0", "rel_path": "/snap_test_dir1/dir0", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:41:59", "first": "2023-10-14T16:42:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 12, "pruned_count": 0, "active": false}] 2023-10-14T16:53:29.107 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir0', 'rel_path': '/snap_test_dir1/dir0', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:41:59', 'first': '2023-10-14T16:42:00', 'last': '2023-10-14T16:53:00', 'last_pruned': None, 'created_count': 12, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:29.108 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 12 2023-10-14T16:53:29.108 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 12 2023-10-14T16:53:29.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:29.109 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-14T16:53:29.170 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:29.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:29.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:29.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:29.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:29.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:29.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:29.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:29.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:29.172 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:29.173 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:29.174 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_01_UTC', 'scheduled-2023-10-14-16_45_01_UTC', 'scheduled-2023-10-14-16_46_01_UTC', 'scheduled-2023-10-14-16_47_01_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_01_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_01_UTC', 'scheduled-2023-10-14-16_53_01_UTC'] 2023-10-14T16:53:29.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir1 2023-10-14T16:53:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.405+0000 7f7492c21640 1 Processor -- start 2023-10-14T16:53:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.406+0000 7f7492c21640 1 -- start start 2023-10-14T16:53:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.407+0000 7f7492c21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f748c15cd70 0x7f748c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.407+0000 7f7492c21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.407+0000 7f7492c21640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f748c15d150 con 0x7f748c0b8480 2023-10-14T16:53:29.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.407+0000 7f7492c21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f748c0d1180 con 0x7f748c15cd70 2023-10-14T16:53:29.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.407+0000 7f7492c21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0d1460 con 0x7f748c0cded0 2023-10-14T16:53:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.408+0000 7f7492420640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.408+0000 7f7491c1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f748c15cd70 0x7f748c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.408+0000 7f7492420640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:51338/0 (socket says 172.21.15.29:51338) 2023-10-14T16:53:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.408+0000 7f7492420640 1 -- 172.21.15.29:0/101043273 learned_addr learned my addr 172.21.15.29:0/101043273 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:29.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.409+0000 7f7492420640 1 -- 172.21.15.29:0/101043273 >> v2:172.21.15.186:3300/0 conn(0x7f748c15cd70 msgr2=0x7f748c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:29.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.409+0000 7f7492420640 1 --2- 172.21.15.29:0/101043273 >> v2:172.21.15.186:3300/0 conn(0x7f748c15cd70 0x7f748c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.409+0000 7f7492420640 1 -- 172.21.15.29:0/101043273 >> v1:172.21.15.186:6789/0 conn(0x7f748c0b8480 legacy=0x7f748c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.409+0000 7f7492420640 1 -- 172.21.15.29:0/101043273 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f748c0d1740 con 0x7f748c0cded0 2023-10-14T16:53:29.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.411+0000 7f7492420640 1 --2- 172.21.15.29:0/101043273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7484009870 tx=0x7f748402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7a01d97b36d4d598 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.411+0000 7f7490c1d640 1 -- 172.21.15.29:0/101043273 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484030840 con 0x7f748c0cded0 2023-10-14T16:53:29.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.411+0000 7f7490c1d640 1 -- 172.21.15.29:0/101043273 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74840309a0 con 0x7f748c0cded0 2023-10-14T16:53:29.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.411+0000 7f7490c1d640 1 -- 172.21.15.29:0/101043273 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7484030d30 con 0x7f748c0cded0 2023-10-14T16:53:29.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 -- 172.21.15.29:0/101043273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 msgr2=0x7f748c0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 --2- 172.21.15.29:0/101043273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7484009870 tx=0x7f748402b980 comp rx=0 tx=0).stop 2023-10-14T16:53:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 -- 172.21.15.29:0/101043273 shutdown_connections 2023-10-14T16:53:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 --2- 172.21.15.29:0/101043273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c0cded0 0x7f748c0b5c90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 --2- 172.21.15.29:0/101043273 >> v2:172.21.15.186:3300/0 conn(0x7f748c15cd70 0x7f748c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 -- 172.21.15.29:0/101043273 >> 172.21.15.29:0/101043273 conn(0x7f748c0cf570 msgr2=0x7f748c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:29.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 -- 172.21.15.29:0/101043273 shutdown_connections 2023-10-14T16:53:29.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.412+0000 7f7492c21640 1 -- 172.21.15.29:0/101043273 wait complete. 2023-10-14T16:53:29.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.414+0000 7f7492c21640 1 Processor -- start 2023-10-14T16:53:29.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.414+0000 7f7492c21640 1 -- start start 2023-10-14T16:53:29.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f748c0cded0 0x7f748c0b31d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c15cd70 0x7f748c0b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c0c8e20 con 0x7f748c0b8480 2023-10-14T16:53:29.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f748c1f2260 con 0x7f748c0cded0 2023-10-14T16:53:29.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7492c21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f748c1f6810 con 0x7f748c15cd70 2023-10-14T16:53:29.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7491c1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7491c1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53628/0 (socket says 172.21.15.29:53628) 2023-10-14T16:53:29.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.415+0000 7f7491c1f640 1 -- 172.21.15.29:0/632976135 learned_addr learned my addr 172.21.15.29:0/632976135 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:29.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7492420640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c15cd70 0x7f748c0b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f749141e640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f748c0cded0 0x7f748c0b31d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 -- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c15cd70 msgr2=0x7f748c0b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c15cd70 0x7f748c0b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 -- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f748c0cded0 msgr2=0x7f748c0b31d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f748c0cded0 0x7f748c0b31d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 -- 172.21.15.29:0/632976135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74840094d0 con 0x7f748c0b8480 2023-10-14T16:53:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f7491c1f640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f747c00efc0 tx=0x7f747c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.416+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c009b80 con 0x7f748c0b8480 2023-10-14T16:53:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.417+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f747c00f040 con 0x7f748c0b8480 2023-10-14T16:53:29.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.417+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f747c0146a0 con 0x7f748c0b8480 2023-10-14T16:53:29.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.417+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f748c1f3230 con 0x7f748c0b8480 2023-10-14T16:53:29.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.417+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f748c1f2750 con 0x7f748c0b8480 2023-10-14T16:53:29.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.418+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f748c04e9b0 con 0x7f748c0b8480 2023-10-14T16:53:29.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.419+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f747c014800 con 0x7f748c0b8480 2023-10-14T16:53:29.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.420+0000 7f748a7fc640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 0x7f747806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.420+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f747c08a710 con 0x7f748c0b8480 2023-10-14T16:53:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.420+0000 7f749141e640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 0x7f747806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.421+0000 7f749141e640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 0x7f747806d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f748c142e10 tx=0x7f7480009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.424+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f747c055320 con 0x7f748c0b8480 2023-10-14T16:53:29.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.643+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f748c0d1d00 con 0x7f747806b070 2023-10-14T16:53:29.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.661+0000 7f748a7fc640 1 -- 172.21.15.29:0/632976135 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f7480001340 con 0x7f747806b070 2023-10-14T16:53:29.661 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:29.662 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:41:59", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:29.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.663+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 msgr2=0x7f747806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.663+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 0x7f747806d530 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f748c142e10 tx=0x7f7480009040 comp rx=0 tx=0).stop 2023-10-14T16:53:29.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.663+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 msgr2=0x7f748c1f62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.663+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f747c00efc0 tx=0x7f747c00c490 comp rx=0 tx=0).stop 2023-10-14T16:53:29.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 shutdown_connections 2023-10-14T16:53:29.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f748c15cd70 0x7f748c0b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f747806b070 0x7f747806d530 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f748c0cded0 0x7f748c0b31d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 --2- 172.21.15.29:0/632976135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f748c0b8480 0x7f748c1f62d0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 >> 172.21.15.29:0/632976135 conn(0x7f748c0cf570 msgr2=0x7f748c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:29.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 shutdown_connections 2023-10-14T16:53:29.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.664+0000 7f7492c21640 1 -- 172.21.15.29:0/632976135 wait complete. 2023-10-14T16:53:29.688 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir1", "rel_path": "/snap_test_dir1/dir1", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:41:59", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:29.688 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir1', 'rel_path': '/snap_test_dir1/dir1', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:41:59', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:29.688 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:29.689 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:29.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:29.690 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-14T16:53:29.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:29.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:29.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:29.744 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:29.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:29.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:29.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:29.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:29.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:29.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:29.747 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:29.748 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_01_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_00_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_01_UTC', 'scheduled-2023-10-14-16_51_01_UTC', 'scheduled-2023-10-14-16_52_00_UTC', 'scheduled-2023-10-14-16_53_00_UTC'] 2023-10-14T16:53:29.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir2 2023-10-14T16:53:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.971+0000 7f6bab59e640 1 Processor -- start 2023-10-14T16:53:29.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.971+0000 7f6bab59e640 1 -- start start 2023-10-14T16:53:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6bab59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6bab59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6bac0ad970 0x7f6bac15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6bab59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bac06c960 con 0x7f6bac15ebc0 2023-10-14T16:53:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6bab59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bac15b9d0 con 0x7f6bac0ad970 2023-10-14T16:53:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6bab59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac15bcb0 con 0x7f6bac06c040 2023-10-14T16:53:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.973+0000 7f6ba9d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6bac0ad970 0x7f6bac15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6baad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6bac15ebc0 0x7f6bac0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39348/0 (socket says 172.21.15.29:39348) 2023-10-14T16:53:29.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6baad9d640 1 -- 172.21.15.29:0/3633702138 learned_addr learned my addr 172.21.15.29:0/3633702138 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6baa59c640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/3633702138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 msgr2=0x7f6bac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6ba9d9b640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/3633702138 >> v1:172.21.15.29:6789/0 conn(0x7f6bac15ebc0 legacy=0x7f6bac0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/3633702138 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac15bf90 con 0x7f6bac0ad970 2023-10-14T16:53:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.974+0000 7f6baa59c640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.975+0000 7f6ba9d9b640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac15e680 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00097d0 tx=0x7f6ba002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df43c51e37790bd2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6ba959a640 1 -- 172.21.15.29:0/3633702138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba002d040 con 0x7f6bac0ad970 2023-10-14T16:53:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6ba959a640 1 -- 172.21.15.29:0/3633702138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0031810 con 0x7f6bac0ad970 2023-10-14T16:53:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6ba959a640 1 -- 172.21.15.29:0/3633702138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba0031ba0 con 0x7f6bac0ad970 2023-10-14T16:53:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 -- 172.21.15.29:0/3633702138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 msgr2=0x7f6bac15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac15e680 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00097d0 tx=0x7f6ba002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 -- 172.21.15.29:0/3633702138 shutdown_connections 2023-10-14T16:53:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac15e680 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 --2- 172.21.15.29:0/3633702138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 -- 172.21.15.29:0/3633702138 >> 172.21.15.29:0/3633702138 conn(0x7f6bac0ce6a0 msgr2=0x7f6bac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 -- 172.21.15.29:0/3633702138 shutdown_connections 2023-10-14T16:53:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.976+0000 7f6bab59e640 1 -- 172.21.15.29:0/3633702138 wait complete. 2023-10-14T16:53:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.978+0000 7f6bab59e640 1 Processor -- start 2023-10-14T16:53:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.978+0000 7f6bab59e640 1 -- start start 2023-10-14T16:53:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac157910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac15ebc0 0x7f6bac0c9bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0c6b70 con 0x7f6bac0ad970 2023-10-14T16:53:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac1f25e0 con 0x7f6bac15ebc0 2023-10-14T16:53:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.979+0000 7f6bab59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bac0aed10 con 0x7f6bac06c040 2023-10-14T16:53:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53648/0 (socket says 172.21.15.29:53648) 2023-10-14T16:53:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/880879032 learned_addr learned my addr 172.21.15.29:0/880879032 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6baa59c640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac157910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6baad9d640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac15ebc0 0x7f6bac0c9bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 msgr2=0x7f6bac157910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac157910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac15ebc0 msgr2=0x7f6bac0c9bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac15ebc0 0x7f6bac0c9bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 -- 172.21.15.29:0/880879032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ba0009400 con 0x7f6bac0ad970 2023-10-14T16:53:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.980+0000 7f6ba9d9b640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098f0 tx=0x7f6ba00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.981+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba003ad60 con 0x7f6bac0ad970 2023-10-14T16:53:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.981+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba00423f0 con 0x7f6bac0ad970 2023-10-14T16:53:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.981+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba00427a0 con 0x7f6bac0ad970 2023-10-14T16:53:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.981+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bac0aeff0 con 0x7f6bac0ad970 2023-10-14T16:53:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.981+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bac0af5b0 con 0x7f6bac0ad970 2023-10-14T16:53:29.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.982+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bac06b330 con 0x7f6bac0ad970 2023-10-14T16:53:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.983+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f6ba00393b0 con 0x7f6bac0ad970 2023-10-14T16:53:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.984+0000 7f6b9affd640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 0x7f6b8c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.984+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f6ba00ab8a0 con 0x7f6bac0ad970 2023-10-14T16:53:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.984+0000 7f6baa59c640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 0x7f6b8c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.985+0000 7f6baa59c640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 0x7f6b8c06d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c006fd0 tx=0x7f6b9c008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:29.988+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ba00764b0 con 0x7f6bac0ad970 2023-10-14T16:53:30.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.207+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir2", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f6bac1f2470 con 0x7f6b8c06b070 2023-10-14T16:53:30.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.224+0000 7f6b9affd640 1 -- 172.21.15.29:0/880879032 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f6bac1f2470 con 0x7f6b8c06b070 2023-10-14T16:53:30.226 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:30.226 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:00", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:30.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 msgr2=0x7f6b8c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 0x7f6b8c06d530 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6b9c006fd0 tx=0x7f6b9c008040 comp rx=0 tx=0).stop 2023-10-14T16:53:30.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 msgr2=0x7f6bac0c9690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f6ba00098f0 tx=0x7f6ba00046c0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 shutdown_connections 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f6b8c06b070 0x7f6b8c06d530 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6bac15ebc0 0x7f6bac0c9bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6bac0ad970 0x7f6bac0c9690 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 --2- 172.21.15.29:0/880879032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6bac06c040 0x7f6bac157910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 >> 172.21.15.29:0/880879032 conn(0x7f6bac0ce6a0 msgr2=0x7f6bac0ca7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:30.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.227+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 shutdown_connections 2023-10-14T16:53:30.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.228+0000 7f6bab59e640 1 -- 172.21.15.29:0/880879032 wait complete. 2023-10-14T16:53:30.247 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir2", "rel_path": "/snap_test_dir1/dir2", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:00", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:30.248 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir2', 'rel_path': '/snap_test_dir1/dir2', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:00', 'first': '2023-10-14T16:43:01', 'last': '2023-10-14T16:53:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:30.248 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:30.248 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:30.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:30.249 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-14T16:53:30.308 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:30.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:30.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:30.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:30.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:30.309 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:30.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:30.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:30.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:30.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:30.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:30.311 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_01_UTC', 'scheduled-2023-10-14-16_45_00_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_01_UTC', 'scheduled-2023-10-14-16_50_01_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_00_UTC', 'scheduled-2023-10-14-16_53_00_UTC'] 2023-10-14T16:53:30.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir3 2023-10-14T16:53:30.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.535+0000 7fa17359e640 1 Processor -- start 2023-10-14T16:53:30.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.535+0000 7fa17359e640 1 -- start start 2023-10-14T16:53:30.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.536+0000 7fa17359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa17406c040 0x7fa17406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.536+0000 7fa17359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 0x7fa17415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa17406c960 con 0x7fa1740ad970 2023-10-14T16:53:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa17415b9d0 con 0x7fa17406c040 2023-10-14T16:53:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17415bcb0 con 0x7fa1740cda80 2023-10-14T16:53:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa171d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa1740ad970 0x7fa17415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39360/0 (socket says 172.21.15.29:39360) 2023-10-14T16:53:30.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa171d9b640 1 -- 172.21.15.29:0/2930738841 learned_addr learned my addr 172.21.15.29:0/2930738841 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:30.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398162503 0 0) 0x7fa17406c960 con 0x7fa1740ad970 2023-10-14T16:53:30.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17259c640 1 --2- 172.21.15.29:0/2930738841 >> v2:172.21.15.186:3300/0 conn(0x7fa17406c040 0x7fa17406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.537+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa160003520 con 0x7fa1740ad970 2023-10-14T16:53:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607749113 0 0) 0x7fa160003520 con 0x7fa1740ad970 2023-10-14T16:53:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 msgr2=0x7fa17415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 0x7fa17415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 >> v2:172.21.15.186:3300/0 conn(0x7fa17406c040 msgr2=0x7fa17406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:30.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 --2- 172.21.15.29:0/2930738841 >> v2:172.21.15.186:3300/0 conn(0x7fa17406c040 0x7fa17406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17415bf90 con 0x7fa1740ad970 2023-10-14T16:53:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa15c001670 con 0x7fa1740ad970 2023-10-14T16:53:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 >> v1:172.21.15.29:6789/0 conn(0x7fa1740ad970 legacy=0x7fa17415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa160003970 0x7fa160003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 0x7fa1600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160004d00 0x7fa160005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17415b9d0 con 0x7fa160003970 2023-10-14T16:53:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17415bcb0 con 0x7fa160004d00 2023-10-14T16:53:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa16000bb10 con 0x7fa1740cda80 2023-10-14T16:53:30.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.538+0000 7fa17259c640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa160003970 0x7fa160003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa172d9d640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160004d00 0x7fa160005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa171d9b640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 0x7fa1600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa17259c640 1 -- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 msgr2=0x7fa1600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa17259c640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cda80 0x7fa1600047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa17259c640 1 -- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160004d00 msgr2=0x7fa160005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:30.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa17259c640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160004d00 0x7fa160005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.539+0000 7fa17259c640 1 -- 172.21.15.29:0/2930738841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa160003520 con 0x7fa160003970 2023-10-14T16:53:30.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.540+0000 7fa17259c640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa160003970 0x7fa160003d90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa164009f90 tx=0x7fa1600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10ae2519ef8d908 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.540+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16402cc20 con 0x7fa160003970 2023-10-14T16:53:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.540+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16400f040 con 0x7fa160003970 2023-10-14T16:53:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.540+0000 7fa17159a640 1 -- 172.21.15.29:0/2930738841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164034440 con 0x7fa160003970 2023-10-14T16:53:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 -- 172.21.15.29:0/2930738841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa160003970 msgr2=0x7fa160003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 --2- 172.21.15.29:0/2930738841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa160003970 0x7fa160003d90 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7fa164009f90 tx=0x7fa1600057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa172d9d640 1 -- 172.21.15.29:0/2930738841 reap_dead start 2023-10-14T16:53:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 -- 172.21.15.29:0/2930738841 shutdown_connections 2023-10-14T16:53:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 -- 172.21.15.29:0/2930738841 >> 172.21.15.29:0/2930738841 conn(0x7fa1740ce6a0 msgr2=0x7fa1740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 -- 172.21.15.29:0/2930738841 shutdown_connections 2023-10-14T16:53:30.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.541+0000 7fa17359e640 1 -- 172.21.15.29:0/2930738841 wait complete. 2023-10-14T16:53:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.543+0000 7fa17359e640 1 Processor -- start 2023-10-14T16:53:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.543+0000 7fa17359e640 1 -- start start 2023-10-14T16:53:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 0x7fa17414ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17414d9d0 0x7fa1740c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b21f0 con 0x7fa17414d4d0 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b2460 con 0x7fa160003970 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b2740 con 0x7fa17414d9d0 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa171d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa171d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53654/0 (socket says 172.21.15.29:53654) 2023-10-14T16:53:30.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa171d9b640 1 -- 172.21.15.29:0/3580392188 learned_addr learned my addr 172.21.15.29:0/3580392188 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.544+0000 7fa17259c640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 0x7fa17414ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 -- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17414d9d0 msgr2=0x7fa1740c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17414d9d0 0x7fa1740c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 -- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 msgr2=0x7fa17414ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 0x7fa17414ceb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 -- 172.21.15.29:0/3580392188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740b2a20 con 0x7fa17414d4d0 2023-10-14T16:53:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa17259c640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 0x7fa17414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa171d9b640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00d810 tx=0x7fa15c00dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:30.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c0074e0 con 0x7fa17414d4d0 2023-10-14T16:53:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa15c007640 con 0x7fa17414d4d0 2023-10-14T16:53:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa15c004a20 con 0x7fa17414d4d0 2023-10-14T16:53:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.545+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1740c84c0 con 0x7fa17414d4d0 2023-10-14T16:53:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.546+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1740c8230 con 0x7fa17414d4d0 2023-10-14T16:53:30.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.547+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1740b2c60 con 0x7fa17414d4d0 2023-10-14T16:53:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.547+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa15c004b80 con 0x7fa17414d4d0 2023-10-14T16:53:30.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.548+0000 7fa16affd640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 0x7fa15406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:30.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.548+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa15c0896a0 con 0x7fa17414d4d0 2023-10-14T16:53:30.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.549+0000 7fa17259c640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 0x7fa15406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:30.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.549+0000 7fa17259c640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 0x7fa15406d530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa164036000 tx=0x7fa16402e000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:30.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.553+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa1740b2c60 con 0x7fa17414d4d0 2023-10-14T16:53:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.770+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir3", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa1740b25d0 con 0x7fa15406b070 2023-10-14T16:53:30.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.784+0000 7fa16affd640 1 -- 172.21.15.29:0/3580392188 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa1740b25d0 con 0x7fa15406b070 2023-10-14T16:53:30.785 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:30.785 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:01", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:30.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.786+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 msgr2=0x7fa15406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.786+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 0x7fa15406d530 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa164036000 tx=0x7fa16402e000 comp rx=0 tx=0).stop 2023-10-14T16:53:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.786+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 msgr2=0x7fa1741e4460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.786+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7fa15c00d810 tx=0x7fa15c00dcd0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 shutdown_connections 2023-10-14T16:53:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17414d9d0 0x7fa1740c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17414d4d0 0x7fa1741e4460 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa15406b070 0x7fa15406d530 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 --2- 172.21.15.29:0/3580392188 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa160003970 0x7fa17414ceb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 >> 172.21.15.29:0/3580392188 conn(0x7fa1740ce6a0 msgr2=0x7fa17415dba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 shutdown_connections 2023-10-14T16:53:30.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:30.787+0000 7fa17359e640 1 -- 172.21.15.29:0/3580392188 wait complete. 2023-10-14T16:53:30.805 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir3", "rel_path": "/snap_test_dir1/dir3", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:01", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:30.806 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir3', 'rel_path': '/snap_test_dir1/dir3', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:01', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:30.806 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:30.806 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:30.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:30.807 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-14T16:53:30.866 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:30.866 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:30.866 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:30.867 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:30.868 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:30.868 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:30.869 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_00_UTC', 'scheduled-2023-10-14-16_46_01_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_01_UTC', 'scheduled-2023-10-14-16_50_00_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_01_UTC', 'scheduled-2023-10-14-16_53_01_UTC'] 2023-10-14T16:53:30.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir4 2023-10-14T16:53:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.101+0000 7f2e8d165640 1 Processor -- start 2023-10-14T16:53:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.101+0000 7f2e8d165640 1 -- start start 2023-10-14T16:53:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8d165640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2e8815d150 0x7f2e880ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8d165640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e8806a4b0 0x7f2e88160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8d165640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e880ac890 con 0x7f2e88160b90 2023-10-14T16:53:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8d165640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e8815d530 con 0x7f2e8806a4b0 2023-10-14T16:53:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8d165640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e88161650 con 0x7f2e8815d150 2023-10-14T16:53:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e8806a4b0 0x7f2e88160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e8806a4b0 0x7f2e88160fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41876/0 (socket says 172.21.15.29:41876) 2023-10-14T16:53:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e8759e640 1 -- 172.21.15.29:0/1420836930 learned_addr learned my addr 172.21.15.29:0/1420836930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.103+0000 7f2e86d9d640 1 --2- 172.21.15.29:0/1420836930 >> v2:172.21.15.186:3300/0 conn(0x7f2e8815d150 0x7f2e880ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.104+0000 7f2e8759e640 1 -- 172.21.15.29:0/1420836930 >> v2:172.21.15.186:3300/0 conn(0x7f2e8815d150 msgr2=0x7f2e880ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.104+0000 7f2e8759e640 1 --2- 172.21.15.29:0/1420836930 >> v2:172.21.15.186:3300/0 conn(0x7f2e8815d150 0x7f2e880ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.104+0000 7f2e8759e640 1 -- 172.21.15.29:0/1420836930 >> v1:172.21.15.186:6789/0 conn(0x7f2e88160b90 legacy=0x7f2e88069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.104+0000 7f2e8759e640 1 -- 172.21.15.29:0/1420836930 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e88161930 con 0x7f2e8806a4b0 2023-10-14T16:53:31.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.105+0000 7f2e8759e640 1 --2- 172.21.15.29:0/1420836930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e8806a4b0 0x7f2e88160fe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2e880b15a0 tx=0x7f2e8002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=149ae190b632bdb4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.105+0000 7f2e85d9b640 1 -- 172.21.15.29:0/1420836930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8002c9f0 con 0x7f2e8806a4b0 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.105+0000 7f2e85d9b640 1 -- 172.21.15.29:0/1420836930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8002cb50 con 0x7f2e8806a4b0 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.105+0000 7f2e85d9b640 1 -- 172.21.15.29:0/1420836930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e800343f0 con 0x7f2e8806a4b0 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 -- 172.21.15.29:0/1420836930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e8806a4b0 msgr2=0x7f2e88160fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 --2- 172.21.15.29:0/1420836930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e8806a4b0 0x7f2e88160fe0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f2e880b15a0 tx=0x7f2e8002be50 comp rx=0 tx=0).stop 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 -- 172.21.15.29:0/1420836930 shutdown_connections 2023-10-14T16:53:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 --2- 172.21.15.29:0/1420836930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e8806a4b0 0x7f2e88160fe0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 --2- 172.21.15.29:0/1420836930 >> v2:172.21.15.186:3300/0 conn(0x7f2e8815d150 0x7f2e880ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 -- 172.21.15.29:0/1420836930 >> 172.21.15.29:0/1420836930 conn(0x7f2e880ce9b0 msgr2=0x7f2e880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 -- 172.21.15.29:0/1420836930 shutdown_connections 2023-10-14T16:53:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.106+0000 7f2e8d165640 1 -- 172.21.15.29:0/1420836930 wait complete. 2023-10-14T16:53:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.109+0000 7f2e8d165640 1 Processor -- start 2023-10-14T16:53:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.109+0000 7f2e8d165640 1 -- start start 2023-10-14T16:53:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 0x7f2e8815d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e8815d150 0x7f2e880b1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e8815a090 con 0x7f2e88160b90 2023-10-14T16:53:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880b1a40 con 0x7f2e8806a4b0 2023-10-14T16:53:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.110+0000 7f2e8d165640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e880b04d0 con 0x7f2e8815d150 2023-10-14T16:53:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41884/0 (socket says 172.21.15.29:41884) 2023-10-14T16:53:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 -- 172.21.15.29:0/2126040206 learned_addr learned my addr 172.21.15.29:0/2126040206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e86d9d640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 0x7f2e8815d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 -- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e8815d150 msgr2=0x7f2e880b1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e8815d150 0x7f2e880b1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 -- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 msgr2=0x7f2e8815d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 0x7f2e8815d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e8759e640 1 -- 172.21.15.29:0/2126040206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e80009b30 con 0x7f2e88160b90 2023-10-14T16:53:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.111+0000 7f2e86d9d640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 0x7f2e8815d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.112+0000 7f2e8759e640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009f40 tx=0x7f2e80004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.112+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e80009510 con 0x7f2e88160b90 2023-10-14T16:53:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.112+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e8003e450 con 0x7f2e88160b90 2023-10-14T16:53:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.112+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e8003e7e0 con 0x7f2e88160b90 2023-10-14T16:53:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.112+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e880aff10 con 0x7f2e88160b90 2023-10-14T16:53:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.113+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e880b16d0 con 0x7f2e88160b90 2023-10-14T16:53:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.114+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e880b1bf0 con 0x7f2e88160b90 2023-10-14T16:53:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.115+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2e8003e940 con 0x7f2e88160b90 2023-10-14T16:53:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.115+0000 7f2e6f7fe640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 0x7f2e7406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.116+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f2e800ac880 con 0x7f2e88160b90 2023-10-14T16:53:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.116+0000 7f2e86d9d640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 0x7f2e7406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.116+0000 7f2e86d9d640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 0x7f2e7406d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.120+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e880b1bf0 con 0x7f2e88160b90 2023-10-14T16:53:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.346+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir4", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f2e880ace30 con 0x7f2e7406b070 2023-10-14T16:53:31.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.364+0000 7f2e6f7fe640 1 -- 172.21.15.29:0/2126040206 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f2e78009530 con 0x7f2e7406b070 2023-10-14T16:53:31.365 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:31.365 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:02", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:31.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 msgr2=0x7f2e7406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 0x7f2e7406d530 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e78006fd0 tx=0x7f2e78008040 comp rx=0 tx=0).stop 2023-10-14T16:53:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 msgr2=0x7f2e880b2410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f2e80009f40 tx=0x7f2e80004a40 comp rx=0 tx=0).stop 2023-10-14T16:53:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 shutdown_connections 2023-10-14T16:53:31.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e88160b90 0x7f2e880b2410 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e8815d150 0x7f2e880b1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2e7406b070 0x7f2e7406d530 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 --2- 172.21.15.29:0/2126040206 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e8806a4b0 0x7f2e8815d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 >> 172.21.15.29:0/2126040206 conn(0x7f2e880ce9b0 msgr2=0x7f2e880c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:31.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 shutdown_connections 2023-10-14T16:53:31.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.367+0000 7f2e8d165640 1 -- 172.21.15.29:0/2126040206 wait complete. 2023-10-14T16:53:31.388 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir4", "rel_path": "/snap_test_dir1/dir4", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:02", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:31.388 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir4', 'rel_path': '/snap_test_dir1/dir4', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:02', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:31.389 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:31.389 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:31.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:31.390 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-14T16:53:31.445 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:31.445 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:31.445 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:31.445 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:31.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:31.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:31.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:31.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:31.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:31.447 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:31.447 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:31.448 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_01_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_00_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_00_UTC', 'scheduled-2023-10-14-16_51_01_UTC', 'scheduled-2023-10-14-16_52_00_UTC', 'scheduled-2023-10-14-16_53_01_UTC'] 2023-10-14T16:53:31.448 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir5 2023-10-14T16:53:31.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.667+0000 7fa494501640 1 Processor -- start 2023-10-14T16:53:31.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.667+0000 7fa494501640 1 -- start start 2023-10-14T16:53:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa494501640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa48c15ad70 0x7fa48c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa494501640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 0x7fa48c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa494501640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa48c0cb030 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa494501640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c0c91c0 con 0x7fa48c15ad70 2023-10-14T16:53:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa494501640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa48c0c94a0 con 0x7fa48c0c9ca0 2023-10-14T16:53:31.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa492a9e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa48c0ca3b0 0x7fa48c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51634/0 (socket says 172.21.15.29:51634) 2023-10-14T16:53:31.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa492a9e640 1 -- 172.21.15.29:0/473847286 learned_addr learned my addr 172.21.15.29:0/473847286 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.669+0000 7fa49229d640 1 --2- 172.21.15.29:0/473847286 >> v2:172.21.15.29:3300/0 conn(0x7fa48c15ad70 0x7fa48c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa491a9c640 1 --2- 172.21.15.29:0/473847286 >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 0x7fa48c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49129b640 1 -- 172.21.15.29:0/473847286 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4030281167 0 0) 0x7fa48c0cb030 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49129b640 1 -- 172.21.15.29:0/473847286 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa478003520 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49229d640 1 -- 172.21.15.29:0/473847286 >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 msgr2=0x7fa48c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:31.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49229d640 1 --2- 172.21.15.29:0/473847286 >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 0x7fa48c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49229d640 1 -- 172.21.15.29:0/473847286 >> v1:172.21.15.29:6789/0 conn(0x7fa48c0ca3b0 legacy=0x7fa48c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa49229d640 1 -- 172.21.15.29:0/473847286 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa48c0c9780 con 0x7fa48c15ad70 2023-10-14T16:53:31.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.670+0000 7fa491a9c640 1 --2- 172.21.15.29:0/473847286 >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 0x7fa48c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:31.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.672+0000 7fa49229d640 1 --2- 172.21.15.29:0/473847286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c15ad70 0x7fa48c069da0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0ce870 tx=0x7fa48802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36afa0f9ae602cd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.672+0000 7fa49129b640 1 -- 172.21.15.29:0/473847286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488031830 con 0x7fa48c15ad70 2023-10-14T16:53:31.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.672+0000 7fa49129b640 1 -- 172.21.15.29:0/473847286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa488031990 con 0x7fa48c15ad70 2023-10-14T16:53:31.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.672+0000 7fa49129b640 1 -- 172.21.15.29:0/473847286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa488031d00 con 0x7fa48c15ad70 2023-10-14T16:53:31.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 -- 172.21.15.29:0/473847286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c15ad70 msgr2=0x7fa48c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 --2- 172.21.15.29:0/473847286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c15ad70 0x7fa48c069da0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0ce870 tx=0x7fa48802b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 -- 172.21.15.29:0/473847286 shutdown_connections 2023-10-14T16:53:31.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 --2- 172.21.15.29:0/473847286 >> v2:172.21.15.186:3300/0 conn(0x7fa48c0c9ca0 0x7fa48c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 --2- 172.21.15.29:0/473847286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c15ad70 0x7fa48c069da0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 -- 172.21.15.29:0/473847286 >> 172.21.15.29:0/473847286 conn(0x7fa48c0ce9b0 msgr2=0x7fa48c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:31.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 -- 172.21.15.29:0/473847286 shutdown_connections 2023-10-14T16:53:31.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.673+0000 7fa494501640 1 -- 172.21.15.29:0/473847286 wait complete. 2023-10-14T16:53:31.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.675+0000 7fa494501640 1 Processor -- start 2023-10-14T16:53:31.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.675+0000 7fa494501640 1 -- start start 2023-10-14T16:53:31.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48c0c9ca0 0x7fa48c15b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 0x7fa48c15cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c15bca0 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c15bf80 con 0x7fa48c15ad70 2023-10-14T16:53:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.676+0000 7fa494501640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa48c15c260 con 0x7fa48c0c9ca0 2023-10-14T16:53:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41904/0 (socket says 172.21.15.29:41904) 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 -- 172.21.15.29:0/3025426846 learned_addr learned my addr 172.21.15.29:0/3025426846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa492a9e640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 0x7fa48c15cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa49229d640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48c0c9ca0 0x7fa48c15b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 -- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48c0c9ca0 msgr2=0x7fa48c15b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48c0c9ca0 0x7fa48c15b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 -- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 msgr2=0x7fa48c15cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 0x7fa48c15cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa491a9c640 1 -- 172.21.15.29:0/3025426846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa488009400 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.677+0000 7fa492a9e640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 0x7fa48c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa491a9c640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00e7e0 tx=0x7fa47c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c004030 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa47c004190 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa47c004520 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa48c15c540 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.678+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa48c15b990 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.679+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa48c04e160 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.680+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa47c00c6d0 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.682+0000 7fa482ffd640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 0x7fa47006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.682+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa47c0898b0 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.682+0000 7fa49229d640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 0x7fa47006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.683+0000 7fa49229d640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 0x7fa47006d530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.686+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa47c0544c0 con 0x7fa48c0ca3b0 2023-10-14T16:53:31.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.905+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir5", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa48c0ad4d0 con 0x7fa47006b070 2023-10-14T16:53:31.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.921+0000 7fa482ffd640 1 -- 172.21.15.29:0/3025426846 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa488031b00 con 0x7fa47006b070 2023-10-14T16:53:31.922 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:31.922 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:02", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 msgr2=0x7fa47006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 0x7fa47006d530 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa488036000 tx=0x7fa48803c040 comp rx=0 tx=0).stop 2023-10-14T16:53:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 msgr2=0x7fa48c153570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7fa47c00e7e0 tx=0x7fa47c00eca0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 shutdown_connections 2023-10-14T16:53:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa47006b070 0x7fa47006d530 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa48c15ad70 0x7fa48c15cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa48c0ca3b0 0x7fa48c153570 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 --2- 172.21.15.29:0/3025426846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa48c0c9ca0 0x7fa48c15b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.924+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 >> 172.21.15.29:0/3025426846 conn(0x7fa48c0ce9b0 msgr2=0x7fa48c0acc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.925+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 shutdown_connections 2023-10-14T16:53:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:31.925+0000 7fa494501640 1 -- 172.21.15.29:0/3025426846 wait complete. 2023-10-14T16:53:31.945 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir5", "rel_path": "/snap_test_dir1/dir5", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:02", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:31.945 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir5', 'rel_path': '/snap_test_dir1/dir5', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:02', 'first': '2023-10-14T16:43:01', 'last': '2023-10-14T16:53:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:31.946 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:31.946 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:31.946 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:31.947 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-14T16:53:32.002 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:32.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:32.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:32.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:32.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:32.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:32.004 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:32.004 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:32.004 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:32.004 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:32.005 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:32.006 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_00_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_01_UTC', 'scheduled-2023-10-14-16_49_01_UTC', 'scheduled-2023-10-14-16_50_01_UTC', 'scheduled-2023-10-14-16_51_01_UTC', 'scheduled-2023-10-14-16_52_00_UTC', 'scheduled-2023-10-14-16_53_01_UTC'] 2023-10-14T16:53:32.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir6 2023-10-14T16:53:32.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.241+0000 7fa5ab40d640 1 Processor -- start 2023-10-14T16:53:32.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.241+0000 7fa5ab40d640 1 -- start start 2023-10-14T16:53:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.242+0000 7fa5ab40d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 0x7fa5a40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.242+0000 7fa5ab40d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 0x7fa5a40cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.242+0000 7fa5ab40d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5a415b9d0 con 0x7fa5a406c040 2023-10-14T16:53:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5ab40d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5a415bcb0 con 0x7fa5a40ccfb0 2023-10-14T16:53:32.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5ab40d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a415bf90 con 0x7fa5a40ad970 2023-10-14T16:53:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5aac0c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 0x7fa5a40cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5a9c0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 0x7fa5a40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5aac0c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 0x7fa5a40cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:49022/0 (socket says 172.21.15.29:49022) 2023-10-14T16:53:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.243+0000 7fa5aac0c640 1 -- 172.21.15.29:0/2253573107 learned_addr learned my addr 172.21.15.29:0/2253573107 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.244+0000 7fa5a9c0a640 1 -- 172.21.15.29:0/2253573107 >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 msgr2=0x7fa5a40cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.244+0000 7fa5a9c0a640 1 --2- 172.21.15.29:0/2253573107 >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 0x7fa5a40cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.244+0000 7fa5a9c0a640 1 -- 172.21.15.29:0/2253573107 >> v1:172.21.15.186:6789/0 conn(0x7fa5a406c040 legacy=0x7fa5a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.244+0000 7fa5a9c0a640 1 -- 172.21.15.29:0/2253573107 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a40cc270 con 0x7fa5a40ad970 2023-10-14T16:53:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.245+0000 7fa5a9c0a640 1 --2- 172.21.15.29:0/2253573107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 0x7fa5a40cca70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5940097d0 tx=0x7fa59402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=270053a4f0e64800 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.246+0000 7fa5a9409640 1 -- 172.21.15.29:0/2253573107 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa59402d040 con 0x7fa5a40ad970 2023-10-14T16:53:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.246+0000 7fa5a9409640 1 -- 172.21.15.29:0/2253573107 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa594031810 con 0x7fa5a40ad970 2023-10-14T16:53:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.246+0000 7fa5a9409640 1 -- 172.21.15.29:0/2253573107 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa594031ba0 con 0x7fa5a40ad970 2023-10-14T16:53:32.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2253573107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 msgr2=0x7fa5a40cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2253573107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 0x7fa5a40cca70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fa5940097d0 tx=0x7fa59402b8b0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2253573107 shutdown_connections 2023-10-14T16:53:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2253573107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ad970 0x7fa5a40cca70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2253573107 >> v2:172.21.15.186:3300/0 conn(0x7fa5a40ccfb0 0x7fa5a40cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2253573107 >> 172.21.15.29:0/2253573107 conn(0x7fa5a40ce6a0 msgr2=0x7fa5a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2253573107 shutdown_connections 2023-10-14T16:53:32.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.247+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2253573107 wait complete. 2023-10-14T16:53:32.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.249+0000 7fa5ab40d640 1 Processor -- start 2023-10-14T16:53:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.249+0000 7fa5ab40d640 1 -- start start 2023-10-14T16:53:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5a40ad970 0x7fa5a4159e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ccfb0 0x7fa5a40b00e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a415af20 con 0x7fa5a406c040 2023-10-14T16:53:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a40c8330 con 0x7fa5a40ad970 2023-10-14T16:53:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5ab40d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a40c6b50 con 0x7fa5a40ccfb0 2023-10-14T16:53:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5aa40b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5aa40b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41920/0 (socket says 172.21.15.29:41920) 2023-10-14T16:53:32.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.250+0000 7fa5aa40b640 1 -- 172.21.15.29:0/2300309672 learned_addr learned my addr 172.21.15.29:0/2300309672 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5a9c0a640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5a40ad970 0x7fa5a4159e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aac0c640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ccfb0 0x7fa5a40b00e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 -- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ccfb0 msgr2=0x7fa5a40b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ccfb0 0x7fa5a40b00e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 -- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5a40ad970 msgr2=0x7fa5a4159e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5a40ad970 0x7fa5a4159e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 -- 172.21.15.29:0/2300309672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa594009400 con 0x7fa5a406c040 2023-10-14T16:53:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.251+0000 7fa5aa40b640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c400 tx=0x7fa5a000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.252+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a00052d0 con 0x7fa5a406c040 2023-10-14T16:53:32.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.252+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0005450 con 0x7fa5a406c040 2023-10-14T16:53:32.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.252+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a40c8560 con 0x7fa5a406c040 2023-10-14T16:53:32.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.252+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0007260 con 0x7fa5a406c040 2023-10-14T16:53:32.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.252+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a40c6500 con 0x7fa5a406c040 2023-10-14T16:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.253+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a406b330 con 0x7fa5a406c040 2023-10-14T16:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.254+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fa5a00073c0 con 0x7fa5a406c040 2023-10-14T16:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.255+0000 7fa59affd640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 0x7fa58806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.255+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fa5a0088f50 con 0x7fa5a406c040 2023-10-14T16:53:32.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.255+0000 7fa5a9c0a640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 0x7fa58806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.256+0000 7fa5a9c0a640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 0x7fa58806d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa594036000 tx=0x7fa59403c040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.259+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5a0053bf0 con 0x7fa5a406c040 2023-10-14T16:53:32.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.475+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir6", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa5a40cc550 con 0x7fa58806b070 2023-10-14T16:53:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.490+0000 7fa59affd640 1 -- 172.21.15.29:0/2300309672 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fa594031a80 con 0x7fa58806b070 2023-10-14T16:53:32.491 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:32.491 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:03", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.492+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 msgr2=0x7fa58806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.492+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 0x7fa58806d530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa594036000 tx=0x7fa59403c040 comp rx=0 tx=0).stop 2023-10-14T16:53:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.492+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 msgr2=0x7fa5a40c7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.492+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7fa5a000c400 tx=0x7fa5a000c8c0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 shutdown_connections 2023-10-14T16:53:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5a40ccfb0 0x7fa5a40b00e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fa58806b070 0x7fa58806d530 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5a40ad970 0x7fa5a4159e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 --2- 172.21.15.29:0/2300309672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5a406c040 0x7fa5a40c7df0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 >> 172.21.15.29:0/2300309672 conn(0x7fa5a40ce6a0 msgr2=0x7fa5a40b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 shutdown_connections 2023-10-14T16:53:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.493+0000 7fa5ab40d640 1 -- 172.21.15.29:0/2300309672 wait complete. 2023-10-14T16:53:32.511 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir6", "rel_path": "/snap_test_dir1/dir6", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:03", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:32.511 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir6', 'rel_path': '/snap_test_dir1/dir6', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:03', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:32.512 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:32.512 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:32.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:32.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-14T16:53:32.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:32.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:32.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:32.571 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:32.571 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:32.571 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:32.571 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:32.571 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:32.572 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:32.572 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:32.572 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:32.573 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_01_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_00_UTC', 'scheduled-2023-10-14-16_48_01_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_00_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_00_UTC', 'scheduled-2023-10-14-16_53_00_UTC'] 2023-10-14T16:53:32.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir7 2023-10-14T16:53:32.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.803+0000 7f56e8cd1640 1 Processor -- start 2023-10-14T16:53:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.804+0000 7f56e8cd1640 1 -- start start 2023-10-14T16:53:32.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e8cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56e406c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e8cd1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e415d910 con 0x7f56e4159950 2023-10-14T16:53:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e8cd1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56e415dbf0 con 0x7f56e4070b60 2023-10-14T16:53:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e8cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e415e490 con 0x7f56e40ca0a0 2023-10-14T16:53:32.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56e4159950 0x7f56e4070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51638/0 (socket says 172.21.15.29:51638) 2023-10-14T16:53:32.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.805+0000 7f56e37fe640 1 -- 172.21.15.29:0/2711196684 learned_addr learned my addr 172.21.15.29:0/2711196684 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e3fff640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56e406c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1562102626 0 0) 0x7f56e415d910 con 0x7f56e4159950 2023-10-14T16:53:32.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0003520 con 0x7f56e4159950 2023-10-14T16:53:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3805719340 0 0) 0x7f56d0003520 con 0x7f56e4159950 2023-10-14T16:53:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 msgr2=0x7f56e406c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56e406c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 >> v1:172.21.15.186:6789/0 conn(0x7f56e4070b60 legacy=0x7f56e406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e415ded0 con 0x7f56e4159950 2023-10-14T16:53:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f56d40019f0 con 0x7f56e4159950 2023-10-14T16:53:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 >> v1:172.21.15.29:6789/0 conn(0x7f56e4159950 legacy=0x7f56e4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56d00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56d00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d0004d00 0x7f56d0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e415e490 con 0x7f56d0003f20 2023-10-14T16:53:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d000bb10 con 0x7f56d0004d00 2023-10-14T16:53:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.806+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d000dee0 con 0x7f56e40ca0a0 2023-10-14T16:53:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56d00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e37fe640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56d00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e3fff640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d0004d00 0x7f56d0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 -- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 msgr2=0x7f56d00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56d00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 -- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d0004d00 msgr2=0x7f56d0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d0004d00 0x7f56d0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e2ffd640 1 -- 172.21.15.29:0/2711196684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d0003520 con 0x7f56d0003f20 2023-10-14T16:53:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e37fe640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40ca0a0 0x7f56d00039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.807+0000 7f56e3fff640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d0004d00 0x7f56d0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.808+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56d00047c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f56cc001750 tx=0x7f56d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bbc746f34107de6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0052d0 con 0x7f56d0003f20 2023-10-14T16:53:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56cc02e050 con 0x7f56d0003f20 2023-10-14T16:53:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e27fc640 1 -- 172.21.15.29:0/2711196684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0035f0 con 0x7f56d0003f20 2023-10-14T16:53:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e8cd1640 1 -- 172.21.15.29:0/2711196684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 msgr2=0x7f56d00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/2711196684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56d00047c0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f56cc001750 tx=0x7f56d00057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e3fff640 1 -- 172.21.15.29:0/2711196684 reap_dead start 2023-10-14T16:53:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e8cd1640 1 -- 172.21.15.29:0/2711196684 shutdown_connections 2023-10-14T16:53:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.809+0000 7f56e8cd1640 1 -- 172.21.15.29:0/2711196684 >> 172.21.15.29:0/2711196684 conn(0x7f56e40ce6a0 msgr2=0x7f56e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:32.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.810+0000 7f56e8cd1640 1 -- 172.21.15.29:0/2711196684 shutdown_connections 2023-10-14T16:53:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.810+0000 7f56e8cd1640 1 -- 172.21.15.29:0/2711196684 wait complete. 2023-10-14T16:53:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.811+0000 7f56e8cd1640 1 Processor -- start 2023-10-14T16:53:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.811+0000 7f56e8cd1640 1 -- start start 2023-10-14T16:53:32.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40b0810 0x7f56e40c6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56e40c6570 0x7f56e406c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e4156140 con 0x7f56d0003f20 2023-10-14T16:53:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40af480 con 0x7f56e40c6570 2023-10-14T16:53:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.812+0000 7f56e8cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56e40af760 con 0x7f56e40b0810 2023-10-14T16:53:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41926/0 (socket says 172.21.15.29:41926) 2023-10-14T16:53:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 -- 172.21.15.29:0/1462671992 learned_addr learned my addr 172.21.15.29:0/1462671992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 -- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40b0810 msgr2=0x7f56e40c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40b0810 0x7f56e40c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 -- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56e40c6570 msgr2=0x7f56e406c1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56e40c6570 0x7f56e406c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 -- 172.21.15.29:0/1462671992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40afa40 con 0x7f56d0003f20 2023-10-14T16:53:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.813+0000 7f56e37fe640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f56d4002170 tx=0x7f56d400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.814+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d4004d60 con 0x7f56d0003f20 2023-10-14T16:53:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.814+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56d400be60 con 0x7f56d0003f20 2023-10-14T16:53:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.814+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56d40072d0 con 0x7f56d0003f20 2023-10-14T16:53:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.814+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56e40afd20 con 0x7f56d0003f20 2023-10-14T16:53:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.814+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56e40b0c90 con 0x7f56d0003f20 2023-10-14T16:53:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.815+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56e404e9b0 con 0x7f56d0003f20 2023-10-14T16:53:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.816+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f56d4007430 con 0x7f56d0003f20 2023-10-14T16:53:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.817+0000 7f56dbfff640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 0x7f56c406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.817+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f56d4089b70 con 0x7f56d0003f20 2023-10-14T16:53:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.817+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 0x7f56c406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.818+0000 7f56e2ffd640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 0x7f56c406d530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56cc001750 tx=0x7f56cc02f000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:32.822+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56d40547a0 con 0x7f56d0003f20 2023-10-14T16:53:33.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.039+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir7", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f56e4159d30 con 0x7f56c406b070 2023-10-14T16:53:33.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.053+0000 7f56dbfff640 1 -- 172.21.15.29:0/1462671992 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f56cc0052d0 con 0x7f56c406b070 2023-10-14T16:53:33.053 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:33.054 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:04", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 msgr2=0x7f56c406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 0x7f56c406d530 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f56cc001750 tx=0x7f56cc02f000 comp rx=0 tx=0).stop 2023-10-14T16:53:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 msgr2=0x7f56e4155c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f56d4002170 tx=0x7f56d400b4a0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 shutdown_connections 2023-10-14T16:53:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f56c406b070 0x7f56c406d530 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56e40c6570 0x7f56e406c1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56e40b0810 0x7f56e40c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.055+0000 7f56e8cd1640 1 --2- 172.21.15.29:0/1462671992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d0003f20 0x7f56e4155c00 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.056+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 >> 172.21.15.29:0/1462671992 conn(0x7f56e40ce6a0 msgr2=0x7f56e41563b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.056+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 shutdown_connections 2023-10-14T16:53:33.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.056+0000 7f56e8cd1640 1 -- 172.21.15.29:0/1462671992 wait complete. 2023-10-14T16:53:33.076 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir7", "rel_path": "/snap_test_dir1/dir7", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:04", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:33.076 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir7', 'rel_path': '/snap_test_dir1/dir7', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:04', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:33.077 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:33.077 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:33.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:33.078 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-14T16:53:33.134 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:33.134 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:33.134 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:33.134 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:33.135 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:33.135 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:33.135 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:33.136 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:33.136 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:33.136 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:33.136 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:33.137 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_01_UTC', 'scheduled-2023-10-14-16_44_01_UTC', 'scheduled-2023-10-14-16_45_01_UTC', 'scheduled-2023-10-14-16_46_00_UTC', 'scheduled-2023-10-14-16_47_01_UTC', 'scheduled-2023-10-14-16_48_01_UTC', 'scheduled-2023-10-14-16_49_01_UTC', 'scheduled-2023-10-14-16_50_00_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_01_UTC', 'scheduled-2023-10-14-16_53_00_UTC'] 2023-10-14T16:53:33.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir8 2023-10-14T16:53:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.354+0000 7fcd5759e640 1 Processor -- start 2023-10-14T16:53:33.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.354+0000 7fcd5759e640 1 -- start start 2023-10-14T16:53:33.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.356+0000 7fcd5759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcd580c9060 0x7fcd5806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.356+0000 7fcd5759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd580ca560 con 0x7fcd580c9440 2023-10-14T16:53:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.356+0000 7fcd5759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd58165b70 con 0x7fcd5806c9e0 2023-10-14T16:53:33.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.356+0000 7fcd5759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd580c9950 con 0x7fcd580c9060 2023-10-14T16:53:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.357+0000 7fcd56d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcd580c9440 0x7fcd58165460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51640/0 (socket says 172.21.15.29:51640) 2023-10-14T16:53:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.357+0000 7fcd56d9d640 1 -- 172.21.15.29:0/1233454712 learned_addr learned my addr 172.21.15.29:0/1233454712 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:33.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.357+0000 7fcd5659c640 1 --2- 172.21.15.29:0/1233454712 >> v2:172.21.15.186:3300/0 conn(0x7fcd580c9060 0x7fcd5806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.357+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4146167580 0 0) 0x7fcd580ca560 con 0x7fcd580c9440 2023-10-14T16:53:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.357+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd40003520 con 0x7fcd580c9440 2023-10-14T16:53:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4182927736 0 0) 0x7fcd40003520 con 0x7fcd580c9440 2023-10-14T16:53:33.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 >> v2:172.21.15.186:3300/0 conn(0x7fcd580c9060 msgr2=0x7fcd5806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 --2- 172.21.15.29:0/1233454712 >> v2:172.21.15.186:3300/0 conn(0x7fcd580c9060 0x7fcd5806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 >> v1:172.21.15.186:6789/0 conn(0x7fcd5806c9e0 legacy=0x7fcd58164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd580c9c30 con 0x7fcd580c9440 2023-10-14T16:53:33.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fcd500019f0 con 0x7fcd580c9440 2023-10-14T16:53:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 >> v1:172.21.15.29:6789/0 conn(0x7fcd580c9440 legacy=0x7fcd58165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd40003970 0x7fcd40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd40005180 0x7fcd4000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd580c9950 con 0x7fcd400047c0 2023-10-14T16:53:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4000c020 con 0x7fcd40003970 2023-10-14T16:53:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd4000c1e0 con 0x7fcd40005180 2023-10-14T16:53:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd55d9b640 1 -- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd40005180 msgr2=0x7fcd4000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:33.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.358+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd40005180 0x7fcd4000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.359+0000 7fcd55d9b640 1 -- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd40003970 msgr2=0x7fcd40003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.359+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd40003970 0x7fcd40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.359+0000 7fcd55d9b640 1 -- 172.21.15.29:0/1233454712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd40003520 con 0x7fcd400047c0 2023-10-14T16:53:33.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.359+0000 7fcd56d9d640 1 -- 172.21.15.29:0/1233454712 reap_dead start 2023-10-14T16:53:33.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd40004c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c001720 tx=0x7fcd400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c609aa3530e7adf2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c0052d0 con 0x7fcd400047c0 2023-10-14T16:53:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4c02e050 con 0x7fcd400047c0 2023-10-14T16:53:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd5559a640 1 -- 172.21.15.29:0/1233454712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4c003830 con 0x7fcd400047c0 2023-10-14T16:53:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd5759e640 1 -- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 msgr2=0x7fcd40004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.360+0000 7fcd5759e640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd40004c40 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c001720 tx=0x7fcd400057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.361+0000 7fcd5759e640 1 -- 172.21.15.29:0/1233454712 shutdown_connections 2023-10-14T16:53:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.361+0000 7fcd5759e640 1 --2- 172.21.15.29:0/1233454712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd40004c40 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.361+0000 7fcd5759e640 1 -- 172.21.15.29:0/1233454712 >> 172.21.15.29:0/1233454712 conn(0x7fcd580ce6a0 msgr2=0x7fcd580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.361+0000 7fcd5759e640 1 -- 172.21.15.29:0/1233454712 shutdown_connections 2023-10-14T16:53:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.361+0000 7fcd5759e640 1 -- 172.21.15.29:0/1233454712 wait complete. 2023-10-14T16:53:33.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.362+0000 7fcd5759e640 1 Processor -- start 2023-10-14T16:53:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.363+0000 7fcd5759e640 1 -- start start 2023-10-14T16:53:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.363+0000 7fcd5759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.363+0000 7fcd5759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd58164380 0x7fcd5814d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd5814d780 0x7fcd581500e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd58163630 con 0x7fcd400047c0 2023-10-14T16:53:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5814dc00 con 0x7fcd5814d780 2023-10-14T16:53:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd581638a0 con 0x7fcd58164380 2023-10-14T16:53:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41942/0 (socket says 172.21.15.29:41942) 2023-10-14T16:53:33.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 -- 172.21.15.29:0/3355510367 learned_addr learned my addr 172.21.15.29:0/3355510367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 -- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd58164380 msgr2=0x7fcd5814d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd58164380 0x7fcd5814d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 -- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd5814d780 msgr2=0x7fcd581500e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd56d9d640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd5814d780 0x7fcd581500e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd5814d780 0x7fcd581500e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.364+0000 7fcd5659c640 1 -- 172.21.15.29:0/3355510367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd58163b80 con 0x7fcd400047c0 2023-10-14T16:53:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd5659c640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800e770 tx=0x7fcd4800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800c730 con 0x7fcd400047c0 2023-10-14T16:53:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd4800c890 con 0x7fcd400047c0 2023-10-14T16:53:33.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd58163e60 con 0x7fcd400047c0 2023-10-14T16:53:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd4800cc40 con 0x7fcd400047c0 2023-10-14T16:53:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.365+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd58161cc0 con 0x7fcd400047c0 2023-10-14T16:53:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.366+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd580c1ed0 con 0x7fcd400047c0 2023-10-14T16:53:33.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.367+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fcd48013470 con 0x7fcd400047c0 2023-10-14T16:53:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.368+0000 7fcd46ffd640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 0x7fcd3806d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.368+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7fcd48089d70 con 0x7fcd400047c0 2023-10-14T16:53:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.368+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 0x7fcd3806d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.369+0000 7fcd55d9b640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 0x7fcd3806d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c033000 tx=0x7fcd4c02f000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.372+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd48054a00 con 0x7fcd400047c0 2023-10-14T16:53:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.590+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir8", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcd580c9f10 con 0x7fcd3806b0a0 2023-10-14T16:53:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.608+0000 7fcd46ffd640 1 -- 172.21.15.29:0/3355510367 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7fcd4c0052d0 con 0x7fcd3806b0a0 2023-10-14T16:53:33.609 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:33.609 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:05", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 msgr2=0x7fcd3806d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 0x7fcd3806d560 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcd4c033000 tx=0x7fcd4c02f000 comp rx=0 tx=0).stop 2023-10-14T16:53:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 msgr2=0x7fcd58161780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7fcd4800e770 tx=0x7fcd4800ec30 comp rx=0 tx=0).stop 2023-10-14T16:53:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 shutdown_connections 2023-10-14T16:53:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fcd3806b0a0 0x7fcd3806d560 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd5814d780 0x7fcd581500e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd58164380 0x7fcd5814d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 --2- 172.21.15.29:0/3355510367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd400047c0 0x7fcd58161780 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 >> 172.21.15.29:0/3355510367 conn(0x7fcd580ce6a0 msgr2=0x7fcd58160c80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 shutdown_connections 2023-10-14T16:53:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.611+0000 7fcd5759e640 1 -- 172.21.15.29:0/3355510367 wait complete. 2023-10-14T16:53:33.631 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir8", "rel_path": "/snap_test_dir1/dir8", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:05", "first": "2023-10-14T16:43:01", "last": "2023-10-14T16:53:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:33.632 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir8', 'rel_path': '/snap_test_dir1/dir8', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:05', 'first': '2023-10-14T16:43:01', 'last': '2023-10-14T16:53:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:33.632 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:33.632 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:33.633 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:33.633 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-14T16:53:33.696 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:33.696 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:33.696 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:33.696 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:33.697 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:33.698 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:33.699 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-16_43_00_UTC', 'scheduled-2023-10-14-16_44_00_UTC', 'scheduled-2023-10-14-16_45_01_UTC', 'scheduled-2023-10-14-16_46_01_UTC', 'scheduled-2023-10-14-16_47_01_UTC', 'scheduled-2023-10-14-16_48_00_UTC', 'scheduled-2023-10-14-16_49_00_UTC', 'scheduled-2023-10-14-16_50_01_UTC', 'scheduled-2023-10-14-16_51_00_UTC', 'scheduled-2023-10-14-16_52_01_UTC', 'scheduled-2023-10-14-16_53_01_UTC'] 2023-10-14T16:53:33.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/dir9 2023-10-14T16:53:33.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.925+0000 7f9bd87f0640 1 Processor -- start 2023-10-14T16:53:33.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.925+0000 7f9bd87f0640 1 -- start start 2023-10-14T16:53:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.926+0000 7f9bd87f0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9bd015ad70 0x7f9bd0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.926+0000 7f9bd87f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bd00caac0 0x7f9bd006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.926+0000 7f9bd87f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bd00cb030 con 0x7f9bd00c9ca0 2023-10-14T16:53:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd87f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00c91c0 con 0x7f9bd00caac0 2023-10-14T16:53:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd87f0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bd00c94a0 con 0x7f9bd015ad70 2023-10-14T16:53:33.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd5d8b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9bd00c9ca0 0x7f9bd00ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51654/0 (socket says 172.21.15.29:51654) 2023-10-14T16:53:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd5d8b640 1 -- 172.21.15.29:0/2983916970 learned_addr learned my addr 172.21.15.29:0/2983916970 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:33.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2983916970 >> v2:172.21.15.29:3300/0 conn(0x7f9bd00caac0 0x7f9bd006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd658c640 1 --2- 172.21.15.29:0/2983916970 >> v2:172.21.15.186:3300/0 conn(0x7f9bd015ad70 0x7f9bd0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd558a640 1 -- 172.21.15.29:0/2983916970 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740701889 0 0) 0x7f9bd00cb030 con 0x7f9bd00c9ca0 2023-10-14T16:53:33.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.927+0000 7f9bd558a640 1 -- 172.21.15.29:0/2983916970 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bbc003520 con 0x7f9bd00c9ca0 2023-10-14T16:53:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.928+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2983916970 >> v2:172.21.15.186:3300/0 conn(0x7f9bd015ad70 msgr2=0x7f9bd0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.928+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2983916970 >> v2:172.21.15.186:3300/0 conn(0x7f9bd015ad70 0x7f9bd0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.928+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2983916970 >> v1:172.21.15.29:6789/0 conn(0x7f9bd00c9ca0 legacy=0x7f9bd00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.928+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2983916970 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd00c9780 con 0x7f9bd00caac0 2023-10-14T16:53:33.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.929+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2983916970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd00caac0 0x7f9bd006a2e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00cf780 tx=0x7f9bc802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3c0a45ce9f33fc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd558a640 1 -- 172.21.15.29:0/2983916970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc802cc70 con 0x7f9bd00caac0 2023-10-14T16:53:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd558a640 1 -- 172.21.15.29:0/2983916970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc802cdd0 con 0x7f9bd00caac0 2023-10-14T16:53:33.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd558a640 1 -- 172.21.15.29:0/2983916970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8034640 con 0x7f9bd00caac0 2023-10-14T16:53:33.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2983916970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd00caac0 msgr2=0x7f9bd006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2983916970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd00caac0 0x7f9bd006a2e0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f9bd00cf780 tx=0x7f9bc802bec0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2983916970 shutdown_connections 2023-10-14T16:53:33.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2983916970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd00caac0 0x7f9bd006a2e0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2983916970 >> v2:172.21.15.186:3300/0 conn(0x7f9bd015ad70 0x7f9bd0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.930+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2983916970 >> 172.21.15.29:0/2983916970 conn(0x7f9bd00ce9b0 msgr2=0x7f9bd00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:33.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.931+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2983916970 shutdown_connections 2023-10-14T16:53:33.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.931+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2983916970 wait complete. 2023-10-14T16:53:33.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.932+0000 7f9bd87f0640 1 Processor -- start 2023-10-14T16:53:33.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.933+0000 7f9bd87f0640 1 -- start start 2023-10-14T16:53:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 0x7f9bd0158ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd00caac0 0x7f9bd015d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd00c4cd0 con 0x7f9bd015ad70 2023-10-14T16:53:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015d490 con 0x7f9bd00caac0 2023-10-14T16:53:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd87f0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd015c8c0 con 0x7f9bd00c9ca0 2023-10-14T16:53:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41956/0 (socket says 172.21.15.29:41956) 2023-10-14T16:53:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2643208875 learned_addr learned my addr 172.21.15.29:0/2643208875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd5d8b640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd00caac0 0x7f9bd015d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd658c640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 0x7f9bd0158ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 msgr2=0x7f9bd0158ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 0x7f9bd0158ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd00caac0 msgr2=0x7f9bd015d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd00caac0 0x7f9bd015d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:33.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd6d8d640 1 -- 172.21.15.29:0/2643208875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc8009a70 con 0x7f9bd015ad70 2023-10-14T16:53:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.934+0000 7f9bd658c640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 0x7f9bd0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bd6d8d640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009e50 tx=0x7f9bc8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc8009450 con 0x7f9bd015ad70 2023-10-14T16:53:33.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bc803f420 con 0x7f9bd015ad70 2023-10-14T16:53:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bc803f7b0 con 0x7f9bd015ad70 2023-10-14T16:53:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd0159530 con 0x7f9bd015ad70 2023-10-14T16:53:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.935+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd015ce40 con 0x7f9bd015ad70 2023-10-14T16:53:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.937+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd004e160 con 0x7f9bd015ad70 2023-10-14T16:53:33.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.937+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9bc803f910 con 0x7f9bd015ad70 2023-10-14T16:53:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.938+0000 7f9bc2ffd640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 0x7f9bb406d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.939+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9bc80ac560 con 0x7f9bd015ad70 2023-10-14T16:53:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.939+0000 7f9bd658c640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 0x7f9bb406d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:33.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.939+0000 7f9bd658c640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 0x7f9bb406d580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc006fd0 tx=0x7f9bcc008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:33.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:33.943+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bc8077270 con 0x7f9bd015ad70 2023-10-14T16:53:34.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.163+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/dir9", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9bd00ad4d0 con 0x7f9bb406b0c0 2023-10-14T16:53:34.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.179+0000 7f9bc2ffd640 1 -- 172.21.15.29:0/2643208875 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+360 (secure 0 0 0) 0x7f9bcc009530 con 0x7f9bb406b0c0 2023-10-14T16:53:34.180 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:34.180 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:05", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 msgr2=0x7f9bb406d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 0x7f9bb406d580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9bcc006fd0 tx=0x7f9bcc008040 comp rx=0 tx=0).stop 2023-10-14T16:53:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 msgr2=0x7f9bd015de60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f9bc8009e50 tx=0x7f9bc8004a40 comp rx=0 tx=0).stop 2023-10-14T16:53:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 shutdown_connections 2023-10-14T16:53:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bd015ad70 0x7f9bd015de60 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9bb406b0c0 0x7f9bb406d580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bd00caac0 0x7f9bd015d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 --2- 172.21.15.29:0/2643208875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bd00c9ca0 0x7f9bd0158ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 >> 172.21.15.29:0/2643208875 conn(0x7f9bd00ce9b0 msgr2=0x7f9bd00af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 shutdown_connections 2023-10-14T16:53:34.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.182+0000 7f9bd87f0640 1 -- 172.21.15.29:0/2643208875 wait complete. 2023-10-14T16:53:34.203 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/dir9", "rel_path": "/snap_test_dir1/dir9", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:42:05", "first": "2023-10-14T16:43:00", "last": "2023-10-14T16:53:01", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": false}] 2023-10-14T16:53:34.203 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/dir9', 'rel_path': '/snap_test_dir1/dir9', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T16:42:05', 'first': '2023-10-14T16:43:00', 'last': '2023-10-14T16:53:01', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': False} 2023-10-14T16:53:34.204 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T16:53:34.204 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T16:53:34.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir0 1m 2023-10-14T16:53:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.432+0000 7f9c3d789640 1 Processor -- start 2023-10-14T16:53:34.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.432+0000 7f9c3d789640 1 -- start start 2023-10-14T16:53:34.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c3d789640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9c3815ad70 0x7f9c38069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c3d789640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c380c9ca0 0x7f9c3806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c3d789640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c380cb030 con 0x7f9c380ca3b0 2023-10-14T16:53:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c3d789640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c380c91c0 con 0x7f9c380c9ca0 2023-10-14T16:53:34.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c3d789640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c380c94a0 con 0x7f9c3815ad70 2023-10-14T16:53:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c367fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c380c9ca0 0x7f9c3806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c377fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9c380ca3b0 0x7f9c380ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51670/0 (socket says 172.21.15.29:51670) 2023-10-14T16:53:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.434+0000 7f9c377fe640 1 -- 172.21.15.29:0/3677860993 learned_addr learned my addr 172.21.15.29:0/3677860993 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c36ffd640 1 --2- 172.21.15.29:0/3677860993 >> v2:172.21.15.186:3300/0 conn(0x7f9c3815ad70 0x7f9c38069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c367fc640 1 -- 172.21.15.29:0/3677860993 >> v2:172.21.15.186:3300/0 conn(0x7f9c3815ad70 msgr2=0x7f9c38069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c367fc640 1 --2- 172.21.15.29:0/3677860993 >> v2:172.21.15.186:3300/0 conn(0x7f9c3815ad70 0x7f9c38069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c367fc640 1 -- 172.21.15.29:0/3677860993 >> v1:172.21.15.29:6789/0 conn(0x7f9c380ca3b0 legacy=0x7f9c380ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c367fc640 1 -- 172.21.15.29:0/3677860993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c380c9780 con 0x7f9c380c9ca0 2023-10-14T16:53:34.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.435+0000 7f9c35ffb640 1 -- 172.21.15.29:0/3677860993 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109591166 0 0) 0x7f9c380cb030 con 0x7f9c380ca3b0 2023-10-14T16:53:34.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.436+0000 7f9c367fc640 1 --2- 172.21.15.29:0/3677860993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c3806a2e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9c380cf780 tx=0x7f9c2c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6304d348178498a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c35ffb640 1 -- 172.21.15.29:0/3677860993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c031830 con 0x7f9c380c9ca0 2023-10-14T16:53:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c35ffb640 1 -- 172.21.15.29:0/3677860993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2c031990 con 0x7f9c380c9ca0 2023-10-14T16:53:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c35ffb640 1 -- 172.21.15.29:0/3677860993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2c031d20 con 0x7f9c380c9ca0 2023-10-14T16:53:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 -- 172.21.15.29:0/3677860993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 msgr2=0x7f9c3806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 --2- 172.21.15.29:0/3677860993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c3806a2e0 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f9c380cf780 tx=0x7f9c2c02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 -- 172.21.15.29:0/3677860993 shutdown_connections 2023-10-14T16:53:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 --2- 172.21.15.29:0/3677860993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c3806a2e0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 --2- 172.21.15.29:0/3677860993 >> v2:172.21.15.186:3300/0 conn(0x7f9c3815ad70 0x7f9c38069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 -- 172.21.15.29:0/3677860993 >> 172.21.15.29:0/3677860993 conn(0x7f9c380ce9b0 msgr2=0x7f9c380cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 -- 172.21.15.29:0/3677860993 shutdown_connections 2023-10-14T16:53:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.437+0000 7f9c3d789640 1 -- 172.21.15.29:0/3677860993 wait complete. 2023-10-14T16:53:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.439+0000 7f9c3d789640 1 Processor -- start 2023-10-14T16:53:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.439+0000 7f9c3d789640 1 -- start start 2023-10-14T16:53:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c380ca3b0 0x7f9c380ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c3815ad70 0x7f9c380cd5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c7ab0 con 0x7f9c380c9ca0 2023-10-14T16:53:34.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c6a10 con 0x7f9c3815ad70 2023-10-14T16:53:34.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.440+0000 7f9c3d789640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c380c6cf0 con 0x7f9c380ca3b0 2023-10-14T16:53:34.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:34.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41974/0 (socket says 172.21.15.29:41974) 2023-10-14T16:53:34.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 -- 172.21.15.29:0/2251113958 learned_addr learned my addr 172.21.15.29:0/2251113958 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:34.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 -- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c380ca3b0 msgr2=0x7f9c380ae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c380ca3b0 0x7f9c380ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 -- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c3815ad70 msgr2=0x7f9c380cd5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c3815ad70 0x7f9c380cd5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.441+0000 7f9c36ffd640 1 -- 172.21.15.29:0/2251113958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c2c009400 con 0x7f9c380c9ca0 2023-10-14T16:53:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c36ffd640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800e870 tx=0x7f9c2800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800c940 con 0x7f9c380c9ca0 2023-10-14T16:53:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c2800caa0 con 0x7f9c380c9ca0 2023-10-14T16:53:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c380c6fd0 con 0x7f9c380c9ca0 2023-10-14T16:53:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c2800ce50 con 0x7f9c380c9ca0 2023-10-14T16:53:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.442+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c380c7590 con 0x7f9c380c9ca0 2023-10-14T16:53:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.443+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c3804e160 con 0x7f9c380c9ca0 2023-10-14T16:53:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.444+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f9c28013470 con 0x7f9c380c9ca0 2023-10-14T16:53:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.445+0000 7f9c1f7fe640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 0x7f9c2406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.445+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f9c28089b80 con 0x7f9c380c9ca0 2023-10-14T16:53:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.446+0000 7f9c367fc640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 0x7f9c2406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.446+0000 7f9c367fc640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 0x7f9c2406d560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c036000 tx=0x7f9c2c030000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.450+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c280548b0 con 0x7f9c380c9ca0 2023-10-14T16:53:34.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.670+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir0", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f9c380c6e60 con 0x7f9c2406b0a0 2023-10-14T16:53:34.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.907+0000 7f9c1f7fe640 1 -- 172.21.15.29:0/2251113958 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f9c380c6e60 con 0x7f9c2406b0a0 2023-10-14T16:53:34.909 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir0 2023-10-14T16:53:34.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.911+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 msgr2=0x7f9c2406d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.911+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 0x7f9c2406d560 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f9c2c036000 tx=0x7f9c2c030000 comp rx=0 tx=0).stop 2023-10-14T16:53:34.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.911+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 msgr2=0x7f9c38158200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:34.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.911+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f9c2800e870 tx=0x7f9c2800ed30 comp rx=0 tx=0).stop 2023-10-14T16:53:34.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 shutdown_connections 2023-10-14T16:53:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f9c2406b0a0 0x7f9c2406d560 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c3815ad70 0x7f9c380cd5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c380ca3b0 0x7f9c380ae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 --2- 172.21.15.29:0/2251113958 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c380c9ca0 0x7f9c38158200 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:34.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 >> 172.21.15.29:0/2251113958 conn(0x7f9c380ce9b0 msgr2=0x7f9c38070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:34.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 shutdown_connections 2023-10-14T16:53:34.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:34.912+0000 7f9c3d789640 1 -- 172.21.15.29:0/2251113958 wait complete. 2023-10-14T16:53:34.937 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir0 2023-10-14T16:53:34.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:34.939 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir0/.snap) 2023-10-14T16:53:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_42_00_UTC 2023-10-14T16:53:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:34.996 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:34.997 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:34.998 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:34.999 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_42_00_UTC 2023-10-14T16:53:35.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.000 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_42_00_UTC) 2023-10-14T16:53:35.075 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:35.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.076 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_43_01_UTC) 2023-10-14T16:53:35.150 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:35.151 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.151 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_44_01_UTC) 2023-10-14T16:53:35.222 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:35.222 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.223 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_45_01_UTC) 2023-10-14T16:53:35.297 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:35.298 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.298 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_46_01_UTC) 2023-10-14T16:53:35.373 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:35.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.374 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_47_01_UTC) 2023-10-14T16:53:35.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:35.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.449 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_48_01_UTC) 2023-10-14T16:53:35.526 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:35.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.528 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:35.607 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:35.607 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.607 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_50_00_UTC) 2023-10-14T16:53:35.684 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:35.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.685 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_51_01_UTC) 2023-10-14T16:53:35.758 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:35.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.759 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_52_01_UTC) 2023-10-14T16:53:35.833 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:35.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.834 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0/.snap/scheduled-2023-10-14-16_53_00_UTC) 2023-10-14T16:53:35.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:35.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir0) 2023-10-14T16:53:35.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir1 1m 2023-10-14T16:53:36.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.203+0000 7ff03e6e6640 1 Processor -- start 2023-10-14T16:53:36.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.203+0000 7ff03e6e6640 1 -- start start 2023-10-14T16:53:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.204+0000 7ff03e6e6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff03815cd70 0x7ff0380d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03e6e6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff0380b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03e6e6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff03815d150 con 0x7ff0380b8480 2023-10-14T16:53:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03e6e6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0380d1180 con 0x7ff03815cd70 2023-10-14T16:53:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03e6e6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380d1460 con 0x7ff0380cded0 2023-10-14T16:53:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03cee3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff0380b8480 0x7ff0380cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51674/0 (socket says 172.21.15.29:51674) 2023-10-14T16:53:36.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03cee3640 1 -- 172.21.15.29:0/513517453 learned_addr learned my addr 172.21.15.29:0/513517453 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03dee5640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff0380b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640841231 0 0) 0x7ff03815d150 con 0x7ff0380b8480 2023-10-14T16:53:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> v2:172.21.15.186:3300/0 conn(0x7ff03815cd70 0x7ff0380d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.205+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff024003520 con 0x7ff0380b8480 2023-10-14T16:53:36.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1804289345 0 0) 0x7ff024003520 con 0x7ff0380b8480 2023-10-14T16:53:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 msgr2=0x7ff0380b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff0380b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 >> v2:172.21.15.186:3300/0 conn(0x7ff03815cd70 msgr2=0x7ff0380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 --2- 172.21.15.29:0/513517453 >> v2:172.21.15.186:3300/0 conn(0x7ff03815cd70 0x7ff0380d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0380d1740 con 0x7ff0380b8480 2023-10-14T16:53:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff034001690 con 0x7ff0380b8480 2023-10-14T16:53:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 >> v1:172.21.15.29:6789/0 conn(0x7ff0380b8480 legacy=0x7ff0380cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff024003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0240047c0 0x7ff024004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff024005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380d1180 con 0x7ff024003970 2023-10-14T16:53:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0380d1460 con 0x7ff0240047c0 2023-10-14T16:53:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02400ba30 con 0x7ff0380cded0 2023-10-14T16:53:36.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> v2:172.21.15.186:3300/0 conn(0x7ff03815cd70 0x7ff0380d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.206+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff024003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03cee3640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0240047c0 0x7ff024004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03dee5640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff024005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03d6e4640 1 -- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 msgr2=0x7ff024005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:36.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff024005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03d6e4640 1 -- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0240047c0 msgr2=0x7ff024004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0240047c0 0x7ff024004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03d6e4640 1 -- 172.21.15.29:0/513517453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff024003520 con 0x7ff024003970 2023-10-14T16:53:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.207+0000 7ff03dee5640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cded0 0x7ff024005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:36.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.208+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff024003d90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0098a0 tx=0x7ff0240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc1326016a16893 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.208+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c0305f0 con 0x7ff024003970 2023-10-14T16:53:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.208+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c030750 con 0x7ff024003970 2023-10-14T16:53:36.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.208+0000 7ff02bfff640 1 -- 172.21.15.29:0/513517453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c030ac0 con 0x7ff024003970 2023-10-14T16:53:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 -- 172.21.15.29:0/513517453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 msgr2=0x7ff024003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/513517453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff024003d90 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff02c0098a0 tx=0x7ff0240057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03dee5640 1 -- 172.21.15.29:0/513517453 reap_dead start 2023-10-14T16:53:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 -- 172.21.15.29:0/513517453 shutdown_connections 2023-10-14T16:53:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 -- 172.21.15.29:0/513517453 >> 172.21.15.29:0/513517453 conn(0x7ff0380cf570 msgr2=0x7ff0380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 -- 172.21.15.29:0/513517453 shutdown_connections 2023-10-14T16:53:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.209+0000 7ff03e6e6640 1 -- 172.21.15.29:0/513517453 wait complete. 2023-10-14T16:53:36.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.211+0000 7ff03e6e6640 1 Processor -- start 2023-10-14T16:53:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.211+0000 7ff03e6e6640 1 -- start start 2023-10-14T16:53:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff038156550 0x7ff0380cb0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 0x7ff0381f20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038159380 con 0x7ff024003970 2023-10-14T16:53:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff038158790 con 0x7ff038156550 2023-10-14T16:53:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.212+0000 7ff03e6e6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0381589e0 con 0x7ff0380cb5e0 2023-10-14T16:53:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41992/0 (socket says 172.21.15.29:41992) 2023-10-14T16:53:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 -- 172.21.15.29:0/2435047313 learned_addr learned my addr 172.21.15.29:0/2435047313 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03dee5640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 0x7ff0381f20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03cee3640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff038156550 0x7ff0380cb0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 -- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 msgr2=0x7ff0381f20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 0x7ff0381f20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 -- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff038156550 msgr2=0x7ff0380cb0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff038156550 0x7ff0380cb0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03d6e4640 1 -- 172.21.15.29:0/2435047313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff038158cc0 con 0x7ff024003970 2023-10-14T16:53:36.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.213+0000 7ff03dee5640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 0x7ff0381f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:36.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff03d6e4640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009900 tx=0x7ff02c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c039a20 con 0x7ff024003970 2023-10-14T16:53:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff02c03f040 con 0x7ff024003970 2023-10-14T16:53:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff02c039dc0 con 0x7ff024003970 2023-10-14T16:53:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff038158fa0 con 0x7ff024003970 2023-10-14T16:53:36.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.214+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0380ccba0 con 0x7ff024003970 2023-10-14T16:53:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.215+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff03804e9b0 con 0x7ff024003970 2023-10-14T16:53:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.216+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff02c04c400 con 0x7ff024003970 2023-10-14T16:53:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.217+0000 7ff029ffb640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 0x7ff01c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:36.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.217+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff02c0abef0 con 0x7ff024003970 2023-10-14T16:53:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.218+0000 7ff03cee3640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 0x7ff01c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.218+0000 7ff03cee3640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 0x7ff01c06d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff03815ffb0 tx=0x7ff034009040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:36.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.222+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff02c076a80 con 0x7ff024003970 2023-10-14T16:53:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.442+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir1", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0380d1d00 con 0x7ff01c06b070 2023-10-14T16:53:36.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.678+0000 7ff029ffb640 1 -- 172.21.15.29:0/2435047313 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff034001d60 con 0x7ff01c06b070 2023-10-14T16:53:36.681 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir1 2023-10-14T16:53:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 msgr2=0x7ff01c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 0x7ff01c06d530 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff03815ffb0 tx=0x7ff034009040 comp rx=0 tx=0).stop 2023-10-14T16:53:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 msgr2=0x7ff0380cc480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:36.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7ff02c009900 tx=0x7ff02c004730 comp rx=0 tx=0).stop 2023-10-14T16:53:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 shutdown_connections 2023-10-14T16:53:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0380cb5e0 0x7ff0381f20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff01c06b070 0x7ff01c06d530 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff038156550 0x7ff0380cb0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 --2- 172.21.15.29:0/2435047313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff024003970 0x7ff0380cc480 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.682+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 >> 172.21.15.29:0/2435047313 conn(0x7ff0380cf570 msgr2=0x7ff0380b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.683+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 shutdown_connections 2023-10-14T16:53:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:36.683+0000 7ff03e6e6640 1 -- 172.21.15.29:0/2435047313 wait complete. 2023-10-14T16:53:36.707 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir1 2023-10-14T16:53:36.708 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:36.708 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir1/.snap) 2023-10-14T16:53:36.770 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:36.770 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:36.770 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:36.771 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:36.772 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:36.772 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:36.772 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:36.772 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:36.773 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:36.773 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:36.847 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:36.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:36.848 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_44_01_UTC) 2023-10-14T16:53:36.924 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:36.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:36.925 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_45_01_UTC) 2023-10-14T16:53:36.999 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:36.999 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.000 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_46_01_UTC) 2023-10-14T16:53:37.074 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:37.075 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.075 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_47_01_UTC) 2023-10-14T16:53:37.140 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:37.141 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.141 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:37.210 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:37.211 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.211 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:37.285 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:37.286 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.286 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_50_01_UTC) 2023-10-14T16:53:37.355 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:37.356 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.356 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:37.432 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:37.433 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.433 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_52_01_UTC) 2023-10-14T16:53:37.512 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:37.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1/.snap/scheduled-2023-10-14-16_53_01_UTC) 2023-10-14T16:53:37.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:37.589 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir1) 2023-10-14T16:53:37.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir2 1m 2023-10-14T16:53:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.880+0000 7ff89f59e640 1 Processor -- start 2023-10-14T16:53:37.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.880+0000 7ff89f59e640 1 -- start start 2023-10-14T16:53:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.881+0000 7ff89f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a015cba0 0x7ff8a0071b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.881+0000 7ff89f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8a00720d0 0x7ff8a00cd800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.881+0000 7ff89f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a00d0500 con 0x7ff8a00cdd40 2023-10-14T16:53:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.881+0000 7ff89f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a00ce1b0 con 0x7ff8a015cba0 2023-10-14T16:53:37.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.881+0000 7ff89f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a00d07c0 con 0x7ff8a00720d0 2023-10-14T16:53:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a015cba0 0x7ff8a0071b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff8a015cba0 0x7ff8a0071b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42002/0 (socket says 172.21.15.29:42002) 2023-10-14T16:53:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 -- 172.21.15.29:0/2451860641 learned_addr learned my addr 172.21.15.29:0/2451860641 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:37.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89dd9b640 1 --2- 172.21.15.29:0/2451860641 >> v2:172.21.15.186:3300/0 conn(0x7ff8a00720d0 0x7ff8a00cd800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 -- 172.21.15.29:0/2451860641 >> v2:172.21.15.186:3300/0 conn(0x7ff8a00720d0 msgr2=0x7ff8a00cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 --2- 172.21.15.29:0/2451860641 >> v2:172.21.15.186:3300/0 conn(0x7ff8a00720d0 0x7ff8a00cd800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.882+0000 7ff89e59c640 1 -- 172.21.15.29:0/2451860641 >> v1:172.21.15.186:6789/0 conn(0x7ff8a00cdd40 legacy=0x7ff8a01607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:37.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.883+0000 7ff89e59c640 1 -- 172.21.15.29:0/2451860641 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a00d0aa0 con 0x7ff8a015cba0 2023-10-14T16:53:37.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.884+0000 7ff89e59c640 1 --2- 172.21.15.29:0/2451860641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a015cba0 0x7ff8a0071b90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00ce3f0 tx=0x7ff89002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74d90d445889a367 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:37.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.884+0000 7ff89d59a640 1 -- 172.21.15.29:0/2451860641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff89002ca80 con 0x7ff8a015cba0 2023-10-14T16:53:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.884+0000 7ff89d59a640 1 -- 172.21.15.29:0/2451860641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89002cbe0 con 0x7ff8a015cba0 2023-10-14T16:53:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.884+0000 7ff89d59a640 1 -- 172.21.15.29:0/2451860641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8900343f0 con 0x7ff8a015cba0 2023-10-14T16:53:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 -- 172.21.15.29:0/2451860641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a015cba0 msgr2=0x7ff8a0071b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:37.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 --2- 172.21.15.29:0/2451860641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a015cba0 0x7ff8a0071b90 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7ff8a00ce3f0 tx=0x7ff89002bee0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 -- 172.21.15.29:0/2451860641 shutdown_connections 2023-10-14T16:53:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 --2- 172.21.15.29:0/2451860641 >> v2:172.21.15.186:3300/0 conn(0x7ff8a00720d0 0x7ff8a00cd800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 --2- 172.21.15.29:0/2451860641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a015cba0 0x7ff8a0071b90 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 -- 172.21.15.29:0/2451860641 >> 172.21.15.29:0/2451860641 conn(0x7ff8a0065ed0 msgr2=0x7ff8a00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:37.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 -- 172.21.15.29:0/2451860641 shutdown_connections 2023-10-14T16:53:37.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.885+0000 7ff89f59e640 1 -- 172.21.15.29:0/2451860641 wait complete. 2023-10-14T16:53:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.887+0000 7ff89f59e640 1 Processor -- start 2023-10-14T16:53:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.887+0000 7ff89f59e640 1 -- start start 2023-10-14T16:53:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8a00720d0 0x7ff8a014b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8a015cba0 0x7ff8a014e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a01f21c0 con 0x7ff8a00cdd40 2023-10-14T16:53:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a014ca80 con 0x7ff8a00720d0 2023-10-14T16:53:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a014b5d0 con 0x7ff8a015cba0 2023-10-14T16:53:37.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42004/0 (socket says 172.21.15.29:42004) 2023-10-14T16:53:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.888+0000 7ff89dd9b640 1 -- 172.21.15.29:0/185743953 learned_addr learned my addr 172.21.15.29:0/185743953 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89ed9d640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8a015cba0 0x7ff8a014e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89e59c640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8a00720d0 0x7ff8a014b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 -- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8a015cba0 msgr2=0x7ff8a014e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8a015cba0 0x7ff8a014e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 -- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8a00720d0 msgr2=0x7ff8a014b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8a00720d0 0x7ff8a014b890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 -- 172.21.15.29:0/185743953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff890009b30 con 0x7ff8a00cdd40 2023-10-14T16:53:37.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.889+0000 7ff89dd9b640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d800 tx=0x7ff88800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.890+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88800b6c0 con 0x7ff8a00cdd40 2023-10-14T16:53:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.890+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff88800b820 con 0x7ff8a00cdd40 2023-10-14T16:53:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.890+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a014e820 con 0x7ff8a00cdd40 2023-10-14T16:53:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.890+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88800bbb0 con 0x7ff8a00cdd40 2023-10-14T16:53:37.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.890+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a014c150 con 0x7ff8a00cdd40 2023-10-14T16:53:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.891+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a014cc10 con 0x7ff8a00cdd40 2023-10-14T16:53:37.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.892+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7ff88800bd10 con 0x7ff8a00cdd40 2023-10-14T16:53:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.893+0000 7ff896ffd640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 0x7ff88006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.893+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7ff88808a0c0 con 0x7ff8a00cdd40 2023-10-14T16:53:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.893+0000 7ff89e59c640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 0x7ff88006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:37.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.894+0000 7ff89e59c640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 0x7ff88006d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff890035000 tx=0x7ff890030000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:37.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:37.897+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff888054d50 con 0x7ff8a00cdd40 2023-10-14T16:53:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.118+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir2", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8a00d1060 con 0x7ff88006b070 2023-10-14T16:53:38.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.362+0000 7ff896ffd640 1 -- 172.21.15.29:0/185743953 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7ff8900440c0 con 0x7ff88006b070 2023-10-14T16:53:38.364 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir2 2023-10-14T16:53:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 msgr2=0x7ff88006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 0x7ff88006d530 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff890035000 tx=0x7ff890030000 comp rx=0 tx=0).stop 2023-10-14T16:53:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 msgr2=0x7ff8a01f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:38.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d800 tx=0x7ff88800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:53:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 shutdown_connections 2023-10-14T16:53:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8a015cba0 0x7ff8a014e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8a00cdd40 0x7ff8a01f1c80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7ff88006b070 0x7ff88006d530 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:38.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 --2- 172.21.15.29:0/185743953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8a00720d0 0x7ff8a014b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 >> 172.21.15.29:0/185743953 conn(0x7ff8a0065ed0 msgr2=0x7ff8a00b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.365+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 shutdown_connections 2023-10-14T16:53:38.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:38.366+0000 7ff89f59e640 1 -- 172.21.15.29:0/185743953 wait complete. 2023-10-14T16:53:38.389 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir2 2023-10-14T16:53:38.389 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.390 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir2/.snap) 2023-10-14T16:53:38.449 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:38.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:38.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:38.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:38.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:38.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:38.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:38.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:38.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:38.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:38.451 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:38.452 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:38.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.453 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_43_01_UTC) 2023-10-14T16:53:38.527 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:38.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.528 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:38.605 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:38.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.606 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_45_00_UTC) 2023-10-14T16:53:38.683 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:38.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.684 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:38.761 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:38.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.762 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:38.838 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:38.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.839 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:38.908 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:38.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:38.983 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:38.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:38.984 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_50_01_UTC) 2023-10-14T16:53:39.060 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:39.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:39.061 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_51_01_UTC) 2023-10-14T16:53:39.135 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:39.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:39.136 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_52_00_UTC) 2023-10-14T16:53:39.207 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:39.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:39.208 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2/.snap/scheduled-2023-10-14-16_53_00_UTC) 2023-10-14T16:53:39.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:39.282 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir2) 2023-10-14T16:53:39.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir3 1m 2023-10-14T16:53:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.578+0000 7f5663d90640 1 Processor -- start 2023-10-14T16:53:39.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.579+0000 7f5663d90640 1 -- start start 2023-10-14T16:53:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.580+0000 7f5663d90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f565c15af80 0x7f565c0cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.580+0000 7f5663d90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c065f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.580+0000 7f5663d90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c15eff0 con 0x7f565c06faf0 2023-10-14T16:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.580+0000 7f5663d90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c066480 con 0x7f565c15af80 2023-10-14T16:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.580+0000 7f5663d90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0d0930 con 0x7f565c065ab0 2023-10-14T16:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f565c15af80 0x7f565c0cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:39.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f565c15af80 0x7f565c0cb1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42012/0 (socket says 172.21.15.29:42012) 2023-10-14T16:53:39.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 -- 172.21.15.29:0/1609337340 learned_addr learned my addr 172.21.15.29:0/1609337340 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 -- 172.21.15.29:0/1609337340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 msgr2=0x7f565c065f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 --2- 172.21.15.29:0/1609337340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c065f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:39.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 -- 172.21.15.29:0/1609337340 >> v1:172.21.15.186:6789/0 conn(0x7f565c06faf0 legacy=0x7f565c15f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.581+0000 7f5661b2c640 1 -- 172.21.15.29:0/1609337340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c15ecc0 con 0x7f565c15af80 2023-10-14T16:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.582+0000 7f5661b2c640 1 --2- 172.21.15.29:0/1609337340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c0cb1c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5658009e80 tx=0x7f565802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c492e701feb026b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.583+0000 7f5660b2a640 1 -- 172.21.15.29:0/1609337340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f565802c9c0 con 0x7f565c15af80 2023-10-14T16:53:39.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.583+0000 7f5660b2a640 1 -- 172.21.15.29:0/1609337340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f565802cb20 con 0x7f565c15af80 2023-10-14T16:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.583+0000 7f5660b2a640 1 -- 172.21.15.29:0/1609337340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56580343f0 con 0x7f565c15af80 2023-10-14T16:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.583+0000 7f5663d90640 1 -- 172.21.15.29:0/1609337340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 msgr2=0x7f565c0cb1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.583+0000 7f5663d90640 1 --2- 172.21.15.29:0/1609337340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c0cb1c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f5658009e80 tx=0x7f565802be20 comp rx=0 tx=0).stop 2023-10-14T16:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 -- 172.21.15.29:0/1609337340 shutdown_connections 2023-10-14T16:53:39.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 --2- 172.21.15.29:0/1609337340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c065f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 --2- 172.21.15.29:0/1609337340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c0cb1c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 -- 172.21.15.29:0/1609337340 >> 172.21.15.29:0/1609337340 conn(0x7f565c0d0ab0 msgr2=0x7f565c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 -- 172.21.15.29:0/1609337340 shutdown_connections 2023-10-14T16:53:39.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.584+0000 7f5663d90640 1 -- 172.21.15.29:0/1609337340 wait complete. 2023-10-14T16:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.585+0000 7f5663d90640 1 Processor -- start 2023-10-14T16:53:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.586+0000 7f5663d90640 1 -- start start 2023-10-14T16:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c0c4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f565c06faf0 0x7f565c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c5a60 con 0x7f565c15af80 2023-10-14T16:53:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c5d40 con 0x7f565c06faf0 2023-10-14T16:53:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5663d90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0c6020 con 0x7f565c065ab0 2023-10-14T16:53:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42022/0 (socket says 172.21.15.29:42022) 2023-10-14T16:53:39.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 -- 172.21.15.29:0/3817293494 learned_addr learned my addr 172.21.15.29:0/3817293494 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566132b640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f565c06faf0 0x7f565c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f5661b2c640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c0c4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 -- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 msgr2=0x7f565c0c4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c0c4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:39.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 -- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f565c06faf0 msgr2=0x7f565c1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f565c06faf0 0x7f565c1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.587+0000 7f566232d640 1 -- 172.21.15.29:0/3817293494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5658009b30 con 0x7f565c15af80 2023-10-14T16:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f5661b2c640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c0c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f566232d640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f565400b410 tx=0x7f565400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:39.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654011830 con 0x7f565c15af80 2023-10-14T16:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56540119b0 con 0x7f565c15af80 2023-10-14T16:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c0c6300 con 0x7f565c15af80 2023-10-14T16:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5654011d60 con 0x7f565c15af80 2023-10-14T16:53:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.588+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c0c54a0 con 0x7f565c15af80 2023-10-14T16:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.590+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c0c6820 con 0x7f565c15af80 2023-10-14T16:53:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.591+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f565401a400 con 0x7f565c15af80 2023-10-14T16:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.592+0000 7f56527fc640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.592+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5967+0+0 (secure 0 0 0) 0x7f5654089c50 con 0x7f565c15af80 2023-10-14T16:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.592+0000 7f5661b2c640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.593+0000 7f5661b2c640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5658035000 tx=0x7f5658030000 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.597+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f565c0c6820 con 0x7f565c15af80 2023-10-14T16:53:39.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:39.823+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir3", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f565c0d0370 con 0x7f564006b070 2023-10-14T16:53:40.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.070+0000 7f56527fc640 1 -- 172.21.15.29:0/3817293494 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f5658009510 con 0x7f564006b070 2023-10-14T16:53:40.072 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir3 2023-10-14T16:53:40.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 msgr2=0x7f564006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:40.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5658035000 tx=0x7f5658030000 comp rx=0 tx=0).stop 2023-10-14T16:53:40.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 msgr2=0x7f565c1f30a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:40.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f565400b410 tx=0x7f565400b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:40.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 shutdown_connections 2023-10-14T16:53:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f565c15af80 0x7f565c1f30a0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f564006b070 0x7f564006d530 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f565c06faf0 0x7f565c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:40.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 --2- 172.21.15.29:0/3817293494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f565c065ab0 0x7f565c0c4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.075+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 >> 172.21.15.29:0/3817293494 conn(0x7f565c0d0ab0 msgr2=0x7f565c15d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.076+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 shutdown_connections 2023-10-14T16:53:40.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:40.076+0000 7f5663d90640 1 -- 172.21.15.29:0/3817293494 wait complete. 2023-10-14T16:53:40.107 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir3 2023-10-14T16:53:40.107 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.108 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir3/.snap) 2023-10-14T16:53:40.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:40.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:40.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:40.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:40.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:40.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:40.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:40.182 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:40.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.182 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:40.255 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:40.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.256 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_44_01_UTC) 2023-10-14T16:53:40.342 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:40.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.343 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_45_00_UTC) 2023-10-14T16:53:40.412 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:40.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.413 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:40.464 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:40.465 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.465 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:40.518 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:40.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.519 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:40.575 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:40.575 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.576 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_49_01_UTC) 2023-10-14T16:53:40.644 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:40.644 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.645 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_50_01_UTC) 2023-10-14T16:53:40.718 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:40.719 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.719 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:40.812 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:40.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.813 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_52_00_UTC) 2023-10-14T16:53:40.881 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:40.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.882 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3/.snap/scheduled-2023-10-14-16_53_00_UTC) 2023-10-14T16:53:40.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:40.966 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir3) 2023-10-14T16:53:41.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir4 1m 2023-10-14T16:53:41.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.213+0000 7f0281161640 1 Processor -- start 2023-10-14T16:53:41.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.214+0000 7f0281161640 1 -- start start 2023-10-14T16:53:41.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0281161640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 0x7f027c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0281161640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f027c0c9ca0 0x7f027c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0281161640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f027c0cb030 con 0x7f027c0ca3b0 2023-10-14T16:53:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0281161640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c0c91c0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0281161640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f027c0c94a0 con 0x7f027c15ad70 2023-10-14T16:53:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0273fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f027c0c9ca0 0x7f027c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0273fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f027c0c9ca0 0x7f027c06a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57202/0 (socket says 172.21.15.29:57202) 2023-10-14T16:53:41.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.215+0000 7f0273fff640 1 -- 172.21.15.29:0/2636484165 learned_addr learned my addr 172.21.15.29:0/2636484165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f027ad9d640 1 --2- 172.21.15.29:0/2636484165 >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 0x7f027c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f0273fff640 1 -- 172.21.15.29:0/2636484165 >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 msgr2=0x7f027c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f0273fff640 1 --2- 172.21.15.29:0/2636484165 >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 0x7f027c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f0273fff640 1 -- 172.21.15.29:0/2636484165 >> v1:172.21.15.186:6789/0 conn(0x7f027c0ca3b0 legacy=0x7f027c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f0273fff640 1 -- 172.21.15.29:0/2636484165 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f027c0c9780 con 0x7f027c0c9ca0 2023-10-14T16:53:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.216+0000 7f027ad9d640 1 --2- 172.21.15.29:0/2636484165 >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 0x7f027c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:41.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.217+0000 7f0273fff640 1 --2- 172.21.15.29:0/2636484165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c06a2e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cf780 tx=0x7f026402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a306aa0c08f3ffe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.217+0000 7f027a59c640 1 -- 172.21.15.29:0/2636484165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264031870 con 0x7f027c0c9ca0 2023-10-14T16:53:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.217+0000 7f027a59c640 1 -- 172.21.15.29:0/2636484165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02640319d0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.217+0000 7f027a59c640 1 -- 172.21.15.29:0/2636484165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0264031d60 con 0x7f027c0c9ca0 2023-10-14T16:53:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 -- 172.21.15.29:0/2636484165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 msgr2=0x7f027c06a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:41.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 --2- 172.21.15.29:0/2636484165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c06a2e0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f027c0cf780 tx=0x7f026402b910 comp rx=0 tx=0).stop 2023-10-14T16:53:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 -- 172.21.15.29:0/2636484165 shutdown_connections 2023-10-14T16:53:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 --2- 172.21.15.29:0/2636484165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c06a2e0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 --2- 172.21.15.29:0/2636484165 >> v2:172.21.15.186:3300/0 conn(0x7f027c15ad70 0x7f027c069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 -- 172.21.15.29:0/2636484165 >> 172.21.15.29:0/2636484165 conn(0x7f027c0ce9b0 msgr2=0x7f027c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 -- 172.21.15.29:0/2636484165 shutdown_connections 2023-10-14T16:53:41.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.218+0000 7f0281161640 1 -- 172.21.15.29:0/2636484165 wait complete. 2023-10-14T16:53:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.220+0000 7f0281161640 1 Processor -- start 2023-10-14T16:53:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.220+0000 7f0281161640 1 -- start start 2023-10-14T16:53:41.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f027c0ca3b0 0x7f027c0c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c15ad70 0x7f027c15d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c1f2580 con 0x7f027c0c9ca0 2023-10-14T16:53:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0c6e70 con 0x7f027c15ad70 2023-10-14T16:53:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f0281161640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f027c0c7150 con 0x7f027c0ca3b0 2023-10-14T16:53:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f027ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57206/0 (socket says 172.21.15.29:57206) 2023-10-14T16:53:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.221+0000 7f027ad9d640 1 -- 172.21.15.29:0/2311704235 learned_addr learned my addr 172.21.15.29:0/2311704235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f0273fff640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f027c0ca3b0 0x7f027c0c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027b59e640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c15ad70 0x7f027c15d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 -- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f027c0ca3b0 msgr2=0x7f027c0c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f027c0ca3b0 0x7f027c0c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 -- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c15ad70 msgr2=0x7f027c15d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c15ad70 0x7f027c15d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 -- 172.21.15.29:0/2311704235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02640094d0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f027ad9d640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c7f0 tx=0x7f026c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c007820 con 0x7f027c0c9ca0 2023-10-14T16:53:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f026c007980 con 0x7f027c0c9ca0 2023-10-14T16:53:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.222+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f026c007cf0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.223+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c0c7430 con 0x7f027c0c9ca0 2023-10-14T16:53:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.223+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c0c49f0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.224+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c04e160 con 0x7f027c0c9ca0 2023-10-14T16:53:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.225+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f026c0113f0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.226+0000 7f02737fe640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 0x7f025c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.226+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 6455+0+0 (secure 0 0 0) 0x7f026c08a140 con 0x7f027c0c9ca0 2023-10-14T16:53:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.226+0000 7f0273fff640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 0x7f025c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:41.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.227+0000 7f0273fff640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 0x7f025c06d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f026403c040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:41.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.231+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f026c054ae0 con 0x7f027c0c9ca0 2023-10-14T16:53:41.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.456+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir4", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f027c0ad4d0 con 0x7f025c06b070 2023-10-14T16:53:41.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.712+0000 7f02737fe640 1 -- 172.21.15.29:0/2311704235 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f0264031810 con 0x7f025c06b070 2023-10-14T16:53:41.714 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir4 2023-10-14T16:53:41.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.715+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 msgr2=0x7f025c06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.715+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 0x7f025c06d530 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0264036000 tx=0x7f026403c040 comp rx=0 tx=0).stop 2023-10-14T16:53:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.715+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 msgr2=0x7f027c0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.715+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f026c00c7f0 tx=0x7f026c00ccb0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 shutdown_connections 2023-10-14T16:53:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f025c06b070 0x7f025c06d530 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f027c15ad70 0x7f027c15d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f027c0ca3b0 0x7f027c0c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 --2- 172.21.15.29:0/2311704235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f027c0c9ca0 0x7f027c0c44b0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:41.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 >> 172.21.15.29:0/2311704235 conn(0x7f027c0ce9b0 msgr2=0x7f027c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 shutdown_connections 2023-10-14T16:53:41.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:41.716+0000 7f0281161640 1 -- 172.21.15.29:0/2311704235 wait complete. 2023-10-14T16:53:41.739 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir4 2023-10-14T16:53:41.740 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:41.740 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir4/.snap) 2023-10-14T16:53:41.798 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:41.798 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:41.799 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:41.799 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:41.799 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:41.799 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:41.799 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:41.800 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:41.800 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:41.800 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:41.800 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:41.801 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:41.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:41.802 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:41.879 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:41.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:41.880 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:41.949 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:41.950 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:41.950 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_45_00_UTC) 2023-10-14T16:53:42.021 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:42.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.022 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_46_01_UTC) 2023-10-14T16:53:42.125 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:42.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.126 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:42.211 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:42.212 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.212 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:42.267 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:42.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.268 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_49_01_UTC) 2023-10-14T16:53:42.316 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:42.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.317 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_50_00_UTC) 2023-10-14T16:53:42.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:42.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.372 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:42.426 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:42.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.427 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_52_01_UTC) 2023-10-14T16:53:42.485 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:42.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.486 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4/.snap/scheduled-2023-10-14-16_53_01_UTC) 2023-10-14T16:53:42.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:42.548 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir4) 2023-10-14T16:53:42.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir5 1m 2023-10-14T16:53:42.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.837+0000 7f703cfb0640 1 Processor -- start 2023-10-14T16:53:42.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.837+0000 7f703cfb0640 1 -- start start 2023-10-14T16:53:42.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.838+0000 7f703cfb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 0x7f7038069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.838+0000 7f703cfb0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70380caac0 0x7f703806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.838+0000 7f703cfb0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70380cb030 con 0x7f70380c9ca0 2023-10-14T16:53:42.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.838+0000 7f703cfb0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70380c91c0 con 0x7f70380caac0 2023-10-14T16:53:42.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.838+0000 7f703cfb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c94a0 con 0x7f703815ad70 2023-10-14T16:53:42.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.839+0000 7f7035d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70380c9ca0 0x7f70380ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53274/0 (socket says 172.21.15.29:53274) 2023-10-14T16:53:42.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.839+0000 7f7035d9b640 1 -- 172.21.15.29:0/408665038 learned_addr learned my addr 172.21.15.29:0/408665038 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:42.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.839+0000 7f703659c640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 0x7f7038069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.839+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3878043242 0 0) 0x7f70380cb030 con 0x7f70380c9ca0 2023-10-14T16:53:42.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7020003520 con 0x7f70380c9ca0 2023-10-14T16:53:42.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f7036d9d640 1 --2- 172.21.15.29:0/408665038 >> v2:172.21.15.186:3300/0 conn(0x7f70380caac0 0x7f703806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2201733863 0 0) 0x7f7020003520 con 0x7f70380c9ca0 2023-10-14T16:53:42.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 msgr2=0x7f7038069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:42.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 0x7f7038069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 >> v2:172.21.15.186:3300/0 conn(0x7f70380caac0 msgr2=0x7f703806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 --2- 172.21.15.29:0/408665038 >> v2:172.21.15.186:3300/0 conn(0x7f70380caac0 0x7f703806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380c9780 con 0x7f70380c9ca0 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f702c001670 con 0x7f70380c9ca0 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 >> v1:172.21.15.29:6789/0 conn(0x7f70380c9ca0 legacy=0x7f70380ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:42.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7020003970 0x7f7020003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 0x7f70200047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f7020005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c91c0 con 0x7f7020004d00 2023-10-14T16:53:42.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c94a0 con 0x7f7020003970 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f702000b980 con 0x7f703815ad70 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.840+0000 7f7036d9d640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f7020005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.841+0000 7f7036d9d640 1 -- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 msgr2=0x7f70200047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.841+0000 7f7036d9d640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f703815ad70 0x7f70200047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.841+0000 7f7036d9d640 1 -- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7020003970 msgr2=0x7f7020003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:42.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.841+0000 7f7036d9d640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7020003970 0x7f7020003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.841+0000 7f7036d9d640 1 -- 172.21.15.29:0/408665038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7020003520 con 0x7f7020004d00 2023-10-14T16:53:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.842+0000 7f7036d9d640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f7020005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7030007390 tx=0x7f70200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28db71a01a946575 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f703002ccf0 con 0x7f7020004d00 2023-10-14T16:53:42.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f703002ce50 con 0x7f7020004d00 2023-10-14T16:53:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f703559a640 1 -- 172.21.15.29:0/408665038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7030034660 con 0x7f7020004d00 2023-10-14T16:53:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f703cfb0640 1 -- 172.21.15.29:0/408665038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 msgr2=0x7f7020005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f703cfb0640 1 --2- 172.21.15.29:0/408665038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f7020005180 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f7030007390 tx=0x7f70200057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.843+0000 7f7036d9d640 1 -- 172.21.15.29:0/408665038 reap_dead start 2023-10-14T16:53:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.844+0000 7f703cfb0640 1 -- 172.21.15.29:0/408665038 shutdown_connections 2023-10-14T16:53:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.844+0000 7f703cfb0640 1 -- 172.21.15.29:0/408665038 >> 172.21.15.29:0/408665038 conn(0x7f70380ce9b0 msgr2=0x7f70380cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.844+0000 7f703cfb0640 1 -- 172.21.15.29:0/408665038 shutdown_connections 2023-10-14T16:53:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.844+0000 7f703cfb0640 1 -- 172.21.15.29:0/408665038 wait complete. 2023-10-14T16:53:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.846+0000 7f703cfb0640 1 Processor -- start 2023-10-14T16:53:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.846+0000 7f703cfb0640 1 -- start start 2023-10-14T16:53:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70380c24c0 0x7f70381f2050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 0x7f70381f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c2950 con 0x7f7020004d00 2023-10-14T16:53:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c2c30 con 0x7f70381f6220 2023-10-14T16:53:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.847+0000 7f703cfb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70380c2f10 con 0x7f70380c24c0 2023-10-14T16:53:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.848+0000 7f703659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.848+0000 7f703659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57230/0 (socket says 172.21.15.29:57230) 2023-10-14T16:53:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.848+0000 7f703659c640 1 -- 172.21.15.29:0/898932889 learned_addr learned my addr 172.21.15.29:0/898932889 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.848+0000 7f7035d9b640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70380c24c0 0x7f70381f2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.848+0000 7f7036d9d640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 0x7f70381f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 -- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70380c24c0 msgr2=0x7f70381f2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70380c24c0 0x7f70381f2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 -- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 msgr2=0x7f70381f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 0x7f70381f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:42.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 -- 172.21.15.29:0/898932889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70380c31f0 con 0x7f7020004d00 2023-10-14T16:53:42.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f7036d9d640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 0x7f70381f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703659c640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f702800e7f0 tx=0x7f702800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702800eea0 con 0x7f7020004d00 2023-10-14T16:53:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702800c940 con 0x7f7020004d00 2023-10-14T16:53:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70380c34d0 con 0x7f7020004d00 2023-10-14T16:53:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.849+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f702800cd10 con 0x7f7020004d00 2023-10-14T16:53:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.850+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703815d050 con 0x7f7020004d00 2023-10-14T16:53:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.851+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f703804e160 con 0x7f7020004d00 2023-10-14T16:53:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.852+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f7028013470 con 0x7f7020004d00 2023-10-14T16:53:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.853+0000 7f7026ffd640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 0x7f701806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.854+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(38..38 src has 1..38) v4 ==== 6663+0+0 (secure 0 0 0) 0x7f7028089d50 con 0x7f7020004d00 2023-10-14T16:53:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.854+0000 7f7035d9b640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 0x7f701806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.855+0000 7f7035d9b640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 0x7f701806d530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7038159200 tx=0x7f702c001b30 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:42.859+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7028054850 con 0x7f7020004d00 2023-10-14T16:53:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.095+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir5", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f7038069da0 con 0x7f701806b070 2023-10-14T16:53:43.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.290+0000 7f7026ffd640 1 -- 172.21.15.29:0/898932889 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f702c004900 con 0x7f701806b070 2023-10-14T16:53:43.292 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir5 2023-10-14T16:53:43.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.293+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 msgr2=0x7f701806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:43.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.293+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 0x7f701806d530 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f7038159200 tx=0x7f702c001b30 comp rx=0 tx=0).stop 2023-10-14T16:53:43.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.293+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 msgr2=0x7f70380afdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:43.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.293+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f702800e7f0 tx=0x7f702800ecb0 comp rx=0 tx=0).stop 2023-10-14T16:53:43.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 shutdown_connections 2023-10-14T16:53:43.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f701806b070 0x7f701806d530 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:43.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70381f6220 0x7f70381f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:43.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70380c24c0 0x7f70381f2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:43.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 --2- 172.21.15.29:0/898932889 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7020004d00 0x7f70380afdf0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:43.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 >> 172.21.15.29:0/898932889 conn(0x7f70380ce9b0 msgr2=0x7f70380b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:43.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 shutdown_connections 2023-10-14T16:53:43.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:43.294+0000 7f703cfb0640 1 -- 172.21.15.29:0/898932889 wait complete. 2023-10-14T16:53:43.316 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir5 2023-10-14T16:53:43.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.317 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir5/.snap) 2023-10-14T16:53:43.372 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:43.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:43.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:43.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:43.373 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:43.374 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:43.374 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:43.374 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:43.374 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:43.374 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:43.375 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:43.376 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:43.376 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.376 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_43_01_UTC) 2023-10-14T16:53:43.444 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:43.445 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.445 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:43.514 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:43.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.515 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_45_00_UTC) 2023-10-14T16:53:43.582 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:43.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.583 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:43.651 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:43.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.652 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:43.704 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:43.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.705 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:43.764 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:43.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.765 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:43.832 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:43.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.833 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_50_00_UTC) 2023-10-14T16:53:43.900 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:43.900 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.901 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_51_01_UTC) 2023-10-14T16:53:43.959 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:43.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:43.960 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_52_00_UTC) 2023-10-14T16:53:44.018 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:44.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:44.018 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5/.snap/scheduled-2023-10-14-16_53_01_UTC) 2023-10-14T16:53:44.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:44.092 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir5) 2023-10-14T16:53:44.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir6 1m 2023-10-14T16:53:44.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.374+0000 7fea03f5a640 1 Processor -- start 2023-10-14T16:53:44.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.374+0000 7fea03f5a640 1 -- start start 2023-10-14T16:53:44.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe9fc0cf710 0x7fe9fc0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 0x7fe9fc0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9fc0d3c60 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9fc0d3980 con 0x7fe9fc0cf710 2023-10-14T16:53:44.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea03f5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0ce630 con 0x7fe9fc0b38e0 2023-10-14T16:53:44.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea014f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea014f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57234/0 (socket says 172.21.15.29:57234) 2023-10-14T16:53:44.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea014f5640 1 -- 172.21.15.29:0/2132056431 learned_addr learned my addr 172.21.15.29:0/2132056431 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:44.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.375+0000 7fea01cf6640 1 --2- 172.21.15.29:0/2132056431 >> v2:172.21.15.186:3300/0 conn(0x7fe9fc0cf710 0x7fe9fc0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea024f7640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 0x7fe9fc0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea014f5640 1 -- 172.21.15.29:0/2132056431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 msgr2=0x7fe9fc0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:44.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea014f5640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 0x7fe9fc0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea014f5640 1 -- 172.21.15.29:0/2132056431 >> v2:172.21.15.186:3300/0 conn(0x7fe9fc0cf710 msgr2=0x7fe9fc0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea014f5640 1 --2- 172.21.15.29:0/2132056431 >> v2:172.21.15.186:3300/0 conn(0x7fe9fc0cf710 0x7fe9fc0b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea014f5640 1 -- 172.21.15.29:0/2132056431 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9fc0ce910 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.376+0000 7fea024f7640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 0x7fe9fc0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.377+0000 7fea014f5640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0d2c20 tx=0x7fe9f002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb8f0c550abfee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:44.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.377+0000 7fea00cf4640 1 -- 172.21.15.29:0/2132056431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f00317e0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.377+0000 7fea00cf4640 1 -- 172.21.15.29:0/2132056431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f0031940 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.377+0000 7fea00cf4640 1 -- 172.21.15.29:0/2132056431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f0031cd0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 -- 172.21.15.29:0/2132056431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 msgr2=0x7fe9fc0b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7fe9fc0d2c20 tx=0x7fe9f002b880 comp rx=0 tx=0).stop 2023-10-14T16:53:44.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 -- 172.21.15.29:0/2132056431 shutdown_connections 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0b38e0 0x7fe9fc0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2132056431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0b3250 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2132056431 >> v2:172.21.15.186:3300/0 conn(0x7fe9fc0cf710 0x7fe9fc0b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 -- 172.21.15.29:0/2132056431 >> 172.21.15.29:0/2132056431 conn(0x7fe9fc0d2cc0 msgr2=0x7fe9fc0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 -- 172.21.15.29:0/2132056431 shutdown_connections 2023-10-14T16:53:44.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.378+0000 7fea03f5a640 1 -- 172.21.15.29:0/2132056431 wait complete. 2023-10-14T16:53:44.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.380+0000 7fea03f5a640 1 Processor -- start 2023-10-14T16:53:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.380+0000 7fea03f5a640 1 -- start start 2023-10-14T16:53:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9fc0b38e0 0x7fe9fc151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0cf710 0x7fe9fc14dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0ca760 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc1522d0 con 0x7fe9fc0b38e0 2023-10-14T16:53:44.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea03f5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9fc0c9e30 con 0x7fe9fc0cf710 2023-10-14T16:53:44.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57246/0 (socket says 172.21.15.29:57246) 2023-10-14T16:53:44.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 -- 172.21.15.29:0/2613340854 learned_addr learned my addr 172.21.15.29:0/2613340854 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea01cf6640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9fc0b38e0 0x7fe9fc151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 -- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0cf710 msgr2=0x7fe9fc14dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0cf710 0x7fe9fc14dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.381+0000 7fea024f7640 1 -- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9fc0b38e0 msgr2=0x7fe9fc151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:44.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fea024f7640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9fc0b38e0 0x7fe9fc151d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fea024f7640 1 -- 172.21.15.29:0/2613340854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9f00094d0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fea024f7640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400cf20 tx=0x7fe9f400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400d840 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9f400d9a0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9f400dd50 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9fc0ca110 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.382+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9fc154bd0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.384+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9fc0ca910 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.384+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fe9f401e3b0 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.385+0000 7fe9fa7fc640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 0x7fe9e806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:44.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.386+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6631+0+0 (secure 0 0 0) 0x7fe9f408a800 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.386+0000 7fea01cf6640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 0x7fe9e806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:44.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.387+0000 7fea01cf6640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 0x7fe9e806d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009770 tx=0x7fe9ec008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:44.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.390+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9fc0ca910 con 0x7fe9fc0cfaf0 2023-10-14T16:53:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.615+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir6", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9fc0ca910 con 0x7fe9e806b070 2023-10-14T16:53:44.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:53:44.768 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:53:44.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.815+0000 7fe9fa7fc640 1 -- 172.21.15.29:0/2613340854 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fe9fc0ca910 con 0x7fe9e806b070 2023-10-14T16:53:44.817 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir6 2023-10-14T16:53:44.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 msgr2=0x7fe9e806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:44.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 0x7fe9e806d530 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fe9ec009770 tx=0x7fe9ec008040 comp rx=0 tx=0).stop 2023-10-14T16:53:44.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 msgr2=0x7fe9fc0cc390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:44.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fe9f400cf20 tx=0x7fe9f400a710 comp rx=0 tx=0).stop 2023-10-14T16:53:44.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 shutdown_connections 2023-10-14T16:53:44.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9fc0cfaf0 0x7fe9fc0cc390 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9fc0cf710 0x7fe9fc14dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fe9e806b070 0x7fe9e806d530 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 --2- 172.21.15.29:0/2613340854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9fc0b38e0 0x7fe9fc151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:44.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 >> 172.21.15.29:0/2613340854 conn(0x7fe9fc0d2cc0 msgr2=0x7fe9fc0b5bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:44.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.818+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 shutdown_connections 2023-10-14T16:53:44.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:44.819+0000 7fea03f5a640 1 -- 172.21.15.29:0/2613340854 wait complete. 2023-10-14T16:53:44.839 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir6 2023-10-14T16:53:44.839 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:44.840 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir6/.snap) 2023-10-14T16:53:44.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:44.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:44.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:44.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:44.888 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:44.889 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:44.889 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:44.890 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:44.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:44.891 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:44.944 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:44.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:44.945 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:45.059 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_45_00_UTC 2023-10-14T16:53:45.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.060 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_45_00_UTC) 2023-10-14T16:53:45.144 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:45.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.145 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:45.198 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:45.198 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.198 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:45.248 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:45.248 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.249 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_48_01_UTC) 2023-10-14T16:53:45.301 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:45.301 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.302 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_49_01_UTC) 2023-10-14T16:53:45.359 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:45.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.360 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_50_01_UTC) 2023-10-14T16:53:45.430 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_51_01_UTC 2023-10-14T16:53:45.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.431 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_51_01_UTC) 2023-10-14T16:53:45.517 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:45.517 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.518 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_52_00_UTC) 2023-10-14T16:53:45.567 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:45.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.568 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6/.snap/scheduled-2023-10-14-16_53_01_UTC) 2023-10-14T16:53:45.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:45.617 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir6) 2023-10-14T16:53:45.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir7 1m 2023-10-14T16:53:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.936+0000 7fbbcfd2c640 1 Processor -- start 2023-10-14T16:53:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.936+0000 7fbbcfd2c640 1 -- start start 2023-10-14T16:53:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.938+0000 7fbbcfd2c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.938+0000 7fbbcfd2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbbc8160b90 0x7fbbc8160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.938+0000 7fbbcfd2c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc80ac890 con 0x7fbbc8069da0 2023-10-14T16:53:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.938+0000 7fbbcfd2c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc815d530 con 0x7fbbc8160b90 2023-10-14T16:53:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.938+0000 7fbbcfd2c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8161650 con 0x7fbbc815d150 2023-10-14T16:53:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbbc8160b90 0x7fbbc8160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcdac8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbbc8160b90 0x7fbbc8160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:37480/0 (socket says 172.21.15.29:37480) 2023-10-14T16:53:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 -- 172.21.15.29:0/1675055133 learned_addr learned my addr 172.21.15.29:0/1675055133 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 -- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 msgr2=0x7fbbc80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 --2- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc80ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 -- 172.21.15.29:0/1675055133 >> v1:172.21.15.186:6789/0 conn(0x7fbbc8069da0 legacy=0x7fbbc806a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:45.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.939+0000 7fbbcd2c7640 1 -- 172.21.15.29:0/1675055133 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc8161930 con 0x7fbbc8160b90 2023-10-14T16:53:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.941+0000 7fbbcd2c7640 1 --2- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8160b90 0x7fbbc8160fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009870 tx=0x7fbbb802b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60d33deb95c11423 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbccac6640 1 -- 172.21.15.29:0/1675055133 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8030840 con 0x7fbbc8160b90 2023-10-14T16:53:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbccac6640 1 -- 172.21.15.29:0/1675055133 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbb80309a0 con 0x7fbbc8160b90 2023-10-14T16:53:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbccac6640 1 -- 172.21.15.29:0/1675055133 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8030d30 con 0x7fbbc8160b90 2023-10-14T16:53:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8160b90 msgr2=0x7fbbc8160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8160b90 0x7fbbc8160fb0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbbb8009870 tx=0x7fbbb802b980 comp rx=0 tx=0).stop 2023-10-14T16:53:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/1675055133 shutdown_connections 2023-10-14T16:53:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8160b90 0x7fbbc8160fb0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/1675055133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc80ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.942+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/1675055133 >> 172.21.15.29:0/1675055133 conn(0x7fbbc80ce9b0 msgr2=0x7fbbc80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.943+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/1675055133 shutdown_connections 2023-10-14T16:53:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.943+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/1675055133 wait complete. 2023-10-14T16:53:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.944+0000 7fbbcfd2c640 1 Processor -- start 2023-10-14T16:53:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.944+0000 7fbbcfd2c640 1 -- start start 2023-10-14T16:53:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 0x7fbbc81e82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc81e69b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81e7200 con 0x7fbbc8155bb0 2023-10-14T16:53:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81e74e0 con 0x7fbbc8069da0 2023-10-14T16:53:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.945+0000 7fbbcfd2c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81e77c0 con 0x7fbbc815d150 2023-10-14T16:53:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57260/0 (socket says 172.21.15.29:57260) 2023-10-14T16:53:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 -- 172.21.15.29:0/2048052132 learned_addr learned my addr 172.21.15.29:0/2048052132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbcdac8640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 0x7fbbc81e82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbcd2c7640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc81e69b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 -- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 msgr2=0x7fbbc81e69b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc81e69b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 -- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 msgr2=0x7fbbc81e82a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 0x7fbbc81e82a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbce2c9640 1 -- 172.21.15.29:0/2048052132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbb80094d0 con 0x7fbbc8155bb0 2023-10-14T16:53:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbcd2c7640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc81e69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.946+0000 7fbbcdac8640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 0x7fbbc81e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbce2c9640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0001eb0 tx=0x7fbbc000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc00052d0 con 0x7fbbc8155bb0 2023-10-14T16:53:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbc0005430 con 0x7fbbc8155bb0 2023-10-14T16:53:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc81e7aa0 con 0x7fbbc8155bb0 2023-10-14T16:53:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0003810 con 0x7fbbc8155bb0 2023-10-14T16:53:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.947+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc81e6ef0 con 0x7fbbc8155bb0 2023-10-14T16:53:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.948+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc81e7fc0 con 0x7fbbc8155bb0 2023-10-14T16:53:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.949+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fbbc0003970 con 0x7fbbc8155bb0 2023-10-14T16:53:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.950+0000 7fbbbe7fc640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 0x7fbbac06d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.950+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6663+0+0 (secure 0 0 0) 0x7fbbc0089430 con 0x7fbbc8155bb0 2023-10-14T16:53:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.950+0000 7fbbcdac8640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 0x7fbbac06d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.951+0000 7fbbcdac8640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 0x7fbbac06d560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4007440 tx=0x7fbbc400b040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:45.955+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbbc81e7fc0 con 0x7fbbc8155bb0 2023-10-14T16:53:46.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.183+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir7", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbc81e7650 con 0x7fbbac06b0a0 2023-10-14T16:53:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.406+0000 7fbbbe7fc640 1 -- 172.21.15.29:0/2048052132 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fbbc81e7650 con 0x7fbbac06b0a0 2023-10-14T16:53:46.408 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir7 2023-10-14T16:53:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.410+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 msgr2=0x7fbbac06d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:46.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.410+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 0x7fbbac06d560 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fbbc4007440 tx=0x7fbbc400b040 comp rx=0 tx=0).stop 2023-10-14T16:53:46.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 msgr2=0x7fbbc8155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:46.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fbbc0001eb0 tx=0x7fbbc000c890 comp rx=0 tx=0).stop 2023-10-14T16:53:46.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 shutdown_connections 2023-10-14T16:53:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbbc8155bb0 0x7fbbc8155370 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbbc815d150 0x7fbbc81e69b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fbbac06b0a0 0x7fbbac06d560 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 --2- 172.21.15.29:0/2048052132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbbc8069da0 0x7fbbc81e82a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 >> 172.21.15.29:0/2048052132 conn(0x7fbbc80ce9b0 msgr2=0x7fbbc81569c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 shutdown_connections 2023-10-14T16:53:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:46.411+0000 7fbbcfd2c640 1 -- 172.21.15.29:0/2048052132 wait complete. 2023-10-14T16:53:46.436 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir7 2023-10-14T16:53:46.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.437 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir7/.snap) 2023-10-14T16:53:46.495 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:46.495 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:46.496 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:46.496 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:46.496 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:46.496 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:46.497 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:46.497 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:46.497 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:46.497 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:46.497 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:46.498 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:46.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.499 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:46.567 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:46.567 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.568 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:46.634 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:46.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.635 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_45_01_UTC) 2023-10-14T16:53:46.703 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:46.703 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.704 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:46.772 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_47_00_UTC 2023-10-14T16:53:46.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.773 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_47_00_UTC) 2023-10-14T16:53:46.843 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:46.845 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.845 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_48_01_UTC) 2023-10-14T16:53:46.917 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:46.917 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.918 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:46.987 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:46.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:46.988 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_50_00_UTC) 2023-10-14T16:53:47.063 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:47.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:47.064 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:47.145 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_52_00_UTC 2023-10-14T16:53:47.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:47.146 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_52_00_UTC) 2023-10-14T16:53:47.207 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:47.207 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:47.208 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7/.snap/scheduled-2023-10-14-16_53_00_UTC) 2023-10-14T16:53:47.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:47.281 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir7) 2023-10-14T16:53:47.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir8 1m 2023-10-14T16:53:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.600+0000 7f93c61f0640 1 Processor -- start 2023-10-14T16:53:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.600+0000 7f93c61f0640 1 -- start start 2023-10-14T16:53:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93c61f0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f93c015ad70 0x7f93c0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93c61f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f93c00c9ca0 0x7f93c006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93c61f0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93c00cb030 con 0x7f93c00ca3b0 2023-10-14T16:53:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93c61f0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c00c91c0 con 0x7f93c00c9ca0 2023-10-14T16:53:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93c61f0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93c00c94a0 con 0x7f93c015ad70 2023-10-14T16:53:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93b7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f93c00c9ca0 0x7f93c006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.601+0000 7f93bffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f93c00ca3b0 0x7f93c00ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53288/0 (socket says 172.21.15.29:53288) 2023-10-14T16:53:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93bffff640 1 -- 172.21.15.29:0/2300379212 learned_addr learned my addr 172.21.15.29:0/2300379212 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93bf7fe640 1 --2- 172.21.15.29:0/2300379212 >> v2:172.21.15.186:3300/0 conn(0x7f93c015ad70 0x7f93c0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93beffd640 1 -- 172.21.15.29:0/2300379212 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988958382 0 0) 0x7f93c00cb030 con 0x7f93c00ca3b0 2023-10-14T16:53:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93b7fff640 1 -- 172.21.15.29:0/2300379212 >> v2:172.21.15.186:3300/0 conn(0x7f93c015ad70 msgr2=0x7f93c0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2300379212 >> v2:172.21.15.186:3300/0 conn(0x7f93c015ad70 0x7f93c0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93b7fff640 1 -- 172.21.15.29:0/2300379212 >> v1:172.21.15.29:6789/0 conn(0x7f93c00ca3b0 legacy=0x7f93c00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:47.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.602+0000 7f93b7fff640 1 -- 172.21.15.29:0/2300379212 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93c00c9780 con 0x7f93c00c9ca0 2023-10-14T16:53:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.603+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2300379212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c006a2e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93c00cf780 tx=0x7f93a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27db0d821d3c19c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93beffd640 1 -- 172.21.15.29:0/2300379212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8031830 con 0x7f93c00c9ca0 2023-10-14T16:53:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93beffd640 1 -- 172.21.15.29:0/2300379212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a8031990 con 0x7f93c00c9ca0 2023-10-14T16:53:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93beffd640 1 -- 172.21.15.29:0/2300379212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8031d20 con 0x7f93c00c9ca0 2023-10-14T16:53:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93c61f0640 1 -- 172.21.15.29:0/2300379212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 msgr2=0x7f93c006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2300379212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c006a2e0 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f93c00cf780 tx=0x7f93a802b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93c61f0640 1 -- 172.21.15.29:0/2300379212 shutdown_connections 2023-10-14T16:53:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2300379212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c006a2e0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.604+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2300379212 >> v2:172.21.15.186:3300/0 conn(0x7f93c015ad70 0x7f93c0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.605+0000 7f93c61f0640 1 -- 172.21.15.29:0/2300379212 >> 172.21.15.29:0/2300379212 conn(0x7f93c00ce9b0 msgr2=0x7f93c00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.605+0000 7f93c61f0640 1 -- 172.21.15.29:0/2300379212 shutdown_connections 2023-10-14T16:53:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.605+0000 7f93c61f0640 1 -- 172.21.15.29:0/2300379212 wait complete. 2023-10-14T16:53:47.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.606+0000 7f93c61f0640 1 Processor -- start 2023-10-14T16:53:47.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.606+0000 7f93c61f0640 1 -- start start 2023-10-14T16:53:47.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 0x7f93c014deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 0x7f93c00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c015c6c0 con 0x7f93c00c9ca0 2023-10-14T16:53:47.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93c014e3f0 con 0x7f93c015ad70 2023-10-14T16:53:47.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.607+0000 7f93c61f0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93c00c8d90 con 0x7f93c00ca3b0 2023-10-14T16:53:47.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57284/0 (socket says 172.21.15.29:57284) 2023-10-14T16:53:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 -- 172.21.15.29:0/2454157209 learned_addr learned my addr 172.21.15.29:0/2454157209 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 0x7f93c014deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bffff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 0x7f93c00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 -- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 msgr2=0x7f93c014deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 0x7f93c014deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 -- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 msgr2=0x7f93c00c5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 0x7f93c00c5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bf7fe640 1 -- 172.21.15.29:0/2454157209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a8009400 con 0x7f93c00c9ca0 2023-10-14T16:53:47.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 0x7f93c014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.608+0000 7f93bffff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 0x7f93c00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93bf7fe640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f93b000d5f0 tx=0x7f93b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b00125f0 con 0x7f93c00c9ca0 2023-10-14T16:53:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b0012750 con 0x7f93c00c9ca0 2023-10-14T16:53:47.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93b0012b00 con 0x7f93c00c9ca0 2023-10-14T16:53:47.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93c015c000 con 0x7f93c00c9ca0 2023-10-14T16:53:47.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.609+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93c00c60a0 con 0x7f93c00c9ca0 2023-10-14T16:53:47.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.610+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93c015be60 con 0x7f93c00c9ca0 2023-10-14T16:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.611+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f93b0012c60 con 0x7f93c00c9ca0 2023-10-14T16:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.612+0000 7f93bcff9640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 0x7f93a006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:47.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.612+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6567+0+0 (secure 0 0 0) 0x7f93b008a200 con 0x7f93c00c9ca0 2023-10-14T16:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.613+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 0x7f93a006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.614+0000 7f93b7fff640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 0x7f93a006d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93a8036000 tx=0x7f93a8008030 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:47.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.616+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f93c015be60 con 0x7f93c00c9ca0 2023-10-14T16:53:47.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:47.836+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir8", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f93c00ad4d0 con 0x7f93a006b070 2023-10-14T16:53:48.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.078+0000 7f93bcff9640 1 -- 172.21.15.29:0/2454157209 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7f93a8045400 con 0x7f93a006b070 2023-10-14T16:53:48.079 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir8 2023-10-14T16:53:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.081+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 msgr2=0x7f93a006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.081+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 0x7f93a006d530 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f93a8036000 tx=0x7f93a8008030 comp rx=0 tx=0).stop 2023-10-14T16:53:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 msgr2=0x7f93c00c8720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:48.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f93b000d5f0 tx=0x7f93b000dab0 comp rx=0 tx=0).stop 2023-10-14T16:53:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 shutdown_connections 2023-10-14T16:53:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f93a006b070 0x7f93a006d530 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93c015ad70 0x7f93c00c5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:48.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f93c00ca3b0 0x7f93c014deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 --2- 172.21.15.29:0/2454157209 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f93c00c9ca0 0x7f93c00c8720 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 >> 172.21.15.29:0/2454157209 conn(0x7f93c00ce9b0 msgr2=0x7f93c0070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 shutdown_connections 2023-10-14T16:53:48.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:48.082+0000 7f93c61f0640 1 -- 172.21.15.29:0/2454157209 wait complete. 2023-10-14T16:53:48.110 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir8 2023-10-14T16:53:48.110 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.111 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir8/.snap) 2023-10-14T16:53:48.178 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:48.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:48.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:48.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:48.181 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:48.182 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_43_01_UTC 2023-10-14T16:53:48.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.183 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_43_01_UTC) 2023-10-14T16:53:48.243 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_44_01_UTC 2023-10-14T16:53:48.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.244 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_44_01_UTC) 2023-10-14T16:53:48.304 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:48.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.305 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_45_01_UTC) 2023-10-14T16:53:48.369 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_46_00_UTC 2023-10-14T16:53:48.370 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.370 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_46_00_UTC) 2023-10-14T16:53:48.446 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:48.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.447 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_47_01_UTC) 2023-10-14T16:53:48.545 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_48_01_UTC 2023-10-14T16:53:48.545 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.546 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_48_01_UTC) 2023-10-14T16:53:48.601 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_49_01_UTC 2023-10-14T16:53:48.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.602 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_49_01_UTC) 2023-10-14T16:53:48.662 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_50_00_UTC 2023-10-14T16:53:48.663 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.663 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_50_00_UTC) 2023-10-14T16:53:48.725 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:48.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.726 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:48.801 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:48.803 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.803 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_52_01_UTC) 2023-10-14T16:53:48.883 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_53_00_UTC 2023-10-14T16:53:48.883 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.884 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8/.snap/scheduled-2023-10-14-16_53_00_UTC) 2023-10-14T16:53:48.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:48.963 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir8) 2023-10-14T16:53:49.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/dir9 1m 2023-10-14T16:53:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.264+0000 7fddd7ff0640 1 Processor -- start 2023-10-14T16:53:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.264+0000 7fddd7ff0640 1 -- start start 2023-10-14T16:53:49.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 0x7fddd00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddd0159e90 0x7fddd006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd006cb10 con 0x7fddd0159e90 2023-10-14T16:53:49.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd015d890 con 0x7fddd0159ab0 2023-10-14T16:53:49.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd7ff0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00cd1f0 con 0x7fddd006c140 2023-10-14T16:53:49.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd558b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddd0159e90 0x7fddd006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd558b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddd0159e90 0x7fddd006fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57300/0 (socket says 172.21.15.29:57300) 2023-10-14T16:53:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.266+0000 7fddd558b640 1 -- 172.21.15.29:0/820153801 learned_addr learned my addr 172.21.15.29:0/820153801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd5d8c640 1 --2- 172.21.15.29:0/820153801 >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 0x7fddd00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd658d640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd558b640 1 -- 172.21.15.29:0/820153801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 msgr2=0x7fddd006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd558b640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd558b640 1 -- 172.21.15.29:0/820153801 >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 msgr2=0x7fddd00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd558b640 1 --2- 172.21.15.29:0/820153801 >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 0x7fddd00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd558b640 1 -- 172.21.15.29:0/820153801 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd00cd4d0 con 0x7fddd0159e90 2023-10-14T16:53:49.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd5d8c640 1 --2- 172.21.15.29:0/820153801 >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 0x7fddd00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.267+0000 7fddd658d640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:49.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.268+0000 7fddd558b640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd006fc50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fddd00cfa00 tx=0x7fddc002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7749c023b8a5b903 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:49.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.268+0000 7fddd4d8a640 1 -- 172.21.15.29:0/820153801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0031750 con 0x7fddd0159e90 2023-10-14T16:53:49.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.268+0000 7fddd4d8a640 1 -- 172.21.15.29:0/820153801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc00318b0 con 0x7fddd0159e90 2023-10-14T16:53:49.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.268+0000 7fddd4d8a640 1 -- 172.21.15.29:0/820153801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc0031c40 con 0x7fddd0159e90 2023-10-14T16:53:49.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 -- 172.21.15.29:0/820153801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 msgr2=0x7fddd006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd006fc50 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fddd00cfa00 tx=0x7fddc002b7f0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 -- 172.21.15.29:0/820153801 shutdown_connections 2023-10-14T16:53:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/820153801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd006fc50 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/820153801 >> v2:172.21.15.186:3300/0 conn(0x7fddd0159ab0 0x7fddd00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 -- 172.21.15.29:0/820153801 >> 172.21.15.29:0/820153801 conn(0x7fddd00ce9b0 msgr2=0x7fddd00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 -- 172.21.15.29:0/820153801 shutdown_connections 2023-10-14T16:53:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.269+0000 7fddd7ff0640 1 -- 172.21.15.29:0/820153801 wait complete. 2023-10-14T16:53:49.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.271+0000 7fddd7ff0640 1 Processor -- start 2023-10-14T16:53:49.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.272+0000 7fddd7ff0640 1 -- start start 2023-10-14T16:53:49.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd00aed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd0159ab0 0x7fddd014c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00c6830 con 0x7fddd0159e90 2023-10-14T16:53:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00c5bb0 con 0x7fddd0159ab0 2023-10-14T16:53:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd7ff0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd00c5e90 con 0x7fddd006c140 2023-10-14T16:53:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd658d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd658d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57306/0 (socket says 172.21.15.29:57306) 2023-10-14T16:53:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd658d640 1 -- 172.21.15.29:0/1642500531 learned_addr learned my addr 172.21.15.29:0/1642500531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.273+0000 7fddd5d8c640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd00aed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd558b640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd0159ab0 0x7fddd014c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 -- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 msgr2=0x7fddd00aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd00aed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 -- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd0159ab0 msgr2=0x7fddd014c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd0159ab0 0x7fddd014c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 -- 172.21.15.29:0/1642500531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc00094d0 con 0x7fddd0159e90 2023-10-14T16:53:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd658d640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fddc800cd10 tx=0x7fddc800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc800f680 con 0x7fddd0159e90 2023-10-14T16:53:49.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc800f800 con 0x7fddd0159e90 2023-10-14T16:53:49.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd00c6170 con 0x7fddd0159e90 2023-10-14T16:53:49.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.274+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddc800fbb0 con 0x7fddd0159e90 2023-10-14T16:53:49.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.275+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd00af250 con 0x7fddd0159e90 2023-10-14T16:53:49.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.276+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd00af440 con 0x7fddd0159e90 2023-10-14T16:53:49.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.277+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7fddc800b040 con 0x7fddd0159e90 2023-10-14T16:53:49.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.278+0000 7fddc67fc640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 0x7fddb406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.279+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 6391+0+0 (secure 0 0 0) 0x7fddc808a780 con 0x7fddd0159e90 2023-10-14T16:53:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.279+0000 7fddd5d8c640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 0x7fddb406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:49.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.280+0000 7fddd5d8c640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 0x7fddb406d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00a990 tx=0x7fddcc008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:49.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.283+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fddc8055160 con 0x7fddd0159e90 2023-10-14T16:53:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.531+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 --> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/dir9", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fddd00cda90 con 0x7fddb406b070 2023-10-14T16:53:49.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.736+0000 7fddc67fc640 1 -- 172.21.15.29:0/1642500531 <== mgr.5352 v2:172.21.15.29:6832/115188615 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+47 (secure 0 0 0) 0x7fddcc00a660 con 0x7fddb406b070 2023-10-14T16:53:49.738 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/dir9 2023-10-14T16:53:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 msgr2=0x7fddb406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 0x7fddb406d530 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fddcc00a990 tx=0x7fddcc008040 comp rx=0 tx=0).stop 2023-10-14T16:53:49.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 msgr2=0x7fddd00c6c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fddc800cd10 tx=0x7fddc800a650 comp rx=0 tx=0).stop 2023-10-14T16:53:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 shutdown_connections 2023-10-14T16:53:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddd0159e90 0x7fddd00c6c50 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fddb406b070 0x7fddb406d530 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddd0159ab0 0x7fddd014c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 --2- 172.21.15.29:0/1642500531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddd006c140 0x7fddd00aed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 >> 172.21.15.29:0/1642500531 conn(0x7fddd00ce9b0 msgr2=0x7fddd00b0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 shutdown_connections 2023-10-14T16:53:49.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:49.740+0000 7fddd7ff0640 1 -- 172.21.15.29:0/1642500531 wait complete. 2023-10-14T16:53:49.761 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/dir9 2023-10-14T16:53:49.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:49.762 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/dir9/.snap) 2023-10-14T16:53:49.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:49.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:49.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:49.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:49.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:49.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:49.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:49.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:49.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:49.818 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:49.818 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:49.819 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_43_00_UTC 2023-10-14T16:53:49.820 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:49.820 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_43_00_UTC) 2023-10-14T16:53:49.893 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_44_00_UTC 2023-10-14T16:53:49.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:49.894 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_44_00_UTC) 2023-10-14T16:53:49.977 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_45_01_UTC 2023-10-14T16:53:49.978 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:49.978 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_45_01_UTC) 2023-10-14T16:53:50.079 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_46_01_UTC 2023-10-14T16:53:50.079 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.079 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_46_01_UTC) 2023-10-14T16:53:50.147 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_47_01_UTC 2023-10-14T16:53:50.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.148 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_47_01_UTC) 2023-10-14T16:53:50.237 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_48_00_UTC 2023-10-14T16:53:50.238 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.238 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_48_00_UTC) 2023-10-14T16:53:50.296 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_49_00_UTC 2023-10-14T16:53:50.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.297 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_49_00_UTC) 2023-10-14T16:53:50.352 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_50_01_UTC 2023-10-14T16:53:50.353 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.353 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_50_01_UTC) 2023-10-14T16:53:50.411 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_51_00_UTC 2023-10-14T16:53:50.411 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.412 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_51_00_UTC) 2023-10-14T16:53:50.479 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_52_01_UTC 2023-10-14T16:53:50.479 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.480 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_52_01_UTC) 2023-10-14T16:53:50.554 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_53_01_UTC 2023-10-14T16:53:50.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.555 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9/.snap/scheduled-2023-10-14-16_53_01_UTC) 2023-10-14T16:53:50.628 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:50.628 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/dir9) 2023-10-14T16:53:50.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T16:53:50.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.920+0000 7f2ecf983640 1 Processor -- start 2023-10-14T16:53:50.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.921+0000 7f2ecf983640 1 -- start start 2023-10-14T16:53:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ec815d150 0x7f2ec80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 0x7f2ec8160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec8069da0 0x7f2ec806a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec815d530 con 0x7f2ec8069da0 2023-10-14T16:53:50.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ec806a6f0 con 0x7f2ec815d150 2023-10-14T16:53:50.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.922+0000 7f2ecf983640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80ac890 con 0x7f2ec8160b90 2023-10-14T16:53:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.923+0000 7f2ecdf20640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec8069da0 0x7f2ec806a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.923+0000 7f2ecdf20640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ec8069da0 0x7f2ec806a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57320/0 (socket says 172.21.15.29:57320) 2023-10-14T16:53:50.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.923+0000 7f2ecdf20640 1 -- 172.21.15.29:0/3757910597 learned_addr learned my addr 172.21.15.29:0/3757910597 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.923+0000 7f2ecd71f640 1 --2- 172.21.15.29:0/3757910597 >> v2:172.21.15.186:3300/0 conn(0x7f2ec815d150 0x7f2ec80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.923+0000 7f2eccf1e640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 0x7f2ec8160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2ecdf20640 1 -- 172.21.15.29:0/3757910597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 msgr2=0x7f2ec8160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 0x7f2ec8160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2ecdf20640 1 -- 172.21.15.29:0/3757910597 >> v2:172.21.15.186:3300/0 conn(0x7f2ec815d150 msgr2=0x7f2ec80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/3757910597 >> v2:172.21.15.186:3300/0 conn(0x7f2ec815d150 0x7f2ec80ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2ecdf20640 1 -- 172.21.15.29:0/3757910597 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec8161650 con 0x7f2ec8069da0 2023-10-14T16:53:50.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.924+0000 7f2eccf1e640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 0x7f2ec8160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.925+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8069da0 0x7f2ec806a1b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80b15a0 tx=0x7f2ec002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fea12a171324089e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:50.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.925+0000 7f2ebffff640 1 -- 172.21.15.29:0/3757910597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec002ca80 con 0x7f2ec8069da0 2023-10-14T16:53:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.925+0000 7f2ebffff640 1 -- 172.21.15.29:0/3757910597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec002cbe0 con 0x7f2ec8069da0 2023-10-14T16:53:50.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.925+0000 7f2ebffff640 1 -- 172.21.15.29:0/3757910597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec00343f0 con 0x7f2ec8069da0 2023-10-14T16:53:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 -- 172.21.15.29:0/3757910597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8069da0 msgr2=0x7f2ec806a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8069da0 0x7f2ec806a1b0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f2ec80b15a0 tx=0x7f2ec002bee0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 -- 172.21.15.29:0/3757910597 shutdown_connections 2023-10-14T16:53:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8069da0 0x7f2ec806a1b0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 --2- 172.21.15.29:0/3757910597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8160b90 0x7f2ec8160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 --2- 172.21.15.29:0/3757910597 >> v2:172.21.15.186:3300/0 conn(0x7f2ec815d150 0x7f2ec80ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 -- 172.21.15.29:0/3757910597 >> 172.21.15.29:0/3757910597 conn(0x7f2ec80ce9b0 msgr2=0x7f2ec80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 -- 172.21.15.29:0/3757910597 shutdown_connections 2023-10-14T16:53:50.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.926+0000 7f2ecf983640 1 -- 172.21.15.29:0/3757910597 wait complete. 2023-10-14T16:53:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.928+0000 7f2ecf983640 1 Processor -- start 2023-10-14T16:53:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.928+0000 7f2ecf983640 1 -- start start 2023-10-14T16:53:50.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8069da0 0x7f2ec80ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec815d150 0x7f2ec815b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80aed70 con 0x7f2ec8160b90 2023-10-14T16:53:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80c2cd0 con 0x7f2ec815d150 2023-10-14T16:53:50.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecf983640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ec80aea90 con 0x7f2ec8069da0 2023-10-14T16:53:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.929+0000 7f2ecdf20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56644/0 (socket says 172.21.15.29:56644) 2023-10-14T16:53:50.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 -- 172.21.15.29:0/198072800 learned_addr learned my addr 172.21.15.29:0/198072800 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2eccf1e640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec815d150 0x7f2ec815b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecd71f640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8069da0 0x7f2ec80ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 -- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8069da0 msgr2=0x7f2ec80ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8069da0 0x7f2ec80ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 -- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec815d150 msgr2=0x7f2ec815b7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:50.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec815d150 0x7f2ec815b7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 -- 172.21.15.29:0/198072800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0009b30 con 0x7f2ec8160b90 2023-10-14T16:53:50.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.930+0000 7f2ecdf20640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009f40 tx=0x7f2ec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.931+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec0009510 con 0x7f2ec8160b90 2023-10-14T16:53:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.931+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ec003e450 con 0x7f2ec8160b90 2023-10-14T16:53:50.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.931+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ec003e800 con 0x7f2ec8160b90 2023-10-14T16:53:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.931+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec80c2680 con 0x7f2ec8160b90 2023-10-14T16:53:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.931+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec80ae780 con 0x7f2ec8160b90 2023-10-14T16:53:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.932+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ec80c24e0 con 0x7f2ec8160b90 2023-10-14T16:53:50.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.933+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 13) v1 ==== 89417+0+0 (secure 0 0 0) 0x7f2ec002cd50 con 0x7f2ec8160b90 2023-10-14T16:53:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.934+0000 7f2ebdffb640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 0x7f2eac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.934+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 7171+0+0 (secure 0 0 0) 0x7f2ec00aca30 con 0x7f2ec8160b90 2023-10-14T16:53:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.934+0000 7f2ecd71f640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 0x7f2eac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:50.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.935+0000 7f2ecd71f640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 0x7f2eac06d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).ready entity=mgr.5352 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:50.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:50.939+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ec80c24e0 con 0x7f2ec8160b90 2023-10-14T16:53:51.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:51.198+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f2ec80c24e0 con 0x7f2ec8160b90 2023-10-14T16:53:52.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.111+0000 7f2ebdffb640 1 -- 172.21.15.29:0/198072800 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v14) v1 ==== 93+0+0 (secure 0 0 0) 0x7f2ec80c24e0 con 0x7f2ec8160b90 2023-10-14T16:53:52.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 msgr2=0x7f2eac06d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 0x7f2eac06d530 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2ec4006fd0 tx=0x7f2ec4008040 comp rx=0 tx=0).stop 2023-10-14T16:53:52.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 msgr2=0x7f2ec80c7250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f2ec0009f40 tx=0x7f2ec0004a40 comp rx=0 tx=0).stop 2023-10-14T16:53:52.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 shutdown_connections 2023-10-14T16:53:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ec8160b90 0x7f2ec80c7250 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f2eac06b070 0x7f2eac06d530 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ec815d150 0x7f2ec815b7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 --2- 172.21.15.29:0/198072800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ec8069da0 0x7f2ec80ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.115+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 >> 172.21.15.29:0/198072800 conn(0x7f2ec80ce9b0 msgr2=0x7f2ec806f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:52.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.116+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 shutdown_connections 2023-10-14T16:53:52.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.116+0000 7f2ecf983640 1 -- 172.21.15.29:0/198072800 wait complete. 2023-10-14T16:53:52.139 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:53:52.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:53:52.257 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:53:52.359 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:53:52.360 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:53:52.360 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:53:52.419 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:53:52.419 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:53:52.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:53:52.650 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:52.650+0000 7fc550830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:53:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.690+0000 7fc0c554e640 1 Processor -- start 2023-10-14T16:53:52.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.690+0000 7fc0c554e640 1 -- start start 2023-10-14T16:53:52.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0c554e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0c015a970 0x7fc0c006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:52.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0c554e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0070660 con 0x7fc0c015d430 2023-10-14T16:53:52.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0c554e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0c0071530 con 0x7fc0c00cd880 2023-10-14T16:53:52.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0c554e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0c015dc60 con 0x7fc0c015a970 2023-10-14T16:53:52.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0beffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0c015a970 0x7fc0c006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:52.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0be7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc0c015d430 0x7fc0c0070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40822/0 (socket says 172.21.15.29:40822) 2023-10-14T16:53:52.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.692+0000 7fc0be7fc640 1 -- 172.21.15.29:0/1619720484 learned_addr learned my addr 172.21.15.29:0/1619720484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:52.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.693+0000 7fc0beffd640 1 -- 172.21.15.29:0/1619720484 >> v1:172.21.15.186:6789/0 conn(0x7fc0c00cd880 legacy=0x7fc0c0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:52.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.693+0000 7fc0bdffb640 1 -- 172.21.15.29:0/1619720484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2277904083 0 0) 0x7fc0c0070660 con 0x7fc0c015d430 2023-10-14T16:53:52.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.693+0000 7fc0beffd640 1 -- 172.21.15.29:0/1619720484 >> v1:172.21.15.29:6789/0 conn(0x7fc0c015d430 legacy=0x7fc0c0070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.693+0000 7fc0beffd640 1 -- 172.21.15.29:0/1619720484 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c015df40 con 0x7fc0c015a970 2023-10-14T16:53:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.694+0000 7fc0beffd640 1 --2- 172.21.15.29:0/1619720484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c006ecd0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00097a0 tx=0x7fc0b002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ea4f62345a4b871 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0bdffb640 1 -- 172.21.15.29:0/1619720484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0031830 con 0x7fc0c015a970 2023-10-14T16:53:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0bdffb640 1 -- 172.21.15.29:0/1619720484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b0031990 con 0x7fc0c015a970 2023-10-14T16:53:52.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0bdffb640 1 -- 172.21.15.29:0/1619720484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0031d20 con 0x7fc0c015a970 2023-10-14T16:53:52.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0c554e640 1 -- 172.21.15.29:0/1619720484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 msgr2=0x7fc0c006ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1619720484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c006ecd0 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7fc0b00097a0 tx=0x7fc0b002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0c554e640 1 -- 172.21.15.29:0/1619720484 shutdown_connections 2023-10-14T16:53:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1619720484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c006ecd0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.695+0000 7fc0c554e640 1 -- 172.21.15.29:0/1619720484 >> 172.21.15.29:0/1619720484 conn(0x7fc0c00cfaa0 msgr2=0x7fc0c00cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.696+0000 7fc0c554e640 1 -- 172.21.15.29:0/1619720484 shutdown_connections 2023-10-14T16:53:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.696+0000 7fc0c554e640 1 -- 172.21.15.29:0/1619720484 wait complete. 2023-10-14T16:53:52.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.698+0000 7fc0c554e640 1 Processor -- start 2023-10-14T16:53:52.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.698+0000 7fc0c554e640 1 -- start start 2023-10-14T16:53:52.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0c00cd880 0x7fc0c015c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c015d430 0x7fc0c0154500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c01409c0 con 0x7fc0c015a970 2023-10-14T16:53:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c0142100 con 0x7fc0c015d430 2023-10-14T16:53:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0c554e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0c00b17d0 con 0x7fc0c00cd880 2023-10-14T16:53:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0be7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0be7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56658/0 (socket says 172.21.15.29:56658) 2023-10-14T16:53:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.699+0000 7fc0be7fc640 1 -- 172.21.15.29:0/1956810684 learned_addr learned my addr 172.21.15.29:0/1956810684 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:52.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0beffd640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0c00cd880 0x7fc0c015c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:52.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0bf7fe640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c015d430 0x7fc0c0154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0c00cd880 msgr2=0x7fc0c015c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0c00cd880 0x7fc0c015c320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c015d430 msgr2=0x7fc0c0154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c015d430 0x7fc0c0154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 -- 172.21.15.29:0/1956810684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0b0009400 con 0x7fc0c015a970 2023-10-14T16:53:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.700+0000 7fc0be7fc640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c4d0 tx=0x7fc0b400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.701+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b40074e0 con 0x7fc0c015a970 2023-10-14T16:53:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.701+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b4007660 con 0x7fc0c015a970 2023-10-14T16:53:52.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.701+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4004a20 con 0x7fc0c015a970 2023-10-14T16:53:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.701+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0c00b1a90 con 0x7fc0c015a970 2023-10-14T16:53:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.701+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0c00b2050 con 0x7fc0c015a970 2023-10-14T16:53:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.702+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0c0140b70 con 0x7fc0c015a970 2023-10-14T16:53:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.703+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fc0b4004b80 con 0x7fc0c015a970 2023-10-14T16:53:52.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.705+0000 7fc0a77fe640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 0x7fc0ac06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:52.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.705+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fc0b4088e80 con 0x7fc0c015a970 2023-10-14T16:53:52.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.705+0000 7fc0beffd640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 msgr2=0x7fc0ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:52.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.705+0000 7fc0beffd640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 0x7fc0ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:52.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.711+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0c0140b70 con 0x7fc0c015a970 2023-10-14T16:53:52.718 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:52.718+0000 7fc550830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:53:52.721 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:52.721+0000 7fcf87c3a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:53:52.788 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:52.787+0000 7fcf87c3a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:53:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.906+0000 7fc0beffd640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 msgr2=0x7fc0ac06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:52.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.906+0000 7fc0beffd640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 0x7fc0ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:52.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.934+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0c004e160 con 0x7fc0c015a970 2023-10-14T16:53:52.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.937+0000 7fc0a77fe640 1 -- 172.21.15.29:0/1956810684 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+17988 (secure 0 0 0) 0x7fc0b4018020 con 0x7fc0c015a970 2023-10-14T16:53:52.939 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:52.939 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:53:52.019590+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":222,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1900972994":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/3481939635":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/4149980298":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2383868706":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6832/3113675785":"2023-10-15T16:41:58.134928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":2},{"begin":10,"length":2},{"begin":13,"length":1},{"begin":15,"length":3},{"begin":22,"length":3},{"begin":28,"length":1},{"begin":31,"length":3},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":45,"length":3},{"begin":49,"length":2},{"begin":54,"length":1},{"begin":58,"length":3},{"begin":62,"length":1},{"begin":64,"length":1},{"begin":67,"length":3},{"begin":71,"length":1},{"begin":73,"length":1},{"begin":76,"length":3},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":88,"length":1},{"begin":91,"length":2},{"begin":94,"length":2},{"begin":97,"length":1},{"begin":100,"length":2},{"begin":104,"length":2},{"begin":108,"length":2},{"begin":111,"length":1},{"begin":168,"length":55}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":15,"length":1},{"begin":28,"length":1},{"begin":39,"length":1},{"begin":49,"length":1},{"begin":58,"length":1},{"begin":68,"length":1},{"begin":78,"length":1},{"begin":88,"length":1},{"begin":100,"length":1},{"begin":214,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:53:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 msgr2=0x7fc0ac06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 0x7fc0ac06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 msgr2=0x7fc0c00b1290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7fc0b400c4d0 tx=0x7fc0b400c990 comp rx=0 tx=0).stop 2023-10-14T16:53:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 shutdown_connections 2023-10-14T16:53:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fc0ac06b020 0x7fc0ac06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0c015d430 0x7fc0c0154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0c015a970 0x7fc0c00b1290 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 --2- 172.21.15.29:0/1956810684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0c00cd880 0x7fc0c015c320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 >> 172.21.15.29:0/1956810684 conn(0x7fc0c00cfaa0 msgr2=0x7fc0c0069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.940+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 shutdown_connections 2023-10-14T16:53:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:52.941+0000 7fc0c554e640 1 -- 172.21.15.29:0/1956810684 wait complete. 2023-10-14T16:53:52.959 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:41:36.429166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:41:37.759317+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 222, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:53:52.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:53:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.168+0000 7f2307fff640 1 Processor -- start 2023-10-14T16:53:53.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.168+0000 7f2307fff640 1 -- start start 2023-10-14T16:53:53.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f2307fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f23080b1ca0 0x7f23080d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f2307fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080d0970 con 0x7f23080cfc40 2023-10-14T16:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f2307fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23080d0ea0 con 0x7f23080b2080 2023-10-14T16:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f2307fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23080d0bc0 con 0x7f23080b1ca0 2023-10-14T16:53:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f23077fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f23080cfc40 0x7f23080b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40828/0 (socket says 172.21.15.29:40828) 2023-10-14T16:53:53.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.170+0000 7f23077fe640 1 -- 172.21.15.29:0/449584494 learned_addr learned my addr 172.21.15.29:0/449584494 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2306ffd640 1 --2- 172.21.15.29:0/449584494 >> v2:172.21.15.186:3300/0 conn(0x7f23080b1ca0 0x7f23080d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202138264 0 0) 0x7f23080d0970 con 0x7f23080cfc40 2023-10-14T16:53:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22f0003520 con 0x7f23080cfc40 2023-10-14T16:53:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1851334614 0 0) 0x7f22f0003520 con 0x7f23080cfc40 2023-10-14T16:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 >> v2:172.21.15.186:3300/0 conn(0x7f23080b1ca0 msgr2=0x7f23080d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 --2- 172.21.15.29:0/449584494 >> v2:172.21.15.186:3300/0 conn(0x7f23080b1ca0 0x7f23080d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 >> v1:172.21.15.186:6789/0 conn(0x7f23080b2080 legacy=0x7f23080ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23080cd940 con 0x7f23080cfc40 2023-10-14T16:53:53.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f23000019f0 con 0x7f23080cfc40 2023-10-14T16:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 >> v1:172.21.15.29:6789/0 conn(0x7f23080cfc40 legacy=0x7f23080b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f0003970 0x7f22f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f22f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.171+0000 7f2305ffb640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f0005180 0x7f22f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23080d0bc0 con 0x7f22f00047c0 2023-10-14T16:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f000c0b0 con 0x7f22f0003970 2023-10-14T16:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f000c270 con 0x7f22f0005180 2023-10-14T16:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f22f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f2306ffd640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f0003970 0x7f22f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 -- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f0005180 msgr2=0x7f22f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f0005180 0x7f22f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 -- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f0003970 msgr2=0x7f22f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f0003970 0x7f22f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23067fc640 1 -- 172.21.15.29:0/449584494 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0003520 con 0x7f22f00047c0 2023-10-14T16:53:53.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f2306ffd640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f0003970 0x7f22f0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.172+0000 7f23077fe640 1 -- 172.21.15.29:0/449584494 reap_dead start 2023-10-14T16:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.173+0000 7f23067fc640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f22f0004c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001750 tx=0x7f22f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5138c74888650e5e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.173+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc0052d0 con 0x7f22f00047c0 2023-10-14T16:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.173+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc02e050 con 0x7f22f00047c0 2023-10-14T16:53:53.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.173+0000 7f2305ffb640 1 -- 172.21.15.29:0/449584494 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22fc003830 con 0x7f22f00047c0 2023-10-14T16:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 -- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 msgr2=0x7f22f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f22f0004c40 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f22fc001750 tx=0x7f22f00057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 -- 172.21.15.29:0/449584494 shutdown_connections 2023-10-14T16:53:53.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 --2- 172.21.15.29:0/449584494 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f22f0004c40 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 -- 172.21.15.29:0/449584494 >> 172.21.15.29:0/449584494 conn(0x7f23080d2b90 msgr2=0x7f23080d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:53.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 -- 172.21.15.29:0/449584494 shutdown_connections 2023-10-14T16:53:53.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.174+0000 7f2307fff640 1 -- 172.21.15.29:0/449584494 wait complete. 2023-10-14T16:53:53.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.176+0000 7f2307fff640 1 Processor -- start 2023-10-14T16:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.176+0000 7f2307fff640 1 -- start start 2023-10-14T16:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080ca220 0x7f23080afec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23080ca6a0 0x7f23080b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308149930 con 0x7f22f00047c0 2023-10-14T16:53:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2308146d10 con 0x7f23080ca220 2023-10-14T16:53:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2307fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2308148d20 con 0x7f23080ca6a0 2023-10-14T16:53:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2306ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56684/0 (socket says 172.21.15.29:56684) 2023-10-14T16:53:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.177+0000 7f2306ffd640 1 -- 172.21.15.29:0/4147972780 learned_addr learned my addr 172.21.15.29:0/4147972780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f23067fc640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080ca220 0x7f23080afec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f23077fe640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23080ca6a0 0x7f23080b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23080ca6a0 msgr2=0x7f23080b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23080ca6a0 0x7f23080b0400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080ca220 msgr2=0x7f23080afec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080ca220 0x7f23080afec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 -- 172.21.15.29:0/4147972780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2308149000 con 0x7f22f00047c0 2023-10-14T16:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.178+0000 7f2306ffd640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e7e0 tx=0x7f22f800eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.179+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f800c8d0 con 0x7f22f00047c0 2023-10-14T16:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.179+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f800ca30 con 0x7f22f00047c0 2023-10-14T16:53:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.179+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f800cda0 con 0x7f22f00047c0 2023-10-14T16:53:53.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.179+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23081492e0 con 0x7f22f00047c0 2023-10-14T16:53:53.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.179+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f230814cd10 con 0x7f22f00047c0 2023-10-14T16:53:53.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.180+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f230804e9b0 con 0x7f22f00047c0 2023-10-14T16:53:53.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.181+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f22f8013410 con 0x7f22f00047c0 2023-10-14T16:53:53.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.182+0000 7f22ef7fe640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 0x7f22f406d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.182+0000 7f23067fc640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 msgr2=0x7f22f406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:53.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.182+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f22f8089d20 con 0x7f22f00047c0 2023-10-14T16:53:53.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.182+0000 7f23067fc640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 0x7f22f406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:53.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.186+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22f80545f0 con 0x7f22f00047c0 2023-10-14T16:53:53.206 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.206+0000 7fc550830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:53:53.258 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.258+0000 7fc550830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:53:53.286 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.285+0000 7fcf87c3a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:53:53.309 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.308+0000 7fc550830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:53:53.342 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.341+0000 7fcf87c3a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:53:53.369 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.369+0000 7fc550830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:53:53.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.382+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2308073a40 con 0x7f22f00047c0 2023-10-14T16:53:53.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.382+0000 7f22ef7fe640 1 -- 172.21.15.29:0/4147972780 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f22f8057c10 con 0x7f22f00047c0 2023-10-14T16:53:53.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.383+0000 7f23067fc640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 msgr2=0x7f22f406d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:53.384 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:53:53.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.383+0000 7f23067fc640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 0x7f22f406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:53.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 msgr2=0x7f22f406d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:53.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 0x7f22f406d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 msgr2=0x7f23080c8e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f22f800e7e0 tx=0x7f22f800eca0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 shutdown_connections 2023-10-14T16:53:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23080ca6a0 0x7f23080b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f22f406b050 0x7f22f406d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23080ca220 0x7f23080afec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.385+0000 7f2307fff640 1 --2- 172.21.15.29:0/4147972780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f00047c0 0x7f23080c8e60 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.386+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 >> 172.21.15.29:0/4147972780 conn(0x7f23080d2b90 msgr2=0x7f23080d1170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:53.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.386+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 shutdown_connections 2023-10-14T16:53:53.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.386+0000 7f2307fff640 1 -- 172.21.15.29:0/4147972780 wait complete. 2023-10-14T16:53:53.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.395+0000 7fcf87c3a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:53:53.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:53:53.418 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.419+0000 7fc550830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:53:53.461 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.461+0000 7fcf87c3a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:53:53.514 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.513+0000 7fcf87c3a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:53:53.529 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.530+0000 7fc550830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:53:53.593 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.593+0000 7fc550830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:53:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.613+0000 7f55ffa44640 1 Processor -- start 2023-10-14T16:53:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.614+0000 7f55ffa44640 1 -- start start 2023-10-14T16:53:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.615+0000 7f55ffa44640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55f8070830 0x7f55f806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.615+0000 7f55ffa44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.615+0000 7f55ffa44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55f8161740 con 0x7f55f80cf5f0 2023-10-14T16:53:53.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55ffa44640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55f80cdbd0 con 0x7f55f8070830 2023-10-14T16:53:53.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55ffa44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f806b0f0 con 0x7f55f8070c10 2023-10-14T16:53:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55fdfe1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55fdfe1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:49356/0 (socket says 172.21.15.29:49356) 2023-10-14T16:53:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55fdfe1640 1 -- 172.21.15.29:0/4158179402 learned_addr learned my addr 172.21.15.29:0/4158179402 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.616+0000 7f55fd7e0640 1 --2- 172.21.15.29:0/4158179402 >> v2:172.21.15.186:3300/0 conn(0x7f55f8070830 0x7f55f806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.617+0000 7f55fdfe1640 1 -- 172.21.15.29:0/4158179402 >> v2:172.21.15.186:3300/0 conn(0x7f55f8070830 msgr2=0x7f55f806d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.617+0000 7f55fdfe1640 1 --2- 172.21.15.29:0/4158179402 >> v2:172.21.15.186:3300/0 conn(0x7f55f8070830 0x7f55f806d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.617+0000 7f55fdfe1640 1 -- 172.21.15.29:0/4158179402 >> v1:172.21.15.186:6789/0 conn(0x7f55f80cf5f0 legacy=0x7f55f806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.617+0000 7f55fdfe1640 1 -- 172.21.15.29:0/4158179402 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f81619b0 con 0x7f55f8070c10 2023-10-14T16:53:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.618+0000 7f55fdfe1640 1 --2- 172.21.15.29:0/4158179402 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009870 tx=0x7f55f002b860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=93c5063f09286e54 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.619+0000 7f55effff640 1 -- 172.21.15.29:0/4158179402 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f00317c0 con 0x7f55f8070c10 2023-10-14T16:53:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.619+0000 7f55effff640 1 -- 172.21.15.29:0/4158179402 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55f0031920 con 0x7f55f8070c10 2023-10-14T16:53:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.619+0000 7f55effff640 1 -- 172.21.15.29:0/4158179402 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55f0031cb0 con 0x7f55f8070c10 2023-10-14T16:53:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 -- 172.21.15.29:0/4158179402 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 msgr2=0x7f55f80cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 --2- 172.21.15.29:0/4158179402 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f55f0009870 tx=0x7f55f002b860 comp rx=0 tx=0).stop 2023-10-14T16:53:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 -- 172.21.15.29:0/4158179402 shutdown_connections 2023-10-14T16:53:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 --2- 172.21.15.29:0/4158179402 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f8070c10 0x7f55f80cd690 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 --2- 172.21.15.29:0/4158179402 >> v2:172.21.15.186:3300/0 conn(0x7f55f8070830 0x7f55f806d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 -- 172.21.15.29:0/4158179402 >> 172.21.15.29:0/4158179402 conn(0x7f55f80cfac0 msgr2=0x7f55f80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 -- 172.21.15.29:0/4158179402 shutdown_connections 2023-10-14T16:53:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.620+0000 7f55ffa44640 1 -- 172.21.15.29:0/4158179402 wait complete. 2023-10-14T16:53:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.622+0000 7f55ffa44640 1 Processor -- start 2023-10-14T16:53:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.622+0000 7f55ffa44640 1 -- start start 2023-10-14T16:53:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8070830 0x7f55f815b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f80cf5f0 0x7f55f80c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c48b0 con 0x7f55f8070c10 2023-10-14T16:53:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c3d30 con 0x7f55f8070830 2023-10-14T16:53:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55ffa44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55f80c4010 con 0x7f55f80cf5f0 2023-10-14T16:53:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56692/0 (socket says 172.21.15.29:56692) 2023-10-14T16:53:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 -- 172.21.15.29:0/3229221278 learned_addr learned my addr 172.21.15.29:0/3229221278 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fd7e0640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8070830 0x7f55f815b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fdfe1640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f80cf5f0 0x7f55f80c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f80cf5f0 msgr2=0x7f55f80c3260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f80cf5f0 0x7f55f80c3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8070830 msgr2=0x7f55f815b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8070830 0x7f55f815b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.623+0000 7f55fcfdf640 1 -- 172.21.15.29:0/3229221278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55f00094d0 con 0x7f55f8070c10 2023-10-14T16:53:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55fcfdf640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c3c0 tx=0x7f55e800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e80052d0 con 0x7f55f8070c10 2023-10-14T16:53:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55e8005430 con 0x7f55f8070c10 2023-10-14T16:53:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55f80c42f0 con 0x7f55f8070c10 2023-10-14T16:53:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55e8003830 con 0x7f55f8070c10 2023-10-14T16:53:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.624+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55f815b680 con 0x7f55f8070c10 2023-10-14T16:53:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.625+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55f804e160 con 0x7f55f8070c10 2023-10-14T16:53:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.626+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f55e8003990 con 0x7f55f8070c10 2023-10-14T16:53:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.628+0000 7f55edffb640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 0x7f55dc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.628+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f55e8089200 con 0x7f55f8070c10 2023-10-14T16:53:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.628+0000 7f55fd7e0640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 msgr2=0x7f55dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.628+0000 7f55fd7e0640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 0x7f55dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:53.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.633+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55e8053c70 con 0x7f55f8070c10 2023-10-14T16:53:53.635 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.635+0000 7fcf87c3a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:53:53.644 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.644+0000 7fc550830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:53:53.702 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.702+0000 7fc550830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:53:53.703 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.702+0000 7fcf87c3a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:53:53.753 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:53.753+0000 7fc550830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:53:53.755 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.755+0000 7fcf87c3a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:53:53.815 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.815+0000 7fcf87c3a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:53:53.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.826+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f55f80b5a40 con 0x7f55f8070c10 2023-10-14T16:53:53.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.827+0000 7f55edffb640 1 -- 172.21.15.29:0/3229221278 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) v1 ==== 104+0+11 (secure 0 0 0) 0x7f55e8057290 con 0x7f55f8070c10 2023-10-14T16:53:53.828 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:53:53.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.829+0000 7f55fd7e0640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 msgr2=0x7f55dc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:53.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.829+0000 7f55fd7e0640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 0x7f55dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:53.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 msgr2=0x7f55dc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:53.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 0x7f55dc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 msgr2=0x7f55f81f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:53.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f55e800c3c0 tx=0x7f55e800c880 comp rx=0 tx=0).stop 2023-10-14T16:53:53.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 shutdown_connections 2023-10-14T16:53:53.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55f80cf5f0 0x7f55f80c3260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55f8070c10 0x7f55f81f21a0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f55dc06b020 0x7f55dc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 --2- 172.21.15.29:0/3229221278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55f8070830 0x7f55f815b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:53.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 >> 172.21.15.29:0/3229221278 conn(0x7f55f80cfac0 msgr2=0x7f55f80cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:53.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 shutdown_connections 2023-10-14T16:53:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:53.830+0000 7f55ffa44640 1 -- 172.21.15.29:0/3229221278 wait complete. 2023-10-14T16:53:53.850 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:53:53.870 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:53.870+0000 7fcf87c3a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:53:54.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.060+0000 7f5ad65c9640 1 Processor -- start 2023-10-14T16:53:54.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.060+0000 7f5ad65c9640 1 -- start start 2023-10-14T16:53:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ad65c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5ad0159ab0 0x7f5ad00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ad65c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad0070360 con 0x7f5ad0159e90 2023-10-14T16:53:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ad65c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad00cd1f0 con 0x7f5ad006c140 2023-10-14T16:53:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ad65c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad015d890 con 0x7f5ad0159ab0 2023-10-14T16:53:54.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ac7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5ad0159e90 0x7f5ad006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40836/0 (socket says 172.21.15.29:40836) 2023-10-14T16:53:54.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.062+0000 7f5ac7fff640 1 -- 172.21.15.29:0/929425520 learned_addr learned my addr 172.21.15.29:0/929425520 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acffff640 1 --2- 172.21.15.29:0/929425520 >> v2:172.21.15.186:3300/0 conn(0x7f5ad0159ab0 0x7f5ad00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291071111 0 0) 0x7f5ad0070360 con 0x7f5ad0159e90 2023-10-14T16:53:54.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5abc003520 con 0x7f5ad0159e90 2023-10-14T16:53:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2894031115 0 0) 0x7f5abc003520 con 0x7f5ad0159e90 2023-10-14T16:53:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 >> v2:172.21.15.186:3300/0 conn(0x7f5ad0159ab0 msgr2=0x7f5ad00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 --2- 172.21.15.29:0/929425520 >> v2:172.21.15.186:3300/0 conn(0x7f5ad0159ab0 0x7f5ad00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 >> v1:172.21.15.186:6789/0 conn(0x7f5ad006c140 legacy=0x7f5ad015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad00cd4d0 con 0x7f5ad0159e90 2023-10-14T16:53:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5ab8001670 con 0x7f5ad0159e90 2023-10-14T16:53:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 >> v1:172.21.15.29:6789/0 conn(0x7f5ad0159e90 legacy=0x7f5ad006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5abc0039c0 0x7f5abc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5abc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5abc0052a0 0x7f5abc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015d890 con 0x7f5abc0047c0 2023-10-14T16:53:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc00e3b0 con 0x7f5abc0052a0 2023-10-14T16:53:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.063+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5abc00e570 con 0x7f5abc0039c0 2023-10-14T16:53:54.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5abc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5acffff640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5abc0039c0 0x7f5abc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ad4b66640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5abc0052a0 0x7f5abc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 -- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5abc0039c0 msgr2=0x7f5abc003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:54.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5abc0039c0 0x7f5abc003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 -- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5abc0052a0 msgr2=0x7f5abc00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5abc0052a0 0x7f5abc00db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ac7fff640 1 -- 172.21.15.29:0/929425520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5abc003520 con 0x7f5abc0047c0 2023-10-14T16:53:54.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.064+0000 7f5ad4b66640 1 -- 172.21.15.29:0/929425520 reap_dead start 2023-10-14T16:53:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.066+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5abc004c40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c0e0 tx=0x7f5abc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5beea45db1190212 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:54.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.066+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8005ad0 con 0x7f5abc0047c0 2023-10-14T16:53:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.066+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab8005c30 con 0x7f5abc0047c0 2023-10-14T16:53:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.066+0000 7f5acf7fe640 1 -- 172.21.15.29:0/929425520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab802ca50 con 0x7f5abc0047c0 2023-10-14T16:53:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 -- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 msgr2=0x7f5abc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5abc004c40 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5ab800c0e0 tx=0x7f5abc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 -- 172.21.15.29:0/929425520 shutdown_connections 2023-10-14T16:53:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/929425520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5abc004c40 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 -- 172.21.15.29:0/929425520 >> 172.21.15.29:0/929425520 conn(0x7f5ad00ce9b0 msgr2=0x7f5ad00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 -- 172.21.15.29:0/929425520 shutdown_connections 2023-10-14T16:53:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.067+0000 7f5ad65c9640 1 -- 172.21.15.29:0/929425520 wait complete. 2023-10-14T16:53:54.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.070+0000 7f5ad65c9640 1 Processor -- start 2023-10-14T16:53:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.070+0000 7f5ad65c9640 1 -- start start 2023-10-14T16:53:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad01ee500 0x7f5ad00c2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015ad90 0x7f5ad01e2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015bf10 con 0x7f5abc0047c0 2023-10-14T16:53:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015b300 con 0x7f5ad01ee500 2023-10-14T16:53:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad65c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad015b5e0 con 0x7f5ad015ad90 2023-10-14T16:53:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5acffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5acffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56716/0 (socket says 172.21.15.29:56716) 2023-10-14T16:53:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5acffff640 1 -- 172.21.15.29:0/4028386613 learned_addr learned my addr 172.21.15.29:0/4028386613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad01ee500 0x7f5ad00c2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.072+0000 7f5ad4b66640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015ad90 0x7f5ad01e2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015ad90 msgr2=0x7f5ad01e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015ad90 0x7f5ad01e2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad01ee500 msgr2=0x7f5ad00c2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad01ee500 0x7f5ad00c2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 -- 172.21.15.29:0/4028386613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad015b8c0 con 0x7f5abc0047c0 2023-10-14T16:53:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acffff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00114e0 tx=0x7f5ac00119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0015870 con 0x7f5abc0047c0 2023-10-14T16:53:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac00159d0 con 0x7f5abc0047c0 2023-10-14T16:53:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0015d40 con 0x7f5abc0047c0 2023-10-14T16:53:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.073+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad015bba0 con 0x7f5abc0047c0 2023-10-14T16:53:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.074+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad00c3290 con 0x7f5abc0047c0 2023-10-14T16:53:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.075+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad004e160 con 0x7f5abc0047c0 2023-10-14T16:53:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.076+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f5ac0013400 con 0x7f5abc0047c0 2023-10-14T16:53:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.077+0000 7f5acd7fa640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 0x7f5ab006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.077+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f5ac008a040 con 0x7f5abc0047c0 2023-10-14T16:53:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.077+0000 7f5ac7fff640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 msgr2=0x7f5ab006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.077+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 0x7f5ab006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.081+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ac0054ac0 con 0x7f5abc0047c0 2023-10-14T16:53:54.142 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.142+0000 7fc550830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:53:54.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.275+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5ad00ccb60 con 0x7f5abc0047c0 2023-10-14T16:53:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.276+0000 7f5acd7fa640 1 -- 172.21.15.29:0/4028386613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ac00580e0 con 0x7f5abc0047c0 2023-10-14T16:53:54.277 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:53:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.278+0000 7f5ac7fff640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 msgr2=0x7f5ab006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.278+0000 7f5ac7fff640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 0x7f5ab006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 msgr2=0x7f5ab006d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 0x7f5ab006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 msgr2=0x7f5ad0156eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f5ac00114e0 tx=0x7f5ac00119a0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 shutdown_connections 2023-10-14T16:53:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015ad90 0x7f5ad01e2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f5ab006b050 0x7f5ab006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad01ee500 0x7f5ad00c2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 --2- 172.21.15.29:0/4028386613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5abc0047c0 0x7f5ad0156eb0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 >> 172.21.15.29:0/4028386613 conn(0x7f5ad00ce9b0 msgr2=0x7f5ad0159130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 shutdown_connections 2023-10-14T16:53:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.279+0000 7f5ad65c9640 1 -- 172.21.15.29:0/4028386613 wait complete. 2023-10-14T16:53:54.284 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.284+0000 7fcf87c3a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:53:54.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:53:54.304 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.304+0000 7fc550830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:53:54.413 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.413+0000 7fc550830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:53:54.458 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.458+0000 7fcf87c3a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:53:54.465 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.465+0000 7fc550830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:53:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.555+0000 7f169c6b4640 1 Processor -- start 2023-10-14T16:53:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.556+0000 7f169c6b4640 1 -- start start 2023-10-14T16:53:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.557+0000 7f169c6b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16940cf5f0 0x7f169406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.557+0000 7f169c6b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1694070c10 0x7f16940cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.557+0000 7f169c6b4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1694161740 con 0x7f1694070830 2023-10-14T16:53:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.557+0000 7f169c6b4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16940cdbd0 con 0x7f16940cf5f0 2023-10-14T16:53:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.557+0000 7f169c6b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f169406b0f0 con 0x7f1694070c10 2023-10-14T16:53:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16940cf5f0 0x7f169406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16940cf5f0 0x7f169406d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:43774/0 (socket says 172.21.15.29:43774) 2023-10-14T16:53:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 -- 172.21.15.29:0/372938396 learned_addr learned my addr 172.21.15.29:0/372938396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f169a450640 1 --1- 172.21.15.29:0/372938396 >> v1:172.21.15.186:6789/0 conn(0x7f1694070830 0x7f169406c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:41966/0 (socket says 172.21.15.29:41966) 2023-10-14T16:53:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f169ac51640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1694070c10 0x7f16940cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 -- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1694070c10 msgr2=0x7f16940cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1694070c10 0x7f16940cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f1699c4f640 1 -- 172.21.15.29:0/372938396 >> v1:172.21.15.186:6789/0 conn(0x7f1694070830 legacy=0x7f169406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.559+0000 7f1699c4f640 1 -- 172.21.15.29:0/372938396 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16941619b0 con 0x7f16940cf5f0 2023-10-14T16:53:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.558+0000 7f169944e640 1 -- 172.21.15.29:0/372938396 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247928396 0 0) 0x7f1694161740 con 0x7f1694070830 2023-10-14T16:53:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.560+0000 7f1699c4f640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16940cf5f0 0x7f169406d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1684009870 tx=0x7f168402b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f5666aca58651ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.560+0000 7f169944e640 1 -- 172.21.15.29:0/372938396 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16840317c0 con 0x7f16940cf5f0 2023-10-14T16:53:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.560+0000 7f169944e640 1 -- 172.21.15.29:0/372938396 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1684031920 con 0x7f16940cf5f0 2023-10-14T16:53:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.560+0000 7f169944e640 1 -- 172.21.15.29:0/372938396 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1684031cb0 con 0x7f16940cf5f0 2023-10-14T16:53:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 -- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16940cf5f0 msgr2=0x7f169406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16940cf5f0 0x7f169406d3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f1684009870 tx=0x7f168402b860 comp rx=0 tx=0).stop 2023-10-14T16:53:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 -- 172.21.15.29:0/372938396 shutdown_connections 2023-10-14T16:53:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1694070c10 0x7f16940cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 --2- 172.21.15.29:0/372938396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16940cf5f0 0x7f169406d3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 -- 172.21.15.29:0/372938396 >> 172.21.15.29:0/372938396 conn(0x7f16940cfac0 msgr2=0x7f16940cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 -- 172.21.15.29:0/372938396 shutdown_connections 2023-10-14T16:53:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.561+0000 7f169c6b4640 1 -- 172.21.15.29:0/372938396 wait complete. 2023-10-14T16:53:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.563+0000 7f169c6b4640 1 Processor -- start 2023-10-14T16:53:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.563+0000 7f169c6b4640 1 -- start start 2023-10-14T16:53:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1694070c10 0x7f16940c8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16940cf5f0 0x7f16940c90e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16940c52a0 con 0x7f1694070830 2023-10-14T16:53:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16941f28a0 con 0x7f1694070c10 2023-10-14T16:53:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169c6b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16940c4fc0 con 0x7f16940cf5f0 2023-10-14T16:53:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169a450640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169a450640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56724/0 (socket says 172.21.15.29:56724) 2023-10-14T16:53:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.564+0000 7f169a450640 1 -- 172.21.15.29:0/2389877801 learned_addr learned my addr 172.21.15.29:0/2389877801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f1699c4f640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1694070c10 0x7f16940c8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169ac51640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16940cf5f0 0x7f16940c90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16940cf5f0 msgr2=0x7f16940c90e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16940cf5f0 0x7f16940c90e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1694070c10 msgr2=0x7f16940c8ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1694070c10 0x7f16940c8ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 -- 172.21.15.29:0/2389877801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16840094d0 con 0x7f1694070830 2023-10-14T16:53:54.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169a450640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f169000c460 tx=0x7f169000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690007480 con 0x7f1694070830 2023-10-14T16:53:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690007600 con 0x7f1694070830 2023-10-14T16:53:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690004a20 con 0x7f1694070830 2023-10-14T16:53:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.565+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1694150820 con 0x7f1694070830 2023-10-14T16:53:54.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.566+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1694150de0 con 0x7f1694070830 2023-10-14T16:53:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.567+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f169404e160 con 0x7f1694070830 2023-10-14T16:53:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.568+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f1690004b80 con 0x7f1694070830 2023-10-14T16:53:54.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.568+0000 7f168affd640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.569+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f1690088d10 con 0x7f1694070830 2023-10-14T16:53:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.569+0000 7f1699c4f640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 msgr2=0x7f167806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.569+0000 7f1699c4f640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:54.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.573+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f16900538d0 con 0x7f1694070830 2023-10-14T16:53:54.576 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.576+0000 7fc550830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:53:54.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.579+0000 7fcf87c3a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:53:54.633 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.633+0000 7fcf87c3a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:53:54.684 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.684+0000 7fc550830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:53:54.749 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.748+0000 7fcf87c3a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:53:54.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.770+0000 7f1699c4f640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 msgr2=0x7f167806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:54.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.770+0000 7f1699c4f640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:54.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.781+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f169415de20 con 0x7f1694070830 2023-10-14T16:53:54.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.782+0000 7f168affd640 1 -- 172.21.15.29:0/2389877801 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 6 entries v48) v1 ==== 80+0+354 (secure 0 0 0) 0x7f1690018080 con 0x7f1694070830 2023-10-14T16:53:54.783 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1900972994 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.783 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3481939635 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.783 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4149980298 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.783 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2383868706 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.784 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/3113675785 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.784 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/3113675785 2023-10-15T16:41:58.134928+0000 2023-10-14T16:53:54.784 INFO:teuthology.orchestra.run.smithi029.stderr:listed 6 entries 2023-10-14T16:53:54.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 msgr2=0x7f167806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 msgr2=0x7f16941f2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f169000c460 tx=0x7f169000c920 comp rx=0 tx=0).stop 2023-10-14T16:53:54.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 shutdown_connections 2023-10-14T16:53:54.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16940cf5f0 0x7f16940c90e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f167806b020 0x7f167806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1694070c10 0x7f16940c8ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 --2- 172.21.15.29:0/2389877801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1694070830 0x7f16941f2360 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:54.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 >> 172.21.15.29:0/2389877801 conn(0x7f16940cfac0 msgr2=0x7f16940ad4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:54.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 shutdown_connections 2023-10-14T16:53:54.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:54.785+0000 7f169c6b4640 1 -- 172.21.15.29:0/2389877801 wait complete. 2023-10-14T16:53:54.804 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:53:54.805 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:54.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T16:53:54.843 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.843+0000 7fc550830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:53:54.867 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:54.867+0000 7fcf87c3a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:53:54.893 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:54.893+0000 7fc550830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:53:54.903 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:53:54.904 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:53:54.904 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:53:54.956 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:53:54.957 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:53:55.012 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:53:55.012 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:53:55.013 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:53:55.018 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:55.019+0000 7fc550830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:53:55.041 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:55.040+0000 7fcf87c3a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:53:55.067 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:53:55.068 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:53:55.068 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:53:55.094 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:55.093+0000 7fcf87c3a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:53:55.122 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:53:55.123 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:53:55.222 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:55.221+0000 7fcf87c3a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:53:55.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.340+0000 7f7988075640 1 Processor -- start 2023-10-14T16:53:55.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.340+0000 7f7988075640 1 -- start start 2023-10-14T16:53:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.341+0000 7f7988075640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 0x7f79800cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.341+0000 7f7988075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79800d1b10 0x7f79800cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7988075640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79800b3940 con 0x7f79800b3e70 2023-10-14T16:53:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7988075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79800b45e0 con 0x7f79800d1b10 2023-10-14T16:53:55.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7988075640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79800b3b90 con 0x7f79800b3070 2023-10-14T16:53:55.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7986612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79800d1b10 0x7f79800cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7986612640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f79800d1b10 0x7f79800cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56734/0 (socket says 172.21.15.29:56734) 2023-10-14T16:53:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7986612640 1 -- 172.21.15.29:0/2332133004 learned_addr learned my addr 172.21.15.29:0/2332133004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.342+0000 7f7985e11640 1 --2- 172.21.15.29:0/2332133004 >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 0x7f79800cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.343+0000 7f7986612640 1 -- 172.21.15.29:0/2332133004 >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 msgr2=0x7f79800cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:55.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.343+0000 7f7986612640 1 --2- 172.21.15.29:0/2332133004 >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 0x7f79800cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.343+0000 7f7986612640 1 -- 172.21.15.29:0/2332133004 >> v1:172.21.15.186:6789/0 conn(0x7f79800b3e70 legacy=0x7f79800c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:55.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.343+0000 7f7986612640 1 -- 172.21.15.29:0/2332133004 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79800d3840 con 0x7f79800d1b10 2023-10-14T16:53:55.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.343+0000 7f7985e11640 1 --2- 172.21.15.29:0/2332133004 >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 0x7f79800cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:55.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.344+0000 7f7986612640 1 --2- 172.21.15.29:0/2332133004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800cfa40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f798008ba60 tx=0x7f797802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ea978f0f5ea166 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:55.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.344+0000 7f7984e0f640 1 -- 172.21.15.29:0/2332133004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797802ca80 con 0x7f79800d1b10 2023-10-14T16:53:55.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.344+0000 7f7984e0f640 1 -- 172.21.15.29:0/2332133004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797802cbe0 con 0x7f79800d1b10 2023-10-14T16:53:55.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.344+0000 7f7984e0f640 1 -- 172.21.15.29:0/2332133004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79780343f0 con 0x7f79800d1b10 2023-10-14T16:53:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 -- 172.21.15.29:0/2332133004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 msgr2=0x7f79800cfa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 --2- 172.21.15.29:0/2332133004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800cfa40 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f798008ba60 tx=0x7f797802bee0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 -- 172.21.15.29:0/2332133004 shutdown_connections 2023-10-14T16:53:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 --2- 172.21.15.29:0/2332133004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800cfa40 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 --2- 172.21.15.29:0/2332133004 >> v2:172.21.15.186:3300/0 conn(0x7f79800b3070 0x7f79800cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 -- 172.21.15.29:0/2332133004 >> 172.21.15.29:0/2332133004 conn(0x7f79800d2d20 msgr2=0x7f79800d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 -- 172.21.15.29:0/2332133004 shutdown_connections 2023-10-14T16:53:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.345+0000 7f7988075640 1 -- 172.21.15.29:0/2332133004 wait complete. 2023-10-14T16:53:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.347+0000 7f7988075640 1 Processor -- start 2023-10-14T16:53:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.347+0000 7f7988075640 1 -- start start 2023-10-14T16:53:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79800b3070 0x7f798015c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f79800b3e70 0x7f7980150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7980150760 con 0x7f79800d1b10 2023-10-14T16:53:55.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79800b0300 con 0x7f79800b3070 2023-10-14T16:53:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7988075640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79800c9e90 con 0x7f79800b3e70 2023-10-14T16:53:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56738/0 (socket says 172.21.15.29:56738) 2023-10-14T16:53:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 -- 172.21.15.29:0/3387509796 learned_addr learned my addr 172.21.15.29:0/3387509796 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:55.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7985e11640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79800b3070 0x7f798015c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f79800b3e70 msgr2=0x7f7980150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f79800b3e70 0x7f7980150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79800b3070 msgr2=0x7f798015c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.348+0000 7f7986612640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79800b3070 0x7f798015c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f7986612640 1 -- 172.21.15.29:0/3387509796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7978009b30 con 0x7f79800d1b10 2023-10-14T16:53:55.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f7986612640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7978009f40 tx=0x7f7978004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:55.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7978009510 con 0x7f79800d1b10 2023-10-14T16:53:55.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f797803e450 con 0x7f79800d1b10 2023-10-14T16:53:55.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f797803e800 con 0x7f79800d1b10 2023-10-14T16:53:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79800ca170 con 0x7f79800d1b10 2023-10-14T16:53:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.349+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f798015ecd0 con 0x7f79800d1b10 2023-10-14T16:53:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.350+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79800cd0e0 con 0x7f79800d1b10 2023-10-14T16:53:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.351+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f797802cd50 con 0x7f79800d1b10 2023-10-14T16:53:55.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.352+0000 7f79767fc640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 0x7f796406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.352+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f79780ac940 con 0x7f79800d1b10 2023-10-14T16:53:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.352+0000 7f7985e11640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 msgr2=0x7f796406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.353+0000 7f7985e11640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 0x7f796406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:55.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.357+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f79780774f0 con 0x7f79800d1b10 2023-10-14T16:53:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.554+0000 7f7985e11640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 msgr2=0x7f796406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:55.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.554+0000 7f7985e11640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 0x7f796406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.592+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79800ca630 con 0x7f79800d1b10 2023-10-14T16:53:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.592+0000 7f79767fc640 1 -- 172.21.15.29:0/3387509796 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f797807ab10 con 0x7f79800d1b10 2023-10-14T16:53:55.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:55.594 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:53:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:53:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 msgr2=0x7f796406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 0x7f796406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 msgr2=0x7f79800c7ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f7978009f40 tx=0x7f7978004a40 comp rx=0 tx=0).stop 2023-10-14T16:53:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 shutdown_connections 2023-10-14T16:53:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f79800d1b10 0x7f79800c7ab0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f79800b3e70 0x7f7980150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f796406b020 0x7f796406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 --2- 172.21.15.29:0/3387509796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79800b3070 0x7f798015c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.595+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 >> 172.21.15.29:0/3387509796 conn(0x7f79800d2d20 msgr2=0x7f79801605d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.596+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 shutdown_connections 2023-10-14T16:53:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.596+0000 7f7988075640 1 -- 172.21.15.29:0/3387509796 wait complete. 2023-10-14T16:53:55.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:53:55.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.831+0000 7fafff0d0640 1 Processor -- start 2023-10-14T16:53:55.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.831+0000 7fafff0d0640 1 -- start start 2023-10-14T16:53:55.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7fafff0d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff806b8c0 0x7faff80cfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7fafff0d0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff806bca0 0x7faff80b0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7fafff0d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faff80cf940 con 0x7faff80b09c0 2023-10-14T16:53:55.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7fafff0d0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faff80ce990 con 0x7faff806b8c0 2023-10-14T16:53:55.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7fafff0d0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff80d0310 con 0x7faff806bca0 2023-10-14T16:53:55.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff806b8c0 0x7faff80cfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faff806b8c0 0x7faff80cfdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56744/0 (socket says 172.21.15.29:56744) 2023-10-14T16:53:55.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 -- 172.21.15.29:0/133967292 learned_addr learned my addr 172.21.15.29:0/133967292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:55.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 -- 172.21.15.29:0/133967292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff806bca0 msgr2=0x7faff80b0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:55.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faff7fff640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff806bca0 0x7faff80b0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff806bca0 0x7faff80b0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.833+0000 7faffce6c640 1 -- 172.21.15.29:0/133967292 >> v1:172.21.15.186:6789/0 conn(0x7faff80b09c0 legacy=0x7faff8160a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:55.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.834+0000 7faffce6c640 1 -- 172.21.15.29:0/133967292 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faff815e290 con 0x7faff806b8c0 2023-10-14T16:53:55.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.835+0000 7faffce6c640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff80cfdd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fafec009850 tx=0x7fafec02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8843a72002160691 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:55.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.835+0000 7faff77fe640 1 -- 172.21.15.29:0/133967292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec004230 con 0x7faff806b8c0 2023-10-14T16:53:55.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.835+0000 7faff77fe640 1 -- 172.21.15.29:0/133967292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec004390 con 0x7faff806b8c0 2023-10-14T16:53:55.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.835+0000 7faff77fe640 1 -- 172.21.15.29:0/133967292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec031830 con 0x7faff806b8c0 2023-10-14T16:53:55.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 -- 172.21.15.29:0/133967292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 msgr2=0x7faff80cfdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff80cfdd0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7fafec009850 tx=0x7fafec02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 -- 172.21.15.29:0/133967292 shutdown_connections 2023-10-14T16:53:55.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff806bca0 0x7faff80b0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 --2- 172.21.15.29:0/133967292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff80cfdd0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 -- 172.21.15.29:0/133967292 >> 172.21.15.29:0/133967292 conn(0x7faff8099ea0 msgr2=0x7faff80d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 -- 172.21.15.29:0/133967292 shutdown_connections 2023-10-14T16:53:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.836+0000 7fafff0d0640 1 -- 172.21.15.29:0/133967292 wait complete. 2023-10-14T16:53:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.838+0000 7fafff0d0640 1 Processor -- start 2023-10-14T16:53:55.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.838+0000 7fafff0d0640 1 -- start start 2023-10-14T16:53:55.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 0x7faff80ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff80b09c0 0x7faff80aec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff80b6aa0 con 0x7faff806b8c0 2023-10-14T16:53:55.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faff80c5970 con 0x7faff806bca0 2023-10-14T16:53:55.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7fafff0d0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faff81f6850 con 0x7faff80b09c0 2023-10-14T16:53:55.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56748/0 (socket says 172.21.15.29:56748) 2023-10-14T16:53:55.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 -- 172.21.15.29:0/122899399 learned_addr learned my addr 172.21.15.29:0/122899399 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:55.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faff7fff640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 0x7faff80ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffd66d640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff80b09c0 0x7faff80aec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:55.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff80b09c0 msgr2=0x7faff80aec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff80b09c0 0x7faff80aec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.839+0000 7faffce6c640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 msgr2=0x7faff80ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:55.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faffce6c640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 0x7faff80ae6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:55.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faffce6c640 1 -- 172.21.15.29:0/122899399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafec009460 con 0x7faff806b8c0 2023-10-14T16:53:55.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faff7fff640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 0x7faff80ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faffce6c640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fafec000c00 tx=0x7fafec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec031830 con 0x7faff806b8c0 2023-10-14T16:53:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafec03f070 con 0x7faff806b8c0 2023-10-14T16:53:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faff81f6b30 con 0x7faff806b8c0 2023-10-14T16:53:55.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafec031bd0 con 0x7faff806b8c0 2023-10-14T16:53:55.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.840+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faff81f70f0 con 0x7faff806b8c0 2023-10-14T16:53:55.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.842+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faff80c5ae0 con 0x7faff806b8c0 2023-10-14T16:53:55.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.843+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7fafec041030 con 0x7faff806b8c0 2023-10-14T16:53:55.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.844+0000 7faff57fa640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 0x7fafdc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:55.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.844+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7fafec0abf50 con 0x7faff806b8c0 2023-10-14T16:53:55.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.844+0000 7faff7fff640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 msgr2=0x7fafdc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:55.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.844+0000 7faff7fff640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 0x7fafdc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:55.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:55.849+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafec076a60 con 0x7faff806b8c0 2023-10-14T16:53:55.862 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:55.862+0000 7fc550830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:53:56.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.045+0000 7faff7fff640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 msgr2=0x7fafdc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.045+0000 7faff7fff640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 0x7fafdc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:56.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.053+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faff80c5ae0 con 0x7faff806b8c0 2023-10-14T16:53:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.055+0000 7faff57fa640 1 -- 172.21.15.29:0/122899399 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) v1 ==== 74+0+17988 (secure 0 0 0) 0x7fafec07a080 con 0x7faff806b8c0 2023-10-14T16:53:56.056 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:56.057 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":48,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:53:52.019590+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":222,"snap_epoch":48,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1900972994":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/3481939635":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/4149980298":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2383868706":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6832/3113675785":"2023-10-15T16:41:58.134928+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":2},{"begin":10,"length":2},{"begin":13,"length":1},{"begin":15,"length":3},{"begin":22,"length":3},{"begin":28,"length":1},{"begin":31,"length":3},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":45,"length":3},{"begin":49,"length":2},{"begin":54,"length":1},{"begin":58,"length":3},{"begin":62,"length":1},{"begin":64,"length":1},{"begin":67,"length":3},{"begin":71,"length":1},{"begin":73,"length":1},{"begin":76,"length":3},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":88,"length":1},{"begin":91,"length":2},{"begin":94,"length":2},{"begin":97,"length":1},{"begin":100,"length":2},{"begin":104,"length":2},{"begin":108,"length":2},{"begin":111,"length":1},{"begin":168,"length":55}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":15,"length":1},{"begin":28,"length":1},{"begin":39,"length":1},{"begin":49,"length":1},{"begin":58,"length":1},{"begin":68,"length":1},{"begin":78,"length":1},{"begin":88,"length":1},{"begin":100,"length":1},{"begin":214,"length":9}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:53:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.057+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 msgr2=0x7fafdc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.057+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 0x7fafdc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.057+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 msgr2=0x7faff8150e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.057+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7fafec000c00 tx=0x7fafec0046c0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.057+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 shutdown_connections 2023-10-14T16:53:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faff80b09c0 0x7faff80aec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7fafdc06b020 0x7fafdc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faff806bca0 0x7faff80ae6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 --2- 172.21.15.29:0/122899399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faff806b8c0 0x7faff8150e40 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 >> 172.21.15.29:0/122899399 conn(0x7faff8099ea0 msgr2=0x7faff815af40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:56.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 shutdown_connections 2023-10-14T16:53:56.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.058+0000 7fafff0d0640 1 -- 172.21.15.29:0/122899399 wait complete. 2023-10-14T16:53:56.078 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:41:36.429166+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:41:37.759317+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 222, 'snap_epoch': 48, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:53:56.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:53:56.086 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.081+0000 7fcf87c3a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:53:56.162 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:56.162+0000 7fc550830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:53:56.220 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:56.220+0000 7fc550830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:53:56.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.292+0000 7f8070221640 1 Processor -- start 2023-10-14T16:53:56.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.292+0000 7f8070221640 1 -- start start 2023-10-14T16:53:56.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f8070221640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80681618b0 0x7f8068065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f8070221640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8068065ff0 0x7f80680ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f8070221640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8068066420 con 0x7f806815d330 2023-10-14T16:53:56.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f8070221640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806806b1e0 con 0x7f8068065ff0 2023-10-14T16:53:56.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f8070221640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806806cee0 con 0x7f80681618b0 2023-10-14T16:53:56.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.294+0000 7f806e7be640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8068065ff0 0x7f80680ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8068065ff0 0x7f80680ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56750/0 (socket says 172.21.15.29:56750) 2023-10-14T16:53:56.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 -- 172.21.15.29:0/3607120352 learned_addr learned my addr 172.21.15.29:0/3607120352 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:56.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806d7bc640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80681618b0 0x7f8068065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 -- 172.21.15.29:0/3607120352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80681618b0 msgr2=0x7f8068065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:56.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80681618b0 0x7f8068065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 -- 172.21.15.29:0/3607120352 >> v1:172.21.15.186:6789/0 conn(0x7f806815d330 legacy=0x7f806806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.295+0000 7f806e7be640 1 -- 172.21.15.29:0/3607120352 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80680d0890 con 0x7f8068065ff0 2023-10-14T16:53:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.297+0000 7f806e7be640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680ce9b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8060009e50 tx=0x7f806002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dcd0f3a2c12cda3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.297+0000 7f806cfbb640 1 -- 172.21.15.29:0/3607120352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f806002c9e0 con 0x7f8068065ff0 2023-10-14T16:53:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.297+0000 7f806cfbb640 1 -- 172.21.15.29:0/3607120352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806002cb40 con 0x7f8068065ff0 2023-10-14T16:53:56.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.297+0000 7f806cfbb640 1 -- 172.21.15.29:0/3607120352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80600343f0 con 0x7f8068065ff0 2023-10-14T16:53:56.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 -- 172.21.15.29:0/3607120352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 msgr2=0x7f80680ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680ce9b0 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f8060009e50 tx=0x7f806002be40 comp rx=0 tx=0).stop 2023-10-14T16:53:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 -- 172.21.15.29:0/3607120352 shutdown_connections 2023-10-14T16:53:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680ce9b0 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 --2- 172.21.15.29:0/3607120352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80681618b0 0x7f8068065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 -- 172.21.15.29:0/3607120352 >> 172.21.15.29:0/3607120352 conn(0x7f80680d0ab0 msgr2=0x7f80680d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 -- 172.21.15.29:0/3607120352 shutdown_connections 2023-10-14T16:53:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.298+0000 7f8070221640 1 -- 172.21.15.29:0/3607120352 wait complete. 2023-10-14T16:53:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.300+0000 7f8070221640 1 Processor -- start 2023-10-14T16:53:56.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.300+0000 7f8070221640 1 -- start start 2023-10-14T16:53:56.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 0x7f806815ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80681618b0 0x7f8068159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80680ca0b0 con 0x7f8068065ff0 2023-10-14T16:53:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80680b2430 con 0x7f80681618b0 2023-10-14T16:53:56.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.301+0000 7f8070221640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80680b2710 con 0x7f806815d330 2023-10-14T16:53:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56760/0 (socket says 172.21.15.29:56760) 2023-10-14T16:53:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 -- 172.21.15.29:0/1078908040 learned_addr learned my addr 172.21.15.29:0/1078908040 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:56.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 msgr2=0x7f806815ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806d7bc640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 0x7f806815ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806e7be640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80681618b0 0x7f8068159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 0x7f806815ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80681618b0 msgr2=0x7f8068159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80681618b0 0x7f8068159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806dfbd640 1 -- 172.21.15.29:0/1078908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8060009b30 con 0x7f8068065ff0 2023-10-14T16:53:56.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.302+0000 7f806d7bc640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 0x7f806815ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f806dfbd640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f806400b3f0 tx=0x7f806400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:56.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064011830 con 0x7f8068065ff0 2023-10-14T16:53:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8064011990 con 0x7f8068065ff0 2023-10-14T16:53:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80680b29f0 con 0x7f8068065ff0 2023-10-14T16:53:56.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064011d40 con 0x7f8068065ff0 2023-10-14T16:53:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.303+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80680b2fb0 con 0x7f8068065ff0 2023-10-14T16:53:56.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.304+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80680b31d0 con 0x7f8068065ff0 2023-10-14T16:53:56.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.305+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 14) v1 ==== 89400+0+0 (secure 0 0 0) 0x7f806401a400 con 0x7f8068065ff0 2023-10-14T16:53:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.306+0000 7f805e7fc640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 0x7f804c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.306+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6647+0+0 (secure 0 0 0) 0x7f8064089b90 con 0x7f8068065ff0 2023-10-14T16:53:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.306+0000 7f806d7bc640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 msgr2=0x7f804c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.306+0000 7f806d7bc640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 0x7f804c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:53:56.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.311+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8064054570 con 0x7f8068065ff0 2023-10-14T16:53:56.335 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:53:56.335+0000 7fc550830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:53:56.404 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.403+0000 7fcf87c3a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:53:56.466 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.465+0000 7fcf87c3a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:53:56.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.507+0000 7f806d7bc640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 msgr2=0x7f804c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/115188615 2023-10-14T16:53:56.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.507+0000 7f806d7bc640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 0x7f804c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:53:56.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.508+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8068161080 con 0x7f8068065ff0 2023-10-14T16:53:56.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.508+0000 7f805e7fc640 1 -- 172.21.15.29:0/1078908040 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80640150a0 con 0x7f8068065ff0 2023-10-14T16:53:56.509 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:53:56.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 msgr2=0x7f804c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:53:56.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 0x7f804c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 msgr2=0x7f80680b03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f806400b3f0 tx=0x7f806400b8b0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 shutdown_connections 2023-10-14T16:53:56.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:6832/115188615,v1:172.21.15.29:6833/115188615] conn(0x7f804c06b020 0x7f804c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80681618b0 0x7f8068159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806815d330 0x7f806815ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 --2- 172.21.15.29:0/1078908040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8068065ff0 0x7f80680b03b0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 >> 172.21.15.29:0/1078908040 conn(0x7f80680d0ab0 msgr2=0x7f80680c6540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:56.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 shutdown_connections 2023-10-14T16:53:56.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.511+0000 7f8070221640 1 -- 172.21.15.29:0/1078908040 wait complete. 2023-10-14T16:53:56.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:53:56.590 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.590+0000 7fcf87c3a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:53:56.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.670+0000 7fcf0bf35640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.670+0000 7fcf0bf35640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.670+0000 7fcf0bf35640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.670+0000 7fcf0bf35640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.670+0000 7fcf0bf35640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.671+0000 7fcf0872e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.671+0000 7fcf0872e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.671+0000 7fcf0872e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.671+0000 7fcf0872e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.674 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:53:56.671+0000 7fcf0872e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:53:56.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.730+0000 7fa86e565640 1 Processor -- start 2023-10-14T16:53:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.730+0000 7fa86e565640 1 -- start start 2023-10-14T16:53:56.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.731+0000 7fa86e565640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.731+0000 7fa86e565640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa8680c9ca0 0x7fa8680ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.731+0000 7fa86e565640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8680cb030 con 0x7fa86815ad70 2023-10-14T16:53:56.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.731+0000 7fa86e565640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8680cae30 con 0x7fa8680c9ca0 2023-10-14T16:53:56.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.731+0000 7fa86e565640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8680c91c0 con 0x7fa86806a4b0 2023-10-14T16:53:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.732+0000 7fa8677fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.732+0000 7fa8677fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:49530/0 (socket says 172.21.15.29:49530) 2023-10-14T16:53:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.732+0000 7fa86cb02640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa8680c9ca0 0x7fa8680ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.732+0000 7fa8677fe640 1 -- 172.21.15.29:0/2372298424 learned_addr learned my addr 172.21.15.29:0/2372298424 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.733+0000 7fa8677fe640 1 -- 172.21.15.29:0/2372298424 >> v2:172.21.15.186:3300/0 conn(0x7fa8680c9ca0 msgr2=0x7fa8680ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.733+0000 7fa8677fe640 1 --2- 172.21.15.29:0/2372298424 >> v2:172.21.15.186:3300/0 conn(0x7fa8680c9ca0 0x7fa8680ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.733+0000 7fa8677fe640 1 -- 172.21.15.29:0/2372298424 >> v1:172.21.15.186:6789/0 conn(0x7fa86815ad70 legacy=0x7fa868069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.733+0000 7fa8677fe640 1 -- 172.21.15.29:0/2372298424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8680c94a0 con 0x7fa86806a4b0 2023-10-14T16:53:56.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.734+0000 7fa8677fe640 1 --2- 172.21.15.29:0/2372298424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009870 tx=0x7fa85c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c487049ce666c4d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.734+0000 7fa866ffd640 1 -- 172.21.15.29:0/2372298424 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c030840 con 0x7fa86806a4b0 2023-10-14T16:53:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.734+0000 7fa866ffd640 1 -- 172.21.15.29:0/2372298424 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa85c0309a0 con 0x7fa86806a4b0 2023-10-14T16:53:56.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.734+0000 7fa866ffd640 1 -- 172.21.15.29:0/2372298424 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa85c030d30 con 0x7fa86806a4b0 2023-10-14T16:53:56.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 -- 172.21.15.29:0/2372298424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 msgr2=0x7fa8680ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 --2- 172.21.15.29:0/2372298424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa85c009870 tx=0x7fa85c02b980 comp rx=0 tx=0).stop 2023-10-14T16:53:56.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 -- 172.21.15.29:0/2372298424 shutdown_connections 2023-10-14T16:53:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 --2- 172.21.15.29:0/2372298424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa86806a4b0 0x7fa8680ca3b0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 --2- 172.21.15.29:0/2372298424 >> v2:172.21.15.186:3300/0 conn(0x7fa8680c9ca0 0x7fa8680ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 -- 172.21.15.29:0/2372298424 >> 172.21.15.29:0/2372298424 conn(0x7fa8680ce9b0 msgr2=0x7fa8680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 -- 172.21.15.29:0/2372298424 shutdown_connections 2023-10-14T16:53:56.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.735+0000 7fa86e565640 1 -- 172.21.15.29:0/2372298424 wait complete. 2023-10-14T16:53:56.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.737+0000 7fa86e565640 1 Processor -- start 2023-10-14T16:53:56.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.737+0000 7fa86e565640 1 -- start start 2023-10-14T16:53:56.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8680c9ca0 0x7fa8680cde60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa86815ad70 0x7fa8680c7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8680cd570 con 0x7fa86806a4b0 2023-10-14T16:53:56.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8680cc9f0 con 0x7fa86815ad70 2023-10-14T16:53:56.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa86e565640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8680cccd0 con 0x7fa8680c9ca0 2023-10-14T16:53:56.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa867fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa867fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56862/0 (socket says 172.21.15.29:56862) 2023-10-14T16:53:56.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.738+0000 7fa867fff640 1 -- 172.21.15.29:0/925168252 learned_addr learned my addr 172.21.15.29:0/925168252 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa8677fe640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8680c9ca0 0x7fa8680cde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa86cb02640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa86815ad70 0x7fa8680c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:56.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 -- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8680c9ca0 msgr2=0x7fa8680cde60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8680c9ca0 0x7fa8680cde60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 -- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa86815ad70 msgr2=0x7fa8680c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa86815ad70 0x7fa8680c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 -- 172.21.15.29:0/925168252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0094d0 con 0x7fa86806a4b0 2023-10-14T16:53:56.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa867fff640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c400 tx=0x7fa85800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:56.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8580052d0 con 0x7fa86806a4b0 2023-10-14T16:53:56.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa858005430 con 0x7fa86806a4b0 2023-10-14T16:53:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8680ccfb0 con 0x7fa86806a4b0 2023-10-14T16:53:56.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa858003830 con 0x7fa86806a4b0 2023-10-14T16:53:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.739+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa86815a2f0 con 0x7fa86806a4b0 2023-10-14T16:53:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.740+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa86804e160 con 0x7fa86806a4b0 2023-10-14T16:53:56.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.742+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa858003990 con 0x7fa86806a4b0 2023-10-14T16:53:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.743+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6862+0+0 (secure 0 0 0) 0x7fa858088b70 con 0x7fa86806a4b0 2023-10-14T16:53:56.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.747+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa858053b00 con 0x7fa86806a4b0 2023-10-14T16:53:56.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.940+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fa8680700e0 con 0x7fa86806a4b0 2023-10-14T16:53:56.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.941+0000 7fa864ff9640 1 -- 172.21.15.29:0/925168252 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v49) v1 ==== 104+0+11 (secure 0 0 0) 0x7fa858057120 con 0x7fa86806a4b0 2023-10-14T16:53:56.943 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:53:56.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 msgr2=0x7fa868159db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fa85800c400 tx=0x7fa85800c8c0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 shutdown_connections 2023-10-14T16:53:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa86815ad70 0x7fa8680c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8680c9ca0 0x7fa8680cde60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 --2- 172.21.15.29:0/925168252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa86806a4b0 0x7fa868159db0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 >> 172.21.15.29:0/925168252 conn(0x7fa8680ce9b0 msgr2=0x7fa86806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 shutdown_connections 2023-10-14T16:53:56.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:56.944+0000 7fa86e565640 1 -- 172.21.15.29:0/925168252 wait complete. 2023-10-14T16:53:56.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:53:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.191+0000 7f1fef8c9640 1 Processor -- start 2023-10-14T16:53:57.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.192+0000 7f1fef8c9640 1 -- start start 2023-10-14T16:53:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.193+0000 7f1fef8c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1fe815ad70 0x7f1fe8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.193+0000 7f1fef8c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fe80c9ca0 0x7f1fe806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.193+0000 7f1fef8c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fe80cb030 con 0x7f1fe80ca3b0 2023-10-14T16:53:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.193+0000 7f1fef8c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80c91c0 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.193+0000 7f1fef8c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fe80c94a0 con 0x7f1fe815ad70 2023-10-14T16:53:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fede66640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1fe80ca3b0 0x7f1fe80ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40846/0 (socket says 172.21.15.29:40846) 2023-10-14T16:53:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fece64640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fe80c9ca0 0x7f1fe806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fede66640 1 -- 172.21.15.29:0/3969617302 learned_addr learned my addr 172.21.15.29:0/3969617302 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fed665640 1 --2- 172.21.15.29:0/3969617302 >> v2:172.21.15.186:3300/0 conn(0x7f1fe815ad70 0x7f1fe8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fece64640 1 -- 172.21.15.29:0/3969617302 >> v2:172.21.15.186:3300/0 conn(0x7f1fe815ad70 msgr2=0x7f1fe8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fece64640 1 --2- 172.21.15.29:0/3969617302 >> v2:172.21.15.186:3300/0 conn(0x7f1fe815ad70 0x7f1fe8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fece64640 1 -- 172.21.15.29:0/3969617302 >> v1:172.21.15.29:6789/0 conn(0x7f1fe80ca3b0 legacy=0x7f1fe80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fece64640 1 -- 172.21.15.29:0/3969617302 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fe80c9780 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.194+0000 7f1fdbfff640 1 -- 172.21.15.29:0/3969617302 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163663043 0 0) 0x7f1fe80cb030 con 0x7f1fe80ca3b0 2023-10-14T16:53:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.195+0000 7f1fece64640 1 --2- 172.21.15.29:0/3969617302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe806a2e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80cf780 tx=0x7f1fdc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bd30d682af9322 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:57.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fdbfff640 1 -- 172.21.15.29:0/3969617302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc031830 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fdbfff640 1 -- 172.21.15.29:0/3969617302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fdc031990 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fdbfff640 1 -- 172.21.15.29:0/3969617302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fdc031d20 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 -- 172.21.15.29:0/3969617302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 msgr2=0x7f1fe806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/3969617302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe806a2e0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f1fe80cf780 tx=0x7f1fdc02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 -- 172.21.15.29:0/3969617302 shutdown_connections 2023-10-14T16:53:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/3969617302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe806a2e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/3969617302 >> v2:172.21.15.186:3300/0 conn(0x7f1fe815ad70 0x7f1fe8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.196+0000 7f1fef8c9640 1 -- 172.21.15.29:0/3969617302 >> 172.21.15.29:0/3969617302 conn(0x7f1fe80ce9b0 msgr2=0x7f1fe80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.197+0000 7f1fef8c9640 1 -- 172.21.15.29:0/3969617302 shutdown_connections 2023-10-14T16:53:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.197+0000 7f1fef8c9640 1 -- 172.21.15.29:0/3969617302 wait complete. 2023-10-14T16:53:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.198+0000 7f1fef8c9640 1 Processor -- start 2023-10-14T16:53:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.199+0000 7f1fef8c9640 1 -- start start 2023-10-14T16:53:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.199+0000 7f1fef8c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.199+0000 7f1fef8c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 0x7f1fe814b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fef8c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 0x7f1fe80cd590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fef8c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe814b350 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fef8c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe815b830 con 0x7f1fe80ca3b0 2023-10-14T16:53:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fef8c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fe815bb10 con 0x7f1fe815ad70 2023-10-14T16:53:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56894/0 (socket says 172.21.15.29:56894) 2023-10-14T16:53:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 -- 172.21.15.29:0/2532852309 learned_addr learned my addr 172.21.15.29:0/2532852309 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fece64640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 0x7f1fe814b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fede66640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 0x7f1fe80cd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 -- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 msgr2=0x7f1fe80cd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 0x7f1fe80cd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 -- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 msgr2=0x7f1fe814b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 0x7f1fe814b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fed665640 1 -- 172.21.15.29:0/2532852309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fdc009400 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fede66640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 0x7f1fe80cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.200+0000 7f1fece64640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 0x7f1fe814b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fed665640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d5f0 tx=0x7f1fe400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe40125f0 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fe4012750 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fe4012b00 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fe80aeb60 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.201+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fe814bda0 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.203+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fe804e160 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.203+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 15) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1fe4012c60 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.204+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 6862+0+0 (secure 0 0 0) 0x7f1fe4089ae0 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.209+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1fe4054a70 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.407+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1fe80b25c0 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.408+0000 7f1fd9ffb640 1 -- 172.21.15.29:0/2532852309 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v49) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1fe4058090 con 0x7f1fe80c9ca0 2023-10-14T16:53:57.409 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:53:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 msgr2=0x7f1fe814ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f1fe400d5f0 tx=0x7f1fe400dab0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 shutdown_connections 2023-10-14T16:53:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fe815ad70 0x7f1fe80cd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fe80ca3b0 0x7f1fe814b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 --2- 172.21.15.29:0/2532852309 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fe80c9ca0 0x7f1fe814ae10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 >> 172.21.15.29:0/2532852309 conn(0x7f1fe80ce9b0 msgr2=0x7f1fe80aca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 shutdown_connections 2023-10-14T16:53:57.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.413+0000 7f1fef8c9640 1 -- 172.21.15.29:0/2532852309 wait complete. 2023-10-14T16:53:57.432 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:53:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.648+0000 7f0764e61640 1 Processor -- start 2023-10-14T16:53:57.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.648+0000 7f0764e61640 1 -- start start 2023-10-14T16:53:57.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f0764e61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 0x7f07601615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f0764e61640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f076015db00 0x7f07600b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f0764e61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07600b0db0 con 0x7f07600b0410 2023-10-14T16:53:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f0764e61640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f076015e3e0 con 0x7f076015db00 2023-10-14T16:53:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f0764e61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07600cd4a0 con 0x7f07600cd0c0 2023-10-14T16:53:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f075dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f07600b0410 0x7f076015d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40862/0 (socket says 172.21.15.29:40862) 2023-10-14T16:53:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.650+0000 7f075dd9b640 1 -- 172.21.15.29:0/1530748675 learned_addr learned my addr 172.21.15.29:0/1530748675 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:57.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075e59c640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 0x7f07601615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542296792 0 0) 0x7f07600b0db0 con 0x7f07600b0410 2023-10-14T16:53:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0748003520 con 0x7f07600b0410 2023-10-14T16:53:57.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075ed9d640 1 --2- 172.21.15.29:0/1530748675 >> v2:172.21.15.186:3300/0 conn(0x7f076015db00 0x7f07600b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3767142560 0 0) 0x7f0748003520 con 0x7f07600b0410 2023-10-14T16:53:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 msgr2=0x7f07601615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 0x7f07601615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 >> v2:172.21.15.186:3300/0 conn(0x7f076015db00 msgr2=0x7f07600b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 --2- 172.21.15.29:0/1530748675 >> v2:172.21.15.186:3300/0 conn(0x7f076015db00 0x7f07600b0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f076015e100 con 0x7f07600b0410 2023-10-14T16:53:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0754001670 con 0x7f07600b0410 2023-10-14T16:53:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 >> v1:172.21.15.29:6789/0 conn(0x7f07600b0410 legacy=0x7f076015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.651+0000 7f075d59a640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0748003970 0x7f0748003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075d59a640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07480047c0 0x7f0748004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075d59a640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 0x7f0748005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076015e3e0 con 0x7f07480047c0 2023-10-14T16:53:57.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600cd4a0 con 0x7f0748003970 2023-10-14T16:53:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f074800ba80 con 0x7f07600cd0c0 2023-10-14T16:53:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075ed9d640 1 --2- 172.21.15.29:0/1530748675 >> v2:172.21.15.186:3300/0 conn(0x7f076015db00 0x7f07600b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07480047c0 0x7f0748004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 -- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 msgr2=0x7f0748005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:57.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600cd0c0 0x7f0748005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 -- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0748003970 msgr2=0x7f0748003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0748003970 0x7f0748003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.652+0000 7f075dd9b640 1 -- 172.21.15.29:0/1530748675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0748003520 con 0x7f07480047c0 2023-10-14T16:53:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.653+0000 7f075dd9b640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07480047c0 0x7f0748004c40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f075400c190 tx=0x7f07480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d62bc755cfac0d09 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:57.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.654+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754005ad0 con 0x7f07480047c0 2023-10-14T16:53:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.654+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0754005c50 con 0x7f07480047c0 2023-10-14T16:53:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.654+0000 7f075d59a640 1 -- 172.21.15.29:0/1530748675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075402c7e0 con 0x7f07480047c0 2023-10-14T16:53:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 -- 172.21.15.29:0/1530748675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07480047c0 msgr2=0x7f0748004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 --2- 172.21.15.29:0/1530748675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07480047c0 0x7f0748004c40 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f075400c190 tx=0x7f07480057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f075ed9d640 1 -- 172.21.15.29:0/1530748675 reap_dead start 2023-10-14T16:53:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 -- 172.21.15.29:0/1530748675 shutdown_connections 2023-10-14T16:53:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 -- 172.21.15.29:0/1530748675 >> 172.21.15.29:0/1530748675 conn(0x7f07600cb410 msgr2=0x7f07600cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 -- 172.21.15.29:0/1530748675 shutdown_connections 2023-10-14T16:53:57.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.655+0000 7f0764e61640 1 -- 172.21.15.29:0/1530748675 wait complete. 2023-10-14T16:53:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.657+0000 7f0764e61640 1 Processor -- start 2023-10-14T16:53:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.657+0000 7f0764e61640 1 -- start start 2023-10-14T16:53:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07480047c0 0x7f07600c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600b4230 0x7f076014fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07600b47a0 con 0x7f07600c7450 2023-10-14T16:53:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076015a3b0 con 0x7f07480047c0 2023-10-14T16:53:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f0764e61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07600c8220 con 0x7f07600b4230 2023-10-14T16:53:57.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f075ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f075ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56914/0 (socket says 172.21.15.29:56914) 2023-10-14T16:53:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.658+0000 7f075ed9d640 1 -- 172.21.15.29:0/2613363020 learned_addr learned my addr 172.21.15.29:0/2613363020 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075dd9b640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600b4230 0x7f076014fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075e59c640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07480047c0 0x7f07600c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 -- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600b4230 msgr2=0x7f076014fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600b4230 0x7f076014fd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 -- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07480047c0 msgr2=0x7f07600c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07480047c0 0x7f07600c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 -- 172.21.15.29:0/2613363020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07600c8500 con 0x7f07600c7450 2023-10-14T16:53:57.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f075ed9d640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f076015dee0 tx=0x7f075800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758007e20 con 0x7f07600c7450 2023-10-14T16:53:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075800f040 con 0x7f07600c7450 2023-10-14T16:53:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07600c87e0 con 0x7f07600c7450 2023-10-14T16:53:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758003bd0 con 0x7f07600c7450 2023-10-14T16:53:57.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.659+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0760150250 con 0x7f07600c7450 2023-10-14T16:53:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.660+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07600c8d00 con 0x7f07600c7450 2023-10-14T16:53:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.661+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0758003d30 con 0x7f07600c7450 2023-10-14T16:53:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.662+0000 7f074effd640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 0x7f074006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.662+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f0758089710 con 0x7f07600c7450 2023-10-14T16:53:57.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.663+0000 7f075e59c640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 0x7f074006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.663+0000 7f075e59c640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 0x7f074006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0750004770 tx=0x7f0750008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:57.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.668+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07600c8d00 con 0x7f07600c7450 2023-10-14T16:53:57.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.909+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07600c8d00 con 0x7f07600c7450 2023-10-14T16:53:57.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.910+0000 7f074effd640 1 -- 172.21.15.29:0/2613363020 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f07600c8d00 con 0x7f07600c7450 2023-10-14T16:53:57.911 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:57.911 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:53:57.912 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:53:57.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 msgr2=0x7f074006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 0x7f074006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0750004770 tx=0x7f0750008040 comp rx=0 tx=0).stop 2023-10-14T16:53:57.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 msgr2=0x7f07600c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f076015dee0 tx=0x7f075800d4a0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 shutdown_connections 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f074006acc0 0x7f074006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07600c7450 0x7f07600c78d0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07600b4230 0x7f076014fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 --2- 172.21.15.29:0/2613363020 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07480047c0 0x7f07600c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 >> 172.21.15.29:0/2613363020 conn(0x7f07600cb410 msgr2=0x7f07600ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 shutdown_connections 2023-10-14T16:53:57.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:57.913+0000 7f0764e61640 1 -- 172.21.15.29:0/2613363020 wait complete. 2023-10-14T16:53:57.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:53:58.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.153+0000 7f1550b6a640 1 Processor -- start 2023-10-14T16:53:58.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.153+0000 7f1550b6a640 1 -- start start 2023-10-14T16:53:58.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f1550b6a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f154c06fb50 0x7f154c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f1550b6a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 0x7f154c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f1550b6a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f154c0ca560 con 0x7f154c0c9060 2023-10-14T16:53:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f1550b6a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f154c0c9950 con 0x7f154c06fb50 2023-10-14T16:53:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f1550b6a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c0c9c30 con 0x7f154c06c9e0 2023-10-14T16:53:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f154affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f154c06fb50 0x7f154c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f154affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f154c06fb50 0x7f154c0cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56918/0 (socket says 172.21.15.29:56918) 2023-10-14T16:53:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.155+0000 7f154affd640 1 -- 172.21.15.29:0/358434164 learned_addr learned my addr 172.21.15.29:0/358434164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154bfff640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 0x7f154c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154affd640 1 -- 172.21.15.29:0/358434164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 msgr2=0x7f154c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154affd640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 0x7f154c0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154affd640 1 -- 172.21.15.29:0/358434164 >> v1:172.21.15.186:6789/0 conn(0x7f154c0c9060 legacy=0x7f154c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154affd640 1 -- 172.21.15.29:0/358434164 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f154c0c9f10 con 0x7f154c06fb50 2023-10-14T16:53:58.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.156+0000 7f154bfff640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 0x7f154c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:58.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.157+0000 7f154affd640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06fb50 0x7f154c0cc8a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1540009870 tx=0x7f154002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cf5f13a79896c3d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.157+0000 7f154a7fc640 1 -- 172.21.15.29:0/358434164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540030840 con 0x7f154c06fb50 2023-10-14T16:53:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f154a7fc640 1 -- 172.21.15.29:0/358434164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15400309a0 con 0x7f154c06fb50 2023-10-14T16:53:58.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f154a7fc640 1 -- 172.21.15.29:0/358434164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1540030d30 con 0x7f154c06fb50 2023-10-14T16:53:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 -- 172.21.15.29:0/358434164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06fb50 msgr2=0x7f154c0cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06fb50 0x7f154c0cc8a0 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f1540009870 tx=0x7f154002b980 comp rx=0 tx=0).stop 2023-10-14T16:53:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 -- 172.21.15.29:0/358434164 shutdown_connections 2023-10-14T16:53:58.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06c9e0 0x7f154c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 --2- 172.21.15.29:0/358434164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06fb50 0x7f154c0cc8a0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.158+0000 7f1550b6a640 1 -- 172.21.15.29:0/358434164 >> 172.21.15.29:0/358434164 conn(0x7f154c0ce6a0 msgr2=0x7f154c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:58.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.159+0000 7f1550b6a640 1 -- 172.21.15.29:0/358434164 shutdown_connections 2023-10-14T16:53:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.159+0000 7f1550b6a640 1 -- 172.21.15.29:0/358434164 wait complete. 2023-10-14T16:53:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.160+0000 7f1550b6a640 1 Processor -- start 2023-10-14T16:53:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.160+0000 7f1550b6a640 1 -- start start 2023-10-14T16:53:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06fb50 0x7f154c163540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f154c0c9060 0x7f154c1fa090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c162ea0 con 0x7f154c06c9e0 2023-10-14T16:53:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f154c162290 con 0x7f154c0c9060 2023-10-14T16:53:58.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.161+0000 7f1550b6a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f154c162570 con 0x7f154c06fb50 2023-10-14T16:53:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56920/0 (socket says 172.21.15.29:56920) 2023-10-14T16:53:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 -- 172.21.15.29:0/2570324831 learned_addr learned my addr 172.21.15.29:0/2570324831 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:58.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 -- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06fb50 msgr2=0x7f154c163540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06fb50 0x7f154c163540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 -- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f154c0c9060 msgr2=0x7f154c1fa090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f154c0c9060 0x7f154c1fa090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 -- 172.21.15.29:0/2570324831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15400094d0 con 0x7f154c06c9e0 2023-10-14T16:53:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.162+0000 7f154b7fe640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c400 tx=0x7f153c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.163+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c0052d0 con 0x7f154c06c9e0 2023-10-14T16:53:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.163+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f153c005430 con 0x7f154c06c9e0 2023-10-14T16:53:58.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.163+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f153c003830 con 0x7f154c06c9e0 2023-10-14T16:53:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.163+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f154c162850 con 0x7f154c06c9e0 2023-10-14T16:53:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.163+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f154c161f20 con 0x7f154c06c9e0 2023-10-14T16:53:58.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.164+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f154c04e9b0 con 0x7f154c06c9e0 2023-10-14T16:53:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.165+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f153c003990 con 0x7f154c06c9e0 2023-10-14T16:53:58.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.166+0000 7f1533fff640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 0x7f153806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.166+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f153c088cd0 con 0x7f154c06c9e0 2023-10-14T16:53:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.166+0000 7f154affd640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 0x7f153806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.167+0000 7f154affd640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 0x7f153806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1540036000 tx=0x7f154003b040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.170+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f153c053b40 con 0x7f154c06c9e0 2023-10-14T16:53:58.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.367+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f154c0b4620 con 0x7f154c06c9e0 2023-10-14T16:53:58.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.370+0000 7f1533fff640 1 -- 172.21.15.29:0/2570324831 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+18357 (secure 0 0 0) 0x7f153c057160 con 0x7f154c06c9e0 2023-10-14T16:53:58.372 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:58.372 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:53:57.599089+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":223,"snap_epoch":50,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3003570167":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3659680710":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/4149980298":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3481939635":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/347213673":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/1900972994":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2383868706":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2733984810":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2615927594":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2453699906":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3327634503":"2023-10-15T16:53:56.592878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":2},{"begin":10,"length":2},{"begin":13,"length":1},{"begin":15,"length":3},{"begin":22,"length":3},{"begin":28,"length":1},{"begin":31,"length":3},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":45,"length":3},{"begin":49,"length":2},{"begin":54,"length":1},{"begin":58,"length":3},{"begin":62,"length":1},{"begin":64,"length":1},{"begin":67,"length":3},{"begin":71,"length":1},{"begin":73,"length":1},{"begin":76,"length":3},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":88,"length":1},{"begin":91,"length":2},{"begin":94,"length":2},{"begin":97,"length":1},{"begin":100,"length":2},{"begin":104,"length":2},{"begin":108,"length":2},{"begin":111,"length":2},{"begin":168,"length":56}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:53:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 msgr2=0x7f153806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 0x7f153806d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f1540036000 tx=0x7f154003b040 comp rx=0 tx=0).stop 2023-10-14T16:53:58.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 msgr2=0x7f154c1f3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f153c00c400 tx=0x7f153c00c8c0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 shutdown_connections 2023-10-14T16:53:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f153806acf0 0x7f153806d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f154c0c9060 0x7f154c1fa090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f154c06fb50 0x7f154c163540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 --2- 172.21.15.29:0/2570324831 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f154c06c9e0 0x7f154c1f3c70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 >> 172.21.15.29:0/2570324831 conn(0x7f154c0ce6a0 msgr2=0x7f154c0c1560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.373+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 shutdown_connections 2023-10-14T16:53:58.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.374+0000 7f1550b6a640 1 -- 172.21.15.29:0/2570324831 wait complete. 2023-10-14T16:53:58.395 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:53:58.396 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:53:58.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.624+0000 7f4201533640 1 Processor -- start 2023-10-14T16:53:58.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.624+0000 7f4201533640 1 -- start start 2023-10-14T16:53:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.626+0000 7f4201533640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41fc159f00 0x7f41fc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.626+0000 7f4201533640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc15d880 0x7f41fc0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.626+0000 7f4201533640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41fc06f7f0 con 0x7f41fc159b20 2023-10-14T16:53:58.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.626+0000 7f4201533640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41fc06c5e0 con 0x7f41fc159f00 2023-10-14T16:53:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.626+0000 7f4201533640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc06c8c0 con 0x7f41fc15d880 2023-10-14T16:53:58.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41fc159f00 0x7f41fc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41fc159f00 0x7f41fc0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56928/0 (socket says 172.21.15.29:56928) 2023-10-14T16:53:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 -- 172.21.15.29:0/1419356699 learned_addr learned my addr 172.21.15.29:0/1419356699 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41fb7fe640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc15d880 0x7f41fc0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 -- 172.21.15.29:0/1419356699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc15d880 msgr2=0x7f41fc0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:58.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc15d880 0x7f41fc0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 -- 172.21.15.29:0/1419356699 >> v1:172.21.15.186:6789/0 conn(0x7f41fc159b20 legacy=0x7f41fc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:58.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.627+0000 7f41f3fff640 1 -- 172.21.15.29:0/1419356699 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41fc06cba0 con 0x7f41fc159f00 2023-10-14T16:53:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.629+0000 7f41f3fff640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0b0410 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f41e400bb10 tx=0x7f41e402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff220a942b5e892 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.630+0000 7f41fa7fc640 1 -- 172.21.15.29:0/1419356699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e402ee70 con 0x7f41fc159f00 2023-10-14T16:53:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.630+0000 7f41fa7fc640 1 -- 172.21.15.29:0/1419356699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e4036440 con 0x7f41fc159f00 2023-10-14T16:53:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.630+0000 7f41fa7fc640 1 -- 172.21.15.29:0/1419356699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e40367d0 con 0x7f41fc159f00 2023-10-14T16:53:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 -- 172.21.15.29:0/1419356699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 msgr2=0x7f41fc0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0b0410 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f41e400bb10 tx=0x7f41e402e0c0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 -- 172.21.15.29:0/1419356699 shutdown_connections 2023-10-14T16:53:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc15d880 0x7f41fc0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 --2- 172.21.15.29:0/1419356699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0b0410 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 -- 172.21.15.29:0/1419356699 >> 172.21.15.29:0/1419356699 conn(0x7f41fc0cb410 msgr2=0x7f41fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:58.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 -- 172.21.15.29:0/1419356699 shutdown_connections 2023-10-14T16:53:58.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.631+0000 7f4201533640 1 -- 172.21.15.29:0/1419356699 wait complete. 2023-10-14T16:53:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.633+0000 7f4201533640 1 Processor -- start 2023-10-14T16:53:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.633+0000 7f4201533640 1 -- start start 2023-10-14T16:53:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 0x7f41fc14c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41fc15d880 0x7f41fc15af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c6a30 con 0x7f41fc159f00 2023-10-14T16:53:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c5e20 con 0x7f41fc15d880 2023-10-14T16:53:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.634+0000 7f4201533640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41fc0c6100 con 0x7f41fc159b20 2023-10-14T16:53:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56934/0 (socket says 172.21.15.29:56934) 2023-10-14T16:53:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 -- 172.21.15.29:0/4071454129 learned_addr learned my addr 172.21.15.29:0/4071454129 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41faffd640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 0x7f41fc14c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 -- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 msgr2=0x7f41fc14c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 0x7f41fc14c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 -- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41fc15d880 msgr2=0x7f41fc15af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41fc15d880 0x7f41fc15af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 -- 172.21.15.29:0/4071454129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e400b770 con 0x7f41fc159f00 2023-10-14T16:53:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41faffd640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 0x7f41fc14c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:58.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.635+0000 7f41f3fff640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f41e400b2c0 tx=0x7f41e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.636+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e403f910 con 0x7f41fc159f00 2023-10-14T16:53:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.636+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41e403fa90 con 0x7f41fc159f00 2023-10-14T16:53:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.636+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41e403fe40 con 0x7f41fc159f00 2023-10-14T16:53:58.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.636+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41fc0c63e0 con 0x7f41fc159f00 2023-10-14T16:53:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.636+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41fc14ccb0 con 0x7f41fc159f00 2023-10-14T16:53:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.637+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41fc04e160 con 0x7f41fc159f00 2023-10-14T16:53:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.638+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 16) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f41e403d070 con 0x7f41fc159f00 2023-10-14T16:53:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.639+0000 7f41f37fe640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 0x7f41dc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:58.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.639+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f41e40abea0 con 0x7f41fc159f00 2023-10-14T16:53:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.639+0000 7f41faffd640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 0x7f41dc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:58.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.640+0000 7f41faffd640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 0x7f41dc06d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00a830 tx=0x7f41ec008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:58.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.644+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f41e4076e30 con 0x7f41fc159f00 2023-10-14T16:53:58.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.888+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41fc0707b0 con 0x7f41fc159f00 2023-10-14T16:53:58.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.888+0000 7f41f37fe640 1 -- 172.21.15.29:0/4071454129 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f41e4043090 con 0x7f41fc159f00 2023-10-14T16:53:58.889 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:58.890 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:53:58.890 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:53:58.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 msgr2=0x7f41dc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 0x7f41dc06d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00a830 tx=0x7f41ec008040 comp rx=0 tx=0).stop 2023-10-14T16:53:58.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 msgr2=0x7f41fc0c2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:58.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f41e400b2c0 tx=0x7f41e40046c0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 shutdown_connections 2023-10-14T16:53:58.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41dc06acf0 0x7f41dc06d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41fc15d880 0x7f41fc15af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41fc159f00 0x7f41fc0c2980 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 --2- 172.21.15.29:0/4071454129 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41fc159b20 0x7f41fc14c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 >> 172.21.15.29:0/4071454129 conn(0x7f41fc0cb410 msgr2=0x7f41fc06bba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:58.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.891+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 shutdown_connections 2023-10-14T16:53:58.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:58.892+0000 7f4201533640 1 -- 172.21.15.29:0/4071454129 wait complete. 2023-10-14T16:53:58.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:53:59.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.134+0000 7f83775dc640 1 Processor -- start 2023-10-14T16:53:59.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.134+0000 7f83775dc640 1 -- start start 2023-10-14T16:53:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f83775dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f837015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f83775dc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f837015ed60 con 0x7f837015ae80 2023-10-14T16:53:59.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f83775dc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83700707e0 con 0x7f837015aaa0 2023-10-14T16:53:59.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f83775dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837006b810 con 0x7f83700ce390 2023-10-14T16:53:59.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f8374b77640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f837015ae80 0x7f8370070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40866/0 (socket says 172.21.15.29:40866) 2023-10-14T16:53:59.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f8374b77640 1 -- 172.21.15.29:0/3348201120 learned_addr learned my addr 172.21.15.29:0/3348201120 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.136+0000 7f8375b79640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f837015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102153591 0 0) 0x7f837015ed60 con 0x7f837015ae80 2023-10-14T16:53:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c003520 con 0x7f837015ae80 2023-10-14T16:53:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2618204843 0 0) 0x7f835c003520 con 0x7f837015ae80 2023-10-14T16:53:59.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 msgr2=0x7f837015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f837015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 >> v1:172.21.15.186:6789/0 conn(0x7f837015aaa0 legacy=0x7f83700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837015ef90 con 0x7f837015ae80 2023-10-14T16:53:59.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8360001690 con 0x7f837015ae80 2023-10-14T16:53:59.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 >> v1:172.21.15.29:6789/0 conn(0x7f837015ae80 legacy=0x7f8370070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835c0039a0 0x7f835c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f835c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f835c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837006b810 con 0x7f835c004d00 2023-10-14T16:53:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00ba60 con 0x7f835c0039a0 2023-10-14T16:53:59.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.137+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f835c00bc20 con 0x7f83700ce390 2023-10-14T16:53:59.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f835c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8374b77640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f835c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375378640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835c0039a0 0x7f835c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 -- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 msgr2=0x7f835c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700ce390 0x7f835c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 -- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835c0039a0 msgr2=0x7f835c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835c0039a0 0x7f835c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375b79640 1 -- 172.21.15.29:0/3348201120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f835c003520 con 0x7f835c004d00 2023-10-14T16:53:59.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.138+0000 7f8375378640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f835c0039a0 0x7f835c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:59.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.139+0000 7f8375b79640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f835c005180 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8368009f90 tx=0x7f835c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dfd35dbbdc3934 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.139+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836802cbd0 con 0x7f835c004d00 2023-10-14T16:53:59.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836800f040 con 0x7f835c004d00 2023-10-14T16:53:59.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f8367fff640 1 -- 172.21.15.29:0/3348201120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8368034440 con 0x7f835c004d00 2023-10-14T16:53:59.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 -- 172.21.15.29:0/3348201120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 msgr2=0x7f835c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 --2- 172.21.15.29:0/3348201120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f835c005180 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7f8368009f90 tx=0x7f835c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f8375b79640 1 -- 172.21.15.29:0/3348201120 reap_dead start 2023-10-14T16:53:59.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 -- 172.21.15.29:0/3348201120 shutdown_connections 2023-10-14T16:53:59.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 -- 172.21.15.29:0/3348201120 >> 172.21.15.29:0/3348201120 conn(0x7f83700cb410 msgr2=0x7f83700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:59.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 -- 172.21.15.29:0/3348201120 shutdown_connections 2023-10-14T16:53:59.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.140+0000 7f83775dc640 1 -- 172.21.15.29:0/3348201120 wait complete. 2023-10-14T16:53:59.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.142+0000 7f83775dc640 1 Processor -- start 2023-10-14T16:53:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.142+0000 7f83775dc640 1 -- start start 2023-10-14T16:53:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 0x7f8370158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 0x7f837015c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8370159600 con 0x7f835c004d00 2023-10-14T16:53:59.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f837015b320 con 0x7f8370159150 2023-10-14T16:53:59.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f83775dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f837015b530 con 0x7f83700c4550 2023-10-14T16:53:59.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f8375378640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f8375378640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56960/0 (socket says 172.21.15.29:56960) 2023-10-14T16:53:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.143+0000 7f8375378640 1 -- 172.21.15.29:0/1684819342 learned_addr learned my addr 172.21.15.29:0/1684819342 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8374b77640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 0x7f8370158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375b79640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 0x7f837015c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 -- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 msgr2=0x7f8370158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 0x7f8370158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 -- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 msgr2=0x7f837015c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 0x7f837015c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 -- 172.21.15.29:0/1684819342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f837015b810 con 0x7f835c004d00 2023-10-14T16:53:59.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8374b77640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 0x7f8370158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:59.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375b79640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 0x7f837015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:59.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8375378640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f836c00d7a0 tx=0x7f836c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.144+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c017070 con 0x7f835c004d00 2023-10-14T16:53:59.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.145+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f836c005070 con 0x7f835c004d00 2023-10-14T16:53:59.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.145+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f837015baf0 con 0x7f835c004d00 2023-10-14T16:53:59.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.145+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f836c005420 con 0x7f835c004d00 2023-10-14T16:53:59.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.145+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f837015c0b0 con 0x7f835c004d00 2023-10-14T16:53:59.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.146+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f837004e160 con 0x7f835c004d00 2023-10-14T16:53:59.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.147+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f836c007030 con 0x7f835c004d00 2023-10-14T16:53:59.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.148+0000 7f8365ffb640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 0x7f835406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.148+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f836c089470 con 0x7f835c004d00 2023-10-14T16:53:59.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.148+0000 7f8374b77640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 0x7f835406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.149+0000 7f8374b77640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 0x7f835406d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83700c3e10 tx=0x7f8360001560 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.152+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f836c053ff0 con 0x7f835c004d00 2023-10-14T16:53:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.390+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83700cdc60 con 0x7f835c004d00 2023-10-14T16:53:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.390+0000 7f8365ffb640 1 -- 172.21.15.29:0/1684819342 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f836c057610 con 0x7f835c004d00 2023-10-14T16:53:59.391 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:59.392 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:41:38.721456+0000","modified":"2023-10-14T16:41:41.828160+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5118},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5118":{"gid":5118,"name":"a","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.29:6837/2277792616","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":2277792616},{"type":"v1","addr":"172.21.15.29:6837","nonce":2277792616}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":2}]} 2023-10-14T16:53:59.392 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 15 2023-10-14T16:53:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.393+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 msgr2=0x7f835406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.393+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 0x7f835406d410 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f83700c3e10 tx=0x7f8360001560 comp rx=0 tx=0).stop 2023-10-14T16:53:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.393+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 msgr2=0x7f83701f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.393+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7f836c00d7a0 tx=0x7f836c00dc60 comp rx=0 tx=0).stop 2023-10-14T16:53:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 shutdown_connections 2023-10-14T16:53:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f835406af50 0x7f835406d410 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8370159150 0x7f837015c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83700c4550 0x7f8370158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 --2- 172.21.15.29:0/1684819342 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f835c004d00 0x7f83701f3760 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 >> 172.21.15.29:0/1684819342 conn(0x7f83700cb410 msgr2=0x7f837006fb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 shutdown_connections 2023-10-14T16:53:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.394+0000 7f83775dc640 1 -- 172.21.15.29:0/1684819342 wait complete. 2023-10-14T16:53:59.416 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:53:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.638+0000 7fe25759e640 1 Processor -- start 2023-10-14T16:53:59.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.638+0000 7fe25759e640 1 -- start start 2023-10-14T16:53:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.639+0000 7fe25759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe2580ccb00 0x7fe2580ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.639+0000 7fe25759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580ad4d0 0x7fe2580ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.639+0000 7fe25759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2580cd4a0 con 0x7fe258153740 2023-10-14T16:53:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.639+0000 7fe25759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe25815ac60 con 0x7fe2580ccb00 2023-10-14T16:53:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.639+0000 7fe25759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe258070030 con 0x7fe2580ad4d0 2023-10-14T16:53:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe25659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe258153740 0x7fe25815a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40868/0 (socket says 172.21.15.29:40868) 2023-10-14T16:53:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe25659c640 1 -- 172.21.15.29:0/2465328511 learned_addr learned my addr 172.21.15.29:0/2465328511 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:53:59.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe255d9b640 1 --2- 172.21.15.29:0/2465328511 >> v2:172.21.15.186:3300/0 conn(0x7fe2580ccb00 0x7fe2580ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe256d9d640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580ad4d0 0x7fe2580ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1698083711 0 0) 0x7fe2580cd4a0 con 0x7fe258153740 2023-10-14T16:53:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.640+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe244005730 con 0x7fe258153740 2023-10-14T16:53:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255314585 0 0) 0x7fe244005730 con 0x7fe258153740 2023-10-14T16:53:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580ad4d0 msgr2=0x7fe2580ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580ad4d0 0x7fe2580ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 >> v2:172.21.15.186:3300/0 conn(0x7fe2580ccb00 msgr2=0x7fe2580ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 --2- 172.21.15.29:0/2465328511 >> v2:172.21.15.186:3300/0 conn(0x7fe2580ccb00 0x7fe2580ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe258070310 con 0x7fe258153740 2023-10-14T16:53:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe248001670 con 0x7fe258153740 2023-10-14T16:53:59.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 >> v1:172.21.15.29:6789/0 conn(0x7fe258153740 legacy=0x7fe25815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe244005b80 0x7fe244005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe244006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe244007390 0x7fe24400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe25815ac60 con 0x7fe2440069d0 2023-10-14T16:53:59.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe258070030 con 0x7fe244005b80 2023-10-14T16:53:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe24400e100 con 0x7fe244007390 2023-10-14T16:53:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe256d9d640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580ad4d0 0x7fe2580ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:53:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe255d9b640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe244006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe25659c640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe244005b80 0x7fe244005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe255d9b640 1 -- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe244007390 msgr2=0x7fe24400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe255d9b640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe244007390 0x7fe24400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe255d9b640 1 -- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe244005b80 msgr2=0x7fe244005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.641+0000 7fe255d9b640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe244005b80 0x7fe244005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.642+0000 7fe255d9b640 1 -- 172.21.15.29:0/2465328511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe244005730 con 0x7fe2440069d0 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.642+0000 7fe25659c640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe244005b80 0x7fe244005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:59.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe255d9b640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe244006e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe240009ac0 tx=0x7fe244007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1859276aa92c3a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24002c6d0 con 0x7fe2440069d0 2023-10-14T16:53:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe24002c830 con 0x7fe2440069d0 2023-10-14T16:53:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25559a640 1 -- 172.21.15.29:0/2465328511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe24002cbc0 con 0x7fe2440069d0 2023-10-14T16:53:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25759e640 1 -- 172.21.15.29:0/2465328511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 msgr2=0x7fe244006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25759e640 1 --2- 172.21.15.29:0/2465328511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe244006e50 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fe240009ac0 tx=0x7fe244007a00 comp rx=0 tx=0).stop 2023-10-14T16:53:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe256d9d640 1 -- 172.21.15.29:0/2465328511 reap_dead start 2023-10-14T16:53:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25759e640 1 -- 172.21.15.29:0/2465328511 shutdown_connections 2023-10-14T16:53:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.643+0000 7fe25759e640 1 -- 172.21.15.29:0/2465328511 >> 172.21.15.29:0/2465328511 conn(0x7fe2580ce6a0 msgr2=0x7fe2580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.644+0000 7fe25759e640 1 -- 172.21.15.29:0/2465328511 shutdown_connections 2023-10-14T16:53:59.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.644+0000 7fe25759e640 1 -- 172.21.15.29:0/2465328511 wait complete. 2023-10-14T16:53:59.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.645+0000 7fe25759e640 1 Processor -- start 2023-10-14T16:53:59.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.646+0000 7fe25759e640 1 -- start start 2023-10-14T16:53:59.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 0x7fe2581f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580c5280 0x7fe2581f2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580b34e0 con 0x7fe2440069d0 2023-10-14T16:53:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2580b2650 con 0x7fe2580c4e00 2023-10-14T16:53:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe25814d9f0 con 0x7fe2580c5280 2023-10-14T16:53:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:56984/0 (socket says 172.21.15.29:56984) 2023-10-14T16:53:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 -- 172.21.15.29:0/675820182 learned_addr learned my addr 172.21.15.29:0/675820182 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:53:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 -- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580c5280 msgr2=0x7fe2581f2bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580c5280 0x7fe2581f2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 -- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 msgr2=0x7fe2581f26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe255d9b640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 0x7fe2581f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 0x7fe2581f26b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe25659c640 1 -- 172.21.15.29:0/675820182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2580b2860 con 0x7fe2440069d0 2023-10-14T16:53:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.647+0000 7fe255d9b640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 0x7fe2581f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:53:59.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe25659c640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe24800da90 tx=0x7fe24800df50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2480074e0 con 0x7fe2440069d0 2023-10-14T16:53:59.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe248007640 con 0x7fe2440069d0 2023-10-14T16:53:59.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2480049e0 con 0x7fe2440069d0 2023-10-14T16:53:59.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2580b2b40 con 0x7fe2440069d0 2023-10-14T16:53:59.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.648+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2580b3100 con 0x7fe2440069d0 2023-10-14T16:53:59.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.649+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2580b32c0 con 0x7fe2440069d0 2023-10-14T16:53:59.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.650+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe248004b40 con 0x7fe2440069d0 2023-10-14T16:53:59.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.651+0000 7fe24effd640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 0x7fe23806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:53:59.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.651+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6906+0+0 (secure 0 0 0) 0x7fe2480895b0 con 0x7fe2440069d0 2023-10-14T16:53:59.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.652+0000 7fe255d9b640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 0x7fe23806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:53:59.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.653+0000 7fe255d9b640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 0x7fe23806d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe240036000 tx=0x7fe240030000 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:53:59.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.656+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe248053fb0 con 0x7fe2440069d0 2023-10-14T16:53:59.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.843+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe25806fc30 con 0x7fe2440069d0 2023-10-14T16:53:59.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.847+0000 7fe24effd640 1 -- 172.21.15.29:0/675820182 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v50) v1 ==== 74+0+18357 (secure 0 0 0) 0x7fe248019090 con 0x7fe2440069d0 2023-10-14T16:53:59.848 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:53:59.848 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":50,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:53:57.599089+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:41:36.429166+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2023-10-14T16:41:37.759317+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":223,"snap_epoch":50,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3003570167":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3659680710":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/4149980298":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3481939635":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/347213673":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/1900972994":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2383868706":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2733984810":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2615927594":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2453699906":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3327634503":"2023-10-15T16:53:56.592878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":3,"length":1},{"begin":6,"length":2},{"begin":10,"length":2},{"begin":13,"length":1},{"begin":15,"length":3},{"begin":22,"length":3},{"begin":28,"length":1},{"begin":31,"length":3},{"begin":35,"length":1},{"begin":37,"length":3},{"begin":45,"length":3},{"begin":49,"length":2},{"begin":54,"length":1},{"begin":58,"length":3},{"begin":62,"length":1},{"begin":64,"length":1},{"begin":67,"length":3},{"begin":71,"length":1},{"begin":73,"length":1},{"begin":76,"length":3},{"begin":80,"length":1},{"begin":85,"length":2},{"begin":88,"length":1},{"begin":91,"length":2},{"begin":94,"length":2},{"begin":97,"length":1},{"begin":100,"length":2},{"begin":104,"length":2},{"begin":108,"length":2},{"begin":111,"length":2},{"begin":168,"length":56}]}],"new_removed_snaps":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 msgr2=0x7fe23806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 0x7fe23806d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe240036000 tx=0x7fe240030000 comp rx=0 tx=0).stop 2023-10-14T16:53:59.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 msgr2=0x7fe25814d4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fe24800da90 tx=0x7fe24800df50 comp rx=0 tx=0).stop 2023-10-14T16:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 shutdown_connections 2023-10-14T16:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe23806af30 0x7fe23806d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe2580c5280 0x7fe2581f2bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe2580c4e00 0x7fe2581f26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 --2- 172.21.15.29:0/675820182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe2440069d0 0x7fe25814d4b0 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 >> 172.21.15.29:0/675820182 conn(0x7fe2580ce6a0 msgr2=0x7fe2580ad960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:53:59.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 shutdown_connections 2023-10-14T16:53:59.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:53:59.849+0000 7fe25759e640 1 -- 172.21.15.29:0/675820182 wait complete. 2023-10-14T16:53:59.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:54:00.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.087+0000 7f2b3cd9e640 1 Processor -- start 2023-10-14T16:54:00.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.087+0000 7f2b3cd9e640 1 -- start start 2023-10-14T16:54:00.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 0x7f2b380c1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 0x7f2b380c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b380ccd80 0x7f2b380cd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b380cb1d0 con 0x7f2b380ccd80 2023-10-14T16:54:00.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b381659b0 con 0x7f2b380c9800 2023-10-14T16:54:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b3cd9e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b38165c90 con 0x7f2b38161320 2023-10-14T16:54:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b36d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b380ccd80 0x7f2b380cd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b36d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b380ccd80 0x7f2b380cd1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57000/0 (socket says 172.21.15.29:57000) 2023-10-14T16:54:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.089+0000 7f2b36d9d640 1 -- 172.21.15.29:0/2561342983 learned_addr learned my addr 172.21.15.29:0/2561342983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b3659c640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 0x7f2b380c1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b36d9d640 1 -- 172.21.15.29:0/2561342983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 msgr2=0x7f2b380c1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b2dd9b640 1 --2- 172.21.15.29:0/2561342983 >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 0x7f2b380c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 0x7f2b380c1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b36d9d640 1 -- 172.21.15.29:0/2561342983 >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 msgr2=0x7f2b380c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/2561342983 >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 0x7f2b380c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b36d9d640 1 -- 172.21.15.29:0/2561342983 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b380cb450 con 0x7f2b380ccd80 2023-10-14T16:54:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b3659c640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 0x7f2b380c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:00.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.090+0000 7f2b2dd9b640 1 --2- 172.21.15.29:0/2561342983 >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 0x7f2b380c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:00.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.091+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b380ccd80 0x7f2b380cd1d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2b3806f6a0 tx=0x7f2b3002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ff67ed8602add1d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.091+0000 7f2b35d9b640 1 -- 172.21.15.29:0/2561342983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30031870 con 0x7f2b380ccd80 2023-10-14T16:54:00.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.091+0000 7f2b35d9b640 1 -- 172.21.15.29:0/2561342983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b300319d0 con 0x7f2b380ccd80 2023-10-14T16:54:00.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.091+0000 7f2b35d9b640 1 -- 172.21.15.29:0/2561342983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30031d60 con 0x7f2b380ccd80 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/2561342983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b380ccd80 msgr2=0x7f2b380cd1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b380ccd80 0x7f2b380cd1d0 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f2b3806f6a0 tx=0x7f2b3002b910 comp rx=0 tx=0).stop 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/2561342983 shutdown_connections 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b380ccd80 0x7f2b380cd1d0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/2561342983 >> v2:172.21.15.186:3300/0 conn(0x7f2b380c9800 0x7f2b380c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/2561342983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b38161320 0x7f2b380c1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/2561342983 >> 172.21.15.29:0/2561342983 conn(0x7f2b3806bc10 msgr2=0x7f2b3806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/2561342983 shutdown_connections 2023-10-14T16:54:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.092+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/2561342983 wait complete. 2023-10-14T16:54:00.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.094+0000 7f2b3cd9e640 1 Processor -- start 2023-10-14T16:54:00.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.094+0000 7f2b3cd9e640 1 -- start start 2023-10-14T16:54:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b380c9800 0x7f2b381551b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b380ccd80 0x7f2b380c2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381fb270 con 0x7f2b38161320 2023-10-14T16:54:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381556f0 con 0x7f2b380c9800 2023-10-14T16:54:00.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.095+0000 7f2b3cd9e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b381facb0 con 0x7f2b380ccd80 2023-10-14T16:54:00.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57014/0 (socket says 172.21.15.29:57014) 2023-10-14T16:54:00.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 -- 172.21.15.29:0/3409393749 learned_addr learned my addr 172.21.15.29:0/3409393749 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:00.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b3659c640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b380c9800 0x7f2b381551b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b2dd9b640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b380ccd80 0x7f2b380c2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 -- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b380ccd80 msgr2=0x7f2b380c2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b380ccd80 0x7f2b380c2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 -- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b380c9800 msgr2=0x7f2b381551b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b380c9800 0x7f2b381551b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 -- 172.21.15.29:0/3409393749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b300094d0 con 0x7f2b38161320 2023-10-14T16:54:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.096+0000 7f2b36d9d640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2b300090b0 tx=0x7f2b300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.097+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b30031800 con 0x7f2b38161320 2023-10-14T16:54:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.097+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b300443f0 con 0x7f2b38161320 2023-10-14T16:54:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.097+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b300447a0 con 0x7f2b38161320 2023-10-14T16:54:00.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.097+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b381faf90 con 0x7f2b38161320 2023-10-14T16:54:00.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.097+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b38161b80 con 0x7f2b38161320 2023-10-14T16:54:00.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.098+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b38161700 con 0x7f2b38161320 2023-10-14T16:54:00.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.099+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2b3004d450 con 0x7f2b38161320 2023-10-14T16:54:00.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.100+0000 7f2b2f7fe640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 0x7f2b1806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.100+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6906+0+0 (secure 0 0 0) 0x7f2b3002d040 con 0x7f2b38161320 2023-10-14T16:54:00.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.100+0000 7f2b3659c640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 0x7f2b1806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.101+0000 7f2b3659c640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 0x7f2b1806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007130 tx=0x7f2b28007060 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.105+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b38161700 con 0x7f2b38161320 2023-10-14T16:54:00.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.343+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2b38161700 con 0x7f2b38161320 2023-10-14T16:54:00.346 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:54:00.345+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:54:00.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.354+0000 7f2b2f7fe640 1 -- 172.21.15.29:0/3409393749 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v16) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2b38161700 con 0x7f2b38161320 2023-10-14T16:54:00.358 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:54:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 msgr2=0x7f2b1806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 0x7f2b1806d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2b28007130 tx=0x7f2b28007060 comp rx=0 tx=0).stop 2023-10-14T16:54:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 msgr2=0x7f2b381fa280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f2b300090b0 tx=0x7f2b300046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 shutdown_connections 2023-10-14T16:54:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b1806af00 0x7f2b1806d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b38161320 0x7f2b381fa280 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b380ccd80 0x7f2b380c2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 --2- 172.21.15.29:0/3409393749 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b380c9800 0x7f2b381551b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 >> 172.21.15.29:0/3409393749 conn(0x7f2b3806bc10 msgr2=0x7f2b380b2000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.360+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 shutdown_connections 2023-10-14T16:54:00.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.362+0000 7f2b3cd9e640 1 -- 172.21.15.29:0/3409393749 wait complete. 2023-10-14T16:54:00.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:54:00.440 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:54:00.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.571+0000 7f0504821640 1 Processor -- start 2023-10-14T16:54:00.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.571+0000 7f0504821640 1 -- start start 2023-10-14T16:54:00.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f050015ad70 0x7f0500069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f050006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05000ca3b0 0x7f05000ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05000cad80 con 0x7f05000ca3b0 2023-10-14T16:54:00.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05000cb030 con 0x7f050015ad70 2023-10-14T16:54:00.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f0504821640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05000c91c0 con 0x7f05000c9ca0 2023-10-14T16:54:00.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f04fed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05000ca3b0 0x7f05000ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f04fed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05000ca3b0 0x7f05000ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57044/0 (socket says 172.21.15.29:57044) 2023-10-14T16:54:00.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.573+0000 7f04fed9d640 1 -- 172.21.15.29:0/217179541 learned_addr learned my addr 172.21.15.29:0/217179541 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:00.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fe59c640 1 --2- 172.21.15.29:0/217179541 >> v2:172.21.15.186:3300/0 conn(0x7f050015ad70 0x7f0500069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04f7fff640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f050006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fed9d640 1 -- 172.21.15.29:0/217179541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 msgr2=0x7f050006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:00.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fed9d640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f050006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fed9d640 1 -- 172.21.15.29:0/217179541 >> v2:172.21.15.186:3300/0 conn(0x7f050015ad70 msgr2=0x7f0500069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:00.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fed9d640 1 --2- 172.21.15.29:0/217179541 >> v2:172.21.15.186:3300/0 conn(0x7f050015ad70 0x7f0500069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.574+0000 7f04fed9d640 1 -- 172.21.15.29:0/217179541 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05000c94a0 con 0x7f05000ca3b0 2023-10-14T16:54:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.575+0000 7f04fed9d640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05000ca3b0 0x7f05000ca840 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f05000cf780 tx=0x7f04f802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c976a56072ef8e89 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f04fdd9b640 1 -- 172.21.15.29:0/217179541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f802ca80 con 0x7f05000ca3b0 2023-10-14T16:54:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f04fdd9b640 1 -- 172.21.15.29:0/217179541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f802cbe0 con 0x7f05000ca3b0 2023-10-14T16:54:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f04fdd9b640 1 -- 172.21.15.29:0/217179541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f80343f0 con 0x7f05000ca3b0 2023-10-14T16:54:00.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 -- 172.21.15.29:0/217179541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05000ca3b0 msgr2=0x7f05000ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05000ca3b0 0x7f05000ca840 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f05000cf780 tx=0x7f04f802bee0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 -- 172.21.15.29:0/217179541 shutdown_connections 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05000ca3b0 0x7f05000ca840 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 --2- 172.21.15.29:0/217179541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f050006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 --2- 172.21.15.29:0/217179541 >> v2:172.21.15.186:3300/0 conn(0x7f050015ad70 0x7f0500069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.576+0000 7f0504821640 1 -- 172.21.15.29:0/217179541 >> 172.21.15.29:0/217179541 conn(0x7f05000ce9b0 msgr2=0x7f05000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:00.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.577+0000 7f0504821640 1 -- 172.21.15.29:0/217179541 shutdown_connections 2023-10-14T16:54:00.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.577+0000 7f0504821640 1 -- 172.21.15.29:0/217179541 wait complete. 2023-10-14T16:54:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.578+0000 7f0504821640 1 Processor -- start 2023-10-14T16:54:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.579+0000 7f0504821640 1 -- start start 2023-10-14T16:54:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.579+0000 7f0504821640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f05000c4a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f0504821640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05000ca3b0 0x7f0500158cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f0504821640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f0504821640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000c6d70 con 0x7f050015ad70 2023-10-14T16:54:00.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f0504821640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05000c7050 con 0x7f05000ca3b0 2023-10-14T16:54:00.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f0504821640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05000c7330 con 0x7f05000c9ca0 2023-10-14T16:54:00.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57060/0 (socket says 172.21.15.29:57060) 2023-10-14T16:54:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 -- 172.21.15.29:0/3324832330 learned_addr learned my addr 172.21.15.29:0/3324832330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04f7fff640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05000ca3b0 0x7f0500158cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fe59c640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f05000c4a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 -- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 msgr2=0x7f05000c4a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f05000c4a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 -- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05000ca3b0 msgr2=0x7f0500158cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05000ca3b0 0x7f0500158cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:00.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.580+0000 7f04fed9d640 1 -- 172.21.15.29:0/3324832330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04f8009b30 con 0x7f050015ad70 2023-10-14T16:54:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f04fed9d640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f04f8036040 tx=0x7f04f8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f8009510 con 0x7f050015ad70 2023-10-14T16:54:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04f803e450 con 0x7f050015ad70 2023-10-14T16:54:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04f803e7e0 con 0x7f050015ad70 2023-10-14T16:54:00.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05000c7610 con 0x7f050015ad70 2023-10-14T16:54:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.581+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05001f28c0 con 0x7f050015ad70 2023-10-14T16:54:00.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.582+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f050004e160 con 0x7f050015ad70 2023-10-14T16:54:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.584+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 17) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f04f802cd50 con 0x7f050015ad70 2023-10-14T16:54:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.584+0000 7f04f6ffd640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 0x7f04e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.585+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6484+0+0 (secure 0 0 0) 0x7f04f80ac580 con 0x7f050015ad70 2023-10-14T16:54:00.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.585+0000 7f04fe59c640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 0x7f04e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.585+0000 7f04fe59c640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 0x7f04e006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04f0006fd0 tx=0x7f04f0008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:00.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.589+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f04f8077230 con 0x7f050015ad70 2023-10-14T16:54:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:00.833+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f05000b42b0 con 0x7f050015ad70 2023-10-14T16:54:01.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.367+0000 7f04f6ffd640 1 -- 172.21.15.29:0/3324832330 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f04f807a850 con 0x7f050015ad70 2023-10-14T16:54:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.372+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 msgr2=0x7f04e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.372+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 0x7f04e006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f04f0006fd0 tx=0x7f04f0008040 comp rx=0 tx=0).stop 2023-10-14T16:54:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.372+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 msgr2=0x7f05001f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:01.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.372+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f04f8036040 tx=0x7f04f8004a40 comp rx=0 tx=0).stop 2023-10-14T16:54:01.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 shutdown_connections 2023-10-14T16:54:01.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f04e006af00 0x7f04e006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f050015ad70 0x7f05001f2380 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05000ca3b0 0x7f0500158cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 --2- 172.21.15.29:0/3324832330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05000c9ca0 0x7f05000c4a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 >> 172.21.15.29:0/3324832330 conn(0x7f05000ce9b0 msgr2=0x7f050015b4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:01.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 shutdown_connections 2023-10-14T16:54:01.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.373+0000 7f0504821640 1 -- 172.21.15.29:0/3324832330 wait complete. 2023-10-14T16:54:01.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:54:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.617+0000 7f7fab39a640 1 Processor -- start 2023-10-14T16:54:01.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.617+0000 7f7fab39a640 1 -- start start 2023-10-14T16:54:01.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa4159950 0x7f7fa415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fa4070450 0x7f7fa40708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 0x7f7fa406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa415e490 con 0x7f7fa4070450 2023-10-14T16:54:01.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fa4070df0 con 0x7f7fa406c100 2023-10-14T16:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fab39a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa415aef0 con 0x7f7fa4159950 2023-10-14T16:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fa9b97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fa4070450 0x7f7fa40708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fa9b97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7fa4070450 0x7f7fa40708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46358/0 (socket says 172.21.15.29:46358) 2023-10-14T16:54:01.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.619+0000 7f7fa9b97640 1 -- 172.21.15.29:0/3057549046 learned_addr learned my addr 172.21.15.29:0/3057549046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7faab99640 1 --2- 172.21.15.29:0/3057549046 >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 0x7f7fa406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7fa9b97640 1 -- 172.21.15.29:0/3057549046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa4159950 msgr2=0x7f7fa415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:01.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7fa9b97640 1 --2- 172.21.15.29:0/3057549046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa4159950 0x7f7fa415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7fa9b97640 1 -- 172.21.15.29:0/3057549046 >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 msgr2=0x7f7fa406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7fa9b97640 1 --2- 172.21.15.29:0/3057549046 >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 0x7f7fa406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7fa9b97640 1 -- 172.21.15.29:0/3057549046 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fa415d910 con 0x7f7fa4070450 2023-10-14T16:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.620+0000 7f7faab99640 1 --2- 172.21.15.29:0/3057549046 >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 0x7f7fa406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:01.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.621+0000 7f7fa9b97640 1 --2- 172.21.15.29:0/3057549046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4070450 0x7f7fa40708b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40b0120 tx=0x7f7f9402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de3aac6ec3361987 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.621+0000 7f7fa9396640 1 -- 172.21.15.29:0/3057549046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f940317c0 con 0x7f7fa4070450 2023-10-14T16:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.621+0000 7f7fa9396640 1 -- 172.21.15.29:0/3057549046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f94031920 con 0x7f7fa4070450 2023-10-14T16:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.621+0000 7f7fa9396640 1 -- 172.21.15.29:0/3057549046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f94031cb0 con 0x7f7fa4070450 2023-10-14T16:54:01.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 -- 172.21.15.29:0/3057549046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4070450 msgr2=0x7f7fa40708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 --2- 172.21.15.29:0/3057549046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4070450 0x7f7fa40708b0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f7fa40b0120 tx=0x7f7f9402b860 comp rx=0 tx=0).stop 2023-10-14T16:54:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 -- 172.21.15.29:0/3057549046 shutdown_connections 2023-10-14T16:54:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 --2- 172.21.15.29:0/3057549046 >> v2:172.21.15.186:3300/0 conn(0x7f7fa406c100 0x7f7fa406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 --2- 172.21.15.29:0/3057549046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4070450 0x7f7fa40708b0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 --2- 172.21.15.29:0/3057549046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa4159950 0x7f7fa415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 -- 172.21.15.29:0/3057549046 >> 172.21.15.29:0/3057549046 conn(0x7f7fa40ce6a0 msgr2=0x7f7fa40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 -- 172.21.15.29:0/3057549046 shutdown_connections 2023-10-14T16:54:01.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.622+0000 7f7fab39a640 1 -- 172.21.15.29:0/3057549046 wait complete. 2023-10-14T16:54:01.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.624+0000 7f7fab39a640 1 Processor -- start 2023-10-14T16:54:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.624+0000 7f7fab39a640 1 -- start start 2023-10-14T16:54:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 0x7f7fa41509b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 0x7f7fa41f0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa4151230 con 0x7f7fa4159950 2023-10-14T16:54:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa41f0540 con 0x7f7fa4070450 2023-10-14T16:54:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.625+0000 7f7fab39a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa40c3690 con 0x7f7fa406c100 2023-10-14T16:54:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46360/0 (socket says 172.21.15.29:46360) 2023-10-14T16:54:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 -- 172.21.15.29:0/2115494502 learned_addr learned my addr 172.21.15.29:0/2115494502 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:01.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7fa9b97640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 0x7f7fa41f0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faa398640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 0x7f7fa41509b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 -- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 msgr2=0x7f7fa41509b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:01.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 0x7f7fa41509b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 -- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 msgr2=0x7f7fa41f0000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 0x7f7fa41f0000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faab99640 1 -- 172.21.15.29:0/2115494502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f940094d0 con 0x7f7fa4159950 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7fa9b97640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 0x7f7fa41f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.626+0000 7f7faa398640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 0x7f7fa41509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:01.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7faab99640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00cd50 tx=0x7f7f9c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:01.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c00f680 con 0x7f7fa4159950 2023-10-14T16:54:01.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f9c00f800 con 0x7f7fa4159950 2023-10-14T16:54:01.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa40c3950 con 0x7f7fa4159950 2023-10-14T16:54:01.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f9c00fbb0 con 0x7f7fa4159950 2023-10-14T16:54:01.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.627+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa40c3f10 con 0x7f7fa4159950 2023-10-14T16:54:01.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.628+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa404e9b0 con 0x7f7fa4159950 2023-10-14T16:54:01.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.630+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7f9c00b040 con 0x7f7fa4159950 2023-10-14T16:54:01.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.631+0000 7f7f9affd640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 0x7f7f8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:01.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.631+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6484+0+0 (secure 0 0 0) 0x7f7f9c08a660 con 0x7f7fa4159950 2023-10-14T16:54:01.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.631+0000 7f7faa398640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 0x7f7f8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:01.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.632+0000 7f7faa398640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 0x7f7f8806d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0006fd0 tx=0x7f7fa0008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:01.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.636+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f9c055140 con 0x7f7fa4159950 2023-10-14T16:54:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:01.833+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7fa40b4620 con 0x7f7fa4159950 2023-10-14T16:54:02.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.371+0000 7f7f9affd640 1 -- 172.21.15.29:0/2115494502 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v53) v1 ==== 181+0+0 (secure 0 0 0) 0x7f7f9c058760 con 0x7f7fa4159950 2023-10-14T16:54:02.372 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:54:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.375+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 msgr2=0x7f7f8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.375+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 0x7f7f8806d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7fa0006fd0 tx=0x7f7fa0008040 comp rx=0 tx=0).stop 2023-10-14T16:54:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.375+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 msgr2=0x7f7fa40c3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.375+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f7f9c00cd50 tx=0x7f7f9c00a650 comp rx=0 tx=0).stop 2023-10-14T16:54:02.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 shutdown_connections 2023-10-14T16:54:02.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f7f8806af90 0x7f7f8806d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7fa4159950 0x7f7fa40c3150 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7fa4070450 0x7f7fa41f0000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 --2- 172.21.15.29:0/2115494502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7fa406c100 0x7f7fa41509b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.377+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 >> 172.21.15.29:0/2115494502 conn(0x7f7fa40ce6a0 msgr2=0x7f7fa40cd390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:02.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.378+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 shutdown_connections 2023-10-14T16:54:02.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.378+0000 7f7fab39a640 1 -- 172.21.15.29:0/2115494502 wait complete. 2023-10-14T16:54:02.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:54:02.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.631+0000 7f5b3d55a640 1 Processor -- start 2023-10-14T16:54:02.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.631+0000 7f5b3d55a640 1 -- start start 2023-10-14T16:54:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.633+0000 7f5b3d55a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 0x7f5b380cdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.633+0000 7f5b3d55a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 0x7f5b3815c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.633+0000 7f5b3d55a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b3806bbc0 con 0x7f5b3806cc60 2023-10-14T16:54:02.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.633+0000 7f5b3d55a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b3806bea0 con 0x7f5b380cfa00 2023-10-14T16:54:02.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.633+0000 7f5b3d55a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3806c180 con 0x7f5b380cca70 2023-10-14T16:54:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.634+0000 7f5b36ffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 0x7f5b380cdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.634+0000 7f5b367fc640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f5b3806cc60 0x7f5b380cf2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:53466/0 (socket says 172.21.15.29:53466) 2023-10-14T16:54:02.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.634+0000 7f5b367fc640 1 -- 172.21.15.29:0/4147567163 learned_addr learned my addr 172.21.15.29:0/4147567163 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:02.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.634+0000 7f5b377fe640 1 --2- 172.21.15.29:0/4147567163 >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 0x7f5b3815c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.635+0000 7f5b36ffd640 1 -- 172.21.15.29:0/4147567163 >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 msgr2=0x7f5b3815c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:02.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.635+0000 7f5b36ffd640 1 --2- 172.21.15.29:0/4147567163 >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 0x7f5b3815c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.635+0000 7f5b36ffd640 1 -- 172.21.15.29:0/4147567163 >> v1:172.21.15.186:6789/0 conn(0x7f5b3806cc60 legacy=0x7f5b380cf2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.635+0000 7f5b36ffd640 1 -- 172.21.15.29:0/4147567163 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3806c460 con 0x7f5b380cca70 2023-10-14T16:54:02.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.635+0000 7f5b377fe640 1 --2- 172.21.15.29:0/4147567163 >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 0x7f5b3815c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:54:02.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.636+0000 7f5b36ffd640 1 --2- 172.21.15.29:0/4147567163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 0x7f5b380cdd70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5b280097a0 tx=0x7f5b2802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdf268537377b2aa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.637+0000 7f5b35ffb640 1 -- 172.21.15.29:0/4147567163 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b2802d040 con 0x7f5b380cca70 2023-10-14T16:54:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.637+0000 7f5b35ffb640 1 -- 172.21.15.29:0/4147567163 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b28031810 con 0x7f5b380cca70 2023-10-14T16:54:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.637+0000 7f5b35ffb640 1 -- 172.21.15.29:0/4147567163 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b28031ba0 con 0x7f5b380cca70 2023-10-14T16:54:02.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 -- 172.21.15.29:0/4147567163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 msgr2=0x7f5b380cdd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/4147567163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 0x7f5b380cdd70 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f5b280097a0 tx=0x7f5b2802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 -- 172.21.15.29:0/4147567163 shutdown_connections 2023-10-14T16:54:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/4147567163 >> v2:172.21.15.186:3300/0 conn(0x7f5b380cfa00 0x7f5b3815c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/4147567163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b380cca70 0x7f5b380cdd70 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 -- 172.21.15.29:0/4147567163 >> 172.21.15.29:0/4147567163 conn(0x7f5b38099e30 msgr2=0x7f5b380d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 -- 172.21.15.29:0/4147567163 shutdown_connections 2023-10-14T16:54:02.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.638+0000 7f5b3d55a640 1 -- 172.21.15.29:0/4147567163 wait complete. 2023-10-14T16:54:02.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.641+0000 7f5b3d55a640 1 Processor -- start 2023-10-14T16:54:02.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.641+0000 7f5b3d55a640 1 -- start start 2023-10-14T16:54:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b3806cc60 0x7f5b380c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b380cfa00 0x7f5b3809e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3809a7a0 con 0x7f5b380cca70 2023-10-14T16:54:02.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b380d1460 con 0x7f5b380cfa00 2023-10-14T16:54:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.642+0000 7f5b3d55a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b3809e600 con 0x7f5b3806cc60 2023-10-14T16:54:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46372/0 (socket says 172.21.15.29:46372) 2023-10-14T16:54:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 -- 172.21.15.29:0/3176799024 learned_addr learned my addr 172.21.15.29:0/3176799024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:02.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b36ffd640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b3806cc60 0x7f5b380c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b377fe640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b380cfa00 0x7f5b3809e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 -- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b3806cc60 msgr2=0x7f5b380c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b3806cc60 0x7f5b380c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 -- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b380cfa00 msgr2=0x7f5b3809e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:02.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b380cfa00 0x7f5b3809e0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:02.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.643+0000 7f5b367fc640 1 -- 172.21.15.29:0/3176799024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b30009a70 con 0x7f5b380cca70 2023-10-14T16:54:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b367fc640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c400 tx=0x7f5b2000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b20005ce0 con 0x7f5b380cca70 2023-10-14T16:54:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b20005e40 con 0x7f5b380cca70 2023-10-14T16:54:02.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b200072d0 con 0x7f5b380cca70 2023-10-14T16:54:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b28009400 con 0x7f5b380cca70 2023-10-14T16:54:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.644+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b380c8ef0 con 0x7f5b380cca70 2023-10-14T16:54:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.645+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b381f24a0 con 0x7f5b380cca70 2023-10-14T16:54:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.646+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5b200074f0 con 0x7f5b380cca70 2023-10-14T16:54:02.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.647+0000 7f5b2f7fe640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 0x7f5b1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:02.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.647+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6046+0+0 (secure 0 0 0) 0x7f5b20088fe0 con 0x7f5b380cca70 2023-10-14T16:54:02.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.648+0000 7f5b36ffd640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 0x7f5b1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:02.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.648+0000 7f5b36ffd640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 0x7f5b1806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b28036000 tx=0x7f5b2803c040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:02.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.652+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b20053d90 con 0x7f5b380cca70 2023-10-14T16:54:02.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:02.848+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5b3804e160 con 0x7f5b380cca70 2023-10-14T16:54:03.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.378+0000 7f5b2f7fe640 1 -- 172.21.15.29:0/3176799024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v54) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5b20017020 con 0x7f5b380cca70 2023-10-14T16:54:03.382 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:54:03.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.383+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 msgr2=0x7f5b1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.383+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 0x7f5b1806d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5b28036000 tx=0x7f5b2803c040 comp rx=0 tx=0).stop 2023-10-14T16:54:03.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.383+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 msgr2=0x7f5b380d0f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.383+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7f5b2000c400 tx=0x7f5b2000c8c0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 shutdown_connections 2023-10-14T16:54:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f5b1806af90 0x7f5b1806d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b380cfa00 0x7f5b3809e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b380cca70 0x7f5b380d0f20 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 --2- 172.21.15.29:0/3176799024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b3806cc60 0x7f5b380c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 >> 172.21.15.29:0/3176799024 conn(0x7f5b38099e30 msgr2=0x7f5b3806ddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:03.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.387+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 shutdown_connections 2023-10-14T16:54:03.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.388+0000 7f5b3d55a640 1 -- 172.21.15.29:0/3176799024 wait complete. 2023-10-14T16:54:03.412 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T16:54:03.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.609+0000 7fda97c83640 1 Processor -- start 2023-10-14T16:54:03.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.609+0000 7fda97c83640 1 -- start start 2023-10-14T16:54:03.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda900adbe0 0x7fda900cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda90070fd0 0x7fda9006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fda900adfc0 0x7fda900b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda900cc600 con 0x7fda900adbe0 2023-10-14T16:54:03.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda90071470 con 0x7fda900adfc0 2023-10-14T16:54:03.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda97c83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda900cc870 con 0x7fda90070fd0 2023-10-14T16:54:03.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda95a1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda900adbe0 0x7fda900cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda95a1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda900adbe0 0x7fda900cd070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46384/0 (socket says 172.21.15.29:46384) 2023-10-14T16:54:03.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.611+0000 7fda95a1f640 1 -- 172.21.15.29:0/3144986147 learned_addr learned my addr 172.21.15.29:0/3144986147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda9521e640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda90070fd0 0x7fda9006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda96220640 1 --2- 172.21.15.29:0/3144986147 >> v2:172.21.15.186:3300/0 conn(0x7fda900adfc0 0x7fda900b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda95a1f640 1 -- 172.21.15.29:0/3144986147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda90070fd0 msgr2=0x7fda9006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:03.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda95a1f640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda90070fd0 0x7fda9006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda95a1f640 1 -- 172.21.15.29:0/3144986147 >> v2:172.21.15.186:3300/0 conn(0x7fda900adfc0 msgr2=0x7fda900b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda95a1f640 1 --2- 172.21.15.29:0/3144986147 >> v2:172.21.15.186:3300/0 conn(0x7fda900adfc0 0x7fda900b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.612+0000 7fda95a1f640 1 -- 172.21.15.29:0/3144986147 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda900ccb50 con 0x7fda900adbe0 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.613+0000 7fda95a1f640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adbe0 0x7fda900cd070 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fda900ca5f0 tx=0x7fda8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a6c58f674da3449 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.613+0000 7fda94a1d640 1 -- 172.21.15.29:0/3144986147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c031870 con 0x7fda900adbe0 2023-10-14T16:54:03.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.613+0000 7fda94a1d640 1 -- 172.21.15.29:0/3144986147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8c0319d0 con 0x7fda900adbe0 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.613+0000 7fda94a1d640 1 -- 172.21.15.29:0/3144986147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda8c031d60 con 0x7fda900adbe0 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 -- 172.21.15.29:0/3144986147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adbe0 msgr2=0x7fda900cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adbe0 0x7fda900cd070 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fda900ca5f0 tx=0x7fda8c02b910 comp rx=0 tx=0).stop 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 -- 172.21.15.29:0/3144986147 shutdown_connections 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 --2- 172.21.15.29:0/3144986147 >> v2:172.21.15.186:3300/0 conn(0x7fda900adfc0 0x7fda900b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda90070fd0 0x7fda9006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 --2- 172.21.15.29:0/3144986147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adbe0 0x7fda900cd070 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 -- 172.21.15.29:0/3144986147 >> 172.21.15.29:0/3144986147 conn(0x7fda900cb410 msgr2=0x7fda900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:03.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 -- 172.21.15.29:0/3144986147 shutdown_connections 2023-10-14T16:54:03.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.614+0000 7fda97c83640 1 -- 172.21.15.29:0/3144986147 wait complete. 2023-10-14T16:54:03.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.616+0000 7fda97c83640 1 Processor -- start 2023-10-14T16:54:03.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.616+0000 7fda97c83640 1 -- start start 2023-10-14T16:54:03.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 0x7fda90164490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda900adbe0 0x7fda9015b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda901649d0 con 0x7fda900adfc0 2023-10-14T16:54:03.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9015cdf0 con 0x7fda90070fd0 2023-10-14T16:54:03.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda97c83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda900c3710 con 0x7fda900adbe0 2023-10-14T16:54:03.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda96220640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda96220640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46392/0 (socket says 172.21.15.29:46392) 2023-10-14T16:54:03.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda96220640 1 -- 172.21.15.29:0/1089550890 learned_addr learned my addr 172.21.15.29:0/1089550890 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:03.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.617+0000 7fda95a1f640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 0x7fda90164490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda9521e640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda900adbe0 0x7fda9015b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 -- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda900adbe0 msgr2=0x7fda9015b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda900adbe0 0x7fda9015b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 -- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 msgr2=0x7fda90164490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 0x7fda90164490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 -- 172.21.15.29:0/1089550890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8c0094d0 con 0x7fda900adfc0 2023-10-14T16:54:03.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda95a1f640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 0x7fda90164490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:03.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda96220640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fda8800cd70 tx=0x7fda8800a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88003b90 con 0x7fda900adfc0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda88003cf0 con 0x7fda900adfc0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.618+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda88004240 con 0x7fda900adfc0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.619+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9015b7e0 con 0x7fda900adfc0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.619+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda900c3430 con 0x7fda900adfc0 2023-10-14T16:54:03.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.620+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9004e160 con 0x7fda900adfc0 2023-10-14T16:54:03.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.621+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fda880043a0 con 0x7fda900adfc0 2023-10-14T16:54:03.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.622+0000 7fda867fc640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 0x7fda7406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:03.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.622+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7fda88012070 con 0x7fda900adfc0 2023-10-14T16:54:03.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.622+0000 7fda95a1f640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 0x7fda7406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:03.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.623+0000 7fda95a1f640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 0x7fda7406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fda8c036000 tx=0x7fda8c03c040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:03.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.626+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda880550a0 con 0x7fda900adfc0 2023-10-14T16:54:03.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.825+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fda900b4910 con 0x7fda900adfc0 2023-10-14T16:54:03.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.831+0000 7fda867fc640 1 -- 172.21.15.29:0/1089550890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v13) v1 ==== 115+0+0 (secure 0 0 0) 0x7fda880586c0 con 0x7fda900adfc0 2023-10-14T16:54:03.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 msgr2=0x7fda7406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 0x7fda7406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fda8c036000 tx=0x7fda8c03c040 comp rx=0 tx=0).stop 2023-10-14T16:54:03.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 msgr2=0x7fda900c4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:03.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fda8800cd70 tx=0x7fda8800a660 comp rx=0 tx=0).stop 2023-10-14T16:54:03.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 shutdown_connections 2023-10-14T16:54:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fda7406af90 0x7fda7406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda900adfc0 0x7fda900c4750 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda900adbe0 0x7fda9015b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 --2- 172.21.15.29:0/1089550890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda90070fd0 0x7fda90164490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:03.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 >> 172.21.15.29:0/1089550890 conn(0x7fda900cb410 msgr2=0x7fda9006b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 shutdown_connections 2023-10-14T16:54:03.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:03.834+0000 7fda97c83640 1 -- 172.21.15.29:0/1089550890 wait complete. 2023-10-14T16:54:03.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T16:54:04.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.086+0000 7fc3e27d7640 1 Processor -- start 2023-10-14T16:54:04.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.086+0000 7fc3e27d7640 1 -- start start 2023-10-14T16:54:04.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e27d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 0x7fc3dc070b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e27d7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3dc0b03b0 0x7fc3dc0cbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e27d7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc071510 con 0x7fc3dc0710b0 2023-10-14T16:54:04.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e27d7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b1720 con 0x7fc3dc0b03b0 2023-10-14T16:54:04.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e27d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b1a00 con 0x7fc3dc159890 2023-10-14T16:54:04.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e0fd4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3dc0710b0 0x7fc3dc0afca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59622/0 (socket says 172.21.15.29:59622) 2023-10-14T16:54:04.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.088+0000 7fc3e0fd4640 1 -- 172.21.15.29:0/2430965875 learned_addr learned my addr 172.21.15.29:0/2430965875 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:04.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 0x7fc3dc070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3e1fd6640 1 --2- 172.21.15.29:0/2430965875 >> v2:172.21.15.186:3300/0 conn(0x7fc3dc0b03b0 0x7fc3dc0cbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 555339752 0 0) 0x7fc3cc001110 con 0x7fc3dc0710b0 2023-10-14T16:54:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3c8005730 con 0x7fc3dc0710b0 2023-10-14T16:54:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3363498599 0 0) 0x7fc3c8005730 con 0x7fc3dc0710b0 2023-10-14T16:54:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 msgr2=0x7fc3dc070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 0x7fc3dc070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 >> v2:172.21.15.186:3300/0 conn(0x7fc3dc0b03b0 msgr2=0x7fc3dc0cbab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 --2- 172.21.15.29:0/2430965875 >> v2:172.21.15.186:3300/0 conn(0x7fc3dc0b03b0 0x7fc3dc0cbab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc0b1ce0 con 0x7fc3dc0710b0 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc3cc001800 con 0x7fc3dc0710b0 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 >> v1:172.21.15.29:6789/0 conn(0x7fc3dc0710b0 legacy=0x7fc3dc0afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3c8005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c80069d0 0x7fc3c8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 0x7fc3c8007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b1720 con 0x7fc3c8005b80 2023-10-14T16:54:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc0b1a00 con 0x7fc3c80069d0 2023-10-14T16:54:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c800dc40 con 0x7fc3dc159890 2023-10-14T16:54:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.089+0000 7fc3e1fd6640 1 --2- 172.21.15.29:0/2430965875 >> v2:172.21.15.186:3300/0 conn(0x7fc3dc0b03b0 0x7fc3dc0cbab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3c8005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 -- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 msgr2=0x7fc3c8007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc159890 0x7fc3c8007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 -- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c80069d0 msgr2=0x7fc3c8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3c80069d0 0x7fc3c8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.090+0000 7fc3e17d5640 1 -- 172.21.15.29:0/2430965875 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c8005730 con 0x7fc3c8005b80 2023-10-14T16:54:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.091+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3c8005fa0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800ba00 tx=0x7fc3c8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84b46bbeaf4bf43b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.091+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8032630 con 0x7fc3c8005b80 2023-10-14T16:54:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d8032790 con 0x7fc3c8005b80 2023-10-14T16:54:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3d3fff640 1 -- 172.21.15.29:0/2430965875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8032b20 con 0x7fc3c8005b80 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 -- 172.21.15.29:0/2430965875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 msgr2=0x7fc3c8005fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/2430965875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3c8005fa0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fc3d800ba00 tx=0x7fc3c8007a00 comp rx=0 tx=0).stop 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e1fd6640 1 -- 172.21.15.29:0/2430965875 reap_dead start 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 -- 172.21.15.29:0/2430965875 shutdown_connections 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 -- 172.21.15.29:0/2430965875 >> 172.21.15.29:0/2430965875 conn(0x7fc3dc069da0 msgr2=0x7fc3dc06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 -- 172.21.15.29:0/2430965875 shutdown_connections 2023-10-14T16:54:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.092+0000 7fc3e27d7640 1 -- 172.21.15.29:0/2430965875 wait complete. 2023-10-14T16:54:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.094+0000 7fc3e27d7640 1 Processor -- start 2023-10-14T16:54:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.094+0000 7fc3e27d7640 1 -- start start 2023-10-14T16:54:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3dc14ac70 0x7fc3dc1492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc06fa60 0x7fc3dc06fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc070420 con 0x7fc3c8005b80 2023-10-14T16:54:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc14eb80 con 0x7fc3dc14ac70 2023-10-14T16:54:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.095+0000 7fc3e27d7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc14be10 con 0x7fc3dc06fa60 2023-10-14T16:54:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46418/0 (socket says 172.21.15.29:46418) 2023-10-14T16:54:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 -- 172.21.15.29:0/1912175355 learned_addr learned my addr 172.21.15.29:0/1912175355 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e1fd6640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc06fa60 0x7fc3dc06fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 -- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc06fa60 msgr2=0x7fc3dc06fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc06fa60 0x7fc3dc06fee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 -- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3dc14ac70 msgr2=0x7fc3dc1492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3dc14ac70 0x7fc3dc1492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.096+0000 7fc3e17d5640 1 -- 172.21.15.29:0/1912175355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc14df30 con 0x7fc3c8005b80 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3e17d5640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8035040 tx=0x7fc3d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d8004220 con 0x7fc3c8005b80 2023-10-14T16:54:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d80043a0 con 0x7fc3c8005b80 2023-10-14T16:54:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3d803bab0 con 0x7fc3c8005b80 2023-10-14T16:54:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc14c2a0 con 0x7fc3c8005b80 2023-10-14T16:54:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.097+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc14b200 con 0x7fc3c8005b80 2023-10-14T16:54:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.098+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc04e9b0 con 0x7fc3c8005b80 2023-10-14T16:54:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.099+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3d803bc10 con 0x7fc3c8005b80 2023-10-14T16:54:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.100+0000 7fc3d1ffb640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 0x7fc3c006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.100+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7fc3d80ac3a0 con 0x7fc3c8005b80 2023-10-14T16:54:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.101+0000 7fc3e0fd4640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 0x7fc3c006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.101+0000 7fc3e0fd4640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 0x7fc3c006d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0012b0 tx=0x7fc3cc0011e0 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.105+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3d8077320 con 0x7fc3c8005b80 2023-10-14T16:54:04.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.314+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fc3dc04e9b0 con 0x7fc3c8005b80 2023-10-14T16:54:04.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.321+0000 7fc3d1ffb640 1 -- 172.21.15.29:0/1912175355 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v14) v1 ==== 120+0+0 (secure 0 0 0) 0x7fc3d807a940 con 0x7fc3c8005b80 2023-10-14T16:54:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.324+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 msgr2=0x7fc3c006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.324+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 0x7fc3c006d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc0012b0 tx=0x7fc3cc0011e0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.324+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 msgr2=0x7fc3dc14d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.324+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fc3d8035040 tx=0x7fc3d8004730 comp rx=0 tx=0).stop 2023-10-14T16:54:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 shutdown_connections 2023-10-14T16:54:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc3c006afc0 0x7fc3c006d480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3dc06fa60 0x7fc3dc06fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3dc14ac70 0x7fc3dc1492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 --2- 172.21.15.29:0/1912175355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3c8005b80 0x7fc3dc14d9f0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 >> 172.21.15.29:0/1912175355 conn(0x7fc3dc069da0 msgr2=0x7fc3dc0cc170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 shutdown_connections 2023-10-14T16:54:04.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.325+0000 7fc3e27d7640 1 -- 172.21.15.29:0/1912175355 wait complete. 2023-10-14T16:54:04.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T16:54:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.552+0000 7f4c893fb640 1 Processor -- start 2023-10-14T16:54:04.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.552+0000 7f4c893fb640 1 -- start start 2023-10-14T16:54:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c8406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c840ad970 0x7f4c8415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4c8415ebc0 0x7f4c8415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8406c960 con 0x7f4c840ad970 2023-10-14T16:54:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8415b9d0 con 0x7f4c8415ebc0 2023-10-14T16:54:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c893fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8415bcb0 con 0x7f4c8406c040 2023-10-14T16:54:04.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c840ad970 0x7f4c8415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c840ad970 0x7f4c8415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46432/0 (socket says 172.21.15.29:46432) 2023-10-14T16:54:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 -- 172.21.15.29:0/2815483627 learned_addr learned my addr 172.21.15.29:0/2815483627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 -- 172.21.15.29:0/2815483627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 msgr2=0x7f4c8406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 --2- 172.21.15.29:0/2815483627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c8406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 -- 172.21.15.29:0/2815483627 >> v2:172.21.15.186:3300/0 conn(0x7f4c8415ebc0 msgr2=0x7f4c8415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.554+0000 7f4c837fe640 1 --2- 172.21.15.29:0/2815483627 >> v2:172.21.15.186:3300/0 conn(0x7f4c8415ebc0 0x7f4c8415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.555+0000 7f4c837fe640 1 -- 172.21.15.29:0/2815483627 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8415bf90 con 0x7f4c840ad970 2023-10-14T16:54:04.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c837fe640 1 --2- 172.21.15.29:0/2815483627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c8415e680 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009870 tx=0x7f4c7802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17de91aa4ae0b4ec server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c82ffd640 1 -- 172.21.15.29:0/2815483627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78031640 con 0x7f4c840ad970 2023-10-14T16:54:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c82ffd640 1 -- 172.21.15.29:0/2815483627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c780317a0 con 0x7f4c840ad970 2023-10-14T16:54:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c82ffd640 1 -- 172.21.15.29:0/2815483627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78031b30 con 0x7f4c840ad970 2023-10-14T16:54:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c893fb640 1 -- 172.21.15.29:0/2815483627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 msgr2=0x7f4c8415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.556+0000 7f4c893fb640 1 --2- 172.21.15.29:0/2815483627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c8415e680 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009870 tx=0x7f4c7802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 -- 172.21.15.29:0/2815483627 shutdown_connections 2023-10-14T16:54:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 --2- 172.21.15.29:0/2815483627 >> v2:172.21.15.186:3300/0 conn(0x7f4c8415ebc0 0x7f4c8415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 --2- 172.21.15.29:0/2815483627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c8415e680 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 --2- 172.21.15.29:0/2815483627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c8406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 -- 172.21.15.29:0/2815483627 >> 172.21.15.29:0/2815483627 conn(0x7f4c840ce6a0 msgr2=0x7f4c840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:04.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 -- 172.21.15.29:0/2815483627 shutdown_connections 2023-10-14T16:54:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.557+0000 7f4c893fb640 1 -- 172.21.15.29:0/2815483627 wait complete. 2023-10-14T16:54:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.558+0000 7f4c893fb640 1 Processor -- start 2023-10-14T16:54:04.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.558+0000 7f4c893fb640 1 -- start start 2023-10-14T16:54:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c84148ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c8415ebc0 0x7f4c840acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c84149020 con 0x7f4c840ad970 2023-10-14T16:54:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c840ca8d0 con 0x7f4c8415ebc0 2023-10-14T16:54:04.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.559+0000 7f4c893fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c840c5810 con 0x7f4c8406c040 2023-10-14T16:54:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46434/0 (socket says 172.21.15.29:46434) 2023-10-14T16:54:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 -- 172.21.15.29:0/1150409657 learned_addr learned my addr 172.21.15.29:0/1150409657 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c83fff640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c84148ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 -- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 msgr2=0x7f4c84148ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c84148ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 -- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c8415ebc0 msgr2=0x7f4c840acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c8415ebc0 0x7f4c840acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 -- 172.21.15.29:0/1150409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c780094d0 con 0x7f4c840ad970 2023-10-14T16:54:04.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.560+0000 7f4c837fe640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009870 tx=0x7f4c780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.561+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c7803ac60 con 0x7f4c840ad970 2023-10-14T16:54:04.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.561+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7803adc0 con 0x7f4c840ad970 2023-10-14T16:54:04.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.561+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c840c5af0 con 0x7f4c840ad970 2023-10-14T16:54:04.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.561+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c78044630 con 0x7f4c840ad970 2023-10-14T16:54:04.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.561+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c840c60b0 con 0x7f4c840ad970 2023-10-14T16:54:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.562+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8406b330 con 0x7f4c840ad970 2023-10-14T16:54:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.563+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c7802f050 con 0x7f4c840ad970 2023-10-14T16:54:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.564+0000 7f4c80ff9640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 0x7f4c7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.564+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f4c780ab8b0 con 0x7f4c840ad970 2023-10-14T16:54:04.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.564+0000 7f4c83fff640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 0x7f4c7006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:04.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.565+0000 7f4c83fff640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 0x7f4c7006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400a770 tx=0x7f4c74008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:04.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.569+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c78076710 con 0x7f4c840ad970 2023-10-14T16:54:04.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.762+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f4c840c62a0 con 0x7f4c840ad970 2023-10-14T16:54:04.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.769+0000 7f4c80ff9640 1 -- 172.21.15.29:0/1150409657 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4c78079d30 con 0x7f4c840ad970 2023-10-14T16:54:04.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.771+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 msgr2=0x7f4c7006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.771+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 0x7f4c7006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4c7400a770 tx=0x7f4c74008040 comp rx=0 tx=0).stop 2023-10-14T16:54:04.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.771+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 msgr2=0x7f4c840ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:04.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.771+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f4c78009870 tx=0x7f4c780046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 shutdown_connections 2023-10-14T16:54:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4c7006af90 0x7f4c7006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c8415ebc0 0x7f4c840acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c840ad970 0x7f4c840ac890 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 --2- 172.21.15.29:0/1150409657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c8406c040 0x7f4c84148ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 >> 172.21.15.29:0/1150409657 conn(0x7f4c840ce6a0 msgr2=0x7f4c8415de00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 shutdown_connections 2023-10-14T16:54:04.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:04.772+0000 7f4c893fb640 1 -- 172.21.15.29:0/1150409657 wait complete. 2023-10-14T16:54:04.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates' 2023-10-14T16:54:05.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.008+0000 7f3614de4640 1 Processor -- start 2023-10-14T16:54:05.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.009+0000 7f3614de4640 1 -- start start 2023-10-14T16:54:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.010+0000 7f3614de4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f36100712c0 0x7f3610070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.010+0000 7f3614de4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100b4620 con 0x7f361015aad0 2023-10-14T16:54:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.010+0000 7f3614de4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100b4340 con 0x7f3610070d70 2023-10-14T16:54:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.010+0000 7f3614de4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361006f520 con 0x7f36100712c0 2023-10-14T16:54:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f361015aad0 0x7f36100cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59626/0 (socket says 172.21.15.29:59626) 2023-10-14T16:54:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360f7fe640 1 -- 172.21.15.29:0/1961294614 learned_addr learned my addr 172.21.15.29:0/1961294614 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360ffff640 1 --2- 172.21.15.29:0/1961294614 >> v2:172.21.15.29:3300/0 conn(0x7f36100712c0 0x7f3610070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360ffff640 1 -- 172.21.15.29:0/1961294614 >> v1:172.21.15.186:6789/0 conn(0x7f3610070d70 legacy=0x7f361006bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360ffff640 1 -- 172.21.15.29:0/1961294614 >> v1:172.21.15.29:6789/0 conn(0x7f361015aad0 legacy=0x7f36100cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.011+0000 7f360ffff640 1 -- 172.21.15.29:0/1961294614 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361006f800 con 0x7f36100712c0 2023-10-14T16:54:05.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.012+0000 7f360ffff640 1 --2- 172.21.15.29:0/1961294614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36100712c0 0x7f3610070000 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f36080097a0 tx=0x7f360802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=754ceeeb46815324 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f360e7fc640 1 -- 172.21.15.29:0/1961294614 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608030610 con 0x7f36100712c0 2023-10-14T16:54:05.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f360e7fc640 1 -- 172.21.15.29:0/1961294614 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608030770 con 0x7f36100712c0 2023-10-14T16:54:05.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f360e7fc640 1 -- 172.21.15.29:0/1961294614 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3608030b00 con 0x7f36100712c0 2023-10-14T16:54:05.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 -- 172.21.15.29:0/1961294614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36100712c0 msgr2=0x7f3610070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 --2- 172.21.15.29:0/1961294614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36100712c0 0x7f3610070000 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f36080097a0 tx=0x7f360802b960 comp rx=0 tx=0).stop 2023-10-14T16:54:05.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 -- 172.21.15.29:0/1961294614 shutdown_connections 2023-10-14T16:54:05.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 --2- 172.21.15.29:0/1961294614 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36100712c0 0x7f3610070000 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 -- 172.21.15.29:0/1961294614 >> 172.21.15.29:0/1961294614 conn(0x7f36100b0a90 msgr2=0x7f36100ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:05.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.013+0000 7f3614de4640 1 -- 172.21.15.29:0/1961294614 shutdown_connections 2023-10-14T16:54:05.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.014+0000 7f3614de4640 1 -- 172.21.15.29:0/1961294614 wait complete. 2023-10-14T16:54:05.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.015+0000 7f3614de4640 1 Processor -- start 2023-10-14T16:54:05.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.015+0000 7f3614de4640 1 -- start start 2023-10-14T16:54:05.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 0x7f361015b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36100712c0 0x7f36101f50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361015b700 con 0x7f361015aad0 2023-10-14T16:54:05.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f361015b9e0 con 0x7f36100712c0 2023-10-14T16:54:05.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.016+0000 7f3614de4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f361015bcc0 con 0x7f3610070d70 2023-10-14T16:54:05.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46450/0 (socket says 172.21.15.29:46450) 2023-10-14T16:54:05.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 -- 172.21.15.29:0/423529721 learned_addr learned my addr 172.21.15.29:0/423529721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:05.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360f7fe640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 0x7f361015b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360effd640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36100712c0 0x7f36101f50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 -- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 msgr2=0x7f361015b120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 0x7f361015b120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 -- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36100712c0 msgr2=0x7f36101f50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36100712c0 0x7f36101f50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360ffff640 1 -- 172.21.15.29:0/423529721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608009400 con 0x7f361015aad0 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.017+0000 7f360f7fe640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 0x7f361015b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f360ffff640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f36080098c0 tx=0x7f3608004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f360803e040 con 0x7f361015aad0 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608039c10 con 0x7f361015aad0 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36080443f0 con 0x7f361015aad0 2023-10-14T16:54:05.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f361015bfa0 con 0x7f361015aad0 2023-10-14T16:54:05.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.018+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36100c4390 con 0x7f361015aad0 2023-10-14T16:54:05.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.019+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f361004e9b0 con 0x7f361015aad0 2023-10-14T16:54:05.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.020+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f360804d3b0 con 0x7f361015aad0 2023-10-14T16:54:05.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.021+0000 7f3607fff640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.021+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f36080abf60 con 0x7f361015aad0 2023-10-14T16:54:05.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.022+0000 7f360f7fe640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.022+0000 7f360f7fe640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 0x7f35f006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3600005c00 tx=0x7f3600009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.027+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3608076dc0 con 0x7f361015aad0 2023-10-14T16:54:05.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.297+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]} v 0) v1 -- 0x7f36100cf520 con 0x7f361015aad0 2023-10-14T16:54:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.408+0000 7f3607fff640 1 -- 172.21.15.29:0/423529721 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_concurrent_snap_creates"]}]=0 v494) v1 ==== 158+0+0 (secure 0 0 0) 0x7f360807a3e0 con 0x7f361015aad0 2023-10-14T16:54:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 msgr2=0x7f35f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 0x7f35f006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3600005c00 tx=0x7f3600009040 comp rx=0 tx=0).stop 2023-10-14T16:54:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 msgr2=0x7f36100c3e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f36080098c0 tx=0x7f3608004730 comp rx=0 tx=0).stop 2023-10-14T16:54:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 shutdown_connections 2023-10-14T16:54:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35f006af90 0x7f35f006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f361015aad0 0x7f36100c3e50 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36100712c0 0x7f36101f50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 --2- 172.21.15.29:0/423529721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3610070d70 0x7f361015b120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.411+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 >> 172.21.15.29:0/423529721 conn(0x7f36100b0a90 msgr2=0x7f36100b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.412+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 shutdown_connections 2023-10-14T16:54:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.412+0000 7f3614de4640 1 -- 172.21.15.29:0/423529721 wait complete. 2023-10-14T16:54:05.433 INFO:tasks.cephfs_test_runner:Test concurrent snap creates in same file-system without db issues ... ok 2023-10-14T16:54:05.434 INFO:tasks.cephfs_test_runner:Starting test: test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test exisitence of multiple scheduled snapshots 2023-10-14T16:54:05.434 INFO:tasks.cephfs_test_runner:test_multi_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:54:05.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:54:05.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.664+0000 7f539662e640 1 Processor -- start 2023-10-14T16:54:05.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.664+0000 7f539662e640 1 -- start start 2023-10-14T16:54:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f539662e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f539015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f539662e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f539006c810 0x7f53900708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f539662e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5390070df0 con 0x7f5390070450 2023-10-14T16:54:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f539662e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539015e490 con 0x7f539006c810 2023-10-14T16:54:05.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f539662e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539015aef0 con 0x7f5390159950 2023-10-14T16:54:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f5394e2b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5390070450 0x7f539006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59640/0 (socket says 172.21.15.29:59640) 2023-10-14T16:54:05.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.666+0000 7f5394e2b640 1 -- 172.21.15.29:0/3193913502 learned_addr learned my addr 172.21.15.29:0/3193913502 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:05.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5395e2d640 1 --2- 172.21.15.29:0/3193913502 >> v2:172.21.15.29:3300/0 conn(0x7f539006c810 0x7f53900708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f539562c640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f539015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5387fff640 1 -- 172.21.15.29:0/3193913502 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584277338 0 0) 0x7f5390070df0 con 0x7f5390070450 2023-10-14T16:54:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5387fff640 1 -- 172.21.15.29:0/3193913502 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f5390070450 2023-10-14T16:54:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5395e2d640 1 -- 172.21.15.29:0/3193913502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 msgr2=0x7f539015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5395e2d640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f539015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5395e2d640 1 -- 172.21.15.29:0/3193913502 >> v1:172.21.15.29:6789/0 conn(0x7f5390070450 legacy=0x7f539006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.667+0000 7f5395e2d640 1 -- 172.21.15.29:0/3193913502 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539006cc80 con 0x7f539006c810 2023-10-14T16:54:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f5395e2d640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900708b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5388009d90 tx=0x7f538802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ea0c3cadeab43d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f5387fff640 1 -- 172.21.15.29:0/3193913502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538802cbd0 con 0x7f539006c810 2023-10-14T16:54:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f5387fff640 1 -- 172.21.15.29:0/3193913502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538802cd30 con 0x7f539006c810 2023-10-14T16:54:05.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f5387fff640 1 -- 172.21.15.29:0/3193913502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538802cea0 con 0x7f539006c810 2023-10-14T16:54:05.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f539662e640 1 -- 172.21.15.29:0/3193913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 msgr2=0x7f53900708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.669+0000 7f539662e640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900708b0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f5388009d90 tx=0x7f538802be20 comp rx=0 tx=0).stop 2023-10-14T16:54:05.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 -- 172.21.15.29:0/3193913502 shutdown_connections 2023-10-14T16:54:05.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900708b0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 --2- 172.21.15.29:0/3193913502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f539015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 -- 172.21.15.29:0/3193913502 >> 172.21.15.29:0/3193913502 conn(0x7f53900ce6a0 msgr2=0x7f53900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 -- 172.21.15.29:0/3193913502 shutdown_connections 2023-10-14T16:54:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.670+0000 7f539662e640 1 -- 172.21.15.29:0/3193913502 wait complete. 2023-10-14T16:54:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.672+0000 7f539662e640 1 Processor -- start 2023-10-14T16:54:05.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.672+0000 7f539662e640 1 -- start start 2023-10-14T16:54:05.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5390070450 0x7f5390155e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f53900c9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900ad590 con 0x7f539006c810 2023-10-14T16:54:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53900c22c0 con 0x7f5390070450 2023-10-14T16:54:05.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539662e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539015bee0 con 0x7f5390159950 2023-10-14T16:54:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46472/0 (socket says 172.21.15.29:46472) 2023-10-14T16:54:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 -- 172.21.15.29:0/2498172864 learned_addr learned my addr 172.21.15.29:0/2498172864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:05.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 -- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 msgr2=0x7f53900c9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f53900c9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 -- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5390070450 msgr2=0x7f5390155e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.673+0000 7f539562c640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5390070450 0x7f5390155e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f539562c640 1 -- 172.21.15.29:0/2498172864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388009a70 con 0x7f539006c810 2023-10-14T16:54:05.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f539562c640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e9b0 tx=0x7f538c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c00c9a0 con 0x7f539006c810 2023-10-14T16:54:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c00cb00 con 0x7f539006c810 2023-10-14T16:54:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53900acac0 con 0x7f539006c810 2023-10-14T16:54:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c013470 con 0x7f539006c810 2023-10-14T16:54:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.674+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539015b5b0 con 0x7f539006c810 2023-10-14T16:54:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.675+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539004e9b0 con 0x7f539006c810 2023-10-14T16:54:05.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.676+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f538c0135d0 con 0x7f539006c810 2023-10-14T16:54:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.677+0000 7f5385ffb640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.678+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f538c089a80 con 0x7f539006c810 2023-10-14T16:54:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.678+0000 7f5394e2b640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.679+0000 7f5394e2b640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 0x7f537406d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f53800011e0 tx=0x7f5380007910 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.682+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f538c054960 con 0x7f539006c810 2023-10-14T16:54:05.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.878+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53900b4620 con 0x7f539006c810 2023-10-14T16:54:05.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.880+0000 7f5385ffb640 1 -- 172.21.15.29:0/2498172864 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v54) v1 ==== 74+0+13983 (secure 0 0 0) 0x7f538c021090 con 0x7f539006c810 2023-10-14T16:54:05.881 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:05.881 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":54,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:54:03.372362+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3003570167":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3659680710":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/3113675785":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/4149980298":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6833/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/3481939635":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/347213673":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/1900972994":"2023-10-15T16:41:58.134928+0000","172.21.15.29:6835/2277792616":"2023-10-15T16:54:00.345356+0000","172.21.15.29:0/2383868706":"2023-10-15T16:41:58.134928+0000","172.21.15.29:0/2733984810":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2615927594":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6832/115188615":"2023-10-15T16:53:56.592878+0000","172.21.15.29:0/2453699906":"2023-10-15T16:53:56.592878+0000","172.21.15.29:6837/2277792616":"2023-10-15T16:54:00.345356+0000","172.21.15.29:0/3327634503":"2023-10-15T16:53:56.592878+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:54:05.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.882+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 msgr2=0x7f537406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.882+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 0x7f537406d480 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f53800011e0 tx=0x7f5380007910 comp rx=0 tx=0).stop 2023-10-14T16:54:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 msgr2=0x7f53900acf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f538c00e9b0 tx=0x7f538c00ee70 comp rx=0 tx=0).stop 2023-10-14T16:54:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 shutdown_connections 2023-10-14T16:54:05.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5390159950 0x7f53900c9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5390070450 0x7f5390155e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 --2- 172.21.15.29:0/2498172864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539006c810 0x7f53900acf00 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 >> 172.21.15.29:0/2498172864 conn(0x7f53900ce6a0 msgr2=0x7f53900cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:05.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 shutdown_connections 2023-10-14T16:54:05.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:05.883+0000 7f539662e640 1 -- 172.21.15.29:0/2498172864 wait complete. 2023-10-14T16:54:05.903 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:54:05.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:54:06.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.131+0000 7f3af66ca640 1 Processor -- start 2023-10-14T16:54:06.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.131+0000 7f3af66ca640 1 -- start start 2023-10-14T16:54:06.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.132+0000 7f3af66ca640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3af0159ad0 0x7f3af0070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af66ca640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015d650 0x7f3af006aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af66ca640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af00cd260 con 0x7f3af006b3e0 2023-10-14T16:54:06.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af66ca640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af0071470 con 0x7f3af015d650 2023-10-14T16:54:06.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af66ca640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3af015de30 con 0x7f3af0159ad0 2023-10-14T16:54:06.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af4ec7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015d650 0x7f3af006aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af4ec7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3af015d650 0x7f3af006aea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46486/0 (socket says 172.21.15.29:46486) 2023-10-14T16:54:06.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.133+0000 7f3af4ec7640 1 -- 172.21.15.29:0/2916580888 learned_addr learned my addr 172.21.15.29:0/2916580888 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:06.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.134+0000 7f3af56c8640 1 --2- 172.21.15.29:0/2916580888 >> v2:172.21.15.186:3300/0 conn(0x7f3af0159ad0 0x7f3af0070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.134+0000 7f3af4ec7640 1 -- 172.21.15.29:0/2916580888 >> v2:172.21.15.186:3300/0 conn(0x7f3af0159ad0 msgr2=0x7f3af0070f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:06.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.134+0000 7f3af4ec7640 1 --2- 172.21.15.29:0/2916580888 >> v2:172.21.15.186:3300/0 conn(0x7f3af0159ad0 0x7f3af0070f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.134+0000 7f3af4ec7640 1 -- 172.21.15.29:0/2916580888 >> v1:172.21.15.186:6789/0 conn(0x7f3af006b3e0 legacy=0x7f3af00c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:06.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.134+0000 7f3af4ec7640 1 -- 172.21.15.29:0/2916580888 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af015e110 con 0x7f3af015d650 2023-10-14T16:54:06.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.135+0000 7f3af4ec7640 1 --2- 172.21.15.29:0/2916580888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015d650 0x7f3af006aea0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3af006d010 tx=0x7f3ae402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47f81a6143389375 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.135+0000 7f3aeffff640 1 -- 172.21.15.29:0/2916580888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4031870 con 0x7f3af015d650 2023-10-14T16:54:06.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.135+0000 7f3aeffff640 1 -- 172.21.15.29:0/2916580888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae40319d0 con 0x7f3af015d650 2023-10-14T16:54:06.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.135+0000 7f3aeffff640 1 -- 172.21.15.29:0/2916580888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4031d60 con 0x7f3af015d650 2023-10-14T16:54:06.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 -- 172.21.15.29:0/2916580888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015d650 msgr2=0x7f3af006aea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 --2- 172.21.15.29:0/2916580888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015d650 0x7f3af006aea0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f3af006d010 tx=0x7f3ae402b910 comp rx=0 tx=0).stop 2023-10-14T16:54:06.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 -- 172.21.15.29:0/2916580888 shutdown_connections 2023-10-14T16:54:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 --2- 172.21.15.29:0/2916580888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af015d650 0x7f3af006aea0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 --2- 172.21.15.29:0/2916580888 >> v2:172.21.15.186:3300/0 conn(0x7f3af0159ad0 0x7f3af0070f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 -- 172.21.15.29:0/2916580888 >> 172.21.15.29:0/2916580888 conn(0x7f3af00b0170 msgr2=0x7f3af00b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 -- 172.21.15.29:0/2916580888 shutdown_connections 2023-10-14T16:54:06.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.136+0000 7f3af66ca640 1 -- 172.21.15.29:0/2916580888 wait complete. 2023-10-14T16:54:06.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.138+0000 7f3af66ca640 1 Processor -- start 2023-10-14T16:54:06.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.138+0000 7f3af66ca640 1 -- start start 2023-10-14T16:54:06.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3af006b3e0 0x7f3af014ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3af015d650 0x7f3af00c7190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af014efd0 con 0x7f3af0159ad0 2023-10-14T16:54:06.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3af014ff30 con 0x7f3af015d650 2023-10-14T16:54:06.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af66ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3af015b5f0 con 0x7f3af006b3e0 2023-10-14T16:54:06.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af4ec7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af4ec7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46502/0 (socket says 172.21.15.29:46502) 2023-10-14T16:54:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.139+0000 7f3af4ec7640 1 -- 172.21.15.29:0/587245679 learned_addr learned my addr 172.21.15.29:0/587245679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af56c8640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3af006b3e0 0x7f3af014ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af5ec9640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3af015d650 0x7f3af00c7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 -- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3af006b3e0 msgr2=0x7f3af014ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3af006b3e0 0x7f3af014ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 -- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3af015d650 msgr2=0x7f3af00c7190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3af015d650 0x7f3af00c7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 -- 172.21.15.29:0/587245679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae40094d0 con 0x7f3af0159ad0 2023-10-14T16:54:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.140+0000 7f3af4ec7640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402b950 tx=0x7f3ae40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.141+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae403ac60 con 0x7f3af0159ad0 2023-10-14T16:54:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.141+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ae403adc0 con 0x7f3af0159ad0 2023-10-14T16:54:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.141+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af015b8d0 con 0x7f3af0159ad0 2023-10-14T16:54:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.141+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ae4044630 con 0x7f3af0159ad0 2023-10-14T16:54:06.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.141+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af015be90 con 0x7f3af0159ad0 2023-10-14T16:54:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.142+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3af004e9b0 con 0x7f3af0159ad0 2023-10-14T16:54:06.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.143+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3ae404d450 con 0x7f3af0159ad0 2023-10-14T16:54:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.144+0000 7f3aedffb640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 0x7f3adc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.145+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f3ae402d040 con 0x7f3af0159ad0 2023-10-14T16:54:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.145+0000 7f3af56c8640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 0x7f3adc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.145+0000 7f3af56c8640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 0x7f3adc06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0006fd0 tx=0x7f3ae0008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.149+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ae40b2080 con 0x7f3af0159ad0 2023-10-14T16:54:06.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.348+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3af00b4600 con 0x7f3af0159ad0 2023-10-14T16:54:06.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.349+0000 7f3aedffb640 1 -- 172.21.15.29:0/587245679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v54) v1 ==== 93+0+10 (secure 0 0 0) 0x7f3ae4035020 con 0x7f3af0159ad0 2023-10-14T16:54:06.350 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:54:06.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 msgr2=0x7f3adc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 0x7f3adc06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3ae0006fd0 tx=0x7f3ae0008040 comp rx=0 tx=0).stop 2023-10-14T16:54:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 msgr2=0x7f3af015ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3ae402b950 tx=0x7f3ae40046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 shutdown_connections 2023-10-14T16:54:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f3adc06af90 0x7f3adc06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3af015d650 0x7f3af00c7190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3af0159ad0 0x7f3af015ab00 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 --2- 172.21.15.29:0/587245679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3af006b3e0 0x7f3af014ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 >> 172.21.15.29:0/587245679 conn(0x7f3af00b0170 msgr2=0x7f3af006b8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:06.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 shutdown_connections 2023-10-14T16:54:06.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.352+0000 7f3af66ca640 1 -- 172.21.15.29:0/587245679 wait complete. 2023-10-14T16:54:06.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-14T16:54:06.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.594+0000 7f52df59e640 1 Processor -- start 2023-10-14T16:54:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.594+0000 7f52df59e640 1 -- start start 2023-10-14T16:54:06.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.595+0000 7f52df59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e01599c0 0x7f52e00ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.595+0000 7f52df59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52e00cda90 0x7f52e006f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.595+0000 7f52df59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52e006bab0 con 0x7f52e00ca6f0 2023-10-14T16:54:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.595+0000 7f52df59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52e006f300 con 0x7f52e00cda90 2023-10-14T16:54:06.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.595+0000 7f52df59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e006bd90 con 0x7f52e01599c0 2023-10-14T16:54:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52e00cda90 0x7f52e006f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52e00cda90 0x7f52e006f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46504/0 (socket says 172.21.15.29:46504) 2023-10-14T16:54:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 -- 172.21.15.29:0/3105203241 learned_addr learned my addr 172.21.15.29:0/3105203241 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 -- 172.21.15.29:0/3105203241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e01599c0 msgr2=0x7f52e00ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 --2- 172.21.15.29:0/3105203241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e01599c0 0x7f52e00ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 -- 172.21.15.29:0/3105203241 >> v1:172.21.15.186:6789/0 conn(0x7f52e00ca6f0 legacy=0x7f52e00cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:06.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.596+0000 7f52ded9d640 1 -- 172.21.15.29:0/3105203241 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52e006c070 con 0x7f52e00cda90 2023-10-14T16:54:06.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.597+0000 7f52ded9d640 1 --2- 172.21.15.29:0/3105203241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e00cda90 0x7f52e006f750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e50 tx=0x7f52d802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4179c8523eb72776 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.597+0000 7f52ddd9b640 1 -- 172.21.15.29:0/3105203241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d802c9f0 con 0x7f52e00cda90 2023-10-14T16:54:06.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52ddd9b640 1 -- 172.21.15.29:0/3105203241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d802cb50 con 0x7f52e00cda90 2023-10-14T16:54:06.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52ddd9b640 1 -- 172.21.15.29:0/3105203241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d80343f0 con 0x7f52e00cda90 2023-10-14T16:54:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 -- 172.21.15.29:0/3105203241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e00cda90 msgr2=0x7f52e006f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 --2- 172.21.15.29:0/3105203241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e00cda90 0x7f52e006f750 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009e50 tx=0x7f52d802be50 comp rx=0 tx=0).stop 2023-10-14T16:54:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 -- 172.21.15.29:0/3105203241 shutdown_connections 2023-10-14T16:54:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 --2- 172.21.15.29:0/3105203241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e00cda90 0x7f52e006f750 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 --2- 172.21.15.29:0/3105203241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e01599c0 0x7f52e00ca1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 -- 172.21.15.29:0/3105203241 >> 172.21.15.29:0/3105203241 conn(0x7f52e00b0a90 msgr2=0x7f52e00ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 -- 172.21.15.29:0/3105203241 shutdown_connections 2023-10-14T16:54:06.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.598+0000 7f52df59e640 1 -- 172.21.15.29:0/3105203241 wait complete. 2023-10-14T16:54:06.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.600+0000 7f52df59e640 1 Processor -- start 2023-10-14T16:54:06.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.600+0000 7f52df59e640 1 -- start start 2023-10-14T16:54:06.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52e00ca6f0 0x7f52e014f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e00cda90 0x7f52e01552a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015bc20 con 0x7f52e01599c0 2023-10-14T16:54:06.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e014fd40 con 0x7f52e00ca6f0 2023-10-14T16:54:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.601+0000 7f52df59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52e015c0b0 con 0x7f52e00cda90 2023-10-14T16:54:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46514/0 (socket says 172.21.15.29:46514) 2023-10-14T16:54:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 -- 172.21.15.29:0/2426764856 learned_addr learned my addr 172.21.15.29:0/2426764856 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:06.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52de59c640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52e00ca6f0 0x7f52e014f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 -- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e00cda90 msgr2=0x7f52e01552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52d5d9b640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e00cda90 0x7f52e01552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e00cda90 0x7f52e01552a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 -- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52e00ca6f0 msgr2=0x7f52e014f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52e00ca6f0 0x7f52e014f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 -- 172.21.15.29:0/2426764856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d8009b30 con 0x7f52e01599c0 2023-10-14T16:54:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.602+0000 7f52ded9d640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f52d8036040 tx=0x7f52d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.603+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d803e450 con 0x7f52e01599c0 2023-10-14T16:54:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.603+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d803e5d0 con 0x7f52e01599c0 2023-10-14T16:54:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.603+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52d803e980 con 0x7f52e01599c0 2023-10-14T16:54:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.603+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e015b010 con 0x7f52e01599c0 2023-10-14T16:54:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.603+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e015b5d0 con 0x7f52e01599c0 2023-10-14T16:54:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.604+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52e004e9b0 con 0x7f52e01599c0 2023-10-14T16:54:06.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.605+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f52d803eae0 con 0x7f52e01599c0 2023-10-14T16:54:06.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.606+0000 7f52d77fe640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 0x7f52c006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:06.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.606+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f52d80ac690 con 0x7f52e01599c0 2023-10-14T16:54:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.606+0000 7f52de59c640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 0x7f52c006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.607+0000 7f52de59c640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 0x7f52c006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52d000b810 tx=0x7f52d0008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:06.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.610+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52d8077600 con 0x7f52e01599c0 2023-10-14T16:54:06.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:06.876+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f52e00cae10 con 0x7f52e01599c0 2023-10-14T16:54:07.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.423+0000 7f52d77fe640 1 -- 172.21.15.29:0/2426764856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v496) v1 ==== 157+0+0 (secure 0 0 0) 0x7f52d807ac20 con 0x7f52e01599c0 2023-10-14T16:54:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 msgr2=0x7f52c006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 0x7f52c006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f52d000b810 tx=0x7f52d0008040 comp rx=0 tx=0).stop 2023-10-14T16:54:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 msgr2=0x7f52e01557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f52d8036040 tx=0x7f52d8004a40 comp rx=0 tx=0).stop 2023-10-14T16:54:07.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 shutdown_connections 2023-10-14T16:54:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f52c006af90 0x7f52c006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52e01599c0 0x7f52e01557e0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52e00cda90 0x7f52e01552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 --2- 172.21.15.29:0/2426764856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52e00ca6f0 0x7f52e014f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 >> 172.21.15.29:0/2426764856 conn(0x7f52e00b0a90 msgr2=0x7f52e00adff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 shutdown_connections 2023-10-14T16:54:07.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.427+0000 7f52df59e640 1 -- 172.21.15.29:0/2426764856 wait complete. 2023-10-14T16:54:07.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:54:07.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.679+0000 7f8a2e9c9640 1 Processor -- start 2023-10-14T16:54:07.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.679+0000 7f8a2e9c9640 1 -- start start 2023-10-14T16:54:07.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2e9c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8a2815ad70 0x7f8a28069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2e9c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a280c9ca0 0x7f8a2806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2e9c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a280cb030 con 0x7f8a280ca3b0 2023-10-14T16:54:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2e9c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280c91c0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2e9c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a280c94a0 con 0x7f8a2815ad70 2023-10-14T16:54:07.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a277fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a280c9ca0 0x7f8a2806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2cf66640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8a280ca3b0 0x7f8a280ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59644/0 (socket says 172.21.15.29:59644) 2023-10-14T16:54:07.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a2cf66640 1 -- 172.21.15.29:0/188572612 learned_addr learned my addr 172.21.15.29:0/188572612 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.681+0000 7f8a27fff640 1 --2- 172.21.15.29:0/188572612 >> v2:172.21.15.186:3300/0 conn(0x7f8a2815ad70 0x7f8a28069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.682+0000 7f8a277fe640 1 -- 172.21.15.29:0/188572612 >> v2:172.21.15.186:3300/0 conn(0x7f8a2815ad70 msgr2=0x7f8a28069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.682+0000 7f8a277fe640 1 --2- 172.21.15.29:0/188572612 >> v2:172.21.15.186:3300/0 conn(0x7f8a2815ad70 0x7f8a28069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.682+0000 7f8a277fe640 1 -- 172.21.15.29:0/188572612 >> v1:172.21.15.29:6789/0 conn(0x7f8a280ca3b0 legacy=0x7f8a280ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.682+0000 7f8a277fe640 1 -- 172.21.15.29:0/188572612 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a280c9780 con 0x7f8a280c9ca0 2023-10-14T16:54:07.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.683+0000 7f8a277fe640 1 --2- 172.21.15.29:0/188572612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a2806a2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8a280cf780 tx=0x7f8a1002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d663c6fc07ade53e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a26ffd640 1 -- 172.21.15.29:0/188572612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10031830 con 0x7f8a280c9ca0 2023-10-14T16:54:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a26ffd640 1 -- 172.21.15.29:0/188572612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a10031990 con 0x7f8a280c9ca0 2023-10-14T16:54:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a26ffd640 1 -- 172.21.15.29:0/188572612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a10031d20 con 0x7f8a280c9ca0 2023-10-14T16:54:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/188572612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 msgr2=0x7f8a2806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/188572612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a2806a2e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8a280cf780 tx=0x7f8a1002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/188572612 shutdown_connections 2023-10-14T16:54:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/188572612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a2806a2e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/188572612 >> v2:172.21.15.186:3300/0 conn(0x7f8a2815ad70 0x7f8a28069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/188572612 >> 172.21.15.29:0/188572612 conn(0x7f8a280ce9b0 msgr2=0x7f8a280cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/188572612 shutdown_connections 2023-10-14T16:54:07.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.684+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/188572612 wait complete. 2023-10-14T16:54:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.686+0000 7f8a2e9c9640 1 Processor -- start 2023-10-14T16:54:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.686+0000 7f8a2e9c9640 1 -- start start 2023-10-14T16:54:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a280ca3b0 0x7f8a28146a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a2815ad70 0x7f8a28146f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280c6720 con 0x7f8a280c9ca0 2023-10-14T16:54:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280c6a00 con 0x7f8a280ca3b0 2023-10-14T16:54:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.687+0000 7f8a2e9c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a280c6ce0 con 0x7f8a2815ad70 2023-10-14T16:54:07.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46540/0 (socket says 172.21.15.29:46540) 2023-10-14T16:54:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 -- 172.21.15.29:0/3736554403 learned_addr learned my addr 172.21.15.29:0/3736554403 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 -- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a2815ad70 msgr2=0x7f8a28146f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a2cf66640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a2815ad70 0x7f8a28146f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a2815ad70 0x7f8a28146f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a277fe640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a280ca3b0 0x7f8a28146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 -- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a280ca3b0 msgr2=0x7f8a28146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a280ca3b0 0x7f8a28146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 -- 172.21.15.29:0/3736554403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a10009400 con 0x7f8a280c9ca0 2023-10-14T16:54:07.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.688+0000 7f8a27fff640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8a18007e40 tx=0x7f8a1800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.689+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a18014620 con 0x7f8a280c9ca0 2023-10-14T16:54:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.689+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a1800ee90 con 0x7f8a280c9ca0 2023-10-14T16:54:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.689+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a180149c0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.689+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a280c6fc0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.689+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a280c87c0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.690+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2804e160 con 0x7f8a280c9ca0 2023-10-14T16:54:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.691+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a18014b20 con 0x7f8a280c9ca0 2023-10-14T16:54:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.692+0000 7f8a24ff9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 0x7f8a0806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.692+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f8a1808a0c0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.692+0000 7f8a277fe640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 0x7f8a0806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:07.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.693+0000 7f8a277fe640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 0x7f8a0806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8a10036000 tx=0x7f8a10008030 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.697+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a180550b0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.897+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f8a280cf1a0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.904+0000 7f8a24ff9640 1 -- 172.21.15.29:0/3736554403 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f8a180586d0 con 0x7f8a280c9ca0 2023-10-14T16:54:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 msgr2=0x7f8a0806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 0x7f8a0806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f8a10036000 tx=0x7f8a10008030 comp rx=0 tx=0).stop 2023-10-14T16:54:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 msgr2=0x7f8a280c8280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:07.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8a18007e40 tx=0x7f8a1800e4b0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 shutdown_connections 2023-10-14T16:54:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a0806af90 0x7f8a0806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a2815ad70 0x7f8a28146f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a280ca3b0 0x7f8a28146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 --2- 172.21.15.29:0/3736554403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a280c9ca0 0x7f8a280c8280 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 >> 172.21.15.29:0/3736554403 conn(0x7f8a280ce9b0 msgr2=0x7f8a28070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 shutdown_connections 2023-10-14T16:54:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:07.907+0000 7f8a2e9c9640 1 -- 172.21.15.29:0/3736554403 wait complete. 2023-10-14T16:54:07.927 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:54:07.928 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:54:08.055 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:54:08.179 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:54:08.180 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:54:08.180 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:54:08.244 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:54:08.245 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:54:08.245 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:54:08.305 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:54:08.306 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:54:08.306 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:54:08.371 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:54:08.372 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:54:08.372 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:54:08.435 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:54:08.436 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:54:08.436 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:54:08.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.672+0000 7f30d058a640 1 Processor -- start 2023-10-14T16:54:08.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.672+0000 7f30d058a640 1 -- start start 2023-10-14T16:54:08.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30d058a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30c80b0410 0x7f30c80b0840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30d058a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c815ce30 con 0x7f30c8158ed0 2023-10-14T16:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30d058a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c815d9d0 con 0x7f30c80ade70 2023-10-14T16:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30d058a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c80b0d80 con 0x7f30c80b0410 2023-10-14T16:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30ce326640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30c8158ed0 0x7f30c80ce1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59646/0 (socket says 172.21.15.29:59646) 2023-10-14T16:54:08.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.674+0000 7f30ce326640 1 -- 172.21.15.29:0/1540766078 learned_addr learned my addr 172.21.15.29:0/1540766078 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:08.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30ceb27640 1 --2- 172.21.15.29:0/1540766078 >> v2:172.21.15.186:3300/0 conn(0x7f30c80b0410 0x7f30c80b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381950184 0 0) 0x7f30c815ce30 con 0x7f30c8158ed0 2023-10-14T16:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30b4003520 con 0x7f30c8158ed0 2023-10-14T16:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4109407006 0 0) 0x7f30b4003520 con 0x7f30c8158ed0 2023-10-14T16:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 >> v2:172.21.15.186:3300/0 conn(0x7f30c80b0410 msgr2=0x7f30c80b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:08.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 --2- 172.21.15.29:0/1540766078 >> v2:172.21.15.186:3300/0 conn(0x7f30c80b0410 0x7f30c80b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 >> v1:172.21.15.186:6789/0 conn(0x7f30c80ade70 legacy=0x7f30c815d2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80c9080 con 0x7f30c8158ed0 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f30c4001670 con 0x7f30c8158ed0 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 >> v1:172.21.15.29:6789/0 conn(0x7f30c8158ed0 legacy=0x7f30c80ce1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 0x7f30b4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30b40047c0 0x7f30b4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40051d0 0x7f30b400ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80b0d80 con 0x7f30b40039c0 2023-10-14T16:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b400d230 con 0x7f30b40051d0 2023-10-14T16:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30b400d3f0 con 0x7f30b40047c0 2023-10-14T16:54:08.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.675+0000 7f30ce326640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 0x7f30b4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ce326640 1 -- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30b40047c0 msgr2=0x7f30b4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ce326640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30b40047c0 0x7f30b4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ce326640 1 -- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40051d0 msgr2=0x7f30b400ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ce326640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40051d0 0x7f30b400ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ce326640 1 -- 172.21.15.29:0/1540766078 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b4003520 con 0x7f30b40039c0 2023-10-14T16:54:08.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.676+0000 7f30ceb27640 1 -- 172.21.15.29:0/1540766078 reap_dead start 2023-10-14T16:54:08.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.677+0000 7f30ce326640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 0x7f30b4003de0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f30c4001eb0 tx=0x7f30b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d20ae6224af2f81 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:08.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.677+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c4005ad0 con 0x7f30b40039c0 2023-10-14T16:54:08.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.677+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c4005c30 con 0x7f30b40039c0 2023-10-14T16:54:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.677+0000 7f30cd324640 1 -- 172.21.15.29:0/1540766078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c402ca00 con 0x7f30b40039c0 2023-10-14T16:54:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 -- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 msgr2=0x7f30b4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 0x7f30b4003de0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f30c4001eb0 tx=0x7f30b40057f0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 -- 172.21.15.29:0/1540766078 shutdown_connections 2023-10-14T16:54:08.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 --2- 172.21.15.29:0/1540766078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30b40039c0 0x7f30b4003de0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 -- 172.21.15.29:0/1540766078 >> 172.21.15.29:0/1540766078 conn(0x7f30c80cb410 msgr2=0x7f30c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 -- 172.21.15.29:0/1540766078 shutdown_connections 2023-10-14T16:54:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.678+0000 7f30d058a640 1 -- 172.21.15.29:0/1540766078 wait complete. 2023-10-14T16:54:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.680+0000 7f30d058a640 1 Processor -- start 2023-10-14T16:54:08.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.680+0000 7f30d058a640 1 -- start start 2023-10-14T16:54:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 0x7f30c80acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c80c8140 0x7f30c814e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c814f260 con 0x7f30c80c85c0 2023-10-14T16:54:08.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80c2540 con 0x7f30b40039c0 2023-10-14T16:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30d058a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c80c2820 con 0x7f30c80c8140 2023-10-14T16:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30ceb27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30ceb27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46570/0 (socket says 172.21.15.29:46570) 2023-10-14T16:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.681+0000 7f30ceb27640 1 -- 172.21.15.29:0/2611110735 learned_addr learned my addr 172.21.15.29:0/2611110735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:08.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ce326640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 0x7f30c80acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30cdb25640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c80c8140 0x7f30c814e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 -- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c80c8140 msgr2=0x7f30c814e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c80c8140 0x7f30c814e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 -- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 msgr2=0x7f30c80acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 0x7f30c80acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 -- 172.21.15.29:0/2611110735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c80c2b00 con 0x7f30c80c85c0 2023-10-14T16:54:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ce326640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 0x7f30c80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.682+0000 7f30ceb27640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f30c80cfa40 tx=0x7f30c000c0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.683+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c000cb90 con 0x7f30c80c85c0 2023-10-14T16:54:08.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.683+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30c000f040 con 0x7f30c80c85c0 2023-10-14T16:54:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.683+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30c0014480 con 0x7f30c80c85c0 2023-10-14T16:54:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.683+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c80c2de0 con 0x7f30c80c85c0 2023-10-14T16:54:08.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.683+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c80c35e0 con 0x7f30c80c85c0 2023-10-14T16:54:08.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.684+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c80c3300 con 0x7f30c80c85c0 2023-10-14T16:54:08.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.685+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30c00145e0 con 0x7f30c80c85c0 2023-10-14T16:54:08.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.686+0000 7f30beffd640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 0x7f30ac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.687+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f30c008a0c0 con 0x7f30c80c85c0 2023-10-14T16:54:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.687+0000 7f30ce326640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 0x7f30ac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.688+0000 7f30ce326640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 0x7f30ac06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30c4034000 tx=0x7f30c4009030 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:08.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.691+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30c80c3300 con 0x7f30c80c85c0 2023-10-14T16:54:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.928+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f30c80c3300 con 0x7f30c80c85c0 2023-10-14T16:54:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.929+0000 7f30beffd640 1 -- 172.21.15.29:0/2611110735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f30c80c3300 con 0x7f30c80c85c0 2023-10-14T16:54:08.930 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:08.930 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2839589697","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2839589697},{"type":"v1","addr":"172.21.15.186:6834","nonce":2839589697}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":4720,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/222509470","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":222509470},{"type":"v1","addr":"172.21.15.186:6835","nonce":222509470}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5115,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1857009766","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1857009766},{"type":"v1","addr":"172.21.15.29:6836","nonce":1857009766}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1442262305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1442262305},{"type":"v1","addr":"172.21.15.29:6837","nonce":1442262305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":17}],"filesystems":[]} 2023-10-14T16:54:08.931 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 17 2023-10-14T16:54:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 msgr2=0x7f30ac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 0x7f30ac06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f30c4034000 tx=0x7f30c4009030 comp rx=0 tx=0).stop 2023-10-14T16:54:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 msgr2=0x7f30c814ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f30c80cfa40 tx=0x7f30c000c0c0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 shutdown_connections 2023-10-14T16:54:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f30ac06afc0 0x7f30ac06d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c80c85c0 0x7f30c814ed20 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c80c8140 0x7f30c814e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 --2- 172.21.15.29:0/2611110735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30b40039c0 0x7f30c80acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 >> 172.21.15.29:0/2611110735 conn(0x7f30c80cb410 msgr2=0x7f30c8155f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 shutdown_connections 2023-10-14T16:54:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:08.932+0000 7f30d058a640 1 -- 172.21.15.29:0/2611110735 wait complete. 2023-10-14T16:54:08.952 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:54:08.953 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:54:08.953 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:54:08.953 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:54:08.954 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:54:08.954 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:54:08.954 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:54:08.955 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:54:08.955 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:54:08.955 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:54:08.956 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:54:08.956 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:54:08.956 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:54:08.956 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:54:08.957 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:54:08.957 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:54:08.971 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:54:08.971 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:54:08.973 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:54:08.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:54:08.977 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:54:08.977 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:54:08.978 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:54:08.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:54:09.021 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:54:09.022 DEBUG:teuthology.parallel:result is None 2023-10-14T16:54:09.025 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:54:09.026 DEBUG:teuthology.parallel:result is None 2023-10-14T16:54:09.029 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:54:09.029 DEBUG:teuthology.parallel:result is None 2023-10-14T16:54:09.032 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:54:09.033 DEBUG:teuthology.parallel:result is None 2023-10-14T16:54:09.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:54:09.186 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:54:09.189 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:54:09.192 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:54:09.200 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:54:09.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.224+0000 7f2b64c44640 1 Processor -- start 2023-10-14T16:54:09.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.224+0000 7f2b64c44640 1 -- start start 2023-10-14T16:54:09.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b64c44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b600caa80 0x7f2b60069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b64c44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b6006a2e0 0x7f2b601650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b64c44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b6006a730 con 0x7f2b600addb0 2023-10-14T16:54:09.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b64c44640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b60161a50 con 0x7f2b600caa80 2023-10-14T16:54:09.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b64c44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b600caef0 con 0x7f2b6006a2e0 2023-10-14T16:54:09.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b5e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2b600addb0 0x7f2b600c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59660/0 (socket says 172.21.15.29:59660) 2023-10-14T16:54:09.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b5e59c640 1 -- 172.21.15.29:0/448936090 learned_addr learned my addr 172.21.15.29:0/448936090 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:09.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.225+0000 7f2b57fff640 1 --2- 172.21.15.29:0/448936090 >> v2:172.21.15.29:3300/0 conn(0x7f2b600caa80 0x7f2b60069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.226+0000 7f2b5ed9d640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b6006a2e0 0x7f2b601650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.226+0000 7f2b57fff640 1 -- 172.21.15.29:0/448936090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b6006a2e0 msgr2=0x7f2b601650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.226+0000 7f2b57fff640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b6006a2e0 0x7f2b601650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.226+0000 7f2b57fff640 1 -- 172.21.15.29:0/448936090 >> v1:172.21.15.29:6789/0 conn(0x7f2b600addb0 legacy=0x7f2b600c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:09.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.226+0000 7f2b57fff640 1 -- 172.21.15.29:0/448936090 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b600cb140 con 0x7f2b600caa80 2023-10-14T16:54:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.227+0000 7f2b57fff640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b600caa80 0x7f2b60069da0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2b480097a0 tx=0x7f2b4802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53fb6931ed825b32 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.227+0000 7f2b5dd9b640 1 -- 172.21.15.29:0/448936090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4802d040 con 0x7f2b600caa80 2023-10-14T16:54:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.227+0000 7f2b5dd9b640 1 -- 172.21.15.29:0/448936090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b48031810 con 0x7f2b600caa80 2023-10-14T16:54:09.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b5dd9b640 1 -- 172.21.15.29:0/448936090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b48031ba0 con 0x7f2b600caa80 2023-10-14T16:54:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 -- 172.21.15.29:0/448936090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b600caa80 msgr2=0x7f2b60069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b600caa80 0x7f2b60069da0 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f2b480097a0 tx=0x7f2b4802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 -- 172.21.15.29:0/448936090 shutdown_connections 2023-10-14T16:54:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b6006a2e0 0x7f2b601650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 --2- 172.21.15.29:0/448936090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b600caa80 0x7f2b60069da0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 -- 172.21.15.29:0/448936090 >> 172.21.15.29:0/448936090 conn(0x7f2b600ce9b0 msgr2=0x7f2b600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 -- 172.21.15.29:0/448936090 shutdown_connections 2023-10-14T16:54:09.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.228+0000 7f2b64c44640 1 -- 172.21.15.29:0/448936090 wait complete. 2023-10-14T16:54:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.230+0000 7f2b64c44640 1 Processor -- start 2023-10-14T16:54:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.230+0000 7f2b64c44640 1 -- start start 2023-10-14T16:54:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 0x7f2b6015d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600caa80 0x7f2b6006f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b600c3c30 con 0x7f2b6006a2e0 2023-10-14T16:54:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b60070450 con 0x7f2b600caa80 2023-10-14T16:54:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b64c44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b600c6b30 con 0x7f2b600addb0 2023-10-14T16:54:09.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b5e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46606/0 (socket says 172.21.15.29:46606) 2023-10-14T16:54:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b5e59c640 1 -- 172.21.15.29:0/849966524 learned_addr learned my addr 172.21.15.29:0/849966524 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.231+0000 7f2b57fff640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 0x7f2b6015d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b5ed9d640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600caa80 0x7f2b6006f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 -- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600caa80 msgr2=0x7f2b6006f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600caa80 0x7f2b6006f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 -- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 msgr2=0x7f2b6015d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:09.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 -- 172.21.15.29:0/849966524 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b48009400 con 0x7f2b600addb0 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b5e59c640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.232+0000 7f2b57fff640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 0x7f2b6015d130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b48033000 tx=0x7f2b480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.233+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4803ac60 con 0x7f2b600addb0 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.233+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4803ade0 con 0x7f2b600addb0 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.233+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b48044630 con 0x7f2b600addb0 2023-10-14T16:54:09.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.233+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b600c6e10 con 0x7f2b600addb0 2023-10-14T16:54:09.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.233+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b6015d670 con 0x7f2b600addb0 2023-10-14T16:54:09.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.234+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b600c3990 con 0x7f2b600addb0 2023-10-14T16:54:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.235+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b4804d450 con 0x7f2b600addb0 2023-10-14T16:54:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.236+0000 7f2b56ffd640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 0x7f2b4006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:09.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.236+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5632+0+0 (secure 0 0 0) 0x7f2b480ab7b0 con 0x7f2b600addb0 2023-10-14T16:54:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.236+0000 7f2b5e59c640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 0x7f2b4006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.237+0000 7f2b5e59c640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 0x7f2b4006d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2b601eea10 tx=0x7f2b50009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:09.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.241+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b600c3990 con 0x7f2b600addb0 2023-10-14T16:54:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:09.457+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f2b600c3d70 con 0x7f2b600addb0 2023-10-14T16:54:10.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.473+0000 7f2b56ffd640 1 -- 172.21.15.29:0/849966524 <== mon.2 v2:172.21.15.186:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v59) v1 ==== 97+0+0 (secure 0 0 0) 0x7f2b600c3d70 con 0x7f2b600addb0 2023-10-14T16:54:10.474 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:54:10.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 msgr2=0x7f2b4006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 0x7f2b4006d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2b601eea10 tx=0x7f2b50009040 comp rx=0 tx=0).stop 2023-10-14T16:54:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 msgr2=0x7f2b6015d130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 0x7f2b6015d130 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2b48033000 tx=0x7f2b480046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 shutdown_connections 2023-10-14T16:54:10.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2b4006afe0 0x7f2b4006d4a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b600caa80 0x7f2b6006f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b600addb0 0x7f2b6015d130 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 --2- 172.21.15.29:0/849966524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b6006a2e0 0x7f2b6015d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 >> 172.21.15.29:0/849966524 conn(0x7f2b600ce9b0 msgr2=0x7f2b600ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:10.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 shutdown_connections 2023-10-14T16:54:10.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.477+0000 7f2b64c44640 1 -- 172.21.15.29:0/849966524 wait complete. 2023-10-14T16:54:10.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:54:10.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.718+0000 7f8cc3fff640 1 Processor -- start 2023-10-14T16:54:10.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.718+0000 7f8cc3fff640 1 -- start start 2023-10-14T16:54:10.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4159950 0x7f8cc415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8cc406c810 0x7f8cc40708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc3fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cc4070df0 con 0x7f8cc4070450 2023-10-14T16:54:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cc415e490 con 0x7f8cc406c810 2023-10-14T16:54:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc415aef0 con 0x7f8cc4159950 2023-10-14T16:54:10.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cba7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8cc4070450 0x7f8cc406c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59674/0 (socket says 172.21.15.29:59674) 2023-10-14T16:54:10.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cba7fc640 1 -- 172.21.15.29:0/619625764 learned_addr learned my addr 172.21.15.29:0/619625764 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:10.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.720+0000 7f8cc37fe640 1 --2- 172.21.15.29:0/619625764 >> v2:172.21.15.29:3300/0 conn(0x7f8cc406c810 0x7f8cc40708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:10.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc2ffd640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4159950 0x7f8cc415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:10.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc27fc640 1 -- 172.21.15.29:0/619625764 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431136204 0 0) 0x7f8cc4070df0 con 0x7f8cc4070450 2023-10-14T16:54:10.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc27fc640 1 -- 172.21.15.29:0/619625764 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8cb0003520 con 0x7f8cc4070450 2023-10-14T16:54:10.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc37fe640 1 -- 172.21.15.29:0/619625764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4159950 msgr2=0x7f8cc415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:10.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc37fe640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4159950 0x7f8cc415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc37fe640 1 -- 172.21.15.29:0/619625764 >> v1:172.21.15.29:6789/0 conn(0x7f8cc4070450 legacy=0x7f8cc406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.721+0000 7f8cc37fe640 1 -- 172.21.15.29:0/619625764 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc406cc80 con 0x7f8cc406c810 2023-10-14T16:54:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.722+0000 7f8cc37fe640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40708b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0097a0 tx=0x7f8cbc02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e1d2939008872cb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.722+0000 7f8cc27fc640 1 -- 172.21.15.29:0/619625764 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc031780 con 0x7f8cc406c810 2023-10-14T16:54:10.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.722+0000 7f8cc27fc640 1 -- 172.21.15.29:0/619625764 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cbc0318e0 con 0x7f8cc406c810 2023-10-14T16:54:10.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.722+0000 7f8cc27fc640 1 -- 172.21.15.29:0/619625764 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cbc031c70 con 0x7f8cc406c810 2023-10-14T16:54:10.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 -- 172.21.15.29:0/619625764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 msgr2=0x7f8cc40708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40708b0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f8cbc0097a0 tx=0x7f8cbc02b820 comp rx=0 tx=0).stop 2023-10-14T16:54:10.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 -- 172.21.15.29:0/619625764 shutdown_connections 2023-10-14T16:54:10.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40708b0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/619625764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4159950 0x7f8cc415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 -- 172.21.15.29:0/619625764 >> 172.21.15.29:0/619625764 conn(0x7f8cc40ce6a0 msgr2=0x7f8cc40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:10.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 -- 172.21.15.29:0/619625764 shutdown_connections 2023-10-14T16:54:10.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.723+0000 7f8cc3fff640 1 -- 172.21.15.29:0/619625764 wait complete. 2023-10-14T16:54:10.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.725+0000 7f8cc3fff640 1 Processor -- start 2023-10-14T16:54:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.725+0000 7f8cc3fff640 1 -- start start 2023-10-14T16:54:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4070450 0x7f8cc40cd720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cc4159950 0x7f8cc41f5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40c5160 con 0x7f8cc406c810 2023-10-14T16:54:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40c4e80 con 0x7f8cc4159950 2023-10-14T16:54:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc40b1460 con 0x7f8cc4070450 2023-10-14T16:54:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:10.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46614/0 (socket says 172.21.15.29:46614) 2023-10-14T16:54:10.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 -- 172.21.15.29:0/122285057 learned_addr learned my addr 172.21.15.29:0/122285057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:10.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 -- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4070450 msgr2=0x7f8cc40cd720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4070450 0x7f8cc40cd720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 -- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cc4159950 msgr2=0x7f8cc41f5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:10.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cc4159950 0x7f8cc41f5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.726+0000 7f8cc2ffd640 1 -- 172.21.15.29:0/122285057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc009400 con 0x7f8cc406c810 2023-10-14T16:54:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cc2ffd640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400d800 tx=0x7f8cb400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400b6c0 con 0x7f8cc406c810 2023-10-14T16:54:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cb400b820 con 0x7f8cc406c810 2023-10-14T16:54:10.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cb400bbb0 con 0x7f8cc406c810 2023-10-14T16:54:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc40b1740 con 0x7f8cc406c810 2023-10-14T16:54:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.727+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc4155290 con 0x7f8cc406c810 2023-10-14T16:54:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.728+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc40b12c0 con 0x7f8cc406c810 2023-10-14T16:54:10.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.729+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8cb400bd10 con 0x7f8cc406c810 2023-10-14T16:54:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.730+0000 7f8cbbfff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 0x7f8ca406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:10.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.730+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f8cb4089a20 con 0x7f8cc406c810 2023-10-14T16:54:10.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.730+0000 7f8cba7fc640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 0x7f8ca406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:10.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.731+0000 7f8cba7fc640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 0x7f8ca406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8cc41f67d0 tx=0x7f8cac001690 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:10.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.735+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cc40b12c0 con 0x7f8cc406c810 2023-10-14T16:54:10.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.994+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8cc40b12c0 con 0x7f8cc406c810 2023-10-14T16:54:10.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.995+0000 7f8cbbfff640 1 -- 172.21.15.29:0/122285057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v11) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8cc40b12c0 con 0x7f8cc406c810 2023-10-14T16:54:10.996 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:10.997 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:54:10.997 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:54:10.997 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:10.997 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:54:10.997 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:54:10.998 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:10.998 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:10.998 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:54:10.998 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:54:10.999 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:54:11.000 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:54:11.000 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:54:11.000 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.001 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.001 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.001 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.002 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:54:11.002 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:54:11.002 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.002 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:54:11.003 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:54:11.003 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.003 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.003 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.003 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.004 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:54:11.004 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:54:11.004 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.004 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.004 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:54:11.005 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:54:11.005 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.005 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.005 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.005 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.006 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:54:11.006 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:54:11.006 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.006 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.007 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.008 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:54:11.008 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:54:11.008 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.008 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.008 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.009 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.009 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.009 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.009 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.009 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:54:11.010 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:54:11.010 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.010 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.010 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.010 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.011 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.011 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.011 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.011 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:54:11.011 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.012 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.013 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.013 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:54:11.013 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:54:11.013 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.013 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.014 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.014 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.014 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.014 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.014 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.015 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:54:11.015 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:54:11.015 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.015 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.015 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.016 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.016 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.016 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.016 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.016 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:54:11.017 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:54:11.017 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.017 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.017 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.017 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.018 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.018 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.018 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.019 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:54:11.019 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:54:11.019 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.019 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:54:11.020 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:54:11.020 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.020 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.021 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.021 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.021 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:54:11.022 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:54:11.022 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.022 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.023 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T16:54:11.023 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T16:54:11.024 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.024 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.024 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.025 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:54:11.025 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:54:11.025 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.026 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.026 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:54:11.026 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:54:11.027 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:54:11.027 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.028 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.028 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:54:11.029 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:54:11.029 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.029 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.029 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:54:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:54:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:54:11.030 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.031 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.031 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.031 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:54:11.032 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:54:11.032 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.032 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:54:11.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:54:11.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:54:11.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:54:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.035 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:54:11.035 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:54:11.035 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.036 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:54:11.036 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:54:11.036 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:54:11.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.037 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.038 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.038 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:54:11.038 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:54:11.039 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.039 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:54:11.041 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:54:11.041 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.042 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:54:11.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:11.043 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:54:11.043 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:54:11.044 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:54:11.044 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:54:11.044 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:54:11.045 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:54:11.045 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.045 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:11.046 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:54:11.046 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:54:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.997+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 msgr2=0x7f8ca406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.997+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 0x7f8ca406d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8cc41f67d0 tx=0x7f8cac001690 comp rx=0 tx=0).stop 2023-10-14T16:54:11.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 msgr2=0x7f8cc40cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f8cb400d800 tx=0x7f8cb400dcc0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 shutdown_connections 2023-10-14T16:54:11.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8ca406af90 0x7f8ca406d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cc4159950 0x7f8cc41f5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cc4070450 0x7f8cc40cd720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 --2- 172.21.15.29:0/122285057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cc406c810 0x7f8cc40cd1e0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 >> 172.21.15.29:0/122285057 conn(0x7f8cc40ce6a0 msgr2=0x7f8cc40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:11.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 shutdown_connections 2023-10-14T16:54:11.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:10.998+0000 7f8cc3fff640 1 -- 172.21.15.29:0/122285057 wait complete. 2023-10-14T16:54:11.054 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:54:11.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.280+0000 7fe751b8a640 1 Processor -- start 2023-10-14T16:54:11.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.280+0000 7fe751b8a640 1 -- start start 2023-10-14T16:54:11.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.282+0000 7fe751b8a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 0x7fe74c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.282+0000 7fe751b8a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.282+0000 7fe751b8a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe74c15ed60 con 0x7fe74c15aaa0 2023-10-14T16:54:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.282+0000 7fe751b8a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe74c06b810 con 0x7fe74c15ae80 2023-10-14T16:54:11.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.282+0000 7fe751b8a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c0707e0 con 0x7fe74c0ce390 2023-10-14T16:54:11.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.283+0000 7fe74affd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 0x7fe74c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.283+0000 7fe74bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.283+0000 7fe74bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:57956/0 (socket says 172.21.15.29:57956) 2023-10-14T16:54:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.283+0000 7fe74bfff640 1 -- 172.21.15.29:0/456605178 learned_addr learned my addr 172.21.15.29:0/456605178 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:11.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74bfff640 1 -- 172.21.15.29:0/456605178 >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 msgr2=0x7fe74c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74bfff640 1 --2- 172.21.15.29:0/456605178 >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 0x7fe74c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74bfff640 1 -- 172.21.15.29:0/456605178 >> v1:172.21.15.186:6789/0 conn(0x7fe74c15aaa0 legacy=0x7fe74c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74a7fc640 1 -- 172.21.15.29:0/456605178 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847046335 0 0) 0x7fe74c15ed60 con 0x7fe74c15aaa0 2023-10-14T16:54:11.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74bfff640 1 -- 172.21.15.29:0/456605178 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe74c15ef90 con 0x7fe74c0ce390 2023-10-14T16:54:11.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.284+0000 7fe74affd640 1 --2- 172.21.15.29:0/456605178 >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 0x7fe74c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:54:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.286+0000 7fe74bfff640 1 --2- 172.21.15.29:0/456605178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe744009e50 tx=0x7fe74402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f1301e795795b20 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.286+0000 7fe74a7fc640 1 -- 172.21.15.29:0/456605178 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74402cb00 con 0x7fe74c0ce390 2023-10-14T16:54:11.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe74a7fc640 1 -- 172.21.15.29:0/456605178 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74402cc60 con 0x7fe74c0ce390 2023-10-14T16:54:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe74a7fc640 1 -- 172.21.15.29:0/456605178 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7440343f0 con 0x7fe74c0ce390 2023-10-14T16:54:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 -- 172.21.15.29:0/456605178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 msgr2=0x7fe74c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 --2- 172.21.15.29:0/456605178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe744009e50 tx=0x7fe74402bf60 comp rx=0 tx=0).stop 2023-10-14T16:54:11.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 -- 172.21.15.29:0/456605178 shutdown_connections 2023-10-14T16:54:11.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 --2- 172.21.15.29:0/456605178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c070f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 --2- 172.21.15.29:0/456605178 >> v2:172.21.15.186:3300/0 conn(0x7fe74c15ae80 0x7fe74c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.287+0000 7fe751b8a640 1 -- 172.21.15.29:0/456605178 >> 172.21.15.29:0/456605178 conn(0x7fe74c0cb410 msgr2=0x7fe74c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:11.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.288+0000 7fe751b8a640 1 -- 172.21.15.29:0/456605178 shutdown_connections 2023-10-14T16:54:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.288+0000 7fe751b8a640 1 -- 172.21.15.29:0/456605178 wait complete. 2023-10-14T16:54:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.290+0000 7fe751b8a640 1 Processor -- start 2023-10-14T16:54:11.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.290+0000 7fe751b8a640 1 -- start start 2023-10-14T16:54:11.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c0c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe74c15aaa0 0x7fe74c0c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c1f2e30 con 0x7fe74c15ae80 2023-10-14T16:54:11.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c1f3110 con 0x7fe74c15aaa0 2023-10-14T16:54:11.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.291+0000 7fe751b8a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe74c1f33f0 con 0x7fe74c0ce390 2023-10-14T16:54:11.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36310/0 (socket says 172.21.15.29:36310) 2023-10-14T16:54:11.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 -- 172.21.15.29:0/129141062 learned_addr learned my addr 172.21.15.29:0/129141062 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:11.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74b7fe640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c0c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74affd640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe74c15aaa0 0x7fe74c0c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 -- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 msgr2=0x7fe74c0c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c0c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 -- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe74c15aaa0 msgr2=0x7fe74c0c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe74c15aaa0 0x7fe74c0c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.292+0000 7fe74bfff640 1 -- 172.21.15.29:0/129141062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe740009400 con 0x7fe74c15ae80 2023-10-14T16:54:11.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe74bfff640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe744036040 tx=0x7fe744004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe744009510 con 0x7fe74c15ae80 2023-10-14T16:54:11.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74403e450 con 0x7fe74c15ae80 2023-10-14T16:54:11.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe744009b30 con 0x7fe74c15ae80 2023-10-14T16:54:11.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe74403e7e0 con 0x7fe74c15ae80 2023-10-14T16:54:11.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.293+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe74c0c42c0 con 0x7fe74c15ae80 2023-10-14T16:54:11.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.294+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe74c04e160 con 0x7fe74c15ae80 2023-10-14T16:54:11.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.296+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe74403e940 con 0x7fe74c15ae80 2023-10-14T16:54:11.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.296+0000 7fe733fff640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 0x7fe73806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.297+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fe7440ac220 con 0x7fe74c15ae80 2023-10-14T16:54:11.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.297+0000 7fe74b7fe640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 0x7fe73806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.297+0000 7fe74b7fe640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 0x7fe73806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe73c000f60 tx=0x7fe73c009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.302+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe744077440 con 0x7fe74c15ae80 2023-10-14T16:54:11.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.502+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe74c0cdc60 con 0x7fe74c15ae80 2023-10-14T16:54:11.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.504+0000 7fe733fff640 1 -- 172.21.15.29:0/129141062 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v59) v1 ==== 74+0+12915 (secure 0 0 0) 0x7fe74407aa60 con 0x7fe74c15ae80 2023-10-14T16:54:11.505 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:11.506 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":59,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:54:10.465287+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":32,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:54:11.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.507+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 msgr2=0x7fe73806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.507+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 0x7fe73806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fe73c000f60 tx=0x7fe73c009040 comp rx=0 tx=0).stop 2023-10-14T16:54:11.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.507+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 msgr2=0x7fe74c1575f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.507+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7fe744036040 tx=0x7fe744004a40 comp rx=0 tx=0).stop 2023-10-14T16:54:11.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 shutdown_connections 2023-10-14T16:54:11.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fe73806af90 0x7fe73806d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe74c15ae80 0x7fe74c1575f0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe74c15aaa0 0x7fe74c0c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 --2- 172.21.15.29:0/129141062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe74c0ce390 0x7fe74c0c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 >> 172.21.15.29:0/129141062 conn(0x7fe74c0cb410 msgr2=0x7fe74c0b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:11.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 shutdown_connections 2023-10-14T16:54:11.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.508+0000 7fe751b8a640 1 -- 172.21.15.29:0/129141062 wait complete. 2023-10-14T16:54:11.526 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:54:11.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:54:11.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.760+0000 7f761b116640 1 Processor -- start 2023-10-14T16:54:11.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.761+0000 7f761b116640 1 -- start start 2023-10-14T16:54:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.762+0000 7f761b116640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f76140b0ea0 0x7f761415a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.762+0000 7f761b116640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140cd970 0x7f761415a830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.762+0000 7f761b116640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f761406be30 con 0x7f76140b3ec0 2023-10-14T16:54:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.762+0000 7f761b116640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f761406ca00 con 0x7f76140b0ea0 2023-10-14T16:54:11.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.762+0000 7f761b116640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76140ada10 con 0x7f76140cd970 2023-10-14T16:54:11.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f761a114640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76140b3ec0 0x7f76140cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39662/0 (socket says 172.21.15.29:39662) 2023-10-14T16:54:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f761a114640 1 -- 172.21.15.29:0/384066669 learned_addr learned my addr 172.21.15.29:0/384066669 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619913640 1 --2- 172.21.15.29:0/384066669 >> v2:172.21.15.29:3300/0 conn(0x7f76140b0ea0 0x7f761415a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619913640 1 -- 172.21.15.29:0/384066669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140cd970 msgr2=0x7f761415a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619913640 1 --2- 172.21.15.29:0/384066669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140cd970 0x7f761415a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619112640 1 -- 172.21.15.29:0/384066669 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520062557 0 0) 0x7f761406be30 con 0x7f76140b3ec0 2023-10-14T16:54:11.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619913640 1 -- 172.21.15.29:0/384066669 >> v1:172.21.15.29:6789/0 conn(0x7f76140b3ec0 legacy=0x7f76140cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.763+0000 7f7619913640 1 -- 172.21.15.29:0/384066669 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f761406c060 con 0x7f76140b0ea0 2023-10-14T16:54:11.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.764+0000 7f7619913640 1 --2- 172.21.15.29:0/384066669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f761415a2f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f76040097a0 tx=0x7f760402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3315cc306bca8f38 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.765+0000 7f7619112640 1 -- 172.21.15.29:0/384066669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760402e070 con 0x7f76140b0ea0 2023-10-14T16:54:11.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.765+0000 7f7619112640 1 -- 172.21.15.29:0/384066669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76040315a0 con 0x7f76140b0ea0 2023-10-14T16:54:11.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.765+0000 7f7619112640 1 -- 172.21.15.29:0/384066669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604031930 con 0x7f76140b0ea0 2023-10-14T16:54:11.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 -- 172.21.15.29:0/384066669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 msgr2=0x7f761415a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 --2- 172.21.15.29:0/384066669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f761415a2f0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f76040097a0 tx=0x7f760402b840 comp rx=0 tx=0).stop 2023-10-14T16:54:11.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 -- 172.21.15.29:0/384066669 shutdown_connections 2023-10-14T16:54:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 --2- 172.21.15.29:0/384066669 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140cd970 0x7f761415a830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 --2- 172.21.15.29:0/384066669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f761415a2f0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 -- 172.21.15.29:0/384066669 >> 172.21.15.29:0/384066669 conn(0x7f76140b0170 msgr2=0x7f76140b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 -- 172.21.15.29:0/384066669 shutdown_connections 2023-10-14T16:54:11.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.766+0000 7f761b116640 1 -- 172.21.15.29:0/384066669 wait complete. 2023-10-14T16:54:11.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.768+0000 7f761b116640 1 Processor -- start 2023-10-14T16:54:11.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.768+0000 7f761b116640 1 -- start start 2023-10-14T16:54:11.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 0x7f76140c7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 0x7f76141f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761415c6d0 con 0x7f76140b0ea0 2023-10-14T16:54:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761415d100 con 0x7f76140cd970 2023-10-14T16:54:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761b116640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76141e3b00 con 0x7f76140b3ec0 2023-10-14T16:54:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761a114640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761a114640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36334/0 (socket says 172.21.15.29:36334) 2023-10-14T16:54:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761a114640 1 -- 172.21.15.29:0/2790117981 learned_addr learned my addr 172.21.15.29:0/2790117981 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761a114640 1 -- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 msgr2=0x7f76140c7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f7619913640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 0x7f76140c7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.769+0000 7f761a915640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 0x7f76141f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a114640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 0x7f76140c7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a114640 1 -- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 msgr2=0x7f76141f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a114640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 0x7f76141f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a114640 1 -- 172.21.15.29:0/2790117981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7604009400 con 0x7f76140b0ea0 2023-10-14T16:54:11.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f7619913640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 0x7f76140c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a915640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 0x7f76141f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761a114640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f761000c430 tx=0x7f761000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76100074e0 con 0x7f76140b0ea0 2023-10-14T16:54:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7610007660 con 0x7f76140b0ea0 2023-10-14T16:54:11.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610004a20 con 0x7f76140b0ea0 2023-10-14T16:54:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76141e3540 con 0x7f76140b0ea0 2023-10-14T16:54:11.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.770+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76140c7350 con 0x7f76140b0ea0 2023-10-14T16:54:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.771+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76140c71b0 con 0x7f76140b0ea0 2023-10-14T16:54:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.772+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7610004b80 con 0x7f76140b0ea0 2023-10-14T16:54:11.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.773+0000 7f760affd640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 0x7f75f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.773+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f76100892b0 con 0x7f76140b0ea0 2023-10-14T16:54:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.774+0000 7f7619913640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 0x7f75f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.774+0000 7f7619913640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 0x7f75f806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7604036000 tx=0x7f7604008030 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:11.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.778+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76140c71b0 con 0x7f76140b0ea0 2023-10-14T16:54:11.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.975+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f76140c71b0 con 0x7f76140b0ea0 2023-10-14T16:54:11.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.976+0000 7f760affd640 1 -- 172.21.15.29:0/2790117981 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v59) v1 ==== 93+0+10 (secure 0 0 0) 0x7f76140c71b0 con 0x7f76140b0ea0 2023-10-14T16:54:11.976 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:54:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 msgr2=0x7f75f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 0x7f75f806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f7604036000 tx=0x7f7604008030 comp rx=0 tx=0).stop 2023-10-14T16:54:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 msgr2=0x7f76140c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f761000c430 tx=0x7f761000c8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 shutdown_connections 2023-10-14T16:54:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f75f806af90 0x7f75f806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76140cd970 0x7f76141f3900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76140b3ec0 0x7f76140c7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 --2- 172.21.15.29:0/2790117981 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76140b0ea0 0x7f76140c50a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.978+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 >> 172.21.15.29:0/2790117981 conn(0x7f76140b0170 msgr2=0x7f761406f270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.979+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 shutdown_connections 2023-10-14T16:54:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:11.979+0000 7f761b116640 1 -- 172.21.15.29:0/2790117981 wait complete. 2023-10-14T16:54:11.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:54:12.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.214+0000 7fd86f6f2640 1 Processor -- start 2023-10-14T16:54:12.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.214+0000 7fd86f6f2640 1 -- start start 2023-10-14T16:54:12.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.216+0000 7fd86f6f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.216+0000 7fd86f6f2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd8680b0b20 0x7fd868164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.216+0000 7fd86f6f2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd868071450 con 0x7fd868070fd0 2023-10-14T16:54:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.216+0000 7fd86f6f2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8680cc600 con 0x7fd8680b0b20 2023-10-14T16:54:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.216+0000 7fd86f6f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680cc8e0 con 0x7fd8680adb50 2023-10-14T16:54:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.217+0000 7fd86d48e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.217+0000 7fd86d48e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:57988/0 (socket says 172.21.15.29:57988) 2023-10-14T16:54:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.217+0000 7fd86d48e640 1 -- 172.21.15.29:0/2760444700 learned_addr learned my addr 172.21.15.29:0/2760444700 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.217+0000 7fd86dc8f640 1 --2- 172.21.15.29:0/2760444700 >> v2:172.21.15.186:3300/0 conn(0x7fd8680b0b20 0x7fd868164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.218+0000 7fd86d48e640 1 -- 172.21.15.29:0/2760444700 >> v2:172.21.15.186:3300/0 conn(0x7fd8680b0b20 msgr2=0x7fd868164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.218+0000 7fd86d48e640 1 --2- 172.21.15.29:0/2760444700 >> v2:172.21.15.186:3300/0 conn(0x7fd8680b0b20 0x7fd868164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.218+0000 7fd86d48e640 1 -- 172.21.15.29:0/2760444700 >> v1:172.21.15.186:6789/0 conn(0x7fd868070fd0 legacy=0x7fd8680b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.218+0000 7fd86d48e640 1 -- 172.21.15.29:0/2760444700 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8680ccbc0 con 0x7fd8680adb50 2023-10-14T16:54:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.219+0000 7fd86d48e640 1 --2- 172.21.15.29:0/2760444700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd864009870 tx=0x7fd86402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a4dbee6b32966239 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.219+0000 7fd85ffff640 1 -- 172.21.15.29:0/2760444700 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864030840 con 0x7fd8680adb50 2023-10-14T16:54:12.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd85ffff640 1 -- 172.21.15.29:0/2760444700 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8640309a0 con 0x7fd8680adb50 2023-10-14T16:54:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd85ffff640 1 -- 172.21.15.29:0/2760444700 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864030d30 con 0x7fd8680adb50 2023-10-14T16:54:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 -- 172.21.15.29:0/2760444700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 msgr2=0x7fd86806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/2760444700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd864009870 tx=0x7fd86402b980 comp rx=0 tx=0).stop 2023-10-14T16:54:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 -- 172.21.15.29:0/2760444700 shutdown_connections 2023-10-14T16:54:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/2760444700 >> v2:172.21.15.186:3300/0 conn(0x7fd8680b0b20 0x7fd868164f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/2760444700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd86806f550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 -- 172.21.15.29:0/2760444700 >> 172.21.15.29:0/2760444700 conn(0x7fd8680cb410 msgr2=0x7fd8680cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.220+0000 7fd86f6f2640 1 -- 172.21.15.29:0/2760444700 shutdown_connections 2023-10-14T16:54:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.221+0000 7fd86f6f2640 1 -- 172.21.15.29:0/2760444700 wait complete. 2023-10-14T16:54:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.223+0000 7fd86f6f2640 1 Processor -- start 2023-10-14T16:54:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.223+0000 7fd86f6f2640 1 -- start start 2023-10-14T16:54:12.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.224+0000 7fd86f6f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.224+0000 7fd86f6f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd8680b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.224+0000 7fd86f6f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8680b0b20 0x7fd86815eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86f6f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8681f7440 con 0x7fd868070fd0 2023-10-14T16:54:12.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86f6f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680b3be0 con 0x7fd8680b0b20 2023-10-14T16:54:12.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86f6f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8680b15b0 con 0x7fd8680adb50 2023-10-14T16:54:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36344/0 (socket says 172.21.15.29:36344) 2023-10-14T16:54:12.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 -- 172.21.15.29:0/1408394057 learned_addr learned my addr 172.21.15.29:0/1408394057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:12.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86dc8f640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8680b0b20 0x7fd86815eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86cc8d640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd8680b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 -- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 msgr2=0x7fd8680b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd8680b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 -- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8680b0b20 msgr2=0x7fd86815eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8680b0b20 0x7fd86815eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.225+0000 7fd86d48e640 1 -- 172.21.15.29:0/1408394057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640094d0 con 0x7fd868070fd0 2023-10-14T16:54:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd86cc8d640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd8680b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd86d48e640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd86402b9c0 tx=0x7fd864004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd864039c90 con 0x7fd868070fd0 2023-10-14T16:54:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd864039df0 con 0x7fd868070fd0 2023-10-14T16:54:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8640446c0 con 0x7fd868070fd0 2023-10-14T16:54:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8680b1800 con 0x7fd868070fd0 2023-10-14T16:54:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.226+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8680b1dc0 con 0x7fd868070fd0 2023-10-14T16:54:12.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.228+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd86806fa10 con 0x7fd868070fd0 2023-10-14T16:54:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.229+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd86402f390 con 0x7fd868070fd0 2023-10-14T16:54:12.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.230+0000 7fd85dffb640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.230+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fd86403e070 con 0x7fd868070fd0 2023-10-14T16:54:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.230+0000 7fd86cc8d640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.231+0000 7fd86cc8d640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 0x7fd84c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8680bd600 tx=0x7fd858009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.234+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd864077010 con 0x7fd868070fd0 2023-10-14T16:54:12.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.438+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fd86804e160 con 0x7fd868070fd0 2023-10-14T16:54:12.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.439+0000 7fd85dffb640 1 -- 172.21.15.29:0/1408394057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v59) v1 ==== 84+0+65 (secure 0 0 0) 0x7fd864034090 con 0x7fd868070fd0 2023-10-14T16:54:12.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:12.440 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:54:12.440 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:12.441 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:54:12.441 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:54:12.441 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:12.441 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:54:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 msgr2=0x7fd84c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 0x7fd84c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fd8680bd600 tx=0x7fd858009040 comp rx=0 tx=0).stop 2023-10-14T16:54:12.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 msgr2=0x7fd8681f6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7fd86402b9c0 tx=0x7fd864004730 comp rx=0 tx=0).stop 2023-10-14T16:54:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 shutdown_connections 2023-10-14T16:54:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd84c06af90 0x7fd84c06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8680b0b20 0x7fd86815eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8680adb50 0x7fd8680b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 --2- 172.21.15.29:0/1408394057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd868070fd0 0x7fd8681f6f00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.442+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 >> 172.21.15.29:0/1408394057 conn(0x7fd8680cb410 msgr2=0x7fd8680adf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.443+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 shutdown_connections 2023-10-14T16:54:12.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.443+0000 7fd86f6f2640 1 -- 172.21.15.29:0/1408394057 wait complete. 2023-10-14T16:54:12.462 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:54:12.463 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:54:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.694+0000 7fc66e8f1640 1 Processor -- start 2023-10-14T16:54:12.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.695+0000 7fc66e8f1640 1 -- start start 2023-10-14T16:54:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.696+0000 7fc66e8f1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc668069da0 0x7fc668070f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.696+0000 7fc66e8f1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680ce2b0 0x7fc6680ce6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.696+0000 7fc66e8f1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6680cde20 con 0x7fc6680c9e80 2023-10-14T16:54:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.696+0000 7fc66e8f1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc668071480 con 0x7fc668069da0 2023-10-14T16:54:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.696+0000 7fc66e8f1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680cab70 con 0x7fc6680ce2b0 2023-10-14T16:54:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc668069da0 0x7fc668070f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc668069da0 0x7fc668070f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36360/0 (socket says 172.21.15.29:36360) 2023-10-14T16:54:12.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 -- 172.21.15.29:0/1455808771 learned_addr learned my addr 172.21.15.29:0/1455808771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:12.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 -- 172.21.15.29:0/1455808771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680ce2b0 msgr2=0x7fc6680ce6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 --2- 172.21.15.29:0/1455808771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680ce2b0 0x7fc6680ce6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 -- 172.21.15.29:0/1455808771 >> v1:172.21.15.186:6789/0 conn(0x7fc6680c9e80 legacy=0x7fc66806a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.697+0000 7fc66d8ef640 1 -- 172.21.15.29:0/1455808771 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6680ca310 con 0x7fc668069da0 2023-10-14T16:54:12.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.698+0000 7fc66d8ef640 1 --2- 172.21.15.29:0/1455808771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc668069da0 0x7fc668070f40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc664009870 tx=0x7fc66402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36e55506b8cde4a2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.698+0000 7fc66c8ed640 1 -- 172.21.15.29:0/1455808771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6640317c0 con 0x7fc668069da0 2023-10-14T16:54:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.699+0000 7fc66c8ed640 1 -- 172.21.15.29:0/1455808771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc664031920 con 0x7fc668069da0 2023-10-14T16:54:12.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.699+0000 7fc66c8ed640 1 -- 172.21.15.29:0/1455808771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc664031cb0 con 0x7fc668069da0 2023-10-14T16:54:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.699+0000 7fc66e8f1640 1 -- 172.21.15.29:0/1455808771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc668069da0 msgr2=0x7fc668070f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.699+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/1455808771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc668069da0 0x7fc668070f40 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7fc664009870 tx=0x7fc66402b860 comp rx=0 tx=0).stop 2023-10-14T16:54:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 -- 172.21.15.29:0/1455808771 shutdown_connections 2023-10-14T16:54:12.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/1455808771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680ce2b0 0x7fc6680ce6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/1455808771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc668069da0 0x7fc668070f40 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 -- 172.21.15.29:0/1455808771 >> 172.21.15.29:0/1455808771 conn(0x7fc6680cf830 msgr2=0x7fc6680cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 -- 172.21.15.29:0/1455808771 shutdown_connections 2023-10-14T16:54:12.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.700+0000 7fc66e8f1640 1 -- 172.21.15.29:0/1455808771 wait complete. 2023-10-14T16:54:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.701+0000 7fc66e8f1640 1 Processor -- start 2023-10-14T16:54:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.702+0000 7fc66e8f1640 1 -- start start 2023-10-14T16:54:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.702+0000 7fc66e8f1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668069da0 0x7fc6680b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e8f1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680c9e80 0x7fc6680b28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e8f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e8f1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc668155660 con 0x7fc6680ce2b0 2023-10-14T16:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e8f1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c9430 con 0x7fc668069da0 2023-10-14T16:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e8f1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6680c88b0 con 0x7fc6680c9e80 2023-10-14T16:54:12.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36368/0 (socket says 172.21.15.29:36368) 2023-10-14T16:54:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 -- 172.21.15.29:0/4197470900 learned_addr learned my addr 172.21.15.29:0/4197470900 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66d8ef640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668069da0 0x7fc6680b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66d0ee640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680c9e80 0x7fc6680b28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 -- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680c9e80 msgr2=0x7fc6680b28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680c9e80 0x7fc6680b28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 -- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668069da0 msgr2=0x7fc6680b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:12.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.703+0000 7fc66e0f0640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668069da0 0x7fc6680b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc66e0f0640 1 -- 172.21.15.29:0/4197470900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6640094d0 con 0x7fc6680ce2b0 2023-10-14T16:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc66e0f0640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc66000b970 tx=0x7fc66000be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc66000cb40 con 0x7fc6680ce2b0 2023-10-14T16:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc66000ccc0 con 0x7fc6680ce2b0 2023-10-14T16:54:12.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc660014680 con 0x7fc6680ce2b0 2023-10-14T16:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6680c8b90 con 0x7fc6680ce2b0 2023-10-14T16:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.704+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6680c9150 con 0x7fc6680ce2b0 2023-10-14T16:54:12.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.705+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6681f39b0 con 0x7fc6680ce2b0 2023-10-14T16:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.706+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc66001d3b0 con 0x7fc6680ce2b0 2023-10-14T16:54:12.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.707+0000 7fc65e7fc640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 0x7fc64c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.707+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fc660089380 con 0x7fc6680ce2b0 2023-10-14T16:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.708+0000 7fc66d8ef640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 0x7fc64c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.708+0000 7fc66d8ef640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 0x7fc64c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc664036000 tx=0x7fc66403d040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:12.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.712+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6681f39b0 con 0x7fc6680ce2b0 2023-10-14T16:54:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:12.908+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fc6681f39b0 con 0x7fc6680ce2b0 2023-10-14T16:54:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.490+0000 7fc65e7fc640 1 -- 172.21.15.29:0/4197470900 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v60) v1 ==== 138+0+0 (secure 0 0 0) 0x7fc6681f39b0 con 0x7fc6680ce2b0 2023-10-14T16:54:13.491 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:54:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 msgr2=0x7fc64c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 0x7fc64c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fc664036000 tx=0x7fc66403d040 comp rx=0 tx=0).stop 2023-10-14T16:54:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 msgr2=0x7fc6680c7b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fc66000b970 tx=0x7fc66000be30 comp rx=0 tx=0).stop 2023-10-14T16:54:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 shutdown_connections 2023-10-14T16:54:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc64c06af90 0x7fc64c06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6680ce2b0 0x7fc6680c7b10 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6680c9e80 0x7fc6680b28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 --2- 172.21.15.29:0/4197470900 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc668069da0 0x7fc6680b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.494+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 >> 172.21.15.29:0/4197470900 conn(0x7fc6680cf830 msgr2=0x7fc6680ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.495+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 shutdown_connections 2023-10-14T16:54:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.495+0000 7fc66e8f1640 1 -- 172.21.15.29:0/4197470900 wait complete. 2023-10-14T16:54:13.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:54:13.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.736+0000 7f7115c46640 1 Processor -- start 2023-10-14T16:54:13.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.736+0000 7f7115c46640 1 -- start start 2023-10-14T16:54:13.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f7115c46640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f711015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f7115c46640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71100707e0 con 0x7f711015aaa0 2023-10-14T16:54:13.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f7115c46640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f711006b7a0 con 0x7f711015ae80 2023-10-14T16:54:13.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f7115c46640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f711015ef20 con 0x7f71100ce390 2023-10-14T16:54:13.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f710f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f711015aaa0 0x7f71100cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39678/0 (socket says 172.21.15.29:39678) 2023-10-14T16:54:13.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.738+0000 7f710f7fe640 1 -- 172.21.15.29:0/1680109086 learned_addr learned my addr 172.21.15.29:0/1680109086 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:13.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710ffff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f711015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3909725116 0 0) 0x7f71100707e0 con 0x7f711015aaa0 2023-10-14T16:54:13.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70fc003520 con 0x7f711015aaa0 2023-10-14T16:54:13.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2421161080 0 0) 0x7f70fc003520 con 0x7f711015aaa0 2023-10-14T16:54:13.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 msgr2=0x7f711015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:13.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f711015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 >> v1:172.21.15.186:6789/0 conn(0x7f711015ae80 legacy=0x7f7110070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711015f200 con 0x7f711015aaa0 2023-10-14T16:54:13.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7100001670 con 0x7f711015aaa0 2023-10-14T16:54:13.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 >> v1:172.21.15.29:6789/0 conn(0x7f711015aaa0 legacy=0x7f71100cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f70fc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70fc003f20 0x7f70fc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc004d00 0x7f70fc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.739+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015ef20 con 0x7f70fc003f20 2023-10-14T16:54:13.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc00bb10 con 0x7f70fc004d00 2023-10-14T16:54:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc00bcd0 con 0x7f71100ce390 2023-10-14T16:54:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70fc003f20 0x7f70fc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f710ffff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc004d00 0x7f70fc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f710f7fe640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f70fc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 -- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 msgr2=0x7f70fc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71100ce390 0x7f70fc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 -- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc004d00 msgr2=0x7f70fc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc004d00 0x7f70fc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.740+0000 7f7107fff640 1 -- 172.21.15.29:0/1680109086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc003520 con 0x7f70fc003f20 2023-10-14T16:54:13.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.742+0000 7f7107fff640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70fc003f20 0x7f70fc0047c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001720 tx=0x7f70fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2dfebbc725b9cf00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:13.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.742+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f80052d0 con 0x7f70fc003f20 2023-10-14T16:54:13.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.742+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f802e050 con 0x7f70fc003f20 2023-10-14T16:54:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.742+0000 7f710effd640 1 -- 172.21.15.29:0/1680109086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f80035f0 con 0x7f70fc003f20 2023-10-14T16:54:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 -- 172.21.15.29:0/1680109086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70fc003f20 msgr2=0x7f70fc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 --2- 172.21.15.29:0/1680109086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70fc003f20 0x7f70fc0047c0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f70f8001720 tx=0x7f70fc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f710ffff640 1 -- 172.21.15.29:0/1680109086 reap_dead start 2023-10-14T16:54:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 -- 172.21.15.29:0/1680109086 shutdown_connections 2023-10-14T16:54:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 -- 172.21.15.29:0/1680109086 >> 172.21.15.29:0/1680109086 conn(0x7f71100cb410 msgr2=0x7f71100cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 -- 172.21.15.29:0/1680109086 shutdown_connections 2023-10-14T16:54:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.743+0000 7f7115c46640 1 -- 172.21.15.29:0/1680109086 wait complete. 2023-10-14T16:54:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.745+0000 7f7115c46640 1 Processor -- start 2023-10-14T16:54:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.745+0000 7f7115c46640 1 -- start start 2023-10-14T16:54:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc003f20 0x7f71100caab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71100acdd0 0x7f71100c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71100ad280 con 0x7f711015b320 2023-10-14T16:54:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f711015c3c0 con 0x7f71100acdd0 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7115c46640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71100caff0 con 0x7f70fc003f20 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7107fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36384/0 (socket says 172.21.15.29:36384) 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.746+0000 7f7107fff640 1 -- 172.21.15.29:0/2111012557 learned_addr learned my addr 172.21.15.29:0/2111012557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f710f7fe640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc003f20 0x7f71100caab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f710ffff640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71100acdd0 0x7f71100c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 -- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc003f20 msgr2=0x7f71100caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc003f20 0x7f71100caab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 -- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71100acdd0 msgr2=0x7f71100c8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71100acdd0 0x7f71100c8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:13.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 -- 172.21.15.29:0/2111012557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f711015ceb0 con 0x7f711015b320 2023-10-14T16:54:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.747+0000 7f7107fff640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f70f802bbe0 tx=0x7f70f8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.748+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f80076c0 con 0x7f711015b320 2023-10-14T16:54:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.748+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f8007a30 con 0x7f711015b320 2023-10-14T16:54:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.748+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f711015b7b0 con 0x7f711015b320 2023-10-14T16:54:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.748+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8006be0 con 0x7f711015b320 2023-10-14T16:54:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.748+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f711015bd70 con 0x7f711015b320 2023-10-14T16:54:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.749+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71100ad1e0 con 0x7f711015b320 2023-10-14T16:54:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.751+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70f8006d40 con 0x7f711015b320 2023-10-14T16:54:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.752+0000 7f710cff9640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.752+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 5271+0+0 (secure 0 0 0) 0x7f70f80ab650 con 0x7f711015b320 2023-10-14T16:54:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.752+0000 7f710f7fe640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.753+0000 7f710f7fe640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 0x7f70f006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7100001c90 tx=0x7f7100009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.755+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70f80765e0 con 0x7f711015b320 2023-10-14T16:54:13.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:13.950+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7110070a30 con 0x7f711015b320 2023-10-14T16:54:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.497+0000 7f710cff9640 1 -- 172.21.15.29:0/2111012557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v61) v1 ==== 170+0+0 (secure 0 0 0) 0x7f70f8079c00 con 0x7f711015b320 2023-10-14T16:54:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:54:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 msgr2=0x7f70f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 0x7f70f006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7100001c90 tx=0x7f7100009040 comp rx=0 tx=0).stop 2023-10-14T16:54:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 msgr2=0x7f71100ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f70f802bbe0 tx=0x7f70f8001120 comp rx=0 tx=0).stop 2023-10-14T16:54:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 shutdown_connections 2023-10-14T16:54:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f70f006af90 0x7f70f006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f71100acdd0 0x7f71100c8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f711015b320 0x7f71100ac890 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 --2- 172.21.15.29:0/2111012557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc003f20 0x7f71100caab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.501+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 >> 172.21.15.29:0/2111012557 conn(0x7f71100cb410 msgr2=0x7f7110070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.502+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 shutdown_connections 2023-10-14T16:54:14.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.502+0000 7f7115c46640 1 -- 172.21.15.29:0/2111012557 wait complete. 2023-10-14T16:54:14.533 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:54:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.727+0000 7f2fe3e0e640 1 Processor -- start 2023-10-14T16:54:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.727+0000 7f2fe3e0e640 1 -- start start 2023-10-14T16:54:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.728+0000 7f2fe3e0e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fdc0ad350 0x7f2fdc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.728+0000 7f2fe3e0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2fdc06a2e0 0x7f2fdc0c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.728+0000 7f2fe3e0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fdc06a730 con 0x7f2fdc15d150 2023-10-14T16:54:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.728+0000 7f2fe3e0e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fdc15d530 con 0x7f2fdc0ad350 2023-10-14T16:54:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.728+0000 7f2fe3e0e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fdc0ac890 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fdc0ad350 0x7f2fdc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fdc0ad350 0x7f2fdc069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36390/0 (socket says 172.21.15.29:36390) 2023-10-14T16:54:14.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 -- 172.21.15.29:0/299299639 learned_addr learned my addr 172.21.15.29:0/299299639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:14.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe23ab640 1 --2- 172.21.15.29:0/299299639 >> v2:172.21.15.186:3300/0 conn(0x7f2fdc06a2e0 0x7f2fdc0c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 -- 172.21.15.29:0/299299639 >> v2:172.21.15.186:3300/0 conn(0x7f2fdc06a2e0 msgr2=0x7f2fdc0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/299299639 >> v2:172.21.15.186:3300/0 conn(0x7f2fdc06a2e0 0x7f2fdc0c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 -- 172.21.15.29:0/299299639 >> v1:172.21.15.186:6789/0 conn(0x7f2fdc15d150 legacy=0x7f2fdc160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.729+0000 7f2fe13a9640 1 -- 172.21.15.29:0/299299639 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fdc0ad7a0 con 0x7f2fdc0ad350 2023-10-14T16:54:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/299299639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc0ad350 0x7f2fdc069da0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0adde0 tx=0x7f2fcc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8cad51b15b1ea2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe0ba8640 1 -- 172.21.15.29:0/299299639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc031870 con 0x7f2fdc0ad350 2023-10-14T16:54:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe0ba8640 1 -- 172.21.15.29:0/299299639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fcc0319d0 con 0x7f2fdc0ad350 2023-10-14T16:54:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe0ba8640 1 -- 172.21.15.29:0/299299639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fcc031d60 con 0x7f2fdc0ad350 2023-10-14T16:54:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/299299639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc0ad350 msgr2=0x7f2fdc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.731+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/299299639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc0ad350 0x7f2fdc069da0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f2fdc0adde0 tx=0x7f2fcc02b910 comp rx=0 tx=0).stop 2023-10-14T16:54:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/299299639 shutdown_connections 2023-10-14T16:54:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/299299639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc0ad350 0x7f2fdc069da0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/299299639 >> v2:172.21.15.186:3300/0 conn(0x7f2fdc06a2e0 0x7f2fdc0c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/299299639 >> 172.21.15.29:0/299299639 conn(0x7f2fdc0ce9b0 msgr2=0x7f2fdc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/299299639 shutdown_connections 2023-10-14T16:54:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.732+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/299299639 wait complete. 2023-10-14T16:54:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.733+0000 7f2fe3e0e640 1 Processor -- start 2023-10-14T16:54:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.734+0000 7f2fe3e0e640 1 -- start start 2023-10-14T16:54:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 0x7f2fdc0c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 0x7f2fdc0c7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0ae420 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0c4b30 con 0x7f2fdc15d150 2023-10-14T16:54:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe3e0e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fdc0c4850 con 0x7f2fdc0ad350 2023-10-14T16:54:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe1baa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe1baa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36392/0 (socket says 172.21.15.29:36392) 2023-10-14T16:54:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe1baa640 1 -- 172.21.15.29:0/1938137193 learned_addr learned my addr 172.21.15.29:0/1938137193 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:14.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe23ab640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 0x7f2fdc0c7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe1baa640 1 -- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 msgr2=0x7f2fdc0c3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 0x7f2fdc0c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.735+0000 7f2fe1baa640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 0x7f2fdc0c3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe1baa640 1 -- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 msgr2=0x7f2fdc0c7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe1baa640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 0x7f2fdc0c7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe1baa640 1 -- 172.21.15.29:0/1938137193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fcc0094d0 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe23ab640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 0x7f2fdc0c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 0x7f2fdc0c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe1baa640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800b410 tx=0x7f2fd800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8011830 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd8011990 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd8011d40 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fdc0b16e0 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.736+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fdc0b1ca0 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.737+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fdc04e160 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.739+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2fd801a400 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.740+0000 7f2fd27fc640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 0x7f2fc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.740+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f2fd80898c0 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.740+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 0x7f2fc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.741+0000 7f2fe13a9640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 0x7f2fc006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc036000 tx=0x7f2fcc03c040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.744+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fd8054880 con 0x7f2fdc06a2e0 2023-10-14T16:54:14.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:54:14.926 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:54:14.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:14.975+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f2fdc0ade10 con 0x7f2fdc06a2e0 2023-10-14T16:54:15.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.531+0000 7f2fd27fc640 1 -- 172.21.15.29:0/1938137193 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 6 and data pool 7 v20) v1 ==== 172+0+0 (secure 0 0 0) 0x7f2fd8015020 con 0x7f2fdc06a2e0 2023-10-14T16:54:15.532 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 6 and data pool 7 2023-10-14T16:54:15.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 msgr2=0x7f2fc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 0x7f2fc006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f2fcc036000 tx=0x7f2fcc03c040 comp rx=0 tx=0).stop 2023-10-14T16:54:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 msgr2=0x7f2fdc1576b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800b410 tx=0x7f2fd800b8d0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 shutdown_connections 2023-10-14T16:54:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f2fc006af90 0x7f2fc006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fdc15d150 0x7f2fdc0c7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fdc0ad350 0x7f2fdc0c3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 --2- 172.21.15.29:0/1938137193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fdc06a2e0 0x7f2fdc1576b0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 >> 172.21.15.29:0/1938137193 conn(0x7f2fdc0ce9b0 msgr2=0x7f2fdc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:15.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 shutdown_connections 2023-10-14T16:54:15.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.535+0000 7f2fe3e0e640 1 -- 172.21.15.29:0/1938137193 wait complete. 2023-10-14T16:54:15.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:54:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.767+0000 7fd634a9f640 1 Processor -- start 2023-10-14T16:54:15.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.768+0000 7fd634a9f640 1 -- start start 2023-10-14T16:54:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.769+0000 7fd634a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.769+0000 7fd634a9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6300ad970 0x7fd63015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.769+0000 7fd634a9f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63006c960 con 0x7fd63015ebc0 2023-10-14T16:54:15.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.769+0000 7fd634a9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd63015b9d0 con 0x7fd6300ad970 2023-10-14T16:54:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.769+0000 7fd634a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63015bcb0 con 0x7fd63006c040 2023-10-14T16:54:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6300ad970 0x7fd63015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6300ad970 0x7fd63015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36404/0 (socket says 172.21.15.29:36404) 2023-10-14T16:54:15.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 -- 172.21.15.29:0/105911709 learned_addr learned my addr 172.21.15.29:0/105911709 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 -- 172.21.15.29:0/105911709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 msgr2=0x7fd63006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 -- 172.21.15.29:0/105911709 >> v1:172.21.15.186:6789/0 conn(0x7fd63015ebc0 legacy=0x7fd6300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.770+0000 7fd62effd640 1 -- 172.21.15.29:0/105911709 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd63015bf90 con 0x7fd6300ad970 2023-10-14T16:54:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.771+0000 7fd62effd640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300ad970 0x7fd63015e680 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd6240098a0 tx=0x7fd62402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c690f15192ea54 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.771+0000 7fd62e7fc640 1 -- 172.21.15.29:0/105911709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624031640 con 0x7fd6300ad970 2023-10-14T16:54:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.771+0000 7fd62e7fc640 1 -- 172.21.15.29:0/105911709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6240317a0 con 0x7fd6300ad970 2023-10-14T16:54:15.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.771+0000 7fd62e7fc640 1 -- 172.21.15.29:0/105911709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd624031b30 con 0x7fd6300ad970 2023-10-14T16:54:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 -- 172.21.15.29:0/105911709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300ad970 msgr2=0x7fd63015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300ad970 0x7fd63015e680 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fd6240098a0 tx=0x7fd62402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 -- 172.21.15.29:0/105911709 shutdown_connections 2023-10-14T16:54:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6300ad970 0x7fd63015e680 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 --2- 172.21.15.29:0/105911709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 -- 172.21.15.29:0/105911709 >> 172.21.15.29:0/105911709 conn(0x7fd6300ce6a0 msgr2=0x7fd6300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:15.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 -- 172.21.15.29:0/105911709 shutdown_connections 2023-10-14T16:54:15.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.772+0000 7fd634a9f640 1 -- 172.21.15.29:0/105911709 wait complete. 2023-10-14T16:54:15.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.774+0000 7fd634a9f640 1 Processor -- start 2023-10-14T16:54:15.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.774+0000 7fd634a9f640 1 -- start start 2023-10-14T16:54:15.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6300ad970 0x7fd6300acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300c7c60 con 0x7fd63015ebc0 2023-10-14T16:54:15.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6300af8a0 con 0x7fd6300ad970 2023-10-14T16:54:15.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.775+0000 7fd634a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd63015b1b0 con 0x7fd63006c040 2023-10-14T16:54:15.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36412/0 (socket says 172.21.15.29:36412) 2023-10-14T16:54:15.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 -- 172.21.15.29:0/2320923765 learned_addr learned my addr 172.21.15.29:0/2320923765 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:15.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62effd640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6300ad970 0x7fd6300acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 -- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 msgr2=0x7fd63015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63015ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 -- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6300ad970 msgr2=0x7fd6300acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6300ad970 0x7fd6300acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.776+0000 7fd62ffff640 1 -- 172.21.15.29:0/2320923765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6240094d0 con 0x7fd63015ebc0 2023-10-14T16:54:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd62ffff640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c780 tx=0x7fd62800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628003d80 con 0x7fd63015ebc0 2023-10-14T16:54:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd62800ce90 con 0x7fd63015ebc0 2023-10-14T16:54:15.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280052b0 con 0x7fd63015ebc0 2023-10-14T16:54:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6300afb20 con 0x7fd63015ebc0 2023-10-14T16:54:15.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.777+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6300b00e0 con 0x7fd63015ebc0 2023-10-14T16:54:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.778+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63006b330 con 0x7fd63015ebc0 2023-10-14T16:54:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.779+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd628005410 con 0x7fd63015ebc0 2023-10-14T16:54:15.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.780+0000 7fd617fff640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.780+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fd628088b80 con 0x7fd63015ebc0 2023-10-14T16:54:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.781+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.781+0000 7fd62f7fe640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 0x7fd61c06d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd620006fd0 tx=0x7fd620008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:15.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.785+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd628053ae0 con 0x7fd63015ebc0 2023-10-14T16:54:15.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.981+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd63004e9b0 con 0x7fd63015ebc0 2023-10-14T16:54:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.984+0000 7fd617fff640 1 -- 172.21.15.29:0/2320923765 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v62) v1 ==== 74+0+16584 (secure 0 0 0) 0x7fd628017090 con 0x7fd63015ebc0 2023-10-14T16:54:15.985 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:15.985 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":62,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:54:14.976271+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:54:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 msgr2=0x7fd61c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 0x7fd61c06d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd620006fd0 tx=0x7fd620008040 comp rx=0 tx=0).stop 2023-10-14T16:54:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 msgr2=0x7fd63015cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fd62800c780 tx=0x7fd62800cc40 comp rx=0 tx=0).stop 2023-10-14T16:54:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 shutdown_connections 2023-10-14T16:54:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd63015ebc0 0x7fd63015cb50 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6300ad970 0x7fd6300acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 --2- 172.21.15.29:0/2320923765 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63006c040 0x7fd63015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 >> 172.21.15.29:0/2320923765 conn(0x7fd6300ce6a0 msgr2=0x7fd6300ca770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 shutdown_connections 2023-10-14T16:54:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:15.987+0000 7fd634a9f640 1 -- 172.21.15.29:0/2320923765 wait complete. 2023-10-14T16:54:16.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:54:16.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.229+0000 7fdf8359e640 1 Processor -- start 2023-10-14T16:54:16.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.229+0000 7fdf8359e640 1 -- start start 2023-10-14T16:54:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.230+0000 7fdf8359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160620 0x7fdf840caf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf8359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840b1ca0 0x7fdf840cd220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf8359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf8406afe0 con 0x7fdf84160240 2023-10-14T16:54:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf8359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf840cf540 con 0x7fdf840b1ca0 2023-10-14T16:54:16.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf8359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840cf820 con 0x7fdf84160620 2023-10-14T16:54:16.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840b1ca0 0x7fdf840cd220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf840b1ca0 0x7fdf840cd220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36424/0 (socket says 172.21.15.29:36424) 2023-10-14T16:54:16.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 -- 172.21.15.29:0/2049655275 learned_addr learned my addr 172.21.15.29:0/2049655275 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:16.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 -- 172.21.15.29:0/2049655275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160620 msgr2=0x7fdf840caf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 --2- 172.21.15.29:0/2049655275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160620 0x7fdf840caf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 -- 172.21.15.29:0/2049655275 >> v1:172.21.15.186:6789/0 conn(0x7fdf84160240 legacy=0x7fdf840ceaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.231+0000 7fdf82d9d640 1 -- 172.21.15.29:0/2049655275 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf840cfb00 con 0x7fdf840b1ca0 2023-10-14T16:54:16.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf82d9d640 1 --2- 172.21.15.29:0/2049655275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf840cd220 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009e50 tx=0x7fdf7c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e54821743016b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8159a640 1 -- 172.21.15.29:0/2049655275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c02c9e0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8159a640 1 -- 172.21.15.29:0/2049655275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf7c02cb40 con 0x7fdf840b1ca0 2023-10-14T16:54:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8159a640 1 -- 172.21.15.29:0/2049655275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf7c0343f0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 -- 172.21.15.29:0/2049655275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 msgr2=0x7fdf840cd220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2049655275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf840cd220 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fdf7c009e50 tx=0x7fdf7c02be40 comp rx=0 tx=0).stop 2023-10-14T16:54:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 -- 172.21.15.29:0/2049655275 shutdown_connections 2023-10-14T16:54:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2049655275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf840cd220 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2049655275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160620 0x7fdf840caf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.233+0000 7fdf8359e640 1 -- 172.21.15.29:0/2049655275 >> 172.21.15.29:0/2049655275 conn(0x7fdf84099a10 msgr2=0x7fdf840d0910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:16.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.234+0000 7fdf8359e640 1 -- 172.21.15.29:0/2049655275 shutdown_connections 2023-10-14T16:54:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.234+0000 7fdf8359e640 1 -- 172.21.15.29:0/2049655275 wait complete. 2023-10-14T16:54:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.235+0000 7fdf8359e640 1 Processor -- start 2023-10-14T16:54:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.235+0000 7fdf8359e640 1 -- start start 2023-10-14T16:54:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 0x7fdf840b5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 0x7fdf840b5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840c8ad0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840c8d70 con 0x7fdf84160620 2023-10-14T16:54:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.236+0000 7fdf8359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf840c9050 con 0x7fdf84160240 2023-10-14T16:54:16.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36434/0 (socket says 172.21.15.29:36434) 2023-10-14T16:54:16.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 -- 172.21.15.29:0/2027039440 learned_addr learned my addr 172.21.15.29:0/2027039440 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:16.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf82d9d640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 0x7fdf840b5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 -- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 msgr2=0x7fdf840b5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf81d9b640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 0x7fdf840b5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 0x7fdf840b5a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 -- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 msgr2=0x7fdf840b5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 0x7fdf840b5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf8259c640 1 -- 172.21.15.29:0/2027039440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c009b30 con 0x7fdf840b1ca0 2023-10-14T16:54:16.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf82d9d640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 0x7fdf840b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.237+0000 7fdf81d9b640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 0x7fdf840b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf8259c640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c430 tx=0x7fdf7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74011790 con 0x7fdf840b1ca0 2023-10-14T16:54:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf740118f0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf74011ce0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf840c9330 con 0x7fdf840b1ca0 2023-10-14T16:54:16.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.238+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf840b64e0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.240+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf840b6670 con 0x7fdf840b1ca0 2023-10-14T16:54:16.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.241+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf7401c460 con 0x7fdf840b1ca0 2023-10-14T16:54:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.241+0000 7fdf72ffd640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 0x7fdf6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.242+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fdf74089820 con 0x7fdf840b1ca0 2023-10-14T16:54:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.242+0000 7fdf81d9b640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 0x7fdf6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.242+0000 7fdf81d9b640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 0x7fdf6406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf840ad560 tx=0x7fdf78008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.246+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf740546f0 con 0x7fdf840b1ca0 2023-10-14T16:54:16.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.445+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf840c9850 con 0x7fdf840b1ca0 2023-10-14T16:54:16.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.449+0000 7fdf72ffd640 1 -- 172.21.15.29:0/2027039440 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16571 (secure 0 0 0) 0x7fdf74057d10 con 0x7fdf840b1ca0 2023-10-14T16:54:16.450 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:16.450 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:54:16.028566+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:54:16.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 msgr2=0x7fdf6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 0x7fdf6406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf840ad560 tx=0x7fdf78008040 comp rx=0 tx=0).stop 2023-10-14T16:54:16.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 msgr2=0x7fdf8415b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fdf7400c430 tx=0x7fdf7400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 shutdown_connections 2023-10-14T16:54:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fdf6406af90 0x7fdf6406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf84160620 0x7fdf840b5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf84160240 0x7fdf840b5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 --2- 172.21.15.29:0/2027039440 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf840b1ca0 0x7fdf8415b180 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 >> 172.21.15.29:0/2027039440 conn(0x7fdf84099a10 msgr2=0x7fdf8415f400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:16.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 shutdown_connections 2023-10-14T16:54:16.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.452+0000 7fdf8359e640 1 -- 172.21.15.29:0/2027039440 wait complete. 2023-10-14T16:54:16.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:54:16.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.705+0000 7fcf84900640 1 Processor -- start 2023-10-14T16:54:16.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.705+0000 7fcf84900640 1 -- start start 2023-10-14T16:54:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.706+0000 7fcf84900640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf8015aa60 0x7fcf800af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.706+0000 7fcf84900640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf800afd60 0x7fcf800cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.706+0000 7fcf84900640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf800b3730 con 0x7fcf800cc820 2023-10-14T16:54:16.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.706+0000 7fcf84900640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800b3980 con 0x7fcf800afd60 2023-10-14T16:54:16.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.706+0000 7fcf84900640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf800b3c60 con 0x7fcf8015aa60 2023-10-14T16:54:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7f59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcf800cc820 0x7fcf800b2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39692/0 (socket says 172.21.15.29:39692) 2023-10-14T16:54:16.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7f59e640 1 -- 172.21.15.29:0/1397634404 learned_addr learned my addr 172.21.15.29:0/1397634404 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1397634404 >> v2:172.21.15.29:3300/0 conn(0x7fcf800afd60 0x7fcf800cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7ed9d640 1 --2- 172.21.15.29:0/1397634404 >> v2:172.21.15.186:3300/0 conn(0x7fcf8015aa60 0x7fcf800af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316552859 0 0) 0x7fcf800b3730 con 0x7fcf800cc820 2023-10-14T16:54:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.707+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf6c003520 con 0x7fcf800cc820 2023-10-14T16:54:16.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270722782 0 0) 0x7fcf6c003520 con 0x7fcf800cc820 2023-10-14T16:54:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 >> v2:172.21.15.186:3300/0 conn(0x7fcf8015aa60 msgr2=0x7fcf800af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 --2- 172.21.15.29:0/1397634404 >> v2:172.21.15.186:3300/0 conn(0x7fcf8015aa60 0x7fcf800af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 >> v2:172.21.15.29:3300/0 conn(0x7fcf800afd60 msgr2=0x7fcf800cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 --2- 172.21.15.29:0/1397634404 >> v2:172.21.15.29:3300/0 conn(0x7fcf800afd60 0x7fcf800cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800b3f40 con 0x7fcf800cc820 2023-10-14T16:54:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fcf78001670 con 0x7fcf800cc820 2023-10-14T16:54:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 >> v1:172.21.15.29:6789/0 conn(0x7fcf800cc820 legacy=0x7fcf800b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf6c0039c0 0x7fcf6c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 0x7fcf6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c005180 0x7fcf6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800b3980 con 0x7fcf6c0047c0 2023-10-14T16:54:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800b3c60 con 0x7fcf6c005180 2023-10-14T16:54:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf6c00c020 con 0x7fcf6c0039c0 2023-10-14T16:54:16.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.708+0000 7fcf7f59e640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c005180 0x7fcf6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 0x7fcf6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7ed9d640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf6c0039c0 0x7fcf6c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf6c0039c0 msgr2=0x7fcf6c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf6c0039c0 0x7fcf6c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c005180 msgr2=0x7fcf6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c005180 0x7fcf6c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1397634404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf6c003520 con 0x7fcf6c0047c0 2023-10-14T16:54:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7f59e640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c005180 0x7fcf6c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:54:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.709+0000 7fcf7f59e640 1 -- 172.21.15.29:0/1397634404 reap_dead start 2023-10-14T16:54:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.712+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 0x7fcf6c004c40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800ba30 tx=0x7fcf6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=315cd8f8c7e1ecc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68032640 con 0x7fcf6c0047c0 2023-10-14T16:54:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf680327a0 con 0x7fcf6c0047c0 2023-10-14T16:54:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf7dd9b640 1 -- 172.21.15.29:0/1397634404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68032b30 con 0x7fcf6c0047c0 2023-10-14T16:54:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 -- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 msgr2=0x7fcf6c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 0x7fcf6c004c40 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fcf6800ba30 tx=0x7fcf6c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 -- 172.21.15.29:0/1397634404 shutdown_connections 2023-10-14T16:54:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 --2- 172.21.15.29:0/1397634404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf6c0047c0 0x7fcf6c004c40 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 -- 172.21.15.29:0/1397634404 >> 172.21.15.29:0/1397634404 conn(0x7fcf80065ed0 msgr2=0x7fcf800662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.713+0000 7fcf84900640 1 -- 172.21.15.29:0/1397634404 shutdown_connections 2023-10-14T16:54:16.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.714+0000 7fcf84900640 1 -- 172.21.15.29:0/1397634404 wait complete. 2023-10-14T16:54:16.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.715+0000 7fcf84900640 1 Processor -- start 2023-10-14T16:54:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.716+0000 7fcf84900640 1 -- start start 2023-10-14T16:54:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.716+0000 7fcf84900640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c0047c0 0x7fcf800b1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf84900640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf84900640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf80151eb0 0x7fcf800b20a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf84900640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c5320 con 0x7fcf8015ae40 2023-10-14T16:54:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf84900640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800c54b0 con 0x7fcf6c0047c0 2023-10-14T16:54:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf84900640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf800b1290 con 0x7fcf80151eb0 2023-10-14T16:54:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf7e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf7e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36456/0 (socket says 172.21.15.29:36456) 2023-10-14T16:54:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1742046057 learned_addr learned my addr 172.21.15.29:0/1742046057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf7f59e640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf80151eb0 0x7fcf800b20a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.717+0000 7fcf7ed9d640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c0047c0 0x7fcf800b1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf80151eb0 msgr2=0x7fcf800b20a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf80151eb0 0x7fcf800b20a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c0047c0 msgr2=0x7fcf800b1a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c0047c0 0x7fcf800b1a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:16.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 -- 172.21.15.29:0/1742046057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf800b14f0 con 0x7fcf8015ae40 2023-10-14T16:54:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf7e59c640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcf6802bec0 tx=0x7fcf68004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf68004220 con 0x7fcf8015ae40 2023-10-14T16:54:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf680043a0 con 0x7fcf8015ae40 2023-10-14T16:54:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf8015d5d0 con 0x7fcf8015ae40 2023-10-14T16:54:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.718+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf6803ca80 con 0x7fcf8015ae40 2023-10-14T16:54:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.719+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf8015db50 con 0x7fcf8015ae40 2023-10-14T16:54:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.720+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf8015dce0 con 0x7fcf8015ae40 2023-10-14T16:54:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.721+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcf6803cbe0 con 0x7fcf8015ae40 2023-10-14T16:54:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.722+0000 7fcf777fe640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 0x7fcf6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.722+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fcf680ac410 con 0x7fcf8015ae40 2023-10-14T16:54:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.722+0000 7fcf7ed9d640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 0x7fcf6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.723+0000 7fcf7ed9d640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 0x7fcf6006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf700132b0 tx=0x7fcf7000b040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.727+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf68077380 con 0x7fcf8015ae40 2023-10-14T16:54:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:16.962+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcf8015dce0 con 0x7fcf8015ae40 2023-10-14T16:54:17.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.568+0000 7fcf777fe640 1 -- 172.21.15.29:0/1742046057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v23) v1 ==== 118+0+0 (secure 0 0 0) 0x7fcf6807a9a0 con 0x7fcf8015ae40 2023-10-14T16:54:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.572+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 msgr2=0x7fcf6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.572+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 0x7fcf6006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fcf700132b0 tx=0x7fcf7000b040 comp rx=0 tx=0).stop 2023-10-14T16:54:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.572+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 msgr2=0x7fcf800b0d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.572+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fcf6802bec0 tx=0x7fcf68004730 comp rx=0 tx=0).stop 2023-10-14T16:54:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 shutdown_connections 2023-10-14T16:54:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fcf6006af90 0x7fcf6006d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf80151eb0 0x7fcf800b20a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf8015ae40 0x7fcf800b0d50 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 --2- 172.21.15.29:0/1742046057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf6c0047c0 0x7fcf800b1a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 >> 172.21.15.29:0/1742046057 conn(0x7fcf80065ed0 msgr2=0x7fcf800662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 shutdown_connections 2023-10-14T16:54:17.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.573+0000 7fcf84900640 1 -- 172.21.15.29:0/1742046057 wait complete. 2023-10-14T16:54:17.595 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:54:17.595 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:54:17.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.815+0000 7f14faa7c640 1 Processor -- start 2023-10-14T16:54:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.815+0000 7f14faa7c640 1 -- start start 2023-10-14T16:54:17.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.816+0000 7f14faa7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f406d570 0x7f14f406c000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.816+0000 7f14faa7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14f40caca0 0x7f14f406c540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14faa7c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14f415b1f0 con 0x7f14f415ae10 2023-10-14T16:54:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14faa7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14f406f650 con 0x7f14f40caca0 2023-10-14T16:54:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14faa7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f406bb70 con 0x7f14f406d570 2023-10-14T16:54:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14f40caca0 0x7f14f406c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14f40caca0 0x7f14f406c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36470/0 (socket says 172.21.15.29:36470) 2023-10-14T16:54:17.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 -- 172.21.15.29:0/3319781233 learned_addr learned my addr 172.21.15.29:0/3319781233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:17.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 -- 172.21.15.29:0/3319781233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f406d570 msgr2=0x7f14f406c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:17.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 --2- 172.21.15.29:0/3319781233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f406d570 0x7f14f406c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 -- 172.21.15.29:0/3319781233 >> v1:172.21.15.186:6789/0 conn(0x7f14f415ae10 legacy=0x7f14f40ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:17.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.817+0000 7f14f9019640 1 -- 172.21.15.29:0/3319781233 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f40b4190 con 0x7f14f40caca0 2023-10-14T16:54:17.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.818+0000 7f14f9019640 1 --2- 172.21.15.29:0/3319781233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f40caca0 0x7f14f406c540 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cb2e2f875db767b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:17.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.819+0000 7f14f37fe640 1 -- 172.21.15.29:0/3319781233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec031640 con 0x7f14f40caca0 2023-10-14T16:54:17.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.819+0000 7f14f37fe640 1 -- 172.21.15.29:0/3319781233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14ec0317a0 con 0x7f14f40caca0 2023-10-14T16:54:17.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.819+0000 7f14f37fe640 1 -- 172.21.15.29:0/3319781233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14ec031b30 con 0x7f14f40caca0 2023-10-14T16:54:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.819+0000 7f14faa7c640 1 -- 172.21.15.29:0/3319781233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f40caca0 msgr2=0x7f14f406c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.819+0000 7f14faa7c640 1 --2- 172.21.15.29:0/3319781233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f40caca0 0x7f14f406c540 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f14ec009870 tx=0x7f14ec02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 -- 172.21.15.29:0/3319781233 shutdown_connections 2023-10-14T16:54:17.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 --2- 172.21.15.29:0/3319781233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f40caca0 0x7f14f406c540 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 --2- 172.21.15.29:0/3319781233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f406d570 0x7f14f406c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 -- 172.21.15.29:0/3319781233 >> 172.21.15.29:0/3319781233 conn(0x7f14f40ce840 msgr2=0x7f14f40cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 -- 172.21.15.29:0/3319781233 shutdown_connections 2023-10-14T16:54:17.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.820+0000 7f14faa7c640 1 -- 172.21.15.29:0/3319781233 wait complete. 2023-10-14T16:54:17.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.822+0000 7f14faa7c640 1 Processor -- start 2023-10-14T16:54:17.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.822+0000 7f14faa7c640 1 -- start start 2023-10-14T16:54:17.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f40caca0 0x7f14f40c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14f415ae10 0x7f14f40ad280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40ce470 con 0x7f14f406d570 2023-10-14T16:54:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cd3d0 con 0x7f14f415ae10 2023-10-14T16:54:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14faa7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14f40cd6b0 con 0x7f14f40caca0 2023-10-14T16:54:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14f8818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:17.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14f8818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36484/0 (socket says 172.21.15.29:36484) 2023-10-14T16:54:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14f8818640 1 -- 172.21.15.29:0/4007915596 learned_addr learned my addr 172.21.15.29:0/4007915596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14f3fff640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f40caca0 0x7f14f40c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.823+0000 7f14f9019640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14f415ae10 0x7f14f40ad280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 -- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f40caca0 msgr2=0x7f14f40c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f40caca0 0x7f14f40c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 -- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14f415ae10 msgr2=0x7f14f40ad280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14f415ae10 0x7f14f40ad280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 -- 172.21.15.29:0/4007915596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14ec0094d0 con 0x7f14f406d570 2023-10-14T16:54:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f8818640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c7a0 tx=0x7f14e400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4003d80 con 0x7f14f406d570 2023-10-14T16:54:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14e4005070 con 0x7f14f406d570 2023-10-14T16:54:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14f40cd990 con 0x7f14f406d570 2023-10-14T16:54:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14e4005420 con 0x7f14f406d570 2023-10-14T16:54:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.824+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14f40cdf50 con 0x7f14f406d570 2023-10-14T16:54:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.826+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14f404e160 con 0x7f14f406d570 2023-10-14T16:54:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.826+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f14e4005580 con 0x7f14f406d570 2023-10-14T16:54:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.827+0000 7f14f17fa640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 0x7f14e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.828+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f14e4088c70 con 0x7f14f406d570 2023-10-14T16:54:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.828+0000 7f14f3fff640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 0x7f14e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.828+0000 7f14f3fff640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 0x7f14e006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14f40b2c30 tx=0x7f14e8008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:17.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:17.832+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14e4053be0 con 0x7f14f406d570 2023-10-14T16:54:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.071+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f14f40b0090 con 0x7f14f406d570 2023-10-14T16:54:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.571+0000 7f14f17fa640 1 -- 172.21.15.29:0/4007915596 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7f14e4057200 con 0x7f14f406d570 2023-10-14T16:54:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 msgr2=0x7f14e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:18.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 0x7f14e006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f14f40b2c30 tx=0x7f14e8008040 comp rx=0 tx=0).stop 2023-10-14T16:54:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 msgr2=0x7f14f4156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f14e400c7a0 tx=0x7f14e400cc60 comp rx=0 tx=0).stop 2023-10-14T16:54:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 shutdown_connections 2023-10-14T16:54:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f14e006af90 0x7f14e006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14f415ae10 0x7f14f40ad280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14f40caca0 0x7f14f40c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 --2- 172.21.15.29:0/4007915596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14f406d570 0x7f14f4156a10 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.575+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 >> 172.21.15.29:0/4007915596 conn(0x7f14f40ce840 msgr2=0x7f14f40b2f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.576+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 shutdown_connections 2023-10-14T16:54:18.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.576+0000 7f14faa7c640 1 -- 172.21.15.29:0/4007915596 wait complete. 2023-10-14T16:54:18.598 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:54:18.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:54:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.828+0000 7f8a874bb640 1 Processor -- start 2023-10-14T16:54:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.829+0000 7f8a874bb640 1 -- start start 2023-10-14T16:54:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 0x7f8a8006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a800ad970 0x7f8a8015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8006c960 con 0x7f8a800ad970 2023-10-14T16:54:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8015b9d0 con 0x7f8a8006c040 2023-10-14T16:54:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.830+0000 7f8a874bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8015bcb0 con 0x7f8a8015ebc0 2023-10-14T16:54:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a800ad970 0x7f8a8015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a800ad970 0x7f8a8015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36492/0 (socket says 172.21.15.29:36492) 2023-10-14T16:54:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 -- 172.21.15.29:0/1822712794 learned_addr learned my addr 172.21.15.29:0/1822712794 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a86cba640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a864b9640 1 --2- 172.21.15.29:0/1822712794 >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 0x7f8a8006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 -- 172.21.15.29:0/1822712794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 msgr2=0x7f8a8015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 -- 172.21.15.29:0/1822712794 >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 msgr2=0x7f8a8006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/1822712794 >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 0x7f8a8006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a85cb8640 1 -- 172.21.15.29:0/1822712794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8015bf90 con 0x7f8a800ad970 2023-10-14T16:54:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.831+0000 7f8a864b9640 1 --2- 172.21.15.29:0/1822712794 >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 0x7f8a8006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.832+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a800ad970 0x7f8a8015e680 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cf350 tx=0x7f8a7002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45742f7687e0680 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.833+0000 7f8a854b7640 1 -- 172.21.15.29:0/1822712794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70031870 con 0x7f8a800ad970 2023-10-14T16:54:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.833+0000 7f8a854b7640 1 -- 172.21.15.29:0/1822712794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a700319d0 con 0x7f8a800ad970 2023-10-14T16:54:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.833+0000 7f8a854b7640 1 -- 172.21.15.29:0/1822712794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70031d60 con 0x7f8a800ad970 2023-10-14T16:54:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.833+0000 7f8a874bb640 1 -- 172.21.15.29:0/1822712794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a800ad970 msgr2=0x7f8a8015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.833+0000 7f8a874bb640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a800ad970 0x7f8a8015e680 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f8a800cf350 tx=0x7f8a7002b910 comp rx=0 tx=0).stop 2023-10-14T16:54:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 -- 172.21.15.29:0/1822712794 shutdown_connections 2023-10-14T16:54:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 --2- 172.21.15.29:0/1822712794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a800ad970 0x7f8a8015e680 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 --2- 172.21.15.29:0/1822712794 >> v2:172.21.15.186:3300/0 conn(0x7f8a8006c040 0x7f8a8006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 -- 172.21.15.29:0/1822712794 >> 172.21.15.29:0/1822712794 conn(0x7f8a800ce6a0 msgr2=0x7f8a800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:18.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 -- 172.21.15.29:0/1822712794 shutdown_connections 2023-10-14T16:54:18.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.834+0000 7f8a874bb640 1 -- 172.21.15.29:0/1822712794 wait complete. 2023-10-14T16:54:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.836+0000 7f8a874bb640 1 Processor -- start 2023-10-14T16:54:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.836+0000 7f8a874bb640 1 -- start start 2023-10-14T16:54:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800ad970 0x7f8a80158440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c85b0 con 0x7f8a8006c040 2023-10-14T16:54:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c6fd0 con 0x7f8a800ad970 2023-10-14T16:54:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.837+0000 7f8a874bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a800c72b0 con 0x7f8a8015ebc0 2023-10-14T16:54:18.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36498/0 (socket says 172.21.15.29:36498) 2023-10-14T16:54:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 -- 172.21.15.29:0/3561801059 learned_addr learned my addr 172.21.15.29:0/3561801059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a86cba640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800ad970 0x7f8a80158440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 -- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 msgr2=0x7f8a8015c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 -- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800ad970 msgr2=0x7f8a80158440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800ad970 0x7f8a80158440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.838+0000 7f8a864b9640 1 -- 172.21.15.29:0/3561801059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a700094d0 con 0x7f8a8006c040 2023-10-14T16:54:18.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a864b9640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c7c0 tx=0x7f8a7c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c007a30 con 0x7f8a8006c040 2023-10-14T16:54:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a7c007b90 con 0x7f8a8006c040 2023-10-14T16:54:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c00ce70 con 0x7f8a8006c040 2023-10-14T16:54:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a800c7590 con 0x7f8a8006c040 2023-10-14T16:54:18.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.839+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a800c7b50 con 0x7f8a8006c040 2023-10-14T16:54:18.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.840+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8006b330 con 0x7f8a8006c040 2023-10-14T16:54:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.841+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a7c0113f0 con 0x7f8a8006c040 2023-10-14T16:54:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.842+0000 7f8a76ffd640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 0x7f8a6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.842+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f8a7c0899d0 con 0x7f8a8006c040 2023-10-14T16:54:18.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.843+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 0x7f8a6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:18.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.843+0000 7f8a85cb8640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 0x7f8a6406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a70036000 tx=0x7f8a7003c040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:18.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:18.847+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a7c054820 con 0x7f8a8006c040 2023-10-14T16:54:19.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.083+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8a8004e9b0 con 0x7f8a8006c040 2023-10-14T16:54:19.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.084+0000 7f8a76ffd640 1 -- 172.21.15.29:0/3561801059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f8a7c01f090 con 0x7f8a8006c040 2023-10-14T16:54:19.085 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:19.085 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:54:19.085 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:54:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.086+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 msgr2=0x7f8a6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.086+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 0x7f8a6406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8a70036000 tx=0x7f8a7003c040 comp rx=0 tx=0).stop 2023-10-14T16:54:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.086+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 msgr2=0x7f8a800c8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.086+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c00c7c0 tx=0x7f8a7c00cc80 comp rx=0 tx=0).stop 2023-10-14T16:54:19.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 shutdown_connections 2023-10-14T16:54:19.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a6406af90 0x7f8a6406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8015ebc0 0x7f8a8015c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a800ad970 0x7f8a80158440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 --2- 172.21.15.29:0/3561801059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8006c040 0x7f8a800c8070 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 >> 172.21.15.29:0/3561801059 conn(0x7f8a800ce6a0 msgr2=0x7f8a800cd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:19.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 shutdown_connections 2023-10-14T16:54:19.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.087+0000 7f8a874bb640 1 -- 172.21.15.29:0/3561801059 wait complete. 2023-10-14T16:54:19.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:54:19.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.337+0000 7f0410d99640 1 Processor -- start 2023-10-14T16:54:19.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.337+0000 7f0410d99640 1 -- start start 2023-10-14T16:54:19.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.338+0000 7f0410d99640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.338+0000 7f0410d99640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 0x7f040c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.338+0000 7f0410d99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f040c06c960 con 0x7f040c0ad970 2023-10-14T16:54:19.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.338+0000 7f0410d99640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c15b9d0 con 0x7f040c06c040 2023-10-14T16:54:19.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.338+0000 7f0410d99640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c15bcb0 con 0x7f040c0cda80 2023-10-14T16:54:19.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f040c0ad970 0x7f040c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39696/0 (socket says 172.21.15.29:39696) 2023-10-14T16:54:19.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040affd640 1 -- 172.21.15.29:0/2698902833 learned_addr learned my addr 172.21.15.29:0/2698902833 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:19.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040b7fe640 1 --2- 172.21.15.29:0/2698902833 >> v2:172.21.15.29:3300/0 conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040bfff640 1 --2- 172.21.15.29:0/2698902833 >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 0x7f040c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040b7fe640 1 -- 172.21.15.29:0/2698902833 >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 msgr2=0x7f040c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:19.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040b7fe640 1 --2- 172.21.15.29:0/2698902833 >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 0x7f040c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.339+0000 7f040b7fe640 1 -- 172.21.15.29:0/2698902833 >> v1:172.21.15.29:6789/0 conn(0x7f040c0ad970 legacy=0x7f040c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.340+0000 7f040b7fe640 1 -- 172.21.15.29:0/2698902833 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c15bf90 con 0x7f040c06c040 2023-10-14T16:54:19.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.340+0000 7f040a7fc640 1 -- 172.21.15.29:0/2698902833 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478697005 0 0) 0x7f040c06c960 con 0x7f040c0ad970 2023-10-14T16:54:19.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.340+0000 7f040bfff640 1 --2- 172.21.15.29:0/2698902833 >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 0x7f040c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.341+0000 7f040b7fe640 1 --2- 172.21.15.29:0/2698902833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c06c420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f040c0cf300 tx=0x7f03fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cffd47d2bb1bf4e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.341+0000 7f040a7fc640 1 -- 172.21.15.29:0/2698902833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc031830 con 0x7f040c06c040 2023-10-14T16:54:19.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.341+0000 7f040a7fc640 1 -- 172.21.15.29:0/2698902833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc031990 con 0x7f040c06c040 2023-10-14T16:54:19.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.341+0000 7f040a7fc640 1 -- 172.21.15.29:0/2698902833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc031d20 con 0x7f040c06c040 2023-10-14T16:54:19.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 -- 172.21.15.29:0/2698902833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 msgr2=0x7f040c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 --2- 172.21.15.29:0/2698902833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c06c420 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f040c0cf300 tx=0x7f03fc02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 -- 172.21.15.29:0/2698902833 shutdown_connections 2023-10-14T16:54:19.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 --2- 172.21.15.29:0/2698902833 >> v2:172.21.15.186:3300/0 conn(0x7f040c0cda80 0x7f040c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 --2- 172.21.15.29:0/2698902833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 -- 172.21.15.29:0/2698902833 >> 172.21.15.29:0/2698902833 conn(0x7f040c0ce6a0 msgr2=0x7f040c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:19.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 -- 172.21.15.29:0/2698902833 shutdown_connections 2023-10-14T16:54:19.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.342+0000 7f0410d99640 1 -- 172.21.15.29:0/2698902833 wait complete. 2023-10-14T16:54:19.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.344+0000 7f0410d99640 1 Processor -- start 2023-10-14T16:54:19.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.344+0000 7f0410d99640 1 -- start start 2023-10-14T16:54:19.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c0ad970 0x7f040c1581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 0x7f040c0b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0b0e90 con 0x7f040c06c040 2023-10-14T16:54:19.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0c7250 con 0x7f040c0cda80 2023-10-14T16:54:19.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f0410d99640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0c6920 con 0x7f040c0ad970 2023-10-14T16:54:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f040b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36506/0 (socket says 172.21.15.29:36506) 2023-10-14T16:54:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f040b7fe640 1 -- 172.21.15.29:0/3015229176 learned_addr learned my addr 172.21.15.29:0/3015229176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f040affd640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c0ad970 0x7f040c1581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.345+0000 7f040bfff640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 0x7f040c0b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 -- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c0ad970 msgr2=0x7f040c1581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c0ad970 0x7f040c1581c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 -- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 msgr2=0x7f040c0b05d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 0x7f040c0b05d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 -- 172.21.15.29:0/3015229176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03fc009400 con 0x7f040c06c040 2023-10-14T16:54:19.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040bfff640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 0x7f040c0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:19.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f040b7fe640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009890 tx=0x7f03fc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc004230 con 0x7f040c06c040 2023-10-14T16:54:19.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03fc004390 con 0x7f040c06c040 2023-10-14T16:54:19.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03fc03a880 con 0x7f040c06c040 2023-10-14T16:54:19.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c0c6c00 con 0x7f040c06c040 2023-10-14T16:54:19.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.346+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c0b0b10 con 0x7f040c06c040 2023-10-14T16:54:19.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.348+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c06b330 con 0x7f040c06c040 2023-10-14T16:54:19.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.348+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03fc03a9e0 con 0x7f040c06c040 2023-10-14T16:54:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.349+0000 7f0403fff640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 0x7f03ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.349+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f03fc0ac2c0 con 0x7f040c06c040 2023-10-14T16:54:19.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.350+0000 7f040affd640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 0x7f03ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.350+0000 7f040affd640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 0x7f03ec06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f040c0c5000 tx=0x7f03f4007ed0 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.354+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03fc077240 con 0x7f040c06c040 2023-10-14T16:54:19.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.553+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f040c04e9b0 con 0x7f040c06c040 2023-10-14T16:54:19.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.556+0000 7f0403fff640 1 -- 172.21.15.29:0/3015229176 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v63) v1 ==== 74+0+16571 (secure 0 0 0) 0x7f03fc07a860 con 0x7f040c06c040 2023-10-14T16:54:19.558 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:19.558 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":63,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:54:16.028566+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:54:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 msgr2=0x7f03ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 0x7f03ec06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f040c0c5000 tx=0x7f03f4007ed0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 msgr2=0x7f040c158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f03fc009890 tx=0x7f03fc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 shutdown_connections 2023-10-14T16:54:19.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f03ec06af90 0x7f03ec06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0cda80 0x7f040c0b05d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c0ad970 0x7f040c1581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 --2- 172.21.15.29:0/3015229176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c06c040 0x7f040c158790 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.559+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 >> 172.21.15.29:0/3015229176 conn(0x7f040c0ce6a0 msgr2=0x7f040c15aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:19.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.560+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 shutdown_connections 2023-10-14T16:54:19.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.560+0000 7f0410d99640 1 -- 172.21.15.29:0/3015229176 wait complete. 2023-10-14T16:54:19.579 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:54:19.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:54:19.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.806+0000 7f4ab9def640 1 Processor -- start 2023-10-14T16:54:19.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.806+0000 7f4ab9def640 1 -- start start 2023-10-14T16:54:19.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.807+0000 7f4ab9def640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4ab40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.807+0000 7f4ab9def640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab415d150 con 0x7f4ab40d0be0 2023-10-14T16:54:19.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.807+0000 7f4ab9def640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab40d1180 con 0x7f4ab415cd70 2023-10-14T16:54:19.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.807+0000 7f4ab9def640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40d1460 con 0x7f4ab40b8480 2023-10-14T16:54:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.808+0000 7f4ab95ee640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4ab40d0be0 0x7f4ab40b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39704/0 (socket says 172.21.15.29:39704) 2023-10-14T16:54:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.808+0000 7f4ab95ee640 1 -- 172.21.15.29:0/321110750 learned_addr learned my addr 172.21.15.29:0/321110750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.808+0000 7f4aabfff640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4ab40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.808+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708187754 0 0) 0x7f4ab415d150 con 0x7f4ab40d0be0 2023-10-14T16:54:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.808+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a9c003520 con 0x7f4ab40d0be0 2023-10-14T16:54:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2626846000 0 0) 0x7f4a9c003520 con 0x7f4ab40d0be0 2023-10-14T16:54:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 msgr2=0x7f4ab40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4ab40cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 >> v1:172.21.15.186:6789/0 conn(0x7f4ab415cd70 legacy=0x7f4ab40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab40d1740 con 0x7f4ab40d0be0 2023-10-14T16:54:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4aac0019f0 con 0x7f4ab40d0be0 2023-10-14T16:54:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 >> v1:172.21.15.29:6789/0 conn(0x7f4ab40d0be0 legacy=0x7f4ab40b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4a9c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c003f70 0x7f4a9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a9c004d00 0x7f4a9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40d1460 con 0x7f4a9c003f70 2023-10-14T16:54:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c00bac0 con 0x7f4a9c004d00 2023-10-14T16:54:19.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a9c00bc80 con 0x7f4ab40b8480 2023-10-14T16:54:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aabfff640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c003f70 0x7f4a9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4ab95ee640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a9c004d00 0x7f4a9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.809+0000 7f4aabfff640 1 -- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 msgr2=0x7f4a9c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4ab8ded640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4a9c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4aabfff640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4a9c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4aabfff640 1 -- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a9c004d00 msgr2=0x7f4a9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4aabfff640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a9c004d00 0x7f4a9c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4aabfff640 1 -- 172.21.15.29:0/321110750 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a9c003520 con 0x7f4a9c003f70 2023-10-14T16:54:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4ab95ee640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a9c004d00 0x7f4a9c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:19.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.810+0000 7f4ab8ded640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ab40b8480 0x7f4a9c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:19.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.811+0000 7f4aabfff640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c003f70 0x7f4a9c0047c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009ac0 tx=0x7f4a9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbae9c933ac7f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.811+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa402c600 con 0x7f4a9c003f70 2023-10-14T16:54:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.811+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa402c760 con 0x7f4a9c003f70 2023-10-14T16:54:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.811+0000 7f4aab7fe640 1 -- 172.21.15.29:0/321110750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa402caf0 con 0x7f4a9c003f70 2023-10-14T16:54:19.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 -- 172.21.15.29:0/321110750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c003f70 msgr2=0x7f4a9c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 --2- 172.21.15.29:0/321110750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a9c003f70 0x7f4a9c0047c0 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4009ac0 tx=0x7f4a9c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab95ee640 1 -- 172.21.15.29:0/321110750 reap_dead start 2023-10-14T16:54:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 -- 172.21.15.29:0/321110750 shutdown_connections 2023-10-14T16:54:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 -- 172.21.15.29:0/321110750 >> 172.21.15.29:0/321110750 conn(0x7f4ab40cf570 msgr2=0x7f4ab40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:19.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 -- 172.21.15.29:0/321110750 shutdown_connections 2023-10-14T16:54:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.812+0000 7f4ab9def640 1 -- 172.21.15.29:0/321110750 wait complete. 2023-10-14T16:54:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.814+0000 7f4ab9def640 1 Processor -- start 2023-10-14T16:54:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.814+0000 7f4ab9def640 1 -- start start 2023-10-14T16:54:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a9c003f70 0x7f4ab40c9d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ab415d750 0x7f4ab415dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40ca450 con 0x7f4ab40cb150 2023-10-14T16:54:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab40cb5d0 con 0x7f4ab415d750 2023-10-14T16:54:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.815+0000 7f4ab9def640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab4157a60 con 0x7f4a9c003f70 2023-10-14T16:54:19.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36522/0 (socket says 172.21.15.29:36522) 2023-10-14T16:54:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 -- 172.21.15.29:0/4061255757 learned_addr learned my addr 172.21.15.29:0/4061255757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4ab8ded640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a9c003f70 0x7f4ab40c9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4ab95ee640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ab415d750 0x7f4ab415dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 -- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a9c003f70 msgr2=0x7f4ab40c9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a9c003f70 0x7f4ab40c9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 -- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ab415d750 msgr2=0x7f4ab415dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ab415d750 0x7f4ab415dbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.816+0000 7f4aabfff640 1 -- 172.21.15.29:0/4061255757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab4157d40 con 0x7f4ab40cb150 2023-10-14T16:54:19.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4aabfff640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4037040 tx=0x7f4aa4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003810 con 0x7f4ab40cb150 2023-10-14T16:54:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa4003970 con 0x7f4ab40cb150 2023-10-14T16:54:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4aa4003ce0 con 0x7f4ab40cb150 2023-10-14T16:54:19.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab4158020 con 0x7f4ab40cb150 2023-10-14T16:54:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.817+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab41524a0 con 0x7f4ab40cb150 2023-10-14T16:54:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.818+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab404e9b0 con 0x7f4ab40cb150 2023-10-14T16:54:19.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.819+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4aa4034440 con 0x7f4ab40cb150 2023-10-14T16:54:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.820+0000 7f4aa97fa640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 0x7f4aa006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.820+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f4aa40ab670 con 0x7f4ab40cb150 2023-10-14T16:54:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.821+0000 7f4ab8ded640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 0x7f4aa006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:19.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.821+0000 7f4ab8ded640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 0x7f4aa006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ab41f5700 tx=0x7f4ab0001190 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:19.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:19.825+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4aa40767a0 con 0x7f4ab40cb150 2023-10-14T16:54:20.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.018+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 --> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ab4157bd0 con 0x7f4aa006af90 2023-10-14T16:54:20.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.029+0000 7f4aa97fa640 1 -- 172.21.15.29:0/4061255757 <== mgr.5763 v2:172.21.15.29:6832/98172895 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286455 (secure 0 0 0) 0x7f4ab4157bd0 con 0x7f4aa006af90 2023-10-14T16:54:20.034 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:20.035 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:54:20.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 msgr2=0x7f4aa006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 0x7f4aa006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f4ab41f5700 tx=0x7f4ab0001190 comp rx=0 tx=0).stop 2023-10-14T16:54:20.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 msgr2=0x7f4ab4151f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f4aa4037040 tx=0x7f4aa4004730 comp rx=0 tx=0).stop 2023-10-14T16:54:20.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 shutdown_connections 2023-10-14T16:54:20.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.034+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f4aa006af90 0x7f4aa006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ab415d750 0x7f4ab415dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ab40cb150 0x7f4ab4151f60 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 --2- 172.21.15.29:0/4061255757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a9c003f70 0x7f4ab40c9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 >> 172.21.15.29:0/4061255757 conn(0x7f4ab40cf570 msgr2=0x7f4ab40b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:20.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 shutdown_connections 2023-10-14T16:54:20.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.035+0000 7f4ab9def640 1 -- 172.21.15.29:0/4061255757 wait complete. 2023-10-14T16:54:20.054 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-14T16:54:18.622468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330060,"kb_used_data":7056,"kb_used_omap":12,"kb_used_meta":322867,"kb_avail":754644660,"statfs":{"total":773094113280,"available":772756131840,"internally_reserved":0,"allocated":7225344,"data_stored":3986092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":330616376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129592"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042521+0000","last_change":"2023-10-14T16:54:15.051309+0000","last_active":"2023-10-14T16:54:16.042521+0000","last_peered":"2023-10-14T16:54:16.042521+0000","last_clean":"2023-10-14T16:54:16.042521+0000","last_became_active":"2023-10-14T16:54:15.051034+0000","last_became_peered":"2023-10-14T16:54:15.051034+0000","last_unstale":"2023-10-14T16:54:16.042521+0000","last_undegraded":"2023-10-14T16:54:16.042521+0000","last_fullsized":"2023-10-14T16:54:16.042521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:24.753631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037016+0000","last_change":"2023-10-14T16:54:15.061821+0000","last_active":"2023-10-14T16:54:16.037016+0000","last_peered":"2023-10-14T16:54:16.037016+0000","last_clean":"2023-10-14T16:54:16.037016+0000","last_became_active":"2023-10-14T16:54:15.061509+0000","last_became_peered":"2023-10-14T16:54:15.061509+0000","last_unstale":"2023-10-14T16:54:16.037016+0000","last_undegraded":"2023-10-14T16:54:16.037016+0000","last_fullsized":"2023-10-14T16:54:16.037016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:05.604809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447927+0000","last_change":"2023-10-14T16:54:15.051176+0000","last_active":"2023-10-14T16:54:16.447927+0000","last_peered":"2023-10-14T16:54:16.447927+0000","last_clean":"2023-10-14T16:54:16.447927+0000","last_became_active":"2023-10-14T16:54:15.050889+0000","last_became_peered":"2023-10-14T16:54:15.050889+0000","last_unstale":"2023-10-14T16:54:16.447927+0000","last_undegraded":"2023-10-14T16:54:16.447927+0000","last_fullsized":"2023-10-14T16:54:16.447927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:02.493144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497293+0000","last_change":"2023-10-14T16:54:15.063187+0000","last_active":"2023-10-14T16:54:17.497293+0000","last_peered":"2023-10-14T16:54:17.497293+0000","last_clean":"2023-10-14T16:54:17.497293+0000","last_became_active":"2023-10-14T16:54:15.062799+0000","last_became_peered":"2023-10-14T16:54:15.062799+0000","last_unstale":"2023-10-14T16:54:17.497293+0000","last_undegraded":"2023-10-14T16:54:17.497293+0000","last_fullsized":"2023-10-14T16:54:17.497293+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:03.980605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036941+0000","last_change":"2023-10-14T16:54:15.064052+0000","last_active":"2023-10-14T16:54:16.036941+0000","last_peered":"2023-10-14T16:54:16.036941+0000","last_clean":"2023-10-14T16:54:16.036941+0000","last_became_active":"2023-10-14T16:54:15.063764+0000","last_became_peered":"2023-10-14T16:54:15.063764+0000","last_unstale":"2023-10-14T16:54:16.036941+0000","last_undegraded":"2023-10-14T16:54:16.036941+0000","last_fullsized":"2023-10-14T16:54:16.036941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:28:58.879612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040375+0000","last_change":"2023-10-14T16:54:15.051894+0000","last_active":"2023-10-14T16:54:16.040375+0000","last_peered":"2023-10-14T16:54:16.040375+0000","last_clean":"2023-10-14T16:54:16.040375+0000","last_became_active":"2023-10-14T16:54:15.050249+0000","last_became_peered":"2023-10-14T16:54:15.050249+0000","last_unstale":"2023-10-14T16:54:16.040375+0000","last_undegraded":"2023-10-14T16:54:16.040375+0000","last_fullsized":"2023-10-14T16:54:16.040375+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:30.616445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041263+0000","last_change":"2023-10-14T16:54:15.049687+0000","last_active":"2023-10-14T16:54:16.041263+0000","last_peered":"2023-10-14T16:54:16.041263+0000","last_clean":"2023-10-14T16:54:16.041263+0000","last_became_active":"2023-10-14T16:54:15.049398+0000","last_became_peered":"2023-10-14T16:54:15.049398+0000","last_unstale":"2023-10-14T16:54:16.041263+0000","last_undegraded":"2023-10-14T16:54:16.041263+0000","last_fullsized":"2023-10-14T16:54:16.041263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:22.844113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496069+0000","last_change":"2023-10-14T16:54:15.061176+0000","last_active":"2023-10-14T16:54:17.496069+0000","last_peered":"2023-10-14T16:54:17.496069+0000","last_clean":"2023-10-14T16:54:17.496069+0000","last_became_active":"2023-10-14T16:54:15.060953+0000","last_became_peered":"2023-10-14T16:54:15.060953+0000","last_unstale":"2023-10-14T16:54:17.496069+0000","last_undegraded":"2023-10-14T16:54:17.496069+0000","last_fullsized":"2023-10-14T16:54:17.496069+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.265449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497733+0000","last_change":"2023-10-14T16:54:15.064214+0000","last_active":"2023-10-14T16:54:17.497733+0000","last_peered":"2023-10-14T16:54:17.497733+0000","last_clean":"2023-10-14T16:54:17.497733+0000","last_became_active":"2023-10-14T16:54:15.063062+0000","last_became_peered":"2023-10-14T16:54:15.063062+0000","last_unstale":"2023-10-14T16:54:17.497733+0000","last_undegraded":"2023-10-14T16:54:17.497733+0000","last_fullsized":"2023-10-14T16:54:17.497733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:31.656932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497838+0000","last_change":"2023-10-14T16:54:15.058689+0000","last_active":"2023-10-14T16:54:17.497838+0000","last_peered":"2023-10-14T16:54:17.497838+0000","last_clean":"2023-10-14T16:54:17.497838+0000","last_became_active":"2023-10-14T16:54:15.058335+0000","last_became_peered":"2023-10-14T16:54:15.058335+0000","last_unstale":"2023-10-14T16:54:17.497838+0000","last_undegraded":"2023-10-14T16:54:17.497838+0000","last_fullsized":"2023-10-14T16:54:17.497838+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:02.180000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497427+0000","last_change":"2023-10-14T16:54:15.056497+0000","last_active":"2023-10-14T16:54:17.497427+0000","last_peered":"2023-10-14T16:54:17.497427+0000","last_clean":"2023-10-14T16:54:17.497427+0000","last_became_active":"2023-10-14T16:54:15.056220+0000","last_became_peered":"2023-10-14T16:54:15.056220+0000","last_unstale":"2023-10-14T16:54:17.497427+0000","last_undegraded":"2023-10-14T16:54:17.497427+0000","last_fullsized":"2023-10-14T16:54:17.497427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:57.184914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036941+0000","last_change":"2023-10-14T16:54:15.044431+0000","last_active":"2023-10-14T16:54:16.036941+0000","last_peered":"2023-10-14T16:54:16.036941+0000","last_clean":"2023-10-14T16:54:16.036941+0000","last_became_active":"2023-10-14T16:54:15.043351+0000","last_became_peered":"2023-10-14T16:54:15.043351+0000","last_unstale":"2023-10-14T16:54:16.036941+0000","last_undegraded":"2023-10-14T16:54:16.036941+0000","last_fullsized":"2023-10-14T16:54:16.036941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:10:47.236262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190098+0000","last_change":"2023-10-14T16:54:15.057376+0000","last_active":"2023-10-14T16:54:16.190098+0000","last_peered":"2023-10-14T16:54:16.190098+0000","last_clean":"2023-10-14T16:54:16.190098+0000","last_became_active":"2023-10-14T16:54:15.056769+0000","last_became_peered":"2023-10-14T16:54:15.056769+0000","last_unstale":"2023-10-14T16:54:16.190098+0000","last_undegraded":"2023-10-14T16:54:16.190098+0000","last_fullsized":"2023-10-14T16:54:16.190098+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:34:46.870634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496582+0000","last_change":"2023-10-14T16:54:15.063500+0000","last_active":"2023-10-14T16:54:17.496582+0000","last_peered":"2023-10-14T16:54:17.496582+0000","last_clean":"2023-10-14T16:54:17.496582+0000","last_became_active":"2023-10-14T16:54:15.063284+0000","last_became_peered":"2023-10-14T16:54:15.063284+0000","last_unstale":"2023-10-14T16:54:17.496582+0000","last_undegraded":"2023-10-14T16:54:17.496582+0000","last_fullsized":"2023-10-14T16:54:17.496582+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:25:21.653999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040268+0000","last_change":"2023-10-14T16:54:15.054006+0000","last_active":"2023-10-14T16:54:16.040268+0000","last_peered":"2023-10-14T16:54:16.040268+0000","last_clean":"2023-10-14T16:54:16.040268+0000","last_became_active":"2023-10-14T16:54:15.053716+0000","last_became_peered":"2023-10-14T16:54:15.053716+0000","last_unstale":"2023-10-14T16:54:16.040268+0000","last_undegraded":"2023-10-14T16:54:16.040268+0000","last_fullsized":"2023-10-14T16:54:16.040268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:27.771769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038021+0000","last_change":"2023-10-14T16:54:15.055763+0000","last_active":"2023-10-14T16:54:16.038021+0000","last_peered":"2023-10-14T16:54:16.038021+0000","last_clean":"2023-10-14T16:54:16.038021+0000","last_became_active":"2023-10-14T16:54:15.055487+0000","last_became_peered":"2023-10-14T16:54:15.055487+0000","last_unstale":"2023-10-14T16:54:16.038021+0000","last_undegraded":"2023-10-14T16:54:16.038021+0000","last_fullsized":"2023-10-14T16:54:16.038021+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:19.439016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447497+0000","last_change":"2023-10-14T16:54:15.053078+0000","last_active":"2023-10-14T16:54:16.447497+0000","last_peered":"2023-10-14T16:54:16.447497+0000","last_clean":"2023-10-14T16:54:16.447497+0000","last_became_active":"2023-10-14T16:54:15.052754+0000","last_became_peered":"2023-10-14T16:54:15.052754+0000","last_unstale":"2023-10-14T16:54:16.447497+0000","last_undegraded":"2023-10-14T16:54:16.447497+0000","last_fullsized":"2023-10-14T16:54:16.447497+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:16.161287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447090+0000","last_change":"2023-10-14T16:54:15.062654+0000","last_active":"2023-10-14T16:54:16.447090+0000","last_peered":"2023-10-14T16:54:16.447090+0000","last_clean":"2023-10-14T16:54:16.447090+0000","last_became_active":"2023-10-14T16:54:15.062346+0000","last_became_peered":"2023-10-14T16:54:15.062346+0000","last_unstale":"2023-10-14T16:54:16.447090+0000","last_undegraded":"2023-10-14T16:54:16.447090+0000","last_fullsized":"2023-10-14T16:54:16.447090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:07.343276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039871+0000","last_change":"2023-10-14T16:54:15.063487+0000","last_active":"2023-10-14T16:54:16.039871+0000","last_peered":"2023-10-14T16:54:16.039871+0000","last_clean":"2023-10-14T16:54:16.039871+0000","last_became_active":"2023-10-14T16:54:15.063164+0000","last_became_peered":"2023-10-14T16:54:15.063164+0000","last_unstale":"2023-10-14T16:54:16.039871+0000","last_undegraded":"2023-10-14T16:54:16.039871+0000","last_fullsized":"2023-10-14T16:54:16.039871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:31.656932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190404+0000","last_change":"2023-10-14T16:54:15.062679+0000","last_active":"2023-10-14T16:54:16.190404+0000","last_peered":"2023-10-14T16:54:16.190404+0000","last_clean":"2023-10-14T16:54:16.190404+0000","last_became_active":"2023-10-14T16:54:15.062400+0000","last_became_peered":"2023-10-14T16:54:15.062400+0000","last_unstale":"2023-10-14T16:54:16.190404+0000","last_undegraded":"2023-10-14T16:54:16.190404+0000","last_fullsized":"2023-10-14T16:54:16.190404+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:39:11.808051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040664+0000","last_change":"2023-10-14T16:54:15.057954+0000","last_active":"2023-10-14T16:54:16.040664+0000","last_peered":"2023-10-14T16:54:16.040664+0000","last_clean":"2023-10-14T16:54:16.040664+0000","last_became_active":"2023-10-14T16:54:15.057680+0000","last_became_peered":"2023-10-14T16:54:15.057680+0000","last_unstale":"2023-10-14T16:54:16.040664+0000","last_undegraded":"2023-10-14T16:54:16.040664+0000","last_fullsized":"2023-10-14T16:54:16.040664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:31.972755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496832+0000","last_change":"2023-10-14T16:54:15.060209+0000","last_active":"2023-10-14T16:54:17.496832+0000","last_peered":"2023-10-14T16:54:17.496832+0000","last_clean":"2023-10-14T16:54:17.496832+0000","last_became_active":"2023-10-14T16:54:15.059731+0000","last_became_peered":"2023-10-14T16:54:15.059731+0000","last_unstale":"2023-10-14T16:54:17.496832+0000","last_undegraded":"2023-10-14T16:54:17.496832+0000","last_fullsized":"2023-10-14T16:54:17.496832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:22:00.401204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038210+0000","last_change":"2023-10-14T16:54:15.051167+0000","last_active":"2023-10-14T16:54:16.038210+0000","last_peered":"2023-10-14T16:54:16.038210+0000","last_clean":"2023-10-14T16:54:16.038210+0000","last_became_active":"2023-10-14T16:54:15.050859+0000","last_became_peered":"2023-10-14T16:54:15.050859+0000","last_unstale":"2023-10-14T16:54:16.038210+0000","last_undegraded":"2023-10-14T16:54:16.038210+0000","last_fullsized":"2023-10-14T16:54:16.038210+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:25:21.653999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038013+0000","last_change":"2023-10-14T16:54:15.050903+0000","last_active":"2023-10-14T16:54:16.038013+0000","last_peered":"2023-10-14T16:54:16.038013+0000","last_clean":"2023-10-14T16:54:16.038013+0000","last_became_active":"2023-10-14T16:54:15.050366+0000","last_became_peered":"2023-10-14T16:54:15.050366+0000","last_unstale":"2023-10-14T16:54:16.038013+0000","last_undegraded":"2023-10-14T16:54:16.038013+0000","last_fullsized":"2023-10-14T16:54:16.038013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:29.103133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038702+0000","last_change":"2023-10-14T16:54:15.060414+0000","last_active":"2023-10-14T16:54:16.038702+0000","last_peered":"2023-10-14T16:54:16.038702+0000","last_clean":"2023-10-14T16:54:16.038702+0000","last_became_active":"2023-10-14T16:54:15.060129+0000","last_became_peered":"2023-10-14T16:54:15.060129+0000","last_unstale":"2023-10-14T16:54:16.038702+0000","last_undegraded":"2023-10-14T16:54:16.038702+0000","last_fullsized":"2023-10-14T16:54:16.038702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:22:00.401204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190301+0000","last_change":"2023-10-14T16:54:15.058846+0000","last_active":"2023-10-14T16:54:16.190301+0000","last_peered":"2023-10-14T16:54:16.190301+0000","last_clean":"2023-10-14T16:54:16.190301+0000","last_became_active":"2023-10-14T16:54:15.058578+0000","last_became_peered":"2023-10-14T16:54:15.058578+0000","last_unstale":"2023-10-14T16:54:16.190301+0000","last_undegraded":"2023-10-14T16:54:16.190301+0000","last_fullsized":"2023-10-14T16:54:16.190301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:24:08.864688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447502+0000","last_change":"2023-10-14T16:54:15.053692+0000","last_active":"2023-10-14T16:54:16.447502+0000","last_peered":"2023-10-14T16:54:16.447502+0000","last_clean":"2023-10-14T16:54:16.447502+0000","last_became_active":"2023-10-14T16:54:15.053451+0000","last_became_peered":"2023-10-14T16:54:15.053451+0000","last_unstale":"2023-10-14T16:54:16.447502+0000","last_undegraded":"2023-10-14T16:54:16.447502+0000","last_fullsized":"2023-10-14T16:54:16.447502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:20.110552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037431+0000","last_change":"2023-10-14T16:54:15.044439+0000","last_active":"2023-10-14T16:54:16.037431+0000","last_peered":"2023-10-14T16:54:16.037431+0000","last_clean":"2023-10-14T16:54:16.037431+0000","last_became_active":"2023-10-14T16:54:15.043944+0000","last_became_peered":"2023-10-14T16:54:15.043944+0000","last_unstale":"2023-10-14T16:54:16.037431+0000","last_undegraded":"2023-10-14T16:54:16.037431+0000","last_fullsized":"2023-10-14T16:54:16.037431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:57:52.855997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040222+0000","last_change":"2023-10-14T16:54:15.054146+0000","last_active":"2023-10-14T16:54:16.040222+0000","last_peered":"2023-10-14T16:54:16.040222+0000","last_clean":"2023-10-14T16:54:16.040222+0000","last_became_active":"2023-10-14T16:54:15.053765+0000","last_became_peered":"2023-10-14T16:54:15.053765+0000","last_unstale":"2023-10-14T16:54:16.040222+0000","last_undegraded":"2023-10-14T16:54:16.040222+0000","last_fullsized":"2023-10-14T16:54:16.040222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:02.180000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446967+0000","last_change":"2023-10-14T16:54:15.056344+0000","last_active":"2023-10-14T16:54:16.446967+0000","last_peered":"2023-10-14T16:54:16.446967+0000","last_clean":"2023-10-14T16:54:16.446967+0000","last_became_active":"2023-10-14T16:54:15.055937+0000","last_became_peered":"2023-10-14T16:54:15.055937+0000","last_unstale":"2023-10-14T16:54:16.446967+0000","last_undegraded":"2023-10-14T16:54:16.446967+0000","last_fullsized":"2023-10-14T16:54:16.446967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:08:28.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040387+0000","last_change":"2023-10-14T16:54:15.052013+0000","last_active":"2023-10-14T16:54:16.040387+0000","last_peered":"2023-10-14T16:54:16.040387+0000","last_clean":"2023-10-14T16:54:16.040387+0000","last_became_active":"2023-10-14T16:54:15.051281+0000","last_became_peered":"2023-10-14T16:54:15.051281+0000","last_unstale":"2023-10-14T16:54:16.040387+0000","last_undegraded":"2023-10-14T16:54:16.040387+0000","last_fullsized":"2023-10-14T16:54:16.040387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:24.048725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.189550+0000","last_change":"2023-10-14T16:54:15.049843+0000","last_active":"2023-10-14T16:54:16.189550+0000","last_peered":"2023-10-14T16:54:16.189550+0000","last_clean":"2023-10-14T16:54:16.189550+0000","last_became_active":"2023-10-14T16:54:15.049480+0000","last_became_peered":"2023-10-14T16:54:15.049480+0000","last_unstale":"2023-10-14T16:54:16.189550+0000","last_undegraded":"2023-10-14T16:54:16.189550+0000","last_fullsized":"2023-10-14T16:54:16.189550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:07:26.170550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.064544+0000","last_change":"2023-10-14T16:54:15.062451+0000","last_active":"2023-10-14T16:54:15.064544+0000","last_peered":"2023-10-14T16:54:15.064544+0000","last_clean":"2023-10-14T16:54:15.064544+0000","last_became_active":"2023-10-14T16:54:15.062162+0000","last_became_peered":"2023-10-14T16:54:15.062162+0000","last_unstale":"2023-10-14T16:54:15.064544+0000","last_undegraded":"2023-10-14T16:54:15.064544+0000","last_fullsized":"2023-10-14T16:54:15.064544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:31.855950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"62'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041302+0000","last_change":"2023-10-14T16:54:14.532761+0000","last_active":"2023-10-14T16:54:16.041302+0000","last_peered":"2023-10-14T16:54:16.041302+0000","last_clean":"2023-10-14T16:54:16.041302+0000","last_became_active":"2023-10-14T16:54:14.532557+0000","last_became_peered":"2023-10-14T16:54:14.532557+0000","last_unstale":"2023-10-14T16:54:16.041302+0000","last_undegraded":"2023-10-14T16:54:16.041302+0000","last_fullsized":"2023-10-14T16:54:16.041302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:03.644829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039111+0000","last_change":"2023-10-14T16:54:15.052455+0000","last_active":"2023-10-14T16:54:16.039111+0000","last_peered":"2023-10-14T16:54:16.039111+0000","last_clean":"2023-10-14T16:54:16.039111+0000","last_became_active":"2023-10-14T16:54:15.052204+0000","last_became_peered":"2023-10-14T16:54:15.052204+0000","last_unstale":"2023-10-14T16:54:16.039111+0000","last_undegraded":"2023-10-14T16:54:16.039111+0000","last_fullsized":"2023-10-14T16:54:16.039111+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:33.172706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447563+0000","last_change":"2023-10-14T16:54:14.527661+0000","last_active":"2023-10-14T16:54:16.447563+0000","last_peered":"2023-10-14T16:54:16.447563+0000","last_clean":"2023-10-14T16:54:16.447563+0000","last_became_active":"2023-10-14T16:54:14.526737+0000","last_became_peered":"2023-10-14T16:54:14.526737+0000","last_unstale":"2023-10-14T16:54:16.447563+0000","last_undegraded":"2023-10-14T16:54:16.447563+0000","last_fullsized":"2023-10-14T16:54:16.447563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:49.014304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037366+0000","last_change":"2023-10-14T16:54:15.057173+0000","last_active":"2023-10-14T16:54:16.037366+0000","last_peered":"2023-10-14T16:54:16.037366+0000","last_clean":"2023-10-14T16:54:16.037366+0000","last_became_active":"2023-10-14T16:54:15.056828+0000","last_became_peered":"2023-10-14T16:54:15.056828+0000","last_unstale":"2023-10-14T16:54:16.037366+0000","last_undegraded":"2023-10-14T16:54:16.037366+0000","last_fullsized":"2023-10-14T16:54:16.037366+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:43:00.550103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.560241+0000","last_change":"2023-10-14T16:54:14.530229+0000","last_active":"2023-10-14T16:54:15.560241+0000","last_peered":"2023-10-14T16:54:15.560241+0000","last_clean":"2023-10-14T16:54:15.560241+0000","last_became_active":"2023-10-14T16:54:14.528566+0000","last_became_peered":"2023-10-14T16:54:14.528566+0000","last_unstale":"2023-10-14T16:54:15.560241+0000","last_undegraded":"2023-10-14T16:54:15.560241+0000","last_fullsized":"2023-10-14T16:54:15.560241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:51.926885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.063981+0000","last_change":"2023-10-14T16:54:15.060562+0000","last_active":"2023-10-14T16:54:15.063981+0000","last_peered":"2023-10-14T16:54:15.063981+0000","last_clean":"2023-10-14T16:54:15.063981+0000","last_became_active":"2023-10-14T16:54:15.060296+0000","last_became_peered":"2023-10-14T16:54:15.060296+0000","last_unstale":"2023-10-14T16:54:15.063981+0000","last_undegraded":"2023-10-14T16:54:15.063981+0000","last_fullsized":"2023-10-14T16:54:15.063981+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:54.303157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041938+0000","last_change":"2023-10-14T16:54:14.540360+0000","last_active":"2023-10-14T16:54:16.041938+0000","last_peered":"2023-10-14T16:54:16.041938+0000","last_clean":"2023-10-14T16:54:16.041938+0000","last_became_active":"2023-10-14T16:54:14.540032+0000","last_became_peered":"2023-10-14T16:54:14.540032+0000","last_unstale":"2023-10-14T16:54:16.041938+0000","last_undegraded":"2023-10-14T16:54:16.041938+0000","last_fullsized":"2023-10-14T16:54:16.041938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:59.533841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446965+0000","last_change":"2023-10-14T16:54:15.058028+0000","last_active":"2023-10-14T16:54:16.446965+0000","last_peered":"2023-10-14T16:54:16.446965+0000","last_clean":"2023-10-14T16:54:16.446965+0000","last_became_active":"2023-10-14T16:54:15.057733+0000","last_became_peered":"2023-10-14T16:54:15.057733+0000","last_unstale":"2023-10-14T16:54:16.446965+0000","last_undegraded":"2023-10-14T16:54:16.446965+0000","last_fullsized":"2023-10-14T16:54:16.446965+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:13.482060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040697+0000","last_change":"2023-10-14T16:54:14.536339+0000","last_active":"2023-10-14T16:54:16.040697+0000","last_peered":"2023-10-14T16:54:16.040697+0000","last_clean":"2023-10-14T16:54:16.040697+0000","last_became_active":"2023-10-14T16:54:14.536064+0000","last_became_peered":"2023-10-14T16:54:14.536064+0000","last_unstale":"2023-10-14T16:54:16.040697+0000","last_undegraded":"2023-10-14T16:54:16.040697+0000","last_fullsized":"2023-10-14T16:54:16.040697+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:04.588548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497096+0000","last_change":"2023-10-14T16:54:15.065694+0000","last_active":"2023-10-14T16:54:17.497096+0000","last_peered":"2023-10-14T16:54:17.497096+0000","last_clean":"2023-10-14T16:54:17.497096+0000","last_became_active":"2023-10-14T16:54:15.061582+0000","last_became_peered":"2023-10-14T16:54:15.061582+0000","last_unstale":"2023-10-14T16:54:17.497096+0000","last_undegraded":"2023-10-14T16:54:17.497096+0000","last_fullsized":"2023-10-14T16:54:17.497096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:28:23.730093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447311+0000","last_change":"2023-10-14T16:54:14.532574+0000","last_active":"2023-10-14T16:54:16.447311+0000","last_peered":"2023-10-14T16:54:16.447311+0000","last_clean":"2023-10-14T16:54:16.447311+0000","last_became_active":"2023-10-14T16:54:14.530216+0000","last_became_peered":"2023-10-14T16:54:14.530216+0000","last_unstale":"2023-10-14T16:54:16.447311+0000","last_undegraded":"2023-10-14T16:54:16.447311+0000","last_fullsized":"2023-10-14T16:54:16.447311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:15:09.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496373+0000","last_change":"2023-10-14T16:54:15.057528+0000","last_active":"2023-10-14T16:54:17.496373+0000","last_peered":"2023-10-14T16:54:17.496373+0000","last_clean":"2023-10-14T16:54:17.496373+0000","last_became_active":"2023-10-14T16:54:15.057291+0000","last_became_peered":"2023-10-14T16:54:15.057291+0000","last_unstale":"2023-10-14T16:54:17.496373+0000","last_undegraded":"2023-10-14T16:54:17.496373+0000","last_fullsized":"2023-10-14T16:54:17.496373+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:19.439016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036660+0000","last_change":"2023-10-14T16:54:14.529766+0000","last_active":"2023-10-14T16:54:16.036660+0000","last_peered":"2023-10-14T16:54:16.036660+0000","last_clean":"2023-10-14T16:54:16.036660+0000","last_became_active":"2023-10-14T16:54:14.529396+0000","last_became_peered":"2023-10-14T16:54:14.529396+0000","last_unstale":"2023-10-14T16:54:16.036660+0000","last_undegraded":"2023-10-14T16:54:16.036660+0000","last_fullsized":"2023-10-14T16:54:16.036660+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:18:45.997572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037509+0000","last_change":"2023-10-14T16:54:15.047954+0000","last_active":"2023-10-14T16:54:16.037509+0000","last_peered":"2023-10-14T16:54:16.037509+0000","last_clean":"2023-10-14T16:54:16.037509+0000","last_became_active":"2023-10-14T16:54:15.047163+0000","last_became_peered":"2023-10-14T16:54:15.047163+0000","last_unstale":"2023-10-14T16:54:16.037509+0000","last_undegraded":"2023-10-14T16:54:16.037509+0000","last_fullsized":"2023-10-14T16:54:16.037509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.265449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496344+0000","last_change":"2023-10-14T16:54:14.530038+0000","last_active":"2023-10-14T16:54:17.496344+0000","last_peered":"2023-10-14T16:54:17.496344+0000","last_clean":"2023-10-14T16:54:17.496344+0000","last_became_active":"2023-10-14T16:54:14.527414+0000","last_became_peered":"2023-10-14T16:54:14.527414+0000","last_unstale":"2023-10-14T16:54:17.496344+0000","last_undegraded":"2023-10-14T16:54:17.496344+0000","last_fullsized":"2023-10-14T16:54:17.496344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:11.383980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447498+0000","last_change":"2023-10-14T16:54:15.062052+0000","last_active":"2023-10-14T16:54:16.447498+0000","last_peered":"2023-10-14T16:54:16.447498+0000","last_clean":"2023-10-14T16:54:16.447498+0000","last_became_active":"2023-10-14T16:54:15.061719+0000","last_became_peered":"2023-10-14T16:54:15.061719+0000","last_unstale":"2023-10-14T16:54:16.447498+0000","last_undegraded":"2023-10-14T16:54:16.447498+0000","last_fullsized":"2023-10-14T16:54:16.447498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:24:05.513503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447719+0000","last_change":"2023-10-14T16:54:14.522023+0000","last_active":"2023-10-14T16:54:16.447719+0000","last_peered":"2023-10-14T16:54:16.447719+0000","last_clean":"2023-10-14T16:54:16.447719+0000","last_became_active":"2023-10-14T16:54:14.521553+0000","last_became_peered":"2023-10-14T16:54:14.521553+0000","last_unstale":"2023-10-14T16:54:16.447719+0000","last_undegraded":"2023-10-14T16:54:16.447719+0000","last_fullsized":"2023-10-14T16:54:16.447719+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:16.152378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042750+0000","last_change":"2023-10-14T16:54:15.048815+0000","last_active":"2023-10-14T16:54:16.042750+0000","last_peered":"2023-10-14T16:54:16.042750+0000","last_clean":"2023-10-14T16:54:16.042750+0000","last_became_active":"2023-10-14T16:54:15.048548+0000","last_became_peered":"2023-10-14T16:54:15.048548+0000","last_unstale":"2023-10-14T16:54:16.042750+0000","last_undegraded":"2023-10-14T16:54:16.042750+0000","last_fullsized":"2023-10-14T16:54:16.042750+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:17:21.807863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497976+0000","last_change":"2023-10-14T16:54:14.532449+0000","last_active":"2023-10-14T16:54:17.497976+0000","last_peered":"2023-10-14T16:54:17.497976+0000","last_clean":"2023-10-14T16:54:17.497976+0000","last_became_active":"2023-10-14T16:54:14.532149+0000","last_became_peered":"2023-10-14T16:54:14.532149+0000","last_unstale":"2023-10-14T16:54:17.497976+0000","last_undegraded":"2023-10-14T16:54:17.497976+0000","last_fullsized":"2023-10-14T16:54:17.497976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:04:58.952522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041001+0000","last_change":"2023-10-14T16:54:15.061083+0000","last_active":"2023-10-14T16:54:16.041001+0000","last_peered":"2023-10-14T16:54:16.041001+0000","last_clean":"2023-10-14T16:54:16.041001+0000","last_became_active":"2023-10-14T16:54:15.060773+0000","last_became_peered":"2023-10-14T16:54:15.060773+0000","last_unstale":"2023-10-14T16:54:16.041001+0000","last_undegraded":"2023-10-14T16:54:16.041001+0000","last_fullsized":"2023-10-14T16:54:16.041001+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"62'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042309+0000","last_change":"2023-10-14T16:54:14.540813+0000","last_active":"2023-10-14T16:54:16.042309+0000","last_peered":"2023-10-14T16:54:16.042309+0000","last_clean":"2023-10-14T16:54:16.042309+0000","last_became_active":"2023-10-14T16:54:14.540503+0000","last_became_peered":"2023-10-14T16:54:14.540503+0000","last_unstale":"2023-10-14T16:54:16.042309+0000","last_undegraded":"2023-10-14T16:54:16.042309+0000","last_fullsized":"2023-10-14T16:54:16.042309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:39:23.483273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041572+0000","last_change":"2023-10-14T16:54:15.054664+0000","last_active":"2023-10-14T16:54:16.041572+0000","last_peered":"2023-10-14T16:54:16.041572+0000","last_clean":"2023-10-14T16:54:16.041572+0000","last_became_active":"2023-10-14T16:54:15.054390+0000","last_became_peered":"2023-10-14T16:54:15.054390+0000","last_unstale":"2023-10-14T16:54:16.041572+0000","last_undegraded":"2023-10-14T16:54:16.041572+0000","last_fullsized":"2023-10-14T16:54:16.041572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:31.972755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"62'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190374+0000","last_change":"2023-10-14T16:54:14.531927+0000","last_active":"2023-10-14T16:54:16.190374+0000","last_peered":"2023-10-14T16:54:16.190374+0000","last_clean":"2023-10-14T16:54:16.190374+0000","last_became_active":"2023-10-14T16:54:14.531623+0000","last_became_peered":"2023-10-14T16:54:14.531623+0000","last_unstale":"2023-10-14T16:54:16.190374+0000","last_undegraded":"2023-10-14T16:54:16.190374+0000","last_fullsized":"2023-10-14T16:54:16.190374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:26.241831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039295+0000","last_change":"2023-10-14T16:54:15.062501+0000","last_active":"2023-10-14T16:54:16.039295+0000","last_peered":"2023-10-14T16:54:16.039295+0000","last_clean":"2023-10-14T16:54:16.039295+0000","last_became_active":"2023-10-14T16:54:15.062231+0000","last_became_peered":"2023-10-14T16:54:15.062231+0000","last_unstale":"2023-10-14T16:54:16.039295+0000","last_undegraded":"2023-10-14T16:54:16.039295+0000","last_fullsized":"2023-10-14T16:54:16.039295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:03.980605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040867+0000","last_change":"2023-10-14T16:54:14.537549+0000","last_active":"2023-10-14T16:54:16.040867+0000","last_peered":"2023-10-14T16:54:16.040867+0000","last_clean":"2023-10-14T16:54:16.040867+0000","last_became_active":"2023-10-14T16:54:14.536132+0000","last_became_peered":"2023-10-14T16:54:14.536132+0000","last_unstale":"2023-10-14T16:54:16.040867+0000","last_undegraded":"2023-10-14T16:54:16.040867+0000","last_fullsized":"2023-10-14T16:54:16.040867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:57:51.839736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497240+0000","last_change":"2023-10-14T16:54:15.055199+0000","last_active":"2023-10-14T16:54:17.497240+0000","last_peered":"2023-10-14T16:54:17.497240+0000","last_clean":"2023-10-14T16:54:17.497240+0000","last_became_active":"2023-10-14T16:54:15.054973+0000","last_became_peered":"2023-10-14T16:54:15.054973+0000","last_unstale":"2023-10-14T16:54:17.497240+0000","last_undegraded":"2023-10-14T16:54:17.497240+0000","last_fullsized":"2023-10-14T16:54:17.497240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:33.172706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041615+0000","last_change":"2023-10-14T16:54:14.540108+0000","last_active":"2023-10-14T16:54:16.041615+0000","last_peered":"2023-10-14T16:54:16.041615+0000","last_clean":"2023-10-14T16:54:16.041615+0000","last_became_active":"2023-10-14T16:54:14.539816+0000","last_became_peered":"2023-10-14T16:54:14.539816+0000","last_unstale":"2023-10-14T16:54:16.041615+0000","last_undegraded":"2023-10-14T16:54:16.041615+0000","last_fullsized":"2023-10-14T16:54:16.041615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:19.435571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040153+0000","last_change":"2023-10-14T16:54:15.054060+0000","last_active":"2023-10-14T16:54:16.040153+0000","last_peered":"2023-10-14T16:54:16.040153+0000","last_clean":"2023-10-14T16:54:16.040153+0000","last_became_active":"2023-10-14T16:54:15.053786+0000","last_became_peered":"2023-10-14T16:54:15.053786+0000","last_unstale":"2023-10-14T16:54:16.040153+0000","last_undegraded":"2023-10-14T16:54:16.040153+0000","last_fullsized":"2023-10-14T16:54:16.040153+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:22.568851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040390+0000","last_change":"2023-10-14T16:54:14.534505+0000","last_active":"2023-10-14T16:54:16.040390+0000","last_peered":"2023-10-14T16:54:16.040390+0000","last_clean":"2023-10-14T16:54:16.040390+0000","last_became_active":"2023-10-14T16:54:14.534188+0000","last_became_peered":"2023-10-14T16:54:14.534188+0000","last_unstale":"2023-10-14T16:54:16.040390+0000","last_undegraded":"2023-10-14T16:54:16.040390+0000","last_fullsized":"2023-10-14T16:54:16.040390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:59.533841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041391+0000","last_change":"2023-10-14T16:54:15.046830+0000","last_active":"2023-10-14T16:54:16.041391+0000","last_peered":"2023-10-14T16:54:16.041391+0000","last_clean":"2023-10-14T16:54:16.041391+0000","last_became_active":"2023-10-14T16:54:15.046529+0000","last_became_peered":"2023-10-14T16:54:15.046529+0000","last_unstale":"2023-10-14T16:54:16.041391+0000","last_undegraded":"2023-10-14T16:54:16.041391+0000","last_fullsized":"2023-10-14T16:54:16.041391+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:02:15.718591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"62'8","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496602+0000","last_change":"2023-10-14T16:54:14.535595+0000","last_active":"2023-10-14T16:54:17.496602+0000","last_peered":"2023-10-14T16:54:17.496602+0000","last_clean":"2023-10-14T16:54:17.496602+0000","last_became_active":"2023-10-14T16:54:14.535287+0000","last_became_peered":"2023-10-14T16:54:14.535287+0000","last_unstale":"2023-10-14T16:54:17.496602+0000","last_undegraded":"2023-10-14T16:54:17.496602+0000","last_fullsized":"2023-10-14T16:54:17.496602+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:16.557551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190603+0000","last_change":"2023-10-14T16:54:15.041597+0000","last_active":"2023-10-14T16:54:16.190603+0000","last_peered":"2023-10-14T16:54:16.190603+0000","last_clean":"2023-10-14T16:54:16.190603+0000","last_became_active":"2023-10-14T16:54:15.041323+0000","last_became_peered":"2023-10-14T16:54:15.041323+0000","last_unstale":"2023-10-14T16:54:16.190603+0000","last_undegraded":"2023-10-14T16:54:16.190603+0000","last_fullsized":"2023-10-14T16:54:16.190603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:04.036749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190483+0000","last_change":"2023-10-14T16:54:14.537867+0000","last_active":"2023-10-14T16:54:16.190483+0000","last_peered":"2023-10-14T16:54:16.190483+0000","last_clean":"2023-10-14T16:54:16.190483+0000","last_became_active":"2023-10-14T16:54:14.537571+0000","last_became_peered":"2023-10-14T16:54:14.537571+0000","last_unstale":"2023-10-14T16:54:16.190483+0000","last_undegraded":"2023-10-14T16:54:16.190483+0000","last_fullsized":"2023-10-14T16:54:16.190483+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:10.454177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497908+0000","last_change":"2023-10-14T16:54:15.065740+0000","last_active":"2023-10-14T16:54:17.497908+0000","last_peered":"2023-10-14T16:54:17.497908+0000","last_clean":"2023-10-14T16:54:17.497908+0000","last_became_active":"2023-10-14T16:54:15.062742+0000","last_became_peered":"2023-10-14T16:54:15.062742+0000","last_unstale":"2023-10-14T16:54:17.497908+0000","last_undegraded":"2023-10-14T16:54:17.497908+0000","last_fullsized":"2023-10-14T16:54:17.497908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:51:07.363554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447408+0000","last_change":"2023-10-14T16:54:14.522092+0000","last_active":"2023-10-14T16:54:16.447408+0000","last_peered":"2023-10-14T16:54:16.447408+0000","last_clean":"2023-10-14T16:54:16.447408+0000","last_became_active":"2023-10-14T16:54:14.521681+0000","last_became_peered":"2023-10-14T16:54:14.521681+0000","last_unstale":"2023-10-14T16:54:16.447408+0000","last_undegraded":"2023-10-14T16:54:16.447408+0000","last_fullsized":"2023-10-14T16:54:16.447408+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:09.280098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039108+0000","last_change":"2023-10-14T16:54:15.056759+0000","last_active":"2023-10-14T16:54:16.039108+0000","last_peered":"2023-10-14T16:54:16.039108+0000","last_clean":"2023-10-14T16:54:16.039108+0000","last_became_active":"2023-10-14T16:54:15.056498+0000","last_became_peered":"2023-10-14T16:54:15.056498+0000","last_unstale":"2023-10-14T16:54:16.039108+0000","last_undegraded":"2023-10-14T16:54:16.039108+0000","last_fullsized":"2023-10-14T16:54:16.039108+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:28:23.730093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037292+0000","last_change":"2023-10-14T16:54:14.531987+0000","last_active":"2023-10-14T16:54:16.037292+0000","last_peered":"2023-10-14T16:54:16.037292+0000","last_clean":"2023-10-14T16:54:16.037292+0000","last_became_active":"2023-10-14T16:54:14.531642+0000","last_became_peered":"2023-10-14T16:54:14.531642+0000","last_unstale":"2023-10-14T16:54:16.037292+0000","last_undegraded":"2023-10-14T16:54:16.037292+0000","last_fullsized":"2023-10-14T16:54:16.037292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:14.574599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039499+0000","last_change":"2023-10-14T16:54:15.054799+0000","last_active":"2023-10-14T16:54:16.039499+0000","last_peered":"2023-10-14T16:54:16.039499+0000","last_clean":"2023-10-14T16:54:16.039499+0000","last_became_active":"2023-10-14T16:54:15.053765+0000","last_became_peered":"2023-10-14T16:54:15.053765+0000","last_unstale":"2023-10-14T16:54:16.039499+0000","last_undegraded":"2023-10-14T16:54:16.039499+0000","last_fullsized":"2023-10-14T16:54:16.039499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:57.184914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041026+0000","last_change":"2023-10-14T16:54:14.533253+0000","last_active":"2023-10-14T16:54:16.041026+0000","last_peered":"2023-10-14T16:54:16.041026+0000","last_clean":"2023-10-14T16:54:16.041026+0000","last_became_active":"2023-10-14T16:54:14.531802+0000","last_became_peered":"2023-10-14T16:54:14.531802+0000","last_unstale":"2023-10-14T16:54:16.041026+0000","last_undegraded":"2023-10-14T16:54:16.041026+0000","last_fullsized":"2023-10-14T16:54:16.041026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:52.590896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040624+0000","last_change":"2023-10-14T16:54:15.052111+0000","last_active":"2023-10-14T16:54:16.040624+0000","last_peered":"2023-10-14T16:54:16.040624+0000","last_clean":"2023-10-14T16:54:16.040624+0000","last_became_active":"2023-10-14T16:54:15.051640+0000","last_became_peered":"2023-10-14T16:54:15.051640+0000","last_unstale":"2023-10-14T16:54:16.040624+0000","last_undegraded":"2023-10-14T16:54:16.040624+0000","last_fullsized":"2023-10-14T16:54:16.040624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:02:15.718591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041103+0000","last_change":"2023-10-14T16:54:14.533578+0000","last_active":"2023-10-14T16:54:16.041103+0000","last_peered":"2023-10-14T16:54:16.041103+0000","last_clean":"2023-10-14T16:54:16.041103+0000","last_became_active":"2023-10-14T16:54:14.533302+0000","last_became_peered":"2023-10-14T16:54:14.533302+0000","last_unstale":"2023-10-14T16:54:16.041103+0000","last_undegraded":"2023-10-14T16:54:16.041103+0000","last_fullsized":"2023-10-14T16:54:16.041103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:39:23.483273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.059228+0000","last_change":"2023-10-14T16:54:15.055856+0000","last_active":"2023-10-14T16:54:15.059228+0000","last_peered":"2023-10-14T16:54:15.059228+0000","last_clean":"2023-10-14T16:54:15.059228+0000","last_became_active":"2023-10-14T16:54:15.053506+0000","last_became_peered":"2023-10-14T16:54:15.053506+0000","last_unstale":"2023-10-14T16:54:15.059228+0000","last_undegraded":"2023-10-14T16:54:15.059228+0000","last_fullsized":"2023-10-14T16:54:15.059228+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:33.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036447+0000","last_change":"2023-10-14T16:54:14.534219+0000","last_active":"2023-10-14T16:54:16.036447+0000","last_peered":"2023-10-14T16:54:16.036447+0000","last_clean":"2023-10-14T16:54:16.036447+0000","last_became_active":"2023-10-14T16:54:14.533915+0000","last_became_peered":"2023-10-14T16:54:14.533915+0000","last_unstale":"2023-10-14T16:54:16.036447+0000","last_undegraded":"2023-10-14T16:54:16.036447+0000","last_fullsized":"2023-10-14T16:54:16.036447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:03.309664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.189853+0000","last_change":"2023-10-14T16:54:15.057372+0000","last_active":"2023-10-14T16:54:16.189853+0000","last_peered":"2023-10-14T16:54:16.189853+0000","last_clean":"2023-10-14T16:54:16.189853+0000","last_became_active":"2023-10-14T16:54:15.056733+0000","last_became_peered":"2023-10-14T16:54:15.056733+0000","last_unstale":"2023-10-14T16:54:16.189853+0000","last_undegraded":"2023-10-14T16:54:16.189853+0000","last_fullsized":"2023-10-14T16:54:16.189853+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:53.103204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496374+0000","last_change":"2023-10-14T16:54:14.532790+0000","last_active":"2023-10-14T16:54:17.496374+0000","last_peered":"2023-10-14T16:54:17.496374+0000","last_clean":"2023-10-14T16:54:17.496374+0000","last_became_active":"2023-10-14T16:54:14.532530+0000","last_became_peered":"2023-10-14T16:54:14.532530+0000","last_unstale":"2023-10-14T16:54:17.496374+0000","last_undegraded":"2023-10-14T16:54:17.496374+0000","last_fullsized":"2023-10-14T16:54:17.496374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:52:24.265637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496434+0000","last_change":"2023-10-14T16:54:15.059647+0000","last_active":"2023-10-14T16:54:17.496434+0000","last_peered":"2023-10-14T16:54:17.496434+0000","last_clean":"2023-10-14T16:54:17.496434+0000","last_became_active":"2023-10-14T16:54:15.059337+0000","last_became_peered":"2023-10-14T16:54:15.059337+0000","last_unstale":"2023-10-14T16:54:17.496434+0000","last_undegraded":"2023-10-14T16:54:17.496434+0000","last_fullsized":"2023-10-14T16:54:17.496434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:29.103133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041667+0000","last_change":"2023-10-14T16:54:14.540165+0000","last_active":"2023-10-14T16:54:16.041667+0000","last_peered":"2023-10-14T16:54:16.041667+0000","last_clean":"2023-10-14T16:54:16.041667+0000","last_became_active":"2023-10-14T16:54:14.539839+0000","last_became_peered":"2023-10-14T16:54:14.539839+0000","last_unstale":"2023-10-14T16:54:16.041667+0000","last_undegraded":"2023-10-14T16:54:16.041667+0000","last_fullsized":"2023-10-14T16:54:16.041667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:17:17.389989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039502+0000","last_change":"2023-10-14T16:54:15.063819+0000","last_active":"2023-10-14T16:54:16.039502+0000","last_peered":"2023-10-14T16:54:16.039502+0000","last_clean":"2023-10-14T16:54:16.039502+0000","last_became_active":"2023-10-14T16:54:15.063498+0000","last_became_peered":"2023-10-14T16:54:15.063498+0000","last_unstale":"2023-10-14T16:54:16.039502+0000","last_undegraded":"2023-10-14T16:54:16.039502+0000","last_fullsized":"2023-10-14T16:54:16.039502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:33.195664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041220+0000","last_change":"2023-10-14T16:54:14.531687+0000","last_active":"2023-10-14T16:54:16.041220+0000","last_peered":"2023-10-14T16:54:16.041220+0000","last_clean":"2023-10-14T16:54:16.041220+0000","last_became_active":"2023-10-14T16:54:14.531351+0000","last_became_peered":"2023-10-14T16:54:14.531351+0000","last_unstale":"2023-10-14T16:54:16.041220+0000","last_undegraded":"2023-10-14T16:54:16.041220+0000","last_fullsized":"2023-10-14T16:54:16.041220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:17:17.389989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497523+0000","last_change":"2023-10-14T16:54:15.056714+0000","last_active":"2023-10-14T16:54:17.497523+0000","last_peered":"2023-10-14T16:54:17.497523+0000","last_clean":"2023-10-14T16:54:17.497523+0000","last_became_active":"2023-10-14T16:54:15.056443+0000","last_became_peered":"2023-10-14T16:54:15.056443+0000","last_unstale":"2023-10-14T16:54:17.497523+0000","last_undegraded":"2023-10-14T16:54:17.497523+0000","last_fullsized":"2023-10-14T16:54:17.497523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:33.195664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042446+0000","last_change":"2023-10-14T16:54:14.541365+0000","last_active":"2023-10-14T16:54:16.042446+0000","last_peered":"2023-10-14T16:54:16.042446+0000","last_clean":"2023-10-14T16:54:16.042446+0000","last_became_active":"2023-10-14T16:54:14.541067+0000","last_became_peered":"2023-10-14T16:54:14.541067+0000","last_unstale":"2023-10-14T16:54:16.042446+0000","last_undegraded":"2023-10-14T16:54:16.042446+0000","last_fullsized":"2023-10-14T16:54:16.042446+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:21.552589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042363+0000","last_change":"2023-10-14T16:54:15.046680+0000","last_active":"2023-10-14T16:54:16.042363+0000","last_peered":"2023-10-14T16:54:16.042363+0000","last_clean":"2023-10-14T16:54:16.042363+0000","last_became_active":"2023-10-14T16:54:15.046389+0000","last_became_peered":"2023-10-14T16:54:15.046389+0000","last_unstale":"2023-10-14T16:54:16.042363+0000","last_undegraded":"2023-10-14T16:54:16.042363+0000","last_fullsized":"2023-10-14T16:54:16.042363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.047903+0000","last_change":"2023-10-14T16:54:14.528607+0000","last_active":"2023-10-14T16:54:15.047903+0000","last_peered":"2023-10-14T16:54:15.047903+0000","last_clean":"2023-10-14T16:54:15.047903+0000","last_became_active":"2023-10-14T16:54:14.528337+0000","last_became_peered":"2023-10-14T16:54:14.528337+0000","last_unstale":"2023-10-14T16:54:15.047903+0000","last_undegraded":"2023-10-14T16:54:15.047903+0000","last_fullsized":"2023-10-14T16:54:15.047903+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:24:37.976867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040518+0000","last_change":"2023-10-14T16:54:15.065200+0000","last_active":"2023-10-14T16:54:16.040518+0000","last_peered":"2023-10-14T16:54:16.040518+0000","last_clean":"2023-10-14T16:54:16.040518+0000","last_became_active":"2023-10-14T16:54:15.064457+0000","last_became_peered":"2023-10-14T16:54:15.064457+0000","last_unstale":"2023-10-14T16:54:16.040518+0000","last_undegraded":"2023-10-14T16:54:16.040518+0000","last_fullsized":"2023-10-14T16:54:16.040518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:20.589977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041879+0000","last_change":"2023-10-14T16:54:14.541134+0000","last_active":"2023-10-14T16:54:16.041879+0000","last_peered":"2023-10-14T16:54:16.041879+0000","last_clean":"2023-10-14T16:54:16.041879+0000","last_became_active":"2023-10-14T16:54:14.540823+0000","last_became_peered":"2023-10-14T16:54:14.540823+0000","last_unstale":"2023-10-14T16:54:16.041879+0000","last_undegraded":"2023-10-14T16:54:16.041879+0000","last_fullsized":"2023-10-14T16:54:16.041879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:26.755508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040559+0000","last_change":"2023-10-14T16:54:15.060432+0000","last_active":"2023-10-14T16:54:16.040559+0000","last_peered":"2023-10-14T16:54:16.040559+0000","last_clean":"2023-10-14T16:54:16.040559+0000","last_became_active":"2023-10-14T16:54:15.059990+0000","last_became_peered":"2023-10-14T16:54:15.059990+0000","last_unstale":"2023-10-14T16:54:16.040559+0000","last_undegraded":"2023-10-14T16:54:16.040559+0000","last_fullsized":"2023-10-14T16:54:16.040559+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:22.844113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447322+0000","last_change":"2023-10-14T16:54:14.528023+0000","last_active":"2023-10-14T16:54:16.447322+0000","last_peered":"2023-10-14T16:54:16.447322+0000","last_clean":"2023-10-14T16:54:16.447322+0000","last_became_active":"2023-10-14T16:54:14.527694+0000","last_became_peered":"2023-10-14T16:54:14.527694+0000","last_unstale":"2023-10-14T16:54:16.447322+0000","last_undegraded":"2023-10-14T16:54:16.447322+0000","last_fullsized":"2023-10-14T16:54:16.447322+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:23:15.354376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036429+0000","last_change":"2023-10-14T16:54:15.046086+0000","last_active":"2023-10-14T16:54:16.036429+0000","last_peered":"2023-10-14T16:54:16.036429+0000","last_clean":"2023-10-14T16:54:16.036429+0000","last_became_active":"2023-10-14T16:54:15.045110+0000","last_became_peered":"2023-10-14T16:54:15.045110+0000","last_unstale":"2023-10-14T16:54:16.036429+0000","last_undegraded":"2023-10-14T16:54:16.036429+0000","last_fullsized":"2023-10-14T16:54:16.036429+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:14.582104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.053701+0000","last_change":"2023-10-14T16:54:14.526389+0000","last_active":"2023-10-14T16:54:15.053701+0000","last_peered":"2023-10-14T16:54:15.053701+0000","last_clean":"2023-10-14T16:54:15.053701+0000","last_became_active":"2023-10-14T16:54:14.526035+0000","last_became_peered":"2023-10-14T16:54:14.526035+0000","last_unstale":"2023-10-14T16:54:15.053701+0000","last_undegraded":"2023-10-14T16:54:15.053701+0000","last_fullsized":"2023-10-14T16:54:15.053701+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:09.280098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446913+0000","last_change":"2023-10-14T16:54:15.057060+0000","last_active":"2023-10-14T16:54:16.446913+0000","last_peered":"2023-10-14T16:54:16.446913+0000","last_clean":"2023-10-14T16:54:16.446913+0000","last_became_active":"2023-10-14T16:54:15.056768+0000","last_became_peered":"2023-10-14T16:54:15.056768+0000","last_unstale":"2023-10-14T16:54:16.446913+0000","last_undegraded":"2023-10-14T16:54:16.446913+0000","last_fullsized":"2023-10-14T16:54:16.446913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:33.350632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496605+0000","last_change":"2023-10-14T16:54:14.534616+0000","last_active":"2023-10-14T16:54:17.496605+0000","last_peered":"2023-10-14T16:54:17.496605+0000","last_clean":"2023-10-14T16:54:17.496605+0000","last_became_active":"2023-10-14T16:54:14.533635+0000","last_became_peered":"2023-10-14T16:54:14.533635+0000","last_unstale":"2023-10-14T16:54:17.496605+0000","last_undegraded":"2023-10-14T16:54:17.496605+0000","last_fullsized":"2023-10-14T16:54:17.496605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:55:08.394535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.498070+0000","last_change":"2023-10-14T16:54:15.058532+0000","last_active":"2023-10-14T16:54:17.498070+0000","last_peered":"2023-10-14T16:54:17.498070+0000","last_clean":"2023-10-14T16:54:17.498070+0000","last_became_active":"2023-10-14T16:54:15.058114+0000","last_became_peered":"2023-10-14T16:54:15.058114+0000","last_unstale":"2023-10-14T16:54:17.498070+0000","last_undegraded":"2023-10-14T16:54:17.498070+0000","last_fullsized":"2023-10-14T16:54:17.498070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:26:43.509435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037292+0000","last_change":"2023-10-14T16:54:14.537565+0000","last_active":"2023-10-14T16:54:16.037292+0000","last_peered":"2023-10-14T16:54:16.037292+0000","last_clean":"2023-10-14T16:54:16.037292+0000","last_became_active":"2023-10-14T16:54:14.537284+0000","last_became_peered":"2023-10-14T16:54:14.537284+0000","last_unstale":"2023-10-14T16:54:16.037292+0000","last_undegraded":"2023-10-14T16:54:16.037292+0000","last_fullsized":"2023-10-14T16:54:16.037292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:59:10.016915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"50'139","reported_seq":286,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.039740+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:54:15.039740+0000","last_peered":"2023-10-14T16:54:15.039740+0000","last_clean":"2023-10-14T16:54:15.039740+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:54:15.039740+0000","last_undegraded":"2023-10-14T16:54:15.039740+0000","last_fullsized":"2023-10-14T16:54:15.039740+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1347584,"data_stored":942111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1779999999999999,"15min":1.147},"min":{"1min":0.84399999999999997,"5min":0.77700000000000002,"15min":0.503},"max":{"1min":1.5840000000000001,"5min":1.897,"15min":1.897},"last":1.365},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.232,"15min":1.2150000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":2.117,"15min":2.262},"last":1.321}]},{"osd":1,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.254,"15min":1.2310000000000001},"min":{"1min":0.91300000000000003,"5min":0.86599999999999999,"15min":0.74399999999999999},"max":{"1min":1.375,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.254},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.258,"15min":1.2270000000000001},"min":{"1min":0.79200000000000004,"5min":0.751,"15min":0.60799999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.0609999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.23,"15min":1.232},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.71999999999999997},"max":{"1min":1.6259999999999999,"5min":1.772,"15min":2.2389999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.329,"15min":1.3160000000000001},"min":{"1min":1.002,"5min":0.93500000000000005,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.1589999999999998,"15min":2.3500000000000001},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4099999999999999,"15min":1.347},"min":{"1min":0.94099999999999995,"5min":0.87,"15min":0.77300000000000002},"max":{"1min":1.8660000000000001,"5min":1.9610000000000001,"15min":2.4159999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4830000000000001,"15min":1.452},"min":{"1min":1.1259999999999999,"5min":1.0840000000000001,"15min":0.83099999999999996},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":2.5819999999999999},"last":1.5169999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.26,"15min":1.2749999999999999},"min":{"1min":0.82999999999999996,"5min":0.76500000000000001,"15min":0.59599999999999997},"max":{"1min":1.8120000000000001,"5min":2.0070000000000001,"15min":2.2959999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.3520000000000001,"15min":1.3160000000000001},"min":{"1min":0.95499999999999996,"5min":0.75,"15min":0.52400000000000002},"max":{"1min":1.718,"5min":1.885,"15min":2.1400000000000001},"last":1.45}]},{"osd":5,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.425,"15min":1.413},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.63100000000000001},"max":{"1min":1.861,"5min":2.0390000000000001,"15min":3.5270000000000001},"last":1.137},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.452},"min":{"1min":1.0940000000000001,"5min":0.95399999999999996,"15min":0.54200000000000004},"max":{"1min":1.9359999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.4159999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5389999999999999},"min":{"1min":1.044,"5min":1.044,"15min":0.82499999999999996},"max":{"1min":1.9890000000000001,"5min":2.1949999999999998,"15min":2.3220000000000001},"last":1.49},{"interface":"front","average":{"1min":1.498,"5min":1.496,"15min":1.4770000000000001},"min":{"1min":1.097,"5min":1.006,"15min":0.56200000000000006},"max":{"1min":2.036,"5min":2.0640000000000001,"15min":2.7040000000000002},"last":1.55}]}]},{"osd":6,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":757760,"data_stored":352325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1479999999999999,"15min":1.171},"min":{"1min":0.88300000000000001,"5min":0.78100000000000003,"15min":0.63700000000000001},"max":{"1min":1.627,"5min":1.655,"15min":2.339},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2430000000000001,"15min":1.214},"min":{"1min":0.88500000000000001,"5min":0.84099999999999997,"15min":0.68799999999999994},"max":{"1min":1.5589999999999999,"5min":1.752,"15min":3.2509999999999999},"last":1.264}]},{"osd":1,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.169,"15min":1.212},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.4159999999999999,"5min":1.857,"15min":4.3280000000000003},"last":1.353},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2589999999999999,"15min":1.3100000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.5429999999999999,"5min":2.1309999999999998,"15min":3.431},"last":1.7230000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3560000000000001,"15min":1.343},"min":{"1min":0.90200000000000002,"5min":0.85999999999999999,"15min":0.75900000000000001},"max":{"1min":1.4530000000000001,"5min":1.9470000000000001,"15min":2.6539999999999999},"last":1.421},{"interface":"front","average":{"1min":1.351,"5min":1.413,"15min":1.413},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6140000000000001,"5min":1.895,"15min":2.7200000000000002},"last":1.462}]},{"osd":3,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.377,"15min":1.3919999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.90100000000000002},"max":{"1min":1.867,"5min":1.867,"15min":2.8319999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.444,"15min":1.4179999999999999},"min":{"1min":1.0609999999999999,"5min":1.05,"15min":0.75600000000000001},"max":{"1min":1.79,"5min":1.827,"15min":3.6059999999999999},"last":1.6259999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3080000000000001,"15min":1.3620000000000001},"min":{"1min":0.94399999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.899,"5min":1.899,"15min":3.0790000000000002},"last":1.849},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4259999999999999,"15min":1.425},"min":{"1min":0.95899999999999996,"5min":0.88700000000000001,"15min":0.82999999999999996},"max":{"1min":1.946,"5min":2.0270000000000001,"15min":2.738},"last":1.524}]},{"osd":5,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3959999999999999,"15min":1.3939999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.83299999999999996},"max":{"1min":2.0619999999999998,"5min":2.105,"15min":3.0110000000000001},"last":1.677},{"interface":"front","average":{"1min":1.611,"5min":1.538,"15min":1.573},"min":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":0.96799999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":3.206},"last":1.9419999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.6020000000000001,"15min":1.589},"min":{"1min":1.3049999999999999,"5min":0.98899999999999999,"15min":0.90900000000000003},"max":{"1min":2.0619999999999998,"5min":2.0880000000000001,"15min":2.6349999999999998},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.5389999999999999,"15min":1.5649999999999999},"min":{"1min":0.999,"5min":0.999,"15min":0.97099999999999997},"max":{"1min":1.9179999999999999,"5min":2.181,"15min":3.1320000000000001},"last":1.5800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":753664,"data_stored":351723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.097,"15min":1.129},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.52200000000000002},"max":{"1min":1.548,"5min":1.655,"15min":3.2000000000000002},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.077,"5min":1.1339999999999999,"15min":1.161},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.57099999999999995},"max":{"1min":1.4930000000000001,"5min":1.665,"15min":4.101},"last":0.82199999999999995}]},{"osd":1,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1539999999999999,"15min":1.212},"min":{"1min":0.88300000000000001,"5min":0.79500000000000004,"15min":0.63400000000000001},"max":{"1min":1.446,"5min":1.613,"15min":3.1659999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.135,"5min":1.1950000000000001,"15min":1.244},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6839999999999999,"5min":1.7609999999999999,"15min":2.2549999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1890000000000001,"15min":1.2470000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.69099999999999995},"max":{"1min":1.4239999999999999,"5min":1.6439999999999999,"15min":4.3639999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3260000000000001,"15min":1.353},"min":{"1min":0.97699999999999998,"5min":0.81499999999999995,"15min":0.79300000000000004},"max":{"1min":1.6619999999999999,"5min":2.1030000000000002,"15min":4.3339999999999996},"last":0.99399999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3700000000000001,"15min":1.3979999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":2.431},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.425,"15min":1.4450000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.88600000000000001},"max":{"1min":1.7589999999999999,"5min":1.8939999999999999,"15min":3.6890000000000001},"last":0.91600000000000004}]},{"osd":4,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.242,"15min":1.3069999999999999},"min":{"1min":0.89700000000000002,"5min":0.79500000000000004,"15min":0.72199999999999998},"max":{"1min":1.8380000000000001,"5min":2.1890000000000001,"15min":2.577},"last":1.103},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3140000000000001,"15min":1.383},"min":{"1min":0.86499999999999999,"5min":0.81299999999999994,"15min":0.66200000000000003},"max":{"1min":1.8360000000000001,"5min":1.887,"15min":2.6030000000000002},"last":1.2729999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.415,"15min":1.4379999999999999},"min":{"1min":1.052,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":1.8799999999999999,"5min":2.2869999999999999,"15min":3.5739999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.516,"15min":1.581},"min":{"1min":1.097,"5min":1.083,"15min":0.81499999999999995},"max":{"1min":1.7989999999999999,"5min":2.2519999999999998,"15min":3.2949999999999999},"last":1.2270000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.502,"15min":1.548},"min":{"1min":1.075,"5min":0.94499999999999995,"15min":0.72299999999999998},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":3.0289999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.53,"15min":1.55},"min":{"1min":1.3149999999999999,"5min":0.91700000000000004,"15min":0.627},"max":{"1min":1.9730000000000001,"5min":2.3980000000000001,"15min":3.7869999999999999},"last":1.329}]}]},{"osd":4,"up_from":13,"seq":55834575019,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58304,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313536,"statfs":{"total":96636764160,"available":96577060864,"internally_reserved":0,"allocated":761856,"data_stored":353265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.165,"15min":1.181},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.61399999999999999},"max":{"1min":1.7050000000000001,"5min":1.7709999999999999,"15min":4.3810000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2370000000000001,"15min":1.254},"min":{"1min":0.83399999999999996,"5min":0.80900000000000005,"15min":0.69799999999999995},"max":{"1min":1.671,"5min":1.7390000000000001,"15min":4.4059999999999997},"last":0.80600000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.284,"15min":1.258},"min":{"1min":0.873,"5min":0.81699999999999995,"15min":0.60699999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":2.8639999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2,"5min":1.2470000000000001,"15min":1.2250000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.754},"max":{"1min":1.6180000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.5780000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.2390000000000001,"15min":1.2330000000000001},"min":{"1min":0.88700000000000001,"5min":0.84599999999999997,"15min":0.65000000000000002},"max":{"1min":1.802,"5min":1.802,"15min":1.8460000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3919999999999999,"15min":1.371},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.72799999999999998},"max":{"1min":1.744,"5min":1.8140000000000001,"15min":2.7759999999999998},"last":1.238}]},{"osd":3,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.335,"15min":1.306},"min":{"1min":0.98099999999999998,"5min":0.97499999999999998,"15min":0.752},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.048},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4199999999999999,"15min":1.4099999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.72199999999999998},"max":{"1min":1.952,"5min":1.952,"15min":2.085},"last":1.0009999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.349,"15min":1.3720000000000001},"min":{"1min":1.0549999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.855,"5min":1.855,"15min":2.2320000000000002},"last":1.504},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.359,"15min":1.371},"min":{"1min":0.90500000000000003,"5min":0.84199999999999997,"15min":0.80400000000000005},"max":{"1min":1.704,"5min":1.7909999999999999,"15min":2.153},"last":1.746}]},{"osd":6,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3959999999999999,"15min":1.421},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.542,"15min":1.5529999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.917,"5min":1.917,"15min":2.6539999999999999},"last":1.3799999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.502},"min":{"1min":1.097,"5min":0.94299999999999995,"15min":0.94199999999999995},"max":{"1min":1.7070000000000001,"5min":2.1269999999999998,"15min":2.2389999999999999},"last":1.9359999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5600000000000001,"15min":1.579},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.3999999999999999},"last":1.829}]}]},{"osd":3,"up_from":13,"seq":55834575019,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40244,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331596,"statfs":{"total":96636764160,"available":96595554304,"internally_reserved":0,"allocated":761856,"data_stored":352291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96499999999999997,"15min":0.93500000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.4239999999999999,"5min":1.5009999999999999,"15min":1.526},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.1719999999999999,"15min":1.1180000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.46500000000000002},"max":{"1min":1.417,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.442}]},{"osd":1,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.95199999999999996,"15min":0.93000000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.33600000000000002},"max":{"1min":1.206,"5min":1.597,"15min":1.7889999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.1240000000000001,"15min":1.115},"min":{"1min":0.438,"5min":0.438,"15min":0.373},"max":{"1min":1.3120000000000001,"5min":1.8129999999999999,"15min":2.294},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.4490000000000001,"15min":1.355},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.51600000000000001},"max":{"1min":1.6739999999999999,"5min":2.0609999999999999,"15min":2.6819999999999999},"last":1.018},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3200000000000001,"15min":1.2629999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.601,"5min":2.0960000000000001,"15min":2.77},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3620000000000001,"15min":1.3740000000000001},"min":{"1min":1.0469999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.569,"5min":1.8480000000000001,"15min":2.8239999999999998},"last":3.4350000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4410000000000001},"min":{"1min":0.92600000000000005,"5min":0.92500000000000004,"15min":0.749},"max":{"1min":1.7350000000000001,"5min":1.8740000000000001,"15min":3.9540000000000002},"last":1.3169999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.77500000000000002},"max":{"1min":1.544,"5min":1.9950000000000001,"15min":3.0190000000000001},"last":2.516},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4450000000000001,"15min":1.462},"min":{"1min":1.1839999999999999,"5min":1.042,"15min":0.77400000000000002},"max":{"1min":1.556,"5min":2.3199999999999998,"15min":3.2490000000000001},"last":3.7410000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.65100000000000002},"max":{"1min":1.655,"5min":2.0579999999999998,"15min":4.0570000000000004},"last":1.246},{"interface":"front","average":{"1min":1.462,"5min":1.544,"15min":1.573},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.79200000000000004},"max":{"1min":1.7490000000000001,"5min":2.1059999999999999,"15min":4.1900000000000004},"last":2.1259999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5109999999999999,"15min":1.5089999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.745},"max":{"1min":1.681,"5min":2.419,"15min":2.9900000000000002},"last":2.871},{"interface":"front","average":{"1min":1.462,"5min":1.556,"15min":1.5569999999999999},"min":{"1min":1.137,"5min":0.95799999999999996,"15min":0.82899999999999996},"max":{"1min":1.8149999999999999,"5min":2.464,"15min":2.9569999999999999},"last":2.907}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40256,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331584,"statfs":{"total":96636764160,"available":96595542016,"internally_reserved":0,"allocated":765952,"data_stored":353287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.34300000000000003},"max":{"1min":1.4950000000000001,"5min":1.659,"15min":1.849},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0920000000000001,"15min":1.0489999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.45900000000000002},"max":{"1min":1.643,"5min":1.6519999999999999,"15min":3.2149999999999999},"last":1.048}]},{"osd":1,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.995,"15min":0.97499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.42899999999999999},"max":{"1min":1.415,"5min":1.518,"15min":2.0419999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2050000000000001,"15min":1.1739999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.42099999999999999},"max":{"1min":1.7090000000000001,"5min":2.069,"15min":2.2890000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2669999999999999,"15min":1.2250000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46100000000000002},"max":{"1min":1.5800000000000001,"5min":1.8069999999999999,"15min":2.1349999999999998},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3420000000000001,"15min":1.3140000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.628},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.506}]},{"osd":4,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4490000000000001,"15min":1.423},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.73099999999999998},"max":{"1min":1.95,"5min":1.95,"15min":2.6259999999999999},"last":1.669},{"interface":"front","average":{"1min":1.391,"5min":1.4199999999999999,"15min":1.381},"min":{"1min":0.93999999999999995,"5min":0.91500000000000004,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.0750000000000002,"15min":3.0459999999999998},"last":1.887}]},{"osd":5,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4299999999999999,"15min":1.407},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.65500000000000003},"max":{"1min":1.875,"5min":2.0019999999999998,"15min":2.2389999999999999},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.5529999999999999,"15min":1.534},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.80000000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.883},"last":1.784}]},{"osd":6,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4930000000000001,"15min":1.472},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.83499999999999996},"max":{"1min":1.929,"5min":2.0289999999999999,"15min":3.1200000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.5640000000000001,"15min":1.54},"min":{"1min":1.002,"5min":1.002,"15min":0.70699999999999996},"max":{"1min":2.02,"5min":2.105,"15min":2.3420000000000001},"last":1.585}]},{"osd":7,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.573,"15min":1.556},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.77400000000000002},"max":{"1min":1.964,"5min":2.109,"15min":3.169},"last":1.444},{"interface":"front","average":{"1min":1.516,"5min":1.6100000000000001,"15min":1.609},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.755},"max":{"1min":1.9590000000000001,"5min":2.165,"15min":2.996},"last":1.8300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58272,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313568,"statfs":{"total":96636764160,"available":96577093632,"internally_reserved":0,"allocated":741376,"data_stored":345379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.93400000000000005,"15min":0.90000000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.35999999999999999},"max":{"1min":1.3839999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.0640000000000001,"15min":1.0509999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1930000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.96499999999999997,"15min":0.94699999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2250000000000001,"5min":1.758,"15min":1.758},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1679999999999999,"15min":1.169},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.45200000000000001},"max":{"1min":1.726,"5min":2.0150000000000001,"15min":2.1499999999999999},"last":0.95499999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.2929999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40999999999999998},"max":{"1min":1.885,"5min":2.198,"15min":2.5339999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.434},"max":{"1min":1.4039999999999999,"5min":1.8680000000000001,"15min":4.6799999999999997},"last":1.2110000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.425,"15min":1.4419999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.83799999999999997},"max":{"1min":1.78,"5min":2.0139999999999998,"15min":4.5789999999999997},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.323,"15min":1.3129999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.74,"5min":1.9279999999999999,"15min":2.0840000000000001},"last":0.98999999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3919999999999999,"15min":1.379},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.84099999999999997},"max":{"1min":1.843,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.502,"15min":1.4990000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.028,"5min":2.302,"15min":2.4980000000000002},"last":1.3740000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5129999999999999,"15min":1.5369999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.872},"max":{"1min":1.698,"5min":2.3479999999999999,"15min":4.6360000000000001},"last":1.625},{"interface":"front","average":{"1min":1.327,"5min":1.4690000000000001,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9019999999999999,"5min":2.1070000000000002,"15min":2.2970000000000002},"last":1.1639999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.589,"15min":1.591},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.79900000000000004},"max":{"1min":2.1400000000000001,"5min":2.411,"15min":2.452},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8819999999999999,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.573}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1335296,"data_stored":935711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89200000000000002,"15min":0.88100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2050000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.085},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.46999999999999997},"max":{"1min":1.6679999999999999,"5min":1.95,"15min":1.95},"last":1.3069999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0820000000000001,"15min":1.0740000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.6359999999999999,"5min":1.962,"15min":1.962},"last":1.034},{"interface":"front","average":{"1min":1.135,"5min":1.2350000000000001,"15min":1.1870000000000001},"min":{"1min":0.874,"5min":0.86099999999999999,"15min":0.42699999999999999},"max":{"1min":1.667,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.111}]},{"osd":3,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.288,"15min":1.254},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.53300000000000003},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3959999999999999,"15min":1.333},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.51700000000000002},"max":{"1min":1.843,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.8120000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2969999999999999,"15min":1.2569999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.63600000000000001},"max":{"1min":1.5680000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.429,"15min":1.4350000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.77800000000000002},"max":{"1min":1.873,"5min":2.101,"15min":2.3460000000000001},"last":1.879}]},{"osd":5,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.3109999999999999,"15min":1.323},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.66100000000000003},"max":{"1min":1.661,"5min":1.8720000000000001,"15min":2.0539999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.5409999999999999,"15min":1.5089999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0230000000000001,"5min":2.395,"15min":2.395},"last":1.474}]},{"osd":6,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.635,"15min":1.577},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":2.0870000000000002,"5min":2.492,"15min":2.492},"last":1.605},{"interface":"front","average":{"1min":1.337,"5min":1.417,"15min":1.391},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.77000000000000002},"max":{"1min":1.696,"5min":1.9790000000000001,"15min":2.1850000000000001},"last":1.671}]},{"osd":7,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.552,"15min":1.534},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83399999999999996},"max":{"1min":2.3069999999999999,"5min":2.3410000000000002,"15min":2.5219999999999998},"last":2.1070000000000002},{"interface":"front","average":{"1min":1.456,"5min":1.5429999999999999,"15min":1.5169999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":2.02,"5min":2.415,"15min":2.415},"last":1.8520000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:54:20.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:54:20.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.325+0000 7f76c34e7640 1 Processor -- start 2023-10-14T16:54:20.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.325+0000 7f76c34e7640 1 -- start start 2023-10-14T16:54:20.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c34e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76bc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c34e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76bc0b0b20 0x7f76bc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c34e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc06f7f0 con 0x7f76bc159f00 2023-10-14T16:54:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c34e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc06c5e0 con 0x7f76bc0b0b20 2023-10-14T16:54:20.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c34e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc06c8c0 con 0x7f76bc159b20 2023-10-14T16:54:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c0a82640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76bc159f00 0x7f76bc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39714/0 (socket says 172.21.15.29:39714) 2023-10-14T16:54:20.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c0a82640 1 -- 172.21.15.29:0/3258940979 learned_addr learned my addr 172.21.15.29:0/3258940979 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:20.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c1283640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76bc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.327+0000 7f76c1a84640 1 --2- 172.21.15.29:0/3258940979 >> v2:172.21.15.186:3300/0 conn(0x7f76bc0b0b20 0x7f76bc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290509733 0 0) 0x7f76bc06f7f0 con 0x7f76bc159f00 2023-10-14T16:54:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a8005730 con 0x7f76bc159f00 2023-10-14T16:54:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70059453 0 0) 0x7f76a8005730 con 0x7f76bc159f00 2023-10-14T16:54:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 msgr2=0x7f76bc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76bc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 >> v2:172.21.15.186:3300/0 conn(0x7f76bc0b0b20 msgr2=0x7f76bc06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 --2- 172.21.15.29:0/3258940979 >> v2:172.21.15.186:3300/0 conn(0x7f76bc0b0b20 0x7f76bc06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc06cba0 con 0x7f76bc159f00 2023-10-14T16:54:20.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f76ac001670 con 0x7f76bc159f00 2023-10-14T16:54:20.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 >> v1:172.21.15.29:6789/0 conn(0x7f76bc159f00 legacy=0x7f76bc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76a8005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a8006100 0x7f76a80069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76a8006f10 0x7f76a8007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc06c5e0 con 0x7f76a8006100 2023-10-14T16:54:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc06c8c0 con 0x7f76a8006f10 2023-10-14T16:54:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.328+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a800db90 con 0x7f76bc159b20 2023-10-14T16:54:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a8006100 0x7f76a80069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c1a84640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76a8006f10 0x7f76a8007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c1283640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76a8005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 -- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 msgr2=0x7f76a8005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76bc159b20 0x7f76a8005b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 -- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76a8006f10 msgr2=0x7f76a8007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76a8006f10 0x7f76a8007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c0a82640 1 -- 172.21.15.29:0/3258940979 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a8005730 con 0x7f76a8006100 2023-10-14T16:54:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.329+0000 7f76c1a84640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76a8006f10 0x7f76a8007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.330+0000 7f76c0a82640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a8006100 0x7f76a80069d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f76bc15a2e0 tx=0x7f76a8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b26b228e65c416b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac005ad0 con 0x7f76a8006100 2023-10-14T16:54:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76ac005c30 con 0x7f76a8006100 2023-10-14T16:54:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76b3fff640 1 -- 172.21.15.29:0/3258940979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76ac02c880 con 0x7f76a8006100 2023-10-14T16:54:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c34e7640 1 -- 172.21.15.29:0/3258940979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a8006100 msgr2=0x7f76a80069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c34e7640 1 --2- 172.21.15.29:0/3258940979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76a8006100 0x7f76a80069d0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f76bc15a2e0 tx=0x7f76a8007a00 comp rx=0 tx=0).stop 2023-10-14T16:54:20.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c1a84640 1 -- 172.21.15.29:0/3258940979 reap_dead start 2023-10-14T16:54:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c34e7640 1 -- 172.21.15.29:0/3258940979 shutdown_connections 2023-10-14T16:54:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c34e7640 1 -- 172.21.15.29:0/3258940979 >> 172.21.15.29:0/3258940979 conn(0x7f76bc0cb410 msgr2=0x7f76bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.331+0000 7f76c34e7640 1 -- 172.21.15.29:0/3258940979 shutdown_connections 2023-10-14T16:54:20.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.332+0000 7f76c34e7640 1 -- 172.21.15.29:0/3258940979 wait complete. 2023-10-14T16:54:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.333+0000 7f76c34e7640 1 Processor -- start 2023-10-14T16:54:20.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.334+0000 7f76c34e7640 1 -- start start 2023-10-14T16:54:20.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.334+0000 7f76c34e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76a8006100 0x7f76bc1ef4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.334+0000 7f76c34e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc0c84c0 0x7f76bc1f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c34e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c34e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1f29f0 con 0x7f76bc1f2520 2023-10-14T16:54:20.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c34e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1f2cd0 con 0x7f76bc0c84c0 2023-10-14T16:54:20.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c34e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc1eecf0 con 0x7f76a8006100 2023-10-14T16:54:20.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36546/0 (socket says 172.21.15.29:36546) 2023-10-14T16:54:20.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 -- 172.21.15.29:0/1971948510 learned_addr learned my addr 172.21.15.29:0/1971948510 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:20.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1283640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76a8006100 0x7f76bc1ef4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c0a82640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc0c84c0 0x7f76bc1f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 -- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76a8006100 msgr2=0x7f76bc1ef4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76a8006100 0x7f76bc1ef4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 -- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc0c84c0 msgr2=0x7f76bc1f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc0c84c0 0x7f76bc1f1fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.335+0000 7f76c1a84640 1 -- 172.21.15.29:0/1971948510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc1eefd0 con 0x7f76bc1f2520 2023-10-14T16:54:20.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76c1a84640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f76bc06d360 tx=0x7f76b40115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b400e030 con 0x7f76bc1f2520 2023-10-14T16:54:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b4007e60 con 0x7f76bc1f2520 2023-10-14T16:54:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76b4016470 con 0x7f76bc1f2520 2023-10-14T16:54:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc1ee9e0 con 0x7f76bc1f2520 2023-10-14T16:54:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.336+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc156400 con 0x7f76bc1f2520 2023-10-14T16:54:20.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.337+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc1ef1c0 con 0x7f76bc1f2520 2023-10-14T16:54:20.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.338+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f76b40165f0 con 0x7f76bc1f2520 2023-10-14T16:54:20.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.339+0000 7f76b1ffb640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 0x7f76a006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.339+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f76b400f070 con 0x7f76bc1f2520 2023-10-14T16:54:20.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.340+0000 7f76c1283640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 0x7f76a006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.340+0000 7f76c1283640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 0x7f76a006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009930 tx=0x7f76b8008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.344+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76b40550b0 con 0x7f76bc1f2520 2023-10-14T16:54:20.465 INFO:tasks.ceph.mds.d.smithi186.stderr:2023-10-14T16:54:20.464+0000 7fc044cf0640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:54:20.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.538+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 --> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f76bc06ce80 con 0x7f76a006afc0 2023-10-14T16:54:20.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.548+0000 7f76b1ffb640 1 -- 172.21.15.29:0/1971948510 <== mgr.5763 v2:172.21.15.29:6832/98172895 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286455 (secure 0 0 0) 0x7f76b80095c0 con 0x7f76a006afc0 2023-10-14T16:54:20.552 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:20.554 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:54:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 msgr2=0x7f76a006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 0x7f76a006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f76b8009930 tx=0x7f76b8008040 comp rx=0 tx=0).stop 2023-10-14T16:54:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 msgr2=0x7f76bc15c2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f76bc06d360 tx=0x7f76b40115a0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 shutdown_connections 2023-10-14T16:54:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f76a006afc0 0x7f76a006d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76bc1f2520 0x7f76bc15c2f0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f76bc0c84c0 0x7f76bc1f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 --2- 172.21.15.29:0/1971948510 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76a8006100 0x7f76bc1ef4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 >> 172.21.15.29:0/1971948510 conn(0x7f76bc0cb410 msgr2=0x7f76bc0cdab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:20.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.553+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 shutdown_connections 2023-10-14T16:54:20.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.554+0000 7f76c34e7640 1 -- 172.21.15.29:0/1971948510 wait complete. 2023-10-14T16:54:20.572 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-14T16:54:18.622468+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":243,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":177,"ondisk_log_size":177,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":144,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":330060,"kb_used_data":7056,"kb_used_omap":12,"kb_used_meta":322867,"kb_avail":754644660,"statfs":{"total":773094113280,"available":772756131840,"internally_reserved":0,"allocated":7225344,"data_stored":3986092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":330616376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.129592"},"pg_stats":[{"pgid":"7.3f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042521+0000","last_change":"2023-10-14T16:54:15.051309+0000","last_active":"2023-10-14T16:54:16.042521+0000","last_peered":"2023-10-14T16:54:16.042521+0000","last_clean":"2023-10-14T16:54:16.042521+0000","last_became_active":"2023-10-14T16:54:15.051034+0000","last_became_peered":"2023-10-14T16:54:15.051034+0000","last_unstale":"2023-10-14T16:54:16.042521+0000","last_undegraded":"2023-10-14T16:54:16.042521+0000","last_fullsized":"2023-10-14T16:54:16.042521+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:24.753631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037016+0000","last_change":"2023-10-14T16:54:15.061821+0000","last_active":"2023-10-14T16:54:16.037016+0000","last_peered":"2023-10-14T16:54:16.037016+0000","last_clean":"2023-10-14T16:54:16.037016+0000","last_became_active":"2023-10-14T16:54:15.061509+0000","last_became_peered":"2023-10-14T16:54:15.061509+0000","last_unstale":"2023-10-14T16:54:16.037016+0000","last_undegraded":"2023-10-14T16:54:16.037016+0000","last_fullsized":"2023-10-14T16:54:16.037016+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:05.604809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447927+0000","last_change":"2023-10-14T16:54:15.051176+0000","last_active":"2023-10-14T16:54:16.447927+0000","last_peered":"2023-10-14T16:54:16.447927+0000","last_clean":"2023-10-14T16:54:16.447927+0000","last_became_active":"2023-10-14T16:54:15.050889+0000","last_became_peered":"2023-10-14T16:54:15.050889+0000","last_unstale":"2023-10-14T16:54:16.447927+0000","last_undegraded":"2023-10-14T16:54:16.447927+0000","last_fullsized":"2023-10-14T16:54:16.447927+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:02.493144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497293+0000","last_change":"2023-10-14T16:54:15.063187+0000","last_active":"2023-10-14T16:54:17.497293+0000","last_peered":"2023-10-14T16:54:17.497293+0000","last_clean":"2023-10-14T16:54:17.497293+0000","last_became_active":"2023-10-14T16:54:15.062799+0000","last_became_peered":"2023-10-14T16:54:15.062799+0000","last_unstale":"2023-10-14T16:54:17.497293+0000","last_undegraded":"2023-10-14T16:54:17.497293+0000","last_fullsized":"2023-10-14T16:54:17.497293+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:03.980605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036941+0000","last_change":"2023-10-14T16:54:15.064052+0000","last_active":"2023-10-14T16:54:16.036941+0000","last_peered":"2023-10-14T16:54:16.036941+0000","last_clean":"2023-10-14T16:54:16.036941+0000","last_became_active":"2023-10-14T16:54:15.063764+0000","last_became_peered":"2023-10-14T16:54:15.063764+0000","last_unstale":"2023-10-14T16:54:16.036941+0000","last_undegraded":"2023-10-14T16:54:16.036941+0000","last_fullsized":"2023-10-14T16:54:16.036941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:28:58.879612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040375+0000","last_change":"2023-10-14T16:54:15.051894+0000","last_active":"2023-10-14T16:54:16.040375+0000","last_peered":"2023-10-14T16:54:16.040375+0000","last_clean":"2023-10-14T16:54:16.040375+0000","last_became_active":"2023-10-14T16:54:15.050249+0000","last_became_peered":"2023-10-14T16:54:15.050249+0000","last_unstale":"2023-10-14T16:54:16.040375+0000","last_undegraded":"2023-10-14T16:54:16.040375+0000","last_fullsized":"2023-10-14T16:54:16.040375+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:30.616445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.39","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041263+0000","last_change":"2023-10-14T16:54:15.049687+0000","last_active":"2023-10-14T16:54:16.041263+0000","last_peered":"2023-10-14T16:54:16.041263+0000","last_clean":"2023-10-14T16:54:16.041263+0000","last_became_active":"2023-10-14T16:54:15.049398+0000","last_became_peered":"2023-10-14T16:54:15.049398+0000","last_unstale":"2023-10-14T16:54:16.041263+0000","last_undegraded":"2023-10-14T16:54:16.041263+0000","last_fullsized":"2023-10-14T16:54:16.041263+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:22.844113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.38","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496069+0000","last_change":"2023-10-14T16:54:15.061176+0000","last_active":"2023-10-14T16:54:17.496069+0000","last_peered":"2023-10-14T16:54:17.496069+0000","last_clean":"2023-10-14T16:54:17.496069+0000","last_became_active":"2023-10-14T16:54:15.060953+0000","last_became_peered":"2023-10-14T16:54:15.060953+0000","last_unstale":"2023-10-14T16:54:17.496069+0000","last_undegraded":"2023-10-14T16:54:17.496069+0000","last_fullsized":"2023-10-14T16:54:17.496069+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.265449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.37","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497733+0000","last_change":"2023-10-14T16:54:15.064214+0000","last_active":"2023-10-14T16:54:17.497733+0000","last_peered":"2023-10-14T16:54:17.497733+0000","last_clean":"2023-10-14T16:54:17.497733+0000","last_became_active":"2023-10-14T16:54:15.063062+0000","last_became_peered":"2023-10-14T16:54:15.063062+0000","last_unstale":"2023-10-14T16:54:17.497733+0000","last_undegraded":"2023-10-14T16:54:17.497733+0000","last_fullsized":"2023-10-14T16:54:17.497733+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:31.656932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.36","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497838+0000","last_change":"2023-10-14T16:54:15.058689+0000","last_active":"2023-10-14T16:54:17.497838+0000","last_peered":"2023-10-14T16:54:17.497838+0000","last_clean":"2023-10-14T16:54:17.497838+0000","last_became_active":"2023-10-14T16:54:15.058335+0000","last_became_peered":"2023-10-14T16:54:15.058335+0000","last_unstale":"2023-10-14T16:54:17.497838+0000","last_undegraded":"2023-10-14T16:54:17.497838+0000","last_fullsized":"2023-10-14T16:54:17.497838+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:02.180000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.35","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497427+0000","last_change":"2023-10-14T16:54:15.056497+0000","last_active":"2023-10-14T16:54:17.497427+0000","last_peered":"2023-10-14T16:54:17.497427+0000","last_clean":"2023-10-14T16:54:17.497427+0000","last_became_active":"2023-10-14T16:54:15.056220+0000","last_became_peered":"2023-10-14T16:54:15.056220+0000","last_unstale":"2023-10-14T16:54:17.497427+0000","last_undegraded":"2023-10-14T16:54:17.497427+0000","last_fullsized":"2023-10-14T16:54:17.497427+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:57.184914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.34","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036941+0000","last_change":"2023-10-14T16:54:15.044431+0000","last_active":"2023-10-14T16:54:16.036941+0000","last_peered":"2023-10-14T16:54:16.036941+0000","last_clean":"2023-10-14T16:54:16.036941+0000","last_became_active":"2023-10-14T16:54:15.043351+0000","last_became_peered":"2023-10-14T16:54:15.043351+0000","last_unstale":"2023-10-14T16:54:16.036941+0000","last_undegraded":"2023-10-14T16:54:16.036941+0000","last_fullsized":"2023-10-14T16:54:16.036941+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:10:47.236262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.33","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190098+0000","last_change":"2023-10-14T16:54:15.057376+0000","last_active":"2023-10-14T16:54:16.190098+0000","last_peered":"2023-10-14T16:54:16.190098+0000","last_clean":"2023-10-14T16:54:16.190098+0000","last_became_active":"2023-10-14T16:54:15.056769+0000","last_became_peered":"2023-10-14T16:54:15.056769+0000","last_unstale":"2023-10-14T16:54:16.190098+0000","last_undegraded":"2023-10-14T16:54:16.190098+0000","last_fullsized":"2023-10-14T16:54:16.190098+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:34:46.870634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.32","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496582+0000","last_change":"2023-10-14T16:54:15.063500+0000","last_active":"2023-10-14T16:54:17.496582+0000","last_peered":"2023-10-14T16:54:17.496582+0000","last_clean":"2023-10-14T16:54:17.496582+0000","last_became_active":"2023-10-14T16:54:15.063284+0000","last_became_peered":"2023-10-14T16:54:15.063284+0000","last_unstale":"2023-10-14T16:54:17.496582+0000","last_undegraded":"2023-10-14T16:54:17.496582+0000","last_fullsized":"2023-10-14T16:54:17.496582+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:25:21.653999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.31","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040268+0000","last_change":"2023-10-14T16:54:15.054006+0000","last_active":"2023-10-14T16:54:16.040268+0000","last_peered":"2023-10-14T16:54:16.040268+0000","last_clean":"2023-10-14T16:54:16.040268+0000","last_became_active":"2023-10-14T16:54:15.053716+0000","last_became_peered":"2023-10-14T16:54:15.053716+0000","last_unstale":"2023-10-14T16:54:16.040268+0000","last_undegraded":"2023-10-14T16:54:16.040268+0000","last_fullsized":"2023-10-14T16:54:16.040268+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:27.771769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.30","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038021+0000","last_change":"2023-10-14T16:54:15.055763+0000","last_active":"2023-10-14T16:54:16.038021+0000","last_peered":"2023-10-14T16:54:16.038021+0000","last_clean":"2023-10-14T16:54:16.038021+0000","last_became_active":"2023-10-14T16:54:15.055487+0000","last_became_peered":"2023-10-14T16:54:15.055487+0000","last_unstale":"2023-10-14T16:54:16.038021+0000","last_undegraded":"2023-10-14T16:54:16.038021+0000","last_fullsized":"2023-10-14T16:54:16.038021+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:19.439016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447497+0000","last_change":"2023-10-14T16:54:15.053078+0000","last_active":"2023-10-14T16:54:16.447497+0000","last_peered":"2023-10-14T16:54:16.447497+0000","last_clean":"2023-10-14T16:54:16.447497+0000","last_became_active":"2023-10-14T16:54:15.052754+0000","last_became_peered":"2023-10-14T16:54:15.052754+0000","last_unstale":"2023-10-14T16:54:16.447497+0000","last_undegraded":"2023-10-14T16:54:16.447497+0000","last_fullsized":"2023-10-14T16:54:16.447497+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:16.161287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447090+0000","last_change":"2023-10-14T16:54:15.062654+0000","last_active":"2023-10-14T16:54:16.447090+0000","last_peered":"2023-10-14T16:54:16.447090+0000","last_clean":"2023-10-14T16:54:16.447090+0000","last_became_active":"2023-10-14T16:54:15.062346+0000","last_became_peered":"2023-10-14T16:54:15.062346+0000","last_unstale":"2023-10-14T16:54:16.447090+0000","last_undegraded":"2023-10-14T16:54:16.447090+0000","last_fullsized":"2023-10-14T16:54:16.447090+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:07.343276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.2d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039871+0000","last_change":"2023-10-14T16:54:15.063487+0000","last_active":"2023-10-14T16:54:16.039871+0000","last_peered":"2023-10-14T16:54:16.039871+0000","last_clean":"2023-10-14T16:54:16.039871+0000","last_became_active":"2023-10-14T16:54:15.063164+0000","last_became_peered":"2023-10-14T16:54:15.063164+0000","last_unstale":"2023-10-14T16:54:16.039871+0000","last_undegraded":"2023-10-14T16:54:16.039871+0000","last_fullsized":"2023-10-14T16:54:16.039871+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:31.656932+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190404+0000","last_change":"2023-10-14T16:54:15.062679+0000","last_active":"2023-10-14T16:54:16.190404+0000","last_peered":"2023-10-14T16:54:16.190404+0000","last_clean":"2023-10-14T16:54:16.190404+0000","last_became_active":"2023-10-14T16:54:15.062400+0000","last_became_peered":"2023-10-14T16:54:15.062400+0000","last_unstale":"2023-10-14T16:54:16.190404+0000","last_undegraded":"2023-10-14T16:54:16.190404+0000","last_fullsized":"2023-10-14T16:54:16.190404+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:39:11.808051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040664+0000","last_change":"2023-10-14T16:54:15.057954+0000","last_active":"2023-10-14T16:54:16.040664+0000","last_peered":"2023-10-14T16:54:16.040664+0000","last_clean":"2023-10-14T16:54:16.040664+0000","last_became_active":"2023-10-14T16:54:15.057680+0000","last_became_peered":"2023-10-14T16:54:15.057680+0000","last_unstale":"2023-10-14T16:54:16.040664+0000","last_undegraded":"2023-10-14T16:54:16.040664+0000","last_fullsized":"2023-10-14T16:54:16.040664+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:31.972755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496832+0000","last_change":"2023-10-14T16:54:15.060209+0000","last_active":"2023-10-14T16:54:17.496832+0000","last_peered":"2023-10-14T16:54:17.496832+0000","last_clean":"2023-10-14T16:54:17.496832+0000","last_became_active":"2023-10-14T16:54:15.059731+0000","last_became_peered":"2023-10-14T16:54:15.059731+0000","last_unstale":"2023-10-14T16:54:17.496832+0000","last_undegraded":"2023-10-14T16:54:17.496832+0000","last_fullsized":"2023-10-14T16:54:17.496832+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:22:00.401204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.29","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038210+0000","last_change":"2023-10-14T16:54:15.051167+0000","last_active":"2023-10-14T16:54:16.038210+0000","last_peered":"2023-10-14T16:54:16.038210+0000","last_clean":"2023-10-14T16:54:16.038210+0000","last_became_active":"2023-10-14T16:54:15.050859+0000","last_became_peered":"2023-10-14T16:54:15.050859+0000","last_unstale":"2023-10-14T16:54:16.038210+0000","last_undegraded":"2023-10-14T16:54:16.038210+0000","last_fullsized":"2023-10-14T16:54:16.038210+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:25:21.653999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.28","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038013+0000","last_change":"2023-10-14T16:54:15.050903+0000","last_active":"2023-10-14T16:54:16.038013+0000","last_peered":"2023-10-14T16:54:16.038013+0000","last_clean":"2023-10-14T16:54:16.038013+0000","last_became_active":"2023-10-14T16:54:15.050366+0000","last_became_peered":"2023-10-14T16:54:15.050366+0000","last_unstale":"2023-10-14T16:54:16.038013+0000","last_undegraded":"2023-10-14T16:54:16.038013+0000","last_fullsized":"2023-10-14T16:54:16.038013+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:29.103133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.27","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.038702+0000","last_change":"2023-10-14T16:54:15.060414+0000","last_active":"2023-10-14T16:54:16.038702+0000","last_peered":"2023-10-14T16:54:16.038702+0000","last_clean":"2023-10-14T16:54:16.038702+0000","last_became_active":"2023-10-14T16:54:15.060129+0000","last_became_peered":"2023-10-14T16:54:15.060129+0000","last_unstale":"2023-10-14T16:54:16.038702+0000","last_undegraded":"2023-10-14T16:54:16.038702+0000","last_fullsized":"2023-10-14T16:54:16.038702+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:22:00.401204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.26","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190301+0000","last_change":"2023-10-14T16:54:15.058846+0000","last_active":"2023-10-14T16:54:16.190301+0000","last_peered":"2023-10-14T16:54:16.190301+0000","last_clean":"2023-10-14T16:54:16.190301+0000","last_became_active":"2023-10-14T16:54:15.058578+0000","last_became_peered":"2023-10-14T16:54:15.058578+0000","last_unstale":"2023-10-14T16:54:16.190301+0000","last_undegraded":"2023-10-14T16:54:16.190301+0000","last_fullsized":"2023-10-14T16:54:16.190301+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:24:08.864688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.25","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447502+0000","last_change":"2023-10-14T16:54:15.053692+0000","last_active":"2023-10-14T16:54:16.447502+0000","last_peered":"2023-10-14T16:54:16.447502+0000","last_clean":"2023-10-14T16:54:16.447502+0000","last_became_active":"2023-10-14T16:54:15.053451+0000","last_became_peered":"2023-10-14T16:54:15.053451+0000","last_unstale":"2023-10-14T16:54:16.447502+0000","last_undegraded":"2023-10-14T16:54:16.447502+0000","last_fullsized":"2023-10-14T16:54:16.447502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:41:20.110552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.24","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037431+0000","last_change":"2023-10-14T16:54:15.044439+0000","last_active":"2023-10-14T16:54:16.037431+0000","last_peered":"2023-10-14T16:54:16.037431+0000","last_clean":"2023-10-14T16:54:16.037431+0000","last_became_active":"2023-10-14T16:54:15.043944+0000","last_became_peered":"2023-10-14T16:54:15.043944+0000","last_unstale":"2023-10-14T16:54:16.037431+0000","last_undegraded":"2023-10-14T16:54:16.037431+0000","last_fullsized":"2023-10-14T16:54:16.037431+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:57:52.855997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.23","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040222+0000","last_change":"2023-10-14T16:54:15.054146+0000","last_active":"2023-10-14T16:54:16.040222+0000","last_peered":"2023-10-14T16:54:16.040222+0000","last_clean":"2023-10-14T16:54:16.040222+0000","last_became_active":"2023-10-14T16:54:15.053765+0000","last_became_peered":"2023-10-14T16:54:15.053765+0000","last_unstale":"2023-10-14T16:54:16.040222+0000","last_undegraded":"2023-10-14T16:54:16.040222+0000","last_fullsized":"2023-10-14T16:54:16.040222+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:02.180000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.22","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446967+0000","last_change":"2023-10-14T16:54:15.056344+0000","last_active":"2023-10-14T16:54:16.446967+0000","last_peered":"2023-10-14T16:54:16.446967+0000","last_clean":"2023-10-14T16:54:16.446967+0000","last_became_active":"2023-10-14T16:54:15.055937+0000","last_became_peered":"2023-10-14T16:54:15.055937+0000","last_unstale":"2023-10-14T16:54:16.446967+0000","last_undegraded":"2023-10-14T16:54:16.446967+0000","last_fullsized":"2023-10-14T16:54:16.446967+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:08:28.333167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.21","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040387+0000","last_change":"2023-10-14T16:54:15.052013+0000","last_active":"2023-10-14T16:54:16.040387+0000","last_peered":"2023-10-14T16:54:16.040387+0000","last_clean":"2023-10-14T16:54:16.040387+0000","last_became_active":"2023-10-14T16:54:15.051281+0000","last_became_peered":"2023-10-14T16:54:15.051281+0000","last_unstale":"2023-10-14T16:54:16.040387+0000","last_undegraded":"2023-10-14T16:54:16.040387+0000","last_fullsized":"2023-10-14T16:54:16.040387+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:24:24.048725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.20","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.189550+0000","last_change":"2023-10-14T16:54:15.049843+0000","last_active":"2023-10-14T16:54:16.189550+0000","last_peered":"2023-10-14T16:54:16.189550+0000","last_clean":"2023-10-14T16:54:16.189550+0000","last_became_active":"2023-10-14T16:54:15.049480+0000","last_became_peered":"2023-10-14T16:54:15.049480+0000","last_unstale":"2023-10-14T16:54:16.189550+0000","last_undegraded":"2023-10-14T16:54:16.189550+0000","last_fullsized":"2023-10-14T16:54:16.189550+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:07:26.170550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.064544+0000","last_change":"2023-10-14T16:54:15.062451+0000","last_active":"2023-10-14T16:54:15.064544+0000","last_peered":"2023-10-14T16:54:15.064544+0000","last_clean":"2023-10-14T16:54:15.064544+0000","last_became_active":"2023-10-14T16:54:15.062162+0000","last_became_peered":"2023-10-14T16:54:15.062162+0000","last_unstale":"2023-10-14T16:54:15.064544+0000","last_undegraded":"2023-10-14T16:54:15.064544+0000","last_fullsized":"2023-10-14T16:54:15.064544+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:31.855950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"62'4","reported_seq":16,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041302+0000","last_change":"2023-10-14T16:54:14.532761+0000","last_active":"2023-10-14T16:54:16.041302+0000","last_peered":"2023-10-14T16:54:16.041302+0000","last_clean":"2023-10-14T16:54:16.041302+0000","last_became_active":"2023-10-14T16:54:14.532557+0000","last_became_peered":"2023-10-14T16:54:14.532557+0000","last_unstale":"2023-10-14T16:54:16.041302+0000","last_undegraded":"2023-10-14T16:54:16.041302+0000","last_fullsized":"2023-10-14T16:54:16.041302+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:03.644829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039111+0000","last_change":"2023-10-14T16:54:15.052455+0000","last_active":"2023-10-14T16:54:16.039111+0000","last_peered":"2023-10-14T16:54:16.039111+0000","last_clean":"2023-10-14T16:54:16.039111+0000","last_became_active":"2023-10-14T16:54:15.052204+0000","last_became_peered":"2023-10-14T16:54:15.052204+0000","last_unstale":"2023-10-14T16:54:16.039111+0000","last_undegraded":"2023-10-14T16:54:16.039111+0000","last_fullsized":"2023-10-14T16:54:16.039111+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:33.172706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1e","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447563+0000","last_change":"2023-10-14T16:54:14.527661+0000","last_active":"2023-10-14T16:54:16.447563+0000","last_peered":"2023-10-14T16:54:16.447563+0000","last_clean":"2023-10-14T16:54:16.447563+0000","last_became_active":"2023-10-14T16:54:14.526737+0000","last_became_peered":"2023-10-14T16:54:14.526737+0000","last_unstale":"2023-10-14T16:54:16.447563+0000","last_undegraded":"2023-10-14T16:54:16.447563+0000","last_fullsized":"2023-10-14T16:54:16.447563+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:49.014304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037366+0000","last_change":"2023-10-14T16:54:15.057173+0000","last_active":"2023-10-14T16:54:16.037366+0000","last_peered":"2023-10-14T16:54:16.037366+0000","last_clean":"2023-10-14T16:54:16.037366+0000","last_became_active":"2023-10-14T16:54:15.056828+0000","last_became_peered":"2023-10-14T16:54:15.056828+0000","last_unstale":"2023-10-14T16:54:16.037366+0000","last_undegraded":"2023-10-14T16:54:16.037366+0000","last_fullsized":"2023-10-14T16:54:16.037366+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:43:00.550103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1d","version":"62'2","reported_seq":13,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.560241+0000","last_change":"2023-10-14T16:54:14.530229+0000","last_active":"2023-10-14T16:54:15.560241+0000","last_peered":"2023-10-14T16:54:15.560241+0000","last_clean":"2023-10-14T16:54:15.560241+0000","last_became_active":"2023-10-14T16:54:14.528566+0000","last_became_peered":"2023-10-14T16:54:14.528566+0000","last_unstale":"2023-10-14T16:54:15.560241+0000","last_undegraded":"2023-10-14T16:54:15.560241+0000","last_fullsized":"2023-10-14T16:54:15.560241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:51.926885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.063981+0000","last_change":"2023-10-14T16:54:15.060562+0000","last_active":"2023-10-14T16:54:15.063981+0000","last_peered":"2023-10-14T16:54:15.063981+0000","last_clean":"2023-10-14T16:54:15.063981+0000","last_became_active":"2023-10-14T16:54:15.060296+0000","last_became_peered":"2023-10-14T16:54:15.060296+0000","last_unstale":"2023-10-14T16:54:15.063981+0000","last_undegraded":"2023-10-14T16:54:15.063981+0000","last_fullsized":"2023-10-14T16:54:15.063981+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:54.303157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041938+0000","last_change":"2023-10-14T16:54:14.540360+0000","last_active":"2023-10-14T16:54:16.041938+0000","last_peered":"2023-10-14T16:54:16.041938+0000","last_clean":"2023-10-14T16:54:16.041938+0000","last_became_active":"2023-10-14T16:54:14.540032+0000","last_became_peered":"2023-10-14T16:54:14.540032+0000","last_unstale":"2023-10-14T16:54:16.041938+0000","last_undegraded":"2023-10-14T16:54:16.041938+0000","last_fullsized":"2023-10-14T16:54:16.041938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:59.533841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446965+0000","last_change":"2023-10-14T16:54:15.058028+0000","last_active":"2023-10-14T16:54:16.446965+0000","last_peered":"2023-10-14T16:54:16.446965+0000","last_clean":"2023-10-14T16:54:16.446965+0000","last_became_active":"2023-10-14T16:54:15.057733+0000","last_became_peered":"2023-10-14T16:54:15.057733+0000","last_unstale":"2023-10-14T16:54:16.446965+0000","last_undegraded":"2023-10-14T16:54:16.446965+0000","last_fullsized":"2023-10-14T16:54:16.446965+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:13.482060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040697+0000","last_change":"2023-10-14T16:54:14.536339+0000","last_active":"2023-10-14T16:54:16.040697+0000","last_peered":"2023-10-14T16:54:16.040697+0000","last_clean":"2023-10-14T16:54:16.040697+0000","last_became_active":"2023-10-14T16:54:14.536064+0000","last_became_peered":"2023-10-14T16:54:14.536064+0000","last_unstale":"2023-10-14T16:54:16.040697+0000","last_undegraded":"2023-10-14T16:54:16.040697+0000","last_fullsized":"2023-10-14T16:54:16.040697+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:04.588548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497096+0000","last_change":"2023-10-14T16:54:15.065694+0000","last_active":"2023-10-14T16:54:17.497096+0000","last_peered":"2023-10-14T16:54:17.497096+0000","last_clean":"2023-10-14T16:54:17.497096+0000","last_became_active":"2023-10-14T16:54:15.061582+0000","last_became_peered":"2023-10-14T16:54:15.061582+0000","last_unstale":"2023-10-14T16:54:17.497096+0000","last_undegraded":"2023-10-14T16:54:17.497096+0000","last_fullsized":"2023-10-14T16:54:17.497096+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:28:23.730093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1a","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447311+0000","last_change":"2023-10-14T16:54:14.532574+0000","last_active":"2023-10-14T16:54:16.447311+0000","last_peered":"2023-10-14T16:54:16.447311+0000","last_clean":"2023-10-14T16:54:16.447311+0000","last_became_active":"2023-10-14T16:54:14.530216+0000","last_became_peered":"2023-10-14T16:54:14.530216+0000","last_unstale":"2023-10-14T16:54:16.447311+0000","last_undegraded":"2023-10-14T16:54:16.447311+0000","last_fullsized":"2023-10-14T16:54:16.447311+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:15:09.132829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496373+0000","last_change":"2023-10-14T16:54:15.057528+0000","last_active":"2023-10-14T16:54:17.496373+0000","last_peered":"2023-10-14T16:54:17.496373+0000","last_clean":"2023-10-14T16:54:17.496373+0000","last_became_active":"2023-10-14T16:54:15.057291+0000","last_became_peered":"2023-10-14T16:54:15.057291+0000","last_unstale":"2023-10-14T16:54:17.496373+0000","last_undegraded":"2023-10-14T16:54:17.496373+0000","last_fullsized":"2023-10-14T16:54:17.496373+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:19.439016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036660+0000","last_change":"2023-10-14T16:54:14.529766+0000","last_active":"2023-10-14T16:54:16.036660+0000","last_peered":"2023-10-14T16:54:16.036660+0000","last_clean":"2023-10-14T16:54:16.036660+0000","last_became_active":"2023-10-14T16:54:14.529396+0000","last_became_peered":"2023-10-14T16:54:14.529396+0000","last_unstale":"2023-10-14T16:54:16.036660+0000","last_undegraded":"2023-10-14T16:54:16.036660+0000","last_fullsized":"2023-10-14T16:54:16.036660+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:18:45.997572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037509+0000","last_change":"2023-10-14T16:54:15.047954+0000","last_active":"2023-10-14T16:54:16.037509+0000","last_peered":"2023-10-14T16:54:16.037509+0000","last_clean":"2023-10-14T16:54:16.037509+0000","last_became_active":"2023-10-14T16:54:15.047163+0000","last_became_peered":"2023-10-14T16:54:15.047163+0000","last_unstale":"2023-10-14T16:54:16.037509+0000","last_undegraded":"2023-10-14T16:54:16.037509+0000","last_fullsized":"2023-10-14T16:54:16.037509+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:53.265449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496344+0000","last_change":"2023-10-14T16:54:14.530038+0000","last_active":"2023-10-14T16:54:17.496344+0000","last_peered":"2023-10-14T16:54:17.496344+0000","last_clean":"2023-10-14T16:54:17.496344+0000","last_became_active":"2023-10-14T16:54:14.527414+0000","last_became_peered":"2023-10-14T16:54:14.527414+0000","last_unstale":"2023-10-14T16:54:17.496344+0000","last_undegraded":"2023-10-14T16:54:17.496344+0000","last_fullsized":"2023-10-14T16:54:17.496344+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:11.383980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447498+0000","last_change":"2023-10-14T16:54:15.062052+0000","last_active":"2023-10-14T16:54:16.447498+0000","last_peered":"2023-10-14T16:54:16.447498+0000","last_clean":"2023-10-14T16:54:16.447498+0000","last_became_active":"2023-10-14T16:54:15.061719+0000","last_became_peered":"2023-10-14T16:54:15.061719+0000","last_unstale":"2023-10-14T16:54:16.447498+0000","last_undegraded":"2023-10-14T16:54:16.447498+0000","last_fullsized":"2023-10-14T16:54:16.447498+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:24:05.513503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447719+0000","last_change":"2023-10-14T16:54:14.522023+0000","last_active":"2023-10-14T16:54:16.447719+0000","last_peered":"2023-10-14T16:54:16.447719+0000","last_clean":"2023-10-14T16:54:16.447719+0000","last_became_active":"2023-10-14T16:54:14.521553+0000","last_became_peered":"2023-10-14T16:54:14.521553+0000","last_unstale":"2023-10-14T16:54:16.447719+0000","last_undegraded":"2023-10-14T16:54:16.447719+0000","last_fullsized":"2023-10-14T16:54:16.447719+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:16.152378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042750+0000","last_change":"2023-10-14T16:54:15.048815+0000","last_active":"2023-10-14T16:54:16.042750+0000","last_peered":"2023-10-14T16:54:16.042750+0000","last_clean":"2023-10-14T16:54:16.042750+0000","last_became_active":"2023-10-14T16:54:15.048548+0000","last_became_peered":"2023-10-14T16:54:15.048548+0000","last_unstale":"2023-10-14T16:54:16.042750+0000","last_undegraded":"2023-10-14T16:54:16.042750+0000","last_fullsized":"2023-10-14T16:54:16.042750+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:17:21.807863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497976+0000","last_change":"2023-10-14T16:54:14.532449+0000","last_active":"2023-10-14T16:54:17.497976+0000","last_peered":"2023-10-14T16:54:17.497976+0000","last_clean":"2023-10-14T16:54:17.497976+0000","last_became_active":"2023-10-14T16:54:14.532149+0000","last_became_peered":"2023-10-14T16:54:14.532149+0000","last_unstale":"2023-10-14T16:54:17.497976+0000","last_undegraded":"2023-10-14T16:54:17.497976+0000","last_fullsized":"2023-10-14T16:54:17.497976+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:04:58.952522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041001+0000","last_change":"2023-10-14T16:54:15.061083+0000","last_active":"2023-10-14T16:54:16.041001+0000","last_peered":"2023-10-14T16:54:16.041001+0000","last_clean":"2023-10-14T16:54:16.041001+0000","last_became_active":"2023-10-14T16:54:15.060773+0000","last_became_peered":"2023-10-14T16:54:15.060773+0000","last_unstale":"2023-10-14T16:54:16.041001+0000","last_undegraded":"2023-10-14T16:54:16.041001+0000","last_fullsized":"2023-10-14T16:54:16.041001+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"62'3","reported_seq":15,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042309+0000","last_change":"2023-10-14T16:54:14.540813+0000","last_active":"2023-10-14T16:54:16.042309+0000","last_peered":"2023-10-14T16:54:16.042309+0000","last_clean":"2023-10-14T16:54:16.042309+0000","last_became_active":"2023-10-14T16:54:14.540503+0000","last_became_peered":"2023-10-14T16:54:14.540503+0000","last_unstale":"2023-10-14T16:54:16.042309+0000","last_undegraded":"2023-10-14T16:54:16.042309+0000","last_fullsized":"2023-10-14T16:54:16.042309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:39:23.483273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041572+0000","last_change":"2023-10-14T16:54:15.054664+0000","last_active":"2023-10-14T16:54:16.041572+0000","last_peered":"2023-10-14T16:54:16.041572+0000","last_clean":"2023-10-14T16:54:16.041572+0000","last_became_active":"2023-10-14T16:54:15.054390+0000","last_became_peered":"2023-10-14T16:54:15.054390+0000","last_unstale":"2023-10-14T16:54:16.041572+0000","last_undegraded":"2023-10-14T16:54:16.041572+0000","last_fullsized":"2023-10-14T16:54:16.041572+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:31.972755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"62'6","reported_seq":18,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190374+0000","last_change":"2023-10-14T16:54:14.531927+0000","last_active":"2023-10-14T16:54:16.190374+0000","last_peered":"2023-10-14T16:54:16.190374+0000","last_clean":"2023-10-14T16:54:16.190374+0000","last_became_active":"2023-10-14T16:54:14.531623+0000","last_became_peered":"2023-10-14T16:54:14.531623+0000","last_unstale":"2023-10-14T16:54:16.190374+0000","last_undegraded":"2023-10-14T16:54:16.190374+0000","last_fullsized":"2023-10-14T16:54:16.190374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:26.241831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039295+0000","last_change":"2023-10-14T16:54:15.062501+0000","last_active":"2023-10-14T16:54:16.039295+0000","last_peered":"2023-10-14T16:54:16.039295+0000","last_clean":"2023-10-14T16:54:16.039295+0000","last_became_active":"2023-10-14T16:54:15.062231+0000","last_became_peered":"2023-10-14T16:54:15.062231+0000","last_unstale":"2023-10-14T16:54:16.039295+0000","last_undegraded":"2023-10-14T16:54:16.039295+0000","last_fullsized":"2023-10-14T16:54:16.039295+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:03.980605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.13","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040867+0000","last_change":"2023-10-14T16:54:14.537549+0000","last_active":"2023-10-14T16:54:16.040867+0000","last_peered":"2023-10-14T16:54:16.040867+0000","last_clean":"2023-10-14T16:54:16.040867+0000","last_became_active":"2023-10-14T16:54:14.536132+0000","last_became_peered":"2023-10-14T16:54:14.536132+0000","last_unstale":"2023-10-14T16:54:16.040867+0000","last_undegraded":"2023-10-14T16:54:16.040867+0000","last_fullsized":"2023-10-14T16:54:16.040867+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:57:51.839736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497240+0000","last_change":"2023-10-14T16:54:15.055199+0000","last_active":"2023-10-14T16:54:17.497240+0000","last_peered":"2023-10-14T16:54:17.497240+0000","last_clean":"2023-10-14T16:54:17.497240+0000","last_became_active":"2023-10-14T16:54:15.054973+0000","last_became_peered":"2023-10-14T16:54:15.054973+0000","last_unstale":"2023-10-14T16:54:17.497240+0000","last_undegraded":"2023-10-14T16:54:17.497240+0000","last_fullsized":"2023-10-14T16:54:17.497240+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:33.172706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041615+0000","last_change":"2023-10-14T16:54:14.540108+0000","last_active":"2023-10-14T16:54:16.041615+0000","last_peered":"2023-10-14T16:54:16.041615+0000","last_clean":"2023-10-14T16:54:16.041615+0000","last_became_active":"2023-10-14T16:54:14.539816+0000","last_became_peered":"2023-10-14T16:54:14.539816+0000","last_unstale":"2023-10-14T16:54:16.041615+0000","last_undegraded":"2023-10-14T16:54:16.041615+0000","last_fullsized":"2023-10-14T16:54:16.041615+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:19.435571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040153+0000","last_change":"2023-10-14T16:54:15.054060+0000","last_active":"2023-10-14T16:54:16.040153+0000","last_peered":"2023-10-14T16:54:16.040153+0000","last_clean":"2023-10-14T16:54:16.040153+0000","last_became_active":"2023-10-14T16:54:15.053786+0000","last_became_peered":"2023-10-14T16:54:15.053786+0000","last_unstale":"2023-10-14T16:54:16.040153+0000","last_undegraded":"2023-10-14T16:54:16.040153+0000","last_fullsized":"2023-10-14T16:54:16.040153+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:22.568851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040390+0000","last_change":"2023-10-14T16:54:14.534505+0000","last_active":"2023-10-14T16:54:16.040390+0000","last_peered":"2023-10-14T16:54:16.040390+0000","last_clean":"2023-10-14T16:54:16.040390+0000","last_became_active":"2023-10-14T16:54:14.534188+0000","last_became_peered":"2023-10-14T16:54:14.534188+0000","last_unstale":"2023-10-14T16:54:16.040390+0000","last_undegraded":"2023-10-14T16:54:16.040390+0000","last_fullsized":"2023-10-14T16:54:16.040390+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:59.533841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041391+0000","last_change":"2023-10-14T16:54:15.046830+0000","last_active":"2023-10-14T16:54:16.041391+0000","last_peered":"2023-10-14T16:54:16.041391+0000","last_clean":"2023-10-14T16:54:16.041391+0000","last_became_active":"2023-10-14T16:54:15.046529+0000","last_became_peered":"2023-10-14T16:54:15.046529+0000","last_unstale":"2023-10-14T16:54:16.041391+0000","last_undegraded":"2023-10-14T16:54:16.041391+0000","last_fullsized":"2023-10-14T16:54:16.041391+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:02:15.718591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"62'8","reported_seq":20,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496602+0000","last_change":"2023-10-14T16:54:14.535595+0000","last_active":"2023-10-14T16:54:17.496602+0000","last_peered":"2023-10-14T16:54:17.496602+0000","last_clean":"2023-10-14T16:54:17.496602+0000","last_became_active":"2023-10-14T16:54:14.535287+0000","last_became_peered":"2023-10-14T16:54:14.535287+0000","last_unstale":"2023-10-14T16:54:17.496602+0000","last_undegraded":"2023-10-14T16:54:17.496602+0000","last_fullsized":"2023-10-14T16:54:17.496602+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:16.557551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190603+0000","last_change":"2023-10-14T16:54:15.041597+0000","last_active":"2023-10-14T16:54:16.190603+0000","last_peered":"2023-10-14T16:54:16.190603+0000","last_clean":"2023-10-14T16:54:16.190603+0000","last_became_active":"2023-10-14T16:54:15.041323+0000","last_became_peered":"2023-10-14T16:54:15.041323+0000","last_unstale":"2023-10-14T16:54:16.190603+0000","last_undegraded":"2023-10-14T16:54:16.190603+0000","last_fullsized":"2023-10-14T16:54:16.190603+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:04.036749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.190483+0000","last_change":"2023-10-14T16:54:14.537867+0000","last_active":"2023-10-14T16:54:16.190483+0000","last_peered":"2023-10-14T16:54:16.190483+0000","last_clean":"2023-10-14T16:54:16.190483+0000","last_became_active":"2023-10-14T16:54:14.537571+0000","last_became_peered":"2023-10-14T16:54:14.537571+0000","last_unstale":"2023-10-14T16:54:16.190483+0000","last_undegraded":"2023-10-14T16:54:16.190483+0000","last_fullsized":"2023-10-14T16:54:16.190483+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:10.454177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497908+0000","last_change":"2023-10-14T16:54:15.065740+0000","last_active":"2023-10-14T16:54:17.497908+0000","last_peered":"2023-10-14T16:54:17.497908+0000","last_clean":"2023-10-14T16:54:17.497908+0000","last_became_active":"2023-10-14T16:54:15.062742+0000","last_became_peered":"2023-10-14T16:54:15.062742+0000","last_unstale":"2023-10-14T16:54:17.497908+0000","last_undegraded":"2023-10-14T16:54:17.497908+0000","last_fullsized":"2023-10-14T16:54:17.497908+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:51:07.363554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447408+0000","last_change":"2023-10-14T16:54:14.522092+0000","last_active":"2023-10-14T16:54:16.447408+0000","last_peered":"2023-10-14T16:54:16.447408+0000","last_clean":"2023-10-14T16:54:16.447408+0000","last_became_active":"2023-10-14T16:54:14.521681+0000","last_became_peered":"2023-10-14T16:54:14.521681+0000","last_unstale":"2023-10-14T16:54:16.447408+0000","last_undegraded":"2023-10-14T16:54:16.447408+0000","last_fullsized":"2023-10-14T16:54:16.447408+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:09.280098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039108+0000","last_change":"2023-10-14T16:54:15.056759+0000","last_active":"2023-10-14T16:54:16.039108+0000","last_peered":"2023-10-14T16:54:16.039108+0000","last_clean":"2023-10-14T16:54:16.039108+0000","last_became_active":"2023-10-14T16:54:15.056498+0000","last_became_peered":"2023-10-14T16:54:15.056498+0000","last_unstale":"2023-10-14T16:54:16.039108+0000","last_undegraded":"2023-10-14T16:54:16.039108+0000","last_fullsized":"2023-10-14T16:54:16.039108+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:28:23.730093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"62'1","reported_seq":13,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037292+0000","last_change":"2023-10-14T16:54:14.531987+0000","last_active":"2023-10-14T16:54:16.037292+0000","last_peered":"2023-10-14T16:54:16.037292+0000","last_clean":"2023-10-14T16:54:16.037292+0000","last_became_active":"2023-10-14T16:54:14.531642+0000","last_became_peered":"2023-10-14T16:54:14.531642+0000","last_unstale":"2023-10-14T16:54:16.037292+0000","last_undegraded":"2023-10-14T16:54:16.037292+0000","last_fullsized":"2023-10-14T16:54:16.037292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:14.574599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039499+0000","last_change":"2023-10-14T16:54:15.054799+0000","last_active":"2023-10-14T16:54:16.039499+0000","last_peered":"2023-10-14T16:54:16.039499+0000","last_clean":"2023-10-14T16:54:16.039499+0000","last_became_active":"2023-10-14T16:54:15.053765+0000","last_became_peered":"2023-10-14T16:54:15.053765+0000","last_unstale":"2023-10-14T16:54:16.039499+0000","last_undegraded":"2023-10-14T16:54:16.039499+0000","last_fullsized":"2023-10-14T16:54:16.039499+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:57.184914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041026+0000","last_change":"2023-10-14T16:54:14.533253+0000","last_active":"2023-10-14T16:54:16.041026+0000","last_peered":"2023-10-14T16:54:16.041026+0000","last_clean":"2023-10-14T16:54:16.041026+0000","last_became_active":"2023-10-14T16:54:14.531802+0000","last_became_peered":"2023-10-14T16:54:14.531802+0000","last_unstale":"2023-10-14T16:54:16.041026+0000","last_undegraded":"2023-10-14T16:54:16.041026+0000","last_fullsized":"2023-10-14T16:54:16.041026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:52.590896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040624+0000","last_change":"2023-10-14T16:54:15.052111+0000","last_active":"2023-10-14T16:54:16.040624+0000","last_peered":"2023-10-14T16:54:16.040624+0000","last_clean":"2023-10-14T16:54:16.040624+0000","last_became_active":"2023-10-14T16:54:15.051640+0000","last_became_peered":"2023-10-14T16:54:15.051640+0000","last_unstale":"2023-10-14T16:54:16.040624+0000","last_undegraded":"2023-10-14T16:54:16.040624+0000","last_fullsized":"2023-10-14T16:54:16.040624+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:02:15.718591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041103+0000","last_change":"2023-10-14T16:54:14.533578+0000","last_active":"2023-10-14T16:54:16.041103+0000","last_peered":"2023-10-14T16:54:16.041103+0000","last_clean":"2023-10-14T16:54:16.041103+0000","last_became_active":"2023-10-14T16:54:14.533302+0000","last_became_peered":"2023-10-14T16:54:14.533302+0000","last_unstale":"2023-10-14T16:54:16.041103+0000","last_undegraded":"2023-10-14T16:54:16.041103+0000","last_fullsized":"2023-10-14T16:54:16.041103+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:39:23.483273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":9,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.059228+0000","last_change":"2023-10-14T16:54:15.055856+0000","last_active":"2023-10-14T16:54:15.059228+0000","last_peered":"2023-10-14T16:54:15.059228+0000","last_clean":"2023-10-14T16:54:15.059228+0000","last_became_active":"2023-10-14T16:54:15.053506+0000","last_became_peered":"2023-10-14T16:54:15.053506+0000","last_unstale":"2023-10-14T16:54:15.059228+0000","last_undegraded":"2023-10-14T16:54:15.059228+0000","last_fullsized":"2023-10-14T16:54:15.059228+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:33.055388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036447+0000","last_change":"2023-10-14T16:54:14.534219+0000","last_active":"2023-10-14T16:54:16.036447+0000","last_peered":"2023-10-14T16:54:16.036447+0000","last_clean":"2023-10-14T16:54:16.036447+0000","last_became_active":"2023-10-14T16:54:14.533915+0000","last_became_peered":"2023-10-14T16:54:14.533915+0000","last_unstale":"2023-10-14T16:54:16.036447+0000","last_undegraded":"2023-10-14T16:54:16.036447+0000","last_fullsized":"2023-10-14T16:54:16.036447+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:03.309664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.189853+0000","last_change":"2023-10-14T16:54:15.057372+0000","last_active":"2023-10-14T16:54:16.189853+0000","last_peered":"2023-10-14T16:54:16.189853+0000","last_clean":"2023-10-14T16:54:16.189853+0000","last_became_active":"2023-10-14T16:54:15.056733+0000","last_became_peered":"2023-10-14T16:54:15.056733+0000","last_unstale":"2023-10-14T16:54:16.189853+0000","last_undegraded":"2023-10-14T16:54:16.189853+0000","last_fullsized":"2023-10-14T16:54:16.189853+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:45:53.103204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496374+0000","last_change":"2023-10-14T16:54:14.532790+0000","last_active":"2023-10-14T16:54:17.496374+0000","last_peered":"2023-10-14T16:54:17.496374+0000","last_clean":"2023-10-14T16:54:17.496374+0000","last_became_active":"2023-10-14T16:54:14.532530+0000","last_became_peered":"2023-10-14T16:54:14.532530+0000","last_unstale":"2023-10-14T16:54:17.496374+0000","last_undegraded":"2023-10-14T16:54:17.496374+0000","last_fullsized":"2023-10-14T16:54:17.496374+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:52:24.265637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496434+0000","last_change":"2023-10-14T16:54:15.059647+0000","last_active":"2023-10-14T16:54:17.496434+0000","last_peered":"2023-10-14T16:54:17.496434+0000","last_clean":"2023-10-14T16:54:17.496434+0000","last_became_active":"2023-10-14T16:54:15.059337+0000","last_became_peered":"2023-10-14T16:54:15.059337+0000","last_unstale":"2023-10-14T16:54:17.496434+0000","last_undegraded":"2023-10-14T16:54:17.496434+0000","last_fullsized":"2023-10-14T16:54:17.496434+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:29.103133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041667+0000","last_change":"2023-10-14T16:54:14.540165+0000","last_active":"2023-10-14T16:54:16.041667+0000","last_peered":"2023-10-14T16:54:16.041667+0000","last_clean":"2023-10-14T16:54:16.041667+0000","last_became_active":"2023-10-14T16:54:14.539839+0000","last_became_peered":"2023-10-14T16:54:14.539839+0000","last_unstale":"2023-10-14T16:54:16.041667+0000","last_undegraded":"2023-10-14T16:54:16.041667+0000","last_fullsized":"2023-10-14T16:54:16.041667+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:17:17.389989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.039502+0000","last_change":"2023-10-14T16:54:15.063819+0000","last_active":"2023-10-14T16:54:16.039502+0000","last_peered":"2023-10-14T16:54:16.039502+0000","last_clean":"2023-10-14T16:54:16.039502+0000","last_became_active":"2023-10-14T16:54:15.063498+0000","last_became_peered":"2023-10-14T16:54:15.063498+0000","last_unstale":"2023-10-14T16:54:16.039502+0000","last_undegraded":"2023-10-14T16:54:16.039502+0000","last_fullsized":"2023-10-14T16:54:16.039502+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:33.195664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041220+0000","last_change":"2023-10-14T16:54:14.531687+0000","last_active":"2023-10-14T16:54:16.041220+0000","last_peered":"2023-10-14T16:54:16.041220+0000","last_clean":"2023-10-14T16:54:16.041220+0000","last_became_active":"2023-10-14T16:54:14.531351+0000","last_became_peered":"2023-10-14T16:54:14.531351+0000","last_unstale":"2023-10-14T16:54:16.041220+0000","last_undegraded":"2023-10-14T16:54:16.041220+0000","last_fullsized":"2023-10-14T16:54:16.041220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:17:17.389989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.497523+0000","last_change":"2023-10-14T16:54:15.056714+0000","last_active":"2023-10-14T16:54:17.497523+0000","last_peered":"2023-10-14T16:54:17.497523+0000","last_clean":"2023-10-14T16:54:17.497523+0000","last_became_active":"2023-10-14T16:54:15.056443+0000","last_became_peered":"2023-10-14T16:54:15.056443+0000","last_unstale":"2023-10-14T16:54:17.497523+0000","last_undegraded":"2023-10-14T16:54:17.497523+0000","last_fullsized":"2023-10-14T16:54:17.497523+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:33.195664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042446+0000","last_change":"2023-10-14T16:54:14.541365+0000","last_active":"2023-10-14T16:54:16.042446+0000","last_peered":"2023-10-14T16:54:16.042446+0000","last_clean":"2023-10-14T16:54:16.042446+0000","last_became_active":"2023-10-14T16:54:14.541067+0000","last_became_peered":"2023-10-14T16:54:14.541067+0000","last_unstale":"2023-10-14T16:54:16.042446+0000","last_undegraded":"2023-10-14T16:54:16.042446+0000","last_fullsized":"2023-10-14T16:54:16.042446+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:21.552589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.042363+0000","last_change":"2023-10-14T16:54:15.046680+0000","last_active":"2023-10-14T16:54:16.042363+0000","last_peered":"2023-10-14T16:54:16.042363+0000","last_clean":"2023-10-14T16:54:16.042363+0000","last_became_active":"2023-10-14T16:54:15.046389+0000","last_became_peered":"2023-10-14T16:54:15.046389+0000","last_unstale":"2023-10-14T16:54:16.042363+0000","last_undegraded":"2023-10-14T16:54:16.042363+0000","last_fullsized":"2023-10-14T16:54:16.042363+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.884141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.047903+0000","last_change":"2023-10-14T16:54:14.528607+0000","last_active":"2023-10-14T16:54:15.047903+0000","last_peered":"2023-10-14T16:54:15.047903+0000","last_clean":"2023-10-14T16:54:15.047903+0000","last_became_active":"2023-10-14T16:54:14.528337+0000","last_became_peered":"2023-10-14T16:54:14.528337+0000","last_unstale":"2023-10-14T16:54:15.047903+0000","last_undegraded":"2023-10-14T16:54:15.047903+0000","last_fullsized":"2023-10-14T16:54:15.047903+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:24:37.976867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040518+0000","last_change":"2023-10-14T16:54:15.065200+0000","last_active":"2023-10-14T16:54:16.040518+0000","last_peered":"2023-10-14T16:54:16.040518+0000","last_clean":"2023-10-14T16:54:16.040518+0000","last_became_active":"2023-10-14T16:54:15.064457+0000","last_became_peered":"2023-10-14T16:54:15.064457+0000","last_unstale":"2023-10-14T16:54:16.040518+0000","last_undegraded":"2023-10-14T16:54:16.040518+0000","last_fullsized":"2023-10-14T16:54:16.040518+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:20.589977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.041879+0000","last_change":"2023-10-14T16:54:14.541134+0000","last_active":"2023-10-14T16:54:16.041879+0000","last_peered":"2023-10-14T16:54:16.041879+0000","last_clean":"2023-10-14T16:54:16.041879+0000","last_became_active":"2023-10-14T16:54:14.540823+0000","last_became_peered":"2023-10-14T16:54:14.540823+0000","last_unstale":"2023-10-14T16:54:16.041879+0000","last_undegraded":"2023-10-14T16:54:16.041879+0000","last_fullsized":"2023-10-14T16:54:16.041879+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:26.755508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.040559+0000","last_change":"2023-10-14T16:54:15.060432+0000","last_active":"2023-10-14T16:54:16.040559+0000","last_peered":"2023-10-14T16:54:16.040559+0000","last_clean":"2023-10-14T16:54:16.040559+0000","last_became_active":"2023-10-14T16:54:15.059990+0000","last_became_peered":"2023-10-14T16:54:15.059990+0000","last_unstale":"2023-10-14T16:54:16.040559+0000","last_undegraded":"2023-10-14T16:54:16.040559+0000","last_fullsized":"2023-10-14T16:54:16.040559+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:22.844113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.447322+0000","last_change":"2023-10-14T16:54:14.528023+0000","last_active":"2023-10-14T16:54:16.447322+0000","last_peered":"2023-10-14T16:54:16.447322+0000","last_clean":"2023-10-14T16:54:16.447322+0000","last_became_active":"2023-10-14T16:54:14.527694+0000","last_became_peered":"2023-10-14T16:54:14.527694+0000","last_unstale":"2023-10-14T16:54:16.447322+0000","last_undegraded":"2023-10-14T16:54:16.447322+0000","last_fullsized":"2023-10-14T16:54:16.447322+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:23:15.354376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.036429+0000","last_change":"2023-10-14T16:54:15.046086+0000","last_active":"2023-10-14T16:54:16.036429+0000","last_peered":"2023-10-14T16:54:16.036429+0000","last_clean":"2023-10-14T16:54:16.036429+0000","last_became_active":"2023-10-14T16:54:15.045110+0000","last_became_peered":"2023-10-14T16:54:15.045110+0000","last_unstale":"2023-10-14T16:54:16.036429+0000","last_undegraded":"2023-10-14T16:54:16.036429+0000","last_fullsized":"2023-10-14T16:54:16.036429+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:14.582104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":11,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.053701+0000","last_change":"2023-10-14T16:54:14.526389+0000","last_active":"2023-10-14T16:54:15.053701+0000","last_peered":"2023-10-14T16:54:15.053701+0000","last_clean":"2023-10-14T16:54:15.053701+0000","last_became_active":"2023-10-14T16:54:14.526035+0000","last_became_peered":"2023-10-14T16:54:14.526035+0000","last_unstale":"2023-10-14T16:54:15.053701+0000","last_undegraded":"2023-10-14T16:54:15.053701+0000","last_fullsized":"2023-10-14T16:54:15.053701+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:09.280098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.446913+0000","last_change":"2023-10-14T16:54:15.057060+0000","last_active":"2023-10-14T16:54:16.446913+0000","last_peered":"2023-10-14T16:54:16.446913+0000","last_clean":"2023-10-14T16:54:16.446913+0000","last_became_active":"2023-10-14T16:54:15.056768+0000","last_became_peered":"2023-10-14T16:54:15.056768+0000","last_unstale":"2023-10-14T16:54:16.446913+0000","last_undegraded":"2023-10-14T16:54:16.446913+0000","last_fullsized":"2023-10-14T16:54:16.446913+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:33.350632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":12,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.496605+0000","last_change":"2023-10-14T16:54:14.534616+0000","last_active":"2023-10-14T16:54:17.496605+0000","last_peered":"2023-10-14T16:54:17.496605+0000","last_clean":"2023-10-14T16:54:17.496605+0000","last_became_active":"2023-10-14T16:54:14.533635+0000","last_became_peered":"2023-10-14T16:54:14.533635+0000","last_unstale":"2023-10-14T16:54:17.496605+0000","last_undegraded":"2023-10-14T16:54:17.496605+0000","last_fullsized":"2023-10-14T16:54:17.496605+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T16:55:08.394535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":11,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:17.498070+0000","last_change":"2023-10-14T16:54:15.058532+0000","last_active":"2023-10-14T16:54:17.498070+0000","last_peered":"2023-10-14T16:54:17.498070+0000","last_clean":"2023-10-14T16:54:17.498070+0000","last_became_active":"2023-10-14T16:54:15.058114+0000","last_became_peered":"2023-10-14T16:54:15.058114+0000","last_unstale":"2023-10-14T16:54:17.498070+0000","last_undegraded":"2023-10-14T16:54:17.498070+0000","last_fullsized":"2023-10-14T16:54:17.498070+0000","mapping_epoch":61,"log_start":"0'0","ondisk_log_start":"0'0","created":61,"last_epoch_clean":62,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:14.490797+0000","last_clean_scrub_stamp":"2023-10-14T16:54:14.490797+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:26:43.509435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"62'2","reported_seq":14,"reported_epoch":63,"state":"active+clean","last_fresh":"2023-10-14T16:54:16.037292+0000","last_change":"2023-10-14T16:54:14.537565+0000","last_active":"2023-10-14T16:54:16.037292+0000","last_peered":"2023-10-14T16:54:16.037292+0000","last_clean":"2023-10-14T16:54:16.037292+0000","last_became_active":"2023-10-14T16:54:14.537284+0000","last_became_peered":"2023-10-14T16:54:14.537284+0000","last_unstale":"2023-10-14T16:54:16.037292+0000","last_undegraded":"2023-10-14T16:54:16.037292+0000","last_fullsized":"2023-10-14T16:54:16.037292+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:54:13.474535+0000","last_clean_scrub_stamp":"2023-10-14T16:54:13.474535+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:59:10.016915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"50'139","reported_seq":286,"reported_epoch":62,"state":"active+clean","last_fresh":"2023-10-14T16:54:15.039740+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:54:15.039740+0000","last_peered":"2023-10-14T16:54:15.039740+0000","last_clean":"2023-10-14T16:54:15.039740+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:54:15.039740+0000","last_undegraded":"2023-10-14T16:54:15.039740+0000","last_fullsized":"2023-10-14T16:54:15.039740+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":139,"log_dups_size":0,"ondisk_log_size":139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":308,"num_read_kb":512,"num_write":199,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":139,"ondisk_log_size":139,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575019,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32504,"kb_used_data":1316,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339336,"statfs":{"total":96636764160,"available":96603480064,"internally_reserved":0,"allocated":1347584,"data_stored":942111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31914443},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1779999999999999,"15min":1.147},"min":{"1min":0.84399999999999997,"5min":0.77700000000000002,"15min":0.503},"max":{"1min":1.5840000000000001,"5min":1.897,"15min":1.897},"last":1.365},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.232,"15min":1.2150000000000001},"min":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.56499999999999995},"max":{"1min":1.669,"5min":2.117,"15min":2.262},"last":1.321}]},{"osd":1,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.254,"15min":1.2310000000000001},"min":{"1min":0.91300000000000003,"5min":0.86599999999999999,"15min":0.74399999999999999},"max":{"1min":1.375,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.254},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.258,"15min":1.2270000000000001},"min":{"1min":0.79200000000000004,"5min":0.751,"15min":0.60799999999999998},"max":{"1min":1.8799999999999999,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":1.0609999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.23,"15min":1.232},"min":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.71999999999999997},"max":{"1min":1.6259999999999999,"5min":1.772,"15min":2.2389999999999999},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.329,"15min":1.3160000000000001},"min":{"1min":1.002,"5min":0.93500000000000005,"15min":0.63300000000000001},"max":{"1min":1.577,"5min":2.1589999999999998,"15min":2.3500000000000001},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.4099999999999999,"15min":1.347},"min":{"1min":0.94099999999999995,"5min":0.87,"15min":0.77300000000000002},"max":{"1min":1.8660000000000001,"5min":1.9610000000000001,"15min":2.4159999999999999},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.4830000000000001,"15min":1.452},"min":{"1min":1.1259999999999999,"5min":1.0840000000000001,"15min":0.83099999999999996},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":2.5819999999999999},"last":1.5169999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3100000000000001,"5min":1.26,"15min":1.2749999999999999},"min":{"1min":0.82999999999999996,"5min":0.76500000000000001,"15min":0.59599999999999997},"max":{"1min":1.8120000000000001,"5min":2.0070000000000001,"15min":2.2959999999999998},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.3520000000000001,"15min":1.3160000000000001},"min":{"1min":0.95499999999999996,"5min":0.75,"15min":0.52400000000000002},"max":{"1min":1.718,"5min":1.885,"15min":2.1400000000000001},"last":1.45}]},{"osd":5,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.425,"15min":1.413},"min":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.63100000000000001},"max":{"1min":1.861,"5min":2.0390000000000001,"15min":3.5270000000000001},"last":1.137},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.4930000000000001,"15min":1.452},"min":{"1min":1.0940000000000001,"5min":0.95399999999999996,"15min":0.54200000000000004},"max":{"1min":1.9359999999999999,"5min":2.1299999999999999,"15min":2.1299999999999999},"last":1.4159999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:53:48 2023","interfaces":[{"interface":"back","average":{"1min":1.5609999999999999,"5min":1.5609999999999999,"15min":1.5389999999999999},"min":{"1min":1.044,"5min":1.044,"15min":0.82499999999999996},"max":{"1min":1.9890000000000001,"5min":2.1949999999999998,"15min":2.3220000000000001},"last":1.49},{"interface":"front","average":{"1min":1.498,"5min":1.496,"15min":1.4770000000000001},"min":{"1min":1.097,"5min":1.006,"15min":0.56200000000000006},"max":{"1min":2.036,"5min":2.0640000000000001,"15min":2.7040000000000002},"last":1.55}]}]},{"osd":6,"up_from":13,"seq":55834575018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31920,"kb_used_data":740,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339920,"statfs":{"total":96636764160,"available":96604078080,"internally_reserved":0,"allocated":757760,"data_stored":352325,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1359999999999999,"5min":1.1479999999999999,"15min":1.171},"min":{"1min":0.88300000000000001,"5min":0.78100000000000003,"15min":0.63700000000000001},"max":{"1min":1.627,"5min":1.655,"15min":2.339},"last":0.75900000000000001},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2430000000000001,"15min":1.214},"min":{"1min":0.88500000000000001,"5min":0.84099999999999997,"15min":0.68799999999999994},"max":{"1min":1.5589999999999999,"5min":1.752,"15min":3.2509999999999999},"last":1.264}]},{"osd":1,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.079,"5min":1.169,"15min":1.212},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.4159999999999999,"5min":1.857,"15min":4.3280000000000003},"last":1.353},{"interface":"front","average":{"1min":1.2250000000000001,"5min":1.2589999999999999,"15min":1.3100000000000001},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":1.5429999999999999,"5min":2.1309999999999998,"15min":3.431},"last":1.7230000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.3560000000000001,"15min":1.343},"min":{"1min":0.90200000000000002,"5min":0.85999999999999999,"15min":0.75900000000000001},"max":{"1min":1.4530000000000001,"5min":1.9470000000000001,"15min":2.6539999999999999},"last":1.421},{"interface":"front","average":{"1min":1.351,"5min":1.413,"15min":1.413},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6140000000000001,"5min":1.895,"15min":2.7200000000000002},"last":1.462}]},{"osd":3,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.377,"15min":1.3919999999999999},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.90100000000000002},"max":{"1min":1.867,"5min":1.867,"15min":2.8319999999999999},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.421,"5min":1.444,"15min":1.4179999999999999},"min":{"1min":1.0609999999999999,"5min":1.05,"15min":0.75600000000000001},"max":{"1min":1.79,"5min":1.827,"15min":3.6059999999999999},"last":1.6259999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.3080000000000001,"15min":1.3620000000000001},"min":{"1min":0.94399999999999995,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.899,"5min":1.899,"15min":3.0790000000000002},"last":1.849},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.4259999999999999,"15min":1.425},"min":{"1min":0.95899999999999996,"5min":0.88700000000000001,"15min":0.82999999999999996},"max":{"1min":1.946,"5min":2.0270000000000001,"15min":2.738},"last":1.524}]},{"osd":5,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.3959999999999999,"15min":1.3939999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.83299999999999996},"max":{"1min":2.0619999999999998,"5min":2.105,"15min":3.0110000000000001},"last":1.677},{"interface":"front","average":{"1min":1.611,"5min":1.538,"15min":1.573},"min":{"1min":1.0469999999999999,"5min":1.0469999999999999,"15min":0.96799999999999997},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":3.206},"last":1.9419999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.6020000000000001,"15min":1.589},"min":{"1min":1.3049999999999999,"5min":0.98899999999999999,"15min":0.90900000000000003},"max":{"1min":2.0619999999999998,"5min":2.0880000000000001,"15min":2.6349999999999998},"last":1.7789999999999999},{"interface":"front","average":{"1min":1.536,"5min":1.5389999999999999,"15min":1.5649999999999999},"min":{"1min":0.999,"5min":0.999,"15min":0.97099999999999997},"max":{"1min":1.9179999999999999,"5min":2.181,"15min":3.1320000000000001},"last":1.5800000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575019,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31916,"kb_used_data":736,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339924,"statfs":{"total":96636764160,"available":96604082176,"internally_reserved":0,"allocated":753664,"data_stored":351723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1060000000000001,"5min":1.097,"15min":1.129},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.52200000000000002},"max":{"1min":1.548,"5min":1.655,"15min":3.2000000000000002},"last":0.76800000000000002},{"interface":"front","average":{"1min":1.077,"5min":1.1339999999999999,"15min":1.161},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.57099999999999995},"max":{"1min":1.4930000000000001,"5min":1.665,"15min":4.101},"last":0.82199999999999995}]},{"osd":1,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.1539999999999999,"15min":1.212},"min":{"1min":0.88300000000000001,"5min":0.79500000000000004,"15min":0.63400000000000001},"max":{"1min":1.446,"5min":1.613,"15min":3.1659999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.135,"5min":1.1950000000000001,"15min":1.244},"min":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":1.6839999999999999,"5min":1.7609999999999999,"15min":2.2549999999999999},"last":0.72699999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1339999999999999,"5min":1.1890000000000001,"15min":1.2470000000000001},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.69099999999999995},"max":{"1min":1.4239999999999999,"5min":1.6439999999999999,"15min":4.3639999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.3260000000000001,"15min":1.353},"min":{"1min":0.97699999999999998,"5min":0.81499999999999995,"15min":0.79300000000000004},"max":{"1min":1.6619999999999999,"5min":2.1030000000000002,"15min":4.3339999999999996},"last":0.99399999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.298,"5min":1.3700000000000001,"15min":1.3979999999999999},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.8120000000000001,"5min":1.8120000000000001,"15min":2.431},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.425,"15min":1.4450000000000001},"min":{"1min":0.90800000000000003,"5min":0.90800000000000003,"15min":0.88600000000000001},"max":{"1min":1.7589999999999999,"5min":1.8939999999999999,"15min":3.6890000000000001},"last":0.91600000000000004}]},{"osd":4,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.242,"15min":1.3069999999999999},"min":{"1min":0.89700000000000002,"5min":0.79500000000000004,"15min":0.72199999999999998},"max":{"1min":1.8380000000000001,"5min":2.1890000000000001,"15min":2.577},"last":1.103},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.3140000000000001,"15min":1.383},"min":{"1min":0.86499999999999999,"5min":0.81299999999999994,"15min":0.66200000000000003},"max":{"1min":1.8360000000000001,"5min":1.887,"15min":2.6030000000000002},"last":1.2729999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.415,"15min":1.4379999999999999},"min":{"1min":1.052,"5min":0.745,"15min":0.68799999999999994},"max":{"1min":1.8799999999999999,"5min":2.2869999999999999,"15min":3.5739999999999998},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.516,"15min":1.581},"min":{"1min":1.097,"5min":1.083,"15min":0.81499999999999995},"max":{"1min":1.7989999999999999,"5min":2.2519999999999998,"15min":3.2949999999999999},"last":1.2270000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:53:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.502,"15min":1.548},"min":{"1min":1.075,"5min":0.94499999999999995,"15min":0.72299999999999998},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":3.0289999999999999},"last":1.4630000000000001},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.53,"15min":1.55},"min":{"1min":1.3149999999999999,"5min":0.91700000000000004,"15min":0.627},"max":{"1min":1.9730000000000001,"5min":2.3980000000000001,"15min":3.7869999999999999},"last":1.329}]}]},{"osd":4,"up_from":13,"seq":55834575019,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58304,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313536,"statfs":{"total":96636764160,"available":96577060864,"internally_reserved":0,"allocated":761856,"data_stored":353265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.165,"15min":1.181},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.61399999999999999},"max":{"1min":1.7050000000000001,"5min":1.7709999999999999,"15min":4.3810000000000002},"last":0.66500000000000004},{"interface":"front","average":{"1min":1.2529999999999999,"5min":1.2370000000000001,"15min":1.254},"min":{"1min":0.83399999999999996,"5min":0.80900000000000005,"15min":0.69799999999999995},"max":{"1min":1.671,"5min":1.7390000000000001,"15min":4.4059999999999997},"last":0.80600000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.2509999999999999,"5min":1.284,"15min":1.258},"min":{"1min":0.873,"5min":0.81699999999999995,"15min":0.60699999999999998},"max":{"1min":1.8640000000000001,"5min":1.8640000000000001,"15min":2.8639999999999999},"last":1.083},{"interface":"front","average":{"1min":1.2,"5min":1.2470000000000001,"15min":1.2250000000000001},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.754},"max":{"1min":1.6180000000000001,"5min":1.7769999999999999,"15min":1.7769999999999999},"last":1.5780000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.2390000000000001,"15min":1.2330000000000001},"min":{"1min":0.88700000000000001,"5min":0.84599999999999997,"15min":0.65000000000000002},"max":{"1min":1.802,"5min":1.802,"15min":1.8460000000000001},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3919999999999999,"15min":1.371},"min":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.72799999999999998},"max":{"1min":1.744,"5min":1.8140000000000001,"15min":2.7759999999999998},"last":1.238}]},{"osd":3,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.284,"5min":1.335,"15min":1.306},"min":{"1min":0.98099999999999998,"5min":0.97499999999999998,"15min":0.752},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.048},"last":1.4490000000000001},{"interface":"front","average":{"1min":1.4370000000000001,"5min":1.4199999999999999,"15min":1.4099999999999999},"min":{"1min":0.89600000000000002,"5min":0.89600000000000002,"15min":0.72199999999999998},"max":{"1min":1.952,"5min":1.952,"15min":2.085},"last":1.0009999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.349,"15min":1.3720000000000001},"min":{"1min":1.0549999999999999,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":1.855,"5min":1.855,"15min":2.2320000000000002},"last":1.504},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.359,"15min":1.371},"min":{"1min":0.90500000000000003,"5min":0.84199999999999997,"15min":0.80400000000000005},"max":{"1min":1.704,"5min":1.7909999999999999,"15min":2.153},"last":1.746}]},{"osd":6,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3959999999999999,"15min":1.421},"min":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.5249999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.542,"15min":1.5529999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":1.917,"5min":1.917,"15min":2.6539999999999999},"last":1.3799999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:39 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.4750000000000001,"15min":1.502},"min":{"1min":1.097,"5min":0.94299999999999995,"15min":0.94199999999999995},"max":{"1min":1.7070000000000001,"5min":2.1269999999999998,"15min":2.2389999999999999},"last":1.9359999999999999},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5600000000000001,"15min":1.579},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"max":{"1min":2.2549999999999999,"5min":2.2549999999999999,"15min":2.3999999999999999},"last":1.829}]}]},{"osd":3,"up_from":13,"seq":55834575019,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40244,"kb_used_data":744,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331596,"statfs":{"total":96636764160,"available":96595554304,"internally_reserved":0,"allocated":761856,"data_stored":352291,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":0.96499999999999997,"15min":0.93500000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.36599999999999999},"max":{"1min":1.4239999999999999,"5min":1.5009999999999999,"15min":1.526},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.1719999999999999,"15min":1.1180000000000001},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.46500000000000002},"max":{"1min":1.417,"5min":1.8799999999999999,"15min":1.8799999999999999},"last":0.442}]},{"osd":1,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.95199999999999996,"15min":0.93000000000000005},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.33600000000000002},"max":{"1min":1.206,"5min":1.597,"15min":1.7889999999999999},"last":1.0960000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.1240000000000001,"15min":1.115},"min":{"1min":0.438,"5min":0.438,"15min":0.373},"max":{"1min":1.3120000000000001,"5min":1.8129999999999999,"15min":2.294},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3360000000000001,"5min":1.4490000000000001,"15min":1.355},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.51600000000000001},"max":{"1min":1.6739999999999999,"5min":2.0609999999999999,"15min":2.6819999999999999},"last":1.018},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.3200000000000001,"15min":1.2629999999999999},"min":{"1min":0.496,"5min":0.496,"15min":0.496},"max":{"1min":1.601,"5min":2.0960000000000001,"15min":2.77},"last":0.58099999999999996}]},{"osd":4,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.276,"5min":1.3620000000000001,"15min":1.3740000000000001},"min":{"1min":1.0469999999999999,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.569,"5min":1.8480000000000001,"15min":2.8239999999999998},"last":3.4350000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.405,"15min":1.4410000000000001},"min":{"1min":0.92600000000000005,"5min":0.92500000000000004,"15min":0.749},"max":{"1min":1.7350000000000001,"5min":1.8740000000000001,"15min":3.9540000000000002},"last":1.3169999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.2749999999999999,"5min":1.403,"15min":1.4159999999999999},"min":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.77500000000000002},"max":{"1min":1.544,"5min":1.9950000000000001,"15min":3.0190000000000001},"last":2.516},{"interface":"front","average":{"1min":1.3999999999999999,"5min":1.4450000000000001,"15min":1.462},"min":{"1min":1.1839999999999999,"5min":1.042,"15min":0.77400000000000002},"max":{"1min":1.556,"5min":2.3199999999999998,"15min":3.2490000000000001},"last":3.7410000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.4530000000000001,"15min":1.4830000000000001},"min":{"1min":0.92500000000000004,"5min":0.92500000000000004,"15min":0.65100000000000002},"max":{"1min":1.655,"5min":2.0579999999999998,"15min":4.0570000000000004},"last":1.246},{"interface":"front","average":{"1min":1.462,"5min":1.544,"15min":1.573},"min":{"1min":0.94499999999999995,"5min":0.94499999999999995,"15min":0.79200000000000004},"max":{"1min":1.7490000000000001,"5min":2.1059999999999999,"15min":4.1900000000000004},"last":2.1259999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:53:37 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.5109999999999999,"15min":1.5089999999999999},"min":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.745},"max":{"1min":1.681,"5min":2.419,"15min":2.9900000000000002},"last":2.871},{"interface":"front","average":{"1min":1.462,"5min":1.556,"15min":1.5569999999999999},"min":{"1min":1.137,"5min":0.95799999999999996,"15min":0.82899999999999996},"max":{"1min":1.8149999999999999,"5min":2.464,"15min":2.9569999999999999},"last":2.907}]}]},{"osd":2,"up_from":13,"seq":55834575019,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40256,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331584,"statfs":{"total":96636764160,"available":96595542016,"internally_reserved":0,"allocated":765952,"data_stored":353287,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91400000000000003,"15min":0.90300000000000002},"min":{"1min":0.63400000000000001,"5min":0.63400000000000001,"15min":0.34300000000000003},"max":{"1min":1.4950000000000001,"5min":1.659,"15min":1.849},"last":0.78300000000000003},{"interface":"front","average":{"1min":1.0109999999999999,"5min":1.0920000000000001,"15min":1.0489999999999999},"min":{"1min":0.625,"5min":0.625,"15min":0.45900000000000002},"max":{"1min":1.643,"5min":1.6519999999999999,"15min":3.2149999999999999},"last":1.048}]},{"osd":1,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":0.93400000000000005,"5min":0.995,"15min":0.97499999999999998},"min":{"1min":0.439,"5min":0.439,"15min":0.42899999999999999},"max":{"1min":1.415,"5min":1.518,"15min":2.0419999999999998},"last":0.90000000000000002},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2050000000000001,"15min":1.1739999999999999},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.42099999999999999},"max":{"1min":1.7090000000000001,"5min":2.069,"15min":2.2890000000000001},"last":0.97799999999999998}]},{"osd":3,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2669999999999999,"15min":1.2250000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46100000000000002},"max":{"1min":1.5800000000000001,"5min":1.8069999999999999,"15min":2.1349999999999998},"last":1.9430000000000001},{"interface":"front","average":{"1min":1.3160000000000001,"5min":1.3420000000000001,"15min":1.3140000000000001},"min":{"1min":0.745,"5min":0.745,"15min":0.628},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.506}]},{"osd":4,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4419999999999999,"5min":1.4490000000000001,"15min":1.423},"min":{"1min":0.82399999999999995,"5min":0.82399999999999995,"15min":0.73099999999999998},"max":{"1min":1.95,"5min":1.95,"15min":2.6259999999999999},"last":1.669},{"interface":"front","average":{"1min":1.391,"5min":1.4199999999999999,"15min":1.381},"min":{"1min":0.93999999999999995,"5min":0.91500000000000004,"15min":0.67900000000000005},"max":{"1min":1.6839999999999999,"5min":2.0750000000000002,"15min":3.0459999999999998},"last":1.887}]},{"osd":5,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.377,"5min":1.4299999999999999,"15min":1.407},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.65500000000000003},"max":{"1min":1.875,"5min":2.0019999999999998,"15min":2.2389999999999999},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.5529999999999999,"15min":1.534},"min":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.80000000000000004},"max":{"1min":2.2730000000000001,"5min":2.2730000000000001,"15min":2.883},"last":1.784}]},{"osd":6,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4930000000000001,"15min":1.472},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.83499999999999996},"max":{"1min":1.929,"5min":2.0289999999999999,"15min":3.1200000000000001},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.5640000000000001,"15min":1.54},"min":{"1min":1.002,"5min":1.002,"15min":0.70699999999999996},"max":{"1min":2.02,"5min":2.105,"15min":2.3420000000000001},"last":1.585}]},{"osd":7,"last update":"Sat Oct 14 16:53:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.573,"15min":1.556},"min":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.77400000000000002},"max":{"1min":1.964,"5min":2.109,"15min":3.169},"last":1.444},{"interface":"front","average":{"1min":1.516,"5min":1.6100000000000001,"15min":1.609},"min":{"1min":0.96399999999999997,"5min":0.96399999999999997,"15min":0.755},"max":{"1min":1.9590000000000001,"5min":2.165,"15min":2.996},"last":1.8300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58272,"kb_used_data":724,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313568,"statfs":{"total":96636764160,"available":96577093632,"internally_reserved":0,"allocated":741376,"data_stored":345379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.77800000000000002,"5min":0.93400000000000005,"15min":0.90000000000000002},"min":{"1min":0.496,"5min":0.496,"15min":0.35999999999999999},"max":{"1min":1.3839999999999999,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.80900000000000005,"5min":1.0640000000000001,"15min":1.0509999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.1930000000000001,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.82799999999999996}]},{"osd":2,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":0.80200000000000005,"5min":0.96499999999999997,"15min":0.94699999999999995},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.2250000000000001,"5min":1.758,"15min":1.758},"last":0.92500000000000004},{"interface":"front","average":{"1min":0.96699999999999997,"5min":1.1679999999999999,"15min":1.169},"min":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.45200000000000001},"max":{"1min":1.726,"5min":2.0150000000000001,"15min":2.1499999999999999},"last":0.95499999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0569999999999999,"5min":1.3080000000000001,"15min":1.2929999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.40999999999999998},"max":{"1min":1.885,"5min":2.198,"15min":2.5339999999999998},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.004,"5min":1.2270000000000001,"15min":1.232},"min":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.434},"max":{"1min":1.4039999999999999,"5min":1.8680000000000001,"15min":4.6799999999999997},"last":1.2110000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.425,"15min":1.4419999999999999},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.83799999999999997},"max":{"1min":1.78,"5min":2.0139999999999998,"15min":4.5789999999999997},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.119,"5min":1.323,"15min":1.3129999999999999},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.74,"5min":1.9279999999999999,"15min":2.0840000000000001},"last":0.98999999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3280000000000001,"5min":1.3919999999999999,"15min":1.379},"min":{"1min":0.89000000000000001,"5min":0.89000000000000001,"15min":0.84099999999999997},"max":{"1min":1.843,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.341,"5min":1.502,"15min":1.4990000000000001},"min":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":2.028,"5min":2.302,"15min":2.4980000000000002},"last":1.3740000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.5129999999999999,"15min":1.5369999999999999},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.872},"max":{"1min":1.698,"5min":2.3479999999999999,"15min":4.6360000000000001},"last":1.625},{"interface":"front","average":{"1min":1.327,"5min":1.4690000000000001,"15min":1.486},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.9019999999999999,"5min":2.1070000000000002,"15min":2.2970000000000002},"last":1.1639999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:54:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4219999999999999,"5min":1.589,"15min":1.591},"min":{"1min":0.90200000000000002,"5min":0.90200000000000002,"15min":0.79900000000000004},"max":{"1min":2.1400000000000001,"5min":2.411,"15min":2.452},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"max":{"1min":1.8819999999999999,"5min":2.6600000000000001,"15min":2.6600000000000001},"last":1.573}]}]},{"osd":0,"up_from":13,"seq":55834575018,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36644,"kb_used_data":1304,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335196,"statfs":{"total":96636764160,"available":96599240704,"internally_reserved":0,"allocated":1335296,"data_stored":935711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.78800000000000003,"5min":0.89200000000000002,"15min":0.88100000000000001},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.2050000000000001,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.79300000000000004},{"interface":"front","average":{"1min":1.0880000000000001,"5min":1.141,"15min":1.085},"min":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.46999999999999997},"max":{"1min":1.6679999999999999,"5min":1.95,"15min":1.95},"last":1.3069999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":0.97099999999999997,"5min":1.0820000000000001,"15min":1.0740000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":1.6359999999999999,"5min":1.962,"15min":1.962},"last":1.034},{"interface":"front","average":{"1min":1.135,"5min":1.2350000000000001,"15min":1.1870000000000001},"min":{"1min":0.874,"5min":0.86099999999999999,"15min":0.42699999999999999},"max":{"1min":1.667,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.111}]},{"osd":3,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.1160000000000001,"5min":1.288,"15min":1.254},"min":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.53300000000000003},"max":{"1min":2.2010000000000001,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.7390000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.3959999999999999,"15min":1.333},"min":{"1min":0.66900000000000004,"5min":0.66900000000000004,"15min":0.51700000000000002},"max":{"1min":1.843,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.8120000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.2969999999999999,"15min":1.2569999999999999},"min":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.63600000000000001},"max":{"1min":1.5680000000000001,"5min":1.8440000000000001,"15min":1.8440000000000001},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.429,"15min":1.4350000000000001},"min":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.77800000000000002},"max":{"1min":1.873,"5min":2.101,"15min":2.3460000000000001},"last":1.879}]},{"osd":5,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.3109999999999999,"15min":1.323},"min":{"1min":0.85499999999999998,"5min":0.85499999999999998,"15min":0.66100000000000003},"max":{"1min":1.661,"5min":1.8720000000000001,"15min":2.0539999999999998},"last":1.252},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.5409999999999999,"15min":1.5089999999999999},"min":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.0230000000000001,"5min":2.395,"15min":2.395},"last":1.474}]},{"osd":6,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.5429999999999999,"5min":1.635,"15min":1.577},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":2.0870000000000002,"5min":2.492,"15min":2.492},"last":1.605},{"interface":"front","average":{"1min":1.337,"5min":1.417,"15min":1.391},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.77000000000000002},"max":{"1min":1.696,"5min":1.9790000000000001,"15min":2.1850000000000001},"last":1.671}]},{"osd":7,"last update":"Sat Oct 14 16:53:41 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.552,"15min":1.534},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83399999999999996},"max":{"1min":2.3069999999999999,"5min":2.3410000000000002,"15min":2.5219999999999998},"last":2.1070000000000002},{"interface":"front","average":{"1min":1.456,"5min":1.5429999999999999,"15min":1.5169999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":2.02,"5min":2.415,"15min":2.415},"last":1.8520000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:54:20.578 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:54:20.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:54:20.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.852+0000 7f13f3fff640 1 Processor -- start 2023-10-14T16:54:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.852+0000 7f13f3fff640 1 -- start start 2023-10-14T16:54:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13f40ca1a0 0x7f13f40cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f13f406f940 0x7f13f4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f3fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13f4161760 con 0x7f13f40ccb90 2023-10-14T16:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f40cd010 con 0x7f13f40ca1a0 2023-10-14T16:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f3fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13f40cd2f0 con 0x7f13f406f940 2023-10-14T16:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13f40ca1a0 0x7f13f40cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f2ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13f40ca1a0 0x7f13f40cc650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36560/0 (socket says 172.21.15.29:36560) 2023-10-14T16:54:20.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.854+0000 7f13f2ffd640 1 -- 172.21.15.29:0/1006780993 learned_addr learned my addr 172.21.15.29:0/1006780993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:20.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.855+0000 7f13f37fe640 1 --2- 172.21.15.29:0/1006780993 >> v2:172.21.15.186:3300/0 conn(0x7f13f406f940 0x7f13f4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.855+0000 7f13f2ffd640 1 -- 172.21.15.29:0/1006780993 >> v2:172.21.15.186:3300/0 conn(0x7f13f406f940 msgr2=0x7f13f4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.855+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/1006780993 >> v2:172.21.15.186:3300/0 conn(0x7f13f406f940 0x7f13f4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.855+0000 7f13f2ffd640 1 -- 172.21.15.29:0/1006780993 >> v1:172.21.15.186:6789/0 conn(0x7f13f40ccb90 legacy=0x7f13f406f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.855+0000 7f13f2ffd640 1 -- 172.21.15.29:0/1006780993 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f40cd5d0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.856+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/1006780993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40cc650 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f13f40addd0 tx=0x7f13e402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91f803361b8d897 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.856+0000 7f13f1ffb640 1 -- 172.21.15.29:0/1006780993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4030860 con 0x7f13f40ca1a0 2023-10-14T16:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f1ffb640 1 -- 172.21.15.29:0/1006780993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e40309c0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f1ffb640 1 -- 172.21.15.29:0/1006780993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13e4030d50 con 0x7f13f40ca1a0 2023-10-14T16:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 -- 172.21.15.29:0/1006780993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 msgr2=0x7f13f40cc650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 --2- 172.21.15.29:0/1006780993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40cc650 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f13f40addd0 tx=0x7f13e402b9a0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 -- 172.21.15.29:0/1006780993 shutdown_connections 2023-10-14T16:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 --2- 172.21.15.29:0/1006780993 >> v2:172.21.15.186:3300/0 conn(0x7f13f406f940 0x7f13f4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 --2- 172.21.15.29:0/1006780993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40cc650 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 -- 172.21.15.29:0/1006780993 >> 172.21.15.29:0/1006780993 conn(0x7f13f40ce6a0 msgr2=0x7f13f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:20.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.857+0000 7f13f3fff640 1 -- 172.21.15.29:0/1006780993 shutdown_connections 2023-10-14T16:54:20.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.858+0000 7f13f3fff640 1 -- 172.21.15.29:0/1006780993 wait complete. 2023-10-14T16:54:20.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.859+0000 7f13f3fff640 1 Processor -- start 2023-10-14T16:54:20.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.859+0000 7f13f3fff640 1 -- start start 2023-10-14T16:54:20.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 0x7f13f40c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 0x7f13f40c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40c4220 con 0x7f13f40ca1a0 2023-10-14T16:54:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40c3350 con 0x7f13f406f940 2023-10-14T16:54:20.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.860+0000 7f13f3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13f40b14c0 con 0x7f13f40ccb90 2023-10-14T16:54:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36568/0 (socket says 172.21.15.29:36568) 2023-10-14T16:54:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 -- 172.21.15.29:0/264327441 learned_addr learned my addr 172.21.15.29:0/264327441 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:20.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 0x7f13f40c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f37fe640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 0x7f13f40c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 -- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 msgr2=0x7f13f40c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 0x7f13f40c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 -- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 msgr2=0x7f13f40c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 0x7f13f40c4640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f27fc640 1 -- 172.21.15.29:0/264327441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e40094d0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f37fe640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 0x7f13f40c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.861+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 0x7f13f40c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:20.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13f27fc640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00ba50 tx=0x7f13dc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc00cbe0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc00cd80 con 0x7f13f40ca1a0 2023-10-14T16:54:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13dc014680 con 0x7f13f40ca1a0 2023-10-14T16:54:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13f40c35a0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.862+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13f40c3b60 con 0x7f13f40ca1a0 2023-10-14T16:54:20.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.863+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13f404e9b0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.864+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13dc0147e0 con 0x7f13f40ca1a0 2023-10-14T16:54:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.865+0000 7f13eb7fe640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:20.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.865+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f13dc089670 con 0x7f13f40ca1a0 2023-10-14T16:54:20.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.865+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:20.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.866+0000 7f13f2ffd640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 0x7f13d406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13e4036000 tx=0x7f13e4008030 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:20.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:20.870+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13dc054690 con 0x7f13f40ca1a0 2023-10-14T16:54:21.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.142+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f13f40c5410 con 0x7f13f40ca1a0 2023-10-14T16:54:21.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.147+0000 7f13eb7fe640 1 -- 172.21.15.29:0/264327441 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v12) v1 ==== 190+0+147 (secure 0 0 0) 0x7f13dc057cb0 con 0x7f13f40ca1a0 2023-10-14T16:54:21.147 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:54:21.148 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:54:21.148 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:54:21.148 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:54:21.148 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:54:21.149 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:54:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 msgr2=0x7f13d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 0x7f13d406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f13e4036000 tx=0x7f13e4008030 comp rx=0 tx=0).stop 2023-10-14T16:54:21.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 msgr2=0x7f13f40c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f13dc00ba50 tx=0x7f13dc00bf10 comp rx=0 tx=0).stop 2023-10-14T16:54:21.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 shutdown_connections 2023-10-14T16:54:21.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f13d406af90 0x7f13d406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13f40ccb90 0x7f13f40c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13f40ca1a0 0x7f13f40c83b0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.149+0000 7f13f3fff640 1 --2- 172.21.15.29:0/264327441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13f406f940 0x7f13f40c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.150+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 >> 172.21.15.29:0/264327441 conn(0x7f13f40ce6a0 msgr2=0x7f13f4163940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.150+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 shutdown_connections 2023-10-14T16:54:21.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.150+0000 7f13f3fff640 1 -- 172.21.15.29:0/264327441 wait complete. 2023-10-14T16:54:21.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:54:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.398+0000 7f35ced78640 1 Processor -- start 2023-10-14T16:54:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.398+0000 7f35ced78640 1 -- start start 2023-10-14T16:54:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.400+0000 7f35ced78640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35c80adb50 0x7f35c806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.400+0000 7f35ced78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c8070fd0 0x7f35c80b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.400+0000 7f35ced78640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35c8071450 con 0x7f35c80adf30 2023-10-14T16:54:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.400+0000 7f35ced78640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35c80cc600 con 0x7f35c80adb50 2023-10-14T16:54:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.400+0000 7f35ced78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c80cc8e0 con 0x7f35c8070fd0 2023-10-14T16:54:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35c80adb50 0x7f35c806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35c80adb50 0x7f35c806f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45292/0 (socket says 172.21.15.29:45292) 2023-10-14T16:54:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 -- 172.21.15.29:0/214707950 learned_addr learned my addr 172.21.15.29:0/214707950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35c7fff640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c8070fd0 0x7f35c80b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 -- 172.21.15.29:0/214707950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c8070fd0 msgr2=0x7f35c80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c8070fd0 0x7f35c80b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 -- 172.21.15.29:0/214707950 >> v1:172.21.15.186:6789/0 conn(0x7f35c80adf30 legacy=0x7f35c8164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.401+0000 7f35ccb14640 1 -- 172.21.15.29:0/214707950 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c80ccbc0 con 0x7f35c80adb50 2023-10-14T16:54:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.403+0000 7f35ccb14640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c80adb50 0x7f35c806f550 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb817b6ed59b3ecb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.403+0000 7f35c77fe640 1 -- 172.21.15.29:0/214707950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc031640 con 0x7f35c80adb50 2023-10-14T16:54:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35c77fe640 1 -- 172.21.15.29:0/214707950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc0317a0 con 0x7f35c80adb50 2023-10-14T16:54:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35c77fe640 1 -- 172.21.15.29:0/214707950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc031b30 con 0x7f35c80adb50 2023-10-14T16:54:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 -- 172.21.15.29:0/214707950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c80adb50 msgr2=0x7f35c806f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c80adb50 0x7f35c806f550 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 -- 172.21.15.29:0/214707950 shutdown_connections 2023-10-14T16:54:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c8070fd0 0x7f35c80b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 --2- 172.21.15.29:0/214707950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c80adb50 0x7f35c806f550 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.404+0000 7f35ced78640 1 -- 172.21.15.29:0/214707950 >> 172.21.15.29:0/214707950 conn(0x7f35c80cb410 msgr2=0x7f35c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.405+0000 7f35ced78640 1 -- 172.21.15.29:0/214707950 shutdown_connections 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.405+0000 7f35ced78640 1 -- 172.21.15.29:0/214707950 wait complete. 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.406+0000 7f35ced78640 1 Processor -- start 2023-10-14T16:54:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.407+0000 7f35ced78640 1 -- start start 2023-10-14T16:54:21.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 0x7f35c8165080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 0x7f35c8160fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c8161500 con 0x7f35c8070fd0 2023-10-14T16:54:21.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c8163fe0 con 0x7f35c80adb50 2023-10-14T16:54:21.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ced78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c81642c0 con 0x7f35c80adf30 2023-10-14T16:54:21.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45300/0 (socket says 172.21.15.29:45300) 2023-10-14T16:54:21.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 -- 172.21.15.29:0/4159970433 learned_addr learned my addr 172.21.15.29:0/4159970433 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:21.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35c7fff640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 0x7f35c8165080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35cd315640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 0x7f35c8160fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 -- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 msgr2=0x7f35c8160fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 0x7f35c8160fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 -- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 msgr2=0x7f35c8165080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.408+0000 7f35ccb14640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 0x7f35c8165080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35ccb14640 1 -- 172.21.15.29:0/4159970433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc0094d0 con 0x7f35c8070fd0 2023-10-14T16:54:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35c7fff640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 0x7f35c8165080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35cd315640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 0x7f35c8160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35ccb14640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc03ac60 con 0x7f35c8070fd0 2023-10-14T16:54:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35bc03adc0 con 0x7f35c8070fd0 2023-10-14T16:54:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35bc044630 con 0x7f35c8070fd0 2023-10-14T16:54:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c81645a0 con 0x7f35c8070fd0 2023-10-14T16:54:21.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.409+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c8164b60 con 0x7f35c8070fd0 2023-10-14T16:54:21.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.411+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c804e160 con 0x7f35c8070fd0 2023-10-14T16:54:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.412+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35bc02f050 con 0x7f35c8070fd0 2023-10-14T16:54:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.412+0000 7f35c57fa640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 0x7f35ac06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.413+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f35bc0ab770 con 0x7f35c8070fd0 2023-10-14T16:54:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.413+0000 7f35c7fff640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 0x7f35ac06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:21.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.414+0000 7f35c7fff640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 0x7f35ac06d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f35b4006fd0 tx=0x7f35b4008040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:21.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.417+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35bc0766b0 con 0x7f35c8070fd0 2023-10-14T16:54:21.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.652+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35c80b4910 con 0x7f35c8070fd0 2023-10-14T16:54:21.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.653+0000 7f35c57fa640 1 -- 172.21.15.29:0/4159970433 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f35bc079cd0 con 0x7f35c8070fd0 2023-10-14T16:54:21.654 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:21.654 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:54:21.655 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:54:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 msgr2=0x7f35ac06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 0x7f35ac06d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f35b4006fd0 tx=0x7f35b4008040 comp rx=0 tx=0).stop 2023-10-14T16:54:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 msgr2=0x7f35c8163a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:21.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f35bc009870 tx=0x7f35bc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 shutdown_connections 2023-10-14T16:54:21.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f35ac06afe0 0x7f35ac06d4a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35c80adf30 0x7f35c8160fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35c80adb50 0x7f35c8165080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 --2- 172.21.15.29:0/4159970433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35c8070fd0 0x7f35c8163a00 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 >> 172.21.15.29:0/4159970433 conn(0x7f35c80cb410 msgr2=0x7f35c80afbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 shutdown_connections 2023-10-14T16:54:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:21.656+0000 7f35ced78640 1 -- 172.21.15.29:0/4159970433 wait complete. 2023-10-14T16:54:22.078 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:54:14.976595+0000', 'modified': '2023-10-14T16:54:18.073045+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5236}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5236': {'gid': 5236, 'name': 'd', 'rank': 0, 'incarnation': 20, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.186:6835/2940163923', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 2940163923}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 2940163923}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [7], 'metadata_pool': 6, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:54:22.079 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:54:22.079 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T16:54:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.321+0000 7fb04ae2e640 1 Processor -- start 2023-10-14T16:54:22.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.321+0000 7fb04ae2e640 1 -- start start 2023-10-14T16:54:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.323+0000 7fb04ae2e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb04415cd70 0x7fb0440d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.323+0000 7fb04ae2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 0x7fb0440cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.323+0000 7fb04ae2e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb04415d150 con 0x7fb0440d0be0 2023-10-14T16:54:22.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.323+0000 7fb04ae2e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0440d1180 con 0x7fb04415cd70 2023-10-14T16:54:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.323+0000 7fb04ae2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440d1460 con 0x7fb0440b8480 2023-10-14T16:54:22.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.324+0000 7fb049e2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb04415cd70 0x7fb0440d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.324+0000 7fb04962b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 0x7fb0440cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.324+0000 7fb049e2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb04415cd70 0x7fb0440d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:40464/0 (socket says 172.21.15.29:40464) 2023-10-14T16:54:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.324+0000 7fb049e2c640 1 -- 172.21.15.29:0/74002467 learned_addr learned my addr 172.21.15.29:0/74002467 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.325+0000 7fb049e2c640 1 -- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 msgr2=0x7fb0440cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.325+0000 7fb049e2c640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 0x7fb0440cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.325+0000 7fb049e2c640 1 -- 172.21.15.29:0/74002467 >> v1:172.21.15.186:6789/0 conn(0x7fb0440d0be0 legacy=0x7fb0440b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.325+0000 7fb049e2c640 1 -- 172.21.15.29:0/74002467 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0440d1740 con 0x7fb04415cd70 2023-10-14T16:54:22.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.325+0000 7fb04962b640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 0x7fb0440cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:54:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.326+0000 7fb049e2c640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb04415cd70 0x7fb0440d2220 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb040009870 tx=0x7fb04002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2925d56d41272603 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.327+0000 7fb048e2a640 1 -- 172.21.15.29:0/74002467 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040030840 con 0x7fb04415cd70 2023-10-14T16:54:22.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.327+0000 7fb048e2a640 1 -- 172.21.15.29:0/74002467 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400309a0 con 0x7fb04415cd70 2023-10-14T16:54:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.327+0000 7fb048e2a640 1 -- 172.21.15.29:0/74002467 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040030d30 con 0x7fb04415cd70 2023-10-14T16:54:22.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 -- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb04415cd70 msgr2=0x7fb0440d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb04415cd70 0x7fb0440d2220 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb040009870 tx=0x7fb04002b980 comp rx=0 tx=0).stop 2023-10-14T16:54:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 -- 172.21.15.29:0/74002467 shutdown_connections 2023-10-14T16:54:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0440b8480 0x7fb0440cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/74002467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb04415cd70 0x7fb0440d2220 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 -- 172.21.15.29:0/74002467 >> 172.21.15.29:0/74002467 conn(0x7fb0440cf570 msgr2=0x7fb0440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 -- 172.21.15.29:0/74002467 shutdown_connections 2023-10-14T16:54:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.328+0000 7fb04ae2e640 1 -- 172.21.15.29:0/74002467 wait complete. 2023-10-14T16:54:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.330+0000 7fb04ae2e640 1 Processor -- start 2023-10-14T16:54:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.330+0000 7fb04ae2e640 1 -- start start 2023-10-14T16:54:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0440d0be0 0x7fb0441f1d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb04415cd70 0x7fb0440cab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0441f22a0 con 0x7fb0440b8480 2023-10-14T16:54:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440b4670 con 0x7fb0440d0be0 2023-10-14T16:54:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.331+0000 7fb04ae2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0440cb510 con 0x7fb04415cd70 2023-10-14T16:54:22.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45310/0 (socket says 172.21.15.29:45310) 2023-10-14T16:54:22.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 -- 172.21.15.29:0/354671944 learned_addr learned my addr 172.21.15.29:0/354671944 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb04962b640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0440d0be0 0x7fb0441f1d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb04a62d640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb04415cd70 0x7fb0440cab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 -- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb04415cd70 msgr2=0x7fb0440cab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb04415cd70 0x7fb0440cab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 -- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0440d0be0 msgr2=0x7fb0441f1d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0440d0be0 0x7fb0441f1d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.332+0000 7fb049e2c640 1 -- 172.21.15.29:0/354671944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb034009400 con 0x7fb0440b8480 2023-10-14T16:54:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb049e2c640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb040000c00 tx=0x7fb040004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb03a7fc640 1 -- 172.21.15.29:0/354671944 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040039c90 con 0x7fb0440b8480 2023-10-14T16:54:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb03a7fc640 1 -- 172.21.15.29:0/354671944 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb040039e10 con 0x7fb0440b8480 2023-10-14T16:54:22.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb03a7fc640 1 -- 172.21.15.29:0/354671944 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040043640 con 0x7fb0440b8480 2023-10-14T16:54:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0400094d0 con 0x7fb0440b8480 2023-10-14T16:54:22.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.333+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440b4300 con 0x7fb0440b8480 2023-10-14T16:54:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.336+0000 7fb03a7fc640 1 -- 172.21.15.29:0/354671944 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0400437a0 con 0x7fb0440b8480 2023-10-14T16:54:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.336+0000 7fb03a7fc640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 0x7fb02806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.337+0000 7fb03a7fc640 1 -- 172.21.15.29:0/354671944 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb04003e070 con 0x7fb0440b8480 2023-10-14T16:54:22.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.337+0000 7fb04962b640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 0x7fb02806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.337+0000 7fb04ec1f740 1 Processor -- start 2023-10-14T16:54:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.337+0000 7fb04962b640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 0x7fb02806d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0441526e0 tx=0x7fb034009190 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.337+0000 7fb04ec1f740 1 -- start start 2023-10-14T16:54:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f53070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f535b0 0x564577f83440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577e737a0 con 0x564577f80550 2023-10-14T16:54:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577ec5740 con 0x564577f535b0 2023-10-14T16:54:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.338+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564577ec2e30 con 0x564577f80170 2023-10-14T16:54:22.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45326/0 (socket says 172.21.15.29:45326) 2023-10-14T16:54:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 -- 172.21.15.29:0/791012437 learned_addr learned my addr 172.21.15.29:0/791012437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb049e2c640 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f53070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 -- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 msgr2=0x564577f53070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f53070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04a62d640 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f535b0 0x564577f83440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 -- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f535b0 msgr2=0x564577f83440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f535b0 0x564577f83440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.339+0000 7fb04962b640 1 -- 172.21.15.29:0/791012437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564577e82910 con 0x564577f80550 2023-10-14T16:54:22.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb04962b640 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb034011f00 tx=0x7fb034014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd956c76c3b50f0e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb0277fe640 1 -- 172.21.15.29:0/791012437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb034014c90 con 0x564577f80550 2023-10-14T16:54:22.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb0277fe640 1 -- 172.21.15.29:0/791012437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034014df0 con 0x564577f80550 2023-10-14T16:54:22.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb0277fe640 1 -- 172.21.15.29:0/791012437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0340215f0 con 0x564577f80550 2023-10-14T16:54:22.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb04ec1f740 1 -- 172.21.15.29:0/791012437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 msgr2=0x564577f82f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.340+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7fb034011f00 tx=0x7fb034014000 comp rx=0 tx=0).stop 2023-10-14T16:54:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 -- 172.21.15.29:0/791012437 shutdown_connections 2023-10-14T16:54:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f535b0 0x564577f83440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80550 0x564577f82f00 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/791012437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f53070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 -- 172.21.15.29:0/791012437 >> 172.21.15.29:0/791012437 conn(0x564577f55380 msgr2=0x564577f9df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 -- 172.21.15.29:0/791012437 shutdown_connections 2023-10-14T16:54:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 -- 172.21.15.29:0/791012437 wait complete. 2023-10-14T16:54:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.341+0000 7fb04ec1f740 1 -- start start 2023-10-14T16:54:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.342+0000 7fb04ec1f740 1 Processor -- start 2023-10-14T16:54:22.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f84080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564577f81f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577ebe720 con 0x564577f82460 2023-10-14T16:54:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577e5c970 con 0x564577f80550 2023-10-14T16:54:22.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564577e5c610 con 0x564577f80170 2023-10-14T16:54:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04ec1f740 10 client.0 _ll_register_callbacks cb 0x564577e7e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:54:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04a62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04a62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45332/0 (socket says 172.21.15.29:45332) 2023-10-14T16:54:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.343+0000 7fb04a62d640 1 -- 172.21.15.29:0/3358789201 learned_addr learned my addr 172.21.15.29:0/3358789201 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 -- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 msgr2=0x564577f84080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f84080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 -- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 msgr2=0x564577f81f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564577f81f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 -- 172.21.15.29:0/3358789201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb034011bb0 con 0x564577f82460 2023-10-14T16:54:22.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001db0 tx=0x7fb03c005420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4cc70e0c0e566d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb024ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:54:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c0020a0 con 0x564577f82460 2023-10-14T16:54:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb03c013570 con 0x564577f82460 2023-10-14T16:54:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.344+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564577ee5160 con 0x564577f82460 2023-10-14T16:54:22.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb03c013730 con 0x564577f82460 2023-10-14T16:54:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 10 client.6120 fetch_fsmap learned FSMap version 24 2023-10-14T16:54:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x564577ebcda0 con 0x564577f82460 2023-10-14T16:54:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564577ee5160 con 0x564577f82460 2023-10-14T16:54:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb03c013890 con 0x564577f82460 2023-10-14T16:54:22.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 10 client.6120 fetch_fsmap finished waiting for FSMap version 24 2023-10-14T16:54:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 10 client.6120 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T16:54:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 20 client.6120 populate_metadata read hostname 'smithi029' 2023-10-14T16:54:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577f6c930 0x564577fbb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 4 client.6120 mds_command: new command op to 5236 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:54:22.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.345+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 --> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564577e90cc0 con 0x564577f6c930 2023-10-14T16:54:22.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.346+0000 7fb049e2c640 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577f6c930 0x564577fbb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.347+0000 7fb049e2c640 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577f6c930 0x564577fbb680 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ee4e4b7af35bd89 server_cookie=f92da751e3d5256f in_seq=0 out_seq=0 2023-10-14T16:54:22.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.347+0000 7fb024ff9640 10 client.6120 ms_handle_connect on v2:172.21.15.186:6834/2940163923 2023-10-14T16:54:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb024ff9640 1 -- 172.21.15.29:0/3358789201 <== mds.0 v2:172.21.15.186:6834/2940163923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564577e90cc0 con 0x564577f6c930 2023-10-14T16:54:22.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb024ff9640 10 client.6120 handle_command_reply: tid=0 2023-10-14T16:54:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 client.6120 shutdown 2023-10-14T16:54:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 msgr2=0x564577f6bae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7fb03c001db0 tx=0x7fb03c005420 comp rx=0 tx=0).stop 2023-10-14T16:54:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 shutdown_connections 2023-10-14T16:54:22.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577f6c930 0x564577fbb680 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577f6bae0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564577f81f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb024ff9640 0 client.6120 ms_handle_reset on v2:172.21.15.186:6834/2940163923 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3358789201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f84080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 >> 172.21.15.29:0/3358789201 conn(0x564577f535b0 msgr2=0x564577f57160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 shutdown_connections 2023-10-14T16:54:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.349+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3358789201 wait complete. 2023-10-14T16:54:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.350+0000 7fb04ec1f740 20 client.6120 trim_cache size 0 max 16384 2023-10-14T16:54:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.366+0000 7fb04ec1f740 1 Processor -- start 2023-10-14T16:54:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 -- start start 2023-10-14T16:54:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80550 0x564577f58e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 0x564577f59360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577c22f20 con 0x564577f80170 2023-10-14T16:54:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577f84370 con 0x564577f82460 2023-10-14T16:54:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.367+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564577fadc90 con 0x564577f80550 2023-10-14T16:54:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45340/0 (socket says 172.21.15.29:45340) 2023-10-14T16:54:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 -- 172.21.15.29:0/2924788221 learned_addr learned my addr 172.21.15.29:0/2924788221 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb04962b640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80550 0x564577f58e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb04a62d640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 0x564577f59360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 -- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80550 msgr2=0x564577f58e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80550 0x564577f58e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 -- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 msgr2=0x564577f59360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 0x564577f59360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb049e2c640 1 -- 172.21.15.29:0/2924788221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564577e5bb80 con 0x564577f80170 2023-10-14T16:54:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.368+0000 7fb04a62d640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 0x564577f59360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb049e2c640 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb0400aaab0 tx=0x7fb040077670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a3d1799d8bf0a45 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb027fff640 1 -- 172.21.15.29:0/2924788221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb040083580 con 0x564577f80170 2023-10-14T16:54:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb027fff640 1 -- 172.21.15.29:0/2924788221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0400ac040 con 0x564577f80170 2023-10-14T16:54:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb027fff640 1 -- 172.21.15.29:0/2924788221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0400796d0 con 0x564577f80170 2023-10-14T16:54:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 -- 172.21.15.29:0/2924788221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 msgr2=0x564577f88290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7fb0400aaab0 tx=0x7fb040077670 comp rx=0 tx=0).stop 2023-10-14T16:54:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 -- 172.21.15.29:0/2924788221 shutdown_connections 2023-10-14T16:54:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f82460 0x564577f59360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80550 0x564577f58e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/2924788221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f80170 0x564577f88290 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.369+0000 7fb04ec1f740 1 -- 172.21.15.29:0/2924788221 >> 172.21.15.29:0/2924788221 conn(0x564577f535b0 msgr2=0x564577f57160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.370+0000 7fb04ec1f740 1 -- 172.21.15.29:0/2924788221 shutdown_connections 2023-10-14T16:54:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.370+0000 7fb04ec1f740 1 -- 172.21.15.29:0/2924788221 wait complete. 2023-10-14T16:54:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.370+0000 7fb04ec1f740 1 -- start start 2023-10-14T16:54:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.370+0000 7fb04ec1f740 1 Processor -- start 2023-10-14T16:54:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f810f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564578009ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577f58990 con 0x564577f82460 2023-10-14T16:54:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564577fad0f0 con 0x564577f80550 2023-10-14T16:54:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564577f81b70 con 0x564577f80170 2023-10-14T16:54:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.371+0000 7fb04ec1f740 10 client.0 _ll_register_callbacks cb 0x7fb03c0039b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:54:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45348/0 (socket says 172.21.15.29:45348) 2023-10-14T16:54:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 -- 172.21.15.29:0/3751360190 learned_addr learned my addr 172.21.15.29:0/3751360190 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 -- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 msgr2=0x564577f810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f810f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 -- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 msgr2=0x564578009ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564578009ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.372+0000 7fb04a62d640 1 -- 172.21.15.29:0/3751360190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0440b4300 con 0x564577f82460 2023-10-14T16:54:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb04a62d640 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb03c013b40 tx=0x7fb03c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e27ab49fc1e98f0d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:54:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb03c013d00 con 0x564577f82460 2023-10-14T16:54:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb03c0020a0 con 0x564577f82460 2023-10-14T16:54:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb034014df0 con 0x564577f82460 2023-10-14T16:54:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb03c006b50 con 0x564577f82460 2023-10-14T16:54:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb04ec1f740 10 client.6126 fetch_fsmap learned FSMap version 24 2023-10-14T16:54:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=2+}) v3 -- 0x564577e81170 con 0x564577f82460 2023-10-14T16:54:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb034014df0 con 0x564577f82460 2023-10-14T16:54:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 24) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fb03c006cb0 con 0x564577f82460 2023-10-14T16:54:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.373+0000 7fb04ec1f740 10 client.6126 fetch_fsmap finished waiting for FSMap version 24 2023-10-14T16:54:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb04ec1f740 10 client.6126 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T16:54:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb04ec1f740 20 client.6126 populate_metadata read hostname 'smithi029' 2023-10-14T16:54:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577fac920 0x564577f7e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb04ec1f740 4 client.6126 mds_command: new command op to 5236 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:54:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 --> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] -- command(tid 0: {"prefix": "status"}) v1 -- 0x564577eb7700 con 0x564577fac920 2023-10-14T16:54:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.374+0000 7fb049e2c640 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577fac920 0x564577f7e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.375+0000 7fb049e2c640 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577fac920 0x564577f7e010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c3971872db49d47 server_cookie=cff5f068f7209f28 in_seq=0 out_seq=0 2023-10-14T16:54:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.375+0000 7fb024ff9640 10 client.6126 ms_handle_connect on v2:172.21.15.186:6834/2940163923 2023-10-14T16:54:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb024ff9640 1 -- 172.21.15.29:0/3751360190 <== mds.0 v2:172.21.15.186:6834/2940163923 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x564577eb7700 con 0x564577fac920 2023-10-14T16:54:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb024ff9640 10 client.6126 handle_command_reply: tid=0 2023-10-14T16:54:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 client.6126 shutdown 2023-10-14T16:54:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 msgr2=0x564577fac3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fb03c013b40 tx=0x7fb03c0014a0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 shutdown_connections 2023-10-14T16:54:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:6834/2940163923,v1:172.21.15.186:6835/2940163923] conn(0x564577fac920 0x564577f7e010 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564577f82460 0x564577fac3e0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564577f80550 0x564578009ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 --2- 172.21.15.29:0/3751360190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564577f80170 0x564577f810f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb024ff9640 0 client.6126 ms_handle_reset on v2:172.21.15.186:6834/2940163923 2023-10-14T16:54:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 >> 172.21.15.29:0/3751360190 conn(0x564577f535b0 msgr2=0x564577fa19c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 shutdown_connections 2023-10-14T16:54:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.376+0000 7fb04ec1f740 1 -- 172.21.15.29:0/3751360190 wait complete. 2023-10-14T16:54:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ec1f740 20 client.6126 trim_cache size 0 max 16384 2023-10-14T16:54:22.388 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:54:22.388 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 5236, 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:54:22.389 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8418794529999998, 2023-10-14T16:54:22.390 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 24, 2023-10-14T16:54:22.390 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 63, 2023-10-14T16:54:22.390 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 63, 2023-10-14T16:54:22.390 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.172665771 2023-10-14T16:54:22.390 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:54:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 msgr2=0x7fb02806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 0x7fb02806d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb0441526e0 tx=0x7fb034009190 comp rx=0 tx=0).stop 2023-10-14T16:54:22.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 msgr2=0x7fb0440c74a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fb040000c00 tx=0x7fb040004730 comp rx=0 tx=0).stop 2023-10-14T16:54:22.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 shutdown_connections 2023-10-14T16:54:22.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fb02806afc0 0x7fb02806d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb04415cd70 0x7fb0440cab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0440d0be0 0x7fb0441f1d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 --2- 172.21.15.29:0/354671944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0440b8480 0x7fb0440c74a0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 >> 172.21.15.29:0/354671944 conn(0x7fb0440cf570 msgr2=0x7fb0440b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 shutdown_connections 2023-10-14T16:54:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.377+0000 7fb04ae2e640 1 -- 172.21.15.29:0/354671944 wait complete. 2023-10-14T16:54:22.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:54:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.623+0000 7f8a92e26640 1 Processor -- start 2023-10-14T16:54:22.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.623+0000 7f8a92e26640 1 -- start start 2023-10-14T16:54:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a92e26640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8a8c159b20 0x7f8a8c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a92e26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a92e26640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a8c06f7f0 con 0x7f8a8c159f00 2023-10-14T16:54:22.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a92e26640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c06c5e0 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a92e26640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8a8c06c8c0 con 0x7f8a8c159b20 2023-10-14T16:54:22.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a913c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a913c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45352/0 (socket says 172.21.15.29:45352) 2023-10-14T16:54:22.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a913c3640 1 -- 172.21.15.29:0/3058117585 learned_addr learned my addr 172.21.15.29:0/3058117585 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.625+0000 7f8a90bc2640 1 --2- 172.21.15.29:0/3058117585 >> v2:172.21.15.186:3300/0 conn(0x7f8a8c159b20 0x7f8a8c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.626+0000 7f8a913c3640 1 -- 172.21.15.29:0/3058117585 >> v2:172.21.15.186:3300/0 conn(0x7f8a8c159b20 msgr2=0x7f8a8c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.626+0000 7f8a913c3640 1 --2- 172.21.15.29:0/3058117585 >> v2:172.21.15.186:3300/0 conn(0x7f8a8c159b20 0x7f8a8c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.626+0000 7f8a913c3640 1 -- 172.21.15.29:0/3058117585 >> v1:172.21.15.186:6789/0 conn(0x7f8a8c159f00 legacy=0x7f8a8c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:22.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.626+0000 7f8a913c3640 1 -- 172.21.15.29:0/3058117585 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a8c06cba0 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.627+0000 7f8a913c3640 1 --2- 172.21.15.29:0/3058117585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0c9340 tx=0x7f8a8402b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3418d858c5e4cd77 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.627+0000 7f8a837fe640 1 -- 172.21.15.29:0/3058117585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031860 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.627+0000 7f8a837fe640 1 -- 172.21.15.29:0/3058117585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a840319c0 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.627+0000 7f8a837fe640 1 -- 172.21.15.29:0/3058117585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a84031d50 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 -- 172.21.15.29:0/3058117585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 msgr2=0x7f8a8c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3058117585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c0c9340 tx=0x7f8a8402b900 comp rx=0 tx=0).stop 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 -- 172.21.15.29:0/3058117585 shutdown_connections 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3058117585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c06fc80 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3058117585 >> v2:172.21.15.186:3300/0 conn(0x7f8a8c159b20 0x7f8a8c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 -- 172.21.15.29:0/3058117585 >> 172.21.15.29:0/3058117585 conn(0x7f8a8c0cb410 msgr2=0x7f8a8c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 -- 172.21.15.29:0/3058117585 shutdown_connections 2023-10-14T16:54:22.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.628+0000 7f8a92e26640 1 -- 172.21.15.29:0/3058117585 wait complete. 2023-10-14T16:54:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.630+0000 7f8a92e26640 1 Processor -- start 2023-10-14T16:54:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.630+0000 7f8a92e26640 1 -- start start 2023-10-14T16:54:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8c159b20 0x7f8a8c0cedb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a8c159f00 0x7f8a8c145720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0af400 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0aedd0 con 0x7f8a8c159f00 2023-10-14T16:54:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a92e26640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8a8c0b3d20 con 0x7f8a8c159b20 2023-10-14T16:54:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45356/0 (socket says 172.21.15.29:45356) 2023-10-14T16:54:22.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 -- 172.21.15.29:0/3463915331 learned_addr learned my addr 172.21.15.29:0/3463915331 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:22.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 -- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8c159b20 msgr2=0x7f8a8c0cedb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8c159b20 0x7f8a8c0cedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 -- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a8c159f00 msgr2=0x7f8a8c145720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.631+0000 7f8a90bc2640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a8c159f00 0x7f8a8c145720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a90bc2640 1 -- 172.21.15.29:0/3463915331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a840094d0 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a90bc2640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a88007ef0 tx=0x7f8a8800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88014620 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a88014780 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a88014b30 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a8c0b4000 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.632+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a8c0c5a60 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.634+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a8c04e160 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.635+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a88014c90 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.635+0000 7f8a817fa640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 0x7f8a7806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.636+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f8a8808a180 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.636+0000 7f8a83fff640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 0x7f8a7806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.636+0000 7f8a83fff640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 0x7f8a7806d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c15d640 tx=0x7f8a7c0079b0 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.640+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8a880550f0 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.879+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f8a8c0c9610 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.881+0000 7f8a817fa640 1 -- 172.21.15.29:0/3463915331 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) v1 ==== 53+0+73 (secure 0 0 0) 0x7f8a88058710 con 0x7f8a8c0b0b20 2023-10-14T16:54:22.881 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:54:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 msgr2=0x7f8a7806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 0x7f8a7806d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c15d640 tx=0x7f8a7c0079b0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 msgr2=0x7f8a8c0ce870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f8a88007ef0 tx=0x7f8a8800e4b0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 shutdown_connections 2023-10-14T16:54:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f8a7806afc0 0x7f8a7806d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8a8c159f00 0x7f8a8c145720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8a8c159b20 0x7f8a8c0cedb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 --2- 172.21.15.29:0/3463915331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8a8c0b0b20 0x7f8a8c0ce870 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.883+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 >> 172.21.15.29:0/3463915331 conn(0x7f8a8c0cb410 msgr2=0x7f8a8c14fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.884+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 shutdown_connections 2023-10-14T16:54:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:22.884+0000 7f8a92e26640 1 -- 172.21.15.29:0/3463915331 wait complete. 2023-10-14T16:54:22.904 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:54:22.904 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:54:22.905 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:54:22.905 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:54:22.905 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:54:22.906 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:54:22.906 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:54:22.906 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:54:22.907 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:54:22.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:22.908 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:54:22.965 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:54:22.965 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:54:22.965 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:54:22.966 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.966 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:54:22.966 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.966 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:54:22.967 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:54:22.967 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:54:22.967 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:54:22.967 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:54:22.967 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:54:22.968 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:54:22.968 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:54:22.968 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:54:22.968 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:54:22.969 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41513sec preferred_lft 41513sec 2023-10-14T16:54:22.969 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:54:22.969 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.969 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:54:22.970 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:54:22.970 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:54:22.970 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:54:22.970 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:54:22.970 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:54:22.971 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:54:22.972 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.972 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:54:22.972 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:54:22.972 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:54:22.972 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:54:22.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:22.974 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:54:23.076 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:54:23.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.081 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:54:23.136 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:54:23.137 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:54:23.137 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:54:23.137 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:54:23.137 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:54:23.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.138 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:54:23.139 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:54:23.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:54:23.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:54:23.140 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:54:23.140 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:54:23.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.336 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:54:23.392 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:54:23.393 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:54:23.394 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:54:23.394 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.395 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:54:23.501 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:54:23.501 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:54:23.559 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:54:23.560 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:54:23.560 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:54:23.617 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:54:23.618 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:54:23.773 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:54:23.773 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:54:23.773 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:54:23.796 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:54:23.797 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:54:23.797 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:54:23.797 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:54:23.797 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:54:23.825 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:54:23.825 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:54:23.826 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:54:23.826 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:54:23.826 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:54:23.826 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:54:23.827 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:54:23.827 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:54:23.827 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:54:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:54:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:54:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:54:23.828 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:54:23.833 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:54:23.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:54:23.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:23.927 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:54:24.012 INFO:teuthology.orchestra.run.smithi029.stdout:client4997 2023-10-14T16:54:24.012 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:54:24.013 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:54:24.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:54:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.308+0000 7f7542859640 1 Processor -- start 2023-10-14T16:54:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.308+0000 7f7542859640 1 -- start start 2023-10-14T16:54:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542859640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f753c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542859640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0ca560 con 0x7f753c06c9e0 2023-10-14T16:54:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542859640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f753c0c9950 con 0x7f753c0c9060 2023-10-14T16:54:24.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542859640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0c9c30 con 0x7f753c06fb50 2023-10-14T16:54:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542058640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f753c06c9e0 0x7f753c0ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60780/0 (socket says 172.21.15.29:60780) 2023-10-14T16:54:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7542058640 1 -- 172.21.15.29:0/200703968 learned_addr learned my addr 172.21.15.29:0/200703968 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.310+0000 7f7541056640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f753c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2799086118 0 0) 0x7f753c0ca560 con 0x7f753c06c9e0 2023-10-14T16:54:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7528003520 con 0x7f753c06c9e0 2023-10-14T16:54:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977458794 0 0) 0x7f7528003520 con 0x7f753c06c9e0 2023-10-14T16:54:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 msgr2=0x7f753c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f753c0cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 >> v1:172.21.15.186:6789/0 conn(0x7f753c0c9060 legacy=0x7f753c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0c9f10 con 0x7f753c06c9e0 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f75340019f0 con 0x7f753c06c9e0 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 >> v1:172.21.15.29:6789/0 conn(0x7f753c06c9e0 legacy=0x7f753c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75280039a0 0x7f7528003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280047c0 0x7f7528004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f7528005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0c9c30 con 0x7f75280039a0 2023-10-14T16:54:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f752800ba30 con 0x7f75280047c0 2023-10-14T16:54:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f752800bbf0 con 0x7f753c06fb50 2023-10-14T16:54:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.311+0000 7f7541857640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75280039a0 0x7f7528003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541056640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280047c0 0x7f7528004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7542058640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f7528005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541857640 1 -- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 msgr2=0x7f7528005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541857640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c06fb50 0x7f7528005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541857640 1 -- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280047c0 msgr2=0x7f7528004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:24.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541857640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280047c0 0x7f7528004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.312+0000 7f7541857640 1 -- 172.21.15.29:0/200703968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7528003520 con 0x7f75280039a0 2023-10-14T16:54:24.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.313+0000 7f7541857640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75280039a0 0x7f7528003dc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7538001720 tx=0x7f75280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4771b607570582e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.313+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75380052d0 con 0x7f75280039a0 2023-10-14T16:54:24.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.313+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753802e050 con 0x7f75280039a0 2023-10-14T16:54:24.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.313+0000 7f7540855640 1 -- 172.21.15.29:0/200703968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75380035f0 con 0x7f75280039a0 2023-10-14T16:54:24.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 -- 172.21.15.29:0/200703968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75280039a0 msgr2=0x7f7528003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 --2- 172.21.15.29:0/200703968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75280039a0 0x7f7528003dc0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f7538001720 tx=0x7f75280057f0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542058640 1 -- 172.21.15.29:0/200703968 reap_dead start 2023-10-14T16:54:24.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 -- 172.21.15.29:0/200703968 shutdown_connections 2023-10-14T16:54:24.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 -- 172.21.15.29:0/200703968 >> 172.21.15.29:0/200703968 conn(0x7f753c0ce6a0 msgr2=0x7f753c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:24.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 -- 172.21.15.29:0/200703968 shutdown_connections 2023-10-14T16:54:24.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.314+0000 7f7542859640 1 -- 172.21.15.29:0/200703968 wait complete. 2023-10-14T16:54:24.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.316+0000 7f7542859640 1 Processor -- start 2023-10-14T16:54:24.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.316+0000 7f7542859640 1 -- start start 2023-10-14T16:54:24.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 0x7f753c0b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 0x7f753c0c85a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0b2df0 con 0x7f753c1f6090 2023-10-14T16:54:24.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0c8ae0 con 0x7f75280039a0 2023-10-14T16:54:24.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.317+0000 7f7542859640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f753c0b21e0 con 0x7f753c0b3280 2023-10-14T16:54:24.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45376/0 (socket says 172.21.15.29:45376) 2023-10-14T16:54:24.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 -- 172.21.15.29:0/1778088826 learned_addr learned my addr 172.21.15.29:0/1778088826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:24.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7541056640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 0x7f753c0c85a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7541857640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 0x7f753c0b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 -- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 msgr2=0x7f753c0c85a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 0x7f753c0c85a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 -- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 msgr2=0x7f753c0b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 0x7f753c0b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 -- 172.21.15.29:0/1778088826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f753c0b24c0 con 0x7f753c1f6090 2023-10-14T16:54:24.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7541056640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 0x7f753c0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:24.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7541857640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 0x7f753c0b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:24.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.318+0000 7f7542058640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7534001370 tx=0x7f753400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.319+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f753400bb80 con 0x7f753c1f6090 2023-10-14T16:54:24.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.319+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f753400bd00 con 0x7f753c1f6090 2023-10-14T16:54:24.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.319+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7534007350 con 0x7f753c1f6090 2023-10-14T16:54:24.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.319+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f753c0b27a0 con 0x7f753c1f6090 2023-10-14T16:54:24.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.319+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f753c1f6540 con 0x7f753c1f6090 2023-10-14T16:54:24.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.320+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f753c0b2fa0 con 0x7f753c1f6090 2023-10-14T16:54:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.321+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75340074b0 con 0x7f753c1f6090 2023-10-14T16:54:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.322+0000 7f752e7fc640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 0x7f752006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.322+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f7534089ac0 con 0x7f753c1f6090 2023-10-14T16:54:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.322+0000 7f7541857640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 0x7f752006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.323+0000 7f7541857640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 0x7f752006d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7538033000 tx=0x7f7538009030 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.326+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f753c0b2fa0 con 0x7f753c1f6090 2023-10-14T16:54:24.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.547+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 --> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f753c0c1f10 con 0x7f752006afc0 2023-10-14T16:54:24.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.550+0000 7f752e7fc640 1 -- 172.21.15.29:0/1778088826 <== mgr.5763 v2:172.21.15.29:6832/98172895 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f75380054c0 con 0x7f752006afc0 2023-10-14T16:54:24.551 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:54:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:54:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:54:24.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:54:24.552 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:54:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 msgr2=0x7f752006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 0x7f752006d480 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f7538033000 tx=0x7f7538009030 comp rx=0 tx=0).stop 2023-10-14T16:54:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 msgr2=0x7f753c0c59a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f7534001370 tx=0x7f753400b4a0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 shutdown_connections 2023-10-14T16:54:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f752006afc0 0x7f752006d480 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f753c1f6090 0x7f753c0c59a0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f753c0b3280 0x7f753c0c85a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 --2- 172.21.15.29:0/1778088826 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75280039a0 0x7f753c0b1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 >> 172.21.15.29:0/1778088826 conn(0x7f753c0ce6a0 msgr2=0x7f753c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 shutdown_connections 2023-10-14T16:54:24.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.553+0000 7f7542859640 1 -- 172.21.15.29:0/1778088826 wait complete. 2023-10-14T16:54:24.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:54:24.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.799+0000 7f9bf09f4640 1 Processor -- start 2023-10-14T16:54:24.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.800+0000 7f9bf09f4640 1 -- start start 2023-10-14T16:54:24.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.801+0000 7f9bf09f4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bec15b320 0x7f9bec0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.801+0000 7f9bf09f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec15b9d0 con 0x7f9bec06c040 2023-10-14T16:54:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.801+0000 7f9bf09f4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bec15bcb0 con 0x7f9bec0ad970 2023-10-14T16:54:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.801+0000 7f9bf09f4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bec15b7d0 con 0x7f9bec15b320 2023-10-14T16:54:24.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9beb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9bec06c040 0x7f9bec06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:60792/0 (socket says 172.21.15.29:60792) 2023-10-14T16:54:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9beb7fe640 1 -- 172.21.15.29:0/2601805586 learned_addr learned my addr 172.21.15.29:0/2601805586 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:54:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9bebfff640 1 --2- 172.21.15.29:0/2601805586 >> v2:172.21.15.29:3300/0 conn(0x7f9bec15b320 0x7f9bec0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 385929518 0 0) 0x7f9bec15b9d0 con 0x7f9bec06c040 2023-10-14T16:54:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd8003520 con 0x7f9bec06c040 2023-10-14T16:54:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9bebfff640 1 -- 172.21.15.29:0/2601805586 >> v1:172.21.15.186:6789/0 conn(0x7f9bec0ad970 legacy=0x7f9bec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3875295041 0 0) 0x7f9bd8003520 con 0x7f9bec06c040 2023-10-14T16:54:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9bebfff640 1 -- 172.21.15.29:0/2601805586 >> v1:172.21.15.29:6789/0 conn(0x7f9bec06c040 legacy=0x7f9bec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.802+0000 7f9bebfff640 1 -- 172.21.15.29:0/2601805586 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec15bf90 con 0x7f9bec15b320 2023-10-14T16:54:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.804+0000 7f9bebfff640 1 --2- 172.21.15.29:0/2601805586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0cd180 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9be40097d0 tx=0x7f9be402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d64bb143ad9a75f6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.804+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be402f050 con 0x7f9bec15b320 2023-10-14T16:54:24.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.804+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be4031500 con 0x7f9bec15b320 2023-10-14T16:54:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.804+0000 7f9beaffd640 1 -- 172.21.15.29:0/2601805586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bdc001670 con 0x7f9bec15b320 2023-10-14T16:54:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 -- 172.21.15.29:0/2601805586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 msgr2=0x7f9bec0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/2601805586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0cd180 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9be40097d0 tx=0x7f9be402b820 comp rx=0 tx=0).stop 2023-10-14T16:54:24.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 -- 172.21.15.29:0/2601805586 shutdown_connections 2023-10-14T16:54:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/2601805586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0cd180 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 -- 172.21.15.29:0/2601805586 >> 172.21.15.29:0/2601805586 conn(0x7f9bec0ce6a0 msgr2=0x7f9bec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:24.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 -- 172.21.15.29:0/2601805586 shutdown_connections 2023-10-14T16:54:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.805+0000 7f9bf09f4640 1 -- 172.21.15.29:0/2601805586 wait complete. 2023-10-14T16:54:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.807+0000 7f9bf09f4640 1 Processor -- start 2023-10-14T16:54:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.807+0000 7f9bf09f4640 1 -- start start 2023-10-14T16:54:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bec06c040 0x7f9bec0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bec0ad970 0x7f9bec0ad070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0ad5b0 con 0x7f9bec15b320 2023-10-14T16:54:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0ae430 con 0x7f9bec06c040 2023-10-14T16:54:24.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bf09f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bec0ae710 con 0x7f9bec0ad970 2023-10-14T16:54:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45388/0 (socket says 172.21.15.29:45388) 2023-10-14T16:54:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9bebfff640 1 -- 172.21.15.29:0/3885077042 learned_addr learned my addr 172.21.15.29:0/3885077042 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.808+0000 7f9beb7fe640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bec06c040 0x7f9bec0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9be2ffd640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bec0ad970 0x7f9bec0ad070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 -- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bec0ad970 msgr2=0x7f9bec0ad070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bec0ad970 0x7f9bec0ad070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 -- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bec06c040 msgr2=0x7f9bec0ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:24.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bec06c040 0x7f9bec0ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 -- 172.21.15.29:0/3885077042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9be4009400 con 0x7f9bec15b320 2023-10-14T16:54:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bebfff640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9be40097d0 tx=0x7f9be40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be4004220 con 0x7f9bec15b320 2023-10-14T16:54:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be40043a0 con 0x7f9bec15b320 2023-10-14T16:54:24.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be403e990 con 0x7f9bec15b320 2023-10-14T16:54:24.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.809+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bec0ae9f0 con 0x7f9bec15b320 2023-10-14T16:54:24.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.810+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bec0b3c10 con 0x7f9bec15b320 2023-10-14T16:54:24.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.811+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bec06b330 con 0x7f9bec15b320 2023-10-14T16:54:24.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.812+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 18) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9be403a040 con 0x7f9bec15b320 2023-10-14T16:54:24.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.813+0000 7f9be8ff9640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 0x7f9bcc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:24.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.813+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f9be40ab590 con 0x7f9bec15b320 2023-10-14T16:54:24.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.813+0000 7f9beb7fe640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 0x7f9bcc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:24.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.814+0000 7f9beb7fe640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 0x7f9bcc06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc000f60 tx=0x7f9bdc009040 comp rx=0 tx=0).ready entity=mgr.5763 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:24.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:24.818+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9be40763f0 con 0x7f9bec15b320 2023-10-14T16:54:25.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:25.063+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f9bec0ae570 con 0x7f9bec15b320 2023-10-14T16:54:26.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.078+0000 7f9be8ff9640 1 -- 172.21.15.29:0/3885077042 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f9bec0ae570 con 0x7f9bec15b320 2023-10-14T16:54:26.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 msgr2=0x7f9bcc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 0x7f9bcc06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9bdc000f60 tx=0x7f9bdc009040 comp rx=0 tx=0).stop 2023-10-14T16:54:26.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 msgr2=0x7f9bec0b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9be40097d0 tx=0x7f9be40046c0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 shutdown_connections 2023-10-14T16:54:26.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f9bcc06af90 0x7f9bcc06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bec15b320 0x7f9bec0b36d0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bec0ad970 0x7f9bec0ad070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 --2- 172.21.15.29:0/3885077042 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bec06c040 0x7f9bec0ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.082+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 >> 172.21.15.29:0/3885077042 conn(0x7f9bec0ce6a0 msgr2=0x7f9bec15cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:26.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.083+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 shutdown_connections 2023-10-14T16:54:26.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.083+0000 7f9bf09f4640 1 -- 172.21.15.29:0/3885077042 wait complete. 2023-10-14T16:54:26.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:54:26.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.313+0000 7f2282a46640 1 Processor -- start 2023-10-14T16:54:26.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.313+0000 7f2282a46640 1 -- start start 2023-10-14T16:54:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.314+0000 7f2282a46640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f227c15aee0 0x7f227c0aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.314+0000 7f2282a46640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.314+0000 7f2282a46640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227c06c140 con 0x7f227c069da0 2023-10-14T16:54:26.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.314+0000 7f2282a46640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f227c15b2c0 con 0x7f227c15aee0 2023-10-14T16:54:26.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.314+0000 7f2282a46640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c06b270 con 0x7f227c0cb810 2023-10-14T16:54:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.315+0000 7f22737fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.315+0000 7f22737fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:60976/0 (socket says 172.21.15.29:60976) 2023-10-14T16:54:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.315+0000 7f22737fe640 1 -- 172.21.15.29:0/2010041975 learned_addr learned my addr 172.21.15.29:0/2010041975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.315+0000 7f227bfff640 1 --2- 172.21.15.29:0/2010041975 >> v2:172.21.15.186:3300/0 conn(0x7f227c15aee0 0x7f227c0aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.316+0000 7f22737fe640 1 -- 172.21.15.29:0/2010041975 >> v2:172.21.15.186:3300/0 conn(0x7f227c15aee0 msgr2=0x7f227c0aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.316+0000 7f22737fe640 1 --2- 172.21.15.29:0/2010041975 >> v2:172.21.15.186:3300/0 conn(0x7f227c15aee0 0x7f227c0aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.316+0000 7f22737fe640 1 -- 172.21.15.29:0/2010041975 >> v1:172.21.15.186:6789/0 conn(0x7f227c069da0 legacy=0x7f227c06a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:26.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.316+0000 7f22737fe640 1 -- 172.21.15.29:0/2010041975 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227c06daf0 con 0x7f227c0cb810 2023-10-14T16:54:26.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.317+0000 7f22737fe640 1 --2- 172.21.15.29:0/2010041975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2264009870 tx=0x7f226402b7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d9802bfe759362a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.317+0000 7f227b7fe640 1 -- 172.21.15.29:0/2010041975 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264031740 con 0x7f227c0cb810 2023-10-14T16:54:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.317+0000 7f227b7fe640 1 -- 172.21.15.29:0/2010041975 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22640318a0 con 0x7f227c0cb810 2023-10-14T16:54:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.317+0000 7f227b7fe640 1 -- 172.21.15.29:0/2010041975 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2264031c30 con 0x7f227c0cb810 2023-10-14T16:54:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 -- 172.21.15.29:0/2010041975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 msgr2=0x7f227c0cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 --2- 172.21.15.29:0/2010041975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2264009870 tx=0x7f226402b7e0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 -- 172.21.15.29:0/2010041975 shutdown_connections 2023-10-14T16:54:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 --2- 172.21.15.29:0/2010041975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c0cb810 0x7f227c0cbc70 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 --2- 172.21.15.29:0/2010041975 >> v2:172.21.15.186:3300/0 conn(0x7f227c15aee0 0x7f227c0aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 -- 172.21.15.29:0/2010041975 >> 172.21.15.29:0/2010041975 conn(0x7f227c0cfac0 msgr2=0x7f227c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:26.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.318+0000 7f2282a46640 1 -- 172.21.15.29:0/2010041975 shutdown_connections 2023-10-14T16:54:26.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.319+0000 7f2282a46640 1 -- 172.21.15.29:0/2010041975 wait complete. 2023-10-14T16:54:26.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.321+0000 7f2282a46640 1 Processor -- start 2023-10-14T16:54:26.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.321+0000 7f2282a46640 1 -- start start 2023-10-14T16:54:26.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c069da0 0x7f227c155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c0cb810 0x7f227c0adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0ad6b0 con 0x7f227c15aee0 2023-10-14T16:54:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f227c155560 con 0x7f227c0cb810 2023-10-14T16:54:26.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.322+0000 7f2282a46640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227c0aca30 con 0x7f227c069da0 2023-10-14T16:54:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f227bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c069da0 0x7f227c155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45410/0 (socket says 172.21.15.29:45410) 2023-10-14T16:54:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 -- 172.21.15.29:0/2888296844 learned_addr learned my addr 172.21.15.29:0/2888296844 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f22737fe640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c0cb810 0x7f227c0adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c069da0 msgr2=0x7f227c155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c069da0 0x7f227c155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c0cb810 msgr2=0x7f227c0adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c0cb810 0x7f227c0adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.323+0000 7f2280fe3640 1 -- 172.21.15.29:0/2888296844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22640094d0 con 0x7f227c15aee0 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f2280fe3640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f227400b970 tx=0x7f227400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:26.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227400cb40 con 0x7f227c15aee0 2023-10-14T16:54:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f227400cca0 con 0x7f227c15aee0 2023-10-14T16:54:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f227c0acd10 con 0x7f227c15aee0 2023-10-14T16:54:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274014680 con 0x7f227c15aee0 2023-10-14T16:54:26.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.324+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f227c0ad2d0 con 0x7f227c15aee0 2023-10-14T16:54:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.325+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f227c0ad7f0 con 0x7f227c15aee0 2023-10-14T16:54:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.327+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f227401d3b0 con 0x7f227c15aee0 2023-10-14T16:54:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.328+0000 7f22797fa640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 0x7f225c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.328+0000 7f227bfff640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 msgr2=0x7f225c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.328+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f2274089640 con 0x7f227c15aee0 2023-10-14T16:54:26.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.328+0000 7f227bfff640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 0x7f225c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:54:26.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.333+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f227c0ad7f0 con 0x7f227c15aee0 2023-10-14T16:54:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.529+0000 7f227bfff640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 msgr2=0x7f225c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:26.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.529+0000 7f227bfff640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 0x7f225c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:54:26.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.534+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f227c0ad7f0 con 0x7f227c15aee0 2023-10-14T16:54:26.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.540+0000 7f22797fa640 1 -- 172.21.15.29:0/2888296844 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v17)=0 v17) v1 ==== 138+0+0 (secure 0 0 0) 0x7f227c0ad7f0 con 0x7f227c15aee0 2023-10-14T16:54:26.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 msgr2=0x7f225c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:54:26.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 0x7f225c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 msgr2=0x7f227c06a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f227400b970 tx=0x7f227400be30 comp rx=0 tx=0).stop 2023-10-14T16:54:26.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 shutdown_connections 2023-10-14T16:54:26.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f225c06afe0 0x7f225c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f227c15aee0 0x7f227c06a1f0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f227c0cb810 0x7f227c0adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 --2- 172.21.15.29:0/2888296844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f227c069da0 0x7f227c155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 >> 172.21.15.29:0/2888296844 conn(0x7f227c0cfac0 msgr2=0x7f227c0c9ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:26.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.542+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 shutdown_connections 2023-10-14T16:54:26.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.543+0000 7f2282a46640 1 -- 172.21.15.29:0/2888296844 wait complete. 2023-10-14T16:54:26.561 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:54:26.600 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:26.599+0000 7ff4f5445040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:54:26.641 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:26.641+0000 7f83d620f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:54:26.654 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:26.653+0000 7ff4f5445040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:54:26.699 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:26.699+0000 7f83d620f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:54:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.776+0000 7fc646973640 1 Processor -- start 2023-10-14T16:54:26.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.776+0000 7fc646973640 1 -- start start 2023-10-14T16:54:26.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc646973640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc646973640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6400b6cd0 0x7fc6400b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc646973640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6400d3980 con 0x7fc6400cfaf0 2023-10-14T16:54:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc646973640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6400d3c60 con 0x7fc6400b6cd0 2023-10-14T16:54:26.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc646973640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6400ce630 con 0x7fc6400cf710 2023-10-14T16:54:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.778+0000 7fc644f10640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6400b6cd0 0x7fc6400b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:60986/0 (socket says 172.21.15.29:60986) 2023-10-14T16:54:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 -- 172.21.15.29:0/1024233475 learned_addr learned my addr 172.21.15.29:0/1024233475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 -- 172.21.15.29:0/1024233475 >> v2:172.21.15.186:3300/0 conn(0x7fc6400b6cd0 msgr2=0x7fc6400b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 --2- 172.21.15.29:0/1024233475 >> v2:172.21.15.186:3300/0 conn(0x7fc6400b6cd0 0x7fc6400b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 -- 172.21.15.29:0/1024233475 >> v1:172.21.15.186:6789/0 conn(0x7fc6400cfaf0 legacy=0x7fc6400d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.779+0000 7fc63ffff640 1 -- 172.21.15.29:0/1024233475 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6400ce910 con 0x7fc6400cf710 2023-10-14T16:54:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.780+0000 7fc63ffff640 1 --2- 172.21.15.29:0/1024233475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc630009850 tx=0x7fc63002b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafacda103bc343c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.781+0000 7fc63f7fe640 1 -- 172.21.15.29:0/1024233475 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630004230 con 0x7fc6400cf710 2023-10-14T16:54:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.781+0000 7fc63f7fe640 1 -- 172.21.15.29:0/1024233475 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc630004390 con 0x7fc6400cf710 2023-10-14T16:54:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.781+0000 7fc63f7fe640 1 -- 172.21.15.29:0/1024233475 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc630031830 con 0x7fc6400cf710 2023-10-14T16:54:26.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.781+0000 7fc646973640 1 -- 172.21.15.29:0/1024233475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 msgr2=0x7fc6400b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.781+0000 7fc646973640 1 --2- 172.21.15.29:0/1024233475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc630009850 tx=0x7fc63002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 -- 172.21.15.29:0/1024233475 shutdown_connections 2023-10-14T16:54:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 --2- 172.21.15.29:0/1024233475 >> v2:172.21.15.186:3300/0 conn(0x7fc6400b6cd0 0x7fc6400b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 --2- 172.21.15.29:0/1024233475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400cf710 0x7fc6400b2d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 -- 172.21.15.29:0/1024233475 >> 172.21.15.29:0/1024233475 conn(0x7fc6400d2cc0 msgr2=0x7fc6400d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 -- 172.21.15.29:0/1024233475 shutdown_connections 2023-10-14T16:54:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.782+0000 7fc646973640 1 -- 172.21.15.29:0/1024233475 wait complete. 2023-10-14T16:54:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.783+0000 7fc646973640 1 Processor -- start 2023-10-14T16:54:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.784+0000 7fc646973640 1 -- start start 2023-10-14T16:54:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.784+0000 7fc646973640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400b6cd0 0x7fc6401f6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.784+0000 7fc646973640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6400cf710 0x7fc640156bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.784+0000 7fc646973640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc646973640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc640159e60 con 0x7fc6400cfaf0 2023-10-14T16:54:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc646973640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc640157110 con 0x7fc6400cf710 2023-10-14T16:54:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc646973640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6401591e0 con 0x7fc6400b6cd0 2023-10-14T16:54:26.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45414/0 (socket says 172.21.15.29:45414) 2023-10-14T16:54:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 -- 172.21.15.29:0/1879652954 learned_addr learned my addr 172.21.15.29:0/1879652954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc63ffff640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400b6cd0 0x7fc6401f6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc637fff640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6400cf710 0x7fc640156bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:26.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400b6cd0 msgr2=0x7fc6401f6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400b6cd0 0x7fc6401f6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6400cf710 msgr2=0x7fc640156bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.785+0000 7fc644f10640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6400cf710 0x7fc640156bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc644f10640 1 -- 172.21.15.29:0/1879652954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc630009460 con 0x7fc6400cfaf0 2023-10-14T16:54:26.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc644f10640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc63800efc0 tx=0x7fc63800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc638009b20 con 0x7fc6400cfaf0 2023-10-14T16:54:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6401594c0 con 0x7fc6400cfaf0 2023-10-14T16:54:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc63800f040 con 0x7fc6400cfaf0 2023-10-14T16:54:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6380146c0 con 0x7fc6400cfaf0 2023-10-14T16:54:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.786+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc640159a80 con 0x7fc6400cfaf0 2023-10-14T16:54:26.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.788+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc64004e160 con 0x7fc6400cfaf0 2023-10-14T16:54:26.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.790+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc638014820 con 0x7fc6400cfaf0 2023-10-14T16:54:26.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.790+0000 7fc63d7fa640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 0x7fc62006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:26.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.790+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fc63808a5b0 con 0x7fc6400cfaf0 2023-10-14T16:54:26.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.791+0000 7fc63ffff640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 msgr2=0x7fc62006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:26.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.791+0000 7fc63ffff640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 0x7fc62006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:54:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.795+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc638055430 con 0x7fc6400cfaf0 2023-10-14T16:54:26.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.991+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fc64015f6e0 con 0x7fc6400cfaf0 2023-10-14T16:54:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.992+0000 7fc63ffff640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 msgr2=0x7fc62006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:26.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.992+0000 7fc63ffff640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 0x7fc62006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:54:26.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:26.996+0000 7fc63d7fa640 1 -- 172.21.15.29:0/1879652954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v18)=0 v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7fc638009e60 con 0x7fc6400cfaf0 2023-10-14T16:54:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 msgr2=0x7fc62006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:54:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 0x7fc62006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 msgr2=0x7fc6400cca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fc63800efc0 tx=0x7fc63800c490 comp rx=0 tx=0).stop 2023-10-14T16:54:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 shutdown_connections 2023-10-14T16:54:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7fc62006b010 0x7fc62006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6400cfaf0 0x7fc6400cca00 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6400cf710 0x7fc640156bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 --2- 172.21.15.29:0/1879652954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6400b6cd0 0x7fc6401f6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 >> 172.21.15.29:0/1879652954 conn(0x7fc6400d2cc0 msgr2=0x7fc6400b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 shutdown_connections 2023-10-14T16:54:27.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.000+0000 7fc646973640 1 -- 172.21.15.29:0/1879652954 wait complete. 2023-10-14T16:54:27.021 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:27.021 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T16:54:27.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-14T16:54:27.102 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.102+0000 7ff4f5445040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:54:27.154 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.154+0000 7ff4f5445040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:54:27.184 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.183+0000 7f83d620f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:54:27.205 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.204+0000 7ff4f5445040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:54:27.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.239+0000 7f83d620f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:54:27.265 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.264+0000 7ff4f5445040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:54:27.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.282+0000 7f421a9c4640 1 Processor -- start 2023-10-14T16:54:27.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.282+0000 7f421a9c4640 1 -- start start 2023-10-14T16:54:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a9c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4214065ab0 0x7f42140c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a9c4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42140ce280 0x7f42140ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a9c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140acd70 con 0x7f421406d220 2023-10-14T16:54:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a9c4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42140ac4a0 con 0x7f42140ce280 2023-10-14T16:54:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a9c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4214065f40 con 0x7f4214065ab0 2023-10-14T16:54:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a1c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42140ce280 0x7f42140ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a1c3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42140ce280 0x7f42140ad200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45426/0 (socket says 172.21.15.29:45426) 2023-10-14T16:54:27.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.284+0000 7f421a1c3640 1 -- 172.21.15.29:0/2449690987 learned_addr learned my addr 172.21.15.29:0/2449690987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.285+0000 7f42191c1640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4214065ab0 0x7f42140c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.285+0000 7f421a1c3640 1 -- 172.21.15.29:0/2449690987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4214065ab0 msgr2=0x7f42140c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.285+0000 7f421a1c3640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4214065ab0 0x7f42140c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.285+0000 7f421a1c3640 1 -- 172.21.15.29:0/2449690987 >> v1:172.21.15.186:6789/0 conn(0x7f421406d220 legacy=0x7f42140cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.285+0000 7f421a1c3640 1 -- 172.21.15.29:0/2449690987 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140ac720 con 0x7f42140ce280 2023-10-14T16:54:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.286+0000 7f421a1c3640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140ce280 0x7f42140ad200 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f420c009e50 tx=0x7f420c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0133269385b0f26 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.286+0000 7f42189c0640 1 -- 172.21.15.29:0/2449690987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c02cb00 con 0x7f42140ce280 2023-10-14T16:54:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.286+0000 7f42189c0640 1 -- 172.21.15.29:0/2449690987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c02cc60 con 0x7f42140ce280 2023-10-14T16:54:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.286+0000 7f42189c0640 1 -- 172.21.15.29:0/2449690987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c0343f0 con 0x7f42140ce280 2023-10-14T16:54:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 -- 172.21.15.29:0/2449690987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140ce280 msgr2=0x7f42140ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140ce280 0x7f42140ad200 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f420c009e50 tx=0x7f420c02bf60 comp rx=0 tx=0).stop 2023-10-14T16:54:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 -- 172.21.15.29:0/2449690987 shutdown_connections 2023-10-14T16:54:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140ce280 0x7f42140ad200 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 --2- 172.21.15.29:0/2449690987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4214065ab0 0x7f42140c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 -- 172.21.15.29:0/2449690987 >> 172.21.15.29:0/2449690987 conn(0x7f42140b0a80 msgr2=0x7f42140b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 -- 172.21.15.29:0/2449690987 shutdown_connections 2023-10-14T16:54:27.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.287+0000 7f421a9c4640 1 -- 172.21.15.29:0/2449690987 wait complete. 2023-10-14T16:54:27.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.289+0000 7f421a9c4640 1 Processor -- start 2023-10-14T16:54:27.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.289+0000 7f421a9c4640 1 -- start start 2023-10-14T16:54:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f421406d220 0x7f4214070840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42140ce280 0x7f42140b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 0x7f4214159570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f421415df60 con 0x7f42140b20f0 2023-10-14T16:54:27.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42140b2720 con 0x7f42140ce280 2023-10-14T16:54:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.290+0000 7f421a9c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42140b2a00 con 0x7f421406d220 2023-10-14T16:54:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 0x7f4214159570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 0x7f4214159570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45430/0 (socket says 172.21.15.29:45430) 2023-10-14T16:54:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 -- 172.21.15.29:0/3050970392 learned_addr learned my addr 172.21.15.29:0/3050970392 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:27.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f421406d220 0x7f4214070840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f42191c1640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42140ce280 0x7f42140b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f421406d220 msgr2=0x7f4214070840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f421406d220 0x7f4214070840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42140ce280 msgr2=0x7f42140b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42140ce280 0x7f42140b1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f420c009b30 con 0x7f42140b20f0 2023-10-14T16:54:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f421406d220 0x7f4214070840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:54:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.291+0000 7f421a1c3640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 0x7f4214159570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f420c009e50 tx=0x7f420c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.292+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c040670 con 0x7f42140b20f0 2023-10-14T16:54:27.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.292+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f420c0407d0 con 0x7f42140b20f0 2023-10-14T16:54:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.292+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f420c040b80 con 0x7f42140b20f0 2023-10-14T16:54:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.292+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42140b2ce0 con 0x7f42140b20f0 2023-10-14T16:54:27.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.292+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42140b32a0 con 0x7f42140b20f0 2023-10-14T16:54:27.298 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.293+0000 7f83d620f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:54:27.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.293+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f421404e9b0 con 0x7f42140b20f0 2023-10-14T16:54:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.294+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 19) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f420c040ce0 con 0x7f42140b20f0 2023-10-14T16:54:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.295+0000 7f420a7fc640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.295+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f420c0ac870 con 0x7f42140b20f0 2023-10-14T16:54:27.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.295+0000 7f42199c2640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:27.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.295+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:54:27.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.299+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f420c0777e0 con 0x7f42140b20f0 2023-10-14T16:54:27.316 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.315+0000 7ff4f5445040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:54:27.358 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.358+0000 7f83d620f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:54:27.411 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.410+0000 7f83d620f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:54:27.426 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.425+0000 7ff4f5445040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:54:27.488 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.488+0000 7ff4f5445040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:54:27.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.496+0000 7f42199c2640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:27.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.496+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:54:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.516+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f42140aca00 con 0x7f41f806afe0 2023-10-14T16:54:27.527 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.527+0000 7f83d620f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:54:27.539 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.538+0000 7ff4f5445040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:54:27.594 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.593+0000 7f83d620f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:54:27.597 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.596+0000 7ff4f5445040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:54:27.645 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.644+0000 7f83d620f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:54:27.647 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:27.646+0000 7ff4f5445040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:54:27.704 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.703+0000 7f83d620f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:54:27.757 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:27.757+0000 7f83d620f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:54:27.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.897+0000 7f42199c2640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:27.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:27.897+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:54:28.027 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.026+0000 7ff4f5445040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:54:28.155 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.154+0000 7f83d620f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:54:28.187 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.186+0000 7ff4f5445040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:54:28.293 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.292+0000 7ff4f5445040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:54:28.322 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.322+0000 7f83d620f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:54:28.343 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.342+0000 7ff4f5445040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:54:28.435 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.434+0000 7f83d620f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:54:28.450 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.449+0000 7ff4f5445040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:54:28.487 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.487+0000 7f83d620f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:54:28.557 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.556+0000 7ff4f5445040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:54:28.601 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.600+0000 7f83d620f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:54:28.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:28.698+0000 7f42199c2640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:28.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:28.698+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:54:28.713 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.712+0000 7ff4f5445040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:54:28.714 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.713+0000 7f83d620f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:54:28.762 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.761+0000 7ff4f5445040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:54:28.877 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.876+0000 7f83d620f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:54:28.880 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:28.879+0000 7ff4f5445040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:54:28.928 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:28.928+0000 7f83d620f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:54:29.053 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:29.052+0000 7f83d620f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:54:29.713 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:29.712+0000 7ff4f5445040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:54:29.936 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:29.936+0000 7f83d620f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:54:30.025 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:30.024+0000 7ff4f5445040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:54:30.083 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:30.082+0000 7ff4f5445040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:54:30.196 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:54:30.195+0000 7ff4f5445040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:54:30.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.247+0000 7f83d620f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:54:30.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:30.300+0000 7f42199c2640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/98172895 2023-10-14T16:54:30.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:30.300+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:54:30.307 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.306+0000 7f83d620f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:54:30.425 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.424+0000 7f83d620f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:54:30.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:30.444+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 20) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f420c0aad60 con 0x7f42140b20f0 2023-10-14T16:54:30.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:30.445+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 msgr2=0x7f41f806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:54:30.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:30.445+0000 7f420a7fc640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/98172895,v1:172.21.15.29:6833/98172895] conn(0x7f41f806afe0 0x7f41f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:30.797 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.796+0000 7f835343c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.797 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.796+0000 7f835343c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.797 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.796+0000 7f835343c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.798 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.796+0000 7f835343c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.798 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.796+0000 7f835343c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.798 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.797+0000 7f8355c81640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.797+0000 7f8355c81640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.797+0000 7f8355c81640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.797+0000 7f8355c81640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:30.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:54:30.797+0000 7f8355c81640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:54:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.458+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 21) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f420c07f290 con 0x7f42140b20f0 2023-10-14T16:54:31.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.459+0000 7f420a7fc640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f41f8035880 0x7f41f8074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.459+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 --> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f41f8035c60 con 0x7f41f8035880 2023-10-14T16:54:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.462+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f41f8035880 0x7f41f8074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.474+0000 7f42199c2640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f41f8035880 0x7f41f8074f30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4210000f60 tx=0x7f42100011c0 comp rx=0 tx=0).ready entity=mgr.6156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:31.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.665+0000 7f420a7fc640 1 -- 172.21.15.29:0/3050970392 <== mgr.6156 v2:172.21.15.29:6832/2060982901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f41f8035c60 con 0x7f41f8035880 2023-10-14T16:54:31.666 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T16:54:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f41f8035880 msgr2=0x7f41f8074f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f41f8035880 0x7f41f8074f30 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4210000f60 tx=0x7f42100011c0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 msgr2=0x7f4214159570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:31.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 --2- 172.21.15.29:0/3050970392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42140b20f0 0x7f4214159570 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f420c009e50 tx=0x7f420c004a40 comp rx=0 tx=0).stop 2023-10-14T16:54:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a1c3640 1 -- 172.21.15.29:0/3050970392 reap_dead start 2023-10-14T16:54:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 shutdown_connections 2023-10-14T16:54:31.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.668+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 >> 172.21.15.29:0/3050970392 conn(0x7f42140b0a80 msgr2=0x7f421415c470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:31.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.669+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 shutdown_connections 2023-10-14T16:54:31.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.669+0000 7f421a9c4640 1 -- 172.21.15.29:0/3050970392 wait complete. 2023-10-14T16:54:31.690 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T16:54:31.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 2m 2023-10-14T16:54:31.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.921+0000 7febe5fd2640 1 Processor -- start 2023-10-14T16:54:31.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.921+0000 7febe5fd2640 1 -- start start 2023-10-14T16:54:31.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.922+0000 7febe5fd2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febe0159950 0x7febe015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.922+0000 7febe5fd2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febe0070450 0x7febe00708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.922+0000 7febe5fd2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe015aef0 con 0x7febe006c100 2023-10-14T16:54:31.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.922+0000 7febe5fd2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe0070e20 con 0x7febe0070450 2023-10-14T16:54:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.922+0000 7febe5fd2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe015e490 con 0x7febe0159950 2023-10-14T16:54:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febe0070450 0x7febe00708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7febe0070450 0x7febe00708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52666/0 (socket says 172.21.15.29:52666) 2023-10-14T16:54:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 -- 172.21.15.29:0/3920188532 learned_addr learned my addr 172.21.15.29:0/3920188532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febe4fd0640 1 --2- 172.21.15.29:0/3920188532 >> v2:172.21.15.186:3300/0 conn(0x7febe0159950 0x7febe015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 -- 172.21.15.29:0/3920188532 >> v2:172.21.15.186:3300/0 conn(0x7febe0159950 msgr2=0x7febe015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 --2- 172.21.15.29:0/3920188532 >> v2:172.21.15.186:3300/0 conn(0x7febe0159950 0x7febe015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.923+0000 7febd7fff640 1 -- 172.21.15.29:0/3920188532 >> v1:172.21.15.186:6789/0 conn(0x7febe006c100 legacy=0x7febe006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.924+0000 7febd7fff640 1 -- 172.21.15.29:0/3920188532 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe015d910 con 0x7febe0070450 2023-10-14T16:54:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.925+0000 7febd7fff640 1 --2- 172.21.15.29:0/3920188532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0070450 0x7febe00708e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7febe00ca950 tx=0x7febd002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfc4a952d181e103 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.925+0000 7febd77fe640 1 -- 172.21.15.29:0/3920188532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd00317e0 con 0x7febe0070450 2023-10-14T16:54:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.925+0000 7febd77fe640 1 -- 172.21.15.29:0/3920188532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd0031940 con 0x7febe0070450 2023-10-14T16:54:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.925+0000 7febd77fe640 1 -- 172.21.15.29:0/3920188532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd0031cd0 con 0x7febe0070450 2023-10-14T16:54:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 -- 172.21.15.29:0/3920188532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0070450 msgr2=0x7febe00708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 --2- 172.21.15.29:0/3920188532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0070450 0x7febe00708e0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7febe00ca950 tx=0x7febd002b880 comp rx=0 tx=0).stop 2023-10-14T16:54:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 -- 172.21.15.29:0/3920188532 shutdown_connections 2023-10-14T16:54:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 --2- 172.21.15.29:0/3920188532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0070450 0x7febe00708e0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 --2- 172.21.15.29:0/3920188532 >> v2:172.21.15.186:3300/0 conn(0x7febe0159950 0x7febe015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 -- 172.21.15.29:0/3920188532 >> 172.21.15.29:0/3920188532 conn(0x7febe00ce6a0 msgr2=0x7febe00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:31.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 -- 172.21.15.29:0/3920188532 shutdown_connections 2023-10-14T16:54:31.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.926+0000 7febe5fd2640 1 -- 172.21.15.29:0/3920188532 wait complete. 2023-10-14T16:54:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.928+0000 7febe5fd2640 1 Processor -- start 2023-10-14T16:54:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.928+0000 7febe5fd2640 1 -- start start 2023-10-14T16:54:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0070450 0x7febe00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febe0159950 0x7febe00c75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00c86a0 con 0x7febe006c100 2023-10-14T16:54:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe00c5e50 con 0x7febe0070450 2023-10-14T16:54:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.929+0000 7febe5fd2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe00c80e0 con 0x7febe0159950 2023-10-14T16:54:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52682/0 (socket says 172.21.15.29:52682) 2023-10-14T16:54:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 -- 172.21.15.29:0/1117375179 learned_addr learned my addr 172.21.15.29:0/1117375179 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febd7fff640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0070450 0x7febe00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 -- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febe0159950 msgr2=0x7febe00c75d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febe0159950 0x7febe00c75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 -- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0070450 msgr2=0x7febe00c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0070450 0x7febe00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 -- 172.21.15.29:0/1117375179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd00094d0 con 0x7febe006c100 2023-10-14T16:54:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.930+0000 7febe4fd0640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7febdc007e70 tx=0x7febdc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.931+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc014620 con 0x7febe006c100 2023-10-14T16:54:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.931+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc014780 con 0x7febe006c100 2023-10-14T16:54:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.931+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febdc014b30 con 0x7febe006c100 2023-10-14T16:54:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.931+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe00c83c0 con 0x7febe006c100 2023-10-14T16:54:31.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.931+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe00c7b10 con 0x7febe006c100 2023-10-14T16:54:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.932+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe00c2340 con 0x7febe006c100 2023-10-14T16:54:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.933+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 21) v1 ==== 89084+0+0 (secure 0 0 0) 0x7febdc014c90 con 0x7febe006c100 2023-10-14T16:54:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.934+0000 7febd57fa640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 0x7febcc06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:31.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.934+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6097+0+0 (secure 0 0 0) 0x7febdc089e40 con 0x7febe006c100 2023-10-14T16:54:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.934+0000 7febd7fff640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 0x7febcc06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.935+0000 7febd7fff640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 0x7febcc06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7febd0036000 tx=0x7febd003d040 comp rx=0 tx=0).ready entity=mgr.6156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:31.939+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febe00c2340 con 0x7febe006c100 2023-10-14T16:54:32.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.158+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 --> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "2m", "target": ["mon-mgr", ""]}) v1 -- 0x7febe015ded0 con 0x7febcc06ad10 2023-10-14T16:54:32.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.320+0000 7febd57fa640 1 -- 172.21.15.29:0/1117375179 <== mgr.6156 v2:172.21.15.29:6832/2060982901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7febd0031b30 con 0x7febcc06ad10 2023-10-14T16:54:32.321 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T16:54:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.323+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 msgr2=0x7febcc06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.323+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 0x7febcc06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7febd0036000 tx=0x7febd003d040 comp rx=0 tx=0).stop 2023-10-14T16:54:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.323+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 msgr2=0x7febe00c4bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.323+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7febdc007e70 tx=0x7febdc00e4b0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 shutdown_connections 2023-10-14T16:54:32.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7febcc06ad10 0x7febcc06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febe0159950 0x7febe00c75d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0070450 0x7febe00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 --2- 172.21.15.29:0/1117375179 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe006c100 0x7febe00c4bd0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 >> 172.21.15.29:0/1117375179 conn(0x7febe00ce6a0 msgr2=0x7febe00af9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:32.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 shutdown_connections 2023-10-14T16:54:32.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.324+0000 7febe5fd2640 1 -- 172.21.15.29:0/1117375179 wait complete. 2023-10-14T16:54:32.346 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T16:54:32.347 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-14-16_55_00 in ~28s... 2023-10-14T16:54:32.347 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-14-16_56_00 in ~88s... 2023-10-14T16:54:32.347 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m', '2m'], expected_retention: [] 2023-10-14T16:54:32.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-14T16:54:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.581+0000 7f579abc5640 1 Processor -- start 2023-10-14T16:54:32.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.581+0000 7f579abc5640 1 -- start start 2023-10-14T16:54:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f579415ad70 0x7f5794069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940c9ca0 0x7f579406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57940ca3b0 0x7f57940ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57940cad80 con 0x7f57940ca3b0 2023-10-14T16:54:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57940cb030 con 0x7f579415ad70 2023-10-14T16:54:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f579abc5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940c91c0 con 0x7f57940c9ca0 2023-10-14T16:54:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f5799162640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57940ca3b0 0x7f57940ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f5799162640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f57940ca3b0 0x7f57940ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52718/0 (socket says 172.21.15.29:52718) 2023-10-14T16:54:32.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.582+0000 7f5799162640 1 -- 172.21.15.29:0/156670522 learned_addr learned my addr 172.21.15.29:0/156670522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5793fff640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940c9ca0 0x7f579406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5798961640 1 --2- 172.21.15.29:0/156670522 >> v2:172.21.15.186:3300/0 conn(0x7f579415ad70 0x7f5794069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5799162640 1 -- 172.21.15.29:0/156670522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940c9ca0 msgr2=0x7f579406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5799162640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940c9ca0 0x7f579406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5799162640 1 -- 172.21.15.29:0/156670522 >> v2:172.21.15.186:3300/0 conn(0x7f579415ad70 msgr2=0x7f5794069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:54:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5799162640 1 --2- 172.21.15.29:0/156670522 >> v2:172.21.15.186:3300/0 conn(0x7f579415ad70 0x7f5794069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.583+0000 7f5799162640 1 -- 172.21.15.29:0/156670522 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57940c94a0 con 0x7f57940ca3b0 2023-10-14T16:54:32.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.584+0000 7f5799162640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940ca3b0 0x7f57940ca840 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f57940cf780 tx=0x7f578c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c09509996f016741 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.584+0000 7f57937fe640 1 -- 172.21.15.29:0/156670522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c02ca80 con 0x7f57940ca3b0 2023-10-14T16:54:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f57937fe640 1 -- 172.21.15.29:0/156670522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f578c02cbe0 con 0x7f57940ca3b0 2023-10-14T16:54:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f57937fe640 1 -- 172.21.15.29:0/156670522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f578c0343f0 con 0x7f57940ca3b0 2023-10-14T16:54:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 -- 172.21.15.29:0/156670522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940ca3b0 msgr2=0x7f57940ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940ca3b0 0x7f57940ca840 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f57940cf780 tx=0x7f578c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 -- 172.21.15.29:0/156670522 shutdown_connections 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940ca3b0 0x7f57940ca840 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 --2- 172.21.15.29:0/156670522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940c9ca0 0x7f579406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 --2- 172.21.15.29:0/156670522 >> v2:172.21.15.186:3300/0 conn(0x7f579415ad70 0x7f5794069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 -- 172.21.15.29:0/156670522 >> 172.21.15.29:0/156670522 conn(0x7f57940ce9b0 msgr2=0x7f57940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:32.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 -- 172.21.15.29:0/156670522 shutdown_connections 2023-10-14T16:54:32.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.585+0000 7f579abc5640 1 -- 172.21.15.29:0/156670522 wait complete. 2023-10-14T16:54:32.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.588+0000 7f579abc5640 1 Processor -- start 2023-10-14T16:54:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.588+0000 7f579abc5640 1 -- start start 2023-10-14T16:54:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940ca3b0 0x7f57940c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 0x7f57940c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940c8a30 con 0x7f57940c9ca0 2023-10-14T16:54:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57940c60d0 con 0x7f579415ad70 2023-10-14T16:54:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f579abc5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57940c63b0 con 0x7f57940ca3b0 2023-10-14T16:54:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f5798961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f5798961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52722/0 (socket says 172.21.15.29:52722) 2023-10-14T16:54:32.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.590+0000 7f5798961640 1 -- 172.21.15.29:0/1851539169 learned_addr learned my addr 172.21.15.29:0/1851539169 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:54:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5793fff640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940ca3b0 0x7f57940c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5799162640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 0x7f57940c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 -- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940ca3b0 msgr2=0x7f57940c7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940ca3b0 0x7f57940c7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 -- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 msgr2=0x7f57940c84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 0x7f57940c84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 -- 172.21.15.29:0/1851539169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c009b30 con 0x7f57940c9ca0 2023-10-14T16:54:32.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5799162640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 0x7f57940c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:54:32.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f5798961640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f578800d5f0 tx=0x7f578800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:32.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.591+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57880125f0 con 0x7f57940c9ca0 2023-10-14T16:54:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.592+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5788012750 con 0x7f57940c9ca0 2023-10-14T16:54:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.592+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57940c6690 con 0x7f57940c9ca0 2023-10-14T16:54:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.592+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5788012b00 con 0x7f57940c9ca0 2023-10-14T16:54:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.592+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57940c6c50 con 0x7f57940c9ca0 2023-10-14T16:54:32.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.593+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f579404e160 con 0x7f57940c9ca0 2023-10-14T16:54:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.594+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 22) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f5788012c60 con 0x7f57940c9ca0 2023-10-14T16:54:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.595+0000 7f57917fa640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 0x7f577806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:54:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.595+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f5788089e10 con 0x7f57940c9ca0 2023-10-14T16:54:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.595+0000 7f5793fff640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 0x7f577806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:54:32.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.596+0000 7f5793fff640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 0x7f577806d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57940cdca0 tx=0x7f5780008040 comp rx=0 tx=0).ready entity=mgr.6156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:54:32.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.599+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5788054c20 con 0x7f57940c9ca0 2023-10-14T16:54:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.818+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 --> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f57940c9780 con 0x7f577806afe0 2023-10-14T16:54:32.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.834+0000 7f57917fa640 1 -- 172.21.15.29:0/1851539169 <== mgr.6156 v2:172.21.15.29:6832/2060982901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+76 (secure 0 0 0) 0x7f578000b640 con 0x7f577806afe0 2023-10-14T16:54:32.835 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:54:32.835 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-14T16:54:32.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 msgr2=0x7f577806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 0x7f577806d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f57940cdca0 tx=0x7f5780008040 comp rx=0 tx=0).stop 2023-10-14T16:54:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 msgr2=0x7f57940c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:54:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f578800d5f0 tx=0x7f578800dab0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 shutdown_connections 2023-10-14T16:54:32.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f577806afe0 0x7f577806d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f579415ad70 0x7f57940c84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57940ca3b0 0x7f57940c7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 --2- 172.21.15.29:0/1851539169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57940c9ca0 0x7f57940c5a80 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:54:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 >> 172.21.15.29:0/1851539169 conn(0x7f57940ce9b0 msgr2=0x7f57940ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:54:32.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 shutdown_connections 2023-10-14T16:54:32.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:54:32.837+0000 7f579abc5640 1 -- 172.21.15.29:0/1851539169 wait complete. 2023-10-14T16:54:32.857 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m", "2m"], "retention": [{}, {}]} 2023-10-14T16:54:32.858 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m', '2m'], 'retention': [{}, {}]} 2023-10-14T16:54:32.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:32.858 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:32.917 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:32.917 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:32.918 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:33.918 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:33.919 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:33.988 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:33.989 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:33.989 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:34.990 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:34.991 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:35.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:35.058 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:35.058 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:36.060 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:36.060 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:36.129 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:36.130 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:36.130 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:37.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:37.132 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:37.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:37.200 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:37.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:38.202 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:38.202 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:38.270 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:38.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:38.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:39.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:39.273 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:39.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:39.341 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:39.342 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:40.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:40.344 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:40.411 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:40.411 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:40.412 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:41.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:41.413 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:41.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:41.481 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:41.482 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:42.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:42.484 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:42.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:42.552 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:42.552 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:43.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:43.554 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:43.623 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:43.623 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:43.623 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:44.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:44.625 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:44.693 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:44.694 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:44.694 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:45.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:54:45.081 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:54:45.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:45.696 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:45.765 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:45.766 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:45.766 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:46.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:46.768 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:46.836 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:46.837 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:46.837 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:47.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:47.839 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:47.905 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:47.906 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:47.906 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:48.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:48.908 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:48.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:48.974 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:48.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:49.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:49.976 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:50.042 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:50.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:50.043 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:51.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:51.045 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:51.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:51.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:51.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:52.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:52.113 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:52.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:52.180 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:52.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:53.182 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:53.182 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:53.250 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:53.251 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:53.251 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:54.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:54.252 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:54.320 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:54.321 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:54.321 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:55.322 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:55.323 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:55.391 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:55.391 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:55.392 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:56.393 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:56.394 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:56.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:56.465 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:56.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:57.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:57.467 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:57.537 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:57.538 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:57.538 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:58.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:58.540 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:58.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:58.607 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:58.608 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:54:59.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:54:59.609 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:54:59.677 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:54:59.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:54:59.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:00.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:55:00.346+0000 7f8345420640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:55:00.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:55:00.346+0000 7f8345420640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:55:00.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:55:00.346+0000 7f8345420640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:55:00.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:55:00.346+0000 7f8345420640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:55:00.348 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:55:00.346+0000 7f8345420640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:55:00.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:00.680 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:00.745 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:00.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:00.747 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:00.747 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:00.748 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-14-16_55_00_UTC'] 2023-10-14T16:55:00.748 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697302472.3470364, now = 1697302500.7484918, timo = 28 2023-10-14T16:55:01.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:01.750 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:01.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:01.817 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:01.818 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:01.818 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:02.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:02.820 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:02.887 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:02.888 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:02.889 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:02.889 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:03.890 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:03.891 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:03.958 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:03.960 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:03.960 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:03.960 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:04.962 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:04.962 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:05.029 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:05.031 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:05.031 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:05.031 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:06.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:06.033 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:06.100 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:06.101 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:06.101 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:06.101 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:07.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:07.103 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:07.171 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:07.172 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:07.173 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:07.173 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:08.174 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:08.175 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:08.241 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:08.243 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:08.243 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:08.243 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:09.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:09.245 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:09.310 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:09.312 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:09.312 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:09.312 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:10.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:10.314 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:10.378 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:10.380 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:10.380 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:10.381 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:11.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:11.382 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:11.450 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:11.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:11.452 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:11.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:12.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:12.454 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:12.519 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:12.520 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:12.521 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:12.521 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:13.522 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:13.523 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:13.590 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:13.591 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:13.592 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:13.592 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:14.593 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:14.594 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:14.663 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:14.664 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:14.665 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:14.665 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:15.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:55:15.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:55:15.666 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:15.667 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:15.733 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:15.735 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:15.736 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:15.736 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:16.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:16.738 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:16.804 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:16.805 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:16.806 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:16.806 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:17.807 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:17.808 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:17.875 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:17.876 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:17.876 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:17.877 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:18.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:18.878 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:18.945 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:18.947 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:18.947 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:18.947 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:19.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:19.949 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:20.015 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:20.016 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:20.017 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:20.017 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:21.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:21.019 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:21.086 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:21.088 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:21.088 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:21.088 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:22.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:22.090 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:22.157 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:22.158 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:22.159 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:22.159 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:23.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:23.161 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:23.227 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:23.229 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:23.229 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:23.230 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:24.231 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:24.232 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:24.293 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:24.294 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:24.295 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:24.295 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:25.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:25.296 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:25.360 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:25.361 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:25.361 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:25.362 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:26.365 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:26.366 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:26.434 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:26.436 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:26.436 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:26.436 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:27.437 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:27.438 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:27.503 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:27.505 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:27.505 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:27.505 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:28.506 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:28.507 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:28.575 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:28.576 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:28.576 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:28.577 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:29.577 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:29.578 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:29.644 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:29.645 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:29.645 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:29.646 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:30.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:30.647 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:30.709 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:30.711 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:30.711 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:30.711 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:31.712 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:31.713 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:31.779 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:31.781 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:31.781 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:31.781 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:32.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:32.783 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:32.848 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:32.850 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:32.850 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:32.850 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:33.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:33.852 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:33.918 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:33.919 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:33.919 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:33.920 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:34.921 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:34.921 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:34.986 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:34.987 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:34.987 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:34.987 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:35.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:35.989 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:36.055 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:36.057 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:36.057 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:36.058 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:37.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:37.059 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:37.129 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:37.130 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:37.131 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:37.131 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:38.132 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:38.133 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:38.199 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:38.200 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:38.201 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:38.201 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:39.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:39.203 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:39.269 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:39.270 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:39.271 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:39.271 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:40.272 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:40.272 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:40.339 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:40.340 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:40.340 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:40.341 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:41.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:41.342 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:41.408 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:41.410 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:41.410 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:41.411 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:42.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:42.413 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:42.479 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:42.481 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:42.481 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:42.481 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:43.482 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:43.483 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:43.551 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:43.552 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:43.553 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:43.553 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:44.554 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:44.555 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:44.620 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:44.621 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:44.622 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:44.622 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:45.376 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:55:45.386 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:55:45.623 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:45.623 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:45.689 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:45.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:45.691 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:45.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:46.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:46.693 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:46.761 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:46.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:46.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:46.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:47.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:47.764 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:47.829 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:47.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:47.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:47.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:48.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:48.833 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:48.900 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:48.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:48.901 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:48.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:49.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:49.903 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:49.969 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:49.970 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:49.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:49.971 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:50.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:50.975 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:51.040 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:51.042 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:51.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:51.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:52.043 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:52.044 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:52.109 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:52.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:52.111 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:52.111 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:53.112 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:53.113 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:53.179 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:53.180 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:53.181 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:53.181 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:54.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:54.183 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:54.249 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:54.251 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:54.251 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:54.251 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:55.253 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:55.253 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:55.319 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:55.320 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:55.320 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:55.320 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:56.321 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:56.322 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:56.389 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:56.390 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:56.390 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:56.391 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:57.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:57.393 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:57.460 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:57.461 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:57.462 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:57.462 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:58.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:58.464 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:58.532 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:58.533 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:58.533 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:58.534 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:55:59.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:55:59.536 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:55:59.602 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:55:59.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:55:59.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:55:59.607 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:56:00.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:00.608 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:56:00.674 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:56:00.674 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_56_00_UTC 2023-10-14T16:56:00.675 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-16_56_00_UTC', 'scheduled-2023-10-14-16_55_00_UTC'} 2023-10-14T16:56:00.676 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-14-16_56_00_UTC'} 2023-10-14T16:56:00.676 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:56:00.676 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-14-16_56_00_UTC'] 2023-10-14T16:56:00.677 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697302472.3470364, now = 1697302560.6771708, timo = 88 2023-10-14T16:56:01.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-14T16:56:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.925+0000 7f4bbed65640 1 Processor -- start 2023-10-14T16:56:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.925+0000 7f4bbed65640 1 -- start start 2023-10-14T16:56:01.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbed65640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4bb815d150 0x7f4bb80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbed65640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4bb8160b90 0x7f4bb8160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbed65640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bb80ac890 con 0x7f4bb8069da0 2023-10-14T16:56:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbed65640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb815d530 con 0x7f4bb815d150 2023-10-14T16:56:01.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbed65640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bb8161650 con 0x7f4bb8160b90 2023-10-14T16:56:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbcb01640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4bb815d150 0x7f4bb80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbcb01640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4bb815d150 0x7f4bb80ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44970/0 (socket says 172.21.15.29:44970) 2023-10-14T16:56:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbcb01640 1 -- 172.21.15.29:0/1186683368 learned_addr learned my addr 172.21.15.29:0/1186683368 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.927+0000 7f4bbd302640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4bb8069da0 0x7f4bb80c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40580/0 (socket says 172.21.15.29:40580) 2023-10-14T16:56:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bb7fff640 1 --2- 172.21.15.29:0/1186683368 >> v2:172.21.15.186:3300/0 conn(0x7f4bb8160b90 0x7f4bb8160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bbcb01640 1 -- 172.21.15.29:0/1186683368 >> v2:172.21.15.186:3300/0 conn(0x7f4bb8160b90 msgr2=0x7f4bb8160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bbcb01640 1 --2- 172.21.15.29:0/1186683368 >> v2:172.21.15.186:3300/0 conn(0x7f4bb8160b90 0x7f4bb8160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bbcb01640 1 -- 172.21.15.29:0/1186683368 >> v1:172.21.15.29:6789/0 conn(0x7f4bb8069da0 legacy=0x7f4bb80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bbcb01640 1 -- 172.21.15.29:0/1186683368 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8161930 con 0x7f4bb815d150 2023-10-14T16:56:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.928+0000 7f4bb77fe640 1 -- 172.21.15.29:0/1186683368 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 123228718 0 0) 0x7f4bb80ac890 con 0x7f4bb8069da0 2023-10-14T16:56:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.929+0000 7f4bbcb01640 1 --2- 172.21.15.29:0/1186683368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb815d150 0x7f4bb80ad350 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80cf990 tx=0x7f4bac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=709d00e8e4db3a1c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.929+0000 7f4bb77fe640 1 -- 172.21.15.29:0/1186683368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac031870 con 0x7f4bb815d150 2023-10-14T16:56:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.929+0000 7f4bb77fe640 1 -- 172.21.15.29:0/1186683368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bac0319d0 con 0x7f4bb815d150 2023-10-14T16:56:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.929+0000 7f4bb77fe640 1 -- 172.21.15.29:0/1186683368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bac031d60 con 0x7f4bb815d150 2023-10-14T16:56:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 -- 172.21.15.29:0/1186683368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb815d150 msgr2=0x7f4bb80ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 --2- 172.21.15.29:0/1186683368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb815d150 0x7f4bb80ad350 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f4bb80cf990 tx=0x7f4bac02b910 comp rx=0 tx=0).stop 2023-10-14T16:56:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 -- 172.21.15.29:0/1186683368 shutdown_connections 2023-10-14T16:56:01.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 --2- 172.21.15.29:0/1186683368 >> v2:172.21.15.186:3300/0 conn(0x7f4bb8160b90 0x7f4bb8160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 --2- 172.21.15.29:0/1186683368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb815d150 0x7f4bb80ad350 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 -- 172.21.15.29:0/1186683368 >> 172.21.15.29:0/1186683368 conn(0x7f4bb80ce9b0 msgr2=0x7f4bb80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 -- 172.21.15.29:0/1186683368 shutdown_connections 2023-10-14T16:56:01.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.930+0000 7f4bbed65640 1 -- 172.21.15.29:0/1186683368 wait complete. 2023-10-14T16:56:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.932+0000 7f4bbed65640 1 Processor -- start 2023-10-14T16:56:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.932+0000 7f4bbed65640 1 -- start start 2023-10-14T16:56:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4bb8069da0 0x7f4bb81f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bb815d150 0x7f4bb81f21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb815e620 con 0x7f4bb8160b90 2023-10-14T16:56:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb81f3900 con 0x7f4bb815d150 2023-10-14T16:56:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbed65640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bb815da10 con 0x7f4bb8069da0 2023-10-14T16:56:01.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbd302640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbd302640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44978/0 (socket says 172.21.15.29:44978) 2023-10-14T16:56:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.933+0000 7f4bbd302640 1 -- 172.21.15.29:0/760877552 learned_addr learned my addr 172.21.15.29:0/760877552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bb7fff640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bb815d150 0x7f4bb81f21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbcb01640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4bb8069da0 0x7f4bb81f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 -- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4bb8069da0 msgr2=0x7f4bb81f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4bb8069da0 0x7f4bb81f33c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 -- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bb815d150 msgr2=0x7f4bb81f21c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bb815d150 0x7f4bb81f21c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 -- 172.21.15.29:0/760877552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bac0094d0 con 0x7f4bb8160b90 2023-10-14T16:56:01.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bbd302640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c7b0 tx=0x7f4bb000cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.934+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000a640 con 0x7f4bb8160b90 2023-10-14T16:56:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.935+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb000a7c0 con 0x7f4bb8160b90 2023-10-14T16:56:01.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.935+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4bb000ab70 con 0x7f4bb8160b90 2023-10-14T16:56:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.935+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bb815dcf0 con 0x7f4bb8160b90 2023-10-14T16:56:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.935+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bb815e2b0 con 0x7f4bb8160b90 2023-10-14T16:56:01.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.936+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bb804e160 con 0x7f4bb8160b90 2023-10-14T16:56:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.937+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f4bb000acd0 con 0x7f4bb8160b90 2023-10-14T16:56:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.938+0000 7f4bb57fa640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 0x7f4b9c06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:01.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.938+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6155+0+0 (secure 0 0 0) 0x7f4bb00890d0 con 0x7f4bb8160b90 2023-10-14T16:56:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.938+0000 7f4bbcb01640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 0x7f4b9c06d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.939+0000 7f4bbcb01640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 0x7f4b9c06d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4bac036000 tx=0x7f4bac03c040 comp rx=0 tx=0).ready entity=mgr.6156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:01.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:01.943+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4bb0053bc0 con 0x7f4bb8160b90 2023-10-14T16:56:02.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.165+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 --> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4bb80c24e0 con 0x7f4b9c06b130 2023-10-14T16:56:02.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.355+0000 7f4bb57fa640 1 -- 172.21.15.29:0/760877552 <== mgr.6156 v2:172.21.15.29:6832/2060982901 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f4bb80c24e0 con 0x7f4b9c06b130 2023-10-14T16:56:02.356 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-14T16:56:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 msgr2=0x7f4b9c06d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 0x7f4b9c06d5f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4bac036000 tx=0x7f4bac03c040 comp rx=0 tx=0).stop 2023-10-14T16:56:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 msgr2=0x7f4bb80b32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f4bb000c7b0 tx=0x7f4bb000cc70 comp rx=0 tx=0).stop 2023-10-14T16:56:02.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 shutdown_connections 2023-10-14T16:56:02.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f4b9c06b130 0x7f4b9c06d5f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4bb8160b90 0x7f4bb80b32e0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4bb815d150 0x7f4bb81f21c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 --2- 172.21.15.29:0/760877552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4bb8069da0 0x7f4bb81f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 >> 172.21.15.29:0/760877552 conn(0x7f4bb80ce9b0 msgr2=0x7f4bb8070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:02.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 shutdown_connections 2023-10-14T16:56:02.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.359+0000 7f4bbed65640 1 -- 172.21.15.29:0/760877552 wait complete. 2023-10-14T16:56:02.382 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-14T16:56:02.382 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:02.383 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:56:02.440 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:56:02.440 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-16_56_00_UTC 2023-10-14T16:56:02.441 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-16_55_00_UTC 2023-10-14T16:56:02.442 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:02.442 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-16_55_00_UTC) 2023-10-14T16:56:02.511 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-16_56_00_UTC 2023-10-14T16:56:02.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:02.512 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-16_56_00_UTC) 2023-10-14T16:56:02.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:02.584 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T16:56:02.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T16:56:02.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.894+0000 7f11cc2d4640 1 Processor -- start 2023-10-14T16:56:02.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.894+0000 7f11cc2d4640 1 -- start start 2023-10-14T16:56:02.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11cc2d4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11cc2d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 0x7f11c40ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11cc2d4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11c40cb030 con 0x7f11c415ad70 2023-10-14T16:56:02.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11cc2d4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11c40cae30 con 0x7f11c406a4b0 2023-10-14T16:56:02.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11cc2d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c40c91c0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.896+0000 7f11c986f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11c986f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:51052/0 (socket says 172.21.15.29:51052) 2023-10-14T16:56:02.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11ca871640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 0x7f11c40ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11c986f640 1 -- 172.21.15.29:0/2738866749 learned_addr learned my addr 172.21.15.29:0/2738866749 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:02.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11ca871640 1 -- 172.21.15.29:0/2738866749 >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 msgr2=0x7f11c40ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:02.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11ca871640 1 --2- 172.21.15.29:0/2738866749 >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.897+0000 7f11ca871640 1 -- 172.21.15.29:0/2738866749 >> v1:172.21.15.186:6789/0 conn(0x7f11c415ad70 legacy=0x7f11c4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.898+0000 7f11ca871640 1 -- 172.21.15.29:0/2738866749 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11c40c94a0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.898+0000 7f11c986f640 1 --2- 172.21.15.29:0/2738866749 >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:56:02.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.899+0000 7f11ca871640 1 --2- 172.21.15.29:0/2738866749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 0x7f11c40ca8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009d90 tx=0x7f11bc02beb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e21a0cdcb0235b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:02.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.899+0000 7f11c906e640 1 -- 172.21.15.29:0/2738866749 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc02cc60 con 0x7f11c40c9ca0 2023-10-14T16:56:02.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.899+0000 7f11c906e640 1 -- 172.21.15.29:0/2738866749 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11bc02cdc0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.899+0000 7f11c906e640 1 -- 172.21.15.29:0/2738866749 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc034660 con 0x7f11c40c9ca0 2023-10-14T16:56:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 -- 172.21.15.29:0/2738866749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 msgr2=0x7f11c40ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/2738866749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 0x7f11c40ca8f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009d90 tx=0x7f11bc02beb0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 -- 172.21.15.29:0/2738866749 shutdown_connections 2023-10-14T16:56:02.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/2738866749 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c40c9ca0 0x7f11c40ca8f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/2738866749 >> v2:172.21.15.186:3300/0 conn(0x7f11c406a4b0 0x7f11c40ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 -- 172.21.15.29:0/2738866749 >> 172.21.15.29:0/2738866749 conn(0x7f11c40ce9b0 msgr2=0x7f11c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 -- 172.21.15.29:0/2738866749 shutdown_connections 2023-10-14T16:56:02.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.900+0000 7f11cc2d4640 1 -- 172.21.15.29:0/2738866749 wait complete. 2023-10-14T16:56:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.903+0000 7f11cc2d4640 1 Processor -- start 2023-10-14T16:56:02.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.903+0000 7f11cc2d4640 1 -- start start 2023-10-14T16:56:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11c406a4b0 0x7f11c41597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c415ad70 0x7f11c40b2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c415e3b0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11c41f63e0 con 0x7f11c406a4b0 2023-10-14T16:56:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.904+0000 7f11cc2d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11c41f6100 con 0x7f11c415ad70 2023-10-14T16:56:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:44984/0 (socket says 172.21.15.29:44984) 2023-10-14T16:56:02.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 -- 172.21.15.29:0/239599557 learned_addr learned my addr 172.21.15.29:0/239599557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11ca070640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11c406a4b0 0x7f11c41597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11ca871640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c415ad70 0x7f11c40b2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 -- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c415ad70 msgr2=0x7f11c40b2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c415ad70 0x7f11c40b2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 -- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11c406a4b0 msgr2=0x7f11c41597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11c406a4b0 0x7f11c41597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 -- 172.21.15.29:0/239599557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b40094d0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.905+0000 7f11c986f640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f11b400e9d0 tx=0x7f11b400ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.906+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b400ca60 con 0x7f11c40c9ca0 2023-10-14T16:56:02.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.906+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11b400cbc0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.906+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4013470 con 0x7f11c40c9ca0 2023-10-14T16:56:02.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.906+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11bc009a70 con 0x7f11c40c9ca0 2023-10-14T16:56:02.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.906+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11c415cef0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.907+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11c404e160 con 0x7f11c40c9ca0 2023-10-14T16:56:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.908+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 24) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f11b40135d0 con 0x7f11c40c9ca0 2023-10-14T16:56:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.909+0000 7f11baffd640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 0x7f11a806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.909+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6155+0+0 (secure 0 0 0) 0x7f11b4089f90 con 0x7f11c40c9ca0 2023-10-14T16:56:02.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.909+0000 7f11ca070640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 0x7f11a806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:02.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.910+0000 7f11ca070640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 0x7f11a806d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000f60 tx=0x7f11c0009040 comp rx=0 tx=0).ready entity=mgr.6156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:02.915+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11b4054b10 con 0x7f11c40c9ca0 2023-10-14T16:56:03.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.162+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f11c40736e0 con 0x7f11c40c9ca0 2023-10-14T16:56:03.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.212+0000 7f11baffd640 1 -- 172.21.15.29:0/239599557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v25) v1 ==== 93+0+0 (secure 0 0 0) 0x7f11b40170a0 con 0x7f11c40c9ca0 2023-10-14T16:56:03.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.215+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 msgr2=0x7f11a806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.215+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 0x7f11a806d5f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f11c0000f60 tx=0x7f11c0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:03.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 msgr2=0x7f11c415d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f11b400e9d0 tx=0x7f11b400ee90 comp rx=0 tx=0).stop 2023-10-14T16:56:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 shutdown_connections 2023-10-14T16:56:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f11a806b130 0x7f11a806d5f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11c415ad70 0x7f11c40b2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11c40c9ca0 0x7f11c415d6f0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 --2- 172.21.15.29:0/239599557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11c406a4b0 0x7f11c41597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 >> 172.21.15.29:0/239599557 conn(0x7f11c40ce9b0 msgr2=0x7f11c4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:03.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 shutdown_connections 2023-10-14T16:56:03.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.216+0000 7f11cc2d4640 1 -- 172.21.15.29:0/239599557 wait complete. 2023-10-14T16:56:03.238 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:56:03.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:03.345 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:03.441 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:56:03.442 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:03.442 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:03.498 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:03.499 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:03.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:03.734 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:03.732+0000 7f22f1153040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:03.766 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:03.766+0000 7ff2c4414040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:03.787 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:03.786+0000 7f22f1153040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:03.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.802+0000 7f431e9a7640 1 Processor -- start 2023-10-14T16:56:03.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.802+0000 7f431e9a7640 1 -- start start 2023-10-14T16:56:03.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e9a7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43180b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e9a7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180d15a0 con 0x7f43180d2640 2023-10-14T16:56:03.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e9a7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180d1880 con 0x7f431815cbe0 2023-10-14T16:56:03.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e9a7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180d1b60 con 0x7f431815cfc0 2023-10-14T16:56:03.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e1a6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f43180d2640 0x7f43180b5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40592/0 (socket says 172.21.15.29:40592) 2023-10-14T16:56:03.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.804+0000 7f431e1a6640 1 -- 172.21.15.29:0/4127849823 learned_addr learned my addr 172.21.15.29:0/4127849823 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:03.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431d1a4640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43180b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310474979 0 0) 0x7f43180d15a0 con 0x7f43180d2640 2023-10-14T16:56:03.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304003520 con 0x7f43180d2640 2023-10-14T16:56:03.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3846865388 0 0) 0x7f43180d1880 con 0x7f431815cbe0 2023-10-14T16:56:03.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43180d15a0 con 0x7f431815cbe0 2023-10-14T16:56:03.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 56054545 0 0) 0x7f4304003520 con 0x7f43180d2640 2023-10-14T16:56:03.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 msgr2=0x7f43180b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:03.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43180b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 >> v1:172.21.15.186:6789/0 conn(0x7f431815cbe0 legacy=0x7f43180d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180d1e40 con 0x7f43180d2640 2023-10-14T16:56:03.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f43100019f0 con 0x7f43180d2640 2023-10-14T16:56:03.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431d1a4640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43180b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:56:03.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 >> v1:172.21.15.29:6789/0 conn(0x7f43180d2640 legacy=0x7f43180b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304003b30 0x7f4304003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.805+0000 7f431c9a3640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431c9a3640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4304004d00 0x7f4304005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180d1b60 con 0x7f4304004d00 2023-10-14T16:56:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f430400ba30 con 0x7f4304003b30 2023-10-14T16:56:03.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f430400bbf0 con 0x7f431815cfc0 2023-10-14T16:56:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431e1a6640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4304004d00 0x7f4304005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431d9a5640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304003b30 0x7f4304003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.806+0000 7f431d1a4640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.807+0000 7f431e1a6640 1 -- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 msgr2=0x7f43040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.807+0000 7f431e1a6640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f431815cfc0 0x7f43040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.807+0000 7f431e1a6640 1 -- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304003b30 msgr2=0x7f4304003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.807+0000 7f431e1a6640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304003b30 0x7f4304003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.807+0000 7f431e1a6640 1 -- 172.21.15.29:0/4127849823 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4308009400 con 0x7f4304004d00 2023-10-14T16:56:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.808+0000 7f431e1a6640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4304004d00 0x7f4304005180 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f431000a710 tx=0x7f43040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=409f564c373a258 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310034630 con 0x7f4304004d00 2023-10-14T16:56:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431002f070 con 0x7f4304004d00 2023-10-14T16:56:03.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431c9a3640 1 -- 172.21.15.29:0/4127849823 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310034a10 con 0x7f4304004d00 2023-10-14T16:56:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431e9a7640 1 -- 172.21.15.29:0/4127849823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4304004d00 msgr2=0x7f4304005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431e9a7640 1 --2- 172.21.15.29:0/4127849823 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4304004d00 0x7f4304005180 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f431000a710 tx=0x7f43040057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.809+0000 7f431e1a6640 1 -- 172.21.15.29:0/4127849823 reap_dead start 2023-10-14T16:56:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.810+0000 7f431e9a7640 1 -- 172.21.15.29:0/4127849823 shutdown_connections 2023-10-14T16:56:03.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.810+0000 7f431e9a7640 1 -- 172.21.15.29:0/4127849823 >> 172.21.15.29:0/4127849823 conn(0x7f43180cf570 msgr2=0x7f43180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.810+0000 7f431e9a7640 1 -- 172.21.15.29:0/4127849823 shutdown_connections 2023-10-14T16:56:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.810+0000 7f431e9a7640 1 -- 172.21.15.29:0/4127849823 wait complete. 2023-10-14T16:56:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.812+0000 7f431e9a7640 1 Processor -- start 2023-10-14T16:56:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.812+0000 7f431e9a7640 1 -- start start 2023-10-14T16:56:03.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304004d00 0x7f43180c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43180b6e00 0x7f43180b7280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43180b77c0 con 0x7f4318158890 2023-10-14T16:56:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43181598f0 con 0x7f4304004d00 2023-10-14T16:56:03.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.813+0000 7f431e9a7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4318159610 con 0x7f43180b6e00 2023-10-14T16:56:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45008/0 (socket says 172.21.15.29:45008) 2023-10-14T16:56:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 -- 172.21.15.29:0/411003940 learned_addr learned my addr 172.21.15.29:0/411003940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:03.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431d9a5640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304004d00 0x7f43180c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431d1a4640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43180b6e00 0x7f43180b7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43180b6e00 msgr2=0x7f43180b7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43180b6e00 0x7f43180b7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304004d00 msgr2=0x7f43180c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304004d00 0x7f43180c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.814+0000 7f431e1a6640 1 -- 172.21.15.29:0/411003940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43180ca5a0 con 0x7f4318158890 2023-10-14T16:56:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f431e1a6640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4310000f80 tx=0x7f4310001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310043070 con 0x7f4318158890 2023-10-14T16:56:03.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4310034640 con 0x7f4318158890 2023-10-14T16:56:03.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4310034a10 con 0x7f4318158890 2023-10-14T16:56:03.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43180ca880 con 0x7f4318158890 2023-10-14T16:56:03.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.815+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43180c8b20 con 0x7f4318158890 2023-10-14T16:56:03.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.816+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43180ca400 con 0x7f4318158890 2023-10-14T16:56:03.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.818+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4310031030 con 0x7f4318158890 2023-10-14T16:56:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.819+0000 7f430e7fc640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 0x7f42fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.819+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 6155+0+0 (secure 0 0 0) 0x7f43100ab2f0 con 0x7f4318158890 2023-10-14T16:56:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.819+0000 7f431d9a5640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 msgr2=0x7f42fc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:03.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.819+0000 7f431d9a5640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 0x7f42fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:03.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:03.824+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43180ca400 con 0x7f4318158890 2023-10-14T16:56:03.828 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:03.828+0000 7ff2c4414040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:04.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.017+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43180cada0 con 0x7f4318158890 2023-10-14T16:56:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.019+0000 7f430e7fc640 1 -- 172.21.15.29:0/411003940 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v66) v1 ==== 74+0+17156 (secure 0 0 0) 0x7f43180cada0 con 0x7f4318158890 2023-10-14T16:56:04.020 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:04.020 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:55:32.231265+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.38","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6833/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:6832/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/470082549":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/128331176":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2125394339":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/3085701795":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/4095875950":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/185522519":"2023-10-15T16:54:30.427919+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.020+0000 7f431d9a5640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 msgr2=0x7f42fc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.020+0000 7f431d9a5640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 0x7f42fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 msgr2=0x7f42fc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 0x7f42fc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 msgr2=0x7f4318158d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f4310000f80 tx=0x7f4310001cb0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 shutdown_connections 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f42fc06af90 0x7f42fc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4318158890 0x7f4318158d10 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43180b6e00 0x7f43180b7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 --2- 172.21.15.29:0/411003940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4304004d00 0x7f43180c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.021+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 >> 172.21.15.29:0/411003940 conn(0x7f43180cf570 msgr2=0x7f43180b1830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.022+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 shutdown_connections 2023-10-14T16:56:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.022+0000 7f431e9a7640 1 -- 172.21.15.29:0/411003940 wait complete. 2023-10-14T16:56:04.041 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:54:12.966365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:54:14.007924+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '63', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:04.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:04.235 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.234+0000 7f22f1153040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.257+0000 7f44c7fff640 1 Processor -- start 2023-10-14T16:56:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.257+0000 7f44c7fff640 1 -- start start 2023-10-14T16:56:04.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44c815ad10 0x7f44c80b1db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c80ce250 0x7f44c809d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c7fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44c8070c40 con 0x7f44c8065cc0 2023-10-14T16:56:04.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44c8066100 con 0x7f44c815ad10 2023-10-14T16:56:04.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c806e4c0 con 0x7f44c80ce250 2023-10-14T16:56:04.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c6ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44c815ad10 0x7f44c80b1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c67fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f44c8065cc0 0x7f44c80cdb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40600/0 (socket says 172.21.15.29:40600) 2023-10-14T16:56:04.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.259+0000 7f44c67fc640 1 -- 172.21.15.29:0/977528709 learned_addr learned my addr 172.21.15.29:0/977528709 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:04.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.260+0000 7f44c6ffd640 1 -- 172.21.15.29:0/977528709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c80ce250 msgr2=0x7f44c809d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:04.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.260+0000 7f44c6ffd640 1 --2- 172.21.15.29:0/977528709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c80ce250 0x7f44c809d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.260+0000 7f44c6ffd640 1 -- 172.21.15.29:0/977528709 >> v1:172.21.15.29:6789/0 conn(0x7f44c8065cc0 legacy=0x7f44c80cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.260+0000 7f44c6ffd640 1 -- 172.21.15.29:0/977528709 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c80b49c0 con 0x7f44c815ad10 2023-10-14T16:56:04.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.261+0000 7f44c6ffd640 1 --2- 172.21.15.29:0/977528709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c815ad10 0x7f44c80b1db0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f44b80097a0 tx=0x7f44b802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1493292b409b7fd5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:04.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.261+0000 7f44c5ffb640 1 -- 172.21.15.29:0/977528709 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b802d040 con 0x7f44c815ad10 2023-10-14T16:56:04.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.261+0000 7f44c5ffb640 1 -- 172.21.15.29:0/977528709 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44b8031810 con 0x7f44c815ad10 2023-10-14T16:56:04.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.261+0000 7f44c5ffb640 1 -- 172.21.15.29:0/977528709 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44b8031ba0 con 0x7f44c815ad10 2023-10-14T16:56:04.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 -- 172.21.15.29:0/977528709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c815ad10 msgr2=0x7f44c80b1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 --2- 172.21.15.29:0/977528709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c815ad10 0x7f44c80b1db0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f44b80097a0 tx=0x7f44b802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 -- 172.21.15.29:0/977528709 shutdown_connections 2023-10-14T16:56:04.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 --2- 172.21.15.29:0/977528709 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c80ce250 0x7f44c809d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 --2- 172.21.15.29:0/977528709 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c815ad10 0x7f44c80b1db0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 -- 172.21.15.29:0/977528709 >> 172.21.15.29:0/977528709 conn(0x7f44c80b0a50 msgr2=0x7f44c80b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:04.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 -- 172.21.15.29:0/977528709 shutdown_connections 2023-10-14T16:56:04.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.262+0000 7f44c7fff640 1 -- 172.21.15.29:0/977528709 wait complete. 2023-10-14T16:56:04.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.265+0000 7f44c7fff640 1 Processor -- start 2023-10-14T16:56:04.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.265+0000 7f44c7fff640 1 -- start start 2023-10-14T16:56:04.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8065cc0 0x7f44c80c36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c815ad10 0x7f44c81f5c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80c1f80 con 0x7f44c80ce250 2023-10-14T16:56:04.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80c53c0 con 0x7f44c8065cc0 2023-10-14T16:56:04.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.266+0000 7f44c7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c80c47b0 con 0x7f44c815ad10 2023-10-14T16:56:04.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.267+0000 7f44c67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.267+0000 7f44c67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45040/0 (socket says 172.21.15.29:45040) 2023-10-14T16:56:04.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.267+0000 7f44c67fc640 1 -- 172.21.15.29:0/2548445833 learned_addr learned my addr 172.21.15.29:0/2548445833 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:04.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.267+0000 7f44c77fe640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c815ad10 0x7f44c81f5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.267+0000 7f44c6ffd640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8065cc0 0x7f44c80c36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.268+0000 7f44c67fc640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c815ad10 msgr2=0x7f44c81f5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.268+0000 7f44c67fc640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c815ad10 0x7f44c81f5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.268+0000 7f44c67fc640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8065cc0 msgr2=0x7f44c80c36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.268+0000 7f44c67fc640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8065cc0 0x7f44c80c36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.268+0000 7f44c67fc640 1 -- 172.21.15.29:0/2548445833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b8009400 con 0x7f44c80ce250 2023-10-14T16:56:04.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.269+0000 7f44c67fc640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00c4a0 tx=0x7f44bc00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:04.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.269+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc005ce0 con 0x7f44c80ce250 2023-10-14T16:56:04.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.270+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44bc005e40 con 0x7f44c80ce250 2023-10-14T16:56:04.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.270+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44bc007270 con 0x7f44c80ce250 2023-10-14T16:56:04.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.270+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44c80c4a90 con 0x7f44c80ce250 2023-10-14T16:56:04.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.270+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44c80c5050 con 0x7f44c80ce250 2023-10-14T16:56:04.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.271+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44c804e9b0 con 0x7f44c80ce250 2023-10-14T16:56:04.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.272+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f44bc007490 con 0x7f44c80ce250 2023-10-14T16:56:04.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.273+0000 7f44af7fe640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 0x7f44b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.273+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6243+0+0 (secure 0 0 0) 0x7f44bc088e00 con 0x7f44c80ce250 2023-10-14T16:56:04.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.273+0000 7f44c6ffd640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 msgr2=0x7f44b406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:04.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.273+0000 7f44c6ffd640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 0x7f44b406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:04.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.282+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44bc053c20 con 0x7f44c80ce250 2023-10-14T16:56:04.298 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.297+0000 7f22f1153040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:04.317 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.314+0000 7ff2c4414040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:04.348 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.347+0000 7f22f1153040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:04.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.369+0000 7ff2c4414040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:04.408 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.407+0000 7f22f1153040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:04.424 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.423+0000 7ff2c4414040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:04.456 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.455+0000 7f22f1153040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.474+0000 7f44c6ffd640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 msgr2=0x7f44b406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.474+0000 7f44c6ffd640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 0x7f44b406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.481+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f44c80c5500 con 0x7f44c80ce250 2023-10-14T16:56:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.482+0000 7f44af7fe640 1 -- 172.21.15.29:0/2548445833 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v67) v1 ==== 93+0+10 (secure 0 0 0) 0x7f44bc017020 con 0x7f44c80ce250 2023-10-14T16:56:04.483 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 msgr2=0x7f44b406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 0x7f44b406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 msgr2=0x7f44c80c1a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f44bc00c4a0 tx=0x7f44bc00c960 comp rx=0 tx=0).stop 2023-10-14T16:56:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 shutdown_connections 2023-10-14T16:56:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f44b406af90 0x7f44b406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44c815ad10 0x7f44c81f5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80ce250 0x7f44c80c1a40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 --2- 172.21.15.29:0/2548445833 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8065cc0 0x7f44c80c36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.484+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 >> 172.21.15.29:0/2548445833 conn(0x7f44c80b0a50 msgr2=0x7f44c80cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.485+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 shutdown_connections 2023-10-14T16:56:04.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.485+0000 7f44c7fff640 1 -- 172.21.15.29:0/2548445833 wait complete. 2023-10-14T16:56:04.491 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.490+0000 7ff2c4414040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:04.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:56:04.544 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.543+0000 7ff2c4414040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:04.567 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.566+0000 7f22f1153040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:04.631 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.630+0000 7f22f1153040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:04.667 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.666+0000 7ff2c4414040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:04.682 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.681+0000 7f22f1153040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.726+0000 7f99f200d640 1 Processor -- start 2023-10-14T16:56:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.727+0000 7f99f200d640 1 -- start start 2023-10-14T16:56:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.728+0000 7f99f200d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 0x7f99ec0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f200d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec15d150 con 0x7f99ec0cded0 2023-10-14T16:56:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f200d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99ec0d1180 con 0x7f99ec0b8480 2023-10-14T16:56:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f200d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0d1460 con 0x7f99ec15cd70 2023-10-14T16:56:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f180c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99ec0cded0 0x7f99ec0b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40616/0 (socket says 172.21.15.29:40616) 2023-10-14T16:56:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f180c640 1 -- 172.21.15.29:0/1971882887 learned_addr learned my addr 172.21.15.29:0/1971882887 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.729+0000 7f99f100b640 1 --2- 172.21.15.29:0/1971882887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 0x7f99ec0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.730+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474942185 0 0) 0x7f99ec15d150 con 0x7f99ec0cded0 2023-10-14T16:56:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.730+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d8003520 con 0x7f99ec0cded0 2023-10-14T16:56:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.730+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3962752532 0 0) 0x7f99ec0d1180 con 0x7f99ec0b8480 2023-10-14T16:56:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.731+0000 7f99f100b640 1 -- 172.21.15.29:0/1971882887 >> v1:172.21.15.186:6789/0 conn(0x7f99ec0b8480 legacy=0x7f99ec0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.731+0000 7f99f100b640 1 -- 172.21.15.29:0/1971882887 >> v1:172.21.15.29:6789/0 conn(0x7f99ec0cded0 legacy=0x7f99ec0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.731+0000 7f99f100b640 1 -- 172.21.15.29:0/1971882887 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99ec0d1740 con 0x7f99ec15cd70 2023-10-14T16:56:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.732+0000 7f99f100b640 1 --2- 172.21.15.29:0/1971882887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 0x7f99ec0d2220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99e80097d0 tx=0x7f99e802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab038c1b3fbb0d69 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.733+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8031890 con 0x7f99ec15cd70 2023-10-14T16:56:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.733+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e80319f0 con 0x7f99ec15cd70 2023-10-14T16:56:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.733+0000 7f99e3fff640 1 -- 172.21.15.29:0/1971882887 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8031d60 con 0x7f99ec15cd70 2023-10-14T16:56:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 -- 172.21.15.29:0/1971882887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 msgr2=0x7f99ec0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 --2- 172.21.15.29:0/1971882887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 0x7f99ec0d2220 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f99e80097d0 tx=0x7f99e802b930 comp rx=0 tx=0).stop 2023-10-14T16:56:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 -- 172.21.15.29:0/1971882887 shutdown_connections 2023-10-14T16:56:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 --2- 172.21.15.29:0/1971882887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec15cd70 0x7f99ec0d2220 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 -- 172.21.15.29:0/1971882887 >> 172.21.15.29:0/1971882887 conn(0x7f99ec0cf570 msgr2=0x7f99ec0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 -- 172.21.15.29:0/1971882887 shutdown_connections 2023-10-14T16:56:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.734+0000 7f99f200d640 1 -- 172.21.15.29:0/1971882887 wait complete. 2023-10-14T16:56:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.736+0000 7f99f200d640 1 Processor -- start 2023-10-14T16:56:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.736+0000 7f99f200d640 1 -- start start 2023-10-14T16:56:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 0x7f99ec15b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 0x7f99ec15bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0b2fc0 con 0x7f99ec0b8480 2023-10-14T16:56:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0b3580 con 0x7f99ec15cd70 2023-10-14T16:56:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f200d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99ec0ca9a0 con 0x7f99ec0cded0 2023-10-14T16:56:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f100b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f100b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45048/0 (socket says 172.21.15.29:45048) 2023-10-14T16:56:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f100b640 1 -- 172.21.15.29:0/1875333149 learned_addr learned my addr 172.21.15.29:0/1875333149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f180c640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 0x7f99ec15bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.737+0000 7f99f080a640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 0x7f99ec15b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 msgr2=0x7f99ec15b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 0x7f99ec15b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 msgr2=0x7f99ec15bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 0x7f99ec15bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 -- 172.21.15.29:0/1875333149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99e8009400 con 0x7f99ec0b8480 2023-10-14T16:56:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f080a640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 0x7f99ec15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f180c640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 0x7f99ec15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f100b640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f99e80098f0 tx=0x7f99e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8003a20 con 0x7f99ec0b8480 2023-10-14T16:56:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99e8003b80 con 0x7f99ec0b8480 2023-10-14T16:56:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99e8006e10 con 0x7f99ec0b8480 2023-10-14T16:56:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99ec0cac80 con 0x7f99ec0b8480 2023-10-14T16:56:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.738+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99ec14e480 con 0x7f99ec0b8480 2023-10-14T16:56:04.744 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.739+0000 7ff2c4414040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:04.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.739+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99ec04e9b0 con 0x7f99ec0b8480 2023-10-14T16:56:04.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.740+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f99e804e440 con 0x7f99ec0b8480 2023-10-14T16:56:04.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.741+0000 7f99e1ffb640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 0x7f99d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:04.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.741+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6243+0+0 (secure 0 0 0) 0x7f99e80ac6e0 con 0x7f99ec0b8480 2023-10-14T16:56:04.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.741+0000 7f99f080a640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 msgr2=0x7f99d006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.741+0000 7f99f080a640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 0x7f99d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:04.746 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.741+0000 7f22f1153040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.745+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99e8077340 con 0x7f99ec0b8480 2023-10-14T16:56:04.793 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:04.792+0000 7f22f1153040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:04.793 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.792+0000 7ff2c4414040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:04.854 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.854+0000 7ff2c4414040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:04.908 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:04.907+0000 7ff2c4414040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:04.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.940+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f99ec0cfa80 con 0x7f99ec0b8480 2023-10-14T16:56:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.941+0000 7f99e1ffb640 1 -- 172.21.15.29:0/1875333149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v67) v1 ==== 104+0+11 (secure 0 0 0) 0x7f99e807a960 con 0x7f99ec0b8480 2023-10-14T16:56:04.942 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:56:04.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.942+0000 7f99f080a640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 msgr2=0x7f99d006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:04.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.942+0000 7f99f080a640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 0x7f99d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:04.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 msgr2=0x7f99d006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 0x7f99d006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 msgr2=0x7f99ec14df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f99e80098f0 tx=0x7f99e80046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 shutdown_connections 2023-10-14T16:56:04.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f99d006af90 0x7f99d006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99ec15cd70 0x7f99ec15bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99ec0cded0 0x7f99ec15b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 --2- 172.21.15.29:0/1875333149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99ec0b8480 0x7f99ec14df40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:04.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 >> 172.21.15.29:0/1875333149 conn(0x7f99ec0cf570 msgr2=0x7f99ec15ad90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 shutdown_connections 2023-10-14T16:56:04.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:04.944+0000 7f99f200d640 1 -- 172.21.15.29:0/1875333149 wait complete. 2023-10-14T16:56:04.963 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:56:05.178 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.177+0000 7f22f1153040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:05.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.228+0000 7f7311d73640 1 Processor -- start 2023-10-14T16:56:05.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.228+0000 7f7311d73640 1 -- start start 2023-10-14T16:56:05.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.229+0000 7f7311d73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f730c0c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f7311d73640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0c9af0 con 0x7f730c0c9150 2023-10-14T16:56:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f7311d73640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0cc6b0 con 0x7f730c1428c0 2023-10-14T16:56:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f7311d73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c06a4b0 con 0x7f730c06bd70 2023-10-14T16:56:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f730affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f730c0c9150 0x7f730c06b660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40618/0 (socket says 172.21.15.29:40618) 2023-10-14T16:56:05.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f730affd640 1 -- 172.21.15.29:0/3228695534 learned_addr learned my addr 172.21.15.29:0/3228695534 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f730bfff640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f730c0c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816131772 0 0) 0x7f730c0c9af0 con 0x7f730c0c9150 2023-10-14T16:56:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.230+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f4005730 con 0x7f730c0c9150 2023-10-14T16:56:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4203566211 0 0) 0x7f72f4005730 con 0x7f730c0c9150 2023-10-14T16:56:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 msgr2=0x7f730c0c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:05.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f730c0c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 >> v1:172.21.15.186:6789/0 conn(0x7f730c1428c0 legacy=0x7f730c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c1599b0 con 0x7f730c0c9150 2023-10-14T16:56:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7300001670 con 0x7f730c0c9150 2023-10-14T16:56:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 >> v1:172.21.15.29:6789/0 conn(0x7f730c0c9150 legacy=0x7f730c06b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f72f4005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f4006130 0x7f72f40069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006f10 0x7f72f4007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c06a4b0 con 0x7f72f4006130 2023-10-14T16:56:05.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f400dc70 con 0x7f72f4006f10 2023-10-14T16:56:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72f400de30 con 0x7f730c06bd70 2023-10-14T16:56:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.231+0000 7f730affd640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f4006130 0x7f72f40069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730b7fe640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f72f4005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730affd640 1 -- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 msgr2=0x7f72f4005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:05.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730affd640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c06bd70 0x7f72f4005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730affd640 1 -- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006f10 msgr2=0x7f72f4007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730affd640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006f10 0x7f72f4007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.232+0000 7f730affd640 1 -- 172.21.15.29:0/3228695534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f4005730 con 0x7f72f4006130 2023-10-14T16:56:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.233+0000 7f730affd640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f4006130 0x7f72f40069d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f730000c240 tx=0x7f72f4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc5dcbd7b2986b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:05.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.233+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7300005ad0 con 0x7f72f4006130 2023-10-14T16:56:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.233+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7300005c30 con 0x7f72f4006130 2023-10-14T16:56:05.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.233+0000 7f730a7fc640 1 -- 172.21.15.29:0/3228695534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f730002caf0 con 0x7f72f4006130 2023-10-14T16:56:05.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 -- 172.21.15.29:0/3228695534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f4006130 msgr2=0x7f72f40069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 --2- 172.21.15.29:0/3228695534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72f4006130 0x7f72f40069d0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f730000c240 tx=0x7f72f4007a00 comp rx=0 tx=0).stop 2023-10-14T16:56:05.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f730bfff640 1 -- 172.21.15.29:0/3228695534 reap_dead start 2023-10-14T16:56:05.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 -- 172.21.15.29:0/3228695534 shutdown_connections 2023-10-14T16:56:05.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 -- 172.21.15.29:0/3228695534 >> 172.21.15.29:0/3228695534 conn(0x7f730c0ce9b0 msgr2=0x7f730c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 -- 172.21.15.29:0/3228695534 shutdown_connections 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.234+0000 7f7311d73640 1 -- 172.21.15.29:0/3228695534 wait complete. 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.236+0000 7f7311d73640 1 Processor -- start 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.236+0000 7f7311d73640 1 -- start start 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006130 0x7f730c0c40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c0ae6c0 0x7f730c0aeb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0c4620 con 0x7f730c159e50 2023-10-14T16:56:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f730c071720 con 0x7f72f4006130 2023-10-14T16:56:05.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f7311d73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0af080 con 0x7f730c0ae6c0 2023-10-14T16:56:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f730bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f730bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45060/0 (socket says 172.21.15.29:45060) 2023-10-14T16:56:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f730bfff640 1 -- 172.21.15.29:0/1505346599 learned_addr learned my addr 172.21.15.29:0/1505346599 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f730b7fe640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006130 0x7f730c0c40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.237+0000 7f730affd640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c0ae6c0 0x7f730c0aeb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c0ae6c0 msgr2=0x7f730c0aeb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c0ae6c0 0x7f730c0aeb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006130 msgr2=0x7f730c0c40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006130 0x7f730c0c40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 -- 172.21.15.29:0/1505346599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c15a840 con 0x7f730c159e50 2023-10-14T16:56:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f730bfff640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f730c06c1c0 tx=0x7f730400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304010040 con 0x7f730c159e50 2023-10-14T16:56:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f730400bce0 con 0x7f730c159e50 2023-10-14T16:56:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7304014640 con 0x7f730c159e50 2023-10-14T16:56:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.238+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c070680 con 0x7f730c159e50 2023-10-14T16:56:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.239+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c070c40 con 0x7f730c159e50 2023-10-14T16:56:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.240+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c04e160 con 0x7f730c159e50 2023-10-14T16:56:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.242+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f73040147a0 con 0x7f730c159e50 2023-10-14T16:56:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.242+0000 7f72f3fff640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 0x7f72f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.243+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 6243+0+0 (secure 0 0 0) 0x7f7304089d20 con 0x7f730c159e50 2023-10-14T16:56:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.243+0000 7f730b7fe640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 msgr2=0x7f72f806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.243+0000 7f730b7fe640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 0x7f72f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.247+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7304054990 con 0x7f730c159e50 2023-10-14T16:56:05.329 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.329+0000 7ff2c4414040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:05.349 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.348+0000 7f22f1153040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:05.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.444+0000 7f730b7fe640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 msgr2=0x7f72f806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:05.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.444+0000 7f730b7fe640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 0x7f72f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:05.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.451+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f730c15a840 con 0x7f730c159e50 2023-10-14T16:56:05.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.452+0000 7f72f3fff640 1 -- 172.21.15.29:0/1505346599 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v68) v1 ==== 100+0+11 (secure 0 0 0) 0x7f730c15a840 con 0x7f730c159e50 2023-10-14T16:56:05.453 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:56:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 msgr2=0x7f72f806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 0x7f72f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 msgr2=0x7f730c15a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f730c06c1c0 tx=0x7f730400b480 comp rx=0 tx=0).stop 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 shutdown_connections 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f72f806afc0 0x7f72f806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f730c159e50 0x7f730c15a300 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f730c0ae6c0 0x7f730c0aeb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 --2- 172.21.15.29:0/1505346599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72f4006130 0x7f730c0c40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 >> 172.21.15.29:0/1505346599 conn(0x7f730c0ce9b0 msgr2=0x7f730c0afd20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:05.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 shutdown_connections 2023-10-14T16:56:05.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.455+0000 7f7311d73640 1 -- 172.21.15.29:0/1505346599 wait complete. 2023-10-14T16:56:05.458 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.456+0000 7f22f1153040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:05.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:56:05.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.507+0000 7ff2c4414040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:05.512 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.511+0000 7f22f1153040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:05.619 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.618+0000 7f22f1153040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:05.623 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.623+0000 7ff2c4414040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:05.678 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.678+0000 7ff2c4414040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.680+0000 7fca4bdcc640 1 Processor -- start 2023-10-14T16:56:05.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.680+0000 7fca4bdcc640 1 -- start start 2023-10-14T16:56:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.681+0000 7fca4bdcc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca441618b0 0x7fca44065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.681+0000 7fca4bdcc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca4406cee0 con 0x7fca44065ff0 2023-10-14T16:56:05.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.681+0000 7fca4bdcc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca440663d0 con 0x7fca4415d330 2023-10-14T16:56:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.681+0000 7fca4bdcc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca4406b1e0 con 0x7fca441618b0 2023-10-14T16:56:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca49367640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca441618b0 0x7fca44065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca4a369640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fca44065ff0 0x7fca440ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40630/0 (socket says 172.21.15.29:40630) 2023-10-14T16:56:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca49367640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca441618b0 0x7fca44065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45064/0 (socket says 172.21.15.29:45064) 2023-10-14T16:56:05.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca4a369640 1 -- 172.21.15.29:0/1589409353 learned_addr learned my addr 172.21.15.29:0/1589409353 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca49367640 1 -- 172.21.15.29:0/1589409353 >> v1:172.21.15.186:6789/0 conn(0x7fca4415d330 legacy=0x7fca4406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca49367640 1 -- 172.21.15.29:0/1589409353 >> v1:172.21.15.29:6789/0 conn(0x7fca44065ff0 legacy=0x7fca440ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca49367640 1 -- 172.21.15.29:0/1589409353 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca440d0890 con 0x7fca441618b0 2023-10-14T16:56:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.682+0000 7fca48b66640 1 -- 172.21.15.29:0/1589409353 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3925562181 0 0) 0x7fca4406cee0 con 0x7fca44065ff0 2023-10-14T16:56:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.683+0000 7fca49367640 1 --2- 172.21.15.29:0/1589409353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca44065ab0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fca340097a0 tx=0x7fca3402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d74b2cfdb3c468c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca48b66640 1 -- 172.21.15.29:0/1589409353 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34031830 con 0x7fca441618b0 2023-10-14T16:56:05.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca48b66640 1 -- 172.21.15.29:0/1589409353 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca34031990 con 0x7fca441618b0 2023-10-14T16:56:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca48b66640 1 -- 172.21.15.29:0/1589409353 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca34031d20 con 0x7fca441618b0 2023-10-14T16:56:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 -- 172.21.15.29:0/1589409353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 msgr2=0x7fca44065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/1589409353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca44065ab0 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fca340097a0 tx=0x7fca3402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 -- 172.21.15.29:0/1589409353 shutdown_connections 2023-10-14T16:56:05.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/1589409353 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca44065ab0 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 -- 172.21.15.29:0/1589409353 >> 172.21.15.29:0/1589409353 conn(0x7fca440d0ab0 msgr2=0x7fca440d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:05.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 -- 172.21.15.29:0/1589409353 shutdown_connections 2023-10-14T16:56:05.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.684+0000 7fca4bdcc640 1 -- 172.21.15.29:0/1589409353 wait complete. 2023-10-14T16:56:05.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.686+0000 7fca4bdcc640 1 Processor -- start 2023-10-14T16:56:05.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.687+0000 7fca4bdcc640 1 -- start start 2023-10-14T16:56:05.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca44065ff0 0x7fca44158350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca4415d330 0x7fca440ae130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440ac890 con 0x7fca441618b0 2023-10-14T16:56:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca440ad3d0 con 0x7fca44065ff0 2023-10-14T16:56:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4bdcc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca44158890 con 0x7fca4415d330 2023-10-14T16:56:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4a369640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4a369640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45066/0 (socket says 172.21.15.29:45066) 2023-10-14T16:56:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca4a369640 1 -- 172.21.15.29:0/3114914542 learned_addr learned my addr 172.21.15.29:0/3114914542 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca49b68640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca44065ff0 0x7fca44158350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.688+0000 7fca49367640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca4415d330 0x7fca440ae130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca4415d330 msgr2=0x7fca440ae130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca4415d330 0x7fca440ae130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca44065ff0 msgr2=0x7fca44158350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca44065ff0 0x7fca44158350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 -- 172.21.15.29:0/3114914542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca34009400 con 0x7fca441618b0 2023-10-14T16:56:05.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4a369640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00c780 tx=0x7fca3c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00ce90 con 0x7fca441618b0 2023-10-14T16:56:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca3c00a640 con 0x7fca441618b0 2023-10-14T16:56:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca3c00a9d0 con 0x7fca441618b0 2023-10-14T16:56:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.689+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca440acaa0 con 0x7fca441618b0 2023-10-14T16:56:05.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.690+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca440ad060 con 0x7fca441618b0 2023-10-14T16:56:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.691+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca440ad540 con 0x7fca441618b0 2023-10-14T16:56:05.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.691+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fca3c00ab30 con 0x7fca441618b0 2023-10-14T16:56:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.692+0000 7fca3a7fc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 0x7fca2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.693+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fca3c088ed0 con 0x7fca441618b0 2023-10-14T16:56:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.693+0000 7fca49b68640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 msgr2=0x7fca2806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:05.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.693+0000 7fca49b68640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 0x7fca2806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:05.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.697+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca3c053c50 con 0x7fca441618b0 2023-10-14T16:56:05.730 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.729+0000 7f22f1153040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:05.799 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.797+0000 7ff2c4414040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:05.890 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.888+0000 7f22f1153040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:05.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.894+0000 7fca49b68640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 msgr2=0x7fca2806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:05.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.894+0000 7fca49b68640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 0x7fca2806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:05.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.912+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fca4404e160 con 0x7fca441618b0 2023-10-14T16:56:05.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.913+0000 7fca3a7fc640 1 -- 172.21.15.29:0/3114914542 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v68) v1 ==== 80+0+463 (secure 0 0 0) 0x7fca3c018070 con 0x7fca441618b0 2023-10-14T16:56:05.913 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/98172895 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/98172895 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/470082549 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/128331176 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2125394339 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.914 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3085701795 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4095875950 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.915 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/185522519 2023-10-15T16:54:30.427919+0000 2023-10-14T16:56:05.915 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T16:56:05.916 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:05.915+0000 7ff2c4414040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:05.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 msgr2=0x7fca2806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:05.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 0x7fca2806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 msgr2=0x7fca4415e890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:05.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fca3c00c780 tx=0x7fca3c00cc40 comp rx=0 tx=0).stop 2023-10-14T16:56:05.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 shutdown_connections 2023-10-14T16:56:05.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fca2806af90 0x7fca2806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca441618b0 0x7fca4415e890 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca4415d330 0x7fca440ae130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 --2- 172.21.15.29:0/3114914542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca44065ff0 0x7fca44158350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:05.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.916+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 >> 172.21.15.29:0/3114914542 conn(0x7fca440d0ab0 msgr2=0x7fca44159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:05.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.917+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 shutdown_connections 2023-10-14T16:56:05.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:05.917+0000 7fca4bdcc640 1 -- 172.21.15.29:0/3114914542 wait complete. 2023-10-14T16:56:05.938 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:05.937+0000 7f22f1153040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:05.942 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:56:05.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:05.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:06.055 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:06.054+0000 7f22f1153040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:06.089 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:56:06.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:06.090 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:56:06.097 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:06.094+0000 7ff2c4414040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:06.140 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:06.141 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:06.148 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:06.148+0000 7ff2c4414040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:06.198 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:06.198 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:06.199 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:06.255 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:06.256 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:06.256 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:06.276 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:06.276+0000 7ff2c4414040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:06.313 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:06.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.525+0000 7f10e174e640 1 Processor -- start 2023-10-14T16:56:06.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.525+0000 7f10e174e640 1 -- start start 2023-10-14T16:56:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.527+0000 7f10e174e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f10dc159b20 0x7f10dc06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.527+0000 7f10e174e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10dc0b0b20 0x7f10dc06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.527+0000 7f10e174e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10dc06f7f0 con 0x7f10dc159f00 2023-10-14T16:56:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.527+0000 7f10e174e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc06c5e0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.527+0000 7f10e174e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10dc06c8c0 con 0x7f10dc159b20 2023-10-14T16:56:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10dc0b0b20 0x7f10dc06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10d27fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f10dc159f00 0x7f10dc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40632/0 (socket says 172.21.15.29:40632) 2023-10-14T16:56:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f10dc0b0b20 0x7f10dc06fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45080/0 (socket says 172.21.15.29:45080) 2023-10-14T16:56:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10d27fc640 1 -- 172.21.15.29:0/3455698754 learned_addr learned my addr 172.21.15.29:0/3455698754 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:06.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10daffd640 1 --2- 172.21.15.29:0/3455698754 >> v2:172.21.15.186:3300/0 conn(0x7f10dc159b20 0x7f10dc06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 -- 172.21.15.29:0/3455698754 >> v2:172.21.15.186:3300/0 conn(0x7f10dc159b20 msgr2=0x7f10dc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 --2- 172.21.15.29:0/3455698754 >> v2:172.21.15.186:3300/0 conn(0x7f10dc159b20 0x7f10dc06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 -- 172.21.15.29:0/3455698754 >> v1:172.21.15.29:6789/0 conn(0x7f10dc159f00 legacy=0x7f10dc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.528+0000 7f10db7fe640 1 -- 172.21.15.29:0/3455698754 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10dc06cba0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.529+0000 7f10db7fe640 1 --2- 172.21.15.29:0/3455698754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc06fc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0c9340 tx=0x7f10d402df20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebdacf31a549fcdf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.530+0000 7f10da7fc640 1 -- 172.21.15.29:0/3455698754 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d402e840 con 0x7f10dc0b0b20 2023-10-14T16:56:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.530+0000 7f10da7fc640 1 -- 172.21.15.29:0/3455698754 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10d402e9a0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.530+0000 7f10da7fc640 1 -- 172.21.15.29:0/3455698754 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10d402ed30 con 0x7f10dc0b0b20 2023-10-14T16:56:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.530+0000 7f10e174e640 1 -- 172.21.15.29:0/3455698754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 msgr2=0x7f10dc06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:06.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.530+0000 7f10e174e640 1 --2- 172.21.15.29:0/3455698754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc06fc80 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f10dc0c9340 tx=0x7f10d402df20 comp rx=0 tx=0).stop 2023-10-14T16:56:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 -- 172.21.15.29:0/3455698754 shutdown_connections 2023-10-14T16:56:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 --2- 172.21.15.29:0/3455698754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc06fc80 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 --2- 172.21.15.29:0/3455698754 >> v2:172.21.15.186:3300/0 conn(0x7f10dc159b20 0x7f10dc06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 -- 172.21.15.29:0/3455698754 >> 172.21.15.29:0/3455698754 conn(0x7f10dc0cb410 msgr2=0x7f10dc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:06.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 -- 172.21.15.29:0/3455698754 shutdown_connections 2023-10-14T16:56:06.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.531+0000 7f10e174e640 1 -- 172.21.15.29:0/3455698754 wait complete. 2023-10-14T16:56:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.533+0000 7f10e174e640 1 Processor -- start 2023-10-14T16:56:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.533+0000 7f10e174e640 1 -- start start 2023-10-14T16:56:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10dc159b20 0x7f10dc0c2d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10dc159f00 0x7f10dc151630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc0c32a0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc15c520 con 0x7f10dc159f00 2023-10-14T16:56:06.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10e174e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10dc1f6950 con 0x7f10dc159b20 2023-10-14T16:56:06.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10daffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45096/0 (socket says 172.21.15.29:45096) 2023-10-14T16:56:06.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10daffd640 1 -- 172.21.15.29:0/1866360105 learned_addr learned my addr 172.21.15.29:0/1866360105 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:06.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10db7fe640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10dc159f00 0x7f10dc151630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.534+0000 7f10d27fc640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10dc159b20 0x7f10dc0c2d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10dc159b20 msgr2=0x7f10dc0c2d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:06.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10dc159b20 0x7f10dc0c2d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10dc159f00 msgr2=0x7f10dc151630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:06.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10dc159f00 0x7f10dc151630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 -- 172.21.15.29:0/1866360105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d400bc80 con 0x7f10dc0b0b20 2023-10-14T16:56:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10daffd640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00e930 tx=0x7f10cc00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.535+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc009690 con 0x7f10dc0b0b20 2023-10-14T16:56:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.536+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10cc00ca00 con 0x7f10dc0b0b20 2023-10-14T16:56:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.536+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10cc00cdb0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.536+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10dc1f6c30 con 0x7f10dc0b0b20 2023-10-14T16:56:06.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.536+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10dc1f71f0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.537+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10dc04e160 con 0x7f10dc0b0b20 2023-10-14T16:56:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.538+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f10cc013470 con 0x7f10dc0b0b20 2023-10-14T16:56:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.539+0000 7f10d3fff640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 0x7f10c006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.539+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f10cc089a60 con 0x7f10dc0b0b20 2023-10-14T16:56:06.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.539+0000 7f10d27fc640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 msgr2=0x7f10c006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.539+0000 7f10d27fc640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 0x7f10c006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:06.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.544+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f10cc054a40 con 0x7f10dc0b0b20 2023-10-14T16:56:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.740+0000 7f10d27fc640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 msgr2=0x7f10c006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.740+0000 7f10d27fc640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 0x7f10c006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:06.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.777+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10dc0b10b0 con 0x7f10dc0b0b20 2023-10-14T16:56:06.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.777+0000 7f10d3fff640 1 -- 172.21.15.29:0/1866360105 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f10cc058060 con 0x7f10dc0b0b20 2023-10-14T16:56:06.778 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:06.779 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:56:06.779 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:56:06.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.779+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 msgr2=0x7f10c006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.779+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 0x7f10c006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.779+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 msgr2=0x7f10dc15ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.779+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f10cc00e930 tx=0x7f10cc00edf0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 shutdown_connections 2023-10-14T16:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f10c006b010 0x7f10c006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f10dc159f00 0x7f10dc151630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10dc159b20 0x7f10dc0c2d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 --2- 172.21.15.29:0/1866360105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f10dc0b0b20 0x7f10dc15ad20 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:06.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 >> 172.21.15.29:0/1866360105 conn(0x7f10dc0cb410 msgr2=0x7f10dc0ad120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 shutdown_connections 2023-10-14T16:56:06.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.780+0000 7f10e174e640 1 -- 172.21.15.29:0/1866360105 wait complete. 2023-10-14T16:56:06.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:06.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:06.887+0000 7f22f1153040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.993+0000 7f6ace9ba640 1 Processor -- start 2023-10-14T16:56:06.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.993+0000 7f6ace9ba640 1 -- start start 2023-10-14T16:56:06.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace9ba640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 0x7f6ac815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace9ba640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ac806c100 0x7f6ac806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:06.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace9ba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ac815aef0 con 0x7f6ac8159950 2023-10-14T16:56:06.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace9ba640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ac815e490 con 0x7f6ac806c100 2023-10-14T16:56:06.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace9ba640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac815d910 con 0x7f6ac8070b60 2023-10-14T16:56:06.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6acd9b8640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6ac8159950 0x7f6ac8070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:53974/0 (socket says 172.21.15.29:53974) 2023-10-14T16:56:06.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6acd9b8640 1 -- 172.21.15.29:0/2822890572 learned_addr learned my addr 172.21.15.29:0/2822890572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:06.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2822890572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 0x7f6ac815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:06.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.995+0000 7f6ace1b9640 1 --2- 172.21.15.29:0/2822890572 >> v2:172.21.15.186:3300/0 conn(0x7f6ac806c100 0x7f6ac806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.996+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2822890572 >> v2:172.21.15.186:3300/0 conn(0x7f6ac806c100 msgr2=0x7f6ac806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.996+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2822890572 >> v2:172.21.15.186:3300/0 conn(0x7f6ac806c100 0x7f6ac806c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.996+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2822890572 >> v1:172.21.15.186:6789/0 conn(0x7f6ac8159950 legacy=0x7f6ac8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.996+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2822890572 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac815dbf0 con 0x7f6ac8070b60 2023-10-14T16:56:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.997+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2822890572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 0x7f6ac815a9b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800ba40 tx=0x7f6ab802dff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e192088ceeb4c31 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.998+0000 7f6acc9b6640 1 -- 172.21.15.29:0/2822890572 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab802eda0 con 0x7f6ac8070b60 2023-10-14T16:56:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.998+0000 7f6acc9b6640 1 -- 172.21.15.29:0/2822890572 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab8036440 con 0x7f6ac8070b60 2023-10-14T16:56:07.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.998+0000 7f6acc9b6640 1 -- 172.21.15.29:0/2822890572 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab80367d0 con 0x7f6ac8070b60 2023-10-14T16:56:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.998+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2822890572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 msgr2=0x7f6ac815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.998+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2822890572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 0x7f6ac815a9b0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800ba40 tx=0x7f6ab802dff0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2822890572 shutdown_connections 2023-10-14T16:56:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2822890572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac8070b60 0x7f6ac815a9b0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2822890572 >> v2:172.21.15.186:3300/0 conn(0x7f6ac806c100 0x7f6ac806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2822890572 >> 172.21.15.29:0/2822890572 conn(0x7f6ac80ce6a0 msgr2=0x7f6ac80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2822890572 shutdown_connections 2023-10-14T16:56:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:06.999+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2822890572 wait complete. 2023-10-14T16:56:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.001+0000 7f6ace9ba640 1 Processor -- start 2023-10-14T16:56:07.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.001+0000 7f6ace9ba640 1 -- start start 2023-10-14T16:56:07.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac806c100 0x7f6ac80aed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac8159950 0x7f6ac81553e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac81f33e0 con 0x7f6ac8070b60 2023-10-14T16:56:07.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80afc70 con 0x7f6ac8159950 2023-10-14T16:56:07.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6ace9ba640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ac80afef0 con 0x7f6ac806c100 2023-10-14T16:56:07.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.002+0000 7f6acd1b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45108/0 (socket says 172.21.15.29:45108) 2023-10-14T16:56:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2432109131 learned_addr learned my addr 172.21.15.29:0/2432109131 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6ace1b9640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac8159950 0x7f6ac81553e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd9b8640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac806c100 0x7f6ac80aed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac806c100 msgr2=0x7f6ac80aed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac806c100 0x7f6ac80aed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac8159950 msgr2=0x7f6ac81553e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac8159950 0x7f6ac81553e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 -- 172.21.15.29:0/2432109131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab800b6a0 con 0x7f6ac8070b60 2023-10-14T16:56:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.003+0000 7f6acd1b7640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800bb60 tx=0x7f6ab80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.004+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab803f640 con 0x7f6ac8070b60 2023-10-14T16:56:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.004+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab803f7a0 con 0x7f6ac8070b60 2023-10-14T16:56:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.004+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab803fb50 con 0x7f6ac8070b60 2023-10-14T16:56:07.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.004+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ac81f2a40 con 0x7f6ac8070b60 2023-10-14T16:56:07.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.004+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ac81f3000 con 0x7f6ac8070b60 2023-10-14T16:56:07.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.005+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ac81f3520 con 0x7f6ac8070b60 2023-10-14T16:56:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.006+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f6ab803d070 con 0x7f6ac8070b60 2023-10-14T16:56:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.007+0000 7f6abe7fc640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.007+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f6ab80abce0 con 0x7f6ac8070b60 2023-10-14T16:56:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.007+0000 7f6acd9b8640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:07.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.007+0000 7f6acd9b8640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:07.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.011+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ac81f3520 con 0x7f6ac8070b60 2023-10-14T16:56:07.146 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.145+0000 7ff2c4414040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:07.190 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:07.187+0000 7f22f1153040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:07.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.208+0000 7f6acd9b8640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:07.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.208+0000 7f6acd9b8640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:07.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.212+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ac81f3520 con 0x7f6ac8070b60 2023-10-14T16:56:07.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.214+0000 7f6abe7fc640 1 -- 172.21.15.29:0/2432109131 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v68) v1 ==== 74+0+17288 (secure 0 0 0) 0x7f6ac81f3520 con 0x7f6ac8070b60 2023-10-14T16:56:07.216 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:07.216 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:05.264246+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.38","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6833/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:6832/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/470082549":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/128331176":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2125394339":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/3085701795":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/4095875950":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/185522519":"2023-10-15T16:54:30.427919+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":7,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:07.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 msgr2=0x7f6aac06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:07.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 msgr2=0x7f6ac80af730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f6ab800bb60 tx=0x7f6ab80046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 shutdown_connections 2023-10-14T16:56:07.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7f6aac06af90 0x7f6aac06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ac8159950 0x7f6ac81553e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ac8070b60 0x7f6ac80af730 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 --2- 172.21.15.29:0/2432109131 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ac806c100 0x7f6ac80aed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 >> 172.21.15.29:0/2432109131 conn(0x7f6ac80ce6a0 msgr2=0x7f6ac80cdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:07.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 shutdown_connections 2023-10-14T16:56:07.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.217+0000 7f6ace9ba640 1 -- 172.21.15.29:0/2432109131 wait complete. 2023-10-14T16:56:07.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:54:12.966365+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:54:14.007924+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:07.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:07.244 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:07.243+0000 7f22f1153040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:07.360 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:07.358+0000 7f22f1153040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:07.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.440+0000 7fa029941640 1 Processor -- start 2023-10-14T16:56:07.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.440+0000 7fa029941640 1 -- start start 2023-10-14T16:56:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.441+0000 7fa029941640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa024159ab0 0x7fa0240ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.441+0000 7fa029941640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa024159e90 0x7fa02406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.441+0000 7fa029941640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0240cd1f0 con 0x7fa02406c140 2023-10-14T16:56:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.441+0000 7fa029941640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02415d890 con 0x7fa024159e90 2023-10-14T16:56:07.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.441+0000 7fa029941640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0240cd4d0 con 0x7fa024159ab0 2023-10-14T16:56:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa024159e90 0x7fa02406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa024159e90 0x7fa02406fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45110/0 (socket says 172.21.15.29:45110) 2023-10-14T16:56:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 -- 172.21.15.29:0/1601609144 learned_addr learned my addr 172.21.15.29:0/1601609144 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa022ffd640 1 --2- 172.21.15.29:0/1601609144 >> v2:172.21.15.186:3300/0 conn(0x7fa024159ab0 0x7fa0240ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 -- 172.21.15.29:0/1601609144 >> v2:172.21.15.186:3300/0 conn(0x7fa024159ab0 msgr2=0x7fa0240ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 --2- 172.21.15.29:0/1601609144 >> v2:172.21.15.186:3300/0 conn(0x7fa024159ab0 0x7fa0240ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 -- 172.21.15.29:0/1601609144 >> v1:172.21.15.186:6789/0 conn(0x7fa02406c140 legacy=0x7fa02415dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.442+0000 7fa01bfff640 1 -- 172.21.15.29:0/1601609144 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0240cd7b0 con 0x7fa024159e90 2023-10-14T16:56:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.443+0000 7fa01bfff640 1 --2- 172.21.15.29:0/1601609144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159e90 0x7fa02406fc50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa0240cfa00 tx=0x7fa00c02b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d467eeda15480361 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.444+0000 7fa0227fc640 1 -- 172.21.15.29:0/1601609144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c031750 con 0x7fa024159e90 2023-10-14T16:56:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.444+0000 7fa0227fc640 1 -- 172.21.15.29:0/1601609144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c0318b0 con 0x7fa024159e90 2023-10-14T16:56:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.444+0000 7fa0227fc640 1 -- 172.21.15.29:0/1601609144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c031c40 con 0x7fa024159e90 2023-10-14T16:56:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.444+0000 7fa029941640 1 -- 172.21.15.29:0/1601609144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159e90 msgr2=0x7fa02406fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.444+0000 7fa029941640 1 --2- 172.21.15.29:0/1601609144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159e90 0x7fa02406fc50 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fa0240cfa00 tx=0x7fa00c02b7f0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 -- 172.21.15.29:0/1601609144 shutdown_connections 2023-10-14T16:56:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 --2- 172.21.15.29:0/1601609144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159e90 0x7fa02406fc50 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 --2- 172.21.15.29:0/1601609144 >> v2:172.21.15.186:3300/0 conn(0x7fa024159ab0 0x7fa0240ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 -- 172.21.15.29:0/1601609144 >> 172.21.15.29:0/1601609144 conn(0x7fa0240ce9b0 msgr2=0x7fa0240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 -- 172.21.15.29:0/1601609144 shutdown_connections 2023-10-14T16:56:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.445+0000 7fa029941640 1 -- 172.21.15.29:0/1601609144 wait complete. 2023-10-14T16:56:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.446+0000 7fa029941640 1 Processor -- start 2023-10-14T16:56:07.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 -- start start 2023-10-14T16:56:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02406c140 0x7fa02415cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024159e90 0x7fa0241f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240c71d0 con 0x7fa024159ab0 2023-10-14T16:56:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240c74b0 con 0x7fa024159e90 2023-10-14T16:56:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.447+0000 7fa029941640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0240c7790 con 0x7fa02406c140 2023-10-14T16:56:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45122/0 (socket says 172.21.15.29:45122) 2023-10-14T16:56:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 -- 172.21.15.29:0/3755947701 learned_addr learned my addr 172.21.15.29:0/3755947701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 -- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02406c140 msgr2=0x7fa02415cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02406c140 0x7fa02415cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 -- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024159e90 msgr2=0x7fa0241f1f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024159e90 0x7fa0241f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.448+0000 7fa01bfff640 1 -- 172.21.15.29:0/3755947701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa00c0094d0 con 0x7fa024159ab0 2023-10-14T16:56:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa01bfff640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa00c000c00 tx=0x7fa00c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c033c80 con 0x7fa024159ab0 2023-10-14T16:56:07.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa00c033de0 con 0x7fa024159ab0 2023-10-14T16:56:07.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0240c2580 con 0x7fa024159ab0 2023-10-14T16:56:07.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa00c03c620 con 0x7fa024159ab0 2023-10-14T16:56:07.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.449+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0240c7a70 con 0x7fa024159ab0 2023-10-14T16:56:07.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.450+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0240c78d0 con 0x7fa024159ab0 2023-10-14T16:56:07.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.451+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 25) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fa00c03a040 con 0x7fa024159ab0 2023-10-14T16:56:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.452+0000 7fa020ff9640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 0x7fa00406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.452+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fa00c0ab6b0 con 0x7fa024159ab0 2023-10-14T16:56:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.452+0000 7fa022ffd640 1 -- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 msgr2=0x7fa00406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2060982901 2023-10-14T16:56:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.452+0000 7fa022ffd640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 0x7fa00406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.456+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0240c78d0 con 0x7fa024159ab0 2023-10-14T16:56:07.467 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.466+0000 7ff2c4414040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:07.528 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.527+0000 7ff2c4414040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:07.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.643+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa0240c78d0 con 0x7fa024159ab0 2023-10-14T16:56:07.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.643+0000 7fa020ff9640 1 -- 172.21.15.29:0/3755947701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v68) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa00c038020 con 0x7fa024159ab0 2023-10-14T16:56:07.644 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 msgr2=0x7fa00406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 0x7fa00406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 msgr2=0x7fa0240c6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fa00c000c00 tx=0x7fa00c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 shutdown_connections 2023-10-14T16:56:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:6832/2060982901,v1:172.21.15.29:6833/2060982901] conn(0x7fa00406af90 0x7fa00406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa024159e90 0x7fa0241f1f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa024159ab0 0x7fa0240c6ad0 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 --2- 172.21.15.29:0/3755947701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02406c140 0x7fa02415cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 >> 172.21.15.29:0/3755947701 conn(0x7fa0240ce9b0 msgr2=0x7fa0240acf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 shutdown_connections 2023-10-14T16:56:07.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.646+0000 7fa029941640 1 -- 172.21.15.29:0/3755947701 wait complete. 2023-10-14T16:56:07.650 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.648+0000 7ff2c4414040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:07.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:56:07.728 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff24945e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff24945e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff24945e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff24945e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff24945e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff246458640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff246458640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff246458640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff246458640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.731 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:07.728+0000 7ff246458640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.856+0000 7fbfa2553640 1 Processor -- start 2023-10-14T16:56:07.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.856+0000 7fbfa2553640 1 -- start start 2023-10-14T16:56:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.857+0000 7fbfa2553640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf9c06c850 0x7fbf9c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.857+0000 7fbfa2553640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c070360 con 0x7fbf9c159e90 2023-10-14T16:56:07.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa2553640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf9c0cd1f0 con 0x7fbf9c159ab0 2023-10-14T16:56:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa2553640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf9c15d890 con 0x7fbf9c06c850 2023-10-14T16:56:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa0af0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf9c06c850 0x7fbf9c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbf9b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf9c159e90 0x7fbf9c06c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40634/0 (socket says 172.21.15.29:40634) 2023-10-14T16:56:07.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbf9b7fe640 1 -- 172.21.15.29:0/4202146271 learned_addr learned my addr 172.21.15.29:0/4202146271 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:07.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa0af0640 1 -- 172.21.15.29:0/4202146271 >> v1:172.21.15.186:6789/0 conn(0x7fbf9c159ab0 legacy=0x7fbf9c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa0af0640 1 -- 172.21.15.29:0/4202146271 >> v1:172.21.15.29:6789/0 conn(0x7fbf9c159e90 legacy=0x7fbf9c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.858+0000 7fbfa0af0640 1 -- 172.21.15.29:0/4202146271 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf9c0cd4d0 con 0x7fbf9c06c850 2023-10-14T16:56:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.859+0000 7fbfa0af0640 1 --2- 172.21.15.29:0/4202146271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c06c850 0x7fbf9c0ce370 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009d90 tx=0x7fbf9402bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6708bd2b4155615c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.859+0000 7fbf9affd640 1 -- 172.21.15.29:0/4202146271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9402cb40 con 0x7fbf9c06c850 2023-10-14T16:56:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbf9affd640 1 -- 172.21.15.29:0/4202146271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9402cca0 con 0x7fbf9c06c850 2023-10-14T16:56:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbf9affd640 1 -- 172.21.15.29:0/4202146271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf94034660 con 0x7fbf9c06c850 2023-10-14T16:56:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 -- 172.21.15.29:0/4202146271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c06c850 msgr2=0x7fbf9c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 --2- 172.21.15.29:0/4202146271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c06c850 0x7fbf9c0ce370 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009d90 tx=0x7fbf9402bd90 comp rx=0 tx=0).stop 2023-10-14T16:56:07.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 -- 172.21.15.29:0/4202146271 shutdown_connections 2023-10-14T16:56:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 --2- 172.21.15.29:0/4202146271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c06c850 0x7fbf9c0ce370 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 -- 172.21.15.29:0/4202146271 >> 172.21.15.29:0/4202146271 conn(0x7fbf9c0ce9b0 msgr2=0x7fbf9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 -- 172.21.15.29:0/4202146271 shutdown_connections 2023-10-14T16:56:07.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.860+0000 7fbfa2553640 1 -- 172.21.15.29:0/4202146271 wait complete. 2023-10-14T16:56:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.862+0000 7fbfa2553640 1 Processor -- start 2023-10-14T16:56:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.862+0000 7fbfa2553640 1 -- start start 2023-10-14T16:56:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 0x7fbf9c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 0x7fbf9c1f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c153af0 con 0x7fbf9c159e90 2023-10-14T16:56:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c154a30 con 0x7fbf9c06c850 2023-10-14T16:56:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa2553640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf9c154d10 con 0x7fbf9c159ab0 2023-10-14T16:56:07.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa0af0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa0af0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45238/0 (socket says 172.21.15.29:45238) 2023-10-14T16:56:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.863+0000 7fbfa0af0640 1 -- 172.21.15.29:0/23643838 learned_addr learned my addr 172.21.15.29:0/23643838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf9bfff640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 0x7fbf9c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf9b7fe640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 0x7fbf9c1f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:07.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 -- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 msgr2=0x7fbf9c1f03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 0x7fbf9c1f03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 -- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 msgr2=0x7fbf9c1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 0x7fbf9c1535b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 -- 172.21.15.29:0/23643838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf94009a70 con 0x7fbf9c159e90 2023-10-14T16:56:07.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf9b7fe640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 0x7fbf9c1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf9bfff640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 0x7fbf9c1535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa0af0640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009d90 tx=0x7fbf94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9403e610 con 0x7fbf9c159e90 2023-10-14T16:56:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf9403e770 con 0x7fbf9c159e90 2023-10-14T16:56:07.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf9403eb20 con 0x7fbf9c159e90 2023-10-14T16:56:07.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.864+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf9c154ff0 con 0x7fbf9c159e90 2023-10-14T16:56:07.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.865+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf9c1555b0 con 0x7fbf9c159e90 2023-10-14T16:56:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.866+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf9c04e160 con 0x7fbf9c159e90 2023-10-14T16:56:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.867+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbf9403b050 con 0x7fbf9c159e90 2023-10-14T16:56:07.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.867+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6527+0+0 (secure 0 0 0) 0x7fbf940abd10 con 0x7fbf9c159e90 2023-10-14T16:56:07.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:07.872+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf940b1020 con 0x7fbf9c159e90 2023-10-14T16:56:08.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.061+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbf9c0cd780 con 0x7fbf9c159e90 2023-10-14T16:56:08.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.062+0000 7fbf98ff9640 1 -- 172.21.15.29:0/23643838 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v69) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbf940390a0 con 0x7fbf9c159e90 2023-10-14T16:56:08.063 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:56:08.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.064+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 msgr2=0x7fbf9c15c000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.064+0000 7fbfa2553640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fbf94009d90 tx=0x7fbf94004a40 comp rx=0 tx=0).stop 2023-10-14T16:56:08.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 shutdown_connections 2023-10-14T16:56:08.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf9c159e90 0x7fbf9c15c000 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf9c159ab0 0x7fbf9c1f03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 --2- 172.21.15.29:0/23643838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf9c06c850 0x7fbf9c1535b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 >> 172.21.15.29:0/23643838 conn(0x7fbf9c0ce9b0 msgr2=0x7fbf9c159010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 shutdown_connections 2023-10-14T16:56:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.065+0000 7fbfa2553640 1 -- 172.21.15.29:0/23643838 wait complete. 2023-10-14T16:56:08.082 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:56:08.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.302+0000 7f7c1a916640 1 Processor -- start 2023-10-14T16:56:08.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.302+0000 7f7c1a916640 1 -- start start 2023-10-14T16:56:08.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c1a916640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7c140ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c1a916640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c140cb030 con 0x7f7c1406a4b0 2023-10-14T16:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c1a916640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c140c91c0 con 0x7f7c1415ad70 2023-10-14T16:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c1a916640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140c94a0 con 0x7f7c140caac0 2023-10-14T16:56:08.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c137fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7c1406a4b0 0x7f7c140ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40642/0 (socket says 172.21.15.29:40642) 2023-10-14T16:56:08.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c137fe640 1 -- 172.21.15.29:0/4221195046 learned_addr learned my addr 172.21.15.29:0/4221195046 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.304+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092946563 0 0) 0x7f7c140cb030 con 0x7f7c1406a4b0 2023-10-14T16:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bfc003520 con 0x7f7c1406a4b0 2023-10-14T16:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7c140ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009779348 0 0) 0x7f7bfc003520 con 0x7f7c1406a4b0 2023-10-14T16:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 msgr2=0x7f7c140ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7c140ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 >> v1:172.21.15.186:6789/0 conn(0x7f7c1415ad70 legacy=0x7f7c14069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c140c9780 con 0x7f7c1406a4b0 2023-10-14T16:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7c08001670 con 0x7f7c1406a4b0 2023-10-14T16:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 >> v1:172.21.15.29:6789/0 conn(0x7f7c1406a4b0 legacy=0x7f7c140ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc0039a0 0x7f7bfc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7bfc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc004d00 0x7f7bfc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140c94a0 con 0x7f7bfc004d00 2023-10-14T16:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc00fe70 con 0x7f7bfc0039a0 2023-10-14T16:56:08.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.305+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc010030 con 0x7f7c140caac0 2023-10-14T16:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc004d00 0x7f7bfc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c137fe640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7bfc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c13fff640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc0039a0 0x7f7bfc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 -- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 msgr2=0x7f7bfc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:08.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c140caac0 0x7f7bfc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 -- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc0039a0 msgr2=0x7f7bfc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:08.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc0039a0 0x7f7bfc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.306+0000 7f7c18eb3640 1 -- 172.21.15.29:0/4221195046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc003520 con 0x7f7bfc004d00 2023-10-14T16:56:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.307+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc004d00 0x7f7bfc005180 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009f90 tx=0x7f7bfc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d8d538c5cc32ba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c02cbd0 con 0x7f7bfc004d00 2023-10-14T16:56:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c00f040 con 0x7f7bfc004d00 2023-10-14T16:56:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c12ffd640 1 -- 172.21.15.29:0/4221195046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c034440 con 0x7f7bfc004d00 2023-10-14T16:56:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c1a916640 1 -- 172.21.15.29:0/4221195046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc004d00 msgr2=0x7f7bfc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c1a916640 1 --2- 172.21.15.29:0/4221195046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc004d00 0x7f7bfc005180 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009f90 tx=0x7f7bfc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c18eb3640 1 -- 172.21.15.29:0/4221195046 reap_dead start 2023-10-14T16:56:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c1a916640 1 -- 172.21.15.29:0/4221195046 shutdown_connections 2023-10-14T16:56:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.308+0000 7f7c1a916640 1 -- 172.21.15.29:0/4221195046 >> 172.21.15.29:0/4221195046 conn(0x7f7c140ce9b0 msgr2=0x7f7c140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.309+0000 7f7c1a916640 1 -- 172.21.15.29:0/4221195046 shutdown_connections 2023-10-14T16:56:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.309+0000 7f7c1a916640 1 -- 172.21.15.29:0/4221195046 wait complete. 2023-10-14T16:56:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.310+0000 7f7c1a916640 1 Processor -- start 2023-10-14T16:56:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.310+0000 7f7c1a916640 1 -- start start 2023-10-14T16:56:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc004d00 0x7f7c140c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c141eef70 0x7f7c141552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c140c8640 con 0x7f7c140c7c50 2023-10-14T16:56:08.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c141ef3d0 con 0x7f7c141eef70 2023-10-14T16:56:08.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.311+0000 7f7c1a916640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c14155810 con 0x7f7bfc004d00 2023-10-14T16:56:08.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45258/0 (socket says 172.21.15.29:45258) 2023-10-14T16:56:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 -- 172.21.15.29:0/1368060613 learned_addr learned my addr 172.21.15.29:0/1368060613 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 -- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc004d00 msgr2=0x7f7c140c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc004d00 0x7f7c140c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 -- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c141eef70 msgr2=0x7f7c141552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c141eef70 0x7f7c141552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 -- 172.21.15.29:0/1368060613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c140c6110 con 0x7f7c140c7c50 2023-10-14T16:56:08.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c18eb3640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c037040 tx=0x7f7c0c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:08.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.312+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c003b90 con 0x7f7c140c7c50 2023-10-14T16:56:08.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.313+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c0c003d10 con 0x7f7c140c7c50 2023-10-14T16:56:08.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.313+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c034b60 con 0x7f7c140c7c50 2023-10-14T16:56:08.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.313+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c140c63f0 con 0x7f7c140c7c50 2023-10-14T16:56:08.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.313+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c140c69b0 con 0x7f7c140c7c50 2023-10-14T16:56:08.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.314+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c1404e160 con 0x7f7c140c7c50 2023-10-14T16:56:08.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.315+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 26) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f7c0c034cc0 con 0x7f7c140c7c50 2023-10-14T16:56:08.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.316+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f7c0c0aad90 con 0x7f7c140c7c50 2023-10-14T16:56:08.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.320+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c0c076010 con 0x7f7c140c7c50 2023-10-14T16:56:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.513+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7c1415e310 con 0x7f7c140c7c50 2023-10-14T16:56:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.514+0000 7f7c10ff9640 1 -- 172.21.15.29:0/1368060613 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v69) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7c0c039020 con 0x7f7c140c7c50 2023-10-14T16:56:08.515 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:56:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 msgr2=0x7f7c140c8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c037040 tx=0x7f7c0c009750 comp rx=0 tx=0).stop 2023-10-14T16:56:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 shutdown_connections 2023-10-14T16:56:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c140c7c50 0x7f7c140c8100 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c141eef70 0x7f7c141552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 --2- 172.21.15.29:0/1368060613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc004d00 0x7f7c140c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 >> 172.21.15.29:0/1368060613 conn(0x7f7c140ce9b0 msgr2=0x7f7c140b16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 shutdown_connections 2023-10-14T16:56:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.517+0000 7f7c1a916640 1 -- 172.21.15.29:0/1368060613 wait complete. 2023-10-14T16:56:08.537 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:08.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.773+0000 7f4ddb59e640 1 Processor -- start 2023-10-14T16:56:08.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.773+0000 7f4ddb59e640 1 -- start start 2023-10-14T16:56:08.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddb59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ddc153740 0x7f4ddc15a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddb59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 0x7f4ddc0ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddb59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ddc0cd4a0 con 0x7f4ddc0ad4d0 2023-10-14T16:56:08.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddb59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc15ac60 con 0x7f4ddc153740 2023-10-14T16:56:08.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddb59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ddc070030 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4dda59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4ddc153740 0x7f4ddc15a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4ddc0ad4d0 0x7f4ddc0ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40652/0 (socket says 172.21.15.29:40652) 2023-10-14T16:56:08.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.775+0000 7f4ddad9d640 1 -- 172.21.15.29:0/3116047221 learned_addr learned my addr 172.21.15.29:0/3116047221 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:08.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dd1d9b640 1 --2- 172.21.15.29:0/3116047221 >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 0x7f4ddc0ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dda59c640 1 -- 172.21.15.29:0/3116047221 >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 msgr2=0x7f4ddc0ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:08.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dda59c640 1 --2- 172.21.15.29:0/3116047221 >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 0x7f4ddc0ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dda59c640 1 -- 172.21.15.29:0/3116047221 >> v1:172.21.15.29:6789/0 conn(0x7f4ddc0ad4d0 legacy=0x7f4ddc0ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dda59c640 1 -- 172.21.15.29:0/3116047221 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ddc070310 con 0x7f4ddc153740 2023-10-14T16:56:08.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dd9d9b640 1 -- 172.21.15.29:0/3116047221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3333021469 0 0) 0x7f4ddc0cd4a0 con 0x7f4ddc0ad4d0 2023-10-14T16:56:08.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.776+0000 7f4dd1d9b640 1 --2- 172.21.15.29:0/3116047221 >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 0x7f4ddc0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:08.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.777+0000 7f4dda59c640 1 --2- 172.21.15.29:0/3116047221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc153740 0x7f4ddc15a0e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc06d6e0 tx=0x7f4dcc02c950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd0703043a96027b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.777+0000 7f4dd9d9b640 1 -- 172.21.15.29:0/3116047221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc031610 con 0x7f4ddc153740 2023-10-14T16:56:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.777+0000 7f4dd9d9b640 1 -- 172.21.15.29:0/3116047221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc031770 con 0x7f4ddc153740 2023-10-14T16:56:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.777+0000 7f4dd9d9b640 1 -- 172.21.15.29:0/3116047221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc031b00 con 0x7f4ddc153740 2023-10-14T16:56:08.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 -- 172.21.15.29:0/3116047221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc153740 msgr2=0x7f4ddc15a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/3116047221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc153740 0x7f4ddc15a0e0 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f4ddc06d6e0 tx=0x7f4dcc02c950 comp rx=0 tx=0).stop 2023-10-14T16:56:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 -- 172.21.15.29:0/3116047221 shutdown_connections 2023-10-14T16:56:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/3116047221 >> v2:172.21.15.186:3300/0 conn(0x7f4ddc0ccb00 0x7f4ddc0ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/3116047221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc153740 0x7f4ddc15a0e0 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 -- 172.21.15.29:0/3116047221 >> 172.21.15.29:0/3116047221 conn(0x7f4ddc0ce6a0 msgr2=0x7f4ddc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 -- 172.21.15.29:0/3116047221 shutdown_connections 2023-10-14T16:56:08.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.778+0000 7f4ddb59e640 1 -- 172.21.15.29:0/3116047221 wait complete. 2023-10-14T16:56:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.780+0000 7f4ddb59e640 1 Processor -- start 2023-10-14T16:56:08.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.780+0000 7f4ddb59e640 1 -- start start 2023-10-14T16:56:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ddc0ad4d0 0x7f4ddc0af1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 0x7f4ddc14fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0c4ea0 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc1583a0 con 0x7f4ddc153740 2023-10-14T16:56:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4ddb59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ddc0af780 con 0x7f4ddc0ad4d0 2023-10-14T16:56:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4dd1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4dd1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45270/0 (socket says 172.21.15.29:45270) 2023-10-14T16:56:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.781+0000 7f4dd1d9b640 1 -- 172.21.15.29:0/216187735 learned_addr learned my addr 172.21.15.29:0/216187735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:08.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4ddad9d640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 0x7f4ddc14fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 -- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ddc0ad4d0 msgr2=0x7f4ddc0af1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ddc0ad4d0 0x7f4ddc0af1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 -- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 msgr2=0x7f4ddc14fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 0x7f4ddc14fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:08.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 -- 172.21.15.29:0/216187735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc009400 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4ddad9d640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 0x7f4ddc14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:08.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd1d9b640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400c820 tx=0x7f4dc400cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:08.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc4007a30 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc4007b90 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.782+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ddc0c47e0 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.783+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc400ce70 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.783+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ddc150070 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.784+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ddc06fc30 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.785+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4dc40113f0 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.786+0000 7f4dd37fe640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 0x7f4dbc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:08.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.786+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f4dc40897d0 con 0x7f4ddc0ccb00 2023-10-14T16:56:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.786+0000 7f4dda59c640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 0x7f4dbc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.787+0000 7f4dda59c640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 0x7f4dbc06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc037000 tx=0x7f4dcc031000 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:08.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:08.791+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4dc40547c0 con 0x7f4ddc0ccb00 2023-10-14T16:56:09.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.025+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ddc04e9b0 con 0x7f4ddc0ccb00 2023-10-14T16:56:09.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.027+0000 7f4dd37fe640 1 -- 172.21.15.29:0/216187735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f4dc4057de0 con 0x7f4ddc0ccb00 2023-10-14T16:56:09.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:09.028 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:56:09.028 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:56:09.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.029+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 msgr2=0x7f4dbc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.029+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 0x7f4dbc06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc037000 tx=0x7f4dcc031000 comp rx=0 tx=0).stop 2023-10-14T16:56:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.029+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 msgr2=0x7f4ddc157e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.029+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7f4dc400c820 tx=0x7f4dc400cc20 comp rx=0 tx=0).stop 2023-10-14T16:56:09.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 shutdown_connections 2023-10-14T16:56:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f4dbc06acc0 0x7f4dbc06d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ddc153740 0x7f4ddc14fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ddc0ccb00 0x7f4ddc157e60 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 --2- 172.21.15.29:0/216187735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4ddc0ad4d0 0x7f4ddc0af1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 >> 172.21.15.29:0/216187735 conn(0x7f4ddc0ce6a0 msgr2=0x7f4ddc151f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:09.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 shutdown_connections 2023-10-14T16:56:09.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.030+0000 7f4ddb59e640 1 -- 172.21.15.29:0/216187735 wait complete. 2023-10-14T16:56:09.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:09.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.275+0000 7f7bec9bf640 1 Processor -- start 2023-10-14T16:56:09.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.276+0000 7f7bec9bf640 1 -- start start 2023-10-14T16:56:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.277+0000 7f7bec9bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7be80cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.277+0000 7f7bec9bf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7be806c9e0 0x7f7be80ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.277+0000 7f7bec9bf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7be80ca560 con 0x7f7be80c9060 2023-10-14T16:56:09.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.277+0000 7f7bec9bf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7be80c9950 con 0x7f7be806c9e0 2023-10-14T16:56:09.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.277+0000 7f7bec9bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80c9c30 con 0x7f7be806fb50 2023-10-14T16:56:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be6d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7be80c9060 0x7f7be806f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40656/0 (socket says 172.21.15.29:40656) 2023-10-14T16:56:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be6d9d640 1 -- 172.21.15.29:0/3198222516 learned_addr learned my addr 172.21.15.29:0/3198222516 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be659c640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7be80cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be759e640 1 --2- 172.21.15.29:0/3198222516 >> v2:172.21.15.186:3300/0 conn(0x7f7be806c9e0 0x7f7be80ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2898770477 0 0) 0x7f7be80ca560 con 0x7f7be80c9060 2023-10-14T16:56:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.278+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bc8003520 con 0x7f7be80c9060 2023-10-14T16:56:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2537122381 0 0) 0x7f7bc8003520 con 0x7f7be80c9060 2023-10-14T16:56:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 msgr2=0x7f7be80cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7be80cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 >> v2:172.21.15.186:3300/0 conn(0x7f7be806c9e0 msgr2=0x7f7be80ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 --2- 172.21.15.29:0/3198222516 >> v2:172.21.15.186:3300/0 conn(0x7f7be806c9e0 0x7f7be80ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80c9f10 con 0x7f7be80c9060 2023-10-14T16:56:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7bd8001670 con 0x7f7be80c9060 2023-10-14T16:56:09.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 >> v1:172.21.15.29:6789/0 conn(0x7f7be80c9060 legacy=0x7f7be806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7bc8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bc8003f40 0x7f7bc80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7bc8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80c9950 con 0x7f7bc8004d00 2023-10-14T16:56:09.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80c9c30 con 0x7f7bc8003f40 2023-10-14T16:56:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bc800b980 con 0x7f7be806fb50 2023-10-14T16:56:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be759e640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7bc8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.279+0000 7f7be659c640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bc8003f40 0x7f7bc80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be6d9d640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7bc8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be759e640 1 -- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 msgr2=0x7f7bc8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be759e640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7bc8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be759e640 1 -- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bc8003f40 msgr2=0x7f7bc80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be759e640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bc8003f40 0x7f7bc80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be759e640 1 -- 172.21.15.29:0/3198222516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bc8003520 con 0x7f7bc8004d00 2023-10-14T16:56:09.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.280+0000 7f7be6d9d640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be806fb50 0x7f7bc8003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.281+0000 7f7be759e640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7bc8005180 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009a90 tx=0x7f7bc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902b8649f24f0ed1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.281+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be002c680 con 0x7f7bc8004d00 2023-10-14T16:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.281+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be002c7e0 con 0x7f7bc8004d00 2023-10-14T16:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.281+0000 7f7be5d9b640 1 -- 172.21.15.29:0/3198222516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7be002cb70 con 0x7f7bc8004d00 2023-10-14T16:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 -- 172.21.15.29:0/3198222516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 msgr2=0x7f7bc8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/3198222516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7bc8005180 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f7be0009a90 tx=0x7f7bc80057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7be759e640 1 -- 172.21.15.29:0/3198222516 reap_dead start 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 -- 172.21.15.29:0/3198222516 shutdown_connections 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 -- 172.21.15.29:0/3198222516 >> 172.21.15.29:0/3198222516 conn(0x7f7be80ce6a0 msgr2=0x7f7be80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 -- 172.21.15.29:0/3198222516 shutdown_connections 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.282+0000 7f7bec9bf640 1 -- 172.21.15.29:0/3198222516 wait complete. 2023-10-14T16:56:09.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.284+0000 7f7bec9bf640 1 Processor -- start 2023-10-14T16:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.284+0000 7f7bec9bf640 1 -- start start 2023-10-14T16:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 0x7f7be80b3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 0x7f7be80b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80c17e0 con 0x7f7bc8004d00 2023-10-14T16:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80b1980 con 0x7f7be815eef0 2023-10-14T16:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7bec9bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7be80b1c60 con 0x7f7be815f330 2023-10-14T16:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.285+0000 7f7be6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45282/0 (socket says 172.21.15.29:45282) 2023-10-14T16:56:09.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 -- 172.21.15.29:0/2356941891 learned_addr learned my addr 172.21.15.29:0/2356941891 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be659c640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 0x7f7be80b3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be759e640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 0x7f7be80b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 -- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 msgr2=0x7f7be80b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 0x7f7be80b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 -- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 msgr2=0x7f7be80b3010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 0x7f7be80b3010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 -- 172.21.15.29:0/2356941891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7be80b1f40 con 0x7f7bc8004d00 2023-10-14T16:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be659c640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 0x7f7be80b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be759e640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 0x7f7be80b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:09.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.286+0000 7f7be6d9d640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8001200 tx=0x7f7bd800ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.288+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800c8d0 con 0x7f7bc8004d00 2023-10-14T16:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.288+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bd800ca30 con 0x7f7bc8004d00 2023-10-14T16:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.288+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bd800cda0 con 0x7f7bc8004d00 2023-10-14T16:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.288+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be80b2220 con 0x7f7bc8004d00 2023-10-14T16:56:09.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.288+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be80b2a20 con 0x7f7bc8004d00 2023-10-14T16:56:09.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.290+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be804e9b0 con 0x7f7bc8004d00 2023-10-14T16:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.291+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7bd801d460 con 0x7f7bc8004d00 2023-10-14T16:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.291+0000 7f7bdf7fe640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 0x7f7bcc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.292+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f7bd8089750 con 0x7f7bc8004d00 2023-10-14T16:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.292+0000 7f7be659c640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 0x7f7bcc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.293+0000 7f7be659c640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 0x7f7bcc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7be81f9920 tx=0x7f7bd000b040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.296+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7bd80546b0 con 0x7f7bc8004d00 2023-10-14T16:56:09.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.491+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7be80b2bb0 con 0x7f7bc8004d00 2023-10-14T16:56:09.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.493+0000 7f7bdf7fe640 1 -- 172.21.15.29:0/2356941891 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v69) v1 ==== 74+0+17724 (secure 0 0 0) 0x7f7bd8057cd0 con 0x7f7bc8004d00 2023-10-14T16:56:09.494 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:09.495 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":69,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:07.650967+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.38","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2114169573":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2811289807":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/1660561604":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2419772294":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6832/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/3005757508":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:6832/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/470082549":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/128331176":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2125394339":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/3085701795":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/4095875950":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2968229624":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/185522519":"2023-10-15T16:54:30.427919+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":7,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:09.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 msgr2=0x7f7bcc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 0x7f7bcc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7be81f9920 tx=0x7f7bd000b040 comp rx=0 tx=0).stop 2023-10-14T16:56:09.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 msgr2=0x7f7be80b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8001200 tx=0x7f7bd800ec80 comp rx=0 tx=0).stop 2023-10-14T16:56:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 shutdown_connections 2023-10-14T16:56:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f7bcc06acc0 0x7f7bcc06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7be815f330 0x7f7be80b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7be815eef0 0x7f7be80b3010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 --2- 172.21.15.29:0/2356941891 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bc8004d00 0x7f7be80b12c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 >> 172.21.15.29:0/2356941891 conn(0x7f7be80ce6a0 msgr2=0x7f7be80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 shutdown_connections 2023-10-14T16:56:09.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.496+0000 7f7bec9bf640 1 -- 172.21.15.29:0/2356941891 wait complete. 2023-10-14T16:56:09.515 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:56:09.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.727+0000 7ff06a82e640 1 Processor -- start 2023-10-14T16:56:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.728+0000 7ff06a82e640 1 -- start start 2023-10-14T16:56:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.729+0000 7ff06a82e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff0640cd870 0x7ff0640718f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.729+0000 7ff06a82e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0640cdc50 0x7ff0640cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.729+0000 7ff06a82e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0640cca60 con 0x7ff06415ac80 2023-10-14T16:56:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.729+0000 7ff06a82e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0640ccd40 con 0x7ff0640cdc50 2023-10-14T16:56:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.729+0000 7ff06a82e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0640cd020 con 0x7ff0640cd870 2023-10-14T16:56:09.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0640cdc50 0x7ff0640cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0640cdc50 0x7ff0640cfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45288/0 (socket says 172.21.15.29:45288) 2023-10-14T16:56:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 -- 172.21.15.29:0/1392821966 learned_addr learned my addr 172.21.15.29:0/1392821966 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:09.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06982c640 1 --2- 172.21.15.29:0/1392821966 >> v2:172.21.15.186:3300/0 conn(0x7ff0640cd870 0x7ff0640718f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 -- 172.21.15.29:0/1392821966 >> v2:172.21.15.186:3300/0 conn(0x7ff0640cd870 msgr2=0x7ff0640718f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 --2- 172.21.15.29:0/1392821966 >> v2:172.21.15.186:3300/0 conn(0x7ff0640cd870 0x7ff0640718f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 -- 172.21.15.29:0/1392821966 >> v1:172.21.15.186:6789/0 conn(0x7ff06415ac80 legacy=0x7ff0640cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.730+0000 7ff06a02d640 1 -- 172.21.15.29:0/1392821966 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0640ce4b0 con 0x7ff0640cdc50 2023-10-14T16:56:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06a02d640 1 --2- 172.21.15.29:0/1392821966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cdc50 0x7ff0640cfab0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009e50 tx=0x7ff05c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac3b7952b8490734 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06882a640 1 -- 172.21.15.29:0/1392821966 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c02ca80 con 0x7ff0640cdc50 2023-10-14T16:56:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06882a640 1 -- 172.21.15.29:0/1392821966 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c02cbe0 con 0x7ff0640cdc50 2023-10-14T16:56:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06882a640 1 -- 172.21.15.29:0/1392821966 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c0343f0 con 0x7ff0640cdc50 2023-10-14T16:56:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06a82e640 1 -- 172.21.15.29:0/1392821966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cdc50 msgr2=0x7ff0640cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.732+0000 7ff06a82e640 1 --2- 172.21.15.29:0/1392821966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cdc50 0x7ff0640cfab0 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7ff05c009e50 tx=0x7ff05c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 -- 172.21.15.29:0/1392821966 shutdown_connections 2023-10-14T16:56:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 --2- 172.21.15.29:0/1392821966 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cdc50 0x7ff0640cfab0 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 --2- 172.21.15.29:0/1392821966 >> v2:172.21.15.186:3300/0 conn(0x7ff0640cd870 0x7ff0640718f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 -- 172.21.15.29:0/1392821966 >> 172.21.15.29:0/1392821966 conn(0x7ff064065ed0 msgr2=0x7ff0640662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 -- 172.21.15.29:0/1392821966 shutdown_connections 2023-10-14T16:56:09.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.733+0000 7ff06a82e640 1 -- 172.21.15.29:0/1392821966 wait complete. 2023-10-14T16:56:09.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.734+0000 7ff06a82e640 1 Processor -- start 2023-10-14T16:56:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.735+0000 7ff06a82e640 1 -- start start 2023-10-14T16:56:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0640cdc50 0x7ff0641f24b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff06415ac80 0x7ff0641f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640c5e90 con 0x7ff0640cd870 2023-10-14T16:56:09.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640c6170 con 0x7ff06415ac80 2023-10-14T16:56:09.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06a82e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0640c6450 con 0x7ff0640cdc50 2023-10-14T16:56:09.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45290/0 (socket says 172.21.15.29:45290) 2023-10-14T16:56:09.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 -- 172.21.15.29:0/242523164 learned_addr learned my addr 172.21.15.29:0/242523164 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 -- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0640cdc50 msgr2=0x7ff0641f24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0640cdc50 0x7ff0641f24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 -- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff06415ac80 msgr2=0x7ff0641f6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff06415ac80 0x7ff0641f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.736+0000 7ff06982c640 1 -- 172.21.15.29:0/242523164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c009b30 con 0x7ff0640cd870 2023-10-14T16:56:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff06982c640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d7d0 tx=0x7ff06000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0600096c0 con 0x7ff0640cd870 2023-10-14T16:56:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff060009820 con 0x7ff0640cd870 2023-10-14T16:56:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff060009bd0 con 0x7ff0640cd870 2023-10-14T16:56:09.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0640c6730 con 0x7ff0640cd870 2023-10-14T16:56:09.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.737+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0641f29f0 con 0x7ff0640cd870 2023-10-14T16:56:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.738+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0640cf8f0 con 0x7ff0640cd870 2023-10-14T16:56:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.739+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff060009d30 con 0x7ff0640cd870 2023-10-14T16:56:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.740+0000 7ff05a7fc640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 0x7ff04806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:09.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.740+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6527+0+0 (secure 0 0 0) 0x7ff060089e00 con 0x7ff0640cd870 2023-10-14T16:56:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.740+0000 7ff06902b640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 0x7ff04806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.741+0000 7ff06902b640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 0x7ff04806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff06406bfc0 tx=0x7ff054007990 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:09.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.745+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff060054df0 con 0x7ff0640cd870 2023-10-14T16:56:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.990+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff06404e9b0 con 0x7ff0640cd870 2023-10-14T16:56:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.991+0000 7ff05a7fc640 1 -- 172.21.15.29:0/242523164 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7ff060058410 con 0x7ff0640cd870 2023-10-14T16:56:09.992 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:09.992 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:56:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:56:09.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.993+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 msgr2=0x7ff04806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 0x7ff04806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff06406bfc0 tx=0x7ff054007990 comp rx=0 tx=0).stop 2023-10-14T16:56:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 msgr2=0x7ff06415da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7ff06000d7d0 tx=0x7ff06000dc90 comp rx=0 tx=0).stop 2023-10-14T16:56:09.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 shutdown_connections 2023-10-14T16:56:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff04806acc0 0x7ff04806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff06415ac80 0x7ff0641f6190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0640cdc50 0x7ff0641f24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 --2- 172.21.15.29:0/242523164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0640cd870 0x7ff06415da90 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:09.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 >> 172.21.15.29:0/242523164 conn(0x7ff064065ed0 msgr2=0x7ff0640d0200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:09.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 shutdown_connections 2023-10-14T16:56:09.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:09.994+0000 7ff06a82e640 1 -- 172.21.15.29:0/242523164 wait complete. 2023-10-14T16:56:10.014 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:10.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.243+0000 7f2338494640 1 Processor -- start 2023-10-14T16:56:10.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.243+0000 7f2338494640 1 -- start start 2023-10-14T16:56:10.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.244+0000 7f2338494640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2330159f00 0x7f23300b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.244+0000 7f2338494640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233006f7f0 con 0x7f233015d880 2023-10-14T16:56:10.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.244+0000 7f2338494640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f233006c5e0 con 0x7f2330159b20 2023-10-14T16:56:10.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.244+0000 7f2338494640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f233006c8c0 con 0x7f2330159f00 2023-10-14T16:56:10.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2335a2f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2330159f00 0x7f23300b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2336a31640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f233015d880 0x7f233006fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40662/0 (socket says 172.21.15.29:40662) 2023-10-14T16:56:10.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2336a31640 1 -- 172.21.15.29:0/1994573948 learned_addr learned my addr 172.21.15.29:0/1994573948 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:10.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2335a2f640 1 -- 172.21.15.29:0/1994573948 >> v1:172.21.15.186:6789/0 conn(0x7f2330159b20 legacy=0x7f233006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:10.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2335a2f640 1 -- 172.21.15.29:0/1994573948 >> v1:172.21.15.29:6789/0 conn(0x7f233015d880 legacy=0x7f233006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.245+0000 7f2335a2f640 1 -- 172.21.15.29:0/1994573948 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f233006cba0 con 0x7f2330159f00 2023-10-14T16:56:10.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.246+0000 7f2335a2f640 1 --2- 172.21.15.29:0/1994573948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159f00 0x7f23300b0410 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f23200097a0 tx=0x7f232002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ccc73b472513c0e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.246+0000 7f233522e640 1 -- 172.21.15.29:0/1994573948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320031830 con 0x7f2330159f00 2023-10-14T16:56:10.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f233522e640 1 -- 172.21.15.29:0/1994573948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2320031990 con 0x7f2330159f00 2023-10-14T16:56:10.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f233522e640 1 -- 172.21.15.29:0/1994573948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2320031d20 con 0x7f2330159f00 2023-10-14T16:56:10.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 -- 172.21.15.29:0/1994573948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159f00 msgr2=0x7f23300b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 --2- 172.21.15.29:0/1994573948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159f00 0x7f23300b0410 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f23200097a0 tx=0x7f232002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 -- 172.21.15.29:0/1994573948 shutdown_connections 2023-10-14T16:56:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 --2- 172.21.15.29:0/1994573948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159f00 0x7f23300b0410 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 -- 172.21.15.29:0/1994573948 >> 172.21.15.29:0/1994573948 conn(0x7f23300cb410 msgr2=0x7f23300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 -- 172.21.15.29:0/1994573948 shutdown_connections 2023-10-14T16:56:10.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.247+0000 7f2338494640 1 -- 172.21.15.29:0/1994573948 wait complete. 2023-10-14T16:56:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.249+0000 7f2338494640 1 Processor -- start 2023-10-14T16:56:10.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.249+0000 7f2338494640 1 -- start start 2023-10-14T16:56:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2330159f00 0x7f2330154a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f233015d880 0x7f23301f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233015a2e0 con 0x7f2330159b20 2023-10-14T16:56:10.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f233015d200 con 0x7f2330159f00 2023-10-14T16:56:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2338494640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23301f3920 con 0x7f233015d880 2023-10-14T16:56:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2336230640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2336230640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45340/0 (socket says 172.21.15.29:45340) 2023-10-14T16:56:10.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.250+0000 7f2336230640 1 -- 172.21.15.29:0/2810413808 learned_addr learned my addr 172.21.15.29:0/2810413808 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 -- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f233015d880 msgr2=0x7f23301f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f233015d880 0x7f23301f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 -- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2330159f00 msgr2=0x7f2330154a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2330159f00 0x7f2330154a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 -- 172.21.15.29:0/2810413808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2320009400 con 0x7f2330159b20 2023-10-14T16:56:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2336230640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f232c00b3f0 tx=0x7f232c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011830 con 0x7f2330159b20 2023-10-14T16:56:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f232c011990 con 0x7f2330159b20 2023-10-14T16:56:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f232c011d40 con 0x7f2330159b20 2023-10-14T16:56:10.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.251+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23301f3c00 con 0x7f2330159b20 2023-10-14T16:56:10.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.252+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f233015a4f0 con 0x7f2330159b20 2023-10-14T16:56:10.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.253+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f233015a6e0 con 0x7f2330159b20 2023-10-14T16:56:10.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.254+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 27) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f232c01a400 con 0x7f2330159b20 2023-10-14T16:56:10.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.254+0000 7f2326ffd640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 0x7f231406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.255+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f232c089480 con 0x7f2330159b20 2023-10-14T16:56:10.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.255+0000 7f2335a2f640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 0x7f231406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.255+0000 7f2335a2f640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 0x7f231406d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2320036000 tx=0x7f2320030000 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.260+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f232c0543e0 con 0x7f2330159b20 2023-10-14T16:56:10.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.306+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f232c0547b0 con 0x7f2330159b20 2023-10-14T16:56:10.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.507+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f233004e160 con 0x7f2330159b20 2023-10-14T16:56:10.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.508+0000 7f2326ffd640 1 -- 172.21.15.29:0/2810413808 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+4913 (secure 0 0 0) 0x7f232c075630 con 0x7f2330159b20 2023-10-14T16:56:10.509 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:10.510 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":24,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:54:14.976595+0000","modified":"2023-10-14T16:54:18.073045+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":5236},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5236":{"gid":5236,"name":"d","rank":0,"incarnation":20,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/2940163923","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2940163923},{"type":"v1","addr":"172.21.15.186:6835","nonce":2940163923}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[7],"metadata_pool":6,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":3}]} 2023-10-14T16:56:10.510 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 24 2023-10-14T16:56:10.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 msgr2=0x7f231406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 0x7f231406d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2320036000 tx=0x7f2320030000 comp rx=0 tx=0).stop 2023-10-14T16:56:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 msgr2=0x7f233015be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f232c00b3f0 tx=0x7f232c00b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 shutdown_connections 2023-10-14T16:56:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f231406acf0 0x7f231406d1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f233015d880 0x7f23301f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2330159f00 0x7f2330154a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 --2- 172.21.15.29:0/2810413808 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2330159b20 0x7f233015be70 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.511+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 >> 172.21.15.29:0/2810413808 conn(0x7f23300cb410 msgr2=0x7f23300ae1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:10.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.512+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 shutdown_connections 2023-10-14T16:56:10.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.512+0000 7f2338494640 1 -- 172.21.15.29:0/2810413808 wait complete. 2023-10-14T16:56:10.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:10.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.752+0000 7f2fe759e640 1 Processor -- start 2023-10-14T16:56:10.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.753+0000 7f2fe759e640 1 -- start start 2023-10-14T16:56:10.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe8161430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2fe80b1d60 0x7f2fe80b4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fe80b4ad0 0x7f2fe806bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80b4f00 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fe80b5c40 con 0x7f2fe80b1d60 2023-10-14T16:56:10.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.754+0000 7f2fe759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80cdb50 con 0x7f2fe815ce10 2023-10-14T16:56:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fe80b4ad0 0x7f2fe806bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fe80b4ad0 0x7f2fe806bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45356/0 (socket says 172.21.15.29:45356) 2023-10-14T16:56:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 -- 172.21.15.29:0/2270321822 learned_addr learned my addr 172.21.15.29:0/2270321822 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fddd9b640 1 --2- 172.21.15.29:0/2270321822 >> v2:172.21.15.186:3300/0 conn(0x7f2fe80b1d60 0x7f2fe80b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe659c640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe8161430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 -- 172.21.15.29:0/2270321822 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 msgr2=0x7f2fe8161430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:10.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe8161430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 -- 172.21.15.29:0/2270321822 >> v2:172.21.15.186:3300/0 conn(0x7f2fe80b1d60 msgr2=0x7f2fe80b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:10.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 --2- 172.21.15.29:0/2270321822 >> v2:172.21.15.186:3300/0 conn(0x7f2fe80b1d60 0x7f2fe80b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.755+0000 7f2fe6d9d640 1 -- 172.21.15.29:0/2270321822 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe8161970 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.757+0000 7f2fe6d9d640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe806bc40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2fe806b250 tx=0x7f2fe002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8a6c32736146a84 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.757+0000 7f2fe5d9b640 1 -- 172.21.15.29:0/2270321822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe002ca70 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.757+0000 7f2fe5d9b640 1 -- 172.21.15.29:0/2270321822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fe002cbd0 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.757+0000 7f2fe5d9b640 1 -- 172.21.15.29:0/2270321822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fe00343f0 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 -- 172.21.15.29:0/2270321822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b4ad0 msgr2=0x7f2fe806bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe806bc40 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f2fe806b250 tx=0x7f2fe002bed0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 -- 172.21.15.29:0/2270321822 shutdown_connections 2023-10-14T16:56:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe806bc40 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 --2- 172.21.15.29:0/2270321822 >> v2:172.21.15.186:3300/0 conn(0x7f2fe80b1d60 0x7f2fe80b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 --2- 172.21.15.29:0/2270321822 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe8161430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 -- 172.21.15.29:0/2270321822 >> 172.21.15.29:0/2270321822 conn(0x7f2fe80cf590 msgr2=0x7f2fe80cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 -- 172.21.15.29:0/2270321822 shutdown_connections 2023-10-14T16:56:10.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.758+0000 7f2fe759e640 1 -- 172.21.15.29:0/2270321822 wait complete. 2023-10-14T16:56:10.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.760+0000 7f2fe759e640 1 Processor -- start 2023-10-14T16:56:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.760+0000 7f2fe759e640 1 -- start start 2023-10-14T16:56:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe80ca630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe815b900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80b13d0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe80c9be0 con 0x7f2fe80b4ad0 2023-10-14T16:56:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fe815e570 con 0x7f2fe815ce10 2023-10-14T16:56:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45372/0 (socket says 172.21.15.29:45372) 2023-10-14T16:56:10.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.761+0000 7f2fe659c640 1 -- 172.21.15.29:0/3241599987 learned_addr learned my addr 172.21.15.29:0/3241599987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fddd9b640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe80ca630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe6d9d640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe815b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 -- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 msgr2=0x7f2fe815b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe815b900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 -- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe80b4ad0 msgr2=0x7f2fe80ca630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe80ca630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 -- 172.21.15.29:0/3241599987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fe0009b30 con 0x7f2fe80b1d60 2023-10-14T16:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe6d9d640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe659c640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800d800 tx=0x7f2fd800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800b6c0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fd800b820 con 0x7f2fe80b1d60 2023-10-14T16:56:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fd800bbb0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.762+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fe815e850 con 0x7f2fe80b1d60 2023-10-14T16:56:10.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.763+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fe80ca110 con 0x7f2fe80b1d60 2023-10-14T16:56:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.764+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fe80ca2d0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.765+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2fd800bd10 con 0x7f2fe80b1d60 2023-10-14T16:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.765+0000 7f2fdf7fe640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 0x7f2fc806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.766+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f2fd8089f30 con 0x7f2fe80b1d60 2023-10-14T16:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.766+0000 7f2fddd9b640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 0x7f2fc806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:10.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.767+0000 7f2fddd9b640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 0x7f2fc806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006fd0 tx=0x7f2fd0008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:10.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.771+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fd8054bc0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.969+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fe8072150 con 0x7f2fe80b1d60 2023-10-14T16:56:10.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.970+0000 7f2fdf7fe640 1 -- 172.21.15.29:0/3241599987 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+17723 (secure 0 0 0) 0x7f2fd80581e0 con 0x7f2fe80b1d60 2023-10-14T16:56:10.972 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:10.972 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":70,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:10.072798+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:54:12.966365+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_data","create_time":"2023-10-14T16:54:14.007924+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"7.6","mappings":[{"from":0,"to":5}]},{"pgid":"7.38","mappings":[{"from":0,"to":4}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2114169573":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2811289807":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/1660561604":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2419772294":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6832/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/3005757508":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:6832/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/470082549":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/128331176":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2125394339":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/3085701795":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/4095875950":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2968229624":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/185522519":"2023-10-15T16:54:30.427919+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[{"pool":7,"snaps":[{"begin":2,"length":4}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:10.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 msgr2=0x7f2fc806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 0x7f2fc806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006fd0 tx=0x7f2fd0008040 comp rx=0 tx=0).stop 2023-10-14T16:56:10.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 msgr2=0x7f2fe815de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7f2fd800d800 tx=0x7f2fd800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 shutdown_connections 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f2fc806af00 0x7f2fc806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fe815ce10 0x7f2fe815b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fe80b4ad0 0x7f2fe80ca630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 --2- 172.21.15.29:0/3241599987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fe80b1d60 0x7f2fe815de70 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.973+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 >> 172.21.15.29:0/3241599987 conn(0x7f2fe80cf590 msgr2=0x7f2fe81609b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.974+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 shutdown_connections 2023-10-14T16:56:10.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:10.974+0000 7f2fe759e640 1 -- 172.21.15.29:0/3241599987 wait complete. 2023-10-14T16:56:10.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:56:11.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.216+0000 7fdbc4c59640 1 Processor -- start 2023-10-14T16:56:11.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.216+0000 7fdbc4c59640 1 -- start start 2023-10-14T16:56:11.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.217+0000 7fdbc4c59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbc4c59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 0x7fdbc00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbc4c59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdbc0070f20 0x7fdbc00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbc4c59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc00cdec0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbc4c59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbc0160a90 con 0x7fdbc0070f20 2023-10-14T16:56:11.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbc4c59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00cd900 con 0x7fdbc00c9230 2023-10-14T16:56:11.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbbf7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbbf7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40338/0 (socket says 172.21.15.29:40338) 2023-10-14T16:56:11.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/2243501759 learned_addr learned my addr 172.21.15.29:0/2243501759 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:11.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2243501759 >> v2:172.21.15.186:3300/0 conn(0x7fdbc0070f20 0x7fdbc00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.218+0000 7fdbb6ffd640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 0x7fdbc00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/2243501759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 msgr2=0x7fdbc00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:11.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 0x7fdbc00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/2243501759 >> v2:172.21.15.186:3300/0 conn(0x7fdbc0070f20 msgr2=0x7fdbc00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2243501759 >> v2:172.21.15.186:3300/0 conn(0x7fdbc0070f20 0x7fdbc00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbbf7fe640 1 -- 172.21.15.29:0/2243501759 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc00cdbe0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.219+0000 7fdbb6ffd640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 0x7fdbc00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.220+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00c8c80 tx=0x7fdbb002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c792e9f45e32d9bb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbbeffd640 1 -- 172.21.15.29:0/2243501759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0031880 con 0x7fdbc015c9f0 2023-10-14T16:56:11.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbbeffd640 1 -- 172.21.15.29:0/2243501759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb00319e0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbbeffd640 1 -- 172.21.15.29:0/2243501759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb0031d70 con 0x7fdbc015c9f0 2023-10-14T16:56:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2243501759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 msgr2=0x7fdbc00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fdbc00c8c80 tx=0x7fdbb002b920 comp rx=0 tx=0).stop 2023-10-14T16:56:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2243501759 shutdown_connections 2023-10-14T16:56:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2243501759 >> v2:172.21.15.186:3300/0 conn(0x7fdbc0070f20 0x7fdbc00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc00c9230 0x7fdbc00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2243501759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc00c8cf0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.221+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2243501759 >> 172.21.15.29:0/2243501759 conn(0x7fdbc00ce6a0 msgr2=0x7fdbc00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.222+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2243501759 shutdown_connections 2023-10-14T16:56:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.222+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2243501759 wait complete. 2023-10-14T16:56:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.223+0000 7fdbc4c59640 1 Processor -- start 2023-10-14T16:56:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.223+0000 7fdbc4c59640 1 -- start start 2023-10-14T16:56:11.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc0070f20 0x7fdbc00c3c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbc00c9230 0x7fdbc0158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00c4170 con 0x7fdbc015c9f0 2023-10-14T16:56:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00c4410 con 0x7fdbc00c9230 2023-10-14T16:56:11.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.224+0000 7fdbc4c59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbc00c46f0 con 0x7fdbc0070f20 2023-10-14T16:56:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40354/0 (socket says 172.21.15.29:40354) 2023-10-14T16:56:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 -- 172.21.15.29:0/2453844187 learned_addr learned my addr 172.21.15.29:0/2453844187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 -- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc0070f20 msgr2=0x7fdbc00c3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc0070f20 0x7fdbc00c3c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 -- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbc00c9230 msgr2=0x7fdbc0158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbc00c9230 0x7fdbc0158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 -- 172.21.15.29:0/2453844187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbb00094d0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbffff640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800d5d0 tx=0x7fdbb800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.225+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb80125f0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.226+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbb8012770 con 0x7fdbc015c9f0 2023-10-14T16:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.226+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbb8012b20 con 0x7fdbc015c9f0 2023-10-14T16:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.226+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbc00c49d0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.226+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbc015ce70 con 0x7fdbc015c9f0 2023-10-14T16:56:11.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.227+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbc015d090 con 0x7fdbc015c9f0 2023-10-14T16:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.228+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdbb8012c80 con 0x7fdbc015c9f0 2023-10-14T16:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.229+0000 7fdbbcff9640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 0x7fdba006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.229+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6527+0+0 (secure 0 0 0) 0x7fdbb8089fc0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.229+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 0x7fdba006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.230+0000 7fdbbf7fe640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 0x7fdba006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0036000 tx=0x7fdbb0030000 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.233+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbb8054bd0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.467+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fdbc004e9b0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.469 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:56:11.468+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:56:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.472+0000 7fdbbcff9640 1 -- 172.21.15.29:0/2453844187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v25) v1 ==== 161+0+0 (secure 0 0 0) 0x7fdbb80581f0 con 0x7fdbc015c9f0 2023-10-14T16:56:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:56:11.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.475+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 msgr2=0x7fdba006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.475+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 0x7fdba006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdbb0036000 tx=0x7fdbb0030000 comp rx=0 tx=0).stop 2023-10-14T16:56:11.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.475+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 msgr2=0x7fdbc015f180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.475+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fdbb800d5d0 tx=0x7fdbb800da90 comp rx=0 tx=0).stop 2023-10-14T16:56:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 shutdown_connections 2023-10-14T16:56:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdba006af00 0x7fdba006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbc015c9f0 0x7fdbc015f180 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbc00c9230 0x7fdbc0158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 --2- 172.21.15.29:0/2453844187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbc0070f20 0x7fdbc00c3c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 >> 172.21.15.29:0/2453844187 conn(0x7fdbc00ce6a0 msgr2=0x7fdbc00c96f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:11.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 shutdown_connections 2023-10-14T16:56:11.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.476+0000 7fdbc4c59640 1 -- 172.21.15.29:0/2453844187 wait complete. 2023-10-14T16:56:11.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:56:11.600 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:56:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.725+0000 7fe6d4c90640 1 Processor -- start 2023-10-14T16:56:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.725+0000 7fe6d4c90640 1 -- start start 2023-10-14T16:56:11.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6d4c90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6d0070b60 0x7fe6d015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6d4c90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe6d006c100 0x7fe6d006c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6d4c90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6d015aef0 con 0x7fe6d0159950 2023-10-14T16:56:11.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6d4c90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d015e490 con 0x7fe6d006c100 2023-10-14T16:56:11.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6d4c90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6d015d910 con 0x7fe6d0070b60 2023-10-14T16:56:11.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6cf7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe6d0159950 0x7fe6d0070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51244/0 (socket says 172.21.15.29:51244) 2023-10-14T16:56:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6cf7fe640 1 -- 172.21.15.29:0/902783079 learned_addr learned my addr 172.21.15.29:0/902783079 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:11.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6cffff640 1 --2- 172.21.15.29:0/902783079 >> v2:172.21.15.29:3300/0 conn(0x7fe6d006c100 0x7fe6d006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.727+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/902783079 >> v2:172.21.15.186:3300/0 conn(0x7fe6d0070b60 0x7fe6d015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6ceffd640 1 -- 172.21.15.29:0/902783079 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179325481 0 0) 0x7fe6c0001110 con 0x7fe6d0159950 2023-10-14T16:56:11.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6ceffd640 1 -- 172.21.15.29:0/902783079 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6bc003520 con 0x7fe6d0159950 2023-10-14T16:56:11.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6cffff640 1 -- 172.21.15.29:0/902783079 >> v2:172.21.15.186:3300/0 conn(0x7fe6d0070b60 msgr2=0x7fe6d015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:11.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6cffff640 1 --2- 172.21.15.29:0/902783079 >> v2:172.21.15.186:3300/0 conn(0x7fe6d0070b60 0x7fe6d015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6cffff640 1 -- 172.21.15.29:0/902783079 >> v1:172.21.15.29:6789/0 conn(0x7fe6d0159950 legacy=0x7fe6d0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.728+0000 7fe6cffff640 1 -- 172.21.15.29:0/902783079 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d015dbf0 con 0x7fe6d006c100 2023-10-14T16:56:11.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.729+0000 7fe6cffff640 1 --2- 172.21.15.29:0/902783079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d006c100 0x7fe6d006c510 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00716a0 tx=0x7fe6c802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2175d258f6b2bcb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.729+0000 7fe6ceffd640 1 -- 172.21.15.29:0/902783079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c802cbe0 con 0x7fe6d006c100 2023-10-14T16:56:11.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.729+0000 7fe6ceffd640 1 -- 172.21.15.29:0/902783079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6c802cd40 con 0x7fe6d006c100 2023-10-14T16:56:11.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.729+0000 7fe6ceffd640 1 -- 172.21.15.29:0/902783079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8034660 con 0x7fe6d006c100 2023-10-14T16:56:11.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 -- 172.21.15.29:0/902783079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d006c100 msgr2=0x7fe6d006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/902783079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d006c100 0x7fe6d006c510 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7fe6d00716a0 tx=0x7fe6c802be30 comp rx=0 tx=0).stop 2023-10-14T16:56:11.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 -- 172.21.15.29:0/902783079 shutdown_connections 2023-10-14T16:56:11.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/902783079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d006c100 0x7fe6d006c510 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/902783079 >> v2:172.21.15.186:3300/0 conn(0x7fe6d0070b60 0x7fe6d015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 -- 172.21.15.29:0/902783079 >> 172.21.15.29:0/902783079 conn(0x7fe6d00ce6a0 msgr2=0x7fe6d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:11.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 -- 172.21.15.29:0/902783079 shutdown_connections 2023-10-14T16:56:11.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.730+0000 7fe6d4c90640 1 -- 172.21.15.29:0/902783079 wait complete. 2023-10-14T16:56:11.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.732+0000 7fe6d4c90640 1 Processor -- start 2023-10-14T16:56:11.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.732+0000 7fe6d4c90640 1 -- start start 2023-10-14T16:56:11.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 0x7fe6d01558e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d0159950 0x7fe6d01f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d014fc60 con 0x7fe6d0070b60 2023-10-14T16:56:11.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d0154ed0 con 0x7fe6d0159950 2023-10-14T16:56:11.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6d4c90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6d00c4610 con 0x7fe6d006c100 2023-10-14T16:56:11.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40372/0 (socket says 172.21.15.29:40372) 2023-10-14T16:56:11.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 -- 172.21.15.29:0/1602854314 learned_addr learned my addr 172.21.15.29:0/1602854314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:11.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 0x7fe6d01558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 -- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 msgr2=0x7fe6d01558e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 0x7fe6d01558e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6cffff640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d0159950 0x7fe6d01f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 -- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d0159950 msgr2=0x7fe6d01f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:11.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.733+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d0159950 0x7fe6d01f5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:11.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6c7fff640 1 -- 172.21.15.29:0/1602854314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c8009a70 con 0x7fe6d0070b60 2023-10-14T16:56:11.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 0x7fe6d01558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:11.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6c7fff640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800d800 tx=0x7fe6b800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800b6c0 con 0x7fe6d0070b60 2023-10-14T16:56:11.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800b820 con 0x7fe6d0070b60 2023-10-14T16:56:11.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800bbb0 con 0x7fe6d0070b60 2023-10-14T16:56:11.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6d01545a0 con 0x7fe6d0070b60 2023-10-14T16:56:11.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.734+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6d0154b60 con 0x7fe6d0070b60 2023-10-14T16:56:11.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.735+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d0155080 con 0x7fe6d0070b60 2023-10-14T16:56:11.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.736+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 28) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe6b800bd10 con 0x7fe6d0070b60 2023-10-14T16:56:11.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.738+0000 7fe6ccff9640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 0x7fe6b006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:11.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.738+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6585+0+0 (secure 0 0 0) 0x7fe6b8089fe0 con 0x7fe6d0070b60 2023-10-14T16:56:11.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.738+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 0x7fe6b006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:11.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.739+0000 7fe6cf7fe640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 0x7fe6b006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0001370 tx=0x7fe6c0009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:11.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.742+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe6d0155080 con 0x7fe6d0070b60 2023-10-14T16:56:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:11.980+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6d0155040 con 0x7fe6d0070b60 2023-10-14T16:56:12.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.485+0000 7fe6ccff9640 1 -- 172.21.15.29:0/1602854314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v26) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6b8054c30 con 0x7fe6d0070b60 2023-10-14T16:56:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.489+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 msgr2=0x7fe6b006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.489+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 0x7fe6b006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe6c0001370 tx=0x7fe6c0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:12.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.489+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 msgr2=0x7fe6d00ad0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.489+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7fe6b800d800 tx=0x7fe6b800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 shutdown_connections 2023-10-14T16:56:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fe6b006af00 0x7fe6b006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6d0159950 0x7fe6d01f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6d0070b60 0x7fe6d00ad0a0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 --2- 172.21.15.29:0/1602854314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6d006c100 0x7fe6d01558e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.490+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 >> 172.21.15.29:0/1602854314 conn(0x7fe6d00ce6a0 msgr2=0x7fe6d00af8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:12.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.491+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 shutdown_connections 2023-10-14T16:56:12.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.491+0000 7fe6d4c90640 1 -- 172.21.15.29:0/1602854314 wait complete. 2023-10-14T16:56:12.517 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:56:12.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.737+0000 7fb45d064640 1 Processor -- start 2023-10-14T16:56:12.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.737+0000 7fb45d064640 1 -- start start 2023-10-14T16:56:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb45d064640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4580cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb45d064640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4580ccfb0 0x7fb4580cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb45d064640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb45815b9d0 con 0x7fb45806c040 2023-10-14T16:56:12.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb45d064640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb45815bcb0 con 0x7fb4580ccfb0 2023-10-14T16:56:12.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb45d064640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb45815bf90 con 0x7fb4580ad970 2023-10-14T16:56:12.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb457fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb45806c040 0x7fb45806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51258/0 (socket says 172.21.15.29:51258) 2023-10-14T16:56:12.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb457fff640 1 -- 172.21.15.29:0/3721914771 learned_addr learned my addr 172.21.15.29:0/3721914771 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:12.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb4577fe640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4580cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.739+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722453333 0 0) 0x7fb45815b9d0 con 0x7fb45806c040 2023-10-14T16:56:12.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb440003520 con 0x7fb45806c040 2023-10-14T16:56:12.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb45c863640 1 --2- 172.21.15.29:0/3721914771 >> v2:172.21.15.186:3300/0 conn(0x7fb4580ccfb0 0x7fb4580cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3451136463 0 0) 0x7fb440003520 con 0x7fb45806c040 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 msgr2=0x7fb4580cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4580cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 >> v2:172.21.15.186:3300/0 conn(0x7fb4580ccfb0 msgr2=0x7fb4580cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 --2- 172.21.15.29:0/3721914771 >> v2:172.21.15.186:3300/0 conn(0x7fb4580ccfb0 0x7fb4580cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4580cc270 con 0x7fb45806c040 2023-10-14T16:56:12.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb448001670 con 0x7fb45806c040 2023-10-14T16:56:12.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 >> v1:172.21.15.29:6789/0 conn(0x7fb45806c040 legacy=0x7fb45806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb440003970 0x7fb440003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb440004d00 0x7fb440005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45815bcb0 con 0x7fb440003970 2023-10-14T16:56:12.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb45815bf90 con 0x7fb440004d00 2023-10-14T16:56:12.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.740+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44000b980 con 0x7fb4580ad970 2023-10-14T16:56:12.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb440003970 0x7fb440003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb45c863640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb440004d00 0x7fb440005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb4577fe640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 -- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 msgr2=0x7fb4400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:12.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4580ad970 0x7fb4400047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 -- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb440004d00 msgr2=0x7fb440005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:12.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb440004d00 0x7fb440005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.741+0000 7fb457fff640 1 -- 172.21.15.29:0/3721914771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb440003520 con 0x7fb440003970 2023-10-14T16:56:12.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.742+0000 7fb457fff640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb440003970 0x7fb440003d90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb44800c110 tx=0x7fb4400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32be4b720b9ddc4d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:12.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb448005ad0 con 0x7fb440003970 2023-10-14T16:56:12.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb448005c50 con 0x7fb440003970 2023-10-14T16:56:12.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb456ffd640 1 -- 172.21.15.29:0/3721914771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb44802c7e0 con 0x7fb440003970 2023-10-14T16:56:12.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 -- 172.21.15.29:0/3721914771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb440003970 msgr2=0x7fb440003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 --2- 172.21.15.29:0/3721914771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb440003970 0x7fb440003d90 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7fb44800c110 tx=0x7fb4400057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45c863640 1 -- 172.21.15.29:0/3721914771 reap_dead start 2023-10-14T16:56:12.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 -- 172.21.15.29:0/3721914771 shutdown_connections 2023-10-14T16:56:12.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 -- 172.21.15.29:0/3721914771 >> 172.21.15.29:0/3721914771 conn(0x7fb4580ce6a0 msgr2=0x7fb4580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:12.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 -- 172.21.15.29:0/3721914771 shutdown_connections 2023-10-14T16:56:12.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.743+0000 7fb45d064640 1 -- 172.21.15.29:0/3721914771 wait complete. 2023-10-14T16:56:12.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.745+0000 7fb45d064640 1 Processor -- start 2023-10-14T16:56:12.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.746+0000 7fb45d064640 1 -- start start 2023-10-14T16:56:12.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.746+0000 7fb45d064640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 0x7fb458158a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.746+0000 7fb45d064640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb458153ee0 0x7fb458158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45d064640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45d064640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb458154b00 con 0x7fb4580c6150 2023-10-14T16:56:12.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45d064640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb458154900 con 0x7fb458153b00 2023-10-14T16:56:12.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45d064640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4580c6e70 con 0x7fb458153ee0 2023-10-14T16:56:12.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40390/0 (socket says 172.21.15.29:40390) 2023-10-14T16:56:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 -- 172.21.15.29:0/981598344 learned_addr learned my addr 172.21.15.29:0/981598344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb457fff640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 0x7fb458158a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 -- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb458153ee0 msgr2=0x7fb458158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb458153ee0 0x7fb458158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 -- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 msgr2=0x7fb458158a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:12.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 0x7fb458158a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:12.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb45c863640 1 -- 172.21.15.29:0/981598344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4580c7150 con 0x7fb4580c6150 2023-10-14T16:56:12.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.747+0000 7fb457fff640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 0x7fb458158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb45c863640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb4580cd390 tx=0x7fb45000f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb45000e040 con 0x7fb4580c6150 2023-10-14T16:56:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb45000f040 con 0x7fb4580c6150 2023-10-14T16:56:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb450015430 con 0x7fb4580c6150 2023-10-14T16:56:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4580c7430 con 0x7fb4580c6150 2023-10-14T16:56:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.748+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4580c6b10 con 0x7fb4580c6150 2023-10-14T16:56:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.749+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb45806b330 con 0x7fb4580c6150 2023-10-14T16:56:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.750+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb450015590 con 0x7fb4580c6150 2023-10-14T16:56:12.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.751+0000 7fb454ff9640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 0x7fb44406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.751+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6585+0+0 (secure 0 0 0) 0x7fb45008a300 con 0x7fb4580c6150 2023-10-14T16:56:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.752+0000 7fb457fff640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 0x7fb44406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.752+0000 7fb457fff640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 0x7fb44406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb448034000 tx=0x7fb448009030 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:12.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.756+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb450054d80 con 0x7fb4580c6150 2023-10-14T16:56:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:12.952+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb458154c40 con 0x7fb4580c6150 2023-10-14T16:56:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.489+0000 7fb454ff9640 1 -- 172.21.15.29:0/981598344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v72) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb458154c40 con 0x7fb4580c6150 2023-10-14T16:56:13.490 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:56:13.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.494+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 msgr2=0x7fb44406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.494+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 0x7fb44406d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fb448034000 tx=0x7fb448009030 comp rx=0 tx=0).stop 2023-10-14T16:56:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.494+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 msgr2=0x7fb4580c65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.494+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7fb4580cd390 tx=0x7fb45000f580 comp rx=0 tx=0).stop 2023-10-14T16:56:13.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 shutdown_connections 2023-10-14T16:56:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb44406af90 0x7fb44406d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4580c6150 0x7fb4580c65d0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb458153ee0 0x7fb458158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 --2- 172.21.15.29:0/981598344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb458153b00 0x7fb458158a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.495+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 >> 172.21.15.29:0/981598344 conn(0x7fb4580ce6a0 msgr2=0x7fb4580ac9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.496+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 shutdown_connections 2023-10-14T16:56:13.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.496+0000 7fb45d064640 1 -- 172.21.15.29:0/981598344 wait complete. 2023-10-14T16:56:13.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:56:13.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.750+0000 7f23b4ad1640 1 Processor -- start 2023-10-14T16:56:13.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.750+0000 7f23b4ad1640 1 -- start start 2023-10-14T16:56:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.751+0000 7f23b4ad1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f23b00cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23b4ad1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b015b9d0 con 0x7f23b006c040 2023-10-14T16:56:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23b4ad1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b015bcb0 con 0x7f23b00ad970 2023-10-14T16:56:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23b4ad1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015b7d0 con 0x7f23b015b320 2023-10-14T16:56:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23af7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f23b006c040 0x7f23b006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51268/0 (socket says 172.21.15.29:51268) 2023-10-14T16:56:13.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23af7fe640 1 -- 172.21.15.29:0/3182028317 learned_addr learned my addr 172.21.15.29:0/3182028317 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23affff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f23b00cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.752+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146219554 0 0) 0x7f23b015b9d0 con 0x7f23b006c040 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c003520 con 0x7f23b006c040 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 682054657 0 0) 0x7f239c003520 con 0x7f23b006c040 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 msgr2=0x7f23b00cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f23b00cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 >> v1:172.21.15.186:6789/0 conn(0x7f23b00ad970 legacy=0x7f23b00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b015bf90 con 0x7f23b006c040 2023-10-14T16:56:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f23a0001670 con 0x7f23b006c040 2023-10-14T16:56:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 >> v1:172.21.15.29:6789/0 conn(0x7f23b006c040 legacy=0x7f23b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f239c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c003f20 0x7f239c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c004d00 0x7f239c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b015b7d0 con 0x7f239c003f20 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239c00ba60 con 0x7f239c004d00 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239c00bc20 con 0x7f23b015b320 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.753+0000 7f23a7fff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c003f20 0x7f239c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23af7fe640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f239c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23affff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c004d00 0x7f239c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23a7fff640 1 -- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 msgr2=0x7f239c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:13.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23a7fff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b015b320 0x7f239c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23a7fff640 1 -- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c004d00 msgr2=0x7f239c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23a7fff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c004d00 0x7f239c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.754+0000 7f23a7fff640 1 -- 172.21.15.29:0/3182028317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c003520 con 0x7f239c003f20 2023-10-14T16:56:13.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.755+0000 7f23a7fff640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c003f20 0x7f239c0047c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2398001720 tx=0x7f239c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ebcbb9cb6915cb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.755+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23980052d0 con 0x7f239c003f20 2023-10-14T16:56:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f239802e050 con 0x7f239c003f20 2023-10-14T16:56:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23aeffd640 1 -- 172.21.15.29:0/3182028317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2398003830 con 0x7f239c003f20 2023-10-14T16:56:13.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 -- 172.21.15.29:0/3182028317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c003f20 msgr2=0x7f239c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/3182028317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239c003f20 0x7f239c0047c0 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f2398001720 tx=0x7f239c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23affff640 1 -- 172.21.15.29:0/3182028317 reap_dead start 2023-10-14T16:56:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 -- 172.21.15.29:0/3182028317 shutdown_connections 2023-10-14T16:56:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 -- 172.21.15.29:0/3182028317 >> 172.21.15.29:0/3182028317 conn(0x7f23b00ce6a0 msgr2=0x7f23b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:13.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 -- 172.21.15.29:0/3182028317 shutdown_connections 2023-10-14T16:56:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.756+0000 7f23b4ad1640 1 -- 172.21.15.29:0/3182028317 wait complete. 2023-10-14T16:56:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.758+0000 7f23b4ad1640 1 Processor -- start 2023-10-14T16:56:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.758+0000 7f23b4ad1640 1 -- start start 2023-10-14T16:56:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c003f20 0x7f23b01e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b00c7110 0x7f23b01529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b0152f00 con 0x7f23b01e5be0 2023-10-14T16:56:13.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b00c7590 con 0x7f239c003f20 2023-10-14T16:56:13.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.759+0000 7f23b4ad1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b01e4b40 con 0x7f23b00c7110 2023-10-14T16:56:13.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40406/0 (socket says 172.21.15.29:40406) 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 -- 172.21.15.29:0/931338661 learned_addr learned my addr 172.21.15.29:0/931338661 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23af7fe640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c003f20 0x7f23b01e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 -- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b00c7110 msgr2=0x7f23b01529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b00c7110 0x7f23b01529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 -- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c003f20 msgr2=0x7f23b01e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c003f20 0x7f23b01e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 -- 172.21.15.29:0/931338661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b01e4e20 con 0x7f23b01e5be0 2023-10-14T16:56:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.760+0000 7f23affff640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f23b0071b50 tx=0x7f23a80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:13.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.761+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a800e070 con 0x7f23b01e5be0 2023-10-14T16:56:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.761+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23a8011e10 con 0x7f23b01e5be0 2023-10-14T16:56:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.761+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23a8003990 con 0x7f23b01e5be0 2023-10-14T16:56:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.761+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b01e5100 con 0x7f23b01e5be0 2023-10-14T16:56:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.761+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b01e56c0 con 0x7f23b01e5be0 2023-10-14T16:56:13.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.762+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b006b330 con 0x7f23b01e5be0 2023-10-14T16:56:13.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.763+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f23a8003af0 con 0x7f23b01e5be0 2023-10-14T16:56:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.764+0000 7f23acff9640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 0x7f239006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.764+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6147+0+0 (secure 0 0 0) 0x7f23a808a1b0 con 0x7f23b01e5be0 2023-10-14T16:56:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.764+0000 7f23af7fe640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 0x7f239006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.765+0000 7f23af7fe640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 0x7f239006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23a0001c90 tx=0x7f23a0009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:13.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.768+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23a8054de0 con 0x7f23b01e5be0 2023-10-14T16:56:13.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:13.965+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f23b004e9b0 con 0x7f23b01e5be0 2023-10-14T16:56:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.493+0000 7f23acff9640 1 -- 172.21.15.29:0/931338661 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v73) v1 ==== 169+0+0 (secure 0 0 0) 0x7f23a8058400 con 0x7f23b01e5be0 2023-10-14T16:56:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:56:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 msgr2=0x7f239006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 0x7f239006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f23a0001c90 tx=0x7f23a0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:14.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 msgr2=0x7f23b00b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f23b0071b50 tx=0x7f23a80115a0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 shutdown_connections 2023-10-14T16:56:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f239006af90 0x7f239006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b01e5be0 0x7f23b00b22e0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23b00c7110 0x7f23b01529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 --2- 172.21.15.29:0/931338661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239c003f20 0x7f23b01e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.497+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 >> 172.21.15.29:0/931338661 conn(0x7f23b00ce6a0 msgr2=0x7f23b015c1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.498+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 shutdown_connections 2023-10-14T16:56:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.498+0000 7f23b4ad1640 1 -- 172.21.15.29:0/931338661 wait complete. 2023-10-14T16:56:14.527 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T16:56:14.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.726+0000 7fce3f59e640 1 Processor -- start 2023-10-14T16:56:14.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.726+0000 7fce3f59e640 1 -- start start 2023-10-14T16:56:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.727+0000 7fce3f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce400b3ec0 0x7fce400cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce4006be30 con 0x7fce400b0ea0 2023-10-14T16:56:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce400ada10 con 0x7fce400b43c0 2023-10-14T16:56:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4006ca00 con 0x7fce400b3ec0 2023-10-14T16:56:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce400b0ea0 0x7fce4015a2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51274/0 (socket says 172.21.15.29:51274) 2023-10-14T16:56:14.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3dd9b640 1 -- 172.21.15.29:0/2245652396 learned_addr learned my addr 172.21.15.29:0/2245652396 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.728+0000 7fce3e59c640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce400b3ec0 0x7fce400cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180085724 0 0) 0x7fce4006be30 con 0x7fce400b0ea0 2023-10-14T16:56:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce28003520 con 0x7fce400b0ea0 2023-10-14T16:56:14.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1526590956 0 0) 0x7fce28003520 con 0x7fce400b0ea0 2023-10-14T16:56:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce400b3ec0 msgr2=0x7fce400cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce400b3ec0 0x7fce400cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 >> v1:172.21.15.186:6789/0 conn(0x7fce400b43c0 legacy=0x7fce4015aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4006c060 con 0x7fce400b0ea0 2023-10-14T16:56:14.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fce34001670 con 0x7fce400b0ea0 2023-10-14T16:56:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 >> v1:172.21.15.29:6789/0 conn(0x7fce400b0ea0 legacy=0x7fce4015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280039a0 0x7fce28003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce28005180 0x7fce2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce4006ca00 con 0x7fce280047c0 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800c050 con 0x7fce28005180 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce2800c210 con 0x7fce280039a0 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.729+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce280047c0 0x7fce28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3e59c640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280039a0 0x7fce28003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3ed9d640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce28005180 0x7fce2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3dd9b640 1 -- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280039a0 msgr2=0x7fce28003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280039a0 0x7fce28003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3dd9b640 1 -- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce28005180 msgr2=0x7fce2800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce28005180 0x7fce2800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3dd9b640 1 -- 172.21.15.29:0/2245652396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28003520 con 0x7fce280047c0 2023-10-14T16:56:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.730+0000 7fce3ed9d640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce28005180 0x7fce2800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.731+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce280047c0 0x7fce28004c40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c210 tx=0x7fce280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c0b8689e14253d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.731+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34005ad0 con 0x7fce280047c0 2023-10-14T16:56:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.731+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce34005c50 con 0x7fce280047c0 2023-10-14T16:56:14.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.731+0000 7fce3d59a640 1 -- 172.21.15.29:0/2245652396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3402c8d0 con 0x7fce280047c0 2023-10-14T16:56:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 -- 172.21.15.29:0/2245652396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce280047c0 msgr2=0x7fce28004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 --2- 172.21.15.29:0/2245652396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce280047c0 0x7fce28004c40 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7fce3400c210 tx=0x7fce280057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3ed9d640 1 -- 172.21.15.29:0/2245652396 reap_dead start 2023-10-14T16:56:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 -- 172.21.15.29:0/2245652396 shutdown_connections 2023-10-14T16:56:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 -- 172.21.15.29:0/2245652396 >> 172.21.15.29:0/2245652396 conn(0x7fce400b0170 msgr2=0x7fce400b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 -- 172.21.15.29:0/2245652396 shutdown_connections 2023-10-14T16:56:14.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.732+0000 7fce3f59e640 1 -- 172.21.15.29:0/2245652396 wait complete. 2023-10-14T16:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.734+0000 7fce3f59e640 1 Processor -- start 2023-10-14T16:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.734+0000 7fce3f59e640 1 -- start start 2023-10-14T16:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 0x7fce40159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 0x7fce400c84f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce400b2400 0x7fce400b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400c4e30 con 0x7fce400c8070 2023-10-14T16:56:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce400b2dc0 con 0x7fce400b2400 2023-10-14T16:56:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.735+0000 7fce3f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce4015d170 con 0x7fce280047c0 2023-10-14T16:56:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 0x7fce400c84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3e59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 0x7fce40159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3e59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 0x7fce40159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:36756/0 (socket says 172.21.15.29:36756) 2023-10-14T16:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3e59c640 1 -- 172.21.15.29:0/1897405998 learned_addr learned my addr 172.21.15.29:0/1897405998 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3ed9d640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce400b2400 0x7fce400b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 -- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 msgr2=0x7fce40159780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 0x7fce40159780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 -- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce400b2400 msgr2=0x7fce400b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce400b2400 0x7fce400b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.736+0000 7fce3dd9b640 1 -- 172.21.15.29:0/1897405998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce4015c560 con 0x7fce400c8070 2023-10-14T16:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce3dd9b640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 0x7fce400c84f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fce34001eb0 tx=0x7fce34001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:14.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce34039070 con 0x7fce400c8070 2023-10-14T16:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3402cce0 con 0x7fce400c8070 2023-10-14T16:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3403c040 con 0x7fce400c8070 2023-10-14T16:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce4015c840 con 0x7fce400c8070 2023-10-14T16:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.737+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce4015ce00 con 0x7fce400c8070 2023-10-14T16:56:14.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.739+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce4004e9b0 con 0x7fce400c8070 2023-10-14T16:56:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.740+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fce34032070 con 0x7fce400c8070 2023-10-14T16:56:14.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.740+0000 7fce2effd640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 0x7fce2006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.741+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fce340aa990 con 0x7fce400c8070 2023-10-14T16:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.741+0000 7fce3e59c640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 0x7fce2006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.742+0000 7fce3e59c640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 0x7fce2006d4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce30003b90 tx=0x7fce3000b040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:14.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.745+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce34075b00 con 0x7fce400c8070 2023-10-14T16:56:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.943+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fce4006abd0 con 0x7fce400c8070 2023-10-14T16:56:14.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.950+0000 7fce2effd640 1 -- 172.21.15.29:0/1897405998 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v19) v1 ==== 115+0+0 (secure 0 0 0) 0x7fce34079120 con 0x7fce400c8070 2023-10-14T16:56:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 msgr2=0x7fce2006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 0x7fce2006d4d0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fce30003b90 tx=0x7fce3000b040 comp rx=0 tx=0).stop 2023-10-14T16:56:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 msgr2=0x7fce400c84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 0x7fce400c84f0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7fce34001eb0 tx=0x7fce34001a00 comp rx=0 tx=0).stop 2023-10-14T16:56:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 shutdown_connections 2023-10-14T16:56:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce2006b010 0x7fce2006d4d0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce400b2400 0x7fce400b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce400c8070 0x7fce400c84f0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 --2- 172.21.15.29:0/1897405998 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce280047c0 0x7fce40159780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 >> 172.21.15.29:0/1897405998 conn(0x7fce400b0170 msgr2=0x7fce4015a9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 shutdown_connections 2023-10-14T16:56:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:14.953+0000 7fce3f59e640 1 -- 172.21.15.29:0/1897405998 wait complete. 2023-10-14T16:56:14.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T16:56:15.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.203+0000 7fce4ffff640 1 Processor -- start 2023-10-14T16:56:15.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.204+0000 7fce4ffff640 1 -- start start 2023-10-14T16:56:15.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.205+0000 7fce4ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fce5006c040 0x7fce5006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.205+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500cda80 0x7fce5015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.205+0000 7fce4ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5006c960 con 0x7fce500ad970 2023-10-14T16:56:15.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce5015b9d0 con 0x7fce5006c040 2023-10-14T16:56:15.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce5015bcb0 con 0x7fce500cda80 2023-10-14T16:56:15.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4e7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce500ad970 0x7fce5015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51282/0 (socket says 172.21.15.29:51282) 2023-10-14T16:56:15.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4e7fc640 1 -- 172.21.15.29:0/2834284160 learned_addr learned my addr 172.21.15.29:0/2834284160 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4effd640 1 --2- 172.21.15.29:0/2834284160 >> v2:172.21.15.186:3300/0 conn(0x7fce5006c040 0x7fce5006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.206+0000 7fce4f7fe640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500cda80 0x7fce5015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486723340 0 0) 0x7fce5006c960 con 0x7fce500ad970 2023-10-14T16:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce3c003520 con 0x7fce500ad970 2023-10-14T16:56:15.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4effd640 1 -- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500cda80 msgr2=0x7fce5015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4effd640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500cda80 0x7fce5015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4effd640 1 -- 172.21.15.29:0/2834284160 >> v1:172.21.15.29:6789/0 conn(0x7fce500ad970 legacy=0x7fce5015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 981872489 0 0) 0x7fce3c003520 con 0x7fce500ad970 2023-10-14T16:56:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.207+0000 7fce4effd640 1 -- 172.21.15.29:0/2834284160 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5015bf90 con 0x7fce5006c040 2023-10-14T16:56:15.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.208+0000 7fce4effd640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce5006c420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d90 tx=0x7fce4002bd70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ed01ffd0889700b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.209+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce38001670 con 0x7fce5006c040 2023-10-14T16:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.209+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4002c8a0 con 0x7fce5006c040 2023-10-14T16:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.209+0000 7fce4dffb640 1 -- 172.21.15.29:0/2834284160 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4002cc30 con 0x7fce5006c040 2023-10-14T16:56:15.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.209+0000 7fce4ffff640 1 -- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 msgr2=0x7fce5006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.209+0000 7fce4ffff640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce5006c420 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fce40009d90 tx=0x7fce4002bd70 comp rx=0 tx=0).stop 2023-10-14T16:56:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 -- 172.21.15.29:0/2834284160 shutdown_connections 2023-10-14T16:56:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500cda80 0x7fce5015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 --2- 172.21.15.29:0/2834284160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce5006c420 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 -- 172.21.15.29:0/2834284160 >> 172.21.15.29:0/2834284160 conn(0x7fce500ce6a0 msgr2=0x7fce500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 -- 172.21.15.29:0/2834284160 shutdown_connections 2023-10-14T16:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.210+0000 7fce4ffff640 1 -- 172.21.15.29:0/2834284160 wait complete. 2023-10-14T16:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.211+0000 7fce4ffff640 1 Processor -- start 2023-10-14T16:56:15.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.212+0000 7fce4ffff640 1 -- start start 2023-10-14T16:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce500c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 0x7fce5014cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce5014d520 con 0x7fce500cda80 2023-10-14T16:56:15.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce500c9410 con 0x7fce5006c040 2023-10-14T16:56:15.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce500c8e90 con 0x7fce500ad970 2023-10-14T16:56:15.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40426/0 (socket says 172.21.15.29:40426) 2023-10-14T16:56:15.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 -- 172.21.15.29:0/1234801552 learned_addr learned my addr 172.21.15.29:0/1234801552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:15.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4effd640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce500c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4e7fc640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 0x7fce5014cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 -- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 msgr2=0x7fce5014cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 0x7fce5014cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 -- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 msgr2=0x7fce500c8130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce500c8130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4f7fe640 1 -- 172.21.15.29:0/1234801552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce40009a70 con 0x7fce500cda80 2023-10-14T16:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.213+0000 7fce4e7fc640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 0x7fce5014cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce4f7fe640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fce4800e960 tx=0x7fce4800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce48009690 con 0x7fce500cda80 2023-10-14T16:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce4800c980 con 0x7fce500cda80 2023-10-14T16:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce4800cd30 con 0x7fce500cda80 2023-10-14T16:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce500c7970 con 0x7fce500cda80 2023-10-14T16:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.214+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce500c7c10 con 0x7fce500cda80 2023-10-14T16:56:15.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.215+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce5006b330 con 0x7fce500cda80 2023-10-14T16:56:15.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.216+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fce48013470 con 0x7fce500cda80 2023-10-14T16:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.217+0000 7fce477fe640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 0x7fce3006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.217+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fce480899d0 con 0x7fce500cda80 2023-10-14T16:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.218+0000 7fce4effd640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 0x7fce3006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.218+0000 7fce4effd640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 0x7fce3006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce40036000 tx=0x7fce40030000 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.222+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce480548f0 con 0x7fce500cda80 2023-10-14T16:56:15.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.421+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fce500cc250 con 0x7fce500cda80 2023-10-14T16:56:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.427+0000 7fce477fe640 1 -- 172.21.15.29:0/1234801552 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v20) v1 ==== 120+0+0 (secure 0 0 0) 0x7fce500cc250 con 0x7fce500cda80 2023-10-14T16:56:15.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 msgr2=0x7fce3006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 0x7fce3006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fce40036000 tx=0x7fce40030000 comp rx=0 tx=0).stop 2023-10-14T16:56:15.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 msgr2=0x7fce5014d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fce4800e960 tx=0x7fce4800ee20 comp rx=0 tx=0).stop 2023-10-14T16:56:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 shutdown_connections 2023-10-14T16:56:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fce3006af90 0x7fce3006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce500cda80 0x7fce5014d690 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce500ad970 0x7fce5014cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 --2- 172.21.15.29:0/1234801552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce5006c040 0x7fce500c8130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 >> 172.21.15.29:0/1234801552 conn(0x7fce500ce6a0 msgr2=0x7fce50070280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 shutdown_connections 2023-10-14T16:56:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.430+0000 7fce4ffff640 1 -- 172.21.15.29:0/1234801552 wait complete. 2023-10-14T16:56:15.449 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T16:56:15.524 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:56:15.534 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:56:15.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.664+0000 7f86c953a640 1 Processor -- start 2023-10-14T16:56:15.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.665+0000 7f86c953a640 1 -- start start 2023-10-14T16:56:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.666+0000 7f86c953a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86c40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c953a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40d18c0 con 0x7f86c40b6cd0 2023-10-14T16:56:15.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c953a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c40d3980 con 0x7f86c40cfaf0 2023-10-14T16:56:15.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c953a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40d3c60 con 0x7f86c40cf710 2023-10-14T16:56:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c37fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f86c40b6cd0 0x7f86c40d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51298/0 (socket says 172.21.15.29:51298) 2023-10-14T16:56:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c37fe640 1 -- 172.21.15.29:0/3088914518 learned_addr learned my addr 172.21.15.29:0/3088914518 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.667+0000 7f86c2ffd640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86c40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345877835 0 0) 0x7f86c40d18c0 con 0x7f86c40b6cd0 2023-10-14T16:56:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86b0003520 con 0x7f86c40b6cd0 2023-10-14T16:56:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3428143285 0 0) 0x7f86b0003520 con 0x7f86c40b6cd0 2023-10-14T16:56:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 msgr2=0x7f86c40b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:15.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86c40b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 >> v1:172.21.15.186:6789/0 conn(0x7f86c40cfaf0 legacy=0x7f86c40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c40ce630 con 0x7f86c40b6cd0 2023-10-14T16:56:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f86bc0019f0 con 0x7f86c40b6cd0 2023-10-14T16:56:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 >> v1:172.21.15.29:6789/0 conn(0x7f86c40b6cd0 legacy=0x7f86c40d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86b00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b0003f20 0x7f86b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40d3c60 con 0x7f86b0004d00 2023-10-14T16:56:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86b000bb10 con 0x7f86b0003f20 2023-10-14T16:56:15.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.668+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86b000bcd0 con 0x7f86c40cf710 2023-10-14T16:56:15.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c2ffd640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86b00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86bbfff640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b0003f20 0x7f86b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 -- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 msgr2=0x7f86b00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:15.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cf710 0x7f86b00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 -- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b0003f20 msgr2=0x7f86b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:15.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86b0003f20 0x7f86b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.669+0000 7f86c37fe640 1 -- 172.21.15.29:0/3088914518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86b0003520 con 0x7f86b0004d00 2023-10-14T16:56:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.670+0000 7f86c37fe640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86b0005180 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00a800 tx=0x7f86b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76f0d9c1a13ad5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc0072d0 con 0x7f86b0004d00 2023-10-14T16:56:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc02f070 con 0x7f86b0004d00 2023-10-14T16:56:15.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c27fc640 1 -- 172.21.15.29:0/3088914518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86bc0076b0 con 0x7f86b0004d00 2023-10-14T16:56:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 -- 172.21.15.29:0/3088914518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 msgr2=0x7f86b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 --2- 172.21.15.29:0/3088914518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86b0005180 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f86bc00a800 tx=0x7f86b00057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c37fe640 1 -- 172.21.15.29:0/3088914518 reap_dead start 2023-10-14T16:56:15.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 -- 172.21.15.29:0/3088914518 shutdown_connections 2023-10-14T16:56:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 -- 172.21.15.29:0/3088914518 >> 172.21.15.29:0/3088914518 conn(0x7f86c40d2cc0 msgr2=0x7f86c40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 -- 172.21.15.29:0/3088914518 shutdown_connections 2023-10-14T16:56:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.671+0000 7f86c953a640 1 -- 172.21.15.29:0/3088914518 wait complete. 2023-10-14T16:56:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.673+0000 7f86c953a640 1 Processor -- start 2023-10-14T16:56:15.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.673+0000 7f86c953a640 1 -- start start 2023-10-14T16:56:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 0x7f86c41531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 0x7f86c406b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40afdd0 con 0x7f86b0004d00 2023-10-14T16:56:15.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40cc420 con 0x7f86c40b1080 2023-10-14T16:56:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c953a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c40b0bf0 con 0x7f86c40cb770 2023-10-14T16:56:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.674+0000 7f86c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40454/0 (socket says 172.21.15.29:40454) 2023-10-14T16:56:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 -- 172.21.15.29:0/59767169 learned_addr learned my addr 172.21.15.29:0/59767169 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:15.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86bbfff640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 0x7f86c41531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c37fe640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 0x7f86c406b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 -- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 msgr2=0x7f86c41531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 0x7f86c41531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 -- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 msgr2=0x7f86c406b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 0x7f86c406b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 -- 172.21.15.29:0/59767169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c40affe0 con 0x7f86b0004d00 2023-10-14T16:56:15.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86bbfff640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 0x7f86c41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:15.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c37fe640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 0x7f86c406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:15.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.675+0000 7f86c2ffd640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f86b40115e0 tx=0x7f86b4011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.676+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b4005da0 con 0x7f86b0004d00 2023-10-14T16:56:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.676+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86b4004020 con 0x7f86b0004d00 2023-10-14T16:56:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.676+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86b40043d0 con 0x7f86b0004d00 2023-10-14T16:56:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.676+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c40b02c0 con 0x7f86b0004d00 2023-10-14T16:56:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.676+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c40b0880 con 0x7f86b0004d00 2023-10-14T16:56:15.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.677+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c404e160 con 0x7f86b0004d00 2023-10-14T16:56:15.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.678+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f86b4003610 con 0x7f86b0004d00 2023-10-14T16:56:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.679+0000 7f86bb7fe640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 0x7f86a406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.679+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f86b4089b20 con 0x7f86b0004d00 2023-10-14T16:56:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.679+0000 7f86bbfff640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 0x7f86a406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.680+0000 7f86bbfff640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 0x7f86a406d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86c406c2a0 tx=0x7f86ac009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:15.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.684+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f86b4054a40 con 0x7f86b0004d00 2023-10-14T16:56:15.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.882+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f86c40b4240 con 0x7f86b0004d00 2023-10-14T16:56:15.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.889+0000 7f86bb7fe640 1 -- 172.21.15.29:0/59767169 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7f86b40220a0 con 0x7f86b0004d00 2023-10-14T16:56:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 msgr2=0x7f86a406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 0x7f86a406d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f86c406c2a0 tx=0x7f86ac009040 comp rx=0 tx=0).stop 2023-10-14T16:56:15.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 msgr2=0x7f86c40af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f86b40115e0 tx=0x7f86b4011aa0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 shutdown_connections 2023-10-14T16:56:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f86a406afc0 0x7f86a406d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86c40b1080 0x7f86c406b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86c40cb770 0x7f86c41531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 --2- 172.21.15.29:0/59767169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86b0004d00 0x7f86c40af890 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.892+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 >> 172.21.15.29:0/59767169 conn(0x7f86c40d2cc0 msgr2=0x7f86c40b7d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.893+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 shutdown_connections 2023-10-14T16:56:15.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:15.893+0000 7f86c953a640 1 -- 172.21.15.29:0/59767169 wait complete. 2023-10-14T16:56:15.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule' 2023-10-14T16:56:16.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.175+0000 7f3b65e59640 1 Processor -- start 2023-10-14T16:56:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.176+0000 7f3b65e59640 1 -- start start 2023-10-14T16:56:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3b60070d70 0x7f3b6006bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b600712c0 0x7f3b6006c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b6006c9c0 con 0x7f3b60070d70 2023-10-14T16:56:16.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b600b4620 con 0x7f3b600712c0 2023-10-14T16:56:16.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.177+0000 7f3b65e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b600b4340 con 0x7f3b6015aad0 2023-10-14T16:56:16.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b64e57640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3b60070d70 0x7f3b6006bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b65658640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b600712c0 0x7f3b6006c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b64e57640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:36796/0 (socket says 172.21.15.29:36796) 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b64e57640 1 -- 172.21.15.29:0/3078888019 learned_addr learned my addr 172.21.15.29:0/3078888019 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 -- 172.21.15.29:0/3078888019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 msgr2=0x7f3b600cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/3078888019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 -- 172.21.15.29:0/3078888019 >> v2:172.21.15.186:3300/0 conn(0x7f3b600712c0 msgr2=0x7f3b6006c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/3078888019 >> v2:172.21.15.186:3300/0 conn(0x7f3b600712c0 0x7f3b6006c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.178+0000 7f3b5ffff640 1 -- 172.21.15.29:0/3078888019 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b6006f520 con 0x7f3b60070d70 2023-10-14T16:56:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.180+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/3078888019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b6006bf40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006f200 tx=0x7f3b5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1379f07d0782559 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.180+0000 7f3b5f7fe640 1 -- 172.21.15.29:0/3078888019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54031830 con 0x7f3b60070d70 2023-10-14T16:56:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.180+0000 7f3b5f7fe640 1 -- 172.21.15.29:0/3078888019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b54031990 con 0x7f3b60070d70 2023-10-14T16:56:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.180+0000 7f3b5f7fe640 1 -- 172.21.15.29:0/3078888019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b54031d20 con 0x7f3b60070d70 2023-10-14T16:56:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 -- 172.21.15.29:0/3078888019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 msgr2=0x7f3b6006bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 --2- 172.21.15.29:0/3078888019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b6006bf40 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f3b6006f200 tx=0x7f3b5402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 -- 172.21.15.29:0/3078888019 shutdown_connections 2023-10-14T16:56:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 --2- 172.21.15.29:0/3078888019 >> v2:172.21.15.186:3300/0 conn(0x7f3b600712c0 0x7f3b6006c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 --2- 172.21.15.29:0/3078888019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b6006bf40 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 --2- 172.21.15.29:0/3078888019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 -- 172.21.15.29:0/3078888019 >> 172.21.15.29:0/3078888019 conn(0x7f3b600b0a90 msgr2=0x7f3b600ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 -- 172.21.15.29:0/3078888019 shutdown_connections 2023-10-14T16:56:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.181+0000 7f3b65e59640 1 -- 172.21.15.29:0/3078888019 wait complete. 2023-10-14T16:56:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.183+0000 7f3b65e59640 1 Processor -- start 2023-10-14T16:56:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.183+0000 7f3b65e59640 1 -- start start 2023-10-14T16:56:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 0x7f3b60151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600c5980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b601f28d0 con 0x7f3b60070d70 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b6015da70 con 0x7f3b600712c0 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b65e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b601f2230 con 0x7f3b6015aad0 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b64e57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b64e57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40468/0 (socket says 172.21.15.29:40468) 2023-10-14T16:56:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b64e57640 1 -- 172.21.15.29:0/713193544 learned_addr learned my addr 172.21.15.29:0/713193544 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.184+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 0x7f3b60151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b65658640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600c5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 -- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 msgr2=0x7f3b600c5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600c5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 -- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 msgr2=0x7f3b60151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 0x7f3b60151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 -- 172.21.15.29:0/713193544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b54009400 con 0x7f3b60070d70 2023-10-14T16:56:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 0x7f3b60151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b65658640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b64e57640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000efc0 tx=0x7f3b5000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b50009b60 con 0x7f3b60070d70 2023-10-14T16:56:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b5000f040 con 0x7f3b60070d70 2023-10-14T16:56:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b500092e0 con 0x7f3b60070d70 2023-10-14T16:56:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b65e59640 1 -- 172.21.15.29:0/713193544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b600c6280 con 0x7f3b60070d70 2023-10-14T16:56:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.185+0000 7f3b65e59640 1 -- 172.21.15.29:0/713193544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b601f24f0 con 0x7f3b60070d70 2023-10-14T16:56:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.187+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b6004e9b0 con 0x7f3b60070d70 2023-10-14T16:56:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.188+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b50014480 con 0x7f3b60070d70 2023-10-14T16:56:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.189+0000 7f3b5d7fa640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 0x7f3b4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.189+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f3b50089f60 con 0x7f3b60070d70 2023-10-14T16:56:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.189+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 0x7f3b4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.190+0000 7f3b5ffff640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 0x7f3b4c06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b54036000 tx=0x7f3b54008030 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.194+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b50054e00 con 0x7f3b60070d70 2023-10-14T16:56:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.466+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]} v 0) v1 -- 0x7f3b600cf520 con 0x7f3b60070d70 2023-10-14T16:56:16.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.512+0000 7f3b5d7fa640 1 -- 172.21.15.29:0/713193544 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_multi_snap_schedule"]}]=0 v578) v1 ==== 154+0+0 (secure 0 0 0) 0x7f3b50058420 con 0x7f3b60070d70 2023-10-14T16:56:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 msgr2=0x7f3b4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 0x7f3b4c06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3b54036000 tx=0x7f3b54008030 comp rx=0 tx=0).stop 2023-10-14T16:56:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 msgr2=0x7f3b601f2cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f3b5000efc0 tx=0x7f3b5000c490 comp rx=0 tx=0).stop 2023-10-14T16:56:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 shutdown_connections 2023-10-14T16:56:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f3b4c06af90 0x7f3b4c06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b6015aad0 0x7f3b600c5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b600712c0 0x7f3b60151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 --2- 172.21.15.29:0/713193544 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b60070d70 0x7f3b601f2cf0 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 >> 172.21.15.29:0/713193544 conn(0x7f3b600b0a90 msgr2=0x7f3b600ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 shutdown_connections 2023-10-14T16:56:16.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.515+0000 7f3b46ffd640 1 -- 172.21.15.29:0/713193544 wait complete. 2023-10-14T16:56:16.537 INFO:tasks.cephfs_test_runner:Test exisitence of multiple scheduled snapshots ... ok 2023-10-14T16:56:16.538 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing non-existing snap schedules failure 2023-10-14T16:56:16.538 INFO:tasks.cephfs_test_runner:test_non_existent_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:56:16.539 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.756+0000 7fd95cb53640 1 Processor -- start 2023-10-14T16:56:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.756+0000 7fd95cb53640 1 -- start start 2023-10-14T16:56:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd95cb53640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95815ae80 0x7fd95815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd95cb53640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9580ce390 0x7fd958070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd95cb53640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd95815ed60 con 0x7fd95815aaa0 2023-10-14T16:56:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd95cb53640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd95806b810 con 0x7fd9580ce390 2023-10-14T16:56:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd95cb53640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580707e0 con 0x7fd95815ae80 2023-10-14T16:56:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.758+0000 7fd956d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9580ce390 0x7fd958070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9580ce390 0x7fd958070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40482/0 (socket says 172.21.15.29:40482) 2023-10-14T16:56:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 -- 172.21.15.29:0/1978308858 learned_addr learned my addr 172.21.15.29:0/1978308858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd955d9b640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95815ae80 0x7fd95815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 -- 172.21.15.29:0/1978308858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95815ae80 msgr2=0x7fd95815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95815ae80 0x7fd95815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 -- 172.21.15.29:0/1978308858 >> v1:172.21.15.186:6789/0 conn(0x7fd95815aaa0 legacy=0x7fd9580cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.759+0000 7fd956d9d640 1 -- 172.21.15.29:0/1978308858 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95815ef90 con 0x7fd9580ce390 2023-10-14T16:56:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.760+0000 7fd956d9d640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9580ce390 0x7fd958070f50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd950009e50 tx=0x7fd95002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeddb200fe2eb7d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.760+0000 7fd95559a640 1 -- 172.21.15.29:0/1978308858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95002ca70 con 0x7fd9580ce390 2023-10-14T16:56:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95559a640 1 -- 172.21.15.29:0/1978308858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd95002cbd0 con 0x7fd9580ce390 2023-10-14T16:56:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95559a640 1 -- 172.21.15.29:0/1978308858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9500343f0 con 0x7fd9580ce390 2023-10-14T16:56:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 -- 172.21.15.29:0/1978308858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9580ce390 msgr2=0x7fd958070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9580ce390 0x7fd958070f50 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd950009e50 tx=0x7fd95002bed0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 -- 172.21.15.29:0/1978308858 shutdown_connections 2023-10-14T16:56:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9580ce390 0x7fd958070f50 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1978308858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95815ae80 0x7fd95815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 -- 172.21.15.29:0/1978308858 >> 172.21.15.29:0/1978308858 conn(0x7fd9580cb410 msgr2=0x7fd9580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 -- 172.21.15.29:0/1978308858 shutdown_connections 2023-10-14T16:56:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.761+0000 7fd95cb53640 1 -- 172.21.15.29:0/1978308858 wait complete. 2023-10-14T16:56:16.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.763+0000 7fd95cb53640 1 Processor -- start 2023-10-14T16:56:16.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.763+0000 7fd95cb53640 1 -- start start 2023-10-14T16:56:16.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9580ce390 0x7fd958155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95815ae80 0x7fd958157380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9581578c0 con 0x7fd95815aaa0 2023-10-14T16:56:16.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd958156740 con 0x7fd95815ae80 2023-10-14T16:56:16.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.764+0000 7fd95cb53640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9580ca870 con 0x7fd9580ce390 2023-10-14T16:56:16.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40486/0 (socket says 172.21.15.29:40486) 2023-10-14T16:56:16.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 -- 172.21.15.29:0/1586388470 learned_addr learned my addr 172.21.15.29:0/1586388470 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd95659c640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9580ce390 0x7fd958155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 -- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9580ce390 msgr2=0x7fd958155ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9580ce390 0x7fd958155ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 -- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95815ae80 msgr2=0x7fd958157380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd956d9d640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95815ae80 0x7fd958157380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95815ae80 0x7fd958157380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.765+0000 7fd955d9b640 1 -- 172.21.15.29:0/1586388470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd950009b30 con 0x7fd95815aaa0 2023-10-14T16:56:16.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd955d9b640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd94000d800 tx=0x7fd94000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000b6c0 con 0x7fd95815aaa0 2023-10-14T16:56:16.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd94000b820 con 0x7fd95815aaa0 2023-10-14T16:56:16.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd94000bbb0 con 0x7fd95815aaa0 2023-10-14T16:56:16.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd95815b380 con 0x7fd95815aaa0 2023-10-14T16:56:16.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.766+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd95815b940 con 0x7fd95815aaa0 2023-10-14T16:56:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.767+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd95804e160 con 0x7fd95815aaa0 2023-10-14T16:56:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.768+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd94000bd10 con 0x7fd95815aaa0 2023-10-14T16:56:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.769+0000 7fd94effd640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 0x7fd93806d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.769+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fd940089d80 con 0x7fd95815aaa0 2023-10-14T16:56:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.769+0000 7fd95659c640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 0x7fd93806d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.770+0000 7fd95659c640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 0x7fd93806d470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd948006fd0 tx=0x7fd948008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:16.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.773+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd940054c90 con 0x7fd95815aaa0 2023-10-14T16:56:16.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.978+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd95806c7a0 con 0x7fd95815aaa0 2023-10-14T16:56:16.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.980+0000 7fd94effd640 1 -- 172.21.15.29:0/1586388470 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v73) v1 ==== 74+0+14043 (secure 0 0 0) 0x7fd9400582b0 con 0x7fd95815aaa0 2023-10-14T16:56:16.981 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:16.981 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":73,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:14.489480+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6834/2940163923":"2023-10-15T16:56:11.469387+0000","172.21.15.29:0/2114169573":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2811289807":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/1660561604":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/2419772294":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6832/2060982901":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/3005757508":"2023-10-15T16:56:07.650936+0000","172.21.15.29:6833/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:6832/98172895":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/470082549":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/128331176":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2125394339":"2023-10-15T16:54:30.427919+0000","172.21.15.186:6835/2940163923":"2023-10-15T16:56:11.469387+0000","172.21.15.29:0/3085701795":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/4095875950":"2023-10-15T16:54:30.427919+0000","172.21.15.29:0/2968229624":"2023-10-15T16:56:07.650936+0000","172.21.15.29:0/185522519":"2023-10-15T16:54:30.427919+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.982+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 msgr2=0x7fd93806d470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.982+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 0x7fd93806d470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fd948006fd0 tx=0x7fd948008040 comp rx=0 tx=0).stop 2023-10-14T16:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.982+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 msgr2=0x7fd9580ca1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.982+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd94000d800 tx=0x7fd94000dcc0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 shutdown_connections 2023-10-14T16:56:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd93806afb0 0x7fd93806d470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95815ae80 0x7fd958157380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95815aaa0 0x7fd9580ca1e0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 --2- 172.21.15.29:0/1586388470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9580ce390 0x7fd958155ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 >> 172.21.15.29:0/1586388470 conn(0x7fd9580cb410 msgr2=0x7fd95806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:16.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 shutdown_connections 2023-10-14T16:56:16.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:16.983+0000 7fd95cb53640 1 -- 172.21.15.29:0/1586388470 wait complete. 2023-10-14T16:56:17.001 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:17.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.230+0000 7f43be5ef640 1 Processor -- start 2023-10-14T16:56:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.230+0000 7f43be5ef640 1 -- start start 2023-10-14T16:56:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43be5ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b80cd260 0x7f43b80ad130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43be5ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b80ce480 con 0x7f43b815cc00 2023-10-14T16:56:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43be5ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43b80cd7b0 con 0x7f43b80ad670 2023-10-14T16:56:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43be5ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43b80cda90 con 0x7f43b80cd260 2023-10-14T16:56:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bcdec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b80cd260 0x7f43b80ad130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bd5ed640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f43b815cc00 0x7f43b80ca410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51304/0 (socket says 172.21.15.29:51304) 2023-10-14T16:56:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bcdec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43b80cd260 0x7f43b80ad130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40498/0 (socket says 172.21.15.29:40498) 2023-10-14T16:56:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bd5ed640 1 -- 172.21.15.29:0/138663398 learned_addr learned my addr 172.21.15.29:0/138663398 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bcdec640 1 -- 172.21.15.29:0/138663398 >> v1:172.21.15.186:6789/0 conn(0x7f43b80ad670 legacy=0x7f43b80c8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bcdec640 1 -- 172.21.15.29:0/138663398 >> v1:172.21.15.29:6789/0 conn(0x7f43b815cc00 legacy=0x7f43b80ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.232+0000 7f43bcdec640 1 -- 172.21.15.29:0/138663398 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b80cdd70 con 0x7f43b80cd260 2023-10-14T16:56:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43bcdec640 1 --2- 172.21.15.29:0/138663398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b80ad130 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f43a80097d0 tx=0x7f43a802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae8896ed58b9e95f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43affff640 1 -- 172.21.15.29:0/138663398 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a802d040 con 0x7f43b80cd260 2023-10-14T16:56:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43affff640 1 -- 172.21.15.29:0/138663398 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a8031810 con 0x7f43b80cd260 2023-10-14T16:56:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43affff640 1 -- 172.21.15.29:0/138663398 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a8031ba0 con 0x7f43b80cd260 2023-10-14T16:56:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43be5ef640 1 -- 172.21.15.29:0/138663398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 msgr2=0x7f43b80ad130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.234+0000 7f43be5ef640 1 --2- 172.21.15.29:0/138663398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b80ad130 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7f43a80097d0 tx=0x7f43a802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.235+0000 7f43be5ef640 1 -- 172.21.15.29:0/138663398 shutdown_connections 2023-10-14T16:56:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.235+0000 7f43be5ef640 1 --2- 172.21.15.29:0/138663398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b80ad130 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.235+0000 7f43be5ef640 1 -- 172.21.15.29:0/138663398 >> 172.21.15.29:0/138663398 conn(0x7f43b80b0170 msgr2=0x7f43b80b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.235+0000 7f43be5ef640 1 -- 172.21.15.29:0/138663398 shutdown_connections 2023-10-14T16:56:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.235+0000 7f43be5ef640 1 -- 172.21.15.29:0/138663398 wait complete. 2023-10-14T16:56:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.237+0000 7f43be5ef640 1 Processor -- start 2023-10-14T16:56:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.237+0000 7f43be5ef640 1 -- start start 2023-10-14T16:56:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43b80ad670 0x7f43b815ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43b815cc00 0x7f43b80c6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b806c730 con 0x7f43b80cd260 2023-10-14T16:56:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b806c170 con 0x7f43b815cc00 2023-10-14T16:56:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43be5ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43b806c450 con 0x7f43b80ad670 2023-10-14T16:56:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40500/0 (socket says 172.21.15.29:40500) 2023-10-14T16:56:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 -- 172.21.15.29:0/332799271 learned_addr learned my addr 172.21.15.29:0/332799271 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:17.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 -- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43b80ad670 msgr2=0x7f43b815ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43b80ad670 0x7f43b815ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 -- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43b815cc00 msgr2=0x7f43b80c6ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43b815cc00 0x7f43b80c6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.238+0000 7f43bcdec640 1 -- 172.21.15.29:0/332799271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a8009400 con 0x7f43b80cd260 2023-10-14T16:56:17.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43bcdec640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a803ad60 con 0x7f43b80cd260 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43a80443f0 con 0x7f43b80cd260 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43a80447a0 con 0x7f43b80cd260 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b806a850 con 0x7f43b80cd260 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.239+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b815b220 con 0x7f43b80cd260 2023-10-14T16:56:17.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.240+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b806ad70 con 0x7f43b80cd260 2023-10-14T16:56:17.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.242+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43a804d450 con 0x7f43b80cd260 2023-10-14T16:56:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.242+0000 7f43adffb640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 0x7f439c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.243+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f43a80abce0 con 0x7f43b80cd260 2023-10-14T16:56:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.243+0000 7f43bd5ed640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 0x7f439c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.244+0000 7f43bd5ed640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 0x7f439c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43b4001550 tx=0x7f43b4007a40 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.247+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43b806ad70 con 0x7f43b80cd260 2023-10-14T16:56:17.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.442+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f43b806ad70 con 0x7f43b80cd260 2023-10-14T16:56:17.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.443+0000 7f43adffb640 1 -- 172.21.15.29:0/332799271 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v73) v1 ==== 93+0+10 (secure 0 0 0) 0x7f43b806ad70 con 0x7f43b80cd260 2023-10-14T16:56:17.444 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:17.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 msgr2=0x7f439c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 0x7f439c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f43b4001550 tx=0x7f43b4007a40 comp rx=0 tx=0).stop 2023-10-14T16:56:17.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 msgr2=0x7f43b806b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f43a8000c00 tx=0x7f43a80046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 shutdown_connections 2023-10-14T16:56:17.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f439c06af90 0x7f439c06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43b815cc00 0x7f43b80c6ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43b80cd260 0x7f43b806b050 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 --2- 172.21.15.29:0/332799271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43b80ad670 0x7f43b815ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 >> 172.21.15.29:0/332799271 conn(0x7f43b80b0170 msgr2=0x7f43b80adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:17.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 shutdown_connections 2023-10-14T16:56:17.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.446+0000 7f43be5ef640 1 -- 172.21.15.29:0/332799271 wait complete. 2023-10-14T16:56:17.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-14T16:56:17.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.693+0000 7f12c1ae7640 1 Processor -- start 2023-10-14T16:56:17.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.693+0000 7f12c1ae7640 1 -- start start 2023-10-14T16:56:17.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.694+0000 7f12c1ae7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 0x7f12bc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12c1ae7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f12bc15d880 0x7f12bc0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12c1ae7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc06f7f0 con 0x7f12bc159b20 2023-10-14T16:56:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12c1ae7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc06c5e0 con 0x7f12bc15d880 2023-10-14T16:56:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12c1ae7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc06c8c0 con 0x7f12bc159f00 2023-10-14T16:56:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f12bc159b20 0x7f12bc06bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51308/0 (socket says 172.21.15.29:51308) 2023-10-14T16:56:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bb7fe640 1 -- 172.21.15.29:0/1136854835 learned_addr learned my addr 172.21.15.29:0/1136854835 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:17.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bbfff640 1 --2- 172.21.15.29:0/1136854835 >> v2:172.21.15.29:3300/0 conn(0x7f12bc15d880 0x7f12bc0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12b3fff640 1 --2- 172.21.15.29:0/1136854835 >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 0x7f12bc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bbfff640 1 -- 172.21.15.29:0/1136854835 >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 msgr2=0x7f12bc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:17.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12baffd640 1 -- 172.21.15.29:0/1136854835 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867243700 0 0) 0x7f12bc06f7f0 con 0x7f12bc159b20 2023-10-14T16:56:17.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bbfff640 1 --2- 172.21.15.29:0/1136854835 >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 0x7f12bc0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bbfff640 1 -- 172.21.15.29:0/1136854835 >> v1:172.21.15.29:6789/0 conn(0x7f12bc159b20 legacy=0x7f12bc06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.695+0000 7f12bbfff640 1 -- 172.21.15.29:0/1136854835 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc06cba0 con 0x7f12bc15d880 2023-10-14T16:56:17.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.696+0000 7f12b3fff640 1 --2- 172.21.15.29:0/1136854835 >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 0x7f12bc0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12bbfff640 1 --2- 172.21.15.29:0/1136854835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc15d880 0x7f12bc0b0950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12bc15d770 tx=0x7f12b402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db7d0ee270eddc4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12baffd640 1 -- 172.21.15.29:0/1136854835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b402cc60 con 0x7f12bc15d880 2023-10-14T16:56:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12baffd640 1 -- 172.21.15.29:0/1136854835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b402cdc0 con 0x7f12bc15d880 2023-10-14T16:56:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12baffd640 1 -- 172.21.15.29:0/1136854835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12b4034660 con 0x7f12bc15d880 2023-10-14T16:56:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12c1ae7640 1 -- 172.21.15.29:0/1136854835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc15d880 msgr2=0x7f12bc0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.697+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/1136854835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc15d880 0x7f12bc0b0950 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f12bc15d770 tx=0x7f12b402beb0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 -- 172.21.15.29:0/1136854835 shutdown_connections 2023-10-14T16:56:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/1136854835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc15d880 0x7f12bc0b0950 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/1136854835 >> v2:172.21.15.186:3300/0 conn(0x7f12bc159f00 0x7f12bc0b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 -- 172.21.15.29:0/1136854835 >> 172.21.15.29:0/1136854835 conn(0x7f12bc0cb410 msgr2=0x7f12bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 -- 172.21.15.29:0/1136854835 shutdown_connections 2023-10-14T16:56:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.698+0000 7f12c1ae7640 1 -- 172.21.15.29:0/1136854835 wait complete. 2023-10-14T16:56:17.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.699+0000 7f12c1ae7640 1 Processor -- start 2023-10-14T16:56:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.699+0000 7f12c1ae7640 1 -- start start 2023-10-14T16:56:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 0x7f12bc15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12bc15d880 0x7f12bc1500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc15d200 con 0x7f12bc159b20 2023-10-14T16:56:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc070820 con 0x7f12bc159f00 2023-10-14T16:56:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.700+0000 7f12c1ae7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12bc070260 con 0x7f12bc15d880 2023-10-14T16:56:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40518/0 (socket says 172.21.15.29:40518) 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 -- 172.21.15.29:0/2814667795 learned_addr learned my addr 172.21.15.29:0/2814667795 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12b3fff640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 0x7f12bc15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bbfff640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12bc15d880 0x7f12bc1500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 -- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12bc15d880 msgr2=0x7f12bc1500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12bc15d880 0x7f12bc1500b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 -- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 msgr2=0x7f12bc15b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 0x7f12bc15b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12bb7fe640 1 -- 172.21.15.29:0/2814667795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12b4009a70 con 0x7f12bc159b20 2023-10-14T16:56:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.701+0000 7f12b3fff640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 0x7f12bc15b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12bb7fe640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c460 tx=0x7f12ac00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac0074e0 con 0x7f12bc159b20 2023-10-14T16:56:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ac007660 con 0x7f12bc159b20 2023-10-14T16:56:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12ac004a20 con 0x7f12bc159b20 2023-10-14T16:56:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc070540 con 0x7f12bc159b20 2023-10-14T16:56:17.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.702+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc1505f0 con 0x7f12bc159b20 2023-10-14T16:56:17.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.703+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc0ae3c0 con 0x7f12bc159b20 2023-10-14T16:56:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.704+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f12ac004b80 con 0x7f12bc159b20 2023-10-14T16:56:17.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.705+0000 7f12b8ff9640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 0x7f129c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.705+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f12ac088920 con 0x7f12bc159b20 2023-10-14T16:56:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.706+0000 7f12b3fff640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 0x7f129c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.706+0000 7f12b3fff640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 0x7f129c06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f12a4006fd0 tx=0x7f12a4008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:17.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.710+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12bc0ae3c0 con 0x7f12bc159b20 2023-10-14T16:56:17.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:17.977+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7f12bc0ae3c0 con 0x7f12bc159b20 2023-10-14T16:56:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.527+0000 7f12b8ff9640 1 -- 172.21.15.29:0/2814667795 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v580) v1 ==== 159+0+0 (secure 0 0 0) 0x7f12bc0ae3c0 con 0x7f12bc159b20 2023-10-14T16:56:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 msgr2=0x7f129c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 0x7f129c06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f12a4006fd0 tx=0x7f12a4008040 comp rx=0 tx=0).stop 2023-10-14T16:56:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 msgr2=0x7f12bc15cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f12ac00c460 tx=0x7f12ac00c920 comp rx=0 tx=0).stop 2023-10-14T16:56:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 shutdown_connections 2023-10-14T16:56:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f129c06af90 0x7f129c06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12bc15d880 0x7f12bc1500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12bc159f00 0x7f12bc15b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 --2- 172.21.15.29:0/2814667795 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f12bc159b20 0x7f12bc15cb90 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.530+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 >> 172.21.15.29:0/2814667795 conn(0x7f12bc0cb410 msgr2=0x7f12bc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.531+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 shutdown_connections 2023-10-14T16:56:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.531+0000 7f12c1ae7640 1 -- 172.21.15.29:0/2814667795 wait complete. 2023-10-14T16:56:18.553 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:56:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.778+0000 7f7777fff640 1 Processor -- start 2023-10-14T16:56:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.778+0000 7f7777fff640 1 -- start start 2023-10-14T16:56:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7777fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f777815dde0 0x7f77780709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7777fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7778070f20 0x7f77780ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7777fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7778160a90 con 0x7f777815c9f0 2023-10-14T16:56:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7777fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7778071350 con 0x7f7778070f20 2023-10-14T16:56:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7777fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77780cdec0 con 0x7f777815dde0 2023-10-14T16:56:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f77777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7778070f20 0x7f77780ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f77777fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7778070f20 0x7f77780ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40534/0 (socket says 172.21.15.29:40534) 2023-10-14T16:56:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f77777fe640 1 -- 172.21.15.29:0/120782083 learned_addr learned my addr 172.21.15.29:0/120782083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.780+0000 7f7776ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f777815c9f0 0x7f77780c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51310/0 (socket says 172.21.15.29:51310) 2023-10-14T16:56:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.781+0000 7f776ffff640 1 --2- 172.21.15.29:0/120782083 >> v2:172.21.15.186:3300/0 conn(0x7f777815dde0 0x7f77780709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.781+0000 7f77777fe640 1 -- 172.21.15.29:0/120782083 >> v2:172.21.15.186:3300/0 conn(0x7f777815dde0 msgr2=0x7f77780709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.781+0000 7f77777fe640 1 --2- 172.21.15.29:0/120782083 >> v2:172.21.15.186:3300/0 conn(0x7f777815dde0 0x7f77780709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.781+0000 7f77777fe640 1 -- 172.21.15.29:0/120782083 >> v1:172.21.15.29:6789/0 conn(0x7f777815c9f0 legacy=0x7f77780c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.781+0000 7f77777fe640 1 -- 172.21.15.29:0/120782083 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77780cd900 con 0x7f7778070f20 2023-10-14T16:56:18.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.782+0000 7f77777fe640 1 --2- 172.21.15.29:0/120782083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f77780ac890 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f77780b1260 tx=0x7f777002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25772fec51fa47f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.782+0000 7f77767fc640 1 -- 172.21.15.29:0/120782083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f777002cb10 con 0x7f7778070f20 2023-10-14T16:56:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.782+0000 7f77767fc640 1 -- 172.21.15.29:0/120782083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f777002cc70 con 0x7f7778070f20 2023-10-14T16:56:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.782+0000 7f77767fc640 1 -- 172.21.15.29:0/120782083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77700343f0 con 0x7f7778070f20 2023-10-14T16:56:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 -- 172.21.15.29:0/120782083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 msgr2=0x7f77780ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 --2- 172.21.15.29:0/120782083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f77780ac890 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f77780b1260 tx=0x7f777002bf70 comp rx=0 tx=0).stop 2023-10-14T16:56:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 -- 172.21.15.29:0/120782083 shutdown_connections 2023-10-14T16:56:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 --2- 172.21.15.29:0/120782083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f77780ac890 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 --2- 172.21.15.29:0/120782083 >> v2:172.21.15.186:3300/0 conn(0x7f777815dde0 0x7f77780709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 -- 172.21.15.29:0/120782083 >> 172.21.15.29:0/120782083 conn(0x7f77780ce6a0 msgr2=0x7f77780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 -- 172.21.15.29:0/120782083 shutdown_connections 2023-10-14T16:56:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.783+0000 7f7777fff640 1 -- 172.21.15.29:0/120782083 wait complete. 2023-10-14T16:56:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.785+0000 7f7777fff640 1 Processor -- start 2023-10-14T16:56:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.785+0000 7f7777fff640 1 -- start start 2023-10-14T16:56:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777815c9f0 0x7f7778150b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f777815dde0 0x7f7778159600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77780c8a40 con 0x7f7778070f20 2023-10-14T16:56:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f777806bab0 con 0x7f777815c9f0 2023-10-14T16:56:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7777fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77781510a0 con 0x7f777815dde0 2023-10-14T16:56:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7776ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40550/0 (socket says 172.21.15.29:40550) 2023-10-14T16:56:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.786+0000 7f7776ffd640 1 -- 172.21.15.29:0/3204462022 learned_addr learned my addr 172.21.15.29:0/3204462022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f77777fe640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f777815dde0 0x7f7778159600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f776ffff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777815c9f0 0x7f7778150b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 -- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f777815dde0 msgr2=0x7f7778159600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f777815dde0 0x7f7778159600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 -- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777815c9f0 msgr2=0x7f7778150b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777815c9f0 0x7f7778150b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 -- 172.21.15.29:0/3204462022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7770009b30 con 0x7f7778070f20 2023-10-14T16:56:18.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7776ffd640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f776800c400 tx=0x7f776800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77680074e0 con 0x7f7778070f20 2023-10-14T16:56:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.787+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7768007640 con 0x7f7778070f20 2023-10-14T16:56:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.788+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f777806bce0 con 0x7f7778070f20 2023-10-14T16:56:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.788+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7768004a00 con 0x7f7778070f20 2023-10-14T16:56:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.788+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f777806c2a0 con 0x7f7778070f20 2023-10-14T16:56:18.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.789+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f777804e9b0 con 0x7f7778070f20 2023-10-14T16:56:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.790+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7768004b60 con 0x7f7778070f20 2023-10-14T16:56:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.791+0000 7f7774ff9640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 0x7f775806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.791+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f77680890e0 con 0x7f7778070f20 2023-10-14T16:56:18.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.791+0000 7f776ffff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 0x7f775806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.792+0000 7f776ffff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 0x7f775806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77780cc770 tx=0x7f7760008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:18.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.795+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f77680541e0 con 0x7f7778070f20 2023-10-14T16:56:18.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.990+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7778160280 con 0x7f7778070f20 2023-10-14T16:56:18.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.996+0000 7f7774ff9640 1 -- 172.21.15.29:0/3204462022 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7768057800 con 0x7f7778070f20 2023-10-14T16:56:19.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 msgr2=0x7f775806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 0x7f775806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f77780cc770 tx=0x7f7760008040 comp rx=0 tx=0).stop 2023-10-14T16:56:19.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 msgr2=0x7f777806b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f776800c400 tx=0x7f776800c8c0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 shutdown_connections 2023-10-14T16:56:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f775806af90 0x7f775806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f777815dde0 0x7f7778159600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f777815c9f0 0x7f7778150b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 --2- 172.21.15.29:0/3204462022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7778070f20 0x7f777806b570 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 >> 172.21.15.29:0/3204462022 conn(0x7f77780ce6a0 msgr2=0x7f77780c9370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:19.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 shutdown_connections 2023-10-14T16:56:19.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:18.999+0000 7f7777fff640 1 -- 172.21.15.29:0/3204462022 wait complete. 2023-10-14T16:56:19.020 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:56:19.021 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:19.138 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:19.261 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:56:19.261 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:19.262 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:19.323 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:19.324 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:19.324 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:19.384 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:19.385 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:19.385 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:19.448 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:19.449 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:19.449 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:19.510 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:19.511 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:56:19.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.736+0000 7fb24d3a5640 1 Processor -- start 2023-10-14T16:56:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.737+0000 7fb24d3a5640 1 -- start start 2023-10-14T16:56:19.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.738+0000 7fb24d3a5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2480ad970 0x7fb2480cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.738+0000 7fb24d3a5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 0x7fb2480cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.738+0000 7fb24d3a5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24815b9d0 con 0x7fb24806c040 2023-10-14T16:56:19.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.738+0000 7fb24d3a5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24815bcb0 con 0x7fb2480ad970 2023-10-14T16:56:19.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.738+0000 7fb24d3a5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24815bf90 con 0x7fb2480ccfb0 2023-10-14T16:56:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2480ad970 0x7fb2480cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2480ad970 0x7fb2480cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40560/0 (socket says 172.21.15.29:40560) 2023-10-14T16:56:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 -- 172.21.15.29:0/3985594781 learned_addr learned my addr 172.21.15.29:0/3985594781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb24cba4640 1 --2- 172.21.15.29:0/3985594781 >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 0x7fb2480cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 -- 172.21.15.29:0/3985594781 >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 msgr2=0x7fb2480cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 --2- 172.21.15.29:0/3985594781 >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 0x7fb2480cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 -- 172.21.15.29:0/3985594781 >> v1:172.21.15.186:6789/0 conn(0x7fb24806c040 legacy=0x7fb24806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb23f7fe640 1 -- 172.21.15.29:0/3985594781 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2480cc270 con 0x7fb2480ad970 2023-10-14T16:56:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.739+0000 7fb24cba4640 1 --2- 172.21.15.29:0/3985594781 >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 0x7fb2480cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:19.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.740+0000 7fb23f7fe640 1 --2- 172.21.15.29:0/3985594781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ad970 0x7fb2480cca70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb230009870 tx=0x7fb23002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9cffb2123a387f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:19.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.741+0000 7fb2477fe640 1 -- 172.21.15.29:0/3985594781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230031870 con 0x7fb2480ad970 2023-10-14T16:56:19.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.741+0000 7fb2477fe640 1 -- 172.21.15.29:0/3985594781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2300319d0 con 0x7fb2480ad970 2023-10-14T16:56:19.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.741+0000 7fb2477fe640 1 -- 172.21.15.29:0/3985594781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb230031d60 con 0x7fb2480ad970 2023-10-14T16:56:19.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.741+0000 7fb24d3a5640 1 -- 172.21.15.29:0/3985594781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ad970 msgr2=0x7fb2480cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.741+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/3985594781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ad970 0x7fb2480cca70 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fb230009870 tx=0x7fb23002b910 comp rx=0 tx=0).stop 2023-10-14T16:56:19.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 -- 172.21.15.29:0/3985594781 shutdown_connections 2023-10-14T16:56:19.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/3985594781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ad970 0x7fb2480cca70 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/3985594781 >> v2:172.21.15.186:3300/0 conn(0x7fb2480ccfb0 0x7fb2480cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 -- 172.21.15.29:0/3985594781 >> 172.21.15.29:0/3985594781 conn(0x7fb2480ce6a0 msgr2=0x7fb2480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 -- 172.21.15.29:0/3985594781 shutdown_connections 2023-10-14T16:56:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.742+0000 7fb24d3a5640 1 -- 172.21.15.29:0/3985594781 wait complete. 2023-10-14T16:56:19.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.744+0000 7fb24d3a5640 1 Processor -- start 2023-10-14T16:56:19.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.744+0000 7fb24d3a5640 1 -- start start 2023-10-14T16:56:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24806c040 0x7fb2480c4110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480ad970 0x7fb2480b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480b3d10 con 0x7fb2480ccfb0 2023-10-14T16:56:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480b4130 con 0x7fb24806c040 2023-10-14T16:56:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24d3a5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2480b3090 con 0x7fb2480ad970 2023-10-14T16:56:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24cba4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24cba4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40564/0 (socket says 172.21.15.29:40564) 2023-10-14T16:56:19.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb24cba4640 1 -- 172.21.15.29:0/2796599381 learned_addr learned my addr 172.21.15.29:0/2796599381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.745+0000 7fb247fff640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24806c040 0x7fb2480c4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb23f7fe640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480ad970 0x7fb2480b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 -- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480ad970 msgr2=0x7fb2480b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480ad970 0x7fb2480b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 -- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24806c040 msgr2=0x7fb2480c4110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24806c040 0x7fb2480c4110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 -- 172.21.15.29:0/2796599381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2300094d0 con 0x7fb2480ccfb0 2023-10-14T16:56:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24cba4640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb24000c7c0 tx=0x7fb24000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240004230 con 0x7fb2480ccfb0 2023-10-14T16:56:19.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb240004390 con 0x7fb2480ccfb0 2023-10-14T16:56:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2480b3370 con 0x7fb2480ccfb0 2023-10-14T16:56:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb240007860 con 0x7fb2480ccfb0 2023-10-14T16:56:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.746+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2480b3930 con 0x7fb2480ccfb0 2023-10-14T16:56:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.748+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24806b330 con 0x7fb2480ccfb0 2023-10-14T16:56:19.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.748+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb2400079c0 con 0x7fb2480ccfb0 2023-10-14T16:56:19.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.749+0000 7fb2457fa640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 0x7fb22806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.750+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fb240089b30 con 0x7fb2480ccfb0 2023-10-14T16:56:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.750+0000 7fb247fff640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 0x7fb22806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.750+0000 7fb247fff640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 0x7fb22806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb248069f00 tx=0x7fb238009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:19.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.754+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb240054aa0 con 0x7fb2480ccfb0 2023-10-14T16:56:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.987+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2480b3e50 con 0x7fb2480ccfb0 2023-10-14T16:56:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.988+0000 7fb2457fa640 1 -- 172.21.15.29:0/2796599381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) v1 ==== 94+0+3696 (secure 0 0 0) 0x7fb2480b3e50 con 0x7fb2480ccfb0 2023-10-14T16:56:19.989 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:19.990 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":26,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4919,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/666656439","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":666656439},{"type":"v1","addr":"172.21.15.29:6835","nonce":666656439}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":18},{"gid":5126,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/110958744","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":110958744},{"type":"v1","addr":"172.21.15.186:6835","nonce":110958744}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":26},{"gid":5230,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1250136678","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1250136678},{"type":"v1","addr":"172.21.15.29:6837","nonce":1250136678}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19},{"gid":5233,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/536820090","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":536820090},{"type":"v1","addr":"172.21.15.186:6833","nonce":536820090}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":19}],"filesystems":[]} 2023-10-14T16:56:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 26 2023-10-14T16:56:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.991+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 msgr2=0x7fb22806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.991+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 0x7fb22806d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb248069f00 tx=0x7fb238009040 comp rx=0 tx=0).stop 2023-10-14T16:56:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.991+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 msgr2=0x7fb2480b2850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.991+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fb24000c7c0 tx=0x7fb24000cc80 comp rx=0 tx=0).stop 2023-10-14T16:56:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 shutdown_connections 2023-10-14T16:56:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb22806af90 0x7fb22806d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480ccfb0 0x7fb2480b2850 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480ad970 0x7fb2480b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 --2- 172.21.15.29:0/2796599381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24806c040 0x7fb2480c4110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 >> 172.21.15.29:0/2796599381 conn(0x7fb2480ce6a0 msgr2=0x7fb24806f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 shutdown_connections 2023-10-14T16:56:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:19.992+0000 7fb24d3a5640 1 -- 172.21.15.29:0/2796599381 wait complete. 2023-10-14T16:56:20.013 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:56:20.013 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:56:20.013 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:56:20.014 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:20.014 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:56:20.014 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:56:20.015 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:56:20.015 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:20.015 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:56:20.016 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:56:20.016 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:56:20.016 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:20.016 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:56:20.017 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:56:20.017 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:56:20.017 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:20.033 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:56:20.033 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:56:20.037 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:56:20.038 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:56:20.039 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:56:20.039 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:56:20.040 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:56:20.040 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:56:20.085 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:56:20.086 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:20.090 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:56:20.090 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:20.093 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:56:20.094 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:20.097 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:56:20.098 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:20.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:56:20.248 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:56:20.252 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:56:20.255 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:56:20.255 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:56:20.257 INFO:tasks.ceph.mds.d.smithi186.stderr:2023-10-14T16:56:20.256+0000 7fb36212e640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6832/0: (98) Address already in use 2023-10-14T16:56:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.289+0000 7fd7cb0af640 1 Processor -- start 2023-10-14T16:56:20.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.289+0000 7fd7cb0af640 1 -- start start 2023-10-14T16:56:20.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7cb0af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7cb0af640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c406a4b0 0x7fd7c41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7cb0af640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c40cb140 con 0x7fd7c40c9000 2023-10-14T16:56:20.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7cb0af640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c4161a50 con 0x7fd7c406a4b0 2023-10-14T16:56:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7cb0af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c406b030 con 0x7fd7c40addb0 2023-10-14T16:56:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c406a4b0 0x7fd7c41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd7c406a4b0 0x7fd7c41650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40614/0 (socket says 172.21.15.29:40614) 2023-10-14T16:56:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 -- 172.21.15.29:0/2996945499 learned_addr learned my addr 172.21.15.29:0/2996945499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:20.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c8e4b640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 -- 172.21.15.29:0/2996945499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 msgr2=0x7fd7c40caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 -- 172.21.15.29:0/2996945499 >> v1:172.21.15.186:6789/0 conn(0x7fd7c40c9000 legacy=0x7fd7c4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:20.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.291+0000 7fd7c964c640 1 -- 172.21.15.29:0/2996945499 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c406b310 con 0x7fd7c406a4b0 2023-10-14T16:56:20.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.292+0000 7fd7c964c640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c41650d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009e50 tx=0x7fd7bc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61950d18facb4091 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:20.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7c37fe640 1 -- 172.21.15.29:0/2996945499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc02cb00 con 0x7fd7c406a4b0 2023-10-14T16:56:20.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7c37fe640 1 -- 172.21.15.29:0/2996945499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7bc02cc60 con 0x7fd7c406a4b0 2023-10-14T16:56:20.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7c37fe640 1 -- 172.21.15.29:0/2996945499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7bc0343f0 con 0x7fd7c406a4b0 2023-10-14T16:56:20.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 -- 172.21.15.29:0/2996945499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 msgr2=0x7fd7c41650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:20.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c41650d0 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fd7bc009e50 tx=0x7fd7bc02bf60 comp rx=0 tx=0).stop 2023-10-14T16:56:20.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 -- 172.21.15.29:0/2996945499 shutdown_connections 2023-10-14T16:56:20.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c41650d0 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:20.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/2996945499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:20.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 -- 172.21.15.29:0/2996945499 >> 172.21.15.29:0/2996945499 conn(0x7fd7c40ce9b0 msgr2=0x7fd7c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:20.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 -- 172.21.15.29:0/2996945499 shutdown_connections 2023-10-14T16:56:20.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.293+0000 7fd7cb0af640 1 -- 172.21.15.29:0/2996945499 wait complete. 2023-10-14T16:56:20.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.295+0000 7fd7cb0af640 1 Processor -- start 2023-10-14T16:56:20.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.295+0000 7fd7cb0af640 1 -- start start 2023-10-14T16:56:20.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 0x7fd7c40b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c41624c0 con 0x7fd7c406a4b0 2023-10-14T16:56:20.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b3e90 con 0x7fd7c40c9000 2023-10-14T16:56:20.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7cb0af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c40b24a0 con 0x7fd7c40addb0 2023-10-14T16:56:20.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.296+0000 7fd7c8e4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40618/0 (socket says 172.21.15.29:40618) 2023-10-14T16:56:20.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 -- 172.21.15.29:0/3698175968 learned_addr learned my addr 172.21.15.29:0/3698175968 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:20.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c964c640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 0x7fd7c40b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c3fff640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 -- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 msgr2=0x7fd7c40b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:20.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 -- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 msgr2=0x7fd7c40b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 0x7fd7c40b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 -- 172.21.15.29:0/3698175968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7bc009b30 con 0x7fd7c406a4b0 2023-10-14T16:56:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c964c640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 0x7fd7c40b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:20.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c8e4b640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800e9b0 tx=0x7fd7b800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.297+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b8009670 con 0x7fd7c406a4b0 2023-10-14T16:56:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.298+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b800c9a0 con 0x7fd7c406a4b0 2023-10-14T16:56:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.298+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7b800cd50 con 0x7fd7c406a4b0 2023-10-14T16:56:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.298+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c4161bd0 con 0x7fd7c406a4b0 2023-10-14T16:56:20.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.298+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c4161e70 con 0x7fd7c406a4b0 2023-10-14T16:56:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.299+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c404e160 con 0x7fd7c406a4b0 2023-10-14T16:56:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.300+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd7b8013470 con 0x7fd7c406a4b0 2023-10-14T16:56:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.300+0000 7fd7c17fa640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 0x7fd7a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:20.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.301+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fd7b80899d0 con 0x7fd7c406a4b0 2023-10-14T16:56:20.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.301+0000 7fd7c3fff640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 0x7fd7a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:20.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.301+0000 7fd7c3fff640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 0x7fd7a806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40c8220 tx=0x7fd7b0009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:20.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.305+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd7b8054980 con 0x7fd7c406a4b0 2023-10-14T16:56:20.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:20.513+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fd7c406c5b0 con 0x7fd7c406a4b0 2023-10-14T16:56:21.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.568+0000 7fd7c17fa640 1 -- 172.21.15.29:0/3698175968 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v78) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd7b8017080 con 0x7fd7c406a4b0 2023-10-14T16:56:21.569 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:56:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.571+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 msgr2=0x7fd7a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.571+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 0x7fd7a806d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fd7c40c8220 tx=0x7fd7b0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.571+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 msgr2=0x7fd7c40c2d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.571+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fd7b800e9b0 tx=0x7fd7b800ee70 comp rx=0 tx=0).stop 2023-10-14T16:56:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 shutdown_connections 2023-10-14T16:56:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd7a806af90 0x7fd7a806d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd7c40c9000 0x7fd7c40b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd7c40addb0 0x7fd7c40b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 --2- 172.21.15.29:0/3698175968 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd7c406a4b0 0x7fd7c40c2d70 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 >> 172.21.15.29:0/3698175968 conn(0x7fd7c40ce9b0 msgr2=0x7fd7c40aedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:21.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 shutdown_connections 2023-10-14T16:56:21.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.572+0000 7fd7cb0af640 1 -- 172.21.15.29:0/3698175968 wait complete. 2023-10-14T16:56:21.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:56:21.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.820+0000 7f31ea1bd640 1 Processor -- start 2023-10-14T16:56:21.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.820+0000 7f31ea1bd640 1 -- start start 2023-10-14T16:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31ea1bd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e415b320 0x7f31e40cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31ea1bd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e415b9d0 con 0x7f31e406c040 2023-10-14T16:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31ea1bd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31e415bcb0 con 0x7f31e40ad970 2023-10-14T16:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31ea1bd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e415b7d0 con 0x7f31e415b320 2023-10-14T16:56:21.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31e91bb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31e406c040 0x7f31e406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56734/0 (socket says 172.21.15.29:56734) 2023-10-14T16:56:21.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.821+0000 7f31e91bb640 1 -- 172.21.15.29:0/2015643125 learned_addr learned my addr 172.21.15.29:0/2015643125 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665690932 0 0) 0x7f31e415b9d0 con 0x7f31e406c040 2023-10-14T16:56:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e415b320 0x7f31e40cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31d0003520 con 0x7f31e406c040 2023-10-14T16:56:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812019508 0 0) 0x7f31d0003520 con 0x7f31e406c040 2023-10-14T16:56:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e415b320 msgr2=0x7f31e40cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e415b320 0x7f31e40cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 >> v1:172.21.15.186:6789/0 conn(0x7f31e40ad970 legacy=0x7f31e40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e415bf90 con 0x7f31e406c040 2023-10-14T16:56:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f31e00019f0 con 0x7f31e406c040 2023-10-14T16:56:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 >> v1:172.21.15.29:6789/0 conn(0x7f31e406c040 legacy=0x7f31e406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31d00039a0 0x7f31d0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31d00047c0 0x7f31d0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.822+0000 7f31dbfff640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31d0005180 0x7f31d000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e415b7d0 con 0x7f31d0005180 2023-10-14T16:56:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31d000c080 con 0x7f31d00047c0 2023-10-14T16:56:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d000c240 con 0x7f31d00039a0 2023-10-14T16:56:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e415b320 0x7f31e40cd180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31d0005180 0x7f31d000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e89ba640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31d00047c0 0x7f31d0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e91bb640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31d00039a0 0x7f31d0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 -- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31d00039a0 msgr2=0x7f31d0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31d00039a0 0x7f31d0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 -- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31d00047c0 msgr2=0x7f31d0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31d00047c0 0x7f31d0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e99bc640 1 -- 172.21.15.29:0/2015643125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d0003520 con 0x7f31d0005180 2023-10-14T16:56:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.823+0000 7f31e91bb640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31d00039a0 0x7f31d0003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31d0005180 0x7f31d000b980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006fd0 tx=0x7f31d00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49bbde53f3149283 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc033600 con 0x7f31d0005180 2023-10-14T16:56:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc02f050 con 0x7f31d0005180 2023-10-14T16:56:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31dbfff640 1 -- 172.21.15.29:0/2015643125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc033980 con 0x7f31d0005180 2023-10-14T16:56:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2015643125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31d0005180 msgr2=0x7f31d000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2015643125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31d0005180 0x7f31d000b980 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7f31dc006fd0 tx=0x7f31d00057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31e99bc640 1 -- 172.21.15.29:0/2015643125 reap_dead start 2023-10-14T16:56:21.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.825+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2015643125 shutdown_connections 2023-10-14T16:56:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.826+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2015643125 >> 172.21.15.29:0/2015643125 conn(0x7f31e40ce6a0 msgr2=0x7f31e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.826+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2015643125 shutdown_connections 2023-10-14T16:56:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.826+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2015643125 wait complete. 2023-10-14T16:56:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.827+0000 7f31ea1bd640 1 Processor -- start 2023-10-14T16:56:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.828+0000 7f31ea1bd640 1 -- start start 2023-10-14T16:56:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.828+0000 7f31ea1bd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e414de50 0x7f31e40c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.828+0000 7f31ea1bd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31e414e230 0x7f31e4154210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.828+0000 7f31ea1bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.828+0000 7f31ea1bd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4153170 con 0x7f31e40c2710 2023-10-14T16:56:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31ea1bd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4153450 con 0x7f31e414e230 2023-10-14T16:56:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31ea1bd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31e4153730 con 0x7f31e414de50 2023-10-14T16:56:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51340/0 (socket says 172.21.15.29:51340) 2023-10-14T16:56:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 -- 172.21.15.29:0/2462955995 learned_addr learned my addr 172.21.15.29:0/2462955995 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 -- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e414de50 msgr2=0x7f31e40c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e91bb640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e414de50 0x7f31e40c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e414de50 0x7f31e40c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 -- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31e414e230 msgr2=0x7f31e4154210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31e414e230 0x7f31e4154210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.829+0000 7f31e99bc640 1 -- 172.21.15.29:0/2462955995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31e4153a10 con 0x7f31e40c2710 2023-10-14T16:56:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31e99bc640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0097c0 tx=0x7f31dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc03d650 con 0x7f31e40c2710 2023-10-14T16:56:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31dc03d7d0 con 0x7f31e40c2710 2023-10-14T16:56:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31dc03db80 con 0x7f31e40c2710 2023-10-14T16:56:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31e4153cf0 con 0x7f31e40c2710 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.830+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31e414e810 con 0x7f31e40c2710 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.831+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31e406b330 con 0x7f31e40c2710 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.832+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f31dc03dce0 con 0x7f31e40c2710 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.834+0000 7f31d9ffb640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 0x7f31c806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.834+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f31dc000e40 con 0x7f31e40c2710 2023-10-14T16:56:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.834+0000 7f31e91bb640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 0x7f31c806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.834+0000 7f31e91bb640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 0x7f31c806d4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e280 tx=0x7f31e0009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:21.838+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f31dc0762c0 con 0x7f31e40c2710 2023-10-14T16:56:22.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.107+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f31e4153f10 con 0x7f31e40c2710 2023-10-14T16:56:22.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.108+0000 7f31d9ffb640 1 -- 172.21.15.29:0/2462955995 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v13) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f31dc00f040 con 0x7f31e40c2710 2023-10-14T16:56:22.109 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:22.109 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:56:22.110 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:56:22.110 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.110 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:56:22.110 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:56:22.111 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:56:22.111 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:56:22.111 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.112 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.112 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.112 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.112 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:56:22.112 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:56:22.113 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:56:22.113 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:56:22.113 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.114 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.114 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.114 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.114 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:56:22.114 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:56:22.115 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:56:22.115 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:56:22.115 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.116 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.116 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.116 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.116 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:56:22.116 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:56:22.117 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:56:22.117 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:56:22.117 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:56:22.118 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.119 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.119 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.119 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.119 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.120 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.120 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:56:22.120 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:56:22.120 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.120 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.121 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.121 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.121 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.121 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:56:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:56:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.123 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.123 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.123 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.123 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.123 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.124 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.125 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.125 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.125 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:56:22.125 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:56:22.125 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.126 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.126 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.126 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.126 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.126 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.127 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.127 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:56:22.127 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:56:22.127 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.127 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.129 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:56:22.129 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:56:22.129 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.129 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.129 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.130 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.130 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.130 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.130 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.130 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:56:22.131 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:56:22.131 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.131 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:56:22.131 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:56:22.131 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.132 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.132 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.132 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.132 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:56:22.132 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:56:22.133 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.133 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.133 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T16:56:22.133 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T16:56:22.133 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.134 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.134 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.134 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:56:22.134 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:56:22.134 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:56:22.135 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:56:22.135 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:56:22.135 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.136 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.136 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:56:22.136 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:56:22.136 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.137 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.137 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:56:22.137 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:56:22.137 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:56:22.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.138 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.138 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:56:22.138 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:56:22.138 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:56:22.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:56:22.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:56:22.139 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:56:22.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.140 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:56:22.140 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:56:22.140 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.141 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.142 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:56:22.142 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:56:22.142 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:56:22.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:56:22.143 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:56:22.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:22.144 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:56:22.144 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:56:22.144 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:56:22.144 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:56:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:56:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:56:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:22.145 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:56:22.146 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:56:22.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 msgr2=0x7f31c806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 0x7f31c806d4d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f31e000e280 tx=0x7f31e0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 msgr2=0x7f31e4154750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f31dc0097c0 tx=0x7f31dc004730 comp rx=0 tx=0).stop 2023-10-14T16:56:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 shutdown_connections 2023-10-14T16:56:22.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f31c806b010 0x7f31c806d4d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f31e40c2710 0x7f31e4154750 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f31e414e230 0x7f31e4154210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 --2- 172.21.15.29:0/2462955995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31e414de50 0x7f31e40c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.111+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 >> 172.21.15.29:0/2462955995 conn(0x7f31e40ce6a0 msgr2=0x7f31e40ccdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:22.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.112+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 shutdown_connections 2023-10-14T16:56:22.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.112+0000 7f31ea1bd640 1 -- 172.21.15.29:0/2462955995 wait complete. 2023-10-14T16:56:22.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.381+0000 7fecfffff640 1 Processor -- start 2023-10-14T16:56:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.381+0000 7fecfffff640 1 -- start start 2023-10-14T16:56:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 0x7fed000cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed000ccfb0 0x7fed000cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0015b9d0 con 0x7fed0006c040 2023-10-14T16:56:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0015bcb0 con 0x7fed000ccfb0 2023-10-14T16:56:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0015bf90 con 0x7fed000ad970 2023-10-14T16:56:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed0006c040 0x7fed0006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56746/0 (socket says 172.21.15.29:56746) 2023-10-14T16:56:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.383+0000 7fecfeffd640 1 -- 172.21.15.29:0/1361620681 learned_addr learned my addr 172.21.15.29:0/1361620681 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 894173169 0 0) 0x7fed0015b9d0 con 0x7fed0006c040 2023-10-14T16:56:22.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fece8003520 con 0x7fed0006c040 2023-10-14T16:56:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfe7fc640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 0x7fed000cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecff7fe640 1 --2- 172.21.15.29:0/1361620681 >> v2:172.21.15.186:3300/0 conn(0x7fed000ccfb0 0x7fed000cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 871216963 0 0) 0x7fece8003520 con 0x7fed0006c040 2023-10-14T16:56:22.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 msgr2=0x7fed000cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 0x7fed000cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 >> v2:172.21.15.186:3300/0 conn(0x7fed000ccfb0 msgr2=0x7fed000cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 --2- 172.21.15.29:0/1361620681 >> v2:172.21.15.186:3300/0 conn(0x7fed000ccfb0 0x7fed000cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000cc270 con 0x7fed0006c040 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fecf0001670 con 0x7fed0006c040 2023-10-14T16:56:22.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 >> v1:172.21.15.29:6789/0 conn(0x7fed0006c040 legacy=0x7fed0006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fece8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 0x7fece80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fece8004d00 0x7fece8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0015bcb0 con 0x7fece8003970 2023-10-14T16:56:22.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed0015bf90 con 0x7fece8004d00 2023-10-14T16:56:22.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.384+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fece800bae0 con 0x7fed000ad970 2023-10-14T16:56:22.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fece8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 -- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 msgr2=0x7fece80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000ad970 0x7fece80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 -- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fece8004d00 msgr2=0x7fece8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fece8004d00 0x7fece8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.385+0000 7fecfeffd640 1 -- 172.21.15.29:0/1361620681 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece8003520 con 0x7fece8003970 2023-10-14T16:56:22.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.386+0000 7fecfeffd640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fece8003d90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c0e0 tx=0x7fece80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78d8a32eca21fcca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.386+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0005ad0 con 0x7fece8003970 2023-10-14T16:56:22.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.386+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf0005c50 con 0x7fece8003970 2023-10-14T16:56:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.386+0000 7fecfdffb640 1 -- 172.21.15.29:0/1361620681 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf002c810 con 0x7fece8003970 2023-10-14T16:56:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 -- 172.21.15.29:0/1361620681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 msgr2=0x7fece8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 --2- 172.21.15.29:0/1361620681 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fece8003d90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c0e0 tx=0x7fece80057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecff7fe640 1 -- 172.21.15.29:0/1361620681 reap_dead start 2023-10-14T16:56:22.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 -- 172.21.15.29:0/1361620681 shutdown_connections 2023-10-14T16:56:22.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 -- 172.21.15.29:0/1361620681 >> 172.21.15.29:0/1361620681 conn(0x7fed000ce6a0 msgr2=0x7fed000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:22.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 -- 172.21.15.29:0/1361620681 shutdown_connections 2023-10-14T16:56:22.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.387+0000 7fecfffff640 1 -- 172.21.15.29:0/1361620681 wait complete. 2023-10-14T16:56:22.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.389+0000 7fecfffff640 1 Processor -- start 2023-10-14T16:56:22.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.389+0000 7fecfffff640 1 -- start start 2023-10-14T16:56:22.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000c7df0 0x7fed000af180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b0370 0x7fed001495e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000c7320 con 0x7fece8003970 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed000c6950 con 0x7fed000c7df0 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed000af6c0 con 0x7fed000b0370 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51372/0 (socket says 172.21.15.29:51372) 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.390+0000 7fecfeffd640 1 -- 172.21.15.29:0/3918602313 learned_addr learned my addr 172.21.15.29:0/3918602313 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:22.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 -- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b0370 msgr2=0x7fed001495e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b0370 0x7fed001495e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 -- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000c7df0 msgr2=0x7fed000af180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000c7df0 0x7fed000af180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 -- 172.21.15.29:0/3918602313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000c83b0 con 0x7fece8003970 2023-10-14T16:56:22.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.391+0000 7fecfeffd640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c0e0 tx=0x7fecf0001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.392+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf002cc20 con 0x7fece8003970 2023-10-14T16:56:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.392+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf002cd80 con 0x7fece8003970 2023-10-14T16:56:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.392+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf0007270 con 0x7fece8003970 2023-10-14T16:56:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.392+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed000c6300 con 0x7fece8003970 2023-10-14T16:56:22.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.392+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed000b07f0 con 0x7fece8003970 2023-10-14T16:56:22.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.393+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed000b0a40 con 0x7fece8003970 2023-10-14T16:56:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.394+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fecf0032070 con 0x7fece8003970 2023-10-14T16:56:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.395+0000 7fece7fff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.396+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fecf003c040 con 0x7fece8003970 2023-10-14T16:56:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.396+0000 7fecfe7fc640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.396+0000 7fecfe7fc640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 0x7fecec06d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006f00 tx=0x7fecf4008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.400+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fecf0075c70 con 0x7fece8003970 2023-10-14T16:56:22.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.597+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed000cc520 con 0x7fece8003970 2023-10-14T16:56:22.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.599+0000 7fece7fff640 1 -- 172.21.15.29:0/3918602313 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v78) v1 ==== 74+0+12915 (secure 0 0 0) 0x7fecf0079290 con 0x7fece8003970 2023-10-14T16:56:22.600 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:22.600 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":78,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:21.565237+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":65,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:22.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.601+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 msgr2=0x7fecec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.601+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 0x7fecec06d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fecf4006f00 tx=0x7fecf4008040 comp rx=0 tx=0).stop 2023-10-14T16:56:22.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.601+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 msgr2=0x7fed000c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.601+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fecf000c0e0 tx=0x7fecf0001930 comp rx=0 tx=0).stop 2023-10-14T16:56:22.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 shutdown_connections 2023-10-14T16:56:22.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed000b0370 0x7fed001495e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed000c7df0 0x7fed000af180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 --2- 172.21.15.29:0/3918602313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fece8003970 0x7fed000c6de0 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 >> 172.21.15.29:0/3918602313 conn(0x7fed000ce6a0 msgr2=0x7fed0006ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:22.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 shutdown_connections 2023-10-14T16:56:22.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.602+0000 7fecfffff640 1 -- 172.21.15.29:0/3918602313 wait complete. 2023-10-14T16:56:22.622 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:22.623 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.857+0000 7f5487e7c640 1 Processor -- start 2023-10-14T16:56:22.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.858+0000 7f5487e7c640 1 -- start start 2023-10-14T16:56:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5487e7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f54800b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5487e7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54800b0df0 0x7f548015d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5487e7c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54800b1230 con 0x7f54801540e0 2023-10-14T16:56:22.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5487e7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54801544c0 con 0x7f54800b0df0 2023-10-14T16:56:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5487e7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54800ccf00 con 0x7f54800cca00 2023-10-14T16:56:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5486419640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54800b0df0 0x7f548015d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5486419640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54800b0df0 0x7f548015d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51388/0 (socket says 172.21.15.29:51388) 2023-10-14T16:56:22.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.859+0000 7f5486419640 1 -- 172.21.15.29:0/1422227525 learned_addr learned my addr 172.21.15.29:0/1422227525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.860+0000 7f5486419640 1 -- 172.21.15.29:0/1422227525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 msgr2=0x7f54800b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.860+0000 7f5486419640 1 --2- 172.21.15.29:0/1422227525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f54800b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.860+0000 7f5486419640 1 -- 172.21.15.29:0/1422227525 >> v1:172.21.15.186:6789/0 conn(0x7f54801540e0 legacy=0x7f54800adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:22.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.860+0000 7f5486419640 1 -- 172.21.15.29:0/1422227525 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54800cd1e0 con 0x7f54800b0df0 2023-10-14T16:56:22.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.861+0000 7f5486419640 1 --2- 172.21.15.29:0/1422227525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f548015d5a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5478009870 tx=0x7f547802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3364eb367f899945 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.861+0000 7f5484c16640 1 -- 172.21.15.29:0/1422227525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478031640 con 0x7f54800b0df0 2023-10-14T16:56:22.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.861+0000 7f5484c16640 1 -- 172.21.15.29:0/1422227525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54780317a0 con 0x7f54800b0df0 2023-10-14T16:56:22.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.861+0000 7f5484c16640 1 -- 172.21.15.29:0/1422227525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478031b30 con 0x7f54800b0df0 2023-10-14T16:56:22.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 -- 172.21.15.29:0/1422227525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 msgr2=0x7f548015d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1422227525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f548015d5a0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f5478009870 tx=0x7f547802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 -- 172.21.15.29:0/1422227525 shutdown_connections 2023-10-14T16:56:22.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1422227525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f548015d5a0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1422227525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f54800b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 -- 172.21.15.29:0/1422227525 >> 172.21.15.29:0/1422227525 conn(0x7f54800cb8b0 msgr2=0x7f54800cbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 -- 172.21.15.29:0/1422227525 shutdown_connections 2023-10-14T16:56:22.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.862+0000 7f5487e7c640 1 -- 172.21.15.29:0/1422227525 wait complete. 2023-10-14T16:56:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.864+0000 7f5487e7c640 1 Processor -- start 2023-10-14T16:56:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.864+0000 7f5487e7c640 1 -- start start 2023-10-14T16:56:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f5480156f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 0x7f54801f6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801f3950 con 0x7f54800b0df0 2023-10-14T16:56:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54801f30b0 con 0x7f54801540e0 2023-10-14T16:56:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5487e7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54801f3390 con 0x7f54800cca00 2023-10-14T16:56:22.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5485c18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5485c18640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51398/0 (socket says 172.21.15.29:51398) 2023-10-14T16:56:22.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.865+0000 7f5485c18640 1 -- 172.21.15.29:0/1564737640 learned_addr learned my addr 172.21.15.29:0/1564737640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5486419640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 0x7f54801f6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485417640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f5480156f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 -- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 msgr2=0x7f5480156f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f5480156f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 -- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 msgr2=0x7f54801f6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 0x7f54801f6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 -- 172.21.15.29:0/1564737640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54780094d0 con 0x7f54800b0df0 2023-10-14T16:56:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5486419640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 0x7f54801f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:22.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5485c18640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f547c00ba50 tx=0x7f547c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c00cbe0 con 0x7f54800b0df0 2023-10-14T16:56:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c00cd80 con 0x7f54800b0df0 2023-10-14T16:56:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547c014680 con 0x7f54800b0df0 2023-10-14T16:56:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.866+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54801f3670 con 0x7f54800b0df0 2023-10-14T16:56:22.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.867+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54801574a0 con 0x7f54800b0df0 2023-10-14T16:56:22.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.868+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54801f5fa0 con 0x7f54800b0df0 2023-10-14T16:56:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.869+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f547c0147e0 con 0x7f54800b0df0 2023-10-14T16:56:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.870+0000 7f54767fc640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 0x7f546406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.870+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f547c0892e0 con 0x7f54800b0df0 2023-10-14T16:56:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.870+0000 7f5485417640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 0x7f546406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:22.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.871+0000 7f5485417640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 0x7f546406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5480071080 tx=0x7f5470008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:22.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:22.877+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54801f5fa0 con 0x7f54800b0df0 2023-10-14T16:56:23.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.080+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f54801f5fa0 con 0x7f54800b0df0 2023-10-14T16:56:23.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.081+0000 7f54767fc640 1 -- 172.21.15.29:0/1564737640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v78) v1 ==== 93+0+10 (secure 0 0 0) 0x7f54801f5fa0 con 0x7f54800b0df0 2023-10-14T16:56:23.082 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 msgr2=0x7f546406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 0x7f546406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5480071080 tx=0x7f5470008040 comp rx=0 tx=0).stop 2023-10-14T16:56:23.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 msgr2=0x7f54801f2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f547c00ba50 tx=0x7f547c00bf10 comp rx=0 tx=0).stop 2023-10-14T16:56:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 shutdown_connections 2023-10-14T16:56:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f546406af90 0x7f546406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54801540e0 0x7f54801f6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54800cca00 0x7f5480156f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 --2- 172.21.15.29:0/1564737640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54800b0df0 0x7f54801f2a20 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.084+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 >> 172.21.15.29:0/1564737640 conn(0x7f54800cb8b0 msgr2=0x7f54800c9460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.085+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 shutdown_connections 2023-10-14T16:56:23.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.085+0000 7f5487e7c640 1 -- 172.21.15.29:0/1564737640 wait complete. 2023-10-14T16:56:23.104 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:56:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.336+0000 7f5d5359e640 1 Processor -- start 2023-10-14T16:56:23.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.336+0000 7f5d5359e640 1 -- start start 2023-10-14T16:56:23.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.338+0000 7f5d5359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5d5406c040 0x7f5d5406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.338+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d5415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.338+0000 7f5d5359e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5406c960 con 0x7f5d5415ebc0 2023-10-14T16:56:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.338+0000 7f5d5359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5415b9d0 con 0x7f5d5406c040 2023-10-14T16:56:23.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.338+0000 7f5d5359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5415bcb0 con 0x7f5d540ad970 2023-10-14T16:56:23.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d52d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5d5415ebc0 0x7f5d540cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56760/0 (socket says 172.21.15.29:56760) 2023-10-14T16:56:23.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d52d9d640 1 -- 172.21.15.29:0/3340058313 learned_addr learned my addr 172.21.15.29:0/3340058313 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d5415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d5259c640 1 --2- 172.21.15.29:0/3340058313 >> v2:172.21.15.186:3300/0 conn(0x7f5d5406c040 0x7f5d5406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 482515625 0 0) 0x7f5d5406c960 con 0x7f5d5415ebc0 2023-10-14T16:56:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d40003520 con 0x7f5d5415ebc0 2023-10-14T16:56:23.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1166479082 0 0) 0x7f5d40003520 con 0x7f5d5415ebc0 2023-10-14T16:56:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 msgr2=0x7f5d5415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d5415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 >> v2:172.21.15.186:3300/0 conn(0x7f5d5406c040 msgr2=0x7f5d5406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.339+0000 7f5d51d9b640 1 --2- 172.21.15.29:0/3340058313 >> v2:172.21.15.186:3300/0 conn(0x7f5d5406c040 0x7f5d5406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5415bf90 con 0x7f5d5415ebc0 2023-10-14T16:56:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5d4c0019f0 con 0x7f5d5415ebc0 2023-10-14T16:56:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 >> v1:172.21.15.29:6789/0 conn(0x7f5d5415ebc0 legacy=0x7f5d540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d40003970 0x7f5d40003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d400047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d40004d00 0x7f5d40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5415b9d0 con 0x7f5d40003970 2023-10-14T16:56:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5415bcb0 con 0x7f5d40004d00 2023-10-14T16:56:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d4000fd90 con 0x7f5d540ad970 2023-10-14T16:56:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d5259c640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d40003970 0x7f5d40003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.340+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d400047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d5259c640 1 -- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 msgr2=0x7f5d400047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d5259c640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d540ad970 0x7f5d400047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d5259c640 1 -- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d40004d00 msgr2=0x7f5d40005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d40004d00 0x7f5d40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d5259c640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d40004d00 0x7f5d40005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.341+0000 7f5d5259c640 1 -- 172.21.15.29:0/3340058313 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d40003520 con 0x7f5d40003970 2023-10-14T16:56:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.342+0000 7f5d5259c640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d40003970 0x7f5d40003d90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009af0 tx=0x7f5d400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bdc5784ee92e51 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4402c710 con 0x7f5d40003970 2023-10-14T16:56:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4402c870 con 0x7f5d40003970 2023-10-14T16:56:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d51d9b640 1 -- 172.21.15.29:0/3340058313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4402cc00 con 0x7f5d40003970 2023-10-14T16:56:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d5359e640 1 -- 172.21.15.29:0/3340058313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d40003970 msgr2=0x7f5d40003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d5359e640 1 --2- 172.21.15.29:0/3340058313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d40003970 0x7f5d40003d90 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f5d44009af0 tx=0x7f5d400057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d52d9d640 1 -- 172.21.15.29:0/3340058313 reap_dead start 2023-10-14T16:56:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d5359e640 1 -- 172.21.15.29:0/3340058313 shutdown_connections 2023-10-14T16:56:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.343+0000 7f5d5359e640 1 -- 172.21.15.29:0/3340058313 >> 172.21.15.29:0/3340058313 conn(0x7f5d540ce6a0 msgr2=0x7f5d540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.344+0000 7f5d5359e640 1 -- 172.21.15.29:0/3340058313 shutdown_connections 2023-10-14T16:56:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.344+0000 7f5d5359e640 1 -- 172.21.15.29:0/3340058313 wait complete. 2023-10-14T16:56:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.345+0000 7f5d5359e640 1 Processor -- start 2023-10-14T16:56:23.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.346+0000 7f5d5359e640 1 -- start start 2023-10-14T16:56:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.346+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40003970 0x7f5d54158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.346+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d541526e0 0x7f5d54151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d5359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d5359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54157450 con 0x7f5d540c79a0 2023-10-14T16:56:23.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d5359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d54157980 con 0x7f5d541526e0 2023-10-14T16:56:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d5359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d541576a0 con 0x7f5d40003970 2023-10-14T16:56:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51422/0 (socket says 172.21.15.29:51422) 2023-10-14T16:56:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 -- 172.21.15.29:0/2600457114 learned_addr learned my addr 172.21.15.29:0/2600457114 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:23.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d4bfff640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d541526e0 0x7f5d54151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40003970 0x7f5d54158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 -- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40003970 msgr2=0x7f5d54158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40003970 0x7f5d54158450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 -- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d541526e0 msgr2=0x7f5d54151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d541526e0 0x7f5d54151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.347+0000 7f5d52d9d640 1 -- 172.21.15.29:0/2600457114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d540c71a0 con 0x7f5d540c79a0 2023-10-14T16:56:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d52d9d640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5d5406d2e0 tx=0x7f5d4c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c004d60 con 0x7f5d540c79a0 2023-10-14T16:56:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c00bb20 con 0x7f5d540c79a0 2023-10-14T16:56:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d4c00beb0 con 0x7f5d540c79a0 2023-10-14T16:56:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d540c7480 con 0x7f5d540c79a0 2023-10-14T16:56:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.348+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d54152440 con 0x7f5d540c79a0 2023-10-14T16:56:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.349+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5404e9b0 con 0x7f5d540c79a0 2023-10-14T16:56:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.351+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5d4c007110 con 0x7f5d540c79a0 2023-10-14T16:56:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.352+0000 7f5d4affd640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 0x7f5d3406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.352+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f5d4c0898a0 con 0x7f5d540c79a0 2023-10-14T16:56:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.352+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 0x7f5d3406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.353+0000 7f5d5259c640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 0x7f5d3406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d44036000 tx=0x7f5d44030000 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.356+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d4c054a40 con 0x7f5d540c79a0 2023-10-14T16:56:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.553+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f5d5406b2b0 con 0x7f5d540c79a0 2023-10-14T16:56:23.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.554+0000 7f5d4affd640 1 -- 172.21.15.29:0/2600457114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v78) v1 ==== 84+0+65 (secure 0 0 0) 0x7f5d4c058060 con 0x7f5d540c79a0 2023-10-14T16:56:23.555 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:23.555 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:56:23.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:23.555 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:56:23.556 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:56:23.556 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:23.556 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:56:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.556+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 msgr2=0x7f5d3406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.556+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 0x7f5d3406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5d44036000 tx=0x7f5d44030000 comp rx=0 tx=0).stop 2023-10-14T16:56:23.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.556+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 msgr2=0x7f5d5415ce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.556+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f5d5406d2e0 tx=0x7f5d4c00b4a0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 shutdown_connections 2023-10-14T16:56:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f5d3406af90 0x7f5d3406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d540c79a0 0x7f5d5415ce60 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d541526e0 0x7f5d54151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 --2- 172.21.15.29:0/2600457114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d40003970 0x7f5d54158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 >> 172.21.15.29:0/2600457114 conn(0x7f5d540ce6a0 msgr2=0x7f5d540cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 shutdown_connections 2023-10-14T16:56:23.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.557+0000 7f5d5359e640 1 -- 172.21.15.29:0/2600457114 wait complete. 2023-10-14T16:56:23.575 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:56:23.576 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:56:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.799+0000 7faea9548640 1 Processor -- start 2023-10-14T16:56:23.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.800+0000 7faea9548640 1 -- start start 2023-10-14T16:56:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea9548640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faea40b0360 0x7faea40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea9548640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 0x7faea40ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea9548640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea415b310 con 0x7faea415ac10 2023-10-14T16:56:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea9548640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea415f800 con 0x7faea40b0360 2023-10-14T16:56:23.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea9548640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea40b0d30 con 0x7faea40ca510 2023-10-14T16:56:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faea40b0360 0x7faea40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faea40b0360 0x7faea40b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51436/0 (socket says 172.21.15.29:51436) 2023-10-14T16:56:23.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.801+0000 7faea27fc640 1 -- 172.21.15.29:0/2222810235 learned_addr learned my addr 172.21.15.29:0/2222810235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:23.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea37fe640 1 --2- 172.21.15.29:0/2222810235 >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 0x7faea40ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea27fc640 1 -- 172.21.15.29:0/2222810235 >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 msgr2=0x7faea40ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea27fc640 1 --2- 172.21.15.29:0/2222810235 >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 0x7faea40ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea27fc640 1 -- 172.21.15.29:0/2222810235 >> v1:172.21.15.29:6789/0 conn(0x7faea415ac10 legacy=0x7faea406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea27fc640 1 -- 172.21.15.29:0/2222810235 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea406f640 con 0x7faea40b0360 2023-10-14T16:56:23.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.802+0000 7faea37fe640 1 --2- 172.21.15.29:0/2222810235 >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 0x7faea40ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:23.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.803+0000 7faea27fc640 1 --2- 172.21.15.29:0/2222810235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40b0360 0x7faea40b07f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faea406d730 tx=0x7fae9802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bffa8904b28bfc8f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.803+0000 7faea1ffb640 1 -- 172.21.15.29:0/2222810235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98031870 con 0x7faea40b0360 2023-10-14T16:56:23.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.803+0000 7faea1ffb640 1 -- 172.21.15.29:0/2222810235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980319d0 con 0x7faea40b0360 2023-10-14T16:56:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.803+0000 7faea1ffb640 1 -- 172.21.15.29:0/2222810235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98031d60 con 0x7faea40b0360 2023-10-14T16:56:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 -- 172.21.15.29:0/2222810235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40b0360 msgr2=0x7faea40b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 --2- 172.21.15.29:0/2222810235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40b0360 0x7faea40b07f0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7faea406d730 tx=0x7fae9802b910 comp rx=0 tx=0).stop 2023-10-14T16:56:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 -- 172.21.15.29:0/2222810235 shutdown_connections 2023-10-14T16:56:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 --2- 172.21.15.29:0/2222810235 >> v2:172.21.15.186:3300/0 conn(0x7faea40ca510 0x7faea40ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 --2- 172.21.15.29:0/2222810235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40b0360 0x7faea40b07f0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 -- 172.21.15.29:0/2222810235 >> 172.21.15.29:0/2222810235 conn(0x7faea40cb360 msgr2=0x7faea40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:23.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 -- 172.21.15.29:0/2222810235 shutdown_connections 2023-10-14T16:56:23.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.804+0000 7faea9548640 1 -- 172.21.15.29:0/2222810235 wait complete. 2023-10-14T16:56:23.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.806+0000 7faea9548640 1 Processor -- start 2023-10-14T16:56:23.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.806+0000 7faea9548640 1 -- start start 2023-10-14T16:56:23.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faea40b0360 0x7faea40b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faea415ac10 0x7faea40cd350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40b1870 con 0x7faea40ca510 2023-10-14T16:56:23.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c5b50 con 0x7faea40b0360 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea9548640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea40c5e30 con 0x7faea415ac10 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51438/0 (socket says 172.21.15.29:51438) 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.807+0000 7faea27fc640 1 -- 172.21.15.29:0/991090959 learned_addr learned my addr 172.21.15.29:0/991090959 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea37fe640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faea415ac10 0x7faea40cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea2ffd640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faea40b0360 0x7faea40b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 -- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faea415ac10 msgr2=0x7faea40cd350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faea415ac10 0x7faea40cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 -- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faea40b0360 msgr2=0x7faea40b23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:23.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faea40b0360 0x7faea40b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 -- 172.21.15.29:0/991090959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae980094d0 con 0x7faea40ca510 2023-10-14T16:56:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea27fc640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fae9802b950 tx=0x7fae980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98031860 con 0x7faea40ca510 2023-10-14T16:56:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae980443f0 con 0x7faea40ca510 2023-10-14T16:56:23.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae980447a0 con 0x7faea40ca510 2023-10-14T16:56:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.808+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea40c6110 con 0x7faea40ca510 2023-10-14T16:56:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.809+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea415a1a0 con 0x7faea40ca510 2023-10-14T16:56:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.810+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea40c6630 con 0x7faea40ca510 2023-10-14T16:56:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.811+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae9804d450 con 0x7faea40ca510 2023-10-14T16:56:23.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.812+0000 7fae8b7fe640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 0x7fae9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:23.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.812+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fae9802d040 con 0x7faea40ca510 2023-10-14T16:56:23.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.812+0000 7faea2ffd640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 0x7fae9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:23.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.813+0000 7faea2ffd640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 0x7fae9006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae94000f60 tx=0x7fae94009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:23.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:23.817+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faea40c6630 con 0x7faea40ca510 2023-10-14T16:56:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.011+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faea40c6630 con 0x7faea40ca510 2023-10-14T16:56:24.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.598+0000 7fae8b7fe640 1 -- 172.21.15.29:0/991090959 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v79) v1 ==== 138+0+0 (secure 0 0 0) 0x7faea40c6630 con 0x7faea40ca510 2023-10-14T16:56:24.599 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:56:24.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 msgr2=0x7fae9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 0x7fae9006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fae94000f60 tx=0x7fae94009040 comp rx=0 tx=0).stop 2023-10-14T16:56:24.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 msgr2=0x7faea40b2910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fae9802b950 tx=0x7fae980046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 shutdown_connections 2023-10-14T16:56:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fae9006af90 0x7fae9006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faea415ac10 0x7faea40cd350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faea40ca510 0x7faea40b2910 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 --2- 172.21.15.29:0/991090959 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faea40b0360 0x7faea40b23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 >> 172.21.15.29:0/991090959 conn(0x7faea40cb360 msgr2=0x7faea40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:24.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 shutdown_connections 2023-10-14T16:56:24.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.602+0000 7faea9548640 1 -- 172.21.15.29:0/991090959 wait complete. 2023-10-14T16:56:24.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:56:24.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.843+0000 7f1d1421c640 1 Processor -- start 2023-10-14T16:56:24.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.844+0000 7f1d1421c640 1 -- start start 2023-10-14T16:56:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.845+0000 7f1d1421c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1d0c15ae80 0x7f1d0c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.845+0000 7f1d1421c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c0707e0 con 0x7f1d0c15aaa0 2023-10-14T16:56:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.845+0000 7f1d1421c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d0c06b7a0 con 0x7f1d0c0ce390 2023-10-14T16:56:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.845+0000 7f1d1421c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d0c15ef20 con 0x7f1d0c15ae80 2023-10-14T16:56:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d11fb8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d0c15aaa0 0x7f1d0c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56782/0 (socket says 172.21.15.29:56782) 2023-10-14T16:56:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d11fb8640 1 -- 172.21.15.29:0/158348811 learned_addr learned my addr 172.21.15.29:0/158348811 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d117b7640 1 --2- 172.21.15.29:0/158348811 >> v2:172.21.15.29:3300/0 conn(0x7f1d0c15ae80 0x7f1d0c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d10fb6640 1 -- 172.21.15.29:0/158348811 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3690917265 0 0) 0x7f1d0c0707e0 con 0x7f1d0c15aaa0 2023-10-14T16:56:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d10fb6640 1 -- 172.21.15.29:0/158348811 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cf8003520 con 0x7f1d0c15aaa0 2023-10-14T16:56:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d117b7640 1 -- 172.21.15.29:0/158348811 >> v1:172.21.15.186:6789/0 conn(0x7f1d0c0ce390 legacy=0x7f1d0c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d117b7640 1 -- 172.21.15.29:0/158348811 >> v1:172.21.15.29:6789/0 conn(0x7f1d0c15aaa0 legacy=0x7f1d0c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.846+0000 7f1d117b7640 1 -- 172.21.15.29:0/158348811 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d0c15f200 con 0x7f1d0c15ae80 2023-10-14T16:56:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d117b7640 1 --2- 172.21.15.29:0/158348811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c15e820 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1d080097a0 tx=0x7f1d0802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff60459215a8497 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d10fb6640 1 -- 172.21.15.29:0/158348811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0802e070 con 0x7f1d0c15ae80 2023-10-14T16:56:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d10fb6640 1 -- 172.21.15.29:0/158348811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d080315a0 con 0x7f1d0c15ae80 2023-10-14T16:56:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d10fb6640 1 -- 172.21.15.29:0/158348811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d08031930 con 0x7f1d0c15ae80 2023-10-14T16:56:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d1421c640 1 -- 172.21.15.29:0/158348811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 msgr2=0x7f1d0c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.848+0000 7f1d1421c640 1 --2- 172.21.15.29:0/158348811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c15e820 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7f1d080097a0 tx=0x7f1d0802b840 comp rx=0 tx=0).stop 2023-10-14T16:56:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.849+0000 7f1d1421c640 1 -- 172.21.15.29:0/158348811 shutdown_connections 2023-10-14T16:56:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.849+0000 7f1d1421c640 1 --2- 172.21.15.29:0/158348811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c15e820 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.849+0000 7f1d1421c640 1 -- 172.21.15.29:0/158348811 >> 172.21.15.29:0/158348811 conn(0x7f1d0c0cb410 msgr2=0x7f1d0c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.849+0000 7f1d1421c640 1 -- 172.21.15.29:0/158348811 shutdown_connections 2023-10-14T16:56:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.849+0000 7f1d1421c640 1 -- 172.21.15.29:0/158348811 wait complete. 2023-10-14T16:56:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.850+0000 7f1d1421c640 1 Processor -- start 2023-10-14T16:56:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.851+0000 7f1d1421c640 1 -- start start 2023-10-14T16:56:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d0c0ce390 0x7f1d0c0c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 0x7f1d0c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c1537b0 con 0x7f1d0c15ae80 2023-10-14T16:56:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0c5b90 con 0x7f1d0c15aaa0 2023-10-14T16:56:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d1421c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0c0c4a00 con 0x7f1d0c0ce390 2023-10-14T16:56:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d127b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d127b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51466/0 (socket says 172.21.15.29:51466) 2023-10-14T16:56:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d127b9640 1 -- 172.21.15.29:0/1238730441 learned_addr learned my addr 172.21.15.29:0/1238730441 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d117b7640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 0x7f1d0c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d11fb8640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d0c0ce390 0x7f1d0c0c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d127b9640 1 -- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d0c0ce390 msgr2=0x7f1d0c0c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.852+0000 7f1d127b9640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d0c0ce390 0x7f1d0c0c8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d127b9640 1 -- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 msgr2=0x7f1d0c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d127b9640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 0x7f1d0c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d127b9640 1 -- 172.21.15.29:0/1238730441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d08009400 con 0x7f1d0c15ae80 2023-10-14T16:56:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d117b7640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 0x7f1d0c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d127b9640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400c780 tx=0x7f1d0400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d04004b60 con 0x7f1d0c15ae80 2023-10-14T16:56:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d0400ce90 con 0x7f1d0c15ae80 2023-10-14T16:56:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d0400a850 con 0x7f1d0c15ae80 2023-10-14T16:56:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0c0c4ce0 con 0x7f1d0c15ae80 2023-10-14T16:56:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.853+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0c0c46f0 con 0x7f1d0c15ae80 2023-10-14T16:56:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.854+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0c0c5d00 con 0x7f1d0c15ae80 2023-10-14T16:56:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.856+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1d04006b50 con 0x7f1d0c15ae80 2023-10-14T16:56:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.857+0000 7f1cfe7fc640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 0x7f1cf006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.857+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5271+0+0 (secure 0 0 0) 0x7f1d04089f00 con 0x7f1d0c15ae80 2023-10-14T16:56:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.857+0000 7f1d11fb8640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 0x7f1cf006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.858+0000 7f1d11fb8640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 0x7f1cf006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d000011e0 tx=0x7f1d00009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:24.862+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d04055130 con 0x7f1d0c15ae80 2023-10-14T16:56:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.058+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1d0c04e160 con 0x7f1d0c15ae80 2023-10-14T16:56:25.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.611+0000 7f1cfe7fc640 1 -- 172.21.15.29:0/1238730441 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v80) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1d04058750 con 0x7f1d0c15ae80 2023-10-14T16:56:25.613 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:56:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 msgr2=0x7f1cf006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 0x7f1cf006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1d000011e0 tx=0x7f1d00009040 comp rx=0 tx=0).stop 2023-10-14T16:56:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 msgr2=0x7f1d0c0acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7f1d0400c780 tx=0x7f1d0400cc40 comp rx=0 tx=0).stop 2023-10-14T16:56:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 shutdown_connections 2023-10-14T16:56:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f1cf006af90 0x7f1cf006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d0c15ae80 0x7f1d0c0acdd0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d0c15aaa0 0x7f1d0c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 --2- 172.21.15.29:0/1238730441 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d0c0ce390 0x7f1d0c0c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.615+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 >> 172.21.15.29:0/1238730441 conn(0x7f1d0c0cb410 msgr2=0x7f1d0c0b0530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.616+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 shutdown_connections 2023-10-14T16:56:25.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.617+0000 7f1d1421c640 1 -- 172.21.15.29:0/1238730441 wait complete. 2023-10-14T16:56:25.655 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:56:25.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.848+0000 7f9a748d1640 1 Processor -- start 2023-10-14T16:56:25.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.848+0000 7f9a748d1640 1 -- start start 2023-10-14T16:56:25.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a748d1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9a7015c9f0 0x7f9a700c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a748d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a7015dde0 0x7f9a700ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a748d1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a70160a90 con 0x7f9a700c9230 2023-10-14T16:56:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a748d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a700cdec0 con 0x7f9a7015dde0 2023-10-14T16:56:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a748d1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a700cd900 con 0x7f9a7015c9f0 2023-10-14T16:56:25.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a6ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a7015dde0 0x7f9a700ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a6ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9a7015dde0 0x7f9a700ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51480/0 (socket says 172.21.15.29:51480) 2023-10-14T16:56:25.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.850+0000 7f9a6ffff640 1 -- 172.21.15.29:0/2285676454 learned_addr learned my addr 172.21.15.29:0/2285676454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6f7fe640 1 --2- 172.21.15.29:0/2285676454 >> v2:172.21.15.186:3300/0 conn(0x7f9a7015c9f0 0x7f9a700c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6ffff640 1 -- 172.21.15.29:0/2285676454 >> v2:172.21.15.186:3300/0 conn(0x7f9a7015c9f0 msgr2=0x7f9a700c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:25.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6ffff640 1 --2- 172.21.15.29:0/2285676454 >> v2:172.21.15.186:3300/0 conn(0x7f9a7015c9f0 0x7f9a700c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6ffff640 1 -- 172.21.15.29:0/2285676454 >> v1:172.21.15.29:6789/0 conn(0x7f9a700c9230 legacy=0x7f9a700709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6ffff640 1 -- 172.21.15.29:0/2285676454 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a700cdbe0 con 0x7f9a7015dde0 2023-10-14T16:56:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.851+0000 7f9a6effd640 1 -- 172.21.15.29:0/2285676454 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787532841 0 0) 0x7f9a70160a90 con 0x7f9a700c9230 2023-10-14T16:56:25.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.852+0000 7f9a6ffff640 1 --2- 172.21.15.29:0/2285676454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a700ac890 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9a700ad4e0 tx=0x7f9a6802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ada7a0fcf9f3ee3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.852+0000 7f9a6effd640 1 -- 172.21.15.29:0/2285676454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6802cb10 con 0x7f9a7015dde0 2023-10-14T16:56:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.852+0000 7f9a6effd640 1 -- 172.21.15.29:0/2285676454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6802cc70 con 0x7f9a7015dde0 2023-10-14T16:56:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.852+0000 7f9a6effd640 1 -- 172.21.15.29:0/2285676454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a680343f0 con 0x7f9a7015dde0 2023-10-14T16:56:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 -- 172.21.15.29:0/2285676454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 msgr2=0x7f9a700ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 --2- 172.21.15.29:0/2285676454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a700ac890 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f9a700ad4e0 tx=0x7f9a6802bf70 comp rx=0 tx=0).stop 2023-10-14T16:56:25.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 -- 172.21.15.29:0/2285676454 shutdown_connections 2023-10-14T16:56:25.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 --2- 172.21.15.29:0/2285676454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a700ac890 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 --2- 172.21.15.29:0/2285676454 >> v2:172.21.15.186:3300/0 conn(0x7f9a7015c9f0 0x7f9a700c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 -- 172.21.15.29:0/2285676454 >> 172.21.15.29:0/2285676454 conn(0x7f9a700ce6a0 msgr2=0x7f9a700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:25.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 -- 172.21.15.29:0/2285676454 shutdown_connections 2023-10-14T16:56:25.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.853+0000 7f9a748d1640 1 -- 172.21.15.29:0/2285676454 wait complete. 2023-10-14T16:56:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.855+0000 7f9a748d1640 1 Processor -- start 2023-10-14T16:56:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.855+0000 7f9a748d1640 1 -- start start 2023-10-14T16:56:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a700c9230 0x7f9a700ae410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a7015c9f0 0x7f9a700b0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7006a210 con 0x7f9a7015dde0 2023-10-14T16:56:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7006a4f0 con 0x7f9a7015c9f0 2023-10-14T16:56:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a748d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a7006a7d0 con 0x7f9a700c9230 2023-10-14T16:56:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51488/0 (socket says 172.21.15.29:51488) 2023-10-14T16:56:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.856+0000 7f9a6ffff640 1 -- 172.21.15.29:0/4084981679 learned_addr learned my addr 172.21.15.29:0/4084981679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a67fff640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a7015c9f0 0x7f9a700b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6f7fe640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a700c9230 0x7f9a700ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 -- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a700c9230 msgr2=0x7f9a700ae410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a700c9230 0x7f9a700ae410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 -- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a7015c9f0 msgr2=0x7f9a700b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:25.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a7015c9f0 0x7f9a700b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:25.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 -- 172.21.15.29:0/4084981679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a68009b30 con 0x7f9a7015dde0 2023-10-14T16:56:25.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.857+0000 7f9a6ffff640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9a68036040 tx=0x7f9a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:25.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.858+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a68009510 con 0x7f9a7015dde0 2023-10-14T16:56:25.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.858+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a7006aab0 con 0x7f9a7015dde0 2023-10-14T16:56:25.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.858+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a6803e450 con 0x7f9a7015dde0 2023-10-14T16:56:25.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.858+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a6803e5e0 con 0x7f9a7015dde0 2023-10-14T16:56:25.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.858+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a700b0f90 con 0x7f9a7015dde0 2023-10-14T16:56:25.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.859+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7006acf0 con 0x7f9a7015dde0 2023-10-14T16:56:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.860+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a6803e740 con 0x7f9a7015dde0 2023-10-14T16:56:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.861+0000 7f9a6cff9640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 0x7f9a5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.861+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f9a680ac720 con 0x7f9a7015dde0 2023-10-14T16:56:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.862+0000 7f9a6f7fe640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 0x7f9a5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:25.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.862+0000 7f9a6f7fe640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 0x7f9a5006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a60006fd0 tx=0x7f9a60008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:25.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:25.866+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a680776e0 con 0x7f9a7015dde0 2023-10-14T16:56:26.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.103+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f9a7006a350 con 0x7f9a7015dde0 2023-10-14T16:56:26.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.634+0000 7f9a6cff9640 1 -- 172.21.15.29:0/4084981679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 8 and data pool 9 v28) v1 ==== 172+0+0 (secure 0 0 0) 0x7f9a7006a350 con 0x7f9a7015dde0 2023-10-14T16:56:26.638 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 8 and data pool 9 2023-10-14T16:56:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.643+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 msgr2=0x7f9a5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.643+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 0x7f9a5006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9a60006fd0 tx=0x7f9a60008040 comp rx=0 tx=0).stop 2023-10-14T16:56:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.643+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 msgr2=0x7f9a70150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:26.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.643+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f9a68036040 tx=0x7f9a68004a40 comp rx=0 tx=0).stop 2023-10-14T16:56:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 shutdown_connections 2023-10-14T16:56:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f9a5006af90 0x7f9a5006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a7015dde0 0x7f9a70150bd0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9a7015c9f0 0x7f9a700b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 --2- 172.21.15.29:0/4084981679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a700c9230 0x7f9a700ae410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.644+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 >> 172.21.15.29:0/4084981679 conn(0x7f9a700ce6a0 msgr2=0x7f9a700c1750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.645+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 shutdown_connections 2023-10-14T16:56:26.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.645+0000 7f9a748d1640 1 -- 172.21.15.29:0/4084981679 wait complete. 2023-10-14T16:56:26.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:26.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.869+0000 7fbf13da9640 1 Processor -- start 2023-10-14T16:56:26.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.870+0000 7fbf13da9640 1 -- start start 2023-10-14T16:56:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf0c15aaa0 0x7fbf0c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15ae80 0x7fbf0c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c06b810 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0c15ed60 con 0x7fbf0c15aaa0 2023-10-14T16:56:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf13da9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c0707e0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf12346640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.871+0000 7fbf12346640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51500/0 (socket says 172.21.15.29:51500) 2023-10-14T16:56:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 -- 172.21.15.29:0/364055666 learned_addr learned my addr 172.21.15.29:0/364055666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf11b45640 1 --2- 172.21.15.29:0/364055666 >> v2:172.21.15.186:3300/0 conn(0x7fbf0c15aaa0 0x7fbf0c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf11344640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15ae80 0x7fbf0c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 -- 172.21.15.29:0/364055666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15ae80 msgr2=0x7fbf0c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:26.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15ae80 0x7fbf0c0cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 -- 172.21.15.29:0/364055666 >> v2:172.21.15.186:3300/0 conn(0x7fbf0c15aaa0 msgr2=0x7fbf0c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 --2- 172.21.15.29:0/364055666 >> v2:172.21.15.186:3300/0 conn(0x7fbf0c15aaa0 0x7fbf0c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.872+0000 7fbf12346640 1 -- 172.21.15.29:0/364055666 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0c15ef90 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.873+0000 7fbf12346640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0cd990 tx=0x7fbf0402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbdbf6965cc718e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.873+0000 7fbf10b43640 1 -- 172.21.15.29:0/364055666 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04031870 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf10b43640 1 -- 172.21.15.29:0/364055666 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf040319d0 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf10b43640 1 -- 172.21.15.29:0/364055666 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04031d60 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 -- 172.21.15.29:0/364055666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c0ccaa0 msgr2=0x7fbf0c070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0cd990 tx=0x7fbf0402b910 comp rx=0 tx=0).stop 2023-10-14T16:56:26.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 -- 172.21.15.29:0/364055666 shutdown_connections 2023-10-14T16:56:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c070f50 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 --2- 172.21.15.29:0/364055666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15ae80 0x7fbf0c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 --2- 172.21.15.29:0/364055666 >> v2:172.21.15.186:3300/0 conn(0x7fbf0c15aaa0 0x7fbf0c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 -- 172.21.15.29:0/364055666 >> 172.21.15.29:0/364055666 conn(0x7fbf0c0cb410 msgr2=0x7fbf0c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:26.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 -- 172.21.15.29:0/364055666 shutdown_connections 2023-10-14T16:56:26.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.874+0000 7fbf13da9640 1 -- 172.21.15.29:0/364055666 wait complete. 2023-10-14T16:56:26.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.876+0000 7fbf13da9640 1 Processor -- start 2023-10-14T16:56:26.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.876+0000 7fbf13da9640 1 -- start start 2023-10-14T16:56:26.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15aaa0 0x7fbf0c155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c1591f0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c159b50 con 0x7fbf0c0ccaa0 2023-10-14T16:56:26.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf13da9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0c070510 con 0x7fbf0c15aaa0 2023-10-14T16:56:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf12346640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf12346640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51506/0 (socket says 172.21.15.29:51506) 2023-10-14T16:56:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.877+0000 7fbf12346640 1 -- 172.21.15.29:0/239218963 learned_addr learned my addr 172.21.15.29:0/239218963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf11b45640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf11344640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15aaa0 0x7fbf0c155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 -- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15aaa0 msgr2=0x7fbf0c155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15aaa0 0x7fbf0c155bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 -- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf0c0ccaa0 msgr2=0x7fbf0c159610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:26.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c159610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 -- 172.21.15.29:0/239218963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040094d0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.878+0000 7fbf12346640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fbf0402b950 tx=0x7fbf040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.879+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0403ac60 con 0x7fbf0c15ae80 2023-10-14T16:56:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.879+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf0403adc0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.879+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf04045690 con 0x7fbf0c15ae80 2023-10-14T16:56:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.879+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf0c1582d0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.879+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf0c158850 con 0x7fbf0c15ae80 2023-10-14T16:56:26.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.880+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0c159330 con 0x7fbf0c15ae80 2023-10-14T16:56:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.881+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbf040457f0 con 0x7fbf0c15ae80 2023-10-14T16:56:26.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.883+0000 7fbefe7fc640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:26.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.883+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbf0402d040 con 0x7fbf0c15ae80 2023-10-14T16:56:26.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.883+0000 7fbf11b45640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:26.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.884+0000 7fbf11b45640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 0x7fbef006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf00006fd0 tx=0x7fbf00008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:26.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:26.887+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbf040defe0 con 0x7fbf0c15ae80 2023-10-14T16:56:27.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.083+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf0c04e160 con 0x7fbf0c15ae80 2023-10-14T16:56:27.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.086+0000 7fbefe7fc640 1 -- 172.21.15.29:0/239218963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16636 (secure 0 0 0) 0x7fbf04035090 con 0x7fbf0c15ae80 2023-10-14T16:56:27.087 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:27.087 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:26.104883+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 msgr2=0x7fbef006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 0x7fbef006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbf00006fd0 tx=0x7fbf00008040 comp rx=0 tx=0).stop 2023-10-14T16:56:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 msgr2=0x7fbf0c06ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fbf0402b950 tx=0x7fbf040046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 shutdown_connections 2023-10-14T16:56:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbef006af90 0x7fbef006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0c15ae80 0x7fbf0c06ccc0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0c15aaa0 0x7fbf0c155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 --2- 172.21.15.29:0/239218963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf0c0ccaa0 0x7fbf0c159610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 >> 172.21.15.29:0/239218963 conn(0x7fbf0c0cb410 msgr2=0x7fbf0c0b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 shutdown_connections 2023-10-14T16:56:27.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.088+0000 7fbf13da9640 1 -- 172.21.15.29:0/239218963 wait complete. 2023-10-14T16:56:27.107 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:27.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.328+0000 7fd077fff640 1 Processor -- start 2023-10-14T16:56:27.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.328+0000 7fd077fff640 1 -- start start 2023-10-14T16:56:27.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.329+0000 7fd077fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0780ad970 0x7fd0780cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.329+0000 7fd077fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07815b9d0 con 0x7fd07806c040 2023-10-14T16:56:27.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.329+0000 7fd077fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd07815bcb0 con 0x7fd0780ccfb0 2023-10-14T16:56:27.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.329+0000 7fd077fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07815bf90 con 0x7fd0780ad970 2023-10-14T16:56:27.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0780ad970 0x7fd0780cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd0780ad970 0x7fd0780cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51520/0 (socket says 172.21.15.29:51520) 2023-10-14T16:56:27.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 -- 172.21.15.29:0/163855820 learned_addr learned my addr 172.21.15.29:0/163855820 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:27.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 -- 172.21.15.29:0/163855820 >> v1:172.21.15.186:6789/0 conn(0x7fd0780ccfb0 legacy=0x7fd0780cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:27.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 -- 172.21.15.29:0/163855820 >> v1:172.21.15.29:6789/0 conn(0x7fd07806c040 legacy=0x7fd07806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.330+0000 7fd0767fc640 1 -- 172.21.15.29:0/163855820 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780cc270 con 0x7fd0780ad970 2023-10-14T16:56:27.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.331+0000 7fd0767fc640 1 --2- 172.21.15.29:0/163855820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780cca70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c0162702dc81f87 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.331+0000 7fd075ffb640 1 -- 172.21.15.29:0/163855820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c030840 con 0x7fd0780ad970 2023-10-14T16:56:27.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd075ffb640 1 -- 172.21.15.29:0/163855820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c0309a0 con 0x7fd0780ad970 2023-10-14T16:56:27.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd075ffb640 1 -- 172.21.15.29:0/163855820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c030d30 con 0x7fd0780ad970 2023-10-14T16:56:27.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 -- 172.21.15.29:0/163855820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 msgr2=0x7fd0780cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 --2- 172.21.15.29:0/163855820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780cca70 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7fd06c0098a0 tx=0x7fd06c02b980 comp rx=0 tx=0).stop 2023-10-14T16:56:27.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 -- 172.21.15.29:0/163855820 shutdown_connections 2023-10-14T16:56:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 --2- 172.21.15.29:0/163855820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780cca70 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 -- 172.21.15.29:0/163855820 >> 172.21.15.29:0/163855820 conn(0x7fd0780ce6a0 msgr2=0x7fd0780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 -- 172.21.15.29:0/163855820 shutdown_connections 2023-10-14T16:56:27.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.332+0000 7fd077fff640 1 -- 172.21.15.29:0/163855820 wait complete. 2023-10-14T16:56:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.334+0000 7fd077fff640 1 Processor -- start 2023-10-14T16:56:27.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.334+0000 7fd077fff640 1 -- start start 2023-10-14T16:56:27.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 0x7fd07815a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0780ccfb0 0x7fd078159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780af530 con 0x7fd0780ad970 2023-10-14T16:56:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07815ad90 con 0x7fd07806c040 2023-10-14T16:56:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd077fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0780b0960 con 0x7fd0780ccfb0 2023-10-14T16:56:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd0767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd0767fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51530/0 (socket says 172.21.15.29:51530) 2023-10-14T16:56:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.335+0000 7fd0767fc640 1 -- 172.21.15.29:0/2428802797 learned_addr learned my addr 172.21.15.29:0/2428802797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd076ffd640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 0x7fd07815a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 -- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0780ccfb0 msgr2=0x7fd078159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0777fe640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0780ccfb0 0x7fd078159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0780ccfb0 0x7fd078159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 -- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 msgr2=0x7fd07815a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 0x7fd07815a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 -- 172.21.15.29:0/2428802797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06c0094d0 con 0x7fd0780ad970 2023-10-14T16:56:27.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd076ffd640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 0x7fd07815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.336+0000 7fd0767fc640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd06c000c00 tx=0x7fd06c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.337+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c039e20 con 0x7fd0780ad970 2023-10-14T16:56:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.337+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd06c043400 con 0x7fd0780ad970 2023-10-14T16:56:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.337+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd06c0437b0 con 0x7fd0780ad970 2023-10-14T16:56:27.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.337+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0780b0c40 con 0x7fd0780ad970 2023-10-14T16:56:27.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.337+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0780af150 con 0x7fd0780ad970 2023-10-14T16:56:27.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.338+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07806b330 con 0x7fd0780ad970 2023-10-14T16:56:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.339+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd06c043910 con 0x7fd0780ad970 2023-10-14T16:56:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.340+0000 7fd05f7fe640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 0x7fd06406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.340+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fd06c03e070 con 0x7fd0780ad970 2023-10-14T16:56:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.340+0000 7fd076ffd640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 0x7fd06406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.341+0000 7fd076ffd640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 0x7fd06406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0780702e0 tx=0x7fd068009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.344+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd06c077990 con 0x7fd0780ad970 2023-10-14T16:56:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.539+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0780af670 con 0x7fd0780ad970 2023-10-14T16:56:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.542+0000 7fd05f7fe640 1 -- 172.21.15.29:0/2428802797 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16636 (secure 0 0 0) 0x7fd0780af670 con 0x7fd0780ad970 2023-10-14T16:56:27.544 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:27.544 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":81,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:26.104883+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:27.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 msgr2=0x7fd06406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 0x7fd06406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0780702e0 tx=0x7fd068009040 comp rx=0 tx=0).stop 2023-10-14T16:56:27.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 msgr2=0x7fd0780b0280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7fd06c000c00 tx=0x7fd06c004730 comp rx=0 tx=0).stop 2023-10-14T16:56:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 shutdown_connections 2023-10-14T16:56:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fd06406af90 0x7fd06406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0780ccfb0 0x7fd078159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780ad970 0x7fd0780b0280 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 --2- 172.21.15.29:0/2428802797 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07806c040 0x7fd07815a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 >> 172.21.15.29:0/2428802797 conn(0x7fd0780ce6a0 msgr2=0x7fd07806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.545+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 shutdown_connections 2023-10-14T16:56:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.546+0000 7fd077fff640 1 -- 172.21.15.29:0/2428802797 wait complete. 2023-10-14T16:56:27.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:56:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.790+0000 7fdaf9a9f640 1 Processor -- start 2023-10-14T16:56:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.790+0000 7fdaf9a9f640 1 -- start start 2023-10-14T16:56:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf9a9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdaf406c040 0x7fdaf406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf9a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf9a9f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf406c960 con 0x7fdaf415ebc0 2023-10-14T16:56:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf9a9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf415b9d0 con 0x7fdaf406c040 2023-10-14T16:56:27.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf9a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415bcb0 con 0x7fdaf40ad970 2023-10-14T16:56:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf8a9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdaf406c040 0x7fdaf406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf8a9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdaf406c040 0x7fdaf406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51536/0 (socket says 172.21.15.29:51536) 2023-10-14T16:56:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3764413449 learned_addr learned my addr 172.21.15.29:0/3764413449 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.792+0000 7fdaf929e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdaf415ebc0 0x7fdaf40cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56826/0 (socket says 172.21.15.29:56826) 2023-10-14T16:56:27.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf3fff640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3764413449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 msgr2=0x7fdaf415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf8a9d640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3764413449 >> v1:172.21.15.29:6789/0 conn(0x7fdaf415ebc0 legacy=0x7fdaf40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3764413449 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf415bf90 con 0x7fdaf406c040 2023-10-14T16:56:27.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.793+0000 7fdaf37fe640 1 -- 172.21.15.29:0/3764413449 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860485667 0 0) 0x7fdaf406c960 con 0x7fdaf415ebc0 2023-10-14T16:56:27.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.794+0000 7fdaf8a9d640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf406c420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdae80098a0 tx=0x7fdae802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a5196e85be9a0d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.795+0000 7fdaf37fe640 1 -- 172.21.15.29:0/3764413449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8030840 con 0x7fdaf406c040 2023-10-14T16:56:27.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.795+0000 7fdaf37fe640 1 -- 172.21.15.29:0/3764413449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae80309a0 con 0x7fdaf406c040 2023-10-14T16:56:27.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.795+0000 7fdaf37fe640 1 -- 172.21.15.29:0/3764413449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae8030d30 con 0x7fdaf406c040 2023-10-14T16:56:27.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.795+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3764413449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 msgr2=0x7fdaf406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.795+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf406c420 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7fdae80098a0 tx=0x7fdae802b980 comp rx=0 tx=0).stop 2023-10-14T16:56:27.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3764413449 shutdown_connections 2023-10-14T16:56:27.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3764413449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf406c420 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3764413449 >> 172.21.15.29:0/3764413449 conn(0x7fdaf40ce6a0 msgr2=0x7fdaf40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3764413449 shutdown_connections 2023-10-14T16:56:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.796+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3764413449 wait complete. 2023-10-14T16:56:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.798+0000 7fdaf9a9f640 1 Processor -- start 2023-10-14T16:56:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.798+0000 7fdaf9a9f640 1 -- start start 2023-10-14T16:56:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf40b26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 0x7fdaf4152750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf415ab50 con 0x7fdaf406c040 2023-10-14T16:56:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40af250 con 0x7fdaf415ebc0 2023-10-14T16:56:27.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.799+0000 7fdaf9a9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf40c6860 con 0x7fdaf40ad970 2023-10-14T16:56:27.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51548/0 (socket says 172.21.15.29:51548) 2023-10-14T16:56:27.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3688784268 learned_addr learned my addr 172.21.15.29:0/3688784268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:27.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf3fff640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf40b26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf929e640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 0x7fdaf4152750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 msgr2=0x7fdaf40b26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf40b26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 msgr2=0x7fdaf4152750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:27.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 0x7fdaf4152750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:27.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf8a9d640 1 -- 172.21.15.29:0/3688784268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae80094d0 con 0x7fdaf406c040 2023-10-14T16:56:27.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.800+0000 7fdaf929e640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 0x7fdaf4152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:27.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf8a9d640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000c00 tx=0x7fdae80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae803e070 con 0x7fdaf406c040 2023-10-14T16:56:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae8004030 con 0x7fdaf406c040 2023-10-14T16:56:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdae80043c0 con 0x7fdaf406c040 2023-10-14T16:56:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf40c6b40 con 0x7fdaf406c040 2023-10-14T16:56:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.801+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf40b2bf0 con 0x7fdaf406c040 2023-10-14T16:56:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.803+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf406b330 con 0x7fdaf406c040 2023-10-14T16:56:27.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.803+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdae8042070 con 0x7fdaf406c040 2023-10-14T16:56:27.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.804+0000 7fdaf17fa640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 0x7fdad806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:27.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.804+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fdae80ab720 con 0x7fdaf406c040 2023-10-14T16:56:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.805+0000 7fdaf3fff640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 0x7fdad806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.806+0000 7fdaf3fff640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 0x7fdad806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdae0006fd0 tx=0x7fdae0008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:27.810+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdae8076580 con 0x7fdaf406c040 2023-10-14T16:56:28.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.050+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdaf40c6d80 con 0x7fdaf406c040 2023-10-14T16:56:28.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.663+0000 7fdaf17fa640 1 -- 172.21.15.29:0/3688784268 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v31) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdaf40c6d80 con 0x7fdaf406c040 2023-10-14T16:56:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 msgr2=0x7fdad806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 0x7fdad806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fdae0006fd0 tx=0x7fdae0008040 comp rx=0 tx=0).stop 2023-10-14T16:56:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 msgr2=0x7fdaf4153310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7fdae8000c00 tx=0x7fdae80046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 shutdown_connections 2023-10-14T16:56:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fdad806af90 0x7fdad806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdaf415ebc0 0x7fdaf4152750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdaf40ad970 0x7fdaf40b26b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 --2- 172.21.15.29:0/3688784268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdaf406c040 0x7fdaf4153310 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.667+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 >> 172.21.15.29:0/3688784268 conn(0x7fdaf40ce6a0 msgr2=0x7fdaf406a380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:28.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.668+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 shutdown_connections 2023-10-14T16:56:28.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.668+0000 7fdaf9a9f640 1 -- 172.21.15.29:0/3688784268 wait complete. 2023-10-14T16:56:28.690 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:56:28.690 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:56:28.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.907+0000 7f85af59e640 1 Processor -- start 2023-10-14T16:56:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.907+0000 7f85af59e640 1 -- start start 2023-10-14T16:56:28.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f85b00ad970 0x7f85b015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85b015ebc0 0x7f85b015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b006c960 con 0x7f85b015ebc0 2023-10-14T16:56:28.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b015b9d0 con 0x7f85b00ad970 2023-10-14T16:56:28.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.908+0000 7f85af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b015bcb0 con 0x7f85b006c040 2023-10-14T16:56:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85b015ebc0 0x7f85b015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f85b015ebc0 0x7f85b015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51560/0 (socket says 172.21.15.29:51560) 2023-10-14T16:56:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 -- 172.21.15.29:0/699001531 learned_addr learned my addr 172.21.15.29:0/699001531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:28.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85a7fff640 1 --2- 172.21.15.29:0/699001531 >> v2:172.21.15.186:3300/0 conn(0x7f85b00ad970 0x7f85b015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85ae59c640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 -- 172.21.15.29:0/699001531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 msgr2=0x7f85b006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 -- 172.21.15.29:0/699001531 >> v2:172.21.15.186:3300/0 conn(0x7f85b00ad970 msgr2=0x7f85b015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:28.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 --2- 172.21.15.29:0/699001531 >> v2:172.21.15.186:3300/0 conn(0x7f85b00ad970 0x7f85b015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.909+0000 7f85aed9d640 1 -- 172.21.15.29:0/699001531 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b015bf90 con 0x7f85b015ebc0 2023-10-14T16:56:28.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.910+0000 7f85aed9d640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b015f050 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e50 tx=0x7f85a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e4add2b10d191cd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85add9b640 1 -- 172.21.15.29:0/699001531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a802ca70 con 0x7f85b015ebc0 2023-10-14T16:56:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85add9b640 1 -- 172.21.15.29:0/699001531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a802cbd0 con 0x7f85b015ebc0 2023-10-14T16:56:28.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85add9b640 1 -- 172.21.15.29:0/699001531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a80343f0 con 0x7f85b015ebc0 2023-10-14T16:56:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 -- 172.21.15.29:0/699001531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 msgr2=0x7f85b015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b015f050 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009e50 tx=0x7f85a802bed0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 -- 172.21.15.29:0/699001531 shutdown_connections 2023-10-14T16:56:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b015f050 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 --2- 172.21.15.29:0/699001531 >> v2:172.21.15.186:3300/0 conn(0x7f85b00ad970 0x7f85b015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 --2- 172.21.15.29:0/699001531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.911+0000 7f85af59e640 1 -- 172.21.15.29:0/699001531 >> 172.21.15.29:0/699001531 conn(0x7f85b00ce6a0 msgr2=0x7f85b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.912+0000 7f85af59e640 1 -- 172.21.15.29:0/699001531 shutdown_connections 2023-10-14T16:56:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.912+0000 7f85af59e640 1 -- 172.21.15.29:0/699001531 wait complete. 2023-10-14T16:56:28.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.913+0000 7f85af59e640 1 Processor -- start 2023-10-14T16:56:28.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.914+0000 7f85af59e640 1 -- start start 2023-10-14T16:56:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.914+0000 7f85af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b00b17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.914+0000 7f85af59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00ad970 0x7f85b0154520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85af59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85af59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b0154a60 con 0x7f85b015ebc0 2023-10-14T16:56:28.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85af59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b01f0390 con 0x7f85b00ad970 2023-10-14T16:56:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b00c7c50 con 0x7f85b006c040 2023-10-14T16:56:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51576/0 (socket says 172.21.15.29:51576) 2023-10-14T16:56:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 -- 172.21.15.29:0/2767649160 learned_addr learned my addr 172.21.15.29:0/2767649160 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:28.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85a7fff640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00ad970 0x7f85b0154520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85ae59c640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b00b17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 -- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 msgr2=0x7f85b00b17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b00b17b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 -- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00ad970 msgr2=0x7f85b0154520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00ad970 0x7f85b0154520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:28.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.915+0000 7f85aed9d640 1 -- 172.21.15.29:0/2767649160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a8009b30 con 0x7f85b015ebc0 2023-10-14T16:56:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85aed9d640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009f70 tx=0x7f85a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8040670 con 0x7f85b015ebc0 2023-10-14T16:56:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8040810 con 0x7f85b015ebc0 2023-10-14T16:56:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85a8040bc0 con 0x7f85b015ebc0 2023-10-14T16:56:28.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b00c7f30 con 0x7f85b015ebc0 2023-10-14T16:56:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.916+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b00af2e0 con 0x7f85b015ebc0 2023-10-14T16:56:28.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.918+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b006b330 con 0x7f85b015ebc0 2023-10-14T16:56:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.919+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85a8040d20 con 0x7f85b015ebc0 2023-10-14T16:56:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.919+0000 7f85a6ffd640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 0x7f859006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.920+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f85a80ac790 con 0x7f85b015ebc0 2023-10-14T16:56:28.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.920+0000 7f85ae59c640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 0x7f859006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.920+0000 7f85ae59c640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 0x7f859006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85a000b740 tx=0x7f85a0008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:28.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:28.924+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85a80777c0 con 0x7f85b015ebc0 2023-10-14T16:56:29.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.166+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f85b00af140 con 0x7f85b015ebc0 2023-10-14T16:56:29.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.666+0000 7f85a6ffd640 1 -- 172.21.15.29:0/2767649160 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v32) v1 ==== 122+0+0 (secure 0 0 0) 0x7f85b00af140 con 0x7f85b015ebc0 2023-10-14T16:56:29.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 msgr2=0x7f859006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 0x7f859006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f85a000b740 tx=0x7f85a0008040 comp rx=0 tx=0).stop 2023-10-14T16:56:29.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 msgr2=0x7f85b00b3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f85a8009f70 tx=0x7f85a8004a40 comp rx=0 tx=0).stop 2023-10-14T16:56:29.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 shutdown_connections 2023-10-14T16:56:29.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.670+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f859006af90 0x7f859006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b015ebc0 0x7f85b00b3530 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b00ad970 0x7f85b0154520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 --2- 172.21.15.29:0/2767649160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b006c040 0x7f85b00b17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 >> 172.21.15.29:0/2767649160 conn(0x7f85b00ce6a0 msgr2=0x7f85b0072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:29.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 shutdown_connections 2023-10-14T16:56:29.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.671+0000 7f85af59e640 1 -- 172.21.15.29:0/2767649160 wait complete. 2023-10-14T16:56:29.694 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:56:29.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:29.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.922+0000 7fcbcaccf640 1 Processor -- start 2023-10-14T16:56:29.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.923+0000 7fcbcaccf640 1 -- start start 2023-10-14T16:56:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.924+0000 7fcbcaccf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbc415ae80 0x7fcbc415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:29.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.924+0000 7fcbcaccf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc415ed60 con 0x7fcbc40ce390 2023-10-14T16:56:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.924+0000 7fcbcaccf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbc40707e0 con 0x7fcbc415aaa0 2023-10-14T16:56:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.924+0000 7fcbcaccf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbc406b810 con 0x7fcbc415ae80 2023-10-14T16:56:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbc415ae80 0x7fcbc415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbc415ae80 0x7fcbc415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51588/0 (socket says 172.21.15.29:51588) 2023-10-14T16:56:29.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 -- 172.21.15.29:0/1282138036 learned_addr learned my addr 172.21.15.29:0/1282138036 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:29.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbc926c640 1 --1- 172.21.15.29:0/1282138036 >> v1:172.21.15.29:6789/0 conn(0x7fcbc40ce390 0x7fcbc4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56842/0 (socket says 172.21.15.29:56842) 2023-10-14T16:56:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 -- 172.21.15.29:0/1282138036 >> v1:172.21.15.186:6789/0 conn(0x7fcbc415aaa0 legacy=0x7fcbc40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 -- 172.21.15.29:0/1282138036 >> v1:172.21.15.29:6789/0 conn(0x7fcbc40ce390 legacy=0x7fcbc4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.925+0000 7fcbbbfff640 1 -- 172.21.15.29:0/1282138036 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbc415ef90 con 0x7fcbc415ae80 2023-10-14T16:56:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbbbfff640 1 --2- 172.21.15.29:0/1282138036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415ae80 0x7fcbc415e820 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009870 tx=0x7fcbb402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11726c1a1864e110 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:29.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbbb7fe640 1 -- 172.21.15.29:0/1282138036 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402f850 con 0x7fcbc415ae80 2023-10-14T16:56:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbbb7fe640 1 -- 172.21.15.29:0/1282138036 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb402f9b0 con 0x7fcbc415ae80 2023-10-14T16:56:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbbb7fe640 1 -- 172.21.15.29:0/1282138036 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402fd40 con 0x7fcbc415ae80 2023-10-14T16:56:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbcaccf640 1 -- 172.21.15.29:0/1282138036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415ae80 msgr2=0x7fcbc415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.927+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/1282138036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415ae80 0x7fcbc415e820 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4009870 tx=0x7fcbb402ba30 comp rx=0 tx=0).stop 2023-10-14T16:56:29.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.928+0000 7fcbcaccf640 1 -- 172.21.15.29:0/1282138036 shutdown_connections 2023-10-14T16:56:29.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.928+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/1282138036 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415ae80 0x7fcbc415e820 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.928+0000 7fcbcaccf640 1 -- 172.21.15.29:0/1282138036 >> 172.21.15.29:0/1282138036 conn(0x7fcbc40cb410 msgr2=0x7fcbc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:29.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.928+0000 7fcbcaccf640 1 -- 172.21.15.29:0/1282138036 shutdown_connections 2023-10-14T16:56:29.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.928+0000 7fcbcaccf640 1 -- 172.21.15.29:0/1282138036 wait complete. 2023-10-14T16:56:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.929+0000 7fcbcaccf640 1 Processor -- start 2023-10-14T16:56:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.930+0000 7fcbcaccf640 1 -- start start 2023-10-14T16:56:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.930+0000 7fcbcaccf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 0x7fcbc41f5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:29.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.930+0000 7fcbcaccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.930+0000 7fcbcaccf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 0x7fcbc40cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbcaccf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41f6f00 con 0x7fcbc415aaa0 2023-10-14T16:56:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbcaccf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41f6380 con 0x7fcbc415ae80 2023-10-14T16:56:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbcaccf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbc41f6660 con 0x7fcbc40ce390 2023-10-14T16:56:29.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51604/0 (socket says 172.21.15.29:51604) 2023-10-14T16:56:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 -- 172.21.15.29:0/3223480034 learned_addr learned my addr 172.21.15.29:0/3223480034 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbc926c640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 0x7fcbc40cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbc8a6b640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 0x7fcbc41f5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:29.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 -- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 msgr2=0x7fcbc41f5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 0x7fcbc41f5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 -- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 msgr2=0x7fcbc40cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 0x7fcbc40cc4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbbbfff640 1 -- 172.21.15.29:0/3223480034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb40094d0 con 0x7fcbc415aaa0 2023-10-14T16:56:29.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbc8a6b640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 0x7fcbc41f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.931+0000 7fcbc926c640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 0x7fcbc40cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbbbfff640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb402f820 con 0x7fcbc415aaa0 2023-10-14T16:56:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbb4041400 con 0x7fcbc415aaa0 2023-10-14T16:56:29.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb40417d0 con 0x7fcbc415aaa0 2023-10-14T16:56:29.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbc41f6940 con 0x7fcbc415aaa0 2023-10-14T16:56:29.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.932+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbc41f5fe0 con 0x7fcbc415aaa0 2023-10-14T16:56:29.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.933+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbc404e160 con 0x7fcbc415aaa0 2023-10-14T16:56:29.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.934+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcbb4041930 con 0x7fcbc415aaa0 2023-10-14T16:56:29.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.935+0000 7fcbb97fa640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 0x7fcbb006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:29.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.935+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fcbb403c070 con 0x7fcbc415aaa0 2023-10-14T16:56:29.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.935+0000 7fcbc8a6b640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 0x7fcbb006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:29.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.936+0000 7fcbc8a6b640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 0x7fcbb006d4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40ce070 tx=0x7fcbc0009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:29.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:29.939+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbb4077630 con 0x7fcbc415aaa0 2023-10-14T16:56:30.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.171+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbc40cdc60 con 0x7fcbc415aaa0 2023-10-14T16:56:30.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.173+0000 7fcbb97fa640 1 -- 172.21.15.29:0/3223480034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fcbb407ac50 con 0x7fcbc415aaa0 2023-10-14T16:56:30.174 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:30.174 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:30.174 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:30.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 msgr2=0x7fcbb006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 0x7fcbb006d4a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fcbc40ce070 tx=0x7fcbc0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 msgr2=0x7fcbc40c27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fcbb4000c00 tx=0x7fcbb4004730 comp rx=0 tx=0).stop 2023-10-14T16:56:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 shutdown_connections 2023-10-14T16:56:30.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fcbb006afe0 0x7fcbb006d4a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbc415ae80 0x7fcbc40cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbc415aaa0 0x7fcbc40c27c0 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 --2- 172.21.15.29:0/3223480034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbc40ce390 0x7fcbc41f5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.175+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 >> 172.21.15.29:0/3223480034 conn(0x7fcbc40cb410 msgr2=0x7fcbc40b29d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:30.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.176+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 shutdown_connections 2023-10-14T16:56:30.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.176+0000 7fcbcaccf640 1 -- 172.21.15.29:0/3223480034 wait complete. 2023-10-14T16:56:30.196 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.416+0000 7ff1f29d2640 1 Processor -- start 2023-10-14T16:56:30.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.417+0000 7ff1f29d2640 1 -- start start 2023-10-14T16:56:30.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.418+0000 7ff1f29d2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.418+0000 7ff1f29d2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 0x7ff1ec0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.418+0000 7ff1f29d2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ec15d150 con 0x7ff1ec15cd70 2023-10-14T16:56:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.418+0000 7ff1f29d2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0d1180 con 0x7ff1ec0b8480 2023-10-14T16:56:30.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.418+0000 7ff1f29d2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1ec0d1460 con 0x7ff1ec0d0be0 2023-10-14T16:56:30.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51612/0 (socket says 172.21.15.29:51612) 2023-10-14T16:56:30.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 -- 172.21.15.29:0/3284863442 learned_addr learned my addr 172.21.15.29:0/3284863442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:30.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f21d1640 1 --2- 172.21.15.29:0/3284863442 >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 0x7ff1ec0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 -- 172.21.15.29:0/3284863442 >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 msgr2=0x7ff1ec0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:30.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 --2- 172.21.15.29:0/3284863442 >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 0x7ff1ec0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 -- 172.21.15.29:0/3284863442 >> v1:172.21.15.186:6789/0 conn(0x7ff1ec15cd70 legacy=0x7ff1ec0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:30.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f11cf640 1 -- 172.21.15.29:0/3284863442 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ec0d1740 con 0x7ff1ec0b8480 2023-10-14T16:56:30.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.419+0000 7ff1f21d1640 1 --2- 172.21.15.29:0/3284863442 >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 0x7ff1ec0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:30.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.420+0000 7ff1f11cf640 1 --2- 172.21.15.29:0/3284863442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0b8910 tx=0x7ff1dc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc422d43c84daff4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f09ce640 1 -- 172.21.15.29:0/3284863442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc0317e0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f09ce640 1 -- 172.21.15.29:0/3284863442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1dc031940 con 0x7ff1ec0b8480 2023-10-14T16:56:30.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f09ce640 1 -- 172.21.15.29:0/3284863442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1dc031cd0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 -- 172.21.15.29:0/3284863442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 msgr2=0x7ff1ec0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/3284863442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0b8910 tx=0x7ff1dc02b880 comp rx=0 tx=0).stop 2023-10-14T16:56:30.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 -- 172.21.15.29:0/3284863442 shutdown_connections 2023-10-14T16:56:30.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/3284863442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec0cd7c0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/3284863442 >> v2:172.21.15.186:3300/0 conn(0x7ff1ec0d0be0 0x7ff1ec0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.421+0000 7ff1f29d2640 1 -- 172.21.15.29:0/3284863442 >> 172.21.15.29:0/3284863442 conn(0x7ff1ec0cf570 msgr2=0x7ff1ec0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:30.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.422+0000 7ff1f29d2640 1 -- 172.21.15.29:0/3284863442 shutdown_connections 2023-10-14T16:56:30.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.422+0000 7ff1f29d2640 1 -- 172.21.15.29:0/3284863442 wait complete. 2023-10-14T16:56:30.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.424+0000 7ff1f29d2640 1 Processor -- start 2023-10-14T16:56:30.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.424+0000 7ff1f29d2640 1 -- start start 2023-10-14T16:56:30.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1ec0d0be0 0x7ff1ec15af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ec15cd70 0x7ff1ec0cbe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec0c9bc0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1482a0 con 0x7ff1ec15cd70 2023-10-14T16:56:30.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f29d2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec15e8c0 con 0x7ff1ec0d0be0 2023-10-14T16:56:30.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.425+0000 7ff1f19d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51618/0 (socket says 172.21.15.29:51618) 2023-10-14T16:56:30.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 -- 172.21.15.29:0/599278508 learned_addr learned my addr 172.21.15.29:0/599278508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:30.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f21d1640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ec15cd70 0x7ff1ec0cbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f11cf640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1ec0d0be0 0x7ff1ec15af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 -- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1ec0d0be0 msgr2=0x7ff1ec15af80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1ec0d0be0 0x7ff1ec15af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 -- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ec15cd70 msgr2=0x7ff1ec0cbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ec15cd70 0x7ff1ec0cbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 -- 172.21.15.29:0/599278508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc0094d0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.426+0000 7ff1f19d0640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c7c0 tx=0x7ff1e800cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.427+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800aa50 con 0x7ff1ec0b8480 2023-10-14T16:56:30.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.427+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e800abb0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.427+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8012420 con 0x7ff1ec0b8480 2023-10-14T16:56:30.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.427+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec15eba0 con 0x7ff1ec0b8480 2023-10-14T16:56:30.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.427+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec15f160 con 0x7ff1ec0b8480 2023-10-14T16:56:30.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.428+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec15f350 con 0x7ff1ec0b8480 2023-10-14T16:56:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.429+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1e8012580 con 0x7ff1ec0b8480 2023-10-14T16:56:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.430+0000 7ff1e27fc640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 0x7ff1d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.430+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff1e8089800 con 0x7ff1ec0b8480 2023-10-14T16:56:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.431+0000 7ff1f11cf640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 0x7ff1d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.431+0000 7ff1f11cf640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 0x7ff1d006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc036000 tx=0x7ff1dc03d040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.435+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1ec15f350 con 0x7ff1ec0b8480 2023-10-14T16:56:30.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.630+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1ec15f350 con 0x7ff1ec0b8480 2023-10-14T16:56:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.632+0000 7ff1e27fc640 1 -- 172.21.15.29:0/599278508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+16623 (secure 0 0 0) 0x7ff1ec15f350 con 0x7ff1ec0b8480 2023-10-14T16:56:30.633 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:30.633 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":82,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:27.632523+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 msgr2=0x7ff1d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 0x7ff1d006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc036000 tx=0x7ff1dc03d040 comp rx=0 tx=0).stop 2023-10-14T16:56:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 msgr2=0x7ff1ec15ff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c7c0 tx=0x7ff1e800cc80 comp rx=0 tx=0).stop 2023-10-14T16:56:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 shutdown_connections 2023-10-14T16:56:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d006af90 0x7ff1d006d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1ec15cd70 0x7ff1ec0cbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1ec0d0be0 0x7ff1ec15af80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 --2- 172.21.15.29:0/599278508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1ec0b8480 0x7ff1ec15ff00 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 >> 172.21.15.29:0/599278508 conn(0x7ff1ec0cf570 msgr2=0x7ff1ec0b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.634+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 shutdown_connections 2023-10-14T16:56:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.635+0000 7ff1f29d2640 1 -- 172.21.15.29:0/599278508 wait complete. 2023-10-14T16:56:30.654 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:56:30.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:56:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.876+0000 7f82b2447640 1 Processor -- start 2023-10-14T16:56:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.876+0000 7f82b2447640 1 -- start start 2023-10-14T16:56:30.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b2447640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82ac0b4c80 0x7f82ac167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b2447640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0d15a0 con 0x7f82ac15cbe0 2023-10-14T16:56:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b2447640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82ac0d1880 con 0x7f82ac15cfc0 2023-10-14T16:56:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b2447640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82ac0d1b60 con 0x7f82ac0b4c80 2023-10-14T16:56:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b1445640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f82ac15cbe0 0x7f82ac0d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56852/0 (socket says 172.21.15.29:56852) 2023-10-14T16:56:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b1445640 1 -- 172.21.15.29:0/4210921169 learned_addr learned my addr 172.21.15.29:0/4210921169 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:30.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f82b1c46640 1 --2- 172.21.15.29:0/4210921169 >> v2:172.21.15.29:3300/0 conn(0x7f82ac0b4c80 0x7f82ac167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.878+0000 7f829ffff640 1 -- 172.21.15.29:0/4210921169 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708465296 0 0) 0x7f82ac0d15a0 con 0x7f82ac15cbe0 2023-10-14T16:56:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.879+0000 7f829ffff640 1 -- 172.21.15.29:0/4210921169 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8298003520 con 0x7f82ac15cbe0 2023-10-14T16:56:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.879+0000 7f82b1c46640 1 -- 172.21.15.29:0/4210921169 >> v1:172.21.15.186:6789/0 conn(0x7f82ac15cfc0 legacy=0x7f82ac0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.879+0000 7f82b1c46640 1 -- 172.21.15.29:0/4210921169 >> v1:172.21.15.29:6789/0 conn(0x7f82ac15cbe0 legacy=0x7f82ac0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.879+0000 7f82b1c46640 1 -- 172.21.15.29:0/4210921169 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82ac0d1e40 con 0x7f82ac0b4c80 2023-10-14T16:56:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f82b1c46640 1 --2- 172.21.15.29:0/4210921169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac0b4c80 0x7f82ac167910 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009d90 tx=0x7f82a402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903640d2f2dd1576 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f829ffff640 1 -- 172.21.15.29:0/4210921169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a402cc60 con 0x7f82ac0b4c80 2023-10-14T16:56:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f829ffff640 1 -- 172.21.15.29:0/4210921169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a402cdc0 con 0x7f82ac0b4c80 2023-10-14T16:56:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f829ffff640 1 -- 172.21.15.29:0/4210921169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4034660 con 0x7f82ac0b4c80 2023-10-14T16:56:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f82b2447640 1 -- 172.21.15.29:0/4210921169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac0b4c80 msgr2=0x7f82ac167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.880+0000 7f82b2447640 1 --2- 172.21.15.29:0/4210921169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac0b4c80 0x7f82ac167910 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009d90 tx=0x7f82a402beb0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.881+0000 7f82b2447640 1 -- 172.21.15.29:0/4210921169 shutdown_connections 2023-10-14T16:56:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.881+0000 7f82b2447640 1 --2- 172.21.15.29:0/4210921169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac0b4c80 0x7f82ac167910 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.881+0000 7f82b2447640 1 -- 172.21.15.29:0/4210921169 >> 172.21.15.29:0/4210921169 conn(0x7f82ac0cf570 msgr2=0x7f82ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.881+0000 7f82b2447640 1 -- 172.21.15.29:0/4210921169 shutdown_connections 2023-10-14T16:56:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.881+0000 7f82b2447640 1 -- 172.21.15.29:0/4210921169 wait complete. 2023-10-14T16:56:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.882+0000 7f82b2447640 1 Processor -- start 2023-10-14T16:56:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.883+0000 7f82b2447640 1 -- start start 2023-10-14T16:56:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.883+0000 7f82b2447640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82ac0b4c80 0x7f82ac159540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.883+0000 7f82b2447640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 0x7f82ac0a9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b2447640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b2447640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac159a80 con 0x7f82ac15cfc0 2023-10-14T16:56:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b2447640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0caa70 con 0x7f82ac15cbe0 2023-10-14T16:56:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b2447640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82ac0c9bc0 con 0x7f82ac0b4c80 2023-10-14T16:56:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51636/0 (socket says 172.21.15.29:51636) 2023-10-14T16:56:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 -- 172.21.15.29:0/1182073384 learned_addr learned my addr 172.21.15.29:0/1182073384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b0c44640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 0x7f82ac0a9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 -- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82ac0b4c80 msgr2=0x7f82ac159540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82ac0b4c80 0x7f82ac159540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 -- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 msgr2=0x7f82ac0a9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 0x7f82ac0a9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b1c46640 1 -- 172.21.15.29:0/1182073384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82a4009a70 con 0x7f82ac15cfc0 2023-10-14T16:56:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.884+0000 7f82b0c44640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 0x7f82ac0a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f82b1c46640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009eb0 tx=0x7f82a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a4009450 con 0x7f82ac15cfc0 2023-10-14T16:56:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82a403f420 con 0x7f82ac15cfc0 2023-10-14T16:56:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82a403f7b0 con 0x7f82ac15cfc0 2023-10-14T16:56:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82ac0c9e60 con 0x7f82ac15cfc0 2023-10-14T16:56:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.885+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82ac0ca420 con 0x7f82ac15cfc0 2023-10-14T16:56:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.887+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82ac04e9b0 con 0x7f82ac15cfc0 2023-10-14T16:56:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.887+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f82a403f910 con 0x7f82ac15cfc0 2023-10-14T16:56:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.888+0000 7f829dffb640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 0x7f829006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.888+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f82a40abe90 con 0x7f82ac15cfc0 2023-10-14T16:56:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.889+0000 7f82b1445640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 0x7f829006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.889+0000 7f82b1445640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 0x7f829006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000f60 tx=0x7f82a80078f0 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:30.892+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f82a4076e90 con 0x7f82ac15cfc0 2023-10-14T16:56:31.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.082+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 --> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82ac0d1260 con 0x7f829006af90 2023-10-14T16:56:31.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.093+0000 7f829dffb640 1 -- 172.21.15.29:0/1182073384 <== mgr.6252 v2:172.21.15.29:6832/3108857887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286440 (secure 0 0 0) 0x7f82a8005890 con 0x7f829006af90 2023-10-14T16:56:31.095 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:31.099 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:56:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 msgr2=0x7f829006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 0x7f829006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f82a8000f60 tx=0x7f82a80078f0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 msgr2=0x7f82ac0aa500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f82a4009eb0 tx=0x7f82a4004a40 comp rx=0 tx=0).stop 2023-10-14T16:56:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 shutdown_connections 2023-10-14T16:56:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f829006af90 0x7f829006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82ac15cfc0 0x7f82ac0aa500 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f82ac15cbe0 0x7f82ac0a9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 --2- 172.21.15.29:0/1182073384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82ac0b4c80 0x7f82ac159540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 >> 172.21.15.29:0/1182073384 conn(0x7f82ac0cf570 msgr2=0x7f82ac0b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 shutdown_connections 2023-10-14T16:56:31.103 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f95d879f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:56:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.100+0000 7f82b2447640 1 -- 172.21.15.29:0/1182073384 wait complete. 2023-10-14T16:56:31.122 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T16:56:29.679347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336084,"kb_used_data":8920,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754638636,"statfs":{"total":773094113280,"available":772749963264,"internally_reserved":0,"allocated":9134080,"data_stored":4949068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":334876216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079554"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637315+0000","last_change":"2023-10-14T16:56:26.666843+0000","last_active":"2023-10-14T16:56:28.637315+0000","last_peered":"2023-10-14T16:56:28.637315+0000","last_clean":"2023-10-14T16:56:28.637315+0000","last_became_active":"2023-10-14T16:56:26.666569+0000","last_became_peered":"2023-10-14T16:56:26.666569+0000","last_unstale":"2023-10-14T16:56:28.637315+0000","last_undegraded":"2023-10-14T16:56:28.637315+0000","last_fullsized":"2023-10-14T16:56:28.637315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:05.992321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636160+0000","last_change":"2023-10-14T16:56:26.663299+0000","last_active":"2023-10-14T16:56:28.636160+0000","last_peered":"2023-10-14T16:56:28.636160+0000","last_clean":"2023-10-14T16:56:28.636160+0000","last_became_active":"2023-10-14T16:56:26.662937+0000","last_became_peered":"2023-10-14T16:56:26.662937+0000","last_unstale":"2023-10-14T16:56:28.636160+0000","last_undegraded":"2023-10-14T16:56:28.636160+0000","last_fullsized":"2023-10-14T16:56:28.636160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:31.219070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633935+0000","last_change":"2023-10-14T16:56:26.651056+0000","last_active":"2023-10-14T16:56:28.633935+0000","last_peered":"2023-10-14T16:56:28.633935+0000","last_clean":"2023-10-14T16:56:28.633935+0000","last_became_active":"2023-10-14T16:56:26.650774+0000","last_became_peered":"2023-10-14T16:56:26.650774+0000","last_unstale":"2023-10-14T16:56:28.633935+0000","last_undegraded":"2023-10-14T16:56:28.633935+0000","last_fullsized":"2023-10-14T16:56:28.633935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:10:46.090271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635597+0000","last_change":"2023-10-14T16:56:26.644485+0000","last_active":"2023-10-14T16:56:28.635597+0000","last_peered":"2023-10-14T16:56:28.635597+0000","last_clean":"2023-10-14T16:56:28.635597+0000","last_became_active":"2023-10-14T16:56:26.642602+0000","last_became_peered":"2023-10-14T16:56:26.642602+0000","last_unstale":"2023-10-14T16:56:28.635597+0000","last_undegraded":"2023-10-14T16:56:28.635597+0000","last_fullsized":"2023-10-14T16:56:28.635597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:18.732584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635103+0000","last_change":"2023-10-14T16:56:26.641166+0000","last_active":"2023-10-14T16:56:28.635103+0000","last_peered":"2023-10-14T16:56:28.635103+0000","last_clean":"2023-10-14T16:56:28.635103+0000","last_became_active":"2023-10-14T16:56:26.640870+0000","last_became_peered":"2023-10-14T16:56:26.640870+0000","last_unstale":"2023-10-14T16:56:28.635103+0000","last_undegraded":"2023-10-14T16:56:28.635103+0000","last_fullsized":"2023-10-14T16:56:28.635103+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:31.698157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646838+0000","last_change":"2023-10-14T16:56:26.655173+0000","last_active":"2023-10-14T16:56:27.646838+0000","last_peered":"2023-10-14T16:56:27.646838+0000","last_clean":"2023-10-14T16:56:27.646838+0000","last_became_active":"2023-10-14T16:56:26.654803+0000","last_became_peered":"2023-10-14T16:56:26.654803+0000","last_unstale":"2023-10-14T16:56:27.646838+0000","last_undegraded":"2023-10-14T16:56:27.646838+0000","last_fullsized":"2023-10-14T16:56:27.646838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:19:32.916043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649703+0000","last_change":"2023-10-14T16:56:26.644806+0000","last_active":"2023-10-14T16:56:27.649703+0000","last_peered":"2023-10-14T16:56:27.649703+0000","last_clean":"2023-10-14T16:56:27.649703+0000","last_became_active":"2023-10-14T16:56:26.640771+0000","last_became_peered":"2023-10-14T16:56:26.640771+0000","last_unstale":"2023-10-14T16:56:27.649703+0000","last_undegraded":"2023-10-14T16:56:27.649703+0000","last_fullsized":"2023-10-14T16:56:27.649703+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:24:27.364564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635894+0000","last_change":"2023-10-14T16:56:26.651766+0000","last_active":"2023-10-14T16:56:28.635894+0000","last_peered":"2023-10-14T16:56:28.635894+0000","last_clean":"2023-10-14T16:56:28.635894+0000","last_became_active":"2023-10-14T16:56:26.651463+0000","last_became_peered":"2023-10-14T16:56:26.651463+0000","last_unstale":"2023-10-14T16:56:28.635894+0000","last_undegraded":"2023-10-14T16:56:28.635894+0000","last_fullsized":"2023-10-14T16:56:28.635894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:52.293330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651038+0000","last_change":"2023-10-14T16:56:26.663236+0000","last_active":"2023-10-14T16:56:27.651038+0000","last_peered":"2023-10-14T16:56:27.651038+0000","last_clean":"2023-10-14T16:56:27.651038+0000","last_became_active":"2023-10-14T16:56:26.662601+0000","last_became_peered":"2023-10-14T16:56:26.662601+0000","last_unstale":"2023-10-14T16:56:27.651038+0000","last_undegraded":"2023-10-14T16:56:27.651038+0000","last_fullsized":"2023-10-14T16:56:27.651038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:00.056508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633348+0000","last_change":"2023-10-14T16:56:26.647304+0000","last_active":"2023-10-14T16:56:28.633348+0000","last_peered":"2023-10-14T16:56:28.633348+0000","last_clean":"2023-10-14T16:56:28.633348+0000","last_became_active":"2023-10-14T16:56:26.646955+0000","last_became_peered":"2023-10-14T16:56:26.646955+0000","last_unstale":"2023-10-14T16:56:28.633348+0000","last_undegraded":"2023-10-14T16:56:28.633348+0000","last_fullsized":"2023-10-14T16:56:28.633348+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:02.946295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634043+0000","last_change":"2023-10-14T16:56:26.667439+0000","last_active":"2023-10-14T16:56:28.634043+0000","last_peered":"2023-10-14T16:56:28.634043+0000","last_clean":"2023-10-14T16:56:28.634043+0000","last_became_active":"2023-10-14T16:56:26.666805+0000","last_became_peered":"2023-10-14T16:56:26.666805+0000","last_unstale":"2023-10-14T16:56:28.634043+0000","last_undegraded":"2023-10-14T16:56:28.634043+0000","last_fullsized":"2023-10-14T16:56:28.634043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:31.962545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635718+0000","last_change":"2023-10-14T16:56:26.647590+0000","last_active":"2023-10-14T16:56:28.635718+0000","last_peered":"2023-10-14T16:56:28.635718+0000","last_clean":"2023-10-14T16:56:28.635718+0000","last_became_active":"2023-10-14T16:56:26.647192+0000","last_became_peered":"2023-10-14T16:56:26.647192+0000","last_unstale":"2023-10-14T16:56:28.635718+0000","last_undegraded":"2023-10-14T16:56:28.635718+0000","last_fullsized":"2023-10-14T16:56:28.635718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:55:35.417582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651960+0000","last_change":"2023-10-14T16:56:26.658224+0000","last_active":"2023-10-14T16:56:27.651960+0000","last_peered":"2023-10-14T16:56:27.651960+0000","last_clean":"2023-10-14T16:56:27.651960+0000","last_became_active":"2023-10-14T16:56:26.657896+0000","last_became_peered":"2023-10-14T16:56:26.657896+0000","last_unstale":"2023-10-14T16:56:27.651960+0000","last_undegraded":"2023-10-14T16:56:27.651960+0000","last_fullsized":"2023-10-14T16:56:27.651960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:45:11.658283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651285+0000","last_change":"2023-10-14T16:56:26.649981+0000","last_active":"2023-10-14T16:56:27.651285+0000","last_peered":"2023-10-14T16:56:27.651285+0000","last_clean":"2023-10-14T16:56:27.651285+0000","last_became_active":"2023-10-14T16:56:26.646819+0000","last_became_peered":"2023-10-14T16:56:26.646819+0000","last_unstale":"2023-10-14T16:56:27.651285+0000","last_undegraded":"2023-10-14T16:56:27.651285+0000","last_fullsized":"2023-10-14T16:56:27.651285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:28.315229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650247+0000","last_change":"2023-10-14T16:56:26.649023+0000","last_active":"2023-10-14T16:56:27.650247+0000","last_peered":"2023-10-14T16:56:27.650247+0000","last_clean":"2023-10-14T16:56:27.650247+0000","last_became_active":"2023-10-14T16:56:26.648716+0000","last_became_peered":"2023-10-14T16:56:26.648716+0000","last_unstale":"2023-10-14T16:56:27.650247+0000","last_undegraded":"2023-10-14T16:56:27.650247+0000","last_fullsized":"2023-10-14T16:56:27.650247+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:47:58.473208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646688+0000","last_change":"2023-10-14T16:56:26.658134+0000","last_active":"2023-10-14T16:56:27.646688+0000","last_peered":"2023-10-14T16:56:27.646688+0000","last_clean":"2023-10-14T16:56:27.646688+0000","last_became_active":"2023-10-14T16:56:26.657731+0000","last_became_peered":"2023-10-14T16:56:26.657731+0000","last_unstale":"2023-10-14T16:56:27.646688+0000","last_undegraded":"2023-10-14T16:56:27.646688+0000","last_fullsized":"2023-10-14T16:56:27.646688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:04:26.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635915+0000","last_change":"2023-10-14T16:56:26.663466+0000","last_active":"2023-10-14T16:56:28.635915+0000","last_peered":"2023-10-14T16:56:28.635915+0000","last_clean":"2023-10-14T16:56:28.635915+0000","last_became_active":"2023-10-14T16:56:26.656381+0000","last_became_peered":"2023-10-14T16:56:26.656381+0000","last_unstale":"2023-10-14T16:56:28.635915+0000","last_undegraded":"2023-10-14T16:56:28.635915+0000","last_fullsized":"2023-10-14T16:56:28.635915+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:04.197142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634030+0000","last_change":"2023-10-14T16:56:26.660673+0000","last_active":"2023-10-14T16:56:28.634030+0000","last_peered":"2023-10-14T16:56:28.634030+0000","last_clean":"2023-10-14T16:56:28.634030+0000","last_became_active":"2023-10-14T16:56:26.660342+0000","last_became_peered":"2023-10-14T16:56:26.660342+0000","last_unstale":"2023-10-14T16:56:28.634030+0000","last_undegraded":"2023-10-14T16:56:28.634030+0000","last_fullsized":"2023-10-14T16:56:28.634030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:24.440386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636805+0000","last_change":"2023-10-14T16:56:26.666156+0000","last_active":"2023-10-14T16:56:28.636805+0000","last_peered":"2023-10-14T16:56:28.636805+0000","last_clean":"2023-10-14T16:56:28.636805+0000","last_became_active":"2023-10-14T16:56:26.665868+0000","last_became_peered":"2023-10-14T16:56:26.665868+0000","last_unstale":"2023-10-14T16:56:28.636805+0000","last_undegraded":"2023-10-14T16:56:28.636805+0000","last_fullsized":"2023-10-14T16:56:28.636805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:04:26.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636391+0000","last_change":"2023-10-14T16:56:26.666482+0000","last_active":"2023-10-14T16:56:28.636391+0000","last_peered":"2023-10-14T16:56:28.636391+0000","last_clean":"2023-10-14T16:56:28.636391+0000","last_became_active":"2023-10-14T16:56:26.665136+0000","last_became_peered":"2023-10-14T16:56:26.665136+0000","last_unstale":"2023-10-14T16:56:28.636391+0000","last_undegraded":"2023-10-14T16:56:28.636391+0000","last_fullsized":"2023-10-14T16:56:28.636391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:34.986146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633746+0000","last_change":"2023-10-14T16:56:26.654125+0000","last_active":"2023-10-14T16:56:28.633746+0000","last_peered":"2023-10-14T16:56:28.633746+0000","last_clean":"2023-10-14T16:56:28.633746+0000","last_became_active":"2023-10-14T16:56:26.652025+0000","last_became_peered":"2023-10-14T16:56:26.652025+0000","last_unstale":"2023-10-14T16:56:28.633746+0000","last_undegraded":"2023-10-14T16:56:28.633746+0000","last_fullsized":"2023-10-14T16:56:28.633746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:53:31.081821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633957+0000","last_change":"2023-10-14T16:56:26.653653+0000","last_active":"2023-10-14T16:56:28.633957+0000","last_peered":"2023-10-14T16:56:28.633957+0000","last_clean":"2023-10-14T16:56:28.633957+0000","last_became_active":"2023-10-14T16:56:26.653382+0000","last_became_peered":"2023-10-14T16:56:26.653382+0000","last_unstale":"2023-10-14T16:56:28.633957+0000","last_undegraded":"2023-10-14T16:56:28.633957+0000","last_fullsized":"2023-10-14T16:56:28.633957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:17.197738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633347+0000","last_change":"2023-10-14T16:56:26.664506+0000","last_active":"2023-10-14T16:56:28.633347+0000","last_peered":"2023-10-14T16:56:28.633347+0000","last_clean":"2023-10-14T16:56:28.633347+0000","last_became_active":"2023-10-14T16:56:26.664204+0000","last_became_peered":"2023-10-14T16:56:26.664204+0000","last_unstale":"2023-10-14T16:56:28.633347+0000","last_undegraded":"2023-10-14T16:56:28.633347+0000","last_fullsized":"2023-10-14T16:56:28.633347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:48.701184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651149+0000","last_change":"2023-10-14T16:56:26.658829+0000","last_active":"2023-10-14T16:56:27.651149+0000","last_peered":"2023-10-14T16:56:27.651149+0000","last_clean":"2023-10-14T16:56:27.651149+0000","last_became_active":"2023-10-14T16:56:26.658167+0000","last_became_peered":"2023-10-14T16:56:26.658167+0000","last_unstale":"2023-10-14T16:56:27.651149+0000","last_undegraded":"2023-10-14T16:56:27.651149+0000","last_fullsized":"2023-10-14T16:56:27.651149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:12:58.344442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637294+0000","last_change":"2023-10-14T16:56:26.644050+0000","last_active":"2023-10-14T16:56:28.637294+0000","last_peered":"2023-10-14T16:56:28.637294+0000","last_clean":"2023-10-14T16:56:28.637294+0000","last_became_active":"2023-10-14T16:56:26.643408+0000","last_became_peered":"2023-10-14T16:56:26.643408+0000","last_unstale":"2023-10-14T16:56:28.637294+0000","last_undegraded":"2023-10-14T16:56:28.637294+0000","last_fullsized":"2023-10-14T16:56:28.637294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:43.080935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636502+0000","last_change":"2023-10-14T16:56:26.640086+0000","last_active":"2023-10-14T16:56:28.636502+0000","last_peered":"2023-10-14T16:56:28.636502+0000","last_clean":"2023-10-14T16:56:28.636502+0000","last_became_active":"2023-10-14T16:56:26.638238+0000","last_became_peered":"2023-10-14T16:56:26.638238+0000","last_unstale":"2023-10-14T16:56:28.636502+0000","last_undegraded":"2023-10-14T16:56:28.636502+0000","last_fullsized":"2023-10-14T16:56:28.636502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:51:45.292889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646645+0000","last_change":"2023-10-14T16:56:26.670716+0000","last_active":"2023-10-14T16:56:27.646645+0000","last_peered":"2023-10-14T16:56:27.646645+0000","last_clean":"2023-10-14T16:56:27.646645+0000","last_became_active":"2023-10-14T16:56:26.661708+0000","last_became_peered":"2023-10-14T16:56:26.661708+0000","last_unstale":"2023-10-14T16:56:27.646645+0000","last_undegraded":"2023-10-14T16:56:27.646645+0000","last_fullsized":"2023-10-14T16:56:27.646645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:33.952293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636786+0000","last_change":"2023-10-14T16:56:26.649342+0000","last_active":"2023-10-14T16:56:28.636786+0000","last_peered":"2023-10-14T16:56:28.636786+0000","last_clean":"2023-10-14T16:56:28.636786+0000","last_became_active":"2023-10-14T16:56:26.648978+0000","last_became_peered":"2023-10-14T16:56:26.648978+0000","last_unstale":"2023-10-14T16:56:28.636786+0000","last_undegraded":"2023-10-14T16:56:28.636786+0000","last_fullsized":"2023-10-14T16:56:28.636786+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:33.952293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635436+0000","last_change":"2023-10-14T16:56:26.656387+0000","last_active":"2023-10-14T16:56:28.635436+0000","last_peered":"2023-10-14T16:56:28.635436+0000","last_clean":"2023-10-14T16:56:28.635436+0000","last_became_active":"2023-10-14T16:56:26.656077+0000","last_became_peered":"2023-10-14T16:56:26.656077+0000","last_unstale":"2023-10-14T16:56:28.635436+0000","last_undegraded":"2023-10-14T16:56:28.635436+0000","last_fullsized":"2023-10-14T16:56:28.635436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:11.525157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633280+0000","last_change":"2023-10-14T16:56:26.652520+0000","last_active":"2023-10-14T16:56:28.633280+0000","last_peered":"2023-10-14T16:56:28.633280+0000","last_clean":"2023-10-14T16:56:28.633280+0000","last_became_active":"2023-10-14T16:56:26.651212+0000","last_became_peered":"2023-10-14T16:56:26.651212+0000","last_unstale":"2023-10-14T16:56:28.633280+0000","last_undegraded":"2023-10-14T16:56:28.633280+0000","last_fullsized":"2023-10-14T16:56:28.633280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:45.300948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634938+0000","last_change":"2023-10-14T16:56:26.657709+0000","last_active":"2023-10-14T16:56:28.634938+0000","last_peered":"2023-10-14T16:56:28.634938+0000","last_clean":"2023-10-14T16:56:28.634938+0000","last_became_active":"2023-10-14T16:56:26.657248+0000","last_became_peered":"2023-10-14T16:56:26.657248+0000","last_unstale":"2023-10-14T16:56:28.634938+0000","last_undegraded":"2023-10-14T16:56:28.634938+0000","last_fullsized":"2023-10-14T16:56:28.634938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:35.156905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646791+0000","last_change":"2023-10-14T16:56:26.673025+0000","last_active":"2023-10-14T16:56:27.646791+0000","last_peered":"2023-10-14T16:56:27.646791+0000","last_clean":"2023-10-14T16:56:27.646791+0000","last_became_active":"2023-10-14T16:56:26.668592+0000","last_became_peered":"2023-10-14T16:56:26.668592+0000","last_unstale":"2023-10-14T16:56:27.646791+0000","last_undegraded":"2023-10-14T16:56:27.646791+0000","last_fullsized":"2023-10-14T16:56:27.646791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:53:35.861811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647282+0000","last_change":"2023-10-14T16:56:26.658692+0000","last_active":"2023-10-14T16:56:27.647282+0000","last_peered":"2023-10-14T16:56:27.647282+0000","last_clean":"2023-10-14T16:56:27.647282+0000","last_became_active":"2023-10-14T16:56:26.657792+0000","last_became_peered":"2023-10-14T16:56:26.657792+0000","last_unstale":"2023-10-14T16:56:27.647282+0000","last_undegraded":"2023-10-14T16:56:27.647282+0000","last_fullsized":"2023-10-14T16:56:27.647282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:58.290941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"81'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647209+0000","last_change":"2023-10-14T16:56:25.643488+0000","last_active":"2023-10-14T16:56:27.647209+0000","last_peered":"2023-10-14T16:56:27.647209+0000","last_clean":"2023-10-14T16:56:27.647209+0000","last_became_active":"2023-10-14T16:56:25.643222+0000","last_became_peered":"2023-10-14T16:56:25.643222+0000","last_unstale":"2023-10-14T16:56:27.647209+0000","last_undegraded":"2023-10-14T16:56:27.647209+0000","last_fullsized":"2023-10-14T16:56:27.647209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:15:37.864783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633735+0000","last_change":"2023-10-14T16:56:26.650938+0000","last_active":"2023-10-14T16:56:28.633735+0000","last_peered":"2023-10-14T16:56:28.633735+0000","last_clean":"2023-10-14T16:56:28.633735+0000","last_became_active":"2023-10-14T16:56:26.648274+0000","last_became_peered":"2023-10-14T16:56:26.648274+0000","last_unstale":"2023-10-14T16:56:28.633735+0000","last_undegraded":"2023-10-14T16:56:28.633735+0000","last_fullsized":"2023-10-14T16:56:28.633735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:28.160672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637392+0000","last_change":"2023-10-14T16:56:25.643219+0000","last_active":"2023-10-14T16:56:28.637392+0000","last_peered":"2023-10-14T16:56:28.637392+0000","last_clean":"2023-10-14T16:56:28.637392+0000","last_became_active":"2023-10-14T16:56:25.642932+0000","last_became_peered":"2023-10-14T16:56:25.642932+0000","last_unstale":"2023-10-14T16:56:28.637392+0000","last_undegraded":"2023-10-14T16:56:28.637392+0000","last_fullsized":"2023-10-14T16:56:28.637392+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.370194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647135+0000","last_change":"2023-10-14T16:56:26.661490+0000","last_active":"2023-10-14T16:56:27.647135+0000","last_peered":"2023-10-14T16:56:27.647135+0000","last_clean":"2023-10-14T16:56:27.647135+0000","last_became_active":"2023-10-14T16:56:26.661194+0000","last_became_peered":"2023-10-14T16:56:26.661194+0000","last_unstale":"2023-10-14T16:56:27.647135+0000","last_undegraded":"2023-10-14T16:56:27.647135+0000","last_fullsized":"2023-10-14T16:56:27.647135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:40.702394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647032+0000","last_change":"2023-10-14T16:56:25.645125+0000","last_active":"2023-10-14T16:56:27.647032+0000","last_peered":"2023-10-14T16:56:27.647032+0000","last_clean":"2023-10-14T16:56:27.647032+0000","last_became_active":"2023-10-14T16:56:25.644080+0000","last_became_peered":"2023-10-14T16:56:25.644080+0000","last_unstale":"2023-10-14T16:56:27.647032+0000","last_undegraded":"2023-10-14T16:56:27.647032+0000","last_fullsized":"2023-10-14T16:56:27.647032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:30.682990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646784+0000","last_change":"2023-10-14T16:56:26.673690+0000","last_active":"2023-10-14T16:56:27.646784+0000","last_peered":"2023-10-14T16:56:27.646784+0000","last_clean":"2023-10-14T16:56:27.646784+0000","last_became_active":"2023-10-14T16:56:26.665731+0000","last_became_peered":"2023-10-14T16:56:26.665731+0000","last_unstale":"2023-10-14T16:56:27.646784+0000","last_undegraded":"2023-10-14T16:56:27.646784+0000","last_fullsized":"2023-10-14T16:56:27.646784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:05.992321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635838+0000","last_change":"2023-10-14T16:56:25.650292+0000","last_active":"2023-10-14T16:56:28.635838+0000","last_peered":"2023-10-14T16:56:28.635838+0000","last_clean":"2023-10-14T16:56:28.635838+0000","last_became_active":"2023-10-14T16:56:25.649957+0000","last_became_peered":"2023-10-14T16:56:25.649957+0000","last_unstale":"2023-10-14T16:56:28.635838+0000","last_undegraded":"2023-10-14T16:56:28.635838+0000","last_fullsized":"2023-10-14T16:56:28.635838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:18:43.362749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634472+0000","last_change":"2023-10-14T16:56:26.653567+0000","last_active":"2023-10-14T16:56:28.634472+0000","last_peered":"2023-10-14T16:56:28.634472+0000","last_clean":"2023-10-14T16:56:28.634472+0000","last_became_active":"2023-10-14T16:56:26.653291+0000","last_became_peered":"2023-10-14T16:56:26.653291+0000","last_unstale":"2023-10-14T16:56:28.634472+0000","last_undegraded":"2023-10-14T16:56:28.634472+0000","last_fullsized":"2023-10-14T16:56:28.634472+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:38:35.976676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650717+0000","last_change":"2023-10-14T16:56:25.636908+0000","last_active":"2023-10-14T16:56:27.650717+0000","last_peered":"2023-10-14T16:56:27.650717+0000","last_clean":"2023-10-14T16:56:27.650717+0000","last_became_active":"2023-10-14T16:56:25.636500+0000","last_became_peered":"2023-10-14T16:56:25.636500+0000","last_unstale":"2023-10-14T16:56:27.650717+0000","last_undegraded":"2023-10-14T16:56:27.650717+0000","last_fullsized":"2023-10-14T16:56:27.650717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:33.815067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635063+0000","last_change":"2023-10-14T16:56:26.666970+0000","last_active":"2023-10-14T16:56:28.635063+0000","last_peered":"2023-10-14T16:56:28.635063+0000","last_clean":"2023-10-14T16:56:28.635063+0000","last_became_active":"2023-10-14T16:56:26.660155+0000","last_became_peered":"2023-10-14T16:56:26.660155+0000","last_unstale":"2023-10-14T16:56:28.635063+0000","last_undegraded":"2023-10-14T16:56:28.635063+0000","last_fullsized":"2023-10-14T16:56:28.635063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:18.391549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647330+0000","last_change":"2023-10-14T16:56:25.640146+0000","last_active":"2023-10-14T16:56:27.647330+0000","last_peered":"2023-10-14T16:56:27.647330+0000","last_clean":"2023-10-14T16:56:27.647330+0000","last_became_active":"2023-10-14T16:56:25.639854+0000","last_became_peered":"2023-10-14T16:56:25.639854+0000","last_unstale":"2023-10-14T16:56:27.647330+0000","last_undegraded":"2023-10-14T16:56:27.647330+0000","last_fullsized":"2023-10-14T16:56:27.647330+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:34.592548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651536+0000","last_change":"2023-10-14T16:56:26.649084+0000","last_active":"2023-10-14T16:56:27.651536+0000","last_peered":"2023-10-14T16:56:27.651536+0000","last_clean":"2023-10-14T16:56:27.651536+0000","last_became_active":"2023-10-14T16:56:26.648783+0000","last_became_peered":"2023-10-14T16:56:26.648783+0000","last_unstale":"2023-10-14T16:56:27.651536+0000","last_undegraded":"2023-10-14T16:56:27.651536+0000","last_fullsized":"2023-10-14T16:56:27.651536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:16.712989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633470+0000","last_change":"2023-10-14T16:56:25.647288+0000","last_active":"2023-10-14T16:56:28.633470+0000","last_peered":"2023-10-14T16:56:28.633470+0000","last_clean":"2023-10-14T16:56:28.633470+0000","last_became_active":"2023-10-14T16:56:25.646903+0000","last_became_peered":"2023-10-14T16:56:25.646903+0000","last_unstale":"2023-10-14T16:56:28.633470+0000","last_undegraded":"2023-10-14T16:56:28.633470+0000","last_fullsized":"2023-10-14T16:56:28.633470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:44:10.364683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634150+0000","last_change":"2023-10-14T16:56:26.643883+0000","last_active":"2023-10-14T16:56:28.634150+0000","last_peered":"2023-10-14T16:56:28.634150+0000","last_clean":"2023-10-14T16:56:28.634150+0000","last_became_active":"2023-10-14T16:56:26.643594+0000","last_became_peered":"2023-10-14T16:56:26.643594+0000","last_unstale":"2023-10-14T16:56:28.634150+0000","last_undegraded":"2023-10-14T16:56:28.634150+0000","last_fullsized":"2023-10-14T16:56:28.634150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:14.882523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649774+0000","last_change":"2023-10-14T16:56:25.643963+0000","last_active":"2023-10-14T16:56:27.649774+0000","last_peered":"2023-10-14T16:56:27.649774+0000","last_clean":"2023-10-14T16:56:27.649774+0000","last_became_active":"2023-10-14T16:56:25.643635+0000","last_became_peered":"2023-10-14T16:56:25.643635+0000","last_unstale":"2023-10-14T16:56:27.649774+0000","last_undegraded":"2023-10-14T16:56:27.649774+0000","last_fullsized":"2023-10-14T16:56:27.649774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:09:30.864016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651404+0000","last_change":"2023-10-14T16:56:26.654103+0000","last_active":"2023-10-14T16:56:27.651404+0000","last_peered":"2023-10-14T16:56:27.651404+0000","last_clean":"2023-10-14T16:56:27.651404+0000","last_became_active":"2023-10-14T16:56:26.653814+0000","last_became_peered":"2023-10-14T16:56:26.653814+0000","last_unstale":"2023-10-14T16:56:27.651404+0000","last_undegraded":"2023-10-14T16:56:27.651404+0000","last_fullsized":"2023-10-14T16:56:27.651404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:45:46.099622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634821+0000","last_change":"2023-10-14T16:56:25.648712+0000","last_active":"2023-10-14T16:56:28.634821+0000","last_peered":"2023-10-14T16:56:28.634821+0000","last_clean":"2023-10-14T16:56:28.634821+0000","last_became_active":"2023-10-14T16:56:25.647900+0000","last_became_peered":"2023-10-14T16:56:25.647900+0000","last_unstale":"2023-10-14T16:56:28.634821+0000","last_undegraded":"2023-10-14T16:56:28.634821+0000","last_fullsized":"2023-10-14T16:56:28.634821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:17.717418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634895+0000","last_change":"2023-10-14T16:56:26.655659+0000","last_active":"2023-10-14T16:56:28.634895+0000","last_peered":"2023-10-14T16:56:28.634895+0000","last_clean":"2023-10-14T16:56:28.634895+0000","last_became_active":"2023-10-14T16:56:26.655265+0000","last_became_peered":"2023-10-14T16:56:26.655265+0000","last_unstale":"2023-10-14T16:56:28.634895+0000","last_undegraded":"2023-10-14T16:56:28.634895+0000","last_fullsized":"2023-10-14T16:56:28.634895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:05.887144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636809+0000","last_change":"2023-10-14T16:56:25.640983+0000","last_active":"2023-10-14T16:56:28.636809+0000","last_peered":"2023-10-14T16:56:28.636809+0000","last_clean":"2023-10-14T16:56:28.636809+0000","last_became_active":"2023-10-14T16:56:25.640119+0000","last_became_peered":"2023-10-14T16:56:25.640119+0000","last_unstale":"2023-10-14T16:56:28.636809+0000","last_undegraded":"2023-10-14T16:56:28.636809+0000","last_fullsized":"2023-10-14T16:56:28.636809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.612413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632957+0000","last_change":"2023-10-14T16:56:26.641261+0000","last_active":"2023-10-14T16:56:28.632957+0000","last_peered":"2023-10-14T16:56:28.632957+0000","last_clean":"2023-10-14T16:56:28.632957+0000","last_became_active":"2023-10-14T16:56:26.640953+0000","last_became_peered":"2023-10-14T16:56:26.640953+0000","last_unstale":"2023-10-14T16:56:28.632957+0000","last_undegraded":"2023-10-14T16:56:28.632957+0000","last_fullsized":"2023-10-14T16:56:28.632957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:14:43.771856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"81'3","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634518+0000","last_change":"2023-10-14T16:56:25.649816+0000","last_active":"2023-10-14T16:56:28.634518+0000","last_peered":"2023-10-14T16:56:28.634518+0000","last_clean":"2023-10-14T16:56:28.634518+0000","last_became_active":"2023-10-14T16:56:25.649028+0000","last_became_peered":"2023-10-14T16:56:25.649028+0000","last_unstale":"2023-10-14T16:56:28.634518+0000","last_undegraded":"2023-10-14T16:56:28.634518+0000","last_fullsized":"2023-10-14T16:56:28.634518+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:07.276627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634283+0000","last_change":"2023-10-14T16:56:26.655469+0000","last_active":"2023-10-14T16:56:28.634283+0000","last_peered":"2023-10-14T16:56:28.634283+0000","last_clean":"2023-10-14T16:56:28.634283+0000","last_became_active":"2023-10-14T16:56:26.655162+0000","last_became_peered":"2023-10-14T16:56:26.655162+0000","last_unstale":"2023-10-14T16:56:28.634283+0000","last_undegraded":"2023-10-14T16:56:28.634283+0000","last_fullsized":"2023-10-14T16:56:28.634283+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.627579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"81'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651119+0000","last_change":"2023-10-14T16:56:25.644403+0000","last_active":"2023-10-14T16:56:27.651119+0000","last_peered":"2023-10-14T16:56:27.651119+0000","last_clean":"2023-10-14T16:56:27.651119+0000","last_became_active":"2023-10-14T16:56:25.643086+0000","last_became_peered":"2023-10-14T16:56:25.643086+0000","last_unstale":"2023-10-14T16:56:27.651119+0000","last_undegraded":"2023-10-14T16:56:27.651119+0000","last_fullsized":"2023-10-14T16:56:27.651119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:14.208391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634219+0000","last_change":"2023-10-14T16:56:26.645925+0000","last_active":"2023-10-14T16:56:28.634219+0000","last_peered":"2023-10-14T16:56:28.634219+0000","last_clean":"2023-10-14T16:56:28.634219+0000","last_became_active":"2023-10-14T16:56:26.645401+0000","last_became_peered":"2023-10-14T16:56:26.645401+0000","last_unstale":"2023-10-14T16:56:28.634219+0000","last_undegraded":"2023-10-14T16:56:28.634219+0000","last_fullsized":"2023-10-14T16:56:28.634219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:37:17.646767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635997+0000","last_change":"2023-10-14T16:56:25.642652+0000","last_active":"2023-10-14T16:56:28.635997+0000","last_peered":"2023-10-14T16:56:28.635997+0000","last_clean":"2023-10-14T16:56:28.635997+0000","last_became_active":"2023-10-14T16:56:25.641433+0000","last_became_peered":"2023-10-14T16:56:25.641433+0000","last_unstale":"2023-10-14T16:56:28.635997+0000","last_undegraded":"2023-10-14T16:56:28.635997+0000","last_fullsized":"2023-10-14T16:56:28.635997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:17.542259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633887+0000","last_change":"2023-10-14T16:56:26.648280+0000","last_active":"2023-10-14T16:56:28.633887+0000","last_peered":"2023-10-14T16:56:28.633887+0000","last_clean":"2023-10-14T16:56:28.633887+0000","last_became_active":"2023-10-14T16:56:26.647953+0000","last_became_peered":"2023-10-14T16:56:26.647953+0000","last_unstale":"2023-10-14T16:56:28.633887+0000","last_undegraded":"2023-10-14T16:56:28.633887+0000","last_fullsized":"2023-10-14T16:56:28.633887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:18.557426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.652091+0000","last_change":"2023-10-14T16:56:25.640577+0000","last_active":"2023-10-14T16:56:27.652091+0000","last_peered":"2023-10-14T16:56:27.652091+0000","last_clean":"2023-10-14T16:56:27.652091+0000","last_became_active":"2023-10-14T16:56:25.639993+0000","last_became_peered":"2023-10-14T16:56:25.639993+0000","last_unstale":"2023-10-14T16:56:27.652091+0000","last_undegraded":"2023-10-14T16:56:27.652091+0000","last_fullsized":"2023-10-14T16:56:27.652091+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:08.972626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649872+0000","last_change":"2023-10-14T16:56:26.647114+0000","last_active":"2023-10-14T16:56:27.649872+0000","last_peered":"2023-10-14T16:56:27.649872+0000","last_clean":"2023-10-14T16:56:27.649872+0000","last_became_active":"2023-10-14T16:56:26.646832+0000","last_became_peered":"2023-10-14T16:56:26.646832+0000","last_unstale":"2023-10-14T16:56:27.649872+0000","last_undegraded":"2023-10-14T16:56:27.649872+0000","last_fullsized":"2023-10-14T16:56:27.649872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:28:52.159302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633700+0000","last_change":"2023-10-14T16:56:25.650013+0000","last_active":"2023-10-14T16:56:28.633700+0000","last_peered":"2023-10-14T16:56:28.633700+0000","last_clean":"2023-10-14T16:56:28.633700+0000","last_became_active":"2023-10-14T16:56:25.649703+0000","last_became_peered":"2023-10-14T16:56:25.649703+0000","last_unstale":"2023-10-14T16:56:28.633700+0000","last_undegraded":"2023-10-14T16:56:28.633700+0000","last_fullsized":"2023-10-14T16:56:28.633700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:54.440032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633376+0000","last_change":"2023-10-14T16:56:26.665210+0000","last_active":"2023-10-14T16:56:28.633376+0000","last_peered":"2023-10-14T16:56:28.633376+0000","last_clean":"2023-10-14T16:56:28.633376+0000","last_became_active":"2023-10-14T16:56:26.664862+0000","last_became_peered":"2023-10-14T16:56:26.664862+0000","last_unstale":"2023-10-14T16:56:28.633376+0000","last_undegraded":"2023-10-14T16:56:28.633376+0000","last_fullsized":"2023-10-14T16:56:28.633376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:53.197186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"81'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632980+0000","last_change":"2023-10-14T16:56:25.649885+0000","last_active":"2023-10-14T16:56:28.632980+0000","last_peered":"2023-10-14T16:56:28.632980+0000","last_clean":"2023-10-14T16:56:28.632980+0000","last_became_active":"2023-10-14T16:56:25.649573+0000","last_became_peered":"2023-10-14T16:56:25.649573+0000","last_unstale":"2023-10-14T16:56:28.632980+0000","last_undegraded":"2023-10-14T16:56:28.632980+0000","last_fullsized":"2023-10-14T16:56:28.632980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:18:43.362749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634305+0000","last_change":"2023-10-14T16:56:26.651016+0000","last_active":"2023-10-14T16:56:28.634305+0000","last_peered":"2023-10-14T16:56:28.634305+0000","last_clean":"2023-10-14T16:56:28.634305+0000","last_became_active":"2023-10-14T16:56:26.650703+0000","last_became_peered":"2023-10-14T16:56:26.650703+0000","last_unstale":"2023-10-14T16:56:28.634305+0000","last_undegraded":"2023-10-14T16:56:28.634305+0000","last_fullsized":"2023-10-14T16:56:28.634305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:35.385361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636295+0000","last_change":"2023-10-14T16:56:25.642862+0000","last_active":"2023-10-14T16:56:28.636295+0000","last_peered":"2023-10-14T16:56:28.636295+0000","last_clean":"2023-10-14T16:56:28.636295+0000","last_became_active":"2023-10-14T16:56:25.642560+0000","last_became_peered":"2023-10-14T16:56:25.642560+0000","last_unstale":"2023-10-14T16:56:28.636295+0000","last_undegraded":"2023-10-14T16:56:28.636295+0000","last_fullsized":"2023-10-14T16:56:28.636295+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:26.189777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636597+0000","last_change":"2023-10-14T16:56:26.645263+0000","last_active":"2023-10-14T16:56:28.636597+0000","last_peered":"2023-10-14T16:56:28.636597+0000","last_clean":"2023-10-14T16:56:28.636597+0000","last_became_active":"2023-10-14T16:56:26.644911+0000","last_became_peered":"2023-10-14T16:56:26.644911+0000","last_unstale":"2023-10-14T16:56:28.636597+0000","last_undegraded":"2023-10-14T16:56:28.636597+0000","last_fullsized":"2023-10-14T16:56:28.636597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:08.291793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647218+0000","last_change":"2023-10-14T16:56:25.641798+0000","last_active":"2023-10-14T16:56:27.647218+0000","last_peered":"2023-10-14T16:56:27.647218+0000","last_clean":"2023-10-14T16:56:27.647218+0000","last_became_active":"2023-10-14T16:56:25.639123+0000","last_became_peered":"2023-10-14T16:56:25.639123+0000","last_unstale":"2023-10-14T16:56:27.647218+0000","last_undegraded":"2023-10-14T16:56:27.647218+0000","last_fullsized":"2023-10-14T16:56:27.647218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:30:32.661864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637100+0000","last_change":"2023-10-14T16:56:26.654982+0000","last_active":"2023-10-14T16:56:28.637100+0000","last_peered":"2023-10-14T16:56:28.637100+0000","last_clean":"2023-10-14T16:56:28.637100+0000","last_became_active":"2023-10-14T16:56:26.652898+0000","last_became_peered":"2023-10-14T16:56:26.652898+0000","last_unstale":"2023-10-14T16:56:28.637100+0000","last_undegraded":"2023-10-14T16:56:28.637100+0000","last_fullsized":"2023-10-14T16:56:28.637100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:53.647679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636240+0000","last_change":"2023-10-14T16:56:25.647622+0000","last_active":"2023-10-14T16:56:28.636240+0000","last_peered":"2023-10-14T16:56:28.636240+0000","last_clean":"2023-10-14T16:56:28.636240+0000","last_became_active":"2023-10-14T16:56:25.647291+0000","last_became_peered":"2023-10-14T16:56:25.647291+0000","last_unstale":"2023-10-14T16:56:28.636240+0000","last_undegraded":"2023-10-14T16:56:28.636240+0000","last_fullsized":"2023-10-14T16:56:28.636240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:11:17.476895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634477+0000","last_change":"2023-10-14T16:56:26.658026+0000","last_active":"2023-10-14T16:56:28.634477+0000","last_peered":"2023-10-14T16:56:28.634477+0000","last_clean":"2023-10-14T16:56:28.634477+0000","last_became_active":"2023-10-14T16:56:26.657716+0000","last_became_peered":"2023-10-14T16:56:26.657716+0000","last_unstale":"2023-10-14T16:56:28.634477+0000","last_undegraded":"2023-10-14T16:56:28.634477+0000","last_fullsized":"2023-10-14T16:56:28.634477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:20.604773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647239+0000","last_change":"2023-10-14T16:56:25.637124+0000","last_active":"2023-10-14T16:56:27.647239+0000","last_peered":"2023-10-14T16:56:27.647239+0000","last_clean":"2023-10-14T16:56:27.647239+0000","last_became_active":"2023-10-14T16:56:25.636785+0000","last_became_peered":"2023-10-14T16:56:25.636785+0000","last_unstale":"2023-10-14T16:56:27.647239+0000","last_undegraded":"2023-10-14T16:56:27.647239+0000","last_fullsized":"2023-10-14T16:56:27.647239+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:15.697823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634561+0000","last_change":"2023-10-14T16:56:26.645678+0000","last_active":"2023-10-14T16:56:28.634561+0000","last_peered":"2023-10-14T16:56:28.634561+0000","last_clean":"2023-10-14T16:56:28.634561+0000","last_became_active":"2023-10-14T16:56:26.644996+0000","last_became_peered":"2023-10-14T16:56:26.644996+0000","last_unstale":"2023-10-14T16:56:28.634561+0000","last_undegraded":"2023-10-14T16:56:28.634561+0000","last_fullsized":"2023-10-14T16:56:28.634561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:23:38.826436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635690+0000","last_change":"2023-10-14T16:56:25.637583+0000","last_active":"2023-10-14T16:56:28.635690+0000","last_peered":"2023-10-14T16:56:28.635690+0000","last_clean":"2023-10-14T16:56:28.635690+0000","last_became_active":"2023-10-14T16:56:25.636540+0000","last_became_peered":"2023-10-14T16:56:25.636540+0000","last_unstale":"2023-10-14T16:56:28.635690+0000","last_undegraded":"2023-10-14T16:56:28.635690+0000","last_fullsized":"2023-10-14T16:56:28.635690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:10:45.075105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646691+0000","last_change":"2023-10-14T16:56:26.660199+0000","last_active":"2023-10-14T16:56:27.646691+0000","last_peered":"2023-10-14T16:56:27.646691+0000","last_clean":"2023-10-14T16:56:27.646691+0000","last_became_active":"2023-10-14T16:56:26.659785+0000","last_became_peered":"2023-10-14T16:56:26.659785+0000","last_unstale":"2023-10-14T16:56:27.646691+0000","last_undegraded":"2023-10-14T16:56:27.646691+0000","last_fullsized":"2023-10-14T16:56:27.646691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:43.080935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632618+0000","last_change":"2023-10-14T16:56:25.640484+0000","last_active":"2023-10-14T16:56:28.632618+0000","last_peered":"2023-10-14T16:56:28.632618+0000","last_clean":"2023-10-14T16:56:28.632618+0000","last_became_active":"2023-10-14T16:56:25.640197+0000","last_became_peered":"2023-10-14T16:56:25.640197+0000","last_unstale":"2023-10-14T16:56:28.632618+0000","last_undegraded":"2023-10-14T16:56:28.632618+0000","last_fullsized":"2023-10-14T16:56:28.632618+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:16:00.739758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647423+0000","last_change":"2023-10-14T16:56:26.659203+0000","last_active":"2023-10-14T16:56:27.647423+0000","last_peered":"2023-10-14T16:56:27.647423+0000","last_clean":"2023-10-14T16:56:27.647423+0000","last_became_active":"2023-10-14T16:56:26.658918+0000","last_became_peered":"2023-10-14T16:56:26.658918+0000","last_unstale":"2023-10-14T16:56:27.647423+0000","last_undegraded":"2023-10-14T16:56:27.647423+0000","last_fullsized":"2023-10-14T16:56:27.647423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:16:01.754924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632670+0000","last_change":"2023-10-14T16:56:25.647872+0000","last_active":"2023-10-14T16:56:28.632670+0000","last_peered":"2023-10-14T16:56:28.632670+0000","last_clean":"2023-10-14T16:56:28.632670+0000","last_became_active":"2023-10-14T16:56:25.647561+0000","last_became_peered":"2023-10-14T16:56:25.647561+0000","last_unstale":"2023-10-14T16:56:28.632670+0000","last_undegraded":"2023-10-14T16:56:28.632670+0000","last_fullsized":"2023-10-14T16:56:28.632670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:08:17.942375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636469+0000","last_change":"2023-10-14T16:56:26.646367+0000","last_active":"2023-10-14T16:56:28.636469+0000","last_peered":"2023-10-14T16:56:28.636469+0000","last_clean":"2023-10-14T16:56:28.636469+0000","last_became_active":"2023-10-14T16:56:26.645998+0000","last_became_peered":"2023-10-14T16:56:26.645998+0000","last_unstale":"2023-10-14T16:56:28.636469+0000","last_undegraded":"2023-10-14T16:56:28.636469+0000","last_fullsized":"2023-10-14T16:56:28.636469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:40.592003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633785+0000","last_change":"2023-10-14T16:56:25.633905+0000","last_active":"2023-10-14T16:56:28.633785+0000","last_peered":"2023-10-14T16:56:28.633785+0000","last_clean":"2023-10-14T16:56:28.633785+0000","last_became_active":"2023-10-14T16:56:25.633610+0000","last_became_peered":"2023-10-14T16:56:25.633610+0000","last_unstale":"2023-10-14T16:56:28.633785+0000","last_undegraded":"2023-10-14T16:56:28.633785+0000","last_fullsized":"2023-10-14T16:56:28.633785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:18.308926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636641+0000","last_change":"2023-10-14T16:56:26.649544+0000","last_active":"2023-10-14T16:56:28.636641+0000","last_peered":"2023-10-14T16:56:28.636641+0000","last_clean":"2023-10-14T16:56:28.636641+0000","last_became_active":"2023-10-14T16:56:26.649109+0000","last_became_peered":"2023-10-14T16:56:26.649109+0000","last_unstale":"2023-10-14T16:56:28.636641+0000","last_undegraded":"2023-10-14T16:56:28.636641+0000","last_fullsized":"2023-10-14T16:56:28.636641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:15.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647314+0000","last_change":"2023-10-14T16:56:25.637970+0000","last_active":"2023-10-14T16:56:27.647314+0000","last_peered":"2023-10-14T16:56:27.647314+0000","last_clean":"2023-10-14T16:56:27.647314+0000","last_became_active":"2023-10-14T16:56:25.637682+0000","last_became_peered":"2023-10-14T16:56:25.637682+0000","last_unstale":"2023-10-14T16:56:27.647314+0000","last_undegraded":"2023-10-14T16:56:27.647314+0000","last_fullsized":"2023-10-14T16:56:27.647314+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:40.709459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633034+0000","last_change":"2023-10-14T16:56:26.648848+0000","last_active":"2023-10-14T16:56:28.633034+0000","last_peered":"2023-10-14T16:56:28.633034+0000","last_clean":"2023-10-14T16:56:28.633034+0000","last_became_active":"2023-10-14T16:56:26.648228+0000","last_became_peered":"2023-10-14T16:56:26.648228+0000","last_unstale":"2023-10-14T16:56:28.633034+0000","last_undegraded":"2023-10-14T16:56:28.633034+0000","last_fullsized":"2023-10-14T16:56:28.633034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:59.217099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651893+0000","last_change":"2023-10-14T16:56:25.632548+0000","last_active":"2023-10-14T16:56:27.651893+0000","last_peered":"2023-10-14T16:56:27.651893+0000","last_clean":"2023-10-14T16:56:27.651893+0000","last_became_active":"2023-10-14T16:56:25.632081+0000","last_became_peered":"2023-10-14T16:56:25.632081+0000","last_unstale":"2023-10-14T16:56:27.651893+0000","last_undegraded":"2023-10-14T16:56:27.651893+0000","last_fullsized":"2023-10-14T16:56:27.651893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:55.130276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646933+0000","last_change":"2023-10-14T16:56:26.666882+0000","last_active":"2023-10-14T16:56:27.646933+0000","last_peered":"2023-10-14T16:56:27.646933+0000","last_clean":"2023-10-14T16:56:27.646933+0000","last_became_active":"2023-10-14T16:56:26.661361+0000","last_became_peered":"2023-10-14T16:56:26.661361+0000","last_unstale":"2023-10-14T16:56:27.646933+0000","last_undegraded":"2023-10-14T16:56:27.646933+0000","last_fullsized":"2023-10-14T16:56:27.646933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:50:02.534701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635934+0000","last_change":"2023-10-14T16:56:25.646176+0000","last_active":"2023-10-14T16:56:28.635934+0000","last_peered":"2023-10-14T16:56:28.635934+0000","last_clean":"2023-10-14T16:56:28.635934+0000","last_became_active":"2023-10-14T16:56:25.645905+0000","last_became_peered":"2023-10-14T16:56:25.645905+0000","last_unstale":"2023-10-14T16:56:28.635934+0000","last_undegraded":"2023-10-14T16:56:28.635934+0000","last_fullsized":"2023-10-14T16:56:28.635934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:28.092816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635212+0000","last_change":"2023-10-14T16:56:26.653173+0000","last_active":"2023-10-14T16:56:28.635212+0000","last_peered":"2023-10-14T16:56:28.635212+0000","last_clean":"2023-10-14T16:56:28.635212+0000","last_became_active":"2023-10-14T16:56:26.650944+0000","last_became_peered":"2023-10-14T16:56:26.650944+0000","last_unstale":"2023-10-14T16:56:28.635212+0000","last_undegraded":"2023-10-14T16:56:28.635212+0000","last_fullsized":"2023-10-14T16:56:28.635212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:42.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632416+0000","last_change":"2023-10-14T16:56:25.644381+0000","last_active":"2023-10-14T16:56:28.632416+0000","last_peered":"2023-10-14T16:56:28.632416+0000","last_clean":"2023-10-14T16:56:28.632416+0000","last_became_active":"2023-10-14T16:56:25.644102+0000","last_became_peered":"2023-10-14T16:56:25.644102+0000","last_unstale":"2023-10-14T16:56:28.632416+0000","last_undegraded":"2023-10-14T16:56:28.632416+0000","last_fullsized":"2023-10-14T16:56:28.632416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:54:12.210244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634068+0000","last_change":"2023-10-14T16:56:26.668231+0000","last_active":"2023-10-14T16:56:28.634068+0000","last_peered":"2023-10-14T16:56:28.634068+0000","last_clean":"2023-10-14T16:56:28.634068+0000","last_became_active":"2023-10-14T16:56:26.667848+0000","last_became_peered":"2023-10-14T16:56:26.667848+0000","last_unstale":"2023-10-14T16:56:28.634068+0000","last_undegraded":"2023-10-14T16:56:28.634068+0000","last_fullsized":"2023-10-14T16:56:28.634068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:27.204944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650470+0000","last_change":"2023-10-14T16:56:25.636516+0000","last_active":"2023-10-14T16:56:27.650470+0000","last_peered":"2023-10-14T16:56:27.650470+0000","last_clean":"2023-10-14T16:56:27.650470+0000","last_became_active":"2023-10-14T16:56:25.636001+0000","last_became_peered":"2023-10-14T16:56:25.636001+0000","last_unstale":"2023-10-14T16:56:27.650470+0000","last_undegraded":"2023-10-14T16:56:27.650470+0000","last_fullsized":"2023-10-14T16:56:27.650470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:01.550372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632196+0000","last_change":"2023-10-14T16:56:26.644627+0000","last_active":"2023-10-14T16:56:28.632196+0000","last_peered":"2023-10-14T16:56:28.632196+0000","last_clean":"2023-10-14T16:56:28.632196+0000","last_became_active":"2023-10-14T16:56:26.641465+0000","last_became_peered":"2023-10-14T16:56:26.641465+0000","last_unstale":"2023-10-14T16:56:28.632196+0000","last_undegraded":"2023-10-14T16:56:28.632196+0000","last_fullsized":"2023-10-14T16:56:28.632196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:11:02.854968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632308+0000","last_change":"2023-10-14T16:56:25.635327+0000","last_active":"2023-10-14T16:56:28.632308+0000","last_peered":"2023-10-14T16:56:28.632308+0000","last_clean":"2023-10-14T16:56:28.632308+0000","last_became_active":"2023-10-14T16:56:25.635060+0000","last_became_peered":"2023-10-14T16:56:25.635060+0000","last_unstale":"2023-10-14T16:56:28.632308+0000","last_undegraded":"2023-10-14T16:56:28.632308+0000","last_fullsized":"2023-10-14T16:56:28.632308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:39.687227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634219+0000","last_change":"2023-10-14T16:56:26.642789+0000","last_active":"2023-10-14T16:56:28.634219+0000","last_peered":"2023-10-14T16:56:28.634219+0000","last_clean":"2023-10-14T16:56:28.634219+0000","last_became_active":"2023-10-14T16:56:26.641371+0000","last_became_peered":"2023-10-14T16:56:26.641371+0000","last_unstale":"2023-10-14T16:56:28.634219+0000","last_undegraded":"2023-10-14T16:56:28.634219+0000","last_fullsized":"2023-10-14T16:56:28.634219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:00:50.426954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635629+0000","last_change":"2023-10-14T16:56:25.644744+0000","last_active":"2023-10-14T16:56:28.635629+0000","last_peered":"2023-10-14T16:56:28.635629+0000","last_clean":"2023-10-14T16:56:28.635629+0000","last_became_active":"2023-10-14T16:56:25.644477+0000","last_became_peered":"2023-10-14T16:56:25.644477+0000","last_unstale":"2023-10-14T16:56:28.635629+0000","last_undegraded":"2023-10-14T16:56:28.635629+0000","last_fullsized":"2023-10-14T16:56:28.635629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:23.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635224+0000","last_change":"2023-10-14T16:56:26.664224+0000","last_active":"2023-10-14T16:56:28.635224+0000","last_peered":"2023-10-14T16:56:28.635224+0000","last_clean":"2023-10-14T16:56:28.635224+0000","last_became_active":"2023-10-14T16:56:26.659567+0000","last_became_peered":"2023-10-14T16:56:26.659567+0000","last_unstale":"2023-10-14T16:56:28.635224+0000","last_undegraded":"2023-10-14T16:56:28.635224+0000","last_fullsized":"2023-10-14T16:56:28.635224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:49.204116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651778+0000","last_change":"2023-10-14T16:56:25.646342+0000","last_active":"2023-10-14T16:56:27.651778+0000","last_peered":"2023-10-14T16:56:27.651778+0000","last_clean":"2023-10-14T16:56:27.651778+0000","last_became_active":"2023-10-14T16:56:25.642275+0000","last_became_peered":"2023-10-14T16:56:25.642275+0000","last_unstale":"2023-10-14T16:56:27.651778+0000","last_undegraded":"2023-10-14T16:56:27.651778+0000","last_fullsized":"2023-10-14T16:56:27.651778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:45.616981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"69'161","reported_seq":383,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-14T16:56:26.642695+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:56:26.642695+0000","last_peered":"2023-10-14T16:56:26.642695+0000","last_clean":"2023-10-14T16:56:26.642695+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:56:26.642695+0000","last_undegraded":"2023-10-14T16:56:26.642695+0000","last_fullsized":"2023-10-14T16:56:26.642695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":1057795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.163},"min":{"1min":0.80000000000000004,"5min":0.56599999999999995,"15min":0.503},"max":{"1min":1.732,"5min":1.732,"15min":1.897},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.143,"15min":1.2010000000000001},"min":{"1min":0.82299999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7230000000000001,"5min":1.8400000000000001,"15min":2.262},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2170000000000001,"15min":1.2330000000000001},"min":{"1min":0.93400000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.6060000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.26,"5min":1.2450000000000001,"15min":1.2509999999999999},"min":{"1min":0.93600000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6020000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.532}]},{"osd":2,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.91700000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.633,"5min":1.651,"15min":2.2389999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.29,"15min":1.3129999999999999},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.6870000000000001,"5min":1.7869999999999999,"15min":2.3500000000000001},"last":1.054}]},{"osd":3,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.3799999999999999,"15min":1.3680000000000001},"min":{"1min":0.95099999999999996,"5min":0.92400000000000004,"15min":0.77300000000000002},"max":{"1min":1.6539999999999999,"5min":1.875,"15min":2.4159999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.4650000000000001,"15min":1.456},"min":{"1min":1.2949999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.861,"5min":1.861,"15min":2.5819999999999999},"last":1.464}]},{"osd":4,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.252,"15min":1.2669999999999999},"min":{"1min":0.83099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.714,"5min":2.105,"15min":2.2959999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.333,"15min":1.321},"min":{"1min":0.90500000000000003,"5min":0.59799999999999998,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":1.899,"15min":2.1400000000000001},"last":1.3200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3700000000000001,"15min":1.393},"min":{"1min":0.92700000000000005,"5min":0.66000000000000003,"15min":0.63100000000000001},"max":{"1min":1.7889999999999999,"5min":1.885,"15min":3.5270000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.4630000000000001,"15min":1.456},"min":{"1min":1.117,"5min":0.61799999999999999,"15min":0.54200000000000004},"max":{"1min":1.8029999999999999,"5min":1.9359999999999999,"15min":2.1299999999999999},"last":1.6559999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.514,"15min":1.5369999999999999},"min":{"1min":1.2210000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.869,"5min":1.9890000000000001,"15min":2.3220000000000001},"last":1.732},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.452,"15min":1.4750000000000001},"min":{"1min":1.1040000000000001,"5min":0.64800000000000002,"15min":0.56200000000000006},"max":{"1min":1.7450000000000001,"5min":2.036,"15min":2.7040000000000002},"last":1.5069999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":473611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.165,"15min":1.1639999999999999},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.63700000000000001},"max":{"1min":1.6140000000000001,"5min":2.137,"15min":2.339},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.23,"5min":1.242,"15min":1.2130000000000001},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6419999999999999,"5min":2.0760000000000001,"15min":3.2509999999999999},"last":1.2569999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.194,"15min":1.2010000000000001},"min":{"1min":0.78400000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.631,"5min":2.802,"15min":4.3280000000000003},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.302},"min":{"1min":0.92800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.7030000000000001,"5min":2.512,"15min":3.431},"last":1.3169999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3899999999999999,"15min":1.3620000000000001},"min":{"1min":0.91300000000000003,"5min":0.81100000000000005,"15min":0.75900000000000001},"max":{"1min":1.9970000000000001,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.4219999999999999,"15min":1.4039999999999999},"min":{"1min":1.028,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6990000000000001,"5min":3.286,"15min":3.286},"last":1.1870000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.395,"15min":1.3819999999999999},"min":{"1min":0.93400000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.669,"5min":2.823,"15min":2.8319999999999999},"last":1.577},{"interface":"front","average":{"1min":1.379,"5min":1.4730000000000001,"15min":1.4239999999999999},"min":{"1min":1.0880000000000001,"5min":0.879,"15min":0.75600000000000001},"max":{"1min":1.774,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.659}]},{"osd":4,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3540000000000001,"15min":1.351},"min":{"1min":0.97399999999999998,"5min":0.81999999999999995,"15min":0.77200000000000002},"max":{"1min":1.7729999999999999,"5min":2.9460000000000002,"15min":3.0790000000000002},"last":1.145},{"interface":"front","average":{"1min":1.427,"5min":1.482,"15min":1.423},"min":{"1min":0.98099999999999998,"5min":0.88700000000000001,"15min":0.82999999999999996},"max":{"1min":1.8009999999999999,"5min":2.5750000000000002,"15min":2.738},"last":1.738}]},{"osd":5,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.421,"15min":1.3859999999999999},"min":{"1min":0.93000000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.665,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5900000000000001,"15min":1.5589999999999999},"min":{"1min":0.98999999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.843,"5min":3.0489999999999999,"15min":3.206},"last":1.4259999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6539999999999999,"15min":1.5880000000000001},"min":{"1min":1.204,"5min":1.117,"15min":0.90900000000000003},"max":{"1min":1.899,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.823},{"interface":"front","average":{"1min":1.512,"5min":1.5469999999999999,"15min":1.546},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.8680000000000001,"5min":2.7000000000000002,"15min":3.1320000000000001},"last":1.9670000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":467919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.119},"min":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.52200000000000002},"max":{"1min":1.5620000000000001,"5min":1.861,"15min":3.2000000000000002},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.1080000000000001,"15min":1.137},"min":{"1min":0.85699999999999998,"5min":0.68999999999999995,"15min":0.57099999999999995},"max":{"1min":1.4750000000000001,"5min":1.665,"15min":4.101},"last":1.1279999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.157,"15min":1.1839999999999999},"min":{"1min":0.91500000000000004,"5min":0.71599999999999997,"15min":0.63400000000000001},"max":{"1min":1.6910000000000001,"5min":1.7090000000000001,"15min":3.1659999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.206,"15min":1.232},"min":{"1min":0.86899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.524,"5min":1.7609999999999999,"15min":2.2549999999999999},"last":1.1970000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.196,"15min":1.212},"min":{"1min":0.81699999999999995,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4399999999999999,"5min":1.7150000000000001,"15min":4.3639999999999999},"last":1.333},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3320000000000001,"15min":1.3280000000000001},"min":{"1min":0.98499999999999999,"5min":0.84699999999999998,"15min":0.79300000000000004},"max":{"1min":1.7450000000000001,"5min":2.1030000000000002,"15min":4.3339999999999996},"last":1.4339999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.347,"15min":1.391},"min":{"1min":1.01,"5min":0.745,"15min":0.745},"max":{"1min":1.609,"5min":1.9279999999999999,"15min":2.431},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3999999999999999,"15min":1.417},"min":{"1min":0.94199999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.6870000000000001,"5min":1.8999999999999999,"15min":3.6890000000000001},"last":0.85199999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2689999999999999,"15min":1.292},"min":{"1min":0.87,"5min":0.79500000000000004,"15min":0.72199999999999998},"max":{"1min":1.5669999999999999,"5min":2.194,"15min":2.577},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.323,"15min":1.3580000000000001},"min":{"1min":0.85999999999999999,"5min":0.69699999999999995,"15min":0.66200000000000003},"max":{"1min":1.673,"5min":1.927,"15min":2.6030000000000002},"last":1.2529999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.401,"15min":1.393},"min":{"1min":0.97999999999999998,"5min":0.68899999999999995,"15min":0.68799999999999994},"max":{"1min":1.71,"5min":2.2869999999999999,"15min":3.5739999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.532,"15min":1.55},"min":{"1min":1.0920000000000001,"5min":1.0760000000000001,"15min":0.81499999999999995},"max":{"1min":1.788,"5min":2.2519999999999998,"15min":3.2949999999999999},"last":1.49}]},{"osd":7,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.506,"15min":1.5229999999999999},"min":{"1min":1.0109999999999999,"5min":0.86399999999999999,"15min":0.72299999999999998},"max":{"1min":1.726,"5min":2.032,"15min":3.0289999999999999},"last":1.403},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.546,"15min":1.526},"min":{"1min":1.147,"5min":1.0349999999999999,"15min":0.627},"max":{"1min":1.7130000000000001,"5min":2.3980000000000001,"15min":3.7869999999999999},"last":1.2130000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58532,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313308,"statfs":{"total":96636764160,"available":96576827392,"internally_reserved":0,"allocated":1003520,"data_stored":475223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1379999999999999,"15min":1.1619999999999999},"min":{"1min":0.72399999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.621,"5min":1.7050000000000001,"15min":4.3810000000000002},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.2609999999999999,"15min":1.24},"min":{"1min":0.86899999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.7170000000000001,"5min":1.972,"15min":4.4059999999999997},"last":1.554}]},{"osd":1,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.264,"15min":1.256},"min":{"1min":0.89400000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2490000000000001,"15min":1.2430000000000001},"min":{"1min":0.90300000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.704,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.90500000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2410000000000001,"15min":1.23},"min":{"1min":0.84799999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.599,"5min":2.169,"15min":2.169},"last":1.752},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.395,"15min":1.3759999999999999},"min":{"1min":0.96999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.823,"5min":1.966,"15min":2.7759999999999998},"last":1.3100000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.302,"15min":1.3120000000000001},"min":{"1min":0.93799999999999994,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.206},{"interface":"front","average":{"1min":1.379,"5min":1.3680000000000001,"15min":1.3979999999999999},"min":{"1min":1.0469999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.764,"5min":1.952,"15min":2.085},"last":1.611}]},{"osd":5,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.371,"15min":1.361},"min":{"1min":0.99099999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.7470000000000001,"5min":1.855,"15min":2.2320000000000002},"last":1.05},{"interface":"front","average":{"1min":1.306,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":1.0589999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.6839999999999999,"5min":1.7909999999999999,"15min":2.153},"last":1.6930000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":1.004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.8560000000000001,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5229999999999999,"15min":1.5369999999999999},"min":{"1min":1.181,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.915,"5min":2.214,"15min":2.6539999999999999},"last":1.6599999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4530000000000001,"15min":1.482},"min":{"1min":1.038,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.794,"5min":2.1269999999999998,"15min":2.2389999999999999},"last":1.49},{"interface":"front","average":{"1min":1.554,"5min":1.5409999999999999,"15min":1.5580000000000001},"min":{"1min":1.157,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.0209999999999999,"5min":2.3130000000000002,"15min":2.3999999999999999},"last":1.893}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40468,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331372,"statfs":{"total":96636764160,"available":96595324928,"internally_reserved":0,"allocated":987136,"data_stored":467323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91900000000000004,"15min":0.92600000000000005},"min":{"1min":0.53000000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.2709999999999999,"5min":1.4239999999999999,"15min":1.526},"last":1.141},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67900000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.4850000000000001,"5min":1.706,"15min":1.8799999999999999},"last":1.091}]},{"osd":1,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90200000000000002,"15min":0.91800000000000004},"min":{"1min":0.53800000000000003,"5min":0.48199999999999998,"15min":0.33600000000000002},"max":{"1min":1.129,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.141,"5min":1.069,"15min":1.1060000000000001},"min":{"1min":0.755,"5min":0.438,"15min":0.373},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.294},"last":1.0509999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3580000000000001,"15min":1.357},"min":{"1min":0.66900000000000004,"5min":0.57899999999999996,"15min":0.51600000000000001},"max":{"1min":1.7310000000000001,"5min":1.966,"15min":2.6819999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.74099999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.8560000000000001,"5min":2.0960000000000001,"15min":2.77},"last":1.373}]},{"osd":4,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3720000000000001,"15min":1.3700000000000001},"min":{"1min":1.0960000000000001,"5min":0.81699999999999995,"15min":0.77300000000000002},"max":{"1min":1.7989999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3959999999999999,"15min":1.4179999999999999},"min":{"1min":0.96799999999999997,"5min":0.76900000000000002,"15min":0.749},"max":{"1min":1.756,"5min":1.8080000000000001,"15min":3.9540000000000002},"last":1.3340000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.4059999999999999},"min":{"1min":0.878,"5min":0.78400000000000003,"15min":0.77500000000000002},"max":{"1min":1.722,"5min":2.516,"15min":3.0190000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.4339999999999999,"15min":1.45},"min":{"1min":1.0640000000000001,"5min":0.80600000000000005,"15min":0.77400000000000002},"max":{"1min":1.629,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5169999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4359999999999999,"15min":1.47},"min":{"1min":1.0620000000000001,"5min":0.78500000000000003,"15min":0.65100000000000002},"max":{"1min":1.9119999999999999,"5min":2.2250000000000001,"15min":4.0570000000000004},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5469999999999999},"min":{"1min":1.101,"5min":0.91400000000000003,"15min":0.79200000000000004},"max":{"1min":2.0070000000000001,"5min":2.1680000000000001,"15min":4.1900000000000004},"last":1.202}]},{"osd":7,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4890000000000001,"15min":1.51},"min":{"1min":0.92600000000000005,"5min":0.85199999999999998,"15min":0.745},"max":{"1min":1.698,"5min":2.871,"15min":2.9900000000000002},"last":1.226},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.55,"15min":1.556},"min":{"1min":1.004,"5min":0.85099999999999998,"15min":0.82899999999999996},"max":{"1min":1.962,"5min":2.907,"15min":2.9569999999999999},"last":1.4039999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40464,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331376,"statfs":{"total":96636764160,"available":96595329024,"internally_reserved":0,"allocated":987136,"data_stored":467323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88800000000000001,"15min":0.90700000000000003},"min":{"1min":0.55800000000000005,"5min":0.38900000000000001,"15min":0.34300000000000003},"max":{"1min":1.698,"5min":1.698,"15min":1.849},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0529999999999999,"15min":1.0489999999999999},"min":{"1min":0.67700000000000005,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.528,"5min":1.8080000000000001,"15min":3.2149999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.64300000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.337,"5min":1.4630000000000001,"15min":2.0419999999999998},"last":0.626},{"interface":"front","average":{"1min":1.288,"5min":1.2150000000000001,"15min":1.1830000000000001},"min":{"1min":0.876,"5min":0.54700000000000004,"15min":0.42099999999999999},"max":{"1min":1.736,"5min":2.069,"15min":2.2890000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1919999999999999,"15min":1.2310000000000001},"min":{"1min":0.72199999999999998,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.6599999999999999,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.359,"5min":1.2969999999999999,"15min":1.3220000000000001},"min":{"1min":1.02,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.7729999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.72299999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.4279999999999999,"15min":1.431},"min":{"1min":1.052,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.913,"5min":1.95,"15min":2.6259999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.341,"5min":1.3520000000000001,"15min":1.375},"min":{"1min":0.90500000000000003,"5min":0.71699999999999997,"15min":0.67900000000000005},"max":{"1min":1.6719999999999999,"5min":2.0750000000000002,"15min":3.0459999999999998},"last":1.179}]},{"osd":5,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":0.91600000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.734,"5min":2.0019999999999998,"15min":2.2389999999999999},"last":1.256},{"interface":"front","average":{"1min":1.575,"5min":1.526,"15min":1.5389999999999999},"min":{"1min":1.0640000000000001,"5min":0.85599999999999998,"15min":0.80000000000000004},"max":{"1min":1.857,"5min":2.2730000000000001,"15min":2.883},"last":1.29}]},{"osd":6,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.4330000000000001,"15min":1.458},"min":{"1min":0.99299999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.8759999999999999,"5min":2.0289999999999999,"15min":3.1200000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5149999999999999,"15min":1.5349999999999999},"min":{"1min":1.145,"5min":0.83599999999999997,"15min":0.70699999999999996},"max":{"1min":1.8560000000000001,"5min":2.105,"15min":2.3420000000000001},"last":1.369}]},{"osd":7,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.5029999999999999,"15min":1.544},"min":{"1min":1.052,"5min":0.86099999999999999,"15min":0.77400000000000002},"max":{"1min":1.7729999999999999,"5min":2.109,"15min":3.169},"last":1.341},{"interface":"front","average":{"1min":1.641,"5min":1.5720000000000001,"15min":1.6040000000000001},"min":{"1min":1.218,"5min":0.81200000000000006,"15min":0.755},"max":{"1min":1.9650000000000001,"5min":2.165,"15min":2.996},"last":1.504}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":995328,"data_stored":473701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89000000000000001,"15min":0.90500000000000003},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.438},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0429999999999999,"15min":1.071},"min":{"1min":0.46600000000000003,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.6819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.94399999999999995,"15min":0.95899999999999996},"min":{"1min":0.68500000000000005,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.655,"5min":1.758,"15min":1.758},"last":0.62},{"interface":"front","average":{"1min":1.054,"5min":1.1279999999999999,"15min":1.171},"min":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.45200000000000001},"max":{"1min":1.5229999999999999,"5min":2.0150000000000001,"15min":2.1499999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.276,"15min":1.2969999999999999},"min":{"1min":0.81299999999999994,"5min":0.49399999999999999,"15min":0.40999999999999998},"max":{"1min":1.6990000000000001,"5min":2.198,"15min":2.5339999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.167,"15min":1.2090000000000001},"min":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.434},"max":{"1min":1.663,"5min":1.8680000000000001,"15min":4.6799999999999997},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.409,"15min":1.429},"min":{"1min":1.0189999999999999,"5min":0.86699999999999999,"15min":0.83799999999999997},"max":{"1min":1.7490000000000001,"5min":2.0139999999999998,"15min":4.5789999999999997},"last":1.234},{"interface":"front","average":{"1min":1.272,"5min":1.2849999999999999,"15min":1.3109999999999999},"min":{"1min":0.88400000000000001,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.0840000000000001},"last":0.88100000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.343,"15min":1.371},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.393,"5min":1.4330000000000001,"15min":1.476},"min":{"1min":0.873,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.78,"5min":2.302,"15min":2.4980000000000002},"last":1.034}]},{"osd":6,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.468,"15min":1.514},"min":{"1min":0.873,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8680000000000001,"5min":2.3479999999999999,"15min":4.6360000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.423,"15min":1.476},"min":{"1min":0.94399999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.1070000000000002,"15min":2.2970000000000002},"last":1.1000000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5389999999999999,"15min":1.577},"min":{"1min":0.91700000000000004,"5min":0.87,"15min":0.79900000000000004},"max":{"1min":2.1930000000000001,"5min":2.3940000000000001,"15min":2.452},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.536,"15min":1.5900000000000001},"min":{"1min":0.95599999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.819,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":0.90400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1601536,"data_stored":1066173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89700000000000002,"15min":0.88600000000000001},"min":{"1min":0.64700000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6719999999999999,"5min":1.774,"15min":2.0979999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.101,"15min":1.091},"min":{"1min":0.75700000000000001,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":1.6859999999999999,"5min":1.95,"15min":1.95},"last":1.1259999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.0600000000000001,"15min":1.083},"min":{"1min":0.91500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5049999999999999,"5min":1.649,"15min":1.962},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.1970000000000001,"15min":1.2},"min":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.42699999999999999},"max":{"1min":1.599,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.248}]},{"osd":3,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.278,"15min":1.272},"min":{"1min":1.0529999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.849,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.452},{"interface":"front","average":{"1min":1.319,"5min":1.3540000000000001,"15min":1.3520000000000001},"min":{"1min":1.0349999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":1.764,"5min":2.593,"15min":2.593},"last":1.365}]},{"osd":4,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.276,"15min":1.272},"min":{"1min":0.96099999999999997,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.8440000000000001},"last":1.415},{"interface":"front","average":{"1min":1.46,"5min":1.4119999999999999,"15min":1.427},"min":{"1min":1.171,"5min":0.88600000000000001,"15min":0.77800000000000002},"max":{"1min":1.667,"5min":2.0019999999999998,"15min":2.3460000000000001},"last":1.7050000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3069999999999999,"15min":1.3220000000000001},"min":{"1min":1.0960000000000001,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":1.8220000000000001,"5min":1.9650000000000001,"15min":2.0539999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.516,"15min":1.52},"min":{"1min":1.236,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.024,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.504}]},{"osd":6,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5840000000000001,"15min":1.5880000000000001},"min":{"1min":1.2010000000000001,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":1.962,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.575},{"interface":"front","average":{"1min":1.419,"5min":1.409,"15min":1.399},"min":{"1min":1.044,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":1.722,"5min":2.0049999999999999,"15min":2.1850000000000001},"last":1.77}]},{"osd":7,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5609999999999999,"15min":1.55},"min":{"1min":1.3049999999999999,"5min":0.83899999999999997,"15min":0.83399999999999996},"max":{"1min":2.266,"5min":2.3410000000000002,"15min":2.5219999999999998},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.528,"15min":1.5249999999999999},"min":{"1min":1.117,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":1.7969999999999999,"5min":2.073,"15min":2.415},"last":2.0579999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:56:31.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:56:31.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.401+0000 7f776b59e640 1 Processor -- start 2023-10-14T16:56:31.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.401+0000 7f776b59e640 1 -- start start 2023-10-14T16:56:31.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.402+0000 7f776b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f776c15cfc0 0x7f776c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.402+0000 7f776b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776c0d2640 0x7f776c0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.402+0000 7f776b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f776c0d15a0 con 0x7f776c15cbe0 2023-10-14T16:56:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.402+0000 7f776b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c0d1880 con 0x7f776c0d2640 2023-10-14T16:56:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.402+0000 7f776b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f776c0d1b60 con 0x7f776c15cfc0 2023-10-14T16:56:31.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776c0d2640 0x7f776c0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f776c0d2640 0x7f776c0b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39738/0 (socket says 172.21.15.29:39738) 2023-10-14T16:56:31.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 -- 172.21.15.29:0/1443743338 learned_addr learned my addr 172.21.15.29:0/1443743338 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:31.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 -- 172.21.15.29:0/1443743338 >> v2:172.21.15.186:3300/0 conn(0x7f776c15cfc0 msgr2=0x7f776c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 --2- 172.21.15.29:0/1443743338 >> v2:172.21.15.186:3300/0 conn(0x7f776c15cfc0 0x7f776c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 -- 172.21.15.29:0/1443743338 >> v1:172.21.15.186:6789/0 conn(0x7f776c15cbe0 legacy=0x7f776c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.403+0000 7f776ad9d640 1 -- 172.21.15.29:0/1443743338 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f776c0d1e40 con 0x7f776c0d2640 2023-10-14T16:56:31.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.405+0000 7f776ad9d640 1 --2- 172.21.15.29:0/1443743338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c0d2640 0x7f776c0b4ab0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f776c0cf4b0 tx=0x7f776402c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfb834c5ce43f8cb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.405+0000 7f7769d9b640 1 -- 172.21.15.29:0/1443743338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f776402cbb0 con 0x7f776c0d2640 2023-10-14T16:56:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.405+0000 7f7769d9b640 1 -- 172.21.15.29:0/1443743338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f776402cd10 con 0x7f776c0d2640 2023-10-14T16:56:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.405+0000 7f7769d9b640 1 -- 172.21.15.29:0/1443743338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7764034610 con 0x7f776c0d2640 2023-10-14T16:56:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 -- 172.21.15.29:0/1443743338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c0d2640 msgr2=0x7f776c0b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 --2- 172.21.15.29:0/1443743338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c0d2640 0x7f776c0b4ab0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f776c0cf4b0 tx=0x7f776402c010 comp rx=0 tx=0).stop 2023-10-14T16:56:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 -- 172.21.15.29:0/1443743338 shutdown_connections 2023-10-14T16:56:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 --2- 172.21.15.29:0/1443743338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c0d2640 0x7f776c0b4ab0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 --2- 172.21.15.29:0/1443743338 >> v2:172.21.15.186:3300/0 conn(0x7f776c15cfc0 0x7f776c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 -- 172.21.15.29:0/1443743338 >> 172.21.15.29:0/1443743338 conn(0x7f776c0cf570 msgr2=0x7f776c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 -- 172.21.15.29:0/1443743338 shutdown_connections 2023-10-14T16:56:31.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.406+0000 7f776b59e640 1 -- 172.21.15.29:0/1443743338 wait complete. 2023-10-14T16:56:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.408+0000 7f776b59e640 1 Processor -- start 2023-10-14T16:56:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.408+0000 7f776b59e640 1 -- start start 2023-10-14T16:56:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c0d2640 0x7f776c1f2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f776c15cfc0 0x7f776c1f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0c6620 con 0x7f776c15cbe0 2023-10-14T16:56:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f776c0c7480 con 0x7f776c0d2640 2023-10-14T16:56:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f776c1f2680 con 0x7f776c15cfc0 2023-10-14T16:56:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f7761d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f7761d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39750/0 (socket says 172.21.15.29:39750) 2023-10-14T16:56:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f7761d9b640 1 -- 172.21.15.29:0/1069335732 learned_addr learned my addr 172.21.15.29:0/1069335732 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.409+0000 7f776a59c640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c0d2640 0x7f776c1f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 -- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f776c15cfc0 msgr2=0x7f776c1f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f776c15cfc0 0x7f776c1f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 -- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c0d2640 msgr2=0x7f776c1f2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c0d2640 0x7f776c1f2140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 -- 172.21.15.29:0/1069335732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7764009b30 con 0x7f776c15cbe0 2023-10-14T16:56:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f7761d9b640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f775400b440 tx=0x7f775400b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754011890 con 0x7f776c15cbe0 2023-10-14T16:56:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77540119f0 con 0x7f776c15cbe0 2023-10-14T16:56:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7754011da0 con 0x7f776c15cbe0 2023-10-14T16:56:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.410+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f776c0c6870 con 0x7f776c15cbe0 2023-10-14T16:56:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.411+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f776c0c6e30 con 0x7f776c15cbe0 2023-10-14T16:56:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.412+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f776c0c7610 con 0x7f776c15cbe0 2023-10-14T16:56:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.413+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f775401a400 con 0x7f776c15cbe0 2023-10-14T16:56:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.414+0000 7f77637fe640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 0x7f774c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.414+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f7754089ae0 con 0x7f776c15cbe0 2023-10-14T16:56:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.414+0000 7f776a59c640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 0x7f774c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.415+0000 7f776a59c640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 0x7f774c06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f775c000f60 tx=0x7f775c009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.419+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7754054a60 con 0x7f776c15cbe0 2023-10-14T16:56:31.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.613+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 --> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f776c0d1260 con 0x7f774c06afc0 2023-10-14T16:56:31.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.630+0000 7f77637fe640 1 -- 172.21.15.29:0/1069335732 <== mgr.6252 v2:172.21.15.29:6832/3108857887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286440 (secure 0 0 0) 0x7f775c001c70 con 0x7f774c06afc0 2023-10-14T16:56:31.632 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:31.636 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:56:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 msgr2=0x7f774c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 0x7f774c06d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f775c000f60 tx=0x7f775c009040 comp rx=0 tx=0).stop 2023-10-14T16:56:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 msgr2=0x7f776c1f5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f775400b440 tx=0x7f775400b900 comp rx=0 tx=0).stop 2023-10-14T16:56:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 shutdown_connections 2023-10-14T16:56:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f774c06afc0 0x7f774c06d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f776c15cfc0 0x7f776c1f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f776c15cbe0 0x7f776c1f5720 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.637+0000 7f776b59e640 1 --2- 172.21.15.29:0/1069335732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f776c0d2640 0x7f776c1f2140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.638+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 >> 172.21.15.29:0/1069335732 conn(0x7f776c0cf570 msgr2=0x7f776c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.638+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 shutdown_connections 2023-10-14T16:56:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.638+0000 7f776b59e640 1 -- 172.21.15.29:0/1069335732 wait complete. 2023-10-14T16:56:31.659 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T16:56:29.679347+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":283,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":199,"ondisk_log_size":199,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":133,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":336084,"kb_used_data":8920,"kb_used_omap":12,"kb_used_meta":327027,"kb_avail":754638636,"statfs":{"total":773094113280,"available":772749963264,"internally_reserved":0,"allocated":9134080,"data_stored":4949068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":334876216},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":28,"apply_latency_ms":28,"commit_latency_ns":28000000,"apply_latency_ns":28000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079554"},"pg_stats":[{"pgid":"9.3f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637315+0000","last_change":"2023-10-14T16:56:26.666843+0000","last_active":"2023-10-14T16:56:28.637315+0000","last_peered":"2023-10-14T16:56:28.637315+0000","last_clean":"2023-10-14T16:56:28.637315+0000","last_became_active":"2023-10-14T16:56:26.666569+0000","last_became_peered":"2023-10-14T16:56:26.666569+0000","last_unstale":"2023-10-14T16:56:28.637315+0000","last_undegraded":"2023-10-14T16:56:28.637315+0000","last_fullsized":"2023-10-14T16:56:28.637315+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:05.992321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636160+0000","last_change":"2023-10-14T16:56:26.663299+0000","last_active":"2023-10-14T16:56:28.636160+0000","last_peered":"2023-10-14T16:56:28.636160+0000","last_clean":"2023-10-14T16:56:28.636160+0000","last_became_active":"2023-10-14T16:56:26.662937+0000","last_became_peered":"2023-10-14T16:56:26.662937+0000","last_unstale":"2023-10-14T16:56:28.636160+0000","last_undegraded":"2023-10-14T16:56:28.636160+0000","last_fullsized":"2023-10-14T16:56:28.636160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:31.219070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.3d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633935+0000","last_change":"2023-10-14T16:56:26.651056+0000","last_active":"2023-10-14T16:56:28.633935+0000","last_peered":"2023-10-14T16:56:28.633935+0000","last_clean":"2023-10-14T16:56:28.633935+0000","last_became_active":"2023-10-14T16:56:26.650774+0000","last_became_peered":"2023-10-14T16:56:26.650774+0000","last_unstale":"2023-10-14T16:56:28.633935+0000","last_undegraded":"2023-10-14T16:56:28.633935+0000","last_fullsized":"2023-10-14T16:56:28.633935+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:10:46.090271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.3c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635597+0000","last_change":"2023-10-14T16:56:26.644485+0000","last_active":"2023-10-14T16:56:28.635597+0000","last_peered":"2023-10-14T16:56:28.635597+0000","last_clean":"2023-10-14T16:56:28.635597+0000","last_became_active":"2023-10-14T16:56:26.642602+0000","last_became_peered":"2023-10-14T16:56:26.642602+0000","last_unstale":"2023-10-14T16:56:28.635597+0000","last_undegraded":"2023-10-14T16:56:28.635597+0000","last_fullsized":"2023-10-14T16:56:28.635597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:18.732584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.3b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635103+0000","last_change":"2023-10-14T16:56:26.641166+0000","last_active":"2023-10-14T16:56:28.635103+0000","last_peered":"2023-10-14T16:56:28.635103+0000","last_clean":"2023-10-14T16:56:28.635103+0000","last_became_active":"2023-10-14T16:56:26.640870+0000","last_became_peered":"2023-10-14T16:56:26.640870+0000","last_unstale":"2023-10-14T16:56:28.635103+0000","last_undegraded":"2023-10-14T16:56:28.635103+0000","last_fullsized":"2023-10-14T16:56:28.635103+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:31.698157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.3a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646838+0000","last_change":"2023-10-14T16:56:26.655173+0000","last_active":"2023-10-14T16:56:27.646838+0000","last_peered":"2023-10-14T16:56:27.646838+0000","last_clean":"2023-10-14T16:56:27.646838+0000","last_became_active":"2023-10-14T16:56:26.654803+0000","last_became_peered":"2023-10-14T16:56:26.654803+0000","last_unstale":"2023-10-14T16:56:27.646838+0000","last_undegraded":"2023-10-14T16:56:27.646838+0000","last_fullsized":"2023-10-14T16:56:27.646838+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:19:32.916043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.39","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649703+0000","last_change":"2023-10-14T16:56:26.644806+0000","last_active":"2023-10-14T16:56:27.649703+0000","last_peered":"2023-10-14T16:56:27.649703+0000","last_clean":"2023-10-14T16:56:27.649703+0000","last_became_active":"2023-10-14T16:56:26.640771+0000","last_became_peered":"2023-10-14T16:56:26.640771+0000","last_unstale":"2023-10-14T16:56:27.649703+0000","last_undegraded":"2023-10-14T16:56:27.649703+0000","last_fullsized":"2023-10-14T16:56:27.649703+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:24:27.364564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.38","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635894+0000","last_change":"2023-10-14T16:56:26.651766+0000","last_active":"2023-10-14T16:56:28.635894+0000","last_peered":"2023-10-14T16:56:28.635894+0000","last_clean":"2023-10-14T16:56:28.635894+0000","last_became_active":"2023-10-14T16:56:26.651463+0000","last_became_peered":"2023-10-14T16:56:26.651463+0000","last_unstale":"2023-10-14T16:56:28.635894+0000","last_undegraded":"2023-10-14T16:56:28.635894+0000","last_fullsized":"2023-10-14T16:56:28.635894+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:40:52.293330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.37","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651038+0000","last_change":"2023-10-14T16:56:26.663236+0000","last_active":"2023-10-14T16:56:27.651038+0000","last_peered":"2023-10-14T16:56:27.651038+0000","last_clean":"2023-10-14T16:56:27.651038+0000","last_became_active":"2023-10-14T16:56:26.662601+0000","last_became_peered":"2023-10-14T16:56:26.662601+0000","last_unstale":"2023-10-14T16:56:27.651038+0000","last_undegraded":"2023-10-14T16:56:27.651038+0000","last_fullsized":"2023-10-14T16:56:27.651038+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:00.056508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.36","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633348+0000","last_change":"2023-10-14T16:56:26.647304+0000","last_active":"2023-10-14T16:56:28.633348+0000","last_peered":"2023-10-14T16:56:28.633348+0000","last_clean":"2023-10-14T16:56:28.633348+0000","last_became_active":"2023-10-14T16:56:26.646955+0000","last_became_peered":"2023-10-14T16:56:26.646955+0000","last_unstale":"2023-10-14T16:56:28.633348+0000","last_undegraded":"2023-10-14T16:56:28.633348+0000","last_fullsized":"2023-10-14T16:56:28.633348+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:02.946295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.35","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634043+0000","last_change":"2023-10-14T16:56:26.667439+0000","last_active":"2023-10-14T16:56:28.634043+0000","last_peered":"2023-10-14T16:56:28.634043+0000","last_clean":"2023-10-14T16:56:28.634043+0000","last_became_active":"2023-10-14T16:56:26.666805+0000","last_became_peered":"2023-10-14T16:56:26.666805+0000","last_unstale":"2023-10-14T16:56:28.634043+0000","last_undegraded":"2023-10-14T16:56:28.634043+0000","last_fullsized":"2023-10-14T16:56:28.634043+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:31.962545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.34","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635718+0000","last_change":"2023-10-14T16:56:26.647590+0000","last_active":"2023-10-14T16:56:28.635718+0000","last_peered":"2023-10-14T16:56:28.635718+0000","last_clean":"2023-10-14T16:56:28.635718+0000","last_became_active":"2023-10-14T16:56:26.647192+0000","last_became_peered":"2023-10-14T16:56:26.647192+0000","last_unstale":"2023-10-14T16:56:28.635718+0000","last_undegraded":"2023-10-14T16:56:28.635718+0000","last_fullsized":"2023-10-14T16:56:28.635718+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:55:35.417582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.33","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651960+0000","last_change":"2023-10-14T16:56:26.658224+0000","last_active":"2023-10-14T16:56:27.651960+0000","last_peered":"2023-10-14T16:56:27.651960+0000","last_clean":"2023-10-14T16:56:27.651960+0000","last_became_active":"2023-10-14T16:56:26.657896+0000","last_became_peered":"2023-10-14T16:56:26.657896+0000","last_unstale":"2023-10-14T16:56:27.651960+0000","last_undegraded":"2023-10-14T16:56:27.651960+0000","last_fullsized":"2023-10-14T16:56:27.651960+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:45:11.658283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.32","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651285+0000","last_change":"2023-10-14T16:56:26.649981+0000","last_active":"2023-10-14T16:56:27.651285+0000","last_peered":"2023-10-14T16:56:27.651285+0000","last_clean":"2023-10-14T16:56:27.651285+0000","last_became_active":"2023-10-14T16:56:26.646819+0000","last_became_peered":"2023-10-14T16:56:26.646819+0000","last_unstale":"2023-10-14T16:56:27.651285+0000","last_undegraded":"2023-10-14T16:56:27.651285+0000","last_fullsized":"2023-10-14T16:56:27.651285+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:28.315229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.31","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650247+0000","last_change":"2023-10-14T16:56:26.649023+0000","last_active":"2023-10-14T16:56:27.650247+0000","last_peered":"2023-10-14T16:56:27.650247+0000","last_clean":"2023-10-14T16:56:27.650247+0000","last_became_active":"2023-10-14T16:56:26.648716+0000","last_became_peered":"2023-10-14T16:56:26.648716+0000","last_unstale":"2023-10-14T16:56:27.650247+0000","last_undegraded":"2023-10-14T16:56:27.650247+0000","last_fullsized":"2023-10-14T16:56:27.650247+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:47:58.473208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.30","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646688+0000","last_change":"2023-10-14T16:56:26.658134+0000","last_active":"2023-10-14T16:56:27.646688+0000","last_peered":"2023-10-14T16:56:27.646688+0000","last_clean":"2023-10-14T16:56:27.646688+0000","last_became_active":"2023-10-14T16:56:26.657731+0000","last_became_peered":"2023-10-14T16:56:26.657731+0000","last_unstale":"2023-10-14T16:56:27.646688+0000","last_undegraded":"2023-10-14T16:56:27.646688+0000","last_fullsized":"2023-10-14T16:56:27.646688+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:04:26.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.2f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635915+0000","last_change":"2023-10-14T16:56:26.663466+0000","last_active":"2023-10-14T16:56:28.635915+0000","last_peered":"2023-10-14T16:56:28.635915+0000","last_clean":"2023-10-14T16:56:28.635915+0000","last_became_active":"2023-10-14T16:56:26.656381+0000","last_became_peered":"2023-10-14T16:56:26.656381+0000","last_unstale":"2023-10-14T16:56:28.635915+0000","last_undegraded":"2023-10-14T16:56:28.635915+0000","last_fullsized":"2023-10-14T16:56:28.635915+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:04.197142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634030+0000","last_change":"2023-10-14T16:56:26.660673+0000","last_active":"2023-10-14T16:56:28.634030+0000","last_peered":"2023-10-14T16:56:28.634030+0000","last_clean":"2023-10-14T16:56:28.634030+0000","last_became_active":"2023-10-14T16:56:26.660342+0000","last_became_peered":"2023-10-14T16:56:26.660342+0000","last_unstale":"2023-10-14T16:56:28.634030+0000","last_undegraded":"2023-10-14T16:56:28.634030+0000","last_fullsized":"2023-10-14T16:56:28.634030+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:24.440386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636805+0000","last_change":"2023-10-14T16:56:26.666156+0000","last_active":"2023-10-14T16:56:28.636805+0000","last_peered":"2023-10-14T16:56:28.636805+0000","last_clean":"2023-10-14T16:56:28.636805+0000","last_became_active":"2023-10-14T16:56:26.665868+0000","last_became_peered":"2023-10-14T16:56:26.665868+0000","last_unstale":"2023-10-14T16:56:28.636805+0000","last_undegraded":"2023-10-14T16:56:28.636805+0000","last_fullsized":"2023-10-14T16:56:28.636805+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:04:26.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.2c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636391+0000","last_change":"2023-10-14T16:56:26.666482+0000","last_active":"2023-10-14T16:56:28.636391+0000","last_peered":"2023-10-14T16:56:28.636391+0000","last_clean":"2023-10-14T16:56:28.636391+0000","last_became_active":"2023-10-14T16:56:26.665136+0000","last_became_peered":"2023-10-14T16:56:26.665136+0000","last_unstale":"2023-10-14T16:56:28.636391+0000","last_undegraded":"2023-10-14T16:56:28.636391+0000","last_fullsized":"2023-10-14T16:56:28.636391+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:34.986146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633746+0000","last_change":"2023-10-14T16:56:26.654125+0000","last_active":"2023-10-14T16:56:28.633746+0000","last_peered":"2023-10-14T16:56:28.633746+0000","last_clean":"2023-10-14T16:56:28.633746+0000","last_became_active":"2023-10-14T16:56:26.652025+0000","last_became_peered":"2023-10-14T16:56:26.652025+0000","last_unstale":"2023-10-14T16:56:28.633746+0000","last_undegraded":"2023-10-14T16:56:28.633746+0000","last_fullsized":"2023-10-14T16:56:28.633746+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:53:31.081821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.2a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633957+0000","last_change":"2023-10-14T16:56:26.653653+0000","last_active":"2023-10-14T16:56:28.633957+0000","last_peered":"2023-10-14T16:56:28.633957+0000","last_clean":"2023-10-14T16:56:28.633957+0000","last_became_active":"2023-10-14T16:56:26.653382+0000","last_became_peered":"2023-10-14T16:56:26.653382+0000","last_unstale":"2023-10-14T16:56:28.633957+0000","last_undegraded":"2023-10-14T16:56:28.633957+0000","last_fullsized":"2023-10-14T16:56:28.633957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:17.197738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.29","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633347+0000","last_change":"2023-10-14T16:56:26.664506+0000","last_active":"2023-10-14T16:56:28.633347+0000","last_peered":"2023-10-14T16:56:28.633347+0000","last_clean":"2023-10-14T16:56:28.633347+0000","last_became_active":"2023-10-14T16:56:26.664204+0000","last_became_peered":"2023-10-14T16:56:26.664204+0000","last_unstale":"2023-10-14T16:56:28.633347+0000","last_undegraded":"2023-10-14T16:56:28.633347+0000","last_fullsized":"2023-10-14T16:56:28.633347+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:44:48.701184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.28","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651149+0000","last_change":"2023-10-14T16:56:26.658829+0000","last_active":"2023-10-14T16:56:27.651149+0000","last_peered":"2023-10-14T16:56:27.651149+0000","last_clean":"2023-10-14T16:56:27.651149+0000","last_became_active":"2023-10-14T16:56:26.658167+0000","last_became_peered":"2023-10-14T16:56:26.658167+0000","last_unstale":"2023-10-14T16:56:27.651149+0000","last_undegraded":"2023-10-14T16:56:27.651149+0000","last_fullsized":"2023-10-14T16:56:27.651149+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:12:58.344442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.27","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637294+0000","last_change":"2023-10-14T16:56:26.644050+0000","last_active":"2023-10-14T16:56:28.637294+0000","last_peered":"2023-10-14T16:56:28.637294+0000","last_clean":"2023-10-14T16:56:28.637294+0000","last_became_active":"2023-10-14T16:56:26.643408+0000","last_became_peered":"2023-10-14T16:56:26.643408+0000","last_unstale":"2023-10-14T16:56:28.637294+0000","last_undegraded":"2023-10-14T16:56:28.637294+0000","last_fullsized":"2023-10-14T16:56:28.637294+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:43.080935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.26","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636502+0000","last_change":"2023-10-14T16:56:26.640086+0000","last_active":"2023-10-14T16:56:28.636502+0000","last_peered":"2023-10-14T16:56:28.636502+0000","last_clean":"2023-10-14T16:56:28.636502+0000","last_became_active":"2023-10-14T16:56:26.638238+0000","last_became_peered":"2023-10-14T16:56:26.638238+0000","last_unstale":"2023-10-14T16:56:28.636502+0000","last_undegraded":"2023-10-14T16:56:28.636502+0000","last_fullsized":"2023-10-14T16:56:28.636502+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:51:45.292889+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.25","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646645+0000","last_change":"2023-10-14T16:56:26.670716+0000","last_active":"2023-10-14T16:56:27.646645+0000","last_peered":"2023-10-14T16:56:27.646645+0000","last_clean":"2023-10-14T16:56:27.646645+0000","last_became_active":"2023-10-14T16:56:26.661708+0000","last_became_peered":"2023-10-14T16:56:26.661708+0000","last_unstale":"2023-10-14T16:56:27.646645+0000","last_undegraded":"2023-10-14T16:56:27.646645+0000","last_fullsized":"2023-10-14T16:56:27.646645+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:33.952293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.24","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636786+0000","last_change":"2023-10-14T16:56:26.649342+0000","last_active":"2023-10-14T16:56:28.636786+0000","last_peered":"2023-10-14T16:56:28.636786+0000","last_clean":"2023-10-14T16:56:28.636786+0000","last_became_active":"2023-10-14T16:56:26.648978+0000","last_became_peered":"2023-10-14T16:56:26.648978+0000","last_unstale":"2023-10-14T16:56:28.636786+0000","last_undegraded":"2023-10-14T16:56:28.636786+0000","last_fullsized":"2023-10-14T16:56:28.636786+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:33.952293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.23","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635436+0000","last_change":"2023-10-14T16:56:26.656387+0000","last_active":"2023-10-14T16:56:28.635436+0000","last_peered":"2023-10-14T16:56:28.635436+0000","last_clean":"2023-10-14T16:56:28.635436+0000","last_became_active":"2023-10-14T16:56:26.656077+0000","last_became_peered":"2023-10-14T16:56:26.656077+0000","last_unstale":"2023-10-14T16:56:28.635436+0000","last_undegraded":"2023-10-14T16:56:28.635436+0000","last_fullsized":"2023-10-14T16:56:28.635436+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:11.525157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.22","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633280+0000","last_change":"2023-10-14T16:56:26.652520+0000","last_active":"2023-10-14T16:56:28.633280+0000","last_peered":"2023-10-14T16:56:28.633280+0000","last_clean":"2023-10-14T16:56:28.633280+0000","last_became_active":"2023-10-14T16:56:26.651212+0000","last_became_peered":"2023-10-14T16:56:26.651212+0000","last_unstale":"2023-10-14T16:56:28.633280+0000","last_undegraded":"2023-10-14T16:56:28.633280+0000","last_fullsized":"2023-10-14T16:56:28.633280+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:45.300948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.21","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634938+0000","last_change":"2023-10-14T16:56:26.657709+0000","last_active":"2023-10-14T16:56:28.634938+0000","last_peered":"2023-10-14T16:56:28.634938+0000","last_clean":"2023-10-14T16:56:28.634938+0000","last_became_active":"2023-10-14T16:56:26.657248+0000","last_became_peered":"2023-10-14T16:56:26.657248+0000","last_unstale":"2023-10-14T16:56:28.634938+0000","last_undegraded":"2023-10-14T16:56:28.634938+0000","last_fullsized":"2023-10-14T16:56:28.634938+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:35.156905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.20","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646791+0000","last_change":"2023-10-14T16:56:26.673025+0000","last_active":"2023-10-14T16:56:27.646791+0000","last_peered":"2023-10-14T16:56:27.646791+0000","last_clean":"2023-10-14T16:56:27.646791+0000","last_became_active":"2023-10-14T16:56:26.668592+0000","last_became_peered":"2023-10-14T16:56:26.668592+0000","last_unstale":"2023-10-14T16:56:27.646791+0000","last_undegraded":"2023-10-14T16:56:27.646791+0000","last_fullsized":"2023-10-14T16:56:27.646791+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:53:35.861811+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647282+0000","last_change":"2023-10-14T16:56:26.658692+0000","last_active":"2023-10-14T16:56:27.647282+0000","last_peered":"2023-10-14T16:56:27.647282+0000","last_clean":"2023-10-14T16:56:27.647282+0000","last_became_active":"2023-10-14T16:56:26.657792+0000","last_became_peered":"2023-10-14T16:56:26.657792+0000","last_unstale":"2023-10-14T16:56:27.647282+0000","last_undegraded":"2023-10-14T16:56:27.647282+0000","last_fullsized":"2023-10-14T16:56:27.647282+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:58.290941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1f","version":"81'4","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647209+0000","last_change":"2023-10-14T16:56:25.643488+0000","last_active":"2023-10-14T16:56:27.647209+0000","last_peered":"2023-10-14T16:56:27.647209+0000","last_clean":"2023-10-14T16:56:27.647209+0000","last_became_active":"2023-10-14T16:56:25.643222+0000","last_became_peered":"2023-10-14T16:56:25.643222+0000","last_unstale":"2023-10-14T16:56:27.647209+0000","last_undegraded":"2023-10-14T16:56:27.647209+0000","last_fullsized":"2023-10-14T16:56:27.647209+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:15:37.864783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633735+0000","last_change":"2023-10-14T16:56:26.650938+0000","last_active":"2023-10-14T16:56:28.633735+0000","last_peered":"2023-10-14T16:56:28.633735+0000","last_clean":"2023-10-14T16:56:28.633735+0000","last_became_active":"2023-10-14T16:56:26.648274+0000","last_became_peered":"2023-10-14T16:56:26.648274+0000","last_unstale":"2023-10-14T16:56:28.633735+0000","last_undegraded":"2023-10-14T16:56:28.633735+0000","last_fullsized":"2023-10-14T16:56:28.633735+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:28.160672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1e","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637392+0000","last_change":"2023-10-14T16:56:25.643219+0000","last_active":"2023-10-14T16:56:28.637392+0000","last_peered":"2023-10-14T16:56:28.637392+0000","last_clean":"2023-10-14T16:56:28.637392+0000","last_became_active":"2023-10-14T16:56:25.642932+0000","last_became_peered":"2023-10-14T16:56:25.642932+0000","last_unstale":"2023-10-14T16:56:28.637392+0000","last_undegraded":"2023-10-14T16:56:28.637392+0000","last_fullsized":"2023-10-14T16:56:28.637392+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.370194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647135+0000","last_change":"2023-10-14T16:56:26.661490+0000","last_active":"2023-10-14T16:56:27.647135+0000","last_peered":"2023-10-14T16:56:27.647135+0000","last_clean":"2023-10-14T16:56:27.647135+0000","last_became_active":"2023-10-14T16:56:26.661194+0000","last_became_peered":"2023-10-14T16:56:26.661194+0000","last_unstale":"2023-10-14T16:56:27.647135+0000","last_undegraded":"2023-10-14T16:56:27.647135+0000","last_fullsized":"2023-10-14T16:56:27.647135+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:40.702394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647032+0000","last_change":"2023-10-14T16:56:25.645125+0000","last_active":"2023-10-14T16:56:27.647032+0000","last_peered":"2023-10-14T16:56:27.647032+0000","last_clean":"2023-10-14T16:56:27.647032+0000","last_became_active":"2023-10-14T16:56:25.644080+0000","last_became_peered":"2023-10-14T16:56:25.644080+0000","last_unstale":"2023-10-14T16:56:27.647032+0000","last_undegraded":"2023-10-14T16:56:27.647032+0000","last_fullsized":"2023-10-14T16:56:27.647032+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:30.682990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646784+0000","last_change":"2023-10-14T16:56:26.673690+0000","last_active":"2023-10-14T16:56:27.646784+0000","last_peered":"2023-10-14T16:56:27.646784+0000","last_clean":"2023-10-14T16:56:27.646784+0000","last_became_active":"2023-10-14T16:56:26.665731+0000","last_became_peered":"2023-10-14T16:56:26.665731+0000","last_unstale":"2023-10-14T16:56:27.646784+0000","last_undegraded":"2023-10-14T16:56:27.646784+0000","last_fullsized":"2023-10-14T16:56:27.646784+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:05.992321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1c","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635838+0000","last_change":"2023-10-14T16:56:25.650292+0000","last_active":"2023-10-14T16:56:28.635838+0000","last_peered":"2023-10-14T16:56:28.635838+0000","last_clean":"2023-10-14T16:56:28.635838+0000","last_became_active":"2023-10-14T16:56:25.649957+0000","last_became_peered":"2023-10-14T16:56:25.649957+0000","last_unstale":"2023-10-14T16:56:28.635838+0000","last_undegraded":"2023-10-14T16:56:28.635838+0000","last_fullsized":"2023-10-14T16:56:28.635838+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:18:43.362749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634472+0000","last_change":"2023-10-14T16:56:26.653567+0000","last_active":"2023-10-14T16:56:28.634472+0000","last_peered":"2023-10-14T16:56:28.634472+0000","last_clean":"2023-10-14T16:56:28.634472+0000","last_became_active":"2023-10-14T16:56:26.653291+0000","last_became_peered":"2023-10-14T16:56:26.653291+0000","last_unstale":"2023-10-14T16:56:28.634472+0000","last_undegraded":"2023-10-14T16:56:28.634472+0000","last_fullsized":"2023-10-14T16:56:28.634472+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:38:35.976676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1b","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650717+0000","last_change":"2023-10-14T16:56:25.636908+0000","last_active":"2023-10-14T16:56:27.650717+0000","last_peered":"2023-10-14T16:56:27.650717+0000","last_clean":"2023-10-14T16:56:27.650717+0000","last_became_active":"2023-10-14T16:56:25.636500+0000","last_became_peered":"2023-10-14T16:56:25.636500+0000","last_unstale":"2023-10-14T16:56:27.650717+0000","last_undegraded":"2023-10-14T16:56:27.650717+0000","last_fullsized":"2023-10-14T16:56:27.650717+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:33.815067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635063+0000","last_change":"2023-10-14T16:56:26.666970+0000","last_active":"2023-10-14T16:56:28.635063+0000","last_peered":"2023-10-14T16:56:28.635063+0000","last_clean":"2023-10-14T16:56:28.635063+0000","last_became_active":"2023-10-14T16:56:26.660155+0000","last_became_peered":"2023-10-14T16:56:26.660155+0000","last_unstale":"2023-10-14T16:56:28.635063+0000","last_undegraded":"2023-10-14T16:56:28.635063+0000","last_fullsized":"2023-10-14T16:56:28.635063+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:18.391549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1a","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647330+0000","last_change":"2023-10-14T16:56:25.640146+0000","last_active":"2023-10-14T16:56:27.647330+0000","last_peered":"2023-10-14T16:56:27.647330+0000","last_clean":"2023-10-14T16:56:27.647330+0000","last_became_active":"2023-10-14T16:56:25.639854+0000","last_became_peered":"2023-10-14T16:56:25.639854+0000","last_unstale":"2023-10-14T16:56:27.647330+0000","last_undegraded":"2023-10-14T16:56:27.647330+0000","last_fullsized":"2023-10-14T16:56:27.647330+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:34.592548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651536+0000","last_change":"2023-10-14T16:56:26.649084+0000","last_active":"2023-10-14T16:56:27.651536+0000","last_peered":"2023-10-14T16:56:27.651536+0000","last_clean":"2023-10-14T16:56:27.651536+0000","last_became_active":"2023-10-14T16:56:26.648783+0000","last_became_peered":"2023-10-14T16:56:26.648783+0000","last_unstale":"2023-10-14T16:56:27.651536+0000","last_undegraded":"2023-10-14T16:56:27.651536+0000","last_fullsized":"2023-10-14T16:56:27.651536+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:16.712989+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633470+0000","last_change":"2023-10-14T16:56:25.647288+0000","last_active":"2023-10-14T16:56:28.633470+0000","last_peered":"2023-10-14T16:56:28.633470+0000","last_clean":"2023-10-14T16:56:28.633470+0000","last_became_active":"2023-10-14T16:56:25.646903+0000","last_became_peered":"2023-10-14T16:56:25.646903+0000","last_unstale":"2023-10-14T16:56:28.633470+0000","last_undegraded":"2023-10-14T16:56:28.633470+0000","last_fullsized":"2023-10-14T16:56:28.633470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:44:10.364683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.19","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634150+0000","last_change":"2023-10-14T16:56:26.643883+0000","last_active":"2023-10-14T16:56:28.634150+0000","last_peered":"2023-10-14T16:56:28.634150+0000","last_clean":"2023-10-14T16:56:28.634150+0000","last_became_active":"2023-10-14T16:56:26.643594+0000","last_became_peered":"2023-10-14T16:56:26.643594+0000","last_unstale":"2023-10-14T16:56:28.634150+0000","last_undegraded":"2023-10-14T16:56:28.634150+0000","last_fullsized":"2023-10-14T16:56:28.634150+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:14.882523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649774+0000","last_change":"2023-10-14T16:56:25.643963+0000","last_active":"2023-10-14T16:56:27.649774+0000","last_peered":"2023-10-14T16:56:27.649774+0000","last_clean":"2023-10-14T16:56:27.649774+0000","last_became_active":"2023-10-14T16:56:25.643635+0000","last_became_peered":"2023-10-14T16:56:25.643635+0000","last_unstale":"2023-10-14T16:56:27.649774+0000","last_undegraded":"2023-10-14T16:56:27.649774+0000","last_fullsized":"2023-10-14T16:56:27.649774+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:09:30.864016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651404+0000","last_change":"2023-10-14T16:56:26.654103+0000","last_active":"2023-10-14T16:56:27.651404+0000","last_peered":"2023-10-14T16:56:27.651404+0000","last_clean":"2023-10-14T16:56:27.651404+0000","last_became_active":"2023-10-14T16:56:26.653814+0000","last_became_peered":"2023-10-14T16:56:26.653814+0000","last_unstale":"2023-10-14T16:56:27.651404+0000","last_undegraded":"2023-10-14T16:56:27.651404+0000","last_fullsized":"2023-10-14T16:56:27.651404+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:45:46.099622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634821+0000","last_change":"2023-10-14T16:56:25.648712+0000","last_active":"2023-10-14T16:56:28.634821+0000","last_peered":"2023-10-14T16:56:28.634821+0000","last_clean":"2023-10-14T16:56:28.634821+0000","last_became_active":"2023-10-14T16:56:25.647900+0000","last_became_peered":"2023-10-14T16:56:25.647900+0000","last_unstale":"2023-10-14T16:56:28.634821+0000","last_undegraded":"2023-10-14T16:56:28.634821+0000","last_fullsized":"2023-10-14T16:56:28.634821+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:17.717418+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634895+0000","last_change":"2023-10-14T16:56:26.655659+0000","last_active":"2023-10-14T16:56:28.634895+0000","last_peered":"2023-10-14T16:56:28.634895+0000","last_clean":"2023-10-14T16:56:28.634895+0000","last_became_active":"2023-10-14T16:56:26.655265+0000","last_became_peered":"2023-10-14T16:56:26.655265+0000","last_unstale":"2023-10-14T16:56:28.634895+0000","last_undegraded":"2023-10-14T16:56:28.634895+0000","last_fullsized":"2023-10-14T16:56:28.634895+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:05.887144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636809+0000","last_change":"2023-10-14T16:56:25.640983+0000","last_active":"2023-10-14T16:56:28.636809+0000","last_peered":"2023-10-14T16:56:28.636809+0000","last_clean":"2023-10-14T16:56:28.636809+0000","last_became_active":"2023-10-14T16:56:25.640119+0000","last_became_peered":"2023-10-14T16:56:25.640119+0000","last_unstale":"2023-10-14T16:56:28.636809+0000","last_undegraded":"2023-10-14T16:56:28.636809+0000","last_fullsized":"2023-10-14T16:56:28.636809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:26.612413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632957+0000","last_change":"2023-10-14T16:56:26.641261+0000","last_active":"2023-10-14T16:56:28.632957+0000","last_peered":"2023-10-14T16:56:28.632957+0000","last_clean":"2023-10-14T16:56:28.632957+0000","last_became_active":"2023-10-14T16:56:26.640953+0000","last_became_peered":"2023-10-14T16:56:26.640953+0000","last_unstale":"2023-10-14T16:56:28.632957+0000","last_undegraded":"2023-10-14T16:56:28.632957+0000","last_fullsized":"2023-10-14T16:56:28.632957+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:14:43.771856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.15","version":"81'3","reported_seq":15,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634518+0000","last_change":"2023-10-14T16:56:25.649816+0000","last_active":"2023-10-14T16:56:28.634518+0000","last_peered":"2023-10-14T16:56:28.634518+0000","last_clean":"2023-10-14T16:56:28.634518+0000","last_became_active":"2023-10-14T16:56:25.649028+0000","last_became_peered":"2023-10-14T16:56:25.649028+0000","last_unstale":"2023-10-14T16:56:28.634518+0000","last_undegraded":"2023-10-14T16:56:28.634518+0000","last_fullsized":"2023-10-14T16:56:28.634518+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:07.276627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634283+0000","last_change":"2023-10-14T16:56:26.655469+0000","last_active":"2023-10-14T16:56:28.634283+0000","last_peered":"2023-10-14T16:56:28.634283+0000","last_clean":"2023-10-14T16:56:28.634283+0000","last_became_active":"2023-10-14T16:56:26.655162+0000","last_became_peered":"2023-10-14T16:56:26.655162+0000","last_unstale":"2023-10-14T16:56:28.634283+0000","last_undegraded":"2023-10-14T16:56:28.634283+0000","last_fullsized":"2023-10-14T16:56:28.634283+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:27.627579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.14","version":"81'6","reported_seq":18,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651119+0000","last_change":"2023-10-14T16:56:25.644403+0000","last_active":"2023-10-14T16:56:27.651119+0000","last_peered":"2023-10-14T16:56:27.651119+0000","last_clean":"2023-10-14T16:56:27.651119+0000","last_became_active":"2023-10-14T16:56:25.643086+0000","last_became_peered":"2023-10-14T16:56:25.643086+0000","last_unstale":"2023-10-14T16:56:27.651119+0000","last_undegraded":"2023-10-14T16:56:27.651119+0000","last_fullsized":"2023-10-14T16:56:27.651119+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:14.208391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634219+0000","last_change":"2023-10-14T16:56:26.645925+0000","last_active":"2023-10-14T16:56:28.634219+0000","last_peered":"2023-10-14T16:56:28.634219+0000","last_clean":"2023-10-14T16:56:28.634219+0000","last_became_active":"2023-10-14T16:56:26.645401+0000","last_became_peered":"2023-10-14T16:56:26.645401+0000","last_unstale":"2023-10-14T16:56:28.634219+0000","last_undegraded":"2023-10-14T16:56:28.634219+0000","last_fullsized":"2023-10-14T16:56:28.634219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:37:17.646767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.13","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635997+0000","last_change":"2023-10-14T16:56:25.642652+0000","last_active":"2023-10-14T16:56:28.635997+0000","last_peered":"2023-10-14T16:56:28.635997+0000","last_clean":"2023-10-14T16:56:28.635997+0000","last_became_active":"2023-10-14T16:56:25.641433+0000","last_became_peered":"2023-10-14T16:56:25.641433+0000","last_unstale":"2023-10-14T16:56:28.635997+0000","last_undegraded":"2023-10-14T16:56:28.635997+0000","last_fullsized":"2023-10-14T16:56:28.635997+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:17.542259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633887+0000","last_change":"2023-10-14T16:56:26.648280+0000","last_active":"2023-10-14T16:56:28.633887+0000","last_peered":"2023-10-14T16:56:28.633887+0000","last_clean":"2023-10-14T16:56:28.633887+0000","last_became_active":"2023-10-14T16:56:26.647953+0000","last_became_peered":"2023-10-14T16:56:26.647953+0000","last_unstale":"2023-10-14T16:56:28.633887+0000","last_undegraded":"2023-10-14T16:56:28.633887+0000","last_fullsized":"2023-10-14T16:56:28.633887+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:18.557426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.652091+0000","last_change":"2023-10-14T16:56:25.640577+0000","last_active":"2023-10-14T16:56:27.652091+0000","last_peered":"2023-10-14T16:56:27.652091+0000","last_clean":"2023-10-14T16:56:27.652091+0000","last_became_active":"2023-10-14T16:56:25.639993+0000","last_became_peered":"2023-10-14T16:56:25.639993+0000","last_unstale":"2023-10-14T16:56:27.652091+0000","last_undegraded":"2023-10-14T16:56:27.652091+0000","last_fullsized":"2023-10-14T16:56:27.652091+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:08.972626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.649872+0000","last_change":"2023-10-14T16:56:26.647114+0000","last_active":"2023-10-14T16:56:27.649872+0000","last_peered":"2023-10-14T16:56:27.649872+0000","last_clean":"2023-10-14T16:56:27.649872+0000","last_became_active":"2023-10-14T16:56:26.646832+0000","last_became_peered":"2023-10-14T16:56:26.646832+0000","last_unstale":"2023-10-14T16:56:27.649872+0000","last_undegraded":"2023-10-14T16:56:27.649872+0000","last_fullsized":"2023-10-14T16:56:27.649872+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:28:52.159302+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633700+0000","last_change":"2023-10-14T16:56:25.650013+0000","last_active":"2023-10-14T16:56:28.633700+0000","last_peered":"2023-10-14T16:56:28.633700+0000","last_clean":"2023-10-14T16:56:28.633700+0000","last_became_active":"2023-10-14T16:56:25.649703+0000","last_became_peered":"2023-10-14T16:56:25.649703+0000","last_unstale":"2023-10-14T16:56:28.633700+0000","last_undegraded":"2023-10-14T16:56:28.633700+0000","last_fullsized":"2023-10-14T16:56:28.633700+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:54.440032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633376+0000","last_change":"2023-10-14T16:56:26.665210+0000","last_active":"2023-10-14T16:56:28.633376+0000","last_peered":"2023-10-14T16:56:28.633376+0000","last_clean":"2023-10-14T16:56:28.633376+0000","last_became_active":"2023-10-14T16:56:26.664862+0000","last_became_peered":"2023-10-14T16:56:26.664862+0000","last_unstale":"2023-10-14T16:56:28.633376+0000","last_undegraded":"2023-10-14T16:56:28.633376+0000","last_fullsized":"2023-10-14T16:56:28.633376+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:53.197186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.10","version":"81'8","reported_seq":20,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632980+0000","last_change":"2023-10-14T16:56:25.649885+0000","last_active":"2023-10-14T16:56:28.632980+0000","last_peered":"2023-10-14T16:56:28.632980+0000","last_clean":"2023-10-14T16:56:28.632980+0000","last_became_active":"2023-10-14T16:56:25.649573+0000","last_became_peered":"2023-10-14T16:56:25.649573+0000","last_unstale":"2023-10-14T16:56:28.632980+0000","last_undegraded":"2023-10-14T16:56:28.632980+0000","last_fullsized":"2023-10-14T16:56:28.632980+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:18:43.362749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634305+0000","last_change":"2023-10-14T16:56:26.651016+0000","last_active":"2023-10-14T16:56:28.634305+0000","last_peered":"2023-10-14T16:56:28.634305+0000","last_clean":"2023-10-14T16:56:28.634305+0000","last_became_active":"2023-10-14T16:56:26.650703+0000","last_became_peered":"2023-10-14T16:56:26.650703+0000","last_unstale":"2023-10-14T16:56:28.634305+0000","last_undegraded":"2023-10-14T16:56:28.634305+0000","last_fullsized":"2023-10-14T16:56:28.634305+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:35.385361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.f","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636295+0000","last_change":"2023-10-14T16:56:25.642862+0000","last_active":"2023-10-14T16:56:28.636295+0000","last_peered":"2023-10-14T16:56:28.636295+0000","last_clean":"2023-10-14T16:56:28.636295+0000","last_became_active":"2023-10-14T16:56:25.642560+0000","last_became_peered":"2023-10-14T16:56:25.642560+0000","last_unstale":"2023-10-14T16:56:28.636295+0000","last_undegraded":"2023-10-14T16:56:28.636295+0000","last_fullsized":"2023-10-14T16:56:28.636295+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:26.189777+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636597+0000","last_change":"2023-10-14T16:56:26.645263+0000","last_active":"2023-10-14T16:56:28.636597+0000","last_peered":"2023-10-14T16:56:28.636597+0000","last_clean":"2023-10-14T16:56:28.636597+0000","last_became_active":"2023-10-14T16:56:26.644911+0000","last_became_peered":"2023-10-14T16:56:26.644911+0000","last_unstale":"2023-10-14T16:56:28.636597+0000","last_undegraded":"2023-10-14T16:56:28.636597+0000","last_fullsized":"2023-10-14T16:56:28.636597+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:08.291793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.e","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647218+0000","last_change":"2023-10-14T16:56:25.641798+0000","last_active":"2023-10-14T16:56:27.647218+0000","last_peered":"2023-10-14T16:56:27.647218+0000","last_clean":"2023-10-14T16:56:27.647218+0000","last_became_active":"2023-10-14T16:56:25.639123+0000","last_became_peered":"2023-10-14T16:56:25.639123+0000","last_unstale":"2023-10-14T16:56:27.647218+0000","last_undegraded":"2023-10-14T16:56:27.647218+0000","last_fullsized":"2023-10-14T16:56:27.647218+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:30:32.661864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.637100+0000","last_change":"2023-10-14T16:56:26.654982+0000","last_active":"2023-10-14T16:56:28.637100+0000","last_peered":"2023-10-14T16:56:28.637100+0000","last_clean":"2023-10-14T16:56:28.637100+0000","last_became_active":"2023-10-14T16:56:26.652898+0000","last_became_peered":"2023-10-14T16:56:26.652898+0000","last_unstale":"2023-10-14T16:56:28.637100+0000","last_undegraded":"2023-10-14T16:56:28.637100+0000","last_fullsized":"2023-10-14T16:56:28.637100+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:53.647679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"81'1","reported_seq":13,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636240+0000","last_change":"2023-10-14T16:56:25.647622+0000","last_active":"2023-10-14T16:56:28.636240+0000","last_peered":"2023-10-14T16:56:28.636240+0000","last_clean":"2023-10-14T16:56:28.636240+0000","last_became_active":"2023-10-14T16:56:25.647291+0000","last_became_peered":"2023-10-14T16:56:25.647291+0000","last_unstale":"2023-10-14T16:56:28.636240+0000","last_undegraded":"2023-10-14T16:56:28.636240+0000","last_fullsized":"2023-10-14T16:56:28.636240+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:11:17.476895+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634477+0000","last_change":"2023-10-14T16:56:26.658026+0000","last_active":"2023-10-14T16:56:28.634477+0000","last_peered":"2023-10-14T16:56:28.634477+0000","last_clean":"2023-10-14T16:56:28.634477+0000","last_became_active":"2023-10-14T16:56:26.657716+0000","last_became_peered":"2023-10-14T16:56:26.657716+0000","last_unstale":"2023-10-14T16:56:28.634477+0000","last_undegraded":"2023-10-14T16:56:28.634477+0000","last_fullsized":"2023-10-14T16:56:28.634477+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:20.604773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647239+0000","last_change":"2023-10-14T16:56:25.637124+0000","last_active":"2023-10-14T16:56:27.647239+0000","last_peered":"2023-10-14T16:56:27.647239+0000","last_clean":"2023-10-14T16:56:27.647239+0000","last_became_active":"2023-10-14T16:56:25.636785+0000","last_became_peered":"2023-10-14T16:56:25.636785+0000","last_unstale":"2023-10-14T16:56:27.647239+0000","last_undegraded":"2023-10-14T16:56:27.647239+0000","last_fullsized":"2023-10-14T16:56:27.647239+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:15.697823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634561+0000","last_change":"2023-10-14T16:56:26.645678+0000","last_active":"2023-10-14T16:56:28.634561+0000","last_peered":"2023-10-14T16:56:28.634561+0000","last_clean":"2023-10-14T16:56:28.634561+0000","last_became_active":"2023-10-14T16:56:26.644996+0000","last_became_peered":"2023-10-14T16:56:26.644996+0000","last_unstale":"2023-10-14T16:56:28.634561+0000","last_undegraded":"2023-10-14T16:56:28.634561+0000","last_fullsized":"2023-10-14T16:56:28.634561+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:23:38.826436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.b","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635690+0000","last_change":"2023-10-14T16:56:25.637583+0000","last_active":"2023-10-14T16:56:28.635690+0000","last_peered":"2023-10-14T16:56:28.635690+0000","last_clean":"2023-10-14T16:56:28.635690+0000","last_became_active":"2023-10-14T16:56:25.636540+0000","last_became_peered":"2023-10-14T16:56:25.636540+0000","last_unstale":"2023-10-14T16:56:28.635690+0000","last_undegraded":"2023-10-14T16:56:28.635690+0000","last_fullsized":"2023-10-14T16:56:28.635690+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:10:45.075105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646691+0000","last_change":"2023-10-14T16:56:26.660199+0000","last_active":"2023-10-14T16:56:27.646691+0000","last_peered":"2023-10-14T16:56:27.646691+0000","last_clean":"2023-10-14T16:56:27.646691+0000","last_became_active":"2023-10-14T16:56:26.659785+0000","last_became_peered":"2023-10-14T16:56:26.659785+0000","last_unstale":"2023-10-14T16:56:27.646691+0000","last_undegraded":"2023-10-14T16:56:27.646691+0000","last_fullsized":"2023-10-14T16:56:27.646691+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:43.080935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632618+0000","last_change":"2023-10-14T16:56:25.640484+0000","last_active":"2023-10-14T16:56:28.632618+0000","last_peered":"2023-10-14T16:56:28.632618+0000","last_clean":"2023-10-14T16:56:28.632618+0000","last_became_active":"2023-10-14T16:56:25.640197+0000","last_became_peered":"2023-10-14T16:56:25.640197+0000","last_unstale":"2023-10-14T16:56:28.632618+0000","last_undegraded":"2023-10-14T16:56:28.632618+0000","last_fullsized":"2023-10-14T16:56:28.632618+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:16:00.739758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647423+0000","last_change":"2023-10-14T16:56:26.659203+0000","last_active":"2023-10-14T16:56:27.647423+0000","last_peered":"2023-10-14T16:56:27.647423+0000","last_clean":"2023-10-14T16:56:27.647423+0000","last_became_active":"2023-10-14T16:56:26.658918+0000","last_became_peered":"2023-10-14T16:56:26.658918+0000","last_unstale":"2023-10-14T16:56:27.647423+0000","last_undegraded":"2023-10-14T16:56:27.647423+0000","last_fullsized":"2023-10-14T16:56:27.647423+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:16:01.754924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632670+0000","last_change":"2023-10-14T16:56:25.647872+0000","last_active":"2023-10-14T16:56:28.632670+0000","last_peered":"2023-10-14T16:56:28.632670+0000","last_clean":"2023-10-14T16:56:28.632670+0000","last_became_active":"2023-10-14T16:56:25.647561+0000","last_became_peered":"2023-10-14T16:56:25.647561+0000","last_unstale":"2023-10-14T16:56:28.632670+0000","last_undegraded":"2023-10-14T16:56:28.632670+0000","last_fullsized":"2023-10-14T16:56:28.632670+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:08:17.942375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636469+0000","last_change":"2023-10-14T16:56:26.646367+0000","last_active":"2023-10-14T16:56:28.636469+0000","last_peered":"2023-10-14T16:56:28.636469+0000","last_clean":"2023-10-14T16:56:28.636469+0000","last_became_active":"2023-10-14T16:56:26.645998+0000","last_became_peered":"2023-10-14T16:56:26.645998+0000","last_unstale":"2023-10-14T16:56:28.636469+0000","last_undegraded":"2023-10-14T16:56:28.636469+0000","last_fullsized":"2023-10-14T16:56:28.636469+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:40.592003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.7","version":"81'2","reported_seq":14,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633785+0000","last_change":"2023-10-14T16:56:25.633905+0000","last_active":"2023-10-14T16:56:28.633785+0000","last_peered":"2023-10-14T16:56:28.633785+0000","last_clean":"2023-10-14T16:56:28.633785+0000","last_became_active":"2023-10-14T16:56:25.633610+0000","last_became_peered":"2023-10-14T16:56:25.633610+0000","last_unstale":"2023-10-14T16:56:28.633785+0000","last_undegraded":"2023-10-14T16:56:28.633785+0000","last_fullsized":"2023-10-14T16:56:28.633785+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:18.308926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.636641+0000","last_change":"2023-10-14T16:56:26.649544+0000","last_active":"2023-10-14T16:56:28.636641+0000","last_peered":"2023-10-14T16:56:28.636641+0000","last_clean":"2023-10-14T16:56:28.636641+0000","last_became_active":"2023-10-14T16:56:26.649109+0000","last_became_peered":"2023-10-14T16:56:26.649109+0000","last_unstale":"2023-10-14T16:56:28.636641+0000","last_undegraded":"2023-10-14T16:56:28.636641+0000","last_fullsized":"2023-10-14T16:56:28.636641+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:15.049514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.647314+0000","last_change":"2023-10-14T16:56:25.637970+0000","last_active":"2023-10-14T16:56:27.647314+0000","last_peered":"2023-10-14T16:56:27.647314+0000","last_clean":"2023-10-14T16:56:27.647314+0000","last_became_active":"2023-10-14T16:56:25.637682+0000","last_became_peered":"2023-10-14T16:56:25.637682+0000","last_unstale":"2023-10-14T16:56:27.647314+0000","last_undegraded":"2023-10-14T16:56:27.647314+0000","last_fullsized":"2023-10-14T16:56:27.647314+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:40.709459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.633034+0000","last_change":"2023-10-14T16:56:26.648848+0000","last_active":"2023-10-14T16:56:28.633034+0000","last_peered":"2023-10-14T16:56:28.633034+0000","last_clean":"2023-10-14T16:56:28.633034+0000","last_became_active":"2023-10-14T16:56:26.648228+0000","last_became_peered":"2023-10-14T16:56:26.648228+0000","last_unstale":"2023-10-14T16:56:28.633034+0000","last_undegraded":"2023-10-14T16:56:28.633034+0000","last_fullsized":"2023-10-14T16:56:28.633034+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:59.217099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651893+0000","last_change":"2023-10-14T16:56:25.632548+0000","last_active":"2023-10-14T16:56:27.651893+0000","last_peered":"2023-10-14T16:56:27.651893+0000","last_clean":"2023-10-14T16:56:27.651893+0000","last_became_active":"2023-10-14T16:56:25.632081+0000","last_became_peered":"2023-10-14T16:56:25.632081+0000","last_unstale":"2023-10-14T16:56:27.651893+0000","last_undegraded":"2023-10-14T16:56:27.651893+0000","last_fullsized":"2023-10-14T16:56:27.651893+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:55.130276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.646933+0000","last_change":"2023-10-14T16:56:26.666882+0000","last_active":"2023-10-14T16:56:27.646933+0000","last_peered":"2023-10-14T16:56:27.646933+0000","last_clean":"2023-10-14T16:56:27.646933+0000","last_became_active":"2023-10-14T16:56:26.661361+0000","last_became_peered":"2023-10-14T16:56:26.661361+0000","last_unstale":"2023-10-14T16:56:27.646933+0000","last_undegraded":"2023-10-14T16:56:27.646933+0000","last_fullsized":"2023-10-14T16:56:27.646933+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:50:02.534701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635934+0000","last_change":"2023-10-14T16:56:25.646176+0000","last_active":"2023-10-14T16:56:28.635934+0000","last_peered":"2023-10-14T16:56:28.635934+0000","last_clean":"2023-10-14T16:56:28.635934+0000","last_became_active":"2023-10-14T16:56:25.645905+0000","last_became_peered":"2023-10-14T16:56:25.645905+0000","last_unstale":"2023-10-14T16:56:28.635934+0000","last_undegraded":"2023-10-14T16:56:28.635934+0000","last_fullsized":"2023-10-14T16:56:28.635934+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:40:28.092816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635212+0000","last_change":"2023-10-14T16:56:26.653173+0000","last_active":"2023-10-14T16:56:28.635212+0000","last_peered":"2023-10-14T16:56:28.635212+0000","last_clean":"2023-10-14T16:56:28.635212+0000","last_became_active":"2023-10-14T16:56:26.650944+0000","last_became_peered":"2023-10-14T16:56:26.650944+0000","last_unstale":"2023-10-14T16:56:28.635212+0000","last_undegraded":"2023-10-14T16:56:28.635212+0000","last_fullsized":"2023-10-14T16:56:28.635212+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:42.474725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632416+0000","last_change":"2023-10-14T16:56:25.644381+0000","last_active":"2023-10-14T16:56:28.632416+0000","last_peered":"2023-10-14T16:56:28.632416+0000","last_clean":"2023-10-14T16:56:28.632416+0000","last_became_active":"2023-10-14T16:56:25.644102+0000","last_became_peered":"2023-10-14T16:56:25.644102+0000","last_unstale":"2023-10-14T16:56:28.632416+0000","last_undegraded":"2023-10-14T16:56:28.632416+0000","last_fullsized":"2023-10-14T16:56:28.632416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:54:12.210244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634068+0000","last_change":"2023-10-14T16:56:26.668231+0000","last_active":"2023-10-14T16:56:28.634068+0000","last_peered":"2023-10-14T16:56:28.634068+0000","last_clean":"2023-10-14T16:56:28.634068+0000","last_became_active":"2023-10-14T16:56:26.667848+0000","last_became_peered":"2023-10-14T16:56:26.667848+0000","last_unstale":"2023-10-14T16:56:28.634068+0000","last_undegraded":"2023-10-14T16:56:28.634068+0000","last_fullsized":"2023-10-14T16:56:28.634068+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:27.204944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.650470+0000","last_change":"2023-10-14T16:56:25.636516+0000","last_active":"2023-10-14T16:56:27.650470+0000","last_peered":"2023-10-14T16:56:27.650470+0000","last_clean":"2023-10-14T16:56:27.650470+0000","last_became_active":"2023-10-14T16:56:25.636001+0000","last_became_peered":"2023-10-14T16:56:25.636001+0000","last_unstale":"2023-10-14T16:56:27.650470+0000","last_undegraded":"2023-10-14T16:56:27.650470+0000","last_fullsized":"2023-10-14T16:56:27.650470+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:01.550372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632196+0000","last_change":"2023-10-14T16:56:26.644627+0000","last_active":"2023-10-14T16:56:28.632196+0000","last_peered":"2023-10-14T16:56:28.632196+0000","last_clean":"2023-10-14T16:56:28.632196+0000","last_became_active":"2023-10-14T16:56:26.641465+0000","last_became_peered":"2023-10-14T16:56:26.641465+0000","last_unstale":"2023-10-14T16:56:28.632196+0000","last_undegraded":"2023-10-14T16:56:28.632196+0000","last_fullsized":"2023-10-14T16:56:28.632196+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:11:02.854968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.632308+0000","last_change":"2023-10-14T16:56:25.635327+0000","last_active":"2023-10-14T16:56:28.632308+0000","last_peered":"2023-10-14T16:56:28.632308+0000","last_clean":"2023-10-14T16:56:28.632308+0000","last_became_active":"2023-10-14T16:56:25.635060+0000","last_became_peered":"2023-10-14T16:56:25.635060+0000","last_unstale":"2023-10-14T16:56:28.632308+0000","last_undegraded":"2023-10-14T16:56:28.632308+0000","last_fullsized":"2023-10-14T16:56:28.632308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:39.687227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.634219+0000","last_change":"2023-10-14T16:56:26.642789+0000","last_active":"2023-10-14T16:56:28.634219+0000","last_peered":"2023-10-14T16:56:28.634219+0000","last_clean":"2023-10-14T16:56:28.634219+0000","last_became_active":"2023-10-14T16:56:26.641371+0000","last_became_peered":"2023-10-14T16:56:26.641371+0000","last_unstale":"2023-10-14T16:56:28.634219+0000","last_undegraded":"2023-10-14T16:56:28.634219+0000","last_fullsized":"2023-10-14T16:56:28.634219+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:00:50.426954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635629+0000","last_change":"2023-10-14T16:56:25.644744+0000","last_active":"2023-10-14T16:56:28.635629+0000","last_peered":"2023-10-14T16:56:28.635629+0000","last_clean":"2023-10-14T16:56:28.635629+0000","last_became_active":"2023-10-14T16:56:25.644477+0000","last_became_peered":"2023-10-14T16:56:25.644477+0000","last_unstale":"2023-10-14T16:56:28.635629+0000","last_undegraded":"2023-10-14T16:56:28.635629+0000","last_fullsized":"2023-10-14T16:56:28.635629+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:23.425220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":11,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:28.635224+0000","last_change":"2023-10-14T16:56:26.664224+0000","last_active":"2023-10-14T16:56:28.635224+0000","last_peered":"2023-10-14T16:56:28.635224+0000","last_clean":"2023-10-14T16:56:28.635224+0000","last_became_active":"2023-10-14T16:56:26.659567+0000","last_became_peered":"2023-10-14T16:56:26.659567+0000","last_unstale":"2023-10-14T16:56:28.635224+0000","last_undegraded":"2023-10-14T16:56:28.635224+0000","last_fullsized":"2023-10-14T16:56:28.635224+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:25.598977+0000","last_clean_scrub_stamp":"2023-10-14T16:56:25.598977+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:05:49.204116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":82,"state":"active+clean","last_fresh":"2023-10-14T16:56:27.651778+0000","last_change":"2023-10-14T16:56:25.646342+0000","last_active":"2023-10-14T16:56:27.651778+0000","last_peered":"2023-10-14T16:56:27.651778+0000","last_clean":"2023-10-14T16:56:27.651778+0000","last_became_active":"2023-10-14T16:56:25.642275+0000","last_became_peered":"2023-10-14T16:56:25.642275+0000","last_unstale":"2023-10-14T16:56:27.651778+0000","last_undegraded":"2023-10-14T16:56:27.651778+0000","last_fullsized":"2023-10-14T16:56:27.651778+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:56:24.583810+0000","last_clean_scrub_stamp":"2023-10-14T16:56:24.583810+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:45.616981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"69'161","reported_seq":383,"reported_epoch":81,"state":"active+clean","last_fresh":"2023-10-14T16:56:26.642695+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:56:26.642695+0000","last_peered":"2023-10-14T16:56:26.642695+0000","last_clean":"2023-10-14T16:56:26.642695+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:56:26.642695+0000","last_undegraded":"2023-10-14T16:56:26.642695+0000","last_fullsized":"2023-10-14T16:56:26.642695+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":8,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":500,"num_read_kb":924,"num_write":239,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":161,"ondisk_log_size":161,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575044,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36884,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334956,"statfs":{"total":96636764160,"available":96598994944,"internally_reserved":0,"allocated":1581056,"data_stored":1057795,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2,"5min":1.1739999999999999,"15min":1.163},"min":{"1min":0.80000000000000004,"5min":0.56599999999999995,"15min":0.503},"max":{"1min":1.732,"5min":1.732,"15min":1.897},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.1339999999999999,"5min":1.143,"15min":1.2010000000000001},"min":{"1min":0.82299999999999995,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7230000000000001,"5min":1.8400000000000001,"15min":2.262},"last":0.76400000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3120000000000001,"5min":1.2170000000000001,"15min":1.2330000000000001},"min":{"1min":0.93400000000000005,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.6060000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.26,"5min":1.2450000000000001,"15min":1.2509999999999999},"min":{"1min":0.93600000000000005,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.6020000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.532}]},{"osd":2,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.224,"15min":1.232},"min":{"1min":0.91700000000000004,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.633,"5min":1.651,"15min":2.2389999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.29,"15min":1.3129999999999999},"min":{"1min":0.80400000000000005,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.6870000000000001,"5min":1.7869999999999999,"15min":2.3500000000000001},"last":1.054}]},{"osd":3,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.3799999999999999,"15min":1.3680000000000001},"min":{"1min":0.95099999999999996,"5min":0.92400000000000004,"15min":0.77300000000000002},"max":{"1min":1.6539999999999999,"5min":1.875,"15min":2.4159999999999999},"last":1.6040000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.4650000000000001,"15min":1.456},"min":{"1min":1.2949999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.861,"5min":1.861,"15min":2.5819999999999999},"last":1.464}]},{"osd":4,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.26,"5min":1.252,"15min":1.2669999999999999},"min":{"1min":0.83099999999999996,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.714,"5min":2.105,"15min":2.2959999999999998},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.335,"5min":1.333,"15min":1.321},"min":{"1min":0.90500000000000003,"5min":0.59799999999999998,"15min":0.52400000000000002},"max":{"1min":1.835,"5min":1.899,"15min":2.1400000000000001},"last":1.3200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.405,"5min":1.3700000000000001,"15min":1.393},"min":{"1min":0.92700000000000005,"5min":0.66000000000000003,"15min":0.63100000000000001},"max":{"1min":1.7889999999999999,"5min":1.885,"15min":3.5270000000000001},"last":1.2809999999999999},{"interface":"front","average":{"1min":1.448,"5min":1.4630000000000001,"15min":1.456},"min":{"1min":1.117,"5min":0.61799999999999999,"15min":0.54200000000000004},"max":{"1min":1.8029999999999999,"5min":1.9359999999999999,"15min":2.1299999999999999},"last":1.6559999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.514,"15min":1.5369999999999999},"min":{"1min":1.2210000000000001,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.869,"5min":1.9890000000000001,"15min":2.3220000000000001},"last":1.732},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.452,"15min":1.4750000000000001},"min":{"1min":1.1040000000000001,"5min":0.64800000000000002,"15min":0.56200000000000006},"max":{"1min":1.7450000000000001,"5min":2.036,"15min":2.7040000000000002},"last":1.5069999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575043,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32156,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339684,"statfs":{"total":96636764160,"available":96603836416,"internally_reserved":0,"allocated":991232,"data_stored":473611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.165,"15min":1.1639999999999999},"min":{"1min":0.82699999999999996,"5min":0.65700000000000003,"15min":0.63700000000000001},"max":{"1min":1.6140000000000001,"5min":2.137,"15min":2.339},"last":0.93400000000000005},{"interface":"front","average":{"1min":1.23,"5min":1.242,"15min":1.2130000000000001},"min":{"1min":0.82699999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.6419999999999999,"5min":2.0760000000000001,"15min":3.2509999999999999},"last":1.2569999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2250000000000001,"5min":1.194,"15min":1.2010000000000001},"min":{"1min":0.78400000000000003,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.631,"5min":2.802,"15min":4.3280000000000003},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3200000000000001,"15min":1.302},"min":{"1min":0.92800000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.7030000000000001,"5min":2.512,"15min":3.431},"last":1.3169999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3740000000000001,"5min":1.3899999999999999,"15min":1.3620000000000001},"min":{"1min":0.91300000000000003,"5min":0.81100000000000005,"15min":0.75900000000000001},"max":{"1min":1.9970000000000001,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.4219999999999999,"15min":1.4039999999999999},"min":{"1min":1.028,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.6990000000000001,"5min":3.286,"15min":3.286},"last":1.1870000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.395,"15min":1.3819999999999999},"min":{"1min":0.93400000000000005,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":1.669,"5min":2.823,"15min":2.8319999999999999},"last":1.577},{"interface":"front","average":{"1min":1.379,"5min":1.4730000000000001,"15min":1.4239999999999999},"min":{"1min":1.0880000000000001,"5min":0.879,"15min":0.75600000000000001},"max":{"1min":1.774,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.659}]},{"osd":4,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.294,"5min":1.3540000000000001,"15min":1.351},"min":{"1min":0.97399999999999998,"5min":0.81999999999999995,"15min":0.77200000000000002},"max":{"1min":1.7729999999999999,"5min":2.9460000000000002,"15min":3.0790000000000002},"last":1.145},{"interface":"front","average":{"1min":1.427,"5min":1.482,"15min":1.423},"min":{"1min":0.98099999999999998,"5min":0.88700000000000001,"15min":0.82999999999999996},"max":{"1min":1.8009999999999999,"5min":2.5750000000000002,"15min":2.738},"last":1.738}]},{"osd":5,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.421,"15min":1.3859999999999999},"min":{"1min":0.93000000000000005,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":1.665,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.5349999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5900000000000001,"15min":1.5589999999999999},"min":{"1min":0.98999999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"max":{"1min":1.843,"5min":3.0489999999999999,"15min":3.206},"last":1.4259999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:55:49 2023","interfaces":[{"interface":"back","average":{"1min":1.6000000000000001,"5min":1.6539999999999999,"15min":1.5880000000000001},"min":{"1min":1.204,"5min":1.117,"15min":0.90900000000000003},"max":{"1min":1.899,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.823},{"interface":"front","average":{"1min":1.512,"5min":1.5469999999999999,"15min":1.546},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":1.8680000000000001,"5min":2.7000000000000002,"15min":3.1320000000000001},"last":1.9670000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575044,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32152,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339688,"statfs":{"total":96636764160,"available":96603840512,"internally_reserved":0,"allocated":987136,"data_stored":467919,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.1080000000000001,"5min":1.1040000000000001,"15min":1.119},"min":{"1min":0.72699999999999998,"5min":0.68799999999999994,"15min":0.52200000000000002},"max":{"1min":1.5620000000000001,"5min":1.861,"15min":3.2000000000000002},"last":1.5409999999999999},{"interface":"front","average":{"1min":1.121,"5min":1.1080000000000001,"15min":1.137},"min":{"1min":0.85699999999999998,"5min":0.68999999999999995,"15min":0.57099999999999995},"max":{"1min":1.4750000000000001,"5min":1.665,"15min":4.101},"last":1.1279999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.157,"15min":1.1839999999999999},"min":{"1min":0.91500000000000004,"5min":0.71599999999999997,"15min":0.63400000000000001},"max":{"1min":1.6910000000000001,"5min":1.7090000000000001,"15min":3.1659999999999999},"last":1.3700000000000001},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.206,"15min":1.232},"min":{"1min":0.86899999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.524,"5min":1.7609999999999999,"15min":2.2549999999999999},"last":1.1970000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.196,"15min":1.212},"min":{"1min":0.81699999999999995,"5min":0.64000000000000001,"15min":0.64000000000000001},"max":{"1min":1.4399999999999999,"5min":1.7150000000000001,"15min":4.3639999999999999},"last":1.333},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.3320000000000001,"15min":1.3280000000000001},"min":{"1min":0.98499999999999999,"5min":0.84699999999999998,"15min":0.79300000000000004},"max":{"1min":1.7450000000000001,"5min":2.1030000000000002,"15min":4.3339999999999996},"last":1.4339999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.347,"15min":1.391},"min":{"1min":1.01,"5min":0.745,"15min":0.745},"max":{"1min":1.609,"5min":1.9279999999999999,"15min":2.431},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.391,"5min":1.3999999999999999,"15min":1.417},"min":{"1min":0.94199999999999995,"5min":0.83799999999999997,"15min":0.83799999999999997},"max":{"1min":1.6870000000000001,"5min":1.8999999999999999,"15min":3.6890000000000001},"last":0.85199999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.2410000000000001,"5min":1.2689999999999999,"15min":1.292},"min":{"1min":0.87,"5min":0.79500000000000004,"15min":0.72199999999999998},"max":{"1min":1.5669999999999999,"5min":2.194,"15min":2.577},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.3420000000000001,"5min":1.323,"15min":1.3580000000000001},"min":{"1min":0.85999999999999999,"5min":0.69699999999999995,"15min":0.66200000000000003},"max":{"1min":1.673,"5min":1.927,"15min":2.6030000000000002},"last":1.2529999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.401,"15min":1.393},"min":{"1min":0.97999999999999998,"5min":0.68899999999999995,"15min":0.68799999999999994},"max":{"1min":1.71,"5min":2.2869999999999999,"15min":3.5739999999999998},"last":1.0860000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.532,"15min":1.55},"min":{"1min":1.0920000000000001,"5min":1.0760000000000001,"15min":0.81499999999999995},"max":{"1min":1.788,"5min":2.2519999999999998,"15min":3.2949999999999999},"last":1.49}]},{"osd":7,"last update":"Sat Oct 14 16:55:52 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.506,"15min":1.5229999999999999},"min":{"1min":1.0109999999999999,"5min":0.86399999999999999,"15min":0.72299999999999998},"max":{"1min":1.726,"5min":2.032,"15min":3.0289999999999999},"last":1.403},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.546,"15min":1.526},"min":{"1min":1.147,"5min":1.0349999999999999,"15min":0.627},"max":{"1min":1.7130000000000001,"5min":2.3980000000000001,"15min":3.7869999999999999},"last":1.2130000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575044,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58532,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313308,"statfs":{"total":96636764160,"available":96576827392,"internally_reserved":0,"allocated":1003520,"data_stored":475223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.1379999999999999,"15min":1.1619999999999999},"min":{"1min":0.72399999999999998,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.621,"5min":1.7050000000000001,"15min":4.3810000000000002},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.3400000000000001,"5min":1.2609999999999999,"15min":1.24},"min":{"1min":0.86899999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.7170000000000001,"5min":1.972,"15min":4.4059999999999997},"last":1.554}]},{"osd":1,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3300000000000001,"5min":1.264,"15min":1.256},"min":{"1min":0.89400000000000002,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":2.0169999999999999,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.2749999999999999,"5min":1.2490000000000001,"15min":1.2430000000000001},"min":{"1min":0.90300000000000002,"5min":0.70499999999999996,"15min":0.70499999999999996},"max":{"1min":1.704,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":0.90500000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2410000000000001,"15min":1.23},"min":{"1min":0.84799999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.599,"5min":2.169,"15min":2.169},"last":1.752},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.395,"15min":1.3759999999999999},"min":{"1min":0.96999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.823,"5min":1.966,"15min":2.7759999999999998},"last":1.3100000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.327,"5min":1.302,"15min":1.3120000000000001},"min":{"1min":0.93799999999999994,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.206},{"interface":"front","average":{"1min":1.379,"5min":1.3680000000000001,"15min":1.3979999999999999},"min":{"1min":1.0469999999999999,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":1.764,"5min":1.952,"15min":2.085},"last":1.611}]},{"osd":5,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.371,"15min":1.361},"min":{"1min":0.99099999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.7470000000000001,"5min":1.855,"15min":2.2320000000000002},"last":1.05},{"interface":"front","average":{"1min":1.306,"5min":1.3140000000000001,"15min":1.3500000000000001},"min":{"1min":1.0589999999999999,"5min":0.57499999999999996,"15min":0.57499999999999996},"max":{"1min":1.6839999999999999,"5min":1.7909999999999999,"15min":2.153},"last":1.6930000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4490000000000001,"5min":1.403,"15min":1.4019999999999999},"min":{"1min":1.004,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.8560000000000001,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5229999999999999,"15min":1.5369999999999999},"min":{"1min":1.181,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.915,"5min":2.214,"15min":2.6539999999999999},"last":1.6599999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.4530000000000001,"15min":1.482},"min":{"1min":1.038,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.794,"5min":2.1269999999999998,"15min":2.2389999999999999},"last":1.49},{"interface":"front","average":{"1min":1.554,"5min":1.5409999999999999,"15min":1.5580000000000001},"min":{"1min":1.157,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.0209999999999999,"5min":2.3130000000000002,"15min":2.3999999999999999},"last":1.893}]}]},{"osd":3,"up_from":13,"seq":55834575044,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40468,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331372,"statfs":{"total":96636764160,"available":96595324928,"internally_reserved":0,"allocated":987136,"data_stored":467323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":0.91400000000000003,"5min":0.91900000000000004,"15min":0.92600000000000005},"min":{"1min":0.53000000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.2709999999999999,"5min":1.4239999999999999,"15min":1.526},"last":1.141},{"interface":"front","average":{"1min":1.1279999999999999,"5min":1.1180000000000001,"15min":1.1240000000000001},"min":{"1min":0.67900000000000005,"5min":0.442,"15min":0.442},"max":{"1min":1.4850000000000001,"5min":1.706,"15min":1.8799999999999999},"last":1.091}]},{"osd":1,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":0.878,"5min":0.90200000000000002,"15min":0.91800000000000004},"min":{"1min":0.53800000000000003,"5min":0.48199999999999998,"15min":0.33600000000000002},"max":{"1min":1.129,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.81299999999999994},{"interface":"front","average":{"1min":1.141,"5min":1.069,"15min":1.1060000000000001},"min":{"1min":0.755,"5min":0.438,"15min":0.373},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.294},"last":1.0509999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3580000000000001,"15min":1.357},"min":{"1min":0.66900000000000004,"5min":0.57899999999999996,"15min":0.51600000000000001},"max":{"1min":1.7310000000000001,"5min":1.966,"15min":2.6819999999999999},"last":1.6080000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.2490000000000001,"15min":1.2689999999999999},"min":{"1min":0.74099999999999999,"5min":0.496,"15min":0.496},"max":{"1min":1.8560000000000001,"5min":2.0960000000000001,"15min":2.77},"last":1.373}]},{"osd":4,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.3720000000000001,"15min":1.3700000000000001},"min":{"1min":1.0960000000000001,"5min":0.81699999999999995,"15min":0.77300000000000002},"max":{"1min":1.7989999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.3959999999999999,"15min":1.4179999999999999},"min":{"1min":0.96799999999999997,"5min":0.76900000000000002,"15min":0.749},"max":{"1min":1.756,"5min":1.8080000000000001,"15min":3.9540000000000002},"last":1.3340000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.3740000000000001,"15min":1.4059999999999999},"min":{"1min":0.878,"5min":0.78400000000000003,"15min":0.77500000000000002},"max":{"1min":1.722,"5min":2.516,"15min":3.0190000000000001},"last":1.3109999999999999},{"interface":"front","average":{"1min":1.345,"5min":1.4339999999999999,"15min":1.45},"min":{"1min":1.0640000000000001,"5min":0.80600000000000005,"15min":0.77400000000000002},"max":{"1min":1.629,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5169999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.425,"5min":1.4359999999999999,"15min":1.47},"min":{"1min":1.0620000000000001,"5min":0.78500000000000003,"15min":0.65100000000000002},"max":{"1min":1.9119999999999999,"5min":2.2250000000000001,"15min":4.0570000000000004},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5109999999999999,"15min":1.5469999999999999},"min":{"1min":1.101,"5min":0.91400000000000003,"15min":0.79200000000000004},"max":{"1min":2.0070000000000001,"5min":2.1680000000000001,"15min":4.1900000000000004},"last":1.202}]},{"osd":7,"last update":"Sat Oct 14 16:55:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.4890000000000001,"15min":1.51},"min":{"1min":0.92600000000000005,"5min":0.85199999999999998,"15min":0.745},"max":{"1min":1.698,"5min":2.871,"15min":2.9900000000000002},"last":1.226},{"interface":"front","average":{"1min":1.5189999999999999,"5min":1.55,"15min":1.556},"min":{"1min":1.004,"5min":0.85099999999999998,"15min":0.82899999999999996},"max":{"1min":1.962,"5min":2.907,"15min":2.9569999999999999},"last":1.4039999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575044,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40464,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331376,"statfs":{"total":96636764160,"available":96595329024,"internally_reserved":0,"allocated":987136,"data_stored":467323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.88800000000000001,"15min":0.90700000000000003},"min":{"1min":0.55800000000000005,"5min":0.38900000000000001,"15min":0.34300000000000003},"max":{"1min":1.698,"5min":1.698,"15min":1.849},"last":0.52400000000000002},{"interface":"front","average":{"1min":1.0840000000000001,"5min":1.0529999999999999,"15min":1.0489999999999999},"min":{"1min":0.67700000000000005,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.528,"5min":1.8080000000000001,"15min":3.2149999999999999},"last":0.55600000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.95299999999999996,"15min":0.98099999999999998},"min":{"1min":0.64300000000000002,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":1.337,"5min":1.4630000000000001,"15min":2.0419999999999998},"last":0.626},{"interface":"front","average":{"1min":1.288,"5min":1.2150000000000001,"15min":1.1830000000000001},"min":{"1min":0.876,"5min":0.54700000000000004,"15min":0.42099999999999999},"max":{"1min":1.736,"5min":2.069,"15min":2.2890000000000001},"last":0.65000000000000002}]},{"osd":3,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.1919999999999999,"15min":1.2310000000000001},"min":{"1min":0.72199999999999998,"5min":0.46500000000000002,"15min":0.46100000000000002},"max":{"1min":1.6599999999999999,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.359,"5min":1.2969999999999999,"15min":1.3220000000000001},"min":{"1min":1.02,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.7729999999999999,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":0.72299999999999998}]},{"osd":4,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.4279999999999999,"15min":1.431},"min":{"1min":1.052,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.913,"5min":1.95,"15min":2.6259999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":1.341,"5min":1.3520000000000001,"15min":1.375},"min":{"1min":0.90500000000000003,"5min":0.71699999999999997,"15min":0.67900000000000005},"max":{"1min":1.6719999999999999,"5min":2.0750000000000002,"15min":3.0459999999999998},"last":1.179}]},{"osd":5,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.45,"5min":1.4079999999999999,"15min":1.4059999999999999},"min":{"1min":0.91600000000000004,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.734,"5min":2.0019999999999998,"15min":2.2389999999999999},"last":1.256},{"interface":"front","average":{"1min":1.575,"5min":1.526,"15min":1.5389999999999999},"min":{"1min":1.0640000000000001,"5min":0.85599999999999998,"15min":0.80000000000000004},"max":{"1min":1.857,"5min":2.2730000000000001,"15min":2.883},"last":1.29}]},{"osd":6,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.4330000000000001,"15min":1.458},"min":{"1min":0.99299999999999999,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.8759999999999999,"5min":2.0289999999999999,"15min":3.1200000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.5609999999999999,"5min":1.5149999999999999,"15min":1.5349999999999999},"min":{"1min":1.145,"5min":0.83599999999999997,"15min":0.70699999999999996},"max":{"1min":1.8560000000000001,"5min":2.105,"15min":2.3420000000000001},"last":1.369}]},{"osd":7,"last update":"Sat Oct 14 16:55:48 2023","interfaces":[{"interface":"back","average":{"1min":1.468,"5min":1.5029999999999999,"15min":1.544},"min":{"1min":1.052,"5min":0.86099999999999999,"15min":0.77400000000000002},"max":{"1min":1.7729999999999999,"5min":2.109,"15min":3.169},"last":1.341},{"interface":"front","average":{"1min":1.641,"5min":1.5720000000000001,"15min":1.6040000000000001},"min":{"1min":1.218,"5min":0.81200000000000006,"15min":0.755},"max":{"1min":1.9650000000000001,"5min":2.165,"15min":2.996},"last":1.504}]}]},{"osd":1,"up_from":13,"seq":55834575044,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58524,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313316,"statfs":{"total":96636764160,"available":96576835584,"internally_reserved":0,"allocated":995328,"data_stored":473701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.89000000000000001,"15min":0.90500000000000003},"min":{"1min":0.61099999999999999,"5min":0.41199999999999998,"15min":0.35999999999999999},"max":{"1min":1.518,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.438},{"interface":"front","average":{"1min":1.0720000000000001,"5min":1.0429999999999999,"15min":1.071},"min":{"1min":0.46600000000000003,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.6819999999999999,"5min":2.0819999999999999,"15min":2.0819999999999999},"last":0.56999999999999995}]},{"osd":2,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.012,"5min":0.94399999999999995,"15min":0.95899999999999996},"min":{"1min":0.68500000000000005,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.655,"5min":1.758,"15min":1.758},"last":0.62},{"interface":"front","average":{"1min":1.054,"5min":1.1279999999999999,"15min":1.171},"min":{"1min":0.56499999999999995,"5min":0.51300000000000001,"15min":0.45200000000000001},"max":{"1min":1.5229999999999999,"5min":2.0150000000000001,"15min":2.1499999999999999},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.266,"5min":1.276,"15min":1.2969999999999999},"min":{"1min":0.81299999999999994,"5min":0.49399999999999999,"15min":0.40999999999999998},"max":{"1min":1.6990000000000001,"5min":2.198,"15min":2.5339999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":1.1180000000000001,"5min":1.167,"15min":1.2090000000000001},"min":{"1min":0.61899999999999999,"5min":0.56799999999999995,"15min":0.434},"max":{"1min":1.663,"5min":1.8680000000000001,"15min":4.6799999999999997},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.409,"15min":1.429},"min":{"1min":1.0189999999999999,"5min":0.86699999999999999,"15min":0.83799999999999997},"max":{"1min":1.7490000000000001,"5min":2.0139999999999998,"15min":4.5789999999999997},"last":1.234},{"interface":"front","average":{"1min":1.272,"5min":1.2849999999999999,"15min":1.3109999999999999},"min":{"1min":0.88400000000000001,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.0840000000000001},"last":0.88100000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.343,"15min":1.371},"min":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.7769999999999999,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":0.80900000000000005},{"interface":"front","average":{"1min":1.393,"5min":1.4330000000000001,"15min":1.476},"min":{"1min":0.873,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.78,"5min":2.302,"15min":2.4980000000000002},"last":1.034}]},{"osd":6,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.423,"5min":1.468,"15min":1.514},"min":{"1min":0.873,"5min":0.79500000000000004,"15min":0.79500000000000004},"max":{"1min":1.8680000000000001,"5min":2.3479999999999999,"15min":4.6360000000000001},"last":0.85899999999999999},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.423,"15min":1.476},"min":{"1min":0.94399999999999995,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.863,"5min":2.1070000000000002,"15min":2.2970000000000002},"last":1.1000000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:05 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.5389999999999999,"15min":1.577},"min":{"1min":0.91700000000000004,"5min":0.87,"15min":0.79900000000000004},"max":{"1min":2.1930000000000001,"5min":2.3940000000000001,"15min":2.452},"last":1.1859999999999999},{"interface":"front","average":{"1min":1.516,"5min":1.536,"15min":1.5900000000000001},"min":{"1min":0.95599999999999996,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.819,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":0.90400000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575043,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36904,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334936,"statfs":{"total":96636764160,"available":96598974464,"internally_reserved":0,"allocated":1601536,"data_stored":1066173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":0.96599999999999997,"5min":0.89700000000000002,"15min":0.88600000000000001},"min":{"1min":0.64700000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.6719999999999999,"5min":1.774,"15min":2.0979999999999999},"last":0.83299999999999996},{"interface":"front","average":{"1min":1.0329999999999999,"5min":1.101,"15min":1.091},"min":{"1min":0.75700000000000001,"5min":0.52000000000000002,"15min":0.46999999999999997},"max":{"1min":1.6859999999999999,"5min":1.95,"15min":1.95},"last":1.1259999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.0600000000000001,"15min":1.083},"min":{"1min":0.91500000000000004,"5min":0.443,"15min":0.443},"max":{"1min":1.5049999999999999,"5min":1.649,"15min":1.962},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.157,"5min":1.1970000000000001,"15min":1.2},"min":{"1min":0.81599999999999995,"5min":0.80000000000000004,"15min":0.42699999999999999},"max":{"1min":1.599,"5min":2.0190000000000001,"15min":2.0190000000000001},"last":1.248}]},{"osd":3,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.278,"15min":1.272},"min":{"1min":1.0529999999999999,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.849,"5min":2.2010000000000001,"15min":2.2010000000000001},"last":1.452},{"interface":"front","average":{"1min":1.319,"5min":1.3540000000000001,"15min":1.3520000000000001},"min":{"1min":1.0349999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":1.764,"5min":2.593,"15min":2.593},"last":1.365}]},{"osd":4,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.2529999999999999,"5min":1.276,"15min":1.272},"min":{"1min":0.96099999999999997,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":1.641,"5min":1.7769999999999999,"15min":1.8440000000000001},"last":1.415},{"interface":"front","average":{"1min":1.46,"5min":1.4119999999999999,"15min":1.427},"min":{"1min":1.171,"5min":0.88600000000000001,"15min":0.77800000000000002},"max":{"1min":1.667,"5min":2.0019999999999998,"15min":2.3460000000000001},"last":1.7050000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3400000000000001,"5min":1.3069999999999999,"15min":1.3220000000000001},"min":{"1min":1.0960000000000001,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":1.8220000000000001,"5min":1.9650000000000001,"15min":2.0539999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.514,"5min":1.516,"15min":1.52},"min":{"1min":1.236,"5min":0.86099999999999999,"15min":0.86099999999999999},"max":{"1min":2.024,"5min":2.5190000000000001,"15min":2.5190000000000001},"last":1.504}]},{"osd":6,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.5840000000000001,"15min":1.5880000000000001},"min":{"1min":1.2010000000000001,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":1.962,"5min":2.5670000000000002,"15min":2.5670000000000002},"last":1.575},{"interface":"front","average":{"1min":1.419,"5min":1.409,"15min":1.399},"min":{"1min":1.044,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":1.722,"5min":2.0049999999999999,"15min":2.1850000000000001},"last":1.77}]},{"osd":7,"last update":"Sat Oct 14 16:55:44 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5609999999999999,"15min":1.55},"min":{"1min":1.3049999999999999,"5min":0.83899999999999997,"15min":0.83399999999999996},"max":{"1min":2.266,"5min":2.3410000000000002,"15min":2.5219999999999998},"last":1.8089999999999999},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.528,"15min":1.5249999999999999},"min":{"1min":1.117,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":1.7969999999999999,"5min":2.073,"15min":2.415},"last":2.0579999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:56:31.666 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:56:31.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:56:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.940+0000 7f03cd421640 1 Processor -- start 2023-10-14T16:56:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.940+0000 7f03cd421640 1 -- start start 2023-10-14T16:56:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.942+0000 7f03cd421640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03c80cc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.942+0000 7f03cd421640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c8161760 con 0x7f03c806f940 2023-10-14T16:56:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.942+0000 7f03cd421640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03c80cd010 con 0x7f03c80ccb90 2023-10-14T16:56:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.942+0000 7f03cd421640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80cd2f0 con 0x7f03c80ca1a0 2023-10-14T16:56:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.943+0000 7f03ccc20640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f03c806f940 0x7f03c8069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43848/0 (socket says 172.21.15.29:43848) 2023-10-14T16:56:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.943+0000 7f03ccc20640 1 -- 172.21.15.29:0/1147919260 learned_addr learned my addr 172.21.15.29:0/1147919260 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.943+0000 7f03c7fff640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03c80cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.943+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152598990 0 0) 0x7f03c8161760 con 0x7f03c806f940 2023-10-14T16:56:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.943+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03b0003520 con 0x7f03c806f940 2023-10-14T16:56:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1427633049 0 0) 0x7f03b0003520 con 0x7f03c806f940 2023-10-14T16:56:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 msgr2=0x7f03c80cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03c80cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 >> v1:172.21.15.186:6789/0 conn(0x7f03c80ccb90 legacy=0x7f03c806f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c80cd5d0 con 0x7f03c806f940 2023-10-14T16:56:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f03c00019f0 con 0x7f03c806f940 2023-10-14T16:56:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 >> v1:172.21.15.29:6789/0 conn(0x7f03c806f940 legacy=0x7f03c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03b00039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03b0003f70 0x7f03b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80cd2f0 con 0x7f03b0004d00 2023-10-14T16:56:31.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b000bb50 con 0x7f03b0003f70 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b000bd10 con 0x7f03c80ca1a0 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.944+0000 7f03ccc20640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03c77fe640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03b0003f70 0x7f03b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03c7fff640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03b00039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03ccc20640 1 -- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 msgr2=0x7f03b00039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:31.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03ccc20640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80ca1a0 0x7f03b00039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03ccc20640 1 -- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03b0003f70 msgr2=0x7f03b00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03ccc20640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03b0003f70 0x7f03b00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.945+0000 7f03ccc20640 1 -- 172.21.15.29:0/1147919260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b0003520 con 0x7f03b0004d00 2023-10-14T16:56:31.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.946+0000 7f03ccc20640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03b0005180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f03c80b4810 tx=0x7f03b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b5ae860ad318f39 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.946+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0034630 con 0x7f03b0004d00 2023-10-14T16:56:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.946+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c002f070 con 0x7f03b0004d00 2023-10-14T16:56:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.946+0000 7f03c6ffd640 1 -- 172.21.15.29:0/1147919260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c00349d0 con 0x7f03b0004d00 2023-10-14T16:56:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 -- 172.21.15.29:0/1147919260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 msgr2=0x7f03b0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 --2- 172.21.15.29:0/1147919260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03b0005180 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7f03c80b4810 tx=0x7f03b00057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03ccc20640 1 -- 172.21.15.29:0/1147919260 reap_dead start 2023-10-14T16:56:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 -- 172.21.15.29:0/1147919260 shutdown_connections 2023-10-14T16:56:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 -- 172.21.15.29:0/1147919260 >> 172.21.15.29:0/1147919260 conn(0x7f03c80ce6a0 msgr2=0x7f03c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 -- 172.21.15.29:0/1147919260 shutdown_connections 2023-10-14T16:56:31.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.947+0000 7f03cd421640 1 -- 172.21.15.29:0/1147919260 wait complete. 2023-10-14T16:56:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.950+0000 7f03cd421640 1 Processor -- start 2023-10-14T16:56:31.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.950+0000 7f03cd421640 1 -- start start 2023-10-14T16:56:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.951+0000 7f03cd421640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.951+0000 7f03cd421640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80c2c90 0x7f03c81f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03cd421640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03c80c3450 0x7f03c80c38b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03cd421640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c80c3df0 con 0x7f03b0004d00 2023-10-14T16:56:31.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03cd421640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81f6f10 con 0x7f03c80c3450 2023-10-14T16:56:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03cd421640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03c81f71f0 con 0x7f03c80c2c90 2023-10-14T16:56:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39762/0 (socket says 172.21.15.29:39762) 2023-10-14T16:56:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 -- 172.21.15.29:0/3710228135 learned_addr learned my addr 172.21.15.29:0/3710228135 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:31.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 -- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80c2c90 msgr2=0x7f03c81f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80c2c90 0x7f03c81f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 -- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03c80c3450 msgr2=0x7f03c80c38b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03c80c3450 0x7f03c80c38b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.952+0000 7f03c7fff640 1 -- 172.21.15.29:0/3710228135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c81f74d0 con 0x7f03b0004d00 2023-10-14T16:56:31.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03c7fff640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f03b8011600 tx=0x7f03b8011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800f8c0 con 0x7f03b0004d00 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03b800fa20 con 0x7f03b0004d00 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b800fdd0 con 0x7f03b0004d00 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03c80c5770 con 0x7f03b0004d00 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.953+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03c815f1f0 con 0x7f03b0004d00 2023-10-14T16:56:31.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.954+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03c804e9b0 con 0x7f03b0004d00 2023-10-14T16:56:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.955+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03b8015460 con 0x7f03b0004d00 2023-10-14T16:56:31.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.956+0000 7f03c4ff9640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 0x7f03b406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:31.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.956+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f03b8089f00 con 0x7f03b0004d00 2023-10-14T16:56:31.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.957+0000 7f03c77fe640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 0x7f03b406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:31.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.957+0000 7f03c77fe640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 0x7f03b406d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03bc007fd0 tx=0x7f03bc009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:31.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:31.960+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03b8054f00 con 0x7f03b0004d00 2023-10-14T16:56:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.237+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f03c806caf0 con 0x7f03b0004d00 2023-10-14T16:56:32.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.241+0000 7f03c4ff9640 1 -- 172.21.15.29:0/3710228135 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v14) v1 ==== 190+0+147 (secure 0 0 0) 0x7f03b80220e0 con 0x7f03b0004d00 2023-10-14T16:56:32.242 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:56:32.243 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:56:32.243 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:56:32.243 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:56:32.243 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:56:32.243 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:56:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 msgr2=0x7f03b406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 0x7f03b406d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f03bc007fd0 tx=0x7f03bc009040 comp rx=0 tx=0).stop 2023-10-14T16:56:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 msgr2=0x7f03c815ecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7f03b8011600 tx=0x7f03b8011ac0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 shutdown_connections 2023-10-14T16:56:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f03b406afc0 0x7f03b406d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03c80c3450 0x7f03c80c38b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03c80c2c90 0x7f03c81f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 --2- 172.21.15.29:0/3710228135 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03b0004d00 0x7f03c815ecb0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 >> 172.21.15.29:0/3710228135 conn(0x7f03c80ce6a0 msgr2=0x7f03c8163540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 shutdown_connections 2023-10-14T16:56:32.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.244+0000 7f03cd421640 1 -- 172.21.15.29:0/3710228135 wait complete. 2023-10-14T16:56:32.264 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:32.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.489+0000 7fb028bbd640 1 Processor -- start 2023-10-14T16:56:32.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.489+0000 7fb028bbd640 1 -- start start 2023-10-14T16:56:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb028bbd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb0240b8480 0x7fb0240cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb028bbd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0240d0be0 0x7fb0240cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb028bbd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb02415d150 con 0x7fb02415cd70 2023-10-14T16:56:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb028bbd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0240d1180 con 0x7fb0240d0be0 2023-10-14T16:56:32.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb028bbd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0240d1460 con 0x7fb0240b8480 2023-10-14T16:56:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb023fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0240d0be0 0x7fb0240cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb0237fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb02415cd70 0x7fb0240d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43864/0 (socket says 172.21.15.29:43864) 2023-10-14T16:56:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb023fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb0240d0be0 0x7fb0240cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39768/0 (socket says 172.21.15.29:39768) 2023-10-14T16:56:32.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb0237fe640 1 -- 172.21.15.29:0/436285952 learned_addr learned my addr 172.21.15.29:0/436285952 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.491+0000 7fb022ffd640 1 --2- 172.21.15.29:0/436285952 >> v2:172.21.15.186:3300/0 conn(0x7fb0240b8480 0x7fb0240cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.492+0000 7fb023fff640 1 -- 172.21.15.29:0/436285952 >> v2:172.21.15.186:3300/0 conn(0x7fb0240b8480 msgr2=0x7fb0240cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.492+0000 7fb023fff640 1 --2- 172.21.15.29:0/436285952 >> v2:172.21.15.186:3300/0 conn(0x7fb0240b8480 0x7fb0240cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.492+0000 7fb023fff640 1 -- 172.21.15.29:0/436285952 >> v1:172.21.15.29:6789/0 conn(0x7fb02415cd70 legacy=0x7fb0240d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.492+0000 7fb023fff640 1 -- 172.21.15.29:0/436285952 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0240d1740 con 0x7fb0240d0be0 2023-10-14T16:56:32.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.492+0000 7fb0227fc640 1 -- 172.21.15.29:0/436285952 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843378528 0 0) 0x7fb02415d150 con 0x7fb02415cd70 2023-10-14T16:56:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.493+0000 7fb023fff640 1 --2- 172.21.15.29:0/436285952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240d0be0 0x7fb0240cdd00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb024066a00 tx=0x7fb01c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821663118f2ce88a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:32.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.493+0000 7fb0227fc640 1 -- 172.21.15.29:0/436285952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c02e070 con 0x7fb0240d0be0 2023-10-14T16:56:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.493+0000 7fb0227fc640 1 -- 172.21.15.29:0/436285952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01c0315a0 con 0x7fb0240d0be0 2023-10-14T16:56:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.493+0000 7fb0227fc640 1 -- 172.21.15.29:0/436285952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01c031930 con 0x7fb0240d0be0 2023-10-14T16:56:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 -- 172.21.15.29:0/436285952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240d0be0 msgr2=0x7fb0240cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 --2- 172.21.15.29:0/436285952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240d0be0 0x7fb0240cdd00 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fb024066a00 tx=0x7fb01c02b840 comp rx=0 tx=0).stop 2023-10-14T16:56:32.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 -- 172.21.15.29:0/436285952 shutdown_connections 2023-10-14T16:56:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 --2- 172.21.15.29:0/436285952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240d0be0 0x7fb0240cdd00 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 --2- 172.21.15.29:0/436285952 >> v2:172.21.15.186:3300/0 conn(0x7fb0240b8480 0x7fb0240cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 -- 172.21.15.29:0/436285952 >> 172.21.15.29:0/436285952 conn(0x7fb0240cf570 msgr2=0x7fb0240cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 -- 172.21.15.29:0/436285952 shutdown_connections 2023-10-14T16:56:32.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.494+0000 7fb028bbd640 1 -- 172.21.15.29:0/436285952 wait complete. 2023-10-14T16:56:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.496+0000 7fb028bbd640 1 Processor -- start 2023-10-14T16:56:32.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.496+0000 7fb028bbd640 1 -- start start 2023-10-14T16:56:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 0x7fb0240cbde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 0x7fb024148ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0240cb9c0 con 0x7fb0240b8480 2023-10-14T16:56:32.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb024159b70 con 0x7fb02415cd70 2023-10-14T16:56:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb028bbd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0240cad40 con 0x7fb0240d0be0 2023-10-14T16:56:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb0237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb0237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39770/0 (socket says 172.21.15.29:39770) 2023-10-14T16:56:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.497+0000 7fb0237fe640 1 -- 172.21.15.29:0/1614161012 learned_addr learned my addr 172.21.15.29:0/1614161012 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:32.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb022ffd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 0x7fb0240cbde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb023fff640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 0x7fb024148ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 -- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 msgr2=0x7fb0240cbde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 0x7fb0240cbde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 -- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 msgr2=0x7fb024148ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 0x7fb024148ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 -- 172.21.15.29:0/1614161012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb01c009400 con 0x7fb0240b8480 2023-10-14T16:56:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb023fff640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 0x7fb024148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb022ffd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 0x7fb0240cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb0237fe640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb01400c8c0 tx=0x7fb01400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:32.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400a640 con 0x7fb0240b8480 2023-10-14T16:56:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb01400a7c0 con 0x7fb0240b8480 2023-10-14T16:56:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.498+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb01400ab70 con 0x7fb0240b8480 2023-10-14T16:56:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.499+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0240cb020 con 0x7fb0240b8480 2023-10-14T16:56:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.499+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0240cb5e0 con 0x7fb0240b8480 2023-10-14T16:56:32.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.500+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02404e9b0 con 0x7fb0240b8480 2023-10-14T16:56:32.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.501+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb01400acd0 con 0x7fb0240b8480 2023-10-14T16:56:32.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.502+0000 7fb00bfff640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 0x7fb01006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:32.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.502+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb014088d30 con 0x7fb0240b8480 2023-10-14T16:56:32.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.502+0000 7fb022ffd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 0x7fb01006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:32.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.503+0000 7fb022ffd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 0x7fb01006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb018006fd0 tx=0x7fb018008040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:32.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.506+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb014053d30 con 0x7fb0240b8480 2023-10-14T16:56:32.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.744+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0240ccbf0 con 0x7fb0240b8480 2023-10-14T16:56:32.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.746+0000 7fb00bfff640 1 -- 172.21.15.29:0/1614161012 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fb014018070 con 0x7fb0240b8480 2023-10-14T16:56:32.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:32.747 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 msgr2=0x7fb01006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 0x7fb01006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb018006fd0 tx=0x7fb018008040 comp rx=0 tx=0).stop 2023-10-14T16:56:32.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 msgr2=0x7fb0241f1db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7fb01400c8c0 tx=0x7fb01400cd80 comp rx=0 tx=0).stop 2023-10-14T16:56:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 shutdown_connections 2023-10-14T16:56:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fb01006af90 0x7fb01006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb02415cd70 0x7fb024148ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0240d0be0 0x7fb0240cbde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 --2- 172.21.15.29:0/1614161012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0240b8480 0x7fb0241f1db0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.748+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 >> 172.21.15.29:0/1614161012 conn(0x7fb0240cf570 msgr2=0x7fb0240cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.749+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 shutdown_connections 2023-10-14T16:56:32.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:32.749+0000 7fb028bbd640 1 -- 172.21.15.29:0/1614161012 wait complete. 2023-10-14T16:56:33.170 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 32, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:56:26.105735+0000', 'modified': '2023-10-14T16:56:29.167922+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6510}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6510': {'gid': 6510, 'name': 'a', 'rank': 0, 'incarnation': 28, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/3113871431', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 3113871431}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 3113871431}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [9], 'metadata_pool': 8, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:56:33.171 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:56:33.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-14T16:56:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.406+0000 7fbb6c1fb640 1 Processor -- start 2023-10-14T16:56:33.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.406+0000 7fbb6c1fb640 1 -- start start 2023-10-14T16:56:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.407+0000 7fbb6c1fb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb6415d030 0x7fbb640d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb6c1fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb640cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb6c1fb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640b5d90 con 0x7fbb6415d410 2023-10-14T16:56:33.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb6c1fb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb640d2b50 con 0x7fbb6415d030 2023-10-14T16:56:33.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb6c1fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cdbf0 con 0x7fbb640cd260 2023-10-14T16:56:33.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb69796640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbb6415d410 0x7fbb64161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43880/0 (socket says 172.21.15.29:43880) 2023-10-14T16:56:33.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb69796640 1 -- 172.21.15.29:0/3542822287 learned_addr learned my addr 172.21.15.29:0/3542822287 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:33.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb6a798640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb640cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.408+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3542822287 >> v2:172.21.15.186:3300/0 conn(0x7fbb6415d030 0x7fbb640d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3099808430 0 0) 0x7fbb640b5d90 con 0x7fbb6415d410 2023-10-14T16:56:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb6415d410 2023-10-14T16:56:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 945322398 0 0) 0x7fbb50003520 con 0x7fbb6415d410 2023-10-14T16:56:33.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 msgr2=0x7fbb640cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb640cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 >> v2:172.21.15.186:3300/0 conn(0x7fbb6415d030 msgr2=0x7fbb640d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 --2- 172.21.15.29:0/3542822287 >> v2:172.21.15.186:3300/0 conn(0x7fbb6415d030 0x7fbb640d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb64160fc0 con 0x7fbb6415d410 2023-10-14T16:56:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fbb54001690 con 0x7fbb6415d410 2023-10-14T16:56:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 >> v1:172.21.15.29:6789/0 conn(0x7fbb6415d410 legacy=0x7fbb64161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb50003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb500047c0 0x7fbb50004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cdbf0 con 0x7fbb50003970 2023-10-14T16:56:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000b980 con 0x7fbb500047c0 2023-10-14T16:56:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.409+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5000baf0 con 0x7fbb640cd260 2023-10-14T16:56:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb50003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb6a798640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69796640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb500047c0 0x7fbb50004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 -- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 msgr2=0x7fbb50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb50005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 -- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb500047c0 msgr2=0x7fbb50004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb500047c0 0x7fbb50004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69f97640 1 -- 172.21.15.29:0/3542822287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb50003520 con 0x7fbb50003970 2023-10-14T16:56:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb6a798640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cd260 0x7fbb50005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.410+0000 7fbb69796640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb500047c0 0x7fbb50004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:33.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.411+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb50003d90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009fc0 tx=0x7fbb500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=963828f0a9318e4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb6002cc60 con 0x7fbb50003970 2023-10-14T16:56:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb6000f040 con 0x7fbb50003970 2023-10-14T16:56:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb68f95640 1 -- 172.21.15.29:0/3542822287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60034440 con 0x7fbb50003970 2023-10-14T16:56:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3542822287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 msgr2=0x7fbb50003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3542822287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb50003d90 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fbb60009fc0 tx=0x7fbb500057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6a798640 1 -- 172.21.15.29:0/3542822287 reap_dead start 2023-10-14T16:56:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3542822287 shutdown_connections 2023-10-14T16:56:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3542822287 >> 172.21.15.29:0/3542822287 conn(0x7fbb640cf720 msgr2=0x7fbb640cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3542822287 shutdown_connections 2023-10-14T16:56:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.412+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3542822287 wait complete. 2023-10-14T16:56:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.414+0000 7fbb6c1fb640 1 Processor -- start 2023-10-14T16:56:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.414+0000 7fbb6c1fb640 1 -- start start 2023-10-14T16:56:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640cc420 0x7fbb6415e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cc8a0 0x7fbb640b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cb0a0 con 0x7fbb50003970 2023-10-14T16:56:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cb380 con 0x7fbb640cc420 2023-10-14T16:56:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.415+0000 7fbb6c1fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb640cb660 con 0x7fbb640cc8a0 2023-10-14T16:56:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39794/0 (socket says 172.21.15.29:39794) 2023-10-14T16:56:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 -- 172.21.15.29:0/3491718634 learned_addr learned my addr 172.21.15.29:0/3491718634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 -- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cc8a0 msgr2=0x7fbb640b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cc8a0 0x7fbb640b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 -- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640cc420 msgr2=0x7fbb6415e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640cc420 0x7fbb6415e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 -- 172.21.15.29:0/3491718634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640cb940 con 0x7fbb50003970 2023-10-14T16:56:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.416+0000 7fbb69f97640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbb60037040 tx=0x7fbb60009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.417+0000 7fbb5a7fc640 1 -- 172.21.15.29:0/3491718634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60034bd0 con 0x7fbb50003970 2023-10-14T16:56:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.417+0000 7fbb5a7fc640 1 -- 172.21.15.29:0/3491718634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60034d30 con 0x7fbb50003970 2023-10-14T16:56:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.417+0000 7fbb5a7fc640 1 -- 172.21.15.29:0/3491718634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60034ea0 con 0x7fbb50003970 2023-10-14T16:56:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.417+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640cbc20 con 0x7fbb50003970 2023-10-14T16:56:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.417+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641554f0 con 0x7fbb50003970 2023-10-14T16:56:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.419+0000 7fbb5a7fc640 1 -- 172.21.15.29:0/3491718634 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbb60040450 con 0x7fbb50003970 2023-10-14T16:56:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.420+0000 7fbb5a7fc640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 0x7fbb4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.420+0000 7fbb5a7fc640 1 -- 172.21.15.29:0/3491718634 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbb600abfe0 con 0x7fbb50003970 2023-10-14T16:56:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.420+0000 7fbb69796640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 0x7fbb4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.421+0000 7fbb69796640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 0x7fbb4806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001b90 tx=0x7fbb54000ed0 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.421+0000 7fbb6ed8d740 1 Processor -- start 2023-10-14T16:56:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 -- start start 2023-10-14T16:56:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222ccd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d22229d3c0 0x55d2222cd250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2221bd7a0 con 0x55d2222c9f80 2023-10-14T16:56:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d22220f740 con 0x55d22229d3c0 2023-10-14T16:56:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.422+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d22220ce30 con 0x55d2222ca360 2023-10-14T16:56:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39800/0 (socket says 172.21.15.29:39800) 2023-10-14T16:56:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 -- 172.21.15.29:0/627971390 learned_addr learned my addr 172.21.15.29:0/627971390 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69796640 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222ccd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 -- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 msgr2=0x55d2222ccd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222ccd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb6a798640 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d22229d3c0 0x55d2222cd250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 -- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d22229d3c0 msgr2=0x55d2222cd250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d22229d3c0 0x55d2222cd250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.423+0000 7fbb69f97640 1 -- 172.21.15.29:0/627971390 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2221cc910 con 0x55d2222c9f80 2023-10-14T16:56:33.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.424+0000 7fbb69f97640 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbb60082a80 tx=0x7fbb6007b3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=338e8ad3583f13d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.424+0000 7fbb477fe640 1 -- 172.21.15.29:0/627971390 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb60082fb0 con 0x55d2222c9f80 2023-10-14T16:56:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.424+0000 7fbb477fe640 1 -- 172.21.15.29:0/627971390 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60009c10 con 0x55d2222c9f80 2023-10-14T16:56:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.424+0000 7fbb477fe640 1 -- 172.21.15.29:0/627971390 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600773c0 con 0x55d2222c9f80 2023-10-14T16:56:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/627971390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 msgr2=0x55d22229ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fbb60082a80 tx=0x7fbb6007b3b0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/627971390 shutdown_connections 2023-10-14T16:56:33.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d22229d3c0 0x55d2222cd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222ccd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/627971390 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d22229ce80 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/627971390 >> 172.21.15.29:0/627971390 conn(0x55d22229f190 msgr2=0x55d2222e7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/627971390 shutdown_connections 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.425+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/627971390 wait complete. 2023-10-14T16:56:33.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.426+0000 7fbb6ed8d740 1 -- start start 2023-10-14T16:56:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.426+0000 7fbb6ed8d740 1 Processor -- start 2023-10-14T16:56:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 0x55d2222ccad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d222208720 con 0x55d2222cc430 2023-10-14T16:56:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2221a6970 con 0x55d2222c9f80 2023-10-14T16:56:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d2221a6610 con 0x55d2222ca360 2023-10-14T16:56:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6ed8d740 10 client.0 _ll_register_callbacks cb 0x55d2221c89d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:56:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6a798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6a798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39816/0 (socket says 172.21.15.29:39816) 2023-10-14T16:56:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb6a798640 1 -- 172.21.15.29:0/4110626539 learned_addr learned my addr 172.21.15.29:0/4110626539 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.427+0000 7fbb69f97640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 0x55d2222ccad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb69796640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 -- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 msgr2=0x55d2222d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 -- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 msgr2=0x55d2222ccad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 0x55d2222ccad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 -- 172.21.15.29:0/4110626539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641554f0 con 0x55d2222cc430 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb69796640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb69f97640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 0x55d2222ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:33.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6a798640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbb64073730 tx=0x7fbb5c012760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c6a1c04a3041e40 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:56:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c004440 con 0x55d2222cc430 2023-10-14T16:56:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbb5c003610 con 0x55d2222cc430 2023-10-14T16:56:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.428+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d22222f160 con 0x55d2222cc430 2023-10-14T16:56:33.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb5c0037d0 con 0x55d2222cc430 2023-10-14T16:56:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 10 client.6660 fetch_fsmap learned FSMap version 32 2023-10-14T16:56:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55d222206da0 con 0x55d2222cc430 2023-10-14T16:56:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d22222f160 con 0x55d2222cc430 2023-10-14T16:56:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbb5c01d060 con 0x55d2222cc430 2023-10-14T16:56:33.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 10 client.6660 fetch_fsmap finished waiting for FSMap version 32 2023-10-14T16:56:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 10 client.6660 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T16:56:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 20 client.6660 populate_metadata read hostname 'smithi029' 2023-10-14T16:56:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d2222fb4b0 0x55d2223050a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 4 client.6660 mds_command: new command op to 6510 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:56:33.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.429+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 --> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d2221dacc0 con 0x55d2222fb4b0 2023-10-14T16:56:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.430+0000 7fbb69f97640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d2222fb4b0 0x55d2223050a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.430+0000 7fbb69f97640 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d2222fb4b0 0x55d2223050a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdf6d7a2d9b621e0 server_cookie=95e07c1f326a64e3 in_seq=0 out_seq=0 2023-10-14T16:56:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.430+0000 7fbb44ff9640 10 client.6660 ms_handle_connect on v2:172.21.15.29:6836/3113871431 2023-10-14T16:56:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb44ff9640 1 -- 172.21.15.29:0/4110626539 <== mds.0 v2:172.21.15.29:6836/3113871431 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d2221dacc0 con 0x55d2222fb4b0 2023-10-14T16:56:33.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb44ff9640 10 client.6660 handle_command_reply: tid=0 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 client.6660 shutdown 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 msgr2=0x55d2222f75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fbb64073730 tx=0x7fbb5c012760 comp rx=0 tx=0).stop 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 shutdown_connections 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d2222fb4b0 0x55d2223050a0 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d2222f75b0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222ca360 0x55d2222d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/4110626539 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222c9f80 0x55d2222ccad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb44ff9640 0 client.6660 ms_handle_reset on v2:172.21.15.29:6836/3113871431 2023-10-14T16:56:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.431+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 >> 172.21.15.29:0/4110626539 conn(0x55d22229d3c0 msgr2=0x55d2222eb220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.432+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 shutdown_connections 2023-10-14T16:56:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.432+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/4110626539 wait complete. 2023-10-14T16:56:33.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.432+0000 7fbb6ed8d740 20 client.6660 trim_cache size 0 max 16384 2023-10-14T16:56:33.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.448+0000 7fbb6ed8d740 1 Processor -- start 2023-10-14T16:56:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 -- start start 2023-10-14T16:56:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2222d0230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 0x55d2222decf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d221f6cf20 con 0x55d2222c9f80 2023-10-14T16:56:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2222cb700 con 0x55d2222ca360 2023-10-14T16:56:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.449+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d2222df230 con 0x55d2222cc430 2023-10-14T16:56:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39824/0 (socket says 172.21.15.29:39824) 2023-10-14T16:56:33.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 -- 172.21.15.29:0/182767038 learned_addr learned my addr 172.21.15.29:0/182767038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 -- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 msgr2=0x55d2222decf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb6a798640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 0x55d2222decf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69796640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2222d0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 0x55d2222decf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 -- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 msgr2=0x55d2222d0230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2222d0230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69f97640 1 -- 172.21.15.29:0/182767038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d2221a5b80 con 0x55d2222c9f80 2023-10-14T16:56:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb69796640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2222d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.450+0000 7fbb6a798640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 0x55d2222decf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb69f97640 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbb6007a9b0 tx=0x7fbb60044000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad9945429cc1cbfb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb47fff640 1 -- 172.21.15.29:0/182767038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600773c0 con 0x55d2222c9f80 2023-10-14T16:56:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb47fff640 1 -- 172.21.15.29:0/182767038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60077520 con 0x55d2222c9f80 2023-10-14T16:56:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb47fff640 1 -- 172.21.15.29:0/182767038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb600aa670 con 0x55d2222c9f80 2023-10-14T16:56:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/182767038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 msgr2=0x55d2222a0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.451+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fbb6007a9b0 tx=0x7fbb60044000 comp rx=0 tx=0).stop 2023-10-14T16:56:33.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/182767038 shutdown_connections 2023-10-14T16:56:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222cc430 0x55d2222decf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2222d0230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/182767038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222c9f80 0x55d2222a0a80 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/182767038 >> 172.21.15.29:0/182767038 conn(0x55d22229d3c0 msgr2=0x55d2222a2b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/182767038 shutdown_connections 2023-10-14T16:56:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/182767038 wait complete. 2023-10-14T16:56:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.452+0000 7fbb6ed8d740 1 -- start start 2023-10-14T16:56:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.453+0000 7fbb6ed8d740 1 Processor -- start 2023-10-14T16:56:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.453+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222c9f80 0x55d2222ea180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.453+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2223533b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6ed8d740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d22234fd90 con 0x55d2222cc430 2023-10-14T16:56:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d2223538f0 con 0x55d2222ca360 2023-10-14T16:56:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6ed8d740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d222353d20 con 0x55d2222c9f80 2023-10-14T16:56:33.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6ed8d740 10 client.0 _ll_register_callbacks cb 0x7fbb5c0094a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6a798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6a798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39840/0 (socket says 172.21.15.29:39840) 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb6a798640 1 -- 172.21.15.29:0/1791374990 learned_addr learned my addr 172.21.15.29:0/1791374990 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb69f97640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222c9f80 0x55d2222ea180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.454+0000 7fbb69796640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2223533b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 -- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222c9f80 msgr2=0x55d2222ea180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222c9f80 0x55d2222ea180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 -- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 msgr2=0x55d2223533b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2223533b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 -- 172.21.15.29:0/1791374990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d2221cc910 con 0x55d2222cc430 2023-10-14T16:56:33.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6a798640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0127a0 tx=0x7fbb5c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92aadfdc55f4ed3a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb44ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:56:33.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c003e30 con 0x55d2222cc430 2023-10-14T16:56:33.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbb5c005990 con 0x55d2222cc430 2023-10-14T16:56:33.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.455+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbb60009c10 con 0x55d2222cc430 2023-10-14T16:56:33.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbb5c005b50 con 0x55d2222cc430 2023-10-14T16:56:33.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 10 client.6666 fetch_fsmap learned FSMap version 32 2023-10-14T16:56:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=2+}) v3 -- 0x55d2221cb170 con 0x55d2222cc430 2023-10-14T16:56:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb60009c10 con 0x55d2222cc430 2023-10-14T16:56:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 32) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fbb5c01d020 con 0x55d2222cc430 2023-10-14T16:56:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 10 client.6666 fetch_fsmap finished waiting for FSMap version 32 2023-10-14T16:56:33.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 10 client.6666 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T16:56:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 20 client.6666 populate_metadata read hostname 'smithi029' 2023-10-14T16:56:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d22229be40 0x55d22229c220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 4 client.6666 mds_command: new command op to 6510 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:56:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 --> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d222201700 con 0x55d22229be40 2023-10-14T16:56:33.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.456+0000 7fbb69f97640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d22229be40 0x55d22229c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb69f97640 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d22229be40 0x55d22229c220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6065fb200cfda260 server_cookie=734887847051b84c in_seq=0 out_seq=0 2023-10-14T16:56:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb44ff9640 10 client.6666 ms_handle_connect on v2:172.21.15.29:6836/3113871431 2023-10-14T16:56:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb44ff9640 1 -- 172.21.15.29:0/1791374990 <== mds.0 v2:172.21.15.29:6836/3113871431 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x55d222201700 con 0x55d22229be40 2023-10-14T16:56:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb44ff9640 10 client.6666 handle_command_reply: tid=0 2023-10-14T16:56:33.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 client.6666 shutdown 2023-10-14T16:56:33.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 msgr2=0x55d222339010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c0127a0 tx=0x7fbb5c004730 comp rx=0 tx=0).stop 2023-10-14T16:56:33.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 shutdown_connections 2023-10-14T16:56:33.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:6836/3113871431,v1:172.21.15.29:6837/3113871431] conn(0x55d22229be40 0x55d22229c220 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d2222cc430 0x55d222339010 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d2222ca360 0x55d2223533b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 --2- 172.21.15.29:0/1791374990 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d2222c9f80 0x55d2222ea180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb44ff9640 0 client.6666 ms_handle_reset on v2:172.21.15.29:6836/3113871431 2023-10-14T16:56:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.457+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 >> 172.21.15.29:0/1791374990 conn(0x55d22229d3c0 msgr2=0x55d2222d0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 shutdown_connections 2023-10-14T16:56:33.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6ed8d740 1 -- 172.21.15.29:0/1791374990 wait complete. 2023-10-14T16:56:33.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6ed8d740 20 client.6666 trim_cache size 0 max 16384 2023-10-14T16:56:33.475 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:56:33.476 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:56:33.476 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:56:33.476 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 6510, 2023-10-14T16:56:33.476 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:56:33.476 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:56:33.477 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:56:33.477 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8172543360000004, 2023-10-14T16:56:33.477 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 32, 2023-10-14T16:56:33.477 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 82, 2023-10-14T16:56:33.477 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 82, 2023-10-14T16:56:33.478 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.200351659000001 2023-10-14T16:56:33.478 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:56:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 msgr2=0x7fbb4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 0x7fbb4806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fbb54001b90 tx=0x7fbb54000ed0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 msgr2=0x7fbb64154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.458+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fbb60037040 tx=0x7fbb60009750 comp rx=0 tx=0).stop 2023-10-14T16:56:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 shutdown_connections 2023-10-14T16:56:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7fbb4806af90 0x7fbb4806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb640cc8a0 0x7fbb640b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640cc420 0x7fbb6415e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 --2- 172.21.15.29:0/3491718634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb50003970 0x7fbb64154fb0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 >> 172.21.15.29:0/3491718634 conn(0x7fbb640cf720 msgr2=0x7fbb640c5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 shutdown_connections 2023-10-14T16:56:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.459+0000 7fbb6c1fb640 1 -- 172.21.15.29:0/3491718634 wait complete. 2023-10-14T16:56:33.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:56:33.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.693+0000 7ff7041dc640 1 Processor -- start 2023-10-14T16:56:33.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.694+0000 7ff7041dc640 1 -- start start 2023-10-14T16:56:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.695+0000 7ff7041dc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 0x7ff6fc06e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.695+0000 7ff7041dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.695+0000 7ff7041dc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc06a840 con 0x7ff6fc0cf4c0 2023-10-14T16:56:33.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.695+0000 7ff7041dc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6fc0cd970 con 0x7ff6fc06e510 2023-10-14T16:56:33.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.695+0000 7ff7041dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0cf8a0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.696+0000 7ff702779640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.696+0000 7ff702779640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:50508/0 (socket says 172.21.15.29:50508) 2023-10-14T16:56:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.696+0000 7ff702779640 1 -- 172.21.15.29:0/2447116957 learned_addr learned my addr 172.21.15.29:0/2447116957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.696+0000 7ff701777640 1 --2- 172.21.15.29:0/2447116957 >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 0x7ff6fc06e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.696+0000 7ff702779640 1 -- 172.21.15.29:0/2447116957 >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 msgr2=0x7ff6fc06e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.697+0000 7ff702779640 1 --2- 172.21.15.29:0/2447116957 >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 0x7ff6fc06e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.697+0000 7ff702779640 1 -- 172.21.15.29:0/2447116957 >> v1:172.21.15.186:6789/0 conn(0x7ff6fc0cf4c0 legacy=0x7ff6fc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.697+0000 7ff702779640 1 -- 172.21.15.29:0/2447116957 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc15c350 con 0x7ff6fc0705d0 2023-10-14T16:56:33.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.697+0000 7ff701777640 1 --2- 172.21.15.29:0/2447116957 >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 0x7ff6fc06e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:33.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.698+0000 7ff702779640 1 --2- 172.21.15.29:0/2447116957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009e50 tx=0x7ff6f402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5d90ffbc3efaca1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.703+0000 7ff700f76640 1 -- 172.21.15.29:0/2447116957 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f402cb00 con 0x7ff6fc0705d0 2023-10-14T16:56:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.703+0000 7ff700f76640 1 -- 172.21.15.29:0/2447116957 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f402cc60 con 0x7ff6fc0705d0 2023-10-14T16:56:33.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.703+0000 7ff700f76640 1 -- 172.21.15.29:0/2447116957 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f40343f0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 -- 172.21.15.29:0/2447116957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 msgr2=0x7ff6fc070a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 --2- 172.21.15.29:0/2447116957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff6f4009e50 tx=0x7ff6f402bf60 comp rx=0 tx=0).stop 2023-10-14T16:56:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 -- 172.21.15.29:0/2447116957 shutdown_connections 2023-10-14T16:56:33.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 --2- 172.21.15.29:0/2447116957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc0705d0 0x7ff6fc070a20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 --2- 172.21.15.29:0/2447116957 >> v2:172.21.15.186:3300/0 conn(0x7ff6fc06e510 0x7ff6fc06e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 -- 172.21.15.29:0/2447116957 >> 172.21.15.29:0/2447116957 conn(0x7ff6fc0b1560 msgr2=0x7ff6fc0b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 -- 172.21.15.29:0/2447116957 shutdown_connections 2023-10-14T16:56:33.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.704+0000 7ff7041dc640 1 -- 172.21.15.29:0/2447116957 wait complete. 2023-10-14T16:56:33.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.706+0000 7ff7041dc640 1 Processor -- start 2023-10-14T16:56:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.706+0000 7ff7041dc640 1 -- start start 2023-10-14T16:56:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc06e510 0x7ff6fc0c98c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6fc0cf4c0 0x7ff6fc149eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0b3e90 con 0x7ff6fc0705d0 2023-10-14T16:56:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0b3b40 con 0x7ff6fc0cf4c0 2023-10-14T16:56:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff7041dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6fc0c6d80 con 0x7ff6fc06e510 2023-10-14T16:56:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff701777640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff701777640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39844/0 (socket says 172.21.15.29:39844) 2023-10-14T16:56:33.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.707+0000 7ff701777640 1 -- 172.21.15.29:0/3443661799 learned_addr learned my addr 172.21.15.29:0/3443661799 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 -- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc06e510 msgr2=0x7ff6fc0c98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc06e510 0x7ff6fc0c98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 -- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6fc0cf4c0 msgr2=0x7ff6fc149eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6fc0cf4c0 0x7ff6fc149eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 -- 172.21.15.29:0/3443661799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f4009b30 con 0x7ff6fc0705d0 2023-10-14T16:56:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff701777640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00c7f0 tx=0x7ff6ec00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec007820 con 0x7ff6fc0705d0 2023-10-14T16:56:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6ec007980 con 0x7ff6fc0705d0 2023-10-14T16:56:33.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6ec007cf0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc14a3f0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.708+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc1f43d0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.710+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc04e160 con 0x7ff6fc0705d0 2023-10-14T16:56:33.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.710+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6ec0113f0 con 0x7ff6fc0705d0 2023-10-14T16:56:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.711+0000 7ff6f27fc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.712+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff6ec089c20 con 0x7ff6fc0705d0 2023-10-14T16:56:33.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.712+0000 7ff701f78640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.713+0000 7ff701f78640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 0x7ff6e006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000f60 tx=0x7ff6f80079b0 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:33.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.716+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6ec054a70 con 0x7ff6fc0705d0 2023-10-14T16:56:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.954+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7ff6fc0af820 con 0x7ff6fc0705d0 2023-10-14T16:56:33.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.955+0000 7ff6f27fc640 1 -- 172.21.15.29:0/3443661799 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v32) v1 ==== 53+0+73 (secure 0 0 0) 0x7ff6ec01f070 con 0x7ff6fc0705d0 2023-10-14T16:56:33.957 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:56:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.957+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 msgr2=0x7ff6e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.957+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 0x7ff6e006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff6f8000f60 tx=0x7ff6f80079b0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 msgr2=0x7ff6fc1f39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7ff6ec00c7f0 tx=0x7ff6ec00ccb0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 shutdown_connections 2023-10-14T16:56:33.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff6e006af90 0x7ff6e006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6fc0cf4c0 0x7ff6fc149eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6fc0705d0 0x7ff6fc1f39f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 --2- 172.21.15.29:0/3443661799 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6fc06e510 0x7ff6fc0c98c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 >> 172.21.15.29:0/3443661799 conn(0x7ff6fc0b1560 msgr2=0x7ff6fc15c600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 shutdown_connections 2023-10-14T16:56:33.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:33.958+0000 7ff7041dc640 1 -- 172.21.15.29:0/3443661799 wait complete. 2023-10-14T16:56:33.979 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:56:33.979 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:56:33.980 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:56:33.980 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:56:33.980 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:33.981 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:56:33.981 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:56:33.981 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:56:33.982 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:56:33.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:33.983 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:56:34.040 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:56:34.040 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:56:34.040 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:56:34.040 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.041 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:56:34.041 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.041 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:56:34.041 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:56:34.041 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:56:34.042 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:56:34.042 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:56:34.042 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:56:34.042 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:56:34.042 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:56:34.043 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:56:34.043 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:56:34.043 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41382sec preferred_lft 41382sec 2023-10-14T16:56:34.043 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:56:34.043 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.044 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:56:34.044 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:56:34.044 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:56:34.044 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:56:34.044 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:56:34.045 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:56:34.045 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.045 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:56:34.045 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.045 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:56:34.046 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:56:34.046 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:56:34.046 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.046 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:56:34.046 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:56:34.047 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:56:34.047 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:56:34.048 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.049 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:56:34.147 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:56:34.152 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.153 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:56:34.213 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:56:34.214 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:56:34.214 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:56:34.214 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:56:34.214 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:56:34.215 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.215 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:56:34.216 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:56:34.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:56:34.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:56:34.217 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:56:34.217 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:56:34.408 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.408 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:56:34.465 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:56:34.466 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:56:34.469 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:56:34.470 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.470 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:56:34.574 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:56:34.575 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:34.637 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:56:34.637 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:56:34.638 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:34.693 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:56:34.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:56:34.838 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:56:34.838 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:56:34.839 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:56:34.863 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:56:34.863 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:56:34.863 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:56:34.863 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:56:34.864 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:56:34.890 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:56:34.890 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:56:34.891 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:56:34.891 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:56:34.891 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:56:34.891 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:56:34.891 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:56:34.892 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:56:34.893 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:56:34.894 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:56:34.894 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:56:34.894 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:56:34.894 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:56:34.897 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:56:34.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:34.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:34.989 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:56:35.060 INFO:teuthology.orchestra.run.smithi029.stdout:client5605 2023-10-14T16:56:35.060 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:35.061 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:35.120 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:56:35.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.343+0000 7f3896db0640 1 Processor -- start 2023-10-14T16:56:35.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.344+0000 7f3896db0640 1 -- start start 2023-10-14T16:56:35.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.345+0000 7f3896db0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f38900caa80 0x7f3890069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.345+0000 7f3896db0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38900caef0 con 0x7f389006a2e0 2023-10-14T16:56:35.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f3896db0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f389006a6f0 con 0x7f38900addb0 2023-10-14T16:56:35.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f3896db0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3890161a50 con 0x7f38900caa80 2023-10-14T16:56:35.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f388ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f38900caa80 0x7f3890069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f389534d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f389006a2e0 0x7f38901650d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43886/0 (socket says 172.21.15.29:43886) 2023-10-14T16:56:35.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f389534d640 1 -- 172.21.15.29:0/1966514640 learned_addr learned my addr 172.21.15.29:0/1966514640 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:35.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f388ffff640 1 -- 172.21.15.29:0/1966514640 >> v1:172.21.15.186:6789/0 conn(0x7f38900addb0 legacy=0x7f38900c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:35.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f388ffff640 1 -- 172.21.15.29:0/1966514640 >> v1:172.21.15.29:6789/0 conn(0x7f389006a2e0 legacy=0x7f38901650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.346+0000 7f388ffff640 1 -- 172.21.15.29:0/1966514640 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38900cb140 con 0x7f38900caa80 2023-10-14T16:56:35.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.347+0000 7f388ffff640 1 --2- 172.21.15.29:0/1966514640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f3890069da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f38800097a0 tx=0x7f388002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e6fbb463922ffd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.348+0000 7f388f7fe640 1 -- 172.21.15.29:0/1966514640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880031830 con 0x7f38900caa80 2023-10-14T16:56:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.348+0000 7f388f7fe640 1 -- 172.21.15.29:0/1966514640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3880031990 con 0x7f38900caa80 2023-10-14T16:56:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.348+0000 7f388f7fe640 1 -- 172.21.15.29:0/1966514640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3880031d20 con 0x7f38900caa80 2023-10-14T16:56:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.348+0000 7f3896db0640 1 -- 172.21.15.29:0/1966514640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 msgr2=0x7f3890069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.348+0000 7f3896db0640 1 --2- 172.21.15.29:0/1966514640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f3890069da0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f38800097a0 tx=0x7f388002b8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.349+0000 7f3896db0640 1 -- 172.21.15.29:0/1966514640 shutdown_connections 2023-10-14T16:56:35.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.349+0000 7f3896db0640 1 --2- 172.21.15.29:0/1966514640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f3890069da0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.349+0000 7f3896db0640 1 -- 172.21.15.29:0/1966514640 >> 172.21.15.29:0/1966514640 conn(0x7f38900ce9b0 msgr2=0x7f38900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:35.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.349+0000 7f3896db0640 1 -- 172.21.15.29:0/1966514640 shutdown_connections 2023-10-14T16:56:35.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.349+0000 7f3896db0640 1 -- 172.21.15.29:0/1966514640 wait complete. 2023-10-14T16:56:35.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.350+0000 7f3896db0640 1 Processor -- start 2023-10-14T16:56:35.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.351+0000 7f3896db0640 1 -- start start 2023-10-14T16:56:35.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f389006a2e0 0x7f389015cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38900addb0 0x7f389015e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389015eb60 con 0x7f38900caa80 2023-10-14T16:56:35.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38900b2d20 con 0x7f389006a2e0 2023-10-14T16:56:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3896db0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38900b3000 con 0x7f38900addb0 2023-10-14T16:56:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f389534d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f389534d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39870/0 (socket says 172.21.15.29:39870) 2023-10-14T16:56:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f389534d640 1 -- 172.21.15.29:0/1770320983 learned_addr learned my addr 172.21.15.29:0/1770320983 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:35.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f3894b4c640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f389006a2e0 0x7f389015cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.352+0000 7f388ffff640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38900addb0 0x7f389015e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 -- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38900addb0 msgr2=0x7f389015e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38900addb0 0x7f389015e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 -- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f389006a2e0 msgr2=0x7f389015cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f389006a2e0 0x7f389015cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 -- 172.21.15.29:0/1770320983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3880009400 con 0x7f38900caa80 2023-10-14T16:56:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f389534d640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f388800cbc0 tx=0x7f388800a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388800ad40 con 0x7f38900caa80 2023-10-14T16:56:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3888012400 con 0x7f38900caa80 2023-10-14T16:56:35.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f388800aea0 con 0x7f38900caa80 2023-10-14T16:56:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.353+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38900b32e0 con 0x7f38900caa80 2023-10-14T16:56:35.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.354+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f389015b200 con 0x7f38900caa80 2023-10-14T16:56:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.355+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38900b3800 con 0x7f38900caa80 2023-10-14T16:56:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.356+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3888007070 con 0x7f38900caa80 2023-10-14T16:56:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.357+0000 7f388d7fa640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 0x7f387c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.357+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f3888089630 con 0x7f38900caa80 2023-10-14T16:56:35.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.358+0000 7f3894b4c640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 0x7f387c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.358+0000 7f3894b4c640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 0x7f387c06d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3884004ea0 tx=0x7f3884009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.362+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38900b3800 con 0x7f38900caa80 2023-10-14T16:56:35.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.581+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 --> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f389006b310 con 0x7f387c06af90 2023-10-14T16:56:35.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.584+0000 7f388d7fa640 1 -- 172.21.15.29:0/1770320983 <== mgr.6252 v2:172.21.15.29:6832/3108857887 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3884005420 con 0x7f387c06af90 2023-10-14T16:56:35.586 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:56:35.586 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:56:35.586 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:56:35.586 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:56:35.587 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:56:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.586+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 msgr2=0x7f387c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.586+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 0x7f387c06d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3884004ea0 tx=0x7f3884009040 comp rx=0 tx=0).stop 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.586+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 msgr2=0x7f38900c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.586+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f388800cbc0 tx=0x7f388800a430 comp rx=0 tx=0).stop 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 shutdown_connections 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f387c06af90 0x7f387c06d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38900caa80 0x7f38900c24a0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38900addb0 0x7f389015e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 --2- 172.21.15.29:0/1770320983 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f389006a2e0 0x7f389015cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 >> 172.21.15.29:0/1770320983 conn(0x7f38900ce9b0 msgr2=0x7f38900ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:35.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 shutdown_connections 2023-10-14T16:56:35.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.587+0000 7f3896db0640 1 -- 172.21.15.29:0/1770320983 wait complete. 2023-10-14T16:56:35.608 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:56:35.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.836+0000 7ff1f81b0640 1 Processor -- start 2023-10-14T16:56:35.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.836+0000 7ff1f81b0640 1 -- start start 2023-10-14T16:56:35.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f81b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 0x7ff1f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f81b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f0070360 con 0x7ff1f0159ab0 2023-10-14T16:56:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f81b0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1f00cd1f0 con 0x7ff1f0159e90 2023-10-14T16:56:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f81b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f015d890 con 0x7ff1f006c850 2023-10-14T16:56:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f5f4c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff1f0159ab0 0x7ff1f006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43902/0 (socket says 172.21.15.29:43902) 2023-10-14T16:56:35.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f5f4c640 1 -- 172.21.15.29:0/1662809175 learned_addr learned my addr 172.21.15.29:0/1662809175 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.838+0000 7ff1f674d640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 0x7ff1f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219464196 0 0) 0x7ff1f0070360 con 0x7ff1f0159ab0 2023-10-14T16:56:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1dc005730 con 0x7ff1f0159ab0 2023-10-14T16:56:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 33665763 0 0) 0x7ff1dc005730 con 0x7ff1f0159ab0 2023-10-14T16:56:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 msgr2=0x7ff1f00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 0x7ff1f00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 >> v1:172.21.15.186:6789/0 conn(0x7ff1f0159e90 legacy=0x7ff1f006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00cd4d0 con 0x7ff1f0159ab0 2023-10-14T16:56:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff1ec001670 con 0x7ff1f0159ab0 2023-10-14T16:56:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 >> v1:172.21.15.29:6789/0 conn(0x7ff1f0159ab0 legacy=0x7ff1f006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 0x7ff1dc005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1dc006130 0x7ff1dc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006f10 0x7ff1dc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f015d890 con 0x7ff1dc006130 2023-10-14T16:56:35.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc00ddd0 con 0x7ff1dc006f10 2023-10-14T16:56:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.839+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1dc00df90 con 0x7ff1f006c850 2023-10-14T16:56:35.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1dc006130 0x7ff1dc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f674d640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006f10 0x7ff1dc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 -- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 msgr2=0x7ff1dc005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f006c850 0x7ff1dc005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 -- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006f10 msgr2=0x7ff1dc007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:35.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006f10 0x7ff1dc007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.840+0000 7ff1f574b640 1 -- 172.21.15.29:0/1662809175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1dc005730 con 0x7ff1dc006130 2023-10-14T16:56:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.841+0000 7ff1f574b640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1dc006130 0x7ff1dc0069d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0001750 tx=0x7ff1dc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0d2acbeeb6168d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e00052d0 con 0x7ff1dc006130 2023-10-14T16:56:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e002e050 con 0x7ff1dc006130 2023-10-14T16:56:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f4f4a640 1 -- 172.21.15.29:0/1662809175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e0003830 con 0x7ff1dc006130 2023-10-14T16:56:35.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1662809175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1dc006130 msgr2=0x7ff1dc0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1662809175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1dc006130 0x7ff1dc0069d0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7ff1e0001750 tx=0x7ff1dc007a00 comp rx=0 tx=0).stop 2023-10-14T16:56:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f674d640 1 -- 172.21.15.29:0/1662809175 reap_dead start 2023-10-14T16:56:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1662809175 shutdown_connections 2023-10-14T16:56:35.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.842+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1662809175 >> 172.21.15.29:0/1662809175 conn(0x7ff1f00ce9b0 msgr2=0x7ff1f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.843+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1662809175 shutdown_connections 2023-10-14T16:56:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.843+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1662809175 wait complete. 2023-10-14T16:56:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.844+0000 7ff1f81b0640 1 Processor -- start 2023-10-14T16:56:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.845+0000 7ff1f81b0640 1 -- start start 2023-10-14T16:56:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.845+0000 7ff1f81b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 0x7ff1f00c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.845+0000 7ff1f81b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f00ac890 0x7ff1f00acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f81b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f81b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f014a650 con 0x7ff1f0149620 2023-10-14T16:56:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f81b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f00ad250 con 0x7ff1dc006130 2023-10-14T16:56:35.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f81b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1f00c8930 con 0x7ff1f00ac890 2023-10-14T16:56:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39890/0 (socket says 172.21.15.29:39890) 2023-10-14T16:56:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 -- 172.21.15.29:0/1340640739 learned_addr learned my addr 172.21.15.29:0/1340640739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:35.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f5f4c640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 0x7ff1f00c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 -- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f00ac890 msgr2=0x7ff1f00acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f00ac890 0x7ff1f00acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 -- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 msgr2=0x7ff1f00c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:35.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 0x7ff1f00c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:35.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f674d640 1 -- 172.21.15.29:0/1340640739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f00c5a80 con 0x7ff1f0149620 2023-10-14T16:56:35.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.846+0000 7ff1f5f4c640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 0x7ff1f00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1f674d640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00ae3e0 tx=0x7ff1e80115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e800e070 con 0x7ff1f0149620 2023-10-14T16:56:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e8011e10 con 0x7ff1f0149620 2023-10-14T16:56:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003990 con 0x7ff1f0149620 2023-10-14T16:56:35.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1f0147de0 con 0x7ff1f0149620 2023-10-14T16:56:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.847+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1f0147790 con 0x7ff1f0149620 2023-10-14T16:56:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.848+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1f01475f0 con 0x7ff1f0149620 2023-10-14T16:56:35.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.849+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 29) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1e8003af0 con 0x7ff1f0149620 2023-10-14T16:56:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.850+0000 7ff1e67fc640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 0x7ff1d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.850+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff1e808a270 con 0x7ff1f0149620 2023-10-14T16:56:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.851+0000 7ff1f5f4c640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 0x7ff1d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.851+0000 7ff1f5f4c640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 0x7ff1d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0011e0 tx=0x7ff1ec009040 comp rx=0 tx=0).ready entity=mgr.6252 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:35.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:35.855+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1f01475f0 con 0x7ff1f0149620 2023-10-14T16:56:36.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.104+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7ff1f01475f0 con 0x7ff1f0149620 2023-10-14T16:56:36.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.142+0000 7ff1e67fc640 1 -- 172.21.15.29:0/1340640739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v30) v1 ==== 92+0+0 (secure 0 0 0) 0x7ff1f01475f0 con 0x7ff1f0149620 2023-10-14T16:56:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 msgr2=0x7ff1d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 0x7ff1d406d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ff1ec0011e0 tx=0x7ff1ec009040 comp rx=0 tx=0).stop 2023-10-14T16:56:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 msgr2=0x7ff1f0149aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7ff1f00ae3e0 tx=0x7ff1e80115a0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 shutdown_connections 2023-10-14T16:56:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7ff1d406af90 0x7ff1d406d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff1f0149620 0x7ff1f0149aa0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1f00ac890 0x7ff1f00acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 --2- 172.21.15.29:0/1340640739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1dc006130 0x7ff1f00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 >> 172.21.15.29:0/1340640739 conn(0x7ff1f00ce9b0 msgr2=0x7ff1f006fc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.146+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 shutdown_connections 2023-10-14T16:56:36.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.147+0000 7ff1f81b0640 1 -- 172.21.15.29:0/1340640739 wait complete. 2023-10-14T16:56:36.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:56:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.375+0000 7f0183be2640 1 Processor -- start 2023-10-14T16:56:36.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.376+0000 7f0183be2640 1 -- start start 2023-10-14T16:56:36.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f0183be2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c1618b0 0x7f017c065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f0183be2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f017c065ff0 0x7f017c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f0183be2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f017c066420 con 0x7f017c15d330 2023-10-14T16:56:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f0183be2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f017c06b1e0 con 0x7f017c065ff0 2023-10-14T16:56:36.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f0183be2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c06cee0 con 0x7f017c1618b0 2023-10-14T16:56:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f018217f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f017c065ff0 0x7f017c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f018217f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f017c065ff0 0x7f017c0ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39924/0 (socket says 172.21.15.29:39924) 2023-10-14T16:56:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f018217f640 1 -- 172.21.15.29:0/4139936496 learned_addr learned my addr 172.21.15.29:0/4139936496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:36.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.378+0000 7f018117d640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c1618b0 0x7f017c065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.379+0000 7f018217f640 1 -- 172.21.15.29:0/4139936496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c1618b0 msgr2=0x7f017c065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:36.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.379+0000 7f018217f640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c1618b0 0x7f017c065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.379+0000 7f018217f640 1 -- 172.21.15.29:0/4139936496 >> v1:172.21.15.186:6789/0 conn(0x7f017c15d330 legacy=0x7f017c06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.379+0000 7f018217f640 1 -- 172.21.15.29:0/4139936496 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f017c0d0890 con 0x7f017c065ff0 2023-10-14T16:56:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.380+0000 7f018217f640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0ce9b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0174009e50 tx=0x7f017402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af17b3543da09312 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.380+0000 7f018097c640 1 -- 172.21.15.29:0/4139936496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f017402c9f0 con 0x7f017c065ff0 2023-10-14T16:56:36.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.380+0000 7f018097c640 1 -- 172.21.15.29:0/4139936496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f017402cb50 con 0x7f017c065ff0 2023-10-14T16:56:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.380+0000 7f018097c640 1 -- 172.21.15.29:0/4139936496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01740343f0 con 0x7f017c065ff0 2023-10-14T16:56:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 -- 172.21.15.29:0/4139936496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 msgr2=0x7f017c0ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0ce9b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f0174009e50 tx=0x7f017402be50 comp rx=0 tx=0).stop 2023-10-14T16:56:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 -- 172.21.15.29:0/4139936496 shutdown_connections 2023-10-14T16:56:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0ce9b0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 --2- 172.21.15.29:0/4139936496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c1618b0 0x7f017c065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.381+0000 7f0183be2640 1 -- 172.21.15.29:0/4139936496 >> 172.21.15.29:0/4139936496 conn(0x7f017c0d0ab0 msgr2=0x7f017c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.382+0000 7f0183be2640 1 -- 172.21.15.29:0/4139936496 shutdown_connections 2023-10-14T16:56:36.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.382+0000 7f0183be2640 1 -- 172.21.15.29:0/4139936496 wait complete. 2023-10-14T16:56:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.384+0000 7f0183be2640 1 Processor -- start 2023-10-14T16:56:36.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.385+0000 7f0183be2640 1 -- start start 2023-10-14T16:56:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c15d330 0x7f017c158fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017c1618b0 0x7f017c1597f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0af7b0 con 0x7f017c065ff0 2023-10-14T16:56:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0c4a70 con 0x7f017c1618b0 2023-10-14T16:56:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.386+0000 7f0183be2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f017c0ca530 con 0x7f017c15d330 2023-10-14T16:56:36.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39938/0 (socket says 172.21.15.29:39938) 2023-10-14T16:56:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 -- 172.21.15.29:0/1321557885 learned_addr learned my addr 172.21.15.29:0/1321557885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018117d640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c15d330 0x7f017c158fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018217f640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017c1618b0 0x7f017c1597f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c15d330 msgr2=0x7f017c158fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c15d330 0x7f017c158fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017c1618b0 msgr2=0x7f017c1597f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017c1618b0 0x7f017c1597f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.387+0000 7f018197e640 1 -- 172.21.15.29:0/1321557885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0174009b30 con 0x7f017c065ff0 2023-10-14T16:56:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f018197e640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178011830 con 0x7f017c065ff0 2023-10-14T16:56:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0178011990 con 0x7f017c065ff0 2023-10-14T16:56:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0178011d40 con 0x7f017c065ff0 2023-10-14T16:56:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017c0ca7d0 con 0x7f017c065ff0 2023-10-14T16:56:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.388+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f017c0adb60 con 0x7f017c065ff0 2023-10-14T16:56:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.389+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017c0af330 con 0x7f017c065ff0 2023-10-14T16:56:36.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.390+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f017801a400 con 0x7f017c065ff0 2023-10-14T16:56:36.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.391+0000 7f01727fc640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 0x7f016006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.391+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f01780895c0 con 0x7f017c065ff0 2023-10-14T16:56:36.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.391+0000 7f018117d640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 msgr2=0x7f016006d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:36.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.391+0000 7f018117d640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 0x7f016006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:36.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.396+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f017c0af330 con 0x7f017c065ff0 2023-10-14T16:56:36.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.592+0000 7f018117d640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 msgr2=0x7f016006d4f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:36.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.592+0000 7f018117d640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 0x7f016006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:36.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.594+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f017c0ad9c0 con 0x7f017c065ff0 2023-10-14T16:56:36.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.599+0000 7f01727fc640 1 -- 172.21.15.29:0/1321557885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v23)=0 v23) v1 ==== 138+0+0 (secure 0 0 0) 0x7f017c0ad9c0 con 0x7f017c065ff0 2023-10-14T16:56:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 msgr2=0x7f016006d4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 0x7f016006d4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 msgr2=0x7f017c0c4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f017800b3f0 tx=0x7f017800b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 shutdown_connections 2023-10-14T16:56:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f016006b030 0x7f016006d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f017c1618b0 0x7f017c1597f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f017c15d330 0x7f017c158fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 --2- 172.21.15.29:0/1321557885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f017c065ff0 0x7f017c0c4530 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 >> 172.21.15.29:0/1321557885 conn(0x7f017c0d0ab0 msgr2=0x7f017c15c3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 shutdown_connections 2023-10-14T16:56:36.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.602+0000 7f0183be2640 1 -- 172.21.15.29:0/1321557885 wait complete. 2023-10-14T16:56:36.621 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:56:36.712 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:36.711+0000 7ff4dfe32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:36.712 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:36.710+0000 7f5394830040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:36.769 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:36.767+0000 7f5394830040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:36.770 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:36.770+0000 7ff4dfe32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.832+0000 7f00964af640 1 Processor -- start 2023-10-14T16:56:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.832+0000 7f00964af640 1 -- start start 2023-10-14T16:56:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.834+0000 7f00964af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.834+0000 7f00964af640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00900c9ca0 0x7f00900ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.834+0000 7f00964af640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00900cb030 con 0x7f009015ad70 2023-10-14T16:56:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.834+0000 7f00964af640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00900cae30 con 0x7f00900c9ca0 2023-10-14T16:56:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.834+0000 7f00964af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00900c91c0 con 0x7f009006a4b0 2023-10-14T16:56:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:50604/0 (socket says 172.21.15.29:50604) 2023-10-14T16:56:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 -- 172.21.15.29:0/1200630563 learned_addr learned my addr 172.21.15.29:0/1200630563 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f0094a4c640 1 --2- 172.21.15.29:0/1200630563 >> v2:172.21.15.186:3300/0 conn(0x7f00900c9ca0 0x7f00900ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 -- 172.21.15.29:0/1200630563 >> v2:172.21.15.186:3300/0 conn(0x7f00900c9ca0 msgr2=0x7f00900ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 --2- 172.21.15.29:0/1200630563 >> v2:172.21.15.186:3300/0 conn(0x7f00900c9ca0 0x7f00900ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 -- 172.21.15.29:0/1200630563 >> v1:172.21.15.186:6789/0 conn(0x7f009015ad70 legacy=0x7f0090069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.835+0000 7f008f7fe640 1 -- 172.21.15.29:0/1200630563 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00900c94a0 con 0x7f009006a4b0 2023-10-14T16:56:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.836+0000 7f008f7fe640 1 --2- 172.21.15.29:0/1200630563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=319191c3d1ac24cd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.837+0000 7f008effd640 1 -- 172.21.15.29:0/1200630563 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084031640 con 0x7f009006a4b0 2023-10-14T16:56:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.837+0000 7f008effd640 1 -- 172.21.15.29:0/1200630563 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00840317a0 con 0x7f009006a4b0 2023-10-14T16:56:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.837+0000 7f008effd640 1 -- 172.21.15.29:0/1200630563 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0084031b30 con 0x7f009006a4b0 2023-10-14T16:56:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.837+0000 7f00964af640 1 -- 172.21.15.29:0/1200630563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 msgr2=0x7f00900ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.837+0000 7f00964af640 1 --2- 172.21.15.29:0/1200630563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f008402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 -- 172.21.15.29:0/1200630563 shutdown_connections 2023-10-14T16:56:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 --2- 172.21.15.29:0/1200630563 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f00900ca3b0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 --2- 172.21.15.29:0/1200630563 >> v2:172.21.15.186:3300/0 conn(0x7f00900c9ca0 0x7f00900ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 -- 172.21.15.29:0/1200630563 >> 172.21.15.29:0/1200630563 conn(0x7f00900ce9b0 msgr2=0x7f00900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:36.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 -- 172.21.15.29:0/1200630563 shutdown_connections 2023-10-14T16:56:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.838+0000 7f00964af640 1 -- 172.21.15.29:0/1200630563 wait complete. 2023-10-14T16:56:36.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.839+0000 7f00964af640 1 Processor -- start 2023-10-14T16:56:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.840+0000 7f00964af640 1 -- start start 2023-10-14T16:56:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.840+0000 7f00964af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f009014ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.840+0000 7f00964af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f00964af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f009015ad70 0x7f00900b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f00964af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00901f3180 con 0x7f00900c9ca0 2023-10-14T16:56:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f00964af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f009015d3f0 con 0x7f009015ad70 2023-10-14T16:56:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f00964af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f009014b350 con 0x7f009006a4b0 2023-10-14T16:56:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39950/0 (socket says 172.21.15.29:39950) 2023-10-14T16:56:36.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 -- 172.21.15.29:0/4040723476 learned_addr learned my addr 172.21.15.29:0/4040723476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:36.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f0094a4c640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f009015ad70 0x7f00900b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008ffff640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f009014ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:36.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 msgr2=0x7f009014ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f009014ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f009015ad70 msgr2=0x7f00900b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:36.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f009015ad70 0x7f00900b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:36.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.841+0000 7f008f7fe640 1 -- 172.21.15.29:0/4040723476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00840094d0 con 0x7f00900c9ca0 2023-10-14T16:56:36.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f008f7fe640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f00840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:36.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008403a930 con 0x7f00900c9ca0 2023-10-14T16:56:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f008403aa90 con 0x7f00900c9ca0 2023-10-14T16:56:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00901f2ea0 con 0x7f00900c9ca0 2023-10-14T16:56:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f008403ae40 con 0x7f00900c9ca0 2023-10-14T16:56:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.842+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f009015d080 con 0x7f00900c9ca0 2023-10-14T16:56:36.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.843+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f009015cc00 con 0x7f00900c9ca0 2023-10-14T16:56:36.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.844+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f008402f050 con 0x7f00900c9ca0 2023-10-14T16:56:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.845+0000 7f008cff9640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 0x7f007c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.845+0000 7f008ffff640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 msgr2=0x7f007c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.845+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f00840ab860 con 0x7f00900c9ca0 2023-10-14T16:56:36.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.845+0000 7f008ffff640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 0x7f007c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:36.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:36.854+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f009015cc00 con 0x7f00900c9ca0 2023-10-14T16:56:37.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.046+0000 7f008ffff640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 msgr2=0x7f007c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:37.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.046+0000 7f008ffff640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 0x7f007c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:37.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.050+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f009015d240 con 0x7f00900c9ca0 2023-10-14T16:56:37.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.054+0000 7f008cff9640 1 -- 172.21.15.29:0/4040723476 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v24)=0 v24) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0084076750 con 0x7f00900c9ca0 2023-10-14T16:56:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 msgr2=0x7f007c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 0x7f007c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 msgr2=0x7f00900ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f0084009870 tx=0x7f00840046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 shutdown_connections 2023-10-14T16:56:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7f007c06afe0 0x7f007c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f009015ad70 0x7f00900b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00900c9ca0 0x7f00900ac890 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 --2- 172.21.15.29:0/4040723476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f009006a4b0 0x7f009014ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 >> 172.21.15.29:0/4040723476 conn(0x7f00900ce9b0 msgr2=0x7f0090069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 shutdown_connections 2023-10-14T16:56:37.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.057+0000 7f00964af640 1 -- 172.21.15.29:0/4040723476 wait complete. 2023-10-14T16:56:37.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:37.077 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T16:56:37.130 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-14T16:56:37.220 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.220+0000 7ff4dfe32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:37.251 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.249+0000 7f5394830040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:37.272 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.272+0000 7ff4dfe32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:37.307 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.305+0000 7f5394830040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:37.322 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.322+0000 7ff4dfe32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.334+0000 7efc0759e640 1 Processor -- start 2023-10-14T16:56:37.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.335+0000 7efc0759e640 1 -- start start 2023-10-14T16:56:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.336+0000 7efc0759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 0x7efc08071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.336+0000 7efc0759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc080d12f0 con 0x7efc08071fa0 2023-10-14T16:56:37.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.336+0000 7efc0759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc080723f0 con 0x7efc0815cc80 2023-10-14T16:56:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.336+0000 7efc0759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080d0500 con 0x7efc080cdf30 2023-10-14T16:56:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc06d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc08071fa0 0x7efc0806aeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43924/0 (socket says 172.21.15.29:43924) 2023-10-14T16:56:37.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc06d9d640 1 -- 172.21.15.29:0/1540232549 learned_addr learned my addr 172.21.15.29:0/1540232549 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:37.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efbfffff640 1 --2- 172.21.15.29:0/1540232549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 0x7efc08071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:37.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255154579 0 0) 0x7efc080d12f0 con 0x7efc08071fa0 2023-10-14T16:56:37.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efbf4003520 con 0x7efc08071fa0 2023-10-14T16:56:37.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754373328 0 0) 0x7efc080723f0 con 0x7efc0815cc80 2023-10-14T16:56:37.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc080d12f0 con 0x7efc0815cc80 2023-10-14T16:56:37.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efbfffff640 1 -- 172.21.15.29:0/1540232549 >> v1:172.21.15.186:6789/0 conn(0x7efc0815cc80 legacy=0x7efc080cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1182243376 0 0) 0x7efbf4003520 con 0x7efc08071fa0 2023-10-14T16:56:37.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efbfffff640 1 -- 172.21.15.29:0/1540232549 >> v1:172.21.15.29:6789/0 conn(0x7efc08071fa0 legacy=0x7efc0806aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.337+0000 7efbfffff640 1 -- 172.21.15.29:0/1540232549 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc080d06e0 con 0x7efc080cdf30 2023-10-14T16:56:37.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.339+0000 7efbfffff640 1 --2- 172.21.15.29:0/1540232549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 0x7efc08071a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efbf00097d0 tx=0x7efbf002b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3df928f1e9fa6ed2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:37.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.340+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc000019f0 con 0x7efc080cdf30 2023-10-14T16:56:37.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.340+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf002e050 con 0x7efc080cdf30 2023-10-14T16:56:37.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.340+0000 7efc05d9b640 1 -- 172.21.15.29:0/1540232549 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0030730 con 0x7efc080cdf30 2023-10-14T16:56:37.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 -- 172.21.15.29:0/1540232549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 msgr2=0x7efc08071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 --2- 172.21.15.29:0/1540232549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 0x7efc08071a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efbf00097d0 tx=0x7efbf002b800 comp rx=0 tx=0).stop 2023-10-14T16:56:37.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 -- 172.21.15.29:0/1540232549 shutdown_connections 2023-10-14T16:56:37.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 --2- 172.21.15.29:0/1540232549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc080cdf30 0x7efc08071a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 -- 172.21.15.29:0/1540232549 >> 172.21.15.29:0/1540232549 conn(0x7efc08065ed0 msgr2=0x7efc080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:37.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 -- 172.21.15.29:0/1540232549 shutdown_connections 2023-10-14T16:56:37.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.341+0000 7efc0759e640 1 -- 172.21.15.29:0/1540232549 wait complete. 2023-10-14T16:56:37.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.344+0000 7efc0759e640 1 Processor -- start 2023-10-14T16:56:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.344+0000 7efc0759e640 1 -- start start 2023-10-14T16:56:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc08071fa0 0x7efc081f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:37.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080cdf30 0x7efc080aea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc0815cc80 0x7efc080b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc0815b760 con 0x7efc080cdf30 2023-10-14T16:56:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc080af670 con 0x7efc0815cc80 2023-10-14T16:56:37.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efc0759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc080ad7a0 con 0x7efc08071fa0 2023-10-14T16:56:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.345+0000 7efbfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080cdf30 0x7efc080aea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080cdf30 0x7efc080aea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39966/0 (socket says 172.21.15.29:39966) 2023-10-14T16:56:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 -- 172.21.15.29:0/4099096114 learned_addr learned my addr 172.21.15.29:0/4099096114 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc08071fa0 0x7efc081f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:37.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efc06d9d640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc0815cc80 0x7efc080b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc08071fa0 msgr2=0x7efc081f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc08071fa0 0x7efc081f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc0815cc80 msgr2=0x7efc080b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc0815cc80 0x7efc080b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:37.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.346+0000 7efbfffff640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efbf0009400 con 0x7efc080cdf30 2023-10-14T16:56:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efbfffff640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc080cdf30 0x7efc080aea10 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7efbf0000c00 tx=0x7efbf00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0004230 con 0x7efc080cdf30 2023-10-14T16:56:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efbf00043b0 con 0x7efc080cdf30 2023-10-14T16:56:37.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efbf0004520 con 0x7efc080cdf30 2023-10-14T16:56:37.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efc0759e640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc080ada80 con 0x7efc080cdf30 2023-10-14T16:56:37.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.347+0000 7efc0759e640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc080ae040 con 0x7efc080cdf30 2023-10-14T16:56:37.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.348+0000 7efc0759e640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc081f23f0 con 0x7efc080cdf30 2023-10-14T16:56:37.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.350+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 30) v1 ==== 89348+0+0 (secure 0 0 0) 0x7efbf0039040 con 0x7efc080cdf30 2023-10-14T16:56:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.351+0000 7efbfeffd640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.351+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 5753+0+0 (secure 0 0 0) 0x7efbf00aae20 con 0x7efc080cdf30 2023-10-14T16:56:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.351+0000 7efc0659c640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.351+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:37.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.356+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc081f23f0 con 0x7efc080cdf30 2023-10-14T16:56:37.363 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.362+0000 7f5394830040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:37.384 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.384+0000 7ff4dfe32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:37.428 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.426+0000 7f5394830040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:37.432 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.433+0000 7ff4dfe32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:37.479 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.478+0000 7f5394830040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:37.542 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.542+0000 7ff4dfe32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:37.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.552+0000 7efc0659c640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:37.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.552+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:37.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.575+0000 7efc0759e640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efc080d0ca0 con 0x7efbe806afe0 2023-10-14T16:56:37.596 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.594+0000 7f5394830040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:37.604 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.604+0000 7ff4dfe32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:37.654 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.654+0000 7ff4dfe32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:37.662 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.660+0000 7f5394830040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:37.712 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.711+0000 7ff4dfe32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:37.714 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.712+0000 7f5394830040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:37.762 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:37.761+0000 7ff4dfe32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:37.773 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.771+0000 7f5394830040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:37.825 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:37.823+0000 7f5394830040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:37.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.953+0000 7efc0659c640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:37.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:37.953+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:56:38.138 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.138+0000 7ff4dfe32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:38.223 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.221+0000 7f5394830040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:38.297 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.297+0000 7ff4dfe32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:38.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.388+0000 7f5394830040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:38.406 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.406+0000 7ff4dfe32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:38.456 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.456+0000 7ff4dfe32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:38.502 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.500+0000 7f5394830040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:38.554 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.553+0000 7f5394830040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:38.562 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.562+0000 7ff4dfe32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:38.666 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.665+0000 7f5394830040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:38.667 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.667+0000 7ff4dfe32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:38.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:38.754+0000 7efc0659c640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:38.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:38.754+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:56:38.779 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.777+0000 7f5394830040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:38.824 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.823+0000 7ff4dfe32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:38.872 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.873+0000 7ff4dfe32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:38.943 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.941+0000 7f5394830040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:38.990 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:38.990+0000 7ff4dfe32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:38.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:38.993+0000 7f5394830040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:39.119 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:39.117+0000 7f5394830040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:39.806 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:39.806+0000 7ff4dfe32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:40.007 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.005+0000 7f5394830040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:40.099 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:40.099+0000 7ff4dfe32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:40.158 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:40.158+0000 7ff4dfe32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:40.272 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:40.272+0000 7ff4dfe32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:40.320 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.319+0000 7f5394830040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:40.356+0000 7efc0659c640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3108857887 2023-10-14T16:56:40.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:40.356+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:56:40.379 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.378+0000 7f5394830040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:40.497 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.496+0000 7f5394830040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:40.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:40.518+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 31) v1 ==== 88957+0+0 (secure 0 0 0) 0x7efbf00b0020 con 0x7efc080cdf30 2023-10-14T16:56:40.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:40.519+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 msgr2=0x7efbe806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:40.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:40.519+0000 7efbfeffd640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/3108857887,v1:172.21.15.29:6833/3108857887] conn(0x7efbe806afe0 0x7efbe806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:40.867 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.865+0000 7f5313c44640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.867 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.865+0000 7f5313c44640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.865+0000 7f5313c44640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.865+0000 7f5313c44640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.865+0000 7f5313c44640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.866+0000 7f5310bfe640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.866+0000 7f5310bfe640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.869 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.866+0000 7f5310bfe640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.869 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.866+0000 7f5310bfe640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:40.869 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:40.866+0000 7f5310bfe640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:41.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.532+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 32) v1 ==== 89084+0+0 (secure 0 0 0) 0x7efbf0076210 con 0x7efc080cdf30 2023-10-14T16:56:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.533+0000 7efbfeffd640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7efbe80358e0 0x7efbe8074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.533+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 --> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7efbe8035cc0 con 0x7efbe80358e0 2023-10-14T16:56:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.540+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7efbe80358e0 0x7efbe8074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.551+0000 7efc0659c640 1 --2- 172.21.15.29:0/4099096114 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7efbe80358e0 0x7efbe8074f30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7efbf80011e0 tx=0x7efbf8009040 comp rx=0 tx=0).ready entity=mgr.6693 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:41.575 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:41.573+0000 7f531ac52640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-14T16:56:41.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.573+0000 7efbfeffd640 1 -- 172.21.15.29:0/4099096114 <== mgr.6693 v2:172.21.15.29:6832/1682700904 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7efbe8035cc0 con 0x7efbe80358e0 2023-10-14T16:56:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-14T16:56:41.598 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T16:56:41.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:41.600 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T16:56:41.663 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T16:56:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.889+0000 7ff45538e640 1 Processor -- start 2023-10-14T16:56:41.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.889+0000 7ff45538e640 1 -- start start 2023-10-14T16:56:41.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.891+0000 7ff45538e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff45006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.891+0000 7ff45538e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45015f4a0 con 0x7ff4500ad970 2023-10-14T16:56:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.891+0000 7ff45538e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff45006c960 con 0x7ff4500cda80 2023-10-14T16:56:41.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.891+0000 7ff45538e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff45015b9d0 con 0x7ff45006c040 2023-10-14T16:56:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff447fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff4500ad970 0x7ff45015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55002/0 (socket says 172.21.15.29:55002) 2023-10-14T16:56:41.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff447fff640 1 -- 172.21.15.29:0/3506626392 learned_addr learned my addr 172.21.15.29:0/3506626392 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff45006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956566488 0 0) 0x7ff45015f4a0 con 0x7ff4500ad970 2023-10-14T16:56:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff43c003520 con 0x7ff4500ad970 2023-10-14T16:56:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2526269103 0 0) 0x7ff43c003520 con 0x7ff4500ad970 2023-10-14T16:56:41.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 msgr2=0x7ff45006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:41.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.892+0000 7ff44f7fe640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff45006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 >> v1:172.21.15.186:6789/0 conn(0x7ff4500cda80 legacy=0x7ff45015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:41.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff45015bcb0 con 0x7ff4500ad970 2023-10-14T16:56:41.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff438001670 con 0x7ff4500ad970 2023-10-14T16:56:41.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 >> v1:172.21.15.29:6789/0 conn(0x7ff4500ad970 legacy=0x7ff45015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff43c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43c0047c0 0x7ff43c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff43c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45015b9d0 con 0x7ff43c0039a0 2023-10-14T16:56:41.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c00ba60 con 0x7ff43c0047c0 2023-10-14T16:56:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff43c00bc20 con 0x7ff45006c040 2023-10-14T16:56:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff43c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff447fff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43c0047c0 0x7ff43c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.893+0000 7ff454b8d640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff43c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff44ffff640 1 -- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 msgr2=0x7ff43c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff43c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff44ffff640 1 -- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43c0047c0 msgr2=0x7ff43c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff43c0047c0 0x7ff43c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff44ffff640 1 -- 172.21.15.29:0/3506626392 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff43c003520 con 0x7ff43c0039a0 2023-10-14T16:56:41.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.894+0000 7ff454b8d640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff45006c040 0x7ff43c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:41.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.895+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff43c003dc0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a90 tx=0x7ff43c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449ac2ff86866953 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:41.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44002c660 con 0x7ff43c0039a0 2023-10-14T16:56:41.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff44002c7c0 con 0x7ff43c0039a0 2023-10-14T16:56:41.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff44f7fe640 1 -- 172.21.15.29:0/3506626392 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff44002cb50 con 0x7ff43c0039a0 2023-10-14T16:56:41.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 -- 172.21.15.29:0/3506626392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 msgr2=0x7ff43c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:41.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 --2- 172.21.15.29:0/3506626392 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff43c003dc0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a90 tx=0x7ff43c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff454b8d640 1 -- 172.21.15.29:0/3506626392 reap_dead start 2023-10-14T16:56:41.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 -- 172.21.15.29:0/3506626392 shutdown_connections 2023-10-14T16:56:41.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 -- 172.21.15.29:0/3506626392 >> 172.21.15.29:0/3506626392 conn(0x7ff4500ce6a0 msgr2=0x7ff4500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:41.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 -- 172.21.15.29:0/3506626392 shutdown_connections 2023-10-14T16:56:41.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.896+0000 7ff45538e640 1 -- 172.21.15.29:0/3506626392 wait complete. 2023-10-14T16:56:41.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.898+0000 7ff45538e640 1 Processor -- start 2023-10-14T16:56:41.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.898+0000 7ff45538e640 1 -- start start 2023-10-14T16:56:41.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4500ae100 0x7ff4500c7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 0x7ff45015a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501f4d50 con 0x7ff43c0039a0 2023-10-14T16:56:41.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff45015af20 con 0x7ff45015a560 2023-10-14T16:56:41.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.899+0000 7ff45538e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4501f40d0 con 0x7ff4500ae100 2023-10-14T16:56:41.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41432/0 (socket says 172.21.15.29:41432) 2023-10-14T16:56:41.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 -- 172.21.15.29:0/3698709007 learned_addr learned my addr 172.21.15.29:0/3698709007 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:41.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff454b8d640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 0x7ff45015a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 -- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4500ae100 msgr2=0x7ff4500c7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:41.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4500ae100 0x7ff4500c7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 -- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 msgr2=0x7ff45015a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 0x7ff45015a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff44ffff640 1 -- 172.21.15.29:0/3698709007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4501f43b0 con 0x7ff43c0039a0 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.900+0000 7ff454b8d640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 0x7ff45015a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff44ffff640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a90 tx=0x7ff440004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440003810 con 0x7ff43c0039a0 2023-10-14T16:56:41.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff440003970 con 0x7ff43c0039a0 2023-10-14T16:56:41.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff440003d00 con 0x7ff43c0039a0 2023-10-14T16:56:41.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4501f4690 con 0x7ff43c0039a0 2023-10-14T16:56:41.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.901+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff45014fe60 con 0x7ff43c0039a0 2023-10-14T16:56:41.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.902+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff45006b330 con 0x7ff43c0039a0 2023-10-14T16:56:41.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.903+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 32) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff440040450 con 0x7ff43c0039a0 2023-10-14T16:56:41.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.904+0000 7ff44d7fa640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 0x7ff43006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:41.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.905+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7ff4400ab5f0 con 0x7ff43c0039a0 2023-10-14T16:56:41.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.905+0000 7ff447fff640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 0x7ff43006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:41.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.906+0000 7ff447fff640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 0x7ff43006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff438001050 tx=0x7ff438000f40 comp rx=0 tx=0).ready entity=mgr.6693 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:41.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:41.910+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff440076970 con 0x7ff43c0039a0 2023-10-14T16:56:42.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.160+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7ff45004e9b0 con 0x7ff43c0039a0 2023-10-14T16:56:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.525+0000 7ff44d7fa640 1 -- 172.21.15.29:0/3698709007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v33) v1 ==== 93+0+0 (secure 0 0 0) 0x7ff4400044e0 con 0x7ff43c0039a0 2023-10-14T16:56:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 msgr2=0x7ff43006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 0x7ff43006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff438001050 tx=0x7ff438000f40 comp rx=0 tx=0).stop 2023-10-14T16:56:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 msgr2=0x7ff45014f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7ff440009a90 tx=0x7ff440004730 comp rx=0 tx=0).stop 2023-10-14T16:56:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 shutdown_connections 2023-10-14T16:56:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff45015a560 0x7ff45015a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff43006ad10 0x7ff43006d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff4500ae100 0x7ff4500c7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 --2- 172.21.15.29:0/3698709007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff43c0039a0 0x7ff45014f920 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.529+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 >> 172.21.15.29:0/3698709007 conn(0x7ff4500ce6a0 msgr2=0x7ff45015c480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.530+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 shutdown_connections 2023-10-14T16:56:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:42.530+0000 7ff45538e640 1 -- 172.21.15.29:0/3698709007 wait complete. 2023-10-14T16:56:42.557 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:56:42.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:42.694 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:42.792 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:56:42.792 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:42.793 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:42.850 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:42.851 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:42.901 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:43.068 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.067+0000 7fbc4882b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:43.086 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.084+0000 7f8f6fe0d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:56:43.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.108+0000 7f83ee059640 1 Processor -- start 2023-10-14T16:56:43.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.109+0000 7f83ee059640 1 -- start start 2023-10-14T16:56:43.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ee059640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ee059640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f83e80b8480 0x7f83e80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ee059640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83e815d150 con 0x7f83e80d0be0 2023-10-14T16:56:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ee059640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83e80d1180 con 0x7f83e80b8480 2023-10-14T16:56:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ee059640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80d1460 con 0x7f83e815cd70 2023-10-14T16:56:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ed858640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f83e80d0be0 0x7f83e80b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55014/0 (socket says 172.21.15.29:55014) 2023-10-14T16:56:43.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.110+0000 7f83ed858640 1 -- 172.21.15.29:0/3088796302 learned_addr learned my addr 172.21.15.29:0/3088796302 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ed057640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ec856640 1 --2- 172.21.15.29:0/3088796302 >> v2:172.21.15.186:3300/0 conn(0x7f83e80b8480 0x7f83e80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83dffff640 1 -- 172.21.15.29:0/3088796302 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310795286 0 0) 0x7f83e815d150 con 0x7f83e80d0be0 2023-10-14T16:56:43.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83dffff640 1 -- 172.21.15.29:0/3088796302 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83d4003520 con 0x7f83e80d0be0 2023-10-14T16:56:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ec856640 1 -- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 msgr2=0x7f83e80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ec856640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e80d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ec856640 1 -- 172.21.15.29:0/3088796302 >> v1:172.21.15.29:6789/0 conn(0x7f83e80d0be0 legacy=0x7f83e80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.111+0000 7f83ec856640 1 -- 172.21.15.29:0/3088796302 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83e80d1740 con 0x7f83e80b8480 2023-10-14T16:56:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.112+0000 7f83ec856640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80cd7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097a0 tx=0x7f83d802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=659bbba090090e5d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.113+0000 7f83dffff640 1 -- 172.21.15.29:0/3088796302 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d802d040 con 0x7f83e80b8480 2023-10-14T16:56:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.113+0000 7f83dffff640 1 -- 172.21.15.29:0/3088796302 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d8031810 con 0x7f83e80b8480 2023-10-14T16:56:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.113+0000 7f83dffff640 1 -- 172.21.15.29:0/3088796302 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d8031ba0 con 0x7f83e80b8480 2023-10-14T16:56:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.113+0000 7f83ee059640 1 -- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 msgr2=0x7f83e80cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.113+0000 7f83ee059640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80cd7c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f83d80097a0 tx=0x7f83d802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 -- 172.21.15.29:0/3088796302 shutdown_connections 2023-10-14T16:56:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80cd7c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 --2- 172.21.15.29:0/3088796302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e80d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 -- 172.21.15.29:0/3088796302 >> 172.21.15.29:0/3088796302 conn(0x7f83e80cf570 msgr2=0x7f83e80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 -- 172.21.15.29:0/3088796302 shutdown_connections 2023-10-14T16:56:43.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.114+0000 7f83ee059640 1 -- 172.21.15.29:0/3088796302 wait complete. 2023-10-14T16:56:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.115+0000 7f83ee059640 1 Processor -- start 2023-10-14T16:56:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.116+0000 7f83ee059640 1 -- start start 2023-10-14T16:56:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.116+0000 7f83ee059640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.116+0000 7f83ee059640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ee059640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e806bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ee059640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e81591c0 con 0x7f83e80d0be0 2023-10-14T16:56:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ee059640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e80c9bc0 con 0x7f83e80b8480 2023-10-14T16:56:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ee059640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8158a90 con 0x7f83e815cd70 2023-10-14T16:56:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41452/0 (socket says 172.21.15.29:41452) 2023-10-14T16:56:43.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 -- 172.21.15.29:0/2029620874 learned_addr learned my addr 172.21.15.29:0/2029620874 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:43.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ed057640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ed858640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e806bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 msgr2=0x7f83e806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e806bbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 msgr2=0x7f83e80c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ec856640 1 -- 172.21.15.29:0/2029620874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8009400 con 0x7f83e80d0be0 2023-10-14T16:56:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ed858640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e806bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:43.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.117+0000 7f83ed057640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ec856640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000c00 tx=0x7f83d80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d803ad60 con 0x7f83e80d0be0 2023-10-14T16:56:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83d80443f0 con 0x7f83e80d0be0 2023-10-14T16:56:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e80c9e80 con 0x7f83e80d0be0 2023-10-14T16:56:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83d80445a0 con 0x7f83e80d0be0 2023-10-14T16:56:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.118+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e80ca440 con 0x7f83e80d0be0 2023-10-14T16:56:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.119+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e80ca630 con 0x7f83e80d0be0 2023-10-14T16:56:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.120+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f83d804d450 con 0x7f83e80d0be0 2023-10-14T16:56:43.128 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.122+0000 7fbc4882b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.121+0000 7f83ddffb640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 0x7f83cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.121+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f83d80abad0 con 0x7f83e80d0be0 2023-10-14T16:56:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.121+0000 7f83ed057640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 msgr2=0x7f83cc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.121+0000 7f83ed057640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 0x7f83cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.125+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83d80767e0 con 0x7f83e80d0be0 2023-10-14T16:56:43.145 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.143+0000 7f8f6fe0d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:56:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.320+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83e804e9b0 con 0x7f83e80d0be0 2023-10-14T16:56:43.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.322+0000 7f83ed057640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 msgr2=0x7f83cc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:43.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.322+0000 7f83ed057640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 0x7f83cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:43.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.323+0000 7f83ddffb640 1 -- 172.21.15.29:0/2029620874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17121 (secure 0 0 0) 0x7f83d8079e00 con 0x7f83e80d0be0 2023-10-14T16:56:43.325 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:43.325 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:40.500779+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/626362321":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/664522093":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/126969037":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6833/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6832/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/1855767981":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/3741726235":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/2072219911":"2023-10-15T16:56:40.500646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 msgr2=0x7f83cc06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 0x7f83cc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 msgr2=0x7f83e8153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f83d8000c00 tx=0x7f83d80046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 shutdown_connections 2023-10-14T16:56:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f83cc06afc0 0x7f83cc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83e815cd70 0x7f83e806bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83e80d0be0 0x7f83e8153340 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 --2- 172.21.15.29:0/2029620874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83e80b8480 0x7f83e80c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.325+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 >> 172.21.15.29:0/2029620874 conn(0x7f83e80cf570 msgr2=0x7f83e80d2220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.326+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 shutdown_connections 2023-10-14T16:56:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.326+0000 7f83ee059640 1 -- 172.21.15.29:0/2029620874 wait complete. 2023-10-14T16:56:43.345 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:56:24.067424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:56:25.117255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:43.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.556+0000 7ff2a8a1f640 1 Processor -- start 2023-10-14T16:56:43.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.556+0000 7ff2a8a1f640 1 -- start start 2023-10-14T16:56:43.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.558+0000 7ff2a8a1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a40cf710 0x7ff2a40b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.558+0000 7ff2a8a1f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2a40b6cd0 0x7ff2a40b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.558+0000 7ff2a8a1f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2a40d3980 con 0x7ff2a40cfaf0 2023-10-14T16:56:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.558+0000 7ff2a8a1f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40d3c60 con 0x7ff2a40cf710 2023-10-14T16:56:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.558+0000 7ff2a8a1f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2a40ce630 con 0x7ff2a40b6cd0 2023-10-14T16:56:43.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a40cf710 0x7ff2a40b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2a40cf710 0x7ff2a40b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41454/0 (socket says 172.21.15.29:41454) 2023-10-14T16:56:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 -- 172.21.15.29:0/1886908697 learned_addr learned my addr 172.21.15.29:0/1886908697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:43.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a2d9d640 1 --2- 172.21.15.29:0/1886908697 >> v2:172.21.15.186:3300/0 conn(0x7ff2a40b6cd0 0x7ff2a40b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 -- 172.21.15.29:0/1886908697 >> v2:172.21.15.186:3300/0 conn(0x7ff2a40b6cd0 msgr2=0x7ff2a40b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:43.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 --2- 172.21.15.29:0/1886908697 >> v2:172.21.15.186:3300/0 conn(0x7ff2a40b6cd0 0x7ff2a40b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 -- 172.21.15.29:0/1886908697 >> v1:172.21.15.186:6789/0 conn(0x7ff2a40cfaf0 legacy=0x7ff2a40d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.559+0000 7ff2a259c640 1 -- 172.21.15.29:0/1886908697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a40ce910 con 0x7ff2a40cf710 2023-10-14T16:56:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.560+0000 7ff2a259c640 1 --2- 172.21.15.29:0/1886908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a40b2d10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40b5890 tx=0x7ff29402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df95118be9152c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a159a640 1 -- 172.21.15.29:0/1886908697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294004230 con 0x7ff2a40cf710 2023-10-14T16:56:43.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a159a640 1 -- 172.21.15.29:0/1886908697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff294004390 con 0x7ff2a40cf710 2023-10-14T16:56:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a159a640 1 -- 172.21.15.29:0/1886908697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff294003810 con 0x7ff2a40cf710 2023-10-14T16:56:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/1886908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 msgr2=0x7ff2a40b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/1886908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a40b2d10 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7ff2a40b5890 tx=0x7ff29402b860 comp rx=0 tx=0).stop 2023-10-14T16:56:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/1886908697 shutdown_connections 2023-10-14T16:56:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/1886908697 >> v2:172.21.15.186:3300/0 conn(0x7ff2a40b6cd0 0x7ff2a40b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/1886908697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a40b2d10 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/1886908697 >> 172.21.15.29:0/1886908697 conn(0x7ff2a40d2cc0 msgr2=0x7ff2a40d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/1886908697 shutdown_connections 2023-10-14T16:56:43.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.561+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/1886908697 wait complete. 2023-10-14T16:56:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.563+0000 7ff2a8a1f640 1 Processor -- start 2023-10-14T16:56:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.563+0000 7ff2a8a1f640 1 -- start start 2023-10-14T16:56:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.564+0000 7ff2a8a1f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40b6cd0 0x7ff2a415ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.564+0000 7ff2a8a1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.564+0000 7ff2a8a1f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a40cfaf0 0x7ff2a40cbda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.564+0000 7ff2a8a1f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a409fb50 con 0x7ff2a40cf710 2023-10-14T16:56:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a8a1f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a409fdf0 con 0x7ff2a40cfaf0 2023-10-14T16:56:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a8a1f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2a40a00d0 con 0x7ff2a40b6cd0 2023-10-14T16:56:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41464/0 (socket says 172.21.15.29:41464) 2023-10-14T16:56:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 -- 172.21.15.29:0/3775657044 learned_addr learned my addr 172.21.15.29:0/3775657044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a2d9d640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a40cfaf0 0x7ff2a40cbda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a259c640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40b6cd0 0x7ff2a415ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 -- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40b6cd0 msgr2=0x7ff2a415ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40b6cd0 0x7ff2a415ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 -- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a40cfaf0 msgr2=0x7ff2a40cbda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a40cfaf0 0x7ff2a40cbda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.565+0000 7ff2a1d9b640 1 -- 172.21.15.29:0/3775657044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff294009460 con 0x7ff2a40cf710 2023-10-14T16:56:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff2a1d9b640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c430 tx=0x7ff29800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298011790 con 0x7ff2a40cf710 2023-10-14T16:56:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2980118f0 con 0x7ff2a40cf710 2023-10-14T16:56:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff298011ce0 con 0x7ff2a40cf710 2023-10-14T16:56:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2a40a03b0 con 0x7ff2a40cf710 2023-10-14T16:56:43.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.566+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2a415b0a0 con 0x7ff2a40cf710 2023-10-14T16:56:43.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.567+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2a404e160 con 0x7ff2a40cf710 2023-10-14T16:56:43.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.568+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff29801c460 con 0x7ff2a40cf710 2023-10-14T16:56:43.575 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.571+0000 7fbc4882b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:43.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.570+0000 7ff292ffd640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 0x7ff28406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:43.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.570+0000 7ff2a259c640 1 -- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 msgr2=0x7ff28406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:43.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.570+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7ff2980898c0 con 0x7ff2a40cf710 2023-10-14T16:56:43.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.570+0000 7ff2a259c640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 0x7ff28406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:43.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.573+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2980546d0 con 0x7ff2a40cf710 2023-10-14T16:56:43.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.621+0000 7f8f6fe0d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:56:43.624 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.623+0000 7fbc4882b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:43.674 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.673+0000 7fbc4882b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:43.677 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.676+0000 7f8f6fe0d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:56:43.730 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.728+0000 7f8f6fe0d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:56:43.735 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.734+0000 7fbc4882b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:43.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.765+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7ff2a40d1080 con 0x7ff2a40cf710 2023-10-14T16:56:43.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.766+0000 7ff292ffd640 1 -- 172.21.15.29:0/3775657044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7ff298057cf0 con 0x7ff2a40cf710 2023-10-14T16:56:43.768 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:43.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.768+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 msgr2=0x7ff28406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:43.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.768+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 0x7ff28406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.768+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 msgr2=0x7ff2a409d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:43.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.768+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7ff29800c430 tx=0x7ff29800c8f0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 shutdown_connections 2023-10-14T16:56:43.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2a40cfaf0 0x7ff2a40cbda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2a40cf710 0x7ff2a409d220 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7ff28406af90 0x7ff28406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 --2- 172.21.15.29:0/3775657044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2a40b6cd0 0x7ff2a415ab60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:43.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 >> 172.21.15.29:0/3775657044 conn(0x7ff2a40d2cc0 msgr2=0x7ff2a40cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:43.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 shutdown_connections 2023-10-14T16:56:43.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:43.769+0000 7ff2a8a1f640 1 -- 172.21.15.29:0/3775657044 wait complete. 2023-10-14T16:56:43.784 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.783+0000 7fbc4882b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:43.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:56:43.795 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.794+0000 7f8f6fe0d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:56:43.848 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.846+0000 7f8f6fe0d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:56:43.894 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.893+0000 7fbc4882b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:43.957 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:43.956+0000 7fbc4882b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:43.966 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:43.965+0000 7f8f6fe0d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:56:44.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.000+0000 7f7c01fd5640 1 Processor -- start 2023-10-14T16:56:44.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.000+0000 7f7c01fd5640 1 -- start start 2023-10-14T16:56:44.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7c01fd5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7bfc06cd70 0x7f7bfc070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7c01fd5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7bfc0ccf80 0x7f7bfc0ae9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7c01fd5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bfc06a770 con 0x7f7bfc15ab70 2023-10-14T16:56:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7c01fd5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0c6930 con 0x7f7bfc06cd70 2023-10-14T16:56:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7c01fd5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bfc0af0a0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7bfc06cd70 0x7f7bfc070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7bfaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7bfc06cd70 0x7f7bfc070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41468/0 (socket says 172.21.15.29:41468) 2023-10-14T16:56:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.002+0000 7f7bfaffd640 1 -- 172.21.15.29:0/114823486 learned_addr learned my addr 172.21.15.29:0/114823486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.003+0000 7f7bfbfff640 1 --2- 172.21.15.29:0/114823486 >> v2:172.21.15.186:3300/0 conn(0x7f7bfc0ccf80 0x7f7bfc0ae9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.003+0000 7f7bfaffd640 1 -- 172.21.15.29:0/114823486 >> v2:172.21.15.186:3300/0 conn(0x7f7bfc0ccf80 msgr2=0x7f7bfc0ae9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:44.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.003+0000 7f7bfaffd640 1 --2- 172.21.15.29:0/114823486 >> v2:172.21.15.186:3300/0 conn(0x7f7bfc0ccf80 0x7f7bfc0ae9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.003+0000 7f7bfaffd640 1 -- 172.21.15.29:0/114823486 >> v1:172.21.15.186:6789/0 conn(0x7f7bfc15ab70 legacy=0x7f7bfc06c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.003+0000 7f7bfaffd640 1 -- 172.21.15.29:0/114823486 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bfc069e40 con 0x7f7bfc06cd70 2023-10-14T16:56:44.008 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.005+0000 7fbc4882b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.004+0000 7f7bfaffd640 1 --2- 172.21.15.29:0/114823486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc06cd70 0x7f7bfc070660 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0ae730 tx=0x7f7bf002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a030db874e9b9f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.004+0000 7f7bfa7fc640 1 -- 172.21.15.29:0/114823486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0030860 con 0x7f7bfc06cd70 2023-10-14T16:56:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.004+0000 7f7bfa7fc640 1 -- 172.21.15.29:0/114823486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf00309c0 con 0x7f7bfc06cd70 2023-10-14T16:56:44.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.004+0000 7f7bfa7fc640 1 -- 172.21.15.29:0/114823486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0030d50 con 0x7f7bfc06cd70 2023-10-14T16:56:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 -- 172.21.15.29:0/114823486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc06cd70 msgr2=0x7f7bfc070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/114823486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc06cd70 0x7f7bfc070660 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f7bfc0ae730 tx=0x7f7bf002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 -- 172.21.15.29:0/114823486 shutdown_connections 2023-10-14T16:56:44.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/114823486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc06cd70 0x7f7bfc070660 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/114823486 >> v2:172.21.15.186:3300/0 conn(0x7f7bfc0ccf80 0x7f7bfc0ae9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 -- 172.21.15.29:0/114823486 >> 172.21.15.29:0/114823486 conn(0x7f7bfc0cc460 msgr2=0x7f7bfc0cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 -- 172.21.15.29:0/114823486 shutdown_connections 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.005+0000 7f7c01fd5640 1 -- 172.21.15.29:0/114823486 wait complete. 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.007+0000 7f7c01fd5640 1 Processor -- start 2023-10-14T16:56:44.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.007+0000 7f7c01fd5640 1 -- start start 2023-10-14T16:56:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc06cd70 0x7f7bfc155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 0x7f7bfc15b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc15bf10 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc1564b0 con 0x7f7bfc15ab70 2023-10-14T16:56:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7c01fd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bfc156320 con 0x7f7bfc06cd70 2023-10-14T16:56:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7bfaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7bfaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41474/0 (socket says 172.21.15.29:41474) 2023-10-14T16:56:44.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.008+0000 7f7bfaffd640 1 -- 172.21.15.29:0/4066233781 learned_addr learned my addr 172.21.15.29:0/4066233781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfbfff640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 0x7f7bfc15b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfb7fe640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc06cd70 0x7f7bfc155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc06cd70 msgr2=0x7f7bfc155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc06cd70 0x7f7bfc155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 msgr2=0x7f7bfc15b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 0x7f7bfc15b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 -- 172.21.15.29:0/4066233781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf00094d0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfbfff640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 0x7f7bfc15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7bfaffd640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002b9e0 tx=0x7f7bf0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0039b60 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.009+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bf0039ce0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.010+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bfc1566c0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.010+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7bf0043650 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.010+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bfc156c80 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.011+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bfc06dc40 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.012+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7bf004c3b0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.013+0000 7f7be3fff640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 0x7f7be806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.014+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f7bf00abf50 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.014+0000 7f7bfb7fe640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 msgr2=0x7f7be806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:44.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.014+0000 7f7bfb7fe640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 0x7f7be806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:44.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.018+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7bf0076ce0 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.035 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.033+0000 7f8f6fe0d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:44.064 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.063+0000 7fbc4882b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:44.087 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.086+0000 7f8f6fe0d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:56:44.114 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.113+0000 7fbc4882b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:44.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.145+0000 7f8f6fe0d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:56:44.199 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.198+0000 7f8f6fe0d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:56:44.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.212+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7bfc04e160 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.213+0000 7f7be3fff640 1 -- 172.21.15.29:0/4066233781 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v83) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7bf0048020 con 0x7f7bfc0ccf80 2023-10-14T16:56:44.215 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:56:44.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.215+0000 7f7bfb7fe640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 msgr2=0x7f7be806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:44.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.215+0000 7f7bfb7fe640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 0x7f7be806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:44.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 msgr2=0x7f7be806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:44.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 0x7f7be806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 msgr2=0x7f7bfc15b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f7bf002b9e0 tx=0x7f7bf0004730 comp rx=0 tx=0).stop 2023-10-14T16:56:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 shutdown_connections 2023-10-14T16:56:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7bfc15ab70 0x7f7bfc15b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7bfc0ccf80 0x7f7bfc15b3a0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f7be806af90 0x7f7be806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 --2- 172.21.15.29:0/4066233781 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7bfc06cd70 0x7f7bfc155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 >> 172.21.15.29:0/4066233781 conn(0x7f7bfc0cc460 msgr2=0x7f7bfc070660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 shutdown_connections 2023-10-14T16:56:44.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.216+0000 7f7c01fd5640 1 -- 172.21.15.29:0/4066233781 wait complete. 2023-10-14T16:56:44.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:56:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.491+0000 7f0034e02640 1 Processor -- start 2023-10-14T16:56:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.492+0000 7f0034e02640 1 -- start start 2023-10-14T16:56:44.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0034e02640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00300ac890 0x7f00300c9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0034e02640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00300c9d40 0x7f0030160a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0034e02640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f003006ad30 con 0x7f00300cb100 2023-10-14T16:56:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0034e02640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f003006a760 con 0x7f00300ac890 2023-10-14T16:56:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0034e02640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00300ae400 con 0x7f00300c9d40 2023-10-14T16:56:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.493+0000 7f0027fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00300ac890 0x7f00300c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00300ac890 0x7f00300c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41480/0 (socket says 172.21.15.29:41480) 2023-10-14T16:56:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 -- 172.21.15.29:0/2092387426 learned_addr learned my addr 172.21.15.29:0/2092387426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f002ed9d640 1 --2- 172.21.15.29:0/2092387426 >> v2:172.21.15.186:3300/0 conn(0x7f00300c9d40 0x7f0030160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 -- 172.21.15.29:0/2092387426 >> v2:172.21.15.186:3300/0 conn(0x7f00300c9d40 msgr2=0x7f0030160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 --2- 172.21.15.29:0/2092387426 >> v2:172.21.15.186:3300/0 conn(0x7f00300c9d40 0x7f0030160a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 -- 172.21.15.29:0/2092387426 >> v1:172.21.15.186:6789/0 conn(0x7f00300cb100 legacy=0x7f003006b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.494+0000 7f0027fff640 1 -- 172.21.15.29:0/2092387426 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00300ae6e0 con 0x7f00300ac890 2023-10-14T16:56:44.498 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.496+0000 7fbc4882b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.495+0000 7f0027fff640 1 --2- 172.21.15.29:0/2092387426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300ac890 0x7f00300c9800 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f003006e8b0 tx=0x7f001802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25853ad4859e68b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.495+0000 7f002dd9b640 1 -- 172.21.15.29:0/2092387426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018031870 con 0x7f00300ac890 2023-10-14T16:56:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.495+0000 7f002dd9b640 1 -- 172.21.15.29:0/2092387426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00180319d0 con 0x7f00300ac890 2023-10-14T16:56:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.495+0000 7f002dd9b640 1 -- 172.21.15.29:0/2092387426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018031d60 con 0x7f00300ac890 2023-10-14T16:56:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 -- 172.21.15.29:0/2092387426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300ac890 msgr2=0x7f00300c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 --2- 172.21.15.29:0/2092387426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300ac890 0x7f00300c9800 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f003006e8b0 tx=0x7f001802b910 comp rx=0 tx=0).stop 2023-10-14T16:56:44.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 -- 172.21.15.29:0/2092387426 shutdown_connections 2023-10-14T16:56:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 --2- 172.21.15.29:0/2092387426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300ac890 0x7f00300c9800 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 --2- 172.21.15.29:0/2092387426 >> v2:172.21.15.186:3300/0 conn(0x7f00300c9d40 0x7f0030160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.496+0000 7f0034e02640 1 -- 172.21.15.29:0/2092387426 >> 172.21.15.29:0/2092387426 conn(0x7f003006bc10 msgr2=0x7f003006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:44.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.497+0000 7f0034e02640 1 -- 172.21.15.29:0/2092387426 shutdown_connections 2023-10-14T16:56:44.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.497+0000 7f0034e02640 1 -- 172.21.15.29:0/2092387426 wait complete. 2023-10-14T16:56:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.499+0000 7f0034e02640 1 Processor -- start 2023-10-14T16:56:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.499+0000 7f0034e02640 1 -- start start 2023-10-14T16:56:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 0x7f003014c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 0x7f003015fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f003014c940 con 0x7f00300c9d40 2023-10-14T16:56:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00300c5c90 con 0x7f00300ac890 2023-10-14T16:56:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.500+0000 7f0034e02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f003015e050 con 0x7f00300cb100 2023-10-14T16:56:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41496/0 (socket says 172.21.15.29:41496) 2023-10-14T16:56:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 -- 172.21.15.29:0/1435282731 learned_addr learned my addr 172.21.15.29:0/1435282731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f002e59c640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 0x7f003014c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 msgr2=0x7f003015fc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f002ed9d640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 0x7f003015fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 0x7f003015fc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 msgr2=0x7f003014c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 0x7f003014c400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f0027fff640 1 -- 172.21.15.29:0/1435282731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00180094d0 con 0x7f00300c9d40 2023-10-14T16:56:44.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f002e59c640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 0x7f003014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.501+0000 7f002ed9d640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 0x7f003015fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0027fff640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0018000c00 tx=0x7f00180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0018031840 con 0x7f00300c9d40 2023-10-14T16:56:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00180443f0 con 0x7f00300c9d40 2023-10-14T16:56:44.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00180447a0 con 0x7f00300c9d40 2023-10-14T16:56:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f003015d7b0 con 0x7f00300c9d40 2023-10-14T16:56:44.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.502+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f003015dd70 con 0x7f00300c9d40 2023-10-14T16:56:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.503+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f003004e160 con 0x7f00300c9d40 2023-10-14T16:56:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.504+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f001804d450 con 0x7f00300c9d40 2023-10-14T16:56:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.505+0000 7f0026ffd640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 0x7f001006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.505+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f001802d040 con 0x7f00300c9d40 2023-10-14T16:56:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.505+0000 7f002e59c640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 msgr2=0x7f001006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.505+0000 7f002e59c640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 0x7f001006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:44.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.509+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0018076f30 con 0x7f00300c9d40 2023-10-14T16:56:44.611 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.610+0000 7f8f6fe0d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:56:44.658 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.657+0000 7fbc4882b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:44.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.703+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f00300ad120 con 0x7f00300c9d40 2023-10-14T16:56:44.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.704+0000 7f0026ffd640 1 -- 172.21.15.29:0/1435282731 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v83) v1 ==== 100+0+11 (secure 0 0 0) 0x7f001807a550 con 0x7f00300c9d40 2023-10-14T16:56:44.706 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:56:44.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f002e59c640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 msgr2=0x7f001006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:44.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f002e59c640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 0x7f001006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:44.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 msgr2=0x7f001006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:44.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 0x7f001006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 msgr2=0x7f003015f750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.706+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f0018000c00 tx=0x7f00180046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 shutdown_connections 2023-10-14T16:56:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f001006af90 0x7f001006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00300cb100 0x7f003015fc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00300c9d40 0x7f003015f750 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 --2- 172.21.15.29:0/1435282731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00300ac890 0x7f003014c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 >> 172.21.15.29:0/1435282731 conn(0x7f003006bc10 msgr2=0x7f00300cd920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 shutdown_connections 2023-10-14T16:56:44.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.707+0000 7f0034e02640 1 -- 172.21.15.29:0/1435282731 wait complete. 2023-10-14T16:56:44.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:56:44.765 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.764+0000 7fbc4882b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:44.785 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.783+0000 7f8f6fe0d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:56:44.815 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.814+0000 7fbc4882b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:44.900 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.899+0000 7f8f6fe0d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:56:44.922 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:44.921+0000 7fbc4882b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:44.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.936+0000 7f649eabb640 1 Processor -- start 2023-10-14T16:56:44.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.936+0000 7f649eabb640 1 -- start start 2023-10-14T16:56:44.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.938+0000 7f649eabb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f649815af80 0x7f64980cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.938+0000 7f649eabb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.938+0000 7f649eabb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649815eff0 con 0x7f649806faf0 2023-10-14T16:56:44.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.938+0000 7f649eabb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6498066480 con 0x7f649815af80 2023-10-14T16:56:44.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.938+0000 7f649eabb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64980d0930 con 0x7f6498065ab0 2023-10-14T16:56:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.939+0000 7f649c857640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f649815af80 0x7f64980cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.939+0000 7f649d058640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.939+0000 7f649d058640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:39232/0 (socket says 172.21.15.29:39232) 2023-10-14T16:56:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.939+0000 7f649d058640 1 -- 172.21.15.29:0/1883797332 learned_addr learned my addr 172.21.15.29:0/1883797332 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.940+0000 7f649d058640 1 -- 172.21.15.29:0/1883797332 >> v2:172.21.15.186:3300/0 conn(0x7f649815af80 msgr2=0x7f64980cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.940+0000 7f649d058640 1 --2- 172.21.15.29:0/1883797332 >> v2:172.21.15.186:3300/0 conn(0x7f649815af80 0x7f64980cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.940+0000 7f649d058640 1 -- 172.21.15.29:0/1883797332 >> v1:172.21.15.186:6789/0 conn(0x7f649806faf0 legacy=0x7f649815f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.940+0000 7f649d058640 1 -- 172.21.15.29:0/1883797332 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f649815ecc0 con 0x7f6498065ab0 2023-10-14T16:56:44.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.941+0000 7f649d058640 1 --2- 172.21.15.29:0/1883797332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6490009e50 tx=0x7f649002bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f88b994347cba918 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.942+0000 7f64977fe640 1 -- 172.21.15.29:0/1883797332 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649002ca70 con 0x7f6498065ab0 2023-10-14T16:56:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.942+0000 7f64977fe640 1 -- 172.21.15.29:0/1883797332 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649002cbd0 con 0x7f6498065ab0 2023-10-14T16:56:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.942+0000 7f64977fe640 1 -- 172.21.15.29:0/1883797332 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64900343f0 con 0x7f6498065ab0 2023-10-14T16:56:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 -- 172.21.15.29:0/1883797332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 msgr2=0x7f6498065f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 --2- 172.21.15.29:0/1883797332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6490009e50 tx=0x7f649002bed0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 -- 172.21.15.29:0/1883797332 shutdown_connections 2023-10-14T16:56:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 --2- 172.21.15.29:0/1883797332 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f6498065f40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 --2- 172.21.15.29:0/1883797332 >> v2:172.21.15.186:3300/0 conn(0x7f649815af80 0x7f64980cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 -- 172.21.15.29:0/1883797332 >> 172.21.15.29:0/1883797332 conn(0x7f64980d0ab0 msgr2=0x7f64980d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 -- 172.21.15.29:0/1883797332 shutdown_connections 2023-10-14T16:56:44.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.943+0000 7f649eabb640 1 -- 172.21.15.29:0/1883797332 wait complete. 2023-10-14T16:56:44.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.945+0000 7f649eabb640 1 Processor -- start 2023-10-14T16:56:44.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.945+0000 7f649eabb640 1 -- start start 2023-10-14T16:56:44.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f64981f2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649806faf0 0x7f64980c7070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981f61c0 con 0x7f649815af80 2023-10-14T16:56:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64981f3120 con 0x7f649806faf0 2023-10-14T16:56:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649eabb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64981f3400 con 0x7f6498065ab0 2023-10-14T16:56:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41502/0 (socket says 172.21.15.29:41502) 2023-10-14T16:56:44.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 -- 172.21.15.29:0/3975396560 learned_addr learned my addr 172.21.15.29:0/3975396560 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:44.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649c857640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f64981f2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f6497fff640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649806faf0 0x7f64980c7070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:44.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 msgr2=0x7f64981f2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f64981f2b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649806faf0 msgr2=0x7f64980c7070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649806faf0 0x7f64980c7070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.946+0000 7f649d058640 1 -- 172.21.15.29:0/3975396560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6490009b30 con 0x7f649815af80 2023-10-14T16:56:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f649d058640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6490036040 tx=0x7f6490004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:44.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6490009510 con 0x7f649815af80 2023-10-14T16:56:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f649003e450 con 0x7f649815af80 2023-10-14T16:56:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f649003e800 con 0x7f649815af80 2023-10-14T16:56:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64981f36e0 con 0x7f649815af80 2023-10-14T16:56:44.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.947+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64980c59e0 con 0x7f649815af80 2023-10-14T16:56:44.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.948+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64981f6330 con 0x7f649815af80 2023-10-14T16:56:44.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.949+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f649003e960 con 0x7f649815af80 2023-10-14T16:56:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.951+0000 7f64957fa640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 0x7f647c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.951+0000 7f649c857640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 msgr2=0x7f647c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.951+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f64900ac680 con 0x7f649815af80 2023-10-14T16:56:44.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.951+0000 7f649c857640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 0x7f647c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:44.958 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:44.955+0000 7f8f6fe0d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:56:44.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:44.955+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f64981f6330 con 0x7f649815af80 2023-10-14T16:56:45.030 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:45.029+0000 7fbc4882b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:45.070 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:45.070+0000 7f8f6fe0d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.152+0000 7f649c857640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 msgr2=0x7f647c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.152+0000 7f649c857640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 0x7f647c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.164+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f64981f6300 con 0x7f649815af80 2023-10-14T16:56:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.165+0000 7f64957fa640 1 -- 172.21.15.29:0/3975396560 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v83) v1 ==== 80+0+467 (secure 0 0 0) 0x7f6490077490 con 0x7f649815af80 2023-10-14T16:56:45.165 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/626362321 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.166 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/664522093 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.166 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/126969037 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.166 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/3108857887 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.166 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/3108857887 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.166 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1855767981 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.167 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3741726235 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.167 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2072219911 2023-10-15T16:56:40.500646+0000 2023-10-14T16:56:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T16:56:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 msgr2=0x7f647c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 0x7f647c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 msgr2=0x7f64981580c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f6490036040 tx=0x7f6490004a40 comp rx=0 tx=0).stop 2023-10-14T16:56:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 shutdown_connections 2023-10-14T16:56:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f649815af80 0x7f64981580c0 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f649806faf0 0x7f64980c7070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f647c06af90 0x7f647c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 --2- 172.21.15.29:0/3975396560 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6498065ab0 0x7f64981f2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 >> 172.21.15.29:0/3975396560 conn(0x7f64980d0ab0 msgr2=0x7f64980d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 shutdown_connections 2023-10-14T16:56:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.167+0000 7f649eabb640 1 -- 172.21.15.29:0/3975396560 wait complete. 2023-10-14T16:56:45.185 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:45.185+0000 7f8f6fe0d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:56:45.186 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:56:45.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:45.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T16:56:45.188 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:45.186+0000 7fbc4882b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:45.236 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:45.235+0000 7fbc4882b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:45.284 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:56:45.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:56:45.285 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:56:45.336 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:45.336 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:45.357 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:45.355+0000 7f8f6fe0d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:56:45.359 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:45.358+0000 7fbc4882b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:45.390 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:45.391 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:45.391 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:45.407 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:45.406+0000 7f8f6fe0d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:56:45.446 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:45.446 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:45.447 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:45.500 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:45.501 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:45.532 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:45.532+0000 7f8f6fe0d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:56:45.705 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:56:45.711 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:56:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.717+0000 7f717b1ac640 1 Processor -- start 2023-10-14T16:56:45.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.717+0000 7f717b1ac640 1 -- start start 2023-10-14T16:56:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.718+0000 7f717b1ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f717415cd70 0x7f71740d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:45.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.718+0000 7f717b1ac640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f717415d150 con 0x7f71740b8480 2023-10-14T16:56:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f717415cd70 0x7f71740d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.718+0000 7f717b1ac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71740d1180 con 0x7f71740cded0 2023-10-14T16:56:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717b1ac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71740d1460 con 0x7f717415cd70 2023-10-14T16:56:45.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f717415cd70 0x7f71740d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41516/0 (socket says 172.21.15.29:41516) 2023-10-14T16:56:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 -- 172.21.15.29:0/2928458838 learned_addr learned my addr 172.21.15.29:0/2928458838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 -- 172.21.15.29:0/2928458838 >> v1:172.21.15.186:6789/0 conn(0x7f71740cded0 legacy=0x7f71740b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 -- 172.21.15.29:0/2928458838 >> v1:172.21.15.29:6789/0 conn(0x7f71740b8480 legacy=0x7f71740cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f717a1aa640 1 -- 172.21.15.29:0/2928458838 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71740d1740 con 0x7f717415cd70 2023-10-14T16:56:45.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.719+0000 7f71791a8640 1 -- 172.21.15.29:0/2928458838 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578764969 0 0) 0x7f717415d150 con 0x7f71740b8480 2023-10-14T16:56:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.720+0000 7f717a1aa640 1 --2- 172.21.15.29:0/2928458838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717415cd70 0x7f71740d2220 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7170009870 tx=0x7f717002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a61cd30ac456181d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f71791a8640 1 -- 172.21.15.29:0/2928458838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170030860 con 0x7f717415cd70 2023-10-14T16:56:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f71791a8640 1 -- 172.21.15.29:0/2928458838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71700309c0 con 0x7f717415cd70 2023-10-14T16:56:45.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f71791a8640 1 -- 172.21.15.29:0/2928458838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7170030d50 con 0x7f717415cd70 2023-10-14T16:56:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f717b1ac640 1 -- 172.21.15.29:0/2928458838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717415cd70 msgr2=0x7f71740d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f717b1ac640 1 --2- 172.21.15.29:0/2928458838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717415cd70 0x7f71740d2220 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f7170009870 tx=0x7f717002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f717b1ac640 1 -- 172.21.15.29:0/2928458838 shutdown_connections 2023-10-14T16:56:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f717b1ac640 1 --2- 172.21.15.29:0/2928458838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f717415cd70 0x7f71740d2220 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.721+0000 7f717b1ac640 1 -- 172.21.15.29:0/2928458838 >> 172.21.15.29:0/2928458838 conn(0x7f71740cf570 msgr2=0x7f71740cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:45.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.722+0000 7f717b1ac640 1 -- 172.21.15.29:0/2928458838 shutdown_connections 2023-10-14T16:56:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.722+0000 7f717b1ac640 1 -- 172.21.15.29:0/2928458838 wait complete. 2023-10-14T16:56:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.723+0000 7f717b1ac640 1 Processor -- start 2023-10-14T16:56:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.723+0000 7f717b1ac640 1 -- start start 2023-10-14T16:56:45.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71740b8480 0x7f717415a320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 0x7f717415f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740aae80 con 0x7f71740cded0 2023-10-14T16:56:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71740aa270 con 0x7f717415cd70 2023-10-14T16:56:45.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.724+0000 7f717b1ac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71740aa550 con 0x7f71740b8480 2023-10-14T16:56:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41524/0 (socket says 172.21.15.29:41524) 2023-10-14T16:56:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 -- 172.21.15.29:0/1047548509 learned_addr learned my addr 172.21.15.29:0/1047548509 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f717a1aa640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71740b8480 0x7f717415a320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:45.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f717a9ab640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 0x7f717415f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71740b8480 msgr2=0x7f717415a320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71740b8480 0x7f717415a320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 msgr2=0x7f717415f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 0x7f717415f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f71799a9640 1 -- 172.21.15.29:0/1047548509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71700094d0 con 0x7f71740cded0 2023-10-14T16:56:45.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.725+0000 7f717a9ab640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 0x7f717415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f71799a9640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f716400c4b0 tx=0x7f716400c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:45.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7164005480 con 0x7f71740cded0 2023-10-14T16:56:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71640035f0 con 0x7f71740cded0 2023-10-14T16:56:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71640039e0 con 0x7f71740cded0 2023-10-14T16:56:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71740aa830 con 0x7f71740cded0 2023-10-14T16:56:45.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.726+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71740a9f00 con 0x7f71740cded0 2023-10-14T16:56:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.727+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71740ab030 con 0x7f71740cded0 2023-10-14T16:56:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.728+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f7164003b40 con 0x7f71740cded0 2023-10-14T16:56:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.730+0000 7f716affd640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 0x7f715806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:45.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.730+0000 7f717a1aa640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 msgr2=0x7f715806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.730+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f7164088e00 con 0x7f71740cded0 2023-10-14T16:56:45.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.730+0000 7f717a1aa640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 0x7f715806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:45.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.735+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f71740ab030 con 0x7f71740cded0 2023-10-14T16:56:45.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.931+0000 7f717a1aa640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 msgr2=0x7f715806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:45.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.931+0000 7f717a1aa640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 0x7f715806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:45.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.968+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f717404e9b0 con 0x7f71740cded0 2023-10-14T16:56:45.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.969+0000 7f716affd640 1 -- 172.21.15.29:0/1047548509 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f7164017070 con 0x7f71740cded0 2023-10-14T16:56:45.970 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:45.971 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:45.971 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.971+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 msgr2=0x7f715806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 0x7f715806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 msgr2=0x7f71740cba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:45.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f716400c4b0 tx=0x7f716400c970 comp rx=0 tx=0).stop 2023-10-14T16:56:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 shutdown_connections 2023-10-14T16:56:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f717415cd70 0x7f717415f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71740cded0 0x7f71740cba40 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f715806afc0 0x7f715806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 --2- 172.21.15.29:0/1047548509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71740b8480 0x7f717415a320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 >> 172.21.15.29:0/1047548509 conn(0x7f71740cf570 msgr2=0x7f71740b25a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 shutdown_connections 2023-10-14T16:56:45.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:45.972+0000 7f717b1ac640 1 -- 172.21.15.29:0/1047548509 wait complete. 2023-10-14T16:56:45.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:46.155 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:46.154+0000 7fbc4882b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:46.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.193+0000 7f3d7d52c640 1 Processor -- start 2023-10-14T16:56:46.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.193+0000 7f3d7d52c640 1 -- start start 2023-10-14T16:56:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.194+0000 7f3d7d52c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d780ad970 0x7f3d780cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.194+0000 7f3d7d52c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d780ccfb0 0x7f3d780cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.194+0000 7f3d7d52c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d7815b9d0 con 0x7f3d7806c040 2023-10-14T16:56:46.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.194+0000 7f3d7d52c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7815bcb0 con 0x7f3d780ccfb0 2023-10-14T16:56:46.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.194+0000 7f3d7d52c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d7815bf90 con 0x7f3d780ad970 2023-10-14T16:56:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d780ccfb0 0x7f3d780cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3d780ccfb0 0x7f3d780cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41530/0 (socket says 172.21.15.29:41530) 2023-10-14T16:56:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 -- 172.21.15.29:0/2217890835 learned_addr learned my addr 172.21.15.29:0/2217890835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:46.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d777fe640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d780ad970 0x7f3d780cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 -- 172.21.15.29:0/2217890835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d780ad970 msgr2=0x7f3d780cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d780ad970 0x7f3d780cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 -- 172.21.15.29:0/2217890835 >> v1:172.21.15.186:6789/0 conn(0x7f3d7806c040 legacy=0x7f3d7806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.195+0000 7f3d7cd2b640 1 -- 172.21.15.29:0/2217890835 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d780cc270 con 0x7f3d780ccfb0 2023-10-14T16:56:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.197+0000 7f3d7cd2b640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ccfb0 0x7f3d780cd440 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009e80 tx=0x7f3d7002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=221c8ed179572359 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.197+0000 7f3d76ffd640 1 -- 172.21.15.29:0/2217890835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d7002caf0 con 0x7f3d780ccfb0 2023-10-14T16:56:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.197+0000 7f3d76ffd640 1 -- 172.21.15.29:0/2217890835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d7002cc50 con 0x7f3d780ccfb0 2023-10-14T16:56:46.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.197+0000 7f3d76ffd640 1 -- 172.21.15.29:0/2217890835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d700343f0 con 0x7f3d780ccfb0 2023-10-14T16:56:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 -- 172.21.15.29:0/2217890835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ccfb0 msgr2=0x7f3d780cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ccfb0 0x7f3d780cd440 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f3d70009e80 tx=0x7f3d7002bf50 comp rx=0 tx=0).stop 2023-10-14T16:56:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 -- 172.21.15.29:0/2217890835 shutdown_connections 2023-10-14T16:56:46.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ccfb0 0x7f3d780cd440 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/2217890835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d780ad970 0x7f3d780cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 -- 172.21.15.29:0/2217890835 >> 172.21.15.29:0/2217890835 conn(0x7f3d780ce6a0 msgr2=0x7f3d780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:46.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 -- 172.21.15.29:0/2217890835 shutdown_connections 2023-10-14T16:56:46.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.198+0000 7f3d7d52c640 1 -- 172.21.15.29:0/2217890835 wait complete. 2023-10-14T16:56:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.200+0000 7f3d7d52c640 1 Processor -- start 2023-10-14T16:56:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.200+0000 7f3d7d52c640 1 -- start start 2023-10-14T16:56:46.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 0x7f3d780c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d780ccfb0 0x7f3d78156160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780c7e20 con 0x7f3d780ad970 2023-10-14T16:56:46.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d781566a0 con 0x7f3d780ccfb0 2023-10-14T16:56:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d7d52c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780c6840 con 0x7f3d7806c040 2023-10-14T16:56:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41542/0 (socket says 172.21.15.29:41542) 2023-10-14T16:56:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.201+0000 7f3d777fe640 1 -- 172.21.15.29:0/1469628339 learned_addr learned my addr 172.21.15.29:0/1469628339 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:46.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d77fff640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 0x7f3d780c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d7cd2b640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d780ccfb0 0x7f3d78156160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 msgr2=0x7f3d780c6180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 0x7f3d780c6180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d780ccfb0 msgr2=0x7f3d78156160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d780ccfb0 0x7f3d78156160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 -- 172.21.15.29:0/1469628339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d70009b30 con 0x7f3d780ad970 2023-10-14T16:56:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d77fff640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 0x7f3d780c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d777fe640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00d800 tx=0x7f3d6c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:46.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.202+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00b6c0 con 0x7f3d780ad970 2023-10-14T16:56:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.203+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d6c00b840 con 0x7f3d780ad970 2023-10-14T16:56:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.203+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d6c00bbf0 con 0x7f3d780ad970 2023-10-14T16:56:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.203+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d780c6b20 con 0x7f3d780ad970 2023-10-14T16:56:46.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.203+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d780c70e0 con 0x7f3d780ad970 2023-10-14T16:56:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.204+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d7804e9b0 con 0x7f3d780ad970 2023-10-14T16:56:46.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.205+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3d6c00bd50 con 0x7f3d780ad970 2023-10-14T16:56:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.206+0000 7f3d74ff9640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 0x7f3d6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.206+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f3d6c089ee0 con 0x7f3d780ad970 2023-10-14T16:56:46.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.206+0000 7f3d77fff640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 msgr2=0x7f3d6406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.206+0000 7f3d77fff640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 0x7f3d6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:46.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.210+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d6c054cf0 con 0x7f3d780ad970 2023-10-14T16:56:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.407+0000 7f3d77fff640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 msgr2=0x7f3d6406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.407+0000 7f3d77fff640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 0x7f3d6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.407+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d780ce300 con 0x7f3d780ad970 2023-10-14T16:56:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.410+0000 7f3d74ff9640 1 -- 172.21.15.29:0/1469628339 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) v1 ==== 74+0+17121 (secure 0 0 0) 0x7f3d6c058310 con 0x7f3d780ad970 2023-10-14T16:56:46.411 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:46.412 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:40.500779+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/626362321":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/664522093":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/126969037":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6833/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6832/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/1855767981":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/3741726235":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/2072219911":"2023-10-15T16:56:40.500646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 msgr2=0x7f3d6406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 0x7f3d6406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 msgr2=0x7f3d780c78e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f3d6c00d800 tx=0x7f3d6c00dcc0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 shutdown_connections 2023-10-14T16:56:46.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d780ccfb0 0x7f3d78156160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d780ad970 0x7f3d780c78e0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7f3d6406af90 0x7f3d6406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 --2- 172.21.15.29:0/1469628339 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d7806c040 0x7f3d780c6180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 >> 172.21.15.29:0/1469628339 conn(0x7f3d780ce6a0 msgr2=0x7f3d78072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:46.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 shutdown_connections 2023-10-14T16:56:46.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.413+0000 7f3d7d52c640 1 -- 172.21.15.29:0/1469628339 wait complete. 2023-10-14T16:56:46.429 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:46.429+0000 7f8f6fe0d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:56:46.434 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:56:24.067424+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 9, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:56:25.117255+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '82', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:46.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:46.451 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:46.450+0000 7fbc4882b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:46.508 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:46.507+0000 7fbc4882b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:46.623 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:56:46.621+0000 7fbc4882b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.658+0000 7fa4e7e83640 1 Processor -- start 2023-10-14T16:56:46.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.658+0000 7fa4e7e83640 1 -- start start 2023-10-14T16:56:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.659+0000 7fa4e7e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4e00b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.659+0000 7fa4e7e83640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e00d3980 con 0x7fa4e00cf710 2023-10-14T16:56:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e7e83640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4e00ce630 con 0x7fa4e00b3420 2023-10-14T16:56:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e7e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00d3c60 con 0x7fa4e00cfaf0 2023-10-14T16:56:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e5c1f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa4e00cf710 0x7fa4e00b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55038/0 (socket says 172.21.15.29:55038) 2023-10-14T16:56:46.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/2462824424 learned_addr learned my addr 172.21.15.29:0/2462824424 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:46.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e6420640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4e00b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 86119869 0 0) 0x7fa4e00d3980 con 0x7fa4e00cf710 2023-10-14T16:56:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.660+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4cc003520 con 0x7fa4e00cf710 2023-10-14T16:56:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1832471210 0 0) 0x7fa4cc003520 con 0x7fa4e00cf710 2023-10-14T16:56:46.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 msgr2=0x7fa4e00b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4e00b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 >> v1:172.21.15.186:6789/0 conn(0x7fa4e00b3420 legacy=0x7fa4e00b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e00ce910 con 0x7fa4e00cf710 2023-10-14T16:56:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa4dc0017f0 con 0x7fa4e00cf710 2023-10-14T16:56:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 >> v1:172.21.15.29:6789/0 conn(0x7fa4e00cf710 legacy=0x7fa4e00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4cc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4cc005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00d3c60 con 0x7fa4cc0039a0 2023-10-14T16:56:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00bb40 con 0x7fa4cc0047c0 2023-10-14T16:56:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4cc00bd00 con 0x7fa4e00cfaf0 2023-10-14T16:56:46.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4cc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.661+0000 7fa4e541e640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e6420640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4cc005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 msgr2=0x7fa4cc005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e00cfaf0 0x7fa4cc005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4cc0047c0 msgr2=0x7fa4cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:46.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4cc0047c0 0x7fa4cc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.662+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/2462824424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc003520 con 0x7fa4cc0039a0 2023-10-14T16:56:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.663+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4cc003dc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc007fb0 tx=0x7fa4cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5266671138591aae server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.663+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc007270 con 0x7fa4cc0039a0 2023-10-14T16:56:46.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.663+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc0073f0 con 0x7fa4cc0039a0 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.663+0000 7fa4e4c1d640 1 -- 172.21.15.29:0/2462824424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc02c7c0 con 0x7fa4cc0039a0 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 -- 172.21.15.29:0/2462824424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 msgr2=0x7fa4cc003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/2462824424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4cc003dc0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc007fb0 tx=0x7fa4cc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e6420640 1 -- 172.21.15.29:0/2462824424 reap_dead start 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 -- 172.21.15.29:0/2462824424 shutdown_connections 2023-10-14T16:56:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 -- 172.21.15.29:0/2462824424 >> 172.21.15.29:0/2462824424 conn(0x7fa4e00d2cc0 msgr2=0x7fa4e00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 -- 172.21.15.29:0/2462824424 shutdown_connections 2023-10-14T16:56:46.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.664+0000 7fa4e7e83640 1 -- 172.21.15.29:0/2462824424 wait complete. 2023-10-14T16:56:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.666+0000 7fa4e7e83640 1 Processor -- start 2023-10-14T16:56:46.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.666+0000 7fa4e7e83640 1 -- start start 2023-10-14T16:56:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 0x7fa4e015b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 0x7fa4e00d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e015b920 con 0x7fa4cc0039a0 2023-10-14T16:56:46.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e00cb110 con 0x7fa4e00af890 2023-10-14T16:56:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e7e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4e01f3090 con 0x7fa4e01f2830 2023-10-14T16:56:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41558/0 (socket says 172.21.15.29:41558) 2023-10-14T16:56:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/4069710189 learned_addr learned my addr 172.21.15.29:0/4069710189 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:46.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e541e640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 0x7fa4e015b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e6420640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 0x7fa4e00d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 msgr2=0x7fa4e00d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 0x7fa4e00d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 msgr2=0x7fa4e015b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 0x7fa4e015b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e5c1f640 1 -- 172.21.15.29:0/4069710189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e01f3370 con 0x7fa4cc0039a0 2023-10-14T16:56:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.667+0000 7fa4e541e640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 0x7fa4e015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4e6420640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 0x7fa4e00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:46.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4e5c1f640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc000ef0 tx=0x7fa4e015d990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc039070 con 0x7fa4cc0039a0 2023-10-14T16:56:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4dc02cbd0 con 0x7fa4cc0039a0 2023-10-14T16:56:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4dc03c040 con 0x7fa4cc0039a0 2023-10-14T16:56:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4e01f3650 con 0x7fa4cc0039a0 2023-10-14T16:56:46.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.668+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4e01f2cb0 con 0x7fa4cc0039a0 2023-10-14T16:56:46.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.670+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4e00afc70 con 0x7fa4cc0039a0 2023-10-14T16:56:46.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.670+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 33) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fa4dc032070 con 0x7fa4cc0039a0 2023-10-14T16:56:46.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.671+0000 7fa4d67fc640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 0x7fa4c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:46.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.671+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fa4dc0aabd0 con 0x7fa4cc0039a0 2023-10-14T16:56:46.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.671+0000 7fa4e541e640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 msgr2=0x7fa4c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:46.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.671+0000 7fa4e541e640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 0x7fa4c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:56:46.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.676+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4e00afc70 con 0x7fa4cc0039a0 2023-10-14T16:56:46.757 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:46.756+0000 7f8f6fe0d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:56:46.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:46.816+0000 7f8f6fe0d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:56:46.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.871+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa4e00afc70 con 0x7fa4cc0039a0 2023-10-14T16:56:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.872+0000 7fa4e541e640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 msgr2=0x7fa4c406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1682700904 2023-10-14T16:56:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.872+0000 7fa4e541e640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 0x7fa4c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:56:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.872+0000 7fa4d67fc640 1 -- 172.21.15.29:0/4069710189 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v83) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa4e00afc70 con 0x7fa4cc0039a0 2023-10-14T16:56:46.873 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 msgr2=0x7fa4c406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:56:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 0x7fa4c406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 msgr2=0x7fa4e00ccaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7fa4dc000ef0 tx=0x7fa4e015d990 comp rx=0 tx=0).stop 2023-10-14T16:56:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 shutdown_connections 2023-10-14T16:56:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:6832/1682700904,v1:172.21.15.29:6833/1682700904] conn(0x7fa4c406af90 0x7fa4c406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4e01f2830 0x7fa4e00d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4e00af890 0x7fa4e015b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 --2- 172.21.15.29:0/4069710189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4cc0039a0 0x7fa4e00ccaf0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 >> 172.21.15.29:0/4069710189 conn(0x7fa4e00d2cc0 msgr2=0x7fa4e015d110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 shutdown_connections 2023-10-14T16:56:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:46.875+0000 7fa4e7e83640 1 -- 172.21.15.29:0/4069710189 wait complete. 2023-10-14T16:56:46.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:56:46.941 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:46.941+0000 7f8f6fe0d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:56:47.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.027+0000 7f8ef4cc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.027+0000 7f8ef4cc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.027+0000 7f8ef4cc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.028 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.027+0000 7f8ef4cc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.029 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.027+0000 7f8ef4cc9640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.029 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.028+0000 7f8ef04c0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.030 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.028+0000 7f8ef04c0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.030 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.028+0000 7f8ef04c0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.030 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.028+0000 7f8ef04c0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.030 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:56:47.028+0000 7f8ef04c0640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:56:47.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.111+0000 7fd50f27c640 1 Processor -- start 2023-10-14T16:56:47.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.111+0000 7fd50f27c640 1 -- start start 2023-10-14T16:56:47.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50f27c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd5080ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50f27c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080cb030 con 0x7fd50815ad70 2023-10-14T16:56:47.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50f27c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080c91c0 con 0x7fd50806a4b0 2023-10-14T16:56:47.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50f27c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c94a0 con 0x7fd5080caac0 2023-10-14T16:56:47.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50d018640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd50815ad70 0x7fd508069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55044/0 (socket says 172.21.15.29:55044) 2023-10-14T16:56:47.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.112+0000 7fd50d018640 1 -- 172.21.15.29:0/505486517 learned_addr learned my addr 172.21.15.29:0/505486517 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:47.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950134626 0 0) 0x7fd5080cb030 con 0x7fd50815ad70 2023-10-14T16:56:47.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4003520 con 0x7fd50815ad70 2023-10-14T16:56:47.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd50d819640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd5080ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192640564 0 0) 0x7fd4f4003520 con 0x7fd50815ad70 2023-10-14T16:56:47.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 msgr2=0x7fd5080ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd5080ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 >> v1:172.21.15.186:6789/0 conn(0x7fd50806a4b0 legacy=0x7fd5080ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080c9780 con 0x7fd50815ad70 2023-10-14T16:56:47.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd5040019f0 con 0x7fd50815ad70 2023-10-14T16:56:47.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 >> v1:172.21.15.29:6789/0 conn(0x7fd50815ad70 legacy=0x7fd508069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f40039a0 0x7fd4f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd4f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c94a0 con 0x7fd4f4004d00 2023-10-14T16:56:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400bb10 con 0x7fd4f40039a0 2023-10-14T16:56:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.113+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f400bca0 con 0x7fd5080caac0 2023-10-14T16:56:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d018640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f40039a0 0x7fd4f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50c817640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd4f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 -- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 msgr2=0x7fd4f40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080caac0 0x7fd4f40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 -- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f40039a0 msgr2=0x7fd4f4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f40039a0 0x7fd4f4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.114+0000 7fd50d819640 1 -- 172.21.15.29:0/505486517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f4003520 con 0x7fd4f4004d00 2023-10-14T16:56:47.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.115+0000 7fd50d819640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd50000ba60 tx=0x7fd4f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab09dbdf7c61f8a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:47.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.115+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500032630 con 0x7fd4f4004d00 2023-10-14T16:56:47.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.115+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd500032790 con 0x7fd4f4004d00 2023-10-14T16:56:47.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.115+0000 7fd4fffff640 1 -- 172.21.15.29:0/505486517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd500032b20 con 0x7fd4f4004d00 2023-10-14T16:56:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 -- 172.21.15.29:0/505486517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 msgr2=0x7fd4f4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 --2- 172.21.15.29:0/505486517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd4f4005180 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7fd50000ba60 tx=0x7fd4f40057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50d819640 1 -- 172.21.15.29:0/505486517 reap_dead start 2023-10-14T16:56:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 -- 172.21.15.29:0/505486517 shutdown_connections 2023-10-14T16:56:47.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 -- 172.21.15.29:0/505486517 >> 172.21.15.29:0/505486517 conn(0x7fd5080ce9b0 msgr2=0x7fd5080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 -- 172.21.15.29:0/505486517 shutdown_connections 2023-10-14T16:56:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.116+0000 7fd50f27c640 1 -- 172.21.15.29:0/505486517 wait complete. 2023-10-14T16:56:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.118+0000 7fd50f27c640 1 Processor -- start 2023-10-14T16:56:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.118+0000 7fd50f27c640 1 -- start start 2023-10-14T16:56:47.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd508156cf0 0x7fd50815c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080c5620 0x7fd50815cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508154890 con 0x7fd4f4004d00 2023-10-14T16:56:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5080c4910 con 0x7fd508156cf0 2023-10-14T16:56:47.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50f27c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd508157180 con 0x7fd5080c5620 2023-10-14T16:56:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41670/0 (socket says 172.21.15.29:41670) 2023-10-14T16:56:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 -- 172.21.15.29:0/1690981923 learned_addr learned my addr 172.21.15.29:0/1690981923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 -- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080c5620 msgr2=0x7fd50815cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080c5620 0x7fd50815cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 -- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd508156cf0 msgr2=0x7fd50815c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd508156cf0 0x7fd50815c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.119+0000 7fd50d018640 1 -- 172.21.15.29:0/1690981923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080c4b60 con 0x7fd4f4004d00 2023-10-14T16:56:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd50d018640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd504007a50 tx=0x7fd50400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:47.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd50400ee90 con 0x7fd4f4004d00 2023-10-14T16:56:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd504013420 con 0x7fd4f4004d00 2023-10-14T16:56:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5040137b0 con 0x7fd4f4004d00 2023-10-14T16:56:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5080c4e40 con 0x7fd4f4004d00 2023-10-14T16:56:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.120+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd508153ac0 con 0x7fd4f4004d00 2023-10-14T16:56:47.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.121+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50804e160 con 0x7fd4f4004d00 2023-10-14T16:56:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.122+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fd504013910 con 0x7fd4f4004d00 2023-10-14T16:56:47.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.124+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7fd504089d40 con 0x7fd4f4004d00 2023-10-14T16:56:47.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.129+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd504054e70 con 0x7fd4f4004d00 2023-10-14T16:56:47.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.323+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd5080736e0 con 0x7fd4f4004d00 2023-10-14T16:56:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.324+0000 7fd4fdffb640 1 -- 172.21.15.29:0/1690981923 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v84) v1 ==== 104+0+11 (secure 0 0 0) 0x7fd504058490 con 0x7fd4f4004d00 2023-10-14T16:56:47.325 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:56:47.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 msgr2=0x7fd50815c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7fd504007a50 tx=0x7fd50400e530 comp rx=0 tx=0).stop 2023-10-14T16:56:47.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 shutdown_connections 2023-10-14T16:56:47.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5080c5620 0x7fd50815cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd508156cf0 0x7fd50815c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 --2- 172.21.15.29:0/1690981923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f4004d00 0x7fd50815c4a0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 >> 172.21.15.29:0/1690981923 conn(0x7fd5080ce9b0 msgr2=0x7fd508069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 shutdown_connections 2023-10-14T16:56:47.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.327+0000 7fd50f27c640 1 -- 172.21.15.29:0/1690981923 wait complete. 2023-10-14T16:56:47.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:56:47.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.578+0000 7ff3317b0640 1 Processor -- start 2023-10-14T16:56:47.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.578+0000 7ff3317b0640 1 -- start start 2023-10-14T16:56:47.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff3317b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 0x7ff32c065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff3317b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff32c065ff0 0x7ff32c0ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff3317b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c066420 con 0x7ff32c15d330 2023-10-14T16:56:47.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff3317b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c06b1e0 con 0x7ff32c065ff0 2023-10-14T16:56:47.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff3317b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c06cee0 con 0x7ff32c1618b0 2023-10-14T16:56:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff32affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff32c15d330 0x7ff32c06c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55048/0 (socket says 172.21.15.29:55048) 2023-10-14T16:56:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.579+0000 7ff32affd640 1 -- 172.21.15.29:0/1307743541 learned_addr learned my addr 172.21.15.29:0/1307743541 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4252672000 0 0) 0x7ff32c066420 con 0x7ff32c15d330 2023-10-14T16:56:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318003520 con 0x7ff32c15d330 2023-10-14T16:56:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/1307743541 >> v2:172.21.15.186:3300/0 conn(0x7ff32c065ff0 0x7ff32c0ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff32a7fc640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 0x7ff32c065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1770119297 0 0) 0x7ff318003520 con 0x7ff32c15d330 2023-10-14T16:56:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 msgr2=0x7ff32c065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 0x7ff32c065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 >> v2:172.21.15.186:3300/0 conn(0x7ff32c065ff0 msgr2=0x7ff32c0ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 --2- 172.21.15.29:0/1307743541 >> v2:172.21.15.186:3300/0 conn(0x7ff32c065ff0 0x7ff32c0ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0d0890 con 0x7ff32c15d330 2023-10-14T16:56:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff31c0019f0 con 0x7ff32c15d330 2023-10-14T16:56:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 >> v1:172.21.15.29:6789/0 conn(0x7ff32c15d330 legacy=0x7ff32c06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 0x7ff318003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff318003ff0 0x7ff3180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff318004d00 0x7ff318005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c06b1e0 con 0x7ff318004d00 2023-10-14T16:56:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c06cee0 con 0x7ff318003ff0 2023-10-14T16:56:47.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.580+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31800bb90 con 0x7ff32c1618b0 2023-10-14T16:56:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff318004d00 0x7ff318005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32a7fc640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff318003ff0 0x7ff3180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 -- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 msgr2=0x7ff318003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff32c1618b0 0x7ff318003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 -- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff318003ff0 msgr2=0x7ff3180047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff318003ff0 0x7ff3180047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32b7fe640 1 -- 172.21.15.29:0/1307743541 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff318003520 con 0x7ff318004d00 2023-10-14T16:56:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.581+0000 7ff32a7fc640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff318003ff0 0x7ff3180047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.582+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff318004d00 0x7ff318005180 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff32400ba30 tx=0x7ff3180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50bcbc4aeaea2818 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.582+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324032680 con 0x7ff318004d00 2023-10-14T16:56:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.582+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3240327e0 con 0x7ff318004d00 2023-10-14T16:56:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.582+0000 7ff329ffb640 1 -- 172.21.15.29:0/1307743541 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324032b70 con 0x7ff318004d00 2023-10-14T16:56:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 -- 172.21.15.29:0/1307743541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff318004d00 msgr2=0x7ff318005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 --2- 172.21.15.29:0/1307743541 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff318004d00 0x7ff318005180 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7ff32400ba30 tx=0x7ff3180057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff32b7fe640 1 -- 172.21.15.29:0/1307743541 reap_dead start 2023-10-14T16:56:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 -- 172.21.15.29:0/1307743541 shutdown_connections 2023-10-14T16:56:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 -- 172.21.15.29:0/1307743541 >> 172.21.15.29:0/1307743541 conn(0x7ff32c0d0ab0 msgr2=0x7ff32c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 -- 172.21.15.29:0/1307743541 shutdown_connections 2023-10-14T16:56:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.583+0000 7ff3317b0640 1 -- 172.21.15.29:0/1307743541 wait complete. 2023-10-14T16:56:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.585+0000 7ff3317b0640 1 Processor -- start 2023-10-14T16:56:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.585+0000 7ff3317b0640 1 -- start start 2023-10-14T16:56:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff318004d00 0x7ff32c0c87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32c0b3c00 0x7ff32c1568f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c1602a0 con 0x7ff32c15f8e0 2023-10-14T16:56:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c0b43a0 con 0x7ff32c0b3c00 2023-10-14T16:56:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff3317b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff32c0b4680 con 0x7ff318004d00 2023-10-14T16:56:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff32b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41692/0 (socket says 172.21.15.29:41692) 2023-10-14T16:56:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.586+0000 7ff32b7fe640 1 -- 172.21.15.29:0/144534534 learned_addr learned my addr 172.21.15.29:0/144534534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32affd640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff318004d00 0x7ff32c0c87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32a7fc640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32c0b3c00 0x7ff32c1568f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 -- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff318004d00 msgr2=0x7ff32c0c87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff318004d00 0x7ff32c0c87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 -- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32c0b3c00 msgr2=0x7ff32c1568f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32c0b3c00 0x7ff32c1568f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 -- 172.21.15.29:0/144534534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c0b4960 con 0x7ff32c15f8e0 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.587+0000 7ff32b7fe640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff324035040 tx=0x7ff324004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.588+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff324004220 con 0x7ff32c15f8e0 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.588+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324004380 con 0x7ff32c15f8e0 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.588+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff32403b830 con 0x7ff32c15f8e0 2023-10-14T16:56:47.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.588+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c0b4c40 con 0x7ff32c15f8e0 2023-10-14T16:56:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.588+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c156e30 con 0x7ff32c15f8e0 2023-10-14T16:56:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.589+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff32c04e160 con 0x7ff32c15f8e0 2023-10-14T16:56:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.590+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 34) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff32403b990 con 0x7ff32c15f8e0 2023-10-14T16:56:47.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.591+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7ff3240abae0 con 0x7ff32c15f8e0 2023-10-14T16:56:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.595+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff324076c10 con 0x7ff32c15f8e0 2023-10-14T16:56:47.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.787+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff32c06f3a0 con 0x7ff32c15f8e0 2023-10-14T16:56:47.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.788+0000 7ff3237fe640 1 -- 172.21.15.29:0/144534534 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v84) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff32407a230 con 0x7ff32c15f8e0 2023-10-14T16:56:47.789 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:56:47.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 msgr2=0x7ff32c15fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7ff324035040 tx=0x7ff324004730 comp rx=0 tx=0).stop 2023-10-14T16:56:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 shutdown_connections 2023-10-14T16:56:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff32c15f8e0 0x7ff32c15fd60 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff32c0b3c00 0x7ff32c1568f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 --2- 172.21.15.29:0/144534534 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff318004d00 0x7ff32c0c87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 >> 172.21.15.29:0/144534534 conn(0x7ff32c0d0ab0 msgr2=0x7ff32c0d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 shutdown_connections 2023-10-14T16:56:47.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:47.791+0000 7ff3317b0640 1 -- 172.21.15.29:0/144534534 wait complete. 2023-10-14T16:56:47.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:48.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.042+0000 7efc5759e640 1 Processor -- start 2023-10-14T16:56:48.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.043+0000 7efc5759e640 1 -- start start 2023-10-14T16:56:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.044+0000 7efc5759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc5815cbe0 0x7efc580d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.044+0000 7efc5759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580d15a0 con 0x7efc580b4c80 2023-10-14T16:56:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.044+0000 7efc5759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580d1880 con 0x7efc5815cfc0 2023-10-14T16:56:48.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.044+0000 7efc5759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc580d1b60 con 0x7efc5815cbe0 2023-10-14T16:56:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.044+0000 7efc5659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efc5815cbe0 0x7efc580d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.045+0000 7efc56d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efc580b4c80 0x7efc580b5c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55054/0 (socket says 172.21.15.29:55054) 2023-10-14T16:56:48.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.045+0000 7efc56d9d640 1 -- 172.21.15.29:0/4131596263 learned_addr learned my addr 172.21.15.29:0/4131596263 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:48.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.045+0000 7efc5659c640 1 -- 172.21.15.29:0/4131596263 >> v1:172.21.15.186:6789/0 conn(0x7efc5815cfc0 legacy=0x7efc580b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.045+0000 7efc5659c640 1 -- 172.21.15.29:0/4131596263 >> v1:172.21.15.29:6789/0 conn(0x7efc580b4c80 legacy=0x7efc580b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.045+0000 7efc5659c640 1 -- 172.21.15.29:0/4131596263 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580d1e40 con 0x7efc5815cbe0 2023-10-14T16:56:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.046+0000 7efc5659c640 1 --2- 172.21.15.29:0/4131596263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cbe0 0x7efc580d0b50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efc48009d90 tx=0x7efc4802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ee0b56889c99d17 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.046+0000 7efc5559a640 1 -- 172.21.15.29:0/4131596263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc4802cc60 con 0x7efc5815cbe0 2023-10-14T16:56:48.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.046+0000 7efc5559a640 1 -- 172.21.15.29:0/4131596263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc4802cdc0 con 0x7efc5815cbe0 2023-10-14T16:56:48.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.046+0000 7efc5559a640 1 -- 172.21.15.29:0/4131596263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc48034660 con 0x7efc5815cbe0 2023-10-14T16:56:48.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 -- 172.21.15.29:0/4131596263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cbe0 msgr2=0x7efc580d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 --2- 172.21.15.29:0/4131596263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cbe0 0x7efc580d0b50 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7efc48009d90 tx=0x7efc4802beb0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 -- 172.21.15.29:0/4131596263 shutdown_connections 2023-10-14T16:56:48.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 --2- 172.21.15.29:0/4131596263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cbe0 0x7efc580d0b50 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 -- 172.21.15.29:0/4131596263 >> 172.21.15.29:0/4131596263 conn(0x7efc580cf570 msgr2=0x7efc580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:48.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 -- 172.21.15.29:0/4131596263 shutdown_connections 2023-10-14T16:56:48.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.047+0000 7efc5759e640 1 -- 172.21.15.29:0/4131596263 wait complete. 2023-10-14T16:56:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.049+0000 7efc5759e640 1 Processor -- start 2023-10-14T16:56:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.049+0000 7efc5759e640 1 -- start start 2023-10-14T16:56:48.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc580b4c80 0x7efc581456d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc5815cbe0 0x7efc580c8da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5815b220 con 0x7efc5815cfc0 2023-10-14T16:56:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc580c6620 con 0x7efc580b4c80 2023-10-14T16:56:48.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc580c6e30 con 0x7efc5815cbe0 2023-10-14T16:56:48.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41710/0 (socket says 172.21.15.29:41710) 2023-10-14T16:56:48.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 -- 172.21.15.29:0/2915985630 learned_addr learned my addr 172.21.15.29:0/2915985630 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:48.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc5659c640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc580b4c80 0x7efc581456d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 -- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc5815cbe0 msgr2=0x7efc580c8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc5815cbe0 0x7efc580c8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.050+0000 7efc56d9d640 1 -- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc580b4c80 msgr2=0x7efc581456d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc56d9d640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc580b4c80 0x7efc581456d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc56d9d640 1 -- 172.21.15.29:0/2915985630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc48009a70 con 0x7efc5815cfc0 2023-10-14T16:56:48.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc56d9d640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c420 tx=0x7efc5000c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc50005ce0 con 0x7efc5815cfc0 2023-10-14T16:56:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc50005e40 con 0x7efc5815cfc0 2023-10-14T16:56:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc500072d0 con 0x7efc5815cfc0 2023-10-14T16:56:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc580c92e0 con 0x7efc5815cfc0 2023-10-14T16:56:48.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.051+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc580c6b50 con 0x7efc5815cfc0 2023-10-14T16:56:48.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.052+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc5804e9b0 con 0x7efc5815cfc0 2023-10-14T16:56:48.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.053+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7efc50007430 con 0x7efc5815cfc0 2023-10-14T16:56:48.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.054+0000 7efc4effd640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 0x7efc3806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.054+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7efc50088930 con 0x7efc5815cfc0 2023-10-14T16:56:48.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.054+0000 7efc5659c640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 0x7efc3806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.055+0000 7efc5659c640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 0x7efc3806d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc48030000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.059+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efc50053a20 con 0x7efc5815cfc0 2023-10-14T16:56:48.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.295+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc5815c5c0 con 0x7efc5815cfc0 2023-10-14T16:56:48.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.296+0000 7efc4effd640 1 -- 172.21.15.29:0/2915985630 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7efc50017090 con 0x7efc5815cfc0 2023-10-14T16:56:48.297 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:48.297 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:48.297 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 msgr2=0x7efc3806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 0x7efc3806d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7efc48036000 tx=0x7efc48030000 comp rx=0 tx=0).stop 2023-10-14T16:56:48.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 msgr2=0x7efc58144da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7efc5000c420 tx=0x7efc5000c8e0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 shutdown_connections 2023-10-14T16:56:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7efc3806acc0 0x7efc3806d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efc5815cfc0 0x7efc58144da0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efc5815cbe0 0x7efc580c8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 --2- 172.21.15.29:0/2915985630 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efc580b4c80 0x7efc581456d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 >> 172.21.15.29:0/2915985630 conn(0x7efc580cf570 msgr2=0x7efc580b4540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:48.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 shutdown_connections 2023-10-14T16:56:48.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.299+0000 7efc5759e640 1 -- 172.21.15.29:0/2915985630 wait complete. 2023-10-14T16:56:48.318 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.539+0000 7f94492e7640 1 Processor -- start 2023-10-14T16:56:48.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.540+0000 7f94492e7640 1 -- start start 2023-10-14T16:56:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f94492e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94440b0b20 0x7f9444164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f94492e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944406f7f0 con 0x7f9444159b20 2023-10-14T16:56:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f94492e7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944406c5e0 con 0x7f9444159f00 2023-10-14T16:56:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f94492e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944406c8c0 con 0x7f94440b0b20 2023-10-14T16:56:48.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f9442ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9444159b20 0x7f944406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55064/0 (socket says 172.21.15.29:55064) 2023-10-14T16:56:48.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.541+0000 7f9442ffd640 1 -- 172.21.15.29:0/60562793 learned_addr learned my addr 172.21.15.29:0/60562793 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f94437fe640 1 --2- 172.21.15.29:0/60562793 >> v2:172.21.15.186:3300/0 conn(0x7f94440b0b20 0x7f9444164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998356926 0 0) 0x7f944406f7f0 con 0x7f9444159b20 2023-10-14T16:56:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9430003520 con 0x7f9444159b20 2023-10-14T16:56:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 377870443 0 0) 0x7f9430003520 con 0x7f9444159b20 2023-10-14T16:56:48.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 >> v2:172.21.15.186:3300/0 conn(0x7f94440b0b20 msgr2=0x7f9444164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 --2- 172.21.15.29:0/60562793 >> v2:172.21.15.186:3300/0 conn(0x7f94440b0b20 0x7f9444164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 >> v1:172.21.15.186:6789/0 conn(0x7f9444159f00 legacy=0x7f94440b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944406cba0 con 0x7f9444159b20 2023-10-14T16:56:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9434001670 con 0x7f9444159b20 2023-10-14T16:56:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.542+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 >> v1:172.21.15.29:6789/0 conn(0x7f9444159b20 legacy=0x7f944406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 0x7f9430004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9430005180 0x7f943000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f944406c8c0 con 0x7f94300047c0 2023-10-14T16:56:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f943000c080 con 0x7f9430005180 2023-10-14T16:56:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f943000c240 con 0x7f94300039c0 2023-10-14T16:56:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 0x7f9430004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f9442ffd640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94437fe640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9430005180 0x7f943000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 -- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300039c0 msgr2=0x7f9430003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300039c0 0x7f9430003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 -- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9430005180 msgr2=0x7f943000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9430005180 0x7f943000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94427fc640 1 -- 172.21.15.29:0/60562793 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9430003520 con 0x7f94300047c0 2023-10-14T16:56:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.543+0000 7f94437fe640 1 -- 172.21.15.29:0/60562793 reap_dead start 2023-10-14T16:56:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f94427fc640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 0x7f9430004c40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f942c001a30 tx=0x7f94300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9617c5578d18327 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c003b70 con 0x7f94300047c0 2023-10-14T16:56:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c003cd0 con 0x7f94300047c0 2023-10-14T16:56:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f9441ffb640 1 -- 172.21.15.29:0/60562793 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c005310 con 0x7f94300047c0 2023-10-14T16:56:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f94492e7640 1 -- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 msgr2=0x7f9430004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.545+0000 7f94492e7640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 0x7f9430004c40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f942c001a30 tx=0x7f94300057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.546+0000 7f94492e7640 1 -- 172.21.15.29:0/60562793 shutdown_connections 2023-10-14T16:56:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.546+0000 7f94492e7640 1 --2- 172.21.15.29:0/60562793 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94300047c0 0x7f9430004c40 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.546+0000 7f94492e7640 1 -- 172.21.15.29:0/60562793 >> 172.21.15.29:0/60562793 conn(0x7f94440cb410 msgr2=0x7f94440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.546+0000 7f94492e7640 1 -- 172.21.15.29:0/60562793 shutdown_connections 2023-10-14T16:56:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.546+0000 7f94492e7640 1 -- 172.21.15.29:0/60562793 wait complete. 2023-10-14T16:56:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.547+0000 7f94492e7640 1 Processor -- start 2023-10-14T16:56:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.548+0000 7f94492e7640 1 -- start start 2023-10-14T16:56:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300047c0 0x7f94440aee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f944406fe90 0x7f9444070310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9444070850 con 0x7f944414d3d0 2023-10-14T16:56:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94440b0270 con 0x7f944406fe90 2023-10-14T16:56:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94492e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f944414d830 con 0x7f94300047c0 2023-10-14T16:56:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41730/0 (socket says 172.21.15.29:41730) 2023-10-14T16:56:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 -- 172.21.15.29:0/1921840916 learned_addr learned my addr 172.21.15.29:0/1921840916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 -- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300047c0 msgr2=0x7f94440aee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300047c0 0x7f94440aee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 -- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f944406fe90 msgr2=0x7f9444070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f944406fe90 0x7f9444070310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.549+0000 7f94427fc640 1 -- 172.21.15.29:0/1921840916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94440af4e0 con 0x7f944414d3d0 2023-10-14T16:56:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f94427fc640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f942c02e230 tx=0x7f942c0014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c042070 con 0x7f944414d3d0 2023-10-14T16:56:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f942c003e40 con 0x7f944414d3d0 2023-10-14T16:56:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944414e700 con 0x7f944414d3d0 2023-10-14T16:56:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f942c006b50 con 0x7f944414d3d0 2023-10-14T16:56:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.550+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944414ecc0 con 0x7f944414d3d0 2023-10-14T16:56:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.551+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94441579e0 con 0x7f944414d3d0 2023-10-14T16:56:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.552+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f942c006cd0 con 0x7f944414d3d0 2023-10-14T16:56:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.553+0000 7f943b7fe640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 0x7f942406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.553+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f942c0aac30 con 0x7f944414d3d0 2023-10-14T16:56:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.553+0000 7f9442ffd640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 0x7f942406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.554+0000 7f9442ffd640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 0x7f942406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9444069740 tx=0x7f9434009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.557+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f942c075be0 con 0x7f944414d3d0 2023-10-14T16:56:48.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.759+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94441579e0 con 0x7f944414d3d0 2023-10-14T16:56:48.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.761+0000 7f943b7fe640 1 -- 172.21.15.29:0/1921840916 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17620 (secure 0 0 0) 0x7f942c079200 con 0x7f944414d3d0 2023-10-14T16:56:48.762 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:48.763 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:46.943991+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2640045513":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3061435492":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/34298176":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6833/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3103792448":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3148536323":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/626362321":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/664522093":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/126969037":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6833/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/266566422":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/1855767981":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/3741726235":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/2072219911":"2023-10-15T16:56:40.500646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:48.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 msgr2=0x7f942406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 0x7f942406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9444069740 tx=0x7f9434009040 comp rx=0 tx=0).stop 2023-10-14T16:56:48.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 msgr2=0x7f94440afd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:48.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f942c02e230 tx=0x7f942c0014a0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 shutdown_connections 2023-10-14T16:56:48.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f942406acc0 0x7f942406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f944406fe90 0x7f9444070310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f944414d3d0 0x7f94440afd30 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 --2- 172.21.15.29:0/1921840916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94300047c0 0x7f94440aee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:48.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 >> 172.21.15.29:0/1921840916 conn(0x7f94440cb410 msgr2=0x7f944415d550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:48.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 shutdown_connections 2023-10-14T16:56:48.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:48.764+0000 7f94492e7640 1 -- 172.21.15.29:0/1921840916 wait complete. 2023-10-14T16:56:48.784 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:56:48.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.009+0000 7f3988947640 1 Processor -- start 2023-10-14T16:56:49.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.010+0000 7f3988947640 1 -- start start 2023-10-14T16:56:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.011+0000 7f3988947640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f398415cfc0 0x7f39840b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.011+0000 7f3988947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 0x7f39840b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3988947640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39840d15a0 con 0x7f398415cbe0 2023-10-14T16:56:49.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3988947640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39840d1880 con 0x7f398415cfc0 2023-10-14T16:56:49.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3988947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39840d1b60 con 0x7f39840d2640 2023-10-14T16:56:49.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3982d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f398415cbe0 0x7f39840d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55070/0 (socket says 172.21.15.29:55070) 2023-10-14T16:56:49.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3982d9d640 1 -- 172.21.15.29:0/2669319819 learned_addr learned my addr 172.21.15.29:0/2669319819 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:49.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f398259c640 1 --2- 172.21.15.29:0/2669319819 >> v2:172.21.15.186:3300/0 conn(0x7f398415cfc0 0x7f39840b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f398359e640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 0x7f39840b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.012+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327420193 0 0) 0x7f39840d15a0 con 0x7f398415cbe0 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f396c003520 con 0x7f398415cbe0 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3440216632 0 0) 0x7f396c003520 con 0x7f398415cbe0 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 msgr2=0x7f39840b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 0x7f39840b4ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 >> v2:172.21.15.186:3300/0 conn(0x7f398415cfc0 msgr2=0x7f39840b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 --2- 172.21.15.29:0/2669319819 >> v2:172.21.15.186:3300/0 conn(0x7f398415cfc0 0x7f39840b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39840d1e40 con 0x7f398415cbe0 2023-10-14T16:56:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3974001670 con 0x7f398415cbe0 2023-10-14T16:56:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 >> v1:172.21.15.29:6789/0 conn(0x7f398415cbe0 legacy=0x7f39840d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 0x7f396c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f396c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c004d00 0x7f396c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840d1880 con 0x7f396c003f40 2023-10-14T16:56:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840d1b60 con 0x7f396c004d00 2023-10-14T16:56:49.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c00b980 con 0x7f39840d2640 2023-10-14T16:56:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.013+0000 7f398259c640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f396c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398359e640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c004d00 0x7f396c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398259c640 1 -- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 msgr2=0x7f396c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398259c640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840d2640 0x7f396c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398259c640 1 -- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c004d00 msgr2=0x7f396c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398259c640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c004d00 0x7f396c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.014+0000 7f398259c640 1 -- 172.21.15.29:0/2669319819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c003520 con 0x7f396c003f40 2023-10-14T16:56:49.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.015+0000 7f398259c640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f396c0047c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3978009ac0 tx=0x7f396c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c19d36e36a8b5dc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.015+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397802c6b0 con 0x7f396c003f40 2023-10-14T16:56:49.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.015+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f397802c810 con 0x7f396c003f40 2023-10-14T16:56:49.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.015+0000 7f3981d9b640 1 -- 172.21.15.29:0/2669319819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f397802cba0 con 0x7f396c003f40 2023-10-14T16:56:49.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 -- 172.21.15.29:0/2669319819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 msgr2=0x7f396c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 --2- 172.21.15.29:0/2669319819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f396c0047c0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f3978009ac0 tx=0x7f396c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f398359e640 1 -- 172.21.15.29:0/2669319819 reap_dead start 2023-10-14T16:56:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 -- 172.21.15.29:0/2669319819 shutdown_connections 2023-10-14T16:56:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 -- 172.21.15.29:0/2669319819 >> 172.21.15.29:0/2669319819 conn(0x7f39840cf570 msgr2=0x7f39840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 -- 172.21.15.29:0/2669319819 shutdown_connections 2023-10-14T16:56:49.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.016+0000 7f3988947640 1 -- 172.21.15.29:0/2669319819 wait complete. 2023-10-14T16:56:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.018+0000 7f3988947640 1 Processor -- start 2023-10-14T16:56:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.018+0000 7f3988947640 1 -- start start 2023-10-14T16:56:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39840b25e0 0x7f39840b2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840a94f0 0x7f39840a9970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39840b2fa0 con 0x7f396c003f40 2023-10-14T16:56:49.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39841f30a0 con 0x7f39840b25e0 2023-10-14T16:56:49.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3988947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39841599d0 con 0x7f39840a94f0 2023-10-14T16:56:49.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.019+0000 7f3982d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41758/0 (socket says 172.21.15.29:41758) 2023-10-14T16:56:49.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 -- 172.21.15.29:0/2836889374 learned_addr learned my addr 172.21.15.29:0/2836889374 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:49.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f398359e640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840a94f0 0x7f39840a9970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 -- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840a94f0 msgr2=0x7f39840a9970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840a94f0 0x7f39840a9970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 -- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39840b25e0 msgr2=0x7f39840b2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f398259c640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39840b25e0 0x7f39840b2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39840b25e0 0x7f39840b2a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 -- 172.21.15.29:0/2836889374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39841f22e0 con 0x7f396c003f40 2023-10-14T16:56:49.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.020+0000 7f3982d9d640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f397400c730 tx=0x7f397400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.021+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39740072d0 con 0x7f396c003f40 2023-10-14T16:56:49.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.021+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3974007430 con 0x7f396c003f40 2023-10-14T16:56:49.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.021+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3974004a20 con 0x7f396c003f40 2023-10-14T16:56:49.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.021+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39841f25c0 con 0x7f396c003f40 2023-10-14T16:56:49.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.021+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39841f2b80 con 0x7f396c003f40 2023-10-14T16:56:49.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.022+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39840b4f10 con 0x7f396c003f40 2023-10-14T16:56:49.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.023+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f397400a5e0 con 0x7f396c003f40 2023-10-14T16:56:49.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.024+0000 7f396b7fe640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 0x7f397006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.025+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f3974088bb0 con 0x7f396c003f40 2023-10-14T16:56:49.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.025+0000 7f398259c640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 0x7f397006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.026+0000 7f398259c640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 0x7f397006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978030000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.029+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3974053bf0 con 0x7f396c003f40 2023-10-14T16:56:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.282+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39840b8550 con 0x7f396c003f40 2023-10-14T16:56:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.283+0000 7f396b7fe640 1 -- 172.21.15.29:0/2836889374 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7f3974018020 con 0x7f396c003f40 2023-10-14T16:56:49.284 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:49.285 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:49.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 msgr2=0x7f397006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 0x7f397006d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3978036000 tx=0x7f3978030000 comp rx=0 tx=0).stop 2023-10-14T16:56:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 msgr2=0x7f3984159490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f397400c730 tx=0x7f397400cbf0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 shutdown_connections 2023-10-14T16:56:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f397006acc0 0x7f397006d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f39840a94f0 0x7f39840a9970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f39840b25e0 0x7f39840b2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 --2- 172.21.15.29:0/2836889374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c003f40 0x7f3984159490 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 >> 172.21.15.29:0/2836889374 conn(0x7f39840cf570 msgr2=0x7f39840d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 shutdown_connections 2023-10-14T16:56:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.286+0000 7f3988947640 1 -- 172.21.15.29:0/2836889374 wait complete. 2023-10-14T16:56:49.307 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.522+0000 7fe4641a9640 1 Processor -- start 2023-10-14T16:56:49.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.522+0000 7fe4641a9640 1 -- start start 2023-10-14T16:56:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe45c0cd0c0 0x7fe45c1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe45c0b0410 0x7fe45c0b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c15d3f0 0x7fe45c15d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c0b0de0 con 0x7fe45c0b0410 2023-10-14T16:56:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe45c15e3e0 con 0x7fe45c0cd0c0 2023-10-14T16:56:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe4641a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0cd4a0 con 0x7fe45c15d3f0 2023-10-14T16:56:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe461744640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe45c0b0410 0x7fe45c0b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe461744640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe45c0b0410 0x7fe45c0b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41782/0 (socket says 172.21.15.29:41782) 2023-10-14T16:56:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.524+0000 7fe461744640 1 -- 172.21.15.29:0/3016325196 learned_addr learned my addr 172.21.15.29:0/3016325196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461744640 1 -- 172.21.15.29:0/3016325196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c15d3f0 msgr2=0x7fe45c15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461744640 1 --2- 172.21.15.29:0/3016325196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c15d3f0 0x7fe45c15d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461744640 1 -- 172.21.15.29:0/3016325196 >> v2:172.21.15.186:3300/0 conn(0x7fe45c0cd0c0 msgr2=0x7fe45c1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461f45640 1 --2- 172.21.15.29:0/3016325196 >> v2:172.21.15.186:3300/0 conn(0x7fe45c0cd0c0 0x7fe45c1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461744640 1 --2- 172.21.15.29:0/3016325196 >> v2:172.21.15.186:3300/0 conn(0x7fe45c0cd0c0 0x7fe45c1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.525+0000 7fe461744640 1 -- 172.21.15.29:0/3016325196 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe45c15e100 con 0x7fe45c0b0410 2023-10-14T16:56:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.526+0000 7fe461744640 1 --2- 172.21.15.29:0/3016325196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c0b0410 0x7fe45c0b08a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0cc4e0 tx=0x7fe44c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c0a71fc1397a64 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.526+0000 7fe460f43640 1 -- 172.21.15.29:0/3016325196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c0317e0 con 0x7fe45c0b0410 2023-10-14T16:56:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe460f43640 1 -- 172.21.15.29:0/3016325196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe44c031940 con 0x7fe45c0b0410 2023-10-14T16:56:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe460f43640 1 -- 172.21.15.29:0/3016325196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe44c031cd0 con 0x7fe45c0b0410 2023-10-14T16:56:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 -- 172.21.15.29:0/3016325196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c0b0410 msgr2=0x7fe45c0b08a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 --2- 172.21.15.29:0/3016325196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c0b0410 0x7fe45c0b08a0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fe45c0cc4e0 tx=0x7fe44c02b880 comp rx=0 tx=0).stop 2023-10-14T16:56:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 -- 172.21.15.29:0/3016325196 shutdown_connections 2023-10-14T16:56:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 --2- 172.21.15.29:0/3016325196 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c15d3f0 0x7fe45c15d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 --2- 172.21.15.29:0/3016325196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c0b0410 0x7fe45c0b08a0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 --2- 172.21.15.29:0/3016325196 >> v2:172.21.15.186:3300/0 conn(0x7fe45c0cd0c0 0x7fe45c1615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 -- 172.21.15.29:0/3016325196 >> 172.21.15.29:0/3016325196 conn(0x7fe45c0cb410 msgr2=0x7fe45c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.527+0000 7fe4641a9640 1 -- 172.21.15.29:0/3016325196 shutdown_connections 2023-10-14T16:56:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.528+0000 7fe4641a9640 1 -- 172.21.15.29:0/3016325196 wait complete. 2023-10-14T16:56:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.529+0000 7fe4641a9640 1 Processor -- start 2023-10-14T16:56:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.530+0000 7fe4641a9640 1 -- start start 2023-10-14T16:56:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.530+0000 7fe4641a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c0b0410 0x7fe45c0b3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.530+0000 7fe4641a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe45c0cd0c0 0x7fe45c0c2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe4641a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe4641a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c1438e0 con 0x7fe45c15d3f0 2023-10-14T16:56:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe4641a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0c5330 con 0x7fe45c0cd0c0 2023-10-14T16:56:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe4641a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe45c0b44a0 con 0x7fe45c0b0410 2023-10-14T16:56:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe462746640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe462746640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41786/0 (socket says 172.21.15.29:41786) 2023-10-14T16:56:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe462746640 1 -- 172.21.15.29:0/1076561225 learned_addr learned my addr 172.21.15.29:0/1076561225 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe461744640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe45c0cd0c0 0x7fe45c0c2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.531+0000 7fe461f45640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c0b0410 0x7fe45c0b3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 -- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c0b0410 msgr2=0x7fe45c0b3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c0b0410 0x7fe45c0b3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 -- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe45c0cd0c0 msgr2=0x7fe45c0c2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe45c0cd0c0 0x7fe45c0c2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 -- 172.21.15.29:0/1076561225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe44c0094d0 con 0x7fe45c15d3f0 2023-10-14T16:56:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe462746640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe45400b970 tx=0x7fe45400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe45400cb40 con 0x7fe45c15d3f0 2023-10-14T16:56:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe45400ccc0 con 0x7fe45c15d3f0 2023-10-14T16:56:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe454014680 con 0x7fe45c15d3f0 2023-10-14T16:56:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe45c0c4290 con 0x7fe45c15d3f0 2023-10-14T16:56:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.532+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45c0c4850 con 0x7fe45c15d3f0 2023-10-14T16:56:49.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.534+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe45c04e160 con 0x7fe45c15d3f0 2023-10-14T16:56:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.535+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 35) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe454007370 con 0x7fe45c15d3f0 2023-10-14T16:56:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.536+0000 7fe4527fc640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 0x7fe44006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.536+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7fe454089500 con 0x7fe45c15d3f0 2023-10-14T16:56:49.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.536+0000 7fe461f45640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 0x7fe44006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.537+0000 7fe461f45640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 0x7fe44006d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe458006fd0 tx=0x7fe458008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:49.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.540+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe454054580 con 0x7fe45c15d3f0 2023-10-14T16:56:49.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.573+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe454054950 con 0x7fe45c15d3f0 2023-10-14T16:56:49.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.777+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe45c0b4910 con 0x7fe45c15d3f0 2023-10-14T16:56:49.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.778+0000 7fe4527fc640 1 -- 172.21.15.29:0/1076561225 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+4916 (secure 0 0 0) 0x7fe454072840 con 0x7fe45c15d3f0 2023-10-14T16:56:49.779 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:49.779 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":32,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27}],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:56:26.105735+0000","modified":"2023-10-14T16:56:29.167922+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6510},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6510":{"gid":6510,"name":"a","rank":0,"incarnation":28,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/3113871431","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3113871431},{"type":"v1","addr":"172.21.15.29:6837","nonce":3113871431}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[9],"metadata_pool":8,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":4}]} 2023-10-14T16:56:49.779 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 32 2023-10-14T16:56:49.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.780+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 msgr2=0x7fe44006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.780+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 0x7fe44006d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe458006fd0 tx=0x7fe458008040 comp rx=0 tx=0).stop 2023-10-14T16:56:49.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.780+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 msgr2=0x7fe45c15c5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:49.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.780+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fe45400b970 tx=0x7fe45400be30 comp rx=0 tx=0).stop 2023-10-14T16:56:49.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 shutdown_connections 2023-10-14T16:56:49.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe44006acf0 0x7fe44006d1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe45c15d3f0 0x7fe45c15c5e0 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe45c0cd0c0 0x7fe45c0c2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 --2- 172.21.15.29:0/1076561225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe45c0b0410 0x7fe45c0b3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:49.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 >> 172.21.15.29:0/1076561225 conn(0x7fe45c0cb410 msgr2=0x7fe45c15b0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:49.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 shutdown_connections 2023-10-14T16:56:49.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:49.781+0000 7fe4641a9640 1 -- 172.21.15.29:0/1076561225 wait complete. 2023-10-14T16:56:49.797 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.018+0000 7f70fc81a640 1 Processor -- start 2023-10-14T16:56:50.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.018+0000 7f70fc81a640 1 -- start start 2023-10-14T16:56:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70fc81a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f70f815b320 0x7f70f80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70fc81a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f815b9d0 con 0x7f70f80ad970 2023-10-14T16:56:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70fc81a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70f815bcb0 con 0x7f70f806c040 2023-10-14T16:56:50.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70fc81a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70f815b7d0 con 0x7f70f815b320 2023-10-14T16:56:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70f80ad970 0x7f70f80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55082/0 (socket says 172.21.15.29:55082) 2023-10-14T16:56:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70effff640 1 -- 172.21.15.29:0/3169022382 learned_addr learned my addr 172.21.15.29:0/3169022382 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:50.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.020+0000 7f70f7fff640 1 --2- 172.21.15.29:0/3169022382 >> v2:172.21.15.29:3300/0 conn(0x7f70f815b320 0x7f70f80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.021+0000 7f70f6ffd640 1 -- 172.21.15.29:0/3169022382 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519272532 0 0) 0x7f70f815b9d0 con 0x7f70f80ad970 2023-10-14T16:56:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.021+0000 7f70f6ffd640 1 -- 172.21.15.29:0/3169022382 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e4003520 con 0x7f70f80ad970 2023-10-14T16:56:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.021+0000 7f70f7fff640 1 -- 172.21.15.29:0/3169022382 >> v1:172.21.15.186:6789/0 conn(0x7f70f806c040 legacy=0x7f70f806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.021+0000 7f70f7fff640 1 -- 172.21.15.29:0/3169022382 >> v1:172.21.15.29:6789/0 conn(0x7f70f80ad970 legacy=0x7f70f80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.021+0000 7f70f7fff640 1 -- 172.21.15.29:0/3169022382 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f815bf90 con 0x7f70f815b320 2023-10-14T16:56:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.022+0000 7f70f7fff640 1 --2- 172.21.15.29:0/3169022382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f815b320 0x7f70f80cd180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009dc0 tx=0x7f70f002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9783d5204256aba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.022+0000 7f70f6ffd640 1 -- 172.21.15.29:0/3169022382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f002cc40 con 0x7f70f815b320 2023-10-14T16:56:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.022+0000 7f70f6ffd640 1 -- 172.21.15.29:0/3169022382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f002cda0 con 0x7f70f815b320 2023-10-14T16:56:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.022+0000 7f70f6ffd640 1 -- 172.21.15.29:0/3169022382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0034660 con 0x7f70f815b320 2023-10-14T16:56:50.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 -- 172.21.15.29:0/3169022382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f815b320 msgr2=0x7f70f80cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 --2- 172.21.15.29:0/3169022382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f815b320 0x7f70f80cd180 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009dc0 tx=0x7f70f002be90 comp rx=0 tx=0).stop 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 -- 172.21.15.29:0/3169022382 shutdown_connections 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 --2- 172.21.15.29:0/3169022382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f815b320 0x7f70f80cd180 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 -- 172.21.15.29:0/3169022382 >> 172.21.15.29:0/3169022382 conn(0x7f70f80ce6a0 msgr2=0x7f70f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 -- 172.21.15.29:0/3169022382 shutdown_connections 2023-10-14T16:56:50.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.023+0000 7f70fc81a640 1 -- 172.21.15.29:0/3169022382 wait complete. 2023-10-14T16:56:50.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.025+0000 7f70fc81a640 1 Processor -- start 2023-10-14T16:56:50.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.025+0000 7f70fc81a640 1 -- start start 2023-10-14T16:56:50.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f80ad970 0x7f70f8155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 0x7f70f80afcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f81564a0 con 0x7f70f806c040 2023-10-14T16:56:50.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80b0510 con 0x7f70f815b320 2023-10-14T16:56:50.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70fc81a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70f80b07f0 con 0x7f70f80ad970 2023-10-14T16:56:50.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41790/0 (socket says 172.21.15.29:41790) 2023-10-14T16:56:50.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.026+0000 7f70f77fe640 1 -- 172.21.15.29:0/2925442191 learned_addr learned my addr 172.21.15.29:0/2925442191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:50.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 -- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f80ad970 msgr2=0x7f70f8155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f7fff640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 0x7f70f80afcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70effff640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f80ad970 0x7f70f8155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f80ad970 0x7f70f8155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 -- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 msgr2=0x7f70f80afcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 0x7f70f80afcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 -- 172.21.15.29:0/2925442191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0009a70 con 0x7f70f806c040 2023-10-14T16:56:50.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f7fff640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 0x7f70f80afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:50.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f77fe640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b3f0 tx=0x7f70e800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8011830 con 0x7f70f806c040 2023-10-14T16:56:50.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8011990 con 0x7f70f806c040 2023-10-14T16:56:50.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70f80adff0 con 0x7f70f806c040 2023-10-14T16:56:50.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.027+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8011d40 con 0x7f70f806c040 2023-10-14T16:56:50.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.028+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70f80ae5b0 con 0x7f70f806c040 2023-10-14T16:56:50.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.029+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70f80ade50 con 0x7f70f806c040 2023-10-14T16:56:50.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.030+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f70e801a400 con 0x7f70f806c040 2023-10-14T16:56:50.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.030+0000 7f70f4ff9640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 0x7f70d806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.031+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f70e8089850 con 0x7f70f806c040 2023-10-14T16:56:50.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.031+0000 7f70effff640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 0x7f70d806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.031+0000 7f70effff640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 0x7f70d806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f70f80c8770 tx=0x7f70e0009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.035+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70f80ade50 con 0x7f70f806c040 2023-10-14T16:56:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.228+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70f80ade50 con 0x7f70f806c040 2023-10-14T16:56:50.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.231+0000 7f70f4ff9640 1 -- 172.21.15.29:0/2925442191 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v84) v1 ==== 74+0+17620 (secure 0 0 0) 0x7f70f80ade50 con 0x7f70f806c040 2023-10-14T16:56:50.232 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:50.232 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":84,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:46.943991+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:56:24.067424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"cephfs_data","create_time":"2023-10-14T16:56:25.117255+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"82","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2640045513":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3061435492":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/34298176":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6833/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3103792448":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3148536323":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/626362321":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/664522093":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/126969037":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6833/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/266566422":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/1855767981":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/3741726235":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/2072219911":"2023-10-15T16:56:40.500646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:50.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.233+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 msgr2=0x7f70d806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.233+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 0x7f70d806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f70f80c8770 tx=0x7f70e0009040 comp rx=0 tx=0).stop 2023-10-14T16:56:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.233+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 msgr2=0x7f70f80aead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f70e800b3f0 tx=0x7f70e800b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 shutdown_connections 2023-10-14T16:56:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f70d806af00 0x7f70d806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70f815b320 0x7f70f80afcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70f80ad970 0x7f70f8155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 --2- 172.21.15.29:0/2925442191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70f806c040 0x7f70f80aead0 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 >> 172.21.15.29:0/2925442191 conn(0x7f70f80ce6a0 msgr2=0x7f70f815dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 shutdown_connections 2023-10-14T16:56:50.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.234+0000 7f70fc81a640 1 -- 172.21.15.29:0/2925442191 wait complete. 2023-10-14T16:56:50.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:56:50.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.467+0000 7f7a27fff640 1 Processor -- start 2023-10-14T16:56:50.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.467+0000 7f7a27fff640 1 -- start start 2023-10-14T16:56:50.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.468+0000 7f7a27fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7a2815cd70 0x7f7a280d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a27fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2815d150 con 0x7f7a280b8480 2023-10-14T16:56:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a27fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a280d1180 con 0x7f7a280cded0 2023-10-14T16:56:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a27fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a280d1460 con 0x7f7a2815cd70 2023-10-14T16:56:50.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a267fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7a280b8480 0x7f7a280cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55090/0 (socket says 172.21.15.29:55090) 2023-10-14T16:56:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a267fc640 1 -- 172.21.15.29:0/2364606239 learned_addr learned my addr 172.21.15.29:0/2364606239 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.469+0000 7f7a26ffd640 1 --2- 172.21.15.29:0/2364606239 >> v2:172.21.15.186:3300/0 conn(0x7f7a2815cd70 0x7f7a280d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417309593 0 0) 0x7f7a2815d150 con 0x7f7a280b8480 2023-10-14T16:56:50.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a14003520 con 0x7f7a280b8480 2023-10-14T16:56:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943522478 0 0) 0x7f7a14003520 con 0x7f7a280b8480 2023-10-14T16:56:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 >> v2:172.21.15.186:3300/0 conn(0x7f7a2815cd70 msgr2=0x7f7a280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 --2- 172.21.15.29:0/2364606239 >> v2:172.21.15.186:3300/0 conn(0x7f7a2815cd70 0x7f7a280d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 >> v1:172.21.15.186:6789/0 conn(0x7f7a280cded0 legacy=0x7f7a280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a280d1740 con 0x7f7a280b8480 2023-10-14T16:56:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7a10001670 con 0x7f7a280b8480 2023-10-14T16:56:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 >> v1:172.21.15.29:6789/0 conn(0x7f7a280b8480 legacy=0x7f7a280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a140039c0 0x7f7a14003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a140047c0 0x7f7a14004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 0x7f7a1400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280d1460 con 0x7f7a14005180 2023-10-14T16:56:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1400c080 con 0x7f7a140047c0 2023-10-14T16:56:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.470+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1400c240 con 0x7f7a140039c0 2023-10-14T16:56:50.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 0x7f7a1400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 -- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a140039c0 msgr2=0x7f7a14003de0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T16:56:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a140039c0 0x7f7a14003de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 -- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a140047c0 msgr2=0x7f7a14004c40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2023-10-14T16:56:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a140047c0 0x7f7a14004c40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.471+0000 7f7a277fe640 1 -- 172.21.15.29:0/2364606239 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14003520 con 0x7f7a14005180 2023-10-14T16:56:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.472+0000 7f7a277fe640 1 -- 172.21.15.29:0/2364606239 reap_dead start 2023-10-14T16:56:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.472+0000 7f7a277fe640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 0x7f7a1400b980 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a20001750 tx=0x7f7a140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aed1290590766c99 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a200052d0 con 0x7f7a14005180 2023-10-14T16:56:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2002e050 con 0x7f7a14005180 2023-10-14T16:56:50.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a25ffb640 1 -- 172.21.15.29:0/2364606239 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a20003830 con 0x7f7a14005180 2023-10-14T16:56:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 -- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 msgr2=0x7f7a1400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 0x7f7a1400b980 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7a20001750 tx=0x7f7a140057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 -- 172.21.15.29:0/2364606239 shutdown_connections 2023-10-14T16:56:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 --2- 172.21.15.29:0/2364606239 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a14005180 0x7f7a1400b980 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 -- 172.21.15.29:0/2364606239 >> 172.21.15.29:0/2364606239 conn(0x7f7a280cf570 msgr2=0x7f7a280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 -- 172.21.15.29:0/2364606239 shutdown_connections 2023-10-14T16:56:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.473+0000 7f7a27fff640 1 -- 172.21.15.29:0/2364606239 wait complete. 2023-10-14T16:56:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.475+0000 7f7a27fff640 1 Processor -- start 2023-10-14T16:56:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.475+0000 7f7a27fff640 1 -- start start 2023-10-14T16:56:50.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a14005180 0x7f7a280a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a280b3b20 0x7f7a280b3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280b44e0 con 0x7f7a281f5590 2023-10-14T16:56:50.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a280aa1b0 con 0x7f7a14005180 2023-10-14T16:56:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a27fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a281f6100 con 0x7f7a280b3b20 2023-10-14T16:56:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a267fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41808/0 (socket says 172.21.15.29:41808) 2023-10-14T16:56:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.476+0000 7f7a267fc640 1 -- 172.21.15.29:0/84048577 learned_addr learned my addr 172.21.15.29:0/84048577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:50.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 -- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a280b3b20 msgr2=0x7f7a280b3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a280b3b20 0x7f7a280b3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 -- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a14005180 msgr2=0x7f7a280a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a14005180 0x7f7a280a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 -- 172.21.15.29:0/84048577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a281f63e0 con 0x7f7a281f5590 2023-10-14T16:56:50.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a267fc640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000c700 tx=0x7f7a1000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.477+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100072d0 con 0x7f7a281f5590 2023-10-14T16:56:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.478+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10007450 con 0x7f7a281f5590 2023-10-14T16:56:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.478+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a100047e0 con 0x7f7a281f5590 2023-10-14T16:56:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.478+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a280ca600 con 0x7f7a281f5590 2023-10-14T16:56:50.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.478+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a280cabc0 con 0x7f7a281f5590 2023-10-14T16:56:50.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.479+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a280d1f40 con 0x7f7a281f5590 2023-10-14T16:56:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.480+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f7a10004960 con 0x7f7a281f5590 2023-10-14T16:56:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.481+0000 7f7a1f7fe640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 0x7f7a0806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.481+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f7a10088c90 con 0x7f7a281f5590 2023-10-14T16:56:50.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.481+0000 7f7a26ffd640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 0x7f7a0806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.482+0000 7f7a26ffd640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 0x7f7a0806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800e110 tx=0x7f7a1800d040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.486+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7a10053970 con 0x7f7a281f5590 2023-10-14T16:56:50.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.727+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f7a2804e9b0 con 0x7f7a281f5590 2023-10-14T16:56:50.729 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:56:50.728+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:56:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.732+0000 7f7a1f7fe640 1 -- 172.21.15.29:0/84048577 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v33) v1 ==== 161+0+0 (secure 0 0 0) 0x7f7a10018090 con 0x7f7a281f5590 2023-10-14T16:56:50.733 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:56:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.735+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 msgr2=0x7f7a0806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.735+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 0x7f7a0806d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7a1800e110 tx=0x7f7a1800d040 comp rx=0 tx=0).stop 2023-10-14T16:56:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.735+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 msgr2=0x7f7a281f5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.735+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f7a1000c700 tx=0x7f7a1000cbc0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 shutdown_connections 2023-10-14T16:56:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7a0806af30 0x7f7a0806d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7a280b3b20 0x7f7a280b3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7a281f5590 0x7f7a281f5a10 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 --2- 172.21.15.29:0/84048577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7a14005180 0x7f7a280a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 >> 172.21.15.29:0/84048577 conn(0x7f7a280cf570 msgr2=0x7f7a280b7ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 shutdown_connections 2023-10-14T16:56:50.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.736+0000 7f7a27fff640 1 -- 172.21.15.29:0/84048577 wait complete. 2023-10-14T16:56:50.744 INFO:tasks.ceph.mds.a.smithi029.stderr: -3696> 2023-10-14T16:56:31.100+0000 7f95d879f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:56:50.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:56:50.798 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:56:50.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.952+0000 7f3a6171d640 1 Processor -- start 2023-10-14T16:56:50.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.953+0000 7f3a6171d640 1 -- start start 2023-10-14T16:56:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.954+0000 7f3a6171d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a5c159ab0 0x7f3a5c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.954+0000 7f3a6171d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c070360 con 0x7f3a5c06c140 2023-10-14T16:56:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.954+0000 7f3a6171d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a5c0cd1f0 con 0x7f3a5c159e90 2023-10-14T16:56:50.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.954+0000 7f3a6171d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a5c15d890 con 0x7f3a5c159ab0 2023-10-14T16:56:50.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3a5c06c140 0x7f3a5c15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40728/0 (socket says 172.21.15.29:40728) 2023-10-14T16:56:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5b7fe640 1 -- 172.21.15.29:0/129222005 learned_addr learned my addr 172.21.15.29:0/129222005 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a5c159ab0 0x7f3a5c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5a7fc640 1 -- 172.21.15.29:0/129222005 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851176937 0 0) 0x7f3a5c070360 con 0x7f3a5c06c140 2023-10-14T16:56:50.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5a7fc640 1 -- 172.21.15.29:0/129222005 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a48005730 con 0x7f3a5c06c140 2023-10-14T16:56:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5affd640 1 -- 172.21.15.29:0/129222005 >> v1:172.21.15.186:6789/0 conn(0x7f3a5c159e90 legacy=0x7f3a5c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5affd640 1 -- 172.21.15.29:0/129222005 >> v1:172.21.15.29:6789/0 conn(0x7f3a5c06c140 legacy=0x7f3a5c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.955+0000 7f3a5affd640 1 -- 172.21.15.29:0/129222005 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a5c0cd4d0 con 0x7f3a5c159ab0 2023-10-14T16:56:50.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a5affd640 1 --2- 172.21.15.29:0/129222005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0ce370 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0097a0 tx=0x7f3a4c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2cdc315fa516af5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a5a7fc640 1 -- 172.21.15.29:0/129222005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c02e070 con 0x7f3a5c159ab0 2023-10-14T16:56:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a5a7fc640 1 -- 172.21.15.29:0/129222005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a4c0315a0 con 0x7f3a5c159ab0 2023-10-14T16:56:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a5a7fc640 1 -- 172.21.15.29:0/129222005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a4c031910 con 0x7f3a5c159ab0 2023-10-14T16:56:50.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a6171d640 1 -- 172.21.15.29:0/129222005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159ab0 msgr2=0x7f3a5c0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.957+0000 7f3a6171d640 1 --2- 172.21.15.29:0/129222005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0ce370 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c0097a0 tx=0x7f3a4c02b840 comp rx=0 tx=0).stop 2023-10-14T16:56:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.958+0000 7f3a6171d640 1 -- 172.21.15.29:0/129222005 shutdown_connections 2023-10-14T16:56:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.958+0000 7f3a6171d640 1 --2- 172.21.15.29:0/129222005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0ce370 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.958+0000 7f3a6171d640 1 -- 172.21.15.29:0/129222005 >> 172.21.15.29:0/129222005 conn(0x7f3a5c0ce9b0 msgr2=0x7f3a5c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:50.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.958+0000 7f3a6171d640 1 -- 172.21.15.29:0/129222005 shutdown_connections 2023-10-14T16:56:50.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.958+0000 7f3a6171d640 1 -- 172.21.15.29:0/129222005 wait complete. 2023-10-14T16:56:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.959+0000 7f3a6171d640 1 Processor -- start 2023-10-14T16:56:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.960+0000 7f3a6171d640 1 -- start start 2023-10-14T16:56:50.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.960+0000 7f3a6171d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a5c06c140 0x7f3a5c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a6171d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a6171d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a6171d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c15a9f0 con 0x7f3a5c159e90 2023-10-14T16:56:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a6171d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c52b0 con 0x7f3a5c159ab0 2023-10-14T16:56:50.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a6171d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a5c0c4d10 con 0x7f3a5c06c140 2023-10-14T16:56:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39098/0 (socket says 172.21.15.29:39098) 2023-10-14T16:56:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 -- 172.21.15.29:0/309750228 learned_addr learned my addr 172.21.15.29:0/309750228 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:50.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a53fff640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 -- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a5c06c140 msgr2=0x7f3a5c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5affd640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a5c06c140 0x7f3a5c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a5c06c140 0x7f3a5c14e0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 -- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 msgr2=0x7f3a5c0c47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:50.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0c47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a5b7fe640 1 -- 172.21.15.29:0/309750228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c009400 con 0x7f3a5c159e90 2023-10-14T16:56:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.961+0000 7f3a53fff640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a5b7fe640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400caa0 tx=0x7f3a5400cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a5400ac00 con 0x7f3a5c159e90 2023-10-14T16:56:50.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a5400ad60 con 0x7f3a5c159e90 2023-10-14T16:56:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a54012620 con 0x7f3a5c159e90 2023-10-14T16:56:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a5c06a2d0 con 0x7f3a5c159e90 2023-10-14T16:56:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.962+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a5c06a890 con 0x7f3a5c159e90 2023-10-14T16:56:50.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.963+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a5c15a750 con 0x7f3a5c159e90 2023-10-14T16:56:50.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.964+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 36) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3a54007070 con 0x7f3a5c159e90 2023-10-14T16:56:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.965+0000 7f3a537fe640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 0x7f3a3c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.965+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f3a54089950 con 0x7f3a5c159e90 2023-10-14T16:56:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.966+0000 7f3a5affd640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 0x7f3a3c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:50.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.966+0000 7f3a5affd640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 0x7f3a3c06d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036000 tx=0x7f3a4c008030 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:50.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:50.970+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a5c15a750 con 0x7f3a5c159e90 2023-10-14T16:56:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.206+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3a5c15ab30 con 0x7f3a5c159e90 2023-10-14T16:56:51.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.745+0000 7f3a537fe640 1 -- 172.21.15.29:0/309750228 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v34) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3a5c15ab30 con 0x7f3a5c159e90 2023-10-14T16:56:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.749+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 msgr2=0x7f3a3c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.749+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 0x7f3a3c06d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f3a4c036000 tx=0x7f3a4c008030 comp rx=0 tx=0).stop 2023-10-14T16:56:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.749+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 msgr2=0x7f3a5c06c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.749+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f3a5400caa0 tx=0x7f3a5400cf60 comp rx=0 tx=0).stop 2023-10-14T16:56:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 shutdown_connections 2023-10-14T16:56:51.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f3a3c06af30 0x7f3a3c06d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a5c159e90 0x7f3a5c06c570 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a5c159ab0 0x7f3a5c0c47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 --2- 172.21.15.29:0/309750228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a5c06c140 0x7f3a5c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.750+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 >> 172.21.15.29:0/309750228 conn(0x7f3a5c0ce9b0 msgr2=0x7f3a5c06b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.751+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 shutdown_connections 2023-10-14T16:56:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:51.751+0000 7f3a6171d640 1 -- 172.21.15.29:0/309750228 wait complete. 2023-10-14T16:56:51.775 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:56:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.052+0000 7f44b7a83640 1 Processor -- start 2023-10-14T16:56:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.052+0000 7f44b7a83640 1 -- start start 2023-10-14T16:56:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00cb8d0 0x7f44b0161650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f44b00ae9d0 0x7f44b00ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44b0069da0 0x7f44b006a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b006a770 con 0x7f44b0069da0 2023-10-14T16:56:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44b01611c0 con 0x7f44b00ae9d0 2023-10-14T16:56:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.053+0000 7f44b7a83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b00cf270 con 0x7f44b00cb8d0 2023-10-14T16:56:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.054+0000 7f44b6020640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44b0069da0 0x7f44b006a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.054+0000 7f44b6020640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f44b0069da0 0x7f44b006a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39102/0 (socket says 172.21.15.29:39102) 2023-10-14T16:56:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.054+0000 7f44b6020640 1 -- 172.21.15.29:0/3031669649 learned_addr learned my addr 172.21.15.29:0/3031669649 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.054+0000 7f44b581f640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00cb8d0 0x7f44b0161650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.054+0000 7f44b501e640 1 --2- 172.21.15.29:0/3031669649 >> v2:172.21.15.186:3300/0 conn(0x7f44b00ae9d0 0x7f44b00ced30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.055+0000 7f44b6020640 1 -- 172.21.15.29:0/3031669649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00cb8d0 msgr2=0x7f44b0161650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.055+0000 7f44b6020640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00cb8d0 0x7f44b0161650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.055+0000 7f44b6020640 1 -- 172.21.15.29:0/3031669649 >> v2:172.21.15.186:3300/0 conn(0x7f44b00ae9d0 msgr2=0x7f44b00ced30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.055+0000 7f44b6020640 1 --2- 172.21.15.29:0/3031669649 >> v2:172.21.15.186:3300/0 conn(0x7f44b00ae9d0 0x7f44b00ced30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.055+0000 7f44b6020640 1 -- 172.21.15.29:0/3031669649 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44b00cf7a0 con 0x7f44b0069da0 2023-10-14T16:56:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.056+0000 7f44b6020640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b006a230 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f44b0070080 tx=0x7f44a802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff1b90ead923fcbe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.056+0000 7f44b481d640 1 -- 172.21.15.29:0/3031669649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a802ca80 con 0x7f44b0069da0 2023-10-14T16:56:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.056+0000 7f44b481d640 1 -- 172.21.15.29:0/3031669649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a802cbe0 con 0x7f44b0069da0 2023-10-14T16:56:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.056+0000 7f44b481d640 1 -- 172.21.15.29:0/3031669649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a80343f0 con 0x7f44b0069da0 2023-10-14T16:56:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 -- 172.21.15.29:0/3031669649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 msgr2=0x7f44b006a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b006a230 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f44b0070080 tx=0x7f44a802bee0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 -- 172.21.15.29:0/3031669649 shutdown_connections 2023-10-14T16:56:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b006a230 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 --2- 172.21.15.29:0/3031669649 >> v2:172.21.15.186:3300/0 conn(0x7f44b00ae9d0 0x7f44b00ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 --2- 172.21.15.29:0/3031669649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00cb8d0 0x7f44b0161650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 -- 172.21.15.29:0/3031669649 >> 172.21.15.29:0/3031669649 conn(0x7f44b00cc470 msgr2=0x7f44b00cc850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 -- 172.21.15.29:0/3031669649 shutdown_connections 2023-10-14T16:56:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.057+0000 7f44b7a83640 1 -- 172.21.15.29:0/3031669649 wait complete. 2023-10-14T16:56:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.059+0000 7f44b7a83640 1 Processor -- start 2023-10-14T16:56:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.059+0000 7f44b7a83640 1 -- start start 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00ae9d0 0x7f44b01f1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 0x7f44b00c6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b00c8930 con 0x7f44b0069da0 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44b00b1910 con 0x7f44b00cb8d0 2023-10-14T16:56:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b7a83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44b01529b0 con 0x7f44b00ae9d0 2023-10-14T16:56:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.060+0000 7f44b581f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39110/0 (socket says 172.21.15.29:39110) 2023-10-14T16:56:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 -- 172.21.15.29:0/960885773 learned_addr learned my addr 172.21.15.29:0/960885773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b501e640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00ae9d0 0x7f44b01f1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b6020640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 0x7f44b00c6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 -- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00ae9d0 msgr2=0x7f44b01f1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00ae9d0 0x7f44b01f1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 -- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 msgr2=0x7f44b00c6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 0x7f44b00c6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 -- 172.21.15.29:0/960885773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a8009b30 con 0x7f44b0069da0 2023-10-14T16:56:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b6020640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 0x7f44b00c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.061+0000 7f44b581f640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e9e0 tx=0x7f44ac00eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.062+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac009670 con 0x7f44b0069da0 2023-10-14T16:56:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.062+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ac00c9d0 con 0x7f44b0069da0 2023-10-14T16:56:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.062+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44ac00cd80 con 0x7f44b0069da0 2023-10-14T16:56:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.062+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44b00b1630 con 0x7f44b0069da0 2023-10-14T16:56:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.062+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44b00c8e40 con 0x7f44b0069da0 2023-10-14T16:56:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.063+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44b004e160 con 0x7f44b0069da0 2023-10-14T16:56:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.064+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f44ac013470 con 0x7f44b0069da0 2023-10-14T16:56:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.065+0000 7f44a67fc640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 0x7f449406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.065+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f44ac089ce0 con 0x7f44b0069da0 2023-10-14T16:56:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.065+0000 7f44b501e640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 0x7f449406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.066+0000 7f44b501e640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 0x7f449406d4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44b01535f0 tx=0x7f44a0008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.070+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44ac054930 con 0x7f44b0069da0 2023-10-14T16:56:52.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.277+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f44b00cd7f0 con 0x7f44b0069da0 2023-10-14T16:56:52.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.749+0000 7f44a67fc640 1 -- 172.21.15.29:0/960885773 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v86) v1 ==== 181+0+0 (secure 0 0 0) 0x7f44ac057f50 con 0x7f44b0069da0 2023-10-14T16:56:52.750 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:56:52.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 msgr2=0x7f449406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 0x7f449406d4a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f44b01535f0 tx=0x7f44a0008040 comp rx=0 tx=0).stop 2023-10-14T16:56:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 msgr2=0x7f44b00c3f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f44ac00e9e0 tx=0x7f44ac00eea0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 shutdown_connections 2023-10-14T16:56:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f449406afe0 0x7f449406d4a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44b00cb8d0 0x7f44b00c6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44b00ae9d0 0x7f44b01f1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 --2- 172.21.15.29:0/960885773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44b0069da0 0x7f44b00c3f10 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.753+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 >> 172.21.15.29:0/960885773 conn(0x7f44b00cc470 msgr2=0x7f44b006d730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.754+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 shutdown_connections 2023-10-14T16:56:52.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:52.755+0000 7f44b7a83640 1 -- 172.21.15.29:0/960885773 wait complete. 2023-10-14T16:56:52.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:56:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.016+0000 7f84167b0640 1 Processor -- start 2023-10-14T16:56:53.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.017+0000 7f84167b0640 1 -- start start 2023-10-14T16:56:53.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.019+0000 7f84167b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 0x7f84100c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.019+0000 7f84167b0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100ce390 con 0x7f84100cbe50 2023-10-14T16:56:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.019+0000 7f84167b0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84100ce670 con 0x7f8410158db0 2023-10-14T16:56:53.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.019+0000 7f84167b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100ce180 con 0x7f84100adde0 2023-10-14T16:56:53.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.020+0000 7f840ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84100cbe50 0x7f84100caba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40734/0 (socket says 172.21.15.29:40734) 2023-10-14T16:56:53.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.020+0000 7f840ffff640 1 -- 172.21.15.29:0/3881277650 learned_addr learned my addr 172.21.15.29:0/3881277650 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:53.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.020+0000 7f840f7fe640 1 --2- 172.21.15.29:0/3881277650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 0x7f84100c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:53.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.021+0000 7f840effd640 1 -- 172.21.15.29:0/3881277650 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4124233133 0 0) 0x7f84100ce390 con 0x7f84100cbe50 2023-10-14T16:56:53.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.021+0000 7f840effd640 1 -- 172.21.15.29:0/3881277650 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83f8003520 con 0x7f84100cbe50 2023-10-14T16:56:53.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.022+0000 7f840f7fe640 1 -- 172.21.15.29:0/3881277650 >> v1:172.21.15.186:6789/0 conn(0x7f8410158db0 legacy=0x7f84100b0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:53.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.022+0000 7f840f7fe640 1 -- 172.21.15.29:0/3881277650 >> v1:172.21.15.29:6789/0 conn(0x7f84100cbe50 legacy=0x7f84100caba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.022+0000 7f840f7fe640 1 -- 172.21.15.29:0/3881277650 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f841006ba70 con 0x7f84100adde0 2023-10-14T16:56:53.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.024+0000 7f840f7fe640 1 --2- 172.21.15.29:0/3881277650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 0x7f84100c92f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84000097d0 tx=0x7f840002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebe60c0ee8aeed72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.025+0000 7f840effd640 1 -- 172.21.15.29:0/3881277650 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840002d040 con 0x7f84100adde0 2023-10-14T16:56:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.025+0000 7f840effd640 1 -- 172.21.15.29:0/3881277650 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8400031810 con 0x7f84100adde0 2023-10-14T16:56:53.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.025+0000 7f840effd640 1 -- 172.21.15.29:0/3881277650 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8400031ba0 con 0x7f84100adde0 2023-10-14T16:56:53.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 -- 172.21.15.29:0/3881277650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 msgr2=0x7f84100c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 --2- 172.21.15.29:0/3881277650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 0x7f84100c92f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f84000097d0 tx=0x7f840002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 -- 172.21.15.29:0/3881277650 shutdown_connections 2023-10-14T16:56:53.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 --2- 172.21.15.29:0/3881277650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100adde0 0x7f84100c92f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 -- 172.21.15.29:0/3881277650 >> 172.21.15.29:0/3881277650 conn(0x7f84100cb360 msgr2=0x7f84100cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:53.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 -- 172.21.15.29:0/3881277650 shutdown_connections 2023-10-14T16:56:53.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.026+0000 7f84167b0640 1 -- 172.21.15.29:0/3881277650 wait complete. 2023-10-14T16:56:53.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.028+0000 7f84167b0640 1 Processor -- start 2023-10-14T16:56:53.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.028+0000 7f84167b0640 1 -- start start 2023-10-14T16:56:53.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:53.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 0x7f84100c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:53.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 0x7f841015bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:53.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100b2b30 con 0x7f84100adde0 2023-10-14T16:56:53.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84100c68b0 con 0x7f8410158db0 2023-10-14T16:56:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f84167b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84100c6b40 con 0x7f84100cbe50 2023-10-14T16:56:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39126/0 (socket says 172.21.15.29:39126) 2023-10-14T16:56:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 -- 172.21.15.29:0/4273096124 learned_addr learned my addr 172.21.15.29:0/4273096124 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:53.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840f7fe640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 0x7f84100c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f8414d4d640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 0x7f841015bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 -- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 msgr2=0x7f84100c6370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 0x7f84100c6370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 -- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 msgr2=0x7f841015bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 0x7f841015bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.029+0000 7f840ffff640 1 -- 172.21.15.29:0/4273096124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8400009400 con 0x7f84100adde0 2023-10-14T16:56:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f840f7fe640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 0x7f84100c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f8414d4d640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 0x7f841015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f840ffff640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f840400c430 tx=0x7f840400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:53.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404005ce0 con 0x7f84100adde0 2023-10-14T16:56:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8404005e40 con 0x7f84100adde0 2023-10-14T16:56:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84040072f0 con 0x7f84100adde0 2023-10-14T16:56:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84100c6e20 con 0x7f84100adde0 2023-10-14T16:56:53.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.030+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84100b2d80 con 0x7f84100adde0 2023-10-14T16:56:53.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.032+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84100c7340 con 0x7f84100adde0 2023-10-14T16:56:53.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.032+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8404007450 con 0x7f84100adde0 2023-10-14T16:56:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.034+0000 7f840cff9640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 0x7f83fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.034+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 6089+0+0 (secure 0 0 0) 0x7f8404088fe0 con 0x7f84100adde0 2023-10-14T16:56:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.034+0000 7f840f7fe640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 0x7f83fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.035+0000 7f840f7fe640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 0x7f83fc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8400036000 tx=0x7f840003c040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:53.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.038+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8404053dc0 con 0x7f84100adde0 2023-10-14T16:56:53.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.239+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f84100b0600 con 0x7f84100adde0 2023-10-14T16:56:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.761+0000 7f840cff9640 1 -- 172.21.15.29:0/4273096124 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v87) v1 ==== 169+0+0 (secure 0 0 0) 0x7f84040170d0 con 0x7f84100adde0 2023-10-14T16:56:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:56:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.767+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 msgr2=0x7f83fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.767+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 0x7f83fc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8400036000 tx=0x7f840003c040 comp rx=0 tx=0).stop 2023-10-14T16:56:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.767+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 msgr2=0x7f841014cb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.767+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f840400c430 tx=0x7f840400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 shutdown_connections 2023-10-14T16:56:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f83fc06af90 0x7f83fc06d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8410158db0 0x7f841015bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84100cbe50 0x7f84100c6370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 --2- 172.21.15.29:0/4273096124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84100adde0 0x7f841014cb90 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.768+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 >> 172.21.15.29:0/4273096124 conn(0x7f84100cb360 msgr2=0x7f841006fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.769+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 shutdown_connections 2023-10-14T16:56:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:53.769+0000 7f84167b0640 1 -- 172.21.15.29:0/4273096124 wait complete. 2023-10-14T16:56:53.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T16:56:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.007+0000 7faab45dc640 1 Processor -- start 2023-10-14T16:56:54.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.008+0000 7faab45dc640 1 -- start start 2023-10-14T16:56:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.009+0000 7faab45dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faaac15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.009+0000 7faab45dc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faaac0ce390 0x7faaac070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.009+0000 7faab45dc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaac0707e0 con 0x7faaac15aaa0 2023-10-14T16:56:54.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.009+0000 7faab45dc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaac06b7a0 con 0x7faaac0ce390 2023-10-14T16:56:54.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.009+0000 7faab45dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaac15ef20 con 0x7faaac15ae80 2023-10-14T16:56:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab2378640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faaac15aaa0 0x7faaac0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40736/0 (socket says 172.21.15.29:40736) 2023-10-14T16:56:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab2378640 1 -- 172.21.15.29:0/2080566703 learned_addr learned my addr 172.21.15.29:0/2080566703 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873320612 0 0) 0x7faaac0707e0 con 0x7faaac15aaa0 2023-10-14T16:56:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa98003520 con 0x7faaac15aaa0 2023-10-14T16:56:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab2b79640 1 --2- 172.21.15.29:0/2080566703 >> v2:172.21.15.186:3300/0 conn(0x7faaac0ce390 0x7faaac070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.010+0000 7faab1b77640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faaac15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1672516971 0 0) 0x7faa98003520 con 0x7faaac15aaa0 2023-10-14T16:56:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 msgr2=0x7faaac15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faaac15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 >> v2:172.21.15.186:3300/0 conn(0x7faaac0ce390 msgr2=0x7faaac070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 --2- 172.21.15.29:0/2080566703 >> v2:172.21.15.186:3300/0 conn(0x7faaac0ce390 0x7faaac070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac15f200 con 0x7faaac15aaa0 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7faaa8001670 con 0x7faaac15aaa0 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 >> v1:172.21.15.29:6789/0 conn(0x7faaac15aaa0 legacy=0x7faaac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faa98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faa980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa98004d00 0x7faa98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac06b7a0 con 0x7faa98003970 2023-10-14T16:56:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac15ef20 con 0x7faa98004d00 2023-10-14T16:56:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa9800b9b0 con 0x7faaac15ae80 2023-10-14T16:56:54.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.011+0000 7faab2378640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faa98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab1b77640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faa980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2b79640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa98004d00 0x7faa98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2378640 1 -- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 msgr2=0x7faa980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2378640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faa980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2378640 1 -- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa98004d00 msgr2=0x7faa98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2378640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa98004d00 0x7faa98005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2378640 1 -- 172.21.15.29:0/2080566703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa98003520 con 0x7faa98003970 2023-10-14T16:56:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab2b79640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa98004d00 0x7faa98005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:54.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.012+0000 7faab1b77640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac15ae80 0x7faa980047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:54.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.013+0000 7faab2378640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faa98003d90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c160 tx=0x7faa980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4d1a4cbebf07b93 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.013+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8005ad0 con 0x7faa98003970 2023-10-14T16:56:54.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.013+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa8005c30 con 0x7faa98003970 2023-10-14T16:56:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.013+0000 7faab1376640 1 -- 172.21.15.29:0/2080566703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa802c830 con 0x7faa98003970 2023-10-14T16:56:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 -- 172.21.15.29:0/2080566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 msgr2=0x7faa98003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 --2- 172.21.15.29:0/2080566703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faa98003d90 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c160 tx=0x7faa980057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab2b79640 1 -- 172.21.15.29:0/2080566703 reap_dead start 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 -- 172.21.15.29:0/2080566703 shutdown_connections 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 -- 172.21.15.29:0/2080566703 >> 172.21.15.29:0/2080566703 conn(0x7faaac0cb410 msgr2=0x7faaac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 -- 172.21.15.29:0/2080566703 shutdown_connections 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.014+0000 7faab45dc640 1 -- 172.21.15.29:0/2080566703 wait complete. 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.016+0000 7faab45dc640 1 Processor -- start 2023-10-14T16:56:54.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.016+0000 7faab45dc640 1 -- start start 2023-10-14T16:56:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac070e50 0x7faaac0712d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac0c7860 0x7faaac0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac071810 con 0x7faa98003970 2023-10-14T16:56:54.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaac14a6b0 con 0x7faaac070e50 2023-10-14T16:56:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab45dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaac14ab60 con 0x7faaac0c7860 2023-10-14T16:56:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab2378640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab2378640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39150/0 (socket says 172.21.15.29:39150) 2023-10-14T16:56:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.017+0000 7faab2378640 1 -- 172.21.15.29:0/3654495422 learned_addr learned my addr 172.21.15.29:0/3654495422 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:54.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab1b77640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac070e50 0x7faaac0712d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2b79640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac0c7860 0x7faaac0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 -- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac0c7860 msgr2=0x7faaac0c7ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac0c7860 0x7faaac0c7ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 -- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac070e50 msgr2=0x7faaac0712d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac070e50 0x7faaac0712d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 -- 172.21.15.29:0/3654495422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaac0c86b0 con 0x7faa98003970 2023-10-14T16:56:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab2378640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c280 tx=0x7faaa8000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8039070 con 0x7faa98003970 2023-10-14T16:56:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaa802cc60 con 0x7faa98003970 2023-10-14T16:56:54.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaa8007070 con 0x7faa98003970 2023-10-14T16:56:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.018+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaac0c8990 con 0x7faa98003970 2023-10-14T16:56:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.019+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaac14adb0 con 0x7faa98003970 2023-10-14T16:56:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.020+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaac14af40 con 0x7faa98003970 2023-10-14T16:56:54.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.021+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faaa8032070 con 0x7faa98003970 2023-10-14T16:56:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.022+0000 7faaa2ffd640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.022+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7faaa803c040 con 0x7faa98003970 2023-10-14T16:56:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.022+0000 7faab1b77640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.023+0000 7faab1b77640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 0x7faa9006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faaac0c4a80 tx=0x7faa9c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.026+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faaa8075950 con 0x7faa98003970 2023-10-14T16:56:54.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.234+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7faaac04e160 con 0x7faa98003970 2023-10-14T16:56:54.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.242+0000 7faaa2ffd640 1 -- 172.21.15.29:0/3654495422 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v25) v1 ==== 115+0+0 (secure 0 0 0) 0x7faaa8078f70 con 0x7faa98003970 2023-10-14T16:56:54.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 msgr2=0x7faa9006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 0x7faa9006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faaac0c4a80 tx=0x7faa9c008040 comp rx=0 tx=0).stop 2023-10-14T16:56:54.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 msgr2=0x7faaac1efce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7faaa800c280 tx=0x7faaa8000fe0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 shutdown_connections 2023-10-14T16:56:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7faa9006af90 0x7faa9006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaac0c7860 0x7faaac0c7ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaac070e50 0x7faaac0712d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 --2- 172.21.15.29:0/3654495422 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa98003970 0x7faaac1efce0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 >> 172.21.15.29:0/3654495422 conn(0x7faaac0cb410 msgr2=0x7faaac15a000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:54.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.245+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 shutdown_connections 2023-10-14T16:56:54.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.246+0000 7faab45dc640 1 -- 172.21.15.29:0/3654495422 wait complete. 2023-10-14T16:56:54.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T16:56:54.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.489+0000 7f34ce116640 1 Processor -- start 2023-10-14T16:56:54.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.489+0000 7f34ce116640 1 -- start start 2023-10-14T16:56:54.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34ce116640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34c8159950 0x7f34c815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34ce116640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f34c8070450 0x7f34c80708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34ce116640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c815aef0 con 0x7f34c806c100 2023-10-14T16:56:54.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34ce116640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c8070e20 con 0x7f34c8159950 2023-10-14T16:56:54.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34ce116640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f34c815e490 con 0x7f34c8070450 2023-10-14T16:56:54.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34cd114640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34c8159950 0x7f34c815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34cd114640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f34c8159950 0x7f34c815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39162/0 (socket says 172.21.15.29:39162) 2023-10-14T16:56:54.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.491+0000 7f34cd114640 1 -- 172.21.15.29:0/4150335441 learned_addr learned my addr 172.21.15.29:0/4150335441 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:54.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.492+0000 7f34cd114640 1 -- 172.21.15.29:0/4150335441 >> v2:172.21.15.186:3300/0 conn(0x7f34c8070450 msgr2=0x7f34c80708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.492+0000 7f34cd114640 1 --2- 172.21.15.29:0/4150335441 >> v2:172.21.15.186:3300/0 conn(0x7f34c8070450 0x7f34c80708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.492+0000 7f34cd114640 1 -- 172.21.15.29:0/4150335441 >> v1:172.21.15.186:6789/0 conn(0x7f34c806c100 legacy=0x7f34c806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.492+0000 7f34cd114640 1 -- 172.21.15.29:0/4150335441 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c815d910 con 0x7f34c8159950 2023-10-14T16:56:54.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.493+0000 7f34cd114640 1 --2- 172.21.15.29:0/4150335441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8159950 0x7f34c815a9b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f34c80cab70 tx=0x7f34b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cff6d37b6b94ba02 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.493+0000 7f34c7fff640 1 -- 172.21.15.29:0/4150335441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b80317e0 con 0x7f34c8159950 2023-10-14T16:56:54.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.493+0000 7f34c7fff640 1 -- 172.21.15.29:0/4150335441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34b8031940 con 0x7f34c8159950 2023-10-14T16:56:54.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.493+0000 7f34c7fff640 1 -- 172.21.15.29:0/4150335441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b8031cd0 con 0x7f34c8159950 2023-10-14T16:56:54.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 -- 172.21.15.29:0/4150335441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8159950 msgr2=0x7f34c815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 --2- 172.21.15.29:0/4150335441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8159950 0x7f34c815a9b0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f34c80cab70 tx=0x7f34b802b880 comp rx=0 tx=0).stop 2023-10-14T16:56:54.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 -- 172.21.15.29:0/4150335441 shutdown_connections 2023-10-14T16:56:54.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 --2- 172.21.15.29:0/4150335441 >> v2:172.21.15.186:3300/0 conn(0x7f34c8070450 0x7f34c80708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 --2- 172.21.15.29:0/4150335441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8159950 0x7f34c815a9b0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 -- 172.21.15.29:0/4150335441 >> 172.21.15.29:0/4150335441 conn(0x7f34c80ce6a0 msgr2=0x7f34c80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:54.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 -- 172.21.15.29:0/4150335441 shutdown_connections 2023-10-14T16:56:54.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.494+0000 7f34ce116640 1 -- 172.21.15.29:0/4150335441 wait complete. 2023-10-14T16:56:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.496+0000 7f34ce116640 1 Processor -- start 2023-10-14T16:56:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.496+0000 7f34ce116640 1 -- start start 2023-10-14T16:56:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c806c100 0x7f34c80c31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 0x7f34c815c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80c21b0 con 0x7f34c8070450 2023-10-14T16:56:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80c2400 con 0x7f34c806c100 2023-10-14T16:56:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34ce116640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34c80c26e0 con 0x7f34c8159950 2023-10-14T16:56:54.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cc913640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cc913640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39164/0 (socket says 172.21.15.29:39164) 2023-10-14T16:56:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cc913640 1 -- 172.21.15.29:0/1771745022 learned_addr learned my addr 172.21.15.29:0/1771745022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cd114640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c806c100 0x7f34c80c31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cd915640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 0x7f34c815c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cc913640 1 -- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 msgr2=0x7f34c815c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.497+0000 7f34cc913640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 0x7f34c815c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34cc913640 1 -- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c806c100 msgr2=0x7f34c80c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34cc913640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c806c100 0x7f34c80c31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34cc913640 1 -- 172.21.15.29:0/1771745022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b80094d0 con 0x7f34c8070450 2023-10-14T16:56:54.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34cd915640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 0x7f34c815c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34cc913640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b4a0 tx=0x7f34bc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc010610 con 0x7f34c8070450 2023-10-14T16:56:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34bc010770 con 0x7f34c8070450 2023-10-14T16:56:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34bc010b20 con 0x7f34c8070450 2023-10-14T16:56:54.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c80c29c0 con 0x7f34c8070450 2023-10-14T16:56:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.498+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c8157c50 con 0x7f34c8070450 2023-10-14T16:56:54.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.499+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c80c2ee0 con 0x7f34c8070450 2023-10-14T16:56:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.500+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f34bc010c80 con 0x7f34c8070450 2023-10-14T16:56:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.501+0000 7f34c5ffb640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 0x7f34b406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.501+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f34bc089340 con 0x7f34c8070450 2023-10-14T16:56:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.502+0000 7f34cd114640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 0x7f34b406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.502+0000 7f34cd114640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 0x7f34b406d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34b8036000 tx=0x7f34b803d040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.506+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34c80c2ee0 con 0x7f34c8070450 2023-10-14T16:56:54.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.701+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f34c80c2ee0 con 0x7f34c8070450 2023-10-14T16:56:54.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.707+0000 7f34c5ffb640 1 -- 172.21.15.29:0/1771745022 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v26) v1 ==== 120+0+0 (secure 0 0 0) 0x7f34c80c2ee0 con 0x7f34c8070450 2023-10-14T16:56:54.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.710+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 msgr2=0x7f34b406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.710+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 0x7f34b406d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f34b8036000 tx=0x7f34b803d040 comp rx=0 tx=0).stop 2023-10-14T16:56:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.710+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 msgr2=0x7f34c8157710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.710+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f34bc00b4a0 tx=0x7f34bc00b960 comp rx=0 tx=0).stop 2023-10-14T16:56:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 shutdown_connections 2023-10-14T16:56:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f34b406afc0 0x7f34b406d480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34c8159950 0x7f34c815c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34c8070450 0x7f34c8157710 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 --2- 172.21.15.29:0/1771745022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f34c806c100 0x7f34c80c31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 >> 172.21.15.29:0/1771745022 conn(0x7f34c80ce6a0 msgr2=0x7f34c815a6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 shutdown_connections 2023-10-14T16:56:54.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.711+0000 7f34ce116640 1 -- 172.21.15.29:0/1771745022 wait complete. 2023-10-14T16:56:54.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T16:56:54.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.961+0000 7f6cf514e640 1 Processor -- start 2023-10-14T16:56:54.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.961+0000 7f6cf514e640 1 -- start start 2023-10-14T16:56:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.963+0000 7f6cf514e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6cf0159290 0x7f6cf00cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.963+0000 7f6cf514e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.963+0000 7f6cf514e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf00ca580 con 0x7f6cf006fa20 2023-10-14T16:56:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.963+0000 7f6cf514e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf00ad5e0 con 0x7f6cf0159290 2023-10-14T16:56:54.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.963+0000 7f6cf514e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf00ca2a0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:44740/0 (socket says 172.21.15.29:44740) 2023-10-14T16:56:54.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 -- 172.21.15.29:0/2992680256 learned_addr learned my addr 172.21.15.29:0/2992680256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6ceffff640 1 --2- 172.21.15.29:0/2992680256 >> v2:172.21.15.186:3300/0 conn(0x7f6cf0159290 0x7f6cf00cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 -- 172.21.15.29:0/2992680256 >> v2:172.21.15.186:3300/0 conn(0x7f6cf0159290 msgr2=0x7f6cf00cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 --2- 172.21.15.29:0/2992680256 >> v2:172.21.15.186:3300/0 conn(0x7f6cf0159290 0x7f6cf00cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 -- 172.21.15.29:0/2992680256 >> v1:172.21.15.186:6789/0 conn(0x7f6cf006fa20 legacy=0x7f6cf00c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.964+0000 7f6cf494d640 1 -- 172.21.15.29:0/2992680256 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf00acf20 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.966+0000 7f6cf494d640 1 --2- 172.21.15.29:0/2992680256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800bb30 tx=0x7f6ce802df10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcc49426e8a63b37 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cef7fe640 1 -- 172.21.15.29:0/2992680256 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce802ecc0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cef7fe640 1 -- 172.21.15.29:0/2992680256 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8032070 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cef7fe640 1 -- 172.21.15.29:0/2992680256 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8038620 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 -- 172.21.15.29:0/2992680256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 msgr2=0x7f6cf00ce0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 --2- 172.21.15.29:0/2992680256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800bb30 tx=0x7f6ce802df10 comp rx=0 tx=0).stop 2023-10-14T16:56:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 -- 172.21.15.29:0/2992680256 shutdown_connections 2023-10-14T16:56:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 --2- 172.21.15.29:0/2992680256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf00c9cf0 0x7f6cf00ce0b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 --2- 172.21.15.29:0/2992680256 >> v2:172.21.15.186:3300/0 conn(0x7f6cf0159290 0x7f6cf00cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 -- 172.21.15.29:0/2992680256 >> 172.21.15.29:0/2992680256 conn(0x7f6cf00b0a80 msgr2=0x7f6cf00b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 -- 172.21.15.29:0/2992680256 shutdown_connections 2023-10-14T16:56:54.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.967+0000 7f6cf514e640 1 -- 172.21.15.29:0/2992680256 wait complete. 2023-10-14T16:56:54.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.969+0000 7f6cf514e640 1 Processor -- start 2023-10-14T16:56:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.969+0000 7f6cf514e640 1 -- start start 2023-10-14T16:56:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf006fa20 0x7f6cf01f56e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf0159290 0x7f6cf0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf01f3eb0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf006cd80 con 0x7f6cf006fa20 2023-10-14T16:56:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.970+0000 7f6cf514e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf015a000 con 0x7f6cf0159290 2023-10-14T16:56:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39172/0 (socket says 172.21.15.29:39172) 2023-10-14T16:56:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 -- 172.21.15.29:0/1658624599 learned_addr learned my addr 172.21.15.29:0/1658624599 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ceffff640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf006fa20 0x7f6cf01f56e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6cf494d640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf0159290 0x7f6cf0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 -- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf0159290 msgr2=0x7f6cf0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf0159290 0x7f6cf0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 -- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf006fa20 msgr2=0x7f6cf01f56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:54.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf006fa20 0x7f6cf01f56e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:54.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.971+0000 7f6ce77fe640 1 -- 172.21.15.29:0/1658624599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce800b760 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6ce77fe640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c450 tx=0x7f6cd800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd80052d0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd8005430 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd8012870 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf015a2e0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.972+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf015a8a0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.973+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf004e9b0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.974+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6cd80129d0 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.976+0000 7f6ced7fa640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 0x7f6cd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:54.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.976+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f6cd8089350 con 0x7f6cf00c9cf0 2023-10-14T16:56:54.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.976+0000 7f6ceffff640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 0x7f6cd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:54.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.977+0000 7f6ceffff640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 0x7f6cd006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0154a50 tx=0x7f6ce000b040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:54.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:54.980+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6cd8054270 con 0x7f6cf00c9cf0 2023-10-14T16:56:55.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.178+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6cf00c5910 con 0x7f6cf00c9cf0 2023-10-14T16:56:55.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.185+0000 7f6ced7fa640 1 -- 172.21.15.29:0/1658624599 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v27) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6cd8057890 con 0x7f6cf00c9cf0 2023-10-14T16:56:55.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 msgr2=0x7f6cd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 0x7f6cd006d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6cf0154a50 tx=0x7f6ce000b040 comp rx=0 tx=0).stop 2023-10-14T16:56:55.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 msgr2=0x7f6cf01f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f6cd800c450 tx=0x7f6cd800c910 comp rx=0 tx=0).stop 2023-10-14T16:56:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 shutdown_connections 2023-10-14T16:56:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6cd006af90 0x7f6cd006d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6cf0159290 0x7f6cf0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf00c9cf0 0x7f6cf01f3970 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 --2- 172.21.15.29:0/1658624599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf006fa20 0x7f6cf01f56e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 >> 172.21.15.29:0/1658624599 conn(0x7f6cf00b0a80 msgr2=0x7f6cf0158760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 shutdown_connections 2023-10-14T16:56:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.188+0000 7f6cf514e640 1 -- 172.21.15.29:0/1658624599 wait complete. 2023-10-14T16:56:55.208 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule' 2023-10-14T16:56:55.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.437+0000 7ff29f015640 1 Processor -- start 2023-10-14T16:56:55.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.437+0000 7ff29f015640 1 -- start start 2023-10-14T16:56:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.438+0000 7ff29f015640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29806c3f0 0x7ff29806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.438+0000 7ff29f015640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff29815e0c0 0x7ff2980b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.438+0000 7ff29f015640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2980cd630 con 0x7ff29815e540 2023-10-14T16:56:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff29f015640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2980ce1b0 con 0x7ff29815e0c0 2023-10-14T16:56:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff29f015640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29806cd10 con 0x7ff29806c3f0 2023-10-14T16:56:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff297fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff29815e0c0 0x7ff2980b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff29d5b2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff29815e540 0x7ff2980b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40744/0 (socket says 172.21.15.29:40744) 2023-10-14T16:56:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff29d5b2640 1 -- 172.21.15.29:0/3385357280 learned_addr learned my addr 172.21.15.29:0/3385357280 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff29cdb1640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29806c3f0 0x7ff29806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff297fff640 1 -- 172.21.15.29:0/3385357280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29806c3f0 msgr2=0x7ff29806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff297fff640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29806c3f0 0x7ff29806c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff297fff640 1 -- 172.21.15.29:0/3385357280 >> v1:172.21.15.29:6789/0 conn(0x7ff29815e540 legacy=0x7ff2980b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff297fff640 1 -- 172.21.15.29:0/3385357280 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2980d2170 con 0x7ff29815e0c0 2023-10-14T16:56:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.439+0000 7ff2977fe640 1 -- 172.21.15.29:0/3385357280 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1932087864 0 0) 0x7ff2980cd630 con 0x7ff29815e540 2023-10-14T16:56:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.441+0000 7ff297fff640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e0c0 0x7ff2980b4730 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097a0 tx=0x7ff28402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27c0f807bcb434be server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.441+0000 7ff2977fe640 1 -- 172.21.15.29:0/3385357280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284031830 con 0x7ff29815e0c0 2023-10-14T16:56:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.441+0000 7ff2977fe640 1 -- 172.21.15.29:0/3385357280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff284031990 con 0x7ff29815e0c0 2023-10-14T16:56:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.441+0000 7ff2977fe640 1 -- 172.21.15.29:0/3385357280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff284031d20 con 0x7ff29815e0c0 2023-10-14T16:56:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 -- 172.21.15.29:0/3385357280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e0c0 msgr2=0x7ff2980b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e0c0 0x7ff2980b4730 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff2840097a0 tx=0x7ff28402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 -- 172.21.15.29:0/3385357280 shutdown_connections 2023-10-14T16:56:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e0c0 0x7ff2980b4730 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 --2- 172.21.15.29:0/3385357280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29806c3f0 0x7ff29806c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 -- 172.21.15.29:0/3385357280 >> 172.21.15.29:0/3385357280 conn(0x7ff2980d2cb0 msgr2=0x7ff2980d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 -- 172.21.15.29:0/3385357280 shutdown_connections 2023-10-14T16:56:55.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.442+0000 7ff29f015640 1 -- 172.21.15.29:0/3385357280 wait complete. 2023-10-14T16:56:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.444+0000 7ff29f015640 1 Processor -- start 2023-10-14T16:56:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.444+0000 7ff29f015640 1 -- start start 2023-10-14T16:56:55.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 0x7ff2980b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29815e0c0 0x7ff2980ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29815a7d0 con 0x7ff29815e540 2023-10-14T16:56:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2980b4560 con 0x7ff29806c3f0 2023-10-14T16:56:55.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.445+0000 7ff29f015640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff298159900 con 0x7ff29815e0c0 2023-10-14T16:56:55.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:55.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39188/0 (socket says 172.21.15.29:39188) 2023-10-14T16:56:55.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 -- 172.21.15.29:0/22471403 learned_addr learned my addr 172.21.15.29:0/22471403 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:55.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29cdb1640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 0x7ff2980b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 -- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29815e0c0 msgr2=0x7ff2980ccb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29815e0c0 0x7ff2980ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 -- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 msgr2=0x7ff2980b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 0x7ff2980b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 -- 172.21.15.29:0/22471403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff284009400 con 0x7ff29815e540 2023-10-14T16:56:55.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29cdb1640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 0x7ff2980b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.446+0000 7ff29d5b2640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff29000c7a0 tx=0x7ff29000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.447+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29000a5e0 con 0x7ff29815e540 2023-10-14T16:56:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.447+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff29000a760 con 0x7ff29815e540 2023-10-14T16:56:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.447+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff29000ab10 con 0x7ff29815e540 2023-10-14T16:56:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.447+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff298159b50 con 0x7ff29815e540 2023-10-14T16:56:55.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.447+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29815a110 con 0x7ff29815e540 2023-10-14T16:56:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.448+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff29815a910 con 0x7ff29815e540 2023-10-14T16:56:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.449+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff29000ac70 con 0x7ff29815e540 2023-10-14T16:56:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.450+0000 7ff2957fa640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 0x7ff27c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:55.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.450+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7ff290088bb0 con 0x7ff29815e540 2023-10-14T16:56:55.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.450+0000 7ff29cdb1640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 0x7ff27c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:55.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.451+0000 7ff29cdb1640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 0x7ff27c06d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff28c006fd0 tx=0x7ff28c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:55.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.455+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff290053ad0 con 0x7ff29815e540 2023-10-14T16:56:55.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:55.722+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]} v 0) v1 -- 0x7ff2980d1f60 con 0x7ff29815e540 2023-10-14T16:56:56.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.140+0000 7ff2957fa640 1 -- 172.21.15.29:0/22471403 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_schedule"]}]=0 v613) v1 ==== 156+0+0 (secure 0 0 0) 0x7ff290018020 con 0x7ff29815e540 2023-10-14T16:56:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.143+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 msgr2=0x7ff27c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.143+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 0x7ff27c06d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7ff28c006fd0 tx=0x7ff28c008040 comp rx=0 tx=0).stop 2023-10-14T16:56:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.143+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 msgr2=0x7ff2980cd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.143+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff29000c7a0 tx=0x7ff29000cc60 comp rx=0 tx=0).stop 2023-10-14T16:56:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 shutdown_connections 2023-10-14T16:56:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff27c06afc0 0x7ff27c06d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff29815e540 0x7ff2980cd0d0 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff29815e0c0 0x7ff2980ccb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 --2- 172.21.15.29:0/22471403 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff29806c3f0 0x7ff2980b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 >> 172.21.15.29:0/22471403 conn(0x7ff2980d2cb0 msgr2=0x7ff29815bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 shutdown_connections 2023-10-14T16:56:56.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.144+0000 7ff29f015640 1 -- 172.21.15.29:0/22471403 wait complete. 2023-10-14T16:56:56.166 INFO:tasks.cephfs_test_runner:Test listing non-existing snap schedules failure ... ok 2023-10-14T16:56:56.167 INFO:tasks.cephfs_test_runner:Starting test: test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules on a non-existing filesystem path failure 2023-10-14T16:56:56.167 INFO:tasks.cephfs_test_runner:test_non_existent_snap_schedule_list (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:56:56.168 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:56:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.393+0000 7f8163327640 1 Processor -- start 2023-10-14T16:56:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.393+0000 7f8163327640 1 -- start start 2023-10-14T16:56:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.394+0000 7f8163327640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 0x7f815c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.394+0000 7f8163327640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c15f4a0 con 0x7f815c0cda80 2023-10-14T16:56:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.395+0000 7f8163327640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f815c06c960 con 0x7f815c0ad970 2023-10-14T16:56:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.395+0000 7f8163327640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c15b9d0 con 0x7f815c06c040 2023-10-14T16:56:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.395+0000 7f8162b26640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f815c0cda80 0x7f815c15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40756/0 (socket says 172.21.15.29:40756) 2023-10-14T16:56:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.395+0000 7f8162b26640 1 -- 172.21.15.29:0/211762852 learned_addr learned my addr 172.21.15.29:0/211762852 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:56.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.395+0000 7f8162325640 1 --2- 172.21.15.29:0/211762852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 0x7f815c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561518683 0 0) 0x7f815c15f4a0 con 0x7f815c0cda80 2023-10-14T16:56:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8148003520 con 0x7f815c0cda80 2023-10-14T16:56:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654169020 0 0) 0x7f815c06c960 con 0x7f815c0ad970 2023-10-14T16:56:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8162325640 1 -- 172.21.15.29:0/211762852 >> v1:172.21.15.186:6789/0 conn(0x7f815c0ad970 legacy=0x7f815c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8162325640 1 -- 172.21.15.29:0/211762852 >> v1:172.21.15.29:6789/0 conn(0x7f815c0cda80 legacy=0x7f815c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.396+0000 7f8162325640 1 -- 172.21.15.29:0/211762852 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f815c15bcb0 con 0x7f815c06c040 2023-10-14T16:56:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.397+0000 7f8162325640 1 --2- 172.21.15.29:0/211762852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 0x7f815c06c420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f81580097d0 tx=0x7f815802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34740502081a31ae server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.398+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158030800 con 0x7f815c06c040 2023-10-14T16:56:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.398+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8158030960 con 0x7f815c06c040 2023-10-14T16:56:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.398+0000 7f8161323640 1 -- 172.21.15.29:0/211762852 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8158030cf0 con 0x7f815c06c040 2023-10-14T16:56:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 -- 172.21.15.29:0/211762852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 msgr2=0x7f815c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 --2- 172.21.15.29:0/211762852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 0x7f815c06c420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f81580097d0 tx=0x7f815802b940 comp rx=0 tx=0).stop 2023-10-14T16:56:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 -- 172.21.15.29:0/211762852 shutdown_connections 2023-10-14T16:56:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 --2- 172.21.15.29:0/211762852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c06c040 0x7f815c06c420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 -- 172.21.15.29:0/211762852 >> 172.21.15.29:0/211762852 conn(0x7f815c0ce6a0 msgr2=0x7f815c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 -- 172.21.15.29:0/211762852 shutdown_connections 2023-10-14T16:56:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.399+0000 7f8163327640 1 -- 172.21.15.29:0/211762852 wait complete. 2023-10-14T16:56:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.401+0000 7f8163327640 1 Processor -- start 2023-10-14T16:56:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.401+0000 7f8163327640 1 -- start start 2023-10-14T16:56:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.402+0000 7f8163327640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f815c06c040 0x7f815c0c9a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.402+0000 7f8163327640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8163327640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 0x7f815c15dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8163327640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c0c9fd0 con 0x7f815c0ad970 2023-10-14T16:56:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8163327640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1f27d0 con 0x7f815c06c040 2023-10-14T16:56:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8163327640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f815c1f2ab0 con 0x7f815c0cda80 2023-10-14T16:56:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39200/0 (socket says 172.21.15.29:39200) 2023-10-14T16:56:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 -- 172.21.15.29:0/3131774005 learned_addr learned my addr 172.21.15.29:0/3131774005 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8162325640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f815c06c040 0x7f815c0c9a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8162b26640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 0x7f815c15dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 -- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 msgr2=0x7f815c15dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 0x7f815c15dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 -- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f815c06c040 msgr2=0x7f815c0c9a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f815c06c040 0x7f815c0c9a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8161b24640 1 -- 172.21.15.29:0/3131774005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8158009400 con 0x7f815c0ad970 2023-10-14T16:56:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.403+0000 7f8162b26640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 0x7f815c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8161b24640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c410 tx=0x7f814c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c0035f0 con 0x7f815c0ad970 2023-10-14T16:56:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f814c003770 con 0x7f815c0ad970 2023-10-14T16:56:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814c003b20 con 0x7f815c0ad970 2023-10-14T16:56:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f815c1f2d90 con 0x7f815c0ad970 2023-10-14T16:56:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.404+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f815c1f3350 con 0x7f815c0ad970 2023-10-14T16:56:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.405+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815c06b330 con 0x7f815c0ad970 2023-10-14T16:56:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.407+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f814c003c80 con 0x7f815c0ad970 2023-10-14T16:56:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.408+0000 7f8152ffd640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.408+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f814c088e30 con 0x7f815c0ad970 2023-10-14T16:56:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.408+0000 7f8162325640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.409+0000 7f8162325640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 0x7f814006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8158036000 tx=0x7f815803b040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.412+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f814c053d50 con 0x7f815c0ad970 2023-10-14T16:56:56.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.608+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f815c04e9b0 con 0x7f815c0ad970 2023-10-14T16:56:56.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.610+0000 7f8152ffd640 1 -- 172.21.15.29:0/3131774005 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v87) v1 ==== 74+0+14042 (secure 0 0 0) 0x7f814c017020 con 0x7f815c0ad970 2023-10-14T16:56:56.611 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:56.611 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":87,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:56:53.750360+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6837/3113871431":"2023-10-15T16:56:50.728930+0000","172.21.15.29:6836/3113871431":"2023-10-15T16:56:50.728930+0000","172.21.15.29:0/2640045513":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3061435492":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/34298176":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6833/1682700904":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3103792448":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/3148536323":"2023-10-15T16:56:46.943924+0000","172.21.15.29:0/626362321":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/664522093":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/126969037":"2023-10-15T16:56:40.500646+0000","172.21.15.29:6833/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/266566422":"2023-10-15T16:56:46.943924+0000","172.21.15.29:6832/3108857887":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/1855767981":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/3741726235":"2023-10-15T16:56:40.500646+0000","172.21.15.29:0/2072219911":"2023-10-15T16:56:40.500646+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:56:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 msgr2=0x7f814006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 0x7f814006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8158036000 tx=0x7f815803b040 comp rx=0 tx=0).stop 2023-10-14T16:56:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 msgr2=0x7f815c15d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f814c00c410 tx=0x7f814c00c8d0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 shutdown_connections 2023-10-14T16:56:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f814006af90 0x7f814006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f815c0cda80 0x7f815c15dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f815c0ad970 0x7f815c15d770 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 --2- 172.21.15.29:0/3131774005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f815c06c040 0x7f815c0c9a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 >> 172.21.15.29:0/3131774005 conn(0x7f815c0ce6a0 msgr2=0x7f815c0b3860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 shutdown_connections 2023-10-14T16:56:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.613+0000 7f8163327640 1 -- 172.21.15.29:0/3131774005 wait complete. 2023-10-14T16:56:56.633 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:56:56.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:56:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.864+0000 7fc0556d1640 1 Processor -- start 2023-10-14T16:56:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.864+0000 7fc0556d1640 1 -- start start 2023-10-14T16:56:56.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.865+0000 7fc0556d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0500b3ec0 0x7fc0500cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.865+0000 7fc0556d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc0500cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.865+0000 7fc0556d1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc05006be30 con 0x7fc0500b0ea0 2023-10-14T16:56:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.865+0000 7fc0556d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc05006ca00 con 0x7fc0500b3ec0 2023-10-14T16:56:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.865+0000 7fc0556d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500ada10 con 0x7fc0500b43c0 2023-10-14T16:56:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0500b3ec0 0x7fc0500cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0500b3ec0 0x7fc0500cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39212/0 (socket says 172.21.15.29:39212) 2023-10-14T16:56:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 -- 172.21.15.29:0/436440158 learned_addr learned my addr 172.21.15.29:0/436440158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 -- 172.21.15.29:0/436440158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 msgr2=0x7fc0500cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc054ed0640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc0500cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc0500cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 -- 172.21.15.29:0/436440158 >> v1:172.21.15.186:6789/0 conn(0x7fc0500b0ea0 legacy=0x7fc05015a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc04ffff640 1 -- 172.21.15.29:0/436440158 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05006c060 con 0x7fc0500b3ec0 2023-10-14T16:56:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.866+0000 7fc054ed0640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc0500cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.867+0000 7fc04ffff640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc0500cd260 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc040009870 tx=0x7fc04002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d1ba5122f763c4bc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.868+0000 7fc04f7fe640 1 -- 172.21.15.29:0/436440158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0400317c0 con 0x7fc0500b3ec0 2023-10-14T16:56:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.868+0000 7fc04f7fe640 1 -- 172.21.15.29:0/436440158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc040031920 con 0x7fc0500b3ec0 2023-10-14T16:56:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.868+0000 7fc04f7fe640 1 -- 172.21.15.29:0/436440158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040031cb0 con 0x7fc0500b3ec0 2023-10-14T16:56:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.868+0000 7fc0556d1640 1 -- 172.21.15.29:0/436440158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 msgr2=0x7fc0500cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.868+0000 7fc0556d1640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc0500cd260 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fc040009870 tx=0x7fc04002b860 comp rx=0 tx=0).stop 2023-10-14T16:56:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 -- 172.21.15.29:0/436440158 shutdown_connections 2023-10-14T16:56:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc0500cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 --2- 172.21.15.29:0/436440158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc0500cd260 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 -- 172.21.15.29:0/436440158 >> 172.21.15.29:0/436440158 conn(0x7fc0500b0170 msgr2=0x7fc0500b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 -- 172.21.15.29:0/436440158 shutdown_connections 2023-10-14T16:56:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.869+0000 7fc0556d1640 1 -- 172.21.15.29:0/436440158 wait complete. 2023-10-14T16:56:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.870+0000 7fc0556d1640 1 Processor -- start 2023-10-14T16:56:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.871+0000 7fc0556d1640 1 -- start start 2023-10-14T16:56:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.871+0000 7fc0556d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 0x7fc0500c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc0556d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc0556d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc050070950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc0556d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500c6d90 con 0x7fc0500b3ec0 2023-10-14T16:56:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc0556d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500c5cf0 con 0x7fc0500b0ea0 2023-10-14T16:56:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc0556d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0500c5fd0 con 0x7fc0500b43c0 2023-10-14T16:56:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc047fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc047fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39220/0 (socket says 172.21.15.29:39220) 2023-10-14T16:56:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc047fff640 1 -- 172.21.15.29:0/2030914529 learned_addr learned my addr 172.21.15.29:0/2030914529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:56.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc04ffff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 0x7fc0500c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc054ed0640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc050070950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc047fff640 1 -- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 msgr2=0x7fc050070950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.872+0000 7fc047fff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc050070950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc047fff640 1 -- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 msgr2=0x7fc0500c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc047fff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 0x7fc0500c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc047fff640 1 -- 172.21.15.29:0/2030914529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0400094d0 con 0x7fc0500b3ec0 2023-10-14T16:56:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc04ffff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 0x7fc0500c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:56:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc047fff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc03800b3f0 tx=0x7fc03800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038011830 con 0x7fc0500b3ec0 2023-10-14T16:56:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc038011990 con 0x7fc0500b3ec0 2023-10-14T16:56:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc038011d40 con 0x7fc0500b3ec0 2023-10-14T16:56:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0500c62b0 con 0x7fc0500b3ec0 2023-10-14T16:56:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.873+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0500c6870 con 0x7fc0500b3ec0 2023-10-14T16:56:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.874+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc05004e9b0 con 0x7fc0500b3ec0 2023-10-14T16:56:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.875+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc03801a400 con 0x7fc0500b3ec0 2023-10-14T16:56:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.876+0000 7fc04d7fa640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 0x7fc03006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.876+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc038089630 con 0x7fc0500b3ec0 2023-10-14T16:56:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.877+0000 7fc04ffff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 0x7fc03006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.877+0000 7fc04ffff640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 0x7fc03006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc040036000 tx=0x7fc04003d040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:56.881+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0380545e0 con 0x7fc0500b3ec0 2023-10-14T16:56:57.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.070+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fc05006abd0 con 0x7fc0500b3ec0 2023-10-14T16:56:57.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.071+0000 7fc04d7fa640 1 -- 172.21.15.29:0/2030914529 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v87) v1 ==== 93+0+10 (secure 0 0 0) 0x7fc038057c00 con 0x7fc0500b3ec0 2023-10-14T16:56:57.072 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:56:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 msgr2=0x7fc03006d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 0x7fc03006d420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fc040036000 tx=0x7fc04003d040 comp rx=0 tx=0).stop 2023-10-14T16:56:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 msgr2=0x7fc05014aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fc03800b3f0 tx=0x7fc03800b8b0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 shutdown_connections 2023-10-14T16:56:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc03006af60 0x7fc03006d420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0500b43c0 0x7fc050070950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0500b3ec0 0x7fc05014aa30 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 --2- 172.21.15.29:0/2030914529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0500b0ea0 0x7fc0500c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 >> 172.21.15.29:0/2030914529 conn(0x7fc0500b0170 msgr2=0x7fc0500af6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 shutdown_connections 2023-10-14T16:56:57.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.074+0000 7fc0556d1640 1 -- 172.21.15.29:0/2030914529 wait complete. 2023-10-14T16:56:57.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-14T16:56:57.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.310+0000 7f7c4be09640 1 Processor -- start 2023-10-14T16:56:57.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.310+0000 7f7c4be09640 1 -- start start 2023-10-14T16:56:57.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4be09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 0x7f7c4406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4be09640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7c44070fd0 0x7f7c440b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4be09640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c44071450 con 0x7f7c440adf30 2023-10-14T16:56:57.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4be09640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c440cc600 con 0x7f7c44070fd0 2023-10-14T16:56:57.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4be09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c440cc8e0 con 0x7f7c440adb50 2023-10-14T16:56:57.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.312+0000 7f7c4a3a6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7c440adf30 0x7f7c44164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40760/0 (socket says 172.21.15.29:40760) 2023-10-14T16:56:57.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c4a3a6640 1 -- 172.21.15.29:0/955401572 learned_addr learned my addr 172.21.15.29:0/955401572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:57.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/955401572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 0x7f7c4406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:57.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c493a4640 1 --2- 172.21.15.29:0/955401572 >> v2:172.21.15.186:3300/0 conn(0x7f7c44070fd0 0x7f7c440b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 811258571 0 0) 0x7f7c44071450 con 0x7f7c440adf30 2023-10-14T16:56:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c30003520 con 0x7f7c440adf30 2023-10-14T16:56:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3914928594 0 0) 0x7f7c30003520 con 0x7f7c440adf30 2023-10-14T16:56:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c49ba5640 1 -- 172.21.15.29:0/955401572 >> v2:172.21.15.186:3300/0 conn(0x7f7c44070fd0 msgr2=0x7f7c440b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:57.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/955401572 >> v2:172.21.15.186:3300/0 conn(0x7f7c44070fd0 0x7f7c440b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c49ba5640 1 -- 172.21.15.29:0/955401572 >> v1:172.21.15.29:6789/0 conn(0x7f7c440adf30 legacy=0x7f7c44164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.313+0000 7f7c49ba5640 1 -- 172.21.15.29:0/955401572 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c440ccbc0 con 0x7f7c440adb50 2023-10-14T16:56:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.315+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/955401572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 0x7f7c4406f550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097a0 tx=0x7f7c4002b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a197ad3753807a6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.315+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0019f0 con 0x7f7c440adb50 2023-10-14T16:56:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.315+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4002e050 con 0x7f7c440adb50 2023-10-14T16:56:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.315+0000 7f7c48ba3640 1 -- 172.21.15.29:0/955401572 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c40030710 con 0x7f7c440adb50 2023-10-14T16:56:57.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 -- 172.21.15.29:0/955401572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 msgr2=0x7f7c4406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 --2- 172.21.15.29:0/955401572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 0x7f7c4406f550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f7c400097a0 tx=0x7f7c4002b790 comp rx=0 tx=0).stop 2023-10-14T16:56:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 -- 172.21.15.29:0/955401572 shutdown_connections 2023-10-14T16:56:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 --2- 172.21.15.29:0/955401572 >> v2:172.21.15.186:3300/0 conn(0x7f7c44070fd0 0x7f7c440b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 --2- 172.21.15.29:0/955401572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adb50 0x7f7c4406f550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 -- 172.21.15.29:0/955401572 >> 172.21.15.29:0/955401572 conn(0x7f7c440cb410 msgr2=0x7f7c440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 -- 172.21.15.29:0/955401572 shutdown_connections 2023-10-14T16:56:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.316+0000 7f7c4be09640 1 -- 172.21.15.29:0/955401572 wait complete. 2023-10-14T16:56:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.318+0000 7f7c4be09640 1 Processor -- start 2023-10-14T16:56:57.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.318+0000 7f7c4be09640 1 -- start start 2023-10-14T16:56:57.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c440adb50 0x7f7c4415f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adf30 0x7f7c441649c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c441620a0 con 0x7f7c44070fd0 2023-10-14T16:56:57.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44162380 con 0x7f7c440adb50 2023-10-14T16:56:57.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c4be09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c44162660 con 0x7f7c440adf30 2023-10-14T16:56:57.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.319+0000 7f7c49ba5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:57.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39232/0 (socket says 172.21.15.29:39232) 2023-10-14T16:56:57.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 -- 172.21.15.29:0/3895174667 learned_addr learned my addr 172.21.15.29:0/3895174667 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:57.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c4a3a6640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adf30 0x7f7c441649c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:57.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 -- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adf30 msgr2=0x7f7c441649c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adf30 0x7f7c441649c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 -- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c440adb50 msgr2=0x7f7c4415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:57.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c440adb50 0x7f7c4415f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:57.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 -- 172.21.15.29:0/3895174667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c40009400 con 0x7f7c44070fd0 2023-10-14T16:56:57.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.320+0000 7f7c49ba5640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7c40000c00 tx=0x7f7c400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:57.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.321+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4003c620 con 0x7f7c44070fd0 2023-10-14T16:56:57.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.321+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4003c7a0 con 0x7f7c44070fd0 2023-10-14T16:56:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.321+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c4003cb50 con 0x7f7c44070fd0 2023-10-14T16:56:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.321+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c44162940 con 0x7f7c44070fd0 2023-10-14T16:56:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.321+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c44161d70 con 0x7f7c44070fd0 2023-10-14T16:56:57.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.322+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c44066de0 con 0x7f7c44070fd0 2023-10-14T16:56:57.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.323+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7c40039070 con 0x7f7c44070fd0 2023-10-14T16:56:57.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.324+0000 7f7c3a7fc640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 0x7f7c2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:57.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.324+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f7c400aafe0 con 0x7f7c44070fd0 2023-10-14T16:56:57.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.324+0000 7f7c493a4640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 0x7f7c2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:57.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.325+0000 7f7c493a4640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 0x7f7c2806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006a90 tx=0x7f7c34006a20 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:57.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.328+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7c40075e30 con 0x7f7c44070fd0 2023-10-14T16:56:57.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:57.598+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f7c4404e160 con 0x7f7c44070fd0 2023-10-14T16:56:58.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.152+0000 7f7c3a7fc640 1 -- 172.21.15.29:0/3895174667 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v615) v1 ==== 169+0+0 (secure 0 0 0) 0x7f7c400370f0 con 0x7f7c44070fd0 2023-10-14T16:56:58.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 msgr2=0x7f7c2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 0x7f7c2806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006a90 tx=0x7f7c34006a20 comp rx=0 tx=0).stop 2023-10-14T16:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 msgr2=0x7f7c44163140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f7c40000c00 tx=0x7f7c400046c0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 shutdown_connections 2023-10-14T16:56:58.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f7c2806af90 0x7f7c2806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.156+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7c440adf30 0x7f7c441649c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.157+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7c440adb50 0x7f7c4415f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.157+0000 7f7c4be09640 1 --2- 172.21.15.29:0/3895174667 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7c44070fd0 0x7f7c44163140 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.157+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 >> 172.21.15.29:0/3895174667 conn(0x7f7c440cb410 msgr2=0x7f7c4406cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:58.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.157+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 shutdown_connections 2023-10-14T16:56:58.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.157+0000 7f7c4be09640 1 -- 172.21.15.29:0/3895174667 wait complete. 2023-10-14T16:56:58.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:56:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.410+0000 7f57923e9640 1 Processor -- start 2023-10-14T16:56:58.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.410+0000 7f57923e9640 1 -- start start 2023-10-14T16:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f57923e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f578c159e90 0x7f578c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f57923e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c070360 con 0x7f578c159ab0 2023-10-14T16:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f57923e9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f578c0cd1f0 con 0x7f578c06c140 2023-10-14T16:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f57923e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f578c15d890 con 0x7f578c159e90 2023-10-14T16:56:58.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f578bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f578c159ab0 0x7f578c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40766/0 (socket says 172.21.15.29:40766) 2023-10-14T16:56:58.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.412+0000 7f578bfff640 1 -- 172.21.15.29:0/4215422852 learned_addr learned my addr 172.21.15.29:0/4215422852 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f5783fff640 1 --2- 172.21.15.29:0/4215422852 >> v2:172.21.15.186:3300/0 conn(0x7f578c159e90 0x7f578c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991846368 0 0) 0x7f578c070360 con 0x7f578c159ab0 2023-10-14T16:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5778003520 con 0x7f578c159ab0 2023-10-14T16:56:58.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591189942 0 0) 0x7f5778003520 con 0x7f578c159ab0 2023-10-14T16:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 >> v2:172.21.15.186:3300/0 conn(0x7f578c159e90 msgr2=0x7f578c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 --2- 172.21.15.29:0/4215422852 >> v2:172.21.15.186:3300/0 conn(0x7f578c159e90 0x7f578c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 >> v1:172.21.15.186:6789/0 conn(0x7f578c06c140 legacy=0x7f578c15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c0cd4d0 con 0x7f578c159ab0 2023-10-14T16:56:58.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f577c001670 con 0x7f578c159ab0 2023-10-14T16:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.413+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 >> v1:172.21.15.29:6789/0 conn(0x7f578c159ab0 legacy=0x7f578c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5778003970 0x7f5778003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57780047c0 0x7f5778004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 0x7f577800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c15d890 con 0x7f57780052a0 2023-10-14T16:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f577800c170 con 0x7f5778003970 2023-10-14T16:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f577800c330 con 0x7f57780047c0 2023-10-14T16:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 0x7f577800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5783fff640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57780047c0 0x7f5778004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578bfff640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5778003970 0x7f5778003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 -- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57780047c0 msgr2=0x7f5778004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f57780047c0 0x7f5778004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 -- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5778003970 msgr2=0x7f5778003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:58.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5778003970 0x7f5778003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f5790986640 1 -- 172.21.15.29:0/4215422852 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5778003520 con 0x7f57780052a0 2023-10-14T16:56:58.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.414+0000 7f578bfff640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5778003970 0x7f5778003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:58.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.415+0000 7f5790986640 1 -- 172.21.15.29:0/4215422852 reap_dead start 2023-10-14T16:56:58.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f5790986640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 0x7f577800b980 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5784001e20 tx=0x7f57780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4520fb425c1cb495 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5784003b70 con 0x7f57780052a0 2023-10-14T16:56:58.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5784003cd0 con 0x7f57780052a0 2023-10-14T16:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f578b7fe640 1 -- 172.21.15.29:0/4215422852 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5784005310 con 0x7f57780052a0 2023-10-14T16:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f57923e9640 1 -- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 msgr2=0x7f577800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f57923e9640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 0x7f577800b980 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7f5784001e20 tx=0x7f57780057f0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f57923e9640 1 -- 172.21.15.29:0/4215422852 shutdown_connections 2023-10-14T16:56:58.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f57923e9640 1 --2- 172.21.15.29:0/4215422852 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f57780052a0 0x7f577800b980 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.416+0000 7f57923e9640 1 -- 172.21.15.29:0/4215422852 >> 172.21.15.29:0/4215422852 conn(0x7f578c0ce9b0 msgr2=0x7f578c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.417+0000 7f57923e9640 1 -- 172.21.15.29:0/4215422852 shutdown_connections 2023-10-14T16:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.417+0000 7f57923e9640 1 -- 172.21.15.29:0/4215422852 wait complete. 2023-10-14T16:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.418+0000 7f57923e9640 1 Processor -- start 2023-10-14T16:56:58.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.419+0000 7f57923e9640 1 -- start start 2023-10-14T16:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.419+0000 7f57923e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57780052a0 0x7f578c0c4760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.419+0000 7f57923e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f57923e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 0x7f578c070140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f57923e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c153c80 con 0x7f578c1529b0 2023-10-14T16:56:58.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f57923e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f578c153370 con 0x7f57780052a0 2023-10-14T16:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f57923e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f578c15af20 con 0x7f578c06fcc0 2023-10-14T16:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39260/0 (socket says 172.21.15.29:39260) 2023-10-14T16:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 -- 172.21.15.29:0/395310328 learned_addr learned my addr 172.21.15.29:0/395310328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:58.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5790986640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 0x7f578c070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f578bfff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57780052a0 0x7f578c0c4760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 -- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 msgr2=0x7f578c070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 0x7f578c070140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 -- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57780052a0 msgr2=0x7f578c0c4760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57780052a0 0x7f578c0c4760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5783fff640 1 -- 172.21.15.29:0/395310328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c15b200 con 0x7f578c1529b0 2023-10-14T16:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.420+0000 7f5790986640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 0x7f578c070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f5783fff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5774010860 tx=0x7f5774010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5774009610 con 0x7f578c1529b0 2023-10-14T16:56:58.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f577400eb60 con 0x7f578c1529b0 2023-10-14T16:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f577401c400 con 0x7f578c1529b0 2023-10-14T16:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f578c15b4e0 con 0x7f578c1529b0 2023-10-14T16:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.421+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f578c15baa0 con 0x7f578c1529b0 2023-10-14T16:56:58.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.422+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f578c0cd790 con 0x7f578c1529b0 2023-10-14T16:56:58.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.423+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f577401c560 con 0x7f578c1529b0 2023-10-14T16:56:58.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.424+0000 7f57897fa640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 0x7f576c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:58.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.424+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f5774089d60 con 0x7f578c1529b0 2023-10-14T16:56:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.424+0000 7f578bfff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 0x7f576c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.425+0000 7f578bfff640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 0x7f576c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f577c001200 tx=0x7f577c009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:58.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.428+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f578c0cd790 con 0x7f578c1529b0 2023-10-14T16:56:58.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.625+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f578c0cd790 con 0x7f578c1529b0 2023-10-14T16:56:58.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.631+0000 7f57897fa640 1 -- 172.21.15.29:0/395310328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v28)=0 v28) v1 ==== 122+0+0 (secure 0 0 0) 0x7f578c0cd790 con 0x7f578c1529b0 2023-10-14T16:56:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 msgr2=0x7f576c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 0x7f576c06d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f577c001200 tx=0x7f577c009040 comp rx=0 tx=0).stop 2023-10-14T16:56:58.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 msgr2=0x7f578c152e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7f5774010860 tx=0x7f5774010d20 comp rx=0 tx=0).stop 2023-10-14T16:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 shutdown_connections 2023-10-14T16:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f576c06afc0 0x7f576c06d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f578c06fcc0 0x7f578c070140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f578c1529b0 0x7f578c152e30 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 --2- 172.21.15.29:0/395310328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f57780052a0 0x7f578c0c4760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 >> 172.21.15.29:0/395310328 conn(0x7f578c0ce9b0 msgr2=0x7f578c06c520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 shutdown_connections 2023-10-14T16:56:58.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:58.634+0000 7f57923e9640 1 -- 172.21.15.29:0/395310328 wait complete. 2023-10-14T16:56:58.655 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:56:58.656 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:58.766 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:56:58.895 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:56:58.895 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:58.896 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:58.959 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:58.959 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:58.960 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:59.019 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:59.019 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:56:59.020 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:59.083 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:56:59.084 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:56:59.084 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:56:59.144 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:56:59.145 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:56:59.145 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:56:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.381+0000 7fc7a6927640 1 Processor -- start 2023-10-14T16:56:59.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.381+0000 7fc7a6927640 1 -- start start 2023-10-14T16:56:59.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 0x7fc7a015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a006c100 0x7fc7a006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a015e490 con 0x7fc7a006c100 2023-10-14T16:56:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc7a0070df0 con 0x7fc7a0159950 2023-10-14T16:56:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6927640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a015aef0 con 0x7fc7a0070450 2023-10-14T16:56:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6126640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a006c100 0x7fc7a006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6126640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc7a006c100 0x7fc7a006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39274/0 (socket says 172.21.15.29:39274) 2023-10-14T16:56:59.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.383+0000 7fc7a6126640 1 -- 172.21.15.29:0/2067783241 learned_addr learned my addr 172.21.15.29:0/2067783241 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a6126640 1 -- 172.21.15.29:0/2067783241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 msgr2=0x7fc7a00708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:59.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a5925640 1 --2- 172.21.15.29:0/2067783241 >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 0x7fc7a015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a5124640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a6126640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a00708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a6126640 1 -- 172.21.15.29:0/2067783241 >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 msgr2=0x7fc7a015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a6126640 1 --2- 172.21.15.29:0/2067783241 >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 0x7fc7a015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a6126640 1 -- 172.21.15.29:0/2067783241 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a015d910 con 0x7fc7a006c100 2023-10-14T16:56:59.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a5925640 1 --2- 172.21.15.29:0/2067783241 >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 0x7fc7a015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.384+0000 7fc7a5124640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.385+0000 7fc7a6126640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a006c570 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00ca950 tx=0x7fc79802be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fd647de3ab8b8a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.385+0000 7fc7a4923640 1 -- 172.21.15.29:0/2067783241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79802c9f0 con 0x7fc7a006c100 2023-10-14T16:56:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.385+0000 7fc7a4923640 1 -- 172.21.15.29:0/2067783241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79802cb50 con 0x7fc7a006c100 2023-10-14T16:56:59.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.385+0000 7fc7a4923640 1 -- 172.21.15.29:0/2067783241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7980343f0 con 0x7fc7a006c100 2023-10-14T16:56:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 -- 172.21.15.29:0/2067783241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 msgr2=0x7fc7a006c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a006c570 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00ca950 tx=0x7fc79802be50 comp rx=0 tx=0).stop 2023-10-14T16:56:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 -- 172.21.15.29:0/2067783241 shutdown_connections 2023-10-14T16:56:59.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a006c570 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 --2- 172.21.15.29:0/2067783241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 --2- 172.21.15.29:0/2067783241 >> v2:172.21.15.186:3300/0 conn(0x7fc7a0159950 0x7fc7a015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 -- 172.21.15.29:0/2067783241 >> 172.21.15.29:0/2067783241 conn(0x7fc7a00ce6a0 msgr2=0x7fc7a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 -- 172.21.15.29:0/2067783241 shutdown_connections 2023-10-14T16:56:59.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.386+0000 7fc7a6927640 1 -- 172.21.15.29:0/2067783241 wait complete. 2023-10-14T16:56:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.388+0000 7fc7a6927640 1 Processor -- start 2023-10-14T16:56:59.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.388+0000 7fc7a6927640 1 -- start start 2023-10-14T16:56:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a01f5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0159950 0x7fc7a01f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01582d0 con 0x7fc7a006c100 2023-10-14T16:56:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a0159040 con 0x7fc7a0159950 2023-10-14T16:56:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a6927640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a01579a0 con 0x7fc7a0070450 2023-10-14T16:56:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a5925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a5925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39284/0 (socket says 172.21.15.29:39284) 2023-10-14T16:56:59.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.389+0000 7fc7a5925640 1 -- 172.21.15.29:0/1456312386 learned_addr learned my addr 172.21.15.29:0/1456312386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5124640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a01f5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a6126640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0159950 0x7fc7a01f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 -- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 msgr2=0x7fc7a01f5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a01f5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 -- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0159950 msgr2=0x7fc7a01f5fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0159950 0x7fc7a01f5fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 -- 172.21.15.29:0/1456312386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc798009b30 con 0x7fc7a006c100 2023-10-14T16:56:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.390+0000 7fc7a5925640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c7f0 tx=0x7fc79c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.391+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c007a30 con 0x7fc7a006c100 2023-10-14T16:56:59.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.391+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc79c007b90 con 0x7fc7a006c100 2023-10-14T16:56:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.391+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc79c00ce40 con 0x7fc7a006c100 2023-10-14T16:56:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.391+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a0157c80 con 0x7fc7a006c100 2023-10-14T16:56:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.391+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a0150090 con 0x7fc7a006c100 2023-10-14T16:56:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.392+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a004e9b0 con 0x7fc7a006c100 2023-10-14T16:56:59.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.393+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc79c0113f0 con 0x7fc7a006c100 2023-10-14T16:56:59.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.394+0000 7fc7967fc640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 0x7fc78406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.394+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc79c089a00 con 0x7fc7a006c100 2023-10-14T16:56:59.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.394+0000 7fc7a5124640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 0x7fc78406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.395+0000 7fc7a5124640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 0x7fc78406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00c5180 tx=0x7fc790008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.399+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc79c0548a0 con 0x7fc7a006c100 2023-10-14T16:56:59.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.632+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc7a00b4620 con 0x7fc7a006c100 2023-10-14T16:56:59.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.636+0000 7fc7967fc640 1 -- 172.21.15.29:0/1456312386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+3702 (secure 0 0 0) 0x7fc79c01f090 con 0x7fc7a006c100 2023-10-14T16:56:59.637 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:56:59.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":34,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5542,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/2546653194","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2546653194},{"type":"v1","addr":"172.21.15.186:6834","nonce":2546653194}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":5545,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/487257298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":487257298},{"type":"v1","addr":"172.21.15.186:6835","nonce":487257298}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6507,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3167200760","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3167200760},{"type":"v1","addr":"172.21.15.29:6835","nonce":3167200760}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":27},{"gid":6915,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3520933724","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3520933724},{"type":"v1","addr":"172.21.15.29:6837","nonce":3520933724}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":34}],"filesystems":[]} 2023-10-14T16:56:59.638 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 34 2023-10-14T16:56:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 msgr2=0x7fc78406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 0x7fc78406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc7a00c5180 tx=0x7fc790008040 comp rx=0 tx=0).stop 2023-10-14T16:56:59.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 msgr2=0x7fc7a00b03c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7fc79c00c7f0 tx=0x7fc79c00cbf0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 shutdown_connections 2023-10-14T16:56:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fc78406af90 0x7fc78406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc7a0159950 0x7fc7a01f5fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc7a0070450 0x7fc7a01f5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 --2- 172.21.15.29:0/1456312386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc7a006c100 0x7fc7a00b03c0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 >> 172.21.15.29:0/1456312386 conn(0x7fc7a00ce6a0 msgr2=0x7fc7a00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:59.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 shutdown_connections 2023-10-14T16:56:59.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.639+0000 7fc7a6927640 1 -- 172.21.15.29:0/1456312386 wait complete. 2023-10-14T16:56:59.660 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:56:59.660 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:56:59.661 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:56:59.661 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:59.661 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:56:59.662 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:56:59.662 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:56:59.662 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:59.662 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:56:59.663 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:56:59.663 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:56:59.663 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:59.664 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:56:59.664 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:56:59.664 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:56:59.665 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:56:59.678 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:56:59.678 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:56:59.679 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:56:59.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:56:59.683 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:56:59.684 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:56:59.691 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:56:59.692 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:56:59.725 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:56:59.726 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:59.732 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:56:59.733 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:59.739 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:56:59.740 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:59.747 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:56:59.748 DEBUG:teuthology.parallel:result is None 2023-10-14T16:56:59.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:56:59.884 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:56:59.886 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:56:59.905 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:56:59.914 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:56:59.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.933+0000 7f60aa86f640 1 Processor -- start 2023-10-14T16:56:59.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.933+0000 7f60aa86f640 1 -- start start 2023-10-14T16:56:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.934+0000 7f60aa86f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 0x7f60a40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.934+0000 7f60aa86f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f60a406c140 0x7f60a406c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.934+0000 7f60aa86f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60a406cad0 con 0x7f60a4159ab0 2023-10-14T16:56:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.934+0000 7f60aa86f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a415d890 con 0x7f60a406c140 2023-10-14T16:56:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.934+0000 7f60aa86f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60a4070360 con 0x7f60a4159e90 2023-10-14T16:56:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f60a4159ab0 0x7f60a406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40774/0 (socket says 172.21.15.29:40774) 2023-10-14T16:56:59.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a3fff640 1 -- 172.21.15.29:0/1364796443 learned_addr learned my addr 172.21.15.29:0/1364796443 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:56:59.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a8e0c640 1 --2- 172.21.15.29:0/1364796443 >> v2:172.21.15.29:3300/0 conn(0x7f60a406c140 0x7f60a406c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a37fe640 1 -- 172.21.15.29:0/1364796443 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 706920587 0 0) 0x7f6094001110 con 0x7f60a4159ab0 2023-10-14T16:56:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a37fe640 1 -- 172.21.15.29:0/1364796443 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6090003520 con 0x7f60a4159ab0 2023-10-14T16:56:59.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f609bfff640 1 --2- 172.21.15.29:0/1364796443 >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 0x7f60a40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a8e0c640 1 -- 172.21.15.29:0/1364796443 >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 msgr2=0x7f60a40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:56:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a8e0c640 1 --2- 172.21.15.29:0/1364796443 >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 0x7f60a40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a8e0c640 1 -- 172.21.15.29:0/1364796443 >> v1:172.21.15.29:6789/0 conn(0x7f60a4159ab0 legacy=0x7f60a406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f60a8e0c640 1 -- 172.21.15.29:0/1364796443 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a40cd1f0 con 0x7f60a406c140 2023-10-14T16:56:59.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.935+0000 7f609bfff640 1 --2- 172.21.15.29:0/1364796443 >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 0x7f60a40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:56:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.936+0000 7f60a8e0c640 1 --2- 172.21.15.29:0/1364796443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a406c590 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f60a406d900 tx=0x7f609c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea267b50e03ddabb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.936+0000 7f60a37fe640 1 -- 172.21.15.29:0/1364796443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c02cb60 con 0x7f60a406c140 2023-10-14T16:56:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.936+0000 7f60a37fe640 1 -- 172.21.15.29:0/1364796443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c02ccc0 con 0x7f60a406c140 2023-10-14T16:56:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60a37fe640 1 -- 172.21.15.29:0/1364796443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f609c034660 con 0x7f60a406c140 2023-10-14T16:56:59.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 -- 172.21.15.29:0/1364796443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 msgr2=0x7f60a406c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 --2- 172.21.15.29:0/1364796443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a406c590 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f60a406d900 tx=0x7f609c02bdb0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 -- 172.21.15.29:0/1364796443 shutdown_connections 2023-10-14T16:56:59.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 --2- 172.21.15.29:0/1364796443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a406c590 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 --2- 172.21.15.29:0/1364796443 >> v2:172.21.15.186:3300/0 conn(0x7f60a4159e90 0x7f60a40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 -- 172.21.15.29:0/1364796443 >> 172.21.15.29:0/1364796443 conn(0x7f60a40ce9b0 msgr2=0x7f60a40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:56:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 -- 172.21.15.29:0/1364796443 shutdown_connections 2023-10-14T16:56:59.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.937+0000 7f60aa86f640 1 -- 172.21.15.29:0/1364796443 wait complete. 2023-10-14T16:56:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.939+0000 7f60aa86f640 1 Processor -- start 2023-10-14T16:56:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.939+0000 7f60aa86f640 1 -- start start 2023-10-14T16:56:59.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4159ab0 0x7f60a4154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4159e90 0x7f60a4154770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40b26e0 con 0x7f60a406c140 2023-10-14T16:56:59.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60a4154570 con 0x7f60a4159e90 2023-10-14T16:56:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60aa86f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60a40b1a60 con 0x7f60a4159ab0 2023-10-14T16:56:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39342/0 (socket says 172.21.15.29:39342) 2023-10-14T16:56:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.940+0000 7f60a3fff640 1 -- 172.21.15.29:0/3146441053 learned_addr learned my addr 172.21.15.29:0/3146441053 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:56:59.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f609bfff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4159ab0 0x7f60a4154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a8e0c640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4159e90 0x7f60a4154770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 -- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4159ab0 msgr2=0x7f60a4154030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4159ab0 0x7f60a4154030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 -- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4159e90 msgr2=0x7f60a4154770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:56:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4159e90 0x7f60a4154770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:56:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 -- 172.21.15.29:0/3146441053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c009a70 con 0x7f60a406c140 2023-10-14T16:56:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a3fff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f609400c700 tx=0x7f609400cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094005ce0 con 0x7f60a406c140 2023-10-14T16:56:59.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.941+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6094005e40 con 0x7f60a406c140 2023-10-14T16:56:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.942+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60a40b1d40 con 0x7f60a406c140 2023-10-14T16:56:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.942+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6094007290 con 0x7f60a406c140 2023-10-14T16:56:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.942+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60a40b2300 con 0x7f60a406c140 2023-10-14T16:56:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.943+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60a404e160 con 0x7f60a406c140 2023-10-14T16:56:59.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.944+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f60940073f0 con 0x7f60a406c140 2023-10-14T16:56:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.944+0000 7f60a17fa640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 0x7f608406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:56:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.945+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f6094088b00 con 0x7f60a406c140 2023-10-14T16:56:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.945+0000 7f609bfff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 0x7f608406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:56:59.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.945+0000 7f609bfff640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 0x7f608406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60a40ad510 tx=0x7f608c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:56:59.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:56:59.949+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f609408ae60 con 0x7f60a406c140 2023-10-14T16:57:00.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:00.166+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f60a406cad0 con 0x7f60a406c140 2023-10-14T16:57:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.167+0000 7f60a17fa640 1 -- 172.21.15.29:0/3146441053 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v92) v1 ==== 97+0+0 (secure 0 0 0) 0x7f60a406cad0 con 0x7f60a406c140 2023-10-14T16:57:01.169 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:57:01.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.171+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 msgr2=0x7f608406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.171+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 0x7f608406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f60a40ad510 tx=0x7f608c008040 comp rx=0 tx=0).stop 2023-10-14T16:57:01.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.171+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 msgr2=0x7f60a415bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.171+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f609400c700 tx=0x7f609400cb00 comp rx=0 tx=0).stop 2023-10-14T16:57:01.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 shutdown_connections 2023-10-14T16:57:01.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f608406af90 0x7f608406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60a4159e90 0x7f60a4154770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60a4159ab0 0x7f60a4154030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 --2- 172.21.15.29:0/3146441053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60a406c140 0x7f60a415bec0 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 >> 172.21.15.29:0/3146441053 conn(0x7f60a40ce9b0 msgr2=0x7f60a40af5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:01.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 shutdown_connections 2023-10-14T16:57:01.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.172+0000 7f60aa86f640 1 -- 172.21.15.29:0/3146441053 wait complete. 2023-10-14T16:57:01.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:57:01.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.413+0000 7fda9ef73640 1 Processor -- start 2023-10-14T16:57:01.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.413+0000 7fda9ef73640 1 -- start start 2023-10-14T16:57:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9ef73640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 0x7fda980cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9ef73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda980ccfb0 0x7fda980cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9ef73640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9815b9d0 con 0x7fda9806c040 2023-10-14T16:57:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9ef73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9815bcb0 con 0x7fda980ccfb0 2023-10-14T16:57:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9ef73640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9815bf90 con 0x7fda980ad970 2023-10-14T16:57:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9e772640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda980ccfb0 0x7fda980cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9e772640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda980ccfb0 0x7fda980cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32968/0 (socket says 172.21.15.29:32968) 2023-10-14T16:57:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.415+0000 7fda9e772640 1 -- 172.21.15.29:0/2394891962 learned_addr learned my addr 172.21.15.29:0/2394891962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:01.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9e772640 1 -- 172.21.15.29:0/2394891962 >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 msgr2=0x7fda980cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:01.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9d770640 1 --2- 172.21.15.29:0/2394891962 >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 0x7fda980cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9e772640 1 --2- 172.21.15.29:0/2394891962 >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 0x7fda980cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9e772640 1 -- 172.21.15.29:0/2394891962 >> v1:172.21.15.186:6789/0 conn(0x7fda9806c040 legacy=0x7fda9806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9e772640 1 -- 172.21.15.29:0/2394891962 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda980cc270 con 0x7fda980ccfb0 2023-10-14T16:57:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.416+0000 7fda9d770640 1 --2- 172.21.15.29:0/2394891962 >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 0x7fda980cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.417+0000 7fda9e772640 1 --2- 172.21.15.29:0/2394891962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda980ccfb0 0x7fda980cd460 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fda90009e50 tx=0x7fda9002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9fc9565a3797f84 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.417+0000 7fda9cf6f640 1 -- 172.21.15.29:0/2394891962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9002ca80 con 0x7fda980ccfb0 2023-10-14T16:57:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.417+0000 7fda9cf6f640 1 -- 172.21.15.29:0/2394891962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9002cbe0 con 0x7fda980ccfb0 2023-10-14T16:57:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.417+0000 7fda9cf6f640 1 -- 172.21.15.29:0/2394891962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda900343f0 con 0x7fda980ccfb0 2023-10-14T16:57:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 -- 172.21.15.29:0/2394891962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda980ccfb0 msgr2=0x7fda980cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2394891962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda980ccfb0 0x7fda980cd460 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7fda90009e50 tx=0x7fda9002bee0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 -- 172.21.15.29:0/2394891962 shutdown_connections 2023-10-14T16:57:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2394891962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda980ccfb0 0x7fda980cd460 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2394891962 >> v2:172.21.15.186:3300/0 conn(0x7fda980ad970 0x7fda980cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 -- 172.21.15.29:0/2394891962 >> 172.21.15.29:0/2394891962 conn(0x7fda980ce6a0 msgr2=0x7fda980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 -- 172.21.15.29:0/2394891962 shutdown_connections 2023-10-14T16:57:01.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.418+0000 7fda9ef73640 1 -- 172.21.15.29:0/2394891962 wait complete. 2023-10-14T16:57:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.420+0000 7fda9ef73640 1 Processor -- start 2023-10-14T16:57:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.420+0000 7fda9ef73640 1 -- start start 2023-10-14T16:57:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda980ad970 0x7fda980b0510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda980ccfb0 0x7fda980b0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980a8720 con 0x7fda9806c040 2023-10-14T16:57:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda980b2980 con 0x7fda980ad970 2023-10-14T16:57:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9ef73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda980b2c60 con 0x7fda980ccfb0 2023-10-14T16:57:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9df71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9df71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32974/0 (socket says 172.21.15.29:32974) 2023-10-14T16:57:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.421+0000 7fda9df71640 1 -- 172.21.15.29:0/2789036496 learned_addr learned my addr 172.21.15.29:0/2789036496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9e772640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda980ccfb0 0x7fda980b0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9d770640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda980ad970 0x7fda980b0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 -- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda980ccfb0 msgr2=0x7fda980b0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda980ccfb0 0x7fda980b0a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 -- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda980ad970 msgr2=0x7fda980b0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda980ad970 0x7fda980b0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 -- 172.21.15.29:0/2789036496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90009b30 con 0x7fda9806c040 2023-10-14T16:57:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.422+0000 7fda9df71640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fda9400c430 tx=0x7fda9400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.423+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94011790 con 0x7fda9806c040 2023-10-14T16:57:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.423+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda940118f0 con 0x7fda9806c040 2023-10-14T16:57:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.423+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda94011ce0 con 0x7fda9806c040 2023-10-14T16:57:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.423+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda980b2f40 con 0x7fda9806c040 2023-10-14T16:57:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.423+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda98156fb0 con 0x7fda9806c040 2023-10-14T16:57:01.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.424+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9806b330 con 0x7fda9806c040 2023-10-14T16:57:01.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.425+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fda9401c460 con 0x7fda9806c040 2023-10-14T16:57:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.426+0000 7fda8e7fc640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 0x7fda7c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.427+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fda940894a0 con 0x7fda9806c040 2023-10-14T16:57:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.427+0000 7fda9d770640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 0x7fda7c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.427+0000 7fda9d770640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 0x7fda7c06d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda980b4770 tx=0x7fda88008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.431+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda940546c0 con 0x7fda9806c040 2023-10-14T16:57:01.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.696+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fda980b3460 con 0x7fda9806c040 2023-10-14T16:57:01.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.697+0000 7fda8e7fc640 1 -- 172.21.15.29:0/2789036496 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fda980b3460 con 0x7fda9806c040 2023-10-14T16:57:01.698 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:01.698 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:57:01.698 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:57:01.698 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.699 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:57:01.699 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:57:01.699 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.699 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.699 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:01.700 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:01.700 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.700 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.700 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.700 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.701 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:57:01.701 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:57:01.701 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.701 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.702 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.703 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:57:01.703 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:57:01.703 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.703 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.704 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:57:01.705 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:57:01.705 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:01.705 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:01.706 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.706 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.706 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.706 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.706 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.707 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.708 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.708 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.708 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:57:01.708 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:57:01.708 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.709 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.710 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.711 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.711 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.711 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.711 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.711 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:57:01.712 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:57:01.712 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.712 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.712 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:57:01.713 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.714 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.714 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.714 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.714 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.715 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.716 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.717 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:57:01.717 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:57:01.717 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.717 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.717 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.718 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.718 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.718 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.718 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.718 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.719 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.720 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.720 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.720 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:57:01.720 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:57:01.720 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.721 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.721 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T16:57:01.721 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T16:57:01.721 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.721 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:57:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:57:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.722 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.723 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:01.723 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:01.723 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:01.723 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.723 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.724 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.724 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:57:01.724 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:57:01.724 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.724 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.725 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:57:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:57:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:01.726 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:01.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:01.727 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:01.727 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.727 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.727 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:01.728 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:57:01.729 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:57:01.730 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:01.730 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:57:01.730 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.730 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.731 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:01.731 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:01.731 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:57:01.731 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:57:01.731 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:01.732 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:01.732 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:57:01.732 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:01.732 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.732 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:01.733 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:57:01.733 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:57:01.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.699+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 msgr2=0x7fda7c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.699+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 0x7fda7c06d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fda980b4770 tx=0x7fda88008040 comp rx=0 tx=0).stop 2023-10-14T16:57:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.699+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 msgr2=0x7fda98156a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.699+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7fda9400c430 tx=0x7fda9400c8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 shutdown_connections 2023-10-14T16:57:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fda7c06afc0 0x7fda7c06d480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda980ccfb0 0x7fda980b0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda980ad970 0x7fda980b0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 --2- 172.21.15.29:0/2789036496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda9806c040 0x7fda98156a70 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 >> 172.21.15.29:0/2789036496 conn(0x7fda980ce6a0 msgr2=0x7fda9806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:01.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 shutdown_connections 2023-10-14T16:57:01.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.700+0000 7fda9ef73640 1 -- 172.21.15.29:0/2789036496 wait complete. 2023-10-14T16:57:01.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:01.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.965+0000 7fb7ea535640 1 Processor -- start 2023-10-14T16:57:01.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.965+0000 7fb7ea535640 1 -- start start 2023-10-14T16:57:01.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.967+0000 7fb7ea535640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e415dde0 0x7fb7e40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.967+0000 7fb7ea535640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 0x7fb7e40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.967+0000 7fb7ea535640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e4160a90 con 0x7fb7e415c9f0 2023-10-14T16:57:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.967+0000 7fb7ea535640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e4071350 con 0x7fb7e415dde0 2023-10-14T16:57:01.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.967+0000 7fb7ea535640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e40cdec0 con 0x7fb7e4070f20 2023-10-14T16:57:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e415dde0 0x7fb7e40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb7e415dde0 0x7fb7e40709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32988/0 (socket says 172.21.15.29:32988) 2023-10-14T16:57:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 -- 172.21.15.29:0/656376682 learned_addr learned my addr 172.21.15.29:0/656376682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e9d34640 1 --2- 172.21.15.29:0/656376682 >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 0x7fb7e40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 -- 172.21.15.29:0/656376682 >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 msgr2=0x7fb7e40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 --2- 172.21.15.29:0/656376682 >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 0x7fb7e40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 -- 172.21.15.29:0/656376682 >> v1:172.21.15.186:6789/0 conn(0x7fb7e415c9f0 legacy=0x7fb7e40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e8d32640 1 -- 172.21.15.29:0/656376682 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e40cd900 con 0x7fb7e415dde0 2023-10-14T16:57:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.968+0000 7fb7e9d34640 1 --2- 172.21.15.29:0/656376682 >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 0x7fb7e40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:01.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.969+0000 7fb7e8d32640 1 --2- 172.21.15.29:0/656376682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e40709e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb7e406fcc0 tx=0x7fb7d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33dd43a237926cf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7dbfff640 1 -- 172.21.15.29:0/656376682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4031870 con 0x7fb7e415dde0 2023-10-14T16:57:01.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7dbfff640 1 -- 172.21.15.29:0/656376682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d40319d0 con 0x7fb7e415dde0 2023-10-14T16:57:01.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7dbfff640 1 -- 172.21.15.29:0/656376682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4031d60 con 0x7fb7e415dde0 2023-10-14T16:57:01.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7ea535640 1 -- 172.21.15.29:0/656376682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 msgr2=0x7fb7e40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7ea535640 1 --2- 172.21.15.29:0/656376682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e40709e0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb7e406fcc0 tx=0x7fb7d402b910 comp rx=0 tx=0).stop 2023-10-14T16:57:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7ea535640 1 -- 172.21.15.29:0/656376682 shutdown_connections 2023-10-14T16:57:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7ea535640 1 --2- 172.21.15.29:0/656376682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e40709e0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.970+0000 7fb7ea535640 1 --2- 172.21.15.29:0/656376682 >> v2:172.21.15.186:3300/0 conn(0x7fb7e4070f20 0x7fb7e40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.971+0000 7fb7ea535640 1 -- 172.21.15.29:0/656376682 >> 172.21.15.29:0/656376682 conn(0x7fb7e40ce6a0 msgr2=0x7fb7e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.971+0000 7fb7ea535640 1 -- 172.21.15.29:0/656376682 shutdown_connections 2023-10-14T16:57:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.971+0000 7fb7ea535640 1 -- 172.21.15.29:0/656376682 wait complete. 2023-10-14T16:57:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.972+0000 7fb7ea535640 1 Processor -- start 2023-10-14T16:57:01.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.972+0000 7fb7ea535640 1 -- start start 2023-10-14T16:57:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 0x7fb7e40c2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb7e415c9f0 0x7fb7e40c3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e406ba70 con 0x7fb7e415dde0 2023-10-14T16:57:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e406adf0 con 0x7fb7e4070f20 2023-10-14T16:57:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.973+0000 7fb7ea535640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e406b0d0 con 0x7fb7e415c9f0 2023-10-14T16:57:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:32998/0 (socket says 172.21.15.29:32998) 2023-10-14T16:57:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 -- 172.21.15.29:0/1514925930 learned_addr learned my addr 172.21.15.29:0/1514925930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 -- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb7e415c9f0 msgr2=0x7fb7e40c3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb7e415c9f0 0x7fb7e40c3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 -- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 msgr2=0x7fb7e40c2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9533640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 0x7fb7e40c2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 0x7fb7e40c2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9d34640 1 -- 172.21.15.29:0/1514925930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d40094d0 con 0x7fb7e415dde0 2023-10-14T16:57:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.974+0000 7fb7e9533640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 0x7fb7e40c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7e9d34640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00cef0 tx=0x7fb7dc00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00d840 con 0x7fb7e415dde0 2023-10-14T16:57:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7dc00d9c0 con 0x7fb7e415dde0 2023-10-14T16:57:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7dc00dd70 con 0x7fb7e415dde0 2023-10-14T16:57:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e406b3b0 con 0x7fb7e415dde0 2023-10-14T16:57:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.975+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e40c3c10 con 0x7fb7e415dde0 2023-10-14T16:57:01.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.976+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e404e9b0 con 0x7fb7e415dde0 2023-10-14T16:57:01.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.977+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb7dc01e3b0 con 0x7fb7e415dde0 2023-10-14T16:57:01.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.978+0000 7fb7d9ffb640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 0x7fb7c806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.978+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fb7dc089fd0 con 0x7fb7e415dde0 2023-10-14T16:57:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.979+0000 7fb7e9533640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 0x7fb7c806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:01.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.979+0000 7fb7e9533640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 0x7fb7c806d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0003eb0 tx=0x7fb7e0009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:01.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:01.983+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb7dc0552b0 con 0x7fb7e415dde0 2023-10-14T16:57:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.182+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7e40ae330 con 0x7fb7e415dde0 2023-10-14T16:57:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.183+0000 7fb7d9ffb640 1 -- 172.21.15.29:0/1514925930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v92) v1 ==== 74+0+12915 (secure 0 0 0) 0x7fb7dc01a090 con 0x7fb7e415dde0 2023-10-14T16:57:02.184 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:02.184 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":92,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:01.164428+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":9,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":80,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.185+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 msgr2=0x7fb7c806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.185+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 0x7fb7c806d480 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb7e0003eb0 tx=0x7fb7e0009040 comp rx=0 tx=0).stop 2023-10-14T16:57:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.185+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 msgr2=0x7fb7e406bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.185+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb7dc00cef0 tx=0x7fb7dc00a710 comp rx=0 tx=0).stop 2023-10-14T16:57:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 shutdown_connections 2023-10-14T16:57:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb7c806afc0 0x7fb7c806d480 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb7e415dde0 0x7fb7e406bdb0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb7e415c9f0 0x7fb7e40c3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 --2- 172.21.15.29:0/1514925930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb7e4070f20 0x7fb7e40c2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 >> 172.21.15.29:0/1514925930 conn(0x7fb7e40ce6a0 msgr2=0x7fb7e40cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 shutdown_connections 2023-10-14T16:57:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.186+0000 7fb7ea535640 1 -- 172.21.15.29:0/1514925930 wait complete. 2023-10-14T16:57:02.206 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:57:02.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:57:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.440+0000 7f6a25034640 1 Processor -- start 2023-10-14T16:57:02.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.440+0000 7f6a25034640 1 -- start start 2023-10-14T16:57:02.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a25034640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6a20159ab0 0x7f6a200ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a25034640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a2006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a25034640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a200cd1f0 con 0x7f6a2006c140 2023-10-14T16:57:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a25034640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2015d890 con 0x7f6a20159ab0 2023-10-14T16:57:02.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a25034640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200cd4d0 con 0x7f6a20159e90 2023-10-14T16:57:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a1f59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6a2006c140 0x7f6a2015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56306/0 (socket says 172.21.15.29:56306) 2023-10-14T16:57:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.442+0000 7f6a1f59e640 1 -- 172.21.15.29:0/1707204661 learned_addr learned my addr 172.21.15.29:0/1707204661 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:02.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1e59c640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a2006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1ed9d640 1 --2- 172.21.15.29:0/1707204661 >> v2:172.21.15.186:3300/0 conn(0x7f6a20159ab0 0x7f6a200ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291740591 0 0) 0x7f6a200cd1f0 con 0x7f6a2006c140 2023-10-14T16:57:02.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a08003520 con 0x7f6a2006c140 2023-10-14T16:57:02.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1466293869 0 0) 0x7f6a08003520 con 0x7f6a2006c140 2023-10-14T16:57:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 msgr2=0x7f6a2006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a2006fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 >> v2:172.21.15.186:3300/0 conn(0x7f6a20159ab0 msgr2=0x7f6a200ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 --2- 172.21.15.29:0/1707204661 >> v2:172.21.15.186:3300/0 conn(0x7f6a20159ab0 0x7f6a200ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a200cd7b0 con 0x7f6a2006c140 2023-10-14T16:57:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6a180019f0 con 0x7f6a2006c140 2023-10-14T16:57:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 >> v1:172.21.15.29:6789/0 conn(0x7f6a2006c140 legacy=0x7f6a2015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08003970 0x7f6a08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a08004d00 0x7f6a08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2015d890 con 0x7f6a08004d00 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a200cd4d0 con 0x7f6a08003970 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.443+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a0800ba50 con 0x7f6a20159e90 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a08004d00 0x7f6a08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1e59c640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1ed9d640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08003970 0x7f6a08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 -- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 msgr2=0x7f6a080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a20159e90 0x7f6a080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 -- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08003970 msgr2=0x7f6a08003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08003970 0x7f6a08003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.444+0000 7f6a1f59e640 1 -- 172.21.15.29:0/1707204661 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a08003520 con 0x7f6a08004d00 2023-10-14T16:57:02.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.445+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a08004d00 0x7f6a08005180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800a750 tx=0x7f6a080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a317d78c5833a6f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18034630 con 0x7f6a08004d00 2023-10-14T16:57:02.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1802f070 con 0x7f6a08004d00 2023-10-14T16:57:02.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a1dd9b640 1 -- 172.21.15.29:0/1707204661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a180349d0 con 0x7f6a08004d00 2023-10-14T16:57:02.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a25034640 1 -- 172.21.15.29:0/1707204661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a08004d00 msgr2=0x7f6a08005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a25034640 1 --2- 172.21.15.29:0/1707204661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a08004d00 0x7f6a08005180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800a750 tx=0x7f6a080057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a1f59e640 1 -- 172.21.15.29:0/1707204661 reap_dead start 2023-10-14T16:57:02.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a25034640 1 -- 172.21.15.29:0/1707204661 shutdown_connections 2023-10-14T16:57:02.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.446+0000 7f6a25034640 1 -- 172.21.15.29:0/1707204661 >> 172.21.15.29:0/1707204661 conn(0x7f6a200ce9b0 msgr2=0x7f6a200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:02.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.447+0000 7f6a25034640 1 -- 172.21.15.29:0/1707204661 shutdown_connections 2023-10-14T16:57:02.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.447+0000 7f6a25034640 1 -- 172.21.15.29:0/1707204661 wait complete. 2023-10-14T16:57:02.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.449+0000 7f6a25034640 1 Processor -- start 2023-10-14T16:57:02.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.449+0000 7f6a25034640 1 -- start start 2023-10-14T16:57:02.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08004d00 0x7f6a20070280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 0x7f6a200c8660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2015c750 con 0x7f6a2015a330 2023-10-14T16:57:02.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2015bb40 con 0x7f6a08004d00 2023-10-14T16:57:02.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a25034640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a2015be20 con 0x7f6a200c81e0 2023-10-14T16:57:02.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a1f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a1f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33018/0 (socket says 172.21.15.29:33018) 2023-10-14T16:57:02.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a1f59e640 1 -- 172.21.15.29:0/3042032635 learned_addr learned my addr 172.21.15.29:0/3042032635 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:02.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a1e59c640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 0x7f6a200c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.450+0000 7f6a1ed9d640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08004d00 0x7f6a20070280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 -- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 msgr2=0x7f6a200c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 0x7f6a200c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 -- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08004d00 msgr2=0x7f6a20070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08004d00 0x7f6a20070280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 -- 172.21.15.29:0/3042032635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2015c100 con 0x7f6a2015a330 2023-10-14T16:57:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1e59c640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 0x7f6a200c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a1f59e640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800a750 tx=0x7f6a2006f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18034430 con 0x7f6a2015a330 2023-10-14T16:57:02.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a18034590 con 0x7f6a2015a330 2023-10-14T16:57:02.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a18034900 con 0x7f6a2015a330 2023-10-14T16:57:02.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.451+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2015c3e0 con 0x7f6a2015a330 2023-10-14T16:57:02.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.452+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a200707c0 con 0x7f6a2015a330 2023-10-14T16:57:02.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.453+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a200709e0 con 0x7f6a2015a330 2023-10-14T16:57:02.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.454+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6a18031030 con 0x7f6a2015a330 2023-10-14T16:57:02.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.455+0000 7f6a077fe640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 0x7f6a0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.455+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f6a180ab8d0 con 0x7f6a2015a330 2023-10-14T16:57:02.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.455+0000 7f6a1ed9d640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 0x7f6a0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.456+0000 7f6a1ed9d640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 0x7f6a0c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009a90 tx=0x7f6a10008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.460+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6a18076ba0 con 0x7f6a2015a330 2023-10-14T16:57:02.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.656+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6a200709e0 con 0x7f6a2015a330 2023-10-14T16:57:02.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.657+0000 7f6a077fe640 1 -- 172.21.15.29:0/3042032635 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v92) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6a1807a1c0 con 0x7f6a2015a330 2023-10-14T16:57:02.658 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:57:02.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 msgr2=0x7f6a0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 0x7f6a0c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f6a10009a90 tx=0x7f6a10008040 comp rx=0 tx=0).stop 2023-10-14T16:57:02.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 msgr2=0x7f6a2015a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7f6a1800a750 tx=0x7f6a2006f7c0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 shutdown_connections 2023-10-14T16:57:02.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6a0c06af90 0x7f6a0c06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6a2015a330 0x7f6a2015a7b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6a200c81e0 0x7f6a200c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 --2- 172.21.15.29:0/3042032635 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6a08004d00 0x7f6a20070280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 >> 172.21.15.29:0/3042032635 conn(0x7f6a200ce9b0 msgr2=0x7f6a20070d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:02.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.660+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 shutdown_connections 2023-10-14T16:57:02.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.661+0000 7f6a25034640 1 -- 172.21.15.29:0/3042032635 wait complete. 2023-10-14T16:57:02.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:57:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.913+0000 7f6285864640 1 Processor -- start 2023-10-14T16:57:02.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.913+0000 7f6285864640 1 -- start start 2023-10-14T16:57:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.914+0000 7f6285864640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f628015b320 0x7f62800cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.914+0000 7f6285864640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628015b9d0 con 0x7f62800ad970 2023-10-14T16:57:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.914+0000 7f6285864640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628015bcb0 con 0x7f628006c040 2023-10-14T16:57:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.914+0000 7f6285864640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628015b7d0 con 0x7f628015b320 2023-10-14T16:57:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f6285063640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f628015b320 0x7f62800cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f627ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f62800ad970 0x7f62800cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56308/0 (socket says 172.21.15.29:56308) 2023-10-14T16:57:02.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f627ffff640 1 -- 172.21.15.29:0/254060186 learned_addr learned my addr 172.21.15.29:0/254060186 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f6285063640 1 -- 172.21.15.29:0/254060186 >> v1:172.21.15.186:6789/0 conn(0x7f628006c040 legacy=0x7f628006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f6285063640 1 -- 172.21.15.29:0/254060186 >> v1:172.21.15.29:6789/0 conn(0x7f62800ad970 legacy=0x7f62800cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.915+0000 7f6285063640 1 -- 172.21.15.29:0/254060186 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628015bf90 con 0x7f628015b320 2023-10-14T16:57:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.916+0000 7f6285063640 1 --2- 172.21.15.29:0/254060186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62800cd180 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6278009dc0 tx=0x7f627802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7d8dde61e3fa041 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f627f7fe640 1 -- 172.21.15.29:0/254060186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627802cc40 con 0x7f628015b320 2023-10-14T16:57:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f627f7fe640 1 -- 172.21.15.29:0/254060186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627802cda0 con 0x7f628015b320 2023-10-14T16:57:02.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f627f7fe640 1 -- 172.21.15.29:0/254060186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6278034660 con 0x7f628015b320 2023-10-14T16:57:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f6285864640 1 -- 172.21.15.29:0/254060186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 msgr2=0x7f62800cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f6285864640 1 --2- 172.21.15.29:0/254060186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62800cd180 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f6278009dc0 tx=0x7f627802be90 comp rx=0 tx=0).stop 2023-10-14T16:57:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f6285864640 1 -- 172.21.15.29:0/254060186 shutdown_connections 2023-10-14T16:57:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f6285864640 1 --2- 172.21.15.29:0/254060186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62800cd180 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.917+0000 7f6285864640 1 -- 172.21.15.29:0/254060186 >> 172.21.15.29:0/254060186 conn(0x7f62800ce6a0 msgr2=0x7f62800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.918+0000 7f6285864640 1 -- 172.21.15.29:0/254060186 shutdown_connections 2023-10-14T16:57:02.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.918+0000 7f6285864640 1 -- 172.21.15.29:0/254060186 wait complete. 2023-10-14T16:57:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.919+0000 7f6285864640 1 Processor -- start 2023-10-14T16:57:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.919+0000 7f6285864640 1 -- start start 2023-10-14T16:57:02.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f628006c040 0x7f62800b0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62800ad970 0x7f62800aeac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800af000 con 0x7f628015b320 2023-10-14T16:57:02.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62800af280 con 0x7f62800ad970 2023-10-14T16:57:02.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.920+0000 7f6285864640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62800af560 con 0x7f628006c040 2023-10-14T16:57:02.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33048/0 (socket says 172.21.15.29:33048) 2023-10-14T16:57:02.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 -- 172.21.15.29:0/404262578 learned_addr learned my addr 172.21.15.29:0/404262578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:02.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6284862640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f628006c040 0x7f62800b0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f627ffff640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62800ad970 0x7f62800aeac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 -- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f628006c040 msgr2=0x7f62800b0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f628006c040 0x7f62800b0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 -- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62800ad970 msgr2=0x7f62800aeac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:02.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62800ad970 0x7f62800aeac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.921+0000 7f6285063640 1 -- 172.21.15.29:0/404262578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6278009a70 con 0x7f628015b320 2023-10-14T16:57:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f6285063640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6278037040 tx=0x7f6278004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627803e610 con 0x7f628015b320 2023-10-14T16:57:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627803e770 con 0x7f628015b320 2023-10-14T16:57:02.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627803eb00 con 0x7f628015b320 2023-10-14T16:57:02.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62800af840 con 0x7f628015b320 2023-10-14T16:57:02.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.922+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62800b1f70 con 0x7f628015b320 2023-10-14T16:57:02.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.923+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628006b330 con 0x7f628015b320 2023-10-14T16:57:02.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.924+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f627803b050 con 0x7f628015b320 2023-10-14T16:57:02.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.925+0000 7f627d7fa640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 0x7f626406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:02.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.925+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f62780abee0 con 0x7f628015b320 2023-10-14T16:57:02.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.926+0000 7f6284862640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 0x7f626406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:02.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.926+0000 7f6284862640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 0x7f626406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6274006fd0 tx=0x7f6274008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:02.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:02.930+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f62780772b0 con 0x7f628015b320 2023-10-14T16:57:03.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.133+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f628004e9b0 con 0x7f628015b320 2023-10-14T16:57:03.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.134+0000 7f627d7fa640 1 -- 172.21.15.29:0/404262578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v92) v1 ==== 84+0+65 (secure 0 0 0) 0x7f627804e060 con 0x7f628015b320 2023-10-14T16:57:03.135 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:03.135 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:57:03.136 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:03.136 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:57:03.136 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:57:03.136 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:03.136 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:57:03.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 msgr2=0x7f626406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 0x7f626406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6274006fd0 tx=0x7f6274008040 comp rx=0 tx=0).stop 2023-10-14T16:57:03.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 msgr2=0x7f62801e6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f6278037040 tx=0x7f6278004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:03.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 shutdown_connections 2023-10-14T16:57:03.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f626406af90 0x7f626406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628015b320 0x7f62801e6210 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f62800ad970 0x7f62800aeac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 --2- 172.21.15.29:0/404262578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f628006c040 0x7f62800b0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.137+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 >> 172.21.15.29:0/404262578 conn(0x7f62800ce6a0 msgr2=0x7f62800cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:03.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.138+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 shutdown_connections 2023-10-14T16:57:03.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.138+0000 7f6285864640 1 -- 172.21.15.29:0/404262578 wait complete. 2023-10-14T16:57:03.158 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:57:03.158 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:57:03.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.385+0000 7fa1145a5640 1 Processor -- start 2023-10-14T16:57:03.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.386+0000 7fa1145a5640 1 -- start start 2023-10-14T16:57:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.387+0000 7fa1145a5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa10c0ce690 0x7fa10c0c9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.387+0000 7fa1145a5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c06adb0 0x7fa10c06b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.387+0000 7fa1145a5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa10c0c9d60 con 0x7fa10c0ae220 2023-10-14T16:57:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.387+0000 7fa1145a5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa10c06b780 con 0x7fa10c0ce690 2023-10-14T16:57:03.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.387+0000 7fa1145a5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c0cae30 con 0x7fa10c06adb0 2023-10-14T16:57:03.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa10c0ce690 0x7fa10c0c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa10c0ce690 0x7fa10c0c9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33050/0 (socket says 172.21.15.29:33050) 2023-10-14T16:57:03.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 -- 172.21.15.29:0/4128132951 learned_addr learned my addr 172.21.15.29:0/4128132951 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:03.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112b42640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c06adb0 0x7fa10c06b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 -- 172.21.15.29:0/4128132951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c06adb0 msgr2=0x7fa10c06b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:03.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c06adb0 0x7fa10c06b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 -- 172.21.15.29:0/4128132951 >> v1:172.21.15.186:6789/0 conn(0x7fa10c0ae220 legacy=0x7fa10c06e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:03.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.388+0000 7fa112341640 1 -- 172.21.15.29:0/4128132951 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa10c0cb110 con 0x7fa10c0ce690 2023-10-14T16:57:03.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.389+0000 7fa112341640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c0ce690 0x7fa10c0c9820 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fa108009870 tx=0x7fa10802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b7ee783235afef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa11133f640 1 -- 172.21.15.29:0/4128132951 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108031640 con 0x7fa10c0ce690 2023-10-14T16:57:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa11133f640 1 -- 172.21.15.29:0/4128132951 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1080317a0 con 0x7fa10c0ce690 2023-10-14T16:57:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa11133f640 1 -- 172.21.15.29:0/4128132951 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108031b30 con 0x7fa10c0ce690 2023-10-14T16:57:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 -- 172.21.15.29:0/4128132951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c0ce690 msgr2=0x7fa10c0c9820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c0ce690 0x7fa10c0c9820 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fa108009870 tx=0x7fa10802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 -- 172.21.15.29:0/4128132951 shutdown_connections 2023-10-14T16:57:03.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c06adb0 0x7fa10c06b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 --2- 172.21.15.29:0/4128132951 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c0ce690 0x7fa10c0c9820 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.390+0000 7fa1145a5640 1 -- 172.21.15.29:0/4128132951 >> 172.21.15.29:0/4128132951 conn(0x7fa10c0b0cc0 msgr2=0x7fa10c0b10d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:03.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.391+0000 7fa1145a5640 1 -- 172.21.15.29:0/4128132951 shutdown_connections 2023-10-14T16:57:03.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.391+0000 7fa1145a5640 1 -- 172.21.15.29:0/4128132951 wait complete. 2023-10-14T16:57:03.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.392+0000 7fa1145a5640 1 Processor -- start 2023-10-14T16:57:03.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.393+0000 7fa1145a5640 1 -- start start 2023-10-14T16:57:03.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.393+0000 7fa1145a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.393+0000 7fa1145a5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 0x7fa10c15c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa1145a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa10c0ce690 0x7fa10c15def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa1145a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c0af540 con 0x7fa10c06adb0 2023-10-14T16:57:03.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa1145a5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c0af010 con 0x7fa10c0ce690 2023-10-14T16:57:03.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa1145a5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa10c09b490 con 0x7fa10c0ae220 2023-10-14T16:57:03.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33062/0 (socket says 172.21.15.29:33062) 2023-10-14T16:57:03.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 -- 172.21.15.29:0/3037837161 learned_addr learned my addr 172.21.15.29:0/3037837161 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:03.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa111b40640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 0x7fa10c15c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112b42640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa10c0ce690 0x7fa10c15def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 -- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 msgr2=0x7fa10c15c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 0x7fa10c15c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 -- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa10c0ce690 msgr2=0x7fa10c15def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa10c0ce690 0x7fa10c15def0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa112341640 1 -- 172.21.15.29:0/3037837161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080094d0 con 0x7fa10c06adb0 2023-10-14T16:57:03.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.394+0000 7fa111b40640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 0x7fa10c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:03.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa112341640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa10802b930 tx=0x7fa1080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa10803ac60 con 0x7fa10c06adb0 2023-10-14T16:57:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10803adc0 con 0x7fa10c06adb0 2023-10-14T16:57:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa108042630 con 0x7fa10c06adb0 2023-10-14T16:57:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa10c0af260 con 0x7fa10c06adb0 2023-10-14T16:57:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.395+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa10c09b120 con 0x7fa10c06adb0 2023-10-14T16:57:03.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.396+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa10c09aca0 con 0x7fa10c06adb0 2023-10-14T16:57:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.397+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa10802f050 con 0x7fa10c06adb0 2023-10-14T16:57:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.399+0000 7fa102ffd640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 0x7fa0f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.399+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fa1080ab200 con 0x7fa10c06adb0 2023-10-14T16:57:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.399+0000 7fa111b40640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 0x7fa0f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.400+0000 7fa111b40640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 0x7fa0f006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0aefc0 tx=0x7fa0fc009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.403+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa10c09aca0 con 0x7fa10c06adb0 2023-10-14T16:57:03.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.607+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa10c09aca0 con 0x7fa10c06adb0 2023-10-14T16:57:03.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.716+0000 7fa102ffd640 1 -- 172.21.15.29:0/3037837161 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v93) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa10c09aca0 con 0x7fa10c06adb0 2023-10-14T16:57:03.718 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:57:03.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 msgr2=0x7fa0f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 0x7fa0f006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa10c0aefc0 tx=0x7fa0fc009040 comp rx=0 tx=0).stop 2023-10-14T16:57:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 msgr2=0x7fa10c09fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa10802b930 tx=0x7fa1080046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 shutdown_connections 2023-10-14T16:57:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fa0f006af90 0x7fa0f006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa10c0ce690 0x7fa10c15def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa10c0ae220 0x7fa10c15c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 --2- 172.21.15.29:0/3037837161 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa10c06adb0 0x7fa10c09fdf0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 >> 172.21.15.29:0/3037837161 conn(0x7fa10c0b0cc0 msgr2=0x7fa10c0afef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:03.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 shutdown_connections 2023-10-14T16:57:03.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.720+0000 7fa1145a5640 1 -- 172.21.15.29:0/3037837161 wait complete. 2023-10-14T16:57:03.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:57:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.972+0000 7f564b59e640 1 Processor -- start 2023-10-14T16:57:03.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.973+0000 7f564b59e640 1 -- start start 2023-10-14T16:57:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.974+0000 7f564b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f564c15cb50 0x7f564c161180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.974+0000 7f564b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b1690 0x7f564c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.974+0000 7f564b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f564c0b4f00 con 0x7f564c0b4ab0 2023-10-14T16:57:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.974+0000 7f564b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564c1616c0 con 0x7f564c15cb50 2023-10-14T16:57:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.974+0000 7f564b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0cdbf0 con 0x7f564c0b1690 2023-10-14T16:57:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f564c15cb50 0x7f564c161180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f564c15cb50 0x7f564c161180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33076/0 (socket says 172.21.15.29:33076) 2023-10-14T16:57:03.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 -- 172.21.15.29:0/2626644236 learned_addr learned my addr 172.21.15.29:0/2626644236 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:03.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f5649d9b640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b1690 0x7f564c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 -- 172.21.15.29:0/2626644236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b1690 msgr2=0x7f564c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:03.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b1690 0x7f564c0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 -- 172.21.15.29:0/2626644236 >> v1:172.21.15.186:6789/0 conn(0x7f564c0b4ab0 legacy=0x7f564c06bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:03.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.975+0000 7f564a59c640 1 -- 172.21.15.29:0/2626644236 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c0cde90 con 0x7f564c15cb50 2023-10-14T16:57:03.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.977+0000 7f564a59c640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c15cb50 0x7f564c161180 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f563c009e50 tx=0x7f563c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f24ac4304c220ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.977+0000 7f564959a640 1 -- 172.21.15.29:0/2626644236 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c02c9f0 con 0x7f564c15cb50 2023-10-14T16:57:03.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.977+0000 7f564959a640 1 -- 172.21.15.29:0/2626644236 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c02cb50 con 0x7f564c15cb50 2023-10-14T16:57:03.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.977+0000 7f564959a640 1 -- 172.21.15.29:0/2626644236 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c0343f0 con 0x7f564c15cb50 2023-10-14T16:57:03.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.977+0000 7f564b59e640 1 -- 172.21.15.29:0/2626644236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c15cb50 msgr2=0x7f564c161180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c15cb50 0x7f564c161180 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7f563c009e50 tx=0x7f563c02be50 comp rx=0 tx=0).stop 2023-10-14T16:57:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 -- 172.21.15.29:0/2626644236 shutdown_connections 2023-10-14T16:57:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b1690 0x7f564c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 --2- 172.21.15.29:0/2626644236 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c15cb50 0x7f564c161180 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 -- 172.21.15.29:0/2626644236 >> 172.21.15.29:0/2626644236 conn(0x7f564c0cf570 msgr2=0x7f564c0cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:03.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 -- 172.21.15.29:0/2626644236 shutdown_connections 2023-10-14T16:57:03.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.978+0000 7f564b59e640 1 -- 172.21.15.29:0/2626644236 wait complete. 2023-10-14T16:57:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.980+0000 7f564b59e640 1 Processor -- start 2023-10-14T16:57:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.980+0000 7f564b59e640 1 -- start start 2023-10-14T16:57:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b4ab0 0x7f564c0cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f564c15cb50 0x7f564c0c6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c14f450 con 0x7f564c0b1690 2023-10-14T16:57:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0c9f10 con 0x7f564c15cb50 2023-10-14T16:57:03.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f564c0cbbd0 con 0x7f564c0b4ab0 2023-10-14T16:57:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33078/0 (socket says 172.21.15.29:33078) 2023-10-14T16:57:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564a59c640 1 -- 172.21.15.29:0/3556302472 learned_addr learned my addr 172.21.15.29:0/3556302472 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f564ad9d640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f564c15cb50 0x7f564c0c6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.981+0000 7f5649d9b640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b4ab0 0x7f564c0cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 -- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b4ab0 msgr2=0x7f564c0cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b4ab0 0x7f564c0cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 -- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f564c15cb50 msgr2=0x7f564c0c6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f564c15cb50 0x7f564c0c6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 -- 172.21.15.29:0/3556302472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f563c009b30 con 0x7f564c0b1690 2023-10-14T16:57:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564a59c640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f563c009f70 tx=0x7f563c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c03e450 con 0x7f564c0b1690 2023-10-14T16:57:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563c03e5d0 con 0x7f564c0b1690 2023-10-14T16:57:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563c03e980 con 0x7f564c0b1690 2023-10-14T16:57:03.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c0cbeb0 con 0x7f564c0b1690 2023-10-14T16:57:03.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.982+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c0cc470 con 0x7f564c0b1690 2023-10-14T16:57:03.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.984+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c04e9b0 con 0x7f564c0b1690 2023-10-14T16:57:03.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.985+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f563c03eae0 con 0x7f564c0b1690 2023-10-14T16:57:03.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.986+0000 7f5642ffd640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 0x7f562c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:03.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.986+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5271+0+0 (secure 0 0 0) 0x7f563c0ac8b0 con 0x7f564c0b1690 2023-10-14T16:57:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.986+0000 7f5649d9b640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 0x7f562c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.987+0000 7f5649d9b640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 0x7f562c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5634006fd0 tx=0x7f5634008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:03.990+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f563c0779e0 con 0x7f564c0b1690 2023-10-14T16:57:04.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.190+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f564c159920 con 0x7f564c0b1690 2023-10-14T16:57:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.722+0000 7f5642ffd640 1 -- 172.21.15.29:0/3556302472 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v94) v1 ==== 170+0+0 (secure 0 0 0) 0x7f563c033e80 con 0x7f564c0b1690 2023-10-14T16:57:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:57:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 msgr2=0x7f562c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 0x7f562c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f5634006fd0 tx=0x7f5634008040 comp rx=0 tx=0).stop 2023-10-14T16:57:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 msgr2=0x7f564c0ccc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7f563c009f70 tx=0x7f563c004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 shutdown_connections 2023-10-14T16:57:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f562c06af90 0x7f562c06d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f564c15cb50 0x7f564c0c6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f564c0b4ab0 0x7f564c0cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 --2- 172.21.15.29:0/3556302472 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f564c0b1690 0x7f564c0ccc70 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 >> 172.21.15.29:0/3556302472 conn(0x7f564c0cf570 msgr2=0x7f564c0b4000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 shutdown_connections 2023-10-14T16:57:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.727+0000 7f564b59e640 1 -- 172.21.15.29:0/3556302472 wait complete. 2023-10-14T16:57:04.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:57:04.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.953+0000 7f590a133640 1 Processor -- start 2023-10-14T16:57:04.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.954+0000 7f590a133640 1 -- start start 2023-10-14T16:57:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f590a133640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f590415af80 0x7f59040cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f590a133640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f590406faf0 0x7f590415f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f590a133640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590415eff0 con 0x7f5904065ab0 2023-10-14T16:57:04.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f590a133640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59040d0930 con 0x7f590406faf0 2023-10-14T16:57:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f590a133640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f590415ecc0 con 0x7f590415af80 2023-10-14T16:57:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f58faffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f590406faf0 0x7f590415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f58faffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f590406faf0 0x7f590415f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33092/0 (socket says 172.21.15.29:33092) 2023-10-14T16:57:04.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.955+0000 7f58faffd640 1 -- 172.21.15.29:0/1129407885 learned_addr learned my addr 172.21.15.29:0/1129407885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.956+0000 7f58faffd640 1 -- 172.21.15.29:0/1129407885 >> v2:172.21.15.186:3300/0 conn(0x7f590415af80 msgr2=0x7f59040cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.956+0000 7f58faffd640 1 --2- 172.21.15.29:0/1129407885 >> v2:172.21.15.186:3300/0 conn(0x7f590415af80 0x7f59040cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.956+0000 7f58faffd640 1 -- 172.21.15.29:0/1129407885 >> v1:172.21.15.186:6789/0 conn(0x7f5904065ab0 legacy=0x7f59040ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:04.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.956+0000 7f58faffd640 1 -- 172.21.15.29:0/1129407885 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f590415b360 con 0x7f590406faf0 2023-10-14T16:57:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.957+0000 7f58faffd640 1 --2- 172.21.15.29:0/1129407885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590406faf0 0x7f590415f480 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f59040c7aa0 tx=0x7f58ec02b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ac3c5696a9c6a1a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.957+0000 7f5902ffd640 1 -- 172.21.15.29:0/1129407885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec031890 con 0x7f590406faf0 2023-10-14T16:57:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f5902ffd640 1 -- 172.21.15.29:0/1129407885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58ec0319f0 con 0x7f590406faf0 2023-10-14T16:57:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f5902ffd640 1 -- 172.21.15.29:0/1129407885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58ec031d80 con 0x7f590406faf0 2023-10-14T16:57:04.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 -- 172.21.15.29:0/1129407885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590406faf0 msgr2=0x7f590415f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 --2- 172.21.15.29:0/1129407885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590406faf0 0x7f590415f480 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7f59040c7aa0 tx=0x7f58ec02b930 comp rx=0 tx=0).stop 2023-10-14T16:57:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 -- 172.21.15.29:0/1129407885 shutdown_connections 2023-10-14T16:57:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 --2- 172.21.15.29:0/1129407885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590406faf0 0x7f590415f480 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 --2- 172.21.15.29:0/1129407885 >> v2:172.21.15.186:3300/0 conn(0x7f590415af80 0x7f59040cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 -- 172.21.15.29:0/1129407885 >> 172.21.15.29:0/1129407885 conn(0x7f59040d0ab0 msgr2=0x7f59040d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 -- 172.21.15.29:0/1129407885 shutdown_connections 2023-10-14T16:57:04.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.958+0000 7f590a133640 1 -- 172.21.15.29:0/1129407885 wait complete. 2023-10-14T16:57:04.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.960+0000 7f590a133640 1 Processor -- start 2023-10-14T16:57:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.960+0000 7f590a133640 1 -- start start 2023-10-14T16:57:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f590406faf0 0x7f59041f3e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590415af80 0x7f59040c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59040b36d0 con 0x7f5904065ab0 2023-10-14T16:57:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59041f5f80 con 0x7f590415af80 2023-10-14T16:57:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f590a133640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59040c9c40 con 0x7f590406faf0 2023-10-14T16:57:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f59037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:04.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f59037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33098/0 (socket says 172.21.15.29:33098) 2023-10-14T16:57:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.961+0000 7f59037fe640 1 -- 172.21.15.29:0/2842893389 learned_addr learned my addr 172.21.15.29:0/2842893389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f58faffd640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f590406faf0 0x7f59041f3e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f5903fff640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590415af80 0x7f59040c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 -- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f590406faf0 msgr2=0x7f59041f3e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:04.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f590406faf0 0x7f59041f3e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 -- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590415af80 msgr2=0x7f59040c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590415af80 0x7f59040c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 -- 172.21.15.29:0/2842893389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58ec0094d0 con 0x7f5904065ab0 2023-10-14T16:57:04.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.962+0000 7f59037fe640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f58f400b4a0 tx=0x7f58f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.963+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4010610 con 0x7f5904065ab0 2023-10-14T16:57:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.963+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58f4010770 con 0x7f5904065ab0 2023-10-14T16:57:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.963+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58f4010b20 con 0x7f5904065ab0 2023-10-14T16:57:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.963+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59040b3990 con 0x7f5904065ab0 2023-10-14T16:57:04.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.963+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59040b3f50 con 0x7f5904065ab0 2023-10-14T16:57:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.964+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f590404e160 con 0x7f5904065ab0 2023-10-14T16:57:04.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.965+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f58f4010c80 con 0x7f5904065ab0 2023-10-14T16:57:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.966+0000 7f5900ff9640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 0x7f58e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.966+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f58f4089470 con 0x7f5904065ab0 2023-10-14T16:57:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.966+0000 7f58faffd640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 0x7f58e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:04.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.967+0000 7f58faffd640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 0x7f58e406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58ec036000 tx=0x7f58ec03c040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:04.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:04.971+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58f40543a0 con 0x7f5904065ab0 2023-10-14T16:57:05.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:05.211+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f59040b4140 con 0x7f5904065ab0 2023-10-14T16:57:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.145+0000 7f5900ff9640 1 -- 172.21.15.29:0/2842893389 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 10 and data pool 11 v36) v1 ==== 174+0+0 (secure 0 0 0) 0x7f58f40579c0 con 0x7f5904065ab0 2023-10-14T16:57:06.146 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 10 and data pool 11 2023-10-14T16:57:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 msgr2=0x7f58e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 0x7f58e406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f58ec036000 tx=0x7f58ec03c040 comp rx=0 tx=0).stop 2023-10-14T16:57:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 msgr2=0x7f59040b6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7f58f400b4a0 tx=0x7f58f400b960 comp rx=0 tx=0).stop 2023-10-14T16:57:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 shutdown_connections 2023-10-14T16:57:06.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f58e406af90 0x7f58e406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f590415af80 0x7f59040c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f590406faf0 0x7f59041f3e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 --2- 172.21.15.29:0/2842893389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5904065ab0 0x7f59040b6220 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 >> 172.21.15.29:0/2842893389 conn(0x7f59040d0ab0 msgr2=0x7f59040af720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 shutdown_connections 2023-10-14T16:57:06.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.150+0000 7f590a133640 1 -- 172.21.15.29:0/2842893389 wait complete. 2023-10-14T16:57:06.177 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:06.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.381+0000 7f8fbeaff640 1 Processor -- start 2023-10-14T16:57:06.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.381+0000 7f8fbeaff640 1 -- start start 2023-10-14T16:57:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbeaff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fb806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbeaff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8fb806c9e0 0x7f8fb806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbeaff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fb80ca560 con 0x7f8fb80c9440 2023-10-14T16:57:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbeaff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fb80c9950 con 0x7f8fb806c9e0 2023-10-14T16:57:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbeaff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c9c30 con 0x7f8fb80c9060 2023-10-14T16:57:06.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbe2fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8fb80c9440 0x7f8fb8164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56314/0 (socket says 172.21.15.29:56314) 2023-10-14T16:57:06.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbe2fe640 1 -- 172.21.15.29:0/854920855 learned_addr learned my addr 172.21.15.29:0/854920855 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/854920855 >> v2:172.21.15.186:3300/0 conn(0x7f8fb806c9e0 0x7f8fb806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.383+0000 7f8fbdafd640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fb806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935979363 0 0) 0x7f8fb80ca560 con 0x7f8fb80c9440 2023-10-14T16:57:06.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa4003520 con 0x7f8fb80c9440 2023-10-14T16:57:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2652764699 0 0) 0x7f8fa4003520 con 0x7f8fb80c9440 2023-10-14T16:57:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 msgr2=0x7f8fb806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fb806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 >> v2:172.21.15.186:3300/0 conn(0x7f8fb806c9e0 msgr2=0x7f8fb806f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 --2- 172.21.15.29:0/854920855 >> v2:172.21.15.186:3300/0 conn(0x7f8fb806c9e0 0x7f8fb806f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80c9f10 con 0x7f8fb80c9440 2023-10-14T16:57:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8fb00019f0 con 0x7f8fb80c9440 2023-10-14T16:57:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 >> v1:172.21.15.29:6789/0 conn(0x7f8fb80c9440 legacy=0x7f8fb8164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa4003970 0x7f8fa4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fa40047c0 0x7f8fa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fa4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c9950 con 0x7f8fa40047c0 2023-10-14T16:57:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c9c30 con 0x7f8fa4003970 2023-10-14T16:57:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.384+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa400b980 con 0x7f8fb80c9060 2023-10-14T16:57:06.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fa40047c0 0x7f8fa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbe2fe640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fa4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 msgr2=0x7f8fa4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fa4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa4003970 msgr2=0x7f8fa4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbdafd640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa4003970 0x7f8fa4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa4003970 0x7f8fa4003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/854920855 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa4003520 con 0x7f8fa40047c0 2023-10-14T16:57:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.385+0000 7f8fbe2fe640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80c9060 0x7f8fa4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:06.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.386+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fa40047c0 0x7f8fa4004c40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009a90 tx=0x7f8fa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c8cee485eac5965 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.386+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa802c6b0 con 0x7f8fa40047c0 2023-10-14T16:57:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa802c810 con 0x7f8fa40047c0 2023-10-14T16:57:06.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbcafb640 1 -- 172.21.15.29:0/854920855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa802cba0 con 0x7f8fa40047c0 2023-10-14T16:57:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 -- 172.21.15.29:0/854920855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fa40047c0 msgr2=0x7f8fa4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/854920855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fa40047c0 0x7f8fa4004c40 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8009a90 tx=0x7f8fa40057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbe2fe640 1 -- 172.21.15.29:0/854920855 reap_dead start 2023-10-14T16:57:06.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 -- 172.21.15.29:0/854920855 shutdown_connections 2023-10-14T16:57:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 -- 172.21.15.29:0/854920855 >> 172.21.15.29:0/854920855 conn(0x7f8fb80ce6a0 msgr2=0x7f8fb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 -- 172.21.15.29:0/854920855 shutdown_connections 2023-10-14T16:57:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.387+0000 7f8fbeaff640 1 -- 172.21.15.29:0/854920855 wait complete. 2023-10-14T16:57:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.389+0000 7f8fbeaff640 1 Processor -- start 2023-10-14T16:57:06.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.389+0000 7f8fbeaff640 1 -- start start 2023-10-14T16:57:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa40047c0 0x7f8fb81f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 0x7f8fb8155bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c80a0 con 0x7f8fb80bd260 2023-10-14T16:57:06.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb80c71d0 con 0x7f8fa40047c0 2023-10-14T16:57:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.390+0000 7f8fbeaff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fb81f7450 con 0x7f8fb80bd6e0 2023-10-14T16:57:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33118/0 (socket says 172.21.15.29:33118) 2023-10-14T16:57:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/3398001257 learned_addr learned my addr 172.21.15.29:0/3398001257 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:06.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbe2fe640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 0x7f8fb8155bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 msgr2=0x7f8fb8155bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 0x7f8fb8155bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa40047c0 msgr2=0x7f8fb81f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa40047c0 0x7f8fb81f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 -- 172.21.15.29:0/3398001257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb80c7420 con 0x7f8fb80bd260 2023-10-14T16:57:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbe2fe640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 0x7f8fb8155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.391+0000 7f8fbd2fc640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8037040 tx=0x7f8fa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.392+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8003810 con 0x7f8fb80bd260 2023-10-14T16:57:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.392+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fa8003970 con 0x7f8fb80bd260 2023-10-14T16:57:06.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.392+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fa8003d20 con 0x7f8fb80bd260 2023-10-14T16:57:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.392+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fb80c7700 con 0x7f8fb80bd260 2023-10-14T16:57:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.392+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fb80c7cc0 con 0x7f8fb80bd260 2023-10-14T16:57:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.393+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fb804e9b0 con 0x7f8fb80bd260 2023-10-14T16:57:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.394+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fa8040450 con 0x7f8fb80bd260 2023-10-14T16:57:06.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.395+0000 7f8fae7fc640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 0x7f8f9c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.395+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(95..95 src has 1..95) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f8fa80abdd0 con 0x7f8fb80bd260 2023-10-14T16:57:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.395+0000 7f8fbdafd640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 0x7f8f9c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.396+0000 7f8fbdafd640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 0x7f8f9c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009870 tx=0x7f8fb4008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.400+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fa8076dc0 con 0x7f8fb80bd260 2023-10-14T16:57:06.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.600+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fb8164840 con 0x7f8fb80bd260 2023-10-14T16:57:06.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.603+0000 7f8fae7fc640 1 -- 172.21.15.29:0/3398001257 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v95) v1 ==== 74+0+16639 (secure 0 0 0) 0x7f8fa80044e0 con 0x7f8fb80bd260 2023-10-14T16:57:06.604 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:06.605 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":95,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:05.213011+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 msgr2=0x7f8f9c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 0x7f8f9c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8fb4009870 tx=0x7f8fb4008040 comp rx=0 tx=0).stop 2023-10-14T16:57:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 msgr2=0x7f8fb8155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f8fa8037040 tx=0x7f8fa8004730 comp rx=0 tx=0).stop 2023-10-14T16:57:06.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 shutdown_connections 2023-10-14T16:57:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f8f9c06af90 0x7f8f9c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fb80bd6e0 0x7f8fb8155bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fb80bd260 0x7f8fb8155680 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 --2- 172.21.15.29:0/3398001257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fa40047c0 0x7f8fb81f68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 >> 172.21.15.29:0/3398001257 conn(0x7f8fb80ce6a0 msgr2=0x7f8fb80cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 shutdown_connections 2023-10-14T16:57:06.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.606+0000 7f8fbeaff640 1 -- 172.21.15.29:0/3398001257 wait complete. 2023-10-14T16:57:06.625 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:06.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.850+0000 7f42f411d640 1 Processor -- start 2023-10-14T16:57:06.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.850+0000 7f42f411d640 1 -- start start 2023-10-14T16:57:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.851+0000 7f42f411d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42ec1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f411d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec15e3e0 con 0x7f42ec15d3f0 2023-10-14T16:57:06.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f411d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42ec0cd4a0 con 0x7f42ec0cd0c0 2023-10-14T16:57:06.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f411d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec15e100 con 0x7f42ec0b0b20 2023-10-14T16:57:06.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f26ba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f42ec15d3f0 0x7f42ec15d860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56330/0 (socket says 172.21.15.29:56330) 2023-10-14T16:57:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f26ba640 1 -- 172.21.15.29:0/2652881652 learned_addr learned my addr 172.21.15.29:0/2652881652 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.852+0000 7f42f16b8640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42ec1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805759411 0 0) 0x7f42ec15e3e0 con 0x7f42ec15d3f0 2023-10-14T16:57:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42d8003520 con 0x7f42ec15d3f0 2023-10-14T16:57:06.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979595402 0 0) 0x7f42d8003520 con 0x7f42ec15d3f0 2023-10-14T16:57:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 msgr2=0x7f42ec1615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42ec1615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 >> v1:172.21.15.186:6789/0 conn(0x7f42ec0cd0c0 legacy=0x7f42ec0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec160b10 con 0x7f42ec15d3f0 2023-10-14T16:57:06.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f42e4001670 con 0x7f42ec15d3f0 2023-10-14T16:57:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 >> v1:172.21.15.29:6789/0 conn(0x7f42ec15d3f0 legacy=0x7f42ec15d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d80039a0 0x7f42d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42d80047c0 0x7f42d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42d80051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec15e100 con 0x7f42d80047c0 2023-10-14T16:57:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800bae0 con 0x7f42d80039a0 2023-10-14T16:57:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42d800bca0 con 0x7f42ec0b0b20 2023-10-14T16:57:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.853+0000 7f42f16b8640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42d80047c0 0x7f42d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f1eb9640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d80039a0 0x7f42d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f16b8640 1 -- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 msgr2=0x7f42d80051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f26ba640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42d80051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f16b8640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42d80051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f16b8640 1 -- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d80039a0 msgr2=0x7f42d8003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:06.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f16b8640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d80039a0 0x7f42d8003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f16b8640 1 -- 172.21.15.29:0/2652881652 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42d8003520 con 0x7f42d80047c0 2023-10-14T16:57:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f26ba640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42ec0b0b20 0x7f42d80051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:06.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.854+0000 7f42f1eb9640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42d80039a0 0x7f42d8003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.855+0000 7f42f16b8640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42d80047c0 0x7f42d8004c40 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009ac0 tx=0x7f42d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59ebdb852f8427ed server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.855+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc02c6b0 con 0x7f42d80047c0 2023-10-14T16:57:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.855+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42dc02c810 con 0x7f42d80047c0 2023-10-14T16:57:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.855+0000 7f42f0eb7640 1 -- 172.21.15.29:0/2652881652 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42dc02cba0 con 0x7f42d80047c0 2023-10-14T16:57:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 -- 172.21.15.29:0/2652881652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42d80047c0 msgr2=0x7f42d8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 --2- 172.21.15.29:0/2652881652 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42d80047c0 0x7f42d8004c40 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f42dc009ac0 tx=0x7f42d80057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f26ba640 1 -- 172.21.15.29:0/2652881652 reap_dead start 2023-10-14T16:57:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 -- 172.21.15.29:0/2652881652 shutdown_connections 2023-10-14T16:57:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 -- 172.21.15.29:0/2652881652 >> 172.21.15.29:0/2652881652 conn(0x7f42ec0cb410 msgr2=0x7f42ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 -- 172.21.15.29:0/2652881652 shutdown_connections 2023-10-14T16:57:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.856+0000 7f42f411d640 1 -- 172.21.15.29:0/2652881652 wait complete. 2023-10-14T16:57:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.858+0000 7f42f411d640 1 Processor -- start 2023-10-14T16:57:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.858+0000 7f42f411d640 1 -- start start 2023-10-14T16:57:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d80047c0 0x7f42ec158700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42ec1f22a0 0x7f42ec0ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec160390 con 0x7f42ec15f9d0 2023-10-14T16:57:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec1567e0 con 0x7f42ec1f22a0 2023-10-14T16:57:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f411d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42ec159910 con 0x7f42d80047c0 2023-10-14T16:57:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.859+0000 7f42f26ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33134/0 (socket says 172.21.15.29:33134) 2023-10-14T16:57:06.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 -- 172.21.15.29:0/151767366 learned_addr learned my addr 172.21.15.29:0/151767366 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f16b8640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42ec1f22a0 0x7f42ec0ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f1eb9640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d80047c0 0x7f42ec158700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 -- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d80047c0 msgr2=0x7f42ec158700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d80047c0 0x7f42ec158700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 -- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42ec1f22a0 msgr2=0x7f42ec0ae470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:06.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42ec1f22a0 0x7f42ec0ae470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 -- 172.21.15.29:0/151767366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42ec159da0 con 0x7f42ec15f9d0 2023-10-14T16:57:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42f26ba640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0cc190 tx=0x7f42e400d9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.860+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4007440 con 0x7f42ec15f9d0 2023-10-14T16:57:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.861+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42e40075c0 con 0x7f42ec15f9d0 2023-10-14T16:57:06.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.861+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f42e4004a20 con 0x7f42ec15f9d0 2023-10-14T16:57:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.861+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42ec158d00 con 0x7f42ec15f9d0 2023-10-14T16:57:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.861+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42ec1592c0 con 0x7f42ec15f9d0 2023-10-14T16:57:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.862+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42ec1f2680 con 0x7f42ec15f9d0 2023-10-14T16:57:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.863+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f42e4004b80 con 0x7f42ec15f9d0 2023-10-14T16:57:06.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.864+0000 7f42e27fc640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 0x7f42d006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.864+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f42e4089400 con 0x7f42ec15f9d0 2023-10-14T16:57:06.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.865+0000 7f42f1eb9640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 0x7f42d006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:06.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.865+0000 7f42f1eb9640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 0x7f42d006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f42e8001d20 tx=0x7f42e8009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:06.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:06.869+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42ec1f2680 con 0x7f42ec15f9d0 2023-10-14T16:57:07.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.068+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f42ec1f2680 con 0x7f42ec15f9d0 2023-10-14T16:57:07.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.071+0000 7f42e27fc640 1 -- 172.21.15.29:0/151767366 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f42ec1f2680 con 0x7f42ec15f9d0 2023-10-14T16:57:07.072 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:07.072 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":96,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:06.745590+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:07.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.073+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 msgr2=0x7f42d006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.073+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 0x7f42d006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f42e8001d20 tx=0x7f42e8009040 comp rx=0 tx=0).stop 2023-10-14T16:57:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 msgr2=0x7f42ec15fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f42ec0cc190 tx=0x7f42e400d9b0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 shutdown_connections 2023-10-14T16:57:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f42d006afc0 0x7f42d006d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42ec15f9d0 0x7f42ec15fe50 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42ec1f22a0 0x7f42ec0ae470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 --2- 172.21.15.29:0/151767366 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f42d80047c0 0x7f42ec158700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 >> 172.21.15.29:0/151767366 conn(0x7f42ec0cb410 msgr2=0x7f42ec0c92d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 shutdown_connections 2023-10-14T16:57:07.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.074+0000 7f42f411d640 1 -- 172.21.15.29:0/151767366 wait complete. 2023-10-14T16:57:07.095 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:57:07.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.322+0000 7f6ed9c51640 1 Processor -- start 2023-10-14T16:57:07.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.323+0000 7f6ed9c51640 1 -- start start 2023-10-14T16:57:07.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.324+0000 7f6ed9c51640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 0x7f6ed4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.324+0000 7f6ed9c51640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.324+0000 7f6ed9c51640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed406a730 con 0x7f6ed415d150 2023-10-14T16:57:07.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.324+0000 7f6ed9c51640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed415d530 con 0x7f6ed40ad350 2023-10-14T16:57:07.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.324+0000 7f6ed9c51640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed40ac890 con 0x7f6ed406a2e0 2023-10-14T16:57:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.325+0000 7f6ed3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.325+0000 7f6ecbfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 0x7f6ed4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.325+0000 7f6ed37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6ed415d150 0x7f6ed4160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:47792/0 (socket says 172.21.15.29:47792) 2023-10-14T16:57:07.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.325+0000 7f6ed37fe640 1 -- 172.21.15.29:0/1822431675 learned_addr learned my addr 172.21.15.29:0/1822431675 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:07.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ed3fff640 1 -- 172.21.15.29:0/1822431675 >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 msgr2=0x7f6ed4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:07.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ed3fff640 1 --2- 172.21.15.29:0/1822431675 >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 0x7f6ed4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ed3fff640 1 -- 172.21.15.29:0/1822431675 >> v1:172.21.15.186:6789/0 conn(0x7f6ed415d150 legacy=0x7f6ed4160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ed2ffd640 1 -- 172.21.15.29:0/1822431675 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119651642 0 0) 0x7f6ec4001110 con 0x7f6ed415d150 2023-10-14T16:57:07.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ed3fff640 1 -- 172.21.15.29:0/1822431675 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed40ad7a0 con 0x7f6ed406a2e0 2023-10-14T16:57:07.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.326+0000 7f6ecbfff640 1 --2- 172.21.15.29:0/1822431675 >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 0x7f6ed4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:57:07.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.328+0000 7f6ed3fff640 1 --2- 172.21.15.29:0/1822431675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40c9000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009d90 tx=0x7f6ecc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7ae4b7a790199338 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:07.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.328+0000 7f6ed2ffd640 1 -- 172.21.15.29:0/1822431675 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc02cc50 con 0x7f6ed406a2e0 2023-10-14T16:57:07.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.328+0000 7f6ed2ffd640 1 -- 172.21.15.29:0/1822431675 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc02cdb0 con 0x7f6ed406a2e0 2023-10-14T16:57:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.328+0000 7f6ed2ffd640 1 -- 172.21.15.29:0/1822431675 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ecc034660 con 0x7f6ed406a2e0 2023-10-14T16:57:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 -- 172.21.15.29:0/1822431675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 msgr2=0x7f6ed40c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/1822431675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40c9000 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f6ecc009d90 tx=0x7f6ecc02bea0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 -- 172.21.15.29:0/1822431675 shutdown_connections 2023-10-14T16:57:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/1822431675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40c9000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/1822431675 >> v2:172.21.15.186:3300/0 conn(0x7f6ed40ad350 0x7f6ed4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 -- 172.21.15.29:0/1822431675 >> 172.21.15.29:0/1822431675 conn(0x7f6ed40ce9b0 msgr2=0x7f6ed40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 -- 172.21.15.29:0/1822431675 shutdown_connections 2023-10-14T16:57:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.329+0000 7f6ed9c51640 1 -- 172.21.15.29:0/1822431675 wait complete. 2023-10-14T16:57:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.331+0000 7f6ed9c51640 1 Processor -- start 2023-10-14T16:57:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.331+0000 7f6ed9c51640 1 -- start start 2023-10-14T16:57:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 0x7f6ed415b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed40b3500 con 0x7f6ed40ad350 2023-10-14T16:57:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed414f1a0 con 0x7f6ed415d150 2023-10-14T16:57:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.332+0000 7f6ed9c51640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed40c7280 con 0x7f6ed406a2e0 2023-10-14T16:57:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33148/0 (socket says 172.21.15.29:33148) 2023-10-14T16:57:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 -- 172.21.15.29:0/833942140 learned_addr learned my addr 172.21.15.29:0/833942140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ed37fe640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ed3fff640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 0x7f6ed415b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 -- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 msgr2=0x7f6ed40ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 -- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 msgr2=0x7f6ed415b8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 0x7f6ed415b8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ecbfff640 1 -- 172.21.15.29:0/833942140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc009400 con 0x7f6ed40ad350 2023-10-14T16:57:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ed37fe640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.333+0000 7f6ed3fff640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 0x7f6ed415b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ecbfff640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00ec10 tx=0x7f6ebc00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc009760 con 0x7f6ed40ad350 2023-10-14T16:57:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc00cc30 con 0x7f6ed40ad350 2023-10-14T16:57:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc009a70 con 0x7f6ed40ad350 2023-10-14T16:57:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc013470 con 0x7f6ed40ad350 2023-10-14T16:57:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.334+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed40b3190 con 0x7f6ed40ad350 2023-10-14T16:57:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.336+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed404e160 con 0x7f6ed40ad350 2023-10-14T16:57:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.337+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ebc0135d0 con 0x7f6ed40ad350 2023-10-14T16:57:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.337+0000 7f6ed0ff9640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 0x7f6eb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.338+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6ebc089a80 con 0x7f6ed40ad350 2023-10-14T16:57:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.338+0000 7f6ed37fe640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 0x7f6eb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.339+0000 7f6ed37fe640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 0x7f6eb406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4000f60 tx=0x7f6ec4009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.342+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ebc0549f0 con 0x7f6ed40ad350 2023-10-14T16:57:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:07.578+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f6ed40b1300 con 0x7f6ed40ad350 2023-10-14T16:57:08.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.158+0000 7f6ed0ff9640 1 -- 172.21.15.29:0/833942140 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v39) v1 ==== 118+0+0 (secure 0 0 0) 0x7f6ebc058010 con 0x7f6ed40ad350 2023-10-14T16:57:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 msgr2=0x7f6eb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 0x7f6eb406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6ec4000f60 tx=0x7f6ec4009040 comp rx=0 tx=0).stop 2023-10-14T16:57:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 msgr2=0x7f6ed414e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00ec10 tx=0x7f6ebc00c520 comp rx=0 tx=0).stop 2023-10-14T16:57:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 shutdown_connections 2023-10-14T16:57:08.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f6eb406af90 0x7f6eb406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ed415d150 0x7f6ed415b8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ed40ad350 0x7f6ed414e560 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 --2- 172.21.15.29:0/833942140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ed406a2e0 0x7f6ed40ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.161+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 >> 172.21.15.29:0/833942140 conn(0x7f6ed40ce9b0 msgr2=0x7f6ed406f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:08.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.162+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 shutdown_connections 2023-10-14T16:57:08.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.162+0000 7f6ed9c51640 1 -- 172.21.15.29:0/833942140 wait complete. 2023-10-14T16:57:08.183 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:57:08.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:57:08.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.416+0000 7f2eb4b6c640 1 Processor -- start 2023-10-14T16:57:08.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.416+0000 7f2eb4b6c640 1 -- start start 2023-10-14T16:57:08.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eb4b6c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2eb006c040 0x7f2eb006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eb4b6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2eb00cda80 0x7f2eb015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eb4b6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2eb006c960 con 0x7f2eb00ad970 2023-10-14T16:57:08.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eb4b6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb015b9d0 con 0x7f2eb00cda80 2023-10-14T16:57:08.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eb4b6c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2eb015bcb0 con 0x7f2eb006c040 2023-10-14T16:57:08.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eaffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2eb00cda80 0x7f2eb015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eaeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2eb00ad970 0x7f2eb015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56346/0 (socket says 172.21.15.29:56346) 2023-10-14T16:57:08.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.418+0000 7f2eaeffd640 1 -- 172.21.15.29:0/1690249232 learned_addr learned my addr 172.21.15.29:0/1690249232 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:08.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eaf7fe640 1 --2- 172.21.15.29:0/1690249232 >> v2:172.21.15.186:3300/0 conn(0x7f2eb006c040 0x7f2eb006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eaffff640 1 -- 172.21.15.29:0/1690249232 >> v2:172.21.15.186:3300/0 conn(0x7f2eb006c040 msgr2=0x7f2eb006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:08.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eaffff640 1 --2- 172.21.15.29:0/1690249232 >> v2:172.21.15.186:3300/0 conn(0x7f2eb006c040 0x7f2eb006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eaffff640 1 -- 172.21.15.29:0/1690249232 >> v1:172.21.15.29:6789/0 conn(0x7f2eb00ad970 legacy=0x7f2eb015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eaffff640 1 -- 172.21.15.29:0/1690249232 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2eb015bf90 con 0x7f2eb00cda80 2023-10-14T16:57:08.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.419+0000 7f2eae7fc640 1 -- 172.21.15.29:0/1690249232 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225942541 0 0) 0x7f2eb006c960 con 0x7f2eb00ad970 2023-10-14T16:57:08.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.420+0000 7f2eaffff640 1 --2- 172.21.15.29:0/1690249232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb015ed90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00cf350 tx=0x7f2ea802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4c4e85a63d3ee37 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.420+0000 7f2eae7fc640 1 -- 172.21.15.29:0/1690249232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea802cc70 con 0x7f2eb00cda80 2023-10-14T16:57:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.420+0000 7f2eae7fc640 1 -- 172.21.15.29:0/1690249232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea802cdd0 con 0x7f2eb00cda80 2023-10-14T16:57:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eae7fc640 1 -- 172.21.15.29:0/1690249232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8034660 con 0x7f2eb00cda80 2023-10-14T16:57:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/1690249232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 msgr2=0x7f2eb015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/1690249232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb015ed90 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f2eb00cf350 tx=0x7f2ea802bec0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/1690249232 shutdown_connections 2023-10-14T16:57:08.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/1690249232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb015ed90 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/1690249232 >> v2:172.21.15.186:3300/0 conn(0x7f2eb006c040 0x7f2eb006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/1690249232 >> 172.21.15.29:0/1690249232 conn(0x7f2eb00ce6a0 msgr2=0x7f2eb00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:08.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/1690249232 shutdown_connections 2023-10-14T16:57:08.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.421+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/1690249232 wait complete. 2023-10-14T16:57:08.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.423+0000 7f2eb4b6c640 1 Processor -- start 2023-10-14T16:57:08.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.423+0000 7f2eb4b6c640 1 -- start start 2023-10-14T16:57:08.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2eb006c040 0x7f2eb014ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eb00ad970 0x7f2eb01f0540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00c6e00 con 0x7f2eb00cda80 2023-10-14T16:57:08.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb01504b0 con 0x7f2eb00ad970 2023-10-14T16:57:08.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.424+0000 7f2eb4b6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2eb00c7070 con 0x7f2eb006c040 2023-10-14T16:57:08.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33170/0 (socket says 172.21.15.29:33170) 2023-10-14T16:57:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 -- 172.21.15.29:0/2054697232 learned_addr learned my addr 172.21.15.29:0/2054697232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaf7fe640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2eb006c040 0x7f2eb014ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaeffd640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eb00ad970 0x7f2eb01f0540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 -- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2eb006c040 msgr2=0x7f2eb014ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2eb006c040 0x7f2eb014ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 -- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eb00ad970 msgr2=0x7f2eb01f0540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eb00ad970 0x7f2eb01f0540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 -- 172.21.15.29:0/2054697232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8009a70 con 0x7f2eb00cda80 2023-10-14T16:57:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.425+0000 7f2eaffff640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2ea802c4d0 tx=0x7f2ea8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.426+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003990 con 0x7f2eb00cda80 2023-10-14T16:57:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.426+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea8003af0 con 0x7f2eb00cda80 2023-10-14T16:57:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.426+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea8003e60 con 0x7f2eb00cda80 2023-10-14T16:57:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.426+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2eb00c7350 con 0x7f2eb00cda80 2023-10-14T16:57:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.426+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2eb00c65f0 con 0x7f2eb00cda80 2023-10-14T16:57:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.427+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2eb006b330 con 0x7f2eb00cda80 2023-10-14T16:57:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.428+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ea8003fe0 con 0x7f2eb00cda80 2023-10-14T16:57:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.429+0000 7f2e97fff640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 0x7f2e9c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:08.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.429+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f2ea80ac760 con 0x7f2eb00cda80 2023-10-14T16:57:08.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.429+0000 7f2eaf7fe640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 0x7f2e9c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:08.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.430+0000 7f2eaf7fe640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 0x7f2e9c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006fd0 tx=0x7f2ea0008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:08.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.433+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ea8077750 con 0x7f2eb00cda80 2023-10-14T16:57:08.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:08.667+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f2eb00c7870 con 0x7f2eb00cda80 2023-10-14T16:57:09.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.161+0000 7f2e97fff640 1 -- 172.21.15.29:0/2054697232 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2ea807ad70 con 0x7f2eb00cda80 2023-10-14T16:57:09.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 msgr2=0x7f2e9c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 0x7f2e9c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0006fd0 tx=0x7f2ea0008040 comp rx=0 tx=0).stop 2023-10-14T16:57:09.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 msgr2=0x7f2eb00aebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f2ea802c4d0 tx=0x7f2ea8004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:09.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 shutdown_connections 2023-10-14T16:57:09.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f2e9c06af90 0x7f2e9c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2eb00cda80 0x7f2eb00aebb0 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2eb00ad970 0x7f2eb01f0540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 --2- 172.21.15.29:0/2054697232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2eb006c040 0x7f2eb014ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.165+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 >> 172.21.15.29:0/2054697232 conn(0x7f2eb00ce6a0 msgr2=0x7f2eb015cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:09.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.166+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 shutdown_connections 2023-10-14T16:57:09.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.166+0000 7f2eb4b6c640 1 -- 172.21.15.29:0/2054697232 wait complete. 2023-10-14T16:57:09.188 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:57:09.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:09.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.416+0000 7ff6110d4640 1 Processor -- start 2023-10-14T16:57:09.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.416+0000 7ff6110d4640 1 -- start start 2023-10-14T16:57:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 0x7ff60c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff60c15ae80 0x7ff60c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c06b810 con 0x7ff60c15ae80 2023-10-14T16:57:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff60c15ed60 con 0x7ff60c15aaa0 2023-10-14T16:57:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff6110d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0707e0 con 0x7ff60c0ccaa0 2023-10-14T16:57:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff60c15ae80 0x7ff60c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff60c15ae80 0x7ff60c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33180/0 (socket says 172.21.15.29:33180) 2023-10-14T16:57:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60a59c640 1 -- 172.21.15.29:0/1824036049 learned_addr learned my addr 172.21.15.29:0/1824036049 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60a59c640 1 -- 172.21.15.29:0/1824036049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 msgr2=0x7ff60c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60a59c640 1 --2- 172.21.15.29:0/1824036049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.418+0000 7ff60ad9d640 1 --2- 172.21.15.29:0/1824036049 >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 0x7ff60c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.419+0000 7ff60a59c640 1 -- 172.21.15.29:0/1824036049 >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 msgr2=0x7ff60c15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.419+0000 7ff60a59c640 1 --2- 172.21.15.29:0/1824036049 >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 0x7ff60c15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.419+0000 7ff60a59c640 1 -- 172.21.15.29:0/1824036049 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c15ef90 con 0x7ff60c15ae80 2023-10-14T16:57:09.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.419+0000 7ff60ad9d640 1 --2- 172.21.15.29:0/1824036049 >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 0x7ff60c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:09.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.420+0000 7ff60a59c640 1 --2- 172.21.15.29:0/1824036049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c0cc560 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0cd990 tx=0x7ff60002b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b1fb1d18f534f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.420+0000 7ff609d9b640 1 -- 172.21.15.29:0/1824036049 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600031880 con 0x7ff60c15ae80 2023-10-14T16:57:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.420+0000 7ff609d9b640 1 -- 172.21.15.29:0/1824036049 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6000319e0 con 0x7ff60c15ae80 2023-10-14T16:57:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.420+0000 7ff609d9b640 1 -- 172.21.15.29:0/1824036049 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff600031d70 con 0x7ff60c15ae80 2023-10-14T16:57:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 -- 172.21.15.29:0/1824036049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 msgr2=0x7ff60c0cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 --2- 172.21.15.29:0/1824036049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c0cc560 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7ff60c0cd990 tx=0x7ff60002b920 comp rx=0 tx=0).stop 2023-10-14T16:57:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 -- 172.21.15.29:0/1824036049 shutdown_connections 2023-10-14T16:57:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 --2- 172.21.15.29:0/1824036049 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 --2- 172.21.15.29:0/1824036049 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c0cc560 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 --2- 172.21.15.29:0/1824036049 >> v2:172.21.15.186:3300/0 conn(0x7ff60c15aaa0 0x7ff60c15e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 -- 172.21.15.29:0/1824036049 >> 172.21.15.29:0/1824036049 conn(0x7ff60c0cb410 msgr2=0x7ff60c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 -- 172.21.15.29:0/1824036049 shutdown_connections 2023-10-14T16:57:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.421+0000 7ff6110d4640 1 -- 172.21.15.29:0/1824036049 wait complete. 2023-10-14T16:57:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.423+0000 7ff6110d4640 1 Processor -- start 2023-10-14T16:57:09.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.423+0000 7ff6110d4640 1 -- start start 2023-10-14T16:57:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c14c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 0x7ff60c14fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c15c560 con 0x7ff60c15ae80 2023-10-14T16:57:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0aa8c0 con 0x7ff60c15aaa0 2023-10-14T16:57:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff6110d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c0aaba0 con 0x7ff60c0ccaa0 2023-10-14T16:57:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33188/0 (socket says 172.21.15.29:33188) 2023-10-14T16:57:09.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 -- 172.21.15.29:0/356311751 learned_addr learned my addr 172.21.15.29:0/356311751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60ad9d640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c14c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60a59c640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 0x7ff60c14fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 -- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 msgr2=0x7ff60c14c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c14c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 -- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 msgr2=0x7ff60c14fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 0x7ff60c14fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.424+0000 7ff60b59e640 1 -- 172.21.15.29:0/356311751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6000094d0 con 0x7ff60c15ae80 2023-10-14T16:57:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff60a59c640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 0x7ff60c14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:09.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff60b59e640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff60400b970 tx=0x7ff60400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff60400cb40 con 0x7ff60c15ae80 2023-10-14T16:57:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff60400ccc0 con 0x7ff60c15ae80 2023-10-14T16:57:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604014680 con 0x7ff60c15ae80 2023-10-14T16:57:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c0aae80 con 0x7ff60c15ae80 2023-10-14T16:57:09.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.425+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c152480 con 0x7ff60c15ae80 2023-10-14T16:57:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.426+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff60c066de0 con 0x7ff60c15ae80 2023-10-14T16:57:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.428+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff60401d3b0 con 0x7ff60c15ae80 2023-10-14T16:57:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.428+0000 7ff5f37fe640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 0x7ff5f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.429+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff6040896e0 con 0x7ff60c15ae80 2023-10-14T16:57:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.429+0000 7ff60ad9d640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 0x7ff5f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.429+0000 7ff60ad9d640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 0x7ff5f806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00a990 tx=0x7ff5fc008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.433+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6040546e0 con 0x7ff60c15ae80 2023-10-14T16:57:09.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.670+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff60c04e160 con 0x7ff60c15ae80 2023-10-14T16:57:09.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.671+0000 7ff5f37fe640 1 -- 172.21.15.29:0/356311751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7ff604057d00 con 0x7ff60c15ae80 2023-10-14T16:57:09.672 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:09.672 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:09.672 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:09.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 msgr2=0x7ff5f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 0x7ff5f806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc00a990 tx=0x7ff5fc008040 comp rx=0 tx=0).stop 2023-10-14T16:57:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 msgr2=0x7ff60c15ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7ff60400b970 tx=0x7ff60400be30 comp rx=0 tx=0).stop 2023-10-14T16:57:09.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 shutdown_connections 2023-10-14T16:57:09.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff5f806af90 0x7ff5f806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.673+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff60c15ae80 0x7ff60c15ca40 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.674+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff60c15aaa0 0x7ff60c14fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.674+0000 7ff6110d4640 1 --2- 172.21.15.29:0/356311751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff60c0ccaa0 0x7ff60c14c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.674+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 >> 172.21.15.29:0/356311751 conn(0x7ff60c0cb410 msgr2=0x7ff60c0b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.674+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 shutdown_connections 2023-10-14T16:57:09.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.674+0000 7ff6110d4640 1 -- 172.21.15.29:0/356311751 wait complete. 2023-10-14T16:57:09.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.918+0000 7f575bfff640 1 Processor -- start 2023-10-14T16:57:09.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.918+0000 7f575bfff640 1 -- start start 2023-10-14T16:57:09.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f575c06c040 0x7f575c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c0cda80 0x7f575c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f575c06c960 con 0x7f575c0ad970 2023-10-14T16:57:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c15b9d0 con 0x7f575c0cda80 2023-10-14T16:57:09.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575bfff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f575c15bcb0 con 0x7f575c06c040 2023-10-14T16:57:09.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c0cda80 0x7f575c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f575c0cda80 0x7f575c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33204/0 (socket says 172.21.15.29:33204) 2023-10-14T16:57:09.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.920+0000 7f575b7fe640 1 -- 172.21.15.29:0/455394851 learned_addr learned my addr 172.21.15.29:0/455394851 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:09.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.921+0000 7f575affd640 1 --2- 172.21.15.29:0/455394851 >> v2:172.21.15.186:3300/0 conn(0x7f575c06c040 0x7f575c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.921+0000 7f575b7fe640 1 -- 172.21.15.29:0/455394851 >> v2:172.21.15.186:3300/0 conn(0x7f575c06c040 msgr2=0x7f575c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:09.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.921+0000 7f575b7fe640 1 --2- 172.21.15.29:0/455394851 >> v2:172.21.15.186:3300/0 conn(0x7f575c06c040 0x7f575c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.921+0000 7f575b7fe640 1 -- 172.21.15.29:0/455394851 >> v1:172.21.15.186:6789/0 conn(0x7f575c0ad970 legacy=0x7f575c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.921+0000 7f575b7fe640 1 -- 172.21.15.29:0/455394851 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c15bf90 con 0x7f575c0cda80 2023-10-14T16:57:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.922+0000 7f575b7fe640 1 --2- 172.21.15.29:0/455394851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c15ed90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cf350 tx=0x7f575402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf22d6f61417ea3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.922+0000 7f5759ffb640 1 -- 172.21.15.29:0/455394851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575402cb20 con 0x7f575c0cda80 2023-10-14T16:57:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f5759ffb640 1 -- 172.21.15.29:0/455394851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f575402cc80 con 0x7f575c0cda80 2023-10-14T16:57:09.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f5759ffb640 1 -- 172.21.15.29:0/455394851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57540343f0 con 0x7f575c0cda80 2023-10-14T16:57:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 -- 172.21.15.29:0/455394851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 msgr2=0x7f575c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 --2- 172.21.15.29:0/455394851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c15ed90 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f575c0cf350 tx=0x7f575402bf80 comp rx=0 tx=0).stop 2023-10-14T16:57:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 -- 172.21.15.29:0/455394851 shutdown_connections 2023-10-14T16:57:09.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 --2- 172.21.15.29:0/455394851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c15ed90 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 --2- 172.21.15.29:0/455394851 >> v2:172.21.15.186:3300/0 conn(0x7f575c06c040 0x7f575c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 -- 172.21.15.29:0/455394851 >> 172.21.15.29:0/455394851 conn(0x7f575c0ce6a0 msgr2=0x7f575c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 -- 172.21.15.29:0/455394851 shutdown_connections 2023-10-14T16:57:09.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.923+0000 7f575bfff640 1 -- 172.21.15.29:0/455394851 wait complete. 2023-10-14T16:57:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.925+0000 7f575bfff640 1 Processor -- start 2023-10-14T16:57:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.925+0000 7f575bfff640 1 -- start start 2023-10-14T16:57:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f575c06c040 0x7f575c0c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f575c0ad970 0x7f575c0b0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0c8630 con 0x7f575c0cda80 2023-10-14T16:57:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b21d0 con 0x7f575c0ad970 2023-10-14T16:57:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f575c0b15c0 con 0x7f575c06c040 2023-10-14T16:57:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.926+0000 7f575b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33212/0 (socket says 172.21.15.29:33212) 2023-10-14T16:57:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 -- 172.21.15.29:0/1660052848 learned_addr learned my addr 172.21.15.29:0/1660052848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575affd640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f575c06c040 0x7f575c0c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575a7fc640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f575c0ad970 0x7f575c0b0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 -- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f575c06c040 msgr2=0x7f575c0c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f575c06c040 0x7f575c0c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 -- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f575c0ad970 msgr2=0x7f575c0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f575c0ad970 0x7f575c0b0710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 -- 172.21.15.29:0/1660052848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5754009b30 con 0x7f575c0cda80 2023-10-14T16:57:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.927+0000 7f575b7fe640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5754036040 tx=0x7f5754004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.928+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5754009510 con 0x7f575c0cda80 2023-10-14T16:57:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.928+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5754007e50 con 0x7f575c0cda80 2023-10-14T16:57:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.928+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f575403e690 con 0x7f575c0cda80 2023-10-14T16:57:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.928+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f575c0b18a0 con 0x7f575c0cda80 2023-10-14T16:57:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.928+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f575c0b1e60 con 0x7f575c0cda80 2023-10-14T16:57:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.929+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f575c06b330 con 0x7f575c0cda80 2023-10-14T16:57:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.930+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f575403e7f0 con 0x7f575c0cda80 2023-10-14T16:57:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.931+0000 7f57437fe640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 0x7f574806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.931+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f57540ac030 con 0x7f575c0cda80 2023-10-14T16:57:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.931+0000 7f575affd640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 0x7f574806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.932+0000 7f575affd640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 0x7f574806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f574c006fd0 tx=0x7f574c008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:09.935+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5754077030 con 0x7f575c0cda80 2023-10-14T16:57:10.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.124+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f575c04e9b0 con 0x7f575c0cda80 2023-10-14T16:57:10.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.127+0000 7f57437fe640 1 -- 172.21.15.29:0/1660052848 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v96) v1 ==== 74+0+16626 (secure 0 0 0) 0x7f575407a650 con 0x7f575c0cda80 2023-10-14T16:57:10.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:10.128 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":96,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:06.745590+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:10.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.129+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 msgr2=0x7f574806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.129+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 0x7f574806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f574c006fd0 tx=0x7f574c008040 comp rx=0 tx=0).stop 2023-10-14T16:57:10.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.129+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 msgr2=0x7f575c0c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.129+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f5754036040 tx=0x7f5754004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:10.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 shutdown_connections 2023-10-14T16:57:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f574806afc0 0x7f574806d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f575c0cda80 0x7f575c0c5770 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f575c0ad970 0x7f575c0b0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 --2- 172.21.15.29:0/1660052848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f575c06c040 0x7f575c0c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 >> 172.21.15.29:0/1660052848 conn(0x7f575c0ce6a0 msgr2=0x7f575c0c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 shutdown_connections 2023-10-14T16:57:10.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.130+0000 7f575bfff640 1 -- 172.21.15.29:0/1660052848 wait complete. 2023-10-14T16:57:10.147 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:57:10.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:57:10.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.369+0000 7fb172b1d640 1 Processor -- start 2023-10-14T16:57:10.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.370+0000 7fb172b1d640 1 -- start start 2023-10-14T16:57:10.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.371+0000 7fb172b1d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb16c0ce390 0x7fb16c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.371+0000 7fb172b1d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16c15ed60 con 0x7fb16c15ae80 2023-10-14T16:57:10.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.371+0000 7fb172b1d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb16c0707e0 con 0x7fb16c15aaa0 2023-10-14T16:57:10.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.371+0000 7fb172b1d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb16c06b810 con 0x7fb16c0ce390 2023-10-14T16:57:10.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb16c0ce390 0x7fb16c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb16c0ce390 0x7fb16c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33226/0 (socket says 172.21.15.29:33226) 2023-10-14T16:57:10.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 -- 172.21.15.29:0/274877184 learned_addr learned my addr 172.21.15.29:0/274877184 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb16bfff640 1 --1- 172.21.15.29:0/274877184 >> v1:172.21.15.29:6789/0 conn(0x7fb16c15ae80 0x7fb16c070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56358/0 (socket says 172.21.15.29:56358) 2023-10-14T16:57:10.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 -- 172.21.15.29:0/274877184 >> v1:172.21.15.186:6789/0 conn(0x7fb16c15aaa0 legacy=0x7fb16c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 -- 172.21.15.29:0/274877184 >> v1:172.21.15.29:6789/0 conn(0x7fb16c15ae80 legacy=0x7fb16c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb1710ba640 1 -- 172.21.15.29:0/274877184 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb16c15ef90 con 0x7fb16c0ce390 2023-10-14T16:57:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.372+0000 7fb16b7fe640 1 -- 172.21.15.29:0/274877184 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541069772 0 0) 0x7fb16c15ed60 con 0x7fb16c15ae80 2023-10-14T16:57:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.373+0000 7fb1710ba640 1 --2- 172.21.15.29:0/274877184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c0ce390 0x7fb16c15e820 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb164009e50 tx=0x7fb16402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3eac883a6931aa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.374+0000 7fb16b7fe640 1 -- 172.21.15.29:0/274877184 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16402cb20 con 0x7fb16c0ce390 2023-10-14T16:57:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.374+0000 7fb16b7fe640 1 -- 172.21.15.29:0/274877184 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16402cc80 con 0x7fb16c0ce390 2023-10-14T16:57:10.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.374+0000 7fb16b7fe640 1 -- 172.21.15.29:0/274877184 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1640343f0 con 0x7fb16c0ce390 2023-10-14T16:57:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.374+0000 7fb172b1d640 1 -- 172.21.15.29:0/274877184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c0ce390 msgr2=0x7fb16c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.374+0000 7fb172b1d640 1 --2- 172.21.15.29:0/274877184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c0ce390 0x7fb16c15e820 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb164009e50 tx=0x7fb16402bf80 comp rx=0 tx=0).stop 2023-10-14T16:57:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.375+0000 7fb172b1d640 1 -- 172.21.15.29:0/274877184 shutdown_connections 2023-10-14T16:57:10.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.375+0000 7fb172b1d640 1 --2- 172.21.15.29:0/274877184 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c0ce390 0x7fb16c15e820 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.375+0000 7fb172b1d640 1 -- 172.21.15.29:0/274877184 >> 172.21.15.29:0/274877184 conn(0x7fb16c0cb410 msgr2=0x7fb16c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.375+0000 7fb172b1d640 1 -- 172.21.15.29:0/274877184 shutdown_connections 2023-10-14T16:57:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.375+0000 7fb172b1d640 1 -- 172.21.15.29:0/274877184 wait complete. 2023-10-14T16:57:10.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.376+0000 7fb172b1d640 1 Processor -- start 2023-10-14T16:57:10.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.377+0000 7fb172b1d640 1 -- start start 2023-10-14T16:57:10.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 0x7fb16c141410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 0x7fb16c1f44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c0c8910 con 0x7fb16c15ae80 2023-10-14T16:57:10.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c1f3d00 con 0x7fb16c15aaa0 2023-10-14T16:57:10.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb172b1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb16c141950 con 0x7fb16c0ce390 2023-10-14T16:57:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33234/0 (socket says 172.21.15.29:33234) 2023-10-14T16:57:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 -- 172.21.15.29:0/2915239654 learned_addr learned my addr 172.21.15.29:0/2915239654 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:10.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1708b9640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 0x7fb16c141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 -- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 msgr2=0x7fb16c141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 0x7fb16c141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 -- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 msgr2=0x7fb16c1f44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 0x7fb16c1f44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 0x7fb16c1f44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1710ba640 1 -- 172.21.15.29:0/2915239654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb164009b30 con 0x7fb16c15ae80 2023-10-14T16:57:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb1708b9640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 0x7fb16c141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.378+0000 7fb16bfff640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 0x7fb16c1f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb1710ba640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb164036040 tx=0x7fb164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb164009510 con 0x7fb16c15ae80 2023-10-14T16:57:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb164007e10 con 0x7fb16c15ae80 2023-10-14T16:57:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb16c0c8350 con 0x7fb16c15ae80 2023-10-14T16:57:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb16403e670 con 0x7fb16c15ae80 2023-10-14T16:57:10.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.379+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb16c1f3fa0 con 0x7fb16c15ae80 2023-10-14T16:57:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.380+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb16c1f3b60 con 0x7fb16c15ae80 2023-10-14T16:57:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.381+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb16403e7d0 con 0x7fb16c15ae80 2023-10-14T16:57:10.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.382+0000 7fb1697fa640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 0x7fb15006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.382+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb1640ac550 con 0x7fb16c15ae80 2023-10-14T16:57:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.383+0000 7fb1708b9640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 0x7fb15006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.383+0000 7fb1708b9640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 0x7fb15006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb160003eb0 tx=0x7fb160009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.387+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb16c1f3b60 con 0x7fb16c15ae80 2023-10-14T16:57:10.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.580+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 --> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb16c15f550 con 0x7fb15006af90 2023-10-14T16:57:10.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.596+0000 7fb1697fa640 1 -- 172.21.15.29:0/2915239654 <== mgr.6765 v2:172.21.15.29:6832/3330033844 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286819 (secure 0 0 0) 0x7fb160005030 con 0x7fb15006af90 2023-10-14T16:57:10.599 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:10.602 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:57:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 msgr2=0x7fb15006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 0x7fb15006d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb160003eb0 tx=0x7fb160009040 comp rx=0 tx=0).stop 2023-10-14T16:57:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 msgr2=0x7fb16c0b0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fb164036040 tx=0x7fb164004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 shutdown_connections 2023-10-14T16:57:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.603+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb15006af90 0x7fb15006d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb16c15ae80 0x7fb16c0b0590 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb16c15aaa0 0x7fb16c1f44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 --2- 172.21.15.29:0/2915239654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb16c0ce390 0x7fb16c141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 >> 172.21.15.29:0/2915239654 conn(0x7fb16c0cb410 msgr2=0x7fb16c0b3330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 shutdown_connections 2023-10-14T16:57:10.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.604+0000 7fb172b1d640 1 -- 172.21.15.29:0/2915239654 wait complete. 2023-10-14T16:57:10.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T16:57:08.976716+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341704,"kb_used_data":10364,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754633016,"statfs":{"total":773094113280,"available":772744208384,"internally_reserved":0,"allocated":10612736,"data_stored":5672460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.258094"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516735+0000","last_change":"2023-10-14T16:57:05.771528+0000","last_active":"2023-10-14T16:57:07.516735+0000","last_peered":"2023-10-14T16:57:07.516735+0000","last_clean":"2023-10-14T16:57:07.516735+0000","last_became_active":"2023-10-14T16:57:05.771118+0000","last_became_peered":"2023-10-14T16:57:05.771118+0000","last_unstale":"2023-10-14T16:57:07.516735+0000","last_undegraded":"2023-10-14T16:57:07.516735+0000","last_fullsized":"2023-10-14T16:57:07.516735+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:10.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764806+0000","last_change":"2023-10-14T16:57:05.756040+0000","last_active":"2023-10-14T16:57:06.764806+0000","last_peered":"2023-10-14T16:57:06.764806+0000","last_clean":"2023-10-14T16:57:06.764806+0000","last_became_active":"2023-10-14T16:57:05.755719+0000","last_became_peered":"2023-10-14T16:57:05.755719+0000","last_unstale":"2023-10-14T16:57:06.764806+0000","last_undegraded":"2023-10-14T16:57:06.764806+0000","last_fullsized":"2023-10-14T16:57:06.764806+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983603+0000","last_change":"2023-10-14T16:57:05.757206+0000","last_active":"2023-10-14T16:57:06.983603+0000","last_peered":"2023-10-14T16:57:06.983603+0000","last_clean":"2023-10-14T16:57:06.983603+0000","last_became_active":"2023-10-14T16:57:05.756917+0000","last_became_peered":"2023-10-14T16:57:05.756917+0000","last_unstale":"2023-10-14T16:57:06.983603+0000","last_undegraded":"2023-10-14T16:57:06.983603+0000","last_fullsized":"2023-10-14T16:57:06.983603+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:39.839365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092759+0000","last_change":"2023-10-14T16:57:05.770843+0000","last_active":"2023-10-14T16:57:07.092759+0000","last_peered":"2023-10-14T16:57:07.092759+0000","last_clean":"2023-10-14T16:57:07.092759+0000","last_became_active":"2023-10-14T16:57:05.770522+0000","last_became_peered":"2023-10-14T16:57:05.770522+0000","last_unstale":"2023-10-14T16:57:07.092759+0000","last_undegraded":"2023-10-14T16:57:07.092759+0000","last_fullsized":"2023-10-14T16:57:07.092759+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:12:22.036610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761123+0000","last_change":"2023-10-14T16:57:05.764891+0000","last_active":"2023-10-14T16:57:06.761123+0000","last_peered":"2023-10-14T16:57:06.761123+0000","last_clean":"2023-10-14T16:57:06.761123+0000","last_became_active":"2023-10-14T16:57:05.764554+0000","last_became_peered":"2023-10-14T16:57:05.764554+0000","last_unstale":"2023-10-14T16:57:06.761123+0000","last_undegraded":"2023-10-14T16:57:06.761123+0000","last_fullsized":"2023-10-14T16:57:06.761123+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:12.225048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763978+0000","last_change":"2023-10-14T16:57:05.757888+0000","last_active":"2023-10-14T16:57:06.763978+0000","last_peered":"2023-10-14T16:57:06.763978+0000","last_clean":"2023-10-14T16:57:06.763978+0000","last_became_active":"2023-10-14T16:57:05.757428+0000","last_became_peered":"2023-10-14T16:57:05.757428+0000","last_unstale":"2023-10-14T16:57:06.763978+0000","last_undegraded":"2023-10-14T16:57:06.763978+0000","last_fullsized":"2023-10-14T16:57:06.763978+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:59.165568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.045973+0000","last_change":"2023-10-14T16:57:05.757383+0000","last_active":"2023-10-14T16:57:07.045973+0000","last_peered":"2023-10-14T16:57:07.045973+0000","last_clean":"2023-10-14T16:57:07.045973+0000","last_became_active":"2023-10-14T16:57:05.756062+0000","last_became_peered":"2023-10-14T16:57:05.756062+0000","last_unstale":"2023-10-14T16:57:07.045973+0000","last_undegraded":"2023-10-14T16:57:07.045973+0000","last_fullsized":"2023-10-14T16:57:07.045973+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:43.315213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763762+0000","last_change":"2023-10-14T16:57:05.768104+0000","last_active":"2023-10-14T16:57:06.763762+0000","last_peered":"2023-10-14T16:57:06.763762+0000","last_clean":"2023-10-14T16:57:06.763762+0000","last_became_active":"2023-10-14T16:57:05.767713+0000","last_became_peered":"2023-10-14T16:57:05.767713+0000","last_unstale":"2023-10-14T16:57:06.763762+0000","last_undegraded":"2023-10-14T16:57:06.763762+0000","last_fullsized":"2023-10-14T16:57:06.763762+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762375+0000","last_change":"2023-10-14T16:57:05.769922+0000","last_active":"2023-10-14T16:57:06.762375+0000","last_peered":"2023-10-14T16:57:06.762375+0000","last_clean":"2023-10-14T16:57:06.762375+0000","last_became_active":"2023-10-14T16:57:05.769340+0000","last_became_peered":"2023-10-14T16:57:05.769340+0000","last_unstale":"2023-10-14T16:57:06.762375+0000","last_undegraded":"2023-10-14T16:57:06.762375+0000","last_fullsized":"2023-10-14T16:57:06.762375+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765403+0000","last_change":"2023-10-14T16:57:05.760925+0000","last_active":"2023-10-14T16:57:06.765403+0000","last_peered":"2023-10-14T16:57:06.765403+0000","last_clean":"2023-10-14T16:57:06.765403+0000","last_became_active":"2023-10-14T16:57:05.760618+0000","last_became_peered":"2023-10-14T16:57:05.760618+0000","last_unstale":"2023-10-14T16:57:06.765403+0000","last_undegraded":"2023-10-14T16:57:06.765403+0000","last_fullsized":"2023-10-14T16:57:06.765403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:27.819255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516563+0000","last_change":"2023-10-14T16:57:05.766151+0000","last_active":"2023-10-14T16:57:07.516563+0000","last_peered":"2023-10-14T16:57:07.516563+0000","last_clean":"2023-10-14T16:57:07.516563+0000","last_became_active":"2023-10-14T16:57:05.765846+0000","last_became_peered":"2023-10-14T16:57:05.765846+0000","last_unstale":"2023-10-14T16:57:07.516563+0000","last_undegraded":"2023-10-14T16:57:07.516563+0000","last_fullsized":"2023-10-14T16:57:07.516563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:01.120633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092955+0000","last_change":"2023-10-14T16:57:05.758969+0000","last_active":"2023-10-14T16:57:07.092955+0000","last_peered":"2023-10-14T16:57:07.092955+0000","last_clean":"2023-10-14T16:57:07.092955+0000","last_became_active":"2023-10-14T16:57:05.758562+0000","last_became_peered":"2023-10-14T16:57:05.758562+0000","last_unstale":"2023-10-14T16:57:07.092955+0000","last_undegraded":"2023-10-14T16:57:07.092955+0000","last_fullsized":"2023-10-14T16:57:07.092955+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:12.881364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516398+0000","last_change":"2023-10-14T16:57:05.769791+0000","last_active":"2023-10-14T16:57:07.516398+0000","last_peered":"2023-10-14T16:57:07.516398+0000","last_clean":"2023-10-14T16:57:07.516398+0000","last_became_active":"2023-10-14T16:57:05.768452+0000","last_became_peered":"2023-10-14T16:57:05.768452+0000","last_unstale":"2023-10-14T16:57:07.516398+0000","last_undegraded":"2023-10-14T16:57:07.516398+0000","last_fullsized":"2023-10-14T16:57:07.516398+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:07.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761146+0000","last_change":"2023-10-14T16:57:05.760655+0000","last_active":"2023-10-14T16:57:06.761146+0000","last_peered":"2023-10-14T16:57:06.761146+0000","last_clean":"2023-10-14T16:57:06.761146+0000","last_became_active":"2023-10-14T16:57:05.759465+0000","last_became_peered":"2023-10-14T16:57:05.759465+0000","last_unstale":"2023-10-14T16:57:06.761146+0000","last_undegraded":"2023-10-14T16:57:06.761146+0000","last_fullsized":"2023-10-14T16:57:06.761146+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:59.165568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515609+0000","last_change":"2023-10-14T16:57:05.769285+0000","last_active":"2023-10-14T16:57:07.515609+0000","last_peered":"2023-10-14T16:57:07.515609+0000","last_clean":"2023-10-14T16:57:07.515609+0000","last_became_active":"2023-10-14T16:57:05.768628+0000","last_became_peered":"2023-10-14T16:57:05.768628+0000","last_unstale":"2023-10-14T16:57:07.515609+0000","last_undegraded":"2023-10-14T16:57:07.515609+0000","last_fullsized":"2023-10-14T16:57:07.515609+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762473+0000","last_change":"2023-10-14T16:57:05.769097+0000","last_active":"2023-10-14T16:57:06.762473+0000","last_peered":"2023-10-14T16:57:06.762473+0000","last_clean":"2023-10-14T16:57:06.762473+0000","last_became_active":"2023-10-14T16:57:05.768589+0000","last_became_peered":"2023-10-14T16:57:05.768589+0000","last_unstale":"2023-10-14T16:57:06.762473+0000","last_undegraded":"2023-10-14T16:57:06.762473+0000","last_fullsized":"2023-10-14T16:57:06.762473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:07.910530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762519+0000","last_change":"2023-10-14T16:57:05.768904+0000","last_active":"2023-10-14T16:57:06.762519+0000","last_peered":"2023-10-14T16:57:06.762519+0000","last_clean":"2023-10-14T16:57:06.762519+0000","last_became_active":"2023-10-14T16:57:05.766522+0000","last_became_peered":"2023-10-14T16:57:05.766522+0000","last_unstale":"2023-10-14T16:57:06.762519+0000","last_undegraded":"2023-10-14T16:57:06.762519+0000","last_fullsized":"2023-10-14T16:57:06.762519+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:39.174579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092664+0000","last_change":"2023-10-14T16:57:05.772673+0000","last_active":"2023-10-14T16:57:07.092664+0000","last_peered":"2023-10-14T16:57:07.092664+0000","last_clean":"2023-10-14T16:57:07.092664+0000","last_became_active":"2023-10-14T16:57:05.772227+0000","last_became_peered":"2023-10-14T16:57:05.772227+0000","last_unstale":"2023-10-14T16:57:07.092664+0000","last_undegraded":"2023-10-14T16:57:07.092664+0000","last_fullsized":"2023-10-14T16:57:07.092664+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:44:06.517776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765125+0000","last_change":"2023-10-14T16:57:05.761360+0000","last_active":"2023-10-14T16:57:06.765125+0000","last_peered":"2023-10-14T16:57:06.765125+0000","last_clean":"2023-10-14T16:57:06.765125+0000","last_became_active":"2023-10-14T16:57:05.760997+0000","last_became_peered":"2023-10-14T16:57:05.760997+0000","last_unstale":"2023-10-14T16:57:06.765125+0000","last_undegraded":"2023-10-14T16:57:06.765125+0000","last_fullsized":"2023-10-14T16:57:06.765125+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516039+0000","last_change":"2023-10-14T16:57:05.764064+0000","last_active":"2023-10-14T16:57:07.516039+0000","last_peered":"2023-10-14T16:57:07.516039+0000","last_clean":"2023-10-14T16:57:07.516039+0000","last_became_active":"2023-10-14T16:57:05.763749+0000","last_became_peered":"2023-10-14T16:57:05.763749+0000","last_unstale":"2023-10-14T16:57:07.516039+0000","last_undegraded":"2023-10-14T16:57:07.516039+0000","last_fullsized":"2023-10-14T16:57:07.516039+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763846+0000","last_change":"2023-10-14T16:57:05.760146+0000","last_active":"2023-10-14T16:57:06.763846+0000","last_peered":"2023-10-14T16:57:06.763846+0000","last_clean":"2023-10-14T16:57:06.763846+0000","last_became_active":"2023-10-14T16:57:05.759870+0000","last_became_peered":"2023-10-14T16:57:05.759870+0000","last_unstale":"2023-10-14T16:57:06.763846+0000","last_undegraded":"2023-10-14T16:57:06.763846+0000","last_fullsized":"2023-10-14T16:57:06.763846+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:07.099457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762737+0000","last_change":"2023-10-14T16:57:05.754153+0000","last_active":"2023-10-14T16:57:06.762737+0000","last_peered":"2023-10-14T16:57:06.762737+0000","last_clean":"2023-10-14T16:57:06.762737+0000","last_became_active":"2023-10-14T16:57:05.753572+0000","last_became_peered":"2023-10-14T16:57:05.753572+0000","last_unstale":"2023-10-14T16:57:06.762737+0000","last_undegraded":"2023-10-14T16:57:06.762737+0000","last_fullsized":"2023-10-14T16:57:06.762737+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:22.350211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516145+0000","last_change":"2023-10-14T16:57:05.764057+0000","last_active":"2023-10-14T16:57:07.516145+0000","last_peered":"2023-10-14T16:57:07.516145+0000","last_clean":"2023-10-14T16:57:07.516145+0000","last_became_active":"2023-10-14T16:57:05.763739+0000","last_became_peered":"2023-10-14T16:57:05.763739+0000","last_unstale":"2023-10-14T16:57:07.516145+0000","last_undegraded":"2023-10-14T16:57:07.516145+0000","last_fullsized":"2023-10-14T16:57:07.516145+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:51:11.837087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515618+0000","last_change":"2023-10-14T16:57:05.773468+0000","last_active":"2023-10-14T16:57:07.515618+0000","last_peered":"2023-10-14T16:57:07.515618+0000","last_clean":"2023-10-14T16:57:07.515618+0000","last_became_active":"2023-10-14T16:57:05.773164+0000","last_became_peered":"2023-10-14T16:57:05.773164+0000","last_unstale":"2023-10-14T16:57:07.515618+0000","last_undegraded":"2023-10-14T16:57:07.515618+0000","last_fullsized":"2023-10-14T16:57:07.515618+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:59.950873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764868+0000","last_change":"2023-10-14T16:57:05.758752+0000","last_active":"2023-10-14T16:57:06.764868+0000","last_peered":"2023-10-14T16:57:06.764868+0000","last_clean":"2023-10-14T16:57:06.764868+0000","last_became_active":"2023-10-14T16:57:05.758376+0000","last_became_peered":"2023-10-14T16:57:05.758376+0000","last_unstale":"2023-10-14T16:57:06.764868+0000","last_undegraded":"2023-10-14T16:57:06.764868+0000","last_fullsized":"2023-10-14T16:57:06.764868+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:24.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046516+0000","last_change":"2023-10-14T16:57:05.770509+0000","last_active":"2023-10-14T16:57:07.046516+0000","last_peered":"2023-10-14T16:57:07.046516+0000","last_clean":"2023-10-14T16:57:07.046516+0000","last_became_active":"2023-10-14T16:57:05.770220+0000","last_became_peered":"2023-10-14T16:57:05.770220+0000","last_unstale":"2023-10-14T16:57:07.046516+0000","last_undegraded":"2023-10-14T16:57:07.046516+0000","last_fullsized":"2023-10-14T16:57:07.046516+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:50.398894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983747+0000","last_change":"2023-10-14T16:57:05.768312+0000","last_active":"2023-10-14T16:57:06.983747+0000","last_peered":"2023-10-14T16:57:06.983747+0000","last_clean":"2023-10-14T16:57:06.983747+0000","last_became_active":"2023-10-14T16:57:05.768004+0000","last_became_peered":"2023-10-14T16:57:05.768004+0000","last_unstale":"2023-10-14T16:57:06.983747+0000","last_undegraded":"2023-10-14T16:57:06.983747+0000","last_fullsized":"2023-10-14T16:57:06.983747+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:59:45.459458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763940+0000","last_change":"2023-10-14T16:57:05.752785+0000","last_active":"2023-10-14T16:57:06.763940+0000","last_peered":"2023-10-14T16:57:06.763940+0000","last_clean":"2023-10-14T16:57:06.763940+0000","last_became_active":"2023-10-14T16:57:05.752401+0000","last_became_peered":"2023-10-14T16:57:05.752401+0000","last_unstale":"2023-10-14T16:57:06.763940+0000","last_undegraded":"2023-10-14T16:57:06.763940+0000","last_fullsized":"2023-10-14T16:57:06.763940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:44:46.584055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983306+0000","last_change":"2023-10-14T16:57:05.770924+0000","last_active":"2023-10-14T16:57:06.983306+0000","last_peered":"2023-10-14T16:57:06.983306+0000","last_clean":"2023-10-14T16:57:06.983306+0000","last_became_active":"2023-10-14T16:57:05.770622+0000","last_became_peered":"2023-10-14T16:57:05.770622+0000","last_unstale":"2023-10-14T16:57:06.983306+0000","last_undegraded":"2023-10-14T16:57:06.983306+0000","last_fullsized":"2023-10-14T16:57:06.983306+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046044+0000","last_change":"2023-10-14T16:57:05.773392+0000","last_active":"2023-10-14T16:57:07.046044+0000","last_peered":"2023-10-14T16:57:07.046044+0000","last_clean":"2023-10-14T16:57:07.046044+0000","last_became_active":"2023-10-14T16:57:05.773108+0000","last_became_peered":"2023-10-14T16:57:05.773108+0000","last_unstale":"2023-10-14T16:57:07.046044+0000","last_undegraded":"2023-10-14T16:57:07.046044+0000","last_fullsized":"2023-10-14T16:57:07.046044+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:10.337141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046408+0000","last_change":"2023-10-14T16:57:05.762887+0000","last_active":"2023-10-14T16:57:07.046408+0000","last_peered":"2023-10-14T16:57:07.046408+0000","last_clean":"2023-10-14T16:57:07.046408+0000","last_became_active":"2023-10-14T16:57:05.762588+0000","last_became_peered":"2023-10-14T16:57:05.762588+0000","last_unstale":"2023-10-14T16:57:07.046408+0000","last_undegraded":"2023-10-14T16:57:07.046408+0000","last_fullsized":"2023-10-14T16:57:07.046408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:32.747286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764110+0000","last_change":"2023-10-14T16:57:05.768996+0000","last_active":"2023-10-14T16:57:06.764110+0000","last_peered":"2023-10-14T16:57:06.764110+0000","last_clean":"2023-10-14T16:57:06.764110+0000","last_became_active":"2023-10-14T16:57:05.768625+0000","last_became_peered":"2023-10-14T16:57:05.768625+0000","last_unstale":"2023-10-14T16:57:06.764110+0000","last_undegraded":"2023-10-14T16:57:06.764110+0000","last_fullsized":"2023-10-14T16:57:06.764110+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:54.341628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516555+0000","last_change":"2023-10-14T16:57:05.773933+0000","last_active":"2023-10-14T16:57:07.516555+0000","last_peered":"2023-10-14T16:57:07.516555+0000","last_clean":"2023-10-14T16:57:07.516555+0000","last_became_active":"2023-10-14T16:57:05.773475+0000","last_became_peered":"2023-10-14T16:57:05.773475+0000","last_unstale":"2023-10-14T16:57:07.516555+0000","last_undegraded":"2023-10-14T16:57:07.516555+0000","last_fullsized":"2023-10-14T16:57:07.516555+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:00.541027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"95'4","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.517014+0000","last_change":"2023-10-14T16:57:04.758777+0000","last_active":"2023-10-14T16:57:07.517014+0000","last_peered":"2023-10-14T16:57:07.517014+0000","last_clean":"2023-10-14T16:57:07.517014+0000","last_became_active":"2023-10-14T16:57:04.758506+0000","last_became_peered":"2023-10-14T16:57:04.758506+0000","last_unstale":"2023-10-14T16:57:07.517014+0000","last_undegraded":"2023-10-14T16:57:07.517014+0000","last_fullsized":"2023-10-14T16:57:07.517014+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:14.220829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762757+0000","last_change":"2023-10-14T16:57:05.769089+0000","last_active":"2023-10-14T16:57:06.762757+0000","last_peered":"2023-10-14T16:57:06.762757+0000","last_clean":"2023-10-14T16:57:06.762757+0000","last_became_active":"2023-10-14T16:57:05.768620+0000","last_became_peered":"2023-10-14T16:57:05.768620+0000","last_unstale":"2023-10-14T16:57:06.762757+0000","last_undegraded":"2023-10-14T16:57:06.762757+0000","last_fullsized":"2023-10-14T16:57:06.762757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:07.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516401+0000","last_change":"2023-10-14T16:57:04.747213+0000","last_active":"2023-10-14T16:57:07.516401+0000","last_peered":"2023-10-14T16:57:07.516401+0000","last_clean":"2023-10-14T16:57:07.516401+0000","last_became_active":"2023-10-14T16:57:04.746927+0000","last_became_peered":"2023-10-14T16:57:04.746927+0000","last_unstale":"2023-10-14T16:57:07.516401+0000","last_undegraded":"2023-10-14T16:57:07.516401+0000","last_fullsized":"2023-10-14T16:57:07.516401+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:58.161053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763951+0000","last_change":"2023-10-14T16:57:05.770939+0000","last_active":"2023-10-14T16:57:06.763951+0000","last_peered":"2023-10-14T16:57:06.763951+0000","last_clean":"2023-10-14T16:57:06.763951+0000","last_became_active":"2023-10-14T16:57:05.768828+0000","last_became_peered":"2023-10-14T16:57:05.768828+0000","last_unstale":"2023-10-14T16:57:06.763951+0000","last_undegraded":"2023-10-14T16:57:06.763951+0000","last_fullsized":"2023-10-14T16:57:06.763951+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:12.225048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764819+0000","last_change":"2023-10-14T16:57:04.758702+0000","last_active":"2023-10-14T16:57:06.764819+0000","last_peered":"2023-10-14T16:57:06.764819+0000","last_clean":"2023-10-14T16:57:06.764819+0000","last_became_active":"2023-10-14T16:57:04.758442+0000","last_became_peered":"2023-10-14T16:57:04.758442+0000","last_unstale":"2023-10-14T16:57:06.764819+0000","last_undegraded":"2023-10-14T16:57:06.764819+0000","last_fullsized":"2023-10-14T16:57:06.764819+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:46.405350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516708+0000","last_change":"2023-10-14T16:57:05.766409+0000","last_active":"2023-10-14T16:57:07.516708+0000","last_peered":"2023-10-14T16:57:07.516708+0000","last_clean":"2023-10-14T16:57:07.516708+0000","last_became_active":"2023-10-14T16:57:05.765856+0000","last_became_peered":"2023-10-14T16:57:05.765856+0000","last_unstale":"2023-10-14T16:57:07.516708+0000","last_undegraded":"2023-10-14T16:57:07.516708+0000","last_fullsized":"2023-10-14T16:57:07.516708+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:05.067018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764563+0000","last_change":"2023-10-14T16:57:04.749625+0000","last_active":"2023-10-14T16:57:06.764563+0000","last_peered":"2023-10-14T16:57:06.764563+0000","last_clean":"2023-10-14T16:57:06.764563+0000","last_became_active":"2023-10-14T16:57:04.747285+0000","last_became_peered":"2023-10-14T16:57:04.747285+0000","last_unstale":"2023-10-14T16:57:06.764563+0000","last_undegraded":"2023-10-14T16:57:06.764563+0000","last_fullsized":"2023-10-14T16:57:06.764563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:09.992738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046355+0000","last_change":"2023-10-14T16:57:05.765059+0000","last_active":"2023-10-14T16:57:07.046355+0000","last_peered":"2023-10-14T16:57:07.046355+0000","last_clean":"2023-10-14T16:57:07.046355+0000","last_became_active":"2023-10-14T16:57:05.764764+0000","last_became_peered":"2023-10-14T16:57:05.764764+0000","last_unstale":"2023-10-14T16:57:07.046355+0000","last_undegraded":"2023-10-14T16:57:07.046355+0000","last_fullsized":"2023-10-14T16:57:07.046355+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:14.104217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983645+0000","last_change":"2023-10-14T16:57:04.754798+0000","last_active":"2023-10-14T16:57:06.983645+0000","last_peered":"2023-10-14T16:57:06.983645+0000","last_clean":"2023-10-14T16:57:06.983645+0000","last_became_active":"2023-10-14T16:57:04.754510+0000","last_became_peered":"2023-10-14T16:57:04.754510+0000","last_unstale":"2023-10-14T16:57:06.983645+0000","last_undegraded":"2023-10-14T16:57:06.983645+0000","last_fullsized":"2023-10-14T16:57:06.983645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:04.062504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764237+0000","last_change":"2023-10-14T16:57:05.762673+0000","last_active":"2023-10-14T16:57:06.764237+0000","last_peered":"2023-10-14T16:57:06.764237+0000","last_clean":"2023-10-14T16:57:06.764237+0000","last_became_active":"2023-10-14T16:57:05.762355+0000","last_became_peered":"2023-10-14T16:57:05.762355+0000","last_unstale":"2023-10-14T16:57:06.764237+0000","last_undegraded":"2023-10-14T16:57:06.764237+0000","last_fullsized":"2023-10-14T16:57:06.764237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:24.150960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764269+0000","last_change":"2023-10-14T16:57:04.743873+0000","last_active":"2023-10-14T16:57:06.764269+0000","last_peered":"2023-10-14T16:57:06.764269+0000","last_clean":"2023-10-14T16:57:06.764269+0000","last_became_active":"2023-10-14T16:57:04.742913+0000","last_became_peered":"2023-10-14T16:57:04.742913+0000","last_unstale":"2023-10-14T16:57:06.764269+0000","last_undegraded":"2023-10-14T16:57:06.764269+0000","last_fullsized":"2023-10-14T16:57:06.764269+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:41.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.760758+0000","last_change":"2023-10-14T16:57:05.747650+0000","last_active":"2023-10-14T16:57:06.760758+0000","last_peered":"2023-10-14T16:57:06.760758+0000","last_clean":"2023-10-14T16:57:06.760758+0000","last_became_active":"2023-10-14T16:57:05.744605+0000","last_became_peered":"2023-10-14T16:57:05.744605+0000","last_unstale":"2023-10-14T16:57:06.760758+0000","last_undegraded":"2023-10-14T16:57:06.760758+0000","last_fullsized":"2023-10-14T16:57:06.760758+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092621+0000","last_change":"2023-10-14T16:57:04.754128+0000","last_active":"2023-10-14T16:57:07.092621+0000","last_peered":"2023-10-14T16:57:07.092621+0000","last_clean":"2023-10-14T16:57:07.092621+0000","last_became_active":"2023-10-14T16:57:04.753778+0000","last_became_peered":"2023-10-14T16:57:04.753778+0000","last_unstale":"2023-10-14T16:57:07.092621+0000","last_undegraded":"2023-10-14T16:57:07.092621+0000","last_fullsized":"2023-10-14T16:57:07.092621+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:38.170064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515849+0000","last_change":"2023-10-14T16:57:05.761399+0000","last_active":"2023-10-14T16:57:07.515849+0000","last_peered":"2023-10-14T16:57:07.515849+0000","last_clean":"2023-10-14T16:57:07.515849+0000","last_became_active":"2023-10-14T16:57:05.761074+0000","last_became_peered":"2023-10-14T16:57:05.761074+0000","last_unstale":"2023-10-14T16:57:07.515849+0000","last_undegraded":"2023-10-14T16:57:07.515849+0000","last_fullsized":"2023-10-14T16:57:07.515849+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.760854+0000","last_change":"2023-10-14T16:57:04.755846+0000","last_active":"2023-10-14T16:57:06.760854+0000","last_peered":"2023-10-14T16:57:06.760854+0000","last_clean":"2023-10-14T16:57:06.760854+0000","last_became_active":"2023-10-14T16:57:04.755364+0000","last_became_peered":"2023-10-14T16:57:04.755364+0000","last_unstale":"2023-10-14T16:57:06.760854+0000","last_undegraded":"2023-10-14T16:57:06.760854+0000","last_fullsized":"2023-10-14T16:57:06.760854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:32.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046526+0000","last_change":"2023-10-14T16:57:05.765644+0000","last_active":"2023-10-14T16:57:07.046526+0000","last_peered":"2023-10-14T16:57:07.046526+0000","last_clean":"2023-10-14T16:57:07.046526+0000","last_became_active":"2023-10-14T16:57:05.765346+0000","last_became_peered":"2023-10-14T16:57:05.765346+0000","last_unstale":"2023-10-14T16:57:07.046526+0000","last_undegraded":"2023-10-14T16:57:07.046526+0000","last_fullsized":"2023-10-14T16:57:07.046526+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:35.129696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765049+0000","last_change":"2023-10-14T16:57:04.757810+0000","last_active":"2023-10-14T16:57:06.765049+0000","last_peered":"2023-10-14T16:57:06.765049+0000","last_clean":"2023-10-14T16:57:06.765049+0000","last_became_active":"2023-10-14T16:57:04.753731+0000","last_became_peered":"2023-10-14T16:57:04.753731+0000","last_unstale":"2023-10-14T16:57:06.765049+0000","last_undegraded":"2023-10-14T16:57:06.765049+0000","last_fullsized":"2023-10-14T16:57:06.765049+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:44:26.122699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516734+0000","last_change":"2023-10-14T16:57:05.774079+0000","last_active":"2023-10-14T16:57:07.516734+0000","last_peered":"2023-10-14T16:57:07.516734+0000","last_clean":"2023-10-14T16:57:07.516734+0000","last_became_active":"2023-10-14T16:57:05.773594+0000","last_became_peered":"2023-10-14T16:57:05.773594+0000","last_unstale":"2023-10-14T16:57:07.516734+0000","last_undegraded":"2023-10-14T16:57:07.516734+0000","last_fullsized":"2023-10-14T16:57:07.516734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:53.913205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765423+0000","last_change":"2023-10-14T16:57:04.757011+0000","last_active":"2023-10-14T16:57:06.765423+0000","last_peered":"2023-10-14T16:57:06.765423+0000","last_clean":"2023-10-14T16:57:06.765423+0000","last_became_active":"2023-10-14T16:57:04.756701+0000","last_became_peered":"2023-10-14T16:57:04.756701+0000","last_unstale":"2023-10-14T16:57:06.765423+0000","last_undegraded":"2023-10-14T16:57:06.765423+0000","last_fullsized":"2023-10-14T16:57:06.765423+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:39:10.076101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764301+0000","last_change":"2023-10-14T16:57:05.769487+0000","last_active":"2023-10-14T16:57:06.764301+0000","last_peered":"2023-10-14T16:57:06.764301+0000","last_clean":"2023-10-14T16:57:06.764301+0000","last_became_active":"2023-10-14T16:57:05.767403+0000","last_became_peered":"2023-10-14T16:57:05.767403+0000","last_unstale":"2023-10-14T16:57:06.764301+0000","last_undegraded":"2023-10-14T16:57:06.764301+0000","last_fullsized":"2023-10-14T16:57:06.764301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"95'3","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983911+0000","last_change":"2023-10-14T16:57:04.757319+0000","last_active":"2023-10-14T16:57:06.983911+0000","last_peered":"2023-10-14T16:57:06.983911+0000","last_clean":"2023-10-14T16:57:06.983911+0000","last_became_active":"2023-10-14T16:57:04.757056+0000","last_became_peered":"2023-10-14T16:57:04.757056+0000","last_unstale":"2023-10-14T16:57:06.983911+0000","last_undegraded":"2023-10-14T16:57:06.983911+0000","last_fullsized":"2023-10-14T16:57:06.983911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:07.937751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761862+0000","last_change":"2023-10-14T16:57:05.756437+0000","last_active":"2023-10-14T16:57:06.761862+0000","last_peered":"2023-10-14T16:57:06.761862+0000","last_clean":"2023-10-14T16:57:06.761862+0000","last_became_active":"2023-10-14T16:57:05.756143+0000","last_became_peered":"2023-10-14T16:57:05.756143+0000","last_unstale":"2023-10-14T16:57:06.761862+0000","last_undegraded":"2023-10-14T16:57:06.761862+0000","last_fullsized":"2023-10-14T16:57:06.761862+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"95'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.093126+0000","last_change":"2023-10-14T16:57:04.747004+0000","last_active":"2023-10-14T16:57:07.093126+0000","last_peered":"2023-10-14T16:57:07.093126+0000","last_clean":"2023-10-14T16:57:07.093126+0000","last_became_active":"2023-10-14T16:57:04.746736+0000","last_became_peered":"2023-10-14T16:57:04.746736+0000","last_unstale":"2023-10-14T16:57:07.093126+0000","last_undegraded":"2023-10-14T16:57:07.093126+0000","last_fullsized":"2023-10-14T16:57:07.093126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:31.895160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515919+0000","last_change":"2023-10-14T16:57:05.763543+0000","last_active":"2023-10-14T16:57:07.515919+0000","last_peered":"2023-10-14T16:57:07.515919+0000","last_clean":"2023-10-14T16:57:07.515919+0000","last_became_active":"2023-10-14T16:57:05.761127+0000","last_became_peered":"2023-10-14T16:57:05.761127+0000","last_unstale":"2023-10-14T16:57:07.515919+0000","last_undegraded":"2023-10-14T16:57:07.515919+0000","last_fullsized":"2023-10-14T16:57:07.515919+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:08.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092710+0000","last_change":"2023-10-14T16:57:04.755308+0000","last_active":"2023-10-14T16:57:07.092710+0000","last_peered":"2023-10-14T16:57:07.092710+0000","last_clean":"2023-10-14T16:57:07.092710+0000","last_became_active":"2023-10-14T16:57:04.755063+0000","last_became_peered":"2023-10-14T16:57:04.755063+0000","last_unstale":"2023-10-14T16:57:07.092710+0000","last_undegraded":"2023-10-14T16:57:07.092710+0000","last_fullsized":"2023-10-14T16:57:07.092710+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:36.586765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046389+0000","last_change":"2023-10-14T16:57:05.760952+0000","last_active":"2023-10-14T16:57:07.046389+0000","last_peered":"2023-10-14T16:57:07.046389+0000","last_clean":"2023-10-14T16:57:07.046389+0000","last_became_active":"2023-10-14T16:57:05.760639+0000","last_became_peered":"2023-10-14T16:57:05.760639+0000","last_unstale":"2023-10-14T16:57:07.046389+0000","last_undegraded":"2023-10-14T16:57:07.046389+0000","last_fullsized":"2023-10-14T16:57:07.046389+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:32.765750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764534+0000","last_change":"2023-10-14T16:57:04.753205+0000","last_active":"2023-10-14T16:57:06.764534+0000","last_peered":"2023-10-14T16:57:06.764534+0000","last_clean":"2023-10-14T16:57:06.764534+0000","last_became_active":"2023-10-14T16:57:04.750580+0000","last_became_peered":"2023-10-14T16:57:04.750580+0000","last_unstale":"2023-10-14T16:57:06.764534+0000","last_undegraded":"2023-10-14T16:57:06.764534+0000","last_fullsized":"2023-10-14T16:57:06.764534+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:42.503829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515606+0000","last_change":"2023-10-14T16:57:05.771757+0000","last_active":"2023-10-14T16:57:07.515606+0000","last_peered":"2023-10-14T16:57:07.515606+0000","last_clean":"2023-10-14T16:57:07.515606+0000","last_became_active":"2023-10-14T16:57:05.770291+0000","last_became_peered":"2023-10-14T16:57:05.770291+0000","last_unstale":"2023-10-14T16:57:07.515606+0000","last_undegraded":"2023-10-14T16:57:07.515606+0000","last_fullsized":"2023-10-14T16:57:07.515606+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046170+0000","last_change":"2023-10-14T16:57:04.755533+0000","last_active":"2023-10-14T16:57:07.046170+0000","last_peered":"2023-10-14T16:57:07.046170+0000","last_clean":"2023-10-14T16:57:07.046170+0000","last_became_active":"2023-10-14T16:57:04.755238+0000","last_became_peered":"2023-10-14T16:57:04.755238+0000","last_unstale":"2023-10-14T16:57:07.046170+0000","last_undegraded":"2023-10-14T16:57:07.046170+0000","last_fullsized":"2023-10-14T16:57:07.046170+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:55.311294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761165+0000","last_change":"2023-10-14T16:57:05.763650+0000","last_active":"2023-10-14T16:57:06.761165+0000","last_peered":"2023-10-14T16:57:06.761165+0000","last_clean":"2023-10-14T16:57:06.761165+0000","last_became_active":"2023-10-14T16:57:05.763344+0000","last_became_peered":"2023-10-14T16:57:05.763344+0000","last_unstale":"2023-10-14T16:57:06.761165+0000","last_undegraded":"2023-10-14T16:57:06.761165+0000","last_fullsized":"2023-10-14T16:57:06.761165+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:54.341628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"95'8","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765372+0000","last_change":"2023-10-14T16:57:04.757605+0000","last_active":"2023-10-14T16:57:06.765372+0000","last_peered":"2023-10-14T16:57:06.765372+0000","last_clean":"2023-10-14T16:57:06.765372+0000","last_became_active":"2023-10-14T16:57:04.757291+0000","last_became_peered":"2023-10-14T16:57:04.757291+0000","last_unstale":"2023-10-14T16:57:06.765372+0000","last_undegraded":"2023-10-14T16:57:06.765372+0000","last_fullsized":"2023-10-14T16:57:06.765372+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:23.406446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765625+0000","last_change":"2023-10-14T16:57:05.754950+0000","last_active":"2023-10-14T16:57:06.765625+0000","last_peered":"2023-10-14T16:57:06.765625+0000","last_clean":"2023-10-14T16:57:06.765625+0000","last_became_active":"2023-10-14T16:57:05.754596+0000","last_became_peered":"2023-10-14T16:57:05.754596+0000","last_unstale":"2023-10-14T16:57:06.765625+0000","last_undegraded":"2023-10-14T16:57:06.765625+0000","last_fullsized":"2023-10-14T16:57:06.765625+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:32.315257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765701+0000","last_change":"2023-10-14T16:57:04.748816+0000","last_active":"2023-10-14T16:57:06.765701+0000","last_peered":"2023-10-14T16:57:06.765701+0000","last_clean":"2023-10-14T16:57:06.765701+0000","last_became_active":"2023-10-14T16:57:04.748406+0000","last_became_peered":"2023-10-14T16:57:04.748406+0000","last_unstale":"2023-10-14T16:57:06.765701+0000","last_undegraded":"2023-10-14T16:57:06.765701+0000","last_fullsized":"2023-10-14T16:57:06.765701+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:52.333555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.517092+0000","last_change":"2023-10-14T16:57:05.765528+0000","last_active":"2023-10-14T16:57:07.517092+0000","last_peered":"2023-10-14T16:57:07.517092+0000","last_clean":"2023-10-14T16:57:07.517092+0000","last_became_active":"2023-10-14T16:57:05.765233+0000","last_became_peered":"2023-10-14T16:57:05.765233+0000","last_unstale":"2023-10-14T16:57:07.517092+0000","last_undegraded":"2023-10-14T16:57:07.517092+0000","last_fullsized":"2023-10-14T16:57:07.517092+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765734+0000","last_change":"2023-10-14T16:57:04.758087+0000","last_active":"2023-10-14T16:57:06.765734+0000","last_peered":"2023-10-14T16:57:06.765734+0000","last_clean":"2023-10-14T16:57:06.765734+0000","last_became_active":"2023-10-14T16:57:04.757802+0000","last_became_peered":"2023-10-14T16:57:04.757802+0000","last_unstale":"2023-10-14T16:57:06.765734+0000","last_undegraded":"2023-10-14T16:57:06.765734+0000","last_fullsized":"2023-10-14T16:57:06.765734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:53.163070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.093039+0000","last_change":"2023-10-14T16:57:05.758955+0000","last_active":"2023-10-14T16:57:07.093039+0000","last_peered":"2023-10-14T16:57:07.093039+0000","last_clean":"2023-10-14T16:57:07.093039+0000","last_became_active":"2023-10-14T16:57:05.758592+0000","last_became_peered":"2023-10-14T16:57:05.758592+0000","last_unstale":"2023-10-14T16:57:07.093039+0000","last_undegraded":"2023-10-14T16:57:07.093039+0000","last_fullsized":"2023-10-14T16:57:07.093039+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:44:50.497921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765099+0000","last_change":"2023-10-14T16:57:04.755600+0000","last_active":"2023-10-14T16:57:06.765099+0000","last_peered":"2023-10-14T16:57:06.765099+0000","last_clean":"2023-10-14T16:57:06.765099+0000","last_became_active":"2023-10-14T16:57:04.755074+0000","last_became_peered":"2023-10-14T16:57:04.755074+0000","last_unstale":"2023-10-14T16:57:06.765099+0000","last_undegraded":"2023-10-14T16:57:06.765099+0000","last_fullsized":"2023-10-14T16:57:06.765099+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:32.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761890+0000","last_change":"2023-10-14T16:57:05.746927+0000","last_active":"2023-10-14T16:57:06.761890+0000","last_peered":"2023-10-14T16:57:06.761890+0000","last_clean":"2023-10-14T16:57:06.761890+0000","last_became_active":"2023-10-14T16:57:05.746202+0000","last_became_peered":"2023-10-14T16:57:05.746202+0000","last_unstale":"2023-10-14T16:57:06.761890+0000","last_undegraded":"2023-10-14T16:57:06.761890+0000","last_fullsized":"2023-10-14T16:57:06.761890+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516217+0000","last_change":"2023-10-14T16:57:04.760857+0000","last_active":"2023-10-14T16:57:07.516217+0000","last_peered":"2023-10-14T16:57:07.516217+0000","last_clean":"2023-10-14T16:57:07.516217+0000","last_became_active":"2023-10-14T16:57:04.760569+0000","last_became_peered":"2023-10-14T16:57:04.760569+0000","last_unstale":"2023-10-14T16:57:07.516217+0000","last_undegraded":"2023-10-14T16:57:07.516217+0000","last_fullsized":"2023-10-14T16:57:07.516217+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:59:58.160172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.984170+0000","last_change":"2023-10-14T16:57:05.763910+0000","last_active":"2023-10-14T16:57:06.984170+0000","last_peered":"2023-10-14T16:57:06.984170+0000","last_clean":"2023-10-14T16:57:06.984170+0000","last_became_active":"2023-10-14T16:57:05.763549+0000","last_became_peered":"2023-10-14T16:57:05.763549+0000","last_unstale":"2023-10-14T16:57:06.984170+0000","last_undegraded":"2023-10-14T16:57:06.984170+0000","last_fullsized":"2023-10-14T16:57:06.984170+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:41.170694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764780+0000","last_change":"2023-10-14T16:57:04.744062+0000","last_active":"2023-10-14T16:57:06.764780+0000","last_peered":"2023-10-14T16:57:06.764780+0000","last_clean":"2023-10-14T16:57:06.764780+0000","last_became_active":"2023-10-14T16:57:04.742678+0000","last_became_peered":"2023-10-14T16:57:04.742678+0000","last_unstale":"2023-10-14T16:57:06.764780+0000","last_undegraded":"2023-10-14T16:57:06.764780+0000","last_fullsized":"2023-10-14T16:57:06.764780+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:52.356756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983641+0000","last_change":"2023-10-14T16:57:05.773880+0000","last_active":"2023-10-14T16:57:06.983641+0000","last_peered":"2023-10-14T16:57:06.983641+0000","last_clean":"2023-10-14T16:57:06.983641+0000","last_became_active":"2023-10-14T16:57:05.773559+0000","last_became_peered":"2023-10-14T16:57:05.773559+0000","last_unstale":"2023-10-14T16:57:06.983641+0000","last_undegraded":"2023-10-14T16:57:06.983641+0000","last_fullsized":"2023-10-14T16:57:06.983641+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:13.121991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046441+0000","last_change":"2023-10-14T16:57:04.755513+0000","last_active":"2023-10-14T16:57:07.046441+0000","last_peered":"2023-10-14T16:57:07.046441+0000","last_clean":"2023-10-14T16:57:07.046441+0000","last_became_active":"2023-10-14T16:57:04.755205+0000","last_became_peered":"2023-10-14T16:57:04.755205+0000","last_unstale":"2023-10-14T16:57:07.046441+0000","last_undegraded":"2023-10-14T16:57:07.046441+0000","last_fullsized":"2023-10-14T16:57:07.046441+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:52.996079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983631+0000","last_change":"2023-10-14T16:57:05.773951+0000","last_active":"2023-10-14T16:57:06.983631+0000","last_peered":"2023-10-14T16:57:06.983631+0000","last_clean":"2023-10-14T16:57:06.983631+0000","last_became_active":"2023-10-14T16:57:05.772474+0000","last_became_peered":"2023-10-14T16:57:05.772474+0000","last_unstale":"2023-10-14T16:57:06.983631+0000","last_undegraded":"2023-10-14T16:57:06.983631+0000","last_fullsized":"2023-10-14T16:57:06.983631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.637099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515427+0000","last_change":"2023-10-14T16:57:04.747342+0000","last_active":"2023-10-14T16:57:07.515427+0000","last_peered":"2023-10-14T16:57:07.515427+0000","last_clean":"2023-10-14T16:57:07.515427+0000","last_became_active":"2023-10-14T16:57:04.747073+0000","last_became_peered":"2023-10-14T16:57:04.747073+0000","last_unstale":"2023-10-14T16:57:07.515427+0000","last_undegraded":"2023-10-14T16:57:07.515427+0000","last_fullsized":"2023-10-14T16:57:07.515427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:12.943790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516296+0000","last_change":"2023-10-14T16:57:05.768476+0000","last_active":"2023-10-14T16:57:07.516296+0000","last_peered":"2023-10-14T16:57:07.516296+0000","last_clean":"2023-10-14T16:57:07.516296+0000","last_became_active":"2023-10-14T16:57:05.767714+0000","last_became_peered":"2023-10-14T16:57:05.767714+0000","last_unstale":"2023-10-14T16:57:07.516296+0000","last_undegraded":"2023-10-14T16:57:07.516296+0000","last_fullsized":"2023-10-14T16:57:07.516296+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:39.174579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765808+0000","last_change":"2023-10-14T16:57:04.748379+0000","last_active":"2023-10-14T16:57:06.765808+0000","last_peered":"2023-10-14T16:57:06.765808+0000","last_clean":"2023-10-14T16:57:06.765808+0000","last_became_active":"2023-10-14T16:57:04.748092+0000","last_became_peered":"2023-10-14T16:57:04.748092+0000","last_unstale":"2023-10-14T16:57:06.765808+0000","last_undegraded":"2023-10-14T16:57:06.765808+0000","last_fullsized":"2023-10-14T16:57:06.765808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:11.290225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046406+0000","last_change":"2023-10-14T16:57:05.759025+0000","last_active":"2023-10-14T16:57:07.046406+0000","last_peered":"2023-10-14T16:57:07.046406+0000","last_clean":"2023-10-14T16:57:07.046406+0000","last_became_active":"2023-10-14T16:57:05.758647+0000","last_became_peered":"2023-10-14T16:57:05.758647+0000","last_unstale":"2023-10-14T16:57:07.046406+0000","last_undegraded":"2023-10-14T16:57:07.046406+0000","last_fullsized":"2023-10-14T16:57:07.046406+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:17.297679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046787+0000","last_change":"2023-10-14T16:57:04.754959+0000","last_active":"2023-10-14T16:57:07.046787+0000","last_peered":"2023-10-14T16:57:07.046787+0000","last_clean":"2023-10-14T16:57:07.046787+0000","last_became_active":"2023-10-14T16:57:04.754656+0000","last_became_peered":"2023-10-14T16:57:04.754656+0000","last_unstale":"2023-10-14T16:57:07.046787+0000","last_undegraded":"2023-10-14T16:57:07.046787+0000","last_fullsized":"2023-10-14T16:57:07.046787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:06:27.317672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516160+0000","last_change":"2023-10-14T16:57:05.773575+0000","last_active":"2023-10-14T16:57:07.516160+0000","last_peered":"2023-10-14T16:57:07.516160+0000","last_clean":"2023-10-14T16:57:07.516160+0000","last_became_active":"2023-10-14T16:57:05.773222+0000","last_became_peered":"2023-10-14T16:57:05.773222+0000","last_unstale":"2023-10-14T16:57:07.516160+0000","last_undegraded":"2023-10-14T16:57:07.516160+0000","last_fullsized":"2023-10-14T16:57:07.516160+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:40.239061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761996+0000","last_change":"2023-10-14T16:57:04.753889+0000","last_active":"2023-10-14T16:57:06.761996+0000","last_peered":"2023-10-14T16:57:06.761996+0000","last_clean":"2023-10-14T16:57:06.761996+0000","last_became_active":"2023-10-14T16:57:04.753487+0000","last_became_peered":"2023-10-14T16:57:04.753487+0000","last_unstale":"2023-10-14T16:57:06.761996+0000","last_undegraded":"2023-10-14T16:57:06.761996+0000","last_fullsized":"2023-10-14T16:57:06.761996+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:00.434810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516852+0000","last_change":"2023-10-14T16:57:05.769678+0000","last_active":"2023-10-14T16:57:07.516852+0000","last_peered":"2023-10-14T16:57:07.516852+0000","last_clean":"2023-10-14T16:57:07.516852+0000","last_became_active":"2023-10-14T16:57:05.769288+0000","last_became_peered":"2023-10-14T16:57:05.769288+0000","last_unstale":"2023-10-14T16:57:07.516852+0000","last_undegraded":"2023-10-14T16:57:07.516852+0000","last_fullsized":"2023-10-14T16:57:07.516852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:03:21.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516056+0000","last_change":"2023-10-14T16:57:04.750718+0000","last_active":"2023-10-14T16:57:07.516056+0000","last_peered":"2023-10-14T16:57:07.516056+0000","last_clean":"2023-10-14T16:57:07.516056+0000","last_became_active":"2023-10-14T16:57:04.750408+0000","last_became_peered":"2023-10-14T16:57:04.750408+0000","last_unstale":"2023-10-14T16:57:07.516056+0000","last_undegraded":"2023-10-14T16:57:07.516056+0000","last_fullsized":"2023-10-14T16:57:07.516056+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:11.220533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761478+0000","last_change":"2023-10-14T16:57:05.765342+0000","last_active":"2023-10-14T16:57:06.761478+0000","last_peered":"2023-10-14T16:57:06.761478+0000","last_clean":"2023-10-14T16:57:06.761478+0000","last_became_active":"2023-10-14T16:57:05.764526+0000","last_became_peered":"2023-10-14T16:57:05.764526+0000","last_unstale":"2023-10-14T16:57:06.761478+0000","last_undegraded":"2023-10-14T16:57:06.761478+0000","last_fullsized":"2023-10-14T16:57:06.761478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761367+0000","last_change":"2023-10-14T16:57:04.755618+0000","last_active":"2023-10-14T16:57:06.761367+0000","last_peered":"2023-10-14T16:57:06.761367+0000","last_clean":"2023-10-14T16:57:06.761367+0000","last_became_active":"2023-10-14T16:57:04.755334+0000","last_became_peered":"2023-10-14T16:57:04.755334+0000","last_unstale":"2023-10-14T16:57:06.761367+0000","last_undegraded":"2023-10-14T16:57:06.761367+0000","last_fullsized":"2023-10-14T16:57:06.761367+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:08:48.464845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764333+0000","last_change":"2023-10-14T16:57:05.765252+0000","last_active":"2023-10-14T16:57:06.764333+0000","last_peered":"2023-10-14T16:57:06.764333+0000","last_clean":"2023-10-14T16:57:06.764333+0000","last_became_active":"2023-10-14T16:57:05.761688+0000","last_became_peered":"2023-10-14T16:57:05.761688+0000","last_unstale":"2023-10-14T16:57:06.764333+0000","last_undegraded":"2023-10-14T16:57:06.764333+0000","last_fullsized":"2023-10-14T16:57:06.764333+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092452+0000","last_change":"2023-10-14T16:57:04.753189+0000","last_active":"2023-10-14T16:57:07.092452+0000","last_peered":"2023-10-14T16:57:07.092452+0000","last_clean":"2023-10-14T16:57:07.092452+0000","last_became_active":"2023-10-14T16:57:04.752822+0000","last_became_peered":"2023-10-14T16:57:04.752822+0000","last_unstale":"2023-10-14T16:57:07.092452+0000","last_undegraded":"2023-10-14T16:57:07.092452+0000","last_fullsized":"2023-10-14T16:57:07.092452+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:05.478120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092873+0000","last_change":"2023-10-14T16:57:05.761047+0000","last_active":"2023-10-14T16:57:07.092873+0000","last_peered":"2023-10-14T16:57:07.092873+0000","last_clean":"2023-10-14T16:57:07.092873+0000","last_became_active":"2023-10-14T16:57:05.758357+0000","last_became_peered":"2023-10-14T16:57:05.758357+0000","last_unstale":"2023-10-14T16:57:07.092873+0000","last_undegraded":"2023-10-14T16:57:07.092873+0000","last_fullsized":"2023-10-14T16:57:07.092873+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:57:45.952268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762363+0000","last_change":"2023-10-14T16:57:04.757128+0000","last_active":"2023-10-14T16:57:06.762363+0000","last_peered":"2023-10-14T16:57:06.762363+0000","last_clean":"2023-10-14T16:57:06.762363+0000","last_became_active":"2023-10-14T16:57:04.756835+0000","last_became_peered":"2023-10-14T16:57:04.756835+0000","last_unstale":"2023-10-14T16:57:06.762363+0000","last_undegraded":"2023-10-14T16:57:06.762363+0000","last_fullsized":"2023-10-14T16:57:06.762363+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:18.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763108+0000","last_change":"2023-10-14T16:57:05.765424+0000","last_active":"2023-10-14T16:57:06.763108+0000","last_peered":"2023-10-14T16:57:06.763108+0000","last_clean":"2023-10-14T16:57:06.763108+0000","last_became_active":"2023-10-14T16:57:05.764897+0000","last_became_peered":"2023-10-14T16:57:05.764897+0000","last_unstale":"2023-10-14T16:57:06.763108+0000","last_undegraded":"2023-10-14T16:57:06.763108+0000","last_fullsized":"2023-10-14T16:57:06.763108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:41.683610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764385+0000","last_change":"2023-10-14T16:57:04.744497+0000","last_active":"2023-10-14T16:57:06.764385+0000","last_peered":"2023-10-14T16:57:06.764385+0000","last_clean":"2023-10-14T16:57:06.764385+0000","last_became_active":"2023-10-14T16:57:04.744208+0000","last_became_peered":"2023-10-14T16:57:04.744208+0000","last_unstale":"2023-10-14T16:57:06.764385+0000","last_undegraded":"2023-10-14T16:57:06.764385+0000","last_fullsized":"2023-10-14T16:57:06.764385+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:34.579935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983971+0000","last_change":"2023-10-14T16:57:05.773085+0000","last_active":"2023-10-14T16:57:06.983971+0000","last_peered":"2023-10-14T16:57:06.983971+0000","last_clean":"2023-10-14T16:57:06.983971+0000","last_became_active":"2023-10-14T16:57:05.772765+0000","last_became_peered":"2023-10-14T16:57:05.772765+0000","last_unstale":"2023-10-14T16:57:06.983971+0000","last_undegraded":"2023-10-14T16:57:06.983971+0000","last_fullsized":"2023-10-14T16:57:06.983971+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:22:58.024289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764591+0000","last_change":"2023-10-14T16:57:04.744312+0000","last_active":"2023-10-14T16:57:06.764591+0000","last_peered":"2023-10-14T16:57:06.764591+0000","last_clean":"2023-10-14T16:57:06.764591+0000","last_became_active":"2023-10-14T16:57:04.743749+0000","last_became_peered":"2023-10-14T16:57:04.743749+0000","last_unstale":"2023-10-14T16:57:06.764591+0000","last_undegraded":"2023-10-14T16:57:06.764591+0000","last_fullsized":"2023-10-14T16:57:06.764591+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:21.298963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"84'172","reported_seq":433,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T16:57:05.751859+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:57:05.751859+0000","last_peered":"2023-10-14T16:57:05.751859+0000","last_clean":"2023-10-14T16:57:05.751859+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:57:05.751859+0000","last_undegraded":"2023-10-14T16:57:05.751859+0000","last_fullsized":"2023-10-14T16:57:05.751859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1773568,"data_stored":1152640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.169,"15min":1.1699999999999999},"min":{"1min":0.66000000000000003,"5min":0.56599999999999995,"15min":0.51500000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.73299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6279999999999999,"5min":1.8400000000000001,"15min":2.117},"last":0.79400000000000004}]},{"osd":1,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1890000000000001,"15min":1.2230000000000001},"min":{"1min":0.73799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3560000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.236,"15min":1.2569999999999999},"min":{"1min":0.66700000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.722,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.026}]},{"osd":2,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1910000000000001,"15min":1.224},"min":{"1min":0.623,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.78900000000000003,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.1589999999999998},"last":0.999}]},{"osd":3,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":1.369},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.996,"5min":1.996,"15min":2.4159999999999999},"last":1.04},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4259999999999999,"15min":1.446},"min":{"1min":0.85599999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0880000000000001},"last":1.1259999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.52600000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.881,"5min":2.105,"15min":2.1819999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.161}]},{"osd":5,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3540000000000001,"15min":1.385},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0390000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4450000000000001,"15min":1.4590000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.141}]},{"osd":6,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.532,"15min":1.542},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":1.077},{"interface":"front","average":{"1min":1.587,"5min":1.4790000000000001,"15min":1.4830000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.1100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575050,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36500,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335340,"statfs":{"total":96636764160,"available":96599388160,"internally_reserved":0,"allocated":1179648,"data_stored":561748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1519999999999999,"15min":1.153},"min":{"1min":0.81799999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.524,"5min":2.137,"15min":2.137},"last":1.327},{"interface":"front","average":{"1min":1.141,"5min":1.222,"15min":1.206},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8100000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1779999999999999,"15min":1.1850000000000001},"min":{"1min":0.77000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.593,"5min":2.802,"15min":2.802},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.395,"5min":2.512,"15min":2.512},"last":1.1240000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.75900000000000001},"max":{"1min":1.5629999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":1.925},{"interface":"front","average":{"1min":1.202,"5min":1.3680000000000001,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5549999999999999,"5min":3.286,"15min":3.286},"last":1.5349999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.363,"15min":1.3660000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.577,"5min":2.823,"15min":2.823},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.4410000000000001,"15min":1.415},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.772}]},{"osd":4,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.3400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.74,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.675},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4790000000000001,"15min":1.4119999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8049999999999999,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.6060000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3999999999999999,"15min":1.3740000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7609999999999999,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8999999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.8740000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.6299999999999999,"15min":1.5760000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.992,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":2.1110000000000002},{"interface":"front","average":{"1min":1.474,"5min":1.5189999999999999,"15min":1.532},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9670000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.8029999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1175552,"data_stored":561746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1100000000000001,"15min":1.1120000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5409999999999999,"5min":1.861,"15min":1.986},"last":1.05},{"interface":"front","average":{"1min":1.127,"5min":1.1060000000000001,"15min":1.129},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.60599999999999998},"max":{"1min":1.6599999999999999,"5min":1.665,"15min":2.234},"last":0.76900000000000002}]},{"osd":1,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1439999999999999,"15min":1.167},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6100000000000001,"5min":1.7090000000000001,"15min":2.1200000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.1830000000000001,"15min":1.216},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7609999999999999,"15min":1.905},"last":0.90000000000000002}]},{"osd":2,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5760000000000001,"5min":1.7150000000000001,"15min":2.375},"last":1.256},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3129999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.577,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":1.6419999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.321,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.72,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.768},{"interface":"front","average":{"1min":1.298,"5min":1.375,"15min":1.3999999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.732,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":1.6919999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.272,"15min":1.284},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.6699999999999999,"5min":2.194,"15min":2.4889999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.335,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9119999999999999,"5min":1.927,"15min":2.5169999999999999},"last":1.431}]},{"osd":6,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.302},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5229999999999999,"15min":1.5349999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.758,"5min":2.1059999999999999,"15min":2.613},"last":1.7350000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.478,"15min":1.5049999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.806,"5min":2.032,"15min":2.605},"last":1.599},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.522,"15min":1.5109999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8280000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.8240000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575051,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58700,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313140,"statfs":{"total":96636764160,"available":96576655360,"internally_reserved":0,"allocated":1175552,"data_stored":561726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.623,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.486,"5min":1.7050000000000001,"15min":1.982},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.23},"min":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.704,"5min":1.972,"15min":2.1099999999999999},"last":1.1339999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.753,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7490000000000001,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":1.534},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.206,"15min":1.24},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.478,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.423}]},{"osd":2,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2250000000000001,"15min":1.226},"min":{"1min":0.73099999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.752,"5min":2.169,"15min":2.169},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3799999999999999,"15min":1.3740000000000001},"min":{"1min":0.83399999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.7759999999999998},"last":1.3740000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2669999999999999,"15min":1.3100000000000001},"min":{"1min":0.77100000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6819999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77800000000000002},"max":{"1min":1.8440000000000001,"5min":1.952,"15min":2.085},"last":1.734}]},{"osd":5,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.361,"15min":1.3520000000000001},"min":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.627},{"interface":"front","average":{"1min":1.329,"5min":1.3080000000000001,"15min":1.343},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":2.153},"last":1.6910000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3779999999999999,"15min":1.389},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9139999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.484},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.504,"15min":1.526},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0960000000000001,"5min":2.214,"15min":2.6539999999999999},"last":1.2929999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4730000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.2389999999999999},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.5469999999999999},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.5660000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40660,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331180,"statfs":{"total":96636764160,"available":96595128320,"internally_reserved":0,"allocated":1179648,"data_stored":561768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.92800000000000005},"min":{"1min":0.47999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0960000000000001,"15min":1.127},"min":{"1min":0.66600000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.403,"5min":1.706,"15min":1.8799999999999999},"last":0.69999999999999996}]},{"osd":1,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":0.90900000000000003},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.33600000000000002},"max":{"1min":1.4410000000000001,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0580000000000001,"15min":1.1060000000000001},"min":{"1min":0.52500000000000002,"5min":0.438,"15min":0.373},"max":{"1min":1.5349999999999999,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":1.1040000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3169999999999999,"15min":1.363},"min":{"1min":0.59599999999999997,"5min":0.57899999999999996,"15min":0.56599999999999995},"max":{"1min":1.7889999999999999,"5min":1.966,"15min":2.0609999999999999},"last":2.024},{"interface":"front","average":{"1min":1.242,"5min":1.228,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.843,"5min":2.0960000000000001,"15min":2.282},"last":1.609}]},{"osd":4,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.353,"15min":1.361},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77300000000000002},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.3939999999999999,"15min":1.4099999999999999},"min":{"1min":0.97699999999999998,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.6919999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3560000000000001,"15min":1.397},"min":{"1min":0.92700000000000005,"5min":0.78400000000000003,"15min":0.77500000000000002},"max":{"1min":1.825,"5min":2.516,"15min":3.0190000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":0.91800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.798,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5489999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4450000000000001,"15min":1.468},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5169999999999999,"15min":1.5389999999999999},"min":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":3.3690000000000002,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.9099999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4810000000000001,"15min":1.5089999999999999},"min":{"1min":1.0349999999999999,"5min":0.85199999999999998,"15min":0.745},"max":{"1min":2.1019999999999999,"5min":2.871,"15min":2.875},"last":1.25},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5409999999999999,"15min":1.552},"min":{"1min":0.98499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.996,"5min":2.907,"15min":2.907},"last":1.341}]}]},{"osd":2,"up_from":13,"seq":55834575051,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40660,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331180,"statfs":{"total":96636764160,"available":96595128320,"internally_reserved":0,"allocated":1183744,"data_stored":563268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.706,"5min":1.706,"15min":1.849},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":1.044},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4990000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.878}]},{"osd":1,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.98199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0419999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.165,"15min":1.175},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.74,"5min":2.0390000000000001,"15min":2.069},"last":1.212}]},{"osd":3,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1539999999999999,"15min":1.23},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6930000000000001,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":1.28},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.242,"15min":1.3149999999999999},"min":{"1min":0.58699999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.52,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1}]},{"osd":4,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.361,"15min":1.4199999999999999},"min":{"1min":0.75,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":2.3159999999999998},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.327,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.038}]},{"osd":5,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.363,"15min":1.399},"min":{"1min":0.73999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.472,"15min":1.53},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7849999999999999,"5min":2.2730000000000001,"15min":2.4689999999999999},"last":1.2629999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.395,"15min":1.4490000000000001},"min":{"1min":0.82399999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.958,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.474,"15min":1.526},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.1629999999999998},"last":1.391}]},{"osd":7,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.0129999999999999,"5min":2.109,"15min":2.109},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5329999999999999,"15min":1.5920000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.482}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58716,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313124,"statfs":{"total":96636764160,"available":96576638976,"internally_reserved":0,"allocated":1191936,"data_stored":563884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.90500000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.708,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.996,"15min":1.0669999999999999},"min":{"1min":0.51500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.5549999999999999,"5min":1.728,"15min":2.0819999999999999},"last":0.73499999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":0.96199999999999997},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.758},"last":0.503},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0780000000000001,"15min":1.165},"min":{"1min":0.59899999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.556,"5min":2.0009999999999999,"15min":2.0150000000000001},"last":0.755}]},{"osd":3,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1919999999999999,"15min":1.2829999999999999},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.724,"5min":2.1549999999999998,"15min":2.198},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.099,"15min":1.1839999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.421,"5min":1.802,"15min":2.004},"last":1.1639999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.361,"15min":1.4059999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.575,"5min":1.9470000000000001,"15min":2.3050000000000002},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2609999999999999,"15min":1.3080000000000001},"min":{"1min":0.83699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.819,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.80900000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0430000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.373,"15min":1.458},"min":{"1min":0.83999999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6100000000000001,"5min":2.2970000000000002,"15min":2.302},"last":1.248}]},{"osd":6,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.403,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6619999999999999,"5min":2.1880000000000002,"15min":2.4169999999999998},"last":1.359},{"interface":"front","average":{"1min":1.399,"5min":1.421,"15min":1.47},"min":{"1min":1.02,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0870000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.2769999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.482,"15min":1.5609999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.827,"5min":2.2290000000000001,"15min":2.411},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.482,"15min":1.575},"min":{"1min":0.81899999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.734,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":1.2230000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1753088,"data_stored":1145680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90700000000000003,"15min":0.89200000000000002},"min":{"1min":0.55600000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.355,"5min":3.355,"15min":3.355},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.0620000000000001,"15min":1.0980000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46999999999999997},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.745}]},{"osd":2,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.59099999999999997,"5min":0.443,"15min":0.443},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1830000000000001,"15min":1.2090000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.42699999999999999},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.278},"min":{"1min":0.69799999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":0.627},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.304,"15min":1.355},"min":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":2.4790000000000001,"5min":2.593,"15min":2.593},"last":1.319}]},{"osd":4,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.302,"15min":1.2909999999999999},"min":{"1min":0.873,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.03},{"interface":"front","average":{"1min":1.456,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.77800000000000002},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.96499999999999997,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4870000000000001,"15min":1.524},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.306,"5min":3.306,"15min":3.306},"last":1.244}]},{"osd":6,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.55,"15min":1.5900000000000001},"min":{"1min":0.92500000000000004,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.391},{"interface":"front","average":{"1min":1.411,"5min":1.3999999999999999,"15min":1.405},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":1.0660000000000001,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":3.698,"5min":3.698,"15min":3.698},"last":1.1040000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:57:10.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:57:10.658 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T16:57:10.657+0000 7f6732e5d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:57:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.889+0000 7fab8699a640 1 Processor -- start 2023-10-14T16:57:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.889+0000 7fab8699a640 1 -- start start 2023-10-14T16:57:10.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab8699a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab8006c040 0x7fab8006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab8699a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8015f4a0 con 0x7fab800ad970 2023-10-14T16:57:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab8699a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab8006c960 con 0x7fab800cda80 2023-10-14T16:57:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab8699a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab8015b9d0 con 0x7fab8006c040 2023-10-14T16:57:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab85197640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab800ad970 0x7fab8015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:56364/0 (socket says 172.21.15.29:56364) 2023-10-14T16:57:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab85197640 1 -- 172.21.15.29:0/1781111928 learned_addr learned my addr 172.21.15.29:0/1781111928 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.891+0000 7fab85998640 1 --2- 172.21.15.29:0/1781111928 >> v2:172.21.15.29:3300/0 conn(0x7fab8006c040 0x7fab8006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.892+0000 7fab84996640 1 -- 172.21.15.29:0/1781111928 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926883147 0 0) 0x7fab8015f4a0 con 0x7fab800ad970 2023-10-14T16:57:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.892+0000 7fab84996640 1 -- 172.21.15.29:0/1781111928 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab6c003520 con 0x7fab800ad970 2023-10-14T16:57:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.892+0000 7fab85998640 1 -- 172.21.15.29:0/1781111928 >> v1:172.21.15.186:6789/0 conn(0x7fab800cda80 legacy=0x7fab8015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.892+0000 7fab85998640 1 -- 172.21.15.29:0/1781111928 >> v1:172.21.15.29:6789/0 conn(0x7fab800ad970 legacy=0x7fab8015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.892+0000 7fab85998640 1 -- 172.21.15.29:0/1781111928 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab8015bcb0 con 0x7fab8006c040 2023-10-14T16:57:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.893+0000 7fab85998640 1 --2- 172.21.15.29:0/1781111928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab8006c040 0x7fab8006c420 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0097a0 tx=0x7fab7c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee447bcdeb53438 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab84996640 1 -- 172.21.15.29:0/1781111928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c031830 con 0x7fab8006c040 2023-10-14T16:57:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab84996640 1 -- 172.21.15.29:0/1781111928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7c031990 con 0x7fab8006c040 2023-10-14T16:57:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab84996640 1 -- 172.21.15.29:0/1781111928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7c031d20 con 0x7fab8006c040 2023-10-14T16:57:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab8699a640 1 -- 172.21.15.29:0/1781111928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab8006c040 msgr2=0x7fab8006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab8699a640 1 --2- 172.21.15.29:0/1781111928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab8006c040 0x7fab8006c420 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fab7c0097a0 tx=0x7fab7c02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab8699a640 1 -- 172.21.15.29:0/1781111928 shutdown_connections 2023-10-14T16:57:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab8699a640 1 --2- 172.21.15.29:0/1781111928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab8006c040 0x7fab8006c420 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.894+0000 7fab8699a640 1 -- 172.21.15.29:0/1781111928 >> 172.21.15.29:0/1781111928 conn(0x7fab800ce6a0 msgr2=0x7fab800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:10.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.895+0000 7fab8699a640 1 -- 172.21.15.29:0/1781111928 shutdown_connections 2023-10-14T16:57:10.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.895+0000 7fab8699a640 1 -- 172.21.15.29:0/1781111928 wait complete. 2023-10-14T16:57:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.896+0000 7fab8699a640 1 Processor -- start 2023-10-14T16:57:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.896+0000 7fab8699a640 1 -- start start 2023-10-14T16:57:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab8006c040 0x7fab80147010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 0x7fab800b0550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801f2580 con 0x7fab800cda80 2023-10-14T16:57:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801f2770 con 0x7fab800ad970 2023-10-14T16:57:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.897+0000 7fab8699a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab801f2a10 con 0x7fab8006c040 2023-10-14T16:57:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33258/0 (socket says 172.21.15.29:33258) 2023-10-14T16:57:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 -- 172.21.15.29:0/1477951712 learned_addr learned my addr 172.21.15.29:0/1477951712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab85998640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab8006c040 0x7fab80147010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab85197640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 0x7fab800b0550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 -- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab8006c040 msgr2=0x7fab80147010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab8006c040 0x7fab80147010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 -- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 msgr2=0x7fab800b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 0x7fab800b0550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab86199640 1 -- 172.21.15.29:0/1477951712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c009400 con 0x7fab800cda80 2023-10-14T16:57:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.898+0000 7fab85197640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 0x7fab800b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab86199640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c780 tx=0x7fab7800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78004b60 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7800ce90 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7800a850 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab801f2cf0 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.899+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab801f32b0 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.900+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab8006b330 con 0x7fab800cda80 2023-10-14T16:57:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.901+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab7800a9b0 con 0x7fab800cda80 2023-10-14T16:57:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.903+0000 7fab767fc640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 0x7fab6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.903+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fab78089a40 con 0x7fab800cda80 2023-10-14T16:57:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.903+0000 7fab85998640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 0x7fab6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.904+0000 7fab85998640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 0x7fab6406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c03c040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:10.908+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab78054910 con 0x7fab800cda80 2023-10-14T16:57:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.101+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 --> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab800cc270 con 0x7fab6406af90 2023-10-14T16:57:11.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.124+0000 7fab767fc640 1 -- 172.21.15.29:0/1477951712 <== mgr.6765 v2:172.21.15.29:6832/3330033844 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286819 (secure 0 0 0) 0x7fab7c044400 con 0x7fab6406af90 2023-10-14T16:57:11.129 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:11.131 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:57:11.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.131+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 msgr2=0x7fab6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.131+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 0x7fab6406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fab7c036000 tx=0x7fab7c03c040 comp rx=0 tx=0).stop 2023-10-14T16:57:11.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.131+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 msgr2=0x7fab800c8c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.131+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fab7800c780 tx=0x7fab7800cc40 comp rx=0 tx=0).stop 2023-10-14T16:57:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 shutdown_connections 2023-10-14T16:57:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fab6406af90 0x7fab6406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab800cda80 0x7fab800c8c30 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab800ad970 0x7fab800b0550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 --2- 172.21.15.29:0/1477951712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab8006c040 0x7fab80147010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 >> 172.21.15.29:0/1477951712 conn(0x7fab800ce6a0 msgr2=0x7fab80159b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 shutdown_connections 2023-10-14T16:57:11.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.132+0000 7fab8699a640 1 -- 172.21.15.29:0/1477951712 wait complete. 2023-10-14T16:57:11.152 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T16:57:10.977706+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":303,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":210,"ondisk_log_size":210,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":184,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":341704,"kb_used_data":10364,"kb_used_omap":12,"kb_used_meta":331187,"kb_avail":754633016,"statfs":{"total":773094113280,"available":772744208384,"internally_reserved":0,"allocated":10612736,"data_stored":5672460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":339136056},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.252759"},"pg_stats":[{"pgid":"11.3f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516735+0000","last_change":"2023-10-14T16:57:05.771528+0000","last_active":"2023-10-14T16:57:07.516735+0000","last_peered":"2023-10-14T16:57:07.516735+0000","last_clean":"2023-10-14T16:57:07.516735+0000","last_became_active":"2023-10-14T16:57:05.771118+0000","last_became_peered":"2023-10-14T16:57:05.771118+0000","last_unstale":"2023-10-14T16:57:07.516735+0000","last_undegraded":"2023-10-14T16:57:07.516735+0000","last_fullsized":"2023-10-14T16:57:07.516735+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:10.918706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.3e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764806+0000","last_change":"2023-10-14T16:57:05.756040+0000","last_active":"2023-10-14T16:57:06.764806+0000","last_peered":"2023-10-14T16:57:06.764806+0000","last_clean":"2023-10-14T16:57:06.764806+0000","last_became_active":"2023-10-14T16:57:05.755719+0000","last_became_peered":"2023-10-14T16:57:05.755719+0000","last_unstale":"2023-10-14T16:57:06.764806+0000","last_undegraded":"2023-10-14T16:57:06.764806+0000","last_fullsized":"2023-10-14T16:57:06.764806+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.3d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983603+0000","last_change":"2023-10-14T16:57:05.757206+0000","last_active":"2023-10-14T16:57:06.983603+0000","last_peered":"2023-10-14T16:57:06.983603+0000","last_clean":"2023-10-14T16:57:06.983603+0000","last_became_active":"2023-10-14T16:57:05.756917+0000","last_became_peered":"2023-10-14T16:57:05.756917+0000","last_unstale":"2023-10-14T16:57:06.983603+0000","last_undegraded":"2023-10-14T16:57:06.983603+0000","last_fullsized":"2023-10-14T16:57:06.983603+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:39.839365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.3c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092759+0000","last_change":"2023-10-14T16:57:05.770843+0000","last_active":"2023-10-14T16:57:07.092759+0000","last_peered":"2023-10-14T16:57:07.092759+0000","last_clean":"2023-10-14T16:57:07.092759+0000","last_became_active":"2023-10-14T16:57:05.770522+0000","last_became_peered":"2023-10-14T16:57:05.770522+0000","last_unstale":"2023-10-14T16:57:07.092759+0000","last_undegraded":"2023-10-14T16:57:07.092759+0000","last_fullsized":"2023-10-14T16:57:07.092759+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:12:22.036610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.3b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761123+0000","last_change":"2023-10-14T16:57:05.764891+0000","last_active":"2023-10-14T16:57:06.761123+0000","last_peered":"2023-10-14T16:57:06.761123+0000","last_clean":"2023-10-14T16:57:06.761123+0000","last_became_active":"2023-10-14T16:57:05.764554+0000","last_became_peered":"2023-10-14T16:57:05.764554+0000","last_unstale":"2023-10-14T16:57:06.761123+0000","last_undegraded":"2023-10-14T16:57:06.761123+0000","last_fullsized":"2023-10-14T16:57:06.761123+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:12.225048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763978+0000","last_change":"2023-10-14T16:57:05.757888+0000","last_active":"2023-10-14T16:57:06.763978+0000","last_peered":"2023-10-14T16:57:06.763978+0000","last_clean":"2023-10-14T16:57:06.763978+0000","last_became_active":"2023-10-14T16:57:05.757428+0000","last_became_peered":"2023-10-14T16:57:05.757428+0000","last_unstale":"2023-10-14T16:57:06.763978+0000","last_undegraded":"2023-10-14T16:57:06.763978+0000","last_fullsized":"2023-10-14T16:57:06.763978+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:59.165568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.39","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.045973+0000","last_change":"2023-10-14T16:57:05.757383+0000","last_active":"2023-10-14T16:57:07.045973+0000","last_peered":"2023-10-14T16:57:07.045973+0000","last_clean":"2023-10-14T16:57:07.045973+0000","last_became_active":"2023-10-14T16:57:05.756062+0000","last_became_peered":"2023-10-14T16:57:05.756062+0000","last_unstale":"2023-10-14T16:57:07.045973+0000","last_undegraded":"2023-10-14T16:57:07.045973+0000","last_fullsized":"2023-10-14T16:57:07.045973+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:43.315213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.38","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763762+0000","last_change":"2023-10-14T16:57:05.768104+0000","last_active":"2023-10-14T16:57:06.763762+0000","last_peered":"2023-10-14T16:57:06.763762+0000","last_clean":"2023-10-14T16:57:06.763762+0000","last_became_active":"2023-10-14T16:57:05.767713+0000","last_became_peered":"2023-10-14T16:57:05.767713+0000","last_unstale":"2023-10-14T16:57:06.763762+0000","last_undegraded":"2023-10-14T16:57:06.763762+0000","last_fullsized":"2023-10-14T16:57:06.763762+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.37","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762375+0000","last_change":"2023-10-14T16:57:05.769922+0000","last_active":"2023-10-14T16:57:06.762375+0000","last_peered":"2023-10-14T16:57:06.762375+0000","last_clean":"2023-10-14T16:57:06.762375+0000","last_became_active":"2023-10-14T16:57:05.769340+0000","last_became_peered":"2023-10-14T16:57:05.769340+0000","last_unstale":"2023-10-14T16:57:06.762375+0000","last_undegraded":"2023-10-14T16:57:06.762375+0000","last_fullsized":"2023-10-14T16:57:06.762375+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.36","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765403+0000","last_change":"2023-10-14T16:57:05.760925+0000","last_active":"2023-10-14T16:57:06.765403+0000","last_peered":"2023-10-14T16:57:06.765403+0000","last_clean":"2023-10-14T16:57:06.765403+0000","last_became_active":"2023-10-14T16:57:05.760618+0000","last_became_peered":"2023-10-14T16:57:05.760618+0000","last_unstale":"2023-10-14T16:57:06.765403+0000","last_undegraded":"2023-10-14T16:57:06.765403+0000","last_fullsized":"2023-10-14T16:57:06.765403+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:27.819255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.35","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516563+0000","last_change":"2023-10-14T16:57:05.766151+0000","last_active":"2023-10-14T16:57:07.516563+0000","last_peered":"2023-10-14T16:57:07.516563+0000","last_clean":"2023-10-14T16:57:07.516563+0000","last_became_active":"2023-10-14T16:57:05.765846+0000","last_became_peered":"2023-10-14T16:57:05.765846+0000","last_unstale":"2023-10-14T16:57:07.516563+0000","last_undegraded":"2023-10-14T16:57:07.516563+0000","last_fullsized":"2023-10-14T16:57:07.516563+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:01.120633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.34","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092955+0000","last_change":"2023-10-14T16:57:05.758969+0000","last_active":"2023-10-14T16:57:07.092955+0000","last_peered":"2023-10-14T16:57:07.092955+0000","last_clean":"2023-10-14T16:57:07.092955+0000","last_became_active":"2023-10-14T16:57:05.758562+0000","last_became_peered":"2023-10-14T16:57:05.758562+0000","last_unstale":"2023-10-14T16:57:07.092955+0000","last_undegraded":"2023-10-14T16:57:07.092955+0000","last_fullsized":"2023-10-14T16:57:07.092955+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:26:12.881364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.33","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516398+0000","last_change":"2023-10-14T16:57:05.769791+0000","last_active":"2023-10-14T16:57:07.516398+0000","last_peered":"2023-10-14T16:57:07.516398+0000","last_clean":"2023-10-14T16:57:07.516398+0000","last_became_active":"2023-10-14T16:57:05.768452+0000","last_became_peered":"2023-10-14T16:57:05.768452+0000","last_unstale":"2023-10-14T16:57:07.516398+0000","last_undegraded":"2023-10-14T16:57:07.516398+0000","last_fullsized":"2023-10-14T16:57:07.516398+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:07.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.32","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761146+0000","last_change":"2023-10-14T16:57:05.760655+0000","last_active":"2023-10-14T16:57:06.761146+0000","last_peered":"2023-10-14T16:57:06.761146+0000","last_clean":"2023-10-14T16:57:06.761146+0000","last_became_active":"2023-10-14T16:57:05.759465+0000","last_became_peered":"2023-10-14T16:57:05.759465+0000","last_unstale":"2023-10-14T16:57:06.761146+0000","last_undegraded":"2023-10-14T16:57:06.761146+0000","last_fullsized":"2023-10-14T16:57:06.761146+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:59.165568+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.31","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515609+0000","last_change":"2023-10-14T16:57:05.769285+0000","last_active":"2023-10-14T16:57:07.515609+0000","last_peered":"2023-10-14T16:57:07.515609+0000","last_clean":"2023-10-14T16:57:07.515609+0000","last_became_active":"2023-10-14T16:57:05.768628+0000","last_became_peered":"2023-10-14T16:57:05.768628+0000","last_unstale":"2023-10-14T16:57:07.515609+0000","last_undegraded":"2023-10-14T16:57:07.515609+0000","last_fullsized":"2023-10-14T16:57:07.515609+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.30","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762473+0000","last_change":"2023-10-14T16:57:05.769097+0000","last_active":"2023-10-14T16:57:06.762473+0000","last_peered":"2023-10-14T16:57:06.762473+0000","last_clean":"2023-10-14T16:57:06.762473+0000","last_became_active":"2023-10-14T16:57:05.768589+0000","last_became_peered":"2023-10-14T16:57:05.768589+0000","last_unstale":"2023-10-14T16:57:06.762473+0000","last_undegraded":"2023-10-14T16:57:06.762473+0000","last_fullsized":"2023-10-14T16:57:06.762473+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:07.910530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762519+0000","last_change":"2023-10-14T16:57:05.768904+0000","last_active":"2023-10-14T16:57:06.762519+0000","last_peered":"2023-10-14T16:57:06.762519+0000","last_clean":"2023-10-14T16:57:06.762519+0000","last_became_active":"2023-10-14T16:57:05.766522+0000","last_became_peered":"2023-10-14T16:57:05.766522+0000","last_unstale":"2023-10-14T16:57:06.762519+0000","last_undegraded":"2023-10-14T16:57:06.762519+0000","last_fullsized":"2023-10-14T16:57:06.762519+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:39.174579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.2e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092664+0000","last_change":"2023-10-14T16:57:05.772673+0000","last_active":"2023-10-14T16:57:07.092664+0000","last_peered":"2023-10-14T16:57:07.092664+0000","last_clean":"2023-10-14T16:57:07.092664+0000","last_became_active":"2023-10-14T16:57:05.772227+0000","last_became_peered":"2023-10-14T16:57:05.772227+0000","last_unstale":"2023-10-14T16:57:07.092664+0000","last_undegraded":"2023-10-14T16:57:07.092664+0000","last_fullsized":"2023-10-14T16:57:07.092664+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:44:06.517776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.2d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765125+0000","last_change":"2023-10-14T16:57:05.761360+0000","last_active":"2023-10-14T16:57:06.765125+0000","last_peered":"2023-10-14T16:57:06.765125+0000","last_clean":"2023-10-14T16:57:06.765125+0000","last_became_active":"2023-10-14T16:57:05.760997+0000","last_became_peered":"2023-10-14T16:57:05.760997+0000","last_unstale":"2023-10-14T16:57:06.765125+0000","last_undegraded":"2023-10-14T16:57:06.765125+0000","last_fullsized":"2023-10-14T16:57:06.765125+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.2c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516039+0000","last_change":"2023-10-14T16:57:05.764064+0000","last_active":"2023-10-14T16:57:07.516039+0000","last_peered":"2023-10-14T16:57:07.516039+0000","last_clean":"2023-10-14T16:57:07.516039+0000","last_became_active":"2023-10-14T16:57:05.763749+0000","last_became_peered":"2023-10-14T16:57:05.763749+0000","last_unstale":"2023-10-14T16:57:07.516039+0000","last_undegraded":"2023-10-14T16:57:07.516039+0000","last_fullsized":"2023-10-14T16:57:07.516039+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:37.591279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763846+0000","last_change":"2023-10-14T16:57:05.760146+0000","last_active":"2023-10-14T16:57:06.763846+0000","last_peered":"2023-10-14T16:57:06.763846+0000","last_clean":"2023-10-14T16:57:06.763846+0000","last_became_active":"2023-10-14T16:57:05.759870+0000","last_became_peered":"2023-10-14T16:57:05.759870+0000","last_unstale":"2023-10-14T16:57:06.763846+0000","last_undegraded":"2023-10-14T16:57:06.763846+0000","last_fullsized":"2023-10-14T16:57:06.763846+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:07.099457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.2a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762737+0000","last_change":"2023-10-14T16:57:05.754153+0000","last_active":"2023-10-14T16:57:06.762737+0000","last_peered":"2023-10-14T16:57:06.762737+0000","last_clean":"2023-10-14T16:57:06.762737+0000","last_became_active":"2023-10-14T16:57:05.753572+0000","last_became_peered":"2023-10-14T16:57:05.753572+0000","last_unstale":"2023-10-14T16:57:06.762737+0000","last_undegraded":"2023-10-14T16:57:06.762737+0000","last_fullsized":"2023-10-14T16:57:06.762737+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:22.350211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.29","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516145+0000","last_change":"2023-10-14T16:57:05.764057+0000","last_active":"2023-10-14T16:57:07.516145+0000","last_peered":"2023-10-14T16:57:07.516145+0000","last_clean":"2023-10-14T16:57:07.516145+0000","last_became_active":"2023-10-14T16:57:05.763739+0000","last_became_peered":"2023-10-14T16:57:05.763739+0000","last_unstale":"2023-10-14T16:57:07.516145+0000","last_undegraded":"2023-10-14T16:57:07.516145+0000","last_fullsized":"2023-10-14T16:57:07.516145+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:51:11.837087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.28","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515618+0000","last_change":"2023-10-14T16:57:05.773468+0000","last_active":"2023-10-14T16:57:07.515618+0000","last_peered":"2023-10-14T16:57:07.515618+0000","last_clean":"2023-10-14T16:57:07.515618+0000","last_became_active":"2023-10-14T16:57:05.773164+0000","last_became_peered":"2023-10-14T16:57:05.773164+0000","last_unstale":"2023-10-14T16:57:07.515618+0000","last_undegraded":"2023-10-14T16:57:07.515618+0000","last_fullsized":"2023-10-14T16:57:07.515618+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:59.950873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.27","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764868+0000","last_change":"2023-10-14T16:57:05.758752+0000","last_active":"2023-10-14T16:57:06.764868+0000","last_peered":"2023-10-14T16:57:06.764868+0000","last_clean":"2023-10-14T16:57:06.764868+0000","last_became_active":"2023-10-14T16:57:05.758376+0000","last_became_peered":"2023-10-14T16:57:05.758376+0000","last_unstale":"2023-10-14T16:57:06.764868+0000","last_undegraded":"2023-10-14T16:57:06.764868+0000","last_fullsized":"2023-10-14T16:57:06.764868+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:24.419019+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.26","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046516+0000","last_change":"2023-10-14T16:57:05.770509+0000","last_active":"2023-10-14T16:57:07.046516+0000","last_peered":"2023-10-14T16:57:07.046516+0000","last_clean":"2023-10-14T16:57:07.046516+0000","last_became_active":"2023-10-14T16:57:05.770220+0000","last_became_peered":"2023-10-14T16:57:05.770220+0000","last_unstale":"2023-10-14T16:57:07.046516+0000","last_undegraded":"2023-10-14T16:57:07.046516+0000","last_fullsized":"2023-10-14T16:57:07.046516+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:50.398894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.25","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983747+0000","last_change":"2023-10-14T16:57:05.768312+0000","last_active":"2023-10-14T16:57:06.983747+0000","last_peered":"2023-10-14T16:57:06.983747+0000","last_clean":"2023-10-14T16:57:06.983747+0000","last_became_active":"2023-10-14T16:57:05.768004+0000","last_became_peered":"2023-10-14T16:57:05.768004+0000","last_unstale":"2023-10-14T16:57:06.983747+0000","last_undegraded":"2023-10-14T16:57:06.983747+0000","last_fullsized":"2023-10-14T16:57:06.983747+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:59:45.459458+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.24","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763940+0000","last_change":"2023-10-14T16:57:05.752785+0000","last_active":"2023-10-14T16:57:06.763940+0000","last_peered":"2023-10-14T16:57:06.763940+0000","last_clean":"2023-10-14T16:57:06.763940+0000","last_became_active":"2023-10-14T16:57:05.752401+0000","last_became_peered":"2023-10-14T16:57:05.752401+0000","last_unstale":"2023-10-14T16:57:06.763940+0000","last_undegraded":"2023-10-14T16:57:06.763940+0000","last_fullsized":"2023-10-14T16:57:06.763940+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:44:46.584055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.23","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983306+0000","last_change":"2023-10-14T16:57:05.770924+0000","last_active":"2023-10-14T16:57:06.983306+0000","last_peered":"2023-10-14T16:57:06.983306+0000","last_clean":"2023-10-14T16:57:06.983306+0000","last_became_active":"2023-10-14T16:57:05.770622+0000","last_became_peered":"2023-10-14T16:57:05.770622+0000","last_unstale":"2023-10-14T16:57:06.983306+0000","last_undegraded":"2023-10-14T16:57:06.983306+0000","last_fullsized":"2023-10-14T16:57:06.983306+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.22","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046044+0000","last_change":"2023-10-14T16:57:05.773392+0000","last_active":"2023-10-14T16:57:07.046044+0000","last_peered":"2023-10-14T16:57:07.046044+0000","last_clean":"2023-10-14T16:57:07.046044+0000","last_became_active":"2023-10-14T16:57:05.773108+0000","last_became_peered":"2023-10-14T16:57:05.773108+0000","last_unstale":"2023-10-14T16:57:07.046044+0000","last_undegraded":"2023-10-14T16:57:07.046044+0000","last_fullsized":"2023-10-14T16:57:07.046044+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:10.337141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.21","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046408+0000","last_change":"2023-10-14T16:57:05.762887+0000","last_active":"2023-10-14T16:57:07.046408+0000","last_peered":"2023-10-14T16:57:07.046408+0000","last_clean":"2023-10-14T16:57:07.046408+0000","last_became_active":"2023-10-14T16:57:05.762588+0000","last_became_peered":"2023-10-14T16:57:05.762588+0000","last_unstale":"2023-10-14T16:57:07.046408+0000","last_undegraded":"2023-10-14T16:57:07.046408+0000","last_fullsized":"2023-10-14T16:57:07.046408+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:32.747286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.20","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764110+0000","last_change":"2023-10-14T16:57:05.768996+0000","last_active":"2023-10-14T16:57:06.764110+0000","last_peered":"2023-10-14T16:57:06.764110+0000","last_clean":"2023-10-14T16:57:06.764110+0000","last_became_active":"2023-10-14T16:57:05.768625+0000","last_became_peered":"2023-10-14T16:57:05.768625+0000","last_unstale":"2023-10-14T16:57:06.764110+0000","last_undegraded":"2023-10-14T16:57:06.764110+0000","last_fullsized":"2023-10-14T16:57:06.764110+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:54.341628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516555+0000","last_change":"2023-10-14T16:57:05.773933+0000","last_active":"2023-10-14T16:57:07.516555+0000","last_peered":"2023-10-14T16:57:07.516555+0000","last_clean":"2023-10-14T16:57:07.516555+0000","last_became_active":"2023-10-14T16:57:05.773475+0000","last_became_peered":"2023-10-14T16:57:05.773475+0000","last_unstale":"2023-10-14T16:57:07.516555+0000","last_undegraded":"2023-10-14T16:57:07.516555+0000","last_fullsized":"2023-10-14T16:57:07.516555+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:00.541027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1f","version":"95'4","reported_seq":16,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.517014+0000","last_change":"2023-10-14T16:57:04.758777+0000","last_active":"2023-10-14T16:57:07.517014+0000","last_peered":"2023-10-14T16:57:07.517014+0000","last_clean":"2023-10-14T16:57:07.517014+0000","last_became_active":"2023-10-14T16:57:04.758506+0000","last_became_peered":"2023-10-14T16:57:04.758506+0000","last_unstale":"2023-10-14T16:57:07.517014+0000","last_undegraded":"2023-10-14T16:57:07.517014+0000","last_fullsized":"2023-10-14T16:57:07.517014+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:19:14.220829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.1f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762757+0000","last_change":"2023-10-14T16:57:05.769089+0000","last_active":"2023-10-14T16:57:06.762757+0000","last_peered":"2023-10-14T16:57:06.762757+0000","last_clean":"2023-10-14T16:57:06.762757+0000","last_became_active":"2023-10-14T16:57:05.768620+0000","last_became_peered":"2023-10-14T16:57:05.768620+0000","last_unstale":"2023-10-14T16:57:06.762757+0000","last_undegraded":"2023-10-14T16:57:06.762757+0000","last_fullsized":"2023-10-14T16:57:06.762757+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:07.433300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.1e","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516401+0000","last_change":"2023-10-14T16:57:04.747213+0000","last_active":"2023-10-14T16:57:07.516401+0000","last_peered":"2023-10-14T16:57:07.516401+0000","last_clean":"2023-10-14T16:57:07.516401+0000","last_became_active":"2023-10-14T16:57:04.746927+0000","last_became_peered":"2023-10-14T16:57:04.746927+0000","last_unstale":"2023-10-14T16:57:07.516401+0000","last_undegraded":"2023-10-14T16:57:07.516401+0000","last_fullsized":"2023-10-14T16:57:07.516401+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:58.161053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.1c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763951+0000","last_change":"2023-10-14T16:57:05.770939+0000","last_active":"2023-10-14T16:57:06.763951+0000","last_peered":"2023-10-14T16:57:06.763951+0000","last_clean":"2023-10-14T16:57:06.763951+0000","last_became_active":"2023-10-14T16:57:05.768828+0000","last_became_peered":"2023-10-14T16:57:05.768828+0000","last_unstale":"2023-10-14T16:57:06.763951+0000","last_undegraded":"2023-10-14T16:57:06.763951+0000","last_fullsized":"2023-10-14T16:57:06.763951+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:12.225048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.1d","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764819+0000","last_change":"2023-10-14T16:57:04.758702+0000","last_active":"2023-10-14T16:57:06.764819+0000","last_peered":"2023-10-14T16:57:06.764819+0000","last_clean":"2023-10-14T16:57:06.764819+0000","last_became_active":"2023-10-14T16:57:04.758442+0000","last_became_peered":"2023-10-14T16:57:04.758442+0000","last_unstale":"2023-10-14T16:57:06.764819+0000","last_undegraded":"2023-10-14T16:57:06.764819+0000","last_fullsized":"2023-10-14T16:57:06.764819+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:46.405350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516708+0000","last_change":"2023-10-14T16:57:05.766409+0000","last_active":"2023-10-14T16:57:07.516708+0000","last_peered":"2023-10-14T16:57:07.516708+0000","last_clean":"2023-10-14T16:57:07.516708+0000","last_became_active":"2023-10-14T16:57:05.765856+0000","last_became_peered":"2023-10-14T16:57:05.765856+0000","last_unstale":"2023-10-14T16:57:07.516708+0000","last_undegraded":"2023-10-14T16:57:07.516708+0000","last_fullsized":"2023-10-14T16:57:07.516708+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:05.067018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.1c","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764563+0000","last_change":"2023-10-14T16:57:04.749625+0000","last_active":"2023-10-14T16:57:06.764563+0000","last_peered":"2023-10-14T16:57:06.764563+0000","last_clean":"2023-10-14T16:57:06.764563+0000","last_became_active":"2023-10-14T16:57:04.747285+0000","last_became_peered":"2023-10-14T16:57:04.747285+0000","last_unstale":"2023-10-14T16:57:06.764563+0000","last_undegraded":"2023-10-14T16:57:06.764563+0000","last_fullsized":"2023-10-14T16:57:06.764563+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:09.992738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.1a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046355+0000","last_change":"2023-10-14T16:57:05.765059+0000","last_active":"2023-10-14T16:57:07.046355+0000","last_peered":"2023-10-14T16:57:07.046355+0000","last_clean":"2023-10-14T16:57:07.046355+0000","last_became_active":"2023-10-14T16:57:05.764764+0000","last_became_peered":"2023-10-14T16:57:05.764764+0000","last_unstale":"2023-10-14T16:57:07.046355+0000","last_undegraded":"2023-10-14T16:57:07.046355+0000","last_fullsized":"2023-10-14T16:57:07.046355+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:14.104217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.1b","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983645+0000","last_change":"2023-10-14T16:57:04.754798+0000","last_active":"2023-10-14T16:57:06.983645+0000","last_peered":"2023-10-14T16:57:06.983645+0000","last_clean":"2023-10-14T16:57:06.983645+0000","last_became_active":"2023-10-14T16:57:04.754510+0000","last_became_peered":"2023-10-14T16:57:04.754510+0000","last_unstale":"2023-10-14T16:57:06.983645+0000","last_undegraded":"2023-10-14T16:57:06.983645+0000","last_fullsized":"2023-10-14T16:57:06.983645+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:04.062504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.1b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764237+0000","last_change":"2023-10-14T16:57:05.762673+0000","last_active":"2023-10-14T16:57:06.764237+0000","last_peered":"2023-10-14T16:57:06.764237+0000","last_clean":"2023-10-14T16:57:06.764237+0000","last_became_active":"2023-10-14T16:57:05.762355+0000","last_became_peered":"2023-10-14T16:57:05.762355+0000","last_unstale":"2023-10-14T16:57:06.764237+0000","last_undegraded":"2023-10-14T16:57:06.764237+0000","last_fullsized":"2023-10-14T16:57:06.764237+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:24.150960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.1a","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764269+0000","last_change":"2023-10-14T16:57:04.743873+0000","last_active":"2023-10-14T16:57:06.764269+0000","last_peered":"2023-10-14T16:57:06.764269+0000","last_clean":"2023-10-14T16:57:06.764269+0000","last_became_active":"2023-10-14T16:57:04.742913+0000","last_became_peered":"2023-10-14T16:57:04.742913+0000","last_unstale":"2023-10-14T16:57:06.764269+0000","last_undegraded":"2023-10-14T16:57:06.764269+0000","last_fullsized":"2023-10-14T16:57:06.764269+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:41.059851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.18","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.760758+0000","last_change":"2023-10-14T16:57:05.747650+0000","last_active":"2023-10-14T16:57:06.760758+0000","last_peered":"2023-10-14T16:57:06.760758+0000","last_clean":"2023-10-14T16:57:06.760758+0000","last_became_active":"2023-10-14T16:57:05.744605+0000","last_became_peered":"2023-10-14T16:57:05.744605+0000","last_unstale":"2023-10-14T16:57:06.760758+0000","last_undegraded":"2023-10-14T16:57:06.760758+0000","last_fullsized":"2023-10-14T16:57:06.760758+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:13.948304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.19","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092621+0000","last_change":"2023-10-14T16:57:04.754128+0000","last_active":"2023-10-14T16:57:07.092621+0000","last_peered":"2023-10-14T16:57:07.092621+0000","last_clean":"2023-10-14T16:57:07.092621+0000","last_became_active":"2023-10-14T16:57:04.753778+0000","last_became_peered":"2023-10-14T16:57:04.753778+0000","last_unstale":"2023-10-14T16:57:07.092621+0000","last_undegraded":"2023-10-14T16:57:07.092621+0000","last_fullsized":"2023-10-14T16:57:07.092621+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:38.170064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.19","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515849+0000","last_change":"2023-10-14T16:57:05.761399+0000","last_active":"2023-10-14T16:57:07.515849+0000","last_peered":"2023-10-14T16:57:07.515849+0000","last_clean":"2023-10-14T16:57:07.515849+0000","last_became_active":"2023-10-14T16:57:05.761074+0000","last_became_peered":"2023-10-14T16:57:05.761074+0000","last_unstale":"2023-10-14T16:57:07.515849+0000","last_undegraded":"2023-10-14T16:57:07.515849+0000","last_fullsized":"2023-10-14T16:57:07.515849+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.18","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.760854+0000","last_change":"2023-10-14T16:57:04.755846+0000","last_active":"2023-10-14T16:57:06.760854+0000","last_peered":"2023-10-14T16:57:06.760854+0000","last_clean":"2023-10-14T16:57:06.760854+0000","last_became_active":"2023-10-14T16:57:04.755364+0000","last_became_peered":"2023-10-14T16:57:04.755364+0000","last_unstale":"2023-10-14T16:57:06.760854+0000","last_undegraded":"2023-10-14T16:57:06.760854+0000","last_fullsized":"2023-10-14T16:57:06.760854+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:32.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.16","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046526+0000","last_change":"2023-10-14T16:57:05.765644+0000","last_active":"2023-10-14T16:57:07.046526+0000","last_peered":"2023-10-14T16:57:07.046526+0000","last_clean":"2023-10-14T16:57:07.046526+0000","last_became_active":"2023-10-14T16:57:05.765346+0000","last_became_peered":"2023-10-14T16:57:05.765346+0000","last_unstale":"2023-10-14T16:57:07.046526+0000","last_undegraded":"2023-10-14T16:57:07.046526+0000","last_fullsized":"2023-10-14T16:57:07.046526+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:35.129696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.17","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765049+0000","last_change":"2023-10-14T16:57:04.757810+0000","last_active":"2023-10-14T16:57:06.765049+0000","last_peered":"2023-10-14T16:57:06.765049+0000","last_clean":"2023-10-14T16:57:06.765049+0000","last_became_active":"2023-10-14T16:57:04.753731+0000","last_became_peered":"2023-10-14T16:57:04.753731+0000","last_unstale":"2023-10-14T16:57:06.765049+0000","last_undegraded":"2023-10-14T16:57:06.765049+0000","last_fullsized":"2023-10-14T16:57:06.765049+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:44:26.122699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.17","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516734+0000","last_change":"2023-10-14T16:57:05.774079+0000","last_active":"2023-10-14T16:57:07.516734+0000","last_peered":"2023-10-14T16:57:07.516734+0000","last_clean":"2023-10-14T16:57:07.516734+0000","last_became_active":"2023-10-14T16:57:05.773594+0000","last_became_peered":"2023-10-14T16:57:05.773594+0000","last_unstale":"2023-10-14T16:57:07.516734+0000","last_undegraded":"2023-10-14T16:57:07.516734+0000","last_fullsized":"2023-10-14T16:57:07.516734+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:53.913205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.16","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765423+0000","last_change":"2023-10-14T16:57:04.757011+0000","last_active":"2023-10-14T16:57:06.765423+0000","last_peered":"2023-10-14T16:57:06.765423+0000","last_clean":"2023-10-14T16:57:06.765423+0000","last_became_active":"2023-10-14T16:57:04.756701+0000","last_became_peered":"2023-10-14T16:57:04.756701+0000","last_unstale":"2023-10-14T16:57:06.765423+0000","last_undegraded":"2023-10-14T16:57:06.765423+0000","last_fullsized":"2023-10-14T16:57:06.765423+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:39:10.076101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.14","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764301+0000","last_change":"2023-10-14T16:57:05.769487+0000","last_active":"2023-10-14T16:57:06.764301+0000","last_peered":"2023-10-14T16:57:06.764301+0000","last_clean":"2023-10-14T16:57:06.764301+0000","last_became_active":"2023-10-14T16:57:05.767403+0000","last_became_peered":"2023-10-14T16:57:05.767403+0000","last_unstale":"2023-10-14T16:57:06.764301+0000","last_undegraded":"2023-10-14T16:57:06.764301+0000","last_fullsized":"2023-10-14T16:57:06.764301+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.15","version":"95'3","reported_seq":15,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983911+0000","last_change":"2023-10-14T16:57:04.757319+0000","last_active":"2023-10-14T16:57:06.983911+0000","last_peered":"2023-10-14T16:57:06.983911+0000","last_clean":"2023-10-14T16:57:06.983911+0000","last_became_active":"2023-10-14T16:57:04.757056+0000","last_became_peered":"2023-10-14T16:57:04.757056+0000","last_unstale":"2023-10-14T16:57:06.983911+0000","last_undegraded":"2023-10-14T16:57:06.983911+0000","last_fullsized":"2023-10-14T16:57:06.983911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:07.937751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"11.15","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761862+0000","last_change":"2023-10-14T16:57:05.756437+0000","last_active":"2023-10-14T16:57:06.761862+0000","last_peered":"2023-10-14T16:57:06.761862+0000","last_clean":"2023-10-14T16:57:06.761862+0000","last_became_active":"2023-10-14T16:57:05.756143+0000","last_became_peered":"2023-10-14T16:57:05.756143+0000","last_unstale":"2023-10-14T16:57:06.761862+0000","last_undegraded":"2023-10-14T16:57:06.761862+0000","last_fullsized":"2023-10-14T16:57:06.761862+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.14","version":"95'6","reported_seq":18,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.093126+0000","last_change":"2023-10-14T16:57:04.747004+0000","last_active":"2023-10-14T16:57:07.093126+0000","last_peered":"2023-10-14T16:57:07.093126+0000","last_clean":"2023-10-14T16:57:07.093126+0000","last_became_active":"2023-10-14T16:57:04.746736+0000","last_became_peered":"2023-10-14T16:57:04.746736+0000","last_unstale":"2023-10-14T16:57:07.093126+0000","last_undegraded":"2023-10-14T16:57:07.093126+0000","last_fullsized":"2023-10-14T16:57:07.093126+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:31.895160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.12","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515919+0000","last_change":"2023-10-14T16:57:05.763543+0000","last_active":"2023-10-14T16:57:07.515919+0000","last_peered":"2023-10-14T16:57:07.515919+0000","last_clean":"2023-10-14T16:57:07.515919+0000","last_became_active":"2023-10-14T16:57:05.761127+0000","last_became_peered":"2023-10-14T16:57:05.761127+0000","last_unstale":"2023-10-14T16:57:07.515919+0000","last_undegraded":"2023-10-14T16:57:07.515919+0000","last_fullsized":"2023-10-14T16:57:07.515919+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:08.942266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.13","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092710+0000","last_change":"2023-10-14T16:57:04.755308+0000","last_active":"2023-10-14T16:57:07.092710+0000","last_peered":"2023-10-14T16:57:07.092710+0000","last_clean":"2023-10-14T16:57:07.092710+0000","last_became_active":"2023-10-14T16:57:04.755063+0000","last_became_peered":"2023-10-14T16:57:04.755063+0000","last_unstale":"2023-10-14T16:57:07.092710+0000","last_undegraded":"2023-10-14T16:57:07.092710+0000","last_fullsized":"2023-10-14T16:57:07.092710+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:36.586765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.13","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046389+0000","last_change":"2023-10-14T16:57:05.760952+0000","last_active":"2023-10-14T16:57:07.046389+0000","last_peered":"2023-10-14T16:57:07.046389+0000","last_clean":"2023-10-14T16:57:07.046389+0000","last_became_active":"2023-10-14T16:57:05.760639+0000","last_became_peered":"2023-10-14T16:57:05.760639+0000","last_unstale":"2023-10-14T16:57:07.046389+0000","last_undegraded":"2023-10-14T16:57:07.046389+0000","last_fullsized":"2023-10-14T16:57:07.046389+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:32.765750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.12","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764534+0000","last_change":"2023-10-14T16:57:04.753205+0000","last_active":"2023-10-14T16:57:06.764534+0000","last_peered":"2023-10-14T16:57:06.764534+0000","last_clean":"2023-10-14T16:57:06.764534+0000","last_became_active":"2023-10-14T16:57:04.750580+0000","last_became_peered":"2023-10-14T16:57:04.750580+0000","last_unstale":"2023-10-14T16:57:06.764534+0000","last_undegraded":"2023-10-14T16:57:06.764534+0000","last_fullsized":"2023-10-14T16:57:06.764534+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:42.503829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.10","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515606+0000","last_change":"2023-10-14T16:57:05.771757+0000","last_active":"2023-10-14T16:57:07.515606+0000","last_peered":"2023-10-14T16:57:07.515606+0000","last_clean":"2023-10-14T16:57:07.515606+0000","last_became_active":"2023-10-14T16:57:05.770291+0000","last_became_peered":"2023-10-14T16:57:05.770291+0000","last_unstale":"2023-10-14T16:57:07.515606+0000","last_undegraded":"2023-10-14T16:57:07.515606+0000","last_fullsized":"2023-10-14T16:57:07.515606+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.11","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046170+0000","last_change":"2023-10-14T16:57:04.755533+0000","last_active":"2023-10-14T16:57:07.046170+0000","last_peered":"2023-10-14T16:57:07.046170+0000","last_clean":"2023-10-14T16:57:07.046170+0000","last_became_active":"2023-10-14T16:57:04.755238+0000","last_became_peered":"2023-10-14T16:57:04.755238+0000","last_unstale":"2023-10-14T16:57:07.046170+0000","last_undegraded":"2023-10-14T16:57:07.046170+0000","last_fullsized":"2023-10-14T16:57:07.046170+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:55.311294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.11","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761165+0000","last_change":"2023-10-14T16:57:05.763650+0000","last_active":"2023-10-14T16:57:06.761165+0000","last_peered":"2023-10-14T16:57:06.761165+0000","last_clean":"2023-10-14T16:57:06.761165+0000","last_became_active":"2023-10-14T16:57:05.763344+0000","last_became_peered":"2023-10-14T16:57:05.763344+0000","last_unstale":"2023-10-14T16:57:06.761165+0000","last_undegraded":"2023-10-14T16:57:06.761165+0000","last_fullsized":"2023-10-14T16:57:06.761165+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:35:54.341628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.10","version":"95'8","reported_seq":20,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765372+0000","last_change":"2023-10-14T16:57:04.757605+0000","last_active":"2023-10-14T16:57:06.765372+0000","last_peered":"2023-10-14T16:57:06.765372+0000","last_clean":"2023-10-14T16:57:06.765372+0000","last_became_active":"2023-10-14T16:57:04.757291+0000","last_became_peered":"2023-10-14T16:57:04.757291+0000","last_unstale":"2023-10-14T16:57:06.765372+0000","last_undegraded":"2023-10-14T16:57:06.765372+0000","last_fullsized":"2023-10-14T16:57:06.765372+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:23.406446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.e","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765625+0000","last_change":"2023-10-14T16:57:05.754950+0000","last_active":"2023-10-14T16:57:06.765625+0000","last_peered":"2023-10-14T16:57:06.765625+0000","last_clean":"2023-10-14T16:57:06.765625+0000","last_became_active":"2023-10-14T16:57:05.754596+0000","last_became_peered":"2023-10-14T16:57:05.754596+0000","last_unstale":"2023-10-14T16:57:06.765625+0000","last_undegraded":"2023-10-14T16:57:06.765625+0000","last_fullsized":"2023-10-14T16:57:06.765625+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:32.315257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"10.f","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765701+0000","last_change":"2023-10-14T16:57:04.748816+0000","last_active":"2023-10-14T16:57:06.765701+0000","last_peered":"2023-10-14T16:57:06.765701+0000","last_clean":"2023-10-14T16:57:06.765701+0000","last_became_active":"2023-10-14T16:57:04.748406+0000","last_became_peered":"2023-10-14T16:57:04.748406+0000","last_unstale":"2023-10-14T16:57:06.765701+0000","last_undegraded":"2023-10-14T16:57:06.765701+0000","last_fullsized":"2023-10-14T16:57:06.765701+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:52.333555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.f","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.517092+0000","last_change":"2023-10-14T16:57:05.765528+0000","last_active":"2023-10-14T16:57:07.517092+0000","last_peered":"2023-10-14T16:57:07.517092+0000","last_clean":"2023-10-14T16:57:07.517092+0000","last_became_active":"2023-10-14T16:57:05.765233+0000","last_became_peered":"2023-10-14T16:57:05.765233+0000","last_unstale":"2023-10-14T16:57:07.517092+0000","last_undegraded":"2023-10-14T16:57:07.517092+0000","last_fullsized":"2023-10-14T16:57:07.517092+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:26:34.523846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.e","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765734+0000","last_change":"2023-10-14T16:57:04.758087+0000","last_active":"2023-10-14T16:57:06.765734+0000","last_peered":"2023-10-14T16:57:06.765734+0000","last_clean":"2023-10-14T16:57:06.765734+0000","last_became_active":"2023-10-14T16:57:04.757802+0000","last_became_peered":"2023-10-14T16:57:04.757802+0000","last_unstale":"2023-10-14T16:57:06.765734+0000","last_undegraded":"2023-10-14T16:57:06.765734+0000","last_fullsized":"2023-10-14T16:57:06.765734+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:53.163070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.c","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.093039+0000","last_change":"2023-10-14T16:57:05.758955+0000","last_active":"2023-10-14T16:57:07.093039+0000","last_peered":"2023-10-14T16:57:07.093039+0000","last_clean":"2023-10-14T16:57:07.093039+0000","last_became_active":"2023-10-14T16:57:05.758592+0000","last_became_peered":"2023-10-14T16:57:05.758592+0000","last_unstale":"2023-10-14T16:57:07.093039+0000","last_undegraded":"2023-10-14T16:57:07.093039+0000","last_fullsized":"2023-10-14T16:57:07.093039+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:44:50.497921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.d","version":"95'1","reported_seq":13,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765099+0000","last_change":"2023-10-14T16:57:04.755600+0000","last_active":"2023-10-14T16:57:06.765099+0000","last_peered":"2023-10-14T16:57:06.765099+0000","last_clean":"2023-10-14T16:57:06.765099+0000","last_became_active":"2023-10-14T16:57:04.755074+0000","last_became_peered":"2023-10-14T16:57:04.755074+0000","last_unstale":"2023-10-14T16:57:06.765099+0000","last_undegraded":"2023-10-14T16:57:06.765099+0000","last_fullsized":"2023-10-14T16:57:06.765099+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:32.173864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.d","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761890+0000","last_change":"2023-10-14T16:57:05.746927+0000","last_active":"2023-10-14T16:57:06.761890+0000","last_peered":"2023-10-14T16:57:06.761890+0000","last_clean":"2023-10-14T16:57:06.761890+0000","last_became_active":"2023-10-14T16:57:05.746202+0000","last_became_peered":"2023-10-14T16:57:05.746202+0000","last_unstale":"2023-10-14T16:57:06.761890+0000","last_undegraded":"2023-10-14T16:57:06.761890+0000","last_fullsized":"2023-10-14T16:57:06.761890+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:31.277373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.c","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516217+0000","last_change":"2023-10-14T16:57:04.760857+0000","last_active":"2023-10-14T16:57:07.516217+0000","last_peered":"2023-10-14T16:57:07.516217+0000","last_clean":"2023-10-14T16:57:07.516217+0000","last_became_active":"2023-10-14T16:57:04.760569+0000","last_became_peered":"2023-10-14T16:57:04.760569+0000","last_unstale":"2023-10-14T16:57:07.516217+0000","last_undegraded":"2023-10-14T16:57:07.516217+0000","last_fullsized":"2023-10-14T16:57:07.516217+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:59:58.160172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"11.b","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.984170+0000","last_change":"2023-10-14T16:57:05.763910+0000","last_active":"2023-10-14T16:57:06.984170+0000","last_peered":"2023-10-14T16:57:06.984170+0000","last_clean":"2023-10-14T16:57:06.984170+0000","last_became_active":"2023-10-14T16:57:05.763549+0000","last_became_peered":"2023-10-14T16:57:05.763549+0000","last_unstale":"2023-10-14T16:57:06.984170+0000","last_undegraded":"2023-10-14T16:57:06.984170+0000","last_fullsized":"2023-10-14T16:57:06.984170+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:41.170694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.a","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764780+0000","last_change":"2023-10-14T16:57:04.744062+0000","last_active":"2023-10-14T16:57:06.764780+0000","last_peered":"2023-10-14T16:57:06.764780+0000","last_clean":"2023-10-14T16:57:06.764780+0000","last_became_active":"2023-10-14T16:57:04.742678+0000","last_became_peered":"2023-10-14T16:57:04.742678+0000","last_unstale":"2023-10-14T16:57:06.764780+0000","last_undegraded":"2023-10-14T16:57:06.764780+0000","last_fullsized":"2023-10-14T16:57:06.764780+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:52.356756+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.8","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983641+0000","last_change":"2023-10-14T16:57:05.773880+0000","last_active":"2023-10-14T16:57:06.983641+0000","last_peered":"2023-10-14T16:57:06.983641+0000","last_clean":"2023-10-14T16:57:06.983641+0000","last_became_active":"2023-10-14T16:57:05.773559+0000","last_became_peered":"2023-10-14T16:57:05.773559+0000","last_unstale":"2023-10-14T16:57:06.983641+0000","last_undegraded":"2023-10-14T16:57:06.983641+0000","last_fullsized":"2023-10-14T16:57:06.983641+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:13.121991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.9","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046441+0000","last_change":"2023-10-14T16:57:04.755513+0000","last_active":"2023-10-14T16:57:07.046441+0000","last_peered":"2023-10-14T16:57:07.046441+0000","last_clean":"2023-10-14T16:57:07.046441+0000","last_became_active":"2023-10-14T16:57:04.755205+0000","last_became_peered":"2023-10-14T16:57:04.755205+0000","last_unstale":"2023-10-14T16:57:07.046441+0000","last_undegraded":"2023-10-14T16:57:07.046441+0000","last_fullsized":"2023-10-14T16:57:07.046441+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:52.996079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.9","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983631+0000","last_change":"2023-10-14T16:57:05.773951+0000","last_active":"2023-10-14T16:57:06.983631+0000","last_peered":"2023-10-14T16:57:06.983631+0000","last_clean":"2023-10-14T16:57:06.983631+0000","last_became_active":"2023-10-14T16:57:05.772474+0000","last_became_peered":"2023-10-14T16:57:05.772474+0000","last_unstale":"2023-10-14T16:57:06.983631+0000","last_undegraded":"2023-10-14T16:57:06.983631+0000","last_fullsized":"2023-10-14T16:57:06.983631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.637099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.8","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.515427+0000","last_change":"2023-10-14T16:57:04.747342+0000","last_active":"2023-10-14T16:57:07.515427+0000","last_peered":"2023-10-14T16:57:07.515427+0000","last_clean":"2023-10-14T16:57:07.515427+0000","last_became_active":"2023-10-14T16:57:04.747073+0000","last_became_peered":"2023-10-14T16:57:04.747073+0000","last_unstale":"2023-10-14T16:57:07.515427+0000","last_undegraded":"2023-10-14T16:57:07.515427+0000","last_fullsized":"2023-10-14T16:57:07.515427+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:47:12.943790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.6","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516296+0000","last_change":"2023-10-14T16:57:05.768476+0000","last_active":"2023-10-14T16:57:07.516296+0000","last_peered":"2023-10-14T16:57:07.516296+0000","last_clean":"2023-10-14T16:57:07.516296+0000","last_became_active":"2023-10-14T16:57:05.767714+0000","last_became_peered":"2023-10-14T16:57:05.767714+0000","last_unstale":"2023-10-14T16:57:07.516296+0000","last_undegraded":"2023-10-14T16:57:07.516296+0000","last_fullsized":"2023-10-14T16:57:07.516296+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:39.174579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"10.7","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.765808+0000","last_change":"2023-10-14T16:57:04.748379+0000","last_active":"2023-10-14T16:57:06.765808+0000","last_peered":"2023-10-14T16:57:06.765808+0000","last_clean":"2023-10-14T16:57:06.765808+0000","last_became_active":"2023-10-14T16:57:04.748092+0000","last_became_peered":"2023-10-14T16:57:04.748092+0000","last_unstale":"2023-10-14T16:57:06.765808+0000","last_undegraded":"2023-10-14T16:57:06.765808+0000","last_fullsized":"2023-10-14T16:57:06.765808+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:11.290225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.7","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046406+0000","last_change":"2023-10-14T16:57:05.759025+0000","last_active":"2023-10-14T16:57:07.046406+0000","last_peered":"2023-10-14T16:57:07.046406+0000","last_clean":"2023-10-14T16:57:07.046406+0000","last_became_active":"2023-10-14T16:57:05.758647+0000","last_became_peered":"2023-10-14T16:57:05.758647+0000","last_unstale":"2023-10-14T16:57:07.046406+0000","last_undegraded":"2023-10-14T16:57:07.046406+0000","last_fullsized":"2023-10-14T16:57:07.046406+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:17.297679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"10.6","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.046787+0000","last_change":"2023-10-14T16:57:04.754959+0000","last_active":"2023-10-14T16:57:07.046787+0000","last_peered":"2023-10-14T16:57:07.046787+0000","last_clean":"2023-10-14T16:57:07.046787+0000","last_became_active":"2023-10-14T16:57:04.754656+0000","last_became_peered":"2023-10-14T16:57:04.754656+0000","last_unstale":"2023-10-14T16:57:07.046787+0000","last_undegraded":"2023-10-14T16:57:07.046787+0000","last_fullsized":"2023-10-14T16:57:07.046787+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:06:27.317672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"11.4","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516160+0000","last_change":"2023-10-14T16:57:05.773575+0000","last_active":"2023-10-14T16:57:07.516160+0000","last_peered":"2023-10-14T16:57:07.516160+0000","last_clean":"2023-10-14T16:57:07.516160+0000","last_became_active":"2023-10-14T16:57:05.773222+0000","last_became_peered":"2023-10-14T16:57:05.773222+0000","last_unstale":"2023-10-14T16:57:07.516160+0000","last_undegraded":"2023-10-14T16:57:07.516160+0000","last_fullsized":"2023-10-14T16:57:07.516160+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:40.239061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.5","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761996+0000","last_change":"2023-10-14T16:57:04.753889+0000","last_active":"2023-10-14T16:57:06.761996+0000","last_peered":"2023-10-14T16:57:06.761996+0000","last_clean":"2023-10-14T16:57:06.761996+0000","last_became_active":"2023-10-14T16:57:04.753487+0000","last_became_peered":"2023-10-14T16:57:04.753487+0000","last_unstale":"2023-10-14T16:57:06.761996+0000","last_undegraded":"2023-10-14T16:57:06.761996+0000","last_fullsized":"2023-10-14T16:57:06.761996+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:00.434810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.5","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516852+0000","last_change":"2023-10-14T16:57:05.769678+0000","last_active":"2023-10-14T16:57:07.516852+0000","last_peered":"2023-10-14T16:57:07.516852+0000","last_clean":"2023-10-14T16:57:07.516852+0000","last_became_active":"2023-10-14T16:57:05.769288+0000","last_became_peered":"2023-10-14T16:57:05.769288+0000","last_unstale":"2023-10-14T16:57:07.516852+0000","last_undegraded":"2023-10-14T16:57:07.516852+0000","last_fullsized":"2023-10-14T16:57:07.516852+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:03:21.942766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"10.4","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.516056+0000","last_change":"2023-10-14T16:57:04.750718+0000","last_active":"2023-10-14T16:57:07.516056+0000","last_peered":"2023-10-14T16:57:07.516056+0000","last_clean":"2023-10-14T16:57:07.516056+0000","last_became_active":"2023-10-14T16:57:04.750408+0000","last_became_peered":"2023-10-14T16:57:04.750408+0000","last_unstale":"2023-10-14T16:57:07.516056+0000","last_undegraded":"2023-10-14T16:57:07.516056+0000","last_fullsized":"2023-10-14T16:57:07.516056+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:18:11.220533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"11.2","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761478+0000","last_change":"2023-10-14T16:57:05.765342+0000","last_active":"2023-10-14T16:57:06.761478+0000","last_peered":"2023-10-14T16:57:06.761478+0000","last_clean":"2023-10-14T16:57:06.761478+0000","last_became_active":"2023-10-14T16:57:05.764526+0000","last_became_peered":"2023-10-14T16:57:05.764526+0000","last_unstale":"2023-10-14T16:57:06.761478+0000","last_undegraded":"2023-10-14T16:57:06.761478+0000","last_fullsized":"2023-10-14T16:57:06.761478+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:22.487882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"10.3","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.761367+0000","last_change":"2023-10-14T16:57:04.755618+0000","last_active":"2023-10-14T16:57:06.761367+0000","last_peered":"2023-10-14T16:57:06.761367+0000","last_clean":"2023-10-14T16:57:06.761367+0000","last_became_active":"2023-10-14T16:57:04.755334+0000","last_became_peered":"2023-10-14T16:57:04.755334+0000","last_unstale":"2023-10-14T16:57:06.761367+0000","last_undegraded":"2023-10-14T16:57:06.761367+0000","last_fullsized":"2023-10-14T16:57:06.761367+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:08:48.464845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"11.3","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764333+0000","last_change":"2023-10-14T16:57:05.765252+0000","last_active":"2023-10-14T16:57:06.764333+0000","last_peered":"2023-10-14T16:57:06.764333+0000","last_clean":"2023-10-14T16:57:06.764333+0000","last_became_active":"2023-10-14T16:57:05.761688+0000","last_became_peered":"2023-10-14T16:57:05.761688+0000","last_unstale":"2023-10-14T16:57:06.764333+0000","last_undegraded":"2023-10-14T16:57:06.764333+0000","last_fullsized":"2023-10-14T16:57:06.764333+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:06.482635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.2","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092452+0000","last_change":"2023-10-14T16:57:04.753189+0000","last_active":"2023-10-14T16:57:07.092452+0000","last_peered":"2023-10-14T16:57:07.092452+0000","last_clean":"2023-10-14T16:57:07.092452+0000","last_became_active":"2023-10-14T16:57:04.752822+0000","last_became_peered":"2023-10-14T16:57:04.752822+0000","last_unstale":"2023-10-14T16:57:07.092452+0000","last_undegraded":"2023-10-14T16:57:07.092452+0000","last_fullsized":"2023-10-14T16:57:07.092452+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:05.478120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"11.0","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:07.092873+0000","last_change":"2023-10-14T16:57:05.761047+0000","last_active":"2023-10-14T16:57:07.092873+0000","last_peered":"2023-10-14T16:57:07.092873+0000","last_clean":"2023-10-14T16:57:07.092873+0000","last_became_active":"2023-10-14T16:57:05.758357+0000","last_became_peered":"2023-10-14T16:57:05.758357+0000","last_unstale":"2023-10-14T16:57:07.092873+0000","last_undegraded":"2023-10-14T16:57:07.092873+0000","last_fullsized":"2023-10-14T16:57:07.092873+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:57:45.952268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"10.1","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.762363+0000","last_change":"2023-10-14T16:57:04.757128+0000","last_active":"2023-10-14T16:57:06.762363+0000","last_peered":"2023-10-14T16:57:06.762363+0000","last_clean":"2023-10-14T16:57:06.762363+0000","last_became_active":"2023-10-14T16:57:04.756835+0000","last_became_peered":"2023-10-14T16:57:04.756835+0000","last_unstale":"2023-10-14T16:57:06.762363+0000","last_undegraded":"2023-10-14T16:57:06.762363+0000","last_fullsized":"2023-10-14T16:57:06.762363+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:18.705559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"11.1","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.763108+0000","last_change":"2023-10-14T16:57:05.765424+0000","last_active":"2023-10-14T16:57:06.763108+0000","last_peered":"2023-10-14T16:57:06.763108+0000","last_clean":"2023-10-14T16:57:06.763108+0000","last_became_active":"2023-10-14T16:57:05.764897+0000","last_became_peered":"2023-10-14T16:57:05.764897+0000","last_unstale":"2023-10-14T16:57:06.763108+0000","last_undegraded":"2023-10-14T16:57:06.763108+0000","last_fullsized":"2023-10-14T16:57:06.763108+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:41.683610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"10.0","version":"0'0","reported_seq":12,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764385+0000","last_change":"2023-10-14T16:57:04.744497+0000","last_active":"2023-10-14T16:57:06.764385+0000","last_peered":"2023-10-14T16:57:06.764385+0000","last_clean":"2023-10-14T16:57:06.764385+0000","last_became_active":"2023-10-14T16:57:04.744208+0000","last_became_peered":"2023-10-14T16:57:04.744208+0000","last_unstale":"2023-10-14T16:57:06.764385+0000","last_undegraded":"2023-10-14T16:57:06.764385+0000","last_fullsized":"2023-10-14T16:57:06.764385+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:34.579935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"11.a","version":"0'0","reported_seq":11,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.983971+0000","last_change":"2023-10-14T16:57:05.773085+0000","last_active":"2023-10-14T16:57:06.983971+0000","last_peered":"2023-10-14T16:57:06.983971+0000","last_clean":"2023-10-14T16:57:06.983971+0000","last_became_active":"2023-10-14T16:57:05.772765+0000","last_became_peered":"2023-10-14T16:57:05.772765+0000","last_unstale":"2023-10-14T16:57:06.983971+0000","last_undegraded":"2023-10-14T16:57:06.983971+0000","last_fullsized":"2023-10-14T16:57:06.983971+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":94,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:04.717048+0000","last_clean_scrub_stamp":"2023-10-14T16:57:04.717048+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:22:58.024289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"10.b","version":"95'2","reported_seq":14,"reported_epoch":96,"state":"active+clean","last_fresh":"2023-10-14T16:57:06.764591+0000","last_change":"2023-10-14T16:57:04.744312+0000","last_active":"2023-10-14T16:57:06.764591+0000","last_peered":"2023-10-14T16:57:06.764591+0000","last_clean":"2023-10-14T16:57:06.764591+0000","last_became_active":"2023-10-14T16:57:04.743749+0000","last_became_peered":"2023-10-14T16:57:04.743749+0000","last_unstale":"2023-10-14T16:57:06.764591+0000","last_undegraded":"2023-10-14T16:57:06.764591+0000","last_fullsized":"2023-10-14T16:57:06.764591+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":93,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:03.712533+0000","last_clean_scrub_stamp":"2023-10-14T16:57:03.712533+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:21.298963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"84'172","reported_seq":433,"reported_epoch":95,"state":"active+clean","last_fresh":"2023-10-14T16:57:05.751859+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:57:05.751859+0000","last_peered":"2023-10-14T16:57:05.751859+0000","last_clean":"2023-10-14T16:57:05.751859+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:57:05.751859+0000","last_undegraded":"2023-10-14T16:57:05.751859+0000","last_fullsized":"2023-10-14T16:57:05.751859+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":11,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":10,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":596,"num_read_kb":1130,"num_write":259,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":172,"ondisk_log_size":172,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575051,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37076,"kb_used_data":1732,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334764,"statfs":{"total":96636764160,"available":96598798336,"internally_reserved":0,"allocated":1773568,"data_stored":1152640,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.169,"15min":1.1699999999999999},"min":{"1min":0.66000000000000003,"5min":0.56599999999999995,"15min":0.51500000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.018,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.73299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6279999999999999,"5min":1.8400000000000001,"15min":2.117},"last":0.79400000000000004}]},{"osd":1,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1890000000000001,"15min":1.2230000000000001},"min":{"1min":0.73799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3560000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.236,"15min":1.2569999999999999},"min":{"1min":0.66700000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.722,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.026}]},{"osd":2,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1910000000000001,"15min":1.224},"min":{"1min":0.623,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.214,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.78900000000000003,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.1589999999999998},"last":0.999}]},{"osd":3,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":1.369},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.996,"5min":1.996,"15min":2.4159999999999999},"last":1.04},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4259999999999999,"15min":1.446},"min":{"1min":0.85599999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0880000000000001},"last":1.1259999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.52600000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.881,"5min":2.105,"15min":2.1819999999999999},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.161}]},{"osd":5,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3540000000000001,"15min":1.385},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0390000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4450000000000001,"15min":1.4590000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.141}]},{"osd":6,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.532,"15min":1.542},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":1.077},{"interface":"front","average":{"1min":1.587,"5min":1.4790000000000001,"15min":1.4830000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.1100000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575050,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36500,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335340,"statfs":{"total":96636764160,"available":96599388160,"internally_reserved":0,"allocated":1179648,"data_stored":561748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1519999999999999,"15min":1.153},"min":{"1min":0.81799999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.524,"5min":2.137,"15min":2.137},"last":1.327},{"interface":"front","average":{"1min":1.141,"5min":1.222,"15min":1.206},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8100000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.93500000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1779999999999999,"15min":1.1850000000000001},"min":{"1min":0.77000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.593,"5min":2.802,"15min":2.802},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.395,"5min":2.512,"15min":2.512},"last":1.1240000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.75900000000000001},"max":{"1min":1.5629999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":1.925},{"interface":"front","average":{"1min":1.202,"5min":1.3680000000000001,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5549999999999999,"5min":3.286,"15min":3.286},"last":1.5349999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.363,"15min":1.3660000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.577,"5min":2.823,"15min":2.823},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.4410000000000001,"15min":1.415},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.772}]},{"osd":4,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.3400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.74,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.675},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4790000000000001,"15min":1.4119999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8049999999999999,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.6060000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3999999999999999,"15min":1.3740000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7609999999999999,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.7430000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8999999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.8740000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.6299999999999999,"15min":1.5760000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.992,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":2.1110000000000002},{"interface":"front","average":{"1min":1.474,"5min":1.5189999999999999,"15min":1.532},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9670000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.8029999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575051,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32332,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339508,"statfs":{"total":96636764160,"available":96603656192,"internally_reserved":0,"allocated":1175552,"data_stored":561746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1100000000000001,"15min":1.1120000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5409999999999999,"5min":1.861,"15min":1.986},"last":1.05},{"interface":"front","average":{"1min":1.127,"5min":1.1060000000000001,"15min":1.129},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.60599999999999998},"max":{"1min":1.6599999999999999,"5min":1.665,"15min":2.234},"last":0.76900000000000002}]},{"osd":1,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1439999999999999,"15min":1.167},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6100000000000001,"5min":1.7090000000000001,"15min":2.1200000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.1830000000000001,"15min":1.216},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7609999999999999,"15min":1.905},"last":0.90000000000000002}]},{"osd":2,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5760000000000001,"5min":1.7150000000000001,"15min":2.375},"last":1.256},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3129999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.577,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":1.6419999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.321,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.72,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.768},{"interface":"front","average":{"1min":1.298,"5min":1.375,"15min":1.3999999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.732,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":1.6919999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.272,"15min":1.284},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.6699999999999999,"5min":2.194,"15min":2.4889999999999999},"last":1.5129999999999999},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.335,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9119999999999999,"5min":1.927,"15min":2.5169999999999999},"last":1.431}]},{"osd":6,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.302},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5229999999999999,"15min":1.5349999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.758,"5min":2.1059999999999999,"15min":2.613},"last":1.7350000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.478,"15min":1.5049999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.806,"5min":2.032,"15min":2.605},"last":1.599},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.522,"15min":1.5109999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8280000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.8240000000000001}]}]},{"osd":4,"up_from":13,"seq":55834575051,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58700,"kb_used_data":1148,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313140,"statfs":{"total":96636764160,"available":96576655360,"internally_reserved":0,"allocated":1175552,"data_stored":561726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.623,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.486,"5min":1.7050000000000001,"15min":1.982},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.23},"min":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.704,"5min":1.972,"15min":2.1099999999999999},"last":1.1339999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.753,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7490000000000001,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":1.534},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.206,"15min":1.24},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.478,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":1.423}]},{"osd":2,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2250000000000001,"15min":1.226},"min":{"1min":0.73099999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.752,"5min":2.169,"15min":2.169},"last":1.3380000000000001},{"interface":"front","average":{"1min":1.319,"5min":1.3799999999999999,"15min":1.3740000000000001},"min":{"1min":0.83399999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.7759999999999998},"last":1.3740000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2669999999999999,"15min":1.3100000000000001},"min":{"1min":0.77100000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6819999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":1.5940000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77800000000000002},"max":{"1min":1.8440000000000001,"5min":1.952,"15min":2.085},"last":1.734}]},{"osd":5,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.361,"15min":1.3520000000000001},"min":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":1.627},{"interface":"front","average":{"1min":1.329,"5min":1.3080000000000001,"15min":1.343},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":2.153},"last":1.6910000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3779999999999999,"15min":1.389},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9139999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":1.484},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.504,"15min":1.526},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0960000000000001,"5min":2.214,"15min":2.6539999999999999},"last":1.2929999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4730000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.2389999999999999},"last":1.7649999999999999},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.5469999999999999},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.5660000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40660,"kb_used_data":1152,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331180,"statfs":{"total":96636764160,"available":96595128320,"internally_reserved":0,"allocated":1179648,"data_stored":561768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.91800000000000004,"15min":0.92800000000000005},"min":{"1min":0.47999999999999998,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0960000000000001,"15min":1.127},"min":{"1min":0.66600000000000004,"5min":0.442,"15min":0.442},"max":{"1min":1.403,"5min":1.706,"15min":1.8799999999999999},"last":0.69999999999999996}]},{"osd":1,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":0.81699999999999995,"5min":0.88200000000000001,"15min":0.90900000000000003},"min":{"1min":0.497,"5min":0.48199999999999998,"15min":0.33600000000000002},"max":{"1min":1.4410000000000001,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.0580000000000001,"15min":1.1060000000000001},"min":{"1min":0.52500000000000002,"5min":0.438,"15min":0.373},"max":{"1min":1.5349999999999999,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":1.1040000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3169999999999999,"15min":1.363},"min":{"1min":0.59599999999999997,"5min":0.57899999999999996,"15min":0.56599999999999995},"max":{"1min":1.7889999999999999,"5min":1.966,"15min":2.0609999999999999},"last":2.024},{"interface":"front","average":{"1min":1.242,"5min":1.228,"15min":1.276},"min":{"1min":0.73199999999999998,"5min":0.496,"15min":0.496},"max":{"1min":1.843,"5min":2.0960000000000001,"15min":2.282},"last":1.609}]},{"osd":4,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.353,"15min":1.361},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.77300000000000002},"max":{"1min":1.641,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.3939999999999999,"15min":1.4099999999999999},"min":{"1min":0.97699999999999998,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.8290000000000002,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.6919999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.3560000000000001,"15min":1.397},"min":{"1min":0.92700000000000005,"5min":0.78400000000000003,"15min":0.77500000000000002},"max":{"1min":1.825,"5min":2.516,"15min":3.0190000000000001},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.4279999999999999,"15min":1.4399999999999999},"min":{"1min":0.91800000000000004,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.798,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.5489999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.496,"5min":1.4450000000000001,"15min":1.468},"min":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.9710000000000001,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.8109999999999999},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.5169999999999999,"15min":1.5389999999999999},"min":{"1min":0.93899999999999995,"5min":0.91400000000000003,"15min":0.91400000000000003},"max":{"1min":3.3690000000000002,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.9099999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4430000000000001,"5min":1.4810000000000001,"15min":1.5089999999999999},"min":{"1min":1.0349999999999999,"5min":0.85199999999999998,"15min":0.745},"max":{"1min":2.1019999999999999,"5min":2.871,"15min":2.875},"last":1.25},{"interface":"front","average":{"1min":1.5069999999999999,"5min":1.5409999999999999,"15min":1.552},"min":{"1min":0.98499999999999999,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.996,"5min":2.907,"15min":2.907},"last":1.341}]}]},{"osd":2,"up_from":13,"seq":55834575051,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40660,"kb_used_data":1156,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331180,"statfs":{"total":96636764160,"available":96595128320,"internally_reserved":0,"allocated":1183744,"data_stored":563268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.706,"5min":1.706,"15min":1.849},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":1.044},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4990000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.878}]},{"osd":1,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.98199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0419999999999998},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.165,"15min":1.175},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.74,"5min":2.0390000000000001,"15min":2.069},"last":1.212}]},{"osd":3,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1539999999999999,"15min":1.23},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6930000000000001,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":1.28},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.242,"15min":1.3149999999999999},"min":{"1min":0.58699999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.52,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1}]},{"osd":4,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.361,"15min":1.4199999999999999},"min":{"1min":0.75,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":2.3159999999999998},"last":1.0820000000000001},{"interface":"front","average":{"1min":1.206,"5min":1.327,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.038}]},{"osd":5,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.363,"15min":1.399},"min":{"1min":0.73999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.1679999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.472,"15min":1.53},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7849999999999999,"5min":2.2730000000000001,"15min":2.4689999999999999},"last":1.2629999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.395,"15min":1.4490000000000001},"min":{"1min":0.82399999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.958,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.474,"15min":1.526},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.1629999999999998},"last":1.391}]},{"osd":7,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.0129999999999999,"5min":2.109,"15min":2.109},"last":1.2310000000000001},{"interface":"front","average":{"1min":1.423,"5min":1.5329999999999999,"15min":1.5920000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.482}]}]},{"osd":1,"up_from":13,"seq":55834575051,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58716,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94313124,"statfs":{"total":96636764160,"available":96576638976,"internally_reserved":0,"allocated":1191936,"data_stored":563884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.90500000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.708,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.996,"15min":1.0669999999999999},"min":{"1min":0.51500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.5549999999999999,"5min":1.728,"15min":2.0819999999999999},"last":0.73499999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":0.96199999999999997},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.758},"last":0.503},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0780000000000001,"15min":1.165},"min":{"1min":0.59899999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.556,"5min":2.0009999999999999,"15min":2.0150000000000001},"last":0.755}]},{"osd":3,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1919999999999999,"15min":1.2829999999999999},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.724,"5min":2.1549999999999998,"15min":2.198},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.099,"15min":1.1839999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.421,"5min":1.802,"15min":2.004},"last":1.1639999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.361,"15min":1.4059999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.575,"5min":1.9470000000000001,"15min":2.3050000000000002},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2609999999999999,"15min":1.3080000000000001},"min":{"1min":0.83699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.819,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.92000000000000004}]},{"osd":5,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.80900000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0430000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.1899999999999999},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.373,"15min":1.458},"min":{"1min":0.83999999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6100000000000001,"5min":2.2970000000000002,"15min":2.302},"last":1.248}]},{"osd":6,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.403,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6619999999999999,"5min":2.1880000000000002,"15min":2.4169999999999998},"last":1.359},{"interface":"front","average":{"1min":1.399,"5min":1.421,"15min":1.47},"min":{"1min":1.02,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0870000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.2769999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.482,"15min":1.5609999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.827,"5min":2.2290000000000001,"15min":2.411},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.482,"15min":1.575},"min":{"1min":0.81899999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.734,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":1.2230000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575050,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37060,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334780,"statfs":{"total":96636764160,"available":96598814720,"internally_reserved":0,"allocated":1753088,"data_stored":1145680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90700000000000003,"15min":0.89200000000000002},"min":{"1min":0.55600000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.355,"5min":3.355,"15min":3.355},"last":0.38800000000000001},{"interface":"front","average":{"1min":1.071,"5min":1.0620000000000001,"15min":1.0980000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46999999999999997},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.745}]},{"osd":2,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.59099999999999997,"5min":0.443,"15min":0.443},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.70499999999999996},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1830000000000001,"15min":1.2090000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.42699999999999999},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.278},"min":{"1min":0.69799999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":0.627},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.304,"15min":1.355},"min":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":2.4790000000000001,"5min":2.593,"15min":2.593},"last":1.319}]},{"osd":4,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.302,"15min":1.2909999999999999},"min":{"1min":0.873,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.03},{"interface":"front","average":{"1min":1.456,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.77800000000000002},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":0.91200000000000003}]},{"osd":5,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.96499999999999997,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":0.88700000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4870000000000001,"15min":1.524},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.306,"5min":3.306,"15min":3.306},"last":1.244}]},{"osd":6,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.55,"15min":1.5900000000000001},"min":{"1min":0.92500000000000004,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.391},{"interface":"front","average":{"1min":1.411,"5min":1.3999999999999999,"15min":1.405},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":0.95699999999999996}]},{"osd":7,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":0.98799999999999999},{"interface":"front","average":{"1min":1.506,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":1.0660000000000001,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":3.698,"5min":3.698,"15min":3.698},"last":1.1040000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":10,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":11,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:57:11.160 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:57:11.160 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:57:11.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.435+0000 7f5683ce0640 1 Processor -- start 2023-10-14T16:57:11.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.435+0000 7f5683ce0640 1 -- start start 2023-10-14T16:57:11.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.436+0000 7f5683ce0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f567c06b8c0 0x7f567c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.436+0000 7f5683ce0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0b0970 con 0x7f567c0b0e00 2023-10-14T16:57:11.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f5683ce0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0cfca0 con 0x7f567c160440 2023-10-14T16:57:11.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f5683ce0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c06b440 con 0x7f567c06b8c0 2023-10-14T16:57:11.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f568227d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f567c0b0e00 0x7f567c0b1250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41090/0 (socket says 172.21.15.29:41090) 2023-10-14T16:57:11.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f568227d640 1 -- 172.21.15.29:0/1678246696 learned_addr learned my addr 172.21.15.29:0/1678246696 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:11.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658900617 0 0) 0x7f567c0b0970 con 0x7f567c0b0e00 2023-10-14T16:57:11.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f568127b640 1 --2- 172.21.15.29:0/1678246696 >> v2:172.21.15.186:3300/0 conn(0x7f567c06b8c0 0x7f567c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.437+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5668003520 con 0x7f567c0b0e00 2023-10-14T16:57:11.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1895883586 0 0) 0x7f5668003520 con 0x7f567c0b0e00 2023-10-14T16:57:11.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 >> v2:172.21.15.186:3300/0 conn(0x7f567c06b8c0 msgr2=0x7f567c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:11.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 --2- 172.21.15.29:0/1678246696 >> v2:172.21.15.186:3300/0 conn(0x7f567c06b8c0 0x7f567c06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 >> v1:172.21.15.186:6789/0 conn(0x7f567c160440 legacy=0x7f567c0d1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0cff80 con 0x7f567c0b0e00 2023-10-14T16:57:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f56740019f0 con 0x7f567c0b0e00 2023-10-14T16:57:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 >> v1:172.21.15.29:6789/0 conn(0x7f567c0b0e00 legacy=0x7f567c0b1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56680039c0 0x7f5668003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f5668004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56680051d0 0x7f566800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c06b440 con 0x7f56680047c0 2023-10-14T16:57:11.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f566800c120 con 0x7f56680051d0 2023-10-14T16:57:11.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f566800c2e0 con 0x7f56680039c0 2023-10-14T16:57:11.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.438+0000 7f568127b640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f5668004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f5681a7c640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56680039c0 0x7f5668003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568227d640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56680051d0 0x7f566800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568127b640 1 -- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56680039c0 msgr2=0x7f5668003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568127b640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56680039c0 0x7f5668003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568127b640 1 -- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56680051d0 msgr2=0x7f566800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:11.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568127b640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56680051d0 0x7f566800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568127b640 1 -- 172.21.15.29:0/1678246696 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5668003520 con 0x7f56680047c0 2023-10-14T16:57:11.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f568227d640 1 -- 172.21.15.29:0/1678246696 reap_dead start 2023-10-14T16:57:11.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.439+0000 7f5681a7c640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56680039c0 0x7f5668003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.440+0000 7f568127b640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f5668004c40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f566c00ba30 tx=0x7f56680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f270a887bae4d28 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.440+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c032640 con 0x7f56680047c0 2023-10-14T16:57:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.440+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f566c0327a0 con 0x7f56680047c0 2023-10-14T16:57:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.440+0000 7f5680a7a640 1 -- 172.21.15.29:0/1678246696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f566c032b30 con 0x7f56680047c0 2023-10-14T16:57:11.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 -- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 msgr2=0x7f5668004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f5668004c40 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f566c00ba30 tx=0x7f56680057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 -- 172.21.15.29:0/1678246696 shutdown_connections 2023-10-14T16:57:11.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 --2- 172.21.15.29:0/1678246696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f5668004c40 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 -- 172.21.15.29:0/1678246696 >> 172.21.15.29:0/1678246696 conn(0x7f567c099ea0 msgr2=0x7f567c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:11.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 -- 172.21.15.29:0/1678246696 shutdown_connections 2023-10-14T16:57:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.441+0000 7f5683ce0640 1 -- 172.21.15.29:0/1678246696 wait complete. 2023-10-14T16:57:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.443+0000 7f5683ce0640 1 Processor -- start 2023-10-14T16:57:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.443+0000 7f5683ce0640 1 -- start start 2023-10-14T16:57:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c0b4120 0x7f567c0b5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c0b6490 0x7f567c1f5de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0b4b80 con 0x7f56680047c0 2023-10-14T16:57:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0c9260 con 0x7f567c0b6490 2023-10-14T16:57:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5683ce0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1529f0 con 0x7f567c0b4120 2023-10-14T16:57:11.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5681a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5681a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46686/0 (socket says 172.21.15.29:46686) 2023-10-14T16:57:11.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.444+0000 7f5681a7c640 1 -- 172.21.15.29:0/3101571276 learned_addr learned my addr 172.21.15.29:0/3101571276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:11.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f568127b640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c0b4120 0x7f567c0b5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 -- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c0b4120 msgr2=0x7f567c0b5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f568227d640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c0b6490 0x7f567c1f5de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c0b4120 0x7f567c0b5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 -- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c0b6490 msgr2=0x7f567c1f5de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c0b6490 0x7f567c1f5de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 -- 172.21.15.29:0/3101571276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0c8c80 con 0x7f56680047c0 2023-10-14T16:57:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.445+0000 7f5681a7c640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f567800d4a0 tx=0x7f567800d960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.446+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678005480 con 0x7f56680047c0 2023-10-14T16:57:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.446+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678006820 con 0x7f56680047c0 2023-10-14T16:57:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.446+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678006c10 con 0x7f56680047c0 2023-10-14T16:57:11.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.446+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c0b45c0 con 0x7f56680047c0 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.446+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c0c8ef0 con 0x7f56680047c0 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.447+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c04e160 con 0x7f56680047c0 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.448+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5678007030 con 0x7f56680047c0 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.449+0000 7f56727fc640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 0x7f566006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.449+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5678088c80 con 0x7f56680047c0 2023-10-14T16:57:11.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.450+0000 7f568127b640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 0x7f566006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.450+0000 7f568127b640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 0x7f566006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f566c00ba00 tx=0x7f566c03d040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.454+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5678053c20 con 0x7f56680047c0 2023-10-14T16:57:11.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.716+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f567c0ce1a0 con 0x7f56680047c0 2023-10-14T16:57:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.720+0000 7f56727fc640 1 -- 172.21.15.29:0/3101571276 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) v1 ==== 190+0+147 (secure 0 0 0) 0x7f5678019020 con 0x7f56680047c0 2023-10-14T16:57:11.721 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:57:11.722 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:57:11.722 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:57:11.722 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:57:11.722 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:57:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:57:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 msgr2=0x7f566006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 0x7f566006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f566c00ba00 tx=0x7f566c03d040 comp rx=0 tx=0).stop 2023-10-14T16:57:11.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 msgr2=0x7f567c1524b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f567800d4a0 tx=0x7f567800d960 comp rx=0 tx=0).stop 2023-10-14T16:57:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 shutdown_connections 2023-10-14T16:57:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f566006afc0 0x7f566006d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c0b6490 0x7f567c1f5de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c0b4120 0x7f567c0b5f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 --2- 172.21.15.29:0/3101571276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56680047c0 0x7f567c1524b0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.723+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 >> 172.21.15.29:0/3101571276 conn(0x7f567c099ea0 msgr2=0x7f567c0ae7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.724+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 shutdown_connections 2023-10-14T16:57:11.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.724+0000 7f5683ce0640 1 -- 172.21.15.29:0/3101571276 wait complete. 2023-10-14T16:57:11.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:11.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.962+0000 7f4243fff640 1 Processor -- start 2023-10-14T16:57:11.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.963+0000 7f4243fff640 1 -- start start 2023-10-14T16:57:11.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f4243fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f424406c040 0x7f424406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f4243fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42440cda80 0x7f424415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f4243fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f424406c960 con 0x7f42440ad970 2023-10-14T16:57:11.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f4243fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424415b9d0 con 0x7f42440cda80 2023-10-14T16:57:11.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f4243fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f424415bcb0 con 0x7f424406c040 2023-10-14T16:57:11.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f42437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42440cda80 0x7f424415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f42437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f42440cda80 0x7f424415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46690/0 (socket says 172.21.15.29:46690) 2023-10-14T16:57:11.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.964+0000 7f42437fe640 1 -- 172.21.15.29:0/1857150302 learned_addr learned my addr 172.21.15.29:0/1857150302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.965+0000 7f4242ffd640 1 --2- 172.21.15.29:0/1857150302 >> v2:172.21.15.186:3300/0 conn(0x7f424406c040 0x7f424406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.965+0000 7f42437fe640 1 -- 172.21.15.29:0/1857150302 >> v2:172.21.15.186:3300/0 conn(0x7f424406c040 msgr2=0x7f424406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.965+0000 7f42437fe640 1 --2- 172.21.15.29:0/1857150302 >> v2:172.21.15.186:3300/0 conn(0x7f424406c040 0x7f424406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.965+0000 7f42437fe640 1 -- 172.21.15.29:0/1857150302 >> v1:172.21.15.29:6789/0 conn(0x7f42440ad970 legacy=0x7f424415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.965+0000 7f42437fe640 1 -- 172.21.15.29:0/1857150302 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f424415bf90 con 0x7f42440cda80 2023-10-14T16:57:11.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.966+0000 7f42437fe640 1 --2- 172.21.15.29:0/1857150302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440cda80 0x7f424415ed90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f42440cf350 tx=0x7f423c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68408210a535bd73 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.966+0000 7f4241ffb640 1 -- 172.21.15.29:0/1857150302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c030860 con 0x7f42440cda80 2023-10-14T16:57:11.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.966+0000 7f4241ffb640 1 -- 172.21.15.29:0/1857150302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f423c0309c0 con 0x7f42440cda80 2023-10-14T16:57:11.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.966+0000 7f4241ffb640 1 -- 172.21.15.29:0/1857150302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423c030d50 con 0x7f42440cda80 2023-10-14T16:57:11.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 -- 172.21.15.29:0/1857150302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440cda80 msgr2=0x7f424415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 --2- 172.21.15.29:0/1857150302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440cda80 0x7f424415ed90 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f42440cf350 tx=0x7f423c02b9a0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 -- 172.21.15.29:0/1857150302 shutdown_connections 2023-10-14T16:57:11.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 --2- 172.21.15.29:0/1857150302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440cda80 0x7f424415ed90 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 --2- 172.21.15.29:0/1857150302 >> v2:172.21.15.186:3300/0 conn(0x7f424406c040 0x7f424406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 -- 172.21.15.29:0/1857150302 >> 172.21.15.29:0/1857150302 conn(0x7f42440ce6a0 msgr2=0x7f42440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:11.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 -- 172.21.15.29:0/1857150302 shutdown_connections 2023-10-14T16:57:11.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.967+0000 7f4243fff640 1 -- 172.21.15.29:0/1857150302 wait complete. 2023-10-14T16:57:11.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.969+0000 7f4243fff640 1 Processor -- start 2023-10-14T16:57:11.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.969+0000 7f4243fff640 1 -- start start 2023-10-14T16:57:11.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f424406c040 0x7f4244156ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42440cda80 0x7f42440c7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440af7c0 con 0x7f42440ad970 2023-10-14T16:57:11.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42440aebb0 con 0x7f42440cda80 2023-10-14T16:57:11.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f4243fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42440aee90 con 0x7f424406c040 2023-10-14T16:57:11.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.970+0000 7f42427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46706/0 (socket says 172.21.15.29:46706) 2023-10-14T16:57:11.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 -- 172.21.15.29:0/1613318654 learned_addr learned my addr 172.21.15.29:0/1613318654 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:11.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f4242ffd640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f424406c040 0x7f4244156ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42437fe640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42440cda80 0x7f42440c7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 -- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f424406c040 msgr2=0x7f4244156ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f424406c040 0x7f4244156ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 -- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42440cda80 msgr2=0x7f42440c7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42440cda80 0x7f42440c7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:11.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 -- 172.21.15.29:0/1613318654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c0094d0 con 0x7f42440ad970 2023-10-14T16:57:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f42427fc640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f423800c7a0 tx=0x7f423800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.971+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4238003b70 con 0x7f42440ad970 2023-10-14T16:57:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.972+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4238003cd0 con 0x7f42440ad970 2023-10-14T16:57:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.972+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f423800aa90 con 0x7f42440ad970 2023-10-14T16:57:11.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.972+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42440af170 con 0x7f42440ad970 2023-10-14T16:57:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.972+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42441571e0 con 0x7f42440ad970 2023-10-14T16:57:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.973+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f424406b330 con 0x7f42440ad970 2023-10-14T16:57:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.974+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f423800abf0 con 0x7f42440ad970 2023-10-14T16:57:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.975+0000 7f422bfff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 0x7f423006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:11.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.975+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f42380895c0 con 0x7f42440ad970 2023-10-14T16:57:11.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.975+0000 7f4242ffd640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 0x7f423006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:11.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.976+0000 7f4242ffd640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 0x7f423006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:11.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:11.980+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42380545c0 con 0x7f42440ad970 2023-10-14T16:57:12.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.217+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f424404e9b0 con 0x7f42440ad970 2023-10-14T16:57:12.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.218+0000 7f422bfff640 1 -- 172.21.15.29:0/1613318654 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f4238057be0 con 0x7f42440ad970 2023-10-14T16:57:12.219 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:12.219 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:12.219 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 msgr2=0x7f423006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 0x7f423006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4234006fd0 tx=0x7f4234008040 comp rx=0 tx=0).stop 2023-10-14T16:57:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 msgr2=0x7f42440c78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f423800c7a0 tx=0x7f423800cc60 comp rx=0 tx=0).stop 2023-10-14T16:57:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 shutdown_connections 2023-10-14T16:57:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f423006af90 0x7f423006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f42440cda80 0x7f42440c7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f42440ad970 0x7f42440c78d0 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 --2- 172.21.15.29:0/1613318654 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f424406c040 0x7f4244156ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 >> 172.21.15.29:0/1613318654 conn(0x7f42440ce6a0 msgr2=0x7f42440b32a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 shutdown_connections 2023-10-14T16:57:12.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.221+0000 7f4243fff640 1 -- 172.21.15.29:0/1613318654 wait complete. 2023-10-14T16:57:12.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:57:05.213523+0000', 'modified': '2023-10-14T16:57:08.669005+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7002}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7002': {'gid': 7002, 'name': 'c', 'rank': 0, 'incarnation': 36, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/1398561557', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 1398561557}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 1398561557}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [11], 'metadata_pool': 10, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:57:12.646 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:57:12.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T16:57:12.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.896+0000 7f464fa2d640 1 Processor -- start 2023-10-14T16:57:12.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.897+0000 7f464fa2d640 1 -- start start 2023-10-14T16:57:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.898+0000 7f464fa2d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f46480ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.898+0000 7f464fa2d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480cb030 con 0x7f46480c9ca0 2023-10-14T16:57:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.898+0000 7f464fa2d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46480c91c0 con 0x7f464815ad70 2023-10-14T16:57:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.898+0000 7f464fa2d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46480c94a0 con 0x7f464806a4b0 2023-10-14T16:57:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f464dfca640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f46480c9ca0 0x7f46480ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41104/0 (socket says 172.21.15.29:41104) 2023-10-14T16:57:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f464dfca640 1 -- 172.21.15.29:0/3560148198 learned_addr learned my addr 172.21.15.29:0/3560148198 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f464cfc8640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f46480ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1075641535 0 0) 0x7f46480cb030 con 0x7f46480c9ca0 2023-10-14T16:57:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4634003520 con 0x7f46480c9ca0 2023-10-14T16:57:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4259746223 0 0) 0x7f4634003520 con 0x7f46480c9ca0 2023-10-14T16:57:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 msgr2=0x7f46480ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f46480ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 >> v1:172.21.15.186:6789/0 conn(0x7f464815ad70 legacy=0x7f4648069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480c9780 con 0x7f46480c9ca0 2023-10-14T16:57:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.899+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f46400019f0 con 0x7f46480c9ca0 2023-10-14T16:57:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 >> v1:172.21.15.29:6789/0 conn(0x7f46480c9ca0 legacy=0x7f46480ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f4634003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f4634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46480c94a0 con 0x7f46340039a0 2023-10-14T16:57:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f463400ba30 con 0x7f46340047c0 2023-10-14T16:57:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463400bbf0 con 0x7f464806a4b0 2023-10-14T16:57:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f464d7c9640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f4634003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f464dfca640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f4634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f464cfc8640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f464d7c9640 1 -- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 msgr2=0x7f4634005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.900+0000 7f464d7c9640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f464806a4b0 0x7f4634005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.901+0000 7f464d7c9640 1 -- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46340047c0 msgr2=0x7f4634004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.901+0000 7f464d7c9640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.901+0000 7f464d7c9640 1 -- 172.21.15.29:0/3560148198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4634003520 con 0x7f46340039a0 2023-10-14T16:57:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.901+0000 7f464cfc8640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46340047c0 0x7f4634004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.902+0000 7f464d7c9640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f4634003dc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4644001720 tx=0x7f46340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46944745c7567995 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.902+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440052d0 con 0x7f46340039a0 2023-10-14T16:57:12.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.902+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464402e050 con 0x7f46340039a0 2023-10-14T16:57:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.902+0000 7f463ffff640 1 -- 172.21.15.29:0/3560148198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440035f0 con 0x7f46340039a0 2023-10-14T16:57:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 -- 172.21.15.29:0/3560148198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 msgr2=0x7f4634003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 --2- 172.21.15.29:0/3560148198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f4634003dc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f4644001720 tx=0x7f46340057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464dfca640 1 -- 172.21.15.29:0/3560148198 reap_dead start 2023-10-14T16:57:12.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 -- 172.21.15.29:0/3560148198 shutdown_connections 2023-10-14T16:57:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 -- 172.21.15.29:0/3560148198 >> 172.21.15.29:0/3560148198 conn(0x7f46480ce9b0 msgr2=0x7f46480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 -- 172.21.15.29:0/3560148198 shutdown_connections 2023-10-14T16:57:12.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.903+0000 7f464fa2d640 1 -- 172.21.15.29:0/3560148198 wait complete. 2023-10-14T16:57:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.905+0000 7f464fa2d640 1 Processor -- start 2023-10-14T16:57:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.905+0000 7f464fa2d640 1 -- start start 2023-10-14T16:57:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46480aff90 0x7f46480a9840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 0x7f46480a9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46480ab870 con 0x7f46340039a0 2023-10-14T16:57:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46480abb10 con 0x7f46480b0410 2023-10-14T16:57:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464fa2d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46480b1210 con 0x7f46480aff90 2023-10-14T16:57:12.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464d7c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464d7c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46728/0 (socket says 172.21.15.29:46728) 2023-10-14T16:57:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464d7c9640 1 -- 172.21.15.29:0/310511289 learned_addr learned my addr 172.21.15.29:0/310511289 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464dfca640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 0x7f46480a9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.906+0000 7f464cfc8640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46480aff90 0x7f46480a9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 -- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46480aff90 msgr2=0x7f46480a9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46480aff90 0x7f46480a9840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 -- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 msgr2=0x7f46480a9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 0x7f46480a9d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 -- 172.21.15.29:0/310511289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46480b0970 con 0x7f46340039a0 2023-10-14T16:57:12.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464dfca640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 0x7f46480a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464d7c9640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4644034040 tx=0x7f4644001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f463dffb640 1 -- 172.21.15.29:0/310511289 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464403c630 con 0x7f46340039a0 2023-10-14T16:57:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f463dffb640 1 -- 172.21.15.29:0/310511289 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464403c7b0 con 0x7f46340039a0 2023-10-14T16:57:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f463dffb640 1 -- 172.21.15.29:0/310511289 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464403cb60 con 0x7f46340039a0 2023-10-14T16:57:12.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.907+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46480b0c50 con 0x7f46340039a0 2023-10-14T16:57:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.908+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46480adff0 con 0x7f46340039a0 2023-10-14T16:57:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.910+0000 7f463dffb640 1 -- 172.21.15.29:0/310511289 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f464403ccc0 con 0x7f46340039a0 2023-10-14T16:57:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.911+0000 7f463dffb640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 0x7f462c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.911+0000 7f463dffb640 1 -- 172.21.15.29:0/310511289 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f46440ab850 con 0x7f46340039a0 2023-10-14T16:57:12.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.911+0000 7f464cfc8640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 0x7f462c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.912+0000 7f464cfc8640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 0x7f462c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4638006ce0 tx=0x7f4638008040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.912+0000 7f46525c0740 1 Processor -- start 2023-10-14T16:57:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.912+0000 7f46525c0740 1 -- start start 2023-10-14T16:57:12.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f32436e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 0x564f32467250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f323577a0 con 0x564f32464360 2023-10-14T16:57:12.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f323a9740 con 0x564f324373c0 2023-10-14T16:57:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f323a6e30 con 0x564f32463f80 2023-10-14T16:57:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f464cfc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f464cfc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46736/0 (socket says 172.21.15.29:46736) 2023-10-14T16:57:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f464cfc8640 1 -- 172.21.15.29:0/1810976059 learned_addr learned my addr 172.21.15.29:0/1810976059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.913+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f32436e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464dfca640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 0x564f32467250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 -- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 msgr2=0x564f32436e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f32436e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 -- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 msgr2=0x564f32467250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 0x564f32467250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 -- 172.21.15.29:0/1810976059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f32366910 con 0x564f32464360 2023-10-14T16:57:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464dfca640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 0x564f32467250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:12.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4638011c00 tx=0x7f4638014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de5d02515f15d4a3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f462affd640 1 -- 172.21.15.29:0/1810976059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638015b40 con 0x564f32464360 2023-10-14T16:57:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f462affd640 1 -- 172.21.15.29:0/1810976059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4638015ca0 con 0x564f32464360 2023-10-14T16:57:12.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.914+0000 7f462affd640 1 -- 172.21.15.29:0/1810976059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4638021640 con 0x564f32464360 2023-10-14T16:57:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 -- 172.21.15.29:0/1810976059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 msgr2=0x564f32466d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f4638011c00 tx=0x7f4638014000 comp rx=0 tx=0).stop 2023-10-14T16:57:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 -- 172.21.15.29:0/1810976059 shutdown_connections 2023-10-14T16:57:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f324373c0 0x564f32467250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32466d10 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 --2- 172.21.15.29:0/1810976059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f32436e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 -- 172.21.15.29:0/1810976059 >> 172.21.15.29:0/1810976059 conn(0x564f32439190 msgr2=0x564f32481d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 -- 172.21.15.29:0/1810976059 shutdown_connections 2023-10-14T16:57:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.915+0000 7f46525c0740 1 -- 172.21.15.29:0/1810976059 wait complete. 2023-10-14T16:57:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.916+0000 7f46525c0740 1 -- start start 2023-10-14T16:57:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.916+0000 7f46525c0740 1 Processor -- start 2023-10-14T16:57:12.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32464360 0x564f32465d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f323a2720 con 0x564f32463f80 2023-10-14T16:57:12.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f32340970 con 0x564f3245bd30 2023-10-14T16:57:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.917+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f32340610 con 0x564f32464360 2023-10-14T16:57:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f46525c0740 10 client.0 _ll_register_callbacks cb 0x564f323629d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:57:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46740/0 (socket says 172.21.15.29:46740) 2023-10-14T16:57:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 -- 172.21.15.29:0/1667315217 learned_addr learned my addr 172.21.15.29:0/1667315217 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 -- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32464360 msgr2=0x564f32465d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32464360 0x564f32465d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 -- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 msgr2=0x564f32466280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 -- 172.21.15.29:0/1667315217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46380118b0 con 0x564f32463f80 2023-10-14T16:57:12.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4644001720 tx=0x7f464407aa90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99f173211040b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.918+0000 7f4623fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:57:12.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46440825d0 con 0x564f32463f80 2023-10-14T16:57:12.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f46440769e0 con 0x564f32463f80 2023-10-14T16:57:12.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f323c9160 con 0x564f32463f80 2023-10-14T16:57:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4644076ba0 con 0x564f32463f80 2023-10-14T16:57:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 10 client.7146 fetch_fsmap learned FSMap version 40 2023-10-14T16:57:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x564f323a0da0 con 0x564f32463f80 2023-10-14T16:57:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x564f323c9160 con 0x564f32463f80 2023-10-14T16:57:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f464403a040 con 0x564f32463f80 2023-10-14T16:57:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 10 client.7146 fetch_fsmap finished waiting for FSMap version 40 2023-10-14T16:57:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 10 client.7146 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:57:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 20 client.7146 populate_metadata read hostname 'smithi029' 2023-10-14T16:57:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f32495880 0x564f32495c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 4 client.7146 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:57:12.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.919+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 --> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564f32374cc0 con 0x564f32495880 2023-10-14T16:57:12.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.920+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f32495880 0x564f32495c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.920+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f32495880 0x564f32495c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be43704e81f9728b server_cookie=9fea07f07bb8cc98 in_seq=0 out_seq=0 2023-10-14T16:57:12.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.920+0000 7f4623fff640 10 client.7146 ms_handle_connect on v2:172.21.15.29:6836/1398561557 2023-10-14T16:57:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.921+0000 7f4623fff640 1 -- 172.21.15.29:0/1667315217 <== mds.0 v2:172.21.15.29:6836/1398561557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x564f32374cc0 con 0x564f32495880 2023-10-14T16:57:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.921+0000 7f4623fff640 10 client.7146 handle_command_reply: tid=0 2023-10-14T16:57:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.921+0000 7f46525c0740 1 client.7146 shutdown 2023-10-14T16:57:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.921+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 msgr2=0x564f324825e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.921+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f4644001720 tx=0x7f464407aa90 comp rx=0 tx=0).stop 2023-10-14T16:57:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 shutdown_connections 2023-10-14T16:57:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f32495880 0x564f32495c60 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32464360 0x564f32465d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 --2- 172.21.15.29:0/1667315217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32463f80 0x564f324825e0 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f4623fff640 0 client.7146 ms_handle_reset on v2:172.21.15.29:6836/1398561557 2023-10-14T16:57:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 >> 172.21.15.29:0/1667315217 conn(0x564f324373c0 msgr2=0x564f32485220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 shutdown_connections 2023-10-14T16:57:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 1 -- 172.21.15.29:0/1667315217 wait complete. 2023-10-14T16:57:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.922+0000 7f46525c0740 20 client.7146 trim_cache size 0 max 16384 2023-10-14T16:57:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.939+0000 7f46525c0740 1 Processor -- start 2023-10-14T16:57:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.939+0000 7f46525c0740 1 -- start start 2023-10-14T16:57:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f3245bd30 0x564f324362b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f32463f80 0x564f3243c040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f32106f20 con 0x564f32464360 2023-10-14T16:57:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f323485c0 con 0x564f32463f80 2023-10-14T16:57:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f3236f890 con 0x564f3245bd30 2023-10-14T16:57:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f464dfca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f464dfca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46750/0 (socket says 172.21.15.29:46750) 2023-10-14T16:57:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.940+0000 7f464dfca640 1 -- 172.21.15.29:0/1020378529 learned_addr learned my addr 172.21.15.29:0/1020378529 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 -- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f3245bd30 msgr2=0x564f324362b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f3245bd30 0x564f324362b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 -- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f32463f80 msgr2=0x564f3243c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f32463f80 0x564f3243c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 -- 172.21.15.29:0/1020378529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564f3233fb80 con 0x564f32464360 2023-10-14T16:57:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f464dfca640 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f46480c5a30 tx=0x7f4640007410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2c6380a5c2330e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f462b7fe640 1 -- 172.21.15.29:0/1020378529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f464000d070 con 0x564f32464360 2023-10-14T16:57:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f462b7fe640 1 -- 172.21.15.29:0/1020378529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640006e60 con 0x564f32464360 2023-10-14T16:57:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.941+0000 7f462b7fe640 1 -- 172.21.15.29:0/1020378529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640014630 con 0x564f32464360 2023-10-14T16:57:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 -- 172.21.15.29:0/1020378529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 msgr2=0x564f32485220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7f46480c5a30 tx=0x7f4640007410 comp rx=0 tx=0).stop 2023-10-14T16:57:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 -- 172.21.15.29:0/1020378529 shutdown_connections 2023-10-14T16:57:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32485220 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f32463f80 0x564f3243c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 --2- 172.21.15.29:0/1020378529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f3245bd30 0x564f324362b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 -- 172.21.15.29:0/1020378529 >> 172.21.15.29:0/1020378529 conn(0x564f324373c0 msgr2=0x564f3249bc90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 -- 172.21.15.29:0/1020378529 shutdown_connections 2023-10-14T16:57:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.942+0000 7f46525c0740 1 -- 172.21.15.29:0/1020378529 wait complete. 2023-10-14T16:57:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.943+0000 7f46525c0740 1 -- start start 2023-10-14T16:57:12.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.943+0000 7f46525c0740 1 Processor -- start 2023-10-14T16:57:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f324776a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f324661a0 con 0x564f32464360 2023-10-14T16:57:12.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x564f32436a40 con 0x564f3245bd30 2023-10-14T16:57:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x564f32478120 con 0x564f32463f80 2023-10-14T16:57:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f46525c0740 10 client.0 _ll_register_callbacks cb 0x7f4644007c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:57:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f464dfca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f464dfca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46754/0 (socket says 172.21.15.29:46754) 2023-10-14T16:57:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f464dfca640 1 -- 172.21.15.29:0/1309719856 learned_addr learned my addr 172.21.15.29:0/1309719856 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.944+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f324776a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 -- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 msgr2=0x564f324776a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f324776a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 -- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 msgr2=0x564f32466630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 -- 172.21.15.29:0/1309719856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46480c9780 con 0x564f32464360 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464cfc8640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f324776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f464dfca640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4640001580 tx=0x7f46400014d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cf1b4fe6d7ee92f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f4623fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:57:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4640026070 con 0x564f32464360 2023-10-14T16:57:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f4640001fd0 con 0x564f32464360 2023-10-14T16:57:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564f3245dfd0 con 0x564f32464360 2023-10-14T16:57:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=40) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4640004b60 con 0x564f32464360 2023-10-14T16:57:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f46525c0740 10 client.7152 fetch_fsmap learned FSMap version 40 2023-10-14T16:57:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.945+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=40,monmap=2+}) v3 -- 0x564f32365170 con 0x564f32464360 2023-10-14T16:57:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4640030020 con 0x564f32464360 2023-10-14T16:57:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 40) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4640029020 con 0x564f32464360 2023-10-14T16:57:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 10 client.7152 fetch_fsmap finished waiting for FSMap version 40 2023-10-14T16:57:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 10 client.7152 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:57:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 20 client.7152 populate_metadata read hostname 'smithi029' 2023-10-14T16:57:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f324945a0 0x564f324f9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 4 client.7152 mds_command: new command op to 7002 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:57:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 --> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] -- command(tid 0: {"prefix": "status"}) v1 -- 0x7f4638015ca0 con 0x564f324945a0 2023-10-14T16:57:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.946+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f324945a0 0x564f324f9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f464d7c9640 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f324945a0 0x564f324f9ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de946692476612a6 server_cookie=32f9ac537f5b4239 in_seq=0 out_seq=0 2023-10-14T16:57:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f4623fff640 10 client.7152 ms_handle_connect on v2:172.21.15.29:6836/1398561557 2023-10-14T16:57:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f4623fff640 1 -- 172.21.15.29:0/1309719856 <== mds.0 v2:172.21.15.29:6836/1398561557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+331 (crc 0 0 0) 0x7f4638015ca0 con 0x564f324945a0 2023-10-14T16:57:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f4623fff640 10 client.7152 handle_command_reply: tid=0 2023-10-14T16:57:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 client.7152 shutdown 2023-10-14T16:57:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 msgr2=0x564f32477be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f4640001580 tx=0x7f46400014d0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 shutdown_connections 2023-10-14T16:57:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:6836/1398561557,v1:172.21.15.29:6837/1398561557] conn(0x564f324945a0 0x564f324f9ba0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x564f32464360 0x564f32477be0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x564f32463f80 0x564f324776a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 --2- 172.21.15.29:0/1309719856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x564f3245bd30 0x564f32466630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f4623fff640 0 client.7152 ms_handle_reset on v2:172.21.15.29:6836/1398561557 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 >> 172.21.15.29:0/1309719856 conn(0x564f324373c0 msgr2=0x564f3243d000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 shutdown_connections 2023-10-14T16:57:12.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.947+0000 7f46525c0740 1 -- 172.21.15.29:0/1309719856 wait complete. 2023-10-14T16:57:12.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f46525c0740 20 client.7152 trim_cache size 0 max 16384 2023-10-14T16:57:12.966 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:57:12.966 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:57:12.967 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:57:12.967 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7002, 2023-10-14T16:57:12.967 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:57:12.967 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:57:12.967 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:57:12.968 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8000015850000004, 2023-10-14T16:57:12.968 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 40, 2023-10-14T16:57:12.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 96, 2023-10-14T16:57:12.968 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 96, 2023-10-14T16:57:12.968 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.054101001999999 2023-10-14T16:57:12.969 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:57:12.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 msgr2=0x7f462c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 0x7f462c06d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4638006ce0 tx=0x7f4638008040 comp rx=0 tx=0).stop 2023-10-14T16:57:12.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 msgr2=0x7f46481593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f4644034040 tx=0x7f4644001120 comp rx=0 tx=0).stop 2023-10-14T16:57:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 shutdown_connections 2023-10-14T16:57:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f462c06af90 0x7f462c06d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46480b0410 0x7f46480a9d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46480aff90 0x7f46480a9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 --2- 172.21.15.29:0/310511289 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46340039a0 0x7f46481593b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 >> 172.21.15.29:0/310511289 conn(0x7f46480ce9b0 msgr2=0x7f464815b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 shutdown_connections 2023-10-14T16:57:12.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:12.948+0000 7f464fa2d640 1 -- 172.21.15.29:0/310511289 wait complete. 2023-10-14T16:57:12.973 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:57:13.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.186+0000 7fb10bfff640 1 Processor -- start 2023-10-14T16:57:13.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.186+0000 7fb10bfff640 1 -- start start 2023-10-14T16:57:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb10c0cdff0 0x7fb10c0b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c0d1080 con 0x7fb10c15d130 2023-10-14T16:57:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb10c0d1360 con 0x7fb10c0d2120 2023-10-14T16:57:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10bfff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb10c15d510 con 0x7fb10c0cdff0 2023-10-14T16:57:13.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb10c15d130 0x7fb10c0cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41108/0 (socket says 172.21.15.29:41108) 2023-10-14T16:57:13.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.188+0000 7fb10affd640 1 -- 172.21.15.29:0/1192759399 learned_addr learned my addr 172.21.15.29:0/1192759399 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.189+0000 7fb10b7fe640 1 --2- 172.21.15.29:0/1192759399 >> v2:172.21.15.186:3300/0 conn(0x7fb10c0cdff0 0x7fb10c0b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.189+0000 7fb10a7fc640 1 -- 172.21.15.29:0/1192759399 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 136025207 0 0) 0x7fb10c0d1080 con 0x7fb10c15d130 2023-10-14T16:57:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.189+0000 7fb10a7fc640 1 -- 172.21.15.29:0/1192759399 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f8003520 con 0x7fb10c15d130 2023-10-14T16:57:13.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.190+0000 7fb10b7fe640 1 -- 172.21.15.29:0/1192759399 >> v1:172.21.15.186:6789/0 conn(0x7fb10c0d2120 legacy=0x7fb10c0b5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.190+0000 7fb10b7fe640 1 -- 172.21.15.29:0/1192759399 >> v1:172.21.15.29:6789/0 conn(0x7fb10c15d130 legacy=0x7fb10c0cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.190+0000 7fb10b7fe640 1 -- 172.21.15.29:0/1192759399 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb10c0d1640 con 0x7fb10c0cdff0 2023-10-14T16:57:13.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.191+0000 7fb10b7fe640 1 --2- 172.21.15.29:0/1192759399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c0cdff0 0x7fb10c0b63c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb104009d90 tx=0x7fb10402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85ace0e89b12515e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.195+0000 7fb10a7fc640 1 -- 172.21.15.29:0/1192759399 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb10402ccf0 con 0x7fb10c0cdff0 2023-10-14T16:57:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10a7fc640 1 -- 172.21.15.29:0/1192759399 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb10402ce50 con 0x7fb10c0cdff0 2023-10-14T16:57:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10a7fc640 1 -- 172.21.15.29:0/1192759399 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb104034660 con 0x7fb10c0cdff0 2023-10-14T16:57:13.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 -- 172.21.15.29:0/1192759399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c0cdff0 msgr2=0x7fb10c0b63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 --2- 172.21.15.29:0/1192759399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c0cdff0 0x7fb10c0b63c0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fb104009d90 tx=0x7fb10402bf40 comp rx=0 tx=0).stop 2023-10-14T16:57:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 -- 172.21.15.29:0/1192759399 shutdown_connections 2023-10-14T16:57:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 --2- 172.21.15.29:0/1192759399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c0cdff0 0x7fb10c0b63c0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 -- 172.21.15.29:0/1192759399 >> 172.21.15.29:0/1192759399 conn(0x7fb10c06bc60 msgr2=0x7fb10c06c070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.196+0000 7fb10bfff640 1 -- 172.21.15.29:0/1192759399 shutdown_connections 2023-10-14T16:57:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.197+0000 7fb10bfff640 1 -- 172.21.15.29:0/1192759399 wait complete. 2023-10-14T16:57:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.198+0000 7fb10bfff640 1 Processor -- start 2023-10-14T16:57:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.198+0000 7fb10bfff640 1 -- start start 2023-10-14T16:57:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c15d130 0x7fb10c0b2de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb10c0b4770 0x7fb10c150d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c1566a0 con 0x7fb10c0cdff0 2023-10-14T16:57:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0b3320 con 0x7fb10c15d130 2023-10-14T16:57:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.199+0000 7fb10bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb10c0b4f70 con 0x7fb10c0b4770 2023-10-14T16:57:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46768/0 (socket says 172.21.15.29:46768) 2023-10-14T16:57:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 -- 172.21.15.29:0/2347257860 learned_addr learned my addr 172.21.15.29:0/2347257860 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:13.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb103fff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c15d130 0x7fb10c0b2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 -- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb10c0b4770 msgr2=0x7fb10c150d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb10c0b4770 0x7fb10c150d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 -- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c15d130 msgr2=0x7fb10c0b2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c15d130 0x7fb10c0b2de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.200+0000 7fb10affd640 1 -- 172.21.15.29:0/2347257860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb104009a70 con 0x7fb10c0cdff0 2023-10-14T16:57:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb10affd640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c500 tx=0x7fb0fc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc005ce0 con 0x7fb10c0cdff0 2023-10-14T16:57:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0fc005e40 con 0x7fb10c0cdff0 2023-10-14T16:57:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc007330 con 0x7fb10c0cdff0 2023-10-14T16:57:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10c0b5250 con 0x7fb10c0cdff0 2023-10-14T16:57:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.201+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb10c151290 con 0x7fb10c0cdff0 2023-10-14T16:57:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.202+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb10c0b5440 con 0x7fb10c0cdff0 2023-10-14T16:57:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.203+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0fc007550 con 0x7fb10c0cdff0 2023-10-14T16:57:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.205+0000 7fb1037fe640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 0x7fb0ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.205+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb0fc088e30 con 0x7fb10c0cdff0 2023-10-14T16:57:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.205+0000 7fb103fff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 0x7fb0ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.206+0000 7fb103fff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 0x7fb0ec06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b7a20 tx=0x7fb0f4009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:13.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.209+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0fc053e50 con 0x7fb10c0cdff0 2023-10-14T16:57:13.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.450+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fb10c04e9b0 con 0x7fb10c0cdff0 2023-10-14T16:57:13.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.451+0000 7fb1037fe640 1 -- 172.21.15.29:0/2347257860 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v40) v1 ==== 53+0+73 (secure 0 0 0) 0x7fb0fc017020 con 0x7fb10c0cdff0 2023-10-14T16:57:13.452 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:57:13.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.453+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 msgr2=0x7fb0ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.453+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 0x7fb0ec06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb10c0b7a20 tx=0x7fb0f4009040 comp rx=0 tx=0).stop 2023-10-14T16:57:13.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.453+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 msgr2=0x7fb10c1e4820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:13.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.453+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb0fc00c500 tx=0x7fb0fc00c9c0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 shutdown_connections 2023-10-14T16:57:13.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fb0ec06af90 0x7fb0ec06d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb10c0b4770 0x7fb10c150d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb10c15d130 0x7fb10c0b2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 --2- 172.21.15.29:0/2347257860 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb10c0cdff0 0x7fb10c1e4820 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:13.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 >> 172.21.15.29:0/2347257860 conn(0x7fb10c06bc60 msgr2=0x7fb10c15c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:13.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 shutdown_connections 2023-10-14T16:57:13.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:13.454+0000 7fb10bfff640 1 -- 172.21.15.29:0/2347257860 wait complete. 2023-10-14T16:57:13.472 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:57:13.472 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:57:13.473 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:57:13.473 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:57:13.473 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:13.474 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:57:13.474 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:57:13.474 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:57:13.475 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:13.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.476 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:57:13.526 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:57:13.527 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:57:13.527 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:57:13.527 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.527 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:57:13.528 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.528 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:13.528 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:13.528 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:57:13.529 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:13.529 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:13.529 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:57:13.529 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:57:13.529 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:13.530 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:57:13.530 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:57:13.530 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41343sec preferred_lft 41343sec 2023-10-14T16:57:13.530 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:57:13.530 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.531 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:13.531 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:13.531 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:57:13.531 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:57:13.531 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:13.532 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:57:13.532 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.532 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:57:13.532 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.532 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:57:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:57:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:57:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.533 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:57:13.533 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:57:13.534 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:57:13.534 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:13.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.536 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:57:13.631 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:57:13.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.636 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:57:13.695 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:57:13.695 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:57:13.695 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:57:13.696 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:57:13.696 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:57:13.696 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.697 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:57:13.697 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:57:13.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:57:13.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:57:13.698 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:57:13.699 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:57:13.902 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.903 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:57:13.957 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:57:13.958 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:57:13.959 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:13.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:13.960 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:57:14.062 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:57:14.063 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:14.122 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:14.123 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:57:14.123 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:14.183 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:57:14.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:14.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:14.334 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:14.335 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:14.335 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:14.358 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:57:14.359 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:57:14.384 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:14.384 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:14.385 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:14.385 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:57:14.385 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:57:14.385 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:57:14.386 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:57:14.386 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:57:14.387 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:57:14.387 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:57:14.387 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:57:14.388 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:57:14.388 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:57:14.391 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:57:14.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:14.392 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:14.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:14.489 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:57:14.558 INFO:teuthology.orchestra.run.smithi029.stdout:client5875 2023-10-14T16:57:14.559 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:14.559 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:14.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:57:14.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.846+0000 7ff9d6651640 1 Processor -- start 2023-10-14T16:57:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.846+0000 7ff9d6651640 1 -- start start 2023-10-14T16:57:14.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.847+0000 7ff9d6651640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.847+0000 7ff9d6651640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 0x7ff9d006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.847+0000 7ff9d6651640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9d00cd1f0 con 0x7ff9d006c140 2023-10-14T16:57:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.847+0000 7ff9d6651640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9d015d890 con 0x7ff9d0159ab0 2023-10-14T16:57:14.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.847+0000 7ff9d6651640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00cd4d0 con 0x7ff9d0159e90 2023-10-14T16:57:14.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.848+0000 7ff9cffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.848+0000 7ff9cf7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 0x7ff9d006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.848+0000 7ff9cffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:53408/0 (socket says 172.21.15.29:53408) 2023-10-14T16:57:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.848+0000 7ff9cffff640 1 -- 172.21.15.29:0/3942824127 learned_addr learned my addr 172.21.15.29:0/3942824127 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:14.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.849+0000 7ff9cf7fe640 1 -- 172.21.15.29:0/3942824127 >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 msgr2=0x7ff9d00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.849+0000 7ff9cf7fe640 1 --2- 172.21.15.29:0/3942824127 >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.849+0000 7ff9cf7fe640 1 -- 172.21.15.29:0/3942824127 >> v1:172.21.15.186:6789/0 conn(0x7ff9d006c140 legacy=0x7ff9d015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.849+0000 7ff9cf7fe640 1 -- 172.21.15.29:0/3942824127 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9d00cd7b0 con 0x7ff9d0159e90 2023-10-14T16:57:14.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.849+0000 7ff9cffff640 1 --2- 172.21.15.29:0/3942824127 >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:57:14.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.850+0000 7ff9cf7fe640 1 --2- 172.21.15.29:0/3942824127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 0x7ff9d006fc50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80097d0 tx=0x7ff9b802b780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6030baaebd9eecda server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:14.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9ceffd640 1 -- 172.21.15.29:0/3942824127 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8030710 con 0x7ff9d0159e90 2023-10-14T16:57:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9ceffd640 1 -- 172.21.15.29:0/3942824127 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9b802e050 con 0x7ff9d0159e90 2023-10-14T16:57:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9ceffd640 1 -- 172.21.15.29:0/3942824127 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9b8030a90 con 0x7ff9d0159e90 2023-10-14T16:57:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 -- 172.21.15.29:0/3942824127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 msgr2=0x7ff9d006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3942824127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 0x7ff9d006fc50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7ff9b80097d0 tx=0x7ff9b802b780 comp rx=0 tx=0).stop 2023-10-14T16:57:14.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 -- 172.21.15.29:0/3942824127 shutdown_connections 2023-10-14T16:57:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3942824127 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159e90 0x7ff9d006fc50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3942824127 >> v2:172.21.15.186:3300/0 conn(0x7ff9d0159ab0 0x7ff9d00ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.851+0000 7ff9d6651640 1 -- 172.21.15.29:0/3942824127 >> 172.21.15.29:0/3942824127 conn(0x7ff9d00ce9b0 msgr2=0x7ff9d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:14.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.852+0000 7ff9d6651640 1 -- 172.21.15.29:0/3942824127 shutdown_connections 2023-10-14T16:57:14.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.852+0000 7ff9d6651640 1 -- 172.21.15.29:0/3942824127 wait complete. 2023-10-14T16:57:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.853+0000 7ff9d6651640 1 Processor -- start 2023-10-14T16:57:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.854+0000 7ff9d6651640 1 -- start start 2023-10-14T16:57:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159ab0 0x7ff9d00b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0159e90 0x7ff9d006b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00b2460 con 0x7ff9d006c140 2023-10-14T16:57:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d01e55d0 con 0x7ff9d0159e90 2023-10-14T16:57:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d6651640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9d00b18e0 con 0x7ff9d0159ab0 2023-10-14T16:57:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46770/0 (socket says 172.21.15.29:46770) 2023-10-14T16:57:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 -- 172.21.15.29:0/3378785586 learned_addr learned my addr 172.21.15.29:0/3378785586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 -- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159ab0 msgr2=0x7ff9d00b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159ab0 0x7ff9d00b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 -- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0159e90 msgr2=0x7ff9d006b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9d4bee640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0159e90 0x7ff9d006b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0159e90 0x7ff9d006b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.855+0000 7ff9cffff640 1 -- 172.21.15.29:0/3378785586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c000bd40 con 0x7ff9d006c140 2023-10-14T16:57:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9cffff640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000e370 tx=0x7ff9c000e830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0010040 con 0x7ff9d006c140 2023-10-14T16:57:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9c0009600 con 0x7ff9d006c140 2023-10-14T16:57:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9c0009970 con 0x7ff9d006c140 2023-10-14T16:57:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9b8009400 con 0x7ff9d006c140 2023-10-14T16:57:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.856+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9d00b1ea0 con 0x7ff9d006c140 2023-10-14T16:57:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.857+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9d004e160 con 0x7ff9d006c140 2023-10-14T16:57:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.859+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff9c0009ad0 con 0x7ff9d006c140 2023-10-14T16:57:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.860+0000 7ff9ccff9640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 0x7ff9b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.860+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff9c008a5e0 con 0x7ff9d006c140 2023-10-14T16:57:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.860+0000 7ff9cf7fe640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 0x7ff9b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.861+0000 7ff9cf7fe640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 0x7ff9b006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8034000 tx=0x7ff9b802f000 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:14.864+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9c0055660 con 0x7ff9d006c140 2023-10-14T16:57:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.086+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 --> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9d00adf70 con 0x7ff9b006af90 2023-10-14T16:57:15.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.089+0000 7ff9ccff9640 1 -- 172.21.15.29:0/3378785586 <== mgr.6765 v2:172.21.15.29:6832/3330033844 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7ff9b8030890 con 0x7ff9b006af90 2023-10-14T16:57:15.090 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:57:15.090 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:15.091 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:57:15.091 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:15.091 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:57:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 msgr2=0x7ff9b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 0x7ff9b006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff9b8034000 tx=0x7ff9b802f000 comp rx=0 tx=0).stop 2023-10-14T16:57:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 msgr2=0x7ff9d00c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000e370 tx=0x7ff9c000e830 comp rx=0 tx=0).stop 2023-10-14T16:57:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 shutdown_connections 2023-10-14T16:57:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7ff9b006af90 0x7ff9b006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9d0159e90 0x7ff9d006b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9d0159ab0 0x7ff9d00b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 --2- 172.21.15.29:0/3378785586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9d006c140 0x7ff9d00c44b0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 >> 172.21.15.29:0/3378785586 conn(0x7ff9d00ce9b0 msgr2=0x7ff9d0070480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:15.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 shutdown_connections 2023-10-14T16:57:15.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.092+0000 7ff9d6651640 1 -- 172.21.15.29:0/3378785586 wait complete. 2023-10-14T16:57:15.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:57:15.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.341+0000 7fe89f22e640 1 Processor -- start 2023-10-14T16:57:15.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.341+0000 7fe89f22e640 1 -- start start 2023-10-14T16:57:15.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.342+0000 7fe89f22e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8980ad970 0x7fe8980cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89f22e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89815b9d0 con 0x7fe89806c040 2023-10-14T16:57:15.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89f22e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe89815bcb0 con 0x7fe8980ccfb0 2023-10-14T16:57:15.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89f22e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe89815bf90 con 0x7fe8980ad970 2023-10-14T16:57:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89e22c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe89806c040 0x7fe89806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41114/0 (socket says 172.21.15.29:41114) 2023-10-14T16:57:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89e22c640 1 -- 172.21.15.29:0/338707558 learned_addr learned my addr 172.21.15.29:0/338707558 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.343+0000 7fe89da2b640 1 --2- 172.21.15.29:0/338707558 >> v2:172.21.15.186:3300/0 conn(0x7fe8980ad970 0x7fe8980cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375881176 0 0) 0x7fe89815b9d0 con 0x7fe89806c040 2023-10-14T16:57:15.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe884003520 con 0x7fe89806c040 2023-10-14T16:57:15.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3859733243 0 0) 0x7fe884003520 con 0x7fe89806c040 2023-10-14T16:57:15.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 >> v2:172.21.15.186:3300/0 conn(0x7fe8980ad970 msgr2=0x7fe8980cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:15.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 --2- 172.21.15.29:0/338707558 >> v2:172.21.15.186:3300/0 conn(0x7fe8980ad970 0x7fe8980cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 >> v1:172.21.15.186:6789/0 conn(0x7fe8980ccfb0 legacy=0x7fe8980cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980cc270 con 0x7fe89806c040 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe894001670 con 0x7fe89806c040 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 >> v1:172.21.15.29:6789/0 conn(0x7fe89806c040 legacy=0x7fe89806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe884003970 0x7fe884003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8840047c0 0x7fe884004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 0x7fe88400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe89815bf90 con 0x7fe884005180 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe88400c020 con 0x7fe8840047c0 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.344+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe88400c1e0 con 0x7fe884003970 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 0x7fe88400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89e22c640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe884003970 0x7fe884003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89da2b640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8840047c0 0x7fe884004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 -- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe884003970 msgr2=0x7fe884003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe884003970 0x7fe884003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 -- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8840047c0 msgr2=0x7fe884004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8840047c0 0x7fe884004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89ea2d640 1 -- 172.21.15.29:0/338707558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe884003520 con 0x7fe884005180 2023-10-14T16:57:15.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89da2b640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8840047c0 0x7fe884004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:15.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.345+0000 7fe89e22c640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe884003970 0x7fe884003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.346+0000 7fe89ea2d640 1 -- 172.21.15.29:0/338707558 reap_dead start 2023-10-14T16:57:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.346+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 0x7fe88400b980 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe890001e50 tx=0x7fe8840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69834dedb0d6c3a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890003b70 con 0x7fe884005180 2023-10-14T16:57:15.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe890003cf0 con 0x7fe884005180 2023-10-14T16:57:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89d22a640 1 -- 172.21.15.29:0/338707558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890005310 con 0x7fe884005180 2023-10-14T16:57:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 -- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 msgr2=0x7fe88400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 0x7fe88400b980 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fe890001e50 tx=0x7fe8840057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 -- 172.21.15.29:0/338707558 shutdown_connections 2023-10-14T16:57:15.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 --2- 172.21.15.29:0/338707558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe884005180 0x7fe88400b980 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 -- 172.21.15.29:0/338707558 >> 172.21.15.29:0/338707558 conn(0x7fe8980ce6a0 msgr2=0x7fe8980ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 -- 172.21.15.29:0/338707558 shutdown_connections 2023-10-14T16:57:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.347+0000 7fe89f22e640 1 -- 172.21.15.29:0/338707558 wait complete. 2023-10-14T16:57:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.349+0000 7fe89f22e640 1 Processor -- start 2023-10-14T16:57:15.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.349+0000 7fe89f22e640 1 -- start start 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884005180 0x7fe89815ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe898157e60 0x7fe8981f02d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981f0c70 con 0x7fe89815c680 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981f0f50 con 0x7fe884005180 2023-10-14T16:57:15.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.350+0000 7fe89f22e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8981f1230 con 0x7fe898157e60 2023-10-14T16:57:15.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46792/0 (socket says 172.21.15.29:46792) 2023-10-14T16:57:15.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 -- 172.21.15.29:0/1534283216 learned_addr learned my addr 172.21.15.29:0/1534283216 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:15.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89da2b640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe898157e60 0x7fe8981f02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 -- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe898157e60 msgr2=0x7fe8981f02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe898157e60 0x7fe8981f02d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89e22c640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884005180 0x7fe89815ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 -- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884005180 msgr2=0x7fe89815ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884005180 0x7fe89815ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:15.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.351+0000 7fe89ea2d640 1 -- 172.21.15.29:0/1534283216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980b23b0 con 0x7fe89815c680 2023-10-14T16:57:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe89ea2d640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe89002c060 tx=0x7fe8900014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe890006d50 con 0x7fe89815c680 2023-10-14T16:57:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe890003b80 con 0x7fe89815c680 2023-10-14T16:57:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8900396a0 con 0x7fe89815c680 2023-10-14T16:57:15.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8980b2690 con 0x7fe89815c680 2023-10-14T16:57:15.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.352+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8981582e0 con 0x7fe89815c680 2023-10-14T16:57:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.353+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8981f1090 con 0x7fe89815c680 2023-10-14T16:57:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.354+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 37) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe890039800 con 0x7fe89815c680 2023-10-14T16:57:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.355+0000 7fe88affd640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 0x7fe87c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.355+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fe8900ab6d0 con 0x7fe89815c680 2023-10-14T16:57:15.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.356+0000 7fe89e22c640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 0x7fe87c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:15.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.356+0000 7fe89e22c640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 0x7fe87c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe894000ff0 tx=0x7fe894009040 comp rx=0 tx=0).ready entity=mgr.6765 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:15.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.360+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8981f1090 con 0x7fe89815c680 2023-10-14T16:57:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:15.604+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fe8981f1090 con 0x7fe89815c680 2023-10-14T16:57:15.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:57:15.844 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:57:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.220+0000 7fe88affd640 1 -- 172.21.15.29:0/1534283216 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v38) v1 ==== 92+0+0 (secure 0 0 0) 0x7fe8981f1090 con 0x7fe89815c680 2023-10-14T16:57:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 msgr2=0x7fe87c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 0x7fe87c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe894000ff0 tx=0x7fe894009040 comp rx=0 tx=0).stop 2023-10-14T16:57:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 msgr2=0x7fe89815cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fe89002c060 tx=0x7fe8900014a0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 shutdown_connections 2023-10-14T16:57:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fe87c06af90 0x7fe87c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe89815c680 0x7fe89815cb00 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe898157e60 0x7fe8981f02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 --2- 172.21.15.29:0/1534283216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe884005180 0x7fe89815ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.223+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 >> 172.21.15.29:0/1534283216 conn(0x7fe8980ce6a0 msgr2=0x7fe8980adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.224+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 shutdown_connections 2023-10-14T16:57:16.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.224+0000 7fe89f22e640 1 -- 172.21.15.29:0/1534283216 wait complete. 2023-10-14T16:57:16.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:57:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.453+0000 7f0e2b9a9640 1 Processor -- start 2023-10-14T16:57:16.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.453+0000 7f0e2b9a9640 1 -- start start 2023-10-14T16:57:16.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e2b9a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0e240ad350 0x7f0e24069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e2b9a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e2406a730 con 0x7f0e2415d150 2023-10-14T16:57:16.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e2b9a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e240ac890 con 0x7f0e2406a2e0 2023-10-14T16:57:16.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e2b9a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e2415d530 con 0x7f0e240ad350 2023-10-14T16:57:16.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e28f44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0e240ad350 0x7f0e24069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e29745640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0e2415d150 0x7f0e24160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41118/0 (socket says 172.21.15.29:41118) 2023-10-14T16:57:16.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.455+0000 7f0e29745640 1 -- 172.21.15.29:0/312759091 learned_addr learned my addr 172.21.15.29:0/312759091 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.456+0000 7f0e28f44640 1 -- 172.21.15.29:0/312759091 >> v1:172.21.15.186:6789/0 conn(0x7f0e2406a2e0 legacy=0x7f0e240c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.456+0000 7f0e28f44640 1 -- 172.21.15.29:0/312759091 >> v1:172.21.15.29:6789/0 conn(0x7f0e2415d150 legacy=0x7f0e24160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.456+0000 7f0e28f44640 1 -- 172.21.15.29:0/312759091 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e240ad7a0 con 0x7f0e240ad350 2023-10-14T16:57:16.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.456+0000 7f0e17fff640 1 -- 172.21.15.29:0/312759091 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183975123 0 0) 0x7f0e20001110 con 0x7f0e2415d150 2023-10-14T16:57:16.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.458+0000 7f0e28f44640 1 --2- 172.21.15.29:0/312759091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24069da0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0e180097a0 tx=0x7f0e1802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3200d568e17321f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.458+0000 7f0e17fff640 1 -- 172.21.15.29:0/312759091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18031830 con 0x7f0e240ad350 2023-10-14T16:57:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.458+0000 7f0e17fff640 1 -- 172.21.15.29:0/312759091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e18031990 con 0x7f0e240ad350 2023-10-14T16:57:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.458+0000 7f0e17fff640 1 -- 172.21.15.29:0/312759091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e18031d20 con 0x7f0e240ad350 2023-10-14T16:57:16.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/312759091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 msgr2=0x7f0e24069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/312759091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24069da0 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f0e180097a0 tx=0x7f0e1802b8d0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/312759091 shutdown_connections 2023-10-14T16:57:16.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/312759091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24069da0 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/312759091 >> 172.21.15.29:0/312759091 conn(0x7f0e240ce9b0 msgr2=0x7f0e240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:16.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/312759091 shutdown_connections 2023-10-14T16:57:16.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.459+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/312759091 wait complete. 2023-10-14T16:57:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.461+0000 7f0e2b9a9640 1 Processor -- start 2023-10-14T16:57:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.461+0000 7f0e2b9a9640 1 -- start start 2023-10-14T16:57:16.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e2406a2e0 0x7f0e24142c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e2415d150 0x7f0e24156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e240b3340 con 0x7f0e240ad350 2023-10-14T16:57:16.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e240b09b0 con 0x7f0e2406a2e0 2023-10-14T16:57:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e2b9a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e240b2d70 con 0x7f0e2415d150 2023-10-14T16:57:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e28f44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e28f44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46842/0 (socket says 172.21.15.29:46842) 2023-10-14T16:57:16.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e28f44640 1 -- 172.21.15.29:0/361695293 learned_addr learned my addr 172.21.15.29:0/361695293 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.462+0000 7f0e29745640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e2406a2e0 0x7f0e24142c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e29f46640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e2415d150 0x7f0e24156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e2415d150 msgr2=0x7f0e24156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e2415d150 0x7f0e24156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e2406a2e0 msgr2=0x7f0e24142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e2406a2e0 0x7f0e24142c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 -- 172.21.15.29:0/361695293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e18009400 con 0x7f0e240ad350 2023-10-14T16:57:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.463+0000 7f0e28f44640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802bee0 tx=0x7f0e180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.464+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e1803ba70 con 0x7f0e240ad350 2023-10-14T16:57:16.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.464+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e1803bbd0 con 0x7f0e240ad350 2023-10-14T16:57:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.464+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e180453f0 con 0x7f0e240ad350 2023-10-14T16:57:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.464+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e240b1180 con 0x7f0e240ad350 2023-10-14T16:57:16.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.464+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e240b0bc0 con 0x7f0e240ad350 2023-10-14T16:57:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.465+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2404e160 con 0x7f0e240ad350 2023-10-14T16:57:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.467+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0e18045550 con 0x7f0e240ad350 2023-10-14T16:57:16.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.468+0000 7f0e15ffb640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 0x7f0e0806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.468+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f0e180abe80 con 0x7f0e240ad350 2023-10-14T16:57:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.468+0000 7f0e29745640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 msgr2=0x7f0e0806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:16.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.468+0000 7f0e29745640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 0x7f0e0806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:16.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.473+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e18076cd0 con 0x7f0e240ad350 2023-10-14T16:57:16.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.669+0000 7f0e29745640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 msgr2=0x7f0e0806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.669+0000 7f0e29745640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 0x7f0e0806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.669+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f0e240ade10 con 0x7f0e240ad350 2023-10-14T16:57:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.675+0000 7f0e15ffb640 1 -- 172.21.15.29:0/361695293 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v29)=0 v29) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0e1807a2f0 con 0x7f0e240ad350 2023-10-14T16:57:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 msgr2=0x7f0e0806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 0x7f0e0806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 msgr2=0x7f0e24145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f0e1802bee0 tx=0x7f0e180046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 shutdown_connections 2023-10-14T16:57:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7f0e0806afe0 0x7f0e0806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e2415d150 0x7f0e24156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e240ad350 0x7f0e24145660 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 --2- 172.21.15.29:0/361695293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e2406a2e0 0x7f0e24142c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 >> 172.21.15.29:0/361695293 conn(0x7f0e240ce9b0 msgr2=0x7f0e240c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 shutdown_connections 2023-10-14T16:57:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.678+0000 7f0e2b9a9640 1 -- 172.21.15.29:0/361695293 wait complete. 2023-10-14T16:57:16.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:57:16.791 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:16.790+0000 7f74a5630040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:16.792 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:16.792+0000 7fc84c62b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:16.849 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:16.848+0000 7f74a5630040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:16.850 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:16.850+0000 7fc84c62b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:16.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.915+0000 7fca7b683640 1 Processor -- start 2023-10-14T16:57:16.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.915+0000 7fca7b683640 1 -- start start 2023-10-14T16:57:16.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7b683640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca7415d030 0x7fca740d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7b683640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fca740cd260 0x7fca740cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7b683640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca740b5d90 con 0x7fca7415d410 2023-10-14T16:57:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7b683640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740d2b50 con 0x7fca7415d030 2023-10-14T16:57:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7b683640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca740cdbf0 con 0x7fca740cd260 2023-10-14T16:57:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca7415d030 0x7fca740d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fca7415d030 0x7fca740d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46856/0 (socket says 172.21.15.29:46856) 2023-10-14T16:57:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 -- 172.21.15.29:0/2358692045 learned_addr learned my addr 172.21.15.29:0/2358692045 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca79c20640 1 --2- 172.21.15.29:0/2358692045 >> v2:172.21.15.186:3300/0 conn(0x7fca740cd260 0x7fca740cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 -- 172.21.15.29:0/2358692045 >> v2:172.21.15.186:3300/0 conn(0x7fca740cd260 msgr2=0x7fca740cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 --2- 172.21.15.29:0/2358692045 >> v2:172.21.15.186:3300/0 conn(0x7fca740cd260 0x7fca740cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 -- 172.21.15.29:0/2358692045 >> v1:172.21.15.186:6789/0 conn(0x7fca7415d410 legacy=0x7fca74161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.917+0000 7fca7941f640 1 -- 172.21.15.29:0/2358692045 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca74160fc0 con 0x7fca7415d030 2023-10-14T16:57:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca7941f640 1 --2- 172.21.15.29:0/2358692045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740d1630 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fca740d0750 tx=0x7fca7002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12fe48e99cd0e521 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca6bfff640 1 -- 172.21.15.29:0/2358692045 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70030860 con 0x7fca7415d030 2023-10-14T16:57:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca6bfff640 1 -- 172.21.15.29:0/2358692045 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca700309c0 con 0x7fca7415d030 2023-10-14T16:57:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca6bfff640 1 -- 172.21.15.29:0/2358692045 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca70030d50 con 0x7fca7415d030 2023-10-14T16:57:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca7b683640 1 -- 172.21.15.29:0/2358692045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 msgr2=0x7fca740d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.919+0000 7fca7b683640 1 --2- 172.21.15.29:0/2358692045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740d1630 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7fca740d0750 tx=0x7fca7002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 -- 172.21.15.29:0/2358692045 shutdown_connections 2023-10-14T16:57:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 --2- 172.21.15.29:0/2358692045 >> v2:172.21.15.186:3300/0 conn(0x7fca740cd260 0x7fca740cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 --2- 172.21.15.29:0/2358692045 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740d1630 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 -- 172.21.15.29:0/2358692045 >> 172.21.15.29:0/2358692045 conn(0x7fca740cf720 msgr2=0x7fca740cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 -- 172.21.15.29:0/2358692045 shutdown_connections 2023-10-14T16:57:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.920+0000 7fca7b683640 1 -- 172.21.15.29:0/2358692045 wait complete. 2023-10-14T16:57:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.921+0000 7fca7b683640 1 Processor -- start 2023-10-14T16:57:16.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 -- start start 2023-10-14T16:57:16.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca740cd260 0x7fca740c75e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 0x7fca740c9000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740c7c00 con 0x7fca7415d030 2023-10-14T16:57:16.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca740b1e70 con 0x7fca7415d410 2023-10-14T16:57:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.922+0000 7fca7b683640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca741600a0 con 0x7fca740cd260 2023-10-14T16:57:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46872/0 (socket says 172.21.15.29:46872) 2023-10-14T16:57:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 -- 172.21.15.29:0/4146999679 learned_addr learned my addr 172.21.15.29:0/4146999679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:16.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca7941f640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca740cd260 0x7fca740c75e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca79c20640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 0x7fca740c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca740cd260 msgr2=0x7fca740c75e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca740cd260 0x7fca740c75e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 msgr2=0x7fca740c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:16.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 0x7fca740c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca78c1e640 1 -- 172.21.15.29:0/4146999679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca700094d0 con 0x7fca7415d030 2023-10-14T16:57:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.923+0000 7fca79c20640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 0x7fca740c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca78c1e640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fca6400b3c0 tx=0x7fca6400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64011830 con 0x7fca7415d030 2023-10-14T16:57:16.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca64011990 con 0x7fca7415d030 2023-10-14T16:57:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca74160360 con 0x7fca7415d030 2023-10-14T16:57:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fca64011d80 con 0x7fca7415d030 2023-10-14T16:57:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.924+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca740c8ae0 con 0x7fca7415d030 2023-10-14T16:57:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.925+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca740c8d00 con 0x7fca7415d030 2023-10-14T16:57:16.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.926+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fca6401b460 con 0x7fca7415d030 2023-10-14T16:57:16.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.927+0000 7fca69ffb640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 0x7fca5806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.927+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fca64089340 con 0x7fca7415d030 2023-10-14T16:57:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.927+0000 7fca7941f640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 msgr2=0x7fca5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.927+0000 7fca7941f640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 0x7fca5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:16.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:16.931+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fca640542a0 con 0x7fca7415d030 2023-10-14T16:57:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.121+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fca7404e160 con 0x7fca7415d030 2023-10-14T16:57:17.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.128+0000 7fca69ffb640 1 -- 172.21.15.29:0/4146999679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v30)=0 v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7fca640578c0 con 0x7fca7415d030 2023-10-14T16:57:17.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.128+0000 7fca7941f640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 msgr2=0x7fca5806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:17.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.128+0000 7fca7941f640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 0x7fca5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:17.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 msgr2=0x7fca5806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:17.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 0x7fca5806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 msgr2=0x7fca740b1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x7fca6400b3c0 tx=0x7fca6400b880 comp rx=0 tx=0).stop 2023-10-14T16:57:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 shutdown_connections 2023-10-14T16:57:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fca5806afe0 0x7fca5806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fca7415d410 0x7fca740c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fca7415d030 0x7fca740b1930 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 --2- 172.21.15.29:0/4146999679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fca740cd260 0x7fca740c75e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 >> 172.21.15.29:0/4146999679 conn(0x7fca740cf720 msgr2=0x7fca7415c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.130+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 shutdown_connections 2023-10-14T16:57:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.131+0000 7fca7b683640 1 -- 172.21.15.29:0/4146999679 wait complete. 2023-10-14T16:57:17.150 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-14T16:57:17.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.302+0000 7f74a5630040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:17.335 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.335+0000 7fc84c62b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:17.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.344+0000 7fbae95b0640 1 Processor -- start 2023-10-14T16:57:17.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.344+0000 7fbae95b0640 1 -- start start 2023-10-14T16:57:17.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbae4159ab0 0x7fbae40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159e90 0x7fbae406fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbae406c140 0x7fbae406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae406cb10 con 0x7fbae406c140 2023-10-14T16:57:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbae415d890 con 0x7fbae4159ab0 2023-10-14T16:57:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.346+0000 7fbae95b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae40cd1f0 con 0x7fbae4159e90 2023-10-14T16:57:17.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbae406c140 0x7fbae406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbae406c140 0x7fbae406c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46884/0 (socket says 172.21.15.29:46884) 2023-10-14T16:57:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 -- 172.21.15.29:0/1635724388 learned_addr learned my addr 172.21.15.29:0/1635724388 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1635724388 >> v2:172.21.15.186:3300/0 conn(0x7fbae4159ab0 0x7fbae40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbada7fc640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159e90 0x7fbae406fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 -- 172.21.15.29:0/1635724388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159e90 msgr2=0x7fbae406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159e90 0x7fbae406fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 -- 172.21.15.29:0/1635724388 >> v2:172.21.15.186:3300/0 conn(0x7fbae4159ab0 msgr2=0x7fbae40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1635724388 >> v2:172.21.15.186:3300/0 conn(0x7fbae4159ab0 0x7fbae40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.347+0000 7fbae37fe640 1 -- 172.21.15.29:0/1635724388 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae40cd4d0 con 0x7fbae406c140 2023-10-14T16:57:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.348+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae406c140 0x7fbae406c5d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009870 tx=0x7fbadc02d9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120ecd0ecb364783 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.348+0000 7fbae27fc640 1 -- 172.21.15.29:0/1635724388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc031810 con 0x7fbae406c140 2023-10-14T16:57:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.348+0000 7fbae27fc640 1 -- 172.21.15.29:0/1635724388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc031970 con 0x7fbae406c140 2023-10-14T16:57:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.348+0000 7fbae27fc640 1 -- 172.21.15.29:0/1635724388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc031d00 con 0x7fbae406c140 2023-10-14T16:57:17.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 -- 172.21.15.29:0/1635724388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae406c140 msgr2=0x7fbae406c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae406c140 0x7fbae406c5d0 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fbadc009870 tx=0x7fbadc02d9f0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 -- 172.21.15.29:0/1635724388 shutdown_connections 2023-10-14T16:57:17.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae406c140 0x7fbae406c5d0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 --2- 172.21.15.29:0/1635724388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159e90 0x7fbae406fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 --2- 172.21.15.29:0/1635724388 >> v2:172.21.15.186:3300/0 conn(0x7fbae4159ab0 0x7fbae40ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 -- 172.21.15.29:0/1635724388 >> 172.21.15.29:0/1635724388 conn(0x7fbae40ce9b0 msgr2=0x7fbae40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 -- 172.21.15.29:0/1635724388 shutdown_connections 2023-10-14T16:57:17.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.349+0000 7fbae95b0640 1 -- 172.21.15.29:0/1635724388 wait complete. 2023-10-14T16:57:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.351+0000 7fbae95b0640 1 Processor -- start 2023-10-14T16:57:17.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.351+0000 7fbae95b0640 1 -- start start 2023-10-14T16:57:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae406c140 0x7fbae4157bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159ab0 0x7fbae40c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae4159e90 0x7fbae40c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae415a3d0 con 0x7fbae4159e90 2023-10-14T16:57:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbae415a6b0 con 0x7fbae406c140 2023-10-14T16:57:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae95b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbae415a990 con 0x7fbae4159ab0 2023-10-14T16:57:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.352+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae4159e90 0x7fbae40c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae4159e90 0x7fbae40c6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46898/0 (socket says 172.21.15.29:46898) 2023-10-14T16:57:17.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 -- 172.21.15.29:0/1992384406 learned_addr learned my addr 172.21.15.29:0/1992384406 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae406c140 0x7fbae4157bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbada7fc640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159ab0 0x7fbae40c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159ab0 msgr2=0x7fbae40c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbae4159ab0 0x7fbae40c6730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae406c140 msgr2=0x7fbae4157bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae406c140 0x7fbae4157bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbadc0094d0 con 0x7fbae4159e90 2023-10-14T16:57:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbae406c140 0x7fbae4157bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:17.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbae37fe640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbae4159e90 0x7fbae40c6c70 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fbadc02da30 tx=0x7fbadc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.353+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc039810 con 0x7fbae4159e90 2023-10-14T16:57:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.354+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbadc039990 con 0x7fbae4159e90 2023-10-14T16:57:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.354+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbadc039d40 con 0x7fbae4159e90 2023-10-14T16:57:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.354+0000 7fbae95b0640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae415ac70 con 0x7fbae4159e90 2023-10-14T16:57:17.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.354+0000 7fbae95b0640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae40c7260 con 0x7fbae4159e90 2023-10-14T16:57:17.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.355+0000 7fbae95b0640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae415ae90 con 0x7fbae4159e90 2023-10-14T16:57:17.362 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.354+0000 7f74a5630040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.357+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 38) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbadc02f3a0 con 0x7fbae4159e90 2023-10-14T16:57:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.358+0000 7fbadbfff640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:17.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.358+0000 7fbae2ffd640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.358+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbadc0aba20 con 0x7fbae4159e90 2023-10-14T16:57:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.358+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:17.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.363+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbadc076880 con 0x7fbae4159e90 2023-10-14T16:57:17.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.394+0000 7fc84c62b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:17.406 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.405+0000 7f74a5630040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:17.449 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.448+0000 7fc84c62b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:17.466 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.465+0000 7f74a5630040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:17.514 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.513+0000 7fc84c62b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:17.514 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.514+0000 7f74a5630040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:17.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.559+0000 7fbae2ffd640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:17.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.559+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:17.566 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.566+0000 7fc84c62b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.579+0000 7fbae95b0640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae40c7450 con 0x7fbac406afe0 2023-10-14T16:57:17.626 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.625+0000 7f74a5630040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:17.684 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.683+0000 7fc84c62b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:17.689 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.688+0000 7f74a5630040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:17.739 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.738+0000 7f74a5630040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:17.751 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.750+0000 7fc84c62b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:17.795 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.795+0000 7f74a5630040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:17.804 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.803+0000 7fc84c62b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:17.845 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:17.844+0000 7f74a5630040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:17.864 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.863+0000 7fc84c62b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:17.917 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:17.917+0000 7fc84c62b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.960+0000 7fbae2ffd640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:17.960+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:57:18.220 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.218+0000 7f74a5630040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:18.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.325+0000 7fc84c62b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:18.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.376+0000 7f74a5630040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:18.485 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.484+0000 7f74a5630040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:18.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.494+0000 7fc84c62b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:18.535 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.534+0000 7f74a5630040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:18.611 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.610+0000 7fc84c62b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:18.641 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.640+0000 7f74a5630040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:18.664 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.663+0000 7fc84c62b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:18.746 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.745+0000 7f74a5630040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:18.761+0000 7fbae2ffd640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:18.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:18.761+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:57:18.778 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.778+0000 7fc84c62b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:18.893 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:18.892+0000 7fc84c62b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:18.900 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.898+0000 7f74a5630040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:18.948 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:18.947+0000 7f74a5630040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:19.059 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:19.058+0000 7fc84c62b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:19.065 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:19.063+0000 7f74a5630040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:19.112 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:19.111+0000 7fc84c62b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:19.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:19.238+0000 7fc84c62b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:19.858 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:19.857+0000 7f74a5630040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:20.130 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:20.129+0000 7fc84c62b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:20.153 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:20.152+0000 7f74a5630040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:20.210 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:20.209+0000 7f74a5630040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:20.322 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:20.321+0000 7f74a5630040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:20.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:20.363+0000 7fbae2ffd640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3330033844 2023-10-14T16:57:20.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:20.363+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:57:20.446 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:20.445+0000 7fc84c62b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:20.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:20.506+0000 7fc84c62b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:20.627 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:20.626+0000 7fc84c62b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:20.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:20.648+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 39) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fbadc0a9f10 con 0x7fbae4159e90 2023-10-14T16:57:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:20.648+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 msgr2=0x7fbac406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:20.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:20.648+0000 7fbadbfff640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/3330033844,v1:172.21.15.29:6833/3330033844] conn(0x7fbac406afe0 0x7fbac406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.008 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7c89e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.009 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7c89e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.009 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7c89e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.009 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7c89e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7c89e3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7cba29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.010 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7cba29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7cba29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7cba29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.011 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.008+0000 7fc7cba29640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.662+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 40) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbadc076c50 con 0x7fbae4159e90 2023-10-14T16:57:21.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.663+0000 7fbadbfff640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fbac40358e0 0x7fbac4074f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.663+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 --> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbae40c7450 con 0x7fbac40358e0 2023-10-14T16:57:21.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.667+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fbac40358e0 0x7fbac4074f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.679+0000 7fbae2ffd640 1 --2- 172.21.15.29:0/1992384406 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fbac40358e0 0x7fbac4074f30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbad4006fd0 tx=0x7fbad4008040 comp rx=0 tx=0).ready entity=mgr.7173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:21.706 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:21.706+0000 7fc7d2a37640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-14T16:57:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.706+0000 7fbadbfff640 1 -- 172.21.15.29:0/1992384406 <== mgr.7173 v2:172.21.15.29:6832/505146228 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7fbae40c7450 con 0x7fbac40358e0 2023-10-14T16:57:21.707 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-14T16:57:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T16:57:21.726 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T16:57:21.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.942+0000 7f7521f89640 1 Processor -- start 2023-10-14T16:57:21.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.942+0000 7f7521f89640 1 -- start start 2023-10-14T16:57:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.943+0000 7f7521f89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f751c0ada00 0x7f751c0713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f7521f89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f751c06bb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f7521f89640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f751c0ae0a0 con 0x7f751c0c98a0 2023-10-14T16:57:21.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f7521f89640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f751c0cd5c0 con 0x7f751c0ada00 2023-10-14T16:57:21.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f7521f89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0cd860 con 0x7f751c0706d0 2023-10-14T16:57:21.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f751affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f751c0c98a0 0x7f751c06b480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40186/0 (socket says 172.21.15.29:40186) 2023-10-14T16:57:21.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f751affd640 1 -- 172.21.15.29:0/472083929 learned_addr learned my addr 172.21.15.29:0/472083929 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f751b7fe640 1 --2- 172.21.15.29:0/472083929 >> v2:172.21.15.186:3300/0 conn(0x7f751c0ada00 0x7f751c0713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.944+0000 7f751bfff640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f751c06bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971680866 0 0) 0x7f751c0ae0a0 con 0x7f751c0c98a0 2023-10-14T16:57:21.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7504003520 con 0x7f751c0c98a0 2023-10-14T16:57:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2115987098 0 0) 0x7f7504003520 con 0x7f751c0c98a0 2023-10-14T16:57:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 msgr2=0x7f751c06bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f751c06bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 >> v2:172.21.15.186:3300/0 conn(0x7f751c0ada00 msgr2=0x7f751c0713c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 --2- 172.21.15.29:0/472083929 >> v2:172.21.15.186:3300/0 conn(0x7f751c0ada00 0x7f751c0713c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c0cdb40 con 0x7f751c0c98a0 2023-10-14T16:57:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7510001670 con 0x7f751c0c98a0 2023-10-14T16:57:21.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 >> v1:172.21.15.29:6789/0 conn(0x7f751c0c98a0 legacy=0x7f751c06b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f7504003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504003ef0 0x7f75040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504004d00 0x7f7504005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0cd5c0 con 0x7f7504003ef0 2023-10-14T16:57:21.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c0cd860 con 0x7f7504004d00 2023-10-14T16:57:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750400b980 con 0x7f751c0706d0 2023-10-14T16:57:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.945+0000 7f751affd640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504003ef0 0x7f75040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751b7fe640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f7504003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751bfff640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504004d00 0x7f7504005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751affd640 1 -- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 msgr2=0x7f7504003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751affd640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c0706d0 0x7f7504003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751affd640 1 -- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504004d00 msgr2=0x7f7504005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751affd640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504004d00 0x7f7504005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.946+0000 7f751affd640 1 -- 172.21.15.29:0/472083929 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7504003520 con 0x7f7504003ef0 2023-10-14T16:57:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.947+0000 7f751affd640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504003ef0 0x7f75040047c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f751000c240 tx=0x7f75040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c1decd6b0de5d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.947+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751002cae0 con 0x7f7504003ef0 2023-10-14T16:57:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.947+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751002cc40 con 0x7f7504003ef0 2023-10-14T16:57:21.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.947+0000 7f751a7fc640 1 -- 172.21.15.29:0/472083929 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510005af0 con 0x7f7504003ef0 2023-10-14T16:57:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 -- 172.21.15.29:0/472083929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504003ef0 msgr2=0x7f75040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 --2- 172.21.15.29:0/472083929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7504003ef0 0x7f75040047c0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f751000c240 tx=0x7f75040057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f751bfff640 1 -- 172.21.15.29:0/472083929 reap_dead start 2023-10-14T16:57:21.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 -- 172.21.15.29:0/472083929 shutdown_connections 2023-10-14T16:57:21.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 -- 172.21.15.29:0/472083929 >> 172.21.15.29:0/472083929 conn(0x7f751c0ce840 msgr2=0x7f751c0cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:21.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 -- 172.21.15.29:0/472083929 shutdown_connections 2023-10-14T16:57:21.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.948+0000 7f7521f89640 1 -- 172.21.15.29:0/472083929 wait complete. 2023-10-14T16:57:21.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.950+0000 7f7521f89640 1 Processor -- start 2023-10-14T16:57:21.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.950+0000 7f7521f89640 1 -- start start 2023-10-14T16:57:21.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504003ef0 0x7f751c0c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 0x7f751c151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14d6e0 con 0x7f751c0c4c40 2023-10-14T16:57:21.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14e510 con 0x7f7504003ef0 2023-10-14T16:57:21.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f7521f89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f751c14d900 con 0x7f751c14d250 2023-10-14T16:57:21.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f751affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f751affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41064/0 (socket says 172.21.15.29:41064) 2023-10-14T16:57:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.951+0000 7f751affd640 1 -- 172.21.15.29:0/971248881 learned_addr learned my addr 172.21.15.29:0/971248881 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751b7fe640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504003ef0 0x7f751c0c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751bfff640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 0x7f751c151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 -- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 msgr2=0x7f751c151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:21.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 0x7f751c151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 -- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504003ef0 msgr2=0x7f751c0c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504003ef0 0x7f751c0c4700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 -- 172.21.15.29:0/971248881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f751c14dbe0 con 0x7f751c0c4c40 2023-10-14T16:57:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751bfff640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 0x7f751c151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:21.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.952+0000 7f751affd640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f751000c240 tx=0x7f7510001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:21.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.953+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7510039070 con 0x7f751c0c4c40 2023-10-14T16:57:21.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.953+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510005c80 con 0x7f751c0c4c40 2023-10-14T16:57:21.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.953+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f751003c040 con 0x7f751c0c4c40 2023-10-14T16:57:21.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.953+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f751c14dec0 con 0x7f751c0c4c40 2023-10-14T16:57:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.953+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751c151540 con 0x7f751c0c4c40 2023-10-14T16:57:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.954+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f751c04e160 con 0x7f751c0c4c40 2023-10-14T16:57:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.955+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 40) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f7510032070 con 0x7f751c0c4c40 2023-10-14T16:57:21.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.956+0000 7f7503fff640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 0x7f750806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.956+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f75100aaa90 con 0x7f751c0c4c40 2023-10-14T16:57:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.956+0000 7f751b7fe640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 0x7f750806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:21.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.957+0000 7f751b7fe640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 0x7f750806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f751c1f34d0 tx=0x7f750c00b040 comp rx=0 tx=0).ready entity=mgr.7173 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:21.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:21.961+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75100796b0 con 0x7f751c0c4c40 2023-10-14T16:57:22.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.209+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f751c14e650 con 0x7f751c0c4c40 2023-10-14T16:57:22.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.660+0000 7f7503fff640 1 -- 172.21.15.29:0/971248881 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v41) v1 ==== 93+0+0 (secure 0 0 0) 0x7f7510046070 con 0x7f751c0c4c40 2023-10-14T16:57:22.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.665+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 msgr2=0x7f750806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:22.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.665+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 0x7f750806d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f751c1f34d0 tx=0x7f750c00b040 comp rx=0 tx=0).stop 2023-10-14T16:57:22.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.665+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 msgr2=0x7f751c14cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:22.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.665+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f751000c240 tx=0x7f7510001a00 comp rx=0 tx=0).stop 2023-10-14T16:57:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 shutdown_connections 2023-10-14T16:57:22.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f750806ad10 0x7f750806d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f751c14d250 0x7f751c151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f751c0c4c40 0x7f751c14cd10 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 --2- 172.21.15.29:0/971248881 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7504003ef0 0x7f751c0c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.666+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 >> 172.21.15.29:0/971248881 conn(0x7f751c0ce840 msgr2=0x7f751c06b480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:22.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.667+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 shutdown_connections 2023-10-14T16:57:22.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:22.667+0000 7f7521f89640 1 -- 172.21.15.29:0/971248881 wait complete. 2023-10-14T16:57:22.699 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:57:22.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:57:22.842 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:57:22.944 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:57:22.945 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:22.945 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:23.003 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:23.003 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:23.056 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:23.195 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.194+0000 7f7f3b013040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:23.224 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.224+0000 7f3a44a45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:23.249 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.248+0000 7f7f3b013040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.276+0000 7f410ff8a640 1 Processor -- start 2023-10-14T16:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.276+0000 7f410ff8a640 1 -- start start 2023-10-14T16:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41080ae220 0x7f41080cabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 0x7f4108069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080705d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f410806a770 con 0x7f41080ae220 2023-10-14T16:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41080cf760 con 0x7f4108160f30 2023-10-14T16:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410ff8a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41081614f0 con 0x7f410806a2e0 2023-10-14T16:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410dd26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41080ae220 0x7f41080cabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410dd26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41080ae220 0x7f41080cabd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41098/0 (socket says 172.21.15.29:41098) 2023-10-14T16:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410dd26640 1 -- 172.21.15.29:0/1729546591 learned_addr learned my addr 172.21.15.29:0/1729546591 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.278+0000 7f410e527640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410dd26640 1 -- 172.21.15.29:0/1729546591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 msgr2=0x7f41080705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410dd26640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080705d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410dd26640 1 -- 172.21.15.29:0/1729546591 >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 msgr2=0x7f4108069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410d525640 1 --2- 172.21.15.29:0/1729546591 >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 0x7f4108069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410dd26640 1 --2- 172.21.15.29:0/1729546591 >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 0x7f4108069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410dd26640 1 -- 172.21.15.29:0/1729546591 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41081617d0 con 0x7f41080ae220 2023-10-14T16:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.279+0000 7f410d525640 1 --2- 172.21.15.29:0/1729546591 >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 0x7f4108069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.280+0000 7f410dd26640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080cabd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f41080719e0 tx=0x7f410402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e2d35681b67b361 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:23.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.280+0000 7f410cd24640 1 -- 172.21.15.29:0/1729546591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41040317c0 con 0x7f41080ae220 2023-10-14T16:57:23.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.280+0000 7f410cd24640 1 -- 172.21.15.29:0/1729546591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104031920 con 0x7f41080ae220 2023-10-14T16:57:23.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.280+0000 7f410cd24640 1 -- 172.21.15.29:0/1729546591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104031cb0 con 0x7f41080ae220 2023-10-14T16:57:23.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 -- 172.21.15.29:0/1729546591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 msgr2=0x7f41080cabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080cabd0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f41080719e0 tx=0x7f410402b860 comp rx=0 tx=0).stop 2023-10-14T16:57:23.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 -- 172.21.15.29:0/1729546591 shutdown_connections 2023-10-14T16:57:23.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080705d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1729546591 >> v2:172.21.15.186:3300/0 conn(0x7f4108160f30 0x7f4108069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1729546591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080cabd0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 -- 172.21.15.29:0/1729546591 >> 172.21.15.29:0/1729546591 conn(0x7f41080cfb70 msgr2=0x7f41080cff50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 -- 172.21.15.29:0/1729546591 shutdown_connections 2023-10-14T16:57:23.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.281+0000 7f410ff8a640 1 -- 172.21.15.29:0/1729546591 wait complete. 2023-10-14T16:57:23.287 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.282+0000 7f3a44a45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.283+0000 7f410ff8a640 1 Processor -- start 2023-10-14T16:57:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.283+0000 7f410ff8a640 1 -- start start 2023-10-14T16:57:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080c64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 0x7f41080b5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080c9490 con 0x7f41080ae220 2023-10-14T16:57:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41080c69e0 con 0x7f4108160f30 2023-10-14T16:57:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410ff8a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41080c8880 con 0x7f410806a2e0 2023-10-14T16:57:23.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410d525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410d525640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41114/0 (socket says 172.21.15.29:41114) 2023-10-14T16:57:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410d525640 1 -- 172.21.15.29:0/1262340577 learned_addr learned my addr 172.21.15.29:0/1262340577 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410dd26640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080c64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410e527640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 0x7f41080b5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410d525640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 msgr2=0x7f41080c64a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.284+0000 7f410d525640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080c64a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410d525640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 msgr2=0x7f41080b5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410d525640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 0x7f41080b5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410d525640 1 -- 172.21.15.29:0/1262340577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040094d0 con 0x7f41080ae220 2023-10-14T16:57:23.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410e527640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 0x7f41080b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410d525640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c7f0 tx=0x7f40f800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8007820 con 0x7f41080ae220 2023-10-14T16:57:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40f8007980 con 0x7f41080ae220 2023-10-14T16:57:23.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41080c8b60 con 0x7f41080ae220 2023-10-14T16:57:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f8007cf0 con 0x7f41080ae220 2023-10-14T16:57:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.285+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41080c9120 con 0x7f41080ae220 2023-10-14T16:57:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.287+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41080cff50 con 0x7f41080ae220 2023-10-14T16:57:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.288+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f40f80113f0 con 0x7f41080ae220 2023-10-14T16:57:23.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.289+0000 7f40fe7fc640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 0x7f40ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.289+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f40f8089d70 con 0x7f41080ae220 2023-10-14T16:57:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.289+0000 7f410dd26640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 msgr2=0x7f40ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.289+0000 7f410dd26640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 0x7f40ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:23.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.293+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40f8054a70 con 0x7f41080ae220 2023-10-14T16:57:23.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.488+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f410804e160 con 0x7f41080ae220 2023-10-14T16:57:23.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.490+0000 7f410dd26640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 msgr2=0x7f40ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:23.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.490+0000 7f410dd26640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 0x7f40ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:23.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.491+0000 7f40fe7fc640 1 -- 172.21.15.29:0/1262340577 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+17127 (secure 0 0 0) 0x7f40f801f090 con 0x7f41080ae220 2023-10-14T16:57:23.493 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:23.493 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":97,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:20.631064+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3356363773":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6833/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6832/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3967724404":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2171619166":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3505521487":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3693879118":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/4176788283":"2023-10-15T16:57:20.630995+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:23.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 msgr2=0x7f40ec06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:23.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 0x7f40ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 msgr2=0x7f41080c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x7f40f800c7f0 tx=0x7f40f800ccb0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 shutdown_connections 2023-10-14T16:57:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f40ec06af90 0x7f40ec06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4108160f30 0x7f41080b5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41080ae220 0x7f41080c7dc0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 --2- 172.21.15.29:0/1262340577 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f410806a2e0 0x7f41080c64a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 >> 172.21.15.29:0/1262340577 conn(0x7f41080cfb70 msgr2=0x7f41080adac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 shutdown_connections 2023-10-14T16:57:23.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.494+0000 7f410ff8a640 1 -- 172.21.15.29:0/1262340577 wait complete. 2023-10-14T16:57:23.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:57:03.662290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:57:04.248899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:57:23.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:57:23.694 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.693+0000 7f7f3b013040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.730+0000 7f6702a90640 1 Processor -- start 2023-10-14T16:57:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.731+0000 7f6702a90640 1 -- start start 2023-10-14T16:57:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6702a90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f66fc0cad80 0x7f66fc15dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6702a90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc0b4c00 con 0x7f66fc159c00 2023-10-14T16:57:23.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6702a90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66fc0c9fc0 con 0x7f66fc0b3530 2023-10-14T16:57:23.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6702a90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66fc0ca2a0 con 0x7f66fc0cad80 2023-10-14T16:57:23.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6701a8e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66fc159c00 0x7f66fc15d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40206/0 (socket says 172.21.15.29:40206) 2023-10-14T16:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f6701a8e640 1 -- 172.21.15.29:0/1109710008 learned_addr learned my addr 172.21.15.29:0/1109710008 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.732+0000 7f670128d640 1 --2- 172.21.15.29:0/1109710008 >> v2:172.21.15.29:3300/0 conn(0x7f66fc0cad80 0x7f66fc15dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.733+0000 7f670128d640 1 -- 172.21.15.29:0/1109710008 >> v1:172.21.15.186:6789/0 conn(0x7f66fc0b3530 legacy=0x7f66fc0b39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.733+0000 7f670128d640 1 -- 172.21.15.29:0/1109710008 >> v1:172.21.15.29:6789/0 conn(0x7f66fc159c00 legacy=0x7f66fc15d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.733+0000 7f670128d640 1 -- 172.21.15.29:0/1109710008 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66fc0ca580 con 0x7f66fc0cad80 2023-10-14T16:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.734+0000 7f670128d640 1 --2- 172.21.15.29:0/1109710008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0cad80 0x7f66fc15dd00 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0097a0 tx=0x7f66ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc8c3c53436bffd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.734+0000 7f6700a8c640 1 -- 172.21.15.29:0/1109710008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec030610 con 0x7f66fc0cad80 2023-10-14T16:57:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.734+0000 7f6700a8c640 1 -- 172.21.15.29:0/1109710008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66ec030770 con 0x7f66fc0cad80 2023-10-14T16:57:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.734+0000 7f6700a8c640 1 -- 172.21.15.29:0/1109710008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66ec030b00 con 0x7f66fc0cad80 2023-10-14T16:57:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 -- 172.21.15.29:0/1109710008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0cad80 msgr2=0x7f66fc15dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 --2- 172.21.15.29:0/1109710008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0cad80 0x7f66fc15dd00 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f66ec0097a0 tx=0x7f66ec02b960 comp rx=0 tx=0).stop 2023-10-14T16:57:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 -- 172.21.15.29:0/1109710008 shutdown_connections 2023-10-14T16:57:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 --2- 172.21.15.29:0/1109710008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0cad80 0x7f66fc15dd00 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 -- 172.21.15.29:0/1109710008 >> 172.21.15.29:0/1109710008 conn(0x7f66fc0b0a50 msgr2=0x7f66fc0b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:23.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 -- 172.21.15.29:0/1109710008 shutdown_connections 2023-10-14T16:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.735+0000 7f6702a90640 1 -- 172.21.15.29:0/1109710008 wait complete. 2023-10-14T16:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.737+0000 7f6702a90640 1 Processor -- start 2023-10-14T16:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.737+0000 7f6702a90640 1 -- start start 2023-10-14T16:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66fc0cad80 0x7f66fc0c7620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66fc159c00 0x7f66fc0c7b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc15ce40 con 0x7f66fc0b3530 2023-10-14T16:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc15c250 con 0x7f66fc0cad80 2023-10-14T16:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6702a90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66fc15c510 con 0x7f66fc159c00 2023-10-14T16:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6701a8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6701a8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41126/0 (socket says 172.21.15.29:41126) 2023-10-14T16:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.738+0000 7f6701a8e640 1 -- 172.21.15.29:0/1870623485 learned_addr learned my addr 172.21.15.29:0/1870623485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f670128d640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66fc0cad80 0x7f66fc0c7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66fc159c00 msgr2=0x7f66fc0c7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66fc159c00 0x7f66fc0c7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66fc0cad80 msgr2=0x7f66fc0c7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66fc0cad80 0x7f66fc0c7620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 -- 172.21.15.29:0/1870623485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66ec009400 con 0x7f66fc0b3530 2023-10-14T16:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6701a8e640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f66f800c480 tx=0x7f66f800c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f8005ce0 con 0x7f66fc0b3530 2023-10-14T16:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66f8005e40 con 0x7f66fc0b3530 2023-10-14T16:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66f80072f0 con 0x7f66fc0b3530 2023-10-14T16:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66fc15c7f0 con 0x7f66fc0b3530 2023-10-14T16:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.739+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66fc156330 con 0x7f66fc0b3530 2023-10-14T16:57:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.741+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66fc04e9b0 con 0x7f66fc0b3530 2023-10-14T16:57:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.741+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f66f8007510 con 0x7f66fc0b3530 2023-10-14T16:57:23.748 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.745+0000 7f7f3b013040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:23.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.742+0000 7f66f27fc640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 0x7f66e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.742+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f66f8088df0 con 0x7f66fc0b3530 2023-10-14T16:57:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.742+0000 7f670128d640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 msgr2=0x7f66e006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.742+0000 7f670128d640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 0x7f66e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:23.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.746+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66f8053ca0 con 0x7f66fc0b3530 2023-10-14T16:57:23.774 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.774+0000 7f3a44a45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:23.796 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.795+0000 7f7f3b013040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:23.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.841+0000 7f3a44a45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:23.856 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.855+0000 7f7f3b013040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:23.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.895+0000 7f3a44a45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:23.905 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:23.904+0000 7f7f3b013040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.943+0000 7f670128d640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 msgr2=0x7f66e006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.943+0000 7f670128d640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 0x7f66e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:23.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.959+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66fc074340 con 0x7f66fc0b3530 2023-10-14T16:57:23.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.960+0000 7f66f27fc640 1 -- 172.21.15.29:0/1870623485 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66f8017020 con 0x7f66fc0b3530 2023-10-14T16:57:23.961 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:57:23.962 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:23.961+0000 7f3a44a45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 msgr2=0x7f66e006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:23.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 0x7f66e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 msgr2=0x7f66fc155df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f66f800c480 tx=0x7f66f800c940 comp rx=0 tx=0).stop 2023-10-14T16:57:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 shutdown_connections 2023-10-14T16:57:23.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f66e006af90 0x7f66e006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66fc159c00 0x7f66fc0c7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66fc0cad80 0x7f66fc0c7620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 --2- 172.21.15.29:0/1870623485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66fc0b3530 0x7f66fc155df0 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.962+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 >> 172.21.15.29:0/1870623485 conn(0x7f66fc0b0a50 msgr2=0x7f66fc071480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:23.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.963+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 shutdown_connections 2023-10-14T16:57:23.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:23.963+0000 7f6702a90640 1 -- 172.21.15.29:0/1870623485 wait complete. 2023-10-14T16:57:23.983 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:57:24.015 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.014+0000 7f7f3b013040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:24.017 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.017+0000 7f3a44a45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:24.078 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.077+0000 7f7f3b013040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:24.127 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.126+0000 7f7f3b013040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:24.141 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.141+0000 7f3a44a45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:24.184 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.183+0000 7f7f3b013040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.188+0000 7f0947353640 1 Processor -- start 2023-10-14T16:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.188+0000 7f0947353640 1 -- start start 2023-10-14T16:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.190+0000 7f0947353640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f094006a4b0 0x7f094016c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.190+0000 7f0947353640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400caef0 con 0x7f09400caa80 2023-10-14T16:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.190+0000 7f0947353640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09400cb140 con 0x7f09400addb0 2023-10-14T16:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.190+0000 7f0947353640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0940161a50 con 0x7f094006a4b0 2023-10-14T16:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f094006a4b0 0x7f094016c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f094006a4b0 0x7f094016c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41136/0 (socket says 172.21.15.29:41136) 2023-10-14T16:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 -- 172.21.15.29:0/777236516 learned_addr learned my addr 172.21.15.29:0/777236516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 -- 172.21.15.29:0/777236516 >> v1:172.21.15.186:6789/0 conn(0x7f09400addb0 legacy=0x7f09400c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 -- 172.21.15.29:0/777236516 >> v1:172.21.15.29:6789/0 conn(0x7f09400caa80 legacy=0x7f0940069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.191+0000 7f09458f0640 1 -- 172.21.15.29:0/777236516 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f094006b030 con 0x7f094006a4b0 2023-10-14T16:57:24.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.193+0000 7f09458f0640 1 --2- 172.21.15.29:0/777236516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f094006a4b0 0x7f094016c400 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f0938009e50 tx=0x7f093802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab0790284f1ac70 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:24.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.193+0000 7f0937fff640 1 -- 172.21.15.29:0/777236516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f093802cbb0 con 0x7f094006a4b0 2023-10-14T16:57:24.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0937fff640 1 -- 172.21.15.29:0/777236516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f093802cd10 con 0x7f094006a4b0 2023-10-14T16:57:24.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0937fff640 1 -- 172.21.15.29:0/777236516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0938034610 con 0x7f094006a4b0 2023-10-14T16:57:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 -- 172.21.15.29:0/777236516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f094006a4b0 msgr2=0x7f094016c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 --2- 172.21.15.29:0/777236516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f094006a4b0 0x7f094016c400 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7f0938009e50 tx=0x7f093802c010 comp rx=0 tx=0).stop 2023-10-14T16:57:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 -- 172.21.15.29:0/777236516 shutdown_connections 2023-10-14T16:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 --2- 172.21.15.29:0/777236516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f094006a4b0 0x7f094016c400 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 -- 172.21.15.29:0/777236516 >> 172.21.15.29:0/777236516 conn(0x7f09400ce9b0 msgr2=0x7f09400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.194+0000 7f0947353640 1 -- 172.21.15.29:0/777236516 shutdown_connections 2023-10-14T16:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.195+0000 7f0947353640 1 -- 172.21.15.29:0/777236516 wait complete. 2023-10-14T16:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.197+0000 7f0947353640 1 Processor -- start 2023-10-14T16:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.197+0000 7f0947353640 1 -- start start 2023-10-14T16:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f094006a4b0 0x7f09400c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09400caa80 0x7f0940070e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09400c85d0 con 0x7f09400addb0 2023-10-14T16:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09401f6b90 con 0x7f09400caa80 2023-10-14T16:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f0947353640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09401f6e70 con 0x7f094006a4b0 2023-10-14T16:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.198+0000 7f09448ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41152/0 (socket says 172.21.15.29:41152) 2023-10-14T16:57:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 -- 172.21.15.29:0/3350704062 learned_addr learned my addr 172.21.15.29:0/3350704062 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09450ef640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f094006a4b0 0x7f09400c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09458f0640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09400caa80 0x7f0940070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 -- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f094006a4b0 msgr2=0x7f09400c7ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f094006a4b0 0x7f09400c7ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 -- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09400caa80 msgr2=0x7f0940070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09400caa80 0x7f0940070e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 -- 172.21.15.29:0/3350704062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0938009b30 con 0x7f09400addb0 2023-10-14T16:57:24.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.199+0000 7f09448ee640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f093000c400 tx=0x7f093000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.200+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09300052d0 con 0x7f09400addb0 2023-10-14T16:57:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.200+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0930005430 con 0x7f09400addb0 2023-10-14T16:57:24.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.200+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0930003830 con 0x7f09400addb0 2023-10-14T16:57:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.200+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09401f7150 con 0x7f09400addb0 2023-10-14T16:57:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.200+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09401f7930 con 0x7f09400addb0 2023-10-14T16:57:24.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.201+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f094004e160 con 0x7f09400addb0 2023-10-14T16:57:24.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.202+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f0930003990 con 0x7f09400addb0 2023-10-14T16:57:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.203+0000 7f0935ffb640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 0x7f092406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.203+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f0930088f40 con 0x7f09400addb0 2023-10-14T16:57:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.203+0000 7f09450ef640 1 -- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 msgr2=0x7f092406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:24.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.203+0000 7f09450ef640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 0x7f092406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:24.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.207+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0930053cb0 con 0x7f09400addb0 2023-10-14T16:57:24.216 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.216+0000 7f3a44a45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:24.234 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.234+0000 7f7f3b013040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:24.274 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.271+0000 7f3a44a45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:24.332 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.331+0000 7f3a44a45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:24.385 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.384+0000 7f3a44a45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:24.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.398+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f094006c5b0 con 0x7f09400addb0 2023-10-14T16:57:24.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.399+0000 7f0935ffb640 1 -- 172.21.15.29:0/3350704062 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v97) v1 ==== 104+0+11 (secure 0 0 0) 0x7f0930017020 con 0x7f09400addb0 2023-10-14T16:57:24.400 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:57:24.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.402+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 msgr2=0x7f092406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:24.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.402+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 0x7f092406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.402+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 msgr2=0x7f09400c53d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.402+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7f093000c400 tx=0x7f093000c8c0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 shutdown_connections 2023-10-14T16:57:24.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f092406af90 0x7f092406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09400caa80 0x7f0940070e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09400addb0 0x7f09400c53d0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 --2- 172.21.15.29:0/3350704062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f094006a4b0 0x7f09400c7ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 >> 172.21.15.29:0/3350704062 conn(0x7f09400ce9b0 msgr2=0x7f09400cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:24.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 shutdown_connections 2023-10-14T16:57:24.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.403+0000 7f0947353640 1 -- 172.21.15.29:0/3350704062 wait complete. 2023-10-14T16:57:24.429 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:57:24.612 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.611+0000 7f7f3b013040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:24.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.696+0000 7ff12cead640 1 Processor -- start 2023-10-14T16:57:24.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.696+0000 7ff12cead640 1 -- start start 2023-10-14T16:57:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff12cead640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff12815aaa0 0x7ff1280b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff12cead640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff1280706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff12cead640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1280cad00 con 0x7ff1280ca8a0 2023-10-14T16:57:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff12cead640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff128070c10 con 0x7ff12815aaa0 2023-10-14T16:57:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff12cead640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1280b2d00 con 0x7ff128070240 2023-10-14T16:57:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.697+0000 7ff1277fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff12815aaa0 0x7ff1280b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff12815aaa0 0x7ff1280b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41158/0 (socket says 172.21.15.29:41158) 2023-10-14T16:57:24.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 -- 172.21.15.29:0/4004113812 learned_addr learned my addr 172.21.15.29:0/4004113812 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:24.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff127fff640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff1280706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 -- 172.21.15.29:0/4004113812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 msgr2=0x7ff1280706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff1280706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 -- 172.21.15.29:0/4004113812 >> v1:172.21.15.29:6789/0 conn(0x7ff1280ca8a0 legacy=0x7ff1280b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.698+0000 7ff1277fe640 1 -- 172.21.15.29:0/4004113812 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1280b2fe0 con 0x7ff12815aaa0 2023-10-14T16:57:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.699+0000 7ff1277fe640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280b2600 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7ff118009e50 tx=0x7ff11802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff4cb7d40e42b8f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.699+0000 7ff1267fc640 1 -- 172.21.15.29:0/4004113812 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff11802ca70 con 0x7ff12815aaa0 2023-10-14T16:57:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.699+0000 7ff1267fc640 1 -- 172.21.15.29:0/4004113812 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff11802cbd0 con 0x7ff12815aaa0 2023-10-14T16:57:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.699+0000 7ff1267fc640 1 -- 172.21.15.29:0/4004113812 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1180343f0 con 0x7ff12815aaa0 2023-10-14T16:57:24.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 -- 172.21.15.29:0/4004113812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 msgr2=0x7ff1280b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280b2600 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x7ff118009e50 tx=0x7ff11802bed0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 -- 172.21.15.29:0/4004113812 shutdown_connections 2023-10-14T16:57:24.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff1280706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 --2- 172.21.15.29:0/4004113812 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280b2600 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 -- 172.21.15.29:0/4004113812 >> 172.21.15.29:0/4004113812 conn(0x7ff1280cc230 msgr2=0x7ff1280cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 -- 172.21.15.29:0/4004113812 shutdown_connections 2023-10-14T16:57:24.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.700+0000 7ff12cead640 1 -- 172.21.15.29:0/4004113812 wait complete. 2023-10-14T16:57:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.702+0000 7ff12cead640 1 Processor -- start 2023-10-14T16:57:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.702+0000 7ff12cead640 1 -- start start 2023-10-14T16:57:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff128154370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280ca8a0 0x7ff128154a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff128069eb0 con 0x7ff12815aaa0 2023-10-14T16:57:24.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1281548b0 con 0x7ff1280ca8a0 2023-10-14T16:57:24.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.703+0000 7ff12cead640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff128158580 con 0x7ff128070240 2023-10-14T16:57:24.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41162/0 (socket says 172.21.15.29:41162) 2023-10-14T16:57:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 -- 172.21.15.29:0/1476959526 learned_addr learned my addr 172.21.15.29:0/1476959526 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff1277fe640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff128154370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff126ffd640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280ca8a0 0x7ff128154a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 -- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 msgr2=0x7ff128154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff128154370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 -- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280ca8a0 msgr2=0x7ff128154a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280ca8a0 0x7ff128154a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.704+0000 7ff127fff640 1 -- 172.21.15.29:0/1476959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff118009b30 con 0x7ff12815aaa0 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff127fff640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff12000d5d0 tx=0x7ff12000da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1200125f0 con 0x7ff12815aaa0 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff120012770 con 0x7ff12815aaa0 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff128156120 con 0x7ff12815aaa0 2023-10-14T16:57:24.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff120012b20 con 0x7ff12815aaa0 2023-10-14T16:57:24.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.705+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1281569c0 con 0x7ff12815aaa0 2023-10-14T16:57:24.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.706+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff12804e9b0 con 0x7ff12815aaa0 2023-10-14T16:57:24.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.707+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff120012c80 con 0x7ff12815aaa0 2023-10-14T16:57:24.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.708+0000 7ff11ffff640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 0x7ff10806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:24.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.708+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7ff120089d80 con 0x7ff12815aaa0 2023-10-14T16:57:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.709+0000 7ff1277fe640 1 -- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 msgr2=0x7ff10806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.709+0000 7ff1277fe640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 0x7ff10806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:24.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.713+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff120054b20 con 0x7ff12815aaa0 2023-10-14T16:57:24.773 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.772+0000 7f7f3b013040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:24.804 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.804+0000 7f3a44a45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:24.881 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.880+0000 7f7f3b013040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:24.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.900+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff128071580 con 0x7ff12815aaa0 2023-10-14T16:57:24.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.901+0000 7ff11ffff640 1 -- 172.21.15.29:0/1476959526 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v97) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff120058140 con 0x7ff12815aaa0 2023-10-14T16:57:24.903 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:57:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 msgr2=0x7ff10806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:24.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 0x7ff10806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 msgr2=0x7ff1280c8540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:24.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7ff12000d5d0 tx=0x7ff12000da90 comp rx=0 tx=0).stop 2023-10-14T16:57:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 shutdown_connections 2023-10-14T16:57:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7ff10806af90 0x7ff10806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff12815aaa0 0x7ff1280c8540 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1280ca8a0 0x7ff128154a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 --2- 172.21.15.29:0/1476959526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff128070240 0x7ff128154370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:24.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 >> 172.21.15.29:0/1476959526 conn(0x7ff1280cc230 msgr2=0x7ff1280c5fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:24.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 shutdown_connections 2023-10-14T16:57:24.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:24.904+0000 7ff12cead640 1 -- 172.21.15.29:0/1476959526 wait complete. 2023-10-14T16:57:24.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:57:24.930 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:24.930+0000 7f7f3b013040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:24.980 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:24.980+0000 7f3a44a45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:25.038 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:25.037+0000 7f7f3b013040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:25.097 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.096+0000 7f3a44a45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:25.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.132+0000 7fd2dc4d4640 1 Processor -- start 2023-10-14T16:57:25.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.132+0000 7fd2dc4d4640 1 -- start start 2023-10-14T16:57:25.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2dc4d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d030 0x7fd2d40d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2dc4d4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2d40cd260 0x7fd2d40cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2dc4d4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2d40b5d90 con 0x7fd2d415d410 2023-10-14T16:57:25.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2dc4d4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2d40d2b50 con 0x7fd2d40cd260 2023-10-14T16:57:25.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2dc4d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4160fc0 con 0x7fd2d415d030 2023-10-14T16:57:25.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2d9a6f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd2d415d410 0x7fd2d4161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40236/0 (socket says 172.21.15.29:40236) 2023-10-14T16:57:25.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2d9a6f640 1 -- 172.21.15.29:0/4203303368 learned_addr learned my addr 172.21.15.29:0/4203303368 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:25.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2daa71640 1 --2- 172.21.15.29:0/4203303368 >> v2:172.21.15.29:3300/0 conn(0x7fd2d40cd260 0x7fd2d40cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.134+0000 7fd2da270640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d030 0x7fd2d40d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.135+0000 7fd2daa71640 1 -- 172.21.15.29:0/4203303368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d030 msgr2=0x7fd2d40d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:25.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.135+0000 7fd2daa71640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d030 0x7fd2d40d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.135+0000 7fd2daa71640 1 -- 172.21.15.29:0/4203303368 >> v1:172.21.15.29:6789/0 conn(0x7fd2d415d410 legacy=0x7fd2d4161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.135+0000 7fd2daa71640 1 -- 172.21.15.29:0/4203303368 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d40d25d0 con 0x7fd2d40cd260 2023-10-14T16:57:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.136+0000 7fd2daa71640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d40cd260 0x7fd2d40cd6f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009d90 tx=0x7fd2cc02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=444edd8a1a2af48e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2d926e640 1 -- 172.21.15.29:0/4203303368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc02cce0 con 0x7fd2d40cd260 2023-10-14T16:57:25.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2d926e640 1 -- 172.21.15.29:0/4203303368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2cc02ce40 con 0x7fd2d40cd260 2023-10-14T16:57:25.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2d926e640 1 -- 172.21.15.29:0/4203303368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2cc034660 con 0x7fd2d40cd260 2023-10-14T16:57:25.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/4203303368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d40cd260 msgr2=0x7fd2d40cd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d40cd260 0x7fd2d40cd6f0 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7fd2cc009d90 tx=0x7fd2cc02bf30 comp rx=0 tx=0).stop 2023-10-14T16:57:25.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/4203303368 shutdown_connections 2023-10-14T16:57:25.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d40cd260 0x7fd2d40cd6f0 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/4203303368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d030 0x7fd2d40d1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/4203303368 >> 172.21.15.29:0/4203303368 conn(0x7fd2d40cf720 msgr2=0x7fd2d40cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:25.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.137+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/4203303368 shutdown_connections 2023-10-14T16:57:25.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.138+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/4203303368 wait complete. 2023-10-14T16:57:25.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.139+0000 7fd2dc4d4640 1 Processor -- start 2023-10-14T16:57:25.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.139+0000 7fd2dc4d4640 1 -- start start 2023-10-14T16:57:25.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2d40cd260 0x7fd2d41f2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d410 0x7fd2d40c9250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d41f0ca0 con 0x7fd2d415d030 2023-10-14T16:57:25.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d4160290 con 0x7fd2d40cd260 2023-10-14T16:57:25.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2dc4d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d41f3510 con 0x7fd2d415d410 2023-10-14T16:57:25.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2d9a6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2d9a6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41178/0 (socket says 172.21.15.29:41178) 2023-10-14T16:57:25.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.140+0000 7fd2d9a6f640 1 -- 172.21.15.29:0/1275271261 learned_addr learned my addr 172.21.15.29:0/1275271261 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2da270640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2d40cd260 0x7fd2d41f2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2daa71640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d410 0x7fd2d40c9250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d410 msgr2=0x7fd2d40c9250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d410 0x7fd2d40c9250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2d40cd260 msgr2=0x7fd2d41f2ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2d40cd260 0x7fd2d41f2ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 -- 172.21.15.29:0/1275271261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2cc009a70 con 0x7fd2d415d030 2023-10-14T16:57:25.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2d9a6f640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400c480 tx=0x7fd2c400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:25.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.141+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c4005ce0 con 0x7fd2d415d030 2023-10-14T16:57:25.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.142+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2c4005e40 con 0x7fd2d415d030 2023-10-14T16:57:25.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.142+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d41f37f0 con 0x7fd2d415d030 2023-10-14T16:57:25.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.142+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2c40072f0 con 0x7fd2d415d030 2023-10-14T16:57:25.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.142+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d41f3db0 con 0x7fd2d415d030 2023-10-14T16:57:25.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.143+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d40d1ef0 con 0x7fd2d415d030 2023-10-14T16:57:25.149 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:25.144+0000 7f7f3b013040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:25.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.144+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd2c4007450 con 0x7fd2d415d030 2023-10-14T16:57:25.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.145+0000 7fd2caffd640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 0x7fd2b806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.145+0000 7fd2da270640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 msgr2=0x7fd2b806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:25.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.146+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fd2c4088d20 con 0x7fd2d415d030 2023-10-14T16:57:25.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.146+0000 7fd2da270640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 0x7fd2b806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:25.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.149+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2c4053c70 con 0x7fd2d415d030 2023-10-14T16:57:25.155 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.155+0000 7f3a44a45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:25.286 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.285+0000 7f3a44a45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:25.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:25.302+0000 7f7f3b013040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:25.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.347+0000 7fd2da270640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 msgr2=0x7fd2b806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:25.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.347+0000 7fd2da270640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 0x7fd2b806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:25.352 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:25.352+0000 7f7f3b013040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:25.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.380+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fd2d40d3df0 con 0x7fd2d415d030 2023-10-14T16:57:25.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.380+0000 7fd2caffd640 1 -- 172.21.15.29:0/1275271261 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v97) v1 ==== 80+0+470 (secure 0 0 0) 0x7fd2c4017020 con 0x7fd2d415d030 2023-10-14T16:57:25.381 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3356363773 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.382 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/3330033844 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.382 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/3330033844 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.382 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3967724404 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.382 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2171619166 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.382 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3505521487 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3693879118 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.383 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4176788283 2023-10-15T16:57:20.630995+0000 2023-10-14T16:57:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T16:57:25.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 msgr2=0x7fd2b806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 0x7fd2b806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 msgr2=0x7fd2d415fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fd2c400c480 tx=0x7fd2c400c940 comp rx=0 tx=0).stop 2023-10-14T16:57:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 shutdown_connections 2023-10-14T16:57:25.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7fd2b806afc0 0x7fd2b806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2d415d410 0x7fd2d40c9250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2d415d030 0x7fd2d415fd50 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 --2- 172.21.15.29:0/1275271261 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2d40cd260 0x7fd2d41f2ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 >> 172.21.15.29:0/1275271261 conn(0x7fd2d40cf720 msgr2=0x7fd2d415c160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:25.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 shutdown_connections 2023-10-14T16:57:25.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.383+0000 7fd2dc4d4640 1 -- 172.21.15.29:0/1275271261 wait complete. 2023-10-14T16:57:25.401 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:57:25.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:25.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:25.403 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.400+0000 7f3a44a45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:25.471 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:25.469+0000 7f7f3b013040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:25.510 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:57:25.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:25.511 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:57:25.560 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:57:25.561 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:25.575 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.575+0000 7f3a44a45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:25.618 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:57:25.618 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:25.619 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:25.628 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.627+0000 7f3a44a45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:25.676 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:57:25.676 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:57:25.677 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:25.730 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:57:25.731 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:25.755 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:25.754+0000 7f3a44a45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:25.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.948+0000 7f9d25d0a640 1 Processor -- start 2023-10-14T16:57:25.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.948+0000 7f9d25d0a640 1 -- start start 2023-10-14T16:57:25.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.949+0000 7f9d25d0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d2015cd70 0x7f9d200d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25d0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d200b8480 0x7f9d200cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25d0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d2015d150 con 0x7f9d200d0be0 2023-10-14T16:57:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25d0a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d200d1180 con 0x7f9d2015cd70 2023-10-14T16:57:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25d0a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200d1460 con 0x7f9d200b8480 2023-10-14T16:57:25.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25509640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9d200d0be0 0x7f9d200b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40250/0 (socket says 172.21.15.29:40250) 2023-10-14T16:57:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d25509640 1 -- 172.21.15.29:0/1014946901 learned_addr learned my addr 172.21.15.29:0/1014946901 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1014946901 >> v2:172.21.15.29:3300/0 conn(0x7f9d2015cd70 0x7f9d200d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.950+0000 7f9d17fff640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d200b8480 0x7f9d200cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.951+0000 7f9d24d08640 1 -- 172.21.15.29:0/1014946901 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d200b8480 msgr2=0x7f9d200cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.951+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d200b8480 0x7f9d200cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.951+0000 7f9d24d08640 1 -- 172.21.15.29:0/1014946901 >> v1:172.21.15.29:6789/0 conn(0x7f9d200d0be0 legacy=0x7f9d200b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.951+0000 7f9d24d08640 1 -- 172.21.15.29:0/1014946901 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d200d1740 con 0x7f9d2015cd70 2023-10-14T16:57:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.952+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d2015cd70 0x7f9d200d2220 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0097a0 tx=0x7f9d1c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6848eafb7586959 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.952+0000 7f9d177fe640 1 -- 172.21.15.29:0/1014946901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c02d040 con 0x7f9d2015cd70 2023-10-14T16:57:25.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.952+0000 7f9d177fe640 1 -- 172.21.15.29:0/1014946901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c031810 con 0x7f9d2015cd70 2023-10-14T16:57:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.952+0000 7f9d177fe640 1 -- 172.21.15.29:0/1014946901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c031ba0 con 0x7f9d2015cd70 2023-10-14T16:57:25.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1014946901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d2015cd70 msgr2=0x7f9d200d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d2015cd70 0x7f9d200d2220 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c0097a0 tx=0x7f9d1c02b8b0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1014946901 shutdown_connections 2023-10-14T16:57:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d200b8480 0x7f9d200cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1014946901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d2015cd70 0x7f9d200d2220 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1014946901 >> 172.21.15.29:0/1014946901 conn(0x7f9d200cf570 msgr2=0x7f9d200cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1014946901 shutdown_connections 2023-10-14T16:57:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.953+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1014946901 wait complete. 2023-10-14T16:57:25.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.955+0000 7f9d25d0a640 1 Processor -- start 2023-10-14T16:57:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.955+0000 7f9d25d0a640 1 -- start start 2023-10-14T16:57:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d200b8480 0x7f9d200b4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d2015cd70 0x7f9d2015d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d20142a00 con 0x7f9d200d0be0 2023-10-14T16:57:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200b53a0 con 0x7f9d200b8480 2023-10-14T16:57:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25d0a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d200c8b30 con 0x7f9d2015cd70 2023-10-14T16:57:25.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41192/0 (socket says 172.21.15.29:41192) 2023-10-14T16:57:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 -- 172.21.15.29:0/1256068068 learned_addr learned my addr 172.21.15.29:0/1256068068 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d25509640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d2015cd70 0x7f9d2015d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d200b8480 0x7f9d200b4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:25.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d2015cd70 msgr2=0x7f9d2015d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d2015cd70 0x7f9d2015d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d200b8480 msgr2=0x7f9d200b4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d200b8480 0x7f9d200b4e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.957+0000 7f9d17fff640 1 -- 172.21.15.29:0/1256068068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c009400 con 0x7f9d200d0be0 2023-10-14T16:57:25.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d17fff640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000e870 tx=0x7f9d1000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d100094f0 con 0x7f9d200d0be0 2023-10-14T16:57:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1000c940 con 0x7f9d200d0be0 2023-10-14T16:57:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d200c8e10 con 0x7f9d200d0be0 2023-10-14T16:57:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1000ccf0 con 0x7f9d200d0be0 2023-10-14T16:57:25.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.958+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d200c93d0 con 0x7f9d200d0be0 2023-10-14T16:57:25.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.959+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d20142b70 con 0x7f9d200d0be0 2023-10-14T16:57:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.960+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f9d10013470 con 0x7f9d200d0be0 2023-10-14T16:57:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.961+0000 7f9d157fa640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 0x7f9d0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.961+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f9d10089ae0 con 0x7f9d200d0be0 2023-10-14T16:57:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.961+0000 7f9d24d08640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 msgr2=0x7f9d0c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:25.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.961+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 0x7f9d0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:25.966+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d20142b70 con 0x7f9d200d0be0 2023-10-14T16:57:26.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.162+0000 7f9d24d08640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 msgr2=0x7f9d0c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:26.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.162+0000 7f9d24d08640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 0x7f9d0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:26.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.197+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d20142b40 con 0x7f9d200d0be0 2023-10-14T16:57:26.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.198+0000 7f9d157fa640 1 -- 172.21.15.29:0/1256068068 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f9d10017090 con 0x7f9d200d0be0 2023-10-14T16:57:26.199 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:26.199 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:26.199 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 msgr2=0x7f9d0c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 0x7f9d0c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 msgr2=0x7f9d2015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7f9d1000e870 tx=0x7f9d1000ed30 comp rx=0 tx=0).stop 2023-10-14T16:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 shutdown_connections 2023-10-14T16:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f9d0c06af90 0x7f9d0c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d2015cd70 0x7f9d2015d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d200d0be0 0x7f9d2015d340 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 --2- 172.21.15.29:0/1256068068 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d200b8480 0x7f9d200b4e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 >> 172.21.15.29:0/1256068068 conn(0x7f9d200cf570 msgr2=0x7f9d200b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 shutdown_connections 2023-10-14T16:57:26.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.201+0000 7f9d25d0a640 1 -- 172.21.15.29:0/1256068068 wait complete. 2023-10-14T16:57:26.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:26.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:26.302+0000 7f7f3b013040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:26.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.468+0000 7f07c7f6c640 1 Processor -- start 2023-10-14T16:57:26.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.469+0000 7f07c7f6c640 1 -- start start 2023-10-14T16:57:26.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.470+0000 7f07c7f6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07c00cd0c0 0x7f07c01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.470+0000 7f07c7f6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c015db00 0x7f07c00b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.470+0000 7f07c7f6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07c015e3e0 con 0x7f07c00b0410 2023-10-14T16:57:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.470+0000 7f07c7f6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07c00b0de0 con 0x7f07c00cd0c0 2023-10-14T16:57:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.470+0000 7f07c7f6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00cd4a0 con 0x7f07c015db00 2023-10-14T16:57:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07c00cd0c0 0x7f07c01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07c00cd0c0 0x7f07c01615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41206/0 (socket says 172.21.15.29:41206) 2023-10-14T16:57:26.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 -- 172.21.15.29:0/3535960866 learned_addr learned my addr 172.21.15.29:0/3535960866 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c6509640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c015db00 0x7f07c00b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 -- 172.21.15.29:0/3535960866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c015db00 msgr2=0x7f07c00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c015db00 0x7f07c00b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 -- 172.21.15.29:0/3535960866 >> v1:172.21.15.29:6789/0 conn(0x7f07c00b0410 legacy=0x7f07c015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.471+0000 7f07c5d08640 1 -- 172.21.15.29:0/3535960866 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07c015e100 con 0x7f07c00cd0c0 2023-10-14T16:57:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.472+0000 7f07c5d08640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c00cd0c0 0x7f07c01615f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e50 tx=0x7f07bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af629d322916a13 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.473+0000 7f07c4d06640 1 -- 172.21.15.29:0/3535960866 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc02ca70 con 0x7f07c00cd0c0 2023-10-14T16:57:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.473+0000 7f07c4d06640 1 -- 172.21.15.29:0/3535960866 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07bc02cbd0 con 0x7f07c00cd0c0 2023-10-14T16:57:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.473+0000 7f07c4d06640 1 -- 172.21.15.29:0/3535960866 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07bc0343f0 con 0x7f07c00cd0c0 2023-10-14T16:57:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.473+0000 7f07c7f6c640 1 -- 172.21.15.29:0/3535960866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c00cd0c0 msgr2=0x7f07c01615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.473+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c00cd0c0 0x7f07c01615f0 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7f07bc009e50 tx=0x7f07bc02bed0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 -- 172.21.15.29:0/3535960866 shutdown_connections 2023-10-14T16:57:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c015db00 0x7f07c00b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/3535960866 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c00cd0c0 0x7f07c01615f0 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 -- 172.21.15.29:0/3535960866 >> 172.21.15.29:0/3535960866 conn(0x7f07c00cb410 msgr2=0x7f07c00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:26.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 -- 172.21.15.29:0/3535960866 shutdown_connections 2023-10-14T16:57:26.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.474+0000 7f07c7f6c640 1 -- 172.21.15.29:0/3535960866 wait complete. 2023-10-14T16:57:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.475+0000 7f07c7f6c640 1 Processor -- start 2023-10-14T16:57:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.476+0000 7f07c7f6c640 1 -- start start 2023-10-14T16:57:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.476+0000 7f07c7f6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c00b0410 0x7f07c00b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.476+0000 7f07c7f6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c00cd0c0 0x7f07c00c5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c7f6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c7f6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00c71b0 con 0x7f07c015db00 2023-10-14T16:57:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c7f6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00c65a0 con 0x7f07c00cd0c0 2023-10-14T16:57:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c7f6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07c00c6880 con 0x7f07c00b0410 2023-10-14T16:57:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41212/0 (socket says 172.21.15.29:41212) 2023-10-14T16:57:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 -- 172.21.15.29:0/860147565 learned_addr learned my addr 172.21.15.29:0/860147565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c5507640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c00cd0c0 0x7f07c00c5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c5d08640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c00b0410 0x7f07c00b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c00b0410 msgr2=0x7f07c00b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c00b0410 0x7f07c00b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c00cd0c0 msgr2=0x7f07c00c5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c00cd0c0 0x7f07c00c5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.477+0000 7f07c6509640 1 -- 172.21.15.29:0/860147565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07bc009b30 con 0x7f07c015db00 2023-10-14T16:57:26.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07c6509640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f07b800d5d0 tx=0x7f07b800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:26.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b80125f0 con 0x7f07c015db00 2023-10-14T16:57:26.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07b8012770 con 0x7f07c015db00 2023-10-14T16:57:26.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07c00c6b60 con 0x7f07c015db00 2023-10-14T16:57:26.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07b8012b20 con 0x7f07c015db00 2023-10-14T16:57:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.478+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07c00c4ec0 con 0x7f07c015db00 2023-10-14T16:57:26.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.480+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07c00c7360 con 0x7f07c015db00 2023-10-14T16:57:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.481+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f07b8012c80 con 0x7f07c015db00 2023-10-14T16:57:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.483+0000 7f07b67fc640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 0x7f07a406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.483+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f07b8089e20 con 0x7f07c015db00 2023-10-14T16:57:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.483+0000 7f07c5d08640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 msgr2=0x7f07a406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:26.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.483+0000 7f07c5d08640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 0x7f07a406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:26.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.488+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07c00c7360 con 0x7f07c015db00 2023-10-14T16:57:26.597 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:26.597+0000 7f7f3b013040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:26.637 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:26.636+0000 7f3a44a45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:26.653 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:26.653+0000 7f7f3b013040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:26.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.682+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f07c0160df0 con 0x7f07c015db00 2023-10-14T16:57:26.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.684+0000 7f07c5d08640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 msgr2=0x7f07a406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:26.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.684+0000 7f07c5d08640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 0x7f07a406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:26.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.685+0000 7f07b67fc640 1 -- 172.21.15.29:0/860147565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+17127 (secure 0 0 0) 0x7f07b8054ba0 con 0x7f07c015db00 2023-10-14T16:57:26.686 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:26.686 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":97,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:20.631064+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3356363773":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6833/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6832/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3967724404":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2171619166":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3505521487":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3693879118":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/4176788283":"2023-10-15T16:57:20.630995+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 msgr2=0x7f07a406d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 0x7f07a406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 msgr2=0x7f07c00c4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7f07b800d5d0 tx=0x7f07b800da90 comp rx=0 tx=0).stop 2023-10-14T16:57:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 shutdown_connections 2023-10-14T16:57:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f07a406afc0 0x7f07a406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07c015db00 0x7f07c00c4980 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07c00cd0c0 0x7f07c00c5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 --2- 172.21.15.29:0/860147565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07c00b0410 0x7f07c00b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 >> 172.21.15.29:0/860147565 conn(0x7f07c00cb410 msgr2=0x7f07c015bf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.687+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 shutdown_connections 2023-10-14T16:57:26.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.688+0000 7f07c7f6c640 1 -- 172.21.15.29:0/860147565 wait complete. 2023-10-14T16:57:26.707 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 10, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:57:03.662290+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '95', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 11, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:57:04.248899+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '96', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:57:26.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:57:26.767 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:26.767+0000 7f7f3b013040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.921+0000 7f46850ec640 1 Processor -- start 2023-10-14T16:57:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.921+0000 7f46850ec640 1 -- start start 2023-10-14T16:57:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46850ec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46800b2080 0x7f46800d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46850ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46850ec640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46800d0970 con 0x7f46800b1ca0 2023-10-14T16:57:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46850ec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46800d0ea0 con 0x7f46800b2080 2023-10-14T16:57:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46850ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46800d0bc0 con 0x7f46800cfc40 2023-10-14T16:57:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.923+0000 7f46848eb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:55084/0 (socket says 172.21.15.29:55084) 2023-10-14T16:57:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 -- 172.21.15.29:0/4207348971 learned_addr learned my addr 172.21.15.29:0/4207348971 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f4677fff640 1 --2- 172.21.15.29:0/4207348971 >> v2:172.21.15.186:3300/0 conn(0x7f46800b2080 0x7f46800d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 -- 172.21.15.29:0/4207348971 >> v2:172.21.15.186:3300/0 conn(0x7f46800b2080 msgr2=0x7f46800d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 --2- 172.21.15.29:0/4207348971 >> v2:172.21.15.186:3300/0 conn(0x7f46800b2080 0x7f46800d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 -- 172.21.15.29:0/4207348971 >> v1:172.21.15.186:6789/0 conn(0x7f46800b1ca0 legacy=0x7f46800ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f46848eb640 1 -- 172.21.15.29:0/4207348971 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46800cd940 con 0x7f46800cfc40 2023-10-14T16:57:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.924+0000 7f467f7fe640 1 -- 172.21.15.29:0/4207348971 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825064271 0 0) 0x7f46800d0970 con 0x7f46800b1ca0 2023-10-14T16:57:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.926+0000 7f46848eb640 1 --2- 172.21.15.29:0/4207348971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e50 tx=0x7f467802bff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=48c1c1a52e1ab9ec server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.926+0000 7f467f7fe640 1 -- 172.21.15.29:0/4207348971 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f467802cb90 con 0x7f46800cfc40 2023-10-14T16:57:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.926+0000 7f467f7fe640 1 -- 172.21.15.29:0/4207348971 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f467802ccf0 con 0x7f46800cfc40 2023-10-14T16:57:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.926+0000 7f467f7fe640 1 -- 172.21.15.29:0/4207348971 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4678034610 con 0x7f46800cfc40 2023-10-14T16:57:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 -- 172.21.15.29:0/4207348971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 msgr2=0x7f46800b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 --2- 172.21.15.29:0/4207348971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f4678009e50 tx=0x7f467802bff0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 -- 172.21.15.29:0/4207348971 shutdown_connections 2023-10-14T16:57:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 --2- 172.21.15.29:0/4207348971 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800cfc40 0x7f46800b24c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 --2- 172.21.15.29:0/4207348971 >> v2:172.21.15.186:3300/0 conn(0x7f46800b2080 0x7f46800d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 -- 172.21.15.29:0/4207348971 >> 172.21.15.29:0/4207348971 conn(0x7f46800d2b90 msgr2=0x7f46800d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 -- 172.21.15.29:0/4207348971 shutdown_connections 2023-10-14T16:57:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.927+0000 7f46850ec640 1 -- 172.21.15.29:0/4207348971 wait complete. 2023-10-14T16:57:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.929+0000 7f46850ec640 1 Processor -- start 2023-10-14T16:57:26.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.929+0000 7f46850ec640 1 -- start start 2023-10-14T16:57:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800b1ca0 0x7f46800d13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 0x7f46801f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468009d380 con 0x7f46800b2080 2023-10-14T16:57:26.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46800c7710 con 0x7f46800cfc40 2023-10-14T16:57:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f46850ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468009d640 con 0x7f46800b1ca0 2023-10-14T16:57:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f4677fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f4677fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41228/0 (socket says 172.21.15.29:41228) 2023-10-14T16:57:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f4677fff640 1 -- 172.21.15.29:0/3285991980 learned_addr learned my addr 172.21.15.29:0/3285991980 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:26.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.930+0000 7f467ffff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800b1ca0 0x7f46800d13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f46848eb640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 0x7f46801f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800b1ca0 msgr2=0x7f46800d13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800b1ca0 0x7f46800d13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 msgr2=0x7f46801f5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:26.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 0x7f46801f5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 -- 172.21.15.29:0/3285991980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4678009b30 con 0x7f46800b2080 2023-10-14T16:57:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f46848eb640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 0x7f46801f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:26.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f4677fff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f466800e870 tx=0x7f466800ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46680094f0 con 0x7f46800b2080 2023-10-14T16:57:26.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f466800c940 con 0x7f46800b2080 2023-10-14T16:57:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f466800ccf0 con 0x7f46800b2080 2023-10-14T16:57:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.931+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468009d920 con 0x7f46800b2080 2023-10-14T16:57:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.932+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46800d1920 con 0x7f46800b2080 2023-10-14T16:57:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.933+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46800c7880 con 0x7f46800b2080 2023-10-14T16:57:26.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.934+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 41) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f4668013470 con 0x7f46800b2080 2023-10-14T16:57:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.934+0000 7f467d7fa640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 0x7f466006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.935+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f4668089a50 con 0x7f46800b2080 2023-10-14T16:57:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.935+0000 7f467ffff640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 msgr2=0x7f466006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:26.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.935+0000 7f467ffff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 0x7f466006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:26.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:26.939+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4668054850 con 0x7f46800b2080 2023-10-14T16:57:26.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:26.977+0000 7f3a44a45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:27.038 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.037+0000 7f3a44a45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:27.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.136+0000 7f467ffff640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 msgr2=0x7f466006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/505146228 2023-10-14T16:57:27.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.136+0000 7f467ffff640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 0x7f466006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:27.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.151+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f468004e9b0 con 0x7f46800b2080 2023-10-14T16:57:27.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.152+0000 7f467d7fa640 1 -- 172.21.15.29:0/3285991980 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v97) v1 ==== 93+0+10 (secure 0 0 0) 0x7f46680170a0 con 0x7f46800b2080 2023-10-14T16:57:27.153 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:57:27.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 msgr2=0x7f466006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:27.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 0x7f466006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 msgr2=0x7f46801f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7f466800e870 tx=0x7f466800ed30 comp rx=0 tx=0).stop 2023-10-14T16:57:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 shutdown_connections 2023-10-14T16:57:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:6832/505146228,v1:172.21.15.29:6833/505146228] conn(0x7f466006af90 0x7f466006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46800cfc40 0x7f46801f5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46800b2080 0x7f46801f5830 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 --2- 172.21.15.29:0/3285991980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46800b1ca0 0x7f46800d13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.154+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 >> 172.21.15.29:0/3285991980 conn(0x7f46800d2b90 msgr2=0x7f468015fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.155+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 shutdown_connections 2023-10-14T16:57:27.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.155+0000 7f46850ec640 1 -- 172.21.15.29:0/3285991980 wait complete. 2023-10-14T16:57:27.159 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.158+0000 7f3a44a45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:27.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:57:27.247 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c9d1f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.247 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c9d1f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.247 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c9d1f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c9d1f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c9d1f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c5516640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c5516640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.249 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c5516640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.249 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c5516640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.249 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:27.246+0000 7f39c5516640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:57:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.372+0000 7f41886bb640 1 Processor -- start 2023-10-14T16:57:27.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.372+0000 7f41886bb640 1 -- start start 2023-10-14T16:57:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f41886bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 0x7f41800ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f41886bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41800c9ca0 0x7f41800ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f41886bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41800cb030 con 0x7f418015ad70 2023-10-14T16:57:27.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f41886bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41800cae30 con 0x7f41800c9ca0 2023-10-14T16:57:27.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f41886bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41800c91c0 con 0x7f418006a4b0 2023-10-14T16:57:27.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f4186c58640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41800c9ca0 0x7f41800ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f4185c56640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 0x7f41800ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f4186c58640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41800c9ca0 0x7f41800ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:60582/0 (socket says 172.21.15.29:60582) 2023-10-14T16:57:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.374+0000 7f4186c58640 1 -- 172.21.15.29:0/2886424721 learned_addr learned my addr 172.21.15.29:0/2886424721 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.375+0000 7f4186c58640 1 -- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 msgr2=0x7f41800ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.375+0000 7f4186c58640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 0x7f41800ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.375+0000 7f4186c58640 1 -- 172.21.15.29:0/2886424721 >> v1:172.21.15.186:6789/0 conn(0x7f418015ad70 legacy=0x7f4180069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.375+0000 7f4186c58640 1 -- 172.21.15.29:0/2886424721 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41800c94a0 con 0x7f41800c9ca0 2023-10-14T16:57:27.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.375+0000 7f4185c56640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 0x7f41800ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:57:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.377+0000 7f4186c58640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800ca8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4178009e50 tx=0x7f417802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e8a578c3ecaa581 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.377+0000 7f4185455640 1 -- 172.21.15.29:0/2886424721 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417802cb00 con 0x7f41800c9ca0 2023-10-14T16:57:27.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.377+0000 7f4185455640 1 -- 172.21.15.29:0/2886424721 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417802cc60 con 0x7f41800c9ca0 2023-10-14T16:57:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.377+0000 7f4185455640 1 -- 172.21.15.29:0/2886424721 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41780343f0 con 0x7f41800c9ca0 2023-10-14T16:57:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 -- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 msgr2=0x7f41800ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800ca8f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f4178009e50 tx=0x7f417802bf60 comp rx=0 tx=0).stop 2023-10-14T16:57:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 -- 172.21.15.29:0/2886424721 shutdown_connections 2023-10-14T16:57:27.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418006a4b0 0x7f41800ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 --2- 172.21.15.29:0/2886424721 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800ca8f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 -- 172.21.15.29:0/2886424721 >> 172.21.15.29:0/2886424721 conn(0x7f41800ce9b0 msgr2=0x7f41800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:27.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 -- 172.21.15.29:0/2886424721 shutdown_connections 2023-10-14T16:57:27.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.378+0000 7f41886bb640 1 -- 172.21.15.29:0/2886424721 wait complete. 2023-10-14T16:57:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.380+0000 7f41886bb640 1 Processor -- start 2023-10-14T16:57:27.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.380+0000 7f41886bb640 1 -- start start 2023-10-14T16:57:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800aa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418015ad70 0x7f41800b30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41800a8950 con 0x7f418006a4b0 2023-10-14T16:57:27.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41800a85b0 con 0x7f41800c9ca0 2023-10-14T16:57:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f41886bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41800aab00 con 0x7f418015ad70 2023-10-14T16:57:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f4186457640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f4186457640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41310/0 (socket says 172.21.15.29:41310) 2023-10-14T16:57:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.381+0000 7f4186457640 1 -- 172.21.15.29:0/4276188301 learned_addr learned my addr 172.21.15.29:0/4276188301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:27.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4185c56640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186c58640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418015ad70 0x7f41800b30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 -- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418015ad70 msgr2=0x7f41800b30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418015ad70 0x7f41800b30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 -- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 msgr2=0x7f41800aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 -- 172.21.15.29:0/4276188301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4170009400 con 0x7f418006a4b0 2023-10-14T16:57:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4186457640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c400 tx=0x7f417c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c0052d0 con 0x7f418006a4b0 2023-10-14T16:57:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f417c005430 con 0x7f418006a4b0 2023-10-14T16:57:27.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.382+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4178009b30 con 0x7f418006a4b0 2023-10-14T16:57:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.383+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f417c003830 con 0x7f418006a4b0 2023-10-14T16:57:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.383+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41800ab0c0 con 0x7f418006a4b0 2023-10-14T16:57:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.384+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f418004e160 con 0x7f418006a4b0 2023-10-14T16:57:27.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.385+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 42) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f417c003990 con 0x7f418006a4b0 2023-10-14T16:57:27.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.386+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f417c088900 con 0x7f418006a4b0 2023-10-14T16:57:27.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.391+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f417c053b20 con 0x7f418006a4b0 2023-10-14T16:57:27.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.588+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f41800700e0 con 0x7f418006a4b0 2023-10-14T16:57:27.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.589+0000 7f4176ffd640 1 -- 172.21.15.29:0/4276188301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) v1 ==== 104+0+11 (secure 0 0 0) 0x7f417c057140 con 0x7f418006a4b0 2023-10-14T16:57:27.590 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:57:27.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 msgr2=0x7f41801f1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f417c00c400 tx=0x7f417c00c8c0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 shutdown_connections 2023-10-14T16:57:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f418015ad70 0x7f41800b30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41800c9ca0 0x7f41800aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 --2- 172.21.15.29:0/4276188301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f418006a4b0 0x7f41801f1480 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.592+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 >> 172.21.15.29:0/4276188301 conn(0x7f41800ce9b0 msgr2=0x7f41800ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.593+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 shutdown_connections 2023-10-14T16:57:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.593+0000 7f41886bb640 1 -- 172.21.15.29:0/4276188301 wait complete. 2023-10-14T16:57:27.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:57:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.843+0000 7f9ac04e5640 1 Processor -- start 2023-10-14T16:57:27.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.843+0000 7f9ac04e5640 1 -- start start 2023-10-14T16:57:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9ac04e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9ac04e5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab80c9ca0 0x7f9ab80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9ac04e5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab80cb030 con 0x7f9ab815ad70 2023-10-14T16:57:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9ac04e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ab80cae30 con 0x7f9ab806a4b0 2023-10-14T16:57:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9ac04e5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80c91c0 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41318/0 (socket says 172.21.15.29:41318) 2023-10-14T16:57:27.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 -- 172.21.15.29:0/2290446454 learned_addr learned my addr 172.21.15.29:0/2290446454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:27.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abea82640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab80c9ca0 0x7f9ab80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 -- 172.21.15.29:0/2290446454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab80c9ca0 msgr2=0x7f9ab80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab80c9ca0 0x7f9ab80ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 -- 172.21.15.29:0/2290446454 >> v1:172.21.15.186:6789/0 conn(0x7f9ab815ad70 legacy=0x7f9ab8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:27.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.845+0000 7f9abda80640 1 -- 172.21.15.29:0/2290446454 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab80c94a0 con 0x7f9ab806a4b0 2023-10-14T16:57:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9abda80640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd86453a9e4055b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9abd27f640 1 -- 172.21.15.29:0/2290446454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8031640 con 0x7f9ab806a4b0 2023-10-14T16:57:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9abd27f640 1 -- 172.21.15.29:0/2290446454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa80317a0 con 0x7f9ab806a4b0 2023-10-14T16:57:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9abd27f640 1 -- 172.21.15.29:0/2290446454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8031b30 con 0x7f9ab806a4b0 2023-10-14T16:57:27.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2290446454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab806a4b0 msgr2=0x7f9ab80ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.847+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa802b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2290446454 shutdown_connections 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab80c9ca0 0x7f9ab80ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2290446454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ca3b0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2290446454 >> 172.21.15.29:0/2290446454 conn(0x7f9ab80ce9b0 msgr2=0x7f9ab80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:27.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2290446454 shutdown_connections 2023-10-14T16:57:27.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.848+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2290446454 wait complete. 2023-10-14T16:57:27.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.849+0000 7f9ac04e5640 1 Processor -- start 2023-10-14T16:57:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 -- start start 2023-10-14T16:57:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ade30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab815ad70 0x7f9ab80c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:27.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab81595d0 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80ad9a0 con 0x7f9ab806a4b0 2023-10-14T16:57:27.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.850+0000 7f9ac04e5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ab80acd90 con 0x7f9ab815ad70 2023-10-14T16:57:27.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41324/0 (socket says 172.21.15.29:41324) 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 -- 172.21.15.29:0/2720146331 learned_addr learned my addr 172.21.15.29:0/2720146331 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abe281640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ade30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 -- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab815ad70 msgr2=0x7f9ab80c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab815ad70 0x7f9ab80c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 -- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806a4b0 msgr2=0x7f9ab80ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:27.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ade30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:27.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.851+0000 7f9abda80640 1 -- 172.21.15.29:0/2720146331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa80094d0 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9abda80640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:27.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa803ac60 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa803ade0 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa8044630 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab80ad070 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.852+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab80ad630 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.853+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab80adb30 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.854+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 42) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9aa802f050 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.855+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f9aa80ab610 con 0x7f9ab80c9ca0 2023-10-14T16:57:27.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:27.860+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9aa8076740 con 0x7f9ab80c9ca0 2023-10-14T16:57:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.058+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9ab804e160 con 0x7f9ab80c9ca0 2023-10-14T16:57:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.059+0000 7f9aaeffd640 1 -- 172.21.15.29:0/2720146331 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9aa8079d60 con 0x7f9ab80c9ca0 2023-10-14T16:57:28.060 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:57:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 msgr2=0x7f9ab80af0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f9aa8009870 tx=0x7f9aa80046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 shutdown_connections 2023-10-14T16:57:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ab815ad70 0x7f9ab80c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ab80c9ca0 0x7f9ab80af0f0 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 --2- 172.21.15.29:0/2720146331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ab806a4b0 0x7f9ab80ade30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 >> 172.21.15.29:0/2720146331 conn(0x7f9ab80ce9b0 msgr2=0x7f9ab80cdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 shutdown_connections 2023-10-14T16:57:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.062+0000 7f9ac04e5640 1 -- 172.21.15.29:0/2720146331 wait complete. 2023-10-14T16:57:28.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:28.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.311+0000 7f5bd3ee0640 1 Processor -- start 2023-10-14T16:57:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.311+0000 7f5bd3ee0640 1 -- start start 2023-10-14T16:57:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.312+0000 7f5bd3ee0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bcc15ae80 0x7f5bcc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.312+0000 7f5bd3ee0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc0ce390 0x7f5bcc070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.312+0000 7f5bd3ee0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bcc15ed60 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.312+0000 7f5bd3ee0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bcc06b810 con 0x7f5bcc15ae80 2023-10-14T16:57:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.312+0000 7f5bd3ee0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0707e0 con 0x7f5bcc0ce390 2023-10-14T16:57:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bcc15ae80 0x7f5bcc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bcc15ae80 0x7f5bcc15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41338/0 (socket says 172.21.15.29:41338) 2023-10-14T16:57:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 -- 172.21.15.29:0/1498334906 learned_addr learned my addr 172.21.15.29:0/1498334906 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd247d640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc0ce390 0x7f5bcc070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 -- 172.21.15.29:0/1498334906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc0ce390 msgr2=0x7f5bcc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:28.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc0ce390 0x7f5bcc070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.313+0000 7f5bd147b640 1 -- 172.21.15.29:0/1498334906 >> v1:172.21.15.186:6789/0 conn(0x7f5bcc15aaa0 legacy=0x7f5bcc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.314+0000 7f5bd147b640 1 -- 172.21.15.29:0/1498334906 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bcc15ef90 con 0x7f5bcc15ae80 2023-10-14T16:57:28.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.315+0000 7f5bd147b640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15ae80 0x7f5bcc15e820 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d291ab6368d2ddbe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.315+0000 7f5bd0c7a640 1 -- 172.21.15.29:0/1498334906 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc030840 con 0x7f5bcc15ae80 2023-10-14T16:57:28.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd0c7a640 1 -- 172.21.15.29:0/1498334906 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0309a0 con 0x7f5bcc15ae80 2023-10-14T16:57:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd0c7a640 1 -- 172.21.15.29:0/1498334906 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc030d30 con 0x7f5bcc15ae80 2023-10-14T16:57:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/1498334906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15ae80 msgr2=0x7f5bcc15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15ae80 0x7f5bcc15e820 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc009870 tx=0x7f5bbc02b980 comp rx=0 tx=0).stop 2023-10-14T16:57:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/1498334906 shutdown_connections 2023-10-14T16:57:28.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc0ce390 0x7f5bcc070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/1498334906 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15ae80 0x7f5bcc15e820 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/1498334906 >> 172.21.15.29:0/1498334906 conn(0x7f5bcc0cb410 msgr2=0x7f5bcc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/1498334906 shutdown_connections 2023-10-14T16:57:28.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.316+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/1498334906 wait complete. 2023-10-14T16:57:28.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.318+0000 7f5bd3ee0640 1 Processor -- start 2023-10-14T16:57:28.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.318+0000 7f5bd3ee0640 1 -- start start 2023-10-14T16:57:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bcc0ce390 0x7f5bcc1560a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc15ae80 0x7f5bcc06cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0c2c90 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0b0670 con 0x7f5bcc0ce390 2023-10-14T16:57:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd3ee0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bcc0c2640 con 0x7f5bcc15ae80 2023-10-14T16:57:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.319+0000 7f5bd147b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41350/0 (socket says 172.21.15.29:41350) 2023-10-14T16:57:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 -- 172.21.15.29:0/3395870954 learned_addr learned my addr 172.21.15.29:0/3395870954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 -- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc15ae80 msgr2=0x7f5bcc06cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc15ae80 0x7f5bcc06cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 -- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bcc0ce390 msgr2=0x7f5bcc1560a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bcc0ce390 0x7f5bcc1560a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 -- 172.21.15.29:0/3395870954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc0094d0 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.320+0000 7f5bd147b640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc000c00 tx=0x7f5bbc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.321+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc039a10 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.321+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc039b70 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.321+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bbc043410 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.321+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bcc0c2920 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.321+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bcc0af240 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.322+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bcc04e160 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.323+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5bbc02f390 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.324+0000 7f5bc27fc640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 0x7f5bb006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.324+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f5bbc0ab6d0 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.324+0000 7f5bd1c7c640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 0x7f5bb006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.325+0000 7f5bd1c7c640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 0x7f5bb006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0c2490 tx=0x7f5bc8007950 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.328+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bbc076680 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.561+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bcc0717e0 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.561+0000 7f5bc27fc640 1 -- 172.21.15.29:0/3395870954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f5bbc048090 con 0x7f5bcc15aaa0 2023-10-14T16:57:28.562 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:28.563 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:28.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.564+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 msgr2=0x7f5bb006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.564+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 0x7f5bb006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f5bcc0c2490 tx=0x7f5bc8007950 comp rx=0 tx=0).stop 2023-10-14T16:57:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 msgr2=0x7f5bcc0afc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f5bbc000c00 tx=0x7f5bbc004730 comp rx=0 tx=0).stop 2023-10-14T16:57:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 shutdown_connections 2023-10-14T16:57:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5bb006acc0 0x7f5bb006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bcc15ae80 0x7f5bcc06cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bcc15aaa0 0x7f5bcc0afc60 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 --2- 172.21.15.29:0/3395870954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bcc0ce390 0x7f5bcc1560a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 >> 172.21.15.29:0/3395870954 conn(0x7f5bcc0cb410 msgr2=0x7f5bcc0b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 shutdown_connections 2023-10-14T16:57:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.565+0000 7f5bd3ee0640 1 -- 172.21.15.29:0/3395870954 wait complete. 2023-10-14T16:57:28.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:28.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.802+0000 7f7d08f9a640 1 Processor -- start 2023-10-14T16:57:28.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.802+0000 7f7d08f9a640 1 -- start start 2023-10-14T16:57:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.803+0000 7f7d08f9a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7d0406c040 0x7f7d0406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.803+0000 7f7d08f9a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d040ad970 0x7f7d0415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.803+0000 7f7d08f9a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d0406c960 con 0x7f7d0415ebc0 2023-10-14T16:57:28.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.803+0000 7f7d08f9a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d0415b9d0 con 0x7f7d0406c040 2023-10-14T16:57:28.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.803+0000 7f7d08f9a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d0415bcb0 con 0x7f7d040ad970 2023-10-14T16:57:28.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d03fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7d0415ebc0 0x7f7d040cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40274/0 (socket says 172.21.15.29:40274) 2023-10-14T16:57:28.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d03fff640 1 -- 172.21.15.29:0/4209718549 learned_addr learned my addr 172.21.15.29:0/4209718549 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:28.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d037fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7d0406c040 0x7f7d0406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d02ffd640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d040ad970 0x7f7d0415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d037fe640 1 -- 172.21.15.29:0/4209718549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d040ad970 msgr2=0x7f7d0415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d037fe640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d040ad970 0x7f7d0415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d037fe640 1 -- 172.21.15.29:0/4209718549 >> v1:172.21.15.29:6789/0 conn(0x7f7d0415ebc0 legacy=0x7f7d040cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d037fe640 1 -- 172.21.15.29:0/4209718549 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d0415bf90 con 0x7f7d0406c040 2023-10-14T16:57:28.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.804+0000 7f7d027fc640 1 -- 172.21.15.29:0/4209718549 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158334514 0 0) 0x7f7d0406c960 con 0x7f7d0415ebc0 2023-10-14T16:57:28.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.805+0000 7f7d037fe640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d0406c040 0x7f7d0406c420 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40097d0 tx=0x7f7cf402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36c1a0a4e92d9a29 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d027fc640 1 -- 172.21.15.29:0/4209718549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf402d040 con 0x7f7d0406c040 2023-10-14T16:57:28.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d027fc640 1 -- 172.21.15.29:0/4209718549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf4031810 con 0x7f7d0406c040 2023-10-14T16:57:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d027fc640 1 -- 172.21.15.29:0/4209718549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf4031ba0 con 0x7f7d0406c040 2023-10-14T16:57:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 -- 172.21.15.29:0/4209718549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d0406c040 msgr2=0x7f7d0406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d0406c040 0x7f7d0406c420 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7f7cf40097d0 tx=0x7f7cf402b8b0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 -- 172.21.15.29:0/4209718549 shutdown_connections 2023-10-14T16:57:28.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d040ad970 0x7f7d0415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/4209718549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d0406c040 0x7f7d0406c420 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 -- 172.21.15.29:0/4209718549 >> 172.21.15.29:0/4209718549 conn(0x7f7d040ce6a0 msgr2=0x7f7d040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:28.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 -- 172.21.15.29:0/4209718549 shutdown_connections 2023-10-14T16:57:28.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.806+0000 7f7d08f9a640 1 -- 172.21.15.29:0/4209718549 wait complete. 2023-10-14T16:57:28.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.808+0000 7f7d08f9a640 1 Processor -- start 2023-10-14T16:57:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.808+0000 7f7d08f9a640 1 -- start start 2023-10-14T16:57:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d0406c040 0x7f7d040c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d0415ebc0 0x7f7d040c91b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c4a10 con 0x7f7d040ad970 2023-10-14T16:57:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d040c6fc0 con 0x7f7d0415ebc0 2023-10-14T16:57:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d08f9a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d04146820 con 0x7f7d0406c040 2023-10-14T16:57:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.809+0000 7f7d02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41372/0 (socket says 172.21.15.29:41372) 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 -- 172.21.15.29:0/645827381 learned_addr learned my addr 172.21.15.29:0/645827381 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 -- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d0406c040 msgr2=0x7f7d040c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d0406c040 0x7f7d040c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 -- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d0415ebc0 msgr2=0x7f7d040c91b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d0415ebc0 0x7f7d040c91b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:28.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 -- 172.21.15.29:0/645827381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4009400 con 0x7f7d040ad970 2023-10-14T16:57:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7d02ffd640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800c7f0 tx=0x7f7cf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8004220 con 0x7f7d040ad970 2023-10-14T16:57:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf8004380 con 0x7f7d040ad970 2023-10-14T16:57:28.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.810+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7cf8007840 con 0x7f7d040ad970 2023-10-14T16:57:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.811+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d040c7260 con 0x7f7d040ad970 2023-10-14T16:57:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.811+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d040c7820 con 0x7f7d040ad970 2023-10-14T16:57:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.812+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d0406b330 con 0x7f7d040ad970 2023-10-14T16:57:28.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.813+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7cf8007bb0 con 0x7f7d040ad970 2023-10-14T16:57:28.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.814+0000 7f7cebfff640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 0x7f7cf006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.814+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f7cf8089460 con 0x7f7d040ad970 2023-10-14T16:57:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.814+0000 7f7d037fe640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 0x7f7cf006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.815+0000 7f7d037fe640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 0x7f7cf006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4036000 tx=0x7f7cf4030000 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:28.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:28.818+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7cf80544a0 con 0x7f7d040ad970 2023-10-14T16:57:29.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.016+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d0404e9b0 con 0x7f7d040ad970 2023-10-14T16:57:29.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.017+0000 7f7cebfff640 1 -- 172.21.15.29:0/645827381 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17625 (secure 0 0 0) 0x7f7cf801e090 con 0x7f7d040ad970 2023-10-14T16:57:29.018 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:29.019 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:27.162074+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/559917725":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6833/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/874374761":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/2164323269":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3749138999":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6832/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3356363773":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6833/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6832/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2136904241":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3967724404":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2171619166":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3763524886":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3505521487":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3693879118":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/4176788283":"2023-10-15T16:57:20.630995+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 msgr2=0x7f7cf006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 0x7f7cf006d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f7cf4036000 tx=0x7f7cf4030000 comp rx=0 tx=0).stop 2023-10-14T16:57:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 msgr2=0x7f7d041462e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7f7cf800c7f0 tx=0x7f7cf800ccb0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 shutdown_connections 2023-10-14T16:57:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f7cf006acf0 0x7f7cf006d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d0415ebc0 0x7f7d040c91b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d040ad970 0x7f7d041462e0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 --2- 172.21.15.29:0/645827381 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d0406c040 0x7f7d040c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 >> 172.21.15.29:0/645827381 conn(0x7f7d040ce6a0 msgr2=0x7f7d040705e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 shutdown_connections 2023-10-14T16:57:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.020+0000 7f7d08f9a640 1 -- 172.21.15.29:0/645827381 wait complete. 2023-10-14T16:57:29.041 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:57:29.042 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.259+0000 7fc62fd8a640 1 Processor -- start 2023-10-14T16:57:29.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.259+0000 7fc62fd8a640 1 -- start start 2023-10-14T16:57:29.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62fd8a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6281593b0 0x7fc62806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62fd8a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6280caa10 0x7fc62806fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62fd8a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc628159790 con 0x7fc62806ffd0 2023-10-14T16:57:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62fd8a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6280c9b20 con 0x7fc6280caa10 2023-10-14T16:57:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62fd8a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6280c9e00 con 0x7fc6281593b0 2023-10-14T16:57:29.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62d325640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6280caa10 0x7fc62806fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62e327640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc62806ffd0 0x7fc62815ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40284/0 (socket says 172.21.15.29:40284) 2023-10-14T16:57:29.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62e327640 1 -- 172.21.15.29:0/1139524382 learned_addr learned my addr 172.21.15.29:0/1139524382 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.261+0000 7fc62db26640 1 --2- 172.21.15.29:0/1139524382 >> v2:172.21.15.186:3300/0 conn(0x7fc6281593b0 0x7fc62806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.262+0000 7fc62d325640 1 -- 172.21.15.29:0/1139524382 >> v2:172.21.15.186:3300/0 conn(0x7fc6281593b0 msgr2=0x7fc62806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.262+0000 7fc62d325640 1 --2- 172.21.15.29:0/1139524382 >> v2:172.21.15.186:3300/0 conn(0x7fc6281593b0 0x7fc62806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.262+0000 7fc62d325640 1 -- 172.21.15.29:0/1139524382 >> v1:172.21.15.29:6789/0 conn(0x7fc62806ffd0 legacy=0x7fc62815ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.262+0000 7fc62d325640 1 -- 172.21.15.29:0/1139524382 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6280ca0e0 con 0x7fc6280caa10 2023-10-14T16:57:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.263+0000 7fc62d325640 1 --2- 172.21.15.29:0/1139524382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62806fa90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ce8f0 tx=0x7fc61802b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57f9ca72c0cebf15 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.263+0000 7fc62cb24640 1 -- 172.21.15.29:0/1139524382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618031520 con 0x7fc6280caa10 2023-10-14T16:57:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.263+0000 7fc62cb24640 1 -- 172.21.15.29:0/1139524382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618031680 con 0x7fc6280caa10 2023-10-14T16:57:29.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.263+0000 7fc62cb24640 1 -- 172.21.15.29:0/1139524382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618031a10 con 0x7fc6280caa10 2023-10-14T16:57:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 -- 172.21.15.29:0/1139524382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 msgr2=0x7fc62806fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/1139524382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62806fa90 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7fc6280ce8f0 tx=0x7fc61802b7c0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 -- 172.21.15.29:0/1139524382 shutdown_connections 2023-10-14T16:57:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/1139524382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62806fa90 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/1139524382 >> v2:172.21.15.186:3300/0 conn(0x7fc6281593b0 0x7fc62806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 -- 172.21.15.29:0/1139524382 >> 172.21.15.29:0/1139524382 conn(0x7fc6280cb410 msgr2=0x7fc6280cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 -- 172.21.15.29:0/1139524382 shutdown_connections 2023-10-14T16:57:29.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.264+0000 7fc62fd8a640 1 -- 172.21.15.29:0/1139524382 wait complete. 2023-10-14T16:57:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.266+0000 7fc62fd8a640 1 Processor -- start 2023-10-14T16:57:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.266+0000 7fc62fd8a640 1 -- start start 2023-10-14T16:57:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 0x7fc62814ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6281593b0 0x7fc628148e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280c6070 con 0x7fc6280caa10 2023-10-14T16:57:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280c5d90 con 0x7fc6281593b0 2023-10-14T16:57:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62fd8a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6280c8130 con 0x7fc62806ffd0 2023-10-14T16:57:29.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62d325640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62d325640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41398/0 (socket says 172.21.15.29:41398) 2023-10-14T16:57:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.267+0000 7fc62d325640 1 -- 172.21.15.29:0/593938553 learned_addr learned my addr 172.21.15.29:0/593938553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62db26640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 0x7fc62814ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62e327640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6281593b0 0x7fc628148e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 -- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 msgr2=0x7fc62814ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 0x7fc62814ffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 -- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6281593b0 msgr2=0x7fc628148e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6281593b0 0x7fc628148e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 -- 172.21.15.29:0/593938553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc618009400 con 0x7fc6280caa10 2023-10-14T16:57:29.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62db26640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 0x7fc62814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:29.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.268+0000 7fc62d325640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc618000c00 tx=0x7fc6180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.269+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc618031520 con 0x7fc6280caa10 2023-10-14T16:57:29.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.269+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc618033c70 con 0x7fc6280caa10 2023-10-14T16:57:29.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.269+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61803c400 con 0x7fc6280caa10 2023-10-14T16:57:29.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.269+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6280c8410 con 0x7fc6280caa10 2023-10-14T16:57:29.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.269+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6280c5a60 con 0x7fc6280caa10 2023-10-14T16:57:29.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.270+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc62804e160 con 0x7fc6280caa10 2023-10-14T16:57:29.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.271+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 43) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fc61803a040 con 0x7fc6280caa10 2023-10-14T16:57:29.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.272+0000 7fc61e7fc640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 0x7fc60c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.272+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7fc6180ab1b0 con 0x7fc6280caa10 2023-10-14T16:57:29.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.272+0000 7fc62db26640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 0x7fc60c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.273+0000 7fc62db26640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 0x7fc60c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.277+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc618076310 con 0x7fc6280caa10 2023-10-14T16:57:29.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.512+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6280b4680 con 0x7fc6280caa10 2023-10-14T16:57:29.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.513+0000 7fc61e7fc640 1 -- 172.21.15.29:0/593938553 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fc61807ee60 con 0x7fc6280caa10 2023-10-14T16:57:29.514 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:29.514 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:29.515 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:29.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.515+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 msgr2=0x7fc60c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.515+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 0x7fc60c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc624006fd0 tx=0x7fc624008040 comp rx=0 tx=0).stop 2023-10-14T16:57:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 msgr2=0x7fc62814cbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fc618000c00 tx=0x7fc6180046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 shutdown_connections 2023-10-14T16:57:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc60c06acc0 0x7fc60c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6281593b0 0x7fc628148e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6280caa10 0x7fc62814cbd0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 --2- 172.21.15.29:0/593938553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc62806ffd0 0x7fc62814ffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 >> 172.21.15.29:0/593938553 conn(0x7fc6280cb410 msgr2=0x7fc6280c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:29.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 shutdown_connections 2023-10-14T16:57:29.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.516+0000 7fc62fd8a640 1 -- 172.21.15.29:0/593938553 wait complete. 2023-10-14T16:57:29.536 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.773+0000 7f670d4bb640 1 Processor -- start 2023-10-14T16:57:29.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.773+0000 7f670d4bb640 1 -- start start 2023-10-14T16:57:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.774+0000 7f670d4bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f670806c040 0x7f670806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.774+0000 7f670d4bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670815f4a0 con 0x7f67080cda80 2023-10-14T16:57:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.774+0000 7f670d4bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f670806c960 con 0x7f67080ad970 2023-10-14T16:57:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.774+0000 7f670d4bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f670815b9d0 con 0x7f670806c040 2023-10-14T16:57:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f670ccba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f67080cda80 0x7f670815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40292/0 (socket says 172.21.15.29:40292) 2023-10-14T16:57:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f670ccba640 1 -- 172.21.15.29:0/3900420526 learned_addr learned my addr 172.21.15.29:0/3900420526 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f6707fff640 1 --2- 172.21.15.29:0/3900420526 >> v2:172.21.15.186:3300/0 conn(0x7f670806c040 0x7f670806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4087474438 0 0) 0x7f670815f4a0 con 0x7f67080cda80 2023-10-14T16:57:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66f0003520 con 0x7f67080cda80 2023-10-14T16:57:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.775+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3583360526 0 0) 0x7f66f0003520 con 0x7f67080cda80 2023-10-14T16:57:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 >> v2:172.21.15.186:3300/0 conn(0x7f670806c040 msgr2=0x7f670806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 --2- 172.21.15.29:0/3900420526 >> v2:172.21.15.186:3300/0 conn(0x7f670806c040 0x7f670806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 >> v1:172.21.15.186:6789/0 conn(0x7f67080ad970 legacy=0x7f670815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f670815bcb0 con 0x7f67080cda80 2023-10-14T16:57:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f67000019f0 con 0x7f67080cda80 2023-10-14T16:57:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 >> v1:172.21.15.29:6789/0 conn(0x7f67080cda80 legacy=0x7f670815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66f0003970 0x7f66f0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 0x7f66f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f0005180 0x7f66f000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f670815b9d0 con 0x7f66f00047c0 2023-10-14T16:57:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f000c020 con 0x7f66f0003970 2023-10-14T16:57:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66f000c1e0 con 0x7f66f0005180 2023-10-14T16:57:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f67077fe640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 0x7f66f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f6707fff640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66f0003970 0x7f66f0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.776+0000 7f670ccba640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f0005180 0x7f66f000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f67077fe640 1 -- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f0005180 msgr2=0x7f66f000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f67077fe640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f0005180 0x7f66f000b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f67077fe640 1 -- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66f0003970 msgr2=0x7f66f0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:29.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f67077fe640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66f0003970 0x7f66f0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f67077fe640 1 -- 172.21.15.29:0/3900420526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0003520 con 0x7f66f00047c0 2023-10-14T16:57:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.777+0000 7f670ccba640 1 -- 172.21.15.29:0/3900420526 reap_dead start 2023-10-14T16:57:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f67077fe640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 0x7f66f0004c40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001720 tx=0x7f66f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=285e2aed5b305c36 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc0052d0 con 0x7f66f00047c0 2023-10-14T16:57:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc02e050 con 0x7f66f00047c0 2023-10-14T16:57:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f6706ffd640 1 -- 172.21.15.29:0/3900420526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc003830 con 0x7f66f00047c0 2023-10-14T16:57:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f670d4bb640 1 -- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 msgr2=0x7f66f0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.778+0000 7f670d4bb640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 0x7f66f0004c40 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f66fc001720 tx=0x7f66f00057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.779+0000 7f670d4bb640 1 -- 172.21.15.29:0/3900420526 shutdown_connections 2023-10-14T16:57:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.779+0000 7f670d4bb640 1 --2- 172.21.15.29:0/3900420526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66f00047c0 0x7f66f0004c40 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.779+0000 7f670d4bb640 1 -- 172.21.15.29:0/3900420526 >> 172.21.15.29:0/3900420526 conn(0x7f67080ce6a0 msgr2=0x7f67080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.779+0000 7f670d4bb640 1 -- 172.21.15.29:0/3900420526 shutdown_connections 2023-10-14T16:57:29.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.779+0000 7f670d4bb640 1 -- 172.21.15.29:0/3900420526 wait complete. 2023-10-14T16:57:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.781+0000 7f670d4bb640 1 Processor -- start 2023-10-14T16:57:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.781+0000 7f670d4bb640 1 -- start start 2023-10-14T16:57:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f00047c0 0x7f67080c8120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 0x7f67080c9160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67080c8660 con 0x7f67080b2b20 2023-10-14T16:57:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6708159410 con 0x7f67080b2fa0 2023-10-14T16:57:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670d4bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67080ca980 con 0x7f66f00047c0 2023-10-14T16:57:29.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41430/0 (socket says 172.21.15.29:41430) 2023-10-14T16:57:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 -- 172.21.15.29:0/2575924670 learned_addr learned my addr 172.21.15.29:0/2575924670 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f6707fff640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f00047c0 0x7f67080c8120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 -- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f00047c0 msgr2=0x7f67080c8120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f670ccba640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 0x7f67080c9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f00047c0 0x7f67080c8120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.782+0000 7f67077fe640 1 -- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 msgr2=0x7f67080c9160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f67077fe640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 0x7f67080c9160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:29.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f67077fe640 1 -- 172.21.15.29:0/2575924670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67080cac60 con 0x7f67080b2b20 2023-10-14T16:57:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f670ccba640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 0x7f67080c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f67077fe640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005040 tx=0x7f66fc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc007a30 con 0x7f67080b2b20 2023-10-14T16:57:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66fc0069e0 con 0x7f67080b2b20 2023-10-14T16:57:29.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66fc006d70 con 0x7f67080b2b20 2023-10-14T16:57:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6708158dc0 con 0x7f67080b2b20 2023-10-14T16:57:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.783+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67080c96a0 con 0x7f67080b2b20 2023-10-14T16:57:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.785+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f670806b330 con 0x7f67080b2b20 2023-10-14T16:57:29.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.786+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f66fc0035f0 con 0x7f67080b2b20 2023-10-14T16:57:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.786+0000 7f6704ff9640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 0x7f66f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.787+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f66fc0abcb0 con 0x7f67080b2b20 2023-10-14T16:57:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.787+0000 7f6707fff640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 0x7f66f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:29.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.788+0000 7f6707fff640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 0x7f66f406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f67080ca770 tx=0x7f66f8008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:29.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:29.791+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f66fc076880 con 0x7f67080b2b20 2023-10-14T16:57:30.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.027+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6708159580 con 0x7f67080b2b20 2023-10-14T16:57:30.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.028+0000 7f6704ff9640 1 -- 172.21.15.29:0/2575924670 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f66fc04b090 con 0x7f67080b2b20 2023-10-14T16:57:30.029 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:30.030 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":40,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:05.213523+0000","modified":"2023-10-14T16:57:08.669005+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7002},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7002":{"gid":7002,"name":"c","rank":0,"incarnation":36,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1398561557","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1398561557},{"type":"v1","addr":"172.21.15.29:6837","nonce":1398561557}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[11],"metadata_pool":10,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":5}]} 2023-10-14T16:57:30.030 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 40 2023-10-14T16:57:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 msgr2=0x7f66f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 0x7f66f406d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f67080ca770 tx=0x7f66f8008040 comp rx=0 tx=0).stop 2023-10-14T16:57:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 msgr2=0x7f67080c8c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f66fc005040 tx=0x7f66fc001120 comp rx=0 tx=0).stop 2023-10-14T16:57:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 shutdown_connections 2023-10-14T16:57:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f66f406af00 0x7f66f406d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67080b2fa0 0x7f67080c9160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67080b2b20 0x7f67080c8c20 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 --2- 172.21.15.29:0/2575924670 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66f00047c0 0x7f67080c8120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.031+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 >> 172.21.15.29:0/2575924670 conn(0x7f67080ce6a0 msgr2=0x7f67080cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.032+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 shutdown_connections 2023-10-14T16:57:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.032+0000 7f670d4bb640 1 -- 172.21.15.29:0/2575924670 wait complete. 2023-10-14T16:57:30.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:30.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.277+0000 7f3f55b90640 1 Processor -- start 2023-10-14T16:57:30.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.277+0000 7f3f55b90640 1 -- start start 2023-10-14T16:57:30.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f55b90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3f5006ecd0 0x7f3f50070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f55b90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3f50071360 0x7f3f500cd880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f55b90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f50070660 con 0x7f3f5015a970 2023-10-14T16:57:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f55b90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5015dc60 con 0x7f3f50071360 2023-10-14T16:57:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f55b90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f5015df40 con 0x7f3f5006ecd0 2023-10-14T16:57:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f4f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3f5015a970 0x7f3f5015d430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40298/0 (socket says 172.21.15.29:40298) 2023-10-14T16:57:30.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.279+0000 7f3f4f7fe640 1 -- 172.21.15.29:0/1738914404 learned_addr learned my addr 172.21.15.29:0/1738914404 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4ffff640 1 --2- 172.21.15.29:0/1738914404 >> v2:172.21.15.29:3300/0 conn(0x7f3f50071360 0x7f3f500cd880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f47fff640 1 --2- 172.21.15.29:0/1738914404 >> v2:172.21.15.186:3300/0 conn(0x7f3f5006ecd0 0x7f3f50070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4effd640 1 -- 172.21.15.29:0/1738914404 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2076936357 0 0) 0x7f3f50070660 con 0x7f3f5015a970 2023-10-14T16:57:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4effd640 1 -- 172.21.15.29:0/1738914404 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c003520 con 0x7f3f5015a970 2023-10-14T16:57:30.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4ffff640 1 -- 172.21.15.29:0/1738914404 >> v2:172.21.15.186:3300/0 conn(0x7f3f5006ecd0 msgr2=0x7f3f50070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4ffff640 1 --2- 172.21.15.29:0/1738914404 >> v2:172.21.15.186:3300/0 conn(0x7f3f5006ecd0 0x7f3f50070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4ffff640 1 -- 172.21.15.29:0/1738914404 >> v1:172.21.15.29:6789/0 conn(0x7f3f5015a970 legacy=0x7f3f5015d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.280+0000 7f3f4ffff640 1 -- 172.21.15.29:0/1738914404 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5015e220 con 0x7f3f50071360 2023-10-14T16:57:30.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.281+0000 7f3f4ffff640 1 --2- 172.21.15.29:0/1738914404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f50071360 0x7f3f500cd880 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3f50071800 tx=0x7f3f4802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d175589b9846553 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.282+0000 7f3f4effd640 1 -- 172.21.15.29:0/1738914404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802cc80 con 0x7f3f50071360 2023-10-14T16:57:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.282+0000 7f3f4effd640 1 -- 172.21.15.29:0/1738914404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4802cde0 con 0x7f3f50071360 2023-10-14T16:57:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.282+0000 7f3f4effd640 1 -- 172.21.15.29:0/1738914404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f48034640 con 0x7f3f50071360 2023-10-14T16:57:30.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.282+0000 7f3f55b90640 1 -- 172.21.15.29:0/1738914404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f50071360 msgr2=0x7f3f500cd880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.282+0000 7f3f55b90640 1 --2- 172.21.15.29:0/1738914404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f50071360 0x7f3f500cd880 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7f3f50071800 tx=0x7f3f4802bed0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 -- 172.21.15.29:0/1738914404 shutdown_connections 2023-10-14T16:57:30.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 --2- 172.21.15.29:0/1738914404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f50071360 0x7f3f500cd880 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 --2- 172.21.15.29:0/1738914404 >> v2:172.21.15.186:3300/0 conn(0x7f3f5006ecd0 0x7f3f50070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 -- 172.21.15.29:0/1738914404 >> 172.21.15.29:0/1738914404 conn(0x7f3f500cfaa0 msgr2=0x7f3f500cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:30.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 -- 172.21.15.29:0/1738914404 shutdown_connections 2023-10-14T16:57:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.283+0000 7f3f55b90640 1 -- 172.21.15.29:0/1738914404 wait complete. 2023-10-14T16:57:30.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.284+0000 7f3f55b90640 1 Processor -- start 2023-10-14T16:57:30.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.285+0000 7f3f55b90640 1 -- start start 2023-10-14T16:57:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f5015a970 0x7f3f500ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f500af500 0x7f3f500ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500afa80 con 0x7f3f5006ecd0 2023-10-14T16:57:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500afd60 con 0x7f3f5015a970 2023-10-14T16:57:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f55b90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f500b0040 con 0x7f3f500af500 2023-10-14T16:57:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41456/0 (socket says 172.21.15.29:41456) 2023-10-14T16:57:30.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 -- 172.21.15.29:0/2083858525 learned_addr learned my addr 172.21.15.29:0/2083858525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 -- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f500af500 msgr2=0x7f3f500ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f500af500 0x7f3f500ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 -- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f5015a970 msgr2=0x7f3f500ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f5015a970 0x7f3f500ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.286+0000 7f3f4f7fe640 1 -- 172.21.15.29:0/2083858525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f48009a70 con 0x7f3f5006ecd0 2023-10-14T16:57:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f4f7fe640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000c390 tx=0x7f3f4000c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40005ce0 con 0x7f3f5006ecd0 2023-10-14T16:57:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f40005e40 con 0x7f3f5006ecd0 2023-10-14T16:57:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f400072f0 con 0x7f3f5006ecd0 2023-10-14T16:57:30.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f500b0320 con 0x7f3f5006ecd0 2023-10-14T16:57:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.287+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f500aeb70 con 0x7f3f5006ecd0 2023-10-14T16:57:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.288+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f500b0840 con 0x7f3f5006ecd0 2023-10-14T16:57:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.289+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3f40007450 con 0x7f3f5006ecd0 2023-10-14T16:57:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.290+0000 7f3f4cff9640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 0x7f3f3006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.290+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f3f40088fa0 con 0x7f3f5006ecd0 2023-10-14T16:57:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.291+0000 7f3f47fff640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 0x7f3f3006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.291+0000 7f3f47fff640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 0x7f3f3006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f500c68e0 tx=0x7f3f38008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.295+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f40053d10 con 0x7f3f5006ecd0 2023-10-14T16:57:30.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.492+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f5004e160 con 0x7f3f5006ecd0 2023-10-14T16:57:30.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.495+0000 7f3f4cff9640 1 -- 172.21.15.29:0/2083858525 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) v1 ==== 74+0+17625 (secure 0 0 0) 0x7f3f40017020 con 0x7f3f5006ecd0 2023-10-14T16:57:30.496 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:30.497 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:27.162074+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":10,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:03.662290+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"95","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":11,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:04.248899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"96","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/559917725":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6833/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/874374761":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/2164323269":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3749138999":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6832/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3356363773":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6833/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6832/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2136904241":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3967724404":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2171619166":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3763524886":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3505521487":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3693879118":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/4176788283":"2023-10-15T16:57:20.630995+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:30.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 msgr2=0x7f3f3006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 0x7f3f3006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3f500c68e0 tx=0x7f3f38008040 comp rx=0 tx=0).stop 2023-10-14T16:57:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 msgr2=0x7f3f500c35a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f3f4000c390 tx=0x7f3f4000c850 comp rx=0 tx=0).stop 2023-10-14T16:57:30.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 shutdown_connections 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f3f3006af00 0x7f3f3006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f500af500 0x7f3f500ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f5015a970 0x7f3f500ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 --2- 172.21.15.29:0/2083858525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f5006ecd0 0x7f3f500c35a0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 >> 172.21.15.29:0/2083858525 conn(0x7f3f500cfaa0 msgr2=0x7f3f500cf010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:30.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 shutdown_connections 2023-10-14T16:57:30.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.498+0000 7f3f55b90640 1 -- 172.21.15.29:0/2083858525 wait complete. 2023-10-14T16:57:30.519 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:57:30.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.748+0000 7f2a92cd1640 1 Processor -- start 2023-10-14T16:57:30.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.748+0000 7f2a92cd1640 1 -- start start 2023-10-14T16:57:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.749+0000 7f2a92cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c159950 0x7f2a8c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a92cd1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a8c070450 0x7f2a8c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a92cd1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8c070df0 con 0x7f2a8c06c100 2023-10-14T16:57:30.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a92cd1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8c15e490 con 0x7f2a8c070450 2023-10-14T16:57:30.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a92cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c15aef0 con 0x7f2a8c159950 2023-10-14T16:57:30.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a914ce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a8c070450 0x7f2a8c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a914ce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2a8c070450 0x7f2a8c0708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41470/0 (socket says 172.21.15.29:41470) 2023-10-14T16:57:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a914ce640 1 -- 172.21.15.29:0/2059906511 learned_addr learned my addr 172.21.15.29:0/2059906511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.750+0000 7f2a91ccf640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c159950 0x7f2a8c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.751+0000 7f2a914ce640 1 -- 172.21.15.29:0/2059906511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c159950 msgr2=0x7f2a8c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:30.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.751+0000 7f2a914ce640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c159950 0x7f2a8c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.751+0000 7f2a914ce640 1 -- 172.21.15.29:0/2059906511 >> v1:172.21.15.186:6789/0 conn(0x7f2a8c06c100 legacy=0x7f2a8c06c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.751+0000 7f2a914ce640 1 -- 172.21.15.29:0/2059906511 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8c06cc80 con 0x7f2a8c070450 2023-10-14T16:57:30.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.752+0000 7f2a914ce640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c070450 0x7f2a8c0708b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009e50 tx=0x7f2a7c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96ff2e213a60e09d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.752+0000 7f2a90ccd640 1 -- 172.21.15.29:0/2059906511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c02ca80 con 0x7f2a8c070450 2023-10-14T16:57:30.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a90ccd640 1 -- 172.21.15.29:0/2059906511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7c02cbe0 con 0x7f2a8c070450 2023-10-14T16:57:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a90ccd640 1 -- 172.21.15.29:0/2059906511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7c0343f0 con 0x7f2a8c070450 2023-10-14T16:57:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 -- 172.21.15.29:0/2059906511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c070450 msgr2=0x7f2a8c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c070450 0x7f2a8c0708b0 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c009e50 tx=0x7f2a7c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 -- 172.21.15.29:0/2059906511 shutdown_connections 2023-10-14T16:57:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c070450 0x7f2a8c0708b0 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/2059906511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c159950 0x7f2a8c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 -- 172.21.15.29:0/2059906511 >> 172.21.15.29:0/2059906511 conn(0x7f2a8c0ce6a0 msgr2=0x7f2a8c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:30.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 -- 172.21.15.29:0/2059906511 shutdown_connections 2023-10-14T16:57:30.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.753+0000 7f2a92cd1640 1 -- 172.21.15.29:0/2059906511 wait complete. 2023-10-14T16:57:30.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.755+0000 7f2a92cd1640 1 Processor -- start 2023-10-14T16:57:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.755+0000 7f2a92cd1640 1 -- start start 2023-10-14T16:57:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c070450 0x7f2a8c14f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a8c159950 0x7f2a8c155260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c0c5770 con 0x7f2a8c06c100 2023-10-14T16:57:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c14fd40 con 0x7f2a8c159950 2023-10-14T16:57:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a92cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8c150020 con 0x7f2a8c070450 2023-10-14T16:57:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a91ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a91ccf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41476/0 (socket says 172.21.15.29:41476) 2023-10-14T16:57:30.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.756+0000 7f2a91ccf640 1 -- 172.21.15.29:0/3059286124 learned_addr learned my addr 172.21.15.29:0/3059286124 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 -- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c070450 msgr2=0x7f2a8c14f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c070450 0x7f2a8c14f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 -- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a8c159950 msgr2=0x7f2a8c155260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a8c159950 0x7f2a8c155260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:30.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 -- 172.21.15.29:0/3059286124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c009b30 con 0x7f2a8c06c100 2023-10-14T16:57:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.757+0000 7f2a91ccf640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800d800 tx=0x7f2a8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.758+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800b6c0 con 0x7f2a8c06c100 2023-10-14T16:57:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.758+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a8800b820 con 0x7f2a8c06c100 2023-10-14T16:57:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.758+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a8800bbb0 con 0x7f2a8c06c100 2023-10-14T16:57:30.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.758+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a8c150300 con 0x7f2a8c06c100 2023-10-14T16:57:30.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.758+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8c150fa0 con 0x7f2a8c06c100 2023-10-14T16:57:30.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.759+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8c04e9b0 con 0x7f2a8c06c100 2023-10-14T16:57:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.760+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2a8800bd10 con 0x7f2a8c06c100 2023-10-14T16:57:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.761+0000 7f2a827fc640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 0x7f2a7006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.761+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 6441+0+0 (secure 0 0 0) 0x7f2a88089e20 con 0x7f2a8c06c100 2023-10-14T16:57:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.761+0000 7f2a914ce640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 0x7f2a7006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:30.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.762+0000 7f2a914ce640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 0x7f2a7006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c035000 tx=0x7f2a7c02e000 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:30.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:30.766+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a88054c90 con 0x7f2a8c06c100 2023-10-14T16:57:31.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.004+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2a8c0b4620 con 0x7f2a8c06c100 2023-10-14T16:57:31.005 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:57:31.005+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:57:31.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.008+0000 7f2a827fc640 1 -- 172.21.15.29:0/3059286124 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2a880582b0 con 0x7f2a8c06c100 2023-10-14T16:57:31.009 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:57:31.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 msgr2=0x7f2a7006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 0x7f2a7006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2a7c035000 tx=0x7f2a7c02e000 comp rx=0 tx=0).stop 2023-10-14T16:57:31.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 msgr2=0x7f2a8c14ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2a8800d800 tx=0x7f2a8800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 shutdown_connections 2023-10-14T16:57:31.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f2a7006af00 0x7f2a7006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a8c159950 0x7f2a8c155260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a8c070450 0x7f2a8c14f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 --2- 172.21.15.29:0/3059286124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a8c06c100 0x7f2a8c14ea10 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 >> 172.21.15.29:0/3059286124 conn(0x7f2a8c0ce6a0 msgr2=0x7f2a8c0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:31.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 shutdown_connections 2023-10-14T16:57:31.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.011+0000 7f2a92cd1640 1 -- 172.21.15.29:0/3059286124 wait complete. 2023-10-14T16:57:31.017 INFO:tasks.ceph.mds.c.smithi029.stderr: -2661> 2023-10-14T16:57:10.657+0000 7f6732e5d640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:57:31.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:57:31.069 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:57:31.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.224+0000 7f07380e5640 1 Processor -- start 2023-10-14T16:57:31.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.224+0000 7f07380e5640 1 -- start start 2023-10-14T16:57:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.225+0000 7f07380e5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0730159ab0 0x7f07300ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.225+0000 7f07380e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0730159e90 0x7f073006fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.225+0000 7f07380e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07300cd1f0 con 0x7f073006c140 2023-10-14T16:57:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.225+0000 7f07380e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073015d890 con 0x7f0730159e90 2023-10-14T16:57:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.225+0000 7f07380e5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07300cd4d0 con 0x7f0730159ab0 2023-10-14T16:57:31.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0736682640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f073006c140 0x7f073015dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43436/0 (socket says 172.21.15.29:43436) 2023-10-14T16:57:31.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0736682640 1 -- 172.21.15.29:0/1989087677 learned_addr learned my addr 172.21.15.29:0/1989087677 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:31.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735680640 1 --2- 172.21.15.29:0/1989087677 >> v2:172.21.15.29:3300/0 conn(0x7f0730159e90 0x7f073006fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735e81640 1 --2- 172.21.15.29:0/1989087677 >> v2:172.21.15.186:3300/0 conn(0x7f0730159ab0 0x7f07300ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0734e7f640 1 -- 172.21.15.29:0/1989087677 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938225654 0 0) 0x7f07300cd1f0 con 0x7f073006c140 2023-10-14T16:57:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735680640 1 -- 172.21.15.29:0/1989087677 >> v2:172.21.15.186:3300/0 conn(0x7f0730159ab0 msgr2=0x7f07300ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735680640 1 --2- 172.21.15.29:0/1989087677 >> v2:172.21.15.186:3300/0 conn(0x7f0730159ab0 0x7f07300ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735680640 1 -- 172.21.15.29:0/1989087677 >> v1:172.21.15.29:6789/0 conn(0x7f073006c140 legacy=0x7f073015dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.226+0000 7f0735680640 1 -- 172.21.15.29:0/1989087677 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07300cd7b0 con 0x7f0730159e90 2023-10-14T16:57:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.227+0000 7f0735680640 1 --2- 172.21.15.29:0/1989087677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159e90 0x7f073006fc50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f07300cfa00 tx=0x7f072002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5756caf6ab65a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f0734e7f640 1 -- 172.21.15.29:0/1989087677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720031520 con 0x7f0730159e90 2023-10-14T16:57:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f0734e7f640 1 -- 172.21.15.29:0/1989087677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720031680 con 0x7f0730159e90 2023-10-14T16:57:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f0734e7f640 1 -- 172.21.15.29:0/1989087677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720031a10 con 0x7f0730159e90 2023-10-14T16:57:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 -- 172.21.15.29:0/1989087677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159e90 msgr2=0x7f073006fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 --2- 172.21.15.29:0/1989087677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159e90 0x7f073006fc50 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f07300cfa00 tx=0x7f072002b7c0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 -- 172.21.15.29:0/1989087677 shutdown_connections 2023-10-14T16:57:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 --2- 172.21.15.29:0/1989087677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159e90 0x7f073006fc50 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 --2- 172.21.15.29:0/1989087677 >> v2:172.21.15.186:3300/0 conn(0x7f0730159ab0 0x7f07300ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 -- 172.21.15.29:0/1989087677 >> 172.21.15.29:0/1989087677 conn(0x7f07300ce9b0 msgr2=0x7f07300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.228+0000 7f07380e5640 1 -- 172.21.15.29:0/1989087677 shutdown_connections 2023-10-14T16:57:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.229+0000 7f07380e5640 1 -- 172.21.15.29:0/1989087677 wait complete. 2023-10-14T16:57:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.231+0000 7f07380e5640 1 Processor -- start 2023-10-14T16:57:31.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.231+0000 7f07380e5640 1 -- start start 2023-10-14T16:57:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073006c140 0x7f073006b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0730159e90 0x7f073014b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c2290 con 0x7f0730159ab0 2023-10-14T16:57:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f073015a290 con 0x7f073006c140 2023-10-14T16:57:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f07380e5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07300c4410 con 0x7f0730159e90 2023-10-14T16:57:31.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f0735680640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f0735680640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35666/0 (socket says 172.21.15.29:35666) 2023-10-14T16:57:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.232+0000 7f0735680640 1 -- 172.21.15.29:0/1689939108 learned_addr learned my addr 172.21.15.29:0/1689939108 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 -- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0730159e90 msgr2=0x7f073014b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0730159e90 0x7f073014b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 -- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073006c140 msgr2=0x7f073006b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073006c140 0x7f073006b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 -- 172.21.15.29:0/1689939108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0720009400 con 0x7f0730159ab0 2023-10-14T16:57:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f0735680640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0720000c00 tx=0x7f07200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720033bd0 con 0x7f0730159ab0 2023-10-14T16:57:31.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0720033d30 con 0x7f0730159ab0 2023-10-14T16:57:31.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.233+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0720033ea0 con 0x7f0730159ab0 2023-10-14T16:57:31.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.234+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07300c46f0 con 0x7f0730159ab0 2023-10-14T16:57:31.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.234+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073015a4e0 con 0x7f0730159ab0 2023-10-14T16:57:31.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.235+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073004e160 con 0x7f0730159ab0 2023-10-14T16:57:31.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.236+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 44) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f072003a040 con 0x7f0730159ab0 2023-10-14T16:57:31.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.237+0000 7f07267fc640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 0x7f071406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:31.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.237+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f07200ab3d0 con 0x7f0730159ab0 2023-10-14T16:57:31.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.237+0000 7f0735e81640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 0x7f071406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:31.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.237+0000 7f0735e81640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 0x7f071406d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f072c0098f0 tx=0x7f072c008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:31.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.241+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0720076200 con 0x7f0730159ab0 2023-10-14T16:57:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:31.466+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f07300ccb60 con 0x7f0730159ab0 2023-10-14T16:57:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.020+0000 7f07267fc640 1 -- 172.21.15.29:0/1689939108 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0720006e10 con 0x7f0730159ab0 2023-10-14T16:57:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.024+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 msgr2=0x7f071406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.024+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 0x7f071406d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f072c0098f0 tx=0x7f072c008040 comp rx=0 tx=0).stop 2023-10-14T16:57:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.024+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 msgr2=0x7f073014af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.024+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f0720000c00 tx=0x7f07200046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 shutdown_connections 2023-10-14T16:57:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f071406af30 0x7f071406d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0730159e90 0x7f073014b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0730159ab0 0x7f073014af50 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 --2- 172.21.15.29:0/1689939108 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f073006c140 0x7f073006b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 >> 172.21.15.29:0/1689939108 conn(0x7f07300ce9b0 msgr2=0x7f07300b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.025+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 shutdown_connections 2023-10-14T16:57:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.026+0000 7f07380e5640 1 -- 172.21.15.29:0/1689939108 wait complete. 2023-10-14T16:57:32.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:57:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.327+0000 7f28cbe83640 1 Processor -- start 2023-10-14T16:57:32.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.327+0000 7f28cbe83640 1 -- start start 2023-10-14T16:57:32.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28cbe83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28c415cf60 0x7f28c406f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28cbe83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28c40b0360 0x7f28c40b07b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28cbe83640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c40b0cf0 con 0x7f28c406f9d0 2023-10-14T16:57:32.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28cbe83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c415d340 con 0x7f28c415cf60 2023-10-14T16:57:32.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28cbe83640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c40cc860 con 0x7f28c40b0360 2023-10-14T16:57:32.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.329+0000 7f28c9c1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28c415cf60 0x7f28c406f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f28c415cf60 0x7f28c406f490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35680/0 (socket says 172.21.15.29:35680) 2023-10-14T16:57:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 -- 172.21.15.29:0/2924124798 learned_addr learned my addr 172.21.15.29:0/2924124798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28ca420640 1 --2- 172.21.15.29:0/2924124798 >> v2:172.21.15.186:3300/0 conn(0x7f28c40b0360 0x7f28c40b07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 -- 172.21.15.29:0/2924124798 >> v2:172.21.15.186:3300/0 conn(0x7f28c40b0360 msgr2=0x7f28c40b07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/2924124798 >> v2:172.21.15.186:3300/0 conn(0x7f28c40b0360 0x7f28c40b07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 -- 172.21.15.29:0/2924124798 >> v1:172.21.15.186:6789/0 conn(0x7f28c406f9d0 legacy=0x7f28c40c9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.330+0000 7f28c9c1f640 1 -- 172.21.15.29:0/2924124798 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40c9620 con 0x7f28c415cf60 2023-10-14T16:57:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.333+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/2924124798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c406f490 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009870 tx=0x7f28c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb1e29ebc68708 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.333+0000 7f28c8c1d640 1 -- 172.21.15.29:0/2924124798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0030860 con 0x7f28c415cf60 2023-10-14T16:57:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.333+0000 7f28c8c1d640 1 -- 172.21.15.29:0/2924124798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c00309c0 con 0x7f28c415cf60 2023-10-14T16:57:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.333+0000 7f28c8c1d640 1 -- 172.21.15.29:0/2924124798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c0030d50 con 0x7f28c415cf60 2023-10-14T16:57:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 -- 172.21.15.29:0/2924124798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 msgr2=0x7f28c406f490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 --2- 172.21.15.29:0/2924124798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c406f490 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009870 tx=0x7f28c002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 -- 172.21.15.29:0/2924124798 shutdown_connections 2023-10-14T16:57:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 --2- 172.21.15.29:0/2924124798 >> v2:172.21.15.186:3300/0 conn(0x7f28c40b0360 0x7f28c40b07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 --2- 172.21.15.29:0/2924124798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c406f490 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 -- 172.21.15.29:0/2924124798 >> 172.21.15.29:0/2924124798 conn(0x7f28c40cb360 msgr2=0x7f28c40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 -- 172.21.15.29:0/2924124798 shutdown_connections 2023-10-14T16:57:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.334+0000 7f28cbe83640 1 -- 172.21.15.29:0/2924124798 wait complete. 2023-10-14T16:57:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.336+0000 7f28cbe83640 1 Processor -- start 2023-10-14T16:57:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.336+0000 7f28cbe83640 1 -- start start 2023-10-14T16:57:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 0x7f28c40c77d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28c40b0360 0x7f28c40b2280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40c6e10 con 0x7f28c415cf60 2023-10-14T16:57:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40c2ee0 con 0x7f28c406f9d0 2023-10-14T16:57:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28cbe83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40c25b0 con 0x7f28c40b0360 2023-10-14T16:57:32.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28ca420640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:32.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28ca420640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35690/0 (socket says 172.21.15.29:35690) 2023-10-14T16:57:32.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.337+0000 7f28ca420640 1 -- 172.21.15.29:0/1808265407 learned_addr learned my addr 172.21.15.29:0/1808265407 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:32.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 -- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28c40b0360 msgr2=0x7f28c40b2280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28c40b0360 0x7f28c40b2280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 -- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 msgr2=0x7f28c40c77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 0x7f28c40c77d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 0x7f28c40c77d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 -- 172.21.15.29:0/1808265407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c00094d0 con 0x7f28c415cf60 2023-10-14T16:57:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 0x7f28c40c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ca420640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00efc0 tx=0x7f28bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc009ba0 con 0x7f28c415cf60 2023-10-14T16:57:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc00f040 con 0x7f28c415cf60 2023-10-14T16:57:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc0146c0 con 0x7f28c415cf60 2023-10-14T16:57:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c40c2890 con 0x7f28c415cf60 2023-10-14T16:57:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.338+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c40c8220 con 0x7f28c415cf60 2023-10-14T16:57:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.339+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c40c3070 con 0x7f28c415cf60 2023-10-14T16:57:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.341+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f28bc014820 con 0x7f28c415cf60 2023-10-14T16:57:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.341+0000 7f28ba7fc640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 0x7f28a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.342+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f28bc08a7d0 con 0x7f28c415cf60 2023-10-14T16:57:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.342+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 0x7f28a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.342+0000 7f28c9c1f640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 0x7f28a806d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f28c0036000 tx=0x7f28c003b040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.346+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28bc055290 con 0x7f28c415cf60 2023-10-14T16:57:32.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:32.543+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f28c404e160 con 0x7f28c415cf60 2023-10-14T16:57:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.023+0000 7f28ba7fc640 1 -- 172.21.15.29:0/1808265407 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) v1 ==== 181+0+0 (secure 0 0 0) 0x7f28bc0588b0 con 0x7f28c415cf60 2023-10-14T16:57:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:57:33.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.027+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 msgr2=0x7f28a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.027+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 0x7f28a806d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f28c0036000 tx=0x7f28c003b040 comp rx=0 tx=0).stop 2023-10-14T16:57:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 msgr2=0x7f28c40c63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 secure :-1 s=READY pgs=1162 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00efc0 tx=0x7f28bc00c490 comp rx=0 tx=0).stop 2023-10-14T16:57:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 shutdown_connections 2023-10-14T16:57:33.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f28a806af90 0x7f28a806d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c415cf60 0x7f28c40c63c0 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28c40b0360 0x7f28c40b2280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 --2- 172.21.15.29:0/1808265407 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c406f9d0 0x7f28c40c77d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 >> 172.21.15.29:0/1808265407 conn(0x7f28c40cb360 msgr2=0x7f28c40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.028+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 shutdown_connections 2023-10-14T16:57:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.029+0000 7f28cbe83640 1 -- 172.21.15.29:0/1808265407 wait complete. 2023-10-14T16:57:33.055 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:57:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.288+0000 7f8fc3fe9640 1 Processor -- start 2023-10-14T16:57:33.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.288+0000 7f8fc3fe9640 1 -- start start 2023-10-14T16:57:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.289+0000 7f8fc3fe9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.289+0000 7f8fc3fe9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc15b310 con 0x7f8fbc15ac10 2023-10-14T16:57:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.289+0000 7f8fc3fe9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fbc06f640 con 0x7f8fbc0ca510 2023-10-14T16:57:33.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.289+0000 7f8fc3fe9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fbc15f800 con 0x7f8fbc0b0360 2023-10-14T16:57:33.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35698/0 (socket says 172.21.15.29:35698) 2023-10-14T16:57:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 -- 172.21.15.29:0/1685073826 learned_addr learned my addr 172.21.15.29:0/1685073826 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1d85640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8fbc15ac10 0x7f8fbc06c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43446/0 (socket says 172.21.15.29:43446) 2023-10-14T16:57:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 -- 172.21.15.29:0/1685073826 >> v1:172.21.15.186:6789/0 conn(0x7f8fbc0ca510 legacy=0x7f8fbc0ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:33.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 -- 172.21.15.29:0/1685073826 >> v1:172.21.15.29:6789/0 conn(0x7f8fbc15ac10 legacy=0x7f8fbc06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.290+0000 7f8fc1584640 1 -- 172.21.15.29:0/1685073826 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fbc15b5b0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.291+0000 7f8fc1584640 1 --2- 172.21.15.29:0/1685073826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009870 tx=0x7f8fac02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a482539d3e783851 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc0d83640 1 -- 172.21.15.29:0/1685073826 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac031870 con 0x7f8fbc0b0360 2023-10-14T16:57:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc0d83640 1 -- 172.21.15.29:0/1685073826 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fac0319d0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc0d83640 1 -- 172.21.15.29:0/1685073826 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fac031d60 con 0x7f8fbc0b0360 2023-10-14T16:57:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1685073826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 msgr2=0x7f8fbc0b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1685073826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 crypto rx=0x7f8fac009870 tx=0x7f8fac02b910 comp rx=0 tx=0).stop 2023-10-14T16:57:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1685073826 shutdown_connections 2023-10-14T16:57:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1685073826 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0b07f0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.292+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1685073826 >> 172.21.15.29:0/1685073826 conn(0x7f8fbc0cb360 msgr2=0x7f8fbc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.293+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1685073826 shutdown_connections 2023-10-14T16:57:33.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.293+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1685073826 wait complete. 2023-10-14T16:57:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.295+0000 7f8fc3fe9640 1 Processor -- start 2023-10-14T16:57:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.295+0000 7f8fc3fe9640 1 -- start start 2023-10-14T16:57:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fbc0ca510 0x7f8fbc0ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 0x7f8fbc0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0cd400 con 0x7f8fbc0b0360 2023-10-14T16:57:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc0ce1e0 con 0x7f8fbc15ac10 2023-10-14T16:57:33.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.296+0000 7f8fc3fe9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fbc156260 con 0x7f8fbc0ca510 2023-10-14T16:57:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35708/0 (socket says 172.21.15.29:35708) 2023-10-14T16:57:33.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 -- 172.21.15.29:0/1992671787 learned_addr learned my addr 172.21.15.29:0/1992671787 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:33.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc2586640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 0x7f8fbc0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1584640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fbc0ca510 0x7f8fbc0ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 -- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fbc0ca510 msgr2=0x7f8fbc0ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fbc0ca510 0x7f8fbc0ccec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 -- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 msgr2=0x7f8fbc0acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:33.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 0x7f8fbc0acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 -- 172.21.15.29:0/1992671787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fac0094d0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc2586640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 0x7f8fbc0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.297+0000 7f8fc1d85640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c3e0 tx=0x7f8fb800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.298+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb80074e0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.298+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb8007640 con 0x7f8fbc0b0360 2023-10-14T16:57:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.298+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fb80049e0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.298+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fbc156500 con 0x7f8fbc0b0360 2023-10-14T16:57:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.298+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fbc1567e0 con 0x7f8fbc0b0360 2023-10-14T16:57:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.299+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fbc04e160 con 0x7f8fbc0b0360 2023-10-14T16:57:33.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.300+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8fb8004b40 con 0x7f8fbc0b0360 2023-10-14T16:57:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.301+0000 7f8fb27fc640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 0x7f8fa006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.301+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6089+0+0 (secure 0 0 0) 0x7f8fb8089310 con 0x7f8fbc0b0360 2023-10-14T16:57:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.301+0000 7f8fc1584640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 0x7f8fa006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:33.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.302+0000 7f8fc1584640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 0x7f8fa006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8fac036000 tx=0x7f8fac03c040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.305+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fb8054120 con 0x7f8fbc0b0360 2023-10-14T16:57:33.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:33.499+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f8fbc06c2e0 con 0x7f8fbc0b0360 2023-10-14T16:57:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.033+0000 7f8fb27fc640 1 -- 172.21.15.29:0/1992671787 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) v1 ==== 169+0+0 (secure 0 0 0) 0x7f8fb8057740 con 0x7f8fbc0b0360 2023-10-14T16:57:34.035 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:57:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.037+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 msgr2=0x7f8fa006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.037+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 0x7f8fa006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8fac036000 tx=0x7f8fac03c040 comp rx=0 tx=0).stop 2023-10-14T16:57:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.037+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 msgr2=0x7f8fbc0cdb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.037+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 crypto rx=0x7f8fb800c3e0 tx=0x7f8fb800c8a0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 shutdown_connections 2023-10-14T16:57:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8fa006af90 0x7f8fa006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fbc15ac10 0x7f8fbc0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fbc0ca510 0x7f8fbc0ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 --2- 172.21.15.29:0/1992671787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fbc0b0360 0x7f8fbc0cdb90 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 >> 172.21.15.29:0/1992671787 conn(0x7f8fbc0cb360 msgr2=0x7f8fbc06b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 shutdown_connections 2023-10-14T16:57:34.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.039+0000 7f8fc3fe9640 1 -- 172.21.15.29:0/1992671787 wait complete. 2023-10-14T16:57:34.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T16:57:34.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.280+0000 7f8e296d1640 1 Processor -- start 2023-10-14T16:57:34.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.281+0000 7f8e296d1640 1 -- start start 2023-10-14T16:57:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.282+0000 7f8e296d1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e2406c630 0x7f8e240cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.282+0000 7f8e296d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e240b0c90 0x7f8e240c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.282+0000 7f8e296d1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e2406f3a0 con 0x7f8e240adb30 2023-10-14T16:57:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.282+0000 7f8e296d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e240c8e70 con 0x7f8e240b0c90 2023-10-14T16:57:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.282+0000 7f8e296d1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e240c9150 con 0x7f8e2406c630 2023-10-14T16:57:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e240b0c90 0x7f8e240c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8e240b0c90 0x7f8e240c9950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35710/0 (socket says 172.21.15.29:35710) 2023-10-14T16:57:34.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 -- 172.21.15.29:0/3929450607 learned_addr learned my addr 172.21.15.29:0/3929450607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:34.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 -- 172.21.15.29:0/3929450607 >> v2:172.21.15.186:3300/0 conn(0x7f8e2406c630 msgr2=0x7f8e240cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 --2- 172.21.15.29:0/3929450607 >> v2:172.21.15.186:3300/0 conn(0x7f8e2406c630 0x7f8e240cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 -- 172.21.15.29:0/3929450607 >> v1:172.21.15.186:6789/0 conn(0x7f8e240adb30 legacy=0x7f8e240cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.283+0000 7f8e28ed0640 1 -- 172.21.15.29:0/3929450607 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e240c9430 con 0x7f8e240b0c90 2023-10-14T16:57:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.284+0000 7f8e28ed0640 1 --2- 172.21.15.29:0/3929450607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240b0c90 0x7f8e240c9950 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f8e24071720 tx=0x7f8e1c02be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8293b4bbd430822e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e237fe640 1 -- 172.21.15.29:0/3929450607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c02c9f0 con 0x7f8e240b0c90 2023-10-14T16:57:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e237fe640 1 -- 172.21.15.29:0/3929450607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e1c02cb50 con 0x7f8e240b0c90 2023-10-14T16:57:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e237fe640 1 -- 172.21.15.29:0/3929450607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e1c0343f0 con 0x7f8e240b0c90 2023-10-14T16:57:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 -- 172.21.15.29:0/3929450607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240b0c90 msgr2=0x7f8e240c9950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 --2- 172.21.15.29:0/3929450607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240b0c90 0x7f8e240c9950 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f8e24071720 tx=0x7f8e1c02be50 comp rx=0 tx=0).stop 2023-10-14T16:57:34.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 -- 172.21.15.29:0/3929450607 shutdown_connections 2023-10-14T16:57:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 --2- 172.21.15.29:0/3929450607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240b0c90 0x7f8e240c9950 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 --2- 172.21.15.29:0/3929450607 >> v2:172.21.15.186:3300/0 conn(0x7f8e2406c630 0x7f8e240cd260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 -- 172.21.15.29:0/3929450607 >> 172.21.15.29:0/3929450607 conn(0x7f8e240b0170 msgr2=0x7f8e240b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 -- 172.21.15.29:0/3929450607 shutdown_connections 2023-10-14T16:57:34.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.285+0000 7f8e296d1640 1 -- 172.21.15.29:0/3929450607 wait complete. 2023-10-14T16:57:34.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.287+0000 7f8e296d1640 1 Processor -- start 2023-10-14T16:57:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.287+0000 7f8e296d1640 1 -- start start 2023-10-14T16:57:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.288+0000 7f8e296d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2406c630 0x7f8e240c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.288+0000 7f8e296d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e296d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e240b0c90 0x7f8e240c4700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e296d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240c2ea0 con 0x7f8e240adb30 2023-10-14T16:57:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e296d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e24160050 con 0x7f8e2406c630 2023-10-14T16:57:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e296d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e240b23e0 con 0x7f8e240b0c90 2023-10-14T16:57:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35724/0 (socket says 172.21.15.29:35724) 2023-10-14T16:57:34.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 -- 172.21.15.29:0/4046804138 learned_addr learned my addr 172.21.15.29:0/4046804138 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e23fff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2406c630 0x7f8e240c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 -- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e240b0c90 msgr2=0x7f8e240c4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e240b0c90 0x7f8e240c4700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 -- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2406c630 msgr2=0x7f8e240c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2406c630 0x7f8e240c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.289+0000 7f8e1bfff640 1 -- 172.21.15.29:0/4046804138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e1c009b30 con 0x7f8e240adb30 2023-10-14T16:57:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e1bfff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00c410 tx=0x7f8e0c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c0035f0 con 0x7f8e240adb30 2023-10-14T16:57:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e0c003750 con 0x7f8e240adb30 2023-10-14T16:57:34.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e0c003b40 con 0x7f8e240adb30 2023-10-14T16:57:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e240c3090 con 0x7f8e240adb30 2023-10-14T16:57:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.290+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e240b2960 con 0x7f8e240adb30 2023-10-14T16:57:34.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.291+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e2404e9b0 con 0x7f8e240adb30 2023-10-14T16:57:34.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.292+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8e0c003ca0 con 0x7f8e240adb30 2023-10-14T16:57:34.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.293+0000 7f8e217fa640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 0x7f8e0406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.293+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f8e0c088a20 con 0x7f8e240adb30 2023-10-14T16:57:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.293+0000 7f8e23fff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 0x7f8e0406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.294+0000 7f8e23fff640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 0x7f8e0406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009890 tx=0x7f8e14008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.298+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e0c053940 con 0x7f8e240adb30 2023-10-14T16:57:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.510+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8e2406d320 con 0x7f8e240adb30 2023-10-14T16:57:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.517+0000 7f8e217fa640 1 -- 172.21.15.29:0/4046804138 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v31) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8e0c056f60 con 0x7f8e240adb30 2023-10-14T16:57:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.520+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 msgr2=0x7f8e0406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.520+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 0x7f8e0406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8e14009890 tx=0x7f8e14008040 comp rx=0 tx=0).stop 2023-10-14T16:57:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.520+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 msgr2=0x7f8e240c41c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.520+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8e0c00c410 tx=0x7f8e0c00c8d0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 shutdown_connections 2023-10-14T16:57:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8e0406af90 0x7f8e0406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e240b0c90 0x7f8e240c4700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e240adb30 0x7f8e240c41c0 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 --2- 172.21.15.29:0/4046804138 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e2406c630 0x7f8e240c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 >> 172.21.15.29:0/4046804138 conn(0x7f8e240b0170 msgr2=0x7f8e2406b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 shutdown_connections 2023-10-14T16:57:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.521+0000 7f8e296d1640 1 -- 172.21.15.29:0/4046804138 wait complete. 2023-10-14T16:57:34.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T16:57:34.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.760+0000 7fd915765640 1 Processor -- start 2023-10-14T16:57:34.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.760+0000 7fd915765640 1 -- start start 2023-10-14T16:57:34.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91015ad70 0x7fd910069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9100c9ca0 0x7fd91006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9100ca3b0 0x7fd9100ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100cad80 con 0x7fd9100ca3b0 2023-10-14T16:57:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9100cb030 con 0x7fd9100c9ca0 2023-10-14T16:57:34.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd915765640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100c91c0 con 0x7fd91015ad70 2023-10-14T16:57:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd90f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9100ca3b0 0x7fd9100ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd90f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9100ca3b0 0x7fd9100ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35728/0 (socket says 172.21.15.29:35728) 2023-10-14T16:57:34.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd90f7fe640 1 -- 172.21.15.29:0/1370043189 learned_addr learned my addr 172.21.15.29:0/1370043189 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd90effd640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91015ad70 0x7fd910069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.762+0000 7fd9067fc640 1 --2- 172.21.15.29:0/1370043189 >> v2:172.21.15.186:3300/0 conn(0x7fd9100c9ca0 0x7fd91006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.763+0000 7fd90f7fe640 1 -- 172.21.15.29:0/1370043189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91015ad70 msgr2=0x7fd910069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:34.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.763+0000 7fd90f7fe640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91015ad70 0x7fd910069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.763+0000 7fd90f7fe640 1 -- 172.21.15.29:0/1370043189 >> v2:172.21.15.186:3300/0 conn(0x7fd9100c9ca0 msgr2=0x7fd91006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.763+0000 7fd90f7fe640 1 --2- 172.21.15.29:0/1370043189 >> v2:172.21.15.186:3300/0 conn(0x7fd9100c9ca0 0x7fd91006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.763+0000 7fd90f7fe640 1 -- 172.21.15.29:0/1370043189 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9100c94a0 con 0x7fd9100ca3b0 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.764+0000 7fd90f7fe640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100ca3b0 0x7fd9100ca840 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd91006af00 tx=0x7fd90802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=466f09b0ae0f6d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.764+0000 7fd90e7fc640 1 -- 172.21.15.29:0/1370043189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908031870 con 0x7fd9100ca3b0 2023-10-14T16:57:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.764+0000 7fd90e7fc640 1 -- 172.21.15.29:0/1370043189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9080319d0 con 0x7fd9100ca3b0 2023-10-14T16:57:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.764+0000 7fd90e7fc640 1 -- 172.21.15.29:0/1370043189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd908031d60 con 0x7fd9100ca3b0 2023-10-14T16:57:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 -- 172.21.15.29:0/1370043189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100ca3b0 msgr2=0x7fd9100ca840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100ca3b0 0x7fd9100ca840 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7fd91006af00 tx=0x7fd90802b910 comp rx=0 tx=0).stop 2023-10-14T16:57:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 -- 172.21.15.29:0/1370043189 shutdown_connections 2023-10-14T16:57:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100ca3b0 0x7fd9100ca840 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 --2- 172.21.15.29:0/1370043189 >> v2:172.21.15.186:3300/0 conn(0x7fd9100c9ca0 0x7fd91006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 --2- 172.21.15.29:0/1370043189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd91015ad70 0x7fd910069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 -- 172.21.15.29:0/1370043189 >> 172.21.15.29:0/1370043189 conn(0x7fd9100ce9b0 msgr2=0x7fd9100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 -- 172.21.15.29:0/1370043189 shutdown_connections 2023-10-14T16:57:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.765+0000 7fd915765640 1 -- 172.21.15.29:0/1370043189 wait complete. 2023-10-14T16:57:34.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.767+0000 7fd915765640 1 Processor -- start 2023-10-14T16:57:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.767+0000 7fd915765640 1 -- start start 2023-10-14T16:57:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9100ca3b0 0x7fd9100cd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 0x7fd9100c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9100ccf90 con 0x7fd9100c9ca0 2023-10-14T16:57:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101517d0 con 0x7fd91015ad70 2023-10-14T16:57:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd915765640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9101519b0 con 0x7fd9100ca3b0 2023-10-14T16:57:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd90effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd90effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35742/0 (socket says 172.21.15.29:35742) 2023-10-14T16:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.768+0000 7fd90effd640 1 -- 172.21.15.29:0/1480480783 learned_addr learned my addr 172.21.15.29:0/1480480783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 -- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9100ca3b0 msgr2=0x7fd9100cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9100ca3b0 0x7fd9100cd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 -- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 msgr2=0x7fd9100c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90f7fe640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 0x7fd9100c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 0x7fd9100c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 -- 172.21.15.29:0/1480480783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9080094d0 con 0x7fd9100c9ca0 2023-10-14T16:57:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90f7fe640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 0x7fd9100c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd90effd640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d5f0 tx=0x7fd90000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9000125f0 con 0x7fd9100c9ca0 2023-10-14T16:57:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd900012750 con 0x7fd9100c9ca0 2023-10-14T16:57:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900012b00 con 0x7fd9100c9ca0 2023-10-14T16:57:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.769+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd910151c90 con 0x7fd9100c9ca0 2023-10-14T16:57:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.770+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9100cd1e0 con 0x7fd9100c9ca0 2023-10-14T16:57:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.771+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd91004e160 con 0x7fd9100c9ca0 2023-10-14T16:57:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.772+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd900012c60 con 0x7fd9100c9ca0 2023-10-14T16:57:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.773+0000 7fd907fff640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.773+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fd900089c70 con 0x7fd9100c9ca0 2023-10-14T16:57:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.773+0000 7fd9067fc640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.774+0000 7fd9067fc640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 0x7fd8f006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80070b0 tx=0x7fd8f80094b0 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:34.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.778+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd900054c20 con 0x7fd9100c9ca0 2023-10-14T16:57:34.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.975+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fd910069da0 con 0x7fd9100c9ca0 2023-10-14T16:57:34.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.980+0000 7fd907fff640 1 -- 172.21.15.29:0/1480480783 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v32) v1 ==== 120+0+0 (secure 0 0 0) 0x7fd900058240 con 0x7fd9100c9ca0 2023-10-14T16:57:34.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 msgr2=0x7fd8f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 0x7fd8f006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fd8f80070b0 tx=0x7fd8f80094b0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 msgr2=0x7fd910151290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:34.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7fd90000d5f0 tx=0x7fd90000dab0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 shutdown_connections 2023-10-14T16:57:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd91015ad70 0x7fd9100c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9100ca3b0 0x7fd9100cd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 --2- 172.21.15.29:0/1480480783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9100c9ca0 0x7fd910151290 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.983+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 >> 172.21.15.29:0/1480480783 conn(0x7fd9100ce9b0 msgr2=0x7fd91015c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.984+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 shutdown_connections 2023-10-14T16:57:34.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:34.984+0000 7fd915765640 1 -- 172.21.15.29:0/1480480783 wait complete. 2023-10-14T16:57:35.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T16:57:35.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.219+0000 7f5a18e44640 1 Processor -- start 2023-10-14T16:57:35.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.219+0000 7f5a18e44640 1 -- start start 2023-10-14T16:57:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.220+0000 7f5a18e44640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5a1415d150 0x7f5a140ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.220+0000 7f5a18e44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a140ac890 con 0x7f5a1406a4b0 2023-10-14T16:57:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a18e44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a14161650 con 0x7f5a14160b90 2023-10-14T16:57:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a18e44640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a1415d530 con 0x7f5a1415d150 2023-10-14T16:57:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a12d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5a1406a4b0 0x7f5a140c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43460/0 (socket says 172.21.15.29:43460) 2023-10-14T16:57:35.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a12d9d640 1 -- 172.21.15.29:0/3417858988 learned_addr learned my addr 172.21.15.29:0/3417858988 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a1259c640 1 --2- 172.21.15.29:0/3417858988 >> v2:172.21.15.186:3300/0 conn(0x7f5a1415d150 0x7f5a140ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392950009 0 0) 0x7f5a140ac890 con 0x7f5a1406a4b0 2023-10-14T16:57:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.221+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00003520 con 0x7f5a1406a4b0 2023-10-14T16:57:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2475844835 0 0) 0x7f5a00003520 con 0x7f5a1406a4b0 2023-10-14T16:57:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 >> v2:172.21.15.186:3300/0 conn(0x7f5a1415d150 msgr2=0x7f5a140ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 --2- 172.21.15.29:0/3417858988 >> v2:172.21.15.186:3300/0 conn(0x7f5a1415d150 0x7f5a140ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 >> v1:172.21.15.186:6789/0 conn(0x7f5a14160b90 legacy=0x7f5a14069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a14161930 con 0x7f5a1406a4b0 2023-10-14T16:57:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5a0c0019f0 con 0x7f5a1406a4b0 2023-10-14T16:57:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 >> v1:172.21.15.29:6789/0 conn(0x7f5a1406a4b0 legacy=0x7f5a140c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000039c0 0x7f5a00003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 0x7f5a00004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a00005180 0x7f5a0000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a1415d530 con 0x7f5a000047c0 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0000c080 con 0x7f5a00005180 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0000c240 con 0x7f5a000039c0 2023-10-14T16:57:35.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.222+0000 7f5a11d9b640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 0x7f5a00004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a12d9d640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a00005180 0x7f5a0000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a1259c640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000039c0 0x7f5a00003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a11d9b640 1 -- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000039c0 msgr2=0x7f5a00003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a11d9b640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000039c0 0x7f5a00003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a11d9b640 1 -- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a00005180 msgr2=0x7f5a0000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a11d9b640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a00005180 0x7f5a0000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a11d9b640 1 -- 172.21.15.29:0/3417858988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a00003520 con 0x7f5a000047c0 2023-10-14T16:57:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.223+0000 7f5a12d9d640 1 -- 172.21.15.29:0/3417858988 reap_dead start 2023-10-14T16:57:35.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.224+0000 7f5a11d9b640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 0x7f5a00004c40 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f5a04001750 tx=0x7f5a000057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=505506583ad4366f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.224+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a040052d0 con 0x7f5a000047c0 2023-10-14T16:57:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.224+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0402e050 con 0x7f5a000047c0 2023-10-14T16:57:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.224+0000 7f5a1159a640 1 -- 172.21.15.29:0/3417858988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a04003830 con 0x7f5a000047c0 2023-10-14T16:57:35.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 -- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 msgr2=0x7f5a00004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 0x7f5a00004c40 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7f5a04001750 tx=0x7f5a000057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 -- 172.21.15.29:0/3417858988 shutdown_connections 2023-10-14T16:57:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 --2- 172.21.15.29:0/3417858988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a000047c0 0x7f5a00004c40 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 -- 172.21.15.29:0/3417858988 >> 172.21.15.29:0/3417858988 conn(0x7f5a140ce9b0 msgr2=0x7f5a140cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:35.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 -- 172.21.15.29:0/3417858988 shutdown_connections 2023-10-14T16:57:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.225+0000 7f5a18e44640 1 -- 172.21.15.29:0/3417858988 wait complete. 2023-10-14T16:57:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.227+0000 7f5a18e44640 1 Processor -- start 2023-10-14T16:57:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.227+0000 7f5a18e44640 1 -- start start 2023-10-14T16:57:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000047c0 0x7f5a140b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b2ca0 0x7f5a140c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a141f3a80 con 0x7f5a140c50a0 2023-10-14T16:57:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140b3120 con 0x7f5a140b2ca0 2023-10-14T16:57:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a18e44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a140c5520 con 0x7f5a000047c0 2023-10-14T16:57:35.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a12d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a12d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35756/0 (socket says 172.21.15.29:35756) 2023-10-14T16:57:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.228+0000 7f5a12d9d640 1 -- 172.21.15.29:0/2797764729 learned_addr learned my addr 172.21.15.29:0/2797764729 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a11d9b640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b2ca0 0x7f5a140c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a1259c640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000047c0 0x7f5a140b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 -- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000047c0 msgr2=0x7f5a140b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000047c0 0x7f5a140b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 -- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b2ca0 msgr2=0x7f5a140c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b2ca0 0x7f5a140c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 -- 172.21.15.29:0/2797764729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a141f29e0 con 0x7f5a140c50a0 2023-10-14T16:57:35.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a12d9d640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5a140c8f90 tx=0x7f5a0c007bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c00eb00 con 0x7f5a140c50a0 2023-10-14T16:57:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a0c00ec80 con 0x7f5a140c50a0 2023-10-14T16:57:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a0c013420 con 0x7f5a140c50a0 2023-10-14T16:57:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.229+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a141f2cc0 con 0x7f5a140c50a0 2023-10-14T16:57:35.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.230+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a141f3280 con 0x7f5a140c50a0 2023-10-14T16:57:35.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.231+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a1404e160 con 0x7f5a140c50a0 2023-10-14T16:57:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.232+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5a0c0135a0 con 0x7f5a140c50a0 2023-10-14T16:57:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.233+0000 7f5a0affd640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 0x7f59f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.233+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f5a0c089c50 con 0x7f5a140c50a0 2023-10-14T16:57:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.233+0000 7f5a1259c640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 0x7f59f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.234+0000 7f5a1259c640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 0x7f59f406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a14069740 tx=0x7f59fc008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.240+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5a0c054bf0 con 0x7f5a140c50a0 2023-10-14T16:57:35.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.436+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f5a140ade10 con 0x7f5a140c50a0 2023-10-14T16:57:35.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.440+0000 7f5a0affd640 1 -- 172.21.15.29:0/2797764729 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v33) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5a0c058210 con 0x7f5a140c50a0 2023-10-14T16:57:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.443+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 msgr2=0x7f59f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.443+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 0x7f59f406d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f5a14069740 tx=0x7f59fc008040 comp rx=0 tx=0).stop 2023-10-14T16:57:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.443+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 msgr2=0x7f5a140c2cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.443+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f5a140c8f90 tx=0x7f5a0c007bc0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 shutdown_connections 2023-10-14T16:57:35.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f59f406af90 0x7f59f406d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5a140c50a0 0x7f5a140c2cd0 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5a140b2ca0 0x7f5a140c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 --2- 172.21.15.29:0/2797764729 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5a000047c0 0x7f5a140b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 >> 172.21.15.29:0/2797764729 conn(0x7f5a140ce9b0 msgr2=0x7f5a14070e00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 shutdown_connections 2023-10-14T16:57:35.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.444+0000 7f5a18e44640 1 -- 172.21.15.29:0/2797764729 wait complete. 2023-10-14T16:57:35.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list' 2023-10-14T16:57:35.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.680+0000 7f84bb303640 1 Processor -- start 2023-10-14T16:57:35.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.680+0000 7f84bb303640 1 -- start start 2023-10-14T16:57:35.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.681+0000 7f84bb303640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84b40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84bb303640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b415b9d0 con 0x7f84b40ccfb0 2023-10-14T16:57:35.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84bb303640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b415bcb0 con 0x7f84b406c040 2023-10-14T16:57:35.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84bb303640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415bf90 con 0x7f84b40ad970 2023-10-14T16:57:35.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84bab02640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84b40ccfb0 0x7f84b40cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43466/0 (socket says 172.21.15.29:43466) 2023-10-14T16:57:35.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84bab02640 1 -- 172.21.15.29:0/2354169865 learned_addr learned my addr 172.21.15.29:0/2354169865 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:35.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84b9b00640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84b40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.682+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851152089 0 0) 0x7f84b415b9d0 con 0x7f84b40ccfb0 2023-10-14T16:57:35.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a0003520 con 0x7f84b40ccfb0 2023-10-14T16:57:35.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2734100006 0 0) 0x7f84a0003520 con 0x7f84b40ccfb0 2023-10-14T16:57:35.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 msgr2=0x7f84b40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84b40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 >> v1:172.21.15.186:6789/0 conn(0x7f84b406c040 legacy=0x7f84b406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b40cc270 con 0x7f84b40ccfb0 2023-10-14T16:57:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f84ac0019f0 con 0x7f84b40ccfb0 2023-10-14T16:57:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 >> v1:172.21.15.29:6789/0 conn(0x7f84b40ccfb0 legacy=0x7f84b40cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a00039a0 0x7f84a0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84a00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a0004d00 0x7f84a0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415bf90 con 0x7f84a00039a0 2023-10-14T16:57:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a000ba60 con 0x7f84a0004d00 2023-10-14T16:57:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a000bc20 con 0x7f84b40ad970 2023-10-14T16:57:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.683+0000 7f84ba301640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a00039a0 0x7f84a0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84b9b00640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84a00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84bab02640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a0004d00 0x7f84a0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84ba301640 1 -- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 msgr2=0x7f84a00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84ba301640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b40ad970 0x7f84a00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84ba301640 1 -- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a0004d00 msgr2=0x7f84a0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:35.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84ba301640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a0004d00 0x7f84a0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.684+0000 7f84ba301640 1 -- 172.21.15.29:0/2354169865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a0003520 con 0x7f84a00039a0 2023-10-14T16:57:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.685+0000 7f84ba301640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a00039a0 0x7f84a0003dc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84b0001750 tx=0x7f84a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fd0a648d76c5bb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00052d0 con 0x7f84a00039a0 2023-10-14T16:57:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b002e050 con 0x7f84a00039a0 2023-10-14T16:57:35.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84b92ff640 1 -- 172.21.15.29:0/2354169865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b00035f0 con 0x7f84a00039a0 2023-10-14T16:57:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 -- 172.21.15.29:0/2354169865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a00039a0 msgr2=0x7f84a0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 --2- 172.21.15.29:0/2354169865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a00039a0 0x7f84a0003dc0 secure :-1 s=READY pgs=1174 cs=0 l=1 rev1=1 crypto rx=0x7f84b0001750 tx=0x7f84a00057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bab02640 1 -- 172.21.15.29:0/2354169865 reap_dead start 2023-10-14T16:57:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 -- 172.21.15.29:0/2354169865 shutdown_connections 2023-10-14T16:57:35.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 -- 172.21.15.29:0/2354169865 >> 172.21.15.29:0/2354169865 conn(0x7f84b40ce6a0 msgr2=0x7f84b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 -- 172.21.15.29:0/2354169865 shutdown_connections 2023-10-14T16:57:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.686+0000 7f84bb303640 1 -- 172.21.15.29:0/2354169865 wait complete. 2023-10-14T16:57:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.688+0000 7f84bb303640 1 Processor -- start 2023-10-14T16:57:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.688+0000 7f84bb303640 1 -- start start 2023-10-14T16:57:35.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00039a0 0x7f84b40c7600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 0x7f84b415da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b415e3f0 con 0x7f84b415df70 2023-10-14T16:57:35.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84b40c7b40 con 0x7f84a00039a0 2023-10-14T16:57:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bb303640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b4153180 con 0x7f84b414f320 2023-10-14T16:57:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bab02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bab02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35770/0 (socket says 172.21.15.29:35770) 2023-10-14T16:57:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.689+0000 7f84bab02640 1 -- 172.21.15.29:0/3938382442 learned_addr learned my addr 172.21.15.29:0/3938382442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:35.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84b9b00640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 0x7f84b415da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84ba301640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00039a0 0x7f84b40c7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 -- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 msgr2=0x7f84b415da30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 0x7f84b415da30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 -- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00039a0 msgr2=0x7f84b40c7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:35.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00039a0 0x7f84b40c7600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 -- 172.21.15.29:0/3938382442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b40cdcd0 con 0x7f84b415df70 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84b9b00640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 0x7f84b415da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bab02640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f84b40cd390 tx=0x7f84ac00e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00ec10 con 0x7f84b415df70 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84ac00ed70 con 0x7f84b415df70 2023-10-14T16:57:35.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84ac00eee0 con 0x7f84b415df70 2023-10-14T16:57:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.690+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b4153460 con 0x7f84b415df70 2023-10-14T16:57:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.691+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b4153a20 con 0x7f84b415df70 2023-10-14T16:57:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.692+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b406b330 con 0x7f84b415df70 2023-10-14T16:57:35.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.692+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f84ac013660 con 0x7f84b415df70 2023-10-14T16:57:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.694+0000 7f84aaffd640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 0x7f849806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:35.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.694+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f84ac089c40 con 0x7f84b415df70 2023-10-14T16:57:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.694+0000 7f84ba301640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 0x7f849806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.695+0000 7f84ba301640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 0x7f849806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84b0033000 tx=0x7f84b0009030 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:35.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.698+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84ac054c20 con 0x7f84b415df70 2023-10-14T16:57:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:35.965+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]} v 0) v1 -- 0x7f84b404e9b0 con 0x7f84b415df70 2023-10-14T16:57:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.165+0000 7f84aaffd640 1 -- 172.21.15.29:0/3938382442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_non_existent_snap_schedule_list"]}]=0 v647) v1 ==== 166+0+0 (secure 0 0 0) 0x7f84ac058240 con 0x7f84b415df70 2023-10-14T16:57:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 msgr2=0x7f849806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 0x7f849806d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f84b0033000 tx=0x7f84b0009030 comp rx=0 tx=0).stop 2023-10-14T16:57:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 msgr2=0x7f84b40c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 crypto rx=0x7f84b40cd390 tx=0x7f84ac00e530 comp rx=0 tx=0).stop 2023-10-14T16:57:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 shutdown_connections 2023-10-14T16:57:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f849806af90 0x7f849806d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84b415df70 0x7f84b40c8cf0 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84b414f320 0x7f84b415da30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 --2- 172.21.15.29:0/3938382442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a00039a0 0x7f84b40c7600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.168+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 >> 172.21.15.29:0/3938382442 conn(0x7f84b40ce6a0 msgr2=0x7f84b415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.169+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 shutdown_connections 2023-10-14T16:57:36.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.169+0000 7f84bb303640 1 -- 172.21.15.29:0/3938382442 wait complete. 2023-10-14T16:57:36.190 INFO:tasks.cephfs_test_runner:Test listing snap schedules on a non-existing filesystem path failure ... ok 2023-10-14T16:57:36.190 INFO:tasks.cephfs_test_runner:Starting test: test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that a non-existent path leads to schedule deactivation after a few retries. 2023-10-14T16:57:36.191 INFO:tasks.cephfs_test_runner:test_schedule_auto_deactivation_for_non_existent_path (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:57:36.191 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:36.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.414+0000 7f6e232e5640 1 Processor -- start 2023-10-14T16:57:36.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.414+0000 7f6e232e5640 1 -- start start 2023-10-14T16:57:36.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.415+0000 7f6e232e5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.415+0000 7f6e232e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c15aef0 con 0x7f6e1c070450 2023-10-14T16:57:36.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.415+0000 7f6e232e5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e1c15d910 con 0x7f6e1c06c810 2023-10-14T16:57:36.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.415+0000 7f6e232e5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e1c15e490 con 0x7f6e1c159950 2023-10-14T16:57:36.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.416+0000 7f6e21ae2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6e1c070450 0x7f6e1c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43482/0 (socket says 172.21.15.29:43482) 2023-10-14T16:57:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.416+0000 7f6e21ae2640 1 -- 172.21.15.29:0/2497235385 learned_addr learned my addr 172.21.15.29:0/2497235385 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.416+0000 7f6e222e3640 1 --2- 172.21.15.29:0/2497235385 >> v2:172.21.15.186:3300/0 conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.416+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1018279490 0 0) 0x7f6e0c001110 con 0x7f6e1c070450 2023-10-14T16:57:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.416+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e08003520 con 0x7f6e1c070450 2023-10-14T16:57:36.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.417+0000 7f6e222e3640 1 -- 172.21.15.29:0/2497235385 >> v1:172.21.15.186:6789/0 conn(0x7f6e1c06c810 legacy=0x7f6e1c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.417+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3670198849 0 0) 0x7f6e08003520 con 0x7f6e1c070450 2023-10-14T16:57:36.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.417+0000 7f6e222e3640 1 -- 172.21.15.29:0/2497235385 >> v1:172.21.15.29:6789/0 conn(0x7f6e1c070450 legacy=0x7f6e1c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.417+0000 7f6e222e3640 1 -- 172.21.15.29:0/2497235385 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e1c15dbf0 con 0x7f6e1c159950 2023-10-14T16:57:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.418+0000 7f6e222e3640 1 --2- 172.21.15.29:0/2497235385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e180097a0 tx=0x7f6e1802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f39d90150c4a3118 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e0c001800 con 0x7f6e1c159950 2023-10-14T16:57:36.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e1802e050 con 0x7f6e1c159950 2023-10-14T16:57:36.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e212e1640 1 -- 172.21.15.29:0/2497235385 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e18030730 con 0x7f6e1c159950 2023-10-14T16:57:36.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 -- 172.21.15.29:0/2497235385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c159950 msgr2=0x7f6e1c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2497235385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6e180097a0 tx=0x7f6e1802b790 comp rx=0 tx=0).stop 2023-10-14T16:57:36.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 -- 172.21.15.29:0/2497235385 shutdown_connections 2023-10-14T16:57:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2497235385 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c159950 0x7f6e1c15a9b0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 -- 172.21.15.29:0/2497235385 >> 172.21.15.29:0/2497235385 conn(0x7f6e1c0ce6a0 msgr2=0x7f6e1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 -- 172.21.15.29:0/2497235385 shutdown_connections 2023-10-14T16:57:36.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.419+0000 7f6e232e5640 1 -- 172.21.15.29:0/2497235385 wait complete. 2023-10-14T16:57:36.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.421+0000 7f6e232e5640 1 Processor -- start 2023-10-14T16:57:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.421+0000 7f6e232e5640 1 -- start start 2023-10-14T16:57:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c06c810 0x7f6e1c15b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e1c070450 0x7f6e1c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0aefd0 con 0x7f6e1c159950 2023-10-14T16:57:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0ae450 con 0x7f6e1c06c810 2023-10-14T16:57:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.422+0000 7f6e232e5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e1c0ae730 con 0x7f6e1c070450 2023-10-14T16:57:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35772/0 (socket says 172.21.15.29:35772) 2023-10-14T16:57:36.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 -- 172.21.15.29:0/2461297400 learned_addr learned my addr 172.21.15.29:0/2461297400 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e222e3640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c06c810 0x7f6e1c15b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e21ae2640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e1c070450 0x7f6e1c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 -- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e1c070450 msgr2=0x7f6e1c0acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e1c070450 0x7f6e1c0acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 -- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c06c810 msgr2=0x7f6e1c15b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c06c810 0x7f6e1c15b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.423+0000 7f6e22ae4640 1 -- 172.21.15.29:0/2461297400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e18009400 con 0x7f6e1c159950 2023-10-14T16:57:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e22ae4640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c7a0 tx=0x7f6e1400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14003b70 con 0x7f6e1c159950 2023-10-14T16:57:36.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e14003cf0 con 0x7f6e1c159950 2023-10-14T16:57:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e1c0aea10 con 0x7f6e1c159950 2023-10-14T16:57:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e14003ea0 con 0x7f6e1c159950 2023-10-14T16:57:36.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.424+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e1c0ae070 con 0x7f6e1c159950 2023-10-14T16:57:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.425+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e1c04e9b0 con 0x7f6e1c159950 2023-10-14T16:57:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.427+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6e1400aa90 con 0x7f6e1c159950 2023-10-14T16:57:36.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.428+0000 7f6e12ffd640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 0x7f6e0006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.428+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f6e140895a0 con 0x7f6e1c159950 2023-10-14T16:57:36.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.428+0000 7f6e222e3640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 0x7f6e0006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.429+0000 7f6e222e3640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 0x7f6e0006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e18034000 tx=0x7f6e1802f000 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.433+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6e14054560 con 0x7f6e1c159950 2023-10-14T16:57:36.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.630+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6e1c1f27d0 con 0x7f6e1c159950 2023-10-14T16:57:36.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.633+0000 7f6e12ffd640 1 -- 172.21.15.29:0/2461297400 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+14046 (secure 0 0 0) 0x7f6e14057b80 con 0x7f6e1c159950 2023-10-14T16:57:36.635 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:36.635 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":101,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:34.025251+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6836/1398561557":"2023-10-15T16:57:31.005727+0000","172.21.15.29:0/559917725":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6833/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/874374761":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/2164323269":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3749138999":"2023-10-15T16:57:27.162005+0000","172.21.15.29:6832/505146228":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3356363773":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6833/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6832/3330033844":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2136904241":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3967724404":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/2171619166":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/3763524886":"2023-10-15T16:57:27.162005+0000","172.21.15.29:0/3505521487":"2023-10-15T16:57:20.630995+0000","172.21.15.29:6837/1398561557":"2023-10-15T16:57:31.005727+0000","172.21.15.29:0/3693879118":"2023-10-15T16:57:20.630995+0000","172.21.15.29:0/4176788283":"2023-10-15T16:57:20.630995+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:36.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 msgr2=0x7f6e0006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 0x7f6e0006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e18034000 tx=0x7f6e1802f000 comp rx=0 tx=0).stop 2023-10-14T16:57:36.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 msgr2=0x7f6e1c0c47c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 secure :-1 s=READY pgs=1177 cs=0 l=1 rev1=1 crypto rx=0x7f6e1400c7a0 tx=0x7f6e1400cc60 comp rx=0 tx=0).stop 2023-10-14T16:57:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 shutdown_connections 2023-10-14T16:57:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f6e0006af90 0x7f6e0006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6e1c159950 0x7f6e1c0c47c0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6e1c070450 0x7f6e1c0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 --2- 172.21.15.29:0/2461297400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6e1c06c810 0x7f6e1c15b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 >> 172.21.15.29:0/2461297400 conn(0x7f6e1c0ce6a0 msgr2=0x7f6e1c072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 shutdown_connections 2023-10-14T16:57:36.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.636+0000 7f6e232e5640 1 -- 172.21.15.29:0/2461297400 wait complete. 2023-10-14T16:57:36.658 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:57:36.658 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:57:36.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.888+0000 7f108e5fb640 1 Processor -- start 2023-10-14T16:57:36.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.889+0000 7f108e5fb640 1 -- start start 2023-10-14T16:57:36.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1088159ab0 0x7f10880ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1088159e90 0x7f108806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f108806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108806cb10 con 0x7f1088159e90 2023-10-14T16:57:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f108815d890 con 0x7f1088159ab0 2023-10-14T16:57:36.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.890+0000 7f108e5fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880cd1f0 con 0x7f108806c140 2023-10-14T16:57:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1088159e90 0x7f108806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1088159e90 0x7f108806fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35784/0 (socket says 172.21.15.29:35784) 2023-10-14T16:57:36.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 -- 172.21.15.29:0/446324913 learned_addr learned my addr 172.21.15.29:0/446324913 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 -- 172.21.15.29:0/446324913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 msgr2=0x7f108806c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f108cb98640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f108806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f1087fff640 1 --2- 172.21.15.29:0/446324913 >> v2:172.21.15.186:3300/0 conn(0x7f1088159ab0 0x7f10880ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f108806c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 -- 172.21.15.29:0/446324913 >> v2:172.21.15.186:3300/0 conn(0x7f1088159ab0 msgr2=0x7f10880ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 --2- 172.21.15.29:0/446324913 >> v2:172.21.15.186:3300/0 conn(0x7f1088159ab0 0x7f10880ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f107ffff640 1 -- 172.21.15.29:0/446324913 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10880cd4d0 con 0x7f1088159e90 2023-10-14T16:57:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.891+0000 7f108cb98640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f108806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.892+0000 7f107ffff640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f108806fc50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f10880cfa00 tx=0x7f107002b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=847fcc8ab29e52fd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.893+0000 7f10877fe640 1 -- 172.21.15.29:0/446324913 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070031750 con 0x7f1088159e90 2023-10-14T16:57:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.893+0000 7f10877fe640 1 -- 172.21.15.29:0/446324913 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10700318b0 con 0x7f1088159e90 2023-10-14T16:57:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.893+0000 7f10877fe640 1 -- 172.21.15.29:0/446324913 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1070031c40 con 0x7f1088159e90 2023-10-14T16:57:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 -- 172.21.15.29:0/446324913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 msgr2=0x7f108806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f108806fc50 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 crypto rx=0x7f10880cfa00 tx=0x7f107002b7f0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 -- 172.21.15.29:0/446324913 shutdown_connections 2023-10-14T16:57:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f108806c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 --2- 172.21.15.29:0/446324913 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f108806fc50 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 --2- 172.21.15.29:0/446324913 >> v2:172.21.15.186:3300/0 conn(0x7f1088159ab0 0x7f10880ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 -- 172.21.15.29:0/446324913 >> 172.21.15.29:0/446324913 conn(0x7f10880ce9b0 msgr2=0x7f10880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 -- 172.21.15.29:0/446324913 shutdown_connections 2023-10-14T16:57:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.894+0000 7f108e5fb640 1 -- 172.21.15.29:0/446324913 wait complete. 2023-10-14T16:57:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.896+0000 7f108e5fb640 1 Processor -- start 2023-10-14T16:57:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.896+0000 7f108e5fb640 1 -- start start 2023-10-14T16:57:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f10880c2670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1088159ab0 0x7f108806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108806b660 con 0x7f1088159e90 2023-10-14T16:57:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f108806bfc0 con 0x7f1088159ab0 2023-10-14T16:57:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108e5fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10880c3550 con 0x7f108806c140 2023-10-14T16:57:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108cb98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108cb98640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35796/0 (socket says 172.21.15.29:35796) 2023-10-14T16:57:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.897+0000 7f108cb98640 1 -- 172.21.15.29:0/2458782186 learned_addr learned my addr 172.21.15.29:0/2458782186 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f1087fff640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f10880c2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f107ffff640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1088159ab0 0x7f108806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 -- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 msgr2=0x7f10880c2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f10880c2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 -- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1088159ab0 msgr2=0x7f108806ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1088159ab0 0x7f108806ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 -- 172.21.15.29:0/2458782186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10700094d0 con 0x7f1088159e90 2023-10-14T16:57:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f1087fff640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f10880c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108cb98640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f108000cd40 tx=0x7f108000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108000f680 con 0x7f1088159e90 2023-10-14T16:57:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f108000f800 con 0x7f1088159e90 2023-10-14T16:57:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f108000fbb0 con 0x7f1088159e90 2023-10-14T16:57:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.898+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f108806b030 con 0x7f1088159e90 2023-10-14T16:57:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.899+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10880c3a60 con 0x7f1088159e90 2023-10-14T16:57:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.900+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f108806b7a0 con 0x7f1088159e90 2023-10-14T16:57:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.901+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f108000b040 con 0x7f1088159e90 2023-10-14T16:57:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.902+0000 7f10857fa640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 0x7f106806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.902+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f108008a280 con 0x7f1088159e90 2023-10-14T16:57:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.902+0000 7f1087fff640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 0x7f106806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.903+0000 7f1087fff640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 0x7f106806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1088159200 tx=0x7f1078008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:36.906+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1080055090 con 0x7f1088159e90 2023-10-14T16:57:37.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.102+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f10880adf20 con 0x7f1088159e90 2023-10-14T16:57:37.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.103+0000 7f10857fa640 1 -- 172.21.15.29:0/2458782186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) v1 ==== 93+0+10 (secure 0 0 0) 0x7f10800586b0 con 0x7f1088159e90 2023-10-14T16:57:37.105 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:57:37.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.105+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 msgr2=0x7f106806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.105+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 0x7f106806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1088159200 tx=0x7f1078008040 comp rx=0 tx=0).stop 2023-10-14T16:57:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.105+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 msgr2=0x7f10880c3010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.105+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 crypto rx=0x7f108000cd40 tx=0x7f108000a650 comp rx=0 tx=0).stop 2023-10-14T16:57:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.105+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 shutdown_connections 2023-10-14T16:57:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f106806afc0 0x7f106806d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1088159e90 0x7f10880c3010 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1088159ab0 0x7f108806ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 --2- 172.21.15.29:0/2458782186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f108806c140 0x7f10880c2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 >> 172.21.15.29:0/2458782186 conn(0x7f10880ce9b0 msgr2=0x7f10880ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:37.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 shutdown_connections 2023-10-14T16:57:37.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.106+0000 7f108e5fb640 1 -- 172.21.15.29:0/2458782186 wait complete. 2023-10-14T16:57:37.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-14T16:57:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.355+0000 7fa73e061640 1 Processor -- start 2023-10-14T16:57:37.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.355+0000 7fa73e061640 1 -- start start 2023-10-14T16:57:37.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.357+0000 7fa73e061640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa738070fd0 0x7fa7380b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:37.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.357+0000 7fa73e061640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa738071450 con 0x7fa7380adf30 2023-10-14T16:57:37.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.357+0000 7fa73e061640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7380cc600 con 0x7fa7380adb50 2023-10-14T16:57:37.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.357+0000 7fa73e061640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7380cc8e0 con 0x7fa738070fd0 2023-10-14T16:57:37.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa736ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa738070fd0 0x7fa7380b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:37.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa7377fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa7380adb50 0x7fa73806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45616/0 (socket says 172.21.15.29:45616) 2023-10-14T16:57:37.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa7377fe640 1 -- 172.21.15.29:0/953070902 learned_addr learned my addr 172.21.15.29:0/953070902 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:37.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa737fff640 1 --1- 172.21.15.29:0/953070902 >> v1:172.21.15.29:6789/0 conn(0x7fa7380adf30 0x7fa738164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43488/0 (socket says 172.21.15.29:43488) 2023-10-14T16:57:37.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa736ffd640 1 -- 172.21.15.29:0/953070902 >> v1:172.21.15.186:6789/0 conn(0x7fa7380adb50 legacy=0x7fa73806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa736ffd640 1 -- 172.21.15.29:0/953070902 >> v1:172.21.15.29:6789/0 conn(0x7fa7380adf30 legacy=0x7fa738164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa736ffd640 1 -- 172.21.15.29:0/953070902 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7380ccbc0 con 0x7fa738070fd0 2023-10-14T16:57:37.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.358+0000 7fa7367fc640 1 -- 172.21.15.29:0/953070902 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151641350 0 0) 0x7fa738071450 con 0x7fa7380adf30 2023-10-14T16:57:37.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.359+0000 7fa736ffd640 1 --2- 172.21.15.29:0/953070902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa738070fd0 0x7fa7380b0410 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa7200097a0 tx=0x7fa72002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa98d7342651e5dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:37.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.360+0000 7fa7367fc640 1 -- 172.21.15.29:0/953070902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720030610 con 0x7fa738070fd0 2023-10-14T16:57:37.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.360+0000 7fa7367fc640 1 -- 172.21.15.29:0/953070902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720030770 con 0x7fa738070fd0 2023-10-14T16:57:37.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.360+0000 7fa7367fc640 1 -- 172.21.15.29:0/953070902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720030b00 con 0x7fa738070fd0 2023-10-14T16:57:37.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.360+0000 7fa73e061640 1 -- 172.21.15.29:0/953070902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa738070fd0 msgr2=0x7fa7380b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.360+0000 7fa73e061640 1 --2- 172.21.15.29:0/953070902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa738070fd0 0x7fa7380b0410 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 crypto rx=0x7fa7200097a0 tx=0x7fa72002b960 comp rx=0 tx=0).stop 2023-10-14T16:57:37.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.361+0000 7fa73e061640 1 -- 172.21.15.29:0/953070902 shutdown_connections 2023-10-14T16:57:37.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.361+0000 7fa73e061640 1 --2- 172.21.15.29:0/953070902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa738070fd0 0x7fa7380b0410 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.361+0000 7fa73e061640 1 -- 172.21.15.29:0/953070902 >> 172.21.15.29:0/953070902 conn(0x7fa7380cb410 msgr2=0x7fa7380cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:37.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.361+0000 7fa73e061640 1 -- 172.21.15.29:0/953070902 shutdown_connections 2023-10-14T16:57:37.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.361+0000 7fa73e061640 1 -- 172.21.15.29:0/953070902 wait complete. 2023-10-14T16:57:37.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.363+0000 7fa73e061640 1 Processor -- start 2023-10-14T16:57:37.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.363+0000 7fa73e061640 1 -- start start 2023-10-14T16:57:37.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 0x7fa7381f65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:37.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:37.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7380adf30 0x7fa7380c4910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:37.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380c3c40 con 0x7fa7380adb50 2023-10-14T16:57:37.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7381f6b20 con 0x7fa7380adf30 2023-10-14T16:57:37.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa73e061640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7380c2fc0 con 0x7fa738070fd0 2023-10-14T16:57:37.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa736ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:37.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa736ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35804/0 (socket says 172.21.15.29:35804) 2023-10-14T16:57:37.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa736ffd640 1 -- 172.21.15.29:0/123959074 learned_addr learned my addr 172.21.15.29:0/123959074 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:37.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa7377fe640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 0x7fa7381f65e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:37.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.364+0000 7fa737fff640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7380adf30 0x7fa7380c4910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:37.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 -- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 msgr2=0x7fa7381f65e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 0x7fa7381f65e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 -- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7380adf30 msgr2=0x7fa7380c4910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:37.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7380adf30 0x7fa7380c4910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:37.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 -- 172.21.15.29:0/123959074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa720009400 con 0x7fa7380adb50 2023-10-14T16:57:37.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa7377fe640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 0x7fa7381f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:37.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa736ffd640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa72002bf70 tx=0x7fa7200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:37.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.365+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72003e070 con 0x7fa7380adb50 2023-10-14T16:57:37.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.366+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa720039a10 con 0x7fa7380adb50 2023-10-14T16:57:37.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.366+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa720039da0 con 0x7fa7380adb50 2023-10-14T16:57:37.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.366+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7380c32a0 con 0x7fa7380adb50 2023-10-14T16:57:37.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.366+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7380c3860 con 0x7fa7380adb50 2023-10-14T16:57:37.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.367+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7380c3d80 con 0x7fa7380adb50 2023-10-14T16:57:37.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.368+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa72004b400 con 0x7fa7380adb50 2023-10-14T16:57:37.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.369+0000 7fa72ffff640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 0x7fa71806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:37.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.369+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fa7200abdb0 con 0x7fa7380adb50 2023-10-14T16:57:37.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.369+0000 7fa7377fe640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 0x7fa71806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:37.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.370+0000 7fa7377fe640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 0x7fa71806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa728005880 tx=0x7fa728009040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:37.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.373+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7380c3d80 con 0x7fa7380adb50 2023-10-14T16:57:37.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:37.638+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7fa7380c3d80 con 0x7fa7380adb50 2023-10-14T16:57:38.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.178+0000 7fa72ffff640 1 -- 172.21.15.29:0/123959074 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v649) v1 ==== 186+0+0 (secure 0 0 0) 0x7fa7380c3d80 con 0x7fa7380adb50 2023-10-14T16:57:38.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 msgr2=0x7fa71806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 0x7fa71806d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa728005880 tx=0x7fa728009040 comp rx=0 tx=0).stop 2023-10-14T16:57:38.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 msgr2=0x7fa7380c43d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 secure :-1 s=READY pgs=1182 cs=0 l=1 rev1=1 crypto rx=0x7fa72002bf70 tx=0x7fa7200046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 shutdown_connections 2023-10-14T16:57:38.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fa71806af90 0x7fa71806d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7380adf30 0x7fa7380c4910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7380adb50 0x7fa7380c43d0 unknown :-1 s=CLOSED pgs=1182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 --2- 172.21.15.29:0/123959074 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa738070fd0 0x7fa7381f65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.181+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 >> 172.21.15.29:0/123959074 conn(0x7fa7380cb410 msgr2=0x7fa7380ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.182+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 shutdown_connections 2023-10-14T16:57:38.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.182+0000 7fa73e061640 1 -- 172.21.15.29:0/123959074 wait complete. 2023-10-14T16:57:38.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:57:38.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.434+0000 7fe4ba4ec640 1 Processor -- start 2023-10-14T16:57:38.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.434+0000 7fe4ba4ec640 1 -- start start 2023-10-14T16:57:38.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4b406c040 0x7fe4b406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 0x7fe4b415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4b415ebc0 0x7fe4b415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b406c960 con 0x7fe4b415ebc0 2023-10-14T16:57:38.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4b415b9d0 con 0x7fe4b406c040 2023-10-14T16:57:38.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.435+0000 7fe4ba4ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b415bcb0 con 0x7fe4b40ad970 2023-10-14T16:57:38.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4b415ebc0 0x7fe4b415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:38.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe4b415ebc0 0x7fe4b415f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35816/0 (socket says 172.21.15.29:35816) 2023-10-14T16:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 -- 172.21.15.29:0/1588948769 learned_addr learned my addr 172.21.15.29:0/1588948769 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b8ce9640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 0x7fe4b415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 -- 172.21.15.29:0/1588948769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 msgr2=0x7fe4b415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 0x7fe4b415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 -- 172.21.15.29:0/1588948769 >> v2:172.21.15.186:3300/0 conn(0x7fe4b406c040 msgr2=0x7fe4b406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b94ea640 1 --2- 172.21.15.29:0/1588948769 >> v2:172.21.15.186:3300/0 conn(0x7fe4b406c040 0x7fe4b406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 --2- 172.21.15.29:0/1588948769 >> v2:172.21.15.186:3300/0 conn(0x7fe4b406c040 0x7fe4b406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b9ceb640 1 -- 172.21.15.29:0/1588948769 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b415bf90 con 0x7fe4b415ebc0 2023-10-14T16:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.436+0000 7fe4b8ce9640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 0x7fe4b415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:38.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.437+0000 7fe4b9ceb640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b415ebc0 0x7fe4b415f050 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009870 tx=0x7fe4ac02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b31cc4527ec9bc43 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:38.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4abfff640 1 -- 172.21.15.29:0/1588948769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac031640 con 0x7fe4b415ebc0 2023-10-14T16:57:38.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4abfff640 1 -- 172.21.15.29:0/1588948769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ac0317a0 con 0x7fe4b415ebc0 2023-10-14T16:57:38.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4abfff640 1 -- 172.21.15.29:0/1588948769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ac031b30 con 0x7fe4b415ebc0 2023-10-14T16:57:38.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/1588948769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b415ebc0 msgr2=0x7fe4b415f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b415ebc0 0x7fe4b415f050 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 crypto rx=0x7fe4ac009870 tx=0x7fe4ac02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/1588948769 shutdown_connections 2023-10-14T16:57:38.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b415ebc0 0x7fe4b415f050 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/1588948769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b40ad970 0x7fe4b415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/1588948769 >> v2:172.21.15.186:3300/0 conn(0x7fe4b406c040 0x7fe4b406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/1588948769 >> 172.21.15.29:0/1588948769 conn(0x7fe4b40ce6a0 msgr2=0x7fe4b40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.438+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/1588948769 shutdown_connections 2023-10-14T16:57:38.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.439+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/1588948769 wait complete. 2023-10-14T16:57:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.440+0000 7fe4ba4ec640 1 Processor -- start 2023-10-14T16:57:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.440+0000 7fe4ba4ec640 1 -- start start 2023-10-14T16:57:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4b40ad970 0x7fe4b414d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b415ebc0 0x7fe4b414dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b20a0 con 0x7fe4b406c040 2023-10-14T16:57:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40ad360 con 0x7fe4b40ad970 2023-10-14T16:57:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.441+0000 7fe4ba4ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4b40b2300 con 0x7fe4b415ebc0 2023-10-14T16:57:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:38.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35832/0 (socket says 172.21.15.29:35832) 2023-10-14T16:57:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 -- 172.21.15.29:0/3116632962 learned_addr learned my addr 172.21.15.29:0/3116632962 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 -- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b415ebc0 msgr2=0x7fe4b414dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b415ebc0 0x7fe4b414dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 -- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4b40ad970 msgr2=0x7fe4b414d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4b40ad970 0x7fe4b414d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 -- 172.21.15.29:0/3116632962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4ac0094d0 con 0x7fe4b406c040 2023-10-14T16:57:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.442+0000 7fe4b94ea640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000cb80 tx=0x7fe4b000cf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.443+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0003b90 con 0x7fe4b406c040 2023-10-14T16:57:38.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.443+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4b0003cf0 con 0x7fe4b406c040 2023-10-14T16:57:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.443+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0003ea0 con 0x7fe4b406c040 2023-10-14T16:57:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.443+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4b40b25e0 con 0x7fe4b406c040 2023-10-14T16:57:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.443+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4b40ad010 con 0x7fe4b406c040 2023-10-14T16:57:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.444+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4b406b330 con 0x7fe4b406c040 2023-10-14T16:57:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.445+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe4b000aad0 con 0x7fe4b406c040 2023-10-14T16:57:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.446+0000 7fe4a9ffb640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 0x7fe49806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.446+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fe4b0089710 con 0x7fe4b406c040 2023-10-14T16:57:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.446+0000 7fe4b8ce9640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 0x7fe49806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.447+0000 7fe4b8ce9640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 0x7fe49806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4006fd0 tx=0x7fe4a4008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:38.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.450+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4b0054520 con 0x7fe4b406c040 2023-10-14T16:57:38.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.646+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe4b40b2b00 con 0x7fe4b406c040 2023-10-14T16:57:38.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.653+0000 7fe4a9ffb640 1 -- 172.21.15.29:0/3116632962 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v34)=0 v34) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe4b40b2b00 con 0x7fe4b406c040 2023-10-14T16:57:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 msgr2=0x7fe49806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 0x7fe49806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe4a4006fd0 tx=0x7fe4a4008040 comp rx=0 tx=0).stop 2023-10-14T16:57:38.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 msgr2=0x7fe4b415a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 crypto rx=0x7fe4b000cb80 tx=0x7fe4b000cf80 comp rx=0 tx=0).stop 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 shutdown_connections 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fe49806afc0 0x7fe49806d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4b415ebc0 0x7fe4b414dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4b40ad970 0x7fe4b414d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 --2- 172.21.15.29:0/3116632962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4b406c040 0x7fe4b415a490 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:38.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 >> 172.21.15.29:0/3116632962 conn(0x7fe4b40ce6a0 msgr2=0x7fe4b40706e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:38.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 shutdown_connections 2023-10-14T16:57:38.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:38.656+0000 7fe4ba4ec640 1 -- 172.21.15.29:0/3116632962 wait complete. 2023-10-14T16:57:38.677 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:57:38.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:57:38.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:57:38.920 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:57:38.920 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:38.921 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:38.983 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:57:38.984 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:57:38.984 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:39.046 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:57:39.046 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:39.046 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:39.110 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:57:39.111 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:57:39.112 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:39.172 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:57:39.172 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:57:39.173 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:39.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.403+0000 7fc89091a640 1 Processor -- start 2023-10-14T16:57:39.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.404+0000 7fc89091a640 1 -- start start 2023-10-14T16:57:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.405+0000 7fc89091a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 0x7fc88c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.405+0000 7fc89091a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc88c0c9440 0x7fc88c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.405+0000 7fc89091a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc88c0ca560 con 0x7fc88c06c9e0 2023-10-14T16:57:39.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.405+0000 7fc89091a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0c9950 con 0x7fc88c0c9440 2023-10-14T16:57:39.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.405+0000 7fc89091a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc88c0c9c30 con 0x7fc88c0c9060 2023-10-14T16:57:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc883fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc88c06c9e0 0x7fc88c164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43500/0 (socket says 172.21.15.29:43500) 2023-10-14T16:57:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc883fff640 1 -- 172.21.15.29:0/1963086616 learned_addr learned my addr 172.21.15.29:0/1963086616 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88bfff640 1 --2- 172.21.15.29:0/1963086616 >> v2:172.21.15.29:3300/0 conn(0x7fc88c0c9440 0x7fc88c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88b7fe640 1 --2- 172.21.15.29:0/1963086616 >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 0x7fc88c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88affd640 1 -- 172.21.15.29:0/1963086616 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365554969 0 0) 0x7fc88c0ca560 con 0x7fc88c06c9e0 2023-10-14T16:57:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88affd640 1 -- 172.21.15.29:0/1963086616 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc878003520 con 0x7fc88c06c9e0 2023-10-14T16:57:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88bfff640 1 -- 172.21.15.29:0/1963086616 >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 msgr2=0x7fc88c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88bfff640 1 --2- 172.21.15.29:0/1963086616 >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 0x7fc88c06f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88bfff640 1 -- 172.21.15.29:0/1963086616 >> v1:172.21.15.29:6789/0 conn(0x7fc88c06c9e0 legacy=0x7fc88c164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88bfff640 1 -- 172.21.15.29:0/1963086616 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88c0c9f10 con 0x7fc88c0c9440 2023-10-14T16:57:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.406+0000 7fc88b7fe640 1 --2- 172.21.15.29:0/1963086616 >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 0x7fc88c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:39.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc88bfff640 1 --2- 172.21.15.29:0/1963086616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c9440 0x7fc88c06f980 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06d490 tx=0x7fc88402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3869f904090fc452 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc88affd640 1 -- 172.21.15.29:0/1963086616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc88402cc80 con 0x7fc88c0c9440 2023-10-14T16:57:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc88affd640 1 -- 172.21.15.29:0/1963086616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc88402cde0 con 0x7fc88c0c9440 2023-10-14T16:57:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc88affd640 1 -- 172.21.15.29:0/1963086616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc884034660 con 0x7fc88c0c9440 2023-10-14T16:57:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc89091a640 1 -- 172.21.15.29:0/1963086616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c9440 msgr2=0x7fc88c06f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.408+0000 7fc89091a640 1 --2- 172.21.15.29:0/1963086616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c9440 0x7fc88c06f980 secure :-1 s=READY pgs=1186 cs=0 l=1 rev1=1 crypto rx=0x7fc88c06d490 tx=0x7fc88402bed0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 -- 172.21.15.29:0/1963086616 shutdown_connections 2023-10-14T16:57:39.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 --2- 172.21.15.29:0/1963086616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c0c9440 0x7fc88c06f980 unknown :-1 s=CLOSED pgs=1186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 --2- 172.21.15.29:0/1963086616 >> v2:172.21.15.186:3300/0 conn(0x7fc88c0c9060 0x7fc88c06f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 -- 172.21.15.29:0/1963086616 >> 172.21.15.29:0/1963086616 conn(0x7fc88c0ce6a0 msgr2=0x7fc88c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:39.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 -- 172.21.15.29:0/1963086616 shutdown_connections 2023-10-14T16:57:39.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.409+0000 7fc89091a640 1 -- 172.21.15.29:0/1963086616 wait complete. 2023-10-14T16:57:39.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.410+0000 7fc89091a640 1 Processor -- start 2023-10-14T16:57:39.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.411+0000 7fc89091a640 1 -- start start 2023-10-14T16:57:39.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c0c9060 0x7fc88c163c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c9440 0x7fc88c0c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c0c8b00 con 0x7fc88c06c9e0 2023-10-14T16:57:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c162bc0 con 0x7fc88c0c9060 2023-10-14T16:57:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc89091a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88c162ea0 con 0x7fc88c0c9440 2023-10-14T16:57:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35848/0 (socket says 172.21.15.29:35848) 2023-10-14T16:57:39.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 -- 172.21.15.29:0/861061411 learned_addr learned my addr 172.21.15.29:0/861061411 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 -- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c9440 msgr2=0x7fc88c0c85c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c9440 0x7fc88c0c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 -- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c0c9060 msgr2=0x7fc88c163c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c0c9060 0x7fc88c163c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.412+0000 7fc88b7fe640 1 -- 172.21.15.29:0/861061411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc884009a70 con 0x7fc88c06c9e0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc88b7fe640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00e7e0 tx=0x7fc87c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c004030 con 0x7fc88c06c9e0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc87c004190 con 0x7fc88c06c9e0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc87c004520 con 0x7fc88c06c9e0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc88c163180 con 0x7fc88c06c9e0 2023-10-14T16:57:39.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.413+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc88c163740 con 0x7fc88c06c9e0 2023-10-14T16:57:39.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.414+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc88c163900 con 0x7fc88c06c9e0 2023-10-14T16:57:39.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.415+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc87c00c6d0 con 0x7fc88c06c9e0 2023-10-14T16:57:39.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.416+0000 7fc888ff9640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 0x7fc86c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.416+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7fc87c089610 con 0x7fc88c06c9e0 2023-10-14T16:57:39.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.417+0000 7fc883fff640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 0x7fc86c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.417+0000 7fc883fff640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 0x7fc86c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc874005880 tx=0x7fc874007950 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.420+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc87c054420 con 0x7fc88c06c9e0 2023-10-14T16:57:39.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.656+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc88c04e9b0 con 0x7fc88c06c9e0 2023-10-14T16:57:39.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.657+0000 7fc888ff9640 1 -- 172.21.15.29:0/861061411 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+3705 (secure 0 0 0) 0x7fc87c020e90 con 0x7fc88c06c9e0 2023-10-14T16:57:39.659 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:39.659 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2780492578","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2780492578},{"type":"v1","addr":"172.21.15.186:6833","nonce":2780492578}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":5803,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3212244122","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3212244122},{"type":"v1","addr":"172.21.15.186:6835","nonce":3212244122}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":6999,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/3664447305","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3664447305},{"type":"v1","addr":"172.21.15.29:6835","nonce":3664447305}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":35},{"gid":7416,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1705537115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1705537115},{"type":"v1","addr":"172.21.15.29:6837","nonce":1705537115}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":42}],"filesystems":[]} 2023-10-14T16:57:39.659 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 42 2023-10-14T16:57:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 msgr2=0x7fc86c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 0x7fc86c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fc874005880 tx=0x7fc874007950 comp rx=0 tx=0).stop 2023-10-14T16:57:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 msgr2=0x7fc88c162560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 secure :-1 s=READY pgs=1187 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00e7e0 tx=0x7fc87c00eca0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 shutdown_connections 2023-10-14T16:57:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fc86c06af90 0x7fc86c06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc88c0c9440 0x7fc88c0c85c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc88c0c9060 0x7fc88c163c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 --2- 172.21.15.29:0/861061411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc88c06c9e0 0x7fc88c162560 unknown :-1 s=CLOSED pgs=1187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 >> 172.21.15.29:0/861061411 conn(0x7fc88c0ce6a0 msgr2=0x7fc88c1609c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 shutdown_connections 2023-10-14T16:57:39.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.660+0000 7fc89091a640 1 -- 172.21.15.29:0/861061411 wait complete. 2023-10-14T16:57:39.683 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:57:39.683 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:57:39.683 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:57:39.684 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:57:39.684 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:57:39.684 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:57:39.685 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:57:39.685 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:57:39.685 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:57:39.685 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:57:39.686 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:57:39.686 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:57:39.686 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:57:39.687 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:57:39.687 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:57:39.687 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:57:39.705 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:57:39.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:57:39.710 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:57:39.710 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:57:39.711 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:57:39.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:57:39.713 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:57:39.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:57:39.760 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:57:39.760 DEBUG:teuthology.parallel:result is None 2023-10-14T16:57:39.761 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:57:39.762 DEBUG:teuthology.parallel:result is None 2023-10-14T16:57:39.768 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:57:39.768 DEBUG:teuthology.parallel:result is None 2023-10-14T16:57:39.769 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:57:39.770 DEBUG:teuthology.parallel:result is None 2023-10-14T16:57:39.770 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:57:39.919 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:57:39.921 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:57:39.930 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:57:39.930 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:57:39.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.963+0000 7f02b2fd5640 1 Processor -- start 2023-10-14T16:57:39.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.963+0000 7f02b2fd5640 1 -- start start 2023-10-14T16:57:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02b2fd5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f02ac0addb0 0x7f02ac0caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02b2fd5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac0cb140 con 0x7f02ac0c9000 2023-10-14T16:57:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02b2fd5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02ac06b030 con 0x7f02ac06a4b0 2023-10-14T16:57:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02b2fd5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02ac161a50 con 0x7f02ac0addb0 2023-10-14T16:57:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02abfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f02ac0c9000 0x7f02ac069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43516/0 (socket says 172.21.15.29:43516) 2023-10-14T16:57:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02abfff640 1 -- 172.21.15.29:0/2351881929 learned_addr learned my addr 172.21.15.29:0/2351881929 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.964+0000 7f02b0d71640 1 --2- 172.21.15.29:0/2351881929 >> v2:172.21.15.186:3300/0 conn(0x7f02ac0addb0 0x7f02ac0caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.965+0000 7f02ab7fe640 1 -- 172.21.15.29:0/2351881929 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337607457 0 0) 0x7f02ac0cb140 con 0x7f02ac0c9000 2023-10-14T16:57:39.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.965+0000 7f02b0d71640 1 -- 172.21.15.29:0/2351881929 >> v1:172.21.15.186:6789/0 conn(0x7f02ac06a4b0 legacy=0x7f02ac1650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.965+0000 7f02b0d71640 1 -- 172.21.15.29:0/2351881929 >> v1:172.21.15.29:6789/0 conn(0x7f02ac0c9000 legacy=0x7f02ac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.965+0000 7f02b0d71640 1 -- 172.21.15.29:0/2351881929 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02ac06b310 con 0x7f02ac0addb0 2023-10-14T16:57:39.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.966+0000 7f02b0d71640 1 --2- 172.21.15.29:0/2351881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac0addb0 0x7f02ac0caa80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009720 tx=0x7f02a002b730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6137b4359b89817 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.966+0000 7f02ab7fe640 1 -- 172.21.15.29:0/2351881929 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0030710 con 0x7f02ac0addb0 2023-10-14T16:57:39.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02ab7fe640 1 -- 172.21.15.29:0/2351881929 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a002e050 con 0x7f02ac0addb0 2023-10-14T16:57:39.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02ab7fe640 1 -- 172.21.15.29:0/2351881929 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a0030a90 con 0x7f02ac0addb0 2023-10-14T16:57:39.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 -- 172.21.15.29:0/2351881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac0addb0 msgr2=0x7f02ac0caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/2351881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac0addb0 0x7f02ac0caa80 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f02a0009720 tx=0x7f02a002b730 comp rx=0 tx=0).stop 2023-10-14T16:57:39.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 -- 172.21.15.29:0/2351881929 shutdown_connections 2023-10-14T16:57:39.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/2351881929 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac0addb0 0x7f02ac0caa80 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 -- 172.21.15.29:0/2351881929 >> 172.21.15.29:0/2351881929 conn(0x7f02ac0ce9b0 msgr2=0x7f02ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:39.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 -- 172.21.15.29:0/2351881929 shutdown_connections 2023-10-14T16:57:39.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.967+0000 7f02b2fd5640 1 -- 172.21.15.29:0/2351881929 wait complete. 2023-10-14T16:57:39.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.969+0000 7f02b2fd5640 1 Processor -- start 2023-10-14T16:57:39.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.969+0000 7f02b2fd5640 1 -- start start 2023-10-14T16:57:39.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 0x7f02ac0b2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02ac0addb0 0x7f02ac157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b2ba0 con 0x7f02ac0c9000 2023-10-14T16:57:39.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b2e40 con 0x7f02ac06a4b0 2023-10-14T16:57:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b2fd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02ac0b3120 con 0x7f02ac0addb0 2023-10-14T16:57:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.970+0000 7f02b1572640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b0d71640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 0x7f02ac0b2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35896/0 (socket says 172.21.15.29:35896) 2023-10-14T16:57:39.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 -- 172.21.15.29:0/1514536768 learned_addr learned my addr 172.21.15.29:0/1514536768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02abfff640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02ac0addb0 0x7f02ac157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 -- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02ac0addb0 msgr2=0x7f02ac157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02ac0addb0 0x7f02ac157d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 -- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 msgr2=0x7f02ac0b2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:39.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 0x7f02ac0b2660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:39.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 -- 172.21.15.29:0/1514536768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a0009400 con 0x7f02ac0c9000 2023-10-14T16:57:39.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b0d71640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 0x7f02ac0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:57:39.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.971+0000 7f02b1572640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c8c0 tx=0x7f02a400cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.972+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4003da0 con 0x7f02ac0c9000 2023-10-14T16:57:39.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.972+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02a4005070 con 0x7f02ac0c9000 2023-10-14T16:57:39.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.972+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02a4005400 con 0x7f02ac0c9000 2023-10-14T16:57:39.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.972+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02ac0b3400 con 0x7f02ac0c9000 2023-10-14T16:57:39.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.972+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02ac0b39c0 con 0x7f02ac0c9000 2023-10-14T16:57:39.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.973+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02ac04e160 con 0x7f02ac0c9000 2023-10-14T16:57:39.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.975+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f02a4005560 con 0x7f02ac0c9000 2023-10-14T16:57:39.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.976+0000 7f02a97fa640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 0x7f029006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:39.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.976+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 5675+0+0 (secure 0 0 0) 0x7f02a40894b0 con 0x7f02ac0c9000 2023-10-14T16:57:39.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.977+0000 7f02b0d71640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 0x7f029006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:39.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.977+0000 7f02b0d71640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 0x7f029006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02a0034000 tx=0x7f02a0008030 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:39.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:39.979+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f02a4054590 con 0x7f02ac0c9000 2023-10-14T16:57:40.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.194+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f02ac06c5b0 con 0x7f02ac0c9000 2023-10-14T16:57:40.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.217+0000 7f02a97fa640 1 -- 172.21.15.29:0/1514536768 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v105) v1 ==== 97+0+0 (secure 0 0 0) 0x7f02a4017020 con 0x7f02ac0c9000 2023-10-14T16:57:40.219 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:57:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 msgr2=0x7f029006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 0x7f029006d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f02a0034000 tx=0x7f02a0008030 comp rx=0 tx=0).stop 2023-10-14T16:57:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 msgr2=0x7f02ac0b3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 crypto rx=0x7f02a400c8c0 tx=0x7f02a400cd80 comp rx=0 tx=0).stop 2023-10-14T16:57:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 shutdown_connections 2023-10-14T16:57:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f029006af90 0x7f029006d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02ac0c9000 0x7f02ac0b3ee0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f02ac0addb0 0x7f02ac157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 --2- 172.21.15.29:0/1514536768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02ac06a4b0 0x7f02ac0b2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 >> 172.21.15.29:0/1514536768 conn(0x7f02ac0ce9b0 msgr2=0x7f02ac06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 shutdown_connections 2023-10-14T16:57:40.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.221+0000 7f02b2fd5640 1 -- 172.21.15.29:0/1514536768 wait complete. 2023-10-14T16:57:40.242 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:57:40.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.450+0000 7f65d1bc4640 1 Processor -- start 2023-10-14T16:57:40.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.450+0000 7f65d1bc4640 1 -- start start 2023-10-14T16:57:40.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.452+0000 7f65d1bc4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65cc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.452+0000 7f65d1bc4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc15f4a0 con 0x7f65cc0ad970 2023-10-14T16:57:40.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.452+0000 7f65d1bc4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65cc06c960 con 0x7f65cc0cda80 2023-10-14T16:57:40.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.452+0000 7f65d1bc4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc15b9d0 con 0x7f65cc06c040 2023-10-14T16:57:40.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.453+0000 7f65cbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65cc0ad970 0x7f65cc15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43528/0 (socket says 172.21.15.29:43528) 2023-10-14T16:57:40.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.453+0000 7f65cbfff640 1 -- 172.21.15.29:0/1446966154 learned_addr learned my addr 172.21.15.29:0/1446966154 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:40.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.453+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65cc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.453+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506692727 0 0) 0x7f65cc15f4a0 con 0x7f65cc0ad970 2023-10-14T16:57:40.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.453+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65bc003520 con 0x7f65cc0ad970 2023-10-14T16:57:40.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1343394306 0 0) 0x7f65bc003520 con 0x7f65cc0ad970 2023-10-14T16:57:40.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 msgr2=0x7f65cc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:40.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65cc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 >> v1:172.21.15.186:6789/0 conn(0x7f65cc0cda80 legacy=0x7f65cc15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc15bcb0 con 0x7f65cc0ad970 2023-10-14T16:57:40.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f65b8001670 con 0x7f65cc0ad970 2023-10-14T16:57:40.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 >> v1:172.21.15.29:6789/0 conn(0x7f65cc0ad970 legacy=0x7f65cc15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0039a0 0x7f65bc003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65bc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc15b9d0 con 0x7f65bc0047c0 2023-10-14T16:57:40.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc00ba30 con 0x7f65bc0039a0 2023-10-14T16:57:40.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65bc00bba0 con 0x7f65cc06c040 2023-10-14T16:57:40.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65cbfff640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0039a0 0x7f65bc003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.454+0000 7f65d13c3640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65bc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65cbfff640 1 -- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 msgr2=0x7f65bc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65cbfff640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc06c040 0x7f65bc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65cbfff640 1 -- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0039a0 msgr2=0x7f65bc003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65cbfff640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0039a0 0x7f65bc003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65cbfff640 1 -- 172.21.15.29:0/1446966154 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65bc003520 con 0x7f65bc0047c0 2023-10-14T16:57:40.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.455+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0039a0 0x7f65bc003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.457+0000 7f65cbfff640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f65b800c160 tx=0x7f65bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1654de23e268031 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.457+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8005ad0 con 0x7f65bc0047c0 2023-10-14T16:57:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.457+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b8005c30 con 0x7f65bc0047c0 2023-10-14T16:57:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.457+0000 7f65cb7fe640 1 -- 172.21.15.29:0/1446966154 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b802c7b0 con 0x7f65bc0047c0 2023-10-14T16:57:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 -- 172.21.15.29:0/1446966154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65bc0047c0 msgr2=0x7f65bc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/1446966154 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65bc0047c0 0x7f65bc004c40 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 crypto rx=0x7f65b800c160 tx=0x7f65bc0057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d13c3640 1 -- 172.21.15.29:0/1446966154 reap_dead start 2023-10-14T16:57:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 -- 172.21.15.29:0/1446966154 shutdown_connections 2023-10-14T16:57:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 -- 172.21.15.29:0/1446966154 >> 172.21.15.29:0/1446966154 conn(0x7f65cc0ce6a0 msgr2=0x7f65cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 -- 172.21.15.29:0/1446966154 shutdown_connections 2023-10-14T16:57:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.458+0000 7f65d1bc4640 1 -- 172.21.15.29:0/1446966154 wait complete. 2023-10-14T16:57:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.460+0000 7f65d1bc4640 1 Processor -- start 2023-10-14T16:57:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.460+0000 7f65d1bc4640 1 -- start start 2023-10-14T16:57:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0047c0 0x7f65cc0c7310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc0c7850 0x7f65cc15d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc1f5d70 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc1f6050 con 0x7f65bc0047c0 2023-10-14T16:57:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d1bc4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65cc1f6330 con 0x7f65cc0c7850 2023-10-14T16:57:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d13c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d13c3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35914/0 (socket says 172.21.15.29:35914) 2023-10-14T16:57:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.461+0000 7f65d13c3640 1 -- 172.21.15.29:0/3670566707 learned_addr learned my addr 172.21.15.29:0/3670566707 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65cbfff640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc0c7850 0x7f65cc15d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0047c0 0x7f65cc0c7310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 -- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc0c7850 msgr2=0x7f65cc15d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc0c7850 0x7f65cc15d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 -- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0047c0 msgr2=0x7f65cc0c7310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0047c0 0x7f65cc0c7310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 -- 172.21.15.29:0/3670566707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65cc1f6610 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65d13c3640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f65bc003970 tx=0x7f65c400eb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.462+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c4003970 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.463+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c4003b10 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.463+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c40050d0 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.463+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65cc1f5900 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.463+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65cc0c5d30 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.464+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65cc06b330 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.465+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65c4005230 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.466+0000 7f65c97fa640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 0x7f65b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.466+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f65c4089320 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.466+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 0x7f65b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.467+0000 7f65d0bc2640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 0x7f65b006d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65cc1f5860 tx=0x7f65c0008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.471+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65c40545d0 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.739+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f65cc1f5af0 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.740+0000 7f65c97fa640 1 -- 172.21.15.29:0/3670566707 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f65c4057bf0 con 0x7f65cc0ae0c0 2023-10-14T16:57:40.742 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:40.742 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:57:40.742 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:57:40.743 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.743 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:57:40.743 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:57:40.743 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.743 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.744 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:40.744 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:40.744 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.744 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.744 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.745 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.745 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:57:40.745 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:57:40.745 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.745 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:40.746 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:40.746 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.746 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.746 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.747 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:40.748 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:40.748 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.748 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.748 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.748 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.749 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:57:40.749 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:57:40.749 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.749 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:40.750 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:40.750 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.750 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.750 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.750 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.751 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:57:40.751 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:57:40.751 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.751 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.752 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.752 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.752 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.752 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.752 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:57:40.753 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:57:40.753 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.753 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.753 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.754 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.754 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.754 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.754 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:57:40.754 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:57:40.755 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.755 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.755 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.755 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.756 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.756 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.756 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:57:40.756 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:57:40.756 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.757 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.758 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:57:40.758 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:57:40.758 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.758 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.759 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.759 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.759 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.759 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.759 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:57:40.760 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:57:40.760 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.760 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.760 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.760 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:57:40.761 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.762 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.762 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.762 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.762 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.763 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.763 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.763 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:57:40.763 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:57:40.763 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.764 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:57:40.764 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:57:40.764 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.764 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.764 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.765 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T16:57:40.766 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T16:57:40.766 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.766 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.766 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.766 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:57:40.767 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:57:40.767 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.767 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.767 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:40.767 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:40.768 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:40.768 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.768 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.768 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.768 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:40.769 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:57:40.770 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:57:40.771 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:57:40.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:57:40.771 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:57:40.771 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:57:40.772 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.772 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.772 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.772 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:57:40.772 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:57:40.773 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:40.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:57:40.773 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:57:40.773 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.774 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.774 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.774 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.774 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:57:40.774 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:57:40.775 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:40.775 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:57:40.775 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.775 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:57:40.776 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:57:40.777 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:57:40.777 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:57:40.777 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.777 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:40.777 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:57:40.778 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:57:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 msgr2=0x7f65b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 0x7f65b006d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f65cc1f5860 tx=0x7f65c0008040 comp rx=0 tx=0).stop 2023-10-14T16:57:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 msgr2=0x7f65cc15d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 crypto rx=0x7f65bc003970 tx=0x7f65c400eb90 comp rx=0 tx=0).stop 2023-10-14T16:57:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 shutdown_connections 2023-10-14T16:57:40.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f65b006af90 0x7f65b006d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65cc0ae0c0 0x7f65cc15d760 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65cc0c7850 0x7f65cc15d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 --2- 172.21.15.29:0/3670566707 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65bc0047c0 0x7f65cc0c7310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 >> 172.21.15.29:0/3670566707 conn(0x7f65cc0ce6a0 msgr2=0x7f65cc0ca3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 shutdown_connections 2023-10-14T16:57:40.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:40.743+0000 7f65d1bc4640 1 -- 172.21.15.29:0/3670566707 wait complete. 2023-10-14T16:57:40.782 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.009+0000 7f8ab4075640 1 Processor -- start 2023-10-14T16:57:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.009+0000 7f8ab4075640 1 -- start start 2023-10-14T16:57:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab4075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8aac15ae80 0x7f8aac15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab4075640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac15ed60 con 0x7f8aac15aaa0 2023-10-14T16:57:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab4075640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aac0707e0 con 0x7f8aac0ce390 2023-10-14T16:57:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab4075640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aac06b810 con 0x7f8aac15ae80 2023-10-14T16:57:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab1e11640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8aac15aaa0 0x7f8aac0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43350/0 (socket says 172.21.15.29:43350) 2023-10-14T16:57:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab1e11640 1 -- 172.21.15.29:0/3498636750 learned_addr learned my addr 172.21.15.29:0/3498636750 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.011+0000 7f8ab1610640 1 --2- 172.21.15.29:0/3498636750 >> v2:172.21.15.29:3300/0 conn(0x7f8aac15ae80 0x7f8aac15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.012+0000 7f8ab1610640 1 -- 172.21.15.29:0/3498636750 >> v1:172.21.15.186:6789/0 conn(0x7f8aac0ce390 legacy=0x7f8aac070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.012+0000 7f8ab1610640 1 -- 172.21.15.29:0/3498636750 >> v1:172.21.15.29:6789/0 conn(0x7f8aac15aaa0 legacy=0x7f8aac0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.012+0000 7f8ab1610640 1 -- 172.21.15.29:0/3498636750 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac15ef90 con 0x7f8aac15ae80 2023-10-14T16:57:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.013+0000 7f8ab1610640 1 --2- 172.21.15.29:0/3498636750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15ae80 0x7f8aac15e820 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0097a0 tx=0x7f8a9c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7184bc3f3cb4a58b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab0e0f640 1 -- 172.21.15.29:0/3498636750 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c030610 con 0x7f8aac15ae80 2023-10-14T16:57:41.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab0e0f640 1 -- 172.21.15.29:0/3498636750 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c030770 con 0x7f8aac15ae80 2023-10-14T16:57:41.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab0e0f640 1 -- 172.21.15.29:0/3498636750 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c030b00 con 0x7f8aac15ae80 2023-10-14T16:57:41.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab4075640 1 -- 172.21.15.29:0/3498636750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15ae80 msgr2=0x7f8aac15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab4075640 1 --2- 172.21.15.29:0/3498636750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15ae80 0x7f8aac15e820 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c0097a0 tx=0x7f8a9c02b960 comp rx=0 tx=0).stop 2023-10-14T16:57:41.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab4075640 1 -- 172.21.15.29:0/3498636750 shutdown_connections 2023-10-14T16:57:41.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab4075640 1 --2- 172.21.15.29:0/3498636750 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15ae80 0x7f8aac15e820 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.014+0000 7f8ab4075640 1 -- 172.21.15.29:0/3498636750 >> 172.21.15.29:0/3498636750 conn(0x7f8aac0cb410 msgr2=0x7f8aac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:41.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.015+0000 7f8ab4075640 1 -- 172.21.15.29:0/3498636750 shutdown_connections 2023-10-14T16:57:41.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.015+0000 7f8ab4075640 1 -- 172.21.15.29:0/3498636750 wait complete. 2023-10-14T16:57:41.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.016+0000 7f8ab4075640 1 Processor -- start 2023-10-14T16:57:41.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.016+0000 7f8ab4075640 1 -- start start 2023-10-14T16:57:41.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8aac0ce390 0x7f8aac0c4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8aac15ae80 0x7f8aac158ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac15b2b0 con 0x7f8aac15aaa0 2023-10-14T16:57:41.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac15c110 con 0x7f8aac0ce390 2023-10-14T16:57:41.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.017+0000 7f8ab4075640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac15b500 con 0x7f8aac15ae80 2023-10-14T16:57:41.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58606/0 (socket says 172.21.15.29:58606) 2023-10-14T16:57:41.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 -- 172.21.15.29:0/367392995 learned_addr learned my addr 172.21.15.29:0/367392995 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:41.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1e11640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8aac0ce390 0x7f8aac0c4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab2612640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8aac15ae80 0x7f8aac158ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 -- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8aac15ae80 msgr2=0x7f8aac158ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8aac15ae80 0x7f8aac158ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 -- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8aac0ce390 msgr2=0x7f8aac0c4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8aac0ce390 0x7f8aac0c4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.018+0000 7f8ab1610640 1 -- 172.21.15.29:0/367392995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a9c009400 con 0x7f8aac15aaa0 2023-10-14T16:57:41.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8ab1610640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009890 tx=0x7f8a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c03e070 con 0x7f8aac15aaa0 2023-10-14T16:57:41.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9c039a40 con 0x7f8aac15aaa0 2023-10-14T16:57:41.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c039dd0 con 0x7f8aac15aaa0 2023-10-14T16:57:41.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac15b7e0 con 0x7f8aac15aaa0 2023-10-14T16:57:41.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.019+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac15bda0 con 0x7f8aac15aaa0 2023-10-14T16:57:41.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.020+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac070a10 con 0x7f8aac15aaa0 2023-10-14T16:57:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.021+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8a9c04b400 con 0x7f8aac15aaa0 2023-10-14T16:57:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.022+0000 7f8aa27fc640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 0x7f8a9006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.022+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f8a9c0ab9f0 con 0x7f8aac15aaa0 2023-10-14T16:57:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.022+0000 7f8ab1e11640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 0x7f8a9006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.023+0000 7f8ab1e11640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 0x7f8a9006d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8005880 tx=0x7f8aa8009040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.027+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8aac070a10 con 0x7f8aac15aaa0 2023-10-14T16:57:41.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.224+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8aac070a10 con 0x7f8aac15aaa0 2023-10-14T16:57:41.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.226+0000 7f8aa27fc640 1 -- 172.21.15.29:0/367392995 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+12917 (secure 0 0 0) 0x7f8aac070a10 con 0x7f8aac15aaa0 2023-10-14T16:57:41.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:41.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":105,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:40.212383+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":11,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":94,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:41.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 msgr2=0x7f8a9006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 0x7f8a9006d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8aa8005880 tx=0x7f8aa8009040 comp rx=0 tx=0).stop 2023-10-14T16:57:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 msgr2=0x7f8aac156d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 secure :-1 s=READY pgs=1197 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009890 tx=0x7f8a9c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 shutdown_connections 2023-10-14T16:57:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8a9006afc0 0x7f8a9006d480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8aac15ae80 0x7f8aac158ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8aac15aaa0 0x7f8aac156d60 unknown :-1 s=CLOSED pgs=1197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 --2- 172.21.15.29:0/367392995 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8aac0ce390 0x7f8aac0c4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 >> 172.21.15.29:0/367392995 conn(0x7f8aac0cb410 msgr2=0x7f8aac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:41.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 shutdown_connections 2023-10-14T16:57:41.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.229+0000 7f8ab4075640 1 -- 172.21.15.29:0/367392995 wait complete. 2023-10-14T16:57:41.249 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:57:41.250 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:57:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.471+0000 7f0b4b59e640 1 Processor -- start 2023-10-14T16:57:41.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.471+0000 7f0b4b59e640 1 -- start start 2023-10-14T16:57:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c06b1e0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4c06f3f0 con 0x7f0b4c0ca010 2023-10-14T16:57:41.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4c06f6d0 con 0x7f0b4c1592e0 2023-10-14T16:57:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58618/0 (socket says 172.21.15.29:58618) 2023-10-14T16:57:41.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.473+0000 7f0b4a59c640 1 -- 172.21.15.29:0/877843385 learned_addr learned my addr 172.21.15.29:0/877843385 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.474+0000 7f0b4a59c640 1 -- 172.21.15.29:0/877843385 >> v1:172.21.15.186:6789/0 conn(0x7f0b4c0ca010 legacy=0x7f0b4c06a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.474+0000 7f0b4a59c640 1 -- 172.21.15.29:0/877843385 >> v1:172.21.15.29:6789/0 conn(0x7f0b4c06ace0 legacy=0x7f0b4c06f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.474+0000 7f0b4a59c640 1 -- 172.21.15.29:0/877843385 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4c0c9810 con 0x7f0b4c1592e0 2023-10-14T16:57:41.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.475+0000 7f0b4a59c640 1 --2- 172.21.15.29:0/877843385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00aa10 tx=0x7f0b3c02cdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f171a0f8f03bbcc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.475+0000 7f0b4959a640 1 -- 172.21.15.29:0/877843385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c02db80 con 0x7f0b4c1592e0 2023-10-14T16:57:41.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4959a640 1 -- 172.21.15.29:0/877843385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c02dce0 con 0x7f0b4c1592e0 2023-10-14T16:57:41.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4959a640 1 -- 172.21.15.29:0/877843385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c035660 con 0x7f0b4c1592e0 2023-10-14T16:57:41.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 -- 172.21.15.29:0/877843385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c1592e0 msgr2=0x7f0b4c0cdd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/877843385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00aa10 tx=0x7f0b3c02cdd0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 -- 172.21.15.29:0/877843385 shutdown_connections 2023-10-14T16:57:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/877843385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c1592e0 0x7f0b4c0cdd90 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 -- 172.21.15.29:0/877843385 >> 172.21.15.29:0/877843385 conn(0x7f0b4c0b0170 msgr2=0x7f0b4c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:41.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 -- 172.21.15.29:0/877843385 shutdown_connections 2023-10-14T16:57:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.476+0000 7f0b4b59e640 1 -- 172.21.15.29:0/877843385 wait complete. 2023-10-14T16:57:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.478+0000 7f0b4b59e640 1 Processor -- start 2023-10-14T16:57:41.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.478+0000 7f0b4b59e640 1 -- start start 2023-10-14T16:57:41.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 0x7f0b4c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b4c1592e0 0x7f0b4c0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c15aca0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c15a090 con 0x7f0b4c0ca010 2023-10-14T16:57:41.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4c15a370 con 0x7f0b4c1592e0 2023-10-14T16:57:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58624/0 (socket says 172.21.15.29:58624) 2023-10-14T16:57:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.479+0000 7f0b4a59c640 1 -- 172.21.15.29:0/89040673 learned_addr learned my addr 172.21.15.29:0/89040673 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4ad9d640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b4c1592e0 0x7f0b4c0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 -- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b4c1592e0 msgr2=0x7f0b4c0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b4c1592e0 0x7f0b4c0acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 -- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 msgr2=0x7f0b4c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b49d9b640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 0x7f0b4c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 0x7f0b4c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 -- 172.21.15.29:0/89040673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c00a670 con 0x7f0b4c06ace0 2023-10-14T16:57:41.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b49d9b640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 0x7f0b4c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:41.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.480+0000 7f0b4a59c640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00a1c0 tx=0x7f0b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.481+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c03f5f0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.481+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b3c03f770 con 0x7f0b4c06ace0 2023-10-14T16:57:41.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.481+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b3c03fb20 con 0x7f0b4c06ace0 2023-10-14T16:57:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.481+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b4c15a650 con 0x7f0b4c06ace0 2023-10-14T16:57:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.481+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b4c159d80 con 0x7f0b4c06ace0 2023-10-14T16:57:41.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.482+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b4c04e9b0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.483+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0b3c03fc80 con 0x7f0b4c06ace0 2023-10-14T16:57:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.484+0000 7f0b3affd640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 0x7f0b2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.484+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f0b3c0abf40 con 0x7f0b4c06ace0 2023-10-14T16:57:41.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.484+0000 7f0b49d9b640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 0x7f0b2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.485+0000 7f0b49d9b640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 0x7f0b2c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0c2140 tx=0x7f0b40009040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.489+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b3c0773b0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.684+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0b4c04e9b0 con 0x7f0b4c06ace0 2023-10-14T16:57:41.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.685+0000 7f0b3affd640 1 -- 172.21.15.29:0/89040673 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v105) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0b3c03a020 con 0x7f0b4c06ace0 2023-10-14T16:57:41.687 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:57:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 msgr2=0x7f0b2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 0x7f0b2c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c0c2140 tx=0x7f0b40009040 comp rx=0 tx=0).stop 2023-10-14T16:57:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 msgr2=0x7f0b4c159840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 secure :-1 s=READY pgs=1200 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c00a1c0 tx=0x7f0b3c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 shutdown_connections 2023-10-14T16:57:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0b2c06af90 0x7f0b2c06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b4c1592e0 0x7f0b4c0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b4c0ca010 0x7f0b4c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 --2- 172.21.15.29:0/89040673 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b4c06ace0 0x7f0b4c159840 unknown :-1 s=CLOSED pgs=1200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 >> 172.21.15.29:0/89040673 conn(0x7f0b4c0b0170 msgr2=0x7f0b4c158870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 shutdown_connections 2023-10-14T16:57:41.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.688+0000 7f0b4b59e640 1 -- 172.21.15.29:0/89040673 wait complete. 2023-10-14T16:57:41.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:57:41.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.918+0000 7f807f59e640 1 Processor -- start 2023-10-14T16:57:41.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.918+0000 7f807f59e640 1 -- start start 2023-10-14T16:57:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f807f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f808015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f807f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808015aef0 con 0x7f8080070450 2023-10-14T16:57:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f807f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f808006cc80 con 0x7f808006c810 2023-10-14T16:57:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f807f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f808015e490 con 0x7f8080159950 2023-10-14T16:57:41.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f8077fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8080070450 0x7f808006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43364/0 (socket says 172.21.15.29:43364) 2023-10-14T16:57:41.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.920+0000 7f8077fff640 1 -- 172.21.15.29:0/1546880337 learned_addr learned my addr 172.21.15.29:0/1546880337 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:41.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807e59c640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f808015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 46570046 0 0) 0x7f8068001110 con 0x7f8080070450 2023-10-14T16:57:41.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f806c003520 con 0x7f8080070450 2023-10-14T16:57:41.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137828883 0 0) 0x7f808006cc80 con 0x7f808006c810 2023-10-14T16:57:41.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8068001110 con 0x7f808006c810 2023-10-14T16:57:41.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306480274 0 0) 0x7f806c003520 con 0x7f8080070450 2023-10-14T16:57:41.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 msgr2=0x7f808015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:41.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f808015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 >> v1:172.21.15.186:6789/0 conn(0x7f808006c810 legacy=0x7f80800c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.921+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808015d910 con 0x7f8080070450 2023-10-14T16:57:41.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8068001800 con 0x7f8080070450 2023-10-14T16:57:41.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 >> v1:172.21.15.29:6789/0 conn(0x7f8080070450 legacy=0x7f808006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f806c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806c004090 0x7f806c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f806c004d00 0x7f806c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f808015e490 con 0x7f806c004d00 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c00bb10 con 0x7f806c004090 2023-10-14T16:57:41.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f806c00bca0 con 0x7f8080159950 2023-10-14T16:57:41.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807ed9d640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f806c004d00 0x7f806c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f807e59c640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f806c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.922+0000 7f8077fff640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806c004090 0x7f806c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.923+0000 7f807ed9d640 1 -- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 msgr2=0x7f806c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:41.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.923+0000 7f807ed9d640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8080159950 0x7f806c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.923+0000 7f807ed9d640 1 -- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806c004090 msgr2=0x7f806c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:41.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.923+0000 7f807ed9d640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f806c004090 0x7f806c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.923+0000 7f807ed9d640 1 -- 172.21.15.29:0/1546880337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f808015d910 con 0x7f806c004d00 2023-10-14T16:57:41.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.924+0000 7f807ed9d640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f806c004d00 0x7f806c005180 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f8078001bb0 tx=0x7f806c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664b0995f799f6f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078003b70 con 0x7f806c004d00 2023-10-14T16:57:41.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8078003cd0 con 0x7f806c004d00 2023-10-14T16:57:41.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807dd9b640 1 -- 172.21.15.29:0/1546880337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807802e9a0 con 0x7f806c004d00 2023-10-14T16:57:41.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807f59e640 1 -- 172.21.15.29:0/1546880337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f806c004d00 msgr2=0x7f806c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807f59e640 1 --2- 172.21.15.29:0/1546880337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f806c004d00 0x7f806c005180 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 crypto rx=0x7f8078001bb0 tx=0x7f806c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.925+0000 7f807ed9d640 1 -- 172.21.15.29:0/1546880337 reap_dead start 2023-10-14T16:57:41.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.926+0000 7f807f59e640 1 -- 172.21.15.29:0/1546880337 shutdown_connections 2023-10-14T16:57:41.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.926+0000 7f807f59e640 1 -- 172.21.15.29:0/1546880337 >> 172.21.15.29:0/1546880337 conn(0x7f80800ce6a0 msgr2=0x7f80800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:41.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.926+0000 7f807f59e640 1 -- 172.21.15.29:0/1546880337 shutdown_connections 2023-10-14T16:57:41.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.926+0000 7f807f59e640 1 -- 172.21.15.29:0/1546880337 wait complete. 2023-10-14T16:57:41.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.928+0000 7f807f59e640 1 Processor -- start 2023-10-14T16:57:41.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.928+0000 7f807f59e640 1 -- start start 2023-10-14T16:57:41.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806c004d00 0x7f80801f1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 0x7f8080154ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801f2400 con 0x7f80801572c0 2023-10-14T16:57:41.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80801550e0 con 0x7f80801eef70 2023-10-14T16:57:41.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80800b34f0 con 0x7f806c004d00 2023-10-14T16:57:41.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58644/0 (socket says 172.21.15.29:58644) 2023-10-14T16:57:41.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807ed9d640 1 -- 172.21.15.29:0/4085404351 learned_addr learned my addr 172.21.15.29:0/4085404351 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:41.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f807e59c640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806c004d00 0x7f80801f1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.929+0000 7f8077fff640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 0x7f8080154ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 -- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806c004d00 msgr2=0x7f80801f1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806c004d00 0x7f80801f1ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 -- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 msgr2=0x7f8080154ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:41.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 0x7f8080154ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:41.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 -- 172.21.15.29:0/4085404351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8080155570 con 0x7f80801572c0 2023-10-14T16:57:41.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f8077fff640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 0x7f8080154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:41.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807ed9d640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f80780020c0 tx=0x7f8078001650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8078003e40 con 0x7f80801572c0 2023-10-14T16:57:41.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f807802e990 con 0x7f80801572c0 2023-10-14T16:57:41.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f807802ed40 con 0x7f80801572c0 2023-10-14T16:57:41.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.930+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8080155b30 con 0x7f80801572c0 2023-10-14T16:57:41.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.931+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80800c9e70 con 0x7f80801572c0 2023-10-14T16:57:41.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.932+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80800c9e70 con 0x7f80801572c0 2023-10-14T16:57:41.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.933+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8078005070 con 0x7f80801572c0 2023-10-14T16:57:41.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.934+0000 7f8076ffd640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 0x7f806006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:41.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.934+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f80780ace70 con 0x7f80801572c0 2023-10-14T16:57:41.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.934+0000 7f807e59c640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 0x7f806006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:41.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.935+0000 7f807e59c640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 0x7f806006d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8080070a70 tx=0x7f8070008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:41.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:41.939+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8078078120 con 0x7f80801572c0 2023-10-14T16:57:42.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.139+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f80800c9e70 con 0x7f80801572c0 2023-10-14T16:57:42.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.140+0000 7f8076ffd640 1 -- 172.21.15.29:0/4085404351 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v105) v1 ==== 84+0+65 (secure 0 0 0) 0x7f807807b740 con 0x7f80801572c0 2023-10-14T16:57:42.142 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:42.142 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:57:42.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:42.142 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:57:42.143 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:57:42.143 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:42.143 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:57:42.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 msgr2=0x7f806006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 0x7f806006d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8080070a70 tx=0x7f8070008040 comp rx=0 tx=0).stop 2023-10-14T16:57:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 msgr2=0x7f80800b2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 secure :-1 s=READY pgs=1203 cs=0 l=1 rev1=1 crypto rx=0x7f80780020c0 tx=0x7f8078001650 comp rx=0 tx=0).stop 2023-10-14T16:57:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 shutdown_connections 2023-10-14T16:57:42.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f806006afc0 0x7f806006d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80801572c0 0x7f80800b2690 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80801eef70 0x7f8080154ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 --2- 172.21.15.29:0/4085404351 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806c004d00 0x7f80801f1ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 >> 172.21.15.29:0/4085404351 conn(0x7f80800ce6a0 msgr2=0x7f80800cd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:42.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 shutdown_connections 2023-10-14T16:57:42.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.143+0000 7f807f59e640 1 -- 172.21.15.29:0/4085404351 wait complete. 2023-10-14T16:57:42.164 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:57:42.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:57:42.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.394+0000 7fac71047640 1 Processor -- start 2023-10-14T16:57:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.394+0000 7fac71047640 1 -- start start 2023-10-14T16:57:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.395+0000 7fac71047640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fac6c15d0a0 0x7fac6c0d11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.395+0000 7fac71047640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c15d480 con 0x7fac6c0b3df0 2023-10-14T16:57:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.395+0000 7fac71047640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac6c0b2950 con 0x7fac6c0d09b0 2023-10-14T16:57:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.395+0000 7fac71047640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac6c0d36f0 con 0x7fac6c15d0a0 2023-10-14T16:57:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.396+0000 7fac70846640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fac6c0b3df0 0x7fac6c0cdef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43374/0 (socket says 172.21.15.29:43374) 2023-10-14T16:57:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.396+0000 7fac70846640 1 -- 172.21.15.29:0/8091290 learned_addr learned my addr 172.21.15.29:0/8091290 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.396+0000 7fac6bfff640 1 --2- 172.21.15.29:0/8091290 >> v2:172.21.15.186:3300/0 conn(0x7fac6c15d0a0 0x7fac6c0d11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.396+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972643241 0 0) 0x7fac64001110 con 0x7fac6c0b3df0 2023-10-14T16:57:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.396+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac54003520 con 0x7fac6c0b3df0 2023-10-14T16:57:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462508245 0 0) 0x7fac54003520 con 0x7fac6c0b3df0 2023-10-14T16:57:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 >> v2:172.21.15.186:3300/0 conn(0x7fac6c15d0a0 msgr2=0x7fac6c0d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 --2- 172.21.15.29:0/8091290 >> v2:172.21.15.186:3300/0 conn(0x7fac6c15d0a0 0x7fac6c0d11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 >> v1:172.21.15.186:6789/0 conn(0x7fac6c0d09b0 legacy=0x7fac6c0cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0d39d0 con 0x7fac6c0b3df0 2023-10-14T16:57:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fac64001800 con 0x7fac6c0b3df0 2023-10-14T16:57:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 >> v1:172.21.15.29:6789/0 conn(0x7fac6c0b3df0 legacy=0x7fac6c0cdef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 0x7fac54003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac540047c0 0x7fac54004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac540051d0 0x7fac5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0d36f0 con 0x7fac54003a30 2023-10-14T16:57:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac5400c100 con 0x7fac540047c0 2023-10-14T16:57:42.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac5400c2c0 con 0x7fac540051d0 2023-10-14T16:57:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.397+0000 7fac6bfff640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 0x7fac54003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6b7fe640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac540047c0 0x7fac54004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac70846640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac540051d0 0x7fac5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6bfff640 1 -- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac540051d0 msgr2=0x7fac5400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6bfff640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac540051d0 0x7fac5400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6bfff640 1 -- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac540047c0 msgr2=0x7fac54004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6bfff640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac540047c0 0x7fac54004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6bfff640 1 -- 172.21.15.29:0/8091290 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac54003520 con 0x7fac54003a30 2023-10-14T16:57:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac6b7fe640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac540047c0 0x7fac54004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.398+0000 7fac70846640 1 -- 172.21.15.29:0/8091290 reap_dead start 2023-10-14T16:57:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.399+0000 7fac6bfff640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 0x7fac54003e50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009fc0 tx=0x7fac540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c583831ae41dd03f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.399+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c02cc20 con 0x7fac54003a30 2023-10-14T16:57:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.399+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac5c00f040 con 0x7fac54003a30 2023-10-14T16:57:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.399+0000 7fac6affd640 1 -- 172.21.15.29:0/8091290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac5c034440 con 0x7fac54003a30 2023-10-14T16:57:42.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 -- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 msgr2=0x7fac54003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 0x7fac54003e50 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 crypto rx=0x7fac5c009fc0 tx=0x7fac540057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 -- 172.21.15.29:0/8091290 shutdown_connections 2023-10-14T16:57:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 --2- 172.21.15.29:0/8091290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac54003a30 0x7fac54003e50 unknown :-1 s=CLOSED pgs=1205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 -- 172.21.15.29:0/8091290 >> 172.21.15.29:0/8091290 conn(0x7fac6c0d2bd0 msgr2=0x7fac6c0d2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:42.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 -- 172.21.15.29:0/8091290 shutdown_connections 2023-10-14T16:57:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.400+0000 7fac71047640 1 -- 172.21.15.29:0/8091290 wait complete. 2023-10-14T16:57:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.402+0000 7fac71047640 1 Processor -- start 2023-10-14T16:57:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.402+0000 7fac71047640 1 -- start start 2023-10-14T16:57:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac54003a30 0x7fac6c1551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 0x7fac6c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c160550 con 0x7fac6c1600d0 2023-10-14T16:57:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c157530 con 0x7fac6c15d620 2023-10-14T16:57:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac71047640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac6c0b6d10 con 0x7fac54003a30 2023-10-14T16:57:42.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.403+0000 7fac6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58668/0 (socket says 172.21.15.29:58668) 2023-10-14T16:57:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 -- 172.21.15.29:0/2119513870 learned_addr learned my addr 172.21.15.29:0/2119513870 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6bfff640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac54003a30 0x7fac6c1551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 -- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac54003a30 msgr2=0x7fac6c1551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac54003a30 0x7fac6c1551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 -- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 msgr2=0x7fac6c0afff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac70846640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 0x7fac6c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 0x7fac6c0afff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 -- 172.21.15.29:0/2119513870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c0b6ff0 con 0x7fac6c1600d0 2023-10-14T16:57:42.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac70846640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 0x7fac6c0afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:42.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.404+0000 7fac6b7fe640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fac6000d480 tx=0x7fac6000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:42.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.405+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac60005480 con 0x7fac6c1600d0 2023-10-14T16:57:42.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.405+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac60003610 con 0x7fac6c1600d0 2023-10-14T16:57:42.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.405+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac600039c0 con 0x7fac6c1600d0 2023-10-14T16:57:42.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.405+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac6c0b72d0 con 0x7fac6c1600d0 2023-10-14T16:57:42.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.405+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac6c15daa0 con 0x7fac6c1600d0 2023-10-14T16:57:42.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.406+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac6c04e9b0 con 0x7fac6c1600d0 2023-10-14T16:57:42.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.407+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fac60003b20 con 0x7fac6c1600d0 2023-10-14T16:57:42.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.408+0000 7fac68ff9640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 0x7fac5806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:42.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.408+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fac60089350 con 0x7fac6c1600d0 2023-10-14T16:57:42.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.409+0000 7fac6bfff640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 0x7fac5806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:42.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.409+0000 7fac6bfff640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 0x7fac5806d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac5c036000 tx=0x7fac5c02e000 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:42.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.413+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac60054650 con 0x7fac6c1600d0 2023-10-14T16:57:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:42.609+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fac6c0cd480 con 0x7fac6c1600d0 2023-10-14T16:57:43.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.236+0000 7fac68ff9640 1 -- 172.21.15.29:0/2119513870 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v106) v1 ==== 138+0+0 (secure 0 0 0) 0x7fac60057c70 con 0x7fac6c1600d0 2023-10-14T16:57:43.239 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:57:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 msgr2=0x7fac5806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 0x7fac5806d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fac5c036000 tx=0x7fac5c02e000 comp rx=0 tx=0).stop 2023-10-14T16:57:43.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 msgr2=0x7fac6c156ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 crypto rx=0x7fac6000d480 tx=0x7fac6000d940 comp rx=0 tx=0).stop 2023-10-14T16:57:43.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 shutdown_connections 2023-10-14T16:57:43.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fac5806afc0 0x7fac5806d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac6c15d620 0x7fac6c0afff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac6c1600d0 0x7fac6c156ff0 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 --2- 172.21.15.29:0/2119513870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac54003a30 0x7fac6c1551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 >> 172.21.15.29:0/2119513870 conn(0x7fac6c0d2bd0 msgr2=0x7fac6c15c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 shutdown_connections 2023-10-14T16:57:43.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.241+0000 7fac71047640 1 -- 172.21.15.29:0/2119513870 wait complete. 2023-10-14T16:57:43.270 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:57:43.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.483+0000 7f145e24b640 1 Processor -- start 2023-10-14T16:57:43.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.484+0000 7f145e24b640 1 -- start start 2023-10-14T16:57:43.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.485+0000 7f145e24b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14580b0b20 0x7f14580cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:43.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.485+0000 7f145e24b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14580cc600 con 0x7f1458070fd0 2023-10-14T16:57:43.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.485+0000 7f145e24b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1458071470 con 0x7f14580adbe0 2023-10-14T16:57:43.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.485+0000 7f145e24b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14580cc870 con 0x7f14580b0b20 2023-10-14T16:57:43.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f144ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1458070fd0 0x7f14580b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43386/0 (socket says 172.21.15.29:43386) 2023-10-14T16:57:43.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f144ffff640 1 -- 172.21.15.29:0/410821975 learned_addr learned my addr 172.21.15.29:0/410821975 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:43.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f1457fff640 1 --2- 172.21.15.29:0/410821975 >> v2:172.21.15.29:3300/0 conn(0x7f14580b0b20 0x7f14580cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f1457fff640 1 -- 172.21.15.29:0/410821975 >> v1:172.21.15.186:6789/0 conn(0x7f14580adbe0 legacy=0x7f145806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f1457fff640 1 -- 172.21.15.29:0/410821975 >> v1:172.21.15.29:6789/0 conn(0x7f1458070fd0 legacy=0x7f14580b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.486+0000 7f1457fff640 1 -- 172.21.15.29:0/410821975 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14580ccb50 con 0x7f14580b0b20 2023-10-14T16:57:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.487+0000 7f1457fff640 1 --2- 172.21.15.29:0/410821975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580b0b20 0x7f14580cd070 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1450009d90 tx=0x7f145002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0b5aa412aa5cff3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.488+0000 7f1456ffd640 1 -- 172.21.15.29:0/410821975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f145002cc60 con 0x7f14580b0b20 2023-10-14T16:57:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.488+0000 7f1456ffd640 1 -- 172.21.15.29:0/410821975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f145002cdc0 con 0x7f14580b0b20 2023-10-14T16:57:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.488+0000 7f1456ffd640 1 -- 172.21.15.29:0/410821975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1450034660 con 0x7f14580b0b20 2023-10-14T16:57:43.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.488+0000 7f145e24b640 1 -- 172.21.15.29:0/410821975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580b0b20 msgr2=0x7f14580cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.488+0000 7f145e24b640 1 --2- 172.21.15.29:0/410821975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580b0b20 0x7f14580cd070 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 crypto rx=0x7f1450009d90 tx=0x7f145002beb0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.489+0000 7f145e24b640 1 -- 172.21.15.29:0/410821975 shutdown_connections 2023-10-14T16:57:43.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.489+0000 7f145e24b640 1 --2- 172.21.15.29:0/410821975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580b0b20 0x7f14580cd070 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.489+0000 7f145e24b640 1 -- 172.21.15.29:0/410821975 >> 172.21.15.29:0/410821975 conn(0x7f14580cb410 msgr2=0x7f14580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:43.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.489+0000 7f145e24b640 1 -- 172.21.15.29:0/410821975 shutdown_connections 2023-10-14T16:57:43.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.489+0000 7f145e24b640 1 -- 172.21.15.29:0/410821975 wait complete. 2023-10-14T16:57:43.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.491+0000 7f145e24b640 1 Processor -- start 2023-10-14T16:57:43.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.491+0000 7f145e24b640 1 -- start start 2023-10-14T16:57:43.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.492+0000 7f145e24b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1458070fd0 0x7f14581f7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:43.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.492+0000 7f145e24b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.492+0000 7f145e24b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 0x7f14580c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.492+0000 7f145e24b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580c9a30 con 0x7f14580adbe0 2023-10-14T16:57:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f145e24b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14580c9750 con 0x7f14580b0b20 2023-10-14T16:57:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f145e24b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1458163150 con 0x7f1458070fd0 2023-10-14T16:57:43.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58682/0 (socket says 172.21.15.29:58682) 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 -- 172.21.15.29:0/2686594297 learned_addr learned my addr 172.21.15.29:0/2686594297 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f1457fff640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 0x7f14580c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f14577fe640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1458070fd0 0x7f14581f7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 -- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1458070fd0 msgr2=0x7f14581f7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1458070fd0 0x7f14581f7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.493+0000 7f144ffff640 1 -- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 msgr2=0x7f14580c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:43.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f144ffff640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 0x7f14580c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:43.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f144ffff640 1 -- 172.21.15.29:0/2686594297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1450009a70 con 0x7f14580adbe0 2023-10-14T16:57:43.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f1457fff640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 0x7f14580c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:43.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f144ffff640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f144000c480 tx=0x7f144000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:43.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1440005ce0 con 0x7f14580adbe0 2023-10-14T16:57:43.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1440005e40 con 0x7f14580adbe0 2023-10-14T16:57:43.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1458163430 con 0x7f14580adbe0 2023-10-14T16:57:43.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14400072f0 con 0x7f14580adbe0 2023-10-14T16:57:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.494+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14581fa0e0 con 0x7f14580adbe0 2023-10-14T16:57:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.496+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1458162fb0 con 0x7f14580adbe0 2023-10-14T16:57:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.497+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1440007450 con 0x7f14580adbe0 2023-10-14T16:57:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.498+0000 7f1454ff9640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 0x7f143806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:43.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.498+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5271+0+0 (secure 0 0 0) 0x7f1440088a80 con 0x7f14580adbe0 2023-10-14T16:57:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.498+0000 7f14577fe640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 0x7f143806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:43.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.499+0000 7f14577fe640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 0x7f143806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1448004ea0 tx=0x7f1448009040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:43.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.504+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1458162fb0 con 0x7f14580adbe0 2023-10-14T16:57:43.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:43.699+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f1458163950 con 0x7f14580adbe0 2023-10-14T16:57:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.243+0000 7f1454ff9640 1 -- 172.21.15.29:0/2686594297 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v107) v1 ==== 170+0+0 (secure 0 0 0) 0x7f1458163950 con 0x7f14580adbe0 2023-10-14T16:57:44.245 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:57:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 msgr2=0x7f143806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 0x7f143806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f1448004ea0 tx=0x7f1448009040 comp rx=0 tx=0).stop 2023-10-14T16:57:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 msgr2=0x7f14581f9ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 secure :-1 s=READY pgs=1209 cs=0 l=1 rev1=1 crypto rx=0x7f144000c480 tx=0x7f144000c940 comp rx=0 tx=0).stop 2023-10-14T16:57:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 shutdown_connections 2023-10-14T16:57:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f143806af90 0x7f143806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14580b0b20 0x7f14580c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14580adbe0 0x7f14581f9ba0 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 --2- 172.21.15.29:0/2686594297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1458070fd0 0x7f14581f7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 >> 172.21.15.29:0/2686594297 conn(0x7f14580cb410 msgr2=0x7f14580b0000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 shutdown_connections 2023-10-14T16:57:44.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.248+0000 7f145e24b640 1 -- 172.21.15.29:0/2686594297 wait complete. 2023-10-14T16:57:44.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:57:44.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.476+0000 7f06ec2da640 1 Processor -- start 2023-10-14T16:57:44.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.476+0000 7f06ec2da640 1 -- start start 2023-10-14T16:57:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ec2da640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f06e40ad350 0x7f06e4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ec2da640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e406a730 con 0x7f06e406a2e0 2023-10-14T16:57:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ec2da640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06e40ac890 con 0x7f06e415d150 2023-10-14T16:57:44.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ec2da640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06e415d530 con 0x7f06e40ad350 2023-10-14T16:57:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ea877640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f06e406a2e0 0x7f06e40c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43390/0 (socket says 172.21.15.29:43390) 2023-10-14T16:57:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06ea877640 1 -- 172.21.15.29:0/3995028387 learned_addr learned my addr 172.21.15.29:0/3995028387 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:44.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.478+0000 7f06e9875640 1 --2- 172.21.15.29:0/3995028387 >> v2:172.21.15.186:3300/0 conn(0x7f06e40ad350 0x7f06e4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:44.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971001884 0 0) 0x7f06dc001490 con 0x7f06e406a2e0 2023-10-14T16:57:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06d0003520 con 0x7f06e406a2e0 2023-10-14T16:57:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9875640 1 -- 172.21.15.29:0/3995028387 >> v1:172.21.15.186:6789/0 conn(0x7f06e415d150 legacy=0x7f06e4160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9875640 1 -- 172.21.15.29:0/3995028387 >> v1:172.21.15.29:6789/0 conn(0x7f06e406a2e0 legacy=0x7f06e40c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3797239581 0 0) 0x7f06d0003520 con 0x7f06e406a2e0 2023-10-14T16:57:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.479+0000 7f06e9875640 1 -- 172.21.15.29:0/3995028387 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e40ad7a0 con 0x7f06e40ad350 2023-10-14T16:57:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.480+0000 7f06e9875640 1 --2- 172.21.15.29:0/3995028387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e40ad350 0x7f06e4069da0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f06d40097a0 tx=0x7f06d402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aec0fb44445f2e19 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.480+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d402f050 con 0x7f06e40ad350 2023-10-14T16:57:44.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.480+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d4031500 con 0x7f06e40ad350 2023-10-14T16:57:44.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.480+0000 7f06e9074640 1 -- 172.21.15.29:0/3995028387 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06dc001b80 con 0x7f06e40ad350 2023-10-14T16:57:44.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 -- 172.21.15.29:0/3995028387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e40ad350 msgr2=0x7f06e4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 --2- 172.21.15.29:0/3995028387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e40ad350 0x7f06e4069da0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f06d40097a0 tx=0x7f06d402b820 comp rx=0 tx=0).stop 2023-10-14T16:57:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 -- 172.21.15.29:0/3995028387 shutdown_connections 2023-10-14T16:57:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 --2- 172.21.15.29:0/3995028387 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e40ad350 0x7f06e4069da0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 -- 172.21.15.29:0/3995028387 >> 172.21.15.29:0/3995028387 conn(0x7f06e40ce9b0 msgr2=0x7f06e40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:44.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 -- 172.21.15.29:0/3995028387 shutdown_connections 2023-10-14T16:57:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.482+0000 7f06ec2da640 1 -- 172.21.15.29:0/3995028387 wait complete. 2023-10-14T16:57:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.483+0000 7f06ec2da640 1 Processor -- start 2023-10-14T16:57:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.483+0000 7f06ec2da640 1 -- start start 2023-10-14T16:57:44.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e406a2e0 0x7f06e41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 0x7f06e40b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e4155810 con 0x7f06e40ad350 2023-10-14T16:57:44.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06e415fdc0 con 0x7f06e406a2e0 2023-10-14T16:57:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ec2da640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06e415f3f0 con 0x7f06e415d150 2023-10-14T16:57:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58698/0 (socket says 172.21.15.29:58698) 2023-10-14T16:57:44.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 -- 172.21.15.29:0/850003656 learned_addr learned my addr 172.21.15.29:0/850003656 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ea877640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 0x7f06e40b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 -- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 msgr2=0x7f06e40b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 0x7f06e40b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 -- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e406a2e0 msgr2=0x7f06e41552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:44.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e406a2e0 0x7f06e41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06e9875640 1 -- 172.21.15.29:0/850003656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06d4009400 con 0x7f06e40ad350 2023-10-14T16:57:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.485+0000 7f06ea877640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 0x7f06e40b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06e9875640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f06d40098c0 tx=0x7f06d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:44.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d40044c0 con 0x7f06e40ad350 2023-10-14T16:57:44.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06d4031670 con 0x7f06e40ad350 2023-10-14T16:57:44.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06d4031a20 con 0x7f06e40ad350 2023-10-14T16:57:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06e40b3440 con 0x7f06e40ad350 2023-10-14T16:57:44.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.486+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06e40b3160 con 0x7f06e40ad350 2023-10-14T16:57:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.487+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06e404e160 con 0x7f06e40ad350 2023-10-14T16:57:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.489+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f06d4031b80 con 0x7f06e40ad350 2023-10-14T16:57:44.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.491+0000 7f06daffd640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 0x7f06c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.491+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5653+0+0 (secure 0 0 0) 0x7f06d40ab8f0 con 0x7f06e40ad350 2023-10-14T16:57:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.491+0000 7f06ea076640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 0x7f06c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.491+0000 7f06ea076640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 0x7f06c806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06e0000f60 tx=0x7f06e00079b0 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:44.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.496+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f06d4076a40 con 0x7f06e40ad350 2023-10-14T16:57:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:44.728+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f06e4156430 con 0x7f06e40ad350 2023-10-14T16:57:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.272+0000 7f06daffd640 1 -- 172.21.15.29:0/850003656 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 12 and data pool 13 v45) v1 ==== 174+0+0 (secure 0 0 0) 0x7f06d407a060 con 0x7f06e40ad350 2023-10-14T16:57:45.273 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 12 and data pool 13 2023-10-14T16:57:45.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.276+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 msgr2=0x7f06c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.276+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 0x7f06c806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f06e0000f60 tx=0x7f06e00079b0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.276+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 msgr2=0x7f06e415f880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.276+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 secure :-1 s=READY pgs=1211 cs=0 l=1 rev1=1 crypto rx=0x7f06d40098c0 tx=0x7f06d40046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 shutdown_connections 2023-10-14T16:57:45.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f06c806af90 0x7f06c806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f06e415d150 0x7f06e40b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f06e40ad350 0x7f06e415f880 unknown :-1 s=CLOSED pgs=1211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 --2- 172.21.15.29:0/850003656 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f06e406a2e0 0x7f06e41552d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 >> 172.21.15.29:0/850003656 conn(0x7f06e40ce9b0 msgr2=0x7f06e406f790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 shutdown_connections 2023-10-14T16:57:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.277+0000 7f06ec2da640 1 -- 172.21.15.29:0/850003656 wait complete. 2023-10-14T16:57:45.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:45.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.494+0000 7f242421f640 1 Processor -- start 2023-10-14T16:57:45.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.494+0000 7f242421f640 1 -- start start 2023-10-14T16:57:45.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 0x7f241c06c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f241c15e0c0 0x7f241c0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b4c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0cd630 con 0x7f241c15e0c0 2023-10-14T16:57:45.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f241c0ce1b0 con 0x7f241c06c3f0 2023-10-14T16:57:45.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.495+0000 7f242421f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c06cd10 con 0x7f241c15e540 2023-10-14T16:57:45.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f241c15e0c0 0x7f241c0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f241c15e0c0 0x7f241c0b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58712/0 (socket says 172.21.15.29:58712) 2023-10-14T16:57:45.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 -- 172.21.15.29:0/3699713361 learned_addr learned my addr 172.21.15.29:0/3699713361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:45.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f2421fbb640 1 --2- 172.21.15.29:0/3699713361 >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 0x7f241c06c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 -- 172.21.15.29:0/3699713361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 msgr2=0x7f241c0b4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24227bc640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 -- 172.21.15.29:0/3699713361 >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 msgr2=0x7f241c06c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:45.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 --2- 172.21.15.29:0/3699713361 >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 0x7f241c06c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24217ba640 1 -- 172.21.15.29:0/3699713361 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c0cd880 con 0x7f241c15e0c0 2023-10-14T16:57:45.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f2421fbb640 1 --2- 172.21.15.29:0/3699713361 >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 0x7f241c06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:45.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.496+0000 7f24227bc640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:45.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.497+0000 7f24217ba640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0b4730 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f241c0d3d00 tx=0x7f240802b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715fe7a5d944a486 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f2420fb9640 1 -- 172.21.15.29:0/3699713361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408031880 con 0x7f241c15e0c0 2023-10-14T16:57:45.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f2420fb9640 1 -- 172.21.15.29:0/3699713361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24080319e0 con 0x7f241c15e0c0 2023-10-14T16:57:45.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f2420fb9640 1 -- 172.21.15.29:0/3699713361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408031d70 con 0x7f241c15e0c0 2023-10-14T16:57:45.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 -- 172.21.15.29:0/3699713361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 msgr2=0x7f241c0b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0b4730 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 crypto rx=0x7f241c0d3d00 tx=0x7f240802b920 comp rx=0 tx=0).stop 2023-10-14T16:57:45.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 -- 172.21.15.29:0/3699713361 shutdown_connections 2023-10-14T16:57:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 --2- 172.21.15.29:0/3699713361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0b4730 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 --2- 172.21.15.29:0/3699713361 >> v2:172.21.15.186:3300/0 conn(0x7f241c06c3f0 0x7f241c06c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 -- 172.21.15.29:0/3699713361 >> 172.21.15.29:0/3699713361 conn(0x7f241c0d2cb0 msgr2=0x7f241c0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:45.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 -- 172.21.15.29:0/3699713361 shutdown_connections 2023-10-14T16:57:45.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.498+0000 7f242421f640 1 -- 172.21.15.29:0/3699713361 wait complete. 2023-10-14T16:57:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.500+0000 7f242421f640 1 Processor -- start 2023-10-14T16:57:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.500+0000 7f242421f640 1 -- start start 2023-10-14T16:57:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 0x7f241c0c8830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1f68f0 con 0x7f241c15e0c0 2023-10-14T16:57:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1f5c70 con 0x7f241c06c3f0 2023-10-14T16:57:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.501+0000 7f242421f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f241c1f5f50 con 0x7f241c15e540 2023-10-14T16:57:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58716/0 (socket says 172.21.15.29:58716) 2023-10-14T16:57:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 -- 172.21.15.29:0/1405229083 learned_addr learned my addr 172.21.15.29:0/1405229083 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f2421fbb640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 0x7f241c0c8830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 -- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 msgr2=0x7f241c0b7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24227bc640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 -- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 msgr2=0x7f241c0c8830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 0x7f241c0c8830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24217ba640 1 -- 172.21.15.29:0/1405229083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24080094d0 con 0x7f241c15e0c0 2023-10-14T16:57:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f24227bc640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:45.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.502+0000 7f2421fbb640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 0x7f241c0c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f24217ba640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f240802b960 tx=0x7f24080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408031870 con 0x7f241c15e0c0 2023-10-14T16:57:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24080443f0 con 0x7f241c15e0c0 2023-10-14T16:57:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c1f6230 con 0x7f241c15e0c0 2023-10-14T16:57:45.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24080447a0 con 0x7f241c15e0c0 2023-10-14T16:57:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.503+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c0c8d70 con 0x7f241c15e0c0 2023-10-14T16:57:45.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.504+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c1f6a30 con 0x7f241c15e0c0 2023-10-14T16:57:45.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.506+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f240804d450 con 0x7f241c15e0c0 2023-10-14T16:57:45.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.506+0000 7f241a7fc640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 0x7f240006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.507+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f24080ac150 con 0x7f241c15e0c0 2023-10-14T16:57:45.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.507+0000 7f2421fbb640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 0x7f240006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.508+0000 7f2421fbb640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 0x7f240006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2410008f00 tx=0x7f2410008450 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.511+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f241c1f6a30 con 0x7f241c15e0c0 2023-10-14T16:57:45.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.700+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f241c1f6a30 con 0x7f241c15e0c0 2023-10-14T16:57:45.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.703+0000 7f241a7fc640 1 -- 172.21.15.29:0/1405229083 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16642 (secure 0 0 0) 0x7f241c1f6a30 con 0x7f241c15e0c0 2023-10-14T16:57:45.704 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:45.705 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":108,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:44.730275+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.705+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 msgr2=0x7f240006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.705+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 0x7f240006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f2410008f00 tx=0x7f2410008450 comp rx=0 tx=0).stop 2023-10-14T16:57:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.705+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 msgr2=0x7f241c0cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.705+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 secure :-1 s=READY pgs=1213 cs=0 l=1 rev1=1 crypto rx=0x7f240802b960 tx=0x7f24080046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 shutdown_connections 2023-10-14T16:57:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f240006af90 0x7f240006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f241c15e540 0x7f241c0b7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f241c15e0c0 0x7f241c0cc370 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 --2- 172.21.15.29:0/1405229083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f241c06c3f0 0x7f241c0c8830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 >> 172.21.15.29:0/1405229083 conn(0x7f241c0d2cb0 msgr2=0x7f241c15ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 shutdown_connections 2023-10-14T16:57:45.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.706+0000 7f242421f640 1 -- 172.21.15.29:0/1405229083 wait complete. 2023-10-14T16:57:45.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.937+0000 7ff96216a640 1 Processor -- start 2023-10-14T16:57:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.937+0000 7ff96216a640 1 -- start start 2023-10-14T16:57:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff96216a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff95c15c9f0 0x7ff95c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff96216a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95c160a90 con 0x7ff95c15dde0 2023-10-14T16:57:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff96216a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff95c0cd900 con 0x7ff95c0c9230 2023-10-14T16:57:45.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff96216a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff95c0cdec0 con 0x7ff95c15c9f0 2023-10-14T16:57:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff961969640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff95c15dde0 0x7ff95c0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:43402/0 (socket says 172.21.15.29:43402) 2023-10-14T16:57:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.939+0000 7ff961969640 1 -- 172.21.15.29:0/2043736854 learned_addr learned my addr 172.21.15.29:0/2043736854 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff961168640 1 --2- 172.21.15.29:0/2043736854 >> v2:172.21.15.186:3300/0 conn(0x7ff95c15c9f0 0x7ff95c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646982737 0 0) 0x7ff95c160a90 con 0x7ff95c15dde0 2023-10-14T16:57:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff948003520 con 0x7ff95c15dde0 2023-10-14T16:57:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1109622228 0 0) 0x7ff948003520 con 0x7ff95c15dde0 2023-10-14T16:57:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 >> v2:172.21.15.186:3300/0 conn(0x7ff95c15c9f0 msgr2=0x7ff95c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 --2- 172.21.15.29:0/2043736854 >> v2:172.21.15.186:3300/0 conn(0x7ff95c15c9f0 0x7ff95c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 >> v1:172.21.15.186:6789/0 conn(0x7ff95c0c9230 legacy=0x7ff95c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c0cdbe0 con 0x7ff95c15dde0 2023-10-14T16:57:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.940+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff9540019f0 con 0x7ff95c15dde0 2023-10-14T16:57:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 >> v1:172.21.15.29:6789/0 conn(0x7ff95c15dde0 legacy=0x7ff95c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480039c0 0x7ff948003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 0x7ff948004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff948005180 0x7ff94800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94800c050 con 0x7ff9480047c0 2023-10-14T16:57:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff94800c1c0 con 0x7ff948005180 2023-10-14T16:57:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff94800c380 con 0x7ff9480039c0 2023-10-14T16:57:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 0x7ff948004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff961168640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480039c0 0x7ff948003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff961969640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff948005180 0x7ff94800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 -- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480039c0 msgr2=0x7ff948003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480039c0 0x7ff948003de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 -- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff948005180 msgr2=0x7ff94800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff948005180 0x7ff94800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff960967640 1 -- 172.21.15.29:0/2043736854 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff948003520 con 0x7ff9480047c0 2023-10-14T16:57:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff961969640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff948005180 0x7ff94800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.941+0000 7ff961969640 1 -- 172.21.15.29:0/2043736854 reap_dead start 2023-10-14T16:57:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff960967640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 0x7ff948004c40 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff94c001750 tx=0x7ff9480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7caa71ad41aef1d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c0052d0 con 0x7ff9480047c0 2023-10-14T16:57:45.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c02e050 con 0x7ff9480047c0 2023-10-14T16:57:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff953fff640 1 -- 172.21.15.29:0/2043736854 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c003830 con 0x7ff9480047c0 2023-10-14T16:57:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff96216a640 1 -- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 msgr2=0x7ff948004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff96216a640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 0x7ff948004c40 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 crypto rx=0x7ff94c001750 tx=0x7ff9480057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff96216a640 1 -- 172.21.15.29:0/2043736854 shutdown_connections 2023-10-14T16:57:45.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff96216a640 1 --2- 172.21.15.29:0/2043736854 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9480047c0 0x7ff948004c40 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.943+0000 7ff96216a640 1 -- 172.21.15.29:0/2043736854 >> 172.21.15.29:0/2043736854 conn(0x7ff95c0ce6a0 msgr2=0x7ff95c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.944+0000 7ff96216a640 1 -- 172.21.15.29:0/2043736854 shutdown_connections 2023-10-14T16:57:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.944+0000 7ff96216a640 1 -- 172.21.15.29:0/2043736854 wait complete. 2023-10-14T16:57:45.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.945+0000 7ff96216a640 1 Processor -- start 2023-10-14T16:57:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.945+0000 7ff96216a640 1 -- start start 2023-10-14T16:57:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 0x7ff95c1f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95c1f4fb0 0x7ff95c0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c06b800 con 0x7ff95c0c8120 2023-10-14T16:57:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c069da0 con 0x7ff95c1f4fb0 2023-10-14T16:57:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.946+0000 7ff96216a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff95c0c85a0 con 0x7ff9480047c0 2023-10-14T16:57:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58726/0 (socket says 172.21.15.29:58726) 2023-10-14T16:57:45.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 -- 172.21.15.29:0/3156927819 learned_addr learned my addr 172.21.15.29:0/3156927819 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff961168640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 0x7ff95c1f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff961969640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95c1f4fb0 0x7ff95c0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 -- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 msgr2=0x7ff95c1f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 0x7ff95c1f2300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 -- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95c1f4fb0 msgr2=0x7ff95c0c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:45.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95c1f4fb0 0x7ff95c0c6d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff960967640 1 -- 172.21.15.29:0/3156927819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95c06ba10 con 0x7ff95c0c8120 2023-10-14T16:57:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.947+0000 7ff961168640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 0x7ff95c1f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff960967640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005040 tx=0x7ff94c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c0076c0 con 0x7ff95c0c8120 2023-10-14T16:57:45.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff94c007a30 con 0x7ff95c0c8120 2023-10-14T16:57:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff94c006be0 con 0x7ff95c0c8120 2023-10-14T16:57:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95c06bcf0 con 0x7ff95c0c8120 2023-10-14T16:57:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.948+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95c06c2b0 con 0x7ff95c0c8120 2023-10-14T16:57:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.950+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95c06feb0 con 0x7ff95c0c8120 2023-10-14T16:57:45.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.951+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff94c006d40 con 0x7ff95c0c8120 2023-10-14T16:57:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.951+0000 7ff951ffb640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 0x7ff94006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:45.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.952+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff94c0ab4c0 con 0x7ff95c0c8120 2023-10-14T16:57:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.952+0000 7ff961168640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 0x7ff94006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.952+0000 7ff961168640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 0x7ff94006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff95c063ab0 tx=0x7ff958008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:45.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:45.956+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff94c076310 con 0x7ff95c0c8120 2023-10-14T16:57:45.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:57:45.991 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:57:46.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.161+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff95c04e9b0 con 0x7ff95c0c8120 2023-10-14T16:57:46.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.164+0000 7ff951ffb640 1 -- 172.21.15.29:0/3156927819 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v108) v1 ==== 74+0+16642 (secure 0 0 0) 0x7ff94c07ee60 con 0x7ff95c0c8120 2023-10-14T16:57:46.165 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:46.165 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":108,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:44.730275+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 msgr2=0x7ff94006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:46.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 0x7ff94006d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7ff95c063ab0 tx=0x7ff958008040 comp rx=0 tx=0).stop 2023-10-14T16:57:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 msgr2=0x7ff95c06b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 secure :-1 s=READY pgs=1216 cs=0 l=1 rev1=1 crypto rx=0x7ff94c005040 tx=0x7ff94c001120 comp rx=0 tx=0).stop 2023-10-14T16:57:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 shutdown_connections 2023-10-14T16:57:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7ff94006afc0 0x7ff94006d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff95c1f4fb0 0x7ff95c0c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff95c0c8120 0x7ff95c06b2c0 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 --2- 172.21.15.29:0/3156927819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9480047c0 0x7ff95c1f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.166+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 >> 172.21.15.29:0/3156927819 conn(0x7ff95c0ce6a0 msgr2=0x7ff95c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:46.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.167+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 shutdown_connections 2023-10-14T16:57:46.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.167+0000 7ff96216a640 1 -- 172.21.15.29:0/3156927819 wait complete. 2023-10-14T16:57:46.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:57:46.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.418+0000 7f042a3c4640 1 Processor -- start 2023-10-14T16:57:46.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.419+0000 7f042a3c4640 1 -- start start 2023-10-14T16:57:46.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f042406c040 0x7f042406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04240ad970 0x7f042415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 0x7f042415f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042406c960 con 0x7f042406c040 2023-10-14T16:57:46.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f042415b9d0 con 0x7f042415ebc0 2023-10-14T16:57:46.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.420+0000 7f042a3c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042415bcb0 con 0x7f04240ad970 2023-10-14T16:57:46.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f042406c040 0x7f042406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f042406c040 0x7f042406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58738/0 (socket says 172.21.15.29:58738) 2023-10-14T16:57:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 -- 172.21.15.29:0/3132263385 learned_addr learned my addr 172.21.15.29:0/3132263385 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f0428bc1640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04240ad970 0x7f042415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f0429bc3640 1 --2- 172.21.15.29:0/3132263385 >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 0x7f042415f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 -- 172.21.15.29:0/3132263385 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04240ad970 msgr2=0x7f042415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:46.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04240ad970 0x7f042415e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 -- 172.21.15.29:0/3132263385 >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 msgr2=0x7f042415f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:46.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 --2- 172.21.15.29:0/3132263385 >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 0x7f042415f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f04293c2640 1 -- 172.21.15.29:0/3132263385 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f042415bf90 con 0x7f042406c040 2023-10-14T16:57:46.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.421+0000 7f0429bc3640 1 --2- 172.21.15.29:0/3132263385 >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 0x7f042415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:46.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.423+0000 7f04293c2640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042406c040 0x7f042406c420 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f0420009870 tx=0x7f042002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2ce194ee68a268f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.423+0000 7f041bfff640 1 -- 172.21.15.29:0/3132263385 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420031640 con 0x7f042406c040 2023-10-14T16:57:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.423+0000 7f041bfff640 1 -- 172.21.15.29:0/3132263385 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04200317a0 con 0x7f042406c040 2023-10-14T16:57:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.423+0000 7f041bfff640 1 -- 172.21.15.29:0/3132263385 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0420031b30 con 0x7f042406c040 2023-10-14T16:57:46.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 -- 172.21.15.29:0/3132263385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042406c040 msgr2=0x7f042406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042406c040 0x7f042406c420 secure :-1 s=READY pgs=1217 cs=0 l=1 rev1=1 crypto rx=0x7f0420009870 tx=0x7f042002b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 -- 172.21.15.29:0/3132263385 shutdown_connections 2023-10-14T16:57:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 --2- 172.21.15.29:0/3132263385 >> v2:172.21.15.186:3300/0 conn(0x7f042415ebc0 0x7f042415f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04240ad970 0x7f042415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 --2- 172.21.15.29:0/3132263385 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042406c040 0x7f042406c420 unknown :-1 s=CLOSED pgs=1217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 -- 172.21.15.29:0/3132263385 >> 172.21.15.29:0/3132263385 conn(0x7f04240ce6a0 msgr2=0x7f04240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 -- 172.21.15.29:0/3132263385 shutdown_connections 2023-10-14T16:57:46.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.424+0000 7f042a3c4640 1 -- 172.21.15.29:0/3132263385 wait complete. 2023-10-14T16:57:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.426+0000 7f042a3c4640 1 Processor -- start 2023-10-14T16:57:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.426+0000 7f042a3c4640 1 -- start start 2023-10-14T16:57:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f042406c040 0x7f042414eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04240ad970 0x7f04240c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04240c7110 con 0x7f042415ebc0 2023-10-14T16:57:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04240c73f0 con 0x7f04240ad970 2023-10-14T16:57:46.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f042a3c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04240c76d0 con 0x7f042406c040 2023-10-14T16:57:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f0429bc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f0429bc3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58750/0 (socket says 172.21.15.29:58750) 2023-10-14T16:57:46.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f0429bc3640 1 -- 172.21.15.29:0/285497758 learned_addr learned my addr 172.21.15.29:0/285497758 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f0428bc1640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04240ad970 0x7f04240c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.427+0000 7f04293c2640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f042406c040 0x7f042414eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 -- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f042406c040 msgr2=0x7f042414eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f042406c040 0x7f042414eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 -- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04240ad970 msgr2=0x7f04240c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04240ad970 0x7f04240c67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 -- 172.21.15.29:0/285497758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04200094d0 con 0x7f042415ebc0 2023-10-14T16:57:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0429bc3640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f041c00cef0 tx=0x7f041c00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:46.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00d840 con 0x7f042415ebc0 2023-10-14T16:57:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041c00d9c0 con 0x7f042415ebc0 2023-10-14T16:57:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04240c79b0 con 0x7f042415ebc0 2023-10-14T16:57:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.428+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041c00dd70 con 0x7f042415ebc0 2023-10-14T16:57:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.429+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042414f060 con 0x7f042415ebc0 2023-10-14T16:57:46.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.430+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f042406b330 con 0x7f042415ebc0 2023-10-14T16:57:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.431+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f041c01e3b0 con 0x7f042415ebc0 2023-10-14T16:57:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.432+0000 7f0419ffb640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 0x7f040806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.432+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f041c08a260 con 0x7f042415ebc0 2023-10-14T16:57:46.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.432+0000 7f04293c2640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 0x7f040806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.433+0000 7f04293c2640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 0x7f040806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0420036000 tx=0x7f042003c040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:46.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.436+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f041c055290 con 0x7f042415ebc0 2023-10-14T16:57:46.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:46.671+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f04240c7ed0 con 0x7f042415ebc0 2023-10-14T16:57:47.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.296+0000 7f0419ffb640 1 -- 172.21.15.29:0/285497758 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v48) v1 ==== 118+0+0 (secure 0 0 0) 0x7f04240c7ed0 con 0x7f042415ebc0 2023-10-14T16:57:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.300+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 msgr2=0x7f040806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:47.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.300+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 0x7f040806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0420036000 tx=0x7f042003c040 comp rx=0 tx=0).stop 2023-10-14T16:57:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.300+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 msgr2=0x7f04240b0310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.300+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 crypto rx=0x7f041c00cef0 tx=0x7f041c00a710 comp rx=0 tx=0).stop 2023-10-14T16:57:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.300+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 shutdown_connections 2023-10-14T16:57:47.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f040806af90 0x7f040806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f042415ebc0 0x7f04240b0310 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04240ad970 0x7f04240c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 --2- 172.21.15.29:0/285497758 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f042406c040 0x7f042414eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 >> 172.21.15.29:0/285497758 conn(0x7f04240ce6a0 msgr2=0x7f04240ca6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:47.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 shutdown_connections 2023-10-14T16:57:47.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.301+0000 7f042a3c4640 1 -- 172.21.15.29:0/285497758 wait complete. 2023-10-14T16:57:47.323 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:57:47.324 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:57:47.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.552+0000 7fb3b34d1640 1 Processor -- start 2023-10-14T16:57:47.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.552+0000 7fb3b34d1640 1 -- start start 2023-10-14T16:57:47.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.553+0000 7fb3b34d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.553+0000 7fb3b34d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.553+0000 7fb3b34d1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3ac0cad00 con 0x7fb3ac070240 2023-10-14T16:57:47.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.553+0000 7fb3b34d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b2d00 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.553+0000 7fb3b34d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac0b2fe0 con 0x7fb3ac0ca8a0 2023-10-14T16:57:47.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:47.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58764/0 (socket says 172.21.15.29:58764) 2023-10-14T16:57:47.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3118452573 learned_addr learned my addr 172.21.15.29:0/3118452573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:47.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3118452573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac0ca8a0 msgr2=0x7fb3ac0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:47.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 --2- 172.21.15.29:0/3118452573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3118452573 >> v1:172.21.15.186:6789/0 conn(0x7fb3ac070240 legacy=0x7fb3ac0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:47.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.554+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3118452573 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ac0b32c0 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.555+0000 7fb3b24cf640 1 --2- 172.21.15.29:0/3118452573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362e908a7d6d6269 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:47.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b14cd640 1 -- 172.21.15.29:0/3118452573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a802ca70 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b14cd640 1 -- 172.21.15.29:0/3118452573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a802cbd0 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b14cd640 1 -- 172.21.15.29:0/3118452573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a80343f0 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3118452573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac15aaa0 msgr2=0x7fb3ac0b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:47.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3118452573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a802bed0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3118452573 shutdown_connections 2023-10-14T16:57:47.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3118452573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3118452573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac15aaa0 0x7fb3ac0b2600 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3118452573 >> 172.21.15.29:0/3118452573 conn(0x7fb3ac0cc230 msgr2=0x7fb3ac0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:47.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3118452573 shutdown_connections 2023-10-14T16:57:47.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.556+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3118452573 wait complete. 2023-10-14T16:57:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.558+0000 7fb3b34d1640 1 Processor -- start 2023-10-14T16:57:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.558+0000 7fb3b34d1640 1 -- start start 2023-10-14T16:57:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac15aaa0 0x7fb3ac0c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac153bc0 con 0x7fb3ac070240 2023-10-14T16:57:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac153320 con 0x7fb3ac0ca8a0 2023-10-14T16:57:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.559+0000 7fb3b34d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ac153600 con 0x7fb3ac15aaa0 2023-10-14T16:57:47.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58772/0 (socket says 172.21.15.29:58772) 2023-10-14T16:57:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3994183933 learned_addr learned my addr 172.21.15.29:0/3994183933 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac15aaa0 msgr2=0x7fb3ac0c9ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:47.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac15aaa0 0x7fb3ac0c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca8a0 msgr2=0x7fb3ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 -- 172.21.15.29:0/3994183933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3a8009b30 con 0x7fb3ac070240 2023-10-14T16:57:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.560+0000 7fb3b24cf640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.561+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a803e450 con 0x7fb3ac070240 2023-10-14T16:57:47.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.561+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a803e5d0 con 0x7fb3ac070240 2023-10-14T16:57:47.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.561+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3a803e980 con 0x7fb3ac070240 2023-10-14T16:57:47.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.561+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ac1538e0 con 0x7fb3ac070240 2023-10-14T16:57:47.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.561+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ac0acdd0 con 0x7fb3ac070240 2023-10-14T16:57:47.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.562+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ac0cd600 con 0x7fb3ac070240 2023-10-14T16:57:47.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.563+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3a803eae0 con 0x7fb3ac070240 2023-10-14T16:57:47.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.564+0000 7fb3a2ffd640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 0x7fb39006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:47.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.564+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fb3a80ac720 con 0x7fb3ac070240 2023-10-14T16:57:47.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.564+0000 7fb3b1cce640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 0x7fb39006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:47.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.565+0000 7fb3b1cce640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 0x7fb39006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac0b3dc0 tx=0x7fb39c006c90 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:47.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.568+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3a80777a0 con 0x7fb3ac070240 2023-10-14T16:57:47.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:47.793+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb3ac0acf60 con 0x7fb3ac070240 2023-10-14T16:57:48.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.299+0000 7fb3a2ffd640 1 -- 172.21.15.29:0/3994183933 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v49) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb3a807adc0 con 0x7fb3ac070240 2023-10-14T16:57:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 msgr2=0x7fb39006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 0x7fb39006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac0b3dc0 tx=0x7fb39c006c90 comp rx=0 tx=0).stop 2023-10-14T16:57:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 msgr2=0x7fb3ac14fde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8009e50 tx=0x7fb3a8004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 shutdown_connections 2023-10-14T16:57:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fb39006af90 0x7fb39006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ac15aaa0 0x7fb3ac0c9ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ac0ca8a0 0x7fb3ac0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 --2- 172.21.15.29:0/3994183933 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ac070240 0x7fb3ac14fde0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.303+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 >> 172.21.15.29:0/3994183933 conn(0x7fb3ac0cc230 msgr2=0x7fb3ac0b1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.304+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 shutdown_connections 2023-10-14T16:57:48.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.304+0000 7fb3b34d1640 1 -- 172.21.15.29:0/3994183933 wait complete. 2023-10-14T16:57:48.325 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:57:48.326 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.545+0000 7f0d3b59e640 1 Processor -- start 2023-10-14T16:57:48.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.546+0000 7f0d3b59e640 1 -- start start 2023-10-14T16:57:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d3c06c040 0x7f0d3c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 0x7f0d3c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c15ebc0 0x7f0d3c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c06c960 con 0x7f0d3c06c040 2023-10-14T16:57:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c15b9d0 con 0x7f0d3c0ad970 2023-10-14T16:57:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c15bcb0 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.547+0000 7f0d3a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d3c06c040 0x7f0d3c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0d3c06c040 0x7f0d3c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58782/0 (socket says 172.21.15.29:58782) 2023-10-14T16:57:48.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 -- 172.21.15.29:0/1837050485 learned_addr learned my addr 172.21.15.29:0/1837050485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d33fff640 1 --2- 172.21.15.29:0/1837050485 >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 0x7f0d3c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3ad9d640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c15ebc0 0x7f0d3c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 -- 172.21.15.29:0/1837050485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c15ebc0 msgr2=0x7f0d3c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c15ebc0 0x7f0d3c15f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 -- 172.21.15.29:0/1837050485 >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 msgr2=0x7f0d3c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:48.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/1837050485 >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 0x7f0d3c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d3a59c640 1 -- 172.21.15.29:0/1837050485 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c15bf90 con 0x7f0d3c06c040 2023-10-14T16:57:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.548+0000 7f0d33fff640 1 --2- 172.21.15.29:0/1837050485 >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 0x7f0d3c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.549+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c06c040 0x7f0d3c06c420 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0cb060 tx=0x7f0d2c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bb7cc237099879f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:48.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d39d9b640 1 -- 172.21.15.29:0/1837050485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c031860 con 0x7f0d3c06c040 2023-10-14T16:57:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d39d9b640 1 -- 172.21.15.29:0/1837050485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c0319c0 con 0x7f0d3c06c040 2023-10-14T16:57:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d39d9b640 1 -- 172.21.15.29:0/1837050485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2c031d50 con 0x7f0d3c06c040 2023-10-14T16:57:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 -- 172.21.15.29:0/1837050485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c06c040 msgr2=0x7f0d3c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c06c040 0x7f0d3c06c420 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 crypto rx=0x7f0d3c0cb060 tx=0x7f0d2c02b900 comp rx=0 tx=0).stop 2023-10-14T16:57:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 -- 172.21.15.29:0/1837050485 shutdown_connections 2023-10-14T16:57:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c15ebc0 0x7f0d3c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/1837050485 >> v2:172.21.15.186:3300/0 conn(0x7f0d3c0ad970 0x7f0d3c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/1837050485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c06c040 0x7f0d3c06c420 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 -- 172.21.15.29:0/1837050485 >> 172.21.15.29:0/1837050485 conn(0x7f0d3c0ce6a0 msgr2=0x7f0d3c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 -- 172.21.15.29:0/1837050485 shutdown_connections 2023-10-14T16:57:48.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.550+0000 7f0d3b59e640 1 -- 172.21.15.29:0/1837050485 wait complete. 2023-10-14T16:57:48.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.552+0000 7f0d3b59e640 1 Processor -- start 2023-10-14T16:57:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.552+0000 7f0d3b59e640 1 -- start start 2023-10-14T16:57:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 0x7f0d3c153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c0ad970 0x7f0d3c152e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c0c6f80 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c153350 con 0x7f0d3c06c040 2023-10-14T16:57:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.553+0000 7f0d3b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d3c1e4960 con 0x7f0d3c0ad970 2023-10-14T16:57:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58788/0 (socket says 172.21.15.29:58788) 2023-10-14T16:57:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 -- 172.21.15.29:0/109336624 learned_addr learned my addr 172.21.15.29:0/109336624 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d33fff640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c0ad970 0x7f0d3c152e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 0x7f0d3c153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 -- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c0ad970 msgr2=0x7f0d3c152e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c0ad970 0x7f0d3c152e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 -- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 msgr2=0x7f0d3c153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 0x7f0d3c153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3ad9d640 1 -- 172.21.15.29:0/109336624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2c0094d0 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.554+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 0x7f0d3c153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d3ad9d640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400cc30 tx=0x7f0d3400a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d3400ce40 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d34003990 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c1e4c40 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d34003d20 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.555+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c1e5200 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.556+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c06b330 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.557+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0d34003e80 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.558+0000 7f0d32ffd640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 0x7f0d1c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:48.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.558+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f0d34089bc0 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.559+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 0x7f0d1c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:48.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.559+0000 7f0d3a59c640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 0x7f0d1c06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c036000 tx=0x7f0d2c03c040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:48.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.563+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d34054ab0 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.798+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d3c04e9b0 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.799+0000 7f0d32ffd640 1 -- 172.21.15.29:0/109336624 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f0d3401d070 con 0x7f0d3c15ebc0 2023-10-14T16:57:48.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:48.800 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:57:48.800 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:57:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 msgr2=0x7f0d1c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 0x7f0d1c06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0d2c036000 tx=0x7f0d2c03c040 comp rx=0 tx=0).stop 2023-10-14T16:57:48.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 msgr2=0x7f0d3c0aec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 secure :-1 s=READY pgs=1222 cs=0 l=1 rev1=1 crypto rx=0x7f0d3400cc30 tx=0x7f0d3400a5c0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 shutdown_connections 2023-10-14T16:57:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f0d1c06afc0 0x7f0d1c06d480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0d3c15ebc0 0x7f0d3c0aec80 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0d3c0ad970 0x7f0d3c152e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 --2- 172.21.15.29:0/109336624 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0d3c06c040 0x7f0d3c153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:48.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 >> 172.21.15.29:0/109336624 conn(0x7f0d3c0ce6a0 msgr2=0x7f0d3c15d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:48.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.802+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 shutdown_connections 2023-10-14T16:57:48.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:48.803+0000 7f0d3b59e640 1 -- 172.21.15.29:0/109336624 wait complete. 2023-10-14T16:57:48.823 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:57:49.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.048+0000 7f502e626640 1 Processor -- start 2023-10-14T16:57:49.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.048+0000 7f502e626640 1 -- start start 2023-10-14T16:57:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502e626640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f50281615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502e626640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f502815d3f0 0x7f502815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502e626640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f502815e3e0 con 0x7f50280cd0c0 2023-10-14T16:57:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502e626640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50280cd4a0 con 0x7f502815d3f0 2023-10-14T16:57:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502e626640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f502815e100 con 0x7f50280b0b20 2023-10-14T16:57:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.050+0000 7f502cbc3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f502815d3f0 0x7f502815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f502815d3f0 0x7f502815d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58794/0 (socket says 172.21.15.29:58794) 2023-10-14T16:57:49.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 -- 172.21.15.29:0/211395121 learned_addr learned my addr 172.21.15.29:0/211395121 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f501ffff640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f50281615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 -- 172.21.15.29:0/211395121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 msgr2=0x7f50281615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f50281615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 -- 172.21.15.29:0/211395121 >> v1:172.21.15.186:6789/0 conn(0x7f50280cd0c0 legacy=0x7f50280b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f502cbc3640 1 -- 172.21.15.29:0/211395121 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5028160b10 con 0x7f502815d3f0 2023-10-14T16:57:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.051+0000 7f501ffff640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f50281615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.052+0000 7f502cbc3640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f502815d3f0 0x7f502815d860 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5020009e50 tx=0x7f502002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6e8151581e3d7b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.052+0000 7f50277fe640 1 -- 172.21.15.29:0/211395121 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f502002c9f0 con 0x7f502815d3f0 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f50277fe640 1 -- 172.21.15.29:0/211395121 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f502002cb50 con 0x7f502815d3f0 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f50277fe640 1 -- 172.21.15.29:0/211395121 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50200343f0 con 0x7f502815d3f0 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 -- 172.21.15.29:0/211395121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f502815d3f0 msgr2=0x7f502815d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f502815d3f0 0x7f502815d860 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 crypto rx=0x7f5020009e50 tx=0x7f502002be50 comp rx=0 tx=0).stop 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 -- 172.21.15.29:0/211395121 shutdown_connections 2023-10-14T16:57:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f502815d3f0 0x7f502815d860 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 --2- 172.21.15.29:0/211395121 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f50281615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 -- 172.21.15.29:0/211395121 >> 172.21.15.29:0/211395121 conn(0x7f50280cb410 msgr2=0x7f50280cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 -- 172.21.15.29:0/211395121 shutdown_connections 2023-10-14T16:57:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.053+0000 7f502e626640 1 -- 172.21.15.29:0/211395121 wait complete. 2023-10-14T16:57:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.055+0000 7f502e626640 1 Processor -- start 2023-10-14T16:57:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.055+0000 7f502e626640 1 -- start start 2023-10-14T16:57:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f5028153090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f502815d3f0 0x7f5028152340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280ae570 con 0x7f50280cd0c0 2023-10-14T16:57:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50280ae850 con 0x7f502815d3f0 2023-10-14T16:57:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f502e626640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50280aeb30 con 0x7f50280b0b20 2023-10-14T16:57:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.056+0000 7f501ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58800/0 (socket says 172.21.15.29:58800) 2023-10-14T16:57:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 -- 172.21.15.29:0/1861038740 learned_addr learned my addr 172.21.15.29:0/1861038740 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:49.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f5027fff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f5028153090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 -- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 msgr2=0x7f5028153090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f5028153090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 -- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f502815d3f0 msgr2=0x7f5028152340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f502815d3f0 0x7f5028152340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 -- 172.21.15.29:0/1861038740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5020009b30 con 0x7f50280cd0c0 2023-10-14T16:57:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f501ffff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f501000c7f0 tx=0x7f501000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.057+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010007820 con 0x7f50280cd0c0 2023-10-14T16:57:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.058+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5010007980 con 0x7f50280cd0c0 2023-10-14T16:57:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.058+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5010007cf0 con 0x7f50280cd0c0 2023-10-14T16:57:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.058+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50280aee10 con 0x7f50280cd0c0 2023-10-14T16:57:49.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.058+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5028152880 con 0x7f50280cd0c0 2023-10-14T16:57:49.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.059+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502804e160 con 0x7f50280cd0c0 2023-10-14T16:57:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.060+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50100113f0 con 0x7f50280cd0c0 2023-10-14T16:57:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.061+0000 7f50257fa640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 0x7f500806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.061+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5010089bf0 con 0x7f50280cd0c0 2023-10-14T16:57:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.061+0000 7f5027fff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 0x7f500806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.062+0000 7f5027fff640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 0x7f500806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5018006fd0 tx=0x7f5018008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.065+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5010054a40 con 0x7f50280cd0c0 2023-10-14T16:57:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.264+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50280b4910 con 0x7f50280cd0c0 2023-10-14T16:57:49.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.267+0000 7f50257fa640 1 -- 172.21.15.29:0/1861038740 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v109) v1 ==== 74+0+16629 (secure 0 0 0) 0x7f501001f090 con 0x7f50280cd0c0 2023-10-14T16:57:49.271 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:49.271 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":109,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:57:46.167027+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:57:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 msgr2=0x7f500806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 0x7f500806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5018006fd0 tx=0x7f5018008040 comp rx=0 tx=0).stop 2023-10-14T16:57:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 msgr2=0x7f5028155470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 crypto rx=0x7f501000c7f0 tx=0x7f501000ccb0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 shutdown_connections 2023-10-14T16:57:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f500806af90 0x7f500806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f502815d3f0 0x7f5028152340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50280cd0c0 0x7f5028155470 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 --2- 172.21.15.29:0/1861038740 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50280b0b20 0x7f5028153090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 >> 172.21.15.29:0/1861038740 conn(0x7f50280cb410 msgr2=0x7f502815ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 shutdown_connections 2023-10-14T16:57:49.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.270+0000 7f502e626640 1 -- 172.21.15.29:0/1861038740 wait complete. 2023-10-14T16:57:49.289 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:57:49.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:57:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.507+0000 7efe6beda640 1 Processor -- start 2023-10-14T16:57:49.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.507+0000 7efe6beda640 1 -- start start 2023-10-14T16:57:49.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe6beda640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe64159b20 0x7efe6406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe6beda640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159f00 0x7efe640b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe6beda640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe6406f7f0 con 0x7efe6406c120 2023-10-14T16:57:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe6beda640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe6406c5e0 con 0x7efe64159b20 2023-10-14T16:57:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe6beda640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6406c8c0 con 0x7efe64159f00 2023-10-14T16:57:49.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe69c76640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe64159b20 0x7efe6406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe69c76640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe64159b20 0x7efe6406bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58804/0 (socket says 172.21.15.29:58804) 2023-10-14T16:57:49.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.509+0000 7efe69c76640 1 -- 172.21.15.29:0/3943498553 learned_addr learned my addr 172.21.15.29:0/3943498553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:49.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.510+0000 7efe69475640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159f00 0x7efe640b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.510+0000 7efe69c76640 1 -- 172.21.15.29:0/3943498553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159f00 msgr2=0x7efe640b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:49.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.510+0000 7efe69c76640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159f00 0x7efe640b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.510+0000 7efe69c76640 1 -- 172.21.15.29:0/3943498553 >> v1:172.21.15.186:6789/0 conn(0x7efe6406c120 legacy=0x7efe640cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:49.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.510+0000 7efe69c76640 1 -- 172.21.15.29:0/3943498553 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6406cba0 con 0x7efe64159b20 2023-10-14T16:57:49.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.511+0000 7efe69c76640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe64159b20 0x7efe6406bbe0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7efe6000bb10 tx=0x7efe6002e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d6ab0e95994779 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.511+0000 7efe68c74640 1 -- 172.21.15.29:0/3943498553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe6002ee70 con 0x7efe64159b20 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.511+0000 7efe68c74640 1 -- 172.21.15.29:0/3943498553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60036440 con 0x7efe64159b20 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.511+0000 7efe68c74640 1 -- 172.21.15.29:0/3943498553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600367d0 con 0x7efe64159b20 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 -- 172.21.15.29:0/3943498553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe64159b20 msgr2=0x7efe6406bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe64159b20 0x7efe6406bbe0 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 crypto rx=0x7efe6000bb10 tx=0x7efe6002e0c0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 -- 172.21.15.29:0/3943498553 shutdown_connections 2023-10-14T16:57:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159f00 0x7efe640b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 --2- 172.21.15.29:0/3943498553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe64159b20 0x7efe6406bbe0 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 -- 172.21.15.29:0/3943498553 >> 172.21.15.29:0/3943498553 conn(0x7efe640cb410 msgr2=0x7efe640cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 -- 172.21.15.29:0/3943498553 shutdown_connections 2023-10-14T16:57:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.512+0000 7efe6beda640 1 -- 172.21.15.29:0/3943498553 wait complete. 2023-10-14T16:57:49.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.514+0000 7efe6beda640 1 Processor -- start 2023-10-14T16:57:49.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.514+0000 7efe6beda640 1 -- start start 2023-10-14T16:57:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159b20 0x7efe640c7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe64159f00 0x7efe640c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6415b400 con 0x7efe6406c120 2023-10-14T16:57:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe6415b6a0 con 0x7efe64159f00 2023-10-14T16:57:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6beda640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe6415b980 con 0x7efe64159b20 2023-10-14T16:57:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe69c76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe69c76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58816/0 (socket says 172.21.15.29:58816) 2023-10-14T16:57:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe69c76640 1 -- 172.21.15.29:0/818475787 learned_addr learned my addr 172.21.15.29:0/818475787 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe69475640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159b20 0x7efe640c7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe69c76640 1 -- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159b20 msgr2=0x7efe640c7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe69c76640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159b20 0x7efe640c7460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.515+0000 7efe6a477640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe64159f00 0x7efe640c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe69c76640 1 -- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe64159f00 msgr2=0x7efe640c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe69c76640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe64159f00 0x7efe640c79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe69c76640 1 -- 172.21.15.29:0/818475787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe6000b770 con 0x7efe6406c120 2023-10-14T16:57:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe69c76640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7efe6000b2c0 tx=0x7efe600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe600348d0 con 0x7efe6406c120 2023-10-14T16:57:49.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe60034a50 con 0x7efe6406c120 2023-10-14T16:57:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe60034e00 con 0x7efe6406c120 2023-10-14T16:57:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe6415bc60 con 0x7efe6406c120 2023-10-14T16:57:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.516+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe64158e10 con 0x7efe6406c120 2023-10-14T16:57:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.517+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe6415c180 con 0x7efe6406c120 2023-10-14T16:57:49.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.518+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe600403e0 con 0x7efe6406c120 2023-10-14T16:57:49.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.519+0000 7efe5a7fc640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 0x7efe4806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:49.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.519+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7efe600abb40 con 0x7efe6406c120 2023-10-14T16:57:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.519+0000 7efe69475640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 0x7efe4806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.520+0000 7efe69475640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 0x7efe4806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe54006fd0 tx=0x7efe54008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.524+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe6415c180 con 0x7efe6406c120 2023-10-14T16:57:49.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.717+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 --> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efe6406ce80 con 0x7efe4806af90 2023-10-14T16:57:49.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.742+0000 7efe5a7fc640 1 -- 172.21.15.29:0/818475787 <== mgr.7257 v2:172.21.15.29:6832/1520757887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287303 (secure 0 0 0) 0x7efe5400f830 con 0x7efe4806af90 2023-10-14T16:57:49.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:49.749 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:57:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 msgr2=0x7efe4806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 0x7efe4806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7efe54006fd0 tx=0x7efe54008040 comp rx=0 tx=0).stop 2023-10-14T16:57:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 msgr2=0x7efe641588d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 crypto rx=0x7efe6000b2c0 tx=0x7efe600046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 shutdown_connections 2023-10-14T16:57:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe4806af90 0x7efe4806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe64159f00 0x7efe640c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe64159b20 0x7efe640c7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 --2- 172.21.15.29:0/818475787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe6406c120 0x7efe641588d0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:49.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 >> 172.21.15.29:0/818475787 conn(0x7efe640cb410 msgr2=0x7efe640cddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:49.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 shutdown_connections 2023-10-14T16:57:49.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:49.748+0000 7efe6beda640 1 -- 172.21.15.29:0/818475787 wait complete. 2023-10-14T16:57:49.770 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-14T16:57:49.194510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347172,"kb_used_data":11688,"kb_used_omap":12,"kb_used_meta":335347,"kb_avail":754627548,"statfs":{"total":773094113280,"available":772738609152,"internally_reserved":0,"allocated":11968512,"data_stored":6336164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":343395896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.954481"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187319+0000","last_change":"2023-10-14T16:57:45.193978+0000","last_active":"2023-10-14T16:57:46.187319+0000","last_peered":"2023-10-14T16:57:46.187319+0000","last_clean":"2023-10-14T16:57:46.187319+0000","last_became_active":"2023-10-14T16:57:45.193699+0000","last_became_peered":"2023-10-14T16:57:45.193699+0000","last_unstale":"2023-10-14T16:57:46.187319+0000","last_undegraded":"2023-10-14T16:57:46.187319+0000","last_fullsized":"2023-10-14T16:57:46.187319+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:43.079137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507674+0000","last_change":"2023-10-14T16:57:45.196412+0000","last_active":"2023-10-14T16:57:46.507674+0000","last_peered":"2023-10-14T16:57:46.507674+0000","last_clean":"2023-10-14T16:57:46.507674+0000","last_became_active":"2023-10-14T16:57:45.195455+0000","last_became_peered":"2023-10-14T16:57:45.195455+0000","last_unstale":"2023-10-14T16:57:46.507674+0000","last_undegraded":"2023-10-14T16:57:46.507674+0000","last_fullsized":"2023-10-14T16:57:46.507674+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:35.836488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153245+0000","last_change":"2023-10-14T16:57:45.199410+0000","last_active":"2023-10-14T16:57:47.153245+0000","last_peered":"2023-10-14T16:57:47.153245+0000","last_clean":"2023-10-14T16:57:47.153245+0000","last_became_active":"2023-10-14T16:57:45.197998+0000","last_became_peered":"2023-10-14T16:57:45.197998+0000","last_unstale":"2023-10-14T16:57:47.153245+0000","last_undegraded":"2023-10-14T16:57:47.153245+0000","last_fullsized":"2023-10-14T16:57:47.153245+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:08.513940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151853+0000","last_change":"2023-10-14T16:57:45.209311+0000","last_active":"2023-10-14T16:57:47.151853+0000","last_peered":"2023-10-14T16:57:47.151853+0000","last_clean":"2023-10-14T16:57:47.151853+0000","last_became_active":"2023-10-14T16:57:45.208004+0000","last_became_peered":"2023-10-14T16:57:45.208004+0000","last_unstale":"2023-10-14T16:57:47.151853+0000","last_undegraded":"2023-10-14T16:57:47.151853+0000","last_fullsized":"2023-10-14T16:57:47.151853+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:06.647893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185600+0000","last_change":"2023-10-14T16:57:45.207850+0000","last_active":"2023-10-14T16:57:46.185600+0000","last_peered":"2023-10-14T16:57:46.185600+0000","last_clean":"2023-10-14T16:57:46.185600+0000","last_became_active":"2023-10-14T16:57:45.207572+0000","last_became_peered":"2023-10-14T16:57:45.207572+0000","last_unstale":"2023-10-14T16:57:46.185600+0000","last_undegraded":"2023-10-14T16:57:46.185600+0000","last_fullsized":"2023-10-14T16:57:46.185600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:56.351969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151297+0000","last_change":"2023-10-14T16:57:45.192374+0000","last_active":"2023-10-14T16:57:47.151297+0000","last_peered":"2023-10-14T16:57:47.151297+0000","last_clean":"2023-10-14T16:57:47.151297+0000","last_became_active":"2023-10-14T16:57:45.192014+0000","last_became_peered":"2023-10-14T16:57:45.192014+0000","last_unstale":"2023-10-14T16:57:47.151297+0000","last_undegraded":"2023-10-14T16:57:47.151297+0000","last_fullsized":"2023-10-14T16:57:47.151297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:28.157778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152042+0000","last_change":"2023-10-14T16:57:45.194958+0000","last_active":"2023-10-14T16:57:47.152042+0000","last_peered":"2023-10-14T16:57:47.152042+0000","last_clean":"2023-10-14T16:57:47.152042+0000","last_became_active":"2023-10-14T16:57:45.194555+0000","last_became_peered":"2023-10-14T16:57:45.194555+0000","last_unstale":"2023-10-14T16:57:47.152042+0000","last_undegraded":"2023-10-14T16:57:47.152042+0000","last_fullsized":"2023-10-14T16:57:47.152042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:47.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151298+0000","last_change":"2023-10-14T16:57:45.193775+0000","last_active":"2023-10-14T16:57:47.151298+0000","last_peered":"2023-10-14T16:57:47.151298+0000","last_clean":"2023-10-14T16:57:47.151298+0000","last_became_active":"2023-10-14T16:57:45.193494+0000","last_became_peered":"2023-10-14T16:57:45.193494+0000","last_unstale":"2023-10-14T16:57:47.151298+0000","last_undegraded":"2023-10-14T16:57:47.151298+0000","last_fullsized":"2023-10-14T16:57:47.151298+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:24.980138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152022+0000","last_change":"2023-10-14T16:57:45.198782+0000","last_active":"2023-10-14T16:57:47.152022+0000","last_peered":"2023-10-14T16:57:47.152022+0000","last_clean":"2023-10-14T16:57:47.152022+0000","last_became_active":"2023-10-14T16:57:45.198474+0000","last_became_peered":"2023-10-14T16:57:45.198474+0000","last_unstale":"2023-10-14T16:57:47.152022+0000","last_undegraded":"2023-10-14T16:57:47.152022+0000","last_fullsized":"2023-10-14T16:57:47.152022+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:38:53.010356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187419+0000","last_change":"2023-10-14T16:57:45.200504+0000","last_active":"2023-10-14T16:57:46.187419+0000","last_peered":"2023-10-14T16:57:46.187419+0000","last_clean":"2023-10-14T16:57:46.187419+0000","last_became_active":"2023-10-14T16:57:45.199305+0000","last_became_peered":"2023-10-14T16:57:45.199305+0000","last_unstale":"2023-10-14T16:57:46.187419+0000","last_undegraded":"2023-10-14T16:57:46.187419+0000","last_fullsized":"2023-10-14T16:57:46.187419+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:34:45.843694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507418+0000","last_change":"2023-10-14T16:57:45.197104+0000","last_active":"2023-10-14T16:57:46.507418+0000","last_peered":"2023-10-14T16:57:46.507418+0000","last_clean":"2023-10-14T16:57:46.507418+0000","last_became_active":"2023-10-14T16:57:45.196743+0000","last_became_peered":"2023-10-14T16:57:45.196743+0000","last_unstale":"2023-10-14T16:57:46.507418+0000","last_undegraded":"2023-10-14T16:57:46.507418+0000","last_fullsized":"2023-10-14T16:57:46.507418+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:54.024111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186622+0000","last_change":"2023-10-14T16:57:45.197933+0000","last_active":"2023-10-14T16:57:46.186622+0000","last_peered":"2023-10-14T16:57:46.186622+0000","last_clean":"2023-10-14T16:57:46.186622+0000","last_became_active":"2023-10-14T16:57:45.195872+0000","last_became_peered":"2023-10-14T16:57:45.195872+0000","last_unstale":"2023-10-14T16:57:46.186622+0000","last_undegraded":"2023-10-14T16:57:46.186622+0000","last_fullsized":"2023-10-14T16:57:46.186622+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:49.720572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152644+0000","last_change":"2023-10-14T16:57:45.200442+0000","last_active":"2023-10-14T16:57:47.152644+0000","last_peered":"2023-10-14T16:57:47.152644+0000","last_clean":"2023-10-14T16:57:47.152644+0000","last_became_active":"2023-10-14T16:57:45.200116+0000","last_became_peered":"2023-10-14T16:57:45.200116+0000","last_unstale":"2023-10-14T16:57:47.152644+0000","last_undegraded":"2023-10-14T16:57:47.152644+0000","last_fullsized":"2023-10-14T16:57:47.152644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:31.865642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151692+0000","last_change":"2023-10-14T16:57:45.201437+0000","last_active":"2023-10-14T16:57:47.151692+0000","last_peered":"2023-10-14T16:57:47.151692+0000","last_clean":"2023-10-14T16:57:47.151692+0000","last_became_active":"2023-10-14T16:57:45.200719+0000","last_became_peered":"2023-10-14T16:57:45.200719+0000","last_unstale":"2023-10-14T16:57:47.151692+0000","last_undegraded":"2023-10-14T16:57:47.151692+0000","last_fullsized":"2023-10-14T16:57:47.151692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:33.688264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151695+0000","last_change":"2023-10-14T16:57:45.200286+0000","last_active":"2023-10-14T16:57:47.151695+0000","last_peered":"2023-10-14T16:57:47.151695+0000","last_clean":"2023-10-14T16:57:47.151695+0000","last_became_active":"2023-10-14T16:57:45.199979+0000","last_became_peered":"2023-10-14T16:57:45.199979+0000","last_unstale":"2023-10-14T16:57:47.151695+0000","last_undegraded":"2023-10-14T16:57:47.151695+0000","last_fullsized":"2023-10-14T16:57:47.151695+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:31:28.269227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185561+0000","last_change":"2023-10-14T16:57:45.189274+0000","last_active":"2023-10-14T16:57:46.185561+0000","last_peered":"2023-10-14T16:57:46.185561+0000","last_clean":"2023-10-14T16:57:46.185561+0000","last_became_active":"2023-10-14T16:57:45.188909+0000","last_became_peered":"2023-10-14T16:57:45.188909+0000","last_unstale":"2023-10-14T16:57:46.185561+0000","last_undegraded":"2023-10-14T16:57:46.185561+0000","last_fullsized":"2023-10-14T16:57:46.185561+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:37.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188619+0000","last_change":"2023-10-14T16:57:45.194206+0000","last_active":"2023-10-14T16:57:46.188619+0000","last_peered":"2023-10-14T16:57:46.188619+0000","last_clean":"2023-10-14T16:57:46.188619+0000","last_became_active":"2023-10-14T16:57:45.193881+0000","last_became_peered":"2023-10-14T16:57:45.193881+0000","last_unstale":"2023-10-14T16:57:46.188619+0000","last_undegraded":"2023-10-14T16:57:46.188619+0000","last_fullsized":"2023-10-14T16:57:46.188619+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:50.439386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152430+0000","last_change":"2023-10-14T16:57:45.199533+0000","last_active":"2023-10-14T16:57:47.152430+0000","last_peered":"2023-10-14T16:57:47.152430+0000","last_clean":"2023-10-14T16:57:47.152430+0000","last_became_active":"2023-10-14T16:57:45.198157+0000","last_became_peered":"2023-10-14T16:57:45.198157+0000","last_unstale":"2023-10-14T16:57:47.152430+0000","last_undegraded":"2023-10-14T16:57:47.152430+0000","last_fullsized":"2023-10-14T16:57:47.152430+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:01.870891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183381+0000","last_change":"2023-10-14T16:57:45.204967+0000","last_active":"2023-10-14T16:57:46.183381+0000","last_peered":"2023-10-14T16:57:46.183381+0000","last_clean":"2023-10-14T16:57:46.183381+0000","last_became_active":"2023-10-14T16:57:45.204632+0000","last_became_peered":"2023-10-14T16:57:45.204632+0000","last_unstale":"2023-10-14T16:57:46.183381+0000","last_undegraded":"2023-10-14T16:57:46.183381+0000","last_fullsized":"2023-10-14T16:57:46.183381+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:46.620137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153053+0000","last_change":"2023-10-14T16:57:45.202681+0000","last_active":"2023-10-14T16:57:47.153053+0000","last_peered":"2023-10-14T16:57:47.153053+0000","last_clean":"2023-10-14T16:57:47.153053+0000","last_became_active":"2023-10-14T16:57:45.202412+0000","last_became_peered":"2023-10-14T16:57:45.202412+0000","last_unstale":"2023-10-14T16:57:47.153053+0000","last_undegraded":"2023-10-14T16:57:47.153053+0000","last_fullsized":"2023-10-14T16:57:47.153053+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:46.620137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151625+0000","last_change":"2023-10-14T16:57:45.211973+0000","last_active":"2023-10-14T16:57:47.151625+0000","last_peered":"2023-10-14T16:57:47.151625+0000","last_clean":"2023-10-14T16:57:47.151625+0000","last_became_active":"2023-10-14T16:57:45.211116+0000","last_became_peered":"2023-10-14T16:57:45.211116+0000","last_unstale":"2023-10-14T16:57:47.151625+0000","last_undegraded":"2023-10-14T16:57:47.151625+0000","last_fullsized":"2023-10-14T16:57:47.151625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:03.931640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507542+0000","last_change":"2023-10-14T16:57:45.208939+0000","last_active":"2023-10-14T16:57:46.507542+0000","last_peered":"2023-10-14T16:57:46.507542+0000","last_clean":"2023-10-14T16:57:46.507542+0000","last_became_active":"2023-10-14T16:57:45.208040+0000","last_became_peered":"2023-10-14T16:57:45.208040+0000","last_unstale":"2023-10-14T16:57:46.507542+0000","last_undegraded":"2023-10-14T16:57:46.507542+0000","last_fullsized":"2023-10-14T16:57:46.507542+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:42:24.525894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186117+0000","last_change":"2023-10-14T16:57:45.198225+0000","last_active":"2023-10-14T16:57:46.186117+0000","last_peered":"2023-10-14T16:57:46.186117+0000","last_clean":"2023-10-14T16:57:46.186117+0000","last_became_active":"2023-10-14T16:57:45.197948+0000","last_became_peered":"2023-10-14T16:57:45.197948+0000","last_unstale":"2023-10-14T16:57:46.186117+0000","last_undegraded":"2023-10-14T16:57:46.186117+0000","last_fullsized":"2023-10-14T16:57:46.186117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:07:03.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151171+0000","last_change":"2023-10-14T16:57:45.205417+0000","last_active":"2023-10-14T16:57:47.151171+0000","last_peered":"2023-10-14T16:57:47.151171+0000","last_clean":"2023-10-14T16:57:47.151171+0000","last_became_active":"2023-10-14T16:57:45.203613+0000","last_became_peered":"2023-10-14T16:57:45.203613+0000","last_unstale":"2023-10-14T16:57:47.151171+0000","last_undegraded":"2023-10-14T16:57:47.151171+0000","last_fullsized":"2023-10-14T16:57:47.151171+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:54.056332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507934+0000","last_change":"2023-10-14T16:57:45.204730+0000","last_active":"2023-10-14T16:57:46.507934+0000","last_peered":"2023-10-14T16:57:46.507934+0000","last_clean":"2023-10-14T16:57:46.507934+0000","last_became_active":"2023-10-14T16:57:45.204442+0000","last_became_peered":"2023-10-14T16:57:45.204442+0000","last_unstale":"2023-10-14T16:57:46.507934+0000","last_undegraded":"2023-10-14T16:57:46.507934+0000","last_fullsized":"2023-10-14T16:57:46.507934+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:02:09.065704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152935+0000","last_change":"2023-10-14T16:57:45.210079+0000","last_active":"2023-10-14T16:57:47.152935+0000","last_peered":"2023-10-14T16:57:47.152935+0000","last_clean":"2023-10-14T16:57:47.152935+0000","last_became_active":"2023-10-14T16:57:45.209737+0000","last_became_peered":"2023-10-14T16:57:45.209737+0000","last_unstale":"2023-10-14T16:57:47.152935+0000","last_undegraded":"2023-10-14T16:57:47.152935+0000","last_fullsized":"2023-10-14T16:57:47.152935+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:03.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187443+0000","last_change":"2023-10-14T16:57:45.195537+0000","last_active":"2023-10-14T16:57:46.187443+0000","last_peered":"2023-10-14T16:57:46.187443+0000","last_clean":"2023-10-14T16:57:46.187443+0000","last_became_active":"2023-10-14T16:57:45.195246+0000","last_became_peered":"2023-10-14T16:57:45.195246+0000","last_unstale":"2023-10-14T16:57:46.187443+0000","last_undegraded":"2023-10-14T16:57:46.187443+0000","last_fullsized":"2023-10-14T16:57:46.187443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:48.462945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186565+0000","last_change":"2023-10-14T16:57:45.194725+0000","last_active":"2023-10-14T16:57:46.186565+0000","last_peered":"2023-10-14T16:57:46.186565+0000","last_clean":"2023-10-14T16:57:46.186565+0000","last_became_active":"2023-10-14T16:57:45.194420+0000","last_became_peered":"2023-10-14T16:57:45.194420+0000","last_unstale":"2023-10-14T16:57:46.186565+0000","last_undegraded":"2023-10-14T16:57:46.186565+0000","last_fullsized":"2023-10-14T16:57:46.186565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:11.835936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507453+0000","last_change":"2023-10-14T16:57:45.197647+0000","last_active":"2023-10-14T16:57:46.507453+0000","last_peered":"2023-10-14T16:57:46.507453+0000","last_clean":"2023-10-14T16:57:46.507453+0000","last_became_active":"2023-10-14T16:57:45.197233+0000","last_became_peered":"2023-10-14T16:57:45.197233+0000","last_unstale":"2023-10-14T16:57:46.507453+0000","last_undegraded":"2023-10-14T16:57:46.507453+0000","last_fullsized":"2023-10-14T16:57:46.507453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:39.243523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507435+0000","last_change":"2023-10-14T16:57:45.195600+0000","last_active":"2023-10-14T16:57:46.507435+0000","last_peered":"2023-10-14T16:57:46.507435+0000","last_clean":"2023-10-14T16:57:46.507435+0000","last_became_active":"2023-10-14T16:57:45.195206+0000","last_became_peered":"2023-10-14T16:57:45.195206+0000","last_unstale":"2023-10-14T16:57:46.507435+0000","last_undegraded":"2023-10-14T16:57:46.507435+0000","last_fullsized":"2023-10-14T16:57:46.507435+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:35:46.266329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185564+0000","last_change":"2023-10-14T16:57:45.206622+0000","last_active":"2023-10-14T16:57:46.185564+0000","last_peered":"2023-10-14T16:57:46.185564+0000","last_clean":"2023-10-14T16:57:46.185564+0000","last_became_active":"2023-10-14T16:57:45.206326+0000","last_became_peered":"2023-10-14T16:57:45.206326+0000","last_unstale":"2023-10-14T16:57:46.185564+0000","last_undegraded":"2023-10-14T16:57:46.185564+0000","last_fullsized":"2023-10-14T16:57:46.185564+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:38.676561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185671+0000","last_change":"2023-10-14T16:57:45.207794+0000","last_active":"2023-10-14T16:57:46.185671+0000","last_peered":"2023-10-14T16:57:46.185671+0000","last_clean":"2023-10-14T16:57:46.185671+0000","last_became_active":"2023-10-14T16:57:45.207469+0000","last_became_peered":"2023-10-14T16:57:45.207469+0000","last_unstale":"2023-10-14T16:57:46.185671+0000","last_undegraded":"2023-10-14T16:57:46.185671+0000","last_fullsized":"2023-10-14T16:57:46.185671+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:03.671640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187738+0000","last_change":"2023-10-14T16:57:45.191306+0000","last_active":"2023-10-14T16:57:46.187738+0000","last_peered":"2023-10-14T16:57:46.187738+0000","last_clean":"2023-10-14T16:57:46.187738+0000","last_became_active":"2023-10-14T16:57:45.190865+0000","last_became_peered":"2023-10-14T16:57:45.190865+0000","last_unstale":"2023-10-14T16:57:46.187738+0000","last_undegraded":"2023-10-14T16:57:46.187738+0000","last_fullsized":"2023-10-14T16:57:46.187738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:51:51.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"108'4","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187695+0000","last_change":"2023-10-14T16:57:44.277173+0000","last_active":"2023-10-14T16:57:46.187695+0000","last_peered":"2023-10-14T16:57:46.187695+0000","last_clean":"2023-10-14T16:57:46.187695+0000","last_became_active":"2023-10-14T16:57:44.276881+0000","last_became_peered":"2023-10-14T16:57:44.276881+0000","last_unstale":"2023-10-14T16:57:46.187695+0000","last_undegraded":"2023-10-14T16:57:46.187695+0000","last_fullsized":"2023-10-14T16:57:46.187695+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:46.417549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187435+0000","last_change":"2023-10-14T16:57:45.205762+0000","last_active":"2023-10-14T16:57:46.187435+0000","last_peered":"2023-10-14T16:57:46.187435+0000","last_clean":"2023-10-14T16:57:46.187435+0000","last_became_active":"2023-10-14T16:57:45.205447+0000","last_became_peered":"2023-10-14T16:57:45.205447+0000","last_unstale":"2023-10-14T16:57:46.187435+0000","last_undegraded":"2023-10-14T16:57:46.187435+0000","last_fullsized":"2023-10-14T16:57:46.187435+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:35:46.266329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188176+0000","last_change":"2023-10-14T16:57:44.284698+0000","last_active":"2023-10-14T16:57:46.188176+0000","last_peered":"2023-10-14T16:57:46.188176+0000","last_clean":"2023-10-14T16:57:46.188176+0000","last_became_active":"2023-10-14T16:57:44.284404+0000","last_became_peered":"2023-10-14T16:57:44.284404+0000","last_unstale":"2023-10-14T16:57:46.188176+0000","last_undegraded":"2023-10-14T16:57:46.188176+0000","last_fullsized":"2023-10-14T16:57:46.188176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:11.254305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186921+0000","last_change":"2023-10-14T16:57:45.207849+0000","last_active":"2023-10-14T16:57:46.186921+0000","last_peered":"2023-10-14T16:57:46.186921+0000","last_clean":"2023-10-14T16:57:46.186921+0000","last_became_active":"2023-10-14T16:57:45.207562+0000","last_became_peered":"2023-10-14T16:57:45.207562+0000","last_unstale":"2023-10-14T16:57:46.186921+0000","last_undegraded":"2023-10-14T16:57:46.186921+0000","last_fullsized":"2023-10-14T16:57:46.186921+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:37.196176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183508+0000","last_change":"2023-10-14T16:57:44.275219+0000","last_active":"2023-10-14T16:57:46.183508+0000","last_peered":"2023-10-14T16:57:46.183508+0000","last_clean":"2023-10-14T16:57:46.183508+0000","last_became_active":"2023-10-14T16:57:44.274959+0000","last_became_peered":"2023-10-14T16:57:44.274959+0000","last_unstale":"2023-10-14T16:57:46.183508+0000","last_undegraded":"2023-10-14T16:57:46.183508+0000","last_fullsized":"2023-10-14T16:57:46.183508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:58.217093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153532+0000","last_change":"2023-10-14T16:57:45.203212+0000","last_active":"2023-10-14T16:57:47.153532+0000","last_peered":"2023-10-14T16:57:47.153532+0000","last_clean":"2023-10-14T16:57:47.153532+0000","last_became_active":"2023-10-14T16:57:45.202722+0000","last_became_peered":"2023-10-14T16:57:45.202722+0000","last_unstale":"2023-10-14T16:57:47.153532+0000","last_undegraded":"2023-10-14T16:57:47.153532+0000","last_fullsized":"2023-10-14T16:57:47.153532+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:39.631477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182321+0000","last_change":"2023-10-14T16:57:44.276612+0000","last_active":"2023-10-14T16:57:46.182321+0000","last_peered":"2023-10-14T16:57:46.182321+0000","last_clean":"2023-10-14T16:57:46.182321+0000","last_became_active":"2023-10-14T16:57:44.275530+0000","last_became_peered":"2023-10-14T16:57:44.275530+0000","last_unstale":"2023-10-14T16:57:46.182321+0000","last_undegraded":"2023-10-14T16:57:46.182321+0000","last_fullsized":"2023-10-14T16:57:46.182321+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:25.916883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507755+0000","last_change":"2023-10-14T16:57:45.195146+0000","last_active":"2023-10-14T16:57:46.507755+0000","last_peered":"2023-10-14T16:57:46.507755+0000","last_clean":"2023-10-14T16:57:46.507755+0000","last_became_active":"2023-10-14T16:57:45.194803+0000","last_became_peered":"2023-10-14T16:57:45.194803+0000","last_unstale":"2023-10-14T16:57:46.507755+0000","last_undegraded":"2023-10-14T16:57:46.507755+0000","last_fullsized":"2023-10-14T16:57:46.507755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:46:33.521273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187662+0000","last_change":"2023-10-14T16:57:44.275851+0000","last_active":"2023-10-14T16:57:46.187662+0000","last_peered":"2023-10-14T16:57:46.187662+0000","last_clean":"2023-10-14T16:57:46.187662+0000","last_became_active":"2023-10-14T16:57:44.275509+0000","last_became_peered":"2023-10-14T16:57:44.275509+0000","last_unstale":"2023-10-14T16:57:46.187662+0000","last_undegraded":"2023-10-14T16:57:46.187662+0000","last_fullsized":"2023-10-14T16:57:46.187662+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:15.938015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187207+0000","last_change":"2023-10-14T16:57:45.194964+0000","last_active":"2023-10-14T16:57:46.187207+0000","last_peered":"2023-10-14T16:57:46.187207+0000","last_clean":"2023-10-14T16:57:46.187207+0000","last_became_active":"2023-10-14T16:57:45.194000+0000","last_became_peered":"2023-10-14T16:57:45.194000+0000","last_unstale":"2023-10-14T16:57:46.187207+0000","last_undegraded":"2023-10-14T16:57:46.187207+0000","last_fullsized":"2023-10-14T16:57:46.187207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:29:39.471553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186071+0000","last_change":"2023-10-14T16:57:44.284835+0000","last_active":"2023-10-14T16:57:46.186071+0000","last_peered":"2023-10-14T16:57:46.186071+0000","last_clean":"2023-10-14T16:57:46.186071+0000","last_became_active":"2023-10-14T16:57:44.284357+0000","last_became_peered":"2023-10-14T16:57:44.284357+0000","last_unstale":"2023-10-14T16:57:46.186071+0000","last_undegraded":"2023-10-14T16:57:46.186071+0000","last_fullsized":"2023-10-14T16:57:46.186071+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:48.844159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152863+0000","last_change":"2023-10-14T16:57:45.206648+0000","last_active":"2023-10-14T16:57:47.152863+0000","last_peered":"2023-10-14T16:57:47.152863+0000","last_clean":"2023-10-14T16:57:47.152863+0000","last_became_active":"2023-10-14T16:57:45.206231+0000","last_became_peered":"2023-10-14T16:57:45.206231+0000","last_unstale":"2023-10-14T16:57:47.152863+0000","last_undegraded":"2023-10-14T16:57:47.152863+0000","last_fullsized":"2023-10-14T16:57:47.152863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:08.163969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.181916+0000","last_change":"2023-10-14T16:57:44.268567+0000","last_active":"2023-10-14T16:57:46.181916+0000","last_peered":"2023-10-14T16:57:46.181916+0000","last_clean":"2023-10-14T16:57:46.181916+0000","last_became_active":"2023-10-14T16:57:44.268199+0000","last_became_peered":"2023-10-14T16:57:44.268199+0000","last_unstale":"2023-10-14T16:57:46.181916+0000","last_undegraded":"2023-10-14T16:57:46.181916+0000","last_fullsized":"2023-10-14T16:57:46.181916+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:35.271857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153484+0000","last_change":"2023-10-14T16:57:45.198466+0000","last_active":"2023-10-14T16:57:47.153484+0000","last_peered":"2023-10-14T16:57:47.153484+0000","last_clean":"2023-10-14T16:57:47.153484+0000","last_became_active":"2023-10-14T16:57:45.198174+0000","last_became_peered":"2023-10-14T16:57:45.198174+0000","last_unstale":"2023-10-14T16:57:47.153484+0000","last_undegraded":"2023-10-14T16:57:47.153484+0000","last_fullsized":"2023-10-14T16:57:47.153484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:36:09.824679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506961+0000","last_change":"2023-10-14T16:57:44.271763+0000","last_active":"2023-10-14T16:57:46.506961+0000","last_peered":"2023-10-14T16:57:46.506961+0000","last_clean":"2023-10-14T16:57:46.506961+0000","last_became_active":"2023-10-14T16:57:44.270409+0000","last_became_peered":"2023-10-14T16:57:44.270409+0000","last_unstale":"2023-10-14T16:57:46.506961+0000","last_undegraded":"2023-10-14T16:57:46.506961+0000","last_fullsized":"2023-10-14T16:57:46.506961+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:07:02.926038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152755+0000","last_change":"2023-10-14T16:57:45.211071+0000","last_active":"2023-10-14T16:57:47.152755+0000","last_peered":"2023-10-14T16:57:47.152755+0000","last_clean":"2023-10-14T16:57:47.152755+0000","last_became_active":"2023-10-14T16:57:45.210725+0000","last_became_peered":"2023-10-14T16:57:45.210725+0000","last_unstale":"2023-10-14T16:57:47.152755+0000","last_undegraded":"2023-10-14T16:57:47.152755+0000","last_fullsized":"2023-10-14T16:57:47.152755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:08.513940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187941+0000","last_change":"2023-10-14T16:57:44.284552+0000","last_active":"2023-10-14T16:57:46.187941+0000","last_peered":"2023-10-14T16:57:46.187941+0000","last_clean":"2023-10-14T16:57:46.187941+0000","last_became_active":"2023-10-14T16:57:44.284161+0000","last_became_peered":"2023-10-14T16:57:44.284161+0000","last_unstale":"2023-10-14T16:57:46.187941+0000","last_undegraded":"2023-10-14T16:57:46.187941+0000","last_fullsized":"2023-10-14T16:57:46.187941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:00.857230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152378+0000","last_change":"2023-10-14T16:57:45.196777+0000","last_active":"2023-10-14T16:57:47.152378+0000","last_peered":"2023-10-14T16:57:47.152378+0000","last_clean":"2023-10-14T16:57:47.152378+0000","last_became_active":"2023-10-14T16:57:45.196456+0000","last_became_peered":"2023-10-14T16:57:45.196456+0000","last_unstale":"2023-10-14T16:57:47.152378+0000","last_undegraded":"2023-10-14T16:57:47.152378+0000","last_fullsized":"2023-10-14T16:57:47.152378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:38:50.125580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187538+0000","last_change":"2023-10-14T16:57:44.274582+0000","last_active":"2023-10-14T16:57:46.187538+0000","last_peered":"2023-10-14T16:57:46.187538+0000","last_clean":"2023-10-14T16:57:46.187538+0000","last_became_active":"2023-10-14T16:57:44.274305+0000","last_became_peered":"2023-10-14T16:57:44.274305+0000","last_unstale":"2023-10-14T16:57:46.187538+0000","last_undegraded":"2023-10-14T16:57:46.187538+0000","last_fullsized":"2023-10-14T16:57:46.187538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:25.916883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182233+0000","last_change":"2023-10-14T16:57:45.204148+0000","last_active":"2023-10-14T16:57:46.182233+0000","last_peered":"2023-10-14T16:57:46.182233+0000","last_clean":"2023-10-14T16:57:46.182233+0000","last_became_active":"2023-10-14T16:57:45.199977+0000","last_became_peered":"2023-10-14T16:57:45.199977+0000","last_unstale":"2023-10-14T16:57:46.182233+0000","last_undegraded":"2023-10-14T16:57:46.182233+0000","last_fullsized":"2023-10-14T16:57:46.182233+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:06.647893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"108'3","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186643+0000","last_change":"2023-10-14T16:57:44.272963+0000","last_active":"2023-10-14T16:57:46.186643+0000","last_peered":"2023-10-14T16:57:46.186643+0000","last_clean":"2023-10-14T16:57:46.186643+0000","last_became_active":"2023-10-14T16:57:44.272665+0000","last_became_peered":"2023-10-14T16:57:44.272665+0000","last_unstale":"2023-10-14T16:57:46.186643+0000","last_undegraded":"2023-10-14T16:57:46.186643+0000","last_fullsized":"2023-10-14T16:57:46.186643+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:32.674604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187770+0000","last_change":"2023-10-14T16:57:45.211198+0000","last_active":"2023-10-14T16:57:46.187770+0000","last_peered":"2023-10-14T16:57:46.187770+0000","last_clean":"2023-10-14T16:57:46.187770+0000","last_became_active":"2023-10-14T16:57:45.210881+0000","last_became_peered":"2023-10-14T16:57:45.210881+0000","last_unstale":"2023-10-14T16:57:46.187770+0000","last_undegraded":"2023-10-14T16:57:46.187770+0000","last_fullsized":"2023-10-14T16:57:46.187770+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:19.759740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"108'6","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151541+0000","last_change":"2023-10-14T16:57:44.282567+0000","last_active":"2023-10-14T16:57:47.151541+0000","last_peered":"2023-10-14T16:57:47.151541+0000","last_clean":"2023-10-14T16:57:47.151541+0000","last_became_active":"2023-10-14T16:57:44.282189+0000","last_became_peered":"2023-10-14T16:57:44.282189+0000","last_unstale":"2023-10-14T16:57:47.151541+0000","last_undegraded":"2023-10-14T16:57:47.151541+0000","last_fullsized":"2023-10-14T16:57:47.151541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:39.048046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182370+0000","last_change":"2023-10-14T16:57:45.204341+0000","last_active":"2023-10-14T16:57:46.182370+0000","last_peered":"2023-10-14T16:57:46.182370+0000","last_clean":"2023-10-14T16:57:46.182370+0000","last_became_active":"2023-10-14T16:57:45.201951+0000","last_became_peered":"2023-10-14T16:57:45.201951+0000","last_unstale":"2023-10-14T16:57:46.182370+0000","last_undegraded":"2023-10-14T16:57:46.182370+0000","last_fullsized":"2023-10-14T16:57:46.182370+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:47.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507347+0000","last_change":"2023-10-14T16:57:44.271855+0000","last_active":"2023-10-14T16:57:46.507347+0000","last_peered":"2023-10-14T16:57:46.507347+0000","last_clean":"2023-10-14T16:57:46.507347+0000","last_became_active":"2023-10-14T16:57:44.270451+0000","last_became_peered":"2023-10-14T16:57:44.270451+0000","last_unstale":"2023-10-14T16:57:46.507347+0000","last_undegraded":"2023-10-14T16:57:46.507347+0000","last_fullsized":"2023-10-14T16:57:46.507347+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:02.657979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152978+0000","last_change":"2023-10-14T16:57:45.209362+0000","last_active":"2023-10-14T16:57:47.152978+0000","last_peered":"2023-10-14T16:57:47.152978+0000","last_clean":"2023-10-14T16:57:47.152978+0000","last_became_active":"2023-10-14T16:57:45.209069+0000","last_became_peered":"2023-10-14T16:57:45.209069+0000","last_unstale":"2023-10-14T16:57:47.152978+0000","last_undegraded":"2023-10-14T16:57:47.152978+0000","last_fullsized":"2023-10-14T16:57:47.152978+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:31:28.269227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153283+0000","last_change":"2023-10-14T16:57:44.282653+0000","last_active":"2023-10-14T16:57:47.153283+0000","last_peered":"2023-10-14T16:57:47.153283+0000","last_clean":"2023-10-14T16:57:47.153283+0000","last_became_active":"2023-10-14T16:57:44.281976+0000","last_became_peered":"2023-10-14T16:57:44.281976+0000","last_unstale":"2023-10-14T16:57:47.153283+0000","last_undegraded":"2023-10-14T16:57:47.153283+0000","last_fullsized":"2023-10-14T16:57:47.153283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:30.851982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506748+0000","last_change":"2023-10-14T16:57:45.203150+0000","last_active":"2023-10-14T16:57:46.506748+0000","last_peered":"2023-10-14T16:57:46.506748+0000","last_clean":"2023-10-14T16:57:46.506748+0000","last_became_active":"2023-10-14T16:57:45.194972+0000","last_became_peered":"2023-10-14T16:57:45.194972+0000","last_unstale":"2023-10-14T16:57:46.506748+0000","last_undegraded":"2023-10-14T16:57:46.506748+0000","last_fullsized":"2023-10-14T16:57:46.506748+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:43.079137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151789+0000","last_change":"2023-10-14T16:57:44.271853+0000","last_active":"2023-10-14T16:57:47.151789+0000","last_peered":"2023-10-14T16:57:47.151789+0000","last_clean":"2023-10-14T16:57:47.151789+0000","last_became_active":"2023-10-14T16:57:44.271524+0000","last_became_peered":"2023-10-14T16:57:44.271524+0000","last_unstale":"2023-10-14T16:57:47.151789+0000","last_undegraded":"2023-10-14T16:57:47.151789+0000","last_fullsized":"2023-10-14T16:57:47.151789+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:18.746079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506665+0000","last_change":"2023-10-14T16:57:45.198791+0000","last_active":"2023-10-14T16:57:46.506665+0000","last_peered":"2023-10-14T16:57:46.506665+0000","last_clean":"2023-10-14T16:57:46.506665+0000","last_became_active":"2023-10-14T16:57:45.198287+0000","last_became_peered":"2023-10-14T16:57:45.198287+0000","last_unstale":"2023-10-14T16:57:46.506665+0000","last_undegraded":"2023-10-14T16:57:46.506665+0000","last_fullsized":"2023-10-14T16:57:46.506665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:04.729022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"108'8","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.181489+0000","last_change":"2023-10-14T16:57:44.282785+0000","last_active":"2023-10-14T16:57:46.181489+0000","last_peered":"2023-10-14T16:57:46.181489+0000","last_clean":"2023-10-14T16:57:46.181489+0000","last_became_active":"2023-10-14T16:57:44.282373+0000","last_became_peered":"2023-10-14T16:57:44.282373+0000","last_unstale":"2023-10-14T16:57:46.181489+0000","last_undegraded":"2023-10-14T16:57:46.181489+0000","last_fullsized":"2023-10-14T16:57:46.181489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:36.357674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153300+0000","last_change":"2023-10-14T16:57:45.197142+0000","last_active":"2023-10-14T16:57:47.153300+0000","last_peered":"2023-10-14T16:57:47.153300+0000","last_clean":"2023-10-14T16:57:47.153300+0000","last_became_active":"2023-10-14T16:57:45.196851+0000","last_became_peered":"2023-10-14T16:57:45.196851+0000","last_unstale":"2023-10-14T16:57:47.153300+0000","last_undegraded":"2023-10-14T16:57:47.153300+0000","last_fullsized":"2023-10-14T16:57:47.153300+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:17.465365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188199+0000","last_change":"2023-10-14T16:57:44.272494+0000","last_active":"2023-10-14T16:57:46.188199+0000","last_peered":"2023-10-14T16:57:46.188199+0000","last_clean":"2023-10-14T16:57:46.188199+0000","last_became_active":"2023-10-14T16:57:44.271104+0000","last_became_peered":"2023-10-14T16:57:44.271104+0000","last_unstale":"2023-10-14T16:57:46.188199+0000","last_undegraded":"2023-10-14T16:57:46.188199+0000","last_fullsized":"2023-10-14T16:57:46.188199+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:45.606476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187663+0000","last_change":"2023-10-14T16:57:45.196109+0000","last_active":"2023-10-14T16:57:46.187663+0000","last_peered":"2023-10-14T16:57:46.187663+0000","last_clean":"2023-10-14T16:57:46.187663+0000","last_became_active":"2023-10-14T16:57:45.195801+0000","last_became_peered":"2023-10-14T16:57:45.195801+0000","last_unstale":"2023-10-14T16:57:46.187663+0000","last_undegraded":"2023-10-14T16:57:46.187663+0000","last_fullsized":"2023-10-14T16:57:46.187663+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:54.024111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183426+0000","last_change":"2023-10-14T16:57:44.269395+0000","last_active":"2023-10-14T16:57:46.183426+0000","last_peered":"2023-10-14T16:57:46.183426+0000","last_clean":"2023-10-14T16:57:46.183426+0000","last_became_active":"2023-10-14T16:57:44.268913+0000","last_became_peered":"2023-10-14T16:57:44.268913+0000","last_unstale":"2023-10-14T16:57:46.183426+0000","last_undegraded":"2023-10-14T16:57:46.183426+0000","last_fullsized":"2023-10-14T16:57:46.183426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:53.042671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187938+0000","last_change":"2023-10-14T16:57:45.207875+0000","last_active":"2023-10-14T16:57:46.187938+0000","last_peered":"2023-10-14T16:57:46.187938+0000","last_clean":"2023-10-14T16:57:46.187938+0000","last_became_active":"2023-10-14T16:57:45.207505+0000","last_became_peered":"2023-10-14T16:57:45.207505+0000","last_unstale":"2023-10-14T16:57:46.187938+0000","last_undegraded":"2023-10-14T16:57:46.187938+0000","last_fullsized":"2023-10-14T16:57:46.187938+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:40.061706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182404+0000","last_change":"2023-10-14T16:57:44.273166+0000","last_active":"2023-10-14T16:57:46.182404+0000","last_peered":"2023-10-14T16:57:46.182404+0000","last_clean":"2023-10-14T16:57:46.182404+0000","last_became_active":"2023-10-14T16:57:44.272862+0000","last_became_peered":"2023-10-14T16:57:44.272862+0000","last_unstale":"2023-10-14T16:57:46.182404+0000","last_undegraded":"2023-10-14T16:57:46.182404+0000","last_fullsized":"2023-10-14T16:57:46.182404+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:02.917979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151662+0000","last_change":"2023-10-14T16:57:45.201634+0000","last_active":"2023-10-14T16:57:47.151662+0000","last_peered":"2023-10-14T16:57:47.151662+0000","last_clean":"2023-10-14T16:57:47.151662+0000","last_became_active":"2023-10-14T16:57:45.201093+0000","last_became_peered":"2023-10-14T16:57:45.201093+0000","last_unstale":"2023-10-14T16:57:47.151662+0000","last_undegraded":"2023-10-14T16:57:47.151662+0000","last_fullsized":"2023-10-14T16:57:47.151662+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:26.930544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152905+0000","last_change":"2023-10-14T16:57:44.282377+0000","last_active":"2023-10-14T16:57:47.152905+0000","last_peered":"2023-10-14T16:57:47.152905+0000","last_clean":"2023-10-14T16:57:47.152905+0000","last_became_active":"2023-10-14T16:57:44.282075+0000","last_became_peered":"2023-10-14T16:57:44.282075+0000","last_unstale":"2023-10-14T16:57:47.152905+0000","last_undegraded":"2023-10-14T16:57:47.152905+0000","last_fullsized":"2023-10-14T16:57:47.152905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:02.609380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151309+0000","last_change":"2023-10-14T16:57:45.198872+0000","last_active":"2023-10-14T16:57:47.151309+0000","last_peered":"2023-10-14T16:57:47.151309+0000","last_clean":"2023-10-14T16:57:47.151309+0000","last_became_active":"2023-10-14T16:57:45.198575+0000","last_became_peered":"2023-10-14T16:57:45.198575+0000","last_unstale":"2023-10-14T16:57:47.151309+0000","last_undegraded":"2023-10-14T16:57:47.151309+0000","last_fullsized":"2023-10-14T16:57:47.151309+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:52.642670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152353+0000","last_change":"2023-10-14T16:57:44.276898+0000","last_active":"2023-10-14T16:57:47.152353+0000","last_peered":"2023-10-14T16:57:47.152353+0000","last_clean":"2023-10-14T16:57:47.152353+0000","last_became_active":"2023-10-14T16:57:44.275527+0000","last_became_peered":"2023-10-14T16:57:44.275527+0000","last_unstale":"2023-10-14T16:57:47.152353+0000","last_undegraded":"2023-10-14T16:57:47.152353+0000","last_fullsized":"2023-10-14T16:57:47.152353+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:46.733073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151279+0000","last_change":"2023-10-14T16:57:45.197567+0000","last_active":"2023-10-14T16:57:47.151279+0000","last_peered":"2023-10-14T16:57:47.151279+0000","last_clean":"2023-10-14T16:57:47.151279+0000","last_became_active":"2023-10-14T16:57:45.197296+0000","last_became_peered":"2023-10-14T16:57:45.197296+0000","last_unstale":"2023-10-14T16:57:47.151279+0000","last_undegraded":"2023-10-14T16:57:47.151279+0000","last_fullsized":"2023-10-14T16:57:47.151279+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:59.230753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151685+0000","last_change":"2023-10-14T16:57:44.278662+0000","last_active":"2023-10-14T16:57:47.151685+0000","last_peered":"2023-10-14T16:57:47.151685+0000","last_clean":"2023-10-14T16:57:47.151685+0000","last_became_active":"2023-10-14T16:57:44.277548+0000","last_became_peered":"2023-10-14T16:57:44.277548+0000","last_unstale":"2023-10-14T16:57:47.151685+0000","last_undegraded":"2023-10-14T16:57:47.151685+0000","last_fullsized":"2023-10-14T16:57:47.151685+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:25.036463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151520+0000","last_change":"2023-10-14T16:57:45.203076+0000","last_active":"2023-10-14T16:57:47.151520+0000","last_peered":"2023-10-14T16:57:47.151520+0000","last_clean":"2023-10-14T16:57:47.151520+0000","last_became_active":"2023-10-14T16:57:45.202775+0000","last_became_peered":"2023-10-14T16:57:45.202775+0000","last_unstale":"2023-10-14T16:57:47.151520+0000","last_undegraded":"2023-10-14T16:57:47.151520+0000","last_fullsized":"2023-10-14T16:57:47.151520+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:37.544968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.150875+0000","last_change":"2023-10-14T16:57:44.277664+0000","last_active":"2023-10-14T16:57:47.150875+0000","last_peered":"2023-10-14T16:57:47.150875+0000","last_clean":"2023-10-14T16:57:47.150875+0000","last_became_active":"2023-10-14T16:57:44.277403+0000","last_became_peered":"2023-10-14T16:57:44.277403+0000","last_unstale":"2023-10-14T16:57:47.150875+0000","last_undegraded":"2023-10-14T16:57:47.150875+0000","last_fullsized":"2023-10-14T16:57:47.150875+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:06.785421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188026+0000","last_change":"2023-10-14T16:57:45.185076+0000","last_active":"2023-10-14T16:57:46.188026+0000","last_peered":"2023-10-14T16:57:46.188026+0000","last_clean":"2023-10-14T16:57:46.188026+0000","last_became_active":"2023-10-14T16:57:45.184616+0000","last_became_peered":"2023-10-14T16:57:45.184616+0000","last_unstale":"2023-10-14T16:57:46.188026+0000","last_undegraded":"2023-10-14T16:57:46.188026+0000","last_fullsized":"2023-10-14T16:57:46.188026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:40.641312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507914+0000","last_change":"2023-10-14T16:57:44.281564+0000","last_active":"2023-10-14T16:57:46.507914+0000","last_peered":"2023-10-14T16:57:46.507914+0000","last_clean":"2023-10-14T16:57:46.507914+0000","last_became_active":"2023-10-14T16:57:44.281219+0000","last_became_peered":"2023-10-14T16:57:44.281219+0000","last_unstale":"2023-10-14T16:57:46.507914+0000","last_undegraded":"2023-10-14T16:57:46.507914+0000","last_fullsized":"2023-10-14T16:57:46.507914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:10.822276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153345+0000","last_change":"2023-10-14T16:57:45.201953+0000","last_active":"2023-10-14T16:57:47.153345+0000","last_peered":"2023-10-14T16:57:47.153345+0000","last_clean":"2023-10-14T16:57:47.153345+0000","last_became_active":"2023-10-14T16:57:45.201670+0000","last_became_peered":"2023-10-14T16:57:45.201670+0000","last_unstale":"2023-10-14T16:57:47.153345+0000","last_undegraded":"2023-10-14T16:57:47.153345+0000","last_fullsized":"2023-10-14T16:57:47.153345+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:25.739658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507783+0000","last_change":"2023-10-14T16:57:44.271652+0000","last_active":"2023-10-14T16:57:46.507783+0000","last_peered":"2023-10-14T16:57:46.507783+0000","last_clean":"2023-10-14T16:57:46.507783+0000","last_became_active":"2023-10-14T16:57:44.270642+0000","last_became_peered":"2023-10-14T16:57:44.270642+0000","last_unstale":"2023-10-14T16:57:46.507783+0000","last_undegraded":"2023-10-14T16:57:46.507783+0000","last_fullsized":"2023-10-14T16:57:46.507783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:06.326274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186973+0000","last_change":"2023-10-14T16:57:45.194622+0000","last_active":"2023-10-14T16:57:46.186973+0000","last_peered":"2023-10-14T16:57:46.186973+0000","last_clean":"2023-10-14T16:57:46.186973+0000","last_became_active":"2023-10-14T16:57:45.194323+0000","last_became_peered":"2023-10-14T16:57:45.194323+0000","last_unstale":"2023-10-14T16:57:46.186973+0000","last_undegraded":"2023-10-14T16:57:46.186973+0000","last_fullsized":"2023-10-14T16:57:46.186973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:04.729022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186901+0000","last_change":"2023-10-14T16:57:44.275895+0000","last_active":"2023-10-14T16:57:46.186901+0000","last_peered":"2023-10-14T16:57:46.186901+0000","last_clean":"2023-10-14T16:57:46.186901+0000","last_became_active":"2023-10-14T16:57:44.275628+0000","last_became_peered":"2023-10-14T16:57:44.275628+0000","last_unstale":"2023-10-14T16:57:46.186901+0000","last_undegraded":"2023-10-14T16:57:46.186901+0000","last_fullsized":"2023-10-14T16:57:46.186901+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:05.634232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188131+0000","last_change":"2023-10-14T16:57:45.200998+0000","last_active":"2023-10-14T16:57:46.188131+0000","last_peered":"2023-10-14T16:57:46.188131+0000","last_clean":"2023-10-14T16:57:46.188131+0000","last_became_active":"2023-10-14T16:57:45.200677+0000","last_became_peered":"2023-10-14T16:57:45.200677+0000","last_unstale":"2023-10-14T16:57:46.188131+0000","last_undegraded":"2023-10-14T16:57:46.188131+0000","last_fullsized":"2023-10-14T16:57:46.188131+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:44.587697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187142+0000","last_change":"2023-10-14T16:57:44.280505+0000","last_active":"2023-10-14T16:57:46.187142+0000","last_peered":"2023-10-14T16:57:46.187142+0000","last_clean":"2023-10-14T16:57:46.187142+0000","last_became_active":"2023-10-14T16:57:44.280158+0000","last_became_peered":"2023-10-14T16:57:44.280158+0000","last_unstale":"2023-10-14T16:57:46.187142+0000","last_undegraded":"2023-10-14T16:57:46.187142+0000","last_fullsized":"2023-10-14T16:57:46.187142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:02.917979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151747+0000","last_change":"2023-10-14T16:57:45.193081+0000","last_active":"2023-10-14T16:57:47.151747+0000","last_peered":"2023-10-14T16:57:47.151747+0000","last_clean":"2023-10-14T16:57:47.151747+0000","last_became_active":"2023-10-14T16:57:45.192763+0000","last_became_peered":"2023-10-14T16:57:45.192763+0000","last_unstale":"2023-10-14T16:57:47.151747+0000","last_undegraded":"2023-10-14T16:57:47.151747+0000","last_fullsized":"2023-10-14T16:57:47.151747+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:20.691373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182775+0000","last_change":"2023-10-14T16:57:44.283925+0000","last_active":"2023-10-14T16:57:46.182775+0000","last_peered":"2023-10-14T16:57:46.182775+0000","last_clean":"2023-10-14T16:57:46.182775+0000","last_became_active":"2023-10-14T16:57:44.283598+0000","last_became_peered":"2023-10-14T16:57:44.283598+0000","last_unstale":"2023-10-14T16:57:46.182775+0000","last_undegraded":"2023-10-14T16:57:46.182775+0000","last_fullsized":"2023-10-14T16:57:46.182775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:24:56.451526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182546+0000","last_change":"2023-10-14T16:57:45.203420+0000","last_active":"2023-10-14T16:57:46.182546+0000","last_peered":"2023-10-14T16:57:46.182546+0000","last_clean":"2023-10-14T16:57:46.182546+0000","last_became_active":"2023-10-14T16:57:45.203102+0000","last_became_peered":"2023-10-14T16:57:45.203102+0000","last_unstale":"2023-10-14T16:57:46.182546+0000","last_undegraded":"2023-10-14T16:57:46.182546+0000","last_fullsized":"2023-10-14T16:57:46.182546+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:01.870891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186227+0000","last_change":"2023-10-14T16:57:44.284606+0000","last_active":"2023-10-14T16:57:46.186227+0000","last_peered":"2023-10-14T16:57:46.186227+0000","last_clean":"2023-10-14T16:57:46.186227+0000","last_became_active":"2023-10-14T16:57:44.284297+0000","last_became_peered":"2023-10-14T16:57:44.284297+0000","last_unstale":"2023-10-14T16:57:46.186227+0000","last_undegraded":"2023-10-14T16:57:46.186227+0000","last_fullsized":"2023-10-14T16:57:46.186227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:11.272769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153267+0000","last_change":"2023-10-14T16:57:45.207861+0000","last_active":"2023-10-14T16:57:47.153267+0000","last_peered":"2023-10-14T16:57:47.153267+0000","last_clean":"2023-10-14T16:57:47.153267+0000","last_became_active":"2023-10-14T16:57:45.207033+0000","last_became_peered":"2023-10-14T16:57:45.207033+0000","last_unstale":"2023-10-14T16:57:47.153267+0000","last_undegraded":"2023-10-14T16:57:47.153267+0000","last_fullsized":"2023-10-14T16:57:47.153267+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:36:09.824679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186416+0000","last_change":"2023-10-14T16:57:44.277999+0000","last_active":"2023-10-14T16:57:46.186416+0000","last_peered":"2023-10-14T16:57:46.186416+0000","last_clean":"2023-10-14T16:57:46.186416+0000","last_became_active":"2023-10-14T16:57:44.277726+0000","last_became_peered":"2023-10-14T16:57:44.277726+0000","last_unstale":"2023-10-14T16:57:46.186416+0000","last_undegraded":"2023-10-14T16:57:46.186416+0000","last_fullsized":"2023-10-14T16:57:46.186416+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:58.217093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186327+0000","last_change":"2023-10-14T16:57:45.199102+0000","last_active":"2023-10-14T16:57:46.186327+0000","last_peered":"2023-10-14T16:57:46.186327+0000","last_clean":"2023-10-14T16:57:46.186327+0000","last_became_active":"2023-10-14T16:57:45.197574+0000","last_became_peered":"2023-10-14T16:57:45.197574+0000","last_unstale":"2023-10-14T16:57:46.186327+0000","last_undegraded":"2023-10-14T16:57:46.186327+0000","last_fullsized":"2023-10-14T16:57:46.186327+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:07.339934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185017+0000","last_change":"2023-10-14T16:57:44.274937+0000","last_active":"2023-10-14T16:57:46.185017+0000","last_peered":"2023-10-14T16:57:46.185017+0000","last_clean":"2023-10-14T16:57:46.185017+0000","last_became_active":"2023-10-14T16:57:44.274590+0000","last_became_peered":"2023-10-14T16:57:44.274590+0000","last_unstale":"2023-10-14T16:57:46.185017+0000","last_undegraded":"2023-10-14T16:57:46.185017+0000","last_fullsized":"2023-10-14T16:57:46.185017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:48.339806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507088+0000","last_change":"2023-10-14T16:57:45.204441+0000","last_active":"2023-10-14T16:57:46.507088+0000","last_peered":"2023-10-14T16:57:46.507088+0000","last_clean":"2023-10-14T16:57:46.507088+0000","last_became_active":"2023-10-14T16:57:45.204130+0000","last_became_peered":"2023-10-14T16:57:45.204130+0000","last_unstale":"2023-10-14T16:57:46.507088+0000","last_undegraded":"2023-10-14T16:57:46.507088+0000","last_fullsized":"2023-10-14T16:57:46.507088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:34:45.843694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153656+0000","last_change":"2023-10-14T16:57:44.283509+0000","last_active":"2023-10-14T16:57:47.153656+0000","last_peered":"2023-10-14T16:57:47.153656+0000","last_clean":"2023-10-14T16:57:47.153656+0000","last_became_active":"2023-10-14T16:57:44.283100+0000","last_became_peered":"2023-10-14T16:57:44.283100+0000","last_unstale":"2023-10-14T16:57:47.153656+0000","last_undegraded":"2023-10-14T16:57:47.153656+0000","last_fullsized":"2023-10-14T16:57:47.153656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:06.785421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'183","reported_seq":481,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-14T16:57:45.187796+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:57:45.187796+0000","last_peered":"2023-10-14T16:57:45.187796+0000","last_clean":"2023-10-14T16:57:45.187796+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:57:45.187796+0000","last_undegraded":"2023-10-14T16:57:45.187796+0000","last_fullsized":"2023-10-14T16:57:45.187796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575058,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1937408,"data_stored":1234264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.169,"15min":1.1699999999999999},"min":{"1min":0.66000000000000003,"5min":0.56599999999999995,"15min":0.51500000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.73299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6279999999999999,"5min":1.8400000000000001,"15min":2.117},"last":1.5680000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1890000000000001,"15min":1.2230000000000001},"min":{"1min":0.73799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3560000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.236,"15min":1.2569999999999999},"min":{"1min":0.66700000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.722,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.3460000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1910000000000001,"15min":1.224},"min":{"1min":0.623,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.27},{"interface":"front","average":{"1min":1.214,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.78900000000000003,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.1589999999999998},"last":1.5249999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":1.369},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.996,"5min":1.996,"15min":2.4159999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4259999999999999,"15min":1.446},"min":{"1min":0.85599999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0880000000000001},"last":1.4319999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.52600000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.881,"5min":2.105,"15min":2.1819999999999999},"last":1.772},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.603}]},{"osd":5,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3540000000000001,"15min":1.385},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0390000000000001},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4450000000000001,"15min":1.4590000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.677}]},{"osd":6,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.532,"15min":1.542},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.587,"5min":1.4790000000000001,"15min":1.4830000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.484}]}]},{"osd":6,"up_from":13,"seq":55834575057,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1351680,"data_stored":644520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1519999999999999,"15min":1.153},"min":{"1min":0.81799999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.524,"5min":2.137,"15min":2.137},"last":1.413},{"interface":"front","average":{"1min":1.141,"5min":1.222,"15min":1.206},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8100000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.2669999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1779999999999999,"15min":1.1850000000000001},"min":{"1min":0.77000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.593,"5min":2.802,"15min":2.802},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.395,"5min":2.512,"15min":2.512},"last":1.0589999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.75900000000000001},"max":{"1min":1.5629999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.3680000000000001,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5549999999999999,"5min":3.286,"15min":3.286},"last":1.4490000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.363,"15min":1.3660000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.577,"5min":2.823,"15min":2.823},"last":1.3},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.4410000000000001,"15min":1.415},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.464}]},{"osd":4,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.3400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.74,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4790000000000001,"15min":1.4119999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8049999999999999,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.135}]},{"osd":5,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3999999999999999,"15min":1.3740000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7609999999999999,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.238},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8999999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.4910000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.6299999999999999,"15min":1.5760000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.992,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.365},{"interface":"front","average":{"1min":1.474,"5min":1.5189999999999999,"15min":1.532},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9670000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.3919999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575058,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":645460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1100000000000001,"15min":1.1120000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5409999999999999,"5min":1.861,"15min":1.986},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.127,"5min":1.1060000000000001,"15min":1.129},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.60599999999999998},"max":{"1min":1.6599999999999999,"5min":1.665,"15min":2.234},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1439999999999999,"15min":1.167},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6100000000000001,"5min":1.7090000000000001,"15min":2.1200000000000001},"last":0.753},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.1830000000000001,"15min":1.216},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7609999999999999,"15min":1.905},"last":1.133}]},{"osd":2,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5760000000000001,"5min":1.7150000000000001,"15min":2.375},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3129999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.577,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.321,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.72,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.083},{"interface":"front","average":{"1min":1.298,"5min":1.375,"15min":1.3999999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.732,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":0.71099999999999997}]},{"osd":4,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.272,"15min":1.284},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.6699999999999999,"5min":2.194,"15min":2.4889999999999999},"last":0.434},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.335,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9119999999999999,"5min":1.927,"15min":2.5169999999999999},"last":1.173}]},{"osd":6,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.302},"last":0.879},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5229999999999999,"15min":1.5349999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.758,"5min":2.1059999999999999,"15min":2.613},"last":0.65500000000000003}]},{"osd":7,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.478,"15min":1.5049999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.806,"5min":2.032,"15min":2.605},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.522,"15min":1.5109999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8280000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.04}]}]},{"osd":4,"up_from":13,"seq":55834575058,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1339392,"data_stored":643826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.623,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.486,"5min":1.7050000000000001,"15min":1.982},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.23},"min":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.704,"5min":1.972,"15min":2.1099999999999999},"last":4.5979999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.753,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7490000000000001,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":6.7880000000000003},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.206,"15min":1.24},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.478,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":5.9530000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2250000000000001,"15min":1.226},"min":{"1min":0.73099999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.752,"5min":2.169,"15min":2.169},"last":5.633},{"interface":"front","average":{"1min":1.319,"5min":1.3799999999999999,"15min":1.3740000000000001},"min":{"1min":0.83399999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.7759999999999998},"last":3.5070000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2669999999999999,"15min":1.3100000000000001},"min":{"1min":0.77100000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6819999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":4.6349999999999998},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77800000000000002},"max":{"1min":1.8440000000000001,"5min":1.952,"15min":2.085},"last":3.5590000000000002}]},{"osd":5,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.361,"15min":1.3520000000000001},"min":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":2.931},{"interface":"front","average":{"1min":1.329,"5min":1.3080000000000001,"15min":1.343},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":2.153},"last":3.2599999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3779999999999999,"15min":1.389},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9139999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":2.323},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.504,"15min":1.526},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0960000000000001,"5min":2.214,"15min":2.6539999999999999},"last":2.9889999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4730000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.2389999999999999},"last":4.7960000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.5469999999999999},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.883}]}]},{"osd":3,"up_from":13,"seq":55834575058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":645438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.879,"15min":0.92600000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.2270000000000001,"5min":1.621,"15min":1.621},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0449999999999999,"15min":1.1319999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.377,"5min":1.593,"15min":1.8799999999999999},"last":1.181}]},{"osd":1,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83299999999999996,"15min":0.91100000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.234,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.026,"15min":1.109},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.772,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":0.98199999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2370000000000001,"15min":1.365},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.024,"5min":2.024,"15min":2.0609999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1719999999999999,"15min":1.2829999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.8240000000000001,"5min":1.8560000000000001,"15min":2.0960000000000001},"last":1.3380000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3240000000000001,"15min":1.365},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8859999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.069},{"interface":"front","average":{"1min":1.25,"5min":1.3540000000000001,"15min":1.407},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.802,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.5880000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.319,"15min":1.3939999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.895,"5min":2.516,"15min":3.0190000000000001},"last":1.448},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.389,"15min":1.4390000000000001},"min":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8879999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.298}]},{"osd":6,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4159999999999999,"15min":1.4690000000000001},"min":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.911,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.488},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4950000000000001,"15min":1.5389999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9810000000000001,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.377}]},{"osd":7,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4399999999999999,"15min":1.5089999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.093,"5min":2.871,"15min":2.875},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.5049999999999999,"15min":1.554},"min":{"1min":0.88200000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2330000000000001,"5min":2.907,"15min":2.907},"last":1.4039999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575058,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44980,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326860,"statfs":{"total":96636764160,"available":96590704640,"internally_reserved":0,"allocated":1343488,"data_stored":644408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.706,"5min":1.706,"15min":1.849},"last":1.101},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":1.044},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4990000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.91000000000000003}]},{"osd":1,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.98199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0419999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.165,"15min":1.175},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.74,"5min":2.0390000000000001,"15min":2.069},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1539999999999999,"15min":1.23},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6930000000000001,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.242,"15min":1.3149999999999999},"min":{"1min":0.58699999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.52,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.4530000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.361,"15min":1.4199999999999999},"min":{"1min":0.75,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":2.3159999999999998},"last":1.534},{"interface":"front","average":{"1min":1.206,"5min":1.327,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.381}]},{"osd":5,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.363,"15min":1.399},"min":{"1min":0.73999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.472,"15min":1.53},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7849999999999999,"5min":2.2730000000000001,"15min":2.4689999999999999},"last":1.3129999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.395,"15min":1.4490000000000001},"min":{"1min":0.82399999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.958,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.474,"15min":1.526},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.1629999999999998},"last":1.4930000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.0129999999999999,"5min":2.109,"15min":2.109},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.5329999999999999,"15min":1.5920000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.27}]}]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58872,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312968,"statfs":{"total":96636764160,"available":96576479232,"internally_reserved":0,"allocated":1351680,"data_stored":644020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.90500000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.708,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.996,"15min":1.0669999999999999},"min":{"1min":0.51500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.5549999999999999,"5min":1.728,"15min":2.0819999999999999},"last":0.80100000000000005}]},{"osd":2,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":0.96199999999999997},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.758},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0780000000000001,"15min":1.165},"min":{"1min":0.59899999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.556,"5min":2.0009999999999999,"15min":2.0150000000000001},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1919999999999999,"15min":1.2829999999999999},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.724,"5min":2.1549999999999998,"15min":2.198},"last":1.637},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.099,"15min":1.1839999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.421,"5min":1.802,"15min":2.004},"last":1.1319999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.361,"15min":1.4059999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.575,"5min":1.9470000000000001,"15min":2.3050000000000002},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2609999999999999,"15min":1.3080000000000001},"min":{"1min":0.83699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.819,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.387}]},{"osd":5,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.80900000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0430000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.165},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.373,"15min":1.458},"min":{"1min":0.83999999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6100000000000001,"5min":2.2970000000000002,"15min":2.302},"last":1.2090000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.403,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6619999999999999,"5min":2.1880000000000002,"15min":2.4169999999999998},"last":1.742},{"interface":"front","average":{"1min":1.399,"5min":1.421,"15min":1.47},"min":{"1min":1.02,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0870000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.605}]},{"osd":7,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.482,"15min":1.5609999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.827,"5min":2.2290000000000001,"15min":2.411},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.482,"15min":1.575},"min":{"1min":0.81899999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.734,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":1.2450000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":1234228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90700000000000003,"15min":0.89200000000000002},"min":{"1min":0.55600000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.355,"5min":3.355,"15min":3.355},"last":1.704},{"interface":"front","average":{"1min":1.071,"5min":1.0620000000000001,"15min":1.0980000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46999999999999997},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.98399999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.59099999999999997,"5min":0.443,"15min":0.443},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1830000000000001,"15min":1.2090000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.42699999999999999},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":1.1479999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.278},"min":{"1min":0.69799999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.444},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.304,"15min":1.355},"min":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":2.4790000000000001,"5min":2.593,"15min":2.593},"last":2.0150000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.302,"15min":1.2909999999999999},"min":{"1min":0.873,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.77800000000000002},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.8200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.96499999999999997,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.264},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4870000000000001,"15min":1.524},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.306,"5min":3.306,"15min":3.306},"last":1.3799999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.55,"15min":1.5900000000000001},"min":{"1min":0.92500000000000004,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.3999999999999999,"15min":1.405},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.528}]},{"osd":7,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.9550000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":1.0660000000000001,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":3.698,"5min":3.698,"15min":3.698},"last":1.8660000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:57:49.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:57:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.041+0000 7f9cb9347640 1 Processor -- start 2023-10-14T16:57:50.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.041+0000 7f9cb9347640 1 -- start start 2023-10-14T16:57:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.043+0000 7f9cb9347640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb415c9f0 0x7f9cb40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.043+0000 7f9cb9347640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cb40c9230 0x7f9cb40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.043+0000 7f9cb9347640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb4160a90 con 0x7f9cb4070f20 2023-10-14T16:57:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.043+0000 7f9cb9347640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb40cdec0 con 0x7f9cb40c9230 2023-10-14T16:57:50.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.043+0000 7f9cb9347640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40cd900 con 0x7f9cb415c9f0 2023-10-14T16:57:50.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cb40c9230 0x7f9cb40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cb40c9230 0x7f9cb40709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58822/0 (socket says 172.21.15.29:58822) 2023-10-14T16:57:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 -- 172.21.15.29:0/3756208191 learned_addr learned my addr 172.21.15.29:0/3756208191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb3fff640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb415c9f0 0x7f9cb40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 -- 172.21.15.29:0/3756208191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb415c9f0 msgr2=0x7f9cb40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:50.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb415c9f0 0x7f9cb40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 -- 172.21.15.29:0/3756208191 >> v1:172.21.15.29:6789/0 conn(0x7f9cb4070f20 legacy=0x7f9cb40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.044+0000 7f9cb37fe640 1 -- 172.21.15.29:0/3756208191 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40cdbe0 con 0x7f9cb40c9230 2023-10-14T16:57:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.046+0000 7f9cb37fe640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb40709e0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009870 tx=0x7f9ca802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=75ea3ac8f1cb9a4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.046+0000 7f9cb2ffd640 1 -- 172.21.15.29:0/3756208191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8030840 con 0x7f9cb40c9230 2023-10-14T16:57:50.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.046+0000 7f9cb2ffd640 1 -- 172.21.15.29:0/3756208191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca80309a0 con 0x7f9cb40c9230 2023-10-14T16:57:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.046+0000 7f9cb2ffd640 1 -- 172.21.15.29:0/3756208191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8030d30 con 0x7f9cb40c9230 2023-10-14T16:57:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 -- 172.21.15.29:0/3756208191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 msgr2=0x7f9cb40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb40709e0 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8009870 tx=0x7f9ca802b980 comp rx=0 tx=0).stop 2023-10-14T16:57:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 -- 172.21.15.29:0/3756208191 shutdown_connections 2023-10-14T16:57:50.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb40709e0 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 --2- 172.21.15.29:0/3756208191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb415c9f0 0x7f9cb40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 -- 172.21.15.29:0/3756208191 >> 172.21.15.29:0/3756208191 conn(0x7f9cb40ce6a0 msgr2=0x7f9cb40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 -- 172.21.15.29:0/3756208191 shutdown_connections 2023-10-14T16:57:50.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.047+0000 7f9cb9347640 1 -- 172.21.15.29:0/3756208191 wait complete. 2023-10-14T16:57:50.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.049+0000 7f9cb9347640 1 Processor -- start 2023-10-14T16:57:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.049+0000 7f9cb9347640 1 -- start start 2023-10-14T16:57:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb4070f20 0x7f9cb406c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cb415c9f0 0x7f9cb415abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c3220 con 0x7f9cb40c9230 2023-10-14T16:57:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c3da0 con 0x7f9cb415c9f0 2023-10-14T16:57:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb9347640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb40c3470 con 0x7f9cb4070f20 2023-10-14T16:57:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58830/0 (socket says 172.21.15.29:58830) 2023-10-14T16:57:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb37fe640 1 -- 172.21.15.29:0/1000147481 learned_addr learned my addr 172.21.15.29:0/1000147481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb3fff640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb4070f20 0x7f9cb406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.050+0000 7f9cb8b46640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cb415c9f0 0x7f9cb415abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 -- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb4070f20 msgr2=0x7f9cb406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb4070f20 0x7f9cb406c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 -- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cb415c9f0 msgr2=0x7f9cb415abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cb415c9f0 0x7f9cb415abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 -- 172.21.15.29:0/1000147481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca80094d0 con 0x7f9cb40c9230 2023-10-14T16:57:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb37fe640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8000c00 tx=0x7f9ca8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8039c90 con 0x7f9cb40c9230 2023-10-14T16:57:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca8039df0 con 0x7f9cb40c9230 2023-10-14T16:57:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ca8043620 con 0x7f9cb40c9230 2023-10-14T16:57:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.051+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb40c3750 con 0x7f9cb40c9230 2023-10-14T16:57:50.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.052+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb406ca50 con 0x7f9cb40c9230 2023-10-14T16:57:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.053+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb404e9b0 con 0x7f9cb40c9230 2023-10-14T16:57:50.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.054+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9ca8043780 con 0x7f9cb40c9230 2023-10-14T16:57:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.054+0000 7f9cb0ff9640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 0x7f9ca006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.055+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f9ca803e070 con 0x7f9cb40c9230 2023-10-14T16:57:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.055+0000 7f9cb3fff640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 0x7f9ca006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.055+0000 7f9cb3fff640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 0x7f9ca006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4006fd0 tx=0x7f9ca4008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.059+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ca8077850 con 0x7f9cb40c9230 2023-10-14T16:57:50.224 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T16:57:50.224+0000 7febe022e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:57:50.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.251+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 --> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cb40704c0 con 0x7f9ca006af90 2023-10-14T16:57:50.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.264+0000 7f9cb0ff9640 1 -- 172.21.15.29:0/1000147481 <== mgr.7257 v2:172.21.15.29:6832/1520757887 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287303 (secure 0 0 0) 0x7f9ca400f860 con 0x7f9ca006af90 2023-10-14T16:57:50.267 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:50.270 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:57:50.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 msgr2=0x7f9ca006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 0x7f9ca006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4006fd0 tx=0x7f9ca4008040 comp rx=0 tx=0).stop 2023-10-14T16:57:50.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 msgr2=0x7f9cb415a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 crypto rx=0x7f9ca8000c00 tx=0x7f9ca8004730 comp rx=0 tx=0).stop 2023-10-14T16:57:50.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 shutdown_connections 2023-10-14T16:57:50.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f9ca006af90 0x7f9ca006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cb415c9f0 0x7f9cb415abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cb40c9230 0x7f9cb415a6b0 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 --2- 172.21.15.29:0/1000147481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cb4070f20 0x7f9cb406c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.271+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 >> 172.21.15.29:0/1000147481 conn(0x7f9cb40ce6a0 msgr2=0x7f9cb4160f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:50.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.272+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 shutdown_connections 2023-10-14T16:57:50.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.272+0000 7f9cb9347640 1 -- 172.21.15.29:0/1000147481 wait complete. 2023-10-14T16:57:50.290 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":25,"stamp":"2023-10-14T16:57:49.194510+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":323,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":170,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":347172,"kb_used_data":11688,"kb_used_omap":12,"kb_used_meta":335347,"kb_avail":754627548,"statfs":{"total":773094113280,"available":772738609152,"internally_reserved":0,"allocated":11968512,"data_stored":6336164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":343395896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.954481"},"pg_stats":[{"pgid":"13.3f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187319+0000","last_change":"2023-10-14T16:57:45.193978+0000","last_active":"2023-10-14T16:57:46.187319+0000","last_peered":"2023-10-14T16:57:46.187319+0000","last_clean":"2023-10-14T16:57:46.187319+0000","last_became_active":"2023-10-14T16:57:45.193699+0000","last_became_peered":"2023-10-14T16:57:45.193699+0000","last_unstale":"2023-10-14T16:57:46.187319+0000","last_undegraded":"2023-10-14T16:57:46.187319+0000","last_fullsized":"2023-10-14T16:57:46.187319+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:43.079137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.3e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507674+0000","last_change":"2023-10-14T16:57:45.196412+0000","last_active":"2023-10-14T16:57:46.507674+0000","last_peered":"2023-10-14T16:57:46.507674+0000","last_clean":"2023-10-14T16:57:46.507674+0000","last_became_active":"2023-10-14T16:57:45.195455+0000","last_became_peered":"2023-10-14T16:57:45.195455+0000","last_unstale":"2023-10-14T16:57:46.507674+0000","last_undegraded":"2023-10-14T16:57:46.507674+0000","last_fullsized":"2023-10-14T16:57:46.507674+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:35.836488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.3d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153245+0000","last_change":"2023-10-14T16:57:45.199410+0000","last_active":"2023-10-14T16:57:47.153245+0000","last_peered":"2023-10-14T16:57:47.153245+0000","last_clean":"2023-10-14T16:57:47.153245+0000","last_became_active":"2023-10-14T16:57:45.197998+0000","last_became_peered":"2023-10-14T16:57:45.197998+0000","last_unstale":"2023-10-14T16:57:47.153245+0000","last_undegraded":"2023-10-14T16:57:47.153245+0000","last_fullsized":"2023-10-14T16:57:47.153245+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:08.513940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.3c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151853+0000","last_change":"2023-10-14T16:57:45.209311+0000","last_active":"2023-10-14T16:57:47.151853+0000","last_peered":"2023-10-14T16:57:47.151853+0000","last_clean":"2023-10-14T16:57:47.151853+0000","last_became_active":"2023-10-14T16:57:45.208004+0000","last_became_peered":"2023-10-14T16:57:45.208004+0000","last_unstale":"2023-10-14T16:57:47.151853+0000","last_undegraded":"2023-10-14T16:57:47.151853+0000","last_fullsized":"2023-10-14T16:57:47.151853+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:06.647893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.3b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185600+0000","last_change":"2023-10-14T16:57:45.207850+0000","last_active":"2023-10-14T16:57:46.185600+0000","last_peered":"2023-10-14T16:57:46.185600+0000","last_clean":"2023-10-14T16:57:46.185600+0000","last_became_active":"2023-10-14T16:57:45.207572+0000","last_became_peered":"2023-10-14T16:57:45.207572+0000","last_unstale":"2023-10-14T16:57:46.185600+0000","last_undegraded":"2023-10-14T16:57:46.185600+0000","last_fullsized":"2023-10-14T16:57:46.185600+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:56.351969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.3a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151297+0000","last_change":"2023-10-14T16:57:45.192374+0000","last_active":"2023-10-14T16:57:47.151297+0000","last_peered":"2023-10-14T16:57:47.151297+0000","last_clean":"2023-10-14T16:57:47.151297+0000","last_became_active":"2023-10-14T16:57:45.192014+0000","last_became_peered":"2023-10-14T16:57:45.192014+0000","last_unstale":"2023-10-14T16:57:47.151297+0000","last_undegraded":"2023-10-14T16:57:47.151297+0000","last_fullsized":"2023-10-14T16:57:47.151297+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:28.157778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.39","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152042+0000","last_change":"2023-10-14T16:57:45.194958+0000","last_active":"2023-10-14T16:57:47.152042+0000","last_peered":"2023-10-14T16:57:47.152042+0000","last_clean":"2023-10-14T16:57:47.152042+0000","last_became_active":"2023-10-14T16:57:45.194555+0000","last_became_peered":"2023-10-14T16:57:45.194555+0000","last_unstale":"2023-10-14T16:57:47.152042+0000","last_undegraded":"2023-10-14T16:57:47.152042+0000","last_fullsized":"2023-10-14T16:57:47.152042+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:47.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.38","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151298+0000","last_change":"2023-10-14T16:57:45.193775+0000","last_active":"2023-10-14T16:57:47.151298+0000","last_peered":"2023-10-14T16:57:47.151298+0000","last_clean":"2023-10-14T16:57:47.151298+0000","last_became_active":"2023-10-14T16:57:45.193494+0000","last_became_peered":"2023-10-14T16:57:45.193494+0000","last_unstale":"2023-10-14T16:57:47.151298+0000","last_undegraded":"2023-10-14T16:57:47.151298+0000","last_fullsized":"2023-10-14T16:57:47.151298+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:24.980138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.37","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152022+0000","last_change":"2023-10-14T16:57:45.198782+0000","last_active":"2023-10-14T16:57:47.152022+0000","last_peered":"2023-10-14T16:57:47.152022+0000","last_clean":"2023-10-14T16:57:47.152022+0000","last_became_active":"2023-10-14T16:57:45.198474+0000","last_became_peered":"2023-10-14T16:57:45.198474+0000","last_unstale":"2023-10-14T16:57:47.152022+0000","last_undegraded":"2023-10-14T16:57:47.152022+0000","last_fullsized":"2023-10-14T16:57:47.152022+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:38:53.010356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.36","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187419+0000","last_change":"2023-10-14T16:57:45.200504+0000","last_active":"2023-10-14T16:57:46.187419+0000","last_peered":"2023-10-14T16:57:46.187419+0000","last_clean":"2023-10-14T16:57:46.187419+0000","last_became_active":"2023-10-14T16:57:45.199305+0000","last_became_peered":"2023-10-14T16:57:45.199305+0000","last_unstale":"2023-10-14T16:57:46.187419+0000","last_undegraded":"2023-10-14T16:57:46.187419+0000","last_fullsized":"2023-10-14T16:57:46.187419+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:34:45.843694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.35","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507418+0000","last_change":"2023-10-14T16:57:45.197104+0000","last_active":"2023-10-14T16:57:46.507418+0000","last_peered":"2023-10-14T16:57:46.507418+0000","last_clean":"2023-10-14T16:57:46.507418+0000","last_became_active":"2023-10-14T16:57:45.196743+0000","last_became_peered":"2023-10-14T16:57:45.196743+0000","last_unstale":"2023-10-14T16:57:46.507418+0000","last_undegraded":"2023-10-14T16:57:46.507418+0000","last_fullsized":"2023-10-14T16:57:46.507418+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:54.024111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.34","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186622+0000","last_change":"2023-10-14T16:57:45.197933+0000","last_active":"2023-10-14T16:57:46.186622+0000","last_peered":"2023-10-14T16:57:46.186622+0000","last_clean":"2023-10-14T16:57:46.186622+0000","last_became_active":"2023-10-14T16:57:45.195872+0000","last_became_peered":"2023-10-14T16:57:45.195872+0000","last_unstale":"2023-10-14T16:57:46.186622+0000","last_undegraded":"2023-10-14T16:57:46.186622+0000","last_fullsized":"2023-10-14T16:57:46.186622+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:49.720572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.33","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152644+0000","last_change":"2023-10-14T16:57:45.200442+0000","last_active":"2023-10-14T16:57:47.152644+0000","last_peered":"2023-10-14T16:57:47.152644+0000","last_clean":"2023-10-14T16:57:47.152644+0000","last_became_active":"2023-10-14T16:57:45.200116+0000","last_became_peered":"2023-10-14T16:57:45.200116+0000","last_unstale":"2023-10-14T16:57:47.152644+0000","last_undegraded":"2023-10-14T16:57:47.152644+0000","last_fullsized":"2023-10-14T16:57:47.152644+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:31.865642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.32","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151692+0000","last_change":"2023-10-14T16:57:45.201437+0000","last_active":"2023-10-14T16:57:47.151692+0000","last_peered":"2023-10-14T16:57:47.151692+0000","last_clean":"2023-10-14T16:57:47.151692+0000","last_became_active":"2023-10-14T16:57:45.200719+0000","last_became_peered":"2023-10-14T16:57:45.200719+0000","last_unstale":"2023-10-14T16:57:47.151692+0000","last_undegraded":"2023-10-14T16:57:47.151692+0000","last_fullsized":"2023-10-14T16:57:47.151692+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:33.688264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.31","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151695+0000","last_change":"2023-10-14T16:57:45.200286+0000","last_active":"2023-10-14T16:57:47.151695+0000","last_peered":"2023-10-14T16:57:47.151695+0000","last_clean":"2023-10-14T16:57:47.151695+0000","last_became_active":"2023-10-14T16:57:45.199979+0000","last_became_peered":"2023-10-14T16:57:45.199979+0000","last_unstale":"2023-10-14T16:57:47.151695+0000","last_undegraded":"2023-10-14T16:57:47.151695+0000","last_fullsized":"2023-10-14T16:57:47.151695+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:31:28.269227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.30","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185561+0000","last_change":"2023-10-14T16:57:45.189274+0000","last_active":"2023-10-14T16:57:46.185561+0000","last_peered":"2023-10-14T16:57:46.185561+0000","last_clean":"2023-10-14T16:57:46.185561+0000","last_became_active":"2023-10-14T16:57:45.188909+0000","last_became_peered":"2023-10-14T16:57:45.188909+0000","last_unstale":"2023-10-14T16:57:46.185561+0000","last_undegraded":"2023-10-14T16:57:46.185561+0000","last_fullsized":"2023-10-14T16:57:46.185561+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:37.442970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188619+0000","last_change":"2023-10-14T16:57:45.194206+0000","last_active":"2023-10-14T16:57:46.188619+0000","last_peered":"2023-10-14T16:57:46.188619+0000","last_clean":"2023-10-14T16:57:46.188619+0000","last_became_active":"2023-10-14T16:57:45.193881+0000","last_became_peered":"2023-10-14T16:57:45.193881+0000","last_unstale":"2023-10-14T16:57:46.188619+0000","last_undegraded":"2023-10-14T16:57:46.188619+0000","last_fullsized":"2023-10-14T16:57:46.188619+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:50.439386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.2e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152430+0000","last_change":"2023-10-14T16:57:45.199533+0000","last_active":"2023-10-14T16:57:47.152430+0000","last_peered":"2023-10-14T16:57:47.152430+0000","last_clean":"2023-10-14T16:57:47.152430+0000","last_became_active":"2023-10-14T16:57:45.198157+0000","last_became_peered":"2023-10-14T16:57:45.198157+0000","last_unstale":"2023-10-14T16:57:47.152430+0000","last_undegraded":"2023-10-14T16:57:47.152430+0000","last_fullsized":"2023-10-14T16:57:47.152430+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:01.870891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183381+0000","last_change":"2023-10-14T16:57:45.204967+0000","last_active":"2023-10-14T16:57:46.183381+0000","last_peered":"2023-10-14T16:57:46.183381+0000","last_clean":"2023-10-14T16:57:46.183381+0000","last_became_active":"2023-10-14T16:57:45.204632+0000","last_became_peered":"2023-10-14T16:57:45.204632+0000","last_unstale":"2023-10-14T16:57:46.183381+0000","last_undegraded":"2023-10-14T16:57:46.183381+0000","last_fullsized":"2023-10-14T16:57:46.183381+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:46.620137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.2c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153053+0000","last_change":"2023-10-14T16:57:45.202681+0000","last_active":"2023-10-14T16:57:47.153053+0000","last_peered":"2023-10-14T16:57:47.153053+0000","last_clean":"2023-10-14T16:57:47.153053+0000","last_became_active":"2023-10-14T16:57:45.202412+0000","last_became_peered":"2023-10-14T16:57:45.202412+0000","last_unstale":"2023-10-14T16:57:47.153053+0000","last_undegraded":"2023-10-14T16:57:47.153053+0000","last_fullsized":"2023-10-14T16:57:47.153053+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:46.620137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151625+0000","last_change":"2023-10-14T16:57:45.211973+0000","last_active":"2023-10-14T16:57:47.151625+0000","last_peered":"2023-10-14T16:57:47.151625+0000","last_clean":"2023-10-14T16:57:47.151625+0000","last_became_active":"2023-10-14T16:57:45.211116+0000","last_became_peered":"2023-10-14T16:57:45.211116+0000","last_unstale":"2023-10-14T16:57:47.151625+0000","last_undegraded":"2023-10-14T16:57:47.151625+0000","last_fullsized":"2023-10-14T16:57:47.151625+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:03.931640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.2a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507542+0000","last_change":"2023-10-14T16:57:45.208939+0000","last_active":"2023-10-14T16:57:46.507542+0000","last_peered":"2023-10-14T16:57:46.507542+0000","last_clean":"2023-10-14T16:57:46.507542+0000","last_became_active":"2023-10-14T16:57:45.208040+0000","last_became_peered":"2023-10-14T16:57:45.208040+0000","last_unstale":"2023-10-14T16:57:46.507542+0000","last_undegraded":"2023-10-14T16:57:46.507542+0000","last_fullsized":"2023-10-14T16:57:46.507542+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:42:24.525894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.29","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186117+0000","last_change":"2023-10-14T16:57:45.198225+0000","last_active":"2023-10-14T16:57:46.186117+0000","last_peered":"2023-10-14T16:57:46.186117+0000","last_clean":"2023-10-14T16:57:46.186117+0000","last_became_active":"2023-10-14T16:57:45.197948+0000","last_became_peered":"2023-10-14T16:57:45.197948+0000","last_unstale":"2023-10-14T16:57:46.186117+0000","last_undegraded":"2023-10-14T16:57:46.186117+0000","last_fullsized":"2023-10-14T16:57:46.186117+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:07:03.939699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.28","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151171+0000","last_change":"2023-10-14T16:57:45.205417+0000","last_active":"2023-10-14T16:57:47.151171+0000","last_peered":"2023-10-14T16:57:47.151171+0000","last_clean":"2023-10-14T16:57:47.151171+0000","last_became_active":"2023-10-14T16:57:45.203613+0000","last_became_peered":"2023-10-14T16:57:45.203613+0000","last_unstale":"2023-10-14T16:57:47.151171+0000","last_undegraded":"2023-10-14T16:57:47.151171+0000","last_fullsized":"2023-10-14T16:57:47.151171+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:54.056332+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.27","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507934+0000","last_change":"2023-10-14T16:57:45.204730+0000","last_active":"2023-10-14T16:57:46.507934+0000","last_peered":"2023-10-14T16:57:46.507934+0000","last_clean":"2023-10-14T16:57:46.507934+0000","last_became_active":"2023-10-14T16:57:45.204442+0000","last_became_peered":"2023-10-14T16:57:45.204442+0000","last_unstale":"2023-10-14T16:57:46.507934+0000","last_undegraded":"2023-10-14T16:57:46.507934+0000","last_fullsized":"2023-10-14T16:57:46.507934+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:02:09.065704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.26","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152935+0000","last_change":"2023-10-14T16:57:45.210079+0000","last_active":"2023-10-14T16:57:47.152935+0000","last_peered":"2023-10-14T16:57:47.152935+0000","last_clean":"2023-10-14T16:57:47.152935+0000","last_became_active":"2023-10-14T16:57:45.209737+0000","last_became_peered":"2023-10-14T16:57:45.209737+0000","last_unstale":"2023-10-14T16:57:47.152935+0000","last_undegraded":"2023-10-14T16:57:47.152935+0000","last_fullsized":"2023-10-14T16:57:47.152935+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:03.623040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"13.25","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187443+0000","last_change":"2023-10-14T16:57:45.195537+0000","last_active":"2023-10-14T16:57:46.187443+0000","last_peered":"2023-10-14T16:57:46.187443+0000","last_clean":"2023-10-14T16:57:46.187443+0000","last_became_active":"2023-10-14T16:57:45.195246+0000","last_became_peered":"2023-10-14T16:57:45.195246+0000","last_unstale":"2023-10-14T16:57:46.187443+0000","last_undegraded":"2023-10-14T16:57:46.187443+0000","last_fullsized":"2023-10-14T16:57:46.187443+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:48.462945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.24","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186565+0000","last_change":"2023-10-14T16:57:45.194725+0000","last_active":"2023-10-14T16:57:46.186565+0000","last_peered":"2023-10-14T16:57:46.186565+0000","last_clean":"2023-10-14T16:57:46.186565+0000","last_became_active":"2023-10-14T16:57:45.194420+0000","last_became_peered":"2023-10-14T16:57:45.194420+0000","last_unstale":"2023-10-14T16:57:46.186565+0000","last_undegraded":"2023-10-14T16:57:46.186565+0000","last_fullsized":"2023-10-14T16:57:46.186565+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:11.835936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.23","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507453+0000","last_change":"2023-10-14T16:57:45.197647+0000","last_active":"2023-10-14T16:57:46.507453+0000","last_peered":"2023-10-14T16:57:46.507453+0000","last_clean":"2023-10-14T16:57:46.507453+0000","last_became_active":"2023-10-14T16:57:45.197233+0000","last_became_peered":"2023-10-14T16:57:45.197233+0000","last_unstale":"2023-10-14T16:57:46.507453+0000","last_undegraded":"2023-10-14T16:57:46.507453+0000","last_fullsized":"2023-10-14T16:57:46.507453+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:39.243523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.22","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507435+0000","last_change":"2023-10-14T16:57:45.195600+0000","last_active":"2023-10-14T16:57:46.507435+0000","last_peered":"2023-10-14T16:57:46.507435+0000","last_clean":"2023-10-14T16:57:46.507435+0000","last_became_active":"2023-10-14T16:57:45.195206+0000","last_became_peered":"2023-10-14T16:57:45.195206+0000","last_unstale":"2023-10-14T16:57:46.507435+0000","last_undegraded":"2023-10-14T16:57:46.507435+0000","last_fullsized":"2023-10-14T16:57:46.507435+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:35:46.266329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.21","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185564+0000","last_change":"2023-10-14T16:57:45.206622+0000","last_active":"2023-10-14T16:57:46.185564+0000","last_peered":"2023-10-14T16:57:46.185564+0000","last_clean":"2023-10-14T16:57:46.185564+0000","last_became_active":"2023-10-14T16:57:45.206326+0000","last_became_peered":"2023-10-14T16:57:45.206326+0000","last_unstale":"2023-10-14T16:57:46.185564+0000","last_undegraded":"2023-10-14T16:57:46.185564+0000","last_fullsized":"2023-10-14T16:57:46.185564+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:45:38.676561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.20","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185671+0000","last_change":"2023-10-14T16:57:45.207794+0000","last_active":"2023-10-14T16:57:46.185671+0000","last_peered":"2023-10-14T16:57:46.185671+0000","last_clean":"2023-10-14T16:57:46.185671+0000","last_became_active":"2023-10-14T16:57:45.207469+0000","last_became_peered":"2023-10-14T16:57:45.207469+0000","last_unstale":"2023-10-14T16:57:46.185671+0000","last_undegraded":"2023-10-14T16:57:46.185671+0000","last_fullsized":"2023-10-14T16:57:46.185671+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:03.671640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187738+0000","last_change":"2023-10-14T16:57:45.191306+0000","last_active":"2023-10-14T16:57:46.187738+0000","last_peered":"2023-10-14T16:57:46.187738+0000","last_clean":"2023-10-14T16:57:46.187738+0000","last_became_active":"2023-10-14T16:57:45.190865+0000","last_became_peered":"2023-10-14T16:57:45.190865+0000","last_unstale":"2023-10-14T16:57:46.187738+0000","last_undegraded":"2023-10-14T16:57:46.187738+0000","last_fullsized":"2023-10-14T16:57:46.187738+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:51:51.357766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1f","version":"108'4","reported_seq":16,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187695+0000","last_change":"2023-10-14T16:57:44.277173+0000","last_active":"2023-10-14T16:57:46.187695+0000","last_peered":"2023-10-14T16:57:46.187695+0000","last_clean":"2023-10-14T16:57:46.187695+0000","last_became_active":"2023-10-14T16:57:44.276881+0000","last_became_peered":"2023-10-14T16:57:44.276881+0000","last_unstale":"2023-10-14T16:57:46.187695+0000","last_undegraded":"2023-10-14T16:57:46.187695+0000","last_fullsized":"2023-10-14T16:57:46.187695+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:46.417549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187435+0000","last_change":"2023-10-14T16:57:45.205762+0000","last_active":"2023-10-14T16:57:46.187435+0000","last_peered":"2023-10-14T16:57:46.187435+0000","last_clean":"2023-10-14T16:57:46.187435+0000","last_became_active":"2023-10-14T16:57:45.205447+0000","last_became_peered":"2023-10-14T16:57:45.205447+0000","last_unstale":"2023-10-14T16:57:46.187435+0000","last_undegraded":"2023-10-14T16:57:46.187435+0000","last_fullsized":"2023-10-14T16:57:46.187435+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:35:46.266329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1e","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188176+0000","last_change":"2023-10-14T16:57:44.284698+0000","last_active":"2023-10-14T16:57:46.188176+0000","last_peered":"2023-10-14T16:57:46.188176+0000","last_clean":"2023-10-14T16:57:46.188176+0000","last_became_active":"2023-10-14T16:57:44.284404+0000","last_became_peered":"2023-10-14T16:57:44.284404+0000","last_unstale":"2023-10-14T16:57:46.188176+0000","last_undegraded":"2023-10-14T16:57:46.188176+0000","last_fullsized":"2023-10-14T16:57:46.188176+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:11.254305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186921+0000","last_change":"2023-10-14T16:57:45.207849+0000","last_active":"2023-10-14T16:57:46.186921+0000","last_peered":"2023-10-14T16:57:46.186921+0000","last_clean":"2023-10-14T16:57:46.186921+0000","last_became_active":"2023-10-14T16:57:45.207562+0000","last_became_peered":"2023-10-14T16:57:45.207562+0000","last_unstale":"2023-10-14T16:57:46.186921+0000","last_undegraded":"2023-10-14T16:57:46.186921+0000","last_fullsized":"2023-10-14T16:57:46.186921+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:37.196176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.1d","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183508+0000","last_change":"2023-10-14T16:57:44.275219+0000","last_active":"2023-10-14T16:57:46.183508+0000","last_peered":"2023-10-14T16:57:46.183508+0000","last_clean":"2023-10-14T16:57:46.183508+0000","last_became_active":"2023-10-14T16:57:44.274959+0000","last_became_peered":"2023-10-14T16:57:44.274959+0000","last_unstale":"2023-10-14T16:57:46.183508+0000","last_undegraded":"2023-10-14T16:57:46.183508+0000","last_fullsized":"2023-10-14T16:57:46.183508+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:58.217093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153532+0000","last_change":"2023-10-14T16:57:45.203212+0000","last_active":"2023-10-14T16:57:47.153532+0000","last_peered":"2023-10-14T16:57:47.153532+0000","last_clean":"2023-10-14T16:57:47.153532+0000","last_became_active":"2023-10-14T16:57:45.202722+0000","last_became_peered":"2023-10-14T16:57:45.202722+0000","last_unstale":"2023-10-14T16:57:47.153532+0000","last_undegraded":"2023-10-14T16:57:47.153532+0000","last_fullsized":"2023-10-14T16:57:47.153532+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:39.631477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1c","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182321+0000","last_change":"2023-10-14T16:57:44.276612+0000","last_active":"2023-10-14T16:57:46.182321+0000","last_peered":"2023-10-14T16:57:46.182321+0000","last_clean":"2023-10-14T16:57:46.182321+0000","last_became_active":"2023-10-14T16:57:44.275530+0000","last_became_peered":"2023-10-14T16:57:44.275530+0000","last_unstale":"2023-10-14T16:57:46.182321+0000","last_undegraded":"2023-10-14T16:57:46.182321+0000","last_fullsized":"2023-10-14T16:57:46.182321+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:25.916883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.1a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507755+0000","last_change":"2023-10-14T16:57:45.195146+0000","last_active":"2023-10-14T16:57:46.507755+0000","last_peered":"2023-10-14T16:57:46.507755+0000","last_clean":"2023-10-14T16:57:46.507755+0000","last_became_active":"2023-10-14T16:57:45.194803+0000","last_became_peered":"2023-10-14T16:57:45.194803+0000","last_unstale":"2023-10-14T16:57:46.507755+0000","last_undegraded":"2023-10-14T16:57:46.507755+0000","last_fullsized":"2023-10-14T16:57:46.507755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:46:33.521273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.1b","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187662+0000","last_change":"2023-10-14T16:57:44.275851+0000","last_active":"2023-10-14T16:57:46.187662+0000","last_peered":"2023-10-14T16:57:46.187662+0000","last_clean":"2023-10-14T16:57:46.187662+0000","last_became_active":"2023-10-14T16:57:44.275509+0000","last_became_peered":"2023-10-14T16:57:44.275509+0000","last_unstale":"2023-10-14T16:57:46.187662+0000","last_undegraded":"2023-10-14T16:57:46.187662+0000","last_fullsized":"2023-10-14T16:57:46.187662+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:15.938015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.1b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187207+0000","last_change":"2023-10-14T16:57:45.194964+0000","last_active":"2023-10-14T16:57:46.187207+0000","last_peered":"2023-10-14T16:57:46.187207+0000","last_clean":"2023-10-14T16:57:46.187207+0000","last_became_active":"2023-10-14T16:57:45.194000+0000","last_became_peered":"2023-10-14T16:57:45.194000+0000","last_unstale":"2023-10-14T16:57:46.187207+0000","last_undegraded":"2023-10-14T16:57:46.187207+0000","last_fullsized":"2023-10-14T16:57:46.187207+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:29:39.471553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.1a","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186071+0000","last_change":"2023-10-14T16:57:44.284835+0000","last_active":"2023-10-14T16:57:46.186071+0000","last_peered":"2023-10-14T16:57:46.186071+0000","last_clean":"2023-10-14T16:57:46.186071+0000","last_became_active":"2023-10-14T16:57:44.284357+0000","last_became_peered":"2023-10-14T16:57:44.284357+0000","last_unstale":"2023-10-14T16:57:46.186071+0000","last_undegraded":"2023-10-14T16:57:46.186071+0000","last_fullsized":"2023-10-14T16:57:46.186071+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:48.844159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.18","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152863+0000","last_change":"2023-10-14T16:57:45.206648+0000","last_active":"2023-10-14T16:57:47.152863+0000","last_peered":"2023-10-14T16:57:47.152863+0000","last_clean":"2023-10-14T16:57:47.152863+0000","last_became_active":"2023-10-14T16:57:45.206231+0000","last_became_peered":"2023-10-14T16:57:45.206231+0000","last_unstale":"2023-10-14T16:57:47.152863+0000","last_undegraded":"2023-10-14T16:57:47.152863+0000","last_fullsized":"2023-10-14T16:57:47.152863+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:10:08.163969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.19","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.181916+0000","last_change":"2023-10-14T16:57:44.268567+0000","last_active":"2023-10-14T16:57:46.181916+0000","last_peered":"2023-10-14T16:57:46.181916+0000","last_clean":"2023-10-14T16:57:46.181916+0000","last_became_active":"2023-10-14T16:57:44.268199+0000","last_became_peered":"2023-10-14T16:57:44.268199+0000","last_unstale":"2023-10-14T16:57:46.181916+0000","last_undegraded":"2023-10-14T16:57:46.181916+0000","last_fullsized":"2023-10-14T16:57:46.181916+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:38:35.271857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.19","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153484+0000","last_change":"2023-10-14T16:57:45.198466+0000","last_active":"2023-10-14T16:57:47.153484+0000","last_peered":"2023-10-14T16:57:47.153484+0000","last_clean":"2023-10-14T16:57:47.153484+0000","last_became_active":"2023-10-14T16:57:45.198174+0000","last_became_peered":"2023-10-14T16:57:45.198174+0000","last_unstale":"2023-10-14T16:57:47.153484+0000","last_undegraded":"2023-10-14T16:57:47.153484+0000","last_fullsized":"2023-10-14T16:57:47.153484+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:36:09.824679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.18","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506961+0000","last_change":"2023-10-14T16:57:44.271763+0000","last_active":"2023-10-14T16:57:46.506961+0000","last_peered":"2023-10-14T16:57:46.506961+0000","last_clean":"2023-10-14T16:57:46.506961+0000","last_became_active":"2023-10-14T16:57:44.270409+0000","last_became_peered":"2023-10-14T16:57:44.270409+0000","last_unstale":"2023-10-14T16:57:46.506961+0000","last_undegraded":"2023-10-14T16:57:46.506961+0000","last_fullsized":"2023-10-14T16:57:46.506961+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:07:02.926038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.16","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152755+0000","last_change":"2023-10-14T16:57:45.211071+0000","last_active":"2023-10-14T16:57:47.152755+0000","last_peered":"2023-10-14T16:57:47.152755+0000","last_clean":"2023-10-14T16:57:47.152755+0000","last_became_active":"2023-10-14T16:57:45.210725+0000","last_became_peered":"2023-10-14T16:57:45.210725+0000","last_unstale":"2023-10-14T16:57:47.152755+0000","last_undegraded":"2023-10-14T16:57:47.152755+0000","last_fullsized":"2023-10-14T16:57:47.152755+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:08.513940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"12.17","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187941+0000","last_change":"2023-10-14T16:57:44.284552+0000","last_active":"2023-10-14T16:57:46.187941+0000","last_peered":"2023-10-14T16:57:46.187941+0000","last_clean":"2023-10-14T16:57:46.187941+0000","last_became_active":"2023-10-14T16:57:44.284161+0000","last_became_peered":"2023-10-14T16:57:44.284161+0000","last_unstale":"2023-10-14T16:57:46.187941+0000","last_undegraded":"2023-10-14T16:57:46.187941+0000","last_fullsized":"2023-10-14T16:57:46.187941+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:00.857230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.17","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152378+0000","last_change":"2023-10-14T16:57:45.196777+0000","last_active":"2023-10-14T16:57:47.152378+0000","last_peered":"2023-10-14T16:57:47.152378+0000","last_clean":"2023-10-14T16:57:47.152378+0000","last_became_active":"2023-10-14T16:57:45.196456+0000","last_became_peered":"2023-10-14T16:57:45.196456+0000","last_unstale":"2023-10-14T16:57:47.152378+0000","last_undegraded":"2023-10-14T16:57:47.152378+0000","last_fullsized":"2023-10-14T16:57:47.152378+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:38:50.125580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.16","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187538+0000","last_change":"2023-10-14T16:57:44.274582+0000","last_active":"2023-10-14T16:57:46.187538+0000","last_peered":"2023-10-14T16:57:46.187538+0000","last_clean":"2023-10-14T16:57:46.187538+0000","last_became_active":"2023-10-14T16:57:44.274305+0000","last_became_peered":"2023-10-14T16:57:44.274305+0000","last_unstale":"2023-10-14T16:57:46.187538+0000","last_undegraded":"2023-10-14T16:57:46.187538+0000","last_fullsized":"2023-10-14T16:57:46.187538+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:25.916883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.14","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182233+0000","last_change":"2023-10-14T16:57:45.204148+0000","last_active":"2023-10-14T16:57:46.182233+0000","last_peered":"2023-10-14T16:57:46.182233+0000","last_clean":"2023-10-14T16:57:46.182233+0000","last_became_active":"2023-10-14T16:57:45.199977+0000","last_became_peered":"2023-10-14T16:57:45.199977+0000","last_unstale":"2023-10-14T16:57:46.182233+0000","last_undegraded":"2023-10-14T16:57:46.182233+0000","last_fullsized":"2023-10-14T16:57:46.182233+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:06.647893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.15","version":"108'3","reported_seq":15,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186643+0000","last_change":"2023-10-14T16:57:44.272963+0000","last_active":"2023-10-14T16:57:46.186643+0000","last_peered":"2023-10-14T16:57:46.186643+0000","last_clean":"2023-10-14T16:57:46.186643+0000","last_became_active":"2023-10-14T16:57:44.272665+0000","last_became_peered":"2023-10-14T16:57:44.272665+0000","last_unstale":"2023-10-14T16:57:46.186643+0000","last_undegraded":"2023-10-14T16:57:46.186643+0000","last_fullsized":"2023-10-14T16:57:46.186643+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:32.674604+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.15","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187770+0000","last_change":"2023-10-14T16:57:45.211198+0000","last_active":"2023-10-14T16:57:46.187770+0000","last_peered":"2023-10-14T16:57:46.187770+0000","last_clean":"2023-10-14T16:57:46.187770+0000","last_became_active":"2023-10-14T16:57:45.210881+0000","last_became_peered":"2023-10-14T16:57:45.210881+0000","last_unstale":"2023-10-14T16:57:46.187770+0000","last_undegraded":"2023-10-14T16:57:46.187770+0000","last_fullsized":"2023-10-14T16:57:46.187770+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:19.759740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.14","version":"108'6","reported_seq":18,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151541+0000","last_change":"2023-10-14T16:57:44.282567+0000","last_active":"2023-10-14T16:57:47.151541+0000","last_peered":"2023-10-14T16:57:47.151541+0000","last_clean":"2023-10-14T16:57:47.151541+0000","last_became_active":"2023-10-14T16:57:44.282189+0000","last_became_peered":"2023-10-14T16:57:44.282189+0000","last_unstale":"2023-10-14T16:57:47.151541+0000","last_undegraded":"2023-10-14T16:57:47.151541+0000","last_fullsized":"2023-10-14T16:57:47.151541+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:39.048046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.12","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182370+0000","last_change":"2023-10-14T16:57:45.204341+0000","last_active":"2023-10-14T16:57:46.182370+0000","last_peered":"2023-10-14T16:57:46.182370+0000","last_clean":"2023-10-14T16:57:46.182370+0000","last_became_active":"2023-10-14T16:57:45.201951+0000","last_became_peered":"2023-10-14T16:57:45.201951+0000","last_unstale":"2023-10-14T16:57:46.182370+0000","last_undegraded":"2023-10-14T16:57:46.182370+0000","last_fullsized":"2023-10-14T16:57:46.182370+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:47.431209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.13","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507347+0000","last_change":"2023-10-14T16:57:44.271855+0000","last_active":"2023-10-14T16:57:46.507347+0000","last_peered":"2023-10-14T16:57:46.507347+0000","last_clean":"2023-10-14T16:57:46.507347+0000","last_became_active":"2023-10-14T16:57:44.270451+0000","last_became_peered":"2023-10-14T16:57:44.270451+0000","last_unstale":"2023-10-14T16:57:46.507347+0000","last_undegraded":"2023-10-14T16:57:46.507347+0000","last_fullsized":"2023-10-14T16:57:46.507347+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:02.657979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.13","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152978+0000","last_change":"2023-10-14T16:57:45.209362+0000","last_active":"2023-10-14T16:57:47.152978+0000","last_peered":"2023-10-14T16:57:47.152978+0000","last_clean":"2023-10-14T16:57:47.152978+0000","last_became_active":"2023-10-14T16:57:45.209069+0000","last_became_peered":"2023-10-14T16:57:45.209069+0000","last_unstale":"2023-10-14T16:57:47.152978+0000","last_undegraded":"2023-10-14T16:57:47.152978+0000","last_fullsized":"2023-10-14T16:57:47.152978+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:31:28.269227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.12","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153283+0000","last_change":"2023-10-14T16:57:44.282653+0000","last_active":"2023-10-14T16:57:47.153283+0000","last_peered":"2023-10-14T16:57:47.153283+0000","last_clean":"2023-10-14T16:57:47.153283+0000","last_became_active":"2023-10-14T16:57:44.281976+0000","last_became_peered":"2023-10-14T16:57:44.281976+0000","last_unstale":"2023-10-14T16:57:47.153283+0000","last_undegraded":"2023-10-14T16:57:47.153283+0000","last_fullsized":"2023-10-14T16:57:47.153283+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:30.851982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.10","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506748+0000","last_change":"2023-10-14T16:57:45.203150+0000","last_active":"2023-10-14T16:57:46.506748+0000","last_peered":"2023-10-14T16:57:46.506748+0000","last_clean":"2023-10-14T16:57:46.506748+0000","last_became_active":"2023-10-14T16:57:45.194972+0000","last_became_peered":"2023-10-14T16:57:45.194972+0000","last_unstale":"2023-10-14T16:57:46.506748+0000","last_undegraded":"2023-10-14T16:57:46.506748+0000","last_fullsized":"2023-10-14T16:57:46.506748+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:43.079137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.11","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151789+0000","last_change":"2023-10-14T16:57:44.271853+0000","last_active":"2023-10-14T16:57:47.151789+0000","last_peered":"2023-10-14T16:57:47.151789+0000","last_clean":"2023-10-14T16:57:47.151789+0000","last_became_active":"2023-10-14T16:57:44.271524+0000","last_became_peered":"2023-10-14T16:57:44.271524+0000","last_unstale":"2023-10-14T16:57:47.151789+0000","last_undegraded":"2023-10-14T16:57:47.151789+0000","last_fullsized":"2023-10-14T16:57:47.151789+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:18.746079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"13.11","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.506665+0000","last_change":"2023-10-14T16:57:45.198791+0000","last_active":"2023-10-14T16:57:46.506665+0000","last_peered":"2023-10-14T16:57:46.506665+0000","last_clean":"2023-10-14T16:57:46.506665+0000","last_became_active":"2023-10-14T16:57:45.198287+0000","last_became_peered":"2023-10-14T16:57:45.198287+0000","last_unstale":"2023-10-14T16:57:46.506665+0000","last_undegraded":"2023-10-14T16:57:46.506665+0000","last_fullsized":"2023-10-14T16:57:46.506665+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:04.729022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.10","version":"108'8","reported_seq":20,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.181489+0000","last_change":"2023-10-14T16:57:44.282785+0000","last_active":"2023-10-14T16:57:46.181489+0000","last_peered":"2023-10-14T16:57:46.181489+0000","last_clean":"2023-10-14T16:57:46.181489+0000","last_became_active":"2023-10-14T16:57:44.282373+0000","last_became_peered":"2023-10-14T16:57:44.282373+0000","last_unstale":"2023-10-14T16:57:46.181489+0000","last_undegraded":"2023-10-14T16:57:46.181489+0000","last_fullsized":"2023-10-14T16:57:46.181489+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:36.357674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.e","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153300+0000","last_change":"2023-10-14T16:57:45.197142+0000","last_active":"2023-10-14T16:57:47.153300+0000","last_peered":"2023-10-14T16:57:47.153300+0000","last_clean":"2023-10-14T16:57:47.153300+0000","last_became_active":"2023-10-14T16:57:45.196851+0000","last_became_peered":"2023-10-14T16:57:45.196851+0000","last_unstale":"2023-10-14T16:57:47.153300+0000","last_undegraded":"2023-10-14T16:57:47.153300+0000","last_fullsized":"2023-10-14T16:57:47.153300+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:17.465365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.f","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188199+0000","last_change":"2023-10-14T16:57:44.272494+0000","last_active":"2023-10-14T16:57:46.188199+0000","last_peered":"2023-10-14T16:57:46.188199+0000","last_clean":"2023-10-14T16:57:46.188199+0000","last_became_active":"2023-10-14T16:57:44.271104+0000","last_became_peered":"2023-10-14T16:57:44.271104+0000","last_unstale":"2023-10-14T16:57:46.188199+0000","last_undegraded":"2023-10-14T16:57:46.188199+0000","last_fullsized":"2023-10-14T16:57:46.188199+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:51:45.606476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.f","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187663+0000","last_change":"2023-10-14T16:57:45.196109+0000","last_active":"2023-10-14T16:57:46.187663+0000","last_peered":"2023-10-14T16:57:46.187663+0000","last_clean":"2023-10-14T16:57:46.187663+0000","last_became_active":"2023-10-14T16:57:45.195801+0000","last_became_peered":"2023-10-14T16:57:45.195801+0000","last_unstale":"2023-10-14T16:57:46.187663+0000","last_undegraded":"2023-10-14T16:57:46.187663+0000","last_fullsized":"2023-10-14T16:57:46.187663+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:54.024111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.e","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.183426+0000","last_change":"2023-10-14T16:57:44.269395+0000","last_active":"2023-10-14T16:57:46.183426+0000","last_peered":"2023-10-14T16:57:46.183426+0000","last_clean":"2023-10-14T16:57:46.183426+0000","last_became_active":"2023-10-14T16:57:44.268913+0000","last_became_peered":"2023-10-14T16:57:44.268913+0000","last_unstale":"2023-10-14T16:57:46.183426+0000","last_undegraded":"2023-10-14T16:57:46.183426+0000","last_fullsized":"2023-10-14T16:57:46.183426+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:53.042671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.d","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187938+0000","last_change":"2023-10-14T16:57:45.207875+0000","last_active":"2023-10-14T16:57:46.187938+0000","last_peered":"2023-10-14T16:57:46.187938+0000","last_clean":"2023-10-14T16:57:46.187938+0000","last_became_active":"2023-10-14T16:57:45.207505+0000","last_became_peered":"2023-10-14T16:57:45.207505+0000","last_unstale":"2023-10-14T16:57:46.187938+0000","last_undegraded":"2023-10-14T16:57:46.187938+0000","last_fullsized":"2023-10-14T16:57:46.187938+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:35:40.061706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.c","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182404+0000","last_change":"2023-10-14T16:57:44.273166+0000","last_active":"2023-10-14T16:57:46.182404+0000","last_peered":"2023-10-14T16:57:46.182404+0000","last_clean":"2023-10-14T16:57:46.182404+0000","last_became_active":"2023-10-14T16:57:44.272862+0000","last_became_peered":"2023-10-14T16:57:44.272862+0000","last_unstale":"2023-10-14T16:57:46.182404+0000","last_undegraded":"2023-10-14T16:57:46.182404+0000","last_fullsized":"2023-10-14T16:57:46.182404+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:02.917979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.a","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151662+0000","last_change":"2023-10-14T16:57:45.201634+0000","last_active":"2023-10-14T16:57:47.151662+0000","last_peered":"2023-10-14T16:57:47.151662+0000","last_clean":"2023-10-14T16:57:47.151662+0000","last_became_active":"2023-10-14T16:57:45.201093+0000","last_became_peered":"2023-10-14T16:57:45.201093+0000","last_unstale":"2023-10-14T16:57:47.151662+0000","last_undegraded":"2023-10-14T16:57:47.151662+0000","last_fullsized":"2023-10-14T16:57:47.151662+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:26.930544+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.b","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152905+0000","last_change":"2023-10-14T16:57:44.282377+0000","last_active":"2023-10-14T16:57:47.152905+0000","last_peered":"2023-10-14T16:57:47.152905+0000","last_clean":"2023-10-14T16:57:47.152905+0000","last_became_active":"2023-10-14T16:57:44.282075+0000","last_became_peered":"2023-10-14T16:57:44.282075+0000","last_unstale":"2023-10-14T16:57:47.152905+0000","last_undegraded":"2023-10-14T16:57:47.152905+0000","last_fullsized":"2023-10-14T16:57:47.152905+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:02.609380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.b","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151309+0000","last_change":"2023-10-14T16:57:45.198872+0000","last_active":"2023-10-14T16:57:47.151309+0000","last_peered":"2023-10-14T16:57:47.151309+0000","last_clean":"2023-10-14T16:57:47.151309+0000","last_became_active":"2023-10-14T16:57:45.198575+0000","last_became_peered":"2023-10-14T16:57:45.198575+0000","last_unstale":"2023-10-14T16:57:47.151309+0000","last_undegraded":"2023-10-14T16:57:47.151309+0000","last_fullsized":"2023-10-14T16:57:47.151309+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:31:52.642670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.a","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.152353+0000","last_change":"2023-10-14T16:57:44.276898+0000","last_active":"2023-10-14T16:57:47.152353+0000","last_peered":"2023-10-14T16:57:47.152353+0000","last_clean":"2023-10-14T16:57:47.152353+0000","last_became_active":"2023-10-14T16:57:44.275527+0000","last_became_peered":"2023-10-14T16:57:44.275527+0000","last_unstale":"2023-10-14T16:57:47.152353+0000","last_undegraded":"2023-10-14T16:57:47.152353+0000","last_fullsized":"2023-10-14T16:57:47.152353+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:46.733073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.8","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151279+0000","last_change":"2023-10-14T16:57:45.197567+0000","last_active":"2023-10-14T16:57:47.151279+0000","last_peered":"2023-10-14T16:57:47.151279+0000","last_clean":"2023-10-14T16:57:47.151279+0000","last_became_active":"2023-10-14T16:57:45.197296+0000","last_became_peered":"2023-10-14T16:57:45.197296+0000","last_unstale":"2023-10-14T16:57:47.151279+0000","last_undegraded":"2023-10-14T16:57:47.151279+0000","last_fullsized":"2023-10-14T16:57:47.151279+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:59.230753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"12.9","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151685+0000","last_change":"2023-10-14T16:57:44.278662+0000","last_active":"2023-10-14T16:57:47.151685+0000","last_peered":"2023-10-14T16:57:47.151685+0000","last_clean":"2023-10-14T16:57:47.151685+0000","last_became_active":"2023-10-14T16:57:44.277548+0000","last_became_peered":"2023-10-14T16:57:44.277548+0000","last_unstale":"2023-10-14T16:57:47.151685+0000","last_undegraded":"2023-10-14T16:57:47.151685+0000","last_fullsized":"2023-10-14T16:57:47.151685+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:25.036463+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"13.9","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151520+0000","last_change":"2023-10-14T16:57:45.203076+0000","last_active":"2023-10-14T16:57:47.151520+0000","last_peered":"2023-10-14T16:57:47.151520+0000","last_clean":"2023-10-14T16:57:47.151520+0000","last_became_active":"2023-10-14T16:57:45.202775+0000","last_became_peered":"2023-10-14T16:57:45.202775+0000","last_unstale":"2023-10-14T16:57:47.151520+0000","last_undegraded":"2023-10-14T16:57:47.151520+0000","last_fullsized":"2023-10-14T16:57:47.151520+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:23:37.544968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.8","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.150875+0000","last_change":"2023-10-14T16:57:44.277664+0000","last_active":"2023-10-14T16:57:47.150875+0000","last_peered":"2023-10-14T16:57:47.150875+0000","last_clean":"2023-10-14T16:57:47.150875+0000","last_became_active":"2023-10-14T16:57:44.277403+0000","last_became_peered":"2023-10-14T16:57:44.277403+0000","last_unstale":"2023-10-14T16:57:47.150875+0000","last_undegraded":"2023-10-14T16:57:47.150875+0000","last_fullsized":"2023-10-14T16:57:47.150875+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:06.785421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"13.6","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188026+0000","last_change":"2023-10-14T16:57:45.185076+0000","last_active":"2023-10-14T16:57:46.188026+0000","last_peered":"2023-10-14T16:57:46.188026+0000","last_clean":"2023-10-14T16:57:46.188026+0000","last_became_active":"2023-10-14T16:57:45.184616+0000","last_became_peered":"2023-10-14T16:57:45.184616+0000","last_unstale":"2023-10-14T16:57:46.188026+0000","last_undegraded":"2023-10-14T16:57:46.188026+0000","last_fullsized":"2023-10-14T16:57:46.188026+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:40.641312+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.7","version":"108'2","reported_seq":14,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507914+0000","last_change":"2023-10-14T16:57:44.281564+0000","last_active":"2023-10-14T16:57:46.507914+0000","last_peered":"2023-10-14T16:57:46.507914+0000","last_clean":"2023-10-14T16:57:46.507914+0000","last_became_active":"2023-10-14T16:57:44.281219+0000","last_became_peered":"2023-10-14T16:57:44.281219+0000","last_unstale":"2023-10-14T16:57:46.507914+0000","last_undegraded":"2023-10-14T16:57:46.507914+0000","last_fullsized":"2023-10-14T16:57:46.507914+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:10.822276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.7","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153345+0000","last_change":"2023-10-14T16:57:45.201953+0000","last_active":"2023-10-14T16:57:47.153345+0000","last_peered":"2023-10-14T16:57:47.153345+0000","last_clean":"2023-10-14T16:57:47.153345+0000","last_became_active":"2023-10-14T16:57:45.201670+0000","last_became_peered":"2023-10-14T16:57:45.201670+0000","last_unstale":"2023-10-14T16:57:47.153345+0000","last_undegraded":"2023-10-14T16:57:47.153345+0000","last_fullsized":"2023-10-14T16:57:47.153345+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:25.739658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.6","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507783+0000","last_change":"2023-10-14T16:57:44.271652+0000","last_active":"2023-10-14T16:57:46.507783+0000","last_peered":"2023-10-14T16:57:46.507783+0000","last_clean":"2023-10-14T16:57:46.507783+0000","last_became_active":"2023-10-14T16:57:44.270642+0000","last_became_peered":"2023-10-14T16:57:44.270642+0000","last_unstale":"2023-10-14T16:57:46.507783+0000","last_undegraded":"2023-10-14T16:57:46.507783+0000","last_fullsized":"2023-10-14T16:57:46.507783+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:06.326274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"13.4","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186973+0000","last_change":"2023-10-14T16:57:45.194622+0000","last_active":"2023-10-14T16:57:46.186973+0000","last_peered":"2023-10-14T16:57:46.186973+0000","last_clean":"2023-10-14T16:57:46.186973+0000","last_became_active":"2023-10-14T16:57:45.194323+0000","last_became_peered":"2023-10-14T16:57:45.194323+0000","last_unstale":"2023-10-14T16:57:46.186973+0000","last_undegraded":"2023-10-14T16:57:46.186973+0000","last_fullsized":"2023-10-14T16:57:46.186973+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:04.729022+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.5","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186901+0000","last_change":"2023-10-14T16:57:44.275895+0000","last_active":"2023-10-14T16:57:46.186901+0000","last_peered":"2023-10-14T16:57:46.186901+0000","last_clean":"2023-10-14T16:57:46.186901+0000","last_became_active":"2023-10-14T16:57:44.275628+0000","last_became_peered":"2023-10-14T16:57:44.275628+0000","last_unstale":"2023-10-14T16:57:46.186901+0000","last_undegraded":"2023-10-14T16:57:46.186901+0000","last_fullsized":"2023-10-14T16:57:46.186901+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:05.634232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.5","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.188131+0000","last_change":"2023-10-14T16:57:45.200998+0000","last_active":"2023-10-14T16:57:46.188131+0000","last_peered":"2023-10-14T16:57:46.188131+0000","last_clean":"2023-10-14T16:57:46.188131+0000","last_became_active":"2023-10-14T16:57:45.200677+0000","last_became_peered":"2023-10-14T16:57:45.200677+0000","last_unstale":"2023-10-14T16:57:46.188131+0000","last_undegraded":"2023-10-14T16:57:46.188131+0000","last_fullsized":"2023-10-14T16:57:46.188131+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:44.587697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"12.4","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.187142+0000","last_change":"2023-10-14T16:57:44.280505+0000","last_active":"2023-10-14T16:57:46.187142+0000","last_peered":"2023-10-14T16:57:46.187142+0000","last_clean":"2023-10-14T16:57:46.187142+0000","last_became_active":"2023-10-14T16:57:44.280158+0000","last_became_peered":"2023-10-14T16:57:44.280158+0000","last_unstale":"2023-10-14T16:57:46.187142+0000","last_undegraded":"2023-10-14T16:57:46.187142+0000","last_fullsized":"2023-10-14T16:57:46.187142+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:53:02.917979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.2","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.151747+0000","last_change":"2023-10-14T16:57:45.193081+0000","last_active":"2023-10-14T16:57:47.151747+0000","last_peered":"2023-10-14T16:57:47.151747+0000","last_clean":"2023-10-14T16:57:47.151747+0000","last_became_active":"2023-10-14T16:57:45.192763+0000","last_became_peered":"2023-10-14T16:57:45.192763+0000","last_unstale":"2023-10-14T16:57:47.151747+0000","last_undegraded":"2023-10-14T16:57:47.151747+0000","last_fullsized":"2023-10-14T16:57:47.151747+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:58:20.691373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"12.3","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182775+0000","last_change":"2023-10-14T16:57:44.283925+0000","last_active":"2023-10-14T16:57:46.182775+0000","last_peered":"2023-10-14T16:57:46.182775+0000","last_clean":"2023-10-14T16:57:46.182775+0000","last_became_active":"2023-10-14T16:57:44.283598+0000","last_became_peered":"2023-10-14T16:57:44.283598+0000","last_unstale":"2023-10-14T16:57:46.182775+0000","last_undegraded":"2023-10-14T16:57:46.182775+0000","last_fullsized":"2023-10-14T16:57:46.182775+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:24:56.451526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"13.3","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.182546+0000","last_change":"2023-10-14T16:57:45.203420+0000","last_active":"2023-10-14T16:57:46.182546+0000","last_peered":"2023-10-14T16:57:46.182546+0000","last_clean":"2023-10-14T16:57:46.182546+0000","last_became_active":"2023-10-14T16:57:45.203102+0000","last_became_peered":"2023-10-14T16:57:45.203102+0000","last_unstale":"2023-10-14T16:57:46.182546+0000","last_undegraded":"2023-10-14T16:57:46.182546+0000","last_fullsized":"2023-10-14T16:57:46.182546+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:41:01.870891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"12.2","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186227+0000","last_change":"2023-10-14T16:57:44.284606+0000","last_active":"2023-10-14T16:57:46.186227+0000","last_peered":"2023-10-14T16:57:46.186227+0000","last_clean":"2023-10-14T16:57:46.186227+0000","last_became_active":"2023-10-14T16:57:44.284297+0000","last_became_peered":"2023-10-14T16:57:44.284297+0000","last_unstale":"2023-10-14T16:57:46.186227+0000","last_undegraded":"2023-10-14T16:57:46.186227+0000","last_fullsized":"2023-10-14T16:57:46.186227+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:11.272769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.0","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153267+0000","last_change":"2023-10-14T16:57:45.207861+0000","last_active":"2023-10-14T16:57:47.153267+0000","last_peered":"2023-10-14T16:57:47.153267+0000","last_clean":"2023-10-14T16:57:47.153267+0000","last_became_active":"2023-10-14T16:57:45.207033+0000","last_became_peered":"2023-10-14T16:57:45.207033+0000","last_unstale":"2023-10-14T16:57:47.153267+0000","last_undegraded":"2023-10-14T16:57:47.153267+0000","last_fullsized":"2023-10-14T16:57:47.153267+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:36:09.824679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"12.1","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186416+0000","last_change":"2023-10-14T16:57:44.277999+0000","last_active":"2023-10-14T16:57:46.186416+0000","last_peered":"2023-10-14T16:57:46.186416+0000","last_clean":"2023-10-14T16:57:46.186416+0000","last_became_active":"2023-10-14T16:57:44.277726+0000","last_became_peered":"2023-10-14T16:57:44.277726+0000","last_unstale":"2023-10-14T16:57:46.186416+0000","last_undegraded":"2023-10-14T16:57:46.186416+0000","last_fullsized":"2023-10-14T16:57:46.186416+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:58.217093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"13.1","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.186327+0000","last_change":"2023-10-14T16:57:45.199102+0000","last_active":"2023-10-14T16:57:46.186327+0000","last_peered":"2023-10-14T16:57:46.186327+0000","last_clean":"2023-10-14T16:57:46.186327+0000","last_became_active":"2023-10-14T16:57:45.197574+0000","last_became_peered":"2023-10-14T16:57:45.197574+0000","last_unstale":"2023-10-14T16:57:46.186327+0000","last_undegraded":"2023-10-14T16:57:46.186327+0000","last_fullsized":"2023-10-14T16:57:46.186327+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:07.339934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"12.0","version":"0'0","reported_seq":12,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.185017+0000","last_change":"2023-10-14T16:57:44.274937+0000","last_active":"2023-10-14T16:57:46.185017+0000","last_peered":"2023-10-14T16:57:46.185017+0000","last_clean":"2023-10-14T16:57:46.185017+0000","last_became_active":"2023-10-14T16:57:44.274590+0000","last_became_peered":"2023-10-14T16:57:44.274590+0000","last_unstale":"2023-10-14T16:57:46.185017+0000","last_undegraded":"2023-10-14T16:57:46.185017+0000","last_fullsized":"2023-10-14T16:57:46.185017+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:37:48.339806+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"13.c","version":"0'0","reported_seq":11,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:46.507088+0000","last_change":"2023-10-14T16:57:45.204441+0000","last_active":"2023-10-14T16:57:46.507088+0000","last_peered":"2023-10-14T16:57:46.507088+0000","last_clean":"2023-10-14T16:57:46.507088+0000","last_became_active":"2023-10-14T16:57:45.204130+0000","last_became_peered":"2023-10-14T16:57:45.204130+0000","last_unstale":"2023-10-14T16:57:46.507088+0000","last_undegraded":"2023-10-14T16:57:46.507088+0000","last_fullsized":"2023-10-14T16:57:46.507088+0000","mapping_epoch":107,"log_start":"0'0","ondisk_log_start":"0'0","created":107,"last_epoch_clean":108,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:44.237727+0000","last_clean_scrub_stamp":"2023-10-14T16:57:44.237727+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:34:45.843694+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"12.d","version":"108'1","reported_seq":13,"reported_epoch":109,"state":"active+clean","last_fresh":"2023-10-14T16:57:47.153656+0000","last_change":"2023-10-14T16:57:44.283509+0000","last_active":"2023-10-14T16:57:47.153656+0000","last_peered":"2023-10-14T16:57:47.153656+0000","last_clean":"2023-10-14T16:57:47.153656+0000","last_became_active":"2023-10-14T16:57:44.283100+0000","last_became_peered":"2023-10-14T16:57:44.283100+0000","last_unstale":"2023-10-14T16:57:47.153656+0000","last_undegraded":"2023-10-14T16:57:47.153656+0000","last_fullsized":"2023-10-14T16:57:47.153656+0000","mapping_epoch":106,"log_start":"0'0","ondisk_log_start":"0'0","created":106,"last_epoch_clean":107,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:57:43.224066+0000","last_clean_scrub_stamp":"2023-10-14T16:57:43.224066+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:38:06.785421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"98'183","reported_seq":481,"reported_epoch":108,"state":"active+clean","last_fresh":"2023-10-14T16:57:45.187796+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:57:45.187796+0000","last_peered":"2023-10-14T16:57:45.187796+0000","last_clean":"2023-10-14T16:57:45.187796+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:57:45.187796+0000","last_undegraded":"2023-10-14T16:57:45.187796+0000","last_fullsized":"2023-10-14T16:57:45.187796+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":13,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":12,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":692,"num_read_kb":1336,"num_write":279,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":183,"ondisk_log_size":183,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575058,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37240,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334600,"statfs":{"total":96636764160,"available":96598630400,"internally_reserved":0,"allocated":1937408,"data_stored":1234264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.169,"15min":1.1699999999999999},"min":{"1min":0.66000000000000003,"5min":0.56599999999999995,"15min":0.51500000000000001},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":1.6259999999999999},{"interface":"front","average":{"1min":1.018,"5min":1.107,"15min":1.1910000000000001},"min":{"1min":0.73299999999999998,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.6279999999999999,"5min":1.8400000000000001,"15min":2.117},"last":1.5680000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0600000000000001,"5min":1.1890000000000001,"15min":1.2230000000000001},"min":{"1min":0.73799999999999999,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.3560000000000001,"5min":1.7809999999999999,"15min":2.2309999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.236,"15min":1.2569999999999999},"min":{"1min":0.66700000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.722,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.3460000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1910000000000001,"15min":1.224},"min":{"1min":0.623,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":2.0950000000000002,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.27},{"interface":"front","average":{"1min":1.214,"5min":1.2629999999999999,"15min":1.3069999999999999},"min":{"1min":0.78900000000000003,"5min":0.72599999999999998,"15min":0.63300000000000001},"max":{"1min":1.7689999999999999,"5min":1.7689999999999999,"15min":2.1589999999999998},"last":1.5249999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2150000000000001,"5min":1.3500000000000001,"15min":1.369},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.996,"5min":1.996,"15min":2.4159999999999999},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.4259999999999999,"15min":1.446},"min":{"1min":0.85599999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":2.0009999999999999,"5min":2.0009999999999999,"15min":2.0880000000000001},"last":1.4319999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.2749999999999999,"15min":1.268},"min":{"1min":0.52600000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.881,"5min":2.105,"15min":2.1819999999999999},"last":1.772},{"interface":"front","average":{"1min":1.3149999999999999,"5min":1.3240000000000001,"15min":1.3240000000000001},"min":{"1min":0.61499999999999999,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.1659999999999999,"5min":2.1659999999999999,"15min":2.1659999999999999},"last":1.603}]},{"osd":5,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.3540000000000001,"15min":1.385},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9159999999999999,"5min":1.9159999999999999,"15min":2.0390000000000001},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.4450000000000001,"15min":1.4590000000000001},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.677}]},{"osd":6,"last update":"Sat Oct 14 16:56:58 2023","interfaces":[{"interface":"back","average":{"1min":1.5669999999999999,"5min":1.532,"15min":1.542},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.492,"5min":2.492,"15min":2.492},"last":1.8500000000000001},{"interface":"front","average":{"1min":1.587,"5min":1.4790000000000001,"15min":1.4830000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":4.4580000000000002,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.484}]}]},{"osd":6,"up_from":13,"seq":55834575057,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36660,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335180,"statfs":{"total":96636764160,"available":96599224320,"internally_reserved":0,"allocated":1351680,"data_stored":644520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.1519999999999999,"15min":1.153},"min":{"1min":0.81799999999999995,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.524,"5min":2.137,"15min":2.137},"last":1.413},{"interface":"front","average":{"1min":1.141,"5min":1.222,"15min":1.206},"min":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.8100000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.2669999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.095,"5min":1.1779999999999999,"15min":1.1850000000000001},"min":{"1min":0.77000000000000002,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.593,"5min":2.802,"15min":2.802},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.282,"15min":1.28},"min":{"1min":0.67300000000000004,"5min":0.438,"15min":0.438},"max":{"1min":1.395,"5min":2.512,"15min":2.512},"last":1.0589999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.351,"15min":1.3540000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.75900000000000001},"max":{"1min":1.5629999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":1.5309999999999999},{"interface":"front","average":{"1min":1.202,"5min":1.3680000000000001,"15min":1.385},"min":{"1min":0.81299999999999994,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":1.5549999999999999,"5min":3.286,"15min":3.286},"last":1.4490000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.363,"15min":1.3660000000000001},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.577,"5min":2.823,"15min":2.823},"last":1.3},{"interface":"front","average":{"1min":1.3109999999999999,"5min":1.4410000000000001,"15min":1.415},"min":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.464}]},{"osd":4,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3500000000000001,"5min":1.369,"15min":1.3400000000000001},"min":{"1min":0.497,"5min":0.497,"15min":0.497},"max":{"1min":1.74,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.3919999999999999,"5min":1.4790000000000001,"15min":1.4119999999999999},"min":{"1min":0.68799999999999994,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8049999999999999,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.135}]},{"osd":5,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.3160000000000001,"5min":1.3999999999999999,"15min":1.3740000000000001},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7609999999999999,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.238},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.5740000000000001,"15min":1.5389999999999999},"min":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.8999999999999999,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.4910000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:50 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.6299999999999999,"15min":1.5760000000000001},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.992,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.365},{"interface":"front","average":{"1min":1.474,"5min":1.5189999999999999,"15min":1.532},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.9670000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.3919999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575058,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32512,"kb_used_data":1324,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339328,"statfs":{"total":96636764160,"available":96603471872,"internally_reserved":0,"allocated":1355776,"data_stored":645460,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.1100000000000001,"15min":1.1120000000000001},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5409999999999999,"5min":1.861,"15min":1.986},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.127,"5min":1.1060000000000001,"15min":1.129},"min":{"1min":0.69499999999999995,"5min":0.68999999999999995,"15min":0.60599999999999998},"max":{"1min":1.6599999999999999,"5min":1.665,"15min":2.234},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.1439999999999999,"15min":1.167},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.6100000000000001,"5min":1.7090000000000001,"15min":2.1200000000000001},"last":0.753},{"interface":"front","average":{"1min":1.0700000000000001,"5min":1.1830000000000001,"15min":1.216},"min":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7070000000000001,"5min":1.7609999999999999,"15min":1.905},"last":1.133}]},{"osd":2,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0900000000000001,"5min":1.1739999999999999,"15min":1.1879999999999999},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5760000000000001,"5min":1.7150000000000001,"15min":2.375},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3240000000000001,"15min":1.3129999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.577,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":0.81799999999999995}]},{"osd":3,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.292,"5min":1.321,"15min":1.3839999999999999},"min":{"1min":0.77200000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.72,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.083},{"interface":"front","average":{"1min":1.298,"5min":1.375,"15min":1.3999999999999999},"min":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.732,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":0.71099999999999997}]},{"osd":4,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.272,"15min":1.284},"min":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":1.6699999999999999,"5min":2.194,"15min":2.4889999999999999},"last":0.434},{"interface":"front","average":{"1min":1.3049999999999999,"5min":1.335,"15min":1.343},"min":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.9119999999999999,"5min":1.927,"15min":2.5169999999999999},"last":1.173}]},{"osd":6,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3799999999999999,"15min":1.3700000000000001},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.302},"last":0.879},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5229999999999999,"15min":1.5349999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.758,"5min":2.1059999999999999,"15min":2.613},"last":0.65500000000000003}]},{"osd":7,"last update":"Sat Oct 14 16:56:53 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.478,"15min":1.5049999999999999},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.806,"5min":2.032,"15min":2.605},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.4490000000000001,"5min":1.522,"15min":1.5109999999999999},"min":{"1min":0.749,"5min":0.749,"15min":0.749},"max":{"1min":1.8280000000000001,"5min":2.3980000000000001,"15min":2.3980000000000001},"last":1.04}]}]},{"osd":4,"up_from":13,"seq":55834575058,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58856,"kb_used_data":1308,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312984,"statfs":{"total":96636764160,"available":96576495616,"internally_reserved":0,"allocated":1339392,"data_stored":643826,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.127,"15min":1.149},"min":{"1min":0.623,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.486,"5min":1.7050000000000001,"15min":1.982},"last":1.9319999999999999},{"interface":"front","average":{"1min":1.2210000000000001,"5min":1.2390000000000001,"15min":1.23},"min":{"1min":0.71199999999999997,"5min":0.66000000000000003,"15min":0.66000000000000003},"max":{"1min":1.704,"5min":1.972,"15min":2.1099999999999999},"last":4.5979999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.208,"5min":1.2470000000000001,"15min":1.2529999999999999},"min":{"1min":0.753,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7490000000000001,"5min":2.0169999999999999,"15min":2.8639999999999999},"last":6.7880000000000003},{"interface":"front","average":{"1min":1.0740000000000001,"5min":1.206,"15min":1.24},"min":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.478,"5min":2.3620000000000001,"15min":2.3620000000000001},"last":5.9530000000000003}]},{"osd":2,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.2250000000000001,"15min":1.226},"min":{"1min":0.73099999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.752,"5min":2.169,"15min":2.169},"last":5.633},{"interface":"front","average":{"1min":1.319,"5min":1.3799999999999999,"15min":1.3740000000000001},"min":{"1min":0.83399999999999996,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.0209999999999999,"5min":2.0209999999999999,"15min":2.7759999999999998},"last":3.5070000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.2070000000000001,"5min":1.2669999999999999,"15min":1.3100000000000001},"min":{"1min":0.77100000000000002,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6819999999999999,"5min":2.1099999999999999,"15min":2.1099999999999999},"last":4.6349999999999998},{"interface":"front","average":{"1min":1.361,"5min":1.359,"15min":1.3959999999999999},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.77800000000000002},"max":{"1min":1.8440000000000001,"5min":1.952,"15min":2.085},"last":3.5590000000000002}]},{"osd":5,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3089999999999999,"5min":1.361,"15min":1.3520000000000001},"min":{"1min":0.64500000000000002,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":2.931},{"interface":"front","average":{"1min":1.329,"5min":1.3080000000000001,"15min":1.343},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.9690000000000001,"5min":1.9690000000000001,"15min":2.153},"last":3.2599999999999998}]},{"osd":6,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.3640000000000001,"5min":1.3779999999999999,"15min":1.389},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9139999999999999,"5min":2.1539999999999999,"15min":2.1930000000000001},"last":2.323},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.504,"15min":1.526},"min":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.0960000000000001,"5min":2.214,"15min":2.6539999999999999},"last":2.9889999999999999}]},{"osd":7,"last update":"Sat Oct 14 16:56:48 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4730000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.0379999999999998,"5min":2.0379999999999998,"15min":2.2389999999999999},"last":4.7960000000000003},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.528,"15min":1.5469999999999999},"min":{"1min":0.82899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.883}]}]},{"osd":3,"up_from":13,"seq":55834575058,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":40820,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94331020,"statfs":{"total":96636764160,"available":96594964480,"internally_reserved":0,"allocated":1351680,"data_stored":645438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.80600000000000005,"5min":0.879,"15min":0.92600000000000005},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.2270000000000001,"5min":1.621,"15min":1.621},"last":1.1220000000000001},{"interface":"front","average":{"1min":0.94899999999999995,"5min":1.0449999999999999,"15min":1.1319999999999999},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.377,"5min":1.593,"15min":1.8799999999999999},"last":1.181}]},{"osd":1,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":0.82599999999999996,"5min":0.83299999999999996,"15min":0.91100000000000003},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.234,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.73599999999999999},{"interface":"front","average":{"1min":1.022,"5min":1.026,"15min":1.109},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.772,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":0.98199999999999998}]},{"osd":2,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.2370000000000001,"15min":1.365},"min":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.024,"5min":2.024,"15min":2.0609999999999999},"last":0.94799999999999995},{"interface":"front","average":{"1min":1.1859999999999999,"5min":1.1719999999999999,"15min":1.2829999999999999},"min":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.8240000000000001,"5min":1.8560000000000001,"15min":2.0960000000000001},"last":1.3380000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3240000000000001,"15min":1.365},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8859999999999999,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.069},{"interface":"front","average":{"1min":1.25,"5min":1.3540000000000001,"15min":1.407},"min":{"1min":0.745,"5min":0.745,"15min":0.745},"max":{"1min":1.802,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.5880000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.2809999999999999,"5min":1.319,"15min":1.3939999999999999},"min":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.895,"5min":2.516,"15min":3.0190000000000001},"last":1.448},{"interface":"front","average":{"1min":1.3260000000000001,"5min":1.389,"15min":1.4390000000000001},"min":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.8879999999999999,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.298}]},{"osd":6,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.4159999999999999,"15min":1.4690000000000001},"min":{"1min":0.80500000000000005,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":1.911,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.488},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.4950000000000001,"15min":1.5389999999999999},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9810000000000001,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.377}]},{"osd":7,"last update":"Sat Oct 14 16:57:42 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4399999999999999,"15min":1.5089999999999999},"min":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.093,"5min":2.871,"15min":2.875},"last":1.5609999999999999},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.5049999999999999,"15min":1.554},"min":{"1min":0.88200000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":2.2330000000000001,"5min":2.907,"15min":2.907},"last":1.4039999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575058,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44980,"kb_used_data":1312,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326860,"statfs":{"total":96636764160,"available":96590704640,"internally_reserved":0,"allocated":1343488,"data_stored":644408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.85199999999999998,"15min":0.90400000000000003},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":1.706,"5min":1.706,"15min":1.849},"last":1.101},{"interface":"front","average":{"1min":0.91300000000000003,"5min":1.012,"15min":1.044},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.4990000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.91000000000000003}]},{"osd":1,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":0.89900000000000002,"5min":0.93400000000000005,"15min":0.98199999999999998},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":1.4430000000000001,"5min":1.4430000000000001,"15min":2.0419999999999998},"last":0.83299999999999996},{"interface":"front","average":{"1min":0.98699999999999999,"5min":1.165,"15min":1.175},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":1.74,"5min":2.0390000000000001,"15min":2.069},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.0760000000000001,"5min":1.1539999999999999,"15min":1.23},"min":{"1min":0.54500000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.6930000000000001,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.96099999999999997},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.242,"15min":1.3149999999999999},"min":{"1min":0.58699999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.52,"5min":2.0459999999999998,"15min":2.0459999999999998},"last":1.4530000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.361,"15min":1.4199999999999999},"min":{"1min":0.75,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.5589999999999999,"5min":1.95,"15min":2.3159999999999998},"last":1.534},{"interface":"front","average":{"1min":1.206,"5min":1.327,"15min":1.3660000000000001},"min":{"1min":0.83399999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.0129999999999999,"5min":2.0750000000000002,"15min":2.0750000000000002},"last":1.381}]},{"osd":5,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.363,"15min":1.399},"min":{"1min":0.73999999999999999,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":2.173,"5min":2.173,"15min":2.173},"last":1.0229999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.472,"15min":1.53},"min":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.7849999999999999,"5min":2.2730000000000001,"15min":2.4689999999999999},"last":1.3129999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.395,"15min":1.4490000000000001},"min":{"1min":0.82399999999999995,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.958,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.5660000000000001},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.474,"15min":1.526},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":2.1629999999999998},"last":1.4930000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:56:49 2023","interfaces":[{"interface":"back","average":{"1min":1.3799999999999999,"5min":1.472,"15min":1.5349999999999999},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2.0129999999999999,"5min":2.109,"15min":2.109},"last":1.6439999999999999},{"interface":"front","average":{"1min":1.423,"5min":1.5329999999999999,"15min":1.5920000000000001},"min":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.27}]}]},{"osd":1,"up_from":13,"seq":55834575058,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58872,"kb_used_data":1320,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312968,"statfs":{"total":96636764160,"available":96576479232,"internally_reserved":0,"allocated":1351680,"data_stored":644020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.81999999999999995,"5min":0.85599999999999998,"15min":0.90500000000000003},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.708,"5min":1.8879999999999999,"15min":1.8879999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.86899999999999999,"5min":0.996,"15min":1.0669999999999999},"min":{"1min":0.51500000000000001,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":1.5549999999999999,"5min":1.728,"15min":2.0819999999999999},"last":0.80100000000000005}]},{"osd":2,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":0.89300000000000002,"5min":0.92900000000000005,"15min":0.96199999999999997},"min":{"1min":0.42999999999999999,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":1.7509999999999999,"5min":1.7509999999999999,"15min":1.758},"last":1.2709999999999999},{"interface":"front","average":{"1min":0.97399999999999998,"5min":1.0780000000000001,"15min":1.165},"min":{"1min":0.59899999999999998,"5min":0.51300000000000001,"15min":0.51300000000000001},"max":{"1min":1.556,"5min":2.0009999999999999,"15min":2.0150000000000001},"last":0.95999999999999996}]},{"osd":3,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.1919999999999999,"15min":1.2829999999999999},"min":{"1min":0.54000000000000004,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.724,"5min":2.1549999999999998,"15min":2.198},"last":1.637},{"interface":"front","average":{"1min":0.94199999999999995,"5min":1.099,"15min":1.1839999999999999},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.421,"5min":1.802,"15min":2.004},"last":1.1319999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.361,"15min":1.4059999999999999},"min":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"max":{"1min":1.575,"5min":1.9470000000000001,"15min":2.3050000000000002},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.2609999999999999,"15min":1.3080000000000001},"min":{"1min":0.83699999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.819,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.387}]},{"osd":5,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2569999999999999,"5min":1.327,"15min":1.365},"min":{"1min":0.80900000000000005,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":2.0430000000000001,"5min":2.3180000000000001,"15min":2.3180000000000001},"last":1.165},{"interface":"front","average":{"1min":1.2769999999999999,"5min":1.373,"15min":1.458},"min":{"1min":0.83999999999999997,"5min":0.76200000000000001,"15min":0.76200000000000001},"max":{"1min":1.6100000000000001,"5min":2.2970000000000002,"15min":2.302},"last":1.2090000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2450000000000001,"5min":1.403,"15min":1.49},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.6619999999999999,"5min":2.1880000000000002,"15min":2.4169999999999998},"last":1.742},{"interface":"front","average":{"1min":1.399,"5min":1.421,"15min":1.47},"min":{"1min":1.02,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":2.0870000000000002,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.605}]},{"osd":7,"last update":"Sat Oct 14 16:57:07 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.482,"15min":1.5609999999999999},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.827,"5min":2.2290000000000001,"15min":2.411},"last":1.5840000000000001},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.482,"15min":1.575},"min":{"1min":0.81899999999999995,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.734,"5min":2.3690000000000002,"15min":2.6600000000000001},"last":1.2450000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575057,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37232,"kb_used_data":1892,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334608,"statfs":{"total":96636764160,"available":96598638592,"internally_reserved":0,"allocated":1937408,"data_stored":1234228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":0.97999999999999998,"5min":0.90700000000000003,"15min":0.89200000000000002},"min":{"1min":0.55600000000000005,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":3.355,"5min":3.355,"15min":3.355},"last":1.704},{"interface":"front","average":{"1min":1.071,"5min":1.0620000000000001,"15min":1.0980000000000001},"min":{"1min":0.504,"5min":0.504,"15min":0.46999999999999997},"max":{"1min":3.1230000000000002,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.98399999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.1220000000000001,"5min":1.0600000000000001,"15min":1.091},"min":{"1min":0.59099999999999997,"5min":0.443,"15min":0.443},"max":{"1min":4.3940000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":1.0640000000000001},{"interface":"front","average":{"1min":1.2010000000000001,"5min":1.1830000000000001,"15min":1.2090000000000001},"min":{"1min":0.63,"5min":0.63,"15min":0.42699999999999999},"max":{"1min":4.2009999999999996,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":1.1479999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.2649999999999999,"15min":1.278},"min":{"1min":0.69799999999999995,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":4.2779999999999996,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.444},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.304,"15min":1.355},"min":{"1min":0.61699999999999999,"5min":0.59399999999999997,"15min":0.51700000000000002},"max":{"1min":2.4790000000000001,"5min":2.593,"15min":2.593},"last":2.0150000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.302,"15min":1.2909999999999999},"min":{"1min":0.873,"5min":0.71099999999999997,"15min":0.63600000000000001},"max":{"1min":5.335,"5min":5.335,"15min":5.335},"last":1.6579999999999999},{"interface":"front","average":{"1min":1.456,"5min":1.409,"15min":1.4259999999999999},"min":{"1min":0.85999999999999999,"5min":0.85999999999999999,"15min":0.77800000000000002},"max":{"1min":3.4609999999999999,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.8200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.2989999999999999,"5min":1.2929999999999999,"15min":1.3200000000000001},"min":{"1min":0.96499999999999997,"5min":0.77500000000000002,"15min":0.66100000000000003},"max":{"1min":2.931,"5min":2.931,"15min":2.931},"last":1.264},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4870000000000001,"15min":1.524},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":3.306,"5min":3.306,"15min":3.306},"last":1.3799999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.55,"15min":1.5900000000000001},"min":{"1min":0.92500000000000004,"5min":0.88900000000000001,"15min":0.69599999999999995},"max":{"1min":3.4329999999999998,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.6060000000000001},{"interface":"front","average":{"1min":1.411,"5min":1.3999999999999999,"15min":1.405},"min":{"1min":0.97299999999999998,"5min":0.79300000000000004,"15min":0.77000000000000002},"max":{"1min":3.5830000000000002,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.528}]},{"osd":7,"last update":"Sat Oct 14 16:56:47 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.5249999999999999,"15min":1.5489999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":2.7469999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.9550000000000001},{"interface":"front","average":{"1min":1.506,"5min":1.5129999999999999,"15min":1.526},"min":{"1min":1.0660000000000001,"5min":0.85699999999999998,"15min":0.70399999999999996},"max":{"1min":3.698,"5min":3.698,"15min":3.698},"last":1.8660000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":12,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":13,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:57:50.296 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:57:50.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:57:50.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.567+0000 7f8cd759e640 1 Processor -- start 2023-10-14T16:57:50.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.568+0000 7f8cd759e640 1 -- start start 2023-10-14T16:57:50.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.569+0000 7f8cd759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.569+0000 7f8cd759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 0x7f8cd80cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.569+0000 7f8cd759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd80cc850 con 0x7f8cd80b4160 2023-10-14T16:57:50.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.569+0000 7f8cd759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd80ccb30 con 0x7f8cd806a4b0 2023-10-14T16:57:50.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.569+0000 7f8cd759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80cce10 con 0x7f8cd815ac70 2023-10-14T16:57:50.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.570+0000 7f8cd6d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 0x7f8cd80cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.570+0000 7f8cd659c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.570+0000 7f8cd659c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:50168/0 (socket says 172.21.15.29:50168) 2023-10-14T16:57:50.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.570+0000 7f8cd659c640 1 -- 172.21.15.29:0/4236282775 learned_addr learned my addr 172.21.15.29:0/4236282775 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:50.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.570+0000 7f8cd6d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 0x7f8cd80cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:36272/0 (socket says 172.21.15.29:36272) 2023-10-14T16:57:50.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.571+0000 7f8cd659c640 1 -- 172.21.15.29:0/4236282775 >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 msgr2=0x7f8cd80cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:50.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.571+0000 7f8cd659c640 1 --2- 172.21.15.29:0/4236282775 >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 0x7f8cd80cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.571+0000 7f8cd659c640 1 -- 172.21.15.29:0/4236282775 >> v1:172.21.15.186:6789/0 conn(0x7f8cd80b4160 legacy=0x7f8cd8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.571+0000 7f8cd659c640 1 -- 172.21.15.29:0/4236282775 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd80cd0f0 con 0x7f8cd815ac70 2023-10-14T16:57:50.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.572+0000 7f8cd659c640 1 --2- 172.21.15.29:0/4236282775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009870 tx=0x7f8cc802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd36481678982fdf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.573+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4236282775 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8030840 con 0x7f8cd815ac70 2023-10-14T16:57:50.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.573+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4236282775 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc80309a0 con 0x7f8cd815ac70 2023-10-14T16:57:50.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.573+0000 7f8cd5d9b640 1 -- 172.21.15.29:0/4236282775 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc8030d30 con 0x7f8cd815ac70 2023-10-14T16:57:50.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 -- 172.21.15.29:0/4236282775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 msgr2=0x7f8cd80cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 --2- 172.21.15.29:0/4236282775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8009870 tx=0x7f8cc802b980 comp rx=0 tx=0).stop 2023-10-14T16:57:50.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 -- 172.21.15.29:0/4236282775 shutdown_connections 2023-10-14T16:57:50.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 --2- 172.21.15.29:0/4236282775 >> v2:172.21.15.186:3300/0 conn(0x7f8cd806a4b0 0x7f8cd80cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 --2- 172.21.15.29:0/4236282775 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd80cc270 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 -- 172.21.15.29:0/4236282775 >> 172.21.15.29:0/4236282775 conn(0x7f8cd80ce6a0 msgr2=0x7f8cd80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:50.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 -- 172.21.15.29:0/4236282775 shutdown_connections 2023-10-14T16:57:50.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.574+0000 7f8cd759e640 1 -- 172.21.15.29:0/4236282775 wait complete. 2023-10-14T16:57:50.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.576+0000 7f8cd759e640 1 Processor -- start 2023-10-14T16:57:50.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.577+0000 7f8cd759e640 1 -- start start 2023-10-14T16:57:50.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cd806a4b0 0x7f8cd80c7ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd814e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80c5ee0 con 0x7f8cd80b4160 2023-10-14T16:57:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80c5c00 con 0x7f8cd806a4b0 2023-10-14T16:57:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.578+0000 7f8cd759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd80b13f0 con 0x7f8cd815ac70 2023-10-14T16:57:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8cd659c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cd806a4b0 0x7f8cd80c7ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58832/0 (socket says 172.21.15.29:58832) 2023-10-14T16:57:50.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 -- 172.21.15.29:0/3820512182 learned_addr learned my addr 172.21.15.29:0/3820512182 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:50.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8cd6d9d640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd814e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 -- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 msgr2=0x7f8cd814e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd814e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 -- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cd806a4b0 msgr2=0x7f8cd80c7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cd806a4b0 0x7f8cd80c7ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8ccffff640 1 -- 172.21.15.29:0/3820512182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc80094d0 con 0x7f8cd80b4160 2023-10-14T16:57:50.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.579+0000 7f8cd6d9d640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd814e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:57:50.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8ccffff640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c450 tx=0x7f8cc000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc00052d0 con 0x7f8cd80b4160 2023-10-14T16:57:50.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc0005430 con 0x7f8cd80b4160 2023-10-14T16:57:50.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0012870 con 0x7f8cd80b4160 2023-10-14T16:57:50.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd80b16d0 con 0x7f8cd80b4160 2023-10-14T16:57:50.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.580+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd80c8400 con 0x7f8cd80b4160 2023-10-14T16:57:50.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.582+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd80b1bf0 con 0x7f8cd80b4160 2023-10-14T16:57:50.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.583+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8cc00129d0 con 0x7f8cd80b4160 2023-10-14T16:57:50.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.584+0000 7f8cceffd640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 0x7f8cb806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:50.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.584+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f8cc0089270 con 0x7f8cd80b4160 2023-10-14T16:57:50.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.585+0000 7f8cd659c640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 0x7f8cb806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:50.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.585+0000 7f8cd659c640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 0x7f8cb806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036000 tx=0x7f8cc8008030 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:50.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.589+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8cd80b1bf0 con 0x7f8cd80b4160 2023-10-14T16:57:50.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.861+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f8cd80b1bf0 con 0x7f8cd80b4160 2023-10-14T16:57:50.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.864+0000 7f8cceffd640 1 -- 172.21.15.29:0/3820512182 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v16) v1 ==== 190+0+147 (secure 0 0 0) 0x7f8cd80b1bf0 con 0x7f8cd80b4160 2023-10-14T16:57:50.866 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:57:50.866 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:57:50.866 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:57:50.867 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:57:50.867 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:57:50.867 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:57:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 msgr2=0x7f8cb806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 0x7f8cb806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f8cc8036000 tx=0x7f8cc8008030 comp rx=0 tx=0).stop 2023-10-14T16:57:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 msgr2=0x7f8cd815e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 secure :-1 s=READY pgs=1230 cs=0 l=1 rev1=1 crypto rx=0x7f8cc000c450 tx=0x7f8cc000c910 comp rx=0 tx=0).stop 2023-10-14T16:57:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 shutdown_connections 2023-10-14T16:57:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f8cb806af90 0x7f8cb806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8cd815ac70 0x7f8cd814e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8cd80b4160 0x7f8cd815e830 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 --2- 172.21.15.29:0/3820512182 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8cd806a4b0 0x7f8cd80c7ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 >> 172.21.15.29:0/3820512182 conn(0x7f8cd80ce6a0 msgr2=0x7f8cd81590c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 shutdown_connections 2023-10-14T16:57:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:50.868+0000 7f8cd759e640 1 -- 172.21.15.29:0/3820512182 wait complete. 2023-10-14T16:57:50.889 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:57:51.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.119+0000 7f43fe832640 1 Processor -- start 2023-10-14T16:57:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.119+0000 7f43fe832640 1 -- start start 2023-10-14T16:57:51.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fe832640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43f815c9f0 0x7f43f80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fe832640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fe832640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43f8160a90 con 0x7f43f80c9230 2023-10-14T16:57:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fe832640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43f80cdec0 con 0x7f43f815c9f0 2023-10-14T16:57:51.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fe832640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80cd900 con 0x7f43f815dde0 2023-10-14T16:57:51.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.121+0000 7f43fd830640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43f815c9f0 0x7f43f80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:51.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f43f815c9f0 0x7f43f80c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51476/0 (socket says 172.21.15.29:51476) 2023-10-14T16:57:51.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 -- 172.21.15.29:0/328272268 learned_addr learned my addr 172.21.15.29:0/328272268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:51.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fe031640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:51.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 -- 172.21.15.29:0/328272268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 msgr2=0x7f43f80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:51.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 -- 172.21.15.29:0/328272268 >> v1:172.21.15.186:6789/0 conn(0x7f43f80c9230 legacy=0x7f43f80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.122+0000 7f43fd830640 1 -- 172.21.15.29:0/328272268 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f80cdbe0 con 0x7f43f815c9f0 2023-10-14T16:57:51.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.123+0000 7f43fd830640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f80c8cf0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f43f4009870 tx=0x7f43f402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbc81a7a4d67fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:51.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fc82e640 1 -- 172.21.15.29:0/328272268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4030840 con 0x7f43f815c9f0 2023-10-14T16:57:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fc82e640 1 -- 172.21.15.29:0/328272268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43f40309a0 con 0x7f43f815c9f0 2023-10-14T16:57:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fc82e640 1 -- 172.21.15.29:0/328272268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43f4030d30 con 0x7f43f815c9f0 2023-10-14T16:57:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 -- 172.21.15.29:0/328272268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 msgr2=0x7f43f80c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f80c8cf0 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 crypto rx=0x7f43f4009870 tx=0x7f43f402b980 comp rx=0 tx=0).stop 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 -- 172.21.15.29:0/328272268 shutdown_connections 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 --2- 172.21.15.29:0/328272268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f80c8cf0 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.124+0000 7f43fe832640 1 -- 172.21.15.29:0/328272268 >> 172.21.15.29:0/328272268 conn(0x7f43f80ce6a0 msgr2=0x7f43f80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.125+0000 7f43fe832640 1 -- 172.21.15.29:0/328272268 shutdown_connections 2023-10-14T16:57:51.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.125+0000 7f43fe832640 1 -- 172.21.15.29:0/328272268 wait complete. 2023-10-14T16:57:51.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.126+0000 7f43fe832640 1 Processor -- start 2023-10-14T16:57:51.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 -- start start 2023-10-14T16:57:51.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 0x7f43f80c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f815e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f806a7e0 con 0x7f43f815c9f0 2023-10-14T16:57:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43f8151130 con 0x7f43f80c9230 2023-10-14T16:57:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.127+0000 7f43fe832640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43f80c28d0 con 0x7f43f815dde0 2023-10-14T16:57:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51478/0 (socket says 172.21.15.29:51478) 2023-10-14T16:57:51.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 -- 172.21.15.29:0/2735424040 learned_addr learned my addr 172.21.15.29:0/2735424040 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 -- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 msgr2=0x7f43f815e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f815e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 -- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 msgr2=0x7f43f80c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd830640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 0x7f43f80c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:51.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 0x7f43f80c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd02f640 1 -- 172.21.15.29:0/2735424040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43f40094d0 con 0x7f43f815c9f0 2023-10-14T16:57:51.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.128+0000 7f43fd830640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 0x7f43f80c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:51.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43fd02f640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c450 tx=0x7f43e800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:51.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e80052d0 con 0x7f43f815c9f0 2023-10-14T16:57:51.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43e8005430 con 0x7f43f815c9f0 2023-10-14T16:57:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43e8012870 con 0x7f43f815c9f0 2023-10-14T16:57:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43f806cc20 con 0x7f43f815c9f0 2023-10-14T16:57:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.129+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43f806a120 con 0x7f43f815c9f0 2023-10-14T16:57:51.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.130+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43f806a920 con 0x7f43f815c9f0 2023-10-14T16:57:51.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.131+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f43e80129d0 con 0x7f43f815c9f0 2023-10-14T16:57:51.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.132+0000 7f43ee7fc640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 0x7f43dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:51.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.132+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f43e80892d0 con 0x7f43f815c9f0 2023-10-14T16:57:51.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.133+0000 7f43fd830640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 0x7f43dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:51.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.133+0000 7f43fd830640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 0x7f43dc06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43f4036000 tx=0x7f43f403b040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:51.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.137+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f43f806a920 con 0x7f43f815c9f0 2023-10-14T16:57:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.374+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43f806a920 con 0x7f43f815c9f0 2023-10-14T16:57:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.375+0000 7f43ee7fc640 1 -- 172.21.15.29:0/2735424040 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f43f806a920 con 0x7f43f815c9f0 2023-10-14T16:57:51.376 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:57:51.377 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:57:51.377 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:57:51.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 msgr2=0x7f43dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 0x7f43dc06d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f43f4036000 tx=0x7f43f403b040 comp rx=0 tx=0).stop 2023-10-14T16:57:51.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 msgr2=0x7f43f8150bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:51.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 crypto rx=0x7f43e800c450 tx=0x7f43e800c910 comp rx=0 tx=0).stop 2023-10-14T16:57:51.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 shutdown_connections 2023-10-14T16:57:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f43dc06af90 0x7f43dc06d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f43f815dde0 0x7f43f815e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f43f815c9f0 0x7f43f8150bf0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 --2- 172.21.15.29:0/2735424040 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f43f80c9230 0x7f43f80c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 >> 172.21.15.29:0/2735424040 conn(0x7f43f80ce6a0 msgr2=0x7f43f80c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.378+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 shutdown_connections 2023-10-14T16:57:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:51.379+0000 7f43fe832640 1 -- 172.21.15.29:0/2735424040 wait complete. 2023-10-14T16:57:51.802 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 49, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:57:44.731151+0000', 'modified': '2023-10-14T16:57:47.795224+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6076}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6076': {'gid': 6076, 'name': 'c', 'rank': 0, 'incarnation': 45, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6836/1784427614', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 1784427614}, {'type': 'v1', 'addr': '172.21.15.29:6836', 'nonce': 1784427614}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [13], 'metadata_pool': 12, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:57:51.803 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:57:51.804 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T16:57:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.038+0000 7f79113fc640 1 Processor -- start 2023-10-14T16:57:52.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.038+0000 7f79113fc640 1 -- start start 2023-10-14T16:57:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f79113fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f790c0caac0 0x7f790c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f79113fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0cb030 con 0x7f790c06a4b0 2023-10-14T16:57:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f79113fc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f790c0c91c0 con 0x7f790c15ad70 2023-10-14T16:57:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f79113fc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f790c0c94a0 con 0x7f790c0caac0 2023-10-14T16:57:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f790b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f790c0caac0 0x7f790c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f790b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f790c0caac0 0x7f790c0ad4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51482/0 (socket says 172.21.15.29:51482) 2023-10-14T16:57:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.040+0000 7f790b7fe640 1 -- 172.21.15.29:0/1602479242 learned_addr learned my addr 172.21.15.29:0/1602479242 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.041+0000 7f790b7fe640 1 -- 172.21.15.29:0/1602479242 >> v1:172.21.15.186:6789/0 conn(0x7f790c15ad70 legacy=0x7f790c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.041+0000 7f790b7fe640 1 -- 172.21.15.29:0/1602479242 >> v1:172.21.15.29:6789/0 conn(0x7f790c06a4b0 legacy=0x7f790c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.041+0000 7f790b7fe640 1 -- 172.21.15.29:0/1602479242 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f790c0c9780 con 0x7f790c0caac0 2023-10-14T16:57:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f790b7fe640 1 --2- 172.21.15.29:0/1602479242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c0caac0 0x7f790c0ad4d0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7904009e50 tx=0x7f790402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=430d88aaceda0c29 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f790a7fc640 1 -- 172.21.15.29:0/1602479242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790402cb10 con 0x7f790c0caac0 2023-10-14T16:57:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f790a7fc640 1 -- 172.21.15.29:0/1602479242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790402cc70 con 0x7f790c0caac0 2023-10-14T16:57:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f790a7fc640 1 -- 172.21.15.29:0/1602479242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79040343f0 con 0x7f790c0caac0 2023-10-14T16:57:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f79113fc640 1 -- 172.21.15.29:0/1602479242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c0caac0 msgr2=0x7f790c0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.042+0000 7f79113fc640 1 --2- 172.21.15.29:0/1602479242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c0caac0 0x7f790c0ad4d0 secure :-1 s=READY pgs=1234 cs=0 l=1 rev1=1 crypto rx=0x7f7904009e50 tx=0x7f790402bf70 comp rx=0 tx=0).stop 2023-10-14T16:57:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.043+0000 7f79113fc640 1 -- 172.21.15.29:0/1602479242 shutdown_connections 2023-10-14T16:57:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.043+0000 7f79113fc640 1 --2- 172.21.15.29:0/1602479242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c0caac0 0x7f790c0ad4d0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.043+0000 7f79113fc640 1 -- 172.21.15.29:0/1602479242 >> 172.21.15.29:0/1602479242 conn(0x7f790c0ce9b0 msgr2=0x7f790c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.043+0000 7f79113fc640 1 -- 172.21.15.29:0/1602479242 shutdown_connections 2023-10-14T16:57:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.043+0000 7f79113fc640 1 -- 172.21.15.29:0/1602479242 wait complete. 2023-10-14T16:57:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.044+0000 7f79113fc640 1 Processor -- start 2023-10-14T16:57:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.045+0000 7f79113fc640 1 -- start start 2023-10-14T16:57:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.045+0000 7f79113fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a4b0 0x7f790c159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.045+0000 7f79113fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0caac0 0x7f790c0aee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f79113fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f79113fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0c7d70 con 0x7f790c15ad70 2023-10-14T16:57:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f79113fc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0af370 con 0x7f790c06a4b0 2023-10-14T16:57:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f79113fc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f790c0c7160 con 0x7f790c0caac0 2023-10-14T16:57:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51498/0 (socket says 172.21.15.29:51498) 2023-10-14T16:57:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 -- 172.21.15.29:0/3592794632 learned_addr learned my addr 172.21.15.29:0/3592794632 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 -- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0caac0 msgr2=0x7f790c0aee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0caac0 0x7f790c0aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 -- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a4b0 msgr2=0x7f790c159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790affd640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a4b0 0x7f790c159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a4b0 0x7f790c159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.046+0000 7f790b7fe640 1 -- 172.21.15.29:0/3592794632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7904009b30 con 0x7f790c15ad70 2023-10-14T16:57:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f790b7fe640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7904036040 tx=0x7f7904004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f79037fe640 1 -- 172.21.15.29:0/3592794632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790403e450 con 0x7f790c15ad70 2023-10-14T16:57:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f79037fe640 1 -- 172.21.15.29:0/3592794632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790403e5d0 con 0x7f790c15ad70 2023-10-14T16:57:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f79037fe640 1 -- 172.21.15.29:0/3592794632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790403e980 con 0x7f790c15ad70 2023-10-14T16:57:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f790c0c7440 con 0x7f790c15ad70 2023-10-14T16:57:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.047+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c0c7a00 con 0x7f790c15ad70 2023-10-14T16:57:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.049+0000 7f79037fe640 1 -- 172.21.15.29:0/3592794632 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f790403eae0 con 0x7f790c15ad70 2023-10-14T16:57:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.051+0000 7f79037fe640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 0x7f78ec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.051+0000 7f79037fe640 1 -- 172.21.15.29:0/3592794632 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f79040ac420 con 0x7f790c15ad70 2023-10-14T16:57:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.051+0000 7f790affd640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 0x7f78ec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.052+0000 7f7913f8b740 1 Processor -- start 2023-10-14T16:57:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.052+0000 7f7913f8b740 1 -- start start 2023-10-14T16:57:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.052+0000 7f790affd640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 0x7f78ec06d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78fc006fd0 tx=0x7f78fc008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b861064e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b861094d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b860f857a0 con 0x55b8610653c0 2023-10-14T16:57:52.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b860fd7740 con 0x55b861091f80 2023-10-14T16:57:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b860fd4e30 con 0x55b861092360 2023-10-14T16:57:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51502/0 (socket says 172.21.15.29:51502) 2023-10-14T16:57:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 -- 172.21.15.29:0/2829683785 learned_addr learned my addr 172.21.15.29:0/2829683785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 -- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 msgr2=0x55b861094d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f7903fff640 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b861094d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790affd640 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b861064e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b861094d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 -- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 msgr2=0x55b861064e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b861064e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.053+0000 7f790b7fe640 1 -- 172.21.15.29:0/2829683785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b860f94910 con 0x55b8610653c0 2023-10-14T16:57:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f790b7fe640 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7904077560 tx=0x7f790407b060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4ec6ce82814e96c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f7900ff9640 1 -- 172.21.15.29:0/2829683785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79040832b0 con 0x55b8610653c0 2023-10-14T16:57:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f7900ff9640 1 -- 172.21.15.29:0/2829683785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790403ed90 con 0x55b8610653c0 2023-10-14T16:57:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f7900ff9640 1 -- 172.21.15.29:0/2829683785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f790403ef00 con 0x55b8610653c0 2023-10-14T16:57:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f7913f8b740 1 -- 172.21.15.29:0/2829683785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 msgr2=0x55b861095250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.054+0000 7f7913f8b740 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 secure :-1 s=READY pgs=1236 cs=0 l=1 rev1=1 crypto rx=0x7f7904077560 tx=0x7f790407b060 comp rx=0 tx=0).stop 2023-10-14T16:57:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 -- 172.21.15.29:0/2829683785 shutdown_connections 2023-10-14T16:57:52.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b8610653c0 0x55b861095250 unknown :-1 s=CLOSED pgs=1236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b861094d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 --2- 172.21.15.29:0/2829683785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b861064e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 -- 172.21.15.29:0/2829683785 >> 172.21.15.29:0/2829683785 conn(0x55b861067190 msgr2=0x55b8610afd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 -- 172.21.15.29:0/2829683785 shutdown_connections 2023-10-14T16:57:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 -- 172.21.15.29:0/2829683785 wait complete. 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.055+0000 7f7913f8b740 1 -- start start 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.056+0000 7f7913f8b740 1 Processor -- start 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b86108a9d0 0x55b86108ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b860fd0720 con 0x55b861091f80 2023-10-14T16:57:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b860f6e970 con 0x55b86108a9d0 2023-10-14T16:57:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b860f6e610 con 0x55b861092360 2023-10-14T16:57:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7913f8b740 10 client.0 _ll_register_callbacks cb 0x55b860f909d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:57:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51516/0 (socket says 172.21.15.29:51516) 2023-10-14T16:57:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 -- 172.21.15.29:0/1758630475 learned_addr learned my addr 172.21.15.29:0/1758630475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f7903fff640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 -- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 msgr2=0x55b86108c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 -- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b86108a9d0 msgr2=0x55b86108ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b86108a9d0 0x55b86108ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.057+0000 7f790affd640 1 -- 172.21.15.29:0/1758630475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f790c0c7a00 con 0x55b861091f80 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f790affd640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cc90 tx=0x7f78fc004680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d716d2a4da66bfd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f78e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc023070 con 0x55b861091f80 2023-10-14T16:57:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f78fc004020 con 0x55b861091f80 2023-10-14T16:57:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b860ff7160 con 0x55b861091f80 2023-10-14T16:57:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78fc019050 con 0x55b861091f80 2023-10-14T16:57:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f7913f8b740 10 client.7647 fetch_fsmap learned FSMap version 49 2023-10-14T16:57:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55b860fceda0 con 0x55b861091f80 2023-10-14T16:57:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.058+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b860ff7160 con 0x55b861091f80 2023-10-14T16:57:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f78fc026020 con 0x55b861091f80 2023-10-14T16:57:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 10 client.7647 fetch_fsmap finished waiting for FSMap version 49 2023-10-14T16:57:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 10 client.7647 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:57:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 20 client.7647 populate_metadata read hostname 'smithi029' 2023-10-14T16:57:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861093820 0x55b861096130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 4 client.7647 mds_command: new command op to 6076 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:57:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 --> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b860fa2cc0 con 0x55b861093820 2023-10-14T16:57:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7903fff640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861093820 0x55b861096130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f7903fff640 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861093820 0x55b861096130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ed3f05bdd42efde server_cookie=b61ad02b5c49370f in_seq=0 out_seq=0 2023-10-14T16:57:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.059+0000 7f78e9ffb640 10 client.7647 ms_handle_connect on v2:172.21.15.29:6834/1784427614 2023-10-14T16:57:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.060+0000 7f78e9ffb640 1 -- 172.21.15.29:0/1758630475 <== mds.0 v2:172.21.15.29:6834/1784427614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b860fa2cc0 con 0x55b861093820 2023-10-14T16:57:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.060+0000 7f78e9ffb640 10 client.7647 handle_command_reply: tid=0 2023-10-14T16:57:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.060+0000 7f7913f8b740 1 client.7647 shutdown 2023-10-14T16:57:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.060+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 msgr2=0x55b8610899f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.060+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cc90 tx=0x7f78fc004680 comp rx=0 tx=0).stop 2023-10-14T16:57:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 shutdown_connections 2023-10-14T16:57:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861093820 0x55b861096130 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b86108a9d0 0x55b86108ccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 --2- 172.21.15.29:0/1758630475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861091f80 0x55b8610899f0 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f78e9ffb640 0 client.7647 ms_handle_reset on v2:172.21.15.29:6834/1784427614 2023-10-14T16:57:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 >> 172.21.15.29:0/1758630475 conn(0x55b8610653c0 msgr2=0x55b8610b3220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 shutdown_connections 2023-10-14T16:57:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 1 -- 172.21.15.29:0/1758630475 wait complete. 2023-10-14T16:57:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.061+0000 7f7913f8b740 20 client.7647 trim_cache size 0 max 16384 2023-10-14T16:57:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.077+0000 7f7913f8b740 1 Processor -- start 2023-10-14T16:57:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 -- start start 2023-10-14T16:57:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b860d34f20 con 0x55b86108a9d0 2023-10-14T16:57:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b861094110 con 0x55b861091f80 2023-10-14T16:57:52.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.078+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b8610b0ad0 con 0x55b861092360 2023-10-14T16:57:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.079+0000 7f790affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.079+0000 7f790affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51522/0 (socket says 172.21.15.29:51522) 2023-10-14T16:57:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.079+0000 7f790affd640 1 -- 172.21.15.29:0/371876924 learned_addr learned my addr 172.21.15.29:0/371876924 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f7903fff640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790b7fe640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 -- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 msgr2=0x55b86108c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 -- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 msgr2=0x55b8610b33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610b33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 -- 172.21.15.29:0/371876924 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b860f6db80 con 0x55b86108a9d0 2023-10-14T16:57:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f7903fff640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.080+0000 7f790affd640 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cf90 tx=0x7f78fc01c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=911ad9712f0a0e0c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f79017fa640 1 -- 172.21.15.29:0/371876924 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc023070 con 0x55b86108a9d0 2023-10-14T16:57:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f79017fa640 1 -- 172.21.15.29:0/371876924 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78fc01a040 con 0x55b86108a9d0 2023-10-14T16:57:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f79017fa640 1 -- 172.21.15.29:0/371876924 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78fc019040 con 0x55b86108a9d0 2023-10-14T16:57:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 -- 172.21.15.29:0/371876924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 msgr2=0x55b86108b9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 secure :-1 s=READY pgs=1238 cs=0 l=1 rev1=1 crypto rx=0x7f78fc00cf90 tx=0x7f78fc01c000 comp rx=0 tx=0).stop 2023-10-14T16:57:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 -- 172.21.15.29:0/371876924 shutdown_connections 2023-10-14T16:57:52.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b861092360 0x55b86108c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610b33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 --2- 172.21.15.29:0/371876924 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b86108a9d0 0x55b86108b9d0 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 -- 172.21.15.29:0/371876924 >> 172.21.15.29:0/371876924 conn(0x55b8610653c0 msgr2=0x55b861064290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 -- 172.21.15.29:0/371876924 shutdown_connections 2023-10-14T16:57:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.081+0000 7f7913f8b740 1 -- 172.21.15.29:0/371876924 wait complete. 2023-10-14T16:57:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.082+0000 7f7913f8b740 1 -- start start 2023-10-14T16:57:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.082+0000 7f7913f8b740 1 Processor -- start 2023-10-14T16:57:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b8610b03b0 0x55b861097240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b861118e40 con 0x55b861092360 2023-10-14T16:57:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b8610902c0 con 0x55b861091f80 2023-10-14T16:57:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b861090d00 con 0x55b8610b03b0 2023-10-14T16:57:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7913f8b740 10 client.0 _ll_register_callbacks cb 0x7f78fc00a3e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:57:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7903fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7903fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51536/0 (socket says 172.21.15.29:51536) 2023-10-14T16:57:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.083+0000 7f7903fff640 1 -- 172.21.15.29:0/3049136158 learned_addr learned my addr 172.21.15.29:0/3049136158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 -- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b8610b03b0 msgr2=0x55b861097240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b8610b03b0 0x55b861097240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 -- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 msgr2=0x55b8610af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f790affd640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610af6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 -- 172.21.15.29:0/3049136158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b860fceda0 con 0x55b861092360 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7903fff640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f78f4001550 tx=0x7f78f4017e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=172d90780419841d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f78e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:57:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78f400c040 con 0x55b861092360 2023-10-14T16:57:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f78f40016e0 con 0x55b861092360 2023-10-14T16:57:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.084+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f790403ed90 con 0x55b861092360 2023-10-14T16:57:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78f40018a0 con 0x55b861092360 2023-10-14T16:57:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 10 client.7653 fetch_fsmap learned FSMap version 49 2023-10-14T16:57:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=2+}) v3 -- 0x55b860f93170 con 0x55b861092360 2023-10-14T16:57:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f790403ed90 con 0x55b861092360 2023-10-14T16:57:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 49) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f78f4001a00 con 0x55b861092360 2023-10-14T16:57:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 10 client.7653 fetch_fsmap finished waiting for FSMap version 49 2023-10-14T16:57:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 10 client.7653 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:57:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 20 client.7653 populate_metadata read hostname 'smithi029' 2023-10-14T16:57:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861097aa0 0x55b8610881c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 4 client.7653 mds_command: new command op to 6076 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:57:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 --> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b860fc9700 con 0x55b861097aa0 2023-10-14T16:57:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.085+0000 7f790affd640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861097aa0 0x55b8610881c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f790affd640 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861097aa0 0x55b8610881c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59c8752b07e8421f server_cookie=8c284ea223e5fc23 in_seq=0 out_seq=0 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f78e9ffb640 10 client.7653 ms_handle_connect on v2:172.21.15.29:6834/1784427614 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f78e9ffb640 1 -- 172.21.15.29:0/3049136158 <== mds.0 v2:172.21.15.29:6834/1784427614 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55b860fc9700 con 0x55b861097aa0 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f78e9ffb640 10 client.7653 handle_command_reply: tid=0 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f7913f8b740 1 client.7653 shutdown 2023-10-14T16:57:52.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 msgr2=0x55b8610afc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 crypto rx=0x7f78f4001550 tx=0x7f78f4017e40 comp rx=0 tx=0).stop 2023-10-14T16:57:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.086+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 shutdown_connections 2023-10-14T16:57:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:6834/1784427614,v1:172.21.15.29:6836/1784427614] conn(0x55b861097aa0 0x55b8610881c0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b8610b03b0 0x55b861097240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b861092360 0x55b8610afc30 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f78e9ffb640 0 client.7653 ms_handle_reset on v2:172.21.15.29:6834/1784427614 2023-10-14T16:57:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 --2- 172.21.15.29:0/3049136158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b861091f80 0x55b8610af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 >> 172.21.15.29:0/3049136158 conn(0x55b8610653c0 msgr2=0x55b8610c2400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 shutdown_connections 2023-10-14T16:57:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 1 -- 172.21.15.29:0/3049136158 wait complete. 2023-10-14T16:57:52.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f7913f8b740 20 client.7653 trim_cache size 0 max 16384 2023-10-14T16:57:52.100 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:57:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:57:52.100 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:57:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 6076, 2023-10-14T16:57:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:57:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:57:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:57:52.101 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8122855700000002, 2023-10-14T16:57:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 49, 2023-10-14T16:57:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 109, 2023-10-14T16:57:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 109, 2023-10-14T16:57:52.102 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 12.151787145 2023-10-14T16:57:52.102 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:57:52.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 msgr2=0x7f78ec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 0x7f78ec06d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f78fc006fd0 tx=0x7f78fc008040 comp rx=0 tx=0).stop 2023-10-14T16:57:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 msgr2=0x7f790c0c8200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.087+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 crypto rx=0x7f7904036040 tx=0x7f7904004a40 comp rx=0 tx=0).stop 2023-10-14T16:57:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 shutdown_connections 2023-10-14T16:57:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f78ec06afc0 0x7f78ec06d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f790c15ad70 0x7f790c0c8200 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f790c0caac0 0x7f790c0aee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 --2- 172.21.15.29:0/3592794632 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f790c06a4b0 0x7f790c159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 >> 172.21.15.29:0/3592794632 conn(0x7f790c0ce9b0 msgr2=0x7f790c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 shutdown_connections 2023-10-14T16:57:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.088+0000 7f79113fc640 1 -- 172.21.15.29:0/3592794632 wait complete. 2023-10-14T16:57:52.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:57:52.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.316+0000 7efe468bb640 1 Processor -- start 2023-10-14T16:57:52.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.316+0000 7efe468bb640 1 -- start start 2023-10-14T16:57:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe468bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe401599c0 0x7efe400ca1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe468bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 0x7efe4006f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe468bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4006bab0 con 0x7efe400ca6f0 2023-10-14T16:57:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe468bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4006f300 con 0x7efe401599c0 2023-10-14T16:57:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe468bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe4006bd90 con 0x7efe400cda90 2023-10-14T16:57:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe458b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe401599c0 0x7efe400ca1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe458b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7efe401599c0 0x7efe400ca1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51540/0 (socket says 172.21.15.29:51540) 2023-10-14T16:57:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe458b9640 1 -- 172.21.15.29:0/3465923100 learned_addr learned my addr 172.21.15.29:0/3465923100 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe458b9640 1 -- 172.21.15.29:0/3465923100 >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 msgr2=0x7efe4006f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.318+0000 7efe460ba640 1 --2- 172.21.15.29:0/3465923100 >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 0x7efe4006f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.319+0000 7efe458b9640 1 --2- 172.21.15.29:0/3465923100 >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 0x7efe4006f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.319+0000 7efe458b9640 1 -- 172.21.15.29:0/3465923100 >> v1:172.21.15.186:6789/0 conn(0x7efe400ca6f0 legacy=0x7efe400cd380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.319+0000 7efe458b9640 1 -- 172.21.15.29:0/3465923100 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4006c070 con 0x7efe401599c0 2023-10-14T16:57:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.319+0000 7efe460ba640 1 --2- 172.21.15.29:0/3465923100 >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 0x7efe4006f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.320+0000 7efe458b9640 1 --2- 172.21.15.29:0/3465923100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe401599c0 0x7efe400ca1b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7efe400c9f30 tx=0x7efe3c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38f51a006328835e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.320+0000 7efe448b7640 1 -- 172.21.15.29:0/3465923100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c0317e0 con 0x7efe401599c0 2023-10-14T16:57:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.320+0000 7efe448b7640 1 -- 172.21.15.29:0/3465923100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c031940 con 0x7efe401599c0 2023-10-14T16:57:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.320+0000 7efe448b7640 1 -- 172.21.15.29:0/3465923100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c031cd0 con 0x7efe401599c0 2023-10-14T16:57:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 -- 172.21.15.29:0/3465923100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe401599c0 msgr2=0x7efe400ca1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 --2- 172.21.15.29:0/3465923100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe401599c0 0x7efe400ca1b0 secure :-1 s=READY pgs=1240 cs=0 l=1 rev1=1 crypto rx=0x7efe400c9f30 tx=0x7efe3c02b880 comp rx=0 tx=0).stop 2023-10-14T16:57:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 -- 172.21.15.29:0/3465923100 shutdown_connections 2023-10-14T16:57:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 --2- 172.21.15.29:0/3465923100 >> v2:172.21.15.186:3300/0 conn(0x7efe400cda90 0x7efe4006f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 --2- 172.21.15.29:0/3465923100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe401599c0 0x7efe400ca1b0 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 -- 172.21.15.29:0/3465923100 >> 172.21.15.29:0/3465923100 conn(0x7efe400b0a90 msgr2=0x7efe400ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 -- 172.21.15.29:0/3465923100 shutdown_connections 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.321+0000 7efe468bb640 1 -- 172.21.15.29:0/3465923100 wait complete. 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.323+0000 7efe468bb640 1 Processor -- start 2023-10-14T16:57:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.323+0000 7efe468bb640 1 -- start start 2023-10-14T16:57:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 0x7efe4014ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe401599c0 0x7efe400a8d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe400c3c20 con 0x7efe400ca6f0 2023-10-14T16:57:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe400c3f00 con 0x7efe400cda90 2023-10-14T16:57:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe468bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe400c41e0 con 0x7efe401599c0 2023-10-14T16:57:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51546/0 (socket says 172.21.15.29:51546) 2023-10-14T16:57:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 -- 172.21.15.29:0/2410883573 learned_addr learned my addr 172.21.15.29:0/2410883573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe450b8640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 0x7efe4014ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 -- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe401599c0 msgr2=0x7efe400a8d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe460ba640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe401599c0 0x7efe400a8d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe401599c0 0x7efe400a8d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 -- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 msgr2=0x7efe4014ea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.325+0000 7efe458b9640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 0x7efe4014ea20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.326+0000 7efe458b9640 1 -- 172.21.15.29:0/2410883573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3c0094d0 con 0x7efe400ca6f0 2023-10-14T16:57:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.326+0000 7efe450b8640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 0x7efe4014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:52.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.326+0000 7efe458b9640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7efe3c02b8c0 tx=0x7efe3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.327+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c0329c0 con 0x7efe400ca6f0 2023-10-14T16:57:52.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.327+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe3c032b40 con 0x7efe400ca6f0 2023-10-14T16:57:52.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.327+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efe3c0453f0 con 0x7efe400ca6f0 2023-10-14T16:57:52.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.327+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe400c44c0 con 0x7efe400ca6f0 2023-10-14T16:57:52.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.327+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4014ef60 con 0x7efe400ca6f0 2023-10-14T16:57:52.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.328+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe40158190 con 0x7efe400ca6f0 2023-10-14T16:57:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.329+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efe3c04d3f0 con 0x7efe400ca6f0 2023-10-14T16:57:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.330+0000 7efe367fc640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 0x7efe2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.330+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7efe3c0ab820 con 0x7efe400ca6f0 2023-10-14T16:57:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.330+0000 7efe450b8640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 0x7efe2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.331+0000 7efe450b8640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 0x7efe2406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efe400c2ca0 tx=0x7efe30009040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:52.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.334+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efe3c076730 con 0x7efe400ca6f0 2023-10-14T16:57:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.568+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7efe400cae20 con 0x7efe400ca6f0 2023-10-14T16:57:52.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.569+0000 7efe367fc640 1 -- 172.21.15.29:0/2410883573 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v49) v1 ==== 53+0+73 (secure 0 0 0) 0x7efe400cae20 con 0x7efe400ca6f0 2023-10-14T16:57:52.570 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:57:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.571+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 msgr2=0x7efe2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.571+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 0x7efe2406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7efe400c2ca0 tx=0x7efe30009040 comp rx=0 tx=0).stop 2023-10-14T16:57:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.571+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 msgr2=0x7efe400c4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.571+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 crypto rx=0x7efe3c02b8c0 tx=0x7efe3c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 shutdown_connections 2023-10-14T16:57:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7efe2406af90 0x7efe2406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efe401599c0 0x7efe400a8d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efe400cda90 0x7efe4014ea20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 --2- 172.21.15.29:0/2410883573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efe400ca6f0 0x7efe400c4cc0 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 >> 172.21.15.29:0/2410883573 conn(0x7efe400b0a90 msgr2=0x7efe4015a0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 shutdown_connections 2023-10-14T16:57:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:52.572+0000 7efe468bb640 1 -- 172.21.15.29:0/2410883573 wait complete. 2023-10-14T16:57:52.591 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:57:52.591 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:57:52.591 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:57:52.592 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:57:52.592 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:52.592 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:57:52.593 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:57:52.593 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:57:52.593 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:52.594 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:52.594 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:57:52.647 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:57:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:57:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:57:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.647 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:57:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.648 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:52.648 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:57:52.648 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:57:52.649 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:57:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41304sec preferred_lft 41304sec 2023-10-14T16:57:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:57:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.650 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:57:52.650 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:57:52.651 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:57:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:57:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:57:52.651 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:57:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.652 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:57:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:57:52.652 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:57:52.653 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:57:52.654 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:52.655 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:57:52.754 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:57:52.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:52.759 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:57:52.822 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:57:52.823 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:57:52.823 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:57:52.823 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:57:52.823 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:57:52.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:52.825 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:57:52.825 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:57:52.825 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:57:52.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:57:52.826 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:57:52.826 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:57:53.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:53.023 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:57:53.078 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:57:53.079 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:57:53.080 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:53.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:53.081 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:57:53.189 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:57:53.190 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:53.249 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:57:53.250 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:57:53.250 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:53.302 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:57:53.302 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:53.303 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:53.445 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:53.445 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:53.446 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:53.468 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:53.468 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:53.469 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:53.469 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:57:53.469 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:57:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:57:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:57:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:57:53.496 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:57:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:57:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:57:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:57:53.497 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:57:53.499 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:57:53.499 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:57:53.499 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:57:53.500 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:57:53.500 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:57:53.503 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:57:53.504 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:53.504 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:57:53.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:57:53.596 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:57:53.665 INFO:teuthology.orchestra.run.smithi029.stdout:client5627 2023-10-14T16:57:53.665 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:57:53.666 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:57:53.723 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:57:53.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.949+0000 7f5bbb147640 1 Processor -- start 2023-10-14T16:57:53.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.949+0000 7f5bbb147640 1 -- start start 2023-10-14T16:57:53.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bbb147640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bb4159950 0x7f5bb415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bbb147640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb4070450 0x7f5bb40708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bbb147640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bb4070df0 con 0x7f5bb406c100 2023-10-14T16:57:53.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bbb147640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bb415e490 con 0x7f5bb4159950 2023-10-14T16:57:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bbb147640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb415aef0 con 0x7f5bb4070450 2023-10-14T16:57:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.951+0000 7f5bba145640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bb4159950 0x7f5bb415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5bb4159950 0x7f5bb415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51554/0 (socket says 172.21.15.29:51554) 2023-10-14T16:57:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 -- 172.21.15.29:0/1540249452 learned_addr learned my addr 172.21.15.29:0/1540249452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bb9944640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb4070450 0x7f5bb40708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 -- 172.21.15.29:0/1540249452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb4070450 msgr2=0x7f5bb40708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb4070450 0x7f5bb40708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 -- 172.21.15.29:0/1540249452 >> v1:172.21.15.29:6789/0 conn(0x7f5bb406c100 legacy=0x7f5bb406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:53.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.952+0000 7f5bba145640 1 -- 172.21.15.29:0/1540249452 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb406cc80 con 0x7f5bb4159950 2023-10-14T16:57:53.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.953+0000 7f5bba145640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4159950 0x7f5bb415a9b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009870 tx=0x7f5bb002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e838cd92345fc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:53.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.953+0000 7f5bb9143640 1 -- 172.21.15.29:0/1540249452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0031640 con 0x7f5bb4159950 2023-10-14T16:57:53.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.953+0000 7f5bb9143640 1 -- 172.21.15.29:0/1540249452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb00317a0 con 0x7f5bb4159950 2023-10-14T16:57:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.953+0000 7f5bb9143640 1 -- 172.21.15.29:0/1540249452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bb0031b30 con 0x7f5bb4159950 2023-10-14T16:57:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 -- 172.21.15.29:0/1540249452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4159950 msgr2=0x7f5bb415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4159950 0x7f5bb415a9b0 secure :-1 s=READY pgs=1243 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0009870 tx=0x7f5bb002b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 -- 172.21.15.29:0/1540249452 shutdown_connections 2023-10-14T16:57:53.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb4070450 0x7f5bb40708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1540249452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4159950 0x7f5bb415a9b0 unknown :-1 s=CLOSED pgs=1243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 -- 172.21.15.29:0/1540249452 >> 172.21.15.29:0/1540249452 conn(0x7f5bb40ce6a0 msgr2=0x7f5bb40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:53.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 -- 172.21.15.29:0/1540249452 shutdown_connections 2023-10-14T16:57:53.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.954+0000 7f5bbb147640 1 -- 172.21.15.29:0/1540249452 wait complete. 2023-10-14T16:57:53.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.956+0000 7f5bbb147640 1 Processor -- start 2023-10-14T16:57:53.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.956+0000 7f5bbb147640 1 -- start start 2023-10-14T16:57:53.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 0x7f5bb40c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 0x7f5bb4158cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40b20e0 con 0x7f5bb4070450 2023-10-14T16:57:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb4159210 con 0x7f5bb4159950 2023-10-14T16:57:53.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bbb147640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bb40b1460 con 0x7f5bb406c100 2023-10-14T16:57:53.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.957+0000 7f5bb9944640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51556/0 (socket says 172.21.15.29:51556) 2023-10-14T16:57:53.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 -- 172.21.15.29:0/1944656902 learned_addr learned my addr 172.21.15.29:0/1944656902 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:53.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bba145640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 0x7f5bb40c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 -- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 msgr2=0x7f5bb40c7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 0x7f5bb40c7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bba946640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 0x7f5bb4158cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 -- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 msgr2=0x7f5bb4158cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:53.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 0x7f5bb4158cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:53.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bb9944640 1 -- 172.21.15.29:0/1944656902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb00094d0 con 0x7f5bb4070450 2023-10-14T16:57:53.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bba145640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 0x7f5bb40c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:53.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.958+0000 7f5bba946640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 0x7f5bb4158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:53.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5bb9944640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c7f0 tx=0x7f5ba400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:53.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4004220 con 0x7f5bb4070450 2023-10-14T16:57:53.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ba40043a0 con 0x7f5bb4070450 2023-10-14T16:57:53.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ba4007860 con 0x7f5bb4070450 2023-10-14T16:57:53.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bb40b1740 con 0x7f5bb4070450 2023-10-14T16:57:53.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.959+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bb40b1d00 con 0x7f5bb4070450 2023-10-14T16:57:53.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.960+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bb40b2220 con 0x7f5bb4070450 2023-10-14T16:57:53.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.961+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5ba4007bd0 con 0x7f5bb4070450 2023-10-14T16:57:53.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.963+0000 7f5baaffd640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 0x7f5b9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:53.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.963+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5ba40898f0 con 0x7f5bb4070450 2023-10-14T16:57:53.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.963+0000 7f5bba145640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 0x7f5b9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:53.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.964+0000 7f5bba145640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 0x7f5b9806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0036000 tx=0x7f5bb003c040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:53.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:53.967+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bb40b2220 con 0x7f5bb4070450 2023-10-14T16:57:54.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.192+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 --> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bb415dbf0 con 0x7f5b9806af90 2023-10-14T16:57:54.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.195+0000 7f5baaffd640 1 -- 172.21.15.29:0/1944656902 <== mgr.7257 v2:172.21.15.29:6832/1520757887 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5bb0031ae0 con 0x7f5b9806af90 2023-10-14T16:57:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:57:54.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:57:54.197 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:57:54.197 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:57:54.197 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:57:54.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 msgr2=0x7f5b9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:54.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 0x7f5b9806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5bb0036000 tx=0x7f5bb003c040 comp rx=0 tx=0).stop 2023-10-14T16:57:54.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 msgr2=0x7f5bb4158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:54.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 secure :-1 s=READY pgs=1244 cs=0 l=1 rev1=1 crypto rx=0x7f5ba400c7f0 tx=0x7f5ba400ccb0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 shutdown_connections 2023-10-14T16:57:54.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b9806af90 0x7f5b9806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bb4159950 0x7f5bb4158cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bb4070450 0x7f5bb4158790 unknown :-1 s=CLOSED pgs=1244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 --2- 172.21.15.29:0/1944656902 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bb406c100 0x7f5bb40c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.198+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 >> 172.21.15.29:0/1944656902 conn(0x7f5bb40ce6a0 msgr2=0x7f5bb40c9930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.199+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 shutdown_connections 2023-10-14T16:57:54.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.199+0000 7f5bbb147640 1 -- 172.21.15.29:0/1944656902 wait complete. 2023-10-14T16:57:54.219 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:57:54.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.451+0000 7f5b9359e640 1 Processor -- start 2023-10-14T16:57:54.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.451+0000 7f5b9359e640 1 -- start start 2023-10-14T16:57:54.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.453+0000 7f5b9359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 0x7f5b940c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.453+0000 7f5b9359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b940c9230 0x7f5b940709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.453+0000 7f5b9359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b94160a90 con 0x7f5b94070f20 2023-10-14T16:57:54.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.453+0000 7f5b9359e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b940cdec0 con 0x7f5b940c9230 2023-10-14T16:57:54.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.453+0000 7f5b9359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940cd900 con 0x7f5b9415c9f0 2023-10-14T16:57:54.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b940c9230 0x7f5b940709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b940c9230 0x7f5b940709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51572/0 (socket says 172.21.15.29:51572) 2023-10-14T16:57:54.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 -- 172.21.15.29:0/2937250026 learned_addr learned my addr 172.21.15.29:0/2937250026 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:54.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b9259c640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 0x7f5b940c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 -- 172.21.15.29:0/2937250026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 msgr2=0x7f5b940c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:54.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 0x7f5b940c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 -- 172.21.15.29:0/2937250026 >> v1:172.21.15.186:6789/0 conn(0x7f5b94070f20 legacy=0x7f5b940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:54.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b8bfff640 1 -- 172.21.15.29:0/2937250026 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b940cdbe0 con 0x7f5b940c9230 2023-10-14T16:57:54.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.454+0000 7f5b9259c640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 0x7f5b940c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:54.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.455+0000 7f5b8bfff640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b940c9230 0x7f5b940709e0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009870 tx=0x7f5b7c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11650f7bbe234157 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:54.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.456+0000 7f5b91d9b640 1 -- 172.21.15.29:0/2937250026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c031640 con 0x7f5b940c9230 2023-10-14T16:57:54.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.456+0000 7f5b91d9b640 1 -- 172.21.15.29:0/2937250026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7c0317a0 con 0x7f5b940c9230 2023-10-14T16:57:54.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.456+0000 7f5b91d9b640 1 -- 172.21.15.29:0/2937250026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b7c031b30 con 0x7f5b940c9230 2023-10-14T16:57:54.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.456+0000 7f5b9359e640 1 -- 172.21.15.29:0/2937250026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b940c9230 msgr2=0x7f5b940709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:54.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.456+0000 7f5b9359e640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b940c9230 0x7f5b940709e0 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c009870 tx=0x7f5b7c02b8f0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 -- 172.21.15.29:0/2937250026 shutdown_connections 2023-10-14T16:57:54.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b940c9230 0x7f5b940709e0 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 --2- 172.21.15.29:0/2937250026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b9415c9f0 0x7f5b940c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 -- 172.21.15.29:0/2937250026 >> 172.21.15.29:0/2937250026 conn(0x7f5b940ce6a0 msgr2=0x7f5b940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:54.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 -- 172.21.15.29:0/2937250026 shutdown_connections 2023-10-14T16:57:54.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.457+0000 7f5b9359e640 1 -- 172.21.15.29:0/2937250026 wait complete. 2023-10-14T16:57:54.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.458+0000 7f5b9359e640 1 Processor -- start 2023-10-14T16:57:54.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.459+0000 7f5b9359e640 1 -- start start 2023-10-14T16:57:54.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.459+0000 7f5b9359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b94070f20 0x7f5b94153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.459+0000 7f5b9359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 0x7f5b940c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b9359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b9359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b940c2eb0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b9359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9406afd0 con 0x7f5b940c9230 2023-10-14T16:57:54.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b9359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b9406a630 con 0x7f5b94070f20 2023-10-14T16:57:54.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51584/0 (socket says 172.21.15.29:51584) 2023-10-14T16:57:54.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 -- 172.21.15.29:0/3617705111 learned_addr learned my addr 172.21.15.29:0/3617705111 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:54.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b9259c640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b94070f20 0x7f5b94153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b8bfff640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 0x7f5b940c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 -- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b94070f20 msgr2=0x7f5b94153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b94070f20 0x7f5b94153b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 -- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 msgr2=0x7f5b940c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 0x7f5b940c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b92d9d640 1 -- 172.21.15.29:0/3617705111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0094d0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.460+0000 7f5b8bfff640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 0x7f5b940c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b92d9d640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00b970 tx=0x7f5b8c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c00cb40 con 0x7f5b9415c9f0 2023-10-14T16:57:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b8c00cca0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b8c014680 con 0x7f5b9415c9f0 2023-10-14T16:57:54.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b9406a910 con 0x7f5b9415c9f0 2023-10-14T16:57:54.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.461+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b941540b0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.463+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b9406b110 con 0x7f5b9415c9f0 2023-10-14T16:57:54.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.463+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 45) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5b8c01d3b0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.464+0000 7f5b8affd640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 0x7f5b7406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:54.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.464+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5b8c0895e0 con 0x7f5b9415c9f0 2023-10-14T16:57:54.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.465+0000 7f5b9259c640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 0x7f5b7406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:54.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.465+0000 7f5b9259c640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 0x7f5b7406d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400a990 tx=0x7f5b84008040 comp rx=0 tx=0).ready entity=mgr.7257 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:54.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.469+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b9406b110 con 0x7f5b9415c9f0 2023-10-14T16:57:54.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:54.716+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f5b9406b110 con 0x7f5b9415c9f0 2023-10-14T16:57:55.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.349+0000 7f5b8affd640 1 -- 172.21.15.29:0/3617705111 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v46) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5b9406b110 con 0x7f5b9415c9f0 2023-10-14T16:57:55.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.353+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 msgr2=0x7f5b7406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.353+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 0x7f5b7406d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400a990 tx=0x7f5b84008040 comp rx=0 tx=0).stop 2023-10-14T16:57:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.353+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 msgr2=0x7f5b940c2710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.353+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c00b970 tx=0x7f5b8c00be30 comp rx=0 tx=0).stop 2023-10-14T16:57:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 shutdown_connections 2023-10-14T16:57:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f5b7406afc0 0x7f5b7406d480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b9415c9f0 0x7f5b940c2710 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b940c9230 0x7f5b940c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 --2- 172.21.15.29:0/3617705111 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b94070f20 0x7f5b94153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 >> 172.21.15.29:0/3617705111 conn(0x7f5b940ce6a0 msgr2=0x7f5b940cc4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 shutdown_connections 2023-10-14T16:57:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.354+0000 7f5b9359e640 1 -- 172.21.15.29:0/3617705111 wait complete. 2023-10-14T16:57:55.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:57:55.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.591+0000 7fcddcfd5640 1 Processor -- start 2023-10-14T16:57:55.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.591+0000 7fcddcfd5640 1 -- start start 2023-10-14T16:57:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.592+0000 7fcddcfd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159e90 0x7fcdd80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.592+0000 7fcddcfd5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcdd806c140 0x7fcdd806c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.592+0000 7fcddcfd5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdd8070360 con 0x7fcdd8159ab0 2023-10-14T16:57:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.592+0000 7fcddcfd5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdd806cb10 con 0x7fcdd806c140 2023-10-14T16:57:55.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.592+0000 7fcddcfd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd815d890 con 0x7fcdd8159e90 2023-10-14T16:57:55.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcdd8159ab0 0x7fcdd806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48598/0 (socket says 172.21.15.29:48598) 2023-10-14T16:57:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd659c640 1 -- 172.21.15.29:0/30753012 learned_addr learned my addr 172.21.15.29:0/30753012 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/30753012 >> v2:172.21.15.29:3300/0 conn(0x7fcdd806c140 0x7fcdd806c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd5d9b640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159e90 0x7fcdd80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd559a640 1 -- 172.21.15.29:0/30753012 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 632858829 0 0) 0x7fcdd8070360 con 0x7fcdd8159ab0 2023-10-14T16:57:55.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd559a640 1 -- 172.21.15.29:0/30753012 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdc0005730 con 0x7fcdd8159ab0 2023-10-14T16:57:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/30753012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159e90 msgr2=0x7fcdd80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159e90 0x7fcdd80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/30753012 >> v1:172.21.15.29:6789/0 conn(0x7fcdd8159ab0 legacy=0x7fcdd806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.593+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/30753012 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd80cd1f0 con 0x7fcdd806c140 2023-10-14T16:57:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.594+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd806c140 0x7fcdd806c5d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00097d0 tx=0x7fcdd002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21827a0d742c946d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:55.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.595+0000 7fcdd559a640 1 -- 172.21.15.29:0/30753012 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00316f0 con 0x7fcdd806c140 2023-10-14T16:57:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.595+0000 7fcdd559a640 1 -- 172.21.15.29:0/30753012 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0031850 con 0x7fcdd806c140 2023-10-14T16:57:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.595+0000 7fcdd559a640 1 -- 172.21.15.29:0/30753012 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0031bc0 con 0x7fcdd806c140 2023-10-14T16:57:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.595+0000 7fcddcfd5640 1 -- 172.21.15.29:0/30753012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd806c140 msgr2=0x7fcdd806c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.595+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd806c140 0x7fcdd806c5d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00097d0 tx=0x7fcdd002b790 comp rx=0 tx=0).stop 2023-10-14T16:57:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 -- 172.21.15.29:0/30753012 shutdown_connections 2023-10-14T16:57:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd806c140 0x7fcdd806c5d0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/30753012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159e90 0x7fcdd80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 -- 172.21.15.29:0/30753012 >> 172.21.15.29:0/30753012 conn(0x7fcdd80ce9b0 msgr2=0x7fcdd80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:55.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 -- 172.21.15.29:0/30753012 shutdown_connections 2023-10-14T16:57:55.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.596+0000 7fcddcfd5640 1 -- 172.21.15.29:0/30753012 wait complete. 2023-10-14T16:57:55.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.597+0000 7fcddcfd5640 1 Processor -- start 2023-10-14T16:57:55.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.598+0000 7fcddcfd5640 1 -- start start 2023-10-14T16:57:55.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.598+0000 7fcddcfd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd806c140 0x7fcdd80ace40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.598+0000 7fcddcfd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159ab0 0x7fcdd81ee0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcddcfd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcddcfd5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80c73a0 con 0x7fcdd8159e90 2023-10-14T16:57:55.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcddcfd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80c6790 con 0x7fcdd806c140 2023-10-14T16:57:55.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcddcfd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd80c6a70 con 0x7fcdd8159ab0 2023-10-14T16:57:55.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51624/0 (socket says 172.21.15.29:51624) 2023-10-14T16:57:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/4213906701 learned_addr learned my addr 172.21.15.29:0/4213906701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd5d9b640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159ab0 0x7fcdd81ee0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd659c640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd806c140 0x7fcdd80ace40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:55.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159ab0 msgr2=0x7fcdd81ee0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159ab0 0x7fcdd81ee0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd806c140 msgr2=0x7fcdd80ace40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd806c140 0x7fcdd80ace40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.599+0000 7fcdd6d9d640 1 -- 172.21.15.29:0/4213906701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0009400 con 0x7fcdd8159e90 2023-10-14T16:57:55.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcdd6d9d640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00098f0 tx=0x7fcdd00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:55.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0004230 con 0x7fcdd8159e90 2023-10-14T16:57:55.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdd0004390 con 0x7fcdd8159e90 2023-10-14T16:57:55.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0003810 con 0x7fcdd8159e90 2023-10-14T16:57:55.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdd80c6d50 con 0x7fcdd8159e90 2023-10-14T16:57:55.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.600+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdd80ad380 con 0x7fcdd8159e90 2023-10-14T16:57:55.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.602+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdd80c7550 con 0x7fcdd8159e90 2023-10-14T16:57:55.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.603+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fcdd003b050 con 0x7fcdd8159e90 2023-10-14T16:57:55.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.604+0000 7fcdc6ffd640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 0x7fcdb806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:55.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.604+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fcdd00ab390 con 0x7fcdd8159e90 2023-10-14T16:57:55.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.604+0000 7fcdd659c640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 msgr2=0x7fcdb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:55.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.604+0000 7fcdd659c640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 0x7fcdb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:55.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.609+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcdd80c7550 con 0x7fcdd8159e90 2023-10-14T16:57:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.803+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fcdd804e160 con 0x7fcdd8159e90 2023-10-14T16:57:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.805+0000 7fcdd659c640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 msgr2=0x7fcdb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:55.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.805+0000 7fcdd659c640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 0x7fcdb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.810+0000 7fcdc6ffd640 1 -- 172.21.15.29:0/4213906701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v35)=0 v35) v1 ==== 138+0+0 (secure 0 0 0) 0x7fcdd0076150 con 0x7fcdd8159e90 2023-10-14T16:57:55.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 msgr2=0x7fcdb806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:55.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 0x7fcdb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 msgr2=0x7fcdd806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00098f0 tx=0x7fcdd00046c0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 shutdown_connections 2023-10-14T16:57:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fcdb806afe0 0x7fcdb806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd8159e90 0x7fcdd806b030 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdd8159ab0 0x7fcdd81ee0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 --2- 172.21.15.29:0/4213906701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd806c140 0x7fcdd80ace40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.812+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 >> 172.21.15.29:0/4213906701 conn(0x7fcdd80ce9b0 msgr2=0x7fcdd80c4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.813+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 shutdown_connections 2023-10-14T16:57:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:55.813+0000 7fcddcfd5640 1 -- 172.21.15.29:0/4213906701 wait complete. 2023-10-14T16:57:55.831 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:57:55.912 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:55.911+0000 7f01b8445040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:55.921 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:55.920+0000 7f1733611040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:57:55.970 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:55.969+0000 7f01b8445040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:55.980 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:55.980+0000 7f1733611040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:57:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.043+0000 7f404ffff640 1 Processor -- start 2023-10-14T16:57:56.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.044+0000 7f404ffff640 1 -- start start 2023-10-14T16:57:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.045+0000 7f404ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f40500b8480 0x7f40500cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.045+0000 7f404ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f405015d150 con 0x7f40500d0be0 2023-10-14T16:57:56.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40500d1180 con 0x7f405015cd70 2023-10-14T16:57:56.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40500d1460 con 0x7f40500b8480 2023-10-14T16:57:56.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f40500d0be0 0x7f40500b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48602/0 (socket says 172.21.15.29:48602) 2023-10-14T16:57:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404f7fe640 1 -- 172.21.15.29:0/3705167326 learned_addr learned my addr 172.21.15.29:0/3705167326 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404e7fc640 1 --2- 172.21.15.29:0/3705167326 >> v2:172.21.15.186:3300/0 conn(0x7f40500b8480 0x7f40500cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.046+0000 7f404dffb640 1 -- 172.21.15.29:0/3705167326 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868151463 0 0) 0x7f405015d150 con 0x7f40500d0be0 2023-10-14T16:57:56.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.047+0000 7f404dffb640 1 -- 172.21.15.29:0/3705167326 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4038003520 con 0x7f40500d0be0 2023-10-14T16:57:56.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.047+0000 7f404e7fc640 1 -- 172.21.15.29:0/3705167326 >> v1:172.21.15.186:6789/0 conn(0x7f405015cd70 legacy=0x7f40500d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.047+0000 7f404e7fc640 1 -- 172.21.15.29:0/3705167326 >> v1:172.21.15.29:6789/0 conn(0x7f40500d0be0 legacy=0x7f40500b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.047+0000 7f404e7fc640 1 -- 172.21.15.29:0/3705167326 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40500d1740 con 0x7f40500b8480 2023-10-14T16:57:56.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.048+0000 7f404e7fc640 1 --2- 172.21.15.29:0/3705167326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40500b8480 0x7f40500cd7c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f40440097a0 tx=0x7f404402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9c2aaedcfbffe88 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:56.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404dffb640 1 -- 172.21.15.29:0/3705167326 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4044031830 con 0x7f40500b8480 2023-10-14T16:57:56.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404dffb640 1 -- 172.21.15.29:0/3705167326 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4044031990 con 0x7f40500b8480 2023-10-14T16:57:56.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404dffb640 1 -- 172.21.15.29:0/3705167326 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4044031d00 con 0x7f40500b8480 2023-10-14T16:57:56.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404ffff640 1 -- 172.21.15.29:0/3705167326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40500b8480 msgr2=0x7f40500cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404ffff640 1 --2- 172.21.15.29:0/3705167326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40500b8480 0x7f40500cd7c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f40440097a0 tx=0x7f404402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404ffff640 1 -- 172.21.15.29:0/3705167326 shutdown_connections 2023-10-14T16:57:56.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404ffff640 1 --2- 172.21.15.29:0/3705167326 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40500b8480 0x7f40500cd7c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.049+0000 7f404ffff640 1 -- 172.21.15.29:0/3705167326 >> 172.21.15.29:0/3705167326 conn(0x7f40500cf570 msgr2=0x7f40500cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:56.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.050+0000 7f404ffff640 1 -- 172.21.15.29:0/3705167326 shutdown_connections 2023-10-14T16:57:56.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.050+0000 7f404ffff640 1 -- 172.21.15.29:0/3705167326 wait complete. 2023-10-14T16:57:56.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.051+0000 7f404ffff640 1 Processor -- start 2023-10-14T16:57:56.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.051+0000 7f404ffff640 1 -- start start 2023-10-14T16:57:56.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40500d0be0 0x7f40500ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f405015cd70 0x7f40500caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015e690 con 0x7f40500b8480 2023-10-14T16:57:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f405015e0c0 con 0x7f405015cd70 2023-10-14T16:57:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.052+0000 7f404ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f405015e8e0 con 0x7f40500d0be0 2023-10-14T16:57:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51634/0 (socket says 172.21.15.29:51634) 2023-10-14T16:57:56.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 -- 172.21.15.29:0/3945101779 learned_addr learned my addr 172.21.15.29:0/3945101779 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:56.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404f7fe640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f405015cd70 0x7f40500caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404e7fc640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40500d0be0 0x7f40500ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40500d0be0 msgr2=0x7f40500ca540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40500d0be0 0x7f40500ca540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f405015cd70 msgr2=0x7f40500caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f405015cd70 0x7f40500caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.053+0000 7f404effd640 1 -- 172.21.15.29:0/3945101779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4044009400 con 0x7f40500b8480 2023-10-14T16:57:56.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f404effd640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f404000c400 tx=0x7f404000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:56.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40400052d0 con 0x7f40500b8480 2023-10-14T16:57:56.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4040005450 con 0x7f40500b8480 2023-10-14T16:57:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4040007260 con 0x7f40500b8480 2023-10-14T16:57:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f405015ebc0 con 0x7f40500b8480 2023-10-14T16:57:56.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.054+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40501f25a0 con 0x7f40500b8480 2023-10-14T16:57:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.055+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f405004e9b0 con 0x7f40500b8480 2023-10-14T16:57:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.057+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f40400073c0 con 0x7f40500b8480 2023-10-14T16:57:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.058+0000 7f40377fe640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 0x7f403c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.058+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f4040088d80 con 0x7f40500b8480 2023-10-14T16:57:56.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.058+0000 7f404e7fc640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 msgr2=0x7f403c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:56.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.058+0000 7f404e7fc640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 0x7f403c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:56.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.063+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4040053bd0 con 0x7f40500b8480 2023-10-14T16:57:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.259+0000 7f404e7fc640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 msgr2=0x7f403c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.259+0000 7f404e7fc640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 0x7f403c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:56.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.263+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f40501506d0 con 0x7f40500b8480 2023-10-14T16:57:56.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.270+0000 7f40377fe640 1 -- 172.21.15.29:0/3945101779 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v36)=0 v36) v1 ==== 133+0+0 (secure 0 0 0) 0x7f40400571f0 con 0x7f40500b8480 2023-10-14T16:57:56.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.272+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 msgr2=0x7f403c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:56.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.272+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 0x7f403c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.272+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 msgr2=0x7f40501500d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 crypto rx=0x7f404000c400 tx=0x7f404000c8c0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 shutdown_connections 2023-10-14T16:57:56.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7f403c06afe0 0x7f403c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f405015cd70 0x7f40500caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40500d0be0 0x7f40500ca540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 --2- 172.21.15.29:0/3945101779 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40500b8480 0x7f40501500d0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 >> 172.21.15.29:0/3945101779 conn(0x7f40500cf570 msgr2=0x7f40500b79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:56.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 shutdown_connections 2023-10-14T16:57:56.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.273+0000 7f404ffff640 1 -- 172.21.15.29:0/3945101779 wait complete. 2023-10-14T16:57:56.291 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /bad-path 1m 2023-10-14T16:57:56.437 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.436+0000 7f1733611040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:56.450 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.449+0000 7f01b8445040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:57:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.481+0000 7fbea23fb640 1 Processor -- start 2023-10-14T16:57:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.481+0000 7fbea23fb640 1 -- start start 2023-10-14T16:57:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbea23fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe9c0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbea23fb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0cb030 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbea23fb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0c91c0 con 0x7fbe9c15ad70 2023-10-14T16:57:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbea23fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c94a0 con 0x7fbe9c0caac0 2023-10-14T16:57:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbe93fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe9c06a4b0 0x7fbe9c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48604/0 (socket says 172.21.15.29:48604) 2023-10-14T16:57:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.483+0000 7fbe93fff640 1 -- 172.21.15.29:0/3857843844 learned_addr learned my addr 172.21.15.29:0/3857843844 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:57:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbea0998640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe9c0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3529149282 0 0) 0x7fbe9c0cb030 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe88003520 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3620748584 0 0) 0x7fbe88003520 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 msgr2=0x7fbe9c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe9c0ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 >> v1:172.21.15.186:6789/0 conn(0x7fbe9c15ad70 legacy=0x7fbe9c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0c9780 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fbe84001670 con 0x7fbe9c06a4b0 2023-10-14T16:57:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 >> v1:172.21.15.29:6789/0 conn(0x7fbe9c06a4b0 legacy=0x7fbe9c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.484+0000 7fbe9b7fe640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe880039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9b7fe640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe88003f20 0x7fbe880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9b7fe640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe88004d00 0x7fbe88005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c0c94a0 con 0x7fbe88004d00 2023-10-14T16:57:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800ba60 con 0x7fbe88003f20 2023-10-14T16:57:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8800bc20 con 0x7fbe9c0caac0 2023-10-14T16:57:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe88004d00 0x7fbe88005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe93fff640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe88003f20 0x7fbe880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe880039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 -- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 msgr2=0x7fbe880039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe880039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 -- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe88003f20 msgr2=0x7fbe880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe88003f20 0x7fbe880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbea0998640 1 -- 172.21.15.29:0/3857843844 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe88003520 con 0x7fbe88004d00 2023-10-14T16:57:56.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.485+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe9c0caac0 0x7fbe880039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:57:56.492 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.488+0000 7f1733611040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.487+0000 7fbea0998640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe88004d00 0x7fbe88005180 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009fc0 tx=0x7fbe880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bf67a6cb5e68f70 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.487+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe9402cbf0 con 0x7fbe88004d00 2023-10-14T16:57:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.487+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe9400f040 con 0x7fbe88004d00 2023-10-14T16:57:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.487+0000 7fbe9b7fe640 1 -- 172.21.15.29:0/3857843844 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe94034440 con 0x7fbe88004d00 2023-10-14T16:57:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 -- 172.21.15.29:0/3857843844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe88004d00 msgr2=0x7fbe88005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 --2- 172.21.15.29:0/3857843844 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe88004d00 0x7fbe88005180 secure :-1 s=READY pgs=1255 cs=0 l=1 rev1=1 crypto rx=0x7fbe94009fc0 tx=0x7fbe880057f0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea0998640 1 -- 172.21.15.29:0/3857843844 reap_dead start 2023-10-14T16:57:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 -- 172.21.15.29:0/3857843844 shutdown_connections 2023-10-14T16:57:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 -- 172.21.15.29:0/3857843844 >> 172.21.15.29:0/3857843844 conn(0x7fbe9c0ce9b0 msgr2=0x7fbe9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:57:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 -- 172.21.15.29:0/3857843844 shutdown_connections 2023-10-14T16:57:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.488+0000 7fbea23fb640 1 -- 172.21.15.29:0/3857843844 wait complete. 2023-10-14T16:57:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.491+0000 7fbea23fb640 1 Processor -- start 2023-10-14T16:57:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.491+0000 7fbea23fb640 1 -- start start 2023-10-14T16:57:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe88004d00 0x7fbe9c1454d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 0x7fbe9c156d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe9c15c4a0 0x7fbe9c15c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c148df0 con 0x7fbe9c156910 2023-10-14T16:57:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c156480 con 0x7fbe9c15c4a0 2023-10-14T16:57:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea23fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe9c155bb0 con 0x7fbe88004d00 2023-10-14T16:57:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 0x7fbe9c156d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 0x7fbe9c156d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51664/0 (socket says 172.21.15.29:51664) 2023-10-14T16:57:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 -- 172.21.15.29:0/3019475527 learned_addr learned my addr 172.21.15.29:0/3019475527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:57:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe88004d00 msgr2=0x7fbe9c1454d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe88004d00 0x7fbe9c1454d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe9c15c4a0 msgr2=0x7fbe9c15c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:57:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbea0998640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe9c15c4a0 0x7fbe9c15c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:57:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe9c15c4a0 0x7fbe9c15c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:57:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.493+0000 7fbe93fff640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c148b10 con 0x7fbe9c156910 2023-10-14T16:57:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbe93fff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 0x7fbe9c156d70 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e7f0 tx=0x7fbe8400ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:57:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400c8d0 con 0x7fbe9c156910 2023-10-14T16:57:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe8400ca30 con 0x7fbe9c156910 2023-10-14T16:57:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe8400cde0 con 0x7fbe9c156910 2023-10-14T16:57:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c155e30 con 0x7fbe9c156910 2023-10-14T16:57:56.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.494+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c145a10 con 0x7fbe9c156910 2023-10-14T16:57:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.495+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c04e160 con 0x7fbe9c156910 2023-10-14T16:57:56.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.497+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 46) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbe8401d460 con 0x7fbe9c156910 2023-10-14T16:57:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.498+0000 7fbe997fa640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:57:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.498+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fbe84089890 con 0x7fbe9c156910 2023-10-14T16:57:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.498+0000 7fbe9bfff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:56.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.498+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:57:56.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.503+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe84054760 con 0x7fbe9c156910 2023-10-14T16:57:56.509 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.509+0000 7f01b8445040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:57:56.540 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.539+0000 7f1733611040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:56.562 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.562+0000 7f01b8445040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:57:56.600 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.599+0000 7f1733611040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:56.629 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.626+0000 7f01b8445040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:57:56.649 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.649+0000 7f1733611040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:56.678 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.677+0000 7f01b8445040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:57:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.699+0000 7fbe9bfff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:56.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.699+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:57:56.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:56.719+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe9c0c9ca0 con 0x7fbe7c06b010 2023-10-14T16:57:56.759 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.759+0000 7f1733611040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:56.795 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.794+0000 7f01b8445040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:57:56.822 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.821+0000 7f1733611040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:56.861 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.860+0000 7f01b8445040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:56.872 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.871+0000 7f1733611040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:56.912 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.911+0000 7f01b8445040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:57:56.929 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.928+0000 7f1733611040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:56.972 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:56.971+0000 7f01b8445040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:57:56.979 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:56.978+0000 7f1733611040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:57.024 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.023+0000 7f01b8445040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:57:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:57.100+0000 7fbe9bfff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:57.100+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:57:57.363 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.362+0000 7f1733611040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:57.423 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.423+0000 7f01b8445040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:57:57.524 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.523+0000 7f1733611040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:57.592 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.591+0000 7f01b8445040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:57:57.630 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.630+0000 7f1733611040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:57.680 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.679+0000 7f1733611040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:57.705 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.704+0000 7f01b8445040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:57:57.757 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.757+0000 7f01b8445040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:57:57.787 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.786+0000 7f1733611040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:57.870 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.870+0000 7f01b8445040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:57.893 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:57.892+0000 7f1733611040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:57.901+0000 7fbe9bfff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:57.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:57.901+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:57:57.981 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:57.981+0000 7f01b8445040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:57:58.050 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:58.049+0000 7f1733611040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:58.099 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:58.098+0000 7f1733611040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:58.145 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:58.144+0000 7f01b8445040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:57:58.196 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:58.196+0000 7f01b8445040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:57:58.218 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:58.216+0000 7f1733611040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:58.321 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:58.320+0000 7f01b8445040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:57:59.033 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:59.031+0000 7f1733611040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:59.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:59.169+0000 7f01b8445040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:57:59.325 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:59.324+0000 7f1733611040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:59.382 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:59.381+0000 7f1733611040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:59.480 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:59.479+0000 7f01b8445040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:57:59.495 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:57:59.494+0000 7f1733611040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.503+0000 7fbe9bfff640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1520757887 2023-10-14T16:57:59.540 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:59.540+0000 7f01b8445040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:57:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.503+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:57:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.567+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 47) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fbe84054b30 con 0x7fbe9c156910 2023-10-14T16:57:59.664 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:57:59.664+0000 7f01b8445040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:57:59.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.683+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 48) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fbe84087ce0 con 0x7fbe9c156910 2023-10-14T16:57:59.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.683+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 msgr2=0x7fbe7c06d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:57:59.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:57:59.683+0000 7fbe997fa640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/1520757887,v1:172.21.15.29:6833/1520757887] conn(0x7fbe7c06b010 0x7fbe7c06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:00.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0136f92640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0136f92640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0136f92640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0136f92640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.013 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0136f92640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.013 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0134f4e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.013 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0134f4e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.013 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0134f4e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.013 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0134f4e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.014 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:58:00.011+0000 7f0134f4e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:58:00.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.694+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 49) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fbe84055400 con 0x7fbe9c156910 2023-10-14T16:58:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.695+0000 7fbe997fa640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fbe7c027d00 0x7fbe7c074f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:00.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.695+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 --> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/bad-path", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe7c00b680 con 0x7fbe7c027d00 2023-10-14T16:58:00.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.703+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fbe7c027d00 0x7fbe7c074f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:00.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.708+0000 7fbe9bfff640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fbe7c027d00 0x7fbe7c074f60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0078d0 tx=0x7fbe8c007d20 comp rx=0 tx=0).ready entity=mgr.7680 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:58:00.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.907+0000 7fbe997fa640 1 -- 172.21.15.29:0/3019475527 <== mgr.7680 v2:172.21.15.29:6832/4145442506 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fbe7c00b680 con 0x7fbe7c027d00 2023-10-14T16:58:00.909 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /bad-path 2023-10-14T16:58:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fbe7c027d00 msgr2=0x7fbe7c074f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fbe7c027d00 0x7fbe7c074f60 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fbe8c0078d0 tx=0x7fbe8c007d20 comp rx=0 tx=0).stop 2023-10-14T16:58:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 msgr2=0x7fbe9c156d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 --2- 172.21.15.29:0/3019475527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe9c156910 0x7fbe9c156d70 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 crypto rx=0x7fbe8400e7f0 tx=0x7fbe8400ecb0 comp rx=0 tx=0).stop 2023-10-14T16:58:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea0998640 1 -- 172.21.15.29:0/3019475527 reap_dead start 2023-10-14T16:58:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 shutdown_connections 2023-10-14T16:58:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 >> 172.21.15.29:0/3019475527 conn(0x7fbe9c0ce9b0 msgr2=0x7fbe9c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:58:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 shutdown_connections 2023-10-14T16:58:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:00.911+0000 7fbea23fb640 1 -- 172.21.15.29:0/3019475527 wait complete. 2023-10-14T16:58:00.933 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /bad-path 2023-10-14T16:58:00.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-14T16:58:01.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.153+0000 7fe776bcc640 1 Processor -- start 2023-10-14T16:58:01.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.153+0000 7fe776bcc640 1 -- start start 2023-10-14T16:58:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe776bcc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe77006a4b0 0x7fe77016c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe776bcc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700caef0 con 0x7fe7700addb0 2023-10-14T16:58:01.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe776bcc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7700cb140 con 0x7fe7700caa80 2023-10-14T16:58:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe776bcc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe770161a50 con 0x7fe77006a4b0 2023-10-14T16:58:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe774968640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe7700addb0 0x7fe7700c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55584/0 (socket says 172.21.15.29:55584) 2023-10-14T16:58:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe774968640 1 -- 172.21.15.29:0/77319612 learned_addr learned my addr 172.21.15.29:0/77319612 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:58:01.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.155+0000 7fe775169640 1 --2- 172.21.15.29:0/77319612 >> v2:172.21.15.29:3300/0 conn(0x7fe77006a4b0 0x7fe77016c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.156+0000 7fe775169640 1 -- 172.21.15.29:0/77319612 >> v1:172.21.15.186:6789/0 conn(0x7fe7700caa80 legacy=0x7fe770069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:58:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.156+0000 7fe775169640 1 -- 172.21.15.29:0/77319612 >> v1:172.21.15.29:6789/0 conn(0x7fe7700addb0 legacy=0x7fe7700c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.156+0000 7fe775169640 1 -- 172.21.15.29:0/77319612 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe77006b030 con 0x7fe77006a4b0 2023-10-14T16:58:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.156+0000 7fe76f7fe640 1 -- 172.21.15.29:0/77319612 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753304876 0 0) 0x7fe7700caef0 con 0x7fe7700addb0 2023-10-14T16:58:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.157+0000 7fe775169640 1 --2- 172.21.15.29:0/77319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77006a4b0 0x7fe77016c400 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=323c8fd8831e14e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:58:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.157+0000 7fe76f7fe640 1 -- 172.21.15.29:0/77319612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76802cc60 con 0x7fe77006a4b0 2023-10-14T16:58:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.157+0000 7fe76f7fe640 1 -- 172.21.15.29:0/77319612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76802cdc0 con 0x7fe77006a4b0 2023-10-14T16:58:01.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe76f7fe640 1 -- 172.21.15.29:0/77319612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe768034660 con 0x7fe77006a4b0 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 -- 172.21.15.29:0/77319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77006a4b0 msgr2=0x7fe77016c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 --2- 172.21.15.29:0/77319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77006a4b0 0x7fe77016c400 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe76802beb0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 -- 172.21.15.29:0/77319612 shutdown_connections 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 --2- 172.21.15.29:0/77319612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe77006a4b0 0x7fe77016c400 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 -- 172.21.15.29:0/77319612 >> 172.21.15.29:0/77319612 conn(0x7fe7700ce9b0 msgr2=0x7fe7700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:58:01.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 -- 172.21.15.29:0/77319612 shutdown_connections 2023-10-14T16:58:01.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.158+0000 7fe776bcc640 1 -- 172.21.15.29:0/77319612 wait complete. 2023-10-14T16:58:01.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.160+0000 7fe776bcc640 1 Processor -- start 2023-10-14T16:58:01.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.160+0000 7fe776bcc640 1 -- start start 2023-10-14T16:58:01.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 0x7fe77015beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:01.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7700addb0 0x7fe7700c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:01.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:01.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770163a30 con 0x7fe7700caa80 2023-10-14T16:58:01.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe770162db0 con 0x7fe77006a4b0 2023-10-14T16:58:01.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe776bcc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe770163090 con 0x7fe7700addb0 2023-10-14T16:58:01.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.161+0000 7fe775169640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:01.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54778/0 (socket says 172.21.15.29:54778) 2023-10-14T16:58:01.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 -- 172.21.15.29:0/337255276 learned_addr learned my addr 172.21.15.29:0/337255276 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:58:01.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe774968640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 0x7fe77015beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:01.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 -- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7700addb0 msgr2=0x7fe7700c24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe76ffff640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7700addb0 0x7fe7700c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:01.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7700addb0 0x7fe7700c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 -- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 msgr2=0x7fe77015beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 0x7fe77015beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 -- 172.21.15.29:0/337255276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe768009a70 con 0x7fe7700caa80 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe774968640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 0x7fe77015beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.162+0000 7fe775169640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe768004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.163+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76803e610 con 0x7fe7700caa80 2023-10-14T16:58:01.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.163+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe76803e790 con 0x7fe7700caa80 2023-10-14T16:58:01.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.163+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe770163370 con 0x7fe7700caa80 2023-10-14T16:58:01.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.163+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe76803eb40 con 0x7fe7700caa80 2023-10-14T16:58:01.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.163+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7700c60e0 con 0x7fe7700caa80 2023-10-14T16:58:01.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.164+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe770163b70 con 0x7fe7700caa80 2023-10-14T16:58:01.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.165+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 49) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fe76803b050 con 0x7fe7700caa80 2023-10-14T16:58:01.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.166+0000 7fe76d7fa640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 0x7fe75406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:58:01.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.166+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fe7680abf70 con 0x7fe7700caa80 2023-10-14T16:58:01.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.166+0000 7fe774968640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 0x7fe75406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:58:01.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.167+0000 7fe774968640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 0x7fe75406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f60 tx=0x7fe764009040 comp rx=0 tx=0).ready entity=mgr.7680 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:58:01.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.171+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe770163b70 con 0x7fe7700caa80 2023-10-14T16:58:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.396+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 --> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe77006b5f0 con 0x7fe75406ad10 2023-10-14T16:58:01.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.410+0000 7fe76d7fa640 1 -- 172.21.15.29:0/337255276 <== mgr.7680 v2:172.21.15.29:6832/4145442506 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+302 (secure 0 0 0) 0x7fe7640013d0 con 0x7fe75406ad10 2023-10-14T16:58:01.411 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:58:01.412 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:58:00", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-14T16:58:01.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.412+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 msgr2=0x7fe75406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.412+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 0x7fe75406d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fe764000f60 tx=0x7fe764009040 comp rx=0 tx=0).stop 2023-10-14T16:58:01.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.412+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 msgr2=0x7fe7700c8440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:58:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.412+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 crypto rx=0x7fe768009d90 tx=0x7fe768004a40 comp rx=0 tx=0).stop 2023-10-14T16:58:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 shutdown_connections 2023-10-14T16:58:01.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fe75406ad10 0x7fe75406d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7700caa80 0x7fe7700c8440 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7700addb0 0x7fe7700c24a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 --2- 172.21.15.29:0/337255276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe77006a4b0 0x7fe77015beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:58:01.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 >> 172.21.15.29:0/337255276 conn(0x7fe7700ce9b0 msgr2=0x7fe7700c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:58:01.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 shutdown_connections 2023-10-14T16:58:01.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:58:01.413+0000 7fe776bcc640 1 -- 172.21.15.29:0/337255276 wait complete. 2023-10-14T16:58:01.433 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:58:00", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-14T16:58:16.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:58:16.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:58:46.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:58:46.301 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:59:00.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:00.938+0000 7f0127733640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:00.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:00.938+0000 7f0127733640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:00.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:00.938+0000 7f0127733640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:00.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:00.938+0000 7f0127733640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:00.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:00.938+0000 7f0127733640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:01.435 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /bad-path 2023-10-14T16:59:01.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.666+0000 7f7853200640 1 Processor -- start 2023-10-14T16:59:01.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.666+0000 7f7853200640 1 -- start start 2023-10-14T16:59:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7853200640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f784c15ad70 0x7f784c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7853200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0c9ca0 0x7f784c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7853200640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f784c0cb030 con 0x7f784c0ca3b0 2023-10-14T16:59:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7853200640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f784c0c91c0 con 0x7f784c15ad70 2023-10-14T16:59:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7853200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0c94a0 con 0x7f784c0c9ca0 2023-10-14T16:59:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.668+0000 7f7850f9c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f784c15ad70 0x7f784c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:01.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f784c15ad70 0x7f784c069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35486/0 (socket says 172.21.15.29:35486) 2023-10-14T16:59:01.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 -- 172.21.15.29:0/2554721281 learned_addr learned my addr 172.21.15.29:0/2554721281 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 -- 172.21.15.29:0/2554721281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0c9ca0 msgr2=0x7f784c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 --2- 172.21.15.29:0/2554721281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0c9ca0 0x7f784c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 -- 172.21.15.29:0/2554721281 >> v1:172.21.15.186:6789/0 conn(0x7f784c0ca3b0 legacy=0x7f784c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.669+0000 7f7850f9c640 1 -- 172.21.15.29:0/2554721281 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f784c0c9780 con 0x7f784c15ad70 2023-10-14T16:59:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.670+0000 7f7850f9c640 1 --2- 172.21.15.29:0/2554721281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c15ad70 0x7f784c069da0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7848009e50 tx=0x7f784802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6dc836aeafdcd2cb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f78437fe640 1 -- 172.21.15.29:0/2554721281 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784802ca70 con 0x7f784c15ad70 2023-10-14T16:59:01.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f78437fe640 1 -- 172.21.15.29:0/2554721281 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784802cbd0 con 0x7f784c15ad70 2023-10-14T16:59:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f78437fe640 1 -- 172.21.15.29:0/2554721281 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78480343f0 con 0x7f784c15ad70 2023-10-14T16:59:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 -- 172.21.15.29:0/2554721281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c15ad70 msgr2=0x7f784c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 --2- 172.21.15.29:0/2554721281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c15ad70 0x7f784c069da0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 crypto rx=0x7f7848009e50 tx=0x7f784802bed0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 -- 172.21.15.29:0/2554721281 shutdown_connections 2023-10-14T16:59:01.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 --2- 172.21.15.29:0/2554721281 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0c9ca0 0x7f784c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 --2- 172.21.15.29:0/2554721281 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c15ad70 0x7f784c069da0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 -- 172.21.15.29:0/2554721281 >> 172.21.15.29:0/2554721281 conn(0x7f784c0ce9b0 msgr2=0x7f784c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 -- 172.21.15.29:0/2554721281 shutdown_connections 2023-10-14T16:59:01.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.671+0000 7f7853200640 1 -- 172.21.15.29:0/2554721281 wait complete. 2023-10-14T16:59:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.673+0000 7f7853200640 1 Processor -- start 2023-10-14T16:59:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.673+0000 7f7853200640 1 -- start start 2023-10-14T16:59:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0ca3b0 0x7f784c0c2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f784c15ad70 0x7f784c0cd700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0c2e50 con 0x7f784c0c9ca0 2023-10-14T16:59:01.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0c3130 con 0x7f784c15ad70 2023-10-14T16:59:01.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7853200640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f784c0c3410 con 0x7f784c0ca3b0 2023-10-14T16:59:01.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.674+0000 7f7850f9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:01.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35500/0 (socket says 172.21.15.29:35500) 2023-10-14T16:59:01.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 -- 172.21.15.29:0/771538835 learned_addr learned my addr 172.21.15.29:0/771538835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 -- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0ca3b0 msgr2=0x7f784c0c2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0ca3b0 0x7f784c0c2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 -- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f784c15ad70 msgr2=0x7f784c0cd700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f784c15ad70 0x7f784c0cd700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 -- 172.21.15.29:0/771538835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7848009b30 con 0x7f784c0c9ca0 2023-10-14T16:59:01.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.675+0000 7f7850f9c640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7848009e50 tx=0x7f7848004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:01.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.676+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784803e450 con 0x7f784c0c9ca0 2023-10-14T16:59:01.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.676+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784803e5b0 con 0x7f784c0c9ca0 2023-10-14T16:59:01.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.676+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f784803e960 con 0x7f784c0c9ca0 2023-10-14T16:59:01.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.676+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f784c0c36f0 con 0x7f784c0c9ca0 2023-10-14T16:59:01.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.676+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f784c15a2d0 con 0x7f784c0c9ca0 2023-10-14T16:59:01.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.677+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f784c0c3c10 con 0x7f784c0c9ca0 2023-10-14T16:59:01.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.678+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 51) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f784803eac0 con 0x7f784c0c9ca0 2023-10-14T16:59:01.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.679+0000 7f78417fa640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 0x7f783806d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:01.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.680+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f78480aca70 con 0x7f784c0c9ca0 2023-10-14T16:59:01.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.680+0000 7f7843fff640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 0x7f783806d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:01.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.680+0000 7f7843fff640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 0x7f783806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f784c0ce280 tx=0x7f783c006c90 comp rx=0 tx=0).ready entity=mgr.7680 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:01.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.685+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f784c0c3c10 con 0x7f784c0c9ca0 2023-10-14T16:59:01.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.906+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 --> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/bad-path", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f784c0c3c10 con 0x7f783806b070 2023-10-14T16:59:01.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.925+0000 7f78417fa640 1 -- 172.21.15.29:0/771538835 <== mgr.7680 v2:172.21.15.29:6832/4145442506 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+303 (secure 0 0 0) 0x7f784c0c3c10 con 0x7f783806b070 2023-10-14T16:59:01.927 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:01.927 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:58:00", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-14T16:59:01.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.928+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 msgr2=0x7f783806d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.928+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 0x7f783806d530 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f784c0ce280 tx=0x7f783c006c90 comp rx=0 tx=0).stop 2023-10-14T16:59:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.928+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 msgr2=0x7f784c159b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.928+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 crypto rx=0x7f7848009e50 tx=0x7f7848004a40 comp rx=0 tx=0).stop 2023-10-14T16:59:01.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 shutdown_connections 2023-10-14T16:59:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f783806b070 0x7f783806d530 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f784c15ad70 0x7f784c0cd700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f784c0ca3b0 0x7f784c0c2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 --2- 172.21.15.29:0/771538835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f784c0c9ca0 0x7f784c159b20 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 >> 172.21.15.29:0/771538835 conn(0x7f784c0ce9b0 msgr2=0x7f784c06b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 shutdown_connections 2023-10-14T16:59:01.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:01.929+0000 7f7853200640 1 -- 172.21.15.29:0/771538835 wait complete. 2023-10-14T16:59:01.949 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/bad-path", "rel_path": "/bad-path", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T16:58:00", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": false}] 2023-10-14T16:59:01.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /bad-path 2023-10-14T16:59:02.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.179+0000 7faf5950a640 1 Processor -- start 2023-10-14T16:59:02.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.179+0000 7faf5950a640 1 -- start start 2023-10-14T16:59:02.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf5950a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 0x7faf54165720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf5950a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf5406f1d0 con 0x7faf54144440 2023-10-14T16:59:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf5950a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf540cd610 con 0x7faf54144060 2023-10-14T16:59:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf5950a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf540cd430 con 0x7faf5415ecb0 2023-10-14T16:59:02.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf4bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faf54144440 0x7faf5415e5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49444/0 (socket says 172.21.15.29:49444) 2023-10-14T16:59:02.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.181+0000 7faf4bfff640 1 -- 172.21.15.29:0/3099843205 learned_addr learned my addr 172.21.15.29:0/3099843205 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:02.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf58d09640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 0x7faf54165720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3656165370 0 0) 0x7faf5406f1d0 con 0x7faf54144440 2023-10-14T16:59:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf40003520 con 0x7faf54144440 2023-10-14T16:59:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 145677544 0 0) 0x7faf40003520 con 0x7faf54144440 2023-10-14T16:59:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 msgr2=0x7faf54165720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:02.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 0x7faf54165720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 >> v1:172.21.15.186:6789/0 conn(0x7faf54144060 legacy=0x7faf540b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf540cd8f0 con 0x7faf54144440 2023-10-14T16:59:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7faf3c001670 con 0x7faf54144440 2023-10-14T16:59:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 >> v1:172.21.15.29:6789/0 conn(0x7faf54144440 legacy=0x7faf5415e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf400039a0 0x7faf40003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf400047c0 0x7faf40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 0x7faf400051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf540cd430 con 0x7faf400047c0 2023-10-14T16:59:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf4000bab0 con 0x7faf400039a0 2023-10-14T16:59:02.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.182+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf4000bc70 con 0x7faf5415ecb0 2023-10-14T16:59:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf400047c0 0x7faf40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf53fff640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf400039a0 0x7faf40003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 -- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 msgr2=0x7faf400051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf5415ecb0 0x7faf400051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 -- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf400039a0 msgr2=0x7faf40003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf400039a0 0x7faf40003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.183+0000 7faf4bfff640 1 -- 172.21.15.29:0/3099843205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf40003520 con 0x7faf400047c0 2023-10-14T16:59:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf4bfff640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf400047c0 0x7faf40004c40 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7faf540ae930 tx=0x7faf400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e554ece6bfee125 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c005ad0 con 0x7faf400047c0 2023-10-14T16:59:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf3c005c30 con 0x7faf400047c0 2023-10-14T16:59:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf537fe640 1 -- 172.21.15.29:0/3099843205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf3c02c850 con 0x7faf400047c0 2023-10-14T16:59:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf5950a640 1 -- 172.21.15.29:0/3099843205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf400047c0 msgr2=0x7faf40004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf5950a640 1 --2- 172.21.15.29:0/3099843205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf400047c0 0x7faf40004c40 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 crypto rx=0x7faf540ae930 tx=0x7faf400057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.185+0000 7faf58d09640 1 -- 172.21.15.29:0/3099843205 reap_dead start 2023-10-14T16:59:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.186+0000 7faf5950a640 1 -- 172.21.15.29:0/3099843205 shutdown_connections 2023-10-14T16:59:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.186+0000 7faf5950a640 1 -- 172.21.15.29:0/3099843205 >> 172.21.15.29:0/3099843205 conn(0x7faf54069da0 msgr2=0x7faf5406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.186+0000 7faf5950a640 1 -- 172.21.15.29:0/3099843205 shutdown_connections 2023-10-14T16:59:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.186+0000 7faf5950a640 1 -- 172.21.15.29:0/3099843205 wait complete. 2023-10-14T16:59:02.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.187+0000 7faf5950a640 1 Processor -- start 2023-10-14T16:59:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.188+0000 7faf5950a640 1 -- start start 2023-10-14T16:59:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.188+0000 7faf5950a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf400047c0 0x7faf541e7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.188+0000 7faf5950a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 0x7faf541f5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf5950a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf5950a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541e5490 con 0x7faf540b0100 2023-10-14T16:59:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf5950a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf541e4ea0 con 0x7faf541f5900 2023-10-14T16:59:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf5950a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf540b0ac0 con 0x7faf400047c0 2023-10-14T16:59:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35516/0 (socket says 172.21.15.29:35516) 2023-10-14T16:59:02.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 -- 172.21.15.29:0/1242156686 learned_addr learned my addr 172.21.15.29:0/1242156686 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf4bfff640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 0x7faf541f5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 -- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf400047c0 msgr2=0x7faf541e7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf400047c0 0x7faf541e7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 -- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 msgr2=0x7faf541f5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 0x7faf541f5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf58d09640 1 -- 172.21.15.29:0/1242156686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf541f6380 con 0x7faf540b0100 2023-10-14T16:59:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.189+0000 7faf4bfff640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 0x7faf541f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf58d09640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7faf5406f180 tx=0x7faf4c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c010040 con 0x7faf540b0100 2023-10-14T16:59:02.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf4c00bce0 con 0x7faf540b0100 2023-10-14T16:59:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf4c014640 con 0x7faf540b0100 2023-10-14T16:59:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf541f6660 con 0x7faf540b0100 2023-10-14T16:59:02.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.190+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf54159f80 con 0x7faf540b0100 2023-10-14T16:59:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.191+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf5415e3a0 con 0x7faf540b0100 2023-10-14T16:59:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.192+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 51) v1 ==== 89405+0+0 (secure 0 0 0) 0x7faf4c0147a0 con 0x7faf540b0100 2023-10-14T16:59:02.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.193+0000 7faf517fa640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 0x7faf3406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.193+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7faf4c08a450 con 0x7faf540b0100 2023-10-14T16:59:02.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.194+0000 7faf53fff640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 0x7faf3406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.194+0000 7faf53fff640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 0x7faf3406d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf44007fd0 tx=0x7faf44009040 comp rx=0 tx=0).ready entity=mgr.7680 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.197+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faf4c055080 con 0x7faf540b0100 2023-10-14T16:59:02.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.388+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 52) v1 ==== 89462+0+0 (secure 0 0 0) 0x7faf4c055450 con 0x7faf540b0100 2023-10-14T16:59:02.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.425+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 --> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/bad-path", "target": ["mon-mgr", ""]}) v1 -- 0x7faf540cdbd0 con 0x7faf3406b070 2023-10-14T16:59:02.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.612+0000 7faf517fa640 1 -- 172.21.15.29:0/1242156686 <== mgr.7680 v2:172.21.15.29:6832/4145442506 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+36 (secure 0 0 0) 0x7faf44007b30 con 0x7faf3406b070 2023-10-14T16:59:02.613 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /bad-path 2023-10-14T16:59:02.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 msgr2=0x7faf3406d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 0x7faf3406d530 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7faf44007fd0 tx=0x7faf44009040 comp rx=0 tx=0).stop 2023-10-14T16:59:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 msgr2=0x7faf540b0580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 crypto rx=0x7faf5406f180 tx=0x7faf4c00b480 comp rx=0 tx=0).stop 2023-10-14T16:59:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 shutdown_connections 2023-10-14T16:59:02.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7faf3406b070 0x7faf3406d530 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faf540b0100 0x7faf540b0580 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faf541f5900 0x7faf541f5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 --2- 172.21.15.29:0/1242156686 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faf400047c0 0x7faf541e7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 >> 172.21.15.29:0/1242156686 conn(0x7faf54069da0 msgr2=0x7faf540cc8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:02.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 shutdown_connections 2023-10-14T16:59:02.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.616+0000 7faf5950a640 1 -- 172.21.15.29:0/1242156686 wait complete. 2023-10-14T16:59:02.638 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /bad-path 2023-10-14T16:59:02.638 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T16:59:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.867+0000 7fb6e22a2640 1 Processor -- start 2023-10-14T16:59:02.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.867+0000 7fb6e22a2640 1 -- start start 2023-10-14T16:59:02.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e22a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6dc159ab0 0x7fb6dc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e22a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc06fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e22a2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6dc0cd1f0 con 0x7fb6dc06c140 2023-10-14T16:59:02.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e22a2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6dc15d890 con 0x7fb6dc159ab0 2023-10-14T16:59:02.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e22a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc0cd4d0 con 0x7fb6dc159e90 2023-10-14T16:59:02.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.869+0000 7fb6e083f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6dc06c140 0x7fb6dc15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49456/0 (socket says 172.21.15.29:49456) 2023-10-14T16:59:02.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6e083f640 1 -- 172.21.15.29:0/4163720382 learned_addr learned my addr 172.21.15.29:0/4163720382 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/4163720382 >> v2:172.21.15.29:3300/0 conn(0x7fb6dc159ab0 0x7fb6dc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc06fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6db7fe640 1 -- 172.21.15.29:0/4163720382 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772620787 0 0) 0x7fb6dc0cd1f0 con 0x7fb6dc06c140 2023-10-14T16:59:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6db7fe640 1 -- 172.21.15.29:0/4163720382 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c8003520 con 0x7fb6dc06c140 2023-10-14T16:59:02.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6dbfff640 1 -- 172.21.15.29:0/4163720382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 msgr2=0x7fb6dc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:02.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc06fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6dbfff640 1 -- 172.21.15.29:0/4163720382 >> v1:172.21.15.29:6789/0 conn(0x7fb6dc06c140 legacy=0x7fb6dc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6dbfff640 1 -- 172.21.15.29:0/4163720382 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6dc0cd7b0 con 0x7fb6dc159ab0 2023-10-14T16:59:02.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.870+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:59:02.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.872+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc0ce370 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0097d0 tx=0x7fb6cc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa9a7fb6e0064c16 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.872+0000 7fb6db7fe640 1 -- 172.21.15.29:0/4163720382 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc0316f0 con 0x7fb6dc159ab0 2023-10-14T16:59:02.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.872+0000 7fb6db7fe640 1 -- 172.21.15.29:0/4163720382 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc031850 con 0x7fb6dc159ab0 2023-10-14T16:59:02.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.872+0000 7fb6db7fe640 1 -- 172.21.15.29:0/4163720382 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc031be0 con 0x7fb6dc159ab0 2023-10-14T16:59:02.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 -- 172.21.15.29:0/4163720382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc159ab0 msgr2=0x7fb6dc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc0ce370 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc0097d0 tx=0x7fb6cc02b790 comp rx=0 tx=0).stop 2023-10-14T16:59:02.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 -- 172.21.15.29:0/4163720382 shutdown_connections 2023-10-14T16:59:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc06fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/4163720382 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc0ce370 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 -- 172.21.15.29:0/4163720382 >> 172.21.15.29:0/4163720382 conn(0x7fb6dc0ce9b0 msgr2=0x7fb6dc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 -- 172.21.15.29:0/4163720382 shutdown_connections 2023-10-14T16:59:02.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.873+0000 7fb6e22a2640 1 -- 172.21.15.29:0/4163720382 wait complete. 2023-10-14T16:59:02.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.875+0000 7fb6e22a2640 1 Processor -- start 2023-10-14T16:59:02.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.875+0000 7fb6e22a2640 1 -- start start 2023-10-14T16:59:02.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc156b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc1e3850 con 0x7fb6dc06c140 2023-10-14T16:59:02.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc150430 con 0x7fb6dc159ab0 2023-10-14T16:59:02.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6e22a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6dc153f50 con 0x7fb6dc159e90 2023-10-14T16:59:02.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35538/0 (socket says 172.21.15.29:35538) 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.876+0000 7fb6dbfff640 1 -- 172.21.15.29:0/3182355113 learned_addr learned my addr 172.21.15.29:0/3182355113 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6e083f640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc156b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 -- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 msgr2=0x7fb6dc156b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc156b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 -- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 msgr2=0x7fb6dc1565e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:02.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc1565e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:02.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 -- 172.21.15.29:0/3182355113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6cc009400 con 0x7fb6dc06c140 2023-10-14T16:59:02.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:02.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6dbfff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc034680 con 0x7fb6dc06c140 2023-10-14T16:59:02.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6cc034820 con 0x7fb6dc06c140 2023-10-14T16:59:02.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.877+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6cc034bd0 con 0x7fb6dc06c140 2023-10-14T16:59:02.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.878+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6dc154230 con 0x7fb6dc06c140 2023-10-14T16:59:02.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.878+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6dc1547f0 con 0x7fb6dc06c140 2023-10-14T16:59:02.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.879+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6dc04e160 con 0x7fb6dc06c140 2023-10-14T16:59:02.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.880+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 52) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fb6cc03b040 con 0x7fb6dc06c140 2023-10-14T16:59:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.881+0000 7fb6d97fa640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 0x7fb6bc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.881+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fb6cc0ab720 con 0x7fb6dc06c140 2023-10-14T16:59:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.881+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 0x7fb6bc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.882+0000 7fb6d3fff640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 0x7fb6bc06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4006fd0 tx=0x7fb6c4008040 comp rx=0 tx=0).ready entity=mgr.7680 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:02.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:02.885+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb6cc076130 con 0x7fb6dc06c140 2023-10-14T16:59:03.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.131+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fb6dc0cee90 con 0x7fb6dc06c140 2023-10-14T16:59:03.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.400+0000 7fb6d97fa640 1 -- 172.21.15.29:0/3182355113 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v53) v1 ==== 93+0+0 (secure 0 0 0) 0x7fb6cc079750 con 0x7fb6dc06c140 2023-10-14T16:59:03.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 msgr2=0x7fb6bc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:03.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 0x7fb6bc06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fb6c4006fd0 tx=0x7fb6c4008040 comp rx=0 tx=0).stop 2023-10-14T16:59:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 msgr2=0x7fb6dc14fef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 crypto rx=0x7fb6cc000c00 tx=0x7fb6cc0046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 shutdown_connections 2023-10-14T16:59:03.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fb6bc06b100 0x7fb6bc06d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6dc159e90 0x7fb6dc156b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6dc159ab0 0x7fb6dc1565e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 --2- 172.21.15.29:0/3182355113 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6dc06c140 0x7fb6dc14fef0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:03.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.405+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 >> 172.21.15.29:0/3182355113 conn(0x7fb6dc0ce9b0 msgr2=0x7fb6dc0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:03.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.408+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 shutdown_connections 2023-10-14T16:59:03.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:03.408+0000 7fb6e22a2640 1 -- 172.21.15.29:0/3182355113 wait complete. 2023-10-14T16:59:03.433 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:59:03.434 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:59:03.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:59:03.647 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:59:03.648 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:03.649 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:03.701 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:03.701 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:03.796 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:03.914 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:03.913+0000 7f25dcc3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:59:03.961 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:03.961+0000 7f6ab5e38040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:59:03.969 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:03.967+0000 7f25dcc3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:59:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.016+0000 7ff1a2294640 1 Processor -- start 2023-10-14T16:59:04.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.017+0000 7ff1a2294640 1 -- start start 2023-10-14T16:59:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.018+0000 7ff1a2294640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19c15ae80 0x7ff19c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.018+0000 7ff1a2294640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff19c0ce390 0x7ff19c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.018+0000 7ff1a2294640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19c15ed60 con 0x7ff19c15aaa0 2023-10-14T16:59:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.018+0000 7ff1a2294640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c06b810 con 0x7ff19c15ae80 2023-10-14T16:59:04.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.018+0000 7ff1a2294640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19c0707e0 con 0x7ff19c0ce390 2023-10-14T16:59:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19c15ae80 0x7ff19c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff19c15ae80 0x7ff19c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35552/0 (socket says 172.21.15.29:35552) 2023-10-14T16:59:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 -- 172.21.15.29:0/2044212817 learned_addr learned my addr 172.21.15.29:0/2044212817 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:04.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff1a0831640 1 --2- 172.21.15.29:0/2044212817 >> v2:172.21.15.186:3300/0 conn(0x7ff19c0ce390 0x7ff19c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 -- 172.21.15.29:0/2044212817 >> v2:172.21.15.186:3300/0 conn(0x7ff19c0ce390 msgr2=0x7ff19c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 --2- 172.21.15.29:0/2044212817 >> v2:172.21.15.186:3300/0 conn(0x7ff19c0ce390 0x7ff19c070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 -- 172.21.15.29:0/2044212817 >> v1:172.21.15.186:6789/0 conn(0x7ff19c15aaa0 legacy=0x7ff19c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:04.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.019+0000 7ff193fff640 1 -- 172.21.15.29:0/2044212817 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19c15ef90 con 0x7ff19c15ae80 2023-10-14T16:59:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.021+0000 7ff193fff640 1 --2- 172.21.15.29:0/2044212817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15e820 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0c2490 tx=0x7ff18402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f79486d89205045 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.021+0000 7ff19b7fe640 1 -- 172.21.15.29:0/2044212817 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184031870 con 0x7ff19c15ae80 2023-10-14T16:59:04.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.021+0000 7ff19b7fe640 1 -- 172.21.15.29:0/2044212817 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1840319d0 con 0x7ff19c15ae80 2023-10-14T16:59:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.021+0000 7ff19b7fe640 1 -- 172.21.15.29:0/2044212817 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184031d60 con 0x7ff19c15ae80 2023-10-14T16:59:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 -- 172.21.15.29:0/2044212817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 msgr2=0x7ff19c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 --2- 172.21.15.29:0/2044212817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15e820 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 crypto rx=0x7ff19c0c2490 tx=0x7ff18402b910 comp rx=0 tx=0).stop 2023-10-14T16:59:04.025 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.022+0000 7f6ab5e38040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:59:04.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 -- 172.21.15.29:0/2044212817 shutdown_connections 2023-10-14T16:59:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 --2- 172.21.15.29:0/2044212817 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15e820 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 --2- 172.21.15.29:0/2044212817 >> v2:172.21.15.186:3300/0 conn(0x7ff19c0ce390 0x7ff19c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 -- 172.21.15.29:0/2044212817 >> 172.21.15.29:0/2044212817 conn(0x7ff19c0cb410 msgr2=0x7ff19c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:04.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 -- 172.21.15.29:0/2044212817 shutdown_connections 2023-10-14T16:59:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.022+0000 7ff1a2294640 1 -- 172.21.15.29:0/2044212817 wait complete. 2023-10-14T16:59:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.025+0000 7ff1a2294640 1 Processor -- start 2023-10-14T16:59:04.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.025+0000 7ff1a2294640 1 -- start start 2023-10-14T16:59:04.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff19c0ce390 0x7ff19c155e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 0x7ff19c1f4130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c15e390 con 0x7ff19c15ae80 2023-10-14T16:59:04.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c15db30 con 0x7ff19c15aaa0 2023-10-14T16:59:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a2294640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19c1f4670 con 0x7ff19c0ce390 2023-10-14T16:59:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff1a0831640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.027+0000 7ff19bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff19c0ce390 0x7ff19c155e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35558/0 (socket says 172.21.15.29:35558) 2023-10-14T16:59:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 -- 172.21.15.29:0/3534816041 learned_addr learned my addr 172.21.15.29:0/3534816041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff193fff640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 0x7ff19c1f4130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff19c0ce390 msgr2=0x7ff19c155e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff19c0ce390 0x7ff19c155e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 msgr2=0x7ff19c1f4130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 0x7ff19c1f4130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 -- 172.21.15.29:0/3534816041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840094d0 con 0x7ff19c15ae80 2023-10-14T16:59:04.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff193fff640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 0x7ff19c1f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:59:04.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.028+0000 7ff1a0831640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff194007e40 tx=0x7ff19400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.029+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff194014620 con 0x7ff19c15ae80 2023-10-14T16:59:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.029+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19400ee90 con 0x7ff19c15ae80 2023-10-14T16:59:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.029+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1940149e0 con 0x7ff19c15ae80 2023-10-14T16:59:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.029+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff19c0c6920 con 0x7ff19c15ae80 2023-10-14T16:59:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.029+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff19c15e0b0 con 0x7ff19c15ae80 2023-10-14T16:59:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.030+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19c04e160 con 0x7ff19c15ae80 2023-10-14T16:59:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.031+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7ff194014b40 con 0x7ff19c15ae80 2023-10-14T16:59:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.033+0000 7ff1997fa640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 0x7ff17c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.033+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7ff19408a3e0 con 0x7ff19c15ae80 2023-10-14T16:59:04.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.033+0000 7ff19bfff640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 msgr2=0x7ff17c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:04.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.033+0000 7ff19bfff640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 0x7ff17c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:04.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.037+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1940550f0 con 0x7ff19c15ae80 2023-10-14T16:59:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.232+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff19c0cdc60 con 0x7ff19c15ae80 2023-10-14T16:59:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.234+0000 7ff19bfff640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 msgr2=0x7ff17c06d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.234+0000 7ff19bfff640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 0x7ff17c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.234+0000 7ff1997fa640 1 -- 172.21.15.29:0/3534816041 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17178 (secure 0 0 0) 0x7ff194058710 con 0x7ff19c15ae80 2023-10-14T16:59:04.235 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:04.236 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":112,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:01.358053+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3340406750":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3651737802":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/583660233":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/2619862699":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3275044227":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6833/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6832/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/680310103":"2023-10-15T16:57:59.666943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.236+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 msgr2=0x7ff17c06d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.236+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 0x7ff17c06d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.236+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 msgr2=0x7ff19c15bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.236+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 crypto rx=0x7ff194007e40 tx=0x7ff19400e4b0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 shutdown_connections 2023-10-14T16:59:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7ff17c06b0b0 0x7ff17c06d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19c15ae80 0x7ff19c15bc70 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff19c15aaa0 0x7ff19c1f4130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 --2- 172.21.15.29:0/3534816041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff19c0ce390 0x7ff19c155e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 >> 172.21.15.29:0/3534816041 conn(0x7ff19c0cb410 msgr2=0x7ff19c15a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 shutdown_connections 2023-10-14T16:59:04.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.237+0000 7ff1a2294640 1 -- 172.21.15.29:0/3534816041 wait complete. 2023-10-14T16:59:04.257 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:57:42.669157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:57:43.755222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:59:04.258 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:59:04.422 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.421+0000 7f25dcc3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:59:04.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.467+0000 7f806564b640 1 Processor -- start 2023-10-14T16:59:04.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.467+0000 7f806564b640 1 -- start start 2023-10-14T16:59:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.468+0000 7f806564b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f806015ad70 0x7f8060069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.468+0000 7f806564b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80600c9ca0 0x7f806006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.468+0000 7f806564b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80600cb030 con 0x7f80600ca3b0 2023-10-14T16:59:04.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.468+0000 7f806564b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600c91c0 con 0x7f80600c9ca0 2023-10-14T16:59:04.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.468+0000 7f806564b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80600c94a0 con 0x7f806015ad70 2023-10-14T16:59:04.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80600ca3b0 0x7f80600ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49478/0 (socket says 172.21.15.29:49478) 2023-10-14T16:59:04.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805f7fe640 1 -- 172.21.15.29:0/2271878193 learned_addr learned my addr 172.21.15.29:0/2271878193 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:04.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805e7fc640 1 --2- 172.21.15.29:0/2271878193 >> v2:172.21.15.29:3300/0 conn(0x7f80600c9ca0 0x7f806006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805effd640 1 --2- 172.21.15.29:0/2271878193 >> v2:172.21.15.186:3300/0 conn(0x7f806015ad70 0x7f8060069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805e7fc640 1 -- 172.21.15.29:0/2271878193 >> v2:172.21.15.186:3300/0 conn(0x7f806015ad70 msgr2=0x7f8060069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:04.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805e7fc640 1 --2- 172.21.15.29:0/2271878193 >> v2:172.21.15.186:3300/0 conn(0x7f806015ad70 0x7f8060069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805e7fc640 1 -- 172.21.15.29:0/2271878193 >> v1:172.21.15.29:6789/0 conn(0x7f80600ca3b0 legacy=0x7f80600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.469+0000 7f805e7fc640 1 -- 172.21.15.29:0/2271878193 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80600c9780 con 0x7f80600c9ca0 2023-10-14T16:59:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.470+0000 7f805e7fc640 1 --2- 172.21.15.29:0/2271878193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600c9ca0 0x7f806006a2e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f80600cf780 tx=0x7f805402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=536cdad95175eb8b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f805dffb640 1 -- 172.21.15.29:0/2271878193 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054031830 con 0x7f80600c9ca0 2023-10-14T16:59:04.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f805dffb640 1 -- 172.21.15.29:0/2271878193 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8054031990 con 0x7f80600c9ca0 2023-10-14T16:59:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f805dffb640 1 -- 172.21.15.29:0/2271878193 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054031d20 con 0x7f80600c9ca0 2023-10-14T16:59:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 -- 172.21.15.29:0/2271878193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600c9ca0 msgr2=0x7f806006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 --2- 172.21.15.29:0/2271878193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600c9ca0 0x7f806006a2e0 secure :-1 s=READY pgs=1288 cs=0 l=1 rev1=1 crypto rx=0x7f80600cf780 tx=0x7f805402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 -- 172.21.15.29:0/2271878193 shutdown_connections 2023-10-14T16:59:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 --2- 172.21.15.29:0/2271878193 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600c9ca0 0x7f806006a2e0 unknown :-1 s=CLOSED pgs=1288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 --2- 172.21.15.29:0/2271878193 >> v2:172.21.15.186:3300/0 conn(0x7f806015ad70 0x7f8060069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.471+0000 7f806564b640 1 -- 172.21.15.29:0/2271878193 >> 172.21.15.29:0/2271878193 conn(0x7f80600ce9b0 msgr2=0x7f80600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.472+0000 7f806564b640 1 -- 172.21.15.29:0/2271878193 shutdown_connections 2023-10-14T16:59:04.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.472+0000 7f806564b640 1 -- 172.21.15.29:0/2271878193 wait complete. 2023-10-14T16:59:04.476 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.474+0000 7f25dcc3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:59:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.473+0000 7f806564b640 1 Processor -- start 2023-10-14T16:59:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.473+0000 7f806564b640 1 -- start start 2023-10-14T16:59:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80600c9ca0 0x7f806015b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015ad70 0x7f8060157c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80601f3a40 con 0x7f80600ca3b0 2023-10-14T16:59:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806015b990 con 0x7f80600c9ca0 2023-10-14T16:59:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.474+0000 7f806564b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80601f3ce0 con 0x7f806015ad70 2023-10-14T16:59:04.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35584/0 (socket says 172.21.15.29:35584) 2023-10-14T16:59:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 -- 172.21.15.29:0/1975577004 learned_addr learned my addr 172.21.15.29:0/1975577004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805effd640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80600c9ca0 0x7f806015b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805f7fe640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015ad70 0x7f8060157c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015ad70 msgr2=0x7f8060157c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015ad70 0x7f8060157c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80600c9ca0 msgr2=0x7f806015b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80600c9ca0 0x7f806015b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.475+0000 7f805e7fc640 1 -- 172.21.15.29:0/1975577004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8054009400 con 0x7f80600ca3b0 2023-10-14T16:59:04.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f805e7fc640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8054009890 tx=0x7f80540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805403ba70 con 0x7f80600ca3b0 2023-10-14T16:59:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805403bbd0 con 0x7f80600ca3b0 2023-10-14T16:59:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80540453f0 con 0x7f80600ca3b0 2023-10-14T16:59:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80601f3fc0 con 0x7f80600ca3b0 2023-10-14T16:59:04.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.476+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806015bc40 con 0x7f80600ca3b0 2023-10-14T16:59:04.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.477+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80601f44e0 con 0x7f80600ca3b0 2023-10-14T16:59:04.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.478+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f8054045550 con 0x7f80600ca3b0 2023-10-14T16:59:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.479+0000 7f80477fe640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 0x7f804c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.480+0000 7f805effd640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 msgr2=0x7f804c06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.480+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f80540ac110 con 0x7f80600ca3b0 2023-10-14T16:59:04.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.480+0000 7f805effd640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 0x7f804c06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:04.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.484+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f80601f44e0 con 0x7f80600ca3b0 2023-10-14T16:59:04.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.505+0000 7f6ab5e38040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:59:04.525 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.524+0000 7f25dcc3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:59:04.561 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.560+0000 7f6ab5e38040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:59:04.586 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.585+0000 7f25dcc3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:59:04.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.613+0000 7f6ab5e38040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:59:04.635 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.634+0000 7f25dcc3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:59:04.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.677+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f80601f44e0 con 0x7f80600ca3b0 2023-10-14T16:59:04.678 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.678+0000 7f6ab5e38040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:59:04.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.678+0000 7f80477fe640 1 -- 172.21.15.29:0/1975577004 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v112) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80601f44e0 con 0x7f80600ca3b0 2023-10-14T16:59:04.679 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:59:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f805effd640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 msgr2=0x7f804c06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f805effd640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 0x7f804c06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 msgr2=0x7f804c06d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:04.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 0x7f804c06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 msgr2=0x7f80601e4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 crypto rx=0x7f8054009890 tx=0x7f80540046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 shutdown_connections 2023-10-14T16:59:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f804c06b0e0 0x7f804c06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f806015ad70 0x7f8060157c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80600ca3b0 0x7f80601e4830 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 --2- 172.21.15.29:0/1975577004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80600c9ca0 0x7f806015b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 >> 172.21.15.29:0/1975577004 conn(0x7f80600ce9b0 msgr2=0x7f80600b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 shutdown_connections 2023-10-14T16:59:04.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.681+0000 7f806564b640 1 -- 172.21.15.29:0/1975577004 wait complete. 2023-10-14T16:59:04.699 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:59:04.732 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.731+0000 7f6ab5e38040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:59:04.744 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.743+0000 7f25dcc3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:59:04.806 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.805+0000 7f25dcc3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:04.850 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.849+0000 7f6ab5e38040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:59:04.855 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.854+0000 7f25dcc3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:59:04.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.911+0000 7f56843a9640 1 Processor -- start 2023-10-14T16:59:04.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.911+0000 7f56843a9640 1 -- start start 2023-10-14T16:59:04.912 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.912+0000 7f25dcc3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:59:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f56843a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f567c15d030 0x7f567c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f56843a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 0x7f567c161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f56843a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f567c0b5d90 con 0x7f567c161990 2023-10-14T16:59:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f56843a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f567c0d2b50 con 0x7f567c15d030 2023-10-14T16:59:04.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f56843a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c160fc0 con 0x7f567c15d410 2023-10-14T16:59:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f5682145640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f567c15d030 0x7f567c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f5682145640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f567c15d030 0x7f567c0d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35594/0 (socket says 172.21.15.29:35594) 2023-10-14T16:59:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f5682145640 1 -- 172.21.15.29:0/1112580471 learned_addr learned my addr 172.21.15.29:0/1112580471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:04.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.913+0000 7f5681944640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 0x7f567c161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5682145640 1 -- 172.21.15.29:0/1112580471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 msgr2=0x7f567c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5682145640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 0x7f567c161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5682145640 1 -- 172.21.15.29:0/1112580471 >> v1:172.21.15.29:6789/0 conn(0x7f567c161990 legacy=0x7f567c0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5682145640 1 -- 172.21.15.29:0/1112580471 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f567c0d25d0 con 0x7f567c15d030 2023-10-14T16:59:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5681143640 1 -- 172.21.15.29:0/1112580471 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1488945217 0 0) 0x7f567c0b5d90 con 0x7f567c161990 2023-10-14T16:59:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.914+0000 7f5681944640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 0x7f567c161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:59:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.915+0000 7f5682145640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c0d1630 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5678009870 tx=0x7f567802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b52947bb7da675b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.915+0000 7f5681143640 1 -- 172.21.15.29:0/1112580471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567802f830 con 0x7f567c15d030 2023-10-14T16:59:04.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.915+0000 7f5681143640 1 -- 172.21.15.29:0/1112580471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f567802f990 con 0x7f567c15d030 2023-10-14T16:59:04.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.915+0000 7f5681143640 1 -- 172.21.15.29:0/1112580471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f567802fd20 con 0x7f567c15d030 2023-10-14T16:59:04.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 -- 172.21.15.29:0/1112580471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 msgr2=0x7f567c0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c0d1630 secure :-1 s=READY pgs=1291 cs=0 l=1 rev1=1 crypto rx=0x7f5678009870 tx=0x7f567802ba10 comp rx=0 tx=0).stop 2023-10-14T16:59:04.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 -- 172.21.15.29:0/1112580471 shutdown_connections 2023-10-14T16:59:04.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c15d410 0x7f567c161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 --2- 172.21.15.29:0/1112580471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c0d1630 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 -- 172.21.15.29:0/1112580471 >> 172.21.15.29:0/1112580471 conn(0x7f567c0cf720 msgr2=0x7f567c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:04.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 -- 172.21.15.29:0/1112580471 shutdown_connections 2023-10-14T16:59:04.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.916+0000 7f56843a9640 1 -- 172.21.15.29:0/1112580471 wait complete. 2023-10-14T16:59:04.921 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.917+0000 7f6ab5e38040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:04.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.918+0000 7f56843a9640 1 Processor -- start 2023-10-14T16:59:04.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.918+0000 7f56843a9640 1 -- start start 2023-10-14T16:59:04.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 0x7f567c0b4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 0x7f567c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c1e6b90 con 0x7f567c15d030 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f567c0ca5c0 con 0x7f567c15d410 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f56843a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f567c152840 con 0x7f567c161990 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35600/0 (socket says 172.21.15.29:35600) 2023-10-14T16:59:04.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 -- 172.21.15.29:0/156681356 learned_addr learned my addr 172.21.15.29:0/156681356 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:04.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682946640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 0x7f567c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5681944640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 0x7f567c0b4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:04.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 msgr2=0x7f567c0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 0x7f567c0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 msgr2=0x7f567c0b4190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 0x7f567c0b4190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682145640 1 -- 172.21.15.29:0/156681356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56780094d0 con 0x7f567c15d030 2023-10-14T16:59:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5681944640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 0x7f567c0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:04.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.919+0000 7f5682946640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 0x7f567c0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f5682145640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5678000c00 tx=0x7f5678004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678037c40 con 0x7f567c15d030 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5678037da0 con 0x7f567c15d030 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5678041640 con 0x7f567c15d030 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f567c0b4d00 con 0x7f567c15d030 2023-10-14T16:59:04.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.920+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f567c1e70c0 con 0x7f567c15d030 2023-10-14T16:59:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.921+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f567c0d3df0 con 0x7f567c15d030 2023-10-14T16:59:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.922+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f56780417a0 con 0x7f567c15d030 2023-10-14T16:59:04.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.924+0000 7f5672ffd640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 0x7f566006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.924+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f567803c070 con 0x7f567c15d030 2023-10-14T16:59:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.924+0000 7f5681944640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 msgr2=0x7f566006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.924+0000 7f5681944640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 0x7f566006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:04.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:04.928+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5678077500 con 0x7f567c15d030 2023-10-14T16:59:04.964 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:04.963+0000 7f25dcc3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:59:04.971 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:04.971+0000 7f6ab5e38040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:59:05.031 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.031+0000 7f6ab5e38040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:59:05.084 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.083+0000 7f6ab5e38040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:59:05.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.123+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f567c0c6420 con 0x7f567c15d030 2023-10-14T16:59:05.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.124+0000 7f5672ffd640 1 -- 172.21.15.29:0/156681356 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v112) v1 ==== 104+0+11 (secure 0 0 0) 0x7f567807ab20 con 0x7f567c15d030 2023-10-14T16:59:05.125 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:59:05.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.125+0000 7f5681944640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 msgr2=0x7f566006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:05.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.125+0000 7f5681944640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 0x7f566006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:05.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 msgr2=0x7f566006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:05.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 0x7f566006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 msgr2=0x7f567c1517f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 secure :-1 s=READY pgs=1292 cs=0 l=1 rev1=1 crypto rx=0x7f5678000c00 tx=0x7f5678004730 comp rx=0 tx=0).stop 2023-10-14T16:59:05.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 shutdown_connections 2023-10-14T16:59:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f566006b0b0 0x7f566006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f567c161990 0x7f567c0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f567c15d410 0x7f567c0b4190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 --2- 172.21.15.29:0/156681356 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f567c15d030 0x7f567c1517f0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 >> 172.21.15.29:0/156681356 conn(0x7f567c0cf720 msgr2=0x7f567c0b5d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:05.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 shutdown_connections 2023-10-14T16:59:05.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.127+0000 7f56843a9640 1 -- 172.21.15.29:0/156681356 wait complete. 2023-10-14T16:59:05.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:59:05.351 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.350+0000 7f25dcc3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:59:05.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.395+0000 7faa0e14e640 1 Processor -- start 2023-10-14T16:59:05.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.395+0000 7faa0e14e640 1 -- start start 2023-10-14T16:59:05.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa08160ad0 0x7faa080ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa080ae260 0x7faa08160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa080ccd80 0x7faa080cd210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa08161900 con 0x7faa080ccd80 2023-10-14T16:59:05.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa080acdd0 con 0x7faa080ae260 2023-10-14T16:59:05.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa0e14e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa081613f0 con 0x7faa08160ad0 2023-10-14T16:59:05.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.397+0000 7faa07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa080ccd80 0x7faa080cd210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa080ccd80 0x7faa080cd210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35602/0 (socket says 172.21.15.29:35602) 2023-10-14T16:59:05.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 -- 172.21.15.29:0/376489757 learned_addr learned my addr 172.21.15.29:0/376489757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:05.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa077fe640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa08160ad0 0x7faa080ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa06ffd640 1 --2- 172.21.15.29:0/376489757 >> v2:172.21.15.186:3300/0 conn(0x7faa080ae260 0x7faa08160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 -- 172.21.15.29:0/376489757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa08160ad0 msgr2=0x7faa080ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa08160ad0 0x7faa080ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 -- 172.21.15.29:0/376489757 >> v2:172.21.15.186:3300/0 conn(0x7faa080ae260 msgr2=0x7faa08160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 --2- 172.21.15.29:0/376489757 >> v2:172.21.15.186:3300/0 conn(0x7faa080ae260 0x7faa08160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.398+0000 7faa07fff640 1 -- 172.21.15.29:0/376489757 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa080ce8a0 con 0x7faa080ccd80 2023-10-14T16:59:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.399+0000 7faa07fff640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa080cd210 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7faa0806f6a0 tx=0x7faa0002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab613c7ecc12f50c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.400+0000 7faa067fc640 1 -- 172.21.15.29:0/376489757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa0002ca70 con 0x7faa080ccd80 2023-10-14T16:59:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.400+0000 7faa067fc640 1 -- 172.21.15.29:0/376489757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa0002cbd0 con 0x7faa080ccd80 2023-10-14T16:59:05.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.400+0000 7faa067fc640 1 -- 172.21.15.29:0/376489757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa000343f0 con 0x7faa080ccd80 2023-10-14T16:59:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 -- 172.21.15.29:0/376489757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 msgr2=0x7faa080cd210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa080cd210 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 crypto rx=0x7faa0806f6a0 tx=0x7faa0002bed0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 -- 172.21.15.29:0/376489757 shutdown_connections 2023-10-14T16:59:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa080cd210 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 --2- 172.21.15.29:0/376489757 >> v2:172.21.15.186:3300/0 conn(0x7faa080ae260 0x7faa08160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 --2- 172.21.15.29:0/376489757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa08160ad0 0x7faa080ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 -- 172.21.15.29:0/376489757 >> 172.21.15.29:0/376489757 conn(0x7faa0806bc10 msgr2=0x7faa0806c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 -- 172.21.15.29:0/376489757 shutdown_connections 2023-10-14T16:59:05.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.401+0000 7faa0e14e640 1 -- 172.21.15.29:0/376489757 wait complete. 2023-10-14T16:59:05.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.403+0000 7faa0e14e640 1 Processor -- start 2023-10-14T16:59:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.404+0000 7faa0e14e640 1 -- start start 2023-10-14T16:59:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa080ae260 0x7faa080b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa08160ad0 0x7faa08159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa080c45d0 con 0x7faa080ccd80 2023-10-14T16:59:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa0815ed00 con 0x7faa08160ad0 2023-10-14T16:59:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa0e14e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa08159fe0 con 0x7faa080ae260 2023-10-14T16:59:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa06ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35610/0 (socket says 172.21.15.29:35610) 2023-10-14T16:59:05.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.405+0000 7faa06ffd640 1 -- 172.21.15.29:0/827875230 learned_addr learned my addr 172.21.15.29:0/827875230 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa07fff640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa08160ad0 0x7faa08159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa077fe640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa080ae260 0x7faa080b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa080ae260 msgr2=0x7faa080b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa080ae260 0x7faa080b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa08160ad0 msgr2=0x7faa08159aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa08160ad0 0x7faa08159aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 -- 172.21.15.29:0/827875230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa00009b30 con 0x7faa080ccd80 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa06ffd640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00d800 tx=0x7fa9fc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc0096a0 con 0x7faa080ccd80 2023-10-14T16:59:05.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9fc009800 con 0x7faa080ccd80 2023-10-14T16:59:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9fc009990 con 0x7faa080ccd80 2023-10-14T16:59:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.406+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa080b4040 con 0x7faa080ccd80 2023-10-14T16:59:05.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.407+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa080b4600 con 0x7faa080ccd80 2023-10-14T16:59:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.408+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa080c4430 con 0x7faa080ccd80 2023-10-14T16:59:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.408+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa9fc009af0 con 0x7faa080ccd80 2023-10-14T16:59:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.410+0000 7fa9effff640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 0x7fa9f406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.410+0000 7faa077fe640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 msgr2=0x7fa9f406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.410+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fa9fc089fc0 con 0x7faa080ccd80 2023-10-14T16:59:05.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.410+0000 7faa077fe640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 0x7fa9f406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:05.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.415+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa080c4430 con 0x7faa080ccd80 2023-10-14T16:59:05.497 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.496+0000 7f6ab5e38040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:59:05.514 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.514+0000 7f25dcc3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:59:05.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.609+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7faa080c4430 con 0x7faa080ccd80 2023-10-14T16:59:05.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.610+0000 7fa9effff640 1 -- 172.21.15.29:0/827875230 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v112) v1 ==== 100+0+11 (secure 0 0 0) 0x7faa080c4430 con 0x7faa080ccd80 2023-10-14T16:59:05.611 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:59:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.611+0000 7faa077fe640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 msgr2=0x7fa9f406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:05.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.611+0000 7faa077fe640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 0x7fa9f406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 msgr2=0x7fa9f406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:05.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 0x7fa9f406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 msgr2=0x7faa0815e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc00d800 tx=0x7fa9fc00dcc0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 shutdown_connections 2023-10-14T16:59:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fa9f406b0e0 0x7fa9f406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa08160ad0 0x7faa08159aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa080ccd80 0x7faa0815e7c0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 --2- 172.21.15.29:0/827875230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa080ae260 0x7faa080b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 >> 172.21.15.29:0/827875230 conn(0x7faa0806bc10 msgr2=0x7faa080cd160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 shutdown_connections 2023-10-14T16:59:05.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.613+0000 7faa0e14e640 1 -- 172.21.15.29:0/827875230 wait complete. 2023-10-14T16:59:05.623 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.622+0000 7f25dcc3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:59:05.632 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T16:59:05.668 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.667+0000 7f6ab5e38040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:59:05.672 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.671+0000 7f25dcc3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:59:05.780 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.779+0000 7f25dcc3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:05.782 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.782+0000 7f6ab5e38040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:59:05.835 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.835+0000 7f6ab5e38040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:59:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.837+0000 7f6112541640 1 Processor -- start 2023-10-14T16:59:05.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.837+0000 7f6112541640 1 -- start start 2023-10-14T16:59:05.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6112541640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f610c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6112541640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f610c0caac0 0x7f610c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6112541640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610c0cb030 con 0x7f610c0c9ca0 2023-10-14T16:59:05.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6112541640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610c0c91c0 con 0x7f610c0caac0 2023-10-14T16:59:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6112541640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c94a0 con 0x7f610c15ad70 2023-10-14T16:59:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6103fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f610c0c9ca0 0x7f610c0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49494/0 (socket says 172.21.15.29:49494) 2023-10-14T16:59:05.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.839+0000 7f6103fff640 1 -- 172.21.15.29:0/1334669597 learned_addr learned my addr 172.21.15.29:0/1334669597 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:05.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173961451 0 0) 0x7f610c0cb030 con 0x7f610c0c9ca0 2023-10-14T16:59:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610bfff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f610c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f8003520 con 0x7f610c0c9ca0 2023-10-14T16:59:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f6110ade640 1 --2- 172.21.15.29:0/1334669597 >> v2:172.21.15.186:3300/0 conn(0x7f610c0caac0 0x7f610c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 280872226 0 0) 0x7f60f8003520 con 0x7f610c0c9ca0 2023-10-14T16:59:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 msgr2=0x7f610c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f610c069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 >> v2:172.21.15.186:3300/0 conn(0x7f610c0caac0 msgr2=0x7f610c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 --2- 172.21.15.29:0/1334669597 >> v2:172.21.15.186:3300/0 conn(0x7f610c0caac0 0x7f610c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0c9780 con 0x7f610c0c9ca0 2023-10-14T16:59:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f60f4001670 con 0x7f610c0c9ca0 2023-10-14T16:59:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 >> v1:172.21.15.29:6789/0 conn(0x7f610c0c9ca0 legacy=0x7f610c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f8003970 0x7f60f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60f80047c0 0x7f60f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f60f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c91c0 con 0x7f60f80047c0 2023-10-14T16:59:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c94a0 con 0x7f60f8003970 2023-10-14T16:59:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60f800ba80 con 0x7f610c15ad70 2023-10-14T16:59:05.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.840+0000 7f6103fff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60f80047c0 0x7f60f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f610bfff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f8003970 0x7f60f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6110ade640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f60f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6103fff640 1 -- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 msgr2=0x7f60f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6103fff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c15ad70 0x7f60f8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6103fff640 1 -- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f8003970 msgr2=0x7f60f8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6103fff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f8003970 0x7f60f8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f6103fff640 1 -- 172.21.15.29:0/1334669597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f8003520 con 0x7f60f80047c0 2023-10-14T16:59:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.841+0000 7f610bfff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f8003970 0x7f60f8003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:59:05.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.842+0000 7f6103fff640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60f80047c0 0x7f60f8004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c0e0 tx=0x7f60f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7881d9a2c7e915f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:05.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f4005ad0 con 0x7f60f80047c0 2023-10-14T16:59:05.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f4005c50 con 0x7f60f80047c0 2023-10-14T16:59:05.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f610b7fe640 1 -- 172.21.15.29:0/1334669597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60f402c7d0 con 0x7f60f80047c0 2023-10-14T16:59:05.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f6112541640 1 -- 172.21.15.29:0/1334669597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60f80047c0 msgr2=0x7f60f8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f6112541640 1 --2- 172.21.15.29:0/1334669597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60f80047c0 0x7f60f8004c40 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 crypto rx=0x7f60f400c0e0 tx=0x7f60f80057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.843+0000 7f6110ade640 1 -- 172.21.15.29:0/1334669597 reap_dead start 2023-10-14T16:59:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.844+0000 7f6112541640 1 -- 172.21.15.29:0/1334669597 shutdown_connections 2023-10-14T16:59:05.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.844+0000 7f6112541640 1 -- 172.21.15.29:0/1334669597 >> 172.21.15.29:0/1334669597 conn(0x7f610c0ce9b0 msgr2=0x7f610c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.844+0000 7f6112541640 1 -- 172.21.15.29:0/1334669597 shutdown_connections 2023-10-14T16:59:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.844+0000 7f6112541640 1 -- 172.21.15.29:0/1334669597 wait complete. 2023-10-14T16:59:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.846+0000 7f6112541640 1 Processor -- start 2023-10-14T16:59:05.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.847+0000 7f6112541640 1 -- start start 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f80047c0 0x7f610c0c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c0c8950 0x7f610c1f2a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c7a50 con 0x7f610c1f2f90 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c6e40 con 0x7f60f80047c0 2023-10-14T16:59:05.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.848+0000 7f6112541640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f610c0c7120 con 0x7f610c0c8950 2023-10-14T16:59:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35630/0 (socket says 172.21.15.29:35630) 2023-10-14T16:59:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 -- 172.21.15.29:0/663673235 learned_addr learned my addr 172.21.15.29:0/663673235 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f610bfff640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f80047c0 0x7f610c0c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6103fff640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c0c8950 0x7f610c1f2a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c0c8950 msgr2=0x7f610c1f2a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c0c8950 0x7f610c1f2a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f80047c0 msgr2=0x7f610c0c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f80047c0 0x7f610c0c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:05.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 -- 172.21.15.29:0/663673235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610c0c7400 con 0x7f610c1f2f90 2023-10-14T16:59:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.849+0000 7f6110ade640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f610c0cf780 tx=0x7f6104007700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.850+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104003ea0 con 0x7f610c1f2f90 2023-10-14T16:59:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.850+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f610400f050 con 0x7f610c1f2f90 2023-10-14T16:59:05.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.850+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610c0c76e0 con 0x7f610c1f2f90 2023-10-14T16:59:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.850+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6104014650 con 0x7f610c1f2f90 2023-10-14T16:59:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.850+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610c14eaa0 con 0x7f610c1f2f90 2023-10-14T16:59:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.851+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610c0c7bc0 con 0x7f610c1f2f90 2023-10-14T16:59:05.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.852+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f61040147b0 con 0x7f610c1f2f90 2023-10-14T16:59:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.853+0000 7f61097fa640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 0x7f60ec06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.853+0000 7f610bfff640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 msgr2=0x7f60ec06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:05.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.853+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f610408a170 con 0x7f610c1f2f90 2023-10-14T16:59:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.853+0000 7f610bfff640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 0x7f60ec06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:05.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:05.857+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6104054bb0 con 0x7f610c1f2f90 2023-10-14T16:59:05.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:05.887+0000 7f25dcc3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:59:05.954 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:05.954+0000 7f6ab5e38040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:06.045 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:06.043+0000 7f25dcc3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:06.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.054+0000 7f610bfff640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 msgr2=0x7f60ec06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:06.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.054+0000 7f610bfff640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 0x7f60ec06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:06.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.066+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f610c04e160 con 0x7f610c1f2f90 2023-10-14T16:59:06.067 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:06.066+0000 7f6ab5e38040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:59:06.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.066+0000 7f61097fa640 1 -- 172.21.15.29:0/663673235 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v112) v1 ==== 80+0+468 (secure 0 0 0) 0x7f61040581d0 con 0x7f610c1f2f90 2023-10-14T16:59:06.068 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3340406750 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.068 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3651737802 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.068 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/583660233 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.068 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2619862699 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.069 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3275044227 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.069 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/1520757887 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.069 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/1520757887 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.069 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/680310103 2023-10-15T16:57:59.666943+0000 2023-10-14T16:59:06.070 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T16:59:06.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 msgr2=0x7f60ec06d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:06.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 0x7f60ec06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 msgr2=0x7f610c14e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:06.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 crypto rx=0x7f610c0cf780 tx=0x7f6104007700 comp rx=0 tx=0).stop 2023-10-14T16:59:06.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 shutdown_connections 2023-10-14T16:59:06.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f60ec06b0e0 0x7f60ec06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f610c1f2f90 0x7f610c14e560 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f610c0c8950 0x7f610c1f2a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 --2- 172.21.15.29:0/663673235 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60f80047c0 0x7f610c0c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 >> 172.21.15.29:0/663673235 conn(0x7f610c0ce9b0 msgr2=0x7f610c070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:06.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 shutdown_connections 2023-10-14T16:59:06.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.069+0000 7f6112541640 1 -- 172.21.15.29:0/663673235 wait complete. 2023-10-14T16:59:06.088 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T16:59:06.088 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:06.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T16:59:06.093 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:06.092+0000 7f25dcc3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:06.212 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:06.210+0000 7f25dcc3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:59:06.234 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T16:59:06.234 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:06.235 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T16:59:06.242 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:06.236+0000 7f6ab5e38040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:06.284 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:59:06.284 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:06.289 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:06.289+0000 7f6ab5e38040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:06.338 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:59:06.339 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:06.339 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:06.393 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:59:06.393 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:59:06.394 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:06.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:06.415+0000 7f6ab5e38040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:59:06.451 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:59:06.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.663+0000 7f3068f67640 1 Processor -- start 2023-10-14T16:59:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.663+0000 7f3068f67640 1 -- start start 2023-10-14T16:59:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.664+0000 7f3068f67640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30640b8480 0x7f30640cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f3068f67640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30640d0be0 0x7f30640cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f3068f67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f306415d150 con 0x7f306415cd70 2023-10-14T16:59:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f3068f67640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30640d1180 con 0x7f30640b8480 2023-10-14T16:59:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f3068f67640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30640d1460 con 0x7f30640d0be0 2023-10-14T16:59:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30640b8480 0x7f30640cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f30640b8480 0x7f30640cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35640/0 (socket says 172.21.15.29:35640) 2023-10-14T16:59:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 -- 172.21.15.29:0/3380127284 learned_addr learned my addr 172.21.15.29:0/3380127284 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 -- 172.21.15.29:0/3380127284 >> v2:172.21.15.186:3300/0 conn(0x7f30640d0be0 msgr2=0x7f30640cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 --2- 172.21.15.29:0/3380127284 >> v2:172.21.15.186:3300/0 conn(0x7f30640d0be0 0x7f30640cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 -- 172.21.15.29:0/3380127284 >> v1:172.21.15.186:6789/0 conn(0x7f306415cd70 legacy=0x7f30640d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.665+0000 7f305bfff640 1 -- 172.21.15.29:0/3380127284 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640d1740 con 0x7f30640b8480 2023-10-14T16:59:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.666+0000 7f305bfff640 1 --2- 172.21.15.29:0/3380127284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30640b8480 0x7f30640cd7c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f30640b8910 tx=0x7f304c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43d189ee5427f4f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.667+0000 7f3062ffd640 1 -- 172.21.15.29:0/3380127284 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c0317e0 con 0x7f30640b8480 2023-10-14T16:59:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.667+0000 7f3062ffd640 1 -- 172.21.15.29:0/3380127284 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304c031940 con 0x7f30640b8480 2023-10-14T16:59:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.667+0000 7f3062ffd640 1 -- 172.21.15.29:0/3380127284 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304c031cd0 con 0x7f30640b8480 2023-10-14T16:59:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.667+0000 7f3068f67640 1 -- 172.21.15.29:0/3380127284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30640b8480 msgr2=0x7f30640cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:06.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.667+0000 7f3068f67640 1 --2- 172.21.15.29:0/3380127284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30640b8480 0x7f30640cd7c0 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 crypto rx=0x7f30640b8910 tx=0x7f304c02b880 comp rx=0 tx=0).stop 2023-10-14T16:59:06.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 -- 172.21.15.29:0/3380127284 shutdown_connections 2023-10-14T16:59:06.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 --2- 172.21.15.29:0/3380127284 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30640b8480 0x7f30640cd7c0 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 --2- 172.21.15.29:0/3380127284 >> v2:172.21.15.186:3300/0 conn(0x7f30640d0be0 0x7f30640cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 -- 172.21.15.29:0/3380127284 >> 172.21.15.29:0/3380127284 conn(0x7f30640cf570 msgr2=0x7f30640cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 -- 172.21.15.29:0/3380127284 shutdown_connections 2023-10-14T16:59:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.668+0000 7f3068f67640 1 -- 172.21.15.29:0/3380127284 wait complete. 2023-10-14T16:59:06.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.669+0000 7f3068f67640 1 Processor -- start 2023-10-14T16:59:06.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 -- start start 2023-10-14T16:59:06.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30640b8480 0x7f30640c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30640d0be0 0x7f30641469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30640ccdd0 con 0x7f306415cd70 2023-10-14T16:59:06.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306415e110 con 0x7f30640d0be0 2023-10-14T16:59:06.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.670+0000 7f3068f67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306415d500 con 0x7f30640b8480 2023-10-14T16:59:06.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:06.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35652/0 (socket says 172.21.15.29:35652) 2023-10-14T16:59:06.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 -- 172.21.15.29:0/4146768086 learned_addr learned my addr 172.21.15.29:0/4146768086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:06.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f305bfff640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30640d0be0 0x7f30641469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:06.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f30637fe640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30640b8480 0x7f30640c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:06.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30640b8480 msgr2=0x7f30640c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:06.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30640b8480 0x7f30640c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30640d0be0 msgr2=0x7f30641469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30640d0be0 0x7f30641469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.671+0000 7f3063fff640 1 -- 172.21.15.29:0/4146768086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0094d0 con 0x7f306415cd70 2023-10-14T16:59:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3063fff640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f305c00ba10 tx=0x7f305c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c00cac0 con 0x7f306415cd70 2023-10-14T16:59:06.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f305c00cc40 con 0x7f306415cd70 2023-10-14T16:59:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f306415d7e0 con 0x7f306415cd70 2023-10-14T16:59:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f305c014440 con 0x7f306415cd70 2023-10-14T16:59:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.672+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f306415dda0 con 0x7f306415cd70 2023-10-14T16:59:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.673+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f306404e9b0 con 0x7f306415cd70 2023-10-14T16:59:06.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.674+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f305c01d3b0 con 0x7f306415cd70 2023-10-14T16:59:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.675+0000 7f3060ff9640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 0x7f304406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.675+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f305c089aa0 con 0x7f306415cd70 2023-10-14T16:59:06.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.675+0000 7f30637fe640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 msgr2=0x7f304406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.675+0000 7f30637fe640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 0x7f304406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:06.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.679+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f305c054660 con 0x7f306415cd70 2023-10-14T16:59:06.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.876+0000 7f30637fe640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 msgr2=0x7f304406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:06.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.876+0000 7f30637fe640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 0x7f304406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.912+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f306415e250 con 0x7f306415cd70 2023-10-14T16:59:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.913+0000 7f3060ff9640 1 -- 172.21.15.29:0/4146768086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f305c019020 con 0x7f306415cd70 2023-10-14T16:59:06.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:06.914 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:59:06.914 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:59:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 msgr2=0x7f304406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 0x7f304406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 msgr2=0x7f3064146ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 secure :-1 s=READY pgs=1299 cs=0 l=1 rev1=1 crypto rx=0x7f305c00ba10 tx=0x7f305c00bed0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 shutdown_connections 2023-10-14T16:59:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f304406b0b0 0x7f304406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306415cd70 0x7f3064146ee0 unknown :-1 s=CLOSED pgs=1299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30640d0be0 0x7f30641469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 --2- 172.21.15.29:0/4146768086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30640b8480 0x7f30640c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 >> 172.21.15.29:0/4146768086 conn(0x7f30640cf570 msgr2=0x7f30640cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 shutdown_connections 2023-10-14T16:59:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:06.915+0000 7f3068f67640 1 -- 172.21.15.29:0/4146768086 wait complete. 2023-10-14T16:59:06.933 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:07.041 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:07.039+0000 7f25dcc3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:59:07.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.125+0000 7fac479d5640 1 Processor -- start 2023-10-14T16:59:07.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.125+0000 7fac479d5640 1 -- start start 2023-10-14T16:59:07.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.126+0000 7fac479d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac4015adf0 0x7fac4015be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.126+0000 7fac479d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 0x7fac4006b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.126+0000 7fac479d5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 0x7fac4015ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.126+0000 7fac479d5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4006dc00 con 0x7fac4015adf0 2023-10-14T16:59:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac479d5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac4006b750 con 0x7fac4015e950 2023-10-14T16:59:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac479d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4015c3a0 con 0x7fac4006adb0 2023-10-14T16:59:07.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac4015adf0 0x7fac4015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fac4015adf0 0x7fac4015be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35654/0 (socket says 172.21.15.29:35654) 2023-10-14T16:59:07.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 -- 172.21.15.29:0/2047148374 learned_addr learned my addr 172.21.15.29:0/2047148374 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:07.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45f72640 1 --2- 172.21.15.29:0/2047148374 >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 0x7fac4015ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 -- 172.21.15.29:0/2047148374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 msgr2=0x7fac4006b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac44f70640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 0x7fac4006b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 0x7fac4006b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 -- 172.21.15.29:0/2047148374 >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 msgr2=0x7fac4015ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:07.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 --2- 172.21.15.29:0/2047148374 >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 0x7fac4015ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45771640 1 -- 172.21.15.29:0/2047148374 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac4006cf80 con 0x7fac4015adf0 2023-10-14T16:59:07.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac45f72640 1 --2- 172.21.15.29:0/2047148374 >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 0x7fac4015ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:07.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.127+0000 7fac44f70640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 0x7fac4006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:07.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.128+0000 7fac45771640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4015adf0 0x7fac4015be60 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fac400cf1f0 tx=0x7fac3c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cae4a91da8a92dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.129+0000 7fac37fff640 1 -- 172.21.15.29:0/2047148374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031880 con 0x7fac4015adf0 2023-10-14T16:59:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.129+0000 7fac37fff640 1 -- 172.21.15.29:0/2047148374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0319e0 con 0x7fac4015adf0 2023-10-14T16:59:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.129+0000 7fac37fff640 1 -- 172.21.15.29:0/2047148374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031d70 con 0x7fac4015adf0 2023-10-14T16:59:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.129+0000 7fac479d5640 1 -- 172.21.15.29:0/2047148374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4015adf0 msgr2=0x7fac4015be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.129+0000 7fac479d5640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4015adf0 0x7fac4015be60 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 crypto rx=0x7fac400cf1f0 tx=0x7fac3c02b920 comp rx=0 tx=0).stop 2023-10-14T16:59:07.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 -- 172.21.15.29:0/2047148374 shutdown_connections 2023-10-14T16:59:07.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 --2- 172.21.15.29:0/2047148374 >> v2:172.21.15.186:3300/0 conn(0x7fac4015e950 0x7fac4015ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4006adb0 0x7fac4006b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 --2- 172.21.15.29:0/2047148374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4015adf0 0x7fac4015be60 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 -- 172.21.15.29:0/2047148374 >> 172.21.15.29:0/2047148374 conn(0x7fac400b0d30 msgr2=0x7fac400b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:07.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 -- 172.21.15.29:0/2047148374 shutdown_connections 2023-10-14T16:59:07.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.130+0000 7fac479d5640 1 -- 172.21.15.29:0/2047148374 wait complete. 2023-10-14T16:59:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.131+0000 7fac479d5640 1 Processor -- start 2023-10-14T16:59:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 -- start start 2023-10-14T16:59:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4015adf0 0x7fac400ccdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac4015e950 0x7fac400cd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac400cb580 con 0x7fac4006adb0 2023-10-14T16:59:07.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac400ce520 con 0x7fac4015e950 2023-10-14T16:59:07.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.132+0000 7fac479d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac4009d4c0 con 0x7fac4015adf0 2023-10-14T16:59:07.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35664/0 (socket says 172.21.15.29:35664) 2023-10-14T16:59:07.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 -- 172.21.15.29:0/822977044 learned_addr learned my addr 172.21.15.29:0/822977044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:07.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4015adf0 msgr2=0x7fac400ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45f72640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac4015e950 0x7fac400cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4015adf0 0x7fac400ccdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac4015e950 msgr2=0x7fac400cd330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac4015e950 0x7fac400cd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 -- 172.21.15.29:0/822977044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac3c0094d0 con 0x7fac4006adb0 2023-10-14T16:59:07.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.133+0000 7fac45771640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fac3c02b960 tx=0x7fac3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.134+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c031850 con 0x7fac4006adb0 2023-10-14T16:59:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.134+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac3c0443f0 con 0x7fac4006adb0 2023-10-14T16:59:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.134+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac400cb0f0 con 0x7fac4006adb0 2023-10-14T16:59:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.134+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac3c0447c0 con 0x7fac4006adb0 2023-10-14T16:59:07.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.134+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac400ca7c0 con 0x7fac4006adb0 2023-10-14T16:59:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.135+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac4004e160 con 0x7fac4006adb0 2023-10-14T16:59:07.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.136+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fac3c044920 con 0x7fac4006adb0 2023-10-14T16:59:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.137+0000 7fac35ffb640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 0x7fac2406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.137+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7fac3c0ac690 con 0x7fac4006adb0 2023-10-14T16:59:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.137+0000 7fac44f70640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 msgr2=0x7fac2406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:07.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.137+0000 7fac44f70640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 0x7fac2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:07.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.141+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fac3c077120 con 0x7fac4006adb0 2023-10-14T16:59:07.261 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.260+0000 7f6ab5e38040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:59:07.334 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:07.333+0000 7f25dcc3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:59:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.335+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac40066000 con 0x7fac4006adb0 2023-10-14T16:59:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.336+0000 7fac35ffb640 1 -- 172.21.15.29:0/822977044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v112) v1 ==== 74+0+17178 (secure 0 0 0) 0x7fac3c035090 con 0x7fac4006adb0 2023-10-14T16:59:07.338 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:07.338 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":112,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:01.358053+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3340406750":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3651737802":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/583660233":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/2619862699":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3275044227":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6833/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6832/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/680310103":"2023-10-15T16:57:59.666943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.338+0000 7fac44f70640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 msgr2=0x7fac2406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.338+0000 7fac44f70640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 0x7fac2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 msgr2=0x7fac2406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 0x7fac2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 msgr2=0x7fac400c4540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 crypto rx=0x7fac3c02b960 tx=0x7fac3c0046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 shutdown_connections 2023-10-14T16:59:07.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7fac2406b0e0 0x7fac2406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fac4015e950 0x7fac400cd330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fac4015adf0 0x7fac400ccdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 --2- 172.21.15.29:0/822977044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fac4006adb0 0x7fac400c4540 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 >> 172.21.15.29:0/822977044 conn(0x7fac400b0d30 msgr2=0x7fac4015a130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 shutdown_connections 2023-10-14T16:59:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.339+0000 7fac479d5640 1 -- 172.21.15.29:0/822977044 wait complete. 2023-10-14T16:59:07.358 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 12, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:57:42.669157+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '108', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 13, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:57:43.755222+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:59:07.359 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:59:07.390 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:07.389+0000 7f25dcc3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:59:07.505 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:07.504+0000 7f25dcc3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:59:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.572+0000 7f099c7ce640 1 Processor -- start 2023-10-14T16:59:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.572+0000 7f099c7ce640 1 -- start start 2023-10-14T16:59:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.573+0000 7f099c7ce640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f099415d150 0x7f09940ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099c7ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0994160b90 0x7f0994160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099c7ce640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09940ac890 con 0x7f0994069da0 2023-10-14T16:59:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099c7ce640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f099415d530 con 0x7f099415d150 2023-10-14T16:59:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099c7ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0994161650 con 0x7f0994160b90 2023-10-14T16:59:07.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099ad6b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0994069da0 0x7f099406a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49502/0 (socket says 172.21.15.29:49502) 2023-10-14T16:59:07.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099ad6b640 1 -- 172.21.15.29:0/1869953017 learned_addr learned my addr 172.21.15.29:0/1869953017 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:07.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f099a56a640 1 --2- 172.21.15.29:0/1869953017 >> v2:172.21.15.186:3300/0 conn(0x7f099415d150 0x7f09940ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.574+0000 7f0999d69640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0994160b90 0x7f0994160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169167589 0 0) 0x7f09940ac890 con 0x7f0994069da0 2023-10-14T16:59:07.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0980003520 con 0x7f0994069da0 2023-10-14T16:59:07.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f099a56a640 1 -- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0994160b90 msgr2=0x7f0994160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f099a56a640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0994160b90 0x7f0994160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f099a56a640 1 -- 172.21.15.29:0/1869953017 >> v1:172.21.15.29:6789/0 conn(0x7f0994069da0 legacy=0x7f099406a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 765288281 0 0) 0x7f0980003520 con 0x7f0994069da0 2023-10-14T16:59:07.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.575+0000 7f099a56a640 1 -- 172.21.15.29:0/1869953017 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0994161930 con 0x7f099415d150 2023-10-14T16:59:07.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.576+0000 7f099a56a640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099415d150 0x7f09940ad350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f09900097a0 tx=0x7f099002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8be578d4dd58218 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.577+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f098c0019f0 con 0x7f099415d150 2023-10-14T16:59:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.577+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099002e050 con 0x7f099415d150 2023-10-14T16:59:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.577+0000 7f0999568640 1 -- 172.21.15.29:0/1869953017 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0990030730 con 0x7f099415d150 2023-10-14T16:59:07.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.577+0000 7f099c7ce640 1 -- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099415d150 msgr2=0x7f09940ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.577+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099415d150 0x7f09940ad350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f09900097a0 tx=0x7f099002b790 comp rx=0 tx=0).stop 2023-10-14T16:59:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 -- 172.21.15.29:0/1869953017 shutdown_connections 2023-10-14T16:59:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0994160b90 0x7f0994160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1869953017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099415d150 0x7f09940ad350 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 -- 172.21.15.29:0/1869953017 >> 172.21.15.29:0/1869953017 conn(0x7f09940ce9b0 msgr2=0x7f09940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 -- 172.21.15.29:0/1869953017 shutdown_connections 2023-10-14T16:59:07.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f099c7ce640 1 -- 172.21.15.29:0/1869953017 wait complete. 2023-10-14T16:59:07.583 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.578+0000 7f6ab5e38040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:59:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.579+0000 7f099c7ce640 1 Processor -- start 2023-10-14T16:59:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.580+0000 7f099c7ce640 1 -- start start 2023-10-14T16:59:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.580+0000 7f099c7ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099c7ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099415d150 0x7f09941f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099c7ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0994160b90 0x7f09940b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099c7ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941f7110 con 0x7f0994069da0 2023-10-14T16:59:07.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099c7ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09940c6110 con 0x7f0994160b90 2023-10-14T16:59:07.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099c7ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09940c63f0 con 0x7f099415d150 2023-10-14T16:59:07.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35674/0 (socket says 172.21.15.29:35674) 2023-10-14T16:59:07.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 -- 172.21.15.29:0/1263151819 learned_addr learned my addr 172.21.15.29:0/1263151819 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:07.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 -- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099415d150 msgr2=0x7f09941f6bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f0999d69640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099415d150 0x7f09941f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099415d150 0x7f09941f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 -- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0994160b90 msgr2=0x7f09940b32f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0994160b90 0x7f09940b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.581+0000 7f099a56a640 1 -- 172.21.15.29:0/1263151819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0990009400 con 0x7f0994069da0 2023-10-14T16:59:07.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f099a56a640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0990000c00 tx=0x7f09900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099003b810 con 0x7f0994069da0 2023-10-14T16:59:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099003b970 con 0x7f0994069da0 2023-10-14T16:59:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099003bd20 con 0x7f0994069da0 2023-10-14T16:59:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09940c66d0 con 0x7f0994069da0 2023-10-14T16:59:07.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.582+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09940c6ed0 con 0x7f0994069da0 2023-10-14T16:59:07.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.583+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09940c7060 con 0x7f0994069da0 2023-10-14T16:59:07.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.584+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 53) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f0990039040 con 0x7f0994069da0 2023-10-14T16:59:07.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.586+0000 7f098affd640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 0x7f097806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:07.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.586+0000 7f0999d69640 1 -- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 msgr2=0x7f097806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4145442506 2023-10-14T16:59:07.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.586+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(112..112 src has 1..112) v4 ==== 6126+0+0 (secure 0 0 0) 0x7f09900ab360 con 0x7f0994069da0 2023-10-14T16:59:07.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.586+0000 7f0999d69640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 0x7f097806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:07.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.589+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0990075ec0 con 0x7f0994069da0 2023-10-14T16:59:07.640 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.639+0000 7f6ab5e38040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:59:07.760 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.759+0000 7f6ab5e38040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:59:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.774+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 54) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0990076290 con 0x7f0994069da0 2023-10-14T16:59:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.775+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 msgr2=0x7f097806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.775+0000 7f098affd640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 0x7f097806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.789+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f099404e160 con 0x7f0994069da0 2023-10-14T16:59:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.790+0000 7f098affd640 1 -- 172.21.15.29:0/1263151819 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v113) v1 ==== 93+0+10 (secure 0 0 0) 0x7f099003ea50 con 0x7f0994069da0 2023-10-14T16:59:07.791 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:59:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 msgr2=0x7f09940c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 crypto rx=0x7f0990000c00 tx=0x7f09900046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 shutdown_connections 2023-10-14T16:59:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:6832/4145442506,v1:172.21.15.29:6833/4145442506] conn(0x7f097806b0b0 0x7f097806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0994160b90 0x7f09940b32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099415d150 0x7f09941f6bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 --2- 172.21.15.29:0/1263151819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0994069da0 0x7f09940c5a80 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 >> 172.21.15.29:0/1263151819 conn(0x7f09940ce9b0 msgr2=0x7f09940cace0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 shutdown_connections 2023-10-14T16:59:07.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:07.793+0000 7f099c7ce640 1 -- 172.21.15.29:0/1263151819 wait complete. 2023-10-14T16:59:07.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T16:59:07.851 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a3b137640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a3b137640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a3b137640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a3b137640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a3b137640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.852 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.850+0000 7f6a37930640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.851+0000 7f6a37930640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.851+0000 7f6a37930640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.851+0000 7f6a37930640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:07.853 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:07.851+0000 7f6a37930640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:08.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.001+0000 7ff6fa141640 1 Processor -- start 2023-10-14T16:59:08.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.001+0000 7ff6fa141640 1 -- start start 2023-10-14T16:59:08.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6fa141640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f406fd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6fa141640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff6f406ade0 0x7ff6f406b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6fa141640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6f40cdc30 con 0x7ff6f40c9800 2023-10-14T16:59:08.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6fa141640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6f406b780 con 0x7ff6f406ade0 2023-10-14T16:59:08.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6fa141640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40ccd80 con 0x7ff6f4071b30 2023-10-14T16:59:08.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6f3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff6f40c9800 0x7ff6f40ce0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49510/0 (socket says 172.21.15.29:49510) 2023-10-14T16:59:08.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6f3fff640 1 -- 172.21.15.29:0/1969099921 learned_addr learned my addr 172.21.15.29:0/1969099921 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:08.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.003+0000 7ff6eaffd640 1 --2- 172.21.15.29:0/1969099921 >> v2:172.21.15.29:3300/0 conn(0x7ff6f406ade0 0x7ff6f406b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6f37fe640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f406fd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6f2ffd640 1 -- 172.21.15.29:0/1969099921 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417182777 0 0) 0x7ff6f40cdc30 con 0x7ff6f40c9800 2023-10-14T16:59:08.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6eaffd640 1 -- 172.21.15.29:0/1969099921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 msgr2=0x7ff6f406fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6eaffd640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f406fd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6eaffd640 1 -- 172.21.15.29:0/1969099921 >> v1:172.21.15.29:6789/0 conn(0x7ff6f40c9800 legacy=0x7ff6f40ce0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.004+0000 7ff6eaffd640 1 -- 172.21.15.29:0/1969099921 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6f40cd020 con 0x7ff6f406ade0 2023-10-14T16:59:08.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.005+0000 7ff6eaffd640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f406b240 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00a940 tx=0x7ff6dc02bc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37dbf99fb653c4d9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6f2ffd640 1 -- 172.21.15.29:0/1969099921 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc02ca30 con 0x7ff6f406ade0 2023-10-14T16:59:08.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6f2ffd640 1 -- 172.21.15.29:0/1969099921 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6dc02cb90 con 0x7ff6f406ade0 2023-10-14T16:59:08.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6f2ffd640 1 -- 172.21.15.29:0/1969099921 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc034440 con 0x7ff6f406ade0 2023-10-14T16:59:08.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 -- 172.21.15.29:0/1969099921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 msgr2=0x7ff6f406b240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f406b240 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc00a940 tx=0x7ff6dc02bc80 comp rx=0 tx=0).stop 2023-10-14T16:59:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 -- 172.21.15.29:0/1969099921 shutdown_connections 2023-10-14T16:59:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f406b240 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 --2- 172.21.15.29:0/1969099921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f406fd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.006+0000 7ff6fa141640 1 -- 172.21.15.29:0/1969099921 >> 172.21.15.29:0/1969099921 conn(0x7ff6f406bc10 msgr2=0x7ff6f406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.007+0000 7ff6fa141640 1 -- 172.21.15.29:0/1969099921 shutdown_connections 2023-10-14T16:59:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.007+0000 7ff6fa141640 1 -- 172.21.15.29:0/1969099921 wait complete. 2023-10-14T16:59:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.008+0000 7ff6fa141640 1 Processor -- start 2023-10-14T16:59:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.008+0000 7ff6fa141640 1 -- start start 2023-10-14T16:59:08.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f40c9800 0x7ff6f40c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40acdd0 con 0x7ff6f406ade0 2023-10-14T16:59:08.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40ad000 con 0x7ff6f40c9800 2023-10-14T16:59:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.009+0000 7ff6fa141640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6f40ad2e0 con 0x7ff6f4071b30 2023-10-14T16:59:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35814/0 (socket says 172.21.15.29:35814) 2023-10-14T16:59:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 -- 172.21.15.29:0/3667445825 learned_addr learned my addr 172.21.15.29:0/3667445825 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:08.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 -- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 msgr2=0x7ff6f40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 -- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f40c9800 msgr2=0x7ff6f40c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f40c9800 0x7ff6f40c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.010+0000 7ff6f37fe640 1 -- 172.21.15.29:0/3667445825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6dc00a5a0 con 0x7ff6f406ade0 2023-10-14T16:59:08.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6f37fe640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400d990 tx=0x7ff6e400dd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4007b10 con 0x7ff6f406ade0 2023-10-14T16:59:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6e4007c70 con 0x7ff6f406ade0 2023-10-14T16:59:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6e4010470 con 0x7ff6f406ade0 2023-10-14T16:59:08.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6f40ad5c0 con 0x7ff6f406ade0 2023-10-14T16:59:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.011+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6f40adb80 con 0x7ff6f406ade0 2023-10-14T16:59:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.012+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6f404e160 con 0x7ff6f406ade0 2023-10-14T16:59:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.013+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff6e40105d0 con 0x7ff6f406ade0 2023-10-14T16:59:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.014+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7ff6e4089900 con 0x7ff6f406ade0 2023-10-14T16:59:08.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.019+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6e40549c0 con 0x7ff6f406ade0 2023-10-14T16:59:08.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.214+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff6f406cab0 con 0x7ff6f406ade0 2023-10-14T16:59:08.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.215+0000 7ff6f0ff9640 1 -- 172.21.15.29:0/3667445825 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v113) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff6e401f020 con 0x7ff6f406ade0 2023-10-14T16:59:08.216 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T16:59:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 msgr2=0x7ff6f41569f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 secure :-1 s=READY pgs=1317 cs=0 l=1 rev1=1 crypto rx=0x7ff6e400d990 tx=0x7ff6e400dd90 comp rx=0 tx=0).stop 2023-10-14T16:59:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 shutdown_connections 2023-10-14T16:59:08.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6f40c9800 0x7ff6f40c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6f4071b30 0x7ff6f40ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 --2- 172.21.15.29:0/3667445825 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6f406ade0 0x7ff6f41569f0 unknown :-1 s=CLOSED pgs=1317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 >> 172.21.15.29:0/3667445825 conn(0x7ff6f406bc10 msgr2=0x7ff6f4071090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 shutdown_connections 2023-10-14T16:59:08.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.218+0000 7ff6fa141640 1 -- 172.21.15.29:0/3667445825 wait complete. 2023-10-14T16:59:08.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T16:59:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.505+0000 7fb035e59640 1 Processor -- start 2023-10-14T16:59:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.506+0000 7fb035e59640 1 -- start start 2023-10-14T16:59:08.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb03006c040 0x7fb03006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb03015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb03015ebc0 0x7fb03015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03006c960 con 0x7fb03006c040 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb03015b9d0 con 0x7fb03015ebc0 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb035e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb03015bcb0 con 0x7fb0300ad970 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.507+0000 7fb034e57640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb03006c040 0x7fb03006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb03006c040 0x7fb03006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35822/0 (socket says 172.21.15.29:35822) 2023-10-14T16:59:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 -- 172.21.15.29:0/849696511 learned_addr learned my addr 172.21.15.29:0/849696511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:08.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb02ffff640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb03015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 -- 172.21.15.29:0/849696511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 msgr2=0x7fb03015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb03015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 -- 172.21.15.29:0/849696511 >> v2:172.21.15.186:3300/0 conn(0x7fb03015ebc0 msgr2=0x7fb03015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 --2- 172.21.15.29:0/849696511 >> v2:172.21.15.186:3300/0 conn(0x7fb03015ebc0 0x7fb03015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.508+0000 7fb034e57640 1 -- 172.21.15.29:0/849696511 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb03015bf90 con 0x7fb03006c040 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.509+0000 7fb034e57640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03006c040 0x7fb03006c420 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb024009870 tx=0x7fb02402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83bb64ee78d38b1c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.509+0000 7fb02f7fe640 1 -- 172.21.15.29:0/849696511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024031640 con 0x7fb03006c040 2023-10-14T16:59:08.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.509+0000 7fb02f7fe640 1 -- 172.21.15.29:0/849696511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0240317a0 con 0x7fb03006c040 2023-10-14T16:59:08.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.509+0000 7fb02f7fe640 1 -- 172.21.15.29:0/849696511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb024031b30 con 0x7fb03006c040 2023-10-14T16:59:08.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 -- 172.21.15.29:0/849696511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03006c040 msgr2=0x7fb03006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03006c040 0x7fb03006c420 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 crypto rx=0x7fb024009870 tx=0x7fb02402b8f0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 -- 172.21.15.29:0/849696511 shutdown_connections 2023-10-14T16:59:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 --2- 172.21.15.29:0/849696511 >> v2:172.21.15.186:3300/0 conn(0x7fb03015ebc0 0x7fb03015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb03015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 --2- 172.21.15.29:0/849696511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03006c040 0x7fb03006c420 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 -- 172.21.15.29:0/849696511 >> 172.21.15.29:0/849696511 conn(0x7fb0300ce6a0 msgr2=0x7fb0300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 -- 172.21.15.29:0/849696511 shutdown_connections 2023-10-14T16:59:08.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.510+0000 7fb035e59640 1 -- 172.21.15.29:0/849696511 wait complete. 2023-10-14T16:59:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.512+0000 7fb035e59640 1 Processor -- start 2023-10-14T16:59:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.512+0000 7fb035e59640 1 -- start start 2023-10-14T16:59:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb03006c040 0x7fb030152050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb0301581b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03015e350 con 0x7fb03015ebc0 2023-10-14T16:59:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb03015e070 con 0x7fb03006c040 2023-10-14T16:59:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0300afcd0 con 0x7fb0300ad970 2023-10-14T16:59:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.513+0000 7fb035658640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35826/0 (socket says 172.21.15.29:35826) 2023-10-14T16:59:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 -- 172.21.15.29:0/299104545 learned_addr learned my addr 172.21.15.29:0/299104545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 -- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 msgr2=0x7fb0301581b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb0301581b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 -- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb03006c040 msgr2=0x7fb030152050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb03006c040 0x7fb030152050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 -- 172.21.15.29:0/299104545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0240094d0 con 0x7fb03015ebc0 2023-10-14T16:59:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb035658640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb02800b970 tx=0x7fb02800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb02800cb40 con 0x7fb03015ebc0 2023-10-14T16:59:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb02800ccc0 con 0x7fb03015ebc0 2023-10-14T16:59:08.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.514+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb028014680 con 0x7fb03015ebc0 2023-10-14T16:59:08.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.515+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0300affb0 con 0x7fb03015ebc0 2023-10-14T16:59:08.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.515+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb030152590 con 0x7fb03015ebc0 2023-10-14T16:59:08.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.516+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb03006b330 con 0x7fb03015ebc0 2023-10-14T16:59:08.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.517+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 54) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb02801d3b0 con 0x7fb03015ebc0 2023-10-14T16:59:08.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.518+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7fb028089460 con 0x7fb03015ebc0 2023-10-14T16:59:08.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.522+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb028054350 con 0x7fb03015ebc0 2023-10-14T16:59:08.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.717+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0300b04d0 con 0x7fb03015ebc0 2023-10-14T16:59:08.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.718+0000 7fb02d7fa640 1 -- 172.21.15.29:0/299104545 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v113) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb0300b04d0 con 0x7fb03015ebc0 2023-10-14T16:59:08.719 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T16:59:08.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 msgr2=0x7fb0301586f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 crypto rx=0x7fb02800b970 tx=0x7fb02800be30 comp rx=0 tx=0).stop 2023-10-14T16:59:08.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 shutdown_connections 2023-10-14T16:59:08.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb03015ebc0 0x7fb0301586f0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0300ad970 0x7fb0301581b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 --2- 172.21.15.29:0/299104545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb03006c040 0x7fb030152050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 >> 172.21.15.29:0/299104545 conn(0x7fb0300ce6a0 msgr2=0x7fb0300cd200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:08.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 shutdown_connections 2023-10-14T16:59:08.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.720+0000 7fb035e59640 1 -- 172.21.15.29:0/299104545 wait complete. 2023-10-14T16:59:08.739 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.952+0000 7f52b7380640 1 Processor -- start 2023-10-14T16:59:08.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.952+0000 7f52b7380640 1 -- start start 2023-10-14T16:59:08.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b7380640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52b015cbe0 0x7f52b00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b7380640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f52b00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b7380640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52b00d15a0 con 0x7f52b00d1090 2023-10-14T16:59:08.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b7380640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52b00d1880 con 0x7f52b015cbe0 2023-10-14T16:59:08.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b7380640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00d1b60 con 0x7f52b015cfc0 2023-10-14T16:59:08.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b6b7f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f52b00d1090 0x7f52b006bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49514/0 (socket says 172.21.15.29:49514) 2023-10-14T16:59:08.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.954+0000 7f52b6b7f640 1 -- 172.21.15.29:0/1108600935 learned_addr learned my addr 172.21.15.29:0/1108600935 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b637e640 1 --2- 172.21.15.29:0/1108600935 >> v2:172.21.15.186:3300/0 conn(0x7f52b015cbe0 0x7f52b00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b5b7d640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f52b00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622979792 0 0) 0x7f52b00d15a0 con 0x7f52b00d1090 2023-10-14T16:59:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f529c003520 con 0x7f52b00d1090 2023-10-14T16:59:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1462206892 0 0) 0x7f529c003520 con 0x7f52b00d1090 2023-10-14T16:59:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 msgr2=0x7f52b00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f52b00b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 >> v2:172.21.15.186:3300/0 conn(0x7f52b015cbe0 msgr2=0x7f52b00d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 --2- 172.21.15.29:0/1108600935 >> v2:172.21.15.186:3300/0 conn(0x7f52b015cbe0 0x7f52b00d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b00d1e40 con 0x7f52b00d1090 2023-10-14T16:59:08.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f52a80019f0 con 0x7f52b00d1090 2023-10-14T16:59:08.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 >> v1:172.21.15.29:6789/0 conn(0x7f52b00d1090 legacy=0x7f52b006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.955+0000 7f52b537c640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c003970 0x7f529c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b537c640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f529c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b537c640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f529c004d00 0x7f529c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00d1880 con 0x7f529c004d00 2023-10-14T16:59:08.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00d1b60 con 0x7f529c003970 2023-10-14T16:59:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f529c00b980 con 0x7f52b015cfc0 2023-10-14T16:59:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f529c004d00 0x7f529c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b5b7d640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f529c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b637e640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c003970 0x7f529c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 -- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 msgr2=0x7f529c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b015cfc0 0x7f529c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 -- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c003970 msgr2=0x7f529c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f529c003970 0x7f529c003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.956+0000 7f52b6b7f640 1 -- 172.21.15.29:0/1108600935 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f529c003520 con 0x7f529c004d00 2023-10-14T16:59:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.957+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f529c004d00 0x7f529c005180 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a830 tx=0x7f529c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32f78782e5e5ec72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8034630 con 0x7f529c004d00 2023-10-14T16:59:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a802f070 con 0x7f529c004d00 2023-10-14T16:59:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b537c640 1 -- 172.21.15.29:0/1108600935 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a80349d0 con 0x7f529c004d00 2023-10-14T16:59:08.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 -- 172.21.15.29:0/1108600935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f529c004d00 msgr2=0x7f529c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 --2- 172.21.15.29:0/1108600935 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f529c004d00 0x7f529c005180 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a830 tx=0x7f529c0057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b6b7f640 1 -- 172.21.15.29:0/1108600935 reap_dead start 2023-10-14T16:59:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 -- 172.21.15.29:0/1108600935 shutdown_connections 2023-10-14T16:59:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 -- 172.21.15.29:0/1108600935 >> 172.21.15.29:0/1108600935 conn(0x7f52b00cf570 msgr2=0x7f52b00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 -- 172.21.15.29:0/1108600935 shutdown_connections 2023-10-14T16:59:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.958+0000 7f52b7380640 1 -- 172.21.15.29:0/1108600935 wait complete. 2023-10-14T16:59:08.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.960+0000 7f52b7380640 1 Processor -- start 2023-10-14T16:59:08.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.960+0000 7f52b7380640 1 -- start start 2023-10-14T16:59:08.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b01f1ed0 0x7f52b01572c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01f6560 0x7f52b01f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0153bd0 con 0x7f52b01f27f0 2023-10-14T16:59:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52b00c8f00 con 0x7f52b01f6560 2023-10-14T16:59:08.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.961+0000 7f52b7380640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52b0157800 con 0x7f52b01f1ed0 2023-10-14T16:59:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35856/0 (socket says 172.21.15.29:35856) 2023-10-14T16:59:08.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 -- 172.21.15.29:0/2502203803 learned_addr learned my addr 172.21.15.29:0/2502203803 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b637e640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b01f1ed0 0x7f52b01572c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b5b7d640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01f6560 0x7f52b01f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 -- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b01f1ed0 msgr2=0x7f52b01572c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b01f1ed0 0x7f52b01572c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 -- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01f6560 msgr2=0x7f52b01f22b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01f6560 0x7f52b01f22b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.962+0000 7f52b6b7f640 1 -- 172.21.15.29:0/2502203803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52b0153610 con 0x7f52b01f27f0 2023-10-14T16:59:08.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52b6b7f640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a830 tx=0x7f52b00b5b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8034840 con 0x7f52b01f27f0 2023-10-14T16:59:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52a80349a0 con 0x7f52b01f27f0 2023-10-14T16:59:08.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52a8034d10 con 0x7f52b01f27f0 2023-10-14T16:59:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52b01538f0 con 0x7f52b01f27f0 2023-10-14T16:59:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.963+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52b015e8e0 con 0x7f52b01f27f0 2023-10-14T16:59:08.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.964+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52b015e460 con 0x7f52b01f27f0 2023-10-14T16:59:08.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.965+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f52a8031030 con 0x7f52b01f27f0 2023-10-14T16:59:08.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.966+0000 7f52a6ffd640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 0x7f529406d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:08.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.967+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7f52a80ab2b0 con 0x7f52b01f27f0 2023-10-14T16:59:08.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.967+0000 7f52b637e640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 0x7f529406d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:08.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.967+0000 7f52b637e640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 0x7f529406d1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009a90 tx=0x7f52ac008040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:08.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:08.971+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52b015e460 con 0x7f52b01f27f0 2023-10-14T16:59:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.207+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52b015e460 con 0x7f52b01f27f0 2023-10-14T16:59:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.209+0000 7f52a6ffd640 1 -- 172.21.15.29:0/2502203803 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f52b015e460 con 0x7f52b01f27f0 2023-10-14T16:59:09.209 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:09.210 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:59:09.210 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:59:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.211+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 msgr2=0x7f529406d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.211+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 0x7f529406d1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f52ac009a90 tx=0x7f52ac008040 comp rx=0 tx=0).stop 2023-10-14T16:59:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.211+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 msgr2=0x7f52b0155060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.211+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 secure :-1 s=READY pgs=1322 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a830 tx=0x7f52b00b5b70 comp rx=0 tx=0).stop 2023-10-14T16:59:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 shutdown_connections 2023-10-14T16:59:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f529406acf0 0x7f529406d1b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52b01f27f0 0x7f52b0155060 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52b01f6560 0x7f52b01f22b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 --2- 172.21.15.29:0/2502203803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52b01f1ed0 0x7f52b01572c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 >> 172.21.15.29:0/2502203803 conn(0x7f52b00cf570 msgr2=0x7f52b00cd590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 shutdown_connections 2023-10-14T16:59:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.212+0000 7f52b7380640 1 -- 172.21.15.29:0/2502203803 wait complete. 2023-10-14T16:59:09.232 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.461+0000 7f0bfffff640 1 Processor -- start 2023-10-14T16:59:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.461+0000 7f0bfffff640 1 -- start start 2023-10-14T16:59:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c0015dde0 0x7f0c000709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0c00070f20 0x7f0c000ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c00160a90 con 0x7f0c0015c9f0 2023-10-14T16:59:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c00071350 con 0x7f0c0015dde0 2023-10-14T16:59:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c000cdec0 con 0x7f0c00070f20 2023-10-14T16:59:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.463+0000 7f0bfe7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c0015dde0 0x7f0c000709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c0015dde0 0x7f0c000709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35862/0 (socket says 172.21.15.29:35862) 2023-10-14T16:59:09.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 -- 172.21.15.29:0/2179929867 learned_addr learned my addr 172.21.15.29:0/2179929867 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bff7fe640 1 --2- 172.21.15.29:0/2179929867 >> v2:172.21.15.186:3300/0 conn(0x7f0c00070f20 0x7f0c000ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 -- 172.21.15.29:0/2179929867 >> v2:172.21.15.186:3300/0 conn(0x7f0c00070f20 msgr2=0x7f0c000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 --2- 172.21.15.29:0/2179929867 >> v2:172.21.15.186:3300/0 conn(0x7f0c00070f20 0x7f0c000ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 -- 172.21.15.29:0/2179929867 >> v1:172.21.15.186:6789/0 conn(0x7f0c0015c9f0 legacy=0x7f0c000c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.464+0000 7f0bfe7fc640 1 -- 172.21.15.29:0/2179929867 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c000cd900 con 0x7f0c0015dde0 2023-10-14T16:59:09.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.465+0000 7f0bfe7fc640 1 --2- 172.21.15.29:0/2179929867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c0015dde0 0x7f0c000709e0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f0c0006fcc0 tx=0x7f0be802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=437366c38cf30acc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.466+0000 7f0bfdffb640 1 -- 172.21.15.29:0/2179929867 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8030860 con 0x7f0c0015dde0 2023-10-14T16:59:09.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.466+0000 7f0bfdffb640 1 -- 172.21.15.29:0/2179929867 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0be80309c0 con 0x7f0c0015dde0 2023-10-14T16:59:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.466+0000 7f0bfdffb640 1 -- 172.21.15.29:0/2179929867 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0be8030d50 con 0x7f0c0015dde0 2023-10-14T16:59:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.466+0000 7f0bfffff640 1 -- 172.21.15.29:0/2179929867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c0015dde0 msgr2=0x7f0c000709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.466+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2179929867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c0015dde0 0x7f0c000709e0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 crypto rx=0x7f0c0006fcc0 tx=0x7f0be802b9a0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 -- 172.21.15.29:0/2179929867 shutdown_connections 2023-10-14T16:59:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2179929867 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c0015dde0 0x7f0c000709e0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2179929867 >> v2:172.21.15.186:3300/0 conn(0x7f0c00070f20 0x7f0c000ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 -- 172.21.15.29:0/2179929867 >> 172.21.15.29:0/2179929867 conn(0x7f0c000ce6a0 msgr2=0x7f0c000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 -- 172.21.15.29:0/2179929867 shutdown_connections 2023-10-14T16:59:09.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.467+0000 7f0bfffff640 1 -- 172.21.15.29:0/2179929867 wait complete. 2023-10-14T16:59:09.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.468+0000 7f0bfffff640 1 Processor -- start 2023-10-14T16:59:09.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.469+0000 7f0bfffff640 1 -- start start 2023-10-14T16:59:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.469+0000 7f0bfffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c0015c9f0 0x7f0c0006c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c0015dde0 0x7f0c0015a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0014d9d0 con 0x7f0c00070f20 2023-10-14T16:59:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c0006cbf0 con 0x7f0c0015c9f0 2023-10-14T16:59:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c001f2690 con 0x7f0c0015dde0 2023-10-14T16:59:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35864/0 (socket says 172.21.15.29:35864) 2023-10-14T16:59:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 -- 172.21.15.29:0/2684988001 learned_addr learned my addr 172.21.15.29:0/2684988001 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bff7fe640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c0015dde0 0x7f0c0015a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfe7fc640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c0015c9f0 0x7f0c0006c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 -- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c0015dde0 msgr2=0x7f0c0015a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c0015dde0 0x7f0c0015a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 -- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c0015c9f0 msgr2=0x7f0c0006c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c0015c9f0 0x7f0c0006c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.470+0000 7f0bfeffd640 1 -- 172.21.15.29:0/2684988001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be80094d0 con 0x7f0c00070f20 2023-10-14T16:59:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bfeffd640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c450 tx=0x7f0bf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf00052d0 con 0x7f0c00070f20 2023-10-14T16:59:09.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bf0005430 con 0x7f0c00070f20 2023-10-14T16:59:09.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c000aee70 con 0x7f0c00070f20 2023-10-14T16:59:09.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0bf0012870 con 0x7f0c00070f20 2023-10-14T16:59:09.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.471+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c00151970 con 0x7f0c00070f20 2023-10-14T16:59:09.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.472+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c0004e9b0 con 0x7f0c00070f20 2023-10-14T16:59:09.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.473+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0bf00129d0 con 0x7f0c00070f20 2023-10-14T16:59:09.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.474+0000 7f0bf7fff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 0x7f0be006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.474+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7f0bf00890c0 con 0x7f0c00070f20 2023-10-14T16:59:09.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.475+0000 7f0bfe7fc640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 0x7f0be006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.476+0000 7f0bfe7fc640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 0x7f0be006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0be8036000 tx=0x7f0be803b040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.479+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0bf0054160 con 0x7f0c00070f20 2023-10-14T16:59:09.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.677+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0c0006a7b0 con 0x7f0c00070f20 2023-10-14T16:59:09.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.680+0000 7f0bf7fff640 1 -- 172.21.15.29:0/2684988001 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17800 (secure 0 0 0) 0x7f0bf0057780 con 0x7f0c00070f20 2023-10-14T16:59:09.681 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:09.681 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:07.763653+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6832/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:6833/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/712849763":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/1941242736":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/199695451":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2087407689":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/3563840307":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/869869993":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2128677149":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/3340406750":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3651737802":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/583660233":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/856743829":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2619862699":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3275044227":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6833/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6832/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/680310103":"2023-10-15T16:57:59.666943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:09.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.682+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 msgr2=0x7f0be006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.682+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 0x7f0be006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0be8036000 tx=0x7f0be803b040 comp rx=0 tx=0).stop 2023-10-14T16:59:09.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.682+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 msgr2=0x7f0c0014d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 secure :-1 s=READY pgs=1324 cs=0 l=1 rev1=1 crypto rx=0x7f0bf000c450 tx=0x7f0bf000c910 comp rx=0 tx=0).stop 2023-10-14T16:59:09.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 shutdown_connections 2023-10-14T16:59:09.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0be006acc0 0x7f0be006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c0015dde0 0x7f0c0015a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c0015c9f0 0x7f0c0006c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 --2- 172.21.15.29:0/2684988001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c00070f20 0x7f0c0014d490 unknown :-1 s=CLOSED pgs=1324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 >> 172.21.15.29:0/2684988001 conn(0x7f0c000ce6a0 msgr2=0x7f0c0015e840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 shutdown_connections 2023-10-14T16:59:09.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.683+0000 7f0bfffff640 1 -- 172.21.15.29:0/2684988001 wait complete. 2023-10-14T16:59:09.702 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T16:59:09.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.932+0000 7ff19c0e7640 1 Processor -- start 2023-10-14T16:59:09.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.932+0000 7ff19c0e7640 1 -- start start 2023-10-14T16:59:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.933+0000 7ff19c0e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff194159b20 0x7ff19406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.933+0000 7ff19c0e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.933+0000 7ff19c0e7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff19406f7f0 con 0x7ff19406c120 2023-10-14T16:59:09.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.933+0000 7ff19c0e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff19406c5e0 con 0x7ff194159b20 2023-10-14T16:59:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.933+0000 7ff19c0e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff19406c8c0 con 0x7ff194159f00 2023-10-14T16:59:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.934+0000 7ff199e83640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff194159b20 0x7ff19406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.934+0000 7ff199682640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.934+0000 7ff199682640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:41952/0 (socket says 172.21.15.29:41952) 2023-10-14T16:59:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.934+0000 7ff199682640 1 -- 172.21.15.29:0/303198232 learned_addr learned my addr 172.21.15.29:0/303198232 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:09.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.935+0000 7ff199e83640 1 -- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 msgr2=0x7ff1940b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.935+0000 7ff199e83640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.935+0000 7ff199e83640 1 -- 172.21.15.29:0/303198232 >> v1:172.21.15.186:6789/0 conn(0x7ff19406c120 legacy=0x7ff1940cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.935+0000 7ff199e83640 1 -- 172.21.15.29:0/303198232 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff19406cba0 con 0x7ff194159b20 2023-10-14T16:59:09.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.935+0000 7ff199682640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:59:09.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.937+0000 7ff199e83640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159b20 0x7ff19406bbe0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19002dac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d573b611eae6f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff198e81640 1 -- 172.21.15.29:0/303198232 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002f880 con 0x7ff194159b20 2023-10-14T16:59:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff198e81640 1 -- 172.21.15.29:0/303198232 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff19002f9e0 con 0x7ff194159b20 2023-10-14T16:59:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff198e81640 1 -- 172.21.15.29:0/303198232 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff19002fd70 con 0x7ff194159b20 2023-10-14T16:59:09.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 -- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159b20 msgr2=0x7ff19406bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159b20 0x7ff19406bbe0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7ff1900097a0 tx=0x7ff19002dac0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 -- 172.21.15.29:0/303198232 shutdown_connections 2023-10-14T16:59:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159f00 0x7ff1940b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/303198232 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159b20 0x7ff19406bbe0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 -- 172.21.15.29:0/303198232 >> 172.21.15.29:0/303198232 conn(0x7ff1940cb410 msgr2=0x7ff1940cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 -- 172.21.15.29:0/303198232 shutdown_connections 2023-10-14T16:59:09.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.938+0000 7ff19c0e7640 1 -- 172.21.15.29:0/303198232 wait complete. 2023-10-14T16:59:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.940+0000 7ff19c0e7640 1 Processor -- start 2023-10-14T16:59:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.940+0000 7ff19c0e7640 1 -- start start 2023-10-14T16:59:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 0x7ff19415c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159f00 0x7ff1940b34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940acbc0 con 0x7ff19406c120 2023-10-14T16:59:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940ace60 con 0x7ff194159f00 2023-10-14T16:59:09.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.941+0000 7ff19c0e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1940ad140 con 0x7ff194159b20 2023-10-14T16:59:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35902/0 (socket says 172.21.15.29:35902) 2023-10-14T16:59:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 -- 172.21.15.29:0/4201455660 learned_addr learned my addr 172.21.15.29:0/4201455660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199682640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 0x7ff19415c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff19a684640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159f00 0x7ff1940b34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 -- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 msgr2=0x7ff19415c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 0x7ff19415c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 -- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159f00 msgr2=0x7ff1940b34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:09.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159f00 0x7ff1940b34a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:09.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 -- 172.21.15.29:0/4201455660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1840094d0 con 0x7ff19406c120 2023-10-14T16:59:09.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199682640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 0x7ff19415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:09.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.942+0000 7ff199e83640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff19002e0d0 tx=0x7ff1900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.943+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190030a90 con 0x7ff19406c120 2023-10-14T16:59:09.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.943+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff190030c10 con 0x7ff19406c120 2023-10-14T16:59:09.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.943+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff190042410 con 0x7ff19406c120 2023-10-14T16:59:09.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.943+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff190009400 con 0x7ff19406c120 2023-10-14T16:59:09.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.943+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1940ad700 con 0x7ff19406c120 2023-10-14T16:59:09.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.944+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff19404e160 con 0x7ff19406c120 2023-10-14T16:59:09.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.945+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 55) v1 ==== 89067+0+0 (secure 0 0 0) 0x7ff190039400 con 0x7ff19406c120 2023-10-14T16:59:09.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.946+0000 7ff18a7fc640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 0x7ff17806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:09.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.946+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7ff1900ab140 con 0x7ff19406c120 2023-10-14T16:59:09.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.946+0000 7ff199682640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 0x7ff17806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:09.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.947+0000 7ff199682640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 0x7ff17806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1940cbe20 tx=0x7ff184008040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:09.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:09.950+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1900761b0 con 0x7ff19406c120 2023-10-14T16:59:10.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.186+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff1940adb30 con 0x7ff19406c120 2023-10-14T16:59:10.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.188+0000 7ff18a7fc640 1 -- 172.21.15.29:0/4201455660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7ff1900797d0 con 0x7ff19406c120 2023-10-14T16:59:10.188 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:10.189 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:59:10.189 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:59:10.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 msgr2=0x7ff17806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 0x7ff17806d180 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7ff1940cbe20 tx=0x7ff184008040 comp rx=0 tx=0).stop 2023-10-14T16:59:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 msgr2=0x7ff19414ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 secure :-1 s=READY pgs=1327 cs=0 l=1 rev1=1 crypto rx=0x7ff19002e0d0 tx=0x7ff1900046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 shutdown_connections 2023-10-14T16:59:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff17806acc0 0x7ff17806d180 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.190+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff194159f00 0x7ff1940b34a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.191+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff194159b20 0x7ff19415c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.191+0000 7ff19c0e7640 1 --2- 172.21.15.29:0/4201455660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff19406c120 0x7ff19414ed30 unknown :-1 s=CLOSED pgs=1327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.191+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 >> 172.21.15.29:0/4201455660 conn(0x7ff1940cb410 msgr2=0x7ff1940c7a70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:10.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.191+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 shutdown_connections 2023-10-14T16:59:10.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.191+0000 7ff19c0e7640 1 -- 172.21.15.29:0/4201455660 wait complete. 2023-10-14T16:59:10.210 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:10.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.429+0000 7f7401e75640 1 Processor -- start 2023-10-14T16:59:10.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.429+0000 7f7401e75640 1 -- start start 2023-10-14T16:59:10.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f7401e75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0cfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f7401e75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc071d80 con 0x7f73fc15ac80 2023-10-14T16:59:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f7401e75640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73fc0cca60 con 0x7f73fc0cd870 2023-10-14T16:59:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f7401e75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0ccd40 con 0x7f73fc0718f0 2023-10-14T16:59:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f73fbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f73fc15ac80 0x7f73fc0cdc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49530/0 (socket says 172.21.15.29:49530) 2023-10-14T16:59:10.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f73fbfff640 1 -- 172.21.15.29:0/1688462012 learned_addr learned my addr 172.21.15.29:0/1688462012 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.431+0000 7f7401674640 1 --2- 172.21.15.29:0/1688462012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0cfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.432+0000 7f7401674640 1 -- 172.21.15.29:0/1688462012 >> v1:172.21.15.186:6789/0 conn(0x7f73fc0cd870 legacy=0x7f73fc0cc2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.432+0000 7f7401674640 1 -- 172.21.15.29:0/1688462012 >> v1:172.21.15.29:6789/0 conn(0x7f73fc15ac80 legacy=0x7f73fc0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.432+0000 7f7401674640 1 -- 172.21.15.29:0/1688462012 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73fc0cd020 con 0x7f73fc0718f0 2023-10-14T16:59:10.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.434+0000 7f7401674640 1 --2- 172.21.15.29:0/1688462012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0cfab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f73f40097d0 tx=0x7f73f402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6054983d228d5c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.441+0000 7f73fb7fe640 1 -- 172.21.15.29:0/1688462012 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4031890 con 0x7f73fc0718f0 2023-10-14T16:59:10.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.441+0000 7f73fb7fe640 1 -- 172.21.15.29:0/1688462012 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f40319f0 con 0x7f73fc0718f0 2023-10-14T16:59:10.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.441+0000 7f73fb7fe640 1 -- 172.21.15.29:0/1688462012 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4031d80 con 0x7f73fc0718f0 2023-10-14T16:59:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 -- 172.21.15.29:0/1688462012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 msgr2=0x7f73fc0cfab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 --2- 172.21.15.29:0/1688462012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0cfab0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f73f40097d0 tx=0x7f73f402b930 comp rx=0 tx=0).stop 2023-10-14T16:59:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 -- 172.21.15.29:0/1688462012 shutdown_connections 2023-10-14T16:59:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 --2- 172.21.15.29:0/1688462012 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0cfab0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 -- 172.21.15.29:0/1688462012 >> 172.21.15.29:0/1688462012 conn(0x7f73fc065ed0 msgr2=0x7f73fc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 -- 172.21.15.29:0/1688462012 shutdown_connections 2023-10-14T16:59:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.442+0000 7f7401e75640 1 -- 172.21.15.29:0/1688462012 wait complete. 2023-10-14T16:59:10.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.445+0000 7f7401e75640 1 Processor -- start 2023-10-14T16:59:10.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.445+0000 7f7401e75640 1 -- start start 2023-10-14T16:59:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.446+0000 7f7401e75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0ca880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401e75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73fc0cd870 0x7f73fc14ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401e75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401e75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc14f100 con 0x7f73fc15ac80 2023-10-14T16:59:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401e75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0c5ee0 con 0x7f73fc0cd870 2023-10-14T16:59:10.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401e75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73fc0c61c0 con 0x7f73fc0718f0 2023-10-14T16:59:10.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401674640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401674640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:35904/0 (socket says 172.21.15.29:35904) 2023-10-14T16:59:10.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.447+0000 7f7401674640 1 -- 172.21.15.29:0/75765022 learned_addr learned my addr 172.21.15.29:0/75765022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:10.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 -- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 msgr2=0x7f73fc0ca880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7400e73640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0ca880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0ca880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 -- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73fc0cd870 msgr2=0x7f73fc14ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73fc0cd870 0x7f73fc14ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 -- 172.21.15.29:0/75765022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f4009400 con 0x7f73fc15ac80 2023-10-14T16:59:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.448+0000 7f7401674640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f73f4034040 tx=0x7f73f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.449+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f4031830 con 0x7f73fc15ac80 2023-10-14T16:59:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.449+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73fc0c64a0 con 0x7f73fc15ac80 2023-10-14T16:59:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.449+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f40443f0 con 0x7f73fc15ac80 2023-10-14T16:59:10.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.449+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f40447a0 con 0x7f73fc15ac80 2023-10-14T16:59:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.449+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73fc0c6a60 con 0x7f73fc15ac80 2023-10-14T16:59:10.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.451+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc0cf8f0 con 0x7f73fc15ac80 2023-10-14T16:59:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.451+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f73f404d450 con 0x7f73fc15ac80 2023-10-14T16:59:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.453+0000 7f73f97fa640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 0x7f73e806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.453+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7f73f40ac2c0 con 0x7f73fc15ac80 2023-10-14T16:59:10.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.453+0000 7f7400e73640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 0x7f73e806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.454+0000 7f7400e73640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 0x7f73e806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73ec001330 tx=0x7f73ec009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.458+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73f4076e30 con 0x7f73fc15ac80 2023-10-14T16:59:10.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.695+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73fc0c6c50 con 0x7f73fc15ac80 2023-10-14T16:59:10.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.696+0000 7f73f97fa640 1 -- 172.21.15.29:0/75765022 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 49 v49) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f73f407a450 con 0x7f73fc15ac80 2023-10-14T16:59:10.698 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:10.698 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":49,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43}],"filesystems":[{"mdsmap":{"epoch":49,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:57:44.731151+0000","modified":"2023-10-14T16:57:47.795224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":6076},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6076":{"gid":6076,"name":"c","rank":0,"incarnation":45,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1784427614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1784427614},{"type":"v1","addr":"172.21.15.29:6836","nonce":1784427614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[13],"metadata_pool":12,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":6}]} 2023-10-14T16:59:10.698 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 49 2023-10-14T16:59:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 msgr2=0x7f73e806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 0x7f73e806d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f73ec001330 tx=0x7f73ec009040 comp rx=0 tx=0).stop 2023-10-14T16:59:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 msgr2=0x7f73fc0b1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 crypto rx=0x7f73f4034040 tx=0x7f73f40046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 shutdown_connections 2023-10-14T16:59:10.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f73e806af00 0x7f73e806d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73fc15ac80 0x7f73fc0b1460 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.699+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73fc0cd870 0x7f73fc14ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.700+0000 7f7401e75640 1 --2- 172.21.15.29:0/75765022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73fc0718f0 0x7f73fc0ca880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.700+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 >> 172.21.15.29:0/75765022 conn(0x7f73fc065ed0 msgr2=0x7f73fc0b4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:10.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.700+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 shutdown_connections 2023-10-14T16:59:10.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.700+0000 7f7401e75640 1 -- 172.21.15.29:0/75765022 wait complete. 2023-10-14T16:59:10.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:10.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.951+0000 7fbb6b589640 1 Processor -- start 2023-10-14T16:59:10.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.951+0000 7fbb6b589640 1 -- start start 2023-10-14T16:59:10.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb6b589640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbb64159b20 0x7fbb6406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb6b589640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb640b0b20 0x7fbb6406fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb6b589640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6406f7f0 con 0x7fbb64159f00 2023-10-14T16:59:10.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb6b589640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6406c5e0 con 0x7fbb640b0b20 2023-10-14T16:59:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb6b589640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6406c8c0 con 0x7fbb64159b20 2023-10-14T16:59:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb69b26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb640b0b20 0x7fbb6406fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb69b26640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbb640b0b20 0x7fbb6406fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41764/0 (socket says 172.21.15.29:41764) 2023-10-14T16:59:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.953+0000 7fbb69b26640 1 -- 172.21.15.29:0/1169225379 learned_addr learned my addr 172.21.15.29:0/1169225379 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:10.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.954+0000 7fbb69325640 1 --2- 172.21.15.29:0/1169225379 >> v2:172.21.15.186:3300/0 conn(0x7fbb64159b20 0x7fbb6406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.954+0000 7fbb69b26640 1 -- 172.21.15.29:0/1169225379 >> v2:172.21.15.186:3300/0 conn(0x7fbb64159b20 msgr2=0x7fbb6406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.954+0000 7fbb69b26640 1 --2- 172.21.15.29:0/1169225379 >> v2:172.21.15.186:3300/0 conn(0x7fbb64159b20 0x7fbb6406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.954+0000 7fbb69b26640 1 -- 172.21.15.29:0/1169225379 >> v1:172.21.15.186:6789/0 conn(0x7fbb64159f00 legacy=0x7fbb640b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.954+0000 7fbb69b26640 1 -- 172.21.15.29:0/1169225379 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6406cba0 con 0x7fbb640b0b20 2023-10-14T16:59:10.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.955+0000 7fbb69b26640 1 --2- 172.21.15.29:0/1169225379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb640b0b20 0x7fbb6406fc80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbb640c9340 tx=0x7fbb5c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d13fe3442dcc891 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.955+0000 7fbb5bfff640 1 -- 172.21.15.29:0/1169225379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c02ca80 con 0x7fbb640b0b20 2023-10-14T16:59:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.955+0000 7fbb5bfff640 1 -- 172.21.15.29:0/1169225379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c02cbe0 con 0x7fbb640b0b20 2023-10-14T16:59:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.955+0000 7fbb5bfff640 1 -- 172.21.15.29:0/1169225379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c0343f0 con 0x7fbb640b0b20 2023-10-14T16:59:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 -- 172.21.15.29:0/1169225379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb640b0b20 msgr2=0x7fbb6406fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 --2- 172.21.15.29:0/1169225379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb640b0b20 0x7fbb6406fc80 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 crypto rx=0x7fbb640c9340 tx=0x7fbb5c02bee0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 -- 172.21.15.29:0/1169225379 shutdown_connections 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 --2- 172.21.15.29:0/1169225379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb640b0b20 0x7fbb6406fc80 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 --2- 172.21.15.29:0/1169225379 >> v2:172.21.15.186:3300/0 conn(0x7fbb64159b20 0x7fbb6406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 -- 172.21.15.29:0/1169225379 >> 172.21.15.29:0/1169225379 conn(0x7fbb640cb410 msgr2=0x7fbb640cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 -- 172.21.15.29:0/1169225379 shutdown_connections 2023-10-14T16:59:10.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.956+0000 7fbb6b589640 1 -- 172.21.15.29:0/1169225379 wait complete. 2023-10-14T16:59:10.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.958+0000 7fbb6b589640 1 Processor -- start 2023-10-14T16:59:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.958+0000 7fbb6b589640 1 -- start start 2023-10-14T16:59:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 0x7fbb640c4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 0x7fbb6415be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6415aaf0 con 0x7fbb64159f00 2023-10-14T16:59:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6415add0 con 0x7fbb640b0b20 2023-10-14T16:59:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.959+0000 7fbb6b589640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb6415b0b0 con 0x7fbb64159b20 2023-10-14T16:59:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41766/0 (socket says 172.21.15.29:41766) 2023-10-14T16:59:10.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 -- 172.21.15.29:0/896481947 learned_addr learned my addr 172.21.15.29:0/896481947 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69325640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 0x7fbb640c4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb68b24640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 0x7fbb6415be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 -- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 msgr2=0x7fbb6415be70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 0x7fbb6415be70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 -- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 msgr2=0x7fbb640c4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 0x7fbb640c4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69b26640 1 -- 172.21.15.29:0/896481947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb5c009b30 con 0x7fbb64159f00 2023-10-14T16:59:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb69325640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 0x7fbb640c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.960+0000 7fbb68b24640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 0x7fbb6415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:10.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb69b26640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c036040 tx=0x7fbb5c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c009510 con 0x7fbb64159f00 2023-10-14T16:59:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c03e450 con 0x7fbb64159f00 2023-10-14T16:59:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb5c03e7e0 con 0x7fbb64159f00 2023-10-14T16:59:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6415b390 con 0x7fbb64159f00 2023-10-14T16:59:10.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.961+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6415b950 con 0x7fbb64159f00 2023-10-14T16:59:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.962+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6415bb40 con 0x7fbb64159f00 2023-10-14T16:59:10.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.963+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fbb5c02cd50 con 0x7fbb64159f00 2023-10-14T16:59:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.964+0000 7fbb59ffb640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 0x7fbb4806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.964+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7fbb5c0ac5e0 con 0x7fbb64159f00 2023-10-14T16:59:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.965+0000 7fbb69325640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 0x7fbb4806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:10.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.965+0000 7fbb69325640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 0x7fbb4806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb60006fd0 tx=0x7fbb60008040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:10.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:10.969+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb5c077250 con 0x7fbb64159f00 2023-10-14T16:59:11.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.167+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb6404e160 con 0x7fbb64159f00 2023-10-14T16:59:11.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.168+0000 7fbb59ffb640 1 -- 172.21.15.29:0/896481947 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v113) v1 ==== 74+0+17800 (secure 0 0 0) 0x7fbb5c03e940 con 0x7fbb64159f00 2023-10-14T16:59:11.170 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:11.170 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:07.763653+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":12,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:57:42.669157+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"108","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":13,"pool_name":"cephfs_data","create_time":"2023-10-14T16:57:43.755222+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"109","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"13.3b","mappings":[{"from":1,"to":3}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6832/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:6833/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/712849763":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/1941242736":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/199695451":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2087407689":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/3563840307":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/869869993":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2128677149":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/3340406750":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3651737802":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/583660233":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/856743829":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2619862699":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3275044227":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6833/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6832/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/680310103":"2023-10-15T16:57:59.666943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:11.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.171+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 msgr2=0x7fbb4806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.171+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 0x7fbb4806d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fbb60006fd0 tx=0x7fbb60008040 comp rx=0 tx=0).stop 2023-10-14T16:59:11.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.171+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 msgr2=0x7fbb640c2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.171+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 crypto rx=0x7fbb5c036040 tx=0x7fbb5c004a40 comp rx=0 tx=0).stop 2023-10-14T16:59:11.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 shutdown_connections 2023-10-14T16:59:11.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fbb4806af00 0x7fbb4806d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb64159f00 0x7fbb640c2730 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb64159b20 0x7fbb6415be70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 --2- 172.21.15.29:0/896481947 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb640b0b20 0x7fbb640c4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 >> 172.21.15.29:0/896481947 conn(0x7fbb640cb410 msgr2=0x7fbb641578e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:11.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 shutdown_connections 2023-10-14T16:59:11.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.172+0000 7fbb6b589640 1 -- 172.21.15.29:0/896481947 wait complete. 2023-10-14T16:59:11.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T16:59:11.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.409+0000 7f35e830f640 1 Processor -- start 2023-10-14T16:59:11.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.409+0000 7f35e830f640 1 -- start start 2023-10-14T16:59:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e830f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35e015ad70 0x7f35e0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e830f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e00c9ca0 0x7f35e006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e830f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e00cb030 con 0x7f35e00ca3b0 2023-10-14T16:59:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e830f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35e00c91c0 con 0x7f35e015ad70 2023-10-14T16:59:11.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e830f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00c94a0 con 0x7f35e00c9ca0 2023-10-14T16:59:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e60ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35e015ad70 0x7f35e0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e60ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35e015ad70 0x7f35e0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41780/0 (socket says 172.21.15.29:41780) 2023-10-14T16:59:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e60ab640 1 -- 172.21.15.29:0/624533002 learned_addr learned my addr 172.21.15.29:0/624533002 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.411+0000 7f35e58aa640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e00c9ca0 0x7f35e006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.412+0000 7f35e60ab640 1 -- 172.21.15.29:0/624533002 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e00c9ca0 msgr2=0x7f35e006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.412+0000 7f35e60ab640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e00c9ca0 0x7f35e006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.412+0000 7f35e60ab640 1 -- 172.21.15.29:0/624533002 >> v1:172.21.15.186:6789/0 conn(0x7f35e00ca3b0 legacy=0x7f35e00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.412+0000 7f35e60ab640 1 -- 172.21.15.29:0/624533002 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e00c9780 con 0x7f35e015ad70 2023-10-14T16:59:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.414+0000 7f35e60ab640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e015ad70 0x7f35e0069da0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009870 tx=0x7f35dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ac2393e5f714dbb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.414+0000 7f35e50a9640 1 -- 172.21.15.29:0/624533002 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc030840 con 0x7f35e015ad70 2023-10-14T16:59:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.414+0000 7f35e50a9640 1 -- 172.21.15.29:0/624533002 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc0309a0 con 0x7f35e015ad70 2023-10-14T16:59:11.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.414+0000 7f35e50a9640 1 -- 172.21.15.29:0/624533002 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc030d30 con 0x7f35e015ad70 2023-10-14T16:59:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 -- 172.21.15.29:0/624533002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e015ad70 msgr2=0x7f35e0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e015ad70 0x7f35e0069da0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 crypto rx=0x7f35dc009870 tx=0x7f35dc02b980 comp rx=0 tx=0).stop 2023-10-14T16:59:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 -- 172.21.15.29:0/624533002 shutdown_connections 2023-10-14T16:59:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e00c9ca0 0x7f35e006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 --2- 172.21.15.29:0/624533002 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e015ad70 0x7f35e0069da0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 -- 172.21.15.29:0/624533002 >> 172.21.15.29:0/624533002 conn(0x7f35e00ce9b0 msgr2=0x7f35e00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 -- 172.21.15.29:0/624533002 shutdown_connections 2023-10-14T16:59:11.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.415+0000 7f35e830f640 1 -- 172.21.15.29:0/624533002 wait complete. 2023-10-14T16:59:11.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.417+0000 7f35e830f640 1 Processor -- start 2023-10-14T16:59:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.417+0000 7f35e830f640 1 -- start start 2023-10-14T16:59:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35e00ca3b0 0x7f35e00b2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 0x7f35e015b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00b3260 con 0x7f35e00c9ca0 2023-10-14T16:59:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00b3540 con 0x7f35e00ca3b0 2023-10-14T16:59:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e830f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e00b3820 con 0x7f35e015ad70 2023-10-14T16:59:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e60ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e60ab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41786/0 (socket says 172.21.15.29:41786) 2023-10-14T16:59:11.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.418+0000 7f35e60ab640 1 -- 172.21.15.29:0/1425949337 learned_addr learned my addr 172.21.15.29:0/1425949337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e58aa640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35e00ca3b0 0x7f35e00b2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e68ac640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 0x7f35e015b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 -- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 msgr2=0x7f35e015b4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 0x7f35e015b4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 -- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35e00ca3b0 msgr2=0x7f35e00b2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35e00ca3b0 0x7f35e00b2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 -- 172.21.15.29:0/1425949337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35dc0094d0 con 0x7f35e00c9ca0 2023-10-14T16:59:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e68ac640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 0x7f35e015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:11.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35e60ab640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.419+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc039a90 con 0x7f35e00c9ca0 2023-10-14T16:59:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.420+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35dc039bf0 con 0x7f35e00c9ca0 2023-10-14T16:59:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.420+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35dc043410 con 0x7f35e00c9ca0 2023-10-14T16:59:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.420+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e00b3b00 con 0x7f35e00c9ca0 2023-10-14T16:59:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.420+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e0154470 con 0x7f35e00c9ca0 2023-10-14T16:59:11.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.421+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e004e160 con 0x7f35e00c9ca0 2023-10-14T16:59:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.422+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f35dc02f390 con 0x7f35e00c9ca0 2023-10-14T16:59:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.423+0000 7f35d6ffd640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 0x7f35c406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.423+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6556+0+0 (secure 0 0 0) 0x7f35dc0abd90 con 0x7f35e00c9ca0 2023-10-14T16:59:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.423+0000 7f35e58aa640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 0x7f35c406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.424+0000 7f35e58aa640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 0x7f35c406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35e01f36d0 tx=0x7f35d0008040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.427+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35dc0768f0 con 0x7f35e00c9ca0 2023-10-14T16:59:11.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.665+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f35e00caa50 con 0x7f35e00c9ca0 2023-10-14T16:59:11.666 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T16:59:11.666+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T16:59:11.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.669+0000 7f35d6ffd640 1 -- 172.21.15.29:0/1425949337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v50) v1 ==== 161+0+0 (secure 0 0 0) 0x7f35dc079f10 con 0x7f35e00c9ca0 2023-10-14T16:59:11.670 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T16:59:11.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.673+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 msgr2=0x7f35c406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.673+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 0x7f35c406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f35e01f36d0 tx=0x7f35d0008040 comp rx=0 tx=0).stop 2023-10-14T16:59:11.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.673+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 msgr2=0x7f35e0153f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.673+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 crypto rx=0x7f35dc000c00 tx=0x7f35dc004730 comp rx=0 tx=0).stop 2023-10-14T16:59:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 shutdown_connections 2023-10-14T16:59:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f35c406af00 0x7f35c406d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35e015ad70 0x7f35e015b4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35e00ca3b0 0x7f35e00b2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 --2- 172.21.15.29:0/1425949337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35e00c9ca0 0x7f35e0153f30 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 >> 172.21.15.29:0/1425949337 conn(0x7f35e00ce9b0 msgr2=0x7f35e015dd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.674+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 shutdown_connections 2023-10-14T16:59:11.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.675+0000 7f35e830f640 1 -- 172.21.15.29:0/1425949337 wait complete. 2023-10-14T16:59:11.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T16:59:11.698 INFO:tasks.ceph.mds.c.smithi029.stderr: -6877> 2023-10-14T16:57:50.224+0000 7febe022e640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:59:11.758 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:59:11.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.898+0000 7f590c947640 1 Processor -- start 2023-10-14T16:59:11.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.898+0000 7f590c947640 1 -- start start 2023-10-14T16:59:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.899+0000 7f590c947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.899+0000 7f590c947640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f59080d0be0 0x7f59080cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.899+0000 7f590c947640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f590815d150 con 0x7f590815cd70 2023-10-14T16:59:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.899+0000 7f590c947640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59080d1180 con 0x7f59080d0be0 2023-10-14T16:59:11.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.899+0000 7f590c947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59080d1460 con 0x7f59080b8480 2023-10-14T16:59:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.900+0000 7f59077fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f590815cd70 0x7f59080d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:53856/0 (socket says 172.21.15.29:53856) 2023-10-14T16:59:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.900+0000 7f59077fe640 1 -- 172.21.15.29:0/1063938276 learned_addr learned my addr 172.21.15.29:0/1063938276 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.900+0000 7f5907fff640 1 --2- 172.21.15.29:0/1063938276 >> v2:172.21.15.186:3300/0 conn(0x7f59080d0be0 0x7f59080cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.900+0000 7f5906ffd640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.901+0000 7f5907fff640 1 -- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 msgr2=0x7f59080cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.901+0000 7f5907fff640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.901+0000 7f5907fff640 1 -- 172.21.15.29:0/1063938276 >> v1:172.21.15.186:6789/0 conn(0x7f590815cd70 legacy=0x7f59080d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.901+0000 7f5907fff640 1 -- 172.21.15.29:0/1063938276 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59080d1740 con 0x7f59080d0be0 2023-10-14T16:59:11.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.902+0000 7f5907fff640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080cdd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59000097a0 tx=0x7f590002b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=60caef0bd2fda75a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.903+0000 7f59067fc640 1 -- 172.21.15.29:0/1063938276 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590002d040 con 0x7f59080d0be0 2023-10-14T16:59:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.903+0000 7f59067fc640 1 -- 172.21.15.29:0/1063938276 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5900031810 con 0x7f59080d0be0 2023-10-14T16:59:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.903+0000 7f59067fc640 1 -- 172.21.15.29:0/1063938276 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5900031ba0 con 0x7f59080d0be0 2023-10-14T16:59:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.903+0000 7f590c947640 1 -- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 msgr2=0x7f59080cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.903+0000 7f590c947640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080cdd00 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f59000097a0 tx=0x7f590002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 -- 172.21.15.29:0/1063938276 shutdown_connections 2023-10-14T16:59:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 --2- 172.21.15.29:0/1063938276 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080cdd00 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 -- 172.21.15.29:0/1063938276 >> 172.21.15.29:0/1063938276 conn(0x7f59080cf570 msgr2=0x7f59080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:11.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 -- 172.21.15.29:0/1063938276 shutdown_connections 2023-10-14T16:59:11.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.904+0000 7f590c947640 1 -- 172.21.15.29:0/1063938276 wait complete. 2023-10-14T16:59:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.905+0000 7f590c947640 1 Processor -- start 2023-10-14T16:59:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.906+0000 7f590c947640 1 -- start start 2023-10-14T16:59:11.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.906+0000 7f590c947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080b3c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f590c947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080b41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f590c947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f590c947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59080c7700 con 0x7f590815cd70 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f590c947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f590815f450 con 0x7f59080d0be0 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f590c947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f590815f730 con 0x7f59080b8480 2023-10-14T16:59:11.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41828/0 (socket says 172.21.15.29:41828) 2023-10-14T16:59:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 -- 172.21.15.29:0/2789530275 learned_addr learned my addr 172.21.15.29:0/2789530275 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f59077fe640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080b3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5906ffd640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080b41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 -- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 msgr2=0x7f59080b3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080b3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 -- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 msgr2=0x7f59080b41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.907+0000 7f5907fff640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080b41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f5907fff640 1 -- 172.21.15.29:0/2789530275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900009400 con 0x7f590815cd70 2023-10-14T16:59:11.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f5907fff640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f590002b8f0 tx=0x7f59000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f590003ace0 con 0x7f590815cd70 2023-10-14T16:59:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f590003ae60 con 0x7f590815cd70 2023-10-14T16:59:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59000443f0 con 0x7f590815cd70 2023-10-14T16:59:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.908+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f590815fa10 con 0x7f590815cd70 2023-10-14T16:59:11.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.909+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f590815ffd0 con 0x7f590815cd70 2023-10-14T16:59:11.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.909+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59081601c0 con 0x7f590815cd70 2023-10-14T16:59:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.911+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 56) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f5900044550 con 0x7f590815cd70 2023-10-14T16:59:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.911+0000 7f58effff640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 0x7f58f406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.912+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f59000abd20 con 0x7f590815cd70 2023-10-14T16:59:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.912+0000 7f59077fe640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 0x7f58f406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:11.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.913+0000 7f59077fe640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 0x7f58f406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005880 tx=0x7f58f8009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:11.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:11.916+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f59000769d0 con 0x7f590815cd70 2023-10-14T16:59:12.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.151+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f590804e9b0 con 0x7f590815cd70 2023-10-14T16:59:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.683+0000 7f58effff640 1 -- 172.21.15.29:0/2789530275 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7f5900079ff0 con 0x7f590815cd70 2023-10-14T16:59:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.686+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 msgr2=0x7f58f406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.686+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 0x7f58f406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005880 tx=0x7f58f8009040 comp rx=0 tx=0).stop 2023-10-14T16:59:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 msgr2=0x7f590815d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 crypto rx=0x7f590002b8f0 tx=0x7f59000046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 shutdown_connections 2023-10-14T16:59:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f58f406af00 0x7f58f406d3c0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f58f8005880 tx=0x7f58f8009040 comp rx=0 tx=0).stop 2023-10-14T16:59:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f590815cd70 0x7f590815d720 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f59080d0be0 0x7f59080b41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 --2- 172.21.15.29:0/2789530275 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59080b8480 0x7f59080b3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 >> 172.21.15.29:0/2789530275 conn(0x7f59080cf570 msgr2=0x7f59080b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 shutdown_connections 2023-10-14T16:59:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.687+0000 7f590c947640 1 -- 172.21.15.29:0/2789530275 wait complete. 2023-10-14T16:59:12.711 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T16:59:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.934+0000 7fe41142e640 1 Processor -- start 2023-10-14T16:59:12.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.934+0000 7fe41142e640 1 -- start start 2023-10-14T16:59:12.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe41142e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe40c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe41142e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe40c0ad970 0x7fe40c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe41142e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe40c06c960 con 0x7fe40c15ebc0 2023-10-14T16:59:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe41142e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe40c15b9d0 con 0x7fe40c0ad970 2023-10-14T16:59:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe41142e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c15bcb0 con 0x7fe40c06c040 2023-10-14T16:59:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe410c2d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe40c15ebc0 0x7fe40c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50248/0 (socket says 172.21.15.29:50248) 2023-10-14T16:59:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.936+0000 7fe410c2d640 1 -- 172.21.15.29:0/1531802001 learned_addr learned my addr 172.21.15.29:0/1531802001 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40bfff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe40c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe403fff640 1 --2- 172.21.15.29:0/1531802001 >> v2:172.21.15.186:3300/0 conn(0x7fe40c0ad970 0x7fe40c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1227595947 0 0) 0x7fe40c06c960 con 0x7fe40c15ebc0 2023-10-14T16:59:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3f8003520 con 0x7fe40c15ebc0 2023-10-14T16:59:12.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 596934790 0 0) 0x7fe3f8003520 con 0x7fe40c15ebc0 2023-10-14T16:59:12.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 msgr2=0x7fe40c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:12.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe40c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 >> v2:172.21.15.186:3300/0 conn(0x7fe40c0ad970 msgr2=0x7fe40c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:12.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 --2- 172.21.15.29:0/1531802001 >> v2:172.21.15.186:3300/0 conn(0x7fe40c0ad970 0x7fe40c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c15bf90 con 0x7fe40c15ebc0 2023-10-14T16:59:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe4040019f0 con 0x7fe40c15ebc0 2023-10-14T16:59:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.937+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 >> v1:172.21.15.29:6789/0 conn(0x7fe40c15ebc0 legacy=0x7fe40c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f8003970 0x7fe3f8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe3f80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f8004d00 0x7fe3f8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c15b9d0 con 0x7fe3f8003970 2023-10-14T16:59:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c15bcb0 con 0x7fe3f8004d00 2023-10-14T16:59:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f800b980 con 0x7fe40c06c040 2023-10-14T16:59:12.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f8003970 0x7fe3f8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe403fff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe3f80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe410c2d640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f8004d00 0x7fe3f8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 -- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 msgr2=0x7fe3f80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe40c06c040 0x7fe3f80047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 -- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f8004d00 msgr2=0x7fe3f8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f8004d00 0x7fe3f8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.938+0000 7fe40bfff640 1 -- 172.21.15.29:0/1531802001 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f8003520 con 0x7fe3f8003970 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.939+0000 7fe40bfff640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f8003970 0x7fe3f8003d90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009ac0 tx=0x7fe3f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448bb0cf4bdfece server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc02c6b0 con 0x7fe3f8003970 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3fc02c810 con 0x7fe3f8003970 2023-10-14T16:59:12.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe40b7fe640 1 -- 172.21.15.29:0/1531802001 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3fc02cba0 con 0x7fe3f8003970 2023-10-14T16:59:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe41142e640 1 -- 172.21.15.29:0/1531802001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f8003970 msgr2=0x7fe3f8003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe41142e640 1 --2- 172.21.15.29:0/1531802001 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f8003970 0x7fe3f8003d90 secure :-1 s=READY pgs=1339 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc009ac0 tx=0x7fe3f80057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe410c2d640 1 -- 172.21.15.29:0/1531802001 reap_dead start 2023-10-14T16:59:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe41142e640 1 -- 172.21.15.29:0/1531802001 shutdown_connections 2023-10-14T16:59:12.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.940+0000 7fe41142e640 1 -- 172.21.15.29:0/1531802001 >> 172.21.15.29:0/1531802001 conn(0x7fe40c0ce6a0 msgr2=0x7fe40c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.941+0000 7fe41142e640 1 -- 172.21.15.29:0/1531802001 shutdown_connections 2023-10-14T16:59:12.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.941+0000 7fe41142e640 1 -- 172.21.15.29:0/1531802001 wait complete. 2023-10-14T16:59:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.942+0000 7fe41142e640 1 Processor -- start 2023-10-14T16:59:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.942+0000 7fe41142e640 1 -- start start 2023-10-14T16:59:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f8003970 0x7fe40c0ace00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe40c15aa70 0x7fe40c0a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0ac970 con 0x7fe40c0ab810 2023-10-14T16:59:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0abd60 con 0x7fe40c15aa70 2023-10-14T16:59:12.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.943+0000 7fe41142e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c0ac040 con 0x7fe3f8003970 2023-10-14T16:59:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41846/0 (socket says 172.21.15.29:41846) 2023-10-14T16:59:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 -- 172.21.15.29:0/2561825741 learned_addr learned my addr 172.21.15.29:0/2561825741 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe40bfff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f8003970 0x7fe40c0ace00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe410c2d640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe40c15aa70 0x7fe40c0a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 -- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f8003970 msgr2=0x7fe40c0ace00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f8003970 0x7fe40c0ace00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 -- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe40c15aa70 msgr2=0x7fe40c0a8be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe40c15aa70 0x7fe40c0a8be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.944+0000 7fe403fff640 1 -- 172.21.15.29:0/2561825741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe40c0ac320 con 0x7fe40c0ab810 2023-10-14T16:59:12.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe403fff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40108e0 tx=0x7fe3f4010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f400eb60 con 0x7fe40c0ab810 2023-10-14T16:59:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f400ecc0 con 0x7fe40c0ab810 2023-10-14T16:59:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f401c400 con 0x7fe40c0ab810 2023-10-14T16:59:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c0ac600 con 0x7fe40c0ab810 2023-10-14T16:59:12.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.945+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c15aef0 con 0x7fe40c0ab810 2023-10-14T16:59:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.946+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c06b330 con 0x7fe40c0ab810 2023-10-14T16:59:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.947+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3f401c560 con 0x7fe40c0ab810 2023-10-14T16:59:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.948+0000 7fe4097fa640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 0x7fe3ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:12.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.949+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fe3f408a120 con 0x7fe40c0ab810 2023-10-14T16:59:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.949+0000 7fe40bfff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 0x7fe3ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.950+0000 7fe40bfff640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 0x7fe3ec06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc036000 tx=0x7fe3fc030000 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:12.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:12.953+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3f4054db0 con 0x7fe40c0ab810 2023-10-14T16:59:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.150+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe40c04e9b0 con 0x7fe40c0ab810 2023-10-14T16:59:13.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.686+0000 7fe4097fa640 1 -- 172.21.15.29:0/2561825741 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v115) v1 ==== 181+0+0 (secure 0 0 0) 0x7fe3f40583d0 con 0x7fe40c0ab810 2023-10-14T16:59:13.688 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T16:59:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 msgr2=0x7fe3ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 0x7fe3ec06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc036000 tx=0x7fe3fc030000 comp rx=0 tx=0).stop 2023-10-14T16:59:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 msgr2=0x7fe40c158f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:13.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 crypto rx=0x7fe3f40108e0 tx=0x7fe3f4010da0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 shutdown_connections 2023-10-14T16:59:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3ec06af90 0x7fe3ec06d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe40c15aa70 0x7fe40c0a8be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe40c0ab810 0x7fe40c158f00 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 --2- 172.21.15.29:0/2561825741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f8003970 0x7fe40c0ace00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.690+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 >> 172.21.15.29:0/2561825741 conn(0x7fe40c0ce6a0 msgr2=0x7fe40c15dd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:13.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.691+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 shutdown_connections 2023-10-14T16:59:13.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.691+0000 7fe41142e640 1 -- 172.21.15.29:0/2561825741 wait complete. 2023-10-14T16:59:13.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T16:59:13.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.948+0000 7f2e14a7e640 1 Processor -- start 2023-10-14T16:59:13.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.948+0000 7f2e14a7e640 1 -- start start 2023-10-14T16:59:13.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e100c9060 0x7f2e1006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e1006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2e100c9440 0x7f2e10164d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e100ca560 con 0x7f2e100c9060 2023-10-14T16:59:13.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e100c9950 con 0x7f2e100c9440 2023-10-14T16:59:13.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e14a7e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c9c30 con 0x7f2e1006c9e0 2023-10-14T16:59:13.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e0f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e100c9060 0x7f2e1006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e0f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2e100c9060 0x7f2e1006f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41862/0 (socket says 172.21.15.29:41862) 2023-10-14T16:59:13.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.950+0000 7f2e0f7fe640 1 -- 172.21.15.29:0/1513856989 learned_addr learned my addr 172.21.15.29:0/1513856989 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:13.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0effd640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e1006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0ffff640 1 --2- 172.21.15.29:0/1513856989 >> v2:172.21.15.186:3300/0 conn(0x7f2e100c9440 0x7f2e10164d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0f7fe640 1 -- 172.21.15.29:0/1513856989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 msgr2=0x7f2e1006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e1006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0f7fe640 1 -- 172.21.15.29:0/1513856989 >> v2:172.21.15.186:3300/0 conn(0x7f2e100c9440 msgr2=0x7f2e10164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/1513856989 >> v2:172.21.15.186:3300/0 conn(0x7f2e100c9440 0x7f2e10164d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.951+0000 7f2e0f7fe640 1 -- 172.21.15.29:0/1513856989 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e100c9f10 con 0x7f2e100c9060 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.952+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9060 0x7f2e1006f440 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2e100ca4d0 tx=0x7f2e0002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8939411076da9892 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e0e7fc640 1 -- 172.21.15.29:0/1513856989 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00031870 con 0x7f2e100c9060 2023-10-14T16:59:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e0e7fc640 1 -- 172.21.15.29:0/1513856989 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e000319d0 con 0x7f2e100c9060 2023-10-14T16:59:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e0e7fc640 1 -- 172.21.15.29:0/1513856989 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e00031d60 con 0x7f2e100c9060 2023-10-14T16:59:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 -- 172.21.15.29:0/1513856989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9060 msgr2=0x7f2e1006f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9060 0x7f2e1006f440 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 crypto rx=0x7f2e100ca4d0 tx=0x7f2e0002b910 comp rx=0 tx=0).stop 2023-10-14T16:59:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 -- 172.21.15.29:0/1513856989 shutdown_connections 2023-10-14T16:59:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/1513856989 >> v2:172.21.15.186:3300/0 conn(0x7f2e100c9440 0x7f2e10164d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e1006f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/1513856989 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9060 0x7f2e1006f440 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.953+0000 7f2e14a7e640 1 -- 172.21.15.29:0/1513856989 >> 172.21.15.29:0/1513856989 conn(0x7f2e100ce6a0 msgr2=0x7f2e100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:13.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.954+0000 7f2e14a7e640 1 -- 172.21.15.29:0/1513856989 shutdown_connections 2023-10-14T16:59:13.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.954+0000 7f2e14a7e640 1 -- 172.21.15.29:0/1513856989 wait complete. 2023-10-14T16:59:13.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.955+0000 7f2e14a7e640 1 Processor -- start 2023-10-14T16:59:13.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.956+0000 7f2e14a7e640 1 -- start start 2023-10-14T16:59:13.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e100c46d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e100c9060 0x7f2e10161510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c3630 con 0x7f2e100c9440 2023-10-14T16:59:13.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c3910 con 0x7f2e100c9060 2023-10-14T16:59:13.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e14a7e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e100c3bf0 con 0x7f2e1006c9e0 2023-10-14T16:59:13.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41878/0 (socket says 172.21.15.29:41878) 2023-10-14T16:59:13.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e0ffff640 1 -- 172.21.15.29:0/756071800 learned_addr learned my addr 172.21.15.29:0/756071800 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:13.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e100c46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.957+0000 7f2e0effd640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e100c9060 0x7f2e10161510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 -- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 msgr2=0x7f2e100c46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:13.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e100c46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 -- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e100c9060 msgr2=0x7f2e10161510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:13.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e100c9060 0x7f2e10161510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:13.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 -- 172.21.15.29:0/756071800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e000094d0 con 0x7f2e100c9440 2023-10-14T16:59:13.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e0ffff640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800cc00 tx=0x7f2e0800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:13.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e0800ce10 con 0x7f2e100c9440 2023-10-14T16:59:13.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e08003990 con 0x7f2e100c9440 2023-10-14T16:59:13.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e08003d20 con 0x7f2e100c9440 2023-10-14T16:59:13.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.958+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e100c3ed0 con 0x7f2e100c9440 2023-10-14T16:59:13.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.959+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e100c30e0 con 0x7f2e100c9440 2023-10-14T16:59:13.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.960+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e100c43f0 con 0x7f2e100c9440 2023-10-14T16:59:13.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.961+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2e08003e80 con 0x7f2e100c9440 2023-10-14T16:59:13.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.962+0000 7f2df7fff640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 0x7f2dfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:13.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.962+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6204+0+0 (secure 0 0 0) 0x7f2e08089eb0 con 0x7f2e100c9440 2023-10-14T16:59:13.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.962+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 0x7f2dfc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:13.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.963+0000 7f2e0f7fe640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 0x7f2dfc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e00036000 tx=0x7f2e0003c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:13.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:13.966+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2e100c43f0 con 0x7f2e100c9440 2023-10-14T16:59:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.168+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f2e100c43f0 con 0x7f2e100c9440 2023-10-14T16:59:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.697+0000 7f2df7fff640 1 -- 172.21.15.29:0/756071800 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v116) v1 ==== 169+0+0 (secure 0 0 0) 0x7f2e100c43f0 con 0x7f2e100c9440 2023-10-14T16:59:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T16:59:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 msgr2=0x7f2dfc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 0x7f2dfc06d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f2e00036000 tx=0x7f2e0003c040 comp rx=0 tx=0).stop 2023-10-14T16:59:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 msgr2=0x7f2e10161a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800cc00 tx=0x7f2e0800a5c0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 shutdown_connections 2023-10-14T16:59:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2dfc06af90 0x7f2dfc06d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2e100c9440 0x7f2e10161a50 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2e100c9060 0x7f2e10161510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 --2- 172.21.15.29:0/756071800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2e1006c9e0 0x7f2e100c46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.701+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 >> 172.21.15.29:0/756071800 conn(0x7f2e100ce6a0 msgr2=0x7f2e1006af30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.702+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 shutdown_connections 2023-10-14T16:59:14.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.702+0000 7f2e14a7e640 1 -- 172.21.15.29:0/756071800 wait complete. 2023-10-14T16:59:14.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T16:59:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.949+0000 7f2a8cc75640 1 Processor -- start 2023-10-14T16:59:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.949+0000 7f2a8cc75640 1 -- start start 2023-10-14T16:59:14.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a8cc75640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2a880b0ea0 0x7f2a8815a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a8cc75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a8806be30 con 0x7f2a880b3ec0 2023-10-14T16:59:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a8cc75640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a880ada10 con 0x7f2a880cd970 2023-10-14T16:59:14.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a8cc75640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a8806ca00 con 0x7f2a880b0ea0 2023-10-14T16:59:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a877fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a880b3ec0 0x7f2a880cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50262/0 (socket says 172.21.15.29:50262) 2023-10-14T16:59:14.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.951+0000 7f2a877fe640 1 -- 172.21.15.29:0/3311801790 learned_addr learned my addr 172.21.15.29:0/3311801790 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3311801790 >> v2:172.21.15.186:3300/0 conn(0x7f2a880b0ea0 0x7f2a8815a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4090681330 0 0) 0x7f2a8806be30 con 0x7f2a880b3ec0 2023-10-14T16:59:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a74003520 con 0x7f2a880b3ec0 2023-10-14T16:59:14.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331082976 0 0) 0x7f2a74003520 con 0x7f2a880b3ec0 2023-10-14T16:59:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 >> v2:172.21.15.186:3300/0 conn(0x7f2a880b0ea0 msgr2=0x7f2a8815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 --2- 172.21.15.29:0/3311801790 >> v2:172.21.15.186:3300/0 conn(0x7f2a880b0ea0 0x7f2a8815a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 >> v1:172.21.15.186:6789/0 conn(0x7f2a880cd970 legacy=0x7f2a8815a830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a8806c060 con 0x7f2a880b3ec0 2023-10-14T16:59:14.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.952+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2a78001670 con 0x7f2a880b3ec0 2023-10-14T16:59:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 >> v1:172.21.15.29:6789/0 conn(0x7f2a880b3ec0 legacy=0x7f2a880cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a74003970 0x7f2a74003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a74005180 0x7f2a7400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a8806ca00 con 0x7f2a740047c0 2023-10-14T16:59:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c120 con 0x7f2a74003970 2023-10-14T16:59:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7400c2e0 con 0x7f2a74005180 2023-10-14T16:59:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 -- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a74005180 msgr2=0x7f2a7400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a74005180 0x7f2a7400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 -- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a74003970 msgr2=0x7f2a74003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a74003970 0x7f2a74003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a7ffff640 1 -- 172.21.15.29:0/3311801790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a74003520 con 0x7f2a740047c0 2023-10-14T16:59:14.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.953+0000 7f2a87fff640 1 -- 172.21.15.29:0/3311801790 reap_dead start 2023-10-14T16:59:14.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a74004c40 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009ac0 tx=0x7f2a740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef62f8fd58097977 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7002c6b0 con 0x7f2a740047c0 2023-10-14T16:59:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a7002c810 con 0x7f2a740047c0 2023-10-14T16:59:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a86ffd640 1 -- 172.21.15.29:0/3311801790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a7002cba0 con 0x7f2a740047c0 2023-10-14T16:59:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 msgr2=0x7f2a74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.955+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a74004c40 secure :-1 s=READY pgs=1344 cs=0 l=1 rev1=1 crypto rx=0x7f2a70009ac0 tx=0x7f2a740057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.956+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3311801790 shutdown_connections 2023-10-14T16:59:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.956+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3311801790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a74004c40 unknown :-1 s=CLOSED pgs=1344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.956+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3311801790 >> 172.21.15.29:0/3311801790 conn(0x7f2a880b0170 msgr2=0x7f2a880b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.956+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3311801790 shutdown_connections 2023-10-14T16:59:14.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.956+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3311801790 wait complete. 2023-10-14T16:59:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.957+0000 7f2a8cc75640 1 Processor -- start 2023-10-14T16:59:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.958+0000 7f2a8cc75640 1 -- start start 2023-10-14T16:59:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 0x7f2a881f2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 0x7f2a880c67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a880af5a0 con 0x7f2a740047c0 2023-10-14T16:59:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a881516f0 con 0x7f2a880af120 2023-10-14T16:59:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a8cc75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a88151970 con 0x7f2a881f2da0 2023-10-14T16:59:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41898/0 (socket says 172.21.15.29:41898) 2023-10-14T16:59:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 -- 172.21.15.29:0/3294659092 learned_addr learned my addr 172.21.15.29:0/3294659092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 0x7f2a881f2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a87fff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 0x7f2a880c67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 -- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 msgr2=0x7f2a880c67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 0x7f2a880c67a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 -- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 msgr2=0x7f2a881f2860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 0x7f2a881f2860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a877fe640 1 -- 172.21.15.29:0/3294659092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a88151c50 con 0x7f2a740047c0 2023-10-14T16:59:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 0x7f2a881f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:14.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.959+0000 7f2a87fff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 0x7f2a880c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a877fe640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f2a78000f60 tx=0x7f2a7800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a780074e0 con 0x7f2a740047c0 2023-10-14T16:59:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a78007660 con 0x7f2a740047c0 2023-10-14T16:59:14.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a78004a20 con 0x7f2a740047c0 2023-10-14T16:59:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a88151f30 con 0x7f2a740047c0 2023-10-14T16:59:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.960+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a8814f320 con 0x7f2a740047c0 2023-10-14T16:59:14.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.962+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a8806b410 con 0x7f2a740047c0 2023-10-14T16:59:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.962+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2a78004b80 con 0x7f2a740047c0 2023-10-14T16:59:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.963+0000 7f2a84ff9640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 0x7f2a6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:14.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.963+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f2a78088e50 con 0x7f2a740047c0 2023-10-14T16:59:14.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.964+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 0x7f2a6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:14.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.964+0000 7f2a7ffff640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 0x7f2a6806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a70036000 tx=0x7f2a70030000 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:14.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:14.968+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a78053d10 con 0x7f2a740047c0 2023-10-14T16:59:15.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.164+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f2a8804e9b0 con 0x7f2a740047c0 2023-10-14T16:59:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.171+0000 7f2a84ff9640 1 -- 172.21.15.29:0/3294659092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v37) v1 ==== 115+0+0 (secure 0 0 0) 0x7f2a78018070 con 0x7f2a740047c0 2023-10-14T16:59:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 msgr2=0x7f2a6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 0x7f2a6806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f2a70036000 tx=0x7f2a70030000 comp rx=0 tx=0).stop 2023-10-14T16:59:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 msgr2=0x7f2a880aebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 crypto rx=0x7f2a78000f60 tx=0x7f2a7800cc20 comp rx=0 tx=0).stop 2023-10-14T16:59:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 shutdown_connections 2023-10-14T16:59:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f2a6806af90 0x7f2a6806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a881f2da0 0x7f2a880c67a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a880af120 0x7f2a881f2860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 --2- 172.21.15.29:0/3294659092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a740047c0 0x7f2a880aebe0 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.174+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 >> 172.21.15.29:0/3294659092 conn(0x7f2a880b0170 msgr2=0x7f2a880705a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.175+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 shutdown_connections 2023-10-14T16:59:15.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.175+0000 7f2a8cc75640 1 -- 172.21.15.29:0/3294659092 wait complete. 2023-10-14T16:59:15.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T16:59:15.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.426+0000 7fea8bfff640 1 Processor -- start 2023-10-14T16:59:15.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.426+0000 7fea8bfff640 1 -- start start 2023-10-14T16:59:15.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea8c0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c15b9d0 con 0x7fea8c0ad970 2023-10-14T16:59:15.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea8c15bcb0 con 0x7fea8c06c040 2023-10-14T16:59:15.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8c15b7d0 con 0x7fea8c15b320 2023-10-14T16:59:15.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8a7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea8c0ad970 0x7fea8c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50276/0 (socket says 172.21.15.29:50276) 2023-10-14T16:59:15.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.428+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3744635453 learned_addr learned my addr 172.21.15.29:0/3744635453 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:15.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea8b7fe640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea8c0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096878225 0 0) 0x7fea8c15b9d0 con 0x7fea8c0ad970 2023-10-14T16:59:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea74003520 con 0x7fea8c0ad970 2023-10-14T16:59:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 20629568 0 0) 0x7fea74003520 con 0x7fea8c0ad970 2023-10-14T16:59:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 msgr2=0x7fea8c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea8c0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 >> v1:172.21.15.186:6789/0 conn(0x7fea8c06c040 legacy=0x7fea8c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c15bf90 con 0x7fea8c0ad970 2023-10-14T16:59:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fea80001670 con 0x7fea8c0ad970 2023-10-14T16:59:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 >> v1:172.21.15.29:6789/0 conn(0x7fea8c0ad970 legacy=0x7fea8c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea740039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea74003f70 0x7fea740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea74004d00 0x7fea74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8c15b7d0 con 0x7fea74003f70 2023-10-14T16:59:15.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7400bac0 con 0x7fea74004d00 2023-10-14T16:59:15.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.429+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7400bc80 con 0x7fea8c15b320 2023-10-14T16:59:15.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea74003f70 0x7fea740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8affd640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea740039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8b7fe640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea74004d00 0x7fea74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 msgr2=0x7fea740039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea8c15b320 0x7fea740039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea74004d00 msgr2=0x7fea74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea74004d00 0x7fea74005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.430+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3744635453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea74003520 con 0x7fea74003f70 2023-10-14T16:59:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.431+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea74003f70 0x7fea740047c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0ade20 tx=0x7fea740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65c7d861af802154 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80005ad0 con 0x7fea74003f70 2023-10-14T16:59:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80005c50 con 0x7fea74003f70 2023-10-14T16:59:15.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea89ffb640 1 -- 172.21.15.29:0/3744635453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea80007070 con 0x7fea74003f70 2023-10-14T16:59:15.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 -- 172.21.15.29:0/3744635453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea74003f70 msgr2=0x7fea740047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3744635453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea74003f70 0x7fea740047c0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 crypto rx=0x7fea8c0ade20 tx=0x7fea740057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8b7fe640 1 -- 172.21.15.29:0/3744635453 reap_dead start 2023-10-14T16:59:15.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 -- 172.21.15.29:0/3744635453 shutdown_connections 2023-10-14T16:59:15.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 -- 172.21.15.29:0/3744635453 >> 172.21.15.29:0/3744635453 conn(0x7fea8c0ce6a0 msgr2=0x7fea8c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 -- 172.21.15.29:0/3744635453 shutdown_connections 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.432+0000 7fea8bfff640 1 -- 172.21.15.29:0/3744635453 wait complete. 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.435+0000 7fea8bfff640 1 Processor -- start 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.435+0000 7fea8bfff640 1 -- start start 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea74003f70 0x7fea8c159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea8c0b12c0 0x7fea8c0c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8c0b1880 con 0x7fea8c159960 2023-10-14T16:59:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea8c0b1b60 con 0x7fea8c0b12c0 2023-10-14T16:59:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.436+0000 7fea8bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea8c0b1e40 con 0x7fea74003f70 2023-10-14T16:59:15.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41926/0 (socket says 172.21.15.29:41926) 2023-10-14T16:59:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3047211170 learned_addr learned my addr 172.21.15.29:0/3047211170 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8affd640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea74003f70 0x7fea8c159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8b7fe640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea8c0b12c0 0x7fea8c0c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea74003f70 msgr2=0x7fea8c159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea74003f70 0x7fea8c159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea8c0b12c0 msgr2=0x7fea8c0c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea8c0b12c0 0x7fea8c0c43a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 -- 172.21.15.29:0/3047211170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c0b2120 con 0x7fea8c159960 2023-10-14T16:59:15.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.437+0000 7fea8a7fc640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c110 tx=0x7fea80001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.438+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8003c040 con 0x7fea8c159960 2023-10-14T16:59:15.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.438+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea80007480 con 0x7fea8c159960 2023-10-14T16:59:15.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.438+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea8002c700 con 0x7fea8c159960 2023-10-14T16:59:15.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.438+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea8c0b2400 con 0x7fea8c159960 2023-10-14T16:59:15.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.438+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea8c159670 con 0x7fea8c159960 2023-10-14T16:59:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.439+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea8c06b330 con 0x7fea8c159960 2023-10-14T16:59:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.440+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fea80032070 con 0x7fea8c159960 2023-10-14T16:59:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.441+0000 7fea737fe640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 0x7fea7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.441+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fea800aaba0 con 0x7fea8c159960 2023-10-14T16:59:15.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.441+0000 7fea8affd640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 0x7fea7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.442+0000 7fea8affd640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 0x7fea7806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fea7c007fd0 tx=0x7fea7c009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.446+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea80075b90 con 0x7fea8c159960 2023-10-14T16:59:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.650+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fea8c04e9b0 con 0x7fea8c159960 2023-10-14T16:59:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.657+0000 7fea737fe640 1 -- 172.21.15.29:0/3047211170 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v38) v1 ==== 120+0+0 (secure 0 0 0) 0x7fea80046020 con 0x7fea8c159960 2023-10-14T16:59:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 msgr2=0x7fea7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 0x7fea7806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fea7c007fd0 tx=0x7fea7c009040 comp rx=0 tx=0).stop 2023-10-14T16:59:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 msgr2=0x7fea8c1e6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 secure :-1 s=READY pgs=1348 cs=0 l=1 rev1=1 crypto rx=0x7fea8000c110 tx=0x7fea80001930 comp rx=0 tx=0).stop 2023-10-14T16:59:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 shutdown_connections 2023-10-14T16:59:15.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fea7806af90 0x7fea7806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea8c0b12c0 0x7fea8c0c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea8c159960 0x7fea8c1e6710 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 --2- 172.21.15.29:0/3047211170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea74003f70 0x7fea8c159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 >> 172.21.15.29:0/3047211170 conn(0x7fea8c0ce6a0 msgr2=0x7fea8c0ca700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 shutdown_connections 2023-10-14T16:59:15.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.661+0000 7fea8bfff640 1 -- 172.21.15.29:0/3047211170 wait complete. 2023-10-14T16:59:15.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T16:59:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.912+0000 7f3d40c90640 1 Processor -- start 2023-10-14T16:59:15.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.912+0000 7f3d40c90640 1 -- start start 2023-10-14T16:59:15.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.913+0000 7f3d40c90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3d3c06c9e0 0x7f3d3c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.913+0000 7f3d40c90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0ca560 con 0x7f3d3c0c9060 2023-10-14T16:59:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.913+0000 7f3d40c90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d3c0c9950 con 0x7f3d3c06fb50 2023-10-14T16:59:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.913+0000 7f3d40c90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c9c30 con 0x7f3d3c06c9e0 2023-10-14T16:59:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.914+0000 7f3d3b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3d3c0c9060 0x7f3d3c06f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50284/0 (socket says 172.21.15.29:50284) 2023-10-14T16:59:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.914+0000 7f3d3b7fe640 1 -- 172.21.15.29:0/556795169 learned_addr learned my addr 172.21.15.29:0/556795169 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.914+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255128185 0 0) 0x7f3d3c0ca560 con 0x7f3d3c0c9060 2023-10-14T16:59:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.914+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d28003520 con 0x7f3d3c0c9060 2023-10-14T16:59:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.914+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/556795169 >> v2:172.21.15.186:3300/0 conn(0x7f3d3c06c9e0 0x7f3d3c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873728245 0 0) 0x7f3d28003520 con 0x7f3d3c0c9060 2023-10-14T16:59:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 >> v2:172.21.15.186:3300/0 conn(0x7f3d3c06c9e0 msgr2=0x7f3d3c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 --2- 172.21.15.29:0/556795169 >> v2:172.21.15.186:3300/0 conn(0x7f3d3c06c9e0 0x7f3d3c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 >> v1:172.21.15.186:6789/0 conn(0x7f3d3c06fb50 legacy=0x7f3d3c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c0c9f10 con 0x7f3d3c0c9060 2023-10-14T16:59:15.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3d2c001670 con 0x7f3d3c0c9060 2023-10-14T16:59:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 >> v1:172.21.15.29:6789/0 conn(0x7f3d3c0c9060 legacy=0x7f3d3c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280039c0 0x7f3d28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 0x7f3d2800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c0c9c30 con 0x7f3d280051d0 2023-10-14T16:59:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2800c100 con 0x7f3d280047c0 2023-10-14T16:59:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d2800c2c0 con 0x7f3d280039c0 2023-10-14T16:59:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.915+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 0x7f3d2800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3b7fe640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280039c0 0x7f3d28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3affd640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3bfff640 1 -- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280039c0 msgr2=0x7f3d28003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280039c0 0x7f3d28003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3bfff640 1 -- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d280047c0 msgr2=0x7f3d28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d280047c0 0x7f3d28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.916+0000 7f3d3bfff640 1 -- 172.21.15.29:0/556795169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28003520 con 0x7f3d280051d0 2023-10-14T16:59:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.917+0000 7f3d3bfff640 1 -- 172.21.15.29:0/556795169 reap_dead start 2023-10-14T16:59:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.917+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 0x7f3d2800b980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c06cde0 tx=0x7f3d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a689d54bac000257 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.917+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34004020 con 0x7f3d280051d0 2023-10-14T16:59:15.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.917+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d34004180 con 0x7f3d280051d0 2023-10-14T16:59:15.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.917+0000 7f3d3a7fc640 1 -- 172.21.15.29:0/556795169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d34003610 con 0x7f3d280051d0 2023-10-14T16:59:15.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 -- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 msgr2=0x7f3d2800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 0x7f3d2800b980 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c06cde0 tx=0x7f3d280057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 -- 172.21.15.29:0/556795169 shutdown_connections 2023-10-14T16:59:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 --2- 172.21.15.29:0/556795169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d280051d0 0x7f3d2800b980 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 -- 172.21.15.29:0/556795169 >> 172.21.15.29:0/556795169 conn(0x7f3d3c0ce6a0 msgr2=0x7f3d3c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:15.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 -- 172.21.15.29:0/556795169 shutdown_connections 2023-10-14T16:59:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.918+0000 7f3d40c90640 1 -- 172.21.15.29:0/556795169 wait complete. 2023-10-14T16:59:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.920+0000 7f3d40c90640 1 Processor -- start 2023-10-14T16:59:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.920+0000 7f3d40c90640 1 -- start start 2023-10-14T16:59:15.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280051d0 0x7f3d3c15d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d3c1622e0 0x7f3d3c162790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c164130 con 0x7f3d3c163770 2023-10-14T16:59:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c163050 con 0x7f3d3c1622e0 2023-10-14T16:59:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d40c90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3c157510 con 0x7f3d280051d0 2023-10-14T16:59:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d3affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41950/0 (socket says 172.21.15.29:41950) 2023-10-14T16:59:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.921+0000 7f3d3affd640 1 -- 172.21.15.29:0/1146392405 learned_addr learned my addr 172.21.15.29:0/1146392405 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3b7fe640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280051d0 0x7f3d3c15d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3bfff640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d3c1622e0 0x7f3d3c162790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 -- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280051d0 msgr2=0x7f3d3c15d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280051d0 0x7f3d3c15d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 -- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d3c1622e0 msgr2=0x7f3d3c162790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d3c1622e0 0x7f3d3c162790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 -- 172.21.15.29:0/1146392405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d3c1577f0 con 0x7f3d3c163770 2023-10-14T16:59:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d3affd640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3d240059a0 tx=0x7f3d2400ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d24003df0 con 0x7f3d3c163770 2023-10-14T16:59:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d2400c6c0 con 0x7f3d3c163770 2023-10-14T16:59:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.922+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d3c157ad0 con 0x7f3d3c163770 2023-10-14T16:59:15.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.923+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d2400ca70 con 0x7f3d3c163770 2023-10-14T16:59:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.923+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d3c162cd0 con 0x7f3d3c163770 2023-10-14T16:59:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.924+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3c0668c0 con 0x7f3d3c163770 2023-10-14T16:59:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.925+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3d2400cbd0 con 0x7f3d3c163770 2023-10-14T16:59:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.926+0000 7f3d33fff640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 0x7f3d1c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.926+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f3d24089bd0 con 0x7f3d3c163770 2023-10-14T16:59:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.926+0000 7f3d3b7fe640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 0x7f3d1c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.927+0000 7f3d3b7fe640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 0x7f3d1c06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0cf5a0 tx=0x7f3d2c009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:15.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:15.930+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3d24054b30 con 0x7f3d3c163770 2023-10-14T16:59:16.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.127+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f3d3c04e9b0 con 0x7f3d3c163770 2023-10-14T16:59:16.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.134+0000 7f3d33fff640 1 -- 172.21.15.29:0/1146392405 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v39) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3d24058150 con 0x7f3d3c163770 2023-10-14T16:59:16.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.136+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 msgr2=0x7f3d1c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.136+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 0x7f3d1c06d480 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3d3c0cf5a0 tx=0x7f3d2c009040 comp rx=0 tx=0).stop 2023-10-14T16:59:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 msgr2=0x7f3d3c163bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 secure :-1 s=READY pgs=1351 cs=0 l=1 rev1=1 crypto rx=0x7f3d240059a0 tx=0x7f3d2400ec60 comp rx=0 tx=0).stop 2023-10-14T16:59:16.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 shutdown_connections 2023-10-14T16:59:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f3d1c06afc0 0x7f3d1c06d480 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3d3c1622e0 0x7f3d3c162790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3d3c163770 0x7f3d3c163bf0 unknown :-1 s=CLOSED pgs=1351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 --2- 172.21.15.29:0/1146392405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3d280051d0 0x7f3d3c15d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 >> 172.21.15.29:0/1146392405 conn(0x7f3d3c0ce6a0 msgr2=0x7f3d3c0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 shutdown_connections 2023-10-14T16:59:16.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.137+0000 7f3d40c90640 1 -- 172.21.15.29:0/1146392405 wait complete. 2023-10-14T16:59:16.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path' 2023-10-14T16:59:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.378+0000 7f99ff478640 1 Processor -- start 2023-10-14T16:59:16.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.379+0000 7f99ff478640 1 -- start start 2023-10-14T16:59:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.380+0000 7f99ff478640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99f80b4c80 0x7f99f8167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.380+0000 7f99ff478640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80d15a0 con 0x7f99f815cfc0 2023-10-14T16:59:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.380+0000 7f99ff478640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99f80d1880 con 0x7f99f815cbe0 2023-10-14T16:59:16.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.380+0000 7f99ff478640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99f80d1b60 con 0x7f99f80b4c80 2023-10-14T16:59:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fdc75640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f99f815cfc0 0x7f99f80b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50300/0 (socket says 172.21.15.29:50300) 2023-10-14T16:59:16.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fdc75640 1 -- 172.21.15.29:0/623185037 learned_addr learned my addr 172.21.15.29:0/623185037 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fec77640 1 --2- 172.21.15.29:0/623185037 >> v2:172.21.15.29:3300/0 conn(0x7f99f80b4c80 0x7f99f8167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fd474640 1 -- 172.21.15.29:0/623185037 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687363853 0 0) 0x7f99f80d15a0 con 0x7f99f815cfc0 2023-10-14T16:59:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fd474640 1 -- 172.21.15.29:0/623185037 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99e4003520 con 0x7f99f815cfc0 2023-10-14T16:59:16.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fec77640 1 -- 172.21.15.29:0/623185037 >> v1:172.21.15.186:6789/0 conn(0x7f99f815cbe0 legacy=0x7f99f80d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fec77640 1 -- 172.21.15.29:0/623185037 >> v1:172.21.15.29:6789/0 conn(0x7f99f815cfc0 legacy=0x7f99f80b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.381+0000 7f99fec77640 1 -- 172.21.15.29:0/623185037 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f80d1e40 con 0x7f99f80b4c80 2023-10-14T16:59:16.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.382+0000 7f99fec77640 1 --2- 172.21.15.29:0/623185037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80b4c80 0x7f99f8167910 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009d90 tx=0x7f99f002bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374bc856c36d2a2b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99fd474640 1 -- 172.21.15.29:0/623185037 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f002ccf0 con 0x7f99f80b4c80 2023-10-14T16:59:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99fd474640 1 -- 172.21.15.29:0/623185037 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f002ce50 con 0x7f99f80b4c80 2023-10-14T16:59:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99fd474640 1 -- 172.21.15.29:0/623185037 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f0034640 con 0x7f99f80b4c80 2023-10-14T16:59:16.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99ff478640 1 -- 172.21.15.29:0/623185037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80b4c80 msgr2=0x7f99f8167910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99ff478640 1 --2- 172.21.15.29:0/623185037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80b4c80 0x7f99f8167910 secure :-1 s=READY pgs=1353 cs=0 l=1 rev1=1 crypto rx=0x7f99f0009d90 tx=0x7f99f002bf40 comp rx=0 tx=0).stop 2023-10-14T16:59:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99ff478640 1 -- 172.21.15.29:0/623185037 shutdown_connections 2023-10-14T16:59:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.383+0000 7f99ff478640 1 --2- 172.21.15.29:0/623185037 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f80b4c80 0x7f99f8167910 unknown :-1 s=CLOSED pgs=1353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.384+0000 7f99ff478640 1 -- 172.21.15.29:0/623185037 >> 172.21.15.29:0/623185037 conn(0x7f99f80cf570 msgr2=0x7f99f80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:16.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.384+0000 7f99ff478640 1 -- 172.21.15.29:0/623185037 shutdown_connections 2023-10-14T16:59:16.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.384+0000 7f99ff478640 1 -- 172.21.15.29:0/623185037 wait complete. 2023-10-14T16:59:16.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.385+0000 7f99ff478640 1 Processor -- start 2023-10-14T16:59:16.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.386+0000 7f99ff478640 1 -- start start 2023-10-14T16:59:16.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.386+0000 7f99ff478640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99f80b4c80 0x7f99f80ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.386+0000 7f99ff478640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 0x7f99f81593f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99ff478640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99ff478640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80cab10 con 0x7f99f815cfc0 2023-10-14T16:59:16.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99ff478640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f80cd170 con 0x7f99f815cbe0 2023-10-14T16:59:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99ff478640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f8159930 con 0x7f99f80b4c80 2023-10-14T16:59:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41968/0 (socket says 172.21.15.29:41968) 2023-10-14T16:59:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 -- 172.21.15.29:0/1818144515 learned_addr learned my addr 172.21.15.29:0/1818144515 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:16.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fdc75640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 0x7f99f81593f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fe476640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99f80b4c80 0x7f99f80ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 -- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99f80b4c80 msgr2=0x7f99f80ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99f80b4c80 0x7f99f80ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 -- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 msgr2=0x7f99f81593f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 0x7f99f81593f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fec77640 1 -- 172.21.15.29:0/1818144515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0009a70 con 0x7f99f815cfc0 2023-10-14T16:59:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.387+0000 7f99fdc75640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 0x7f99f81593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99fec77640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f99f002dfd0 tx=0x7f99f0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f0003b90 con 0x7f99f815cfc0 2023-10-14T16:59:16.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f0003d10 con 0x7f99f815cfc0 2023-10-14T16:59:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f0004270 con 0x7f99f815cfc0 2023-10-14T16:59:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99f80cad20 con 0x7f99f815cfc0 2023-10-14T16:59:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.388+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99f80cb2e0 con 0x7f99f815cfc0 2023-10-14T16:59:16.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.389+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99f804e9b0 con 0x7f99f815cfc0 2023-10-14T16:59:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.391+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f99f003b050 con 0x7f99f815cfc0 2023-10-14T16:59:16.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.392+0000 7f99eeffd640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.392+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f99f00abb20 con 0x7f99f815cfc0 2023-10-14T16:59:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.392+0000 7f99fe476640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.393+0000 7f99fe476640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 0x7f99dc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99f4007fd0 tx=0x7f99f4009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.397+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99f0076bd0 con 0x7f99f815cfc0 2023-10-14T16:59:16.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:59:16.454 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:59:16.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.667+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]} v 0) v1 -- 0x7f99f80cdc80 con 0x7f99f815cfc0 2023-10-14T16:59:16.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.711+0000 7f99eeffd640 1 -- 172.21.15.29:0/1818144515 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_schedule_auto_deactivation_for_non_existent_path"]}]=0 v716) v1 ==== 183+0+0 (secure 0 0 0) 0x7f99f00390f0 con 0x7f99f815cfc0 2023-10-14T16:59:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 msgr2=0x7f99dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 0x7f99dc06d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f99f4007fd0 tx=0x7f99f4009040 comp rx=0 tx=0).stop 2023-10-14T16:59:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 msgr2=0x7f99f80a9c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 crypto rx=0x7f99f002dfd0 tx=0x7f99f0004a40 comp rx=0 tx=0).stop 2023-10-14T16:59:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 shutdown_connections 2023-10-14T16:59:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f815cfc0 0x7f99f80a9c50 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f815cbe0 0x7f99f81593f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 --2- 172.21.15.29:0/1818144515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99f80b4c80 0x7f99f80ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 >> 172.21.15.29:0/1818144515 conn(0x7f99f80cf570 msgr2=0x7f99f80b7520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 shutdown_connections 2023-10-14T16:59:16.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.714+0000 7f99ff478640 1 -- 172.21.15.29:0/1818144515 wait complete. 2023-10-14T16:59:16.735 INFO:tasks.cephfs_test_runner:Test that a non-existent path leads to schedule deactivation after a few retries. ... ok 2023-10-14T16:59:16.735 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test existence of a scheduled snapshot 2023-10-14T16:59:16.736 INFO:tasks.cephfs_test_runner:test_snap_schedule (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T16:59:16.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.946+0000 7fa674809640 1 Processor -- start 2023-10-14T16:59:16.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.947+0000 7fa674809640 1 -- start start 2023-10-14T16:59:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa674809640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa670159b20 0x7fa67006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa674809640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6700b0b20 0x7fa67006fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa674809640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa67006f7f0 con 0x7fa670159f00 2023-10-14T16:59:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa674809640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67006c5e0 con 0x7fa670159b20 2023-10-14T16:59:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa674809640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa67006c8c0 con 0x7fa6700b0b20 2023-10-14T16:59:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa66e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa670159b20 0x7fa67006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa66e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa670159b20 0x7fa67006bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41980/0 (socket says 172.21.15.29:41980) 2023-10-14T16:59:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa66e59c640 1 -- 172.21.15.29:0/1442835268 learned_addr learned my addr 172.21.15.29:0/1442835268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.948+0000 7fa66ed9d640 1 --2- 172.21.15.29:0/1442835268 >> v2:172.21.15.186:3300/0 conn(0x7fa6700b0b20 0x7fa67006fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.949+0000 7fa66e59c640 1 -- 172.21.15.29:0/1442835268 >> v2:172.21.15.186:3300/0 conn(0x7fa6700b0b20 msgr2=0x7fa67006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.949+0000 7fa66e59c640 1 --2- 172.21.15.29:0/1442835268 >> v2:172.21.15.186:3300/0 conn(0x7fa6700b0b20 0x7fa67006fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.949+0000 7fa66e59c640 1 -- 172.21.15.29:0/1442835268 >> v1:172.21.15.186:6789/0 conn(0x7fa670159f00 legacy=0x7fa6700b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.949+0000 7fa66e59c640 1 -- 172.21.15.29:0/1442835268 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa67006cba0 con 0x7fa670159b20 2023-10-14T16:59:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.950+0000 7fa66e59c640 1 --2- 172.21.15.29:0/1442835268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa67006bbe0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa6700c8bb0 tx=0x7fa66002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d33efe84cc770a9f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.950+0000 7fa66dd9b640 1 -- 172.21.15.29:0/1442835268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660031860 con 0x7fa670159b20 2023-10-14T16:59:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.950+0000 7fa66dd9b640 1 -- 172.21.15.29:0/1442835268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6600319c0 con 0x7fa670159b20 2023-10-14T16:59:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.950+0000 7fa66dd9b640 1 -- 172.21.15.29:0/1442835268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa660031d50 con 0x7fa670159b20 2023-10-14T16:59:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 -- 172.21.15.29:0/1442835268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 msgr2=0x7fa67006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 --2- 172.21.15.29:0/1442835268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa67006bbe0 secure :-1 s=READY pgs=1355 cs=0 l=1 rev1=1 crypto rx=0x7fa6700c8bb0 tx=0x7fa66002b900 comp rx=0 tx=0).stop 2023-10-14T16:59:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 -- 172.21.15.29:0/1442835268 shutdown_connections 2023-10-14T16:59:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 --2- 172.21.15.29:0/1442835268 >> v2:172.21.15.186:3300/0 conn(0x7fa6700b0b20 0x7fa67006fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 --2- 172.21.15.29:0/1442835268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa67006bbe0 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 -- 172.21.15.29:0/1442835268 >> 172.21.15.29:0/1442835268 conn(0x7fa6700cb410 msgr2=0x7fa6700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 -- 172.21.15.29:0/1442835268 shutdown_connections 2023-10-14T16:59:16.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.951+0000 7fa674809640 1 -- 172.21.15.29:0/1442835268 wait complete. 2023-10-14T16:59:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.952+0000 7fa674809640 1 Processor -- start 2023-10-14T16:59:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.953+0000 7fa674809640 1 -- start start 2023-10-14T16:59:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6700b0b20 0x7fa6700b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa670159f00 0x7fa670158890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6701590e0 con 0x7fa670159b20 2023-10-14T16:59:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700b1df0 con 0x7fa670159f00 2023-10-14T16:59:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa674809640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6700b20d0 con 0x7fa6700b0b20 2023-10-14T16:59:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa667fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa667fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41990/0 (socket says 172.21.15.29:41990) 2023-10-14T16:59:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa667fff640 1 -- 172.21.15.29:0/2185848839 learned_addr learned my addr 172.21.15.29:0/2185848839 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa66ed9d640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa670159f00 0x7fa670158890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa66e59c640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6700b0b20 0x7fa6700b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa667fff640 1 -- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6700b0b20 msgr2=0x7fa6700b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.954+0000 7fa667fff640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6700b0b20 0x7fa6700b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa667fff640 1 -- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa670159f00 msgr2=0x7fa670158890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa667fff640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa670159f00 0x7fa670158890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa667fff640 1 -- 172.21.15.29:0/2185848839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6600094d0 con 0x7fa670159b20 2023-10-14T16:59:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa667fff640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c400 tx=0x7fa65800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658011790 con 0x7fa670159b20 2023-10-14T16:59:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658011930 con 0x7fa670159b20 2023-10-14T16:59:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6700b23b0 con 0x7fa670159b20 2023-10-14T16:59:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa658011ce0 con 0x7fa670159b20 2023-10-14T16:59:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.955+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa670158dd0 con 0x7fa670159b20 2023-10-14T16:59:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.956+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa67004e160 con 0x7fa670159b20 2023-10-14T16:59:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.957+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa65801c460 con 0x7fa670159b20 2023-10-14T16:59:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.958+0000 7fa666ffd640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 0x7fa65006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.958+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fa6580897f0 con 0x7fa670159b20 2023-10-14T16:59:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.958+0000 7fa66e59c640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 0x7fa65006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.959+0000 7fa66e59c640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 0x7fa65006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa660036000 tx=0x7fa66003c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:16.962+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6580546b0 con 0x7fa670159b20 2023-10-14T16:59:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.151+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6700b10b0 con 0x7fa670159b20 2023-10-14T16:59:17.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.153+0000 7fa666ffd640 1 -- 172.21.15.29:0/2185848839 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v116) v1 ==== 74+0+14176 (secure 0 0 0) 0x7fa658057cd0 con 0x7fa670159b20 2023-10-14T16:59:17.154 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:17.154 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":116,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:14.687959+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6832/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:6833/4145442506":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/712849763":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/1941242736":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/199695451":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2087407689":"2023-10-15T16:59:07.763572+0000","172.21.15.29:6836/1784427614":"2023-10-15T16:59:11.666967+0000","172.21.15.29:0/3563840307":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/869869993":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2128677149":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/3340406750":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3651737802":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/583660233":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6834/1784427614":"2023-10-15T16:59:11.666967+0000","172.21.15.29:0/856743829":"2023-10-15T16:59:07.763572+0000","172.21.15.29:0/2619862699":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/3275044227":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6833/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:6832/1520757887":"2023-10-15T16:57:59.666943+0000","172.21.15.29:0/680310103":"2023-10-15T16:57:59.666943+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:17.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.155+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 msgr2=0x7fa65006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.155+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 0x7fa65006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fa660036000 tx=0x7fa66003c040 comp rx=0 tx=0).stop 2023-10-14T16:59:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.155+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 msgr2=0x7fa6700c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.155+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 crypto rx=0x7fa65800c400 tx=0x7fa65800c8c0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 shutdown_connections 2023-10-14T16:59:17.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa65006af90 0x7fa65006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa670159f00 0x7fa670158890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa670159b20 0x7fa6700c5a60 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 --2- 172.21.15.29:0/2185848839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6700b0b20 0x7fa6700b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 >> 172.21.15.29:0/2185848839 conn(0x7fa6700cb410 msgr2=0x7fa6700ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:17.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 shutdown_connections 2023-10-14T16:59:17.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.156+0000 7fa674809640 1 -- 172.21.15.29:0/2185848839 wait complete. 2023-10-14T16:59:17.175 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:59:17.176 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:59:17.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.391+0000 7f38b8778640 1 Processor -- start 2023-10-14T16:59:17.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.391+0000 7f38b8778640 1 -- start start 2023-10-14T16:59:17.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.392+0000 7f38b8778640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38b00c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.392+0000 7f38b8778640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00ac890 con 0x7f38b00ad350 2023-10-14T16:59:17.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.392+0000 7f38b8778640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38b00ad7a0 con 0x7f38b015d150 2023-10-14T16:59:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.392+0000 7f38b8778640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b015d530 con 0x7f38b006a4b0 2023-10-14T16:59:17.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.393+0000 7f38b5d13640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38b00ad350 0x7f38b0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50306/0 (socket says 172.21.15.29:50306) 2023-10-14T16:59:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.393+0000 7f38b5d13640 1 -- 172.21.15.29:0/528772230 learned_addr learned my addr 172.21.15.29:0/528772230 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.393+0000 7f38b6d15640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38b00c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.393+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424931593 0 0) 0x7f38b00ac890 con 0x7f38b00ad350 2023-10-14T16:59:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.393+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3898003520 con 0x7f38b00ad350 2023-10-14T16:59:17.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286628825 0 0) 0x7f3898003520 con 0x7f38b00ad350 2023-10-14T16:59:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 msgr2=0x7f38b00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38b00c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 >> v1:172.21.15.186:6789/0 conn(0x7f38b015d150 legacy=0x7f38b0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b0161650 con 0x7f38b00ad350 2023-10-14T16:59:17.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f38a4001670 con 0x7f38b00ad350 2023-10-14T16:59:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 >> v1:172.21.15.29:6789/0 conn(0x7f38b00ad350 legacy=0x7f38b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38980039a0 0x7f3898003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3898004d00 0x7f3898005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b015d530 con 0x7f38980039a0 2023-10-14T16:59:17.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f389800ba90 con 0x7f3898004d00 2023-10-14T16:59:17.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f389800bc50 con 0x7f38b006a4b0 2023-10-14T16:59:17.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.394+0000 7f38b6514640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38980039a0 0x7f3898003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b5d13640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6d15640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3898004d00 0x7f3898005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6514640 1 -- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 msgr2=0x7f38980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6514640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b006a4b0 0x7f38980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6514640 1 -- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3898004d00 msgr2=0x7f3898005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6514640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3898004d00 0x7f3898005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.395+0000 7f38b6514640 1 -- 172.21.15.29:0/528772230 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3898003520 con 0x7f38980039a0 2023-10-14T16:59:17.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.396+0000 7f38b6514640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38980039a0 0x7f3898003dc0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f38a0001e20 tx=0x7f38980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45da088b4366e76d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.396+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0003b70 con 0x7f38980039a0 2023-10-14T16:59:17.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.396+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a0003cd0 con 0x7f38980039a0 2023-10-14T16:59:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.396+0000 7f38b5512640 1 -- 172.21.15.29:0/528772230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a0005310 con 0x7f38980039a0 2023-10-14T16:59:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 -- 172.21.15.29:0/528772230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38980039a0 msgr2=0x7f3898003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 --2- 172.21.15.29:0/528772230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38980039a0 0x7f3898003dc0 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 crypto rx=0x7f38a0001e20 tx=0x7f38980057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b6d15640 1 -- 172.21.15.29:0/528772230 reap_dead start 2023-10-14T16:59:17.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 -- 172.21.15.29:0/528772230 shutdown_connections 2023-10-14T16:59:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 -- 172.21.15.29:0/528772230 >> 172.21.15.29:0/528772230 conn(0x7f38b00ce9b0 msgr2=0x7f38b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 -- 172.21.15.29:0/528772230 shutdown_connections 2023-10-14T16:59:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.397+0000 7f38b8778640 1 -- 172.21.15.29:0/528772230 wait complete. 2023-10-14T16:59:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.399+0000 7f38b8778640 1 Processor -- start 2023-10-14T16:59:17.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.399+0000 7f38b8778640 1 -- start start 2023-10-14T16:59:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 0x7f38b00afeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b00af6b0 0x7f38b0154f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b00b3c10 con 0x7f38b00b3250 2023-10-14T16:59:17.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b00b10a0 con 0x7f38980039a0 2023-10-14T16:59:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.400+0000 7f38b8778640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b00b0490 con 0x7f38b00af6b0 2023-10-14T16:59:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42010/0 (socket says 172.21.15.29:42010) 2023-10-14T16:59:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 -- 172.21.15.29:0/846252495 learned_addr learned my addr 172.21.15.29:0/846252495 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:17.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6514640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 0x7f38b00afeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b5d13640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b00af6b0 0x7f38b0154f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 -- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b00af6b0 msgr2=0x7f38b0154f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b00af6b0 0x7f38b0154f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 -- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 msgr2=0x7f38b00afeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 0x7f38b00afeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 -- 172.21.15.29:0/846252495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b00b0770 con 0x7f38b00b3250 2023-10-14T16:59:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6514640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 0x7f38b00afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.401+0000 7f38b6d15640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f38b00cf870 tx=0x7f38a8011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.402+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8004500 con 0x7f38b00b3250 2023-10-14T16:59:17.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.402+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a8003610 con 0x7f38b00b3250 2023-10-14T16:59:17.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.402+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a80039e0 con 0x7f38b00b3250 2023-10-14T16:59:17.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.402+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b00b0a50 con 0x7f38b00b3250 2023-10-14T16:59:17.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.402+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b0155480 con 0x7f38b00b3250 2023-10-14T16:59:17.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.403+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b004e160 con 0x7f38b00b3250 2023-10-14T16:59:17.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.404+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f38a8003b40 con 0x7f38b00b3250 2023-10-14T16:59:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.405+0000 7f38aeffd640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 0x7f389c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.405+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f38a8089800 con 0x7f38b00b3250 2023-10-14T16:59:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.405+0000 7f38b6514640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 0x7f389c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.406+0000 7f38b6514640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 0x7f389c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38a0034000 tx=0x7f38a0039040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.409+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38a80546c0 con 0x7f38b00b3250 2023-10-14T16:59:17.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.602+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f38b015d670 con 0x7f38b00b3250 2023-10-14T16:59:17.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.603+0000 7f38aeffd640 1 -- 172.21.15.29:0/846252495 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v116) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38a8057ce0 con 0x7f38b00b3250 2023-10-14T16:59:17.604 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:59:17.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 msgr2=0x7f389c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 0x7f389c06d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f38a0034000 tx=0x7f38a0039040 comp rx=0 tx=0).stop 2023-10-14T16:59:17.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 msgr2=0x7f38b00b36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 crypto rx=0x7f38b00cf870 tx=0x7f38a8011920 comp rx=0 tx=0).stop 2023-10-14T16:59:17.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 shutdown_connections 2023-10-14T16:59:17.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f389c06af90 0x7f389c06d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38b00b3250 0x7f38b00b36d0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38b00af6b0 0x7f38b0154f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 --2- 172.21.15.29:0/846252495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38980039a0 0x7f38b00afeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 >> 172.21.15.29:0/846252495 conn(0x7f38b00ce9b0 msgr2=0x7f38b015b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:17.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.606+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 shutdown_connections 2023-10-14T16:59:17.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.607+0000 7f38b8778640 1 -- 172.21.15.29:0/846252495 wait complete. 2023-10-14T16:59:17.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-14T16:59:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.851+0000 7f4e3302e640 1 Processor -- start 2023-10-14T16:59:17.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.851+0000 7f4e3302e640 1 -- start start 2023-10-14T16:59:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3302e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c070d70 0x7f4e2c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3302e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e2c0712c0 0x7f4e2c06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3302e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c06c9c0 con 0x7f4e2c15aad0 2023-10-14T16:59:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3302e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e2c0b4620 con 0x7f4e2c0712c0 2023-10-14T16:59:17.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3302e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0b4340 con 0x7f4e2c070d70 2023-10-14T16:59:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3202c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e2c15aad0 0x7f4e2c0cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50318/0 (socket says 172.21.15.29:50318) 2023-10-14T16:59:17.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3202c640 1 -- 172.21.15.29:0/3644638718 learned_addr learned my addr 172.21.15.29:0/3644638718 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3282d640 1 --2- 172.21.15.29:0/3644638718 >> v2:172.21.15.29:3300/0 conn(0x7f4e2c0712c0 0x7f4e2c06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.853+0000 7f4e3182b640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c070d70 0x7f4e2c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3102a640 1 -- 172.21.15.29:0/3644638718 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007501460 0 0) 0x7f4e2c06c9c0 con 0x7f4e2c15aad0 2023-10-14T16:59:17.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3102a640 1 -- 172.21.15.29:0/3644638718 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e18003520 con 0x7f4e2c15aad0 2023-10-14T16:59:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3282d640 1 -- 172.21.15.29:0/3644638718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c070d70 msgr2=0x7f4e2c06bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3282d640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c070d70 0x7f4e2c06bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3282d640 1 -- 172.21.15.29:0/3644638718 >> v1:172.21.15.29:6789/0 conn(0x7f4e2c15aad0 legacy=0x7f4e2c0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.854+0000 7f4e3282d640 1 -- 172.21.15.29:0/3644638718 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c06f520 con 0x7f4e2c0712c0 2023-10-14T16:59:17.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.855+0000 7f4e3282d640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c0712c0 0x7f4e2c06c480 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d90 tx=0x7f4e2402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d6c77c2e2d262d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.855+0000 7f4e3102a640 1 -- 172.21.15.29:0/3644638718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2402cce0 con 0x7f4e2c0712c0 2023-10-14T16:59:17.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.855+0000 7f4e3102a640 1 -- 172.21.15.29:0/3644638718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2402ce40 con 0x7f4e2c0712c0 2023-10-14T16:59:17.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.855+0000 7f4e3102a640 1 -- 172.21.15.29:0/3644638718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24034660 con 0x7f4e2c0712c0 2023-10-14T16:59:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 -- 172.21.15.29:0/3644638718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c0712c0 msgr2=0x7f4e2c06c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c0712c0 0x7f4e2c06c480 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d90 tx=0x7f4e2402bf30 comp rx=0 tx=0).stop 2023-10-14T16:59:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 -- 172.21.15.29:0/3644638718 shutdown_connections 2023-10-14T16:59:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c0712c0 0x7f4e2c06c480 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 --2- 172.21.15.29:0/3644638718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c070d70 0x7f4e2c06bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 -- 172.21.15.29:0/3644638718 >> 172.21.15.29:0/3644638718 conn(0x7f4e2c0b0a90 msgr2=0x7f4e2c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:17.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 -- 172.21.15.29:0/3644638718 shutdown_connections 2023-10-14T16:59:17.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.856+0000 7f4e3302e640 1 -- 172.21.15.29:0/3644638718 wait complete. 2023-10-14T16:59:17.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.858+0000 7f4e3302e640 1 Processor -- start 2023-10-14T16:59:17.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.858+0000 7f4e3302e640 1 -- start start 2023-10-14T16:59:17.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c070d70 0x7f4e2c1464b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c0712c0 0x7f4e2c15ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0b2da0 con 0x7f4e2c15aad0 2023-10-14T16:59:17.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0b2190 con 0x7f4e2c070d70 2023-10-14T16:59:17.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3302e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e2c0b2470 con 0x7f4e2c0712c0 2023-10-14T16:59:17.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3282d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3282d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42020/0 (socket says 172.21.15.29:42020) 2023-10-14T16:59:17.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.859+0000 7f4e3282d640 1 -- 172.21.15.29:0/2778522694 learned_addr learned my addr 172.21.15.29:0/2778522694 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:17.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3182b640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c0712c0 0x7f4e2c15ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3202c640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c070d70 0x7f4e2c1464b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 -- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c0712c0 msgr2=0x7f4e2c15ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c0712c0 0x7f4e2c15ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 -- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c070d70 msgr2=0x7f4e2c1464b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c070d70 0x7f4e2c1464b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 -- 172.21.15.29:0/2778522694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24009a70 con 0x7f4e2c15aad0 2023-10-14T16:59:17.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.860+0000 7f4e3282d640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d90 tx=0x7f4e24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.861+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2403e670 con 0x7f4e2c15aad0 2023-10-14T16:59:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.861+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2403e7f0 con 0x7f4e2c15aad0 2023-10-14T16:59:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.861+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2403eba0 con 0x7f4e2c15aad0 2023-10-14T16:59:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.861+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c0b2750 con 0x7f4e2c15aad0 2023-10-14T16:59:17.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.861+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c0b3230 con 0x7f4e2c15aad0 2023-10-14T16:59:17.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.862+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c0ade70 con 0x7f4e2c15aad0 2023-10-14T16:59:17.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.863+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e2403b040 con 0x7f4e2c15aad0 2023-10-14T16:59:17.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.865+0000 7f4e22ffd640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:17.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.865+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f4e240ac190 con 0x7f4e2c15aad0 2023-10-14T16:59:17.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.865+0000 7f4e3202c640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:17.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.866+0000 7f4e3202c640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e280011e0 tx=0x7f4e28009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:17.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:17.869+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e24077050 con 0x7f4e2c15aad0 2023-10-14T16:59:18.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.136+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f4e2c04e9b0 con 0x7f4e2c15aad0 2023-10-14T16:59:18.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.717+0000 7f4e22ffd640 1 -- 172.21.15.29:0/2778522694 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v718) v1 ==== 151+0+0 (secure 0 0 0) 0x7f4e24009450 con 0x7f4e2c15aad0 2023-10-14T16:59:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 msgr2=0x7f4e1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 0x7f4e1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f4e280011e0 tx=0x7f4e28009040 comp rx=0 tx=0).stop 2023-10-14T16:59:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 msgr2=0x7f4e2c15d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 secure :-1 s=READY pgs=1362 cs=0 l=1 rev1=1 crypto rx=0x7f4e24009d90 tx=0x7f4e24004a40 comp rx=0 tx=0).stop 2023-10-14T16:59:18.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 shutdown_connections 2023-10-14T16:59:18.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1006af90 0x7f4e1006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e2c15aad0 0x7f4e2c15d410 unknown :-1 s=CLOSED pgs=1362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e2c0712c0 0x7f4e2c15ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 --2- 172.21.15.29:0/2778522694 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e2c070d70 0x7f4e2c1464b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 >> 172.21.15.29:0/2778522694 conn(0x7f4e2c0b0a90 msgr2=0x7f4e2c0ad230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 shutdown_connections 2023-10-14T16:59:18.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.721+0000 7f4e3302e640 1 -- 172.21.15.29:0/2778522694 wait complete. 2023-10-14T16:59:18.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T16:59:18.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.971+0000 7ff852275640 1 Processor -- start 2023-10-14T16:59:18.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.971+0000 7ff852275640 1 -- start start 2023-10-14T16:59:18.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff852275640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff84c159950 0x7ff84c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff852275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84c070450 0x7ff84c0708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff852275640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff84c15aef0 con 0x7ff84c06c100 2023-10-14T16:59:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff852275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c070e20 con 0x7ff84c070450 2023-10-14T16:59:18.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff852275640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff84c15e490 con 0x7ff84c159950 2023-10-14T16:59:18.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff850a72640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84c070450 0x7ff84c0708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:18.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff850a72640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff84c070450 0x7ff84c0708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42022/0 (socket says 172.21.15.29:42022) 2023-10-14T16:59:18.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.973+0000 7ff850a72640 1 -- 172.21.15.29:0/4206135641 learned_addr learned my addr 172.21.15.29:0/4206135641 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:18.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.974+0000 7ff850a72640 1 -- 172.21.15.29:0/4206135641 >> v2:172.21.15.186:3300/0 conn(0x7ff84c159950 msgr2=0x7ff84c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:18.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.974+0000 7ff850a72640 1 --2- 172.21.15.29:0/4206135641 >> v2:172.21.15.186:3300/0 conn(0x7ff84c159950 0x7ff84c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.974+0000 7ff850a72640 1 -- 172.21.15.29:0/4206135641 >> v1:172.21.15.186:6789/0 conn(0x7ff84c06c100 legacy=0x7ff84c06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:18.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.974+0000 7ff850a72640 1 -- 172.21.15.29:0/4206135641 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c15d910 con 0x7ff84c070450 2023-10-14T16:59:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.975+0000 7ff850a72640 1 --2- 172.21.15.29:0/4206135641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c070450 0x7ff84c0708e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0ca950 tx=0x7ff83c02da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7746de26be2beede server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.975+0000 7ff843fff640 1 -- 172.21.15.29:0/4206135641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c02f850 con 0x7ff84c070450 2023-10-14T16:59:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.975+0000 7ff843fff640 1 -- 172.21.15.29:0/4206135641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff83c02f9b0 con 0x7ff84c070450 2023-10-14T16:59:18.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.975+0000 7ff843fff640 1 -- 172.21.15.29:0/4206135641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff83c02fd40 con 0x7ff84c070450 2023-10-14T16:59:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 -- 172.21.15.29:0/4206135641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c070450 msgr2=0x7ff84c0708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 --2- 172.21.15.29:0/4206135641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c070450 0x7ff84c0708e0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 crypto rx=0x7ff84c0ca950 tx=0x7ff83c02da90 comp rx=0 tx=0).stop 2023-10-14T16:59:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 -- 172.21.15.29:0/4206135641 shutdown_connections 2023-10-14T16:59:18.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 --2- 172.21.15.29:0/4206135641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c070450 0x7ff84c0708e0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 --2- 172.21.15.29:0/4206135641 >> v2:172.21.15.186:3300/0 conn(0x7ff84c159950 0x7ff84c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 -- 172.21.15.29:0/4206135641 >> 172.21.15.29:0/4206135641 conn(0x7ff84c0ce6a0 msgr2=0x7ff84c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 -- 172.21.15.29:0/4206135641 shutdown_connections 2023-10-14T16:59:18.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.976+0000 7ff852275640 1 -- 172.21.15.29:0/4206135641 wait complete. 2023-10-14T16:59:18.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.978+0000 7ff852275640 1 Processor -- start 2023-10-14T16:59:18.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.978+0000 7ff852275640 1 -- start start 2023-10-14T16:59:18.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c070450 0x7ff84c0c8510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 0x7ff84c15c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c1f34c0 con 0x7ff84c06c100 2023-10-14T16:59:18.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c0c8a50 con 0x7ff84c070450 2023-10-14T16:59:18.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff852275640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c1f2c20 con 0x7ff84c159950 2023-10-14T16:59:18.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.979+0000 7ff851273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42028/0 (socket says 172.21.15.29:42028) 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 -- 172.21.15.29:0/81887533 learned_addr learned my addr 172.21.15.29:0/81887533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff850a72640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c070450 0x7ff84c0c8510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851a74640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 0x7ff84c15c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 -- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 msgr2=0x7ff84c15c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:18.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 0x7ff84c15c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 -- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c070450 msgr2=0x7ff84c0c8510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:18.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c070450 0x7ff84c0c8510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:18.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 -- 172.21.15.29:0/81887533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83c0094d0 con 0x7ff84c06c100 2023-10-14T16:59:18.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851a74640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 0x7ff84c15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:18.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.980+0000 7ff851273640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7ff84800b410 tx=0x7ff84800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:18.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.981+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848011830 con 0x7ff84c06c100 2023-10-14T16:59:18.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.981+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8480119b0 con 0x7ff84c06c100 2023-10-14T16:59:18.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.981+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848011d60 con 0x7ff84c06c100 2023-10-14T16:59:18.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.981+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c1f2f00 con 0x7ff84c06c100 2023-10-14T16:59:18.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.981+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c0c22e0 con 0x7ff84c06c100 2023-10-14T16:59:18.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.982+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c04e9b0 con 0x7ff84c06c100 2023-10-14T16:59:18.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.983+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff84801a400 con 0x7ff84c06c100 2023-10-14T16:59:18.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.984+0000 7ff841ffb640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 0x7ff83006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:18.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.984+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7ff8480898c0 con 0x7ff84c06c100 2023-10-14T16:59:18.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.985+0000 7ff850a72640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 0x7ff83006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:18.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.985+0000 7ff850a72640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 0x7ff83006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff83c038000 tx=0x7ff83c03b040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:18.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:18.990+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8480548b0 con 0x7ff84c06c100 2023-10-14T16:59:19.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.185+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7ff84c0b4620 con 0x7ff84c06c100 2023-10-14T16:59:19.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.193+0000 7ff841ffb640 1 -- 172.21.15.29:0/81887533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v40)=0 v40) v1 ==== 122+0+0 (secure 0 0 0) 0x7ff848015020 con 0x7ff84c06c100 2023-10-14T16:59:19.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 msgr2=0x7ff83006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 0x7ff83006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff83c038000 tx=0x7ff83c03b040 comp rx=0 tx=0).stop 2023-10-14T16:59:19.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 msgr2=0x7ff84c1f2520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 crypto rx=0x7ff84800b410 tx=0x7ff84800b8d0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 shutdown_connections 2023-10-14T16:59:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83006af90 0x7ff83006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff84c159950 0x7ff84c15c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff84c070450 0x7ff84c0c8510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 --2- 172.21.15.29:0/81887533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff84c06c100 0x7ff84c1f2520 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 >> 172.21.15.29:0/81887533 conn(0x7ff84c0ce6a0 msgr2=0x7ff84c0cd510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:19.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 shutdown_connections 2023-10-14T16:59:19.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.196+0000 7ff852275640 1 -- 172.21.15.29:0/81887533 wait complete. 2023-10-14T16:59:19.215 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T16:59:19.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:59:19.320 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T16:59:19.443 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T16:59:19.443 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:19.444 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:19.506 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:59:19.507 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:59:19.507 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:19.569 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:59:19.569 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:19.570 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:19.633 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T16:59:19.634 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T16:59:19.634 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:19.694 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T16:59:19.695 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T16:59:19.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.925+0000 7fb40931d640 1 Processor -- start 2023-10-14T16:59:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.925+0000 7fb40931d640 1 -- start start 2023-10-14T16:59:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb40931d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4041615f0 0x7fb40415db00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb40931d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb40415e3e0 con 0x7fb4040b0b20 2023-10-14T16:59:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb40931d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4040cd4a0 con 0x7fb4040cd0c0 2023-10-14T16:59:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb40931d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb40415e100 con 0x7fb4041615f0 2023-10-14T16:59:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb4027fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb4040b0b20 0x7fb40415d3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50324/0 (socket says 172.21.15.29:50324) 2023-10-14T16:59:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.927+0000 7fb4027fc640 1 -- 172.21.15.29:0/4274228618 learned_addr learned my addr 172.21.15.29:0/4274228618 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb4037fe640 1 --2- 172.21.15.29:0/4274228618 >> v2:172.21.15.186:3300/0 conn(0x7fb4041615f0 0x7fb40415db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433560547 0 0) 0x7fb40415e3e0 con 0x7fb4040b0b20 2023-10-14T16:59:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3ec003520 con 0x7fb4040b0b20 2023-10-14T16:59:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 893626410 0 0) 0x7fb3ec003520 con 0x7fb4040b0b20 2023-10-14T16:59:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 >> v2:172.21.15.186:3300/0 conn(0x7fb4041615f0 msgr2=0x7fb40415db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 --2- 172.21.15.29:0/4274228618 >> v2:172.21.15.186:3300/0 conn(0x7fb4041615f0 0x7fb40415db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 >> v1:172.21.15.186:6789/0 conn(0x7fb4040cd0c0 legacy=0x7fb4040b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404160b10 con 0x7fb4040b0b20 2023-10-14T16:59:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb3f8001670 con 0x7fb4040b0b20 2023-10-14T16:59:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 >> v1:172.21.15.29:6789/0 conn(0x7fb4040b0b20 legacy=0x7fb40415d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 0x7fb3ec003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec0047c0 0x7fb3ec004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ec005180 0x7fb3ec00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40415e100 con 0x7fb3ec003970 2023-10-14T16:59:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec00c0b0 con 0x7fb3ec005180 2023-10-14T16:59:19.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.928+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec00c270 con 0x7fb3ec0047c0 2023-10-14T16:59:19.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 0x7fb3ec003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb4037fe640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ec005180 0x7fb3ec00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 -- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec0047c0 msgr2=0x7fb3ec004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec0047c0 0x7fb3ec004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 -- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ec005180 msgr2=0x7fb3ec00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ec005180 0x7fb3ec00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb402ffd640 1 -- 172.21.15.29:0/4274228618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec003520 con 0x7fb3ec003970 2023-10-14T16:59:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb4037fe640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3ec005180 0x7fb3ec00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.929+0000 7fb4037fe640 1 -- 172.21.15.29:0/4274228618 reap_dead start 2023-10-14T16:59:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.930+0000 7fb402ffd640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 0x7fb3ec003d70 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4003db0 tx=0x7fb3ec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399567d91954ce17 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f4032600 con 0x7fb3ec003970 2023-10-14T16:59:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f402e050 con 0x7fb3ec003970 2023-10-14T16:59:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb401ffb640 1 -- 172.21.15.29:0/4274228618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f40329e0 con 0x7fb3ec003970 2023-10-14T16:59:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 -- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 msgr2=0x7fb3ec003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 0x7fb3ec003d70 secure :-1 s=READY pgs=1366 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4003db0 tx=0x7fb3ec0057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 -- 172.21.15.29:0/4274228618 shutdown_connections 2023-10-14T16:59:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 --2- 172.21.15.29:0/4274228618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb3ec003970 0x7fb3ec003d70 unknown :-1 s=CLOSED pgs=1366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 -- 172.21.15.29:0/4274228618 >> 172.21.15.29:0/4274228618 conn(0x7fb4040cb410 msgr2=0x7fb4040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.931+0000 7fb40931d640 1 -- 172.21.15.29:0/4274228618 shutdown_connections 2023-10-14T16:59:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.932+0000 7fb40931d640 1 -- 172.21.15.29:0/4274228618 wait complete. 2023-10-14T16:59:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.933+0000 7fb40931d640 1 Processor -- start 2023-10-14T16:59:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.933+0000 7fb40931d640 1 -- start start 2023-10-14T16:59:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.934+0000 7fb40931d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec003970 0x7fb4040c2e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.934+0000 7fb40931d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.934+0000 7fb40931d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4040c6b40 0x7fb404154510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb40931d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040c29e0 con 0x7fb40406c7a0 2023-10-14T16:59:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb40931d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040c2400 con 0x7fb4040c6b40 2023-10-14T16:59:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb40931d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4040c33b0 con 0x7fb3ec003970 2023-10-14T16:59:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42054/0 (socket says 172.21.15.29:42054) 2023-10-14T16:59:19.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 -- 172.21.15.29:0/2135197942 learned_addr learned my addr 172.21.15.29:0/2135197942 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4037fe640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4040c6b40 0x7fb404154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 -- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec003970 msgr2=0x7fb4040c2e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec003970 0x7fb4040c2e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 -- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4040c6b40 msgr2=0x7fb404154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4040c6b40 0x7fb404154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.935+0000 7fb4027fc640 1 -- 172.21.15.29:0/2135197942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4040c6fe0 con 0x7fb40406c7a0 2023-10-14T16:59:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb4027fc640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800c7a0 tx=0x7fb3f800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f80074e0 con 0x7fb40406c7a0 2023-10-14T16:59:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3f8007640 con 0x7fb40406c7a0 2023-10-14T16:59:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8004a00 con 0x7fb40406c7a0 2023-10-14T16:59:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4040c2670 con 0x7fb40406c7a0 2023-10-14T16:59:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.936+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4040c5b80 con 0x7fb40406c7a0 2023-10-14T16:59:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.937+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb40404e160 con 0x7fb40406c7a0 2023-10-14T16:59:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.938+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb3f8004b60 con 0x7fb40406c7a0 2023-10-14T16:59:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.939+0000 7fb3eb7fe640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 0x7fb3f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.939+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7fb3f8088b50 con 0x7fb40406c7a0 2023-10-14T16:59:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.940+0000 7fb402ffd640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 0x7fb3f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.940+0000 7fb402ffd640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 0x7fb3f006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402f000 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:19.944+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb3f8053aa0 con 0x7fb40406c7a0 2023-10-14T16:59:20.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.184+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4040c2b90 con 0x7fb40406c7a0 2023-10-14T16:59:20.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.185+0000 7fb3eb7fe640 1 -- 172.21.15.29:0/2135197942 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+3699 (secure 0 0 0) 0x7fb4040c2b90 con 0x7fb40406c7a0 2023-10-14T16:59:20.186 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:20.186 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":51,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5570,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3150611835","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3150611835},{"type":"v1","addr":"172.21.15.29:6837","nonce":3150611835}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":44},{"gid":6070,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/359597586","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":359597586},{"type":"v1","addr":"172.21.15.186:6833","nonce":359597586}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":6073,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3502679270","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3502679270},{"type":"v1","addr":"172.21.15.186:6835","nonce":3502679270}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":43},{"gid":7941,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/568431459","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":568431459},{"type":"v1","addr":"172.21.15.29:6836","nonce":568431459}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":51}],"filesystems":[]} 2023-10-14T16:59:20.186 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 51 2023-10-14T16:59:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 msgr2=0x7fb3f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 0x7fb3f006d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb3f4034000 tx=0x7fb3f402f000 comp rx=0 tx=0).stop 2023-10-14T16:59:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 msgr2=0x7fb4040c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800c7a0 tx=0x7fb3f800cc60 comp rx=0 tx=0).stop 2023-10-14T16:59:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 shutdown_connections 2023-10-14T16:59:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fb3f006af90 0x7fb3f006d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4040c6b40 0x7fb404154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb40406c7a0 0x7fb4040c6600 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 --2- 172.21.15.29:0/2135197942 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb3ec003970 0x7fb4040c2e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.188+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 >> 172.21.15.29:0/2135197942 conn(0x7fb4040cb410 msgr2=0x7fb4040c9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.189+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 shutdown_connections 2023-10-14T16:59:20.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.189+0000 7fb40931d640 1 -- 172.21.15.29:0/2135197942 wait complete. 2023-10-14T16:59:20.209 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T16:59:20.209 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T16:59:20.209 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T16:59:20.210 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:59:20.210 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T16:59:20.210 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T16:59:20.210 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T16:59:20.211 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:59:20.211 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T16:59:20.211 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T16:59:20.212 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T16:59:20.212 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:59:20.212 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T16:59:20.212 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T16:59:20.213 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T16:59:20.213 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T16:59:20.227 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T16:59:20.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T16:59:20.230 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T16:59:20.231 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T16:59:20.232 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T16:59:20.232 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T16:59:20.233 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T16:59:20.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T16:59:20.278 INFO:tasks.ceph.mds.a:Started 2023-10-14T16:59:20.279 DEBUG:teuthology.parallel:result is None 2023-10-14T16:59:20.282 INFO:tasks.ceph.mds.b:Started 2023-10-14T16:59:20.282 DEBUG:teuthology.parallel:result is None 2023-10-14T16:59:20.286 INFO:tasks.ceph.mds.c:Started 2023-10-14T16:59:20.287 DEBUG:teuthology.parallel:result is None 2023-10-14T16:59:20.289 INFO:tasks.ceph.mds.d:Started 2023-10-14T16:59:20.289 DEBUG:teuthology.parallel:result is None 2023-10-14T16:59:20.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T16:59:20.443 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T16:59:20.447 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T16:59:20.447 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T16:59:20.456 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T16:59:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.485+0000 7f69c15b0640 1 Processor -- start 2023-10-14T16:59:20.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.485+0000 7f69c15b0640 1 -- start start 2023-10-14T16:59:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.486+0000 7f69c15b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69bc15ad70 0x7f69bc069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.486+0000 7f69c15b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc0caac0 0x7f69bc06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.486+0000 7f69c15b0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69bc0cb030 con 0x7f69bc0c9ca0 2023-10-14T16:59:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.486+0000 7f69c15b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69bc0c91c0 con 0x7f69bc15ad70 2023-10-14T16:59:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.486+0000 7f69c15b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc0c94a0 con 0x7f69bc0caac0 2023-10-14T16:59:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69baffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f69bc15ad70 0x7f69bc069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69b3fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f69bc0c9ca0 0x7f69bc0ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:53934/0 (socket says 172.21.15.29:53934) 2023-10-14T16:59:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69b3fff640 1 -- 172.21.15.29:0/2158794325 learned_addr learned my addr 172.21.15.29:0/2158794325 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69bb7fe640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc0caac0 0x7f69bc06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69baffd640 1 -- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc0caac0 msgr2=0x7f69bc06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69baffd640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc0caac0 0x7f69bc06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69baffd640 1 -- 172.21.15.29:0/2158794325 >> v1:172.21.15.186:6789/0 conn(0x7f69bc0c9ca0 legacy=0x7f69bc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69ba7fc640 1 -- 172.21.15.29:0/2158794325 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2510529132 0 0) 0x7f69bc0cb030 con 0x7f69bc0c9ca0 2023-10-14T16:59:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.487+0000 7f69baffd640 1 -- 172.21.15.29:0/2158794325 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc0c9780 con 0x7f69bc15ad70 2023-10-14T16:59:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.488+0000 7f69baffd640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc15ad70 0x7f69bc069da0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097a0 tx=0x7f69ac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c87ea7dac484f48 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69ba7fc640 1 -- 172.21.15.29:0/2158794325 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac031830 con 0x7f69bc15ad70 2023-10-14T16:59:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69ba7fc640 1 -- 172.21.15.29:0/2158794325 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69ac031990 con 0x7f69bc15ad70 2023-10-14T16:59:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69ba7fc640 1 -- 172.21.15.29:0/2158794325 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac031d20 con 0x7f69bc15ad70 2023-10-14T16:59:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 -- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc15ad70 msgr2=0x7f69bc069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc15ad70 0x7f69bc069da0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f69ac0097a0 tx=0x7f69ac02b8d0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 -- 172.21.15.29:0/2158794325 shutdown_connections 2023-10-14T16:59:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc0caac0 0x7f69bc06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 --2- 172.21.15.29:0/2158794325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc15ad70 0x7f69bc069da0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.489+0000 7f69c15b0640 1 -- 172.21.15.29:0/2158794325 >> 172.21.15.29:0/2158794325 conn(0x7f69bc0ce9b0 msgr2=0x7f69bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.490+0000 7f69c15b0640 1 -- 172.21.15.29:0/2158794325 shutdown_connections 2023-10-14T16:59:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.490+0000 7f69c15b0640 1 -- 172.21.15.29:0/2158794325 wait complete. 2023-10-14T16:59:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.491+0000 7f69c15b0640 1 Processor -- start 2023-10-14T16:59:20.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.491+0000 7f69c15b0640 1 -- start start 2023-10-14T16:59:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc0c9ca0 0x7f69bc159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc15ad70 0x7f69bc0af030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc159dc0 con 0x7f69bc0caac0 2023-10-14T16:59:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc15c720 con 0x7f69bc0c9ca0 2023-10-14T16:59:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.492+0000 7f69c15b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc15ca00 con 0x7f69bc15ad70 2023-10-14T16:59:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42088/0 (socket says 172.21.15.29:42088) 2023-10-14T16:59:20.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 -- 172.21.15.29:0/4258940329 learned_addr learned my addr 172.21.15.29:0/4258940329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69baffd640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc0c9ca0 0x7f69bc159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69bb7fe640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc15ad70 0x7f69bc0af030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 -- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc15ad70 msgr2=0x7f69bc0af030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc15ad70 0x7f69bc0af030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 -- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc0c9ca0 msgr2=0x7f69bc159880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc0c9ca0 0x7f69bc159880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.493+0000 7f69b3fff640 1 -- 172.21.15.29:0/4258940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac009400 con 0x7f69bc0caac0 2023-10-14T16:59:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69b3fff640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f69a400c4d0 tx=0x7f69a400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a40074e0 con 0x7f69bc0caac0 2023-10-14T16:59:20.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69a4007660 con 0x7f69bc0caac0 2023-10-14T16:59:20.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69a4004a20 con 0x7f69bc0caac0 2023-10-14T16:59:20.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc15cce0 con 0x7f69bc0caac0 2023-10-14T16:59:20.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.494+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc15d2a0 con 0x7f69bc0caac0 2023-10-14T16:59:20.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.495+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69bc04e160 con 0x7f69bc0caac0 2023-10-14T16:59:20.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.496+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f69a4004b80 con 0x7f69bc0caac0 2023-10-14T16:59:20.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.498+0000 7f69b37fe640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 0x7f699c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:20.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.498+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 5761+0+0 (secure 0 0 0) 0x7f69a4088960 con 0x7f69bc0caac0 2023-10-14T16:59:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.498+0000 7f69baffd640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 0x7f699c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:20.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.499+0000 7f69baffd640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 0x7f699c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69ac036000 tx=0x7f69ac03c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:20.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.503+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f69a40538d0 con 0x7f69bc0caac0 2023-10-14T16:59:20.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:20.704+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f69bc0ca880 con 0x7f69bc0caac0 2023-10-14T16:59:21.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.755+0000 7f69b37fe640 1 -- 172.21.15.29:0/4258940329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v121) v1 ==== 97+0+0 (secure 0 0 0) 0x7f69a4018020 con 0x7f69bc0caac0 2023-10-14T16:59:21.757 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T16:59:21.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 msgr2=0x7f699c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 0x7f699c06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f69ac036000 tx=0x7f69ac03c040 comp rx=0 tx=0).stop 2023-10-14T16:59:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 msgr2=0x7f69bc1f2550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 crypto rx=0x7f69a400c4d0 tx=0x7f69a400c990 comp rx=0 tx=0).stop 2023-10-14T16:59:21.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 shutdown_connections 2023-10-14T16:59:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f699c06af90 0x7f699c06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69bc15ad70 0x7f69bc0af030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69bc0caac0 0x7f69bc1f2550 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:21.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 --2- 172.21.15.29:0/4258940329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f69bc0c9ca0 0x7f69bc159880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.760+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 >> 172.21.15.29:0/4258940329 conn(0x7f69bc0ce9b0 msgr2=0x7f69bc070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.761+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 shutdown_connections 2023-10-14T16:59:21.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:21.761+0000 7f69c15b0640 1 -- 172.21.15.29:0/4258940329 wait complete. 2023-10-14T16:59:21.781 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T16:59:22.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.010+0000 7f1541e0d640 1 Processor -- start 2023-10-14T16:59:22.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.010+0000 7f1541e0d640 1 -- start start 2023-10-14T16:59:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.011+0000 7f1541e0d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f153c0b8480 0x7f153c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f1541e0d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c0d0be0 0x7f153c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f1541e0d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f153c15d150 con 0x7f153c15cd70 2023-10-14T16:59:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f1541e0d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f153c0d1180 con 0x7f153c0b8480 2023-10-14T16:59:22.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f1541e0d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c0d1460 con 0x7f153c0d0be0 2023-10-14T16:59:22.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f153bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f153c0b8480 0x7f153c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f153bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f153c0b8480 0x7f153c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54492/0 (socket says 172.21.15.29:54492) 2023-10-14T16:59:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f153bfff640 1 -- 172.21.15.29:0/4061631445 learned_addr learned my addr 172.21.15.29:0/4061631445 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.012+0000 7f154160c640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c0d0be0 0x7f153c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.013+0000 7f153bfff640 1 -- 172.21.15.29:0/4061631445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c0d0be0 msgr2=0x7f153c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:22.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.013+0000 7f153bfff640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c0d0be0 0x7f153c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.013+0000 7f153bfff640 1 -- 172.21.15.29:0/4061631445 >> v1:172.21.15.186:6789/0 conn(0x7f153c15cd70 legacy=0x7f153c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.013+0000 7f153bfff640 1 -- 172.21.15.29:0/4061631445 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153c0d1740 con 0x7f153c0b8480 2023-10-14T16:59:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.014+0000 7f153bfff640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0b8480 0x7f153c0cd7c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f1524009870 tx=0x7f152402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1705174ac26475f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.014+0000 7f153b7fe640 1 -- 172.21.15.29:0/4061631445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524030840 con 0x7f153c0b8480 2023-10-14T16:59:22.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.014+0000 7f153b7fe640 1 -- 172.21.15.29:0/4061631445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15240309a0 con 0x7f153c0b8480 2023-10-14T16:59:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.014+0000 7f153b7fe640 1 -- 172.21.15.29:0/4061631445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524030d30 con 0x7f153c0b8480 2023-10-14T16:59:22.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 -- 172.21.15.29:0/4061631445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0b8480 msgr2=0x7f153c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0b8480 0x7f153c0cd7c0 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 crypto rx=0x7f1524009870 tx=0x7f152402b980 comp rx=0 tx=0).stop 2023-10-14T16:59:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 -- 172.21.15.29:0/4061631445 shutdown_connections 2023-10-14T16:59:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c0d0be0 0x7f153c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 --2- 172.21.15.29:0/4061631445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0b8480 0x7f153c0cd7c0 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 -- 172.21.15.29:0/4061631445 >> 172.21.15.29:0/4061631445 conn(0x7f153c0cf570 msgr2=0x7f153c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 -- 172.21.15.29:0/4061631445 shutdown_connections 2023-10-14T16:59:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.015+0000 7f1541e0d640 1 -- 172.21.15.29:0/4061631445 wait complete. 2023-10-14T16:59:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.016+0000 7f1541e0d640 1 Processor -- start 2023-10-14T16:59:22.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.017+0000 7f1541e0d640 1 -- start start 2023-10-14T16:59:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.017+0000 7f1541e0d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153c0b8480 0x7f153c0cc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.017+0000 7f1541e0d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.017+0000 7f1541e0d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c15cd70 0x7f153c0b42d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f1541e0d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f5e30 con 0x7f153c0d0be0 2023-10-14T16:59:22.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f1541e0d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f6110 con 0x7f153c0b8480 2023-10-14T16:59:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f1541e0d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f153c1f63f0 con 0x7f153c15cd70 2023-10-14T16:59:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54496/0 (socket says 172.21.15.29:54496) 2023-10-14T16:59:22.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 -- 172.21.15.29:0/2893661470 learned_addr learned my addr 172.21.15.29:0/2893661470 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 -- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c15cd70 msgr2=0x7f153c0b42d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c15cd70 0x7f153c0b42d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 -- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153c0b8480 msgr2=0x7f153c0cc8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f1540e0b640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153c0b8480 0x7f153c0cc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153c0b8480 0x7f153c0cc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.018+0000 7f153bfff640 1 -- 172.21.15.29:0/2893661470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15240094d0 con 0x7f153c0d0be0 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f153bfff640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f1524000c00 tx=0x7f1524004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524039a90 con 0x7f153c0d0be0 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1524039bf0 con 0x7f153c0d0be0 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1524043410 con 0x7f153c0d0be0 2023-10-14T16:59:22.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153c1f66d0 con 0x7f153c0d0be0 2023-10-14T16:59:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.019+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f153c1f59a0 con 0x7f153c0d0be0 2023-10-14T16:59:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.020+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f153c1f5be0 con 0x7f153c0d0be0 2023-10-14T16:59:22.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.021+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f152402f390 con 0x7f153c0d0be0 2023-10-14T16:59:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.022+0000 7f15397fa640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 0x7f151c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.022+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f15240ab790 con 0x7f153c0d0be0 2023-10-14T16:59:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.022+0000 7f1540e0b640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 0x7f151c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.023+0000 7f1540e0b640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 0x7f151c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f152c007fd0 tx=0x7f152c009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.027+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1524076930 con 0x7f153c0d0be0 2023-10-14T16:59:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.292+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f153c04e9b0 con 0x7f153c0d0be0 2023-10-14T16:59:22.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.294+0000 7f15397fa640 1 -- 172.21.15.29:0/2893661470 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v16) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f1524079f50 con 0x7f153c0d0be0 2023-10-14T16:59:22.295 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:22.295 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:59:22.295 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T16:59:22.295 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.295 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T16:59:22.296 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T16:59:22.296 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.296 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.296 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:59:22.296 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:59:22.297 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.297 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.297 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.297 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.297 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T16:59:22.298 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T16:59:22.298 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:59:22.298 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:59:22.299 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.299 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.299 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.299 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.299 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:59:22.300 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.301 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.301 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.301 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.301 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T16:59:22.301 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T16:59:22.302 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.302 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.302 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:59:22.302 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:59:22.302 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.303 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.303 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.303 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.303 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T16:59:22.303 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.304 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.305 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.305 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T16:59:22.305 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T16:59:22.305 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.305 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.306 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.306 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.306 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.306 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.306 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.307 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T16:59:22.307 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T16:59:22.307 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.307 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.307 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.308 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.308 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.308 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.308 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.308 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.309 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.310 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.310 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.310 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T16:59:22.310 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T16:59:22.310 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.311 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.311 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.311 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.311 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.311 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.312 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.313 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.313 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.313 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.313 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.313 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T16:59:22.314 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T16:59:22.314 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.314 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.314 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T16:59:22.315 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T16:59:22.316 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T16:59:22.316 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.316 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.316 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T16:59:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T16:59:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.317 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.318 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T16:59:22.318 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T16:59:22.318 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.318 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.318 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.319 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T16:59:22.319 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T16:59:22.319 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.319 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.319 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.320 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T16:59:22.321 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T16:59:22.321 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.321 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.321 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:59:22.321 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:59:22.322 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:59:22.322 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.322 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.322 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.322 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T16:59:22.323 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T16:59:22.324 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.324 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.324 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.324 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T16:59:22.324 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T16:59:22.325 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:59:22.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T16:59:22.325 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T16:59:22.325 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.326 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.326 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.326 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.326 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T16:59:22.326 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T16:59:22.327 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.327 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:59:22.327 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:59:22.327 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.327 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.328 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T16:59:22.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:22.328 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T16:59:22.328 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T16:59:22.328 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:22.329 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T16:59:22.330 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:59:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 msgr2=0x7f151c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 0x7f151c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f152c007fd0 tx=0x7f152c009040 comp rx=0 tx=0).stop 2023-10-14T16:59:22.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 msgr2=0x7f153c1f1c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 secure :-1 s=READY pgs=1375 cs=0 l=1 rev1=1 crypto rx=0x7f1524000c00 tx=0x7f1524004730 comp rx=0 tx=0).stop 2023-10-14T16:59:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 shutdown_connections 2023-10-14T16:59:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f151c06af90 0x7f151c06d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f153c15cd70 0x7f153c0b42d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f153c0d0be0 0x7f153c1f1c80 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 --2- 172.21.15.29:0/2893661470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f153c0b8480 0x7f153c0cc8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 >> 172.21.15.29:0/2893661470 conn(0x7f153c0cf570 msgr2=0x7f153c0b6e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 shutdown_connections 2023-10-14T16:59:22.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.296+0000 7f1541e0d640 1 -- 172.21.15.29:0/2893661470 wait complete. 2023-10-14T16:59:22.334 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.567+0000 7f519e789640 1 Processor -- start 2023-10-14T16:59:22.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.567+0000 7f519e789640 1 -- start start 2023-10-14T16:59:22.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f519e789640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51980cd0c0 0x7f51981615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f519e789640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980b0410 0x7f51980b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f519e789640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51980b0db0 con 0x7f519815d3f0 2023-10-14T16:59:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f519e789640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519815e3e0 con 0x7f51980cd0c0 2023-10-14T16:59:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f519e789640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980cd4a0 con 0x7f51980b0410 2023-10-14T16:59:22.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f5197fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51980cd0c0 0x7f51981615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f5197fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51980cd0c0 0x7f51981615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54500/0 (socket says 172.21.15.29:54500) 2023-10-14T16:59:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.569+0000 7f5197fff640 1 -- 172.21.15.29:0/2509735546 learned_addr learned my addr 172.21.15.29:0/2509735546 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.570+0000 7f51977fe640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980b0410 0x7f51980b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.570+0000 7f5197fff640 1 -- 172.21.15.29:0/2509735546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980b0410 msgr2=0x7f51980b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:22.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.570+0000 7f5197fff640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980b0410 0x7f51980b0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.570+0000 7f5197fff640 1 -- 172.21.15.29:0/2509735546 >> v1:172.21.15.186:6789/0 conn(0x7f519815d3f0 legacy=0x7f519815d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:22.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.570+0000 7f5197fff640 1 -- 172.21.15.29:0/2509735546 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519815e100 con 0x7f51980cd0c0 2023-10-14T16:59:22.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.571+0000 7f5197fff640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51980cd0c0 0x7f51981615f0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf3b476a3cb147b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.571+0000 7f5196ffd640 1 -- 172.21.15.29:0/2509735546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51880317c0 con 0x7f51980cd0c0 2023-10-14T16:59:22.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.571+0000 7f5196ffd640 1 -- 172.21.15.29:0/2509735546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188031920 con 0x7f51980cd0c0 2023-10-14T16:59:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.571+0000 7f5196ffd640 1 -- 172.21.15.29:0/2509735546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188031cb0 con 0x7f51980cd0c0 2023-10-14T16:59:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 -- 172.21.15.29:0/2509735546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51980cd0c0 msgr2=0x7f51981615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51980cd0c0 0x7f51981615f0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 crypto rx=0x7f5188009870 tx=0x7f518802b860 comp rx=0 tx=0).stop 2023-10-14T16:59:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 -- 172.21.15.29:0/2509735546 shutdown_connections 2023-10-14T16:59:22.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980b0410 0x7f51980b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 --2- 172.21.15.29:0/2509735546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51980cd0c0 0x7f51981615f0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 -- 172.21.15.29:0/2509735546 >> 172.21.15.29:0/2509735546 conn(0x7f51980cb410 msgr2=0x7f51980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:22.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 -- 172.21.15.29:0/2509735546 shutdown_connections 2023-10-14T16:59:22.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.572+0000 7f519e789640 1 -- 172.21.15.29:0/2509735546 wait complete. 2023-10-14T16:59:22.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.574+0000 7f519e789640 1 Processor -- start 2023-10-14T16:59:22.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.574+0000 7f519e789640 1 -- start start 2023-10-14T16:59:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51980b0410 0x7f5198155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980cd0c0 0x7f51980c94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51981566a0 con 0x7f519815d3f0 2023-10-14T16:59:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b4280 con 0x7f51980b0410 2023-10-14T16:59:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519e789640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51980b3fa0 con 0x7f51980cd0c0 2023-10-14T16:59:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.575+0000 7f519cd26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54506/0 (socket says 172.21.15.29:54506) 2023-10-14T16:59:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 -- 172.21.15.29:0/651985519 learned_addr learned my addr 172.21.15.29:0/651985519 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f5197fff640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51980b0410 0x7f5198155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f51977fe640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980cd0c0 0x7f51980c94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 -- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980cd0c0 msgr2=0x7f51980c94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980cd0c0 0x7f51980c94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 -- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51980b0410 msgr2=0x7f5198155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51980b0410 0x7f5198155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 -- 172.21.15.29:0/651985519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51880094d0 con 0x7f519815d3f0 2023-10-14T16:59:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.576+0000 7f519cd26640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f519000c780 tx=0x7f519000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.577+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5190004b60 con 0x7f519815d3f0 2023-10-14T16:59:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.577+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51981e63a0 con 0x7f519815d3f0 2023-10-14T16:59:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.577+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f519000ce90 con 0x7f519815d3f0 2023-10-14T16:59:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.577+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f519000a850 con 0x7f519815d3f0 2023-10-14T16:59:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.577+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51981e6960 con 0x7f519815d3f0 2023-10-14T16:59:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.578+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519804e160 con 0x7f519815d3f0 2023-10-14T16:59:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.579+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f519000a9b0 con 0x7f519815d3f0 2023-10-14T16:59:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.580+0000 7f5194ff9640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 0x7f518406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.580+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f5190089670 con 0x7f519815d3f0 2023-10-14T16:59:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.580+0000 7f5197fff640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 0x7f518406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.581+0000 7f5197fff640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 0x7f518406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5188036000 tx=0x7f518803d040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.584+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51900548a0 con 0x7f519815d3f0 2023-10-14T16:59:22.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.783+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51980c5820 con 0x7f519815d3f0 2023-10-14T16:59:22.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.785+0000 7f5194ff9640 1 -- 172.21.15.29:0/651985519 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v121) v1 ==== 74+0+12925 (secure 0 0 0) 0x7f5190057ec0 con 0x7f519815d3f0 2023-10-14T16:59:22.786 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:22.786 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":121,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:21.752785+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":13,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":111,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":107,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 msgr2=0x7f518406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 0x7f518406d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5188036000 tx=0x7f518803d040 comp rx=0 tx=0).stop 2023-10-14T16:59:22.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 msgr2=0x7f51980c9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:22.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 crypto rx=0x7f519000c780 tx=0x7f519000cc40 comp rx=0 tx=0).stop 2023-10-14T16:59:22.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 shutdown_connections 2023-10-14T16:59:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f518406af90 0x7f518406d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f519815d3f0 0x7f51980c9a10 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51980cd0c0 0x7f51980c94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 --2- 172.21.15.29:0/651985519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51980b0410 0x7f5198155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:22.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 >> 172.21.15.29:0/651985519 conn(0x7f51980cb410 msgr2=0x7f519815c340 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 shutdown_connections 2023-10-14T16:59:22.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:22.788+0000 7f519e789640 1 -- 172.21.15.29:0/651985519 wait complete. 2023-10-14T16:59:22.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T16:59:22.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T16:59:23.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.041+0000 7fe699ac2640 1 Processor -- start 2023-10-14T16:59:23.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.041+0000 7fe699ac2640 1 -- start start 2023-10-14T16:59:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe699ac2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe69415d150 0x7fe6940ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe699ac2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694160b90 0x7fe694160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe699ac2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6940ac890 con 0x7fe694069da0 2023-10-14T16:59:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe699ac2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe69415d530 con 0x7fe69415d150 2023-10-14T16:59:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe699ac2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe694161650 con 0x7fe694160b90 2023-10-14T16:59:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe693fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe694069da0 0x7fe69406a1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32986/0 (socket says 172.21.15.29:32986) 2023-10-14T16:59:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe693fff640 1 -- 172.21.15.29:0/494428146 learned_addr learned my addr 172.21.15.29:0/494428146 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe6937fe640 1 --2- 172.21.15.29:0/494428146 >> v2:172.21.15.29:3300/0 conn(0x7fe69415d150 0x7fe6940ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.043+0000 7fe68bfff640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694160b90 0x7fe694160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.044+0000 7fe6937fe640 1 -- 172.21.15.29:0/494428146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694160b90 msgr2=0x7fe694160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.044+0000 7fe6937fe640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694160b90 0x7fe694160fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.044+0000 7fe6937fe640 1 -- 172.21.15.29:0/494428146 >> v1:172.21.15.29:6789/0 conn(0x7fe694069da0 legacy=0x7fe69406a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.044+0000 7fe6937fe640 1 -- 172.21.15.29:0/494428146 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe694161930 con 0x7fe69415d150 2023-10-14T16:59:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.045+0000 7fe6937fe640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940ad350 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fcdedcdd237ec4b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.045+0000 7fe692ffd640 1 -- 172.21.15.29:0/494428146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe68402d040 con 0x7fe69415d150 2023-10-14T16:59:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.045+0000 7fe692ffd640 1 -- 172.21.15.29:0/494428146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe684031810 con 0x7fe69415d150 2023-10-14T16:59:23.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.045+0000 7fe692ffd640 1 -- 172.21.15.29:0/494428146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe684031ba0 con 0x7fe69415d150 2023-10-14T16:59:23.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 -- 172.21.15.29:0/494428146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 msgr2=0x7fe6940ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940ad350 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 crypto rx=0x7fe6840097a0 tx=0x7fe68402b8b0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 -- 172.21.15.29:0/494428146 shutdown_connections 2023-10-14T16:59:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694160b90 0x7fe694160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 --2- 172.21.15.29:0/494428146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940ad350 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 -- 172.21.15.29:0/494428146 >> 172.21.15.29:0/494428146 conn(0x7fe6940ce9b0 msgr2=0x7fe6940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 -- 172.21.15.29:0/494428146 shutdown_connections 2023-10-14T16:59:23.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.046+0000 7fe699ac2640 1 -- 172.21.15.29:0/494428146 wait complete. 2023-10-14T16:59:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.048+0000 7fe699ac2640 1 Processor -- start 2023-10-14T16:59:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.049+0000 7fe699ac2640 1 -- start start 2023-10-14T16:59:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694069da0 0x7fe69414ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe694160b90 0x7fe6940c80f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c7c60 con 0x7fe69415d150 2023-10-14T16:59:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c7020 con 0x7fe694160b90 2023-10-14T16:59:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe699ac2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6940c7300 con 0x7fe694069da0 2023-10-14T16:59:23.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe68bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe68bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54536/0 (socket says 172.21.15.29:54536) 2023-10-14T16:59:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe68bfff640 1 -- 172.21.15.29:0/106112075 learned_addr learned my addr 172.21.15.29:0/106112075 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.050+0000 7fe6937fe640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694069da0 0x7fe69414ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe693fff640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe694160b90 0x7fe6940c80f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 -- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694069da0 msgr2=0x7fe69414ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694069da0 0x7fe69414ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 -- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe694160b90 msgr2=0x7fe6940c80f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe694160b90 0x7fe6940c80f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 -- 172.21.15.29:0/106112075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe684009400 con 0x7fe69415d150 2023-10-14T16:59:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe68bfff640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00d5f0 tx=0x7fe67c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c0125f0 con 0x7fe69415d150 2023-10-14T16:59:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe67c012750 con 0x7fe69415d150 2023-10-14T16:59:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6940c7610 con 0x7fe69415d150 2023-10-14T16:59:23.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.051+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe67c012b00 con 0x7fe69415d150 2023-10-14T16:59:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.052+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe69414b2e0 con 0x7fe69415d150 2023-10-14T16:59:23.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.053+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe694066de0 con 0x7fe69415d150 2023-10-14T16:59:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.054+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe67c012c60 con 0x7fe69415d150 2023-10-14T16:59:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.055+0000 7fe690ff9640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 0x7fe67406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.055+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4901+0+0 (secure 0 0 0) 0x7fe67c089990 con 0x7fe69415d150 2023-10-14T16:59:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.055+0000 7fe6937fe640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 0x7fe67406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.056+0000 7fe6937fe640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 0x7fe67406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe68403c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.059+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe67c054bc0 con 0x7fe69415d150 2023-10-14T16:59:23.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.254+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe69404e160 con 0x7fe69415d150 2023-10-14T16:59:23.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.254+0000 7fe690ff9640 1 -- 172.21.15.29:0/106112075 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v121) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe67c0581e0 con 0x7fe69415d150 2023-10-14T16:59:23.255 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T16:59:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 msgr2=0x7fe67406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 0x7fe67406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe684036000 tx=0x7fe68403c040 comp rx=0 tx=0).stop 2023-10-14T16:59:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 msgr2=0x7fe6940c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 crypto rx=0x7fe67c00d5f0 tx=0x7fe67c00dab0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 shutdown_connections 2023-10-14T16:59:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe67406af90 0x7fe67406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe694160b90 0x7fe6940c80f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe69415d150 0x7fe6940c6460 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 --2- 172.21.15.29:0/106112075 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe694069da0 0x7fe69414ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.257+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 >> 172.21.15.29:0/106112075 conn(0x7fe6940ce9b0 msgr2=0x7fe6940b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.258+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 shutdown_connections 2023-10-14T16:59:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.258+0000 7fe699ac2640 1 -- 172.21.15.29:0/106112075 wait complete. 2023-10-14T16:59:23.277 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T16:59:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.510+0000 7f0c799e2640 1 Processor -- start 2023-10-14T16:59:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.510+0000 7f0c799e2640 1 -- start start 2023-10-14T16:59:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c799e2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c740ad970 0x7f0c740cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c799e2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0c740ccfb0 0x7f0c740cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c799e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c7415b9d0 con 0x7f0c7406c040 2023-10-14T16:59:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c799e2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7415bcb0 con 0x7f0c740ad970 2023-10-14T16:59:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c799e2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c7415bf90 con 0x7f0c740ccfb0 2023-10-14T16:59:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c73fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c740ad970 0x7f0c740cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c73fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0c740ad970 0x7f0c740cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54542/0 (socket says 172.21.15.29:54542) 2023-10-14T16:59:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.512+0000 7f0c73fff640 1 -- 172.21.15.29:0/1081136339 learned_addr learned my addr 172.21.15.29:0/1081136339 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.513+0000 7f0c791e1640 1 --2- 172.21.15.29:0/1081136339 >> v2:172.21.15.186:3300/0 conn(0x7f0c740ccfb0 0x7f0c740cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.513+0000 7f0c73fff640 1 -- 172.21.15.29:0/1081136339 >> v2:172.21.15.186:3300/0 conn(0x7f0c740ccfb0 msgr2=0x7f0c740cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.513+0000 7f0c73fff640 1 --2- 172.21.15.29:0/1081136339 >> v2:172.21.15.186:3300/0 conn(0x7f0c740ccfb0 0x7f0c740cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.513+0000 7f0c73fff640 1 -- 172.21.15.29:0/1081136339 >> v1:172.21.15.186:6789/0 conn(0x7f0c7406c040 legacy=0x7f0c7406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.513+0000 7f0c73fff640 1 -- 172.21.15.29:0/1081136339 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c740cc270 con 0x7f0c740ad970 2023-10-14T16:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.514+0000 7f0c73fff640 1 --2- 172.21.15.29:0/1081136339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ad970 0x7f0c740cca70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009870 tx=0x7f0c6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc20ba4940cc95f9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c737fe640 1 -- 172.21.15.29:0/1081136339 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60030860 con 0x7f0c740ad970 2023-10-14T16:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c737fe640 1 -- 172.21.15.29:0/1081136339 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c600309c0 con 0x7f0c740ad970 2023-10-14T16:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c737fe640 1 -- 172.21.15.29:0/1081136339 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c60030d50 con 0x7f0c740ad970 2023-10-14T16:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 -- 172.21.15.29:0/1081136339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ad970 msgr2=0x7f0c740cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 --2- 172.21.15.29:0/1081136339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ad970 0x7f0c740cca70 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 crypto rx=0x7f0c60009870 tx=0x7f0c6002b9a0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 -- 172.21.15.29:0/1081136339 shutdown_connections 2023-10-14T16:59:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 --2- 172.21.15.29:0/1081136339 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ad970 0x7f0c740cca70 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 --2- 172.21.15.29:0/1081136339 >> v2:172.21.15.186:3300/0 conn(0x7f0c740ccfb0 0x7f0c740cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 -- 172.21.15.29:0/1081136339 >> 172.21.15.29:0/1081136339 conn(0x7f0c740ce6a0 msgr2=0x7f0c740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.515+0000 7f0c799e2640 1 -- 172.21.15.29:0/1081136339 shutdown_connections 2023-10-14T16:59:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.516+0000 7f0c799e2640 1 -- 172.21.15.29:0/1081136339 wait complete. 2023-10-14T16:59:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.517+0000 7f0c799e2640 1 Processor -- start 2023-10-14T16:59:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.518+0000 7f0c799e2640 1 -- start start 2023-10-14T16:59:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.518+0000 7f0c799e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 0x7f0c740b3510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c799e2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 0x7f0c740c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c799e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c799e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740b1120 con 0x7f0c740ccfb0 2023-10-14T16:59:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c799e2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c740b1640 con 0x7f0c7406c040 2023-10-14T16:59:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c799e2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7414e450 con 0x7f0c740ad970 2023-10-14T16:59:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54544/0 (socket says 172.21.15.29:54544) 2023-10-14T16:59:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 -- 172.21.15.29:0/977524097 learned_addr learned my addr 172.21.15.29:0/977524097 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c789e0640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 0x7f0c740b3510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c73fff640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 0x7f0c740c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 -- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 msgr2=0x7f0c740c7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 0x7f0c740c7a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 -- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 msgr2=0x7f0c740b3510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 0x7f0c740b3510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c791e1640 1 -- 172.21.15.29:0/977524097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c600094d0 con 0x7f0c740ccfb0 2023-10-14T16:59:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c73fff640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 0x7f0c740c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.519+0000 7f0c789e0640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 0x7f0c740b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c791e1640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007e60 tx=0x7f0c6c00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c014420 con 0x7f0c740ccfb0 2023-10-14T16:59:23.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c6c0145a0 con 0x7f0c740ccfb0 2023-10-14T16:59:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7414ea10 con 0x7f0c740ccfb0 2023-10-14T16:59:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c6c014950 con 0x7f0c740ccfb0 2023-10-14T16:59:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.520+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c740b4040 con 0x7f0c740ccfb0 2023-10-14T16:59:23.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.521+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c7406b330 con 0x7f0c740ccfb0 2023-10-14T16:59:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.522+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c6c014ab0 con 0x7f0c740ccfb0 2023-10-14T16:59:23.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.523+0000 7f0c717fa640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 0x7f0c5806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.523+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f0c6c089df0 con 0x7f0c740ccfb0 2023-10-14T16:59:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.524+0000 7f0c789e0640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 0x7f0c5806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.524+0000 7f0c789e0640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 0x7f0c5806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c68000f60 tx=0x7f0c68009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.528+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0c6c055010 con 0x7f0c740ccfb0 2023-10-14T16:59:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.724+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0c7414eba0 con 0x7f0c740ccfb0 2023-10-14T16:59:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.725+0000 7f0c717fa640 1 -- 172.21.15.29:0/977524097 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v121) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0c6c058630 con 0x7f0c740ccfb0 2023-10-14T16:59:23.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:23.726 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:59:23.726 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:23.727 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T16:59:23.727 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T16:59:23.727 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:23.727 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:59:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 msgr2=0x7f0c5806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 0x7f0c5806d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c68000f60 tx=0x7f0c68009040 comp rx=0 tx=0).stop 2023-10-14T16:59:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 msgr2=0x7f0c740c7fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 crypto rx=0x7f0c6c007e60 tx=0x7f0c6c00e4b0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 shutdown_connections 2023-10-14T16:59:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f0c5806af90 0x7f0c5806d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c740ccfb0 0x7f0c740c7fb0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c740ad970 0x7f0c740c7a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 --2- 172.21.15.29:0/977524097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c7406c040 0x7f0c740b3510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 >> 172.21.15.29:0/977524097 conn(0x7f0c740ce6a0 msgr2=0x7f0c7415c270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 shutdown_connections 2023-10-14T16:59:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.728+0000 7f0c799e2640 1 -- 172.21.15.29:0/977524097 wait complete. 2023-10-14T16:59:23.749 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T16:59:23.750 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T16:59:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.987+0000 7f419496a640 1 Processor -- start 2023-10-14T16:59:23.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.987+0000 7f419496a640 1 -- start start 2023-10-14T16:59:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.988+0000 7f419496a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f419006cb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f419496a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41900ca310 0x7f419006d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:23.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f419496a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41900c8e70 con 0x7f41900cae40 2023-10-14T16:59:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f419496a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41900b4170 con 0x7f41900ca310 2023-10-14T16:59:23.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f419496a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41900adde0 con 0x7f419015ad80 2023-10-14T16:59:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f418ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41900ca310 0x7f419006d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f418ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41900ca310 0x7f419006d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54560/0 (socket says 172.21.15.29:54560) 2023-10-14T16:59:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f418ffff640 1 -- 172.21.15.29:0/3649211970 learned_addr learned my addr 172.21.15.29:0/3649211970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.989+0000 7f418f7fe640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f419006cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.990+0000 7f418ffff640 1 -- 172.21.15.29:0/3649211970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 msgr2=0x7f419006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.990+0000 7f418ffff640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f419006cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.990+0000 7f418ffff640 1 -- 172.21.15.29:0/3649211970 >> v1:172.21.15.186:6789/0 conn(0x7f41900cae40 legacy=0x7f419006b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.990+0000 7f418ffff640 1 -- 172.21.15.29:0/3649211970 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41900b4410 con 0x7f41900ca310 2023-10-14T16:59:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.991+0000 7f418ffff640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f419006d0b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4188009870 tx=0x7f418802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abebc4a457c9c297 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:23.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.991+0000 7f418effd640 1 -- 172.21.15.29:0/3649211970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188030840 con 0x7f41900ca310 2023-10-14T16:59:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.991+0000 7f418effd640 1 -- 172.21.15.29:0/3649211970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41880309a0 con 0x7f41900ca310 2023-10-14T16:59:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.991+0000 7f418effd640 1 -- 172.21.15.29:0/3649211970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188030d30 con 0x7f41900ca310 2023-10-14T16:59:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 -- 172.21.15.29:0/3649211970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 msgr2=0x7f419006d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f419006d0b0 secure :-1 s=READY pgs=1383 cs=0 l=1 rev1=1 crypto rx=0x7f4188009870 tx=0x7f418802b980 comp rx=0 tx=0).stop 2023-10-14T16:59:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 -- 172.21.15.29:0/3649211970 shutdown_connections 2023-10-14T16:59:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f419006d0b0 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 --2- 172.21.15.29:0/3649211970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f419006cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 -- 172.21.15.29:0/3649211970 >> 172.21.15.29:0/3649211970 conn(0x7f41900ce820 msgr2=0x7f41900cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 -- 172.21.15.29:0/3649211970 shutdown_connections 2023-10-14T16:59:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.992+0000 7f419496a640 1 -- 172.21.15.29:0/3649211970 wait complete. 2023-10-14T16:59:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.994+0000 7f419496a640 1 Processor -- start 2023-10-14T16:59:24.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.994+0000 7f419496a640 1 -- start start 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 0x7f419014b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f41901f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419015ddc0 con 0x7f41900ca310 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f419015e070 con 0x7f41900cae40 2023-10-14T16:59:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.995+0000 7f419496a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f419015e350 con 0x7f419015ad80 2023-10-14T16:59:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54574/0 (socket says 172.21.15.29:54574) 2023-10-14T16:59:24.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 -- 172.21.15.29:0/703087522 learned_addr learned my addr 172.21.15.29:0/703087522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f4186ffd640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 0x7f419014b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 -- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 msgr2=0x7f41901f1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:24.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f41901f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 -- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 msgr2=0x7f419014b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 0x7f419014b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 -- 172.21.15.29:0/703087522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41880094d0 con 0x7f41900ca310 2023-10-14T16:59:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f4186ffd640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 0x7f419014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:24.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.996+0000 7f418f7fe640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4180007e60 tx=0x7f418000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.997+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4180014620 con 0x7f41900ca310 2023-10-14T16:59:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.997+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f418000ee90 con 0x7f41900ca310 2023-10-14T16:59:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.997+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41800149e0 con 0x7f41900ca310 2023-10-14T16:59:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.997+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4190157e70 con 0x7f41900ca310 2023-10-14T16:59:24.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.997+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41901f2080 con 0x7f41900ca310 2023-10-14T16:59:24.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.998+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4190158390 con 0x7f41900ca310 2023-10-14T16:59:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:23.999+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4180014b40 con 0x7f41900ca310 2023-10-14T16:59:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.000+0000 7f418cff9640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 0x7f417006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.000+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 4901+0+0 (secure 0 0 0) 0x7f4180089ee0 con 0x7f41900ca310 2023-10-14T16:59:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.000+0000 7f4186ffd640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 0x7f417006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.001+0000 7f4186ffd640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 0x7f417006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41900cf370 tx=0x7f4178009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:24.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.004+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4190158390 con 0x7f41900ca310 2023-10-14T16:59:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.202+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f4190158390 con 0x7f41900ca310 2023-10-14T16:59:24.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.786+0000 7f418cff9640 1 -- 172.21.15.29:0/703087522 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v122) v1 ==== 138+0+0 (secure 0 0 0) 0x7f4180058b90 con 0x7f41900ca310 2023-10-14T16:59:24.787 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T16:59:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 msgr2=0x7f417006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 0x7f417006d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f41900cf370 tx=0x7f4178009040 comp rx=0 tx=0).stop 2023-10-14T16:59:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 msgr2=0x7f4190148230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:24.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 crypto rx=0x7f4180007e60 tx=0x7f418000e4b0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 shutdown_connections 2023-10-14T16:59:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f417006af90 0x7f417006d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f419015ad80 0x7f41901f1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41900cae40 0x7f419014b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 --2- 172.21.15.29:0/703087522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41900ca310 0x7f4190148230 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 >> 172.21.15.29:0/703087522 conn(0x7f41900ce820 msgr2=0x7f41900aeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 shutdown_connections 2023-10-14T16:59:24.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:24.790+0000 7f419496a640 1 -- 172.21.15.29:0/703087522 wait complete. 2023-10-14T16:59:24.819 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T16:59:25.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.043+0000 7effc155e640 1 Processor -- start 2023-10-14T16:59:25.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.044+0000 7effc155e640 1 -- start start 2023-10-14T16:59:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.045+0000 7effc155e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.045+0000 7effc155e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effbc0c9230 0x7effbc070bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.045+0000 7effc155e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effbc070760 con 0x7effbc071130 2023-10-14T16:59:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.045+0000 7effc155e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effbc0715c0 con 0x7effbc0c9230 2023-10-14T16:59:25.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.045+0000 7effc155e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0cdac0 con 0x7effbc15dc90 2023-10-14T16:59:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.046+0000 7effbb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effbc0c9230 0x7effbc070bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.046+0000 7effbbfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.046+0000 7effbb7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7effbc0c9230 0x7effbc070bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:59524/0 (socket says 172.21.15.29:59524) 2023-10-14T16:59:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.046+0000 7effbb7fe640 1 -- 172.21.15.29:0/1146373153 learned_addr learned my addr 172.21.15.29:0/1146373153 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.047+0000 7effbb7fe640 1 -- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 msgr2=0x7effbc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.047+0000 7effbb7fe640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.047+0000 7effbb7fe640 1 -- 172.21.15.29:0/1146373153 >> v1:172.21.15.186:6789/0 conn(0x7effbc071130 legacy=0x7effbc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.047+0000 7effbb7fe640 1 -- 172.21.15.29:0/1146373153 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effbc070110 con 0x7effbc0c9230 2023-10-14T16:59:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.047+0000 7effbbfff640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:59:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.049+0000 7effbb7fe640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc070bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7effb0009870 tx=0x7effb002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a983362524b0160 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.049+0000 7effbaffd640 1 -- 172.21.15.29:0/1146373153 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0030840 con 0x7effbc0c9230 2023-10-14T16:59:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.049+0000 7effbaffd640 1 -- 172.21.15.29:0/1146373153 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effb00309a0 con 0x7effbc0c9230 2023-10-14T16:59:25.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.049+0000 7effbaffd640 1 -- 172.21.15.29:0/1146373153 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effb0030d30 con 0x7effbc0c9230 2023-10-14T16:59:25.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 -- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 msgr2=0x7effbc070bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc070bf0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7effb0009870 tx=0x7effb002b980 comp rx=0 tx=0).stop 2023-10-14T16:59:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 -- 172.21.15.29:0/1146373153 shutdown_connections 2023-10-14T16:59:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc070bf0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 --2- 172.21.15.29:0/1146373153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 -- 172.21.15.29:0/1146373153 >> 172.21.15.29:0/1146373153 conn(0x7effbc0ce6a0 msgr2=0x7effbc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.050+0000 7effc155e640 1 -- 172.21.15.29:0/1146373153 shutdown_connections 2023-10-14T16:59:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.051+0000 7effc155e640 1 -- 172.21.15.29:0/1146373153 wait complete. 2023-10-14T16:59:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.052+0000 7effc155e640 1 Processor -- start 2023-10-14T16:59:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.052+0000 7effc155e640 1 -- start start 2023-10-14T16:59:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc0c44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc1448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b12c0 con 0x7effbc071130 2023-10-14T16:59:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0c1e50 con 0x7effbc0c9230 2023-10-14T16:59:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.053+0000 7effc155e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effbc0b1580 con 0x7effbc15dc90 2023-10-14T16:59:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54590/0 (socket says 172.21.15.29:54590) 2023-10-14T16:59:25.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 -- 172.21.15.29:0/679984767 learned_addr learned my addr 172.21.15.29:0/679984767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:25.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbb7fe640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc0c44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effc0d5d640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc1448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 -- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 msgr2=0x7effbc1448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc1448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 -- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 msgr2=0x7effbc0c44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc0c44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effbbfff640 1 -- 172.21.15.29:0/679984767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac009a70 con 0x7effbc071130 2023-10-14T16:59:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.054+0000 7effc0d5d640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc1448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effbbfff640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7effac00efc0 tx=0x7effac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac014680 con 0x7effbc071130 2023-10-14T16:59:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac00f040 con 0x7effbc071130 2023-10-14T16:59:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effac014a00 con 0x7effbc071130 2023-10-14T16:59:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb00094d0 con 0x7effbc071130 2023-10-14T16:59:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.055+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effbc0c3fb0 con 0x7effbc071130 2023-10-14T16:59:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.056+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effbc0c4140 con 0x7effbc071130 2023-10-14T16:59:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.058+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7effac014b60 con 0x7effbc071130 2023-10-14T16:59:25.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.060+0000 7effb8ff9640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 0x7effa806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.060+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 5271+0+0 (secure 0 0 0) 0x7effac08abf0 con 0x7effbc071130 2023-10-14T16:59:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.060+0000 7effbb7fe640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 0x7effa806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:25.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.061+0000 7effbb7fe640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 0x7effa806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effb0036000 tx=0x7effb0008030 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.064+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7effac055ca0 con 0x7effbc071130 2023-10-14T16:59:25.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.266+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7effbc04e9b0 con 0x7effbc071130 2023-10-14T16:59:25.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.793+0000 7effb8ff9640 1 -- 172.21.15.29:0/679984767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v123) v1 ==== 170+0+0 (secure 0 0 0) 0x7effac0592c0 con 0x7effbc071130 2023-10-14T16:59:25.794 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T16:59:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 msgr2=0x7effa806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 0x7effa806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7effb0036000 tx=0x7effb0008030 comp rx=0 tx=0).stop 2023-10-14T16:59:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 msgr2=0x7effbc0c1910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 crypto rx=0x7effac00efc0 tx=0x7effac00c490 comp rx=0 tx=0).stop 2023-10-14T16:59:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 shutdown_connections 2023-10-14T16:59:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7effa806af90 0x7effa806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7effbc15dc90 0x7effbc1448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7effbc0c9230 0x7effbc0c44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 --2- 172.21.15.29:0/679984767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7effbc071130 0x7effbc0c1910 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.796+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 >> 172.21.15.29:0/679984767 conn(0x7effbc0ce6a0 msgr2=0x7effbc152d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.797+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 shutdown_connections 2023-10-14T16:59:25.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:25.797+0000 7effc155e640 1 -- 172.21.15.29:0/679984767 wait complete. 2023-10-14T16:59:25.837 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T16:59:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.027+0000 7fcc3afe3640 1 Processor -- start 2023-10-14T16:59:26.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.027+0000 7fcc3afe3640 1 -- start start 2023-10-14T16:59:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.029+0000 7fcc3afe3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 0x7fcc340caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.029+0000 7fcc3afe3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 0x7fcc34069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.029+0000 7fcc3afe3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc340cb140 con 0x7fcc3406a2e0 2023-10-14T16:59:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.029+0000 7fcc3afe3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc34161a50 con 0x7fcc340addb0 2023-10-14T16:59:26.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.029+0000 7fcc3afe3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc3406a750 con 0x7fcc340c9000 2023-10-14T16:59:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc38d7f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 0x7fcc340caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc33fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 0x7fcc34069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc38d7f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 0x7fcc340caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:59532/0 (socket says 172.21.15.29:59532) 2023-10-14T16:59:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc39580640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcc3406a2e0 0x7fcc340cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:42794/0 (socket says 172.21.15.29:42794) 2023-10-14T16:59:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc38d7f640 1 -- 172.21.15.29:0/977163264 learned_addr learned my addr 172.21.15.29:0/977163264 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc337fe640 1 -- 172.21.15.29:0/977163264 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858644775 0 0) 0x7fcc340cb140 con 0x7fcc3406a2e0 2023-10-14T16:59:26.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.030+0000 7fcc337fe640 1 -- 172.21.15.29:0/977163264 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc1c003520 con 0x7fcc3406a2e0 2023-10-14T16:59:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.031+0000 7fcc33fff640 1 -- 172.21.15.29:0/977163264 >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 msgr2=0x7fcc340caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.031+0000 7fcc33fff640 1 --2- 172.21.15.29:0/977163264 >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 0x7fcc340caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.031+0000 7fcc33fff640 1 -- 172.21.15.29:0/977163264 >> v1:172.21.15.186:6789/0 conn(0x7fcc3406a2e0 legacy=0x7fcc340cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.031+0000 7fcc33fff640 1 -- 172.21.15.29:0/977163264 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc3406b030 con 0x7fcc340c9000 2023-10-14T16:59:26.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.032+0000 7fcc33fff640 1 --2- 172.21.15.29:0/977163264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 0x7fcc34069da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=96ba6958f5cd64e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.032+0000 7fcc337fe640 1 -- 172.21.15.29:0/977163264 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2402d040 con 0x7fcc340c9000 2023-10-14T16:59:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc337fe640 1 -- 172.21.15.29:0/977163264 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc24031810 con 0x7fcc340c9000 2023-10-14T16:59:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc337fe640 1 -- 172.21.15.29:0/977163264 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc24031ba0 con 0x7fcc340c9000 2023-10-14T16:59:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 -- 172.21.15.29:0/977163264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 msgr2=0x7fcc34069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/977163264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 0x7fcc34069da0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2402b8b0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 -- 172.21.15.29:0/977163264 shutdown_connections 2023-10-14T16:59:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/977163264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340c9000 0x7fcc34069da0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/977163264 >> v2:172.21.15.186:3300/0 conn(0x7fcc340addb0 0x7fcc340caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 -- 172.21.15.29:0/977163264 >> 172.21.15.29:0/977163264 conn(0x7fcc340ce9b0 msgr2=0x7fcc340cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:26.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 -- 172.21.15.29:0/977163264 shutdown_connections 2023-10-14T16:59:26.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.033+0000 7fcc3afe3640 1 -- 172.21.15.29:0/977163264 wait complete. 2023-10-14T16:59:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.035+0000 7fcc3afe3640 1 Processor -- start 2023-10-14T16:59:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.035+0000 7fcc3afe3640 1 -- start start 2023-10-14T16:59:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340addb0 0x7fcc340c6520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc340c9000 0x7fcc340c6a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340c4600 con 0x7fcc3406a2e0 2023-10-14T16:59:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc341644f0 con 0x7fcc340c9000 2023-10-14T16:59:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.036+0000 7fcc3afe3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc340c39f0 con 0x7fcc340addb0 2023-10-14T16:59:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54598/0 (socket says 172.21.15.29:54598) 2023-10-14T16:59:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 -- 172.21.15.29:0/807061064 learned_addr learned my addr 172.21.15.29:0/807061064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc39580640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc340c9000 0x7fcc340c6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc33fff640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340addb0 0x7fcc340c6520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 -- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340addb0 msgr2=0x7fcc340c6520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340addb0 0x7fcc340c6520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 -- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc340c9000 msgr2=0x7fcc340c6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc340c9000 0x7fcc340c6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.037+0000 7fcc38d7f640 1 -- 172.21.15.29:0/807061064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc24009400 con 0x7fcc3406a2e0 2023-10-14T16:59:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc38d7f640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009fd0 tx=0x7fcc2800ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:26.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800c940 con 0x7fcc3406a2e0 2023-10-14T16:59:26.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc2800caa0 con 0x7fcc3406a2e0 2023-10-14T16:59:26.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc2800ce30 con 0x7fcc3406a2e0 2023-10-14T16:59:26.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc340c3cd0 con 0x7fcc3406a2e0 2023-10-14T16:59:26.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.038+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc340c4290 con 0x7fcc3406a2e0 2023-10-14T16:59:26.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.040+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc3404e160 con 0x7fcc3406a2e0 2023-10-14T16:59:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.042+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcc28013470 con 0x7fcc3406a2e0 2023-10-14T16:59:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.043+0000 7fcc317fa640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 0x7fcc2006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.043+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 5653+0+0 (secure 0 0 0) 0x7fcc2808a170 con 0x7fcc3406a2e0 2023-10-14T16:59:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.043+0000 7fcc33fff640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 0x7fcc2006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:26.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.044+0000 7fcc33fff640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 0x7fcc2006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2403c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:26.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.048+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcc28055120 con 0x7fcc3406a2e0 2023-10-14T16:59:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.280+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fcc3406c5b0 con 0x7fcc3406a2e0 2023-10-14T16:59:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.822+0000 7fcc317fa640 1 -- 172.21.15.29:0/807061064 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 14 and data pool 15 v53) v1 ==== 174+0+0 (secure 0 0 0) 0x7fcc28058740 con 0x7fcc3406a2e0 2023-10-14T16:59:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 14 and data pool 15 2023-10-14T16:59:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.839+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 msgr2=0x7fcc2006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.839+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 0x7fcc2006d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fcc240097a0 tx=0x7fcc2403c040 comp rx=0 tx=0).stop 2023-10-14T16:59:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.839+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 msgr2=0x7fcc340c4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.839+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 secure :-1 s=READY pgs=1386 cs=0 l=1 rev1=1 crypto rx=0x7fcc28009fd0 tx=0x7fcc2800ed40 comp rx=0 tx=0).stop 2023-10-14T16:59:26.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 shutdown_connections 2023-10-14T16:59:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fcc2006af90 0x7fcc2006d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc340c9000 0x7fcc340c6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc340addb0 0x7fcc340c6520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 --2- 172.21.15.29:0/807061064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc3406a2e0 0x7fcc340c4a90 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:26.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.840+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 >> 172.21.15.29:0/807061064 conn(0x7fcc340ce9b0 msgr2=0x7fcc340af3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.841+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 shutdown_connections 2023-10-14T16:59:26.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:26.841+0000 7fcc3afe3640 1 -- 172.21.15.29:0/807061064 wait complete. 2023-10-14T16:59:26.870 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.052+0000 7f56aeef2640 1 Processor -- start 2023-10-14T16:59:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.052+0000 7f56aeef2640 1 -- start start 2023-10-14T16:59:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56aeef2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56a80b0b20 0x7f56a81615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56aeef2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a815e3e0 con 0x7f56a80cd0c0 2023-10-14T16:59:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56aeef2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56a80cd4a0 con 0x7f56a815d3f0 2023-10-14T16:59:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56aeef2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56a815e100 con 0x7f56a80b0b20 2023-10-14T16:59:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56acc8e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56a80cd0c0 0x7f56a80b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:32988/0 (socket says 172.21.15.29:32988) 2023-10-14T16:59:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.054+0000 7f56acc8e640 1 -- 172.21.15.29:0/2211970847 learned_addr learned my addr 172.21.15.29:0/2211970847 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a7fff640 1 --2- 172.21.15.29:0/2211970847 >> v2:172.21.15.186:3300/0 conn(0x7f56a80b0b20 0x7f56a81615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3263933084 0 0) 0x7f56a815e3e0 con 0x7f56a80cd0c0 2023-10-14T16:59:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5690003520 con 0x7f56a80cd0c0 2023-10-14T16:59:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 568232119 0 0) 0x7f5690003520 con 0x7f56a80cd0c0 2023-10-14T16:59:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 >> v2:172.21.15.186:3300/0 conn(0x7f56a80b0b20 msgr2=0x7f56a81615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 --2- 172.21.15.29:0/2211970847 >> v2:172.21.15.186:3300/0 conn(0x7f56a80b0b20 0x7f56a81615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 >> v1:172.21.15.186:6789/0 conn(0x7f56a815d3f0 legacy=0x7f56a815d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a8160b10 con 0x7f56a80cd0c0 2023-10-14T16:59:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f569c001670 con 0x7f56a80cd0c0 2023-10-14T16:59:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 >> v1:172.21.15.29:6789/0 conn(0x7f56a80cd0c0 legacy=0x7f56a80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 0x7f5690003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56900047c0 0x7f5690004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690005180 0x7f569000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a815e100 con 0x7f5690003970 2023-10-14T16:59:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f569000c120 con 0x7f56900047c0 2023-10-14T16:59:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.055+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f569000c2e0 con 0x7f5690005180 2023-10-14T16:59:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 0x7f5690003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56a7fff640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56900047c0 0x7f5690004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 -- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690005180 msgr2=0x7f569000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690005180 0x7f569000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 -- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56900047c0 msgr2=0x7f5690004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56900047c0 0x7f5690004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56acc8e640 1 -- 172.21.15.29:0/2211970847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690003520 con 0x7f5690003970 2023-10-14T16:59:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.056+0000 7f56ad48f640 1 -- 172.21.15.29:0/2211970847 reap_dead start 2023-10-14T16:59:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.057+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 0x7f5690003d90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c240 tx=0x7f56900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea9577264b32f4a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.057+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c005ad0 con 0x7f5690003970 2023-10-14T16:59:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.057+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f569c005c30 con 0x7f5690003970 2023-10-14T16:59:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.057+0000 7f56a77fe640 1 -- 172.21.15.29:0/2211970847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569c02caf0 con 0x7f5690003970 2023-10-14T16:59:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 -- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 msgr2=0x7f5690003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 0x7f5690003d90 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 crypto rx=0x7f569c00c240 tx=0x7f56900057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 -- 172.21.15.29:0/2211970847 shutdown_connections 2023-10-14T16:59:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2211970847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5690003970 0x7f5690003d90 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 -- 172.21.15.29:0/2211970847 >> 172.21.15.29:0/2211970847 conn(0x7f56a80cb410 msgr2=0x7f56a80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 -- 172.21.15.29:0/2211970847 shutdown_connections 2023-10-14T16:59:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.058+0000 7f56aeef2640 1 -- 172.21.15.29:0/2211970847 wait complete. 2023-10-14T16:59:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.060+0000 7f56aeef2640 1 Processor -- start 2023-10-14T16:59:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.060+0000 7f56aeef2640 1 -- start start 2023-10-14T16:59:27.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690003970 0x7f56a80c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56a806cae0 0x7f56a81f5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a806cf90 con 0x7f56a80c8760 2023-10-14T16:59:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80c7d90 con 0x7f56a806cae0 2023-10-14T16:59:27.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.061+0000 7f56aeef2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56a80c5a60 con 0x7f5690003970 2023-10-14T16:59:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54616/0 (socket says 172.21.15.29:54616) 2023-10-14T16:59:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 -- 172.21.15.29:0/2147104874 learned_addr learned my addr 172.21.15.29:0/2147104874 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690003970 0x7f56a80c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 -- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690003970 msgr2=0x7f56a80c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56ad48f640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56a806cae0 0x7f56a81f5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690003970 0x7f56a80c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 -- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56a806cae0 msgr2=0x7f56a81f5dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56a806cae0 0x7f56a81f5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 -- 172.21.15.29:0/2147104874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56a80c7780 con 0x7f56a80c8760 2023-10-14T16:59:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.062+0000 7f56a7fff640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f56980107c0 tx=0x7f5698010c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.063+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5698009630 con 0x7f56a80c8760 2023-10-14T16:59:27.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.063+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5698010ed0 con 0x7f56a80c8760 2023-10-14T16:59:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.063+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f569800ed80 con 0x7f56a80c8760 2023-10-14T16:59:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.063+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a81f2fa0 con 0x7f56a80c8760 2023-10-14T16:59:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.063+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a80c5fe0 con 0x7f56a80c8760 2023-10-14T16:59:27.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.064+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a804e160 con 0x7f56a80c8760 2023-10-14T16:59:27.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.065+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f569801c400 con 0x7f56a80c8760 2023-10-14T16:59:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.066+0000 7f56a57fa640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 0x7f569406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.066+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5698089be0 con 0x7f56a80c8760 2023-10-14T16:59:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.066+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 0x7f569406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.067+0000 7f56acc8e640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 0x7f569406d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f569c034000 tx=0x7f569c009030 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.071+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5698054b50 con 0x7f56a80c8760 2023-10-14T16:59:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.270+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56a80b4910 con 0x7f56a80c8760 2023-10-14T16:59:27.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.273+0000 7f56a57fa640 1 -- 172.21.15.29:0/2147104874 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16590 (secure 0 0 0) 0x7f5698058170 con 0x7f56a80c8760 2023-10-14T16:59:27.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:27.275 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":124,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:26.282267+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 msgr2=0x7f569406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 0x7f569406d480 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f569c034000 tx=0x7f569c009030 comp rx=0 tx=0).stop 2023-10-14T16:59:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 msgr2=0x7f56a806c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 secure :-1 s=READY pgs=1389 cs=0 l=1 rev1=1 crypto rx=0x7f56980107c0 tx=0x7f5698010c80 comp rx=0 tx=0).stop 2023-10-14T16:59:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 shutdown_connections 2023-10-14T16:59:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f569406afc0 0x7f569406d480 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56a806cae0 0x7f56a81f5dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56a80c8760 0x7f56a806c5a0 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 --2- 172.21.15.29:0/2147104874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5690003970 0x7f56a80c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 >> 172.21.15.29:0/2147104874 conn(0x7f56a80cb410 msgr2=0x7f56a815aff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.276+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 shutdown_connections 2023-10-14T16:59:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.277+0000 7f56aeef2640 1 -- 172.21.15.29:0/2147104874 wait complete. 2023-10-14T16:59:27.296 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.523+0000 7fdf9b37a640 1 Processor -- start 2023-10-14T16:59:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.523+0000 7fdf9b37a640 1 -- start start 2023-10-14T16:59:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9b37a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 0x7fdf940ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9b37a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf940ca560 con 0x7fdf940c9060 2023-10-14T16:59:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9b37a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf940c9950 con 0x7fdf9406fb50 2023-10-14T16:59:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9b37a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940c9c30 con 0x7fdf9406c9e0 2023-10-14T16:59:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9a378640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdf940c9060 0x7fdf9406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33004/0 (socket says 172.21.15.29:33004) 2023-10-14T16:59:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.525+0000 7fdf9a378640 1 -- 172.21.15.29:0/3625866152 learned_addr learned my addr 172.21.15.29:0/3625866152 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf9ab79640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 0x7fdf940ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2257810627 0 0) 0x7fdf940ca560 con 0x7fdf940c9060 2023-10-14T16:59:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf80003520 con 0x7fdf940c9060 2023-10-14T16:59:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47885893 0 0) 0x7fdf80003520 con 0x7fdf940c9060 2023-10-14T16:59:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 msgr2=0x7fdf940ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 0x7fdf940ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 >> v1:172.21.15.186:6789/0 conn(0x7fdf9406fb50 legacy=0x7fdf940cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940c9f10 con 0x7fdf940c9060 2023-10-14T16:59:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdf90001670 con 0x7fdf940c9060 2023-10-14T16:59:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 >> v1:172.21.15.29:6789/0 conn(0x7fdf940c9060 legacy=0x7fdf9406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800039a0 0x7fdf80003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf800047c0 0x7fdf80004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 0x7fdf80005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940c9c30 con 0x7fdf800047c0 2023-10-14T16:59:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8000ba60 con 0x7fdf800039a0 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.526+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf8000bc20 con 0x7fdf9406c9e0 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf800047c0 0x7fdf80004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 -- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 msgr2=0x7fdf80005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf9406c9e0 0x7fdf80005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf9a378640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800039a0 0x7fdf80003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 -- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800039a0 msgr2=0x7fdf80003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800039a0 0x7fdf80003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf99b77640 1 -- 172.21.15.29:0/3625866152 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf80003520 con 0x7fdf800047c0 2023-10-14T16:59:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.527+0000 7fdf9a378640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800039a0 0x7fdf80003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.528+0000 7fdf99b77640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf800047c0 0x7fdf80004c40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdf84001720 tx=0x7fdf800057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4650976d90f67a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.528+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf840052d0 con 0x7fdf800047c0 2023-10-14T16:59:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.528+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8402e050 con 0x7fdf800047c0 2023-10-14T16:59:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.528+0000 7fdf99376640 1 -- 172.21.15.29:0/3625866152 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf840035f0 con 0x7fdf800047c0 2023-10-14T16:59:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 -- 172.21.15.29:0/3625866152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf800047c0 msgr2=0x7fdf80004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/3625866152 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf800047c0 0x7fdf80004c40 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 crypto rx=0x7fdf84001720 tx=0x7fdf800057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9ab79640 1 -- 172.21.15.29:0/3625866152 reap_dead start 2023-10-14T16:59:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 -- 172.21.15.29:0/3625866152 shutdown_connections 2023-10-14T16:59:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 -- 172.21.15.29:0/3625866152 >> 172.21.15.29:0/3625866152 conn(0x7fdf940ce6a0 msgr2=0x7fdf940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 -- 172.21.15.29:0/3625866152 shutdown_connections 2023-10-14T16:59:27.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.529+0000 7fdf9b37a640 1 -- 172.21.15.29:0/3625866152 wait complete. 2023-10-14T16:59:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.531+0000 7fdf9b37a640 1 Processor -- start 2023-10-14T16:59:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.531+0000 7fdf9b37a640 1 -- start start 2023-10-14T16:59:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800047c0 0x7fdf94156de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf941faac0 0x7fdf94162e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf941fb030 con 0x7fdf94162730 2023-10-14T16:59:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940b12c0 con 0x7fdf800047c0 2023-10-14T16:59:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9b37a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf940c5e70 con 0x7fdf941faac0 2023-10-14T16:59:27.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9ab79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.532+0000 7fdf9ab79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54648/0 (socket says 172.21.15.29:54648) 2023-10-14T16:59:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 -- 172.21.15.29:0/1468761752 learned_addr learned my addr 172.21.15.29:0/1468761752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9a378640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800047c0 0x7fdf94156de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf99b77640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf941faac0 0x7fdf94162e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 -- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf941faac0 msgr2=0x7fdf94162e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf941faac0 0x7fdf94162e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 -- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800047c0 msgr2=0x7fdf94156de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800047c0 0x7fdf94156de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 -- 172.21.15.29:0/1468761752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf940b3690 con 0x7fdf94162730 2023-10-14T16:59:27.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.533+0000 7fdf9ab79640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdf940c8c80 tx=0x7fdf8c00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.534+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c010040 con 0x7fdf94162730 2023-10-14T16:59:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.534+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf8c00bae0 con 0x7fdf94162730 2023-10-14T16:59:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.534+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf8c00be70 con 0x7fdf94162730 2023-10-14T16:59:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.534+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf940b3970 con 0x7fdf94162730 2023-10-14T16:59:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.534+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf940aeaf0 con 0x7fdf94162730 2023-10-14T16:59:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.535+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf940b1750 con 0x7fdf94162730 2023-10-14T16:59:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.536+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdf8c01d400 con 0x7fdf94162730 2023-10-14T16:59:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.537+0000 7fdf8affd640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 0x7fdf7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.537+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fdf8c089fc0 con 0x7fdf94162730 2023-10-14T16:59:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.538+0000 7fdf9a378640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 0x7fdf7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:27.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.538+0000 7fdf9a378640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 0x7fdf7806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf900017a0 tx=0x7fdf90009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:27.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.542+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf940b1750 con 0x7fdf94162730 2023-10-14T16:59:27.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.741+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf940b1750 con 0x7fdf94162730 2023-10-14T16:59:27.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.744+0000 7fdf8affd640 1 -- 172.21.15.29:0/1468761752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+16590 (secure 0 0 0) 0x7fdf940b1750 con 0x7fdf94162730 2023-10-14T16:59:27.745 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:27.745 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":124,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:26.282267+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 msgr2=0x7fdf7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 0x7fdf7806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fdf900017a0 tx=0x7fdf90009040 comp rx=0 tx=0).stop 2023-10-14T16:59:27.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 msgr2=0x7fdf94163370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:27.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 crypto rx=0x7fdf940c8c80 tx=0x7fdf8c00b480 comp rx=0 tx=0).stop 2023-10-14T16:59:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 shutdown_connections 2023-10-14T16:59:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fdf7806af90 0x7fdf7806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf94162730 0x7fdf94163370 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf941faac0 0x7fdf94162e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 --2- 172.21.15.29:0/1468761752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf800047c0 0x7fdf94156de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:27.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 >> 172.21.15.29:0/1468761752 conn(0x7fdf940ce6a0 msgr2=0x7fdf9406b280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:27.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 shutdown_connections 2023-10-14T16:59:27.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:27.747+0000 7fdf9b37a640 1 -- 172.21.15.29:0/1468761752 wait complete. 2023-10-14T16:59:27.769 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T16:59:28.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.003+0000 7fe403844640 1 Processor -- start 2023-10-14T16:59:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.004+0000 7fe403844640 1 -- start start 2023-10-14T16:59:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.005+0000 7fe403844640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.005+0000 7fe403844640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 0x7fe3fc06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.005+0000 7fe403844640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3fc070360 con 0x7fe3fc159e90 2023-10-14T16:59:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.005+0000 7fe403844640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3fc06cb10 con 0x7fe3fc06c140 2023-10-14T16:59:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.005+0000 7fe403844640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc15d890 con 0x7fe3fc159ab0 2023-10-14T16:59:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.006+0000 7fe4015e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.006+0000 7fe401de1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 0x7fe3fc06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.006+0000 7fe4015e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:42758/0 (socket says 172.21.15.29:42758) 2023-10-14T16:59:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.006+0000 7fe4015e0640 1 -- 172.21.15.29:0/910661716 learned_addr learned my addr 172.21.15.29:0/910661716 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.007+0000 7fe4015e0640 1 -- 172.21.15.29:0/910661716 >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 msgr2=0x7fe3fc06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.007+0000 7fe4015e0640 1 --2- 172.21.15.29:0/910661716 >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 0x7fe3fc06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.007+0000 7fe4015e0640 1 -- 172.21.15.29:0/910661716 >> v1:172.21.15.186:6789/0 conn(0x7fe3fc159e90 legacy=0x7fe3fc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.007+0000 7fe4015e0640 1 -- 172.21.15.29:0/910661716 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3fc0cd1f0 con 0x7fe3fc159ab0 2023-10-14T16:59:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.007+0000 7fe401de1640 1 --2- 172.21.15.29:0/910661716 >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 0x7fe3fc06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T16:59:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.008+0000 7fe4015e0640 1 --2- 172.21.15.29:0/910661716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d180 tx=0x7fe3ec02dfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4867e05a15d638e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.009+0000 7fe3fbfff640 1 -- 172.21.15.29:0/910661716 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec02e950 con 0x7fe3fc159ab0 2023-10-14T16:59:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.009+0000 7fe3fbfff640 1 -- 172.21.15.29:0/910661716 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3ec02eab0 con 0x7fe3fc159ab0 2023-10-14T16:59:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.009+0000 7fe3fbfff640 1 -- 172.21.15.29:0/910661716 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ec02ee40 con 0x7fe3fc159ab0 2023-10-14T16:59:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.009+0000 7fe403844640 1 -- 172.21.15.29:0/910661716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 msgr2=0x7fe3fc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.009+0000 7fe403844640 1 --2- 172.21.15.29:0/910661716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec00d180 tx=0x7fe3ec02dfb0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 -- 172.21.15.29:0/910661716 shutdown_connections 2023-10-14T16:59:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 --2- 172.21.15.29:0/910661716 >> v2:172.21.15.186:3300/0 conn(0x7fe3fc06c140 0x7fe3fc06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 --2- 172.21.15.29:0/910661716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159ab0 0x7fe3fc0ce370 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 -- 172.21.15.29:0/910661716 >> 172.21.15.29:0/910661716 conn(0x7fe3fc0ce9b0 msgr2=0x7fe3fc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 -- 172.21.15.29:0/910661716 shutdown_connections 2023-10-14T16:59:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.010+0000 7fe403844640 1 -- 172.21.15.29:0/910661716 wait complete. 2023-10-14T16:59:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.012+0000 7fe403844640 1 Processor -- start 2023-10-14T16:59:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.012+0000 7fe403844640 1 -- start start 2023-10-14T16:59:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3fc06c140 0x7fe3fc1568d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159e90 0x7fe3fc15bf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc0c4b90 con 0x7fe3fc159ab0 2023-10-14T16:59:28.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc0c48b0 con 0x7fe3fc06c140 2023-10-14T16:59:28.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe403844640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3fc0c2580 con 0x7fe3fc159e90 2023-10-14T16:59:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe400ddf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe400ddf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54662/0 (socket says 172.21.15.29:54662) 2023-10-14T16:59:28.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.013+0000 7fe400ddf640 1 -- 172.21.15.29:0/85093165 learned_addr learned my addr 172.21.15.29:0/85093165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe4015e0640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3fc06c140 0x7fe3fc1568d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe401de1640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159e90 0x7fe3fc15bf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 -- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159e90 msgr2=0x7fe3fc15bf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159e90 0x7fe3fc15bf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 -- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3fc06c140 msgr2=0x7fe3fc1568d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3fc06c140 0x7fe3fc1568d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 -- 172.21.15.29:0/85093165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f4009400 con 0x7fe3fc159ab0 2023-10-14T16:59:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.014+0000 7fe400ddf640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c430 tx=0x7fe3f000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.015+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011790 con 0x7fe3fc159ab0 2023-10-14T16:59:28.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.015+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3f00118f0 con 0x7fe3fc159ab0 2023-10-14T16:59:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.015+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3ec00bd40 con 0x7fe3fc159ab0 2023-10-14T16:59:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.015+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3f0011ce0 con 0x7fe3fc159ab0 2023-10-14T16:59:28.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.015+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3fc0c2b40 con 0x7fe3fc159ab0 2023-10-14T16:59:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.016+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3fc04e160 con 0x7fe3fc159ab0 2023-10-14T16:59:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.017+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3f001c460 con 0x7fe3fc159ab0 2023-10-14T16:59:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.018+0000 7fe3f9ffb640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 0x7fe3e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:28.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.018+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fe3f00896d0 con 0x7fe3fc159ab0 2023-10-14T16:59:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.019+0000 7fe4015e0640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 0x7fe3e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:28.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.019+0000 7fe4015e0640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 0x7fe3e806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec037000 tx=0x7fe3ec032000 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:28.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.023+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3f0054640 con 0x7fe3fc159ab0 2023-10-14T16:59:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.258+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe3fc0ccb60 con 0x7fe3fc159ab0 2023-10-14T16:59:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.873+0000 7fe3f9ffb640 1 -- 172.21.15.29:0/85093165 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe3f0057c60 con 0x7fe3fc159ab0 2023-10-14T16:59:28.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.877+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 msgr2=0x7fe3e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.877+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 0x7fe3e806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fe3ec037000 tx=0x7fe3ec032000 comp rx=0 tx=0).stop 2023-10-14T16:59:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.877+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 msgr2=0x7fe3fc15ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.877+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 crypto rx=0x7fe3f000c430 tx=0x7fe3f000c8f0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 shutdown_connections 2023-10-14T16:59:28.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fe3e806af90 0x7fe3e806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3fc159e90 0x7fe3fc15bf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3fc159ab0 0x7fe3fc15ba10 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 --2- 172.21.15.29:0/85093165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3fc06c140 0x7fe3fc1568d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 >> 172.21.15.29:0/85093165 conn(0x7fe3fc0ce9b0 msgr2=0x7fe3fc0b0480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:28.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 shutdown_connections 2023-10-14T16:59:28.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:28.878+0000 7fe403844640 1 -- 172.21.15.29:0/85093165 wait complete. 2023-10-14T16:59:28.900 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T16:59:28.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T16:59:29.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.131+0000 7fa024d93640 1 Processor -- start 2023-10-14T16:59:29.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.131+0000 7fa024d93640 1 -- start start 2023-10-14T16:59:29.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.132+0000 7fa024d93640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa02006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.132+0000 7fa024d93640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa0200cda80 0x7fa02015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.132+0000 7fa024d93640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa02006c960 con 0x7fa0200ad970 2023-10-14T16:59:29.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.132+0000 7fa024d93640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa02015b9d0 con 0x7fa0200cda80 2023-10-14T16:59:29.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.132+0000 7fa024d93640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa02015bcb0 con 0x7fa02006c040 2023-10-14T16:59:29.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa0200ad970 0x7fa02015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33010/0 (socket says 172.21.15.29:33010) 2023-10-14T16:59:29.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01effd640 1 -- 172.21.15.29:0/3483419991 learned_addr learned my addr 172.21.15.29:0/3483419991 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:29.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01ffff640 1 --2- 172.21.15.29:0/3483419991 >> v2:172.21.15.186:3300/0 conn(0x7fa0200cda80 0x7fa02015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01f7fe640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa02006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988416613 0 0) 0x7fa02006c960 con 0x7fa0200ad970 2023-10-14T16:59:29.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.133+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa008003520 con 0x7fa0200ad970 2023-10-14T16:59:29.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1106119163 0 0) 0x7fa008003520 con 0x7fa0200ad970 2023-10-14T16:59:29.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 msgr2=0x7fa02006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:29.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa02006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 >> v2:172.21.15.186:3300/0 conn(0x7fa0200cda80 msgr2=0x7fa02015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:29.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 --2- 172.21.15.29:0/3483419991 >> v2:172.21.15.186:3300/0 conn(0x7fa0200cda80 0x7fa02015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa02015bf90 con 0x7fa0200ad970 2023-10-14T16:59:29.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa014001670 con 0x7fa0200ad970 2023-10-14T16:59:29.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 >> v1:172.21.15.29:6789/0 conn(0x7fa0200ad970 legacy=0x7fa02015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa008003970 0x7fa008003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa0080047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa008005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02015b9d0 con 0x7fa008004d00 2023-10-14T16:59:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa02015bcb0 con 0x7fa008003970 2023-10-14T16:59:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa00800b980 con 0x7fa02006c040 2023-10-14T16:59:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.134+0000 7fa01ffff640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa008005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01f7fe640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa008003970 0x7fa008003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01effd640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa0080047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01ffff640 1 -- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 msgr2=0x7fa0080047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01ffff640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa02006c040 0x7fa0080047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01ffff640 1 -- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa008003970 msgr2=0x7fa008003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01ffff640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa008003970 0x7fa008003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.135+0000 7fa01ffff640 1 -- 172.21.15.29:0/3483419991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa008003520 con 0x7fa008004d00 2023-10-14T16:59:29.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.136+0000 7fa01ffff640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa008005180 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa018009fc0 tx=0x7fa0080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b53dc3aaff0ec46 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:29.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.136+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa01802cc60 con 0x7fa008004d00 2023-10-14T16:59:29.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.136+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa01800f040 con 0x7fa008004d00 2023-10-14T16:59:29.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.136+0000 7fa01e7fc640 1 -- 172.21.15.29:0/3483419991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa018034440 con 0x7fa008004d00 2023-10-14T16:59:29.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 -- 172.21.15.29:0/3483419991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 msgr2=0x7fa008005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 --2- 172.21.15.29:0/3483419991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa008005180 secure :-1 s=READY pgs=1395 cs=0 l=1 rev1=1 crypto rx=0x7fa018009fc0 tx=0x7fa0080057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa01ffff640 1 -- 172.21.15.29:0/3483419991 reap_dead start 2023-10-14T16:59:29.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 -- 172.21.15.29:0/3483419991 shutdown_connections 2023-10-14T16:59:29.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 -- 172.21.15.29:0/3483419991 >> 172.21.15.29:0/3483419991 conn(0x7fa0200ce6a0 msgr2=0x7fa0200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:29.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 -- 172.21.15.29:0/3483419991 shutdown_connections 2023-10-14T16:59:29.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.137+0000 7fa024d93640 1 -- 172.21.15.29:0/3483419991 wait complete. 2023-10-14T16:59:29.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.139+0000 7fa024d93640 1 Processor -- start 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.139+0000 7fa024d93640 1 -- start start 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa020146a30 0x7fa0200ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0200acdd0 0x7fa0200c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0200ad250 con 0x7fa008004d00 2023-10-14T16:59:29.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0201578c0 con 0x7fa0200acdd0 2023-10-14T16:59:29.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa024d93640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa020157ba0 con 0x7fa020146a30 2023-10-14T16:59:29.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa01f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa01f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54680/0 (socket says 172.21.15.29:54680) 2023-10-14T16:59:29.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.140+0000 7fa01f7fe640 1 -- 172.21.15.29:0/2920330394 learned_addr learned my addr 172.21.15.29:0/2920330394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:29.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 -- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa020146a30 msgr2=0x7fa0200ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa020146a30 0x7fa0200ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 -- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0200acdd0 msgr2=0x7fa0200c67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01ffff640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0200acdd0 0x7fa0200c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0200acdd0 0x7fa0200c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 -- 172.21.15.29:0/2920330394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa020157e80 con 0x7fa008004d00 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa01f7fe640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa010011540 tx=0x7fa010011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010015930 con 0x7fa008004d00 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa010015a90 con 0x7fa008004d00 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa010015e60 con 0x7fa008004d00 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa020158160 con 0x7fa008004d00 2023-10-14T16:59:29.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.141+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa020158960 con 0x7fa008004d00 2023-10-14T16:59:29.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.142+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa02004e9b0 con 0x7fa008004d00 2023-10-14T16:59:29.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.143+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa01000f460 con 0x7fa008004d00 2023-10-14T16:59:29.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.144+0000 7fa007fff640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 0x7fa00c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:29.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.145+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fa010089ff0 con 0x7fa008004d00 2023-10-14T16:59:29.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.145+0000 7fa01effd640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 0x7fa00c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:29.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.145+0000 7fa01effd640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 0x7fa00c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa02006c700 tx=0x7fa014001430 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:29.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.149+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa010054f40 con 0x7fa008004d00 2023-10-14T16:59:29.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.374+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa02006b2b0 con 0x7fa008004d00 2023-10-14T16:59:29.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.876+0000 7fa007fff640 1 -- 172.21.15.29:0/2920330394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v57) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa010058560 con 0x7fa008004d00 2023-10-14T16:59:29.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 msgr2=0x7fa00c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 0x7fa00c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fa02006c700 tx=0x7fa014001430 comp rx=0 tx=0).stop 2023-10-14T16:59:29.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 msgr2=0x7fa0200c8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:29.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 crypto rx=0x7fa010011540 tx=0x7fa010011a00 comp rx=0 tx=0).stop 2023-10-14T16:59:29.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 shutdown_connections 2023-10-14T16:59:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa00c06af90 0x7fa00c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0200acdd0 0x7fa0200c67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa020146a30 0x7fa0200ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 --2- 172.21.15.29:0/2920330394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa008004d00 0x7fa0200c8130 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:29.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 >> 172.21.15.29:0/2920330394 conn(0x7fa0200ce6a0 msgr2=0x7fa0200ca4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 shutdown_connections 2023-10-14T16:59:29.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:29.881+0000 7fa024d93640 1 -- 172.21.15.29:0/2920330394 wait complete. 2023-10-14T16:59:29.903 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T16:59:29.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.125+0000 7f6f09cbb640 1 Processor -- start 2023-10-14T16:59:30.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.125+0000 7f6f09cbb640 1 -- start start 2023-10-14T16:59:30.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f09cbb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f0406c040 0x7f6f0406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f09cbb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040cda80 0x7f6f0415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f09cbb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f0406c960 con 0x7f6f040ad970 2023-10-14T16:59:30.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f09cbb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f0415b9d0 con 0x7f6f0406c040 2023-10-14T16:59:30.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f09cbb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f0415bcb0 con 0x7f6f040cda80 2023-10-14T16:59:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f08cb9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6f0406c040 0x7f6f0406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f03fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6f040ad970 0x7f6f0415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33022/0 (socket says 172.21.15.29:33022) 2023-10-14T16:59:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.127+0000 7f6f03fff640 1 -- 172.21.15.29:0/1977926170 learned_addr learned my addr 172.21.15.29:0/1977926170 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:30.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.128+0000 7f6f094ba640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040cda80 0x7f6f0415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.128+0000 7f6f08cb9640 1 -- 172.21.15.29:0/1977926170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040cda80 msgr2=0x7f6f0415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.128+0000 7f6f08cb9640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040cda80 0x7f6f0415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.128+0000 7f6f08cb9640 1 -- 172.21.15.29:0/1977926170 >> v1:172.21.15.29:6789/0 conn(0x7f6f040ad970 legacy=0x7f6f0415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.128+0000 7f6f08cb9640 1 -- 172.21.15.29:0/1977926170 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f0415bf90 con 0x7f6f0406c040 2023-10-14T16:59:30.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.129+0000 7f6f08cb9640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0406c040 0x7f6f0406c420 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80097a0 tx=0x7f6ef802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=377f737442ecb967 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.129+0000 7f6f037fe640 1 -- 172.21.15.29:0/1977926170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef802d040 con 0x7f6f0406c040 2023-10-14T16:59:30.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.129+0000 7f6f037fe640 1 -- 172.21.15.29:0/1977926170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ef8031810 con 0x7f6f0406c040 2023-10-14T16:59:30.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.129+0000 7f6f037fe640 1 -- 172.21.15.29:0/1977926170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ef8031ba0 con 0x7f6f0406c040 2023-10-14T16:59:30.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1977926170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0406c040 msgr2=0x7f6f0406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0406c040 0x7f6f0406c420 secure :-1 s=READY pgs=1398 cs=0 l=1 rev1=1 crypto rx=0x7f6ef80097a0 tx=0x7f6ef802b8b0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1977926170 shutdown_connections 2023-10-14T16:59:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040cda80 0x7f6f0415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1977926170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f0406c040 0x7f6f0406c420 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1977926170 >> 172.21.15.29:0/1977926170 conn(0x7f6f040ce6a0 msgr2=0x7f6f040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1977926170 shutdown_connections 2023-10-14T16:59:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.130+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1977926170 wait complete. 2023-10-14T16:59:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.132+0000 7f6f09cbb640 1 Processor -- start 2023-10-14T16:59:30.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.132+0000 7f6f09cbb640 1 -- start start 2023-10-14T16:59:30.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f0406c040 0x7f6f040c7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040ad970 0x7f6f040b2410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040b3b70 con 0x7f6f040cda80 2023-10-14T16:59:30.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040c7620 con 0x7f6f0406c040 2023-10-14T16:59:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f09cbb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f040c7010 con 0x7f6f040ad970 2023-10-14T16:59:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f094ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f094ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54686/0 (socket says 172.21.15.29:54686) 2023-10-14T16:59:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f094ba640 1 -- 172.21.15.29:0/1431860314 learned_addr learned my addr 172.21.15.29:0/1431860314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:30.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f03fff640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040ad970 0x7f6f040b2410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f094ba640 1 -- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040ad970 msgr2=0x7f6f040b2410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f08cb9640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f0406c040 0x7f6f040c7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.133+0000 7f6f094ba640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040ad970 0x7f6f040b2410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f094ba640 1 -- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f0406c040 msgr2=0x7f6f040c7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f094ba640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f0406c040 0x7f6f040c7ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f094ba640 1 -- 172.21.15.29:0/1431860314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ef8009400 con 0x7f6f040cda80 2023-10-14T16:59:30.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f094ba640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6efc007e60 tx=0x7f6efc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc014420 con 0x7f6f040cda80 2023-10-14T16:59:30.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6efc0145a0 con 0x7f6f040cda80 2023-10-14T16:59:30.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6efc014950 con 0x7f6f040cda80 2023-10-14T16:59:30.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f040b3890 con 0x7f6f040cda80 2023-10-14T16:59:30.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.134+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f040b2950 con 0x7f6f040cda80 2023-10-14T16:59:30.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.135+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f0406b330 con 0x7f6f040cda80 2023-10-14T16:59:30.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.136+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6efc014ab0 con 0x7f6f040cda80 2023-10-14T16:59:30.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.138+0000 7f6f017fa640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 0x7f6ee806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.138+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f6efc08a190 con 0x7f6f040cda80 2023-10-14T16:59:30.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.138+0000 7f6f08cb9640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 0x7f6ee806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.139+0000 7f6f08cb9640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 0x7f6ee806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8036000 tx=0x7f6ef803c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.142+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6efc055060 con 0x7f6f040cda80 2023-10-14T16:59:30.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.383+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f040c6e70 con 0x7f6f040cda80 2023-10-14T16:59:30.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.384+0000 7f6f017fa640 1 -- 172.21.15.29:0/1431860314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f6f040c6e70 con 0x7f6f040cda80 2023-10-14T16:59:30.385 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:30.385 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T16:59:30.386 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T16:59:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.387+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 msgr2=0x7f6ee806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.387+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 0x7f6ee806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6ef8036000 tx=0x7f6ef803c040 comp rx=0 tx=0).stop 2023-10-14T16:59:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.387+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 msgr2=0x7f6f04159f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.387+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 crypto rx=0x7f6efc007e60 tx=0x7f6efc00e4b0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 shutdown_connections 2023-10-14T16:59:30.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f6ee806af90 0x7f6ee806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f040cda80 0x7f6f04159f80 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f040ad970 0x7f6f040b2410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 --2- 172.21.15.29:0/1431860314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f0406c040 0x7f6f040c7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 >> 172.21.15.29:0/1431860314 conn(0x7f6f040ce6a0 msgr2=0x7f6f0415c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:30.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 shutdown_connections 2023-10-14T16:59:30.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.388+0000 7f6f09cbb640 1 -- 172.21.15.29:0/1431860314 wait complete. 2023-10-14T16:59:30.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T16:59:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.629+0000 7f4e3b547640 1 Processor -- start 2023-10-14T16:59:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.629+0000 7f4e3b547640 1 -- start start 2023-10-14T16:59:30.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.630+0000 7f4e3b547640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4e34159950 0x7f4e3415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3b547640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e34070450 0x7f4e340708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3b547640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e3415aef0 con 0x7f4e3406c100 2023-10-14T16:59:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3b547640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e34070e20 con 0x7f4e34070450 2023-10-14T16:59:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3b547640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e3415e490 con 0x7f4e34159950 2023-10-14T16:59:30.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3ad46640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e3406c100 0x7f4e3406c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33036/0 (socket says 172.21.15.29:33036) 2023-10-14T16:59:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3ad46640 1 -- 172.21.15.29:0/1585725914 learned_addr learned my addr 172.21.15.29:0/1585725914 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e39d44640 1 --2- 172.21.15.29:0/1585725914 >> v2:172.21.15.29:3300/0 conn(0x7f4e34070450 0x7f4e340708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e3a545640 1 --2- 172.21.15.29:0/1585725914 >> v2:172.21.15.186:3300/0 conn(0x7f4e34159950 0x7f4e3415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.631+0000 7f4e39543640 1 -- 172.21.15.29:0/1585725914 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298451700 0 0) 0x7f4e2c001110 con 0x7f4e3406c100 2023-10-14T16:59:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.632+0000 7f4e39543640 1 -- 172.21.15.29:0/1585725914 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e20003520 con 0x7f4e3406c100 2023-10-14T16:59:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.632+0000 7f4e39d44640 1 -- 172.21.15.29:0/1585725914 >> v2:172.21.15.186:3300/0 conn(0x7f4e34159950 msgr2=0x7f4e3415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.632+0000 7f4e39d44640 1 --2- 172.21.15.29:0/1585725914 >> v2:172.21.15.186:3300/0 conn(0x7f4e34159950 0x7f4e3415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.632+0000 7f4e39d44640 1 -- 172.21.15.29:0/1585725914 >> v1:172.21.15.29:6789/0 conn(0x7f4e3406c100 legacy=0x7f4e3406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.632+0000 7f4e39d44640 1 -- 172.21.15.29:0/1585725914 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3415d910 con 0x7f4e34070450 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.633+0000 7f4e39d44640 1 --2- 172.21.15.29:0/1585725914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34070450 0x7f4e340708e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ca950 tx=0x7f4e2402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0265a346b41a23 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.633+0000 7f4e39543640 1 -- 172.21.15.29:0/1585725914 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2402e070 con 0x7f4e34070450 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.633+0000 7f4e39543640 1 -- 172.21.15.29:0/1585725914 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e240315a0 con 0x7f4e34070450 2023-10-14T16:59:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.633+0000 7f4e39543640 1 -- 172.21.15.29:0/1585725914 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e24031930 con 0x7f4e34070450 2023-10-14T16:59:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 -- 172.21.15.29:0/1585725914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34070450 msgr2=0x7f4e340708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 --2- 172.21.15.29:0/1585725914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34070450 0x7f4e340708e0 secure :-1 s=READY pgs=1401 cs=0 l=1 rev1=1 crypto rx=0x7f4e340ca950 tx=0x7f4e2402b840 comp rx=0 tx=0).stop 2023-10-14T16:59:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 -- 172.21.15.29:0/1585725914 shutdown_connections 2023-10-14T16:59:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 --2- 172.21.15.29:0/1585725914 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34070450 0x7f4e340708e0 unknown :-1 s=CLOSED pgs=1401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 --2- 172.21.15.29:0/1585725914 >> v2:172.21.15.186:3300/0 conn(0x7f4e34159950 0x7f4e3415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 -- 172.21.15.29:0/1585725914 >> 172.21.15.29:0/1585725914 conn(0x7f4e340ce6a0 msgr2=0x7f4e340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 -- 172.21.15.29:0/1585725914 shutdown_connections 2023-10-14T16:59:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.634+0000 7f4e3b547640 1 -- 172.21.15.29:0/1585725914 wait complete. 2023-10-14T16:59:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.636+0000 7f4e3b547640 1 Processor -- start 2023-10-14T16:59:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.636+0000 7f4e3b547640 1 -- start start 2023-10-14T16:59:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e3406c100 0x7f4e34152ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e34070450 0x7f4e3414f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340c8d80 con 0x7f4e34159950 2023-10-14T16:59:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e34153020 con 0x7f4e34070450 2023-10-14T16:59:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3b547640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e340c4760 con 0x7f4e3406c100 2023-10-14T16:59:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3ad46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3ad46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54708/0 (socket says 172.21.15.29:54708) 2023-10-14T16:59:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e3ad46640 1 -- 172.21.15.29:0/3494710997 learned_addr learned my addr 172.21.15.29:0/3494710997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.637+0000 7f4e39d44640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e34070450 0x7f4e3414f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3a545640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e3406c100 0x7f4e34152ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 -- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e3406c100 msgr2=0x7f4e34152ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e3406c100 0x7f4e34152ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 -- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e34070450 msgr2=0x7f4e3414f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e34070450 0x7f4e3414f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 -- 172.21.15.29:0/3494710997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e24009400 con 0x7f4e34159950 2023-10-14T16:59:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3ad46640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c700 tx=0x7f4e2c00cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c005ab0 con 0x7f4e34159950 2023-10-14T16:59:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c005c10 con 0x7f4e34159950 2023-10-14T16:59:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c001d50 con 0x7f4e34159950 2023-10-14T16:59:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.638+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e340c8aa0 con 0x7f4e34159950 2023-10-14T16:59:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.639+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e340c4ce0 con 0x7f4e34159950 2023-10-14T16:59:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.640+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e340c4f00 con 0x7f4e34159950 2023-10-14T16:59:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.641+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e2c007070 con 0x7f4e34159950 2023-10-14T16:59:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.642+0000 7f4e2affd640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 0x7f4e1806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.642+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f4e2c088bb0 con 0x7f4e34159950 2023-10-14T16:59:30.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.642+0000 7f4e3a545640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 0x7f4e1806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:30.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.643+0000 7f4e3a545640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 0x7f4e1806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e341f6050 tx=0x7f4e300039b0 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:30.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.647+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e2c053b20 con 0x7f4e34159950 2023-10-14T16:59:30.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.845+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e3415aef0 con 0x7f4e34159950 2023-10-14T16:59:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.847+0000 7f4e2affd640 1 -- 172.21.15.29:0/3494710997 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v125) v1 ==== 74+0+16577 (secure 0 0 0) 0x7f4e3415aef0 con 0x7f4e34159950 2023-10-14T16:59:30.848 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:30.848 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":125,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:27.822166+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T16:59:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.849+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 msgr2=0x7f4e1806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.849+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 0x7f4e1806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4e341f6050 tx=0x7f4e300039b0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.849+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 msgr2=0x7f4e34155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.849+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c700 tx=0x7f4e2c00cbc0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 shutdown_connections 2023-10-14T16:59:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f4e1806af90 0x7f4e1806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e34159950 0x7f4e34155390 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e34070450 0x7f4e3414f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 --2- 172.21.15.29:0/3494710997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e3406c100 0x7f4e34152ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 >> 172.21.15.29:0/3494710997 conn(0x7f4e340ce6a0 msgr2=0x7f4e3406b9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 shutdown_connections 2023-10-14T16:59:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:30.850+0000 7f4e3b547640 1 -- 172.21.15.29:0/3494710997 wait complete. 2023-10-14T16:59:30.871 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T16:59:30.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:59:31.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.097+0000 7f535b59e640 1 Processor -- start 2023-10-14T16:59:31.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.097+0000 7f535b59e640 1 -- start start 2023-10-14T16:59:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.099+0000 7f535b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f535c15dde0 0x7f535c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.099+0000 7f535b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f535c070f20 0x7f535c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.099+0000 7f535b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f535c160a90 con 0x7f535c15c9f0 2023-10-14T16:59:31.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.099+0000 7f535b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535c071350 con 0x7f535c15dde0 2023-10-14T16:59:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.099+0000 7f535b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f535c0cdec0 con 0x7f535c070f20 2023-10-14T16:59:31.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f535c15dde0 0x7f535c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f535c15dde0 0x7f535c0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52626/0 (socket says 172.21.15.29:52626) 2023-10-14T16:59:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 -- 172.21.15.29:0/2950426775 learned_addr learned my addr 172.21.15.29:0/2950426775 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f535ad9d640 1 --2- 172.21.15.29:0/2950426775 >> v2:172.21.15.186:3300/0 conn(0x7f535c070f20 0x7f535c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 -- 172.21.15.29:0/2950426775 >> v2:172.21.15.186:3300/0 conn(0x7f535c070f20 msgr2=0x7f535c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 --2- 172.21.15.29:0/2950426775 >> v2:172.21.15.186:3300/0 conn(0x7f535c070f20 0x7f535c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 -- 172.21.15.29:0/2950426775 >> v1:172.21.15.186:6789/0 conn(0x7f535c15c9f0 legacy=0x7f535c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.100+0000 7f5359d9b640 1 -- 172.21.15.29:0/2950426775 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0cd900 con 0x7f535c15dde0 2023-10-14T16:59:31.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.101+0000 7f5359d9b640 1 --2- 172.21.15.29:0/2950426775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15dde0 0x7f535c0709e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f535c06fcc0 tx=0x7f535002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3465c9303b60b291 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.102+0000 7f535959a640 1 -- 172.21.15.29:0/2950426775 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350031870 con 0x7f535c15dde0 2023-10-14T16:59:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.102+0000 7f535959a640 1 -- 172.21.15.29:0/2950426775 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53500319d0 con 0x7f535c15dde0 2023-10-14T16:59:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.102+0000 7f535959a640 1 -- 172.21.15.29:0/2950426775 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350031d60 con 0x7f535c15dde0 2023-10-14T16:59:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.102+0000 7f535b59e640 1 -- 172.21.15.29:0/2950426775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15dde0 msgr2=0x7f535c0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.102+0000 7f535b59e640 1 --2- 172.21.15.29:0/2950426775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15dde0 0x7f535c0709e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 crypto rx=0x7f535c06fcc0 tx=0x7f535002b910 comp rx=0 tx=0).stop 2023-10-14T16:59:31.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 -- 172.21.15.29:0/2950426775 shutdown_connections 2023-10-14T16:59:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 --2- 172.21.15.29:0/2950426775 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15dde0 0x7f535c0709e0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 --2- 172.21.15.29:0/2950426775 >> v2:172.21.15.186:3300/0 conn(0x7f535c070f20 0x7f535c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 -- 172.21.15.29:0/2950426775 >> 172.21.15.29:0/2950426775 conn(0x7f535c0ce6a0 msgr2=0x7f535c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 -- 172.21.15.29:0/2950426775 shutdown_connections 2023-10-14T16:59:31.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.103+0000 7f535b59e640 1 -- 172.21.15.29:0/2950426775 wait complete. 2023-10-14T16:59:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.105+0000 7f535b59e640 1 Processor -- start 2023-10-14T16:59:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.105+0000 7f535b59e640 1 -- start start 2023-10-14T16:59:31.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 0x7f535c06ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f535c15dde0 0x7f535c157020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c155450 con 0x7f535c15c9f0 2023-10-14T16:59:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f535c1f3270 con 0x7f535c15dde0 2023-10-14T16:59:31.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f535c154c30 con 0x7f535c070f20 2023-10-14T16:59:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f5359d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f5359d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52628/0 (socket says 172.21.15.29:52628) 2023-10-14T16:59:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f5359d9b640 1 -- 172.21.15.29:0/2916826742 learned_addr learned my addr 172.21.15.29:0/2916826742 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.107+0000 7f535a59c640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 0x7f535c06ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 -- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 msgr2=0x7f535c06ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 0x7f535c06ba60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f535ad9d640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f535c15dde0 0x7f535c157020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 -- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f535c15dde0 msgr2=0x7f535c157020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f535c15dde0 0x7f535c157020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 -- 172.21.15.29:0/2916826742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53500094d0 con 0x7f535c15c9f0 2023-10-14T16:59:31.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f535a59c640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 0x7f535c06ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f5359d9b640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f53500090b0 tx=0x7f53500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535003ad20 con 0x7f535c15c9f0 2023-10-14T16:59:31.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535003ae80 con 0x7f535c15c9f0 2023-10-14T16:59:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f535c154f10 con 0x7f535c15c9f0 2023-10-14T16:59:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53500445f0 con 0x7f535c15c9f0 2023-10-14T16:59:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.108+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f535c1f2ef0 con 0x7f535c15c9f0 2023-10-14T16:59:31.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.110+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f535c1555c0 con 0x7f535c15c9f0 2023-10-14T16:59:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.111+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f535004d450 con 0x7f535c15c9f0 2023-10-14T16:59:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.112+0000 7f534affd640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 0x7f533c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.112+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f535002d040 con 0x7f535c15c9f0 2023-10-14T16:59:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.112+0000 7f535a59c640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 0x7f533c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.113+0000 7f535a59c640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 0x7f533c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f535c15b560 tx=0x7f534c009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.117+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53500b2080 con 0x7f535c15c9f0 2023-10-14T16:59:31.225 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T16:59:31.224+0000 7f6a32401640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T16:59:31.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.308+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 --> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f535c154ab0 con 0x7f533c06af90 2023-10-14T16:59:31.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.336+0000 7f534affd640 1 -- 172.21.15.29:0/2916826742 <== mgr.7797 v2:172.21.15.29:6832/2416849365 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287014 (secure 0 0 0) 0x7f534c005050 con 0x7f533c06af90 2023-10-14T16:59:31.341 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:31.343 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:59:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.341+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 msgr2=0x7f533c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.341+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 0x7f533c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f535c15b560 tx=0x7f534c009040 comp rx=0 tx=0).stop 2023-10-14T16:59:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 msgr2=0x7f535c150bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 crypto rx=0x7f53500090b0 tx=0x7f53500046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 shutdown_connections 2023-10-14T16:59:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f533c06af90 0x7f533c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f535c15dde0 0x7f535c157020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f535c15c9f0 0x7f535c150bd0 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 --2- 172.21.15.29:0/2916826742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f535c070f20 0x7f535c06ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 >> 172.21.15.29:0/2916826742 conn(0x7f535c0ce6a0 msgr2=0x7f535c0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 shutdown_connections 2023-10-14T16:59:31.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.342+0000 7f535b59e640 1 -- 172.21.15.29:0/2916826742 wait complete. 2023-10-14T16:59:31.364 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T16:59:29.797869+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":348744,"kb_used_data":13276,"kb_used_omap":12,"kb_used_meta":335347,"kb_avail":754625976,"statfs":{"total":773094113280,"available":772736999424,"internally_reserved":0,"allocated":13594624,"data_stored":7144204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":343395896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.009651"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025134+0000","last_change":"2023-10-14T16:59:26.833468+0000","last_active":"2023-10-14T16:59:28.025134+0000","last_peered":"2023-10-14T16:59:28.025134+0000","last_clean":"2023-10-14T16:59:28.025134+0000","last_became_active":"2023-10-14T16:59:26.833202+0000","last_became_peered":"2023-10-14T16:59:26.833202+0000","last_unstale":"2023-10-14T16:59:28.025134+0000","last_undegraded":"2023-10-14T16:59:28.025134+0000","last_fullsized":"2023-10-14T16:59:28.025134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027284+0000","last_change":"2023-10-14T16:59:26.830463+0000","last_active":"2023-10-14T16:59:28.027284+0000","last_peered":"2023-10-14T16:59:28.027284+0000","last_clean":"2023-10-14T16:59:28.027284+0000","last_became_active":"2023-10-14T16:59:26.829935+0000","last_became_peered":"2023-10-14T16:59:26.829935+0000","last_unstale":"2023-10-14T16:59:28.027284+0000","last_undegraded":"2023-10-14T16:59:28.027284+0000","last_fullsized":"2023-10-14T16:59:28.027284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025283+0000","last_change":"2023-10-14T16:59:26.833878+0000","last_active":"2023-10-14T16:59:28.025283+0000","last_peered":"2023-10-14T16:59:28.025283+0000","last_clean":"2023-10-14T16:59:28.025283+0000","last_became_active":"2023-10-14T16:59:26.832776+0000","last_became_peered":"2023-10-14T16:59:26.832776+0000","last_unstale":"2023-10-14T16:59:28.025283+0000","last_undegraded":"2023-10-14T16:59:28.025283+0000","last_fullsized":"2023-10-14T16:59:28.025283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026490+0000","last_change":"2023-10-14T16:59:26.844268+0000","last_active":"2023-10-14T16:59:28.026490+0000","last_peered":"2023-10-14T16:59:28.026490+0000","last_clean":"2023-10-14T16:59:28.026490+0000","last_became_active":"2023-10-14T16:59:26.843557+0000","last_became_peered":"2023-10-14T16:59:26.843557+0000","last_unstale":"2023-10-14T16:59:28.026490+0000","last_undegraded":"2023-10-14T16:59:28.026490+0000","last_fullsized":"2023-10-14T16:59:28.026490+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.418824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833986+0000","last_change":"2023-10-14T16:59:26.836961+0000","last_active":"2023-10-14T16:59:27.833986+0000","last_peered":"2023-10-14T16:59:27.833986+0000","last_clean":"2023-10-14T16:59:27.833986+0000","last_became_active":"2023-10-14T16:59:26.836686+0000","last_became_peered":"2023-10-14T16:59:26.836686+0000","last_unstale":"2023-10-14T16:59:27.833986+0000","last_undegraded":"2023-10-14T16:59:27.833986+0000","last_fullsized":"2023-10-14T16:59:27.833986+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027771+0000","last_change":"2023-10-14T16:59:26.830441+0000","last_active":"2023-10-14T16:59:28.027771+0000","last_peered":"2023-10-14T16:59:28.027771+0000","last_clean":"2023-10-14T16:59:28.027771+0000","last_became_active":"2023-10-14T16:59:26.830126+0000","last_became_peered":"2023-10-14T16:59:26.830126+0000","last_unstale":"2023-10-14T16:59:28.027771+0000","last_undegraded":"2023-10-14T16:59:28.027771+0000","last_fullsized":"2023-10-14T16:59:28.027771+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:31:04.254209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027680+0000","last_change":"2023-10-14T16:59:26.835428+0000","last_active":"2023-10-14T16:59:28.027680+0000","last_peered":"2023-10-14T16:59:28.027680+0000","last_clean":"2023-10-14T16:59:28.027680+0000","last_became_active":"2023-10-14T16:59:26.834940+0000","last_became_peered":"2023-10-14T16:59:26.834940+0000","last_unstale":"2023-10-14T16:59:28.027680+0000","last_undegraded":"2023-10-14T16:59:28.027680+0000","last_fullsized":"2023-10-14T16:59:28.027680+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:25.591632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026311+0000","last_change":"2023-10-14T16:59:26.831040+0000","last_active":"2023-10-14T16:59:28.026311+0000","last_peered":"2023-10-14T16:59:28.026311+0000","last_clean":"2023-10-14T16:59:28.026311+0000","last_became_active":"2023-10-14T16:59:26.830617+0000","last_became_peered":"2023-10-14T16:59:26.830617+0000","last_unstale":"2023-10-14T16:59:28.026311+0000","last_undegraded":"2023-10-14T16:59:28.026311+0000","last_fullsized":"2023-10-14T16:59:28.026311+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833406+0000","last_change":"2023-10-14T16:59:26.841845+0000","last_active":"2023-10-14T16:59:27.833406+0000","last_peered":"2023-10-14T16:59:27.833406+0000","last_clean":"2023-10-14T16:59:27.833406+0000","last_became_active":"2023-10-14T16:59:26.841487+0000","last_became_peered":"2023-10-14T16:59:26.841487+0000","last_unstale":"2023-10-14T16:59:27.833406+0000","last_undegraded":"2023-10-14T16:59:27.833406+0000","last_fullsized":"2023-10-14T16:59:27.833406+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:20:50.738855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833046+0000","last_change":"2023-10-14T16:59:26.822372+0000","last_active":"2023-10-14T16:59:27.833046+0000","last_peered":"2023-10-14T16:59:27.833046+0000","last_clean":"2023-10-14T16:59:27.833046+0000","last_became_active":"2023-10-14T16:59:26.822022+0000","last_became_peered":"2023-10-14T16:59:26.822022+0000","last_unstale":"2023-10-14T16:59:27.833046+0000","last_undegraded":"2023-10-14T16:59:27.833046+0000","last_fullsized":"2023-10-14T16:59:27.833046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027992+0000","last_change":"2023-10-14T16:59:26.831067+0000","last_active":"2023-10-14T16:59:28.027992+0000","last_peered":"2023-10-14T16:59:28.027992+0000","last_clean":"2023-10-14T16:59:28.027992+0000","last_became_active":"2023-10-14T16:59:26.830799+0000","last_became_peered":"2023-10-14T16:59:26.830799+0000","last_unstale":"2023-10-14T16:59:28.027992+0000","last_undegraded":"2023-10-14T16:59:28.027992+0000","last_fullsized":"2023-10-14T16:59:28.027992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:30.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833343+0000","last_change":"2023-10-14T16:59:26.827015+0000","last_active":"2023-10-14T16:59:27.833343+0000","last_peered":"2023-10-14T16:59:27.833343+0000","last_clean":"2023-10-14T16:59:27.833343+0000","last_became_active":"2023-10-14T16:59:26.826717+0000","last_became_peered":"2023-10-14T16:59:26.826717+0000","last_unstale":"2023-10-14T16:59:27.833343+0000","last_undegraded":"2023-10-14T16:59:27.833343+0000","last_fullsized":"2023-10-14T16:59:27.833343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:58:02.836519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026056+0000","last_change":"2023-10-14T16:59:26.836746+0000","last_active":"2023-10-14T16:59:28.026056+0000","last_peered":"2023-10-14T16:59:28.026056+0000","last_clean":"2023-10-14T16:59:28.026056+0000","last_became_active":"2023-10-14T16:59:26.836463+0000","last_became_peered":"2023-10-14T16:59:26.836463+0000","last_unstale":"2023-10-14T16:59:28.026056+0000","last_undegraded":"2023-10-14T16:59:28.026056+0000","last_fullsized":"2023-10-14T16:59:28.026056+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:10.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828347+0000","last_change":"2023-10-14T16:59:26.832929+0000","last_active":"2023-10-14T16:59:27.828347+0000","last_peered":"2023-10-14T16:59:27.828347+0000","last_clean":"2023-10-14T16:59:27.828347+0000","last_became_active":"2023-10-14T16:59:26.832555+0000","last_became_peered":"2023-10-14T16:59:26.832555+0000","last_unstale":"2023-10-14T16:59:27.828347+0000","last_undegraded":"2023-10-14T16:59:27.828347+0000","last_fullsized":"2023-10-14T16:59:27.828347+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:06.528696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025926+0000","last_change":"2023-10-14T16:59:26.844457+0000","last_active":"2023-10-14T16:59:28.025926+0000","last_peered":"2023-10-14T16:59:28.025926+0000","last_clean":"2023-10-14T16:59:28.025926+0000","last_became_active":"2023-10-14T16:59:26.843998+0000","last_became_peered":"2023-10-14T16:59:26.843998+0000","last_unstale":"2023-10-14T16:59:28.025926+0000","last_undegraded":"2023-10-14T16:59:28.025926+0000","last_fullsized":"2023-10-14T16:59:28.025926+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:05.566426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027358+0000","last_change":"2023-10-14T16:59:26.827938+0000","last_active":"2023-10-14T16:59:28.027358+0000","last_peered":"2023-10-14T16:59:28.027358+0000","last_clean":"2023-10-14T16:59:28.027358+0000","last_became_active":"2023-10-14T16:59:26.827654+0000","last_became_peered":"2023-10-14T16:59:26.827654+0000","last_unstale":"2023-10-14T16:59:28.027358+0000","last_undegraded":"2023-10-14T16:59:28.027358+0000","last_fullsized":"2023-10-14T16:59:28.027358+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.418824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027574+0000","last_change":"2023-10-14T16:59:26.839529+0000","last_active":"2023-10-14T16:59:28.027574+0000","last_peered":"2023-10-14T16:59:28.027574+0000","last_clean":"2023-10-14T16:59:28.027574+0000","last_became_active":"2023-10-14T16:59:26.839266+0000","last_became_peered":"2023-10-14T16:59:26.839266+0000","last_unstale":"2023-10-14T16:59:28.027574+0000","last_undegraded":"2023-10-14T16:59:28.027574+0000","last_fullsized":"2023-10-14T16:59:28.027574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:45.122421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026518+0000","last_change":"2023-10-14T16:59:26.846505+0000","last_active":"2023-10-14T16:59:28.026518+0000","last_peered":"2023-10-14T16:59:28.026518+0000","last_clean":"2023-10-14T16:59:28.026518+0000","last_became_active":"2023-10-14T16:59:26.841179+0000","last_became_peered":"2023-10-14T16:59:26.841179+0000","last_unstale":"2023-10-14T16:59:28.026518+0000","last_undegraded":"2023-10-14T16:59:28.026518+0000","last_fullsized":"2023-10-14T16:59:28.026518+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:09.743350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829397+0000","last_change":"2023-10-14T16:59:26.831037+0000","last_active":"2023-10-14T16:59:27.829397+0000","last_peered":"2023-10-14T16:59:27.829397+0000","last_clean":"2023-10-14T16:59:27.829397+0000","last_became_active":"2023-10-14T16:59:26.830706+0000","last_became_peered":"2023-10-14T16:59:26.830706+0000","last_unstale":"2023-10-14T16:59:27.829397+0000","last_undegraded":"2023-10-14T16:59:27.829397+0000","last_fullsized":"2023-10-14T16:59:27.829397+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:47.921758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833719+0000","last_change":"2023-10-14T16:59:26.839109+0000","last_active":"2023-10-14T16:59:27.833719+0000","last_peered":"2023-10-14T16:59:27.833719+0000","last_clean":"2023-10-14T16:59:27.833719+0000","last_became_active":"2023-10-14T16:59:26.837731+0000","last_became_peered":"2023-10-14T16:59:26.837731+0000","last_unstale":"2023-10-14T16:59:27.833719+0000","last_undegraded":"2023-10-14T16:59:27.833719+0000","last_fullsized":"2023-10-14T16:59:27.833719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:32.136376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829296+0000","last_change":"2023-10-14T16:59:26.829711+0000","last_active":"2023-10-14T16:59:27.829296+0000","last_peered":"2023-10-14T16:59:27.829296+0000","last_clean":"2023-10-14T16:59:27.829296+0000","last_became_active":"2023-10-14T16:59:26.829426+0000","last_became_peered":"2023-10-14T16:59:26.829426+0000","last_unstale":"2023-10-14T16:59:27.829296+0000","last_undegraded":"2023-10-14T16:59:27.829296+0000","last_fullsized":"2023-10-14T16:59:27.829296+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:31.674138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026931+0000","last_change":"2023-10-14T16:59:26.829499+0000","last_active":"2023-10-14T16:59:28.026931+0000","last_peered":"2023-10-14T16:59:28.026931+0000","last_clean":"2023-10-14T16:59:28.026931+0000","last_became_active":"2023-10-14T16:59:26.829201+0000","last_became_peered":"2023-10-14T16:59:26.829201+0000","last_unstale":"2023-10-14T16:59:28.026931+0000","last_undegraded":"2023-10-14T16:59:28.026931+0000","last_fullsized":"2023-10-14T16:59:28.026931+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026935+0000","last_change":"2023-10-14T16:59:26.827178+0000","last_active":"2023-10-14T16:59:28.026935+0000","last_peered":"2023-10-14T16:59:28.026935+0000","last_clean":"2023-10-14T16:59:28.026935+0000","last_became_active":"2023-10-14T16:59:26.826867+0000","last_became_peered":"2023-10-14T16:59:26.826867+0000","last_unstale":"2023-10-14T16:59:28.026935+0000","last_undegraded":"2023-10-14T16:59:28.026935+0000","last_fullsized":"2023-10-14T16:59:28.026935+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027725+0000","last_change":"2023-10-14T16:59:26.819042+0000","last_active":"2023-10-14T16:59:28.027725+0000","last_peered":"2023-10-14T16:59:28.027725+0000","last_clean":"2023-10-14T16:59:28.027725+0000","last_became_active":"2023-10-14T16:59:26.818257+0000","last_became_peered":"2023-10-14T16:59:26.818257+0000","last_unstale":"2023-10-14T16:59:28.027725+0000","last_undegraded":"2023-10-14T16:59:28.027725+0000","last_fullsized":"2023-10-14T16:59:28.027725+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:03:55.302030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834305+0000","last_change":"2023-10-14T16:59:26.833604+0000","last_active":"2023-10-14T16:59:27.834305+0000","last_peered":"2023-10-14T16:59:27.834305+0000","last_clean":"2023-10-14T16:59:27.834305+0000","last_became_active":"2023-10-14T16:59:26.833310+0000","last_became_peered":"2023-10-14T16:59:26.833310+0000","last_unstale":"2023-10-14T16:59:27.834305+0000","last_undegraded":"2023-10-14T16:59:27.834305+0000","last_fullsized":"2023-10-14T16:59:27.834305+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027426+0000","last_change":"2023-10-14T16:59:26.830208+0000","last_active":"2023-10-14T16:59:28.027426+0000","last_peered":"2023-10-14T16:59:28.027426+0000","last_clean":"2023-10-14T16:59:28.027426+0000","last_became_active":"2023-10-14T16:59:26.829137+0000","last_became_peered":"2023-10-14T16:59:26.829137+0000","last_unstale":"2023-10-14T16:59:28.027426+0000","last_undegraded":"2023-10-14T16:59:28.027426+0000","last_fullsized":"2023-10-14T16:59:28.027426+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028354+0000","last_change":"2023-10-14T16:59:26.835636+0000","last_active":"2023-10-14T16:59:28.028354+0000","last_peered":"2023-10-14T16:59:28.028354+0000","last_clean":"2023-10-14T16:59:28.028354+0000","last_became_active":"2023-10-14T16:59:26.834941+0000","last_became_peered":"2023-10-14T16:59:26.834941+0000","last_unstale":"2023-10-14T16:59:28.028354+0000","last_undegraded":"2023-10-14T16:59:28.028354+0000","last_fullsized":"2023-10-14T16:59:28.028354+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:01.607198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835074+0000","last_change":"2023-10-14T16:59:26.842610+0000","last_active":"2023-10-14T16:59:27.835074+0000","last_peered":"2023-10-14T16:59:27.835074+0000","last_clean":"2023-10-14T16:59:27.835074+0000","last_became_active":"2023-10-14T16:59:26.842190+0000","last_became_peered":"2023-10-14T16:59:26.842190+0000","last_unstale":"2023-10-14T16:59:27.835074+0000","last_undegraded":"2023-10-14T16:59:27.835074+0000","last_fullsized":"2023-10-14T16:59:27.835074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027167+0000","last_change":"2023-10-14T16:59:26.827805+0000","last_active":"2023-10-14T16:59:28.027167+0000","last_peered":"2023-10-14T16:59:28.027167+0000","last_clean":"2023-10-14T16:59:28.027167+0000","last_became_active":"2023-10-14T16:59:26.827485+0000","last_became_peered":"2023-10-14T16:59:26.827485+0000","last_unstale":"2023-10-14T16:59:28.027167+0000","last_undegraded":"2023-10-14T16:59:28.027167+0000","last_fullsized":"2023-10-14T16:59:28.027167+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828353+0000","last_change":"2023-10-14T16:59:26.831691+0000","last_active":"2023-10-14T16:59:27.828353+0000","last_peered":"2023-10-14T16:59:27.828353+0000","last_clean":"2023-10-14T16:59:27.828353+0000","last_became_active":"2023-10-14T16:59:26.831327+0000","last_became_peered":"2023-10-14T16:59:26.831327+0000","last_unstale":"2023-10-14T16:59:27.828353+0000","last_undegraded":"2023-10-14T16:59:27.828353+0000","last_fullsized":"2023-10-14T16:59:27.828353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:33:34.191228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026123+0000","last_change":"2023-10-14T16:59:26.834390+0000","last_active":"2023-10-14T16:59:28.026123+0000","last_peered":"2023-10-14T16:59:28.026123+0000","last_clean":"2023-10-14T16:59:28.026123+0000","last_became_active":"2023-10-14T16:59:26.833279+0000","last_became_peered":"2023-10-14T16:59:26.833279+0000","last_unstale":"2023-10-14T16:59:28.026123+0000","last_undegraded":"2023-10-14T16:59:28.026123+0000","last_fullsized":"2023-10-14T16:59:28.026123+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:28.965220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025056+0000","last_change":"2023-10-14T16:59:26.828525+0000","last_active":"2023-10-14T16:59:28.025056+0000","last_peered":"2023-10-14T16:59:28.025056+0000","last_clean":"2023-10-14T16:59:28.025056+0000","last_became_active":"2023-10-14T16:59:26.827994+0000","last_became_peered":"2023-10-14T16:59:26.827994+0000","last_unstale":"2023-10-14T16:59:28.025056+0000","last_undegraded":"2023-10-14T16:59:28.025056+0000","last_fullsized":"2023-10-14T16:59:28.025056+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829453+0000","last_change":"2023-10-14T16:59:26.836936+0000","last_active":"2023-10-14T16:59:27.829453+0000","last_peered":"2023-10-14T16:59:27.829453+0000","last_clean":"2023-10-14T16:59:27.829453+0000","last_became_active":"2023-10-14T16:59:26.836046+0000","last_became_peered":"2023-10-14T16:59:26.836046+0000","last_unstale":"2023-10-14T16:59:27.829453+0000","last_undegraded":"2023-10-14T16:59:27.829453+0000","last_fullsized":"2023-10-14T16:59:27.829453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:04.971542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"124'4","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027528+0000","last_change":"2023-10-14T16:59:25.831858+0000","last_active":"2023-10-14T16:59:28.027528+0000","last_peered":"2023-10-14T16:59:28.027528+0000","last_clean":"2023-10-14T16:59:28.027528+0000","last_became_active":"2023-10-14T16:59:25.831562+0000","last_became_peered":"2023-10-14T16:59:25.831562+0000","last_unstale":"2023-10-14T16:59:28.027528+0000","last_undegraded":"2023-10-14T16:59:28.027528+0000","last_fullsized":"2023-10-14T16:59:28.027528+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:03.957057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026660+0000","last_change":"2023-10-14T16:59:26.835727+0000","last_active":"2023-10-14T16:59:28.026660+0000","last_peered":"2023-10-14T16:59:28.026660+0000","last_clean":"2023-10-14T16:59:28.026660+0000","last_became_active":"2023-10-14T16:59:26.835453+0000","last_became_peered":"2023-10-14T16:59:26.835453+0000","last_unstale":"2023-10-14T16:59:28.026660+0000","last_undegraded":"2023-10-14T16:59:28.026660+0000","last_fullsized":"2023-10-14T16:59:28.026660+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:25.591632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028593+0000","last_change":"2023-10-14T16:59:25.835359+0000","last_active":"2023-10-14T16:59:28.028593+0000","last_peered":"2023-10-14T16:59:28.028593+0000","last_clean":"2023-10-14T16:59:28.028593+0000","last_became_active":"2023-10-14T16:59:25.835118+0000","last_became_peered":"2023-10-14T16:59:25.835118+0000","last_unstale":"2023-10-14T16:59:28.028593+0000","last_undegraded":"2023-10-14T16:59:28.028593+0000","last_fullsized":"2023-10-14T16:59:28.028593+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:00:38.918023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025147+0000","last_change":"2023-10-14T16:59:26.824038+0000","last_active":"2023-10-14T16:59:28.025147+0000","last_peered":"2023-10-14T16:59:28.025147+0000","last_clean":"2023-10-14T16:59:28.025147+0000","last_became_active":"2023-10-14T16:59:26.823697+0000","last_became_peered":"2023-10-14T16:59:26.823697+0000","last_unstale":"2023-10-14T16:59:28.025147+0000","last_undegraded":"2023-10-14T16:59:28.025147+0000","last_fullsized":"2023-10-14T16:59:28.025147+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027253+0000","last_change":"2023-10-14T16:59:25.834259+0000","last_active":"2023-10-14T16:59:28.027253+0000","last_peered":"2023-10-14T16:59:28.027253+0000","last_clean":"2023-10-14T16:59:28.027253+0000","last_became_active":"2023-10-14T16:59:25.833994+0000","last_became_peered":"2023-10-14T16:59:25.833994+0000","last_unstale":"2023-10-14T16:59:28.027253+0000","last_undegraded":"2023-10-14T16:59:28.027253+0000","last_fullsized":"2023-10-14T16:59:28.027253+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:41.997518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834836+0000","last_change":"2023-10-14T16:59:26.839609+0000","last_active":"2023-10-14T16:59:27.834836+0000","last_peered":"2023-10-14T16:59:27.834836+0000","last_clean":"2023-10-14T16:59:27.834836+0000","last_became_active":"2023-10-14T16:59:26.838166+0000","last_became_peered":"2023-10-14T16:59:26.838166+0000","last_unstale":"2023-10-14T16:59:27.834836+0000","last_undegraded":"2023-10-14T16:59:27.834836+0000","last_fullsized":"2023-10-14T16:59:27.834836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:57.033678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027213+0000","last_change":"2023-10-14T16:59:25.827668+0000","last_active":"2023-10-14T16:59:28.027213+0000","last_peered":"2023-10-14T16:59:28.027213+0000","last_clean":"2023-10-14T16:59:28.027213+0000","last_became_active":"2023-10-14T16:59:25.827152+0000","last_became_peered":"2023-10-14T16:59:25.827152+0000","last_unstale":"2023-10-14T16:59:28.027213+0000","last_undegraded":"2023-10-14T16:59:28.027213+0000","last_fullsized":"2023-10-14T16:59:28.027213+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:08.691851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028855+0000","last_change":"2023-10-14T16:59:26.835455+0000","last_active":"2023-10-14T16:59:28.028855+0000","last_peered":"2023-10-14T16:59:28.028855+0000","last_clean":"2023-10-14T16:59:28.028855+0000","last_became_active":"2023-10-14T16:59:26.835096+0000","last_became_peered":"2023-10-14T16:59:26.835096+0000","last_unstale":"2023-10-14T16:59:28.028855+0000","last_undegraded":"2023-10-14T16:59:28.028855+0000","last_fullsized":"2023-10-14T16:59:28.028855+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:08.624193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025205+0000","last_change":"2023-10-14T16:59:25.829120+0000","last_active":"2023-10-14T16:59:28.025205+0000","last_peered":"2023-10-14T16:59:28.025205+0000","last_clean":"2023-10-14T16:59:28.025205+0000","last_became_active":"2023-10-14T16:59:25.826156+0000","last_became_peered":"2023-10-14T16:59:25.826156+0000","last_unstale":"2023-10-14T16:59:28.025205+0000","last_undegraded":"2023-10-14T16:59:28.025205+0000","last_fullsized":"2023-10-14T16:59:28.025205+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:46.907273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027582+0000","last_change":"2023-10-14T16:59:26.824727+0000","last_active":"2023-10-14T16:59:28.027582+0000","last_peered":"2023-10-14T16:59:28.027582+0000","last_clean":"2023-10-14T16:59:28.027582+0000","last_became_active":"2023-10-14T16:59:26.824131+0000","last_became_peered":"2023-10-14T16:59:26.824131+0000","last_unstale":"2023-10-14T16:59:28.027582+0000","last_undegraded":"2023-10-14T16:59:28.027582+0000","last_fullsized":"2023-10-14T16:59:28.027582+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:09.743350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027006+0000","last_change":"2023-10-14T16:59:25.833297+0000","last_active":"2023-10-14T16:59:28.027006+0000","last_peered":"2023-10-14T16:59:28.027006+0000","last_clean":"2023-10-14T16:59:28.027006+0000","last_became_active":"2023-10-14T16:59:25.832997+0000","last_became_peered":"2023-10-14T16:59:25.832997+0000","last_unstale":"2023-10-14T16:59:28.027006+0000","last_undegraded":"2023-10-14T16:59:28.027006+0000","last_fullsized":"2023-10-14T16:59:28.027006+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025901+0000","last_change":"2023-10-14T16:59:26.833981+0000","last_active":"2023-10-14T16:59:28.025901+0000","last_peered":"2023-10-14T16:59:28.025901+0000","last_clean":"2023-10-14T16:59:28.025901+0000","last_became_active":"2023-10-14T16:59:26.833621+0000","last_became_peered":"2023-10-14T16:59:26.833621+0000","last_unstale":"2023-10-14T16:59:28.025901+0000","last_undegraded":"2023-10-14T16:59:28.025901+0000","last_fullsized":"2023-10-14T16:59:28.025901+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:37.096065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832914+0000","last_change":"2023-10-14T16:59:25.824007+0000","last_active":"2023-10-14T16:59:27.832914+0000","last_peered":"2023-10-14T16:59:27.832914+0000","last_clean":"2023-10-14T16:59:27.832914+0000","last_became_active":"2023-10-14T16:59:25.823704+0000","last_became_peered":"2023-10-14T16:59:25.823704+0000","last_unstale":"2023-10-14T16:59:27.832914+0000","last_undegraded":"2023-10-14T16:59:27.832914+0000","last_fullsized":"2023-10-14T16:59:27.832914+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833684+0000","last_change":"2023-10-14T16:59:26.833788+0000","last_active":"2023-10-14T16:59:27.833684+0000","last_peered":"2023-10-14T16:59:27.833684+0000","last_clean":"2023-10-14T16:59:27.833684+0000","last_became_active":"2023-10-14T16:59:26.833210+0000","last_became_peered":"2023-10-14T16:59:26.833210+0000","last_unstale":"2023-10-14T16:59:27.833684+0000","last_undegraded":"2023-10-14T16:59:27.833684+0000","last_fullsized":"2023-10-14T16:59:27.833684+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:55.425290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.024760+0000","last_change":"2023-10-14T16:59:25.827524+0000","last_active":"2023-10-14T16:59:28.024760+0000","last_peered":"2023-10-14T16:59:28.024760+0000","last_clean":"2023-10-14T16:59:28.024760+0000","last_became_active":"2023-10-14T16:59:25.827227+0000","last_became_peered":"2023-10-14T16:59:25.827227+0000","last_unstale":"2023-10-14T16:59:28.024760+0000","last_undegraded":"2023-10-14T16:59:28.024760+0000","last_fullsized":"2023-10-14T16:59:28.024760+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:59.894118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027526+0000","last_change":"2023-10-14T16:59:26.827014+0000","last_active":"2023-10-14T16:59:28.027526+0000","last_peered":"2023-10-14T16:59:28.027526+0000","last_clean":"2023-10-14T16:59:28.027526+0000","last_became_active":"2023-10-14T16:59:26.826625+0000","last_became_peered":"2023-10-14T16:59:26.826625+0000","last_unstale":"2023-10-14T16:59:28.027526+0000","last_undegraded":"2023-10-14T16:59:28.027526+0000","last_fullsized":"2023-10-14T16:59:28.027526+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025372+0000","last_change":"2023-10-14T16:59:25.823768+0000","last_active":"2023-10-14T16:59:28.025372+0000","last_peered":"2023-10-14T16:59:28.025372+0000","last_clean":"2023-10-14T16:59:28.025372+0000","last_became_active":"2023-10-14T16:59:25.823498+0000","last_became_peered":"2023-10-14T16:59:25.823498+0000","last_unstale":"2023-10-14T16:59:28.025372+0000","last_undegraded":"2023-10-14T16:59:28.025372+0000","last_fullsized":"2023-10-14T16:59:28.025372+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833771+0000","last_change":"2023-10-14T16:59:26.834533+0000","last_active":"2023-10-14T16:59:27.833771+0000","last_peered":"2023-10-14T16:59:27.833771+0000","last_clean":"2023-10-14T16:59:27.833771+0000","last_became_active":"2023-10-14T16:59:26.832120+0000","last_became_peered":"2023-10-14T16:59:26.832120+0000","last_unstale":"2023-10-14T16:59:27.833771+0000","last_undegraded":"2023-10-14T16:59:27.833771+0000","last_fullsized":"2023-10-14T16:59:27.833771+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:08.624193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026531+0000","last_change":"2023-10-14T16:59:25.826357+0000","last_active":"2023-10-14T16:59:28.026531+0000","last_peered":"2023-10-14T16:59:28.026531+0000","last_clean":"2023-10-14T16:59:28.026531+0000","last_became_active":"2023-10-14T16:59:25.824619+0000","last_became_peered":"2023-10-14T16:59:25.824619+0000","last_unstale":"2023-10-14T16:59:28.026531+0000","last_undegraded":"2023-10-14T16:59:28.026531+0000","last_fullsized":"2023-10-14T16:59:28.026531+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:00:38.918023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835115+0000","last_change":"2023-10-14T16:59:26.838832+0000","last_active":"2023-10-14T16:59:27.835115+0000","last_peered":"2023-10-14T16:59:27.835115+0000","last_clean":"2023-10-14T16:59:27.835115+0000","last_became_active":"2023-10-14T16:59:26.838529+0000","last_became_peered":"2023-10-14T16:59:26.838529+0000","last_unstale":"2023-10-14T16:59:27.835115+0000","last_undegraded":"2023-10-14T16:59:27.835115+0000","last_fullsized":"2023-10-14T16:59:27.835115+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"124'3","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.029017+0000","last_change":"2023-10-14T16:59:25.832164+0000","last_active":"2023-10-14T16:59:28.029017+0000","last_peered":"2023-10-14T16:59:28.029017+0000","last_clean":"2023-10-14T16:59:28.029017+0000","last_became_active":"2023-10-14T16:59:25.831848+0000","last_became_peered":"2023-10-14T16:59:25.831848+0000","last_unstale":"2023-10-14T16:59:28.029017+0000","last_undegraded":"2023-10-14T16:59:28.029017+0000","last_fullsized":"2023-10-14T16:59:28.029017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:27.950735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028703+0000","last_change":"2023-10-14T16:59:26.835807+0000","last_active":"2023-10-14T16:59:28.028703+0000","last_peered":"2023-10-14T16:59:28.028703+0000","last_clean":"2023-10-14T16:59:28.028703+0000","last_became_active":"2023-10-14T16:59:26.835355+0000","last_became_peered":"2023-10-14T16:59:26.835355+0000","last_unstale":"2023-10-14T16:59:28.028703+0000","last_undegraded":"2023-10-14T16:59:28.028703+0000","last_fullsized":"2023-10-14T16:59:28.028703+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:02.563762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"124'6","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833969+0000","last_change":"2023-10-14T16:59:25.825788+0000","last_active":"2023-10-14T16:59:27.833969+0000","last_peered":"2023-10-14T16:59:27.833969+0000","last_clean":"2023-10-14T16:59:27.833969+0000","last_became_active":"2023-10-14T16:59:25.825483+0000","last_became_peered":"2023-10-14T16:59:25.825483+0000","last_unstale":"2023-10-14T16:59:27.833969+0000","last_undegraded":"2023-10-14T16:59:27.833969+0000","last_fullsized":"2023-10-14T16:59:27.833969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:49.388877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025978+0000","last_change":"2023-10-14T16:59:26.839521+0000","last_active":"2023-10-14T16:59:28.025978+0000","last_peered":"2023-10-14T16:59:28.025978+0000","last_clean":"2023-10-14T16:59:28.025978+0000","last_became_active":"2023-10-14T16:59:26.837562+0000","last_became_peered":"2023-10-14T16:59:26.837562+0000","last_unstale":"2023-10-14T16:59:28.025978+0000","last_undegraded":"2023-10-14T16:59:28.025978+0000","last_fullsized":"2023-10-14T16:59:28.025978+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:46:15.037570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829131+0000","last_change":"2023-10-14T16:59:25.819010+0000","last_active":"2023-10-14T16:59:27.829131+0000","last_peered":"2023-10-14T16:59:27.829131+0000","last_clean":"2023-10-14T16:59:27.829131+0000","last_became_active":"2023-10-14T16:59:25.818696+0000","last_became_peered":"2023-10-14T16:59:25.818696+0000","last_unstale":"2023-10-14T16:59:27.829131+0000","last_undegraded":"2023-10-14T16:59:27.829131+0000","last_fullsized":"2023-10-14T16:59:27.829131+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:37:20.595779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025054+0000","last_change":"2023-10-14T16:59:26.835333+0000","last_active":"2023-10-14T16:59:28.025054+0000","last_peered":"2023-10-14T16:59:28.025054+0000","last_clean":"2023-10-14T16:59:28.025054+0000","last_became_active":"2023-10-14T16:59:26.835030+0000","last_became_peered":"2023-10-14T16:59:26.835030+0000","last_unstale":"2023-10-14T16:59:28.025054+0000","last_undegraded":"2023-10-14T16:59:28.025054+0000","last_fullsized":"2023-10-14T16:59:28.025054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828896+0000","last_change":"2023-10-14T16:59:25.819245+0000","last_active":"2023-10-14T16:59:27.828896+0000","last_peered":"2023-10-14T16:59:27.828896+0000","last_clean":"2023-10-14T16:59:27.828896+0000","last_became_active":"2023-10-14T16:59:25.818991+0000","last_became_peered":"2023-10-14T16:59:25.818991+0000","last_unstale":"2023-10-14T16:59:27.828896+0000","last_undegraded":"2023-10-14T16:59:27.828896+0000","last_fullsized":"2023-10-14T16:59:27.828896+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:13.967958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.831755+0000","last_change":"2023-10-14T16:59:26.832899+0000","last_active":"2023-10-14T16:59:27.831755+0000","last_peered":"2023-10-14T16:59:27.831755+0000","last_clean":"2023-10-14T16:59:27.831755+0000","last_became_active":"2023-10-14T16:59:26.831690+0000","last_became_peered":"2023-10-14T16:59:26.831690+0000","last_unstale":"2023-10-14T16:59:27.831755+0000","last_undegraded":"2023-10-14T16:59:27.831755+0000","last_fullsized":"2023-10-14T16:59:27.831755+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:28.174078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833893+0000","last_change":"2023-10-14T16:59:25.819470+0000","last_active":"2023-10-14T16:59:27.833893+0000","last_peered":"2023-10-14T16:59:27.833893+0000","last_clean":"2023-10-14T16:59:27.833893+0000","last_became_active":"2023-10-14T16:59:25.818906+0000","last_became_peered":"2023-10-14T16:59:25.818906+0000","last_unstale":"2023-10-14T16:59:27.833893+0000","last_undegraded":"2023-10-14T16:59:27.833893+0000","last_fullsized":"2023-10-14T16:59:27.833893+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:29:50.193783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828329+0000","last_change":"2023-10-14T16:59:26.832426+0000","last_active":"2023-10-14T16:59:27.828329+0000","last_peered":"2023-10-14T16:59:27.828329+0000","last_clean":"2023-10-14T16:59:27.828329+0000","last_became_active":"2023-10-14T16:59:26.832039+0000","last_became_peered":"2023-10-14T16:59:26.832039+0000","last_unstale":"2023-10-14T16:59:27.828329+0000","last_undegraded":"2023-10-14T16:59:27.828329+0000","last_fullsized":"2023-10-14T16:59:27.828329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:09.706336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"124'8","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.024862+0000","last_change":"2023-10-14T16:59:25.829739+0000","last_active":"2023-10-14T16:59:28.024862+0000","last_peered":"2023-10-14T16:59:28.024862+0000","last_clean":"2023-10-14T16:59:28.024862+0000","last_became_active":"2023-10-14T16:59:25.829471+0000","last_became_peered":"2023-10-14T16:59:25.829471+0000","last_unstale":"2023-10-14T16:59:28.024862+0000","last_undegraded":"2023-10-14T16:59:28.024862+0000","last_fullsized":"2023-10-14T16:59:28.024862+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:05.514211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834597+0000","last_change":"2023-10-14T16:59:26.824269+0000","last_active":"2023-10-14T16:59:27.834597+0000","last_peered":"2023-10-14T16:59:27.834597+0000","last_clean":"2023-10-14T16:59:27.834597+0000","last_became_active":"2023-10-14T16:59:26.823966+0000","last_became_peered":"2023-10-14T16:59:26.823966+0000","last_unstale":"2023-10-14T16:59:27.834597+0000","last_undegraded":"2023-10-14T16:59:27.834597+0000","last_fullsized":"2023-10-14T16:59:27.834597+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834500+0000","last_change":"2023-10-14T16:59:25.821235+0000","last_active":"2023-10-14T16:59:27.834500+0000","last_peered":"2023-10-14T16:59:27.834500+0000","last_clean":"2023-10-14T16:59:27.834500+0000","last_became_active":"2023-10-14T16:59:25.820946+0000","last_became_peered":"2023-10-14T16:59:25.820946+0000","last_unstale":"2023-10-14T16:59:27.834500+0000","last_undegraded":"2023-10-14T16:59:27.834500+0000","last_fullsized":"2023-10-14T16:59:27.834500+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:05.024120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028275+0000","last_change":"2023-10-14T16:59:26.834618+0000","last_active":"2023-10-14T16:59:28.028275+0000","last_peered":"2023-10-14T16:59:28.028275+0000","last_clean":"2023-10-14T16:59:28.028275+0000","last_became_active":"2023-10-14T16:59:26.833681+0000","last_became_peered":"2023-10-14T16:59:26.833681+0000","last_unstale":"2023-10-14T16:59:28.028275+0000","last_undegraded":"2023-10-14T16:59:28.028275+0000","last_fullsized":"2023-10-14T16:59:28.028275+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:32.136376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833685+0000","last_change":"2023-10-14T16:59:25.827686+0000","last_active":"2023-10-14T16:59:27.833685+0000","last_peered":"2023-10-14T16:59:27.833685+0000","last_clean":"2023-10-14T16:59:27.833685+0000","last_became_active":"2023-10-14T16:59:25.827075+0000","last_became_peered":"2023-10-14T16:59:25.827075+0000","last_unstale":"2023-10-14T16:59:27.833685+0000","last_undegraded":"2023-10-14T16:59:27.833685+0000","last_fullsized":"2023-10-14T16:59:27.833685+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:33.544429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028892+0000","last_change":"2023-10-14T16:59:26.829685+0000","last_active":"2023-10-14T16:59:28.028892+0000","last_peered":"2023-10-14T16:59:28.028892+0000","last_clean":"2023-10-14T16:59:28.028892+0000","last_became_active":"2023-10-14T16:59:26.829407+0000","last_became_peered":"2023-10-14T16:59:26.829407+0000","last_unstale":"2023-10-14T16:59:28.028892+0000","last_undegraded":"2023-10-14T16:59:28.028892+0000","last_fullsized":"2023-10-14T16:59:28.028892+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:24.636655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027309+0000","last_change":"2023-10-14T16:59:25.827438+0000","last_active":"2023-10-14T16:59:28.027309+0000","last_peered":"2023-10-14T16:59:28.027309+0000","last_clean":"2023-10-14T16:59:28.027309+0000","last_became_active":"2023-10-14T16:59:25.826774+0000","last_became_peered":"2023-10-14T16:59:25.826774+0000","last_unstale":"2023-10-14T16:59:28.027309+0000","last_undegraded":"2023-10-14T16:59:28.027309+0000","last_fullsized":"2023-10-14T16:59:28.027309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:01.225446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828570+0000","last_change":"2023-10-14T16:59:26.843752+0000","last_active":"2023-10-14T16:59:27.828570+0000","last_peered":"2023-10-14T16:59:27.828570+0000","last_clean":"2023-10-14T16:59:27.828570+0000","last_became_active":"2023-10-14T16:59:26.843492+0000","last_became_peered":"2023-10-14T16:59:26.843492+0000","last_unstale":"2023-10-14T16:59:27.828570+0000","last_undegraded":"2023-10-14T16:59:27.828570+0000","last_fullsized":"2023-10-14T16:59:27.828570+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:19.093526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027113+0000","last_change":"2023-10-14T16:59:25.833911+0000","last_active":"2023-10-14T16:59:28.027113+0000","last_peered":"2023-10-14T16:59:28.027113+0000","last_clean":"2023-10-14T16:59:28.027113+0000","last_became_active":"2023-10-14T16:59:25.833619+0000","last_became_peered":"2023-10-14T16:59:25.833619+0000","last_unstale":"2023-10-14T16:59:28.027113+0000","last_undegraded":"2023-10-14T16:59:28.027113+0000","last_fullsized":"2023-10-14T16:59:28.027113+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:33.544429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828997+0000","last_change":"2023-10-14T16:59:26.834354+0000","last_active":"2023-10-14T16:59:27.828997+0000","last_peered":"2023-10-14T16:59:27.828997+0000","last_clean":"2023-10-14T16:59:27.828997+0000","last_became_active":"2023-10-14T16:59:26.834082+0000","last_became_peered":"2023-10-14T16:59:26.834082+0000","last_unstale":"2023-10-14T16:59:27.828997+0000","last_undegraded":"2023-10-14T16:59:27.828997+0000","last_fullsized":"2023-10-14T16:59:27.828997+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:34.558914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028571+0000","last_change":"2023-10-14T16:59:25.819027+0000","last_active":"2023-10-14T16:59:28.028571+0000","last_peered":"2023-10-14T16:59:28.028571+0000","last_clean":"2023-10-14T16:59:28.028571+0000","last_became_active":"2023-10-14T16:59:25.818445+0000","last_became_peered":"2023-10-14T16:59:25.818445+0000","last_unstale":"2023-10-14T16:59:28.028571+0000","last_undegraded":"2023-10-14T16:59:28.028571+0000","last_fullsized":"2023-10-14T16:59:28.028571+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:58:01.822034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828326+0000","last_change":"2023-10-14T16:59:26.842649+0000","last_active":"2023-10-14T16:59:27.828326+0000","last_peered":"2023-10-14T16:59:27.828326+0000","last_clean":"2023-10-14T16:59:27.828326+0000","last_became_active":"2023-10-14T16:59:26.842362+0000","last_became_peered":"2023-10-14T16:59:26.842362+0000","last_unstale":"2023-10-14T16:59:27.828326+0000","last_undegraded":"2023-10-14T16:59:27.828326+0000","last_fullsized":"2023-10-14T16:59:27.828326+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:55.593083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028436+0000","last_change":"2023-10-14T16:59:25.828774+0000","last_active":"2023-10-14T16:59:28.028436+0000","last_peered":"2023-10-14T16:59:28.028436+0000","last_clean":"2023-10-14T16:59:28.028436+0000","last_became_active":"2023-10-14T16:59:25.827658+0000","last_became_peered":"2023-10-14T16:59:25.827658+0000","last_unstale":"2023-10-14T16:59:28.028436+0000","last_undegraded":"2023-10-14T16:59:28.028436+0000","last_fullsized":"2023-10-14T16:59:28.028436+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:20:49.724370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.831980+0000","last_change":"2023-10-14T16:59:26.824040+0000","last_active":"2023-10-14T16:59:27.831980+0000","last_peered":"2023-10-14T16:59:27.831980+0000","last_clean":"2023-10-14T16:59:27.831980+0000","last_became_active":"2023-10-14T16:59:26.823748+0000","last_became_peered":"2023-10-14T16:59:26.823748+0000","last_unstale":"2023-10-14T16:59:27.831980+0000","last_undegraded":"2023-10-14T16:59:27.831980+0000","last_fullsized":"2023-10-14T16:59:27.831980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:04.971542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026525+0000","last_change":"2023-10-14T16:59:25.833466+0000","last_active":"2023-10-14T16:59:28.026525+0000","last_peered":"2023-10-14T16:59:28.026525+0000","last_clean":"2023-10-14T16:59:28.026525+0000","last_became_active":"2023-10-14T16:59:25.833102+0000","last_became_peered":"2023-10-14T16:59:25.833102+0000","last_unstale":"2023-10-14T16:59:28.026525+0000","last_undegraded":"2023-10-14T16:59:28.026525+0000","last_fullsized":"2023-10-14T16:59:28.026525+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:59.894118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027574+0000","last_change":"2023-10-14T16:59:26.830254+0000","last_active":"2023-10-14T16:59:28.027574+0000","last_peered":"2023-10-14T16:59:28.027574+0000","last_clean":"2023-10-14T16:59:28.027574+0000","last_became_active":"2023-10-14T16:59:26.829510+0000","last_became_peered":"2023-10-14T16:59:26.829510+0000","last_unstale":"2023-10-14T16:59:28.027574+0000","last_undegraded":"2023-10-14T16:59:28.027574+0000","last_fullsized":"2023-10-14T16:59:28.027574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.029493+0000","last_change":"2023-10-14T16:59:25.828330+0000","last_active":"2023-10-14T16:59:28.029493+0000","last_peered":"2023-10-14T16:59:28.029493+0000","last_clean":"2023-10-14T16:59:28.029493+0000","last_became_active":"2023-10-14T16:59:25.826649+0000","last_became_peered":"2023-10-14T16:59:25.826649+0000","last_unstale":"2023-10-14T16:59:28.029493+0000","last_undegraded":"2023-10-14T16:59:28.029493+0000","last_fullsized":"2023-10-14T16:59:28.029493+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.410805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025273+0000","last_change":"2023-10-14T16:59:26.838082+0000","last_active":"2023-10-14T16:59:28.025273+0000","last_peered":"2023-10-14T16:59:28.025273+0000","last_clean":"2023-10-14T16:59:28.025273+0000","last_became_active":"2023-10-14T16:59:26.837421+0000","last_became_peered":"2023-10-14T16:59:26.837421+0000","last_unstale":"2023-10-14T16:59:28.025273+0000","last_undegraded":"2023-10-14T16:59:28.025273+0000","last_fullsized":"2023-10-14T16:59:28.025273+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027637+0000","last_change":"2023-10-14T16:59:25.833012+0000","last_active":"2023-10-14T16:59:28.027637+0000","last_peered":"2023-10-14T16:59:28.027637+0000","last_clean":"2023-10-14T16:59:28.027637+0000","last_became_active":"2023-10-14T16:59:25.832587+0000","last_became_peered":"2023-10-14T16:59:25.832587+0000","last_unstale":"2023-10-14T16:59:28.027637+0000","last_undegraded":"2023-10-14T16:59:28.027637+0000","last_fullsized":"2023-10-14T16:59:28.027637+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:46.907273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025042+0000","last_change":"2023-10-14T16:59:26.831084+0000","last_active":"2023-10-14T16:59:28.025042+0000","last_peered":"2023-10-14T16:59:28.025042+0000","last_clean":"2023-10-14T16:59:28.025042+0000","last_became_active":"2023-10-14T16:59:26.830747+0000","last_became_peered":"2023-10-14T16:59:26.830747+0000","last_unstale":"2023-10-14T16:59:28.025042+0000","last_undegraded":"2023-10-14T16:59:28.025042+0000","last_fullsized":"2023-10-14T16:59:28.025042+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027345+0000","last_change":"2023-10-14T16:59:25.829782+0000","last_active":"2023-10-14T16:59:28.027345+0000","last_peered":"2023-10-14T16:59:28.027345+0000","last_clean":"2023-10-14T16:59:28.027345+0000","last_became_active":"2023-10-14T16:59:25.828845+0000","last_became_peered":"2023-10-14T16:59:25.828845+0000","last_unstale":"2023-10-14T16:59:28.027345+0000","last_undegraded":"2023-10-14T16:59:28.027345+0000","last_fullsized":"2023-10-14T16:59:28.027345+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:54.578598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833158+0000","last_change":"2023-10-14T16:59:26.842333+0000","last_active":"2023-10-14T16:59:27.833158+0000","last_peered":"2023-10-14T16:59:27.833158+0000","last_clean":"2023-10-14T16:59:27.833158+0000","last_became_active":"2023-10-14T16:59:26.841940+0000","last_became_peered":"2023-10-14T16:59:26.841940+0000","last_unstale":"2023-10-14T16:59:27.833158+0000","last_undegraded":"2023-10-14T16:59:27.833158+0000","last_fullsized":"2023-10-14T16:59:27.833158+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835334+0000","last_change":"2023-10-14T16:59:25.830715+0000","last_active":"2023-10-14T16:59:27.835334+0000","last_peered":"2023-10-14T16:59:27.835334+0000","last_clean":"2023-10-14T16:59:27.835334+0000","last_became_active":"2023-10-14T16:59:25.830416+0000","last_became_peered":"2023-10-14T16:59:25.830416+0000","last_unstale":"2023-10-14T16:59:27.835334+0000","last_undegraded":"2023-10-14T16:59:27.835334+0000","last_fullsized":"2023-10-14T16:59:27.835334+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:01.225446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828979+0000","last_change":"2023-10-14T16:59:26.828796+0000","last_active":"2023-10-14T16:59:27.828979+0000","last_peered":"2023-10-14T16:59:27.828979+0000","last_clean":"2023-10-14T16:59:27.828979+0000","last_became_active":"2023-10-14T16:59:26.827950+0000","last_became_peered":"2023-10-14T16:59:26.827950+0000","last_unstale":"2023-10-14T16:59:27.828979+0000","last_undegraded":"2023-10-14T16:59:27.828979+0000","last_fullsized":"2023-10-14T16:59:27.828979+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:02.239931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027209+0000","last_change":"2023-10-14T16:59:25.830261+0000","last_active":"2023-10-14T16:59:28.027209+0000","last_peered":"2023-10-14T16:59:28.027209+0000","last_clean":"2023-10-14T16:59:28.027209+0000","last_became_active":"2023-10-14T16:59:25.829995+0000","last_became_peered":"2023-10-14T16:59:25.829995+0000","last_unstale":"2023-10-14T16:59:28.027209+0000","last_undegraded":"2023-10-14T16:59:28.027209+0000","last_fullsized":"2023-10-14T16:59:28.027209+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:18.079041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834564+0000","last_change":"2023-10-14T16:59:26.832998+0000","last_active":"2023-10-14T16:59:27.834564+0000","last_peered":"2023-10-14T16:59:27.834564+0000","last_clean":"2023-10-14T16:59:27.834564+0000","last_became_active":"2023-10-14T16:59:26.832647+0000","last_became_peered":"2023-10-14T16:59:26.832647+0000","last_unstale":"2023-10-14T16:59:27.834564+0000","last_undegraded":"2023-10-14T16:59:27.834564+0000","last_fullsized":"2023-10-14T16:59:27.834564+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:55.390471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832593+0000","last_change":"2023-10-14T16:59:25.823773+0000","last_active":"2023-10-14T16:59:27.832593+0000","last_peered":"2023-10-14T16:59:27.832593+0000","last_clean":"2023-10-14T16:59:27.832593+0000","last_became_active":"2023-10-14T16:59:25.823133+0000","last_became_peered":"2023-10-14T16:59:25.823133+0000","last_unstale":"2023-10-14T16:59:27.832593+0000","last_undegraded":"2023-10-14T16:59:27.832593+0000","last_fullsized":"2023-10-14T16:59:27.832593+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:05.514211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828325+0000","last_change":"2023-10-14T16:59:26.831575+0000","last_active":"2023-10-14T16:59:27.828325+0000","last_peered":"2023-10-14T16:59:27.828325+0000","last_clean":"2023-10-14T16:59:27.828325+0000","last_became_active":"2023-10-14T16:59:26.831242+0000","last_became_peered":"2023-10-14T16:59:26.831242+0000","last_unstale":"2023-10-14T16:59:27.828325+0000","last_undegraded":"2023-10-14T16:59:27.828325+0000","last_fullsized":"2023-10-14T16:59:27.828325+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:11:00.908603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026101+0000","last_change":"2023-10-14T16:59:25.822607+0000","last_active":"2023-10-14T16:59:28.026101+0000","last_peered":"2023-10-14T16:59:28.026101+0000","last_clean":"2023-10-14T16:59:28.026101+0000","last_became_active":"2023-10-14T16:59:25.822333+0000","last_became_peered":"2023-10-14T16:59:25.822333+0000","last_unstale":"2023-10-14T16:59:28.026101+0000","last_undegraded":"2023-10-14T16:59:28.026101+0000","last_fullsized":"2023-10-14T16:59:28.026101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:50:04.964615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832528+0000","last_change":"2023-10-14T16:59:26.837575+0000","last_active":"2023-10-14T16:59:27.832528+0000","last_peered":"2023-10-14T16:59:27.832528+0000","last_clean":"2023-10-14T16:59:27.832528+0000","last_became_active":"2023-10-14T16:59:26.837279+0000","last_became_peered":"2023-10-14T16:59:26.837279+0000","last_unstale":"2023-10-14T16:59:27.832528+0000","last_undegraded":"2023-10-14T16:59:27.832528+0000","last_fullsized":"2023-10-14T16:59:27.832528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828462+0000","last_change":"2023-10-14T16:59:25.829157+0000","last_active":"2023-10-14T16:59:27.828462+0000","last_peered":"2023-10-14T16:59:27.828462+0000","last_clean":"2023-10-14T16:59:27.828462+0000","last_became_active":"2023-10-14T16:59:25.828035+0000","last_became_peered":"2023-10-14T16:59:25.828035+0000","last_unstale":"2023-10-14T16:59:27.828462+0000","last_undegraded":"2023-10-14T16:59:27.828462+0000","last_fullsized":"2023-10-14T16:59:27.828462+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:30.973459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829558+0000","last_change":"2023-10-14T16:59:26.840023+0000","last_active":"2023-10-14T16:59:27.829558+0000","last_peered":"2023-10-14T16:59:27.829558+0000","last_clean":"2023-10-14T16:59:27.829558+0000","last_became_active":"2023-10-14T16:59:26.838025+0000","last_became_peered":"2023-10-14T16:59:26.838025+0000","last_unstale":"2023-10-14T16:59:27.829558+0000","last_undegraded":"2023-10-14T16:59:27.829558+0000","last_fullsized":"2023-10-14T16:59:27.829558+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027984+0000","last_change":"2023-10-14T16:59:25.833874+0000","last_active":"2023-10-14T16:59:28.027984+0000","last_peered":"2023-10-14T16:59:28.027984+0000","last_clean":"2023-10-14T16:59:28.027984+0000","last_became_active":"2023-10-14T16:59:25.833552+0000","last_became_peered":"2023-10-14T16:59:25.833552+0000","last_unstale":"2023-10-14T16:59:28.027984+0000","last_undegraded":"2023-10-14T16:59:28.027984+0000","last_fullsized":"2023-10-14T16:59:28.027984+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:27:27.739184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"113'205","reported_seq":574,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T16:59:26.806960+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:59:26.806960+0000","last_peered":"2023-10-14T16:59:26.806960+0000","last_clean":"2023-10-14T16:59:26.806960+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:59:26.806960+0000","last_undegraded":"2023-10-14T16:59:26.806960+0000","last_fullsized":"2023-10-14T16:59:26.806960+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":2146304,"data_stored":1336104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1699999999999999,"15min":1.1830000000000001},"min":{"1min":0.77900000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.607,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.163},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.115,"15min":1.198},"min":{"1min":0.876,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7370000000000001,"5min":1.871,"15min":2.117},"last":1.427}]},{"osd":1,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.177,"15min":1.2270000000000001},"min":{"1min":0.87,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":1.379},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.98199999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.7170000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.3009999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1739999999999999,"15min":1.222},"min":{"1min":0.89000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.647,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.25},{"interface":"front","average":{"1min":1.333,"5min":1.276,"15min":1.3049999999999999},"min":{"1min":1.012,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.575,"5min":1.893,"15min":2.1589999999999998},"last":1.4610000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.3169999999999999,"15min":1.367},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.988,"5min":2.056,"15min":2.4159999999999999},"last":1.339},{"interface":"front","average":{"1min":1.444,"5min":1.4179999999999999,"15min":1.45},"min":{"1min":1.0249999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.841,"5min":2.379,"15min":2.379},"last":1.6000000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2989999999999999,"15min":1.2789999999999999},"min":{"1min":0.85699999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.7370000000000001,"5min":2.105,"15min":2.1819999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.347,"15min":1.3340000000000001},"min":{"1min":0.93400000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.3819999999999999,"15min":1.3859999999999999},"min":{"1min":0.84999999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9299999999999999,"5min":2.278,"15min":2.278},"last":1.544},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.444,"15min":1.462},"min":{"1min":0.98199999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7909999999999999,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.5109999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5189999999999999,"15min":1.5429999999999999},"min":{"1min":1.1240000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.917,"5min":2.492,"15min":2.492},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.508,"15min":1.4870000000000001},"min":{"1min":1.014,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":2.3540000000000001,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.1140000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575076,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36856,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334984,"statfs":{"total":96636764160,"available":96599023616,"internally_reserved":0,"allocated":1552384,"data_stored":746276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.139,"15min":1.159},"min":{"1min":0.78400000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7330000000000001,"5min":2.137,"15min":2.137},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.1890000000000001,"15min":1.2110000000000001},"min":{"1min":0.85699999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.806,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.1519999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.169,"15min":1.177},"min":{"1min":0.90900000000000003,"5min":0.73499999999999999,"15min":0.41299999999999998},"max":{"1min":1.359,"5min":2.802,"15min":2.802},"last":1.099},{"interface":"front","average":{"1min":1.286,"5min":1.258,"15min":1.2749999999999999},"min":{"1min":0.86399999999999999,"5min":0.67300000000000004,"15min":0.438},"max":{"1min":1.9810000000000001,"5min":2.512,"15min":2.512},"last":1.5009999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.361,"15min":1.3500000000000001},"min":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.7669999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3460000000000001,"15min":1.383},"min":{"1min":1.125,"5min":0.81299999999999994,"15min":0.46000000000000002},"max":{"1min":1.786,"5min":3.286,"15min":3.286},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.345,"15min":1.3660000000000001},"min":{"1min":0.90500000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.823,"15min":2.823},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.407},"min":{"1min":1.071,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7350000000000001,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.27}]},{"osd":4,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.391,"15min":1.3400000000000001},"min":{"1min":0.92600000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.9299999999999999,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.47,"15min":1.4119999999999999},"min":{"1min":1.1040000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8660000000000001,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.85599999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.891,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5760000000000001,"15min":1.5509999999999999},"min":{"1min":1.0980000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.028,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.585}]},{"osd":7,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.603,"15min":1.5800000000000001},"min":{"1min":1.2070000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.9419999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.538},{"interface":"front","average":{"1min":1.583,"5min":1.5289999999999999,"15min":1.5329999999999999},"min":{"1min":1.153,"5min":0.623,"15min":0.623},"max":{"1min":1.9790000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.3500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":739350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.079,"15min":1.1080000000000001},"min":{"1min":0.71199999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.486,"5min":1.861,"15min":1.986},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.107,"15min":1.133},"min":{"1min":0.88400000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":2.0430000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1850000000000001,"15min":1.1759999999999999},"min":{"1min":0.84499999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":2.1200000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.25,"5min":1.1739999999999999,"15min":1.206},"min":{"1min":0.746,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.905},"last":1.1419999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.113,"15min":1.1819999999999999},"min":{"1min":0.65900000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5529999999999999,"5min":1.7150000000000001,"15min":2.375},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3140000000000001,"15min":1.3180000000000001},"min":{"1min":0.81399999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6319999999999999,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":1.3939999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.347,"15min":1.3839999999999999},"min":{"1min":0.83699999999999997,"5min":0.77200000000000002,"15min":0.745},"max":{"1min":1.8600000000000001,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.492},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3049999999999999,"15min":1.3899999999999999},"min":{"1min":0.86899999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8149999999999999,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":1.6579999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3089999999999999,"15min":1.2889999999999999},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.744,"5min":2.194,"15min":2.4889999999999999},"last":1.575},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.3540000000000001,"15min":1.3400000000000001},"min":{"1min":0.90100000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.748,"5min":1.927,"15min":2.5169999999999999},"last":1.333}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2809999999999999,"15min":1.3540000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6399999999999999,"5min":1.9650000000000001,"15min":2.302},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5109999999999999,"15min":1.53},"min":{"1min":0.85299999999999998,"5min":0.63,"15min":0.63},"max":{"1min":1.8220000000000001,"5min":2.1059999999999999,"15min":2.613},"last":1.512}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4730000000000001,"15min":1.506},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.956,"5min":2.032,"15min":2.605},"last":1.625},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.452,"15min":1.4970000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.8129999999999999,"5min":1.8979999999999999,"15min":2.3980000000000001},"last":1.548}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1564672,"data_stored":747360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.133,"15min":1.169},"min":{"1min":0.94599999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4950000000000001,"5min":1.9319999999999999,"15min":1.982},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.2669999999999999,"15min":1.238},"min":{"1min":0.83999999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.097,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":1.0669999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3160000000000001,"15min":1.29},"min":{"1min":0.74099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7030000000000001,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.194},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.242,"15min":1.254},"min":{"1min":0.69599999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.49,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.329}]},{"osd":2,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.93899999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6339999999999999,"5min":5.633,"15min":5.633},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.3799999999999999,"15min":1.377},"min":{"1min":1.05,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.242,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.282}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.304,"15min":1.3200000000000001},"min":{"1min":0.997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6910000000000001,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.401},{"interface":"front","average":{"1min":1.504,"5min":1.3879999999999999,"15min":1.4119999999999999},"min":{"1min":1.1419999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8560000000000001,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.6200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.3959999999999999,"15min":1.3600000000000001},"min":{"1min":1.0589999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.3849999999999998,"5min":2.931,"15min":2.931},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.339,"15min":1.3500000000000001},"min":{"1min":0.93200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.177,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.1399999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.4299999999999999,"15min":1.3979999999999999},"min":{"1min":0.97399999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9159999999999999,"5min":2.323,"15min":2.323},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.651,"5min":1.542,"15min":1.532},"min":{"1min":0.94999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.4620000000000002,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.359}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.5209999999999999,"15min":1.496},"min":{"1min":0.88400000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.3039999999999998,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.222},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.556,"15min":1.552},"min":{"1min":1.054,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.4450000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1560576,"data_stored":747258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.86799999999999999,"15min":0.92600000000000005},"min":{"1min":0.72099999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.393,"5min":1.621,"15min":1.621},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0529999999999999,"15min":1.131},"min":{"1min":0.67900000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4419999999999999,"5min":1.593,"15min":1.8799999999999999},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.86099999999999999,"15min":0.91400000000000003},"min":{"1min":0.53400000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7609999999999999,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.165,"5min":1.056,"15min":1.1140000000000001},"min":{"1min":0.61299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.7470000000000001,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":1.1359999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.236,"15min":1.357},"min":{"1min":0.64600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7569999999999999,"5min":2.024,"15min":2.0609999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.218,"15min":1.284},"min":{"1min":0.73499999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.0960000000000001},"last":1.883}]},{"osd":4,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.343,"15min":1.365},"min":{"1min":0.98999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8700000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.3660000000000001,"15min":1.417},"min":{"1min":0.90100000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.843,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.4530000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.3520000000000001,"15min":1.3939999999999999},"min":{"1min":1.036,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.867,"5min":2.516,"15min":3.0190000000000001},"last":1.508},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4059999999999999,"15min":1.4430000000000001},"min":{"1min":1.0469999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.571}]},{"osd":6,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.454,"15min":1.478},"min":{"1min":1.107,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.125,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.534,"15min":1.5509999999999999},"min":{"1min":0.96799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.1909999999999998,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.804}]},{"osd":7,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.46,"15min":1.512},"min":{"1min":0.94499999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9670000000000001,"5min":2.871,"15min":2.875},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.5329999999999999,"15min":1.5580000000000001},"min":{"1min":1.2370000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.9790000000000001,"5min":2.907,"15min":2.907},"last":1.756}]}]},{"osd":2,"up_from":13,"seq":55834575077,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45176,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326664,"statfs":{"total":96636764160,"available":96590503936,"internally_reserved":0,"allocated":1548288,"data_stored":745658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.85599999999999998,"15min":0.90200000000000002},"min":{"1min":0.67600000000000005,"5min":0.316,"15min":0.316},"max":{"1min":1.6699999999999999,"5min":1.706,"15min":1.7649999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.048,"5min":0.97599999999999998,"15min":1.0409999999999999},"min":{"1min":0.76800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.7170000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.83999999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.90600000000000003,"15min":0.97699999999999998},"min":{"1min":0.73399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.544,"5min":1.544,"15min":2.0419999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.254,"5min":1.115,"15min":1.173},"min":{"1min":0.90200000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5569999999999999,"5min":2.0390000000000001,"15min":2.069},"last":1.4870000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1100000000000001,"15min":1.2190000000000001},"min":{"1min":0.72499999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.696,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.1870000000000001,"15min":1.3},"min":{"1min":1.0720000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.657,"5min":1.7729999999999999,"15min":2.0459999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.3200000000000001,"15min":1.413},"min":{"1min":1.1870000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.784,"5min":1.913,"15min":2.3159999999999998},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.2689999999999999,"15min":1.3580000000000001},"min":{"1min":0.90700000000000003,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8400000000000001,"5min":2.0129999999999999,"15min":2.0750000000000002},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3129999999999999,"15min":1.389},"min":{"1min":1.0600000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.698,"5min":2.173,"15min":2.173},"last":1.069},{"interface":"front","average":{"1min":1.55,"5min":1.4239999999999999,"15min":1.5209999999999999},"min":{"1min":1.1719999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.889,"5min":1.889,"15min":2.4689999999999999},"last":1.3029999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.345,"15min":1.4350000000000001},"min":{"1min":1.018,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0289999999999999},"last":1.026},{"interface":"front","average":{"1min":1.516,"5min":1.444,"15min":1.5149999999999999},"min":{"1min":1.2729999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.802,"5min":2.0720000000000001,"15min":2.105},"last":1.224}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.417,"15min":1.5189999999999999},"min":{"1min":1.034,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.9099999999999999,"5min":2.0129999999999999,"15min":2.109},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.502,"15min":1.5780000000000001},"min":{"1min":1.1990000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.8819999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.359}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59056,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312784,"statfs":{"total":96636764160,"available":96576290816,"internally_reserved":0,"allocated":1544192,"data_stored":745624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.85399999999999998,"15min":0.90400000000000003},"min":{"1min":0.501,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97099999999999997,"5min":2.113,"15min":2.113},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.05,"5min":0.99099999999999999,"15min":1.0669999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.419,"5min":1.728,"15min":2.0819999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.2929999999999999,"5min":1.7509999999999999,"15min":1.758},"last":1.048},{"interface":"front","average":{"1min":1.226,"5min":1.1180000000000001,"15min":1.1719999999999999},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.51300000000000001},"max":{"1min":1.6339999999999999,"5min":2.008,"15min":2.0150000000000001},"last":1.4059999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.216,"15min":1.296},"min":{"1min":0.77100000000000002,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.7949999999999999,"5min":1.984,"15min":2.198},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.127,"15min":1.1919999999999999},"min":{"1min":0.73599999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.974,"5min":1.974,"15min":2.004},"last":1.2889999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.365,"15min":1.4039999999999999},"min":{"1min":0.871,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.8799999999999999,"5min":2.052,"15min":2.3050000000000002},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3080000000000001,"15min":1.323},"min":{"1min":1.008,"5min":0.66900000000000004,"15min":0.53900000000000003},"max":{"1min":1.9119999999999999,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.339}]},{"osd":5,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3240000000000001,"15min":1.373},"min":{"1min":0.84499999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9399999999999999,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":1.194},{"interface":"front","average":{"1min":1.452,"5min":1.389,"15min":1.458},"min":{"1min":1.155,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.911,"5min":2.1480000000000001,"15min":2.302},"last":1.746}]},{"osd":6,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4399999999999999,"15min":1.4950000000000001},"min":{"1min":1.246,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.4169999999999998},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4370000000000001,"15min":1.4770000000000001},"min":{"1min":0.92500000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.833,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":1.4590000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.492,"15min":1.5629999999999999},"min":{"1min":1.2889999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.901,"5min":2.2410000000000001,"15min":2.411},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.498,"15min":1.5760000000000001},"min":{"1min":0.98999999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":1.4910000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2142208,"data_stored":1336574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.89800000000000002},"min":{"1min":0.56200000000000006,"5min":0.309,"15min":0.309},"max":{"1min":1.403,"5min":3.355,"15min":3.355},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.99099999999999999,"15min":1.0840000000000001},"min":{"1min":0.78600000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7270000000000001,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0629999999999999,"15min":1.087},"min":{"1min":0.62,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.379,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.107,"15min":1.196},"min":{"1min":0.71299999999999997,"5min":0.432,"15min":0.432},"max":{"1min":1.6950000000000001,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.84999999999999998}]},{"osd":3,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.238,"15min":1.2789999999999999},"min":{"1min":0.77000000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.7549999999999999,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.252,"15min":1.3480000000000001},"min":{"1min":0.76600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9219999999999999,"5min":2.593,"15min":2.593},"last":1.095}]},{"osd":4,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.276,"15min":1.2849999999999999},"min":{"1min":0.81000000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6579999999999999,"5min":5.335,"15min":5.335},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.83999999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7290000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.0269999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2669999999999999,"15min":1.3160000000000001},"min":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7949999999999999,"5min":2.931,"15min":2.931},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4279999999999999,"15min":1.5069999999999999},"min":{"1min":1.0960000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.254,"5min":3.306,"15min":3.306},"last":1.1279999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5049999999999999,"15min":1.5840000000000001},"min":{"1min":1.054,"5min":0.92500000000000004,"15min":0.88900000000000001},"max":{"1min":2.4889999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3580000000000001,"15min":1.399},"min":{"1min":0.83299999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.732,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.4390000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.552},"min":{"1min":1.04,"5min":0.747,"15min":0.747},"max":{"1min":1.8819999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.482,"15min":1.52},"min":{"1min":0.78400000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.875,"5min":3.698,"15min":3.698},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:59:31.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T16:59:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.639+0000 7f381b341640 1 Processor -- start 2023-10-14T16:59:31.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.639+0000 7f381b341640 1 -- start start 2023-10-14T16:59:31.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f381415aaa0 0x7f381415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 0x7f38140cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38140ccaa0 0x7f3814070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381406b810 con 0x7f381415aaa0 2023-10-14T16:59:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381415ed60 con 0x7f381415ae80 2023-10-14T16:59:31.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.640+0000 7f381b341640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38140707e0 con 0x7f38140ccaa0 2023-10-14T16:59:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f381415aaa0 0x7f381415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f381415aaa0 0x7f381415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52634/0 (socket says 172.21.15.29:52634) 2023-10-14T16:59:31.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 -- 172.21.15.29:0/1311137252 learned_addr learned my addr 172.21.15.29:0/1311137252 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 -- 172.21.15.29:0/1311137252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38140ccaa0 msgr2=0x7f3814070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 --2- 172.21.15.29:0/1311137252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38140ccaa0 0x7f3814070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 -- 172.21.15.29:0/1311137252 >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 msgr2=0x7f38140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:31.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38188dc640 1 --2- 172.21.15.29:0/1311137252 >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 0x7f38140cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 --2- 172.21.15.29:0/1311137252 >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 0x7f38140cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38190dd640 1 -- 172.21.15.29:0/1311137252 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381415ef90 con 0x7f381415aaa0 2023-10-14T16:59:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.641+0000 7f38188dc640 1 --2- 172.21.15.29:0/1311137252 >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 0x7f38140cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.642+0000 7f38190dd640 1 --2- 172.21.15.29:0/1311137252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f381415e820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f38140c59c0 tx=0x7f380402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a64d67d8ae8685db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.643+0000 7f380bfff640 1 -- 172.21.15.29:0/1311137252 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804031870 con 0x7f381415aaa0 2023-10-14T16:59:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.643+0000 7f380bfff640 1 -- 172.21.15.29:0/1311137252 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38040319d0 con 0x7f381415aaa0 2023-10-14T16:59:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.643+0000 7f380bfff640 1 -- 172.21.15.29:0/1311137252 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3804031d60 con 0x7f381415aaa0 2023-10-14T16:59:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.643+0000 7f381b341640 1 -- 172.21.15.29:0/1311137252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 msgr2=0x7f381415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.643+0000 7f381b341640 1 --2- 172.21.15.29:0/1311137252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f381415e820 secure :-1 s=READY pgs=1405 cs=0 l=1 rev1=1 crypto rx=0x7f38140c59c0 tx=0x7f380402b910 comp rx=0 tx=0).stop 2023-10-14T16:59:31.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 -- 172.21.15.29:0/1311137252 shutdown_connections 2023-10-14T16:59:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 --2- 172.21.15.29:0/1311137252 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38140ccaa0 0x7f3814070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 --2- 172.21.15.29:0/1311137252 >> v2:172.21.15.186:3300/0 conn(0x7f381415ae80 0x7f38140cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 --2- 172.21.15.29:0/1311137252 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f381415e820 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 -- 172.21.15.29:0/1311137252 >> 172.21.15.29:0/1311137252 conn(0x7f38140cb410 msgr2=0x7f38140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:31.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 -- 172.21.15.29:0/1311137252 shutdown_connections 2023-10-14T16:59:31.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.644+0000 7f381b341640 1 -- 172.21.15.29:0/1311137252 wait complete. 2023-10-14T16:59:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.645+0000 7f381b341640 1 Processor -- start 2023-10-14T16:59:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.646+0000 7f381b341640 1 -- start start 2023-10-14T16:59:31.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.646+0000 7f381b341640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 0x7f381414de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.646+0000 7f381b341640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f381b341640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f381415ae80 0x7f38140c86a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f381b341640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381415df50 con 0x7f381415aaa0 2023-10-14T16:59:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f381b341640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38140ca600 con 0x7f38140ccaa0 2023-10-14T16:59:31.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f381b341640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381414e3b0 con 0x7f381415ae80 2023-10-14T16:59:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52648/0 (socket says 172.21.15.29:52648) 2023-10-14T16:59:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 -- 172.21.15.29:0/441820165 learned_addr learned my addr 172.21.15.29:0/441820165 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38190dd640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 0x7f381414de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 -- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f381415ae80 msgr2=0x7f38140c86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38198de640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f381415ae80 0x7f38140c86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f381415ae80 0x7f38140c86a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 -- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 msgr2=0x7f381414de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 0x7f381414de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38188dc640 1 -- 172.21.15.29:0/441820165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38040094d0 con 0x7f381415aaa0 2023-10-14T16:59:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.647+0000 7f38190dd640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 0x7f381414de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f38188dc640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f381000c7f0 tx=0x7f381000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f381000cea0 con 0x7f381415aaa0 2023-10-14T16:59:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3810007a30 con 0x7f381415aaa0 2023-10-14T16:59:31.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381415d990 con 0x7f381415aaa0 2023-10-14T16:59:31.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810007dc0 con 0x7f381415aaa0 2023-10-14T16:59:31.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.648+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38140c5dd0 con 0x7f381415aaa0 2023-10-14T16:59:31.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.649+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38140c5c30 con 0x7f381415aaa0 2023-10-14T16:59:31.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.650+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f38100113f0 con 0x7f381415aaa0 2023-10-14T16:59:31.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.651+0000 7f3809ffb640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 0x7f380006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:31.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.651+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f3810089a20 con 0x7f381415aaa0 2023-10-14T16:59:31.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.651+0000 7f38190dd640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 0x7f380006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:31.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.652+0000 7f38190dd640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 0x7f380006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3804036000 tx=0x7f380403c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:31.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.655+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f38140c5c30 con 0x7f381415aaa0 2023-10-14T16:59:31.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.849+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 --> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f381415f550 con 0x7f380006af90 2023-10-14T16:59:31.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.872+0000 7f3809ffb640 1 -- 172.21.15.29:0/441820165 <== mgr.7797 v2:172.21.15.29:6832/2416849365 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287014 (secure 0 0 0) 0x7f3804031cc0 con 0x7f380006af90 2023-10-14T16:59:31.877 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:31.878 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T16:59:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.878+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 msgr2=0x7f380006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.878+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 0x7f380006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3804036000 tx=0x7f380403c040 comp rx=0 tx=0).stop 2023-10-14T16:59:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.878+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 msgr2=0x7f38140c8160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.878+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 crypto rx=0x7f381000c7f0 tx=0x7f381000ccb0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 shutdown_connections 2023-10-14T16:59:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f380006af90 0x7f380006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f381415ae80 0x7f38140c86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f381415aaa0 0x7f38140c8160 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 --2- 172.21.15.29:0/441820165 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f38140ccaa0 0x7f381414de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 >> 172.21.15.29:0/441820165 conn(0x7f38140cb410 msgr2=0x7f381406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 shutdown_connections 2023-10-14T16:59:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:31.879+0000 7f381b341640 1 -- 172.21.15.29:0/441820165 wait complete. 2023-10-14T16:59:31.901 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T16:59:31.798672+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":363,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":243,"ondisk_log_size":243,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":177,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":348744,"kb_used_data":13276,"kb_used_omap":12,"kb_used_meta":335347,"kb_avail":754625976,"statfs":{"total":773094113280,"available":772736999424,"internally_reserved":0,"allocated":13594624,"data_stored":7144204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":343395896},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":29,"apply_latency_ms":29,"commit_latency_ns":29000000,"apply_latency_ns":29000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002964"},"pg_stats":[{"pgid":"15.3f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025134+0000","last_change":"2023-10-14T16:59:26.833468+0000","last_active":"2023-10-14T16:59:28.025134+0000","last_peered":"2023-10-14T16:59:28.025134+0000","last_clean":"2023-10-14T16:59:28.025134+0000","last_became_active":"2023-10-14T16:59:26.833202+0000","last_became_peered":"2023-10-14T16:59:26.833202+0000","last_unstale":"2023-10-14T16:59:28.025134+0000","last_undegraded":"2023-10-14T16:59:28.025134+0000","last_fullsized":"2023-10-14T16:59:28.025134+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027284+0000","last_change":"2023-10-14T16:59:26.830463+0000","last_active":"2023-10-14T16:59:28.027284+0000","last_peered":"2023-10-14T16:59:28.027284+0000","last_clean":"2023-10-14T16:59:28.027284+0000","last_became_active":"2023-10-14T16:59:26.829935+0000","last_became_peered":"2023-10-14T16:59:26.829935+0000","last_unstale":"2023-10-14T16:59:28.027284+0000","last_undegraded":"2023-10-14T16:59:28.027284+0000","last_fullsized":"2023-10-14T16:59:28.027284+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025283+0000","last_change":"2023-10-14T16:59:26.833878+0000","last_active":"2023-10-14T16:59:28.025283+0000","last_peered":"2023-10-14T16:59:28.025283+0000","last_clean":"2023-10-14T16:59:28.025283+0000","last_became_active":"2023-10-14T16:59:26.832776+0000","last_became_peered":"2023-10-14T16:59:26.832776+0000","last_unstale":"2023-10-14T16:59:28.025283+0000","last_undegraded":"2023-10-14T16:59:28.025283+0000","last_fullsized":"2023-10-14T16:59:28.025283+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.3c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026490+0000","last_change":"2023-10-14T16:59:26.844268+0000","last_active":"2023-10-14T16:59:28.026490+0000","last_peered":"2023-10-14T16:59:28.026490+0000","last_clean":"2023-10-14T16:59:28.026490+0000","last_became_active":"2023-10-14T16:59:26.843557+0000","last_became_peered":"2023-10-14T16:59:26.843557+0000","last_unstale":"2023-10-14T16:59:28.026490+0000","last_undegraded":"2023-10-14T16:59:28.026490+0000","last_fullsized":"2023-10-14T16:59:28.026490+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.418824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.3b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833986+0000","last_change":"2023-10-14T16:59:26.836961+0000","last_active":"2023-10-14T16:59:27.833986+0000","last_peered":"2023-10-14T16:59:27.833986+0000","last_clean":"2023-10-14T16:59:27.833986+0000","last_became_active":"2023-10-14T16:59:26.836686+0000","last_became_peered":"2023-10-14T16:59:26.836686+0000","last_unstale":"2023-10-14T16:59:27.833986+0000","last_undegraded":"2023-10-14T16:59:27.833986+0000","last_fullsized":"2023-10-14T16:59:27.833986+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.3a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027771+0000","last_change":"2023-10-14T16:59:26.830441+0000","last_active":"2023-10-14T16:59:28.027771+0000","last_peered":"2023-10-14T16:59:28.027771+0000","last_clean":"2023-10-14T16:59:28.027771+0000","last_became_active":"2023-10-14T16:59:26.830126+0000","last_became_peered":"2023-10-14T16:59:26.830126+0000","last_unstale":"2023-10-14T16:59:28.027771+0000","last_undegraded":"2023-10-14T16:59:28.027771+0000","last_fullsized":"2023-10-14T16:59:28.027771+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:31:04.254209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.39","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027680+0000","last_change":"2023-10-14T16:59:26.835428+0000","last_active":"2023-10-14T16:59:28.027680+0000","last_peered":"2023-10-14T16:59:28.027680+0000","last_clean":"2023-10-14T16:59:28.027680+0000","last_became_active":"2023-10-14T16:59:26.834940+0000","last_became_peered":"2023-10-14T16:59:26.834940+0000","last_unstale":"2023-10-14T16:59:28.027680+0000","last_undegraded":"2023-10-14T16:59:28.027680+0000","last_fullsized":"2023-10-14T16:59:28.027680+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:25.591632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.38","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026311+0000","last_change":"2023-10-14T16:59:26.831040+0000","last_active":"2023-10-14T16:59:28.026311+0000","last_peered":"2023-10-14T16:59:28.026311+0000","last_clean":"2023-10-14T16:59:28.026311+0000","last_became_active":"2023-10-14T16:59:26.830617+0000","last_became_peered":"2023-10-14T16:59:26.830617+0000","last_unstale":"2023-10-14T16:59:28.026311+0000","last_undegraded":"2023-10-14T16:59:28.026311+0000","last_fullsized":"2023-10-14T16:59:28.026311+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.37","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833406+0000","last_change":"2023-10-14T16:59:26.841845+0000","last_active":"2023-10-14T16:59:27.833406+0000","last_peered":"2023-10-14T16:59:27.833406+0000","last_clean":"2023-10-14T16:59:27.833406+0000","last_became_active":"2023-10-14T16:59:26.841487+0000","last_became_peered":"2023-10-14T16:59:26.841487+0000","last_unstale":"2023-10-14T16:59:27.833406+0000","last_undegraded":"2023-10-14T16:59:27.833406+0000","last_fullsized":"2023-10-14T16:59:27.833406+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:20:50.738855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.36","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833046+0000","last_change":"2023-10-14T16:59:26.822372+0000","last_active":"2023-10-14T16:59:27.833046+0000","last_peered":"2023-10-14T16:59:27.833046+0000","last_clean":"2023-10-14T16:59:27.833046+0000","last_became_active":"2023-10-14T16:59:26.822022+0000","last_became_peered":"2023-10-14T16:59:26.822022+0000","last_unstale":"2023-10-14T16:59:27.833046+0000","last_undegraded":"2023-10-14T16:59:27.833046+0000","last_fullsized":"2023-10-14T16:59:27.833046+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.35","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027992+0000","last_change":"2023-10-14T16:59:26.831067+0000","last_active":"2023-10-14T16:59:28.027992+0000","last_peered":"2023-10-14T16:59:28.027992+0000","last_clean":"2023-10-14T16:59:28.027992+0000","last_became_active":"2023-10-14T16:59:26.830799+0000","last_became_peered":"2023-10-14T16:59:26.830799+0000","last_unstale":"2023-10-14T16:59:28.027992+0000","last_undegraded":"2023-10-14T16:59:28.027992+0000","last_fullsized":"2023-10-14T16:59:28.027992+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:30.567980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.34","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833343+0000","last_change":"2023-10-14T16:59:26.827015+0000","last_active":"2023-10-14T16:59:27.833343+0000","last_peered":"2023-10-14T16:59:27.833343+0000","last_clean":"2023-10-14T16:59:27.833343+0000","last_became_active":"2023-10-14T16:59:26.826717+0000","last_became_peered":"2023-10-14T16:59:26.826717+0000","last_unstale":"2023-10-14T16:59:27.833343+0000","last_undegraded":"2023-10-14T16:59:27.833343+0000","last_fullsized":"2023-10-14T16:59:27.833343+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:58:02.836519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.33","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026056+0000","last_change":"2023-10-14T16:59:26.836746+0000","last_active":"2023-10-14T16:59:28.026056+0000","last_peered":"2023-10-14T16:59:28.026056+0000","last_clean":"2023-10-14T16:59:28.026056+0000","last_became_active":"2023-10-14T16:59:26.836463+0000","last_became_peered":"2023-10-14T16:59:26.836463+0000","last_unstale":"2023-10-14T16:59:28.026056+0000","last_undegraded":"2023-10-14T16:59:28.026056+0000","last_fullsized":"2023-10-14T16:59:28.026056+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:27:10.626221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.32","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828347+0000","last_change":"2023-10-14T16:59:26.832929+0000","last_active":"2023-10-14T16:59:27.828347+0000","last_peered":"2023-10-14T16:59:27.828347+0000","last_clean":"2023-10-14T16:59:27.828347+0000","last_became_active":"2023-10-14T16:59:26.832555+0000","last_became_peered":"2023-10-14T16:59:26.832555+0000","last_unstale":"2023-10-14T16:59:27.828347+0000","last_undegraded":"2023-10-14T16:59:27.828347+0000","last_fullsized":"2023-10-14T16:59:27.828347+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:06.528696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.31","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025926+0000","last_change":"2023-10-14T16:59:26.844457+0000","last_active":"2023-10-14T16:59:28.025926+0000","last_peered":"2023-10-14T16:59:28.025926+0000","last_clean":"2023-10-14T16:59:28.025926+0000","last_became_active":"2023-10-14T16:59:26.843998+0000","last_became_peered":"2023-10-14T16:59:26.843998+0000","last_unstale":"2023-10-14T16:59:28.025926+0000","last_undegraded":"2023-10-14T16:59:28.025926+0000","last_fullsized":"2023-10-14T16:59:28.025926+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:05.566426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.30","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027358+0000","last_change":"2023-10-14T16:59:26.827938+0000","last_active":"2023-10-14T16:59:28.027358+0000","last_peered":"2023-10-14T16:59:28.027358+0000","last_clean":"2023-10-14T16:59:28.027358+0000","last_became_active":"2023-10-14T16:59:26.827654+0000","last_became_peered":"2023-10-14T16:59:26.827654+0000","last_unstale":"2023-10-14T16:59:28.027358+0000","last_undegraded":"2023-10-14T16:59:28.027358+0000","last_fullsized":"2023-10-14T16:59:28.027358+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.418824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.2f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027574+0000","last_change":"2023-10-14T16:59:26.839529+0000","last_active":"2023-10-14T16:59:28.027574+0000","last_peered":"2023-10-14T16:59:28.027574+0000","last_clean":"2023-10-14T16:59:28.027574+0000","last_became_active":"2023-10-14T16:59:26.839266+0000","last_became_peered":"2023-10-14T16:59:26.839266+0000","last_unstale":"2023-10-14T16:59:28.027574+0000","last_undegraded":"2023-10-14T16:59:28.027574+0000","last_fullsized":"2023-10-14T16:59:28.027574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:45.122421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.2e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026518+0000","last_change":"2023-10-14T16:59:26.846505+0000","last_active":"2023-10-14T16:59:28.026518+0000","last_peered":"2023-10-14T16:59:28.026518+0000","last_clean":"2023-10-14T16:59:28.026518+0000","last_became_active":"2023-10-14T16:59:26.841179+0000","last_became_peered":"2023-10-14T16:59:26.841179+0000","last_unstale":"2023-10-14T16:59:28.026518+0000","last_undegraded":"2023-10-14T16:59:28.026518+0000","last_fullsized":"2023-10-14T16:59:28.026518+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:09.743350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.2d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829397+0000","last_change":"2023-10-14T16:59:26.831037+0000","last_active":"2023-10-14T16:59:27.829397+0000","last_peered":"2023-10-14T16:59:27.829397+0000","last_clean":"2023-10-14T16:59:27.829397+0000","last_became_active":"2023-10-14T16:59:26.830706+0000","last_became_peered":"2023-10-14T16:59:26.830706+0000","last_unstale":"2023-10-14T16:59:27.829397+0000","last_undegraded":"2023-10-14T16:59:27.829397+0000","last_fullsized":"2023-10-14T16:59:27.829397+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:47.921758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833719+0000","last_change":"2023-10-14T16:59:26.839109+0000","last_active":"2023-10-14T16:59:27.833719+0000","last_peered":"2023-10-14T16:59:27.833719+0000","last_clean":"2023-10-14T16:59:27.833719+0000","last_became_active":"2023-10-14T16:59:26.837731+0000","last_became_peered":"2023-10-14T16:59:26.837731+0000","last_unstale":"2023-10-14T16:59:27.833719+0000","last_undegraded":"2023-10-14T16:59:27.833719+0000","last_fullsized":"2023-10-14T16:59:27.833719+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:32.136376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.2b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829296+0000","last_change":"2023-10-14T16:59:26.829711+0000","last_active":"2023-10-14T16:59:27.829296+0000","last_peered":"2023-10-14T16:59:27.829296+0000","last_clean":"2023-10-14T16:59:27.829296+0000","last_became_active":"2023-10-14T16:59:26.829426+0000","last_became_peered":"2023-10-14T16:59:26.829426+0000","last_unstale":"2023-10-14T16:59:27.829296+0000","last_undegraded":"2023-10-14T16:59:27.829296+0000","last_fullsized":"2023-10-14T16:59:27.829296+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:31.674138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.2a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026931+0000","last_change":"2023-10-14T16:59:26.829499+0000","last_active":"2023-10-14T16:59:28.026931+0000","last_peered":"2023-10-14T16:59:28.026931+0000","last_clean":"2023-10-14T16:59:28.026931+0000","last_became_active":"2023-10-14T16:59:26.829201+0000","last_became_peered":"2023-10-14T16:59:26.829201+0000","last_unstale":"2023-10-14T16:59:28.026931+0000","last_undegraded":"2023-10-14T16:59:28.026931+0000","last_fullsized":"2023-10-14T16:59:28.026931+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.29","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026935+0000","last_change":"2023-10-14T16:59:26.827178+0000","last_active":"2023-10-14T16:59:28.026935+0000","last_peered":"2023-10-14T16:59:28.026935+0000","last_clean":"2023-10-14T16:59:28.026935+0000","last_became_active":"2023-10-14T16:59:26.826867+0000","last_became_peered":"2023-10-14T16:59:26.826867+0000","last_unstale":"2023-10-14T16:59:28.026935+0000","last_undegraded":"2023-10-14T16:59:28.026935+0000","last_fullsized":"2023-10-14T16:59:28.026935+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.28","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027725+0000","last_change":"2023-10-14T16:59:26.819042+0000","last_active":"2023-10-14T16:59:28.027725+0000","last_peered":"2023-10-14T16:59:28.027725+0000","last_clean":"2023-10-14T16:59:28.027725+0000","last_became_active":"2023-10-14T16:59:26.818257+0000","last_became_peered":"2023-10-14T16:59:26.818257+0000","last_unstale":"2023-10-14T16:59:28.027725+0000","last_undegraded":"2023-10-14T16:59:28.027725+0000","last_fullsized":"2023-10-14T16:59:28.027725+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:03:55.302030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.27","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834305+0000","last_change":"2023-10-14T16:59:26.833604+0000","last_active":"2023-10-14T16:59:27.834305+0000","last_peered":"2023-10-14T16:59:27.834305+0000","last_clean":"2023-10-14T16:59:27.834305+0000","last_became_active":"2023-10-14T16:59:26.833310+0000","last_became_peered":"2023-10-14T16:59:26.833310+0000","last_unstale":"2023-10-14T16:59:27.834305+0000","last_undegraded":"2023-10-14T16:59:27.834305+0000","last_fullsized":"2023-10-14T16:59:27.834305+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:29:25.751446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.26","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027426+0000","last_change":"2023-10-14T16:59:26.830208+0000","last_active":"2023-10-14T16:59:28.027426+0000","last_peered":"2023-10-14T16:59:28.027426+0000","last_clean":"2023-10-14T16:59:28.027426+0000","last_became_active":"2023-10-14T16:59:26.829137+0000","last_became_peered":"2023-10-14T16:59:26.829137+0000","last_unstale":"2023-10-14T16:59:28.027426+0000","last_undegraded":"2023-10-14T16:59:28.027426+0000","last_fullsized":"2023-10-14T16:59:28.027426+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.25","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028354+0000","last_change":"2023-10-14T16:59:26.835636+0000","last_active":"2023-10-14T16:59:28.028354+0000","last_peered":"2023-10-14T16:59:28.028354+0000","last_clean":"2023-10-14T16:59:28.028354+0000","last_became_active":"2023-10-14T16:59:26.834941+0000","last_became_peered":"2023-10-14T16:59:26.834941+0000","last_unstale":"2023-10-14T16:59:28.028354+0000","last_undegraded":"2023-10-14T16:59:28.028354+0000","last_fullsized":"2023-10-14T16:59:28.028354+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:33:01.607198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.24","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835074+0000","last_change":"2023-10-14T16:59:26.842610+0000","last_active":"2023-10-14T16:59:27.835074+0000","last_peered":"2023-10-14T16:59:27.835074+0000","last_clean":"2023-10-14T16:59:27.835074+0000","last_became_active":"2023-10-14T16:59:26.842190+0000","last_became_peered":"2023-10-14T16:59:26.842190+0000","last_unstale":"2023-10-14T16:59:27.835074+0000","last_undegraded":"2023-10-14T16:59:27.835074+0000","last_fullsized":"2023-10-14T16:59:27.835074+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.23","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027167+0000","last_change":"2023-10-14T16:59:26.827805+0000","last_active":"2023-10-14T16:59:28.027167+0000","last_peered":"2023-10-14T16:59:28.027167+0000","last_clean":"2023-10-14T16:59:28.027167+0000","last_became_active":"2023-10-14T16:59:26.827485+0000","last_became_peered":"2023-10-14T16:59:26.827485+0000","last_unstale":"2023-10-14T16:59:28.027167+0000","last_undegraded":"2023-10-14T16:59:28.027167+0000","last_fullsized":"2023-10-14T16:59:28.027167+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.22","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828353+0000","last_change":"2023-10-14T16:59:26.831691+0000","last_active":"2023-10-14T16:59:27.828353+0000","last_peered":"2023-10-14T16:59:27.828353+0000","last_clean":"2023-10-14T16:59:27.828353+0000","last_became_active":"2023-10-14T16:59:26.831327+0000","last_became_peered":"2023-10-14T16:59:26.831327+0000","last_unstale":"2023-10-14T16:59:27.828353+0000","last_undegraded":"2023-10-14T16:59:27.828353+0000","last_fullsized":"2023-10-14T16:59:27.828353+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:33:34.191228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.21","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026123+0000","last_change":"2023-10-14T16:59:26.834390+0000","last_active":"2023-10-14T16:59:28.026123+0000","last_peered":"2023-10-14T16:59:28.026123+0000","last_clean":"2023-10-14T16:59:28.026123+0000","last_became_active":"2023-10-14T16:59:26.833279+0000","last_became_peered":"2023-10-14T16:59:26.833279+0000","last_unstale":"2023-10-14T16:59:28.026123+0000","last_undegraded":"2023-10-14T16:59:28.026123+0000","last_fullsized":"2023-10-14T16:59:28.026123+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:28.965220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.20","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025056+0000","last_change":"2023-10-14T16:59:26.828525+0000","last_active":"2023-10-14T16:59:28.025056+0000","last_peered":"2023-10-14T16:59:28.025056+0000","last_clean":"2023-10-14T16:59:28.025056+0000","last_became_active":"2023-10-14T16:59:26.827994+0000","last_became_peered":"2023-10-14T16:59:26.827994+0000","last_unstale":"2023-10-14T16:59:28.025056+0000","last_undegraded":"2023-10-14T16:59:28.025056+0000","last_fullsized":"2023-10-14T16:59:28.025056+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:43.671644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829453+0000","last_change":"2023-10-14T16:59:26.836936+0000","last_active":"2023-10-14T16:59:27.829453+0000","last_peered":"2023-10-14T16:59:27.829453+0000","last_clean":"2023-10-14T16:59:27.829453+0000","last_became_active":"2023-10-14T16:59:26.836046+0000","last_became_peered":"2023-10-14T16:59:26.836046+0000","last_unstale":"2023-10-14T16:59:27.829453+0000","last_undegraded":"2023-10-14T16:59:27.829453+0000","last_fullsized":"2023-10-14T16:59:27.829453+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:04.971542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1f","version":"124'4","reported_seq":16,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027528+0000","last_change":"2023-10-14T16:59:25.831858+0000","last_active":"2023-10-14T16:59:28.027528+0000","last_peered":"2023-10-14T16:59:28.027528+0000","last_clean":"2023-10-14T16:59:28.027528+0000","last_became_active":"2023-10-14T16:59:25.831562+0000","last_became_peered":"2023-10-14T16:59:25.831562+0000","last_unstale":"2023-10-14T16:59:28.027528+0000","last_undegraded":"2023-10-14T16:59:28.027528+0000","last_fullsized":"2023-10-14T16:59:28.027528+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:03.957057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026660+0000","last_change":"2023-10-14T16:59:26.835727+0000","last_active":"2023-10-14T16:59:28.026660+0000","last_peered":"2023-10-14T16:59:28.026660+0000","last_clean":"2023-10-14T16:59:28.026660+0000","last_became_active":"2023-10-14T16:59:26.835453+0000","last_became_peered":"2023-10-14T16:59:26.835453+0000","last_unstale":"2023-10-14T16:59:28.026660+0000","last_undegraded":"2023-10-14T16:59:28.026660+0000","last_fullsized":"2023-10-14T16:59:28.026660+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:25.591632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.1e","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028593+0000","last_change":"2023-10-14T16:59:25.835359+0000","last_active":"2023-10-14T16:59:28.028593+0000","last_peered":"2023-10-14T16:59:28.028593+0000","last_clean":"2023-10-14T16:59:28.028593+0000","last_became_active":"2023-10-14T16:59:25.835118+0000","last_became_peered":"2023-10-14T16:59:25.835118+0000","last_unstale":"2023-10-14T16:59:28.028593+0000","last_undegraded":"2023-10-14T16:59:28.028593+0000","last_fullsized":"2023-10-14T16:59:28.028593+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:00:38.918023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.1c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025147+0000","last_change":"2023-10-14T16:59:26.824038+0000","last_active":"2023-10-14T16:59:28.025147+0000","last_peered":"2023-10-14T16:59:28.025147+0000","last_clean":"2023-10-14T16:59:28.025147+0000","last_became_active":"2023-10-14T16:59:26.823697+0000","last_became_peered":"2023-10-14T16:59:26.823697+0000","last_unstale":"2023-10-14T16:59:28.025147+0000","last_undegraded":"2023-10-14T16:59:28.025147+0000","last_fullsized":"2023-10-14T16:59:28.025147+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.1d","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027253+0000","last_change":"2023-10-14T16:59:25.834259+0000","last_active":"2023-10-14T16:59:28.027253+0000","last_peered":"2023-10-14T16:59:28.027253+0000","last_clean":"2023-10-14T16:59:28.027253+0000","last_became_active":"2023-10-14T16:59:25.833994+0000","last_became_peered":"2023-10-14T16:59:25.833994+0000","last_unstale":"2023-10-14T16:59:28.027253+0000","last_undegraded":"2023-10-14T16:59:28.027253+0000","last_fullsized":"2023-10-14T16:59:28.027253+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:41.997518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834836+0000","last_change":"2023-10-14T16:59:26.839609+0000","last_active":"2023-10-14T16:59:27.834836+0000","last_peered":"2023-10-14T16:59:27.834836+0000","last_clean":"2023-10-14T16:59:27.834836+0000","last_became_active":"2023-10-14T16:59:26.838166+0000","last_became_peered":"2023-10-14T16:59:26.838166+0000","last_unstale":"2023-10-14T16:59:27.834836+0000","last_undegraded":"2023-10-14T16:59:27.834836+0000","last_fullsized":"2023-10-14T16:59:27.834836+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:57.033678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.1c","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027213+0000","last_change":"2023-10-14T16:59:25.827668+0000","last_active":"2023-10-14T16:59:28.027213+0000","last_peered":"2023-10-14T16:59:28.027213+0000","last_clean":"2023-10-14T16:59:28.027213+0000","last_became_active":"2023-10-14T16:59:25.827152+0000","last_became_peered":"2023-10-14T16:59:25.827152+0000","last_unstale":"2023-10-14T16:59:28.027213+0000","last_undegraded":"2023-10-14T16:59:28.027213+0000","last_fullsized":"2023-10-14T16:59:28.027213+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:08.691851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.1a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028855+0000","last_change":"2023-10-14T16:59:26.835455+0000","last_active":"2023-10-14T16:59:28.028855+0000","last_peered":"2023-10-14T16:59:28.028855+0000","last_clean":"2023-10-14T16:59:28.028855+0000","last_became_active":"2023-10-14T16:59:26.835096+0000","last_became_peered":"2023-10-14T16:59:26.835096+0000","last_unstale":"2023-10-14T16:59:28.028855+0000","last_undegraded":"2023-10-14T16:59:28.028855+0000","last_fullsized":"2023-10-14T16:59:28.028855+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:08.624193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.1b","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025205+0000","last_change":"2023-10-14T16:59:25.829120+0000","last_active":"2023-10-14T16:59:28.025205+0000","last_peered":"2023-10-14T16:59:28.025205+0000","last_clean":"2023-10-14T16:59:28.025205+0000","last_became_active":"2023-10-14T16:59:25.826156+0000","last_became_peered":"2023-10-14T16:59:25.826156+0000","last_unstale":"2023-10-14T16:59:28.025205+0000","last_undegraded":"2023-10-14T16:59:28.025205+0000","last_fullsized":"2023-10-14T16:59:28.025205+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:46.907273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.1b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027582+0000","last_change":"2023-10-14T16:59:26.824727+0000","last_active":"2023-10-14T16:59:28.027582+0000","last_peered":"2023-10-14T16:59:28.027582+0000","last_clean":"2023-10-14T16:59:28.027582+0000","last_became_active":"2023-10-14T16:59:26.824131+0000","last_became_peered":"2023-10-14T16:59:26.824131+0000","last_unstale":"2023-10-14T16:59:28.027582+0000","last_undegraded":"2023-10-14T16:59:28.027582+0000","last_fullsized":"2023-10-14T16:59:28.027582+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:09.743350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.1a","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027006+0000","last_change":"2023-10-14T16:59:25.833297+0000","last_active":"2023-10-14T16:59:28.027006+0000","last_peered":"2023-10-14T16:59:28.027006+0000","last_clean":"2023-10-14T16:59:28.027006+0000","last_became_active":"2023-10-14T16:59:25.832997+0000","last_became_peered":"2023-10-14T16:59:25.832997+0000","last_unstale":"2023-10-14T16:59:28.027006+0000","last_undegraded":"2023-10-14T16:59:28.027006+0000","last_fullsized":"2023-10-14T16:59:28.027006+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.18","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025901+0000","last_change":"2023-10-14T16:59:26.833981+0000","last_active":"2023-10-14T16:59:28.025901+0000","last_peered":"2023-10-14T16:59:28.025901+0000","last_clean":"2023-10-14T16:59:28.025901+0000","last_became_active":"2023-10-14T16:59:26.833621+0000","last_became_peered":"2023-10-14T16:59:26.833621+0000","last_unstale":"2023-10-14T16:59:28.025901+0000","last_undegraded":"2023-10-14T16:59:28.025901+0000","last_fullsized":"2023-10-14T16:59:28.025901+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:37.096065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.19","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832914+0000","last_change":"2023-10-14T16:59:25.824007+0000","last_active":"2023-10-14T16:59:27.832914+0000","last_peered":"2023-10-14T16:59:27.832914+0000","last_clean":"2023-10-14T16:59:27.832914+0000","last_became_active":"2023-10-14T16:59:25.823704+0000","last_became_peered":"2023-10-14T16:59:25.823704+0000","last_unstale":"2023-10-14T16:59:27.832914+0000","last_undegraded":"2023-10-14T16:59:27.832914+0000","last_fullsized":"2023-10-14T16:59:27.832914+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.19","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833684+0000","last_change":"2023-10-14T16:59:26.833788+0000","last_active":"2023-10-14T16:59:27.833684+0000","last_peered":"2023-10-14T16:59:27.833684+0000","last_clean":"2023-10-14T16:59:27.833684+0000","last_became_active":"2023-10-14T16:59:26.833210+0000","last_became_peered":"2023-10-14T16:59:26.833210+0000","last_unstale":"2023-10-14T16:59:27.833684+0000","last_undegraded":"2023-10-14T16:59:27.833684+0000","last_fullsized":"2023-10-14T16:59:27.833684+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:55.425290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.18","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.024760+0000","last_change":"2023-10-14T16:59:25.827524+0000","last_active":"2023-10-14T16:59:28.024760+0000","last_peered":"2023-10-14T16:59:28.024760+0000","last_clean":"2023-10-14T16:59:28.024760+0000","last_became_active":"2023-10-14T16:59:25.827227+0000","last_became_peered":"2023-10-14T16:59:25.827227+0000","last_unstale":"2023-10-14T16:59:28.024760+0000","last_undegraded":"2023-10-14T16:59:28.024760+0000","last_fullsized":"2023-10-14T16:59:28.024760+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:59.894118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.16","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027526+0000","last_change":"2023-10-14T16:59:26.827014+0000","last_active":"2023-10-14T16:59:28.027526+0000","last_peered":"2023-10-14T16:59:28.027526+0000","last_clean":"2023-10-14T16:59:28.027526+0000","last_became_active":"2023-10-14T16:59:26.826625+0000","last_became_peered":"2023-10-14T16:59:26.826625+0000","last_unstale":"2023-10-14T16:59:28.027526+0000","last_undegraded":"2023-10-14T16:59:28.027526+0000","last_fullsized":"2023-10-14T16:59:28.027526+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.17","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025372+0000","last_change":"2023-10-14T16:59:25.823768+0000","last_active":"2023-10-14T16:59:28.025372+0000","last_peered":"2023-10-14T16:59:28.025372+0000","last_clean":"2023-10-14T16:59:28.025372+0000","last_became_active":"2023-10-14T16:59:25.823498+0000","last_became_peered":"2023-10-14T16:59:25.823498+0000","last_unstale":"2023-10-14T16:59:28.025372+0000","last_undegraded":"2023-10-14T16:59:28.025372+0000","last_fullsized":"2023-10-14T16:59:28.025372+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:30.659653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.17","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833771+0000","last_change":"2023-10-14T16:59:26.834533+0000","last_active":"2023-10-14T16:59:27.833771+0000","last_peered":"2023-10-14T16:59:27.833771+0000","last_clean":"2023-10-14T16:59:27.833771+0000","last_became_active":"2023-10-14T16:59:26.832120+0000","last_became_peered":"2023-10-14T16:59:26.832120+0000","last_unstale":"2023-10-14T16:59:27.833771+0000","last_undegraded":"2023-10-14T16:59:27.833771+0000","last_fullsized":"2023-10-14T16:59:27.833771+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:56:08.624193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.16","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026531+0000","last_change":"2023-10-14T16:59:25.826357+0000","last_active":"2023-10-14T16:59:28.026531+0000","last_peered":"2023-10-14T16:59:28.026531+0000","last_clean":"2023-10-14T16:59:28.026531+0000","last_became_active":"2023-10-14T16:59:25.824619+0000","last_became_peered":"2023-10-14T16:59:25.824619+0000","last_unstale":"2023-10-14T16:59:28.026531+0000","last_undegraded":"2023-10-14T16:59:28.026531+0000","last_fullsized":"2023-10-14T16:59:28.026531+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:00:38.918023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.14","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835115+0000","last_change":"2023-10-14T16:59:26.838832+0000","last_active":"2023-10-14T16:59:27.835115+0000","last_peered":"2023-10-14T16:59:27.835115+0000","last_clean":"2023-10-14T16:59:27.835115+0000","last_became_active":"2023-10-14T16:59:26.838529+0000","last_became_peered":"2023-10-14T16:59:26.838529+0000","last_unstale":"2023-10-14T16:59:27.835115+0000","last_undegraded":"2023-10-14T16:59:27.835115+0000","last_fullsized":"2023-10-14T16:59:27.835115+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.15","version":"124'3","reported_seq":15,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.029017+0000","last_change":"2023-10-14T16:59:25.832164+0000","last_active":"2023-10-14T16:59:28.029017+0000","last_peered":"2023-10-14T16:59:28.029017+0000","last_clean":"2023-10-14T16:59:28.029017+0000","last_became_active":"2023-10-14T16:59:25.831848+0000","last_became_peered":"2023-10-14T16:59:25.831848+0000","last_unstale":"2023-10-14T16:59:28.029017+0000","last_undegraded":"2023-10-14T16:59:28.029017+0000","last_fullsized":"2023-10-14T16:59:28.029017+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:27.950735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"15.15","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028703+0000","last_change":"2023-10-14T16:59:26.835807+0000","last_active":"2023-10-14T16:59:28.028703+0000","last_peered":"2023-10-14T16:59:28.028703+0000","last_clean":"2023-10-14T16:59:28.028703+0000","last_became_active":"2023-10-14T16:59:26.835355+0000","last_became_peered":"2023-10-14T16:59:26.835355+0000","last_unstale":"2023-10-14T16:59:28.028703+0000","last_undegraded":"2023-10-14T16:59:28.028703+0000","last_fullsized":"2023-10-14T16:59:28.028703+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:02.563762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.14","version":"124'6","reported_seq":18,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833969+0000","last_change":"2023-10-14T16:59:25.825788+0000","last_active":"2023-10-14T16:59:27.833969+0000","last_peered":"2023-10-14T16:59:27.833969+0000","last_clean":"2023-10-14T16:59:27.833969+0000","last_became_active":"2023-10-14T16:59:25.825483+0000","last_became_peered":"2023-10-14T16:59:25.825483+0000","last_unstale":"2023-10-14T16:59:27.833969+0000","last_undegraded":"2023-10-14T16:59:27.833969+0000","last_fullsized":"2023-10-14T16:59:27.833969+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:49.388877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.12","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025978+0000","last_change":"2023-10-14T16:59:26.839521+0000","last_active":"2023-10-14T16:59:28.025978+0000","last_peered":"2023-10-14T16:59:28.025978+0000","last_clean":"2023-10-14T16:59:28.025978+0000","last_became_active":"2023-10-14T16:59:26.837562+0000","last_became_peered":"2023-10-14T16:59:26.837562+0000","last_unstale":"2023-10-14T16:59:28.025978+0000","last_undegraded":"2023-10-14T16:59:28.025978+0000","last_fullsized":"2023-10-14T16:59:28.025978+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:46:15.037570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"14.13","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829131+0000","last_change":"2023-10-14T16:59:25.819010+0000","last_active":"2023-10-14T16:59:27.829131+0000","last_peered":"2023-10-14T16:59:27.829131+0000","last_clean":"2023-10-14T16:59:27.829131+0000","last_became_active":"2023-10-14T16:59:25.818696+0000","last_became_peered":"2023-10-14T16:59:25.818696+0000","last_unstale":"2023-10-14T16:59:27.829131+0000","last_undegraded":"2023-10-14T16:59:27.829131+0000","last_fullsized":"2023-10-14T16:59:27.829131+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:37:20.595779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.13","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025054+0000","last_change":"2023-10-14T16:59:26.835333+0000","last_active":"2023-10-14T16:59:28.025054+0000","last_peered":"2023-10-14T16:59:28.025054+0000","last_clean":"2023-10-14T16:59:28.025054+0000","last_became_active":"2023-10-14T16:59:26.835030+0000","last_became_peered":"2023-10-14T16:59:26.835030+0000","last_unstale":"2023-10-14T16:59:28.025054+0000","last_undegraded":"2023-10-14T16:59:28.025054+0000","last_fullsized":"2023-10-14T16:59:28.025054+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:49:53.037517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.12","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828896+0000","last_change":"2023-10-14T16:59:25.819245+0000","last_active":"2023-10-14T16:59:27.828896+0000","last_peered":"2023-10-14T16:59:27.828896+0000","last_clean":"2023-10-14T16:59:27.828896+0000","last_became_active":"2023-10-14T16:59:25.818991+0000","last_became_peered":"2023-10-14T16:59:25.818991+0000","last_unstale":"2023-10-14T16:59:27.828896+0000","last_undegraded":"2023-10-14T16:59:27.828896+0000","last_fullsized":"2023-10-14T16:59:27.828896+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:13.967958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.10","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.831755+0000","last_change":"2023-10-14T16:59:26.832899+0000","last_active":"2023-10-14T16:59:27.831755+0000","last_peered":"2023-10-14T16:59:27.831755+0000","last_clean":"2023-10-14T16:59:27.831755+0000","last_became_active":"2023-10-14T16:59:26.831690+0000","last_became_peered":"2023-10-14T16:59:26.831690+0000","last_unstale":"2023-10-14T16:59:27.831755+0000","last_undegraded":"2023-10-14T16:59:27.831755+0000","last_fullsized":"2023-10-14T16:59:27.831755+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:28.174078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.11","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833893+0000","last_change":"2023-10-14T16:59:25.819470+0000","last_active":"2023-10-14T16:59:27.833893+0000","last_peered":"2023-10-14T16:59:27.833893+0000","last_clean":"2023-10-14T16:59:27.833893+0000","last_became_active":"2023-10-14T16:59:25.818906+0000","last_became_peered":"2023-10-14T16:59:25.818906+0000","last_unstale":"2023-10-14T16:59:27.833893+0000","last_undegraded":"2023-10-14T16:59:27.833893+0000","last_fullsized":"2023-10-14T16:59:27.833893+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:29:50.193783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.11","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828329+0000","last_change":"2023-10-14T16:59:26.832426+0000","last_active":"2023-10-14T16:59:27.828329+0000","last_peered":"2023-10-14T16:59:27.828329+0000","last_clean":"2023-10-14T16:59:27.828329+0000","last_became_active":"2023-10-14T16:59:26.832039+0000","last_became_peered":"2023-10-14T16:59:26.832039+0000","last_unstale":"2023-10-14T16:59:27.828329+0000","last_undegraded":"2023-10-14T16:59:27.828329+0000","last_fullsized":"2023-10-14T16:59:27.828329+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:09.706336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.10","version":"124'8","reported_seq":20,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.024862+0000","last_change":"2023-10-14T16:59:25.829739+0000","last_active":"2023-10-14T16:59:28.024862+0000","last_peered":"2023-10-14T16:59:28.024862+0000","last_clean":"2023-10-14T16:59:28.024862+0000","last_became_active":"2023-10-14T16:59:25.829471+0000","last_became_peered":"2023-10-14T16:59:25.829471+0000","last_unstale":"2023-10-14T16:59:28.024862+0000","last_undegraded":"2023-10-14T16:59:28.024862+0000","last_fullsized":"2023-10-14T16:59:28.024862+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:05.514211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"15.f","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834597+0000","last_change":"2023-10-14T16:59:26.824269+0000","last_active":"2023-10-14T16:59:27.834597+0000","last_peered":"2023-10-14T16:59:27.834597+0000","last_clean":"2023-10-14T16:59:27.834597+0000","last_became_active":"2023-10-14T16:59:26.823966+0000","last_became_peered":"2023-10-14T16:59:26.823966+0000","last_unstale":"2023-10-14T16:59:27.834597+0000","last_undegraded":"2023-10-14T16:59:27.834597+0000","last_fullsized":"2023-10-14T16:59:27.834597+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:33:50.683607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.e","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834500+0000","last_change":"2023-10-14T16:59:25.821235+0000","last_active":"2023-10-14T16:59:27.834500+0000","last_peered":"2023-10-14T16:59:27.834500+0000","last_clean":"2023-10-14T16:59:27.834500+0000","last_became_active":"2023-10-14T16:59:25.820946+0000","last_became_peered":"2023-10-14T16:59:25.820946+0000","last_unstale":"2023-10-14T16:59:27.834500+0000","last_undegraded":"2023-10-14T16:59:27.834500+0000","last_fullsized":"2023-10-14T16:59:27.834500+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:05.024120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"15.c","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028275+0000","last_change":"2023-10-14T16:59:26.834618+0000","last_active":"2023-10-14T16:59:28.028275+0000","last_peered":"2023-10-14T16:59:28.028275+0000","last_clean":"2023-10-14T16:59:28.028275+0000","last_became_active":"2023-10-14T16:59:26.833681+0000","last_became_peered":"2023-10-14T16:59:26.833681+0000","last_unstale":"2023-10-14T16:59:28.028275+0000","last_undegraded":"2023-10-14T16:59:28.028275+0000","last_fullsized":"2023-10-14T16:59:28.028275+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:32.136376+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"14.d","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833685+0000","last_change":"2023-10-14T16:59:25.827686+0000","last_active":"2023-10-14T16:59:27.833685+0000","last_peered":"2023-10-14T16:59:27.833685+0000","last_clean":"2023-10-14T16:59:27.833685+0000","last_became_active":"2023-10-14T16:59:25.827075+0000","last_became_peered":"2023-10-14T16:59:25.827075+0000","last_unstale":"2023-10-14T16:59:27.833685+0000","last_undegraded":"2023-10-14T16:59:27.833685+0000","last_fullsized":"2023-10-14T16:59:27.833685+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:33.544429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.d","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028892+0000","last_change":"2023-10-14T16:59:26.829685+0000","last_active":"2023-10-14T16:59:28.028892+0000","last_peered":"2023-10-14T16:59:28.028892+0000","last_clean":"2023-10-14T16:59:28.028892+0000","last_became_active":"2023-10-14T16:59:26.829407+0000","last_became_peered":"2023-10-14T16:59:26.829407+0000","last_unstale":"2023-10-14T16:59:28.028892+0000","last_undegraded":"2023-10-14T16:59:28.028892+0000","last_fullsized":"2023-10-14T16:59:28.028892+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:24.636655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"14.c","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027309+0000","last_change":"2023-10-14T16:59:25.827438+0000","last_active":"2023-10-14T16:59:28.027309+0000","last_peered":"2023-10-14T16:59:28.027309+0000","last_clean":"2023-10-14T16:59:28.027309+0000","last_became_active":"2023-10-14T16:59:25.826774+0000","last_became_peered":"2023-10-14T16:59:25.826774+0000","last_unstale":"2023-10-14T16:59:28.027309+0000","last_undegraded":"2023-10-14T16:59:28.027309+0000","last_fullsized":"2023-10-14T16:59:28.027309+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:01.225446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.a","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828570+0000","last_change":"2023-10-14T16:59:26.843752+0000","last_active":"2023-10-14T16:59:27.828570+0000","last_peered":"2023-10-14T16:59:27.828570+0000","last_clean":"2023-10-14T16:59:27.828570+0000","last_became_active":"2023-10-14T16:59:26.843492+0000","last_became_peered":"2023-10-14T16:59:26.843492+0000","last_unstale":"2023-10-14T16:59:27.828570+0000","last_undegraded":"2023-10-14T16:59:27.828570+0000","last_fullsized":"2023-10-14T16:59:27.828570+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:19.093526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.b","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027113+0000","last_change":"2023-10-14T16:59:25.833911+0000","last_active":"2023-10-14T16:59:28.027113+0000","last_peered":"2023-10-14T16:59:28.027113+0000","last_clean":"2023-10-14T16:59:28.027113+0000","last_became_active":"2023-10-14T16:59:25.833619+0000","last_became_peered":"2023-10-14T16:59:25.833619+0000","last_unstale":"2023-10-14T16:59:28.027113+0000","last_undegraded":"2023-10-14T16:59:28.027113+0000","last_fullsized":"2023-10-14T16:59:28.027113+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:33.544429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.b","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828997+0000","last_change":"2023-10-14T16:59:26.834354+0000","last_active":"2023-10-14T16:59:27.828997+0000","last_peered":"2023-10-14T16:59:27.828997+0000","last_clean":"2023-10-14T16:59:27.828997+0000","last_became_active":"2023-10-14T16:59:26.834082+0000","last_became_peered":"2023-10-14T16:59:26.834082+0000","last_unstale":"2023-10-14T16:59:27.828997+0000","last_undegraded":"2023-10-14T16:59:27.828997+0000","last_fullsized":"2023-10-14T16:59:27.828997+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:40:34.558914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.a","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028571+0000","last_change":"2023-10-14T16:59:25.819027+0000","last_active":"2023-10-14T16:59:28.028571+0000","last_peered":"2023-10-14T16:59:28.028571+0000","last_clean":"2023-10-14T16:59:28.028571+0000","last_became_active":"2023-10-14T16:59:25.818445+0000","last_became_peered":"2023-10-14T16:59:25.818445+0000","last_unstale":"2023-10-14T16:59:28.028571+0000","last_undegraded":"2023-10-14T16:59:28.028571+0000","last_fullsized":"2023-10-14T16:59:28.028571+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:58:01.822034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.8","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828326+0000","last_change":"2023-10-14T16:59:26.842649+0000","last_active":"2023-10-14T16:59:27.828326+0000","last_peered":"2023-10-14T16:59:27.828326+0000","last_clean":"2023-10-14T16:59:27.828326+0000","last_became_active":"2023-10-14T16:59:26.842362+0000","last_became_peered":"2023-10-14T16:59:26.842362+0000","last_unstale":"2023-10-14T16:59:27.828326+0000","last_undegraded":"2023-10-14T16:59:27.828326+0000","last_fullsized":"2023-10-14T16:59:27.828326+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:55.593083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.9","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.028436+0000","last_change":"2023-10-14T16:59:25.828774+0000","last_active":"2023-10-14T16:59:28.028436+0000","last_peered":"2023-10-14T16:59:28.028436+0000","last_clean":"2023-10-14T16:59:28.028436+0000","last_became_active":"2023-10-14T16:59:25.827658+0000","last_became_peered":"2023-10-14T16:59:25.827658+0000","last_unstale":"2023-10-14T16:59:28.028436+0000","last_undegraded":"2023-10-14T16:59:28.028436+0000","last_fullsized":"2023-10-14T16:59:28.028436+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:20:49.724370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.9","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.831980+0000","last_change":"2023-10-14T16:59:26.824040+0000","last_active":"2023-10-14T16:59:27.831980+0000","last_peered":"2023-10-14T16:59:27.831980+0000","last_clean":"2023-10-14T16:59:27.831980+0000","last_became_active":"2023-10-14T16:59:26.823748+0000","last_became_peered":"2023-10-14T16:59:26.823748+0000","last_unstale":"2023-10-14T16:59:27.831980+0000","last_undegraded":"2023-10-14T16:59:27.831980+0000","last_fullsized":"2023-10-14T16:59:27.831980+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:13:04.971542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.8","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026525+0000","last_change":"2023-10-14T16:59:25.833466+0000","last_active":"2023-10-14T16:59:28.026525+0000","last_peered":"2023-10-14T16:59:28.026525+0000","last_clean":"2023-10-14T16:59:28.026525+0000","last_became_active":"2023-10-14T16:59:25.833102+0000","last_became_peered":"2023-10-14T16:59:25.833102+0000","last_unstale":"2023-10-14T16:59:28.026525+0000","last_undegraded":"2023-10-14T16:59:28.026525+0000","last_fullsized":"2023-10-14T16:59:28.026525+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:10:59.894118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.6","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027574+0000","last_change":"2023-10-14T16:59:26.830254+0000","last_active":"2023-10-14T16:59:28.027574+0000","last_peered":"2023-10-14T16:59:28.027574+0000","last_clean":"2023-10-14T16:59:28.027574+0000","last_became_active":"2023-10-14T16:59:26.829510+0000","last_became_peered":"2023-10-14T16:59:26.829510+0000","last_unstale":"2023-10-14T16:59:28.027574+0000","last_undegraded":"2023-10-14T16:59:28.027574+0000","last_fullsized":"2023-10-14T16:59:28.027574+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:23:18.956709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"14.7","version":"124'2","reported_seq":14,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.029493+0000","last_change":"2023-10-14T16:59:25.828330+0000","last_active":"2023-10-14T16:59:28.029493+0000","last_peered":"2023-10-14T16:59:28.029493+0000","last_clean":"2023-10-14T16:59:28.029493+0000","last_became_active":"2023-10-14T16:59:25.826649+0000","last_became_peered":"2023-10-14T16:59:25.826649+0000","last_unstale":"2023-10-14T16:59:28.029493+0000","last_undegraded":"2023-10-14T16:59:28.029493+0000","last_fullsized":"2023-10-14T16:59:28.029493+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:42:54.410805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"15.7","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025273+0000","last_change":"2023-10-14T16:59:26.838082+0000","last_active":"2023-10-14T16:59:28.025273+0000","last_peered":"2023-10-14T16:59:28.025273+0000","last_clean":"2023-10-14T16:59:28.025273+0000","last_became_active":"2023-10-14T16:59:26.837421+0000","last_became_peered":"2023-10-14T16:59:26.837421+0000","last_unstale":"2023-10-14T16:59:28.025273+0000","last_undegraded":"2023-10-14T16:59:28.025273+0000","last_fullsized":"2023-10-14T16:59:28.025273+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:54.179919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.6","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027637+0000","last_change":"2023-10-14T16:59:25.833012+0000","last_active":"2023-10-14T16:59:28.027637+0000","last_peered":"2023-10-14T16:59:28.027637+0000","last_clean":"2023-10-14T16:59:28.027637+0000","last_became_active":"2023-10-14T16:59:25.832587+0000","last_became_peered":"2023-10-14T16:59:25.832587+0000","last_unstale":"2023-10-14T16:59:28.027637+0000","last_undegraded":"2023-10-14T16:59:28.027637+0000","last_fullsized":"2023-10-14T16:59:28.027637+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:09:46.907273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.4","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.025042+0000","last_change":"2023-10-14T16:59:26.831084+0000","last_active":"2023-10-14T16:59:28.025042+0000","last_peered":"2023-10-14T16:59:28.025042+0000","last_clean":"2023-10-14T16:59:28.025042+0000","last_became_active":"2023-10-14T16:59:26.830747+0000","last_became_peered":"2023-10-14T16:59:26.830747+0000","last_unstale":"2023-10-14T16:59:28.025042+0000","last_undegraded":"2023-10-14T16:59:28.025042+0000","last_fullsized":"2023-10-14T16:59:28.025042+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"14.5","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027345+0000","last_change":"2023-10-14T16:59:25.829782+0000","last_active":"2023-10-14T16:59:28.027345+0000","last_peered":"2023-10-14T16:59:28.027345+0000","last_clean":"2023-10-14T16:59:28.027345+0000","last_became_active":"2023-10-14T16:59:25.828845+0000","last_became_peered":"2023-10-14T16:59:25.828845+0000","last_unstale":"2023-10-14T16:59:28.027345+0000","last_undegraded":"2023-10-14T16:59:28.027345+0000","last_fullsized":"2023-10-14T16:59:28.027345+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:54.578598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.5","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.833158+0000","last_change":"2023-10-14T16:59:26.842333+0000","last_active":"2023-10-14T16:59:27.833158+0000","last_peered":"2023-10-14T16:59:27.833158+0000","last_clean":"2023-10-14T16:59:27.833158+0000","last_became_active":"2023-10-14T16:59:26.841940+0000","last_became_peered":"2023-10-14T16:59:26.841940+0000","last_unstale":"2023-10-14T16:59:27.833158+0000","last_undegraded":"2023-10-14T16:59:27.833158+0000","last_fullsized":"2023-10-14T16:59:27.833158+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:55.086162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.4","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.835334+0000","last_change":"2023-10-14T16:59:25.830715+0000","last_active":"2023-10-14T16:59:27.835334+0000","last_peered":"2023-10-14T16:59:27.835334+0000","last_clean":"2023-10-14T16:59:27.835334+0000","last_became_active":"2023-10-14T16:59:25.830416+0000","last_became_peered":"2023-10-14T16:59:25.830416+0000","last_unstale":"2023-10-14T16:59:27.835334+0000","last_undegraded":"2023-10-14T16:59:27.835334+0000","last_fullsized":"2023-10-14T16:59:27.835334+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:01.225446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.2","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828979+0000","last_change":"2023-10-14T16:59:26.828796+0000","last_active":"2023-10-14T16:59:27.828979+0000","last_peered":"2023-10-14T16:59:27.828979+0000","last_clean":"2023-10-14T16:59:27.828979+0000","last_became_active":"2023-10-14T16:59:26.827950+0000","last_became_peered":"2023-10-14T16:59:26.827950+0000","last_unstale":"2023-10-14T16:59:27.828979+0000","last_undegraded":"2023-10-14T16:59:27.828979+0000","last_fullsized":"2023-10-14T16:59:27.828979+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:02.239931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.3","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027209+0000","last_change":"2023-10-14T16:59:25.830261+0000","last_active":"2023-10-14T16:59:28.027209+0000","last_peered":"2023-10-14T16:59:28.027209+0000","last_clean":"2023-10-14T16:59:28.027209+0000","last_became_active":"2023-10-14T16:59:25.829995+0000","last_became_peered":"2023-10-14T16:59:25.829995+0000","last_unstale":"2023-10-14T16:59:28.027209+0000","last_undegraded":"2023-10-14T16:59:28.027209+0000","last_fullsized":"2023-10-14T16:59:28.027209+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:18.079041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"15.3","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.834564+0000","last_change":"2023-10-14T16:59:26.832998+0000","last_active":"2023-10-14T16:59:27.834564+0000","last_peered":"2023-10-14T16:59:27.834564+0000","last_clean":"2023-10-14T16:59:27.834564+0000","last_became_active":"2023-10-14T16:59:26.832647+0000","last_became_peered":"2023-10-14T16:59:26.832647+0000","last_unstale":"2023-10-14T16:59:27.834564+0000","last_undegraded":"2023-10-14T16:59:27.834564+0000","last_fullsized":"2023-10-14T16:59:27.834564+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:55.390471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"14.2","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832593+0000","last_change":"2023-10-14T16:59:25.823773+0000","last_active":"2023-10-14T16:59:27.832593+0000","last_peered":"2023-10-14T16:59:27.832593+0000","last_clean":"2023-10-14T16:59:27.832593+0000","last_became_active":"2023-10-14T16:59:25.823133+0000","last_became_peered":"2023-10-14T16:59:25.823133+0000","last_unstale":"2023-10-14T16:59:27.832593+0000","last_undegraded":"2023-10-14T16:59:27.832593+0000","last_fullsized":"2023-10-14T16:59:27.832593+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:05.514211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"15.0","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828325+0000","last_change":"2023-10-14T16:59:26.831575+0000","last_active":"2023-10-14T16:59:27.828325+0000","last_peered":"2023-10-14T16:59:27.828325+0000","last_clean":"2023-10-14T16:59:27.828325+0000","last_became_active":"2023-10-14T16:59:26.831242+0000","last_became_peered":"2023-10-14T16:59:26.831242+0000","last_unstale":"2023-10-14T16:59:27.828325+0000","last_undegraded":"2023-10-14T16:59:27.828325+0000","last_fullsized":"2023-10-14T16:59:27.828325+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:11:00.908603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.1","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.026101+0000","last_change":"2023-10-14T16:59:25.822607+0000","last_active":"2023-10-14T16:59:28.026101+0000","last_peered":"2023-10-14T16:59:28.026101+0000","last_clean":"2023-10-14T16:59:28.026101+0000","last_became_active":"2023-10-14T16:59:25.822333+0000","last_became_peered":"2023-10-14T16:59:25.822333+0000","last_unstale":"2023-10-14T16:59:28.026101+0000","last_undegraded":"2023-10-14T16:59:28.026101+0000","last_fullsized":"2023-10-14T16:59:28.026101+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:50:04.964615+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"15.1","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.832528+0000","last_change":"2023-10-14T16:59:26.837575+0000","last_active":"2023-10-14T16:59:27.832528+0000","last_peered":"2023-10-14T16:59:27.832528+0000","last_clean":"2023-10-14T16:59:27.832528+0000","last_became_active":"2023-10-14T16:59:26.837279+0000","last_became_peered":"2023-10-14T16:59:26.837279+0000","last_unstale":"2023-10-14T16:59:27.832528+0000","last_undegraded":"2023-10-14T16:59:27.832528+0000","last_fullsized":"2023-10-14T16:59:27.832528+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"14.0","version":"0'0","reported_seq":12,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.828462+0000","last_change":"2023-10-14T16:59:25.829157+0000","last_active":"2023-10-14T16:59:27.828462+0000","last_peered":"2023-10-14T16:59:27.828462+0000","last_clean":"2023-10-14T16:59:27.828462+0000","last_became_active":"2023-10-14T16:59:25.828035+0000","last_became_peered":"2023-10-14T16:59:25.828035+0000","last_unstale":"2023-10-14T16:59:27.828462+0000","last_undegraded":"2023-10-14T16:59:27.828462+0000","last_fullsized":"2023-10-14T16:59:27.828462+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:30.973459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"15.e","version":"0'0","reported_seq":11,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:27.829558+0000","last_change":"2023-10-14T16:59:26.840023+0000","last_active":"2023-10-14T16:59:27.829558+0000","last_peered":"2023-10-14T16:59:27.829558+0000","last_clean":"2023-10-14T16:59:27.829558+0000","last_became_active":"2023-10-14T16:59:26.838025+0000","last_became_peered":"2023-10-14T16:59:26.838025+0000","last_unstale":"2023-10-14T16:59:27.829558+0000","last_undegraded":"2023-10-14T16:59:27.829558+0000","last_fullsized":"2023-10-14T16:59:27.829558+0000","mapping_epoch":123,"log_start":"0'0","ondisk_log_start":"0'0","created":123,"last_epoch_clean":124,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:25.786285+0000","last_clean_scrub_stamp":"2023-10-14T16:59:25.786285+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:43:26.121797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"14.f","version":"124'1","reported_seq":13,"reported_epoch":125,"state":"active+clean","last_fresh":"2023-10-14T16:59:28.027984+0000","last_change":"2023-10-14T16:59:25.833874+0000","last_active":"2023-10-14T16:59:28.027984+0000","last_peered":"2023-10-14T16:59:28.027984+0000","last_clean":"2023-10-14T16:59:28.027984+0000","last_became_active":"2023-10-14T16:59:25.833552+0000","last_became_peered":"2023-10-14T16:59:25.833552+0000","last_unstale":"2023-10-14T16:59:28.027984+0000","last_undegraded":"2023-10-14T16:59:28.027984+0000","last_fullsized":"2023-10-14T16:59:28.027984+0000","mapping_epoch":122,"log_start":"0'0","ondisk_log_start":"0'0","created":122,"last_epoch_clean":123,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:59:24.771800+0000","last_clean_scrub_stamp":"2023-10-14T16:59:24.771800+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:27:27.739184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"113'205","reported_seq":574,"reported_epoch":124,"state":"active+clean","last_fresh":"2023-10-14T16:59:26.806960+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T16:59:26.806960+0000","last_peered":"2023-10-14T16:59:26.806960+0000","last_clean":"2023-10-14T16:59:26.806960+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T16:59:26.806960+0000","last_undegraded":"2023-10-14T16:59:26.806960+0000","last_fullsized":"2023-10-14T16:59:26.806960+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":15,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":14,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":884,"num_read_kb":1748,"num_write":319,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":205,"ondisk_log_size":205,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575077,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37440,"kb_used_data":2096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334400,"statfs":{"total":96636764160,"available":96598425600,"internally_reserved":0,"allocated":2146304,"data_stored":1336104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.1639999999999999,"5min":1.1699999999999999,"15min":1.1830000000000001},"min":{"1min":0.77900000000000003,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.607,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.163},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.115,"15min":1.198},"min":{"1min":0.876,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":1.7370000000000001,"5min":1.871,"15min":2.117},"last":1.427}]},{"osd":1,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.177,"15min":1.2270000000000001},"min":{"1min":0.87,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8520000000000001,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":1.379},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.98199999999999998,"5min":0.57799999999999996,"15min":0.57799999999999996},"max":{"1min":1.7170000000000001,"5min":2.0259999999999998,"15min":2.0259999999999998},"last":1.3009999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2010000000000001,"5min":1.1739999999999999,"15min":1.222},"min":{"1min":0.89000000000000001,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.647,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.25},{"interface":"front","average":{"1min":1.333,"5min":1.276,"15min":1.3049999999999999},"min":{"1min":1.012,"5min":0.72599999999999998,"15min":0.72599999999999998},"max":{"1min":1.575,"5min":1.893,"15min":2.1589999999999998},"last":1.4610000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.3169999999999999,"15min":1.367},"min":{"1min":1.0640000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.988,"5min":2.056,"15min":2.4159999999999999},"last":1.339},{"interface":"front","average":{"1min":1.444,"5min":1.4179999999999999,"15min":1.45},"min":{"1min":1.0249999999999999,"5min":0.72199999999999998,"15min":0.72199999999999998},"max":{"1min":1.841,"5min":2.379,"15min":2.379},"last":1.6000000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.2869999999999999,"5min":1.2989999999999999,"15min":1.2789999999999999},"min":{"1min":0.85699999999999998,"5min":0.51800000000000002,"15min":0.51800000000000002},"max":{"1min":1.7370000000000001,"5min":2.105,"15min":2.1819999999999999},"last":0.89400000000000002},{"interface":"front","average":{"1min":1.3819999999999999,"5min":1.347,"15min":1.3340000000000001},"min":{"1min":0.93400000000000005,"5min":0.59799999999999998,"15min":0.59799999999999998},"max":{"1min":2.2410000000000001,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":0.93799999999999994}]},{"osd":5,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.367,"5min":1.3819999999999999,"15min":1.3859999999999999},"min":{"1min":0.84999999999999998,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.9299999999999999,"5min":2.278,"15min":2.278},"last":1.544},{"interface":"front","average":{"1min":1.4550000000000001,"5min":1.444,"15min":1.462},"min":{"1min":0.98199999999999998,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":1.7909999999999999,"5min":2.0600000000000001,"15min":2.1299999999999999},"last":1.5109999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:02 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.5189999999999999,"15min":1.5429999999999999},"min":{"1min":1.1240000000000001,"5min":0.621,"15min":0.621},"max":{"1min":1.917,"5min":2.492,"15min":2.492},"last":1.5669999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.508,"15min":1.4870000000000001},"min":{"1min":1.014,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":2.3540000000000001,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.1140000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575076,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36856,"kb_used_data":1516,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334984,"statfs":{"total":96636764160,"available":96599023616,"internally_reserved":0,"allocated":1552384,"data_stored":746276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.139,"15min":1.159},"min":{"1min":0.78400000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.7330000000000001,"5min":2.137,"15min":2.137},"last":1.4510000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.1890000000000001,"15min":1.2110000000000001},"min":{"1min":0.85699999999999998,"5min":0.68500000000000005,"15min":0.68500000000000005},"max":{"1min":1.806,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":1.1519999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.169,"15min":1.177},"min":{"1min":0.90900000000000003,"5min":0.73499999999999999,"15min":0.41299999999999998},"max":{"1min":1.359,"5min":2.802,"15min":2.802},"last":1.099},{"interface":"front","average":{"1min":1.286,"5min":1.258,"15min":1.2749999999999999},"min":{"1min":0.86399999999999999,"5min":0.67300000000000004,"15min":0.438},"max":{"1min":1.9810000000000001,"5min":2.512,"15min":2.512},"last":1.5009999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.339,"5min":1.361,"15min":1.3500000000000001},"min":{"1min":0.91700000000000004,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.7669999999999999,"5min":3.3290000000000002,"15min":3.3290000000000002},"last":0.90200000000000002},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.3460000000000001,"15min":1.383},"min":{"1min":1.125,"5min":0.81299999999999994,"15min":0.46000000000000002},"max":{"1min":1.786,"5min":3.286,"15min":3.286},"last":0.96899999999999997}]},{"osd":3,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.345,"15min":1.3660000000000001},"min":{"1min":0.90500000000000003,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.853,"5min":2.823,"15min":2.823},"last":1.3779999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.399,"15min":1.407},"min":{"1min":1.071,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.7350000000000001,"5min":4.2610000000000001,"15min":4.2610000000000001},"last":1.27}]},{"osd":4,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.391,"15min":1.3400000000000001},"min":{"1min":0.92600000000000005,"5min":0.497,"15min":0.497},"max":{"1min":1.9299999999999999,"5min":2.9460000000000002,"15min":2.9460000000000002},"last":1.1850000000000001},{"interface":"front","average":{"1min":1.454,"5min":1.47,"15min":1.4119999999999999},"min":{"1min":1.1040000000000001,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.8660000000000001,"5min":2.5750000000000002,"15min":2.5750000000000002},"last":1.4159999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.3939999999999999,"15min":1.3799999999999999},"min":{"1min":0.85599999999999998,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.891,"5min":3.4060000000000001,"15min":3.4060000000000001},"last":1.2390000000000001},{"interface":"front","average":{"1min":1.5600000000000001,"5min":1.5760000000000001,"15min":1.5509999999999999},"min":{"1min":1.0980000000000001,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":2.028,"5min":3.0489999999999999,"15min":3.0489999999999999},"last":1.585}]},{"osd":7,"last update":"Sat Oct 14 16:58:55 2023","interfaces":[{"interface":"back","average":{"1min":1.591,"5min":1.603,"15min":1.5800000000000001},"min":{"1min":1.2070000000000001,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.9419999999999999,"5min":2.8570000000000002,"15min":2.8570000000000002},"last":1.538},{"interface":"front","average":{"1min":1.583,"5min":1.5289999999999999,"15min":1.5329999999999999},"min":{"1min":1.153,"5min":0.623,"15min":0.623},"max":{"1min":1.9790000000000001,"5min":2.7000000000000002,"15min":2.7000000000000002},"last":1.3500000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575077,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32680,"kb_used_data":1500,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94339160,"statfs":{"total":96636764160,"available":96603299840,"internally_reserved":0,"allocated":1536000,"data_stored":739350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.119,"5min":1.079,"15min":1.1080000000000001},"min":{"1min":0.71199999999999997,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.486,"5min":1.861,"15min":1.986},"last":1.2210000000000001},{"interface":"front","average":{"1min":1.212,"5min":1.107,"15min":1.133},"min":{"1min":0.88400000000000001,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.7729999999999999,"5min":1.7729999999999999,"15min":2.0430000000000001},"last":0.99399999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2470000000000001,"5min":1.1850000000000001,"15min":1.1759999999999999},"min":{"1min":0.84499999999999997,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.744,"5min":1.744,"15min":2.1200000000000001},"last":0.94899999999999995},{"interface":"front","average":{"1min":1.25,"5min":1.1739999999999999,"15min":1.206},"min":{"1min":0.746,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.7629999999999999,"5min":1.7629999999999999,"15min":1.905},"last":1.1419999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.113,"15min":1.1819999999999999},"min":{"1min":0.65900000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.5529999999999999,"5min":1.7150000000000001,"15min":2.375},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.323,"5min":1.3140000000000001,"15min":1.3180000000000001},"min":{"1min":0.81399999999999995,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6319999999999999,"5min":2.0059999999999998,"15min":2.3100000000000001},"last":1.3939999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.431,"5min":1.347,"15min":1.3839999999999999},"min":{"1min":0.83699999999999997,"5min":0.77200000000000002,"15min":0.745},"max":{"1min":1.8600000000000001,"5min":1.9279999999999999,"15min":2.4180000000000001},"last":1.492},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.3049999999999999,"15min":1.3899999999999999},"min":{"1min":0.86899999999999999,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.8149999999999999,"5min":1.8999999999999999,"15min":2.1429999999999998},"last":1.6579999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3089999999999999,"15min":1.2889999999999999},"min":{"1min":0.63600000000000001,"5min":0.434,"15min":0.434},"max":{"1min":1.744,"5min":2.194,"15min":2.4889999999999999},"last":1.575},{"interface":"front","average":{"1min":1.4019999999999999,"5min":1.3540000000000001,"15min":1.3400000000000001},"min":{"1min":0.90100000000000002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.748,"5min":1.927,"15min":2.5169999999999999},"last":1.333}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2130000000000001,"5min":1.2809999999999999,"15min":1.3540000000000001},"min":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.6399999999999999,"5min":1.9650000000000001,"15min":2.302},"last":1.4530000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.5109999999999999,"15min":1.53},"min":{"1min":0.85299999999999998,"5min":0.63,"15min":0.63},"max":{"1min":1.8220000000000001,"5min":2.1059999999999999,"15min":2.613},"last":1.512}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.4730000000000001,"15min":1.506},"min":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.956,"5min":2.032,"15min":2.605},"last":1.625},{"interface":"front","average":{"1min":1.4359999999999999,"5min":1.452,"15min":1.4970000000000001},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.8129999999999999,"5min":1.8979999999999999,"15min":2.3980000000000001},"last":1.548}]}]},{"osd":4,"up_from":13,"seq":55834575077,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59076,"kb_used_data":1528,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312764,"statfs":{"total":96636764160,"available":96576270336,"internally_reserved":0,"allocated":1564672,"data_stored":747360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.141,"5min":1.133,"15min":1.169},"min":{"1min":0.94599999999999995,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.4950000000000001,"5min":1.9319999999999999,"15min":1.982},"last":1.7270000000000001},{"interface":"front","average":{"1min":1.302,"5min":1.2669999999999999,"15min":1.238},"min":{"1min":0.83999999999999997,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":2.097,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":1.0669999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3,"5min":1.3160000000000001,"15min":1.29},"min":{"1min":0.74099999999999999,"5min":0.48199999999999998,"15min":0.48199999999999998},"max":{"1min":1.7030000000000001,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.194},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.242,"15min":1.254},"min":{"1min":0.69599999999999995,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.49,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.329}]},{"osd":2,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.274,"5min":1.2829999999999999,"15min":1.252},"min":{"1min":0.93899999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.6339999999999999,"5min":5.633,"15min":5.633},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.4319999999999999,"5min":1.3799999999999999,"15min":1.377},"min":{"1min":1.05,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":2.242,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.282}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.304,"15min":1.3200000000000001},"min":{"1min":0.997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.6910000000000001,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.401},{"interface":"front","average":{"1min":1.504,"5min":1.3879999999999999,"15min":1.4119999999999999},"min":{"1min":1.1419999999999999,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.8560000000000001,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.6200000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.48,"5min":1.3959999999999999,"15min":1.3600000000000001},"min":{"1min":1.0589999999999999,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.3849999999999998,"5min":2.931,"15min":2.931},"last":1.4239999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.339,"15min":1.3500000000000001},"min":{"1min":0.93200000000000005,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.177,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.1399999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5109999999999999,"5min":1.4299999999999999,"15min":1.3979999999999999},"min":{"1min":0.97399999999999998,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.9159999999999999,"5min":2.323,"15min":2.323},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.651,"5min":1.542,"15min":1.532},"min":{"1min":0.94999999999999996,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":2.4620000000000002,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.359}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.609,"5min":1.5209999999999999,"15min":1.496},"min":{"1min":0.88400000000000001,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":2.3039999999999998,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.222},{"interface":"front","average":{"1min":1.6339999999999999,"5min":1.556,"15min":1.552},"min":{"1min":1.054,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.004,"5min":2.3130000000000002,"15min":2.3130000000000002},"last":1.4450000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575077,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41028,"kb_used_data":1524,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330812,"statfs":{"total":96636764160,"available":96594751488,"internally_reserved":0,"allocated":1560576,"data_stored":747258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.86799999999999999,"15min":0.92600000000000005},"min":{"1min":0.72099999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.393,"5min":1.621,"15min":1.621},"last":0.74399999999999999},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.0529999999999999,"15min":1.131},"min":{"1min":0.67900000000000005,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.4419999999999999,"5min":1.593,"15min":1.8799999999999999},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":0.96199999999999997,"5min":0.86099999999999999,"15min":0.91400000000000003},"min":{"1min":0.53400000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7609999999999999,"5min":2.1720000000000002,"15min":2.1720000000000002},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.165,"5min":1.056,"15min":1.1140000000000001},"min":{"1min":0.61299999999999999,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.7470000000000001,"5min":1.9159999999999999,"15min":1.9350000000000001},"last":1.1359999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3320000000000001,"5min":1.236,"15min":1.357},"min":{"1min":0.64600000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.7569999999999999,"5min":2.024,"15min":2.0609999999999999},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.3080000000000001,"5min":1.218,"15min":1.284},"min":{"1min":0.73499999999999999,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.8720000000000001,"5min":1.8720000000000001,"15min":2.0960000000000001},"last":1.883}]},{"osd":4,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.3720000000000001,"5min":1.343,"15min":1.365},"min":{"1min":0.98999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8700000000000001,"5min":3.4350000000000001,"15min":3.4350000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.3660000000000001,"15min":1.417},"min":{"1min":0.90100000000000002,"5min":0.745,"15min":0.745},"max":{"1min":1.843,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.4530000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.4390000000000001,"5min":1.3520000000000001,"15min":1.3939999999999999},"min":{"1min":1.036,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.867,"5min":2.516,"15min":3.0190000000000001},"last":1.508},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4059999999999999,"15min":1.4430000000000001},"min":{"1min":1.0469999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"max":{"1min":1.889,"5min":3.7410000000000001,"15min":3.7410000000000001},"last":1.571}]},{"osd":6,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.454,"15min":1.478},"min":{"1min":1.107,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.125,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.6559999999999999,"5min":1.534,"15min":1.5509999999999999},"min":{"1min":0.96799999999999997,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.1909999999999998,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.804}]},{"osd":7,"last update":"Sat Oct 14 16:58:43 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.46,"15min":1.512},"min":{"1min":0.94499999999999995,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":1.9670000000000001,"5min":2.871,"15min":2.875},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.601,"5min":1.5329999999999999,"15min":1.5580000000000001},"min":{"1min":1.2370000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"max":{"1min":1.9790000000000001,"5min":2.907,"15min":2.907},"last":1.756}]}]},{"osd":2,"up_from":13,"seq":55834575077,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45176,"kb_used_data":1512,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326664,"statfs":{"total":96636764160,"available":96590503936,"internally_reserved":0,"allocated":1548288,"data_stored":745658,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":44693958},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.85599999999999998,"15min":0.90200000000000002},"min":{"1min":0.67600000000000005,"5min":0.316,"15min":0.316},"max":{"1min":1.6699999999999999,"5min":1.706,"15min":1.7649999999999999},"last":0.92300000000000004},{"interface":"front","average":{"1min":1.048,"5min":0.97599999999999998,"15min":1.0409999999999999},"min":{"1min":0.76800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.7170000000000001,"5min":1.8080000000000001,"15min":1.8080000000000001},"last":0.83999999999999997}]},{"osd":1,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94799999999999995,"5min":0.90600000000000003,"15min":0.97699999999999998},"min":{"1min":0.73399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.544,"5min":1.544,"15min":2.0419999999999998},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.254,"5min":1.115,"15min":1.173},"min":{"1min":0.90200000000000002,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.5569999999999999,"5min":2.0390000000000001,"15min":2.069},"last":1.4870000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1100000000000001,"15min":1.2190000000000001},"min":{"1min":0.72499999999999998,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.696,"5min":1.9430000000000001,"15min":2.1349999999999998},"last":0.67200000000000004},{"interface":"front","average":{"1min":1.3200000000000001,"5min":1.1870000000000001,"15min":1.3},"min":{"1min":1.0720000000000001,"5min":0.46899999999999997,"15min":0.46899999999999997},"max":{"1min":1.657,"5min":1.7729999999999999,"15min":2.0459999999999998},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.3200000000000001,"15min":1.413},"min":{"1min":1.1870000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.784,"5min":1.913,"15min":2.3159999999999998},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.387,"5min":1.2689999999999999,"15min":1.3580000000000001},"min":{"1min":0.90700000000000003,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.8400000000000001,"5min":2.0129999999999999,"15min":2.0750000000000002},"last":0.96299999999999997}]},{"osd":5,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.355,"5min":1.3129999999999999,"15min":1.389},"min":{"1min":1.0600000000000001,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.698,"5min":2.173,"15min":2.173},"last":1.069},{"interface":"front","average":{"1min":1.55,"5min":1.4239999999999999,"15min":1.5209999999999999},"min":{"1min":1.1719999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.889,"5min":1.889,"15min":2.4689999999999999},"last":1.3029999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4259999999999999,"5min":1.345,"15min":1.4350000000000001},"min":{"1min":1.018,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":1.9590000000000001,"5min":1.9590000000000001,"15min":2.0289999999999999},"last":1.026},{"interface":"front","average":{"1min":1.516,"5min":1.444,"15min":1.5149999999999999},"min":{"1min":1.2729999999999999,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.802,"5min":2.0720000000000001,"15min":2.105},"last":1.224}]},{"osd":7,"last update":"Sat Oct 14 16:58:54 2023","interfaces":[{"interface":"back","average":{"1min":1.548,"5min":1.417,"15min":1.5189999999999999},"min":{"1min":1.034,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.9099999999999999,"5min":2.0129999999999999,"15min":2.109},"last":1.2689999999999999},{"interface":"front","average":{"1min":1.6020000000000001,"5min":1.502,"15min":1.5780000000000001},"min":{"1min":1.1990000000000001,"5min":0.74099999999999999,"15min":0.74099999999999999},"max":{"1min":1.8819999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":1.359}]}]},{"osd":1,"up_from":13,"seq":55834575077,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59056,"kb_used_data":1508,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312784,"statfs":{"total":96636764160,"available":96576290816,"internally_reserved":0,"allocated":1544192,"data_stored":745624,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.85399999999999998,"15min":0.90400000000000003},"min":{"1min":0.501,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.97099999999999997,"5min":2.113,"15min":2.113},"last":1.2250000000000001},{"interface":"front","average":{"1min":1.05,"5min":0.99099999999999999,"15min":1.0669999999999999},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.419,"5min":1.728,"15min":2.0819999999999999},"last":0.77500000000000002}]},{"osd":2,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.92900000000000005,"15min":0.95799999999999996},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.2929999999999999,"5min":1.7509999999999999,"15min":1.758},"last":1.048},{"interface":"front","average":{"1min":1.226,"5min":1.1180000000000001,"15min":1.1719999999999999},"min":{"1min":0.64000000000000001,"5min":0.56499999999999995,"15min":0.51300000000000001},"max":{"1min":1.6339999999999999,"5min":2.008,"15min":2.0150000000000001},"last":1.4059999999999999}]},{"osd":3,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.216,"15min":1.296},"min":{"1min":0.77100000000000002,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.7949999999999999,"5min":1.984,"15min":2.198},"last":1.5189999999999999},{"interface":"front","average":{"1min":1.3680000000000001,"5min":1.127,"15min":1.1919999999999999},"min":{"1min":0.73599999999999999,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.974,"5min":1.974,"15min":2.004},"last":1.2889999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.472,"5min":1.365,"15min":1.4039999999999999},"min":{"1min":0.871,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.8799999999999999,"5min":2.052,"15min":2.3050000000000002},"last":1.1559999999999999},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.3080000000000001,"15min":1.323},"min":{"1min":1.008,"5min":0.66900000000000004,"15min":0.53900000000000003},"max":{"1min":1.9119999999999999,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":1.339}]},{"osd":5,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.3240000000000001,"15min":1.373},"min":{"1min":0.84499999999999997,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9399999999999999,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":1.194},{"interface":"front","average":{"1min":1.452,"5min":1.389,"15min":1.458},"min":{"1min":1.155,"5min":0.83999999999999997,"15min":0.76200000000000001},"max":{"1min":1.911,"5min":2.1480000000000001,"15min":2.302},"last":1.746}]},{"osd":6,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.617,"5min":1.4399999999999999,"15min":1.4950000000000001},"min":{"1min":1.246,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.9650000000000001,"5min":1.9650000000000001,"15min":2.4169999999999998},"last":1.3720000000000001},{"interface":"front","average":{"1min":1.4950000000000001,"5min":1.4370000000000001,"15min":1.4770000000000001},"min":{"1min":0.92500000000000004,"5min":0.84199999999999997,"15min":0.84199999999999997},"max":{"1min":1.833,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":1.4590000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:09 2023","interfaces":[{"interface":"back","average":{"1min":1.5700000000000001,"5min":1.492,"15min":1.5629999999999999},"min":{"1min":1.2889999999999999,"5min":0.81399999999999995,"15min":0.81399999999999995},"max":{"1min":1.901,"5min":2.2410000000000001,"15min":2.411},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.653,"5min":1.498,"15min":1.5760000000000001},"min":{"1min":0.98999999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":2.0910000000000002,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":1.4910000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575076,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37432,"kb_used_data":2092,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334408,"statfs":{"total":96636764160,"available":96598433792,"internally_reserved":0,"allocated":2142208,"data_stored":1336574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.91500000000000004,"15min":0.89800000000000002},"min":{"1min":0.56200000000000006,"5min":0.309,"15min":0.309},"max":{"1min":1.403,"5min":3.355,"15min":3.355},"last":0.64600000000000002},{"interface":"front","average":{"1min":1.0980000000000001,"5min":0.99099999999999999,"15min":1.0840000000000001},"min":{"1min":0.78600000000000003,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.7270000000000001,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.89100000000000001}]},{"osd":2,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.0149999999999999,"5min":1.0629999999999999,"15min":1.087},"min":{"1min":0.62,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.379,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":1.0820000000000001,"5min":1.107,"15min":1.196},"min":{"1min":0.71299999999999997,"5min":0.432,"15min":0.432},"max":{"1min":1.6950000000000001,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.84999999999999998}]},{"osd":3,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.238,"15min":1.2789999999999999},"min":{"1min":0.77000000000000002,"5min":0.502,"15min":0.502},"max":{"1min":1.7549999999999999,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.252,"15min":1.3480000000000001},"min":{"1min":0.76600000000000001,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.9219999999999999,"5min":2.593,"15min":2.593},"last":1.095}]},{"osd":4,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.276,"15min":1.2849999999999999},"min":{"1min":0.81000000000000005,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.6579999999999999,"5min":5.335,"15min":5.335},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3959999999999999,"15min":1.427},"min":{"1min":0.83999999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7290000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.0269999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2669999999999999,"15min":1.3160000000000001},"min":{"1min":0.74099999999999999,"5min":0.67600000000000005,"15min":0.67600000000000005},"max":{"1min":1.7949999999999999,"5min":2.931,"15min":2.931},"last":0.98499999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.4279999999999999,"15min":1.5069999999999999},"min":{"1min":1.0960000000000001,"5min":0.72399999999999998,"15min":0.72399999999999998},"max":{"1min":2.254,"5min":3.306,"15min":3.306},"last":1.1279999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.5049999999999999,"15min":1.5840000000000001},"min":{"1min":1.054,"5min":0.92500000000000004,"15min":0.88900000000000001},"max":{"1min":2.4889999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.3580000000000001,"15min":1.399},"min":{"1min":0.83299999999999996,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.732,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.4390000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:58:53 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.518,"15min":1.552},"min":{"1min":1.04,"5min":0.747,"15min":0.747},"max":{"1min":1.8819999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.4690000000000001,"5min":1.482,"15min":1.52},"min":{"1min":0.78400000000000003,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.875,"5min":3.698,"15min":3.698},"last":1.1419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":14,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":15,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T16:59:31.907 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T16:59:31.908 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T16:59:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.189+0000 7fa0b5f89640 1 Processor -- start 2023-10-14T16:59:32.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.189+0000 7fa0b5f89640 1 -- start start 2023-10-14T16:59:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.190+0000 7fa0b5f89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.190+0000 7fa0b5f89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 0x7fa0b006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.190+0000 7fa0b5f89640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0b00cc560 con 0x7fa0b006fa90 2023-10-14T16:59:32.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.190+0000 7fa0b5f89640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa0b006fee0 con 0x7fa0b00c9520 2023-10-14T16:59:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.190+0000 7fa0b5f89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b008adf0 con 0x7fa0b008aa10 2023-10-14T16:59:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.191+0000 7fa0af7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.191+0000 7fa0aeffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 0x7fa0b006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.191+0000 7fa0af7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:43604/0 (socket says 172.21.15.29:43604) 2023-10-14T16:59:32.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.191+0000 7fa0af7fe640 1 -- 172.21.15.29:0/2375539473 learned_addr learned my addr 172.21.15.29:0/2375539473 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.192+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3017715905 0 0) 0x7fa0b00cc560 con 0x7fa0b006fa90 2023-10-14T16:59:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.191+0000 7fa0aeffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 0x7fa0b006f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:50340/0 (socket says 172.21.15.29:50340) 2023-10-14T16:59:32.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.353+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa098003500 con 0x7fa0b006fa90 2023-10-14T16:59:32.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.354+0000 7fa0af7fe640 1 -- 172.21.15.29:0/2375539473 >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 msgr2=0x7fa0b006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:32.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.354+0000 7fa0af7fe640 1 --2- 172.21.15.29:0/2375539473 >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 0x7fa0b006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.354+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1098118171 0 0) 0x7fa098003500 con 0x7fa0b006fa90 2023-10-14T16:59:32.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.355+0000 7fa0af7fe640 1 -- 172.21.15.29:0/2375539473 >> v1:172.21.15.186:6789/0 conn(0x7fa0b006fa90 legacy=0x7fa0b00cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.355+0000 7fa0af7fe640 1 -- 172.21.15.29:0/2375539473 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b00cc790 con 0x7fa0b008aa10 2023-10-14T16:59:32.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.358+0000 7fa0af7fe640 1 --2- 172.21.15.29:0/2375539473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c1c0 tx=0x7fa0a0009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=475a5b2b5cc6ab37 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.358+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8001670 con 0x7fa0b008aa10 2023-10-14T16:59:32.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.359+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a002cb80 con 0x7fa0b008aa10 2023-10-14T16:59:32.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.359+0000 7fa0ae7fc640 1 -- 172.21.15.29:0/2375539473 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a00343f0 con 0x7fa0b008aa10 2023-10-14T16:59:32.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 -- 172.21.15.29:0/2375539473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 msgr2=0x7fa0b00c8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/2375539473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fa0a000c1c0 tx=0x7fa0a0009c40 comp rx=0 tx=0).stop 2023-10-14T16:59:32.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 -- 172.21.15.29:0/2375539473 shutdown_connections 2023-10-14T16:59:32.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/2375539473 >> v2:172.21.15.186:3300/0 conn(0x7fa0b00c9520 0x7fa0b006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/2375539473 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b008aa10 0x7fa0b00c8fe0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 -- 172.21.15.29:0/2375539473 >> 172.21.15.29:0/2375539473 conn(0x7fa0b00ce990 msgr2=0x7fa0b00ceda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:32.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 -- 172.21.15.29:0/2375539473 shutdown_connections 2023-10-14T16:59:32.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.360+0000 7fa0b5f89640 1 -- 172.21.15.29:0/2375539473 wait complete. 2023-10-14T16:59:32.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.363+0000 7fa0b5f89640 1 Processor -- start 2023-10-14T16:59:32.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.364+0000 7fa0b5f89640 1 -- start start 2023-10-14T16:59:32.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0b006fa90 0x7fa0b00b0b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b00c9520 0x7fa0b00899c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b3f30 con 0x7fa0b008aa10 2023-10-14T16:59:32.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b10b0 con 0x7fa0b006fa90 2023-10-14T16:59:32.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0b5f89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b00b12e0 con 0x7fa0b00c9520 2023-10-14T16:59:32.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.365+0000 7fa0aeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52652/0 (socket says 172.21.15.29:52652) 2023-10-14T16:59:32.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 -- 172.21.15.29:0/3374826167 learned_addr learned my addr 172.21.15.29:0/3374826167 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:32.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0af7fe640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0b006fa90 0x7fa0b00b0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0affff640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b00c9520 0x7fa0b00899c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 -- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b00c9520 msgr2=0x7fa0b00899c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b00c9520 0x7fa0b00899c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 -- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0b006fa90 msgr2=0x7fa0b00b0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0b006fa90 0x7fa0b00b0b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.366+0000 7fa0aeffd640 1 -- 172.21.15.29:0/3374826167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0a00078e0 con 0x7fa0b008aa10 2023-10-14T16:59:32.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa0aeffd640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e870 tx=0x7fa0a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400ee60 con 0x7fa0b008aa10 2023-10-14T16:59:32.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0a400c940 con 0x7fa0b008aa10 2023-10-14T16:59:32.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b00c5170 con 0x7fa0b008aa10 2023-10-14T16:59:32.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a400cd10 con 0x7fa0b008aa10 2023-10-14T16:59:32.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.367+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b0089f60 con 0x7fa0b008aa10 2023-10-14T16:59:32.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.369+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b004e160 con 0x7fa0b008aa10 2023-10-14T16:59:32.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.370+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa0a4013470 con 0x7fa0b008aa10 2023-10-14T16:59:32.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.371+0000 7fa097fff640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 0x7fa09c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.371+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7fa0a40896c0 con 0x7fa0b008aa10 2023-10-14T16:59:32.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.371+0000 7fa0af7fe640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 0x7fa09c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.373+0000 7fa0af7fe640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 0x7fa09c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0035000 tx=0x7fa0a0030000 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.376+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0a40545a0 con 0x7fa0b008aa10 2023-10-14T16:59:32.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.653+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fa0b0089f00 con 0x7fa0b008aa10 2023-10-14T16:59:32.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.658+0000 7fa097fff640 1 -- 172.21.15.29:0/3374826167 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v17) v1 ==== 190+0+147 (secure 0 0 0) 0x7fa0a40170b0 con 0x7fa0b008aa10 2023-10-14T16:59:32.659 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T16:59:32.659 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T16:59:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T16:59:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T16:59:32.660 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T16:59:32.660 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T16:59:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.662+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 msgr2=0x7fa09c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.662+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 0x7fa09c06d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0035000 tx=0x7fa0a0030000 comp rx=0 tx=0).stop 2023-10-14T16:59:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.662+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 msgr2=0x7fa0b00c4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.662+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 secure :-1 s=READY pgs=1407 cs=0 l=1 rev1=1 crypto rx=0x7fa0a400e870 tx=0x7fa0a400ed30 comp rx=0 tx=0).stop 2023-10-14T16:59:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.662+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 shutdown_connections 2023-10-14T16:59:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7fa09c06afc0 0x7fa09c06d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa0b00c9520 0x7fa0b00899c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0b008aa10 0x7fa0b00c4c30 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 --2- 172.21.15.29:0/3374826167 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa0b006fa90 0x7fa0b00b0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 >> 172.21.15.29:0/3374826167 conn(0x7fa0b00ce990 msgr2=0x7fa0b006d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 shutdown_connections 2023-10-14T16:59:32.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.663+0000 7fa0b5f89640 1 -- 172.21.15.29:0/3374826167 wait complete. 2023-10-14T16:59:32.684 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T16:59:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.908+0000 7f9e0c859640 1 Processor -- start 2023-10-14T16:59:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.908+0000 7f9e0c859640 1 -- start start 2023-10-14T16:59:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159950 0x7f9e0815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c100 0x7f9e0806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0815e490 con 0x7f9e08159950 2023-10-14T16:59:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e08070df0 con 0x7f9e0806c100 2023-10-14T16:59:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e0c859640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0815aef0 con 0x7f9e08070450 2023-10-14T16:59:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e077fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159950 0x7f9e0815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e077fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e08159950 0x7f9e0815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52658/0 (socket says 172.21.15.29:52658) 2023-10-14T16:59:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.910+0000 7f9e077fe640 1 -- 172.21.15.29:0/2785938315 learned_addr learned my addr 172.21.15.29:0/2785938315 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e07fff640 1 --2- 172.21.15.29:0/2785938315 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c100 0x7f9e0806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e06ffd640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e077fe640 1 -- 172.21.15.29:0/2785938315 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 msgr2=0x7f9e080708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e077fe640 1 -- 172.21.15.29:0/2785938315 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c100 msgr2=0x7f9e0806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2785938315 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c100 0x7f9e0806c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.911+0000 7f9e077fe640 1 -- 172.21.15.29:0/2785938315 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0815d910 con 0x7f9e08159950 2023-10-14T16:59:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.912+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815a9b0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f9e080b0120 tx=0x7f9df802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28dcfcf36552661a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.913+0000 7f9e067fc640 1 -- 172.21.15.29:0/2785938315 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df80317c0 con 0x7f9e08159950 2023-10-14T16:59:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.913+0000 7f9e067fc640 1 -- 172.21.15.29:0/2785938315 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9df8031920 con 0x7f9e08159950 2023-10-14T16:59:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.913+0000 7f9e067fc640 1 -- 172.21.15.29:0/2785938315 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9df8031cb0 con 0x7f9e08159950 2023-10-14T16:59:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.913+0000 7f9e0c859640 1 -- 172.21.15.29:0/2785938315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 msgr2=0x7f9e0815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.913+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815a9b0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 crypto rx=0x7f9e080b0120 tx=0x7f9df802b860 comp rx=0 tx=0).stop 2023-10-14T16:59:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 -- 172.21.15.29:0/2785938315 shutdown_connections 2023-10-14T16:59:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2785938315 >> v2:172.21.15.186:3300/0 conn(0x7f9e0806c100 0x7f9e0806c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2785938315 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815a9b0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 -- 172.21.15.29:0/2785938315 >> 172.21.15.29:0/2785938315 conn(0x7f9e080ce6a0 msgr2=0x7f9e080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 -- 172.21.15.29:0/2785938315 shutdown_connections 2023-10-14T16:59:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.914+0000 7f9e0c859640 1 -- 172.21.15.29:0/2785938315 wait complete. 2023-10-14T16:59:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.915+0000 7f9e0c859640 1 Processor -- start 2023-10-14T16:59:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 -- start start 2023-10-14T16:59:32.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 0x7f9e0815b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080c21b0 con 0x7f9e08159950 2023-10-14T16:59:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0814c680 con 0x7f9e0806c100 2023-10-14T16:59:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.916+0000 7f9e0c859640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e080c2400 con 0x7f9e08070450 2023-10-14T16:59:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52672/0 (socket says 172.21.15.29:52672) 2023-10-14T16:59:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 -- 172.21.15.29:0/2770375954 learned_addr learned my addr 172.21.15.29:0/2770375954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e06ffd640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 0x7f9e0815b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 -- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 msgr2=0x7f9e080c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 -- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 msgr2=0x7f9e0815b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 0x7f9e0815b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e07fff640 1 -- 172.21.15.29:0/2770375954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df80094d0 con 0x7f9e08159950 2023-10-14T16:59:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.917+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 0x7f9e0815b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9e07fff640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007e60 tx=0x7f9e0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00014420 con 0x7f9e08159950 2023-10-14T16:59:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e000145a0 con 0x7f9e08159950 2023-10-14T16:59:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e00014950 con 0x7f9e08159950 2023-10-14T16:59:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e080c26e0 con 0x7f9e08159950 2023-10-14T16:59:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.918+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e080c29c0 con 0x7f9e08159950 2023-10-14T16:59:32.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.920+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0804e9b0 con 0x7f9e08159950 2023-10-14T16:59:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.920+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9e00014ab0 con 0x7f9e08159950 2023-10-14T16:59:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.921+0000 7f9deffff640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 0x7f9df406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.921+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f9e0008a080 con 0x7f9e08159950 2023-10-14T16:59:32.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.922+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 0x7f9df406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.923+0000 7f9e077fe640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 0x7f9df406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9df8036000 tx=0x7f9df803d040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:32.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:32.926+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e00055080 con 0x7f9e08159950 2023-10-14T16:59:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.158+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e080b4620 con 0x7f9e08159950 2023-10-14T16:59:33.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.159+0000 7f9deffff640 1 -- 172.21.15.29:0/2770375954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f9e000586a0 con 0x7f9e08159950 2023-10-14T16:59:33.168 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:33.168 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T16:59:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T16:59:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 msgr2=0x7f9df406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 0x7f9df406d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9df8036000 tx=0x7f9df803d040 comp rx=0 tx=0).stop 2023-10-14T16:59:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 msgr2=0x7f9e0815cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 crypto rx=0x7f9e00007e60 tx=0x7f9e0000e4b0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 shutdown_connections 2023-10-14T16:59:33.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f9df406af90 0x7f9df406d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e08159950 0x7f9e0815cde0 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e08070450 0x7f9e080c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 --2- 172.21.15.29:0/2770375954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e0806c100 0x7f9e0815b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.162+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 >> 172.21.15.29:0/2770375954 conn(0x7f9e080ce6a0 msgr2=0x7f9e080cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.163+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 shutdown_connections 2023-10-14T16:59:33.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.163+0000 7f9e0c859640 1 -- 172.21.15.29:0/2770375954 wait complete. 2023-10-14T16:59:33.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 57, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T16:59:26.282689+0000', 'modified': '2023-10-14T16:59:29.375892+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8037}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8037': {'gid': 8037, 'name': 'c', 'rank': 0, 'incarnation': 53, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/80282461', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 80282461}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 80282461}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [15], 'metadata_pool': 14, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T16:59:33.584 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T16:59:33.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T16:59:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.825+0000 7f475ab16640 1 Processor -- start 2023-10-14T16:59:33.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.825+0000 7f475ab16640 1 -- start start 2023-10-14T16:59:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.827+0000 7f475ab16640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f475415ad70 0x7f4754069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.827+0000 7f475ab16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540c9ca0 0x7f475406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.827+0000 7f475ab16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47540cb030 con 0x7f47540ca3b0 2023-10-14T16:59:33.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.827+0000 7f475ab16640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47540c91c0 con 0x7f475415ad70 2023-10-14T16:59:33.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.827+0000 7f475ab16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540c94a0 con 0x7f47540c9ca0 2023-10-14T16:59:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47590b3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f47540ca3b0 0x7f47540ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47834/0 (socket says 172.21.15.29:47834) 2023-10-14T16:59:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47590b3640 1 -- 172.21.15.29:0/891306096 learned_addr learned my addr 172.21.15.29:0/891306096 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:33.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47588b2640 1 --2- 172.21.15.29:0/891306096 >> v2:172.21.15.29:3300/0 conn(0x7f475415ad70 0x7f4754069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f474bfff640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540c9ca0 0x7f475406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f474b7fe640 1 -- 172.21.15.29:0/891306096 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186474801 0 0) 0x7f47540cb030 con 0x7f47540ca3b0 2023-10-14T16:59:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f474b7fe640 1 -- 172.21.15.29:0/891306096 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473c003520 con 0x7f47540ca3b0 2023-10-14T16:59:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47588b2640 1 -- 172.21.15.29:0/891306096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540c9ca0 msgr2=0x7f475406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47588b2640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540c9ca0 0x7f475406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47588b2640 1 -- 172.21.15.29:0/891306096 >> v1:172.21.15.29:6789/0 conn(0x7f47540ca3b0 legacy=0x7f47540ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.828+0000 7f47588b2640 1 -- 172.21.15.29:0/891306096 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47540c9780 con 0x7f475415ad70 2023-10-14T16:59:33.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f47588b2640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475415ad70 0x7f4754069da0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a23b52bc407ed1f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f474b7fe640 1 -- 172.21.15.29:0/891306096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f475002d040 con 0x7f475415ad70 2023-10-14T16:59:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f474b7fe640 1 -- 172.21.15.29:0/891306096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4750031810 con 0x7f475415ad70 2023-10-14T16:59:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f474b7fe640 1 -- 172.21.15.29:0/891306096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750031ba0 con 0x7f475415ad70 2023-10-14T16:59:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f475ab16640 1 -- 172.21.15.29:0/891306096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475415ad70 msgr2=0x7f4754069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.830+0000 7f475ab16640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475415ad70 0x7f4754069da0 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475002b8b0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 -- 172.21.15.29:0/891306096 shutdown_connections 2023-10-14T16:59:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540c9ca0 0x7f475406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 --2- 172.21.15.29:0/891306096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f475415ad70 0x7f4754069da0 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 -- 172.21.15.29:0/891306096 >> 172.21.15.29:0/891306096 conn(0x7f47540ce9b0 msgr2=0x7f47540cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 -- 172.21.15.29:0/891306096 shutdown_connections 2023-10-14T16:59:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.831+0000 7f475ab16640 1 -- 172.21.15.29:0/891306096 wait complete. 2023-10-14T16:59:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.833+0000 7f475ab16640 1 Processor -- start 2023-10-14T16:59:33.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.833+0000 7f475ab16640 1 -- start start 2023-10-14T16:59:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540ca3b0 0x7f475415b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f475415ad70 0x7f47540b2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b1470 con 0x7f47540c9ca0 2023-10-14T16:59:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b08f0 con 0x7f475415ad70 2023-10-14T16:59:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f475ab16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47540b0bd0 con 0x7f47540ca3b0 2023-10-14T16:59:33.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.834+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52682/0 (socket says 172.21.15.29:52682) 2023-10-14T16:59:33.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 -- 172.21.15.29:0/2334039625 learned_addr learned my addr 172.21.15.29:0/2334039625 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:33.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f474bfff640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540ca3b0 0x7f475415b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 -- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540ca3b0 msgr2=0x7f475415b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540ca3b0 0x7f475415b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 -- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f475415ad70 msgr2=0x7f47540b2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f475415ad70 0x7f47540b2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 -- 172.21.15.29:0/2334039625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750009400 con 0x7f47540c9ca0 2023-10-14T16:59:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.835+0000 7f47588b2640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f47500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.836+0000 7f47497fa640 1 -- 172.21.15.29:0/2334039625 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750006e10 con 0x7f47540c9ca0 2023-10-14T16:59:33.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.836+0000 7f47497fa640 1 -- 172.21.15.29:0/2334039625 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f475003ad60 con 0x7f47540c9ca0 2023-10-14T16:59:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.836+0000 7f47497fa640 1 -- 172.21.15.29:0/2334039625 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750044630 con 0x7f47540c9ca0 2023-10-14T16:59:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.836+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47540b0eb0 con 0x7f47540c9ca0 2023-10-14T16:59:33.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.836+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540c7410 con 0x7f47540c9ca0 2023-10-14T16:59:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.838+0000 7f47497fa640 1 -- 172.21.15.29:0/2334039625 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f475004d450 con 0x7f47540c9ca0 2023-10-14T16:59:33.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.839+0000 7f47497fa640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 0x7f474006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.839+0000 7f47497fa640 1 -- 172.21.15.29:0/2334039625 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f47500aba40 con 0x7f47540c9ca0 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.839+0000 7f474bfff640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 0x7f474006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.840+0000 7f474bfff640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 0x7f474006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47540c68c0 tx=0x7f4744008040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.840+0000 7f475d781740 1 Processor -- start 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.840+0000 7f475d781740 1 -- start start 2023-10-14T16:59:33.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611e6120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611b67d0 0x55dd611e6660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd610d27a0 con 0x55dd611e3390 2023-10-14T16:59:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd61124740 con 0x55dd611b67d0 2023-10-14T16:59:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.841+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd61121e30 con 0x55dd611e3770 2023-10-14T16:59:33.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52698/0 (socket says 172.21.15.29:52698) 2023-10-14T16:59:33.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 -- 172.21.15.29:0/1881232724 learned_addr learned my addr 172.21.15.29:0/1881232724 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:33.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 -- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 msgr2=0x55dd611e6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611e6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 -- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611b67d0 msgr2=0x55dd611e6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611b67d0 0x55dd611e6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 -- 172.21.15.29:0/1881232724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd610e1910 con 0x55dd611e3390 2023-10-14T16:59:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f47588b2640 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475007a090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67127b44406d1fa0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.842+0000 7f473affd640 1 -- 172.21.15.29:0/1881232724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750082970 con 0x55dd611e3390 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f473affd640 1 -- 172.21.15.29:0/1881232724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500a9e90 con 0x55dd611e3390 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f473affd640 1 -- 172.21.15.29:0/1881232724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47500aa000 con 0x55dd611e3390 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 -- 172.21.15.29:0/1881232724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 msgr2=0x55dd611b6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 crypto rx=0x7f47500097a0 tx=0x7f475007a090 comp rx=0 tx=0).stop 2023-10-14T16:59:33.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 -- 172.21.15.29:0/1881232724 shutdown_connections 2023-10-14T16:59:33.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611b67d0 0x55dd611e6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611e6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 --2- 172.21.15.29:0/1881232724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b6290 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 -- 172.21.15.29:0/1881232724 >> 172.21.15.29:0/1881232724 conn(0x55dd611b85a0 msgr2=0x55dd61201120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 -- 172.21.15.29:0/1881232724 shutdown_connections 2023-10-14T16:59:33.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.843+0000 7f475d781740 1 -- 172.21.15.29:0/1881232724 wait complete. 2023-10-14T16:59:33.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.844+0000 7f475d781740 1 -- start start 2023-10-14T16:59:33.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.844+0000 7f475d781740 1 Processor -- start 2023-10-14T16:59:33.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611dc470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611ccc00 0x55dd611cd080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd6111d720 con 0x55dd611e3390 2023-10-14T16:59:33.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd610bb970 con 0x55dd611ccc00 2023-10-14T16:59:33.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd610bb610 con 0x55dd611e3770 2023-10-14T16:59:33.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.845+0000 7f475d781740 10 client.0 _ll_register_callbacks cb 0x55dd610dd9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:59:33.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52700/0 (socket says 172.21.15.29:52700) 2023-10-14T16:59:33.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 -- 172.21.15.29:0/542503260 learned_addr learned my addr 172.21.15.29:0/542503260 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:33.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 -- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 msgr2=0x55dd611dc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611dc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 -- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611ccc00 msgr2=0x55dd611cd080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611ccc00 0x55dd611cd080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 -- 172.21.15.29:0/542503260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47540c7410 con 0x55dd611e3390 2023-10-14T16:59:33.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47588b2640 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f475002be40 tx=0x7f475002bdd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b5246217fb544e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.846+0000 7f47337fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:59:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750087070 con 0x55dd611e3390 2023-10-14T16:59:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f4750082970 con 0x55dd611e3390 2023-10-14T16:59:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dd61144160 con 0x55dd611e3390 2023-10-14T16:59:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f47500aa000 con 0x55dd611e3390 2023-10-14T16:59:33.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 10 client.8157 fetch_fsmap learned FSMap version 57 2023-10-14T16:59:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55dd6111bda0 con 0x55dd611e3390 2023-10-14T16:59:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55dd61144160 con 0x55dd611e3390 2023-10-14T16:59:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f4750079b60 con 0x55dd611e3390 2023-10-14T16:59:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 10 client.8157 fetch_fsmap finished waiting for FSMap version 57 2023-10-14T16:59:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 10 client.8157 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:59:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 20 client.8157 populate_metadata read hostname 'smithi029' 2023-10-14T16:59:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611c9e20 0x55dd611ca200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 4 client.8157 mds_command: new command op to 8037 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T16:59:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.847+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 --> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dd610efcc0 con 0x55dd611c9e20 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.848+0000 7f474bfff640 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611c9e20 0x55dd611ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.848+0000 7f474bfff640 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611c9e20 0x55dd611ca200 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=955093eee50074ca server_cookie=5fbfb48146c9dabf in_seq=0 out_seq=0 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.848+0000 7f47337fe640 10 client.8157 ms_handle_connect on v2:172.21.15.29:6836/80282461 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f47337fe640 1 -- 172.21.15.29:0/542503260 <== mds.0 v2:172.21.15.29:6836/80282461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55dd610efcc0 con 0x55dd611c9e20 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f47337fe640 10 client.8157 handle_command_reply: tid=0 2023-10-14T16:59:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 client.8157 shutdown 2023-10-14T16:59:33.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 msgr2=0x55dd611def50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 crypto rx=0x7f475002be40 tx=0x7f475002bdd0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 shutdown_connections 2023-10-14T16:59:33.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611c9e20 0x55dd611ca200 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611ccc00 0x55dd611cd080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611dc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 --2- 172.21.15.29:0/542503260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611def50 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f47337fe640 0 client.8157 ms_handle_reset on v2:172.21.15.29:6836/80282461 2023-10-14T16:59:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.849+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 >> 172.21.15.29:0/542503260 conn(0x55dd611b67d0 msgr2=0x55dd611ba350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.850+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 shutdown_connections 2023-10-14T16:59:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.850+0000 7f475d781740 1 -- 172.21.15.29:0/542503260 wait complete. 2023-10-14T16:59:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.850+0000 7f475d781740 20 client.8157 trim_cache size 0 max 16384 2023-10-14T16:59:33.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.867+0000 7f475d781740 1 Processor -- start 2023-10-14T16:59:33.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.867+0000 7f475d781740 1 -- start start 2023-10-14T16:59:33.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3390 0x55dd611cd000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd60e81f20 con 0x55dd611ccc00 2023-10-14T16:59:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd611e4ac0 con 0x55dd611e3390 2023-10-14T16:59:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd611e2a10 con 0x55dd611e3770 2023-10-14T16:59:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52702/0 (socket says 172.21.15.29:52702) 2023-10-14T16:59:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 -- 172.21.15.29:0/2164883982 learned_addr learned my addr 172.21.15.29:0/2164883982 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f474bfff640 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3390 0x55dd611cd000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 -- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 msgr2=0x55dd611b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 -- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3390 msgr2=0x55dd611cd000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3390 0x55dd611cd000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.868+0000 7f47588b2640 1 -- 172.21.15.29:0/2164883982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dd610bab80 con 0x55dd611ccc00 2023-10-14T16:59:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f47588b2640 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4750085040 tx=0x7f4750091000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7029c2adcab30b98 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f473b7fe640 1 -- 172.21.15.29:0/2164883982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750087040 con 0x55dd611ccc00 2023-10-14T16:59:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f473b7fe640 1 -- 172.21.15.29:0/2164883982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500aa000 con 0x55dd611ccc00 2023-10-14T16:59:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f473b7fe640 1 -- 172.21.15.29:0/2164883982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4750082970 con 0x55dd611ccc00 2023-10-14T16:59:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f475d781740 1 -- 172.21.15.29:0/2164883982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 msgr2=0x55dd611c93b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.869+0000 7f475d781740 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 crypto rx=0x7f4750085040 tx=0x7f4750091000 comp rx=0 tx=0).stop 2023-10-14T16:59:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 -- 172.21.15.29:0/2164883982 shutdown_connections 2023-10-14T16:59:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611e3770 0x55dd611b0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3390 0x55dd611cd000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 --2- 172.21.15.29:0/2164883982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611ccc00 0x55dd611c93b0 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 -- 172.21.15.29:0/2164883982 >> 172.21.15.29:0/2164883982 conn(0x55dd611b67d0 msgr2=0x55dd611ea0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 -- 172.21.15.29:0/2164883982 shutdown_connections 2023-10-14T16:59:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 -- 172.21.15.29:0/2164883982 wait complete. 2023-10-14T16:59:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 -- start start 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.870+0000 7f475d781740 1 Processor -- start 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611ccc00 0x55dd61200b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3770 0x55dd611daee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd611ea6d0 con 0x55dd611e3390 2023-10-14T16:59:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dd61201510 con 0x55dd611e3770 2023-10-14T16:59:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.871+0000 7f475d781740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55dd61271c10 con 0x55dd611ccc00 2023-10-14T16:59:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f475d781740 10 client.0 _ll_register_callbacks cb 0x7f47500a6070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T16:59:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52706/0 (socket says 172.21.15.29:52706) 2023-10-14T16:59:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 -- 172.21.15.29:0/430236723 learned_addr learned my addr 172.21.15.29:0/430236723 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 -- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611ccc00 msgr2=0x55dd61200b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611ccc00 0x55dd61200b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 -- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3770 msgr2=0x55dd611daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3770 0x55dd611daee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 -- 172.21.15.29:0/430236723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dd6111bda0 con 0x55dd611e3390 2023-10-14T16:59:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f474bfff640 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f475406a7f0 tx=0x7f4744009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f70225217fa95a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.872+0000 7f47337fe640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T16:59:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744023480 con 0x55dd611e3390 2023-10-14T16:59:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f474400e840 con 0x55dd611e3390 2023-10-14T16:59:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f47500a9e90 con 0x55dd611e3390 2023-10-14T16:59:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=57) v2 ==== 24+0+0 (secure 0 0 0) 0x7f474400ea00 con 0x55dd611e3390 2023-10-14T16:59:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f475d781740 10 client.8163 fetch_fsmap learned FSMap version 57 2023-10-14T16:59:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=57,monmap=2+}) v3 -- 0x55dd610e0170 con 0x55dd611e3390 2023-10-14T16:59:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47500a9e90 con 0x55dd611e3390 2023-10-14T16:59:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.873+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 57) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f474400eb60 con 0x55dd611e3390 2023-10-14T16:59:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 10 client.8163 fetch_fsmap finished waiting for FSMap version 57 2023-10-14T16:59:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 10 client.8163 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T16:59:33.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 20 client.8163 populate_metadata read hostname 'smithi029' 2023-10-14T16:59:33.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611b5b60 0x55dd611db420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:33.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 4 client.8163 mds_command: new command op to 8037 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T16:59:33.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 --> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55dd61116700 con 0x55dd611b5b60 2023-10-14T16:59:33.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f47588b2640 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611b5b60 0x55dd611db420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:33.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f47588b2640 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611b5b60 0x55dd611db420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=932062c799eb06a0 server_cookie=179233d520c95466 in_seq=0 out_seq=0 2023-10-14T16:59:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.874+0000 7f47337fe640 10 client.8163 ms_handle_connect on v2:172.21.15.29:6836/80282461 2023-10-14T16:59:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f47337fe640 1 -- 172.21.15.29:0/430236723 <== mds.0 v2:172.21.15.29:6836/80282461 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55dd61116700 con 0x55dd611b5b60 2023-10-14T16:59:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f47337fe640 10 client.8163 handle_command_reply: tid=0 2023-10-14T16:59:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 client.8163 shutdown 2023-10-14T16:59:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 msgr2=0x55dd611b55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 crypto rx=0x7f475406a7f0 tx=0x7f4744009d80 comp rx=0 tx=0).stop 2023-10-14T16:59:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 shutdown_connections 2023-10-14T16:59:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:6836/80282461,v1:172.21.15.29:6837/80282461] conn(0x55dd611b5b60 0x55dd611db420 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55dd611e3770 0x55dd611daee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55dd611e3390 0x55dd611b55b0 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 --2- 172.21.15.29:0/430236723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55dd611ccc00 0x55dd61200b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f47337fe640 0 client.8163 ms_handle_reset on v2:172.21.15.29:6836/80282461 2023-10-14T16:59:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 >> 172.21.15.29:0/430236723 conn(0x55dd611b67d0 msgr2=0x55dd611b0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 shutdown_connections 2023-10-14T16:59:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 1 -- 172.21.15.29:0/430236723 wait complete. 2023-10-14T16:59:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.875+0000 7f475d781740 20 client.8163 trim_cache size 0 max 16384 2023-10-14T16:59:33.892 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T16:59:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T16:59:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T16:59:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8037, 2023-10-14T16:59:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T16:59:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T16:59:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T16:59:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 7.0516175759999999, 2023-10-14T16:59:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 57, 2023-10-14T16:59:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 125, 2023-10-14T16:59:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 125, 2023-10-14T16:59:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.422707578000001 2023-10-14T16:59:33.894 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T16:59:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 msgr2=0x7f474006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 0x7f474006d480 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f47540c68c0 tx=0x7f4744008040 comp rx=0 tx=0).stop 2023-10-14T16:59:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 msgr2=0x7f47540c6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 secure :-1 s=READY pgs=1412 cs=0 l=1 rev1=1 crypto rx=0x7f4750000c00 tx=0x7f47500046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 shutdown_connections 2023-10-14T16:59:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f474006afc0 0x7f474006d480 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f475415ad70 0x7f47540b2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47540ca3b0 0x7f475415b730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 --2- 172.21.15.29:0/2334039625 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47540c9ca0 0x7f47540c6ed0 unknown :-1 s=CLOSED pgs=1412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 >> 172.21.15.29:0/2334039625 conn(0x7f47540ce9b0 msgr2=0x7f47540afb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 shutdown_connections 2023-10-14T16:59:33.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:33.876+0000 7f475ab16640 1 -- 172.21.15.29:0/2334039625 wait complete. 2023-10-14T16:59:33.898 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T16:59:34.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.115+0000 7f351e490640 1 Processor -- start 2023-10-14T16:59:34.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.115+0000 7f351e490640 1 -- start start 2023-10-14T16:59:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.116+0000 7f351e490640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f351815cdb0 0x7f35180cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.116+0000 7f351e490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180c8cf0 con 0x7f35180acfa0 2023-10-14T16:59:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.116+0000 7f351e490640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35180ccca0 con 0x7f35180c9a20 2023-10-14T16:59:34.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.116+0000 7f351e490640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351815d190 con 0x7f351815cdb0 2023-10-14T16:59:34.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351dc8f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35180acfa0 0x7f351806fea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47836/0 (socket says 172.21.15.29:47836) 2023-10-14T16:59:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351d48e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f351815cdb0 0x7f35180cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351dc8f640 1 -- 172.21.15.29:0/3694217504 learned_addr learned my addr 172.21.15.29:0/3694217504 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f350ffff640 1 -- 172.21.15.29:0/3694217504 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 135651613 0 0) 0x7f35180c8cf0 con 0x7f35180acfa0 2023-10-14T16:59:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f350ffff640 1 -- 172.21.15.29:0/3694217504 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3504003520 con 0x7f35180acfa0 2023-10-14T16:59:34.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351d48e640 1 -- 172.21.15.29:0/3694217504 >> v1:172.21.15.186:6789/0 conn(0x7f35180c9a20 legacy=0x7f35180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351d48e640 1 -- 172.21.15.29:0/3694217504 >> v1:172.21.15.29:6789/0 conn(0x7f35180acfa0 legacy=0x7f351806fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.117+0000 7f351d48e640 1 -- 172.21.15.29:0/3694217504 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35180c8f40 con 0x7f351815cdb0 2023-10-14T16:59:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.119+0000 7f351d48e640 1 --2- 172.21.15.29:0/3694217504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f35180cc270 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f35140097a0 tx=0x7f351402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c4732cd3b954d91 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.119+0000 7f350ffff640 1 -- 172.21.15.29:0/3694217504 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514031830 con 0x7f351815cdb0 2023-10-14T16:59:34.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.119+0000 7f350ffff640 1 -- 172.21.15.29:0/3694217504 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3514031990 con 0x7f351815cdb0 2023-10-14T16:59:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.119+0000 7f350ffff640 1 -- 172.21.15.29:0/3694217504 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3514031d20 con 0x7f351815cdb0 2023-10-14T16:59:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 -- 172.21.15.29:0/3694217504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 msgr2=0x7f35180cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 --2- 172.21.15.29:0/3694217504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f35180cc270 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 crypto rx=0x7f35140097a0 tx=0x7f351402b8d0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 -- 172.21.15.29:0/3694217504 shutdown_connections 2023-10-14T16:59:34.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 --2- 172.21.15.29:0/3694217504 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f35180cc270 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 -- 172.21.15.29:0/3694217504 >> 172.21.15.29:0/3694217504 conn(0x7f35180b0a90 msgr2=0x7f35180ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 -- 172.21.15.29:0/3694217504 shutdown_connections 2023-10-14T16:59:34.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.120+0000 7f351e490640 1 -- 172.21.15.29:0/3694217504 wait complete. 2023-10-14T16:59:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.122+0000 7f351e490640 1 Processor -- start 2023-10-14T16:59:34.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.122+0000 7f351e490640 1 -- start start 2023-10-14T16:59:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35180acfa0 0x7f3518157d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 0x7f351815e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:34.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180c60f0 con 0x7f351815cdb0 2023-10-14T16:59:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35180c63d0 con 0x7f35180c9a20 2023-10-14T16:59:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351e490640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f351806b3d0 con 0x7f35180acfa0 2023-10-14T16:59:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351dc8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351dc8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52728/0 (socket says 172.21.15.29:52728) 2023-10-14T16:59:34.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.123+0000 7f351dc8f640 1 -- 172.21.15.29:0/2317594837 learned_addr learned my addr 172.21.15.29:0/2317594837 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351cc8d640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 0x7f351815e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 -- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35180acfa0 msgr2=0x7f3518157d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351d48e640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35180acfa0 0x7f3518157d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35180acfa0 0x7f3518157d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 -- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 msgr2=0x7f351815e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 0x7f351815e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 -- 172.21.15.29:0/2317594837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3514009400 con 0x7f351815cdb0 2023-10-14T16:59:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351cc8d640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 0x7f351815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f351dc8f640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f351000cb40 tx=0x7f351000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.124+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f351000aca0 con 0x7f351815cdb0 2023-10-14T16:59:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.125+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f351000ae00 con 0x7f351815cdb0 2023-10-14T16:59:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.125+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351806b6b0 con 0x7f351815cdb0 2023-10-14T16:59:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.125+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3510012620 con 0x7f351815cdb0 2023-10-14T16:59:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.125+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35180c66b0 con 0x7f351815cdb0 2023-10-14T16:59:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.126+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180c94e0 con 0x7f351815cdb0 2023-10-14T16:59:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.127+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3510007070 con 0x7f351815cdb0 2023-10-14T16:59:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.128+0000 7f350dffb640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 0x7f34fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.128+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f35100898a0 con 0x7f351815cdb0 2023-10-14T16:59:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.128+0000 7f351d48e640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 0x7f34fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.129+0000 7f351d48e640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 0x7f34fc06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3514036000 tx=0x7f351403c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.133+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35180c94e0 con 0x7f351815cdb0 2023-10-14T16:59:34.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.367+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f351806bbd0 con 0x7f351815cdb0 2023-10-14T16:59:34.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.368+0000 7f350dffb640 1 -- 172.21.15.29:0/2317594837 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v57) v1 ==== 53+0+73 (secure 0 0 0) 0x7f351806bbd0 con 0x7f351815cdb0 2023-10-14T16:59:34.370 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T16:59:34.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 msgr2=0x7f34fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 0x7f34fc06d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f3514036000 tx=0x7f351403c040 comp rx=0 tx=0).stop 2023-10-14T16:59:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 msgr2=0x7f351815e990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 crypto rx=0x7f351000cb40 tx=0x7f351000a060 comp rx=0 tx=0).stop 2023-10-14T16:59:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 shutdown_connections 2023-10-14T16:59:34.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f34fc06af90 0x7f34fc06d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f351815cdb0 0x7f351815e990 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35180c9a20 0x7f351815e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 --2- 172.21.15.29:0/2317594837 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35180acfa0 0x7f3518157d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.370+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 >> 172.21.15.29:0/2317594837 conn(0x7f35180b0a90 msgr2=0x7f35180af860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:34.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.371+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 shutdown_connections 2023-10-14T16:59:34.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:34.371+0000 7f351e490640 1 -- 172.21.15.29:0/2317594837 wait complete. 2023-10-14T16:59:34.391 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T16:59:34.392 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T16:59:34.392 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T16:59:34.393 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T16:59:34.393 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T16:59:34.393 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T16:59:34.393 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T16:59:34.394 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T16:59:34.394 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:59:34.395 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.395 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T16:59:34.451 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T16:59:34.452 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T16:59:34.452 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T16:59:34.452 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.453 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T16:59:34.453 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.454 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:59:34.454 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:59:34.454 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T16:59:34.454 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:59:34.454 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T16:59:34.455 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T16:59:34.455 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T16:59:34.455 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T16:59:34.455 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T16:59:34.456 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T16:59:34.456 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41202sec preferred_lft 41202sec 2023-10-14T16:59:34.457 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T16:59:34.457 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.457 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T16:59:34.457 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:59:34.457 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T16:59:34.458 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T16:59:34.458 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T16:59:34.458 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T16:59:34.458 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.459 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T16:59:34.459 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.459 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:59:34.460 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T16:59:34.462 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T16:59:34.462 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.462 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T16:59:34.463 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T16:59:34.463 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T16:59:34.463 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T16:59:34.464 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.464 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T16:59:34.565 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T16:59:34.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.571 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T16:59:34.630 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T16:59:34.631 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T16:59:34.631 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T16:59:34.631 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T16:59:34.631 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T16:59:34.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.632 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T16:59:34.633 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T16:59:34.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T16:59:34.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T16:59:34.634 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T16:59:34.635 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T16:59:34.834 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.835 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T16:59:34.890 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T16:59:34.890 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T16:59:34.891 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:59:34.892 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:34.892 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T16:59:34.994 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:59:34.994 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T16:59:35.052 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T16:59:35.052 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T16:59:35.053 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:59:35.110 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T16:59:35.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:35.111 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:59:35.275 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:59:35.275 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:59:35.276 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:59:35.297 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:59:35.297 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:59:35.298 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:59:35.298 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:59:35.298 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:59:35.326 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T16:59:35.326 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T16:59:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T16:59:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T16:59:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T16:59:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T16:59:35.327 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T16:59:35.328 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T16:59:35.331 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T16:59:35.331 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T16:59:35.332 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T16:59:35.332 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T16:59:35.332 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T16:59:35.337 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T16:59:35.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:35.337 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T16:59:35.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:35.442 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T16:59:35.504 INFO:teuthology.orchestra.run.smithi029.stdout:client5888 2023-10-14T16:59:35.505 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T16:59:35.505 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T16:59:35.565 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T16:59:35.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.797+0000 7f504f59e640 1 Processor -- start 2023-10-14T16:59:35.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.798+0000 7f504f59e640 1 -- start start 2023-10-14T16:59:35.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.799+0000 7f504f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f50500ccb00 0x7f50500ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.799+0000 7f504f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50500ad4d0 0x7f50500ad960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.799+0000 7f504f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50500cd4a0 con 0x7f5050153740 2023-10-14T16:59:35.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f505015ac60 con 0x7f50500ad4d0 2023-10-14T16:59:35.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5050070030 con 0x7f50500ccb00 2023-10-14T16:59:35.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5050153740 0x7f505015a0e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47850/0 (socket says 172.21.15.29:47850) 2023-10-14T16:59:35.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504e59c640 1 -- 172.21.15.29:0/1107215613 learned_addr learned my addr 172.21.15.29:0/1107215613 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:35.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504ed9d640 1 --2- 172.21.15.29:0/1107215613 >> v2:172.21.15.29:3300/0 conn(0x7f50500ad4d0 0x7f50500ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504dd9b640 1 --2- 172.21.15.29:0/1107215613 >> v2:172.21.15.186:3300/0 conn(0x7f50500ccb00 0x7f50500ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504d59a640 1 -- 172.21.15.29:0/1107215613 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662183100 0 0) 0x7f50500cd4a0 con 0x7f5050153740 2023-10-14T16:59:35.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504d59a640 1 -- 172.21.15.29:0/1107215613 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f503c003520 con 0x7f5050153740 2023-10-14T16:59:35.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504ed9d640 1 -- 172.21.15.29:0/1107215613 >> v2:172.21.15.186:3300/0 conn(0x7f50500ccb00 msgr2=0x7f50500ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:35.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.800+0000 7f504ed9d640 1 --2- 172.21.15.29:0/1107215613 >> v2:172.21.15.186:3300/0 conn(0x7f50500ccb00 0x7f50500ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.801+0000 7f504ed9d640 1 -- 172.21.15.29:0/1107215613 >> v1:172.21.15.29:6789/0 conn(0x7f5050153740 legacy=0x7f505015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:35.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.801+0000 7f504ed9d640 1 -- 172.21.15.29:0/1107215613 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5050070310 con 0x7f50500ad4d0 2023-10-14T16:59:35.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.802+0000 7f504ed9d640 1 --2- 172.21.15.29:0/1107215613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ad4d0 0x7f50500ad960 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f505006d6e0 tx=0x7f504802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=654c610addbe1caf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:35.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.802+0000 7f504d59a640 1 -- 172.21.15.29:0/1107215613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048031830 con 0x7f50500ad4d0 2023-10-14T16:59:35.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.802+0000 7f504d59a640 1 -- 172.21.15.29:0/1107215613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048031990 con 0x7f50500ad4d0 2023-10-14T16:59:35.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.802+0000 7f504d59a640 1 -- 172.21.15.29:0/1107215613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048031d20 con 0x7f50500ad4d0 2023-10-14T16:59:35.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 -- 172.21.15.29:0/1107215613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ad4d0 msgr2=0x7f50500ad960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 --2- 172.21.15.29:0/1107215613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ad4d0 0x7f50500ad960 secure :-1 s=READY pgs=1421 cs=0 l=1 rev1=1 crypto rx=0x7f505006d6e0 tx=0x7f504802b8d0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 -- 172.21.15.29:0/1107215613 shutdown_connections 2023-10-14T16:59:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 --2- 172.21.15.29:0/1107215613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ad4d0 0x7f50500ad960 unknown :-1 s=CLOSED pgs=1421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 --2- 172.21.15.29:0/1107215613 >> v2:172.21.15.186:3300/0 conn(0x7f50500ccb00 0x7f50500ccf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 -- 172.21.15.29:0/1107215613 >> 172.21.15.29:0/1107215613 conn(0x7f50500ce6a0 msgr2=0x7f50500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:35.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 -- 172.21.15.29:0/1107215613 shutdown_connections 2023-10-14T16:59:35.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.803+0000 7f504f59e640 1 -- 172.21.15.29:0/1107215613 wait complete. 2023-10-14T16:59:35.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.805+0000 7f504f59e640 1 Processor -- start 2023-10-14T16:59:35.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.805+0000 7f504f59e640 1 -- start start 2023-10-14T16:59:35.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50500ad4d0 0x7f505014ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5050153740 0x7f50500c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5050152da0 con 0x7f50500ccb00 2023-10-14T16:59:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50500b1fc0 con 0x7f50500ad4d0 2023-10-14T16:59:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50500b22a0 con 0x7f5050153740 2023-10-14T16:59:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52750/0 (socket says 172.21.15.29:52750) 2023-10-14T16:59:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504dd9b640 1 -- 172.21.15.29:0/3149508730 learned_addr learned my addr 172.21.15.29:0/3149508730 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504e59c640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50500ad4d0 0x7f505014ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.806+0000 7f504ed9d640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5050153740 0x7f50500c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 -- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5050153740 msgr2=0x7f50500c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:35.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5050153740 0x7f50500c2710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 -- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50500ad4d0 msgr2=0x7f505014ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:35.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50500ad4d0 0x7f505014ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:35.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 -- 172.21.15.29:0/3149508730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009400 con 0x7f50500ccb00 2023-10-14T16:59:35.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504dd9b640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f503800d800 tx=0x7f503800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:35.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503800b6c0 con 0x7f50500ccb00 2023-10-14T16:59:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f503800b820 con 0x7f50500ccb00 2023-10-14T16:59:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50500b2580 con 0x7f50500ccb00 2023-10-14T16:59:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f503800bbb0 con 0x7f50500ccb00 2023-10-14T16:59:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.807+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50501520b0 con 0x7f50500ccb00 2023-10-14T16:59:35.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.808+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f505004e9b0 con 0x7f50500ccb00 2023-10-14T16:59:35.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.810+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f503800bd10 con 0x7f50500ccb00 2023-10-14T16:59:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.811+0000 7f5046ffd640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 0x7f503006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.811+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f5038089df0 con 0x7f50500ccb00 2023-10-14T16:59:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.811+0000 7f504e59c640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 0x7f503006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.811+0000 7f504e59c640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 0x7f503006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5040001560 tx=0x7f5040009040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:35.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:35.815+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5038054cc0 con 0x7f50500ccb00 2023-10-14T16:59:36.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.038+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 --> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f50500b1e40 con 0x7f503006af90 2023-10-14T16:59:36.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.040+0000 7f5046ffd640 1 -- 172.21.15.29:0/3149508730 <== mgr.7797 v2:172.21.15.29:6832/2416849365 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f504000ec40 con 0x7f503006af90 2023-10-14T16:59:36.042 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T16:59:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T16:59:36.042 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T16:59:36.043 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T16:59:36.043 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T16:59:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 msgr2=0x7f503006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 0x7f503006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5040001560 tx=0x7f5040009040 comp rx=0 tx=0).stop 2023-10-14T16:59:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 msgr2=0x7f50500c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:36.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 secure :-1 s=READY pgs=1422 cs=0 l=1 rev1=1 crypto rx=0x7f503800d800 tx=0x7f503800dcc0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 shutdown_connections 2023-10-14T16:59:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f503006af90 0x7f503006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5050153740 0x7f50500c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50500ccb00 0x7f50500c21d0 unknown :-1 s=CLOSED pgs=1422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 --2- 172.21.15.29:0/3149508730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50500ad4d0 0x7f505014ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 >> 172.21.15.29:0/3149508730 conn(0x7f50500ce6a0 msgr2=0x7f50500af8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.043+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 shutdown_connections 2023-10-14T16:59:36.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.044+0000 7f504f59e640 1 -- 172.21.15.29:0/3149508730 wait complete. 2023-10-14T16:59:36.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T16:59:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.291+0000 7f63cb541640 1 Processor -- start 2023-10-14T16:59:36.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.291+0000 7f63cb541640 1 -- start start 2023-10-14T16:59:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63cb541640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c40addb0 0x7f63c40caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63cb541640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 0x7f63c4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63cb541640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c406a6f0 con 0x7f63c406a2e0 2023-10-14T16:59:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63cb541640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c4161a50 con 0x7f63c40addb0 2023-10-14T16:59:36.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63cb541640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c40cb140 con 0x7f63c40c9000 2023-10-14T16:59:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63c92dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c40addb0 0x7f63c40caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63c92dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63c40addb0 0x7f63c40caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52756/0 (socket says 172.21.15.29:52756) 2023-10-14T16:59:36.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63c92dd640 1 -- 172.21.15.29:0/3683861143 learned_addr learned my addr 172.21.15.29:0/3683861143 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.293+0000 7f63c8adc640 1 --2- 172.21.15.29:0/3683861143 >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 0x7f63c4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.294+0000 7f63c92dd640 1 -- 172.21.15.29:0/3683861143 >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 msgr2=0x7f63c4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.294+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3683861143 >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 0x7f63c4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.294+0000 7f63c92dd640 1 -- 172.21.15.29:0/3683861143 >> v1:172.21.15.186:6789/0 conn(0x7f63c406a2e0 legacy=0x7f63c40cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.294+0000 7f63c92dd640 1 -- 172.21.15.29:0/3683861143 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c406b030 con 0x7f63c40addb0 2023-10-14T16:59:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.294+0000 7f63c8adc640 1 --2- 172.21.15.29:0/3683861143 >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 0x7f63c4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.295+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3683861143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40caa80 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f63c40cdbe0 tx=0x7f63c002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1072ae37f04d56b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.295+0000 7f63bbfff640 1 -- 172.21.15.29:0/3683861143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0031870 con 0x7f63c40addb0 2023-10-14T16:59:36.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.295+0000 7f63bbfff640 1 -- 172.21.15.29:0/3683861143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63c00319d0 con 0x7f63c40addb0 2023-10-14T16:59:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.295+0000 7f63bbfff640 1 -- 172.21.15.29:0/3683861143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63c0031d60 con 0x7f63c40addb0 2023-10-14T16:59:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 -- 172.21.15.29:0/3683861143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 msgr2=0x7f63c40caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 --2- 172.21.15.29:0/3683861143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40caa80 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 crypto rx=0x7f63c40cdbe0 tx=0x7f63c002b910 comp rx=0 tx=0).stop 2023-10-14T16:59:36.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 -- 172.21.15.29:0/3683861143 shutdown_connections 2023-10-14T16:59:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 --2- 172.21.15.29:0/3683861143 >> v2:172.21.15.186:3300/0 conn(0x7f63c40c9000 0x7f63c4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 --2- 172.21.15.29:0/3683861143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40caa80 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 -- 172.21.15.29:0/3683861143 >> 172.21.15.29:0/3683861143 conn(0x7f63c40ce9b0 msgr2=0x7f63c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 -- 172.21.15.29:0/3683861143 shutdown_connections 2023-10-14T16:59:36.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.296+0000 7f63cb541640 1 -- 172.21.15.29:0/3683861143 wait complete. 2023-10-14T16:59:36.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.298+0000 7f63cb541640 1 Processor -- start 2023-10-14T16:59:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.298+0000 7f63cb541640 1 -- start start 2023-10-14T16:59:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 0x7f63c41f7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c40c9000 0x7f63c40c4890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40c0eb0 con 0x7f63c40addb0 2023-10-14T16:59:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c41f7c90 con 0x7f63c406a2e0 2023-10-14T16:59:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63cb541640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40b1a60 con 0x7f63c40c9000 2023-10-14T16:59:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63c8adc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:36.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63c8adc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52768/0 (socket says 172.21.15.29:52768) 2023-10-14T16:59:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63c8adc640 1 -- 172.21.15.29:0/3316141120 learned_addr learned my addr 172.21.15.29:0/3316141120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.299+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 0x7f63c41f7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 -- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c40c9000 msgr2=0x7f63c40c4890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c40c9000 0x7f63c40c4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 -- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 msgr2=0x7f63c41f7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 0x7f63c41f7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 -- 172.21.15.29:0/3316141120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c00094d0 con 0x7f63c40addb0 2023-10-14T16:59:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 0x7f63c41f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63c8adc640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c7f0 tx=0x7f63b400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:36.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007820 con 0x7f63c40addb0 2023-10-14T16:59:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4007980 con 0x7f63c40addb0 2023-10-14T16:59:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b4007cf0 con 0x7f63c40addb0 2023-10-14T16:59:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c40b1d40 con 0x7f63c40addb0 2023-10-14T16:59:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.300+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c40b2300 con 0x7f63c40addb0 2023-10-14T16:59:36.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.302+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c404e160 con 0x7f63c40addb0 2023-10-14T16:59:36.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.303+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 57) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f63b40113f0 con 0x7f63c40addb0 2023-10-14T16:59:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.304+0000 7f63b9ffb640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 0x7f63a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.304+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f63b4089c20 con 0x7f63c40addb0 2023-10-14T16:59:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.304+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 0x7f63a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:36.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.305+0000 7f63c92dd640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 0x7f63a806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63c0036000 tx=0x7f63c003c040 comp rx=0 tx=0).ready entity=mgr.7797 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:36.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.308+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63b4054a70 con 0x7f63c40addb0 2023-10-14T16:59:36.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:36.558+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f63c40ad560 con 0x7f63c40addb0 2023-10-14T16:59:37.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.228+0000 7f63b9ffb640 1 -- 172.21.15.29:0/3316141120 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v58) v1 ==== 92+0+0 (secure 0 0 0) 0x7f63b401f0a0 con 0x7f63c40addb0 2023-10-14T16:59:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 msgr2=0x7f63a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 0x7f63a806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f63c0036000 tx=0x7f63c003c040 comp rx=0 tx=0).stop 2023-10-14T16:59:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 msgr2=0x7f63c40c4350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 crypto rx=0x7f63b400c7f0 tx=0x7f63b400ccb0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 shutdown_connections 2023-10-14T16:59:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f63a806af90 0x7f63a806d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c40c9000 0x7f63c40c4890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c40addb0 0x7f63c40c4350 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 --2- 172.21.15.29:0/3316141120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c406a2e0 0x7f63c41f7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.232+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 >> 172.21.15.29:0/3316141120 conn(0x7f63c40ce9b0 msgr2=0x7f63c40ae190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.233+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 shutdown_connections 2023-10-14T16:59:37.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.233+0000 7f63cb541640 1 -- 172.21.15.29:0/3316141120 wait complete. 2023-10-14T16:59:37.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T16:59:37.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.477+0000 7ff85c961640 1 Processor -- start 2023-10-14T16:59:37.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.477+0000 7ff85c961640 1 -- start start 2023-10-14T16:59:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.478+0000 7ff85c961640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff858159860 0x7ff858159c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.478+0000 7ff85c961640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8580cddb0 0x7ff858070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.478+0000 7ff85c961640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff85815dfa0 con 0x7ff8580cf370 2023-10-14T16:59:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.478+0000 7ff85c961640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85815a180 con 0x7ff858159860 2023-10-14T16:59:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.478+0000 7ff85c961640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff85815e280 con 0x7ff8580cddb0 2023-10-14T16:59:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff858159860 0x7ff858159c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff858159860 0x7ff858159c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52814/0 (socket says 172.21.15.29:52814) 2023-10-14T16:59:37.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 -- 172.21.15.29:0/3037156618 learned_addr learned my addr 172.21.15.29:0/3037156618 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 -- 172.21.15.29:0/3037156618 >> v2:172.21.15.186:3300/0 conn(0x7ff8580cddb0 msgr2=0x7ff858070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 --2- 172.21.15.29:0/3037156618 >> v2:172.21.15.186:3300/0 conn(0x7ff8580cddb0 0x7ff858070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 -- 172.21.15.29:0/3037156618 >> v1:172.21.15.186:6789/0 conn(0x7ff8580cf370 legacy=0x7ff85815d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.479+0000 7ff85659c640 1 -- 172.21.15.29:0/3037156618 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85815e560 con 0x7ff858159860 2023-10-14T16:59:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.480+0000 7ff85659c640 1 --2- 172.21.15.29:0/3037156618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff858159860 0x7ff858159c40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff8580c6a40 tx=0x7ff84002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc674c989211e1cc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.480+0000 7ff855d9b640 1 -- 172.21.15.29:0/3037156618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840031870 con 0x7ff858159860 2023-10-14T16:59:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.480+0000 7ff855d9b640 1 -- 172.21.15.29:0/3037156618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8400319d0 con 0x7ff858159860 2023-10-14T16:59:37.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.480+0000 7ff855d9b640 1 -- 172.21.15.29:0/3037156618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840031d60 con 0x7ff858159860 2023-10-14T16:59:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 -- 172.21.15.29:0/3037156618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff858159860 msgr2=0x7ff858159c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 --2- 172.21.15.29:0/3037156618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff858159860 0x7ff858159c40 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 crypto rx=0x7ff8580c6a40 tx=0x7ff84002b910 comp rx=0 tx=0).stop 2023-10-14T16:59:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 -- 172.21.15.29:0/3037156618 shutdown_connections 2023-10-14T16:59:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 --2- 172.21.15.29:0/3037156618 >> v2:172.21.15.186:3300/0 conn(0x7ff8580cddb0 0x7ff858070660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 --2- 172.21.15.29:0/3037156618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff858159860 0x7ff858159c40 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.481+0000 7ff85c961640 1 -- 172.21.15.29:0/3037156618 >> 172.21.15.29:0/3037156618 conn(0x7ff8580cc460 msgr2=0x7ff8580cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.482+0000 7ff85c961640 1 -- 172.21.15.29:0/3037156618 shutdown_connections 2023-10-14T16:59:37.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.482+0000 7ff85c961640 1 -- 172.21.15.29:0/3037156618 wait complete. 2023-10-14T16:59:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.483+0000 7ff85c961640 1 Processor -- start 2023-10-14T16:59:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.483+0000 7ff85c961640 1 -- start start 2023-10-14T16:59:37.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8580cddb0 0x7ff8580c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 0x7ff8581f4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff858145600 con 0x7ff8580cf370 2023-10-14T16:59:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85815a620 con 0x7ff858159860 2023-10-14T16:59:37.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.484+0000 7ff85c961640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8580c59d0 con 0x7ff8580cddb0 2023-10-14T16:59:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52826/0 (socket says 172.21.15.29:52826) 2023-10-14T16:59:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 -- 172.21.15.29:0/788202777 learned_addr learned my addr 172.21.15.29:0/788202777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff856d9d640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 0x7ff8581f4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff85659c640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8580cddb0 0x7ff8580c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8580cddb0 msgr2=0x7ff8580c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8580cddb0 0x7ff8580c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 msgr2=0x7ff8581f4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 0x7ff8581f4bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff84dd9b640 1 -- 172.21.15.29:0/788202777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8400094d0 con 0x7ff8580cf370 2023-10-14T16:59:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.485+0000 7ff856d9d640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 0x7ff8581f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T16:59:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff84dd9b640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c400 tx=0x7ff84800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8480035f0 con 0x7ff8580cf370 2023-10-14T16:59:37.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff848003750 con 0x7ff8580cf370 2023-10-14T16:59:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8580c5cb0 con 0x7ff8580cf370 2023-10-14T16:59:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff848003b40 con 0x7ff8580cf370 2023-10-14T16:59:37.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.486+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8580c6270 con 0x7ff8580cf370 2023-10-14T16:59:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.488+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8580c9e60 con 0x7ff8580cf370 2023-10-14T16:59:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.489+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 58) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff848003ca0 con 0x7ff8580cf370 2023-10-14T16:59:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.489+0000 7ff84f7fe640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 0x7ff83806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.490+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff848088e50 con 0x7ff8580cf370 2023-10-14T16:59:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.490+0000 7ff85659c640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 msgr2=0x7ff83806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.490+0000 7ff85659c640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 0x7ff83806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:37.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.494+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff848053dd0 con 0x7ff8580cf370 2023-10-14T16:59:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.691+0000 7ff85659c640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 msgr2=0x7ff83806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:37.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.691+0000 7ff85659c640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 0x7ff83806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:37.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.706+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff85804e160 con 0x7ff8580cf370 2023-10-14T16:59:37.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.711+0000 7ff84f7fe640 1 -- 172.21.15.29:0/788202777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v41)=0 v41) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff848017020 con 0x7ff8580cf370 2023-10-14T16:59:37.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 msgr2=0x7ff83806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:37.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 0x7ff83806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 msgr2=0x7ff8580c4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 secure :-1 s=READY pgs=1428 cs=0 l=1 rev1=1 crypto rx=0x7ff84800c400 tx=0x7ff84800c8c0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 shutdown_connections 2023-10-14T16:59:37.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff83806afe0 0x7ff83806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff858159860 0x7ff8581f4bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8580cf370 0x7ff8580c4ee0 unknown :-1 s=CLOSED pgs=1428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 --2- 172.21.15.29:0/788202777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8580cddb0 0x7ff8580c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 >> 172.21.15.29:0/788202777 conn(0x7ff8580cc460 msgr2=0x7ff8580cc840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:37.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 shutdown_connections 2023-10-14T16:59:37.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.714+0000 7ff85c961640 1 -- 172.21.15.29:0/788202777 wait complete. 2023-10-14T16:59:37.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T16:59:37.748 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:37.748+0000 7f0150033040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:59:37.787 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:37.785+0000 7f7c7762b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T16:59:37.803 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:37.802+0000 7f0150033040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:59:37.846 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:37.843+0000 7f7c7762b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T16:59:37.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.944+0000 7ff066809640 1 Processor -- start 2023-10-14T16:59:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.944+0000 7ff066809640 1 -- start start 2023-10-14T16:59:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600cf710 0x7ff0600b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0600cfaf0 0x7ff0600b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff0600b38e0 0x7ff0600d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0600d3c60 con 0x7ff0600cfaf0 2023-10-14T16:59:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0600d3980 con 0x7ff0600b38e0 2023-10-14T16:59:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.945+0000 7ff066809640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600ce630 con 0x7ff0600cf710 2023-10-14T16:59:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0600cfaf0 0x7ff0600b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0600cfaf0 0x7ff0600b3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52832/0 (socket says 172.21.15.29:52832) 2023-10-14T16:59:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 -- 172.21.15.29:0/3605786798 learned_addr learned my addr 172.21.15.29:0/3605786798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05ffff640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600cf710 0x7ff0600b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 -- 172.21.15.29:0/3605786798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600cf710 msgr2=0x7ff0600b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600cf710 0x7ff0600b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff064da6640 1 --2- 172.21.15.29:0/3605786798 >> v2:172.21.15.186:3300/0 conn(0x7ff0600b38e0 0x7ff0600d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 -- 172.21.15.29:0/3605786798 >> v2:172.21.15.186:3300/0 conn(0x7ff0600b38e0 msgr2=0x7ff0600d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/3605786798 >> v2:172.21.15.186:3300/0 conn(0x7ff0600b38e0 0x7ff0600d0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.946+0000 7ff05f7fe640 1 -- 172.21.15.29:0/3605786798 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0600ce910 con 0x7ff0600cfaf0 2023-10-14T16:59:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.948+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cfaf0 0x7ff0600b3250 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ff06006cd60 tx=0x7ff05402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=788cc269602df2e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.948+0000 7ff05effd640 1 -- 172.21.15.29:0/3605786798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0540317e0 con 0x7ff0600cfaf0 2023-10-14T16:59:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.948+0000 7ff05effd640 1 -- 172.21.15.29:0/3605786798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054031940 con 0x7ff0600cfaf0 2023-10-14T16:59:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.948+0000 7ff05effd640 1 -- 172.21.15.29:0/3605786798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054031cd0 con 0x7ff0600cfaf0 2023-10-14T16:59:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 -- 172.21.15.29:0/3605786798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cfaf0 msgr2=0x7ff0600b3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cfaf0 0x7ff0600b3250 secure :-1 s=READY pgs=1429 cs=0 l=1 rev1=1 crypto rx=0x7ff06006cd60 tx=0x7ff05402b880 comp rx=0 tx=0).stop 2023-10-14T16:59:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 -- 172.21.15.29:0/3605786798 shutdown_connections 2023-10-14T16:59:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 --2- 172.21.15.29:0/3605786798 >> v2:172.21.15.186:3300/0 conn(0x7ff0600b38e0 0x7ff0600d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cfaf0 0x7ff0600b3250 unknown :-1 s=CLOSED pgs=1429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 --2- 172.21.15.29:0/3605786798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600cf710 0x7ff0600b2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 -- 172.21.15.29:0/3605786798 >> 172.21.15.29:0/3605786798 conn(0x7ff0600d2cc0 msgr2=0x7ff0600d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 -- 172.21.15.29:0/3605786798 shutdown_connections 2023-10-14T16:59:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.949+0000 7ff066809640 1 -- 172.21.15.29:0/3605786798 wait complete. 2023-10-14T16:59:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.951+0000 7ff066809640 1 Processor -- start 2023-10-14T16:59:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.951+0000 7ff066809640 1 -- start start 2023-10-14T16:59:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600b38e0 0x7ff0600b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 0x7ff0601543a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600b2940 con 0x7ff0600cf710 2023-10-14T16:59:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff06009cb40 con 0x7ff0600cfaf0 2023-10-14T16:59:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff066809640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0600b20a0 con 0x7ff0600b38e0 2023-10-14T16:59:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff05f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff05f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52842/0 (socket says 172.21.15.29:52842) 2023-10-14T16:59:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff05f7fe640 1 -- 172.21.15.29:0/2011467435 learned_addr learned my addr 172.21.15.29:0/2011467435 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff05ffff640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600b38e0 0x7ff0600b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.952+0000 7ff064da6640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 0x7ff0601543a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600b38e0 msgr2=0x7ff0600b1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600b38e0 0x7ff0600b1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 msgr2=0x7ff0601543a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 0x7ff0601543a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 -- 172.21.15.29:0/2011467435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0540094d0 con 0x7ff0600cf710 2023-10-14T16:59:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff064da6640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 0x7ff0601543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05f7fe640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7ff05402b8c0 tx=0x7ff0540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:37.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054032880 con 0x7ff0600cf710 2023-10-14T16:59:37.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff054032a00 con 0x7ff0600cf710 2023-10-14T16:59:37.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0600b2380 con 0x7ff0600cf710 2023-10-14T16:59:37.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.953+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff054032db0 con 0x7ff0600cf710 2023-10-14T16:59:37.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.954+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff06014e530 con 0x7ff0600cf710 2023-10-14T16:59:37.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.955+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06009c9a0 con 0x7ff0600cf710 2023-10-14T16:59:37.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.955+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 58) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff0540453f0 con 0x7ff0600cf710 2023-10-14T16:59:37.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.956+0000 7ff05cff9640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 0x7ff04c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:37.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.956+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7ff0540abc70 con 0x7ff0600cf710 2023-10-14T16:59:37.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.956+0000 7ff05ffff640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 msgr2=0x7ff04c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:37.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.957+0000 7ff05ffff640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 0x7ff04c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:37.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:37.960+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff06009c9a0 con 0x7ff0600cf710 2023-10-14T16:59:38.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.153+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff06009c9a0 con 0x7ff0600cf710 2023-10-14T16:59:38.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.158+0000 7ff05ffff640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 msgr2=0x7ff04c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:38.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.158+0000 7ff05ffff640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 0x7ff04c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:38.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.162+0000 7ff05cff9640 1 -- 172.21.15.29:0/2011467435 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v42)=0 v42) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff06009c9a0 con 0x7ff0600cf710 2023-10-14T16:59:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 msgr2=0x7ff04c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:38.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 0x7ff04c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 msgr2=0x7ff060153e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 crypto rx=0x7ff05402b8c0 tx=0x7ff0540046c0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 shutdown_connections 2023-10-14T16:59:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7ff04c06afe0 0x7ff04c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0600cfaf0 0x7ff0601543a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0600cf710 0x7ff060153e60 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 --2- 172.21.15.29:0/2011467435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0600b38e0 0x7ff0600b1690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.164+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 >> 172.21.15.29:0/2011467435 conn(0x7ff0600d2cc0 msgr2=0x7ff0600d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.165+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 shutdown_connections 2023-10-14T16:59:38.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.165+0000 7ff066809640 1 -- 172.21.15.29:0/2011467435 wait complete. 2023-10-14T16:59:38.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:38.183 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T16:59:38.238 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-14T16:59:38.262 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.261+0000 7f0150033040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:59:38.316 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.315+0000 7f0150033040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:59:38.319 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.318+0000 7f7c7762b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T16:59:38.367 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.366+0000 7f0150033040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:59:38.374 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.373+0000 7f7c7762b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T16:59:38.427 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.426+0000 7f7c7762b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T16:59:38.429 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.428+0000 7f0150033040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:59:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.449+0000 7f74f0e59640 1 Processor -- start 2023-10-14T16:59:38.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.449+0000 7f74f0e59640 1 -- start start 2023-10-14T16:59:38.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.451+0000 7f74f0e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74ec0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.451+0000 7f74f0e59640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ec15d150 con 0x7f74ec0b8480 2023-10-14T16:59:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.451+0000 7f74f0e59640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ec0d1180 con 0x7f74ec15cd70 2023-10-14T16:59:38.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.451+0000 7f74f0e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0d1460 con 0x7f74ec0cded0 2023-10-14T16:59:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74e3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f74ec0b8480 0x7f74ec0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47864/0 (socket says 172.21.15.29:47864) 2023-10-14T16:59:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74e3fff640 1 -- 172.21.15.29:0/3514793081 learned_addr learned my addr 172.21.15.29:0/3514793081 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74ebfff640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74ec0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687221119 0 0) 0x7f74ec15d150 con 0x7f74ec0b8480 2023-10-14T16:59:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d8003520 con 0x7f74ec0b8480 2023-10-14T16:59:38.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2494174835 0 0) 0x7f74d8003520 con 0x7f74ec0b8480 2023-10-14T16:59:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 msgr2=0x7f74ec0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74ec0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 >> v1:172.21.15.186:6789/0 conn(0x7f74ec15cd70 legacy=0x7f74ec0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec0d1740 con 0x7f74ec0b8480 2023-10-14T16:59:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.452+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f74d4001690 con 0x7f74ec0b8480 2023-10-14T16:59:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 >> v1:172.21.15.29:6789/0 conn(0x7f74ec0b8480 legacy=0x7f74ec0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d80039a0 0x7f74d8003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80047c0 0x7f74d8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74d8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec0d1460 con 0x7f74d80039a0 2023-10-14T16:59:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d800ba60 con 0x7f74d80047c0 2023-10-14T16:59:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d800bc20 con 0x7f74ec0cded0 2023-10-14T16:59:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d80039a0 0x7f74d8003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74e3fff640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80047c0 0x7f74d8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74ebfff640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74d8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 msgr2=0x7f74d8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec0cded0 0x7f74d8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80047c0 msgr2=0x7f74d8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80047c0 0x7f74d8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.453+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3514793081 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d8003520 con 0x7f74d80039a0 2023-10-14T16:59:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.454+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d80039a0 0x7f74d8003dc0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f74dc001750 tx=0x7f74d80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aba3c848ecd7fd5a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc0052d0 con 0x7f74d80039a0 2023-10-14T16:59:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74dc02e050 con 0x7f74d80039a0 2023-10-14T16:59:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74eaffd640 1 -- 172.21.15.29:0/3514793081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74dc0035f0 con 0x7f74d80039a0 2023-10-14T16:59:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74f0e59640 1 -- 172.21.15.29:0/3514793081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d80039a0 msgr2=0x7f74d8003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74f0e59640 1 --2- 172.21.15.29:0/3514793081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74d80039a0 0x7f74d8003dc0 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 crypto rx=0x7f74dc001750 tx=0x7f74d80057f0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74ebfff640 1 -- 172.21.15.29:0/3514793081 reap_dead start 2023-10-14T16:59:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74f0e59640 1 -- 172.21.15.29:0/3514793081 shutdown_connections 2023-10-14T16:59:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.455+0000 7f74f0e59640 1 -- 172.21.15.29:0/3514793081 >> 172.21.15.29:0/3514793081 conn(0x7f74ec0cf570 msgr2=0x7f74ec0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.456+0000 7f74f0e59640 1 -- 172.21.15.29:0/3514793081 shutdown_connections 2023-10-14T16:59:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.456+0000 7f74f0e59640 1 -- 172.21.15.29:0/3514793081 wait complete. 2023-10-14T16:59:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.457+0000 7f74f0e59640 1 Processor -- start 2023-10-14T16:59:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.457+0000 7f74f0e59640 1 -- start start 2023-10-14T16:59:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80039a0 0x7f74ec1faa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec070b70 0x7f74ec15f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 0x7f74ec15f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec15b120 con 0x7f74ec070ff0 2023-10-14T16:59:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec1601a0 con 0x7f74d80039a0 2023-10-14T16:59:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.458+0000 7f74f0e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ec15fec0 con 0x7f74ec070b70 2023-10-14T16:59:38.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 0x7f74ec15f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 0x7f74ec15f7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52864/0 (socket says 172.21.15.29:52864) 2023-10-14T16:59:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 -- 172.21.15.29:0/3998239396 learned_addr learned my addr 172.21.15.29:0/3998239396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80039a0 0x7f74ec1faa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74e3fff640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec070b70 0x7f74ec15f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:38.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec070b70 msgr2=0x7f74ec15f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec070b70 0x7f74ec15f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80039a0 msgr2=0x7f74ec1faa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74d80039a0 0x7f74ec1faa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74ebfff640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ec1f9fd0 con 0x7f74ec070ff0 2023-10-14T16:59:38.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.459+0000 7f74e3fff640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ec070b70 0x7f74ec15f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T16:59:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74ebfff640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 0x7f74ec15f7d0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0cd0f0 tx=0x7f74e400f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4003990 con 0x7f74ec070ff0 2023-10-14T16:59:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e4011040 con 0x7f74ec070ff0 2023-10-14T16:59:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e4003d50 con 0x7f74ec070ff0 2023-10-14T16:59:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ec1fa2b0 con 0x7f74ec070ff0 2023-10-14T16:59:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.460+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ec158b10 con 0x7f74ec070ff0 2023-10-14T16:59:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.461+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ec04e9b0 con 0x7f74ec070ff0 2023-10-14T16:59:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.462+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 58) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f74e4003eb0 con 0x7f74ec070ff0 2023-10-14T16:59:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.464+0000 7f74e8ff9640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.464+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.464+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5753+0+0 (secure 0 0 0) 0x7f74e4089d60 con 0x7f74ec070ff0 2023-10-14T16:59:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.464+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T16:59:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.468+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f74e4054bb0 con 0x7f74ec070ff0 2023-10-14T16:59:38.480 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.479+0000 7f0150033040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:59:38.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.492+0000 7f7c7762b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T16:59:38.545 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.543+0000 7f7c7762b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T16:59:38.592 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.591+0000 7f0150033040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:59:38.655 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.654+0000 7f0150033040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:38.660 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.659+0000 7f7c7762b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T16:59:38.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.665+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:38.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.665+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T16:59:38.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:38.686+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f74ec0d1d00 con 0x7f74cc06b010 2023-10-14T16:59:38.705 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.704+0000 7f0150033040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:59:38.726 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.724+0000 7f7c7762b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:38.763 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.763+0000 7f0150033040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:59:38.777 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.775+0000 7f7c7762b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T16:59:38.815 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:38.814+0000 7f0150033040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:59:38.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.834+0000 7f7c7762b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T16:59:38.888 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:38.886+0000 7f7c7762b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T16:59:39.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:39.066+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:39.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:39.066+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T16:59:39.201 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.200+0000 7f0150033040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:59:39.282 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.280+0000 7f7c7762b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T16:59:39.361 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.361+0000 7f0150033040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:59:39.450 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.448+0000 7f7c7762b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T16:59:39.472 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.471+0000 7f0150033040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:59:39.523 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.522+0000 7f0150033040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:59:39.562 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.560+0000 7f7c7762b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T16:59:39.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.612+0000 7f7c7762b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T16:59:39.632 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.631+0000 7f0150033040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:39.725 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.724+0000 7f7c7762b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:39.740 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.739+0000 7f0150033040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:59:39.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.834+0000 7f7c7762b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T16:59:39.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:39.867+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:39.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:39.867+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T16:59:39.898 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.897+0000 7f0150033040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:39.949 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:39.948+0000 7f0150033040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:39.999 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:39.997+0000 7f7c7762b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T16:59:40.049 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:40.048+0000 7f7c7762b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T16:59:40.070 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:40.068+0000 7f0150033040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:59:40.173 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:40.171+0000 7f7c7762b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T16:59:40.904 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:40.903+0000 7f0150033040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:59:41.049 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.048+0000 7f7c7762b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T16:59:41.204 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:41.203+0000 7f0150033040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:59:41.262 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:41.261+0000 7f0150033040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:59:41.357 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.355+0000 7f7c7762b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T16:59:41.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T16:59:41.376+0000 7f0150033040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:59:41.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.415+0000 7f7c7762b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T16:59:41.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.446+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 59) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f74e4054f80 con 0x7f74ec070ff0 2023-10-14T16:59:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.469+0000 7f74eb7fe640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2416849365 2023-10-14T16:59:41.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.469+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T16:59:41.539 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.538+0000 7f7c7762b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T16:59:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.557+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 60) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f74e4087b20 con 0x7f74ec070ff0 2023-10-14T16:59:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.557+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 msgr2=0x7f74cc06d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T16:59:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:41.557+0000 7f74e8ff9640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/2416849365,v1:172.21.15.29:6833/2416849365] conn(0x7f74cc06b010 0x7f74cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:41.879 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.877+0000 7f7bf3990640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.879 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.877+0000 7f7bf3990640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.880 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.877+0000 7f7bf3990640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.880 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.877+0000 7f7bf3990640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.880 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.877+0000 7f7bf3990640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.878+0000 7f7bf69d6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.878+0000 7f7bf69d6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.878+0000 7f7bf69d6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.878+0000 7f7bf69d6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:41.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T16:59:41.878+0000 7f7bf69d6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T16:59:42.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.565+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 61) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f74e4055850 con 0x7f74ec070ff0 2023-10-14T16:59:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.566+0000 7f74e8ff9640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f74cc01d1f0 0x7f74cc074f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:42.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.566+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 --> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f74cc00b680 con 0x7f74cc01d1f0 2023-10-14T16:59:42.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.576+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f74cc01d1f0 0x7f74cc074f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:42.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.584+0000 7f74eb7fe640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f74cc01d1f0 0x7f74cc074f60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74dc033000 tx=0x7f74dc009030 comp rx=0 tx=0).ready entity=mgr.8199 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:42.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.774+0000 7f74e8ff9640 1 -- 172.21.15.29:0/3998239396 <== mgr.8199 v2:172.21.15.29:6832/1622969171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f74cc00b680 con 0x7f74cc01d1f0 2023-10-14T16:59:42.777 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T16:59:42.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f74cc01d1f0 msgr2=0x7f74cc074f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f74cc01d1f0 0x7f74cc074f60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f74dc033000 tx=0x7f74dc009030 comp rx=0 tx=0).stop 2023-10-14T16:59:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 msgr2=0x7f74ec15f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 --2- 172.21.15.29:0/3998239396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ec070ff0 0x7f74ec15f7d0 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 crypto rx=0x7f74ec0cd0f0 tx=0x7f74e400f4a0 comp rx=0 tx=0).stop 2023-10-14T16:59:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74ebfff640 1 -- 172.21.15.29:0/3998239396 reap_dead start 2023-10-14T16:59:42.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 shutdown_connections 2023-10-14T16:59:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 >> 172.21.15.29:0/3998239396 conn(0x7f74ec0cf570 msgr2=0x7f74ec0d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 shutdown_connections 2023-10-14T16:59:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:42.778+0000 7f74f0e59640 1 -- 172.21.15.29:0/3998239396 wait complete. 2023-10-14T16:59:42.801 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T16:59:42.801 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-14-17_00_00 in ~18s... 2023-10-14T16:59:42.802 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-14T16:59:42.802 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-14T16:59:43.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.022+0000 7ffb519a9640 1 Processor -- start 2023-10-14T16:59:43.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.022+0000 7ffb519a9640 1 -- start start 2023-10-14T16:59:43.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.023+0000 7ffb519a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffb4c15ad70 0x7ffb4c069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.023+0000 7ffb519a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffb4c0c9ca0 0x7ffb4c06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb519a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb4c0cb030 con 0x7ffb4c0ca3b0 2023-10-14T16:59:43.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb519a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c91c0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb519a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c94a0 con 0x7ffb4c0c9ca0 2023-10-14T16:59:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb4b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffb4c0ca3b0 0x7ffb4c0ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58428/0 (socket says 172.21.15.29:58428) 2023-10-14T16:59:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb4b7fe640 1 -- 172.21.15.29:0/2401012254 learned_addr learned my addr 172.21.15.29:0/2401012254 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T16:59:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb4affd640 1 --2- 172.21.15.29:0/2401012254 >> v2:172.21.15.29:3300/0 conn(0x7ffb4c15ad70 0x7ffb4c069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.024+0000 7ffb4a7fc640 1 --2- 172.21.15.29:0/2401012254 >> v2:172.21.15.186:3300/0 conn(0x7ffb4c0c9ca0 0x7ffb4c06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.025+0000 7ffb4affd640 1 -- 172.21.15.29:0/2401012254 >> v2:172.21.15.186:3300/0 conn(0x7ffb4c0c9ca0 msgr2=0x7ffb4c06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T16:59:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.025+0000 7ffb4affd640 1 --2- 172.21.15.29:0/2401012254 >> v2:172.21.15.186:3300/0 conn(0x7ffb4c0c9ca0 0x7ffb4c06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.025+0000 7ffb4affd640 1 -- 172.21.15.29:0/2401012254 >> v1:172.21.15.29:6789/0 conn(0x7ffb4c0ca3b0 legacy=0x7ffb4c0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.025+0000 7ffb4affd640 1 -- 172.21.15.29:0/2401012254 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb4c0c9780 con 0x7ffb4c15ad70 2023-10-14T16:59:43.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.026+0000 7ffb4affd640 1 --2- 172.21.15.29:0/2401012254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c069da0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0ce870 tx=0x7ffb3c02bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=622595401a3e916b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.026+0000 7ffb49ffb640 1 -- 172.21.15.29:0/2401012254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c02cc70 con 0x7ffb4c15ad70 2023-10-14T16:59:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.026+0000 7ffb49ffb640 1 -- 172.21.15.29:0/2401012254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb3c02cdd0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.026+0000 7ffb49ffb640 1 -- 172.21.15.29:0/2401012254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb3c034660 con 0x7ffb4c15ad70 2023-10-14T16:59:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 -- 172.21.15.29:0/2401012254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 msgr2=0x7ffb4c069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 --2- 172.21.15.29:0/2401012254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c069da0 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 crypto rx=0x7ffb4c0ce870 tx=0x7ffb3c02bec0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 -- 172.21.15.29:0/2401012254 shutdown_connections 2023-10-14T16:59:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 --2- 172.21.15.29:0/2401012254 >> v2:172.21.15.186:3300/0 conn(0x7ffb4c0c9ca0 0x7ffb4c06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 --2- 172.21.15.29:0/2401012254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c069da0 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 -- 172.21.15.29:0/2401012254 >> 172.21.15.29:0/2401012254 conn(0x7ffb4c0ce9b0 msgr2=0x7ffb4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:43.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 -- 172.21.15.29:0/2401012254 shutdown_connections 2023-10-14T16:59:43.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.027+0000 7ffb519a9640 1 -- 172.21.15.29:0/2401012254 wait complete. 2023-10-14T16:59:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.029+0000 7ffb519a9640 1 Processor -- start 2023-10-14T16:59:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.029+0000 7ffb519a9640 1 -- start start 2023-10-14T16:59:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0c9ca0 0x7ffb4c1f1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ca3b0 0x7ffb4c0c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c1f24e0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c154980 con 0x7ffb4c0c9ca0 2023-10-14T16:59:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb519a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb4c0c6520 con 0x7ffb4c0ca3b0 2023-10-14T16:59:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.030+0000 7ffb4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41940/0 (socket says 172.21.15.29:41940) 2023-10-14T16:59:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 -- 172.21.15.29:0/3769846706 learned_addr learned my addr 172.21.15.29:0/3769846706 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T16:59:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4a7fc640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ca3b0 0x7ffb4c0c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4affd640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0c9ca0 0x7ffb4c1f1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 -- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ca3b0 msgr2=0x7ffb4c0c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ca3b0 0x7ffb4c0c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 -- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0c9ca0 msgr2=0x7ffb4c1f1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0c9ca0 0x7ffb4c1f1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 -- 172.21.15.29:0/3769846706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3c009a70 con 0x7ffb4c15ad70 2023-10-14T16:59:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.031+0000 7ffb4b7fe640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c420 tx=0x7ffb4400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:43.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.032+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb44005ce0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.032+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb44005e40 con 0x7ffb4c15ad70 2023-10-14T16:59:43.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.032+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb4c0c6800 con 0x7ffb4c15ad70 2023-10-14T16:59:43.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.032+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb440072d0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.032+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb4c15b4f0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.033+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb4c04e160 con 0x7ffb4c15ad70 2023-10-14T16:59:43.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.034+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 61) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ffb44007430 con 0x7ffb4c15ad70 2023-10-14T16:59:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.035+0000 7ffb33fff640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 0x7ffb3806d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T16:59:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.035+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7ffb44088960 con 0x7ffb4c15ad70 2023-10-14T16:59:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.035+0000 7ffb4affd640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 0x7ffb3806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T16:59:43.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.036+0000 7ffb4affd640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 0x7ffb3806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c036000 tx=0x7ffb3c030000 comp rx=0 tx=0).ready entity=mgr.8199 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T16:59:43.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.040+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb44053ba0 con 0x7ffb4c15ad70 2023-10-14T16:59:43.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.256+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 --> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb4c0ad4d0 con 0x7ffb3806ad40 2023-10-14T16:59:43.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.270+0000 7ffb33fff640 1 -- 172.21.15.29:0/3769846706 <== mgr.8199 v2:172.21.15.29:6832/1622969171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+66 (secure 0 0 0) 0x7ffb3c02ca70 con 0x7ffb3806ad40 2023-10-14T16:59:43.272 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T16:59:43.272 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-14T16:59:43.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 msgr2=0x7ffb3806d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 0x7ffb3806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ffb3c036000 tx=0x7ffb3c030000 comp rx=0 tx=0).stop 2023-10-14T16:59:43.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 msgr2=0x7ffb4c0ae970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 crypto rx=0x7ffb4400c420 tx=0x7ffb4400c8e0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 shutdown_connections 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7ffb3806ad40 0x7ffb3806d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb4c15ad70 0x7ffb4c0ae970 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb4c0ca3b0 0x7ffb4c0c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 --2- 172.21.15.29:0/3769846706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb4c0c9ca0 0x7ffb4c1f1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T16:59:43.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 >> 172.21.15.29:0/3769846706 conn(0x7ffb4c0ce9b0 msgr2=0x7ffb4c159f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T16:59:43.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.273+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 shutdown_connections 2023-10-14T16:59:43.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T16:59:43.274+0000 7ffb519a9640 1 -- 172.21.15.29:0/3769846706 wait complete. 2023-10-14T16:59:43.293 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{}]} 2023-10-14T16:59:43.293 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{}]} 2023-10-14T16:59:43.294 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:43.294 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:43.347 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:43.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:43.347 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:44.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:44.349 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:44.417 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:44.418 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:44.418 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:45.419 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:45.420 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:45.487 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:45.488 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:45.488 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:46.489 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:46.490 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:46.555 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:46.556 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:46.556 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:46.592 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:59:46.601 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T16:59:47.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:47.557 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:47.620 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:47.621 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:47.621 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:48.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:48.623 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:48.690 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:48.691 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:48.691 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:49.692 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:49.693 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:49.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:49.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:49.762 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:50.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:50.764 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:50.829 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:50.830 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:50.830 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:51.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:51.832 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:51.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:51.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:51.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:52.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:52.902 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:52.969 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:52.970 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:52.970 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:53.971 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:53.972 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:54.038 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:54.039 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:54.039 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:55.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:55.041 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:55.108 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:55.108 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:55.109 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:56.109 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:56.110 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:56.177 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:56.177 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:56.178 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:57.179 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:57.179 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:57.247 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:57.247 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:57.248 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:58.249 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:58.249 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:58.315 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:58.315 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:58.315 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T16:59:59.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T16:59:59.317 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T16:59:59.384 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T16:59:59.385 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T16:59:59.385 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T17:00:00.386 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:00.387 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T17:00:00.451 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T17:00:00.451 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T17:00:00.452 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T17:00:00.802 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:00.802+0000 7f7be6175640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:00.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:00.802+0000 7f7be6175640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:00.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:00.802+0000 7f7be6175640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:00.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:00.802+0000 7f7be6175640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:00.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:00.802+0000 7f7be6175640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:01.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:01.454 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T17:00:01.520 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_00_00_UTC 2023-10-14T17:00:01.522 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-17_00_00_UTC'} 2023-10-14T17:00:01.523 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-14-17_00_00_UTC'} 2023-10-14T17:00:01.523 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T17:00:01.524 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-14-17_00_00_UTC'] 2023-10-14T17:00:01.524 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697302782.8018987, now = 1697302801.524581, timo = 18 2023-10-14T17:00:02.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-14T17:00:02.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.772+0000 7f2a63fff640 1 Processor -- start 2023-10-14T17:00:02.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.772+0000 7f2a63fff640 1 -- start start 2023-10-14T17:00:02.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a640709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a63fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a64160a90 con 0x7f2a6415c9f0 2023-10-14T17:00:02.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a63fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a640cd900 con 0x7f2a64070f20 2023-10-14T17:00:02.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a63fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640cdec0 con 0x7f2a6415dde0 2023-10-14T17:00:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a62ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2a6415c9f0 0x7f2a640c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42426/0 (socket says 172.21.15.29:42426) 2023-10-14T17:00:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.774+0000 7f2a62ffd640 1 -- 172.21.15.29:0/1132178470 learned_addr learned my addr 172.21.15.29:0/1132178470 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:02.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a5a7fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a640709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2649578040 0 0) 0x7f2a64160a90 con 0x7f2a6415c9f0 2023-10-14T17:00:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a50003520 con 0x7f2a6415c9f0 2023-10-14T17:00:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3931907445 0 0) 0x7f2a50003520 con 0x7f2a6415c9f0 2023-10-14T17:00:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 msgr2=0x7f2a640709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:02.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a640709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 >> v1:172.21.15.186:6789/0 conn(0x7f2a64070f20 legacy=0x7f2a640ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a640cdbe0 con 0x7f2a6415c9f0 2023-10-14T17:00:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2a540019f0 con 0x7f2a6415c9f0 2023-10-14T17:00:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 >> v1:172.21.15.29:6789/0 conn(0x7f2a6415c9f0 legacy=0x7f2a640c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:02.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a500039a0 0x7f2a50003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a500047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a50004d00 0x7f2a50005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5000ba30 con 0x7f2a500039a0 2023-10-14T17:00:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5000bba0 con 0x7f2a50004d00 2023-10-14T17:00:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.775+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a5000bd60 con 0x7f2a6415dde0 2023-10-14T17:00:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a500039a0 0x7f2a50003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a5a7fc640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a500047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 -- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 msgr2=0x7f2a500047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a6415dde0 0x7f2a500047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 -- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a50004d00 msgr2=0x7f2a50005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a637fe640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a50004d00 0x7f2a50005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a50004d00 0x7f2a50005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a62ffd640 1 -- 172.21.15.29:0/1132178470 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a50003520 con 0x7f2a500039a0 2023-10-14T17:00:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.776+0000 7f2a637fe640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a50004d00 0x7f2a50005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.777+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a500039a0 0x7f2a50003dc0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f2a54002230 tx=0x7f2a500057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=125afe0ede4f71d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:02.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54034630 con 0x7f2a500039a0 2023-10-14T17:00:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a5402f070 con 0x7f2a500039a0 2023-10-14T17:00:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a627fc640 1 -- 172.21.15.29:0/1132178470 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a54034a10 con 0x7f2a500039a0 2023-10-14T17:00:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 -- 172.21.15.29:0/1132178470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a500039a0 msgr2=0x7f2a50003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 --2- 172.21.15.29:0/1132178470 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a500039a0 0x7f2a50003dc0 secure :-1 s=READY pgs=1455 cs=0 l=1 rev1=1 crypto rx=0x7f2a54002230 tx=0x7f2a500057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a637fe640 1 -- 172.21.15.29:0/1132178470 reap_dead start 2023-10-14T17:00:02.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 -- 172.21.15.29:0/1132178470 shutdown_connections 2023-10-14T17:00:02.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 -- 172.21.15.29:0/1132178470 >> 172.21.15.29:0/1132178470 conn(0x7f2a640ce6a0 msgr2=0x7f2a640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:02.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 -- 172.21.15.29:0/1132178470 shutdown_connections 2023-10-14T17:00:02.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.778+0000 7f2a63fff640 1 -- 172.21.15.29:0/1132178470 wait complete. 2023-10-14T17:00:02.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.780+0000 7f2a63fff640 1 Processor -- start 2023-10-14T17:00:02.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.780+0000 7f2a63fff640 1 -- start start 2023-10-14T17:00:02.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a500039a0 0x7f2a64150af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a640af030 0x7f2a640af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a640afa20 con 0x7f2a641f6a20 2023-10-14T17:00:02.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a64151030 con 0x7f2a500039a0 2023-10-14T17:00:02.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.781+0000 7f2a63fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a641f3970 con 0x7f2a640af030 2023-10-14T17:00:02.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36542/0 (socket says 172.21.15.29:36542) 2023-10-14T17:00:02.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 -- 172.21.15.29:0/108612292 learned_addr learned my addr 172.21.15.29:0/108612292 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 -- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a640af030 msgr2=0x7f2a640af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a640af030 0x7f2a640af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 -- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a500039a0 msgr2=0x7f2a64150af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a500039a0 0x7f2a64150af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.782+0000 7f2a5a7fc640 1 -- 172.21.15.29:0/108612292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a641f2950 con 0x7f2a641f6a20 2023-10-14T17:00:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a5a7fc640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c011560 tx=0x7f2a4c011a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c004220 con 0x7f2a641f6a20 2023-10-14T17:00:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a4c004380 con 0x7f2a641f6a20 2023-10-14T17:00:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a641f2c30 con 0x7f2a641f6a20 2023-10-14T17:00:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c015870 con 0x7f2a641f6a20 2023-10-14T17:00:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.783+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a641f31f0 con 0x7f2a641f6a20 2023-10-14T17:00:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.784+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6404e9b0 con 0x7f2a641f6a20 2023-10-14T17:00:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.785+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 89462+0+0 (secure 0 0 0) 0x7f2a4c0159d0 con 0x7f2a641f6a20 2023-10-14T17:00:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.786+0000 7f2a60ff9640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 0x7f2a4406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.786+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f2a4c089df0 con 0x7f2a641f6a20 2023-10-14T17:00:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.787+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 0x7f2a4406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:02.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.787+0000 7f2a62ffd640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 0x7f2a4406d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a54034000 tx=0x7f2a54009030 comp rx=0 tx=0).ready entity=mgr.8199 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:02.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:02.791+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2a4c054910 con 0x7f2a641f6a20 2023-10-14T17:00:03.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.010+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 --> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a640c7260 con 0x7f2a4406b100 2023-10-14T17:00:03.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.202+0000 7f2a60ff9640 1 -- 172.21.15.29:0/108612292 <== mgr.8199 v2:172.21.15.29:6832/1622969171 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f2a640c7260 con 0x7f2a4406b100 2023-10-14T17:00:03.203 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-14T17:00:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 msgr2=0x7f2a4406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:03.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 0x7f2a4406d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f2a54034000 tx=0x7f2a54009030 comp rx=0 tx=0).stop 2023-10-14T17:00:03.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 msgr2=0x7f2a640c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c011560 tx=0x7f2a4c011a20 comp rx=0 tx=0).stop 2023-10-14T17:00:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 shutdown_connections 2023-10-14T17:00:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f2a4406b100 0x7f2a4406d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2a640af030 0x7f2a640af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2a641f6a20 0x7f2a640c5770 unknown :-1 s=CLOSED pgs=1456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 --2- 172.21.15.29:0/108612292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2a500039a0 0x7f2a64150af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 >> 172.21.15.29:0/108612292 conn(0x7f2a640ce6a0 msgr2=0x7f2a640ad130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 shutdown_connections 2023-10-14T17:00:03.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.205+0000 7f2a63fff640 1 -- 172.21.15.29:0/108612292 wait complete. 2023-10-14T17:00:03.226 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-14T17:00:03.227 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:03.227 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T17:00:03.283 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_00_00_UTC 2023-10-14T17:00:03.284 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_00_00_UTC 2023-10-14T17:00:03.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:03.285 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_00_00_UTC) 2023-10-14T17:00:03.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:03.351 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T17:00:03.413 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T17:00:03.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.643+0000 7fd6e9573640 1 Processor -- start 2023-10-14T17:00:03.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.643+0000 7fd6e9573640 1 -- start start 2023-10-14T17:00:03.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6e4159950 0x7fd6e415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd6e4070450 0x7fd6e40708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e406c100 0x7fd6e406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e4070e20 con 0x7fd6e4159950 2023-10-14T17:00:03.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6e415e490 con 0x7fd6e4070450 2023-10-14T17:00:03.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e9573640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e415aef0 con 0x7fd6e406c100 2023-10-14T17:00:03.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6e4159950 0x7fd6e415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd6e4159950 0x7fd6e415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36544/0 (socket says 172.21.15.29:36544) 2023-10-14T17:00:03.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.645+0000 7fd6e3fff640 1 -- 172.21.15.29:0/1630532290 learned_addr learned my addr 172.21.15.29:0/1630532290 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:03.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e8d72640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e406c100 0x7fd6e406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6dbfff640 1 --2- 172.21.15.29:0/1630532290 >> v2:172.21.15.186:3300/0 conn(0x7fd6e4070450 0x7fd6e40708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e3fff640 1 -- 172.21.15.29:0/1630532290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e406c100 msgr2=0x7fd6e406c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:03.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e406c100 0x7fd6e406c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e3fff640 1 -- 172.21.15.29:0/1630532290 >> v2:172.21.15.186:3300/0 conn(0x7fd6e4070450 msgr2=0x7fd6e40708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:03.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1630532290 >> v2:172.21.15.186:3300/0 conn(0x7fd6e4070450 0x7fd6e40708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.646+0000 7fd6e3fff640 1 -- 172.21.15.29:0/1630532290 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e415d910 con 0x7fd6e4159950 2023-10-14T17:00:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.647+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e415a9b0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40cab70 tx=0x7fd6d402b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6b043aae1cef1a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e37fe640 1 -- 172.21.15.29:0/1630532290 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d40317c0 con 0x7fd6e4159950 2023-10-14T17:00:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e37fe640 1 -- 172.21.15.29:0/1630532290 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4031920 con 0x7fd6e4159950 2023-10-14T17:00:03.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e37fe640 1 -- 172.21.15.29:0/1630532290 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6d4031cb0 con 0x7fd6e4159950 2023-10-14T17:00:03.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 -- 172.21.15.29:0/1630532290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 msgr2=0x7fd6e415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e415a9b0 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 crypto rx=0x7fd6e40cab70 tx=0x7fd6d402b860 comp rx=0 tx=0).stop 2023-10-14T17:00:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 -- 172.21.15.29:0/1630532290 shutdown_connections 2023-10-14T17:00:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e406c100 0x7fd6e406c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1630532290 >> v2:172.21.15.186:3300/0 conn(0x7fd6e4070450 0x7fd6e40708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1630532290 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e415a9b0 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 -- 172.21.15.29:0/1630532290 >> 172.21.15.29:0/1630532290 conn(0x7fd6e40ce6a0 msgr2=0x7fd6e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.648+0000 7fd6e9573640 1 -- 172.21.15.29:0/1630532290 shutdown_connections 2023-10-14T17:00:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.649+0000 7fd6e9573640 1 -- 172.21.15.29:0/1630532290 wait complete. 2023-10-14T17:00:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.650+0000 7fd6e9573640 1 Processor -- start 2023-10-14T17:00:03.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.650+0000 7fd6e9573640 1 -- start start 2023-10-14T17:00:03.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.651+0000 7fd6e9573640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 0x7fd6e4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.651+0000 7fd6e9573640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e4070450 0x7fd6e40b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e9573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e9573640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40b12c0 con 0x7fd6e4159950 2023-10-14T17:00:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e9573640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40c3a00 con 0x7fd6e406c100 2023-10-14T17:00:03.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e9573640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6e40c2e80 con 0x7fd6e4070450 2023-10-14T17:00:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36548/0 (socket says 172.21.15.29:36548) 2023-10-14T17:00:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 -- 172.21.15.29:0/1768763557 learned_addr learned my addr 172.21.15.29:0/1768763557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 0x7fd6e4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6dbfff640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e4070450 0x7fd6e40b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 -- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e4070450 msgr2=0x7fd6e40b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e4070450 0x7fd6e40b0170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 -- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 msgr2=0x7fd6e4154ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 0x7fd6e4154ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e8d72640 1 -- 172.21.15.29:0/1768763557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6d40094d0 con 0x7fd6e4159950 2023-10-14T17:00:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.652+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 0x7fd6e4154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e8d72640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc007e60 tx=0x7fd6dc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc014420 con 0x7fd6e4159950 2023-10-14T17:00:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6dc0145a0 con 0x7fd6e4159950 2023-10-14T17:00:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6dc014950 con 0x7fd6e4159950 2023-10-14T17:00:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6e40c3160 con 0x7fd6e4159950 2023-10-14T17:00:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.653+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6e40c3720 con 0x7fd6e4159950 2023-10-14T17:00:03.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.654+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6e404e9b0 con 0x7fd6e4159950 2023-10-14T17:00:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.655+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 64) v1 ==== 89462+0+0 (secure 0 0 0) 0x7fd6dc014ab0 con 0x7fd6e4159950 2023-10-14T17:00:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.656+0000 7fd6e17fa640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 0x7fd6c406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.656+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fd6dc08a590 con 0x7fd6e4159950 2023-10-14T17:00:03.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.657+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 0x7fd6c406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.657+0000 7fd6e3fff640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 0x7fd6c406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4036000 tx=0x7fd6d403d040 comp rx=0 tx=0).ready entity=mgr.8199 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.661+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6dc055180 con 0x7fd6e4159950 2023-10-14T17:00:03.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:03.908+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fd6e40b4620 con 0x7fd6e4159950 2023-10-14T17:00:04.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.318+0000 7fd6e17fa640 1 -- 172.21.15.29:0/1768763557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v65) v1 ==== 93+0+0 (secure 0 0 0) 0x7fd6dc0587a0 con 0x7fd6e4159950 2023-10-14T17:00:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 msgr2=0x7fd6c406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 0x7fd6c406d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fd6d4036000 tx=0x7fd6d403d040 comp rx=0 tx=0).stop 2023-10-14T17:00:04.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 msgr2=0x7fd6e40b06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc007e60 tx=0x7fd6dc00e4b0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 shutdown_connections 2023-10-14T17:00:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fd6c406b100 0x7fd6c406d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6e4159950 0x7fd6e40b06b0 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6e4070450 0x7fd6e40b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 --2- 172.21.15.29:0/1768763557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6e406c100 0x7fd6e4154ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.321+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 >> 172.21.15.29:0/1768763557 conn(0x7fd6e40ce6a0 msgr2=0x7fd6e40cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.322+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 shutdown_connections 2023-10-14T17:00:04.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.322+0000 7fd6e9573640 1 -- 172.21.15.29:0/1768763557 wait complete. 2023-10-14T17:00:04.348 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:00:04.348 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:04.444 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:04.553 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:00:04.554 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:04.554 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:04.611 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:04.612 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:04.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:04.846 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:04.845+0000 7f83bfc2f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:04.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.865+0000 7fa5af59e640 1 Processor -- start 2023-10-14T17:00:04.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.866+0000 7fa5af59e640 1 -- start start 2023-10-14T17:00:04.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.867+0000 7fa5af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 0x7fa5b00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.867+0000 7fa5af59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 0x7fa5b00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.867+0000 7fa5af59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5b015d150 con 0x7fa5b00b8480 2023-10-14T17:00:04.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.867+0000 7fa5af59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5b00d1180 con 0x7fa5b00cded0 2023-10-14T17:00:04.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.867+0000 7fa5af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00d1460 con 0x7fa5b015cd70 2023-10-14T17:00:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ae59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 0x7fa5b00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5add9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fa5b00b8480 0x7fa5b00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:36412/0 (socket says 172.21.15.29:36412) 2023-10-14T17:00:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5aed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 0x7fa5b00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:04.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5add9b640 1 -- 172.21.15.29:0/733334575 learned_addr learned my addr 172.21.15.29:0/733334575 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:04.870 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fb84642e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:04.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ad59a640 1 -- 172.21.15.29:0/733334575 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885970070 0 0) 0x7fa5b015d150 con 0x7fa5b00b8480 2023-10-14T17:00:04.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ad59a640 1 -- 172.21.15.29:0/733334575 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa598003520 con 0x7fa5b00b8480 2023-10-14T17:00:04.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ae59c640 1 -- 172.21.15.29:0/733334575 >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 msgr2=0x7fa5b00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:04.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ae59c640 1 --2- 172.21.15.29:0/733334575 >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 0x7fa5b00b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ae59c640 1 -- 172.21.15.29:0/733334575 >> v1:172.21.15.186:6789/0 conn(0x7fa5b00b8480 legacy=0x7fa5b00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5ae59c640 1 -- 172.21.15.29:0/733334575 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b00d1740 con 0x7fa5b015cd70 2023-10-14T17:00:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.868+0000 7fa5aed9d640 1 --2- 172.21.15.29:0/733334575 >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 0x7fa5b00b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:00:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.870+0000 7fa5ae59c640 1 --2- 172.21.15.29:0/733334575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 0x7fa5b00d2220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00097a0 tx=0x7fa5a002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b0fb9f55ecb91a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.870+0000 7fa5ad59a640 1 -- 172.21.15.29:0/733334575 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a002d040 con 0x7fa5b015cd70 2023-10-14T17:00:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.870+0000 7fa5ad59a640 1 -- 172.21.15.29:0/733334575 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a0031810 con 0x7fa5b015cd70 2023-10-14T17:00:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.870+0000 7fa5ad59a640 1 -- 172.21.15.29:0/733334575 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a0031b80 con 0x7fa5b015cd70 2023-10-14T17:00:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 -- 172.21.15.29:0/733334575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 msgr2=0x7fa5b00d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 --2- 172.21.15.29:0/733334575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 0x7fa5b00d2220 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fa5a00097a0 tx=0x7fa5a002b8b0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 -- 172.21.15.29:0/733334575 shutdown_connections 2023-10-14T17:00:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 --2- 172.21.15.29:0/733334575 >> v2:172.21.15.186:3300/0 conn(0x7fa5b00cded0 0x7fa5b00b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 --2- 172.21.15.29:0/733334575 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b015cd70 0x7fa5b00d2220 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 -- 172.21.15.29:0/733334575 >> 172.21.15.29:0/733334575 conn(0x7fa5b00cf570 msgr2=0x7fa5b00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 -- 172.21.15.29:0/733334575 shutdown_connections 2023-10-14T17:00:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.871+0000 7fa5af59e640 1 -- 172.21.15.29:0/733334575 wait complete. 2023-10-14T17:00:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.873+0000 7fa5af59e640 1 Processor -- start 2023-10-14T17:00:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.873+0000 7fa5af59e640 1 -- start start 2023-10-14T17:00:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b00b8480 0x7fa5b00cbe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b00cded0 0x7fa5b00ca5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b015a8b0 con 0x7fa5b015cd70 2023-10-14T17:00:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00cb680 con 0x7fa5b00cded0 2023-10-14T17:00:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b00cade0 con 0x7fa5b00b8480 2023-10-14T17:00:04.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36588/0 (socket says 172.21.15.29:36588) 2023-10-14T17:00:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5aed9d640 1 -- 172.21.15.29:0/49983771 learned_addr learned my addr 172.21.15.29:0/49983771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5ae59c640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b00b8480 0x7fa5b00cbe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:04.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.874+0000 7fa5add9b640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b00cded0 0x7fa5b00ca5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b00b8480 msgr2=0x7fa5b00cbe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b00b8480 0x7fa5b00cbe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b00cded0 msgr2=0x7fa5b00ca5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b00cded0 0x7fa5b00ca5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:04.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 -- 172.21.15.29:0/49983771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a8009a70 con 0x7fa5b015cd70 2023-10-14T17:00:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5aed9d640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800efc0 tx=0x7fa5a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8009cc0 con 0x7fa5b015cd70 2023-10-14T17:00:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5a800f040 con 0x7fa5b015cd70 2023-10-14T17:00:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a0009400 con 0x7fa5b015cd70 2023-10-14T17:00:04.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a8003990 con 0x7fa5b015cd70 2023-10-14T17:00:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.875+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b00cb3a0 con 0x7fa5b015cd70 2023-10-14T17:00:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.877+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5b004e9b0 con 0x7fa5b015cd70 2023-10-14T17:00:04.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.878+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa5a8014480 con 0x7fa5b015cd70 2023-10-14T17:00:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.879+0000 7fa59effd640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 0x7fa59006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.879+0000 7fa5ae59c640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 msgr2=0x7fa59006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.879+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fa5a808a820 con 0x7fa5b015cd70 2023-10-14T17:00:04.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.879+0000 7fa5ae59c640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 0x7fa59006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:04.883+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5a8055360 con 0x7fa5b015cd70 2023-10-14T17:00:04.901 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:04.901+0000 7f83bfc2f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:04.928 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:04.927+0000 7fb84642e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:05.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.080+0000 7fa5ae59c640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 msgr2=0x7fa59006d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.080+0000 7fa5ae59c640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 0x7fa59006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:05.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.080+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5b00c9370 con 0x7fa5b015cd70 2023-10-14T17:00:05.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.082+0000 7fa59effd640 1 -- 172.21.15.29:0/49983771 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17077 (secure 0 0 0) 0x7fa5a8058980 con 0x7fa5b015cd70 2023-10-14T17:00:05.083 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:05.083 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":126,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:41.541979+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6832/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:6833/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3986638800":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/354517400":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3653150355":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3890360705":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/2790183016":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3481988951":"2023-10-15T16:59:41.541911+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:05.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 msgr2=0x7fa59006d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 0x7fa59006d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 msgr2=0x7fa5b01f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 crypto rx=0x7fa5a800efc0 tx=0x7fa5a800c490 comp rx=0 tx=0).stop 2023-10-14T17:00:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 shutdown_connections 2023-10-14T17:00:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa59006b0e0 0x7fa59006d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5b015cd70 0x7fa5b01f2200 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5b00cded0 0x7fa5b00ca5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 --2- 172.21.15.29:0/49983771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5b00b8480 0x7fa5b00cbe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.084+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 >> 172.21.15.29:0/49983771 conn(0x7fa5b00cf570 msgr2=0x7fa5b015be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.085+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 shutdown_connections 2023-10-14T17:00:05.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.085+0000 7fa5af59e640 1 -- 172.21.15.29:0/49983771 wait complete. 2023-10-14T17:00:05.105 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:59:24.261633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:59:25.325453+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:05.106 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.321+0000 7fa59a780640 1 Processor -- start 2023-10-14T17:00:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.321+0000 7fa59a780640 1 -- start start 2023-10-14T17:00:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 0x7fa5940ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5940ce960 0x7fa5940b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940d0970 con 0x7fa5940ce960 2023-10-14T17:00:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5940d0ea0 con 0x7fa5940b2080 2023-10-14T17:00:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59a780640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940d0bc0 con 0x7fa5940b1ca0 2023-10-14T17:00:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5940ce960 0x7fa5940b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa5940ce960 0x7fa5940b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36594/0 (socket says 172.21.15.29:36594) 2023-10-14T17:00:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 -- 172.21.15.29:0/1266974713 learned_addr learned my addr 172.21.15.29:0/1266974713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa59977e640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa598f7d640 1 --2- 172.21.15.29:0/1266974713 >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 0x7fa5940ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 -- 172.21.15.29:0/1266974713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 msgr2=0x7fa5940d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 -- 172.21.15.29:0/1266974713 >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 msgr2=0x7fa5940ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1266974713 >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 0x7fa5940ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa599f7f640 1 -- 172.21.15.29:0/1266974713 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5940cd940 con 0x7fa5940ce960 2023-10-14T17:00:05.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.323+0000 7fa598f7d640 1 --2- 172.21.15.29:0/1266974713 >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 0x7fa5940ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.325+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa5940b24c0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa594067300 tx=0x7fa58c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77238ec1a258182 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.325+0000 7fa58bfff640 1 -- 172.21.15.29:0/1266974713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c02cb00 con 0x7fa5940ce960 2023-10-14T17:00:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.325+0000 7fa58bfff640 1 -- 172.21.15.29:0/1266974713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c02cc60 con 0x7fa5940ce960 2023-10-14T17:00:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.325+0000 7fa58bfff640 1 -- 172.21.15.29:0/1266974713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c0343f0 con 0x7fa5940ce960 2023-10-14T17:00:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 -- 172.21.15.29:0/1266974713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 msgr2=0x7fa5940b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa5940b24c0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 crypto rx=0x7fa594067300 tx=0x7fa58c02bf60 comp rx=0 tx=0).stop 2023-10-14T17:00:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 -- 172.21.15.29:0/1266974713 shutdown_connections 2023-10-14T17:00:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa5940b24c0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 --2- 172.21.15.29:0/1266974713 >> v2:172.21.15.186:3300/0 conn(0x7fa5940b2080 0x7fa5940ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 --2- 172.21.15.29:0/1266974713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 -- 172.21.15.29:0/1266974713 >> 172.21.15.29:0/1266974713 conn(0x7fa5940d2b90 msgr2=0x7fa5940d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 -- 172.21.15.29:0/1266974713 shutdown_connections 2023-10-14T17:00:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.326+0000 7fa59a780640 1 -- 172.21.15.29:0/1266974713 wait complete. 2023-10-14T17:00:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.327+0000 7fa59a780640 1 Processor -- start 2023-10-14T17:00:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.328+0000 7fa59a780640 1 -- start start 2023-10-14T17:00:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940c76d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 0x7fa5940c8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940c66c0 con 0x7fa5940ce960 2023-10-14T17:00:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940c6910 con 0x7fa5940b2080 2023-10-14T17:00:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59a780640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5940c6bf0 con 0x7fa5940b1ca0 2023-10-14T17:00:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36604/0 (socket says 172.21.15.29:36604) 2023-10-14T17:00:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 -- 172.21.15.29:0/1542138307 learned_addr learned my addr 172.21.15.29:0/1542138307 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa598f7d640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 0x7fa5940c8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa59977e640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940c76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 msgr2=0x7fa5940c76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940c76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 msgr2=0x7fa5940c8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.329+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 0x7fa5940c8990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa599f7f640 1 -- 172.21.15.29:0/1542138307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa58c009b30 con 0x7fa5940ce960 2023-10-14T17:00:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa598f7d640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 0x7fa5940c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa599f7f640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fa58c036040 tx=0x7fa58c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c040670 con 0x7fa5940ce960 2023-10-14T17:00:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa58c0407d0 con 0x7fa5940ce960 2023-10-14T17:00:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5940c6ed0 con 0x7fa5940ce960 2023-10-14T17:00:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.330+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c040b80 con 0x7fa5940ce960 2023-10-14T17:00:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.331+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa594157460 con 0x7fa5940ce960 2023-10-14T17:00:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.331+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5940c73f0 con 0x7fa5940ce960 2023-10-14T17:00:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.333+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fa58c040ce0 con 0x7fa5940ce960 2023-10-14T17:00:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.334+0000 7fa589ffb640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 0x7fa57806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.335+0000 7fa59977e640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 msgr2=0x7fa57806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.335+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fa58c0acbc0 con 0x7fa5940ce960 2023-10-14T17:00:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.335+0000 7fa59977e640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 0x7fa57806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.339+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa5940c73f0 con 0x7fa5940ce960 2023-10-14T17:00:05.360 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.360+0000 7f83bfc2f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:05.413 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.412+0000 7f83bfc2f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:05.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.415+0000 7fb84642e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:05.463 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.462+0000 7f83bfc2f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:05.472 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.471+0000 7fb84642e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:05.524 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.524+0000 7f83bfc2f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:05.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.524+0000 7fb84642e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:05.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.536+0000 7fa59977e640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 msgr2=0x7fa57806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:05.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.536+0000 7fa59977e640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 0x7fa57806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.536+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5940c73f0 con 0x7fa5940ce960 2023-10-14T17:00:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.536+0000 7fa589ffb640 1 -- 172.21.15.29:0/1542138307 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5940c73f0 con 0x7fa5940ce960 2023-10-14T17:00:05.538 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 msgr2=0x7fa57806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:05.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 0x7fa57806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 msgr2=0x7fa594157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 secure :-1 s=READY pgs=1464 cs=0 l=1 rev1=1 crypto rx=0x7fa58c036040 tx=0x7fa58c004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:05.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 shutdown_connections 2023-10-14T17:00:05.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fa57806b0b0 0x7fa57806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa5940ce960 0x7fa594157f40 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa5940b2080 0x7fa5940c8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 --2- 172.21.15.29:0/1542138307 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa5940b1ca0 0x7fa5940c76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 >> 172.21.15.29:0/1542138307 conn(0x7fa5940d2b90 msgr2=0x7fa5940d3020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:05.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 shutdown_connections 2023-10-14T17:00:05.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.539+0000 7fa59a780640 1 -- 172.21.15.29:0/1542138307 wait complete. 2023-10-14T17:00:05.558 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:00:05.573 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.573+0000 7f83bfc2f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:05.591 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.591+0000 7fb84642e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:05.645 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.644+0000 7fb84642e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:05.684 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.683+0000 7f83bfc2f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:05.746 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.745+0000 7f83bfc2f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:05.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.757+0000 7fb25121a640 1 Processor -- start 2023-10-14T17:00:05.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.757+0000 7fb25121a640 1 -- start start 2023-10-14T17:00:05.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb25121a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb25121a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb24c0ce280 0x7fb24c0ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb25121a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb24c0acd70 con 0x7fb24c065ab0 2023-10-14T17:00:05.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb25121a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb24c0ac4a0 con 0x7fb24c0ce280 2023-10-14T17:00:05.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb25121a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c065f40 con 0x7fb24c06d220 2023-10-14T17:00:05.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb24b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb24c065ab0 0x7fb24c0cdb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42434/0 (socket says 172.21.15.29:42434) 2023-10-14T17:00:05.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb24b7fe640 1 -- 172.21.15.29:0/1789718238 learned_addr learned my addr 172.21.15.29:0/1789718238 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:05.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb250a19640 1 --2- 172.21.15.29:0/1789718238 >> v2:172.21.15.29:3300/0 conn(0x7fb24c0ce280 0x7fb24c0ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.759+0000 7fb24bfff640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb24affd640 1 -- 172.21.15.29:0/1789718238 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 871977836 0 0) 0x7fb24c0acd70 con 0x7fb24c065ab0 2023-10-14T17:00:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb24affd640 1 -- 172.21.15.29:0/1789718238 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb234003520 con 0x7fb24c065ab0 2023-10-14T17:00:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb250a19640 1 -- 172.21.15.29:0/1789718238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 msgr2=0x7fb24c0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb250a19640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb250a19640 1 -- 172.21.15.29:0/1789718238 >> v1:172.21.15.29:6789/0 conn(0x7fb24c065ab0 legacy=0x7fb24c0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.760+0000 7fb250a19640 1 -- 172.21.15.29:0/1789718238 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb24c0ac720 con 0x7fb24c0ce280 2023-10-14T17:00:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.762+0000 7fb250a19640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0ce280 0x7fb24c0ad200 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb244009d90 tx=0x7fb24402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5bb2f9cb18696d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.762+0000 7fb24affd640 1 -- 172.21.15.29:0/1789718238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24402cc60 con 0x7fb24c0ce280 2023-10-14T17:00:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.762+0000 7fb24affd640 1 -- 172.21.15.29:0/1789718238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24402cdc0 con 0x7fb24c0ce280 2023-10-14T17:00:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.762+0000 7fb24affd640 1 -- 172.21.15.29:0/1789718238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244034660 con 0x7fb24c0ce280 2023-10-14T17:00:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 -- 172.21.15.29:0/1789718238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0ce280 msgr2=0x7fb24c0ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0ce280 0x7fb24c0ad200 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 crypto rx=0x7fb244009d90 tx=0x7fb24402beb0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 -- 172.21.15.29:0/1789718238 shutdown_connections 2023-10-14T17:00:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0ce280 0x7fb24c0ad200 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 --2- 172.21.15.29:0/1789718238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c0c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 -- 172.21.15.29:0/1789718238 >> 172.21.15.29:0/1789718238 conn(0x7fb24c0b0a80 msgr2=0x7fb24c0b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 -- 172.21.15.29:0/1789718238 shutdown_connections 2023-10-14T17:00:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.763+0000 7fb25121a640 1 -- 172.21.15.29:0/1789718238 wait complete. 2023-10-14T17:00:05.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.764+0000 7fb84642e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.765+0000 7fb25121a640 1 Processor -- start 2023-10-14T17:00:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.765+0000 7fb25121a640 1 -- start start 2023-10-14T17:00:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c06c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24c0ce280 0x7fb24c0b1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c143250 con 0x7fb24c0b20f0 2023-10-14T17:00:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c0b2580 con 0x7fb24c0ce280 2023-10-14T17:00:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.766+0000 7fb25121a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24c1f2840 con 0x7fb24c06d220 2023-10-14T17:00:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36614/0 (socket says 172.21.15.29:36614) 2023-10-14T17:00:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 -- 172.21.15.29:0/2853329802 learned_addr learned my addr 172.21.15.29:0/2853329802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb24bfff640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c06c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb24b7fe640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24c0ce280 0x7fb24c0b1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 msgr2=0x7fb24c06c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c06c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24c0ce280 msgr2=0x7fb24c0b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24c0ce280 0x7fb24c0b1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.767+0000 7fb250a19640 1 -- 172.21.15.29:0/2853329802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb244009a70 con 0x7fb24c0b20f0 2023-10-14T17:00:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb250a19640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb244009d90 tx=0x7fb244004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244040650 con 0x7fb24c0b20f0 2023-10-14T17:00:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2440407b0 con 0x7fb24c0b20f0 2023-10-14T17:00:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244040b60 con 0x7fb24c0b20f0 2023-10-14T17:00:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb24c1f3810 con 0x7fb24c0b20f0 2023-10-14T17:00:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.768+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb24c1f2d30 con 0x7fb24c0b20f0 2023-10-14T17:00:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.769+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb24c0ae430 con 0x7fb24c0b20f0 2023-10-14T17:00:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.771+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fb244040cc0 con 0x7fb24c0b20f0 2023-10-14T17:00:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.772+0000 7fb248ff9640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 0x7fb23806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.772+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fb2440ac830 con 0x7fb24c0b20f0 2023-10-14T17:00:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.772+0000 7fb24bfff640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 msgr2=0x7fb23806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.772+0000 7fb24bfff640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 0x7fb23806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.777+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2440774c0 con 0x7fb24c0b20f0 2023-10-14T17:00:05.795 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.794+0000 7f83bfc2f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:05.837 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.837+0000 7fb84642e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:05.856 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.855+0000 7f83bfc2f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:05.889 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.889+0000 7fb84642e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:05.907 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:05.906+0000 7f83bfc2f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:05.949 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:05.949+0000 7fb84642e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.973+0000 7fb24bfff640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 msgr2=0x7fb23806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.973+0000 7fb24bfff640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 0x7fb23806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:05.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.974+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb24c04e9b0 con 0x7fb24c0b20f0 2023-10-14T17:00:05.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.975+0000 7fb248ff9640 1 -- 172.21.15.29:0/2853329802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v126) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb244009450 con 0x7fb24c0b20f0 2023-10-14T17:00:05.976 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:00:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 msgr2=0x7fb23806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 0x7fb23806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 msgr2=0x7fb24c1424e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 crypto rx=0x7fb244009d90 tx=0x7fb244004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 shutdown_connections 2023-10-14T17:00:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fb23806b0b0 0x7fb23806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb24c0b20f0 0x7fb24c1424e0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb24c0ce280 0x7fb24c0b1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 --2- 172.21.15.29:0/2853329802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb24c06d220 0x7fb24c06c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.978+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 >> 172.21.15.29:0/2853329802 conn(0x7fb24c0b0a80 msgr2=0x7fb24c0ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.979+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 shutdown_connections 2023-10-14T17:00:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:05.979+0000 7fb25121a640 1 -- 172.21.15.29:0/2853329802 wait complete. 2023-10-14T17:00:05.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:00:06.004 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.003+0000 7fb84642e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.210+0000 7f9e51ebb640 1 Processor -- start 2023-10-14T17:00:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.210+0000 7f9e51ebb640 1 -- start start 2023-10-14T17:00:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e51ebb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e4c0b8480 0x7f9e4c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e51ebb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e4c0d0be0 0x7f9e4c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e51ebb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e4c15d150 con 0x7f9e4c15cd70 2023-10-14T17:00:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e51ebb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4c0d1180 con 0x7f9e4c0b8480 2023-10-14T17:00:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e51ebb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e4c0d1460 con 0x7f9e4c0d0be0 2023-10-14T17:00:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e50eb9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9e4c15cd70 0x7f9e4c0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42446/0 (socket says 172.21.15.29:42446) 2023-10-14T17:00:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e50eb9640 1 -- 172.21.15.29:0/2063431218 learned_addr learned my addr 172.21.15.29:0/2063431218 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.212+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/2063431218 >> v2:172.21.15.29:3300/0 conn(0x7f9e4c0b8480 0x7f9e4c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e516ba640 1 --2- 172.21.15.29:0/2063431218 >> v2:172.21.15.186:3300/0 conn(0x7f9e4c0d0be0 0x7f9e4c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e4bfff640 1 -- 172.21.15.29:0/2063431218 >> v2:172.21.15.186:3300/0 conn(0x7f9e4c0d0be0 msgr2=0x7f9e4c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/2063431218 >> v2:172.21.15.186:3300/0 conn(0x7f9e4c0d0be0 0x7f9e4c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e4bfff640 1 -- 172.21.15.29:0/2063431218 >> v1:172.21.15.29:6789/0 conn(0x7f9e4c15cd70 legacy=0x7f9e4c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e4bfff640 1 -- 172.21.15.29:0/2063431218 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e4c0d1740 con 0x7f9e4c0b8480 2023-10-14T17:00:06.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.213+0000 7f9e4b7fe640 1 -- 172.21.15.29:0/2063431218 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184915030 0 0) 0x7f9e4c15d150 con 0x7f9e4c15cd70 2023-10-14T17:00:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.214+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/2063431218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c0cd7c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0b8910 tx=0x7f9e3802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81918a9dee02fa79 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.214+0000 7f9e4b7fe640 1 -- 172.21.15.29:0/2063431218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e3802e070 con 0x7f9e4c0b8480 2023-10-14T17:00:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e4b7fe640 1 -- 172.21.15.29:0/2063431218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e380315a0 con 0x7f9e4c0b8480 2023-10-14T17:00:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e4b7fe640 1 -- 172.21.15.29:0/2063431218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e38031930 con 0x7f9e4c0b8480 2023-10-14T17:00:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 -- 172.21.15.29:0/2063431218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 msgr2=0x7f9e4c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/2063431218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c0cd7c0 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c0b8910 tx=0x7f9e3802b840 comp rx=0 tx=0).stop 2023-10-14T17:00:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 -- 172.21.15.29:0/2063431218 shutdown_connections 2023-10-14T17:00:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/2063431218 >> v2:172.21.15.186:3300/0 conn(0x7f9e4c0d0be0 0x7f9e4c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/2063431218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c0cd7c0 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 -- 172.21.15.29:0/2063431218 >> 172.21.15.29:0/2063431218 conn(0x7f9e4c0cf570 msgr2=0x7f9e4c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 -- 172.21.15.29:0/2063431218 shutdown_connections 2023-10-14T17:00:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.215+0000 7f9e51ebb640 1 -- 172.21.15.29:0/2063431218 wait complete. 2023-10-14T17:00:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.217+0000 7f9e51ebb640 1 Processor -- start 2023-10-14T17:00:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.217+0000 7f9e51ebb640 1 -- start start 2023-10-14T17:00:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e4c0d0be0 0x7f9e4c14d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4c15cd70 0x7f9e4c15ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c15e930 con 0x7f9e4c0b8480 2023-10-14T17:00:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c15dcb0 con 0x7f9e4c0d0be0 2023-10-14T17:00:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.218+0000 7f9e51ebb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4c15df90 con 0x7f9e4c15cd70 2023-10-14T17:00:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36636/0 (socket says 172.21.15.29:36636) 2023-10-14T17:00:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 -- 172.21.15.29:0/1156104334 learned_addr learned my addr 172.21.15.29:0/1156104334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e4c0d0be0 0x7f9e4c14d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e516ba640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4c15cd70 0x7f9e4c15ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4c15cd70 msgr2=0x7f9e4c15ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4c15cd70 0x7f9e4c15ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e4c0d0be0 msgr2=0x7f9e4c14d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e4c0d0be0 0x7f9e4c14d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 -- 172.21.15.29:0/1156104334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e38009400 con 0x7f9e4c0b8480 2023-10-14T17:00:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.219+0000 7f9e50eb9640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000c4d0 tx=0x7f9e4000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.220+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e400074e0 con 0x7f9e4c0b8480 2023-10-14T17:00:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.220+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e40007660 con 0x7f9e4c0b8480 2023-10-14T17:00:06.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.220+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e40004a20 con 0x7f9e4c0b8480 2023-10-14T17:00:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.220+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e4c15e270 con 0x7f9e4c0b8480 2023-10-14T17:00:06.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.220+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e4c15d880 con 0x7f9e4c0b8480 2023-10-14T17:00:06.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.221+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c15da70 con 0x7f9e4c0b8480 2023-10-14T17:00:06.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.222+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f9e40004b80 con 0x7f9e4c0b8480 2023-10-14T17:00:06.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.223+0000 7f9e497fa640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 0x7f9e3006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.223+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f9e40088cf0 con 0x7f9e4c0b8480 2023-10-14T17:00:06.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.223+0000 7f9e4bfff640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 msgr2=0x7f9e3006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:06.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.223+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 0x7f9e3006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:06.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.227+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e4c15da70 con 0x7f9e4c0b8480 2023-10-14T17:00:06.290 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.289+0000 7f83bfc2f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:06.413 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.413+0000 7fb84642e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:06.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.423+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9e4c15da70 con 0x7f9e4c0b8480 2023-10-14T17:00:06.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.424+0000 7f9e4bfff640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 msgr2=0x7f9e3006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:06.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.424+0000 7f9e4bfff640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 0x7f9e3006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:06.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.424+0000 7f9e497fa640 1 -- 172.21.15.29:0/1156104334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v126) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9e4c15da70 con 0x7f9e4c0b8480 2023-10-14T17:00:06.425 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:00:06.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 msgr2=0x7f9e3006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:06.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 0x7f9e3006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 msgr2=0x7f9e4c15d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 crypto rx=0x7f9e4000c4d0 tx=0x7f9e4000c990 comp rx=0 tx=0).stop 2023-10-14T17:00:06.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 shutdown_connections 2023-10-14T17:00:06.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f9e3006b0b0 0x7f9e3006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e4c15cd70 0x7f9e4c15ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e4c0d0be0 0x7f9e4c14d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 --2- 172.21.15.29:0/1156104334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e4c0b8480 0x7f9e4c15d340 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 >> 172.21.15.29:0/1156104334 conn(0x7f9e4c0cf570 msgr2=0x7f9e4c15be10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:06.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 shutdown_connections 2023-10-14T17:00:06.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.427+0000 7f9e51ebb640 1 -- 172.21.15.29:0/1156104334 wait complete. 2023-10-14T17:00:06.445 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T17:00:06.453 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.452+0000 7f83bfc2f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:06.563 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.562+0000 7f83bfc2f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:06.586 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.585+0000 7fb84642e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:06.614 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.613+0000 7f83bfc2f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:06.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.653+0000 7fc82c87e640 1 Processor -- start 2023-10-14T17:00:06.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.653+0000 7fc82c87e640 1 -- start start 2023-10-14T17:00:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc82c87e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc828070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc82c87e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8280adc00 0x7fc828070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc82c87e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8280ae1a0 con 0x7fc828160c80 2023-10-14T17:00:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc82c87e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8280ae480 con 0x7fc8280adc00 2023-10-14T17:00:06.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc82c87e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8280ae760 con 0x7fc8280ad7a0 2023-10-14T17:00:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc827fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8280adc00 0x7fc828070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc827fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc8280adc00 0x7fc828070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36648/0 (socket says 172.21.15.29:36648) 2023-10-14T17:00:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc827fff640 1 -- 172.21.15.29:0/383298862 learned_addr learned my addr 172.21.15.29:0/383298862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc8277fe640 1 --1- 172.21.15.29:0/383298862 >> v1:172.21.15.186:6789/0 conn(0x7fc828160c80 0x7fc828161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:36426/0 (socket says 172.21.15.29:36426) 2023-10-14T17:00:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.655+0000 7fc826ffd640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc828070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.656+0000 7fc827fff640 1 -- 172.21.15.29:0/383298862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 msgr2=0x7fc828070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.656+0000 7fc827fff640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc828070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.656+0000 7fc827fff640 1 -- 172.21.15.29:0/383298862 >> v1:172.21.15.186:6789/0 conn(0x7fc828160c80 legacy=0x7fc828161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.656+0000 7fc827fff640 1 -- 172.21.15.29:0/383298862 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8280aea40 con 0x7fc8280adc00 2023-10-14T17:00:06.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.657+0000 7fc827fff640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc828070780 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc820009e50 tx=0x7fc82002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a523ea2e1d39f1e1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.657+0000 7fc8267fc640 1 -- 172.21.15.29:0/383298862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc82002cb00 con 0x7fc8280adc00 2023-10-14T17:00:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.657+0000 7fc8267fc640 1 -- 172.21.15.29:0/383298862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82002cc60 con 0x7fc8280adc00 2023-10-14T17:00:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.657+0000 7fc8267fc640 1 -- 172.21.15.29:0/383298862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8200343f0 con 0x7fc8280adc00 2023-10-14T17:00:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 -- 172.21.15.29:0/383298862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 msgr2=0x7fc828070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc828070780 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 crypto rx=0x7fc820009e50 tx=0x7fc82002bf60 comp rx=0 tx=0).stop 2023-10-14T17:00:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 -- 172.21.15.29:0/383298862 shutdown_connections 2023-10-14T17:00:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc828070780 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 --2- 172.21.15.29:0/383298862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc828070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 -- 172.21.15.29:0/383298862 >> 172.21.15.29:0/383298862 conn(0x7fc8280cc230 msgr2=0x7fc8280cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 -- 172.21.15.29:0/383298862 shutdown_connections 2023-10-14T17:00:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.658+0000 7fc82c87e640 1 -- 172.21.15.29:0/383298862 wait complete. 2023-10-14T17:00:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.660+0000 7fc82c87e640 1 Processor -- start 2023-10-14T17:00:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.660+0000 7fc82c87e640 1 -- start start 2023-10-14T17:00:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc8280b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 0x7fc8280c5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc82815c100 con 0x7fc8280adc00 2023-10-14T17:00:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8280b4020 con 0x7fc828160c80 2023-10-14T17:00:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc82c87e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8280c5ff0 con 0x7fc8280ad7a0 2023-10-14T17:00:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc826ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36650/0 (socket says 172.21.15.29:36650) 2023-10-14T17:00:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc826ffd640 1 -- 172.21.15.29:0/634861437 learned_addr learned my addr 172.21.15.29:0/634861437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc827fff640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 0x7fc8280c5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc8277fe640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc8280b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:06.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.661+0000 7fc826ffd640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 msgr2=0x7fc8280b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc826ffd640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc8280b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc826ffd640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 msgr2=0x7fc8280c5ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc826ffd640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 0x7fc8280c5ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc826ffd640 1 -- 172.21.15.29:0/634861437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc820009b30 con 0x7fc8280adc00 2023-10-14T17:00:06.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc827fff640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 0x7fc8280c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc8277fe640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc8280b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc826ffd640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00d800 tx=0x7fc81c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c00b6c0 con 0x7fc8280adc00 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc81c00b820 con 0x7fc8280adc00 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc81c00bbb0 con 0x7fc8280adc00 2023-10-14T17:00:06.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8280b3410 con 0x7fc8280adc00 2023-10-14T17:00:06.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.662+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8280b39d0 con 0x7fc8280adc00 2023-10-14T17:00:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.664+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc82804e9b0 con 0x7fc8280adc00 2023-10-14T17:00:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.664+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fc81c00bd10 con 0x7fc8280adc00 2023-10-14T17:00:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.665+0000 7fc80ffff640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 0x7fc81406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.666+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fc81c08a1c0 con 0x7fc8280adc00 2023-10-14T17:00:06.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.666+0000 7fc8277fe640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 msgr2=0x7fc81406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.666+0000 7fc8277fe640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 0x7fc81406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:06.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.670+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc81c054ce0 con 0x7fc8280adc00 2023-10-14T17:00:06.703 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.702+0000 7fb84642e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:06.722 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.721+0000 7f83bfc2f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:06.756 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.755+0000 7fb84642e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:06.833 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.832+0000 7f83bfc2f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.867+0000 7fc8277fe640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 msgr2=0x7fc81406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:06.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.867+0000 7fc8277fe640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 0x7fc81406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:06.876 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.876+0000 7fb84642e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:06.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.883+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fc8280cf230 con 0x7fc8280adc00 2023-10-14T17:00:06.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.884+0000 7fc80ffff640 1 -- 172.21.15.29:0/634861437 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v126) v1 ==== 80+0+469 (secure 0 0 0) 0x7fc81c058300 con 0x7fc8280adc00 2023-10-14T17:00:06.885 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/2416849365 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.885 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/2416849365 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.885 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3986638800 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.885 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/354517400 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.886 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3653150355 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.886 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3890360705 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.886 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2790183016 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.886 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3481988951 2023-10-15T16:59:41.541911+0000 2023-10-14T17:00:06.887 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T17:00:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 msgr2=0x7fc81406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 0x7fc81406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 msgr2=0x7fc82815bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 crypto rx=0x7fc81c00d800 tx=0x7fc81c00dcc0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 shutdown_connections 2023-10-14T17:00:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fc81406b0b0 0x7fc81406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc828160c80 0x7fc8280c5ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc8280adc00 0x7fc82815bbc0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.886+0000 7fc82c87e640 1 --2- 172.21.15.29:0/634861437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc8280ad7a0 0x7fc8280b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:06.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.887+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 >> 172.21.15.29:0/634861437 conn(0x7fc8280cc230 msgr2=0x7fc82815d180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.887+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 shutdown_connections 2023-10-14T17:00:06.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:06.887+0000 7fc82c87e640 1 -- 172.21.15.29:0/634861437 wait complete. 2023-10-14T17:00:06.906 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T17:00:06.906 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:06.907 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:06.993 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:06.991+0000 7f83bfc2f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:06.994 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:06.993+0000 7fb84642e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:07.006 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T17:00:07.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:07.007 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T17:00:07.043 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:07.043+0000 7f83bfc2f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:07.057 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:07.057 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:07.111 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:07.112 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:07.112 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:07.168 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:07.167+0000 7f83bfc2f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:07.168 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:07.168+0000 7fb84642e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:07.171 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:00:07.171 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:07.171 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:07.220 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:07.220+0000 7fb84642e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:07.225 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:07.225 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:07.350 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:07.349+0000 7fb84642e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.444+0000 7f522f59e640 1 Processor -- start 2023-10-14T17:00:07.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.445+0000 7f522f59e640 1 -- start start 2023-10-14T17:00:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52300cdf10 0x7f5230071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f523006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52300d12b0 con 0x7f523015cdb0 2023-10-14T17:00:07.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f523015d190 con 0x7f52300cdf10 2023-10-14T17:00:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52300d04b0 con 0x7f5230071fa0 2023-10-14T17:00:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f523015cdb0 0x7f52300cd800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42448/0 (socket says 172.21.15.29:42448) 2023-10-14T17:00:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.446+0000 7f522e59c640 1 -- 172.21.15.29:0/4050262715 learned_addr learned my addr 172.21.15.29:0/4050262715 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:07.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> v2:172.21.15.186:3300/0 conn(0x7f52300cdf10 0x7f5230071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522ed9d640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f523006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4105292931 0 0) 0x7f52300d12b0 con 0x7f523015cdb0 2023-10-14T17:00:07.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c003520 con 0x7f523015cdb0 2023-10-14T17:00:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3332861085 0 0) 0x7f521c003520 con 0x7f523015cdb0 2023-10-14T17:00:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 msgr2=0x7f523006aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f523006aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 >> v2:172.21.15.186:3300/0 conn(0x7f52300cdf10 msgr2=0x7f5230071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:07.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 --2- 172.21.15.29:0/4050262715 >> v2:172.21.15.186:3300/0 conn(0x7f52300cdf10 0x7f5230071a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5230072410 con 0x7f523015cdb0 2023-10-14T17:00:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f52200017f0 con 0x7f523015cdb0 2023-10-14T17:00:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> v2:172.21.15.186:3300/0 conn(0x7f52300cdf10 0x7f5230071a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:00:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 >> v1:172.21.15.29:6789/0 conn(0x7f523015cdb0 legacy=0x7f52300cd800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f521c003970 0x7f521c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f521c0047c0 0x7f521c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f521c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c00b980 con 0x7f521c0047c0 2023-10-14T17:00:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521c00bac0 con 0x7f521c003970 2023-10-14T17:00:07.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.447+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f521c00bc50 con 0x7f5230071fa0 2023-10-14T17:00:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f521c0047c0 0x7f521c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f522e59c640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f521c003970 0x7f521c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f522ed9d640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f521c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 -- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 msgr2=0x7f521c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:07.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f521c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 -- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f521c003970 msgr2=0x7f521c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f521c003970 0x7f521c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f5227fff640 1 -- 172.21.15.29:0/4050262715 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c003520 con 0x7f521c0047c0 2023-10-14T17:00:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f522e59c640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f521c003970 0x7f521c003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.448+0000 7f522ed9d640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5230071fa0 0x7f521c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.449+0000 7f5227fff640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f521c0047c0 0x7f521c004c40 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f5218009c90 tx=0x7f521c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac483730c120836d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521802c980 con 0x7f521c0047c0 2023-10-14T17:00:07.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f521802cae0 con 0x7f521c0047c0 2023-10-14T17:00:07.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522dd9b640 1 -- 172.21.15.29:0/4050262715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521802ce90 con 0x7f521c0047c0 2023-10-14T17:00:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 -- 172.21.15.29:0/4050262715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f521c0047c0 msgr2=0x7f521c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 --2- 172.21.15.29:0/4050262715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f521c0047c0 0x7f521c004c40 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 crypto rx=0x7f5218009c90 tx=0x7f521c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522ed9d640 1 -- 172.21.15.29:0/4050262715 reap_dead start 2023-10-14T17:00:07.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 -- 172.21.15.29:0/4050262715 shutdown_connections 2023-10-14T17:00:07.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 -- 172.21.15.29:0/4050262715 >> 172.21.15.29:0/4050262715 conn(0x7f5230065ed0 msgr2=0x7f52300662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:07.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 -- 172.21.15.29:0/4050262715 shutdown_connections 2023-10-14T17:00:07.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.450+0000 7f522f59e640 1 -- 172.21.15.29:0/4050262715 wait complete. 2023-10-14T17:00:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.452+0000 7f522f59e640 1 Processor -- start 2023-10-14T17:00:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.452+0000 7f522f59e640 1 -- start start 2023-10-14T17:00:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f521c0047c0 0x7f5230160820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f523015eaf0 0x7f52300c8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52301600d0 con 0x7f523015e690 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523015f050 con 0x7f523015eaf0 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f522f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f523015f330 con 0x7f521c0047c0 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f5227fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f5227fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36664/0 (socket says 172.21.15.29:36664) 2023-10-14T17:00:07.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.453+0000 7f5227fff640 1 -- 172.21.15.29:0/2862580963 learned_addr learned my addr 172.21.15.29:0/2862580963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f522ed9d640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f523015eaf0 0x7f52300c8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f522e59c640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f521c0047c0 0x7f5230160820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f521c0047c0 msgr2=0x7f5230160820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f521c0047c0 0x7f5230160820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f523015eaf0 msgr2=0x7f52300c8800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f523015eaf0 0x7f52300c8800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 -- 172.21.15.29:0/2862580963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523015f610 con 0x7f523015e690 2023-10-14T17:00:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5227fff640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f521802c460 tx=0x7f5218004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.454+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218034c80 con 0x7f523015e690 2023-10-14T17:00:07.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.455+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5218034de0 con 0x7f523015e690 2023-10-14T17:00:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.455+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523015f8f0 con 0x7f523015e690 2023-10-14T17:00:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.455+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5218040690 con 0x7f523015e690 2023-10-14T17:00:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.455+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5230160d60 con 0x7f523015e690 2023-10-14T17:00:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.456+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523004e9b0 con 0x7f523015e690 2023-10-14T17:00:07.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.457+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f52180407f0 con 0x7f523015e690 2023-10-14T17:00:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.458+0000 7f5226ffd640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 0x7f521006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.458+0000 7f522e59c640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 msgr2=0x7f521006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.458+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f52180abfd0 con 0x7f523015e690 2023-10-14T17:00:07.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.458+0000 7f522e59c640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 0x7f521006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:07.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.463+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5218076c30 con 0x7f523015e690 2023-10-14T17:00:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.659+0000 7f522e59c640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 msgr2=0x7f521006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:07.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.659+0000 7f522e59c640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 0x7f521006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:07.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.697+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52300b6860 con 0x7f523015e690 2023-10-14T17:00:07.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.698+0000 7f5226ffd640 1 -- 172.21.15.29:0/2862580963 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f5218044090 con 0x7f523015e690 2023-10-14T17:00:07.698 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:07.699 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T17:00:07.699 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T17:00:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 msgr2=0x7f521006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 0x7f521006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 msgr2=0x7f52300c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 secure :-1 s=READY pgs=1475 cs=0 l=1 rev1=1 crypto rx=0x7f521802c460 tx=0x7f5218004730 comp rx=0 tx=0).stop 2023-10-14T17:00:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 shutdown_connections 2023-10-14T17:00:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f521006b0b0 0x7f521006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f523015eaf0 0x7f52300c8800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f523015e690 0x7f52300c82c0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 --2- 172.21.15.29:0/2862580963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f521c0047c0 0x7f5230160820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 >> 172.21.15.29:0/2862580963 conn(0x7f5230065ed0 msgr2=0x7f52300cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 shutdown_connections 2023-10-14T17:00:07.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.700+0000 7f522f59e640 1 -- 172.21.15.29:0/2862580963 wait complete. 2023-10-14T17:00:07.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:07.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.908+0000 7f5c3c932640 1 Processor -- start 2023-10-14T17:00:07.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.909+0000 7f5c3c932640 1 -- start start 2023-10-14T17:00:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.910+0000 7f5c3c932640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5c380cc2e0 0x7f5c380cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.910+0000 7f5c3c932640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380b3730 con 0x7f5c380b4740 2023-10-14T17:00:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.910+0000 7f5c3c932640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c380b3980 con 0x7f5c3815aa60 2023-10-14T17:00:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.910+0000 7f5c3c932640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c380b3c60 con 0x7f5c380cc2e0 2023-10-14T17:00:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.910+0000 7f5c3759e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5c380b4740 0x7f5c380b4bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42464/0 (socket says 172.21.15.29:42464) 2023-10-14T17:00:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c3759e640 1 -- 172.21.15.29:0/195267519 learned_addr learned my addr 172.21.15.29:0/195267519 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:07.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c3659c640 1 --2- 172.21.15.29:0/195267519 >> v2:172.21.15.186:3300/0 conn(0x7f5c380cc2e0 0x7f5c380cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c35d9b640 1 -- 172.21.15.29:0/195267519 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3798836823 0 0) 0x7f5c380b3730 con 0x7f5c380b4740 2023-10-14T17:00:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c35d9b640 1 -- 172.21.15.29:0/195267519 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c20003520 con 0x7f5c380b4740 2023-10-14T17:00:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c3659c640 1 -- 172.21.15.29:0/195267519 >> v1:172.21.15.186:6789/0 conn(0x7f5c3815aa60 legacy=0x7f5c380af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c3659c640 1 -- 172.21.15.29:0/195267519 >> v1:172.21.15.29:6789/0 conn(0x7f5c380b4740 legacy=0x7f5c380b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.911+0000 7f5c3659c640 1 -- 172.21.15.29:0/195267519 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c380b3f40 con 0x7f5c380cc2e0 2023-10-14T17:00:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.912+0000 7f5c3659c640 1 --2- 172.21.15.29:0/195267519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c380cc2e0 0x7f5c380cc750 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a639c29d9b4b91b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.913+0000 7f5c35d9b640 1 -- 172.21.15.29:0/195267519 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c030610 con 0x7f5c380cc2e0 2023-10-14T17:00:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.913+0000 7f5c35d9b640 1 -- 172.21.15.29:0/195267519 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c030770 con 0x7f5c380cc2e0 2023-10-14T17:00:07.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.913+0000 7f5c35d9b640 1 -- 172.21.15.29:0/195267519 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c030ae0 con 0x7f5c380cc2e0 2023-10-14T17:00:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 -- 172.21.15.29:0/195267519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c380cc2e0 msgr2=0x7f5c380cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 --2- 172.21.15.29:0/195267519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c380cc2e0 0x7f5c380cc750 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c02b960 comp rx=0 tx=0).stop 2023-10-14T17:00:07.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 -- 172.21.15.29:0/195267519 shutdown_connections 2023-10-14T17:00:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 --2- 172.21.15.29:0/195267519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c380cc2e0 0x7f5c380cc750 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 -- 172.21.15.29:0/195267519 >> 172.21.15.29:0/195267519 conn(0x7f5c38065ed0 msgr2=0x7f5c380662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:07.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 -- 172.21.15.29:0/195267519 shutdown_connections 2023-10-14T17:00:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.914+0000 7f5c3c932640 1 -- 172.21.15.29:0/195267519 wait complete. 2023-10-14T17:00:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.916+0000 7f5c3c932640 1 Processor -- start 2023-10-14T17:00:07.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.916+0000 7f5c3c932640 1 -- start start 2023-10-14T17:00:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 0x7f5c3814b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c3815aa60 0x7f5c38151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c67a0 con 0x7f5c380cc2e0 2023-10-14T17:00:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c6a80 con 0x7f5c3815aa60 2023-10-14T17:00:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3c932640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380c6d60 con 0x7f5c380b4740 2023-10-14T17:00:07.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.917+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36666/0 (socket says 172.21.15.29:36666) 2023-10-14T17:00:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 -- 172.21.15.29:0/3882160545 learned_addr learned my addr 172.21.15.29:0/3882160545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3759e640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c3815aa60 0x7f5c38151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c36d9d640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 0x7f5c3814b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:07.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 msgr2=0x7f5c3814b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 0x7f5c3814b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c3815aa60 msgr2=0x7f5c38151cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c3815aa60 0x7f5c38151cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 -- 172.21.15.29:0/3882160545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c2c009400 con 0x7f5c380cc2e0 2023-10-14T17:00:07.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c36d9d640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 0x7f5c3814b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.918+0000 7f5c3659c640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.919+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c040070 con 0x7f5c380cc2e0 2023-10-14T17:00:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.919+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c2c006e10 con 0x7f5c380cc2e0 2023-10-14T17:00:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.919+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2c030ea0 con 0x7f5c380cc2e0 2023-10-14T17:00:07.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.919+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c380c7040 con 0x7f5c380cc2e0 2023-10-14T17:00:07.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.919+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c380c5a50 con 0x7f5c380cc2e0 2023-10-14T17:00:07.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.920+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c3804e9b0 con 0x7f5c380cc2e0 2023-10-14T17:00:07.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.921+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7f5c2c003810 con 0x7f5c380cc2e0 2023-10-14T17:00:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.922+0000 7f5c1f7fe640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 0x7f5c2406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.923+0000 7f5c36d9d640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 msgr2=0x7f5c2406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:07.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.923+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7f5c2c0ab540 con 0x7f5c380cc2e0 2023-10-14T17:00:07.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.923+0000 7f5c36d9d640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 0x7f5c2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:07.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:07.927+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c2c076250 con 0x7f5c380cc2e0 2023-10-14T17:00:08.008 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:08.007+0000 7f83bfc2f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:08.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.118+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c3815df00 con 0x7f5c380cc2e0 2023-10-14T17:00:08.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.120+0000 7f5c1f7fe640 1 -- 172.21.15.29:0/3882160545 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+17077 (secure 0 0 0) 0x7f5c2c079870 con 0x7f5c380cc2e0 2023-10-14T17:00:08.122 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:08.122 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":126,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T16:59:41.541979+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"125","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6832/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:6833/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3986638800":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/354517400":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3653150355":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3890360705":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/2790183016":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3481988951":"2023-10-15T16:59:41.541911+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c36d9d640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 msgr2=0x7f5c2406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c36d9d640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 0x7f5c2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 msgr2=0x7f5c2406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 0x7f5c2406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 msgr2=0x7f5c38151780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 crypto rx=0x7f5c2c0097a0 tx=0x7f5c2c0046c0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 shutdown_connections 2023-10-14T17:00:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7f5c2406b0e0 0x7f5c2406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c3815aa60 0x7f5c38151cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c380cc2e0 0x7f5c38151780 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 --2- 172.21.15.29:0/3882160545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c380b4740 0x7f5c3814b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 >> 172.21.15.29:0/3882160545 conn(0x7f5c38065ed0 msgr2=0x7f5c3806dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 shutdown_connections 2023-10-14T17:00:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.124+0000 7f5c3c932640 1 -- 172.21.15.29:0/3882160545 wait complete. 2023-10-14T17:00:08.150 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 14, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T16:59:24.261633+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '124', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 15, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T16:59:25.325453+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '125', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:08.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:08.220 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.219+0000 7fb84642e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:08.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:08.302+0000 7f83bfc2f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.360+0000 7fe8cdd5e640 1 Processor -- start 2023-10-14T17:00:08.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.360+0000 7fe8cdd5e640 1 -- start start 2023-10-14T17:00:08.361 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:08.360+0000 7f83bfc2f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8cdd5e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8cdd5e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8c80cb810 0x7fe8c80cbc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8cdd5e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8c806c140 con 0x7fe8c8069da0 2023-10-14T17:00:08.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8cdd5e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8c815b2c0 con 0x7fe8c80cb810 2023-10-14T17:00:08.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8cdd5e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c806b270 con 0x7fe8c815aee0 2023-10-14T17:00:08.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8c6ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8c80cb810 0x7fe8c80cbc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8c6ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8c80cb810 0x7fe8c80cbc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:48276/0 (socket says 172.21.15.29:48276) 2023-10-14T17:00:08.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8c7fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe8c8069da0 0x7fe8c806a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:36430/0 (socket says 172.21.15.29:36430) 2023-10-14T17:00:08.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/1571714676 learned_addr learned my addr 172.21.15.29:0/1571714676 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:08.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.362+0000 7fe8c77fe640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.363+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 msgr2=0x7fe8c80aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.363+0000 7fe8c6ffd640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.363+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/1571714676 >> v1:172.21.15.186:6789/0 conn(0x7fe8c8069da0 legacy=0x7fe8c806a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.363+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/1571714676 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8c806daf0 con 0x7fe8c80cb810 2023-10-14T17:00:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.364+0000 7fe8c6ffd640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c80cb810 0x7fe8c80cbc70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009870 tx=0x7fe8b002b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=101d6346b24c180f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.365+0000 7fe8c67fc640 1 -- 172.21.15.29:0/1571714676 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b00317c0 con 0x7fe8c80cb810 2023-10-14T17:00:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.365+0000 7fe8c67fc640 1 -- 172.21.15.29:0/1571714676 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b0031920 con 0x7fe8c80cb810 2023-10-14T17:00:08.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.365+0000 7fe8c67fc640 1 -- 172.21.15.29:0/1571714676 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0031cb0 con 0x7fe8c80cb810 2023-10-14T17:00:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c80cb810 msgr2=0x7fe8c80cbc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c80cb810 0x7fe8c80cbc70 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fe8b0009870 tx=0x7fe8b002b860 comp rx=0 tx=0).stop 2023-10-14T17:00:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/1571714676 shutdown_connections 2023-10-14T17:00:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c80cb810 0x7fe8c80cbc70 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/1571714676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80aeef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/1571714676 >> 172.21.15.29:0/1571714676 conn(0x7fe8c80cfac0 msgr2=0x7fe8c80cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/1571714676 shutdown_connections 2023-10-14T17:00:08.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.366+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/1571714676 wait complete. 2023-10-14T17:00:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.369+0000 7fe8cdd5e640 1 Processor -- start 2023-10-14T17:00:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.369+0000 7fe8cdd5e640 1 -- start start 2023-10-14T17:00:08.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c8069da0 0x7fe8c814bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80c7580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c81544f0 con 0x7fe8c80cb810 2023-10-14T17:00:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80c94f0 con 0x7fe8c8069da0 2023-10-14T17:00:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8cdd5e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8c80c97d0 con 0x7fe8c815aee0 2023-10-14T17:00:08.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8c6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36676/0 (socket says 172.21.15.29:36676) 2023-10-14T17:00:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.370+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/3010851401 learned_addr learned my addr 172.21.15.29:0/3010851401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c77fe640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c8069da0 0x7fe8c814bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c7fff640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 msgr2=0x7fe8c80c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c8069da0 msgr2=0x7fe8c814bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c8069da0 0x7fe8c814bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 -- 172.21.15.29:0/3010851401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8b00094d0 con 0x7fe8c80cb810 2023-10-14T17:00:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.371+0000 7fe8c6ffd640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fe8b002b8a0 tx=0x7fe8b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.372+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0032860 con 0x7fe8c80cb810 2023-10-14T17:00:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.372+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8b00329c0 con 0x7fe8c80cb810 2023-10-14T17:00:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.372+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8b0032d70 con 0x7fe8c80cb810 2023-10-14T17:00:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.372+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8c8154180 con 0x7fe8c80cb810 2023-10-14T17:00:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.372+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8c80c91a0 con 0x7fe8c80cb810 2023-10-14T17:00:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.374+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8c8153fe0 con 0x7fe8c80cb810 2023-10-14T17:00:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.374+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 65) v1 ==== 89445+0+0 (secure 0 0 0) 0x7fe8b004e450 con 0x7fe8c80cb810 2023-10-14T17:00:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.375+0000 7fe8bffff640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 0x7fe8a806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.375+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 6097+0+0 (secure 0 0 0) 0x7fe8b00abd20 con 0x7fe8c80cb810 2023-10-14T17:00:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.375+0000 7fe8c77fe640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 msgr2=0x7fe8a806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.375+0000 7fe8c77fe640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 0x7fe8a806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.379+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe8c8153fe0 con 0x7fe8c80cb810 2023-10-14T17:00:08.477 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:08.476+0000 7f83bfc2f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:08.567 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.566+0000 7fb84642e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:08.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.576+0000 7fe8c77fe640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 msgr2=0x7fe8a806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1622969171 2023-10-14T17:00:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.576+0000 7fe8c77fe640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 0x7fe8a806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:08.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.591+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe8c8153fe0 con 0x7fe8c80cb810 2023-10-14T17:00:08.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.592+0000 7fe8bffff640 1 -- 172.21.15.29:0/3010851401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v126) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe8c8153fe0 con 0x7fe8c80cb810 2023-10-14T17:00:08.593 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:08.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.594+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 msgr2=0x7fe8a806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.594+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 0x7fe8a806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.594+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 msgr2=0x7fe8c8154980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.594+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 crypto rx=0x7fe8b002b8a0 tx=0x7fe8b00046c0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 shutdown_connections 2023-10-14T17:00:08.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:6832/1622969171,v1:172.21.15.29:6833/1622969171] conn(0x7fe8a806b0b0 0x7fe8a806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8c815aee0 0x7fe8c80c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8c80cb810 0x7fe8c8154980 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 --2- 172.21.15.29:0/3010851401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8c8069da0 0x7fe8c814bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 >> 172.21.15.29:0/3010851401 conn(0x7fe8c80cfac0 msgr2=0x7fe8c8158070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:08.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 shutdown_connections 2023-10-14T17:00:08.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.595+0000 7fe8cdd5e640 1 -- 172.21.15.29:0/3010851401 wait complete. 2023-10-14T17:00:08.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:00:08.629 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.629+0000 7fb84642e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:08.754 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.753+0000 7fb84642e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:08.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.821+0000 7f3c17190640 1 Processor -- start 2023-10-14T17:00:08.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.822+0000 7f3c17190640 1 -- start start 2023-10-14T17:00:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c17190640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3c100addb0 0x7f3c100caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c17190640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c1006a4b0 0x7f3c101650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c17190640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c100cb140 con 0x7f3c100c9000 2023-10-14T17:00:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c17190640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c10161a50 con 0x7f3c1006a4b0 2023-10-14T17:00:08.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c17190640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1006b030 con 0x7f3c100addb0 2023-10-14T17:00:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c1572d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c1006a4b0 0x7f3c101650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c1572d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c1006a4b0 0x7f3c101650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36722/0 (socket says 172.21.15.29:36722) 2023-10-14T17:00:08.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c1572d640 1 -- 172.21.15.29:0/3075801701 learned_addr learned my addr 172.21.15.29:0/3075801701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.823+0000 7f3c14f2c640 1 --2- 172.21.15.29:0/3075801701 >> v2:172.21.15.186:3300/0 conn(0x7f3c100addb0 0x7f3c100caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.824+0000 7f3c1572d640 1 -- 172.21.15.29:0/3075801701 >> v2:172.21.15.186:3300/0 conn(0x7f3c100addb0 msgr2=0x7f3c100caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.824+0000 7f3c1572d640 1 --2- 172.21.15.29:0/3075801701 >> v2:172.21.15.186:3300/0 conn(0x7f3c100addb0 0x7f3c100caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.824+0000 7f3c1572d640 1 -- 172.21.15.29:0/3075801701 >> v1:172.21.15.186:6789/0 conn(0x7f3c100c9000 legacy=0x7f3c10069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.824+0000 7f3c1572d640 1 -- 172.21.15.29:0/3075801701 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1006b310 con 0x7f3c1006a4b0 2023-10-14T17:00:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.825+0000 7f3c1572d640 1 --2- 172.21.15.29:0/3075801701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1006a4b0 0x7f3c101650d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f3c100b15a0 tx=0x7f3c0802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8864bbce793b4ca4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:08.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.826+0000 7f3c0f7fe640 1 -- 172.21.15.29:0/3075801701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0802ca80 con 0x7f3c1006a4b0 2023-10-14T17:00:08.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.826+0000 7f3c0f7fe640 1 -- 172.21.15.29:0/3075801701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0802cbe0 con 0x7f3c1006a4b0 2023-10-14T17:00:08.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.826+0000 7f3c0f7fe640 1 -- 172.21.15.29:0/3075801701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c080343f0 con 0x7f3c1006a4b0 2023-10-14T17:00:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 -- 172.21.15.29:0/3075801701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1006a4b0 msgr2=0x7f3c101650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 --2- 172.21.15.29:0/3075801701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1006a4b0 0x7f3c101650d0 secure :-1 s=READY pgs=1484 cs=0 l=1 rev1=1 crypto rx=0x7f3c100b15a0 tx=0x7f3c0802bee0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 -- 172.21.15.29:0/3075801701 shutdown_connections 2023-10-14T17:00:08.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 --2- 172.21.15.29:0/3075801701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c1006a4b0 0x7f3c101650d0 unknown :-1 s=CLOSED pgs=1484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 --2- 172.21.15.29:0/3075801701 >> v2:172.21.15.186:3300/0 conn(0x7f3c100addb0 0x7f3c100caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 -- 172.21.15.29:0/3075801701 >> 172.21.15.29:0/3075801701 conn(0x7f3c100ce9b0 msgr2=0x7f3c100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:08.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 -- 172.21.15.29:0/3075801701 shutdown_connections 2023-10-14T17:00:08.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.827+0000 7f3c17190640 1 -- 172.21.15.29:0/3075801701 wait complete. 2023-10-14T17:00:08.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.829+0000 7f3c17190640 1 Processor -- start 2023-10-14T17:00:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.829+0000 7f3c17190640 1 -- start start 2023-10-14T17:00:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c1006a4b0 0x7f3c100b15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c100addb0 0x7f3c100b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:08.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100b1ae0 con 0x7f3c100c9000 2023-10-14T17:00:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100b2740 con 0x7f3c100addb0 2023-10-14T17:00:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.830+0000 7f3c17190640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c100b1cf0 con 0x7f3c1006a4b0 2023-10-14T17:00:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36724/0 (socket says 172.21.15.29:36724) 2023-10-14T17:00:08.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 -- 172.21.15.29:0/2057410508 learned_addr learned my addr 172.21.15.29:0/2057410508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c0ffff640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c100addb0 0x7f3c100b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c14f2c640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c1006a4b0 0x7f3c100b15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 -- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c1006a4b0 msgr2=0x7f3c100b15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c1006a4b0 0x7f3c100b15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 -- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c100addb0 msgr2=0x7f3c100b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c100addb0 0x7f3c100b2200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 -- 172.21.15.29:0/2057410508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c08009b30 con 0x7f3c100c9000 2023-10-14T17:00:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.831+0000 7f3c1572d640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.832+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c08009510 con 0x7f3c100c9000 2023-10-14T17:00:08.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.832+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0803e450 con 0x7f3c100c9000 2023-10-14T17:00:08.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.832+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c0803e7e0 con 0x7f3c100c9000 2023-10-14T17:00:08.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.832+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c10157da0 con 0x7f3c100c9000 2023-10-14T17:00:08.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.832+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c100c5e90 con 0x7f3c100c9000 2023-10-14T17:00:08.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.833+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1004e160 con 0x7f3c100c9000 2023-10-14T17:00:08.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.834+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f3c0802cd50 con 0x7f3c100c9000 2023-10-14T17:00:08.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.835+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f3c080abfb0 con 0x7f3c100c9000 2023-10-14T17:00:08.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:08.839+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c08077090 con 0x7f3c100c9000 2023-10-14T17:00:08.861 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.860+0000 7fb7ca012640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.861 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.860+0000 7fb7ca012640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.861 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.860+0000 7fb7ca012640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.860+0000 7fb7ca012640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.860+0000 7fb7ca012640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.861+0000 7fb7c800e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.862 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.861+0000 7fb7c800e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.863 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.861+0000 7fb7c800e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.863 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.861+0000 7fb7c800e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:08.863 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:08.861+0000 7fb7c800e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:09.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.043+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3c1006c5b0 con 0x7f3c100c9000 2023-10-14T17:00:09.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.044+0000 7f3c0d7fa640 1 -- 172.21.15.29:0/2057410508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v127) v1 ==== 104+0+11 (secure 0 0 0) 0x7f3c0807a6b0 con 0x7f3c100c9000 2023-10-14T17:00:09.045 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:00:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 msgr2=0x7f3c10157860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 crypto rx=0x7f3c08036040 tx=0x7f3c08004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 shutdown_connections 2023-10-14T17:00:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c100c9000 0x7f3c10157860 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c100addb0 0x7f3c100b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.046+0000 7f3c17190640 1 --2- 172.21.15.29:0/2057410508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c1006a4b0 0x7f3c100b15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.047+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 >> 172.21.15.29:0/2057410508 conn(0x7f3c100ce9b0 msgr2=0x7f3c100ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.047+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 shutdown_connections 2023-10-14T17:00:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.047+0000 7f3c17190640 1 -- 172.21.15.29:0/2057410508 wait complete. 2023-10-14T17:00:09.065 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:00:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.340+0000 7f99cb17c640 1 Processor -- start 2023-10-14T17:00:09.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.340+0000 7f99cb17c640 1 -- start start 2023-10-14T17:00:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.342+0000 7f99cb17c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99c415d330 0x7f99c41618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.342+0000 7f99cb17c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f99c406c7d0 0x7f99c4065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.342+0000 7f99cb17c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99c40d0890 con 0x7f99c406cc80 2023-10-14T17:00:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.342+0000 7f99cb17c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c406b1e0 con 0x7f99c415d330 2023-10-14T17:00:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.342+0000 7f99cb17c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99c40d05b0 con 0x7f99c406c7d0 2023-10-14T17:00:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99c415d330 0x7f99c41618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99c415d330 0x7f99c41618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36800/0 (socket says 172.21.15.29:36800) 2023-10-14T17:00:09.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 -- 172.21.15.29:0/1043816521 learned_addr learned my addr 172.21.15.29:0/1043816521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c3fff640 1 --2- 172.21.15.29:0/1043816521 >> v2:172.21.15.186:3300/0 conn(0x7f99c406c7d0 0x7f99c4065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 -- 172.21.15.29:0/1043816521 >> v2:172.21.15.186:3300/0 conn(0x7f99c406c7d0 msgr2=0x7f99c4065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 --2- 172.21.15.29:0/1043816521 >> v2:172.21.15.186:3300/0 conn(0x7f99c406c7d0 0x7f99c4065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 -- 172.21.15.29:0/1043816521 >> v1:172.21.15.186:6789/0 conn(0x7f99c406cc80 legacy=0x7f99c40ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:09.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.343+0000 7f99c8f18640 1 -- 172.21.15.29:0/1043816521 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99c4160dd0 con 0x7f99c415d330 2023-10-14T17:00:09.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.344+0000 7f99c8f18640 1 --2- 172.21.15.29:0/1043816521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41618b0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f99c40b6bc0 tx=0x7f99b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aad92262ff5f560e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:09.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99c37fe640 1 -- 172.21.15.29:0/1043816521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b80317e0 con 0x7f99c415d330 2023-10-14T17:00:09.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99c37fe640 1 -- 172.21.15.29:0/1043816521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99b8031940 con 0x7f99c415d330 2023-10-14T17:00:09.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99c37fe640 1 -- 172.21.15.29:0/1043816521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99b8031cd0 con 0x7f99c415d330 2023-10-14T17:00:09.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 -- 172.21.15.29:0/1043816521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 msgr2=0x7f99c41618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 --2- 172.21.15.29:0/1043816521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41618b0 secure :-1 s=READY pgs=1492 cs=0 l=1 rev1=1 crypto rx=0x7f99c40b6bc0 tx=0x7f99b802b880 comp rx=0 tx=0).stop 2023-10-14T17:00:09.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 -- 172.21.15.29:0/1043816521 shutdown_connections 2023-10-14T17:00:09.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 --2- 172.21.15.29:0/1043816521 >> v2:172.21.15.186:3300/0 conn(0x7f99c406c7d0 0x7f99c4065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 --2- 172.21.15.29:0/1043816521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41618b0 unknown :-1 s=CLOSED pgs=1492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 -- 172.21.15.29:0/1043816521 >> 172.21.15.29:0/1043816521 conn(0x7f99c40d0ab0 msgr2=0x7f99c40d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:09.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.345+0000 7f99cb17c640 1 -- 172.21.15.29:0/1043816521 shutdown_connections 2023-10-14T17:00:09.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.346+0000 7f99cb17c640 1 -- 172.21.15.29:0/1043816521 wait complete. 2023-10-14T17:00:09.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.347+0000 7f99cb17c640 1 Processor -- start 2023-10-14T17:00:09.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.347+0000 7f99cb17c640 1 -- start start 2023-10-14T17:00:09.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99c406c7d0 0x7f99c4155240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99c406cc80 0x7f99c40ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40c6f60 con 0x7f99c415d330 2023-10-14T17:00:09.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40b17f0 con 0x7f99c406cc80 2023-10-14T17:00:09.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.348+0000 7f99cb17c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99c40b1ad0 con 0x7f99c406c7d0 2023-10-14T17:00:09.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36808/0 (socket says 172.21.15.29:36808) 2023-10-14T17:00:09.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 -- 172.21.15.29:0/2997646011 learned_addr learned my addr 172.21.15.29:0/2997646011 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:09.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c8f18640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99c406c7d0 0x7f99c4155240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c3fff640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99c406cc80 0x7f99c40ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 -- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99c406c7d0 msgr2=0x7f99c4155240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99c406c7d0 0x7f99c4155240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 -- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99c406cc80 msgr2=0x7f99c40ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99c406cc80 0x7f99c40ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.349+0000 7f99c9719640 1 -- 172.21.15.29:0/2997646011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80094d0 con 0x7f99c415d330 2023-10-14T17:00:09.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99c9719640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00b970 tx=0x7f99bc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:09.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc00cb40 con 0x7f99c415d330 2023-10-14T17:00:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99bc00cca0 con 0x7f99c415d330 2023-10-14T17:00:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99bc014680 con 0x7f99c415d330 2023-10-14T17:00:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99c40c46d0 con 0x7f99c415d330 2023-10-14T17:00:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.350+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99c40ca650 con 0x7f99c415d330 2023-10-14T17:00:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.351+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99c40c4bf0 con 0x7f99c415d330 2023-10-14T17:00:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.352+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 66) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f99bc01d3b0 con 0x7f99c415d330 2023-10-14T17:00:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.353+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 6527+0+0 (secure 0 0 0) 0x7f99bc0891b0 con 0x7f99c415d330 2023-10-14T17:00:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.358+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99c40c4bf0 con 0x7f99c415d330 2023-10-14T17:00:09.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.554+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99c40c4bf0 con 0x7f99c415d330 2023-10-14T17:00:09.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.555+0000 7f99c17fa640 1 -- 172.21.15.29:0/2997646011 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v127) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99c40c4bf0 con 0x7f99c415d330 2023-10-14T17:00:09.556 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:00:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 msgr2=0x7f99c41f28d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 crypto rx=0x7f99bc00b970 tx=0x7f99bc00be30 comp rx=0 tx=0).stop 2023-10-14T17:00:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 shutdown_connections 2023-10-14T17:00:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99c415d330 0x7f99c41f28d0 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99c406cc80 0x7f99c40ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 --2- 172.21.15.29:0/2997646011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99c406c7d0 0x7f99c4155240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 >> 172.21.15.29:0/2997646011 conn(0x7f99c40d0ab0 msgr2=0x7f99c415e550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 shutdown_connections 2023-10-14T17:00:09.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.558+0000 7f99cb17c640 1 -- 172.21.15.29:0/2997646011 wait complete. 2023-10-14T17:00:09.579 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:09.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.813+0000 7f627a43f640 1 Processor -- start 2023-10-14T17:00:09.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.813+0000 7f627a43f640 1 -- start start 2023-10-14T17:00:09.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f627a43f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f627406c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f627a43f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f627415e0c0 0x7f62740b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f627a43f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62740cd630 con 0x7f627415e540 2023-10-14T17:00:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f627a43f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62740ce1b0 con 0x7f627415e0c0 2023-10-14T17:00:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f627a43f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f627406cd10 con 0x7f627406c3f0 2023-10-14T17:00:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f62737fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f627415e0c0 0x7f62740b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f6273fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f627406c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f62737fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f627415e0c0 0x7f62740b4730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36820/0 (socket says 172.21.15.29:36820) 2023-10-14T17:00:09.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f62737fe640 1 -- 172.21.15.29:0/2703532231 learned_addr learned my addr 172.21.15.29:0/2703532231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:09.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.816+0000 7f62789dc640 1 --1- 172.21.15.29:0/2703532231 >> v1:172.21.15.29:6789/0 conn(0x7f627415e540 0x7f62740b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42476/0 (socket says 172.21.15.29:42476) 2023-10-14T17:00:09.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.817+0000 7f62737fe640 1 -- 172.21.15.29:0/2703532231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 msgr2=0x7f627406c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:09.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.817+0000 7f62737fe640 1 --2- 172.21.15.29:0/2703532231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f627406c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.817+0000 7f62737fe640 1 -- 172.21.15.29:0/2703532231 >> v1:172.21.15.29:6789/0 conn(0x7f627415e540 legacy=0x7f62740b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.817+0000 7f62737fe640 1 -- 172.21.15.29:0/2703532231 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62740d2170 con 0x7f627415e0c0 2023-10-14T17:00:09.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.818+0000 7f62737fe640 1 --2- 172.21.15.29:0/2703532231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f62740b4730 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b23df9f83cd72f5b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:09.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.818+0000 7f6272ffd640 1 -- 172.21.15.29:0/2703532231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030840 con 0x7f627415e0c0 2023-10-14T17:00:09.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.818+0000 7f6272ffd640 1 -- 172.21.15.29:0/2703532231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62680309a0 con 0x7f627415e0c0 2023-10-14T17:00:09.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.818+0000 7f6272ffd640 1 -- 172.21.15.29:0/2703532231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268030d30 con 0x7f627415e0c0 2023-10-14T17:00:09.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 -- 172.21.15.29:0/2703532231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 msgr2=0x7f62740b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 --2- 172.21.15.29:0/2703532231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f62740b4730 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 crypto rx=0x7f6268009870 tx=0x7f626802b980 comp rx=0 tx=0).stop 2023-10-14T17:00:09.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 -- 172.21.15.29:0/2703532231 shutdown_connections 2023-10-14T17:00:09.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 --2- 172.21.15.29:0/2703532231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f62740b4730 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 --2- 172.21.15.29:0/2703532231 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f627406c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 -- 172.21.15.29:0/2703532231 >> 172.21.15.29:0/2703532231 conn(0x7f62740d2cb0 msgr2=0x7f62740d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:09.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 -- 172.21.15.29:0/2703532231 shutdown_connections 2023-10-14T17:00:09.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.819+0000 7f627a43f640 1 -- 172.21.15.29:0/2703532231 wait complete. 2023-10-14T17:00:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.821+0000 7f627a43f640 1 Processor -- start 2023-10-14T17:00:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.821+0000 7f627a43f640 1 -- start start 2023-10-14T17:00:09.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f62741f3220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 0x7f627415bff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62741f3760 con 0x7f627415e0c0 2023-10-14T17:00:09.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62741552d0 con 0x7f627415e540 2023-10-14T17:00:09.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.822+0000 7f627a43f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62740c7b40 con 0x7f627406c3f0 2023-10-14T17:00:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36824/0 (socket says 172.21.15.29:36824) 2023-10-14T17:00:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 -- 172.21.15.29:0/3107388893 learned_addr learned my addr 172.21.15.29:0/3107388893 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f6273fff640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f62741f3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62789dc640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 0x7f627415bff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 -- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 msgr2=0x7f62741f3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f62741f3220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 -- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 msgr2=0x7f627415bff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 0x7f627415bff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:09.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 -- 172.21.15.29:0/3107388893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62680094d0 con 0x7f627415e0c0 2023-10-14T17:00:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f6273fff640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f62741f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62789dc640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 0x7f627415bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.823+0000 7f62737fe640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6268000c00 tx=0x7f6268004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.824+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268039c90 con 0x7f627415e0c0 2023-10-14T17:00:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.824+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6268039df0 con 0x7f627415e0c0 2023-10-14T17:00:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.824+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6268043640 con 0x7f627415e0c0 2023-10-14T17:00:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.824+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62740c7e20 con 0x7f627415e0c0 2023-10-14T17:00:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.824+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62740c83e0 con 0x7f627415e0c0 2023-10-14T17:00:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.825+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6274159420 con 0x7f627415e0c0 2023-10-14T17:00:09.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.826+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f62680437a0 con 0x7f627415e0c0 2023-10-14T17:00:09.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.827+0000 7f6270ff9640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 0x7f626006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:09.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.827+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7f626803e070 con 0x7f627415e0c0 2023-10-14T17:00:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.827+0000 7f6273fff640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 0x7f626006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.828+0000 7f6273fff640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 0x7f626006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62740b8b90 tx=0x7f6264008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:09.832+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6274159420 con 0x7f627415e0c0 2023-10-14T17:00:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.086+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6274159420 con 0x7f627415e0c0 2023-10-14T17:00:10.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.087+0000 7f6270ff9640 1 -- 172.21.15.29:0/3107388893 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f6274159420 con 0x7f627415e0c0 2023-10-14T17:00:10.087 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:10.088 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T17:00:10.088 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T17:00:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 msgr2=0x7f626006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 0x7f626006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f62740b8b90 tx=0x7f6264008040 comp rx=0 tx=0).stop 2023-10-14T17:00:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 msgr2=0x7f6274154d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 crypto rx=0x7f6268000c00 tx=0x7f6268004730 comp rx=0 tx=0).stop 2023-10-14T17:00:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 shutdown_connections 2023-10-14T17:00:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f626006acc0 0x7f626006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f627415e540 0x7f627415bff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f627415e0c0 0x7f6274154d90 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 --2- 172.21.15.29:0/3107388893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f627406c3f0 0x7f62741f3220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 >> 172.21.15.29:0/3107388893 conn(0x7f62740d2cb0 msgr2=0x7f62740b81f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 shutdown_connections 2023-10-14T17:00:10.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.089+0000 7f627a43f640 1 -- 172.21.15.29:0/3107388893 wait complete. 2023-10-14T17:00:10.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:10.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.314+0000 7feadffff640 1 Processor -- start 2023-10-14T17:00:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.314+0000 7feadffff640 1 -- start start 2023-10-14T17:00:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feae0167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00ae1a0 con 0x7feae0160c80 2023-10-14T17:00:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feae00ae480 con 0x7feae00ad7a0 2023-10-14T17:00:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ae760 con 0x7feae0070950 2023-10-14T17:00:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feae0160c80 0x7feae0161060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42488/0 (socket says 172.21.15.29:42488) 2023-10-14T17:00:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.315+0000 7feadeffd640 1 -- 172.21.15.29:0/2153396546 learned_addr learned my addr 172.21.15.29:0/2153396546 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381835671 0 0) 0x7feae00ae1a0 con 0x7feae0160c80 2023-10-14T17:00:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feacc003520 con 0x7feae0160c80 2023-10-14T17:00:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feadf7fe640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feae0167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1397490281 0 0) 0x7feacc003520 con 0x7feae0160c80 2023-10-14T17:00:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 msgr2=0x7feae0167b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feae0167b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 >> v1:172.21.15.186:6789/0 conn(0x7feae00ad7a0 legacy=0x7feae0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae00aea40 con 0x7feae0160c80 2023-10-14T17:00:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fead00019f0 con 0x7feae0160c80 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.316+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 >> v1:172.21.15.29:6789/0 conn(0x7feae0160c80 legacy=0x7feae0161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc0039a0 0x7feacc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0047c0 0x7feacc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feacc005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae00ae760 con 0x7feacc0047c0 2023-10-14T17:00:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feacc00bb40 con 0x7feacc0039a0 2023-10-14T17:00:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feacc00bd00 con 0x7feae0070950 2023-10-14T17:00:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0047c0 0x7feacc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feadf7fe640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feacc005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feadeffd640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc0039a0 0x7feacc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 -- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 msgr2=0x7feacc005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feae0070950 0x7feacc005210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 -- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc0039a0 msgr2=0x7feacc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc0039a0 0x7feacc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feade7fc640 1 -- 172.21.15.29:0/2153396546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feacc003520 con 0x7feacc0047c0 2023-10-14T17:00:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.317+0000 7feadeffd640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feacc0039a0 0x7feacc003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.319+0000 7feade7fc640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0047c0 0x7feacc004c40 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7feac8001750 tx=0x7feacc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45fc3ca6cdf40a03 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.319+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac80052d0 con 0x7feacc0047c0 2023-10-14T17:00:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.319+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feac802e050 con 0x7feacc0047c0 2023-10-14T17:00:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.319+0000 7feaddffb640 1 -- 172.21.15.29:0/2153396546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feac80035f0 con 0x7feacc0047c0 2023-10-14T17:00:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 -- 172.21.15.29:0/2153396546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0047c0 msgr2=0x7feacc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 --2- 172.21.15.29:0/2153396546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feacc0047c0 0x7feacc004c40 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 crypto rx=0x7feac8001750 tx=0x7feacc0057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadf7fe640 1 -- 172.21.15.29:0/2153396546 reap_dead start 2023-10-14T17:00:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 -- 172.21.15.29:0/2153396546 shutdown_connections 2023-10-14T17:00:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 -- 172.21.15.29:0/2153396546 >> 172.21.15.29:0/2153396546 conn(0x7feae00cc230 msgr2=0x7feae00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 -- 172.21.15.29:0/2153396546 shutdown_connections 2023-10-14T17:00:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.320+0000 7feadffff640 1 -- 172.21.15.29:0/2153396546 wait complete. 2023-10-14T17:00:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.322+0000 7feadffff640 1 Processor -- start 2023-10-14T17:00:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.322+0000 7feadffff640 1 -- start start 2023-10-14T17:00:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feacc0047c0 0x7feae015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feae006b7d0 0x7feae006bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae0160010 con 0x7feae015f670 2023-10-14T17:00:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae006c250 con 0x7feae006b7d0 2023-10-14T17:00:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feae006c530 con 0x7feacc0047c0 2023-10-14T17:00:10.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36840/0 (socket says 172.21.15.29:36840) 2023-10-14T17:00:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadf7fe640 1 -- 172.21.15.29:0/902737191 learned_addr learned my addr 172.21.15.29:0/902737191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.323+0000 7feadeffd640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feacc0047c0 0x7feae015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feade7fc640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feae006b7d0 0x7feae006bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 -- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feacc0047c0 msgr2=0x7feae015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feacc0047c0 0x7feae015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 -- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feae006b7d0 msgr2=0x7feae006bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feae006b7d0 0x7feae006bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 -- 172.21.15.29:0/902737191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae006c810 con 0x7feae015f670 2023-10-14T17:00:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7feadf7fe640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7feae00cf780 tx=0x7fead800f6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead8004440 con 0x7feae015f670 2023-10-14T17:00:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fead8003610 con 0x7feae015f670 2023-10-14T17:00:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.324+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fead80039c0 con 0x7feae015f670 2023-10-14T17:00:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.325+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae006caf0 con 0x7feae015f670 2023-10-14T17:00:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.325+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae015ba40 con 0x7feae015f670 2023-10-14T17:00:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.326+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae006d010 con 0x7feae015f670 2023-10-14T17:00:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.327+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fead8003b20 con 0x7feae015f670 2023-10-14T17:00:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.328+0000 7fead77fe640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 0x7feac006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.328+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fead8089580 con 0x7feae015f670 2023-10-14T17:00:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.329+0000 7feadeffd640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 0x7feac006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.329+0000 7feadeffd640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 0x7feac006d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fead0000f80 tx=0x7fead0009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.333+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feae006d010 con 0x7feae015f670 2023-10-14T17:00:10.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.536+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feae006d010 con 0x7feae015f670 2023-10-14T17:00:10.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.539+0000 7fead77fe640 1 -- 172.21.15.29:0/902737191 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17813 (secure 0 0 0) 0x7feae006d010 con 0x7feae015f670 2023-10-14T17:00:10.540 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:10.540 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":128,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:09.764614+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1686923517":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3236529658":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/681365668":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/2259811471":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6833/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3200714056":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3964845078":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:6833/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3986638800":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/354517400":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3653150355":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/226673613":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3890360705":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/2790183016":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/1217266964":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3481988951":"2023-10-15T16:59:41.541911+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:10.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 msgr2=0x7feac006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 0x7feac006d1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fead0000f80 tx=0x7fead0009040 comp rx=0 tx=0).stop 2023-10-14T17:00:10.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 msgr2=0x7feae015fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 secure :-1 s=READY pgs=1499 cs=0 l=1 rev1=1 crypto rx=0x7feae00cf780 tx=0x7fead800f6f0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 shutdown_connections 2023-10-14T17:00:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7feac006ad10 0x7feac006d1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feae015f670 0x7feae015fad0 unknown :-1 s=CLOSED pgs=1499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feae006b7d0 0x7feae006bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 --2- 172.21.15.29:0/902737191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feacc0047c0 0x7feae015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 >> 172.21.15.29:0/902737191 conn(0x7feae00cc230 msgr2=0x7feae015c330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 shutdown_connections 2023-10-14T17:00:10.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.542+0000 7feadffff640 1 -- 172.21.15.29:0/902737191 wait complete. 2023-10-14T17:00:10.563 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T17:00:10.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:10.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.796+0000 7f6ac2978640 1 Processor -- start 2023-10-14T17:00:10.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.796+0000 7f6ac2978640 1 -- start start 2023-10-14T17:00:10.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.797+0000 7f6ac2978640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6abc0cfc10 0x7f6abc0d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6ac2978640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0b3940 con 0x7f6abc0b3070 2023-10-14T17:00:10.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6ac2978640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc0b45e0 con 0x7f6abc0b3e70 2023-10-14T17:00:10.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6ac2978640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc0c7880 con 0x7f6abc0cfc10 2023-10-14T17:00:10.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6abbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6abc0b3070 0x7f6abc0cf500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42490/0 (socket says 172.21.15.29:42490) 2023-10-14T17:00:10.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6abbfff640 1 -- 172.21.15.29:0/3576707946 learned_addr learned my addr 172.21.15.29:0/3576707946 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:10.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3543563187 0 0) 0x7f6abc0b3940 con 0x7f6abc0b3070 2023-10-14T17:00:10.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6aa4003520 con 0x7f6abc0b3070 2023-10-14T17:00:10.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.798+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/3576707946 >> v2:172.21.15.186:3300/0 conn(0x7f6abc0cfc10 0x7f6abc0d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4159052345 0 0) 0x7f6aa4003520 con 0x7f6abc0b3070 2023-10-14T17:00:10.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 >> v2:172.21.15.186:3300/0 conn(0x7f6abc0cfc10 msgr2=0x7f6abc0d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 --2- 172.21.15.29:0/3576707946 >> v2:172.21.15.186:3300/0 conn(0x7f6abc0cfc10 0x7f6abc0d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 >> v1:172.21.15.186:6789/0 conn(0x7f6abc0b3e70 legacy=0x7f6abc0d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0b3b90 con 0x7f6abc0b3070 2023-10-14T17:00:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6aac0017f0 con 0x7f6abc0b3070 2023-10-14T17:00:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 >> v1:172.21.15.29:6789/0 conn(0x7f6abc0b3070 legacy=0x7f6abc0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa4003970 0x7f6aa4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 0x7f6aa4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4005180 0x7f6aa400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0c7880 con 0x7f6aa40047c0 2023-10-14T17:00:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa400c0d0 con 0x7f6aa4005180 2023-10-14T17:00:10.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa400c290 con 0x7f6aa4003970 2023-10-14T17:00:10.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.799+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 0x7f6aa4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abbfff640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa4003970 0x7f6aa4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6ac0f15640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4005180 0x7f6aa400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abb7fe640 1 -- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa4003970 msgr2=0x7f6aa4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa4003970 0x7f6aa4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abb7fe640 1 -- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4005180 msgr2=0x7f6aa400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6aa4005180 0x7f6aa400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6ac0f15640 1 -- 172.21.15.29:0/3576707946 reap_dead start 2023-10-14T17:00:10.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.800+0000 7f6abb7fe640 1 -- 172.21.15.29:0/3576707946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa4003520 con 0x7f6aa40047c0 2023-10-14T17:00:10.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.801+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 0x7f6aa4004c40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ba30 tx=0x7f6aa40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e5775e0e6c5dd0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.801+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0032640 con 0x7f6aa40047c0 2023-10-14T17:00:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.801+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab00327a0 con 0x7f6aa40047c0 2023-10-14T17:00:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.801+0000 7f6abaffd640 1 -- 172.21.15.29:0/3576707946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab0032b30 con 0x7f6aa40047c0 2023-10-14T17:00:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 -- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 msgr2=0x7f6aa4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 0x7f6aa4004c40 secure :-1 s=READY pgs=1501 cs=0 l=1 rev1=1 crypto rx=0x7f6ab000ba30 tx=0x7f6aa40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 -- 172.21.15.29:0/3576707946 shutdown_connections 2023-10-14T17:00:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 --2- 172.21.15.29:0/3576707946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6aa40047c0 0x7f6aa4004c40 unknown :-1 s=CLOSED pgs=1501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 -- 172.21.15.29:0/3576707946 >> 172.21.15.29:0/3576707946 conn(0x7f6abc0d2d20 msgr2=0x7f6abc0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 -- 172.21.15.29:0/3576707946 shutdown_connections 2023-10-14T17:00:10.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.802+0000 7f6ac2978640 1 -- 172.21.15.29:0/3576707946 wait complete. 2023-10-14T17:00:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.804+0000 7f6ac2978640 1 Processor -- start 2023-10-14T17:00:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.804+0000 7f6ac2978640 1 -- start start 2023-10-14T17:00:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa40047c0 0x7f6abc0af8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 0x7f6abc0b6a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc06be80 con 0x7f6abc06b4c0 2023-10-14T17:00:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0afed0 con 0x7f6abc1541d0 2023-10-14T17:00:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac2978640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0b01b0 con 0x7f6aa40047c0 2023-10-14T17:00:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.805+0000 7f6ac0f15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36848/0 (socket says 172.21.15.29:36848) 2023-10-14T17:00:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 -- 172.21.15.29:0/765343842 learned_addr learned my addr 172.21.15.29:0/765343842 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6abbfff640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa40047c0 0x7f6abc0af8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 0x7f6abc0b6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 -- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa40047c0 msgr2=0x7f6abc0af8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa40047c0 0x7f6abc0af8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 -- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 msgr2=0x7f6abc0b6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 0x7f6abc0b6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 -- 172.21.15.29:0/765343842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc0b0490 con 0x7f6abc06b4c0 2023-10-14T17:00:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6abb7fe640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 0x7f6abc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.806+0000 7f6ac0f15640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0cd300 tx=0x7f6ab400d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.807+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab4003d80 con 0x7f6abc06b4c0 2023-10-14T17:00:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.807+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab4005070 con 0x7f6abc06b4c0 2023-10-14T17:00:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.807+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40053e0 con 0x7f6abc06b4c0 2023-10-14T17:00:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.807+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc0b0770 con 0x7f6abc06b4c0 2023-10-14T17:00:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.807+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc0b6f50 con 0x7f6abc06b4c0 2023-10-14T17:00:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.808+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0b0d10 con 0x7f6abc06b4c0 2023-10-14T17:00:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.809+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6ab4005540 con 0x7f6abc06b4c0 2023-10-14T17:00:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.810+0000 7f6ab8ff9640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 0x7f6aa806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.810+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7f6ab4088df0 con 0x7f6abc06b4c0 2023-10-14T17:00:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.811+0000 7f6abbfff640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 0x7f6aa806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.811+0000 7f6abbfff640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 0x7f6aa806d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0011d0 tx=0x7f6aac009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:10.815+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6abc0b0d10 con 0x7f6abc06b4c0 2023-10-14T17:00:11.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.058+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6abc0b0d10 con 0x7f6abc06b4c0 2023-10-14T17:00:11.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.059+0000 7f6ab8ff9640 1 -- 172.21.15.29:0/765343842 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7f6abc0b0d10 con 0x7f6abc06b4c0 2023-10-14T17:00:11.060 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:11.060 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T17:00:11.061 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T17:00:11.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 msgr2=0x7f6aa806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 0x7f6aa806d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6aac0011d0 tx=0x7f6aac009040 comp rx=0 tx=0).stop 2023-10-14T17:00:11.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 msgr2=0x7f6abc06b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0cd300 tx=0x7f6ab400d9d0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 shutdown_connections 2023-10-14T17:00:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6aa806acf0 0x7f6aa806d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6abc06b4c0 0x7f6abc06b940 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6abc1541d0 0x7f6abc0b6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 --2- 172.21.15.29:0/765343842 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6aa40047c0 0x7f6abc0af8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 >> 172.21.15.29:0/765343842 conn(0x7f6abc0d2d20 msgr2=0x7f6abc06a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 shutdown_connections 2023-10-14T17:00:11.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.062+0000 7f6ac2978640 1 -- 172.21.15.29:0/765343842 wait complete. 2023-10-14T17:00:11.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:11.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.310+0000 7fe477fff640 1 Processor -- start 2023-10-14T17:00:11.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.310+0000 7fe477fff640 1 -- start start 2023-10-14T17:00:11.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.311+0000 7fe477fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe477fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47815aef0 con 0x7fe478070450 2023-10-14T17:00:11.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe477fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47815d910 con 0x7fe47806c810 2023-10-14T17:00:11.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe477fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe47815e490 con 0x7fe478159950 2023-10-14T17:00:11.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe4767fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe478070450 0x7fe47806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46526/0 (socket says 172.21.15.29:46526) 2023-10-14T17:00:11.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe4767fc640 1 -- 172.21.15.29:0/3384919901 learned_addr learned my addr 172.21.15.29:0/3384919901 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe476ffd640 1 --2- 172.21.15.29:0/3384919901 >> v2:172.21.15.29:3300/0 conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.312+0000 7fe475ffb640 1 -- 172.21.15.29:0/3384919901 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482586679 0 0) 0x7fe460001110 con 0x7fe478070450 2023-10-14T17:00:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.313+0000 7fe475ffb640 1 -- 172.21.15.29:0/3384919901 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe464003520 con 0x7fe478070450 2023-10-14T17:00:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.313+0000 7fe476ffd640 1 -- 172.21.15.29:0/3384919901 >> v1:172.21.15.186:6789/0 conn(0x7fe47806c810 legacy=0x7fe4780c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.313+0000 7fe476ffd640 1 -- 172.21.15.29:0/3384919901 >> v1:172.21.15.29:6789/0 conn(0x7fe478070450 legacy=0x7fe47806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.313+0000 7fe476ffd640 1 -- 172.21.15.29:0/3384919901 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47815dbf0 con 0x7fe478159950 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.314+0000 7fe476ffd640 1 --2- 172.21.15.29:0/3384919901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47815a9b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe468009d90 tx=0x7fe46802be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1861c1c1126b74f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.314+0000 7fe475ffb640 1 -- 172.21.15.29:0/3384919901 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46802cbd0 con 0x7fe478159950 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.314+0000 7fe475ffb640 1 -- 172.21.15.29:0/3384919901 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46802cd30 con 0x7fe478159950 2023-10-14T17:00:11.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.314+0000 7fe475ffb640 1 -- 172.21.15.29:0/3384919901 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe46802cea0 con 0x7fe478159950 2023-10-14T17:00:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 -- 172.21.15.29:0/3384919901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 msgr2=0x7fe47815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 --2- 172.21.15.29:0/3384919901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47815a9b0 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 crypto rx=0x7fe468009d90 tx=0x7fe46802be20 comp rx=0 tx=0).stop 2023-10-14T17:00:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 -- 172.21.15.29:0/3384919901 shutdown_connections 2023-10-14T17:00:11.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 --2- 172.21.15.29:0/3384919901 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47815a9b0 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 -- 172.21.15.29:0/3384919901 >> 172.21.15.29:0/3384919901 conn(0x7fe4780ce6a0 msgr2=0x7fe4780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 -- 172.21.15.29:0/3384919901 shutdown_connections 2023-10-14T17:00:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.315+0000 7fe477fff640 1 -- 172.21.15.29:0/3384919901 wait complete. 2023-10-14T17:00:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.317+0000 7fe477fff640 1 Processor -- start 2023-10-14T17:00:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.317+0000 7fe477fff640 1 -- start start 2023-10-14T17:00:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe47806c810 0x7fe4780b3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe478070450 0x7fe47806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c5770 con 0x7fe478159950 2023-10-14T17:00:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c5d30 con 0x7fe47806c810 2023-10-14T17:00:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe477fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4780c3ef0 con 0x7fe478070450 2023-10-14T17:00:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54730/0 (socket says 172.21.15.29:54730) 2023-10-14T17:00:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 -- 172.21.15.29:0/4055038256 learned_addr learned my addr 172.21.15.29:0/4055038256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 -- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe478070450 msgr2=0x7fe47806ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe478070450 0x7fe47806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 -- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe47806c810 msgr2=0x7fe4780b3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe47806c810 0x7fe4780b3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.318+0000 7fe4777fe640 1 -- 172.21.15.29:0/4055038256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe468009a70 con 0x7fe478159950 2023-10-14T17:00:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe4777fe640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe47000c440 tx=0x7fe47000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470011790 con 0x7fe478159950 2023-10-14T17:00:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4700118f0 con 0x7fe478159950 2023-10-14T17:00:11.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe470011ca0 con 0x7fe478159950 2023-10-14T17:00:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4780c41d0 con 0x7fe478159950 2023-10-14T17:00:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.319+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4780b3f40 con 0x7fe478159950 2023-10-14T17:00:11.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.320+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4780b4100 con 0x7fe478159950 2023-10-14T17:00:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.321+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 67) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fe47001c460 con 0x7fe478159950 2023-10-14T17:00:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.322+0000 7fe46f7fe640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 0x7fe45806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.322+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fe470089460 con 0x7fe478159950 2023-10-14T17:00:11.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.323+0000 7fe476ffd640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 0x7fe45806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.323+0000 7fe476ffd640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 0x7fe45806d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe468036000 tx=0x7fe46802e000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.327+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4700544e0 con 0x7fe478159950 2023-10-14T17:00:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.357+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe4700548b0 con 0x7fe478159950 2023-10-14T17:00:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.568+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4780af940 con 0x7fe478159950 2023-10-14T17:00:11.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.570+0000 7fe46f7fe640 1 -- 172.21.15.29:0/4055038256 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+4915 (secure 0 0 0) 0x7fe470070e10 con 0x7fe478159950 2023-10-14T17:00:11.570 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:11.571 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":57,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52}],"filesystems":[{"mdsmap":{"epoch":57,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T16:59:26.282689+0000","modified":"2023-10-14T16:59:29.375892+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8037},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8037":{"gid":8037,"name":"c","rank":0,"incarnation":53,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/80282461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":80282461},{"type":"v1","addr":"172.21.15.29:6837","nonce":80282461}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[15],"metadata_pool":14,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":7}]} 2023-10-14T17:00:11.571 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 57 2023-10-14T17:00:11.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.572+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 msgr2=0x7fe45806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.572+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 0x7fe45806d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fe468036000 tx=0x7fe46802e000 comp rx=0 tx=0).stop 2023-10-14T17:00:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.572+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 msgr2=0x7fe47806b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.572+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 secure :-1 s=READY pgs=1507 cs=0 l=1 rev1=1 crypto rx=0x7fe47000c440 tx=0x7fe47000c900 comp rx=0 tx=0).stop 2023-10-14T17:00:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 shutdown_connections 2023-10-14T17:00:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fe45806acf0 0x7fe45806d1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe478159950 0x7fe47806b2e0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe478070450 0x7fe47806ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 --2- 172.21.15.29:0/4055038256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe47806c810 0x7fe4780b3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 >> 172.21.15.29:0/4055038256 conn(0x7fe4780ce6a0 msgr2=0x7fe47815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 shutdown_connections 2023-10-14T17:00:11.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.573+0000 7fe477fff640 1 -- 172.21.15.29:0/4055038256 wait complete. 2023-10-14T17:00:11.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.813+0000 7ff204c47640 1 Processor -- start 2023-10-14T17:00:11.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.813+0000 7ff204c47640 1 -- start start 2023-10-14T17:00:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.814+0000 7ff204c47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff20006c040 0x7ff20006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.814+0000 7ff204c47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000cda80 0x7ff20015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.814+0000 7ff204c47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20006c960 con 0x7ff2000ad970 2023-10-14T17:00:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.814+0000 7ff204c47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20015b9d0 con 0x7ff2000cda80 2023-10-14T17:00:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.814+0000 7ff204c47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20015bcb0 con 0x7ff20006c040 2023-10-14T17:00:11.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000cda80 0x7ff20015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2000cda80 0x7ff20015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54742/0 (socket says 172.21.15.29:54742) 2023-10-14T17:00:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 -- 172.21.15.29:0/2378289367 learned_addr learned my addr 172.21.15.29:0/2378289367 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 -- 172.21.15.29:0/2378289367 >> v2:172.21.15.186:3300/0 conn(0x7ff20006c040 msgr2=0x7ff20006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 --2- 172.21.15.29:0/2378289367 >> v2:172.21.15.186:3300/0 conn(0x7ff20006c040 0x7ff20006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 -- 172.21.15.29:0/2378289367 >> v1:172.21.15.186:6789/0 conn(0x7ff2000ad970 legacy=0x7ff20015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.815+0000 7ff1fffff640 1 -- 172.21.15.29:0/2378289367 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20015bf90 con 0x7ff2000cda80 2023-10-14T17:00:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.816+0000 7ff1fffff640 1 --2- 172.21.15.29:0/2378289367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000cda80 0x7ff20015ed90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff2000cf350 tx=0x7ff1f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109230bf99346881 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/2378289367 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031870 con 0x7ff2000cda80 2023-10-14T17:00:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/2378289367 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f80319d0 con 0x7ff2000cda80 2023-10-14T17:00:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff1fe7fc640 1 -- 172.21.15.29:0/2378289367 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f8031d60 con 0x7ff2000cda80 2023-10-14T17:00:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 -- 172.21.15.29:0/2378289367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000cda80 msgr2=0x7ff20015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 --2- 172.21.15.29:0/2378289367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000cda80 0x7ff20015ed90 secure :-1 s=READY pgs=1508 cs=0 l=1 rev1=1 crypto rx=0x7ff2000cf350 tx=0x7ff1f802b910 comp rx=0 tx=0).stop 2023-10-14T17:00:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 -- 172.21.15.29:0/2378289367 shutdown_connections 2023-10-14T17:00:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 --2- 172.21.15.29:0/2378289367 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000cda80 0x7ff20015ed90 unknown :-1 s=CLOSED pgs=1508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 --2- 172.21.15.29:0/2378289367 >> v2:172.21.15.186:3300/0 conn(0x7ff20006c040 0x7ff20006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 -- 172.21.15.29:0/2378289367 >> 172.21.15.29:0/2378289367 conn(0x7ff2000ce6a0 msgr2=0x7ff2000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 -- 172.21.15.29:0/2378289367 shutdown_connections 2023-10-14T17:00:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.817+0000 7ff204c47640 1 -- 172.21.15.29:0/2378289367 wait complete. 2023-10-14T17:00:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.819+0000 7ff204c47640 1 Processor -- start 2023-10-14T17:00:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.819+0000 7ff204c47640 1 -- start start 2023-10-14T17:00:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20006c040 0x7ff200153c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 0x7ff20015eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff200154140 con 0x7ff2000ad970 2023-10-14T17:00:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001f0610 con 0x7ff20006c040 2023-10-14T17:00:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff204c47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2001f08f0 con 0x7ff2000cda80 2023-10-14T17:00:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.820+0000 7ff1feffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54758/0 (socket says 172.21.15.29:54758) 2023-10-14T17:00:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 -- 172.21.15.29:0/1866170611 learned_addr learned my addr 172.21.15.29:0/1866170611 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20006c040 0x7ff200153c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1fffff640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 0x7ff20015eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 -- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 msgr2=0x7ff20015eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 0x7ff20015eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 -- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20006c040 msgr2=0x7ff200153c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20006c040 0x7ff200153c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 -- 172.21.15.29:0/1866170611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1f80094d0 con 0x7ff2000ad970 2023-10-14T17:00:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1fffff640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 0x7ff20015eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.821+0000 7ff1feffd640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c400 tx=0x7ff1e800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.822+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e80035f0 con 0x7ff2000ad970 2023-10-14T17:00:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.822+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1e8003750 con 0x7ff2000ad970 2023-10-14T17:00:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.822+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1e8003b40 con 0x7ff2000ad970 2023-10-14T17:00:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.822+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2001f0bd0 con 0x7ff2000ad970 2023-10-14T17:00:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.822+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2001f1190 con 0x7ff2000ad970 2023-10-14T17:00:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.823+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20006b330 con 0x7ff2000ad970 2023-10-14T17:00:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.824+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7ff1e8003ca0 con 0x7ff2000ad970 2023-10-14T17:00:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.825+0000 7ff1f7fff640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 0x7ff1e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.825+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7ff1e8088bf0 con 0x7ff2000ad970 2023-10-14T17:00:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.825+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 0x7ff1e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.826+0000 7ff1ff7fe640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 0x7ff1e006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2000c8470 tx=0x7ff1f00094f0 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:11.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:11.829+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1e8053980 con 0x7ff2000ad970 2023-10-14T17:00:12.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.018+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff20004e9b0 con 0x7ff2000ad970 2023-10-14T17:00:12.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.020+0000 7ff1f7fff640 1 -- 172.21.15.29:0/1866170611 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v128) v1 ==== 74+0+17813 (secure 0 0 0) 0x7ff1e8056fa0 con 0x7ff2000ad970 2023-10-14T17:00:12.021 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:12.021 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":128,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:09.764614+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":14,"pool_name":"cephfs_metadata","create_time":"2023-10-14T16:59:24.261633+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"124","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":15,"pool_name":"cephfs_data","create_time":"2023-10-14T16:59:25.325453+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"128","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":128,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1686923517":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3236529658":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/681365668":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/2259811471":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6833/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3200714056":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3964845078":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:6833/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3986638800":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/354517400":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3653150355":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/226673613":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3890360705":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/2790183016":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/1217266964":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3481988951":"2023-10-15T16:59:41.541911+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:12.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 msgr2=0x7ff1e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 0x7ff1e006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2000c8470 tx=0x7ff1f00094f0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 msgr2=0x7ff20015e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 crypto rx=0x7ff1e800c400 tx=0x7ff1e800c8c0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 shutdown_connections 2023-10-14T17:00:12.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff1e006af00 0x7ff1e006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2000cda80 0x7ff20015eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2000ad970 0x7ff20015e5f0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 --2- 172.21.15.29:0/1866170611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20006c040 0x7ff200153c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 >> 172.21.15.29:0/1866170611 conn(0x7ff2000ce6a0 msgr2=0x7ff20015d380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:12.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 shutdown_connections 2023-10-14T17:00:12.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.023+0000 7ff204c47640 1 -- 172.21.15.29:0/1866170611 wait complete. 2023-10-14T17:00:12.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T17:00:12.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.264+0000 7f0b79883640 1 Processor -- start 2023-10-14T17:00:12.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.264+0000 7f0b79883640 1 -- start start 2023-10-14T17:00:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.265+0000 7f0b79883640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b740b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.265+0000 7f0b79883640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b740d3980 con 0x7f0b740cf710 2023-10-14T17:00:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.265+0000 7f0b79883640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b740ce630 con 0x7f0b740cfaf0 2023-10-14T17:00:12.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.265+0000 7f0b79883640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740d3c60 con 0x7f0b740b3420 2023-10-14T17:00:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.266+0000 7f0b72ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b740cf710 0x7f0b740b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46530/0 (socket says 172.21.15.29:46530) 2023-10-14T17:00:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.266+0000 7f0b72ffd640 1 -- 172.21.15.29:0/666439189 learned_addr learned my addr 172.21.15.29:0/666439189 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.266+0000 7f0b727fc640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b740b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.266+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1942008487 0 0) 0x7f0b740d3980 con 0x7f0b740cf710 2023-10-14T17:00:12.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.266+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b60003520 con 0x7f0b740cf710 2023-10-14T17:00:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2560226384 0 0) 0x7f0b60003520 con 0x7f0b740cf710 2023-10-14T17:00:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 msgr2=0x7f0b740b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b740b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 >> v1:172.21.15.186:6789/0 conn(0x7f0b740cfaf0 legacy=0x7f0b740b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b740ce910 con 0x7f0b740cf710 2023-10-14T17:00:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0b5c0017f0 con 0x7f0b740cf710 2023-10-14T17:00:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 >> v1:172.21.15.29:6789/0 conn(0x7f0b740cf710 legacy=0x7f0b740b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b600039a0 0x7f0b60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b60004d00 0x7f0b60005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b740d3c60 con 0x7f0b600039a0 2023-10-14T17:00:12.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6000bab0 con 0x7f0b60004d00 2023-10-14T17:00:12.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6000bc70 con 0x7f0b740b3420 2023-10-14T17:00:12.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.267+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b600039a0 0x7f0b60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b727fc640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b737fe640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b60004d00 0x7f0b60005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b72ffd640 1 -- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 msgr2=0x7f0b600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b72ffd640 1 -- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b60004d00 msgr2=0x7f0b60005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b60004d00 0x7f0b60005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b72ffd640 1 -- 172.21.15.29:0/666439189 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b60003520 con 0x7f0b600039a0 2023-10-14T17:00:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b727fc640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b740b3420 0x7f0b600047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:12.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.268+0000 7f0b737fe640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b60004d00 0x7f0b60005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.269+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b600039a0 0x7f0b60003dc0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00c190 tx=0x7f0b600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=552e971176c6fd88 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.269+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c007270 con 0x7f0b600039a0 2023-10-14T17:00:12.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.269+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5c0073f0 con 0x7f0b600039a0 2023-10-14T17:00:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.269+0000 7f0b71ffb640 1 -- 172.21.15.29:0/666439189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b5c02c7e0 con 0x7f0b600039a0 2023-10-14T17:00:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 -- 172.21.15.29:0/666439189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b600039a0 msgr2=0x7f0b60003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 --2- 172.21.15.29:0/666439189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b600039a0 0x7f0b60003dc0 secure :-1 s=READY pgs=1511 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c00c190 tx=0x7f0b600057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b737fe640 1 -- 172.21.15.29:0/666439189 reap_dead start 2023-10-14T17:00:12.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 -- 172.21.15.29:0/666439189 shutdown_connections 2023-10-14T17:00:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 -- 172.21.15.29:0/666439189 >> 172.21.15.29:0/666439189 conn(0x7f0b740d2cc0 msgr2=0x7f0b740d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 -- 172.21.15.29:0/666439189 shutdown_connections 2023-10-14T17:00:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.270+0000 7f0b79883640 1 -- 172.21.15.29:0/666439189 wait complete. 2023-10-14T17:00:12.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.272+0000 7f0b79883640 1 Processor -- start 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.272+0000 7f0b79883640 1 -- start start 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b600039a0 0x7f0b740b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 0x7f0b7406b960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b741f39e0 con 0x7f0b740ca7a0 2023-10-14T17:00:12.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b741f2e60 con 0x7f0b740cac20 2023-10-14T17:00:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b79883640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b741f3140 con 0x7f0b600039a0 2023-10-14T17:00:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54766/0 (socket says 172.21.15.29:54766) 2023-10-14T17:00:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 -- 172.21.15.29:0/3652359509 learned_addr learned my addr 172.21.15.29:0/3652359509 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:12.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b600039a0 0x7f0b740b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b737fe640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 0x7f0b7406b960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 -- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b600039a0 msgr2=0x7f0b740b2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b600039a0 0x7f0b740b2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 -- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 msgr2=0x7f0b7406b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 0x7f0b7406b960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b727fc640 1 -- 172.21.15.29:0/3652359509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b741f3420 con 0x7f0b740ca7a0 2023-10-14T17:00:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.273+0000 7f0b737fe640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 0x7f0b7406b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b727fc640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f0b640108b0 tx=0x7f0b64010d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6400eb60 con 0x7f0b740ca7a0 2023-10-14T17:00:12.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6400ecc0 con 0x7f0b740ca7a0 2023-10-14T17:00:12.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6401c400 con 0x7f0b740ca7a0 2023-10-14T17:00:12.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b741f3700 con 0x7f0b740ca7a0 2023-10-14T17:00:12.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.274+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b741f6150 con 0x7f0b740ca7a0 2023-10-14T17:00:12.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.275+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b741f3c20 con 0x7f0b740ca7a0 2023-10-14T17:00:12.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.276+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0b6401c560 con 0x7f0b740ca7a0 2023-10-14T17:00:12.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.277+0000 7f0b6bfff640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 0x7f0b5406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.277+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 6583+0+0 (secure 0 0 0) 0x7f0b6408a070 con 0x7f0b740ca7a0 2023-10-14T17:00:12.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.277+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 0x7f0b5406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.278+0000 7f0b72ffd640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 0x7f0b5406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c034000 tx=0x7f0b5c009030 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.282+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b640bcfe0 con 0x7f0b740ca7a0 2023-10-14T17:00:12.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.518+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0b7404e160 con 0x7f0b740ca7a0 2023-10-14T17:00:12.520 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T17:00:12.519+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T17:00:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.523+0000 7f0b6bfff640 1 -- 172.21.15.29:0/3652359509 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v58) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0b741f3c20 con 0x7f0b740ca7a0 2023-10-14T17:00:12.524 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T17:00:12.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 msgr2=0x7f0b5406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 0x7f0b5406d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c034000 tx=0x7f0b5c009030 comp rx=0 tx=0).stop 2023-10-14T17:00:12.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 msgr2=0x7f0b7406b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 crypto rx=0x7f0b640108b0 tx=0x7f0b64010d70 comp rx=0 tx=0).stop 2023-10-14T17:00:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 shutdown_connections 2023-10-14T17:00:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b5406af00 0x7f0b5406d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b740cac20 0x7f0b7406b960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b740ca7a0 0x7f0b7406b420 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 --2- 172.21.15.29:0/3652359509 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b600039a0 0x7f0b740b2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 >> 172.21.15.29:0/3652359509 conn(0x7f0b740d2cc0 msgr2=0x7f0b740b7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 shutdown_connections 2023-10-14T17:00:12.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.526+0000 7f0b79883640 1 -- 172.21.15.29:0/3652359509 wait complete. 2023-10-14T17:00:12.548 INFO:tasks.ceph.mds.c.smithi029.stderr: -6003> 2023-10-14T16:59:31.224+0000 7f6a32401640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:00:12.554 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T17:00:12.602 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:00:12.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.755+0000 7f21470a2640 1 Processor -- start 2023-10-14T17:00:12.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.755+0000 7f21470a2640 1 -- start start 2023-10-14T17:00:12.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.756+0000 7f21470a2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f21400caa80 0x7f2140069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f21470a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f21401650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f21470a2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f214006a730 con 0x7f21400addb0 2023-10-14T17:00:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f21470a2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2140161a50 con 0x7f21400caa80 2023-10-14T17:00:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f21470a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21400caef0 con 0x7f214006a2e0 2023-10-14T17:00:12.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f2144e3e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f21400addb0 0x7f21400c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46556/0 (socket says 172.21.15.29:46556) 2023-10-14T17:00:12.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f2144e3e640 1 -- 172.21.15.29:0/1421847986 learned_addr learned my addr 172.21.15.29:0/1421847986 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f214563f640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f21401650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.757+0000 7f213ffff640 1 --2- 172.21.15.29:0/1421847986 >> v2:172.21.15.186:3300/0 conn(0x7f21400caa80 0x7f2140069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1910999469 0 0) 0x7f2134001110 con 0x7f21400addb0 2023-10-14T17:00:12.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2130003520 con 0x7f21400addb0 2023-10-14T17:00:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1090975133 0 0) 0x7f2130003520 con 0x7f21400addb0 2023-10-14T17:00:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 msgr2=0x7f21401650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f21401650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 >> v2:172.21.15.186:3300/0 conn(0x7f21400caa80 msgr2=0x7f2140069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 --2- 172.21.15.29:0/1421847986 >> v2:172.21.15.186:3300/0 conn(0x7f21400caa80 0x7f2140069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400cb140 con 0x7f21400addb0 2023-10-14T17:00:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2134001800 con 0x7f21400addb0 2023-10-14T17:00:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 >> v1:172.21.15.29:6789/0 conn(0x7f21400addb0 legacy=0x7f21400c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2130003970 0x7f2130003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21300047c0 0x7f2130004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f2130005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2140161a50 con 0x7f21300047c0 2023-10-14T17:00:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21400caef0 con 0x7f2130003970 2023-10-14T17:00:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.758+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f213000b980 con 0x7f214006a2e0 2023-10-14T17:00:12.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21300047c0 0x7f2130004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f214563f640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f2130005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f2144e3e640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2130003970 0x7f2130003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 -- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 msgr2=0x7f2130005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f214006a2e0 0x7f2130005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 -- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2130003970 msgr2=0x7f2130003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2130003970 0x7f2130003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.759+0000 7f213ffff640 1 -- 172.21.15.29:0/1421847986 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2130003520 con 0x7f21300047c0 2023-10-14T17:00:12.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.760+0000 7f213ffff640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21300047c0 0x7f2130004c40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ac0 tx=0x7f21300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97ad4db986e2553 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c02c6d0 con 0x7f21300047c0 2023-10-14T17:00:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c02c830 con 0x7f21300047c0 2023-10-14T17:00:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f213f7fe640 1 -- 172.21.15.29:0/1421847986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c02cbc0 con 0x7f21300047c0 2023-10-14T17:00:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f21470a2640 1 -- 172.21.15.29:0/1421847986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21300047c0 msgr2=0x7f2130004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f21470a2640 1 --2- 172.21.15.29:0/1421847986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21300047c0 0x7f2130004c40 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 crypto rx=0x7f212c009ac0 tx=0x7f21300057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f214563f640 1 -- 172.21.15.29:0/1421847986 reap_dead start 2023-10-14T17:00:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f21470a2640 1 -- 172.21.15.29:0/1421847986 shutdown_connections 2023-10-14T17:00:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.761+0000 7f21470a2640 1 -- 172.21.15.29:0/1421847986 >> 172.21.15.29:0/1421847986 conn(0x7f21400ce9b0 msgr2=0x7f21400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.762+0000 7f21470a2640 1 -- 172.21.15.29:0/1421847986 shutdown_connections 2023-10-14T17:00:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.762+0000 7f21470a2640 1 -- 172.21.15.29:0/1421847986 wait complete. 2023-10-14T17:00:12.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.763+0000 7f21470a2640 1 Processor -- start 2023-10-14T17:00:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 -- start start 2023-10-14T17:00:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 0x7f214015cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 0x7f21401f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214015d330 con 0x7f214015e6f0 2023-10-14T17:00:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214015b330 con 0x7f21300047c0 2023-10-14T17:00:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.764+0000 7f21470a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21400c3cb0 con 0x7f21400c4d50 2023-10-14T17:00:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54794/0 (socket says 172.21.15.29:54794) 2023-10-14T17:00:12.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 -- 172.21.15.29:0/3856585329 learned_addr learned my addr 172.21.15.29:0/3856585329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f214563f640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 0x7f21401f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f2144e3e640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 0x7f214015cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 -- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 msgr2=0x7f21401f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 0x7f21401f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 -- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 msgr2=0x7f214015cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 0x7f214015cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f213ffff640 1 -- 172.21.15.29:0/3856585329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21400c3f90 con 0x7f214015e6f0 2023-10-14T17:00:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f214563f640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 0x7f21401f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.765+0000 7f2144e3e640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 0x7f214015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:12.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f213ffff640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f212c02c2d0 tx=0x7f212c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c003810 con 0x7f214015e6f0 2023-10-14T17:00:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c003970 con 0x7f214015e6f0 2023-10-14T17:00:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f212c003d20 con 0x7f214015e6f0 2023-10-14T17:00:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21400c4270 con 0x7f214015e6f0 2023-10-14T17:00:12.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.766+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21400c4830 con 0x7f214015e6f0 2023-10-14T17:00:12.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.767+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21400c49c0 con 0x7f214015e6f0 2023-10-14T17:00:12.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.768+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 68) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f212c040450 con 0x7f214015e6f0 2023-10-14T17:00:12.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.769+0000 7f213d7fa640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 0x7f212406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:12.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.769+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6641+0+0 (secure 0 0 0) 0x7f212c0aba80 con 0x7f214015e6f0 2023-10-14T17:00:12.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.770+0000 7f2144e3e640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 0x7f212406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:12.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.770+0000 7f2144e3e640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 0x7f212406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2134000c00 tx=0x7f2134009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:12.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:12.774+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f212c076720 con 0x7f214015e6f0 2023-10-14T17:00:13.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.013+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f214004e160 con 0x7f214015e6f0 2023-10-14T17:00:13.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.542+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f212c076af0 con 0x7f214015e6f0 2023-10-14T17:00:13.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.547+0000 7f213d7fa640 1 -- 172.21.15.29:0/3856585329 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v59) v1 ==== 104+0+0 (secure 0 0 0) 0x7f212c08e820 con 0x7f214015e6f0 2023-10-14T17:00:13.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 msgr2=0x7f212406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 0x7f212406d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2134000c00 tx=0x7f2134009040 comp rx=0 tx=0).stop 2023-10-14T17:00:13.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 msgr2=0x7f214015a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 crypto rx=0x7f212c02c2d0 tx=0x7f212c004730 comp rx=0 tx=0).stop 2023-10-14T17:00:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 shutdown_connections 2023-10-14T17:00:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f212406af00 0x7f212406d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21400c4d50 0x7f21401f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f214015e6f0 0x7f214015a770 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 --2- 172.21.15.29:0/3856585329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21300047c0 0x7f214015cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 >> 172.21.15.29:0/3856585329 conn(0x7f21400ce9b0 msgr2=0x7f2140072ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 shutdown_connections 2023-10-14T17:00:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.552+0000 7f21470a2640 1 -- 172.21.15.29:0/3856585329 wait complete. 2023-10-14T17:00:13.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T17:00:13.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.848+0000 7f33f1ed4640 1 Processor -- start 2023-10-14T17:00:13.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.848+0000 7f33f1ed4640 1 -- start start 2023-10-14T17:00:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33f1ed4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f33ec15ac10 0x7f33ec06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33f1ed4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0b0360 0x7f33ec0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33f1ed4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33ec0b0d00 con 0x7f33ec0ca510 2023-10-14T17:00:13.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33f1ed4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33ec15f800 con 0x7f33ec15ac10 2023-10-14T17:00:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33f1ed4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec15b310 con 0x7f33ec0b0360 2023-10-14T17:00:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.850+0000 7f33eb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f33ec15ac10 0x7f33ec06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f33ec15ac10 0x7f33ec06c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54808/0 (socket says 172.21.15.29:54808) 2023-10-14T17:00:13.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 -- 172.21.15.29:0/155529864 learned_addr learned my addr 172.21.15.29:0/155529864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33e2ffd640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0b0360 0x7f33ec0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 -- 172.21.15.29:0/155529864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0b0360 msgr2=0x7f33ec0b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:13.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0b0360 0x7f33ec0b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 -- 172.21.15.29:0/155529864 >> v1:172.21.15.29:6789/0 conn(0x7f33ec0ca510 legacy=0x7f33ec0ca940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.851+0000 7f33eb7fe640 1 -- 172.21.15.29:0/155529864 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33ec06f640 con 0x7f33ec15ac10 2023-10-14T17:00:13.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.852+0000 7f33eb7fe640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec15ac10 0x7f33ec06c730 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009e50 tx=0x7f33dc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2a8125a261e3bc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:13.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.852+0000 7f33eaffd640 1 -- 172.21.15.29:0/155529864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc02ca80 con 0x7f33ec15ac10 2023-10-14T17:00:13.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33eaffd640 1 -- 172.21.15.29:0/155529864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc02cbe0 con 0x7f33ec15ac10 2023-10-14T17:00:13.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33eaffd640 1 -- 172.21.15.29:0/155529864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc0343f0 con 0x7f33ec15ac10 2023-10-14T17:00:13.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 -- 172.21.15.29:0/155529864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec15ac10 msgr2=0x7f33ec06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec15ac10 0x7f33ec06c730 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 crypto rx=0x7f33dc009e50 tx=0x7f33dc02bee0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 -- 172.21.15.29:0/155529864 shutdown_connections 2023-10-14T17:00:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0b0360 0x7f33ec0b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/155529864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec15ac10 0x7f33ec06c730 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 -- 172.21.15.29:0/155529864 >> 172.21.15.29:0/155529864 conn(0x7f33ec0cb360 msgr2=0x7f33ec0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.853+0000 7f33f1ed4640 1 -- 172.21.15.29:0/155529864 shutdown_connections 2023-10-14T17:00:13.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.854+0000 7f33f1ed4640 1 -- 172.21.15.29:0/155529864 wait complete. 2023-10-14T17:00:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.855+0000 7f33f1ed4640 1 Processor -- start 2023-10-14T17:00:13.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.855+0000 7f33f1ed4640 1 -- start start 2023-10-14T17:00:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0ca510 0x7f33ec0cc4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec15ac10 0x7f33ec0b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec1538f0 con 0x7f33ec0b0360 2023-10-14T17:00:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0b3b80 con 0x7f33ec15ac10 2023-10-14T17:00:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.856+0000 7f33f1ed4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33ec0ccb70 con 0x7f33ec0ca510 2023-10-14T17:00:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54818/0 (socket says 172.21.15.29:54818) 2023-10-14T17:00:13.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 -- 172.21.15.29:0/902362311 learned_addr learned my addr 172.21.15.29:0/902362311 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 -- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0ca510 msgr2=0x7f33ec0cc4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33e2ffd640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0ca510 0x7f33ec0cc4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0ca510 0x7f33ec0cc4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33ebfff640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec15ac10 0x7f33ec0b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 -- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec15ac10 msgr2=0x7f33ec0b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec15ac10 0x7f33ec0b3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 -- 172.21.15.29:0/902362311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc009b30 con 0x7f33ec0b0360 2023-10-14T17:00:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.857+0000 7f33eb7fe640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036040 tx=0x7f33dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:13.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.858+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc03e450 con 0x7f33ec0b0360 2023-10-14T17:00:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.858+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33dc03e5f0 con 0x7f33ec0b0360 2023-10-14T17:00:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.858+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33dc03e9a0 con 0x7f33ec0b0360 2023-10-14T17:00:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.858+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33ec0cce50 con 0x7f33ec0b0360 2023-10-14T17:00:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.858+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33ec0cd410 con 0x7f33ec0b0360 2023-10-14T17:00:13.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.859+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33ec04e160 con 0x7f33ec0b0360 2023-10-14T17:00:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.860+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f33dc02cd50 con 0x7f33ec0b0360 2023-10-14T17:00:13.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.861+0000 7f33e8ff9640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 0x7f33cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.861+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6641+0+0 (secure 0 0 0) 0x7f33dc0ac820 con 0x7f33ec0b0360 2023-10-14T17:00:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.861+0000 7f33e2ffd640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 0x7f33cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.862+0000 7f33e2ffd640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 0x7f33cc06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0ce130 tx=0x7f33d4008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:13.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:13.865+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f33dc077400 con 0x7f33ec0b0360 2023-10-14T17:00:14.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.063+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f33ec06c2e0 con 0x7f33ec0b0360 2023-10-14T17:00:14.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.550+0000 7f33e8ff9640 1 -- 172.21.15.29:0/902362311 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v130) v1 ==== 181+0+0 (secure 0 0 0) 0x7f33dc07aa20 con 0x7f33ec0b0360 2023-10-14T17:00:14.552 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T17:00:14.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.554+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 msgr2=0x7f33cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.554+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 0x7f33cc06d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f33ec0ce130 tx=0x7f33d4008040 comp rx=0 tx=0).stop 2023-10-14T17:00:14.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.554+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 msgr2=0x7f33ec0cdbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.554+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 secure :-1 s=READY pgs=1521 cs=0 l=1 rev1=1 crypto rx=0x7f33dc036040 tx=0x7f33dc004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:14.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 shutdown_connections 2023-10-14T17:00:14.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f33cc06af90 0x7f33cc06d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33ec15ac10 0x7f33ec0b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33ec0ca510 0x7f33ec0cc4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 --2- 172.21.15.29:0/902362311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33ec0b0360 0x7f33ec0cdbf0 unknown :-1 s=CLOSED pgs=1521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 >> 172.21.15.29:0/902362311 conn(0x7f33ec0cb360 msgr2=0x7f33ec15e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:14.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 shutdown_connections 2023-10-14T17:00:14.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.555+0000 7f33f1ed4640 1 -- 172.21.15.29:0/902362311 wait complete. 2023-10-14T17:00:14.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T17:00:14.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.787+0000 7fb5d5c02640 1 Processor -- start 2023-10-14T17:00:14.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.788+0000 7fb5d5c02640 1 -- start start 2023-10-14T17:00:14.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.789+0000 7fb5d5c02640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb5d0159da0 0x7fb5d00c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.789+0000 7fb5d5c02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.789+0000 7fb5d5c02640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5d015dfb0 con 0x7fb5d01599c0 2023-10-14T17:00:14.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.789+0000 7fb5d5c02640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d015e290 con 0x7fb5d0159da0 2023-10-14T17:00:14.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.789+0000 7fb5d5c02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d015e570 con 0x7fb5d00cce20 2023-10-14T17:00:14.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.790+0000 7fb5cffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.790+0000 7fb5cffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:49588/0 (socket says 172.21.15.29:49588) 2023-10-14T17:00:14.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.790+0000 7fb5cffff640 1 -- 172.21.15.29:0/2152745414 learned_addr learned my addr 172.21.15.29:0/2152745414 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:14.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.790+0000 7fb5c7fff640 1 --2- 172.21.15.29:0/2152745414 >> v2:172.21.15.186:3300/0 conn(0x7fb5d0159da0 0x7fb5d00c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.791+0000 7fb5cffff640 1 -- 172.21.15.29:0/2152745414 >> v2:172.21.15.186:3300/0 conn(0x7fb5d0159da0 msgr2=0x7fb5d00c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:14.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.791+0000 7fb5cffff640 1 --2- 172.21.15.29:0/2152745414 >> v2:172.21.15.186:3300/0 conn(0x7fb5d0159da0 0x7fb5d00c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.791+0000 7fb5cffff640 1 -- 172.21.15.29:0/2152745414 >> v1:172.21.15.186:6789/0 conn(0x7fb5d01599c0 legacy=0x7fb5d015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.791+0000 7fb5cffff640 1 -- 172.21.15.29:0/2152745414 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d006fda0 con 0x7fb5d00cce20 2023-10-14T17:00:14.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.792+0000 7fb5cffff640 1 --2- 172.21.15.29:0/2152745414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009e50 tx=0x7fb5c802bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=615031b72aa0aff6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:14.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5ceffd640 1 -- 172.21.15.29:0/2152745414 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c802cb00 con 0x7fb5d00cce20 2023-10-14T17:00:14.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5ceffd640 1 -- 172.21.15.29:0/2152745414 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c802cc60 con 0x7fb5d00cce20 2023-10-14T17:00:14.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5ceffd640 1 -- 172.21.15.29:0/2152745414 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c80343f0 con 0x7fb5d00cce20 2023-10-14T17:00:14.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 -- 172.21.15.29:0/2152745414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 msgr2=0x7fb5d006c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/2152745414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8009e50 tx=0x7fb5c802bf60 comp rx=0 tx=0).stop 2023-10-14T17:00:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 -- 172.21.15.29:0/2152745414 shutdown_connections 2023-10-14T17:00:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/2152745414 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d00cce20 0x7fb5d006c0d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/2152745414 >> v2:172.21.15.186:3300/0 conn(0x7fb5d0159da0 0x7fb5d00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.793+0000 7fb5d5c02640 1 -- 172.21.15.29:0/2152745414 >> 172.21.15.29:0/2152745414 conn(0x7fb5d00cb360 msgr2=0x7fb5d00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:14.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.794+0000 7fb5d5c02640 1 -- 172.21.15.29:0/2152745414 shutdown_connections 2023-10-14T17:00:14.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.794+0000 7fb5d5c02640 1 -- 172.21.15.29:0/2152745414 wait complete. 2023-10-14T17:00:14.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.796+0000 7fb5d5c02640 1 Processor -- start 2023-10-14T17:00:14.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.796+0000 7fb5d5c02640 1 -- start start 2023-10-14T17:00:14.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 0x7fb5d01f5900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d01599c0 0x7fb5d015a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d015c230 con 0x7fb5d0159da0 2023-10-14T17:00:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d01f6810 con 0x7fb5d00cce20 2023-10-14T17:00:14.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5d5c02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d01f5f70 con 0x7fb5d01599c0 2023-10-14T17:00:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5cffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54830/0 (socket says 172.21.15.29:54830) 2023-10-14T17:00:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5cffff640 1 -- 172.21.15.29:0/446798804 learned_addr learned my addr 172.21.15.29:0/446798804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.797+0000 7fb5c7fff640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d01599c0 0x7fb5d015a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cf7fe640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 0x7fb5d01f5900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 -- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d01599c0 msgr2=0x7fb5d015a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d01599c0 0x7fb5d015a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 -- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 msgr2=0x7fb5d01f5900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:14.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 0x7fb5d01f5900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:14.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 -- 172.21.15.29:0/446798804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5c8009b30 con 0x7fb5d0159da0 2023-10-14T17:00:14.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cf7fe640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 0x7fb5d01f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:14.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5cffff640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8036040 tx=0x7fb5c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:14.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c8009510 con 0x7fb5d0159da0 2023-10-14T17:00:14.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5c803e450 con 0x7fb5d0159da0 2023-10-14T17:00:14.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c803e820 con 0x7fb5d0159da0 2023-10-14T17:00:14.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.798+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d01f6250 con 0x7fb5d0159da0 2023-10-14T17:00:14.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.799+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d015ab20 con 0x7fb5d0159da0 2023-10-14T17:00:14.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.800+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d015c000 con 0x7fb5d0159da0 2023-10-14T17:00:14.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.801+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb5c803e980 con 0x7fb5d0159da0 2023-10-14T17:00:14.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.802+0000 7fb5ccff9640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 0x7fb5b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:14.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.802+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6203+0+0 (secure 0 0 0) 0x7fb5c80ac6c0 con 0x7fb5d0159da0 2023-10-14T17:00:14.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.803+0000 7fb5cf7fe640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 0x7fb5b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:14.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.803+0000 7fb5cf7fe640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 0x7fb5b006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0000f60 tx=0x7fb5c0009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:14.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:14.807+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb5d015c000 con 0x7fb5d0159da0 2023-10-14T17:00:15.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.012+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fb5d015c000 con 0x7fb5d0159da0 2023-10-14T17:00:15.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.561+0000 7fb5ccff9640 1 -- 172.21.15.29:0/446798804 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v131) v1 ==== 169+0+0 (secure 0 0 0) 0x7fb5d015c000 con 0x7fb5d0159da0 2023-10-14T17:00:15.562 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T17:00:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 msgr2=0x7fb5b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 0x7fb5b006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb5c0000f60 tx=0x7fb5c0009040 comp rx=0 tx=0).stop 2023-10-14T17:00:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 msgr2=0x7fb5d0149b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 crypto rx=0x7fb5c8036040 tx=0x7fb5c8004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 shutdown_connections 2023-10-14T17:00:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fb5b006af90 0x7fb5b006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5d0159da0 0x7fb5d0149b90 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5d01599c0 0x7fb5d015a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 --2- 172.21.15.29:0/446798804 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb5d00cce20 0x7fb5d01f5900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 >> 172.21.15.29:0/446798804 conn(0x7fb5d00cb360 msgr2=0x7fb5d00cc6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 shutdown_connections 2023-10-14T17:00:15.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.565+0000 7fb5d5c02640 1 -- 172.21.15.29:0/446798804 wait complete. 2023-10-14T17:00:15.594 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T17:00:15.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.795+0000 7faa20b0a640 1 Processor -- start 2023-10-14T17:00:15.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.795+0000 7faa20b0a640 1 -- start start 2023-10-14T17:00:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa20b0a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa1c06ace0 0x7faa1c06f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa20b0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c06b1e0 con 0x7faa1c1592e0 2023-10-14T17:00:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa20b0a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa1c06f3f0 con 0x7faa1c0ca010 2023-10-14T17:00:15.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa20b0a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa1c06f6d0 con 0x7faa1c06ace0 2023-10-14T17:00:15.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa1b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa1c1592e0 0x7faa1c0cdd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46574/0 (socket says 172.21.15.29:46574) 2023-10-14T17:00:15.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa1b7fe640 1 -- 172.21.15.29:0/163810102 learned_addr learned my addr 172.21.15.29:0/163810102 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:15.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.797+0000 7faa1bfff640 1 --2- 172.21.15.29:0/163810102 >> v2:172.21.15.29:3300/0 conn(0x7faa1c06ace0 0x7faa1c06f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:15.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.798+0000 7faa1affd640 1 -- 172.21.15.29:0/163810102 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319791652 0 0) 0x7faa1c06b1e0 con 0x7faa1c1592e0 2023-10-14T17:00:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.798+0000 7faa1bfff640 1 -- 172.21.15.29:0/163810102 >> v1:172.21.15.186:6789/0 conn(0x7faa1c0ca010 legacy=0x7faa1c06a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.798+0000 7faa1bfff640 1 -- 172.21.15.29:0/163810102 >> v1:172.21.15.29:6789/0 conn(0x7faa1c1592e0 legacy=0x7faa1c0cdd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.798+0000 7faa1bfff640 1 -- 172.21.15.29:0/163810102 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa1c0c9810 con 0x7faa1c06ace0 2023-10-14T17:00:15.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.799+0000 7faa1bfff640 1 --2- 172.21.15.29:0/163810102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c06ace0 0x7faa1c06f9b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7faa14009d90 tx=0x7faa1402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95d27fbc06388591 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.799+0000 7faa1affd640 1 -- 172.21.15.29:0/163810102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1402cbd0 con 0x7faa1c06ace0 2023-10-14T17:00:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.799+0000 7faa1affd640 1 -- 172.21.15.29:0/163810102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa1402cd30 con 0x7faa1c06ace0 2023-10-14T17:00:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.799+0000 7faa1affd640 1 -- 172.21.15.29:0/163810102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa1402cea0 con 0x7faa1c06ace0 2023-10-14T17:00:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 -- 172.21.15.29:0/163810102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c06ace0 msgr2=0x7faa1c06f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 --2- 172.21.15.29:0/163810102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c06ace0 0x7faa1c06f9b0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 crypto rx=0x7faa14009d90 tx=0x7faa1402be20 comp rx=0 tx=0).stop 2023-10-14T17:00:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 -- 172.21.15.29:0/163810102 shutdown_connections 2023-10-14T17:00:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 --2- 172.21.15.29:0/163810102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c06ace0 0x7faa1c06f9b0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 -- 172.21.15.29:0/163810102 >> 172.21.15.29:0/163810102 conn(0x7faa1c0b0170 msgr2=0x7faa1c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 -- 172.21.15.29:0/163810102 shutdown_connections 2023-10-14T17:00:15.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.800+0000 7faa20b0a640 1 -- 172.21.15.29:0/163810102 wait complete. 2023-10-14T17:00:15.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.802+0000 7faa20b0a640 1 Processor -- start 2023-10-14T17:00:15.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.802+0000 7faa20b0a640 1 -- start start 2023-10-14T17:00:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa1c06ace0 0x7faa1c0c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa1c1592e0 0x7faa1c159840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:15.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c5750 con 0x7faa1c0ca010 2023-10-14T17:00:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c59f0 con 0x7faa1c1592e0 2023-10-14T17:00:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa20b0a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa1c0c4150 con 0x7faa1c06ace0 2023-10-14T17:00:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa13fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54854/0 (socket says 172.21.15.29:54854) 2023-10-14T17:00:15.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.803+0000 7faa13fff640 1 -- 172.21.15.29:0/836016701 learned_addr learned my addr 172.21.15.29:0/836016701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 -- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa1c06ace0 msgr2=0x7faa1c0c3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa1c06ace0 0x7faa1c0c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 -- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa1c1592e0 msgr2=0x7faa1c159840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa1c1592e0 0x7faa1c159840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:15.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 -- 172.21.15.29:0/836016701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa14009a70 con 0x7faa1c0ca010 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa13fff640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c430 tx=0x7faa0400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04011790 con 0x7faa1c0ca010 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa040118f0 con 0x7faa1c0ca010 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa04011ce0 con 0x7faa1c0ca010 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.804+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa1c0c43d0 con 0x7faa1c0ca010 2023-10-14T17:00:15.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.805+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa1c0c4990 con 0x7faa1c0ca010 2023-10-14T17:00:15.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.806+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa1c0c4bb0 con 0x7faa1c0ca010 2023-10-14T17:00:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.807+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa0401c460 con 0x7faa1c0ca010 2023-10-14T17:00:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.808+0000 7faa18ff9640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 0x7fa9fc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.808+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7faa04089720 con 0x7faa1c0ca010 2023-10-14T17:00:15.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.808+0000 7faa1b7fe640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 0x7fa9fc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:15.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.809+0000 7faa1b7fe640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 0x7fa9fc06d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa0c007fd0 tx=0x7faa0c007920 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:15.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:15.813+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa040546e0 con 0x7faa1c0ca010 2023-10-14T17:00:16.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.013+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7faa1c0c5b60 con 0x7faa1c0ca010 2023-10-14T17:00:16.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.018+0000 7faa18ff9640 1 -- 172.21.15.29:0/836016701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v43) v1 ==== 115+0+0 (secure 0 0 0) 0x7faa04057d00 con 0x7faa1c0ca010 2023-10-14T17:00:16.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 msgr2=0x7fa9fc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 0x7fa9fc06d4a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7faa0c007fd0 tx=0x7faa0c007920 comp rx=0 tx=0).stop 2023-10-14T17:00:16.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 msgr2=0x7faa1c0c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 crypto rx=0x7faa0400c430 tx=0x7faa0400c8f0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 shutdown_connections 2023-10-14T17:00:16.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fa9fc06afe0 0x7fa9fc06d4a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa1c1592e0 0x7faa1c159840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa1c0ca010 0x7faa1c0c7680 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 --2- 172.21.15.29:0/836016701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa1c06ace0 0x7faa1c0c3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 >> 172.21.15.29:0/836016701 conn(0x7faa1c0b0170 msgr2=0x7faa1c06f910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:16.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 shutdown_connections 2023-10-14T17:00:16.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.022+0000 7faa20b0a640 1 -- 172.21.15.29:0/836016701 wait complete. 2023-10-14T17:00:16.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T17:00:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.270+0000 7f07d727c640 1 Processor -- start 2023-10-14T17:00:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.270+0000 7f07d727c640 1 -- start start 2023-10-14T17:00:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d015ad70 0x7f07d0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f07d00c9ca0 0x7f07d006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 0x7f07d00ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d00cad80 con 0x7f07d015ad70 2023-10-14T17:00:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07d00cb030 con 0x7f07d00c9ca0 2023-10-14T17:00:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d727c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00c91c0 con 0x7f07d00ca3b0 2023-10-14T17:00:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d5018640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d015ad70 0x7f07d0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d5018640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07d015ad70 0x7f07d0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54862/0 (socket says 172.21.15.29:54862) 2023-10-14T17:00:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.272+0000 7f07d5018640 1 -- 172.21.15.29:0/603690298 learned_addr learned my addr 172.21.15.29:0/603690298 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d4817640 1 --2- 172.21.15.29:0/603690298 >> v2:172.21.15.186:3300/0 conn(0x7f07d00c9ca0 0x7f07d006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5018640 1 -- 172.21.15.29:0/603690298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 msgr2=0x7f07d00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5819640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 0x7f07d00ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5018640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 0x7f07d00ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5018640 1 -- 172.21.15.29:0/603690298 >> v2:172.21.15.186:3300/0 conn(0x7f07d00c9ca0 msgr2=0x7f07d006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5018640 1 --2- 172.21.15.29:0/603690298 >> v2:172.21.15.186:3300/0 conn(0x7f07d00c9ca0 0x7f07d006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5018640 1 -- 172.21.15.29:0/603690298 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d00c94a0 con 0x7f07d015ad70 2023-10-14T17:00:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.273+0000 7f07d5819640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 0x7f07d00ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.274+0000 7f07d5018640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d0069da0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f07d00ce870 tx=0x7f07cc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06f68c9e2d7a27 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.274+0000 7f07c7fff640 1 -- 172.21.15.29:0/603690298 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc02ca70 con 0x7f07d015ad70 2023-10-14T17:00:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.274+0000 7f07c7fff640 1 -- 172.21.15.29:0/603690298 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07cc02cbd0 con 0x7f07d015ad70 2023-10-14T17:00:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.274+0000 7f07c7fff640 1 -- 172.21.15.29:0/603690298 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07cc0343f0 con 0x7f07d015ad70 2023-10-14T17:00:16.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.275+0000 7f07d727c640 1 -- 172.21.15.29:0/603690298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 msgr2=0x7f07d0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.275+0000 7f07d727c640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d0069da0 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 crypto rx=0x7f07d00ce870 tx=0x7f07cc02bed0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 -- 172.21.15.29:0/603690298 shutdown_connections 2023-10-14T17:00:16.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00ca3b0 0x7f07d00ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 --2- 172.21.15.29:0/603690298 >> v2:172.21.15.186:3300/0 conn(0x7f07d00c9ca0 0x7f07d006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 --2- 172.21.15.29:0/603690298 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d0069da0 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 -- 172.21.15.29:0/603690298 >> 172.21.15.29:0/603690298 conn(0x7f07d00ce9b0 msgr2=0x7f07d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 -- 172.21.15.29:0/603690298 shutdown_connections 2023-10-14T17:00:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.276+0000 7f07d727c640 1 -- 172.21.15.29:0/603690298 wait complete. 2023-10-14T17:00:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.278+0000 7f07d727c640 1 Processor -- start 2023-10-14T17:00:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.278+0000 7f07d727c640 1 -- start start 2023-10-14T17:00:16.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00c9ca0 0x7f07d00aeee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d00ca3b0 0x7f07d00c51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00af420 con 0x7f07d015ad70 2023-10-14T17:00:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00aea50 con 0x7f07d00ca3b0 2023-10-14T17:00:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d727c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07d00ae120 con 0x7f07d00c9ca0 2023-10-14T17:00:16.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d5819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d5819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54868/0 (socket says 172.21.15.29:54868) 2023-10-14T17:00:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d5819640 1 -- 172.21.15.29:0/2943556491 learned_addr learned my addr 172.21.15.29:0/2943556491 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:16.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.280+0000 7f07d5018640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00c9ca0 0x7f07d00aeee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d4817640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d00ca3b0 0x7f07d00c51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 -- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00c9ca0 msgr2=0x7f07d00aeee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00c9ca0 0x7f07d00aeee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 -- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d00ca3b0 msgr2=0x7f07d00c51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d00ca3b0 0x7f07d00c51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 -- 172.21.15.29:0/2943556491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07cc009b30 con 0x7f07d015ad70 2023-10-14T17:00:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d5819640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c7f0 tx=0x7f07c800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8007820 con 0x7f07d015ad70 2023-10-14T17:00:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07c8007980 con 0x7f07d015ad70 2023-10-14T17:00:16.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07c8007d30 con 0x7f07d015ad70 2023-10-14T17:00:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.281+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07d00ae400 con 0x7f07d015ad70 2023-10-14T17:00:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.282+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07d00a9500 con 0x7f07d015ad70 2023-10-14T17:00:16.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.283+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07d00aec00 con 0x7f07d015ad70 2023-10-14T17:00:16.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.284+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f07c80113f0 con 0x7f07d015ad70 2023-10-14T17:00:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.285+0000 7f07c5ffb640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 0x7f07b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.285+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f07c8089be0 con 0x7f07d015ad70 2023-10-14T17:00:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.285+0000 7f07d5018640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 0x7f07b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.286+0000 7f07d5018640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 0x7f07b406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07cc035000 tx=0x7f07cc030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.289+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07d00aec00 con 0x7f07d015ad70 2023-10-14T17:00:16.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.488+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f07d00aebc0 con 0x7f07d015ad70 2023-10-14T17:00:16.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.494+0000 7f07c5ffb640 1 -- 172.21.15.29:0/2943556491 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v44) v1 ==== 120+0+0 (secure 0 0 0) 0x7f07c8054a30 con 0x7f07d015ad70 2023-10-14T17:00:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 msgr2=0x7f07b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 0x7f07b406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f07cc035000 tx=0x7f07cc030000 comp rx=0 tx=0).stop 2023-10-14T17:00:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 msgr2=0x7f07d015c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 secure :-1 s=READY pgs=1527 cs=0 l=1 rev1=1 crypto rx=0x7f07c800c7f0 tx=0x7f07c800ccb0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 shutdown_connections 2023-10-14T17:00:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f07b406af90 0x7f07b406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07d015ad70 0x7f07d015c980 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07d00ca3b0 0x7f07d00c51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 --2- 172.21.15.29:0/2943556491 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07d00c9ca0 0x7f07d00aeee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 >> 172.21.15.29:0/2943556491 conn(0x7f07d00ce9b0 msgr2=0x7f07d00b34c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 shutdown_connections 2023-10-14T17:00:16.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.497+0000 7f07d727c640 1 -- 172.21.15.29:0/2943556491 wait complete. 2023-10-14T17:00:16.518 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T17:00:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.739+0000 7f741cd21640 1 Processor -- start 2023-10-14T17:00:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.739+0000 7f741cd21640 1 -- start start 2023-10-14T17:00:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.740+0000 7f741cd21640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7418159950 0x7f741815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.740+0000 7f741cd21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f74180708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.740+0000 7f741cd21640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7418070df0 con 0x7f741806c100 2023-10-14T17:00:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.740+0000 7f741cd21640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f741815e490 con 0x7f7418159950 2023-10-14T17:00:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.740+0000 7f741cd21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f741815aef0 con 0x7f7418070450 2023-10-14T17:00:16.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f7417fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f741806c100 0x7f741806c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46580/0 (socket says 172.21.15.29:46580) 2023-10-14T17:00:16.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f7417fff640 1 -- 172.21.15.29:0/2059162484 learned_addr learned my addr 172.21.15.29:0/2059162484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:16.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f74177fe640 1 --2- 172.21.15.29:0/2059162484 >> v2:172.21.15.186:3300/0 conn(0x7f7418159950 0x7f741815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f740ffff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f74180708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704743619 0 0) 0x7f7418070df0 con 0x7f741806c100 2023-10-14T17:00:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.741+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7404003520 con 0x7f741806c100 2023-10-14T17:00:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1979977492 0 0) 0x7f7404003520 con 0x7f741806c100 2023-10-14T17:00:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 msgr2=0x7f74180708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f74180708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 >> v2:172.21.15.186:3300/0 conn(0x7f7418159950 msgr2=0x7f741815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 --2- 172.21.15.29:0/2059162484 >> v2:172.21.15.186:3300/0 conn(0x7f7418159950 0x7f741815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f741806cc80 con 0x7f741806c100 2023-10-14T17:00:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f74100019f0 con 0x7f741806c100 2023-10-14T17:00:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 >> v1:172.21.15.29:6789/0 conn(0x7f741806c100 legacy=0x7f741806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f7404003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404003f40 0x7f74040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7404004d00 0x7f7404005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741815e490 con 0x7f7404004d00 2023-10-14T17:00:16.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741815aef0 con 0x7f7404003f40 2023-10-14T17:00:16.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f740400dc40 con 0x7f7418070450 2023-10-14T17:00:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f7417fff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7404004d00 0x7f7404005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.742+0000 7f74177fe640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f7404003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f740ffff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404003f40 0x7f74040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f7417fff640 1 -- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 msgr2=0x7f7404003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f7417fff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7418070450 0x7f7404003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f7417fff640 1 -- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404003f40 msgr2=0x7f74040047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f7417fff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404003f40 0x7f74040047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f7417fff640 1 -- 172.21.15.29:0/2059162484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7404003520 con 0x7f7404004d00 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.743+0000 7f740ffff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404003f40 0x7f74040047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.744+0000 7f7417fff640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7404004d00 0x7f7404005180 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f741000a790 tx=0x7f74040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ef693f841240cf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.744+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410034630 con 0x7f7404004d00 2023-10-14T17:00:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.744+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f741002f070 con 0x7f7404004d00 2023-10-14T17:00:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.744+0000 7f7416ffd640 1 -- 172.21.15.29:0/2059162484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7410034a30 con 0x7f7404004d00 2023-10-14T17:00:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 -- 172.21.15.29:0/2059162484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7404004d00 msgr2=0x7f7404005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 --2- 172.21.15.29:0/2059162484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7404004d00 0x7f7404005180 secure :-1 s=READY pgs=1529 cs=0 l=1 rev1=1 crypto rx=0x7f741000a790 tx=0x7f74040057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f7417fff640 1 -- 172.21.15.29:0/2059162484 reap_dead start 2023-10-14T17:00:16.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 -- 172.21.15.29:0/2059162484 shutdown_connections 2023-10-14T17:00:16.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 -- 172.21.15.29:0/2059162484 >> 172.21.15.29:0/2059162484 conn(0x7f74180ce6a0 msgr2=0x7f74180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:16.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 -- 172.21.15.29:0/2059162484 shutdown_connections 2023-10-14T17:00:16.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.745+0000 7f741cd21640 1 -- 172.21.15.29:0/2059162484 wait complete. 2023-10-14T17:00:16.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.747+0000 7f741cd21640 1 Processor -- start 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.747+0000 7f741cd21640 1 -- start start 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404004d00 0x7f7418157420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 0x7f74180c6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74180c25a0 con 0x7f74180c5a40 2023-10-14T17:00:16.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418156180 con 0x7f7404004d00 2023-10-14T17:00:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f741cd21640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74180c5ec0 con 0x7f74180c29c0 2023-10-14T17:00:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54880/0 (socket says 172.21.15.29:54880) 2023-10-14T17:00:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 -- 172.21.15.29:0/2974605011 learned_addr learned my addr 172.21.15.29:0/2974605011 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:16.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f7417fff640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 0x7f74180c6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f74177fe640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404004d00 0x7f7418157420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 -- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 msgr2=0x7f74180c6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 0x7f74180c6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 -- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404004d00 msgr2=0x7f7418157420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.748+0000 7f740ffff640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404004d00 0x7f7418157420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f740ffff640 1 -- 172.21.15.29:0/2974605011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418157b20 con 0x7f74180c5a40 2023-10-14T17:00:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f7417fff640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 0x7f74180c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f740ffff640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f74000108c0 tx=0x7f7400010d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740000eb00 con 0x7f74180c5a40 2023-10-14T17:00:16.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f740000ec60 con 0x7f74180c5a40 2023-10-14T17:00:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740001c400 con 0x7f74180c5a40 2023-10-14T17:00:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7418157e00 con 0x7f74180c5a40 2023-10-14T17:00:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.749+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74180c21c0 con 0x7f74180c5a40 2023-10-14T17:00:16.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.750+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741804e9b0 con 0x7f74180c5a40 2023-10-14T17:00:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.752+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f740001c560 con 0x7f74180c5a40 2023-10-14T17:00:16.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.752+0000 7f7414ff9640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 0x7f73f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.753+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7400089ce0 con 0x7f74180c5a40 2023-10-14T17:00:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.753+0000 7f74177fe640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 0x7f73f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.754+0000 7f74177fe640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 0x7f73f806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f74080099d0 tx=0x7f7408008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:16.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.757+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7400054c40 con 0x7f74180c5a40 2023-10-14T17:00:16.776 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:00:16.784 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:00:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.958+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f74180b4620 con 0x7f74180c5a40 2023-10-14T17:00:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.964+0000 7f7414ff9640 1 -- 172.21.15.29:0/2974605011 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v45) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7400058260 con 0x7f74180c5a40 2023-10-14T17:00:16.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 msgr2=0x7f73f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 0x7f73f806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f74080099d0 tx=0x7f7408008040 comp rx=0 tx=0).stop 2023-10-14T17:00:16.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 msgr2=0x7f7418155c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:16.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 crypto rx=0x7f74000108c0 tx=0x7f7400010d80 comp rx=0 tx=0).stop 2023-10-14T17:00:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 shutdown_connections 2023-10-14T17:00:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f73f806af90 0x7f73f806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74180c29c0 0x7f74180c6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74180c5a40 0x7f7418155c40 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 --2- 172.21.15.29:0/2974605011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7404004d00 0x7f7418157420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:16.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 >> 172.21.15.29:0/2974605011 conn(0x7f74180ce6a0 msgr2=0x7f7418070830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:16.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 shutdown_connections 2023-10-14T17:00:16.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:16.967+0000 7f741cd21640 1 -- 172.21.15.29:0/2974605011 wait complete. 2023-10-14T17:00:16.987 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule' 2023-10-14T17:00:17.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.217+0000 7f0b36916640 1 Processor -- start 2023-10-14T17:00:17.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.217+0000 7f0b36916640 1 -- start start 2023-10-14T17:00:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b36916640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b300ca0a0 0x7f0b3006c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b36916640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3015d910 con 0x7f0b30159950 2023-10-14T17:00:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b36916640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3015dbf0 con 0x7f0b30070b60 2023-10-14T17:00:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b36916640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3015e490 con 0x7f0b300ca0a0 2023-10-14T17:00:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b35914640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b30159950 0x7f0b30070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46584/0 (socket says 172.21.15.29:46584) 2023-10-14T17:00:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b35914640 1 -- 172.21.15.29:0/3136963484 learned_addr learned my addr 172.21.15.29:0/3136963484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b36115640 1 --2- 172.21.15.29:0/3136963484 >> v2:172.21.15.186:3300/0 conn(0x7f0b300ca0a0 0x7f0b3006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2742797584 0 0) 0x7f0b3015d910 con 0x7f0b30159950 2023-10-14T17:00:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.219+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b1c005730 con 0x7f0b30159950 2023-10-14T17:00:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126012947 0 0) 0x7f0b1c005730 con 0x7f0b30159950 2023-10-14T17:00:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 >> v2:172.21.15.186:3300/0 conn(0x7f0b300ca0a0 msgr2=0x7f0b3006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 --2- 172.21.15.29:0/3136963484 >> v2:172.21.15.186:3300/0 conn(0x7f0b300ca0a0 0x7f0b3006c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 >> v1:172.21.15.186:6789/0 conn(0x7f0b30070b60 legacy=0x7f0b3006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3015ded0 con 0x7f0b30159950 2023-10-14T17:00:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0b2c001670 con 0x7f0b30159950 2023-10-14T17:00:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 >> v1:172.21.15.29:6789/0 conn(0x7f0b30159950 legacy=0x7f0b30070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b1c005b80 0x7f0b1c005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b1c0069d0 0x7f0b1c006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b1c00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3015e490 con 0x7f0b1c007390 2023-10-14T17:00:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c00e330 con 0x7f0b1c005b80 2023-10-14T17:00:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c00e4f0 con 0x7f0b1c0069d0 2023-10-14T17:00:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.220+0000 7f0b36115640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b1c00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b35113640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b1c0069d0 0x7f0b1c006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b36115640 1 -- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b1c0069d0 msgr2=0x7f0b1c006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b36115640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b1c0069d0 0x7f0b1c006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b36115640 1 -- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b1c005b80 msgr2=0x7f0b1c005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b36115640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b1c005b80 0x7f0b1c005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.221+0000 7f0b36115640 1 -- 172.21.15.29:0/3136963484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b1c005730 con 0x7f0b1c007390 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.222+0000 7f0b36115640 1 -- 172.21.15.29:0/3136963484 reap_dead start 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.222+0000 7f0b36115640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b1c00db90 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009fc0 tx=0x7f0b1c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=192ff91bd457b195 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.222+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2802cc60 con 0x7f0b1c007390 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.222+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2800f040 con 0x7f0b1c007390 2023-10-14T17:00:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.222+0000 7f0b34912640 1 -- 172.21.15.29:0/3136963484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b28003990 con 0x7f0b1c007390 2023-10-14T17:00:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 -- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 msgr2=0x7f0b1c00db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b1c00db90 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 crypto rx=0x7f0b28009fc0 tx=0x7f0b1c007a00 comp rx=0 tx=0).stop 2023-10-14T17:00:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 -- 172.21.15.29:0/3136963484 shutdown_connections 2023-10-14T17:00:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 --2- 172.21.15.29:0/3136963484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b1c00db90 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 -- 172.21.15.29:0/3136963484 >> 172.21.15.29:0/3136963484 conn(0x7f0b300ce6a0 msgr2=0x7f0b300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 -- 172.21.15.29:0/3136963484 shutdown_connections 2023-10-14T17:00:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.223+0000 7f0b36916640 1 -- 172.21.15.29:0/3136963484 wait complete. 2023-10-14T17:00:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.225+0000 7f0b36916640 1 Processor -- start 2023-10-14T17:00:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.225+0000 7f0b36916640 1 -- start start 2023-10-14T17:00:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b301497f0 0x7f0b30149c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b300c3bc0 0x7f0b300c4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300c5590 con 0x7f0b1c007390 2023-10-14T17:00:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3014a1b0 con 0x7f0b301497f0 2023-10-14T17:00:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b36916640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b300c4980 con 0x7f0b300c3bc0 2023-10-14T17:00:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b35914640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b35914640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54898/0 (socket says 172.21.15.29:54898) 2023-10-14T17:00:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.226+0000 7f0b35914640 1 -- 172.21.15.29:0/3274939760 learned_addr learned my addr 172.21.15.29:0/3274939760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35113640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b301497f0 0x7f0b30149c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b36115640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b300c3bc0 0x7f0b300c4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 -- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b300c3bc0 msgr2=0x7f0b300c4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b300c3bc0 0x7f0b300c4040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 -- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b301497f0 msgr2=0x7f0b30149c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b301497f0 0x7f0b30149c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 -- 172.21.15.29:0/3274939760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b300c4c60 con 0x7f0b1c007390 2023-10-14T17:00:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b35914640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c730 tx=0x7f0b2c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0074e0 con 0x7f0b1c007390 2023-10-14T17:00:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b2c007660 con 0x7f0b1c007390 2023-10-14T17:00:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c004a20 con 0x7f0b1c007390 2023-10-14T17:00:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.227+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b300c4f40 con 0x7f0b1c007390 2023-10-14T17:00:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.228+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b300c4580 con 0x7f0b1c007390 2023-10-14T17:00:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.229+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3004e9b0 con 0x7f0b1c007390 2023-10-14T17:00:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.230+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0b2c004b80 con 0x7f0b1c007390 2023-10-14T17:00:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.231+0000 7f0b267fc640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 0x7f0b1406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.231+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f0b2c088b70 con 0x7f0b1c007390 2023-10-14T17:00:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.231+0000 7f0b35113640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 0x7f0b1406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.232+0000 7f0b35113640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 0x7f0b1406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.235+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b2c053a20 con 0x7f0b1c007390 2023-10-14T17:00:17.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.504+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]} v 0) v1 -- 0x7f0b300b4620 con 0x7f0b1c007390 2023-10-14T17:00:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.573+0000 7f0b267fc640 1 -- 172.21.15.29:0/3274939760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule"]}]=0 v764) v1 ==== 148+0+0 (secure 0 0 0) 0x7f0b2c018020 con 0x7f0b1c007390 2023-10-14T17:00:17.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 msgr2=0x7f0b1406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 0x7f0b1406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0b20007fd0 tx=0x7f0b20009040 comp rx=0 tx=0).stop 2023-10-14T17:00:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 msgr2=0x7f0b30147080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c00c730 tx=0x7f0b2c00cbf0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 shutdown_connections 2023-10-14T17:00:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0b1406afc0 0x7f0b1406d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b300c3bc0 0x7f0b300c4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b301497f0 0x7f0b30149c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 --2- 172.21.15.29:0/3274939760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b1c007390 0x7f0b30147080 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 >> 172.21.15.29:0/3274939760 conn(0x7f0b300ce6a0 msgr2=0x7f0b300c8f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:17.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.576+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 shutdown_connections 2023-10-14T17:00:17.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.577+0000 7f0b36916640 1 -- 172.21.15.29:0/3274939760 wait complete. 2023-10-14T17:00:17.598 INFO:tasks.cephfs_test_runner:Test existence of a scheduled snapshot ... ok 2023-10-14T17:00:17.599 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T17:00:17.599 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:17.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.825+0000 7faac707e640 1 Processor -- start 2023-10-14T17:00:17.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.825+0000 7faac707e640 1 -- start start 2023-10-14T17:00:17.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac707e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faac015c9f0 0x7faac00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac707e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 0x7faac00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac707e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faac0160a90 con 0x7faac0070f20 2023-10-14T17:00:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac707e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac00cdec0 con 0x7faac015c9f0 2023-10-14T17:00:17.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac707e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faac00cd900 con 0x7faac00c9230 2023-10-14T17:00:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac607c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faac015c9f0 0x7faac00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac607c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faac015c9f0 0x7faac00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54908/0 (socket says 172.21.15.29:54908) 2023-10-14T17:00:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac687d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faac0070f20 0x7faac00ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46598/0 (socket says 172.21.15.29:46598) 2023-10-14T17:00:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.827+0000 7faac607c640 1 -- 172.21.15.29:0/2728020819 learned_addr learned my addr 172.21.15.29:0/2728020819 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:17.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac587b640 1 --2- 172.21.15.29:0/2728020819 >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 0x7faac00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac607c640 1 -- 172.21.15.29:0/2728020819 >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 msgr2=0x7faac00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac607c640 1 --2- 172.21.15.29:0/2728020819 >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 0x7faac00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac607c640 1 -- 172.21.15.29:0/2728020819 >> v1:172.21.15.29:6789/0 conn(0x7faac0070f20 legacy=0x7faac00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac607c640 1 -- 172.21.15.29:0/2728020819 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faac00cdbe0 con 0x7faac015c9f0 2023-10-14T17:00:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac507a640 1 -- 172.21.15.29:0/2728020819 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1060078443 0 0) 0x7faac0160a90 con 0x7faac0070f20 2023-10-14T17:00:17.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.828+0000 7faac587b640 1 --2- 172.21.15.29:0/2728020819 >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 0x7faac00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:17.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.829+0000 7faac607c640 1 --2- 172.21.15.29:0/2728020819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac015c9f0 0x7faac00c8cf0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7faac00b1260 tx=0x7faabc02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04de7df342fdbad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.829+0000 7faac507a640 1 -- 172.21.15.29:0/2728020819 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc030870 con 0x7faac015c9f0 2023-10-14T17:00:17.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.829+0000 7faac507a640 1 -- 172.21.15.29:0/2728020819 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc0309d0 con 0x7faac015c9f0 2023-10-14T17:00:17.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.829+0000 7faac507a640 1 -- 172.21.15.29:0/2728020819 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc030d60 con 0x7faac015c9f0 2023-10-14T17:00:17.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 -- 172.21.15.29:0/2728020819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac015c9f0 msgr2=0x7faac00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 --2- 172.21.15.29:0/2728020819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac015c9f0 0x7faac00c8cf0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 crypto rx=0x7faac00b1260 tx=0x7faabc02b9b0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 -- 172.21.15.29:0/2728020819 shutdown_connections 2023-10-14T17:00:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 --2- 172.21.15.29:0/2728020819 >> v2:172.21.15.186:3300/0 conn(0x7faac00c9230 0x7faac00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 --2- 172.21.15.29:0/2728020819 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac015c9f0 0x7faac00c8cf0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 -- 172.21.15.29:0/2728020819 >> 172.21.15.29:0/2728020819 conn(0x7faac00ce6a0 msgr2=0x7faac00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:17.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 -- 172.21.15.29:0/2728020819 shutdown_connections 2023-10-14T17:00:17.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.830+0000 7faac707e640 1 -- 172.21.15.29:0/2728020819 wait complete. 2023-10-14T17:00:17.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.832+0000 7faac707e640 1 Processor -- start 2023-10-14T17:00:17.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.832+0000 7faac707e640 1 -- start start 2023-10-14T17:00:17.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c9230 0x7faac00aeeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015c9f0 0x7faac00af3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac006b830 con 0x7faac0070f20 2023-10-14T17:00:17.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faac006abb0 con 0x7faac015c9f0 2023-10-14T17:00:17.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac707e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faac006ae90 con 0x7faac00c9230 2023-10-14T17:00:17.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac607c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac607c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54914/0 (socket says 172.21.15.29:54914) 2023-10-14T17:00:17.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.833+0000 7faac607c640 1 -- 172.21.15.29:0/2972209242 learned_addr learned my addr 172.21.15.29:0/2972209242 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:17.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac687d640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015c9f0 0x7faac00af3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac587b640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c9230 0x7faac00aeeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 -- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c9230 msgr2=0x7faac00aeeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c9230 0x7faac00aeeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 -- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015c9f0 msgr2=0x7faac00af3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:17.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015c9f0 0x7faac00af3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:17.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 -- 172.21.15.29:0/2972209242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faabc0094d0 con 0x7faac0070f20 2023-10-14T17:00:17.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faac607c640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7faabc02b9f0 tx=0x7faabc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.834+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc030860 con 0x7faac0070f20 2023-10-14T17:00:17.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.835+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faabc039d90 con 0x7faac0070f20 2023-10-14T17:00:17.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.835+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faabc043650 con 0x7faac0070f20 2023-10-14T17:00:17.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.835+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faac006b170 con 0x7faac0070f20 2023-10-14T17:00:17.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.835+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faac014f080 con 0x7faac0070f20 2023-10-14T17:00:17.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.836+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faac006b970 con 0x7faac0070f20 2023-10-14T17:00:17.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.837+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faabc02f3c0 con 0x7faac0070f20 2023-10-14T17:00:17.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.838+0000 7faab6ffd640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 0x7faaa406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:17.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.838+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7faabc0abdc0 con 0x7faac0070f20 2023-10-14T17:00:17.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.838+0000 7faac587b640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 0x7faaa406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:17.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.839+0000 7faac587b640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 0x7faaa406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faac00c6ec0 tx=0x7faab0008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:17.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:17.842+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faac006b970 con 0x7faac0070f20 2023-10-14T17:00:18.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.044+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faac006b970 con 0x7faac0070f20 2023-10-14T17:00:18.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.045+0000 7faab6ffd640 1 -- 172.21.15.29:0/2972209242 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v131) v1 ==== 74+0+14220 (secure 0 0 0) 0x7faac006b970 con 0x7faac0070f20 2023-10-14T17:00:18.046 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:18.047 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":131,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:15.552121+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6837/80282461":"2023-10-15T17:00:12.520237+0000","172.21.15.29:6836/80282461":"2023-10-15T17:00:12.520237+0000","172.21.15.29:0/1686923517":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3236529658":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/681365668":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/2259811471":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6833/1622969171":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3200714056":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3964845078":"2023-10-15T17:00:08.757402+0000","172.21.15.29:6832/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:6833/2416849365":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3986638800":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/354517400":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/3653150355":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/226673613":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3890360705":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/2790183016":"2023-10-15T16:59:41.541911+0000","172.21.15.29:0/1217266964":"2023-10-15T17:00:08.757402+0000","172.21.15.29:0/3481988951":"2023-10-15T16:59:41.541911+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 msgr2=0x7faaa406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 0x7faaa406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faac00c6ec0 tx=0x7faab0008040 comp rx=0 tx=0).stop 2023-10-14T17:00:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 msgr2=0x7faac014eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 crypto rx=0x7faabc02b9f0 tx=0x7faabc004730 comp rx=0 tx=0).stop 2023-10-14T17:00:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 shutdown_connections 2023-10-14T17:00:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7faaa406af90 0x7faaa406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faac015c9f0 0x7faac00af3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faac00c9230 0x7faac00aeeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 --2- 172.21.15.29:0/2972209242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faac0070f20 0x7faac014eb40 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.048+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 >> 172.21.15.29:0/2972209242 conn(0x7faac00ce6a0 msgr2=0x7faac00ca350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:18.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.049+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 shutdown_connections 2023-10-14T17:00:18.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.049+0000 7faac707e640 1 -- 172.21.15.29:0/2972209242 wait complete. 2023-10-14T17:00:18.068 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:18.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:18.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.303+0000 7f55b2bbd640 1 Processor -- start 2023-10-14T17:00:18.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.303+0000 7f55b2bbd640 1 -- start start 2023-10-14T17:00:18.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.304+0000 7f55b2bbd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55ac159ad0 0x7f55ac070f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b2bbd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac0ca270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b2bbd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55ac0cd260 con 0x7f55ac15d650 2023-10-14T17:00:18.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b2bbd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55ac071470 con 0x7f55ac159ad0 2023-10-14T17:00:18.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b2bbd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0ca7b0 con 0x7f55ac0c9de0 2023-10-14T17:00:18.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b1bbb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55ac159ad0 0x7f55ac070f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b13ba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55ac15d650 0x7f55ac06aea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46604/0 (socket says 172.21.15.29:46604) 2023-10-14T17:00:18.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b13ba640 1 -- 172.21.15.29:0/164642730 learned_addr learned my addr 172.21.15.29:0/164642730 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b23bc640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac0ca270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b1bbb640 1 -- 172.21.15.29:0/164642730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 msgr2=0x7f55ac0ca270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b1bbb640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac0ca270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b1bbb640 1 -- 172.21.15.29:0/164642730 >> v1:172.21.15.29:6789/0 conn(0x7f55ac15d650 legacy=0x7f55ac06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b1bbb640 1 -- 172.21.15.29:0/164642730 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55ac15de30 con 0x7f55ac159ad0 2023-10-14T17:00:18.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.305+0000 7f55b23bc640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:18.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.307+0000 7f55b1bbb640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac070f30 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009d90 tx=0x7f55a802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96dc0ad6358abf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.307+0000 7f55b0bb9640 1 -- 172.21.15.29:0/164642730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a802cc60 con 0x7f55ac159ad0 2023-10-14T17:00:18.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.307+0000 7f55b0bb9640 1 -- 172.21.15.29:0/164642730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55a802cdc0 con 0x7f55ac159ad0 2023-10-14T17:00:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.307+0000 7f55b0bb9640 1 -- 172.21.15.29:0/164642730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55a8034660 con 0x7f55ac159ad0 2023-10-14T17:00:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 -- 172.21.15.29:0/164642730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 msgr2=0x7f55ac070f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac070f30 secure :-1 s=READY pgs=1538 cs=0 l=1 rev1=1 crypto rx=0x7f55a8009d90 tx=0x7f55a802beb0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 -- 172.21.15.29:0/164642730 shutdown_connections 2023-10-14T17:00:18.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac0ca270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/164642730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac070f30 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 -- 172.21.15.29:0/164642730 >> 172.21.15.29:0/164642730 conn(0x7f55ac0b0170 msgr2=0x7f55ac0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:18.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 -- 172.21.15.29:0/164642730 shutdown_connections 2023-10-14T17:00:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.308+0000 7f55b2bbd640 1 -- 172.21.15.29:0/164642730 wait complete. 2023-10-14T17:00:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.310+0000 7f55b2bbd640 1 Processor -- start 2023-10-14T17:00:18.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.310+0000 7f55b2bbd640 1 -- start start 2023-10-14T17:00:18.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac150bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 0x7f55ac1574e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0b3000 con 0x7f55ac159ad0 2023-10-14T17:00:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac0b3840 con 0x7f55ac15d650 2023-10-14T17:00:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b2bbd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55ac151110 con 0x7f55ac0c9de0 2023-10-14T17:00:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54934/0 (socket says 172.21.15.29:54934) 2023-10-14T17:00:18.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 -- 172.21.15.29:0/2680643636 learned_addr learned my addr 172.21.15.29:0/2680643636 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:18.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b1bbb640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac150bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b23bc640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 0x7f55ac1574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 -- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 msgr2=0x7f55ac150bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac150bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 -- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 msgr2=0x7f55ac1574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 0x7f55ac1574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b13ba640 1 -- 172.21.15.29:0/2680643636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a8009a70 con 0x7f55ac159ad0 2023-10-14T17:00:18.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.311+0000 7f55b23bc640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 0x7f55ac1574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:18.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55b13ba640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c4a0 tx=0x7f559c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c005ce0 con 0x7f55ac159ad0 2023-10-14T17:00:18.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c005e40 con 0x7f55ac159ad0 2023-10-14T17:00:18.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f559c007270 con 0x7f55ac159ad0 2023-10-14T17:00:18.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55ac1f36c0 con 0x7f55ac159ad0 2023-10-14T17:00:18.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.312+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55ac1f33e0 con 0x7f55ac159ad0 2023-10-14T17:00:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.314+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55ac0b39f0 con 0x7f55ac159ad0 2023-10-14T17:00:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.314+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f559c007490 con 0x7f55ac159ad0 2023-10-14T17:00:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.315+0000 7f55a27fc640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 0x7f559006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.316+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f559c088c90 con 0x7f55ac159ad0 2023-10-14T17:00:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.316+0000 7f55b1bbb640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 0x7f559006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.316+0000 7f55b1bbb640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 0x7f559006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55a8036000 tx=0x7f55a8030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.320+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f559c053bf0 con 0x7f55ac159ad0 2023-10-14T17:00:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.515+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f55ac04e9b0 con 0x7f55ac159ad0 2023-10-14T17:00:18.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.516+0000 7f55a27fc640 1 -- 172.21.15.29:0/2680643636 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v131) v1 ==== 93+0+10 (secure 0 0 0) 0x7f559c017020 con 0x7f55ac159ad0 2023-10-14T17:00:18.517 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 msgr2=0x7f559006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 0x7f559006d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f55a8036000 tx=0x7f55a8030000 comp rx=0 tx=0).stop 2023-10-14T17:00:18.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 msgr2=0x7f55ac0b3cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 crypto rx=0x7f559c00c4a0 tx=0x7f559c00c960 comp rx=0 tx=0).stop 2023-10-14T17:00:18.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 shutdown_connections 2023-10-14T17:00:18.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f559006af90 0x7f559006d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55ac15d650 0x7f55ac1574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55ac159ad0 0x7f55ac0b3cd0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 --2- 172.21.15.29:0/2680643636 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55ac0c9de0 0x7f55ac150bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.518+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 >> 172.21.15.29:0/2680643636 conn(0x7f55ac0b0170 msgr2=0x7f55ac06c080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.519+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 shutdown_connections 2023-10-14T17:00:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.519+0000 7f55b2bbd640 1 -- 172.21.15.29:0/2680643636 wait complete. 2023-10-14T17:00:18.540 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-14T17:00:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.774+0000 7fcb36b47640 1 Processor -- start 2023-10-14T17:00:18.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.774+0000 7fcb36b47640 1 -- start start 2023-10-14T17:00:18.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.776+0000 7fcb36b47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 0x7fcb3006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.776+0000 7fcb36b47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcb3015e540 0x7fcb300b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.776+0000 7fcb36b47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb300cd630 con 0x7fcb3015e0c0 2023-10-14T17:00:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.776+0000 7fcb36b47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb300ce1b0 con 0x7fcb3015e540 2023-10-14T17:00:18.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.776+0000 7fcb36b47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb3006cd10 con 0x7fcb3006c3f0 2023-10-14T17:00:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb348e3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 0x7fcb3006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb348e3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 0x7fcb3006c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:49674/0 (socket says 172.21.15.29:49674) 2023-10-14T17:00:18.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb348e3640 1 -- 172.21.15.29:0/155732790 learned_addr learned my addr 172.21.15.29:0/155732790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb350e4640 1 --2- 172.21.15.29:0/155732790 >> v2:172.21.15.186:3300/0 conn(0x7fcb3015e540 0x7fcb300b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb350e4640 1 -- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 msgr2=0x7fcb3006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb350e4640 1 --2- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 0x7fcb3006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb350e4640 1 -- 172.21.15.29:0/155732790 >> v1:172.21.15.186:6789/0 conn(0x7fcb3015e0c0 legacy=0x7fcb300b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.777+0000 7fcb350e4640 1 -- 172.21.15.29:0/155732790 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb300d2170 con 0x7fcb3015e540 2023-10-14T17:00:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.779+0000 7fcb350e4640 1 --2- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb300b1710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcb280097a0 tx=0x7fcb2802b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9e6f80f6db77574 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.779+0000 7fcb2f7fe640 1 -- 172.21.15.29:0/155732790 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb2802d040 con 0x7fcb3015e540 2023-10-14T17:00:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb2f7fe640 1 -- 172.21.15.29:0/155732790 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb28031810 con 0x7fcb3015e540 2023-10-14T17:00:18.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb2f7fe640 1 -- 172.21.15.29:0/155732790 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb28031ba0 con 0x7fcb3015e540 2023-10-14T17:00:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 -- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 msgr2=0x7fcb300b1710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 --2- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb300b1710 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fcb280097a0 tx=0x7fcb2802b8b0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 -- 172.21.15.29:0/155732790 shutdown_connections 2023-10-14T17:00:18.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 --2- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb300b1710 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 --2- 172.21.15.29:0/155732790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3006c3f0 0x7fcb3006c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.780+0000 7fcb36b47640 1 -- 172.21.15.29:0/155732790 >> 172.21.15.29:0/155732790 conn(0x7fcb300d2cb0 msgr2=0x7fcb300d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.781+0000 7fcb36b47640 1 -- 172.21.15.29:0/155732790 shutdown_connections 2023-10-14T17:00:18.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.781+0000 7fcb36b47640 1 -- 172.21.15.29:0/155732790 wait complete. 2023-10-14T17:00:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.782+0000 7fcb36b47640 1 Processor -- start 2023-10-14T17:00:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.783+0000 7fcb36b47640 1 -- start start 2023-10-14T17:00:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3015e0c0 0x7fcb30160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb3015c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300b73e0 con 0x7fcb3006c3f0 2023-10-14T17:00:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300c9cf0 con 0x7fcb3015e540 2023-10-14T17:00:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb36b47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb300b75c0 con 0x7fcb3015e0c0 2023-10-14T17:00:18.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54946/0 (socket says 172.21.15.29:54946) 2023-10-14T17:00:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 -- 172.21.15.29:0/2975997741 learned_addr learned my addr 172.21.15.29:0/2975997741 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb2ffff640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3015e0c0 0x7fcb30160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 -- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3015e0c0 msgr2=0x7fcb30160a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb350e4640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb3015c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3015e0c0 0x7fcb30160a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 -- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 msgr2=0x7fcb3015c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb3015c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.784+0000 7fcb348e3640 1 -- 172.21.15.29:0/2975997741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb28009400 con 0x7fcb3006c3f0 2023-10-14T17:00:18.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb350e4640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb3015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb348e3640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009cf0 tx=0x7fcb2000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb20009b60 con 0x7fcb3006c3f0 2023-10-14T17:00:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb2000f040 con 0x7fcb3006c3f0 2023-10-14T17:00:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb300b78a0 con 0x7fcb3006c3f0 2023-10-14T17:00:18.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb200092e0 con 0x7fcb3006c3f0 2023-10-14T17:00:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.785+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb3015ab30 con 0x7fcb3006c3f0 2023-10-14T17:00:18.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.787+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb3015acf0 con 0x7fcb3006c3f0 2023-10-14T17:00:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.787+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcb20014480 con 0x7fcb3006c3f0 2023-10-14T17:00:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.789+0000 7fcb2d7fa640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 0x7fcb1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.789+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7fcb20089fe0 con 0x7fcb3006c3f0 2023-10-14T17:00:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.789+0000 7fcb2ffff640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 0x7fcb1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:18.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.790+0000 7fcb2ffff640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 0x7fcb1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb30156fd0 tx=0x7fcb18009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:18.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:18.793+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcb20054e20 con 0x7fcb3006c3f0 2023-10-14T17:00:19.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.059+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fcb3004e160 con 0x7fcb3006c3f0 2023-10-14T17:00:19.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.587+0000 7fcb2d7fa640 1 -- 172.21.15.29:0/2975997741 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v766) v1 ==== 163+0+0 (secure 0 0 0) 0x7fcb20058440 con 0x7fcb3006c3f0 2023-10-14T17:00:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 msgr2=0x7fcb1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 0x7fcb1006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcb30156fd0 tx=0x7fcb18009040 comp rx=0 tx=0).stop 2023-10-14T17:00:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 msgr2=0x7fcb30156530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 secure :-1 s=READY pgs=1540 cs=0 l=1 rev1=1 crypto rx=0x7fcb20009cf0 tx=0x7fcb2000c490 comp rx=0 tx=0).stop 2023-10-14T17:00:19.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 shutdown_connections 2023-10-14T17:00:19.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fcb1006af90 0x7fcb1006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcb3015e540 0x7fcb3015c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcb3015e0c0 0x7fcb30160a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 --2- 172.21.15.29:0/2975997741 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcb3006c3f0 0x7fcb30156530 unknown :-1 s=CLOSED pgs=1540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 >> 172.21.15.29:0/2975997741 conn(0x7fcb300d2cb0 msgr2=0x7fcb300b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:19.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 shutdown_connections 2023-10-14T17:00:19.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.591+0000 7fcb36b47640 1 -- 172.21.15.29:0/2975997741 wait complete. 2023-10-14T17:00:19.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T17:00:19.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.851+0000 7f6977fff640 1 Processor -- start 2023-10-14T17:00:19.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.851+0000 7f6977fff640 1 -- start start 2023-10-14T17:00:19.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.852+0000 7f6977fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.852+0000 7f6977fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f69780d0be0 0x7f69780cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.852+0000 7f6977fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f697815d150 con 0x7f697815cd70 2023-10-14T17:00:19.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.852+0000 7f6977fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69780d1180 con 0x7f69780d0be0 2023-10-14T17:00:19.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.852+0000 7f6977fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69780d1460 con 0x7f69780b8480 2023-10-14T17:00:19.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f6976ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f697815cd70 0x7f69780d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46610/0 (socket says 172.21.15.29:46610) 2023-10-14T17:00:19.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f6976ffd640 1 -- 172.21.15.29:0/14628163 learned_addr learned my addr 172.21.15.29:0/14628163 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:19.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69777fe640 1 --2- 172.21.15.29:0/14628163 >> v2:172.21.15.29:3300/0 conn(0x7f69780d0be0 0x7f69780cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f696ffff640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69767fc640 1 -- 172.21.15.29:0/14628163 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908130349 0 0) 0x7f697815d150 con 0x7f697815cd70 2023-10-14T17:00:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69777fe640 1 -- 172.21.15.29:0/14628163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 msgr2=0x7f69780cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69777fe640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69777fe640 1 -- 172.21.15.29:0/14628163 >> v1:172.21.15.29:6789/0 conn(0x7f697815cd70 legacy=0x7f69780d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.853+0000 7f69777fe640 1 -- 172.21.15.29:0/14628163 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69780d1740 con 0x7f69780d0be0 2023-10-14T17:00:19.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f69777fe640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f69780cdd00 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f6970009d90 tx=0x7f697002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f24970519dc8e39 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f69767fc640 1 -- 172.21.15.29:0/14628163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697002cc50 con 0x7f69780d0be0 2023-10-14T17:00:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f69767fc640 1 -- 172.21.15.29:0/14628163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f697002cdb0 con 0x7f69780d0be0 2023-10-14T17:00:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f69767fc640 1 -- 172.21.15.29:0/14628163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6970034660 con 0x7f69780d0be0 2023-10-14T17:00:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f6977fff640 1 -- 172.21.15.29:0/14628163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 msgr2=0x7f69780cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.855+0000 7f6977fff640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f69780cdd00 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 crypto rx=0x7f6970009d90 tx=0x7f697002bea0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 -- 172.21.15.29:0/14628163 shutdown_connections 2023-10-14T17:00:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f69780cdd00 unknown :-1 s=CLOSED pgs=1542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 --2- 172.21.15.29:0/14628163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 -- 172.21.15.29:0/14628163 >> 172.21.15.29:0/14628163 conn(0x7f69780cf570 msgr2=0x7f69780cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:19.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 -- 172.21.15.29:0/14628163 shutdown_connections 2023-10-14T17:00:19.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.856+0000 7f6977fff640 1 -- 172.21.15.29:0/14628163 wait complete. 2023-10-14T17:00:19.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.857+0000 7f6977fff640 1 Processor -- start 2023-10-14T17:00:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.858+0000 7f6977fff640 1 -- start start 2023-10-14T17:00:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.858+0000 7f6977fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cc220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6977fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6977fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 0x7f69781f1a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6977fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69780cbe00 con 0x7f69780d0be0 2023-10-14T17:00:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6977fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6978160360 con 0x7f697815cd70 2023-10-14T17:00:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6977fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6978145a70 con 0x7f69780b8480 2023-10-14T17:00:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54976/0 (socket says 172.21.15.29:54976) 2023-10-14T17:00:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 -- 172.21.15.29:0/2659147301 learned_addr learned my addr 172.21.15.29:0/2659147301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f6976ffd640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cc220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f69777fe640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 0x7f69781f1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 -- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 msgr2=0x7f69780cc220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cc220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 -- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 msgr2=0x7f69781f1a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 0x7f69781f1a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f696ffff640 1 -- 172.21.15.29:0/2659147301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6970009a70 con 0x7f69780d0be0 2023-10-14T17:00:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.859+0000 7f69777fe640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 0x7f69781f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:19.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f696ffff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f696000d620 tx=0x7f696000dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000b620 con 0x7f69780d0be0 2023-10-14T17:00:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696000b780 con 0x7f69780d0be0 2023-10-14T17:00:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696000bb30 con 0x7f69780d0be0 2023-10-14T17:00:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69780cb180 con 0x7f69780d0be0 2023-10-14T17:00:19.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.860+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69780cb740 con 0x7f69780d0be0 2023-10-14T17:00:19.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.861+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f697804e9b0 con 0x7f69780d0be0 2023-10-14T17:00:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.863+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f696000bc90 con 0x7f69780d0be0 2023-10-14T17:00:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.863+0000 7f696f7fe640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 0x7f695806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.864+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f6960089cc0 con 0x7f69780d0be0 2023-10-14T17:00:19.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.864+0000 7f6976ffd640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 0x7f695806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.865+0000 7f6976ffd640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 0x7f695806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6968000f60 tx=0x7f6968007b50 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:19.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:19.868+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6960090080 con 0x7f69780d0be0 2023-10-14T17:00:20.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.067+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f69781f1fa0 con 0x7f69780d0be0 2023-10-14T17:00:20.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.072+0000 7f696f7fe640 1 -- 172.21.15.29:0/2659147301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v46)=0 v46) v1 ==== 122+0+0 (secure 0 0 0) 0x7f6960054c00 con 0x7f69780d0be0 2023-10-14T17:00:20.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 msgr2=0x7f695806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 0x7f695806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6968000f60 tx=0x7f6968007b50 comp rx=0 tx=0).stop 2023-10-14T17:00:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 msgr2=0x7f697815fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 secure :-1 s=READY pgs=1543 cs=0 l=1 rev1=1 crypto rx=0x7f696000d620 tx=0x7f696000dae0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 shutdown_connections 2023-10-14T17:00:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.075+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f695806af90 0x7f695806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f697815cd70 0x7f69781f1a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f69780d0be0 0x7f697815fe20 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 --2- 172.21.15.29:0/2659147301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f69780b8480 0x7f69780cc220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 >> 172.21.15.29:0/2659147301 conn(0x7f69780cf570 msgr2=0x7f697815bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 shutdown_connections 2023-10-14T17:00:20.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.076+0000 7f6977fff640 1 -- 172.21.15.29:0/2659147301 wait complete. 2023-10-14T17:00:20.096 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:00:20.097 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:20.217 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:20.347 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:00:20.348 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:20.348 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:20.409 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:00:20.409 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:20.409 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:20.470 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:20.470 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:20.471 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:20.534 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:00:20.535 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:20.535 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:20.597 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:20.598 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T17:00:20.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.832+0000 7f7dbe3ba640 1 Processor -- start 2023-10-14T17:00:20.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.832+0000 7f7dbe3ba640 1 -- start start 2023-10-14T17:00:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbe3ba640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7db815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbe3ba640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db815aef0 con 0x7f7db806c100 2023-10-14T17:00:20.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbe3ba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db806cc60 con 0x7f7db8159950 2023-10-14T17:00:20.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbe3ba640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db815e490 con 0x7f7db8070b60 2023-10-14T17:00:20.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbdbb9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7db806c100 0x7f7db806c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46614/0 (socket says 172.21.15.29:46614) 2023-10-14T17:00:20.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.834+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/2181028549 learned_addr learned my addr 172.21.15.29:0/2181028549 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:20.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7dbcbb7640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7db815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640184265 0 0) 0x7f7db0001490 con 0x7f7db806c100 2023-10-14T17:00:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da4003520 con 0x7f7db806c100 2023-10-14T17:00:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3916811096 0 0) 0x7f7da4003520 con 0x7f7db806c100 2023-10-14T17:00:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 msgr2=0x7f7db815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:20.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7db815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 >> v1:172.21.15.186:6789/0 conn(0x7f7db8159950 legacy=0x7f7db8070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db815d910 con 0x7f7db806c100 2023-10-14T17:00:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7db0001b80 con 0x7f7db806c100 2023-10-14T17:00:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 >> v1:172.21.15.29:6789/0 conn(0x7f7db806c100 legacy=0x7f7db806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da40039a0 0x7f7da4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7da40047c0 0x7f7da4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7da4005230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db815e490 con 0x7f7da40039a0 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7da400bb40 con 0x7f7da40047c0 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.835+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7da400df10 con 0x7f7db8070b60 2023-10-14T17:00:20.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da40039a0 0x7f7da4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbcbb7640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7da40047c0 0x7f7da4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbdbb9640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7da4005230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 -- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 msgr2=0x7f7da4005230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7da4005230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 -- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7da40047c0 msgr2=0x7f7da4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7da40047c0 0x7f7da4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbd3b8640 1 -- 172.21.15.29:0/2181028549 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da4003520 con 0x7f7da40039a0 2023-10-14T17:00:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbcbb7640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7da40047c0 0x7f7da4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:20.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.836+0000 7f7dbdbb9640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7db8070b60 0x7f7da4005230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.837+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da40039a0 0x7f7da4003dc0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7db4001750 tx=0x7f7da40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=724c9fb53fe1c874 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40052d0 con 0x7f7da40039a0 2023-10-14T17:00:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db402e050 con 0x7f7da40039a0 2023-10-14T17:00:20.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7daffff640 1 -- 172.21.15.29:0/2181028549 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db40035f0 con 0x7f7da40039a0 2023-10-14T17:00:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/2181028549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da40039a0 msgr2=0x7f7da4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/2181028549 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7da40039a0 0x7f7da4003dc0 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 crypto rx=0x7f7db4001750 tx=0x7f7da40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/2181028549 reap_dead start 2023-10-14T17:00:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/2181028549 shutdown_connections 2023-10-14T17:00:20.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/2181028549 >> 172.21.15.29:0/2181028549 conn(0x7f7db80ce6a0 msgr2=0x7f7db80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.838+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/2181028549 shutdown_connections 2023-10-14T17:00:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.839+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/2181028549 wait complete. 2023-10-14T17:00:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.840+0000 7f7dbe3ba640 1 Processor -- start 2023-10-14T17:00:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.840+0000 7f7dbe3ba640 1 -- start start 2023-10-14T17:00:20.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.841+0000 7f7dbe3ba640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7da40039a0 0x7f7db80c6e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.841+0000 7f7dbe3ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7db81f3690 0x7f7db80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.841+0000 7f7dbe3ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbe3ba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db80ad250 con 0x7f7db80acdd0 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbe3ba640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db81f2980 con 0x7f7db81f3690 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbe3ba640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7db81f2bd0 con 0x7f7da40039a0 2023-10-14T17:00:20.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54984/0 (socket says 172.21.15.29:54984) 2023-10-14T17:00:20.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/321330093 learned_addr learned my addr 172.21.15.29:0/321330093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:20.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7da40039a0 0x7f7db80c6e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbcbb7640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7db81f3690 0x7f7db80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7da40039a0 msgr2=0x7f7db80c6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7da40039a0 0x7f7db80c6e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7db81f3690 msgr2=0x7f7db80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7db81f3690 0x7f7db80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.842+0000 7f7dbdbb9640 1 -- 172.21.15.29:0/321330093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db81f6c50 con 0x7f7db80acdd0 2023-10-14T17:00:20.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dbdbb9640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7da4004740 tx=0x7f7db000b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db000bdb0 con 0x7f7db80acdd0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7db0004b60 con 0x7f7db80acdd0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db81f2e90 con 0x7f7db80acdd0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7db0007070 con 0x7f7db80acdd0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.843+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db80c7370 con 0x7f7db80acdd0 2023-10-14T17:00:20.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.844+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db81f33b0 con 0x7f7db80acdd0 2023-10-14T17:00:20.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.845+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7db00071d0 con 0x7f7db80acdd0 2023-10-14T17:00:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.846+0000 7f7dadffb640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 0x7f7d9c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.846+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f7db0089b20 con 0x7f7db80acdd0 2023-10-14T17:00:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.847+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 0x7f7d9c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.847+0000 7f7dbd3b8640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 0x7f7d9c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7db4033000 tx=0x7f7db4009030 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:20.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:20.851+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7db81f33b0 con 0x7f7db80acdd0 2023-10-14T17:00:21.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.087+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7db81f33b0 con 0x7f7db80acdd0 2023-10-14T17:00:21.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.088+0000 7f7dadffb640 1 -- 172.21.15.29:0/321330093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 59 v59) v1 ==== 94+0+3705 (secure 0 0 0) 0x7f7db00548c0 con 0x7f7db80acdd0 2023-10-14T17:00:21.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:21.089 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":59,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5813,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1192161367","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1192161367},{"type":"v1","addr":"172.21.15.186:6833","nonce":1192161367}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":5819,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2025683014","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2025683014},{"type":"v1","addr":"172.21.15.186:6835","nonce":2025683014}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":6340,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1360201357","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1360201357},{"type":"v1","addr":"172.21.15.29:6835","nonce":1360201357}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":52},{"gid":8442,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1457539168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1457539168},{"type":"v1","addr":"172.21.15.29:6837","nonce":1457539168}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":59}],"filesystems":[]} 2023-10-14T17:00:21.090 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 59 2023-10-14T17:00:21.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 msgr2=0x7f7d9c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 0x7f7d9c06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7db4033000 tx=0x7f7db4009030 comp rx=0 tx=0).stop 2023-10-14T17:00:21.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 msgr2=0x7f7db81f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 crypto rx=0x7f7da4004740 tx=0x7f7db000b4a0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 shutdown_connections 2023-10-14T17:00:21.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f7d9c06af90 0x7f7d9c06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7db80acdd0 0x7f7db81f5fb0 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7db81f3690 0x7f7db80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 --2- 172.21.15.29:0/321330093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7da40039a0 0x7f7db80c6e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.091+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 >> 172.21.15.29:0/321330093 conn(0x7f7db80ce6a0 msgr2=0x7f7db80c9620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:21.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.092+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 shutdown_connections 2023-10-14T17:00:21.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.092+0000 7f7dbe3ba640 1 -- 172.21.15.29:0/321330093 wait complete. 2023-10-14T17:00:21.113 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T17:00:21.113 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T17:00:21.113 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T17:00:21.114 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:00:21.114 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T17:00:21.114 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T17:00:21.115 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T17:00:21.115 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:00:21.115 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T17:00:21.115 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T17:00:21.116 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T17:00:21.116 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:00:21.116 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T17:00:21.117 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T17:00:21.117 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T17:00:21.117 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:00:21.133 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T17:00:21.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T17:00:21.134 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T17:00:21.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T17:00:21.138 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T17:00:21.138 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T17:00:21.141 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T17:00:21.141 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T17:00:21.182 INFO:tasks.ceph.mds.a:Started 2023-10-14T17:00:21.183 DEBUG:teuthology.parallel:result is None 2023-10-14T17:00:21.188 INFO:tasks.ceph.mds.d:Started 2023-10-14T17:00:21.189 DEBUG:teuthology.parallel:result is None 2023-10-14T17:00:21.190 INFO:tasks.ceph.mds.c:Started 2023-10-14T17:00:21.190 DEBUG:teuthology.parallel:result is None 2023-10-14T17:00:21.196 INFO:tasks.ceph.mds.b:Started 2023-10-14T17:00:21.197 DEBUG:teuthology.parallel:result is None 2023-10-14T17:00:21.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T17:00:21.348 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:00:21.351 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T17:00:21.352 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T17:00:21.357 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T17:00:21.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.390+0000 7f1f85906640 1 Processor -- start 2023-10-14T17:00:21.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.390+0000 7f1f85906640 1 -- start start 2023-10-14T17:00:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85906640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1f800c9060 0x7f1f8006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85906640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f800c9440 0x7f1f8006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85906640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f800ca560 con 0x7f1f8006c9e0 2023-10-14T17:00:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85906640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f800c9950 con 0x7f1f800c9440 2023-10-14T17:00:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85906640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f800c9c30 con 0x7f1f800c9060 2023-10-14T17:00:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.391+0000 7f1f85105640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f800c9440 0x7f1f8006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f800c9440 0x7f1f8006f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54938/0 (socket says 172.21.15.29:54938) 2023-10-14T17:00:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 -- 172.21.15.29:0/3290163925 learned_addr learned my addr 172.21.15.29:0/3290163925 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:21.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f84904640 1 --2- 172.21.15.29:0/3290163925 >> v2:172.21.15.186:3300/0 conn(0x7f1f800c9060 0x7f1f8006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 -- 172.21.15.29:0/3290163925 >> v2:172.21.15.186:3300/0 conn(0x7f1f800c9060 msgr2=0x7f1f8006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 --2- 172.21.15.29:0/3290163925 >> v2:172.21.15.186:3300/0 conn(0x7f1f800c9060 0x7f1f8006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 -- 172.21.15.29:0/3290163925 >> v1:172.21.15.186:6789/0 conn(0x7f1f8006c9e0 legacy=0x7f1f80164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.392+0000 7f1f85105640 1 -- 172.21.15.29:0/3290163925 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f800c9f10 con 0x7f1f800c9440 2023-10-14T17:00:21.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.393+0000 7f1f85105640 1 --2- 172.21.15.29:0/3290163925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800c9440 0x7f1f8006f980 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006d490 tx=0x7f1f7802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a86f4cd5a489ad3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:21.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.393+0000 7f1f7f7fe640 1 -- 172.21.15.29:0/3290163925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7802cb20 con 0x7f1f800c9440 2023-10-14T17:00:21.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.393+0000 7f1f7f7fe640 1 -- 172.21.15.29:0/3290163925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7802cc80 con 0x7f1f800c9440 2023-10-14T17:00:21.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.393+0000 7f1f7f7fe640 1 -- 172.21.15.29:0/3290163925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f780343f0 con 0x7f1f800c9440 2023-10-14T17:00:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 -- 172.21.15.29:0/3290163925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800c9440 msgr2=0x7f1f8006f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 --2- 172.21.15.29:0/3290163925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800c9440 0x7f1f8006f980 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 crypto rx=0x7f1f8006d490 tx=0x7f1f7802bf80 comp rx=0 tx=0).stop 2023-10-14T17:00:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 -- 172.21.15.29:0/3290163925 shutdown_connections 2023-10-14T17:00:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 --2- 172.21.15.29:0/3290163925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f800c9440 0x7f1f8006f980 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 --2- 172.21.15.29:0/3290163925 >> v2:172.21.15.186:3300/0 conn(0x7f1f800c9060 0x7f1f8006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 -- 172.21.15.29:0/3290163925 >> 172.21.15.29:0/3290163925 conn(0x7f1f800ce6a0 msgr2=0x7f1f800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 -- 172.21.15.29:0/3290163925 shutdown_connections 2023-10-14T17:00:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.394+0000 7f1f85906640 1 -- 172.21.15.29:0/3290163925 wait complete. 2023-10-14T17:00:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.396+0000 7f1f85906640 1 Processor -- start 2023-10-14T17:00:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.396+0000 7f1f85906640 1 -- start start 2023-10-14T17:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f800c9060 0x7f1f801f78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f800c9440 0x7f1f800c7000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8015e4e0 con 0x7f1f8006c9e0 2023-10-14T17:00:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800c66c0 con 0x7f1f800c9440 2023-10-14T17:00:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85906640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f800c2090 con 0x7f1f800c9060 2023-10-14T17:00:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f84904640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f84904640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54940/0 (socket says 172.21.15.29:54940) 2023-10-14T17:00:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f84904640 1 -- 172.21.15.29:0/2070004518 learned_addr learned my addr 172.21.15.29:0/2070004518 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f7ffff640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f800c9060 0x7f1f801f78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.397+0000 7f1f85105640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f800c9440 0x7f1f800c7000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 -- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f800c9060 msgr2=0x7f1f801f78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f800c9060 0x7f1f801f78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 -- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f800c9440 msgr2=0x7f1f800c7000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f800c9440 0x7f1f800c7000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 -- 172.21.15.29:0/2070004518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f78009b30 con 0x7f1f8006c9e0 2023-10-14T17:00:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f84904640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400e9b0 tx=0x7f1f7400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f74009670 con 0x7f1f8006c9e0 2023-10-14T17:00:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f7400c9a0 con 0x7f1f8006c9e0 2023-10-14T17:00:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.398+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f7400cd50 con 0x7f1f8006c9e0 2023-10-14T17:00:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.399+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f800c2330 con 0x7f1f8006c9e0 2023-10-14T17:00:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.399+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f800c28f0 con 0x7f1f8006c9e0 2023-10-14T17:00:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.400+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f80070210 con 0x7f1f8006c9e0 2023-10-14T17:00:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.401+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f74013470 con 0x7f1f8006c9e0 2023-10-14T17:00:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.402+0000 7f1f7d7fa640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 0x7f1f6406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.402+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 5789+0+0 (secure 0 0 0) 0x7f1f740898b0 con 0x7f1f8006c9e0 2023-10-14T17:00:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.402+0000 7f1f7ffff640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 0x7f1f6406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.403+0000 7f1f7ffff640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 0x7f1f6406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f801f9c70 tx=0x7f1f6c009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.406+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f74054940 con 0x7f1f8006c9e0 2023-10-14T17:00:21.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:21.619+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f1f8004e9b0 con 0x7f1f8006c9e0 2023-10-14T17:00:22.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.633+0000 7f1f7d7fa640 1 -- 172.21.15.29:0/2070004518 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f1f74017080 con 0x7f1f8006c9e0 2023-10-14T17:00:22.634 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T17:00:22.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 msgr2=0x7f1f6406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 0x7f1f6406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f1f801f9c70 tx=0x7f1f6c009040 comp rx=0 tx=0).stop 2023-10-14T17:00:22.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 msgr2=0x7f1f800c6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 secure :-1 s=READY pgs=1554 cs=0 l=1 rev1=1 crypto rx=0x7f1f7400e9b0 tx=0x7f1f7400ee70 comp rx=0 tx=0).stop 2023-10-14T17:00:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 shutdown_connections 2023-10-14T17:00:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f1f6406af90 0x7f1f6406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f800c9440 0x7f1f800c7000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f800c9060 0x7f1f801f78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 --2- 172.21.15.29:0/2070004518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f8006c9e0 0x7f1f800c6180 unknown :-1 s=CLOSED pgs=1554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 >> 172.21.15.29:0/2070004518 conn(0x7f1f800ce6a0 msgr2=0x7f1f800adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 shutdown_connections 2023-10-14T17:00:22.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.637+0000 7f1f85906640 1 -- 172.21.15.29:0/2070004518 wait complete. 2023-10-14T17:00:22.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T17:00:22.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.880+0000 7f6c6f9dc640 1 Processor -- start 2023-10-14T17:00:22.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.880+0000 7f6c6f9dc640 1 -- start start 2023-10-14T17:00:22.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.881+0000 7f6c6f9dc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6806c3f0 0x7f6c6806c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6f9dc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c680cd630 con 0x7f6c6815e540 2023-10-14T17:00:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6f9dc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c680ce1b0 con 0x7f6c6815e0c0 2023-10-14T17:00:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6f9dc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c6806cd10 con 0x7f6c6806c3f0 2023-10-14T17:00:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6806c3f0 0x7f6c6806c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:22.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c6806c3f0 0x7f6c6806c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54942/0 (socket says 172.21.15.29:54942) 2023-10-14T17:00:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 -- 172.21.15.29:0/3925344967 learned_addr learned my addr 172.21.15.29:0/3925344967 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:22.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 -- 172.21.15.29:0/3925344967 >> v1:172.21.15.186:6789/0 conn(0x7f6c6815e0c0 legacy=0x7f6c680b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 -- 172.21.15.29:0/3925344967 >> v1:172.21.15.29:6789/0 conn(0x7f6c6815e540 legacy=0x7f6c680b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.882+0000 7f6c6d778640 1 -- 172.21.15.29:0/3925344967 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c680d2170 con 0x7f6c6806c3f0 2023-10-14T17:00:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.884+0000 7f6c6d778640 1 --2- 172.21.15.29:0/3925344967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6806c3f0 0x7f6c6806c7d0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009870 tx=0x7f6c6402b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a90baa31d5794165 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:22.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.884+0000 7f6c5ffff640 1 -- 172.21.15.29:0/3925344967 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64030850 con 0x7f6c6806c3f0 2023-10-14T17:00:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.884+0000 7f6c5ffff640 1 -- 172.21.15.29:0/3925344967 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c640309b0 con 0x7f6c6806c3f0 2023-10-14T17:00:22.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.884+0000 7f6c5ffff640 1 -- 172.21.15.29:0/3925344967 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64030d40 con 0x7f6c6806c3f0 2023-10-14T17:00:22.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3925344967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6806c3f0 msgr2=0x7f6c6806c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3925344967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6806c3f0 0x7f6c6806c7d0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 crypto rx=0x7f6c64009870 tx=0x7f6c6402b990 comp rx=0 tx=0).stop 2023-10-14T17:00:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3925344967 shutdown_connections 2023-10-14T17:00:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3925344967 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6806c3f0 0x7f6c6806c7d0 unknown :-1 s=CLOSED pgs=1556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3925344967 >> 172.21.15.29:0/3925344967 conn(0x7f6c680d2cb0 msgr2=0x7f6c680d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:22.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3925344967 shutdown_connections 2023-10-14T17:00:22.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.885+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3925344967 wait complete. 2023-10-14T17:00:22.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.887+0000 7f6c6f9dc640 1 Processor -- start 2023-10-14T17:00:22.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.887+0000 7f6c6f9dc640 1 -- start start 2023-10-14T17:00:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c6806c3f0 0x7f6c681f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c6815e0c0 0x7f6c680c9f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:22.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c681e8350 con 0x7f6c6815e540 2023-10-14T17:00:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c680cc160 con 0x7f6c6815e0c0 2023-10-14T17:00:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6f9dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c68159200 con 0x7f6c6806c3f0 2023-10-14T17:00:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:22.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54958/0 (socket says 172.21.15.29:54958) 2023-10-14T17:00:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 -- 172.21.15.29:0/3160761991 learned_addr learned my addr 172.21.15.29:0/3160761991 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6d778640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c6806c3f0 0x7f6c681f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6cf77640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c6815e0c0 0x7f6c680c9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 -- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c6806c3f0 msgr2=0x7f6c681f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c6806c3f0 0x7f6c681f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 -- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c6815e0c0 msgr2=0x7f6c680c9f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.888+0000 7f6c6df79640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c6815e0c0 0x7f6c680c9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c6df79640 1 -- 172.21.15.29:0/3160761991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c640094d0 con 0x7f6c6815e540 2023-10-14T17:00:22.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c6df79640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000c7a0 tx=0x7f6c6000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60003b70 con 0x7f6c6815e540 2023-10-14T17:00:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c60003cf0 con 0x7f6c6815e540 2023-10-14T17:00:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c60003ea0 con 0x7f6c6815e540 2023-10-14T17:00:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c680cab00 con 0x7f6c6815e540 2023-10-14T17:00:22.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.889+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c680cb0c0 con 0x7f6c6815e540 2023-10-14T17:00:22.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.891+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c6804e160 con 0x7f6c6815e540 2023-10-14T17:00:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.892+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6c6000aa90 con 0x7f6c6815e540 2023-10-14T17:00:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.893+0000 7f6c5dffb640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 0x7f6c4c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.893+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f6c60089230 con 0x7f6c6815e540 2023-10-14T17:00:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.893+0000 7f6c6d778640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 0x7f6c4c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:22.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.893+0000 7f6c6d778640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 0x7f6c4c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c64036000 tx=0x7f6c6403b040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:22.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:22.897+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c60054560 con 0x7f6c6815e540 2023-10-14T17:00:23.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.162+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6c6815dbc0 con 0x7f6c6815e540 2023-10-14T17:00:23.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.163+0000 7f6c5dffb640 1 -- 172.21.15.29:0/3160761991 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6c60057b80 con 0x7f6c6815e540 2023-10-14T17:00:23.164 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:23.164 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:00:23.165 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T17:00:23.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.165 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T17:00:23.165 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T17:00:23.165 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.166 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.166 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:00:23.166 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:00:23.166 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.167 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T17:00:23.167 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T17:00:23.167 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:00:23.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:00:23.168 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.168 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.169 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.169 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T17:00:23.169 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T17:00:23.169 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:00:23.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:00:23.170 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.170 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.171 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.171 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T17:00:23.171 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T17:00:23.171 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T17:00:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T17:00:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.174 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.174 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.174 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.175 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T17:00:23.175 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T17:00:23.175 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.175 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.175 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.176 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.176 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.176 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T17:00:23.177 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T17:00:23.177 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.177 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.177 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.177 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.178 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.178 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T17:00:23.178 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T17:00:23.179 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.179 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.179 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.179 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.180 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.181 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.181 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.181 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.181 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.181 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.182 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T17:00:23.182 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T17:00:23.182 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.182 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.182 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.183 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.183 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.183 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.183 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.183 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T17:00:23.184 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T17:00:23.184 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.184 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.184 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.184 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.185 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.185 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.185 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.185 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T17:00:23.185 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T17:00:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:00:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:00:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.186 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.187 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.187 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.187 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T17:00:23.187 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T17:00:23.187 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.188 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T17:00:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T17:00:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.189 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.190 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T17:00:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T17:00:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:00:23.191 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:00:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:00:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.192 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:00:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:00:23.194 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.194 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.194 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.194 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T17:00:23.194 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T17:00:23.195 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.195 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:00:23.195 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:00:23.195 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:00:23.195 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T17:00:23.196 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.197 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:00:23.198 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:23.198 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T17:00:23.198 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T17:00:23.198 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:00:23.198 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:00:23.199 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:00:23.199 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:00:23.199 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.199 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:23.199 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T17:00:23.200 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:00:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 msgr2=0x7f6c4c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 0x7f6c4c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6c64036000 tx=0x7f6c6403b040 comp rx=0 tx=0).stop 2023-10-14T17:00:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 msgr2=0x7f6c680ca4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 crypto rx=0x7f6c6000c7a0 tx=0x7f6c6000cc60 comp rx=0 tx=0).stop 2023-10-14T17:00:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 shutdown_connections 2023-10-14T17:00:23.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f6c4c06af90 0x7f6c4c06d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c6815e540 0x7f6c680ca4d0 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c6815e0c0 0x7f6c680c9f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 --2- 172.21.15.29:0/3160761991 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c6806c3f0 0x7f6c681f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 >> 172.21.15.29:0/3160761991 conn(0x7f6c680d2cb0 msgr2=0x7f6c680d0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:23.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 shutdown_connections 2023-10-14T17:00:23.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.166+0000 7f6c6f9dc640 1 -- 172.21.15.29:0/3160761991 wait complete. 2023-10-14T17:00:23.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:23.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.441+0000 7fab81c9f640 1 Processor -- start 2023-10-14T17:00:23.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.441+0000 7fab81c9f640 1 -- start start 2023-10-14T17:00:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab81c9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c0ad970 0x7fab7c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab81c9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab7c0ccfb0 0x7fab7c0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab81c9f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c15b9d0 con 0x7fab7c06c040 2023-10-14T17:00:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab81c9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab7c15bcb0 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab81c9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c15bf90 con 0x7fab7c0ad970 2023-10-14T17:00:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab8149e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab7c0ccfb0 0x7fab7c0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab80c9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab7c06c040 0x7fab7c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57280/0 (socket says 172.21.15.29:57280) 2023-10-14T17:00:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.443+0000 7fab80c9d640 1 -- 172.21.15.29:0/1554261938 learned_addr learned my addr 172.21.15.29:0/1554261938 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab7bfff640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c0ad970 0x7fab7c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab8149e640 1 -- 172.21.15.29:0/1554261938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c0ad970 msgr2=0x7fab7c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab8149e640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c0ad970 0x7fab7c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab8149e640 1 -- 172.21.15.29:0/1554261938 >> v1:172.21.15.29:6789/0 conn(0x7fab7c06c040 legacy=0x7fab7c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab8149e640 1 -- 172.21.15.29:0/1554261938 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0cc270 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.444+0000 7fab7b7fe640 1 -- 172.21.15.29:0/1554261938 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157494416 0 0) 0x7fab7c15b9d0 con 0x7fab7c06c040 2023-10-14T17:00:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.445+0000 7fab8149e640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c0cd440 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fab74009dc0 tx=0x7fab7402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524c45f8d1dbfffe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.445+0000 7fab7b7fe640 1 -- 172.21.15.29:0/1554261938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7402cc40 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.445+0000 7fab7b7fe640 1 -- 172.21.15.29:0/1554261938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab7402cda0 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.445+0000 7fab7b7fe640 1 -- 172.21.15.29:0/1554261938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74034660 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 -- 172.21.15.29:0/1554261938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ccfb0 msgr2=0x7fab7c0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c0cd440 secure :-1 s=READY pgs=1559 cs=0 l=1 rev1=1 crypto rx=0x7fab74009dc0 tx=0x7fab7402be90 comp rx=0 tx=0).stop 2023-10-14T17:00:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 -- 172.21.15.29:0/1554261938 shutdown_connections 2023-10-14T17:00:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c0cd440 unknown :-1 s=CLOSED pgs=1559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1554261938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c0ad970 0x7fab7c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 -- 172.21.15.29:0/1554261938 >> 172.21.15.29:0/1554261938 conn(0x7fab7c0ce6a0 msgr2=0x7fab7c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 -- 172.21.15.29:0/1554261938 shutdown_connections 2023-10-14T17:00:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.446+0000 7fab81c9f640 1 -- 172.21.15.29:0/1554261938 wait complete. 2023-10-14T17:00:23.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.448+0000 7fab81c9f640 1 Processor -- start 2023-10-14T17:00:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.448+0000 7fab81c9f640 1 -- start start 2023-10-14T17:00:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 0x7fab7c1f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c06fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c1f60a0 con 0x7fab7c0ad970 2023-10-14T17:00:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c1f6a00 con 0x7fab7c0ccfb0 2023-10-14T17:00:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab81c9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c1f5a20 con 0x7fab7c06c040 2023-10-14T17:00:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab7bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab7bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54980/0 (socket says 172.21.15.29:54980) 2023-10-14T17:00:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.449+0000 7fab7bfff640 1 -- 172.21.15.29:0/1560234521 learned_addr learned my addr 172.21.15.29:0/1560234521 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab80c9d640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 0x7fab7c1f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 -- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 msgr2=0x7fab7c1f64c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 0x7fab7c1f64c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 -- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7c0ccfb0 msgr2=0x7fab7c06fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab8149e640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c06fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c06fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 -- 172.21.15.29:0/1560234521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74009a70 con 0x7fab7c0ad970 2023-10-14T17:00:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab80c9d640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 0x7fab7c1f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:23.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.450+0000 7fab7bfff640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fab6800d800 tx=0x7fab6800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.451+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800b6c0 con 0x7fab7c0ad970 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.451+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6800b820 con 0x7fab7c0ad970 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.451+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c1f2fd0 con 0x7fab7c0ad970 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.451+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6800bbb0 con 0x7fab7c0ad970 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.451+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c1f5cc0 con 0x7fab7c0ad970 2023-10-14T17:00:23.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.452+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c06b330 con 0x7fab7c0ad970 2023-10-14T17:00:23.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.453+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab6800bd10 con 0x7fab7c0ad970 2023-10-14T17:00:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.454+0000 7fab797fa640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 0x7fab6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.454+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fab68089a90 con 0x7fab7c0ad970 2023-10-14T17:00:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.455+0000 7fab80c9d640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 0x7fab6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.455+0000 7fab80c9d640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 0x7fab6006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.459+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab68054c90 con 0x7fab7c0ad970 2023-10-14T17:00:23.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.659+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fab7c1f5e50 con 0x7fab7c0ad970 2023-10-14T17:00:23.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.661+0000 7fab797fa640 1 -- 172.21.15.29:0/1560234521 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+12970 (secure 0 0 0) 0x7fab680582b0 con 0x7fab7c0ad970 2023-10-14T17:00:23.662 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:23.662 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":136,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:22.629607+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":123,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:23.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 msgr2=0x7fab6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 0x7fab6006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fab70000f60 tx=0x7fab70009040 comp rx=0 tx=0).stop 2023-10-14T17:00:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 msgr2=0x7fab7c06f700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 crypto rx=0x7fab6800d800 tx=0x7fab6800dcc0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 shutdown_connections 2023-10-14T17:00:23.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fab6006af90 0x7fab6006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7c0ccfb0 0x7fab7c06fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab7c0ad970 0x7fab7c06f700 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 --2- 172.21.15.29:0/1560234521 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7c06c040 0x7fab7c1f64c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 >> 172.21.15.29:0/1560234521 conn(0x7fab7c0ce6a0 msgr2=0x7fab7c15dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:23.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 shutdown_connections 2023-10-14T17:00:23.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.664+0000 7fab81c9f640 1 -- 172.21.15.29:0/1560234521 wait complete. 2023-10-14T17:00:23.685 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:23.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:23.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.913+0000 7f0fda62d640 1 Processor -- start 2023-10-14T17:00:23.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.914+0000 7f0fda62d640 1 -- start start 2023-10-14T17:00:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.915+0000 7f0fda62d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 0x7f0fd40ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.915+0000 7f0fda62d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0fd40adc00 0x7f0fd40cc620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.915+0000 7f0fda62d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd40ccf60 con 0x7f0fd40ccb60 2023-10-14T17:00:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.915+0000 7f0fda62d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd40711b0 con 0x7f0fd40adc00 2023-10-14T17:00:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.915+0000 7f0fda62d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd4070530 con 0x7f0fd415cef0 2023-10-14T17:00:23.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.916+0000 7f0fd8bca640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0fd40ccb60 0x7f0fd406f480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57282/0 (socket says 172.21.15.29:57282) 2023-10-14T17:00:23.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.916+0000 7f0fd8bca640 1 -- 172.21.15.29:0/2141783130 learned_addr learned my addr 172.21.15.29:0/2141783130 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:23.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.916+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/2141783130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 0x7f0fd40ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.916+0000 7f0fd37fe640 1 --2- 172.21.15.29:0/2141783130 >> v2:172.21.15.186:3300/0 conn(0x7f0fd40adc00 0x7f0fd40cc620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.916+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2763032156 0 0) 0x7f0fd40ccf60 con 0x7f0fd40ccb60 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc0003520 con 0x7f0fd40ccb60 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3072195957 0 0) 0x7f0fc0003520 con 0x7f0fd40ccb60 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd3fff640 1 -- 172.21.15.29:0/2141783130 >> v2:172.21.15.186:3300/0 conn(0x7f0fd40adc00 msgr2=0x7f0fd40cc620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/2141783130 >> v2:172.21.15.186:3300/0 conn(0x7f0fd40adc00 0x7f0fd40cc620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd3fff640 1 -- 172.21.15.29:0/2141783130 >> v1:172.21.15.29:6789/0 conn(0x7f0fd40ccb60 legacy=0x7f0fd406f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.917+0000 7f0fd3fff640 1 -- 172.21.15.29:0/2141783130 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd4070810 con 0x7f0fd415cef0 2023-10-14T17:00:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.918+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/2141783130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 0x7f0fd40ad2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402bce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60cdc5adc7a49831 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fcc001670 con 0x7f0fd415cef0 2023-10-14T17:00:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc402c810 con 0x7f0fd415cef0 2023-10-14T17:00:23.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fd2ffd640 1 -- 172.21.15.29:0/2141783130 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc402cba0 con 0x7f0fd415cef0 2023-10-14T17:00:23.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 -- 172.21.15.29:0/2141783130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 msgr2=0x7f0fd40ad2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 --2- 172.21.15.29:0/2141783130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 0x7f0fd40ad2a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4009d90 tx=0x7f0fc402bce0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 -- 172.21.15.29:0/2141783130 shutdown_connections 2023-10-14T17:00:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 --2- 172.21.15.29:0/2141783130 >> v2:172.21.15.186:3300/0 conn(0x7f0fd40adc00 0x7f0fd40cc620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 --2- 172.21.15.29:0/2141783130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd415cef0 0x7f0fd40ad2a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 -- 172.21.15.29:0/2141783130 >> 172.21.15.29:0/2141783130 conn(0x7f0fd40ce840 msgr2=0x7f0fd40cec20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.919+0000 7f0fda62d640 1 -- 172.21.15.29:0/2141783130 shutdown_connections 2023-10-14T17:00:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.920+0000 7f0fda62d640 1 -- 172.21.15.29:0/2141783130 wait complete. 2023-10-14T17:00:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.921+0000 7f0fda62d640 1 Processor -- start 2023-10-14T17:00:23.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.921+0000 7f0fda62d640 1 -- start start 2023-10-14T17:00:23.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40adc00 0x7f0fd4158f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415cef0 0x7f0fd41f3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd41555f0 con 0x7f0fd40ccb60 2023-10-14T17:00:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40c4310 con 0x7f0fd415cef0 2023-10-14T17:00:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.922+0000 7f0fda62d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd40c7620 con 0x7f0fd40adc00 2023-10-14T17:00:23.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54988/0 (socket says 172.21.15.29:54988) 2023-10-14T17:00:23.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 -- 172.21.15.29:0/1607865617 learned_addr learned my addr 172.21.15.29:0/1607865617 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:23.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40adc00 0x7f0fd4158f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd8bca640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415cef0 0x7f0fd41f3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 -- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40adc00 msgr2=0x7f0fd4158f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40adc00 0x7f0fd4158f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 -- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415cef0 msgr2=0x7f0fd41f3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415cef0 0x7f0fd41f3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.923+0000 7f0fd37fe640 1 -- 172.21.15.29:0/1607865617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4009a70 con 0x7f0fd40ccb60 2023-10-14T17:00:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fd37fe640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e8a0 tx=0x7f0fbc00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc0094f0 con 0x7f0fd40ccb60 2023-10-14T17:00:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fbc00c940 con 0x7f0fd40ccb60 2023-10-14T17:00:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fbc00ccf0 con 0x7f0fd40ccb60 2023-10-14T17:00:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd40c7900 con 0x7f0fd40ccb60 2023-10-14T17:00:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.924+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd41f22e0 con 0x7f0fd40ccb60 2023-10-14T17:00:23.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.925+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd404e160 con 0x7f0fd40ccb60 2023-10-14T17:00:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.926+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0fbc013470 con 0x7f0fd40ccb60 2023-10-14T17:00:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.927+0000 7f0fd0ff9640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 0x7f0fb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.927+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f0fbc0895a0 con 0x7f0fd40ccb60 2023-10-14T17:00:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.928+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 0x7f0fb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.928+0000 7f0fd3fff640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 0x7f0fb406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4036000 tx=0x7f0fc4030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:23.932+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fbc054830 con 0x7f0fd40ccb60 2023-10-14T17:00:24.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.125+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0fd40afa30 con 0x7f0fd40ccb60 2023-10-14T17:00:24.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.126+0000 7f0fd0ff9640 1 -- 172.21.15.29:0/1607865617 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v136) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0fbc057e50 con 0x7f0fd40ccb60 2023-10-14T17:00:24.128 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:24.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 msgr2=0x7f0fb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 0x7f0fb406d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0fc4036000 tx=0x7f0fc4030000 comp rx=0 tx=0).stop 2023-10-14T17:00:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 msgr2=0x7f0fd40c3dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 secure :-1 s=READY pgs=1562 cs=0 l=1 rev1=1 crypto rx=0x7f0fbc00e8a0 tx=0x7f0fbc00ed60 comp rx=0 tx=0).stop 2023-10-14T17:00:24.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 shutdown_connections 2023-10-14T17:00:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb406af90 0x7f0fb406d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fd415cef0 0x7f0fd41f3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd40ccb60 0x7f0fd40c3dd0 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 --2- 172.21.15.29:0/1607865617 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd40adc00 0x7f0fd4158f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 >> 172.21.15.29:0/1607865617 conn(0x7f0fd40ce840 msgr2=0x7f0fd40cb2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:24.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 shutdown_connections 2023-10-14T17:00:24.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.129+0000 7f0fda62d640 1 -- 172.21.15.29:0/1607865617 wait complete. 2023-10-14T17:00:24.148 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T17:00:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.375+0000 7f2f5f07c640 1 Processor -- start 2023-10-14T17:00:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.375+0000 7f2f5f07c640 1 -- start start 2023-10-14T17:00:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.376+0000 7f2f5f07c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f581593b0 0x7f2f5806f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5f07c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f580cc920 0x7f2f580ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5f07c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f58159790 con 0x7f2f580caa10 2023-10-14T17:00:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5f07c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f580c9b20 con 0x7f2f580cc920 2023-10-14T17:00:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5f07c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f580c9e00 con 0x7f2f581593b0 2023-10-14T17:00:24.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5d619640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f580cc920 0x7f2f580ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5d619640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f580cc920 0x7f2f580ccdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54992/0 (socket says 172.21.15.29:54992) 2023-10-14T17:00:24.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5d619640 1 -- 172.21.15.29:0/1110479531 learned_addr learned my addr 172.21.15.29:0/1110479531 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:24.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.377+0000 7f2f5ce18640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f581593b0 0x7f2f5806f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.378+0000 7f2f5d619640 1 -- 172.21.15.29:0/1110479531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f581593b0 msgr2=0x7f2f5806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.378+0000 7f2f5d619640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f581593b0 0x7f2f5806f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.378+0000 7f2f5d619640 1 -- 172.21.15.29:0/1110479531 >> v1:172.21.15.186:6789/0 conn(0x7f2f580caa10 legacy=0x7f2f5806fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.378+0000 7f2f5d619640 1 -- 172.21.15.29:0/1110479531 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f580ca0e0 con 0x7f2f580cc920 2023-10-14T17:00:24.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.379+0000 7f2f5d619640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580cc920 0x7f2f580ccdb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000d150 tx=0x7f2f5002dfb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5ca10b892fa9a34 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.379+0000 7f2f577fe640 1 -- 172.21.15.29:0/1110479531 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f5002ed60 con 0x7f2f580cc920 2023-10-14T17:00:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f577fe640 1 -- 172.21.15.29:0/1110479531 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f50032070 con 0x7f2f580cc920 2023-10-14T17:00:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f577fe640 1 -- 172.21.15.29:0/1110479531 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50038620 con 0x7f2f580cc920 2023-10-14T17:00:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1110479531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580cc920 msgr2=0x7f2f580ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580cc920 0x7f2f580ccdb0 secure :-1 s=READY pgs=1563 cs=0 l=1 rev1=1 crypto rx=0x7f2f5000d150 tx=0x7f2f5002dfb0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1110479531 shutdown_connections 2023-10-14T17:00:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f580cc920 0x7f2f580ccdb0 unknown :-1 s=CLOSED pgs=1563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1110479531 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f581593b0 0x7f2f5806f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.380+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1110479531 >> 172.21.15.29:0/1110479531 conn(0x7f2f580cb410 msgr2=0x7f2f580cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.381+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1110479531 shutdown_connections 2023-10-14T17:00:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.381+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1110479531 wait complete. 2023-10-14T17:00:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.382+0000 7f2f5f07c640 1 Processor -- start 2023-10-14T17:00:24.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.383+0000 7f2f5f07c640 1 -- start start 2023-10-14T17:00:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.383+0000 7f2f5f07c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f580caa10 0x7f2f5815ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.383+0000 7f2f5f07c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f580cc920 0x7f2f58152f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.383+0000 7f2f5f07c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5f07c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f5815b050 con 0x7f2f581593b0 2023-10-14T17:00:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5f07c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f58155630 con 0x7f2f580caa10 2023-10-14T17:00:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5f07c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f581534b0 con 0x7f2f580cc920 2023-10-14T17:00:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55002/0 (socket says 172.21.15.29:55002) 2023-10-14T17:00:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 -- 172.21.15.29:0/1026850578 learned_addr learned my addr 172.21.15.29:0/1026850578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 -- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f580cc920 msgr2=0x7f2f58152f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f580cc920 0x7f2f58152f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5ce18640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f580caa10 0x7f2f5815ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 -- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f580caa10 msgr2=0x7f2f5815ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f580caa10 0x7f2f5815ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.384+0000 7f2f5d619640 1 -- 172.21.15.29:0/1026850578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5000bd40 con 0x7f2f581593b0 2023-10-14T17:00:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f5d619640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f2f5003a040 tx=0x7f2f50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50046070 con 0x7f2f581593b0 2023-10-14T17:00:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f5000b3f0 con 0x7f2f581593b0 2023-10-14T17:00:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f580c71a0 con 0x7f2f581593b0 2023-10-14T17:00:24.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f50008d10 con 0x7f2f581593b0 2023-10-14T17:00:24.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.385+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f580c7760 con 0x7f2f581593b0 2023-10-14T17:00:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.386+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f581557e0 con 0x7f2f581593b0 2023-10-14T17:00:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.387+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f50042430 con 0x7f2f581593b0 2023-10-14T17:00:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.389+0000 7f2f557fa640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.389+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f2f500ab690 con 0x7f2f581593b0 2023-10-14T17:00:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.389+0000 7f2f5ce18640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.390+0000 7f2f5ce18640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 0x7f2f4406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c006fd0 tx=0x7f2f4c008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.393+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f50076890 con 0x7f2f581593b0 2023-10-14T17:00:24.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.591+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2f5804e160 con 0x7f2f581593b0 2023-10-14T17:00:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.592+0000 7f2f557fa640 1 -- 172.21.15.29:0/1026850578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+65 (secure 0 0 0) 0x7f2f50079eb0 con 0x7f2f581593b0 2023-10-14T17:00:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:24.593 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:00:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:24.593 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T17:00:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T17:00:24.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:24.594 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:00:24.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.594+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 msgr2=0x7f2f4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.594+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 0x7f2f4406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c006fd0 tx=0x7f2f4c008040 comp rx=0 tx=0).stop 2023-10-14T17:00:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.594+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 msgr2=0x7f2f580afb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.594+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 crypto rx=0x7f2f5003a040 tx=0x7f2f50004ab0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 shutdown_connections 2023-10-14T17:00:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2f4406af90 0x7f2f4406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f581593b0 0x7f2f580afb80 unknown :-1 s=CLOSED pgs=1564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f580cc920 0x7f2f58152f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 --2- 172.21.15.29:0/1026850578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f580caa10 0x7f2f5815ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 >> 172.21.15.29:0/1026850578 conn(0x7f2f580cb410 msgr2=0x7f2f5815d830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:24.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 shutdown_connections 2023-10-14T17:00:24.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.595+0000 7f2f5f07c640 1 -- 172.21.15.29:0/1026850578 wait complete. 2023-10-14T17:00:24.615 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T17:00:24.616 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T17:00:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.849+0000 7fefd88c6640 1 Processor -- start 2023-10-14T17:00:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.849+0000 7fefd88c6640 1 -- start start 2023-10-14T17:00:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fefd415ac10 0x7fefd406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefd40b0360 0x7fefd40b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40ca510 0x7fefd40ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd40b0d30 con 0x7fefd40b0360 2023-10-14T17:00:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fefd415b310 con 0x7fefd415ac10 2023-10-14T17:00:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd88c6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd406f640 con 0x7fefd40ca510 2023-10-14T17:00:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd1d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefd40b0360 0x7fefd40b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd1d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fefd40b0360 0x7fefd40b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55012/0 (socket says 172.21.15.29:55012) 2023-10-14T17:00:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.851+0000 7fefd1d9b640 1 -- 172.21.15.29:0/2084611206 learned_addr learned my addr 172.21.15.29:0/2084611206 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd2d9d640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40ca510 0x7fefd40ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd259c640 1 --2- 172.21.15.29:0/2084611206 >> v2:172.21.15.186:3300/0 conn(0x7fefd415ac10 0x7fefd406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd1d9b640 1 -- 172.21.15.29:0/2084611206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40ca510 msgr2=0x7fefd40ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40ca510 0x7fefd40ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd1d9b640 1 -- 172.21.15.29:0/2084611206 >> v2:172.21.15.186:3300/0 conn(0x7fefd415ac10 msgr2=0x7fefd406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/2084611206 >> v2:172.21.15.186:3300/0 conn(0x7fefd415ac10 0x7fefd406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.852+0000 7fefd1d9b640 1 -- 172.21.15.29:0/2084611206 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd415b5b0 con 0x7fefd40b0360 2023-10-14T17:00:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.854+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40b0360 0x7fefd40b07f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fefd406d540 tx=0x7fefbc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cba11dc0cecf7c81 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.854+0000 7fefd159a640 1 -- 172.21.15.29:0/2084611206 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc0317e0 con 0x7fefd40b0360 2023-10-14T17:00:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.854+0000 7fefd159a640 1 -- 172.21.15.29:0/2084611206 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc031940 con 0x7fefd40b0360 2023-10-14T17:00:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.854+0000 7fefd159a640 1 -- 172.21.15.29:0/2084611206 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc031cd0 con 0x7fefd40b0360 2023-10-14T17:00:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 -- 172.21.15.29:0/2084611206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40b0360 msgr2=0x7fefd40b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40b0360 0x7fefd40b07f0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 crypto rx=0x7fefd406d540 tx=0x7fefbc02b880 comp rx=0 tx=0).stop 2023-10-14T17:00:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 -- 172.21.15.29:0/2084611206 shutdown_connections 2023-10-14T17:00:24.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40ca510 0x7fefd40ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 --2- 172.21.15.29:0/2084611206 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40b0360 0x7fefd40b07f0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 --2- 172.21.15.29:0/2084611206 >> v2:172.21.15.186:3300/0 conn(0x7fefd415ac10 0x7fefd406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 -- 172.21.15.29:0/2084611206 >> 172.21.15.29:0/2084611206 conn(0x7fefd40cb360 msgr2=0x7fefd40cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 -- 172.21.15.29:0/2084611206 shutdown_connections 2023-10-14T17:00:24.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.855+0000 7fefd88c6640 1 -- 172.21.15.29:0/2084611206 wait complete. 2023-10-14T17:00:24.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.857+0000 7fefd88c6640 1 Processor -- start 2023-10-14T17:00:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.858+0000 7fefd88c6640 1 -- start start 2023-10-14T17:00:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40b0360 0x7fefd40c4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefd415ac10 0x7fefd40af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40c3320 con 0x7fefd40ca510 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40c38a0 con 0x7fefd415ac10 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd88c6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fefd40c7bc0 con 0x7fefd40b0360 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55020/0 (socket says 172.21.15.29:55020) 2023-10-14T17:00:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd1d9b640 1 -- 172.21.15.29:0/3493799031 learned_addr learned my addr 172.21.15.29:0/3493799031 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.859+0000 7fefd259c640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40b0360 0x7fefd40c4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd2d9d640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefd415ac10 0x7fefd40af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 -- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40b0360 msgr2=0x7fefd40c4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40b0360 0x7fefd40c4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 -- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefd415ac10 msgr2=0x7fefd40af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefd415ac10 0x7fefd40af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 -- 172.21.15.29:0/3493799031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefbc0094d0 con 0x7fefd40ca510 2023-10-14T17:00:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefd1d9b640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fefbc02b8c0 tx=0x7fefbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc032880 con 0x7fefd40ca510 2023-10-14T17:00:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefbc032a20 con 0x7fefd40ca510 2023-10-14T17:00:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.860+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefbc032dd0 con 0x7fefd40ca510 2023-10-14T17:00:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.861+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd40c7e60 con 0x7fefd40ca510 2023-10-14T17:00:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.861+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd40c3530 con 0x7fefd40ca510 2023-10-14T17:00:24.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.862+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd40c5390 con 0x7fefd40ca510 2023-10-14T17:00:24.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.863+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefbc0453f0 con 0x7fefd40ca510 2023-10-14T17:00:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.864+0000 7fefcaffd640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 0x7fefb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.864+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fefbc0ab710 con 0x7fefd40ca510 2023-10-14T17:00:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.865+0000 7fefd259c640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 0x7fefb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:24.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.865+0000 7fefd259c640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 0x7fefb406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:24.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:24.869+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefbc076950 con 0x7fefd40ca510 2023-10-14T17:00:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.067+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fefd404e160 con 0x7fefd40ca510 2023-10-14T17:00:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.652+0000 7fefcaffd640 1 -- 172.21.15.29:0/3493799031 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v137) v1 ==== 138+0+0 (secure 0 0 0) 0x7fefbc079f70 con 0x7fefd40ca510 2023-10-14T17:00:25.653 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T17:00:25.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.655+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 msgr2=0x7fefb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.655+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 0x7fefb406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fefc4006fd0 tx=0x7fefc4008040 comp rx=0 tx=0).stop 2023-10-14T17:00:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.655+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 msgr2=0x7fefd40c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.655+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 crypto rx=0x7fefbc02b8c0 tx=0x7fefbc0046c0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 shutdown_connections 2023-10-14T17:00:25.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fefb406af90 0x7fefb406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fefd415ac10 0x7fefd40af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fefd40ca510 0x7fefd40c7680 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 --2- 172.21.15.29:0/3493799031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fefd40b0360 0x7fefd40c4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 >> 172.21.15.29:0/3493799031 conn(0x7fefd40cb360 msgr2=0x7fefd40caa60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:25.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 shutdown_connections 2023-10-14T17:00:25.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.656+0000 7fefd88c6640 1 -- 172.21.15.29:0/3493799031 wait complete. 2023-10-14T17:00:25.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T17:00:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.892+0000 7f2d8c190640 1 Processor -- start 2023-10-14T17:00:25.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.893+0000 7f2d8c190640 1 -- start start 2023-10-14T17:00:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.894+0000 7f2d8c190640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 0x7f2d840b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.894+0000 7f2d8c190640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.894+0000 7f2d8c190640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8415e3c0 con 0x7f2d8415d3f0 2023-10-14T17:00:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.894+0000 7f2d8c190640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8415e0e0 con 0x7f2d840ccfc0 2023-10-14T17:00:25.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.894+0000 7f2d8c190640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84160af0 con 0x7f2d841615d0 2023-10-14T17:00:25.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.895+0000 7f2d89f2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 0x7f2d840b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.895+0000 7f2d8972b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.895+0000 7f2d89f2c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 0x7f2d840b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:38400/0 (socket says 172.21.15.29:38400) 2023-10-14T17:00:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.895+0000 7f2d89f2c640 1 -- 172.21.15.29:0/764015532 learned_addr learned my addr 172.21.15.29:0/764015532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.896+0000 7f2d8972b640 1 -- 172.21.15.29:0/764015532 >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 msgr2=0x7f2d840b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.896+0000 7f2d8972b640 1 --2- 172.21.15.29:0/764015532 >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 0x7f2d840b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.896+0000 7f2d8972b640 1 -- 172.21.15.29:0/764015532 >> v1:172.21.15.186:6789/0 conn(0x7f2d8415d3f0 legacy=0x7f2d8415d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.896+0000 7f2d8972b640 1 -- 172.21.15.29:0/764015532 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d84160dd0 con 0x7f2d841615d0 2023-10-14T17:00:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.897+0000 7f2d8972b640 1 --2- 172.21.15.29:0/764015532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840b0950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2d740097a0 tx=0x7f2d7402b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f354b8e8b12c879 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.898+0000 7f2d88f2a640 1 -- 172.21.15.29:0/764015532 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7402e070 con 0x7f2d841615d0 2023-10-14T17:00:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.898+0000 7f2d88f2a640 1 -- 172.21.15.29:0/764015532 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d740315a0 con 0x7f2d841615d0 2023-10-14T17:00:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.898+0000 7f2d88f2a640 1 -- 172.21.15.29:0/764015532 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74031930 con 0x7f2d841615d0 2023-10-14T17:00:25.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 -- 172.21.15.29:0/764015532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 msgr2=0x7f2d840b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 --2- 172.21.15.29:0/764015532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840b0950 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2d740097a0 tx=0x7f2d7402b840 comp rx=0 tx=0).stop 2023-10-14T17:00:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 -- 172.21.15.29:0/764015532 shutdown_connections 2023-10-14T17:00:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 --2- 172.21.15.29:0/764015532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840b0950 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 --2- 172.21.15.29:0/764015532 >> v2:172.21.15.186:3300/0 conn(0x7f2d840ccfc0 0x7f2d840b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 -- 172.21.15.29:0/764015532 >> 172.21.15.29:0/764015532 conn(0x7f2d840cb410 msgr2=0x7f2d840cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 -- 172.21.15.29:0/764015532 shutdown_connections 2023-10-14T17:00:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.899+0000 7f2d8c190640 1 -- 172.21.15.29:0/764015532 wait complete. 2023-10-14T17:00:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.901+0000 7f2d8c190640 1 Processor -- start 2023-10-14T17:00:25.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.902+0000 7f2d8c190640 1 -- start start 2023-10-14T17:00:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8415d3f0 0x7f2d8406bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840c22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840c3040 con 0x7f2d840ccfc0 2023-10-14T17:00:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840c3ef0 con 0x7f2d8415d3f0 2023-10-14T17:00:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d8c190640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d840c41d0 con 0x7f2d841615d0 2023-10-14T17:00:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d89f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d89f2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55036/0 (socket says 172.21.15.29:55036) 2023-10-14T17:00:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.903+0000 7f2d89f2c640 1 -- 172.21.15.29:0/1471536954 learned_addr learned my addr 172.21.15.29:0/1471536954 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d8a72d640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840c22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d8972b640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8415d3f0 0x7f2d8406bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 -- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 msgr2=0x7f2d840c22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840c22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 -- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8415d3f0 msgr2=0x7f2d8406bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8415d3f0 0x7f2d8406bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 -- 172.21.15.29:0/1471536954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d74009400 con 0x7f2d840ccfc0 2023-10-14T17:00:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d8a72d640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.904+0000 7f2d89f2c640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009fd0 tx=0x7f2d8000ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.905+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000c940 con 0x7f2d840ccfc0 2023-10-14T17:00:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.905+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d8000caa0 con 0x7f2d840ccfc0 2023-10-14T17:00:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.905+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d8000ce30 con 0x7f2d840ccfc0 2023-10-14T17:00:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.905+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d840c44b0 con 0x7f2d840ccfc0 2023-10-14T17:00:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.905+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8415a210 con 0x7f2d840ccfc0 2023-10-14T17:00:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.906+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8404e160 con 0x7f2d840ccfc0 2023-10-14T17:00:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.907+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d80013470 con 0x7f2d840ccfc0 2023-10-14T17:00:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.908+0000 7f2d7a7fc640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 0x7f2d6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.908+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5299+0+0 (secure 0 0 0) 0x7f2d80089700 con 0x7f2d840ccfc0 2023-10-14T17:00:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.909+0000 7f2d8972b640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 0x7f2d6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.909+0000 7f2d8972b640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 0x7f2d6806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d74036000 tx=0x7f2d7403d040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:25.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:25.916+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d800548c0 con 0x7f2d840ccfc0 2023-10-14T17:00:26.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.120+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f2d840b4910 con 0x7f2d840ccfc0 2023-10-14T17:00:26.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.671+0000 7f2d7a7fc640 1 -- 172.21.15.29:0/1471536954 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v138) v1 ==== 170+0+0 (secure 0 0 0) 0x7f2d80057ee0 con 0x7f2d840ccfc0 2023-10-14T17:00:26.672 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T17:00:26.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.676+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 msgr2=0x7f2d6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.676+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 0x7f2d6806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2d74036000 tx=0x7f2d7403d040 comp rx=0 tx=0).stop 2023-10-14T17:00:26.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.676+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 msgr2=0x7f2d84159cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.676+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 crypto rx=0x7f2d80009fd0 tx=0x7f2d8000ed40 comp rx=0 tx=0).stop 2023-10-14T17:00:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 shutdown_connections 2023-10-14T17:00:26.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2d6806af90 0x7f2d6806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d841615d0 0x7f2d840c22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d8415d3f0 0x7f2d8406bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 --2- 172.21.15.29:0/1471536954 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d840ccfc0 0x7f2d84159cd0 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.677+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 >> 172.21.15.29:0/1471536954 conn(0x7f2d840cb410 msgr2=0x7f2d840cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:26.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.678+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 shutdown_connections 2023-10-14T17:00:26.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.678+0000 7f2d8c190640 1 -- 172.21.15.29:0/1471536954 wait complete. 2023-10-14T17:00:26.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T17:00:26.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.905+0000 7f052e7cc640 1 Processor -- start 2023-10-14T17:00:26.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.906+0000 7f052e7cc640 1 -- start start 2023-10-14T17:00:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.907+0000 7f052e7cc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05280ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.907+0000 7f052e7cc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05280cb030 con 0x7f052815ad70 2023-10-14T17:00:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.907+0000 7f052e7cc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05280c91c0 con 0x7f05280c9ca0 2023-10-14T17:00:26.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.907+0000 7f052e7cc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05280c94a0 con 0x7f052806a4b0 2023-10-14T17:00:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.908+0000 7f0527fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f052815ad70 0x7f0528069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57294/0 (socket says 172.21.15.29:57294) 2023-10-14T17:00:26.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.908+0000 7f0527fff640 1 -- 172.21.15.29:0/3077069824 learned_addr learned my addr 172.21.15.29:0/3077069824 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.908+0000 7f051ffff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05280ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.908+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492996848 0 0) 0x7f05280cb030 con 0x7f052815ad70 2023-10-14T17:00:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.908+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0514003520 con 0x7f052815ad70 2023-10-14T17:00:26.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 135806554 0 0) 0x7f0514003520 con 0x7f052815ad70 2023-10-14T17:00:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 msgr2=0x7f05280ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05280ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 >> v1:172.21.15.186:6789/0 conn(0x7f05280c9ca0 legacy=0x7f05280ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05280c9780 con 0x7f052815ad70 2023-10-14T17:00:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0518001670 con 0x7f052815ad70 2023-10-14T17:00:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 >> v1:172.21.15.29:6789/0 conn(0x7f052815ad70 legacy=0x7f0528069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05140039a0 0x7f0514003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05140047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0514004d00 0x7f0514005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05280c94a0 con 0x7f05140039a0 2023-10-14T17:00:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f051400ba30 con 0x7f0514004d00 2023-10-14T17:00:26.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f051400bbf0 con 0x7f052806a4b0 2023-10-14T17:00:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f0527fff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05140039a0 0x7f0514003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f051ffff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05140047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.909+0000 7f052cd69640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0514004d00 0x7f0514005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.910+0000 7f0527fff640 1 -- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 msgr2=0x7f05140047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.910+0000 7f0527fff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f052806a4b0 0x7f05140047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.910+0000 7f0527fff640 1 -- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0514004d00 msgr2=0x7f0514005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.910+0000 7f0527fff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0514004d00 0x7f0514005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.910+0000 7f0527fff640 1 -- 172.21.15.29:0/3077069824 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0514003520 con 0x7f05140039a0 2023-10-14T17:00:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f0527fff640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05140039a0 0x7f0514003dc0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f0518007fd0 tx=0x7f05140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2b39ee0124e09a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0518005ad0 con 0x7f05140039a0 2023-10-14T17:00:26.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0518005c30 con 0x7f05140039a0 2023-10-14T17:00:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f05277fe640 1 -- 172.21.15.29:0/3077069824 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051802cab0 con 0x7f05140039a0 2023-10-14T17:00:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f052e7cc640 1 -- 172.21.15.29:0/3077069824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05140039a0 msgr2=0x7f0514003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.911+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3077069824 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05140039a0 0x7f0514003dc0 secure :-1 s=READY pgs=1569 cs=0 l=1 rev1=1 crypto rx=0x7f0518007fd0 tx=0x7f05140057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.912+0000 7f052cd69640 1 -- 172.21.15.29:0/3077069824 reap_dead start 2023-10-14T17:00:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.912+0000 7f052e7cc640 1 -- 172.21.15.29:0/3077069824 shutdown_connections 2023-10-14T17:00:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.912+0000 7f052e7cc640 1 -- 172.21.15.29:0/3077069824 >> 172.21.15.29:0/3077069824 conn(0x7f05280ce9b0 msgr2=0x7f05280cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.912+0000 7f052e7cc640 1 -- 172.21.15.29:0/3077069824 shutdown_connections 2023-10-14T17:00:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.912+0000 7f052e7cc640 1 -- 172.21.15.29:0/3077069824 wait complete. 2023-10-14T17:00:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.914+0000 7f052e7cc640 1 Processor -- start 2023-10-14T17:00:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.914+0000 7f052e7cc640 1 -- start start 2023-10-14T17:00:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05140039a0 0x7f052814a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 0x7f05280c8030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05280b02c0 con 0x7f05280b0b60 2023-10-14T17:00:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05280ce620 con 0x7f05280b06e0 2023-10-14T17:00:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.915+0000 7f052e7cc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052814a560 con 0x7f05140039a0 2023-10-14T17:00:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55062/0 (socket says 172.21.15.29:55062) 2023-10-14T17:00:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 -- 172.21.15.29:0/3249826145 learned_addr learned my addr 172.21.15.29:0/3249826145 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f0527fff640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05140039a0 0x7f052814a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f051ffff640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 0x7f05280c8030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 -- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05140039a0 msgr2=0x7f052814a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05140039a0 0x7f052814a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 -- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 msgr2=0x7f05280c8030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:26.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 0x7f05280c8030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 -- 172.21.15.29:0/3249826145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05280b3a80 con 0x7f05280b0b60 2023-10-14T17:00:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f051ffff640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 0x7f05280c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.916+0000 7f052cd69640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f05280c5a30 tx=0x7f052000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.917+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0520003b70 con 0x7f05280b0b60 2023-10-14T17:00:26.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.917+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0520003cd0 con 0x7f05280b0b60 2023-10-14T17:00:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.917+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05200052b0 con 0x7f05280b0b60 2023-10-14T17:00:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.917+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05280b3d60 con 0x7f05280b0b60 2023-10-14T17:00:26.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.917+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05280afb00 con 0x7f05280b0b60 2023-10-14T17:00:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.918+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05280b0400 con 0x7f05280b0b60 2023-10-14T17:00:26.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.919+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0520005410 con 0x7f05280b0b60 2023-10-14T17:00:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.920+0000 7f05257fa640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 0x7f050806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.920+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5681+0+0 (secure 0 0 0) 0x7f0520089300 con 0x7f05280b0b60 2023-10-14T17:00:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.920+0000 7f0527fff640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 0x7f050806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:26.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.921+0000 7f0527fff640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 0x7f050806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0518034000 tx=0x7f0518009030 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:26.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:26.925+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05280b0400 con 0x7f05280b0b60 2023-10-14T17:00:27.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.163+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f05280b0400 con 0x7f05280b0b60 2023-10-14T17:00:27.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.695+0000 7f05257fa640 1 -- 172.21.15.29:0/3249826145 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 16 and data pool 17 v62) v1 ==== 174+0+0 (secure 0 0 0) 0x7f05280b0400 con 0x7f05280b0b60 2023-10-14T17:00:27.698 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 16 and data pool 17 2023-10-14T17:00:27.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.707+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 msgr2=0x7f050806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.707+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 0x7f050806d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0518034000 tx=0x7f0518009030 comp rx=0 tx=0).stop 2023-10-14T17:00:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.707+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 msgr2=0x7f05280c8570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.707+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 crypto rx=0x7f05280c5a30 tx=0x7f052000dc00 comp rx=0 tx=0).stop 2023-10-14T17:00:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 shutdown_connections 2023-10-14T17:00:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f050806af90 0x7f050806d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05280b0b60 0x7f05280c8570 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05280b06e0 0x7f05280c8030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 --2- 172.21.15.29:0/3249826145 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05140039a0 0x7f052814a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.708+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 >> 172.21.15.29:0/3249826145 conn(0x7f05280ce9b0 msgr2=0x7f05280c52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:27.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.709+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 shutdown_connections 2023-10-14T17:00:27.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.709+0000 7f052e7cc640 1 -- 172.21.15.29:0/3249826145 wait complete. 2023-10-14T17:00:27.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:27.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.935+0000 7f1034933640 1 Processor -- start 2023-10-14T17:00:27.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.935+0000 7f1034933640 1 -- start start 2023-10-14T17:00:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.936+0000 7f1034933640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f103015aaa0 0x7f103015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.936+0000 7f1034933640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103015ed60 con 0x7f103015ae80 2023-10-14T17:00:27.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.936+0000 7f1034933640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300707e0 con 0x7f10300ce390 2023-10-14T17:00:27.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.936+0000 7f1034933640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f103006b810 con 0x7f103015aaa0 2023-10-14T17:00:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.937+0000 7f1027fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f103015ae80 0x7f10300cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57298/0 (socket says 172.21.15.29:57298) 2023-10-14T17:00:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.937+0000 7f1027fff640 1 -- 172.21.15.29:0/3443855117 learned_addr learned my addr 172.21.15.29:0/3443855117 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:27.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.937+0000 7f102e59c640 1 --2- 172.21.15.29:0/3443855117 >> v2:172.21.15.186:3300/0 conn(0x7f103015aaa0 0x7f103015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.937+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551225220 0 0) 0x7f103015ed60 con 0x7f103015ae80 2023-10-14T17:00:27.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c003520 con 0x7f103015ae80 2023-10-14T17:00:27.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3331859716 0 0) 0x7f101c003520 con 0x7f103015ae80 2023-10-14T17:00:27.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 >> v2:172.21.15.186:3300/0 conn(0x7f103015aaa0 msgr2=0x7f103015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:27.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 --2- 172.21.15.29:0/3443855117 >> v2:172.21.15.186:3300/0 conn(0x7f103015aaa0 0x7f103015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 >> v1:172.21.15.186:6789/0 conn(0x7f10300ce390 legacy=0x7f1030070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103015ef90 con 0x7f103015ae80 2023-10-14T17:00:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1018001670 con 0x7f103015ae80 2023-10-14T17:00:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 >> v1:172.21.15.29:6789/0 conn(0x7f103015ae80 legacy=0x7f10300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 0x7f101c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c0047c0 0x7f101c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103006b810 con 0x7f101c003970 2023-10-14T17:00:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f101c00c020 con 0x7f101c0047c0 2023-10-14T17:00:27.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.938+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f101c00c1e0 con 0x7f101c005180 2023-10-14T17:00:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 0x7f101c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f1027fff640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c0047c0 0x7f101c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102ed9d640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 -- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f101c005180 msgr2=0x7f101c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:27.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 -- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c0047c0 msgr2=0x7f101c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c0047c0 0x7f101c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102e59c640 1 -- 172.21.15.29:0/3443855117 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c003520 con 0x7f101c003970 2023-10-14T17:00:27.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102ed9d640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f101c005180 0x7f101c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:27.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.939+0000 7f102ed9d640 1 -- 172.21.15.29:0/3443855117 reap_dead start 2023-10-14T17:00:27.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.941+0000 7f102e59c640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 0x7f101c003d90 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ac0 tx=0x7f101c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=494d8a7c5e490f47 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.941+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102002c6b0 con 0x7f101c003970 2023-10-14T17:00:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.941+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f102002c810 con 0x7f101c003970 2023-10-14T17:00:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.941+0000 7f102dd9b640 1 -- 172.21.15.29:0/3443855117 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102002cba0 con 0x7f101c003970 2023-10-14T17:00:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 -- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 msgr2=0x7f101c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 0x7f101c003d90 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 crypto rx=0x7f1020009ac0 tx=0x7f101c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 -- 172.21.15.29:0/3443855117 shutdown_connections 2023-10-14T17:00:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 --2- 172.21.15.29:0/3443855117 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f101c003970 0x7f101c003d90 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 -- 172.21.15.29:0/3443855117 >> 172.21.15.29:0/3443855117 conn(0x7f10300cb410 msgr2=0x7f10300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:27.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 -- 172.21.15.29:0/3443855117 shutdown_connections 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.942+0000 7f1034933640 1 -- 172.21.15.29:0/3443855117 wait complete. 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.944+0000 7f1034933640 1 Processor -- start 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.944+0000 7f1034933640 1 -- start start 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 0x7f10300c3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10301e4af0 0x7f10301f2990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301e4f70 con 0x7f1030158c40 2023-10-14T17:00:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10301f3e60 con 0x7f101c003970 2023-10-14T17:00:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1034933640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10300c4740 con 0x7f10301e4af0 2023-10-14T17:00:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1027fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1027fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55084/0 (socket says 172.21.15.29:55084) 2023-10-14T17:00:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.945+0000 7f1027fff640 1 -- 172.21.15.29:0/1541347017 learned_addr learned my addr 172.21.15.29:0/1541347017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f102ed9d640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10301e4af0 0x7f10301f2990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f102e59c640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 0x7f10300c3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 -- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10301e4af0 msgr2=0x7f10301f2990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10301e4af0 0x7f10301f2990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 -- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 msgr2=0x7f10300c3ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 0x7f10300c3ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 -- 172.21.15.29:0/1541347017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300c4a20 con 0x7f1030158c40 2023-10-14T17:00:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f102e59c640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 0x7f10300c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:27.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1027fff640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f101800e7a0 tx=0x7f101800ec60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101800c870 con 0x7f1030158c40 2023-10-14T17:00:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f101800c9d0 con 0x7f1030158c40 2023-10-14T17:00:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f101800cd80 con 0x7f1030158c40 2023-10-14T17:00:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.946+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10300c4d00 con 0x7f1030158c40 2023-10-14T17:00:27.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.947+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10300c4430 con 0x7f1030158c40 2023-10-14T17:00:27.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.948+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f103004e160 con 0x7f1030158c40 2023-10-14T17:00:27.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.949+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f101801d460 con 0x7f1030158c40 2023-10-14T17:00:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.950+0000 7f1026ffd640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 0x7f101006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.950+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f10180896b0 con 0x7f1030158c40 2023-10-14T17:00:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.950+0000 7f102e59c640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 0x7f101006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.951+0000 7f102e59c640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 0x7f101006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1020036000 tx=0x7f1020030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:27.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:27.954+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1018054690 con 0x7f1030158c40 2023-10-14T17:00:28.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.147+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10300cdc60 con 0x7f1030158c40 2023-10-14T17:00:28.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.150+0000 7f1026ffd640 1 -- 172.21.15.29:0/1541347017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16643 (secure 0 0 0) 0x7f1018057cb0 con 0x7f1030158c40 2023-10-14T17:00:28.151 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:28.151 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:27.164850+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:28.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 msgr2=0x7f101006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 0x7f101006d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1020036000 tx=0x7f1020030000 comp rx=0 tx=0).stop 2023-10-14T17:00:28.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 msgr2=0x7f10301e45b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 crypto rx=0x7f101800e7a0 tx=0x7f101800ec60 comp rx=0 tx=0).stop 2023-10-14T17:00:28.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 shutdown_connections 2023-10-14T17:00:28.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.152+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f101006af90 0x7f101006d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f10301e4af0 0x7f10301f2990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1030158c40 0x7f10301e45b0 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 --2- 172.21.15.29:0/1541347017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f101c003970 0x7f10300c3ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 >> 172.21.15.29:0/1541347017 conn(0x7f10300cb410 msgr2=0x7f103006f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:28.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 shutdown_connections 2023-10-14T17:00:28.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.153+0000 7f1034933640 1 -- 172.21.15.29:0/1541347017 wait complete. 2023-10-14T17:00:28.171 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:28.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.388+0000 7f129f31d640 1 Processor -- start 2023-10-14T17:00:28.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.388+0000 7f129f31d640 1 -- start start 2023-10-14T17:00:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129f31d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 0x7f1298164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129f31d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129806f7f0 con 0x7f1298159f00 2023-10-14T17:00:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129f31d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129806c5e0 con 0x7f1298159b20 2023-10-14T17:00:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129f31d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129806c8c0 con 0x7f12980b0b20 2023-10-14T17:00:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129c8b8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1298159f00 0x7f12980b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57300/0 (socket says 172.21.15.29:57300) 2023-10-14T17:00:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129c8b8640 1 -- 172.21.15.29:0/1793457041 learned_addr learned my addr 172.21.15.29:0/1793457041 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f129d8ba640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 0x7f1298164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.390+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53649909 0 0) 0x7f129806f7f0 con 0x7f1298159f00 2023-10-14T17:00:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1284003520 con 0x7f1298159f00 2023-10-14T17:00:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3684186176 0 0) 0x7f1284003520 con 0x7f1298159f00 2023-10-14T17:00:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 msgr2=0x7f1298164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 0x7f1298164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 >> v1:172.21.15.186:6789/0 conn(0x7f1298159b20 legacy=0x7f129806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129806cba0 con 0x7f1298159f00 2023-10-14T17:00:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f128c001670 con 0x7f1298159f00 2023-10-14T17:00:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 >> v1:172.21.15.29:6789/0 conn(0x7f1298159f00 legacy=0x7f12980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 0x7f12840039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f12840047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1284004d00 0x7f1284005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129806c8c0 con 0x7f1284003f20 2023-10-14T17:00:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f128400ba60 con 0x7f1284004d00 2023-10-14T17:00:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.391+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f128400bc20 con 0x7f12980b0b20 2023-10-14T17:00:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f12840047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 -- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 msgr2=0x7f12840039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980b0b20 0x7f12840039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 -- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1284004d00 msgr2=0x7f1284005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129d8ba640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1284004d00 0x7f1284005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1284004d00 0x7f1284005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.392+0000 7f129c8b8640 1 -- 172.21.15.29:0/1793457041 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1284003520 con 0x7f1284003f20 2023-10-14T17:00:28.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.393+0000 7f129c8b8640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f12840047c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f128c00c210 tx=0x7f12840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f8f31fac5fa425b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c005ad0 con 0x7f1284003f20 2023-10-14T17:00:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c005c30 con 0x7f1284003f20 2023-10-14T17:00:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f128bfff640 1 -- 172.21.15.29:0/1793457041 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f128c02cad0 con 0x7f1284003f20 2023-10-14T17:00:28.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 -- 172.21.15.29:0/1793457041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 msgr2=0x7f12840047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 --2- 172.21.15.29:0/1793457041 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f12840047c0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 crypto rx=0x7f128c00c210 tx=0x7f12840057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129d8ba640 1 -- 172.21.15.29:0/1793457041 reap_dead start 2023-10-14T17:00:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 -- 172.21.15.29:0/1793457041 shutdown_connections 2023-10-14T17:00:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 -- 172.21.15.29:0/1793457041 >> 172.21.15.29:0/1793457041 conn(0x7f12980cb410 msgr2=0x7f12980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:28.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 -- 172.21.15.29:0/1793457041 shutdown_connections 2023-10-14T17:00:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.394+0000 7f129f31d640 1 -- 172.21.15.29:0/1793457041 wait complete. 2023-10-14T17:00:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.396+0000 7f129f31d640 1 Processor -- start 2023-10-14T17:00:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.396+0000 7f129f31d640 1 -- start start 2023-10-14T17:00:28.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980c4eb0 0x7f129814ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12981557d0 0x7f12980ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1298156190 con 0x7f1284003f20 2023-10-14T17:00:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1298156470 con 0x7f12981557d0 2023-10-14T17:00:28.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.397+0000 7f129f31d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1298156750 con 0x7f12980c4eb0 2023-10-14T17:00:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55104/0 (socket says 172.21.15.29:55104) 2023-10-14T17:00:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 -- 172.21.15.29:0/3858051596 learned_addr learned my addr 172.21.15.29:0/3858051596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 -- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980c4eb0 msgr2=0x7f129814ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980c4eb0 0x7f129814ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 -- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12981557d0 msgr2=0x7f12980ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12981557d0 0x7f12980ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 -- 172.21.15.29:0/3858051596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298156a30 con 0x7f1284003f20 2023-10-14T17:00:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.398+0000 7f129d0b9640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f129400c500 tx=0x7f129400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.399+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12940052d0 con 0x7f1284003f20 2023-10-14T17:00:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.399+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294005430 con 0x7f1284003f20 2023-10-14T17:00:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.399+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1294012870 con 0x7f1284003f20 2023-10-14T17:00:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.399+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1298156d10 con 0x7f1284003f20 2023-10-14T17:00:28.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.399+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1298155c80 con 0x7f1284003f20 2023-10-14T17:00:28.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.400+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1298157770 con 0x7f1284003f20 2023-10-14T17:00:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.401+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f12940129d0 con 0x7f1284003f20 2023-10-14T17:00:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.402+0000 7f1289ffb640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 0x7f127c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.402+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f1294089350 con 0x7f1284003f20 2023-10-14T17:00:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.402+0000 7f129c8b8640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 0x7f127c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.403+0000 7f129c8b8640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 0x7f127c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f128c034000 tx=0x7f128c007000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.406+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1294054330 con 0x7f1284003f20 2023-10-14T17:00:28.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.603+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f129804e160 con 0x7f1284003f20 2023-10-14T17:00:28.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.606+0000 7f1289ffb640 1 -- 172.21.15.29:0/3858051596 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16643 (secure 0 0 0) 0x7f1294057950 con 0x7f1284003f20 2023-10-14T17:00:28.607 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:28.607 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:27.164850+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 msgr2=0x7f127c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 0x7f127c06d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f128c034000 tx=0x7f128c007000 comp rx=0 tx=0).stop 2023-10-14T17:00:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 msgr2=0x7f1298157230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 secure :-1 s=READY pgs=1576 cs=0 l=1 rev1=1 crypto rx=0x7f129400c500 tx=0x7f129400c9c0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 shutdown_connections 2023-10-14T17:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f127c06af90 0x7f127c06d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12981557d0 0x7f12980ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12980c4eb0 0x7f129814ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 --2- 172.21.15.29:0/3858051596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1284003f20 0x7f1298157230 unknown :-1 s=CLOSED pgs=1576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.609+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 >> 172.21.15.29:0/3858051596 conn(0x7f12980cb410 msgr2=0x7f12980ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.610+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 shutdown_connections 2023-10-14T17:00:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.610+0000 7f129f31d640 1 -- 172.21.15.29:0/3858051596 wait complete. 2023-10-14T17:00:28.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T17:00:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.857+0000 7f2cf049b640 1 Processor -- start 2023-10-14T17:00:28.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.857+0000 7f2cf049b640 1 -- start start 2023-10-14T17:00:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2cf049b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2ce815ad70 0x7f2ce8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2cf049b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2cf049b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ce80cb030 con 0x7f2ce80ca3b0 2023-10-14T17:00:28.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2cf049b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80c91c0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2cf049b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ce80c94a0 con 0x7f2ce815ad70 2023-10-14T17:00:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2ceda36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2ceda36640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55120/0 (socket says 172.21.15.29:55120) 2023-10-14T17:00:28.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.859+0000 7f2ceda36640 1 -- 172.21.15.29:0/2018146492 learned_addr learned my addr 172.21.15.29:0/2018146492 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:28.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.860+0000 7f2cee237640 1 --2- 172.21.15.29:0/2018146492 >> v2:172.21.15.186:3300/0 conn(0x7f2ce815ad70 0x7f2ce8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.860+0000 7f2ceda36640 1 -- 172.21.15.29:0/2018146492 >> v2:172.21.15.186:3300/0 conn(0x7f2ce815ad70 msgr2=0x7f2ce8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.860+0000 7f2ceda36640 1 --2- 172.21.15.29:0/2018146492 >> v2:172.21.15.186:3300/0 conn(0x7f2ce815ad70 0x7f2ce8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.860+0000 7f2ceda36640 1 -- 172.21.15.29:0/2018146492 >> v1:172.21.15.186:6789/0 conn(0x7f2ce80ca3b0 legacy=0x7f2ce80ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.860+0000 7f2ceda36640 1 -- 172.21.15.29:0/2018146492 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce80c9780 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.861+0000 7f2ceda36640 1 --2- 172.21.15.29:0/2018146492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80cf780 tx=0x7f2cd802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a93bae8e75af7eb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.862+0000 7f2ced235640 1 -- 172.21.15.29:0/2018146492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031870 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.862+0000 7f2ced235640 1 -- 172.21.15.29:0/2018146492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cd80319d0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.862+0000 7f2ced235640 1 -- 172.21.15.29:0/2018146492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8031d60 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.862+0000 7f2cf049b640 1 -- 172.21.15.29:0/2018146492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 msgr2=0x7f2ce806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.862+0000 7f2cf049b640 1 --2- 172.21.15.29:0/2018146492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 crypto rx=0x7f2ce80cf780 tx=0x7f2cd802b910 comp rx=0 tx=0).stop 2023-10-14T17:00:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 -- 172.21.15.29:0/2018146492 shutdown_connections 2023-10-14T17:00:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 --2- 172.21.15.29:0/2018146492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce806a2e0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 --2- 172.21.15.29:0/2018146492 >> v2:172.21.15.186:3300/0 conn(0x7f2ce815ad70 0x7f2ce8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 -- 172.21.15.29:0/2018146492 >> 172.21.15.29:0/2018146492 conn(0x7f2ce80ce9b0 msgr2=0x7f2ce80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:28.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 -- 172.21.15.29:0/2018146492 shutdown_connections 2023-10-14T17:00:28.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.863+0000 7f2cf049b640 1 -- 172.21.15.29:0/2018146492 wait complete. 2023-10-14T17:00:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.865+0000 7f2cf049b640 1 Processor -- start 2023-10-14T17:00:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.865+0000 7f2cf049b640 1 -- start start 2023-10-14T17:00:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.866+0000 7f2cf049b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.866+0000 7f2cf049b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ce80ca3b0 0x7f2ce80ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cf049b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ce815ad70 0x7f2ce80ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cf049b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80c7f80 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cf049b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80af270 con 0x7f2ce815ad70 2023-10-14T17:00:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cf049b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ce80c75e0 con 0x7f2ce80ca3b0 2023-10-14T17:00:28.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55130/0 (socket says 172.21.15.29:55130) 2023-10-14T17:00:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 -- 172.21.15.29:0/1601940210 learned_addr learned my addr 172.21.15.29:0/1601940210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 -- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ce80ca3b0 msgr2=0x7f2ce80ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ce80ca3b0 0x7f2ce80ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 -- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ce815ad70 msgr2=0x7f2ce80ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ce815ad70 0x7f2ce80ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.867+0000 7f2cee237640 1 -- 172.21.15.29:0/1601940210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cd80094d0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cee237640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4007e40 tx=0x7f2ce400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce4014620 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ce400ee90 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ce40149c0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ce80c78c0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.868+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ce80ae870 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.870+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ce804e160 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.871+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ce4014b20 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.871+0000 7f2cdeffd640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 0x7f2ccc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.872+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f2ce408a1a0 con 0x7f2ce80c9ca0 2023-10-14T17:00:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.872+0000 7f2ceda36640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 0x7f2ccc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:28.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.872+0000 7f2ceda36640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 0x7f2ccc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8036000 tx=0x7f2cd8030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:28.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:28.876+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ce40550e0 con 0x7f2ce80c9ca0 2023-10-14T17:00:29.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.102+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f2ce80c5810 con 0x7f2ce80c9ca0 2023-10-14T17:00:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.716+0000 7f2cdeffd640 1 -- 172.21.15.29:0/1601940210 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v65) v1 ==== 118+0+0 (secure 0 0 0) 0x7f2ce4058700 con 0x7f2ce80c9ca0 2023-10-14T17:00:29.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.719+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 msgr2=0x7f2ccc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.719+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 0x7f2ccc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f2cd8036000 tx=0x7f2cd8030000 comp rx=0 tx=0).stop 2023-10-14T17:00:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.719+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 msgr2=0x7f2ce80aec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.719+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 crypto rx=0x7f2ce4007e40 tx=0x7f2ce400e4b0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 shutdown_connections 2023-10-14T17:00:29.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f2ccc06af90 0x7f2ccc06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ce815ad70 0x7f2ce80ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ce80ca3b0 0x7f2ce80ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 --2- 172.21.15.29:0/1601940210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ce80c9ca0 0x7f2ce80aec90 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 >> 172.21.15.29:0/1601940210 conn(0x7f2ce80ce9b0 msgr2=0x7f2ce80ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:29.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 shutdown_connections 2023-10-14T17:00:29.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.720+0000 7f2cf049b640 1 -- 172.21.15.29:0/1601940210 wait complete. 2023-10-14T17:00:29.741 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T17:00:29.741 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T17:00:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.973+0000 7fc04e12e640 1 Processor -- start 2023-10-14T17:00:29.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.974+0000 7fc04e12e640 1 -- start start 2023-10-14T17:00:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.975+0000 7fc04e12e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc04815cd70 0x7fc0480d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.975+0000 7fc04e12e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0480b8480 0x7fc0480cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.975+0000 7fc04e12e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc04815d150 con 0x7fc0480d0be0 2023-10-14T17:00:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.975+0000 7fc04e12e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0480d1180 con 0x7fc04815cd70 2023-10-14T17:00:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.975+0000 7fc04e12e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480d1460 con 0x7fc0480b8480 2023-10-14T17:00:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc04d92d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc0480d0be0 0x7fc0480b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57312/0 (socket says 172.21.15.29:57312) 2023-10-14T17:00:29.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc04d92d640 1 -- 172.21.15.29:0/1573001840 learned_addr learned my addr 172.21.15.29:0/1573001840 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:29.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc04d12c640 1 --2- 172.21.15.29:0/1573001840 >> v2:172.21.15.186:3300/0 conn(0x7fc04815cd70 0x7fc0480d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc04c92b640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0480b8480 0x7fc0480cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406729867 0 0) 0x7fc04815d150 con 0x7fc0480d0be0 2023-10-14T17:00:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.976+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc034003520 con 0x7fc0480d0be0 2023-10-14T17:00:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269605472 0 0) 0x7fc034003520 con 0x7fc0480d0be0 2023-10-14T17:00:29.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0480b8480 msgr2=0x7fc0480cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0480b8480 0x7fc0480cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 >> v2:172.21.15.186:3300/0 conn(0x7fc04815cd70 msgr2=0x7fc0480d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1573001840 >> v2:172.21.15.186:3300/0 conn(0x7fc04815cd70 0x7fc0480d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480d1740 con 0x7fc0480d0be0 2023-10-14T17:00:29.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc0400019f0 con 0x7fc0480d0be0 2023-10-14T17:00:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 >> v1:172.21.15.29:6789/0 conn(0x7fc0480d0be0 legacy=0x7fc0480b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034003970 0x7fc034003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc034005180 0x7fc03400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480d1180 con 0x7fc034005180 2023-10-14T17:00:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480d1460 con 0x7fc0340047c0 2023-10-14T17:00:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc03400c020 con 0x7fc034003970 2023-10-14T17:00:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc04d12c640 1 --2- 172.21.15.29:0/1573001840 >> v2:172.21.15.186:3300/0 conn(0x7fc04815cd70 0x7fc0480d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc04c92b640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0480b8480 0x7fc0480cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:29.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.977+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc034005180 0x7fc03400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04d92d640 1 -- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034003970 msgr2=0x7fc034003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034003970 0x7fc034003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04c92b640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04d92d640 1 -- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0340047c0 msgr2=0x7fc034004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:29.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0340047c0 0x7fc034004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.978+0000 7fc04d92d640 1 -- 172.21.15.29:0/1573001840 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc034003520 con 0x7fc034005180 2023-10-14T17:00:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.979+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc034005180 0x7fc03400b980 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc04000a790 tx=0x7fc0340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1f5d7bcde79296e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.979+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040034630 con 0x7fc034005180 2023-10-14T17:00:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.979+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc04002f070 con 0x7fc034005180 2023-10-14T17:00:29.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.979+0000 7fc03ffff640 1 -- 172.21.15.29:0/1573001840 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040034a30 con 0x7fc034005180 2023-10-14T17:00:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 -- 172.21.15.29:0/1573001840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc034005180 msgr2=0x7fc03400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1573001840 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc034005180 0x7fc03400b980 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 crypto rx=0x7fc04000a790 tx=0x7fc0340057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04d92d640 1 -- 172.21.15.29:0/1573001840 reap_dead start 2023-10-14T17:00:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 -- 172.21.15.29:0/1573001840 shutdown_connections 2023-10-14T17:00:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 -- 172.21.15.29:0/1573001840 >> 172.21.15.29:0/1573001840 conn(0x7fc0480cf570 msgr2=0x7fc0480cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:29.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 -- 172.21.15.29:0/1573001840 shutdown_connections 2023-10-14T17:00:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.980+0000 7fc04e12e640 1 -- 172.21.15.29:0/1573001840 wait complete. 2023-10-14T17:00:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.982+0000 7fc04e12e640 1 Processor -- start 2023-10-14T17:00:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.982+0000 7fc04e12e640 1 -- start start 2023-10-14T17:00:29.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034005180 0x7fc048145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 0x7fc0480cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480cc0a0 con 0x7fc0481f2780 2023-10-14T17:00:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0481f3140 con 0x7fc0480c8600 2023-10-14T17:00:29.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04e12e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0480c8a80 con 0x7fc034005180 2023-10-14T17:00:29.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04d92d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04d92d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55154/0 (socket says 172.21.15.29:55154) 2023-10-14T17:00:29.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04d92d640 1 -- 172.21.15.29:0/1609647360 learned_addr learned my addr 172.21.15.29:0/1609647360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:29.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.983+0000 7fc04c92b640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 0x7fc0480cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d12c640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034005180 0x7fc048145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 -- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034005180 msgr2=0x7fc048145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034005180 0x7fc048145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 -- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 msgr2=0x7fc0480cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:29.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 0x7fc0480cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:29.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 -- 172.21.15.29:0/1609647360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0480cb150 con 0x7fc0481f2780 2023-10-14T17:00:29.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04c92b640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 0x7fc0480cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:29.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04d92d640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fc04002cba0 tx=0x7fc040001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:29.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040034640 con 0x7fc0481f2780 2023-10-14T17:00:29.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0400347a0 con 0x7fc0481f2780 2023-10-14T17:00:29.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc040034b10 con 0x7fc0481f2780 2023-10-14T17:00:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0480cb420 con 0x7fc0481f2780 2023-10-14T17:00:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.984+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0480cb9e0 con 0x7fc0481f2780 2023-10-14T17:00:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.986+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0480cc1e0 con 0x7fc0481f2780 2023-10-14T17:00:29.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.987+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc040031030 con 0x7fc0481f2780 2023-10-14T17:00:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.988+0000 7fc03dffb640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 0x7fc02c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:29.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.988+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc0400abd40 con 0x7fc0481f2780 2023-10-14T17:00:29.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.988+0000 7fc04d12c640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 0x7fc02c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:29.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.989+0000 7fc04d12c640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 0x7fc02c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc04400d8d0 tx=0x7fc044008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:29.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:29.994+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0480cc1e0 con 0x7fc0481f2780 2023-10-14T17:00:30.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.230+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc0480cc1e0 con 0x7fc0481f2780 2023-10-14T17:00:30.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.719+0000 7fc03dffb640 1 -- 172.21.15.29:0/1609647360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v66) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc0480cc1e0 con 0x7fc0481f2780 2023-10-14T17:00:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 msgr2=0x7fc02c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 0x7fc02c06d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc04400d8d0 tx=0x7fc044008040 comp rx=0 tx=0).stop 2023-10-14T17:00:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 msgr2=0x7fc0481f2c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:30.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 secure :-1 s=READY pgs=1581 cs=0 l=1 rev1=1 crypto rx=0x7fc04002cba0 tx=0x7fc040001cb0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 shutdown_connections 2023-10-14T17:00:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fc02c06af90 0x7fc02c06d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0481f2780 0x7fc0481f2c00 unknown :-1 s=CLOSED pgs=1581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0480c8600 0x7fc0480cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 --2- 172.21.15.29:0/1609647360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc034005180 0x7fc048145660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.722+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 >> 172.21.15.29:0/1609647360 conn(0x7fc0480cf570 msgr2=0x7fc04815f6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.723+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 shutdown_connections 2023-10-14T17:00:30.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.723+0000 7fc04e12e640 1 -- 172.21.15.29:0/1609647360 wait complete. 2023-10-14T17:00:30.745 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T17:00:30.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:30.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.977+0000 7fdb6274e640 1 Processor -- start 2023-10-14T17:00:30.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.977+0000 7fdb6274e640 1 -- start start 2023-10-14T17:00:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3070 0x7fdb5c0cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 0x7fdb5c0cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdb5c0b3450 0x7fdb5c0c6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0b3940 con 0x7fdb5c0b3450 2023-10-14T17:00:30.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb5c0b45e0 con 0x7fdb5c0b3e70 2023-10-14T17:00:30.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.978+0000 7fdb6274e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0b3b90 con 0x7fdb5c0b3070 2023-10-14T17:00:30.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdb5c0b3450 0x7fdb5c0c6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 0x7fdb5c0cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb537fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 0x7fdb5c0cfa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:49724/0 (socket says 172.21.15.29:49724) 2023-10-14T17:00:30.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb537fe640 1 -- 172.21.15.29:0/4261984438 learned_addr learned my addr 172.21.15.29:0/4261984438 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:30.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb5bfff640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3070 0x7fdb5c0cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 -- 172.21.15.29:0/4261984438 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3070 msgr2=0x7fdb5c0cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:30.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3070 0x7fdb5c0cf500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 -- 172.21.15.29:0/4261984438 >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 msgr2=0x7fdb5c0cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:30.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 --2- 172.21.15.29:0/4261984438 >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 0x7fdb5c0cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.979+0000 7fdb60ceb640 1 -- 172.21.15.29:0/4261984438 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c0d3840 con 0x7fdb5c0b3450 2023-10-14T17:00:30.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb60ceb640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0c6870 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c08bf20 tx=0x7fdb5402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=872e103c25a1bb8c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:30.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb5b7fe640 1 -- 172.21.15.29:0/4261984438 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5402cbd0 con 0x7fdb5c0b3450 2023-10-14T17:00:30.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb5b7fe640 1 -- 172.21.15.29:0/4261984438 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb5402cd30 con 0x7fdb5c0b3450 2023-10-14T17:00:30.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb5b7fe640 1 -- 172.21.15.29:0/4261984438 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5402cea0 con 0x7fdb5c0b3450 2023-10-14T17:00:30.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb6274e640 1 -- 172.21.15.29:0/4261984438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3450 msgr2=0x7fdb5c0c6870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.981+0000 7fdb6274e640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0c6870 secure :-1 s=READY pgs=1582 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c08bf20 tx=0x7fdb5402be20 comp rx=0 tx=0).stop 2023-10-14T17:00:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 -- 172.21.15.29:0/4261984438 shutdown_connections 2023-10-14T17:00:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0c6870 unknown :-1 s=CLOSED pgs=1582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 --2- 172.21.15.29:0/4261984438 >> v2:172.21.15.186:3300/0 conn(0x7fdb5c0b3e70 0x7fdb5c0cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 --2- 172.21.15.29:0/4261984438 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3070 0x7fdb5c0cf500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 -- 172.21.15.29:0/4261984438 >> 172.21.15.29:0/4261984438 conn(0x7fdb5c0d2d20 msgr2=0x7fdb5c0d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 -- 172.21.15.29:0/4261984438 shutdown_connections 2023-10-14T17:00:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.982+0000 7fdb6274e640 1 -- 172.21.15.29:0/4261984438 wait complete. 2023-10-14T17:00:30.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.983+0000 7fdb6274e640 1 Processor -- start 2023-10-14T17:00:30.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.984+0000 7fdb6274e640 1 -- start start 2023-10-14T17:00:30.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.984+0000 7fdb6274e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.984+0000 7fdb6274e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb6274e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 0x7fdb5c15ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb6274e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c0b1c50 con 0x7fdb5c0b3070 2023-10-14T17:00:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb6274e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c15f370 con 0x7fdb5c0b3450 2023-10-14T17:00:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb6274e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c15fdb0 con 0x7fdb5c0b3e70 2023-10-14T17:00:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58040/0 (socket says 172.21.15.29:58040) 2023-10-14T17:00:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb5bfff640 1 -- 172.21.15.29:0/3098360058 learned_addr learned my addr 172.21.15.29:0/3098360058 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb537fe640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.985+0000 7fdb60ceb640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 0x7fdb5c15ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 -- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 msgr2=0x7fdb5c15ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 0x7fdb5c15ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 -- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 msgr2=0x7fdb5c0b0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0b0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 -- 172.21.15.29:0/3098360058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb54009a70 con 0x7fdb5c0b3070 2023-10-14T17:00:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb60ceb640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 0x7fdb5c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb537fe640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb5bfff640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00e8e0 tx=0x7fdb4c00eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00c9a0 con 0x7fdb5c0b3070 2023-10-14T17:00:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb4c00cb00 con 0x7fdb5c0b3070 2023-10-14T17:00:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c0b1040 con 0x7fdb5c0b3070 2023-10-14T17:00:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb4c00ce90 con 0x7fdb5c0b3070 2023-10-14T17:00:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.986+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c0b1600 con 0x7fdb5c0b3070 2023-10-14T17:00:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.988+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c04e160 con 0x7fdb5c0b3070 2023-10-14T17:00:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.988+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdb4c005990 con 0x7fdb5c0b3070 2023-10-14T17:00:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.989+0000 7fdb597fa640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 0x7fdb3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.989+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdb4c089cb0 con 0x7fdb5c0b3070 2023-10-14T17:00:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.990+0000 7fdb537fe640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 0x7fdb3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.990+0000 7fdb537fe640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 0x7fdb3c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c1f55f0 tx=0x7fdb44008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:30.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:30.994+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdb4c054bf0 con 0x7fdb5c0b3070 2023-10-14T17:00:31.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.226+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb5c06b4a0 con 0x7fdb5c0b3070 2023-10-14T17:00:31.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.227+0000 7fdb597fa640 1 -- 172.21.15.29:0/3098360058 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fdb4c020020 con 0x7fdb5c0b3070 2023-10-14T17:00:31.228 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:31.228 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:31.229 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:31.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.229+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 msgr2=0x7fdb3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 0x7fdb3c06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fdb5c1f55f0 tx=0x7fdb44008040 comp rx=0 tx=0).stop 2023-10-14T17:00:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 msgr2=0x7fdb5c15f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 secure :-1 s=READY pgs=1583 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c00e8e0 tx=0x7fdb4c00eda0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 shutdown_connections 2023-10-14T17:00:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fdb3c06af90 0x7fdb3c06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdb5c0b3e70 0x7fdb5c15ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdb5c0b3450 0x7fdb5c0b0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 --2- 172.21.15.29:0/3098360058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdb5c0b3070 0x7fdb5c15f550 unknown :-1 s=CLOSED pgs=1583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 >> 172.21.15.29:0/3098360058 conn(0x7fdb5c0d2d20 msgr2=0x7fdb5c06bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 shutdown_connections 2023-10-14T17:00:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.230+0000 7fdb6274e640 1 -- 172.21.15.29:0/3098360058 wait complete. 2023-10-14T17:00:31.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.461+0000 7f22fe101640 1 Processor -- start 2023-10-14T17:00:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.461+0000 7f22fe101640 1 -- start start 2023-10-14T17:00:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22fe101640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f815adf0 0x7f22f815be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22fe101640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22f8070dc0 0x7f22f806b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22fe101640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22f806b750 con 0x7f22f806adb0 2023-10-14T17:00:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22fe101640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22f806dc00 con 0x7f22f8070dc0 2023-10-14T17:00:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22fe101640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f815c3a0 con 0x7f22f815adf0 2023-10-14T17:00:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22f7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22f8070dc0 0x7f22f806b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22f7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22f8070dc0 0x7f22f806b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58050/0 (socket says 172.21.15.29:58050) 2023-10-14T17:00:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.463+0000 7f22f7fff640 1 -- 172.21.15.29:0/3292923912 learned_addr learned my addr 172.21.15.29:0/3292923912 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.464+0000 7f22f77fe640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f815adf0 0x7f22f815be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.464+0000 7f22f7fff640 1 -- 172.21.15.29:0/3292923912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f815adf0 msgr2=0x7f22f815be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.464+0000 7f22f7fff640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f815adf0 0x7f22f815be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.464+0000 7f22f7fff640 1 -- 172.21.15.29:0/3292923912 >> v1:172.21.15.186:6789/0 conn(0x7f22f806adb0 legacy=0x7f22f815e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.464+0000 7f22f7fff640 1 -- 172.21.15.29:0/3292923912 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f806cf80 con 0x7f22f8070dc0 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.465+0000 7f22f7fff640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f8070dc0 0x7f22f806b210 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009e50 tx=0x7f22f002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b20c163fa6e3b0ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.465+0000 7f22f67fc640 1 -- 172.21.15.29:0/3292923912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f002ca70 con 0x7f22f8070dc0 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.465+0000 7f22f67fc640 1 -- 172.21.15.29:0/3292923912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f002cbd0 con 0x7f22f8070dc0 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.465+0000 7f22f67fc640 1 -- 172.21.15.29:0/3292923912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22f00343f0 con 0x7f22f8070dc0 2023-10-14T17:00:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 -- 172.21.15.29:0/3292923912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f8070dc0 msgr2=0x7f22f806b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f8070dc0 0x7f22f806b210 secure :-1 s=READY pgs=1584 cs=0 l=1 rev1=1 crypto rx=0x7f22f0009e50 tx=0x7f22f002bed0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 -- 172.21.15.29:0/3292923912 shutdown_connections 2023-10-14T17:00:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f8070dc0 0x7f22f806b210 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 --2- 172.21.15.29:0/3292923912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f815adf0 0x7f22f815be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 -- 172.21.15.29:0/3292923912 >> 172.21.15.29:0/3292923912 conn(0x7f22f80b0d30 msgr2=0x7f22f80b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:31.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 -- 172.21.15.29:0/3292923912 shutdown_connections 2023-10-14T17:00:31.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.466+0000 7f22fe101640 1 -- 172.21.15.29:0/3292923912 wait complete. 2023-10-14T17:00:31.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.468+0000 7f22fe101640 1 Processor -- start 2023-10-14T17:00:31.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.468+0000 7f22fe101640 1 -- start start 2023-10-14T17:00:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f8070dc0 0x7f22f80b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 0x7f22f81efc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80c6ec0 con 0x7f22f806adb0 2023-10-14T17:00:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80b4c90 con 0x7f22f815adf0 2023-10-14T17:00:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22fe101640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22f80c65b0 con 0x7f22f8070dc0 2023-10-14T17:00:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22f77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58060/0 (socket says 172.21.15.29:58060) 2023-10-14T17:00:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.469+0000 7f22f77fe640 1 -- 172.21.15.29:0/4202319072 learned_addr learned my addr 172.21.15.29:0/4202319072 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f6ffd640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f8070dc0 0x7f22f80b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f7fff640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 0x7f22f81efc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 -- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f8070dc0 msgr2=0x7f22f80b4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f8070dc0 0x7f22f80b4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 -- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 msgr2=0x7f22f81efc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 0x7f22f81efc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 -- 172.21.15.29:0/4202319072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22f0009b30 con 0x7f22f806adb0 2023-10-14T17:00:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f7fff640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 0x7f22f81efc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:31.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22f77fe640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d800 tx=0x7f22e800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.470+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800b6c0 con 0x7f22f806adb0 2023-10-14T17:00:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.471+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22e800b820 con 0x7f22f806adb0 2023-10-14T17:00:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.471+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22e800bbb0 con 0x7f22f806adb0 2023-10-14T17:00:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.471+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22f80b2420 con 0x7f22f806adb0 2023-10-14T17:00:31.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.471+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22f80c6b50 con 0x7f22f806adb0 2023-10-14T17:00:31.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.472+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22f80c6d10 con 0x7f22f806adb0 2023-10-14T17:00:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.473+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f22e800bd10 con 0x7f22f806adb0 2023-10-14T17:00:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.474+0000 7f22dffff640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 0x7f22e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.475+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f22e8089d40 con 0x7f22f806adb0 2023-10-14T17:00:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.475+0000 7f22f6ffd640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 0x7f22e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.475+0000 7f22f6ffd640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 0x7f22e406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22ec006fd0 tx=0x7f22ec008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.479+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22e8054c90 con 0x7f22f806adb0 2023-10-14T17:00:31.618 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T17:00:31.617+0000 7f6484c97640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:00:31.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.677+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22f804e160 con 0x7f22f806adb0 2023-10-14T17:00:31.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.680+0000 7f22dffff640 1 -- 172.21.15.29:0/4202319072 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16630 (secure 0 0 0) 0x7f22e80582b0 con 0x7f22f806adb0 2023-10-14T17:00:31.681 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:31.681 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":140,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:28.694398+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.682+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 msgr2=0x7f22e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.682+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 0x7f22e406d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f22ec006fd0 tx=0x7f22ec008040 comp rx=0 tx=0).stop 2023-10-14T17:00:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.682+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 msgr2=0x7f22f80b1e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.682+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 secure :-1 s=READY pgs=1585 cs=0 l=1 rev1=1 crypto rx=0x7f22e800d800 tx=0x7f22e800dcc0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 shutdown_connections 2023-10-14T17:00:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f22e406af90 0x7f22e406d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22f815adf0 0x7f22f81efc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22f8070dc0 0x7f22f80b4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 --2- 172.21.15.29:0/4202319072 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22f806adb0 0x7f22f80b1e40 unknown :-1 s=CLOSED pgs=1585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 >> 172.21.15.29:0/4202319072 conn(0x7f22f80b0d30 msgr2=0x7f22f80aeca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 shutdown_connections 2023-10-14T17:00:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.683+0000 7f22fe101640 1 -- 172.21.15.29:0/4202319072 wait complete. 2023-10-14T17:00:31.702 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T17:00:31.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:00:31.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.925+0000 7f5b1759e640 1 Processor -- start 2023-10-14T17:00:31.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.925+0000 7f5b1759e640 1 -- start start 2023-10-14T17:00:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.926+0000 7f5b1759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b180d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.926+0000 7f5b1759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180d15a0 con 0x7f5b1815cfc0 2023-10-14T17:00:31.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.926+0000 7f5b1759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b180d1880 con 0x7f5b180b4c80 2023-10-14T17:00:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.926+0000 7f5b1759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180d1b60 con 0x7f5b1815cbe0 2023-10-14T17:00:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.927+0000 7f5b15d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5b1815cfc0 0x7f5b180b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35544/0 (socket says 172.21.15.29:35544) 2023-10-14T17:00:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.927+0000 7f5b15d9b640 1 -- 172.21.15.29:0/3222610555 learned_addr learned my addr 172.21.15.29:0/3222610555 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:31.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.927+0000 7f5b1659c640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b180d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.927+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1053504106 0 0) 0x7f5b180d15a0 con 0x7f5b1815cfc0 2023-10-14T17:00:31.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.927+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b04003520 con 0x7f5b1815cfc0 2023-10-14T17:00:31.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1927920581 0 0) 0x7f5b04003520 con 0x7f5b1815cfc0 2023-10-14T17:00:31.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 msgr2=0x7f5b180d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:31.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b180d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 >> v1:172.21.15.186:6789/0 conn(0x7f5b180b4c80 legacy=0x7f5b180b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b180d1e40 con 0x7f5b1815cfc0 2023-10-14T17:00:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5b00001670 con 0x7f5b1815cfc0 2023-10-14T17:00:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 >> v1:172.21.15.29:6789/0 conn(0x7f5b1815cfc0 legacy=0x7f5b180b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b040039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b04003ee0 0x7f5b040047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04004d00 0x7f5b04005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180d1b60 con 0x7f5b04003ee0 2023-10-14T17:00:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0400b9e0 con 0x7f5b04004d00 2023-10-14T17:00:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0400bba0 con 0x7f5b1815cbe0 2023-10-14T17:00:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.928+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b04003ee0 0x7f5b040047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b1659c640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b040039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b16d9d640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04004d00 0x7f5b04005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b15d9b640 1 -- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 msgr2=0x7f5b040039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815cbe0 0x7f5b040039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b15d9b640 1 -- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04004d00 msgr2=0x7f5b04005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:31.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04004d00 0x7f5b04005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.929+0000 7f5b15d9b640 1 -- 172.21.15.29:0/3222610555 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b04003520 con 0x7f5b04003ee0 2023-10-14T17:00:31.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.930+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b04003ee0 0x7f5b040047c0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5b1815d3a0 tx=0x7f5b040057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8fba5703e692ef5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.930+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b00005ad0 con 0x7f5b04003ee0 2023-10-14T17:00:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.930+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b00005c50 con 0x7f5b04003ee0 2023-10-14T17:00:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.930+0000 7f5b1559a640 1 -- 172.21.15.29:0/3222610555 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0002c7d0 con 0x7f5b04003ee0 2023-10-14T17:00:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 -- 172.21.15.29:0/3222610555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b04003ee0 msgr2=0x7f5b040047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 --2- 172.21.15.29:0/3222610555 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b04003ee0 0x7f5b040047c0 secure :-1 s=READY pgs=1587 cs=0 l=1 rev1=1 crypto rx=0x7f5b1815d3a0 tx=0x7f5b040057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b16d9d640 1 -- 172.21.15.29:0/3222610555 reap_dead start 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 -- 172.21.15.29:0/3222610555 shutdown_connections 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 -- 172.21.15.29:0/3222610555 >> 172.21.15.29:0/3222610555 conn(0x7f5b180cf570 msgr2=0x7f5b180cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 -- 172.21.15.29:0/3222610555 shutdown_connections 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.931+0000 7f5b1759e640 1 -- 172.21.15.29:0/3222610555 wait complete. 2023-10-14T17:00:31.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.933+0000 7f5b1759e640 1 Processor -- start 2023-10-14T17:00:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.933+0000 7f5b1759e640 1 -- start start 2023-10-14T17:00:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 0x7f5b181f1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 0x7f5b181f4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b180c87a0 con 0x7f5b181f4fa0 2023-10-14T17:00:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815d960 con 0x7f5b04003ee0 2023-10-14T17:00:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b1759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b1815dc40 con 0x7f5b1815d3a0 2023-10-14T17:00:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b16d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b16d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58084/0 (socket says 172.21.15.29:58084) 2023-10-14T17:00:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.934+0000 7f5b16d9d640 1 -- 172.21.15.29:0/1401503629 learned_addr learned my addr 172.21.15.29:0/1401503629 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 0x7f5b181f4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b1659c640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 0x7f5b181f1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 -- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 msgr2=0x7f5b181f4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 0x7f5b181f4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 -- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 msgr2=0x7f5b181f1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 0x7f5b181f1ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 -- 172.21.15.29:0/1401503629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1815df20 con 0x7f5b181f4fa0 2023-10-14T17:00:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b15d9b640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 0x7f5b181f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:31.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b1659c640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 0x7f5b181f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b16d9d640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b5b70 tx=0x7f5b1000e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b100052e0 con 0x7f5b181f4fa0 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b10005440 con 0x7f5b181f4fa0 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b10003810 con 0x7f5b181f4fa0 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.935+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b1815e200 con 0x7f5b181f4fa0 2023-10-14T17:00:31.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.936+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b1815ef40 con 0x7f5b181f4fa0 2023-10-14T17:00:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.937+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b1815e720 con 0x7f5b181f4fa0 2023-10-14T17:00:31.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.938+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5b10003970 con 0x7f5b181f4fa0 2023-10-14T17:00:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.939+0000 7f5b0effd640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 0x7f5af806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.939+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f5b10088f20 con 0x7f5b181f4fa0 2023-10-14T17:00:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.939+0000 7f5b1659c640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 0x7f5af806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.940+0000 7f5b1659c640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 0x7f5af806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b08007440 tx=0x7f5b0800b040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:31.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:31.944+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b1815e720 con 0x7f5b181f4fa0 2023-10-14T17:00:32.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.136+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 --> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5b1815e720 con 0x7f5af806af90 2023-10-14T17:00:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.165+0000 7f5b0effd640 1 -- 172.21.15.29:0/1401503629 <== mgr.8298 v2:172.21.15.29:6832/3738876524 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286875 (secure 0 0 0) 0x7f5b1815e720 con 0x7f5af806af90 2023-10-14T17:00:32.169 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:32.171 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:00:32.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.171+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 msgr2=0x7f5af806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.171+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 0x7f5af806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f5b08007440 tx=0x7f5b0800b040 comp rx=0 tx=0).stop 2023-10-14T17:00:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.171+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 msgr2=0x7f5b1815ea00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.171+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 secure :-1 s=READY pgs=1588 cs=0 l=1 rev1=1 crypto rx=0x7f5b180b5b70 tx=0x7f5b1000e820 comp rx=0 tx=0).stop 2023-10-14T17:00:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 shutdown_connections 2023-10-14T17:00:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f5af806af90 0x7f5af806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b181f4fa0 0x7f5b1815ea00 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b1815d3a0 0x7f5b181f4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 --2- 172.21.15.29:0/1401503629 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b04003ee0 0x7f5b181f1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 >> 172.21.15.29:0/1401503629 conn(0x7f5b180cf570 msgr2=0x7f5b180cd930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 shutdown_connections 2023-10-14T17:00:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.172+0000 7f5b1759e640 1 -- 172.21.15.29:0/1401503629 wait complete. 2023-10-14T17:00:32.193 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T17:00:30.799910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":359952,"kb_used_data":14732,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754614768,"statfs":{"total":773094113280,"available":772725522432,"internally_reserved":0,"allocated":15085568,"data_stored":7890104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":353357368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.145870"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719339+0000","last_change":"2023-10-14T17:00:27.711292+0000","last_active":"2023-10-14T17:00:27.719339+0000","last_peered":"2023-10-14T17:00:27.719339+0000","last_clean":"2023-10-14T17:00:27.719339+0000","last_became_active":"2023-10-14T17:00:27.710978+0000","last_became_peered":"2023-10-14T17:00:27.710978+0000","last_unstale":"2023-10-14T17:00:27.719339+0000","last_undegraded":"2023-10-14T17:00:27.719339+0000","last_fullsized":"2023-10-14T17:00:27.719339+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:03:16.305171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712365+0000","last_change":"2023-10-14T17:00:27.709657+0000","last_active":"2023-10-14T17:00:27.712365+0000","last_peered":"2023-10-14T17:00:27.712365+0000","last_clean":"2023-10-14T17:00:27.712365+0000","last_became_active":"2023-10-14T17:00:27.709342+0000","last_became_peered":"2023-10-14T17:00:27.709342+0000","last_unstale":"2023-10-14T17:00:27.712365+0000","last_undegraded":"2023-10-14T17:00:27.712365+0000","last_fullsized":"2023-10-14T17:00:27.712365+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:37.963501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712252+0000","last_change":"2023-10-14T17:00:27.711863+0000","last_active":"2023-10-14T17:00:28.712252+0000","last_peered":"2023-10-14T17:00:28.712252+0000","last_clean":"2023-10-14T17:00:28.712252+0000","last_became_active":"2023-10-14T17:00:27.711220+0000","last_became_peered":"2023-10-14T17:00:27.711220+0000","last_unstale":"2023-10-14T17:00:28.712252+0000","last_undegraded":"2023-10-14T17:00:28.712252+0000","last_fullsized":"2023-10-14T17:00:28.712252+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:32:32.026173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.710200+0000","last_change":"2023-10-14T17:00:27.705462+0000","last_active":"2023-10-14T17:00:27.710200+0000","last_peered":"2023-10-14T17:00:27.710200+0000","last_clean":"2023-10-14T17:00:27.710200+0000","last_became_active":"2023-10-14T17:00:27.705146+0000","last_became_peered":"2023-10-14T17:00:27.705146+0000","last_unstale":"2023-10-14T17:00:27.710200+0000","last_undegraded":"2023-10-14T17:00:27.710200+0000","last_fullsized":"2023-10-14T17:00:27.710200+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:51.688094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712707+0000","last_change":"2023-10-14T17:00:27.710529+0000","last_active":"2023-10-14T17:00:27.712707+0000","last_peered":"2023-10-14T17:00:27.712707+0000","last_clean":"2023-10-14T17:00:27.712707+0000","last_became_active":"2023-10-14T17:00:27.708905+0000","last_became_peered":"2023-10-14T17:00:27.708905+0000","last_unstale":"2023-10-14T17:00:27.712707+0000","last_undegraded":"2023-10-14T17:00:27.712707+0000","last_fullsized":"2023-10-14T17:00:27.712707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:37.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724267+0000","last_change":"2023-10-14T17:00:27.712586+0000","last_active":"2023-10-14T17:00:27.724267+0000","last_peered":"2023-10-14T17:00:27.724267+0000","last_clean":"2023-10-14T17:00:27.724267+0000","last_became_active":"2023-10-14T17:00:27.712321+0000","last_became_peered":"2023-10-14T17:00:27.712321+0000","last_unstale":"2023-10-14T17:00:27.724267+0000","last_undegraded":"2023-10-14T17:00:27.724267+0000","last_fullsized":"2023-10-14T17:00:27.724267+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:50.052495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706315+0000","last_change":"2023-10-14T17:00:27.702020+0000","last_active":"2023-10-14T17:00:27.706315+0000","last_peered":"2023-10-14T17:00:27.706315+0000","last_clean":"2023-10-14T17:00:27.706315+0000","last_became_active":"2023-10-14T17:00:27.701441+0000","last_became_peered":"2023-10-14T17:00:27.701441+0000","last_unstale":"2023-10-14T17:00:27.706315+0000","last_undegraded":"2023-10-14T17:00:27.706315+0000","last_fullsized":"2023-10-14T17:00:27.706315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:27.066569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712951+0000","last_change":"2023-10-14T17:00:27.713815+0000","last_active":"2023-10-14T17:00:28.712951+0000","last_peered":"2023-10-14T17:00:28.712951+0000","last_clean":"2023-10-14T17:00:28.712951+0000","last_became_active":"2023-10-14T17:00:27.711147+0000","last_became_peered":"2023-10-14T17:00:27.711147+0000","last_unstale":"2023-10-14T17:00:28.712951+0000","last_undegraded":"2023-10-14T17:00:28.712951+0000","last_fullsized":"2023-10-14T17:00:28.712951+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:30.874300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720638+0000","last_change":"2023-10-14T17:00:27.718112+0000","last_active":"2023-10-14T17:00:27.720638+0000","last_peered":"2023-10-14T17:00:27.720638+0000","last_clean":"2023-10-14T17:00:27.720638+0000","last_became_active":"2023-10-14T17:00:27.715211+0000","last_became_peered":"2023-10-14T17:00:27.715211+0000","last_unstale":"2023-10-14T17:00:27.720638+0000","last_undegraded":"2023-10-14T17:00:27.720638+0000","last_fullsized":"2023-10-14T17:00:27.720638+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:05.215574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720155+0000","last_change":"2023-10-14T17:00:27.714466+0000","last_active":"2023-10-14T17:00:27.720155+0000","last_peered":"2023-10-14T17:00:27.720155+0000","last_clean":"2023-10-14T17:00:27.720155+0000","last_became_active":"2023-10-14T17:00:27.714185+0000","last_became_peered":"2023-10-14T17:00:27.714185+0000","last_unstale":"2023-10-14T17:00:27.720155+0000","last_undegraded":"2023-10-14T17:00:27.720155+0000","last_fullsized":"2023-10-14T17:00:27.720155+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.704307+0000","last_change":"2023-10-14T17:00:27.702645+0000","last_active":"2023-10-14T17:00:27.704307+0000","last_peered":"2023-10-14T17:00:27.704307+0000","last_clean":"2023-10-14T17:00:27.704307+0000","last_became_active":"2023-10-14T17:00:27.702340+0000","last_became_peered":"2023-10-14T17:00:27.702340+0000","last_unstale":"2023-10-14T17:00:27.704307+0000","last_undegraded":"2023-10-14T17:00:27.704307+0000","last_fullsized":"2023-10-14T17:00:27.704307+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:23.818618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.721315+0000","last_change":"2023-10-14T17:00:27.716704+0000","last_active":"2023-10-14T17:00:27.721315+0000","last_peered":"2023-10-14T17:00:27.721315+0000","last_clean":"2023-10-14T17:00:27.721315+0000","last_became_active":"2023-10-14T17:00:27.716431+0000","last_became_peered":"2023-10-14T17:00:27.716431+0000","last_unstale":"2023-10-14T17:00:27.721315+0000","last_undegraded":"2023-10-14T17:00:27.721315+0000","last_fullsized":"2023-10-14T17:00:27.721315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:20.722396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711762+0000","last_change":"2023-10-14T17:00:27.712169+0000","last_active":"2023-10-14T17:00:28.711762+0000","last_peered":"2023-10-14T17:00:28.711762+0000","last_clean":"2023-10-14T17:00:28.711762+0000","last_became_active":"2023-10-14T17:00:27.711716+0000","last_became_peered":"2023-10-14T17:00:27.711716+0000","last_unstale":"2023-10-14T17:00:28.711762+0000","last_undegraded":"2023-10-14T17:00:28.711762+0000","last_fullsized":"2023-10-14T17:00:28.711762+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:26.430276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.710103+0000","last_change":"2023-10-14T17:00:27.714801+0000","last_active":"2023-10-14T17:00:28.710103+0000","last_peered":"2023-10-14T17:00:28.710103+0000","last_clean":"2023-10-14T17:00:28.710103+0000","last_became_active":"2023-10-14T17:00:27.714076+0000","last_became_peered":"2023-10-14T17:00:27.714076+0000","last_unstale":"2023-10-14T17:00:28.710103+0000","last_undegraded":"2023-10-14T17:00:28.710103+0000","last_fullsized":"2023-10-14T17:00:28.710103+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:16:30.342922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.725268+0000","last_change":"2023-10-14T17:00:27.719950+0000","last_active":"2023-10-14T17:00:27.725268+0000","last_peered":"2023-10-14T17:00:27.725268+0000","last_clean":"2023-10-14T17:00:27.725268+0000","last_became_active":"2023-10-14T17:00:27.719656+0000","last_became_peered":"2023-10-14T17:00:27.719656+0000","last_unstale":"2023-10-14T17:00:27.725268+0000","last_undegraded":"2023-10-14T17:00:27.725268+0000","last_fullsized":"2023-10-14T17:00:27.725268+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:06.021014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709303+0000","last_change":"2023-10-14T17:00:27.707309+0000","last_active":"2023-10-14T17:00:27.709303+0000","last_peered":"2023-10-14T17:00:27.709303+0000","last_clean":"2023-10-14T17:00:27.709303+0000","last_became_active":"2023-10-14T17:00:27.706994+0000","last_became_peered":"2023-10-14T17:00:27.706994+0000","last_unstale":"2023-10-14T17:00:27.709303+0000","last_undegraded":"2023-10-14T17:00:27.709303+0000","last_fullsized":"2023-10-14T17:00:27.709303+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:25.770850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712254+0000","last_change":"2023-10-14T17:00:27.707603+0000","last_active":"2023-10-14T17:00:27.712254+0000","last_peered":"2023-10-14T17:00:27.712254+0000","last_clean":"2023-10-14T17:00:27.712254+0000","last_became_active":"2023-10-14T17:00:27.707316+0000","last_became_peered":"2023-10-14T17:00:27.707316+0000","last_unstale":"2023-10-14T17:00:27.712254+0000","last_undegraded":"2023-10-14T17:00:27.712254+0000","last_fullsized":"2023-10-14T17:00:27.712254+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:15.286260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711615+0000","last_change":"2023-10-14T17:00:27.699374+0000","last_active":"2023-10-14T17:00:28.711615+0000","last_peered":"2023-10-14T17:00:28.711615+0000","last_clean":"2023-10-14T17:00:28.711615+0000","last_became_active":"2023-10-14T17:00:27.697480+0000","last_became_peered":"2023-10-14T17:00:27.697480+0000","last_unstale":"2023-10-14T17:00:28.711615+0000","last_undegraded":"2023-10-14T17:00:28.711615+0000","last_fullsized":"2023-10-14T17:00:28.711615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:31.630069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712569+0000","last_change":"2023-10-14T17:00:27.713718+0000","last_active":"2023-10-14T17:00:28.712569+0000","last_peered":"2023-10-14T17:00:28.712569+0000","last_clean":"2023-10-14T17:00:28.712569+0000","last_became_active":"2023-10-14T17:00:27.713413+0000","last_became_peered":"2023-10-14T17:00:27.713413+0000","last_unstale":"2023-10-14T17:00:28.712569+0000","last_undegraded":"2023-10-14T17:00:28.712569+0000","last_fullsized":"2023-10-14T17:00:28.712569+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:24.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712603+0000","last_change":"2023-10-14T17:00:27.710293+0000","last_active":"2023-10-14T17:00:27.712603+0000","last_peered":"2023-10-14T17:00:27.712603+0000","last_clean":"2023-10-14T17:00:27.712603+0000","last_became_active":"2023-10-14T17:00:27.709700+0000","last_became_peered":"2023-10-14T17:00:27.709700+0000","last_unstale":"2023-10-14T17:00:27.712603+0000","last_undegraded":"2023-10-14T17:00:27.712603+0000","last_fullsized":"2023-10-14T17:00:27.712603+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:06.433862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720326+0000","last_change":"2023-10-14T17:00:27.719603+0000","last_active":"2023-10-14T17:00:27.720326+0000","last_peered":"2023-10-14T17:00:27.720326+0000","last_clean":"2023-10-14T17:00:27.720326+0000","last_became_active":"2023-10-14T17:00:27.719341+0000","last_became_peered":"2023-10-14T17:00:27.719341+0000","last_unstale":"2023-10-14T17:00:27.720326+0000","last_undegraded":"2023-10-14T17:00:27.720326+0000","last_fullsized":"2023-10-14T17:00:27.720326+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:40.799944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706938+0000","last_change":"2023-10-14T17:00:27.712322+0000","last_active":"2023-10-14T17:00:28.706938+0000","last_peered":"2023-10-14T17:00:28.706938+0000","last_clean":"2023-10-14T17:00:28.706938+0000","last_became_active":"2023-10-14T17:00:27.712049+0000","last_became_peered":"2023-10-14T17:00:27.712049+0000","last_unstale":"2023-10-14T17:00:28.706938+0000","last_undegraded":"2023-10-14T17:00:28.706938+0000","last_fullsized":"2023-10-14T17:00:28.706938+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:24.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712290+0000","last_change":"2023-10-14T17:00:27.710507+0000","last_active":"2023-10-14T17:00:27.712290+0000","last_peered":"2023-10-14T17:00:27.712290+0000","last_clean":"2023-10-14T17:00:27.712290+0000","last_became_active":"2023-10-14T17:00:27.710225+0000","last_became_peered":"2023-10-14T17:00:27.710225+0000","last_unstale":"2023-10-14T17:00:27.712290+0000","last_undegraded":"2023-10-14T17:00:27.712290+0000","last_fullsized":"2023-10-14T17:00:27.712290+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:29.621105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706399+0000","last_change":"2023-10-14T17:00:27.720944+0000","last_active":"2023-10-14T17:00:28.706399+0000","last_peered":"2023-10-14T17:00:28.706399+0000","last_clean":"2023-10-14T17:00:28.706399+0000","last_became_active":"2023-10-14T17:00:27.720652+0000","last_became_peered":"2023-10-14T17:00:27.720652+0000","last_unstale":"2023-10-14T17:00:28.706399+0000","last_undegraded":"2023-10-14T17:00:28.706399+0000","last_fullsized":"2023-10-14T17:00:28.706399+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:10.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.702817+0000","last_change":"2023-10-14T17:00:27.701211+0000","last_active":"2023-10-14T17:00:27.702817+0000","last_peered":"2023-10-14T17:00:27.702817+0000","last_clean":"2023-10-14T17:00:27.702817+0000","last_became_active":"2023-10-14T17:00:27.700076+0000","last_became_peered":"2023-10-14T17:00:27.700076+0000","last_unstale":"2023-10-14T17:00:27.702817+0000","last_undegraded":"2023-10-14T17:00:27.702817+0000","last_fullsized":"2023-10-14T17:00:27.702817+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:11.493657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709721+0000","last_change":"2023-10-14T17:00:27.706133+0000","last_active":"2023-10-14T17:00:27.709721+0000","last_peered":"2023-10-14T17:00:27.709721+0000","last_clean":"2023-10-14T17:00:27.709721+0000","last_became_active":"2023-10-14T17:00:27.704391+0000","last_became_peered":"2023-10-14T17:00:27.704391+0000","last_unstale":"2023-10-14T17:00:27.709721+0000","last_undegraded":"2023-10-14T17:00:27.709721+0000","last_fullsized":"2023-10-14T17:00:27.709721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:23.280133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724015+0000","last_change":"2023-10-14T17:00:27.715949+0000","last_active":"2023-10-14T17:00:27.724015+0000","last_peered":"2023-10-14T17:00:27.724015+0000","last_clean":"2023-10-14T17:00:27.724015+0000","last_became_active":"2023-10-14T17:00:27.715672+0000","last_became_peered":"2023-10-14T17:00:27.715672+0000","last_unstale":"2023-10-14T17:00:27.724015+0000","last_undegraded":"2023-10-14T17:00:27.724015+0000","last_fullsized":"2023-10-14T17:00:27.724015+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:19:26.034995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707181+0000","last_change":"2023-10-14T17:00:27.723716+0000","last_active":"2023-10-14T17:00:28.707181+0000","last_peered":"2023-10-14T17:00:28.707181+0000","last_clean":"2023-10-14T17:00:28.707181+0000","last_became_active":"2023-10-14T17:00:27.723386+0000","last_became_peered":"2023-10-14T17:00:27.723386+0000","last_unstale":"2023-10-14T17:00:28.707181+0000","last_undegraded":"2023-10-14T17:00:28.707181+0000","last_fullsized":"2023-10-14T17:00:28.707181+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:16.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712655+0000","last_change":"2023-10-14T17:00:27.710388+0000","last_active":"2023-10-14T17:00:27.712655+0000","last_peered":"2023-10-14T17:00:27.712655+0000","last_clean":"2023-10-14T17:00:27.712655+0000","last_became_active":"2023-10-14T17:00:27.709675+0000","last_became_peered":"2023-10-14T17:00:27.709675+0000","last_unstale":"2023-10-14T17:00:27.712655+0000","last_undegraded":"2023-10-14T17:00:27.712655+0000","last_fullsized":"2023-10-14T17:00:27.712655+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:03:29.832656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.722684+0000","last_change":"2023-10-14T17:00:27.721858+0000","last_active":"2023-10-14T17:00:27.722684+0000","last_peered":"2023-10-14T17:00:27.722684+0000","last_clean":"2023-10-14T17:00:27.722684+0000","last_became_active":"2023-10-14T17:00:27.721519+0000","last_became_peered":"2023-10-14T17:00:27.721519+0000","last_unstale":"2023-10-14T17:00:27.722684+0000","last_undegraded":"2023-10-14T17:00:27.722684+0000","last_fullsized":"2023-10-14T17:00:27.722684+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:37.963501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709460+0000","last_change":"2023-10-14T17:00:27.706505+0000","last_active":"2023-10-14T17:00:27.709460+0000","last_peered":"2023-10-14T17:00:27.709460+0000","last_clean":"2023-10-14T17:00:27.709460+0000","last_became_active":"2023-10-14T17:00:27.706191+0000","last_became_peered":"2023-10-14T17:00:27.706191+0000","last_unstale":"2023-10-14T17:00:27.709460+0000","last_undegraded":"2023-10-14T17:00:27.709460+0000","last_fullsized":"2023-10-14T17:00:27.709460+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:32.958910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.725492+0000","last_change":"2023-10-14T17:00:27.721161+0000","last_active":"2023-10-14T17:00:27.725492+0000","last_peered":"2023-10-14T17:00:27.725492+0000","last_clean":"2023-10-14T17:00:27.725492+0000","last_became_active":"2023-10-14T17:00:27.720548+0000","last_became_peered":"2023-10-14T17:00:27.720548+0000","last_unstale":"2023-10-14T17:00:27.725492+0000","last_undegraded":"2023-10-14T17:00:27.725492+0000","last_fullsized":"2023-10-14T17:00:27.725492+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:31:40.798584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.723440+0000","last_change":"2023-10-14T17:00:27.714454+0000","last_active":"2023-10-14T17:00:27.723440+0000","last_peered":"2023-10-14T17:00:27.723440+0000","last_clean":"2023-10-14T17:00:27.723440+0000","last_became_active":"2023-10-14T17:00:27.711861+0000","last_became_peered":"2023-10-14T17:00:27.711861+0000","last_unstale":"2023-10-14T17:00:27.723440+0000","last_undegraded":"2023-10-14T17:00:27.723440+0000","last_fullsized":"2023-10-14T17:00:27.723440+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:25.770850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.729653+0000","last_change":"2023-10-14T17:00:26.702397+0000","last_active":"2023-10-14T17:00:27.729653+0000","last_peered":"2023-10-14T17:00:27.729653+0000","last_clean":"2023-10-14T17:00:27.729653+0000","last_became_active":"2023-10-14T17:00:26.701862+0000","last_became_peered":"2023-10-14T17:00:26.701862+0000","last_unstale":"2023-10-14T17:00:27.729653+0000","last_undegraded":"2023-10-14T17:00:27.729653+0000","last_fullsized":"2023-10-14T17:00:27.729653+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:39.785613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712404+0000","last_change":"2023-10-14T17:00:27.706152+0000","last_active":"2023-10-14T17:00:28.712404+0000","last_peered":"2023-10-14T17:00:28.712404+0000","last_clean":"2023-10-14T17:00:28.712404+0000","last_became_active":"2023-10-14T17:00:27.705830+0000","last_became_peered":"2023-10-14T17:00:27.705830+0000","last_unstale":"2023-10-14T17:00:28.712404+0000","last_undegraded":"2023-10-14T17:00:28.712404+0000","last_fullsized":"2023-10-14T17:00:28.712404+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:24.963274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712269+0000","last_change":"2023-10-14T17:00:26.703407+0000","last_active":"2023-10-14T17:00:28.712269+0000","last_peered":"2023-10-14T17:00:28.712269+0000","last_clean":"2023-10-14T17:00:28.712269+0000","last_became_active":"2023-10-14T17:00:26.703079+0000","last_became_peered":"2023-10-14T17:00:26.703079+0000","last_unstale":"2023-10-14T17:00:28.712269+0000","last_undegraded":"2023-10-14T17:00:28.712269+0000","last_fullsized":"2023-10-14T17:00:28.712269+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:46:19.839679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.727970+0000","last_change":"2023-10-14T17:00:27.721431+0000","last_active":"2023-10-14T17:00:27.727970+0000","last_peered":"2023-10-14T17:00:27.727970+0000","last_clean":"2023-10-14T17:00:27.727970+0000","last_became_active":"2023-10-14T17:00:27.721168+0000","last_became_peered":"2023-10-14T17:00:27.721168+0000","last_unstale":"2023-10-14T17:00:27.727970+0000","last_undegraded":"2023-10-14T17:00:27.727970+0000","last_fullsized":"2023-10-14T17:00:27.727970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:46:01.758782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.735885+0000","last_change":"2023-10-14T17:00:26.710602+0000","last_active":"2023-10-14T17:00:27.735885+0000","last_peered":"2023-10-14T17:00:27.735885+0000","last_clean":"2023-10-14T17:00:27.735885+0000","last_became_active":"2023-10-14T17:00:26.710315+0000","last_became_peered":"2023-10-14T17:00:26.710315+0000","last_unstale":"2023-10-14T17:00:27.735885+0000","last_undegraded":"2023-10-14T17:00:27.735885+0000","last_fullsized":"2023-10-14T17:00:27.735885+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:48.413979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712840+0000","last_change":"2023-10-14T17:00:27.713830+0000","last_active":"2023-10-14T17:00:28.712840+0000","last_peered":"2023-10-14T17:00:28.712840+0000","last_clean":"2023-10-14T17:00:28.712840+0000","last_became_active":"2023-10-14T17:00:27.713198+0000","last_became_peered":"2023-10-14T17:00:27.713198+0000","last_unstale":"2023-10-14T17:00:28.712840+0000","last_undegraded":"2023-10-14T17:00:28.712840+0000","last_fullsized":"2023-10-14T17:00:28.712840+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:16.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.737748+0000","last_change":"2023-10-14T17:00:26.710197+0000","last_active":"2023-10-14T17:00:27.737748+0000","last_peered":"2023-10-14T17:00:27.737748+0000","last_clean":"2023-10-14T17:00:27.737748+0000","last_became_active":"2023-10-14T17:00:26.709462+0000","last_became_peered":"2023-10-14T17:00:26.709462+0000","last_unstale":"2023-10-14T17:00:27.737748+0000","last_undegraded":"2023-10-14T17:00:27.737748+0000","last_fullsized":"2023-10-14T17:00:27.737748+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:51:16.222918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720586+0000","last_change":"2023-10-14T17:00:27.715688+0000","last_active":"2023-10-14T17:00:27.720586+0000","last_peered":"2023-10-14T17:00:27.720586+0000","last_clean":"2023-10-14T17:00:27.720586+0000","last_became_active":"2023-10-14T17:00:27.715387+0000","last_became_peered":"2023-10-14T17:00:27.715387+0000","last_unstale":"2023-10-14T17:00:27.720586+0000","last_undegraded":"2023-10-14T17:00:27.720586+0000","last_fullsized":"2023-10-14T17:00:27.720586+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:31.630069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.734954+0000","last_change":"2023-10-14T17:00:26.701479+0000","last_active":"2023-10-14T17:00:27.734954+0000","last_peered":"2023-10-14T17:00:27.734954+0000","last_clean":"2023-10-14T17:00:27.734954+0000","last_became_active":"2023-10-14T17:00:26.700997+0000","last_became_peered":"2023-10-14T17:00:26.700997+0000","last_unstale":"2023-10-14T17:00:27.734954+0000","last_undegraded":"2023-10-14T17:00:27.734954+0000","last_fullsized":"2023-10-14T17:00:27.734954+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:05.832205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712637+0000","last_change":"2023-10-14T17:00:27.709667+0000","last_active":"2023-10-14T17:00:27.712637+0000","last_peered":"2023-10-14T17:00:27.712637+0000","last_clean":"2023-10-14T17:00:27.712637+0000","last_became_active":"2023-10-14T17:00:27.709389+0000","last_became_peered":"2023-10-14T17:00:27.709389+0000","last_unstale":"2023-10-14T17:00:27.712637+0000","last_undegraded":"2023-10-14T17:00:27.712637+0000","last_fullsized":"2023-10-14T17:00:27.712637+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:47:15.905006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707041+0000","last_change":"2023-10-14T17:00:26.703372+0000","last_active":"2023-10-14T17:00:28.707041+0000","last_peered":"2023-10-14T17:00:28.707041+0000","last_clean":"2023-10-14T17:00:28.707041+0000","last_became_active":"2023-10-14T17:00:26.703096+0000","last_became_peered":"2023-10-14T17:00:26.703096+0000","last_unstale":"2023-10-14T17:00:28.707041+0000","last_undegraded":"2023-10-14T17:00:28.707041+0000","last_fullsized":"2023-10-14T17:00:28.707041+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:16:34.226749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719368+0000","last_change":"2023-10-14T17:00:27.713122+0000","last_active":"2023-10-14T17:00:27.719368+0000","last_peered":"2023-10-14T17:00:27.719368+0000","last_clean":"2023-10-14T17:00:27.719368+0000","last_became_active":"2023-10-14T17:00:27.712805+0000","last_became_peered":"2023-10-14T17:00:27.712805+0000","last_unstale":"2023-10-14T17:00:27.719368+0000","last_undegraded":"2023-10-14T17:00:27.719368+0000","last_fullsized":"2023-10-14T17:00:27.719368+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:19:26.034995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.691343+0000","last_change":"2023-10-14T17:00:26.694095+0000","last_active":"2023-10-14T17:00:27.691343+0000","last_peered":"2023-10-14T17:00:27.691343+0000","last_clean":"2023-10-14T17:00:27.691343+0000","last_became_active":"2023-10-14T17:00:26.693729+0000","last_became_peered":"2023-10-14T17:00:26.693729+0000","last_unstale":"2023-10-14T17:00:27.691343+0000","last_undegraded":"2023-10-14T17:00:27.691343+0000","last_fullsized":"2023-10-14T17:00:27.691343+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:08:25.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706334+0000","last_change":"2023-10-14T17:00:27.702208+0000","last_active":"2023-10-14T17:00:27.706334+0000","last_peered":"2023-10-14T17:00:27.706334+0000","last_clean":"2023-10-14T17:00:27.706334+0000","last_became_active":"2023-10-14T17:00:27.701445+0000","last_became_peered":"2023-10-14T17:00:27.701445+0000","last_unstale":"2023-10-14T17:00:27.706334+0000","last_undegraded":"2023-10-14T17:00:27.706334+0000","last_fullsized":"2023-10-14T17:00:27.706334+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:16:33.235089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.690298+0000","last_change":"2023-10-14T17:00:26.686342+0000","last_active":"2023-10-14T17:00:27.690298+0000","last_peered":"2023-10-14T17:00:27.690298+0000","last_clean":"2023-10-14T17:00:27.690298+0000","last_became_active":"2023-10-14T17:00:26.685947+0000","last_became_peered":"2023-10-14T17:00:26.685947+0000","last_unstale":"2023-10-14T17:00:27.690298+0000","last_undegraded":"2023-10-14T17:00:27.690298+0000","last_fullsized":"2023-10-14T17:00:27.690298+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706771+0000","last_change":"2023-10-14T17:00:27.702814+0000","last_active":"2023-10-14T17:00:27.706771+0000","last_peered":"2023-10-14T17:00:27.706771+0000","last_clean":"2023-10-14T17:00:27.706771+0000","last_became_active":"2023-10-14T17:00:27.702502+0000","last_became_peered":"2023-10-14T17:00:27.702502+0000","last_unstale":"2023-10-14T17:00:27.706771+0000","last_undegraded":"2023-10-14T17:00:27.706771+0000","last_fullsized":"2023-10-14T17:00:27.706771+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:29.621105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.694144+0000","last_change":"2023-10-14T17:00:26.707443+0000","last_active":"2023-10-14T17:00:27.694144+0000","last_peered":"2023-10-14T17:00:27.694144+0000","last_clean":"2023-10-14T17:00:27.694144+0000","last_became_active":"2023-10-14T17:00:26.706297+0000","last_became_peered":"2023-10-14T17:00:26.706297+0000","last_unstale":"2023-10-14T17:00:27.694144+0000","last_undegraded":"2023-10-14T17:00:27.694144+0000","last_fullsized":"2023-10-14T17:00:27.694144+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:31:39.784253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712615+0000","last_change":"2023-10-14T17:00:27.707703+0000","last_active":"2023-10-14T17:00:28.712615+0000","last_peered":"2023-10-14T17:00:28.712615+0000","last_clean":"2023-10-14T17:00:28.712615+0000","last_became_active":"2023-10-14T17:00:27.707412+0000","last_became_peered":"2023-10-14T17:00:27.707412+0000","last_unstale":"2023-10-14T17:00:28.712615+0000","last_undegraded":"2023-10-14T17:00:28.712615+0000","last_fullsized":"2023-10-14T17:00:28.712615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:20.722396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706883+0000","last_change":"2023-10-14T17:00:26.701323+0000","last_active":"2023-10-14T17:00:28.706883+0000","last_peered":"2023-10-14T17:00:28.706883+0000","last_clean":"2023-10-14T17:00:28.706883+0000","last_became_active":"2023-10-14T17:00:26.701051+0000","last_became_peered":"2023-10-14T17:00:26.701051+0000","last_unstale":"2023-10-14T17:00:28.706883+0000","last_undegraded":"2023-10-14T17:00:28.706883+0000","last_fullsized":"2023-10-14T17:00:28.706883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:14.458842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707250+0000","last_change":"2023-10-14T17:00:27.712414+0000","last_active":"2023-10-14T17:00:28.707250+0000","last_peered":"2023-10-14T17:00:28.707250+0000","last_clean":"2023-10-14T17:00:28.707250+0000","last_became_active":"2023-10-14T17:00:27.707590+0000","last_became_peered":"2023-10-14T17:00:27.707590+0000","last_unstale":"2023-10-14T17:00:28.707250+0000","last_undegraded":"2023-10-14T17:00:28.707250+0000","last_fullsized":"2023-10-14T17:00:28.707250+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:30.874300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.736188+0000","last_change":"2023-10-14T17:00:26.708689+0000","last_active":"2023-10-14T17:00:27.736188+0000","last_peered":"2023-10-14T17:00:27.736188+0000","last_clean":"2023-10-14T17:00:27.736188+0000","last_became_active":"2023-10-14T17:00:26.707678+0000","last_became_peered":"2023-10-14T17:00:26.707678+0000","last_unstale":"2023-10-14T17:00:27.736188+0000","last_undegraded":"2023-10-14T17:00:27.736188+0000","last_fullsized":"2023-10-14T17:00:27.736188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:16:32.220758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.701334+0000","last_change":"2023-10-14T17:00:27.698367+0000","last_active":"2023-10-14T17:00:27.701334+0000","last_peered":"2023-10-14T17:00:27.701334+0000","last_clean":"2023-10-14T17:00:27.701334+0000","last_became_active":"2023-10-14T17:00:27.698060+0000","last_became_peered":"2023-10-14T17:00:27.698060+0000","last_unstale":"2023-10-14T17:00:27.701334+0000","last_undegraded":"2023-10-14T17:00:27.701334+0000","last_fullsized":"2023-10-14T17:00:27.701334+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:47:15.905006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.738281+0000","last_change":"2023-10-14T17:00:26.705089+0000","last_active":"2023-10-14T17:00:27.738281+0000","last_peered":"2023-10-14T17:00:27.738281+0000","last_clean":"2023-10-14T17:00:27.738281+0000","last_became_active":"2023-10-14T17:00:26.704625+0000","last_became_peered":"2023-10-14T17:00:26.704625+0000","last_unstale":"2023-10-14T17:00:27.738281+0000","last_undegraded":"2023-10-14T17:00:27.738281+0000","last_fullsized":"2023-10-14T17:00:27.738281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:06.589035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706920+0000","last_change":"2023-10-14T17:00:27.711701+0000","last_active":"2023-10-14T17:00:28.706920+0000","last_peered":"2023-10-14T17:00:28.706920+0000","last_clean":"2023-10-14T17:00:28.706920+0000","last_became_active":"2023-10-14T17:00:27.711392+0000","last_became_peered":"2023-10-14T17:00:27.711392+0000","last_unstale":"2023-10-14T17:00:28.706920+0000","last_undegraded":"2023-10-14T17:00:28.706920+0000","last_fullsized":"2023-10-14T17:00:28.706920+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:32:32.026173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.736014+0000","last_change":"2023-10-14T17:00:26.707567+0000","last_active":"2023-10-14T17:00:27.736014+0000","last_peered":"2023-10-14T17:00:27.736014+0000","last_clean":"2023-10-14T17:00:27.736014+0000","last_became_active":"2023-10-14T17:00:26.707306+0000","last_became_peered":"2023-10-14T17:00:26.707306+0000","last_unstale":"2023-10-14T17:00:27.736014+0000","last_undegraded":"2023-10-14T17:00:27.736014+0000","last_fullsized":"2023-10-14T17:00:27.736014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:59.169566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706625+0000","last_change":"2023-10-14T17:00:27.711844+0000","last_active":"2023-10-14T17:00:28.706625+0000","last_peered":"2023-10-14T17:00:28.706625+0000","last_clean":"2023-10-14T17:00:28.706625+0000","last_became_active":"2023-10-14T17:00:27.711271+0000","last_became_peered":"2023-10-14T17:00:27.711271+0000","last_unstale":"2023-10-14T17:00:28.706625+0000","last_undegraded":"2023-10-14T17:00:28.706625+0000","last_fullsized":"2023-10-14T17:00:28.706625+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:13.083572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711476+0000","last_change":"2023-10-14T17:00:26.696467+0000","last_active":"2023-10-14T17:00:28.711476+0000","last_peered":"2023-10-14T17:00:28.711476+0000","last_clean":"2023-10-14T17:00:28.711476+0000","last_became_active":"2023-10-14T17:00:26.694771+0000","last_became_peered":"2023-10-14T17:00:26.694771+0000","last_unstale":"2023-10-14T17:00:28.711476+0000","last_undegraded":"2023-10-14T17:00:28.711476+0000","last_fullsized":"2023-10-14T17:00:28.711476+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:49.038164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706454+0000","last_change":"2023-10-14T17:00:27.718385+0000","last_active":"2023-10-14T17:00:28.706454+0000","last_peered":"2023-10-14T17:00:28.706454+0000","last_clean":"2023-10-14T17:00:28.706454+0000","last_became_active":"2023-10-14T17:00:27.716541+0000","last_became_peered":"2023-10-14T17:00:27.716541+0000","last_unstale":"2023-10-14T17:00:28.706454+0000","last_undegraded":"2023-10-14T17:00:28.706454+0000","last_fullsized":"2023-10-14T17:00:28.706454+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:39.359877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":19,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.737331+0000","last_change":"2023-10-14T17:00:26.696555+0000","last_active":"2023-10-14T17:00:27.737331+0000","last_peered":"2023-10-14T17:00:27.737331+0000","last_clean":"2023-10-14T17:00:27.737331+0000","last_became_active":"2023-10-14T17:00:26.696252+0000","last_became_peered":"2023-10-14T17:00:26.696252+0000","last_unstale":"2023-10-14T17:00:27.737331+0000","last_undegraded":"2023-10-14T17:00:27.737331+0000","last_fullsized":"2023-10-14T17:00:27.737331+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:56.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.714664+0000","last_change":"2023-10-14T17:00:27.712967+0000","last_active":"2023-10-14T17:00:27.714664+0000","last_peered":"2023-10-14T17:00:27.714664+0000","last_clean":"2023-10-14T17:00:27.714664+0000","last_became_active":"2023-10-14T17:00:27.711569+0000","last_became_peered":"2023-10-14T17:00:27.711569+0000","last_unstale":"2023-10-14T17:00:27.714664+0000","last_undegraded":"2023-10-14T17:00:27.714664+0000","last_fullsized":"2023-10-14T17:00:27.714664+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:09.491629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.733754+0000","last_change":"2023-10-14T17:00:26.700894+0000","last_active":"2023-10-14T17:00:27.733754+0000","last_peered":"2023-10-14T17:00:27.733754+0000","last_clean":"2023-10-14T17:00:27.733754+0000","last_became_active":"2023-10-14T17:00:26.700460+0000","last_became_peered":"2023-10-14T17:00:26.700460+0000","last_unstale":"2023-10-14T17:00:27.733754+0000","last_undegraded":"2023-10-14T17:00:27.733754+0000","last_fullsized":"2023-10-14T17:00:27.733754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:23.081298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724589+0000","last_change":"2023-10-14T17:00:27.718095+0000","last_active":"2023-10-14T17:00:27.724589+0000","last_peered":"2023-10-14T17:00:27.724589+0000","last_clean":"2023-10-14T17:00:27.724589+0000","last_became_active":"2023-10-14T17:00:27.717779+0000","last_became_peered":"2023-10-14T17:00:27.717779+0000","last_unstale":"2023-10-14T17:00:27.724589+0000","last_undegraded":"2023-10-14T17:00:27.724589+0000","last_fullsized":"2023-10-14T17:00:27.724589+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.731039+0000","last_change":"2023-10-14T17:00:26.702217+0000","last_active":"2023-10-14T17:00:27.731039+0000","last_peered":"2023-10-14T17:00:27.731039+0000","last_clean":"2023-10-14T17:00:27.731039+0000","last_became_active":"2023-10-14T17:00:26.701945+0000","last_became_peered":"2023-10-14T17:00:26.701945+0000","last_unstale":"2023-10-14T17:00:27.731039+0000","last_undegraded":"2023-10-14T17:00:27.731039+0000","last_fullsized":"2023-10-14T17:00:27.731039+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:33:36.818209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.714037+0000","last_change":"2023-10-14T17:00:27.708636+0000","last_active":"2023-10-14T17:00:28.714037+0000","last_peered":"2023-10-14T17:00:28.714037+0000","last_clean":"2023-10-14T17:00:28.714037+0000","last_became_active":"2023-10-14T17:00:27.708304+0000","last_became_peered":"2023-10-14T17:00:27.708304+0000","last_unstale":"2023-10-14T17:00:28.714037+0000","last_undegraded":"2023-10-14T17:00:28.714037+0000","last_fullsized":"2023-10-14T17:00:28.714037+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:32:27.367690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711711+0000","last_change":"2023-10-14T17:00:26.696701+0000","last_active":"2023-10-14T17:00:28.711711+0000","last_peered":"2023-10-14T17:00:28.711711+0000","last_clean":"2023-10-14T17:00:28.711711+0000","last_became_active":"2023-10-14T17:00:26.695741+0000","last_became_peered":"2023-10-14T17:00:26.695741+0000","last_unstale":"2023-10-14T17:00:28.711711+0000","last_undegraded":"2023-10-14T17:00:28.711711+0000","last_fullsized":"2023-10-14T17:00:28.711711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:22.265802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713162+0000","last_change":"2023-10-14T17:00:27.710579+0000","last_active":"2023-10-14T17:00:28.713162+0000","last_peered":"2023-10-14T17:00:28.713162+0000","last_clean":"2023-10-14T17:00:28.713162+0000","last_became_active":"2023-10-14T17:00:27.710288+0000","last_became_peered":"2023-10-14T17:00:27.710288+0000","last_unstale":"2023-10-14T17:00:28.713162+0000","last_undegraded":"2023-10-14T17:00:28.713162+0000","last_fullsized":"2023-10-14T17:00:28.713162+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:35.948608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.683720+0000","last_change":"2023-10-14T17:00:26.706134+0000","last_active":"2023-10-14T17:00:27.683720+0000","last_peered":"2023-10-14T17:00:27.683720+0000","last_clean":"2023-10-14T17:00:27.683720+0000","last_became_active":"2023-10-14T17:00:26.705886+0000","last_became_peered":"2023-10-14T17:00:26.705886+0000","last_unstale":"2023-10-14T17:00:27.683720+0000","last_undegraded":"2023-10-14T17:00:27.683720+0000","last_fullsized":"2023-10-14T17:00:27.683720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:22.804287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.710478+0000","last_change":"2023-10-14T17:00:27.716777+0000","last_active":"2023-10-14T17:00:28.710478+0000","last_peered":"2023-10-14T17:00:28.710478+0000","last_clean":"2023-10-14T17:00:28.710478+0000","last_became_active":"2023-10-14T17:00:27.715904+0000","last_became_peered":"2023-10-14T17:00:27.715904+0000","last_unstale":"2023-10-14T17:00:28.710478+0000","last_undegraded":"2023-10-14T17:00:28.710478+0000","last_fullsized":"2023-10-14T17:00:28.710478+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:13.083572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.735608+0000","last_change":"2023-10-14T17:00:26.698897+0000","last_active":"2023-10-14T17:00:27.735608+0000","last_peered":"2023-10-14T17:00:27.735608+0000","last_clean":"2023-10-14T17:00:27.735608+0000","last_became_active":"2023-10-14T17:00:26.698601+0000","last_became_peered":"2023-10-14T17:00:26.698601+0000","last_unstale":"2023-10-14T17:00:27.735608+0000","last_undegraded":"2023-10-14T17:00:27.735608+0000","last_fullsized":"2023-10-14T17:00:27.735608+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719219+0000","last_change":"2023-10-14T17:00:27.710137+0000","last_active":"2023-10-14T17:00:27.719219+0000","last_peered":"2023-10-14T17:00:27.719219+0000","last_clean":"2023-10-14T17:00:27.719219+0000","last_became_active":"2023-10-14T17:00:27.709841+0000","last_became_peered":"2023-10-14T17:00:27.709841+0000","last_unstale":"2023-10-14T17:00:27.719219+0000","last_undegraded":"2023-10-14T17:00:27.719219+0000","last_fullsized":"2023-10-14T17:00:27.719219+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:55.637606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.692426+0000","last_change":"2023-10-14T17:00:26.705121+0000","last_active":"2023-10-14T17:00:27.692426+0000","last_peered":"2023-10-14T17:00:27.692426+0000","last_clean":"2023-10-14T17:00:27.692426+0000","last_became_active":"2023-10-14T17:00:26.704834+0000","last_became_peered":"2023-10-14T17:00:26.704834+0000","last_unstale":"2023-10-14T17:00:27.692426+0000","last_undegraded":"2023-10-14T17:00:27.692426+0000","last_fullsized":"2023-10-14T17:00:27.692426+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:17.378788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713259+0000","last_change":"2023-10-14T17:00:27.715539+0000","last_active":"2023-10-14T17:00:28.713259+0000","last_peered":"2023-10-14T17:00:28.713259+0000","last_clean":"2023-10-14T17:00:28.713259+0000","last_became_active":"2023-10-14T17:00:27.714489+0000","last_became_peered":"2023-10-14T17:00:27.714489+0000","last_unstale":"2023-10-14T17:00:28.713259+0000","last_undegraded":"2023-10-14T17:00:28.713259+0000","last_fullsized":"2023-10-14T17:00:28.713259+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:20.652106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.697629+0000","last_change":"2023-10-14T17:00:26.698957+0000","last_active":"2023-10-14T17:00:27.697629+0000","last_peered":"2023-10-14T17:00:27.697629+0000","last_clean":"2023-10-14T17:00:27.697629+0000","last_became_active":"2023-10-14T17:00:26.698336+0000","last_became_peered":"2023-10-14T17:00:26.698336+0000","last_unstale":"2023-10-14T17:00:27.697629+0000","last_undegraded":"2023-10-14T17:00:27.697629+0000","last_fullsized":"2023-10-14T17:00:27.697629+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:22.265802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706545+0000","last_change":"2023-10-14T17:00:27.722973+0000","last_active":"2023-10-14T17:00:28.706545+0000","last_peered":"2023-10-14T17:00:28.706545+0000","last_clean":"2023-10-14T17:00:28.706545+0000","last_became_active":"2023-10-14T17:00:27.721756+0000","last_became_peered":"2023-10-14T17:00:27.721756+0000","last_unstale":"2023-10-14T17:00:28.706545+0000","last_undegraded":"2023-10-14T17:00:28.706545+0000","last_fullsized":"2023-10-14T17:00:28.706545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:16:30.342922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.699170+0000","last_change":"2023-10-14T17:00:26.698460+0000","last_active":"2023-10-14T17:00:27.699170+0000","last_peered":"2023-10-14T17:00:27.699170+0000","last_clean":"2023-10-14T17:00:27.699170+0000","last_became_active":"2023-10-14T17:00:26.697829+0000","last_became_peered":"2023-10-14T17:00:26.697829+0000","last_unstale":"2023-10-14T17:00:27.699170+0000","last_undegraded":"2023-10-14T17:00:27.699170+0000","last_fullsized":"2023-10-14T17:00:27.699170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:00.973366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711929+0000","last_change":"2023-10-14T17:00:27.710864+0000","last_active":"2023-10-14T17:00:28.711929+0000","last_peered":"2023-10-14T17:00:28.711929+0000","last_clean":"2023-10-14T17:00:28.711929+0000","last_became_active":"2023-10-14T17:00:27.708866+0000","last_became_peered":"2023-10-14T17:00:27.708866+0000","last_unstale":"2023-10-14T17:00:28.711929+0000","last_undegraded":"2023-10-14T17:00:28.711929+0000","last_fullsized":"2023-10-14T17:00:28.711929+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:32.958910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706965+0000","last_change":"2023-10-14T17:00:26.688428+0000","last_active":"2023-10-14T17:00:28.706965+0000","last_peered":"2023-10-14T17:00:28.706965+0000","last_clean":"2023-10-14T17:00:28.706965+0000","last_became_active":"2023-10-14T17:00:26.686124+0000","last_became_peered":"2023-10-14T17:00:26.686124+0000","last_unstale":"2023-10-14T17:00:28.706965+0000","last_undegraded":"2023-10-14T17:00:28.706965+0000","last_fullsized":"2023-10-14T17:00:28.706965+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:31.467942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712922+0000","last_change":"2023-10-14T17:00:27.706650+0000","last_active":"2023-10-14T17:00:27.712922+0000","last_peered":"2023-10-14T17:00:27.712922+0000","last_clean":"2023-10-14T17:00:27.712922+0000","last_became_active":"2023-10-14T17:00:27.705419+0000","last_became_peered":"2023-10-14T17:00:27.705419+0000","last_unstale":"2023-10-14T17:00:27.712922+0000","last_undegraded":"2023-10-14T17:00:27.712922+0000","last_fullsized":"2023-10-14T17:00:27.712922+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:37:37.242371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712347+0000","last_change":"2023-10-14T17:00:26.700612+0000","last_active":"2023-10-14T17:00:28.712347+0000","last_peered":"2023-10-14T17:00:28.712347+0000","last_clean":"2023-10-14T17:00:28.712347+0000","last_became_active":"2023-10-14T17:00:26.699664+0000","last_became_peered":"2023-10-14T17:00:26.699664+0000","last_unstale":"2023-10-14T17:00:28.712347+0000","last_undegraded":"2023-10-14T17:00:28.712347+0000","last_fullsized":"2023-10-14T17:00:28.712347+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.721211+0000","last_change":"2023-10-14T17:00:27.711496+0000","last_active":"2023-10-14T17:00:27.721211+0000","last_peered":"2023-10-14T17:00:27.721211+0000","last_clean":"2023-10-14T17:00:27.721211+0000","last_became_active":"2023-10-14T17:00:27.711174+0000","last_became_peered":"2023-10-14T17:00:27.711174+0000","last_unstale":"2023-10-14T17:00:27.721211+0000","last_undegraded":"2023-10-14T17:00:27.721211+0000","last_fullsized":"2023-10-14T17:00:27.721211+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:55:18.587372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.702641+0000","last_change":"2023-10-14T17:00:26.701564+0000","last_active":"2023-10-14T17:00:27.702641+0000","last_peered":"2023-10-14T17:00:27.702641+0000","last_clean":"2023-10-14T17:00:27.702641+0000","last_became_active":"2023-10-14T17:00:26.701274+0000","last_became_peered":"2023-10-14T17:00:26.701274+0000","last_unstale":"2023-10-14T17:00:27.702641+0000","last_undegraded":"2023-10-14T17:00:27.702641+0000","last_fullsized":"2023-10-14T17:00:27.702641+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:05.832205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711478+0000","last_change":"2023-10-14T17:00:27.707319+0000","last_active":"2023-10-14T17:00:28.711478+0000","last_peered":"2023-10-14T17:00:28.711478+0000","last_clean":"2023-10-14T17:00:28.711478+0000","last_became_active":"2023-10-14T17:00:27.707007+0000","last_became_peered":"2023-10-14T17:00:27.707007+0000","last_unstale":"2023-10-14T17:00:28.711478+0000","last_undegraded":"2023-10-14T17:00:28.711478+0000","last_fullsized":"2023-10-14T17:00:28.711478+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707364+0000","last_change":"2023-10-14T17:00:26.709728+0000","last_active":"2023-10-14T17:00:28.707364+0000","last_peered":"2023-10-14T17:00:28.707364+0000","last_clean":"2023-10-14T17:00:28.707364+0000","last_became_active":"2023-10-14T17:00:26.709344+0000","last_became_peered":"2023-10-14T17:00:26.709344+0000","last_unstale":"2023-10-14T17:00:28.707364+0000","last_undegraded":"2023-10-14T17:00:28.707364+0000","last_fullsized":"2023-10-14T17:00:28.707364+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:04.405861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.699124+0000","last_change":"2023-10-14T17:00:27.698357+0000","last_active":"2023-10-14T17:00:27.699124+0000","last_peered":"2023-10-14T17:00:27.699124+0000","last_clean":"2023-10-14T17:00:27.699124+0000","last_became_active":"2023-10-14T17:00:27.697981+0000","last_became_peered":"2023-10-14T17:00:27.697981+0000","last_unstale":"2023-10-14T17:00:27.699124+0000","last_undegraded":"2023-10-14T17:00:27.699124+0000","last_fullsized":"2023-10-14T17:00:27.699124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:27.066569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.683342+0000","last_change":"2023-10-14T17:00:26.698103+0000","last_active":"2023-10-14T17:00:27.683342+0000","last_peered":"2023-10-14T17:00:27.683342+0000","last_clean":"2023-10-14T17:00:27.683342+0000","last_became_active":"2023-10-14T17:00:26.697804+0000","last_became_peered":"2023-10-14T17:00:26.697804+0000","last_unstale":"2023-10-14T17:00:27.683342+0000","last_undegraded":"2023-10-14T17:00:27.683342+0000","last_fullsized":"2023-10-14T17:00:27.683342+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:34.312839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.717604+0000","last_change":"2023-10-14T17:00:27.715742+0000","last_active":"2023-10-14T17:00:27.717604+0000","last_peered":"2023-10-14T17:00:27.717604+0000","last_clean":"2023-10-14T17:00:27.717604+0000","last_became_active":"2023-10-14T17:00:27.713626+0000","last_became_peered":"2023-10-14T17:00:27.713626+0000","last_unstale":"2023-10-14T17:00:27.717604+0000","last_undegraded":"2023-10-14T17:00:27.717604+0000","last_fullsized":"2023-10-14T17:00:27.717604+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:10.610786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.691819+0000","last_change":"2023-10-14T17:00:26.709571+0000","last_active":"2023-10-14T17:00:27.691819+0000","last_peered":"2023-10-14T17:00:27.691819+0000","last_clean":"2023-10-14T17:00:27.691819+0000","last_became_active":"2023-10-14T17:00:26.709247+0000","last_became_peered":"2023-10-14T17:00:26.709247+0000","last_unstale":"2023-10-14T17:00:27.691819+0000","last_undegraded":"2023-10-14T17:00:27.691819+0000","last_fullsized":"2023-10-14T17:00:27.691819+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:04:55.155135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713546+0000","last_change":"2023-10-14T17:00:27.714781+0000","last_active":"2023-10-14T17:00:28.713546+0000","last_peered":"2023-10-14T17:00:28.713546+0000","last_clean":"2023-10-14T17:00:28.713546+0000","last_became_active":"2023-10-14T17:00:27.714100+0000","last_became_peered":"2023-10-14T17:00:27.714100+0000","last_unstale":"2023-10-14T17:00:28.713546+0000","last_undegraded":"2023-10-14T17:00:28.713546+0000","last_fullsized":"2023-10-14T17:00:28.713546+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:56:41.032208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.689194+0000","last_change":"2023-10-14T17:00:26.701230+0000","last_active":"2023-10-14T17:00:27.689194+0000","last_peered":"2023-10-14T17:00:27.689194+0000","last_clean":"2023-10-14T17:00:27.689194+0000","last_became_active":"2023-10-14T17:00:26.700875+0000","last_became_peered":"2023-10-14T17:00:26.700875+0000","last_unstale":"2023-10-14T17:00:27.689194+0000","last_undegraded":"2023-10-14T17:00:27.689194+0000","last_fullsized":"2023-10-14T17:00:27.689194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:13:55.243575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719965+0000","last_change":"2023-10-14T17:00:27.713305+0000","last_active":"2023-10-14T17:00:27.719965+0000","last_peered":"2023-10-14T17:00:27.719965+0000","last_clean":"2023-10-14T17:00:27.719965+0000","last_became_active":"2023-10-14T17:00:27.713039+0000","last_became_peered":"2023-10-14T17:00:27.713039+0000","last_unstale":"2023-10-14T17:00:27.719965+0000","last_undegraded":"2023-10-14T17:00:27.719965+0000","last_fullsized":"2023-10-14T17:00:27.719965+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:24.963274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.696317+0000","last_change":"2023-10-14T17:00:26.698598+0000","last_active":"2023-10-14T17:00:27.696317+0000","last_peered":"2023-10-14T17:00:27.696317+0000","last_clean":"2023-10-14T17:00:27.696317+0000","last_became_active":"2023-10-14T17:00:26.698199+0000","last_became_peered":"2023-10-14T17:00:26.698199+0000","last_unstale":"2023-10-14T17:00:27.696317+0000","last_undegraded":"2023-10-14T17:00:27.696317+0000","last_fullsized":"2023-10-14T17:00:27.696317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:24:40.243390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706721+0000","last_change":"2023-10-14T17:00:27.718278+0000","last_active":"2023-10-14T17:00:28.706721+0000","last_peered":"2023-10-14T17:00:28.706721+0000","last_clean":"2023-10-14T17:00:28.706721+0000","last_became_active":"2023-10-14T17:00:27.717956+0000","last_became_peered":"2023-10-14T17:00:27.717956+0000","last_unstale":"2023-10-14T17:00:28.706721+0000","last_undegraded":"2023-10-14T17:00:28.706721+0000","last_fullsized":"2023-10-14T17:00:28.706721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:26.430276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.685650+0000","last_change":"2023-10-14T17:00:26.705431+0000","last_active":"2023-10-14T17:00:27.685650+0000","last_peered":"2023-10-14T17:00:27.685650+0000","last_clean":"2023-10-14T17:00:27.685650+0000","last_became_active":"2023-10-14T17:00:26.704776+0000","last_became_peered":"2023-10-14T17:00:26.704776+0000","last_unstale":"2023-10-14T17:00:27.685650+0000","last_undegraded":"2023-10-14T17:00:27.685650+0000","last_fullsized":"2023-10-14T17:00:27.685650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:05.419531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"128'227","reported_seq":666,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713909+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:00:28.713909+0000","last_peered":"2023-10-14T17:00:28.713909+0000","last_clean":"2023-10-14T17:00:28.713909+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:00:28.713909+0000","last_undegraded":"2023-10-14T17:00:28.713909+0000","last_fullsized":"2023-10-14T17:00:28.713909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575088,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37628,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334212,"statfs":{"total":96636764160,"available":96598233088,"internally_reserved":0,"allocated":2338816,"data_stored":1433035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1519999999999999,"15min":1.1699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.7110000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.1339999999999999,"15min":1.194},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.38900000000000001},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":2.117},"last":0.745}]},{"osd":1,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1859999999999999,"15min":1.222},"min":{"1min":0.74299999999999999,"5min":0.71399999999999997,"15min":0.56299999999999994},"max":{"1min":1.72,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.224,"15min":1.254},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57799999999999996},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0259999999999998},"last":1.1599999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.181,"15min":1.2150000000000001},"min":{"1min":0.70799999999999996,"5min":0.623,"15min":0.47099999999999997},"max":{"1min":1.4419999999999999,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.577},{"interface":"front","average":{"1min":1.151,"5min":1.266,"15min":1.294},"min":{"1min":0.83199999999999996,"5min":0.78900000000000003,"15min":0.72599999999999998},"max":{"1min":1.6080000000000001,"5min":1.893,"15min":2.1589999999999998},"last":1.4430000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3149999999999999,"15min":1.357},"min":{"1min":0.85899999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.657,"5min":2.056,"15min":2.4159999999999999},"last":1.018},{"interface":"front","average":{"1min":1.417,"5min":1.4330000000000001,"15min":1.446},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.72199999999999998},"max":{"1min":1.7949999999999999,"5min":2.379,"15min":2.379},"last":1.343}]},{"osd":4,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3,"15min":1.28},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.1819999999999999},"last":1.286},{"interface":"front","average":{"1min":1.226,"5min":1.3400000000000001,"15min":1.327},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.59799999999999998},"max":{"1min":1.748,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.73}]},{"osd":5,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.405,"15min":1.391},"min":{"1min":0.55200000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.877,"5min":2.278,"15min":2.278},"last":1.546},{"interface":"front","average":{"1min":1.383,"5min":1.444,"15min":1.4590000000000001},"min":{"1min":0.71199999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.3919999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5389999999999999,"15min":1.544},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.2959999999999998,"5min":2.492,"15min":2.492},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.512,"15min":1.48},"min":{"1min":0.80200000000000005,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":1.984,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.8320000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575087,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1740800,"data_stored":842121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.123,"15min":1.1439999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.524,"5min":1.7330000000000001,"15min":2.137},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.4770000000000001,"5min":1.8100000000000001,"15min":2.0760000000000001},"last":1.0349999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1479999999999999,"15min":1.1759999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.41299999999999998},"max":{"1min":1.5580000000000001,"5min":1.631,"15min":2.802},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.236,"15min":1.276},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.438},"max":{"1min":1.6899999999999999,"5min":1.9810000000000001,"15min":2.512},"last":1.181}]},{"osd":2,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3100000000000001,"15min":1.3420000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.667,"5min":1.9970000000000001,"15min":3.3290000000000002},"last":1.361},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3140000000000001,"15min":1.3819999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.46000000000000002},"max":{"1min":1.855,"5min":1.855,"15min":3.286},"last":1.649}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.302,"15min":1.361},"min":{"1min":0.68200000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8260000000000001,"5min":1.9890000000000001,"15min":2.823},"last":1.478},{"interface":"front","average":{"1min":1.349,"5min":1.3500000000000001,"15min":1.403},"min":{"1min":0.84899999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.589,"5min":1.774,"15min":4.2610000000000001},"last":1.5069999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":0.60199999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.774,"5min":1.9299999999999999,"15min":2.9460000000000002},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.4119999999999999,"15min":1.415},"min":{"1min":0.70999999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":2.5750000000000002},"last":1.605}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.335,"15min":1.3720000000000001},"min":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7829999999999999,"5min":1.891,"15min":3.4060000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.52,"15min":1.544},"min":{"1min":0.747,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.9219999999999999,"5min":2.2559999999999998,"15min":3.0489999999999999},"last":1.5760000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.536,"15min":1.573},"min":{"1min":0.76900000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8009999999999999,"5min":2.1110000000000002,"15min":2.8570000000000002},"last":1.714},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4930000000000001,"15min":1.5269999999999999},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.623},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.7000000000000002},"last":1.542}]}]},{"osd":5,"up_from":13,"seq":55834575088,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1732608,"data_stored":836283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.0780000000000001,"15min":1.109},"min":{"1min":0.69399999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5569999999999999,"5min":1.5620000000000001,"15min":1.986},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1559999999999999,"15min":1.139},"min":{"1min":0.747,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":2.0430000000000001},"last":1.1990000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.224,"15min":1.1870000000000001},"min":{"1min":0.874,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.7290000000000001,"5min":1.744,"15min":2.1200000000000001},"last":1.661},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.175,"15min":1.1990000000000001},"min":{"1min":0.80900000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.635,"5min":1.7629999999999999,"15min":1.905},"last":1.8999999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.115,"15min":1.1719999999999999},"min":{"1min":0.68100000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":2.375},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.306,"15min":1.319},"min":{"1min":0.81299999999999994,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.788,"5min":1.788,"15min":2.3100000000000001},"last":1.8200000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3540000000000001,"15min":1.381},"min":{"1min":0.92600000000000005,"5min":0.77200000000000002,"15min":0.745},"max":{"1min":1.962,"5min":1.962,"15min":2.4180000000000001},"last":2.1139999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.329,"15min":1.3899999999999999},"min":{"1min":0.88300000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.819,"5min":1.819,"15min":2.1429999999999998},"last":1.9650000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.319,"15min":1.292},"min":{"1min":1.0569999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.665,"5min":1.744,"15min":2.4889999999999999},"last":2.1709999999999998},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3700000000000001,"15min":1.3420000000000001},"min":{"1min":1.002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8879999999999999,"5min":1.9119999999999999,"15min":2.5169999999999999},"last":2.2240000000000002}]},{"osd":6,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2749999999999999,"15min":1.3460000000000001},"min":{"1min":0.82999999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.8999999999999999,"5min":1.9650000000000001,"15min":2.2869999999999999},"last":2.403},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5069999999999999,"15min":1.5249999999999999},"min":{"1min":0.88400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":1.994,"5min":1.994,"15min":2.613},"last":2.0459999999999998}]},{"osd":7,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.472,"15min":1.5009999999999999},"min":{"1min":1.1439999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.605},"last":2.29},{"interface":"front","average":{"1min":1.446,"5min":1.4390000000000001,"15min":1.4910000000000001},"min":{"1min":1.0209999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.3980000000000001},"last":2.48}]}]},{"osd":4,"up_from":13,"seq":55834575088,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59240,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312600,"statfs":{"total":96636764160,"available":96576102400,"internally_reserved":0,"allocated":1724416,"data_stored":835679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1519999999999999,"15min":1.1739999999999999},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":1.7270000000000001,"5min":1.9319999999999999,"15min":1.982},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.177,"5min":1.2649999999999999,"15min":1.2330000000000001},"min":{"1min":0.73799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6579999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.80400000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.323,"15min":1.2869999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.48199999999999998},"max":{"1min":1.607,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.226,"15min":1.2490000000000001},"min":{"1min":0.71899999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.639,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":0.95299999999999996}]},{"osd":2,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.262,"15min":1.252},"min":{"1min":0.68000000000000005,"5min":0.67600000000000005,"15min":0.64700000000000002},"max":{"1min":1.5940000000000001,"5min":5.633,"15min":5.633},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3779999999999999,"15min":1.375},"min":{"1min":0.93100000000000005,"5min":0.81100000000000005,"15min":0.71999999999999997},"max":{"1min":1.7210000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.1830000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.304,"15min":1.3149999999999999},"min":{"1min":0.79500000000000004,"5min":0.77100000000000002,"15min":0.71699999999999997},"max":{"1min":1.8080000000000001,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.288},{"interface":"front","average":{"1min":1.369,"5min":1.4119999999999999,"15min":1.409},"min":{"1min":0.85099999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.6200000000000001,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.1220000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.401,"15min":1.365},"min":{"1min":1.012,"5min":0.64500000000000002,"15min":0.52900000000000003},"max":{"1min":1.724,"5min":2.931,"15min":2.931},"last":1.534},{"interface":"front","average":{"1min":1.288,"5min":1.347,"15min":1.347},"min":{"1min":0.86599999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.77,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.0920000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.431,"15min":1.395},"min":{"1min":0.997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6879999999999999,"5min":2.323,"15min":2.323},"last":1.768},{"interface":"front","average":{"1min":1.492,"5min":1.544,"15min":1.53},"min":{"1min":1.093,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8109999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.637}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.51,"15min":1.4890000000000001},"min":{"1min":0.92700000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.8420000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5589999999999999,"15min":1.554},"min":{"1min":1.1379999999999999,"5min":0.82899999999999996,"15min":0.79000000000000004},"max":{"1min":1.9319999999999999,"5min":2.0209999999999999,"15min":2.3130000000000002},"last":1.6839999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575088,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41220,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330620,"statfs":{"total":96636764160,"available":96594554880,"internally_reserved":0,"allocated":1753088,"data_stored":843733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90000000000000002,"15min":0.92000000000000004},"min":{"1min":0.51400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.264,"5min":1.621,"15min":1.621},"last":1.407},{"interface":"front","average":{"1min":1.036,"5min":1.0600000000000001,"15min":1.1240000000000001},"min":{"1min":0.59899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.8799999999999999},"last":1.2290000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.876,"15min":0.90800000000000003},"min":{"1min":0.56000000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.635,"5min":1.7609999999999999,"15min":2.1720000000000002},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0900000000000001,"15min":1.101},"min":{"1min":0.56799999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.419,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.018}]},{"osd":2,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2689999999999999,"15min":1.347},"min":{"1min":0.77200000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.762,"5min":2.024,"15min":2.0609999999999999},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.242,"15min":1.27},"min":{"1min":0.71599999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.883,"5min":1.883,"15min":2.0960000000000001},"last":0.97099999999999997}]},{"osd":4,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.321,"15min":1.353},"min":{"1min":0.97299999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.913,"5min":1.913,"15min":3.4350000000000001},"last":1.276},{"interface":"front","average":{"1min":1.397,"5min":1.389,"15min":1.407},"min":{"1min":0.94099999999999995,"5min":0.745,"15min":0.745},"max":{"1min":1.845,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.1859999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.339,"15min":1.387},"min":{"1min":0.98699999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.702,"5min":1.895,"15min":2.516},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.387,"15min":1.4359999999999999},"min":{"1min":1.0549999999999999,"5min":0.82699999999999996,"15min":0.80600000000000005},"max":{"1min":1.849,"5min":1.889,"15min":3.7410000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4690000000000001,"15min":1.472},"min":{"1min":0.90600000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.1110000000000002,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.556,"15min":1.548},"min":{"1min":1.252,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.923,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.653}]},{"osd":7,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4610000000000001,"15min":1.498},"min":{"1min":0.90900000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.871},"last":1.585},{"interface":"front","average":{"1min":1.496,"5min":1.514,"15min":1.546},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.911,"5min":2.2330000000000001,"15min":2.907},"last":1.5600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575088,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":1724416,"data_stored":835679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.90000000000000002},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.429,"5min":1.706,"15min":1.706},"last":1.323},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":1.0369999999999999},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6870000000000001,"5min":1.726,"15min":1.8080000000000001},"last":1.3999999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90100000000000002,"15min":0.96599999999999997},"min":{"1min":0.42799999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.2889999999999999,"5min":1.544,"15min":2.0419999999999998},"last":1.532},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.121,"15min":1.1659999999999999},"min":{"1min":0.45200000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.492,"5min":1.74,"15min":2.069},"last":1.909}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1180000000000001,"15min":1.208},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.46500000000000002},"max":{"1min":1.7010000000000001,"5min":1.833,"15min":2.1349999999999998},"last":2.0339999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.2070000000000001,"15min":1.2889999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.46899999999999997},"max":{"1min":1.6559999999999999,"5min":1.7729999999999999,"15min":2.0459999999999998},"last":1.7010000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.403},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7070000000000001,"5min":1.913,"15min":2.3159999999999998},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.2949999999999999,"15min":1.357},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0750000000000002},"last":2.4089999999999998}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2969999999999999,"15min":1.3759999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59199999999999997},"max":{"1min":1.5980000000000001,"5min":2.173,"15min":2.173},"last":2.7890000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4419999999999999,"15min":1.512},"min":{"1min":0.61199999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.827,"5min":1.889,"15min":2.4460000000000002},"last":2.5939999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.357,"15min":1.429},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.966,"5min":1.966,"15min":2.0289999999999999},"last":2.9079999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.4330000000000001,"15min":1.5069999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.7230000000000001,"5min":2.0720000000000001,"15min":2.105},"last":2.9620000000000002}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4199999999999999,"15min":1.51},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.109},"last":2.6579999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4870000000000001,"15min":1.5700000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.7110000000000001,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":2.5369999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59240,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312600,"statfs":{"total":96636764160,"available":96576102400,"internally_reserved":0,"allocated":1732608,"data_stored":835803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.90000000000000002},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2909999999999999,"5min":2.113,"15min":2.113},"last":0.441},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.93999999999999995,"15min":1.052},"min":{"1min":0.45100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":2.0819999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.91800000000000004,"15min":0.94699999999999995},"min":{"1min":0.52000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.224,"5min":1.7509999999999999,"15min":1.758},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.0880000000000001,"15min":1.1559999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.4570000000000001,"5min":2.008,"15min":2.0150000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.738,"5min":1.984,"15min":2.198},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0820000000000001,"15min":1.1759999999999999},"min":{"1min":0.496,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4239999999999999,"5min":1.974,"15min":2.004},"last":0.495}]},{"osd":4,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3300000000000001,"15min":1.399},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9179999999999999,"5min":2.052,"15min":2.3050000000000002},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.29,"15min":1.3089999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.53900000000000003},"max":{"1min":1.6970000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.90300000000000002}]},{"osd":5,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3300000000000001,"15min":1.361},"min":{"1min":0.85499999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.8759999999999999,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":0.754},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3939999999999999,"15min":1.45},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.76200000000000001},"max":{"1min":1.867,"5min":2.1480000000000001,"15min":2.302},"last":0.79100000000000004}]},{"osd":6,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.419,"15min":1.4870000000000001},"min":{"1min":0.82999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.121,"5min":2.121,"15min":2.4169999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4319999999999999,"15min":1.4670000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.7829999999999999,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":0.84799999999999998}]},{"osd":7,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4730000000000001,"15min":1.548},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.786,"5min":2.2410000000000001,"15min":2.411},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.4730000000000001,"15min":1.5660000000000001},"min":{"1min":0.88500000000000001,"5min":0.81899999999999995,"15min":0.61599999999999999},"max":{"1min":2.0529999999999999,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575087,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2338816,"data_stored":1427771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.90700000000000003,"15min":0.89700000000000002},"min":{"1min":0.54100000000000004,"5min":0.309,"15min":0.309},"max":{"1min":1.635,"5min":3.355,"15min":3.355},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.97699999999999998,"15min":1.075},"min":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3859999999999999,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.054,"15min":1.0760000000000001},"min":{"1min":0.46800000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4279999999999999,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":1.292},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.091,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.607,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":1.032}]},{"osd":3,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.268},"min":{"1min":0.50800000000000001,"5min":0.502,"15min":0.502},"max":{"1min":1.6839999999999999,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.236,"15min":1.3360000000000001},"min":{"1min":0.57599999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.7350000000000001,"5min":2.581,"15min":2.593},"last":1.355}]},{"osd":4,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.268,"15min":1.278},"min":{"1min":0.75600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.833,"5min":5.335,"15min":5.335},"last":1.117},{"interface":"front","average":{"1min":1.286,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.4319999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.254,"15min":1.3069999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.698,"5min":2.931,"15min":2.931},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3859999999999999,"15min":1.4850000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8160000000000001,"5min":3.306,"15min":3.306},"last":1.6779999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.472,"15min":1.5649999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.9039999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3500000000000001,"15min":1.393},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.895,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.2150000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4930000000000001,"15min":1.538},"min":{"1min":0.79500000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.8919999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.4590000000000001,"15min":1.5109999999999999},"min":{"1min":0.93000000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7589999999999999,"5min":3.698,"15min":3.698},"last":1.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:00:32.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:00:32.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.464+0000 7f3124844640 1 Processor -- start 2023-10-14T17:00:32.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.464+0000 7f3124844640 1 -- start start 2023-10-14T17:00:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f3124844640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f312015aaa0 0x7f312015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f3124844640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f312015ae80 0x7f31200cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f3124844640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312015ed60 con 0x7f31200ccaa0 2023-10-14T17:00:32.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f3124844640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f312006b810 con 0x7f312015aaa0 2023-10-14T17:00:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f3124844640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31200707e0 con 0x7f312015ae80 2023-10-14T17:00:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f311e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f312015aaa0 0x7f312015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.466+0000 7f311ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f31200ccaa0 0x7f3120070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35548/0 (socket says 172.21.15.29:35548) 2023-10-14T17:00:32.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311ed9d640 1 -- 172.21.15.29:0/1200612519 learned_addr learned my addr 172.21.15.29:0/1200612519 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:32.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311dd9b640 1 --2- 172.21.15.29:0/1200612519 >> v2:172.21.15.186:3300/0 conn(0x7f312015ae80 0x7f31200cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311e59c640 1 -- 172.21.15.29:0/1200612519 >> v2:172.21.15.186:3300/0 conn(0x7f312015ae80 msgr2=0x7f31200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:32.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311e59c640 1 --2- 172.21.15.29:0/1200612519 >> v2:172.21.15.186:3300/0 conn(0x7f312015ae80 0x7f31200cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311e59c640 1 -- 172.21.15.29:0/1200612519 >> v1:172.21.15.29:6789/0 conn(0x7f31200ccaa0 legacy=0x7f3120070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311e59c640 1 -- 172.21.15.29:0/1200612519 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f312015ef90 con 0x7f312015aaa0 2023-10-14T17:00:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.467+0000 7f311d59a640 1 -- 172.21.15.29:0/1200612519 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470883235 0 0) 0x7f312015ed60 con 0x7f31200ccaa0 2023-10-14T17:00:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.468+0000 7f311e59c640 1 --2- 172.21.15.29:0/1200612519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f312015e820 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f31200c59c0 tx=0x7f311002bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea16ca615451a4d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:32.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.468+0000 7f311d59a640 1 -- 172.21.15.29:0/1200612519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f311002cc70 con 0x7f312015aaa0 2023-10-14T17:00:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f311d59a640 1 -- 172.21.15.29:0/1200612519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f311002cdd0 con 0x7f312015aaa0 2023-10-14T17:00:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f311d59a640 1 -- 172.21.15.29:0/1200612519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3110034660 con 0x7f312015aaa0 2023-10-14T17:00:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 -- 172.21.15.29:0/1200612519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 msgr2=0x7f312015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 --2- 172.21.15.29:0/1200612519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f312015e820 secure :-1 s=READY pgs=1590 cs=0 l=1 rev1=1 crypto rx=0x7f31200c59c0 tx=0x7f311002bec0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 -- 172.21.15.29:0/1200612519 shutdown_connections 2023-10-14T17:00:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 --2- 172.21.15.29:0/1200612519 >> v2:172.21.15.186:3300/0 conn(0x7f312015ae80 0x7f31200cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 --2- 172.21.15.29:0/1200612519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f312015e820 unknown :-1 s=CLOSED pgs=1590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 -- 172.21.15.29:0/1200612519 >> 172.21.15.29:0/1200612519 conn(0x7f31200cb410 msgr2=0x7f31200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 -- 172.21.15.29:0/1200612519 shutdown_connections 2023-10-14T17:00:32.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.469+0000 7f3124844640 1 -- 172.21.15.29:0/1200612519 wait complete. 2023-10-14T17:00:32.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.471+0000 7f3124844640 1 Processor -- start 2023-10-14T17:00:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.472+0000 7f3124844640 1 -- start start 2023-10-14T17:00:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.472+0000 7f3124844640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 0x7f312015db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.472+0000 7f3124844640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.472+0000 7f3124844640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312015ae80 0x7f31200c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f3124844640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312015c040 con 0x7f312015aaa0 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f3124844640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312015e040 con 0x7f312015ae80 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f3124844640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312015b3c0 con 0x7f31200ccaa0 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58112/0 (socket says 172.21.15.29:58112) 2023-10-14T17:00:32.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 -- 172.21.15.29:0/3734849474 learned_addr learned my addr 172.21.15.29:0/3734849474 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311e59c640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 0x7f312015db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 -- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 msgr2=0x7f312015db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 0x7f312015db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311ed9d640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312015ae80 0x7f31200c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 -- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312015ae80 msgr2=0x7f31200c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312015ae80 0x7f31200c1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311dd9b640 1 -- 172.21.15.29:0/3734849474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3110009a70 con 0x7f312015aaa0 2023-10-14T17:00:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.473+0000 7f311e59c640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 0x7f312015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f311dd9b640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f311400d5c0 tx=0x7f311400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:32.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31140125f0 con 0x7f312015aaa0 2023-10-14T17:00:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114012750 con 0x7f312015aaa0 2023-10-14T17:00:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3114012b00 con 0x7f312015aaa0 2023-10-14T17:00:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f312015b6a0 con 0x7f312015aaa0 2023-10-14T17:00:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.474+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f312015bc60 con 0x7f312015aaa0 2023-10-14T17:00:32.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.475+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312004e160 con 0x7f312015aaa0 2023-10-14T17:00:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.476+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3114012c60 con 0x7f312015aaa0 2023-10-14T17:00:32.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.477+0000 7f310effd640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 0x7f310006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.477+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3114089d40 con 0x7f312015aaa0 2023-10-14T17:00:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.478+0000 7f311e59c640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 0x7f310006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.478+0000 7f311e59c640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 0x7f310006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3110036000 tx=0x7f3110030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:32.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.482+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3114054c80 con 0x7f312015aaa0 2023-10-14T17:00:32.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.676+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 --> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f312015f550 con 0x7f310006af90 2023-10-14T17:00:32.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.699+0000 7f310effd640 1 -- 172.21.15.29:0/3734849474 <== mgr.8298 v2:172.21.15.29:6832/3738876524 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286875 (secure 0 0 0) 0x7f31100345c0 con 0x7f310006af90 2023-10-14T17:00:32.700 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:32.705 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:00:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.705+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 msgr2=0x7f310006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.705+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 0x7f310006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3110036000 tx=0x7f3110030000 comp rx=0 tx=0).stop 2023-10-14T17:00:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.705+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 msgr2=0x7f31201f3c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.705+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 secure :-1 s=READY pgs=1591 cs=0 l=1 rev1=1 crypto rx=0x7f311400d5c0 tx=0x7f311400da80 comp rx=0 tx=0).stop 2023-10-14T17:00:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 shutdown_connections 2023-10-14T17:00:32.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f310006af90 0x7f310006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f312015ae80 0x7f31200c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f312015aaa0 0x7f31201f3c90 unknown :-1 s=CLOSED pgs=1591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 --2- 172.21.15.29:0/3734849474 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f31200ccaa0 0x7f312015db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 >> 172.21.15.29:0/3734849474 conn(0x7f31200cb410 msgr2=0x7f3120070e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 shutdown_connections 2023-10-14T17:00:32.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:32.706+0000 7f3124844640 1 -- 172.21.15.29:0/3734849474 wait complete. 2023-10-14T17:00:32.726 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T17:00:30.799910+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":403,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":265,"ondisk_log_size":265,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":359952,"kb_used_data":14732,"kb_used_omap":12,"kb_used_meta":345075,"kb_avail":754614768,"statfs":{"total":773094113280,"available":772725522432,"internally_reserved":0,"allocated":15085568,"data_stored":7890104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":353357368},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.145870"},"pg_stats":[{"pgid":"17.3f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719339+0000","last_change":"2023-10-14T17:00:27.711292+0000","last_active":"2023-10-14T17:00:27.719339+0000","last_peered":"2023-10-14T17:00:27.719339+0000","last_clean":"2023-10-14T17:00:27.719339+0000","last_became_active":"2023-10-14T17:00:27.710978+0000","last_became_peered":"2023-10-14T17:00:27.710978+0000","last_unstale":"2023-10-14T17:00:27.719339+0000","last_undegraded":"2023-10-14T17:00:27.719339+0000","last_fullsized":"2023-10-14T17:00:27.719339+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:03:16.305171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712365+0000","last_change":"2023-10-14T17:00:27.709657+0000","last_active":"2023-10-14T17:00:27.712365+0000","last_peered":"2023-10-14T17:00:27.712365+0000","last_clean":"2023-10-14T17:00:27.712365+0000","last_became_active":"2023-10-14T17:00:27.709342+0000","last_became_peered":"2023-10-14T17:00:27.709342+0000","last_unstale":"2023-10-14T17:00:27.712365+0000","last_undegraded":"2023-10-14T17:00:27.712365+0000","last_fullsized":"2023-10-14T17:00:27.712365+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:37.963501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.3d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712252+0000","last_change":"2023-10-14T17:00:27.711863+0000","last_active":"2023-10-14T17:00:28.712252+0000","last_peered":"2023-10-14T17:00:28.712252+0000","last_clean":"2023-10-14T17:00:28.712252+0000","last_became_active":"2023-10-14T17:00:27.711220+0000","last_became_peered":"2023-10-14T17:00:27.711220+0000","last_unstale":"2023-10-14T17:00:28.712252+0000","last_undegraded":"2023-10-14T17:00:28.712252+0000","last_fullsized":"2023-10-14T17:00:28.712252+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:32:32.026173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.3c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.710200+0000","last_change":"2023-10-14T17:00:27.705462+0000","last_active":"2023-10-14T17:00:27.710200+0000","last_peered":"2023-10-14T17:00:27.710200+0000","last_clean":"2023-10-14T17:00:27.710200+0000","last_became_active":"2023-10-14T17:00:27.705146+0000","last_became_peered":"2023-10-14T17:00:27.705146+0000","last_unstale":"2023-10-14T17:00:27.710200+0000","last_undegraded":"2023-10-14T17:00:27.710200+0000","last_fullsized":"2023-10-14T17:00:27.710200+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:51.688094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.3b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712707+0000","last_change":"2023-10-14T17:00:27.710529+0000","last_active":"2023-10-14T17:00:27.712707+0000","last_peered":"2023-10-14T17:00:27.712707+0000","last_clean":"2023-10-14T17:00:27.712707+0000","last_became_active":"2023-10-14T17:00:27.708905+0000","last_became_peered":"2023-10-14T17:00:27.708905+0000","last_unstale":"2023-10-14T17:00:27.712707+0000","last_undegraded":"2023-10-14T17:00:27.712707+0000","last_fullsized":"2023-10-14T17:00:27.712707+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:37.885263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.3a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724267+0000","last_change":"2023-10-14T17:00:27.712586+0000","last_active":"2023-10-14T17:00:27.724267+0000","last_peered":"2023-10-14T17:00:27.724267+0000","last_clean":"2023-10-14T17:00:27.724267+0000","last_became_active":"2023-10-14T17:00:27.712321+0000","last_became_peered":"2023-10-14T17:00:27.712321+0000","last_unstale":"2023-10-14T17:00:27.724267+0000","last_undegraded":"2023-10-14T17:00:27.724267+0000","last_fullsized":"2023-10-14T17:00:27.724267+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:50.052495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.39","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706315+0000","last_change":"2023-10-14T17:00:27.702020+0000","last_active":"2023-10-14T17:00:27.706315+0000","last_peered":"2023-10-14T17:00:27.706315+0000","last_clean":"2023-10-14T17:00:27.706315+0000","last_became_active":"2023-10-14T17:00:27.701441+0000","last_became_peered":"2023-10-14T17:00:27.701441+0000","last_unstale":"2023-10-14T17:00:27.706315+0000","last_undegraded":"2023-10-14T17:00:27.706315+0000","last_fullsized":"2023-10-14T17:00:27.706315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:27.066569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.38","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712951+0000","last_change":"2023-10-14T17:00:27.713815+0000","last_active":"2023-10-14T17:00:28.712951+0000","last_peered":"2023-10-14T17:00:28.712951+0000","last_clean":"2023-10-14T17:00:28.712951+0000","last_became_active":"2023-10-14T17:00:27.711147+0000","last_became_peered":"2023-10-14T17:00:27.711147+0000","last_unstale":"2023-10-14T17:00:28.712951+0000","last_undegraded":"2023-10-14T17:00:28.712951+0000","last_fullsized":"2023-10-14T17:00:28.712951+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:30.874300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.37","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720638+0000","last_change":"2023-10-14T17:00:27.718112+0000","last_active":"2023-10-14T17:00:27.720638+0000","last_peered":"2023-10-14T17:00:27.720638+0000","last_clean":"2023-10-14T17:00:27.720638+0000","last_became_active":"2023-10-14T17:00:27.715211+0000","last_became_peered":"2023-10-14T17:00:27.715211+0000","last_unstale":"2023-10-14T17:00:27.720638+0000","last_undegraded":"2023-10-14T17:00:27.720638+0000","last_fullsized":"2023-10-14T17:00:27.720638+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:05.215574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.36","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720155+0000","last_change":"2023-10-14T17:00:27.714466+0000","last_active":"2023-10-14T17:00:27.720155+0000","last_peered":"2023-10-14T17:00:27.720155+0000","last_clean":"2023-10-14T17:00:27.720155+0000","last_became_active":"2023-10-14T17:00:27.714185+0000","last_became_peered":"2023-10-14T17:00:27.714185+0000","last_unstale":"2023-10-14T17:00:27.720155+0000","last_undegraded":"2023-10-14T17:00:27.720155+0000","last_fullsized":"2023-10-14T17:00:27.720155+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.35","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.704307+0000","last_change":"2023-10-14T17:00:27.702645+0000","last_active":"2023-10-14T17:00:27.704307+0000","last_peered":"2023-10-14T17:00:27.704307+0000","last_clean":"2023-10-14T17:00:27.704307+0000","last_became_active":"2023-10-14T17:00:27.702340+0000","last_became_peered":"2023-10-14T17:00:27.702340+0000","last_unstale":"2023-10-14T17:00:27.704307+0000","last_undegraded":"2023-10-14T17:00:27.704307+0000","last_fullsized":"2023-10-14T17:00:27.704307+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:23.818618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.34","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.721315+0000","last_change":"2023-10-14T17:00:27.716704+0000","last_active":"2023-10-14T17:00:27.721315+0000","last_peered":"2023-10-14T17:00:27.721315+0000","last_clean":"2023-10-14T17:00:27.721315+0000","last_became_active":"2023-10-14T17:00:27.716431+0000","last_became_peered":"2023-10-14T17:00:27.716431+0000","last_unstale":"2023-10-14T17:00:27.721315+0000","last_undegraded":"2023-10-14T17:00:27.721315+0000","last_fullsized":"2023-10-14T17:00:27.721315+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:20.722396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.33","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711762+0000","last_change":"2023-10-14T17:00:27.712169+0000","last_active":"2023-10-14T17:00:28.711762+0000","last_peered":"2023-10-14T17:00:28.711762+0000","last_clean":"2023-10-14T17:00:28.711762+0000","last_became_active":"2023-10-14T17:00:27.711716+0000","last_became_peered":"2023-10-14T17:00:27.711716+0000","last_unstale":"2023-10-14T17:00:28.711762+0000","last_undegraded":"2023-10-14T17:00:28.711762+0000","last_fullsized":"2023-10-14T17:00:28.711762+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:26.430276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.32","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.710103+0000","last_change":"2023-10-14T17:00:27.714801+0000","last_active":"2023-10-14T17:00:28.710103+0000","last_peered":"2023-10-14T17:00:28.710103+0000","last_clean":"2023-10-14T17:00:28.710103+0000","last_became_active":"2023-10-14T17:00:27.714076+0000","last_became_peered":"2023-10-14T17:00:27.714076+0000","last_unstale":"2023-10-14T17:00:28.710103+0000","last_undegraded":"2023-10-14T17:00:28.710103+0000","last_fullsized":"2023-10-14T17:00:28.710103+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:16:30.342922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.31","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.725268+0000","last_change":"2023-10-14T17:00:27.719950+0000","last_active":"2023-10-14T17:00:27.725268+0000","last_peered":"2023-10-14T17:00:27.725268+0000","last_clean":"2023-10-14T17:00:27.725268+0000","last_became_active":"2023-10-14T17:00:27.719656+0000","last_became_peered":"2023-10-14T17:00:27.719656+0000","last_unstale":"2023-10-14T17:00:27.725268+0000","last_undegraded":"2023-10-14T17:00:27.725268+0000","last_fullsized":"2023-10-14T17:00:27.725268+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:42:06.021014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.30","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709303+0000","last_change":"2023-10-14T17:00:27.707309+0000","last_active":"2023-10-14T17:00:27.709303+0000","last_peered":"2023-10-14T17:00:27.709303+0000","last_clean":"2023-10-14T17:00:27.709303+0000","last_became_active":"2023-10-14T17:00:27.706994+0000","last_became_peered":"2023-10-14T17:00:27.706994+0000","last_unstale":"2023-10-14T17:00:27.709303+0000","last_undegraded":"2023-10-14T17:00:27.709303+0000","last_fullsized":"2023-10-14T17:00:27.709303+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:25.770850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.2f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712254+0000","last_change":"2023-10-14T17:00:27.707603+0000","last_active":"2023-10-14T17:00:27.712254+0000","last_peered":"2023-10-14T17:00:27.712254+0000","last_clean":"2023-10-14T17:00:27.712254+0000","last_became_active":"2023-10-14T17:00:27.707316+0000","last_became_peered":"2023-10-14T17:00:27.707316+0000","last_unstale":"2023-10-14T17:00:27.712254+0000","last_undegraded":"2023-10-14T17:00:27.712254+0000","last_fullsized":"2023-10-14T17:00:27.712254+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:15.286260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2e","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711615+0000","last_change":"2023-10-14T17:00:27.699374+0000","last_active":"2023-10-14T17:00:28.711615+0000","last_peered":"2023-10-14T17:00:28.711615+0000","last_clean":"2023-10-14T17:00:28.711615+0000","last_became_active":"2023-10-14T17:00:27.697480+0000","last_became_peered":"2023-10-14T17:00:27.697480+0000","last_unstale":"2023-10-14T17:00:28.711615+0000","last_undegraded":"2023-10-14T17:00:28.711615+0000","last_fullsized":"2023-10-14T17:00:28.711615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:31.630069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.2d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712569+0000","last_change":"2023-10-14T17:00:27.713718+0000","last_active":"2023-10-14T17:00:28.712569+0000","last_peered":"2023-10-14T17:00:28.712569+0000","last_clean":"2023-10-14T17:00:28.712569+0000","last_became_active":"2023-10-14T17:00:27.713413+0000","last_became_peered":"2023-10-14T17:00:27.713413+0000","last_unstale":"2023-10-14T17:00:28.712569+0000","last_undegraded":"2023-10-14T17:00:28.712569+0000","last_fullsized":"2023-10-14T17:00:28.712569+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:24.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.2c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712603+0000","last_change":"2023-10-14T17:00:27.710293+0000","last_active":"2023-10-14T17:00:27.712603+0000","last_peered":"2023-10-14T17:00:27.712603+0000","last_clean":"2023-10-14T17:00:27.712603+0000","last_became_active":"2023-10-14T17:00:27.709700+0000","last_became_peered":"2023-10-14T17:00:27.709700+0000","last_unstale":"2023-10-14T17:00:27.712603+0000","last_undegraded":"2023-10-14T17:00:27.712603+0000","last_fullsized":"2023-10-14T17:00:27.712603+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:06.433862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720326+0000","last_change":"2023-10-14T17:00:27.719603+0000","last_active":"2023-10-14T17:00:27.720326+0000","last_peered":"2023-10-14T17:00:27.720326+0000","last_clean":"2023-10-14T17:00:27.720326+0000","last_became_active":"2023-10-14T17:00:27.719341+0000","last_became_peered":"2023-10-14T17:00:27.719341+0000","last_unstale":"2023-10-14T17:00:27.720326+0000","last_undegraded":"2023-10-14T17:00:27.720326+0000","last_fullsized":"2023-10-14T17:00:27.720326+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:40.799944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.2a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706938+0000","last_change":"2023-10-14T17:00:27.712322+0000","last_active":"2023-10-14T17:00:28.706938+0000","last_peered":"2023-10-14T17:00:28.706938+0000","last_clean":"2023-10-14T17:00:28.706938+0000","last_became_active":"2023-10-14T17:00:27.712049+0000","last_became_peered":"2023-10-14T17:00:27.712049+0000","last_unstale":"2023-10-14T17:00:28.706938+0000","last_undegraded":"2023-10-14T17:00:28.706938+0000","last_fullsized":"2023-10-14T17:00:28.706938+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:24.309686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.29","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712290+0000","last_change":"2023-10-14T17:00:27.710507+0000","last_active":"2023-10-14T17:00:27.712290+0000","last_peered":"2023-10-14T17:00:27.712290+0000","last_clean":"2023-10-14T17:00:27.712290+0000","last_became_active":"2023-10-14T17:00:27.710225+0000","last_became_peered":"2023-10-14T17:00:27.710225+0000","last_unstale":"2023-10-14T17:00:27.712290+0000","last_undegraded":"2023-10-14T17:00:27.712290+0000","last_fullsized":"2023-10-14T17:00:27.712290+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:29.621105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.28","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706399+0000","last_change":"2023-10-14T17:00:27.720944+0000","last_active":"2023-10-14T17:00:28.706399+0000","last_peered":"2023-10-14T17:00:28.706399+0000","last_clean":"2023-10-14T17:00:28.706399+0000","last_became_active":"2023-10-14T17:00:27.720652+0000","last_became_peered":"2023-10-14T17:00:27.720652+0000","last_unstale":"2023-10-14T17:00:28.706399+0000","last_undegraded":"2023-10-14T17:00:28.706399+0000","last_fullsized":"2023-10-14T17:00:28.706399+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:10.530928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.27","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.702817+0000","last_change":"2023-10-14T17:00:27.701211+0000","last_active":"2023-10-14T17:00:27.702817+0000","last_peered":"2023-10-14T17:00:27.702817+0000","last_clean":"2023-10-14T17:00:27.702817+0000","last_became_active":"2023-10-14T17:00:27.700076+0000","last_became_peered":"2023-10-14T17:00:27.700076+0000","last_unstale":"2023-10-14T17:00:27.702817+0000","last_undegraded":"2023-10-14T17:00:27.702817+0000","last_fullsized":"2023-10-14T17:00:27.702817+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:11.493657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.26","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709721+0000","last_change":"2023-10-14T17:00:27.706133+0000","last_active":"2023-10-14T17:00:27.709721+0000","last_peered":"2023-10-14T17:00:27.709721+0000","last_clean":"2023-10-14T17:00:27.709721+0000","last_became_active":"2023-10-14T17:00:27.704391+0000","last_became_peered":"2023-10-14T17:00:27.704391+0000","last_unstale":"2023-10-14T17:00:27.709721+0000","last_undegraded":"2023-10-14T17:00:27.709721+0000","last_fullsized":"2023-10-14T17:00:27.709721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:23.280133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.25","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724015+0000","last_change":"2023-10-14T17:00:27.715949+0000","last_active":"2023-10-14T17:00:27.724015+0000","last_peered":"2023-10-14T17:00:27.724015+0000","last_clean":"2023-10-14T17:00:27.724015+0000","last_became_active":"2023-10-14T17:00:27.715672+0000","last_became_peered":"2023-10-14T17:00:27.715672+0000","last_unstale":"2023-10-14T17:00:27.724015+0000","last_undegraded":"2023-10-14T17:00:27.724015+0000","last_fullsized":"2023-10-14T17:00:27.724015+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:19:26.034995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.24","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707181+0000","last_change":"2023-10-14T17:00:27.723716+0000","last_active":"2023-10-14T17:00:28.707181+0000","last_peered":"2023-10-14T17:00:28.707181+0000","last_clean":"2023-10-14T17:00:28.707181+0000","last_became_active":"2023-10-14T17:00:27.723386+0000","last_became_peered":"2023-10-14T17:00:27.723386+0000","last_unstale":"2023-10-14T17:00:28.707181+0000","last_undegraded":"2023-10-14T17:00:28.707181+0000","last_fullsized":"2023-10-14T17:00:28.707181+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:16.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.23","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712655+0000","last_change":"2023-10-14T17:00:27.710388+0000","last_active":"2023-10-14T17:00:27.712655+0000","last_peered":"2023-10-14T17:00:27.712655+0000","last_clean":"2023-10-14T17:00:27.712655+0000","last_became_active":"2023-10-14T17:00:27.709675+0000","last_became_peered":"2023-10-14T17:00:27.709675+0000","last_unstale":"2023-10-14T17:00:27.712655+0000","last_undegraded":"2023-10-14T17:00:27.712655+0000","last_fullsized":"2023-10-14T17:00:27.712655+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:03:29.832656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.22","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.722684+0000","last_change":"2023-10-14T17:00:27.721858+0000","last_active":"2023-10-14T17:00:27.722684+0000","last_peered":"2023-10-14T17:00:27.722684+0000","last_clean":"2023-10-14T17:00:27.722684+0000","last_became_active":"2023-10-14T17:00:27.721519+0000","last_became_peered":"2023-10-14T17:00:27.721519+0000","last_unstale":"2023-10-14T17:00:27.722684+0000","last_undegraded":"2023-10-14T17:00:27.722684+0000","last_fullsized":"2023-10-14T17:00:27.722684+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:38:37.963501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.21","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.709460+0000","last_change":"2023-10-14T17:00:27.706505+0000","last_active":"2023-10-14T17:00:27.709460+0000","last_peered":"2023-10-14T17:00:27.709460+0000","last_clean":"2023-10-14T17:00:27.709460+0000","last_became_active":"2023-10-14T17:00:27.706191+0000","last_became_peered":"2023-10-14T17:00:27.706191+0000","last_unstale":"2023-10-14T17:00:27.709460+0000","last_undegraded":"2023-10-14T17:00:27.709460+0000","last_fullsized":"2023-10-14T17:00:27.709460+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:32.958910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.20","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.725492+0000","last_change":"2023-10-14T17:00:27.721161+0000","last_active":"2023-10-14T17:00:27.725492+0000","last_peered":"2023-10-14T17:00:27.725492+0000","last_clean":"2023-10-14T17:00:27.725492+0000","last_became_active":"2023-10-14T17:00:27.720548+0000","last_became_peered":"2023-10-14T17:00:27.720548+0000","last_unstale":"2023-10-14T17:00:27.725492+0000","last_undegraded":"2023-10-14T17:00:27.725492+0000","last_fullsized":"2023-10-14T17:00:27.725492+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:31:40.798584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.723440+0000","last_change":"2023-10-14T17:00:27.714454+0000","last_active":"2023-10-14T17:00:27.723440+0000","last_peered":"2023-10-14T17:00:27.723440+0000","last_clean":"2023-10-14T17:00:27.723440+0000","last_became_active":"2023-10-14T17:00:27.711861+0000","last_became_peered":"2023-10-14T17:00:27.711861+0000","last_unstale":"2023-10-14T17:00:27.723440+0000","last_undegraded":"2023-10-14T17:00:27.723440+0000","last_fullsized":"2023-10-14T17:00:27.723440+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:28:25.770850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1f","version":"139'4","reported_seq":15,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.729653+0000","last_change":"2023-10-14T17:00:26.702397+0000","last_active":"2023-10-14T17:00:27.729653+0000","last_peered":"2023-10-14T17:00:27.729653+0000","last_clean":"2023-10-14T17:00:27.729653+0000","last_became_active":"2023-10-14T17:00:26.701862+0000","last_became_peered":"2023-10-14T17:00:26.701862+0000","last_unstale":"2023-10-14T17:00:27.729653+0000","last_undegraded":"2023-10-14T17:00:27.729653+0000","last_fullsized":"2023-10-14T17:00:27.729653+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:39.785613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712404+0000","last_change":"2023-10-14T17:00:27.706152+0000","last_active":"2023-10-14T17:00:28.712404+0000","last_peered":"2023-10-14T17:00:28.712404+0000","last_clean":"2023-10-14T17:00:28.712404+0000","last_became_active":"2023-10-14T17:00:27.705830+0000","last_became_peered":"2023-10-14T17:00:27.705830+0000","last_unstale":"2023-10-14T17:00:28.712404+0000","last_undegraded":"2023-10-14T17:00:28.712404+0000","last_fullsized":"2023-10-14T17:00:28.712404+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:24.963274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.1e","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712269+0000","last_change":"2023-10-14T17:00:26.703407+0000","last_active":"2023-10-14T17:00:28.712269+0000","last_peered":"2023-10-14T17:00:28.712269+0000","last_clean":"2023-10-14T17:00:28.712269+0000","last_became_active":"2023-10-14T17:00:26.703079+0000","last_became_peered":"2023-10-14T17:00:26.703079+0000","last_unstale":"2023-10-14T17:00:28.712269+0000","last_undegraded":"2023-10-14T17:00:28.712269+0000","last_fullsized":"2023-10-14T17:00:28.712269+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:46:19.839679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.727970+0000","last_change":"2023-10-14T17:00:27.721431+0000","last_active":"2023-10-14T17:00:27.727970+0000","last_peered":"2023-10-14T17:00:27.727970+0000","last_clean":"2023-10-14T17:00:27.727970+0000","last_became_active":"2023-10-14T17:00:27.721168+0000","last_became_peered":"2023-10-14T17:00:27.721168+0000","last_unstale":"2023-10-14T17:00:27.727970+0000","last_undegraded":"2023-10-14T17:00:27.727970+0000","last_fullsized":"2023-10-14T17:00:27.727970+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:46:01.758782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.1d","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.735885+0000","last_change":"2023-10-14T17:00:26.710602+0000","last_active":"2023-10-14T17:00:27.735885+0000","last_peered":"2023-10-14T17:00:27.735885+0000","last_clean":"2023-10-14T17:00:27.735885+0000","last_became_active":"2023-10-14T17:00:26.710315+0000","last_became_peered":"2023-10-14T17:00:26.710315+0000","last_unstale":"2023-10-14T17:00:27.735885+0000","last_undegraded":"2023-10-14T17:00:27.735885+0000","last_fullsized":"2023-10-14T17:00:27.735885+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:48.413979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712840+0000","last_change":"2023-10-14T17:00:27.713830+0000","last_active":"2023-10-14T17:00:28.712840+0000","last_peered":"2023-10-14T17:00:28.712840+0000","last_clean":"2023-10-14T17:00:28.712840+0000","last_became_active":"2023-10-14T17:00:27.713198+0000","last_became_peered":"2023-10-14T17:00:27.713198+0000","last_unstale":"2023-10-14T17:00:28.712840+0000","last_undegraded":"2023-10-14T17:00:28.712840+0000","last_fullsized":"2023-10-14T17:00:28.712840+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:33:16.468079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1c","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.737748+0000","last_change":"2023-10-14T17:00:26.710197+0000","last_active":"2023-10-14T17:00:27.737748+0000","last_peered":"2023-10-14T17:00:27.737748+0000","last_clean":"2023-10-14T17:00:27.737748+0000","last_became_active":"2023-10-14T17:00:26.709462+0000","last_became_peered":"2023-10-14T17:00:26.709462+0000","last_unstale":"2023-10-14T17:00:27.737748+0000","last_undegraded":"2023-10-14T17:00:27.737748+0000","last_fullsized":"2023-10-14T17:00:27.737748+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:51:16.222918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.720586+0000","last_change":"2023-10-14T17:00:27.715688+0000","last_active":"2023-10-14T17:00:27.720586+0000","last_peered":"2023-10-14T17:00:27.720586+0000","last_clean":"2023-10-14T17:00:27.720586+0000","last_became_active":"2023-10-14T17:00:27.715387+0000","last_became_peered":"2023-10-14T17:00:27.715387+0000","last_unstale":"2023-10-14T17:00:27.720586+0000","last_undegraded":"2023-10-14T17:00:27.720586+0000","last_fullsized":"2023-10-14T17:00:27.720586+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:31.630069+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.1b","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.734954+0000","last_change":"2023-10-14T17:00:26.701479+0000","last_active":"2023-10-14T17:00:27.734954+0000","last_peered":"2023-10-14T17:00:27.734954+0000","last_clean":"2023-10-14T17:00:27.734954+0000","last_became_active":"2023-10-14T17:00:26.700997+0000","last_became_peered":"2023-10-14T17:00:26.700997+0000","last_unstale":"2023-10-14T17:00:27.734954+0000","last_undegraded":"2023-10-14T17:00:27.734954+0000","last_fullsized":"2023-10-14T17:00:27.734954+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:05.832205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712637+0000","last_change":"2023-10-14T17:00:27.709667+0000","last_active":"2023-10-14T17:00:27.712637+0000","last_peered":"2023-10-14T17:00:27.712637+0000","last_clean":"2023-10-14T17:00:27.712637+0000","last_became_active":"2023-10-14T17:00:27.709389+0000","last_became_peered":"2023-10-14T17:00:27.709389+0000","last_unstale":"2023-10-14T17:00:27.712637+0000","last_undegraded":"2023-10-14T17:00:27.712637+0000","last_fullsized":"2023-10-14T17:00:27.712637+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:47:15.905006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.1a","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707041+0000","last_change":"2023-10-14T17:00:26.703372+0000","last_active":"2023-10-14T17:00:28.707041+0000","last_peered":"2023-10-14T17:00:28.707041+0000","last_clean":"2023-10-14T17:00:28.707041+0000","last_became_active":"2023-10-14T17:00:26.703096+0000","last_became_peered":"2023-10-14T17:00:26.703096+0000","last_unstale":"2023-10-14T17:00:28.707041+0000","last_undegraded":"2023-10-14T17:00:28.707041+0000","last_fullsized":"2023-10-14T17:00:28.707041+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:16:34.226749+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719368+0000","last_change":"2023-10-14T17:00:27.713122+0000","last_active":"2023-10-14T17:00:27.719368+0000","last_peered":"2023-10-14T17:00:27.719368+0000","last_clean":"2023-10-14T17:00:27.719368+0000","last_became_active":"2023-10-14T17:00:27.712805+0000","last_became_peered":"2023-10-14T17:00:27.712805+0000","last_unstale":"2023-10-14T17:00:27.719368+0000","last_undegraded":"2023-10-14T17:00:27.719368+0000","last_fullsized":"2023-10-14T17:00:27.719368+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:19:26.034995+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.19","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.691343+0000","last_change":"2023-10-14T17:00:26.694095+0000","last_active":"2023-10-14T17:00:27.691343+0000","last_peered":"2023-10-14T17:00:27.691343+0000","last_clean":"2023-10-14T17:00:27.691343+0000","last_became_active":"2023-10-14T17:00:26.693729+0000","last_became_peered":"2023-10-14T17:00:26.693729+0000","last_unstale":"2023-10-14T17:00:27.691343+0000","last_undegraded":"2023-10-14T17:00:27.691343+0000","last_fullsized":"2023-10-14T17:00:27.691343+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:08:25.444737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706334+0000","last_change":"2023-10-14T17:00:27.702208+0000","last_active":"2023-10-14T17:00:27.706334+0000","last_peered":"2023-10-14T17:00:27.706334+0000","last_clean":"2023-10-14T17:00:27.706334+0000","last_became_active":"2023-10-14T17:00:27.701445+0000","last_became_peered":"2023-10-14T17:00:27.701445+0000","last_unstale":"2023-10-14T17:00:27.706334+0000","last_undegraded":"2023-10-14T17:00:27.706334+0000","last_fullsized":"2023-10-14T17:00:27.706334+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:16:33.235089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.18","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.690298+0000","last_change":"2023-10-14T17:00:26.686342+0000","last_active":"2023-10-14T17:00:27.690298+0000","last_peered":"2023-10-14T17:00:27.690298+0000","last_clean":"2023-10-14T17:00:27.690298+0000","last_became_active":"2023-10-14T17:00:26.685947+0000","last_became_peered":"2023-10-14T17:00:26.685947+0000","last_unstale":"2023-10-14T17:00:27.690298+0000","last_undegraded":"2023-10-14T17:00:27.690298+0000","last_fullsized":"2023-10-14T17:00:27.690298+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.706771+0000","last_change":"2023-10-14T17:00:27.702814+0000","last_active":"2023-10-14T17:00:27.706771+0000","last_peered":"2023-10-14T17:00:27.706771+0000","last_clean":"2023-10-14T17:00:27.706771+0000","last_became_active":"2023-10-14T17:00:27.702502+0000","last_became_peered":"2023-10-14T17:00:27.702502+0000","last_unstale":"2023-10-14T17:00:27.706771+0000","last_undegraded":"2023-10-14T17:00:27.706771+0000","last_fullsized":"2023-10-14T17:00:27.706771+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:29.621105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.17","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.694144+0000","last_change":"2023-10-14T17:00:26.707443+0000","last_active":"2023-10-14T17:00:27.694144+0000","last_peered":"2023-10-14T17:00:27.694144+0000","last_clean":"2023-10-14T17:00:27.694144+0000","last_became_active":"2023-10-14T17:00:26.706297+0000","last_became_peered":"2023-10-14T17:00:26.706297+0000","last_unstale":"2023-10-14T17:00:27.694144+0000","last_undegraded":"2023-10-14T17:00:27.694144+0000","last_fullsized":"2023-10-14T17:00:27.694144+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:31:39.784253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712615+0000","last_change":"2023-10-14T17:00:27.707703+0000","last_active":"2023-10-14T17:00:28.712615+0000","last_peered":"2023-10-14T17:00:28.712615+0000","last_clean":"2023-10-14T17:00:28.712615+0000","last_became_active":"2023-10-14T17:00:27.707412+0000","last_became_peered":"2023-10-14T17:00:27.707412+0000","last_unstale":"2023-10-14T17:00:28.712615+0000","last_undegraded":"2023-10-14T17:00:28.712615+0000","last_fullsized":"2023-10-14T17:00:28.712615+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:10:20.722396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.16","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706883+0000","last_change":"2023-10-14T17:00:26.701323+0000","last_active":"2023-10-14T17:00:28.706883+0000","last_peered":"2023-10-14T17:00:28.706883+0000","last_clean":"2023-10-14T17:00:28.706883+0000","last_became_active":"2023-10-14T17:00:26.701051+0000","last_became_peered":"2023-10-14T17:00:26.701051+0000","last_unstale":"2023-10-14T17:00:28.706883+0000","last_undegraded":"2023-10-14T17:00:28.706883+0000","last_fullsized":"2023-10-14T17:00:28.706883+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:14.458842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707250+0000","last_change":"2023-10-14T17:00:27.712414+0000","last_active":"2023-10-14T17:00:28.707250+0000","last_peered":"2023-10-14T17:00:28.707250+0000","last_clean":"2023-10-14T17:00:28.707250+0000","last_became_active":"2023-10-14T17:00:27.707590+0000","last_became_peered":"2023-10-14T17:00:27.707590+0000","last_unstale":"2023-10-14T17:00:28.707250+0000","last_undegraded":"2023-10-14T17:00:28.707250+0000","last_fullsized":"2023-10-14T17:00:28.707250+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:30.874300+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.15","version":"139'3","reported_seq":14,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.736188+0000","last_change":"2023-10-14T17:00:26.708689+0000","last_active":"2023-10-14T17:00:27.736188+0000","last_peered":"2023-10-14T17:00:27.736188+0000","last_clean":"2023-10-14T17:00:27.736188+0000","last_became_active":"2023-10-14T17:00:26.707678+0000","last_became_peered":"2023-10-14T17:00:26.707678+0000","last_unstale":"2023-10-14T17:00:27.736188+0000","last_undegraded":"2023-10-14T17:00:27.736188+0000","last_fullsized":"2023-10-14T17:00:27.736188+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:16:32.220758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.701334+0000","last_change":"2023-10-14T17:00:27.698367+0000","last_active":"2023-10-14T17:00:27.701334+0000","last_peered":"2023-10-14T17:00:27.701334+0000","last_clean":"2023-10-14T17:00:27.701334+0000","last_became_active":"2023-10-14T17:00:27.698060+0000","last_became_peered":"2023-10-14T17:00:27.698060+0000","last_unstale":"2023-10-14T17:00:27.701334+0000","last_undegraded":"2023-10-14T17:00:27.701334+0000","last_fullsized":"2023-10-14T17:00:27.701334+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:47:15.905006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"16.14","version":"139'6","reported_seq":17,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.738281+0000","last_change":"2023-10-14T17:00:26.705089+0000","last_active":"2023-10-14T17:00:27.738281+0000","last_peered":"2023-10-14T17:00:27.738281+0000","last_clean":"2023-10-14T17:00:27.738281+0000","last_became_active":"2023-10-14T17:00:26.704625+0000","last_became_peered":"2023-10-14T17:00:26.704625+0000","last_unstale":"2023-10-14T17:00:27.738281+0000","last_undegraded":"2023-10-14T17:00:27.738281+0000","last_fullsized":"2023-10-14T17:00:27.738281+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:06.589035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706920+0000","last_change":"2023-10-14T17:00:27.711701+0000","last_active":"2023-10-14T17:00:28.706920+0000","last_peered":"2023-10-14T17:00:28.706920+0000","last_clean":"2023-10-14T17:00:28.706920+0000","last_became_active":"2023-10-14T17:00:27.711392+0000","last_became_peered":"2023-10-14T17:00:27.711392+0000","last_unstale":"2023-10-14T17:00:28.706920+0000","last_undegraded":"2023-10-14T17:00:28.706920+0000","last_fullsized":"2023-10-14T17:00:28.706920+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:32:32.026173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.13","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.736014+0000","last_change":"2023-10-14T17:00:26.707567+0000","last_active":"2023-10-14T17:00:27.736014+0000","last_peered":"2023-10-14T17:00:27.736014+0000","last_clean":"2023-10-14T17:00:27.736014+0000","last_became_active":"2023-10-14T17:00:26.707306+0000","last_became_peered":"2023-10-14T17:00:26.707306+0000","last_unstale":"2023-10-14T17:00:27.736014+0000","last_undegraded":"2023-10-14T17:00:27.736014+0000","last_fullsized":"2023-10-14T17:00:27.736014+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:59.169566+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706625+0000","last_change":"2023-10-14T17:00:27.711844+0000","last_active":"2023-10-14T17:00:28.706625+0000","last_peered":"2023-10-14T17:00:28.706625+0000","last_clean":"2023-10-14T17:00:28.706625+0000","last_became_active":"2023-10-14T17:00:27.711271+0000","last_became_peered":"2023-10-14T17:00:27.711271+0000","last_unstale":"2023-10-14T17:00:28.706625+0000","last_undegraded":"2023-10-14T17:00:28.706625+0000","last_fullsized":"2023-10-14T17:00:28.706625+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:13.083572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.12","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711476+0000","last_change":"2023-10-14T17:00:26.696467+0000","last_active":"2023-10-14T17:00:28.711476+0000","last_peered":"2023-10-14T17:00:28.711476+0000","last_clean":"2023-10-14T17:00:28.711476+0000","last_became_active":"2023-10-14T17:00:26.694771+0000","last_became_peered":"2023-10-14T17:00:26.694771+0000","last_unstale":"2023-10-14T17:00:28.711476+0000","last_undegraded":"2023-10-14T17:00:28.711476+0000","last_fullsized":"2023-10-14T17:00:28.711476+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:15:49.038164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706454+0000","last_change":"2023-10-14T17:00:27.718385+0000","last_active":"2023-10-14T17:00:28.706454+0000","last_peered":"2023-10-14T17:00:28.706454+0000","last_clean":"2023-10-14T17:00:28.706454+0000","last_became_active":"2023-10-14T17:00:27.716541+0000","last_became_peered":"2023-10-14T17:00:27.716541+0000","last_unstale":"2023-10-14T17:00:28.706454+0000","last_undegraded":"2023-10-14T17:00:28.706454+0000","last_fullsized":"2023-10-14T17:00:28.706454+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:39.359877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.10","version":"139'8","reported_seq":19,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.737331+0000","last_change":"2023-10-14T17:00:26.696555+0000","last_active":"2023-10-14T17:00:27.737331+0000","last_peered":"2023-10-14T17:00:27.737331+0000","last_clean":"2023-10-14T17:00:27.737331+0000","last_became_active":"2023-10-14T17:00:26.696252+0000","last_became_peered":"2023-10-14T17:00:26.696252+0000","last_unstale":"2023-10-14T17:00:27.737331+0000","last_undegraded":"2023-10-14T17:00:27.737331+0000","last_fullsized":"2023-10-14T17:00:27.737331+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:28:56.886782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.714664+0000","last_change":"2023-10-14T17:00:27.712967+0000","last_active":"2023-10-14T17:00:27.714664+0000","last_peered":"2023-10-14T17:00:27.714664+0000","last_clean":"2023-10-14T17:00:27.714664+0000","last_became_active":"2023-10-14T17:00:27.711569+0000","last_became_peered":"2023-10-14T17:00:27.711569+0000","last_unstale":"2023-10-14T17:00:27.714664+0000","last_undegraded":"2023-10-14T17:00:27.714664+0000","last_fullsized":"2023-10-14T17:00:27.714664+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:09.491629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"16.f","version":"139'1","reported_seq":12,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.733754+0000","last_change":"2023-10-14T17:00:26.700894+0000","last_active":"2023-10-14T17:00:27.733754+0000","last_peered":"2023-10-14T17:00:27.733754+0000","last_clean":"2023-10-14T17:00:27.733754+0000","last_became_active":"2023-10-14T17:00:26.700460+0000","last_became_peered":"2023-10-14T17:00:26.700460+0000","last_unstale":"2023-10-14T17:00:27.733754+0000","last_undegraded":"2023-10-14T17:00:27.733754+0000","last_fullsized":"2023-10-14T17:00:27.733754+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:23.081298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.724589+0000","last_change":"2023-10-14T17:00:27.718095+0000","last_active":"2023-10-14T17:00:27.724589+0000","last_peered":"2023-10-14T17:00:27.724589+0000","last_clean":"2023-10-14T17:00:27.724589+0000","last_became_active":"2023-10-14T17:00:27.717779+0000","last_became_peered":"2023-10-14T17:00:27.717779+0000","last_unstale":"2023-10-14T17:00:27.724589+0000","last_undegraded":"2023-10-14T17:00:27.724589+0000","last_fullsized":"2023-10-14T17:00:27.724589+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"16.e","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.731039+0000","last_change":"2023-10-14T17:00:26.702217+0000","last_active":"2023-10-14T17:00:27.731039+0000","last_peered":"2023-10-14T17:00:27.731039+0000","last_clean":"2023-10-14T17:00:27.731039+0000","last_became_active":"2023-10-14T17:00:26.701945+0000","last_became_peered":"2023-10-14T17:00:26.701945+0000","last_unstale":"2023-10-14T17:00:27.731039+0000","last_undegraded":"2023-10-14T17:00:27.731039+0000","last_fullsized":"2023-10-14T17:00:27.731039+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:33:36.818209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.714037+0000","last_change":"2023-10-14T17:00:27.708636+0000","last_active":"2023-10-14T17:00:28.714037+0000","last_peered":"2023-10-14T17:00:28.714037+0000","last_clean":"2023-10-14T17:00:28.714037+0000","last_became_active":"2023-10-14T17:00:27.708304+0000","last_became_peered":"2023-10-14T17:00:27.708304+0000","last_unstale":"2023-10-14T17:00:28.714037+0000","last_undegraded":"2023-10-14T17:00:28.714037+0000","last_fullsized":"2023-10-14T17:00:28.714037+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:32:27.367690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.d","version":"139'1","reported_seq":13,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711711+0000","last_change":"2023-10-14T17:00:26.696701+0000","last_active":"2023-10-14T17:00:28.711711+0000","last_peered":"2023-10-14T17:00:28.711711+0000","last_clean":"2023-10-14T17:00:28.711711+0000","last_became_active":"2023-10-14T17:00:26.695741+0000","last_became_peered":"2023-10-14T17:00:26.695741+0000","last_unstale":"2023-10-14T17:00:28.711711+0000","last_undegraded":"2023-10-14T17:00:28.711711+0000","last_fullsized":"2023-10-14T17:00:28.711711+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:22.265802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.d","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713162+0000","last_change":"2023-10-14T17:00:27.710579+0000","last_active":"2023-10-14T17:00:28.713162+0000","last_peered":"2023-10-14T17:00:28.713162+0000","last_clean":"2023-10-14T17:00:28.713162+0000","last_became_active":"2023-10-14T17:00:27.710288+0000","last_became_peered":"2023-10-14T17:00:27.710288+0000","last_unstale":"2023-10-14T17:00:28.713162+0000","last_undegraded":"2023-10-14T17:00:28.713162+0000","last_fullsized":"2023-10-14T17:00:28.713162+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:35.948608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.c","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.683720+0000","last_change":"2023-10-14T17:00:26.706134+0000","last_active":"2023-10-14T17:00:27.683720+0000","last_peered":"2023-10-14T17:00:27.683720+0000","last_clean":"2023-10-14T17:00:27.683720+0000","last_became_active":"2023-10-14T17:00:26.705886+0000","last_became_peered":"2023-10-14T17:00:26.705886+0000","last_unstale":"2023-10-14T17:00:27.683720+0000","last_undegraded":"2023-10-14T17:00:27.683720+0000","last_fullsized":"2023-10-14T17:00:27.683720+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:22.804287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.710478+0000","last_change":"2023-10-14T17:00:27.716777+0000","last_active":"2023-10-14T17:00:28.710478+0000","last_peered":"2023-10-14T17:00:28.710478+0000","last_clean":"2023-10-14T17:00:28.710478+0000","last_became_active":"2023-10-14T17:00:27.715904+0000","last_became_peered":"2023-10-14T17:00:27.715904+0000","last_unstale":"2023-10-14T17:00:28.710478+0000","last_undegraded":"2023-10-14T17:00:28.710478+0000","last_fullsized":"2023-10-14T17:00:28.710478+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:13.083572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.b","version":"139'2","reported_seq":13,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.735608+0000","last_change":"2023-10-14T17:00:26.698897+0000","last_active":"2023-10-14T17:00:27.735608+0000","last_peered":"2023-10-14T17:00:27.735608+0000","last_clean":"2023-10-14T17:00:27.735608+0000","last_became_active":"2023-10-14T17:00:26.698601+0000","last_became_peered":"2023-10-14T17:00:26.698601+0000","last_unstale":"2023-10-14T17:00:27.735608+0000","last_undegraded":"2023-10-14T17:00:27.735608+0000","last_fullsized":"2023-10-14T17:00:27.735608+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719219+0000","last_change":"2023-10-14T17:00:27.710137+0000","last_active":"2023-10-14T17:00:27.719219+0000","last_peered":"2023-10-14T17:00:27.719219+0000","last_clean":"2023-10-14T17:00:27.719219+0000","last_became_active":"2023-10-14T17:00:27.709841+0000","last_became_peered":"2023-10-14T17:00:27.709841+0000","last_unstale":"2023-10-14T17:00:27.719219+0000","last_undegraded":"2023-10-14T17:00:27.719219+0000","last_fullsized":"2023-10-14T17:00:27.719219+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:53:55.637606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.a","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.692426+0000","last_change":"2023-10-14T17:00:26.705121+0000","last_active":"2023-10-14T17:00:27.692426+0000","last_peered":"2023-10-14T17:00:27.692426+0000","last_clean":"2023-10-14T17:00:27.692426+0000","last_became_active":"2023-10-14T17:00:26.704834+0000","last_became_peered":"2023-10-14T17:00:26.704834+0000","last_unstale":"2023-10-14T17:00:27.692426+0000","last_undegraded":"2023-10-14T17:00:27.692426+0000","last_fullsized":"2023-10-14T17:00:27.692426+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:51:17.378788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713259+0000","last_change":"2023-10-14T17:00:27.715539+0000","last_active":"2023-10-14T17:00:28.713259+0000","last_peered":"2023-10-14T17:00:28.713259+0000","last_clean":"2023-10-14T17:00:28.713259+0000","last_became_active":"2023-10-14T17:00:27.714489+0000","last_became_peered":"2023-10-14T17:00:27.714489+0000","last_unstale":"2023-10-14T17:00:28.713259+0000","last_undegraded":"2023-10-14T17:00:28.713259+0000","last_fullsized":"2023-10-14T17:00:28.713259+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:20.652106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.9","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.697629+0000","last_change":"2023-10-14T17:00:26.698957+0000","last_active":"2023-10-14T17:00:27.697629+0000","last_peered":"2023-10-14T17:00:27.697629+0000","last_clean":"2023-10-14T17:00:27.697629+0000","last_became_active":"2023-10-14T17:00:26.698336+0000","last_became_peered":"2023-10-14T17:00:26.698336+0000","last_unstale":"2023-10-14T17:00:27.697629+0000","last_undegraded":"2023-10-14T17:00:27.697629+0000","last_fullsized":"2023-10-14T17:00:27.697629+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:22.265802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706545+0000","last_change":"2023-10-14T17:00:27.722973+0000","last_active":"2023-10-14T17:00:28.706545+0000","last_peered":"2023-10-14T17:00:28.706545+0000","last_clean":"2023-10-14T17:00:28.706545+0000","last_became_active":"2023-10-14T17:00:27.721756+0000","last_became_peered":"2023-10-14T17:00:27.721756+0000","last_unstale":"2023-10-14T17:00:28.706545+0000","last_undegraded":"2023-10-14T17:00:28.706545+0000","last_fullsized":"2023-10-14T17:00:28.706545+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:16:30.342922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.8","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.699170+0000","last_change":"2023-10-14T17:00:26.698460+0000","last_active":"2023-10-14T17:00:27.699170+0000","last_peered":"2023-10-14T17:00:27.699170+0000","last_clean":"2023-10-14T17:00:27.699170+0000","last_became_active":"2023-10-14T17:00:26.697829+0000","last_became_peered":"2023-10-14T17:00:26.697829+0000","last_unstale":"2023-10-14T17:00:27.699170+0000","last_undegraded":"2023-10-14T17:00:27.699170+0000","last_fullsized":"2023-10-14T17:00:27.699170+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:00.973366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711929+0000","last_change":"2023-10-14T17:00:27.710864+0000","last_active":"2023-10-14T17:00:28.711929+0000","last_peered":"2023-10-14T17:00:28.711929+0000","last_clean":"2023-10-14T17:00:28.711929+0000","last_became_active":"2023-10-14T17:00:27.708866+0000","last_became_peered":"2023-10-14T17:00:27.708866+0000","last_unstale":"2023-10-14T17:00:28.711929+0000","last_undegraded":"2023-10-14T17:00:28.711929+0000","last_fullsized":"2023-10-14T17:00:28.711929+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:32.958910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.7","version":"139'2","reported_seq":14,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706965+0000","last_change":"2023-10-14T17:00:26.688428+0000","last_active":"2023-10-14T17:00:28.706965+0000","last_peered":"2023-10-14T17:00:28.706965+0000","last_clean":"2023-10-14T17:00:28.706965+0000","last_became_active":"2023-10-14T17:00:26.686124+0000","last_became_peered":"2023-10-14T17:00:26.686124+0000","last_unstale":"2023-10-14T17:00:28.706965+0000","last_undegraded":"2023-10-14T17:00:28.706965+0000","last_fullsized":"2023-10-14T17:00:28.706965+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:31.467942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.712922+0000","last_change":"2023-10-14T17:00:27.706650+0000","last_active":"2023-10-14T17:00:27.712922+0000","last_peered":"2023-10-14T17:00:27.712922+0000","last_clean":"2023-10-14T17:00:27.712922+0000","last_became_active":"2023-10-14T17:00:27.705419+0000","last_became_peered":"2023-10-14T17:00:27.705419+0000","last_unstale":"2023-10-14T17:00:27.712922+0000","last_undegraded":"2023-10-14T17:00:27.712922+0000","last_fullsized":"2023-10-14T17:00:27.712922+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:37:37.242371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.6","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.712347+0000","last_change":"2023-10-14T17:00:26.700612+0000","last_active":"2023-10-14T17:00:28.712347+0000","last_peered":"2023-10-14T17:00:28.712347+0000","last_clean":"2023-10-14T17:00:28.712347+0000","last_became_active":"2023-10-14T17:00:26.699664+0000","last_became_peered":"2023-10-14T17:00:26.699664+0000","last_unstale":"2023-10-14T17:00:28.712347+0000","last_undegraded":"2023-10-14T17:00:28.712347+0000","last_fullsized":"2023-10-14T17:00:28.712347+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:59:06.522576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.721211+0000","last_change":"2023-10-14T17:00:27.711496+0000","last_active":"2023-10-14T17:00:27.721211+0000","last_peered":"2023-10-14T17:00:27.721211+0000","last_clean":"2023-10-14T17:00:27.721211+0000","last_became_active":"2023-10-14T17:00:27.711174+0000","last_became_peered":"2023-10-14T17:00:27.711174+0000","last_unstale":"2023-10-14T17:00:27.721211+0000","last_undegraded":"2023-10-14T17:00:27.721211+0000","last_fullsized":"2023-10-14T17:00:27.721211+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:55:18.587372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.5","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.702641+0000","last_change":"2023-10-14T17:00:26.701564+0000","last_active":"2023-10-14T17:00:27.702641+0000","last_peered":"2023-10-14T17:00:27.702641+0000","last_clean":"2023-10-14T17:00:27.702641+0000","last_became_active":"2023-10-14T17:00:26.701274+0000","last_became_peered":"2023-10-14T17:00:26.701274+0000","last_unstale":"2023-10-14T17:00:27.702641+0000","last_undegraded":"2023-10-14T17:00:27.702641+0000","last_fullsized":"2023-10-14T17:00:27.702641+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:51:05.832205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.711478+0000","last_change":"2023-10-14T17:00:27.707319+0000","last_active":"2023-10-14T17:00:28.711478+0000","last_peered":"2023-10-14T17:00:28.711478+0000","last_clean":"2023-10-14T17:00:28.711478+0000","last_became_active":"2023-10-14T17:00:27.707007+0000","last_became_peered":"2023-10-14T17:00:27.707007+0000","last_unstale":"2023-10-14T17:00:28.711478+0000","last_undegraded":"2023-10-14T17:00:28.711478+0000","last_fullsized":"2023-10-14T17:00:28.711478+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:36:21.164580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"16.4","version":"0'0","reported_seq":12,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.707364+0000","last_change":"2023-10-14T17:00:26.709728+0000","last_active":"2023-10-14T17:00:28.707364+0000","last_peered":"2023-10-14T17:00:28.707364+0000","last_clean":"2023-10-14T17:00:28.707364+0000","last_became_active":"2023-10-14T17:00:26.709344+0000","last_became_peered":"2023-10-14T17:00:26.709344+0000","last_unstale":"2023-10-14T17:00:28.707364+0000","last_undegraded":"2023-10-14T17:00:28.707364+0000","last_fullsized":"2023-10-14T17:00:28.707364+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:04.405861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.699124+0000","last_change":"2023-10-14T17:00:27.698357+0000","last_active":"2023-10-14T17:00:27.699124+0000","last_peered":"2023-10-14T17:00:27.699124+0000","last_clean":"2023-10-14T17:00:27.699124+0000","last_became_active":"2023-10-14T17:00:27.697981+0000","last_became_peered":"2023-10-14T17:00:27.697981+0000","last_unstale":"2023-10-14T17:00:27.699124+0000","last_undegraded":"2023-10-14T17:00:27.699124+0000","last_fullsized":"2023-10-14T17:00:27.699124+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:27.066569+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.3","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.683342+0000","last_change":"2023-10-14T17:00:26.698103+0000","last_active":"2023-10-14T17:00:27.683342+0000","last_peered":"2023-10-14T17:00:27.683342+0000","last_clean":"2023-10-14T17:00:27.683342+0000","last_became_active":"2023-10-14T17:00:26.697804+0000","last_became_peered":"2023-10-14T17:00:26.697804+0000","last_unstale":"2023-10-14T17:00:27.683342+0000","last_undegraded":"2023-10-14T17:00:27.683342+0000","last_fullsized":"2023-10-14T17:00:27.683342+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:15:34.312839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.717604+0000","last_change":"2023-10-14T17:00:27.715742+0000","last_active":"2023-10-14T17:00:27.717604+0000","last_peered":"2023-10-14T17:00:27.717604+0000","last_clean":"2023-10-14T17:00:27.717604+0000","last_became_active":"2023-10-14T17:00:27.713626+0000","last_became_peered":"2023-10-14T17:00:27.713626+0000","last_unstale":"2023-10-14T17:00:27.717604+0000","last_undegraded":"2023-10-14T17:00:27.717604+0000","last_fullsized":"2023-10-14T17:00:27.717604+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:10.610786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"16.2","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.691819+0000","last_change":"2023-10-14T17:00:26.709571+0000","last_active":"2023-10-14T17:00:27.691819+0000","last_peered":"2023-10-14T17:00:27.691819+0000","last_clean":"2023-10-14T17:00:27.691819+0000","last_became_active":"2023-10-14T17:00:26.709247+0000","last_became_peered":"2023-10-14T17:00:26.709247+0000","last_unstale":"2023-10-14T17:00:27.691819+0000","last_undegraded":"2023-10-14T17:00:27.691819+0000","last_fullsized":"2023-10-14T17:00:27.691819+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:04:55.155135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713546+0000","last_change":"2023-10-14T17:00:27.714781+0000","last_active":"2023-10-14T17:00:28.713546+0000","last_peered":"2023-10-14T17:00:28.713546+0000","last_clean":"2023-10-14T17:00:28.713546+0000","last_became_active":"2023-10-14T17:00:27.714100+0000","last_became_peered":"2023-10-14T17:00:27.714100+0000","last_unstale":"2023-10-14T17:00:28.713546+0000","last_undegraded":"2023-10-14T17:00:28.713546+0000","last_fullsized":"2023-10-14T17:00:28.713546+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:56:41.032208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"16.1","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.689194+0000","last_change":"2023-10-14T17:00:26.701230+0000","last_active":"2023-10-14T17:00:27.689194+0000","last_peered":"2023-10-14T17:00:27.689194+0000","last_clean":"2023-10-14T17:00:27.689194+0000","last_became_active":"2023-10-14T17:00:26.700875+0000","last_became_peered":"2023-10-14T17:00:26.700875+0000","last_unstale":"2023-10-14T17:00:27.689194+0000","last_undegraded":"2023-10-14T17:00:27.689194+0000","last_fullsized":"2023-10-14T17:00:27.689194+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:13:55.243575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.1","version":"0'0","reported_seq":9,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.719965+0000","last_change":"2023-10-14T17:00:27.713305+0000","last_active":"2023-10-14T17:00:27.719965+0000","last_peered":"2023-10-14T17:00:27.719965+0000","last_clean":"2023-10-14T17:00:27.719965+0000","last_became_active":"2023-10-14T17:00:27.713039+0000","last_became_peered":"2023-10-14T17:00:27.713039+0000","last_unstale":"2023-10-14T17:00:27.719965+0000","last_undegraded":"2023-10-14T17:00:27.719965+0000","last_fullsized":"2023-10-14T17:00:27.719965+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:24.963274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"16.0","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.696317+0000","last_change":"2023-10-14T17:00:26.698598+0000","last_active":"2023-10-14T17:00:27.696317+0000","last_peered":"2023-10-14T17:00:27.696317+0000","last_clean":"2023-10-14T17:00:27.696317+0000","last_became_active":"2023-10-14T17:00:26.698199+0000","last_became_peered":"2023-10-14T17:00:26.698199+0000","last_unstale":"2023-10-14T17:00:27.696317+0000","last_undegraded":"2023-10-14T17:00:27.696317+0000","last_fullsized":"2023-10-14T17:00:27.696317+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:24:40.243390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":11,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.706721+0000","last_change":"2023-10-14T17:00:27.718278+0000","last_active":"2023-10-14T17:00:28.706721+0000","last_peered":"2023-10-14T17:00:28.706721+0000","last_clean":"2023-10-14T17:00:28.706721+0000","last_became_active":"2023-10-14T17:00:27.717956+0000","last_became_peered":"2023-10-14T17:00:27.717956+0000","last_unstale":"2023-10-14T17:00:28.706721+0000","last_undegraded":"2023-10-14T17:00:28.706721+0000","last_fullsized":"2023-10-14T17:00:28.706721+0000","mapping_epoch":138,"log_start":"0'0","ondisk_log_start":"0'0","created":138,"last_epoch_clean":139,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:26.653721+0000","last_clean_scrub_stamp":"2023-10-14T17:00:26.653721+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:53:26.430276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"16.11","version":"0'0","reported_seq":11,"reported_epoch":139,"state":"active+clean","last_fresh":"2023-10-14T17:00:27.685650+0000","last_change":"2023-10-14T17:00:26.705431+0000","last_active":"2023-10-14T17:00:27.685650+0000","last_peered":"2023-10-14T17:00:27.685650+0000","last_clean":"2023-10-14T17:00:27.685650+0000","last_became_active":"2023-10-14T17:00:26.704776+0000","last_became_peered":"2023-10-14T17:00:26.704776+0000","last_unstale":"2023-10-14T17:00:27.685650+0000","last_undegraded":"2023-10-14T17:00:27.685650+0000","last_fullsized":"2023-10-14T17:00:27.685650+0000","mapping_epoch":137,"log_start":"0'0","ondisk_log_start":"0'0","created":137,"last_epoch_clean":138,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:00:25.639390+0000","last_clean_scrub_stamp":"2023-10-14T17:00:25.639390+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:05.419531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"128'227","reported_seq":666,"reported_epoch":140,"state":"active+clean","last_fresh":"2023-10-14T17:00:28.713909+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:00:28.713909+0000","last_peered":"2023-10-14T17:00:28.713909+0000","last_clean":"2023-10-14T17:00:28.713909+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:00:28.713909+0000","last_undegraded":"2023-10-14T17:00:28.713909+0000","last_fullsized":"2023-10-14T17:00:28.713909+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":227,"log_dups_size":0,"ondisk_log_size":227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":17,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":16,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1076,"num_read_kb":2160,"num_write":359,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":227,"ondisk_log_size":227,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575088,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37628,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334212,"statfs":{"total":96636764160,"available":96598233088,"internally_reserved":0,"allocated":2338816,"data_stored":1433035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.1519999999999999,"15min":1.1699999999999999},"min":{"1min":0.504,"5min":0.504,"15min":0.504},"max":{"1min":1.7110000000000001,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.103,"5min":1.1339999999999999,"15min":1.194},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.38900000000000001},"max":{"1min":1.5800000000000001,"5min":1.871,"15min":2.117},"last":0.745}]},{"osd":1,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.1859999999999999,"15min":1.222},"min":{"1min":0.74299999999999999,"5min":0.71399999999999997,"15min":0.56299999999999994},"max":{"1min":1.72,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.2430000000000001,"5min":1.224,"15min":1.254},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.57799999999999996},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":2.0259999999999998},"last":1.1599999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.149,"5min":1.181,"15min":1.2150000000000001},"min":{"1min":0.70799999999999996,"5min":0.623,"15min":0.47099999999999997},"max":{"1min":1.4419999999999999,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.577},{"interface":"front","average":{"1min":1.151,"5min":1.266,"15min":1.294},"min":{"1min":0.83199999999999996,"5min":0.78900000000000003,"15min":0.72599999999999998},"max":{"1min":1.6080000000000001,"5min":1.893,"15min":2.1589999999999998},"last":1.4430000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.3149999999999999,"15min":1.357},"min":{"1min":0.85899999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.657,"5min":2.056,"15min":2.4159999999999999},"last":1.018},{"interface":"front","average":{"1min":1.417,"5min":1.4330000000000001,"15min":1.446},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.72199999999999998},"max":{"1min":1.7949999999999999,"5min":2.379,"15min":2.379},"last":1.343}]},{"osd":4,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.216,"5min":1.3,"15min":1.28},"min":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":2.0960000000000001,"5min":2.0960000000000001,"15min":2.1819999999999999},"last":1.286},{"interface":"front","average":{"1min":1.226,"5min":1.3400000000000001,"15min":1.327},"min":{"1min":0.66600000000000004,"5min":0.61499999999999999,"15min":0.59799999999999998},"max":{"1min":1.748,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.73}]},{"osd":5,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.405,"15min":1.391},"min":{"1min":0.55200000000000005,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.877,"5min":2.278,"15min":2.278},"last":1.546},{"interface":"front","average":{"1min":1.383,"5min":1.444,"15min":1.4590000000000001},"min":{"1min":0.71199999999999997,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.2029999999999998,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.3919999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:06 2023","interfaces":[{"interface":"back","average":{"1min":1.5249999999999999,"5min":1.5389999999999999,"15min":1.544},"min":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":2.2959999999999998,"5min":2.492,"15min":2.492},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.512,"15min":1.48},"min":{"1min":0.80200000000000005,"5min":0.59399999999999997,"15min":0.58399999999999996},"max":{"1min":1.984,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.8320000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575087,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1700,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1740800,"data_stored":842121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0269999999999999,"5min":1.123,"15min":1.1439999999999999},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.524,"5min":1.7330000000000001,"15min":2.137},"last":1.4019999999999999},{"interface":"front","average":{"1min":1.0800000000000001,"5min":1.159,"15min":1.2050000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":1.4770000000000001,"5min":1.8100000000000001,"15min":2.0760000000000001},"last":1.0349999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.1479999999999999,"15min":1.1759999999999999},"min":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.41299999999999998},"max":{"1min":1.5580000000000001,"5min":1.631,"15min":2.802},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.236,"15min":1.276},"min":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.438},"max":{"1min":1.6899999999999999,"5min":1.9810000000000001,"15min":2.512},"last":1.181}]},{"osd":2,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.262,"5min":1.3100000000000001,"15min":1.3420000000000001},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":1.667,"5min":1.9970000000000001,"15min":3.3290000000000002},"last":1.361},{"interface":"front","average":{"1min":1.3220000000000001,"5min":1.3140000000000001,"15min":1.3819999999999999},"min":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.46000000000000002},"max":{"1min":1.855,"5min":1.855,"15min":3.286},"last":1.649}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.302,"15min":1.361},"min":{"1min":0.68200000000000005,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":1.8260000000000001,"5min":1.9890000000000001,"15min":2.823},"last":1.478},{"interface":"front","average":{"1min":1.349,"5min":1.3500000000000001,"15min":1.403},"min":{"1min":0.84899999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":1.589,"5min":1.774,"15min":4.2610000000000001},"last":1.5069999999999999}]},{"osd":4,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.3120000000000001,"15min":1.331},"min":{"1min":0.60199999999999998,"5min":0.497,"15min":0.497},"max":{"1min":1.774,"5min":1.9299999999999999,"15min":2.9460000000000002},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.361,"5min":1.4119999999999999,"15min":1.415},"min":{"1min":0.70999999999999996,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":1.9259999999999999,"5min":1.9259999999999999,"15min":2.5750000000000002},"last":1.605}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.25,"5min":1.335,"15min":1.3720000000000001},"min":{"1min":0.58399999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.7829999999999999,"5min":1.891,"15min":3.4060000000000001},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.52,"15min":1.544},"min":{"1min":0.747,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":1.9219999999999999,"5min":2.2559999999999998,"15min":3.0489999999999999},"last":1.5760000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.421,"5min":1.536,"15min":1.573},"min":{"1min":0.76900000000000002,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.8009999999999999,"5min":2.1110000000000002,"15min":2.8570000000000002},"last":1.714},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.4930000000000001,"15min":1.5269999999999999},"min":{"1min":0.64300000000000002,"5min":0.623,"15min":0.623},"max":{"1min":2.0270000000000001,"5min":2.0270000000000001,"15min":2.7000000000000002},"last":1.542}]}]},{"osd":5,"up_from":13,"seq":55834575088,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37040,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334800,"statfs":{"total":96636764160,"available":96598835200,"internally_reserved":0,"allocated":1732608,"data_stored":836283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1419999999999999,"5min":1.0780000000000001,"15min":1.109},"min":{"1min":0.69399999999999995,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.5569999999999999,"5min":1.5620000000000001,"15min":1.986},"last":0.90900000000000003},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.1559999999999999,"15min":1.139},"min":{"1min":0.747,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":1.8029999999999999,"5min":1.8029999999999999,"15min":2.0430000000000001},"last":1.1990000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.224,"15min":1.1870000000000001},"min":{"1min":0.874,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.7290000000000001,"5min":1.744,"15min":2.1200000000000001},"last":1.661},{"interface":"front","average":{"1min":1.1699999999999999,"5min":1.175,"15min":1.1990000000000001},"min":{"1min":0.80900000000000005,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":1.635,"5min":1.7629999999999999,"15min":1.905},"last":1.8999999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.115,"15min":1.1719999999999999},"min":{"1min":0.68100000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.681,"5min":1.681,"15min":2.375},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.306,"15min":1.319},"min":{"1min":0.81299999999999994,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.788,"5min":1.788,"15min":2.3100000000000001},"last":1.8200000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.3819999999999999,"5min":1.3540000000000001,"15min":1.381},"min":{"1min":0.92600000000000005,"5min":0.77200000000000002,"15min":0.745},"max":{"1min":1.962,"5min":1.962,"15min":2.4180000000000001},"last":2.1139999999999999},{"interface":"front","average":{"1min":1.4279999999999999,"5min":1.329,"15min":1.3899999999999999},"min":{"1min":0.88300000000000001,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.819,"5min":1.819,"15min":2.1429999999999998},"last":1.9650000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.319,"15min":1.292},"min":{"1min":1.0569999999999999,"5min":0.434,"15min":0.434},"max":{"1min":1.665,"5min":1.744,"15min":2.4889999999999999},"last":2.1709999999999998},{"interface":"front","average":{"1min":1.4299999999999999,"5min":1.3700000000000001,"15min":1.3420000000000001},"min":{"1min":1.002,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":1.8879999999999999,"5min":1.9119999999999999,"15min":2.5169999999999999},"last":2.2240000000000002}]},{"osd":6,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2829999999999999,"5min":1.2749999999999999,"15min":1.3460000000000001},"min":{"1min":0.82999999999999996,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":1.8999999999999999,"5min":1.9650000000000001,"15min":2.2869999999999999},"last":2.403},{"interface":"front","average":{"1min":1.5269999999999999,"5min":1.5069999999999999,"15min":1.5249999999999999},"min":{"1min":0.88400000000000001,"5min":0.63,"15min":0.63},"max":{"1min":1.994,"5min":1.994,"15min":2.613},"last":2.0459999999999998}]},{"osd":7,"last update":"Sat Oct 14 16:59:55 2023","interfaces":[{"interface":"back","average":{"1min":1.484,"5min":1.472,"15min":1.5009999999999999},"min":{"1min":1.1439999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.605},"last":2.29},{"interface":"front","average":{"1min":1.446,"5min":1.4390000000000001,"15min":1.4910000000000001},"min":{"1min":1.0209999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.3980000000000001},"last":2.48}]}]},{"osd":4,"up_from":13,"seq":55834575088,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59240,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312600,"statfs":{"total":96636764160,"available":96576102400,"internally_reserved":0,"allocated":1724416,"data_stored":835679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1559999999999999,"5min":1.1519999999999999,"15min":1.1739999999999999},"min":{"1min":0.68000000000000005,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":1.7270000000000001,"5min":1.9319999999999999,"15min":1.982},"last":0.72199999999999998},{"interface":"front","average":{"1min":1.177,"5min":1.2649999999999999,"15min":1.2330000000000001},"min":{"1min":0.73799999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.6579999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.80400000000000005}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1919999999999999,"5min":1.323,"15min":1.2869999999999999},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.48199999999999998},"max":{"1min":1.607,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.3959999999999999},{"interface":"front","average":{"1min":1.1519999999999999,"5min":1.226,"15min":1.2490000000000001},"min":{"1min":0.71899999999999997,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.639,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":0.95299999999999996}]},{"osd":2,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1850000000000001,"5min":1.262,"15min":1.252},"min":{"1min":0.68000000000000005,"5min":0.67600000000000005,"15min":0.64700000000000002},"max":{"1min":1.5940000000000001,"5min":5.633,"15min":5.633},"last":1.2330000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3779999999999999,"15min":1.375},"min":{"1min":0.93100000000000005,"5min":0.81100000000000005,"15min":0.71999999999999997},"max":{"1min":1.7210000000000001,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.1830000000000001}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.2350000000000001,"5min":1.304,"15min":1.3149999999999999},"min":{"1min":0.79500000000000004,"5min":0.77100000000000002,"15min":0.71699999999999997},"max":{"1min":1.8080000000000001,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.288},{"interface":"front","average":{"1min":1.369,"5min":1.4119999999999999,"15min":1.409},"min":{"1min":0.85099999999999998,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":1.6200000000000001,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.1220000000000001}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.401,"15min":1.365},"min":{"1min":1.012,"5min":0.64500000000000002,"15min":0.52900000000000003},"max":{"1min":1.724,"5min":2.931,"15min":2.931},"last":1.534},{"interface":"front","average":{"1min":1.288,"5min":1.347,"15min":1.347},"min":{"1min":0.86599999999999999,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.77,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.0920000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.431,"15min":1.395},"min":{"1min":0.997,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.6879999999999999,"5min":2.323,"15min":2.323},"last":1.768},{"interface":"front","average":{"1min":1.492,"5min":1.544,"15min":1.53},"min":{"1min":1.093,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.8109999999999999,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.637}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.51,"15min":1.4890000000000001},"min":{"1min":0.92700000000000005,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.8420000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4430000000000001},{"interface":"front","average":{"1min":1.556,"5min":1.5589999999999999,"15min":1.554},"min":{"1min":1.1379999999999999,"5min":0.82899999999999996,"15min":0.79000000000000004},"max":{"1min":1.9319999999999999,"5min":2.0209999999999999,"15min":2.3130000000000002},"last":1.6839999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575088,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41220,"kb_used_data":1712,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330620,"statfs":{"total":96636764160,"available":96594554880,"internally_reserved":0,"allocated":1753088,"data_stored":843733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":40434120},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.90700000000000003,"5min":0.90000000000000002,"15min":0.92000000000000004},"min":{"1min":0.51400000000000001,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.264,"5min":1.621,"15min":1.621},"last":1.407},{"interface":"front","average":{"1min":1.036,"5min":1.0600000000000001,"15min":1.1240000000000001},"min":{"1min":0.59899999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":1.52,"5min":1.52,"15min":1.8799999999999999},"last":1.2290000000000001}]},{"osd":1,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":0.89800000000000002,"5min":0.876,"15min":0.90800000000000003},"min":{"1min":0.56000000000000005,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.635,"5min":1.7609999999999999,"15min":2.1720000000000002},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.0489999999999999,"5min":1.0900000000000001,"15min":1.101},"min":{"1min":0.56799999999999995,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":1.419,"5min":1.9159999999999999,"15min":1.9159999999999999},"last":1.018}]},{"osd":2,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.2689999999999999,"15min":1.347},"min":{"1min":0.77200000000000002,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":1.762,"5min":2.024,"15min":2.0609999999999999},"last":1.7130000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.242,"15min":1.27},"min":{"1min":0.71599999999999997,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.883,"5min":1.883,"15min":2.0960000000000001},"last":0.97099999999999997}]},{"osd":4,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.321,"15min":1.353},"min":{"1min":0.97299999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.913,"5min":1.913,"15min":3.4350000000000001},"last":1.276},{"interface":"front","average":{"1min":1.397,"5min":1.389,"15min":1.407},"min":{"1min":0.94099999999999995,"5min":0.745,"15min":0.745},"max":{"1min":1.845,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.1859999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.3220000000000001,"5min":1.339,"15min":1.387},"min":{"1min":0.98699999999999999,"5min":0.76300000000000001,"15min":0.76300000000000001},"max":{"1min":1.702,"5min":1.895,"15min":2.516},"last":1.4970000000000001},{"interface":"front","average":{"1min":1.415,"5min":1.387,"15min":1.4359999999999999},"min":{"1min":1.0549999999999999,"5min":0.82699999999999996,"15min":0.80600000000000005},"max":{"1min":1.849,"5min":1.889,"15min":3.7410000000000001},"last":1.4570000000000001}]},{"osd":6,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4990000000000001,"5min":1.4690000000000001,"15min":1.472},"min":{"1min":0.90600000000000003,"5min":0.77800000000000002,"15min":0.77800000000000002},"max":{"1min":2.1110000000000002,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.5329999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.556,"15min":1.548},"min":{"1min":1.252,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.923,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.653}]},{"osd":7,"last update":"Sat Oct 14 16:59:44 2023","interfaces":[{"interface":"back","average":{"1min":1.4690000000000001,"5min":1.4610000000000001,"15min":1.498},"min":{"1min":0.90900000000000003,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.871},"last":1.585},{"interface":"front","average":{"1min":1.496,"5min":1.514,"15min":1.546},"min":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"max":{"1min":1.911,"5min":2.2330000000000001,"15min":2.907},"last":1.5600000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575088,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46752,"kb_used_data":1684,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94325088,"statfs":{"total":96636764160,"available":96588890112,"internally_reserved":0,"allocated":1724416,"data_stored":835679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.93100000000000005,"5min":0.88300000000000001,"15min":0.90000000000000002},"min":{"1min":0.40000000000000002,"5min":0.316,"15min":0.316},"max":{"1min":1.429,"5min":1.706,"15min":1.706},"last":1.323},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.98699999999999999,"15min":1.0369999999999999},"min":{"1min":0.44,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":1.6870000000000001,"5min":1.726,"15min":1.8080000000000001},"last":1.3999999999999999}]},{"osd":1,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.90100000000000002,"15min":0.96599999999999997},"min":{"1min":0.42799999999999999,"5min":0.372,"15min":0.372},"max":{"1min":1.2889999999999999,"5min":1.544,"15min":2.0419999999999998},"last":1.532},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.121,"15min":1.1659999999999999},"min":{"1min":0.45200000000000001,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.492,"5min":1.74,"15min":2.069},"last":1.909}]},{"osd":3,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.0529999999999999,"5min":1.1180000000000001,"15min":1.208},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.46500000000000002},"max":{"1min":1.7010000000000001,"5min":1.833,"15min":2.1349999999999998},"last":2.0339999999999998},{"interface":"front","average":{"1min":1.222,"5min":1.2070000000000001,"15min":1.2889999999999999},"min":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.46899999999999997},"max":{"1min":1.6559999999999999,"5min":1.7729999999999999,"15min":2.0459999999999998},"last":1.7010000000000001}]},{"osd":4,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.3320000000000001,"15min":1.403},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.7070000000000001,"5min":1.913,"15min":2.3159999999999998},"last":2.2040000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.2949999999999999,"15min":1.357},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.0310000000000001,"5min":2.0310000000000001,"15min":2.0750000000000002},"last":2.4089999999999998}]},{"osd":5,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.1839999999999999,"5min":1.2969999999999999,"15min":1.3759999999999999},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.59199999999999997},"max":{"1min":1.5980000000000001,"5min":2.173,"15min":2.173},"last":2.7890000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.4419999999999999,"15min":1.512},"min":{"1min":0.61199999999999999,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":1.827,"5min":1.889,"15min":2.4460000000000002},"last":2.5939999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.357,"15min":1.429},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":1.966,"5min":1.966,"15min":2.0289999999999999},"last":2.9079999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.4330000000000001,"15min":1.5069999999999999},"min":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":1.7230000000000001,"5min":2.0720000000000001,"15min":2.105},"last":2.9620000000000002}]},{"osd":7,"last update":"Sat Oct 14 16:59:58 2023","interfaces":[{"interface":"back","average":{"1min":1.387,"5min":1.4199999999999999,"15min":1.51},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.0939999999999999,"5min":2.0939999999999999,"15min":2.109},"last":2.6579999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.4870000000000001,"15min":1.5700000000000001},"min":{"1min":0.624,"5min":0.624,"15min":0.624},"max":{"1min":1.7110000000000001,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":2.5369999999999999}]}]},{"osd":1,"up_from":13,"seq":55834575088,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59240,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312600,"statfs":{"total":96636764160,"available":96576102400,"internally_reserved":0,"allocated":1732608,"data_stored":835803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.85799999999999998,"15min":0.90000000000000002},"min":{"1min":0.38700000000000001,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.2909999999999999,"5min":2.113,"15min":2.113},"last":0.441},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.93999999999999995,"15min":1.052},"min":{"1min":0.45100000000000001,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.6559999999999999,"5min":1.6819999999999999,"15min":2.0819999999999999},"last":0.42599999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":0.81299999999999994,"5min":0.91800000000000004,"15min":0.94699999999999995},"min":{"1min":0.52000000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.224,"5min":1.7509999999999999,"15min":1.758},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.96499999999999997,"5min":1.0880000000000001,"15min":1.1559999999999999},"min":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.4570000000000001,"5min":2.008,"15min":2.0150000000000001},"last":0.60899999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.103,"5min":1.1919999999999999,"15min":1.2809999999999999},"min":{"1min":0.57399999999999995,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.738,"5min":1.984,"15min":2.198},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.94299999999999995,"5min":1.0820000000000001,"15min":1.1759999999999999},"min":{"1min":0.496,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.4239999999999999,"5min":1.974,"15min":2.004},"last":0.495}]},{"osd":4,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.3300000000000001,"15min":1.399},"min":{"1min":0.628,"5min":0.628,"15min":0.628},"max":{"1min":1.9179999999999999,"5min":2.052,"15min":2.3050000000000002},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.29,"15min":1.3089999999999999},"min":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.53900000000000003},"max":{"1min":1.6970000000000001,"5min":1.9430000000000001,"15min":1.9430000000000001},"last":0.90300000000000002}]},{"osd":5,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.3300000000000001,"15min":1.361},"min":{"1min":0.85499999999999998,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.8759999999999999,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":0.754},{"interface":"front","average":{"1min":1.3839999999999999,"5min":1.3939999999999999,"15min":1.45},"min":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.76200000000000001},"max":{"1min":1.867,"5min":2.1480000000000001,"15min":2.302},"last":0.79100000000000004}]},{"osd":6,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.419,"15min":1.4870000000000001},"min":{"1min":0.82999999999999996,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.121,"5min":2.121,"15min":2.4169999999999998},"last":0.77300000000000002},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4319999999999999,"15min":1.4670000000000001},"min":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.7829999999999999,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":0.84799999999999998}]},{"osd":7,"last update":"Sat Oct 14 17:00:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3879999999999999,"5min":1.4730000000000001,"15min":1.548},"min":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.786,"5min":2.2410000000000001,"15min":2.411},"last":0.92100000000000004},{"interface":"front","average":{"1min":1.385,"5min":1.4730000000000001,"15min":1.5660000000000001},"min":{"1min":0.88500000000000001,"5min":0.81899999999999995,"15min":0.61599999999999999},"max":{"1min":2.0529999999999999,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":0.871}]}]},{"osd":0,"up_from":13,"seq":55834575087,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41792,"kb_used_data":2284,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330048,"statfs":{"total":96636764160,"available":96593969152,"internally_reserved":0,"allocated":2338816,"data_stored":1427771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.90700000000000003,"15min":0.89700000000000002},"min":{"1min":0.54100000000000004,"5min":0.309,"15min":0.309},"max":{"1min":1.635,"5min":3.355,"15min":3.355},"last":0.73899999999999999},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.97699999999999998,"15min":1.075},"min":{"1min":0.40200000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.3859999999999999,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":1.1459999999999999}]},{"osd":2,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":0.94599999999999995,"5min":1.054,"15min":1.0760000000000001},"min":{"1min":0.46800000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.4279999999999999,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":1.292},{"interface":"front","average":{"1min":1.0920000000000001,"5min":1.091,"15min":1.1779999999999999},"min":{"1min":0.47299999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.607,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":1.032}]},{"osd":3,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2330000000000001,"15min":1.268},"min":{"1min":0.50800000000000001,"5min":0.502,"15min":0.502},"max":{"1min":1.6839999999999999,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":1.5269999999999999},{"interface":"front","average":{"1min":1.2290000000000001,"5min":1.236,"15min":1.3360000000000001},"min":{"1min":0.57599999999999996,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.7350000000000001,"5min":2.581,"15min":2.593},"last":1.355}]},{"osd":4,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.268,"15min":1.278},"min":{"1min":0.75600000000000001,"5min":0.65700000000000003,"15min":0.65700000000000003},"max":{"1min":1.833,"5min":5.335,"15min":5.335},"last":1.117},{"interface":"front","average":{"1min":1.286,"5min":1.385,"15min":1.4139999999999999},"min":{"1min":0.73999999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.7,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.4319999999999999}]},{"osd":5,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.254,"15min":1.3069999999999999},"min":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":1.698,"5min":2.931,"15min":2.931},"last":1.0740000000000001},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3859999999999999,"15min":1.4850000000000001},"min":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.8160000000000001,"5min":3.306,"15min":3.306},"last":1.6779999999999999}]},{"osd":6,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.365,"5min":1.472,"15min":1.5649999999999999},"min":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":1.9039999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.3240000000000001,"5min":1.3500000000000001,"15min":1.393},"min":{"1min":0.79000000000000004,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.895,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.2150000000000001}]},{"osd":7,"last update":"Sat Oct 14 16:59:54 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.4930000000000001,"15min":1.538},"min":{"1min":0.79500000000000004,"5min":0.747,"15min":0.747},"max":{"1min":1.8919999999999999,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.6279999999999999},{"interface":"front","average":{"1min":1.411,"5min":1.4590000000000001,"15min":1.5109999999999999},"min":{"1min":0.93000000000000005,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.7589999999999999,"5min":3.698,"15min":3.698},"last":1.2629999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":16,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:00:32.732 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T17:00:32.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T17:00:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.008+0000 7ff70df4e640 1 Processor -- start 2023-10-14T17:00:33.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.008+0000 7ff70df4e640 1 -- start start 2023-10-14T17:00:33.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.010+0000 7ff70df4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff70815ac70 0x7ff7080cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.010+0000 7ff70df4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7080cc850 con 0x7ff7080b4160 2023-10-14T17:00:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.010+0000 7ff70df4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7080ccb30 con 0x7ff70806a4b0 2023-10-14T17:00:33.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.010+0000 7ff70df4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7080cce10 con 0x7ff70815ac70 2023-10-14T17:00:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.010+0000 7ff70cf4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff70815ac70 0x7ff7080cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.011+0000 7ff70cf4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff70815ac70 0x7ff7080cc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58116/0 (socket says 172.21.15.29:58116) 2023-10-14T17:00:33.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.011+0000 7ff70cf4c640 1 -- 172.21.15.29:0/1552303188 learned_addr learned my addr 172.21.15.29:0/1552303188 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.011+0000 7ff70cf4c640 1 -- 172.21.15.29:0/1552303188 >> v1:172.21.15.186:6789/0 conn(0x7ff70806a4b0 legacy=0x7ff70806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.011+0000 7ff70cf4c640 1 -- 172.21.15.29:0/1552303188 >> v1:172.21.15.29:6789/0 conn(0x7ff7080b4160 legacy=0x7ff708069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.011+0000 7ff70cf4c640 1 -- 172.21.15.29:0/1552303188 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080cd0f0 con 0x7ff70815ac70 2023-10-14T17:00:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.012+0000 7ff70cf4c640 1 --2- 172.21.15.29:0/1552303188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff70815ac70 0x7ff7080cc270 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009870 tx=0x7ff6fc02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9deee828e7de8465 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.013+0000 7ff7077fe640 1 -- 172.21.15.29:0/1552303188 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc030860 con 0x7ff70815ac70 2023-10-14T17:00:33.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.013+0000 7ff7077fe640 1 -- 172.21.15.29:0/1552303188 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6fc0309c0 con 0x7ff70815ac70 2023-10-14T17:00:33.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.013+0000 7ff7077fe640 1 -- 172.21.15.29:0/1552303188 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6fc030d50 con 0x7ff70815ac70 2023-10-14T17:00:33.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.013+0000 7ff70df4e640 1 -- 172.21.15.29:0/1552303188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff70815ac70 msgr2=0x7ff7080cc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.013+0000 7ff70df4e640 1 --2- 172.21.15.29:0/1552303188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff70815ac70 0x7ff7080cc270 secure :-1 s=READY pgs=1593 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc009870 tx=0x7ff6fc02b9a0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.014+0000 7ff70df4e640 1 -- 172.21.15.29:0/1552303188 shutdown_connections 2023-10-14T17:00:33.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.014+0000 7ff70df4e640 1 --2- 172.21.15.29:0/1552303188 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff70815ac70 0x7ff7080cc270 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.014+0000 7ff70df4e640 1 -- 172.21.15.29:0/1552303188 >> 172.21.15.29:0/1552303188 conn(0x7ff7080ce6a0 msgr2=0x7ff7080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:33.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.014+0000 7ff70df4e640 1 -- 172.21.15.29:0/1552303188 shutdown_connections 2023-10-14T17:00:33.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.014+0000 7ff70df4e640 1 -- 172.21.15.29:0/1552303188 wait complete. 2023-10-14T17:00:33.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.015+0000 7ff70df4e640 1 Processor -- start 2023-10-14T17:00:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.016+0000 7ff70df4e640 1 -- start start 2023-10-14T17:00:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff70806a4b0 0x7ff7080c8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 0x7ff7080c5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080c8640 con 0x7ff7080b4160 2023-10-14T17:00:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080adb40 con 0x7ff70815ac70 2023-10-14T17:00:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70df4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff7081e5440 con 0x7ff70806a4b0 2023-10-14T17:00:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58124/0 (socket says 172.21.15.29:58124) 2023-10-14T17:00:33.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 -- 172.21.15.29:0/4094389 learned_addr learned my addr 172.21.15.29:0/4094389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70d74d640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 0x7ff7080c5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 -- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff70806a4b0 msgr2=0x7ff7080c8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70cf4c640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff70806a4b0 0x7ff7080c8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff70806a4b0 0x7ff7080c8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 -- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 msgr2=0x7ff7080c5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 0x7ff7080c5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff707fff640 1 -- 172.21.15.29:0/4094389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc0094d0 con 0x7ff7080b4160 2023-10-14T17:00:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.017+0000 7ff70d74d640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 0x7ff7080c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff707fff640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c450 tx=0x7ff6f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f40052d0 con 0x7ff7080b4160 2023-10-14T17:00:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6f4005430 con 0x7ff7080b4160 2023-10-14T17:00:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6f4012870 con 0x7ff7080b4160 2023-10-14T17:00:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7081e5720 con 0x7ff7080b4160 2023-10-14T17:00:33.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.018+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7081e5ce0 con 0x7ff7080b4160 2023-10-14T17:00:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.019+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff70815a4d0 con 0x7ff7080b4160 2023-10-14T17:00:33.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.020+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff6f40129d0 con 0x7ff7080b4160 2023-10-14T17:00:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.021+0000 7ff7057fa640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 0x7ff6ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.021+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff6f40893c0 con 0x7ff7080b4160 2023-10-14T17:00:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.022+0000 7ff70cf4c640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 0x7ff6ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.022+0000 7ff70cf4c640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 0x7ff6ec06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc036000 tx=0x7ff6fc03b040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.026+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6f4054270 con 0x7ff7080b4160 2023-10-14T17:00:33.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.297+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff7080adcb0 con 0x7ff7080b4160 2023-10-14T17:00:33.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.302+0000 7ff7057fa640 1 -- 172.21.15.29:0/4094389 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v19) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff6f4057890 con 0x7ff7080b4160 2023-10-14T17:00:33.303 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T17:00:33.303 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T17:00:33.303 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T17:00:33.303 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T17:00:33.304 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T17:00:33.304 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T17:00:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 msgr2=0x7ff6ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 0x7ff6ec06d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff6fc036000 tx=0x7ff6fc03b040 comp rx=0 tx=0).stop 2023-10-14T17:00:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 msgr2=0x7ff7080c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 secure :-1 s=READY pgs=1594 cs=0 l=1 rev1=1 crypto rx=0x7ff6f400c450 tx=0x7ff6f400c910 comp rx=0 tx=0).stop 2023-10-14T17:00:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 shutdown_connections 2023-10-14T17:00:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff6ec06af90 0x7ff6ec06d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff70815ac70 0x7ff7080c5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff7080b4160 0x7ff7080c5770 unknown :-1 s=CLOSED pgs=1594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 --2- 172.21.15.29:0/4094389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff70806a4b0 0x7ff7080c8100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 >> 172.21.15.29:0/4094389 conn(0x7ff7080ce6a0 msgr2=0x7ff70815e350 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 shutdown_connections 2023-10-14T17:00:33.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.305+0000 7ff70df4e640 1 -- 172.21.15.29:0/4094389 wait complete. 2023-10-14T17:00:33.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:33.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.544+0000 7fd3ae6ca640 1 Processor -- start 2023-10-14T17:00:33.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.544+0000 7fd3ae6ca640 1 -- start start 2023-10-14T17:00:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.545+0000 7fd3ae6ca640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd3a815c9f0 0x7fd3a80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3ae6ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a8160a90 con 0x7fd3a80c9230 2023-10-14T17:00:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3ae6ca640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a80cd900 con 0x7fd3a815dde0 2023-10-14T17:00:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3ae6ca640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a80cdec0 con 0x7fd3a815c9f0 2023-10-14T17:00:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3acec7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd3a80c9230 0x7fd3a80709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35562/0 (socket says 172.21.15.29:35562) 2023-10-14T17:00:33.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3acec7640 1 -- 172.21.15.29:0/3489895551 learned_addr learned my addr 172.21.15.29:0/3489895551 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:33.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.546+0000 7fd3ad6c8640 1 --2- 172.21.15.29:0/3489895551 >> v2:172.21.15.186:3300/0 conn(0x7fd3a815c9f0 0x7fd3a80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 684858717 0 0) 0x7fd3a8160a90 con 0x7fd3a80c9230 2023-10-14T17:00:33.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd394003520 con 0x7fd3a80c9230 2023-10-14T17:00:33.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943161885 0 0) 0x7fd394003520 con 0x7fd3a80c9230 2023-10-14T17:00:33.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 >> v2:172.21.15.186:3300/0 conn(0x7fd3a815c9f0 msgr2=0x7fd3a80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 --2- 172.21.15.29:0/3489895551 >> v2:172.21.15.186:3300/0 conn(0x7fd3a815c9f0 0x7fd3a80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 >> v1:172.21.15.186:6789/0 conn(0x7fd3a815dde0 legacy=0x7fd3a80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80cdbe0 con 0x7fd3a80c9230 2023-10-14T17:00:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd398001670 con 0x7fd3a80c9230 2023-10-14T17:00:33.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 >> v1:172.21.15.29:6789/0 conn(0x7fd3a80c9230 legacy=0x7fd3a80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940039c0 0x7fd394003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3940047c0 0x7fd394004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd39400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39400c050 con 0x7fd394005180 2023-10-14T17:00:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39400c1c0 con 0x7fd3940039c0 2023-10-14T17:00:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.547+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39400c380 con 0x7fd3940047c0 2023-10-14T17:00:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd39400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3ad6c8640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940039c0 0x7fd394003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3acec7640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3940047c0 0x7fd394004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 -- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3940047c0 msgr2=0x7fd394004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3940047c0 0x7fd394004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 -- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940039c0 msgr2=0x7fd394003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3940039c0 0x7fd394003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.548+0000 7fd3adec9640 1 -- 172.21.15.29:0/3489895551 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394003520 con 0x7fd394005180 2023-10-14T17:00:33.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.549+0000 7fd3adec9640 1 -- 172.21.15.29:0/3489895551 reap_dead start 2023-10-14T17:00:33.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.549+0000 7fd3adec9640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd39400b980 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806fce0 tx=0x7fd3940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53d79b8a10ebf07d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a0003b70 con 0x7fd394005180 2023-10-14T17:00:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a0003cd0 con 0x7fd394005180 2023-10-14T17:00:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd39ffff640 1 -- 172.21.15.29:0/3489895551 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a00052f0 con 0x7fd394005180 2023-10-14T17:00:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 msgr2=0x7fd39400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd39400b980 secure :-1 s=READY pgs=1596 cs=0 l=1 rev1=1 crypto rx=0x7fd3a806fce0 tx=0x7fd3940057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/3489895551 shutdown_connections 2023-10-14T17:00:33.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/3489895551 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd39400b980 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.550+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/3489895551 >> 172.21.15.29:0/3489895551 conn(0x7fd3a80ce6a0 msgr2=0x7fd3a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:33.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.551+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/3489895551 shutdown_connections 2023-10-14T17:00:33.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.551+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/3489895551 wait complete. 2023-10-14T17:00:33.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.552+0000 7fd3ae6ca640 1 Processor -- start 2023-10-14T17:00:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.553+0000 7fd3ae6ca640 1 -- start start 2023-10-14T17:00:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.553+0000 7fd3ae6ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.553+0000 7fd3ae6ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a80c96f0 0x7fd3a80c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ae6ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a806b710 0x7fd3a806bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ae6ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80c8a00 con 0x7fd394005180 2023-10-14T17:00:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ae6ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80ca0b0 con 0x7fd3a806b710 2023-10-14T17:00:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ae6ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a806c0d0 con 0x7fd3a80c96f0 2023-10-14T17:00:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58134/0 (socket says 172.21.15.29:58134) 2023-10-14T17:00:33.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 -- 172.21.15.29:0/4141872061 learned_addr learned my addr 172.21.15.29:0/4141872061 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3adec9640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a806b710 0x7fd3a806bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 -- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a80c96f0 msgr2=0x7fd3a80c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a80c96f0 0x7fd3a80c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 -- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a806b710 msgr2=0x7fd3a806bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a806b710 0x7fd3a806bb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.554+0000 7fd3ad6c8640 1 -- 172.21.15.29:0/4141872061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80c7d80 con 0x7fd394005180 2023-10-14T17:00:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd3ad6c8640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40118e0 tx=0x7fd3a4011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a400faf0 con 0x7fd394005180 2023-10-14T17:00:33.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a400fc50 con 0x7fd394005180 2023-10-14T17:00:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a80c8060 con 0x7fd394005180 2023-10-14T17:00:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3a4014440 con 0x7fd394005180 2023-10-14T17:00:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.555+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a80c8620 con 0x7fd394005180 2023-10-14T17:00:33.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.556+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a804e9b0 con 0x7fd394005180 2023-10-14T17:00:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.557+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd3a401d400 con 0x7fd394005180 2023-10-14T17:00:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.558+0000 7fd39dffb640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 0x7fd38c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:33.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.558+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd3a408a100 con 0x7fd394005180 2023-10-14T17:00:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.559+0000 7fd3acec7640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 0x7fd38c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.559+0000 7fd3acec7640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 0x7fd38c06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd398001eb0 tx=0x7fd398000ef0 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:33.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.563+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3a4054fb0 con 0x7fd394005180 2023-10-14T17:00:33.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.798+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3a8150e90 con 0x7fd394005180 2023-10-14T17:00:33.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.799+0000 7fd39dffb640 1 -- 172.21.15.29:0/4141872061 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7fd3a40585d0 con 0x7fd394005180 2023-10-14T17:00:33.800 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:33.800 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:33.800 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.801+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 msgr2=0x7fd38c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.801+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 0x7fd38c06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd398001eb0 tx=0x7fd398000ef0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 msgr2=0x7fd3a80c7680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 secure :-1 s=READY pgs=1597 cs=0 l=1 rev1=1 crypto rx=0x7fd3a40118e0 tx=0x7fd3a4011da0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 shutdown_connections 2023-10-14T17:00:33.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fd38c06afc0 0x7fd38c06d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3a806b710 0x7fd3a806bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3a80c96f0 0x7fd3a80c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 --2- 172.21.15.29:0/4141872061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd394005180 0x7fd3a80c7680 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:33.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 >> 172.21.15.29:0/4141872061 conn(0x7fd3a80ce6a0 msgr2=0x7fd3a815fd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:33.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 shutdown_connections 2023-10-14T17:00:33.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:33.802+0000 7fd3ae6ca640 1 -- 172.21.15.29:0/4141872061 wait complete. 2023-10-14T17:00:34.226 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 66, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T17:00:27.165729+0000', 'modified': '2023-10-14T17:00:30.232420+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8541}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8541': {'gid': 8541, 'name': 'c', 'rank': 0, 'incarnation': 62, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6836/1532396905', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 1532396905}, {'type': 'v1', 'addr': '172.21.15.29:6836', 'nonce': 1532396905}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [17], 'metadata_pool': 16, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T17:00:34.227 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T17:00:34.227 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T17:00:34.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.466+0000 7f65cc846640 1 Processor -- start 2023-10-14T17:00:34.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.466+0000 7f65cc846640 1 -- start start 2023-10-14T17:00:34.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.467+0000 7f65cc846640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 0x7f65c8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65cc846640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f65c806a2e0 0x7f65c80c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65cc846640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65c806a6f0 con 0x7f65c815d150 2023-10-14T17:00:34.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65cc846640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65c815d530 con 0x7f65c806a2e0 2023-10-14T17:00:34.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65cc846640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80ac890 con 0x7f65c80ad350 2023-10-14T17:00:34.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65c659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f65c815d150 0x7f65c8160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35564/0 (socket says 172.21.15.29:35564) 2023-10-14T17:00:34.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65c659c640 1 -- 172.21.15.29:0/2020295157 learned_addr learned my addr 172.21.15.29:0/2020295157 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298984633 0 0) 0x7f65c806a6f0 con 0x7f65c815d150 2023-10-14T17:00:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65b4003520 con 0x7f65c815d150 2023-10-14T17:00:34.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.468+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/2020295157 >> v2:172.21.15.186:3300/0 conn(0x7f65c806a2e0 0x7f65c80c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 0x7f65c8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1041106977 0 0) 0x7f65b4003520 con 0x7f65c815d150 2023-10-14T17:00:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 msgr2=0x7f65c8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 0x7f65c8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 >> v2:172.21.15.186:3300/0 conn(0x7f65c806a2e0 msgr2=0x7f65c80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 --2- 172.21.15.29:0/2020295157 >> v2:172.21.15.186:3300/0 conn(0x7f65c806a2e0 0x7f65c80c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c8161650 con 0x7f65c815d150 2023-10-14T17:00:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f65b80019f0 con 0x7f65c815d150 2023-10-14T17:00:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 >> v1:172.21.15.29:6789/0 conn(0x7f65c815d150 legacy=0x7f65c8160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 0x7f65b4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65b4004d00 0x7f65b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c815d530 con 0x7f65b4003ef0 2023-10-14T17:00:34.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c80ac890 con 0x7f65b4004d00 2023-10-14T17:00:34.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65b400b9d0 con 0x7f65c80ad350 2023-10-14T17:00:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.469+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65b4004d00 0x7f65b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c5d9b640 1 -- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 msgr2=0x7f65b4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c80ad350 0x7f65b4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c5d9b640 1 -- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65b4004d00 msgr2=0x7f65b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65b4004d00 0x7f65b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.470+0000 7f65c5d9b640 1 -- 172.21.15.29:0/2020295157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65b4003520 con 0x7f65b4003ef0 2023-10-14T17:00:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.471+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65b40047c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba30 tx=0x7f65b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ece5c960107c1ae2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.471+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0032680 con 0x7f65b4003ef0 2023-10-14T17:00:34.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.471+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b00327e0 con 0x7f65b4003ef0 2023-10-14T17:00:34.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65c559a640 1 -- 172.21.15.29:0/2020295157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b0032b70 con 0x7f65b4003ef0 2023-10-14T17:00:34.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 -- 172.21.15.29:0/2020295157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 msgr2=0x7f65b40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 --2- 172.21.15.29:0/2020295157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65b40047c0 secure :-1 s=READY pgs=1599 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba30 tx=0x7f65b40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65c6d9d640 1 -- 172.21.15.29:0/2020295157 reap_dead start 2023-10-14T17:00:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 -- 172.21.15.29:0/2020295157 shutdown_connections 2023-10-14T17:00:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 -- 172.21.15.29:0/2020295157 >> 172.21.15.29:0/2020295157 conn(0x7f65c80ce9b0 msgr2=0x7f65c80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 -- 172.21.15.29:0/2020295157 shutdown_connections 2023-10-14T17:00:34.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.472+0000 7f65cc846640 1 -- 172.21.15.29:0/2020295157 wait complete. 2023-10-14T17:00:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.474+0000 7f65cc846640 1 Processor -- start 2023-10-14T17:00:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.474+0000 7f65cc846640 1 -- start start 2023-10-14T17:00:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 0x7f65c80b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c81f2530 0x7f65c80b2b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c81f29e0 con 0x7f65b4003ef0 2023-10-14T17:00:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65c81f2cc0 con 0x7f65c815fe70 2023-10-14T17:00:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65cc846640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65c81f2fa0 con 0x7f65c81f2530 2023-10-14T17:00:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.475+0000 7f65c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58154/0 (socket says 172.21.15.29:58154) 2023-10-14T17:00:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 -- 172.21.15.29:0/978341432 learned_addr learned my addr 172.21.15.29:0/978341432 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 0x7f65c80b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c81f2530 0x7f65c80b2b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 -- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c81f2530 msgr2=0x7f65c80b2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c81f2530 0x7f65c80b2b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 -- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 msgr2=0x7f65c80b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 0x7f65c80b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 -- 172.21.15.29:0/978341432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65c81f3280 con 0x7f65b4003ef0 2023-10-14T17:00:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 0x7f65c80b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.476+0000 7f65c659c640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f65b800caa0 tx=0x7f65b800cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.477+0000 7f65beffd640 1 -- 172.21.15.29:0/978341432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800a7f0 con 0x7f65b4003ef0 2023-10-14T17:00:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.477+0000 7f65beffd640 1 -- 172.21.15.29:0/978341432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65b800a950 con 0x7f65b4003ef0 2023-10-14T17:00:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.477+0000 7f65beffd640 1 -- 172.21.15.29:0/978341432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b800ad40 con 0x7f65b4003ef0 2023-10-14T17:00:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.477+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65c80b1000 con 0x7f65b4003ef0 2023-10-14T17:00:34.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.477+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c81545c0 con 0x7f65b4003ef0 2023-10-14T17:00:34.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.479+0000 7f65beffd640 1 -- 172.21.15.29:0/978341432 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65b80070d0 con 0x7f65b4003ef0 2023-10-14T17:00:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.481+0000 7f65beffd640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 0x7f65a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.481+0000 7f65beffd640 1 -- 172.21.15.29:0/978341432 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f65b8089110 con 0x7f65b4003ef0 2023-10-14T17:00:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.481+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 0x7f65a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.482+0000 7f65cf3dd740 1 Processor -- start 2023-10-14T17:00:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.482+0000 7f65cf3dd740 1 -- start start 2023-10-14T17:00:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.482+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 0x7f65a806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba00 tx=0x7f65b0030000 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e62ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e65ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e54b7a0 con 0x56233e62b3c0 2023-10-14T17:00:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e59d740 con 0x56233e657f80 2023-10-14T17:00:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56233e59ae30 con 0x56233e658360 2023-10-14T17:00:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58160/0 (socket says 172.21.15.29:58160) 2023-10-14T17:00:34.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65c6d9d640 1 -- 172.21.15.29:0/2853705634 learned_addr learned my addr 172.21.15.29:0/2853705634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.483+0000 7f65c659c640 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e62ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 -- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 msgr2=0x56233e65ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e65ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 -- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 msgr2=0x56233e62ae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e62ae80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 -- 172.21.15.29:0/2853705634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56233e55a910 con 0x56233e62b3c0 2023-10-14T17:00:34.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c659c640 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e62ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cb400 tx=0x7f65c0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277134b1d4b0bb90 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65a7fff640 1 -- 172.21.15.29:0/2853705634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0012c40 con 0x56233e62b3c0 2023-10-14T17:00:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65a7fff640 1 -- 172.21.15.29:0/2853705634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0012da0 con 0x56233e62b3c0 2023-10-14T17:00:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.484+0000 7f65a7fff640 1 -- 172.21.15.29:0/2853705634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0015690 con 0x56233e62b3c0 2023-10-14T17:00:34.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 -- 172.21.15.29:0/2853705634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 msgr2=0x56233e65b250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 secure :-1 s=READY pgs=1601 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cb400 tx=0x7f65c0004730 comp rx=0 tx=0).stop 2023-10-14T17:00:34.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 -- 172.21.15.29:0/2853705634 shutdown_connections 2023-10-14T17:00:34.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e62b3c0 0x56233e65b250 unknown :-1 s=CLOSED pgs=1601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e65ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/2853705634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e62ae80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 -- 172.21.15.29:0/2853705634 >> 172.21.15.29:0/2853705634 conn(0x56233e62d190 msgr2=0x56233e675d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 -- 172.21.15.29:0/2853705634 shutdown_connections 2023-10-14T17:00:34.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.485+0000 7f65cf3dd740 1 -- 172.21.15.29:0/2853705634 wait complete. 2023-10-14T17:00:34.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.486+0000 7f65cf3dd740 1 -- start start 2023-10-14T17:00:34.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.486+0000 7f65cf3dd740 1 Processor -- start 2023-10-14T17:00:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 0x56233e64c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 0x56233e66e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e596720 con 0x56233e658360 2023-10-14T17:00:34.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e534970 con 0x56233e670880 2023-10-14T17:00:34.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56233e534610 con 0x56233e657f80 2023-10-14T17:00:34.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65cf3dd740 10 client.0 _ll_register_callbacks cb 0x56233e5569d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:00:34.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65c5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65c5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58162/0 (socket says 172.21.15.29:58162) 2023-10-14T17:00:34.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.487+0000 7f65c5d9b640 1 -- 172.21.15.29:0/3620765910 learned_addr learned my addr 172.21.15.29:0/3620765910 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 0x56233e66e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c659c640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 0x56233e64c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 -- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 msgr2=0x56233e64c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 0x56233e64c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 -- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 msgr2=0x56233e66e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 0x56233e66e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 -- 172.21.15.29:0/3620765910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65c0009610 con 0x56233e658360 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 0x56233e66e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c659c640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 0x56233e64c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f65b000bb50 tx=0x7f65b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4745ba07485aaaba server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:00:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.488+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b004c070 con 0x56233e658360 2023-10-14T17:00:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f65b0030ab0 con 0x56233e658360 2023-10-14T17:00:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56233e5bd160 con 0x56233e658360 2023-10-14T17:00:34.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65b0040070 con 0x56233e658360 2023-10-14T17:00:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 10 client.8679 fetch_fsmap learned FSMap version 66 2023-10-14T17:00:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x56233e594da0 con 0x56233e658360 2023-10-14T17:00:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56233e5bd160 con 0x56233e658360 2023-10-14T17:00:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f65b002e070 con 0x56233e658360 2023-10-14T17:00:34.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 10 client.8679 fetch_fsmap finished waiting for FSMap version 66 2023-10-14T17:00:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 10 client.8679 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:00:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 20 client.8679 populate_metadata read hostname 'smithi029' 2023-10-14T17:00:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e65e020 0x56233e693030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 4 client.8679 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T17:00:34.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.489+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 --> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56233e568cc0 con 0x56233e65e020 2023-10-14T17:00:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.490+0000 7f65c659c640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e65e020 0x56233e693030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.490+0000 7f65c659c640 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e65e020 0x56233e693030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=71cd0186cd0c4300 server_cookie=7f2c70c6808ff792 in_seq=0 out_seq=0 2023-10-14T17:00:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.490+0000 7f65a57fa640 10 client.8679 ms_handle_connect on v2:172.21.15.29:6834/1532396905 2023-10-14T17:00:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65a57fa640 1 -- 172.21.15.29:0/3620765910 <== mds.0 v2:172.21.15.29:6834/1532396905 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56233e568cc0 con 0x56233e65e020 2023-10-14T17:00:34.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65a57fa640 10 client.8679 handle_command_reply: tid=0 2023-10-14T17:00:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 client.8679 shutdown 2023-10-14T17:00:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 msgr2=0x56233e670340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 secure :-1 s=READY pgs=1602 cs=0 l=1 rev1=1 crypto rx=0x7f65b000bb50 tx=0x7f65b0004730 comp rx=0 tx=0).stop 2023-10-14T17:00:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 shutdown_connections 2023-10-14T17:00:34.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e65e020 0x56233e693030 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e670880 0x56233e66e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e658360 0x56233e670340 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3620765910 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e657f80 0x56233e64c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65a57fa640 0 client.8679 ms_handle_reset on v2:172.21.15.29:6834/1532396905 2023-10-14T17:00:34.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 >> 172.21.15.29:0/3620765910 conn(0x56233e62b3c0 msgr2=0x56233e65ad00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 shutdown_connections 2023-10-14T17:00:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.491+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3620765910 wait complete. 2023-10-14T17:00:34.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.492+0000 7f65cf3dd740 20 client.8679 trim_cache size 0 max 16384 2023-10-14T17:00:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.509+0000 7f65cf3dd740 1 Processor -- start 2023-10-14T17:00:34.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.509+0000 7f65cf3dd740 1 -- start start 2023-10-14T17:00:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e64b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e62eda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e66fb80 con 0x56233e670880 2023-10-14T17:00:34.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e2faf20 con 0x56233e657f80 2023-10-14T17:00:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56233e66fa00 con 0x56233e658360 2023-10-14T17:00:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58176/0 (socket says 172.21.15.29:58176) 2023-10-14T17:00:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.510+0000 7f65c6d9d640 1 -- 172.21.15.29:0/3992572747 learned_addr learned my addr 172.21.15.29:0/3992572747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c659c640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e64b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e62eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 -- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 msgr2=0x56233e62eda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e62eda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 -- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 msgr2=0x56233e64b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e64b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 -- 172.21.15.29:0/3992572747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56233e533b80 con 0x56233e670880 2023-10-14T17:00:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e62eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65c6d9d640 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cf870 tx=0x7f65c0015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2230cb38da1ca3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.511+0000 7f65bcff9640 1 -- 172.21.15.29:0/3992572747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0012c40 con 0x56233e670880 2023-10-14T17:00:34.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65bcff9640 1 -- 172.21.15.29:0/3992572747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0024070 con 0x56233e670880 2023-10-14T17:00:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65bcff9640 1 -- 172.21.15.29:0/3992572747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65c0015670 con 0x56233e670880 2023-10-14T17:00:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3992572747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 msgr2=0x56233e68e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 secure :-1 s=READY pgs=1603 cs=0 l=1 rev1=1 crypto rx=0x7f65c80cf870 tx=0x7f65c0015000 comp rx=0 tx=0).stop 2023-10-14T17:00:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3992572747 shutdown_connections 2023-10-14T17:00:34.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e670880 0x56233e68e6a0 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e658360 0x56233e62eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/3992572747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e657f80 0x56233e64b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3992572747 >> 172.21.15.29:0/3992572747 conn(0x56233e62b3c0 msgr2=0x56233e66e620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3992572747 shutdown_connections 2023-10-14T17:00:34.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.512+0000 7f65cf3dd740 1 -- 172.21.15.29:0/3992572747 wait complete. 2023-10-14T17:00:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.513+0000 7f65cf3dd740 1 -- start start 2023-10-14T17:00:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.513+0000 7f65cf3dd740 1 Processor -- start 2023-10-14T17:00:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e658360 0x56233e6de050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e670880 0x56233e66e200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e66bc80 con 0x56233e657f80 2023-10-14T17:00:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56233e656c30 con 0x56233e658360 2023-10-14T17:00:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56233e65c500 con 0x56233e670880 2023-10-14T17:00:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65cf3dd740 10 client.0 _ll_register_callbacks cb 0x7f65b40036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:00:34.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.514+0000 7f65c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58178/0 (socket says 172.21.15.29:58178) 2023-10-14T17:00:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 -- 172.21.15.29:0/1181038803 learned_addr learned my addr 172.21.15.29:0/1181038803 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 -- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e670880 msgr2=0x56233e66e200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e670880 0x56233e66e200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 -- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e658360 msgr2=0x56233e6de050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e658360 0x56233e6de050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 -- 172.21.15.29:0/1181038803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56233e55a910 con 0x56233e657f80 2023-10-14T17:00:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65c659c640 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f65b80013d0 tx=0x7f65b805e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a79c3adbe54a50b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65a57fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:00:34.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65b8087540 con 0x56233e657f80 2023-10-14T17:00:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f65b80876a0 con 0x56233e657f80 2023-10-14T17:00:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.515+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65c0012da0 con 0x56233e657f80 2023-10-14T17:00:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=66) v2 ==== 24+0+0 (secure 0 0 0) 0x7f65b805fb80 con 0x56233e657f80 2023-10-14T17:00:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 10 client.8685 fetch_fsmap learned FSMap version 66 2023-10-14T17:00:34.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=66,monmap=2+}) v3 -- 0x56233e559170 con 0x56233e657f80 2023-10-14T17:00:34.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65c0012da0 con 0x56233e657f80 2023-10-14T17:00:34.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 66) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f65b805cba0 con 0x56233e657f80 2023-10-14T17:00:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 10 client.8685 fetch_fsmap finished waiting for FSMap version 66 2023-10-14T17:00:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 10 client.8685 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:00:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 20 client.8685 populate_metadata read hostname 'smithi029' 2023-10-14T17:00:34.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e6db020 0x56233e66e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 4 client.8685 mds_command: new command op to 8541 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T17:00:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 --> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56233e58f700 con 0x56233e6db020 2023-10-14T17:00:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.516+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e6db020 0x56233e66e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65c5d9b640 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e6db020 0x56233e66e740 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15624f15fce375f2 server_cookie=c8397836c924011f in_seq=0 out_seq=0 2023-10-14T17:00:34.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65a57fa640 10 client.8685 ms_handle_connect on v2:172.21.15.29:6834/1532396905 2023-10-14T17:00:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65a57fa640 1 -- 172.21.15.29:0/1181038803 <== mds.0 v2:172.21.15.29:6834/1532396905 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56233e58f700 con 0x56233e6db020 2023-10-14T17:00:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65a57fa640 10 client.8685 handle_command_reply: tid=0 2023-10-14T17:00:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65cf3dd740 1 client.8685 shutdown 2023-10-14T17:00:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 msgr2=0x56233e65baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.517+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 secure :-1 s=READY pgs=1604 cs=0 l=1 rev1=1 crypto rx=0x7f65b80013d0 tx=0x7f65b805e040 comp rx=0 tx=0).stop 2023-10-14T17:00:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 shutdown_connections 2023-10-14T17:00:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:6834/1532396905,v1:172.21.15.29:6836/1532396905] conn(0x56233e6db020 0x56233e66e740 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56233e670880 0x56233e66e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56233e658360 0x56233e6de050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 --2- 172.21.15.29:0/1181038803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56233e657f80 0x56233e65baf0 unknown :-1 s=CLOSED pgs=1604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65a57fa640 0 client.8685 ms_handle_reset on v2:172.21.15.29:6834/1532396905 2023-10-14T17:00:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 >> 172.21.15.29:0/1181038803 conn(0x56233e62b3c0 msgr2=0x56233e671900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 shutdown_connections 2023-10-14T17:00:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 1 -- 172.21.15.29:0/1181038803 wait complete. 2023-10-14T17:00:34.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.518+0000 7f65cf3dd740 20 client.8685 trim_cache size 0 max 16384 2023-10-14T17:00:34.535 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:00:34.535 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T17:00:34.535 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T17:00:34.535 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8541, 2023-10-14T17:00:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T17:00:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T17:00:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T17:00:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8198214979999996, 2023-10-14T17:00:34.536 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 66, 2023-10-14T17:00:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 140, 2023-10-14T17:00:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 140, 2023-10-14T17:00:34.537 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.163773159 2023-10-14T17:00:34.537 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:00:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 msgr2=0x7f65a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 0x7f65a806d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f65b000ba00 tx=0x7f65b0030000 comp rx=0 tx=0).stop 2023-10-14T17:00:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 msgr2=0x7f65c80b0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 secure :-1 s=READY pgs=1600 cs=0 l=1 rev1=1 crypto rx=0x7f65b800caa0 tx=0x7f65b800cf60 comp rx=0 tx=0).stop 2023-10-14T17:00:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 shutdown_connections 2023-10-14T17:00:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f65a806af90 0x7f65a806d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65c81f2530 0x7f65c80b2b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65c815fe70 0x7f65c80b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 --2- 172.21.15.29:0/978341432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65b4003ef0 0x7f65c80b0ac0 unknown :-1 s=CLOSED pgs=1600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 >> 172.21.15.29:0/978341432 conn(0x7f65c80ce9b0 msgr2=0x7f65c8158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 shutdown_connections 2023-10-14T17:00:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.519+0000 7f65cc846640 1 -- 172.21.15.29:0/978341432 wait complete. 2023-10-14T17:00:34.542 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T17:00:34.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.768+0000 7f0dbf6a9640 1 Processor -- start 2023-10-14T17:00:34.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.768+0000 7f0dbf6a9640 1 -- start start 2023-10-14T17:00:34.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbf6a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0db80cfc10 0x7f0db80d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbf6a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80b3940 con 0x7f0db80b3e70 2023-10-14T17:00:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbf6a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db80b45e0 con 0x7f0db80b3070 2023-10-14T17:00:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbf6a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db80c7880 con 0x7f0db80cfc10 2023-10-14T17:00:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbdc46640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0db80b3e70 0x7f0db80d1040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35566/0 (socket says 172.21.15.29:35566) 2023-10-14T17:00:34.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbdc46640 1 -- 172.21.15.29:0/1783072058 learned_addr learned my addr 172.21.15.29:0/1783072058 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:34.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.770+0000 7f0dbcc44640 1 --2- 172.21.15.29:0/1783072058 >> v2:172.21.15.186:3300/0 conn(0x7f0db80cfc10 0x7f0db80d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1198668361 0 0) 0x7f0db80b3940 con 0x7f0db80b3e70 2023-10-14T17:00:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0da4003520 con 0x7f0db80b3e70 2023-10-14T17:00:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3485257564 0 0) 0x7f0da4003520 con 0x7f0db80b3e70 2023-10-14T17:00:34.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 >> v2:172.21.15.186:3300/0 conn(0x7f0db80cfc10 msgr2=0x7f0db80d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 --2- 172.21.15.29:0/1783072058 >> v2:172.21.15.186:3300/0 conn(0x7f0db80cfc10 0x7f0db80d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 >> v1:172.21.15.186:6789/0 conn(0x7f0db80b3070 legacy=0x7f0db80cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80b3b90 con 0x7f0db80b3e70 2023-10-14T17:00:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0db00019f0 con 0x7f0db80b3e70 2023-10-14T17:00:34.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 >> v1:172.21.15.29:6789/0 conn(0x7f0db80b3e70 legacy=0x7f0db80d1040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4003970 0x7f0da4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40047c0 0x7f0da4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 0x7f0da400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80c7880 con 0x7f0da4005180 2023-10-14T17:00:34.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da400c020 con 0x7f0da4003970 2023-10-14T17:00:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.771+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da400c1e0 con 0x7f0da40047c0 2023-10-14T17:00:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 0x7f0da400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbd445640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4003970 0x7f0da4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbcc44640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40047c0 0x7f0da4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 -- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40047c0 msgr2=0x7f0da4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40047c0 0x7f0da4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 -- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4003970 msgr2=0x7f0da4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4003970 0x7f0da4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.772+0000 7f0dbdc46640 1 -- 172.21.15.29:0/1783072058 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da4003520 con 0x7f0da4005180 2023-10-14T17:00:34.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.773+0000 7f0dbdc46640 1 -- 172.21.15.29:0/1783072058 reap_dead start 2023-10-14T17:00:34.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.773+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 0x7f0da400b980 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a830 tx=0x7f0da40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c56f768a9be47e8b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db00072d0 con 0x7f0da4005180 2023-10-14T17:00:34.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0db002f070 con 0x7f0da4005180 2023-10-14T17:00:34.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0daffff640 1 -- 172.21.15.29:0/1783072058 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0db0007650 con 0x7f0da4005180 2023-10-14T17:00:34.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 msgr2=0x7f0da400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 0x7f0da400b980 secure :-1 s=READY pgs=1606 cs=0 l=1 rev1=1 crypto rx=0x7f0db000a830 tx=0x7f0da40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1783072058 shutdown_connections 2023-10-14T17:00:34.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1783072058 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4005180 0x7f0da400b980 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1783072058 >> 172.21.15.29:0/1783072058 conn(0x7f0db80d2d20 msgr2=0x7f0db80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:34.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1783072058 shutdown_connections 2023-10-14T17:00:34.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.774+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1783072058 wait complete. 2023-10-14T17:00:34.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.776+0000 7f0dbf6a9640 1 Processor -- start 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.776+0000 7f0dbf6a9640 1 -- start start 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 0x7f0db81e8a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 0x7f0db80b7390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80b8540 con 0x7f0db80b8960 2023-10-14T17:00:34.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80b5f70 con 0x7f0db815f630 2023-10-14T17:00:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.777+0000 7f0dbf6a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db80b59b0 con 0x7f0da4005180 2023-10-14T17:00:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58202/0 (socket says 172.21.15.29:58202) 2023-10-14T17:00:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 -- 172.21.15.29:0/1273057771 learned_addr learned my addr 172.21.15.29:0/1273057771 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:34.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 0x7f0db80b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 -- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 msgr2=0x7f0db81e8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbd445640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 0x7f0db81e8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 0x7f0db81e8a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 -- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 msgr2=0x7f0db80b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:34.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 0x7f0db80b7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 -- 172.21.15.29:0/1273057771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db80b5c90 con 0x7f0db80b8960 2023-10-14T17:00:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbd445640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 0x7f0db81e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbdc46640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 0x7f0db80b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.778+0000 7f0dbcc44640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0da8010890 tx=0x7f0da8010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.779+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da8009610 con 0x7f0db80b8960 2023-10-14T17:00:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.779+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da800eb00 con 0x7f0db80b8960 2023-10-14T17:00:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.779+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0da800ee90 con 0x7f0db80b8960 2023-10-14T17:00:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.779+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db80b8160 con 0x7f0db80b8960 2023-10-14T17:00:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.779+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db815fae0 con 0x7f0db80b8960 2023-10-14T17:00:34.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.780+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db804e160 con 0x7f0db80b8960 2023-10-14T17:00:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.781+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0da801c400 con 0x7f0db80b8960 2023-10-14T17:00:34.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.782+0000 7f0dadffb640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 0x7f0d9c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.782+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f0da8089d00 con 0x7f0db80b8960 2023-10-14T17:00:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.783+0000 7f0dbd445640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 0x7f0d9c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.783+0000 7f0dbd445640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 0x7f0d9c06d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b24f0 tx=0x7f0db4009040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:34.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:34.786+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0da8054c40 con 0x7f0db80b8960 2023-10-14T17:00:35.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.024+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f0db80d2410 con 0x7f0db80b8960 2023-10-14T17:00:35.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.024+0000 7f0dadffb640 1 -- 172.21.15.29:0/1273057771 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v66) v1 ==== 53+0+73 (secure 0 0 0) 0x7f0da8058260 con 0x7f0db80b8960 2023-10-14T17:00:35.025 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T17:00:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 msgr2=0x7f0d9c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 0x7f0d9c06d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0db80b24f0 tx=0x7f0db4009040 comp rx=0 tx=0).stop 2023-10-14T17:00:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 msgr2=0x7f0db815fe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 secure :-1 s=READY pgs=1607 cs=0 l=1 rev1=1 crypto rx=0x7f0da8010890 tx=0x7f0da8010d50 comp rx=0 tx=0).stop 2023-10-14T17:00:35.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 shutdown_connections 2023-10-14T17:00:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0d9c06afc0 0x7f0d9c06d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0db815f630 0x7f0db80b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0db80b8960 0x7f0db815fe30 unknown :-1 s=CLOSED pgs=1607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:35.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 --2- 172.21.15.29:0/1273057771 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da4005180 0x7f0db81e8a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 >> 172.21.15.29:0/1273057771 conn(0x7f0db80d2d20 msgr2=0x7f0db80cd9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 shutdown_connections 2023-10-14T17:00:35.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:35.027+0000 7f0dbf6a9640 1 -- 172.21.15.29:0/1273057771 wait complete. 2023-10-14T17:00:35.046 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T17:00:35.047 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T17:00:35.047 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T17:00:35.048 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T17:00:35.048 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:35.048 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T17:00:35.049 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T17:00:35.049 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T17:00:35.049 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:00:35.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.050 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T17:00:35.105 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T17:00:35.105 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T17:00:35.106 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T17:00:35.106 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.106 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T17:00:35.106 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.106 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:00:35.107 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:00:35.107 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T17:00:35.107 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:00:35.107 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:00:35.108 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T17:00:35.108 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T17:00:35.108 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T17:00:35.108 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T17:00:35.108 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41141sec preferred_lft 41141sec 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:00:35.109 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T17:00:35.110 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T17:00:35.110 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:00:35.110 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T17:00:35.110 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.110 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T17:00:35.111 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.111 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:00:35.111 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T17:00:35.111 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T17:00:35.111 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.112 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:00:35.112 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T17:00:35.112 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T17:00:35.112 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:00:35.113 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.114 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T17:00:35.213 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T17:00:35.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.218 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T17:00:35.280 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T17:00:35.280 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T17:00:35.281 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T17:00:35.281 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T17:00:35.281 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T17:00:35.281 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.282 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T17:00:35.282 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T17:00:35.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T17:00:35.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T17:00:35.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T17:00:35.284 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T17:00:35.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.485 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T17:00:35.546 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T17:00:35.546 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T17:00:35.546 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:00:35.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.547 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T17:00:35.650 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:00:35.651 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:35.714 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:00:35.714 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:00:35.715 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:35.773 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T17:00:35.774 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.774 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:00:35.920 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:00:35.921 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:00:35.921 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:00:35.943 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:00:35.943 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:00:35.944 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:00:35.944 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:00:35.944 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:00:35.969 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:00:35.969 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:00:35.969 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:00:35.969 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:00:35.970 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:00:35.970 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T17:00:35.970 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T17:00:35.970 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T17:00:35.973 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T17:00:35.974 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T17:00:35.974 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T17:00:35.974 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T17:00:35.974 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T17:00:35.978 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T17:00:35.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:35.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:36.076 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:36.076 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T17:00:36.152 INFO:teuthology.orchestra.run.smithi029.stdout:client8694 2023-10-14T17:00:36.152 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:36.153 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:36.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T17:00:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.440+0000 7f3219fa0640 1 Processor -- start 2023-10-14T17:00:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.440+0000 7f3219fa0640 1 -- start start 2023-10-14T17:00:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.441+0000 7f3219fa0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.441+0000 7f3219fa0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f32140cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.441+0000 7f3219fa0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f321415d150 con 0x7f32140d0be0 2023-10-14T17:00:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.441+0000 7f3219fa0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32140d1180 con 0x7f321415cd70 2023-10-14T17:00:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.441+0000 7f3219fa0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32140d1460 con 0x7f32140b8480 2023-10-14T17:00:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.442+0000 7f3218f9e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.442+0000 7f3218f9e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:49884/0 (socket says 172.21.15.29:49884) 2023-10-14T17:00:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.442+0000 7f3218f9e640 1 -- 172.21.15.29:0/3525694946 learned_addr learned my addr 172.21.15.29:0/3525694946 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.442+0000 7f3213fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f32140cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.443+0000 7f3213fff640 1 -- 172.21.15.29:0/3525694946 >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 msgr2=0x7f32140d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.443+0000 7f3213fff640 1 --2- 172.21.15.29:0/3525694946 >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.443+0000 7f3213fff640 1 -- 172.21.15.29:0/3525694946 >> v1:172.21.15.186:6789/0 conn(0x7f32140d0be0 legacy=0x7f32140b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.443+0000 7f3213fff640 1 -- 172.21.15.29:0/3525694946 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32140d1740 con 0x7f32140b8480 2023-10-14T17:00:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.443+0000 7f3218f9e640 1 --2- 172.21.15.29:0/3525694946 >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:00:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.445+0000 7f3213fff640 1 --2- 172.21.15.29:0/3525694946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f32140cd7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32000097a0 tx=0x7f320002b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=38a4aea224f85fe0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.446+0000 7f32137fe640 1 -- 172.21.15.29:0/3525694946 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f320002d040 con 0x7f32140b8480 2023-10-14T17:00:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.446+0000 7f32137fe640 1 -- 172.21.15.29:0/3525694946 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3200031810 con 0x7f32140b8480 2023-10-14T17:00:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.446+0000 7f32137fe640 1 -- 172.21.15.29:0/3525694946 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200031ba0 con 0x7f32140b8480 2023-10-14T17:00:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.446+0000 7f3219fa0640 1 -- 172.21.15.29:0/3525694946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 msgr2=0x7f32140cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.446+0000 7f3219fa0640 1 --2- 172.21.15.29:0/3525694946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f32140cd7c0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32000097a0 tx=0x7f320002b8b0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 -- 172.21.15.29:0/3525694946 shutdown_connections 2023-10-14T17:00:36.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 --2- 172.21.15.29:0/3525694946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f32140cd7c0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 --2- 172.21.15.29:0/3525694946 >> v2:172.21.15.186:3300/0 conn(0x7f321415cd70 0x7f32140d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 -- 172.21.15.29:0/3525694946 >> 172.21.15.29:0/3525694946 conn(0x7f32140cf570 msgr2=0x7f32140cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 -- 172.21.15.29:0/3525694946 shutdown_connections 2023-10-14T17:00:36.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.447+0000 7f3219fa0640 1 -- 172.21.15.29:0/3525694946 wait complete. 2023-10-14T17:00:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.448+0000 7f3219fa0640 1 Processor -- start 2023-10-14T17:00:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.449+0000 7f3219fa0640 1 -- start start 2023-10-14T17:00:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f3214156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321415cd70 0x7f3214151900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3214157a80 con 0x7f32140d0be0 2023-10-14T17:00:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32140cb740 con 0x7f321415cd70 2023-10-14T17:00:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3219fa0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32140cb180 con 0x7f32140b8480 2023-10-14T17:00:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3213fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3213fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58208/0 (socket says 172.21.15.29:58208) 2023-10-14T17:00:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3213fff640 1 -- 172.21.15.29:0/2780156115 learned_addr learned my addr 172.21.15.29:0/2780156115 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f321979f640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321415cd70 0x7f3214151900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.450+0000 7f3218f9e640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f3214156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 -- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 msgr2=0x7f3214156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f3214156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 -- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321415cd70 msgr2=0x7f3214151900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321415cd70 0x7f3214151900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 -- 172.21.15.29:0/2780156115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32080094d0 con 0x7f32140d0be0 2023-10-14T17:00:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3213fff640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f3200000c00 tx=0x7f32000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200006e10 con 0x7f32140d0be0 2023-10-14T17:00:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f320003ad60 con 0x7f32140d0be0 2023-10-14T17:00:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3200009400 con 0x7f32140d0be0 2023-10-14T17:00:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.451+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3200044630 con 0x7f32140d0be0 2023-10-14T17:00:36.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.452+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3214157cb0 con 0x7f32140d0be0 2023-10-14T17:00:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.453+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f321404e9b0 con 0x7f32140d0be0 2023-10-14T17:00:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.454+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f320004d450 con 0x7f32140d0be0 2023-10-14T17:00:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.455+0000 7f32117fa640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 0x7f31f806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.455+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f32000abb90 con 0x7f32140d0be0 2023-10-14T17:00:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.455+0000 7f3218f9e640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 0x7f31f806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.456+0000 7f3218f9e640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 0x7f31f806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3208009cf0 tx=0x7f3208008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.460+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32000768b0 con 0x7f32140d0be0 2023-10-14T17:00:36.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.681+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 --> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f32140d1d00 con 0x7f31f806af90 2023-10-14T17:00:36.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.684+0000 7f32117fa640 1 -- 172.21.15.29:0/2780156115 <== mgr.8298 v2:172.21.15.29:6832/3738876524 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3208009940 con 0x7f31f806af90 2023-10-14T17:00:36.686 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:00:36.686 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:00:36.687 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T17:00:36.687 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:00:36.687 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:00:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 msgr2=0x7f31f806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 0x7f31f806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3208009cf0 tx=0x7f3208008040 comp rx=0 tx=0).stop 2023-10-14T17:00:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 msgr2=0x7f32140ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 secure :-1 s=READY pgs=1609 cs=0 l=1 rev1=1 crypto rx=0x7f3200000c00 tx=0x7f32000046c0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 shutdown_connections 2023-10-14T17:00:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f31f806af90 0x7f31f806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f321415cd70 0x7f3214151900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32140d0be0 0x7f32140ca930 unknown :-1 s=CLOSED pgs=1609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 --2- 172.21.15.29:0/2780156115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32140b8480 0x7f3214156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 >> 172.21.15.29:0/2780156115 conn(0x7f32140cf570 msgr2=0x7f32140b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 shutdown_connections 2023-10-14T17:00:36.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.687+0000 7f3219fa0640 1 -- 172.21.15.29:0/2780156115 wait complete. 2023-10-14T17:00:36.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T17:00:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.941+0000 7f97aed5e640 1 Processor -- start 2023-10-14T17:00:36.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.941+0000 7f97aed5e640 1 -- start start 2023-10-14T17:00:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.943+0000 7f97aed5e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97a80c9060 0x7f97a806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.943+0000 7f97aed5e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f97a806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.943+0000 7f97aed5e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80ca560 con 0x7f97a806c9e0 2023-10-14T17:00:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.943+0000 7f97aed5e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a80c9950 con 0x7f97a80c9060 2023-10-14T17:00:36.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.943+0000 7f97aed5e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c9c30 con 0x7f97a80c9440 2023-10-14T17:00:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97ad55b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f97a806c9e0 0x7f97a8164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35582/0 (socket says 172.21.15.29:35582) 2023-10-14T17:00:36.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97ad55b640 1 -- 172.21.15.29:0/781602015 learned_addr learned my addr 172.21.15.29:0/781602015 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97add5c640 1 --2- 172.21.15.29:0/781602015 >> v2:172.21.15.186:3300/0 conn(0x7f97a80c9060 0x7f97a806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97ae55d640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f97a806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148982175 0 0) 0x7f97a80ca560 con 0x7f97a806c9e0 2023-10-14T17:00:36.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9794003520 con 0x7f97a806c9e0 2023-10-14T17:00:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.944+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 40168586 0 0) 0x7f9794003520 con 0x7f97a806c9e0 2023-10-14T17:00:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 msgr2=0x7f97a806f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f97a806f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 >> v2:172.21.15.186:3300/0 conn(0x7f97a80c9060 msgr2=0x7f97a806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:36.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 --2- 172.21.15.29:0/781602015 >> v2:172.21.15.186:3300/0 conn(0x7f97a80c9060 0x7f97a806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80c9f10 con 0x7f97a806c9e0 2023-10-14T17:00:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9798001670 con 0x7f97a806c9e0 2023-10-14T17:00:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 >> v1:172.21.15.29:6789/0 conn(0x7f97a806c9e0 legacy=0x7f97a8164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f9794003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794003ef0 0x7f97940047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9794004d00 0x7f9794005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c9950 con 0x7f9794004d00 2023-10-14T17:00:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c9c30 con 0x7f9794003ef0 2023-10-14T17:00:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979400b980 con 0x7f97a80c9440 2023-10-14T17:00:36.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97ae55d640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9794004d00 0x7f9794005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97ad55b640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794003ef0 0x7f97940047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.945+0000 7f97add5c640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f9794003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97ae55d640 1 -- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 msgr2=0x7f9794003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97ae55d640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f9794003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97ae55d640 1 -- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794003ef0 msgr2=0x7f97940047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97ae55d640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794003ef0 0x7f97940047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97ae55d640 1 -- 172.21.15.29:0/781602015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9794003520 con 0x7f9794004d00 2023-10-14T17:00:36.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.946+0000 7f97add5c640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80c9440 0x7f9794003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.947+0000 7f97ae55d640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9794004d00 0x7f9794005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009f90 tx=0x7f97940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a6596137d42886 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.947+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a002cbc0 con 0x7f9794004d00 2023-10-14T17:00:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.947+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a000f040 con 0x7f9794004d00 2023-10-14T17:00:36.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.947+0000 7f97acd5a640 1 -- 172.21.15.29:0/781602015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0034440 con 0x7f9794004d00 2023-10-14T17:00:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 -- 172.21.15.29:0/781602015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9794004d00 msgr2=0x7f9794005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 --2- 172.21.15.29:0/781602015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9794004d00 0x7f9794005180 secure :-1 s=READY pgs=1611 cs=0 l=1 rev1=1 crypto rx=0x7f97a0009f90 tx=0x7f97940057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97ae55d640 1 -- 172.21.15.29:0/781602015 reap_dead start 2023-10-14T17:00:36.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 -- 172.21.15.29:0/781602015 shutdown_connections 2023-10-14T17:00:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 -- 172.21.15.29:0/781602015 >> 172.21.15.29:0/781602015 conn(0x7f97a80ce6a0 msgr2=0x7f97a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 -- 172.21.15.29:0/781602015 shutdown_connections 2023-10-14T17:00:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.948+0000 7f97aed5e640 1 -- 172.21.15.29:0/781602015 wait complete. 2023-10-14T17:00:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.950+0000 7f97aed5e640 1 Processor -- start 2023-10-14T17:00:36.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.950+0000 7f97aed5e640 1 -- start start 2023-10-14T17:00:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794004d00 0x7f97a8161560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80bf840 0x7f97a815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c8300 con 0x7f97a80c7940 2023-10-14T17:00:36.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80bfdd0 con 0x7f9794004d00 2023-10-14T17:00:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97aed5e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a80c00b0 con 0x7f97a80bf840 2023-10-14T17:00:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97ae55d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97ae55d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58224/0 (socket says 172.21.15.29:58224) 2023-10-14T17:00:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.951+0000 7f97ae55d640 1 -- 172.21.15.29:0/1801159533 learned_addr learned my addr 172.21.15.29:0/1801159533 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:36.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97add5c640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794004d00 0x7f97a8161560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ad55b640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80bf840 0x7f97a815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 -- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80bf840 msgr2=0x7f97a815fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80bf840 0x7f97a815fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 -- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794004d00 msgr2=0x7f97a8161560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794004d00 0x7f97a8161560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:36.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 -- 172.21.15.29:0/1801159533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a80c0390 con 0x7f97a80c7940 2023-10-14T17:00:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.952+0000 7f97ae55d640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f97a0037040 tx=0x7f97a0009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.953+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a0034cd0 con 0x7f97a80c7940 2023-10-14T17:00:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.953+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97a002de90 con 0x7f97a80c7940 2023-10-14T17:00:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.953+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a80c0670 con 0x7f97a80c7940 2023-10-14T17:00:36.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.953+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97a003e660 con 0x7f97a80c7940 2023-10-14T17:00:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.953+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a8161aa0 con 0x7f97a80c7940 2023-10-14T17:00:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.954+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a80c0b90 con 0x7f97a80c7940 2023-10-14T17:00:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.955+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 69) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f97a003e7c0 con 0x7f97a80c7940 2023-10-14T17:00:36.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.956+0000 7f979e7fc640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 0x7f978c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.956+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f97a00ab9f0 con 0x7f97a80c7940 2023-10-14T17:00:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.957+0000 7f97add5c640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 0x7f978c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:36.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.957+0000 7f97add5c640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 0x7f978c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009b20 tx=0x7f97a4008040 comp rx=0 tx=0).ready entity=mgr.8298 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:36.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:36.961+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97a80c0b90 con 0x7f97a80c7940 2023-10-14T17:00:37.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.209+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f97a80c0b90 con 0x7f97a80c7940 2023-10-14T17:00:37.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.869+0000 7f979e7fc640 1 -- 172.21.15.29:0/1801159533 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v70) v1 ==== 92+0+0 (secure 0 0 0) 0x7f97a80c0b90 con 0x7f97a80c7940 2023-10-14T17:00:37.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.873+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 msgr2=0x7f978c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:37.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.873+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 0x7f978c06d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f97a4009b20 tx=0x7f97a4008040 comp rx=0 tx=0).stop 2023-10-14T17:00:37.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 msgr2=0x7f97a80c7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:37.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 secure :-1 s=READY pgs=1612 cs=0 l=1 rev1=1 crypto rx=0x7f97a0037040 tx=0x7f97a0009750 comp rx=0 tx=0).stop 2023-10-14T17:00:37.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 shutdown_connections 2023-10-14T17:00:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f978c06af90 0x7f978c06d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a80c7940 0x7f97a80c7dc0 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a80bf840 0x7f97a815fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:37.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 --2- 172.21.15.29:0/1801159533 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9794004d00 0x7f97a8161560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:37.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 >> 172.21.15.29:0/1801159533 conn(0x7f97a80ce6a0 msgr2=0x7f97a80c40a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:37.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 shutdown_connections 2023-10-14T17:00:37.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:37.874+0000 7f97aed5e640 1 -- 172.21.15.29:0/1801159533 wait complete. 2023-10-14T17:00:37.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T17:00:38.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.110+0000 7ff859f78640 1 Processor -- start 2023-10-14T17:00:38.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.111+0000 7ff859f78640 1 -- start start 2023-10-14T17:00:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85415af80 0x7ff8540cb1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff85406faf0 0x7ff85415f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff854065ab0 0x7ff854065f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8540d0930 con 0x7ff854065ab0 2023-10-14T17:00:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff854066450 con 0x7ff85406faf0 2023-10-14T17:00:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff859f78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85415eff0 con 0x7ff85415af80 2023-10-14T17:00:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.112+0000 7ff853fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff854065ab0 0x7ff854065f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff854065ab0 0x7ff854065f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58242/0 (socket says 172.21.15.29:58242) 2023-10-14T17:00:38.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 -- 172.21.15.29:0/105944500 learned_addr learned my addr 172.21.15.29:0/105944500 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff8537fe640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85415af80 0x7ff8540cb1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff84affd640 1 --2- 172.21.15.29:0/105944500 >> v2:172.21.15.186:3300/0 conn(0x7ff85406faf0 0x7ff85415f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 -- 172.21.15.29:0/105944500 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85415af80 msgr2=0x7ff8540cb1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85415af80 0x7ff8540cb1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 -- 172.21.15.29:0/105944500 >> v2:172.21.15.186:3300/0 conn(0x7ff85406faf0 msgr2=0x7ff85415f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 --2- 172.21.15.29:0/105944500 >> v2:172.21.15.186:3300/0 conn(0x7ff85406faf0 0x7ff85415f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.113+0000 7ff853fff640 1 -- 172.21.15.29:0/105944500 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff85415ecc0 con 0x7ff854065ab0 2023-10-14T17:00:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.115+0000 7ff853fff640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff854065f10 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff8540b35b0 tx=0x7ff84c0302e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3ca343db134ce3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.115+0000 7ff852ffd640 1 -- 172.21.15.29:0/105944500 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c030e80 con 0x7ff854065ab0 2023-10-14T17:00:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.115+0000 7ff852ffd640 1 -- 172.21.15.29:0/105944500 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84c0383f0 con 0x7ff854065ab0 2023-10-14T17:00:38.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.115+0000 7ff852ffd640 1 -- 172.21.15.29:0/105944500 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84c038760 con 0x7ff854065ab0 2023-10-14T17:00:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 -- 172.21.15.29:0/105944500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 msgr2=0x7ff854065f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff854065f10 secure :-1 s=READY pgs=1616 cs=0 l=1 rev1=1 crypto rx=0x7ff8540b35b0 tx=0x7ff84c0302e0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 -- 172.21.15.29:0/105944500 shutdown_connections 2023-10-14T17:00:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff854065f10 unknown :-1 s=CLOSED pgs=1616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 --2- 172.21.15.29:0/105944500 >> v2:172.21.15.186:3300/0 conn(0x7ff85406faf0 0x7ff85415f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 --2- 172.21.15.29:0/105944500 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85415af80 0x7ff8540cb1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 -- 172.21.15.29:0/105944500 >> 172.21.15.29:0/105944500 conn(0x7ff8540d0ab0 msgr2=0x7ff8540d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 -- 172.21.15.29:0/105944500 shutdown_connections 2023-10-14T17:00:38.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.116+0000 7ff859f78640 1 -- 172.21.15.29:0/105944500 wait complete. 2023-10-14T17:00:38.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.118+0000 7ff859f78640 1 Processor -- start 2023-10-14T17:00:38.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.118+0000 7ff859f78640 1 -- start start 2023-10-14T17:00:38.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 0x7ff85406c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 0x7ff85414c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff85406cc90 con 0x7ff854065ab0 2023-10-14T17:00:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8540c85c0 con 0x7ff85415af80 2023-10-14T17:00:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.119+0000 7ff859f78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff85406ca10 con 0x7ff85406faf0 2023-10-14T17:00:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58246/0 (socket says 172.21.15.29:58246) 2023-10-14T17:00:38.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 -- 172.21.15.29:0/2650902163 learned_addr learned my addr 172.21.15.29:0/2650902163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 msgr2=0x7ff85406c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff853fff640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 0x7ff85414c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff84affd640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 0x7ff85406c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 0x7ff85406c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 msgr2=0x7ff85414c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 0x7ff85414c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 -- 172.21.15.29:0/2650902163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c009b30 con 0x7ff854065ab0 2023-10-14T17:00:38.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff853fff640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 0x7ff85414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:38.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff84affd640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 0x7ff85406c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:38.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.120+0000 7ff8537fe640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff84400d800 tx=0x7ff84400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.121+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400b6c0 con 0x7ff854065ab0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.121+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff84400b820 con 0x7ff854065ab0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.121+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8540c7c90 con 0x7ff854065ab0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.121+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84400bbb0 con 0x7ff854065ab0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.121+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8540c8250 con 0x7ff854065ab0 2023-10-14T17:00:38.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.122+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff85404e160 con 0x7ff854065ab0 2023-10-14T17:00:38.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.123+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff84400bd10 con 0x7ff854065ab0 2023-10-14T17:00:38.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.124+0000 7ff850ff9640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 0x7ff83406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.124+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7ff844089d90 con 0x7ff854065ab0 2023-10-14T17:00:38.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.124+0000 7ff84affd640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 msgr2=0x7ff83406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:38.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.124+0000 7ff84affd640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 0x7ff83406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:38.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.133+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff844054c40 con 0x7ff854065ab0 2023-10-14T17:00:38.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.325+0000 7ff84affd640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 msgr2=0x7ff83406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:38.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.325+0000 7ff84affd640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 0x7ff83406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:38.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.333+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7ff85415b960 con 0x7ff854065ab0 2023-10-14T17:00:38.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.339+0000 7ff850ff9640 1 -- 172.21.15.29:0/2650902163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v47)=0 v47) v1 ==== 138+0+0 (secure 0 0 0) 0x7ff844058260 con 0x7ff854065ab0 2023-10-14T17:00:38.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 msgr2=0x7ff83406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:38.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 0x7ff83406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 msgr2=0x7ff85414d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 secure :-1 s=READY pgs=1617 cs=0 l=1 rev1=1 crypto rx=0x7ff84400d800 tx=0x7ff84400dcc0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 shutdown_connections 2023-10-14T17:00:38.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7ff83406afe0 0x7ff83406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff85415af80 0x7ff85414c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff85406faf0 0x7ff85406c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 --2- 172.21.15.29:0/2650902163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff854065ab0 0x7ff85414d1d0 unknown :-1 s=CLOSED pgs=1617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 >> 172.21.15.29:0/2650902163 conn(0x7ff8540d0ab0 msgr2=0x7ff854157800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:38.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 shutdown_connections 2023-10-14T17:00:38.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.341+0000 7ff859f78640 1 -- 172.21.15.29:0/2650902163 wait complete. 2023-10-14T17:00:38.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T17:00:38.433 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:38.432+0000 7f8d19c2b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:38.439 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:38.439+0000 7ff66ae11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:38.491 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:38.489+0000 7f8d19c2b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:38.498 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:38.498+0000 7ff66ae11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:38.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.581+0000 7f0fdd62d640 1 Processor -- start 2023-10-14T17:00:38.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.581+0000 7f0fdd62d640 1 -- start start 2023-10-14T17:00:38.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.583+0000 7f0fdd62d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 0x7f0fd81618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.583+0000 7f0fdd62d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0fd80661c0 0x7f0fd80ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.583+0000 7f0fdd62d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fd80d0890 con 0x7f0fd806c7d0 2023-10-14T17:00:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.583+0000 7f0fdd62d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fd806b1e0 con 0x7f0fd80661c0 2023-10-14T17:00:38.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.583+0000 7f0fdd62d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80d05b0 con 0x7f0fd815d330 2023-10-14T17:00:38.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd67fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0fd806c7d0 0x7f0fd8065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35596/0 (socket says 172.21.15.29:35596) 2023-10-14T17:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd67fc640 1 -- 172.21.15.29:0/109995077 learned_addr learned my addr 172.21.15.29:0/109995077 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd77fe640 1 --2- 172.21.15.29:0/109995077 >> v2:172.21.15.186:3300/0 conn(0x7f0fd80661c0 0x7f0fd80ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 0x7f0fd81618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 168824078 0 0) 0x7f0fd80d0890 con 0x7f0fd806c7d0 2023-10-14T17:00:38.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fc4003520 con 0x7f0fd806c7d0 2023-10-14T17:00:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 493272181 0 0) 0x7f0fc4003520 con 0x7f0fd806c7d0 2023-10-14T17:00:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 msgr2=0x7f0fd81618b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 0x7f0fd81618b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 >> v2:172.21.15.186:3300/0 conn(0x7f0fd80661c0 msgr2=0x7f0fd80ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.584+0000 7f0fd5ffb640 1 --2- 172.21.15.29:0/109995077 >> v2:172.21.15.186:3300/0 conn(0x7f0fd80661c0 0x7f0fd80ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd8160dd0 con 0x7f0fd806c7d0 2023-10-14T17:00:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0fc0001670 con 0x7f0fd806c7d0 2023-10-14T17:00:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 >> v1:172.21.15.29:6789/0 conn(0x7f0fd806c7d0 legacy=0x7f0fd8065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 0x7f0fc4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd806b1e0 con 0x7f0fc4003970 2023-10-14T17:00:38.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80d05b0 con 0x7f0fc40047c0 2023-10-14T17:00:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc400b980 con 0x7f0fd815d330 2023-10-14T17:00:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd67fc640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 -- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 msgr2=0x7f0fc4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd815d330 0x7f0fc4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 -- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc40047c0 msgr2=0x7f0fc4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd6ffd640 1 -- 172.21.15.29:0/109995077 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4003520 con 0x7f0fc4003970 2023-10-14T17:00:38.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.585+0000 7f0fd67fc640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc40047c0 0x7f0fc4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:38.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.586+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800e170 tx=0x7f0fc40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5fdb37d262f04345 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc802cc20 con 0x7f0fc4003970 2023-10-14T17:00:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc800f040 con 0x7f0fc4003970 2023-10-14T17:00:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fd5ffb640 1 -- 172.21.15.29:0/109995077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc8034440 con 0x7f0fc4003970 2023-10-14T17:00:38.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fdd62d640 1 -- 172.21.15.29:0/109995077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc4003970 msgr2=0x7f0fc4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/109995077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fc4003970 0x7f0fc4003d90 secure :-1 s=READY pgs=1619 cs=0 l=1 rev1=1 crypto rx=0x7f0fc800e170 tx=0x7f0fc40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.587+0000 7f0fd77fe640 1 -- 172.21.15.29:0/109995077 reap_dead start 2023-10-14T17:00:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.588+0000 7f0fdd62d640 1 -- 172.21.15.29:0/109995077 shutdown_connections 2023-10-14T17:00:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.588+0000 7f0fdd62d640 1 -- 172.21.15.29:0/109995077 >> 172.21.15.29:0/109995077 conn(0x7f0fd80d0ab0 msgr2=0x7f0fd80d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:38.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.588+0000 7f0fdd62d640 1 -- 172.21.15.29:0/109995077 shutdown_connections 2023-10-14T17:00:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.588+0000 7f0fdd62d640 1 -- 172.21.15.29:0/109995077 wait complete. 2023-10-14T17:00:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.589+0000 7f0fdd62d640 1 Processor -- start 2023-10-14T17:00:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.590+0000 7f0fdd62d640 1 -- start start 2023-10-14T17:00:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc4003970 0x7f0fd8154b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd80b4040 0x7f0fd80b44c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd81e6660 con 0x7f0fd81e9040 2023-10-14T17:00:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b6410 con 0x7f0fc4003970 2023-10-14T17:00:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fdd62d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fd80b5890 con 0x7f0fd80b4040 2023-10-14T17:00:38.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58252/0 (socket says 172.21.15.29:58252) 2023-10-14T17:00:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 -- 172.21.15.29:0/2234240423 learned_addr learned my addr 172.21.15.29:0/2234240423 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd77fe640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd80b4040 0x7f0fd80b44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc4003970 0x7f0fd8154b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd80b4040 msgr2=0x7f0fd80b44c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd80b4040 0x7f0fd80b44c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc4003970 msgr2=0x7f0fd8154b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc4003970 0x7f0fd8154b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.591+0000 7f0fd67fc640 1 -- 172.21.15.29:0/2234240423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fd80b5b70 con 0x7f0fd81e9040 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fd67fc640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000d8b0 tx=0x7f0fc000dd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc00016d0 con 0x7f0fd81e9040 2023-10-14T17:00:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fc00074e0 con 0x7f0fd81e9040 2023-10-14T17:00:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fc0004a00 con 0x7f0fd81e9040 2023-10-14T17:00:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fd80b5e50 con 0x7f0fd81e9040 2023-10-14T17:00:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.592+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fd81f63b0 con 0x7f0fd81e9040 2023-10-14T17:00:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.593+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fd804e160 con 0x7f0fd81e9040 2023-10-14T17:00:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.595+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0fc0004b60 con 0x7f0fd81e9040 2023-10-14T17:00:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.596+0000 7f0fcf7fe640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 0x7f0fb806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.596+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f0fc00893c0 con 0x7f0fd81e9040 2023-10-14T17:00:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.596+0000 7f0fd6ffd640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 msgr2=0x7f0fb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.596+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 0x7f0fb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:38.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.601+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0fc0057c80 con 0x7f0fd81e9040 2023-10-14T17:00:38.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.793+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f0fd8161390 con 0x7f0fd81e9040 2023-10-14T17:00:38.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.797+0000 7f0fd6ffd640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 msgr2=0x7f0fb806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:38.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.797+0000 7f0fd6ffd640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 0x7f0fb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:38.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.800+0000 7f0fcf7fe640 1 -- 172.21.15.29:0/2234240423 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v48)=0 v48) v1 ==== 133+0+0 (secure 0 0 0) 0x7f0fc0057810 con 0x7f0fd81e9040 2023-10-14T17:00:38.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.802+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 msgr2=0x7f0fb806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.802+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 0x7f0fb806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 msgr2=0x7f0fd80b3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 secure :-1 s=READY pgs=1620 cs=0 l=1 rev1=1 crypto rx=0x7f0fc000d8b0 tx=0x7f0fc000dd70 comp rx=0 tx=0).stop 2023-10-14T17:00:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 shutdown_connections 2023-10-14T17:00:38.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7f0fb806afe0 0x7f0fb806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0fd80b4040 0x7f0fd80b44c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0fd81e9040 0x7f0fd80b3b00 unknown :-1 s=CLOSED pgs=1620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 --2- 172.21.15.29:0/2234240423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0fc4003970 0x7f0fd8154b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:38.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 >> 172.21.15.29:0/2234240423 conn(0x7f0fd80d0ab0 msgr2=0x7f0fd815d710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:38.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 shutdown_connections 2023-10-14T17:00:38.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:38.803+0000 7f0fdd62d640 1 -- 172.21.15.29:0/2234240423 wait complete. 2023-10-14T17:00:38.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:38.822 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/minutes) 2023-10-14T17:00:38.881 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/minutes 1m 2023-10-14T17:00:38.948 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:38.947+0000 7ff66ae11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:38.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:38.976+0000 7f8d19c2b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:38.999 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:38.998+0000 7ff66ae11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:39.032 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.031+0000 7f8d19c2b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:39.048 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.047+0000 7ff66ae11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:39.086 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.085+0000 7f8d19c2b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:39.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.090+0000 7fef7b013640 1 Processor -- start 2023-10-14T17:00:39.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.090+0000 7fef7b013640 1 -- start start 2023-10-14T17:00:39.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7b013640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fef7415e350 0x7fef740cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7b013640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fef74070970 0x7fef740ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7b013640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef7406fbc0 con 0x7fef7415e980 2023-10-14T17:00:39.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7b013640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef7415e730 con 0x7fef74070970 2023-10-14T17:00:39.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7b013640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef740c9920 con 0x7fef7415e350 2023-10-14T17:00:39.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef79810640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fef74070970 0x7fef740ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:39.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7a812640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fef7415e980 0x7fef7415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35602/0 (socket says 172.21.15.29:35602) 2023-10-14T17:00:39.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.092+0000 7fef7a812640 1 -- 172.21.15.29:0/3067413932 learned_addr learned my addr 172.21.15.29:0/3067413932 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:39.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef7a011640 1 --2- 172.21.15.29:0/3067413932 >> v2:172.21.15.186:3300/0 conn(0x7fef7415e350 0x7fef740cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:39.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef79810640 1 -- 172.21.15.29:0/3067413932 >> v2:172.21.15.186:3300/0 conn(0x7fef7415e350 msgr2=0x7fef740cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:39.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef79810640 1 --2- 172.21.15.29:0/3067413932 >> v2:172.21.15.186:3300/0 conn(0x7fef7415e350 0x7fef740cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:39.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef79810640 1 -- 172.21.15.29:0/3067413932 >> v1:172.21.15.29:6789/0 conn(0x7fef7415e980 legacy=0x7fef7415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:39.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef7900f640 1 -- 172.21.15.29:0/3067413932 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798981462 0 0) 0x7fef7406fbc0 con 0x7fef7415e980 2023-10-14T17:00:39.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.093+0000 7fef79810640 1 -- 172.21.15.29:0/3067413932 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef740c9c00 con 0x7fef74070970 2023-10-14T17:00:39.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.094+0000 7fef79810640 1 --2- 172.21.15.29:0/3067413932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef74070970 0x7fef740ce080 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fef74067270 tx=0x7fef6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3803146dc4806c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:39.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.094+0000 7fef7900f640 1 -- 172.21.15.29:0/3067413932 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6802e070 con 0x7fef74070970 2023-10-14T17:00:39.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.094+0000 7fef7900f640 1 -- 172.21.15.29:0/3067413932 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef680315a0 con 0x7fef74070970 2023-10-14T17:00:39.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.094+0000 7fef7900f640 1 -- 172.21.15.29:0/3067413932 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef68031930 con 0x7fef74070970 2023-10-14T17:00:39.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 -- 172.21.15.29:0/3067413932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef74070970 msgr2=0x7fef740ce080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:39.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 --2- 172.21.15.29:0/3067413932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef74070970 0x7fef740ce080 secure :-1 s=READY pgs=1622 cs=0 l=1 rev1=1 crypto rx=0x7fef74067270 tx=0x7fef6802b840 comp rx=0 tx=0).stop 2023-10-14T17:00:39.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 -- 172.21.15.29:0/3067413932 shutdown_connections 2023-10-14T17:00:39.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 --2- 172.21.15.29:0/3067413932 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef74070970 0x7fef740ce080 unknown :-1 s=CLOSED pgs=1622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:39.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 --2- 172.21.15.29:0/3067413932 >> v2:172.21.15.186:3300/0 conn(0x7fef7415e350 0x7fef740cdb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:39.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 -- 172.21.15.29:0/3067413932 >> 172.21.15.29:0/3067413932 conn(0x7fef740b0a50 msgr2=0x7fef740b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:39.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 -- 172.21.15.29:0/3067413932 shutdown_connections 2023-10-14T17:00:39.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.095+0000 7fef7b013640 1 -- 172.21.15.29:0/3067413932 wait complete. 2023-10-14T17:00:39.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.098+0000 7fef7b013640 1 Processor -- start 2023-10-14T17:00:39.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.098+0000 7fef7b013640 1 -- start start 2023-10-14T17:00:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef74070970 0x7fef740c1860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7415e350 0x7fef740c69c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 0x7fef740c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740c7be0 con 0x7fef7415e980 2023-10-14T17:00:39.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef740c7590 con 0x7fef7415e350 2023-10-14T17:00:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7b013640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef740c6410 con 0x7fef74070970 2023-10-14T17:00:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.100+0000 7fef7a812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 0x7fef740c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a011640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef74070970 0x7fef740c1860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 0x7fef740c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:58274/0 (socket says 172.21.15.29:58274) 2023-10-14T17:00:39.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 -- 172.21.15.29:0/2373305485 learned_addr learned my addr 172.21.15.29:0/2373305485 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:39.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef79810640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7415e350 0x7fef740c69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:39.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef74070970 msgr2=0x7fef740c1860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:39.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef74070970 0x7fef740c1860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:39.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7415e350 msgr2=0x7fef740c69c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:39.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef7415e350 0x7fef740c69c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef68009400 con 0x7fef7415e980 2023-10-14T17:00:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef7a812640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 0x7fef740c6f00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00c7e0 tx=0x7fef6c00cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.101+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00a640 con 0x7fef7415e980 2023-10-14T17:00:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.102+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6c00a7c0 con 0x7fef7415e980 2023-10-14T17:00:39.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.102+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6c00ab70 con 0x7fef7415e980 2023-10-14T17:00:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.102+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef740c7800 con 0x7fef7415e980 2023-10-14T17:00:39.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.102+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef740c1da0 con 0x7fef7415e980 2023-10-14T17:00:39.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.103+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef7406e500 con 0x7fef7415e980 2023-10-14T17:00:39.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.104+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 70) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fef6c00acd0 con 0x7fef7415e980 2023-10-14T17:00:39.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.105+0000 7fef72ffd640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:39.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.106+0000 7fef7a011640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.106+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fef6c088ee0 con 0x7fef7415e980 2023-10-14T17:00:39.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.106+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:39.110 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.108+0000 7ff66ae11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:39.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.109+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef6c053db0 con 0x7fef7415e980 2023-10-14T17:00:39.156 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.154+0000 7f8d19c2b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:39.157 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.156+0000 7ff66ae11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:39.207 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.206+0000 7f8d19c2b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:39.266 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.265+0000 7ff66ae11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.307+0000 7fef7a011640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:39.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.307+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:39.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.322+0000 7f8d19c2b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:39.327 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.326+0000 7ff66ae11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:39.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.326+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fef740ca1c0 con 0x7fef6006afe0 2023-10-14T17:00:39.376 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.376+0000 7ff66ae11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:39.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.388+0000 7f8d19c2b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:39.433 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.433+0000 7ff66ae11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:39.442 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.440+0000 7f8d19c2b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:39.483 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.482+0000 7ff66ae11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:39.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.499+0000 7f8d19c2b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:39.553 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.551+0000 7f8d19c2b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:39.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.708+0000 7fef7a011640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:39.708+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T17:00:39.856 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:39.855+0000 7ff66ae11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:39.952 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:39.950+0000 7f8d19c2b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:40.013 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.013+0000 7ff66ae11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:40.119 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.117+0000 7f8d19c2b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:40.119 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.118+0000 7ff66ae11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:40.168 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.167+0000 7ff66ae11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:40.232 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.230+0000 7f8d19c2b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:40.275 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.274+0000 7ff66ae11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:40.285 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.283+0000 7f8d19c2b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:40.382 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.381+0000 7ff66ae11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:40.397 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.396+0000 7f8d19c2b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:40.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.507+0000 7f8d19c2b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:40.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:40.509+0000 7fef7a011640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:40.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:40.509+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T17:00:40.536 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.535+0000 7ff66ae11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:40.585 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.584+0000 7ff66ae11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:40.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.671+0000 7f8d19c2b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:40.702 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:40.701+0000 7ff66ae11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:40.724 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.723+0000 7f8d19c2b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:40.850 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:40.848+0000 7f8d19c2b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:41.520 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:41.519+0000 7ff66ae11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:41.739 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:41.738+0000 7f8d19c2b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:41.810 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:41.809+0000 7ff66ae11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:41.867 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:41.865+0000 7ff66ae11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:41.978 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:41.977+0000 7ff66ae11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:42.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.046+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 71) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fef6c054180 con 0x7fef7415e980 2023-10-14T17:00:42.054 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.053+0000 7f8d19c2b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:42.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.111+0000 7fef7a011640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3738876524 2023-10-14T17:00:42.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.111+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T17:00:42.114 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.113+0000 7f8d19c2b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:42.233 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.231+0000 7f8d19c2b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:42.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.250+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 72) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fef6c05f8c0 con 0x7fef7415e980 2023-10-14T17:00:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.250+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 msgr2=0x7fef6006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:42.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:42.250+0000 7fef72ffd640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/3738876524,v1:172.21.15.29:6833/3738876524] conn(0x7fef6006afe0 0x7fef6006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:42.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.603+0000 7f8c98000640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.603+0000 7f8c98000640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.603+0000 7f8c98000640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.606 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.603+0000 7f8c98000640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.606 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.603+0000 7f8c98000640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.606 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.604+0000 7f8c95fbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.604+0000 7f8c95fbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.604+0000 7f8c95fbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.604+0000 7f8c95fbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:42.607 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:42.604+0000 7f8c95fbc640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:43.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.262+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 73) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fef6c054a50 con 0x7fef7415e980 2023-10-14T17:00:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.263+0000 7fef72ffd640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fef6001d1c0 0x7fef60074f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.264+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/minutes", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fef6001d5a0 con 0x7fef6001d1c0 2023-10-14T17:00:43.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.269+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fef6001d1c0 0x7fef60074f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.279+0000 7fef7a011640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fef6001d1c0 0x7fef60074f90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fef64006fd0 tx=0x7fef64008040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:43.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.483+0000 7fef72ffd640 1 -- 172.21.15.29:0/2373305485 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fef6001d5a0 con 0x7fef6001d1c0 2023-10-14T17:00:43.486 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/minutes 2023-10-14T17:00:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fef6001d1c0 msgr2=0x7fef60074f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fef6001d1c0 0x7fef60074f90 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fef64006fd0 tx=0x7fef64008040 comp rx=0 tx=0).stop 2023-10-14T17:00:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 msgr2=0x7fef740c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 --2- 172.21.15.29:0/2373305485 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef7415e980 0x7fef740c6f00 secure :-1 s=READY pgs=1623 cs=0 l=1 rev1=1 crypto rx=0x7fef6c00c7e0 tx=0x7fef6c00cca0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7a812640 1 -- 172.21.15.29:0/2373305485 reap_dead start 2023-10-14T17:00:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 shutdown_connections 2023-10-14T17:00:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 >> 172.21.15.29:0/2373305485 conn(0x7fef740b0a50 msgr2=0x7fef7415bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 shutdown_connections 2023-10-14T17:00:43.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.487+0000 7fef7b013640 1 -- 172.21.15.29:0/2373305485 wait complete. 2023-10-14T17:00:43.510 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/minutes 2023-10-14T17:00:43.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:43.511 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/hourly) 2023-10-14T17:00:43.570 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/hourly 1h 2023-10-14T17:00:43.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.808+0000 7eff16ea2640 1 Processor -- start 2023-10-14T17:00:43.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.808+0000 7eff16ea2640 1 -- start start 2023-10-14T17:00:43.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff16ea2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7eff100c95b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff16ea2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100c9af0 con 0x7eff101428c0 2023-10-14T17:00:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff16ea2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff100cc6b0 con 0x7eff1006b660 2023-10-14T17:00:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff16ea2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff1006a4b0 con 0x7eff100c9150 2023-10-14T17:00:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff14c3e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7eff101428c0 0x7eff10069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53502/0 (socket says 172.21.15.29:53502) 2023-10-14T17:00:43.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff14c3e640 1 -- 172.21.15.29:0/1783967984 learned_addr learned my addr 172.21.15.29:0/1783967984 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:43.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3751909874 0 0) 0x7eff100c9af0 con 0x7eff101428c0 2023-10-14T17:00:43.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.810+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efef8005730 con 0x7eff101428c0 2023-10-14T17:00:43.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7eff100c95b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 218763876 0 0) 0x7efef8005730 con 0x7eff101428c0 2023-10-14T17:00:43.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 msgr2=0x7eff100c95b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:43.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7eff100c95b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 >> v1:172.21.15.186:6789/0 conn(0x7eff1006b660 legacy=0x7eff101643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff101599b0 con 0x7eff101428c0 2023-10-14T17:00:43.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7eff000019f0 con 0x7eff101428c0 2023-10-14T17:00:43.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 >> v1:172.21.15.29:6789/0 conn(0x7eff101428c0 legacy=0x7eff10069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7efef8005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef80069d0 0x7efef8006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7efef8007420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff1006a4b0 con 0x7efef8005bb0 2023-10-14T17:00:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef800dd20 con 0x7efef80069d0 2023-10-14T17:00:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efef800dee0 con 0x7eff100c9150 2023-10-14T17:00:43.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.811+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7efef8005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef80069d0 0x7efef8006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff1543f640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7efef8007420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff14c3e640 1 -- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 msgr2=0x7efef8007420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:43.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7efef8007420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff14c3e640 1 -- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef80069d0 msgr2=0x7efef8006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:43.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efef80069d0 0x7efef8006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff14c3e640 1 -- 172.21.15.29:0/1783967984 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef8005730 con 0x7efef8005bb0 2023-10-14T17:00:43.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.812+0000 7eff1543f640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff100c9150 0x7efef8007420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:43.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.813+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7efef8005fd0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7eff0000a760 tx=0x7efef8007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6bccbdf6eeb64a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:43.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.813+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00034630 con 0x7efef8005bb0 2023-10-14T17:00:43.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.813+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff0002f070 con 0x7efef8005bb0 2023-10-14T17:00:43.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.813+0000 7eff0f7fe640 1 -- 172.21.15.29:0/1783967984 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00034a10 con 0x7efef8005bb0 2023-10-14T17:00:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 -- 172.21.15.29:0/1783967984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 msgr2=0x7efef8005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1783967984 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7efef8005fd0 secure :-1 s=READY pgs=1638 cs=0 l=1 rev1=1 crypto rx=0x7eff0000a760 tx=0x7efef8007a00 comp rx=0 tx=0).stop 2023-10-14T17:00:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff1543f640 1 -- 172.21.15.29:0/1783967984 reap_dead start 2023-10-14T17:00:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 -- 172.21.15.29:0/1783967984 shutdown_connections 2023-10-14T17:00:43.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 -- 172.21.15.29:0/1783967984 >> 172.21.15.29:0/1783967984 conn(0x7eff100ce9b0 msgr2=0x7eff100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 -- 172.21.15.29:0/1783967984 shutdown_connections 2023-10-14T17:00:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.814+0000 7eff16ea2640 1 -- 172.21.15.29:0/1783967984 wait complete. 2023-10-14T17:00:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.816+0000 7eff16ea2640 1 Processor -- start 2023-10-14T17:00:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.816+0000 7eff16ea2640 1 -- start start 2023-10-14T17:00:43.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 0x7eff101579f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff101e1ad0 0x7eff1015be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff100c6ef0 con 0x7efef8005bb0 2023-10-14T17:00:43.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff100c8310 con 0x7eff101581f0 2023-10-14T17:00:43.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff16ea2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff100c6bc0 con 0x7eff101e1ad0 2023-10-14T17:00:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff14c3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.817+0000 7eff14c3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53324/0 (socket says 172.21.15.29:53324) 2023-10-14T17:00:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 -- 172.21.15.29:0/1674802447 learned_addr learned my addr 172.21.15.29:0/1674802447 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 0x7eff101579f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff1543f640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff101e1ad0 0x7eff1015be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 -- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff101e1ad0 msgr2=0x7eff1015be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff101e1ad0 0x7eff1015be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 -- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 msgr2=0x7eff101579f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 0x7eff101579f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:43.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff14c3e640 1 -- 172.21.15.29:0/1674802447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff100c8890 con 0x7efef8005bb0 2023-10-14T17:00:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.818+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 0x7eff101579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff14c3e640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7eff00000f80 tx=0x7eff00001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00034430 con 0x7efef8005bb0 2023-10-14T17:00:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff00034590 con 0x7efef8005bb0 2023-10-14T17:00:43.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff00034920 con 0x7efef8005bb0 2023-10-14T17:00:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff100c85b0 con 0x7efef8005bb0 2023-10-14T17:00:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.819+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff10157f30 con 0x7efef8005bb0 2023-10-14T17:00:43.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.820+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff1004e160 con 0x7efef8005bb0 2023-10-14T17:00:43.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.821+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 73) v1 ==== 89084+0+0 (secure 0 0 0) 0x7eff00031030 con 0x7efef8005bb0 2023-10-14T17:00:43.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.823+0000 7eff0d7fa640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 0x7efefc06d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:43.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.823+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7eff000ab860 con 0x7efef8005bb0 2023-10-14T17:00:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.823+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 0x7efefc06d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.824+0000 7eff0ffff640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 0x7efefc06d250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7eff100c4e50 tx=0x7eff04008040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:43.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:43.828+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7eff00076a60 con 0x7efef8005bb0 2023-10-14T17:00:44.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.046+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/hourly", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7eff1006c570 con 0x7efefc06ad90 2023-10-14T17:00:44.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.241+0000 7eff0d7fa640 1 -- 172.21.15.29:0/1674802447 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7eff0400b220 con 0x7efefc06ad90 2023-10-14T17:00:44.244 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/hourly 2023-10-14T17:00:44.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.244+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 msgr2=0x7efefc06d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.244+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 0x7efefc06d250 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7eff100c4e50 tx=0x7eff04008040 comp rx=0 tx=0).stop 2023-10-14T17:00:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.244+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 msgr2=0x7eff100c7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.244+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 secure :-1 s=READY pgs=1639 cs=0 l=1 rev1=1 crypto rx=0x7eff00000f80 tx=0x7eff00001cb0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 shutdown_connections 2023-10-14T17:00:44.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7efefc06ad90 0x7efefc06d250 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff101e1ad0 0x7eff1015be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff101581f0 0x7eff101579f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 --2- 172.21.15.29:0/1674802447 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efef8005bb0 0x7eff100c7380 unknown :-1 s=CLOSED pgs=1639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 >> 172.21.15.29:0/1674802447 conn(0x7eff100ce9b0 msgr2=0x7eff100cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:44.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 shutdown_connections 2023-10-14T17:00:44.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.245+0000 7eff16ea2640 1 -- 172.21.15.29:0/1674802447 wait complete. 2023-10-14T17:00:44.270 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/hourly 2023-10-14T17:00:44.271 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:44.271 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/daily) 2023-10-14T17:00:44.328 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/daily 1d 2023-10-14T17:00:44.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.532+0000 7f175a040640 1 Processor -- start 2023-10-14T17:00:44.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.532+0000 7f175a040640 1 -- start start 2023-10-14T17:00:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.533+0000 7f175a040640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1754159950 0x7f175415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.533+0000 7f175a040640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17540708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.533+0000 7f175a040640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1754070df0 con 0x7f175406c100 2023-10-14T17:00:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.533+0000 7f175a040640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f175415e490 con 0x7f1754159950 2023-10-14T17:00:44.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.533+0000 7f175a040640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f175415aef0 con 0x7f1754070450 2023-10-14T17:00:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.534+0000 7f175903e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1754159950 0x7f175415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.534+0000 7f175883d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17540708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.534+0000 7f175903e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1754159950 0x7f175415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:37674/0 (socket says 172.21.15.29:37674) 2023-10-14T17:00:44.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.534+0000 7f175903e640 1 -- 172.21.15.29:0/3188126078 learned_addr learned my addr 172.21.15.29:0/3188126078 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.535+0000 7f175903e640 1 -- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 msgr2=0x7f17540708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.535+0000 7f175903e640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17540708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.535+0000 7f175903e640 1 -- 172.21.15.29:0/3188126078 >> v1:172.21.15.186:6789/0 conn(0x7f175406c100 legacy=0x7f175406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.535+0000 7f175903e640 1 -- 172.21.15.29:0/3188126078 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f175406cc80 con 0x7f1754159950 2023-10-14T17:00:44.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.535+0000 7f175883d640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17540708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:00:44.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.536+0000 7f175903e640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175415a9b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1750009e50 tx=0x7f175002bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5f564e5db90e2990 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.537+0000 7f174bfff640 1 -- 172.21.15.29:0/3188126078 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175002ca80 con 0x7f1754159950 2023-10-14T17:00:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.537+0000 7f174bfff640 1 -- 172.21.15.29:0/3188126078 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175002cbe0 con 0x7f1754159950 2023-10-14T17:00:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.537+0000 7f174bfff640 1 -- 172.21.15.29:0/3188126078 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17500343f0 con 0x7f1754159950 2023-10-14T17:00:44.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 -- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 msgr2=0x7f175415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175415a9b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1750009e50 tx=0x7f175002bee0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 -- 172.21.15.29:0/3188126078 shutdown_connections 2023-10-14T17:00:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17540708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 --2- 172.21.15.29:0/3188126078 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175415a9b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 -- 172.21.15.29:0/3188126078 >> 172.21.15.29:0/3188126078 conn(0x7f17540ce6a0 msgr2=0x7f17540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 -- 172.21.15.29:0/3188126078 shutdown_connections 2023-10-14T17:00:44.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.538+0000 7f175a040640 1 -- 172.21.15.29:0/3188126078 wait complete. 2023-10-14T17:00:44.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.540+0000 7f175a040640 1 Processor -- start 2023-10-14T17:00:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.540+0000 7f175a040640 1 -- start start 2023-10-14T17:00:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17541658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175414b130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1754165e30 con 0x7f175406c100 2023-10-14T17:00:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17541660d0 con 0x7f1754159950 2023-10-14T17:00:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175a040640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17541663b0 con 0x7f1754070450 2023-10-14T17:00:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175903e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175903e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53350/0 (socket says 172.21.15.29:53350) 2023-10-14T17:00:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175903e640 1 -- 172.21.15.29:0/3934073449 learned_addr learned my addr 172.21.15.29:0/3934073449 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.541+0000 7f175983f640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175414b130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175883d640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17541658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 -- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 msgr2=0x7f17541658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17541658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 -- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 msgr2=0x7f175414b130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175414b130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 -- 172.21.15.29:0/3934073449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1744009400 con 0x7f175406c100 2023-10-14T17:00:44.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f175903e640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f1750000f80 tx=0x7f1750004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:44.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.542+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175003e450 con 0x7f175406c100 2023-10-14T17:00:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.543+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f175003e5f0 con 0x7f175406c100 2023-10-14T17:00:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.543+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f175003e9a0 con 0x7f175406c100 2023-10-14T17:00:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.543+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1750009b30 con 0x7f175406c100 2023-10-14T17:00:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.543+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1754166970 con 0x7f175406c100 2023-10-14T17:00:44.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.544+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f175404e9b0 con 0x7f175406c100 2023-10-14T17:00:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.545+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f175002cd50 con 0x7f175406c100 2023-10-14T17:00:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.546+0000 7f1749ffb640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 0x7f173806d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.546+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f17500ac380 con 0x7f175406c100 2023-10-14T17:00:44.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.546+0000 7f175883d640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 0x7f173806d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:44.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.547+0000 7f175883d640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 0x7f173806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f175414e800 tx=0x7f1744008040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:44.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.551+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1750077330 con 0x7f175406c100 2023-10-14T17:00:44.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.770+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/daily", "snap_schedule": "1d", "target": ["mon-mgr", ""]}) v1 -- 0x7f175415dbf0 con 0x7f173806b010 2023-10-14T17:00:44.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.960+0000 7f1749ffb640 1 -- 172.21.15.29:0/3934073449 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f17440098b0 con 0x7f173806b010 2023-10-14T17:00:44.962 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/daily 2023-10-14T17:00:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 msgr2=0x7f173806d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 0x7f173806d4d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f175414e800 tx=0x7f1744008040 comp rx=0 tx=0).stop 2023-10-14T17:00:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 msgr2=0x7f17541675a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 secure :-1 s=READY pgs=1642 cs=0 l=1 rev1=1 crypto rx=0x7f1750000f80 tx=0x7f1750004a40 comp rx=0 tx=0).stop 2023-10-14T17:00:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 shutdown_connections 2023-10-14T17:00:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f173806b010 0x7f173806d4d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1754159950 0x7f175414b130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1754070450 0x7f17541658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 --2- 172.21.15.29:0/3934073449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f175406c100 0x7f17541675a0 unknown :-1 s=CLOSED pgs=1642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:44.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 >> 172.21.15.29:0/3934073449 conn(0x7f17540ce6a0 msgr2=0x7f17540cd4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:44.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 shutdown_connections 2023-10-14T17:00:44.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:44.964+0000 7f175a040640 1 -- 172.21.15.29:0/3934073449 wait complete. 2023-10-14T17:00:44.987 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/daily 2023-10-14T17:00:44.987 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:44.988 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/weekly) 2023-10-14T17:00:45.051 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/weekly 1w 2023-10-14T17:00:45.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.276+0000 7f2edec6f640 1 Processor -- start 2023-10-14T17:00:45.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.276+0000 7f2edec6f640 1 -- start start 2023-10-14T17:00:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.278+0000 7f2edec6f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ed815ad70 0x7f2ed8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.278+0000 7f2edec6f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80cb030 con 0x7f2ed80caac0 2023-10-14T17:00:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.278+0000 7f2edec6f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ed80c91c0 con 0x7f2ed80c9ca0 2023-10-14T17:00:45.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.278+0000 7f2edec6f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ed80c94a0 con 0x7f2ed815ad70 2023-10-14T17:00:45.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edca0b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2ed815ad70 0x7f2ed8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:45.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edd20c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2ed80caac0 0x7f2ed80ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53514/0 (socket says 172.21.15.29:53514) 2023-10-14T17:00:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edd20c640 1 -- 172.21.15.29:0/626496979 learned_addr learned my addr 172.21.15.29:0/626496979 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edca0b640 1 -- 172.21.15.29:0/626496979 >> v1:172.21.15.186:6789/0 conn(0x7f2ed80c9ca0 legacy=0x7f2ed80ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edca0b640 1 -- 172.21.15.29:0/626496979 >> v1:172.21.15.29:6789/0 conn(0x7f2ed80caac0 legacy=0x7f2ed80ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.279+0000 7f2edca0b640 1 -- 172.21.15.29:0/626496979 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed80c9780 con 0x7f2ed815ad70 2023-10-14T17:00:45.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.281+0000 7f2edca0b640 1 --2- 172.21.15.29:0/626496979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed8069da0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0097a0 tx=0x7f2ecc02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e110ef20b9871a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:45.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.281+0000 7f2ed77fe640 1 -- 172.21.15.29:0/626496979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc030610 con 0x7f2ed815ad70 2023-10-14T17:00:45.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.281+0000 7f2ed77fe640 1 -- 172.21.15.29:0/626496979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ecc030770 con 0x7f2ed815ad70 2023-10-14T17:00:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.281+0000 7f2ed77fe640 1 -- 172.21.15.29:0/626496979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ecc030b00 con 0x7f2ed815ad70 2023-10-14T17:00:45.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 -- 172.21.15.29:0/626496979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 msgr2=0x7f2ed8069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 --2- 172.21.15.29:0/626496979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed8069da0 secure :-1 s=READY pgs=1644 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc0097a0 tx=0x7f2ecc02b960 comp rx=0 tx=0).stop 2023-10-14T17:00:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 -- 172.21.15.29:0/626496979 shutdown_connections 2023-10-14T17:00:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 --2- 172.21.15.29:0/626496979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed8069da0 unknown :-1 s=CLOSED pgs=1644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 -- 172.21.15.29:0/626496979 >> 172.21.15.29:0/626496979 conn(0x7f2ed80ce9b0 msgr2=0x7f2ed80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:45.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 -- 172.21.15.29:0/626496979 shutdown_connections 2023-10-14T17:00:45.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.282+0000 7f2edec6f640 1 -- 172.21.15.29:0/626496979 wait complete. 2023-10-14T17:00:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.284+0000 7f2edec6f640 1 Processor -- start 2023-10-14T17:00:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.284+0000 7f2edec6f640 1 -- start start 2023-10-14T17:00:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed80c9ca0 0x7f2ed81f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:45.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 0x7f2ed80ad230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed815c040 con 0x7f2ed815ad70 2023-10-14T17:00:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed815b450 con 0x7f2ed80caac0 2023-10-14T17:00:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.285+0000 7f2edec6f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed81f2470 con 0x7f2ed80c9ca0 2023-10-14T17:00:45.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:45.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53362/0 (socket says 172.21.15.29:53362) 2023-10-14T17:00:45.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 -- 172.21.15.29:0/4496518 learned_addr learned my addr 172.21.15.29:0/4496518 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:45.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2ed7fff640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 0x7f2ed80ad230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:45.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edca0b640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed80c9ca0 0x7f2ed81f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:45.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 -- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed80c9ca0 msgr2=0x7f2ed81f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed80c9ca0 0x7f2ed81f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 -- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 msgr2=0x7f2ed80ad230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 0x7f2ed80ad230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 -- 172.21.15.29:0/4496518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ecc009400 con 0x7f2ed815ad70 2023-10-14T17:00:45.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2ed7fff640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 0x7f2ed80ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:45.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.286+0000 7f2edd20c640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000cbf0 tx=0x7f2ed000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.287+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000ad40 con 0x7f2ed815ad70 2023-10-14T17:00:45.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.287+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed0012400 con 0x7f2ed815ad70 2023-10-14T17:00:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.287+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed000aea0 con 0x7f2ed815ad70 2023-10-14T17:00:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.287+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed815b6a0 con 0x7f2ed815ad70 2023-10-14T17:00:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.287+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed815bc60 con 0x7f2ed815ad70 2023-10-14T17:00:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.288+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed815c180 con 0x7f2ed815ad70 2023-10-14T17:00:45.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.289+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f2ed0007070 con 0x7f2ed815ad70 2023-10-14T17:00:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.290+0000 7f2ed57fa640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 0x7f2ebc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.290+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f2ed0089760 con 0x7f2ed815ad70 2023-10-14T17:00:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.290+0000 7f2edca0b640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 0x7f2ebc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:45.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.291+0000 7f2edca0b640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 0x7f2ebc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc036000 tx=0x7f2ecc03b040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:45.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.295+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ed815c180 con 0x7f2ed815ad70 2023-10-14T17:00:45.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.517+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/weekly", "snap_schedule": "1w", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ed806a2e0 con 0x7f2ebc06afe0 2023-10-14T17:00:45.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.701+0000 7f2ed57fa640 1 -- 172.21.15.29:0/4496518 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7f2ecc030410 con 0x7f2ebc06afe0 2023-10-14T17:00:45.702 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/weekly 2023-10-14T17:00:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.703+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 msgr2=0x7f2ebc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.703+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 0x7f2ebc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f2ecc036000 tx=0x7f2ecc03b040 comp rx=0 tx=0).stop 2023-10-14T17:00:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.703+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 msgr2=0x7f2ed815dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.703+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 secure :-1 s=READY pgs=1645 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000cbf0 tx=0x7f2ed000a430 comp rx=0 tx=0).stop 2023-10-14T17:00:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 shutdown_connections 2023-10-14T17:00:45.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f2ebc06afe0 0x7f2ebc06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ed815ad70 0x7f2ed815dad0 unknown :-1 s=CLOSED pgs=1645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ed80caac0 0x7f2ed80ad230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 --2- 172.21.15.29:0/4496518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ed80c9ca0 0x7f2ed81f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 >> 172.21.15.29:0/4496518 conn(0x7f2ed80ce9b0 msgr2=0x7f2ed8159f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:45.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 shutdown_connections 2023-10-14T17:00:45.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:45.704+0000 7f2edec6f640 1 -- 172.21.15.29:0/4496518 wait complete. 2023-10-14T17:00:45.728 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/weekly 2023-10-14T17:00:45.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:45.729 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/monthly) 2023-10-14T17:00:45.789 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/monthly 1M 2023-10-14T17:00:46.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.024+0000 7fade5290640 1 Processor -- start 2023-10-14T17:00:46.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.024+0000 7fade5290640 1 -- start start 2023-10-14T17:00:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7fade5290640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fade015ad70 0x7fade0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7fade5290640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7fade5290640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade00cb030 con 0x7fade00c9ca0 2023-10-14T17:00:46.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7fade5290640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade00c91c0 con 0x7fade015ad70 2023-10-14T17:00:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7fade5290640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c94a0 con 0x7fade00caac0 2023-10-14T17:00:46.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7faddeffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fade015ad70 0x7fade0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7faddf7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7faddf7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:44994/0 (socket says 172.21.15.29:44994) 2023-10-14T17:00:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.026+0000 7faddf7fe640 1 -- 172.21.15.29:0/2716651666 learned_addr learned my addr 172.21.15.29:0/2716651666 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:46.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.027+0000 7faddf7fe640 1 -- 172.21.15.29:0/2716651666 >> v2:172.21.15.186:3300/0 conn(0x7fade015ad70 msgr2=0x7fade0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.027+0000 7faddf7fe640 1 --2- 172.21.15.29:0/2716651666 >> v2:172.21.15.186:3300/0 conn(0x7fade015ad70 0x7fade0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.027+0000 7faddf7fe640 1 -- 172.21.15.29:0/2716651666 >> v1:172.21.15.186:6789/0 conn(0x7fade00c9ca0 legacy=0x7fade00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.027+0000 7faddf7fe640 1 -- 172.21.15.29:0/2716651666 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade00c9780 con 0x7fade00caac0 2023-10-14T17:00:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.028+0000 7faddf7fe640 1 --2- 172.21.15.29:0/2716651666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009870 tx=0x7fadd802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff815b8b8d40f2b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.028+0000 7fadddffb640 1 -- 172.21.15.29:0/2716651666 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8030840 con 0x7fade00caac0 2023-10-14T17:00:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fadddffb640 1 -- 172.21.15.29:0/2716651666 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd80309a0 con 0x7fade00caac0 2023-10-14T17:00:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fadddffb640 1 -- 172.21.15.29:0/2716651666 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd8030d30 con 0x7fade00caac0 2023-10-14T17:00:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 -- 172.21.15.29:0/2716651666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 msgr2=0x7fade006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 --2- 172.21.15.29:0/2716651666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fadd8009870 tx=0x7fadd802b980 comp rx=0 tx=0).stop 2023-10-14T17:00:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 -- 172.21.15.29:0/2716651666 shutdown_connections 2023-10-14T17:00:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 --2- 172.21.15.29:0/2716651666 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade00caac0 0x7fade006a2e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 --2- 172.21.15.29:0/2716651666 >> v2:172.21.15.186:3300/0 conn(0x7fade015ad70 0x7fade0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 -- 172.21.15.29:0/2716651666 >> 172.21.15.29:0/2716651666 conn(0x7fade00ce9b0 msgr2=0x7fade00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:46.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 -- 172.21.15.29:0/2716651666 shutdown_connections 2023-10-14T17:00:46.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.029+0000 7fade5290640 1 -- 172.21.15.29:0/2716651666 wait complete. 2023-10-14T17:00:46.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.031+0000 7fade5290640 1 Processor -- start 2023-10-14T17:00:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.031+0000 7fade5290640 1 -- start start 2023-10-14T17:00:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fade00caac0 0x7fade01f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade015ad70 0x7fade00c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c7e80 con 0x7fade00c9ca0 2023-10-14T17:00:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c7010 con 0x7fade00caac0 2023-10-14T17:00:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.032+0000 7fade5290640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade00c7270 con 0x7fade015ad70 2023-10-14T17:00:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53364/0 (socket says 172.21.15.29:53364) 2023-10-14T17:00:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 -- 172.21.15.29:0/3396699031 learned_addr learned my addr 172.21.15.29:0/3396699031 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7fadde7fc640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fade00caac0 0x7fade01f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddf7fe640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade015ad70 0x7fade00c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 -- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade015ad70 msgr2=0x7fade00c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade015ad70 0x7fade00c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 -- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fade00caac0 msgr2=0x7fade01f2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fade00caac0 0x7fade01f2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.033+0000 7faddeffd640 1 -- 172.21.15.29:0/3396699031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd80094d0 con 0x7fade00c9ca0 2023-10-14T17:00:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7faddeffd640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fadd000efc0 tx=0x7fadd000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd0009b80 con 0x7fade00c9ca0 2023-10-14T17:00:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd000f040 con 0x7fade00c9ca0 2023-10-14T17:00:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fadd00146a0 con 0x7fade00c9ca0 2023-10-14T17:00:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade00c7550 con 0x7fade00c9ca0 2023-10-14T17:00:46.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.034+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade00c7b10 con 0x7fade00c9ca0 2023-10-14T17:00:46.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.035+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade00ad480 con 0x7fade00c9ca0 2023-10-14T17:00:46.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.036+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 74) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fadd0014800 con 0x7fade00c9ca0 2023-10-14T17:00:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.038+0000 7fadd77fe640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 0x7fadc006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.038+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fadd008a890 con 0x7fade00c9ca0 2023-10-14T17:00:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.038+0000 7fadde7fc640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 0x7fadc006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.039+0000 7fadde7fc640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 0x7fadc006d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fade015a2f0 tx=0x7fadc8009040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.042+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fadd0055570 con 0x7fade00c9ca0 2023-10-14T17:00:46.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.261+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/monthly", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fade00c7fc0 con 0x7fadc006b010 2023-10-14T17:00:46.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.299+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fadd0055940 con 0x7fade00c9ca0 2023-10-14T17:00:46.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.458+0000 7fadd77fe640 1 -- 172.21.15.29:0/3396699031 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+45 (secure 0 0 0) 0x7fadc8001360 con 0x7fadc006b010 2023-10-14T17:00:46.459 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/monthly 2023-10-14T17:00:46.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 msgr2=0x7fadc006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 0x7fadc006d4d0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fade015a2f0 tx=0x7fadc8009040 comp rx=0 tx=0).stop 2023-10-14T17:00:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 msgr2=0x7fade00b31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 secure :-1 s=READY pgs=1646 cs=0 l=1 rev1=1 crypto rx=0x7fadd000efc0 tx=0x7fadd000c490 comp rx=0 tx=0).stop 2023-10-14T17:00:46.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 shutdown_connections 2023-10-14T17:00:46.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fadc006b010 0x7fadc006d4d0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fade015ad70 0x7fade00c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fade00caac0 0x7fade01f2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 --2- 172.21.15.29:0/3396699031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fade00c9ca0 0x7fade00b31a0 unknown :-1 s=CLOSED pgs=1646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 >> 172.21.15.29:0/3396699031 conn(0x7fade00ce9b0 msgr2=0x7fade00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:46.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 shutdown_connections 2023-10-14T17:00:46.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.461+0000 7fade5290640 1 -- 172.21.15.29:0/3396699031 wait complete. 2023-10-14T17:00:46.484 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/monthly 2023-10-14T17:00:46.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:46.485 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/yearly) 2023-10-14T17:00:46.543 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/yearly 1Y 2023-10-14T17:00:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.767+0000 7fcee4901640 1 Processor -- start 2023-10-14T17:00:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.768+0000 7fcee4901640 1 -- start start 2023-10-14T17:00:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.769+0000 7fcee4901640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcee006a4b0 0x7fcee00c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.769+0000 7fcee4901640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee00ac890 con 0x7fcee00ad350 2023-10-14T17:00:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.769+0000 7fcee4901640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee00ad7a0 con 0x7fcee015d150 2023-10-14T17:00:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.769+0000 7fcee4901640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcee015d530 con 0x7fcee006a4b0 2023-10-14T17:00:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.770+0000 7fced5d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcee00ad350 0x7fcee0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53530/0 (socket says 172.21.15.29:53530) 2023-10-14T17:00:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.770+0000 7fced5d9b640 1 -- 172.21.15.29:0/2839315669 learned_addr learned my addr 172.21.15.29:0/2839315669 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.770+0000 7fceded9d640 1 --2- 172.21.15.29:0/2839315669 >> v2:172.21.15.29:3300/0 conn(0x7fcee006a4b0 0x7fcee00c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.770+0000 7fceddd9b640 1 -- 172.21.15.29:0/2839315669 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078302117 0 0) 0x7fcee00ac890 con 0x7fcee00ad350 2023-10-14T17:00:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.770+0000 7fceddd9b640 1 -- 172.21.15.29:0/2839315669 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc003520 con 0x7fcee00ad350 2023-10-14T17:00:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.771+0000 7fceded9d640 1 -- 172.21.15.29:0/2839315669 >> v1:172.21.15.186:6789/0 conn(0x7fcee015d150 legacy=0x7fcee0160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.771+0000 7fceded9d640 1 -- 172.21.15.29:0/2839315669 >> v1:172.21.15.29:6789/0 conn(0x7fcee00ad350 legacy=0x7fcee0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.771+0000 7fceded9d640 1 -- 172.21.15.29:0/2839315669 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee0161650 con 0x7fcee006a4b0 2023-10-14T17:00:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.772+0000 7fceded9d640 1 --2- 172.21.15.29:0/2839315669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee00c9000 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fced80097a0 tx=0x7fced802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24930a25d0da87b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.772+0000 7fceddd9b640 1 -- 172.21.15.29:0/2839315669 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8030610 con 0x7fcee006a4b0 2023-10-14T17:00:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.772+0000 7fceddd9b640 1 -- 172.21.15.29:0/2839315669 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced8030770 con 0x7fcee006a4b0 2023-10-14T17:00:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.772+0000 7fceddd9b640 1 -- 172.21.15.29:0/2839315669 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced8030b00 con 0x7fcee006a4b0 2023-10-14T17:00:46.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 -- 172.21.15.29:0/2839315669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 msgr2=0x7fcee00c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 --2- 172.21.15.29:0/2839315669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee00c9000 secure :-1 s=READY pgs=1648 cs=0 l=1 rev1=1 crypto rx=0x7fced80097a0 tx=0x7fced802b960 comp rx=0 tx=0).stop 2023-10-14T17:00:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 -- 172.21.15.29:0/2839315669 shutdown_connections 2023-10-14T17:00:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 --2- 172.21.15.29:0/2839315669 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee00c9000 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 -- 172.21.15.29:0/2839315669 >> 172.21.15.29:0/2839315669 conn(0x7fcee00ce9b0 msgr2=0x7fcee00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 -- 172.21.15.29:0/2839315669 shutdown_connections 2023-10-14T17:00:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.773+0000 7fcee4901640 1 -- 172.21.15.29:0/2839315669 wait complete. 2023-10-14T17:00:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.775+0000 7fcee4901640 1 Processor -- start 2023-10-14T17:00:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.775+0000 7fcee4901640 1 -- start start 2023-10-14T17:00:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 0x7fcee00c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcee015d150 0x7fcee01592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c7150 con 0x7fcee006a4b0 2023-10-14T17:00:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c6540 con 0x7fcee00ad350 2023-10-14T17:00:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcee4901640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcee00c6820 con 0x7fcee015d150 2023-10-14T17:00:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcede59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcede59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53378/0 (socket says 172.21.15.29:53378) 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fcede59c640 1 -- 172.21.15.29:0/1238008813 learned_addr learned my addr 172.21.15.29:0/1238008813 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.776+0000 7fceded9d640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcee015d150 0x7fcee01592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fced5d9b640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 0x7fcee00c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 -- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcee015d150 msgr2=0x7fcee01592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcee015d150 0x7fcee01592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 -- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 msgr2=0x7fcee00c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 0x7fcee00c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 -- 172.21.15.29:0/1238008813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced8009400 con 0x7fcee006a4b0 2023-10-14T17:00:46.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fced5d9b640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 0x7fcee00c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcede59c640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fced000c870 tx=0x7fced000cd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0003b70 con 0x7fcee006a4b0 2023-10-14T17:00:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0003cf0 con 0x7fcee006a4b0 2023-10-14T17:00:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced000aa90 con 0x7fcee006a4b0 2023-10-14T17:00:46.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.777+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee00c6b00 con 0x7fcee006a4b0 2023-10-14T17:00:46.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.778+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee00aff40 con 0x7fcee006a4b0 2023-10-14T17:00:46.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.779+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee00c58d0 con 0x7fcee006a4b0 2023-10-14T17:00:46.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.780+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fced000abf0 con 0x7fcee006a4b0 2023-10-14T17:00:46.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.780+0000 7fced77fe640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 0x7fcec006d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:46.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.781+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fced0089a10 con 0x7fcee006a4b0 2023-10-14T17:00:46.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.781+0000 7fced5d9b640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 0x7fcec006d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:46.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.782+0000 7fced5d9b640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 0x7fcec006d560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcee00b4900 tx=0x7fcec8009040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:46.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:46.785+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fced0054610 con 0x7fcee006a4b0 2023-10-14T17:00:46.919 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:00:46.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:00:47.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.005+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/yearly", "snap_schedule": "1Y", "target": ["mon-mgr", ""]}) v1 -- 0x7fcee00acb50 con 0x7fcec006b0a0 2023-10-14T17:00:47.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.178+0000 7fced77fe640 1 -- 172.21.15.29:0/1238008813 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+44 (secure 0 0 0) 0x7fcec8001d20 con 0x7fcec006b0a0 2023-10-14T17:00:47.180 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1/yearly 2023-10-14T17:00:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 msgr2=0x7fcec006d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 0x7fcec006d560 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fcee00b4900 tx=0x7fcec8009040 comp rx=0 tx=0).stop 2023-10-14T17:00:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 msgr2=0x7fcee0154f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 secure :-1 s=READY pgs=1649 cs=0 l=1 rev1=1 crypto rx=0x7fced000c870 tx=0x7fced000cd30 comp rx=0 tx=0).stop 2023-10-14T17:00:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 shutdown_connections 2023-10-14T17:00:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcec006b0a0 0x7fcec006d560 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcee015d150 0x7fcee01592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcee00ad350 0x7fcee00c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 --2- 172.21.15.29:0/1238008813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcee006a4b0 0x7fcee0154f20 unknown :-1 s=CLOSED pgs=1649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 >> 172.21.15.29:0/1238008813 conn(0x7fcee00ce9b0 msgr2=0x7fcee0156070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 shutdown_connections 2023-10-14T17:00:47.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.182+0000 7fcee4901640 1 -- 172.21.15.29:0/1238008813 wait complete. 2023-10-14T17:00:47.204 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1/yearly 2023-10-14T17:00:47.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:47.205 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/bad_period_spec) 2023-10-14T17:00:47.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1X 2023-10-14T17:00:47.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.495+0000 7fccc7df5640 1 Processor -- start 2023-10-14T17:00:47.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.495+0000 7fccc7df5640 1 -- start start 2023-10-14T17:00:47.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc7df5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fccc006a4b0 0x7fccc00ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc7df5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccc00c9ca0 0x7fccc00ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc7df5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00cb030 con 0x7fccc015ad70 2023-10-14T17:00:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc7df5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc00cae30 con 0x7fccc00c9ca0 2023-10-14T17:00:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc7df5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccc00c91c0 con 0x7fccc006a4b0 2023-10-14T17:00:47.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.497+0000 7fccc6392640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccc00c9ca0 0x7fccc00ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fccc00c9ca0 0x7fccc00ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53384/0 (socket says 172.21.15.29:53384) 2023-10-14T17:00:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 -- 172.21.15.29:0/1088462787 learned_addr learned my addr 172.21.15.29:0/1088462787 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc5390640 1 --2- 172.21.15.29:0/1088462787 >> v2:172.21.15.186:3300/0 conn(0x7fccc006a4b0 0x7fccc00ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 -- 172.21.15.29:0/1088462787 >> v2:172.21.15.186:3300/0 conn(0x7fccc006a4b0 msgr2=0x7fccc00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:47.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 --2- 172.21.15.29:0/1088462787 >> v2:172.21.15.186:3300/0 conn(0x7fccc006a4b0 0x7fccc00ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 -- 172.21.15.29:0/1088462787 >> v1:172.21.15.186:6789/0 conn(0x7fccc015ad70 legacy=0x7fccc0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:47.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.498+0000 7fccc6392640 1 -- 172.21.15.29:0/1088462787 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc00c94a0 con 0x7fccc00c9ca0 2023-10-14T17:00:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.499+0000 7fccc6392640 1 --2- 172.21.15.29:0/1088462787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc00c9ca0 0x7fccc00ca8f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ce870 tx=0x7fccb802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5390e7114e221be0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.499+0000 7fccc4b8f640 1 -- 172.21.15.29:0/1088462787 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb802ca80 con 0x7fccc00c9ca0 2023-10-14T17:00:47.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc4b8f640 1 -- 172.21.15.29:0/1088462787 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb802cbe0 con 0x7fccc00c9ca0 2023-10-14T17:00:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc4b8f640 1 -- 172.21.15.29:0/1088462787 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb80343f0 con 0x7fccc00c9ca0 2023-10-14T17:00:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 -- 172.21.15.29:0/1088462787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc00c9ca0 msgr2=0x7fccc00ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 --2- 172.21.15.29:0/1088462787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc00c9ca0 0x7fccc00ca8f0 secure :-1 s=READY pgs=1650 cs=0 l=1 rev1=1 crypto rx=0x7fccc00ce870 tx=0x7fccb802bee0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 -- 172.21.15.29:0/1088462787 shutdown_connections 2023-10-14T17:00:47.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 --2- 172.21.15.29:0/1088462787 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc00c9ca0 0x7fccc00ca8f0 unknown :-1 s=CLOSED pgs=1650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 --2- 172.21.15.29:0/1088462787 >> v2:172.21.15.186:3300/0 conn(0x7fccc006a4b0 0x7fccc00ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 -- 172.21.15.29:0/1088462787 >> 172.21.15.29:0/1088462787 conn(0x7fccc00ce9b0 msgr2=0x7fccc00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 -- 172.21.15.29:0/1088462787 shutdown_connections 2023-10-14T17:00:47.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.500+0000 7fccc7df5640 1 -- 172.21.15.29:0/1088462787 wait complete. 2023-10-14T17:00:47.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.502+0000 7fccc7df5640 1 Processor -- start 2023-10-14T17:00:47.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.502+0000 7fccc7df5640 1 -- start start 2023-10-14T17:00:47.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc006a4b0 0x7fccc0151000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc00c9ca0 0x7fccc00ad210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc015ad70 0x7fccc00ad750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00cde40 con 0x7fccc015ad70 2023-10-14T17:00:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00ccf90 con 0x7fccc006a4b0 2023-10-14T17:00:47.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc7df5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccc00cd230 con 0x7fccc00c9ca0 2023-10-14T17:00:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc6392640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc015ad70 0x7fccc00ad750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.503+0000 7fccc6392640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc015ad70 0x7fccc00ad750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53398/0 (socket says 172.21.15.29:53398) 2023-10-14T17:00:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 -- 172.21.15.29:0/3439458898 learned_addr learned my addr 172.21.15.29:0/3439458898 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 -- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc00c9ca0 msgr2=0x7fccc00ad210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fccc00c9ca0 0x7fccc00ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 -- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc006a4b0 msgr2=0x7fccc0151000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc5b91640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc006a4b0 0x7fccc0151000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc006a4b0 0x7fccc0151000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 -- 172.21.15.29:0/3439458898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb8009b30 con 0x7fccc015ad70 2023-10-14T17:00:47.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc5b91640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fccc006a4b0 0x7fccc0151000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:47.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.504+0000 7fccc6392640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fccc015ad70 0x7fccc00ad750 secure :-1 s=READY pgs=1651 cs=0 l=1 rev1=1 crypto rx=0x7fccb8036040 tx=0x7fccb8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:47.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.505+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb8009510 con 0x7fccc015ad70 2023-10-14T17:00:47.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.505+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccb803e450 con 0x7fccc015ad70 2023-10-14T17:00:47.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.505+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fccb803e7e0 con 0x7fccc015ad70 2023-10-14T17:00:47.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.505+0000 7fccc7df5640 1 -- 172.21.15.29:0/3439458898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc00cd510 con 0x7fccc015ad70 2023-10-14T17:00:47.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.505+0000 7fccc7df5640 1 -- 172.21.15.29:0/3439458898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc00cdad0 con 0x7fccc015ad70 2023-10-14T17:00:47.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.506+0000 7fccc7df5640 1 -- 172.21.15.29:0/3439458898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc00cdff0 con 0x7fccc015ad70 2023-10-14T17:00:47.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.507+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fccb802cd50 con 0x7fccc015ad70 2023-10-14T17:00:47.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.508+0000 7fccb67fc640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcca406b0a0 0x7fcca406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.508+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fccb80ac440 con 0x7fccc015ad70 2023-10-14T17:00:47.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.508+0000 7fccc5b91640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcca406b0a0 0x7fcca406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.509+0000 7fccc5b91640 1 --2- 172.21.15.29:0/3439458898 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcca406b0a0 0x7fcca406d560 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fccbc00a830 tx=0x7fccbc008040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:47.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.512+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fccc00cdff0 con 0x7fccc015ad70 2023-10-14T17:00:47.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.735+0000 7fccc7df5640 1 -- 172.21.15.29:0/3439458898 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1X", "target": ["mon-mgr", ""]}) v1 -- 0x7fccc00c9780 con 0x7fcca406b0a0 2023-10-14T17:00:47.738 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:47.737+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-14T17:00:47.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.738+0000 7fccb67fc640 1 -- 172.21.15.29:0/3439458898 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - multiplier should be one of h,d,w,M,Y) v1 ==== 74+0+0 (secure 0 0 0) 0x7fccbc00adb0 con 0x7fcca406b0a0 2023-10-14T17:00:47.739 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - multiplier should be one of h,d,w,M,Y 2023-10-14T17:00:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:47.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1MM 2023-10-14T17:00:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.982+0000 7fbfd887b640 1 Processor -- start 2023-10-14T17:00:47.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.983+0000 7fbfd887b640 1 -- start start 2023-10-14T17:00:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.984+0000 7fbfd887b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd406c040 0x7fbfd406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.984+0000 7fbfd887b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbfd40ad970 0x7fbfd415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.984+0000 7fbfd887b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfd406c960 con 0x7fbfd415ebc0 2023-10-14T17:00:47.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.984+0000 7fbfd887b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfd415b9d0 con 0x7fbfd40ad970 2023-10-14T17:00:47.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.984+0000 7fbfd887b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd415bcb0 con 0x7fbfd406c040 2023-10-14T17:00:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd359e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbfd415ebc0 0x7fbfd40cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53544/0 (socket says 172.21.15.29:53544) 2023-10-14T17:00:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd359e640 1 -- 172.21.15.29:0/1706914489 learned_addr learned my addr 172.21.15.29:0/1706914489 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd2d9d640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd406c040 0x7fbfd406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd259c640 1 --2- 172.21.15.29:0/1706914489 >> v2:172.21.15.29:3300/0 conn(0x7fbfd40ad970 0x7fbfd415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd1d9b640 1 -- 172.21.15.29:0/1706914489 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317395208 0 0) 0x7fbfd406c960 con 0x7fbfd415ebc0 2023-10-14T17:00:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.985+0000 7fbfd1d9b640 1 -- 172.21.15.29:0/1706914489 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0003520 con 0x7fbfd415ebc0 2023-10-14T17:00:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.986+0000 7fbfd259c640 1 -- 172.21.15.29:0/1706914489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd406c040 msgr2=0x7fbfd406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:47.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.986+0000 7fbfd259c640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd406c040 0x7fbfd406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.986+0000 7fbfd259c640 1 -- 172.21.15.29:0/1706914489 >> v1:172.21.15.29:6789/0 conn(0x7fbfd415ebc0 legacy=0x7fbfd40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.986+0000 7fbfd259c640 1 -- 172.21.15.29:0/1706914489 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd415bf90 con 0x7fbfd40ad970 2023-10-14T17:00:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.987+0000 7fbfd259c640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415e680 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0097d0 tx=0x7fbfbc02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16cbc1185ab5b39e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:47.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.987+0000 7fbfd1d9b640 1 -- 172.21.15.29:0/1706914489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc02d040 con 0x7fbfd40ad970 2023-10-14T17:00:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.987+0000 7fbfd1d9b640 1 -- 172.21.15.29:0/1706914489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc031810 con 0x7fbfd40ad970 2023-10-14T17:00:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.987+0000 7fbfd1d9b640 1 -- 172.21.15.29:0/1706914489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc031ba0 con 0x7fbfd40ad970 2023-10-14T17:00:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 -- 172.21.15.29:0/1706914489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 msgr2=0x7fbfd415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415e680 secure :-1 s=READY pgs=1653 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc0097d0 tx=0x7fbfbc02b8b0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 -- 172.21.15.29:0/1706914489 shutdown_connections 2023-10-14T17:00:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415e680 unknown :-1 s=CLOSED pgs=1653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 --2- 172.21.15.29:0/1706914489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd406c040 0x7fbfd406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 -- 172.21.15.29:0/1706914489 >> 172.21.15.29:0/1706914489 conn(0x7fbfd40ce6a0 msgr2=0x7fbfd40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 -- 172.21.15.29:0/1706914489 shutdown_connections 2023-10-14T17:00:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.988+0000 7fbfd887b640 1 -- 172.21.15.29:0/1706914489 wait complete. 2023-10-14T17:00:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.990+0000 7fbfd887b640 1 Processor -- start 2023-10-14T17:00:47.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.990+0000 7fbfd887b640 1 -- start start 2023-10-14T17:00:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfd406c040 0x7fbfd40c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd415ebc0 0x7fbfd4159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40c7950 con 0x7fbfd40ad970 2023-10-14T17:00:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40b0e70 con 0x7fbfd406c040 2023-10-14T17:00:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd887b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd40b0b90 con 0x7fbfd415ebc0 2023-10-14T17:00:47.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53424/0 (socket says 172.21.15.29:53424) 2023-10-14T17:00:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd259c640 1 -- 172.21.15.29:0/329422994 learned_addr learned my addr 172.21.15.29:0/329422994 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd359e640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd415ebc0 0x7fbfd4159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd259c640 1 -- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd415ebc0 msgr2=0x7fbfd4159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.991+0000 7fbfd2d9d640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfd406c040 0x7fbfd40c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd259c640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd415ebc0 0x7fbfd4159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd259c640 1 -- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfd406c040 msgr2=0x7fbfd40c7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd259c640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbfd406c040 0x7fbfd40c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:47.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd259c640 1 -- 172.21.15.29:0/329422994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfbc009400 con 0x7fbfd40ad970 2023-10-14T17:00:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd359e640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbfd415ebc0 0x7fbfd4159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd259c640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbfd40ad970 0x7fbfd415aad0 secure :-1 s=READY pgs=1654 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc000c00 tx=0x7fbfbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc03ad60 con 0x7fbfd40ad970 2023-10-14T17:00:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfbc0443f0 con 0x7fbfd40ad970 2023-10-14T17:00:48.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfbc0445a0 con 0x7fbfd40ad970 2023-10-14T17:00:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd887b640 1 -- 172.21.15.29:0/329422994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd40c7300 con 0x7fbfd40ad970 2023-10-14T17:00:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.992+0000 7fbfd887b640 1 -- 172.21.15.29:0/329422994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd415a230 con 0x7fbfd40ad970 2023-10-14T17:00:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.994+0000 7fbfd887b640 1 -- 172.21.15.29:0/329422994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd406b330 con 0x7fbfd40ad970 2023-10-14T17:00:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.994+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fbfbc04d450 con 0x7fbfd40ad970 2023-10-14T17:00:48.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.995+0000 7fbfcb7fe640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fbfb406b0a0 0x7fbfb406d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.995+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fbfbc0abd20 con 0x7fbfd40ad970 2023-10-14T17:00:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.996+0000 7fbfd2d9d640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fbfb406b0a0 0x7fbfb406d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:47.997+0000 7fbfd2d9d640 1 --2- 172.21.15.29:0/329422994 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fbfb406b0a0 0x7fbfb406d560 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fbfd41f5e90 tx=0x7fbfc400b040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.000+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbfbc076890 con 0x7fbfd40ad970 2023-10-14T17:00:48.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.222+0000 7fbfd887b640 1 -- 172.21.15.29:0/329422994 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1MM", "target": ["mon-mgr", ""]}) v1 -- 0x7fbfd40cc550 con 0x7fbfb406b0a0 2023-10-14T17:00:48.224 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:48.224+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.224+0000 7fbfcb7fe640 1 -- 172.21.15.29:0/329422994 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fbfc4009cb0 con 0x7fbfb406b0a0 2023-10-14T17:00:48.225 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:48.246 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:48.248 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec 1 2023-10-14T17:00:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.485+0000 7f6b183e2640 1 Processor -- start 2023-10-14T17:00:48.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.485+0000 7f6b183e2640 1 -- start start 2023-10-14T17:00:48.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b183e2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b100ccf80 0x7f6b10070660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b183e2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b1006a770 con 0x7f6b1015ab70 2023-10-14T17:00:48.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b183e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b100af0a0 con 0x7f6b1006cd70 2023-10-14T17:00:48.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b183e2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b100c6930 con 0x7f6b100ccf80 2023-10-14T17:00:48.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b1697f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b100ccf80 0x7f6b10070660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b1697f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b100ccf80 0x7f6b10070660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53436/0 (socket says 172.21.15.29:53436) 2023-10-14T17:00:48.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.487+0000 7f6b1697f640 1 -- 172.21.15.29:0/4175473809 learned_addr learned my addr 172.21.15.29:0/4175473809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.488+0000 7f6b1697f640 1 -- 172.21.15.29:0/4175473809 >> v1:172.21.15.186:6789/0 conn(0x7f6b1006cd70 legacy=0x7f6b1015b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.488+0000 7f6b1697f640 1 -- 172.21.15.29:0/4175473809 >> v1:172.21.15.29:6789/0 conn(0x7f6b1015ab70 legacy=0x7f6b1006c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.488+0000 7f6b1697f640 1 -- 172.21.15.29:0/4175473809 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b10069e40 con 0x7f6b100ccf80 2023-10-14T17:00:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.489+0000 7f6b1697f640 1 --2- 172.21.15.29:0/4175473809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b100ccf80 0x7f6b10070660 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009e50 tx=0x7f6b0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ac7507a5bcff79 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.489+0000 7f6b1517c640 1 -- 172.21.15.29:0/4175473809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0802cb10 con 0x7f6b100ccf80 2023-10-14T17:00:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.489+0000 7f6b1517c640 1 -- 172.21.15.29:0/4175473809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0802cc70 con 0x7f6b100ccf80 2023-10-14T17:00:48.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.489+0000 7f6b1517c640 1 -- 172.21.15.29:0/4175473809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b080343f0 con 0x7f6b100ccf80 2023-10-14T17:00:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 -- 172.21.15.29:0/4175473809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b100ccf80 msgr2=0x7f6b10070660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 --2- 172.21.15.29:0/4175473809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b100ccf80 0x7f6b10070660 secure :-1 s=READY pgs=1656 cs=0 l=1 rev1=1 crypto rx=0x7f6b08009e50 tx=0x7f6b0802bf70 comp rx=0 tx=0).stop 2023-10-14T17:00:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 -- 172.21.15.29:0/4175473809 shutdown_connections 2023-10-14T17:00:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 --2- 172.21.15.29:0/4175473809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b100ccf80 0x7f6b10070660 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 -- 172.21.15.29:0/4175473809 >> 172.21.15.29:0/4175473809 conn(0x7f6b100cc460 msgr2=0x7f6b100cc870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:48.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 -- 172.21.15.29:0/4175473809 shutdown_connections 2023-10-14T17:00:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.490+0000 7f6b183e2640 1 -- 172.21.15.29:0/4175473809 wait complete. 2023-10-14T17:00:48.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.492+0000 7f6b183e2640 1 Processor -- start 2023-10-14T17:00:48.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.492+0000 7f6b183e2640 1 -- start start 2023-10-14T17:00:48.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1006cd70 0x7f6b100c7710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b100ccf80 0x7f6b10157320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015ab70 0x7f6b100c9140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b10157860 con 0x7f6b1015ab70 2023-10-14T17:00:48.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100c9680 con 0x7f6b1006cd70 2023-10-14T17:00:48.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.493+0000 7f6b183e2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b100b4b40 con 0x7f6b100ccf80 2023-10-14T17:00:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015ab70 0x7f6b100c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015ab70 0x7f6b100c9140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53450/0 (socket says 172.21.15.29:53450) 2023-10-14T17:00:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 -- 172.21.15.29:0/3220262682 learned_addr learned my addr 172.21.15.29:0/3220262682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 -- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b100ccf80 msgr2=0x7f6b10157320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1597d640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b100ccf80 0x7f6b10157320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1617e640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1006cd70 0x7f6b100c7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b100ccf80 0x7f6b10157320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 -- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1006cd70 msgr2=0x7f6b100c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1006cd70 0x7f6b100c7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1697f640 1 -- 172.21.15.29:0/3220262682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b08009b30 con 0x7f6b1015ab70 2023-10-14T17:00:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.494+0000 7f6b1617e640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b1006cd70 0x7f6b100c7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b1697f640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b1015ab70 0x7f6b100c9140 secure :-1 s=READY pgs=1657 cs=0 l=1 rev1=1 crypto rx=0x7f6b08036040 tx=0x7f6b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0803e450 con 0x7f6b1015ab70 2023-10-14T17:00:48.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b0803e5d0 con 0x7f6b1015ab70 2023-10-14T17:00:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b0803e980 con 0x7f6b1015ab70 2023-10-14T17:00:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b183e2640 1 -- 172.21.15.29:0/3220262682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b100c6c30 con 0x7f6b1015ab70 2023-10-14T17:00:48.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.495+0000 7f6b183e2640 1 -- 172.21.15.29:0/3220262682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b100c71f0 con 0x7f6b1015ab70 2023-10-14T17:00:48.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.496+0000 7f6b183e2640 1 -- 172.21.15.29:0/3220262682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b1006dc40 con 0x7f6b1015ab70 2023-10-14T17:00:48.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.498+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f6b0803eae0 con 0x7f6b1015ab70 2023-10-14T17:00:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.498+0000 7f6b06ffd640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f6af406b070 0x7f6af406d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.499+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f6b080ac9c0 con 0x7f6b1015ab70 2023-10-14T17:00:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.499+0000 7f6b1617e640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f6af406b070 0x7f6af406d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.499+0000 7f6b1617e640 1 --2- 172.21.15.29:0/3220262682 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f6af406b070 0x7f6af406d530 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6b100c2bb0 tx=0x7f6b0c009040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.503+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b08077770 con 0x7f6b1015ab70 2023-10-14T17:00:48.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.722+0000 7f6b183e2640 1 -- 172.21.15.29:0/3220262682 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6b100c6ab0 con 0x7f6af406b070 2023-10-14T17:00:48.725 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:48.725+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:48.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.725+0000 7f6b06ffd640 1 -- 172.21.15.29:0/3220262682 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6b0c001340 con 0x7f6af406b070 2023-10-14T17:00:48.726 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:48.739 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:48.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec M 2023-10-14T17:00:48.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.973+0000 7f9ac4947640 1 Processor -- start 2023-10-14T17:00:48.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.974+0000 7f9ac4947640 1 -- start start 2023-10-14T17:00:48.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.975+0000 7f9ac4947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9ac00cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.975+0000 7f9ac4947640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c9e0 0x7f9ac00ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.975+0000 7f9ac4947640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ac00ca560 con 0x7f9ac00c9060 2023-10-14T17:00:48.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.975+0000 7f9ac4947640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ac00c9950 con 0x7f9ac006c9e0 2023-10-14T17:00:48.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.975+0000 7f9ac4947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c9c30 con 0x7f9ac006fb50 2023-10-14T17:00:48.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.976+0000 7f9abf7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9ac00c9060 0x7f9ac006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53568/0 (socket says 172.21.15.29:53568) 2023-10-14T17:00:48.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.976+0000 7f9abf7fe640 1 -- 172.21.15.29:0/922987723 learned_addr learned my addr 172.21.15.29:0/922987723 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:48.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.976+0000 7f9abeffd640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9ac00cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.976+0000 7f9abffff640 1 --2- 172.21.15.29:0/922987723 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c9e0 0x7f9ac00ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.976+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460138434 0 0) 0x7f9ac00ca560 con 0x7f9ac00c9060 2023-10-14T17:00:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa8003520 con 0x7f9ac00c9060 2023-10-14T17:00:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 336065038 0 0) 0x7f9aa8003520 con 0x7f9ac00c9060 2023-10-14T17:00:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 msgr2=0x7f9ac00cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9ac00cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c9e0 msgr2=0x7f9ac00ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:48.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 --2- 172.21.15.29:0/922987723 >> v2:172.21.15.186:3300/0 conn(0x7f9ac006c9e0 0x7f9ac00ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00c9f10 con 0x7f9ac00c9060 2023-10-14T17:00:48.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9ab0001670 con 0x7f9ac00c9060 2023-10-14T17:00:48.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 >> v1:172.21.15.29:6789/0 conn(0x7f9ac00c9060 legacy=0x7f9ac006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9aa8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aa8003f40 0x7f9aa80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c9950 con 0x7f9aa8004d00 2023-10-14T17:00:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c9c30 con 0x7f9aa8003f40 2023-10-14T17:00:48.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.977+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aa800b980 con 0x7f9ac006fb50 2023-10-14T17:00:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abf7fe640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9aa8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abeffd640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aa8003f40 0x7f9aa80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 -- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 msgr2=0x7f9aa8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:48.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9ac006fb50 0x7f9aa8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 -- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aa8003f40 msgr2=0x7f9aa80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9aa8003f40 0x7f9aa80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.978+0000 7f9abffff640 1 -- 172.21.15.29:0/922987723 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa8003520 con 0x7f9aa8004d00 2023-10-14T17:00:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.979+0000 7f9abffff640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009f90 tx=0x7f9aa80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58fcedc426c9aa2e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab802cbd0 con 0x7f9aa8004d00 2023-10-14T17:00:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab800f040 con 0x7f9aa8004d00 2023-10-14T17:00:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9abe7fc640 1 -- 172.21.15.29:0/922987723 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8034440 con 0x7f9aa8004d00 2023-10-14T17:00:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9ac4947640 1 -- 172.21.15.29:0/922987723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aa8004d00 msgr2=0x7f9aa8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9ac4947640 1 --2- 172.21.15.29:0/922987723 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9aa8004d00 0x7f9aa8005180 secure :-1 s=READY pgs=1659 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8009f90 tx=0x7f9aa80057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9abffff640 1 -- 172.21.15.29:0/922987723 reap_dead start 2023-10-14T17:00:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9ac4947640 1 -- 172.21.15.29:0/922987723 shutdown_connections 2023-10-14T17:00:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9ac4947640 1 -- 172.21.15.29:0/922987723 >> 172.21.15.29:0/922987723 conn(0x7f9ac00ce6a0 msgr2=0x7f9ac00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.980+0000 7f9ac4947640 1 -- 172.21.15.29:0/922987723 shutdown_connections 2023-10-14T17:00:48.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.981+0000 7f9ac4947640 1 -- 172.21.15.29:0/922987723 wait complete. 2023-10-14T17:00:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.982+0000 7f9ac4947640 1 Processor -- start 2023-10-14T17:00:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.982+0000 7f9ac4947640 1 -- start start 2023-10-14T17:00:48.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9aa8004d00 0x7f9ac00c14b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01640d0 0x7f9ac0156d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac0161e40 0x7f9ac01622c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0161a20 con 0x7f9ac0161e40 2023-10-14T17:00:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac00c37c0 con 0x7f9ac01640d0 2023-10-14T17:00:48.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.983+0000 7f9ac4947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ac0162800 con 0x7f9aa8004d00 2023-10-14T17:00:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac0161e40 0x7f9ac01622c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac0161e40 0x7f9ac01622c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53476/0 (socket says 172.21.15.29:53476) 2023-10-14T17:00:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 -- 172.21.15.29:0/325247260 learned_addr learned my addr 172.21.15.29:0/325247260 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:48.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 -- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9aa8004d00 msgr2=0x7f9ac00c14b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abf7fe640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9aa8004d00 0x7f9ac00c14b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abeffd640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01640d0 0x7f9ac0156d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9aa8004d00 0x7f9ac00c14b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 -- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01640d0 msgr2=0x7f9ac0156d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:48.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01640d0 0x7f9ac0156d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abffff640 1 -- 172.21.15.29:0/325247260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ac00c3200 con 0x7f9ac0161e40 2023-10-14T17:00:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abf7fe640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9aa8004d00 0x7f9ac00c14b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.984+0000 7f9abeffd640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9ac01640d0 0x7f9ac0156d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9abffff640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9ac0161e40 0x7f9ac01622c0 secure :-1 s=READY pgs=1660 cs=0 l=1 rev1=1 crypto rx=0x7f9ab8002230 tx=0x7f9ab8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:48.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8046070 con 0x7f9ac0161e40 2023-10-14T17:00:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ab8035ae0 con 0x7f9ac0161e40 2023-10-14T17:00:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ab8035e70 con 0x7f9ac0161e40 2023-10-14T17:00:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9ac4947640 1 -- 172.21.15.29:0/325247260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ac00c34e0 con 0x7f9ac0161e40 2023-10-14T17:00:48.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.985+0000 7f9ac4947640 1 -- 172.21.15.29:0/325247260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ac00c19f0 con 0x7f9ac0161e40 2023-10-14T17:00:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.986+0000 7f9ac4947640 1 -- 172.21.15.29:0/325247260 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ac0161b60 con 0x7f9ac0161e40 2023-10-14T17:00:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.987+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f9ab803b050 con 0x7f9ac0161e40 2023-10-14T17:00:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.988+0000 7f9aa7fff640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f9aac06b070 0x7f9aac06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.988+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f9ab80aaeb0 con 0x7f9ac0161e40 2023-10-14T17:00:48.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.989+0000 7f9abf7fe640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f9aac06b070 0x7f9aac06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.989+0000 7f9abf7fe640 1 --2- 172.21.15.29:0/325247260 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f9aac06b070 0x7f9aac06d530 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9ab0000c00 tx=0x7f9ab0000f50 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:48.993+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9ac0161b60 con 0x7f9ac0161e40 2023-10-14T17:00:49.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.208+0000 7f9ac4947640 1 -- 172.21.15.29:0/325247260 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "M", "target": ["mon-mgr", ""]}) v1 -- 0x7f9ac00c1f10 con 0x7f9aac06b070 2023-10-14T17:00:49.211 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:49.211+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.211+0000 7f9aa7fff640 1 -- 172.21.15.29:0/325247260 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7f9ab0001ca0 con 0x7f9aac06b070 2023-10-14T17:00:49.212 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:49.225 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:49.226 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec -1m 2023-10-14T17:00:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.456+0000 7fcf1fc75640 1 Processor -- start 2023-10-14T17:00:49.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.456+0000 7fcf1fc75640 1 -- start start 2023-10-14T17:00:49.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1fc75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf1815ad70 0x7fcf18069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1fc75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf180caac0 0x7fcf1806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1fc75640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf180cb030 con 0x7fcf180c9ca0 2023-10-14T17:00:49.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1fc75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf180c91c0 con 0x7fcf180caac0 2023-10-14T17:00:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1fc75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180c94a0 con 0x7fcf1815ad70 2023-10-14T17:00:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.458+0000 7fcf1e212640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf180caac0 0x7fcf1806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf180caac0 0x7fcf1806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53490/0 (socket says 172.21.15.29:53490) 2023-10-14T17:00:49.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 -- 172.21.15.29:0/1922786938 learned_addr learned my addr 172.21.15.29:0/1922786938 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1da11640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf1815ad70 0x7fcf18069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 -- 172.21.15.29:0/1922786938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf1815ad70 msgr2=0x7fcf18069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf1815ad70 0x7fcf18069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 -- 172.21.15.29:0/1922786938 >> v1:172.21.15.186:6789/0 conn(0x7fcf180c9ca0 legacy=0x7fcf180ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:49.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.459+0000 7fcf1e212640 1 -- 172.21.15.29:0/1922786938 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf180c9780 con 0x7fcf180caac0 2023-10-14T17:00:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.460+0000 7fcf1e212640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180caac0 0x7fcf1806a2e0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009e50 tx=0x7fcf1002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6991b3de1360689 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1ca0f640 1 -- 172.21.15.29:0/1922786938 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf1002ca70 con 0x7fcf180caac0 2023-10-14T17:00:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1ca0f640 1 -- 172.21.15.29:0/1922786938 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf1002cbd0 con 0x7fcf180caac0 2023-10-14T17:00:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1ca0f640 1 -- 172.21.15.29:0/1922786938 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf100343f0 con 0x7fcf180caac0 2023-10-14T17:00:49.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 -- 172.21.15.29:0/1922786938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180caac0 msgr2=0x7fcf1806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180caac0 0x7fcf1806a2e0 secure :-1 s=READY pgs=1661 cs=0 l=1 rev1=1 crypto rx=0x7fcf10009e50 tx=0x7fcf1002bed0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 -- 172.21.15.29:0/1922786938 shutdown_connections 2023-10-14T17:00:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180caac0 0x7fcf1806a2e0 unknown :-1 s=CLOSED pgs=1661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 --2- 172.21.15.29:0/1922786938 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf1815ad70 0x7fcf18069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 -- 172.21.15.29:0/1922786938 >> 172.21.15.29:0/1922786938 conn(0x7fcf180ce9b0 msgr2=0x7fcf180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.461+0000 7fcf1fc75640 1 -- 172.21.15.29:0/1922786938 shutdown_connections 2023-10-14T17:00:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.462+0000 7fcf1fc75640 1 -- 172.21.15.29:0/1922786938 wait complete. 2023-10-14T17:00:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.463+0000 7fcf1fc75640 1 Processor -- start 2023-10-14T17:00:49.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.464+0000 7fcf1fc75640 1 -- start start 2023-10-14T17:00:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.464+0000 7fcf1fc75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180c9ca0 0x7fcf180aeea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.464+0000 7fcf1fc75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf180caac0 0x7fcf180c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1fc75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1815ad70 0x7fcf18159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1fc75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180c6990 con 0x7fcf180c9ca0 2023-10-14T17:00:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1fc75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180c7c40 con 0x7fcf1815ad70 2023-10-14T17:00:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1fc75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf180c6ba0 con 0x7fcf180caac0 2023-10-14T17:00:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1da11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180c9ca0 0x7fcf180aeea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1da11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180c9ca0 0x7fcf180aeea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53504/0 (socket says 172.21.15.29:53504) 2023-10-14T17:00:49.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1da11640 1 -- 172.21.15.29:0/2767420805 learned_addr learned my addr 172.21.15.29:0/2767420805 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1e212640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1815ad70 0x7fcf18159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.465+0000 7fcf1d210640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf180caac0 0x7fcf180c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 -- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf180caac0 msgr2=0x7fcf180c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf180caac0 0x7fcf180c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 -- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1815ad70 msgr2=0x7fcf18159ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf1815ad70 0x7fcf18159ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 -- 172.21.15.29:0/2767420805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf10009b30 con 0x7fcf180c9ca0 2023-10-14T17:00:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1da11640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf180c9ca0 0x7fcf180aeea0 secure :-1 s=READY pgs=1662 cs=0 l=1 rev1=1 crypto rx=0x7fcf1400c820 tx=0x7fcf1400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14004020 con 0x7fcf180c9ca0 2023-10-14T17:00:49.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf14004180 con 0x7fcf180c9ca0 2023-10-14T17:00:49.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf14007830 con 0x7fcf180c9ca0 2023-10-14T17:00:49.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1fc75640 1 -- 172.21.15.29:0/2767420805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf180c6e80 con 0x7fcf180c9ca0 2023-10-14T17:00:49.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.466+0000 7fcf1fc75640 1 -- 172.21.15.29:0/2767420805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf180c7440 con 0x7fcf180c9ca0 2023-10-14T17:00:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.467+0000 7fcf1fc75640 1 -- 172.21.15.29:0/2767420805 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf1804e160 con 0x7fcf180c9ca0 2023-10-14T17:00:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.469+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fcf14007990 con 0x7fcf180c9ca0 2023-10-14T17:00:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.469+0000 7fcf0e7fc640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcefc06b070 0x7fcefc06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.470+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fcf14089930 con 0x7fcf180c9ca0 2023-10-14T17:00:49.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.470+0000 7fcf1d210640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcefc06b070 0x7fcefc06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.471+0000 7fcf1d210640 1 --2- 172.21.15.29:0/2767420805 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fcefc06b070 0x7fcefc06d530 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fcf180af4f0 tx=0x7fcf08009040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.474+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf140544b0 con 0x7fcf180c9ca0 2023-10-14T17:00:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.692+0000 7fcf1fc75640 1 -- 172.21.15.29:0/2767420805 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "-1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fcf180ad4d0 con 0x7fcefc06b070 2023-10-14T17:00:49.694 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:49.694+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-14T17:00:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.694+0000 7fcf0e7fc640 1 -- 172.21.15.29:0/2767420805 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc.) v1 ==== 96+0+0 (secure 0 0 0) 0x7fcf080053f0 con 0x7fcefc06b070 2023-10-14T17:00:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period must be a non-zero positive value e.g. 1h or 4d etc. 2023-10-14T17:00:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:49.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1/bad_period_spec '' 2023-10-14T17:00:49.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.920+0000 7fdd5b59e640 1 Processor -- start 2023-10-14T17:00:49.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.920+0000 7fdd5b59e640 1 -- start start 2023-10-14T17:00:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.921+0000 7fdd5b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.921+0000 7fdd5b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdd5c0ca8a0 0x7fdd5c0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.921+0000 7fdd5b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 0x7fdd5c0706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.921+0000 7fdd5b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c070c10 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cad00 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0b2d00 con 0x7fdd5c070240 2023-10-14T17:00:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53520/0 (socket says 172.21.15.29:53520) 2023-10-14T17:00:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 -- 172.21.15.29:0/2994892099 learned_addr learned my addr 172.21.15.29:0/2994892099 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 -- 172.21.15.29:0/2994892099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 msgr2=0x7fdd5c0706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 --2- 172.21.15.29:0/2994892099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 0x7fdd5c0706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 -- 172.21.15.29:0/2994892099 >> v2:172.21.15.186:3300/0 conn(0x7fdd5c0ca8a0 msgr2=0x7fdd5c0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 --2- 172.21.15.29:0/2994892099 >> v2:172.21.15.186:3300/0 conn(0x7fdd5c0ca8a0 0x7fdd5c0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.922+0000 7fdd5a59c640 1 -- 172.21.15.29:0/2994892099 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd5c0b2fe0 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.923+0000 7fdd5a59c640 1 --2- 172.21.15.29:0/2994892099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0cf650 tx=0x7fdd4c02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bd3d3a0b0ac3da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.923+0000 7fdd5959a640 1 -- 172.21.15.29:0/2994892099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c031880 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5959a640 1 -- 172.21.15.29:0/2994892099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4c0319e0 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5959a640 1 -- 172.21.15.29:0/2994892099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4c031d70 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2994892099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c15aaa0 msgr2=0x7fdd5c0b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 --2- 172.21.15.29:0/2994892099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 secure :-1 s=READY pgs=1663 cs=0 l=1 rev1=1 crypto rx=0x7fdd5c0cf650 tx=0x7fdd4c02b920 comp rx=0 tx=0).stop 2023-10-14T17:00:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2994892099 shutdown_connections 2023-10-14T17:00:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 --2- 172.21.15.29:0/2994892099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 0x7fdd5c0706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 --2- 172.21.15.29:0/2994892099 >> v2:172.21.15.186:3300/0 conn(0x7fdd5c0ca8a0 0x7fdd5c0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 --2- 172.21.15.29:0/2994892099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c15aaa0 0x7fdd5c0b2600 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2994892099 >> 172.21.15.29:0/2994892099 conn(0x7fdd5c0cc230 msgr2=0x7fdd5c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.924+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2994892099 shutdown_connections 2023-10-14T17:00:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.925+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2994892099 wait complete. 2023-10-14T17:00:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.926+0000 7fdd5b59e640 1 Processor -- start 2023-10-14T17:00:49.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.926+0000 7fdd5b59e640 1 -- start start 2023-10-14T17:00:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 0x7fdd5c150d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c0ca8a0 0x7fdd5c0cea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd5c15aaa0 0x7fdd5c1f2320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0ce5b0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cd9a0 con 0x7fdd5c15aaa0 2023-10-14T17:00:49.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.927+0000 7fdd5b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd5c0cdc80 con 0x7fdd5c070240 2023-10-14T17:00:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c0ca8a0 0x7fdd5c0cea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c0ca8a0 0x7fdd5c0cea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53532/0 (socket says 172.21.15.29:53532) 2023-10-14T17:00:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 -- 172.21.15.29:0/2823015857 learned_addr learned my addr 172.21.15.29:0/2823015857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:49.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 -- 172.21.15.29:0/2823015857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 msgr2=0x7fdd5c150d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd5c070240 0x7fdd5c150d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 -- 172.21.15.29:0/2823015857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd5c15aaa0 msgr2=0x7fdd5c1f2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:49.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd5c15aaa0 0x7fdd5c1f2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 -- 172.21.15.29:0/2823015857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4c0094d0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.928+0000 7fdd59d9b640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd5c0ca8a0 0x7fdd5c0cea40 secure :-1 s=READY pgs=1664 cs=0 l=1 rev1=1 crypto rx=0x7fdd4400ba50 tx=0x7fdd4400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.929+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd4400cbe0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.929+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd4400cd60 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.929+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd44014680 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.929+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2823015857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5c0cdf60 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.929+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2823015857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd5c0cef80 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.930+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2823015857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd5c04e9b0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.931+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7fdd4401d3b0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.932+0000 7fdd52ffd640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fdd3c06b070 0x7fdd3c06d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:49.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.932+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fdd44089ab0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:49.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.932+0000 7fdd5a59c640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fdd3c06b070 0x7fdd3c06d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:49.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.933+0000 7fdd5a59c640 1 --2- 172.21.15.29:0/2823015857 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7fdd3c06b070 0x7fdd3c06d530 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fdd4c036000 tx=0x7fdd4c030000 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:49.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:49.936+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd440547b0 con 0x7fdd5c0ca8a0 2023-10-14T17:00:50.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.150+0000 7fdd5b59e640 1 -- 172.21.15.29:0/2823015857 --> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1/bad_period_spec", "snap_schedule": "", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd5c0b35a0 con 0x7fdd3c06b070 2023-10-14T17:00:50.154 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:50.153+0000 7f8ca0010640 -1 mgr.server reply reply (2) No such file or directory invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:50.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.153+0000 7fdd52ffd640 1 -- 172.21.15.29:0/2823015857 <== mgr.8709 v2:172.21.15.29:6832/795882907 1 ==== mgr_command_reply(tid 0: -2 invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc.) v1 ==== 138+0+0 (secure 0 0 0) 0x7fdd4c031c50 con 0x7fdd3c06b070 2023-10-14T17:00:50.155 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: invalid schedule specified - period should be non-zero positive value and multiplier should be one of h,d,w,M,Y e.g. 1h or 4d etc. 2023-10-14T17:00:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:00:50.166 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.167 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/minutes) 2023-10-14T17:00:50.224 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.224 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/hourly) 2023-10-14T17:00:50.287 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.288 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/daily) 2023-10-14T17:00:50.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.347 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/weekly) 2023-10-14T17:00:50.404 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.405 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/monthly) 2023-10-14T17:00:50.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.464 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/yearly) 2023-10-14T17:00:50.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:50.521 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/bad_period_spec) 2023-10-14T17:00:50.580 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T17:00:50.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.813+0000 7f3971973640 1 Processor -- start 2023-10-14T17:00:50.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.814+0000 7f3971973640 1 -- start start 2023-10-14T17:00:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971973640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f396c06c040 0x7f396c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971973640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f396c0ad970 0x7f396c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971973640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c06c960 con 0x7f396c15ebc0 2023-10-14T17:00:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971973640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c15b9d0 con 0x7f396c0ad970 2023-10-14T17:00:50.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971973640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f396c15bcb0 con 0x7f396c06c040 2023-10-14T17:00:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f396bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f396c0ad970 0x7f396c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971172640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f396c15ebc0 0x7f396c0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53578/0 (socket says 172.21.15.29:53578) 2023-10-14T17:00:50.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.815+0000 7f3971172640 1 -- 172.21.15.29:0/222247078 learned_addr learned my addr 172.21.15.29:0/222247078 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f3970971640 1 --2- 172.21.15.29:0/222247078 >> v2:172.21.15.186:3300/0 conn(0x7f396c06c040 0x7f396c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f396bfff640 1 -- 172.21.15.29:0/222247078 >> v2:172.21.15.186:3300/0 conn(0x7f396c06c040 msgr2=0x7f396c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f396bfff640 1 --2- 172.21.15.29:0/222247078 >> v2:172.21.15.186:3300/0 conn(0x7f396c06c040 0x7f396c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f396bfff640 1 -- 172.21.15.29:0/222247078 >> v1:172.21.15.29:6789/0 conn(0x7f396c15ebc0 legacy=0x7f396c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f396bfff640 1 -- 172.21.15.29:0/222247078 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c15bf90 con 0x7f396c0ad970 2023-10-14T17:00:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.816+0000 7f396b7fe640 1 -- 172.21.15.29:0/222247078 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1161572556 0 0) 0x7f396c06c960 con 0x7f396c15ebc0 2023-10-14T17:00:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.817+0000 7f396bfff640 1 --2- 172.21.15.29:0/222247078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c0ad970 0x7f396c15e680 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f396c0cf350 tx=0x7f395802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85eaf40a70ce3f72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f396b7fe640 1 -- 172.21.15.29:0/222247078 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958031830 con 0x7f396c0ad970 2023-10-14T17:00:50.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f396b7fe640 1 -- 172.21.15.29:0/222247078 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3958031990 con 0x7f396c0ad970 2023-10-14T17:00:50.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f396b7fe640 1 -- 172.21.15.29:0/222247078 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3958031d20 con 0x7f396c0ad970 2023-10-14T17:00:50.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 -- 172.21.15.29:0/222247078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c0ad970 msgr2=0x7f396c15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:50.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 --2- 172.21.15.29:0/222247078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c0ad970 0x7f396c15e680 secure :-1 s=READY pgs=1666 cs=0 l=1 rev1=1 crypto rx=0x7f396c0cf350 tx=0x7f395802b8d0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 -- 172.21.15.29:0/222247078 shutdown_connections 2023-10-14T17:00:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 --2- 172.21.15.29:0/222247078 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c0ad970 0x7f396c15e680 unknown :-1 s=CLOSED pgs=1666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 --2- 172.21.15.29:0/222247078 >> v2:172.21.15.186:3300/0 conn(0x7f396c06c040 0x7f396c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 -- 172.21.15.29:0/222247078 >> 172.21.15.29:0/222247078 conn(0x7f396c0ce6a0 msgr2=0x7f396c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 -- 172.21.15.29:0/222247078 shutdown_connections 2023-10-14T17:00:50.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.818+0000 7f3971973640 1 -- 172.21.15.29:0/222247078 wait complete. 2023-10-14T17:00:50.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.820+0000 7f3971973640 1 Processor -- start 2023-10-14T17:00:50.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.820+0000 7f3971973640 1 -- start start 2023-10-14T17:00:50.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 0x7f396c1f0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 0x7f396c1f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0aee30 con 0x7f396c15ebc0 2023-10-14T17:00:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c0c5000 con 0x7f396c06c040 2023-10-14T17:00:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.821+0000 7f3971973640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c158b00 con 0x7f396c0ad970 2023-10-14T17:00:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53556/0 (socket says 172.21.15.29:53556) 2023-10-14T17:00:50.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 -- 172.21.15.29:0/1004773328 learned_addr learned my addr 172.21.15.29:0/1004773328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:50.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3970971640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 0x7f396c1f0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f396bfff640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 0x7f396c1f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 -- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 msgr2=0x7f396c1f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:50.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 0x7f396c1f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 -- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 msgr2=0x7f396c1f0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 0x7f396c1f0df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3971172640 1 -- 172.21.15.29:0/1004773328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3958009400 con 0x7f396c15ebc0 2023-10-14T17:00:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f396bfff640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 0x7f396c1f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:50.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.822+0000 7f3970971640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 0x7f396c1f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f3971172640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f396400c7e0 tx=0x7f396400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396400a640 con 0x7f396c15ebc0 2023-10-14T17:00:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396400a7c0 con 0x7f396c15ebc0 2023-10-14T17:00:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f396400ab70 con 0x7f396c15ebc0 2023-10-14T17:00:50.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c0af040 con 0x7f396c15ebc0 2023-10-14T17:00:50.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.823+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c0af600 con 0x7f396c15ebc0 2023-10-14T17:00:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.825+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c06b330 con 0x7f396c15ebc0 2023-10-14T17:00:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.826+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 75) v1 ==== 89405+0+0 (secure 0 0 0) 0x7f396400acd0 con 0x7f396c15ebc0 2023-10-14T17:00:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.827+0000 7f39697fa640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 0x7f395006d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:50.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.827+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f3964089150 con 0x7f396c15ebc0 2023-10-14T17:00:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.827+0000 7f3970971640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 0x7f395006d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.828+0000 7f3970971640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 0x7f395006d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3960006fd0 tx=0x7f3960008040 comp rx=0 tx=0).ready entity=mgr.8709 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:50.831+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3964053de0 con 0x7f396c15ebc0 2023-10-14T17:00:51.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.077+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f396c04e9b0 con 0x7f396c15ebc0 2023-10-14T17:00:51.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.316+0000 7f39697fa640 1 -- 172.21.15.29:0/1004773328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v76) v1 ==== 93+0+0 (secure 0 0 0) 0x7f39640180a0 con 0x7f396c15ebc0 2023-10-14T17:00:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.320+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 msgr2=0x7f395006d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.320+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 0x7f395006d530 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3960006fd0 tx=0x7f3960008040 comp rx=0 tx=0).stop 2023-10-14T17:00:51.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.320+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 msgr2=0x7f396c158250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.320+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 secure :-1 s=READY pgs=1667 cs=0 l=1 rev1=1 crypto rx=0x7f396400c7e0 tx=0x7f396400cca0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 shutdown_connections 2023-10-14T17:00:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f395006b070 0x7f395006d530 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f396c15ebc0 0x7f396c158250 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f396c0ad970 0x7f396c1f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 --2- 172.21.15.29:0/1004773328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f396c06c040 0x7f396c1f0df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 >> 172.21.15.29:0/1004773328 conn(0x7f396c0ce6a0 msgr2=0x7f396c0c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:51.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.321+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 shutdown_connections 2023-10-14T17:00:51.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.322+0000 7f3971973640 1 -- 172.21.15.29:0/1004773328 wait complete. 2023-10-14T17:00:51.347 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:00:51.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:51.456 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:00:51.567 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:00:51.568 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:51.568 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:51.627 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:51.628 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:51.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:51.883 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:51.883+0000 7f775fcda040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:51.902 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:51.901+0000 7f1477037040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:00:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.928+0000 7faaf096b640 1 Processor -- start 2023-10-14T17:00:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.929+0000 7faaf096b640 1 -- start start 2023-10-14T17:00:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaf096b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faaec0ad970 0x7faaec0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaf096b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 0x7faaec0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaf096b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaec15b9d0 con 0x7faaec06c040 2023-10-14T17:00:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaf096b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaec15bcb0 con 0x7faaec0ad970 2023-10-14T17:00:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaf096b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec15bf90 con 0x7faaec0ccfb0 2023-10-14T17:00:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaead9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faaec06c040 0x7faaec06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39928/0 (socket says 172.21.15.29:39928) 2023-10-14T17:00:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.930+0000 7faaead9d640 1 -- 172.21.15.29:0/3434546858 learned_addr learned my addr 172.21.15.29:0/3434546858 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faae3fff640 1 --2- 172.21.15.29:0/3434546858 >> v2:172.21.15.186:3300/0 conn(0x7faaec0ad970 0x7faaec0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947834558 0 0) 0x7faaec15b9d0 con 0x7faaec06c040 2023-10-14T17:00:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 0x7faaec0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faad8003520 con 0x7faaec06c040 2023-10-14T17:00:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1848507418 0 0) 0x7faad8003520 con 0x7faaec06c040 2023-10-14T17:00:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 msgr2=0x7faaec0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:51.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 0x7faaec0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 >> v2:172.21.15.186:3300/0 conn(0x7faaec0ad970 msgr2=0x7faaec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 --2- 172.21.15.29:0/3434546858 >> v2:172.21.15.186:3300/0 conn(0x7faaec0ad970 0x7faaec0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0cc270 con 0x7faaec06c040 2023-10-14T17:00:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7faadc001670 con 0x7faaec06c040 2023-10-14T17:00:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 >> v1:172.21.15.29:6789/0 conn(0x7faaec06c040 legacy=0x7faaec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.931+0000 7faaea59c640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 0x7faad8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaea59c640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8003f40 0x7faad80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaea59c640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faad8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec15bcb0 con 0x7faad8004d00 2023-10-14T17:00:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec15bf90 con 0x7faad8003f40 2023-10-14T17:00:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faad800ba30 con 0x7faaec0ccfb0 2023-10-14T17:00:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faad8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 -- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 msgr2=0x7faad8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0ccfb0 0x7faad8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 -- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8003f40 msgr2=0x7faad80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faad8003f40 0x7faad80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.932+0000 7faaeb59e640 1 -- 172.21.15.29:0/3434546858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faad8003520 con 0x7faad8004d00 2023-10-14T17:00:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.933+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faad8005180 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7faae4009fc0 tx=0x7faad80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94d9c7b2f826b6f8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae402cc30 con 0x7faad8004d00 2023-10-14T17:00:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faae400f040 con 0x7faad8004d00 2023-10-14T17:00:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaea59c640 1 -- 172.21.15.29:0/3434546858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faae4034440 con 0x7faad8004d00 2023-10-14T17:00:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 -- 172.21.15.29:0/3434546858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 msgr2=0x7faad8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 --2- 172.21.15.29:0/3434546858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faad8005180 secure :-1 s=READY pgs=1673 cs=0 l=1 rev1=1 crypto rx=0x7faae4009fc0 tx=0x7faad80057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaeb59e640 1 -- 172.21.15.29:0/3434546858 reap_dead start 2023-10-14T17:00:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 -- 172.21.15.29:0/3434546858 shutdown_connections 2023-10-14T17:00:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 -- 172.21.15.29:0/3434546858 >> 172.21.15.29:0/3434546858 conn(0x7faaec0ce6a0 msgr2=0x7faaec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 -- 172.21.15.29:0/3434546858 shutdown_connections 2023-10-14T17:00:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.934+0000 7faaf096b640 1 -- 172.21.15.29:0/3434546858 wait complete. 2023-10-14T17:00:51.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.936+0000 7faaf096b640 1 Processor -- start 2023-10-14T17:00:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.936+0000 7faaf096b640 1 -- start start 2023-10-14T17:00:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0c73d0 0x7faaec0c4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaec0c48a0 0x7faaec0c6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0c3560 con 0x7faad8004d00 2023-10-14T17:00:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0c40c0 con 0x7faaec0c48a0 2023-10-14T17:00:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaf096b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaec0c4d20 con 0x7faaec0c73d0 2023-10-14T17:00:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaead9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38798/0 (socket says 172.21.15.29:38798) 2023-10-14T17:00:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.937+0000 7faaead9d640 1 -- 172.21.15.29:0/3089941716 learned_addr learned my addr 172.21.15.29:0/3089941716 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faae3fff640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0c73d0 0x7faaec0c4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaeb59e640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaec0c48a0 0x7faaec0c6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0c73d0 msgr2=0x7faaec0c4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0c73d0 0x7faaec0c4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaec0c48a0 msgr2=0x7faaec0c6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaec0c48a0 0x7faaec0c6110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 -- 172.21.15.29:0/3089941716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec0c5c80 con 0x7faad8004d00 2023-10-14T17:00:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faaead9d640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c7a0 tx=0x7faadc00cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.938+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc0074e0 con 0x7faad8004d00 2023-10-14T17:00:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.939+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faadc007640 con 0x7faad8004d00 2023-10-14T17:00:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.939+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faadc004a20 con 0x7faad8004d00 2023-10-14T17:00:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.939+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaec156080 con 0x7faad8004d00 2023-10-14T17:00:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.939+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaec0c3280 con 0x7faad8004d00 2023-10-14T17:00:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.940+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaec06b330 con 0x7faad8004d00 2023-10-14T17:00:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.941+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7faadc004b80 con 0x7faad8004d00 2023-10-14T17:00:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.942+0000 7faae37fe640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 0x7faacc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.942+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7faadc088e90 con 0x7faad8004d00 2023-10-14T17:00:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.942+0000 7faae3fff640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 msgr2=0x7faacc06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.942+0000 7faae3fff640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 0x7faacc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:51.951 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:51.943+0000 7f775fcda040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:51.946+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faadc053b90 con 0x7faad8004d00 2023-10-14T17:00:51.961 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:51.961+0000 7f1477037040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:00:52.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.143+0000 7faae3fff640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 msgr2=0x7faacc06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:52.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.143+0000 7faae3fff640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 0x7faacc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.143+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faaec04e9b0 con 0x7faad8004d00 2023-10-14T17:00:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.146+0000 7faae37fe640 1 -- 172.21.15.29:0/3089941716 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17129 (secure 0 0 0) 0x7faadc018020 con 0x7faad8004d00 2023-10-14T17:00:52.147 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:52.148 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:42.236313+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1488377546":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6832/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/532161644":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3029691751":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/932642367":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3424087599":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6833/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/4142001408":"2023-10-15T17:00:42.236245+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:52.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 msgr2=0x7faacc06d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:52.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 0x7faacc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 msgr2=0x7faaec155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 secure :-1 s=READY pgs=1674 cs=0 l=1 rev1=1 crypto rx=0x7faadc00c7a0 tx=0x7faadc00cc60 comp rx=0 tx=0).stop 2023-10-14T17:00:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 shutdown_connections 2023-10-14T17:00:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7faacc06b050 0x7faacc06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faaec0c48a0 0x7faaec0c6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faaec0c73d0 0x7faaec0c4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 --2- 172.21.15.29:0/3089941716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faad8004d00 0x7faaec155b40 unknown :-1 s=CLOSED pgs=1674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 >> 172.21.15.29:0/3089941716 conn(0x7faaec0ce6a0 msgr2=0x7faaec070590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:52.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 shutdown_connections 2023-10-14T17:00:52.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.149+0000 7faaf096b640 1 -- 172.21.15.29:0/3089941716 wait complete. 2023-10-14T17:00:52.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:00:25.126064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:00:26.177199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:52.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:52.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.378+0000 7f236d5b0640 1 Processor -- start 2023-10-14T17:00:52.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.379+0000 7f236d5b0640 1 -- start start 2023-10-14T17:00:52.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.380+0000 7f236d5b0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f23681618b0 0x7f2368065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.380+0000 7f236d5b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368065ff0 0x7f23680ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.380+0000 7f236d5b0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f236806cee0 con 0x7f236815d330 2023-10-14T17:00:52.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.380+0000 7f236d5b0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236806b1e0 con 0x7f2368065ff0 2023-10-14T17:00:52.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.380+0000 7f236d5b0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23680663d0 con 0x7f23681618b0 2023-10-14T17:00:52.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368065ff0 0x7f23680ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2368065ff0 0x7f23680ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38804/0 (socket says 172.21.15.29:38804) 2023-10-14T17:00:52.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 -- 172.21.15.29:0/1190767090 learned_addr learned my addr 172.21.15.29:0/1190767090 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:52.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f235ffff640 1 --2- 172.21.15.29:0/1190767090 >> v2:172.21.15.186:3300/0 conn(0x7f23681618b0 0x7f2368065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 -- 172.21.15.29:0/1190767090 >> v2:172.21.15.186:3300/0 conn(0x7f23681618b0 msgr2=0x7f2368065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:52.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 --2- 172.21.15.29:0/1190767090 >> v2:172.21.15.186:3300/0 conn(0x7f23681618b0 0x7f2368065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 -- 172.21.15.29:0/1190767090 >> v1:172.21.15.186:6789/0 conn(0x7f236815d330 legacy=0x7f236806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:52.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.381+0000 7f23677fe640 1 -- 172.21.15.29:0/1190767090 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23680d0890 con 0x7f2368065ff0 2023-10-14T17:00:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.382+0000 7f23677fe640 1 --2- 172.21.15.29:0/1190767090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368065ff0 0x7f23680ce9b0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f23680b6bc0 tx=0x7f236002be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9763ec50e4d2293f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f23667fc640 1 -- 172.21.15.29:0/1190767090 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f236002c9f0 con 0x7f2368065ff0 2023-10-14T17:00:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f23667fc640 1 -- 172.21.15.29:0/1190767090 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f236002cb50 con 0x7f2368065ff0 2023-10-14T17:00:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f23667fc640 1 -- 172.21.15.29:0/1190767090 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23600343f0 con 0x7f2368065ff0 2023-10-14T17:00:52.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 -- 172.21.15.29:0/1190767090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368065ff0 msgr2=0x7f23680ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 --2- 172.21.15.29:0/1190767090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368065ff0 0x7f23680ce9b0 secure :-1 s=READY pgs=1675 cs=0 l=1 rev1=1 crypto rx=0x7f23680b6bc0 tx=0x7f236002be50 comp rx=0 tx=0).stop 2023-10-14T17:00:52.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 -- 172.21.15.29:0/1190767090 shutdown_connections 2023-10-14T17:00:52.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 --2- 172.21.15.29:0/1190767090 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2368065ff0 0x7f23680ce9b0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 --2- 172.21.15.29:0/1190767090 >> v2:172.21.15.186:3300/0 conn(0x7f23681618b0 0x7f2368065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.383+0000 7f236d5b0640 1 -- 172.21.15.29:0/1190767090 >> 172.21.15.29:0/1190767090 conn(0x7f23680d0ab0 msgr2=0x7f23680d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:52.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.384+0000 7f236d5b0640 1 -- 172.21.15.29:0/1190767090 shutdown_connections 2023-10-14T17:00:52.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.384+0000 7f236d5b0640 1 -- 172.21.15.29:0/1190767090 wait complete. 2023-10-14T17:00:52.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.386+0000 7f236d5b0640 1 Processor -- start 2023-10-14T17:00:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.386+0000 7f236d5b0640 1 -- start start 2023-10-14T17:00:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2368065ff0 0x7f23681571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23681618b0 0x7f23680c9830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23681f6ef0 con 0x7f236815d330 2023-10-14T17:00:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23681f6370 con 0x7f23681618b0 2023-10-14T17:00:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f236d5b0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23681f6650 con 0x7f2368065ff0 2023-10-14T17:00:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f235ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f235ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38820/0 (socket says 172.21.15.29:38820) 2023-10-14T17:00:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.388+0000 7f235ffff640 1 -- 172.21.15.29:0/4028299092 learned_addr learned my addr 172.21.15.29:0/4028299092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f2366ffd640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2368065ff0 0x7f23681571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f23677fe640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23681618b0 0x7f23680c9830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2368065ff0 msgr2=0x7f23681571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2368065ff0 0x7f23681571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23681618b0 msgr2=0x7f23680c9830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23681618b0 0x7f23680c9830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 -- 172.21.15.29:0/4028299092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2360009b30 con 0x7f236815d330 2023-10-14T17:00:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235ffff640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f235000d5f0 tx=0x7f235000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.389+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23500125f0 con 0x7f236815d330 2023-10-14T17:00:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.390+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2350012750 con 0x7f236815d330 2023-10-14T17:00:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.390+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2350012b00 con 0x7f236815d330 2023-10-14T17:00:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.390+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23681f6930 con 0x7f236815d330 2023-10-14T17:00:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.390+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23681f6000 con 0x7f236815d330 2023-10-14T17:00:52.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.391+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f236804e160 con 0x7f236815d330 2023-10-14T17:00:52.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.392+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f2350012c60 con 0x7f236815d330 2023-10-14T17:00:52.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.393+0000 7f235f7fe640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 0x7f234806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.393+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f2350089f70 con 0x7f236815d330 2023-10-14T17:00:52.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.393+0000 7f2366ffd640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 msgr2=0x7f234806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:52.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.393+0000 7f2366ffd640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 0x7f234806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:52.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.399+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2350054bf0 con 0x7f236815d330 2023-10-14T17:00:52.415 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.414+0000 7f1477037040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:52.423 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.423+0000 7f775fcda040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:00:52.467 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.467+0000 7f1477037040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:52.479 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.478+0000 7f775fcda040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:00:52.517 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.516+0000 7f1477037040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:52.532 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.531+0000 7f775fcda040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:00:52.578 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.577+0000 7f1477037040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:52.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.593+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f236806f3a0 con 0x7f236815d330 2023-10-14T17:00:52.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.594+0000 7f2366ffd640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 msgr2=0x7f234806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.594+0000 7f2366ffd640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 0x7f234806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.595+0000 7f235f7fe640 1 -- 172.21.15.29:0/4028299092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2350058210 con 0x7f236815d330 2023-10-14T17:00:52.596 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:52.596 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.596+0000 7f775fcda040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:00:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 msgr2=0x7f234806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 0x7f234806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 msgr2=0x7f23680c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 secure :-1 s=READY pgs=1676 cs=0 l=1 rev1=1 crypto rx=0x7f235000d5f0 tx=0x7f235000dab0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 shutdown_connections 2023-10-14T17:00:52.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f234806b020 0x7f234806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23681618b0 0x7f23680c9830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f236815d330 0x7f23680c92f0 unknown :-1 s=CLOSED pgs=1676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 --2- 172.21.15.29:0/4028299092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2368065ff0 0x7f23681571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.597+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 >> 172.21.15.29:0/4028299092 conn(0x7f23680d0ab0 msgr2=0x7f23680cfc20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:52.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.598+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 shutdown_connections 2023-10-14T17:00:52.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.598+0000 7f236d5b0640 1 -- 172.21.15.29:0/4028299092 wait complete. 2023-10-14T17:00:52.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:00:52.627 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.626+0000 7f1477037040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:52.650 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.649+0000 7f775fcda040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:00:52.737 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.736+0000 7f1477037040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:52.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.767+0000 7f775fcda040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:00:52.799 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.799+0000 7f1477037040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.830+0000 7f8500189640 1 Processor -- start 2023-10-14T17:00:52.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.830+0000 7f8500189640 1 -- start start 2023-10-14T17:00:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.832+0000 7f8500189640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84f80b3420 0x7f84f80b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.832+0000 7f8500189640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f80d3980 con 0x7f84f80cfaf0 2023-10-14T17:00:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.832+0000 7f8500189640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84f80ce630 con 0x7f84f80cf710 2023-10-14T17:00:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.832+0000 7f8500189640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f80d3c60 con 0x7f84f80b3420 2023-10-14T17:00:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fe726640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84f80cfaf0 0x7f84f80b6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39930/0 (socket says 172.21.15.29:39930) 2023-10-14T17:00:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fe726640 1 -- 172.21.15.29:0/881435724 learned_addr learned my addr 172.21.15.29:0/881435724 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fd724640 1 --2- 172.21.15.29:0/881435724 >> v2:172.21.15.186:3300/0 conn(0x7f84f80b3420 0x7f84f80b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3276130781 0 0) 0x7f84f80d3980 con 0x7f84f80cfaf0 2023-10-14T17:00:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84e4003520 con 0x7f84f80cfaf0 2023-10-14T17:00:52.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3831520130 0 0) 0x7f84e4003520 con 0x7f84f80cfaf0 2023-10-14T17:00:52.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 >> v2:172.21.15.186:3300/0 conn(0x7f84f80b3420 msgr2=0x7f84f80b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:52.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 --2- 172.21.15.29:0/881435724 >> v2:172.21.15.186:3300/0 conn(0x7f84f80b3420 0x7f84f80b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 >> v1:172.21.15.186:6789/0 conn(0x7f84f80cf710 legacy=0x7f84f80b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f80ce910 con 0x7f84f80cfaf0 2023-10-14T17:00:52.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f84f00019f0 con 0x7f84f80cfaf0 2023-10-14T17:00:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 >> v1:172.21.15.29:6789/0 conn(0x7f84f80cfaf0 legacy=0x7f84f80b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.833+0000 7f84fcf23640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84e4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fcf23640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e40047c0 0x7f84e4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fcf23640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e40051d0 0x7f84e400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80d3c60 con 0x7f84e40039c0 2023-10-14T17:00:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400c120 con 0x7f84e40047c0 2023-10-14T17:00:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e400c2e0 con 0x7f84e40051d0 2023-10-14T17:00:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84e4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 -- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e40051d0 msgr2=0x7f84e400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:52.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e40051d0 0x7f84e400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 -- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e40047c0 msgr2=0x7f84e4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e40047c0 0x7f84e4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fdf25640 1 -- 172.21.15.29:0/881435724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e4003520 con 0x7f84e40039c0 2023-10-14T17:00:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.834+0000 7f84fe726640 1 -- 172.21.15.29:0/881435724 reap_dead start 2023-10-14T17:00:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.835+0000 7f84fdf25640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84e4003de0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f84f4007fd0 tx=0x7f84e40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e6e6da10e0a32f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:52.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.835+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f40052d0 con 0x7f84e40039c0 2023-10-14T17:00:52.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f402e050 con 0x7f84e40039c0 2023-10-14T17:00:52.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f84fcf23640 1 -- 172.21.15.29:0/881435724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4003640 con 0x7f84e40039c0 2023-10-14T17:00:52.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 -- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 msgr2=0x7f84e4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84e4003de0 secure :-1 s=READY pgs=1678 cs=0 l=1 rev1=1 crypto rx=0x7f84f4007fd0 tx=0x7f84e40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 -- 172.21.15.29:0/881435724 shutdown_connections 2023-10-14T17:00:52.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 --2- 172.21.15.29:0/881435724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84e4003de0 unknown :-1 s=CLOSED pgs=1678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 -- 172.21.15.29:0/881435724 >> 172.21.15.29:0/881435724 conn(0x7f84f80d2cc0 msgr2=0x7f84f80d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:52.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 -- 172.21.15.29:0/881435724 shutdown_connections 2023-10-14T17:00:52.844 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f775fcda040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.836+0000 7f8500189640 1 -- 172.21.15.29:0/881435724 wait complete. 2023-10-14T17:00:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.838+0000 7f8500189640 1 Processor -- start 2023-10-14T17:00:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.838+0000 7f8500189640 1 -- start start 2023-10-14T17:00:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84f80c7200 0x7f84f8153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84f80c8bd0 0x7f84f80c9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f8160390 con 0x7f84e40039c0 2023-10-14T17:00:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80b0490 con 0x7f84f80c7200 2023-10-14T17:00:52.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f8500189640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f80c7680 con 0x7f84f80c8bd0 2023-10-14T17:00:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f84fdf25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f84fdf25640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38844/0 (socket says 172.21.15.29:38844) 2023-10-14T17:00:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.839+0000 7f84fdf25640 1 -- 172.21.15.29:0/3540680578 learned_addr learned my addr 172.21.15.29:0/3540680578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fe726640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84f80c8bd0 0x7f84f80c9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fd724640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84f80c7200 0x7f84f8153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:52.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 -- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84f80c8bd0 msgr2=0x7f84f80c9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84f80c8bd0 0x7f84f80c9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 -- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84f80c7200 msgr2=0x7f84f8153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:52.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84f80c7200 0x7f84f8153260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:52.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 -- 172.21.15.29:0/3540680578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f81537a0 con 0x7f84e40039c0 2023-10-14T17:00:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.840+0000 7f84fdf25640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000ef0 tx=0x7f84f4001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.841+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4006ea0 con 0x7f84e40039c0 2023-10-14T17:00:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.841+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f4007a80 con 0x7f84e40039c0 2023-10-14T17:00:52.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.841+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84f4007450 con 0x7f84e40039c0 2023-10-14T17:00:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.841+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f815faf0 con 0x7f84e40039c0 2023-10-14T17:00:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.841+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f81600b0 con 0x7f84e40039c0 2023-10-14T17:00:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.842+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f806bb30 con 0x7f84e40039c0 2023-10-14T17:00:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.843+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f84f40075b0 con 0x7f84e40039c0 2023-10-14T17:00:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.844+0000 7f84ee7fc640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 0x7f84dc06d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.845+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f84f40abb10 con 0x7f84e40039c0 2023-10-14T17:00:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.845+0000 7f84fd724640 1 -- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 msgr2=0x7f84dc06d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:52.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.845+0000 7f84fd724640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 0x7f84dc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:52.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:52.849+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84f4076790 con 0x7f84e40039c0 2023-10-14T17:00:52.853 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.849+0000 7f1477037040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:52.891 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.891+0000 7f775fcda040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:00:52.909 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.908+0000 7f1477037040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:52.953 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:52.951+0000 7f775fcda040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:00:52.958 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:52.958+0000 7f1477037040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:53.004 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.004+0000 7f775fcda040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:00:53.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.042+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84f80b8aa0 con 0x7f84e40039c0 2023-10-14T17:00:53.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.043+0000 7f84ee7fc640 1 -- 172.21.15.29:0/3540680578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v141) v1 ==== 104+0+11 (secure 0 0 0) 0x7f84f404b0a0 con 0x7f84e40039c0 2023-10-14T17:00:53.043 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:00:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 msgr2=0x7f84dc06d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:53.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 0x7f84dc06d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 msgr2=0x7f84f815f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 secure :-1 s=READY pgs=1679 cs=0 l=1 rev1=1 crypto rx=0x7f84f4000ef0 tx=0x7f84f4001170 comp rx=0 tx=0).stop 2023-10-14T17:00:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 shutdown_connections 2023-10-14T17:00:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84dc06b050 0x7f84dc06d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84f80c8bd0 0x7f84f80c9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84f80c7200 0x7f84f8153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 --2- 172.21.15.29:0/3540680578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e40039c0 0x7f84f815f460 unknown :-1 s=CLOSED pgs=1679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 >> 172.21.15.29:0/3540680578 conn(0x7f84f80d2cc0 msgr2=0x7f84f80cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.045+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 shutdown_connections 2023-10-14T17:00:53.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.046+0000 7f8500189640 1 -- 172.21.15.29:0/3540680578 wait complete. 2023-10-14T17:00:53.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:00:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.323+0000 7f8eb693a640 1 Processor -- start 2023-10-14T17:00:53.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.323+0000 7f8eb693a640 1 -- start start 2023-10-14T17:00:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eb693a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb00cf710 0x7f8eb00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eb693a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb00d18c0 con 0x7f8eb00b6cd0 2023-10-14T17:00:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eb693a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8eb00d3980 con 0x7f8eb00cfaf0 2023-10-14T17:00:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eb693a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8eb00d3c60 con 0x7f8eb00cf710 2023-10-14T17:00:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eaffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb00cf710 0x7f8eb00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eaffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8eb00cf710 0x7f8eb00b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38850/0 (socket says 172.21.15.29:38850) 2023-10-14T17:00:53.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.325+0000 7f8eaffff640 1 -- 172.21.15.29:0/3690475168 learned_addr learned my addr 172.21.15.29:0/3690475168 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.326+0000 7f8eaffff640 1 -- 172.21.15.29:0/3690475168 >> v1:172.21.15.186:6789/0 conn(0x7f8eb00cfaf0 legacy=0x7f8eb00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.326+0000 7f8eaffff640 1 -- 172.21.15.29:0/3690475168 >> v1:172.21.15.29:6789/0 conn(0x7f8eb00b6cd0 legacy=0x7f8eb00d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.326+0000 7f8eaffff640 1 -- 172.21.15.29:0/3690475168 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8eb00ce630 con 0x7f8eb00cf710 2023-10-14T17:00:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.327+0000 7f8eaffff640 1 --2- 172.21.15.29:0/3690475168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb00b2d10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009850 tx=0x7f8ea002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f831cef5bb5185d5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:53.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.327+0000 7f8eaf7fe640 1 -- 172.21.15.29:0/3690475168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0004230 con 0x7f8eb00cf710 2023-10-14T17:00:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.327+0000 7f8eaf7fe640 1 -- 172.21.15.29:0/3690475168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ea0004390 con 0x7f8eb00cf710 2023-10-14T17:00:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.327+0000 7f8eaf7fe640 1 -- 172.21.15.29:0/3690475168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ea0003810 con 0x7f8eb00cf710 2023-10-14T17:00:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 -- 172.21.15.29:0/3690475168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 msgr2=0x7f8eb00b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 --2- 172.21.15.29:0/3690475168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb00b2d10 secure :-1 s=READY pgs=1681 cs=0 l=1 rev1=1 crypto rx=0x7f8ea0009850 tx=0x7f8ea002b8f0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 -- 172.21.15.29:0/3690475168 shutdown_connections 2023-10-14T17:00:53.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 --2- 172.21.15.29:0/3690475168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb00b2d10 unknown :-1 s=CLOSED pgs=1681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 -- 172.21.15.29:0/3690475168 >> 172.21.15.29:0/3690475168 conn(0x7f8eb00d2cc0 msgr2=0x7f8eb00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:53.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 -- 172.21.15.29:0/3690475168 shutdown_connections 2023-10-14T17:00:53.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.328+0000 7f8eb693a640 1 -- 172.21.15.29:0/3690475168 wait complete. 2023-10-14T17:00:53.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.330+0000 7f8eb693a640 1 Processor -- start 2023-10-14T17:00:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.330+0000 7f8eb693a640 1 -- start start 2023-10-14T17:00:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8eb00b6cd0 0x7f8eb00b2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8eb00cfaf0 0x7f8eb0155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0155bb0 con 0x7f8eb00cf710 2023-10-14T17:00:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0155e80 con 0x7f8eb00b6cd0 2023-10-14T17:00:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8eb693a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8eb0156160 con 0x7f8eb00cfaf0 2023-10-14T17:00:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8ea7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38864/0 (socket says 172.21.15.29:38864) 2023-10-14T17:00:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.331+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271502633 learned_addr learned my addr 172.21.15.29:0/2271502633 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8eaffff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8eb00b6cd0 0x7f8eb00b2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8eb4ed7640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8eb00cfaf0 0x7f8eb0155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8eb00cfaf0 msgr2=0x7f8eb0155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8eb00cfaf0 0x7f8eb0155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8eb00b6cd0 msgr2=0x7f8eb00b2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8eb00b6cd0 0x7f8eb00b2570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 -- 172.21.15.29:0/2271502633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea0009460 con 0x7f8eb00cf710 2023-10-14T17:00:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.332+0000 7f8ea7fff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800b3f0 tx=0x7f8e9800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:53.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.333+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98011830 con 0x7f8eb00cf710 2023-10-14T17:00:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.333+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e98011990 con 0x7f8eb00cf710 2023-10-14T17:00:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.333+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98011d40 con 0x7f8eb00cf710 2023-10-14T17:00:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.333+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8eb0156440 con 0x7f8eb00cf710 2023-10-14T17:00:53.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.333+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8eb0143d00 con 0x7f8eb00cf710 2023-10-14T17:00:53.339 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.335+0000 7f1477037040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.334+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8eb0143f20 con 0x7f8eb00cf710 2023-10-14T17:00:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.336+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f8e9801a400 con 0x7f8eb00cf710 2023-10-14T17:00:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.337+0000 7f8ead7fa640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 0x7f8e9006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.337+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f8e98089ce0 con 0x7f8eb00cf710 2023-10-14T17:00:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.337+0000 7f8eaffff640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 msgr2=0x7f8e9006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.337+0000 7f8eaffff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 0x7f8e9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:53.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.341+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e980548d0 con 0x7f8eb00cf710 2023-10-14T17:00:53.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.414+0000 7f775fcda040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:00:53.498 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.497+0000 7f1477037040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.538+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8eb004e160 con 0x7f8eb00cf710 2023-10-14T17:00:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.538+0000 7f8eaffff640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 msgr2=0x7f8e9006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.538+0000 7f8eaffff640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 0x7f8e9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:53.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.538+0000 7f8ead7fa640 1 -- 172.21.15.29:0/2271502633 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v141) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8e980150a0 con 0x7f8eb00cf710 2023-10-14T17:00:53.540 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:00:53.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 msgr2=0x7f8e9006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:53.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 0x7f8e9006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 msgr2=0x7f8eb01575b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 secure :-1 s=READY pgs=1682 cs=0 l=1 rev1=1 crypto rx=0x7f8e9800b3f0 tx=0x7f8e9800b8b0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 shutdown_connections 2023-10-14T17:00:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f8e9006b020 0x7f8e9006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8eb00cfaf0 0x7f8eb0155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8eb00cf710 0x7f8eb01575b0 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 --2- 172.21.15.29:0/2271502633 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8eb00b6cd0 0x7f8eb00b2570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 >> 172.21.15.29:0/2271502633 conn(0x7f8eb00d2cc0 msgr2=0x7f8eb00b7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 shutdown_connections 2023-10-14T17:00:53.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.541+0000 7f8eb693a640 1 -- 172.21.15.29:0/2271502633 wait complete. 2023-10-14T17:00:53.560 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T17:00:53.586 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.585+0000 7f775fcda040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:00:53.605 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.604+0000 7f1477037040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:53.655 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.654+0000 7f1477037040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:53.701 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.701+0000 7f775fcda040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:00:53.755 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.754+0000 7f775fcda040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:00:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.760+0000 7f32d60bb640 1 Processor -- start 2023-10-14T17:00:53.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.760+0000 7f32d60bb640 1 -- start start 2023-10-14T17:00:53.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.762+0000 7f32d60bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f32d00661c0 0x7f32d00cb070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.762+0000 7f32d60bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00cd3a0 con 0x7f32d0165e90 2023-10-14T17:00:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.762+0000 7f32d60bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32d00ceb30 con 0x7f32d00ccfc0 2023-10-14T17:00:53.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.762+0000 7f32d60bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32d006cee0 con 0x7f32d00661c0 2023-10-14T17:00:53.764 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.762+0000 7f1477037040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:53.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.762+0000 7f32ceffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32d0165e90 0x7f32d0065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39946/0 (socket says 172.21.15.29:39946) 2023-10-14T17:00:53.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.763+0000 7f32ceffd640 1 -- 172.21.15.29:0/740527169 learned_addr learned my addr 172.21.15.29:0/740527169 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:53.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.763+0000 7f32cffff640 1 --2- 172.21.15.29:0/740527169 >> v2:172.21.15.186:3300/0 conn(0x7f32d00661c0 0x7f32d00cb070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.763+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1706972618 0 0) 0x7f32d00cd3a0 con 0x7f32d0165e90 2023-10-14T17:00:53.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.763+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32bc003520 con 0x7f32d0165e90 2023-10-14T17:00:53.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.763+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143171771 0 0) 0x7f32bc003520 con 0x7f32d0165e90 2023-10-14T17:00:53.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 >> v2:172.21.15.186:3300/0 conn(0x7f32d00661c0 msgr2=0x7f32d00cb070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:53.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 --2- 172.21.15.29:0/740527169 >> v2:172.21.15.186:3300/0 conn(0x7f32d00661c0 0x7f32d00cb070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 >> v1:172.21.15.186:6789/0 conn(0x7f32d00ccfc0 legacy=0x7f32d006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00cee10 con 0x7f32d0165e90 2023-10-14T17:00:53.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f32c00019f0 con 0x7f32d0165e90 2023-10-14T17:00:53.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 >> v1:172.21.15.29:6789/0 conn(0x7f32d0165e90 legacy=0x7f32d0065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32bc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32bc005180 0x7f32bc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d006cee0 con 0x7f32bc003970 2023-10-14T17:00:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc00c020 con 0x7f32bc0047c0 2023-10-14T17:00:53.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32bc00c1e0 con 0x7f32bc005180 2023-10-14T17:00:53.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32bc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.764+0000 7f32ceffd640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cf7fe640 1 -- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32bc005180 msgr2=0x7f32bc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:53.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32bc005180 0x7f32bc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cf7fe640 1 -- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32bc0047c0 msgr2=0x7f32bc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32bc0047c0 0x7f32bc004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cf7fe640 1 -- 172.21.15.29:0/740527169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00cee10 con 0x7f32bc003970 2023-10-14T17:00:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.765+0000 7f32cffff640 1 -- 172.21.15.29:0/740527169 reap_dead start 2023-10-14T17:00:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.767+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32bc003d90 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f32b8001750 tx=0x7f32bc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=701fa2a79962762c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:53.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.767+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80052d0 con 0x7f32bc003970 2023-10-14T17:00:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.767+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b802e050 con 0x7f32bc003970 2023-10-14T17:00:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.767+0000 7f32ce7fc640 1 -- 172.21.15.29:0/740527169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8003830 con 0x7f32bc003970 2023-10-14T17:00:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 -- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 msgr2=0x7f32bc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32bc003d90 secure :-1 s=READY pgs=1684 cs=0 l=1 rev1=1 crypto rx=0x7f32b8001750 tx=0x7f32bc0057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 -- 172.21.15.29:0/740527169 shutdown_connections 2023-10-14T17:00:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 --2- 172.21.15.29:0/740527169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32bc003d90 unknown :-1 s=CLOSED pgs=1684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 -- 172.21.15.29:0/740527169 >> 172.21.15.29:0/740527169 conn(0x7f32d00d0ab0 msgr2=0x7f32d00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 -- 172.21.15.29:0/740527169 shutdown_connections 2023-10-14T17:00:53.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.768+0000 7f32d60bb640 1 -- 172.21.15.29:0/740527169 wait complete. 2023-10-14T17:00:53.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.770+0000 7f32d60bb640 1 Processor -- start 2023-10-14T17:00:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.770+0000 7f32d60bb640 1 -- start start 2023-10-14T17:00:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32d00c4f60 0x7f32d00c6850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d00c53e0 0x7f32d00c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c5e90 con 0x7f32bc003970 2023-10-14T17:00:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32d0162b60 con 0x7f32d00c53e0 2023-10-14T17:00:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.771+0000 7f32d60bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32d00c58d0 con 0x7f32d00c4f60 2023-10-14T17:00:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38882/0 (socket says 172.21.15.29:38882) 2023-10-14T17:00:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 -- 172.21.15.29:0/3372338756 learned_addr learned my addr 172.21.15.29:0/3372338756 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cffff640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d00c53e0 0x7f32d00c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32ceffd640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32d00c4f60 0x7f32d00c6850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32d00c4f60 msgr2=0x7f32d00c6850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32d00c4f60 0x7f32d00c6850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d00c53e0 msgr2=0x7f32d00c6d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d00c53e0 0x7f32d00c6d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 -- 172.21.15.29:0/3372338756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32d00c5bb0 con 0x7f32bc003970 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.772+0000 7f32cf7fe640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005040 tx=0x7f32b8001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.773+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b80076c0 con 0x7f32bc003970 2023-10-14T17:00:53.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.773+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b80069e0 con 0x7f32bc003970 2023-10-14T17:00:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.773+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b8006d90 con 0x7f32bc003970 2023-10-14T17:00:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.773+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32d0162d70 con 0x7f32bc003970 2023-10-14T17:00:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.773+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32d015d150 con 0x7f32bc003970 2023-10-14T17:00:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.774+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32d00c5df0 con 0x7f32bc003970 2023-10-14T17:00:53.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.775+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f32b80035f0 con 0x7f32bc003970 2023-10-14T17:00:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.777+0000 7f32c7fff640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 0x7f32b006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.777+0000 7f32ceffd640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 msgr2=0x7f32b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.777+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f32b80ab820 con 0x7f32bc003970 2023-10-14T17:00:53.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.777+0000 7f32ceffd640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 0x7f32b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:53.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.780+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32b80764e0 con 0x7f32bc003970 2023-10-14T17:00:53.872 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:53.871+0000 7f1477037040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:53.877 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.876+0000 7f775fcda040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:53.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.978+0000 7f32ceffd640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 msgr2=0x7f32b006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:53.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.978+0000 7f32ceffd640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 0x7f32b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:53.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.987+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f32d004e160 con 0x7f32bc003970 2023-10-14T17:00:53.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.988+0000 7f32c7fff640 1 -- 172.21.15.29:0/3372338756 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v141) v1 ==== 80+0+468 (secure 0 0 0) 0x7f32b804b1f0 con 0x7f32bc003970 2023-10-14T17:00:53.989 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1488377546 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.989 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/3738876524 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.990 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/532161644 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.990 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3029691751 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.990 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/932642367 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.990 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3424087599 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.991 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/3738876524 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.991 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4142001408 2023-10-15T17:00:42.236245+0000 2023-10-14T17:00:53.991 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T17:00:53.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 msgr2=0x7f32b006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:53.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 0x7f32b006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.992 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f775fcda040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:00:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 msgr2=0x7f32d015cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:53.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 secure :-1 s=READY pgs=1685 cs=0 l=1 rev1=1 crypto rx=0x7f32b8005040 tx=0x7f32b8001120 comp rx=0 tx=0).stop 2023-10-14T17:00:53.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 shutdown_connections 2023-10-14T17:00:53.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f32b006b020 0x7f32b006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32d00c53e0 0x7f32d00c6d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32d00c4f60 0x7f32d00c6850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 --2- 172.21.15.29:0/3372338756 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32bc003970 0x7f32d015cc10 unknown :-1 s=CLOSED pgs=1685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:53.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.990+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 >> 172.21.15.29:0/3372338756 conn(0x7f32d00d0ab0 msgr2=0x7f32d00c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:53.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.991+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 shutdown_connections 2023-10-14T17:00:53.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:53.991+0000 7f32d60bb640 1 -- 172.21.15.29:0/3372338756 wait complete. 2023-10-14T17:00:54.009 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T17:00:54.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:54.010 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T17:00:54.028 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:54.027+0000 7f1477037040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:54.076 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:54.075+0000 7f1477037040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:54.159 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T17:00:54.160 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:00:54.160 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T17:00:54.161 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:54.161+0000 7f775fcda040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:00:54.196 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:54.194+0000 7f1477037040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:54.207 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:54.207 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:54.216 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:54.215+0000 7f775fcda040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:00:54.262 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:54.263 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:00:54.263 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:54.323 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:00:54.323 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:00:54.324 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:00:54.343 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:54.342+0000 7f775fcda040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:00:54.376 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:00:54.377 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:54.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.590+0000 7f84e1c9b640 1 Processor -- start 2023-10-14T17:00:54.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.590+0000 7f84e1c9b640 1 -- start start 2023-10-14T17:00:54.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84dc159c10 0x7f84dc15ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84dc06df10 0x7f84dc06adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 0x7f84dc06b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc159ff0 con 0x7f84dc159c10 2023-10-14T17:00:54.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc06e3a0 con 0x7f84dc06df10 2023-10-14T17:00:54.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84e1c9b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc06e680 con 0x7f84dc0c9850 2023-10-14T17:00:54.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84dc159c10 0x7f84dc15ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84db7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f84dc159c10 0x7f84dc15ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38896/0 (socket says 172.21.15.29:38896) 2023-10-14T17:00:54.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.591+0000 7f84db7fe640 1 -- 172.21.15.29:0/2711176648 learned_addr learned my addr 172.21.15.29:0/2711176648 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84daffd640 1 --2- 172.21.15.29:0/2711176648 >> v2:172.21.15.186:3300/0 conn(0x7f84dc06df10 0x7f84dc06adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84dbfff640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 0x7f84dc06b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84db7fe640 1 -- 172.21.15.29:0/2711176648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 msgr2=0x7f84dc06b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:54.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 0x7f84dc06b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84db7fe640 1 -- 172.21.15.29:0/2711176648 >> v2:172.21.15.186:3300/0 conn(0x7f84dc06df10 msgr2=0x7f84dc06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2711176648 >> v2:172.21.15.186:3300/0 conn(0x7f84dc06df10 0x7f84dc06adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84db7fe640 1 -- 172.21.15.29:0/2711176648 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc15a220 con 0x7f84dc159c10 2023-10-14T17:00:54.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.592+0000 7f84dbfff640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 0x7f84dc06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.593+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc159c10 0x7f84dc15ad00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f84dc065ab0 tx=0x7f84cc02db10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=408d3a656db5d901 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.593+0000 7f84da7fc640 1 -- 172.21.15.29:0/2711176648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc02f8d0 con 0x7f84dc159c10 2023-10-14T17:00:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.593+0000 7f84da7fc640 1 -- 172.21.15.29:0/2711176648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc02fa30 con 0x7f84dc159c10 2023-10-14T17:00:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.593+0000 7f84da7fc640 1 -- 172.21.15.29:0/2711176648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc02fdc0 con 0x7f84dc159c10 2023-10-14T17:00:54.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2711176648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc159c10 msgr2=0x7f84dc15ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc159c10 0x7f84dc15ad00 secure :-1 s=READY pgs=1686 cs=0 l=1 rev1=1 crypto rx=0x7f84dc065ab0 tx=0x7f84cc02db10 comp rx=0 tx=0).stop 2023-10-14T17:00:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2711176648 shutdown_connections 2023-10-14T17:00:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc0c9850 0x7f84dc06b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2711176648 >> v2:172.21.15.186:3300/0 conn(0x7f84dc06df10 0x7f84dc06adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2711176648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc159c10 0x7f84dc15ad00 unknown :-1 s=CLOSED pgs=1686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2711176648 >> 172.21.15.29:0/2711176648 conn(0x7f84dc0b0c80 msgr2=0x7f84dc0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2711176648 shutdown_connections 2023-10-14T17:00:54.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.594+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2711176648 wait complete. 2023-10-14T17:00:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.596+0000 7f84e1c9b640 1 Processor -- start 2023-10-14T17:00:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.596+0000 7f84e1c9b640 1 -- start start 2023-10-14T17:00:54.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 0x7f84dc0ce400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0b42a0 0x7f84dc0b2bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0b3c60 con 0x7f84dc0c9850 2023-10-14T17:00:54.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0b3f40 con 0x7f84dc0b42a0 2023-10-14T17:00:54.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84e1c9b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc09dfd0 con 0x7f84dc159c10 2023-10-14T17:00:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38900/0 (socket says 172.21.15.29:38900) 2023-10-14T17:00:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 -- 172.21.15.29:0/2851926243 learned_addr learned my addr 172.21.15.29:0/2851926243 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84dbfff640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0b42a0 0x7f84dc0b2bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84daffd640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 0x7f84dc0ce400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 msgr2=0x7f84dc0ce400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 0x7f84dc0ce400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0b42a0 msgr2=0x7f84dc0b2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0b42a0 0x7f84dc0b2bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84db7fe640 1 -- 172.21.15.29:0/2851926243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc0094d0 con 0x7f84dc0c9850 2023-10-14T17:00:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.597+0000 7f84daffd640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 0x7f84dc0ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84db7fe640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000c00 tx=0x7f84cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc037c60 con 0x7f84dc0c9850 2023-10-14T17:00:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc037e00 con 0x7f84dc0c9850 2023-10-14T17:00:54.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc09e2b0 con 0x7f84dc0c9850 2023-10-14T17:00:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84cc040660 con 0x7f84dc0c9850 2023-10-14T17:00:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.598+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc0b47f0 con 0x7f84dc0c9850 2023-10-14T17:00:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.600+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc0b40b0 con 0x7f84dc0c9850 2023-10-14T17:00:54.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.600+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f84cc0407c0 con 0x7f84dc0c9850 2023-10-14T17:00:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.602+0000 7f84c3fff640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 0x7f84c806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.602+0000 7f84daffd640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 msgr2=0x7f84c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:54.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.602+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f84cc033070 con 0x7f84dc0c9850 2023-10-14T17:00:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.602+0000 7f84daffd640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 0x7f84c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:54.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.606+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84cc079010 con 0x7f84dc0c9850 2023-10-14T17:00:54.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.803+0000 7f84daffd640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 msgr2=0x7f84c806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:54.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.803+0000 7f84daffd640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 0x7f84c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:54.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.839+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84dc0b4980 con 0x7f84dc0c9850 2023-10-14T17:00:54.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.840+0000 7f84c3fff640 1 -- 172.21.15.29:0/2851926243 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f84cc07c630 con 0x7f84dc0c9850 2023-10-14T17:00:54.840 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:54.840 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:54.841 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:54.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 msgr2=0x7f84c806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:54.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 0x7f84c806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 msgr2=0x7f84dc0cdec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:54.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 secure :-1 s=READY pgs=1687 cs=0 l=1 rev1=1 crypto rx=0x7f84cc000c00 tx=0x7f84cc0046c0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 shutdown_connections 2023-10-14T17:00:54.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f84c806b020 0x7f84c806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84dc0b42a0 0x7f84dc0b2bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84dc159c10 0x7f84dc0ce400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 --2- 172.21.15.29:0/2851926243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84dc0c9850 0x7f84dc0cdec0 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:54.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 >> 172.21.15.29:0/2851926243 conn(0x7f84dc0b0c80 msgr2=0x7f84dc0b01d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:54.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 shutdown_connections 2023-10-14T17:00:54.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:54.842+0000 7f84e1c9b640 1 -- 172.21.15.29:0/2851926243 wait complete. 2023-10-14T17:00:54.859 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:55.035 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:55.034+0000 7f1477037040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:55.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.052+0000 7f3ab29d5640 1 Processor -- start 2023-10-14T17:00:55.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.052+0000 7f3ab29d5640 1 -- start start 2023-10-14T17:00:55.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3ab29d5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3aac0b0b20 0x7f3aac164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3ab29d5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac06f7f0 con 0x7f3aac159f00 2023-10-14T17:00:55.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3ab29d5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aac06c5e0 con 0x7f3aac159b20 2023-10-14T17:00:55.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3ab29d5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aac06c8c0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3aab7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3aac159f00 0x7f3aac0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39952/0 (socket says 172.21.15.29:39952) 2023-10-14T17:00:55.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3aab7fe640 1 -- 172.21.15.29:0/2847485998 learned_addr learned my addr 172.21.15.29:0/2847485998 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:55.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.054+0000 7f3ab0f72640 1 --2- 172.21.15.29:0/2847485998 >> v2:172.21.15.186:3300/0 conn(0x7f3aac0b0b20 0x7f3aac164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.055+0000 7f3aaaffd640 1 -- 172.21.15.29:0/2847485998 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 754878296 0 0) 0x7f3aac06f7f0 con 0x7f3aac159f00 2023-10-14T17:00:55.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.055+0000 7f3aaaffd640 1 -- 172.21.15.29:0/2847485998 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a94003520 con 0x7f3aac159f00 2023-10-14T17:00:55.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.055+0000 7f3ab0f72640 1 -- 172.21.15.29:0/2847485998 >> v1:172.21.15.186:6789/0 conn(0x7f3aac159b20 legacy=0x7f3aac06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.055+0000 7f3ab0f72640 1 -- 172.21.15.29:0/2847485998 >> v1:172.21.15.29:6789/0 conn(0x7f3aac159f00 legacy=0x7f3aac0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.055+0000 7f3ab0f72640 1 -- 172.21.15.29:0/2847485998 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aac06cba0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.056+0000 7f3ab0f72640 1 --2- 172.21.15.29:0/2847485998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0b0b20 0x7f3aac164b40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009d90 tx=0x7f3aa402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f1bb0f5798efffe2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3aaaffd640 1 -- 172.21.15.29:0/2847485998 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa402cd00 con 0x7f3aac0b0b20 2023-10-14T17:00:55.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3aaaffd640 1 -- 172.21.15.29:0/2847485998 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa402ce60 con 0x7f3aac0b0b20 2023-10-14T17:00:55.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3aaaffd640 1 -- 172.21.15.29:0/2847485998 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4034640 con 0x7f3aac0b0b20 2023-10-14T17:00:55.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2847485998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0b0b20 msgr2=0x7f3aac164b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2847485998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0b0b20 0x7f3aac164b40 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f3aa4009d90 tx=0x7f3aa402bf50 comp rx=0 tx=0).stop 2023-10-14T17:00:55.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2847485998 shutdown_connections 2023-10-14T17:00:55.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2847485998 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac0b0b20 0x7f3aac164b40 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2847485998 >> 172.21.15.29:0/2847485998 conn(0x7f3aac0cb410 msgr2=0x7f3aac0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.057+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2847485998 shutdown_connections 2023-10-14T17:00:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.058+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2847485998 wait complete. 2023-10-14T17:00:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.059+0000 7f3ab29d5640 1 Processor -- start 2023-10-14T17:00:55.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.059+0000 7f3ab29d5640 1 -- start start 2023-10-14T17:00:55.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac159b20 0x7f3aac0b3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac159f00 0x7f3aac0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0b3720 con 0x7f3aac0b0b20 2023-10-14T17:00:55.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0b2b10 con 0x7f3aac159b20 2023-10-14T17:00:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.060+0000 7f3ab29d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aac0b2df0 con 0x7f3aac159f00 2023-10-14T17:00:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38908/0 (socket says 172.21.15.29:38908) 2023-10-14T17:00:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 -- 172.21.15.29:0/2976758586 learned_addr learned my addr 172.21.15.29:0/2976758586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:55.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aab7fe640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac159b20 0x7f3aac0b3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3ab0f72640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac159f00 0x7f3aac0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac159f00 msgr2=0x7f3aac0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac159f00 0x7f3aac0af6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac159b20 msgr2=0x7f3aac0b3bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac159b20 0x7f3aac0b3bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.061+0000 7f3aabfff640 1 -- 172.21.15.29:0/2976758586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa4009a70 con 0x7f3aac0b0b20 2023-10-14T17:00:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3aabfff640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00c450 tx=0x7f3a9c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c0052d0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a9c005430 con 0x7f3aac0b0b20 2023-10-14T17:00:55.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a9c012870 con 0x7f3aac0b0b20 2023-10-14T17:00:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aac0b30d0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.062+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aac0b2730 con 0x7f3aac0b0b20 2023-10-14T17:00:55.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.063+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aac04e160 con 0x7f3aac0b0b20 2023-10-14T17:00:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.065+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f3a9c0129d0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.065+0000 7f3aa8ff9640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 0x7f3a9806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.065+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f3a9c089610 con 0x7f3aac0b0b20 2023-10-14T17:00:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.066+0000 7f3aab7fe640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 msgr2=0x7f3a9806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:55.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.066+0000 7f3aab7fe640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 0x7f3a9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:55.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.074+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a9c054200 con 0x7f3aac0b0b20 2023-10-14T17:00:55.211 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.209+0000 7f775fcda040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:00:55.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.267+0000 7f3aab7fe640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 msgr2=0x7f3a9806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:55.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.267+0000 7f3aab7fe640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 0x7f3a9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:55.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.272+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3aac0b10b0 con 0x7f3aac0b0b20 2023-10-14T17:00:55.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.274+0000 7f3aa8ff9640 1 -- 172.21.15.29:0/2976758586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v141) v1 ==== 74+0+17129 (secure 0 0 0) 0x7f3a9c057820 con 0x7f3aac0b0b20 2023-10-14T17:00:55.275 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:55.275 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:42.236313+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1488377546":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6832/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/532161644":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3029691751":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/932642367":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3424087599":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6833/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/4142001408":"2023-10-15T17:00:42.236245+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 msgr2=0x7f3a9806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 0x7f3a9806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 msgr2=0x7f3aac1569d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 secure :-1 s=READY pgs=1689 cs=0 l=1 rev1=1 crypto rx=0x7f3a9c00c450 tx=0x7f3a9c00c910 comp rx=0 tx=0).stop 2023-10-14T17:00:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 shutdown_connections 2023-10-14T17:00:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f3a9806b020 0x7f3a9806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3aac159f00 0x7f3aac0af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3aac159b20 0x7f3aac0b3bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 --2- 172.21.15.29:0/2976758586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3aac0b0b20 0x7f3aac1569d0 unknown :-1 s=CLOSED pgs=1689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 >> 172.21.15.29:0/2976758586 conn(0x7f3aac0cb410 msgr2=0x7f3aac06d370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 shutdown_connections 2023-10-14T17:00:55.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.277+0000 7f3ab29d5640 1 -- 172.21.15.29:0/2976758586 wait complete. 2023-10-14T17:00:55.297 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 16, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:00:25.126064+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 17, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:00:26.177199+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:00:55.298 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:00:55.332 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:55.331+0000 7f1477037040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:55.390 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:55.389+0000 7f1477037040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:55.503 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:00:55.502+0000 7f1477037040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:55.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.510+0000 7f30de29f640 1 Processor -- start 2023-10-14T17:00:55.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.510+0000 7f30de29f640 1 -- start start 2023-10-14T17:00:55.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.512+0000 7f30de29f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30d80b1ca0 0x7f30d80d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.512+0000 7f30de29f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.512+0000 7f30de29f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30d80d0970 con 0x7f30d80ce960 2023-10-14T17:00:55.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.512+0000 7f30de29f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30d80d0ea0 con 0x7f30d80b1ca0 2023-10-14T17:00:55.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.512+0000 7f30de29f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80d0bc0 con 0x7f30d80b2080 2023-10-14T17:00:55.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.513+0000 7f30dd29d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30d80b1ca0 0x7f30d80d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.513+0000 7f30dd29d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30d80b1ca0 0x7f30d80d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:56868/0 (socket says 172.21.15.29:56868) 2023-10-14T17:00:55.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.513+0000 7f30dd29d640 1 -- 172.21.15.29:0/2291222346 learned_addr learned my addr 172.21.15.29:0/2291222346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:55.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.513+0000 7f30dca9c640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.514+0000 7f30dd29d640 1 -- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 msgr2=0x7f30d80ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:55.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.514+0000 7f30dd29d640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.514+0000 7f30dd29d640 1 -- 172.21.15.29:0/2291222346 >> v1:172.21.15.186:6789/0 conn(0x7f30d80ce960 legacy=0x7f30d80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.514+0000 7f30dd29d640 1 -- 172.21.15.29:0/2291222346 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d80cd940 con 0x7f30d80b1ca0 2023-10-14T17:00:55.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.515+0000 7f30dd29d640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80d36b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009870 tx=0x7f30d402ba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2e413dacc1940ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.516+0000 7f30cffff640 1 -- 172.21.15.29:0/2291222346 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402f840 con 0x7f30d80b1ca0 2023-10-14T17:00:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.516+0000 7f30cffff640 1 -- 172.21.15.29:0/2291222346 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d402f9a0 con 0x7f30d80b1ca0 2023-10-14T17:00:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.516+0000 7f30cffff640 1 -- 172.21.15.29:0/2291222346 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d402fd30 con 0x7f30d80b1ca0 2023-10-14T17:00:55.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.516+0000 7f30de29f640 1 -- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 msgr2=0x7f30d80d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.516+0000 7f30de29f640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80d36b0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f30d4009870 tx=0x7f30d402ba20 comp rx=0 tx=0).stop 2023-10-14T17:00:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 -- 172.21.15.29:0/2291222346 shutdown_connections 2023-10-14T17:00:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 --2- 172.21.15.29:0/2291222346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80d36b0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 -- 172.21.15.29:0/2291222346 >> 172.21.15.29:0/2291222346 conn(0x7f30d80d2b90 msgr2=0x7f30d80d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 -- 172.21.15.29:0/2291222346 shutdown_connections 2023-10-14T17:00:55.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.517+0000 7f30de29f640 1 -- 172.21.15.29:0/2291222346 wait complete. 2023-10-14T17:00:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.519+0000 7f30de29f640 1 Processor -- start 2023-10-14T17:00:55.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.519+0000 7f30de29f640 1 -- start start 2023-10-14T17:00:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80c70d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80c7610 con 0x7f30d80ce960 2023-10-14T17:00:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80c7840 con 0x7f30d80b1ca0 2023-10-14T17:00:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.520+0000 7f30de29f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30d80c7b20 con 0x7f30d80b2080 2023-10-14T17:00:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38916/0 (socket says 172.21.15.29:38916) 2023-10-14T17:00:55.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 -- 172.21.15.29:0/4245892190 learned_addr learned my addr 172.21.15.29:0/4245892190 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:55.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dd29d640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dca9c640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80c70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 msgr2=0x7f30d80c70d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80c70d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 msgr2=0x7f30d80a03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80a03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.521+0000 7f30dda9e640 1 -- 172.21.15.29:0/4245892190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30d40094d0 con 0x7f30d80ce960 2023-10-14T17:00:55.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30dda9e640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c6b0 tx=0x7f30d000cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0003b70 con 0x7f30d80ce960 2023-10-14T17:00:55.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30d0003cf0 con 0x7f30d80ce960 2023-10-14T17:00:55.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30d0003ea0 con 0x7f30d80ce960 2023-10-14T17:00:55.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30d80c7e00 con 0x7f30d80ce960 2023-10-14T17:00:55.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.522+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30d809c970 con 0x7f30d80ce960 2023-10-14T17:00:55.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.524+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30d804e9b0 con 0x7f30d80ce960 2023-10-14T17:00:55.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.524+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 76) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f30d0005310 con 0x7f30d80ce960 2023-10-14T17:00:55.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.525+0000 7f30cdffb640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 0x7f30bc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.525+0000 7f30dd29d640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 msgr2=0x7f30bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:55.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.525+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f30d0089750 con 0x7f30d80ce960 2023-10-14T17:00:55.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.525+0000 7f30dd29d640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 0x7f30bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:00:55.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.530+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30d0054470 con 0x7f30d80ce960 2023-10-14T17:00:55.564 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.564+0000 7f775fcda040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:00:55.625 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.624+0000 7f775fcda040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:00:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.726+0000 7f30dd29d640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 msgr2=0x7f30bc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/795882907 2023-10-14T17:00:55.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.726+0000 7f30dd29d640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 0x7f30bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:00:55.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.734+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f30d8073a40 con 0x7f30d80ce960 2023-10-14T17:00:55.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.735+0000 7f30cdffb640 1 -- 172.21.15.29:0/4245892190 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v141) v1 ==== 93+0+10 (secure 0 0 0) 0x7f30d0057a90 con 0x7f30d80ce960 2023-10-14T17:00:55.735 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:00:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 msgr2=0x7f30bc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:00:55.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 0x7f30bc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 msgr2=0x7f30d81f1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 secure :-1 s=READY pgs=1690 cs=0 l=1 rev1=1 crypto rx=0x7f30d000c6b0 tx=0x7f30d000cb70 comp rx=0 tx=0).stop 2023-10-14T17:00:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 shutdown_connections 2023-10-14T17:00:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:6832/795882907,v1:172.21.15.29:6833/795882907] conn(0x7f30bc06b020 0x7f30bc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30d80ce960 0x7f30d81f1f80 unknown :-1 s=CLOSED pgs=1690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30d80b2080 0x7f30d80c70d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 --2- 172.21.15.29:0/4245892190 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30d80b1ca0 0x7f30d80a03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.737+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 >> 172.21.15.29:0/4245892190 conn(0x7f30d80d2b90 msgr2=0x7f30d80b1280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.738+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 shutdown_connections 2023-10-14T17:00:55.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.738+0000 7f30de29f640 1 -- 172.21.15.29:0/4245892190 wait complete. 2023-10-14T17:00:55.746 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.746+0000 7f775fcda040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:00:55.758 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:00:55.834 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.832+0000 7f76e0f61640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.835 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.832+0000 7f76e0f61640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.835 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.832+0000 7f76e0f61640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.832+0000 7f76e0f61640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e0f61640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e4f69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.837 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e4f69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.837 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e4f69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.837 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e4f69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.838 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:00:55.833+0000 7f76e4f69640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:00:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.958+0000 7f4f0d6ca640 1 Processor -- start 2023-10-14T17:00:55.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.958+0000 7f4f0d6ca640 1 -- start start 2023-10-14T17:00:55.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4f0d6ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4f0806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4f0d6ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0815f4a0 con 0x7f4f080ad970 2023-10-14T17:00:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4f0d6ca640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f0806c960 con 0x7f4f080cda80 2023-10-14T17:00:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4f0d6ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0815b9d0 con 0x7f4f0806c040 2023-10-14T17:00:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4efffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4f080ad970 0x7f4f0815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39960/0 (socket says 172.21.15.29:39960) 2023-10-14T17:00:55.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4efffff640 1 -- 172.21.15.29:0/3860051982 learned_addr learned my addr 172.21.15.29:0/3860051982 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.960+0000 7f4f07fff640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4f0806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219667026 0 0) 0x7f4f0815f4a0 con 0x7f4f080ad970 2023-10-14T17:00:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ef4003520 con 0x7f4f080ad970 2023-10-14T17:00:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2842286528 0 0) 0x7f4ef4003520 con 0x7f4f080ad970 2023-10-14T17:00:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 msgr2=0x7f4f0806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4f0806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 >> v1:172.21.15.186:6789/0 conn(0x7f4f080cda80 legacy=0x7f4f0815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f0815bcb0 con 0x7f4f080ad970 2023-10-14T17:00:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4ef0001670 con 0x7f4f080ad970 2023-10-14T17:00:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 >> v1:172.21.15.29:6789/0 conn(0x7f4f080ad970 legacy=0x7f4f0815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef40039a0 0x7f4ef4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4ef40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4ef4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0815b9d0 con 0x7f4ef4004d00 2023-10-14T17:00:55.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef400ba60 con 0x7f4ef40039a0 2023-10-14T17:00:55.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef400bc20 con 0x7f4f0806c040 2023-10-14T17:00:55.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.961+0000 7f4f0cec9640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4ef4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f07fff640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef40039a0 0x7f4ef4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f0cec9640 1 -- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 msgr2=0x7f4ef40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4efffff640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4ef40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f0cec9640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0806c040 0x7f4ef40047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f0cec9640 1 -- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef40039a0 msgr2=0x7f4ef4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:55.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f0cec9640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4ef40039a0 0x7f4ef4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.962+0000 7f4f0cec9640 1 -- 172.21.15.29:0/3860051982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ef4003520 con 0x7f4ef4004d00 2023-10-14T17:00:55.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.963+0000 7f4f0cec9640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4ef4005180 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f4f00001e20 tx=0x7f4ef40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d47fff27ec94a406 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.963+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00003b70 con 0x7f4ef4004d00 2023-10-14T17:00:55.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f00003cd0 con 0x7f4ef4004d00 2023-10-14T17:00:55.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f077fe640 1 -- 172.21.15.29:0/3860051982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f00005310 con 0x7f4ef4004d00 2023-10-14T17:00:55.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/3860051982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 msgr2=0x7f4ef4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 --2- 172.21.15.29:0/3860051982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4ef4005180 secure :-1 s=READY pgs=1703 cs=0 l=1 rev1=1 crypto rx=0x7f4f00001e20 tx=0x7f4ef40057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0cec9640 1 -- 172.21.15.29:0/3860051982 reap_dead start 2023-10-14T17:00:55.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/3860051982 shutdown_connections 2023-10-14T17:00:55.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/3860051982 >> 172.21.15.29:0/3860051982 conn(0x7f4f080ce6a0 msgr2=0x7f4f080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:55.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/3860051982 shutdown_connections 2023-10-14T17:00:55.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.964+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/3860051982 wait complete. 2023-10-14T17:00:55.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.966+0000 7f4f0d6ca640 1 Processor -- start 2023-10-14T17:00:55.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.966+0000 7f4f0d6ca640 1 -- start start 2023-10-14T17:00:55.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f080c9700 0x7f4f080acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0814d850 0x7f4f0814dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:55.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080b3c50 con 0x7f4ef4004d00 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f080b2180 con 0x7f4f080c9700 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f0d6ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f0814e210 con 0x7f4f0814d850 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39048/0 (socket says 172.21.15.29:39048) 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.967+0000 7f4f07fff640 1 -- 172.21.15.29:0/1840541112 learned_addr learned my addr 172.21.15.29:0/1840541112 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:55.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4efffff640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f080c9700 0x7f4f080acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 -- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0814d850 msgr2=0x7f4f0814dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0814d850 0x7f4f0814dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 -- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f080c9700 msgr2=0x7f4f080acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f080c9700 0x7f4f080acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:55.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 -- 172.21.15.29:0/1840541112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f080c5d30 con 0x7f4ef4004d00 2023-10-14T17:00:55.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f07fff640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80115b0 tx=0x7f4ef8011a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:55.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.968+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8015870 con 0x7f4ef4004d00 2023-10-14T17:00:55.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.969+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ef80159d0 con 0x7f4ef4004d00 2023-10-14T17:00:55.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.969+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f080ca280 con 0x7f4ef4004d00 2023-10-14T17:00:55.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.969+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ef8015d40 con 0x7f4ef4004d00 2023-10-14T17:00:55.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.969+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f080c9fa0 con 0x7f4ef4004d00 2023-10-14T17:00:55.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.970+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f0806b330 con 0x7f4ef4004d00 2023-10-14T17:00:55.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.971+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4ef800f480 con 0x7f4ef4004d00 2023-10-14T17:00:55.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.972+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f4ef8089be0 con 0x7f4ef4004d00 2023-10-14T17:00:55.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:55.976+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ef8054d80 con 0x7f4ef4004d00 2023-10-14T17:00:56.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.169+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4f0804e9b0 con 0x7f4ef4004d00 2023-10-14T17:00:56.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.170+0000 7f4f057fa640 1 -- 172.21.15.29:0/1840541112 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v142) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4ef8022020 con 0x7f4ef4004d00 2023-10-14T17:00:56.171 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:00:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 msgr2=0x7f4f081502d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 secure :-1 s=READY pgs=1704 cs=0 l=1 rev1=1 crypto rx=0x7f4ef80115b0 tx=0x7f4ef8011a70 comp rx=0 tx=0).stop 2023-10-14T17:00:56.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 shutdown_connections 2023-10-14T17:00:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f0814d850 0x7f4f0814dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f080c9700 0x7f4f080acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 --2- 172.21.15.29:0/1840541112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4ef4004d00 0x7f4f081502d0 unknown :-1 s=CLOSED pgs=1704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 >> 172.21.15.29:0/1840541112 conn(0x7f4f080ce6a0 msgr2=0x7f4f0806f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:56.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 shutdown_connections 2023-10-14T17:00:56.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.172+0000 7f4f0d6ca640 1 -- 172.21.15.29:0/1840541112 wait complete. 2023-10-14T17:00:56.189 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:00:56.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.398+0000 7f4a28c90640 1 Processor -- start 2023-10-14T17:00:56.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.398+0000 7f4a28c90640 1 -- start start 2023-10-14T17:00:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a28c90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4a2406a4b0 0x7f4a240ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a28c90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a240c9ca0 0x7f4a240ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a28c90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a240cb030 con 0x7f4a2415ad70 2023-10-14T17:00:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a28c90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a240cae30 con 0x7f4a2406a4b0 2023-10-14T17:00:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a28c90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a240c91c0 con 0x7f4a240c9ca0 2023-10-14T17:00:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a2259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4a2415ad70 0x7f4a24069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39972/0 (socket says 172.21.15.29:39972) 2023-10-14T17:00:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a2259c640 1 -- 172.21.15.29:0/2827671279 learned_addr learned my addr 172.21.15.29:0/2827671279 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.400+0000 7f4a1bfff640 1 --2- 172.21.15.29:0/2827671279 >> v2:172.21.15.29:3300/0 conn(0x7f4a2406a4b0 0x7f4a240ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a22d9d640 1 --2- 172.21.15.29:0/2827671279 >> v2:172.21.15.186:3300/0 conn(0x7f4a240c9ca0 0x7f4a240ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a21d9b640 1 -- 172.21.15.29:0/2827671279 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2922365570 0 0) 0x7f4a240cb030 con 0x7f4a2415ad70 2023-10-14T17:00:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a21d9b640 1 -- 172.21.15.29:0/2827671279 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a10003520 con 0x7f4a2415ad70 2023-10-14T17:00:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a1bfff640 1 -- 172.21.15.29:0/2827671279 >> v2:172.21.15.186:3300/0 conn(0x7f4a240c9ca0 msgr2=0x7f4a240ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a1bfff640 1 --2- 172.21.15.29:0/2827671279 >> v2:172.21.15.186:3300/0 conn(0x7f4a240c9ca0 0x7f4a240ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a1bfff640 1 -- 172.21.15.29:0/2827671279 >> v1:172.21.15.29:6789/0 conn(0x7f4a2415ad70 legacy=0x7f4a24069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.401+0000 7f4a1bfff640 1 -- 172.21.15.29:0/2827671279 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a240c94a0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.402+0000 7f4a1bfff640 1 --2- 172.21.15.29:0/2827671279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ca3b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4a2406f270 tx=0x7f4a0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b79cfc88e89de422 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.402+0000 7f4a21d9b640 1 -- 172.21.15.29:0/2827671279 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c031830 con 0x7f4a2406a4b0 2023-10-14T17:00:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.402+0000 7f4a21d9b640 1 -- 172.21.15.29:0/2827671279 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a0c031990 con 0x7f4a2406a4b0 2023-10-14T17:00:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.402+0000 7f4a21d9b640 1 -- 172.21.15.29:0/2827671279 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a0c031d20 con 0x7f4a2406a4b0 2023-10-14T17:00:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 -- 172.21.15.29:0/2827671279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 msgr2=0x7f4a240ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 --2- 172.21.15.29:0/2827671279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ca3b0 secure :-1 s=READY pgs=1706 cs=0 l=1 rev1=1 crypto rx=0x7f4a2406f270 tx=0x7f4a0c02b8d0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 -- 172.21.15.29:0/2827671279 shutdown_connections 2023-10-14T17:00:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 --2- 172.21.15.29:0/2827671279 >> v2:172.21.15.186:3300/0 conn(0x7f4a240c9ca0 0x7f4a240ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 --2- 172.21.15.29:0/2827671279 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ca3b0 unknown :-1 s=CLOSED pgs=1706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 -- 172.21.15.29:0/2827671279 >> 172.21.15.29:0/2827671279 conn(0x7f4a240ce9b0 msgr2=0x7f4a240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 -- 172.21.15.29:0/2827671279 shutdown_connections 2023-10-14T17:00:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.403+0000 7f4a28c90640 1 -- 172.21.15.29:0/2827671279 wait complete. 2023-10-14T17:00:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.405+0000 7f4a28c90640 1 Processor -- start 2023-10-14T17:00:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.405+0000 7f4a28c90640 1 -- start start 2023-10-14T17:00:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a240c9ca0 0x7f4a240c2eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a2415ad70 0x7f4a24158b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240c24a0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240c2a20 con 0x7f4a240c9ca0 2023-10-14T17:00:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a28c90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a240c8a40 con 0x7f4a2415ad70 2023-10-14T17:00:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39058/0 (socket says 172.21.15.29:39058) 2023-10-14T17:00:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 -- 172.21.15.29:0/1065918246 learned_addr learned my addr 172.21.15.29:0/1065918246 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a1bfff640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a240c9ca0 0x7f4a240c2eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 -- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a2415ad70 msgr2=0x7f4a24158b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a2415ad70 0x7f4a24158b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 -- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a240c9ca0 msgr2=0x7f4a240c2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a240c9ca0 0x7f4a240c2eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.406+0000 7f4a2259c640 1 -- 172.21.15.29:0/1065918246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a0c009400 con 0x7f4a2406a4b0 2023-10-14T17:00:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a2259c640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400c440 tx=0x7f4a1400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14005ce0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a14005e40 con 0x7f4a2406a4b0 2023-10-14T17:00:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a140072b0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a240acf90 con 0x7f4a2406a4b0 2023-10-14T17:00:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.407+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a240c26b0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.408+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a2404e160 con 0x7f4a2406a4b0 2023-10-14T17:00:56.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.409+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 77) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f4a14007410 con 0x7f4a2406a4b0 2023-10-14T17:00:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.410+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f4a140889e0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.414+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a14053bb0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.599+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f4a24069e40 con 0x7f4a2406a4b0 2023-10-14T17:00:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.600+0000 7f4a1affd640 1 -- 172.21.15.29:0/1065918246 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v142) v1 ==== 100+0+11 (secure 0 0 0) 0x7f4a140170a0 con 0x7f4a2406a4b0 2023-10-14T17:00:56.601 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:00:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.602+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 msgr2=0x7f4a240ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.602+0000 7f4a28c90640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 secure :-1 s=READY pgs=1707 cs=0 l=1 rev1=1 crypto rx=0x7f4a1400c440 tx=0x7f4a1400c900 comp rx=0 tx=0).stop 2023-10-14T17:00:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 shutdown_connections 2023-10-14T17:00:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a2415ad70 0x7f4a24158b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a240c9ca0 0x7f4a240c2eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 --2- 172.21.15.29:0/1065918246 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a2406a4b0 0x7f4a240ac890 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 >> 172.21.15.29:0/1065918246 conn(0x7f4a240ce9b0 msgr2=0x7f4a240afbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 shutdown_connections 2023-10-14T17:00:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.603+0000 7f4a28c90640 1 -- 172.21.15.29:0/1065918246 wait complete. 2023-10-14T17:00:56.620 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:56.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.844+0000 7f36802c9640 1 Processor -- start 2023-10-14T17:00:56.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.844+0000 7f36802c9640 1 -- start start 2023-10-14T17:00:56.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f36802c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f367815ae80 0x7f367815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f36802c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367815ed60 con 0x7f36780ce390 2023-10-14T17:00:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f36802c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36780707e0 con 0x7f367815aaa0 2023-10-14T17:00:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f36802c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367806b810 con 0x7f367815ae80 2023-10-14T17:00:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f367e866640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f36780ce390 0x7f3678070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39988/0 (socket says 172.21.15.29:39988) 2023-10-14T17:00:56.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f367e866640 1 -- 172.21.15.29:0/3073877774 learned_addr learned my addr 172.21.15.29:0/3073877774 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.846+0000 7f367d864640 1 --2- 172.21.15.29:0/3073877774 >> v2:172.21.15.186:3300/0 conn(0x7f367815ae80 0x7f367815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115801049 0 0) 0x7f367815ed60 con 0x7f36780ce390 2023-10-14T17:00:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3664003520 con 0x7f36780ce390 2023-10-14T17:00:56.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3598994606 0 0) 0x7f3664003520 con 0x7f36780ce390 2023-10-14T17:00:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 >> v2:172.21.15.186:3300/0 conn(0x7f367815ae80 msgr2=0x7f367815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 --2- 172.21.15.29:0/3073877774 >> v2:172.21.15.186:3300/0 conn(0x7f367815ae80 0x7f367815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 >> v1:172.21.15.186:6789/0 conn(0x7f367815aaa0 legacy=0x7f36780cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367815ef90 con 0x7f36780ce390 2023-10-14T17:00:56.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f36700019f0 con 0x7f36780ce390 2023-10-14T17:00:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 >> v1:172.21.15.29:6789/0 conn(0x7f36780ce390 legacy=0x7f3678070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3664003970 0x7f3664003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 0x7f3664004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3664005180 0x7f366400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f367806b810 con 0x7f36640047c0 2023-10-14T17:00:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f366400c020 con 0x7f3664003970 2023-10-14T17:00:56.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f366400c1e0 con 0x7f3664005180 2023-10-14T17:00:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.847+0000 7f367d864640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 0x7f3664004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367e065640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3664003970 0x7f3664003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367d864640 1 -- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3664005180 msgr2=0x7f366400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367d864640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3664005180 0x7f366400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367d864640 1 -- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3664003970 msgr2=0x7f3664003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:56.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367d864640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3664003970 0x7f3664003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367d864640 1 -- 172.21.15.29:0/3073877774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3664003520 con 0x7f36640047c0 2023-10-14T17:00:56.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.848+0000 7f367e866640 1 -- 172.21.15.29:0/3073877774 reap_dead start 2023-10-14T17:00:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.849+0000 7f367d864640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 0x7f3664004c40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f366c009a90 tx=0x7f36640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c446be26efddeaf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.849+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c02c6b0 con 0x7f36640047c0 2023-10-14T17:00:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.849+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f366c02c810 con 0x7f36640047c0 2023-10-14T17:00:56.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.849+0000 7f367d063640 1 -- 172.21.15.29:0/3073877774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f366c02cba0 con 0x7f36640047c0 2023-10-14T17:00:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 -- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 msgr2=0x7f3664004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 0x7f3664004c40 secure :-1 s=READY pgs=1709 cs=0 l=1 rev1=1 crypto rx=0x7f366c009a90 tx=0x7f36640057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 -- 172.21.15.29:0/3073877774 shutdown_connections 2023-10-14T17:00:56.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 --2- 172.21.15.29:0/3073877774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36640047c0 0x7f3664004c40 unknown :-1 s=CLOSED pgs=1709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 -- 172.21.15.29:0/3073877774 >> 172.21.15.29:0/3073877774 conn(0x7f36780cb410 msgr2=0x7f36780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 -- 172.21.15.29:0/3073877774 shutdown_connections 2023-10-14T17:00:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.850+0000 7f36802c9640 1 -- 172.21.15.29:0/3073877774 wait complete. 2023-10-14T17:00:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.852+0000 7f36802c9640 1 Processor -- start 2023-10-14T17:00:56.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.852+0000 7f36802c9640 1 -- start start 2023-10-14T17:00:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 0x7f367814fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36780c67b0 0x7f36780c9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36780c7b90 con 0x7f36780c71d0 2023-10-14T17:00:56.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36781502f0 con 0x7f36640047c0 2023-10-14T17:00:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f36802c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36780ca890 con 0x7f36780c67b0 2023-10-14T17:00:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f367e866640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f367e866640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39070/0 (socket says 172.21.15.29:39070) 2023-10-14T17:00:56.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f367e866640 1 -- 172.21.15.29:0/3697400713 learned_addr learned my addr 172.21.15.29:0/3697400713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.853+0000 7f367e065640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 0x7f367814fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367d864640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36780c67b0 0x7f36780c9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 -- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36780c67b0 msgr2=0x7f36780c9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36780c67b0 0x7f36780c9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 -- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 msgr2=0x7f367814fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 0x7f367814fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 -- 172.21.15.29:0/3697400713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36780c9c80 con 0x7f36780c71d0 2023-10-14T17:00:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e065640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 0x7f367814fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:00:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f367e866640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f36780c5a10 tx=0x7f367000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:56.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367000cec0 con 0x7f36780c71d0 2023-10-14T17:00:56.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f367000a610 con 0x7f36780c71d0 2023-10-14T17:00:56.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f367000a9e0 con 0x7f36780c71d0 2023-10-14T17:00:56.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36780c9f60 con 0x7f36780c71d0 2023-10-14T17:00:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.854+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36780ca520 con 0x7f36780c71d0 2023-10-14T17:00:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.855+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367804e160 con 0x7f36780c71d0 2023-10-14T17:00:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.856+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f367000ab40 con 0x7f36780c71d0 2023-10-14T17:00:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.857+0000 7f366affd640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 0x7f365c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:56.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.858+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f3670088ef0 con 0x7f36780c71d0 2023-10-14T17:00:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.858+0000 7f367e065640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 0x7f365c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.858+0000 7f367e065640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 0x7f365c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3678069740 tx=0x7f3674009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:56.862+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f36700579a0 con 0x7f36780c71d0 2023-10-14T17:00:57.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.095+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36780cad70 con 0x7f36780c71d0 2023-10-14T17:00:57.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.096+0000 7f366affd640 1 -- 172.21.15.29:0/3697400713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f367001a070 con 0x7f36780c71d0 2023-10-14T17:00:57.097 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:57.097 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:57.098 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:57.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 msgr2=0x7f365c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 0x7f365c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3678069740 tx=0x7f3674009040 comp rx=0 tx=0).stop 2023-10-14T17:00:57.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 msgr2=0x7f36780c7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 secure :-1 s=READY pgs=1710 cs=0 l=1 rev1=1 crypto rx=0x7f36780c5a10 tx=0x7f367000ccd0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 shutdown_connections 2023-10-14T17:00:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f365c06acc0 0x7f365c06d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36780c71d0 0x7f36780c7650 unknown :-1 s=CLOSED pgs=1710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36780c67b0 0x7f36780c9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 --2- 172.21.15.29:0/3697400713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36640047c0 0x7f367814fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 >> 172.21.15.29:0/3697400713 conn(0x7f36780cb410 msgr2=0x7f36780ccfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:57.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 shutdown_connections 2023-10-14T17:00:57.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.099+0000 7f36802c9640 1 -- 172.21.15.29:0/3697400713 wait complete. 2023-10-14T17:00:57.118 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.342+0000 7f601f4c9640 1 Processor -- start 2023-10-14T17:00:57.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.342+0000 7f601f4c9640 1 -- start start 2023-10-14T17:00:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.343+0000 7f601f4c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f601815aaa0 0x7f601815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.343+0000 7f601f4c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f6018070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.343+0000 7f601f4c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f601815ed60 con 0x7f601815ae80 2023-10-14T17:00:57.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.343+0000 7f601f4c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f601806b810 con 0x7f601815aaa0 2023-10-14T17:00:57.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.343+0000 7f601f4c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60180707e0 con 0x7f60180ce390 2023-10-14T17:00:57.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.344+0000 7f601d265640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f601815aaa0 0x7f601815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.344+0000 7f601da66640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f6018070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.344+0000 7f601d265640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f601815aaa0 0x7f601815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:57050/0 (socket says 172.21.15.29:57050) 2023-10-14T17:00:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.344+0000 7f601d265640 1 -- 172.21.15.29:0/1209682607 learned_addr learned my addr 172.21.15.29:0/1209682607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.345+0000 7f601d265640 1 -- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 msgr2=0x7f6018070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.345+0000 7f601d265640 1 --2- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f6018070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.345+0000 7f601d265640 1 -- 172.21.15.29:0/1209682607 >> v1:172.21.15.186:6789/0 conn(0x7f601815ae80 legacy=0x7f60180cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.345+0000 7f601d265640 1 -- 172.21.15.29:0/1209682607 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601815ef90 con 0x7f601815aaa0 2023-10-14T17:00:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.346+0000 7f601d265640 1 --2- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815aaa0 0x7f601815e820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6014009e50 tx=0x7f601402bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91c00996e66d6bb2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f600ffff640 1 -- 172.21.15.29:0/1209682607 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601402cb00 con 0x7f601815aaa0 2023-10-14T17:00:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f600ffff640 1 -- 172.21.15.29:0/1209682607 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f601402cc60 con 0x7f601815aaa0 2023-10-14T17:00:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f600ffff640 1 -- 172.21.15.29:0/1209682607 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60140343f0 con 0x7f601815aaa0 2023-10-14T17:00:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f601f4c9640 1 -- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815aaa0 msgr2=0x7f601815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f601f4c9640 1 --2- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815aaa0 0x7f601815e820 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6014009e50 tx=0x7f601402bf60 comp rx=0 tx=0).stop 2023-10-14T17:00:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f601f4c9640 1 -- 172.21.15.29:0/1209682607 shutdown_connections 2023-10-14T17:00:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.347+0000 7f601f4c9640 1 --2- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f6018070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.348+0000 7f601f4c9640 1 --2- 172.21.15.29:0/1209682607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815aaa0 0x7f601815e820 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.348+0000 7f601f4c9640 1 -- 172.21.15.29:0/1209682607 >> 172.21.15.29:0/1209682607 conn(0x7f60180cb410 msgr2=0x7f60180cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.348+0000 7f601f4c9640 1 -- 172.21.15.29:0/1209682607 shutdown_connections 2023-10-14T17:00:57.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.348+0000 7f601f4c9640 1 -- 172.21.15.29:0/1209682607 wait complete. 2023-10-14T17:00:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.350+0000 7f601f4c9640 1 Processor -- start 2023-10-14T17:00:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.350+0000 7f601f4c9640 1 -- start start 2023-10-14T17:00:57.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f601815cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815ae80 0x7f60180c1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60180c24c0 con 0x7f601815aaa0 2023-10-14T17:00:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601815bc90 con 0x7f601815ae80 2023-10-14T17:00:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601f4c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f601815bf70 con 0x7f60180ce390 2023-10-14T17:00:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.351+0000 7f601ca64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39074/0 (socket says 172.21.15.29:39074) 2023-10-14T17:00:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 -- 172.21.15.29:0/3086982503 learned_addr learned my addr 172.21.15.29:0/3086982503 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601d265640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f601815cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601da66640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815ae80 0x7f60180c1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 -- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 msgr2=0x7f601815cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f601815cc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 -- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815ae80 msgr2=0x7f60180c1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815ae80 0x7f60180c1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 -- 172.21.15.29:0/3086982503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6014009b30 con 0x7f601815aaa0 2023-10-14T17:00:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.352+0000 7f601ca64640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f600800c3c0 tx=0x7f600800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.354+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60080052d0 con 0x7f601815aaa0 2023-10-14T17:00:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.354+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6008005450 con 0x7f601815aaa0 2023-10-14T17:00:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.354+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6008003850 con 0x7f601815aaa0 2023-10-14T17:00:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.354+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601815c250 con 0x7f601815aaa0 2023-10-14T17:00:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.354+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601815d1a0 con 0x7f601815aaa0 2023-10-14T17:00:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.355+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60180c2670 con 0x7f601815aaa0 2023-10-14T17:00:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.356+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f60080039b0 con 0x7f601815aaa0 2023-10-14T17:00:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.357+0000 7f600dffb640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 0x7f5ffc06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.357+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f6008088af0 con 0x7f601815aaa0 2023-10-14T17:00:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.357+0000 7f601d265640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 0x7f5ffc06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.358+0000 7f601d265640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 0x7f5ffc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6014035000 tx=0x7f6014030000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.361+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60180c2670 con 0x7f601815aaa0 2023-10-14T17:00:57.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.558+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f601804e160 con 0x7f601815aaa0 2023-10-14T17:00:57.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.560+0000 7f600dffb640 1 -- 172.21.15.29:0/3086982503 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+17689 (secure 0 0 0) 0x7f6008017090 con 0x7f601815aaa0 2023-10-14T17:00:57.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:57.561 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":142,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:55.748687+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/960844541":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/502397156":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/2049846967":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/4089732493":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3980098395":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6832/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3374539339":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/1488377546":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6832/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/532161644":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3029691751":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/932642367":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3424087599":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/2982227449":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/4142001408":"2023-10-15T17:00:42.236245+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:57.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.562+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 msgr2=0x7f5ffc06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.562+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 0x7f5ffc06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6014035000 tx=0x7f6014030000 comp rx=0 tx=0).stop 2023-10-14T17:00:57.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.562+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 msgr2=0x7f601815b320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.562+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 secure :-1 s=READY pgs=1711 cs=0 l=1 rev1=1 crypto rx=0x7f600800c3c0 tx=0x7f600800c880 comp rx=0 tx=0).stop 2023-10-14T17:00:57.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 shutdown_connections 2023-10-14T17:00:57.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f5ffc06acc0 0x7f5ffc06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f601815ae80 0x7f60180c1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f601815aaa0 0x7f601815b320 unknown :-1 s=CLOSED pgs=1711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 --2- 172.21.15.29:0/3086982503 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60180ce390 0x7f601815cc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 >> 172.21.15.29:0/3086982503 conn(0x7f60180cb410 msgr2=0x7f60180b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:57.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 shutdown_connections 2023-10-14T17:00:57.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.563+0000 7f601f4c9640 1 -- 172.21.15.29:0/3086982503 wait complete. 2023-10-14T17:00:57.584 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T17:00:57.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:57.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.808+0000 7f9b56799640 1 Processor -- start 2023-10-14T17:00:57.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.808+0000 7f9b56799640 1 -- start start 2023-10-14T17:00:57.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.809+0000 7f9b56799640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b500d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.809+0000 7f9b56799640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9b5015cfc0 0x7f9b500b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.809+0000 7f9b56799640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b500d15a0 con 0x7f9b500d1090 2023-10-14T17:00:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.809+0000 7f9b56799640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b500d1880 con 0x7f9b5015cfc0 2023-10-14T17:00:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.809+0000 7f9b56799640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500d1b60 con 0x7f9b5015cbe0 2023-10-14T17:00:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b55f98640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b500d1090 0x7f9b5006bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40000/0 (socket says 172.21.15.29:40000) 2023-10-14T17:00:57.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b55f98640 1 -- 172.21.15.29:0/2637548600 learned_addr learned my addr 172.21.15.29:0/2637548600 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:57.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b54f96640 1 --2- 172.21.15.29:0/2637548600 >> v2:172.21.15.186:3300/0 conn(0x7f9b5015cfc0 0x7f9b500b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3575323094 0 0) 0x7f9b500d15a0 con 0x7f9b500d1090 2023-10-14T17:00:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b38003520 con 0x7f9b500d1090 2023-10-14T17:00:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.810+0000 7f9b55797640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b500d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2383938461 0 0) 0x7f9b38003520 con 0x7f9b500d1090 2023-10-14T17:00:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 msgr2=0x7f9b500d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b500d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 >> v2:172.21.15.186:3300/0 conn(0x7f9b5015cfc0 msgr2=0x7f9b500b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 --2- 172.21.15.29:0/2637548600 >> v2:172.21.15.186:3300/0 conn(0x7f9b5015cfc0 0x7f9b500b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500d1e40 con 0x7f9b500d1090 2023-10-14T17:00:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9b40001670 con 0x7f9b500d1090 2023-10-14T17:00:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 >> v1:172.21.15.29:6789/0 conn(0x7f9b500d1090 legacy=0x7f9b5006bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b38003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b38003f40 0x7f9b380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38004d00 0x7f9b38005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500d1880 con 0x7f9b38003f40 2023-10-14T17:00:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b500d1b60 con 0x7f9b38004d00 2023-10-14T17:00:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3800bac0 con 0x7f9b5015cbe0 2023-10-14T17:00:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.811+0000 7f9b54f96640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b38003f40 0x7f9b380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b55797640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b38003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b54f96640 1 -- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 msgr2=0x7f9b38003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b54f96640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015cbe0 0x7f9b38003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b54f96640 1 -- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38004d00 msgr2=0x7f9b38005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b54f96640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38004d00 0x7f9b38005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.812+0000 7f9b54f96640 1 -- 172.21.15.29:0/2637548600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b38003520 con 0x7f9b38003f40 2023-10-14T17:00:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.813+0000 7f9b54f96640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b38003f40 0x7f9b380047c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009a90 tx=0x7f9b380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b7a6fcb789bcdf0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.813+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4402c6b0 con 0x7f9b38003f40 2023-10-14T17:00:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.813+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b4402c810 con 0x7f9b38003f40 2023-10-14T17:00:57.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.813+0000 7f9b4ffff640 1 -- 172.21.15.29:0/2637548600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b4402cba0 con 0x7f9b38003f40 2023-10-14T17:00:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 -- 172.21.15.29:0/2637548600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b38003f40 msgr2=0x7f9b380047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 --2- 172.21.15.29:0/2637548600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b38003f40 0x7f9b380047c0 secure :-1 s=READY pgs=1713 cs=0 l=1 rev1=1 crypto rx=0x7f9b44009a90 tx=0x7f9b380057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b55f98640 1 -- 172.21.15.29:0/2637548600 reap_dead start 2023-10-14T17:00:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 -- 172.21.15.29:0/2637548600 shutdown_connections 2023-10-14T17:00:57.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 -- 172.21.15.29:0/2637548600 >> 172.21.15.29:0/2637548600 conn(0x7f9b500cf570 msgr2=0x7f9b500cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 -- 172.21.15.29:0/2637548600 shutdown_connections 2023-10-14T17:00:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.814+0000 7f9b56799640 1 -- 172.21.15.29:0/2637548600 wait complete. 2023-10-14T17:00:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.816+0000 7f9b56799640 1 Processor -- start 2023-10-14T17:00:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.817+0000 7f9b56799640 1 -- start start 2023-10-14T17:00:57.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38003f40 0x7f9b50153310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015d820 0x7f9b5015dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b501508a0 con 0x7f9b50150420 2023-10-14T17:00:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5015d4e0 con 0x7f9b38003f40 2023-10-14T17:00:57.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b56799640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b50154510 con 0x7f9b5015d820 2023-10-14T17:00:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b54f96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b54f96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39094/0 (socket says 172.21.15.29:39094) 2023-10-14T17:00:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b54f96640 1 -- 172.21.15.29:0/1651689175 learned_addr learned my addr 172.21.15.29:0/1651689175 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b55797640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38003f40 0x7f9b50153310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b54f96640 1 -- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015d820 msgr2=0x7f9b5015dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.818+0000 7f9b54f96640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015d820 0x7f9b5015dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b54f96640 1 -- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38003f40 msgr2=0x7f9b50153310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b54f96640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38003f40 0x7f9b50153310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b54f96640 1 -- 172.21.15.29:0/1651689175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b500b74f0 con 0x7f9b50150420 2023-10-14T17:00:57.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b54f96640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9b44037040 tx=0x7f9b44004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44003810 con 0x7f9b50150420 2023-10-14T17:00:57.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b44003970 con 0x7f9b50150420 2023-10-14T17:00:57.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b44003ce0 con 0x7f9b50150420 2023-10-14T17:00:57.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b50153900 con 0x7f9b50150420 2023-10-14T17:00:57.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.819+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b50153ec0 con 0x7f9b50150420 2023-10-14T17:00:57.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.821+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b5004e9b0 con 0x7f9b50150420 2023-10-14T17:00:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.822+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f9b44047450 con 0x7f9b50150420 2023-10-14T17:00:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.823+0000 7f9b4dffb640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 0x7f9b3c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.823+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f9b440ab950 con 0x7f9b50150420 2023-10-14T17:00:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.823+0000 7f9b55797640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 0x7f9b3c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:57.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.824+0000 7f9b55797640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 0x7f9b3c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b48004350 tx=0x7f9b4800b040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:57.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:57.828+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b44076990 con 0x7f9b50150420 2023-10-14T17:00:58.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.068+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b500b4ec0 con 0x7f9b50150420 2023-10-14T17:00:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.069+0000 7f9b4dffb640 1 -- 172.21.15.29:0/1651689175 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f9b440044e0 con 0x7f9b50150420 2023-10-14T17:00:58.070 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:58.070 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:58.070 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:58.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.071+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 msgr2=0x7f9b3c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.071+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 0x7f9b3c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9b48004350 tx=0x7f9b4800b040 comp rx=0 tx=0).stop 2023-10-14T17:00:58.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.071+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 msgr2=0x7f9b500b6fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.071+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 secure :-1 s=READY pgs=1714 cs=0 l=1 rev1=1 crypto rx=0x7f9b44037040 tx=0x7f9b44004730 comp rx=0 tx=0).stop 2023-10-14T17:00:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 shutdown_connections 2023-10-14T17:00:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f9b3c06acc0 0x7f9b3c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b5015d820 0x7f9b5015dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b50150420 0x7f9b500b6fb0 unknown :-1 s=CLOSED pgs=1714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 --2- 172.21.15.29:0/1651689175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b38003f40 0x7f9b50153310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 >> 172.21.15.29:0/1651689175 conn(0x7f9b500cf570 msgr2=0x7f9b501520c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 shutdown_connections 2023-10-14T17:00:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.072+0000 7f9b56799640 1 -- 172.21.15.29:0/1651689175 wait complete. 2023-10-14T17:00:58.091 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:00:58.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.308+0000 7f32c51fb640 1 Processor -- start 2023-10-14T17:00:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.308+0000 7f32c51fb640 1 -- start start 2023-10-14T17:00:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.309+0000 7f32c51fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32c00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.309+0000 7f32c51fb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c015b9d0 con 0x7f32c006c040 2023-10-14T17:00:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.309+0000 7f32c51fb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32c015bcb0 con 0x7f32c00ccfb0 2023-10-14T17:00:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.309+0000 7f32c51fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c015bf90 con 0x7f32c00ad970 2023-10-14T17:00:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.310+0000 7f32bffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f32c006c040 0x7f32c006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40014/0 (socket says 172.21.15.29:40014) 2023-10-14T17:00:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.310+0000 7f32bffff640 1 -- 172.21.15.29:0/2405900130 learned_addr learned my addr 172.21.15.29:0/2405900130 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.310+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379796332 0 0) 0x7f32c015b9d0 con 0x7f32c006c040 2023-10-14T17:00:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.310+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32ac003520 con 0x7f32c006c040 2023-10-14T17:00:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.310+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32c00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064762532 0 0) 0x7f32ac003520 con 0x7f32c006c040 2023-10-14T17:00:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 msgr2=0x7f32c00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32c00cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 >> v1:172.21.15.186:6789/0 conn(0x7f32c00ccfb0 legacy=0x7f32c00cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00cc270 con 0x7f32c006c040 2023-10-14T17:00:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f32b0001670 con 0x7f32c006c040 2023-10-14T17:00:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 >> v1:172.21.15.29:6789/0 conn(0x7f32c006c040 legacy=0x7f32c006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32ac004d00 0x7f32ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c015bf90 con 0x7f32ac0039a0 2023-10-14T17:00:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00bbc0 con 0x7f32ac004d00 2023-10-14T17:00:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32ac00bd80 con 0x7f32c00ad970 2023-10-14T17:00:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32bffff640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32bffff640 1 -- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 msgr2=0x7f32ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32bffff640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c00ad970 0x7f32ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32bffff640 1 -- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32ac004d00 msgr2=0x7f32ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.311+0000 7f32c49fa640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32ac004d00 0x7f32ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.312+0000 7f32bffff640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32ac004d00 0x7f32ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.312+0000 7f32bffff640 1 -- 172.21.15.29:0/2405900130 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ac003520 con 0x7f32ac0039a0 2023-10-14T17:00:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.312+0000 7f32c49fa640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32ac004d00 0x7f32ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32bffff640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32ac003dc0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f32b0007fa0 tx=0x7f32ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40283c152261657a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0005ad0 con 0x7f32ac0039a0 2023-10-14T17:00:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0005c50 con 0x7f32ac0039a0 2023-10-14T17:00:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32beffd640 1 -- 172.21.15.29:0/2405900130 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b002c790 con 0x7f32ac0039a0 2023-10-14T17:00:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32c51fb640 1 -- 172.21.15.29:0/2405900130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 msgr2=0x7f32ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.313+0000 7f32c51fb640 1 --2- 172.21.15.29:0/2405900130 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32ac003dc0 secure :-1 s=READY pgs=1718 cs=0 l=1 rev1=1 crypto rx=0x7f32b0007fa0 tx=0x7f32ac0057f0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.314+0000 7f32c49fa640 1 -- 172.21.15.29:0/2405900130 reap_dead start 2023-10-14T17:00:58.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.314+0000 7f32c51fb640 1 -- 172.21.15.29:0/2405900130 shutdown_connections 2023-10-14T17:00:58.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.314+0000 7f32c51fb640 1 -- 172.21.15.29:0/2405900130 >> 172.21.15.29:0/2405900130 conn(0x7f32c00ce6a0 msgr2=0x7f32c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:58.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.314+0000 7f32c51fb640 1 -- 172.21.15.29:0/2405900130 shutdown_connections 2023-10-14T17:00:58.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.314+0000 7f32c51fb640 1 -- 172.21.15.29:0/2405900130 wait complete. 2023-10-14T17:00:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.316+0000 7f32c51fb640 1 Processor -- start 2023-10-14T17:00:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.316+0000 7f32c51fb640 1 -- start start 2023-10-14T17:00:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 0x7f32c015ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32c00c7940 0x7f32c00c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c00c8300 con 0x7f32ac0039a0 2023-10-14T17:00:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32c00af200 con 0x7f32c00c7940 2023-10-14T17:00:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32c51fb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32c01e9600 con 0x7f32c01e9180 2023-10-14T17:00:58.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39158/0 (socket says 172.21.15.29:39158) 2023-10-14T17:00:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 -- 172.21.15.29:0/906251890 learned_addr learned my addr 172.21.15.29:0/906251890 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 0x7f32c015ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 -- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 msgr2=0x7f32c015ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 0x7f32c015ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 -- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32c00c7940 msgr2=0x7f32c00c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32c00c7940 0x7f32c00c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bffff640 1 -- 172.21.15.29:0/906251890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32c00cdcd0 con 0x7f32ac0039a0 2023-10-14T17:00:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.317+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 0x7f32c015ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:58.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32bffff640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f32b000c1a0 tx=0x7f32b0001870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b003c040 con 0x7f32ac0039a0 2023-10-14T17:00:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b002dc30 con 0x7f32ac0039a0 2023-10-14T17:00:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f32b0007070 con 0x7f32ac0039a0 2023-10-14T17:00:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32c00c8e70 con 0x7f32ac0039a0 2023-10-14T17:00:58.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.318+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32c00aebb0 con 0x7f32ac0039a0 2023-10-14T17:00:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.319+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32c006b330 con 0x7f32ac0039a0 2023-10-14T17:00:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.320+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 78) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f32b0032070 con 0x7f32ac0039a0 2023-10-14T17:00:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.321+0000 7f32bcff9640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 0x7f32a006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.321+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f32b00aae80 con 0x7f32ac0039a0 2023-10-14T17:00:58.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.321+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 0x7f32a006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.322+0000 7f32bf7fe640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 0x7f32a006d200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f32c00c9480 tx=0x7f32a8008040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.325+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f32b0076030 con 0x7f32ac0039a0 2023-10-14T17:00:58.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.372+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 79) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f32b0076400 con 0x7f32ac0039a0 2023-10-14T17:00:58.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.565+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32c004e9b0 con 0x7f32ac0039a0 2023-10-14T17:00:58.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.566+0000 7f32bcff9640 1 -- 172.21.15.29:0/906251890 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+4921 (secure 0 0 0) 0x7f32b007c960 con 0x7f32ac0039a0 2023-10-14T17:00:58.567 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:58.567 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":66,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:00:27.165729+0000","modified":"2023-10-14T17:00:30.232420+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8541},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8541":{"gid":8541,"name":"c","rank":0,"incarnation":62,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/1532396905","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1532396905},{"type":"v1","addr":"172.21.15.29:6836","nonce":1532396905}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[17],"metadata_pool":16,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":8}]} 2023-10-14T17:00:58.567 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 66 2023-10-14T17:00:58.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.568+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 msgr2=0x7f32a006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.568+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 0x7f32a006d200 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f32c00c9480 tx=0x7f32a8008040 comp rx=0 tx=0).stop 2023-10-14T17:00:58.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.568+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 msgr2=0x7f32c0073160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.568+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 secure :-1 s=READY pgs=1719 cs=0 l=1 rev1=1 crypto rx=0x7f32b000c1a0 tx=0x7f32b0001870 comp rx=0 tx=0).stop 2023-10-14T17:00:58.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 shutdown_connections 2023-10-14T17:00:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f32a006ad40 0x7f32a006d200 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f32c00c7940 0x7f32c00c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f32c01e9180 0x7f32c015ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 --2- 172.21.15.29:0/906251890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f32ac0039a0 0x7f32c0073160 unknown :-1 s=CLOSED pgs=1719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 >> 172.21.15.29:0/906251890 conn(0x7f32c00ce6a0 msgr2=0x7f32c015d270 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:58.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 shutdown_connections 2023-10-14T17:00:58.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.569+0000 7f32c51fb640 1 -- 172.21.15.29:0/906251890 wait complete. 2023-10-14T17:00:58.589 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:00:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.805+0000 7f97a9503640 1 Processor -- start 2023-10-14T17:00:58.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.806+0000 7f97a9503640 1 -- start start 2023-10-14T17:00:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.807+0000 7f97a9503640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 0x7f97a406a250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.807+0000 7f97a9503640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f97a4145ab0 0x7f97a4145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.807+0000 7f97a9503640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a415dc60 con 0x7f97a40c6310 2023-10-14T17:00:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.807+0000 7f97a9503640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a406a790 con 0x7f97a4145ab0 2023-10-14T17:00:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.807+0000 7f97a9503640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a415d370 con 0x7f97a4069da0 2023-10-14T17:00:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f97a40c6310 0x7f97a40cd4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40026/0 (socket says 172.21.15.29:40026) 2023-10-14T17:00:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a3fff640 1 -- 172.21.15.29:0/4180051441 learned_addr learned my addr 172.21.15.29:0/4180051441 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:00:58.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a8d02640 1 --2- 172.21.15.29:0/4180051441 >> v2:172.21.15.29:3300/0 conn(0x7f97a4145ab0 0x7f97a4145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a37fe640 1 -- 172.21.15.29:0/4180051441 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446646405 0 0) 0x7f97a415dc60 con 0x7f97a40c6310 2023-10-14T17:00:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a37fe640 1 -- 172.21.15.29:0/4180051441 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9790003520 con 0x7f97a40c6310 2023-10-14T17:00:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a8d02640 1 -- 172.21.15.29:0/4180051441 >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 msgr2=0x7f97a406a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f979b7fe640 1 --2- 172.21.15.29:0/4180051441 >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 0x7f97a406a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a8d02640 1 --2- 172.21.15.29:0/4180051441 >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 0x7f97a406a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a8d02640 1 -- 172.21.15.29:0/4180051441 >> v1:172.21.15.29:6789/0 conn(0x7f97a40c6310 legacy=0x7f97a40cd4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f97a8d02640 1 -- 172.21.15.29:0/4180051441 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a40c66f0 con 0x7f97a4145ab0 2023-10-14T17:00:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.808+0000 7f979b7fe640 1 --2- 172.21.15.29:0/4180051441 >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 0x7f97a406a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.809+0000 7f97a8d02640 1 --2- 172.21.15.29:0/4180051441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4145ab0 0x7f97a4145ee0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f97a406ae80 tx=0x7f979c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46966896df008e01 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a37fe640 1 -- 172.21.15.29:0/4180051441 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c02e070 con 0x7f97a4145ab0 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a37fe640 1 -- 172.21.15.29:0/4180051441 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c0315a0 con 0x7f97a4145ab0 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a37fe640 1 -- 172.21.15.29:0/4180051441 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c031930 con 0x7f97a4145ab0 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 -- 172.21.15.29:0/4180051441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4145ab0 msgr2=0x7f97a4145ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 --2- 172.21.15.29:0/4180051441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4145ab0 0x7f97a4145ee0 secure :-1 s=READY pgs=1721 cs=0 l=1 rev1=1 crypto rx=0x7f97a406ae80 tx=0x7f979c02b840 comp rx=0 tx=0).stop 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 -- 172.21.15.29:0/4180051441 shutdown_connections 2023-10-14T17:00:58.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 --2- 172.21.15.29:0/4180051441 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4145ab0 0x7f97a4145ee0 unknown :-1 s=CLOSED pgs=1721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 --2- 172.21.15.29:0/4180051441 >> v2:172.21.15.186:3300/0 conn(0x7f97a4069da0 0x7f97a406a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 -- 172.21.15.29:0/4180051441 >> 172.21.15.29:0/4180051441 conn(0x7f97a40cf910 msgr2=0x7f97a40cfcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 -- 172.21.15.29:0/4180051441 shutdown_connections 2023-10-14T17:00:58.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.810+0000 7f97a9503640 1 -- 172.21.15.29:0/4180051441 wait complete. 2023-10-14T17:00:58.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.812+0000 7f97a9503640 1 Processor -- start 2023-10-14T17:00:58.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.812+0000 7f97a9503640 1 -- start start 2023-10-14T17:00:58.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a40c6310 0x7f97a415c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 0x7f97a40c69e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40b3c30 con 0x7f97a4069da0 2023-10-14T17:00:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40b3f10 con 0x7f97a4145ab0 2023-10-14T17:00:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a9503640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a40b41f0 con 0x7f97a40c6310 2023-10-14T17:00:58.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39180/0 (socket says 172.21.15.29:39180) 2023-10-14T17:00:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.813+0000 7f97a3fff640 1 -- 172.21.15.29:0/2650251034 learned_addr learned my addr 172.21.15.29:0/2650251034 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a8d02640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 0x7f97a40c69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f979b7fe640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a40c6310 0x7f97a415c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 -- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a40c6310 msgr2=0x7f97a415c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a40c6310 0x7f97a415c880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 -- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 msgr2=0x7f97a40c69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:58.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 0x7f97a40c69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:58.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 -- 172.21.15.29:0/2650251034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c009400 con 0x7f97a4069da0 2023-10-14T17:00:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a8d02640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 0x7f97a40c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a3fff640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f979400c410 tx=0x7f979400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9794005ce0 con 0x7f97a4069da0 2023-10-14T17:00:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9794005e40 con 0x7f97a4069da0 2023-10-14T17:00:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97940072f0 con 0x7f97a4069da0 2023-10-14T17:00:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.814+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a40b44d0 con 0x7f97a4069da0 2023-10-14T17:00:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.815+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a40b3900 con 0x7f97a4069da0 2023-10-14T17:00:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.816+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a40b49f0 con 0x7f97a4069da0 2023-10-14T17:00:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.817+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 79) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9794007510 con 0x7f97a4069da0 2023-10-14T17:00:58.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.818+0000 7f97a17fa640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 0x7f978406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:58.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.818+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f9794089100 con 0x7f97a4069da0 2023-10-14T17:00:58.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.818+0000 7f979b7fe640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 0x7f978406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:58.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.819+0000 7f979b7fe640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 0x7f978406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97a406e0f0 tx=0x7f978c008430 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:58.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:58.822+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f97a40b49f0 con 0x7f97a4069da0 2023-10-14T17:00:59.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.022+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97a40b49f0 con 0x7f97a4069da0 2023-10-14T17:00:59.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.026+0000 7f97a17fa640 1 -- 172.21.15.29:0/2650251034 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v142) v1 ==== 74+0+17689 (secure 0 0 0) 0x7f97a40b49f0 con 0x7f97a4069da0 2023-10-14T17:00:59.027 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:00:59.027 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":142,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:00:55.748687+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":16,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:00:25.126064+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"cephfs_data","create_time":"2023-10-14T17:00:26.177199+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/960844541":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/502397156":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/2049846967":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/4089732493":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3980098395":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6832/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3374539339":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/1488377546":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6832/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/532161644":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3029691751":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/932642367":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3424087599":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/2982227449":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/4142001408":"2023-10-15T17:00:42.236245+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:00:59.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 msgr2=0x7f978406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 0x7f978406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f97a406e0f0 tx=0x7f978c008430 comp rx=0 tx=0).stop 2023-10-14T17:00:59.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 msgr2=0x7f97a40c91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 secure :-1 s=READY pgs=1722 cs=0 l=1 rev1=1 crypto rx=0x7f979400c410 tx=0x7f979400c8d0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 shutdown_connections 2023-10-14T17:00:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f978406af00 0x7f978406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97a4145ab0 0x7f97a40c69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97a40c6310 0x7f97a415c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 --2- 172.21.15.29:0/2650251034 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97a4069da0 0x7f97a40c91f0 unknown :-1 s=CLOSED pgs=1722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 >> 172.21.15.29:0/2650251034 conn(0x7f97a40cf910 msgr2=0x7f97a40b0890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 shutdown_connections 2023-10-14T17:00:59.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.029+0000 7f97a9503640 1 -- 172.21.15.29:0/2650251034 wait complete. 2023-10-14T17:00:59.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T17:00:59.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.257+0000 7f3c0e0bb640 1 Processor -- start 2023-10-14T17:00:59.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.258+0000 7f3c0e0bb640 1 -- start start 2023-10-14T17:00:59.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.259+0000 7f3c0e0bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c0815c9f0 0x7f3c080c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.259+0000 7f3c0e0bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c08160a90 con 0x7f3c080c9230 2023-10-14T17:00:59.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.259+0000 7f3c0e0bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c080cd900 con 0x7f3c0815dde0 2023-10-14T17:00:59.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.259+0000 7f3c0e0bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c080cdec0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c0815c9f0 0x7f3c080c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c0815c9f0 0x7f3c080c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39184/0 (socket says 172.21.15.29:39184) 2023-10-14T17:00:59.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 -- 172.21.15.29:0/1806734700 learned_addr learned my addr 172.21.15.29:0/1806734700 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:59.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0c8b8640 1 --1- 172.21.15.29:0/1806734700 >> v1:172.21.15.29:6789/0 conn(0x7f3c080c9230 0x7f3c080709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40040/0 (socket says 172.21.15.29:40040) 2023-10-14T17:00:59.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 -- 172.21.15.29:0/1806734700 >> v1:172.21.15.186:6789/0 conn(0x7f3c0815dde0 legacy=0x7f3c080ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:59.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 -- 172.21.15.29:0/1806734700 >> v1:172.21.15.29:6789/0 conn(0x7f3c080c9230 legacy=0x7f3c080709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.260+0000 7f3c0d0b9640 1 -- 172.21.15.29:0/1806734700 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c080cdbe0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.261+0000 7f3c0d0b9640 1 --2- 172.21.15.29:0/1806734700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c080c8cf0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009870 tx=0x7f3c0402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=292d5d209dc53a6e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3bfbfff640 1 -- 172.21.15.29:0/1806734700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04030860 con 0x7f3c0815c9f0 2023-10-14T17:00:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3bfbfff640 1 -- 172.21.15.29:0/1806734700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c040309c0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3bfbfff640 1 -- 172.21.15.29:0/1806734700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c04030d50 con 0x7f3c0815c9f0 2023-10-14T17:00:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/1806734700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 msgr2=0x7f3c080c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/1806734700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c080c8cf0 secure :-1 s=READY pgs=1724 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009870 tx=0x7f3c0402b9a0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/1806734700 shutdown_connections 2023-10-14T17:00:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/1806734700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c080c8cf0 unknown :-1 s=CLOSED pgs=1724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/1806734700 >> 172.21.15.29:0/1806734700 conn(0x7f3c080ce6a0 msgr2=0x7f3c080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:59.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/1806734700 shutdown_connections 2023-10-14T17:00:59.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.262+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/1806734700 wait complete. 2023-10-14T17:00:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.264+0000 7f3c0e0bb640 1 Processor -- start 2023-10-14T17:00:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.264+0000 7f3c0e0bb640 1 -- start start 2023-10-14T17:00:59.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 0x7f3c08148f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c0815dde0 0x7f3c080c68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c2d90 con 0x7f3c0815c9f0 2023-10-14T17:00:59.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c4150 con 0x7f3c080c9230 2023-10-14T17:00:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0e0bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c080c46d0 con 0x7f3c0815dde0 2023-10-14T17:00:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.265+0000 7f3c0c8b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39188/0 (socket says 172.21.15.29:39188) 2023-10-14T17:00:59.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 -- 172.21.15.29:0/2999955233 learned_addr learned my addr 172.21.15.29:0/2999955233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 -- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c0815dde0 msgr2=0x7f3c080c68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0d0b9640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 0x7f3c08148f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0d8ba640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c0815dde0 0x7f3c080c68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c0815dde0 0x7f3c080c68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 -- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 msgr2=0x7f3c08148f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 0x7f3c08148f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 -- 172.21.15.29:0/2999955233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c040094d0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0d0b9640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 0x7f3c08148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:00:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.266+0000 7f3c0c8b8640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00c500 tx=0x7f3bfc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.267+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc007500 con 0x7f3c0815c9f0 2023-10-14T17:00:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.267+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bfc007660 con 0x7f3c0815c9f0 2023-10-14T17:00:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.267+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bfc0049e0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.267+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0814c5f0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.267+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0814c310 con 0x7f3c0815c9f0 2023-10-14T17:00:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.268+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0804e9b0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.269+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 79) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3bfc004c00 con 0x7f3c0815c9f0 2023-10-14T17:00:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.270+0000 7f3bf9ffb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 0x7f3bec06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.270+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 6512+0+0 (secure 0 0 0) 0x7f3bfc088af0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.270+0000 7f3c0d0b9640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 0x7f3bec06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.271+0000 7f3c0d0b9640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 0x7f3bec06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403b040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.275+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3bfc0538f0 con 0x7f3c0815c9f0 2023-10-14T17:00:59.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.500+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c080c4810 con 0x7f3c0815c9f0 2023-10-14T17:00:59.502 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T17:00:59.502+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T17:00:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.505+0000 7f3bf9ffb640 1 -- 172.21.15.29:0/2999955233 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v67) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3bfc018090 con 0x7f3c0815c9f0 2023-10-14T17:00:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T17:00:59.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.508+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 msgr2=0x7f3bec06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.508+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 0x7f3bec06d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403b040 comp rx=0 tx=0).stop 2023-10-14T17:00:59.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.508+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 msgr2=0x7f3c0814b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.508+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 secure :-1 s=READY pgs=1725 cs=0 l=1 rev1=1 crypto rx=0x7f3bfc00c500 tx=0x7f3bfc00c9c0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 shutdown_connections 2023-10-14T17:00:59.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3bec06af00 0x7f3bec06d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c0815dde0 0x7f3c080c68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c0815c9f0 0x7f3c0814b6c0 unknown :-1 s=CLOSED pgs=1725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 --2- 172.21.15.29:0/2999955233 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c080c9230 0x7f3c08148f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 >> 172.21.15.29:0/2999955233 conn(0x7f3c080ce6a0 msgr2=0x7f3c080aecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:59.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 shutdown_connections 2023-10-14T17:00:59.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.509+0000 7f3c0e0bb640 1 -- 172.21.15.29:0/2999955233 wait complete. 2023-10-14T17:00:59.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T17:00:59.541 INFO:tasks.ceph.mds.c.smithi029.stderr: -8586> 2023-10-14T17:00:31.617+0000 7f6484c97640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:00:59.618 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:00:59.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.750+0000 7f54e316a640 1 Processor -- start 2023-10-14T17:00:59.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.750+0000 7f54e316a640 1 -- start start 2023-10-14T17:00:59.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.751+0000 7f54e316a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc0c92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.751+0000 7f54e316a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54dc0c97f0 0x7f54dc06f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.751+0000 7f54e316a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54dc0ccc80 con 0x7f54dc16cea0 2023-10-14T17:00:59.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.751+0000 7f54e316a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54dc0c9c60 con 0x7f54dc0c97f0 2023-10-14T17:00:59.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.751+0000 7f54e316a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0cdb00 con 0x7f54dc08a450 2023-10-14T17:00:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.752+0000 7f54e1967640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc0c92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.752+0000 7f54e1967640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc0c92b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:34292/0 (socket says 172.21.15.29:34292) 2023-10-14T17:00:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.752+0000 7f54e2969640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54dc0c97f0 0x7f54dc06f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.752+0000 7f54e1967640 1 -- 172.21.15.29:0/3582237419 learned_addr learned my addr 172.21.15.29:0/3582237419 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:59.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.753+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819333221 0 0) 0x7f54dc0ccc80 con 0x7f54dc16cea0 2023-10-14T17:00:59.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.824+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54c8003500 con 0x7f54dc16cea0 2023-10-14T17:00:59.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.825+0000 7f54e2969640 1 -- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 msgr2=0x7f54dc0c92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:00:59.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.825+0000 7f54e2969640 1 --2- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc0c92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.825+0000 7f54e2969640 1 -- 172.21.15.29:0/3582237419 >> v1:172.21.15.186:6789/0 conn(0x7f54dc16cea0 legacy=0x7f54dc0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.825+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 <== mon.1 v1:172.21.15.186:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4194442857 0 0) 0x7f54c8003500 con 0x7f54dc16cea0 2023-10-14T17:00:59.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.825+0000 7f54e2969640 1 -- 172.21.15.29:0/3582237419 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc06c530 con 0x7f54dc0c97f0 2023-10-14T17:00:59.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.827+0000 7f54e2969640 1 --2- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc0c97f0 0x7f54dc06f230 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f54d400f370 tx=0x7f54d40300e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=827355d33a441958 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.828+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d8001670 con 0x7f54dc0c97f0 2023-10-14T17:00:59.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.829+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54d4030c10 con 0x7f54dc0c97f0 2023-10-14T17:00:59.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.829+0000 7f54e1166640 1 -- 172.21.15.29:0/3582237419 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54d400d0d0 con 0x7f54dc0c97f0 2023-10-14T17:00:59.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 -- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc0c97f0 msgr2=0x7f54dc06f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 --2- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc0c97f0 0x7f54dc06f230 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f54d400f370 tx=0x7f54d40300e0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 -- 172.21.15.29:0/3582237419 shutdown_connections 2023-10-14T17:00:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 --2- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc0c92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 --2- 172.21.15.29:0/3582237419 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc0c97f0 0x7f54dc06f230 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 -- 172.21.15.29:0/3582237419 >> 172.21.15.29:0/3582237419 conn(0x7f54dc0ce6a0 msgr2=0x7f54dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 -- 172.21.15.29:0/3582237419 shutdown_connections 2023-10-14T17:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.830+0000 7f54e316a640 1 -- 172.21.15.29:0/3582237419 wait complete. 2023-10-14T17:00:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.833+0000 7f54e316a640 1 Processor -- start 2023-10-14T17:00:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.833+0000 7f54e316a640 1 -- start start 2023-10-14T17:00:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.834+0000 7f54e316a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc089c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e316a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e316a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc16cea0 0x7f54dc0c7510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e316a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc08a140 con 0x7f54dc0c97f0 2023-10-14T17:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e316a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc167f30 con 0x7f54dc16cea0 2023-10-14T17:00:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e316a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54dc0c29d0 con 0x7f54dc08a450 2023-10-14T17:00:59.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e1967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e1967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39214/0 (socket says 172.21.15.29:39214) 2023-10-14T17:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e1967640 1 -- 172.21.15.29:0/3546330025 learned_addr learned my addr 172.21.15.29:0/3546330025 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e2168640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc089c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.835+0000 7f54e2969640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc16cea0 0x7f54dc0c7510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 -- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 msgr2=0x7f54dc089c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc089c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 -- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc16cea0 msgr2=0x7f54dc0c7510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc16cea0 0x7f54dc0c7510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:00:59.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 -- 172.21.15.29:0/3546330025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d400f050 con 0x7f54dc0c97f0 2023-10-14T17:00:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.836+0000 7f54e1967640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00afd0 tx=0x7f54cc00bb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.837+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc00ecf0 con 0x7f54dc0c97f0 2023-10-14T17:00:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.837+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54cc00ee50 con 0x7f54dc0c97f0 2023-10-14T17:00:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.837+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54cc0166b0 con 0x7f54dc0c97f0 2023-10-14T17:00:59.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.837+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54dc0c2cb0 con 0x7f54dc0c97f0 2023-10-14T17:00:59.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.837+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54dc0c3270 con 0x7f54dc0c97f0 2023-10-14T17:00:59.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.838+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54dc04e9b0 con 0x7f54dc0c97f0 2023-10-14T17:00:59.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.840+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 79) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f54cc016810 con 0x7f54dc0c97f0 2023-10-14T17:00:59.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.841+0000 7f54d2ffd640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 0x7f54c006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:00:59.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.841+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f54cc010040 con 0x7f54dc0c97f0 2023-10-14T17:00:59.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.841+0000 7f54e2168640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 0x7f54c006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:00:59.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.842+0000 7f54e2168640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 0x7f54c006d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54d8001ad0 tx=0x7f54d8001a60 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:00:59.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:00:59.846+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54cc055070 con 0x7f54dc0c97f0 2023-10-14T17:01:00.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.086+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f54dc0c8b40 con 0x7f54dc0c97f0 2023-10-14T17:01:00.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.521+0000 7f54d2ffd640 1 -- 172.21.15.29:0/3546330025 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v68) v1 ==== 104+0+0 (secure 0 0 0) 0x7f54cc058690 con 0x7f54dc0c97f0 2023-10-14T17:01:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.526+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 msgr2=0x7f54c006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.526+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 0x7f54c006d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f54d8001ad0 tx=0x7f54d8001a60 comp rx=0 tx=0).stop 2023-10-14T17:01:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.526+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 msgr2=0x7f54dc0c3a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.526+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 secure :-1 s=READY pgs=1729 cs=0 l=1 rev1=1 crypto rx=0x7f54cc00afd0 tx=0x7f54cc00bb40 comp rx=0 tx=0).stop 2023-10-14T17:01:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 shutdown_connections 2023-10-14T17:01:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f54c006af30 0x7f54c006d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54dc16cea0 0x7f54dc0c7510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54dc0c97f0 0x7f54dc0c3a70 unknown :-1 s=CLOSED pgs=1729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 --2- 172.21.15.29:0/3546330025 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54dc08a450 0x7f54dc089c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 >> 172.21.15.29:0/3546330025 conn(0x7f54dc0ce6a0 msgr2=0x7f54dc06fcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:00.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 shutdown_connections 2023-10-14T17:01:00.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.527+0000 7f54e316a640 1 -- 172.21.15.29:0/3546330025 wait complete. 2023-10-14T17:01:00.550 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T17:01:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.819+0000 7f61e63dd640 1 Processor -- start 2023-10-14T17:01:00.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.820+0000 7f61e63dd640 1 -- start start 2023-10-14T17:01:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.821+0000 7f61e63dd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f61e0159950 0x7f61e015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.821+0000 7f61e63dd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e015aef0 con 0x7f61e006c810 2023-10-14T17:01:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61e63dd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61e015d910 con 0x7f61e0070450 2023-10-14T17:01:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61e63dd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61e015e490 con 0x7f61e0159950 2023-10-14T17:01:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61e5bdc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61e006c810 0x7f61e00c9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40056/0 (socket says 172.21.15.29:40056) 2023-10-14T17:01:00.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61e5bdc640 1 -- 172.21.15.29:0/3807900015 learned_addr learned my addr 172.21.15.29:0/3807900015 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:00.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61e53db640 1 --2- 172.21.15.29:0/3807900015 >> v2:172.21.15.186:3300/0 conn(0x7f61e0159950 0x7f61e015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409211063 0 0) 0x7f61d8001490 con 0x7f61e006c810 2023-10-14T17:01:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.822+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61c8003520 con 0x7f61e006c810 2023-10-14T17:01:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2061441146 0 0) 0x7f61c8003520 con 0x7f61e006c810 2023-10-14T17:01:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 >> v2:172.21.15.186:3300/0 conn(0x7f61e0159950 msgr2=0x7f61e015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:00.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 --2- 172.21.15.29:0/3807900015 >> v2:172.21.15.186:3300/0 conn(0x7f61e0159950 0x7f61e015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 >> v1:172.21.15.186:6789/0 conn(0x7f61e0070450 legacy=0x7f61e006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e015dbf0 con 0x7f61e006c810 2023-10-14T17:01:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f61d8001b80 con 0x7f61e006c810 2023-10-14T17:01:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 >> v1:172.21.15.29:6789/0 conn(0x7f61e006c810 legacy=0x7f61e00c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61c8003970 0x7f61c8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 0x7f61c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c8005180 0x7f61c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e015e490 con 0x7f61c80047c0 2023-10-14T17:01:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61c800c120 con 0x7f61c8003970 2023-10-14T17:01:00.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61c800e4f0 con 0x7f61c8005180 2023-10-14T17:01:00.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.823+0000 7f61e4bda640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 0x7f61c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e53db640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61c8003970 0x7f61c8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e4bda640 1 -- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c8005180 msgr2=0x7f61c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:00.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e5bdc640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c8005180 0x7f61c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e4bda640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c8005180 0x7f61c800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e4bda640 1 -- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61c8003970 msgr2=0x7f61c8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:00.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e4bda640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61c8003970 0x7f61c8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e4bda640 1 -- 172.21.15.29:0/3807900015 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61c8003520 con 0x7f61c80047c0 2023-10-14T17:01:00.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.824+0000 7f61e5bdc640 1 -- 172.21.15.29:0/3807900015 reap_dead start 2023-10-14T17:01:00.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.825+0000 7f61e4bda640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 0x7f61c8004c40 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f61d4001750 tx=0x7f61c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f19d339948ed2db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:00.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.825+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40052d0 con 0x7f61c80047c0 2023-10-14T17:01:00.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d402e050 con 0x7f61c80047c0 2023-10-14T17:01:00.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61dffff640 1 -- 172.21.15.29:0/3807900015 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4003830 con 0x7f61c80047c0 2023-10-14T17:01:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 -- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 msgr2=0x7f61c8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 0x7f61c8004c40 secure :-1 s=READY pgs=1731 cs=0 l=1 rev1=1 crypto rx=0x7f61d4001750 tx=0x7f61c80057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 -- 172.21.15.29:0/3807900015 shutdown_connections 2023-10-14T17:01:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 --2- 172.21.15.29:0/3807900015 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61c80047c0 0x7f61c8004c40 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 -- 172.21.15.29:0/3807900015 >> 172.21.15.29:0/3807900015 conn(0x7f61e00ce6a0 msgr2=0x7f61e00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:00.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 -- 172.21.15.29:0/3807900015 shutdown_connections 2023-10-14T17:01:00.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.826+0000 7f61e63dd640 1 -- 172.21.15.29:0/3807900015 wait complete. 2023-10-14T17:01:00.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.828+0000 7f61e63dd640 1 Processor -- start 2023-10-14T17:01:00.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.828+0000 7f61e63dd640 1 -- start start 2023-10-14T17:01:00.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c80047c0 0x7f61e0143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e0147060 0x7f61e006ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00c8640 con 0x7f61e0155700 2023-10-14T17:01:00.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e00c3d10 con 0x7f61e0147060 2023-10-14T17:01:00.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.829+0000 7f61e63dd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61e0144b40 con 0x7f61c80047c0 2023-10-14T17:01:00.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39234/0 (socket says 172.21.15.29:39234) 2023-10-14T17:01:00.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 -- 172.21.15.29:0/1296945005 learned_addr learned my addr 172.21.15.29:0/1296945005 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:00.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 -- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c80047c0 msgr2=0x7f61e0143ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c80047c0 0x7f61e0143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 -- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e0147060 msgr2=0x7f61e006ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e5bdc640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e0147060 0x7f61e006ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e0147060 0x7f61e006ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.830+0000 7f61e4bda640 1 -- 172.21.15.29:0/1296945005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e00c80c0 con 0x7f61e0155700 2023-10-14T17:01:00.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61e4bda640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005040 tx=0x7f61d4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:00.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d40076c0 con 0x7f61e0155700 2023-10-14T17:01:00.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61d4007a30 con 0x7f61e0155700 2023-10-14T17:01:00.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d4006be0 con 0x7f61e0155700 2023-10-14T17:01:00.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e00c3700 con 0x7f61e0155700 2023-10-14T17:01:00.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.831+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e00c39a0 con 0x7f61e0155700 2023-10-14T17:01:00.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.832+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61e00c85a0 con 0x7f61e0155700 2023-10-14T17:01:00.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.833+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f61d4006d40 con 0x7f61e0155700 2023-10-14T17:01:00.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.834+0000 7f61ddffb640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:00.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.834+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 6598+0+0 (secure 0 0 0) 0x7f61d40abcd0 con 0x7f61e0155700 2023-10-14T17:01:00.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.835+0000 7f61e53db640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:00.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.835+0000 7f61e53db640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 0x7f61cc06d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61d0006e80 tx=0x7f61d0006df0 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:00.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:00.839+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61d40768f0 con 0x7f61e0155700 2023-10-14T17:01:01.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.034+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f61e004e9b0 con 0x7f61e0155700 2023-10-14T17:01:01.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.524+0000 7f61ddffb640 1 -- 172.21.15.29:0/1296945005 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v144) v1 ==== 181+0+0 (secure 0 0 0) 0x7f61d404b020 con 0x7f61e0155700 2023-10-14T17:01:01.526 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T17:01:01.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 msgr2=0x7f61cc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 0x7f61cc06d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f61d0006e80 tx=0x7f61d0006df0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 msgr2=0x7f61e0144600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 secure :-1 s=READY pgs=1732 cs=0 l=1 rev1=1 crypto rx=0x7f61d4005040 tx=0x7f61d4001120 comp rx=0 tx=0).stop 2023-10-14T17:01:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 shutdown_connections 2023-10-14T17:01:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f61cc06afc0 0x7f61cc06d480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61e0147060 0x7f61e006ba60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61e0155700 0x7f61e0144600 unknown :-1 s=CLOSED pgs=1732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 --2- 172.21.15.29:0/1296945005 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61c80047c0 0x7f61e0143ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.528+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 >> 172.21.15.29:0/1296945005 conn(0x7f61e00ce6a0 msgr2=0x7f61e006aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.529+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 shutdown_connections 2023-10-14T17:01:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.529+0000 7f61e63dd640 1 -- 172.21.15.29:0/1296945005 wait complete. 2023-10-14T17:01:01.552 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T17:01:01.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.777+0000 7f38561ff640 1 Processor -- start 2023-10-14T17:01:01.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.777+0000 7f38561ff640 1 -- start start 2023-10-14T17:01:01.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.778+0000 7f38561ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f385006c040 0x7f385006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:01.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.778+0000 7f38561ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385015f4a0 con 0x7f38500cda80 2023-10-14T17:01:01.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.778+0000 7f38561ff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f385006c960 con 0x7f38500ad970 2023-10-14T17:01:01.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.778+0000 7f38561ff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f385015b9d0 con 0x7f385006c040 2023-10-14T17:01:01.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38551fd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f385006c040 0x7f385006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:01.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38559fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38500cda80 0x7f385015ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38570/0 (socket says 172.21.15.29:38570) 2023-10-14T17:01:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38559fe640 1 -- 172.21.15.29:0/3277688432 learned_addr learned my addr 172.21.15.29:0/3277688432 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38551fd640 1 -- 172.21.15.29:0/3277688432 >> v1:172.21.15.186:6789/0 conn(0x7f38500ad970 legacy=0x7f385015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38551fd640 1 -- 172.21.15.29:0/3277688432 >> v1:172.21.15.29:6789/0 conn(0x7f38500cda80 legacy=0x7f385015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f38551fd640 1 -- 172.21.15.29:0/3277688432 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385015bcb0 con 0x7f385006c040 2023-10-14T17:01:01.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.779+0000 7f3847fff640 1 -- 172.21.15.29:0/3277688432 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677067529 0 0) 0x7f385015f4a0 con 0x7f38500cda80 2023-10-14T17:01:01.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.780+0000 7f38551fd640 1 --2- 172.21.15.29:0/3277688432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385006c040 0x7f385006c420 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097a0 tx=0x7f384c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39bee79561b1f54 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f3847fff640 1 -- 172.21.15.29:0/3277688432 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c031830 con 0x7f385006c040 2023-10-14T17:01:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f3847fff640 1 -- 172.21.15.29:0/3277688432 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384c031990 con 0x7f385006c040 2023-10-14T17:01:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f3847fff640 1 -- 172.21.15.29:0/3277688432 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384c031d20 con 0x7f385006c040 2023-10-14T17:01:01.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f38561ff640 1 -- 172.21.15.29:0/3277688432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385006c040 msgr2=0x7f385006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f38561ff640 1 --2- 172.21.15.29:0/3277688432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385006c040 0x7f385006c420 secure :-1 s=READY pgs=1734 cs=0 l=1 rev1=1 crypto rx=0x7f384c0097a0 tx=0x7f384c02b8d0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f38561ff640 1 -- 172.21.15.29:0/3277688432 shutdown_connections 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f38561ff640 1 --2- 172.21.15.29:0/3277688432 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f385006c040 0x7f385006c420 unknown :-1 s=CLOSED pgs=1734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.781+0000 7f38561ff640 1 -- 172.21.15.29:0/3277688432 >> 172.21.15.29:0/3277688432 conn(0x7f38500ce6a0 msgr2=0x7f38500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.782+0000 7f38561ff640 1 -- 172.21.15.29:0/3277688432 shutdown_connections 2023-10-14T17:01:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.782+0000 7f38561ff640 1 -- 172.21.15.29:0/3277688432 wait complete. 2023-10-14T17:01:01.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.783+0000 7f38561ff640 1 Processor -- start 2023-10-14T17:01:01.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.784+0000 7f38561ff640 1 -- start start 2023-10-14T17:01:01.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f385006c040 0x7f38500b0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:01.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38500ad970 0x7f38500b3af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:01.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:01.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500b2820 con 0x7f38500cda80 2023-10-14T17:01:01.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38500b3660 con 0x7f385006c040 2023-10-14T17:01:01.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38561ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38500b2a50 con 0x7f38500ad970 2023-10-14T17:01:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38559fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38559fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39218/0 (socket says 172.21.15.29:39218) 2023-10-14T17:01:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38559fe640 1 -- 172.21.15.29:0/868946437 learned_addr learned my addr 172.21.15.29:0/868946437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.785+0000 7f38551fd640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f385006c040 0x7f38500b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:01.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38549fc640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38500ad970 0x7f38500b3af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:01.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 -- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38500ad970 msgr2=0x7f38500b3af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38500ad970 0x7f38500b3af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 -- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f385006c040 msgr2=0x7f38500b0850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:01.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f385006c040 0x7f38500b0850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:01.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 -- 172.21.15.29:0/868946437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f384c009400 con 0x7f38500cda80 2023-10-14T17:01:01.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38559fe640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f384800c7e0 tx=0x7f384800cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:01.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800a640 con 0x7f38500cda80 2023-10-14T17:01:01.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f384800a7c0 con 0x7f38500cda80 2023-10-14T17:01:01.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38500b2d30 con 0x7f38500cda80 2023-10-14T17:01:01.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.786+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800ab70 con 0x7f38500cda80 2023-10-14T17:01:01.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.787+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38500b32f0 con 0x7f38500cda80 2023-10-14T17:01:01.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.788+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385006b330 con 0x7f38500cda80 2023-10-14T17:01:01.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.789+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f384800acd0 con 0x7f38500cda80 2023-10-14T17:01:01.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.790+0000 7f3845ffb640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 0x7f383406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:01.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.791+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 6160+0+0 (secure 0 0 0) 0x7f3848089180 con 0x7f38500cda80 2023-10-14T17:01:01.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.791+0000 7f38551fd640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 0x7f383406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:01.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.792+0000 7f38551fd640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 0x7f383406d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f384c036000 tx=0x7f384c03c040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:01.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.795+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3848053f70 con 0x7f38500cda80 2023-10-14T17:01:01.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:01.991+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f385004e9b0 con 0x7f38500cda80 2023-10-14T17:01:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.534+0000 7f3845ffb640 1 -- 172.21.15.29:0/868946437 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v145) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3848057590 con 0x7f38500cda80 2023-10-14T17:01:02.536 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T17:01:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 msgr2=0x7f383406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 0x7f383406d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f384c036000 tx=0x7f384c03c040 comp rx=0 tx=0).stop 2023-10-14T17:01:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 msgr2=0x7f38500b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 secure :-1 s=READY pgs=1735 cs=0 l=1 rev1=1 crypto rx=0x7f384800c7e0 tx=0x7f384800cca0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 shutdown_connections 2023-10-14T17:01:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f383406afc0 0x7f383406d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38500cda80 0x7f38500b22e0 unknown :-1 s=CLOSED pgs=1735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f38500ad970 0x7f38500b3af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 --2- 172.21.15.29:0/868946437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f385006c040 0x7f38500b0850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.538+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 >> 172.21.15.29:0/868946437 conn(0x7f38500ce6a0 msgr2=0x7f385006f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.539+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 shutdown_connections 2023-10-14T17:01:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.539+0000 7f38561ff640 1 -- 172.21.15.29:0/868946437 wait complete. 2023-10-14T17:01:02.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T17:01:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.779+0000 7f1f25df4640 1 Processor -- start 2023-10-14T17:01:02.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.779+0000 7f1f25df4640 1 -- start start 2023-10-14T17:01:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f25df4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1f200c8cd0 0x7f1f200c9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f25df4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200c9660 con 0x7f1f200cd260 2023-10-14T17:01:02.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f25df4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200ce480 con 0x7f1f2015cc00 2023-10-14T17:01:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f25df4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f200cd7b0 con 0x7f1f200c8cd0 2023-10-14T17:01:02.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f17fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1f200cd260 0x7f1f200ad130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38580/0 (socket says 172.21.15.29:38580) 2023-10-14T17:01:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f17fff640 1 -- 172.21.15.29:0/3899239310 learned_addr learned my addr 172.21.15.29:0/3899239310 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:02.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.781+0000 7f1f255f3640 1 --2- 172.21.15.29:0/3899239310 >> v2:172.21.15.29:3300/0 conn(0x7f1f200c8cd0 0x7f1f200c9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:02.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.782+0000 7f1f177fe640 1 -- 172.21.15.29:0/3899239310 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2826819342 0 0) 0x7f1f10001110 con 0x7f1f200cd260 2023-10-14T17:01:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.782+0000 7f1f177fe640 1 -- 172.21.15.29:0/3899239310 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f08003520 con 0x7f1f200cd260 2023-10-14T17:01:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.782+0000 7f1f255f3640 1 -- 172.21.15.29:0/3899239310 >> v1:172.21.15.186:6789/0 conn(0x7f1f2015cc00 legacy=0x7f1f200ca410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.782+0000 7f1f255f3640 1 -- 172.21.15.29:0/3899239310 >> v1:172.21.15.29:6789/0 conn(0x7f1f200cd260 legacy=0x7f1f200ad130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.782+0000 7f1f255f3640 1 -- 172.21.15.29:0/3899239310 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200cda90 con 0x7f1f200c8cd0 2023-10-14T17:01:02.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.783+0000 7f1f255f3640 1 --2- 172.21.15.29:0/3899239310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200c8cd0 0x7f1f200c9120 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009dc0 tx=0x7f1f1802bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f2c47d1ebc7f16 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.783+0000 7f1f177fe640 1 -- 172.21.15.29:0/3899239310 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1802ccd0 con 0x7f1f200c8cd0 2023-10-14T17:01:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.783+0000 7f1f177fe640 1 -- 172.21.15.29:0/3899239310 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1802ce30 con 0x7f1f200c8cd0 2023-10-14T17:01:02.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f177fe640 1 -- 172.21.15.29:0/3899239310 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f18034660 con 0x7f1f200c8cd0 2023-10-14T17:01:02.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 -- 172.21.15.29:0/3899239310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200c8cd0 msgr2=0x7f1f200c9120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 --2- 172.21.15.29:0/3899239310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200c8cd0 0x7f1f200c9120 secure :-1 s=READY pgs=1737 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009dc0 tx=0x7f1f1802bf20 comp rx=0 tx=0).stop 2023-10-14T17:01:02.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 -- 172.21.15.29:0/3899239310 shutdown_connections 2023-10-14T17:01:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 --2- 172.21.15.29:0/3899239310 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200c8cd0 0x7f1f200c9120 unknown :-1 s=CLOSED pgs=1737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 -- 172.21.15.29:0/3899239310 >> 172.21.15.29:0/3899239310 conn(0x7f1f200b0170 msgr2=0x7f1f200b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:02.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 -- 172.21.15.29:0/3899239310 shutdown_connections 2023-10-14T17:01:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.784+0000 7f1f25df4640 1 -- 172.21.15.29:0/3899239310 wait complete. 2023-10-14T17:01:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.786+0000 7f1f25df4640 1 Processor -- start 2023-10-14T17:01:02.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.786+0000 7f1f25df4640 1 -- start start 2023-10-14T17:01:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f200c8cd0 0x7f1f200b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f2015cc00 0x7f1f200b3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:02.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f201455f0 con 0x7f1f200cd260 2023-10-14T17:01:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200c4b40 con 0x7f1f200c8cd0 2023-10-14T17:01:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f25df4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20158ee0 con 0x7f1f2015cc00 2023-10-14T17:01:02.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39236/0 (socket says 172.21.15.29:39236) 2023-10-14T17:01:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.787+0000 7f1f17fff640 1 -- 172.21.15.29:0/1748071484 learned_addr learned my addr 172.21.15.29:0/1748071484 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:02.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f24df2640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f200c8cd0 0x7f1f200b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f255f3640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f2015cc00 0x7f1f200b3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 -- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f2015cc00 msgr2=0x7f1f200b3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f2015cc00 0x7f1f200b3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 -- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f200c8cd0 msgr2=0x7f1f200b3040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:02.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f200c8cd0 0x7f1f200b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 -- 172.21.15.29:0/1748071484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f18009a70 con 0x7f1f200cd260 2023-10-14T17:01:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f17fff640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c700 tx=0x7f1f1000cb00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10005ce0 con 0x7f1f200cd260 2023-10-14T17:01:02.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10005e40 con 0x7f1f200cd260 2023-10-14T17:01:02.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10007290 con 0x7f1f200cd260 2023-10-14T17:01:02.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.788+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2015f590 con 0x7f1f200cd260 2023-10-14T17:01:02.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.789+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f200b2840 con 0x7f1f200cd260 2023-10-14T17:01:02.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.790+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f2004e9b0 con 0x7f1f200cd260 2023-10-14T17:01:02.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.791+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1f100074b0 con 0x7f1f200cd260 2023-10-14T17:01:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.792+0000 7f1f157fa640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 0x7f1f0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.792+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f1f10088c80 con 0x7f1f200cd260 2023-10-14T17:01:02.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.792+0000 7f1f24df2640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 0x7f1f0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:02.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.793+0000 7f1f24df2640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 0x7f1f0c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c000f60 tx=0x7f1f1c009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:02.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.797+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1f100575e0 con 0x7f1f200cd260 2023-10-14T17:01:02.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:02.995+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f1f200c9660 con 0x7f1f200cd260 2023-10-14T17:01:03.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.002+0000 7f1f157fa640 1 -- 172.21.15.29:0/1748071484 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v49) v1 ==== 115+0+0 (secure 0 0 0) 0x7f1f200c9660 con 0x7f1f200cd260 2023-10-14T17:01:03.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.006+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 msgr2=0x7f1f0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.006+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 0x7f1f0c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f1f1c000f60 tx=0x7f1f1c009040 comp rx=0 tx=0).stop 2023-10-14T17:01:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.006+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 msgr2=0x7f1f200b3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.006+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 secure :-1 s=READY pgs=1738 cs=0 l=1 rev1=1 crypto rx=0x7f1f1000c700 tx=0x7f1f1000cb00 comp rx=0 tx=0).stop 2023-10-14T17:01:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 shutdown_connections 2023-10-14T17:01:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1f0c06af90 0x7f1f0c06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1f2015cc00 0x7f1f200b3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1f200cd260 0x7f1f200b3580 unknown :-1 s=CLOSED pgs=1738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 --2- 172.21.15.29:0/1748071484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1f200c8cd0 0x7f1f200b3040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 >> 172.21.15.29:0/1748071484 conn(0x7f1f200b0170 msgr2=0x7f1f200af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:03.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 shutdown_connections 2023-10-14T17:01:03.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.007+0000 7f1f25df4640 1 -- 172.21.15.29:0/1748071484 wait complete. 2023-10-14T17:01:03.027 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T17:01:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.256+0000 7fa4c8aa2640 1 Processor -- start 2023-10-14T17:01:03.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.256+0000 7fa4c8aa2640 1 -- start start 2023-10-14T17:01:03.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.257+0000 7fa4c8aa2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4c40ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.257+0000 7fa4c8aa2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa4c406c140 0x7fa4c406c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.257+0000 7fa4c8aa2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4c4070360 con 0x7fa4c4159ab0 2023-10-14T17:01:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.257+0000 7fa4c8aa2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4c406cb10 con 0x7fa4c406c140 2023-10-14T17:01:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.257+0000 7fa4c8aa2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415d890 con 0x7fa4c4159e90 2023-10-14T17:01:03.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa4c4159ab0 0x7fa4c406fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38594/0 (socket says 172.21.15.29:38594) 2023-10-14T17:01:03.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c259c640 1 -- 172.21.15.29:0/3529542024 learned_addr learned my addr 172.21.15.29:0/3529542024 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:03.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2144056252 0 0) 0x7fa4c4070360 con 0x7fa4c4159ab0 2023-10-14T17:01:03.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4b0003520 con 0x7fa4c4159ab0 2023-10-14T17:01:03.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4c40ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c2d9d640 1 --2- 172.21.15.29:0/3529542024 >> v2:172.21.15.186:3300/0 conn(0x7fa4c406c140 0x7fa4c406c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.258+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3053843902 0 0) 0x7fa4b0003520 con 0x7fa4c4159ab0 2023-10-14T17:01:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 msgr2=0x7fa4c40ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4c40ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 >> v2:172.21.15.186:3300/0 conn(0x7fa4c406c140 msgr2=0x7fa4c406c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 --2- 172.21.15.29:0/3529542024 >> v2:172.21.15.186:3300/0 conn(0x7fa4c406c140 0x7fa4c406c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c40cd1f0 con 0x7fa4c4159ab0 2023-10-14T17:01:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa4b4001670 con 0x7fa4c4159ab0 2023-10-14T17:01:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 >> v1:172.21.15.29:6789/0 conn(0x7fa4c4159ab0 legacy=0x7fa4c406fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4b0003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0003fa0 0x7fa4b00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4b0004d00 0x7fa4b0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c406cb10 con 0x7fa4b0003fa0 2023-10-14T17:01:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415d890 con 0x7fa4b0004d00 2023-10-14T17:01:03.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4b000bbf0 con 0x7fa4c4159e90 2023-10-14T17:01:03.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0003fa0 0x7fa4b00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c259c640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4b0003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4c2d9d640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4b0004d00 0x7fa4b0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 -- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 msgr2=0x7fa4b0003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:03.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4c4159e90 0x7fa4b0003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 -- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4b0004d00 msgr2=0x7fa4b0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4b0004d00 0x7fa4b0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.259+0000 7fa4bbfff640 1 -- 172.21.15.29:0/3529542024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4b0003520 con 0x7fa4b0003fa0 2023-10-14T17:01:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0003fa0 0x7fa4b00047c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00ba30 tx=0x7fa4b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3da06660e4fd7c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac032640 con 0x7fa4b0003fa0 2023-10-14T17:01:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac0327a0 con 0x7fa4b0003fa0 2023-10-14T17:01:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4c1d9b640 1 -- 172.21.15.29:0/3529542024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac032b30 con 0x7fa4b0003fa0 2023-10-14T17:01:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/3529542024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0003fa0 msgr2=0x7fa4b00047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.261+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/3529542024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4b0003fa0 0x7fa4b00047c0 secure :-1 s=READY pgs=1740 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac00ba30 tx=0x7fa4b00057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.262+0000 7fa4c2d9d640 1 -- 172.21.15.29:0/3529542024 reap_dead start 2023-10-14T17:01:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.262+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/3529542024 shutdown_connections 2023-10-14T17:01:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.262+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/3529542024 >> 172.21.15.29:0/3529542024 conn(0x7fa4c40ce9b0 msgr2=0x7fa4c40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.262+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/3529542024 shutdown_connections 2023-10-14T17:01:03.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.262+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/3529542024 wait complete. 2023-10-14T17:01:03.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.263+0000 7fa4c8aa2640 1 Processor -- start 2023-10-14T17:01:03.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.264+0000 7fa4c8aa2640 1 -- start start 2023-10-14T17:01:03.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.264+0000 7fa4c8aa2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4b0003fa0 0x7fa4c4154f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c8aa2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c8aa2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4c40c0e30 0x7fa4c415a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c8aa2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c40c12b0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c8aa2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415b020 con 0x7fa4c40c0e30 2023-10-14T17:01:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c8aa2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4c415b300 con 0x7fa4b0003fa0 2023-10-14T17:01:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39260/0 (socket says 172.21.15.29:39260) 2023-10-14T17:01:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 -- 172.21.15.29:0/4152741937 learned_addr learned my addr 172.21.15.29:0/4152741937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c259c640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4b0003fa0 0x7fa4c4154f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4c2d9d640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4c40c0e30 0x7fa4c415a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 -- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4b0003fa0 msgr2=0x7fa4c4154f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4b0003fa0 0x7fa4c4154f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 -- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4c40c0e30 msgr2=0x7fa4c415a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4c40c0e30 0x7fa4c415a330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.265+0000 7fa4bbfff640 1 -- 172.21.15.29:0/4152741937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4c415b5e0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4bbfff640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac02bec0 tx=0x7fa4ac004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac004220 con 0x7fa4c415c0c0 2023-10-14T17:01:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4ac004380 con 0x7fa4c415c0c0 2023-10-14T17:01:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c415b8c0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4ac03caa0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.266+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4c41554c0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.267+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4c404e160 con 0x7fa4c415c0c0 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.268+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa4ac03cc00 con 0x7fa4c415c0c0 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.269+0000 7fa4baffd640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 0x7fa4a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.269+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fa4ac0ac270 con 0x7fa4c415c0c0 2023-10-14T17:01:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.269+0000 7fa4c259c640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 0x7fa4a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.270+0000 7fa4c259c640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 0x7fa4a406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4001eb0 tx=0x7fa4b4009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.274+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4ac076fa0 con 0x7fa4c415c0c0 2023-10-14T17:01:03.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.473+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa4c404e160 con 0x7fa4c415c0c0 2023-10-14T17:01:03.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.480+0000 7fa4baffd640 1 -- 172.21.15.29:0/4152741937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v50) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa4ac006c30 con 0x7fa4c415c0c0 2023-10-14T17:01:03.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 msgr2=0x7fa4a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 0x7fa4a406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa4b4001eb0 tx=0x7fa4b4009040 comp rx=0 tx=0).stop 2023-10-14T17:01:03.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 msgr2=0x7fa4c40c08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 secure :-1 s=READY pgs=1741 cs=0 l=1 rev1=1 crypto rx=0x7fa4ac02bec0 tx=0x7fa4ac004730 comp rx=0 tx=0).stop 2023-10-14T17:01:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 shutdown_connections 2023-10-14T17:01:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa4a406af90 0x7fa4a406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa4c40c0e30 0x7fa4c415a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4c415c0c0 0x7fa4c40c08f0 unknown :-1 s=CLOSED pgs=1741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 --2- 172.21.15.29:0/4152741937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa4b0003fa0 0x7fa4c4154f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.483+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 >> 172.21.15.29:0/4152741937 conn(0x7fa4c40ce9b0 msgr2=0x7fa4c406a630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.484+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 shutdown_connections 2023-10-14T17:01:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.484+0000 7fa4c8aa2640 1 -- 172.21.15.29:0/4152741937 wait complete. 2023-10-14T17:01:03.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T17:01:03.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.738+0000 7fa6ca085640 1 Processor -- start 2023-10-14T17:01:03.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.738+0000 7fa6ca085640 1 -- start start 2023-10-14T17:01:03.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.739+0000 7fa6ca085640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6c415cd70 0x7fa6c40d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6ca085640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6ca085640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6c415d150 con 0x7fa6c40b8480 2023-10-14T17:01:03.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6ca085640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c40d1180 con 0x7fa6c415cd70 2023-10-14T17:01:03.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6ca085640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c40d1460 con 0x7fa6c40cded0 2023-10-14T17:01:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6c9083640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa6c415cd70 0x7fa6c40d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6c9884640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.740+0000 7fa6c9884640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:58328/0 (socket says 172.21.15.29:58328) 2023-10-14T17:01:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.741+0000 7fa6c9884640 1 -- 172.21.15.29:0/4077949975 learned_addr learned my addr 172.21.15.29:0/4077949975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.741+0000 7fa6c9884640 1 -- 172.21.15.29:0/4077949975 >> v2:172.21.15.186:3300/0 conn(0x7fa6c415cd70 msgr2=0x7fa6c40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.741+0000 7fa6c9884640 1 --2- 172.21.15.29:0/4077949975 >> v2:172.21.15.186:3300/0 conn(0x7fa6c415cd70 0x7fa6c40d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.741+0000 7fa6c9884640 1 -- 172.21.15.29:0/4077949975 >> v1:172.21.15.186:6789/0 conn(0x7fa6c40b8480 legacy=0x7fa6c40cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.741+0000 7fa6c9884640 1 -- 172.21.15.29:0/4077949975 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c40d1740 con 0x7fa6c40cded0 2023-10-14T17:01:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.742+0000 7fa6c9884640 1 --2- 172.21.15.29:0/4077949975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009e50 tx=0x7fa6bc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed82ab958596df7a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.743+0000 7fa6bbfff640 1 -- 172.21.15.29:0/4077949975 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc02cb00 con 0x7fa6c40cded0 2023-10-14T17:01:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.743+0000 7fa6bbfff640 1 -- 172.21.15.29:0/4077949975 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc02cc60 con 0x7fa6c40cded0 2023-10-14T17:01:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.743+0000 7fa6bbfff640 1 -- 172.21.15.29:0/4077949975 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6bc0343f0 con 0x7fa6c40cded0 2023-10-14T17:01:03.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 -- 172.21.15.29:0/4077949975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 msgr2=0x7fa6c40b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 --2- 172.21.15.29:0/4077949975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa6bc009e50 tx=0x7fa6bc02bf60 comp rx=0 tx=0).stop 2023-10-14T17:01:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 -- 172.21.15.29:0/4077949975 shutdown_connections 2023-10-14T17:01:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 --2- 172.21.15.29:0/4077949975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c40cded0 0x7fa6c40b5c90 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 --2- 172.21.15.29:0/4077949975 >> v2:172.21.15.186:3300/0 conn(0x7fa6c415cd70 0x7fa6c40d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 -- 172.21.15.29:0/4077949975 >> 172.21.15.29:0/4077949975 conn(0x7fa6c40cf570 msgr2=0x7fa6c40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 -- 172.21.15.29:0/4077949975 shutdown_connections 2023-10-14T17:01:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.744+0000 7fa6ca085640 1 -- 172.21.15.29:0/4077949975 wait complete. 2023-10-14T17:01:03.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.746+0000 7fa6ca085640 1 Processor -- start 2023-10-14T17:01:03.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.747+0000 7fa6ca085640 1 -- start start 2023-10-14T17:01:03.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c40b8480 0x7fa6c415edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c415cd70 0x7fa6c40b0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c41f66d0 con 0x7fa6c40cded0 2023-10-14T17:01:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c414db10 con 0x7fa6c40b8480 2023-10-14T17:01:03.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6ca085640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c41f5b50 con 0x7fa6c415cd70 2023-10-14T17:01:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6c8882640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6c8882640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39270/0 (socket says 172.21.15.29:39270) 2023-10-14T17:01:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6c8882640 1 -- 172.21.15.29:0/3196071415 learned_addr learned my addr 172.21.15.29:0/3196071415 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.748+0000 7fa6c9884640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c415cd70 0x7fa6c40b0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c9083640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c40b8480 0x7fa6c415edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 -- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c415cd70 msgr2=0x7fa6c40b0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c415cd70 0x7fa6c40b0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 -- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c40b8480 msgr2=0x7fa6c415edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c40b8480 0x7fa6c415edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 -- 172.21.15.29:0/3196071415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6bc009b30 con 0x7fa6c40cded0 2023-10-14T17:01:03.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6c8882640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c390 tx=0x7fa6b400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b40052d0 con 0x7fa6c40cded0 2023-10-14T17:01:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6b4005430 con 0x7fa6c40cded0 2023-10-14T17:01:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.749+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6b4003830 con 0x7fa6c40cded0 2023-10-14T17:01:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.750+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c41f5e30 con 0x7fa6c40cded0 2023-10-14T17:01:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.750+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c41f63f0 con 0x7fa6c40cded0 2023-10-14T17:01:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.751+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c404e9b0 con 0x7fa6c40cded0 2023-10-14T17:01:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.752+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa6b4003990 con 0x7fa6c40cded0 2023-10-14T17:01:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.753+0000 7fa6b9ffb640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 0x7fa6a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.753+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fa6b4088df0 con 0x7fa6c40cded0 2023-10-14T17:01:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.753+0000 7fa6c9083640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 0x7fa6a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:03.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.754+0000 7fa6c9083640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 0x7fa6a806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009ad0 tx=0x7fa6c0008040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:03.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.758+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa6b4053cc0 con 0x7fa6c40cded0 2023-10-14T17:01:03.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.954+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa6c415c3b0 con 0x7fa6c40cded0 2023-10-14T17:01:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.961+0000 7fa6b9ffb640 1 -- 172.21.15.29:0/3196071415 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v51) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa6b40572e0 con 0x7fa6c40cded0 2023-10-14T17:01:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 msgr2=0x7fa6a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 0x7fa6a806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa6c0009ad0 tx=0x7fa6c0008040 comp rx=0 tx=0).stop 2023-10-14T17:01:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 msgr2=0x7fa6c414d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 secure :-1 s=READY pgs=1742 cs=0 l=1 rev1=1 crypto rx=0x7fa6b400c390 tx=0x7fa6b400c850 comp rx=0 tx=0).stop 2023-10-14T17:01:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 shutdown_connections 2023-10-14T17:01:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa6a806af90 0x7fa6a806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6c415cd70 0x7fa6c40b0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa6c40cded0 0x7fa6c414d5d0 unknown :-1 s=CLOSED pgs=1742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 --2- 172.21.15.29:0/3196071415 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6c40b8480 0x7fa6c415edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:03.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.964+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 >> 172.21.15.29:0/3196071415 conn(0x7fa6c40cf570 msgr2=0x7fa6c40b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:03.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.965+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 shutdown_connections 2023-10-14T17:01:03.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:03.965+0000 7fa6ca085640 1 -- 172.21.15.29:0/3196071415 wait complete. 2023-10-14T17:01:03.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods' 2023-10-14T17:01:04.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.206+0000 7fe312d85640 1 Processor -- start 2023-10-14T17:01:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.206+0000 7fe312d85640 1 -- start start 2023-10-14T17:01:04.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe30c15aad0 0x7fe30c0cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe30c070d70 0x7fe30c06bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c0712c0 0x7fe30c06c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30c06c9c0 con 0x7fe30c070d70 2023-10-14T17:01:04.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe30c0b4620 con 0x7fe30c15aad0 2023-10-14T17:01:04.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe312d85640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c0b4340 con 0x7fe30c0712c0 2023-10-14T17:01:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe311582640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe30c070d70 0x7fe30c06bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.207+0000 7fe311582640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe30c070d70 0x7fe30c06bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39280/0 (socket says 172.21.15.29:39280) 2023-10-14T17:01:04.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 -- 172.21.15.29:0/4002548431 learned_addr learned my addr 172.21.15.29:0/4002548431 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe312584640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c0712c0 0x7fe30c06c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311d83640 1 --2- 172.21.15.29:0/4002548431 >> v2:172.21.15.186:3300/0 conn(0x7fe30c15aad0 0x7fe30c0cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 -- 172.21.15.29:0/4002548431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c0712c0 msgr2=0x7fe30c06c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c0712c0 0x7fe30c06c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 -- 172.21.15.29:0/4002548431 >> v2:172.21.15.186:3300/0 conn(0x7fe30c15aad0 msgr2=0x7fe30c0cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 --2- 172.21.15.29:0/4002548431 >> v2:172.21.15.186:3300/0 conn(0x7fe30c15aad0 0x7fe30c0cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.208+0000 7fe311582640 1 -- 172.21.15.29:0/4002548431 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c06f520 con 0x7fe30c070d70 2023-10-14T17:01:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.209+0000 7fe311582640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c06bf40 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0cf4f0 tx=0x7fe2fc02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4be710252e40ae0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.209+0000 7fe310d81640 1 -- 172.21.15.29:0/4002548431 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc030850 con 0x7fe30c070d70 2023-10-14T17:01:04.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe310d81640 1 -- 172.21.15.29:0/4002548431 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe2fc0309b0 con 0x7fe30c070d70 2023-10-14T17:01:04.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe310d81640 1 -- 172.21.15.29:0/4002548431 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2fc030d40 con 0x7fe30c070d70 2023-10-14T17:01:04.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 -- 172.21.15.29:0/4002548431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 msgr2=0x7fe30c06bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c06bf40 secure :-1 s=READY pgs=1743 cs=0 l=1 rev1=1 crypto rx=0x7fe30c0cf4f0 tx=0x7fe2fc02b990 comp rx=0 tx=0).stop 2023-10-14T17:01:04.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 -- 172.21.15.29:0/4002548431 shutdown_connections 2023-10-14T17:01:04.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c0712c0 0x7fe30c06c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 --2- 172.21.15.29:0/4002548431 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c06bf40 unknown :-1 s=CLOSED pgs=1743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 --2- 172.21.15.29:0/4002548431 >> v2:172.21.15.186:3300/0 conn(0x7fe30c15aad0 0x7fe30c0cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 -- 172.21.15.29:0/4002548431 >> 172.21.15.29:0/4002548431 conn(0x7fe30c0b0a90 msgr2=0x7fe30c0ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 -- 172.21.15.29:0/4002548431 shutdown_connections 2023-10-14T17:01:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.210+0000 7fe312d85640 1 -- 172.21.15.29:0/4002548431 wait complete. 2023-10-14T17:01:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.212+0000 7fe312d85640 1 Processor -- start 2023-10-14T17:01:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.212+0000 7fe312d85640 1 -- start start 2023-10-14T17:01:04.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 0x7fe30c0b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 0x7fe30c0c84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c15b1c0 con 0x7fe30c070d70 2023-10-14T17:01:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c0b39a0 con 0x7fe30c0712c0 2023-10-14T17:01:04.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.213+0000 7fe312d85640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe30c15aeb0 con 0x7fe30c15aad0 2023-10-14T17:01:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39292/0 (socket says 172.21.15.29:39292) 2023-10-14T17:01:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 -- 172.21.15.29:0/3211177786 learned_addr learned my addr 172.21.15.29:0/3211177786 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311582640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 0x7fe30c0b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 -- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 msgr2=0x7fe30c0c84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe312584640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 0x7fe30c0c84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 0x7fe30c0c84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 -- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 msgr2=0x7fe30c0b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 0x7fe30c0b3460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 -- 172.21.15.29:0/3211177786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc0094d0 con 0x7fe30c070d70 2023-10-14T17:01:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311582640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 0x7fe30c0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe312584640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 0x7fe30c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.214+0000 7fe311d83640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d5f0 tx=0x7fe30800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.215+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3080125f0 con 0x7fe30c070d70 2023-10-14T17:01:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.215+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe308012750 con 0x7fe30c070d70 2023-10-14T17:01:04.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.215+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe30c0c8a20 con 0x7fe30c070d70 2023-10-14T17:01:04.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.215+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe308012b00 con 0x7fe30c070d70 2023-10-14T17:01:04.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.215+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe30c15d6b0 con 0x7fe30c070d70 2023-10-14T17:01:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.216+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe30c15b430 con 0x7fe30c070d70 2023-10-14T17:01:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.217+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe308012c60 con 0x7fe30c070d70 2023-10-14T17:01:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.219+0000 7fe3027fc640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.219+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fe308089cc0 con 0x7fe30c070d70 2023-10-14T17:01:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.219+0000 7fe311582640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.220+0000 7fe311582640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 0x7fe2f006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc036000 tx=0x7fe2fc03b040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.223+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe308090080 con 0x7fe30c070d70 2023-10-14T17:01:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.491+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]} v 0) v1 -- 0x7fe30c04e9b0 con 0x7fe30c070d70 2023-10-14T17:01:04.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.547+0000 7fe3027fc640 1 -- 172.21.15.29:0/3211177786 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_all_periods"]}]=0 v807) v1 ==== 160+0+0 (secure 0 0 0) 0x7fe308054b90 con 0x7fe30c070d70 2023-10-14T17:01:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 msgr2=0x7fe2f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 0x7fe2f006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe2fc036000 tx=0x7fe2fc03b040 comp rx=0 tx=0).stop 2023-10-14T17:01:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 msgr2=0x7fe30c15d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 secure :-1 s=READY pgs=1744 cs=0 l=1 rev1=1 crypto rx=0x7fe30800d5f0 tx=0x7fe30800dab0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 shutdown_connections 2023-10-14T17:01:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe2f006af90 0x7fe2f006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe30c15aad0 0x7fe30c0c84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe30c0712c0 0x7fe30c0b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 --2- 172.21.15.29:0/3211177786 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe30c070d70 0x7fe30c15d0d0 unknown :-1 s=CLOSED pgs=1744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 >> 172.21.15.29:0/3211177786 conn(0x7fe30c0b0a90 msgr2=0x7fe30c0ae1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:04.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 shutdown_connections 2023-10-14T17:01:04.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.551+0000 7fe312d85640 1 -- 172.21.15.29:0/3211177786 wait complete. 2023-10-14T17:01:04.571 INFO:tasks.cephfs_test_runner:test_snap_schedule_all_periods (tasks.cephfs.test_snap_schedules.TestSnapSchedules) ... ok 2023-10-14T17:01:04.572 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that number of snaps retained are as per user spec. 2023-10-14T17:01:04.572 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_number_of_snaps_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T17:01:04.573 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:01:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.789+0000 7fef6e68a640 1 Processor -- start 2023-10-14T17:01:04.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.789+0000 7fef6e68a640 1 -- start start 2023-10-14T17:01:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6e68a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef6815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6e68a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fef6806c810 0x7fef680708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6e68a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef68070df0 con 0x7fef68070450 2023-10-14T17:01:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6e68a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef6815e490 con 0x7fef6806c810 2023-10-14T17:01:04.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6e68a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815aef0 con 0x7fef68159950 2023-10-14T17:01:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6ce87640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fef68070450 0x7fef6806c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38606/0 (socket says 172.21.15.29:38606) 2023-10-14T17:01:04.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.791+0000 7fef6ce87640 1 -- 172.21.15.29:0/3662160487 learned_addr learned my addr 172.21.15.29:0/3662160487 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef6de89640 1 --2- 172.21.15.29:0/3662160487 >> v2:172.21.15.186:3300/0 conn(0x7fef6806c810 0x7fef680708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef6d688640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef6815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3804474844 0 0) 0x7fef68070df0 con 0x7fef68070450 2023-10-14T17:01:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef54003520 con 0x7fef68070450 2023-10-14T17:01:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171217147 0 0) 0x7fef54003520 con 0x7fef68070450 2023-10-14T17:01:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 msgr2=0x7fef6815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef6815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 >> v2:172.21.15.186:3300/0 conn(0x7fef6806c810 msgr2=0x7fef680708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 --2- 172.21.15.29:0/3662160487 >> v2:172.21.15.186:3300/0 conn(0x7fef6806c810 0x7fef680708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef6806cc80 con 0x7fef68070450 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fef58001670 con 0x7fef68070450 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.792+0000 7fef6de89640 1 --2- 172.21.15.29:0/3662160487 >> v2:172.21.15.186:3300/0 conn(0x7fef6806c810 0x7fef680708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 >> v1:172.21.15.29:6789/0 conn(0x7fef68070450 legacy=0x7fef6806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef54003970 0x7fef54003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef54004d00 0x7fef54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815e490 con 0x7fef54003970 2023-10-14T17:01:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815aef0 con 0x7fef54004d00 2023-10-14T17:01:04.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef5400ba60 con 0x7fef68159950 2023-10-14T17:01:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef6d688640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef54003970 0x7fef54003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef6ce87640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.793+0000 7fef6de89640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef54004d00 0x7fef54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.794+0000 7fef6d688640 1 -- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 msgr2=0x7fef540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.794+0000 7fef6d688640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef68159950 0x7fef540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.794+0000 7fef6d688640 1 -- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef54004d00 msgr2=0x7fef54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.794+0000 7fef6d688640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef54004d00 0x7fef54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.794+0000 7fef6d688640 1 -- 172.21.15.29:0/3662160487 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef54003520 con 0x7fef54003970 2023-10-14T17:01:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.795+0000 7fef6d688640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef54003970 0x7fef54003d90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fef64009ac0 tx=0x7fef540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d0c6a68a00c97ee server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.795+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6402c6b0 con 0x7fef54003970 2023-10-14T17:01:04.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef6402c810 con 0x7fef54003970 2023-10-14T17:01:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef5ffff640 1 -- 172.21.15.29:0/3662160487 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef6402cba0 con 0x7fef54003970 2023-10-14T17:01:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 -- 172.21.15.29:0/3662160487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef54003970 msgr2=0x7fef54003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 --2- 172.21.15.29:0/3662160487 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef54003970 0x7fef54003d90 secure :-1 s=READY pgs=1746 cs=0 l=1 rev1=1 crypto rx=0x7fef64009ac0 tx=0x7fef540057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6de89640 1 -- 172.21.15.29:0/3662160487 reap_dead start 2023-10-14T17:01:04.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 -- 172.21.15.29:0/3662160487 shutdown_connections 2023-10-14T17:01:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 -- 172.21.15.29:0/3662160487 >> 172.21.15.29:0/3662160487 conn(0x7fef680ce6a0 msgr2=0x7fef680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 -- 172.21.15.29:0/3662160487 shutdown_connections 2023-10-14T17:01:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.796+0000 7fef6e68a640 1 -- 172.21.15.29:0/3662160487 wait complete. 2023-10-14T17:01:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.798+0000 7fef6e68a640 1 Processor -- start 2023-10-14T17:01:04.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.798+0000 7fef6e68a640 1 -- start start 2023-10-14T17:01:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef54003970 0x7fef680c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef681f2030 0x7fef681f24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c7310 con 0x7fef6815cfd0 2023-10-14T17:01:04.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef6815d4d0 con 0x7fef681f2030 2023-10-14T17:01:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.799+0000 7fef6e68a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef680c7560 con 0x7fef54003970 2023-10-14T17:01:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39300/0 (socket says 172.21.15.29:39300) 2023-10-14T17:01:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 -- 172.21.15.29:0/950774394 learned_addr learned my addr 172.21.15.29:0/950774394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:04.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6d688640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef54003970 0x7fef680c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6de89640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef681f2030 0x7fef681f24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 -- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef54003970 msgr2=0x7fef680c4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef54003970 0x7fef680c4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 -- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef681f2030 msgr2=0x7fef681f24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef681f2030 0x7fef681f24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:04.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.800+0000 7fef6ce87640 1 -- 172.21.15.29:0/950774394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef680c7840 con 0x7fef6815cfd0 2023-10-14T17:01:04.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef6ce87640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fef58001050 tx=0x7fef5800ec30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800ee80 con 0x7fef6815cfd0 2023-10-14T17:01:04.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef5800c8d0 con 0x7fef6815cfd0 2023-10-14T17:01:04.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef5800cc60 con 0x7fef6815cfd0 2023-10-14T17:01:04.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef680c7b20 con 0x7fef6815cfd0 2023-10-14T17:01:04.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.801+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef68151f60 con 0x7fef6815cfd0 2023-10-14T17:01:04.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.802+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef6804e9b0 con 0x7fef6815cfd0 2023-10-14T17:01:04.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.803+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fef5801d460 con 0x7fef6815cfd0 2023-10-14T17:01:04.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.804+0000 7fef5dffb640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 0x7fef4c06d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:04.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.804+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fef58089710 con 0x7fef6815cfd0 2023-10-14T17:01:04.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.804+0000 7fef6d688640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 0x7fef4c06d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:04.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.805+0000 7fef6d688640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 0x7fef4c06d4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef64036000 tx=0x7fef64030000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:04.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:04.809+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fef580546f0 con 0x7fef6815cfd0 2023-10-14T17:01:05.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.006+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef68156140 con 0x7fef6815cfd0 2023-10-14T17:01:05.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.007+0000 7fef5dffb640 1 -- 172.21.15.29:0/950774394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+14159 (secure 0 0 0) 0x7fef58001610 con 0x7fef6815cfd0 2023-10-14T17:01:05.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:05.009 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":145,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:01:02.526045+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6836/1532396905":"2023-10-15T17:00:59.502290+0000","172.21.15.29:6834/1532396905":"2023-10-15T17:00:59.502290+0000","172.21.15.29:0/960844541":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/502397156":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/2049846967":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/4089732493":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3980098395":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6832/795882907":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/3374539339":"2023-10-15T17:00:55.748629+0000","172.21.15.29:0/1488377546":"2023-10-15T17:00:42.236245+0000","172.21.15.29:6832/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/532161644":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3029691751":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/932642367":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/3424087599":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/2982227449":"2023-10-15T17:00:55.748629+0000","172.21.15.29:6833/3738876524":"2023-10-15T17:00:42.236245+0000","172.21.15.29:0/4142001408":"2023-10-15T17:00:42.236245+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:01:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 msgr2=0x7fef4c06d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 0x7fef4c06d4c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fef64036000 tx=0x7fef64030000 comp rx=0 tx=0).stop 2023-10-14T17:01:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 msgr2=0x7fef681f1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 secure :-1 s=READY pgs=1747 cs=0 l=1 rev1=1 crypto rx=0x7fef58001050 tx=0x7fef5800ec30 comp rx=0 tx=0).stop 2023-10-14T17:01:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 shutdown_connections 2023-10-14T17:01:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fef4c06b000 0x7fef4c06d4c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fef681f2030 0x7fef681f24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fef6815cfd0 0x7fef681f1af0 unknown :-1 s=CLOSED pgs=1747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 --2- 172.21.15.29:0/950774394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fef54003970 0x7fef680c4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 >> 172.21.15.29:0/950774394 conn(0x7fef680ce6a0 msgr2=0x7fef6806bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 shutdown_connections 2023-10-14T17:01:05.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.010+0000 7fef6e68a640 1 -- 172.21.15.29:0/950774394 wait complete. 2023-10-14T17:01:05.031 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:01:05.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:01:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.263+0000 7fec84e83640 1 Processor -- start 2023-10-14T17:01:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.263+0000 7fec84e83640 1 -- start start 2023-10-14T17:01:05.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec84e83640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec8015d150 0x7fec800ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec84e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80160b90 0x7fec80160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec84e83640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec800ac890 con 0x7fec80069da0 2023-10-14T17:01:05.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec84e83640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec8015d530 con 0x7fec8015d150 2023-10-14T17:01:05.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec84e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec80161650 con 0x7fec80160b90 2023-10-14T17:01:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec8015d150 0x7fec800ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec8015d150 0x7fec800ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39306/0 (socket says 172.21.15.29:39306) 2023-10-14T17:01:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.265+0000 7fec7e59c640 1 -- 172.21.15.29:0/3555362476 learned_addr learned my addr 172.21.15.29:0/3555362476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.266+0000 7fec75d9b640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80160b90 0x7fec80160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.266+0000 7fec7e59c640 1 -- 172.21.15.29:0/3555362476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80160b90 msgr2=0x7fec80160fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.266+0000 7fec7e59c640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80160b90 0x7fec80160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.266+0000 7fec7e59c640 1 -- 172.21.15.29:0/3555362476 >> v1:172.21.15.186:6789/0 conn(0x7fec80069da0 legacy=0x7fec8006a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.266+0000 7fec7e59c640 1 -- 172.21.15.29:0/3555362476 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec80161930 con 0x7fec8015d150 2023-10-14T17:01:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.267+0000 7fec7e59c640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800ad350 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a69dfb1c012ca231 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.267+0000 7fec7dd9b640 1 -- 172.21.15.29:0/3555362476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70031640 con 0x7fec8015d150 2023-10-14T17:01:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.267+0000 7fec7dd9b640 1 -- 172.21.15.29:0/3555362476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec700317a0 con 0x7fec8015d150 2023-10-14T17:01:05.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.267+0000 7fec7dd9b640 1 -- 172.21.15.29:0/3555362476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec70031b30 con 0x7fec8015d150 2023-10-14T17:01:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 -- 172.21.15.29:0/3555362476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 msgr2=0x7fec800ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800ad350 secure :-1 s=READY pgs=1748 cs=0 l=1 rev1=1 crypto rx=0x7fec70009870 tx=0x7fec7002b8f0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 -- 172.21.15.29:0/3555362476 shutdown_connections 2023-10-14T17:01:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80160b90 0x7fec80160fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 --2- 172.21.15.29:0/3555362476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800ad350 unknown :-1 s=CLOSED pgs=1748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 -- 172.21.15.29:0/3555362476 >> 172.21.15.29:0/3555362476 conn(0x7fec800ce9b0 msgr2=0x7fec800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 -- 172.21.15.29:0/3555362476 shutdown_connections 2023-10-14T17:01:05.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.268+0000 7fec84e83640 1 -- 172.21.15.29:0/3555362476 wait complete. 2023-10-14T17:01:05.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.270+0000 7fec84e83640 1 Processor -- start 2023-10-14T17:01:05.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.271+0000 7fec84e83640 1 -- start start 2023-10-14T17:01:05.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80069da0 0x7fec801f22c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec80160b90 0x7fec80147680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800c38b0 con 0x7fec8015d150 2023-10-14T17:01:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec800c3b90 con 0x7fec80160b90 2023-10-14T17:01:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec84e83640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec800c3e70 con 0x7fec80069da0 2023-10-14T17:01:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec75d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39308/0 (socket says 172.21.15.29:39308) 2023-10-14T17:01:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec75d9b640 1 -- 172.21.15.29:0/2380326570 learned_addr learned my addr 172.21.15.29:0/2380326570 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.272+0000 7fec7e59c640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80069da0 0x7fec801f22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 -- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80069da0 msgr2=0x7fec801f22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec7ed9d640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec80160b90 0x7fec80147680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80069da0 0x7fec801f22c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 -- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec80160b90 msgr2=0x7fec80147680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec80160b90 0x7fec80147680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 -- 172.21.15.29:0/2380326570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec700094d0 con 0x7fec8015d150 2023-10-14T17:01:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec75d9b640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fec6800ccd0 tx=0x7fec6800a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68003990 con 0x7fec8015d150 2023-10-14T17:01:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec68003af0 con 0x7fec8015d150 2023-10-14T17:01:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec68004020 con 0x7fec8015d150 2023-10-14T17:01:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec800c4150 con 0x7fec8015d150 2023-10-14T17:01:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.273+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec8014b960 con 0x7fec8015d150 2023-10-14T17:01:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.275+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec8004e160 con 0x7fec8015d150 2023-10-14T17:01:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.276+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fec68004180 con 0x7fec8015d150 2023-10-14T17:01:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.276+0000 7fec777fe640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 0x7fec6006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.277+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fec68089d20 con 0x7fec8015d150 2023-10-14T17:01:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.277+0000 7fec7e59c640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 0x7fec6006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.277+0000 7fec7e59c640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 0x7fec6006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec70036000 tx=0x7fec7003c040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.281+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fec68054ae0 con 0x7fec8015d150 2023-10-14T17:01:05.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.476+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fec800ade10 con 0x7fec8015d150 2023-10-14T17:01:05.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.477+0000 7fec777fe640 1 -- 172.21.15.29:0/2380326570 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v145) v1 ==== 93+0+10 (secure 0 0 0) 0x7fec6801d070 con 0x7fec8015d150 2023-10-14T17:01:05.478 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:01:05.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.479+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 msgr2=0x7fec6006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.479+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 0x7fec6006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fec70036000 tx=0x7fec7003c040 comp rx=0 tx=0).stop 2023-10-14T17:01:05.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.479+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 msgr2=0x7fec800c2ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.479+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 secure :-1 s=READY pgs=1749 cs=0 l=1 rev1=1 crypto rx=0x7fec6800ccd0 tx=0x7fec6800a5c0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 shutdown_connections 2023-10-14T17:01:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fec6006afc0 0x7fec6006d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec80160b90 0x7fec80147680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec8015d150 0x7fec800c2ef0 unknown :-1 s=CLOSED pgs=1749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 --2- 172.21.15.29:0/2380326570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec80069da0 0x7fec801f22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 >> 172.21.15.29:0/2380326570 conn(0x7fec800ce9b0 msgr2=0x7fec80070f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:05.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 shutdown_connections 2023-10-14T17:01:05.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.480+0000 7fec84e83640 1 -- 172.21.15.29:0/2380326570 wait complete. 2023-10-14T17:01:05.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-14T17:01:05.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.718+0000 7fb0f69a7640 1 Processor -- start 2023-10-14T17:01:05.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.718+0000 7fb0f69a7640 1 -- start start 2023-10-14T17:01:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.719+0000 7fb0f69a7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0f006fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.719+0000 7fb0f69a7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f00cdca0 con 0x7fb0f01599c0 2023-10-14T17:01:05.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.719+0000 7fb0f69a7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f006bab0 con 0x7fb0f00cd380 2023-10-14T17:01:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.719+0000 7fb0f69a7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f006f300 con 0x7fb0f00cd810 2023-10-14T17:01:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f59a5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb0f01599c0 0x7fb0f00ca1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38620/0 (socket says 172.21.15.29:38620) 2023-10-14T17:01:05.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f59a5640 1 -- 172.21.15.29:0/343040224 learned_addr learned my addr 172.21.15.29:0/343040224 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214731241 0 0) 0x7fb0f00cdca0 con 0x7fb0f01599c0 2023-10-14T17:01:05.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0dc003520 con 0x7fb0f01599c0 2023-10-14T17:01:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0f006fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873151040 0 0) 0x7fb0dc003520 con 0x7fb0f01599c0 2023-10-14T17:01:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 msgr2=0x7fb0f006fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0f006fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 >> v1:172.21.15.186:6789/0 conn(0x7fb0f00cd380 legacy=0x7fb0f006f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f006bd90 con 0x7fb0f01599c0 2023-10-14T17:01:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.720+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb0ec001670 con 0x7fb0f01599c0 2023-10-14T17:01:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 >> v1:172.21.15.29:6789/0 conn(0x7fb0f01599c0 legacy=0x7fb0f00ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0dc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0dc003f20 0x7fb0dc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0dc004d00 0x7fb0dc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f006f300 con 0x7fb0dc004d00 2023-10-14T17:01:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc00bb10 con 0x7fb0dc003f20 2023-10-14T17:01:05.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0dc00bcd0 con 0x7fb0f00cd810 2023-10-14T17:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0dc004d00 0x7fb0dc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f59a5640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0dc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f51a4640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0dc003f20 0x7fb0dc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f61a6640 1 -- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 msgr2=0x7fb0dc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:05.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f00cd810 0x7fb0dc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f61a6640 1 -- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0dc003f20 msgr2=0x7fb0dc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.721+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0dc003f20 0x7fb0dc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.722+0000 7fb0f61a6640 1 -- 172.21.15.29:0/343040224 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0dc003520 con 0x7fb0dc004d00 2023-10-14T17:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0dc004d00 0x7fb0dc005180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009fc0 tx=0x7fb0dc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccf4f67e18f51d25 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e802ee30 con 0x7fb0dc004d00 2023-10-14T17:01:05.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e800f040 con 0x7fb0dc004d00 2023-10-14T17:01:05.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f49a3640 1 -- 172.21.15.29:0/343040224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8036680 con 0x7fb0dc004d00 2023-10-14T17:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f69a7640 1 -- 172.21.15.29:0/343040224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0dc004d00 msgr2=0x7fb0dc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/343040224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0dc004d00 0x7fb0dc005180 secure :-1 s=READY pgs=1751 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8009fc0 tx=0x7fb0dc0057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.723+0000 7fb0f61a6640 1 -- 172.21.15.29:0/343040224 reap_dead start 2023-10-14T17:01:05.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.724+0000 7fb0f69a7640 1 -- 172.21.15.29:0/343040224 shutdown_connections 2023-10-14T17:01:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.724+0000 7fb0f69a7640 1 -- 172.21.15.29:0/343040224 >> 172.21.15.29:0/343040224 conn(0x7fb0f00b0a90 msgr2=0x7fb0f00ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.724+0000 7fb0f69a7640 1 -- 172.21.15.29:0/343040224 shutdown_connections 2023-10-14T17:01:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.724+0000 7fb0f69a7640 1 -- 172.21.15.29:0/343040224 wait complete. 2023-10-14T17:01:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.725+0000 7fb0f69a7640 1 Processor -- start 2023-10-14T17:01:05.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.726+0000 7fb0f69a7640 1 -- start start 2023-10-14T17:01:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.726+0000 7fb0f69a7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0dc004d00 0x7fb0f00c7500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f69a7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0f014ec40 0x7fb0f00c7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f69a7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f69a7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f00c85a0 con 0x7fb0f00c8120 2023-10-14T17:01:05.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f69a7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f014f0c0 con 0x7fb0f014ec40 2023-10-14T17:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f69a7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f00c6460 con 0x7fb0dc004d00 2023-10-14T17:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39326/0 (socket says 172.21.15.29:39326) 2023-10-14T17:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 -- 172.21.15.29:0/484839634 learned_addr learned my addr 172.21.15.29:0/484839634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:05.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f59a5640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0dc004d00 0x7fb0f00c7500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 -- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0dc004d00 msgr2=0x7fb0f00c7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0dc004d00 0x7fb0f00c7500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 -- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0f014ec40 msgr2=0x7fb0f00c7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0f014ec40 0x7fb0f00c7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:05.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.727+0000 7fb0f61a6640 1 -- 172.21.15.29:0/484839634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f00c6740 con 0x7fb0f00c8120 2023-10-14T17:01:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0f61a6640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8002230 tx=0x7fb0e8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8043070 con 0x7fb0f00c8120 2023-10-14T17:01:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e8036b50 con 0x7fb0f00c8120 2023-10-14T17:01:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e8003990 con 0x7fb0f00c8120 2023-10-14T17:01:05.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f00c6a20 con 0x7fb0f00c8120 2023-10-14T17:01:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.728+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f00c6fe0 con 0x7fb0f00c8120 2023-10-14T17:01:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.729+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f004e9b0 con 0x7fb0f00c8120 2023-10-14T17:01:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.730+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb0e803d070 con 0x7fb0f00c8120 2023-10-14T17:01:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.731+0000 7fb0e67fc640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:05.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.731+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fb0e80ab4c0 con 0x7fb0f00c8120 2023-10-14T17:01:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.731+0000 7fb0f59a5640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.732+0000 7fb0f59a5640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 0x7fb0d406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000f60 tx=0x7fb0ec000ed0 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:05.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:05.737+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0e8076390 con 0x7fb0f00c8120 2023-10-14T17:01:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.008+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7fb0f0159150 con 0x7fb0f00c8120 2023-10-14T17:01:06.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.562+0000 7fb0e67fc640 1 -- 172.21.15.29:0/484839634 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v809) v1 ==== 181+0+0 (secure 0 0 0) 0x7fb0e803f460 con 0x7fb0f00c8120 2023-10-14T17:01:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.566+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 msgr2=0x7fb0d406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.566+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 0x7fb0d406d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec000f60 tx=0x7fb0ec000ed0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.566+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 msgr2=0x7fb0f00c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.566+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 secure :-1 s=READY pgs=1752 cs=0 l=1 rev1=1 crypto rx=0x7fb0e8002230 tx=0x7fb0e8009750 comp rx=0 tx=0).stop 2023-10-14T17:01:06.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 shutdown_connections 2023-10-14T17:01:06.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f00c8120 0x7fb0f00c5770 unknown :-1 s=CLOSED pgs=1752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0f014ec40 0x7fb0f00c7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 --2- 172.21.15.29:0/484839634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0dc004d00 0x7fb0f00c7500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 >> 172.21.15.29:0/484839634 conn(0x7fb0f00b0a90 msgr2=0x7fb0f00cc1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 shutdown_connections 2023-10-14T17:01:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.567+0000 7fb0f69a7640 1 -- 172.21.15.29:0/484839634 wait complete. 2023-10-14T17:01:06.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T17:01:06.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.817+0000 7f16a71dd640 1 Processor -- start 2023-10-14T17:01:06.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.818+0000 7f16a71dd640 1 -- start start 2023-10-14T17:01:06.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.819+0000 7f16a71dd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f16a00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.819+0000 7f16a71dd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f16a00cded0 0x7f16a00b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.819+0000 7f16a71dd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16a015d150 con 0x7f16a00b8480 2023-10-14T17:01:06.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.819+0000 7f16a71dd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16a00d1180 con 0x7f16a00cded0 2023-10-14T17:01:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.819+0000 7f16a71dd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00d1460 con 0x7f16a015cd70 2023-10-14T17:01:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a59da640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f16a00b8480 0x7f16a00cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38626/0 (socket says 172.21.15.29:38626) 2023-10-14T17:01:06.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a59da640 1 -- 172.21.15.29:0/3259499258 learned_addr learned my addr 172.21.15.29:0/3259499258 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a61db640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f16a00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1804909550 0 0) 0x7f16a015d150 con 0x7f16a00b8480 2023-10-14T17:01:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a69dc640 1 --2- 172.21.15.29:0/3259499258 >> v2:172.21.15.186:3300/0 conn(0x7f16a00cded0 0x7f16a00b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f168c003520 con 0x7f16a00b8480 2023-10-14T17:01:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1711551570 0 0) 0x7f168c003520 con 0x7f16a00b8480 2023-10-14T17:01:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 msgr2=0x7f16a00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:06.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f16a00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 >> v2:172.21.15.186:3300/0 conn(0x7f16a00cded0 msgr2=0x7f16a00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 --2- 172.21.15.29:0/3259499258 >> v2:172.21.15.186:3300/0 conn(0x7f16a00cded0 0x7f16a00b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a00d1740 con 0x7f16a00b8480 2023-10-14T17:01:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1690001670 con 0x7f16a00b8480 2023-10-14T17:01:06.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.820+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 >> v1:172.21.15.29:6789/0 conn(0x7f16a00b8480 legacy=0x7f16a00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f168c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f168c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f168c004d00 0x7f168c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00d1180 con 0x7f168c003ff0 2023-10-14T17:01:06.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a00d1460 con 0x7f168c004d00 2023-10-14T17:01:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168c00bae0 con 0x7f16a015cd70 2023-10-14T17:01:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f168c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a61db640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f168c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a69dc640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f168c004d00 0x7f168c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 -- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 msgr2=0x7f168c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f168c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 -- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f168c004d00 msgr2=0x7f168c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f168c004d00 0x7f168c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a59da640 1 -- 172.21.15.29:0/3259499258 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168c003520 con 0x7f168c003ff0 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.821+0000 7f16a61db640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a015cd70 0x7f168c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:06.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.822+0000 7f16a59da640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f168c0047c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f169000c110 tx=0x7f168c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fec07662204cef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1690005ad0 con 0x7f168c003ff0 2023-10-14T17:01:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1690005c50 con 0x7f168c003ff0 2023-10-14T17:01:06.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a51d9640 1 -- 172.21.15.29:0/3259499258 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169002c830 con 0x7f168c003ff0 2023-10-14T17:01:06.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a71dd640 1 -- 172.21.15.29:0/3259499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 msgr2=0x7f168c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3259499258 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f168c0047c0 secure :-1 s=READY pgs=1754 cs=0 l=1 rev1=1 crypto rx=0x7f169000c110 tx=0x7f168c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a69dc640 1 -- 172.21.15.29:0/3259499258 reap_dead start 2023-10-14T17:01:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a71dd640 1 -- 172.21.15.29:0/3259499258 shutdown_connections 2023-10-14T17:01:06.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.823+0000 7f16a71dd640 1 -- 172.21.15.29:0/3259499258 >> 172.21.15.29:0/3259499258 conn(0x7f16a00cf570 msgr2=0x7f16a00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.824+0000 7f16a71dd640 1 -- 172.21.15.29:0/3259499258 shutdown_connections 2023-10-14T17:01:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.824+0000 7f16a71dd640 1 -- 172.21.15.29:0/3259499258 wait complete. 2023-10-14T17:01:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.825+0000 7f16a71dd640 1 Processor -- start 2023-10-14T17:01:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.826+0000 7f16a71dd640 1 -- start start 2023-10-14T17:01:06.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.826+0000 7f16a71dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.826+0000 7f16a71dd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16a01f3350 0x7f16a00b4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.826+0000 7f16a71dd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a00a9760 0x7f16a0158bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a71dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01f4560 con 0x7f168c003ff0 2023-10-14T17:01:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a71dd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01f38e0 con 0x7f16a01f3350 2023-10-14T17:01:06.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a71dd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16a01f3bc0 con 0x7f16a00a9760 2023-10-14T17:01:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39348/0 (socket says 172.21.15.29:39348) 2023-10-14T17:01:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 -- 172.21.15.29:0/3129070994 learned_addr learned my addr 172.21.15.29:0/3129070994 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 -- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a00a9760 msgr2=0x7f16a0158bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a00a9760 0x7f16a0158bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 -- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16a01f3350 msgr2=0x7f16a00b4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16a01f3350 0x7f16a00b4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.827+0000 7f16a61db640 1 -- 172.21.15.29:0/3129070994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16a01f3ea0 con 0x7f168c003ff0 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f16a61db640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f169c00e7e0 tx=0x7f169c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c00c6d0 con 0x7f168c003ff0 2023-10-14T17:01:06.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f169c00c830 con 0x7f168c003ff0 2023-10-14T17:01:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f169c00cba0 con 0x7f168c003ff0 2023-10-14T17:01:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16a01f4180 con 0x7f168c003ff0 2023-10-14T17:01:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.828+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16a00a9be0 con 0x7f168c003ff0 2023-10-14T17:01:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.829+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16a00a9e60 con 0x7f168c003ff0 2023-10-14T17:01:06.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.830+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f169c00cd00 con 0x7f168c003ff0 2023-10-14T17:01:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.831+0000 7f1696ffd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 0x7f168406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.831+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7f169c0894e0 con 0x7f168c003ff0 2023-10-14T17:01:06.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.831+0000 7f16a59da640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 0x7f168406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.832+0000 7f16a59da640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 0x7f168406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1690034000 tx=0x7f169002e000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:06.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:06.835+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f169c0543d0 con 0x7f168c003ff0 2023-10-14T17:01:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.036+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f16a004e9b0 con 0x7f168c003ff0 2023-10-14T17:01:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.043+0000 7f1696ffd640 1 -- 172.21.15.29:0/3129070994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v52)=0 v52) v1 ==== 122+0+0 (secure 0 0 0) 0x7f169c0579f0 con 0x7f168c003ff0 2023-10-14T17:01:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 msgr2=0x7f168406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 0x7f168406d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f1690034000 tx=0x7f169002e000 comp rx=0 tx=0).stop 2023-10-14T17:01:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 msgr2=0x7f16a015d340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 secure :-1 s=READY pgs=1755 cs=0 l=1 rev1=1 crypto rx=0x7f169c00e7e0 tx=0x7f169c00eca0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 shutdown_connections 2023-10-14T17:01:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f168406af90 0x7f168406d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f16a00a9760 0x7f16a0158bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f16a01f3350 0x7f16a00b4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 --2- 172.21.15.29:0/3129070994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f168c003ff0 0x7f16a015d340 unknown :-1 s=CLOSED pgs=1755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.046+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 >> 172.21.15.29:0/3129070994 conn(0x7f16a00cf570 msgr2=0x7f16a00b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.047+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 shutdown_connections 2023-10-14T17:01:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.047+0000 7f16a71dd640 1 -- 172.21.15.29:0/3129070994 wait complete. 2023-10-14T17:01:07.066 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:01:07.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:01:07.186 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:01:07.315 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:01:07.315 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:01:07.316 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:01:07.378 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:01:07.379 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:01:07.379 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:01:07.437 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:01:07.438 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:01:07.438 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:01:07.502 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:01:07.503 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:01:07.503 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:01:07.563 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:01:07.563 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T17:01:07.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:01:07.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.806+0000 7ff20bfff640 1 Processor -- start 2023-10-14T17:01:07.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.806+0000 7ff20bfff640 1 -- start start 2023-10-14T17:01:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c15f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c06c960 con 0x7ff20c06c040 2023-10-14T17:01:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c15b9d0 con 0x7ff20c0ad970 2023-10-14T17:01:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15bcb0 con 0x7ff20c15ebc0 2023-10-14T17:01:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39350/0 (socket says 172.21.15.29:39350) 2023-10-14T17:01:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.808+0000 7ff20affd640 1 -- 172.21.15.29:0/177325340 learned_addr learned my addr 172.21.15.29:0/177325340 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.809+0000 7ff20affd640 1 -- 172.21.15.29:0/177325340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 msgr2=0x7ff20c15f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:07.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.809+0000 7ff20affd640 1 --2- 172.21.15.29:0/177325340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c15f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.809+0000 7ff20affd640 1 -- 172.21.15.29:0/177325340 >> v2:172.21.15.186:3300/0 conn(0x7ff20c0ad970 msgr2=0x7ff20c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.809+0000 7ff20affd640 1 --2- 172.21.15.29:0/177325340 >> v2:172.21.15.186:3300/0 conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.809+0000 7ff20affd640 1 -- 172.21.15.29:0/177325340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c15bf90 con 0x7ff20c06c040 2023-10-14T17:01:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.810+0000 7ff20affd640 1 --2- 172.21.15.29:0/177325340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0cb060 tx=0x7ff1fc02b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=393b1e13cb7babc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:07.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.810+0000 7ff20a7fc640 1 -- 172.21.15.29:0/177325340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc031880 con 0x7ff20c06c040 2023-10-14T17:01:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.810+0000 7ff20a7fc640 1 -- 172.21.15.29:0/177325340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc0319e0 con 0x7ff20c06c040 2023-10-14T17:01:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.810+0000 7ff20a7fc640 1 -- 172.21.15.29:0/177325340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc031d70 con 0x7ff20c06c040 2023-10-14T17:01:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 -- 172.21.15.29:0/177325340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 msgr2=0x7ff20c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 --2- 172.21.15.29:0/177325340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 secure :-1 s=READY pgs=1756 cs=0 l=1 rev1=1 crypto rx=0x7ff20c0cb060 tx=0x7ff1fc02b920 comp rx=0 tx=0).stop 2023-10-14T17:01:07.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 -- 172.21.15.29:0/177325340 shutdown_connections 2023-10-14T17:01:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 --2- 172.21.15.29:0/177325340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c15f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 --2- 172.21.15.29:0/177325340 >> v2:172.21.15.186:3300/0 conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 --2- 172.21.15.29:0/177325340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=CLOSED pgs=1756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 -- 172.21.15.29:0/177325340 >> 172.21.15.29:0/177325340 conn(0x7ff20c0ce6a0 msgr2=0x7ff20c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:07.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 -- 172.21.15.29:0/177325340 shutdown_connections 2023-10-14T17:01:07.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.811+0000 7ff20bfff640 1 -- 172.21.15.29:0/177325340 wait complete. 2023-10-14T17:01:07.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.813+0000 7ff20bfff640 1 Processor -- start 2023-10-14T17:01:07.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.813+0000 7ff20bfff640 1 -- start start 2023-10-14T17:01:07.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c06c040 0x7ff20c151be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15a1e0 con 0x7ff20c0ad970 2023-10-14T17:01:07.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c152120 con 0x7ff20c06c040 2023-10-14T17:01:07.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff20bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0aed80 con 0x7ff20c15ebc0 2023-10-14T17:01:07.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.814+0000 7ff203fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:07.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39364/0 (socket says 172.21.15.29:39364) 2023-10-14T17:01:07.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 -- 172.21.15.29:0/4050191103 learned_addr learned my addr 172.21.15.29:0/4050191103 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:07.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:07.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff20affd640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c06c040 0x7ff20c151be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:07.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 -- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 msgr2=0x7ff20c0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:07.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 -- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c06c040 msgr2=0x7ff20c151be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:07.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c06c040 0x7ff20c151be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:07.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 -- 172.21.15.29:0/4050191103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0094d0 con 0x7ff20c0ad970 2023-10-14T17:01:07.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.815+0000 7ff203fff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400b410 tx=0x7ff1f400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:07.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.816+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4011830 con 0x7ff20c0ad970 2023-10-14T17:01:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.816+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1f4011990 con 0x7ff20c0ad970 2023-10-14T17:01:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.816+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1f4011d40 con 0x7ff20c0ad970 2023-10-14T17:01:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.816+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c0af060 con 0x7ff20c0ad970 2023-10-14T17:01:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.816+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c0c58f0 con 0x7ff20c0ad970 2023-10-14T17:01:07.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.817+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c06b330 con 0x7ff20c0ad970 2023-10-14T17:01:07.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.818+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1f401a400 con 0x7ff20c0ad970 2023-10-14T17:01:07.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.819+0000 7ff2037fe640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:07.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.819+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7ff1f40899b0 con 0x7ff20c0ad970 2023-10-14T17:01:07.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.819+0000 7ff20affd640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:07.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.820+0000 7ff20affd640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 0x7ff1ec06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03c040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:07.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:07.824+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1f4054880 con 0x7ff20c0ad970 2023-10-14T17:01:08.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.065+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff20c0c5750 con 0x7ff20c0ad970 2023-10-14T17:01:08.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.066+0000 7ff2037fe640 1 -- 172.21.15.29:0/4050191103 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+3705 (secure 0 0 0) 0x7ff20c0c5750 con 0x7ff20c0ad970 2023-10-14T17:01:08.067 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:08.067 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":68,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6065,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/2528550276","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2528550276},{"type":"v1","addr":"172.21.15.186:6833","nonce":2528550276}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6068,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3747078297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3747078297},{"type":"v1","addr":"172.21.15.29:6837","nonce":3747078297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":6071,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1600913396","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1600913396},{"type":"v1","addr":"172.21.15.186:6835","nonce":1600913396}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":61},{"gid":9009,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3850576614","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3850576614},{"type":"v1","addr":"172.21.15.29:6836","nonce":3850576614}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":68}],"filesystems":[]} 2023-10-14T17:01:08.067 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 68 2023-10-14T17:01:08.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.068+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 msgr2=0x7ff1ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.068+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 0x7ff1ec06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc036000 tx=0x7ff1fc03c040 comp rx=0 tx=0).stop 2023-10-14T17:01:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.068+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 msgr2=0x7ff20c0afd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.068+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 secure :-1 s=READY pgs=1757 cs=0 l=1 rev1=1 crypto rx=0x7ff1f400b410 tx=0x7ff1f400b8d0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 shutdown_connections 2023-10-14T17:01:08.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c15ebc0 0x7ff20c0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c0ad970 0x7ff20c0afd00 unknown :-1 s=CLOSED pgs=1757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 --2- 172.21.15.29:0/4050191103 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c06c040 0x7ff20c151be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 >> 172.21.15.29:0/4050191103 conn(0x7ff20c0ce6a0 msgr2=0x7ff20c0b3990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:08.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 shutdown_connections 2023-10-14T17:01:08.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.069+0000 7ff20bfff640 1 -- 172.21.15.29:0/4050191103 wait complete. 2023-10-14T17:01:08.090 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T17:01:08.090 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T17:01:08.090 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T17:01:08.091 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:01:08.091 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T17:01:08.091 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T17:01:08.091 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T17:01:08.092 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:01:08.092 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T17:01:08.092 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T17:01:08.093 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T17:01:08.093 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:01:08.093 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T17:01:08.093 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T17:01:08.094 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T17:01:08.094 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:01:08.108 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T17:01:08.109 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T17:01:08.122 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T17:01:08.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T17:01:08.123 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T17:01:08.123 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T17:01:08.125 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T17:01:08.125 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T17:01:08.171 INFO:tasks.ceph.mds.c:Started 2023-10-14T17:01:08.171 DEBUG:teuthology.parallel:result is None 2023-10-14T17:01:08.172 INFO:tasks.ceph.mds.b:Started 2023-10-14T17:01:08.173 DEBUG:teuthology.parallel:result is None 2023-10-14T17:01:08.179 INFO:tasks.ceph.mds.a:Started 2023-10-14T17:01:08.179 DEBUG:teuthology.parallel:result is None 2023-10-14T17:01:08.180 INFO:tasks.ceph.mds.d:Started 2023-10-14T17:01:08.180 DEBUG:teuthology.parallel:result is None 2023-10-14T17:01:08.181 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T17:01:08.334 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:01:08.335 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T17:01:08.338 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T17:01:08.347 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T17:01:08.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.376+0000 7fc3c2cfc640 1 Processor -- start 2023-10-14T17:01:08.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.376+0000 7fc3c2cfc640 1 -- start start 2023-10-14T17:01:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.377+0000 7fc3c2cfc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15cba0 0x7fc3bc071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.377+0000 7fc3c2cfc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.377+0000 7fc3c2cfc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc06bb60 con 0x7fc3bc0ccdc0 2023-10-14T17:01:08.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.377+0000 7fc3c2cfc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc0716b0 con 0x7fc3bc15cba0 2023-10-14T17:01:08.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.377+0000 7fc3c2cfc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc0d09e0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c1cfa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15cba0 0x7fc3bc071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:58506/0 (socket says 172.21.15.29:58506) 2023-10-14T17:01:08.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 -- 172.21.15.29:0/61218160 learned_addr learned my addr 172.21.15.29:0/61218160 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c14f9640 1 --1- 172.21.15.29:0/61218160 >> v1:172.21.15.186:6789/0 conn(0x7fc3bc0ccdc0 0x7fc3bc0cd840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:47462/0 (socket says 172.21.15.29:47462) 2023-10-14T17:01:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 -- 172.21.15.29:0/61218160 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15cba0 msgr2=0x7fc3bc071bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 --2- 172.21.15.29:0/61218160 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15cba0 0x7fc3bc071bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 -- 172.21.15.29:0/61218160 >> v1:172.21.15.186:6789/0 conn(0x7fc3bc0ccdc0 legacy=0x7fc3bc0cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.378+0000 7fc3c24fb640 1 -- 172.21.15.29:0/61218160 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc06af50 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.379+0000 7fc3c24fb640 1 --2- 172.21.15.29:0/61218160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009e50 tx=0x7fc3b402bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=83369ce1e0130fd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.379+0000 7fc3c0cf8640 1 -- 172.21.15.29:0/61218160 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b402cb00 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c0cf8640 1 -- 172.21.15.29:0/61218160 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b402cc60 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c0cf8640 1 -- 172.21.15.29:0/61218160 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b40343f0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/61218160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 msgr2=0x7fc3bc0cbf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/61218160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4009e50 tx=0x7fc3b402bf60 comp rx=0 tx=0).stop 2023-10-14T17:01:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/61218160 shutdown_connections 2023-10-14T17:01:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/61218160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0cdf50 0x7fc3bc0cbf40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/61218160 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15cba0 0x7fc3bc071bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/61218160 >> 172.21.15.29:0/61218160 conn(0x7fc3bc065ed0 msgr2=0x7fc3bc0662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:08.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/61218160 shutdown_connections 2023-10-14T17:01:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.380+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/61218160 wait complete. 2023-10-14T17:01:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.382+0000 7fc3c2cfc640 1 Processor -- start 2023-10-14T17:01:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.382+0000 7fc3c2cfc640 1 -- start start 2023-10-14T17:01:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0ccdc0 0x7fc3bc159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15cba0 0x7fc3bc157590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc1595c0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc158bf0 con 0x7fc3bc15cba0 2023-10-14T17:01:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c2cfc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc157fe0 con 0x7fc3bc0ccdc0 2023-10-14T17:01:08.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.383+0000 7fc3c14f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39384/0 (socket says 172.21.15.29:39384) 2023-10-14T17:01:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 -- 172.21.15.29:0/1574020660 learned_addr learned my addr 172.21.15.29:0/1574020660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c24fb640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15cba0 0x7fc3bc157590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c1cfa640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0ccdc0 0x7fc3bc159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 -- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0ccdc0 msgr2=0x7fc3bc159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0ccdc0 0x7fc3bc159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 -- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15cba0 msgr2=0x7fc3bc157590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15cba0 0x7fc3bc157590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:08.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 -- 172.21.15.29:0/1574020660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3b4009b30 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.384+0000 7fc3c14f9640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00c3e0 tx=0x7fc3ac00c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.385+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0074e0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.385+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3ac007640 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.385+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc1582c0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.385+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac0049e0 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.385+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc158880 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.386+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3bc158d80 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.387+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3ac004b40 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.388+0000 7fc3b27fc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 0x7fc3a006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.388+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5746+0+0 (secure 0 0 0) 0x7fc3ac089070 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.388+0000 7fc3c1cfa640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 0x7fc3a006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.389+0000 7fc3c1cfa640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 0x7fc3a006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009b40 tx=0x7fc3b8009190 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:08.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.393+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3ac054000 con 0x7fc3bc0cdf50 2023-10-14T17:01:08.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:08.603+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc3bc15cf80 con 0x7fc3bc0cdf50 2023-10-14T17:01:09.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.613+0000 7fc3b27fc640 1 -- 172.21.15.29:0/1574020660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v150) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc3ac057620 con 0x7fc3bc0cdf50 2023-10-14T17:01:09.614 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T17:01:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 msgr2=0x7fc3a006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 0x7fc3a006d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc3b8009b40 tx=0x7fc3b8009190 comp rx=0 tx=0).stop 2023-10-14T17:01:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 msgr2=0x7fc3bc149ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 secure :-1 s=READY pgs=1764 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac00c3e0 tx=0x7fc3ac00c8a0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 shutdown_connections 2023-10-14T17:01:09.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc3a006afc0 0x7fc3a006d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15cba0 0x7fc3bc157590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc0cdf50 0x7fc3bc149ae0 unknown :-1 s=CLOSED pgs=1764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 --2- 172.21.15.29:0/1574020660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3bc0ccdc0 0x7fc3bc159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 >> 172.21.15.29:0/1574020660 conn(0x7fc3bc065ed0 msgr2=0x7fc3bc06c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 shutdown_connections 2023-10-14T17:01:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.617+0000 7fc3c2cfc640 1 -- 172.21.15.29:0/1574020660 wait complete. 2023-10-14T17:01:09.639 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T17:01:09.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.858+0000 7fd63f3cb640 1 Processor -- start 2023-10-14T17:01:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.859+0000 7fd63f3cb640 1 -- start start 2023-10-14T17:01:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63f3cb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6380d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63f3cb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd63815d150 con 0x7fd6380b8480 2023-10-14T17:01:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63f3cb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6380d1180 con 0x7fd6380cded0 2023-10-14T17:01:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63f3cb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380d1460 con 0x7fd63815cd70 2023-10-14T17:01:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63dbc8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd6380b8480 0x7fd6380cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38632/0 (socket says 172.21.15.29:38632) 2023-10-14T17:01:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.860+0000 7fd63dbc8640 1 -- 172.21.15.29:0/662339022 learned_addr learned my addr 172.21.15.29:0/662339022 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3706783077 0 0) 0x7fd63815d150 con 0x7fd6380b8480 2023-10-14T17:01:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd624003520 con 0x7fd6380b8480 2023-10-14T17:01:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6380d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2744487673 0 0) 0x7fd624003520 con 0x7fd6380b8480 2023-10-14T17:01:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 msgr2=0x7fd6380d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6380d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 >> v1:172.21.15.186:6789/0 conn(0x7fd6380cded0 legacy=0x7fd6380b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6380d1740 con 0x7fd6380b8480 2023-10-14T17:01:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd628001670 con 0x7fd6380b8480 2023-10-14T17:01:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 >> v1:172.21.15.29:6789/0 conn(0x7fd6380b8480 legacy=0x7fd6380cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6240039a0 0x7fd624003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6240047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd624004d00 0x7fd624005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6380d1460 con 0x7fd6240039a0 2023-10-14T17:01:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400bb10 con 0x7fd624004d00 2023-10-14T17:01:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd62400bcd0 con 0x7fd63815cd70 2023-10-14T17:01:09.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.861+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6240039a0 0x7fd624003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63dbc8640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6240047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63e3c9640 1 -- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 msgr2=0x7fd6240047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:09.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815cd70 0x7fd6240047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63e3c9640 1 -- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd624004d00 msgr2=0x7fd624005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:09.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63ebca640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd624004d00 0x7fd624005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd624004d00 0x7fd624005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63e3c9640 1 -- 172.21.15.29:0/662339022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd624003520 con 0x7fd6240039a0 2023-10-14T17:01:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.862+0000 7fd63ebca640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd624004d00 0x7fd624005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.863+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6240039a0 0x7fd624003da0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fd634007360 tx=0x7fd6240057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c3b568d0673145 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.863+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd63402ccd0 con 0x7fd6240039a0 2023-10-14T17:01:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.863+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd63402ce30 con 0x7fd6240039a0 2023-10-14T17:01:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.863+0000 7fd63d3c7640 1 -- 172.21.15.29:0/662339022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd634034660 con 0x7fd6240039a0 2023-10-14T17:01:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 -- 172.21.15.29:0/662339022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6240039a0 msgr2=0x7fd624003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/662339022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6240039a0 0x7fd624003da0 secure :-1 s=READY pgs=1766 cs=0 l=1 rev1=1 crypto rx=0x7fd634007360 tx=0x7fd6240057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63ebca640 1 -- 172.21.15.29:0/662339022 reap_dead start 2023-10-14T17:01:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 -- 172.21.15.29:0/662339022 shutdown_connections 2023-10-14T17:01:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 -- 172.21.15.29:0/662339022 >> 172.21.15.29:0/662339022 conn(0x7fd6380cf570 msgr2=0x7fd6380cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 -- 172.21.15.29:0/662339022 shutdown_connections 2023-10-14T17:01:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.864+0000 7fd63f3cb640 1 -- 172.21.15.29:0/662339022 wait complete. 2023-10-14T17:01:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.866+0000 7fd63f3cb640 1 Processor -- start 2023-10-14T17:01:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.866+0000 7fd63f3cb640 1 -- start start 2023-10-14T17:01:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6240039a0 0x7fd63815f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815d340 0x7fd6380b5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd638156830 con 0x7fd6381f2790 2023-10-14T17:01:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381f2d50 con 0x7fd6240039a0 2023-10-14T17:01:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.867+0000 7fd63f3cb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6381f3030 con 0x7fd63815d340 2023-10-14T17:01:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63e3c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6240039a0 0x7fd63815f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39412/0 (socket says 172.21.15.29:39412) 2023-10-14T17:01:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 -- 172.21.15.29:0/687635658 learned_addr learned my addr 172.21.15.29:0/687635658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63ebca640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815d340 0x7fd6380b5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 -- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815d340 msgr2=0x7fd6380b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815d340 0x7fd6380b5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 -- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6240039a0 msgr2=0x7fd63815f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6240039a0 0x7fd63815f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.868+0000 7fd63dbc8640 1 -- 172.21.15.29:0/687635658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6381f3310 con 0x7fd6381f2790 2023-10-14T17:01:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd63dbc8640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fd62800d810 tx=0x7fd62800dcd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6280074e0 con 0x7fd6381f2790 2023-10-14T17:01:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd628007640 con 0x7fd6381f2790 2023-10-14T17:01:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628004a00 con 0x7fd6381f2790 2023-10-14T17:01:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6381f35f0 con 0x7fd6381f2790 2023-10-14T17:01:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.869+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd638158430 con 0x7fd6381f2790 2023-10-14T17:01:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.870+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd63804e9b0 con 0x7fd6381f2790 2023-10-14T17:01:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.871+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd628004b60 con 0x7fd6381f2790 2023-10-14T17:01:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.872+0000 7fd62effd640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.872+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fd628088e20 con 0x7fd6381f2790 2023-10-14T17:01:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.873+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:09.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.873+0000 7fd63e3c9640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 0x7fd61c06d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd634036000 tx=0x7fd634030000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:09.877+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd6280540f0 con 0x7fd6381f2790 2023-10-14T17:01:10.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.143+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fd6380b3450 con 0x7fd6381f2790 2023-10-14T17:01:10.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.144+0000 7fd62effd640 1 -- 172.21.15.29:0/687635658 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v19) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fd628019090 con 0x7fd6381f2790 2023-10-14T17:01:10.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:10.145 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:01:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T17:01:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T17:01:10.146 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.147 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T17:01:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T17:01:10.148 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:01:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:01:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.149 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T17:01:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T17:01:10.150 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.151 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T17:01:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T17:01:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.152 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:01:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:01:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T17:01:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T17:01:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.154 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.155 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.156 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T17:01:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T17:01:10.157 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.158 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.159 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.160 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T17:01:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T17:01:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.161 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T17:01:10.162 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T17:01:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.163 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.164 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.165 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T17:01:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T17:01:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:01:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:01:10.166 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T17:01:10.167 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T17:01:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T17:01:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.169 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:01:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:01:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:01:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.171 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:01:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:01:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:01:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T17:01:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T17:01:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:01:10.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:01:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:01:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:01:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.174 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:01:10.175 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:01:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.176 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.177 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:01:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T17:01:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T17:01:10.178 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:01:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:01:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:01:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:01:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.179 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:10.180 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T17:01:10.180 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:01:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 msgr2=0x7fd61c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 0x7fd61c06d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fd634036000 tx=0x7fd634030000 comp rx=0 tx=0).stop 2023-10-14T17:01:10.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 msgr2=0x7fd638157d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 secure :-1 s=READY pgs=1767 cs=0 l=1 rev1=1 crypto rx=0x7fd62800d810 tx=0x7fd62800dcd0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 shutdown_connections 2023-10-14T17:01:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd61c06afc0 0x7fd61c06d480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd63815d340 0x7fd6380b5c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6381f2790 0x7fd638157d10 unknown :-1 s=CLOSED pgs=1767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 --2- 172.21.15.29:0/687635658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6240039a0 0x7fd63815f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 >> 172.21.15.29:0/687635658 conn(0x7fd6380cf570 msgr2=0x7fd6380b2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 shutdown_connections 2023-10-14T17:01:10.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.147+0000 7fd63f3cb640 1 -- 172.21.15.29:0/687635658 wait complete. 2023-10-14T17:01:10.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:01:10.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.407+0000 7fec0cc59640 1 Processor -- start 2023-10-14T17:01:10.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.407+0000 7fec0cc59640 1 -- start start 2023-10-14T17:01:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 0x7fec0806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec080ad970 0x7fec0815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec0815ebc0 0x7fec0815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0806c960 con 0x7fec0815ebc0 2023-10-14T17:01:10.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec0815b9d0 con 0x7fec0806c040 2023-10-14T17:01:10.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec0cc59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec0815bcb0 con 0x7fec080ad970 2023-10-14T17:01:10.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec0815ebc0 0x7fec0815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec07fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fec0815ebc0 0x7fec0815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39418/0 (socket says 172.21.15.29:39418) 2023-10-14T17:01:10.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.409+0000 7fec07fff640 1 -- 172.21.15.29:0/4101207489 learned_addr learned my addr 172.21.15.29:0/4101207489 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec077fe640 1 --2- 172.21.15.29:0/4101207489 >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 0x7fec0806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7febfffff640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec080ad970 0x7fec0815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec07fff640 1 -- 172.21.15.29:0/4101207489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec080ad970 msgr2=0x7fec0815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec07fff640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec080ad970 0x7fec0815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec07fff640 1 -- 172.21.15.29:0/4101207489 >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 msgr2=0x7fec0806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec07fff640 1 --2- 172.21.15.29:0/4101207489 >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 0x7fec0806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec07fff640 1 -- 172.21.15.29:0/4101207489 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec0815bf90 con 0x7fec0815ebc0 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.410+0000 7fec077fe640 1 --2- 172.21.15.29:0/4101207489 >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 0x7fec0806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.411+0000 7fec07fff640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0815ebc0 0x7fec0815f050 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fec00009e50 tx=0x7fec0002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff864f769780b5a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.411+0000 7fec06ffd640 1 -- 172.21.15.29:0/4101207489 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec0002ca80 con 0x7fec0815ebc0 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.411+0000 7fec06ffd640 1 -- 172.21.15.29:0/4101207489 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec0002cbe0 con 0x7fec0815ebc0 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.411+0000 7fec06ffd640 1 -- 172.21.15.29:0/4101207489 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec000343f0 con 0x7fec0815ebc0 2023-10-14T17:01:10.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 -- 172.21.15.29:0/4101207489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0815ebc0 msgr2=0x7fec0815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0815ebc0 0x7fec0815f050 secure :-1 s=READY pgs=1768 cs=0 l=1 rev1=1 crypto rx=0x7fec00009e50 tx=0x7fec0002bee0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 -- 172.21.15.29:0/4101207489 shutdown_connections 2023-10-14T17:01:10.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0815ebc0 0x7fec0815f050 unknown :-1 s=CLOSED pgs=1768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 --2- 172.21.15.29:0/4101207489 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec080ad970 0x7fec0815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 --2- 172.21.15.29:0/4101207489 >> v2:172.21.15.186:3300/0 conn(0x7fec0806c040 0x7fec0806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 -- 172.21.15.29:0/4101207489 >> 172.21.15.29:0/4101207489 conn(0x7fec080ce6a0 msgr2=0x7fec080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 -- 172.21.15.29:0/4101207489 shutdown_connections 2023-10-14T17:01:10.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.412+0000 7fec0cc59640 1 -- 172.21.15.29:0/4101207489 wait complete. 2023-10-14T17:01:10.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.414+0000 7fec0cc59640 1 Processor -- start 2023-10-14T17:01:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.414+0000 7fec0cc59640 1 -- start start 2023-10-14T17:01:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec080ad970 0x7fec08151830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec0815ebc0 0x7fec08152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec08153180 con 0x7fec0806c040 2023-10-14T17:01:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec08153390 con 0x7fec080ad970 2023-10-14T17:01:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec0cc59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec08153670 con 0x7fec0815ebc0 2023-10-14T17:01:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.415+0000 7fec077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39424/0 (socket says 172.21.15.29:39424) 2023-10-14T17:01:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 -- 172.21.15.29:0/1820448147 learned_addr learned my addr 172.21.15.29:0/1820448147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 -- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec0815ebc0 msgr2=0x7fec08152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec0815ebc0 0x7fec08152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 -- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec080ad970 msgr2=0x7fec08151830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec080ad970 0x7fec08151830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 -- 172.21.15.29:0/1820448147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec00009b30 con 0x7fec0806c040 2023-10-14T17:01:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec077fe640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7febf800c7f0 tx=0x7febf800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.416+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8007820 con 0x7fec0806c040 2023-10-14T17:01:10.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.417+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf8007980 con 0x7fec0806c040 2023-10-14T17:01:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.417+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf8007d30 con 0x7fec0806c040 2023-10-14T17:01:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.417+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec08153950 con 0x7fec0806c040 2023-10-14T17:01:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.417+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec08153f10 con 0x7fec0806c040 2023-10-14T17:01:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.418+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec0806b330 con 0x7fec0806c040 2023-10-14T17:01:10.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.419+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febf80113f0 con 0x7fec0806c040 2023-10-14T17:01:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.420+0000 7fec04ff9640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 0x7febe806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.420+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4929+0+0 (secure 0 0 0) 0x7febf8089960 con 0x7fec0806c040 2023-10-14T17:01:10.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.421+0000 7febfffff640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 0x7febe806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.421+0000 7febfffff640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 0x7febe806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec0815b900 tx=0x7febf0008040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.425+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febf8054a60 con 0x7fec0806c040 2023-10-14T17:01:10.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.623+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec0804e9b0 con 0x7fec0806c040 2023-10-14T17:01:10.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.624+0000 7fec04ff9640 1 -- 172.21.15.29:0/1820448147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v150) v1 ==== 74+0+12970 (secure 0 0 0) 0x7febf8058080 con 0x7fec0806c040 2023-10-14T17:01:10.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:10.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":150,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:01:09.611079+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":17,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:01:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 msgr2=0x7febe806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 0x7febe806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fec0815b900 tx=0x7febf0008040 comp rx=0 tx=0).stop 2023-10-14T17:01:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 msgr2=0x7fec08154e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 secure :-1 s=READY pgs=1769 cs=0 l=1 rev1=1 crypto rx=0x7febf800c7f0 tx=0x7febf800ccb0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 shutdown_connections 2023-10-14T17:01:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7febe806af90 0x7febe806d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fec0815ebc0 0x7fec08152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fec080ad970 0x7fec08151830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 --2- 172.21.15.29:0/1820448147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fec0806c040 0x7fec08154e20 unknown :-1 s=CLOSED pgs=1769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 >> 172.21.15.29:0/1820448147 conn(0x7fec080ce6a0 msgr2=0x7fec0815d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.626+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 shutdown_connections 2023-10-14T17:01:10.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.627+0000 7fec0cc59640 1 -- 172.21.15.29:0/1820448147 wait complete. 2023-10-14T17:01:10.648 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:01:10.649 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:01:10.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.887+0000 7f264a4ec640 1 Processor -- start 2023-10-14T17:01:10.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.887+0000 7f264a4ec640 1 -- start start 2023-10-14T17:01:10.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.888+0000 7f264a4ec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f264415e350 0x7f26440cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.888+0000 7f264a4ec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264406fbc0 con 0x7f264415f090 2023-10-14T17:01:10.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.889+0000 7f264a4ec640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440c9920 con 0x7f2644070970 2023-10-14T17:01:10.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.889+0000 7f264a4ec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f264415e730 con 0x7f264415e350 2023-10-14T17:01:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.889+0000 7f2649ceb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f264415f090 0x7f26440ca9c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38648/0 (socket says 172.21.15.29:38648) 2023-10-14T17:01:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.889+0000 7f2649ceb640 1 -- 172.21.15.29:0/3620119919 learned_addr learned my addr 172.21.15.29:0/3620119919 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.889+0000 7f26494ea640 1 --2- 172.21.15.29:0/3620119919 >> v2:172.21.15.186:3300/0 conn(0x7f264415e350 0x7f26440cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805485300 0 0) 0x7f264406fbc0 con 0x7f264415f090 2023-10-14T17:01:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2634003520 con 0x7f264415f090 2023-10-14T17:01:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3179565774 0 0) 0x7f2634003520 con 0x7f264415f090 2023-10-14T17:01:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 >> v2:172.21.15.186:3300/0 conn(0x7f264415e350 msgr2=0x7f26440cdb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 --2- 172.21.15.29:0/3620119919 >> v2:172.21.15.186:3300/0 conn(0x7f264415e350 0x7f26440cdb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 >> v1:172.21.15.186:6789/0 conn(0x7f2644070970 legacy=0x7f264415e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440c9c00 con 0x7f264415f090 2023-10-14T17:01:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f263c001670 con 0x7f264415f090 2023-10-14T17:01:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 >> v1:172.21.15.29:6789/0 conn(0x7f264415f090 legacy=0x7f26440ca9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2634003970 0x7f2634003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26340047c0 0x7f2634004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 0x7f263400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263400c0f0 con 0x7f2634005180 2023-10-14T17:01:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263400c260 con 0x7f26340047c0 2023-10-14T17:01:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.890+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f263400c420 con 0x7f2634003970 2023-10-14T17:01:10.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 0x7f263400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f26494ea640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2634003970 0x7f2634003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2648ce9640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26340047c0 0x7f2634004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 -- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2634003970 msgr2=0x7f2634003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2634003970 0x7f2634003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 -- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26340047c0 msgr2=0x7f2634004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26340047c0 0x7f2634004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2649ceb640 1 -- 172.21.15.29:0/3620119919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634003520 con 0x7f2634005180 2023-10-14T17:01:10.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.891+0000 7f2648ce9640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f26340047c0 0x7f2634004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.892+0000 7f2649ceb640 1 -- 172.21.15.29:0/3620119919 reap_dead start 2023-10-14T17:01:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.892+0000 7f2649ceb640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 0x7f263400b980 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f263c00c240 tx=0x7f26340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25b4a7ec5826f831 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c005ad0 con 0x7f2634005180 2023-10-14T17:01:10.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c005c70 con 0x7f2634005180 2023-10-14T17:01:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f2643fff640 1 -- 172.21.15.29:0/3620119919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c02c8b0 con 0x7f2634005180 2023-10-14T17:01:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 -- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 msgr2=0x7f263400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 0x7f263400b980 secure :-1 s=READY pgs=1771 cs=0 l=1 rev1=1 crypto rx=0x7f263c00c240 tx=0x7f26340057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 -- 172.21.15.29:0/3620119919 shutdown_connections 2023-10-14T17:01:10.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 --2- 172.21.15.29:0/3620119919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2634005180 0x7f263400b980 unknown :-1 s=CLOSED pgs=1771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 -- 172.21.15.29:0/3620119919 >> 172.21.15.29:0/3620119919 conn(0x7f26440b0a50 msgr2=0x7f26440b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 -- 172.21.15.29:0/3620119919 shutdown_connections 2023-10-14T17:01:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.893+0000 7f264a4ec640 1 -- 172.21.15.29:0/3620119919 wait complete. 2023-10-14T17:01:10.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.895+0000 7f264a4ec640 1 Processor -- start 2023-10-14T17:01:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.896+0000 7f264a4ec640 1 -- start start 2023-10-14T17:01:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.896+0000 7f264a4ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2634005180 0x7f2644157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.896+0000 7f264a4ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f264409f5e0 0x7f26441f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.896+0000 7f264a4ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f264a4ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c2a20 con 0x7f26441f3ff0 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f264a4ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26441583c0 con 0x7f2634005180 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f264a4ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26440c2c70 con 0x7f264409f5e0 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39452/0 (socket says 172.21.15.29:39452) 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 -- 172.21.15.29:0/762114573 learned_addr learned my addr 172.21.15.29:0/762114573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:10.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 -- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f264409f5e0 msgr2=0x7f26441f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f264409f5e0 0x7f26441f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 -- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2634005180 msgr2=0x7f2644157e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f26494ea640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2634005180 0x7f2644157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2634005180 0x7f2644157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:10.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.897+0000 7f2649ceb640 1 -- 172.21.15.29:0/762114573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440c2f50 con 0x7f26441f3ff0 2023-10-14T17:01:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f2649ceb640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f263c00c330 tx=0x7f263c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c0049e0 con 0x7f26441f3ff0 2023-10-14T17:01:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f263c004b40 con 0x7f26441f3ff0 2023-10-14T17:01:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26440c3230 con 0x7f26441f3ff0 2023-10-14T17:01:10.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f263c02de80 con 0x7f26441f3ff0 2023-10-14T17:01:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.898+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26440c37f0 con 0x7f26441f3ff0 2023-10-14T17:01:10.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.899+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f264406e500 con 0x7f26441f3ff0 2023-10-14T17:01:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.900+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f263c032070 con 0x7f26441f3ff0 2023-10-14T17:01:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.902+0000 7f2641ffb640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 0x7f262806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.902+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f263c03c070 con 0x7f26441f3ff0 2023-10-14T17:01:10.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.902+0000 7f26494ea640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 0x7f262806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.903+0000 7f26494ea640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 0x7f262806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2638003a50 tx=0x7f263800b040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:10.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:10.907+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f263c075a40 con 0x7f26441f3ff0 2023-10-14T17:01:11.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.101+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f26440ce4f0 con 0x7f26441f3ff0 2023-10-14T17:01:11.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.102+0000 7f2641ffb640 1 -- 172.21.15.29:0/762114573 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v150) v1 ==== 93+0+10 (secure 0 0 0) 0x7f263c039020 con 0x7f26441f3ff0 2023-10-14T17:01:11.103 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:01:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 msgr2=0x7f262806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 0x7f262806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2638003a50 tx=0x7f263800b040 comp rx=0 tx=0).stop 2023-10-14T17:01:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 msgr2=0x7f26440c3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 secure :-1 s=READY pgs=1772 cs=0 l=1 rev1=1 crypto rx=0x7f263c00c330 tx=0x7f263c001a00 comp rx=0 tx=0).stop 2023-10-14T17:01:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 shutdown_connections 2023-10-14T17:01:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f262806af90 0x7f262806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f26441f3ff0 0x7f26440c3d10 unknown :-1 s=CLOSED pgs=1772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f264409f5e0 0x7f26441f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 --2- 172.21.15.29:0/762114573 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2634005180 0x7f2644157e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.105+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 >> 172.21.15.29:0/762114573 conn(0x7f26440b0a50 msgr2=0x7f264415f470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.106+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 shutdown_connections 2023-10-14T17:01:11.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.106+0000 7f264a4ec640 1 -- 172.21.15.29:0/762114573 wait complete. 2023-10-14T17:01:11.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T17:01:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.342+0000 7f8e9ffff640 1 Processor -- start 2023-10-14T17:01:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.342+0000 7f8e9ffff640 1 -- start start 2023-10-14T17:01:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.344+0000 7f8e9ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.344+0000 7f8e9ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ea00cda80 0x7f8ea015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.344+0000 7f8e9ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ea006c960 con 0x7f8ea00ad970 2023-10-14T17:01:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.344+0000 7f8e9ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ea015b9d0 con 0x7f8ea00cda80 2023-10-14T17:01:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.344+0000 7f8e9ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea015bcb0 con 0x7f8ea006c040 2023-10-14T17:01:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.345+0000 7f8e9effd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.345+0000 7f8e9f7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8ea00cda80 0x7f8ea015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.345+0000 7f8e9effd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:34342/0 (socket says 172.21.15.29:34342) 2023-10-14T17:01:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.345+0000 7f8e9effd640 1 -- 172.21.15.29:0/1703480391 learned_addr learned my addr 172.21.15.29:0/1703480391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:11.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.346+0000 7f8e9effd640 1 -- 172.21.15.29:0/1703480391 >> v2:172.21.15.186:3300/0 conn(0x7f8ea00cda80 msgr2=0x7f8ea015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:11.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.346+0000 7f8e9effd640 1 --2- 172.21.15.29:0/1703480391 >> v2:172.21.15.186:3300/0 conn(0x7f8ea00cda80 0x7f8ea015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.346+0000 7f8e9effd640 1 -- 172.21.15.29:0/1703480391 >> v1:172.21.15.186:6789/0 conn(0x7f8ea00ad970 legacy=0x7f8ea015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.346+0000 7f8e9effd640 1 -- 172.21.15.29:0/1703480391 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ea015bf90 con 0x7f8ea006c040 2023-10-14T17:01:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.347+0000 7f8e9effd640 1 --2- 172.21.15.29:0/1703480391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009870 tx=0x7f8e9002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b8c076d3e48a32 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.348+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1703480391 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90030840 con 0x7f8ea006c040 2023-10-14T17:01:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.348+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1703480391 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e900309a0 con 0x7f8ea006c040 2023-10-14T17:01:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.348+0000 7f8e9e7fc640 1 -- 172.21.15.29:0/1703480391 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e90030d30 con 0x7f8ea006c040 2023-10-14T17:01:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.348+0000 7f8e9ffff640 1 -- 172.21.15.29:0/1703480391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 msgr2=0x7f8ea006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.348+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/1703480391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8e90009870 tx=0x7f8e9002b980 comp rx=0 tx=0).stop 2023-10-14T17:01:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 -- 172.21.15.29:0/1703480391 shutdown_connections 2023-10-14T17:01:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/1703480391 >> v2:172.21.15.186:3300/0 conn(0x7f8ea00cda80 0x7f8ea015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/1703480391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea006c040 0x7f8ea006c420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 -- 172.21.15.29:0/1703480391 >> 172.21.15.29:0/1703480391 conn(0x7f8ea00ce6a0 msgr2=0x7f8ea00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 -- 172.21.15.29:0/1703480391 shutdown_connections 2023-10-14T17:01:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.349+0000 7f8e9ffff640 1 -- 172.21.15.29:0/1703480391 wait complete. 2023-10-14T17:01:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.351+0000 7f8e9ffff640 1 Processor -- start 2023-10-14T17:01:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.351+0000 7f8e9ffff640 1 -- start start 2023-10-14T17:01:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea00ad970 0x7f8ea0153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 0x7f8ea00afb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00ca9b0 con 0x7f8ea00b00d0 2023-10-14T17:01:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea00c64b0 con 0x7f8ea00cda80 2023-10-14T17:01:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea0153fa0 con 0x7f8ea00ad970 2023-10-14T17:01:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47130/0 (socket says 172.21.15.29:47130) 2023-10-14T17:01:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9f7fe640 1 -- 172.21.15.29:0/2969578263 learned_addr learned my addr 172.21.15.29:0/2969578263 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e9effd640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea00ad970 0x7f8ea0153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.353+0000 7f8e97fff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 0x7f8ea00afb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 -- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea00ad970 msgr2=0x7f8ea0153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea00ad970 0x7f8ea0153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 -- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 msgr2=0x7f8ea00afb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 0x7f8ea00afb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 -- 172.21.15.29:0/2969578263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e900094d0 con 0x7f8ea00b00d0 2023-10-14T17:01:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e97fff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 0x7f8ea00afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:01:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.354+0000 7f8e9f7fe640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009dd0 tx=0x7f8e9800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.355+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98014480 con 0x7f8ea00b00d0 2023-10-14T17:01:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.355+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8e9800f040 con 0x7f8ea00b00d0 2023-10-14T17:01:11.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.355+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea00c58a0 con 0x7f8ea00b00d0 2023-10-14T17:01:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.355+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e98014840 con 0x7f8ea00b00d0 2023-10-14T17:01:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.355+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea00c5e60 con 0x7f8ea00b00d0 2023-10-14T17:01:11.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.357+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea006b330 con 0x7f8ea00b00d0 2023-10-14T17:01:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.358+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8e980149a0 con 0x7f8ea00b00d0 2023-10-14T17:01:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.359+0000 7f8e977fe640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.359+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4929+0+0 (secure 0 0 0) 0x7f8e9808a340 con 0x7f8ea00b00d0 2023-10-14T17:01:11.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.360+0000 7f8e9effd640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.360+0000 7f8e9effd640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 0x7f8e8006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8e90036000 tx=0x7f8e90008030 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.364+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e980554c0 con 0x7f8ea00b00d0 2023-10-14T17:01:11.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.573+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8ea00c6660 con 0x7f8ea00b00d0 2023-10-14T17:01:11.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.574+0000 7f8e977fe640 1 -- 172.21.15.29:0/2969578263 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v150) v1 ==== 84+0+65 (secure 0 0 0) 0x7f8ea00c6660 con 0x7f8ea00b00d0 2023-10-14T17:01:11.575 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:11.575 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:01:11.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:11.576 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T17:01:11.576 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T17:01:11.576 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:11.576 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:01:11.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 msgr2=0x7f8e8006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 0x7f8e8006d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f8e90036000 tx=0x7f8e90008030 comp rx=0 tx=0).stop 2023-10-14T17:01:11.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 msgr2=0x7f8ea00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 secure :-1 s=READY pgs=1773 cs=0 l=1 rev1=1 crypto rx=0x7f8e98009dd0 tx=0x7f8e9800c490 comp rx=0 tx=0).stop 2023-10-14T17:01:11.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 shutdown_connections 2023-10-14T17:01:11.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f8e8006af90 0x7f8e8006d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ea00b00d0 0x7f8ea00ac890 unknown :-1 s=CLOSED pgs=1773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ea00cda80 0x7f8ea00afb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 --2- 172.21.15.29:0/2969578263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ea00ad970 0x7f8ea0153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.577+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 >> 172.21.15.29:0/2969578263 conn(0x7f8ea00ce6a0 msgr2=0x7f8ea00cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.578+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 shutdown_connections 2023-10-14T17:01:11.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.578+0000 7f8e9ffff640 1 -- 172.21.15.29:0/2969578263 wait complete. 2023-10-14T17:01:11.597 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T17:01:11.598 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T17:01:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.832+0000 7fe965421640 1 Processor -- start 2023-10-14T17:01:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.832+0000 7fe965421640 1 -- start start 2023-10-14T17:01:11.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe965421640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe96006d150 0x7fe9600b2590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe965421640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe96015b580 con 0x7fe960159ae0 2023-10-14T17:01:11.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe965421640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9600b2f00 con 0x7fe9600b2ad0 2023-10-14T17:01:11.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe965421640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe96015a970 con 0x7fe96006d150 2023-10-14T17:01:11.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe95effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe960159ae0 0x7fe960159ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53958/0 (socket says 172.21.15.29:53958) 2023-10-14T17:01:11.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.834+0000 7fe95effd640 1 -- 172.21.15.29:0/4246285898 learned_addr learned my addr 172.21.15.29:0/4246285898 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:11.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95e7fc640 1 --2- 172.21.15.29:0/4246285898 >> v2:172.21.15.186:3300/0 conn(0x7fe96006d150 0x7fe9600b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471334741 0 0) 0x7fe96015b580 con 0x7fe960159ae0 2023-10-14T17:01:11.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe948005730 con 0x7fe960159ae0 2023-10-14T17:01:11.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591750880 0 0) 0x7fe948005730 con 0x7fe960159ae0 2023-10-14T17:01:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 >> v2:172.21.15.186:3300/0 conn(0x7fe96006d150 msgr2=0x7fe9600b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 --2- 172.21.15.29:0/4246285898 >> v2:172.21.15.186:3300/0 conn(0x7fe96006d150 0x7fe9600b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 >> v1:172.21.15.186:6789/0 conn(0x7fe9600b2ad0 legacy=0x7fe96015ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe96015ac50 con 0x7fe960159ae0 2023-10-14T17:01:11.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe9500019f0 con 0x7fe960159ae0 2023-10-14T17:01:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 >> v1:172.21.15.29:6789/0 conn(0x7fe960159ae0 legacy=0x7fe960159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe948005b80 0x7fe948005fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9480069d0 0x7fe948006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 0x7fe94800db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe96015a970 con 0x7fe948007390 2023-10-14T17:01:11.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe94800e260 con 0x7fe948005b80 2023-10-14T17:01:11.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.835+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe94800e420 con 0x7fe9480069d0 2023-10-14T17:01:11.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 0x7fe94800db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95effd640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe948005b80 0x7fe948005fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95e7fc640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9480069d0 0x7fe948006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 -- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9480069d0 msgr2=0x7fe948006e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:11.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9480069d0 0x7fe948006e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 -- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe948005b80 msgr2=0x7fe948005fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:11.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe948005b80 0x7fe948005fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.836+0000 7fe95f7fe640 1 -- 172.21.15.29:0/4246285898 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe948005730 con 0x7fe948007390 2023-10-14T17:01:11.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.837+0000 7fe95f7fe640 1 -- 172.21.15.29:0/4246285898 reap_dead start 2023-10-14T17:01:11.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.837+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 0x7fe94800db90 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe958001750 tx=0x7fe948007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a6efaed2e7cd064 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580052d0 con 0x7fe948007390 2023-10-14T17:01:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe95802e050 con 0x7fe948007390 2023-10-14T17:01:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe95dffb640 1 -- 172.21.15.29:0/4246285898 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958003830 con 0x7fe948007390 2023-10-14T17:01:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 -- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 msgr2=0x7fe94800db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 0x7fe94800db90 secure :-1 s=READY pgs=1775 cs=0 l=1 rev1=1 crypto rx=0x7fe958001750 tx=0x7fe948007a00 comp rx=0 tx=0).stop 2023-10-14T17:01:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 -- 172.21.15.29:0/4246285898 shutdown_connections 2023-10-14T17:01:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 --2- 172.21.15.29:0/4246285898 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe948007390 0x7fe94800db90 unknown :-1 s=CLOSED pgs=1775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 -- 172.21.15.29:0/4246285898 >> 172.21.15.29:0/4246285898 conn(0x7fe960069da0 msgr2=0x7fe96006a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.838+0000 7fe965421640 1 -- 172.21.15.29:0/4246285898 shutdown_connections 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.839+0000 7fe965421640 1 -- 172.21.15.29:0/4246285898 wait complete. 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.840+0000 7fe965421640 1 Processor -- start 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.840+0000 7fe965421640 1 -- start start 2023-10-14T17:01:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe948007390 0x7fe9600ce2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 0x7fe9601500b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601505f0 con 0x7fe960151c80 2023-10-14T17:01:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe960070c30 con 0x7fe96014fc30 2023-10-14T17:01:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.841+0000 7fe965421640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9601518b0 con 0x7fe948007390 2023-10-14T17:01:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47142/0 (socket says 172.21.15.29:47142) 2023-10-14T17:01:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 -- 172.21.15.29:0/98082937 learned_addr learned my addr 172.21.15.29:0/98082937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95effd640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe948007390 0x7fe9600ce2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95e7fc640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 0x7fe9601500b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 -- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe948007390 msgr2=0x7fe9600ce2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe948007390 0x7fe9600ce2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 -- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 msgr2=0x7fe9601500b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:11.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 0x7fe9601500b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:11.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95f7fe640 1 -- 172.21.15.29:0/98082937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe960151420 con 0x7fe960151c80 2023-10-14T17:01:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.842+0000 7fe95e7fc640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 0x7fe9601500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe95f7fe640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe95802bd00 tx=0x7fe958001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9580052e0 con 0x7fe960151c80 2023-10-14T17:01:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe958005460 con 0x7fe960151c80 2023-10-14T17:01:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe958007270 con 0x7fe960151c80 2023-10-14T17:01:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe960150810 con 0x7fe960151c80 2023-10-14T17:01:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.843+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe960150dd0 con 0x7fe960151c80 2023-10-14T17:01:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.844+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9601515b0 con 0x7fe960151c80 2023-10-14T17:01:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.845+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe9580073d0 con 0x7fe960151c80 2023-10-14T17:01:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.846+0000 7fe9477fe640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 0x7fe94c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.846+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 4929+0+0 (secure 0 0 0) 0x7fe9580aa490 con 0x7fe960151c80 2023-10-14T17:01:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.846+0000 7fe95effd640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 0x7fe94c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.847+0000 7fe95effd640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 0x7fe94c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe950002230 tx=0x7fe950009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:11.850+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe958075630 con 0x7fe960151c80 2023-10-14T17:01:12.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.046+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fe96004e160 con 0x7fe960151c80 2023-10-14T17:01:12.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.634+0000 7fe9477fe640 1 -- 172.21.15.29:0/98082937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v151) v1 ==== 138+0+0 (secure 0 0 0) 0x7fe958078c50 con 0x7fe960151c80 2023-10-14T17:01:12.635 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T17:01:12.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 msgr2=0x7fe94c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 0x7fe94c06d480 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe950002230 tx=0x7fe950009040 comp rx=0 tx=0).stop 2023-10-14T17:01:12.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 msgr2=0x7fe960152100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 secure :-1 s=READY pgs=1776 cs=0 l=1 rev1=1 crypto rx=0x7fe95802bd00 tx=0x7fe958001120 comp rx=0 tx=0).stop 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 shutdown_connections 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe94c06afc0 0x7fe94c06d480 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe960151c80 0x7fe960152100 unknown :-1 s=CLOSED pgs=1776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe96014fc30 0x7fe9601500b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 --2- 172.21.15.29:0/98082937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe948007390 0x7fe9600ce2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 >> 172.21.15.29:0/98082937 conn(0x7fe960069da0 msgr2=0x7fe96006c4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:12.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 shutdown_connections 2023-10-14T17:01:12.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.638+0000 7fe965421640 1 -- 172.21.15.29:0/98082937 wait complete. 2023-10-14T17:01:12.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T17:01:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.886+0000 7fc25ffff640 1 Processor -- start 2023-10-14T17:01:12.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.887+0000 7fc25ffff640 1 -- start start 2023-10-14T17:01:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc2600b1690 0x7fc2600b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2600b4f00 con 0x7fc26015cb50 2023-10-14T17:01:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2600cdbf0 con 0x7fc2600b4ab0 2023-10-14T17:01:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2600cde90 con 0x7fc2600b1690 2023-10-14T17:01:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc26015cb50 0x7fc260161180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53968/0 (socket says 172.21.15.29:53968) 2023-10-14T17:01:12.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.888+0000 7fc25effd640 1 -- 172.21.15.29:0/2092521578 learned_addr learned my addr 172.21.15.29:0/2092521578 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25e7fc640 1 --2- 172.21.15.29:0/2092521578 >> v2:172.21.15.186:3300/0 conn(0x7fc2600b1690 0x7fc2600b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2330928371 0 0) 0x7fc250001490 con 0x7fc26015cb50 2023-10-14T17:01:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc24c003520 con 0x7fc26015cb50 2023-10-14T17:01:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1465393680 0 0) 0x7fc24c003520 con 0x7fc26015cb50 2023-10-14T17:01:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 >> v2:172.21.15.186:3300/0 conn(0x7fc2600b1690 msgr2=0x7fc2600b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 --2- 172.21.15.29:0/2092521578 >> v2:172.21.15.186:3300/0 conn(0x7fc2600b1690 0x7fc2600b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 >> v1:172.21.15.186:6789/0 conn(0x7fc2600b4ab0 legacy=0x7fc2600d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2600ce170 con 0x7fc26015cb50 2023-10-14T17:01:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc250001b80 con 0x7fc26015cb50 2023-10-14T17:01:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 >> v1:172.21.15.29:6789/0 conn(0x7fc26015cb50 legacy=0x7fc260161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.889+0000 7fc25dffb640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc24c003970 0x7fc24c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25dffb640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0047c0 0x7fc24c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25dffb640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 0x7fc24c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600cde90 con 0x7fc24c0052a0 2023-10-14T17:01:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c00c170 con 0x7fc24c0047c0 2023-10-14T17:01:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc24c00c330 con 0x7fc24c003970 2023-10-14T17:01:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 0x7fc24c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25effd640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc24c003970 0x7fc24c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25e7fc640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0047c0 0x7fc24c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 -- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc24c003970 msgr2=0x7fc24c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc24c003970 0x7fc24c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 -- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0047c0 msgr2=0x7fc24c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0047c0 0x7fc24c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.890+0000 7fc25f7fe640 1 -- 172.21.15.29:0/2092521578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc24c003520 con 0x7fc24c0052a0 2023-10-14T17:01:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.891+0000 7fc25f7fe640 1 -- 172.21.15.29:0/2092521578 reap_dead start 2023-10-14T17:01:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 0x7fc24c00b980 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc258007fd0 tx=0x7fc24c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8588e71227d355b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2580052d0 con 0x7fc24c0052a0 2023-10-14T17:01:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25802e050 con 0x7fc24c0052a0 2023-10-14T17:01:12.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25dffb640 1 -- 172.21.15.29:0/2092521578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc258003640 con 0x7fc24c0052a0 2023-10-14T17:01:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25ffff640 1 -- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 msgr2=0x7fc24c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.892+0000 7fc25ffff640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 0x7fc24c00b980 secure :-1 s=READY pgs=1778 cs=0 l=1 rev1=1 crypto rx=0x7fc258007fd0 tx=0x7fc24c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.893+0000 7fc25ffff640 1 -- 172.21.15.29:0/2092521578 shutdown_connections 2023-10-14T17:01:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.893+0000 7fc25ffff640 1 --2- 172.21.15.29:0/2092521578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc24c0052a0 0x7fc24c00b980 unknown :-1 s=CLOSED pgs=1778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.893+0000 7fc25ffff640 1 -- 172.21.15.29:0/2092521578 >> 172.21.15.29:0/2092521578 conn(0x7fc2600cf570 msgr2=0x7fc2600cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.893+0000 7fc25ffff640 1 -- 172.21.15.29:0/2092521578 shutdown_connections 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.893+0000 7fc25ffff640 1 -- 172.21.15.29:0/2092521578 wait complete. 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.894+0000 7fc25ffff640 1 Processor -- start 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.895+0000 7fc25ffff640 1 -- start start 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0052a0 0x7fc26015ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc26015f7e0 0x7fc2600b2da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2600b3760 con 0x7fc2600b32e0 2023-10-14T17:01:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26015b210 con 0x7fc24c0052a0 2023-10-14T17:01:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc26015fc60 con 0x7fc26015f7e0 2023-10-14T17:01:12.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47162/0 (socket says 172.21.15.29:47162) 2023-10-14T17:01:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 -- 172.21.15.29:0/1789048890 learned_addr learned my addr 172.21.15.29:0/1789048890 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25e7fc640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc26015f7e0 0x7fc2600b2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25effd640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0052a0 0x7fc26015ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 -- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc26015f7e0 msgr2=0x7fc2600b2da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc26015f7e0 0x7fc2600b2da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 -- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0052a0 msgr2=0x7fc26015ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0052a0 0x7fc26015ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.896+0000 7fc25f7fe640 1 -- 172.21.15.29:0/1789048890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26015ea20 con 0x7fc2600b32e0 2023-10-14T17:01:12.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc25f7fe640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc258001860 tx=0x7fc258001170 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25803d600 con 0x7fc2600b32e0 2023-10-14T17:01:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc25803d760 con 0x7fc2600b32e0 2023-10-14T17:01:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc25803db10 con 0x7fc2600b32e0 2023-10-14T17:01:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26015ed00 con 0x7fc2600b32e0 2023-10-14T17:01:12.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.897+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26015f2c0 con 0x7fc2600b32e0 2023-10-14T17:01:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.898+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26004e9b0 con 0x7fc2600b32e0 2023-10-14T17:01:12.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.900+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc258032400 con 0x7fc2600b32e0 2023-10-14T17:01:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.901+0000 7fc2577fe640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 0x7fc24006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.901+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5299+0+0 (secure 0 0 0) 0x7fc2580aa8d0 con 0x7fc2600b32e0 2023-10-14T17:01:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.901+0000 7fc25effd640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 0x7fc24006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.902+0000 7fc25effd640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 0x7fc24006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2500021a0 tx=0x7fc250009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:12.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:12.905+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc258075b70 con 0x7fc2600b32e0 2023-10-14T17:01:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.102+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fc2600d0790 con 0x7fc2600b32e0 2023-10-14T17:01:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.642+0000 7fc2577fe640 1 -- 172.21.15.29:0/1789048890 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v152) v1 ==== 170+0+0 (secure 0 0 0) 0x7fc2580420d0 con 0x7fc2600b32e0 2023-10-14T17:01:13.643 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T17:01:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.645+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 msgr2=0x7fc24006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:13.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.645+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 0x7fc24006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc2500021a0 tx=0x7fc250009040 comp rx=0 tx=0).stop 2023-10-14T17:01:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.645+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 msgr2=0x7fc260159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.645+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 secure :-1 s=READY pgs=1779 cs=0 l=1 rev1=1 crypto rx=0x7fc258001860 tx=0x7fc258001170 comp rx=0 tx=0).stop 2023-10-14T17:01:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 shutdown_connections 2023-10-14T17:01:13.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc24006af90 0x7fc24006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2600b32e0 0x7fc260159410 unknown :-1 s=CLOSED pgs=1779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc26015f7e0 0x7fc2600b2da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 --2- 172.21.15.29:0/1789048890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc24c0052a0 0x7fc26015ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 >> 172.21.15.29:0/1789048890 conn(0x7fc2600cf570 msgr2=0x7fc2600cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:13.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 shutdown_connections 2023-10-14T17:01:13.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.646+0000 7fc25ffff640 1 -- 172.21.15.29:0/1789048890 wait complete. 2023-10-14T17:01:13.682 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T17:01:13.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.875+0000 7fee1ea78640 1 Processor -- start 2023-10-14T17:01:13.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.875+0000 7fee1ea78640 1 -- start start 2023-10-14T17:01:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.876+0000 7fee1ea78640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee18159950 0x7fee1815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.876+0000 7fee1ea78640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 0x7fee180708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1ea78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee1806c100 0x7fee1806c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1ea78640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee18070e20 con 0x7fee18159950 2023-10-14T17:01:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1ea78640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee1815e490 con 0x7fee18070450 2023-10-14T17:01:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1ea78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee1815aef0 con 0x7fee1806c100 2023-10-14T17:01:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee18159950 0x7fee1815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fee18159950 0x7fee1815a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47176/0 (socket says 172.21.15.29:47176) 2023-10-14T17:01:13.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 -- 172.21.15.29:0/4148354897 learned_addr learned my addr 172.21.15.29:0/4148354897 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1d275640 1 --2- 172.21.15.29:0/4148354897 >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 0x7fee180708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 -- 172.21.15.29:0/4148354897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee1806c100 msgr2=0x7fee1806c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 --2- 172.21.15.29:0/4148354897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee1806c100 0x7fee1806c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 -- 172.21.15.29:0/4148354897 >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 msgr2=0x7fee180708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:13.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 --2- 172.21.15.29:0/4148354897 >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 0x7fee180708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1da76640 1 -- 172.21.15.29:0/4148354897 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1815d910 con 0x7fee18159950 2023-10-14T17:01:13.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.877+0000 7fee1d275640 1 --2- 172.21.15.29:0/4148354897 >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 0x7fee180708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:13.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.879+0000 7fee1da76640 1 --2- 172.21.15.29:0/4148354897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee18159950 0x7fee1815a9b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fee180cab70 tx=0x7fee1402e030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2ad1de81c5c17b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:13.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.879+0000 7fee1ca74640 1 -- 172.21.15.29:0/4148354897 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1402ede0 con 0x7fee18159950 2023-10-14T17:01:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.879+0000 7fee1ca74640 1 -- 172.21.15.29:0/4148354897 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee14036440 con 0x7fee18159950 2023-10-14T17:01:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.879+0000 7fee1ca74640 1 -- 172.21.15.29:0/4148354897 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee140367d0 con 0x7fee18159950 2023-10-14T17:01:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 -- 172.21.15.29:0/4148354897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee18159950 msgr2=0x7fee1815a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 --2- 172.21.15.29:0/4148354897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee18159950 0x7fee1815a9b0 secure :-1 s=READY pgs=1780 cs=0 l=1 rev1=1 crypto rx=0x7fee180cab70 tx=0x7fee1402e030 comp rx=0 tx=0).stop 2023-10-14T17:01:13.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 -- 172.21.15.29:0/4148354897 shutdown_connections 2023-10-14T17:01:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 --2- 172.21.15.29:0/4148354897 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee1806c100 0x7fee1806c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 --2- 172.21.15.29:0/4148354897 >> v2:172.21.15.186:3300/0 conn(0x7fee18070450 0x7fee180708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 --2- 172.21.15.29:0/4148354897 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee18159950 0x7fee1815a9b0 unknown :-1 s=CLOSED pgs=1780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 -- 172.21.15.29:0/4148354897 >> 172.21.15.29:0/4148354897 conn(0x7fee180ce6a0 msgr2=0x7fee180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:13.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 -- 172.21.15.29:0/4148354897 shutdown_connections 2023-10-14T17:01:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.880+0000 7fee1ea78640 1 -- 172.21.15.29:0/4148354897 wait complete. 2023-10-14T17:01:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.882+0000 7fee1ea78640 1 Processor -- start 2023-10-14T17:01:13.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.882+0000 7fee1ea78640 1 -- start start 2023-10-14T17:01:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee18070450 0x7fee180b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee18159950 0x7fee180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee180b3e70 con 0x7fee1806c100 2023-10-14T17:01:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee180b2dd0 con 0x7fee18070450 2023-10-14T17:01:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1ea78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee180b30b0 con 0x7fee18159950 2023-10-14T17:01:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1da76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1da76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47192/0 (socket says 172.21.15.29:47192) 2023-10-14T17:01:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1da76640 1 -- 172.21.15.29:0/3134821499 learned_addr learned my addr 172.21.15.29:0/3134821499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1e277640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee18159950 0x7fee180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.883+0000 7fee1da76640 1 -- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee18159950 msgr2=0x7fee180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1da76640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee18159950 0x7fee180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1da76640 1 -- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee18070450 msgr2=0x7fee180b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1da76640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee18070450 0x7fee180b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1da76640 1 -- 172.21.15.29:0/3134821499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1400b770 con 0x7fee1806c100 2023-10-14T17:01:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1da76640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fee1402e070 tx=0x7fee140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1403d760 con 0x7fee1806c100 2023-10-14T17:01:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee1403d8c0 con 0x7fee1806c100 2023-10-14T17:01:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee1403dc70 con 0x7fee1806c100 2023-10-14T17:01:13.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee180b3390 con 0x7fee1806c100 2023-10-14T17:01:13.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.884+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee180b3950 con 0x7fee1806c100 2023-10-14T17:01:13.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.885+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee1804e9b0 con 0x7fee1806c100 2023-10-14T17:01:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.887+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fee14049360 con 0x7fee1806c100 2023-10-14T17:01:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.888+0000 7fee0e7fc640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 0x7fedfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.888+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5681+0+0 (secure 0 0 0) 0x7fee140ab720 con 0x7fee1806c100 2023-10-14T17:01:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.888+0000 7fee1d275640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 0x7fedfc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.889+0000 7fee1d275640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 0x7fedfc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee180ad450 tx=0x7fee080083d0 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:13.894+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fee14076760 con 0x7fee1806c100 2023-10-14T17:01:14.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.131+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fee180b4620 con 0x7fee1806c100 2023-10-14T17:01:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.696+0000 7fee0e7fc640 1 -- 172.21.15.29:0/3134821499 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 18 and data pool 19 v71) v1 ==== 174+0+0 (secure 0 0 0) 0x7fee1403b0c0 con 0x7fee1806c100 2023-10-14T17:01:14.699 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 18 and data pool 19 2023-10-14T17:01:14.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.706+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 msgr2=0x7fedfc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.706+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 0x7fedfc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fee180ad450 tx=0x7fee080083d0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.706+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 msgr2=0x7fee1814da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.706+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 secure :-1 s=READY pgs=1781 cs=0 l=1 rev1=1 crypto rx=0x7fee1402e070 tx=0x7fee140046c0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 shutdown_connections 2023-10-14T17:01:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fedfc06af90 0x7fedfc06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fee18159950 0x7fee180ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fee18070450 0x7fee180b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 --2- 172.21.15.29:0/3134821499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fee1806c100 0x7fee1814da30 unknown :-1 s=CLOSED pgs=1781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.707+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 >> 172.21.15.29:0/3134821499 conn(0x7fee180ce6a0 msgr2=0x7fee1815d0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.710+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 shutdown_connections 2023-10-14T17:01:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.710+0000 7fee1ea78640 1 -- 172.21.15.29:0/3134821499 wait complete. 2023-10-14T17:01:14.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:01:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.925+0000 7f3e3ed0f640 1 Processor -- start 2023-10-14T17:01:14.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.925+0000 7f3e3ed0f640 1 -- start start 2023-10-14T17:01:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3ed0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e380b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3ed0f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3e380adf30 0x7f3e380b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3ed0f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38071450 con 0x7f3e380adb50 2023-10-14T17:01:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3ed0f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e380cc600 con 0x7f3e380adf30 2023-10-14T17:01:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3ed0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380cc8e0 con 0x7f3e38070fd0 2023-10-14T17:01:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3caab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3e380adb50 0x7f3e3806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53972/0 (socket says 172.21.15.29:53972) 2023-10-14T17:01:14.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3caab640 1 -- 172.21.15.29:0/3122530675 learned_addr learned my addr 172.21.15.29:0/3122530675 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e3d2ac640 1 --2- 172.21.15.29:0/3122530675 >> v2:172.21.15.186:3300/0 conn(0x7f3e380adf30 0x7f3e380b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.927+0000 7f3e37fff640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e380b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728397851 0 0) 0x7f3e38071450 con 0x7f3e380adb50 2023-10-14T17:01:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e28003520 con 0x7f3e380adb50 2023-10-14T17:01:14.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 464030391 0 0) 0x7f3e28003520 con 0x7f3e380adb50 2023-10-14T17:01:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 msgr2=0x7f3e380b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e380b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 >> v2:172.21.15.186:3300/0 conn(0x7f3e380adf30 msgr2=0x7f3e380b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 --2- 172.21.15.29:0/3122530675 >> v2:172.21.15.186:3300/0 conn(0x7f3e380adf30 0x7f3e380b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380ccbc0 con 0x7f3e380adb50 2023-10-14T17:01:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3e2c001670 con 0x7f3e380adb50 2023-10-14T17:01:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 >> v1:172.21.15.29:6789/0 conn(0x7f3e380adb50 legacy=0x7f3e3806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e28003970 0x7f3e28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e280047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28004d00 0x7f3e28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380cc600 con 0x7f3e28003970 2023-10-14T17:01:14.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380cc8e0 con 0x7f3e28004d00 2023-10-14T17:01:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.928+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2800b980 con 0x7f3e38070fd0 2023-10-14T17:01:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e28003970 0x7f3e28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e37fff640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e280047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3d2ac640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28004d00 0x7f3e28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 -- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 msgr2=0x7f3e280047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e280047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 -- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28004d00 msgr2=0x7f3e28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28004d00 0x7f3e28005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3caab640 1 -- 172.21.15.29:0/3122530675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e28003520 con 0x7f3e28003970 2023-10-14T17:01:14.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e3d2ac640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28004d00 0x7f3e28005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:01:14.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.929+0000 7f3e37fff640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e38070fd0 0x7f3e280047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:01:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.931+0000 7f3e3caab640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e28003970 0x7f3e28003d90 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c160 tx=0x7f3e280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faa69c60b632aa48 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.931+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c005ad0 con 0x7f3e28003970 2023-10-14T17:01:14.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.931+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2c005c30 con 0x7f3e28003970 2023-10-14T17:01:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.931+0000 7f3e377fe640 1 -- 172.21.15.29:0/3122530675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2c02c810 con 0x7f3e28003970 2023-10-14T17:01:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/3122530675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e28003970 msgr2=0x7f3e28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/3122530675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e28003970 0x7f3e28003d90 secure :-1 s=READY pgs=1783 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c00c160 tx=0x7f3e280057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3d2ac640 1 -- 172.21.15.29:0/3122530675 reap_dead start 2023-10-14T17:01:14.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/3122530675 shutdown_connections 2023-10-14T17:01:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/3122530675 >> 172.21.15.29:0/3122530675 conn(0x7f3e380cb410 msgr2=0x7f3e380cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/3122530675 shutdown_connections 2023-10-14T17:01:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.932+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/3122530675 wait complete. 2023-10-14T17:01:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.934+0000 7f3e3ed0f640 1 Processor -- start 2023-10-14T17:01:14.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.934+0000 7f3e3ed0f640 1 -- start start 2023-10-14T17:01:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 0x7f3e3815f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e381fa3d0 0x7f3e381fa850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3815fe00 con 0x7f3e380be420 2023-10-14T17:01:14.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e381fad90 con 0x7f3e28003970 2023-10-14T17:01:14.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.935+0000 7f3e3ed0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e380c1680 con 0x7f3e381fa3d0 2023-10-14T17:01:14.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47220/0 (socket says 172.21.15.29:47220) 2023-10-14T17:01:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 -- 172.21.15.29:0/935424650 learned_addr learned my addr 172.21.15.29:0/935424650 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e3caab640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 0x7f3e3815f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e3d2ac640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e381fa3d0 0x7f3e381fa850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 -- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e381fa3d0 msgr2=0x7f3e381fa850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e381fa3d0 0x7f3e381fa850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 -- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 msgr2=0x7f3e3815f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:14.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 0x7f3e3815f8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:14.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 -- 172.21.15.29:0/935424650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e380c11f0 con 0x7f3e380be420 2023-10-14T17:01:14.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e3caab640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 0x7f3e3815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.936+0000 7f3e37fff640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f3e24011600 tx=0x7f3e24011ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.937+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400f920 con 0x7f3e380be420 2023-10-14T17:01:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.937+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e2400fa80 con 0x7f3e380be420 2023-10-14T17:01:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.937+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e2400fe10 con 0x7f3e380be420 2023-10-14T17:01:14.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.937+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e380c05e0 con 0x7f3e380be420 2023-10-14T17:01:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.937+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e380c0ba0 con 0x7f3e380be420 2023-10-14T17:01:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.938+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e380c1380 con 0x7f3e380be420 2023-10-14T17:01:14.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.939+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3e24015460 con 0x7f3e380be420 2023-10-14T17:01:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.940+0000 7f3e357fa640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 0x7f3e1c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.940+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3e24089fe0 con 0x7f3e380be420 2023-10-14T17:01:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.941+0000 7f3e3caab640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 0x7f3e1c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:14.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.941+0000 7f3e3caab640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 0x7f3e1c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c001050 tx=0x7f3e2c007000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:14.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:14.945+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e24054f60 con 0x7f3e380be420 2023-10-14T17:01:15.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.138+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e380c4780 con 0x7f3e380be420 2023-10-14T17:01:15.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.141+0000 7f3e357fa640 1 -- 172.21.15.29:0/935424650 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16695 (secure 0 0 0) 0x7f3e24022090 con 0x7f3e380be420 2023-10-14T17:01:15.142 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:15.142 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":153,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:01:14.132944+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:01:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.143+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 msgr2=0x7f3e1c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.143+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 0x7f3e1c06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3e2c001050 tx=0x7f3e2c007000 comp rx=0 tx=0).stop 2023-10-14T17:01:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.143+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 msgr2=0x7f3e380bfd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.143+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 secure :-1 s=READY pgs=1784 cs=0 l=1 rev1=1 crypto rx=0x7f3e24011600 tx=0x7f3e24011ac0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 shutdown_connections 2023-10-14T17:01:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f3e1c06af90 0x7f3e1c06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e381fa3d0 0x7f3e381fa850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e380be420 0x7f3e380bfd60 unknown :-1 s=CLOSED pgs=1784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 --2- 172.21.15.29:0/935424650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e28003970 0x7f3e3815f8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 >> 172.21.15.29:0/935424650 conn(0x7f3e380cb410 msgr2=0x7f3e380ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 shutdown_connections 2023-10-14T17:01:15.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.144+0000 7f3e3ed0f640 1 -- 172.21.15.29:0/935424650 wait complete. 2023-10-14T17:01:15.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:01:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.383+0000 7f88c5dd0640 1 Processor -- start 2023-10-14T17:01:15.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.383+0000 7f88c5dd0640 1 -- start start 2023-10-14T17:01:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.384+0000 7f88c5dd0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.384+0000 7f88c5dd0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c00d0be0 0x7f88c00cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.384+0000 7f88c5dd0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c015d150 con 0x7f88c015cd70 2023-10-14T17:01:15.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88c5dd0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c00d1180 con 0x7f88c00d0be0 2023-10-14T17:01:15.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88c5dd0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00d1460 con 0x7f88c00b8480 2023-10-14T17:01:15.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88c55cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c00d0be0 0x7f88c00cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88c4dce640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f88c015cd70 0x7f88c00d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:53986/0 (socket says 172.21.15.29:53986) 2023-10-14T17:01:15.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88c4dce640 1 -- 172.21.15.29:0/2464064644 learned_addr learned my addr 172.21.15.29:0/2464064644 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.385+0000 7f88bffff640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.386+0000 7f88c55cf640 1 -- 172.21.15.29:0/2464064644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 msgr2=0x7f88c00cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.386+0000 7f88c55cf640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c00cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.386+0000 7f88c55cf640 1 -- 172.21.15.29:0/2464064644 >> v1:172.21.15.29:6789/0 conn(0x7f88c015cd70 legacy=0x7f88c00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.386+0000 7f88c55cf640 1 -- 172.21.15.29:0/2464064644 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c00d1740 con 0x7f88c00d0be0 2023-10-14T17:01:15.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.386+0000 7f88bf7fe640 1 -- 172.21.15.29:0/2464064644 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460375051 0 0) 0x7f88c015d150 con 0x7f88c015cd70 2023-10-14T17:01:15.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.387+0000 7f88c55cf640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c00d0be0 0x7f88c00cdd00 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=574105b2feb79ed7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.387+0000 7f88bf7fe640 1 -- 172.21.15.29:0/2464064644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b802cc60 con 0x7f88c00d0be0 2023-10-14T17:01:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.387+0000 7f88bf7fe640 1 -- 172.21.15.29:0/2464064644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b802cdc0 con 0x7f88c00d0be0 2023-10-14T17:01:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.387+0000 7f88bf7fe640 1 -- 172.21.15.29:0/2464064644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8034660 con 0x7f88c00d0be0 2023-10-14T17:01:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 -- 172.21.15.29:0/2464064644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c00d0be0 msgr2=0x7f88c00cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c00d0be0 0x7f88c00cdd00 secure :-1 s=READY pgs=1786 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b802beb0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 -- 172.21.15.29:0/2464064644 shutdown_connections 2023-10-14T17:01:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c00d0be0 0x7f88c00cdd00 unknown :-1 s=CLOSED pgs=1786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/2464064644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c00cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 -- 172.21.15.29:0/2464064644 >> 172.21.15.29:0/2464064644 conn(0x7f88c00cf570 msgr2=0x7f88c00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 -- 172.21.15.29:0/2464064644 shutdown_connections 2023-10-14T17:01:15.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.388+0000 7f88c5dd0640 1 -- 172.21.15.29:0/2464064644 wait complete. 2023-10-14T17:01:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.389+0000 7f88c5dd0640 1 Processor -- start 2023-10-14T17:01:15.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.390+0000 7f88c5dd0640 1 -- start start 2023-10-14T17:01:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c0155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c00d0be0 0x7f88c00c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00ca100 con 0x7f88c015cd70 2023-10-14T17:01:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00ca3a0 con 0x7f88c00d0be0 2023-10-14T17:01:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c5dd0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c00ca680 con 0x7f88c00b8480 2023-10-14T17:01:15.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47238/0 (socket says 172.21.15.29:47238) 2023-10-14T17:01:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 -- 172.21.15.29:0/4138955413 learned_addr learned my addr 172.21.15.29:0/4138955413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88bffff640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c00d0be0 0x7f88c00c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c4dce640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c0155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 -- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 msgr2=0x7f88c0155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c0155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 -- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c00d0be0 msgr2=0x7f88c00c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c00d0be0 0x7f88c00c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.391+0000 7f88c55cf640 1 -- 172.21.15.29:0/4138955413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b8009a70 con 0x7f88c015cd70 2023-10-14T17:01:15.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88c55cf640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8003b90 con 0x7f88c015cd70 2023-10-14T17:01:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8003d10 con 0x7f88c015cd70 2023-10-14T17:01:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b8004270 con 0x7f88c015cd70 2023-10-14T17:01:15.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c00ca960 con 0x7f88c015cd70 2023-10-14T17:01:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.392+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c00caf20 con 0x7f88c015cd70 2023-10-14T17:01:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.393+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c004e9b0 con 0x7f88c015cd70 2023-10-14T17:01:15.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.394+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f88b803b050 con 0x7f88c015cd70 2023-10-14T17:01:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.395+0000 7f88bd7fa640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 0x7f88a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.396+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(153..153 src has 1..153) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f88b80abcf0 con 0x7f88c015cd70 2023-10-14T17:01:15.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.396+0000 7f88c4dce640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 0x7f88a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.396+0000 7f88c4dce640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 0x7f88a406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88b4007fd0 tx=0x7f88b4009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.400+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88b80b0070 con 0x7f88c015cd70 2023-10-14T17:01:15.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.597+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c00cb110 con 0x7f88c015cd70 2023-10-14T17:01:15.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.599+0000 7f88bd7fa640 1 -- 172.21.15.29:0/4138955413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v153) v1 ==== 74+0+16695 (secure 0 0 0) 0x7f88b8039090 con 0x7f88c015cd70 2023-10-14T17:01:15.600 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:15.600 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":153,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:01:14.132944+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:01:15.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 msgr2=0x7f88a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 0x7f88a406d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f88b4007fd0 tx=0x7f88b4009040 comp rx=0 tx=0).stop 2023-10-14T17:01:15.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 msgr2=0x7f88c015f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 secure :-1 s=READY pgs=1787 cs=0 l=1 rev1=1 crypto rx=0x7f88b8009d90 tx=0x7f88b8004a40 comp rx=0 tx=0).stop 2023-10-14T17:01:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 shutdown_connections 2023-10-14T17:01:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f88a406af90 0x7f88a406d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c015cd70 0x7f88c015f590 unknown :-1 s=CLOSED pgs=1787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c00d0be0 0x7f88c00c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 --2- 172.21.15.29:0/4138955413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c00b8480 0x7f88c0155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 >> 172.21.15.29:0/4138955413 conn(0x7f88c00cf570 msgr2=0x7f88c00cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:15.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 shutdown_connections 2023-10-14T17:01:15.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.602+0000 7f88c5dd0640 1 -- 172.21.15.29:0/4138955413 wait complete. 2023-10-14T17:01:15.624 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T17:01:15.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.848+0000 7f70288e6640 1 Processor -- start 2023-10-14T17:01:15.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.848+0000 7f70288e6640 1 -- start start 2023-10-14T17:01:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f70288e6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f702406fb50 0x7f70240cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f70288e6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240ca560 con 0x7f70240c9060 2023-10-14T17:01:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f70288e6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70240c9950 con 0x7f702406c9e0 2023-10-14T17:01:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f70288e6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70240c9c30 con 0x7f702406fb50 2023-10-14T17:01:15.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f7022d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70240c9060 0x7f702406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54002/0 (socket says 172.21.15.29:54002) 2023-10-14T17:01:15.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.849+0000 7f7022d9d640 1 -- 172.21.15.29:0/249139704 learned_addr learned my addr 172.21.15.29:0/249139704 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:15.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2775878583 0 0) 0x7f70240ca560 con 0x7f70240c9060 2023-10-14T17:01:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7010003520 con 0x7f70240c9060 2023-10-14T17:01:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f701a59c640 1 --2- 172.21.15.29:0/249139704 >> v2:172.21.15.186:3300/0 conn(0x7f702406fb50 0x7f70240cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1298327827 0 0) 0x7f7010003520 con 0x7f70240c9060 2023-10-14T17:01:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 >> v2:172.21.15.186:3300/0 conn(0x7f702406fb50 msgr2=0x7f70240cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 --2- 172.21.15.29:0/249139704 >> v2:172.21.15.186:3300/0 conn(0x7f702406fb50 0x7f70240cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 >> v1:172.21.15.186:6789/0 conn(0x7f702406c9e0 legacy=0x7f70240ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240c9f10 con 0x7f70240c9060 2023-10-14T17:01:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7014001670 con 0x7f70240c9060 2023-10-14T17:01:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 >> v1:172.21.15.29:6789/0 conn(0x7f70240c9060 legacy=0x7f702406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 0x7f7010003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70100047c0 0x7f7010004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100051d0 0x7f701000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70240c9c30 con 0x7f70100039c0 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f701000c120 con 0x7f70100051d0 2023-10-14T17:01:15.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.850+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f701000c2e0 con 0x7f70100047c0 2023-10-14T17:01:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 0x7f7010003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 -- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70100047c0 msgr2=0x7f7010004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70100047c0 0x7f7010004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 -- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100051d0 msgr2=0x7f701000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100051d0 0x7f701000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f7022d9d640 1 -- 172.21.15.29:0/249139704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7010003520 con 0x7f70100039c0 2023-10-14T17:01:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.851+0000 7f702359e640 1 -- 172.21.15.29:0/249139704 reap_dead start 2023-10-14T17:01:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.852+0000 7f7022d9d640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 0x7f7010003de0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f701400c0e0 tx=0x7f70100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b09ab473b712063a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.852+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7014005ad0 con 0x7f70100039c0 2023-10-14T17:01:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7014005c30 con 0x7f70100039c0 2023-10-14T17:01:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f702259c640 1 -- 172.21.15.29:0/249139704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f701402ca50 con 0x7f70100039c0 2023-10-14T17:01:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 -- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 msgr2=0x7f7010003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 0x7f7010003de0 secure :-1 s=READY pgs=1789 cs=0 l=1 rev1=1 crypto rx=0x7f701400c0e0 tx=0x7f70100057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 -- 172.21.15.29:0/249139704 shutdown_connections 2023-10-14T17:01:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 --2- 172.21.15.29:0/249139704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70100039c0 0x7f7010003de0 unknown :-1 s=CLOSED pgs=1789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 -- 172.21.15.29:0/249139704 >> 172.21.15.29:0/249139704 conn(0x7f70240ce6a0 msgr2=0x7f70240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:15.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 -- 172.21.15.29:0/249139704 shutdown_connections 2023-10-14T17:01:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.853+0000 7f70288e6640 1 -- 172.21.15.29:0/249139704 wait complete. 2023-10-14T17:01:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.855+0000 7f70288e6640 1 Processor -- start 2023-10-14T17:01:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.855+0000 7f70288e6640 1 -- start start 2023-10-14T17:01:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100039c0 0x7f70240c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70240b3650 0x7f70240b3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70240c6130 con 0x7f70240c5770 2023-10-14T17:01:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70241f7700 con 0x7f70100039c0 2023-10-14T17:01:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f70288e6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70241f6ab0 con 0x7f70240b3650 2023-10-14T17:01:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f701a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f701a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47254/0 (socket says 172.21.15.29:47254) 2023-10-14T17:01:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.856+0000 7f701a59c640 1 -- 172.21.15.29:0/3198947525 learned_addr learned my addr 172.21.15.29:0/3198947525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 -- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70240b3650 msgr2=0x7f70240b3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70240b3650 0x7f70240b3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 -- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100039c0 msgr2=0x7f70240c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100039c0 0x7f70240c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 -- 172.21.15.29:0/3198947525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70240b3ff0 con 0x7f70240c5770 2023-10-14T17:01:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701a59c640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f700c00d830 tx=0x7f700c00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00b6c0 con 0x7f70240c5770 2023-10-14T17:01:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f700c00b820 con 0x7f70240c5770 2023-10-14T17:01:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70240c6330 con 0x7f70240c5770 2023-10-14T17:01:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700c00bbd0 con 0x7f70240c5770 2023-10-14T17:01:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.857+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70240c68f0 con 0x7f70240c5770 2023-10-14T17:01:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.859+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f702404e9b0 con 0x7f70240c5770 2023-10-14T17:01:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.859+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f700c01e460 con 0x7f70240c5770 2023-10-14T17:01:15.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.860+0000 7f701bfff640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 0x7f700406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.860+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f700c089ea0 con 0x7f70240c5770 2023-10-14T17:01:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.861+0000 7f7022d9d640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 0x7f700406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.862+0000 7f7022d9d640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 0x7f700406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7014034000 tx=0x7f7014039040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:15.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:15.865+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f700c054df0 con 0x7f70240c5770 2023-10-14T17:01:16.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.090+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f70240c1a80 con 0x7f70240c5770 2023-10-14T17:01:16.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.692+0000 7f701bfff640 1 -- 172.21.15.29:0/3198947525 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v73) v1 ==== 118+0+0 (secure 0 0 0) 0x7f700c058410 con 0x7f70240c5770 2023-10-14T17:01:16.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.695+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 msgr2=0x7f700406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.695+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 0x7f700406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f7014034000 tx=0x7f7014039040 comp rx=0 tx=0).stop 2023-10-14T17:01:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.695+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 msgr2=0x7f70240c5bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.695+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 secure :-1 s=READY pgs=1790 cs=0 l=1 rev1=1 crypto rx=0x7f700c00d830 tx=0x7f700c00dcf0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 shutdown_connections 2023-10-14T17:01:16.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f700406af90 0x7f700406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70240b3650 0x7f70240b3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70240c5770 0x7f70240c5bf0 unknown :-1 s=CLOSED pgs=1790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 --2- 172.21.15.29:0/3198947525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70100039c0 0x7f70240c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 >> 172.21.15.29:0/3198947525 conn(0x7f70240ce6a0 msgr2=0x7f702415e100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:16.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 shutdown_connections 2023-10-14T17:01:16.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.696+0000 7f70288e6640 1 -- 172.21.15.29:0/3198947525 wait complete. 2023-10-14T17:01:16.719 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T17:01:16.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T17:01:16.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.948+0000 7f969b8ec640 1 Processor -- start 2023-10-14T17:01:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.948+0000 7f969b8ec640 1 -- start start 2023-10-14T17:01:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.949+0000 7f969b8ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 0x7f9694069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.949+0000 7f969b8ec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.949+0000 7f969b8ec640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96940caec0 con 0x7f96940addb0 2023-10-14T17:01:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.949+0000 7f969b8ec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9694161a50 con 0x7f969406a2e0 2023-10-14T17:01:16.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.949+0000 7f969b8ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f969406a750 con 0x7f96940caa80 2023-10-14T17:01:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.950+0000 7f9699e89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:16.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.950+0000 7f9699e89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:56382/0 (socket says 172.21.15.29:56382) 2023-10-14T17:01:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.950+0000 7f9699688640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f96940addb0 0x7f96940c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:54900/0 (socket says 172.21.15.29:54900) 2023-10-14T17:01:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.950+0000 7f9699e89640 1 -- 172.21.15.29:0/2751671616 learned_addr learned my addr 172.21.15.29:0/2751671616 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.950+0000 7f9698e87640 1 --2- 172.21.15.29:0/2751671616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 0x7f9694069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:16.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.951+0000 7f9698e87640 1 -- 172.21.15.29:0/2751671616 >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 msgr2=0x7f96941650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.951+0000 7f9698e87640 1 --2- 172.21.15.29:0/2751671616 >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.951+0000 7f9698e87640 1 -- 172.21.15.29:0/2751671616 >> v1:172.21.15.186:6789/0 conn(0x7f96940addb0 legacy=0x7f96940c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.951+0000 7f9698e87640 1 -- 172.21.15.29:0/2751671616 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96940cb140 con 0x7f96940caa80 2023-10-14T17:01:16.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.951+0000 7f9699e89640 1 --2- 172.21.15.29:0/2751671616 >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:01:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.952+0000 7f9698e87640 1 --2- 172.21.15.29:0/2751671616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 0x7f9694069da0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96880097a0 tx=0x7f968802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de92e602022cf75f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f9693fff640 1 -- 172.21.15.29:0/2751671616 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968802d040 con 0x7f96940caa80 2023-10-14T17:01:16.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f9693fff640 1 -- 172.21.15.29:0/2751671616 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9688031810 con 0x7f96940caa80 2023-10-14T17:01:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f9693fff640 1 -- 172.21.15.29:0/2751671616 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9688031ba0 con 0x7f96940caa80 2023-10-14T17:01:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 -- 172.21.15.29:0/2751671616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 msgr2=0x7f9694069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 --2- 172.21.15.29:0/2751671616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 0x7f9694069da0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f96880097a0 tx=0x7f968802b8b0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 -- 172.21.15.29:0/2751671616 shutdown_connections 2023-10-14T17:01:16.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 --2- 172.21.15.29:0/2751671616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940caa80 0x7f9694069da0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 --2- 172.21.15.29:0/2751671616 >> v2:172.21.15.186:3300/0 conn(0x7f969406a2e0 0x7f96941650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 -- 172.21.15.29:0/2751671616 >> 172.21.15.29:0/2751671616 conn(0x7f96940ce9b0 msgr2=0x7f96940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.953+0000 7f969b8ec640 1 -- 172.21.15.29:0/2751671616 shutdown_connections 2023-10-14T17:01:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.954+0000 7f969b8ec640 1 -- 172.21.15.29:0/2751671616 wait complete. 2023-10-14T17:01:16.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.955+0000 7f969b8ec640 1 Processor -- start 2023-10-14T17:01:16.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.956+0000 7f969b8ec640 1 -- start start 2023-10-14T17:01:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.956+0000 7f969b8ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.956+0000 7f969b8ec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940addb0 0x7f96940aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f969b8ec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96940caa80 0x7f969415aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f969b8ec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941fa580 con 0x7f969406a2e0 2023-10-14T17:01:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f969b8ec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96941fa860 con 0x7f96940caa80 2023-10-14T17:01:16.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f969b8ec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96941fab40 con 0x7f96940addb0 2023-10-14T17:01:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47262/0 (socket says 172.21.15.29:47262) 2023-10-14T17:01:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 -- 172.21.15.29:0/3159801219 learned_addr learned my addr 172.21.15.29:0/3159801219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 -- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940addb0 msgr2=0x7f96940aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940addb0 0x7f96940aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 -- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96940caa80 msgr2=0x7f969415aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96940caa80 0x7f969415aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.957+0000 7f9699688640 1 -- 172.21.15.29:0/3159801219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f968c0094d0 con 0x7f969406a2e0 2023-10-14T17:01:16.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f9699688640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f968400c800 tx=0x7f968400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400a640 con 0x7f969406a2e0 2023-10-14T17:01:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f968400a7a0 con 0x7f969406a2e0 2023-10-14T17:01:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f968400ab50 con 0x7f969406a2e0 2023-10-14T17:01:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9688009400 con 0x7f969406a2e0 2023-10-14T17:01:16.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.958+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96940c7040 con 0x7f969406a2e0 2023-10-14T17:01:16.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.959+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96940c7280 con 0x7f969406a2e0 2023-10-14T17:01:16.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.961+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f968400acb0 con 0x7f969406a2e0 2023-10-14T17:01:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.962+0000 7f9691ffb640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 0x7f968006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.962+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f9684088f50 con 0x7f969406a2e0 2023-10-14T17:01:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.962+0000 7f9698e87640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 0x7f968006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:16.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.963+0000 7f9698e87640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 0x7f968006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f96880097a0 tx=0x7f9688030000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:16.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:16.967+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96940c7280 con 0x7f969406a2e0 2023-10-14T17:01:17.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:01:17.087 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:01:17.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.202+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f96940c7280 con 0x7f969406a2e0 2023-10-14T17:01:17.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.695+0000 7f9691ffb640 1 -- 172.21.15.29:0/3159801219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v74) v1 ==== 122+0+0 (secure 0 0 0) 0x7f96940c7280 con 0x7f969406a2e0 2023-10-14T17:01:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.698+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 msgr2=0x7f968006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.698+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 0x7f968006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f96880097a0 tx=0x7f9688030000 comp rx=0 tx=0).stop 2023-10-14T17:01:17.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.698+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 msgr2=0x7f969415e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.698+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 secure :-1 s=READY pgs=1791 cs=0 l=1 rev1=1 crypto rx=0x7f968400c800 tx=0x7f968400ccc0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 shutdown_connections 2023-10-14T17:01:17.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f968006af90 0x7f968006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96940caa80 0x7f969415aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96940addb0 0x7f96940aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 --2- 172.21.15.29:0/3159801219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f969406a2e0 0x7f969415e4f0 unknown :-1 s=CLOSED pgs=1791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 >> 172.21.15.29:0/3159801219 conn(0x7f96940ce9b0 msgr2=0x7f96940cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:17.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 shutdown_connections 2023-10-14T17:01:17.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.699+0000 7f969b8ec640 1 -- 172.21.15.29:0/3159801219 wait complete. 2023-10-14T17:01:17.720 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T17:01:17.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:01:17.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.939+0000 7efde5d61640 1 Processor -- start 2023-10-14T17:01:17.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.939+0000 7efde5d61640 1 -- start start 2023-10-14T17:01:17.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.940+0000 7efde5d61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efde015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.940+0000 7efde5d61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde015ed60 con 0x7efde015aaa0 2023-10-14T17:01:17.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.940+0000 7efde5d61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efde00707e0 con 0x7efde015ae80 2023-10-14T17:01:17.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.940+0000 7efde5d61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde006b810 con 0x7efde00ce390 2023-10-14T17:01:17.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efddf7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7efde015aaa0 0x7efde00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54018/0 (socket says 172.21.15.29:54018) 2023-10-14T17:01:17.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efddf7fe640 1 -- 172.21.15.29:0/556422599 learned_addr learned my addr 172.21.15.29:0/556422599 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:17.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734345646 0 0) 0x7efde015ed60 con 0x7efde015aaa0 2023-10-14T17:01:17.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efdc8003520 con 0x7efde015aaa0 2023-10-14T17:01:17.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efddffff640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efde015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.941+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 900163839 0 0) 0x7efdc8003520 con 0x7efde015aaa0 2023-10-14T17:01:17.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 msgr2=0x7efde015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:17.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efde015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 >> v1:172.21.15.186:6789/0 conn(0x7efde015ae80 legacy=0x7efde0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde015ef90 con 0x7efde015aaa0 2023-10-14T17:01:17.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7efdd0001690 con 0x7efde015aaa0 2023-10-14T17:01:17.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 >> v1:172.21.15.29:6789/0 conn(0x7efde015aaa0 legacy=0x7efde00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efdc80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8003fd0 0x7efdc80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc8004d00 0x7efdc8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde006b810 con 0x7efdc8004d00 2023-10-14T17:01:17.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdc800bb40 con 0x7efdc8003fd0 2023-10-14T17:01:17.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdc800bd00 con 0x7efde00ce390 2023-10-14T17:01:17.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddffff640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc8004d00 0x7efdc8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddf7fe640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efdc80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddffff640 1 -- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 msgr2=0x7efdc80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:17.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddffff640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde00ce390 0x7efdc80039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddeffd640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8003fd0 0x7efdc80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.942+0000 7efddffff640 1 -- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8003fd0 msgr2=0x7efdc80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.943+0000 7efddffff640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8003fd0 0x7efdc80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.943+0000 7efddffff640 1 -- 172.21.15.29:0/556422599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdc8003520 con 0x7efdc8004d00 2023-10-14T17:01:17.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.943+0000 7efddeffd640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8003fd0 0x7efdc80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:17.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efddffff640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc8004d00 0x7efdc8005180 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7efdd8007360 tx=0x7efdc80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da46956e36e22dd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd802ccd0 con 0x7efdc8004d00 2023-10-14T17:01:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd802ce30 con 0x7efdc8004d00 2023-10-14T17:01:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efdde7fc640 1 -- 172.21.15.29:0/556422599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd8034660 con 0x7efdc8004d00 2023-10-14T17:01:17.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efde5d61640 1 -- 172.21.15.29:0/556422599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc8004d00 msgr2=0x7efdc8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efde5d61640 1 --2- 172.21.15.29:0/556422599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efdc8004d00 0x7efdc8005180 secure :-1 s=READY pgs=1793 cs=0 l=1 rev1=1 crypto rx=0x7efdd8007360 tx=0x7efdc80057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.944+0000 7efddffff640 1 -- 172.21.15.29:0/556422599 reap_dead start 2023-10-14T17:01:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.945+0000 7efde5d61640 1 -- 172.21.15.29:0/556422599 shutdown_connections 2023-10-14T17:01:17.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.945+0000 7efde5d61640 1 -- 172.21.15.29:0/556422599 >> 172.21.15.29:0/556422599 conn(0x7efde00cb410 msgr2=0x7efde00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.945+0000 7efde5d61640 1 -- 172.21.15.29:0/556422599 shutdown_connections 2023-10-14T17:01:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.945+0000 7efde5d61640 1 -- 172.21.15.29:0/556422599 wait complete. 2023-10-14T17:01:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.947+0000 7efde5d61640 1 Processor -- start 2023-10-14T17:01:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.947+0000 7efde5d61640 1 -- start start 2023-10-14T17:01:17.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8004d00 0x7efde00cc720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde0148df0 0x7efde0149280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c50f0 con 0x7efde014c850 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efde01497c0 con 0x7efdc8004d00 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efde5d61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efde00c4470 con 0x7efde0148df0 2023-10-14T17:01:17.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47288/0 (socket says 172.21.15.29:47288) 2023-10-14T17:01:17.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddeffd640 1 -- 172.21.15.29:0/1894798333 learned_addr learned my addr 172.21.15.29:0/1894798333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:17.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddffff640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde0148df0 0x7efde0149280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddf7fe640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8004d00 0x7efde00cc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddeffd640 1 -- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde0148df0 msgr2=0x7efde0149280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.948+0000 7efddeffd640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde0148df0 0x7efde0149280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efddeffd640 1 -- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8004d00 msgr2=0x7efde00cc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efddeffd640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8004d00 0x7efde00cc720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:17.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efddeffd640 1 -- 172.21.15.29:0/1894798333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efde00c4750 con 0x7efde014c850 2023-10-14T17:01:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efddeffd640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d7a0 tx=0x7efdd400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4017070 con 0x7efde014c850 2023-10-14T17:01:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdd4005070 con 0x7efde014c850 2023-10-14T17:01:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdd4005420 con 0x7efde014c850 2023-10-14T17:01:17.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde00c4a30 con 0x7efde014c850 2023-10-14T17:01:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.949+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efde00ccc60 con 0x7efde014c850 2023-10-14T17:01:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.951+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efde00c5230 con 0x7efde014c850 2023-10-14T17:01:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.951+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7efdd4007030 con 0x7efde014c850 2023-10-14T17:01:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.952+0000 7efdc7fff640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 0x7efdcc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:17.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.952+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7efdd40892a0 con 0x7efde014c850 2023-10-14T17:01:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.953+0000 7efddf7fe640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 0x7efdcc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:17.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.954+0000 7efddf7fe640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 0x7efdcc06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001220 tx=0x7efdd0009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:17.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:17.959+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efde00c5230 con 0x7efde014c850 2023-10-14T17:01:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.195+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efde00c5230 con 0x7efde014c850 2023-10-14T17:01:18.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.197+0000 7efdc7fff640 1 -- 172.21.15.29:0/1894798333 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7efde00c5230 con 0x7efde014c850 2023-10-14T17:01:18.198 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:18.198 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:01:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:01:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.199+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 msgr2=0x7efdcc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.199+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 0x7efdcc06d480 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7efdd0001220 tx=0x7efdd0009040 comp rx=0 tx=0).stop 2023-10-14T17:01:18.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.199+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 msgr2=0x7efde00c3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.199+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 secure :-1 s=READY pgs=1794 cs=0 l=1 rev1=1 crypto rx=0x7efdd400d7a0 tx=0x7efdd400dc60 comp rx=0 tx=0).stop 2023-10-14T17:01:18.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 shutdown_connections 2023-10-14T17:01:18.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7efdcc06afc0 0x7efdcc06d480 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efde0148df0 0x7efde0149280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efde014c850 0x7efde00c3b00 unknown :-1 s=CLOSED pgs=1794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 --2- 172.21.15.29:0/1894798333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efdc8004d00 0x7efde00cc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 >> 172.21.15.29:0/1894798333 conn(0x7efde00cb410 msgr2=0x7efde00b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 shutdown_connections 2023-10-14T17:01:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.200+0000 7efde5d61640 1 -- 172.21.15.29:0/1894798333 wait complete. 2023-10-14T17:01:18.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:01:18.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.453+0000 7f306f1dd640 1 Processor -- start 2023-10-14T17:01:18.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.453+0000 7f306f1dd640 1 -- start start 2023-10-14T17:01:18.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.454+0000 7f306f1dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f306815cbe0 0x7f30680d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.454+0000 7f306f1dd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f306815cfc0 0x7f30680b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306f1dd640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30680d15a0 con 0x7f30680d1090 2023-10-14T17:01:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306f1dd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680d1880 con 0x7f306815cbe0 2023-10-14T17:01:18.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306f1dd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30680d1b60 con 0x7f306815cfc0 2023-10-14T17:01:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f306815cbe0 0x7f30680d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f306815cbe0 0x7f30680d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47298/0 (socket says 172.21.15.29:47298) 2023-10-14T17:01:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 -- 172.21.15.29:0/2533286722 learned_addr learned my addr 172.21.15.29:0/2533286722 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 -- 172.21.15.29:0/2533286722 >> v2:172.21.15.186:3300/0 conn(0x7f306815cfc0 msgr2=0x7f30680b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306d9da640 1 --2- 172.21.15.29:0/2533286722 >> v2:172.21.15.186:3300/0 conn(0x7f306815cfc0 0x7f30680b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 --2- 172.21.15.29:0/2533286722 >> v2:172.21.15.186:3300/0 conn(0x7f306815cfc0 0x7f30680b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 -- 172.21.15.29:0/2533286722 >> v1:172.21.15.186:6789/0 conn(0x7f30680d1090 legacy=0x7f306806bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.455+0000 7f306e1db640 1 -- 172.21.15.29:0/2533286722 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30680d1e40 con 0x7f306815cbe0 2023-10-14T17:01:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.456+0000 7f306e1db640 1 --2- 172.21.15.29:0/2533286722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cbe0 0x7f30680d0b50 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f30680cf4b0 tx=0x7f306402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde3b6870a3f48f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.457+0000 7f306d1d9640 1 -- 172.21.15.29:0/2533286722 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064030860 con 0x7f306815cbe0 2023-10-14T17:01:18.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.457+0000 7f306d1d9640 1 -- 172.21.15.29:0/2533286722 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30640309c0 con 0x7f306815cbe0 2023-10-14T17:01:18.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.457+0000 7f306d1d9640 1 -- 172.21.15.29:0/2533286722 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3064030d50 con 0x7f306815cbe0 2023-10-14T17:01:18.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 -- 172.21.15.29:0/2533286722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cbe0 msgr2=0x7f30680d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 --2- 172.21.15.29:0/2533286722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cbe0 0x7f30680d0b50 secure :-1 s=READY pgs=1795 cs=0 l=1 rev1=1 crypto rx=0x7f30680cf4b0 tx=0x7f306402b9a0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 -- 172.21.15.29:0/2533286722 shutdown_connections 2023-10-14T17:01:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 --2- 172.21.15.29:0/2533286722 >> v2:172.21.15.186:3300/0 conn(0x7f306815cfc0 0x7f30680b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 --2- 172.21.15.29:0/2533286722 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cbe0 0x7f30680d0b50 unknown :-1 s=CLOSED pgs=1795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 -- 172.21.15.29:0/2533286722 >> 172.21.15.29:0/2533286722 conn(0x7f30680cf570 msgr2=0x7f30680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 -- 172.21.15.29:0/2533286722 shutdown_connections 2023-10-14T17:01:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.458+0000 7f306f1dd640 1 -- 172.21.15.29:0/2533286722 wait complete. 2023-10-14T17:01:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.460+0000 7f306f1dd640 1 Processor -- start 2023-10-14T17:01:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.460+0000 7f306f1dd640 1 -- start start 2023-10-14T17:01:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30680d1090 0x7f3068151ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f306815cbe0 0x7f30680c7ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3068152530 con 0x7f306815cfc0 2023-10-14T17:01:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f306815f1c0 con 0x7f30680d1090 2023-10-14T17:01:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306f1dd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f306815f4a0 con 0x7f306815cbe0 2023-10-14T17:01:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306e9dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306e9dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47304/0 (socket says 172.21.15.29:47304) 2023-10-14T17:01:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306e9dc640 1 -- 172.21.15.29:0/500699443 learned_addr learned my addr 172.21.15.29:0/500699443 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.461+0000 7f306e1db640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30680d1090 0x7f3068151ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306d9da640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f306815cbe0 0x7f30680c7ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 -- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f306815cbe0 msgr2=0x7f30680c7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f306815cbe0 0x7f30680c7ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 -- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30680d1090 msgr2=0x7f3068151ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30680d1090 0x7f3068151ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 -- 172.21.15.29:0/500699443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30640094d0 con 0x7f306815cfc0 2023-10-14T17:01:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f306e9dc640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f306000c780 tx=0x7f306000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060003d80 con 0x7f306815cfc0 2023-10-14T17:01:18.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.462+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f306000ce30 con 0x7f306815cfc0 2023-10-14T17:01:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.463+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3068152b60 con 0x7f306815cfc0 2023-10-14T17:01:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.463+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060005290 con 0x7f306815cfc0 2023-10-14T17:01:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.463+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3068153120 con 0x7f306815cfc0 2023-10-14T17:01:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.464+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3068153340 con 0x7f306815cfc0 2023-10-14T17:01:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.465+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30600053f0 con 0x7f306815cfc0 2023-10-14T17:01:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.466+0000 7f305effd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 0x7f304c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.466+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f3060088ef0 con 0x7f306815cfc0 2023-10-14T17:01:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.467+0000 7f306e1db640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 0x7f304c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.467+0000 7f306e1db640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 0x7f304c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3064036000 tx=0x7f3064008030 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.471+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3060053ed0 con 0x7f306815cfc0 2023-10-14T17:01:18.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.669+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f306804e9b0 con 0x7f306815cfc0 2023-10-14T17:01:18.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.672+0000 7f305effd640 1 -- 172.21.15.29:0/500699443 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v154) v1 ==== 74+0+16682 (secure 0 0 0) 0x7f3060017070 con 0x7f306815cfc0 2023-10-14T17:01:18.673 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:18.673 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":154,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:01:15.665578+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"154","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:01:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 msgr2=0x7f304c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 0x7f304c06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f3064036000 tx=0x7f3064008030 comp rx=0 tx=0).stop 2023-10-14T17:01:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 msgr2=0x7f30680d2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 secure :-1 s=READY pgs=1796 cs=0 l=1 rev1=1 crypto rx=0x7f306000c780 tx=0x7f306000cc40 comp rx=0 tx=0).stop 2023-10-14T17:01:18.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 shutdown_connections 2023-10-14T17:01:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f304c06af90 0x7f304c06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f306815cfc0 0x7f30680d2640 unknown :-1 s=CLOSED pgs=1796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f306815cbe0 0x7f30680c7ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 --2- 172.21.15.29:0/500699443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30680d1090 0x7f3068151ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 >> 172.21.15.29:0/500699443 conn(0x7f30680cf570 msgr2=0x7f30680b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 shutdown_connections 2023-10-14T17:01:18.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.675+0000 7f306f1dd640 1 -- 172.21.15.29:0/500699443 wait complete. 2023-10-14T17:01:18.695 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T17:01:18.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:01:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.923+0000 7fab24941640 1 Processor -- start 2023-10-14T17:01:18.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.923+0000 7fab24941640 1 -- start start 2023-10-14T17:01:18.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab24941640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fab2015ae80 0x7fab2015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab24941640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab200ce390 0x7fab20070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab24941640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab2015ed60 con 0x7fab2015aaa0 2023-10-14T17:01:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab24941640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab2006b810 con 0x7fab200ce390 2023-10-14T17:01:18.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab24941640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab200707e0 con 0x7fab2015ae80 2023-10-14T17:01:18.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab1e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab2015aaa0 0x7fab200cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54024/0 (socket says 172.21.15.29:54024) 2023-10-14T17:01:18.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab1e59c640 1 -- 172.21.15.29:0/343106653 learned_addr learned my addr 172.21.15.29:0/343106653 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:18.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.925+0000 7fab1ed9d640 1 --2- 172.21.15.29:0/343106653 >> v2:172.21.15.29:3300/0 conn(0x7fab200ce390 0x7fab20070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab17fff640 1 --2- 172.21.15.29:0/343106653 >> v2:172.21.15.186:3300/0 conn(0x7fab2015ae80 0x7fab2015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1dd9b640 1 -- 172.21.15.29:0/343106653 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952799526 0 0) 0x7fab2015ed60 con 0x7fab2015aaa0 2023-10-14T17:01:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1dd9b640 1 -- 172.21.15.29:0/343106653 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab0c003520 con 0x7fab2015aaa0 2023-10-14T17:01:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1ed9d640 1 -- 172.21.15.29:0/343106653 >> v2:172.21.15.186:3300/0 conn(0x7fab2015ae80 msgr2=0x7fab2015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:18.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1ed9d640 1 --2- 172.21.15.29:0/343106653 >> v2:172.21.15.186:3300/0 conn(0x7fab2015ae80 0x7fab2015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1ed9d640 1 -- 172.21.15.29:0/343106653 >> v1:172.21.15.29:6789/0 conn(0x7fab2015aaa0 legacy=0x7fab200cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.926+0000 7fab1ed9d640 1 -- 172.21.15.29:0/343106653 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab2015ef90 con 0x7fab200ce390 2023-10-14T17:01:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.927+0000 7fab1ed9d640 1 --2- 172.21.15.29:0/343106653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200ce390 0x7fab20070f50 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fab200c59c0 tx=0x7fab1802bec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47599df77bbe9fd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.927+0000 7fab1dd9b640 1 -- 172.21.15.29:0/343106653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab1802cc70 con 0x7fab200ce390 2023-10-14T17:01:18.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.927+0000 7fab1dd9b640 1 -- 172.21.15.29:0/343106653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab1802cdd0 con 0x7fab200ce390 2023-10-14T17:01:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.927+0000 7fab1dd9b640 1 -- 172.21.15.29:0/343106653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab18034660 con 0x7fab200ce390 2023-10-14T17:01:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 -- 172.21.15.29:0/343106653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200ce390 msgr2=0x7fab20070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 --2- 172.21.15.29:0/343106653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200ce390 0x7fab20070f50 secure :-1 s=READY pgs=1798 cs=0 l=1 rev1=1 crypto rx=0x7fab200c59c0 tx=0x7fab1802bec0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 -- 172.21.15.29:0/343106653 shutdown_connections 2023-10-14T17:01:18.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 --2- 172.21.15.29:0/343106653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab200ce390 0x7fab20070f50 unknown :-1 s=CLOSED pgs=1798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 --2- 172.21.15.29:0/343106653 >> v2:172.21.15.186:3300/0 conn(0x7fab2015ae80 0x7fab2015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 -- 172.21.15.29:0/343106653 >> 172.21.15.29:0/343106653 conn(0x7fab200cb410 msgr2=0x7fab200cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:18.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 -- 172.21.15.29:0/343106653 shutdown_connections 2023-10-14T17:01:18.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.928+0000 7fab24941640 1 -- 172.21.15.29:0/343106653 wait complete. 2023-10-14T17:01:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.930+0000 7fab24941640 1 Processor -- start 2023-10-14T17:01:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.930+0000 7fab24941640 1 -- start start 2023-10-14T17:01:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 0x7fab200cac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 0x7fab200c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200ca850 con 0x7fab2015aaa0 2023-10-14T17:01:18.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab200c51b0 con 0x7fab2015ae80 2023-10-14T17:01:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.931+0000 7fab24941640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab200b21d0 con 0x7fab200ce390 2023-10-14T17:01:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47336/0 (socket says 172.21.15.29:47336) 2023-10-14T17:01:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 -- 172.21.15.29:0/664212094 learned_addr learned my addr 172.21.15.29:0/664212094 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab1e59c640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 0x7fab200cac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab1ed9d640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 0x7fab200c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 -- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 msgr2=0x7fab200cac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 0x7fab200cac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 -- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 msgr2=0x7fab200c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 0x7fab200c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 -- 172.21.15.29:0/664212094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab18009a70 con 0x7fab2015aaa0 2023-10-14T17:01:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab1ed9d640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 0x7fab200c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:01:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab1e59c640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 0x7fab200cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.932+0000 7fab17fff640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d5c0 tx=0x7fab0800da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.933+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab080125f0 con 0x7fab2015aaa0 2023-10-14T17:01:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.933+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab08012750 con 0x7fab2015aaa0 2023-10-14T17:01:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.933+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab08012b00 con 0x7fab2015aaa0 2023-10-14T17:01:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.933+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab201592e0 con 0x7fab2015aaa0 2023-10-14T17:01:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.933+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab200ca470 con 0x7fab2015aaa0 2023-10-14T17:01:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.934+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab2004e160 con 0x7fab2015aaa0 2023-10-14T17:01:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.935+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fab08012c60 con 0x7fab2015aaa0 2023-10-14T17:01:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.936+0000 7fab16ffd640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 0x7fab0006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.937+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fab08089c20 con 0x7fab2015aaa0 2023-10-14T17:01:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.937+0000 7fab1e59c640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 0x7fab0006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.938+0000 7fab1e59c640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 0x7fab0006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab100011e0 tx=0x7fab10009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:18.941+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab08054b60 con 0x7fab2015aaa0 2023-10-14T17:01:19.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.136+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 --> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab2015f550 con 0x7fab0006afc0 2023-10-14T17:01:19.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.152+0000 7fab16ffd640 1 -- 172.21.15.29:0/664212094 <== mgr.8862 v2:172.21.15.29:6832/3121942650 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287351 (secure 0 0 0) 0x7fab100013f0 con 0x7fab0006afc0 2023-10-14T17:01:19.157 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:19.158 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:01:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 msgr2=0x7fab0006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 0x7fab0006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fab100011e0 tx=0x7fab10009040 comp rx=0 tx=0).stop 2023-10-14T17:01:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 msgr2=0x7fab201572e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 secure :-1 s=READY pgs=1799 cs=0 l=1 rev1=1 crypto rx=0x7fab0800d5c0 tx=0x7fab0800da80 comp rx=0 tx=0).stop 2023-10-14T17:01:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 shutdown_connections 2023-10-14T17:01:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fab0006afc0 0x7fab0006d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab2015ae80 0x7fab200c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab2015aaa0 0x7fab201572e0 unknown :-1 s=CLOSED pgs=1799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 --2- 172.21.15.29:0/664212094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab200ce390 0x7fab200cac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 >> 172.21.15.29:0/664212094 conn(0x7fab200cb410 msgr2=0x7fab2006ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 shutdown_connections 2023-10-14T17:01:19.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.159+0000 7fab24941640 1 -- 172.21.15.29:0/664212094 wait complete. 2023-10-14T17:01:19.181 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T17:01:17.783896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":365520,"kb_used_data":16152,"kb_used_omap":12,"kb_used_meta":349235,"kb_avail":754609200,"statfs":{"total":773094113280,"available":772719820800,"internally_reserved":0,"allocated":16539648,"data_stored":8607348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":357617208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.147386"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683173+0000","last_change":"2023-10-14T17:01:14.708690+0000","last_active":"2023-10-14T17:01:15.683173+0000","last_peered":"2023-10-14T17:01:15.683173+0000","last_clean":"2023-10-14T17:01:15.683173+0000","last_became_active":"2023-10-14T17:01:14.706344+0000","last_became_peered":"2023-10-14T17:01:14.706344+0000","last_unstale":"2023-10-14T17:01:15.683173+0000","last_undegraded":"2023-10-14T17:01:15.683173+0000","last_fullsized":"2023-10-14T17:01:15.683173+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:15:18.416581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781172+0000","last_change":"2023-10-14T17:01:14.706736+0000","last_active":"2023-10-14T17:01:16.781172+0000","last_peered":"2023-10-14T17:01:16.781172+0000","last_clean":"2023-10-14T17:01:16.781172+0000","last_became_active":"2023-10-14T17:01:14.706410+0000","last_became_peered":"2023-10-14T17:01:14.706410+0000","last_unstale":"2023-10-14T17:01:16.781172+0000","last_undegraded":"2023-10-14T17:01:16.781172+0000","last_fullsized":"2023-10-14T17:01:16.781172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:16.187550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691514+0000","last_change":"2023-10-14T17:01:14.699733+0000","last_active":"2023-10-14T17:01:15.691514+0000","last_peered":"2023-10-14T17:01:15.691514+0000","last_clean":"2023-10-14T17:01:15.691514+0000","last_became_active":"2023-10-14T17:01:14.699434+0000","last_became_peered":"2023-10-14T17:01:14.699434+0000","last_unstale":"2023-10-14T17:01:15.691514+0000","last_undegraded":"2023-10-14T17:01:15.691514+0000","last_fullsized":"2023-10-14T17:01:15.691514+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:50.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682866+0000","last_change":"2023-10-14T17:01:14.704804+0000","last_active":"2023-10-14T17:01:15.682866+0000","last_peered":"2023-10-14T17:01:15.682866+0000","last_clean":"2023-10-14T17:01:15.682866+0000","last_became_active":"2023-10-14T17:01:14.704528+0000","last_became_peered":"2023-10-14T17:01:14.704528+0000","last_unstale":"2023-10-14T17:01:15.682866+0000","last_undegraded":"2023-10-14T17:01:15.682866+0000","last_fullsized":"2023-10-14T17:01:15.682866+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:13.440233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103731+0000","last_change":"2023-10-14T17:01:14.691121+0000","last_active":"2023-10-14T17:01:16.103731+0000","last_peered":"2023-10-14T17:01:16.103731+0000","last_clean":"2023-10-14T17:01:16.103731+0000","last_became_active":"2023-10-14T17:01:14.690818+0000","last_became_peered":"2023-10-14T17:01:14.690818+0000","last_unstale":"2023-10-14T17:01:16.103731+0000","last_undegraded":"2023-10-14T17:01:16.103731+0000","last_fullsized":"2023-10-14T17:01:16.103731+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:23.272935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688303+0000","last_change":"2023-10-14T17:01:14.716137+0000","last_active":"2023-10-14T17:01:15.688303+0000","last_peered":"2023-10-14T17:01:15.688303+0000","last_clean":"2023-10-14T17:01:15.688303+0000","last_became_active":"2023-10-14T17:01:14.715782+0000","last_became_peered":"2023-10-14T17:01:14.715782+0000","last_unstale":"2023-10-14T17:01:15.688303+0000","last_undegraded":"2023-10-14T17:01:15.688303+0000","last_fullsized":"2023-10-14T17:01:15.688303+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:12:01.894640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.687528+0000","last_change":"2023-10-14T17:01:14.694354+0000","last_active":"2023-10-14T17:01:15.687528+0000","last_peered":"2023-10-14T17:01:15.687528+0000","last_clean":"2023-10-14T17:01:15.687528+0000","last_became_active":"2023-10-14T17:01:14.694020+0000","last_became_peered":"2023-10-14T17:01:14.694020+0000","last_unstale":"2023-10-14T17:01:15.687528+0000","last_undegraded":"2023-10-14T17:01:15.687528+0000","last_fullsized":"2023-10-14T17:01:15.687528+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:01.039334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690444+0000","last_change":"2023-10-14T17:01:14.684523+0000","last_active":"2023-10-14T17:01:15.690444+0000","last_peered":"2023-10-14T17:01:15.690444+0000","last_clean":"2023-10-14T17:01:15.690444+0000","last_became_active":"2023-10-14T17:01:14.684255+0000","last_became_peered":"2023-10-14T17:01:14.684255+0000","last_unstale":"2023-10-14T17:01:15.690444+0000","last_undegraded":"2023-10-14T17:01:15.690444+0000","last_fullsized":"2023-10-14T17:01:15.690444+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:13.440233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691657+0000","last_change":"2023-10-14T17:01:14.698523+0000","last_active":"2023-10-14T17:01:15.691657+0000","last_peered":"2023-10-14T17:01:15.691657+0000","last_clean":"2023-10-14T17:01:15.691657+0000","last_became_active":"2023-10-14T17:01:14.698191+0000","last_became_peered":"2023-10-14T17:01:14.698191+0000","last_unstale":"2023-10-14T17:01:15.691657+0000","last_undegraded":"2023-10-14T17:01:15.691657+0000","last_fullsized":"2023-10-14T17:01:15.691657+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:14.490172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781353+0000","last_change":"2023-10-14T17:01:14.698747+0000","last_active":"2023-10-14T17:01:16.781353+0000","last_peered":"2023-10-14T17:01:16.781353+0000","last_clean":"2023-10-14T17:01:16.781353+0000","last_became_active":"2023-10-14T17:01:14.697703+0000","last_became_peered":"2023-10-14T17:01:14.697703+0000","last_unstale":"2023-10-14T17:01:16.781353+0000","last_undegraded":"2023-10-14T17:01:16.781353+0000","last_fullsized":"2023-10-14T17:01:16.781353+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:15.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781023+0000","last_change":"2023-10-14T17:01:14.705837+0000","last_active":"2023-10-14T17:01:16.781023+0000","last_peered":"2023-10-14T17:01:16.781023+0000","last_clean":"2023-10-14T17:01:16.781023+0000","last_became_active":"2023-10-14T17:01:14.705512+0000","last_became_peered":"2023-10-14T17:01:14.705512+0000","last_unstale":"2023-10-14T17:01:16.781023+0000","last_undegraded":"2023-10-14T17:01:16.781023+0000","last_fullsized":"2023-10-14T17:01:16.781023+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:08.038231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691163+0000","last_change":"2023-10-14T17:01:14.700479+0000","last_active":"2023-10-14T17:01:15.691163+0000","last_peered":"2023-10-14T17:01:15.691163+0000","last_clean":"2023-10-14T17:01:15.691163+0000","last_became_active":"2023-10-14T17:01:14.700212+0000","last_became_peered":"2023-10-14T17:01:14.700212+0000","last_unstale":"2023-10-14T17:01:15.691163+0000","last_undegraded":"2023-10-14T17:01:15.691163+0000","last_fullsized":"2023-10-14T17:01:15.691163+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:49.455799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683209+0000","last_change":"2023-10-14T17:01:14.702926+0000","last_active":"2023-10-14T17:01:15.683209+0000","last_peered":"2023-10-14T17:01:15.683209+0000","last_clean":"2023-10-14T17:01:15.683209+0000","last_became_active":"2023-10-14T17:01:14.702627+0000","last_became_peered":"2023-10-14T17:01:14.702627+0000","last_unstale":"2023-10-14T17:01:15.683209+0000","last_undegraded":"2023-10-14T17:01:15.683209+0000","last_fullsized":"2023-10-14T17:01:15.683209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:49.455799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780695+0000","last_change":"2023-10-14T17:01:14.716570+0000","last_active":"2023-10-14T17:01:16.780695+0000","last_peered":"2023-10-14T17:01:16.780695+0000","last_clean":"2023-10-14T17:01:16.780695+0000","last_became_active":"2023-10-14T17:01:14.716297+0000","last_became_peered":"2023-10-14T17:01:14.716297+0000","last_unstale":"2023-10-14T17:01:16.780695+0000","last_undegraded":"2023-10-14T17:01:16.780695+0000","last_fullsized":"2023-10-14T17:01:16.780695+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:08.736748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682502+0000","last_change":"2023-10-14T17:01:14.707371+0000","last_active":"2023-10-14T17:01:15.682502+0000","last_peered":"2023-10-14T17:01:15.682502+0000","last_clean":"2023-10-14T17:01:15.682502+0000","last_became_active":"2023-10-14T17:01:14.705122+0000","last_became_peered":"2023-10-14T17:01:14.705122+0000","last_unstale":"2023-10-14T17:01:15.682502+0000","last_undegraded":"2023-10-14T17:01:15.682502+0000","last_fullsized":"2023-10-14T17:01:15.682502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:58.474822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780497+0000","last_change":"2023-10-14T17:01:14.710074+0000","last_active":"2023-10-14T17:01:16.780497+0000","last_peered":"2023-10-14T17:01:16.780497+0000","last_clean":"2023-10-14T17:01:16.780497+0000","last_became_active":"2023-10-14T17:01:14.709716+0000","last_became_peered":"2023-10-14T17:01:14.709716+0000","last_unstale":"2023-10-14T17:01:16.780497+0000","last_undegraded":"2023-10-14T17:01:16.780497+0000","last_fullsized":"2023-10-14T17:01:16.780497+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:46.408470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688942+0000","last_change":"2023-10-14T17:01:14.699675+0000","last_active":"2023-10-14T17:01:15.688942+0000","last_peered":"2023-10-14T17:01:15.688942+0000","last_clean":"2023-10-14T17:01:15.688942+0000","last_became_active":"2023-10-14T17:01:14.699408+0000","last_became_peered":"2023-10-14T17:01:14.699408+0000","last_unstale":"2023-10-14T17:01:15.688942+0000","last_undegraded":"2023-10-14T17:01:15.688942+0000","last_fullsized":"2023-10-14T17:01:15.688942+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:52.401357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782259+0000","last_change":"2023-10-14T17:01:14.702416+0000","last_active":"2023-10-14T17:01:16.782259+0000","last_peered":"2023-10-14T17:01:16.782259+0000","last_clean":"2023-10-14T17:01:16.782259+0000","last_became_active":"2023-10-14T17:01:14.695421+0000","last_became_peered":"2023-10-14T17:01:14.695421+0000","last_unstale":"2023-10-14T17:01:16.782259+0000","last_undegraded":"2023-10-14T17:01:16.782259+0000","last_fullsized":"2023-10-14T17:01:16.782259+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:47.122460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781204+0000","last_change":"2023-10-14T17:01:14.699636+0000","last_active":"2023-10-14T17:01:16.781204+0000","last_peered":"2023-10-14T17:01:16.781204+0000","last_clean":"2023-10-14T17:01:16.781204+0000","last_became_active":"2023-10-14T17:01:14.699313+0000","last_became_peered":"2023-10-14T17:01:14.699313+0000","last_unstale":"2023-10-14T17:01:16.781204+0000","last_undegraded":"2023-10-14T17:01:16.781204+0000","last_fullsized":"2023-10-14T17:01:16.781204+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:39:25.218441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782580+0000","last_change":"2023-10-14T17:01:14.697845+0000","last_active":"2023-10-14T17:01:16.782580+0000","last_peered":"2023-10-14T17:01:16.782580+0000","last_clean":"2023-10-14T17:01:16.782580+0000","last_became_active":"2023-10-14T17:01:14.691997+0000","last_became_peered":"2023-10-14T17:01:14.691997+0000","last_unstale":"2023-10-14T17:01:16.782580+0000","last_undegraded":"2023-10-14T17:01:16.782580+0000","last_fullsized":"2023-10-14T17:01:16.782580+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:42.434427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103671+0000","last_change":"2023-10-14T17:01:14.692032+0000","last_active":"2023-10-14T17:01:16.103671+0000","last_peered":"2023-10-14T17:01:16.103671+0000","last_clean":"2023-10-14T17:01:16.103671+0000","last_became_active":"2023-10-14T17:01:14.691734+0000","last_became_peered":"2023-10-14T17:01:14.691734+0000","last_unstale":"2023-10-14T17:01:16.103671+0000","last_undegraded":"2023-10-14T17:01:16.103671+0000","last_fullsized":"2023-10-14T17:01:16.103671+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:19:32.149779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781557+0000","last_change":"2023-10-14T17:01:14.712560+0000","last_active":"2023-10-14T17:01:16.781557+0000","last_peered":"2023-10-14T17:01:16.781557+0000","last_clean":"2023-10-14T17:01:16.781557+0000","last_became_active":"2023-10-14T17:01:14.712232+0000","last_became_peered":"2023-10-14T17:01:14.712232+0000","last_unstale":"2023-10-14T17:01:16.781557+0000","last_undegraded":"2023-10-14T17:01:16.781557+0000","last_fullsized":"2023-10-14T17:01:16.781557+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:31.057718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690860+0000","last_change":"2023-10-14T17:01:14.699364+0000","last_active":"2023-10-14T17:01:15.690860+0000","last_peered":"2023-10-14T17:01:15.690860+0000","last_clean":"2023-10-14T17:01:15.690860+0000","last_became_active":"2023-10-14T17:01:14.698632+0000","last_became_peered":"2023-10-14T17:01:14.698632+0000","last_unstale":"2023-10-14T17:01:15.690860+0000","last_undegraded":"2023-10-14T17:01:15.690860+0000","last_fullsized":"2023-10-14T17:01:15.690860+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:15:18.416581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780156+0000","last_change":"2023-10-14T17:01:14.702111+0000","last_active":"2023-10-14T17:01:16.780156+0000","last_peered":"2023-10-14T17:01:16.780156+0000","last_clean":"2023-10-14T17:01:16.780156+0000","last_became_active":"2023-10-14T17:01:14.699054+0000","last_became_peered":"2023-10-14T17:01:14.699054+0000","last_unstale":"2023-10-14T17:01:16.780156+0000","last_undegraded":"2023-10-14T17:01:16.780156+0000","last_fullsized":"2023-10-14T17:01:16.780156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:52.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781782+0000","last_change":"2023-10-14T17:01:14.705363+0000","last_active":"2023-10-14T17:01:16.781782+0000","last_peered":"2023-10-14T17:01:16.781782+0000","last_clean":"2023-10-14T17:01:16.781782+0000","last_became_active":"2023-10-14T17:01:14.705099+0000","last_became_peered":"2023-10-14T17:01:14.705099+0000","last_unstale":"2023-10-14T17:01:16.781782+0000","last_undegraded":"2023-10-14T17:01:16.781782+0000","last_fullsized":"2023-10-14T17:01:16.781782+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:41:03.672709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782361+0000","last_change":"2023-10-14T17:01:14.703194+0000","last_active":"2023-10-14T17:01:16.782361+0000","last_peered":"2023-10-14T17:01:16.782361+0000","last_clean":"2023-10-14T17:01:16.782361+0000","last_became_active":"2023-10-14T17:01:14.702678+0000","last_became_peered":"2023-10-14T17:01:14.702678+0000","last_unstale":"2023-10-14T17:01:16.782361+0000","last_undegraded":"2023-10-14T17:01:16.782361+0000","last_fullsized":"2023-10-14T17:01:16.782361+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:12.686427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682857+0000","last_change":"2023-10-14T17:01:14.709448+0000","last_active":"2023-10-14T17:01:15.682857+0000","last_peered":"2023-10-14T17:01:15.682857+0000","last_clean":"2023-10-14T17:01:15.682857+0000","last_became_active":"2023-10-14T17:01:14.709194+0000","last_became_peered":"2023-10-14T17:01:14.709194+0000","last_unstale":"2023-10-14T17:01:15.682857+0000","last_undegraded":"2023-10-14T17:01:15.682857+0000","last_fullsized":"2023-10-14T17:01:15.682857+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:57.591951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782055+0000","last_change":"2023-10-14T17:01:14.709973+0000","last_active":"2023-10-14T17:01:16.782055+0000","last_peered":"2023-10-14T17:01:16.782055+0000","last_clean":"2023-10-14T17:01:16.782055+0000","last_became_active":"2023-10-14T17:01:14.709767+0000","last_became_peered":"2023-10-14T17:01:14.709767+0000","last_unstale":"2023-10-14T17:01:16.782055+0000","last_undegraded":"2023-10-14T17:01:16.782055+0000","last_fullsized":"2023-10-14T17:01:16.782055+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:08.038231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683055+0000","last_change":"2023-10-14T17:01:14.706746+0000","last_active":"2023-10-14T17:01:15.683055+0000","last_peered":"2023-10-14T17:01:15.683055+0000","last_clean":"2023-10-14T17:01:15.683055+0000","last_became_active":"2023-10-14T17:01:14.706098+0000","last_became_peered":"2023-10-14T17:01:14.706098+0000","last_unstale":"2023-10-14T17:01:15.683055+0000","last_undegraded":"2023-10-14T17:01:15.683055+0000","last_fullsized":"2023-10-14T17:01:15.683055+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:52.102810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682645+0000","last_change":"2023-10-14T17:01:14.711463+0000","last_active":"2023-10-14T17:01:15.682645+0000","last_peered":"2023-10-14T17:01:15.682645+0000","last_clean":"2023-10-14T17:01:15.682645+0000","last_became_active":"2023-10-14T17:01:14.707670+0000","last_became_peered":"2023-10-14T17:01:14.707670+0000","last_unstale":"2023-10-14T17:01:15.682645+0000","last_undegraded":"2023-10-14T17:01:15.682645+0000","last_fullsized":"2023-10-14T17:01:15.682645+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:02.267425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103462+0000","last_change":"2023-10-14T17:01:14.700351+0000","last_active":"2023-10-14T17:01:16.103462+0000","last_peered":"2023-10-14T17:01:16.103462+0000","last_clean":"2023-10-14T17:01:16.103462+0000","last_became_active":"2023-10-14T17:01:14.699066+0000","last_became_peered":"2023-10-14T17:01:14.699066+0000","last_unstale":"2023-10-14T17:01:16.103462+0000","last_undegraded":"2023-10-14T17:01:16.103462+0000","last_fullsized":"2023-10-14T17:01:16.103462+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:35.014489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688017+0000","last_change":"2023-10-14T17:01:14.709522+0000","last_active":"2023-10-14T17:01:15.688017+0000","last_peered":"2023-10-14T17:01:15.688017+0000","last_clean":"2023-10-14T17:01:15.688017+0000","last_became_active":"2023-10-14T17:01:14.709153+0000","last_became_peered":"2023-10-14T17:01:14.709153+0000","last_unstale":"2023-10-14T17:01:15.688017+0000","last_undegraded":"2023-10-14T17:01:15.688017+0000","last_fullsized":"2023-10-14T17:01:15.688017+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:47:07.835175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782450+0000","last_change":"2023-10-14T17:01:14.702985+0000","last_active":"2023-10-14T17:01:16.782450+0000","last_peered":"2023-10-14T17:01:16.782450+0000","last_clean":"2023-10-14T17:01:16.782450+0000","last_became_active":"2023-10-14T17:01:14.702041+0000","last_became_peered":"2023-10-14T17:01:14.702041+0000","last_unstale":"2023-10-14T17:01:16.782450+0000","last_undegraded":"2023-10-14T17:01:16.782450+0000","last_fullsized":"2023-10-14T17:01:16.782450+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:35.014489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":16,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104532+0000","last_change":"2023-10-14T17:01:13.678943+0000","last_active":"2023-10-14T17:01:16.104532+0000","last_peered":"2023-10-14T17:01:16.104532+0000","last_clean":"2023-10-14T17:01:16.104532+0000","last_became_active":"2023-10-14T17:01:13.678667+0000","last_became_peered":"2023-10-14T17:01:13.678667+0000","last_unstale":"2023-10-14T17:01:16.104532+0000","last_undegraded":"2023-10-14T17:01:16.104532+0000","last_fullsized":"2023-10-14T17:01:16.104532+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:36.098987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691672+0000","last_change":"2023-10-14T17:01:14.702130+0000","last_active":"2023-10-14T17:01:15.691672+0000","last_peered":"2023-10-14T17:01:15.691672+0000","last_clean":"2023-10-14T17:01:15.691672+0000","last_became_active":"2023-10-14T17:01:14.701851+0000","last_became_peered":"2023-10-14T17:01:14.701851+0000","last_unstale":"2023-10-14T17:01:15.691672+0000","last_undegraded":"2023-10-14T17:01:15.691672+0000","last_fullsized":"2023-10-14T17:01:15.691672+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:51:28.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782350+0000","last_change":"2023-10-14T17:01:13.676855+0000","last_active":"2023-10-14T17:01:16.782350+0000","last_peered":"2023-10-14T17:01:16.782350+0000","last_clean":"2023-10-14T17:01:16.782350+0000","last_became_active":"2023-10-14T17:01:13.675799+0000","last_became_peered":"2023-10-14T17:01:13.675799+0000","last_unstale":"2023-10-14T17:01:16.782350+0000","last_undegraded":"2023-10-14T17:01:16.782350+0000","last_fullsized":"2023-10-14T17:01:16.782350+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:07.218287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104340+0000","last_change":"2023-10-14T17:01:14.696159+0000","last_active":"2023-10-14T17:01:16.104340+0000","last_peered":"2023-10-14T17:01:16.104340+0000","last_clean":"2023-10-14T17:01:16.104340+0000","last_became_active":"2023-10-14T17:01:14.694787+0000","last_became_peered":"2023-10-14T17:01:14.694787+0000","last_unstale":"2023-10-14T17:01:16.104340+0000","last_undegraded":"2023-10-14T17:01:16.104340+0000","last_fullsized":"2023-10-14T17:01:16.104340+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:21.050171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781325+0000","last_change":"2023-10-14T17:01:13.673283+0000","last_active":"2023-10-14T17:01:16.781325+0000","last_peered":"2023-10-14T17:01:16.781325+0000","last_clean":"2023-10-14T17:01:16.781325+0000","last_became_active":"2023-10-14T17:01:13.672243+0000","last_became_peered":"2023-10-14T17:01:13.672243+0000","last_unstale":"2023-10-14T17:01:16.781325+0000","last_undegraded":"2023-10-14T17:01:16.781325+0000","last_fullsized":"2023-10-14T17:01:16.781325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:07.218287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104337+0000","last_change":"2023-10-14T17:01:14.705210+0000","last_active":"2023-10-14T17:01:16.104337+0000","last_peered":"2023-10-14T17:01:16.104337+0000","last_clean":"2023-10-14T17:01:16.104337+0000","last_became_active":"2023-10-14T17:01:14.704897+0000","last_became_peered":"2023-10-14T17:01:14.704897+0000","last_unstale":"2023-10-14T17:01:16.104337+0000","last_undegraded":"2023-10-14T17:01:16.104337+0000","last_fullsized":"2023-10-14T17:01:16.104337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:29.099700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781923+0000","last_change":"2023-10-14T17:01:13.672487+0000","last_active":"2023-10-14T17:01:16.781923+0000","last_peered":"2023-10-14T17:01:16.781923+0000","last_clean":"2023-10-14T17:01:16.781923+0000","last_became_active":"2023-10-14T17:01:13.672192+0000","last_became_peered":"2023-10-14T17:01:13.672192+0000","last_unstale":"2023-10-14T17:01:16.781923+0000","last_undegraded":"2023-10-14T17:01:16.781923+0000","last_fullsized":"2023-10-14T17:01:16.781923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:08.909223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781260+0000","last_change":"2023-10-14T17:01:14.700281+0000","last_active":"2023-10-14T17:01:16.781260+0000","last_peered":"2023-10-14T17:01:16.781260+0000","last_clean":"2023-10-14T17:01:16.781260+0000","last_became_active":"2023-10-14T17:01:14.695051+0000","last_became_peered":"2023-10-14T17:01:14.695051+0000","last_unstale":"2023-10-14T17:01:16.781260+0000","last_undegraded":"2023-10-14T17:01:16.781260+0000","last_fullsized":"2023-10-14T17:01:16.781260+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:44:25.638199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780526+0000","last_change":"2023-10-14T17:01:13.670963+0000","last_active":"2023-10-14T17:01:16.780526+0000","last_peered":"2023-10-14T17:01:16.780526+0000","last_clean":"2023-10-14T17:01:16.780526+0000","last_became_active":"2023-10-14T17:01:13.669597+0000","last_became_peered":"2023-10-14T17:01:13.669597+0000","last_unstale":"2023-10-14T17:01:16.780526+0000","last_undegraded":"2023-10-14T17:01:16.780526+0000","last_fullsized":"2023-10-14T17:01:16.780526+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:48.441183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688096+0000","last_change":"2023-10-14T17:01:14.700042+0000","last_active":"2023-10-14T17:01:15.688096+0000","last_peered":"2023-10-14T17:01:15.688096+0000","last_clean":"2023-10-14T17:01:15.688096+0000","last_became_active":"2023-10-14T17:01:14.699771+0000","last_became_peered":"2023-10-14T17:01:14.699771+0000","last_unstale":"2023-10-14T17:01:15.688096+0000","last_undegraded":"2023-10-14T17:01:15.688096+0000","last_fullsized":"2023-10-14T17:01:15.688096+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:46:11.670317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781383+0000","last_change":"2023-10-14T17:01:13.671646+0000","last_active":"2023-10-14T17:01:16.781383+0000","last_peered":"2023-10-14T17:01:16.781383+0000","last_clean":"2023-10-14T17:01:16.781383+0000","last_became_active":"2023-10-14T17:01:13.671357+0000","last_became_peered":"2023-10-14T17:01:13.671357+0000","last_unstale":"2023-10-14T17:01:16.781383+0000","last_undegraded":"2023-10-14T17:01:16.781383+0000","last_fullsized":"2023-10-14T17:01:16.781383+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:00.652084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690515+0000","last_change":"2023-10-14T17:01:14.685455+0000","last_active":"2023-10-14T17:01:15.690515+0000","last_peered":"2023-10-14T17:01:15.690515+0000","last_clean":"2023-10-14T17:01:15.690515+0000","last_became_active":"2023-10-14T17:01:14.684945+0000","last_became_peered":"2023-10-14T17:01:14.684945+0000","last_unstale":"2023-10-14T17:01:15.690515+0000","last_undegraded":"2023-10-14T17:01:15.690515+0000","last_fullsized":"2023-10-14T17:01:15.690515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:05:43.150631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.687414+0000","last_change":"2023-10-14T17:01:13.676361+0000","last_active":"2023-10-14T17:01:15.687414+0000","last_peered":"2023-10-14T17:01:15.687414+0000","last_clean":"2023-10-14T17:01:15.687414+0000","last_became_active":"2023-10-14T17:01:13.675084+0000","last_became_peered":"2023-10-14T17:01:13.675084+0000","last_unstale":"2023-10-14T17:01:15.687414+0000","last_undegraded":"2023-10-14T17:01:15.687414+0000","last_fullsized":"2023-10-14T17:01:15.687414+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:30.043102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781288+0000","last_change":"2023-10-14T17:01:14.704549+0000","last_active":"2023-10-14T17:01:16.781288+0000","last_peered":"2023-10-14T17:01:16.781288+0000","last_clean":"2023-10-14T17:01:16.781288+0000","last_became_active":"2023-10-14T17:01:14.704239+0000","last_became_peered":"2023-10-14T17:01:14.704239+0000","last_unstale":"2023-10-14T17:01:16.781288+0000","last_undegraded":"2023-10-14T17:01:16.781288+0000","last_fullsized":"2023-10-14T17:01:16.781288+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:13:00.705305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780166+0000","last_change":"2023-10-14T17:01:13.679963+0000","last_active":"2023-10-14T17:01:16.780166+0000","last_peered":"2023-10-14T17:01:16.780166+0000","last_clean":"2023-10-14T17:01:16.780166+0000","last_became_active":"2023-10-14T17:01:13.677746+0000","last_became_peered":"2023-10-14T17:01:13.677746+0000","last_unstale":"2023-10-14T17:01:16.780166+0000","last_undegraded":"2023-10-14T17:01:16.780166+0000","last_fullsized":"2023-10-14T17:01:16.780166+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:03.404311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682948+0000","last_change":"2023-10-14T17:01:14.706038+0000","last_active":"2023-10-14T17:01:15.682948+0000","last_peered":"2023-10-14T17:01:15.682948+0000","last_clean":"2023-10-14T17:01:15.682948+0000","last_became_active":"2023-10-14T17:01:14.705682+0000","last_became_peered":"2023-10-14T17:01:14.705682+0000","last_unstale":"2023-10-14T17:01:15.682948+0000","last_undegraded":"2023-10-14T17:01:15.682948+0000","last_fullsized":"2023-10-14T17:01:15.682948+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:05:43.150631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782597+0000","last_change":"2023-10-14T17:01:13.674826+0000","last_active":"2023-10-14T17:01:16.782597+0000","last_peered":"2023-10-14T17:01:16.782597+0000","last_clean":"2023-10-14T17:01:16.782597+0000","last_became_active":"2023-10-14T17:01:13.669834+0000","last_became_peered":"2023-10-14T17:01:13.669834+0000","last_unstale":"2023-10-14T17:01:16.782597+0000","last_undegraded":"2023-10-14T17:01:16.782597+0000","last_fullsized":"2023-10-14T17:01:16.782597+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:22.778177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782473+0000","last_change":"2023-10-14T17:01:14.702348+0000","last_active":"2023-10-14T17:01:16.782473+0000","last_peered":"2023-10-14T17:01:16.782473+0000","last_clean":"2023-10-14T17:01:16.782473+0000","last_became_active":"2023-10-14T17:01:14.702071+0000","last_became_peered":"2023-10-14T17:01:14.702071+0000","last_unstale":"2023-10-14T17:01:16.782473+0000","last_undegraded":"2023-10-14T17:01:16.782473+0000","last_fullsized":"2023-10-14T17:01:16.782473+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:31.057718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782483+0000","last_change":"2023-10-14T17:01:13.680495+0000","last_active":"2023-10-14T17:01:16.782483+0000","last_peered":"2023-10-14T17:01:16.782483+0000","last_clean":"2023-10-14T17:01:16.782483+0000","last_became_active":"2023-10-14T17:01:13.680166+0000","last_became_peered":"2023-10-14T17:01:13.680166+0000","last_unstale":"2023-10-14T17:01:16.782483+0000","last_undegraded":"2023-10-14T17:01:16.782483+0000","last_fullsized":"2023-10-14T17:01:16.782483+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:10.245115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104502+0000","last_change":"2023-10-14T17:01:14.702121+0000","last_active":"2023-10-14T17:01:16.104502+0000","last_peered":"2023-10-14T17:01:16.104502+0000","last_clean":"2023-10-14T17:01:16.104502+0000","last_became_active":"2023-10-14T17:01:14.701467+0000","last_became_peered":"2023-10-14T17:01:14.701467+0000","last_unstale":"2023-10-14T17:01:16.104502+0000","last_undegraded":"2023-10-14T17:01:16.104502+0000","last_fullsized":"2023-10-14T17:01:16.104502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:17:29.054000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":15,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781940+0000","last_change":"2023-10-14T17:01:13.674735+0000","last_active":"2023-10-14T17:01:16.781940+0000","last_peered":"2023-10-14T17:01:16.781940+0000","last_clean":"2023-10-14T17:01:16.781940+0000","last_became_active":"2023-10-14T17:01:13.672714+0000","last_became_peered":"2023-10-14T17:01:13.672714+0000","last_unstale":"2023-10-14T17:01:16.781940+0000","last_undegraded":"2023-10-14T17:01:16.781940+0000","last_fullsized":"2023-10-14T17:01:16.781940+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:11.468417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104368+0000","last_change":"2023-10-14T17:01:14.694003+0000","last_active":"2023-10-14T17:01:16.104368+0000","last_peered":"2023-10-14T17:01:16.104368+0000","last_clean":"2023-10-14T17:01:16.104368+0000","last_became_active":"2023-10-14T17:01:14.692409+0000","last_became_peered":"2023-10-14T17:01:14.692409+0000","last_unstale":"2023-10-14T17:01:16.104368+0000","last_undegraded":"2023-10-14T17:01:16.104368+0000","last_fullsized":"2023-10-14T17:01:16.104368+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:51.762425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":18,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691070+0000","last_change":"2023-10-14T17:01:13.681462+0000","last_active":"2023-10-14T17:01:15.691070+0000","last_peered":"2023-10-14T17:01:15.691070+0000","last_clean":"2023-10-14T17:01:15.691070+0000","last_became_active":"2023-10-14T17:01:13.681189+0000","last_became_peered":"2023-10-14T17:01:13.681189+0000","last_unstale":"2023-10-14T17:01:15.691070+0000","last_undegraded":"2023-10-14T17:01:15.691070+0000","last_fullsized":"2023-10-14T17:01:15.691070+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:01.871554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683212+0000","last_change":"2023-10-14T17:01:14.711498+0000","last_active":"2023-10-14T17:01:15.683212+0000","last_peered":"2023-10-14T17:01:15.683212+0000","last_clean":"2023-10-14T17:01:15.683212+0000","last_became_active":"2023-10-14T17:01:14.711165+0000","last_became_peered":"2023-10-14T17:01:14.711165+0000","last_unstale":"2023-10-14T17:01:15.683212+0000","last_undegraded":"2023-10-14T17:01:15.683212+0000","last_fullsized":"2023-10-14T17:01:15.683212+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:12.485256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682709+0000","last_change":"2023-10-14T17:01:13.670668+0000","last_active":"2023-10-14T17:01:15.682709+0000","last_peered":"2023-10-14T17:01:15.682709+0000","last_clean":"2023-10-14T17:01:15.682709+0000","last_became_active":"2023-10-14T17:01:13.669219+0000","last_became_peered":"2023-10-14T17:01:13.669219+0000","last_unstale":"2023-10-14T17:01:15.682709+0000","last_undegraded":"2023-10-14T17:01:15.682709+0000","last_fullsized":"2023-10-14T17:01:15.682709+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:01.871554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780042+0000","last_change":"2023-10-14T17:01:14.691782+0000","last_active":"2023-10-14T17:01:16.780042+0000","last_peered":"2023-10-14T17:01:16.780042+0000","last_clean":"2023-10-14T17:01:16.780042+0000","last_became_active":"2023-10-14T17:01:14.691111+0000","last_became_peered":"2023-10-14T17:01:14.691111+0000","last_unstale":"2023-10-14T17:01:16.780042+0000","last_undegraded":"2023-10-14T17:01:16.780042+0000","last_fullsized":"2023-10-14T17:01:16.780042+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:27:18.745622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781613+0000","last_change":"2023-10-14T17:01:13.679508+0000","last_active":"2023-10-14T17:01:16.781613+0000","last_peered":"2023-10-14T17:01:16.781613+0000","last_clean":"2023-10-14T17:01:16.781613+0000","last_became_active":"2023-10-14T17:01:13.679026+0000","last_became_peered":"2023-10-14T17:01:13.679026+0000","last_unstale":"2023-10-14T17:01:16.781613+0000","last_undegraded":"2023-10-14T17:01:16.781613+0000","last_fullsized":"2023-10-14T17:01:16.781613+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:49.397746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690733+0000","last_change":"2023-10-14T17:01:14.699908+0000","last_active":"2023-10-14T17:01:15.690733+0000","last_peered":"2023-10-14T17:01:15.690733+0000","last_clean":"2023-10-14T17:01:15.690733+0000","last_became_active":"2023-10-14T17:01:14.699636+0000","last_became_peered":"2023-10-14T17:01:14.699636+0000","last_unstale":"2023-10-14T17:01:15.690733+0000","last_undegraded":"2023-10-14T17:01:15.690733+0000","last_fullsized":"2023-10-14T17:01:15.690733+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:52.102810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":20,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690791+0000","last_change":"2023-10-14T17:01:13.677498+0000","last_active":"2023-10-14T17:01:15.690791+0000","last_peered":"2023-10-14T17:01:15.690791+0000","last_clean":"2023-10-14T17:01:15.690791+0000","last_became_active":"2023-10-14T17:01:13.677227+0000","last_became_peered":"2023-10-14T17:01:13.677227+0000","last_unstale":"2023-10-14T17:01:15.690791+0000","last_undegraded":"2023-10-14T17:01:15.690791+0000","last_fullsized":"2023-10-14T17:01:15.690791+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:01.252808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691566+0000","last_change":"2023-10-14T17:01:14.703429+0000","last_active":"2023-10-14T17:01:15.691566+0000","last_peered":"2023-10-14T17:01:15.691566+0000","last_clean":"2023-10-14T17:01:15.691566+0000","last_became_active":"2023-10-14T17:01:14.703020+0000","last_became_peered":"2023-10-14T17:01:14.703020+0000","last_unstale":"2023-10-14T17:01:15.691566+0000","last_undegraded":"2023-10-14T17:01:15.691566+0000","last_fullsized":"2023-10-14T17:01:15.691566+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:12.485256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782659+0000","last_change":"2023-10-14T17:01:13.679977+0000","last_active":"2023-10-14T17:01:16.782659+0000","last_peered":"2023-10-14T17:01:16.782659+0000","last_clean":"2023-10-14T17:01:16.782659+0000","last_became_active":"2023-10-14T17:01:13.679694+0000","last_became_peered":"2023-10-14T17:01:13.679694+0000","last_unstale":"2023-10-14T17:01:16.782659+0000","last_undegraded":"2023-10-14T17:01:16.782659+0000","last_fullsized":"2023-10-14T17:01:16.782659+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:04.039372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782530+0000","last_change":"2023-10-14T17:01:14.707457+0000","last_active":"2023-10-14T17:01:16.782530+0000","last_peered":"2023-10-14T17:01:16.782530+0000","last_clean":"2023-10-14T17:01:16.782530+0000","last_became_active":"2023-10-14T17:01:14.707094+0000","last_became_peered":"2023-10-14T17:01:14.707094+0000","last_unstale":"2023-10-14T17:01:16.782530+0000","last_undegraded":"2023-10-14T17:01:16.782530+0000","last_fullsized":"2023-10-14T17:01:16.782530+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:12.819926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780728+0000","last_change":"2023-10-14T17:01:13.678974+0000","last_active":"2023-10-14T17:01:16.780728+0000","last_peered":"2023-10-14T17:01:16.780728+0000","last_clean":"2023-10-14T17:01:16.780728+0000","last_became_active":"2023-10-14T17:01:13.678701+0000","last_became_peered":"2023-10-14T17:01:13.678701+0000","last_unstale":"2023-10-14T17:01:16.780728+0000","last_undegraded":"2023-10-14T17:01:16.780728+0000","last_fullsized":"2023-10-14T17:01:16.780728+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:29:06.055561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782660+0000","last_change":"2023-10-14T17:01:14.699556+0000","last_active":"2023-10-14T17:01:16.782660+0000","last_peered":"2023-10-14T17:01:16.782660+0000","last_clean":"2023-10-14T17:01:16.782660+0000","last_became_active":"2023-10-14T17:01:14.699268+0000","last_became_peered":"2023-10-14T17:01:14.699268+0000","last_unstale":"2023-10-14T17:01:16.782660+0000","last_undegraded":"2023-10-14T17:01:16.782660+0000","last_fullsized":"2023-10-14T17:01:16.782660+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:42.187894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688581+0000","last_change":"2023-10-14T17:01:13.667702+0000","last_active":"2023-10-14T17:01:15.688581+0000","last_peered":"2023-10-14T17:01:15.688581+0000","last_clean":"2023-10-14T17:01:15.688581+0000","last_became_active":"2023-10-14T17:01:13.666669+0000","last_became_peered":"2023-10-14T17:01:13.666669+0000","last_unstale":"2023-10-14T17:01:15.688581+0000","last_undegraded":"2023-10-14T17:01:15.688581+0000","last_fullsized":"2023-10-14T17:01:15.688581+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:11.805309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782096+0000","last_change":"2023-10-14T17:01:14.711230+0000","last_active":"2023-10-14T17:01:16.782096+0000","last_peered":"2023-10-14T17:01:16.782096+0000","last_clean":"2023-10-14T17:01:16.782096+0000","last_became_active":"2023-10-14T17:01:14.710905+0000","last_became_peered":"2023-10-14T17:01:14.710905+0000","last_unstale":"2023-10-14T17:01:16.782096+0000","last_undegraded":"2023-10-14T17:01:16.782096+0000","last_fullsized":"2023-10-14T17:01:16.782096+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:25.273827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782125+0000","last_change":"2023-10-14T17:01:13.673875+0000","last_active":"2023-10-14T17:01:16.782125+0000","last_peered":"2023-10-14T17:01:16.782125+0000","last_clean":"2023-10-14T17:01:16.782125+0000","last_became_active":"2023-10-14T17:01:13.673565+0000","last_became_peered":"2023-10-14T17:01:13.673565+0000","last_unstale":"2023-10-14T17:01:16.782125+0000","last_undegraded":"2023-10-14T17:01:16.782125+0000","last_fullsized":"2023-10-14T17:01:16.782125+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:30:07.618113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682606+0000","last_change":"2023-10-14T17:01:14.707548+0000","last_active":"2023-10-14T17:01:15.682606+0000","last_peered":"2023-10-14T17:01:15.682606+0000","last_clean":"2023-10-14T17:01:15.682606+0000","last_became_active":"2023-10-14T17:01:14.703382+0000","last_became_peered":"2023-10-14T17:01:14.703382+0000","last_unstale":"2023-10-14T17:01:15.682606+0000","last_undegraded":"2023-10-14T17:01:15.682606+0000","last_fullsized":"2023-10-14T17:01:15.682606+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:16.813821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781991+0000","last_change":"2023-10-14T17:01:13.674470+0000","last_active":"2023-10-14T17:01:16.781991+0000","last_peered":"2023-10-14T17:01:16.781991+0000","last_clean":"2023-10-14T17:01:16.781991+0000","last_became_active":"2023-10-14T17:01:13.668958+0000","last_became_peered":"2023-10-14T17:01:13.668958+0000","last_unstale":"2023-10-14T17:01:16.781991+0000","last_undegraded":"2023-10-14T17:01:16.781991+0000","last_fullsized":"2023-10-14T17:01:16.781991+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:51.575158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688398+0000","last_change":"2023-10-14T17:01:14.693436+0000","last_active":"2023-10-14T17:01:15.688398+0000","last_peered":"2023-10-14T17:01:15.688398+0000","last_clean":"2023-10-14T17:01:15.688398+0000","last_became_active":"2023-10-14T17:01:14.693115+0000","last_became_peered":"2023-10-14T17:01:14.693115+0000","last_unstale":"2023-10-14T17:01:15.688398+0000","last_undegraded":"2023-10-14T17:01:15.688398+0000","last_fullsized":"2023-10-14T17:01:15.688398+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:02.454338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781498+0000","last_change":"2023-10-14T17:01:13.681995+0000","last_active":"2023-10-14T17:01:16.781498+0000","last_peered":"2023-10-14T17:01:16.781498+0000","last_clean":"2023-10-14T17:01:16.781498+0000","last_became_active":"2023-10-14T17:01:13.681706+0000","last_became_peered":"2023-10-14T17:01:13.681706+0000","last_unstale":"2023-10-14T17:01:16.781498+0000","last_undegraded":"2023-10-14T17:01:16.781498+0000","last_fullsized":"2023-10-14T17:01:16.781498+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:03:40.425410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780092+0000","last_change":"2023-10-14T17:01:14.699116+0000","last_active":"2023-10-14T17:01:16.780092+0000","last_peered":"2023-10-14T17:01:16.780092+0000","last_clean":"2023-10-14T17:01:16.780092+0000","last_became_active":"2023-10-14T17:01:14.698573+0000","last_became_peered":"2023-10-14T17:01:14.698573+0000","last_unstale":"2023-10-14T17:01:16.780092+0000","last_undegraded":"2023-10-14T17:01:16.780092+0000","last_fullsized":"2023-10-14T17:01:16.780092+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:20.512490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780771+0000","last_change":"2023-10-14T17:01:13.675277+0000","last_active":"2023-10-14T17:01:16.780771+0000","last_peered":"2023-10-14T17:01:16.780771+0000","last_clean":"2023-10-14T17:01:16.780771+0000","last_became_active":"2023-10-14T17:01:13.674996+0000","last_became_peered":"2023-10-14T17:01:13.674996+0000","last_unstale":"2023-10-14T17:01:16.780771+0000","last_undegraded":"2023-10-14T17:01:16.780771+0000","last_fullsized":"2023-10-14T17:01:16.780771+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:48.375524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103514+0000","last_change":"2023-10-14T17:01:14.701784+0000","last_active":"2023-10-14T17:01:16.103514+0000","last_peered":"2023-10-14T17:01:16.103514+0000","last_clean":"2023-10-14T17:01:16.103514+0000","last_became_active":"2023-10-14T17:01:14.699415+0000","last_became_peered":"2023-10-14T17:01:14.699415+0000","last_unstale":"2023-10-14T17:01:16.103514+0000","last_undegraded":"2023-10-14T17:01:16.103514+0000","last_fullsized":"2023-10-14T17:01:16.103514+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:59:23.345688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780559+0000","last_change":"2023-10-14T17:01:13.677783+0000","last_active":"2023-10-14T17:01:16.780559+0000","last_peered":"2023-10-14T17:01:16.780559+0000","last_clean":"2023-10-14T17:01:16.780559+0000","last_became_active":"2023-10-14T17:01:13.677532+0000","last_became_peered":"2023-10-14T17:01:13.677532+0000","last_unstale":"2023-10-14T17:01:16.780559+0000","last_undegraded":"2023-10-14T17:01:16.780559+0000","last_fullsized":"2023-10-14T17:01:16.780559+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:22.778177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688962+0000","last_change":"2023-10-14T17:01:14.696006+0000","last_active":"2023-10-14T17:01:15.688962+0000","last_peered":"2023-10-14T17:01:15.688962+0000","last_clean":"2023-10-14T17:01:15.688962+0000","last_became_active":"2023-10-14T17:01:14.695590+0000","last_became_peered":"2023-10-14T17:01:14.695590+0000","last_unstale":"2023-10-14T17:01:15.688962+0000","last_undegraded":"2023-10-14T17:01:15.688962+0000","last_fullsized":"2023-10-14T17:01:15.688962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:19.463438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683255+0000","last_change":"2023-10-14T17:01:13.676826+0000","last_active":"2023-10-14T17:01:15.683255+0000","last_peered":"2023-10-14T17:01:15.683255+0000","last_clean":"2023-10-14T17:01:15.683255+0000","last_became_active":"2023-10-14T17:01:13.676542+0000","last_became_peered":"2023-10-14T17:01:13.676542+0000","last_unstale":"2023-10-14T17:01:15.683255+0000","last_undegraded":"2023-10-14T17:01:15.683255+0000","last_fullsized":"2023-10-14T17:01:15.683255+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:52.400410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683190+0000","last_change":"2023-10-14T17:01:14.710330+0000","last_active":"2023-10-14T17:01:15.683190+0000","last_peered":"2023-10-14T17:01:15.683190+0000","last_clean":"2023-10-14T17:01:15.683190+0000","last_became_active":"2023-10-14T17:01:14.707021+0000","last_became_peered":"2023-10-14T17:01:14.707021+0000","last_unstale":"2023-10-14T17:01:15.683190+0000","last_undegraded":"2023-10-14T17:01:15.683190+0000","last_fullsized":"2023-10-14T17:01:15.683190+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:50.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104208+0000","last_change":"2023-10-14T17:01:13.679044+0000","last_active":"2023-10-14T17:01:16.104208+0000","last_peered":"2023-10-14T17:01:16.104208+0000","last_clean":"2023-10-14T17:01:16.104208+0000","last_became_active":"2023-10-14T17:01:13.678165+0000","last_became_peered":"2023-10-14T17:01:13.678165+0000","last_unstale":"2023-10-14T17:01:16.104208+0000","last_undegraded":"2023-10-14T17:01:16.104208+0000","last_fullsized":"2023-10-14T17:01:16.104208+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:06.024536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781782+0000","last_change":"2023-10-14T17:01:14.696611+0000","last_active":"2023-10-14T17:01:16.781782+0000","last_peered":"2023-10-14T17:01:16.781782+0000","last_clean":"2023-10-14T17:01:16.781782+0000","last_became_active":"2023-10-14T17:01:14.696361+0000","last_became_peered":"2023-10-14T17:01:14.696361+0000","last_unstale":"2023-10-14T17:01:16.781782+0000","last_undegraded":"2023-10-14T17:01:16.781782+0000","last_fullsized":"2023-10-14T17:01:16.781782+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:12.483034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782104+0000","last_change":"2023-10-14T17:01:13.679306+0000","last_active":"2023-10-14T17:01:16.782104+0000","last_peered":"2023-10-14T17:01:16.782104+0000","last_clean":"2023-10-14T17:01:16.782104+0000","last_became_active":"2023-10-14T17:01:13.677185+0000","last_became_peered":"2023-10-14T17:01:13.677185+0000","last_unstale":"2023-10-14T17:01:16.782104+0000","last_undegraded":"2023-10-14T17:01:16.782104+0000","last_fullsized":"2023-10-14T17:01:16.782104+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:03:03.127006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688317+0000","last_change":"2023-10-14T17:01:14.703929+0000","last_active":"2023-10-14T17:01:15.688317+0000","last_peered":"2023-10-14T17:01:15.688317+0000","last_clean":"2023-10-14T17:01:15.688317+0000","last_became_active":"2023-10-14T17:01:14.703593+0000","last_became_peered":"2023-10-14T17:01:14.703593+0000","last_unstale":"2023-10-14T17:01:15.688317+0000","last_undegraded":"2023-10-14T17:01:15.688317+0000","last_fullsized":"2023-10-14T17:01:15.688317+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:23.174484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780956+0000","last_change":"2023-10-14T17:01:13.678873+0000","last_active":"2023-10-14T17:01:16.780956+0000","last_peered":"2023-10-14T17:01:16.780956+0000","last_clean":"2023-10-14T17:01:16.780956+0000","last_became_active":"2023-10-14T17:01:13.678510+0000","last_became_peered":"2023-10-14T17:01:13.678510+0000","last_unstale":"2023-10-14T17:01:16.780956+0000","last_undegraded":"2023-10-14T17:01:16.780956+0000","last_fullsized":"2023-10-14T17:01:16.780956+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:41:02.658093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103500+0000","last_change":"2023-10-14T17:01:14.695952+0000","last_active":"2023-10-14T17:01:16.103500+0000","last_peered":"2023-10-14T17:01:16.103500+0000","last_clean":"2023-10-14T17:01:16.103500+0000","last_became_active":"2023-10-14T17:01:14.694848+0000","last_became_peered":"2023-10-14T17:01:14.694848+0000","last_unstale":"2023-10-14T17:01:16.103500+0000","last_undegraded":"2023-10-14T17:01:16.103500+0000","last_fullsized":"2023-10-14T17:01:16.103500+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:42.187894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781783+0000","last_change":"2023-10-14T17:01:13.678809+0000","last_active":"2023-10-14T17:01:16.781783+0000","last_peered":"2023-10-14T17:01:16.781783+0000","last_clean":"2023-10-14T17:01:16.781783+0000","last_became_active":"2023-10-14T17:01:13.678239+0000","last_became_peered":"2023-10-14T17:01:13.678239+0000","last_unstale":"2023-10-14T17:01:16.781783+0000","last_undegraded":"2023-10-14T17:01:16.781783+0000","last_fullsized":"2023-10-14T17:01:16.781783+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:51:27.959578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782058+0000","last_change":"2023-10-14T17:01:14.699332+0000","last_active":"2023-10-14T17:01:16.782058+0000","last_peered":"2023-10-14T17:01:16.782058+0000","last_clean":"2023-10-14T17:01:16.782058+0000","last_became_active":"2023-10-14T17:01:14.699029+0000","last_became_peered":"2023-10-14T17:01:14.699029+0000","last_unstale":"2023-10-14T17:01:16.782058+0000","last_undegraded":"2023-10-14T17:01:16.782058+0000","last_fullsized":"2023-10-14T17:01:16.782058+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:39:25.218441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103691+0000","last_change":"2023-10-14T17:01:13.681545+0000","last_active":"2023-10-14T17:01:16.103691+0000","last_peered":"2023-10-14T17:01:16.103691+0000","last_clean":"2023-10-14T17:01:16.103691+0000","last_became_active":"2023-10-14T17:01:13.681079+0000","last_became_peered":"2023-10-14T17:01:13.681079+0000","last_unstale":"2023-10-14T17:01:16.103691+0000","last_undegraded":"2023-10-14T17:01:16.103691+0000","last_fullsized":"2023-10-14T17:01:16.103691+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:46.107843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780679+0000","last_change":"2023-10-14T17:01:14.698436+0000","last_active":"2023-10-14T17:01:16.780679+0000","last_peered":"2023-10-14T17:01:16.780679+0000","last_clean":"2023-10-14T17:01:16.780679+0000","last_became_active":"2023-10-14T17:01:14.697004+0000","last_became_peered":"2023-10-14T17:01:14.697004+0000","last_unstale":"2023-10-14T17:01:16.780679+0000","last_undegraded":"2023-10-14T17:01:16.780679+0000","last_fullsized":"2023-10-14T17:01:16.780679+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:25.252771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780987+0000","last_change":"2023-10-14T17:01:13.682552+0000","last_active":"2023-10-14T17:01:16.780987+0000","last_peered":"2023-10-14T17:01:16.780987+0000","last_clean":"2023-10-14T17:01:16.780987+0000","last_became_active":"2023-10-14T17:01:13.681681+0000","last_became_peered":"2023-10-14T17:01:13.681681+0000","last_unstale":"2023-10-14T17:01:16.780987+0000","last_undegraded":"2023-10-14T17:01:16.780987+0000","last_fullsized":"2023-10-14T17:01:16.780987+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:08.135964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781493+0000","last_change":"2023-10-14T17:01:14.693152+0000","last_active":"2023-10-14T17:01:16.781493+0000","last_peered":"2023-10-14T17:01:16.781493+0000","last_clean":"2023-10-14T17:01:16.781493+0000","last_became_active":"2023-10-14T17:01:14.692791+0000","last_became_peered":"2023-10-14T17:01:14.692791+0000","last_unstale":"2023-10-14T17:01:16.781493+0000","last_undegraded":"2023-10-14T17:01:16.781493+0000","last_fullsized":"2023-10-14T17:01:16.781493+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:03:24.860980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780680+0000","last_change":"2023-10-14T17:01:13.672545+0000","last_active":"2023-10-14T17:01:16.780680+0000","last_peered":"2023-10-14T17:01:16.780680+0000","last_clean":"2023-10-14T17:01:16.780680+0000","last_became_active":"2023-10-14T17:01:13.672246+0000","last_became_peered":"2023-10-14T17:01:13.672246+0000","last_unstale":"2023-10-14T17:01:16.780680+0000","last_undegraded":"2023-10-14T17:01:16.780680+0000","last_fullsized":"2023-10-14T17:01:16.780680+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:04.039372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103486+0000","last_change":"2023-10-14T17:01:14.706496+0000","last_active":"2023-10-14T17:01:16.103486+0000","last_peered":"2023-10-14T17:01:16.103486+0000","last_clean":"2023-10-14T17:01:16.103486+0000","last_became_active":"2023-10-14T17:01:14.705848+0000","last_became_peered":"2023-10-14T17:01:14.705848+0000","last_unstale":"2023-10-14T17:01:16.103486+0000","last_undegraded":"2023-10-14T17:01:16.103486+0000","last_fullsized":"2023-10-14T17:01:16.103486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:42.434427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103869+0000","last_change":"2023-10-14T17:01:13.679084+0000","last_active":"2023-10-14T17:01:16.103869+0000","last_peered":"2023-10-14T17:01:16.103869+0000","last_clean":"2023-10-14T17:01:16.103869+0000","last_became_active":"2023-10-14T17:01:13.678707+0000","last_became_peered":"2023-10-14T17:01:13.678707+0000","last_unstale":"2023-10-14T17:01:16.103869+0000","last_undegraded":"2023-10-14T17:01:16.103869+0000","last_fullsized":"2023-10-14T17:01:16.103869+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:45.393854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"142'249","reported_seq":756,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T17:01:14.673941+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:01:14.673941+0000","last_peered":"2023-10-14T17:01:14.673941+0000","last_clean":"2023-10-14T17:01:14.673941+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:01:14.673941+0000","last_undegraded":"2023-10-14T17:01:14.673941+0000","last_fullsized":"2023-10-14T17:01:14.673941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37816,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334024,"statfs":{"total":96636764160,"available":96598040576,"internally_reserved":0,"allocated":2535424,"data_stored":1523528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1319999999999999,"15min":1.1599999999999999},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.631,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.1799999999999999},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.38900000000000001},"max":{"1min":1.6180000000000001,"5min":1.871,"15min":2.117},"last":1.452}]},{"osd":1,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.159,"15min":1.216},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8260000000000001,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.147,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.64500000000000002,"5min":0.61499999999999999,"15min":0.57799999999999996},"max":{"1min":1.7370000000000001,"5min":1.9870000000000001,"15min":2.0259999999999998},"last":1.738}]},{"osd":2,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.149,"15min":1.2130000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.47099999999999997},"max":{"1min":1.6679999999999999,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2549999999999999,"15min":1.2909999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.72599999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.1589999999999998},"last":0.998}]},{"osd":3,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.278,"15min":1.351},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6339999999999999,"5min":2.056,"15min":2.4159999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4039999999999999,"15min":1.444},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.056,"5min":2.379,"15min":2.379},"last":1.1619999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.306,"15min":1.278},"min":{"1min":0.59099999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.7230000000000001,"5min":2.0960000000000001,"15min":2.1819999999999999},"last":2.0790000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.339,"15min":1.3360000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.99,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.6779999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3959999999999999,"15min":1.3859999999999999},"min":{"1min":0.51200000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.96,"5min":2.278,"15min":2.278},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.444,"15min":1.456},"min":{"1min":0.70399999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.1240000000000001,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.9359999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5309999999999999,"15min":1.536},"min":{"1min":0.63200000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.8560000000000001,"5min":2.492,"15min":2.492},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4970000000000001,"15min":1.482},"min":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8979999999999999,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.9670000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575095,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37224,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334616,"statfs":{"total":96636764160,"available":96598646784,"internally_reserved":0,"allocated":1929216,"data_stored":932522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.1779999999999999,"15min":1.1659999999999999},"min":{"1min":0.65700000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":10.119,"5min":10.119,"15min":10.119},"last":1.262},{"interface":"front","average":{"1min":1.472,"5min":1.2070000000000001,"15min":1.2210000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":8.6329999999999991,"5min":8.6329999999999991,"15min":8.6329999999999991},"last":0.86199999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.69299999999999995,"5min":0.67000000000000004,"15min":0.41299999999999998},"max":{"1min":9.6869999999999994,"5min":9.6869999999999994,"15min":9.6869999999999994},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.306,"15min":1.302},"min":{"1min":0.69899999999999995,"5min":0.53900000000000003,"15min":0.438},"max":{"1min":10.231,"5min":10.231,"15min":10.231},"last":1.863}]},{"osd":2,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.343,"15min":1.355},"min":{"1min":0.95299999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.96,"5min":6.96,"15min":6.96},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.395,"15min":1.4039999999999999},"min":{"1min":0.90000000000000002,"5min":0.72399999999999998,"15min":0.46000000000000002},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.3939999999999999,"15min":1.385},"min":{"1min":0.85699999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":10.307,"5min":10.307,"15min":10.307},"last":1.492},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4019999999999999,"15min":1.4239999999999999},"min":{"1min":0.85799999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":8.4830000000000005,"5min":8.4830000000000005,"15min":8.4830000000000005},"last":1.099}]},{"osd":4,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":1.4219999999999999,"15min":1.371},"min":{"1min":0.97399999999999998,"5min":0.497,"15min":0.497},"max":{"1min":10.48,"5min":10.48,"15min":10.48},"last":1.74},{"interface":"front","average":{"1min":1.8100000000000001,"5min":1.4890000000000001,"15min":1.4490000000000001},"min":{"1min":1.0309999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":9.2230000000000008,"5min":9.2230000000000008,"15min":9.2230000000000008},"last":1.3640000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.4430000000000001,"15min":1.409},"min":{"1min":0.88200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.988,"5min":1.611,"15min":1.5760000000000001},"min":{"1min":1.2549999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":10.388999999999999,"5min":10.388999999999999,"15min":10.388999999999999},"last":1.9910000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":1.601,"15min":1.6000000000000001},"min":{"1min":1.2569999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.8179999999999996,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.579,"15min":1.554},"min":{"1min":1.1990000000000001,"5min":0.623,"15min":0.623},"max":{"1min":9.9789999999999992,"5min":9.9789999999999992,"15min":9.9789999999999992},"last":1.6759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575096,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":924650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0640000000000001,"15min":1.103},"min":{"1min":0.623,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.829,"5min":1.829,"15min":1.986},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.242,"5min":1.1799999999999999,"15min":1.149},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.0720000000000001}]},{"osd":1,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.214,"15min":1.1910000000000001},"min":{"1min":0.78800000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.177,"15min":1.208},"min":{"1min":0.71499999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.3250000000000002,"5min":3.3250000000000002,"15min":3.3250000000000002},"last":0.92900000000000005}]},{"osd":2,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.115,"15min":1.177},"min":{"1min":0.80500000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.625,"5min":1.681,"15min":2.375},"last":0.748},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.292,"15min":1.3149999999999999},"min":{"1min":0.82899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.3100000000000001},"last":1.0089999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.377,"15min":1.3839999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.3320000000000001,"15min":1.3899999999999999},"min":{"1min":0.877,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":1.1140000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3440000000000001,"15min":1.296},"min":{"1min":0.66400000000000003,"5min":0.434,"15min":0.434},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.4889999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.401,"15min":1.357},"min":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.891,"5min":3.891,"15min":3.891},"last":1.3720000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.345,"15min":1.3700000000000001},"min":{"1min":0.90500000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":6.6600000000000001,"5min":6.6600000000000001,"15min":6.6600000000000001},"last":1.218},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.532,"15min":1.5309999999999999},"min":{"1min":0.875,"5min":0.63,"15min":0.63},"max":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":1.4550000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.502,"15min":1.5049999999999999},"min":{"1min":0.82799999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.554},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.5049999999999999,"15min":1.514},"min":{"1min":1.0369999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":6.5359999999999996,"5min":6.5359999999999996,"15min":6.5359999999999996},"last":1.4159999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575096,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":930980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1379999999999999,"15min":1.1659999999999999},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":1.643,"5min":1.9319999999999999,"15min":1.982},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.2330000000000001,"15min":1.23},"min":{"1min":0.73599999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8029999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.754}]},{"osd":1,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.306,"15min":1.2809999999999999},"min":{"1min":0.85999999999999999,"5min":0.69099999999999995,"15min":0.48199999999999998},"max":{"1min":1.607,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.212,"15min":1.2390000000000001},"min":{"1min":0.92000000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7430000000000001,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.355}]},{"osd":2,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.248,"15min":1.2410000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.64700000000000002},"max":{"1min":1.865,"5min":5.633,"15min":5.633},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.84999999999999998,"5min":0.81100000000000005,"15min":0.71999999999999997},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.302}]},{"osd":3,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3100000000000001,"15min":1.3160000000000001},"min":{"1min":1.1240000000000001,"5min":0.77100000000000002,"15min":0.71699999999999997},"max":{"1min":1.825,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.238},{"interface":"front","average":{"1min":1.343,"5min":1.405,"15min":1.403},"min":{"1min":0.92400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.0779999999999998,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.6459999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4179999999999999,"15min":1.375},"min":{"1min":1.032,"5min":0.64500000000000002,"15min":0.52900000000000003},"max":{"1min":1.877,"5min":2.931,"15min":2.931},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.361,"15min":1.343},"min":{"1min":1.0920000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.732,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.5609999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4330000000000001,"15min":1.399},"min":{"1min":1.238,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.768,"5min":2.323,"15min":2.323},"last":1.526},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.544,"15min":1.53},"min":{"1min":1.125,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.988,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.595}]},{"osd":7,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.536,"15min":1.492},"min":{"1min":1.256,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.9790000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.556,"15min":1.5529999999999999},"min":{"1min":1.1659999999999999,"5min":0.82899999999999996,"15min":0.79000000000000004},"max":{"1min":1.804,"5min":2.004,"15min":2.3130000000000002},"last":1.397}]}]},{"osd":3,"up_from":13,"seq":55834575096,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45528,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326312,"statfs":{"total":96636764160,"available":96590143488,"internally_reserved":0,"allocated":1912832,"data_stored":924698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44693960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.90700000000000003,"15min":0.92200000000000004},"min":{"1min":0.39900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.079},{"interface":"front","average":{"1min":1.143,"5min":1.0629999999999999,"15min":1.1220000000000001},"min":{"1min":0.48099999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.90700000000000003,"15min":0.91400000000000003},"min":{"1min":0.51000000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.048},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0740000000000001,"15min":1.1000000000000001},"min":{"1min":0.53200000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.97299999999999998}]},{"osd":2,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.56000000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.9910000000000001,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.198,"15min":1.248},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.6859999999999999,"5min":1.883,"15min":2.0960000000000001},"last":1.2250000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3320000000000001,"15min":1.353},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":3.4350000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3620000000000001,"15min":1.3979999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8320000000000001,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.351}]},{"osd":5,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.339,"15min":1.3779999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.516},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.4179999999999999,"15min":1.4390000000000001},"min":{"1min":0.88100000000000001,"5min":0.82699999999999996,"15min":0.80600000000000005},"max":{"1min":2.7360000000000002,"5min":2.7360000000000002,"15min":3.7410000000000001},"last":1.7829999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.46,"15min":1.462},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.4129999999999998,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.5620000000000001,"15min":1.5469999999999999},"min":{"1min":0.93899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.5089999999999999,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.5029999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.464,"15min":1.484},"min":{"1min":0.90000000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.871},"last":1.718},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.506,"15min":1.536},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.907},"last":1.5349999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575096,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46948,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324892,"statfs":{"total":96636764160,"available":96588689408,"internally_reserved":0,"allocated":1925120,"data_stored":931526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.32400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.6879999999999999,"5min":1.706,"15min":1.706},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.98199999999999998,"15min":1.03},"min":{"1min":0.55000000000000004,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.0189999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.874,"15min":0.95699999999999996},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":1.532,"5min":1.544,"15min":2.0419999999999998},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.083,"15min":1.161},"min":{"1min":0.53100000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.909,"5min":1.909,"15min":2.069},"last":0.92300000000000004}]},{"osd":3,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.105,"15min":1.196},"min":{"1min":0.49199999999999999,"5min":0.48999999999999999,"15min":0.46500000000000002},"max":{"1min":2.3769999999999998,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2809999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":1.75,"5min":1.75,"15min":2.0459999999999998},"last":1.3660000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3160000000000001,"15min":1.3959999999999999},"min":{"1min":0.78200000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3159999999999998},"last":1.536},{"interface":"front","average":{"1min":1.208,"5min":1.268,"15min":1.3460000000000001},"min":{"1min":0.88600000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.508}]},{"osd":5,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.2809999999999999,"15min":1.3720000000000001},"min":{"1min":0.76000000000000001,"5min":0.67200000000000004,"15min":0.59199999999999997},"max":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.508,"15min":1.536},"min":{"1min":0.97199999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":1.476}]},{"osd":6,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.4279999999999999,"15min":1.444},"min":{"1min":0.74399999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":7.0099999999999998,"5min":7.0099999999999998,"15min":7.0099999999999998},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.4810000000000001,"15min":1.5249999999999999},"min":{"1min":1.0089999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":7.3179999999999996,"5min":7.3179999999999996,"15min":7.3179999999999996},"last":1.4219999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.4339999999999999,"15min":1.506},"min":{"1min":1.042,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.607},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.4670000000000001,"15min":1.5660000000000001},"min":{"1min":1,"5min":0.624,"15min":0.624},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.71}]}]},{"osd":1,"up_from":13,"seq":55834575096,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":1904640,"data_stored":924538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.85399999999999998,"15min":0.89800000000000002},"min":{"1min":0.441,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.6759999999999999,"5min":2.113,"15min":2.113},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":1.0369999999999999},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.0819999999999999},"last":0.629}]},{"osd":2,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88500000000000001,"15min":0.93899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5449999999999999,"5min":1.7509999999999999,"15min":1.758},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.0820000000000001,"15min":1.1459999999999999},"min":{"1min":0.59199999999999997,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.423,"5min":2.008,"15min":2.0150000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.167,"15min":1.2649999999999999},"min":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.615,"5min":1.984,"15min":2.198},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0800000000000001,"15min":1.165},"min":{"1min":0.495,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.974,"15min":2.004},"last":0.995}]},{"osd":4,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3089999999999999,"15min":1.385},"min":{"1min":0.72899999999999998,"5min":0.628,"15min":0.628},"max":{"1min":1.8140000000000001,"5min":2.052,"15min":2.3050000000000002},"last":1.429},{"interface":"front","average":{"1min":1.206,"5min":1.2769999999999999,"15min":1.3},"min":{"1min":0.83699999999999997,"5min":0.64100000000000001,"15min":0.53900000000000003},"max":{"1min":1.6319999999999999,"5min":1.9119999999999999,"15min":1.9430000000000001},"last":1.3280000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3029999999999999,"15min":1.345},"min":{"1min":0.754,"5min":0.754,"15min":0.63100000000000001},"max":{"1min":1.9390000000000001,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":1.393},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3859999999999999,"15min":1.444},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.76200000000000001},"max":{"1min":1.8080000000000001,"5min":2.1480000000000001,"15min":2.302},"last":1.5800000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.405,"15min":1.4710000000000001},"min":{"1min":0.77300000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.839,"5min":2.121,"15min":2.4169999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4299999999999999,"15min":1.458},"min":{"1min":0.84799999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.8580000000000001,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":1.607}]},{"osd":7,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4670000000000001,"15min":1.5389999999999999},"min":{"1min":0.92100000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.891,"5min":2.2410000000000001,"15min":2.411},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4670000000000001,"15min":1.552},"min":{"1min":0.871,"5min":0.81899999999999995,"15min":0.61599999999999999},"max":{"1min":1.976,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":1.6459999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575095,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2498560,"data_stored":1514906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90500000000000003,"15min":0.90400000000000003},"min":{"1min":0.46700000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.577,"5min":3.355,"15min":3.355},"last":0.433},{"interface":"front","average":{"1min":1.05,"5min":0.98099999999999998,"15min":1.0649999999999999},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.9219999999999999,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.016,"15min":1.071},"min":{"1min":0.53700000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3720000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0549999999999999,"15min":1.1639999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.8149999999999999,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.755}]},{"osd":3,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.1970000000000001,"15min":1.2689999999999999},"min":{"1min":0.59299999999999997,"5min":0.502,"15min":0.502},"max":{"1min":1.6870000000000001,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.3220000000000001},"min":{"1min":0.55400000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0550000000000002,"5min":2.581,"15min":2.593},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2470000000000001,"15min":1.272},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7290000000000001,"5min":5.335,"15min":5.335},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.363,"15min":1.4079999999999999},"min":{"1min":0.86299999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9490000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.254}]},{"osd":5,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.24,"15min":1.3},"min":{"1min":0.83499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.2160000000000002,"5min":2.931,"15min":2.931},"last":1.21},{"interface":"front","average":{"1min":1.405,"5min":1.3640000000000001,"15min":1.4790000000000001},"min":{"1min":0.92300000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.9790000000000001,"5min":3.306,"15min":3.306},"last":0.95299999999999996}]},{"osd":6,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.46,"15min":1.5589999999999999},"min":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":2.1619999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.996},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3280000000000001,"15min":1.3859999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.911,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.03}]},{"osd":7,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.462,"15min":1.5349999999999999},"min":{"1min":0.78900000000000003,"5min":0.747,"15min":0.747},"max":{"1min":1.8160000000000001,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4339999999999999,"15min":1.508},"min":{"1min":0.71599999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.113,"5min":3.698,"15min":3.698},"last":0.96899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:01:19.187 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:01:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.449+0000 7f35f691d640 1 Processor -- start 2023-10-14T17:01:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.449+0000 7f35f691d640 1 -- start start 2023-10-14T17:01:19.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.451+0000 7f35f691d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 0x7f35f015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.451+0000 7f35f691d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f015ed60 con 0x7f35f015aaa0 2023-10-14T17:01:19.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.451+0000 7f35f691d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35f00707e0 con 0x7f35f015ae80 2023-10-14T17:01:19.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.451+0000 7f35f691d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f006b810 con 0x7f35f00ce390 2023-10-14T17:01:19.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35f015aaa0 0x7f35f00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54030/0 (socket says 172.21.15.29:54030) 2023-10-14T17:01:19.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35effff640 1 -- 172.21.15.29:0/4285981093 learned_addr learned my addr 172.21.15.29:0/4285981093 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35f4eba640 1 --2- 172.21.15.29:0/4285981093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 0x7f35f015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35eeffd640 1 -- 172.21.15.29:0/4285981093 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53149954 0 0) 0x7f35f015ed60 con 0x7f35f015aaa0 2023-10-14T17:01:19.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35eeffd640 1 -- 172.21.15.29:0/4285981093 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35d8003520 con 0x7f35f015aaa0 2023-10-14T17:01:19.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35f4eba640 1 -- 172.21.15.29:0/4285981093 >> v1:172.21.15.186:6789/0 conn(0x7f35f015ae80 legacy=0x7f35f0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.452+0000 7f35f4eba640 1 -- 172.21.15.29:0/4285981093 >> v1:172.21.15.29:6789/0 conn(0x7f35f015aaa0 legacy=0x7f35f00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.453+0000 7f35f4eba640 1 -- 172.21.15.29:0/4285981093 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35f015ef90 con 0x7f35f00ce390 2023-10-14T17:01:19.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.454+0000 7f35f4eba640 1 --2- 172.21.15.29:0/4285981093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 0x7f35f015e820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f35e80097d0 tx=0x7f35e802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80b8a2b9b0a645b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35eeffd640 1 -- 172.21.15.29:0/4285981093 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8031890 con 0x7f35f00ce390 2023-10-14T17:01:19.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35eeffd640 1 -- 172.21.15.29:0/4285981093 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e80319f0 con 0x7f35f00ce390 2023-10-14T17:01:19.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35eeffd640 1 -- 172.21.15.29:0/4285981093 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e8031d60 con 0x7f35f00ce390 2023-10-14T17:01:19.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 -- 172.21.15.29:0/4285981093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 msgr2=0x7f35f015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 --2- 172.21.15.29:0/4285981093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 0x7f35f015e820 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f35e80097d0 tx=0x7f35e802b930 comp rx=0 tx=0).stop 2023-10-14T17:01:19.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 -- 172.21.15.29:0/4285981093 shutdown_connections 2023-10-14T17:01:19.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 --2- 172.21.15.29:0/4285981093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f00ce390 0x7f35f015e820 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 -- 172.21.15.29:0/4285981093 >> 172.21.15.29:0/4285981093 conn(0x7f35f00cb410 msgr2=0x7f35f00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:19.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 -- 172.21.15.29:0/4285981093 shutdown_connections 2023-10-14T17:01:19.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.455+0000 7f35f691d640 1 -- 172.21.15.29:0/4285981093 wait complete. 2023-10-14T17:01:19.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.457+0000 7f35f691d640 1 Processor -- start 2023-10-14T17:01:19.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.457+0000 7f35f691d640 1 -- start start 2023-10-14T17:01:19.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35f015aaa0 0x7f35f00b2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 0x7f35f00b2880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00ad080 con 0x7f35f00ce390 2023-10-14T17:01:19.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c8570 con 0x7f35f015aaa0 2023-10-14T17:01:19.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.458+0000 7f35f691d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35f00c8850 con 0x7f35f015ae80 2023-10-14T17:01:19.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47346/0 (socket says 172.21.15.29:47346) 2023-10-14T17:01:19.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 -- 172.21.15.29:0/3598461873 learned_addr learned my addr 172.21.15.29:0/3598461873 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:19.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35ef7fe640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35f015aaa0 0x7f35f00b2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35f4eba640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 0x7f35f00b2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 -- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 msgr2=0x7f35f00b2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 0x7f35f00b2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 -- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35f015aaa0 msgr2=0x7f35f00b2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35f015aaa0 0x7f35f00b2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35effff640 1 -- 172.21.15.29:0/3598461873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e8009400 con 0x7f35f00ce390 2023-10-14T17:01:19.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.459+0000 7f35f4eba640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 0x7f35f00b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:19.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35effff640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f35e000cbb0 tx=0x7f35e000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e000ad40 con 0x7f35f00ce390 2023-10-14T17:01:19.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0012400 con 0x7f35f00ce390 2023-10-14T17:01:19.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35e000aea0 con 0x7f35f00ce390 2023-10-14T17:01:19.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35f01f6360 con 0x7f35f00ce390 2023-10-14T17:01:19.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.460+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35f01f6920 con 0x7f35f00ce390 2023-10-14T17:01:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.461+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35f004e160 con 0x7f35f00ce390 2023-10-14T17:01:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.462+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f35e0007070 con 0x7f35f00ce390 2023-10-14T17:01:19.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.463+0000 7f35ecff9640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 0x7f35dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.464+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f35e0089690 con 0x7f35f00ce390 2023-10-14T17:01:19.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.464+0000 7f35ef7fe640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 0x7f35dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.464+0000 7f35ef7fe640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 0x7f35dc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f35e4007fd0 tx=0x7f35e4009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.468+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f35e0054720 con 0x7f35f00ce390 2023-10-14T17:01:19.608 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-14T17:01:19.608+0000 7fdd2a223640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:01:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.660+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 --> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35f015f550 con 0x7f35dc06af90 2023-10-14T17:01:19.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.672+0000 7f35ecff9640 1 -- 172.21.15.29:0/3598461873 <== mgr.8862 v2:172.21.15.29:6832/3121942650 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287351 (secure 0 0 0) 0x7f35e40050d0 con 0x7f35dc06af90 2023-10-14T17:01:19.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:19.678 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:01:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 msgr2=0x7f35dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 0x7f35dc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f35e4007fd0 tx=0x7f35e4009040 comp rx=0 tx=0).stop 2023-10-14T17:01:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 msgr2=0x7f35f01f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 secure :-1 s=READY pgs=1801 cs=0 l=1 rev1=1 crypto rx=0x7f35e000cbb0 tx=0x7f35e000a430 comp rx=0 tx=0).stop 2023-10-14T17:01:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 shutdown_connections 2023-10-14T17:01:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f35dc06af90 0x7f35dc06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35f015ae80 0x7f35f00b2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35f015aaa0 0x7f35f00b2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 --2- 172.21.15.29:0/3598461873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35f00ce390 0x7f35f01f6e40 unknown :-1 s=CLOSED pgs=1801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 >> 172.21.15.29:0/3598461873 conn(0x7f35f00cb410 msgr2=0x7f35f00b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 shutdown_connections 2023-10-14T17:01:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.677+0000 7f35f691d640 1 -- 172.21.15.29:0/3598461873 wait complete. 2023-10-14T17:01:19.695 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T17:01:17.783896+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":443,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":287,"ondisk_log_size":287,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":118,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":365520,"kb_used_data":16152,"kb_used_omap":12,"kb_used_meta":349235,"kb_avail":754609200,"statfs":{"total":773094113280,"available":772719820800,"internally_reserved":0,"allocated":16539648,"data_stored":8607348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":357617208},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.147386"},"pg_stats":[{"pgid":"19.3f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683173+0000","last_change":"2023-10-14T17:01:14.708690+0000","last_active":"2023-10-14T17:01:15.683173+0000","last_peered":"2023-10-14T17:01:15.683173+0000","last_clean":"2023-10-14T17:01:15.683173+0000","last_became_active":"2023-10-14T17:01:14.706344+0000","last_became_peered":"2023-10-14T17:01:14.706344+0000","last_unstale":"2023-10-14T17:01:15.683173+0000","last_undegraded":"2023-10-14T17:01:15.683173+0000","last_fullsized":"2023-10-14T17:01:15.683173+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:15:18.416581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781172+0000","last_change":"2023-10-14T17:01:14.706736+0000","last_active":"2023-10-14T17:01:16.781172+0000","last_peered":"2023-10-14T17:01:16.781172+0000","last_clean":"2023-10-14T17:01:16.781172+0000","last_became_active":"2023-10-14T17:01:14.706410+0000","last_became_peered":"2023-10-14T17:01:14.706410+0000","last_unstale":"2023-10-14T17:01:16.781172+0000","last_undegraded":"2023-10-14T17:01:16.781172+0000","last_fullsized":"2023-10-14T17:01:16.781172+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:48:16.187550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691514+0000","last_change":"2023-10-14T17:01:14.699733+0000","last_active":"2023-10-14T17:01:15.691514+0000","last_peered":"2023-10-14T17:01:15.691514+0000","last_clean":"2023-10-14T17:01:15.691514+0000","last_became_active":"2023-10-14T17:01:14.699434+0000","last_became_peered":"2023-10-14T17:01:14.699434+0000","last_unstale":"2023-10-14T17:01:15.691514+0000","last_undegraded":"2023-10-14T17:01:15.691514+0000","last_fullsized":"2023-10-14T17:01:15.691514+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:50.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.3c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682866+0000","last_change":"2023-10-14T17:01:14.704804+0000","last_active":"2023-10-14T17:01:15.682866+0000","last_peered":"2023-10-14T17:01:15.682866+0000","last_clean":"2023-10-14T17:01:15.682866+0000","last_became_active":"2023-10-14T17:01:14.704528+0000","last_became_peered":"2023-10-14T17:01:14.704528+0000","last_unstale":"2023-10-14T17:01:15.682866+0000","last_undegraded":"2023-10-14T17:01:15.682866+0000","last_fullsized":"2023-10-14T17:01:15.682866+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:13.440233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.3b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103731+0000","last_change":"2023-10-14T17:01:14.691121+0000","last_active":"2023-10-14T17:01:16.103731+0000","last_peered":"2023-10-14T17:01:16.103731+0000","last_clean":"2023-10-14T17:01:16.103731+0000","last_became_active":"2023-10-14T17:01:14.690818+0000","last_became_peered":"2023-10-14T17:01:14.690818+0000","last_unstale":"2023-10-14T17:01:16.103731+0000","last_undegraded":"2023-10-14T17:01:16.103731+0000","last_fullsized":"2023-10-14T17:01:16.103731+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:23.272935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.3a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688303+0000","last_change":"2023-10-14T17:01:14.716137+0000","last_active":"2023-10-14T17:01:15.688303+0000","last_peered":"2023-10-14T17:01:15.688303+0000","last_clean":"2023-10-14T17:01:15.688303+0000","last_became_active":"2023-10-14T17:01:14.715782+0000","last_became_peered":"2023-10-14T17:01:14.715782+0000","last_unstale":"2023-10-14T17:01:15.688303+0000","last_undegraded":"2023-10-14T17:01:15.688303+0000","last_fullsized":"2023-10-14T17:01:15.688303+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:12:01.894640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.39","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.687528+0000","last_change":"2023-10-14T17:01:14.694354+0000","last_active":"2023-10-14T17:01:15.687528+0000","last_peered":"2023-10-14T17:01:15.687528+0000","last_clean":"2023-10-14T17:01:15.687528+0000","last_became_active":"2023-10-14T17:01:14.694020+0000","last_became_peered":"2023-10-14T17:01:14.694020+0000","last_unstale":"2023-10-14T17:01:15.687528+0000","last_undegraded":"2023-10-14T17:01:15.687528+0000","last_fullsized":"2023-10-14T17:01:15.687528+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:01.039334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.38","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690444+0000","last_change":"2023-10-14T17:01:14.684523+0000","last_active":"2023-10-14T17:01:15.690444+0000","last_peered":"2023-10-14T17:01:15.690444+0000","last_clean":"2023-10-14T17:01:15.690444+0000","last_became_active":"2023-10-14T17:01:14.684255+0000","last_became_peered":"2023-10-14T17:01:14.684255+0000","last_unstale":"2023-10-14T17:01:15.690444+0000","last_undegraded":"2023-10-14T17:01:15.690444+0000","last_fullsized":"2023-10-14T17:01:15.690444+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:13.440233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.37","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691657+0000","last_change":"2023-10-14T17:01:14.698523+0000","last_active":"2023-10-14T17:01:15.691657+0000","last_peered":"2023-10-14T17:01:15.691657+0000","last_clean":"2023-10-14T17:01:15.691657+0000","last_became_active":"2023-10-14T17:01:14.698191+0000","last_became_peered":"2023-10-14T17:01:14.698191+0000","last_unstale":"2023-10-14T17:01:15.691657+0000","last_undegraded":"2023-10-14T17:01:15.691657+0000","last_fullsized":"2023-10-14T17:01:15.691657+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:14.490172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.36","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781353+0000","last_change":"2023-10-14T17:01:14.698747+0000","last_active":"2023-10-14T17:01:16.781353+0000","last_peered":"2023-10-14T17:01:16.781353+0000","last_clean":"2023-10-14T17:01:16.781353+0000","last_became_active":"2023-10-14T17:01:14.697703+0000","last_became_peered":"2023-10-14T17:01:14.697703+0000","last_unstale":"2023-10-14T17:01:16.781353+0000","last_undegraded":"2023-10-14T17:01:16.781353+0000","last_fullsized":"2023-10-14T17:01:16.781353+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:17:15.318116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.35","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781023+0000","last_change":"2023-10-14T17:01:14.705837+0000","last_active":"2023-10-14T17:01:16.781023+0000","last_peered":"2023-10-14T17:01:16.781023+0000","last_clean":"2023-10-14T17:01:16.781023+0000","last_became_active":"2023-10-14T17:01:14.705512+0000","last_became_peered":"2023-10-14T17:01:14.705512+0000","last_unstale":"2023-10-14T17:01:16.781023+0000","last_undegraded":"2023-10-14T17:01:16.781023+0000","last_fullsized":"2023-10-14T17:01:16.781023+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:08.038231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.34","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691163+0000","last_change":"2023-10-14T17:01:14.700479+0000","last_active":"2023-10-14T17:01:15.691163+0000","last_peered":"2023-10-14T17:01:15.691163+0000","last_clean":"2023-10-14T17:01:15.691163+0000","last_became_active":"2023-10-14T17:01:14.700212+0000","last_became_peered":"2023-10-14T17:01:14.700212+0000","last_unstale":"2023-10-14T17:01:15.691163+0000","last_undegraded":"2023-10-14T17:01:15.691163+0000","last_fullsized":"2023-10-14T17:01:15.691163+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:49.455799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.33","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683209+0000","last_change":"2023-10-14T17:01:14.702926+0000","last_active":"2023-10-14T17:01:15.683209+0000","last_peered":"2023-10-14T17:01:15.683209+0000","last_clean":"2023-10-14T17:01:15.683209+0000","last_became_active":"2023-10-14T17:01:14.702627+0000","last_became_peered":"2023-10-14T17:01:14.702627+0000","last_unstale":"2023-10-14T17:01:15.683209+0000","last_undegraded":"2023-10-14T17:01:15.683209+0000","last_fullsized":"2023-10-14T17:01:15.683209+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:49.455799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.32","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780695+0000","last_change":"2023-10-14T17:01:14.716570+0000","last_active":"2023-10-14T17:01:16.780695+0000","last_peered":"2023-10-14T17:01:16.780695+0000","last_clean":"2023-10-14T17:01:16.780695+0000","last_became_active":"2023-10-14T17:01:14.716297+0000","last_became_peered":"2023-10-14T17:01:14.716297+0000","last_unstale":"2023-10-14T17:01:16.780695+0000","last_undegraded":"2023-10-14T17:01:16.780695+0000","last_fullsized":"2023-10-14T17:01:16.780695+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:08.736748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.31","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682502+0000","last_change":"2023-10-14T17:01:14.707371+0000","last_active":"2023-10-14T17:01:15.682502+0000","last_peered":"2023-10-14T17:01:15.682502+0000","last_clean":"2023-10-14T17:01:15.682502+0000","last_became_active":"2023-10-14T17:01:14.705122+0000","last_became_peered":"2023-10-14T17:01:14.705122+0000","last_unstale":"2023-10-14T17:01:15.682502+0000","last_undegraded":"2023-10-14T17:01:15.682502+0000","last_fullsized":"2023-10-14T17:01:15.682502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:58.474822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.30","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780497+0000","last_change":"2023-10-14T17:01:14.710074+0000","last_active":"2023-10-14T17:01:16.780497+0000","last_peered":"2023-10-14T17:01:16.780497+0000","last_clean":"2023-10-14T17:01:16.780497+0000","last_became_active":"2023-10-14T17:01:14.709716+0000","last_became_peered":"2023-10-14T17:01:14.709716+0000","last_unstale":"2023-10-14T17:01:16.780497+0000","last_undegraded":"2023-10-14T17:01:16.780497+0000","last_fullsized":"2023-10-14T17:01:16.780497+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:46.408470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688942+0000","last_change":"2023-10-14T17:01:14.699675+0000","last_active":"2023-10-14T17:01:15.688942+0000","last_peered":"2023-10-14T17:01:15.688942+0000","last_clean":"2023-10-14T17:01:15.688942+0000","last_became_active":"2023-10-14T17:01:14.699408+0000","last_became_peered":"2023-10-14T17:01:14.699408+0000","last_unstale":"2023-10-14T17:01:15.688942+0000","last_undegraded":"2023-10-14T17:01:15.688942+0000","last_fullsized":"2023-10-14T17:01:15.688942+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:13:52.401357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.2e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782259+0000","last_change":"2023-10-14T17:01:14.702416+0000","last_active":"2023-10-14T17:01:16.782259+0000","last_peered":"2023-10-14T17:01:16.782259+0000","last_clean":"2023-10-14T17:01:16.782259+0000","last_became_active":"2023-10-14T17:01:14.695421+0000","last_became_peered":"2023-10-14T17:01:14.695421+0000","last_unstale":"2023-10-14T17:01:16.782259+0000","last_undegraded":"2023-10-14T17:01:16.782259+0000","last_fullsized":"2023-10-14T17:01:16.782259+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:47.122460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781204+0000","last_change":"2023-10-14T17:01:14.699636+0000","last_active":"2023-10-14T17:01:16.781204+0000","last_peered":"2023-10-14T17:01:16.781204+0000","last_clean":"2023-10-14T17:01:16.781204+0000","last_became_active":"2023-10-14T17:01:14.699313+0000","last_became_peered":"2023-10-14T17:01:14.699313+0000","last_unstale":"2023-10-14T17:01:16.781204+0000","last_undegraded":"2023-10-14T17:01:16.781204+0000","last_fullsized":"2023-10-14T17:01:16.781204+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:39:25.218441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782580+0000","last_change":"2023-10-14T17:01:14.697845+0000","last_active":"2023-10-14T17:01:16.782580+0000","last_peered":"2023-10-14T17:01:16.782580+0000","last_clean":"2023-10-14T17:01:16.782580+0000","last_became_active":"2023-10-14T17:01:14.691997+0000","last_became_peered":"2023-10-14T17:01:14.691997+0000","last_unstale":"2023-10-14T17:01:16.782580+0000","last_undegraded":"2023-10-14T17:01:16.782580+0000","last_fullsized":"2023-10-14T17:01:16.782580+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:42.434427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.2b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103671+0000","last_change":"2023-10-14T17:01:14.692032+0000","last_active":"2023-10-14T17:01:16.103671+0000","last_peered":"2023-10-14T17:01:16.103671+0000","last_clean":"2023-10-14T17:01:16.103671+0000","last_became_active":"2023-10-14T17:01:14.691734+0000","last_became_peered":"2023-10-14T17:01:14.691734+0000","last_unstale":"2023-10-14T17:01:16.103671+0000","last_undegraded":"2023-10-14T17:01:16.103671+0000","last_fullsized":"2023-10-14T17:01:16.103671+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:19:32.149779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.2a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781557+0000","last_change":"2023-10-14T17:01:14.712560+0000","last_active":"2023-10-14T17:01:16.781557+0000","last_peered":"2023-10-14T17:01:16.781557+0000","last_clean":"2023-10-14T17:01:16.781557+0000","last_became_active":"2023-10-14T17:01:14.712232+0000","last_became_peered":"2023-10-14T17:01:14.712232+0000","last_unstale":"2023-10-14T17:01:16.781557+0000","last_undegraded":"2023-10-14T17:01:16.781557+0000","last_fullsized":"2023-10-14T17:01:16.781557+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:31.057718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.29","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690860+0000","last_change":"2023-10-14T17:01:14.699364+0000","last_active":"2023-10-14T17:01:15.690860+0000","last_peered":"2023-10-14T17:01:15.690860+0000","last_clean":"2023-10-14T17:01:15.690860+0000","last_became_active":"2023-10-14T17:01:14.698632+0000","last_became_peered":"2023-10-14T17:01:14.698632+0000","last_unstale":"2023-10-14T17:01:15.690860+0000","last_undegraded":"2023-10-14T17:01:15.690860+0000","last_fullsized":"2023-10-14T17:01:15.690860+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:15:18.416581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.28","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780156+0000","last_change":"2023-10-14T17:01:14.702111+0000","last_active":"2023-10-14T17:01:16.780156+0000","last_peered":"2023-10-14T17:01:16.780156+0000","last_clean":"2023-10-14T17:01:16.780156+0000","last_became_active":"2023-10-14T17:01:14.699054+0000","last_became_peered":"2023-10-14T17:01:14.699054+0000","last_unstale":"2023-10-14T17:01:16.780156+0000","last_undegraded":"2023-10-14T17:01:16.780156+0000","last_fullsized":"2023-10-14T17:01:16.780156+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:52.589774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.27","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781782+0000","last_change":"2023-10-14T17:01:14.705363+0000","last_active":"2023-10-14T17:01:16.781782+0000","last_peered":"2023-10-14T17:01:16.781782+0000","last_clean":"2023-10-14T17:01:16.781782+0000","last_became_active":"2023-10-14T17:01:14.705099+0000","last_became_peered":"2023-10-14T17:01:14.705099+0000","last_unstale":"2023-10-14T17:01:16.781782+0000","last_undegraded":"2023-10-14T17:01:16.781782+0000","last_fullsized":"2023-10-14T17:01:16.781782+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:41:03.672709+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.26","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782361+0000","last_change":"2023-10-14T17:01:14.703194+0000","last_active":"2023-10-14T17:01:16.782361+0000","last_peered":"2023-10-14T17:01:16.782361+0000","last_clean":"2023-10-14T17:01:16.782361+0000","last_became_active":"2023-10-14T17:01:14.702678+0000","last_became_peered":"2023-10-14T17:01:14.702678+0000","last_unstale":"2023-10-14T17:01:16.782361+0000","last_undegraded":"2023-10-14T17:01:16.782361+0000","last_fullsized":"2023-10-14T17:01:16.782361+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:12.686427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.25","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682857+0000","last_change":"2023-10-14T17:01:14.709448+0000","last_active":"2023-10-14T17:01:15.682857+0000","last_peered":"2023-10-14T17:01:15.682857+0000","last_clean":"2023-10-14T17:01:15.682857+0000","last_became_active":"2023-10-14T17:01:14.709194+0000","last_became_peered":"2023-10-14T17:01:14.709194+0000","last_unstale":"2023-10-14T17:01:15.682857+0000","last_undegraded":"2023-10-14T17:01:15.682857+0000","last_fullsized":"2023-10-14T17:01:15.682857+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:57.591951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.24","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782055+0000","last_change":"2023-10-14T17:01:14.709973+0000","last_active":"2023-10-14T17:01:16.782055+0000","last_peered":"2023-10-14T17:01:16.782055+0000","last_clean":"2023-10-14T17:01:16.782055+0000","last_became_active":"2023-10-14T17:01:14.709767+0000","last_became_peered":"2023-10-14T17:01:14.709767+0000","last_unstale":"2023-10-14T17:01:16.782055+0000","last_undegraded":"2023-10-14T17:01:16.782055+0000","last_fullsized":"2023-10-14T17:01:16.782055+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:08:08.038231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.23","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683055+0000","last_change":"2023-10-14T17:01:14.706746+0000","last_active":"2023-10-14T17:01:15.683055+0000","last_peered":"2023-10-14T17:01:15.683055+0000","last_clean":"2023-10-14T17:01:15.683055+0000","last_became_active":"2023-10-14T17:01:14.706098+0000","last_became_peered":"2023-10-14T17:01:14.706098+0000","last_unstale":"2023-10-14T17:01:15.683055+0000","last_undegraded":"2023-10-14T17:01:15.683055+0000","last_fullsized":"2023-10-14T17:01:15.683055+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:52.102810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.22","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682645+0000","last_change":"2023-10-14T17:01:14.711463+0000","last_active":"2023-10-14T17:01:15.682645+0000","last_peered":"2023-10-14T17:01:15.682645+0000","last_clean":"2023-10-14T17:01:15.682645+0000","last_became_active":"2023-10-14T17:01:14.707670+0000","last_became_peered":"2023-10-14T17:01:14.707670+0000","last_unstale":"2023-10-14T17:01:15.682645+0000","last_undegraded":"2023-10-14T17:01:15.682645+0000","last_fullsized":"2023-10-14T17:01:15.682645+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:02.267425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.21","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103462+0000","last_change":"2023-10-14T17:01:14.700351+0000","last_active":"2023-10-14T17:01:16.103462+0000","last_peered":"2023-10-14T17:01:16.103462+0000","last_clean":"2023-10-14T17:01:16.103462+0000","last_became_active":"2023-10-14T17:01:14.699066+0000","last_became_peered":"2023-10-14T17:01:14.699066+0000","last_unstale":"2023-10-14T17:01:16.103462+0000","last_undegraded":"2023-10-14T17:01:16.103462+0000","last_fullsized":"2023-10-14T17:01:16.103462+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:35.014489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.20","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688017+0000","last_change":"2023-10-14T17:01:14.709522+0000","last_active":"2023-10-14T17:01:15.688017+0000","last_peered":"2023-10-14T17:01:15.688017+0000","last_clean":"2023-10-14T17:01:15.688017+0000","last_became_active":"2023-10-14T17:01:14.709153+0000","last_became_peered":"2023-10-14T17:01:14.709153+0000","last_unstale":"2023-10-14T17:01:15.688017+0000","last_undegraded":"2023-10-14T17:01:15.688017+0000","last_fullsized":"2023-10-14T17:01:15.688017+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:47:07.835175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.1e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782450+0000","last_change":"2023-10-14T17:01:14.702985+0000","last_active":"2023-10-14T17:01:16.782450+0000","last_peered":"2023-10-14T17:01:16.782450+0000","last_clean":"2023-10-14T17:01:16.782450+0000","last_became_active":"2023-10-14T17:01:14.702041+0000","last_became_peered":"2023-10-14T17:01:14.702041+0000","last_unstale":"2023-10-14T17:01:16.782450+0000","last_undegraded":"2023-10-14T17:01:16.782450+0000","last_fullsized":"2023-10-14T17:01:16.782450+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:35.014489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1f","version":"153'4","reported_seq":16,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104532+0000","last_change":"2023-10-14T17:01:13.678943+0000","last_active":"2023-10-14T17:01:16.104532+0000","last_peered":"2023-10-14T17:01:16.104532+0000","last_clean":"2023-10-14T17:01:16.104532+0000","last_became_active":"2023-10-14T17:01:13.678667+0000","last_became_peered":"2023-10-14T17:01:13.678667+0000","last_unstale":"2023-10-14T17:01:16.104532+0000","last_undegraded":"2023-10-14T17:01:16.104532+0000","last_fullsized":"2023-10-14T17:01:16.104532+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:36.098987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.1f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691672+0000","last_change":"2023-10-14T17:01:14.702130+0000","last_active":"2023-10-14T17:01:15.691672+0000","last_peered":"2023-10-14T17:01:15.691672+0000","last_clean":"2023-10-14T17:01:15.691672+0000","last_became_active":"2023-10-14T17:01:14.701851+0000","last_became_peered":"2023-10-14T17:01:14.701851+0000","last_unstale":"2023-10-14T17:01:15.691672+0000","last_undegraded":"2023-10-14T17:01:15.691672+0000","last_fullsized":"2023-10-14T17:01:15.691672+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:51:28.974195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.1e","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782350+0000","last_change":"2023-10-14T17:01:13.676855+0000","last_active":"2023-10-14T17:01:16.782350+0000","last_peered":"2023-10-14T17:01:16.782350+0000","last_clean":"2023-10-14T17:01:16.782350+0000","last_became_active":"2023-10-14T17:01:13.675799+0000","last_became_peered":"2023-10-14T17:01:13.675799+0000","last_unstale":"2023-10-14T17:01:16.782350+0000","last_undegraded":"2023-10-14T17:01:16.782350+0000","last_fullsized":"2023-10-14T17:01:16.782350+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:07.218287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.1c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104340+0000","last_change":"2023-10-14T17:01:14.696159+0000","last_active":"2023-10-14T17:01:16.104340+0000","last_peered":"2023-10-14T17:01:16.104340+0000","last_clean":"2023-10-14T17:01:16.104340+0000","last_became_active":"2023-10-14T17:01:14.694787+0000","last_became_peered":"2023-10-14T17:01:14.694787+0000","last_unstale":"2023-10-14T17:01:16.104340+0000","last_undegraded":"2023-10-14T17:01:16.104340+0000","last_fullsized":"2023-10-14T17:01:16.104340+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:21.050171+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1d","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781325+0000","last_change":"2023-10-14T17:01:13.673283+0000","last_active":"2023-10-14T17:01:16.781325+0000","last_peered":"2023-10-14T17:01:16.781325+0000","last_clean":"2023-10-14T17:01:16.781325+0000","last_became_active":"2023-10-14T17:01:13.672243+0000","last_became_peered":"2023-10-14T17:01:13.672243+0000","last_unstale":"2023-10-14T17:01:16.781325+0000","last_undegraded":"2023-10-14T17:01:16.781325+0000","last_fullsized":"2023-10-14T17:01:16.781325+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:00:07.218287+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.1d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104337+0000","last_change":"2023-10-14T17:01:14.705210+0000","last_active":"2023-10-14T17:01:16.104337+0000","last_peered":"2023-10-14T17:01:16.104337+0000","last_clean":"2023-10-14T17:01:16.104337+0000","last_became_active":"2023-10-14T17:01:14.704897+0000","last_became_peered":"2023-10-14T17:01:14.704897+0000","last_unstale":"2023-10-14T17:01:16.104337+0000","last_undegraded":"2023-10-14T17:01:16.104337+0000","last_fullsized":"2023-10-14T17:01:16.104337+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:29.099700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.1c","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781923+0000","last_change":"2023-10-14T17:01:13.672487+0000","last_active":"2023-10-14T17:01:16.781923+0000","last_peered":"2023-10-14T17:01:16.781923+0000","last_clean":"2023-10-14T17:01:16.781923+0000","last_became_active":"2023-10-14T17:01:13.672192+0000","last_became_peered":"2023-10-14T17:01:13.672192+0000","last_unstale":"2023-10-14T17:01:16.781923+0000","last_undegraded":"2023-10-14T17:01:16.781923+0000","last_fullsized":"2023-10-14T17:01:16.781923+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:45:08.909223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781260+0000","last_change":"2023-10-14T17:01:14.700281+0000","last_active":"2023-10-14T17:01:16.781260+0000","last_peered":"2023-10-14T17:01:16.781260+0000","last_clean":"2023-10-14T17:01:16.781260+0000","last_became_active":"2023-10-14T17:01:14.695051+0000","last_became_peered":"2023-10-14T17:01:14.695051+0000","last_unstale":"2023-10-14T17:01:16.781260+0000","last_undegraded":"2023-10-14T17:01:16.781260+0000","last_fullsized":"2023-10-14T17:01:16.781260+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:44:25.638199+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1b","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780526+0000","last_change":"2023-10-14T17:01:13.670963+0000","last_active":"2023-10-14T17:01:16.780526+0000","last_peered":"2023-10-14T17:01:16.780526+0000","last_clean":"2023-10-14T17:01:16.780526+0000","last_became_active":"2023-10-14T17:01:13.669597+0000","last_became_peered":"2023-10-14T17:01:13.669597+0000","last_unstale":"2023-10-14T17:01:16.780526+0000","last_undegraded":"2023-10-14T17:01:16.780526+0000","last_fullsized":"2023-10-14T17:01:16.780526+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:34:48.441183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.1b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688096+0000","last_change":"2023-10-14T17:01:14.700042+0000","last_active":"2023-10-14T17:01:15.688096+0000","last_peered":"2023-10-14T17:01:15.688096+0000","last_clean":"2023-10-14T17:01:15.688096+0000","last_became_active":"2023-10-14T17:01:14.699771+0000","last_became_peered":"2023-10-14T17:01:14.699771+0000","last_unstale":"2023-10-14T17:01:15.688096+0000","last_undegraded":"2023-10-14T17:01:15.688096+0000","last_fullsized":"2023-10-14T17:01:15.688096+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:46:11.670317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.1a","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781383+0000","last_change":"2023-10-14T17:01:13.671646+0000","last_active":"2023-10-14T17:01:16.781383+0000","last_peered":"2023-10-14T17:01:16.781383+0000","last_clean":"2023-10-14T17:01:16.781383+0000","last_became_active":"2023-10-14T17:01:13.671357+0000","last_became_peered":"2023-10-14T17:01:13.671357+0000","last_unstale":"2023-10-14T17:01:16.781383+0000","last_undegraded":"2023-10-14T17:01:16.781383+0000","last_fullsized":"2023-10-14T17:01:16.781383+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:00.652084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.18","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690515+0000","last_change":"2023-10-14T17:01:14.685455+0000","last_active":"2023-10-14T17:01:15.690515+0000","last_peered":"2023-10-14T17:01:15.690515+0000","last_clean":"2023-10-14T17:01:15.690515+0000","last_became_active":"2023-10-14T17:01:14.684945+0000","last_became_peered":"2023-10-14T17:01:14.684945+0000","last_unstale":"2023-10-14T17:01:15.690515+0000","last_undegraded":"2023-10-14T17:01:15.690515+0000","last_fullsized":"2023-10-14T17:01:15.690515+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:05:43.150631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.19","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.687414+0000","last_change":"2023-10-14T17:01:13.676361+0000","last_active":"2023-10-14T17:01:15.687414+0000","last_peered":"2023-10-14T17:01:15.687414+0000","last_clean":"2023-10-14T17:01:15.687414+0000","last_became_active":"2023-10-14T17:01:13.675084+0000","last_became_peered":"2023-10-14T17:01:13.675084+0000","last_unstale":"2023-10-14T17:01:15.687414+0000","last_undegraded":"2023-10-14T17:01:15.687414+0000","last_fullsized":"2023-10-14T17:01:15.687414+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:30.043102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.19","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781288+0000","last_change":"2023-10-14T17:01:14.704549+0000","last_active":"2023-10-14T17:01:16.781288+0000","last_peered":"2023-10-14T17:01:16.781288+0000","last_clean":"2023-10-14T17:01:16.781288+0000","last_became_active":"2023-10-14T17:01:14.704239+0000","last_became_peered":"2023-10-14T17:01:14.704239+0000","last_unstale":"2023-10-14T17:01:16.781288+0000","last_undegraded":"2023-10-14T17:01:16.781288+0000","last_fullsized":"2023-10-14T17:01:16.781288+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:13:00.705305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.18","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780166+0000","last_change":"2023-10-14T17:01:13.679963+0000","last_active":"2023-10-14T17:01:16.780166+0000","last_peered":"2023-10-14T17:01:16.780166+0000","last_clean":"2023-10-14T17:01:16.780166+0000","last_became_active":"2023-10-14T17:01:13.677746+0000","last_became_peered":"2023-10-14T17:01:13.677746+0000","last_unstale":"2023-10-14T17:01:16.780166+0000","last_undegraded":"2023-10-14T17:01:16.780166+0000","last_fullsized":"2023-10-14T17:01:16.780166+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:03.404311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.16","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682948+0000","last_change":"2023-10-14T17:01:14.706038+0000","last_active":"2023-10-14T17:01:15.682948+0000","last_peered":"2023-10-14T17:01:15.682948+0000","last_clean":"2023-10-14T17:01:15.682948+0000","last_became_active":"2023-10-14T17:01:14.705682+0000","last_became_peered":"2023-10-14T17:01:14.705682+0000","last_unstale":"2023-10-14T17:01:15.682948+0000","last_undegraded":"2023-10-14T17:01:15.682948+0000","last_fullsized":"2023-10-14T17:01:15.682948+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:05:43.150631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.17","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782597+0000","last_change":"2023-10-14T17:01:13.674826+0000","last_active":"2023-10-14T17:01:16.782597+0000","last_peered":"2023-10-14T17:01:16.782597+0000","last_clean":"2023-10-14T17:01:16.782597+0000","last_became_active":"2023-10-14T17:01:13.669834+0000","last_became_peered":"2023-10-14T17:01:13.669834+0000","last_unstale":"2023-10-14T17:01:16.782597+0000","last_undegraded":"2023-10-14T17:01:16.782597+0000","last_fullsized":"2023-10-14T17:01:16.782597+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:22.778177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.17","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782473+0000","last_change":"2023-10-14T17:01:14.702348+0000","last_active":"2023-10-14T17:01:16.782473+0000","last_peered":"2023-10-14T17:01:16.782473+0000","last_clean":"2023-10-14T17:01:16.782473+0000","last_became_active":"2023-10-14T17:01:14.702071+0000","last_became_peered":"2023-10-14T17:01:14.702071+0000","last_unstale":"2023-10-14T17:01:16.782473+0000","last_undegraded":"2023-10-14T17:01:16.782473+0000","last_fullsized":"2023-10-14T17:01:16.782473+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:57:31.057718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.16","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782483+0000","last_change":"2023-10-14T17:01:13.680495+0000","last_active":"2023-10-14T17:01:16.782483+0000","last_peered":"2023-10-14T17:01:16.782483+0000","last_clean":"2023-10-14T17:01:16.782483+0000","last_became_active":"2023-10-14T17:01:13.680166+0000","last_became_peered":"2023-10-14T17:01:13.680166+0000","last_unstale":"2023-10-14T17:01:16.782483+0000","last_undegraded":"2023-10-14T17:01:16.782483+0000","last_fullsized":"2023-10-14T17:01:16.782483+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:10.245115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.14","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104502+0000","last_change":"2023-10-14T17:01:14.702121+0000","last_active":"2023-10-14T17:01:16.104502+0000","last_peered":"2023-10-14T17:01:16.104502+0000","last_clean":"2023-10-14T17:01:16.104502+0000","last_became_active":"2023-10-14T17:01:14.701467+0000","last_became_peered":"2023-10-14T17:01:14.701467+0000","last_unstale":"2023-10-14T17:01:16.104502+0000","last_undegraded":"2023-10-14T17:01:16.104502+0000","last_fullsized":"2023-10-14T17:01:16.104502+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:17:29.054000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.15","version":"153'3","reported_seq":15,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781940+0000","last_change":"2023-10-14T17:01:13.674735+0000","last_active":"2023-10-14T17:01:16.781940+0000","last_peered":"2023-10-14T17:01:16.781940+0000","last_clean":"2023-10-14T17:01:16.781940+0000","last_became_active":"2023-10-14T17:01:13.672714+0000","last_became_peered":"2023-10-14T17:01:13.672714+0000","last_unstale":"2023-10-14T17:01:16.781940+0000","last_undegraded":"2023-10-14T17:01:16.781940+0000","last_fullsized":"2023-10-14T17:01:16.781940+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:11.468417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.15","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104368+0000","last_change":"2023-10-14T17:01:14.694003+0000","last_active":"2023-10-14T17:01:16.104368+0000","last_peered":"2023-10-14T17:01:16.104368+0000","last_clean":"2023-10-14T17:01:16.104368+0000","last_became_active":"2023-10-14T17:01:14.692409+0000","last_became_peered":"2023-10-14T17:01:14.692409+0000","last_unstale":"2023-10-14T17:01:16.104368+0000","last_undegraded":"2023-10-14T17:01:16.104368+0000","last_fullsized":"2023-10-14T17:01:16.104368+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:51.762425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.14","version":"153'6","reported_seq":18,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691070+0000","last_change":"2023-10-14T17:01:13.681462+0000","last_active":"2023-10-14T17:01:15.691070+0000","last_peered":"2023-10-14T17:01:15.691070+0000","last_clean":"2023-10-14T17:01:15.691070+0000","last_became_active":"2023-10-14T17:01:13.681189+0000","last_became_peered":"2023-10-14T17:01:13.681189+0000","last_unstale":"2023-10-14T17:01:15.691070+0000","last_undegraded":"2023-10-14T17:01:15.691070+0000","last_fullsized":"2023-10-14T17:01:15.691070+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:01.871554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.13","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683212+0000","last_change":"2023-10-14T17:01:14.711498+0000","last_active":"2023-10-14T17:01:15.683212+0000","last_peered":"2023-10-14T17:01:15.683212+0000","last_clean":"2023-10-14T17:01:15.683212+0000","last_became_active":"2023-10-14T17:01:14.711165+0000","last_became_peered":"2023-10-14T17:01:14.711165+0000","last_unstale":"2023-10-14T17:01:15.683212+0000","last_undegraded":"2023-10-14T17:01:15.683212+0000","last_fullsized":"2023-10-14T17:01:15.683212+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:12.485256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.12","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682709+0000","last_change":"2023-10-14T17:01:13.670668+0000","last_active":"2023-10-14T17:01:15.682709+0000","last_peered":"2023-10-14T17:01:15.682709+0000","last_clean":"2023-10-14T17:01:15.682709+0000","last_became_active":"2023-10-14T17:01:13.669219+0000","last_became_peered":"2023-10-14T17:01:13.669219+0000","last_unstale":"2023-10-14T17:01:15.682709+0000","last_undegraded":"2023-10-14T17:01:15.682709+0000","last_fullsized":"2023-10-14T17:01:15.682709+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:48:01.871554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.10","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780042+0000","last_change":"2023-10-14T17:01:14.691782+0000","last_active":"2023-10-14T17:01:16.780042+0000","last_peered":"2023-10-14T17:01:16.780042+0000","last_clean":"2023-10-14T17:01:16.780042+0000","last_became_active":"2023-10-14T17:01:14.691111+0000","last_became_peered":"2023-10-14T17:01:14.691111+0000","last_unstale":"2023-10-14T17:01:16.780042+0000","last_undegraded":"2023-10-14T17:01:16.780042+0000","last_fullsized":"2023-10-14T17:01:16.780042+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:27:18.745622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.11","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781613+0000","last_change":"2023-10-14T17:01:13.679508+0000","last_active":"2023-10-14T17:01:16.781613+0000","last_peered":"2023-10-14T17:01:16.781613+0000","last_clean":"2023-10-14T17:01:16.781613+0000","last_became_active":"2023-10-14T17:01:13.679026+0000","last_became_peered":"2023-10-14T17:01:13.679026+0000","last_unstale":"2023-10-14T17:01:16.781613+0000","last_undegraded":"2023-10-14T17:01:16.781613+0000","last_fullsized":"2023-10-14T17:01:16.781613+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:49.397746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.11","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690733+0000","last_change":"2023-10-14T17:01:14.699908+0000","last_active":"2023-10-14T17:01:15.690733+0000","last_peered":"2023-10-14T17:01:15.690733+0000","last_clean":"2023-10-14T17:01:15.690733+0000","last_became_active":"2023-10-14T17:01:14.699636+0000","last_became_peered":"2023-10-14T17:01:14.699636+0000","last_unstale":"2023-10-14T17:01:15.690733+0000","last_undegraded":"2023-10-14T17:01:15.690733+0000","last_fullsized":"2023-10-14T17:01:15.690733+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:52.102810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.10","version":"153'8","reported_seq":20,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.690791+0000","last_change":"2023-10-14T17:01:13.677498+0000","last_active":"2023-10-14T17:01:15.690791+0000","last_peered":"2023-10-14T17:01:15.690791+0000","last_clean":"2023-10-14T17:01:15.690791+0000","last_became_active":"2023-10-14T17:01:13.677227+0000","last_became_peered":"2023-10-14T17:01:13.677227+0000","last_unstale":"2023-10-14T17:01:15.690791+0000","last_undegraded":"2023-10-14T17:01:15.690791+0000","last_fullsized":"2023-10-14T17:01:15.690791+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:01.252808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"19.e","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.691566+0000","last_change":"2023-10-14T17:01:14.703429+0000","last_active":"2023-10-14T17:01:15.691566+0000","last_peered":"2023-10-14T17:01:15.691566+0000","last_clean":"2023-10-14T17:01:15.691566+0000","last_became_active":"2023-10-14T17:01:14.703020+0000","last_became_peered":"2023-10-14T17:01:14.703020+0000","last_unstale":"2023-10-14T17:01:15.691566+0000","last_undegraded":"2023-10-14T17:01:15.691566+0000","last_fullsized":"2023-10-14T17:01:15.691566+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:12.485256+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"18.f","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782659+0000","last_change":"2023-10-14T17:01:13.679977+0000","last_active":"2023-10-14T17:01:16.782659+0000","last_peered":"2023-10-14T17:01:16.782659+0000","last_clean":"2023-10-14T17:01:16.782659+0000","last_became_active":"2023-10-14T17:01:13.679694+0000","last_became_peered":"2023-10-14T17:01:13.679694+0000","last_unstale":"2023-10-14T17:01:16.782659+0000","last_undegraded":"2023-10-14T17:01:16.782659+0000","last_fullsized":"2023-10-14T17:01:16.782659+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:04.039372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.f","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782530+0000","last_change":"2023-10-14T17:01:14.707457+0000","last_active":"2023-10-14T17:01:16.782530+0000","last_peered":"2023-10-14T17:01:16.782530+0000","last_clean":"2023-10-14T17:01:16.782530+0000","last_became_active":"2023-10-14T17:01:14.707094+0000","last_became_peered":"2023-10-14T17:01:14.707094+0000","last_unstale":"2023-10-14T17:01:16.782530+0000","last_undegraded":"2023-10-14T17:01:16.782530+0000","last_fullsized":"2023-10-14T17:01:16.782530+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:12.819926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.e","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780728+0000","last_change":"2023-10-14T17:01:13.678974+0000","last_active":"2023-10-14T17:01:16.780728+0000","last_peered":"2023-10-14T17:01:16.780728+0000","last_clean":"2023-10-14T17:01:16.780728+0000","last_became_active":"2023-10-14T17:01:13.678701+0000","last_became_peered":"2023-10-14T17:01:13.678701+0000","last_unstale":"2023-10-14T17:01:16.780728+0000","last_undegraded":"2023-10-14T17:01:16.780728+0000","last_fullsized":"2023-10-14T17:01:16.780728+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:29:06.055561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.c","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782660+0000","last_change":"2023-10-14T17:01:14.699556+0000","last_active":"2023-10-14T17:01:16.782660+0000","last_peered":"2023-10-14T17:01:16.782660+0000","last_clean":"2023-10-14T17:01:16.782660+0000","last_became_active":"2023-10-14T17:01:14.699268+0000","last_became_peered":"2023-10-14T17:01:14.699268+0000","last_unstale":"2023-10-14T17:01:16.782660+0000","last_undegraded":"2023-10-14T17:01:16.782660+0000","last_fullsized":"2023-10-14T17:01:16.782660+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:42.187894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.d","version":"153'1","reported_seq":13,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688581+0000","last_change":"2023-10-14T17:01:13.667702+0000","last_active":"2023-10-14T17:01:15.688581+0000","last_peered":"2023-10-14T17:01:15.688581+0000","last_clean":"2023-10-14T17:01:15.688581+0000","last_became_active":"2023-10-14T17:01:13.666669+0000","last_became_peered":"2023-10-14T17:01:13.666669+0000","last_unstale":"2023-10-14T17:01:15.688581+0000","last_undegraded":"2023-10-14T17:01:15.688581+0000","last_fullsized":"2023-10-14T17:01:15.688581+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:11.805309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"19.d","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782096+0000","last_change":"2023-10-14T17:01:14.711230+0000","last_active":"2023-10-14T17:01:16.782096+0000","last_peered":"2023-10-14T17:01:16.782096+0000","last_clean":"2023-10-14T17:01:16.782096+0000","last_became_active":"2023-10-14T17:01:14.710905+0000","last_became_peered":"2023-10-14T17:01:14.710905+0000","last_unstale":"2023-10-14T17:01:16.782096+0000","last_undegraded":"2023-10-14T17:01:16.782096+0000","last_fullsized":"2023-10-14T17:01:16.782096+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:25.273827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.c","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782125+0000","last_change":"2023-10-14T17:01:13.673875+0000","last_active":"2023-10-14T17:01:16.782125+0000","last_peered":"2023-10-14T17:01:16.782125+0000","last_clean":"2023-10-14T17:01:16.782125+0000","last_became_active":"2023-10-14T17:01:13.673565+0000","last_became_peered":"2023-10-14T17:01:13.673565+0000","last_unstale":"2023-10-14T17:01:16.782125+0000","last_undegraded":"2023-10-14T17:01:16.782125+0000","last_fullsized":"2023-10-14T17:01:16.782125+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:30:07.618113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.a","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.682606+0000","last_change":"2023-10-14T17:01:14.707548+0000","last_active":"2023-10-14T17:01:15.682606+0000","last_peered":"2023-10-14T17:01:15.682606+0000","last_clean":"2023-10-14T17:01:15.682606+0000","last_became_active":"2023-10-14T17:01:14.703382+0000","last_became_peered":"2023-10-14T17:01:14.703382+0000","last_unstale":"2023-10-14T17:01:15.682606+0000","last_undegraded":"2023-10-14T17:01:15.682606+0000","last_fullsized":"2023-10-14T17:01:15.682606+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:16.813821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.b","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781991+0000","last_change":"2023-10-14T17:01:13.674470+0000","last_active":"2023-10-14T17:01:16.781991+0000","last_peered":"2023-10-14T17:01:16.781991+0000","last_clean":"2023-10-14T17:01:16.781991+0000","last_became_active":"2023-10-14T17:01:13.668958+0000","last_became_peered":"2023-10-14T17:01:13.668958+0000","last_unstale":"2023-10-14T17:01:16.781991+0000","last_undegraded":"2023-10-14T17:01:16.781991+0000","last_fullsized":"2023-10-14T17:01:16.781991+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:25:51.575158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"19.b","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688398+0000","last_change":"2023-10-14T17:01:14.693436+0000","last_active":"2023-10-14T17:01:15.688398+0000","last_peered":"2023-10-14T17:01:15.688398+0000","last_clean":"2023-10-14T17:01:15.688398+0000","last_became_active":"2023-10-14T17:01:14.693115+0000","last_became_peered":"2023-10-14T17:01:14.693115+0000","last_unstale":"2023-10-14T17:01:15.688398+0000","last_undegraded":"2023-10-14T17:01:15.688398+0000","last_fullsized":"2023-10-14T17:01:15.688398+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:02.454338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.a","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781498+0000","last_change":"2023-10-14T17:01:13.681995+0000","last_active":"2023-10-14T17:01:16.781498+0000","last_peered":"2023-10-14T17:01:16.781498+0000","last_clean":"2023-10-14T17:01:16.781498+0000","last_became_active":"2023-10-14T17:01:13.681706+0000","last_became_peered":"2023-10-14T17:01:13.681706+0000","last_unstale":"2023-10-14T17:01:16.781498+0000","last_undegraded":"2023-10-14T17:01:16.781498+0000","last_fullsized":"2023-10-14T17:01:16.781498+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:03:40.425410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.8","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780092+0000","last_change":"2023-10-14T17:01:14.699116+0000","last_active":"2023-10-14T17:01:16.780092+0000","last_peered":"2023-10-14T17:01:16.780092+0000","last_clean":"2023-10-14T17:01:16.780092+0000","last_became_active":"2023-10-14T17:01:14.698573+0000","last_became_peered":"2023-10-14T17:01:14.698573+0000","last_unstale":"2023-10-14T17:01:16.780092+0000","last_undegraded":"2023-10-14T17:01:16.780092+0000","last_fullsized":"2023-10-14T17:01:16.780092+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:20.512490+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.9","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780771+0000","last_change":"2023-10-14T17:01:13.675277+0000","last_active":"2023-10-14T17:01:16.780771+0000","last_peered":"2023-10-14T17:01:16.780771+0000","last_clean":"2023-10-14T17:01:16.780771+0000","last_became_active":"2023-10-14T17:01:13.674996+0000","last_became_peered":"2023-10-14T17:01:13.674996+0000","last_unstale":"2023-10-14T17:01:16.780771+0000","last_undegraded":"2023-10-14T17:01:16.780771+0000","last_fullsized":"2023-10-14T17:01:16.780771+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:48.375524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.9","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103514+0000","last_change":"2023-10-14T17:01:14.701784+0000","last_active":"2023-10-14T17:01:16.103514+0000","last_peered":"2023-10-14T17:01:16.103514+0000","last_clean":"2023-10-14T17:01:16.103514+0000","last_became_active":"2023-10-14T17:01:14.699415+0000","last_became_peered":"2023-10-14T17:01:14.699415+0000","last_unstale":"2023-10-14T17:01:16.103514+0000","last_undegraded":"2023-10-14T17:01:16.103514+0000","last_fullsized":"2023-10-14T17:01:16.103514+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:59:23.345688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.8","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780559+0000","last_change":"2023-10-14T17:01:13.677783+0000","last_active":"2023-10-14T17:01:16.780559+0000","last_peered":"2023-10-14T17:01:16.780559+0000","last_clean":"2023-10-14T17:01:16.780559+0000","last_became_active":"2023-10-14T17:01:13.677532+0000","last_became_peered":"2023-10-14T17:01:13.677532+0000","last_unstale":"2023-10-14T17:01:16.780559+0000","last_undegraded":"2023-10-14T17:01:16.780559+0000","last_fullsized":"2023-10-14T17:01:16.780559+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:22.778177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.6","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688962+0000","last_change":"2023-10-14T17:01:14.696006+0000","last_active":"2023-10-14T17:01:15.688962+0000","last_peered":"2023-10-14T17:01:15.688962+0000","last_clean":"2023-10-14T17:01:15.688962+0000","last_became_active":"2023-10-14T17:01:14.695590+0000","last_became_peered":"2023-10-14T17:01:14.695590+0000","last_unstale":"2023-10-14T17:01:15.688962+0000","last_undegraded":"2023-10-14T17:01:15.688962+0000","last_fullsized":"2023-10-14T17:01:15.688962+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:19.463438+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.7","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683255+0000","last_change":"2023-10-14T17:01:13.676826+0000","last_active":"2023-10-14T17:01:15.683255+0000","last_peered":"2023-10-14T17:01:15.683255+0000","last_clean":"2023-10-14T17:01:15.683255+0000","last_became_active":"2023-10-14T17:01:13.676542+0000","last_became_peered":"2023-10-14T17:01:13.676542+0000","last_unstale":"2023-10-14T17:01:15.683255+0000","last_undegraded":"2023-10-14T17:01:15.683255+0000","last_fullsized":"2023-10-14T17:01:15.683255+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:48:52.400410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"19.7","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.683190+0000","last_change":"2023-10-14T17:01:14.710330+0000","last_active":"2023-10-14T17:01:15.683190+0000","last_peered":"2023-10-14T17:01:15.683190+0000","last_clean":"2023-10-14T17:01:15.683190+0000","last_became_active":"2023-10-14T17:01:14.707021+0000","last_became_peered":"2023-10-14T17:01:14.707021+0000","last_unstale":"2023-10-14T17:01:15.683190+0000","last_undegraded":"2023-10-14T17:01:15.683190+0000","last_fullsized":"2023-10-14T17:01:15.683190+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:50:50.412362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"18.6","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.104208+0000","last_change":"2023-10-14T17:01:13.679044+0000","last_active":"2023-10-14T17:01:16.104208+0000","last_peered":"2023-10-14T17:01:16.104208+0000","last_clean":"2023-10-14T17:01:16.104208+0000","last_became_active":"2023-10-14T17:01:13.678165+0000","last_became_peered":"2023-10-14T17:01:13.678165+0000","last_unstale":"2023-10-14T17:01:16.104208+0000","last_undegraded":"2023-10-14T17:01:16.104208+0000","last_fullsized":"2023-10-14T17:01:16.104208+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:06.024536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.4","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781782+0000","last_change":"2023-10-14T17:01:14.696611+0000","last_active":"2023-10-14T17:01:16.781782+0000","last_peered":"2023-10-14T17:01:16.781782+0000","last_clean":"2023-10-14T17:01:16.781782+0000","last_became_active":"2023-10-14T17:01:14.696361+0000","last_became_peered":"2023-10-14T17:01:14.696361+0000","last_unstale":"2023-10-14T17:01:16.781782+0000","last_undegraded":"2023-10-14T17:01:16.781782+0000","last_fullsized":"2023-10-14T17:01:16.781782+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:24:12.483034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.5","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782104+0000","last_change":"2023-10-14T17:01:13.679306+0000","last_active":"2023-10-14T17:01:16.782104+0000","last_peered":"2023-10-14T17:01:16.782104+0000","last_clean":"2023-10-14T17:01:16.782104+0000","last_became_active":"2023-10-14T17:01:13.677185+0000","last_became_peered":"2023-10-14T17:01:13.677185+0000","last_unstale":"2023-10-14T17:01:16.782104+0000","last_undegraded":"2023-10-14T17:01:16.782104+0000","last_fullsized":"2023-10-14T17:01:16.782104+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:03:03.127006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.5","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:15.688317+0000","last_change":"2023-10-14T17:01:14.703929+0000","last_active":"2023-10-14T17:01:15.688317+0000","last_peered":"2023-10-14T17:01:15.688317+0000","last_clean":"2023-10-14T17:01:15.688317+0000","last_became_active":"2023-10-14T17:01:14.703593+0000","last_became_peered":"2023-10-14T17:01:14.703593+0000","last_unstale":"2023-10-14T17:01:15.688317+0000","last_undegraded":"2023-10-14T17:01:15.688317+0000","last_fullsized":"2023-10-14T17:01:15.688317+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:37:23.174484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"18.4","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780956+0000","last_change":"2023-10-14T17:01:13.678873+0000","last_active":"2023-10-14T17:01:16.780956+0000","last_peered":"2023-10-14T17:01:16.780956+0000","last_clean":"2023-10-14T17:01:16.780956+0000","last_became_active":"2023-10-14T17:01:13.678510+0000","last_became_peered":"2023-10-14T17:01:13.678510+0000","last_unstale":"2023-10-14T17:01:16.780956+0000","last_undegraded":"2023-10-14T17:01:16.780956+0000","last_fullsized":"2023-10-14T17:01:16.780956+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:41:02.658093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.2","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103500+0000","last_change":"2023-10-14T17:01:14.695952+0000","last_active":"2023-10-14T17:01:16.103500+0000","last_peered":"2023-10-14T17:01:16.103500+0000","last_clean":"2023-10-14T17:01:16.103500+0000","last_became_active":"2023-10-14T17:01:14.694848+0000","last_became_peered":"2023-10-14T17:01:14.694848+0000","last_unstale":"2023-10-14T17:01:16.103500+0000","last_undegraded":"2023-10-14T17:01:16.103500+0000","last_fullsized":"2023-10-14T17:01:16.103500+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:47:42.187894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.3","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781783+0000","last_change":"2023-10-14T17:01:13.678809+0000","last_active":"2023-10-14T17:01:16.781783+0000","last_peered":"2023-10-14T17:01:16.781783+0000","last_clean":"2023-10-14T17:01:16.781783+0000","last_became_active":"2023-10-14T17:01:13.678239+0000","last_became_peered":"2023-10-14T17:01:13.678239+0000","last_unstale":"2023-10-14T17:01:16.781783+0000","last_undegraded":"2023-10-14T17:01:16.781783+0000","last_fullsized":"2023-10-14T17:01:16.781783+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:51:27.959578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"19.3","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.782058+0000","last_change":"2023-10-14T17:01:14.699332+0000","last_active":"2023-10-14T17:01:16.782058+0000","last_peered":"2023-10-14T17:01:16.782058+0000","last_clean":"2023-10-14T17:01:16.782058+0000","last_became_active":"2023-10-14T17:01:14.699029+0000","last_became_peered":"2023-10-14T17:01:14.699029+0000","last_unstale":"2023-10-14T17:01:16.782058+0000","last_undegraded":"2023-10-14T17:01:16.782058+0000","last_fullsized":"2023-10-14T17:01:16.782058+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:39:25.218441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"18.2","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103691+0000","last_change":"2023-10-14T17:01:13.681545+0000","last_active":"2023-10-14T17:01:16.103691+0000","last_peered":"2023-10-14T17:01:16.103691+0000","last_clean":"2023-10-14T17:01:16.103691+0000","last_became_active":"2023-10-14T17:01:13.681079+0000","last_became_peered":"2023-10-14T17:01:13.681079+0000","last_unstale":"2023-10-14T17:01:16.103691+0000","last_undegraded":"2023-10-14T17:01:16.103691+0000","last_fullsized":"2023-10-14T17:01:16.103691+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:46.107843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"19.0","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780679+0000","last_change":"2023-10-14T17:01:14.698436+0000","last_active":"2023-10-14T17:01:16.780679+0000","last_peered":"2023-10-14T17:01:16.780679+0000","last_clean":"2023-10-14T17:01:16.780679+0000","last_became_active":"2023-10-14T17:01:14.697004+0000","last_became_peered":"2023-10-14T17:01:14.697004+0000","last_unstale":"2023-10-14T17:01:16.780679+0000","last_undegraded":"2023-10-14T17:01:16.780679+0000","last_fullsized":"2023-10-14T17:01:16.780679+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:50:25.252771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"18.1","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780987+0000","last_change":"2023-10-14T17:01:13.682552+0000","last_active":"2023-10-14T17:01:16.780987+0000","last_peered":"2023-10-14T17:01:16.780987+0000","last_clean":"2023-10-14T17:01:16.780987+0000","last_became_active":"2023-10-14T17:01:13.681681+0000","last_became_peered":"2023-10-14T17:01:13.681681+0000","last_unstale":"2023-10-14T17:01:16.780987+0000","last_undegraded":"2023-10-14T17:01:16.780987+0000","last_fullsized":"2023-10-14T17:01:16.780987+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:30:08.135964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"19.1","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.781493+0000","last_change":"2023-10-14T17:01:14.693152+0000","last_active":"2023-10-14T17:01:16.781493+0000","last_peered":"2023-10-14T17:01:16.781493+0000","last_clean":"2023-10-14T17:01:16.781493+0000","last_became_active":"2023-10-14T17:01:14.692791+0000","last_became_peered":"2023-10-14T17:01:14.692791+0000","last_unstale":"2023-10-14T17:01:16.781493+0000","last_undegraded":"2023-10-14T17:01:16.781493+0000","last_fullsized":"2023-10-14T17:01:16.781493+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:03:24.860980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"18.0","version":"0'0","reported_seq":12,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.780680+0000","last_change":"2023-10-14T17:01:13.672545+0000","last_active":"2023-10-14T17:01:16.780680+0000","last_peered":"2023-10-14T17:01:16.780680+0000","last_clean":"2023-10-14T17:01:16.780680+0000","last_became_active":"2023-10-14T17:01:13.672246+0000","last_became_peered":"2023-10-14T17:01:13.672246+0000","last_unstale":"2023-10-14T17:01:16.780680+0000","last_undegraded":"2023-10-14T17:01:16.780680+0000","last_fullsized":"2023-10-14T17:01:16.780680+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:04.039372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"19.12","version":"0'0","reported_seq":11,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103486+0000","last_change":"2023-10-14T17:01:14.706496+0000","last_active":"2023-10-14T17:01:16.103486+0000","last_peered":"2023-10-14T17:01:16.103486+0000","last_clean":"2023-10-14T17:01:16.103486+0000","last_became_active":"2023-10-14T17:01:14.705848+0000","last_became_peered":"2023-10-14T17:01:14.705848+0000","last_unstale":"2023-10-14T17:01:16.103486+0000","last_undegraded":"2023-10-14T17:01:16.103486+0000","last_fullsized":"2023-10-14T17:01:16.103486+0000","mapping_epoch":152,"log_start":"0'0","ondisk_log_start":"0'0","created":152,"last_epoch_clean":153,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:13.634886+0000","last_clean_scrub_stamp":"2023-10-14T17:01:13.634886+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:41:42.434427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"18.13","version":"153'2","reported_seq":14,"reported_epoch":154,"state":"active+clean","last_fresh":"2023-10-14T17:01:16.103869+0000","last_change":"2023-10-14T17:01:13.679084+0000","last_active":"2023-10-14T17:01:16.103869+0000","last_peered":"2023-10-14T17:01:16.103869+0000","last_clean":"2023-10-14T17:01:16.103869+0000","last_became_active":"2023-10-14T17:01:13.678707+0000","last_became_peered":"2023-10-14T17:01:13.678707+0000","last_unstale":"2023-10-14T17:01:16.103869+0000","last_undegraded":"2023-10-14T17:01:16.103869+0000","last_fullsized":"2023-10-14T17:01:16.103869+0000","mapping_epoch":151,"log_start":"0'0","ondisk_log_start":"0'0","created":151,"last_epoch_clean":152,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:01:12.620269+0000","last_clean_scrub_stamp":"2023-10-14T17:01:12.620269+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:38:45.393854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"142'249","reported_seq":756,"reported_epoch":153,"state":"active+clean","last_fresh":"2023-10-14T17:01:14.673941+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:01:14.673941+0000","last_peered":"2023-10-14T17:01:14.673941+0000","last_clean":"2023-10-14T17:01:14.673941+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:01:14.673941+0000","last_undegraded":"2023-10-14T17:01:14.673941+0000","last_fullsized":"2023-10-14T17:01:14.673941+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":249,"log_dups_size":0,"ondisk_log_size":249,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":19,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":18,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1268,"num_read_kb":2572,"num_write":399,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":249,"ondisk_log_size":249,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575096,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37816,"kb_used_data":2476,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334024,"statfs":{"total":96636764160,"available":96598040576,"internally_reserved":0,"allocated":2535424,"data_stored":1523528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":36174283},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.101,"5min":1.1319999999999999,"15min":1.1599999999999999},"min":{"1min":0.59899999999999998,"5min":0.504,"15min":0.504},"max":{"1min":1.631,"5min":2.1320000000000001,"15min":2.1320000000000001},"last":1.6200000000000001},{"interface":"front","average":{"1min":1.0940000000000001,"5min":1.1259999999999999,"15min":1.1799999999999999},"min":{"1min":0.68500000000000005,"5min":0.57599999999999996,"15min":0.38900000000000001},"max":{"1min":1.6180000000000001,"5min":1.871,"15min":2.117},"last":1.452}]},{"osd":1,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.177,"5min":1.159,"15min":1.216},"min":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"max":{"1min":1.8260000000000001,"5min":1.8520000000000001,"15min":2.2309999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":1.147,"5min":1.2010000000000001,"15min":1.2430000000000001},"min":{"1min":0.64500000000000002,"5min":0.61499999999999999,"15min":0.57799999999999996},"max":{"1min":1.7370000000000001,"5min":1.9870000000000001,"15min":2.0259999999999998},"last":1.738}]},{"osd":2,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.153,"5min":1.149,"15min":1.2130000000000001},"min":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.47099999999999997},"max":{"1min":1.6679999999999999,"5min":2.0950000000000002,"15min":2.2389999999999999},"last":1.2490000000000001},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.2549999999999999,"15min":1.2909999999999999},"min":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.72599999999999998},"max":{"1min":2.0089999999999999,"5min":2.0089999999999999,"15min":2.1589999999999998},"last":0.998}]},{"osd":3,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.278,"15min":1.351},"min":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"max":{"1min":1.6339999999999999,"5min":2.056,"15min":2.4159999999999999},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.4039999999999999,"15min":1.444},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.72199999999999998},"max":{"1min":2.056,"5min":2.379,"15min":2.379},"last":1.1619999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.2909999999999999,"5min":1.306,"15min":1.278},"min":{"1min":0.59099999999999997,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":1.7230000000000001,"5min":2.0960000000000001,"15min":2.1819999999999999},"last":2.0790000000000002},{"interface":"front","average":{"1min":1.3300000000000001,"5min":1.339,"15min":1.3360000000000001},"min":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"max":{"1min":1.99,"5min":2.2410000000000001,"15min":2.2410000000000001},"last":1.6779999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.361,"5min":1.3959999999999999,"15min":1.3859999999999999},"min":{"1min":0.51200000000000001,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.96,"5min":2.278,"15min":2.278},"last":1.8400000000000001},{"interface":"front","average":{"1min":1.4510000000000001,"5min":1.444,"15min":1.456},"min":{"1min":0.70399999999999996,"5min":0.56999999999999995,"15min":0.56999999999999995},"max":{"1min":2.1240000000000001,"5min":2.2029999999999998,"15min":2.2029999999999998},"last":1.9359999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:01:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4590000000000001,"5min":1.5309999999999999,"15min":1.536},"min":{"1min":0.63200000000000001,"5min":0.53400000000000003,"15min":0.53400000000000003},"max":{"1min":1.8560000000000001,"5min":2.492,"15min":2.492},"last":2.0099999999999998},{"interface":"front","average":{"1min":1.4239999999999999,"5min":1.4970000000000001,"15min":1.482},"min":{"1min":0.61599999999999999,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.8979999999999999,"5min":4.4580000000000002,"15min":4.4580000000000002},"last":1.9670000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575095,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37224,"kb_used_data":1884,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334616,"statfs":{"total":96636764160,"available":96598646784,"internally_reserved":0,"allocated":1929216,"data_stored":932522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36174279},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.512,"5min":1.1779999999999999,"15min":1.1659999999999999},"min":{"1min":0.65700000000000003,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":10.119,"5min":10.119,"15min":10.119},"last":1.262},{"interface":"front","average":{"1min":1.472,"5min":1.2070000000000001,"15min":1.2210000000000001},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"max":{"1min":8.6329999999999991,"5min":8.6329999999999991,"15min":8.6329999999999991},"last":0.86199999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.69299999999999995,"5min":0.67000000000000004,"15min":0.41299999999999998},"max":{"1min":9.6869999999999994,"5min":9.6869999999999994,"15min":9.6869999999999994},"last":0.73499999999999999},{"interface":"front","average":{"1min":1.6659999999999999,"5min":1.306,"15min":1.302},"min":{"1min":0.69899999999999995,"5min":0.53900000000000003,"15min":0.438},"max":{"1min":10.231,"5min":10.231,"15min":10.231},"last":1.863}]},{"osd":2,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.343,"15min":1.355},"min":{"1min":0.95299999999999996,"5min":0.69799999999999995,"15min":0.69799999999999995},"max":{"1min":6.96,"5min":6.96,"15min":6.96},"last":1.2270000000000001},{"interface":"front","average":{"1min":1.7450000000000001,"5min":1.395,"15min":1.4039999999999999},"min":{"1min":0.90000000000000002,"5min":0.72399999999999998,"15min":0.46000000000000002},"max":{"1min":9.8949999999999996,"5min":9.8949999999999996,"15min":9.8949999999999996},"last":1.1899999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.7549999999999999,"5min":1.3939999999999999,"15min":1.385},"min":{"1min":0.85699999999999998,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":10.307,"5min":10.307,"15min":10.307},"last":1.492},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.4019999999999999,"15min":1.4239999999999999},"min":{"1min":0.85799999999999998,"5min":0.64700000000000002,"15min":0.64700000000000002},"max":{"1min":8.4830000000000005,"5min":8.4830000000000005,"15min":8.4830000000000005},"last":1.099}]},{"osd":4,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.845,"5min":1.4219999999999999,"15min":1.371},"min":{"1min":0.97399999999999998,"5min":0.497,"15min":0.497},"max":{"1min":10.48,"5min":10.48,"15min":10.48},"last":1.74},{"interface":"front","average":{"1min":1.8100000000000001,"5min":1.4890000000000001,"15min":1.4490000000000001},"min":{"1min":1.0309999999999999,"5min":0.68799999999999994,"15min":0.68799999999999994},"max":{"1min":9.2230000000000008,"5min":9.2230000000000008,"15min":9.2230000000000008},"last":1.3640000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.843,"5min":1.4430000000000001,"15min":1.409},"min":{"1min":0.88200000000000001,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":9.8179999999999996,"5min":9.8179999999999996,"15min":9.8179999999999996},"last":1.4079999999999999},{"interface":"front","average":{"1min":1.988,"5min":1.611,"15min":1.5760000000000001},"min":{"1min":1.2549999999999999,"5min":0.54600000000000004,"15min":0.54600000000000004},"max":{"1min":10.388999999999999,"5min":10.388999999999999,"15min":10.388999999999999},"last":1.9910000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:01:01 2023","interfaces":[{"interface":"back","average":{"1min":1.9219999999999999,"5min":1.601,"15min":1.6000000000000001},"min":{"1min":1.2569999999999999,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":8.8179999999999996,"5min":8.8179999999999996,"15min":8.8179999999999996},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.9399999999999999,"5min":1.579,"15min":1.554},"min":{"1min":1.1990000000000001,"5min":0.623,"15min":0.623},"max":{"1min":9.9789999999999992,"5min":9.9789999999999992,"15min":9.9789999999999992},"last":1.6759999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575096,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37212,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334628,"statfs":{"total":96636764160,"available":96598659072,"internally_reserved":0,"allocated":1912832,"data_stored":924650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.0640000000000001,"15min":1.103},"min":{"1min":0.623,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.829,"5min":1.829,"15min":1.986},"last":0.66200000000000003},{"interface":"front","average":{"1min":1.242,"5min":1.1799999999999999,"15min":1.149},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":3.5259999999999998,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":1.0720000000000001}]},{"osd":1,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.214,"15min":1.1910000000000001},"min":{"1min":0.78800000000000003,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":2.6629999999999998,"5min":2.6629999999999998,"15min":2.6629999999999998},"last":1.1599999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.177,"15min":1.208},"min":{"1min":0.71499999999999997,"5min":0.47299999999999998,"15min":0.47299999999999998},"max":{"1min":3.3250000000000002,"5min":3.3250000000000002,"15min":3.3250000000000002},"last":0.92900000000000005}]},{"osd":2,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.2030000000000001,"5min":1.115,"15min":1.177},"min":{"1min":0.80500000000000005,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":1.625,"5min":1.681,"15min":2.375},"last":0.748},{"interface":"front","average":{"1min":1.2370000000000001,"5min":1.292,"15min":1.3149999999999999},"min":{"1min":0.82899999999999996,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.8200000000000001,"5min":1.8200000000000001,"15min":2.3100000000000001},"last":1.0089999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.4570000000000001,"5min":1.377,"15min":1.3839999999999999},"min":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"max":{"1min":3.7839999999999998,"5min":3.7839999999999998,"15min":3.7839999999999998},"last":1.2749999999999999},{"interface":"front","average":{"1min":1.405,"5min":1.3320000000000001,"15min":1.3899999999999999},"min":{"1min":0.877,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":2.5910000000000002,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":1.1140000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.3440000000000001,"15min":1.296},"min":{"1min":0.66400000000000003,"5min":0.434,"15min":0.434},"max":{"1min":2.1709999999999998,"5min":2.1709999999999998,"15min":2.4889999999999999},"last":1.3180000000000001},{"interface":"front","average":{"1min":1.496,"5min":1.401,"15min":1.357},"min":{"1min":0.64100000000000001,"5min":0.58099999999999996,"15min":0.58099999999999996},"max":{"1min":3.891,"5min":3.891,"15min":3.891},"last":1.3720000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.6759999999999999,"5min":1.345,"15min":1.3700000000000001},"min":{"1min":0.90500000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"max":{"1min":6.6600000000000001,"5min":6.6600000000000001,"15min":6.6600000000000001},"last":1.218},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.532,"15min":1.5309999999999999},"min":{"1min":0.875,"5min":0.63,"15min":0.63},"max":{"1min":5.2039999999999997,"5min":5.2039999999999997,"15min":5.2039999999999997},"last":1.4550000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:00:56 2023","interfaces":[{"interface":"back","average":{"1min":1.599,"5min":1.502,"15min":1.5049999999999999},"min":{"1min":0.82799999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":3.9929999999999999,"5min":3.9929999999999999,"15min":3.9929999999999999},"last":1.554},{"interface":"front","average":{"1min":1.8140000000000001,"5min":1.5049999999999999,"15min":1.514},"min":{"1min":1.0369999999999999,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":6.5359999999999996,"5min":6.5359999999999996,"15min":6.5359999999999996},"last":1.4159999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575096,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59432,"kb_used_data":1876,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312408,"statfs":{"total":96636764160,"available":96575905792,"internally_reserved":0,"allocated":1921024,"data_stored":930980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.1140000000000001,"5min":1.1379999999999999,"15min":1.1659999999999999},"min":{"1min":0.70399999999999996,"5min":0.623,"15min":0.55400000000000005},"max":{"1min":1.643,"5min":1.9319999999999999,"15min":1.982},"last":1.1160000000000001},{"interface":"front","average":{"1min":1.177,"5min":1.2330000000000001,"15min":1.23},"min":{"1min":0.73599999999999999,"5min":0.63900000000000001,"15min":0.63900000000000001},"max":{"1min":1.8029999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"last":0.754}]},{"osd":1,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.306,"15min":1.2809999999999999},"min":{"1min":0.85999999999999999,"5min":0.69099999999999995,"15min":0.48199999999999998},"max":{"1min":1.607,"5min":6.7880000000000003,"15min":6.7880000000000003},"last":1.4870000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.212,"15min":1.2390000000000001},"min":{"1min":0.92000000000000004,"5min":0.60499999999999998,"15min":0.60499999999999998},"max":{"1min":1.7430000000000001,"5min":5.9530000000000003,"15min":5.9530000000000003},"last":1.355}]},{"osd":2,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.127,"5min":1.248,"15min":1.2410000000000001},"min":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.64700000000000002},"max":{"1min":1.865,"5min":5.633,"15min":5.633},"last":1.7230000000000001},{"interface":"front","average":{"1min":1.383,"5min":1.3640000000000001,"15min":1.377},"min":{"1min":0.84999999999999998,"5min":0.81100000000000005,"15min":0.71999999999999997},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":1.302}]},{"osd":3,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3100000000000001,"15min":1.3160000000000001},"min":{"1min":1.1240000000000001,"5min":0.77100000000000002,"15min":0.71699999999999997},"max":{"1min":1.825,"5min":4.6349999999999998,"15min":4.6349999999999998},"last":1.238},{"interface":"front","average":{"1min":1.343,"5min":1.405,"15min":1.403},"min":{"1min":0.92400000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"max":{"1min":2.0779999999999998,"5min":3.5590000000000002,"15min":3.5590000000000002},"last":1.6459999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.486,"5min":1.4179999999999999,"15min":1.375},"min":{"1min":1.032,"5min":0.64500000000000002,"15min":0.52900000000000003},"max":{"1min":1.877,"5min":2.931,"15min":2.931},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.3759999999999999,"5min":1.361,"15min":1.343},"min":{"1min":1.0920000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.732,"5min":3.2599999999999998,"15min":3.2599999999999998},"last":1.5609999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.4330000000000001,"15min":1.399},"min":{"1min":1.238,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.768,"5min":2.323,"15min":2.323},"last":1.526},{"interface":"front","average":{"1min":1.5580000000000001,"5min":1.544,"15min":1.53},"min":{"1min":1.125,"5min":0.55900000000000005,"15min":0.55900000000000005},"max":{"1min":1.988,"5min":2.9889999999999999,"15min":2.9889999999999999},"last":1.595}]},{"osd":7,"last update":"Sat Oct 14 17:00:59 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.536,"15min":1.492},"min":{"1min":1.256,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.9790000000000001,"5min":4.7960000000000003,"15min":4.7960000000000003},"last":1.4379999999999999},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.556,"15min":1.5529999999999999},"min":{"1min":1.1659999999999999,"5min":0.82899999999999996,"15min":0.79000000000000004},"max":{"1min":1.804,"5min":2.004,"15min":2.3130000000000002},"last":1.397}]}]},{"osd":3,"up_from":13,"seq":55834575096,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":45528,"kb_used_data":1868,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":94326312,"statfs":{"total":96636764160,"available":96590143488,"internally_reserved":0,"allocated":1912832,"data_stored":924698,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":44693960},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":0.94699999999999995,"5min":0.90700000000000003,"15min":0.92200000000000004},"min":{"1min":0.39900000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":1.9350000000000001,"5min":1.9350000000000001,"15min":1.9350000000000001},"last":1.079},{"interface":"front","average":{"1min":1.143,"5min":1.0629999999999999,"15min":1.1220000000000001},"min":{"1min":0.48099999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":2.0870000000000002,"5min":2.0870000000000002,"15min":2.0870000000000002},"last":0.93400000000000005}]},{"osd":1,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.032,"5min":0.90700000000000003,"15min":0.91400000000000003},"min":{"1min":0.51000000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.5840000000000001,"5min":2.5840000000000001,"15min":2.5840000000000001},"last":1.048},{"interface":"front","average":{"1min":1.0629999999999999,"5min":1.0740000000000001,"15min":1.1000000000000001},"min":{"1min":0.53200000000000003,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":2.2949999999999999,"5min":2.2949999999999999,"15min":2.2949999999999999},"last":0.97299999999999998}]},{"osd":2,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.282,"5min":1.2589999999999999,"15min":1.3380000000000001},"min":{"1min":0.56000000000000005,"5min":0.49199999999999999,"15min":0.49199999999999999},"max":{"1min":2.9910000000000001,"5min":2.9910000000000001,"15min":2.9910000000000001},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.093,"5min":1.198,"15min":1.248},"min":{"1min":0.57999999999999996,"5min":0.48299999999999998,"15min":0.48299999999999998},"max":{"1min":1.6859999999999999,"5min":1.883,"15min":2.0960000000000001},"last":1.2250000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3320000000000001,"15min":1.353},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":2.3420000000000001,"5min":2.3420000000000001,"15min":3.4350000000000001},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.3620000000000001,"15min":1.3979999999999999},"min":{"1min":0.59499999999999997,"5min":0.59499999999999997,"15min":0.59499999999999997},"max":{"1min":1.8320000000000001,"5min":2.8290000000000002,"15min":2.8290000000000002},"last":1.351}]},{"osd":5,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.3480000000000001,"5min":1.339,"15min":1.3779999999999999},"min":{"1min":0.755,"5min":0.755,"15min":0.755},"max":{"1min":2.4630000000000001,"5min":2.4630000000000001,"15min":2.516},"last":1.4370000000000001},{"interface":"front","average":{"1min":1.5,"5min":1.4179999999999999,"15min":1.4390000000000001},"min":{"1min":0.88100000000000001,"5min":0.82699999999999996,"15min":0.80600000000000005},"max":{"1min":2.7360000000000002,"5min":2.7360000000000002,"15min":3.7410000000000001},"last":1.7829999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.46,"15min":1.462},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"max":{"1min":2.4129999999999998,"5min":2.9710000000000001,"15min":2.9710000000000001},"last":1.6819999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.5620000000000001,"15min":1.5469999999999999},"min":{"1min":0.93899999999999995,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":2.5089999999999999,"5min":3.3690000000000002,"15min":3.3690000000000002},"last":1.5029999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:00:45 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.464,"15min":1.484},"min":{"1min":0.90000000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"max":{"1min":2.6549999999999998,"5min":2.6549999999999998,"15min":2.871},"last":1.718},{"interface":"front","average":{"1min":1.4770000000000001,"5min":1.506,"15min":1.536},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.8479999999999999,"5min":2.8479999999999999,"15min":2.907},"last":1.5349999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575096,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":46948,"kb_used_data":1880,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324892,"statfs":{"total":96636764160,"available":96588689408,"internally_reserved":0,"allocated":1925120,"data_stored":931526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.91600000000000004,"5min":0.88400000000000001,"15min":0.89600000000000002},"min":{"1min":0.32400000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.6879999999999999,"5min":1.706,"15min":1.706},"last":0.77600000000000002},{"interface":"front","average":{"1min":1.0589999999999999,"5min":0.98199999999999998,"15min":1.03},"min":{"1min":0.55000000000000004,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":2.2160000000000002,"5min":2.2160000000000002,"15min":2.2160000000000002},"last":1.0189999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":0.89100000000000001,"5min":0.874,"15min":0.95699999999999996},"min":{"1min":0.435,"5min":0.372,"15min":0.372},"max":{"1min":1.532,"5min":1.544,"15min":2.0419999999999998},"last":1.0629999999999999},{"interface":"front","average":{"1min":1.101,"5min":1.083,"15min":1.161},"min":{"1min":0.53100000000000003,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":1.909,"5min":1.909,"15min":2.069},"last":0.92300000000000004}]},{"osd":3,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.105,"15min":1.196},"min":{"1min":0.49199999999999999,"5min":0.48999999999999999,"15min":0.46500000000000002},"max":{"1min":2.3769999999999998,"5min":2.3769999999999998,"15min":2.3769999999999998},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.1890000000000001,"5min":1.173,"15min":1.2809999999999999},"min":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.46899999999999997},"max":{"1min":1.75,"5min":1.75,"15min":2.0459999999999998},"last":1.3660000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.3160000000000001,"15min":1.3959999999999999},"min":{"1min":0.78200000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":2.3029999999999999,"5min":2.3029999999999999,"15min":2.3159999999999998},"last":1.536},{"interface":"front","average":{"1min":1.208,"5min":1.268,"15min":1.3460000000000001},"min":{"1min":0.88600000000000001,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":2.4089999999999998,"5min":2.4089999999999998,"15min":2.4089999999999998},"last":1.508}]},{"osd":5,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3680000000000001,"5min":1.2809999999999999,"15min":1.3720000000000001},"min":{"1min":0.76000000000000001,"5min":0.67200000000000004,"15min":0.59199999999999997},"max":{"1min":2.7890000000000001,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.903,"5min":1.508,"15min":1.536},"min":{"1min":0.97199999999999998,"5min":0.56000000000000005,"15min":0.56000000000000005},"max":{"1min":7.5759999999999996,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":1.476}]},{"osd":6,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.7170000000000001,"5min":1.4279999999999999,"15min":1.444},"min":{"1min":0.74399999999999999,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":7.0099999999999998,"5min":7.0099999999999998,"15min":7.0099999999999998},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.8029999999999999,"5min":1.4810000000000001,"15min":1.5249999999999999},"min":{"1min":1.0089999999999999,"5min":0.57399999999999995,"15min":0.57399999999999995},"max":{"1min":7.3179999999999996,"5min":7.3179999999999996,"15min":7.3179999999999996},"last":1.4219999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:01:00 2023","interfaces":[{"interface":"back","average":{"1min":1.538,"5min":1.4339999999999999,"15min":1.506},"min":{"1min":1.042,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.6579999999999999,"5min":2.6579999999999999,"15min":2.6579999999999999},"last":1.607},{"interface":"front","average":{"1min":1.5429999999999999,"5min":1.4670000000000001,"15min":1.5660000000000001},"min":{"1min":1,"5min":0.624,"15min":0.624},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":2.5369999999999999},"last":1.71}]}]},{"osd":1,"up_from":13,"seq":55834575096,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":1860,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":1904640,"data_stored":924538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.85399999999999998,"15min":0.89800000000000002},"min":{"1min":0.441,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.6759999999999999,"5min":2.113,"15min":2.113},"last":0.85599999999999998},{"interface":"front","average":{"1min":0.95899999999999996,"5min":0.91800000000000004,"15min":1.0369999999999999},"min":{"1min":0.42599999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.0819999999999999},"last":0.629}]},{"osd":2,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":0.84899999999999998,"5min":0.88500000000000001,"15min":0.93899999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.5449999999999999,"5min":1.7509999999999999,"15min":1.758},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.028,"5min":1.0820000000000001,"15min":1.1459999999999999},"min":{"1min":0.59199999999999997,"5min":0.51900000000000002,"15min":0.51300000000000001},"max":{"1min":1.423,"5min":2.008,"15min":2.0150000000000001},"last":0.66500000000000004}]},{"osd":3,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.145,"5min":1.167,"15min":1.2649999999999999},"min":{"1min":0.55600000000000005,"5min":0.54000000000000004,"15min":0.49399999999999999},"max":{"1min":1.615,"5min":1.984,"15min":2.198},"last":0.93999999999999995},{"interface":"front","average":{"1min":1.1060000000000001,"5min":1.0800000000000001,"15min":1.165},"min":{"1min":0.495,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":1.5760000000000001,"5min":1.974,"15min":2.004},"last":0.995}]},{"osd":4,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.3089999999999999,"15min":1.385},"min":{"1min":0.72899999999999998,"5min":0.628,"15min":0.628},"max":{"1min":1.8140000000000001,"5min":2.052,"15min":2.3050000000000002},"last":1.429},{"interface":"front","average":{"1min":1.206,"5min":1.2769999999999999,"15min":1.3},"min":{"1min":0.83699999999999997,"5min":0.64100000000000001,"15min":0.53900000000000003},"max":{"1min":1.6319999999999999,"5min":1.9119999999999999,"15min":1.9430000000000001},"last":1.3280000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.222,"5min":1.3029999999999999,"15min":1.345},"min":{"1min":0.754,"5min":0.754,"15min":0.63100000000000001},"max":{"1min":1.9390000000000001,"5min":2.0430000000000001,"15min":2.3180000000000001},"last":1.393},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.3859999999999999,"15min":1.444},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.76200000000000001},"max":{"1min":1.8080000000000001,"5min":2.1480000000000001,"15min":2.302},"last":1.5800000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.405,"15min":1.4710000000000001},"min":{"1min":0.77300000000000002,"5min":0.65900000000000003,"15min":0.65900000000000003},"max":{"1min":1.839,"5min":2.121,"15min":2.4169999999999998},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.4219999999999999,"5min":1.4299999999999999,"15min":1.458},"min":{"1min":0.84799999999999998,"5min":0.78600000000000003,"15min":0.78600000000000003},"max":{"1min":1.8580000000000001,"5min":2.0870000000000002,"15min":2.1070000000000002},"last":1.607}]},{"osd":7,"last update":"Sat Oct 14 17:01:12 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4670000000000001,"15min":1.5389999999999999},"min":{"1min":0.92100000000000004,"5min":0.71999999999999997,"15min":0.71999999999999997},"max":{"1min":1.891,"5min":2.2410000000000001,"15min":2.411},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.4870000000000001,"5min":1.4670000000000001,"15min":1.552},"min":{"1min":0.871,"5min":0.81899999999999995,"15min":0.61599999999999999},"max":{"1min":1.976,"5min":2.0910000000000002,"15min":2.6600000000000001},"last":1.6459999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575095,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41944,"kb_used_data":2440,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329896,"statfs":{"total":96636764160,"available":96593813504,"internally_reserved":0,"allocated":2498560,"data_stored":1514906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":0.90500000000000003,"15min":0.90400000000000003},"min":{"1min":0.46700000000000003,"5min":0.309,"15min":0.309},"max":{"1min":1.577,"5min":3.355,"15min":3.355},"last":0.433},{"interface":"front","average":{"1min":1.05,"5min":0.98099999999999998,"15min":1.0649999999999999},"min":{"1min":0.45800000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":1.9219999999999999,"5min":3.1230000000000002,"15min":3.1230000000000002},"last":0.65000000000000002}]},{"osd":2,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.016,"15min":1.071},"min":{"1min":0.53700000000000003,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.3720000000000001,"5min":4.3940000000000001,"15min":4.3940000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.97799999999999998,"5min":1.0549999999999999,"15min":1.1639999999999999},"min":{"1min":0.48299999999999998,"5min":0.432,"15min":0.432},"max":{"1min":1.8149999999999999,"5min":4.2009999999999996,"15min":4.2009999999999996},"last":0.755}]},{"osd":3,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2430000000000001,"5min":1.1970000000000001,"15min":1.2689999999999999},"min":{"1min":0.59299999999999997,"5min":0.502,"15min":0.502},"max":{"1min":1.6870000000000001,"5min":4.2779999999999996,"15min":4.2779999999999996},"last":0.79200000000000004},{"interface":"front","average":{"1min":1.2150000000000001,"5min":1.2150000000000001,"15min":1.3220000000000001},"min":{"1min":0.55400000000000005,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":2.0550000000000002,"5min":2.581,"15min":2.593},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.147,"5min":1.2470000000000001,"15min":1.272},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7290000000000001,"5min":5.335,"15min":5.335},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.3480000000000001,"5min":1.363,"15min":1.4079999999999999},"min":{"1min":0.86299999999999999,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":1.9490000000000001,"5min":3.4609999999999999,"15min":3.4609999999999999},"last":1.254}]},{"osd":5,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.24,"15min":1.3},"min":{"1min":0.83499999999999996,"5min":0.56200000000000006,"15min":0.56200000000000006},"max":{"1min":2.2160000000000002,"5min":2.931,"15min":2.931},"last":1.21},{"interface":"front","average":{"1min":1.405,"5min":1.3640000000000001,"15min":1.4790000000000001},"min":{"1min":0.92300000000000004,"5min":0.69099999999999995,"15min":0.69099999999999995},"max":{"1min":1.9790000000000001,"5min":3.306,"15min":3.306},"last":0.95299999999999996}]},{"osd":6,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.46,"15min":1.5589999999999999},"min":{"1min":0.84899999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"max":{"1min":2.1619999999999999,"5min":3.4329999999999998,"15min":3.4329999999999998},"last":0.996},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.3280000000000001,"15min":1.3859999999999999},"min":{"1min":0.76500000000000001,"5min":0.76500000000000001,"15min":0.76500000000000001},"max":{"1min":1.911,"5min":3.5830000000000002,"15min":3.5830000000000002},"last":1.03}]},{"osd":7,"last update":"Sat Oct 14 17:00:55 2023","interfaces":[{"interface":"back","average":{"1min":1.4950000000000001,"5min":1.462,"15min":1.5349999999999999},"min":{"1min":0.78900000000000003,"5min":0.747,"15min":0.747},"max":{"1min":1.8160000000000001,"5min":2.7469999999999999,"15min":2.7469999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.4430000000000001,"5min":1.4339999999999999,"15min":1.508},"min":{"1min":0.71599999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.113,"5min":3.698,"15min":3.698},"last":0.96899999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":18,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":19,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:01:19.701 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T17:01:19.701 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T17:01:19.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.974+0000 7f85d4c72640 1 Processor -- start 2023-10-14T17:01:19.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.974+0000 7f85d4c72640 1 -- start start 2023-10-14T17:01:19.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.975+0000 7f85d4c72640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f85d015ac70 0x7f85d00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.975+0000 7f85d4c72640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85d00cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.975+0000 7f85d4c72640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00cc850 con 0x7f85d00b4160 2023-10-14T17:01:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.975+0000 7f85d4c72640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85d00ccb30 con 0x7f85d015ac70 2023-10-14T17:01:19.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.975+0000 7f85d4c72640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00cce10 con 0x7f85d006a4b0 2023-10-14T17:01:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85ceffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f85d00b4160 0x7f85d0069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54038/0 (socket says 172.21.15.29:54038) 2023-10-14T17:01:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85ceffd640 1 -- 172.21.15.29:0/2849878678 learned_addr learned my addr 172.21.15.29:0/2849878678 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85cffff640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85d00cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2956955712 0 0) 0x7f85d00cc850 con 0x7f85d00b4160 2023-10-14T17:01:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b8003520 con 0x7f85d00b4160 2023-10-14T17:01:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.976+0000 7f85cf7fe640 1 --2- 172.21.15.29:0/2849878678 >> v2:172.21.15.186:3300/0 conn(0x7f85d015ac70 0x7f85d00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1612392052 0 0) 0x7f85b8003520 con 0x7f85d00b4160 2023-10-14T17:01:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 msgr2=0x7f85d00cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:19.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85d00cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 >> v2:172.21.15.186:3300/0 conn(0x7f85d015ac70 msgr2=0x7f85d00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 --2- 172.21.15.29:0/2849878678 >> v2:172.21.15.186:3300/0 conn(0x7f85d015ac70 0x7f85d00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00cd0f0 con 0x7f85d00b4160 2023-10-14T17:01:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f85c4001670 con 0x7f85d00b4160 2023-10-14T17:01:19.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 >> v1:172.21.15.29:6789/0 conn(0x7f85d00b4160 legacy=0x7f85d0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85b8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b8003ef0 0x7f85b80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b8004d00 0x7f85b8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00ccb30 con 0x7f85b8004d00 2023-10-14T17:01:19.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00cce10 con 0x7f85b8003ef0 2023-10-14T17:01:19.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b800b980 con 0x7f85d006a4b0 2023-10-14T17:01:19.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85cffff640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b8004d00 0x7f85b8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.977+0000 7f85cf7fe640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85b8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85ceffd640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b8003ef0 0x7f85b80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85cffff640 1 -- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 msgr2=0x7f85b8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:19.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85cffff640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85d006a4b0 0x7f85b8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85cffff640 1 -- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b8003ef0 msgr2=0x7f85b80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85cffff640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b8003ef0 0x7f85b80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85cffff640 1 -- 172.21.15.29:0/2849878678 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b8003520 con 0x7f85b8004d00 2023-10-14T17:01:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.978+0000 7f85ceffd640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85b8003ef0 0x7f85b80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:19.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.979+0000 7f85cffff640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b8004d00 0x7f85b8005180 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f85d006d350 tx=0x7f85b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=962626736cb69d72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.979+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c802c650 con 0x7f85b8004d00 2023-10-14T17:01:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.979+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c802c7b0 con 0x7f85b8004d00 2023-10-14T17:01:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.979+0000 7f85ce7fc640 1 -- 172.21.15.29:0/2849878678 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c802cb40 con 0x7f85b8004d00 2023-10-14T17:01:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 -- 172.21.15.29:0/2849878678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b8004d00 msgr2=0x7f85b8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 --2- 172.21.15.29:0/2849878678 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85b8004d00 0x7f85b8005180 secure :-1 s=READY pgs=1803 cs=0 l=1 rev1=1 crypto rx=0x7f85d006d350 tx=0x7f85b80057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85cffff640 1 -- 172.21.15.29:0/2849878678 reap_dead start 2023-10-14T17:01:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 -- 172.21.15.29:0/2849878678 shutdown_connections 2023-10-14T17:01:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 -- 172.21.15.29:0/2849878678 >> 172.21.15.29:0/2849878678 conn(0x7f85d00ce6a0 msgr2=0x7f85d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 -- 172.21.15.29:0/2849878678 shutdown_connections 2023-10-14T17:01:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.980+0000 7f85d4c72640 1 -- 172.21.15.29:0/2849878678 wait complete. 2023-10-14T17:01:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.982+0000 7f85d4c72640 1 Processor -- start 2023-10-14T17:01:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.982+0000 7f85d4c72640 1 -- start start 2023-10-14T17:01:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b8004d00 0x7f85d00c6660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85d00c4d80 0x7f85d00b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c6ba0 con 0x7f85d00c5770 2023-10-14T17:01:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00708f0 con 0x7f85d00c4d80 2023-10-14T17:01:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85d4c72640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d00c3a80 con 0x7f85b8004d00 2023-10-14T17:01:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85ceffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47362/0 (socket says 172.21.15.29:47362) 2023-10-14T17:01:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85ceffd640 1 -- 172.21.15.29:0/3044504965 learned_addr learned my addr 172.21.15.29:0/3044504965 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85cf7fe640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b8004d00 0x7f85d00c6660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.983+0000 7f85cffff640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85d00c4d80 0x7f85d00b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 -- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b8004d00 msgr2=0x7f85d00c6660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b8004d00 0x7f85d00c6660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 -- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85d00c4d80 msgr2=0x7f85d00b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85d00c4d80 0x7f85d00b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 -- 172.21.15.29:0/3044504965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00c3d60 con 0x7f85d00c5770 2023-10-14T17:01:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85ceffd640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f85c400d8c0 tx=0x7f85c400dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4007560 con 0x7f85d00c5770 2023-10-14T17:01:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85c40076c0 con 0x7f85d00c5770 2023-10-14T17:01:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85c4004a00 con 0x7f85d00c5770 2023-10-14T17:01:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d00c4040 con 0x7f85d00c5770 2023-10-14T17:01:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.984+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d00c3730 con 0x7f85d00c5770 2023-10-14T17:01:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.986+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d00c3bc0 con 0x7f85d00c5770 2023-10-14T17:01:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.987+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f85c4004b60 con 0x7f85d00c5770 2023-10-14T17:01:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.988+0000 7f85b7fff640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 0x7f85bc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.989+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f85c4089530 con 0x7f85d00c5770 2023-10-14T17:01:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.989+0000 7f85cf7fe640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 0x7f85bc06d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.989+0000 7f85cf7fe640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 0x7f85bc06d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85d01f53b0 tx=0x7f85c000b040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:19.992+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f85c404efe0 con 0x7f85d00c5770 2023-10-14T17:01:20.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.261+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f85d004e9b0 con 0x7f85d00c5770 2023-10-14T17:01:20.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.271+0000 7f85b7fff640 1 -- 172.21.15.29:0/3044504965 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v21) v1 ==== 190+0+147 (secure 0 0 0) 0x7f85c40190b0 con 0x7f85d00c5770 2023-10-14T17:01:20.272 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T17:01:20.272 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T17:01:20.273 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T17:01:20.273 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T17:01:20.273 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T17:01:20.273 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T17:01:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 msgr2=0x7f85bc06d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 0x7f85bc06d4d0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f85d01f53b0 tx=0x7f85c000b040 comp rx=0 tx=0).stop 2023-10-14T17:01:20.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 msgr2=0x7f85d00c4840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 secure :-1 s=READY pgs=1804 cs=0 l=1 rev1=1 crypto rx=0x7f85c400d8c0 tx=0x7f85c400dd80 comp rx=0 tx=0).stop 2023-10-14T17:01:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 shutdown_connections 2023-10-14T17:01:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f85bc06b010 0x7f85bc06d4d0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85d00c4d80 0x7f85d00b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85d00c5770 0x7f85d00c4840 unknown :-1 s=CLOSED pgs=1804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 --2- 172.21.15.29:0/3044504965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85b8004d00 0x7f85d00c6660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 >> 172.21.15.29:0/3044504965 conn(0x7f85d00ce6a0 msgr2=0x7f85d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 shutdown_connections 2023-10-14T17:01:20.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.274+0000 7f85d4c72640 1 -- 172.21.15.29:0/3044504965 wait complete. 2023-10-14T17:01:20.294 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:01:20.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.514+0000 7f1cc33c9640 1 Processor -- start 2023-10-14T17:01:20.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.514+0000 7f1cc33c9640 1 -- start start 2023-10-14T17:01:20.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.516+0000 7f1cc33c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 0x7f1cbc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.516+0000 7f1cc33c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.516+0000 7f1cc33c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cbc15b9d0 con 0x7f1cbc06c040 2023-10-14T17:01:20.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.516+0000 7f1cc33c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cbc15bcb0 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.516+0000 7f1cc33c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc15bf90 con 0x7f1cbc0ad970 2023-10-14T17:01:20.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.517+0000 7f1cc1bc6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 0x7f1cbc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.517+0000 7f1cc2bc8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.517+0000 7f1cc2bc8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:56472/0 (socket says 172.21.15.29:56472) 2023-10-14T17:01:20.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.517+0000 7f1cc2bc8640 1 -- 172.21.15.29:0/3859646936 learned_addr learned my addr 172.21.15.29:0/3859646936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:20.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.518+0000 7f1cc2bc8640 1 -- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 msgr2=0x7f1cbc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:20.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.518+0000 7f1cc2bc8640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 0x7f1cbc0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.518+0000 7f1cc2bc8640 1 -- 172.21.15.29:0/3859646936 >> v1:172.21.15.186:6789/0 conn(0x7f1cbc06c040 legacy=0x7f1cbc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.518+0000 7f1cc2bc8640 1 -- 172.21.15.29:0/3859646936 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc0cc270 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.518+0000 7f1cc1bc6640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 0x7f1cbc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:01:20.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.519+0000 7f1cc2bc8640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009e80 tx=0x7f1cb402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=308a08f12a7a9dc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.520+0000 7f1cc13c5640 1 -- 172.21.15.29:0/3859646936 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb402cae0 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.520+0000 7f1cc13c5640 1 -- 172.21.15.29:0/3859646936 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb402cc40 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.520+0000 7f1cc13c5640 1 -- 172.21.15.29:0/3859646936 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb40343f0 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.520+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 msgr2=0x7f1cbc0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.520+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7f1cb4009e80 tx=0x7f1cb402bf40 comp rx=0 tx=0).stop 2023-10-14T17:01:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3859646936 shutdown_connections 2023-10-14T17:01:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc0ad970 0x7f1cbc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3859646936 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0cd440 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3859646936 >> 172.21.15.29:0/3859646936 conn(0x7f1cbc0ce6a0 msgr2=0x7f1cbc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3859646936 shutdown_connections 2023-10-14T17:01:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.521+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3859646936 wait complete. 2023-10-14T17:01:20.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.523+0000 7f1cc33c9640 1 Processor -- start 2023-10-14T17:01:20.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.523+0000 7f1cc33c9640 1 -- start start 2023-10-14T17:01:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 0x7f1cbc0aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0ae840 con 0x7f1cbc0ad970 2023-10-14T17:01:20.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0c7b80 con 0x7f1cbc0ccfb0 2023-10-14T17:01:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.524+0000 7f1cc33c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc0c78a0 con 0x7f1cbc06c040 2023-10-14T17:01:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47364/0 (socket says 172.21.15.29:47364) 2023-10-14T17:01:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 -- 172.21.15.29:0/3463744033 learned_addr learned my addr 172.21.15.29:0/3463744033 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:20.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc23c7640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 0x7f1cbc0aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 -- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 msgr2=0x7f1cbc0aecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc2bc8640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 0x7f1cbc0aecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 -- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 msgr2=0x7f1cbc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc1bc6640 1 -- 172.21.15.29:0/3463744033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cac009400 con 0x7f1cbc0ad970 2023-10-14T17:01:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.525+0000 7f1cc23c7640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 0x7f1cbc0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cc1bc6640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00ebe0 tx=0x7f1cac00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:20.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac009760 con 0x7f1cbc0ad970 2023-10-14T17:01:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cac00cc30 con 0x7f1cbc0ad970 2023-10-14T17:01:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cb4009b30 con 0x7f1cbc0ad970 2023-10-14T17:01:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cac00eee0 con 0x7f1cbc0ad970 2023-10-14T17:01:20.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.526+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cbc0ae4d0 con 0x7f1cbc0ad970 2023-10-14T17:01:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.527+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cbc06b330 con 0x7f1cbc0ad970 2023-10-14T17:01:20.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.528+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1cac013470 con 0x7f1cbc0ad970 2023-10-14T17:01:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.529+0000 7f1cb2ffd640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 0x7f1ca006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.529+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f1cac089ae0 con 0x7f1cbc0ad970 2023-10-14T17:01:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.530+0000 7f1cc23c7640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 0x7f1ca006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:20.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.530+0000 7f1cc23c7640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 0x7f1ca006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1e9660 tx=0x7f1cb8009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:20.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.536+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cac054a20 con 0x7f1cbc0ad970 2023-10-14T17:01:20.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.767+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cbc04e9b0 con 0x7f1cbc0ad970 2023-10-14T17:01:20.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.768+0000 7f1cb2ffd640 1 -- 172.21.15.29:0/3463744033 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f1cac058040 con 0x7f1cbc0ad970 2023-10-14T17:01:20.768 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:20.769 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:01:20.769 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:01:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.770+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 msgr2=0x7f1ca006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.770+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 0x7f1ca006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1cbc1e9660 tx=0x7f1cb8009040 comp rx=0 tx=0).stop 2023-10-14T17:01:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.770+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 msgr2=0x7f1cbc0c6f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:20.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.770+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 secure :-1 s=READY pgs=1805 cs=0 l=1 rev1=1 crypto rx=0x7f1cac00ebe0 tx=0x7f1cac00c520 comp rx=0 tx=0).stop 2023-10-14T17:01:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 shutdown_connections 2023-10-14T17:01:20.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f1ca006af90 0x7f1ca006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cbc0ccfb0 0x7f1cbc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cbc0ad970 0x7f1cbc0c6f50 unknown :-1 s=CLOSED pgs=1805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 --2- 172.21.15.29:0/3463744033 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cbc06c040 0x7f1cbc0aecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:20.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 >> 172.21.15.29:0/3463744033 conn(0x7f1cbc0ce6a0 msgr2=0x7f1cbc0cd3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:20.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 shutdown_connections 2023-10-14T17:01:20.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:20.771+0000 7f1cc33c9640 1 -- 172.21.15.29:0/3463744033 wait complete. 2023-10-14T17:01:21.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 74, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T17:01:14.133527+0000', 'modified': '2023-10-14T17:01:17.203772+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9105}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9105': {'gid': 9105, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/1190525894', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 1190525894}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 1190525894}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [19], 'metadata_pool': 18, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T17:01:21.193 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T17:01:21.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-14T17:01:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.434+0000 7fa71f59e640 1 Processor -- start 2023-10-14T17:01:21.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.434+0000 7fa71f59e640 1 -- start start 2023-10-14T17:01:21.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7200ad970 0x7fa7200cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72015b9d0 con 0x7fa7200ccfb0 2023-10-14T17:01:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa72015bcb0 con 0x7fa72006c040 2023-10-14T17:01:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa72015bf90 con 0x7fa7200ad970 2023-10-14T17:01:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa715d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7200ad970 0x7fa7200cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa7200ccfb0 0x7fa7200cd460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41248/0 (socket says 172.21.15.29:41248) 2023-10-14T17:01:21.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.436+0000 7fa71ed9d640 1 -- 172.21.15.29:0/903236091 learned_addr learned my addr 172.21.15.29:0/903236091 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.437+0000 7fa715d9b640 1 -- 172.21.15.29:0/903236091 >> v1:172.21.15.186:6789/0 conn(0x7fa72006c040 legacy=0x7fa72006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.437+0000 7fa715d9b640 1 -- 172.21.15.29:0/903236091 >> v1:172.21.15.29:6789/0 conn(0x7fa7200ccfb0 legacy=0x7fa7200cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.437+0000 7fa71dd9b640 1 -- 172.21.15.29:0/903236091 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3551226441 0 0) 0x7fa72015b9d0 con 0x7fa7200ccfb0 2023-10-14T17:01:21.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.437+0000 7fa715d9b640 1 -- 172.21.15.29:0/903236091 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7200cc270 con 0x7fa7200ad970 2023-10-14T17:01:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.438+0000 7fa715d9b640 1 --2- 172.21.15.29:0/903236091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7200ad970 0x7fa7200cca70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fa708009dc0 tx=0x7fa70802be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3fece634b84510 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.438+0000 7fa71dd9b640 1 -- 172.21.15.29:0/903236091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70802cc40 con 0x7fa7200ad970 2023-10-14T17:01:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.438+0000 7fa71dd9b640 1 -- 172.21.15.29:0/903236091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa70802cda0 con 0x7fa7200ad970 2023-10-14T17:01:21.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71dd9b640 1 -- 172.21.15.29:0/903236091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708034660 con 0x7fa7200ad970 2023-10-14T17:01:21.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 -- 172.21.15.29:0/903236091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7200ad970 msgr2=0x7fa7200cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 --2- 172.21.15.29:0/903236091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7200ad970 0x7fa7200cca70 secure :-1 s=READY pgs=1807 cs=0 l=1 rev1=1 crypto rx=0x7fa708009dc0 tx=0x7fa70802be90 comp rx=0 tx=0).stop 2023-10-14T17:01:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 -- 172.21.15.29:0/903236091 shutdown_connections 2023-10-14T17:01:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 --2- 172.21.15.29:0/903236091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7200ad970 0x7fa7200cca70 unknown :-1 s=CLOSED pgs=1807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 -- 172.21.15.29:0/903236091 >> 172.21.15.29:0/903236091 conn(0x7fa7200ce6a0 msgr2=0x7fa7200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 -- 172.21.15.29:0/903236091 shutdown_connections 2023-10-14T17:01:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.439+0000 7fa71f59e640 1 -- 172.21.15.29:0/903236091 wait complete. 2023-10-14T17:01:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.441+0000 7fa71f59e640 1 Processor -- start 2023-10-14T17:01:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.441+0000 7fa71f59e640 1 -- start start 2023-10-14T17:01:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7200ad970 0x7fa7200adde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7200ccfb0 0x7fa7200ae320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7200c7e10 con 0x7fa72006c040 2023-10-14T17:01:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa720156ab0 con 0x7fa7200ad970 2023-10-14T17:01:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa720156d90 con 0x7fa7200ccfb0 2023-10-14T17:01:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37902/0 (socket says 172.21.15.29:37902) 2023-10-14T17:01:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.442+0000 7fa71e59c640 1 -- 172.21.15.29:0/625822660 learned_addr learned my addr 172.21.15.29:0/625822660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa715d9b640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7200ad970 0x7fa7200adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 -- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7200ccfb0 msgr2=0x7fa7200ae320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7200ccfb0 0x7fa7200ae320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 -- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7200ad970 msgr2=0x7fa7200adde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7200ad970 0x7fa7200adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 -- 172.21.15.29:0/625822660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa708009a70 con 0x7fa72006c040 2023-10-14T17:01:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa71e59c640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b3f0 tx=0x7fa71000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.443+0000 7fa7177fe640 1 -- 172.21.15.29:0/625822660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710011830 con 0x7fa72006c040 2023-10-14T17:01:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.444+0000 7fa7177fe640 1 -- 172.21.15.29:0/625822660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710011990 con 0x7fa72006c040 2023-10-14T17:01:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.444+0000 7fa7177fe640 1 -- 172.21.15.29:0/625822660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710011d40 con 0x7fa72006c040 2023-10-14T17:01:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.444+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7200af9e0 con 0x7fa72006c040 2023-10-14T17:01:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.444+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7200aff40 con 0x7fa72006c040 2023-10-14T17:01:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.446+0000 7fa7177fe640 1 -- 172.21.15.29:0/625822660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa71001a400 con 0x7fa72006c040 2023-10-14T17:01:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.447+0000 7fa7177fe640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 0x7fa70006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.447+0000 7fa7177fe640 1 -- 172.21.15.29:0/625822660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa710089690 con 0x7fa72006c040 2023-10-14T17:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.447+0000 7fa715d9b640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 0x7fa70006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.448+0000 7fa72766e740 1 Processor -- start 2023-10-14T17:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.448+0000 7fa715d9b640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 0x7fa70006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa708036000 tx=0x7fa708030000 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.448+0000 7fa72766e740 1 -- start start 2023-10-14T17:01:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 0x55b80e897f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e8685b0 0x55b80e898440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e7887a0 con 0x55b80e895170 2023-10-14T17:01:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e7da740 con 0x55b80e895550 2023-10-14T17:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b80e7d7e30 con 0x55b80e8685b0 2023-10-14T17:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37914/0 (socket says 172.21.15.29:37914) 2023-10-14T17:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa71e59c640 1 -- 172.21.15.29:0/736371277 learned_addr learned my addr 172.21.15.29:0/736371277 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa71e59c640 1 -- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e8685b0 msgr2=0x55b80e898440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.449+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e8685b0 0x55b80e898440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa715d9b640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 0x55b80e897f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa71e59c640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e8685b0 0x55b80e898440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa71e59c640 1 -- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 msgr2=0x55b80e897f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa71e59c640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 0x55b80e897f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa71e59c640 1 -- 172.21.15.29:0/736371277 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b80e797910 con 0x55b80e895170 2023-10-14T17:01:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa715d9b640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 0x55b80e897f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa71e59c640 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa710057f00 tx=0x7fa710057c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5d7b6fd3302099e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa6fffff640 1 -- 172.21.15.29:0/736371277 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa71005c170 con 0x55b80e895170 2023-10-14T17:01:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa6fffff640 1 -- 172.21.15.29:0/736371277 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa710060500 con 0x55b80e895170 2023-10-14T17:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.450+0000 7fa6fffff640 1 -- 172.21.15.29:0/736371277 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710054910 con 0x55b80e895170 2023-10-14T17:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 -- 172.21.15.29:0/736371277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 msgr2=0x55b80e868070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 secure :-1 s=READY pgs=1809 cs=0 l=1 rev1=1 crypto rx=0x7fa710057f00 tx=0x7fa710057c70 comp rx=0 tx=0).stop 2023-10-14T17:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 -- 172.21.15.29:0/736371277 shutdown_connections 2023-10-14T17:01:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e8685b0 0x55b80e898440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895550 0x55b80e897f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 --2- 172.21.15.29:0/736371277 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e868070 unknown :-1 s=CLOSED pgs=1809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 -- 172.21.15.29:0/736371277 >> 172.21.15.29:0/736371277 conn(0x55b80e86a380 msgr2=0x55b80e8b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 -- 172.21.15.29:0/736371277 shutdown_connections 2023-10-14T17:01:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.451+0000 7fa72766e740 1 -- 172.21.15.29:0/736371277 wait complete. 2023-10-14T17:01:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.452+0000 7fa72766e740 1 -- start start 2023-10-14T17:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.452+0000 7fa72766e740 1 Processor -- start 2023-10-14T17:01:21.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895170 0x55b80e8b1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e89a480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e7d3720 con 0x55b80e8b3be0 2023-10-14T17:01:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e771970 con 0x55b80e895170 2023-10-14T17:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b80e771610 con 0x55b80e895550 2023-10-14T17:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa72766e740 10 client.0 _ll_register_callbacks cb 0x55b80e7939d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa71ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa71ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37928/0 (socket says 172.21.15.29:37928) 2023-10-14T17:01:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.453+0000 7fa71ed9d640 1 -- 172.21.15.29:0/1632855955 learned_addr learned my addr 172.21.15.29:0/1632855955 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71e59c640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895170 0x55b80e8b1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa715d9b640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e89a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 -- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 msgr2=0x55b80e89a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e89a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 -- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895170 msgr2=0x55b80e8b1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895170 0x55b80e8b1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 -- 172.21.15.29:0/1632855955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7200aff40 con 0x55b80e8b3be0 2023-10-14T17:01:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa718001d90 tx=0x7fa71800d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=586260aa63e8920b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa6fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:01:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.454+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7180072d0 con 0x55b80e8b3be0 2023-10-14T17:01:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa718007430 con 0x55b80e8b3be0 2023-10-14T17:01:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b80e7fa160 con 0x55b80e8b3be0 2023-10-14T17:01:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa718007e00 con 0x55b80e8b3be0 2023-10-14T17:01:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 10 client.9225 fetch_fsmap learned FSMap version 74 2023-10-14T17:01:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55b80e7d1da0 con 0x55b80e8b3be0 2023-10-14T17:01:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b80e7fa160 con 0x55b80e8b3be0 2023-10-14T17:01:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa7180047e0 con 0x55b80e8b3be0 2023-10-14T17:01:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 10 client.9225 fetch_fsmap finished waiting for FSMap version 74 2023-10-14T17:01:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 10 client.9225 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T17:01:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 20 client.9225 populate_metadata read hostname 'smithi029' 2023-10-14T17:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8adde0 0x55b80e8c62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 4 client.9225 mds_command: new command op to 9105 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T17:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.455+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 --> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b80e7a5cc0 con 0x55b80e8adde0 2023-10-14T17:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.456+0000 7fa71e59c640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8adde0 0x55b80e8c62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.456+0000 7fa71e59c640 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8adde0 0x55b80e8c62b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d72df386b6e9e73c server_cookie=62157f65da6970b1 in_seq=0 out_seq=0 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.456+0000 7fa6fd7fa640 10 client.9225 ms_handle_connect on v2:172.21.15.29:6836/1190525894 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/1632855955 <== mds.0 v2:172.21.15.29:6836/1190525894 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b80e7a5cc0 con 0x55b80e8adde0 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa6fd7fa640 10 client.9225 handle_command_reply: tid=0 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 client.9225 shutdown 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 msgr2=0x55b80e89a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 secure :-1 s=READY pgs=1810 cs=0 l=1 rev1=1 crypto rx=0x7fa718001d90 tx=0x7fa71800d520 comp rx=0 tx=0).stop 2023-10-14T17:01:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 shutdown_connections 2023-10-14T17:01:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8adde0 0x55b80e8c62b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e8b3be0 0x55b80e89a9c0 unknown :-1 s=CLOSED pgs=1810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e89a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 --2- 172.21.15.29:0/1632855955 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e895170 0x55b80e8b1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa6fd7fa640 0 client.9225 ms_handle_reset on v2:172.21.15.29:6836/1190525894 2023-10-14T17:01:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 >> 172.21.15.29:0/1632855955 conn(0x55b80e8685b0 msgr2=0x55b80e897ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 shutdown_connections 2023-10-14T17:01:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.457+0000 7fa72766e740 1 -- 172.21.15.29:0/1632855955 wait complete. 2023-10-14T17:01:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.458+0000 7fa72766e740 20 client.9225 trim_cache size 0 max 16384 2023-10-14T17:01:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.474+0000 7fa72766e740 1 Processor -- start 2023-10-14T17:01:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.475+0000 7fa72766e740 1 -- start start 2023-10-14T17:01:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.475+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895170 0x55b80e86be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.475+0000 7fa72766e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.475+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e8cc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa72766e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e537f20 con 0x55b80e895550 2023-10-14T17:01:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e86b750 con 0x55b80e8b3be0 2023-10-14T17:01:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b80e869de0 con 0x55b80e895170 2023-10-14T17:01:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37932/0 (socket says 172.21.15.29:37932) 2023-10-14T17:01:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 -- 172.21.15.29:0/853365270 learned_addr learned my addr 172.21.15.29:0/853365270 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e8cc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 -- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895170 msgr2=0x55b80e86be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895170 0x55b80e86be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 -- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 msgr2=0x55b80e8cc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e8cc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa715d9b640 1 -- 172.21.15.29:0/853365270 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b80e770b80 con 0x55b80e895550 2023-10-14T17:01:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.476+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e8cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.477+0000 7fa715d9b640 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa70802cd70 tx=0x7fa70802e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6e5a9c0444309e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.477+0000 7fa714d99640 1 -- 172.21.15.29:0/853365270 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa708034d20 con 0x55b80e895550 2023-10-14T17:01:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.477+0000 7fa714d99640 1 -- 172.21.15.29:0/853365270 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa708034e80 con 0x55b80e895550 2023-10-14T17:01:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.477+0000 7fa714d99640 1 -- 172.21.15.29:0/853365270 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa70804d6c0 con 0x55b80e895550 2023-10-14T17:01:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 -- 172.21.15.29:0/853365270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 msgr2=0x55b80e8cc200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 secure :-1 s=READY pgs=1811 cs=0 l=1 rev1=1 crypto rx=0x7fa70802cd70 tx=0x7fa70802e000 comp rx=0 tx=0).stop 2023-10-14T17:01:21.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 -- 172.21.15.29:0/853365270 shutdown_connections 2023-10-14T17:01:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e8cc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895550 0x55b80e8cc200 unknown :-1 s=CLOSED pgs=1811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 --2- 172.21.15.29:0/853365270 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895170 0x55b80e86be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 -- 172.21.15.29:0/853365270 >> 172.21.15.29:0/853365270 conn(0x55b80e8685b0 msgr2=0x55b80e863810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 -- 172.21.15.29:0/853365270 shutdown_connections 2023-10-14T17:01:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.478+0000 7fa72766e740 1 -- 172.21.15.29:0/853365270 wait complete. 2023-10-14T17:01:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.479+0000 7fa72766e740 1 -- start start 2023-10-14T17:01:21.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.479+0000 7fa72766e740 1 Processor -- start 2023-10-14T17:01:21.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e8d45c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e86d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e8b57a0 con 0x55b80e895170 2023-10-14T17:01:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b80e8d4fa0 con 0x55b80e8b3be0 2023-10-14T17:01:21.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b80e8980f0 con 0x55b80e895550 2023-10-14T17:01:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa72766e740 10 client.0 _ll_register_callbacks cb 0x7fa718007bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:01:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa71e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37942/0 (socket says 172.21.15.29:37942) 2023-10-14T17:01:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.480+0000 7fa71e59c640 1 -- 172.21.15.29:0/2646447830 learned_addr learned my addr 172.21.15.29:0/2646447830 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71ed9d640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e86d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa715d9b640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e8d45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 -- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 msgr2=0x55b80e8d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e8d45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 -- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 msgr2=0x55b80e86d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e86d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 -- 172.21.15.29:0/2646447830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa708048040 con 0x55b80e895170 2023-10-14T17:01:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa71e59c640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fa710054700 tx=0x7fa71000e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1b1109378ad0898 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa6fd7fa640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:01:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa710011ea0 con 0x55b80e895170 2023-10-14T17:01:21.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fa710087ac0 con 0x55b80e895170 2023-10-14T17:01:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.481+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b80e7cc700 con 0x55b80e895170 2023-10-14T17:01:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=74) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa710087c80 con 0x55b80e895170 2023-10-14T17:01:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 10 client.9231 fetch_fsmap learned FSMap version 74 2023-10-14T17:01:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=74,monmap=2+}) v3 -- 0x55b80e796170 con 0x55b80e895170 2023-10-14T17:01:21.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b80e7cc700 con 0x55b80e895170 2023-10-14T17:01:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 74) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa71006fd20 con 0x55b80e895170 2023-10-14T17:01:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 10 client.9231 fetch_fsmap finished waiting for FSMap version 74 2023-10-14T17:01:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 10 client.9231 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T17:01:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 20 client.9231 populate_metadata read hostname 'smithi029' 2023-10-14T17:01:21.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8d27f0 0x55b80e86d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 4 client.9231 mds_command: new command op to 9105 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T17:01:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 --> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b80e7cc420 con 0x55b80e8d27f0 2023-10-14T17:01:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.482+0000 7fa715d9b640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8d27f0 0x55b80e86d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa715d9b640 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8d27f0 0x55b80e86d6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dff4542943b506b5 server_cookie=3347bde488e98973 in_seq=0 out_seq=0 2023-10-14T17:01:21.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa6fd7fa640 10 client.9231 ms_handle_connect on v2:172.21.15.29:6836/1190525894 2023-10-14T17:01:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa6fd7fa640 1 -- 172.21.15.29:0/2646447830 <== mds.0 v2:172.21.15.29:6836/1190525894 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+333 (crc 0 0 0) 0x55b80e7cc420 con 0x55b80e8d27f0 2023-10-14T17:01:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa6fd7fa640 10 client.9231 handle_command_reply: tid=0 2023-10-14T17:01:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa72766e740 1 client.9231 shutdown 2023-10-14T17:01:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 msgr2=0x55b80e8d4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.483+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 secure :-1 s=READY pgs=1812 cs=0 l=1 rev1=1 crypto rx=0x7fa710054700 tx=0x7fa71000e000 comp rx=0 tx=0).stop 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 shutdown_connections 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b80e8b3be0 0x55b80e86d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b80e895550 0x55b80e8d45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:6836/1190525894,v1:172.21.15.29:6837/1190525894] conn(0x55b80e8d27f0 0x55b80e86d6f0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 --2- 172.21.15.29:0/2646447830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b80e895170 0x55b80e8d4080 unknown :-1 s=CLOSED pgs=1812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 >> 172.21.15.29:0/2646447830 conn(0x55b80e8685b0 msgr2=0x55b80e863810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa6fd7fa640 0 client.9231 ms_handle_reset on v2:172.21.15.29:6836/1190525894 2023-10-14T17:01:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 shutdown_connections 2023-10-14T17:01:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 1 -- 172.21.15.29:0/2646447830 wait complete. 2023-10-14T17:01:21.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.484+0000 7fa72766e740 20 client.9231 trim_cache size 0 max 16384 2023-10-14T17:01:21.498 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:01:21.498 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T17:01:21.498 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T17:01:21.498 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9105, 2023-10-14T17:01:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T17:01:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T17:01:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T17:01:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.7887084140000002, 2023-10-14T17:01:21.499 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 74, 2023-10-14T17:01:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 154, 2023-10-14T17:01:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 154, 2023-10-14T17:01:21.500 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.139202596000001 2023-10-14T17:01:21.500 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:01:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 msgr2=0x7fa70006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 0x7fa70006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa708036000 tx=0x7fa708030000 comp rx=0 tx=0).stop 2023-10-14T17:01:21.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 msgr2=0x7fa72015cc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 secure :-1 s=READY pgs=1808 cs=0 l=1 rev1=1 crypto rx=0x7fa71000b3f0 tx=0x7fa71000b8b0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 shutdown_connections 2023-10-14T17:01:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fa70006af90 0x7fa70006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7200ccfb0 0x7fa7200ae320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7200ad970 0x7fa7200adde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 --2- 172.21.15.29:0/625822660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa72006c040 0x7fa72015cc80 unknown :-1 s=CLOSED pgs=1808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 >> 172.21.15.29:0/625822660 conn(0x7fa7200ce6a0 msgr2=0x7fa72006c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 shutdown_connections 2023-10-14T17:01:21.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.485+0000 7fa71f59e640 1 -- 172.21.15.29:0/625822660 wait complete. 2023-10-14T17:01:21.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T17:01:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.727+0000 7f5397b61640 1 Processor -- start 2023-10-14T17:01:21.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.727+0000 7f5397b61640 1 -- start start 2023-10-14T17:01:21.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f5397b61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f539015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f5397b61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539015ed60 con 0x7f539015aaa0 2023-10-14T17:01:21.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f5397b61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53900707e0 con 0x7f539015ae80 2023-10-14T17:01:21.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f5397b61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539006b810 con 0x7f53900ce390 2023-10-14T17:01:21.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f53958fd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f539015aaa0 0x7f53900cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41252/0 (socket says 172.21.15.29:41252) 2023-10-14T17:01:21.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.729+0000 7f53958fd640 1 -- 172.21.15.29:0/3843818030 learned_addr learned my addr 172.21.15.29:0/3843818030 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53960fe640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f539015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2260257030 0 0) 0x7f539015ed60 con 0x7f539015aaa0 2023-10-14T17:01:21.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f539015aaa0 2023-10-14T17:01:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1321680970 0 0) 0x7f537c003520 con 0x7f539015aaa0 2023-10-14T17:01:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 msgr2=0x7f539015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f539015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 >> v1:172.21.15.186:6789/0 conn(0x7f539015ae80 legacy=0x7f5390070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539015ef90 con 0x7f539015aaa0 2023-10-14T17:01:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f538c001690 con 0x7f539015aaa0 2023-10-14T17:01:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 >> v1:172.21.15.29:6789/0 conn(0x7f539015aaa0 legacy=0x7f53900cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f537c0039a0 0x7f537c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f537c0047c0 0x7f537c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.730+0000 7f53948fb640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f537c005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539006b810 con 0x7f537c0047c0 2023-10-14T17:01:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f537c00bb10 con 0x7f537c0039a0 2023-10-14T17:01:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f537c00bcd0 con 0x7f53900ce390 2023-10-14T17:01:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f537c0047c0 0x7f537c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53958fd640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f537c0039a0 0x7f537c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 -- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 msgr2=0x7f537c005210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53960fe640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f537c005210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53900ce390 0x7f537c005210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 -- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f537c0039a0 msgr2=0x7f537c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f537c0039a0 0x7f537c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.731+0000 7f53950fc640 1 -- 172.21.15.29:0/3843818030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f537c003520 con 0x7f537c0047c0 2023-10-14T17:01:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.732+0000 7f53958fd640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f537c0039a0 0x7f537c003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:21.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f53950fc640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f537c0047c0 0x7f537c004c40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5380001720 tx=0x7f537c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795e5a2ddd6d6009 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800052d0 con 0x7f537c0047c0 2023-10-14T17:01:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538002e050 con 0x7f537c0047c0 2023-10-14T17:01:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f53948fb640 1 -- 172.21.15.29:0/3843818030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53800035f0 con 0x7f537c0047c0 2023-10-14T17:01:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f5397b61640 1 -- 172.21.15.29:0/3843818030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f537c0047c0 msgr2=0x7f537c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.733+0000 7f5397b61640 1 --2- 172.21.15.29:0/3843818030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f537c0047c0 0x7f537c004c40 secure :-1 s=READY pgs=1814 cs=0 l=1 rev1=1 crypto rx=0x7f5380001720 tx=0x7f537c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.734+0000 7f53960fe640 1 -- 172.21.15.29:0/3843818030 reap_dead start 2023-10-14T17:01:21.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.734+0000 7f5397b61640 1 -- 172.21.15.29:0/3843818030 shutdown_connections 2023-10-14T17:01:21.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.734+0000 7f5397b61640 1 -- 172.21.15.29:0/3843818030 >> 172.21.15.29:0/3843818030 conn(0x7f53900cb410 msgr2=0x7f53900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.734+0000 7f5397b61640 1 -- 172.21.15.29:0/3843818030 shutdown_connections 2023-10-14T17:01:21.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.734+0000 7f5397b61640 1 -- 172.21.15.29:0/3843818030 wait complete. 2023-10-14T17:01:21.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.735+0000 7f5397b61640 1 Processor -- start 2023-10-14T17:01:21.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.736+0000 7f5397b61640 1 -- start start 2023-10-14T17:01:21.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f537c0047c0 0x7f5390153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 0x7f539015c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5390154410 con 0x7f53900c2e30 2023-10-14T17:01:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539015c7f0 con 0x7f539015bce0 2023-10-14T17:01:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f5397b61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539015cad0 con 0x7f537c0047c0 2023-10-14T17:01:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37954/0 (socket says 172.21.15.29:37954) 2023-10-14T17:01:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 -- 172.21.15.29:0/3758981000 learned_addr learned my addr 172.21.15.29:0/3758981000 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53958fd640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f537c0047c0 0x7f5390153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53950fc640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 0x7f539015c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 -- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f537c0047c0 msgr2=0x7f5390153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f537c0047c0 0x7f5390153ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 -- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 msgr2=0x7f539015c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 0x7f539015c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53960fe640 1 -- 172.21.15.29:0/3758981000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539015cdb0 con 0x7f53900c2e30 2023-10-14T17:01:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.737+0000 7f53950fc640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 0x7f539015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:21.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f53960fe640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f53900c5990 tx=0x7f5388009d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388007cf0 con 0x7f53900c2e30 2023-10-14T17:01:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5388007e70 con 0x7f53900c2e30 2023-10-14T17:01:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5388014620 con 0x7f53900c2e30 2023-10-14T17:01:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53900c2910 con 0x7f53900c2e30 2023-10-14T17:01:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.738+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53900c2570 con 0x7f53900c2e30 2023-10-14T17:01:21.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.740+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539004e160 con 0x7f53900c2e30 2023-10-14T17:01:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.740+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f538801d3b0 con 0x7f53900c2e30 2023-10-14T17:01:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.741+0000 7f53867fc640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.742+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7f5388089590 con 0x7f53900c2e30 2023-10-14T17:01:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.742+0000 7f53958fd640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.743+0000 7f53958fd640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 0x7f537406d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f538c001eb0 tx=0x7f538c009040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:21.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.746+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5388054440 con 0x7f53900c2e30 2023-10-14T17:01:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.984+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f539015e350 con 0x7f53900c2e30 2023-10-14T17:01:21.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.985+0000 7f53867fc640 1 -- 172.21.15.29:0/3758981000 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v74) v1 ==== 53+0+73 (secure 0 0 0) 0x7f5388057a60 con 0x7f53900c2e30 2023-10-14T17:01:21.986 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T17:01:21.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 msgr2=0x7f537406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 0x7f537406d480 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f538c001eb0 tx=0x7f538c009040 comp rx=0 tx=0).stop 2023-10-14T17:01:21.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 msgr2=0x7f53900c32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:21.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 secure :-1 s=READY pgs=1815 cs=0 l=1 rev1=1 crypto rx=0x7f53900c5990 tx=0x7f5388009d50 comp rx=0 tx=0).stop 2023-10-14T17:01:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 shutdown_connections 2023-10-14T17:01:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7f537406afc0 0x7f537406d480 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53900c2e30 0x7f53900c32b0 unknown :-1 s=CLOSED pgs=1815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f539015bce0 0x7f539015c140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 --2- 172.21.15.29:0/3758981000 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f537c0047c0 0x7f5390153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 >> 172.21.15.29:0/3758981000 conn(0x7f53900cb410 msgr2=0x7f53900b2940 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 shutdown_connections 2023-10-14T17:01:21.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:21.988+0000 7f5397b61640 1 -- 172.21.15.29:0/3758981000 wait complete. 2023-10-14T17:01:22.006 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T17:01:22.006 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T17:01:22.007 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T17:01:22.007 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T17:01:22.007 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T17:01:22.008 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T17:01:22.008 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T17:01:22.009 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T17:01:22.009 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:01:22.009 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.010 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T17:01:22.066 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T17:01:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T17:01:22.066 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T17:01:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T17:01:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.067 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:01:22.067 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:01:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T17:01:22.068 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:01:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:01:22.068 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T17:01:22.068 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T17:01:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T17:01:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T17:01:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T17:01:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 41094sec preferred_lft 41094sec 2023-10-14T17:01:22.069 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T17:01:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.070 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:01:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:01:22.070 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T17:01:22.070 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T17:01:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:01:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T17:01:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T17:01:22.071 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.072 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:01:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T17:01:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T17:01:22.072 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.072 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:01:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T17:01:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T17:01:22.073 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:01:22.074 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.074 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T17:01:22.176 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T17:01:22.180 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.180 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T17:01:22.242 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T17:01:22.243 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T17:01:22.243 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T17:01:22.243 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T17:01:22.243 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T17:01:22.244 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.244 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T17:01:22.244 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T17:01:22.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T17:01:22.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T17:01:22.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T17:01:22.246 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T17:01:22.453 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.454 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T17:01:22.509 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T17:01:22.510 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T17:01:22.510 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:01:22.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.511 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T17:01:22.613 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:01:22.613 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T17:01:22.675 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:01:22.675 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:01:22.676 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:01:22.735 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T17:01:22.736 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:01:22.879 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:01:22.879 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:01:22.879 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:01:22.902 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:01:22.902 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:01:22.902 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:01:22.902 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:01:22.903 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:01:22.931 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:01:22.931 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:01:22.931 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:01:22.932 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:01:22.932 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:01:22.932 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T17:01:22.932 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T17:01:22.932 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T17:01:22.935 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T17:01:22.935 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T17:01:22.936 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T17:01:22.936 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T17:01:22.936 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T17:01:22.940 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T17:01:22.940 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:22.941 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:01:23.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:23.042 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T17:01:23.107 INFO:teuthology.orchestra.run.smithi029.stdout:client7126 2023-10-14T17:01:23.107 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:01:23.108 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:01:23.157 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T17:01:23.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.430+0000 7fe0b6644640 1 Processor -- start 2023-10-14T17:01:23.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.430+0000 7fe0b6644640 1 -- start start 2023-10-14T17:01:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0b6644640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe0b00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0b6644640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b0070360 con 0x7fe0b0159e90 2023-10-14T17:01:23.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0b6644640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0b00cd1f0 con 0x7fe0b0159ab0 2023-10-14T17:01:23.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0b6644640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015d890 con 0x7fe0b006c850 2023-10-14T17:01:23.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0a77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0b0159e90 0x7fe0b006c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41260/0 (socket says 172.21.15.29:41260) 2023-10-14T17:01:23.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.432+0000 7fe0a77fe640 1 -- 172.21.15.29:0/1247274087 learned_addr learned my addr 172.21.15.29:0/1247274087 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0b4be1640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe0b00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205887127 0 0) 0x7fe0b0070360 con 0x7fe0b0159e90 2023-10-14T17:01:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c005730 con 0x7fe0b0159e90 2023-10-14T17:01:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3767214969 0 0) 0x7fe09c005730 con 0x7fe0b0159e90 2023-10-14T17:01:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 msgr2=0x7fe0b00ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe0b00ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 >> v1:172.21.15.186:6789/0 conn(0x7fe0b0159ab0 legacy=0x7fe0b006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00cd4d0 con 0x7fe0b0159e90 2023-10-14T17:01:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe098001670 con 0x7fe0b0159e90 2023-10-14T17:01:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 >> v1:172.21.15.29:6789/0 conn(0x7fe0b0159e90 legacy=0x7fe0b006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe09c005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe09c0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe09c006f10 0x7fe09c007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.433+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b015d890 con 0x7fe09c0060f0 2023-10-14T17:01:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c010040 con 0x7fe09c006f10 2023-10-14T17:01:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09c0101d0 con 0x7fe0b006c850 2023-10-14T17:01:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe09c0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0affff640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe09c005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 -- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 msgr2=0x7fe09c005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b006c850 0x7fe09c005bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 -- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe09c006f10 msgr2=0x7fe09c007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe09c006f10 0x7fe09c007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.434+0000 7fe0a77fe640 1 -- 172.21.15.29:0/1247274087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c005730 con 0x7fe09c0060f0 2023-10-14T17:01:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.435+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe09c0069d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c0e0 tx=0x7fe09c007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56e49fa71bc9fb00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.435+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe098005ad0 con 0x7fe09c0060f0 2023-10-14T17:01:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe098005c30 con 0x7fe09c0060f0 2023-10-14T17:01:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0af7fe640 1 -- 172.21.15.29:0/1247274087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe09802ca50 con 0x7fe09c0060f0 2023-10-14T17:01:23.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 -- 172.21.15.29:0/1247274087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 msgr2=0x7fe09c0069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 --2- 172.21.15.29:0/1247274087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe09c0069d0 secure :-1 s=READY pgs=1817 cs=0 l=1 rev1=1 crypto rx=0x7fe09800c0e0 tx=0x7fe09c007a00 comp rx=0 tx=0).stop 2023-10-14T17:01:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b4be1640 1 -- 172.21.15.29:0/1247274087 reap_dead start 2023-10-14T17:01:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 -- 172.21.15.29:0/1247274087 shutdown_connections 2023-10-14T17:01:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 -- 172.21.15.29:0/1247274087 >> 172.21.15.29:0/1247274087 conn(0x7fe0b00ce9b0 msgr2=0x7fe0b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:23.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 -- 172.21.15.29:0/1247274087 shutdown_connections 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.436+0000 7fe0b6644640 1 -- 172.21.15.29:0/1247274087 wait complete. 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.438+0000 7fe0b6644640 1 Processor -- start 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.438+0000 7fe0b6644640 1 -- start start 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b0156ca0 0x7fe0b00c2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0b00c42c0 0x7fe0b00c4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00c3f80 con 0x7fe09c0060f0 2023-10-14T17:01:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00c4c80 con 0x7fe0b00c42c0 2023-10-14T17:01:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0b6644640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0b00c36e0 con 0x7fe0b0156ca0 2023-10-14T17:01:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37972/0 (socket says 172.21.15.29:37972) 2023-10-14T17:01:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.439+0000 7fe0affff640 1 -- 172.21.15.29:0/61147225 learned_addr learned my addr 172.21.15.29:0/61147225 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b0156ca0 0x7fe0b00c2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 -- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b0156ca0 msgr2=0x7fe0b00c2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0b4be1640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0b00c42c0 0x7fe0b00c4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b0156ca0 0x7fe0b00c2ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 -- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0b00c42c0 msgr2=0x7fe0b00c4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0b00c42c0 0x7fe0b00c4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 -- 172.21.15.29:0/61147225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0b00c39c0 con 0x7fe09c0060f0 2023-10-14T17:01:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.440+0000 7fe0affff640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000d8e0 tx=0x7fe0a000dda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.441+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a0003da0 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.441+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a000b6b0 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.441+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0a000ba60 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.441+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0b00c3ca0 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.441+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0b00c33e0 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.442+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0b004e160 con 0x7fe09c0060f0 2023-10-14T17:01:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.443+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe0a000bbc0 con 0x7fe09c0060f0 2023-10-14T17:01:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.444+0000 7fe0ad7fa640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 0x7fe09006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.444+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fe0a0089d00 con 0x7fe09c0060f0 2023-10-14T17:01:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.444+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 0x7fe09006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.445+0000 7fe0a77fe640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 0x7fe09006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe098034000 tx=0x7fe098009030 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.448+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0a0054aa0 con 0x7fe09c0060f0 2023-10-14T17:01:23.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.668+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 --> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0b015a5f0 con 0x7fe09006af90 2023-10-14T17:01:23.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.671+0000 7fe0ad7fa640 1 -- 172.21.15.29:0/61147225 <== mgr.8862 v2:172.21.15.29:6832/3121942650 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe0b015a5f0 con 0x7fe09006af90 2023-10-14T17:01:23.671 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:01:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:01:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T17:01:23.672 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:01:23.672 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:01:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.673+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 msgr2=0x7fe09006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.673+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 0x7fe09006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fe098034000 tx=0x7fe098009030 comp rx=0 tx=0).stop 2023-10-14T17:01:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.673+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 msgr2=0x7fe0b00c1c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.673+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 secure :-1 s=READY pgs=1818 cs=0 l=1 rev1=1 crypto rx=0x7fe0a000d8e0 tx=0x7fe0a000dda0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 shutdown_connections 2023-10-14T17:01:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fe09006af90 0x7fe09006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0b00c42c0 0x7fe0b00c4740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0b0156ca0 0x7fe0b00c2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 --2- 172.21.15.29:0/61147225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe09c0060f0 0x7fe0b00c1c10 unknown :-1 s=CLOSED pgs=1818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 >> 172.21.15.29:0/61147225 conn(0x7fe0b00ce9b0 msgr2=0x7fe0b0158be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:23.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 shutdown_connections 2023-10-14T17:01:23.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.674+0000 7fe0b6644640 1 -- 172.21.15.29:0/61147225 wait complete. 2023-10-14T17:01:23.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T17:01:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.926+0000 7fd2956c9640 1 Processor -- start 2023-10-14T17:01:23.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.926+0000 7fd2956c9640 1 -- start start 2023-10-14T17:01:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd29015ad70 0x7fd290069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2900c9ca0 0x7fd29006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2900ca3b0 0x7fd2900ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2900cad80 con 0x7fd29015ad70 2023-10-14T17:01:23.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2900cb030 con 0x7fd2900c9ca0 2023-10-14T17:01:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd2956c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c91c0 con 0x7fd2900ca3b0 2023-10-14T17:01:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd28effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd29015ad70 0x7fd290069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd28effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd29015ad70 0x7fd290069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37986/0 (socket says 172.21.15.29:37986) 2023-10-14T17:01:23.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.928+0000 7fd28effd640 1 -- 172.21.15.29:0/3977983112 learned_addr learned my addr 172.21.15.29:0/3977983112 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28effd640 1 -- 172.21.15.29:0/3977983112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2900ca3b0 msgr2=0x7fd2900ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28f7fe640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2900ca3b0 0x7fd2900ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd287fff640 1 --2- 172.21.15.29:0/3977983112 >> v2:172.21.15.186:3300/0 conn(0x7fd2900c9ca0 0x7fd29006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28effd640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2900ca3b0 0x7fd2900ca840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28effd640 1 -- 172.21.15.29:0/3977983112 >> v2:172.21.15.186:3300/0 conn(0x7fd2900c9ca0 msgr2=0x7fd29006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28effd640 1 --2- 172.21.15.29:0/3977983112 >> v2:172.21.15.186:3300/0 conn(0x7fd2900c9ca0 0x7fd29006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.929+0000 7fd28effd640 1 -- 172.21.15.29:0/3977983112 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900c94a0 con 0x7fd29015ad70 2023-10-14T17:01:23.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.930+0000 7fd28effd640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd29015ad70 0x7fd290069da0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd2900ce870 tx=0x7fd28002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adb80fa3e73830ca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.930+0000 7fd28e7fc640 1 -- 172.21.15.29:0/3977983112 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280031870 con 0x7fd29015ad70 2023-10-14T17:01:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.930+0000 7fd28e7fc640 1 -- 172.21.15.29:0/3977983112 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2800319d0 con 0x7fd29015ad70 2023-10-14T17:01:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.930+0000 7fd28e7fc640 1 -- 172.21.15.29:0/3977983112 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd280031d60 con 0x7fd29015ad70 2023-10-14T17:01:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 -- 172.21.15.29:0/3977983112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd29015ad70 msgr2=0x7fd290069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd29015ad70 0x7fd290069da0 secure :-1 s=READY pgs=1819 cs=0 l=1 rev1=1 crypto rx=0x7fd2900ce870 tx=0x7fd28002b910 comp rx=0 tx=0).stop 2023-10-14T17:01:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 -- 172.21.15.29:0/3977983112 shutdown_connections 2023-10-14T17:01:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2900ca3b0 0x7fd2900ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 --2- 172.21.15.29:0/3977983112 >> v2:172.21.15.186:3300/0 conn(0x7fd2900c9ca0 0x7fd29006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 --2- 172.21.15.29:0/3977983112 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd29015ad70 0x7fd290069da0 unknown :-1 s=CLOSED pgs=1819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 -- 172.21.15.29:0/3977983112 >> 172.21.15.29:0/3977983112 conn(0x7fd2900ce9b0 msgr2=0x7fd2900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 -- 172.21.15.29:0/3977983112 shutdown_connections 2023-10-14T17:01:23.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.931+0000 7fd2956c9640 1 -- 172.21.15.29:0/3977983112 wait complete. 2023-10-14T17:01:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.933+0000 7fd2956c9640 1 Processor -- start 2023-10-14T17:01:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.933+0000 7fd2956c9640 1 -- start start 2023-10-14T17:01:23.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2900c9ca0 0x7fd290159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd29015ad70 0x7fd2900c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c6380 con 0x7fd2900ca3b0 2023-10-14T17:01:23.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c6660 con 0x7fd2900c9ca0 2023-10-14T17:01:23.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd2956c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900c6940 con 0x7fd29015ad70 2023-10-14T17:01:23.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd287fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd287fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:37998/0 (socket says 172.21.15.29:37998) 2023-10-14T17:01:23.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.934+0000 7fd287fff640 1 -- 172.21.15.29:0/2065658268 learned_addr learned my addr 172.21.15.29:0/2065658268 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:23.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 -- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd29015ad70 msgr2=0x7fd2900c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd28f7fe640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd29015ad70 0x7fd2900c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd28effd640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2900c9ca0 0x7fd290159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd29015ad70 0x7fd2900c7420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 -- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2900c9ca0 msgr2=0x7fd290159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:23.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2900c9ca0 0x7fd290159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:23.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 -- 172.21.15.29:0/2065658268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2800094d0 con 0x7fd2900ca3b0 2023-10-14T17:01:23.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd287fff640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd27800c7f0 tx=0x7fd27800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278004220 con 0x7fd2900ca3b0 2023-10-14T17:01:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd278004380 con 0x7fd2900ca3b0 2023-10-14T17:01:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd278007860 con 0x7fd2900ca3b0 2023-10-14T17:01:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.935+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2900c6c20 con 0x7fd2900ca3b0 2023-10-14T17:01:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.936+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29015a4d0 con 0x7fd2900ca3b0 2023-10-14T17:01:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.937+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2900c7140 con 0x7fd2900ca3b0 2023-10-14T17:01:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.938+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 80) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd278007bd0 con 0x7fd2900ca3b0 2023-10-14T17:01:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.939+0000 7fd2877fe640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 0x7fd27006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.939+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fd278089890 con 0x7fd2900ca3b0 2023-10-14T17:01:23.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.939+0000 7fd28effd640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 0x7fd27006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:23.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.940+0000 7fd28effd640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 0x7fd27006d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd280036000 tx=0x7fd28003c040 comp rx=0 tx=0).ready entity=mgr.8862 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:23.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:23.943+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2900c7140 con 0x7fd2900ca3b0 2023-10-14T17:01:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.190+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fd2900c7140 con 0x7fd2900ca3b0 2023-10-14T17:01:24.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.856+0000 7fd2877fe640 1 -- 172.21.15.29:0/2065658268 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v81) v1 ==== 92+0+0 (secure 0 0 0) 0x7fd2900c7140 con 0x7fd2900ca3b0 2023-10-14T17:01:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.859+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 msgr2=0x7fd27006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.859+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 0x7fd27006d4a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd280036000 tx=0x7fd28003c040 comp rx=0 tx=0).stop 2023-10-14T17:01:24.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.859+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 msgr2=0x7fd2900c5a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.859+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 secure :-1 s=READY pgs=1820 cs=0 l=1 rev1=1 crypto rx=0x7fd27800c7f0 tx=0x7fd27800ccb0 comp rx=0 tx=0).stop 2023-10-14T17:01:24.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 shutdown_connections 2023-10-14T17:01:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fd27006afe0 0x7fd27006d4a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd29015ad70 0x7fd2900c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2900ca3b0 0x7fd2900c5a80 unknown :-1 s=CLOSED pgs=1820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 --2- 172.21.15.29:0/2065658268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2900c9ca0 0x7fd290159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:24.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 >> 172.21.15.29:0/2065658268 conn(0x7fd2900ce9b0 msgr2=0x7fd2900b3b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 shutdown_connections 2023-10-14T17:01:24.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:24.860+0000 7fd2956c9640 1 -- 172.21.15.29:0/2065658268 wait complete. 2023-10-14T17:01:24.893 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T17:01:25.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.099+0000 7fb279ed1640 1 Processor -- start 2023-10-14T17:01:25.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.099+0000 7fb279ed1640 1 -- start start 2023-10-14T17:01:25.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb279ed1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb27415aa60 0x7fb2740af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb279ed1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740b3730 con 0x7fb2740afd60 2023-10-14T17:01:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb279ed1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2740b3980 con 0x7fb2740cc9f0 2023-10-14T17:01:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb279ed1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2740b3c60 con 0x7fb27415aa60 2023-10-14T17:01:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb273fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2740afd60 0x7fb2740cc2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41268/0 (socket says 172.21.15.29:41268) 2023-10-14T17:01:25.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.101+0000 7fb273fff640 1 -- 172.21.15.29:0/3249749245 learned_addr learned my addr 172.21.15.29:0/3249749245 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb278ecf640 1 --2- 172.21.15.29:0/3249749245 >> v2:172.21.15.186:3300/0 conn(0x7fb27415aa60 0x7fb2740af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb2737fe640 1 -- 172.21.15.29:0/3249749245 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458739791 0 0) 0x7fb2740b3730 con 0x7fb2740afd60 2023-10-14T17:01:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb2737fe640 1 -- 172.21.15.29:0/3249749245 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb264003520 con 0x7fb2740afd60 2023-10-14T17:01:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb278ecf640 1 -- 172.21.15.29:0/3249749245 >> v1:172.21.15.186:6789/0 conn(0x7fb2740cc9f0 legacy=0x7fb2740b4740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb278ecf640 1 -- 172.21.15.29:0/3249749245 >> v1:172.21.15.29:6789/0 conn(0x7fb2740afd60 legacy=0x7fb2740cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.102+0000 7fb278ecf640 1 -- 172.21.15.29:0/3249749245 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2740b3f40 con 0x7fb27415aa60 2023-10-14T17:01:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.103+0000 7fb278ecf640 1 --2- 172.21.15.29:0/3249749245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740af820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb2680097a0 tx=0x7fb26802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19879705c37a1593 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.104+0000 7fb2737fe640 1 -- 172.21.15.29:0/3249749245 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268030610 con 0x7fb27415aa60 2023-10-14T17:01:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.104+0000 7fb2737fe640 1 -- 172.21.15.29:0/3249749245 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb268030770 con 0x7fb27415aa60 2023-10-14T17:01:25.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.104+0000 7fb2737fe640 1 -- 172.21.15.29:0/3249749245 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb268030ae0 con 0x7fb27415aa60 2023-10-14T17:01:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.104+0000 7fb279ed1640 1 -- 172.21.15.29:0/3249749245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 msgr2=0x7fb2740af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.104+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3249749245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740af820 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fb2680097a0 tx=0x7fb26802b960 comp rx=0 tx=0).stop 2023-10-14T17:01:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.105+0000 7fb279ed1640 1 -- 172.21.15.29:0/3249749245 shutdown_connections 2023-10-14T17:01:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.105+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3249749245 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740af820 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.105+0000 7fb279ed1640 1 -- 172.21.15.29:0/3249749245 >> 172.21.15.29:0/3249749245 conn(0x7fb274065ed0 msgr2=0x7fb2740662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:25.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.105+0000 7fb279ed1640 1 -- 172.21.15.29:0/3249749245 shutdown_connections 2023-10-14T17:01:25.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.105+0000 7fb279ed1640 1 -- 172.21.15.29:0/3249749245 wait complete. 2023-10-14T17:01:25.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.107+0000 7fb279ed1640 1 Processor -- start 2023-10-14T17:01:25.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.107+0000 7fb279ed1640 1 -- start start 2023-10-14T17:01:25.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2740afd60 0x7fb2741588a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740b1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b24e0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740ad360 con 0x7fb27415aa60 2023-10-14T17:01:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb279ed1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2740b2230 con 0x7fb2740afd60 2023-10-14T17:01:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb273fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb273fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38006/0 (socket says 172.21.15.29:38006) 2023-10-14T17:01:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.108+0000 7fb273fff640 1 -- 172.21.15.29:0/3838968569 learned_addr learned my addr 172.21.15.29:0/3838968569 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb2796d0640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740b1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb278ecf640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2740afd60 0x7fb2741588a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2740afd60 msgr2=0x7fb2741588a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2740afd60 0x7fb2741588a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 msgr2=0x7fb2740b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740b1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 -- 172.21.15.29:0/3838968569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb268009400 con 0x7fb2740cc9f0 2023-10-14T17:01:25.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb273fff640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fb26000c500 tx=0x7fb26000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:25.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.109+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260005ce0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.110+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb260005e40 con 0x7fb2740cc9f0 2023-10-14T17:01:25.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.110+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb260007330 con 0x7fb2740cc9f0 2023-10-14T17:01:25.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.110+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2740aca30 con 0x7fb2740cc9f0 2023-10-14T17:01:25.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.110+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2740acff0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.111+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb27404e9b0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.113+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 81) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fb260007490 con 0x7fb2740cc9f0 2023-10-14T17:01:25.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.113+0000 7fb2717fa640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 0x7fb25806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.114+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fb260088f40 con 0x7fb2740cc9f0 2023-10-14T17:01:25.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.114+0000 7fb278ecf640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 msgr2=0x7fb25806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:25.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.114+0000 7fb278ecf640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 0x7fb25806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:01:25.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.123+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb260053ec0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.315+0000 7fb278ecf640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 msgr2=0x7fb25806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:25.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.315+0000 7fb278ecf640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 0x7fb25806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:01:25.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.323+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb2740b41f0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.329+0000 7fb2717fa640 1 -- 172.21.15.29:0/3838968569 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v53)=0 v53) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb2600170d0 con 0x7fb2740cc9f0 2023-10-14T17:01:25.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 msgr2=0x7fb25806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:01:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 0x7fb25806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 msgr2=0x7fb2741582b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 secure :-1 s=READY pgs=1824 cs=0 l=1 rev1=1 crypto rx=0x7fb26000c500 tx=0x7fb26000c9c0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 shutdown_connections 2023-10-14T17:01:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fb25806afe0 0x7fb25806d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb27415aa60 0x7fb2740b1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2740cc9f0 0x7fb2741582b0 unknown :-1 s=CLOSED pgs=1824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 --2- 172.21.15.29:0/3838968569 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2740afd60 0x7fb2741588a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.332+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 >> 172.21.15.29:0/3838968569 conn(0x7fb274065ed0 msgr2=0x7fb27406c730 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.333+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 shutdown_connections 2023-10-14T17:01:25.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.333+0000 7fb279ed1640 1 -- 172.21.15.29:0/3838968569 wait complete. 2023-10-14T17:01:25.352 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T17:01:25.419 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:25.418+0000 7fd4adab2040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:01:25.431 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:25.431+0000 7f4c0d82e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:01:25.472 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:25.471+0000 7fd4adab2040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:01:25.490 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:25.489+0000 7f4c0d82e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:01:25.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.558+0000 7fdd9ed05640 1 Processor -- start 2023-10-14T17:01:25.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.558+0000 7fdd9ed05640 1 -- start start 2023-10-14T17:01:25.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9ed05640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd980cf710 0x7fdd980b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9ed05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 0x7fdd980b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9ed05640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd980d3980 con 0x7fdd980b38e0 2023-10-14T17:01:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9ed05640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd980d3c60 con 0x7fdd980cf710 2023-10-14T17:01:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9ed05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980ce630 con 0x7fdd980cfaf0 2023-10-14T17:01:25.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9caa1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd980cf710 0x7fdd980b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9caa1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd980cf710 0x7fdd980b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38012/0 (socket says 172.21.15.29:38012) 2023-10-14T17:01:25.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3780089524 learned_addr learned my addr 172.21.15.29:0/3780089524 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:25.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 0x7fdd980b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3780089524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 msgr2=0x7fdd980b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:25.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.560+0000 7fdd9caa1640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 0x7fdd980b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.561+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3780089524 >> v1:172.21.15.186:6789/0 conn(0x7fdd980b38e0 legacy=0x7fdd980d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.561+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3780089524 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd980ce910 con 0x7fdd980cf710 2023-10-14T17:01:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.561+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 0x7fdd980b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.562+0000 7fdd9caa1640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980cf710 0x7fdd980b2d10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009850 tx=0x7fdd9402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=464372c8d33fe85 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.562+0000 7fdd8f7fe640 1 -- 172.21.15.29:0/3780089524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd9402e070 con 0x7fdd980cf710 2023-10-14T17:01:25.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.562+0000 7fdd8f7fe640 1 -- 172.21.15.29:0/3780089524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd94004030 con 0x7fdd980cf710 2023-10-14T17:01:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.562+0000 7fdd8f7fe640 1 -- 172.21.15.29:0/3780089524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94004420 con 0x7fdd980cf710 2023-10-14T17:01:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3780089524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980cf710 msgr2=0x7fdd980b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980cf710 0x7fdd980b2d10 secure :-1 s=READY pgs=1825 cs=0 l=1 rev1=1 crypto rx=0x7fdd94009850 tx=0x7fdd9402b840 comp rx=0 tx=0).stop 2023-10-14T17:01:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3780089524 shutdown_connections 2023-10-14T17:01:25.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cfaf0 0x7fdd980b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3780089524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980cf710 0x7fdd980b2d10 unknown :-1 s=CLOSED pgs=1825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3780089524 >> 172.21.15.29:0/3780089524 conn(0x7fdd980d2cc0 msgr2=0x7fdd980d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3780089524 shutdown_connections 2023-10-14T17:01:25.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.563+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3780089524 wait complete. 2023-10-14T17:01:25.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.564+0000 7fdd9ed05640 1 Processor -- start 2023-10-14T17:01:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.565+0000 7fdd9ed05640 1 -- start start 2023-10-14T17:01:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.565+0000 7fdd9ed05640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9ed05640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 0x7fdd980c67f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9ed05640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 0x7fdd981f2630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9ed05640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980c7d70 con 0x7fdd980b38e0 2023-10-14T17:01:25.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9ed05640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980c7160 con 0x7fdd980cfaf0 2023-10-14T17:01:25.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9ed05640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd980c7440 con 0x7fdd980cf710 2023-10-14T17:01:25.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38018/0 (socket says 172.21.15.29:38018) 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3738968300 learned_addr learned my addr 172.21.15.29:0/3738968300 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 0x7fdd980c67f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9d2a2640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 0x7fdd981f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 msgr2=0x7fdd980c67f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 0x7fdd980c67f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 msgr2=0x7fdd981f2630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 0x7fdd981f2630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9caa1640 1 -- 172.21.15.29:0/3738968300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd94009460 con 0x7fdd980b38e0 2023-10-14T17:01:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 0x7fdd980c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.566+0000 7fdd9d2a2640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 0x7fdd981f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:25.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd9caa1640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fdd94035040 tx=0x7fdd940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd940039a0 con 0x7fdd980b38e0 2023-10-14T17:01:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd94003b00 con 0x7fdd980b38e0 2023-10-14T17:01:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd980c7720 con 0x7fdd980b38e0 2023-10-14T17:01:25.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd94003e90 con 0x7fdd980b38e0 2023-10-14T17:01:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.567+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9815ec30 con 0x7fdd980b38e0 2023-10-14T17:01:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.569+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9804e160 con 0x7fdd980b38e0 2023-10-14T17:01:25.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.570+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 81) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdd94006040 con 0x7fdd980b38e0 2023-10-14T17:01:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.571+0000 7fdd8d7fa640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 0x7fdd8406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.571+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fdd940ab3b0 con 0x7fdd980b38e0 2023-10-14T17:01:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.571+0000 7fdd8ffff640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 msgr2=0x7fdd8406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:25.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.571+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 0x7fdd8406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:01:25.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.576+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd94076270 con 0x7fdd980b38e0 2023-10-14T17:01:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.772+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdd980d0a10 con 0x7fdd980b38e0 2023-10-14T17:01:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.772+0000 7fdd8ffff640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 msgr2=0x7fdd8406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.772+0000 7fdd8ffff640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 0x7fdd8406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:01:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.777+0000 7fdd8d7fa640 1 -- 172.21.15.29:0/3738968300 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v54)=0 v54) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdd94079890 con 0x7fdd980b38e0 2023-10-14T17:01:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 msgr2=0x7fdd8406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:01:25.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 0x7fdd8406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 msgr2=0x7fdd9815e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 secure :-1 s=READY pgs=1826 cs=0 l=1 rev1=1 crypto rx=0x7fdd94035040 tx=0x7fdd940046c0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 shutdown_connections 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fdd8406afe0 0x7fdd8406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd980cfaf0 0x7fdd981f2630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd980cf710 0x7fdd980c67f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 --2- 172.21.15.29:0/3738968300 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd980b38e0 0x7fdd9815e6f0 unknown :-1 s=CLOSED pgs=1826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 >> 172.21.15.29:0/3738968300 conn(0x7fdd980d2cc0 msgr2=0x7fdd980d1210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.779+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 shutdown_connections 2023-10-14T17:01:25.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:25.780+0000 7fdd9ed05640 1 -- 172.21.15.29:0/3738968300 wait complete. 2023-10-14T17:01:25.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:01:25.800 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T17:01:25.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1 1m 2023-10-14T17:01:25.922 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:25.921+0000 7fd4adab2040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:01:25.974 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:25.973+0000 7fd4adab2040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:01:25.977 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:25.977+0000 7f4c0d82e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:01:26.023 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.022+0000 7fd4adab2040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:01:26.034 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.033+0000 7f4c0d82e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:01:26.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.054+0000 7fc91d283640 1 Processor -- start 2023-10-14T17:01:26.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.054+0000 7fc91d283640 1 -- start start 2023-10-14T17:01:26.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.056+0000 7fc91d283640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9180b0ba0 0x7fc91806b300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.056+0000 7fc91d283640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc9180c20f0 0x7fc9180c2520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc91d283640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9180c2a60 con 0x7fc918161290 2023-10-14T17:01:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc91d283640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9180cd400 con 0x7fc9180b0ba0 2023-10-14T17:01:26.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc91d283640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9180b1060 con 0x7fc9180c20f0 2023-10-14T17:01:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc90ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9180b0ba0 0x7fc91806b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc90ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc9180b0ba0 0x7fc91806b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38030/0 (socket says 172.21.15.29:38030) 2023-10-14T17:01:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc90ffff640 1 -- 172.21.15.29:0/3472531755 learned_addr learned my addr 172.21.15.29:0/3472531755 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:26.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc9177fe640 1 --2- 172.21.15.29:0/3472531755 >> v2:172.21.15.186:3300/0 conn(0x7fc9180c20f0 0x7fc9180c2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc90ffff640 1 -- 172.21.15.29:0/3472531755 >> v2:172.21.15.186:3300/0 conn(0x7fc9180c20f0 msgr2=0x7fc9180c2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.057+0000 7fc90ffff640 1 --2- 172.21.15.29:0/3472531755 >> v2:172.21.15.186:3300/0 conn(0x7fc9180c20f0 0x7fc9180c2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.058+0000 7fc90ffff640 1 -- 172.21.15.29:0/3472531755 >> v1:172.21.15.186:6789/0 conn(0x7fc918161290 legacy=0x7fc9180cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.058+0000 7fc90ffff640 1 -- 172.21.15.29:0/3472531755 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9180b1340 con 0x7fc9180b0ba0 2023-10-14T17:01:26.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.059+0000 7fc90ffff640 1 --2- 172.21.15.29:0/3472531755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9180b0ba0 0x7fc91806b300 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc9180abe90 tx=0x7fc90002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76bc66f0bb7b43 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.059+0000 7fc9167fc640 1 -- 172.21.15.29:0/3472531755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900030860 con 0x7fc9180b0ba0 2023-10-14T17:01:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.059+0000 7fc9167fc640 1 -- 172.21.15.29:0/3472531755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9000309c0 con 0x7fc9180b0ba0 2023-10-14T17:01:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.059+0000 7fc9167fc640 1 -- 172.21.15.29:0/3472531755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc900030d50 con 0x7fc9180b0ba0 2023-10-14T17:01:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 -- 172.21.15.29:0/3472531755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9180b0ba0 msgr2=0x7fc91806b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:26.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 --2- 172.21.15.29:0/3472531755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9180b0ba0 0x7fc91806b300 secure :-1 s=READY pgs=1827 cs=0 l=1 rev1=1 crypto rx=0x7fc9180abe90 tx=0x7fc90002b9a0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 -- 172.21.15.29:0/3472531755 shutdown_connections 2023-10-14T17:01:26.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 --2- 172.21.15.29:0/3472531755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc9180b0ba0 0x7fc91806b300 unknown :-1 s=CLOSED pgs=1827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 --2- 172.21.15.29:0/3472531755 >> v2:172.21.15.186:3300/0 conn(0x7fc9180c20f0 0x7fc9180c2520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 -- 172.21.15.29:0/3472531755 >> 172.21.15.29:0/3472531755 conn(0x7fc9180cbba0 msgr2=0x7fc9180cbf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:26.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 -- 172.21.15.29:0/3472531755 shutdown_connections 2023-10-14T17:01:26.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.060+0000 7fc91d283640 1 -- 172.21.15.29:0/3472531755 wait complete. 2023-10-14T17:01:26.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.062+0000 7fc91d283640 1 Processor -- start 2023-10-14T17:01:26.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.062+0000 7fc91d283640 1 -- start start 2023-10-14T17:01:26.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9180b0ba0 0x7fc918156730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9180c20f0 0x7fc918070680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 0x7fc91815ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc918163b40 con 0x7fc918161290 2023-10-14T17:01:26.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9181636b0 con 0x7fc9180b0ba0 2023-10-14T17:01:26.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc91d283640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc918156c70 con 0x7fc9180c20f0 2023-10-14T17:01:26.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc9177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 0x7fc91815ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:26.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc9177fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 0x7fc91815ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38032/0 (socket says 172.21.15.29:38032) 2023-10-14T17:01:26.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.063+0000 7fc9177fe640 1 -- 172.21.15.29:0/3361793557 learned_addr learned my addr 172.21.15.29:0/3361793557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:26.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc90ffff640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9180c20f0 0x7fc918070680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:26.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9180b0ba0 0x7fc918156730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:26.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9180c20f0 msgr2=0x7fc918070680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:26.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9180c20f0 0x7fc918070680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9180b0ba0 msgr2=0x7fc918156730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:26.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc9180b0ba0 0x7fc918156730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:26.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000094d0 con 0x7fc918161290 2023-10-14T17:01:26.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc90ffff640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc9180c20f0 0x7fc918070680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:26.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc9177fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 0x7fc91815ed80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc910007e70 tx=0x7fc91000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:26.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910014620 con 0x7fc918161290 2023-10-14T17:01:26.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9100147a0 con 0x7fc918161290 2023-10-14T17:01:26.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc910014b50 con 0x7fc918161290 2023-10-14T17:01:26.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.064+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc918070bc0 con 0x7fc918161290 2023-10-14T17:01:26.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.065+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918162d80 con 0x7fc918161290 2023-10-14T17:01:26.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.066+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91804e160 con 0x7fc918161290 2023-10-14T17:01:26.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.067+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 81) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc910014cb0 con 0x7fc918161290 2023-10-14T17:01:26.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.068+0000 7fc90f7fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:26.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.068+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(154..154 src has 1..154) v4 ==== 5781+0+0 (secure 0 0 0) 0x7fc91008a120 con 0x7fc918161290 2023-10-14T17:01:26.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.068+0000 7fc916ffd640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:26.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.068+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:01:26.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.072+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc910055100 con 0x7fc918161290 2023-10-14T17:01:26.084 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.083+0000 7fd4adab2040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:01:26.089 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.089+0000 7f4c0d82e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:01:26.132 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.131+0000 7fd4adab2040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:01:26.155 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.154+0000 7f4c0d82e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:01:26.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.207+0000 7f4c0d82e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:01:26.242 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.241+0000 7fd4adab2040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:01:26.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.269+0000 7fc916ffd640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.269+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:01:26.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.288+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc9180b1900 con 0x7fc8f806afe0 2023-10-14T17:01:26.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.302+0000 7fd4adab2040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:01:26.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.325+0000 7f4c0d82e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:01:26.353 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.352+0000 7fd4adab2040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:01:26.393 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.393+0000 7f4c0d82e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:01:26.410 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.409+0000 7fd4adab2040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:01:26.446 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.445+0000 7f4c0d82e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:01:26.459 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.458+0000 7fd4adab2040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:01:26.505 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.505+0000 7f4c0d82e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:01:26.558 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.558+0000 7f4c0d82e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:01:26.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.670+0000 7fc916ffd640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:26.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:26.670+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T17:01:26.835 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.834+0000 7fd4adab2040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:01:26.964 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:26.963+0000 7f4c0d82e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:01:26.994 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:26.993+0000 7fd4adab2040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:01:27.100 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.099+0000 7fd4adab2040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:01:27.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.133+0000 7f4c0d82e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:01:27.149 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.148+0000 7fd4adab2040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:01:27.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.248+0000 7f4c0d82e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:01:27.255 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.254+0000 7fd4adab2040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:01:27.302 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.301+0000 7f4c0d82e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:01:27.363 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.362+0000 7fd4adab2040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:01:27.417 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.416+0000 7f4c0d82e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:01:27.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:27.471+0000 7fc916ffd640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:27.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:27.471+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T17:01:27.518 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.517+0000 7fd4adab2040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:01:27.530 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.529+0000 7f4c0d82e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:01:27.566 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.565+0000 7fd4adab2040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:01:27.684 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:27.683+0000 7fd4adab2040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:01:27.697 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.696+0000 7f4c0d82e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:01:27.750 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.749+0000 7f4c0d82e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:01:27.876 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:27.875+0000 7f4c0d82e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:01:28.491 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:28.490+0000 7fd4adab2040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:01:28.773 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:28.772+0000 7f4c0d82e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:01:28.782 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:28.781+0000 7fd4adab2040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:01:28.838 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:28.837+0000 7fd4adab2040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:01:28.951 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:01:28.950+0000 7fd4adab2040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:01:29.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.022+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 82) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fc9100554d0 con 0x7fc918161290 2023-10-14T17:01:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.072+0000 7fc916ffd640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/3121942650 2023-10-14T17:01:29.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.072+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T17:01:29.091 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.090+0000 7f4c0d82e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:01:29.151 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.151+0000 7f4c0d82e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:01:29.271 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.270+0000 7f4c0d82e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:01:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.290+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 83) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fc9100708e0 con 0x7fc918161290 2023-10-14T17:01:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.291+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 msgr2=0x7fc8f806d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:01:29.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:29.291+0000 7fc90f7fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/3121942650,v1:172.21.15.29:6833/3121942650] conn(0x7fc8f806afe0 0x7fc8f806d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:29.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.607+0000 7f4b89b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.607+0000 7f4b89b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.607+0000 7f4b89b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.609 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.607+0000 7f4b89b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.607+0000 7f4b89b66640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.608+0000 7f4b8c3ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.608+0000 7f4b8c3ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.608+0000 7f4b8c3ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.611 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.608+0000 7f4b8c3ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:29.611 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:01:29.608+0000 7f4b8c3ab640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:01:30.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.300+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 84) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc910055da0 con 0x7fc918161290 2023-10-14T17:01:30.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.302+0000 7fc90f7fe640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc8f801d1c0 0x7fc8f8074f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.302+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 --> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8f801d5a0 con 0x7fc8f801d1c0 2023-10-14T17:01:30.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.309+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc8f801d1c0 0x7fc8f8074f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.321+0000 7fc916ffd640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc8f801d1c0 0x7fc8f8074f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc91815dce0 tx=0x7fc908001120 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:30.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.509+0000 7fc90f7fe640 1 -- 172.21.15.29:0/3361793557 <== mgr.9255 v2:172.21.15.29:6832/2309931697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fc8f801d5a0 con 0x7fc8f801d1c0 2023-10-14T17:01:30.510 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1 2023-10-14T17:01:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc8f801d1c0 msgr2=0x7fc8f8074f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc8f801d1c0 0x7fc8f8074f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc91815dce0 tx=0x7fc908001120 comp rx=0 tx=0).stop 2023-10-14T17:01:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 msgr2=0x7fc91815ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 --2- 172.21.15.29:0/3361793557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc918161290 0x7fc91815ed80 secure :-1 s=READY pgs=1828 cs=0 l=1 rev1=1 crypto rx=0x7fc910007e70 tx=0x7fc91000e4b0 comp rx=0 tx=0).stop 2023-10-14T17:01:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc9177fe640 1 -- 172.21.15.29:0/3361793557 reap_dead start 2023-10-14T17:01:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 shutdown_connections 2023-10-14T17:01:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 >> 172.21.15.29:0/3361793557 conn(0x7fc9180cbba0 msgr2=0x7fc9180b03c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.512+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 shutdown_connections 2023-10-14T17:01:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.513+0000 7fc91d283640 1 -- 172.21.15.29:0/3361793557 wait complete. 2023-10-14T17:01:30.535 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1 2023-10-14T17:01:30.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add /snap_test_dir1 55n 2023-10-14T17:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.766+0000 7f5483815640 1 Processor -- start 2023-10-14T17:01:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.766+0000 7f5483815640 1 -- start start 2023-10-14T17:01:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f5483815640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f547c0cac20 0x7f547c0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f5483815640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c15b310 con 0x7f547c15ac10 2023-10-14T17:01:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f5483815640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547c06f640 con 0x7f547c0b0360 2023-10-14T17:01:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f5483815640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f547c15f800 con 0x7f547c0cac20 2023-10-14T17:01:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f54815b1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f547c15ac10 0x7f547c06c730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41272/0 (socket says 172.21.15.29:41272) 2023-10-14T17:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f5481db2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f547c0cac20 0x7f547c0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.768+0000 7f54815b1640 1 -- 172.21.15.29:0/3452298803 learned_addr learned my addr 172.21.15.29:0/3452298803 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.769+0000 7f5481db2640 1 -- 172.21.15.29:0/3452298803 >> v1:172.21.15.186:6789/0 conn(0x7f547c0b0360 legacy=0x7f547c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.769+0000 7f5481db2640 1 -- 172.21.15.29:0/3452298803 >> v1:172.21.15.29:6789/0 conn(0x7f547c15ac10 legacy=0x7f547c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.769+0000 7f5481db2640 1 -- 172.21.15.29:0/3452298803 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c15b5b0 con 0x7f547c0cac20 2023-10-14T17:01:30.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.769+0000 7f5473fff640 1 -- 172.21.15.29:0/3452298803 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678485876 0 0) 0x7f547c15b310 con 0x7f547c15ac10 2023-10-14T17:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.770+0000 7f5481db2640 1 --2- 172.21.15.29:0/3452298803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c0b07f0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f547402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2185af019933009e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.770+0000 7f5473fff640 1 -- 172.21.15.29:0/3452298803 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547402cc60 con 0x7f547c0cac20 2023-10-14T17:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5473fff640 1 -- 172.21.15.29:0/3452298803 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547402cdc0 con 0x7f547c0cac20 2023-10-14T17:01:30.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5473fff640 1 -- 172.21.15.29:0/3452298803 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5474034660 con 0x7f547c0cac20 2023-10-14T17:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 -- 172.21.15.29:0/3452298803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 msgr2=0x7f547c0b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 --2- 172.21.15.29:0/3452298803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c0b07f0 secure :-1 s=READY pgs=1843 cs=0 l=1 rev1=1 crypto rx=0x7f5474009d90 tx=0x7f547402beb0 comp rx=0 tx=0).stop 2023-10-14T17:01:30.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 -- 172.21.15.29:0/3452298803 shutdown_connections 2023-10-14T17:01:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 --2- 172.21.15.29:0/3452298803 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c0b07f0 unknown :-1 s=CLOSED pgs=1843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 -- 172.21.15.29:0/3452298803 >> 172.21.15.29:0/3452298803 conn(0x7f547c0cb360 msgr2=0x7f547c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 -- 172.21.15.29:0/3452298803 shutdown_connections 2023-10-14T17:01:30.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.771+0000 7f5483815640 1 -- 172.21.15.29:0/3452298803 wait complete. 2023-10-14T17:01:30.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.774+0000 7f5483815640 1 Processor -- start 2023-10-14T17:01:30.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.774+0000 7f5483815640 1 -- start start 2023-10-14T17:01:30.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f547c0b0360 0x7f547c157df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547c15ac10 0x7f547c158c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c158330 con 0x7f547c0cac20 2023-10-14T17:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0c7e60 con 0x7f547c15ac10 2023-10-14T17:01:30.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5483815640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f547c0c8140 con 0x7f547c0b0360 2023-10-14T17:01:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5480db0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5480db0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38204/0 (socket says 172.21.15.29:38204) 2023-10-14T17:01:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.775+0000 7f5480db0640 1 -- 172.21.15.29:0/571827558 learned_addr learned my addr 172.21.15.29:0/571827558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5481db2640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547c15ac10 0x7f547c158c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f54815b1640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f547c0b0360 0x7f547c157df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 -- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f547c0b0360 msgr2=0x7f547c157df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f547c0b0360 0x7f547c157df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 -- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547c15ac10 msgr2=0x7f547c158c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547c15ac10 0x7f547c158c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:30.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 -- 172.21.15.29:0/571827558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5474009a70 con 0x7f547c0cac20 2023-10-14T17:01:30.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5480db0640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c450 tx=0x7f546c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.776+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c0052d0 con 0x7f547c0cac20 2023-10-14T17:01:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.777+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f546c005430 con 0x7f547c0cac20 2023-10-14T17:01:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.777+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c012870 con 0x7f547c0cac20 2023-10-14T17:01:30.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.777+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f547c0c8420 con 0x7f547c0cac20 2023-10-14T17:01:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.777+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f547c15c270 con 0x7f547c0cac20 2023-10-14T17:01:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.778+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f547c04e160 con 0x7f547c0cac20 2023-10-14T17:01:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.779+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 84) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f546c0129d0 con 0x7f547c0cac20 2023-10-14T17:01:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.780+0000 7f5471ffb640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 0x7f546006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:30.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.780+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6125+0+0 (secure 0 0 0) 0x7f546c089040 con 0x7f547c0cac20 2023-10-14T17:01:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.780+0000 7f54815b1640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 0x7f546006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:30.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.781+0000 7f54815b1640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 0x7f546006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5478000f60 tx=0x7f5478009040 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:30.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:30.785+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f546c0bbfe0 con 0x7f547c0cac20 2023-10-14T17:01:31.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.023+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 --> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "/snap_test_dir1", "retention_spec_or_period": "55n", "target": ["mon-mgr", ""]}) v1 -- 0x7f547c15bb70 con 0x7f546006ad10 2023-10-14T17:01:31.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.196+0000 7f5471ffb640 1 -- 172.21.15.29:0/571827558 <== mgr.9255 v2:172.21.15.29:6832/2309931697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+40 (secure 0 0 0) 0x7f5478001200 con 0x7f546006ad10 2023-10-14T17:01:31.197 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path /snap_test_dir1 2023-10-14T17:01:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 msgr2=0x7f546006d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 0x7f546006d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5478000f60 tx=0x7f5478009040 comp rx=0 tx=0).stop 2023-10-14T17:01:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 msgr2=0x7f547c158750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 secure :-1 s=READY pgs=1844 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c450 tx=0x7f546c00c910 comp rx=0 tx=0).stop 2023-10-14T17:01:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 shutdown_connections 2023-10-14T17:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f546006ad10 0x7f546006d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f547c15ac10 0x7f547c158c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f547c0cac20 0x7f547c158750 unknown :-1 s=CLOSED pgs=1844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 --2- 172.21.15.29:0/571827558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f547c0b0360 0x7f547c157df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 >> 172.21.15.29:0/571827558 conn(0x7f547c0cb360 msgr2=0x7f547c0ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 shutdown_connections 2023-10-14T17:01:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.200+0000 7f5483815640 1 -- 172.21.15.29:0/571827558 wait complete. 2023-10-14T17:01:31.221 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path /snap_test_dir1 2023-10-14T17:01:31.222 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [] 2023-10-14T17:01:31.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json /snap_test_dir1 2023-10-14T17:01:31.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.452+0000 7fed89289640 1 Processor -- start 2023-10-14T17:01:31.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.452+0000 7fed89289640 1 -- start start 2023-10-14T17:01:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.453+0000 7fed89289640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed8415ad70 0x7fed84069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.453+0000 7fed89289640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed840c9ca0 0x7fed8406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.453+0000 7fed89289640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed840cb030 con 0x7fed840ca3b0 2023-10-14T17:01:31.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.453+0000 7fed89289640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed840c91c0 con 0x7fed840c9ca0 2023-10-14T17:01:31.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.453+0000 7fed89289640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed840c94a0 con 0x7fed8415ad70 2023-10-14T17:01:31.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed840c9ca0 0x7fed8406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:31.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed840c9ca0 0x7fed8406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60570/0 (socket says 172.21.15.29:60570) 2023-10-14T17:01:31.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 -- 172.21.15.29:0/2287655561 learned_addr learned my addr 172.21.15.29:0/2287655561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:31.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 -- 172.21.15.29:0/2287655561 >> v2:172.21.15.186:3300/0 conn(0x7fed8415ad70 msgr2=0x7fed84069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:31.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 --2- 172.21.15.29:0/2287655561 >> v2:172.21.15.186:3300/0 conn(0x7fed8415ad70 0x7fed84069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 -- 172.21.15.29:0/2287655561 >> v1:172.21.15.186:6789/0 conn(0x7fed840ca3b0 legacy=0x7fed840ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:01:31.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.454+0000 7fed7a7fc640 1 -- 172.21.15.29:0/2287655561 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed840c9780 con 0x7fed840c9ca0 2023-10-14T17:01:31.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.455+0000 7fed7a7fc640 1 --2- 172.21.15.29:0/2287655561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed8406a2e0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fed840cf780 tx=0x7fed6c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1a9b21e9150352 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:31.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed827fc640 1 -- 172.21.15.29:0/2287655561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c031870 con 0x7fed840c9ca0 2023-10-14T17:01:31.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed827fc640 1 -- 172.21.15.29:0/2287655561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed6c0319d0 con 0x7fed840c9ca0 2023-10-14T17:01:31.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed827fc640 1 -- 172.21.15.29:0/2287655561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed6c031d60 con 0x7fed840c9ca0 2023-10-14T17:01:31.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 -- 172.21.15.29:0/2287655561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 msgr2=0x7fed8406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 --2- 172.21.15.29:0/2287655561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed8406a2e0 secure :-1 s=READY pgs=1847 cs=0 l=1 rev1=1 crypto rx=0x7fed840cf780 tx=0x7fed6c02b910 comp rx=0 tx=0).stop 2023-10-14T17:01:31.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 -- 172.21.15.29:0/2287655561 shutdown_connections 2023-10-14T17:01:31.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 --2- 172.21.15.29:0/2287655561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed8406a2e0 unknown :-1 s=CLOSED pgs=1847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 --2- 172.21.15.29:0/2287655561 >> v2:172.21.15.186:3300/0 conn(0x7fed8415ad70 0x7fed84069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 -- 172.21.15.29:0/2287655561 >> 172.21.15.29:0/2287655561 conn(0x7fed840ce9b0 msgr2=0x7fed840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 -- 172.21.15.29:0/2287655561 shutdown_connections 2023-10-14T17:01:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.456+0000 7fed89289640 1 -- 172.21.15.29:0/2287655561 wait complete. 2023-10-14T17:01:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.458+0000 7fed89289640 1 Processor -- start 2023-10-14T17:01:31.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.458+0000 7fed89289640 1 -- start start 2023-10-14T17:01:31.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed840ca3b0 0x7fed8415b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 0x7fed841f62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed84154510 con 0x7fed840c9ca0 2023-10-14T17:01:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed840b3550 con 0x7fed8415ad70 2023-10-14T17:01:31.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed89289640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed840b3830 con 0x7fed840ca3b0 2023-10-14T17:01:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.459+0000 7fed82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60584/0 (socket says 172.21.15.29:60584) 2023-10-14T17:01:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 -- 172.21.15.29:0/2043174428 learned_addr learned my addr 172.21.15.29:0/2043174428 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:01:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed7a7fc640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed840ca3b0 0x7fed8415b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:31.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 -- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed840ca3b0 msgr2=0x7fed8415b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed840ca3b0 0x7fed8415b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 -- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 msgr2=0x7fed841f62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed837fe640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 0x7fed841f62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 0x7fed841f62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed82ffd640 1 -- 172.21.15.29:0/2043174428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed6c0094d0 con 0x7fed840c9ca0 2023-10-14T17:01:31.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.460+0000 7fed837fe640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 0x7fed841f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:01:31.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed82ffd640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fed7400ba50 tx=0x7fed7400bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:31.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed7400cbe0 con 0x7fed840c9ca0 2023-10-14T17:01:31.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed7400cd60 con 0x7fed840c9ca0 2023-10-14T17:01:31.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed74014680 con 0x7fed840c9ca0 2023-10-14T17:01:31.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed841f5aa0 con 0x7fed840c9ca0 2023-10-14T17:01:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.461+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed841f67e0 con 0x7fed840c9ca0 2023-10-14T17:01:31.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.462+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed841f5fc0 con 0x7fed840c9ca0 2023-10-14T17:01:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.463+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 85) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fed7401d3b0 con 0x7fed840c9ca0 2023-10-14T17:01:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.464+0000 7fed7bfff640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 0x7fed6406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:01:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.464+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(155..155 src has 1..155) v4 ==== 6125+0+0 (secure 0 0 0) 0x7fed740898f0 con 0x7fed840c9ca0 2023-10-14T17:01:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.465+0000 7fed7a7fc640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 0x7fed6406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:01:31.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.465+0000 7fed7a7fc640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 0x7fed6406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed6c036000 tx=0x7fed6c03c040 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:01:31.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.469+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed841f5fc0 con 0x7fed840c9ca0 2023-10-14T17:01:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.691+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 --> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fed841f5920 con 0x7fed6406afe0 2023-10-14T17:01:31.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.709+0000 7fed7bfff640 1 -- 172.21.15.29:0/2043174428 <== mgr.9255 v2:172.21.15.29:6832/2309931697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+74 (secure 0 0 0) 0x7fed6c031cc0 con 0x7fed6406afe0 2023-10-14T17:01:31.710 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:01:31.710 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-14T17:01:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.711+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 msgr2=0x7fed6406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.711+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 0x7fed6406d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fed6c036000 tx=0x7fed6c03c040 comp rx=0 tx=0).stop 2023-10-14T17:01:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 msgr2=0x7fed840b2f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:01:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 secure :-1 s=READY pgs=1848 cs=0 l=1 rev1=1 crypto rx=0x7fed7400ba50 tx=0x7fed7400bf10 comp rx=0 tx=0).stop 2023-10-14T17:01:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 shutdown_connections 2023-10-14T17:01:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fed6406afe0 0x7fed6406d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed8415ad70 0x7fed841f62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed840ca3b0 0x7fed8415b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 --2- 172.21.15.29:0/2043174428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed840c9ca0 0x7fed840b2f70 unknown :-1 s=CLOSED pgs=1848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:01:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 >> 172.21.15.29:0/2043174428 conn(0x7fed840ce9b0 msgr2=0x7fed840ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:01:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 shutdown_connections 2023-10-14T17:01:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:01:31.712+0000 7fed89289640 1 -- 172.21.15.29:0/2043174428 wait complete. 2023-10-14T17:01:31.731 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "/snap_test_dir1", "schedule": ["1m"], "retention": [{"n": 55}]} 2023-10-14T17:01:31.731 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': '/snap_test_dir1', 'schedule': ['1m'], 'retention': [{'n': 55}]} 2023-10-14T17:01:47.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:01:47.244 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:02:00.224 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:02:00.223+0000 7f4b7bb4a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:02:00.224 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:02:00.223+0000 7f4b7bb4a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:02:00.224 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:02:00.223+0000 7f4b7bb4a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:02:00.225 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:02:00.223+0000 7f4b7bb4a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:02:00.225 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:02:00.223+0000 7f4b7bb4a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:02:17.444 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:02:17.455 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:02:47.596 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:02:47.606 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:03:17.746 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:03:17.756 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:03:47.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:03:47.933 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:04:18.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:04:18.099 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:04:48.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:04:48.298 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:05:18.433 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:05:18.443 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:05:48.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:05:48.615 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:06:18.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:06:18.776 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:06:48.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:06:48.927 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:07:19.090 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:07:19.100 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:07:49.249 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:07:49.259 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:08:19.402 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:08:19.413 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:08:49.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:08:49.569 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:09:19.709 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:09:19.719 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:09:49.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:09:49.879 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:10:20.025 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:10:20.037 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:10:50.184 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:10:50.194 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:11:20.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:11:20.346 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:11:50.495 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:11:50.505 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:12:20.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:12:20.653 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:12:50.812 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:12:50.822 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:13:20.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:13:21.005 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:13:51.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:13:51.164 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:14:21.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:14:21.323 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:14:51.479 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:14:51.489 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:15:21.708 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:15:21.721 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:15:51.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:15:51.865 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:16:22.006 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:16:22.016 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:16:52.224 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:16:52.234 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:17:22.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:17:22.466 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:17:52.661 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:17:52.671 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:18:22.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:18:22.894 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:18:53.105 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:18:53.115 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:19:23.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:19:23.330 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:19:53.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:19:53.534 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:20:23.763 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:20:23.773 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:20:53.962 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:20:53.973 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:21:24.174 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:21:24.184 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:21:54.380 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:21:54.391 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:22:24.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:22:24.620 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:22:54.817 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:22:54.830 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:23:25.039 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:23:25.049 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:23:55.261 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:23:55.271 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:24:25.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:24:25.452 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:24:55.593 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:24:55.603 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:25:25.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:25:25.747 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:25:55.951 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:25:55.961 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:26:26.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:26:26.108 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:26:56.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:26:56.339 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:27:26.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:27:26.496 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:27:56.724 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:27:56.733 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:28:27.046 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:28:27.056 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:28:57.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:28:57.224 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:29:27.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:29:27.380 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:29:57.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:29:57.596 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:30:27.792 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:30:27.802 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:30:58.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:30:58.041 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:31:28.245 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:31:28.255 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:31:58.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:31:58.485 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:32:28.631 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:32:28.641 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:32:58.845 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:32:58.855 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:33:29.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:33:29.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:33:59.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:33:59.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:34:29.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:34:29.437 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:34:59.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:34:59.595 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:35:29.808 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:35:29.819 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:36:00.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:36:00.042 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:36:30.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:36:30.248 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:37:00.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:37:00.421 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:37:30.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:37:30.626 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:38:00.847 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:38:00.857 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:38:31.077 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:38:31.088 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:39:01.300 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:39:01.310 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:39:31.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:39:31.522 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:40:01.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:40:01.846 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:40:32.049 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:40:32.060 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:41:02.263 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:41:02.273 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:41:32.486 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:41:32.497 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:42:02.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:42:02.641 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:42:32.839 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:42:32.849 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:43:03.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:43:03.088 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:43:33.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:43:33.318 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:44:03.530 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:44:03.539 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:44:33.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:44:33.712 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:45:03.914 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:45:03.925 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:45:34.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:45:34.157 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:46:04.385 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:46:04.395 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:46:34.591 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:46:34.601 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:47:04.755 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:47:04.765 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:47:34.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:47:34.962 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:48:05.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:48:05.176 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:48:35.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:48:35.409 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:49:05.549 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:49:05.559 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:49:35.740 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:49:35.750 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:50:05.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:50:05.965 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:50:36.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:50:36.189 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:51:06.390 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:51:06.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:51:36.597 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:51:36.607 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:52:06.744 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:52:06.754 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:52:36.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:52:36.978 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:53:07.188 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:53:07.198 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:53:37.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:53:37.411 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:54:07.548 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:54:07.558 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:54:37.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:54:37.797 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:55:07.934 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:55:07.945 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:55:38.190 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:55:38.200 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:56:08.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:56:08.430 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:56:38.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:56:38.636 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:57:08.832 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:57:08.842 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:57:15.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:15.733 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T17:57:15.807 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_03_00_UTC 2023-10-14T17:57:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_04_00_UTC 2023-10-14T17:57:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_05_00_UTC 2023-10-14T17:57:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_06_01_UTC 2023-10-14T17:57:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_07_00_UTC 2023-10-14T17:57:15.808 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_08_00_UTC 2023-10-14T17:57:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_09_00_UTC 2023-10-14T17:57:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_10_00_UTC 2023-10-14T17:57:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_11_00_UTC 2023-10-14T17:57:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_12_00_UTC 2023-10-14T17:57:15.809 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_13_00_UTC 2023-10-14T17:57:15.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_14_00_UTC 2023-10-14T17:57:15.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_15_00_UTC 2023-10-14T17:57:15.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_16_01_UTC 2023-10-14T17:57:15.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_17_00_UTC 2023-10-14T17:57:15.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_18_00_UTC 2023-10-14T17:57:15.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_19_00_UTC 2023-10-14T17:57:15.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_20_00_UTC 2023-10-14T17:57:15.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_21_01_UTC 2023-10-14T17:57:15.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_22_00_UTC 2023-10-14T17:57:15.811 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_23_00_UTC 2023-10-14T17:57:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_24_00_UTC 2023-10-14T17:57:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_25_00_UTC 2023-10-14T17:57:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_26_00_UTC 2023-10-14T17:57:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_27_00_UTC 2023-10-14T17:57:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_28_00_UTC 2023-10-14T17:57:15.813 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_29_00_UTC 2023-10-14T17:57:15.813 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_30_00_UTC 2023-10-14T17:57:15.813 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_31_00_UTC 2023-10-14T17:57:15.813 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_32_00_UTC 2023-10-14T17:57:15.813 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_33_00_UTC 2023-10-14T17:57:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_34_00_UTC 2023-10-14T17:57:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_35_00_UTC 2023-10-14T17:57:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_36_00_UTC 2023-10-14T17:57:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_37_00_UTC 2023-10-14T17:57:15.814 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_38_00_UTC 2023-10-14T17:57:15.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_39_00_UTC 2023-10-14T17:57:15.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_40_00_UTC 2023-10-14T17:57:15.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_41_00_UTC 2023-10-14T17:57:15.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_42_00_UTC 2023-10-14T17:57:15.815 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_43_00_UTC 2023-10-14T17:57:15.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_44_00_UTC 2023-10-14T17:57:15.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_45_00_UTC 2023-10-14T17:57:15.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_46_00_UTC 2023-10-14T17:57:15.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_47_00_UTC 2023-10-14T17:57:15.816 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_48_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_49_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_50_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_51_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_52_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_53_00_UTC 2023-10-14T17:57:15.817 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_54_00_UTC 2023-10-14T17:57:15.818 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_55_00_UTC 2023-10-14T17:57:15.818 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_56_00_UTC 2023-10-14T17:57:15.818 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_57_00_UTC 2023-10-14T17:57:15.819 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-17_03_00_UTC', 'scheduled-2023-10-14-17_04_00_UTC', 'scheduled-2023-10-14-17_05_00_UTC', 'scheduled-2023-10-14-17_06_01_UTC', 'scheduled-2023-10-14-17_07_00_UTC', 'scheduled-2023-10-14-17_08_00_UTC', 'scheduled-2023-10-14-17_09_00_UTC', 'scheduled-2023-10-14-17_10_00_UTC', 'scheduled-2023-10-14-17_11_00_UTC', 'scheduled-2023-10-14-17_12_00_UTC', 'scheduled-2023-10-14-17_13_00_UTC', 'scheduled-2023-10-14-17_14_00_UTC', 'scheduled-2023-10-14-17_15_00_UTC', 'scheduled-2023-10-14-17_16_01_UTC', 'scheduled-2023-10-14-17_17_00_UTC', 'scheduled-2023-10-14-17_18_00_UTC', 'scheduled-2023-10-14-17_19_00_UTC', 'scheduled-2023-10-14-17_20_00_UTC', 'scheduled-2023-10-14-17_21_01_UTC', 'scheduled-2023-10-14-17_22_00_UTC', 'scheduled-2023-10-14-17_23_00_UTC', 'scheduled-2023-10-14-17_24_00_UTC', 'scheduled-2023-10-14-17_25_00_UTC', 'scheduled-2023-10-14-17_26_00_UTC', 'scheduled-2023-10-14-17_27_00_UTC', 'scheduled-2023-10-14-17_28_00_UTC', 'scheduled-2023-10-14-17_29_00_UTC', 'scheduled-2023-10-14-17_30_00_UTC', 'scheduled-2023-10-14-17_31_00_UTC', 'scheduled-2023-10-14-17_32_00_UTC', 'scheduled-2023-10-14-17_33_00_UTC', 'scheduled-2023-10-14-17_34_00_UTC', 'scheduled-2023-10-14-17_35_00_UTC', 'scheduled-2023-10-14-17_36_00_UTC', 'scheduled-2023-10-14-17_37_00_UTC', 'scheduled-2023-10-14-17_38_00_UTC', 'scheduled-2023-10-14-17_39_00_UTC', 'scheduled-2023-10-14-17_40_00_UTC', 'scheduled-2023-10-14-17_41_00_UTC', 'scheduled-2023-10-14-17_42_00_UTC', 'scheduled-2023-10-14-17_43_00_UTC', 'scheduled-2023-10-14-17_44_00_UTC', 'scheduled-2023-10-14-17_45_00_UTC', 'scheduled-2023-10-14-17_46_00_UTC', 'scheduled-2023-10-14-17_47_00_UTC', 'scheduled-2023-10-14-17_48_00_UTC', 'scheduled-2023-10-14-17_49_00_UTC', 'scheduled-2023-10-14-17_50_00_UTC', 'scheduled-2023-10-14-17_51_00_UTC', 'scheduled-2023-10-14-17_52_00_UTC', 'scheduled-2023-10-14-17_53_00_UTC', 'scheduled-2023-10-14-17_54_00_UTC', 'scheduled-2023-10-14-17_55_00_UTC', 'scheduled-2023-10-14-17_56_00_UTC', 'scheduled-2023-10-14-17_57_00_UTC'] 2023-10-14T17:57:15.820 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1 2023-10-14T17:57:16.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.045+0000 7f30251cb640 1 Processor -- start 2023-10-14T17:57:16.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.045+0000 7f30251cb640 1 -- start start 2023-10-14T17:57:16.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f30251cb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f302006c040 0x7f302006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f30251cb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200cda80 0x7f302015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f30251cb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302006c960 con 0x7f30200ad970 2023-10-14T17:57:16.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f30251cb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f302015b9d0 con 0x7f302006c040 2023-10-14T17:57:16.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f30251cb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f302015bcb0 con 0x7f30200cda80 2023-10-14T17:57:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f301ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f302006c040 0x7f302006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f301ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f302006c040 0x7f302006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59170/0 (socket says 172.21.15.29:59170) 2023-10-14T17:57:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.047+0000 7f301ffff640 1 -- 172.21.15.29:0/2728091626 learned_addr learned my addr 172.21.15.29:0/2728091626 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.048+0000 7f30249ca640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200cda80 0x7f302015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.048+0000 7f301ffff640 1 -- 172.21.15.29:0/2728091626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200cda80 msgr2=0x7f302015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:16.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.048+0000 7f301ffff640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200cda80 0x7f302015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.048+0000 7f301ffff640 1 -- 172.21.15.29:0/2728091626 >> v1:172.21.15.186:6789/0 conn(0x7f30200ad970 legacy=0x7f302015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.048+0000 7f301ffff640 1 -- 172.21.15.29:0/2728091626 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302015bf90 con 0x7f302006c040 2023-10-14T17:57:16.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.049+0000 7f301ffff640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f302006c040 0x7f302006c420 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f3010009870 tx=0x7f301002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4315b595805fa65 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f301effd640 1 -- 172.21.15.29:0/2728091626 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010030840 con 0x7f302006c040 2023-10-14T17:57:16.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f301effd640 1 -- 172.21.15.29:0/2728091626 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30100309a0 con 0x7f302006c040 2023-10-14T17:57:16.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f301effd640 1 -- 172.21.15.29:0/2728091626 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3010030d30 con 0x7f302006c040 2023-10-14T17:57:16.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 -- 172.21.15.29:0/2728091626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f302006c040 msgr2=0x7f302006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f302006c040 0x7f302006c420 secure :-1 s=READY pgs=1851 cs=0 l=1 rev1=1 crypto rx=0x7f3010009870 tx=0x7f301002b980 comp rx=0 tx=0).stop 2023-10-14T17:57:16.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 -- 172.21.15.29:0/2728091626 shutdown_connections 2023-10-14T17:57:16.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200cda80 0x7f302015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 --2- 172.21.15.29:0/2728091626 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f302006c040 0x7f302006c420 unknown :-1 s=CLOSED pgs=1851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 -- 172.21.15.29:0/2728091626 >> 172.21.15.29:0/2728091626 conn(0x7f30200ce6a0 msgr2=0x7f30200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:16.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 -- 172.21.15.29:0/2728091626 shutdown_connections 2023-10-14T17:57:16.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.050+0000 7f30251cb640 1 -- 172.21.15.29:0/2728091626 wait complete. 2023-10-14T17:57:16.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.052+0000 7f30251cb640 1 Processor -- start 2023-10-14T17:57:16.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.053+0000 7f30251cb640 1 -- start start 2023-10-14T17:57:16.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f302006c040 0x7f302014d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200ad970 0x7f30200b06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f302014ddb0 con 0x7f30200cda80 2023-10-14T17:57:16.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30200aeff0 con 0x7f302006c040 2023-10-14T17:57:16.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30251cb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30200af2d0 con 0x7f30200ad970 2023-10-14T17:57:16.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59184/0 (socket says 172.21.15.29:59184) 2023-10-14T17:57:16.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 -- 172.21.15.29:0/3762530583 learned_addr learned my addr 172.21.15.29:0/3762530583 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:16.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f301ffff640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f302006c040 0x7f302014d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f301f7fe640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200ad970 0x7f30200b06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 -- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200ad970 msgr2=0x7f30200b06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200ad970 0x7f30200b06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 -- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f302006c040 msgr2=0x7f302014d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f302006c040 0x7f302014d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.054+0000 7f30249ca640 1 -- 172.21.15.29:0/3762530583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100094d0 con 0x7f30200cda80 2023-10-14T17:57:16.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f30249ca640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f3018007e60 tx=0x7f301800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018014420 con 0x7f30200cda80 2023-10-14T17:57:16.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30180145a0 con 0x7f30200cda80 2023-10-14T17:57:16.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30200af5b0 con 0x7f30200cda80 2023-10-14T17:57:16.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3018014950 con 0x7f30200cda80 2023-10-14T17:57:16.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.055+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30200afb70 con 0x7f30200cda80 2023-10-14T17:57:16.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.056+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f302006b330 con 0x7f30200cda80 2023-10-14T17:57:16.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.057+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f3018014ab0 con 0x7f30200cda80 2023-10-14T17:57:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.058+0000 7f301cff9640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 0x7f300c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.058+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f301808a2f0 con 0x7f30200cda80 2023-10-14T17:57:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.059+0000 7f301ffff640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 0x7f300c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.059+0000 7f301ffff640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 0x7f300c06d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3010036000 tx=0x7f3010008030 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.063+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3018055050 con 0x7f30200cda80 2023-10-14T17:57:16.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.281+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 --> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f30200cc550 con 0x7f300c06afe0 2023-10-14T17:57:16.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.302+0000 7f301cff9640 1 -- 172.21.15.29:0/3762530583 <== mgr.9255 v2:172.21.15.29:6832/2309931697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+373 (secure 0 0 0) 0x7f3010030b90 con 0x7f300c06afe0 2023-10-14T17:57:16.304 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:16.304 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:01:30", "first": "2023-10-14T17:02:00", "last": "2023-10-14T17:57:00", "last_pruned": "2023-10-14T17:57:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-14T17:57:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 msgr2=0x7f300c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 0x7f300c06d4a0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f3010036000 tx=0x7f3010008030 comp rx=0 tx=0).stop 2023-10-14T17:57:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 msgr2=0x7f30200ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 secure :-1 s=READY pgs=1852 cs=0 l=1 rev1=1 crypto rx=0x7f3018007e60 tx=0x7f301800e4b0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 shutdown_connections 2023-10-14T17:57:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30200cda80 0x7f30200ac890 unknown :-1 s=CLOSED pgs=1852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30200ad970 0x7f30200b06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f300c06afe0 0x7f300c06d4a0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 --2- 172.21.15.29:0/3762530583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f302006c040 0x7f302014d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 >> 172.21.15.29:0/3762530583 conn(0x7f30200ce6a0 msgr2=0x7f302015e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 shutdown_connections 2023-10-14T17:57:16.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.305+0000 7f30251cb640 1 -- 172.21.15.29:0/3762530583 wait complete. 2023-10-14T17:57:16.323 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1", "rel_path": "/snap_test_dir1", "schedule": "1m", "retention": {"n": 55}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:01:30", "first": "2023-10-14T17:02:00", "last": "2023-10-14T17:57:00", "last_pruned": "2023-10-14T17:57:00", "created_count": 56, "pruned_count": 1, "active": true}] 2023-10-14T17:57:16.324 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1', 'rel_path': '/snap_test_dir1', 'schedule': '1m', 'retention': {'n': 55}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T17:01:30', 'first': '2023-10-14T17:02:00', 'last': '2023-10-14T17:57:00', 'last_pruned': '2023-10-14T17:57:00', 'created_count': 56, 'pruned_count': 1, 'active': True} 2023-10-14T17:57:16.324 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 55 2023-10-14T17:57:16.325 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 56 2023-10-14T17:57:16.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1 2023-10-14T17:57:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.552+0000 7f1c670ff640 1 Processor -- start 2023-10-14T17:57:16.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.552+0000 7f1c670ff640 1 -- start start 2023-10-14T17:57:16.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c670ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c600cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c670ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600ca560 con 0x7f1c6006c9e0 2023-10-14T17:57:16.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c670ff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c600c9950 con 0x7f1c600c9060 2023-10-14T17:57:16.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c670ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c9c30 con 0x7f1c6006fb50 2023-10-14T17:57:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c668fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1c6006c9e0 0x7f1c600ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:49160/0 (socket says 172.21.15.29:49160) 2023-10-14T17:57:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.554+0000 7f1c668fe640 1 -- 172.21.15.29:0/2621622483 learned_addr learned my addr 172.21.15.29:0/2621622483 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c658fc640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c600cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720625801 0 0) 0x7f1c600ca560 con 0x7f1c6006c9e0 2023-10-14T17:57:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c4c003520 con 0x7f1c6006c9e0 2023-10-14T17:57:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2872990924 0 0) 0x7f1c4c003520 con 0x7f1c6006c9e0 2023-10-14T17:57:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 msgr2=0x7f1c600cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c600cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 >> v1:172.21.15.186:6789/0 conn(0x7f1c600c9060 legacy=0x7f1c6006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c600c9f10 con 0x7f1c6006c9e0 2023-10-14T17:57:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1c58001670 con 0x7f1c6006c9e0 2023-10-14T17:57:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 >> v1:172.21.15.29:6789/0 conn(0x7f1c6006c9e0 legacy=0x7f1c600ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.555+0000 7f1c650fb640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c4c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c9c30 con 0x7f1c4c0047c0 2023-10-14T17:57:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c00ba60 con 0x7f1c4c0039a0 2023-10-14T17:57:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c4c00bc20 con 0x7f1c6006fb50 2023-10-14T17:57:16.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c660fd640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c668fe640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c4c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 -- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 msgr2=0x7f1c4c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c6006fb50 0x7f1c4c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 -- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c4c0039a0 msgr2=0x7f1c4c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c4c0039a0 0x7f1c4c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.556+0000 7f1c658fc640 1 -- 172.21.15.29:0/2621622483 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c4c003520 con 0x7f1c4c0047c0 2023-10-14T17:57:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.557+0000 7f1c658fc640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009a90 tx=0x7f1c4c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cff45baa27a4f75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5002c680 con 0x7f1c4c0047c0 2023-10-14T17:57:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c5002c7e0 con 0x7f1c4c0047c0 2023-10-14T17:57:16.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c650fb640 1 -- 172.21.15.29:0/2621622483 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c5002cb70 con 0x7f1c4c0047c0 2023-10-14T17:57:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 -- 172.21.15.29:0/2621622483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0047c0 msgr2=0x7f1c4c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 --2- 172.21.15.29:0/2621622483 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c4c0047c0 0x7f1c4c004c40 secure :-1 s=READY pgs=1854 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009a90 tx=0x7f1c4c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c668fe640 1 -- 172.21.15.29:0/2621622483 reap_dead start 2023-10-14T17:57:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 -- 172.21.15.29:0/2621622483 shutdown_connections 2023-10-14T17:57:16.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 -- 172.21.15.29:0/2621622483 >> 172.21.15.29:0/2621622483 conn(0x7f1c600ce6a0 msgr2=0x7f1c600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 -- 172.21.15.29:0/2621622483 shutdown_connections 2023-10-14T17:57:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.558+0000 7f1c670ff640 1 -- 172.21.15.29:0/2621622483 wait complete. 2023-10-14T17:57:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.560+0000 7f1c670ff640 1 Processor -- start 2023-10-14T17:57:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.560+0000 7f1c670ff640 1 -- start start 2023-10-14T17:57:16.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c4c0047c0 0x7f1c60152630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 0x7f1c60162b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c3210 con 0x7f1c6006a7d0 2023-10-14T17:57:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c600c81e0 con 0x7f1c601fa9b0 2023-10-14T17:57:16.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.561+0000 7f1c670ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c601fadf0 con 0x7f1c4c0047c0 2023-10-14T17:57:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59208/0 (socket says 172.21.15.29:59208) 2023-10-14T17:57:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 -- 172.21.15.29:0/874726120 learned_addr learned my addr 172.21.15.29:0/874726120 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:16.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c660fd640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c4c0047c0 0x7f1c60152630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c658fc640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 0x7f1c60162b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 -- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c4c0047c0 msgr2=0x7f1c60152630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c4c0047c0 0x7f1c60152630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 -- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 msgr2=0x7f1c60162b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:16.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 0x7f1c60162b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c668fe640 1 -- 172.21.15.29:0/874726120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c60152c10 con 0x7f1c6006a7d0 2023-10-14T17:57:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.562+0000 7f1c658fc640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 0x7f1c60162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c668fe640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f1c600cf7a0 tx=0x7f1c5800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c58007440 con 0x7f1c6006a7d0 2023-10-14T17:57:16.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c580075e0 con 0x7f1c6006a7d0 2023-10-14T17:57:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c60152ef0 con 0x7f1c6006a7d0 2023-10-14T17:57:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c58004a20 con 0x7f1c6006a7d0 2023-10-14T17:57:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.563+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c601534b0 con 0x7f1c6006a7d0 2023-10-14T17:57:16.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.564+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6004e9b0 con 0x7f1c6006a7d0 2023-10-14T17:57:16.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.565+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f1c58004b80 con 0x7f1c6006a7d0 2023-10-14T17:57:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.566+0000 7f1c56ffd640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 0x7f1c4406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.566+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(158..158 src has 1..158) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f1c58089800 con 0x7f1c6006a7d0 2023-10-14T17:57:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.567+0000 7f1c660fd640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 0x7f1c4406d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.567+0000 7f1c660fd640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 0x7f1c4406d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c007fd0 tx=0x7f1c5c009040 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:16.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.571+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c580543c0 con 0x7f1c6006a7d0 2023-10-14T17:57:16.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:16.799+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 --> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f1c600c1f10 con 0x7f1c4406afe0 2023-10-14T17:57:17.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.027+0000 7f1c56ffd640 1 -- 172.21.15.29:0/874726120 <== mgr.9255 v2:172.21.15.29:6832/2309931697 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+42 (secure 0 0 0) 0x7f1c5c005780 con 0x7f1c4406afe0 2023-10-14T17:57:17.029 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1 2023-10-14T17:57:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 msgr2=0x7f1c4406d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 0x7f1c4406d4a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c007fd0 tx=0x7f1c5c009040 comp rx=0 tx=0).stop 2023-10-14T17:57:17.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 msgr2=0x7f1c600c3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 secure :-1 s=READY pgs=1855 cs=0 l=1 rev1=1 crypto rx=0x7f1c600cf7a0 tx=0x7f1c5800d9d0 comp rx=0 tx=0).stop 2023-10-14T17:57:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 shutdown_connections 2023-10-14T17:57:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c6006a7d0 0x7f1c600c3630 unknown :-1 s=CLOSED pgs=1855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:17.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f1c4406afe0 0x7f1c4406d4a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c601fa9b0 0x7f1c60162b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 --2- 172.21.15.29:0/874726120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c4c0047c0 0x7f1c60152630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:17.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 >> 172.21.15.29:0/874726120 conn(0x7f1c600ce6a0 msgr2=0x7f1c600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.031+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 shutdown_connections 2023-10-14T17:57:17.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:17.032+0000 7f1c670ff640 1 -- 172.21.15.29:0/874726120 wait complete. 2023-10-14T17:57:17.055 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1 2023-10-14T17:57:17.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.056 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T17:57:17.118 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_03_00_UTC 2023-10-14T17:57:17.118 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_04_00_UTC 2023-10-14T17:57:17.118 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_05_00_UTC 2023-10-14T17:57:17.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_06_01_UTC 2023-10-14T17:57:17.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_07_00_UTC 2023-10-14T17:57:17.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_08_00_UTC 2023-10-14T17:57:17.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_09_00_UTC 2023-10-14T17:57:17.119 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_10_00_UTC 2023-10-14T17:57:17.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_11_00_UTC 2023-10-14T17:57:17.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_12_00_UTC 2023-10-14T17:57:17.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_13_00_UTC 2023-10-14T17:57:17.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_14_00_UTC 2023-10-14T17:57:17.120 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_15_00_UTC 2023-10-14T17:57:17.121 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_16_01_UTC 2023-10-14T17:57:17.121 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_17_00_UTC 2023-10-14T17:57:17.121 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_18_00_UTC 2023-10-14T17:57:17.121 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_19_00_UTC 2023-10-14T17:57:17.121 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_20_00_UTC 2023-10-14T17:57:17.122 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_21_01_UTC 2023-10-14T17:57:17.122 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_22_00_UTC 2023-10-14T17:57:17.122 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_23_00_UTC 2023-10-14T17:57:17.122 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_24_00_UTC 2023-10-14T17:57:17.123 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_25_00_UTC 2023-10-14T17:57:17.123 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_26_00_UTC 2023-10-14T17:57:17.123 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_27_00_UTC 2023-10-14T17:57:17.123 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_28_00_UTC 2023-10-14T17:57:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_29_00_UTC 2023-10-14T17:57:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_30_00_UTC 2023-10-14T17:57:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_31_00_UTC 2023-10-14T17:57:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_32_00_UTC 2023-10-14T17:57:17.124 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_33_00_UTC 2023-10-14T17:57:17.125 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_34_00_UTC 2023-10-14T17:57:17.125 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_35_00_UTC 2023-10-14T17:57:17.125 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_36_00_UTC 2023-10-14T17:57:17.125 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_37_00_UTC 2023-10-14T17:57:17.126 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_38_00_UTC 2023-10-14T17:57:17.126 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_39_00_UTC 2023-10-14T17:57:17.126 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_40_00_UTC 2023-10-14T17:57:17.126 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_41_00_UTC 2023-10-14T17:57:17.126 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_42_00_UTC 2023-10-14T17:57:17.127 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_43_00_UTC 2023-10-14T17:57:17.127 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_44_00_UTC 2023-10-14T17:57:17.127 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_45_00_UTC 2023-10-14T17:57:17.127 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_46_00_UTC 2023-10-14T17:57:17.127 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_47_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_48_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_49_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_50_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_51_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_52_00_UTC 2023-10-14T17:57:17.128 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_53_00_UTC 2023-10-14T17:57:17.129 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_54_00_UTC 2023-10-14T17:57:17.129 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_55_00_UTC 2023-10-14T17:57:17.129 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_56_00_UTC 2023-10-14T17:57:17.129 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_57_00_UTC 2023-10-14T17:57:17.130 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_03_00_UTC 2023-10-14T17:57:17.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.132 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_03_00_UTC) 2023-10-14T17:57:17.210 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_04_00_UTC 2023-10-14T17:57:17.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.211 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_04_00_UTC) 2023-10-14T17:57:17.290 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_05_00_UTC 2023-10-14T17:57:17.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.291 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_05_00_UTC) 2023-10-14T17:57:17.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_06_01_UTC 2023-10-14T17:57:17.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.372 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_06_01_UTC) 2023-10-14T17:57:17.448 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_07_00_UTC 2023-10-14T17:57:17.448 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.449 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_07_00_UTC) 2023-10-14T17:57:17.527 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_08_00_UTC 2023-10-14T17:57:17.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.528 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_08_00_UTC) 2023-10-14T17:57:17.601 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_09_00_UTC 2023-10-14T17:57:17.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.602 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_09_00_UTC) 2023-10-14T17:57:17.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_10_00_UTC 2023-10-14T17:57:17.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.669 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_10_00_UTC) 2023-10-14T17:57:17.746 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_11_00_UTC 2023-10-14T17:57:17.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.747 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_11_00_UTC) 2023-10-14T17:57:17.823 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_12_00_UTC 2023-10-14T17:57:17.824 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.824 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_12_00_UTC) 2023-10-14T17:57:17.896 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_13_00_UTC 2023-10-14T17:57:17.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.897 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_13_00_UTC) 2023-10-14T17:57:17.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_14_00_UTC 2023-10-14T17:57:17.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:17.976 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_14_00_UTC) 2023-10-14T17:57:18.055 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_15_00_UTC 2023-10-14T17:57:18.056 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.056 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_15_00_UTC) 2023-10-14T17:57:18.136 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_16_01_UTC 2023-10-14T17:57:18.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.137 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_16_01_UTC) 2023-10-14T17:57:18.217 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_17_00_UTC 2023-10-14T17:57:18.218 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.218 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_17_00_UTC) 2023-10-14T17:57:18.292 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_18_00_UTC 2023-10-14T17:57:18.292 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.293 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_18_00_UTC) 2023-10-14T17:57:18.371 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_19_00_UTC 2023-10-14T17:57:18.372 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.372 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_19_00_UTC) 2023-10-14T17:57:18.447 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_20_00_UTC 2023-10-14T17:57:18.447 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.447 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_20_00_UTC) 2023-10-14T17:57:18.523 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_21_01_UTC 2023-10-14T17:57:18.524 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.524 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_21_01_UTC) 2023-10-14T17:57:18.600 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_22_00_UTC 2023-10-14T17:57:18.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.601 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_22_00_UTC) 2023-10-14T17:57:18.679 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_23_00_UTC 2023-10-14T17:57:18.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.680 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_23_00_UTC) 2023-10-14T17:57:18.776 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_24_00_UTC 2023-10-14T17:57:18.776 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.776 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_24_00_UTC) 2023-10-14T17:57:18.845 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_25_00_UTC 2023-10-14T17:57:18.846 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.846 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_25_00_UTC) 2023-10-14T17:57:18.913 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_26_00_UTC 2023-10-14T17:57:18.913 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.914 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_26_00_UTC) 2023-10-14T17:57:18.972 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_27_00_UTC 2023-10-14T17:57:18.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:18.974 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_27_00_UTC) 2023-10-14T17:57:19.029 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_28_00_UTC 2023-10-14T17:57:19.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.030 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_28_00_UTC) 2023-10-14T17:57:19.090 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_29_00_UTC 2023-10-14T17:57:19.091 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.091 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_29_00_UTC) 2023-10-14T17:57:19.160 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_30_00_UTC 2023-10-14T17:57:19.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.162 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_30_00_UTC) 2023-10-14T17:57:19.239 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_31_00_UTC 2023-10-14T17:57:19.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.240 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_31_00_UTC) 2023-10-14T17:57:19.338 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_32_00_UTC 2023-10-14T17:57:19.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.339 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_32_00_UTC) 2023-10-14T17:57:19.429 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_33_00_UTC 2023-10-14T17:57:19.430 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.430 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_33_00_UTC) 2023-10-14T17:57:19.483 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_34_00_UTC 2023-10-14T17:57:19.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.484 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_34_00_UTC) 2023-10-14T17:57:19.539 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_35_00_UTC 2023-10-14T17:57:19.539 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.540 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_35_00_UTC) 2023-10-14T17:57:19.601 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_36_00_UTC 2023-10-14T17:57:19.602 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.603 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_36_00_UTC) 2023-10-14T17:57:19.667 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_37_00_UTC 2023-10-14T17:57:19.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.667 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_37_00_UTC) 2023-10-14T17:57:19.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_38_00_UTC 2023-10-14T17:57:19.756 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.756 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_38_00_UTC) 2023-10-14T17:57:19.831 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_39_00_UTC 2023-10-14T17:57:19.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.832 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_39_00_UTC) 2023-10-14T17:57:19.907 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_40_00_UTC 2023-10-14T17:57:19.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.907 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_40_00_UTC) 2023-10-14T17:57:19.962 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_41_00_UTC 2023-10-14T17:57:19.963 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:19.963 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_41_00_UTC) 2023-10-14T17:57:20.024 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_42_00_UTC 2023-10-14T17:57:20.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.025 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_42_00_UTC) 2023-10-14T17:57:20.081 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_43_00_UTC 2023-10-14T17:57:20.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.082 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_43_00_UTC) 2023-10-14T17:57:20.139 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_44_00_UTC 2023-10-14T17:57:20.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.140 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_44_00_UTC) 2023-10-14T17:57:20.202 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_45_00_UTC 2023-10-14T17:57:20.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.203 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_45_00_UTC) 2023-10-14T17:57:20.266 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_46_00_UTC 2023-10-14T17:57:20.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.268 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_46_00_UTC) 2023-10-14T17:57:20.328 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_47_00_UTC 2023-10-14T17:57:20.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.329 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_47_00_UTC) 2023-10-14T17:57:20.391 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_48_00_UTC 2023-10-14T17:57:20.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.392 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_48_00_UTC) 2023-10-14T17:57:20.458 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_49_00_UTC 2023-10-14T17:57:20.458 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.459 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_49_00_UTC) 2023-10-14T17:57:20.526 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_50_00_UTC 2023-10-14T17:57:20.527 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.528 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_50_00_UTC) 2023-10-14T17:57:20.598 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_51_00_UTC 2023-10-14T17:57:20.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.599 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_51_00_UTC) 2023-10-14T17:57:20.668 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_52_00_UTC 2023-10-14T17:57:20.669 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.669 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_52_00_UTC) 2023-10-14T17:57:20.756 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_53_00_UTC 2023-10-14T17:57:20.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.757 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_53_00_UTC) 2023-10-14T17:57:20.852 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_54_00_UTC 2023-10-14T17:57:20.852 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.853 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_54_00_UTC) 2023-10-14T17:57:20.921 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_55_00_UTC 2023-10-14T17:57:20.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.922 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_55_00_UTC) 2023-10-14T17:57:20.968 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_56_00_UTC 2023-10-14T17:57:20.968 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:20.968 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_56_00_UTC) 2023-10-14T17:57:21.067 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-17_57_00_UTC 2023-10-14T17:57:21.068 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:21.068 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-17_57_00_UTC) 2023-10-14T17:57:21.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:21.125 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T17:57:21.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T17:57:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.391+0000 7fc5e3fff640 1 Processor -- start 2023-10-14T17:57:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.391+0000 7fc5e3fff640 1 -- start start 2023-10-14T17:57:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc5e4070b60 0x7fc5e415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e3fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e415aef0 con 0x7fc5e4159950 2023-10-14T17:57:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e3fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5e415d910 con 0x7fc5e406c100 2023-10-14T17:57:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e3fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5e415e490 con 0x7fc5e4070b60 2023-10-14T17:57:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc5e4159950 0x7fc5e4070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34908/0 (socket says 172.21.15.29:34908) 2023-10-14T17:57:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.393+0000 7fc5e2ffd640 1 -- 172.21.15.29:0/670430763 learned_addr learned my addr 172.21.15.29:0/670430763 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1072626922 0 0) 0x7fc5d4001490 con 0x7fc5e4159950 2023-10-14T17:57:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5d0005730 con 0x7fc5e4159950 2023-10-14T17:57:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 380605776 0 0) 0x7fc5d0005730 con 0x7fc5e4159950 2023-10-14T17:57:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 >> v2:172.21.15.186:3300/0 conn(0x7fc5e4070b60 msgr2=0x7fc5e415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 --2- 172.21.15.29:0/670430763 >> v2:172.21.15.186:3300/0 conn(0x7fc5e4070b60 0x7fc5e415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 >> v1:172.21.15.186:6789/0 conn(0x7fc5e406c100 legacy=0x7fc5e406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e415dbf0 con 0x7fc5e4159950 2023-10-14T17:57:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc5d4001b80 con 0x7fc5e4159950 2023-10-14T17:57:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 >> v1:172.21.15.29:6789/0 conn(0x7fc5e4159950 legacy=0x7fc5e4070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d0005b80 0x7fc5d0005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 0x7fc5d0006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5d0007590 0x7fc5d000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e415e490 con 0x7fc5d00069d0 2023-10-14T17:57:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d000e430 con 0x7fc5d0007590 2023-10-14T17:57:21.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.394+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d000e5f0 con 0x7fc5d0005b80 2023-10-14T17:57:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 0x7fc5d0006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5e2ffd640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d0005b80 0x7fc5d0005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5e37fe640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5d0007590 0x7fc5d000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 -- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d0005b80 msgr2=0x7fc5d0005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d0005b80 0x7fc5d0005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 -- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5d0007590 msgr2=0x7fc5d000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5d0007590 0x7fc5d000db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5da7fc640 1 -- 172.21.15.29:0/670430763 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d0005730 con 0x7fc5d00069d0 2023-10-14T17:57:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5e37fe640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5d0007590 0x7fc5d000db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5e37fe640 1 -- 172.21.15.29:0/670430763 reap_dead start 2023-10-14T17:57:21.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.395+0000 7fc5e2ffd640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d0005b80 0x7fc5d0005f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:21.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.396+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 0x7fc5d0006e50 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009860 tx=0x7fc5d0007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7156c9c22c565494 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc032600 con 0x7fc5d00069d0 2023-10-14T17:57:21.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc02e050 con 0x7fc5d00069d0 2023-10-14T17:57:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e27fc640 1 -- 172.21.15.29:0/670430763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc0329c0 con 0x7fc5d00069d0 2023-10-14T17:57:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 -- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 msgr2=0x7fc5d0006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 0x7fc5d0006e50 secure :-1 s=READY pgs=1857 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc009860 tx=0x7fc5d0007a00 comp rx=0 tx=0).stop 2023-10-14T17:57:21.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 -- 172.21.15.29:0/670430763 shutdown_connections 2023-10-14T17:57:21.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/670430763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5d00069d0 0x7fc5d0006e50 unknown :-1 s=CLOSED pgs=1857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 -- 172.21.15.29:0/670430763 >> 172.21.15.29:0/670430763 conn(0x7fc5e40ce6a0 msgr2=0x7fc5e40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 -- 172.21.15.29:0/670430763 shutdown_connections 2023-10-14T17:57:21.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.397+0000 7fc5e3fff640 1 -- 172.21.15.29:0/670430763 wait complete. 2023-10-14T17:57:21.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.399+0000 7fc5e3fff640 1 Processor -- start 2023-10-14T17:57:21.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.399+0000 7fc5e3fff640 1 -- start start 2023-10-14T17:57:21.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d00069d0 0x7fc5e4154fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5e40ac890 0x7fc5e40acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40ad250 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e40c21b0 con 0x7fc5e40ac890 2023-10-14T17:57:21.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.400+0000 7fc5e3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5e415b130 con 0x7fc5d00069d0 2023-10-14T17:57:21.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39082/0 (socket says 172.21.15.29:39082) 2023-10-14T17:57:21.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 -- 172.21.15.29:0/375522558 learned_addr learned my addr 172.21.15.29:0/375522558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:21.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5e2ffd640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d00069d0 0x7fc5e4154fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 -- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d00069d0 msgr2=0x7fc5e4154fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d00069d0 0x7fc5e4154fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 -- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5e40ac890 msgr2=0x7fc5e40acd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5e40ac890 0x7fc5e40acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.401+0000 7fc5da7fc640 1 -- 172.21.15.29:0/375522558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5e415b410 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5da7fc640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc000c00 tx=0x7fc5cc0047d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:21.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc03c940 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5cc03caa0 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5cc03ce10 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5e415b6f0 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.402+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5e415bcb0 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.403+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5e404e9b0 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.404+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 88) v1 ==== 89360+0+0 (secure 0 0 0) 0x7fc5cc039040 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.405+0000 7fc5e0ff9640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 0x7fc5c406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:21.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.405+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(162..162 src has 1..162) v4 ==== 6271+0+0 (secure 0 0 0) 0x7fc5cc0ab390 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.405+0000 7fc5e2ffd640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 0x7fc5c406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.406+0000 7fc5e2ffd640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 0x7fc5c406d4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5e41f5830 tx=0x7fc5d4009040 comp rx=0 tx=0).ready entity=mgr.9255 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:21.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.410+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5cc079a70 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.658+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc5e40ad600 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.739+0000 7fc5e0ff9640 1 -- 172.21.15.29:0/375522558 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v89) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc5cc079600 con 0x7fc5e40b3bb0 2023-10-14T17:57:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.745+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 msgr2=0x7fc5c406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.745+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 0x7fc5c406d4d0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5e41f5830 tx=0x7fc5d4009040 comp rx=0 tx=0).stop 2023-10-14T17:57:21.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.745+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 msgr2=0x7fc5e406b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:21.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.745+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 secure :-1 s=READY pgs=1858 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc000c00 tx=0x7fc5cc0047d0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 shutdown_connections 2023-10-14T17:57:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fc5c406b010 0x7fc5c406d4d0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5e40ac890 0x7fc5e40acd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5e40b3bb0 0x7fc5e406b2f0 unknown :-1 s=CLOSED pgs=1858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/375522558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5d00069d0 0x7fc5e4154fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:21.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.746+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 >> 172.21.15.29:0/375522558 conn(0x7fc5e40ce6a0 msgr2=0x7fc5e4070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:21.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.747+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 shutdown_connections 2023-10-14T17:57:21.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:21.747+0000 7fc5e3fff640 1 -- 172.21.15.29:0/375522558 wait complete. 2023-10-14T17:57:21.784 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:57:21.784 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:57:21.891 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:57:21.983 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:57:21.983 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:21.984 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:22.028 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:22.028 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:22.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:22.343 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:22.341+0000 7f8af2e13040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:57:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.362+0000 7fd0b6eb4640 1 Processor -- start 2023-10-14T17:57:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.362+0000 7fd0b6eb4640 1 -- start start 2023-10-14T17:57:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.363+0000 7fd0b6eb4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd0b00b0b20 0x7fd0b0164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.363+0000 7fd0b6eb4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b006f7f0 con 0x7fd0b0159b20 2023-10-14T17:57:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.363+0000 7fd0b6eb4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b006c5e0 con 0x7fd0b0159f00 2023-10-14T17:57:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.363+0000 7fd0b6eb4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b006c8c0 con 0x7fd0b00b0b20 2023-10-14T17:57:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0b4c50640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0b0159b20 0x7fd0b006bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34918/0 (socket says 172.21.15.29:34918) 2023-10-14T17:57:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0b4c50640 1 -- 172.21.15.29:0/2616437448 learned_addr learned my addr 172.21.15.29:0/2616437448 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:22.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1917073535 0 0) 0x7fd0b006f7f0 con 0x7fd0b0159b20 2023-10-14T17:57:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0a0003520 con 0x7fd0b0159b20 2023-10-14T17:57:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0b5451640 1 --2- 172.21.15.29:0/2616437448 >> v2:172.21.15.186:3300/0 conn(0x7fd0b00b0b20 0x7fd0b0164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 699498154 0 0) 0x7fd0a0003520 con 0x7fd0b0159b20 2023-10-14T17:57:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 >> v2:172.21.15.186:3300/0 conn(0x7fd0b00b0b20 msgr2=0x7fd0b0164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 --2- 172.21.15.29:0/2616437448 >> v2:172.21.15.186:3300/0 conn(0x7fd0b00b0b20 0x7fd0b0164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 >> v1:172.21.15.186:6789/0 conn(0x7fd0b0159f00 legacy=0x7fd0b00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b006cba0 con 0x7fd0b0159b20 2023-10-14T17:57:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd0a4001670 con 0x7fd0b0159b20 2023-10-14T17:57:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.364+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 >> v1:172.21.15.29:6789/0 conn(0x7fd0b0159b20 legacy=0x7fd0b006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00039c0 0x7fd0a0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0a00047c0 0x7fd0a0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 0x7fd0a000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b006c8c0 con 0x7fd0a00051d0 2023-10-14T17:57:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a000e330 con 0x7fd0a00039c0 2023-10-14T17:57:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0a000e4f0 con 0x7fd0a00047c0 2023-10-14T17:57:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0b5451640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 0x7fd0a000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0b4c50640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00039c0 0x7fd0a0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.365+0000 7fd0affff640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0a00047c0 0x7fd0a0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0b5451640 1 -- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0a00047c0 msgr2=0x7fd0a0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0b5451640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0a00047c0 0x7fd0a0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0b5451640 1 -- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00039c0 msgr2=0x7fd0a0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0b5451640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00039c0 0x7fd0a0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0b5451640 1 -- 172.21.15.29:0/2616437448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0a0003520 con 0x7fd0a00051d0 2023-10-14T17:57:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.366+0000 7fd0affff640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0a00047c0 0x7fd0a0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.367+0000 7fd0b5451640 1 -- 172.21.15.29:0/2616437448 reap_dead start 2023-10-14T17:57:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.367+0000 7fd0b5451640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 0x7fd0a000db90 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8007360 tx=0x7fd0a00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a655582ac0cbf397 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a802ccf0 con 0x7fd0a00051d0 2023-10-14T17:57:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a802ce90 con 0x7fd0a00051d0 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0af7fe640 1 -- 172.21.15.29:0/2616437448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a8003bd0 con 0x7fd0a00051d0 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 msgr2=0x7fd0a000db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 0x7fd0a000db90 secure :-1 s=READY pgs=1861 cs=0 l=1 rev1=1 crypto rx=0x7fd0a8007360 tx=0x7fd0a00057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2616437448 shutdown_connections 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2616437448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0a00051d0 0x7fd0a000db90 unknown :-1 s=CLOSED pgs=1861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2616437448 >> 172.21.15.29:0/2616437448 conn(0x7fd0b00cb410 msgr2=0x7fd0b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:22.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2616437448 shutdown_connections 2023-10-14T17:57:22.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.368+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2616437448 wait complete. 2023-10-14T17:57:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.370+0000 7fd0b6eb4640 1 Processor -- start 2023-10-14T17:57:22.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.370+0000 7fd0b6eb4640 1 -- start start 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00051d0 0x7fd0b00c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0b00c5a60 0x7fd0b00c5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00ad250 con 0x7fd0b00ac890 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00c6450 con 0x7fd0a00051d0 2023-10-14T17:57:22.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.371+0000 7fd0b6eb4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b00b1ab0 con 0x7fd0b00c5a60 2023-10-14T17:57:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39124/0 (socket says 172.21.15.29:39124) 2023-10-14T17:57:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 -- 172.21.15.29:0/2932514683 learned_addr learned my addr 172.21.15.29:0/2932514683 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0b00c5a60 msgr2=0x7fd0b00c5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0b00c5a60 0x7fd0b00c5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00051d0 msgr2=0x7fd0b00c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00051d0 0x7fd0b00c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 -- 172.21.15.29:0/2932514683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b00b1d90 con 0x7fd0b00ac890 2023-10-14T17:57:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0affff640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00e4b0 tx=0x7fd09c00e970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:22.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c0052e0 con 0x7fd0b00ac890 2023-10-14T17:57:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.372+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09c005440 con 0x7fd0b00ac890 2023-10-14T17:57:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.373+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd09c003830 con 0x7fd0b00ac890 2023-10-14T17:57:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.373+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b00b2070 con 0x7fd0b00ac890 2023-10-14T17:57:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.373+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b00b2630 con 0x7fd0b00ac890 2023-10-14T17:57:22.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.374+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b00b27c0 con 0x7fd0b00ac890 2023-10-14T17:57:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.375+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd09c003990 con 0x7fd0b00ac890 2023-10-14T17:57:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.376+0000 7fd0ad7fa640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 0x7fd09406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.376+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(163..163 src has 1..163) v4 ==== 6271+0+0 (secure 0 0 0) 0x7fd09c089700 con 0x7fd0b00ac890 2023-10-14T17:57:22.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.376+0000 7fd0b4c50640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 msgr2=0x7fd09406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.376+0000 7fd0b4c50640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 0x7fd09406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:22.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.380+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd09c054530 con 0x7fd0b00ac890 2023-10-14T17:57:22.387 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:22.386+0000 7f473bada040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:57:22.398 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:22.396+0000 7f8af2e13040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:57:22.444 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:22.443+0000 7f473bada040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:57:22.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.577+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd0b004e160 con 0x7fd0b00ac890 2023-10-14T17:57:22.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.577+0000 7fd0b4c50640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 msgr2=0x7fd09406d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.577+0000 7fd0b4c50640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 0x7fd09406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:22.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.578+0000 7fd0ad7fa640 1 -- 172.21.15.29:0/2932514683 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v163) v1 ==== 74+0+17389 (secure 0 0 0) 0x7fd09c057b50 con 0x7fd0b00ac890 2023-10-14T17:57:22.580 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:22.581 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":163,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:21.709428+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"163","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":106,"snap_epoch":163,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.33","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3251497707":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/80005070":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/2913425840":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/3470978219":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/1610794407":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6833/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6832/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/542195718":"2023-10-15T17:01:29.273525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":48},{"begin":59,"length":48}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":36,"length":15},{"begin":92,"length":15}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 msgr2=0x7fd09406d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 0x7fd09406d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 msgr2=0x7fd0b00acd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 secure :-1 s=READY pgs=1862 cs=0 l=1 rev1=1 crypto rx=0x7fd09c00e4b0 tx=0x7fd09c00e970 comp rx=0 tx=0).stop 2023-10-14T17:57:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 shutdown_connections 2023-10-14T17:57:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd0b00c5a60 0x7fd0b00c5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0b00ac890 0x7fd0b00acd10 unknown :-1 s=CLOSED pgs=1862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fd09406afc0 0x7fd09406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 --2- 172.21.15.29:0/2932514683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0a00051d0 0x7fd0b00c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.581+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 >> 172.21.15.29:0/2932514683 conn(0x7fd0b00cb410 msgr2=0x7fd0b0157e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:22.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.582+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 shutdown_connections 2023-10-14T17:57:22.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.582+0000 7fd0b6eb4640 1 -- 172.21.15.29:0/2932514683 wait complete. 2023-10-14T17:57:22.601 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:01:12.106201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:01:13.162539+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '163', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 106, 'snap_epoch': 163, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:57:22.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:57:22.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.824+0000 7f2399b0f640 1 Processor -- start 2023-10-14T17:57:22.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.824+0000 7f2399b0f640 1 -- start start 2023-10-14T17:57:22.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2399b0f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415d410 0x7f23940d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2399b0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2399b0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23940b5d90 con 0x7f239415d030 2023-10-14T17:57:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2399b0f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23940d2b50 con 0x7f239415d410 2023-10-14T17:57:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2399b0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940cdbf0 con 0x7f23940cd260 2023-10-14T17:57:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415d410 0x7f23940d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415d410 0x7f23940d1630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39128/0 (socket says 172.21.15.29:39128) 2023-10-14T17:57:22.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 -- 172.21.15.29:0/700195398 learned_addr learned my addr 172.21.15.29:0/700195398 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f2393fff640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 -- 172.21.15.29:0/700195398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 msgr2=0x7f23940cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940cd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 -- 172.21.15.29:0/700195398 >> v1:172.21.15.186:6789/0 conn(0x7f239415d030 legacy=0x7f2394161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:22.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.826+0000 7f238affd640 1 -- 172.21.15.29:0/700195398 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2394160fc0 con 0x7f239415d410 2023-10-14T17:57:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f238affd640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f23940d1630 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2394160cd0 tx=0x7f237c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94ff33ee295a312c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/700195398 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c030610 con 0x7f239415d410 2023-10-14T17:57:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/700195398 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c030770 con 0x7f239415d410 2023-10-14T17:57:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/700195398 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c030b00 con 0x7f239415d410 2023-10-14T17:57:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f2399b0f640 1 -- 172.21.15.29:0/700195398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 msgr2=0x7f23940d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.828+0000 7f2399b0f640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f23940d1630 secure :-1 s=READY pgs=1863 cs=0 l=1 rev1=1 crypto rx=0x7f2394160cd0 tx=0x7f237c02b960 comp rx=0 tx=0).stop 2023-10-14T17:57:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 -- 172.21.15.29:0/700195398 shutdown_connections 2023-10-14T17:57:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 --2- 172.21.15.29:0/700195398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f23940d1630 unknown :-1 s=CLOSED pgs=1863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 -- 172.21.15.29:0/700195398 >> 172.21.15.29:0/700195398 conn(0x7f23940cf720 msgr2=0x7f23940cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 -- 172.21.15.29:0/700195398 shutdown_connections 2023-10-14T17:57:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.829+0000 7f2399b0f640 1 -- 172.21.15.29:0/700195398 wait complete. 2023-10-14T17:57:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.830+0000 7f2399b0f640 1 Processor -- start 2023-10-14T17:57:22.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 -- start start 2023-10-14T17:57:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415d030 0x7f23940c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239415d840 con 0x7f239415d410 2023-10-14T17:57:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239415da90 con 0x7f239415d030 2023-10-14T17:57:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.831+0000 7f2399b0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f239415dd70 con 0x7f23940cd260 2023-10-14T17:57:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39134/0 (socket says 172.21.15.29:39134) 2023-10-14T17:57:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 -- 172.21.15.29:0/2085613994 learned_addr learned my addr 172.21.15.29:0/2085613994 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f23937fe640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f238affd640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415d030 0x7f23940c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 msgr2=0x7f23940c6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940c6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415d030 msgr2=0x7f23940c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415d030 0x7f23940c72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.832+0000 7f2393fff640 1 -- 172.21.15.29:0/2085613994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f237c0094d0 con 0x7f239415d410 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2393fff640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f238c00cc30 tx=0x7f238c00a5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c00ce40 con 0x7f239415d410 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f238c003990 con 0x7f239415d410 2023-10-14T17:57:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f238c003d20 con 0x7f239415d410 2023-10-14T17:57:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239415e050 con 0x7f239415d410 2023-10-14T17:57:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.833+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239415e610 con 0x7f239415d410 2023-10-14T17:57:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.834+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23940b7690 con 0x7f239415d410 2023-10-14T17:57:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.836+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f238c003e80 con 0x7f239415d410 2023-10-14T17:57:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.837+0000 7f2390ff9640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 0x7f237406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.837+0000 7f23937fe640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 msgr2=0x7f237406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.837+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f238c089f40 con 0x7f239415d410 2023-10-14T17:57:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.837+0000 7f23937fe640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 0x7f237406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:22.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:22.842+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f238c054b00 con 0x7f239415d410 2023-10-14T17:57:22.864 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:22.862+0000 7f8af2e13040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:57:22.919 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:22.917+0000 7f8af2e13040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:57:22.946 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:22.945+0000 7f473bada040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:57:22.969 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:22.967+0000 7f8af2e13040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:57:23.002 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.001+0000 7f473bada040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:57:23.028 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.027+0000 7f8af2e13040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:57:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.038+0000 7f23937fe640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 msgr2=0x7f237406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:23.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.038+0000 7f23937fe640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 0x7f237406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.039+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f239415e800 con 0x7f239415d410 2023-10-14T17:57:23.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.039+0000 7f2390ff9640 1 -- 172.21.15.29:0/2085613994 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f238c01d070 con 0x7f239415d410 2023-10-14T17:57:23.041 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:57:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 msgr2=0x7f237406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 0x7f237406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 msgr2=0x7f2394065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 secure :-1 s=READY pgs=1864 cs=0 l=1 rev1=1 crypto rx=0x7f238c00cc30 tx=0x7f238c00a5c0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 shutdown_connections 2023-10-14T17:57:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415d410 0x7f2394065ab0 unknown :-1 s=CLOSED pgs=1864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f237406af90 0x7f237406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415d030 0x7f23940c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 --2- 172.21.15.29:0/2085613994 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940cd260 0x7f23940c6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 >> 172.21.15.29:0/2085613994 conn(0x7f23940cf720 msgr2=0x7f23940b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 shutdown_connections 2023-10-14T17:57:23.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.042+0000 7f2399b0f640 1 -- 172.21.15.29:0/2085613994 wait complete. 2023-10-14T17:57:23.055 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.054+0000 7f473bada040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:57:23.061 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:57:23.077 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.075+0000 7f8af2e13040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:57:23.120 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.119+0000 7f473bada040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:57:23.173 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.172+0000 7f473bada040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:57:23.185 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.184+0000 7f8af2e13040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:57:23.247 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.246+0000 7f8af2e13040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.261+0000 7f058f67c640 1 Processor -- start 2023-10-14T17:57:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.261+0000 7f058f67c640 1 -- start start 2023-10-14T17:57:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058f67c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f05881618b0 0x7f0588065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058f67c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058806cee0 con 0x7f0588065ff0 2023-10-14T17:57:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058f67c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05880663d0 con 0x7f058815d330 2023-10-14T17:57:23.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058f67c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058806b1e0 con 0x7f05881618b0 2023-10-14T17:57:23.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058dc19640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0588065ff0 0x7f05880ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34920/0 (socket says 172.21.15.29:34920) 2023-10-14T17:57:23.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.263+0000 7f058dc19640 1 -- 172.21.15.29:0/3453360443 learned_addr learned my addr 172.21.15.29:0/3453360443 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:23.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490277882 0 0) 0x7f058806cee0 con 0x7f0588065ff0 2023-10-14T17:57:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f058cc17640 1 --2- 172.21.15.29:0/3453360443 >> v2:172.21.15.186:3300/0 conn(0x7f05881618b0 0x7f0588065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574003520 con 0x7f0588065ff0 2023-10-14T17:57:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 767401260 0 0) 0x7f0574003520 con 0x7f0588065ff0 2023-10-14T17:57:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 >> v2:172.21.15.186:3300/0 conn(0x7f05881618b0 msgr2=0x7f0588065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:23.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 --2- 172.21.15.29:0/3453360443 >> v2:172.21.15.186:3300/0 conn(0x7f05881618b0 0x7f0588065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 >> v1:172.21.15.186:6789/0 conn(0x7f058815d330 legacy=0x7f058806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880d0890 con 0x7f0588065ff0 2023-10-14T17:57:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f05800019f0 con 0x7f0588065ff0 2023-10-14T17:57:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 >> v1:172.21.15.29:6789/0 conn(0x7f0588065ff0 legacy=0x7f05880ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0574003970 0x7f0574003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05740047c0 0x7f0574004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 0x7f057400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058806b1e0 con 0x7f05740052a0 2023-10-14T17:57:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057400c170 con 0x7f05740047c0 2023-10-14T17:57:23.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.264+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057400c330 con 0x7f0574003970 2023-10-14T17:57:23.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 0x7f057400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058d418640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0574003970 0x7f0574003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058cc17640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05740047c0 0x7f0574004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 -- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0574003970 msgr2=0x7f0574003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:23.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0574003970 0x7f0574003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 -- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05740047c0 msgr2=0x7f0574004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f05740047c0 0x7f0574004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.265+0000 7f058dc19640 1 -- 172.21.15.29:0/3453360443 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0574003520 con 0x7f05740052a0 2023-10-14T17:57:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.266+0000 7f058dc19640 1 -- 172.21.15.29:0/3453360443 reap_dead start 2023-10-14T17:57:23.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.266+0000 7f058dc19640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 0x7f057400b980 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f058000a800 tx=0x7f05740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db1faea28daf496 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:23.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.266+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800072d0 con 0x7f05740052a0 2023-10-14T17:57:23.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058002f070 con 0x7f05740052a0 2023-10-14T17:57:23.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f057ffff640 1 -- 172.21.15.29:0/3453360443 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580007670 con 0x7f05740052a0 2023-10-14T17:57:23.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 -- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 msgr2=0x7f057400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 0x7f057400b980 secure :-1 s=READY pgs=1866 cs=0 l=1 rev1=1 crypto rx=0x7f058000a800 tx=0x7f05740057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 -- 172.21.15.29:0/3453360443 shutdown_connections 2023-10-14T17:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 --2- 172.21.15.29:0/3453360443 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05740052a0 0x7f057400b980 unknown :-1 s=CLOSED pgs=1866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 -- 172.21.15.29:0/3453360443 >> 172.21.15.29:0/3453360443 conn(0x7f05880d0ab0 msgr2=0x7f05880d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:23.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 -- 172.21.15.29:0/3453360443 shutdown_connections 2023-10-14T17:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.267+0000 7f058f67c640 1 -- 172.21.15.29:0/3453360443 wait complete. 2023-10-14T17:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.269+0000 7f058f67c640 1 Processor -- start 2023-10-14T17:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.269+0000 7f058f67c640 1 -- start start 2023-10-14T17:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05740052a0 0x7f058815a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0588159420 0x7f058815b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058815a850 con 0x7f05880af190 2023-10-14T17:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05880ae0f0 con 0x7f0588159420 2023-10-14T17:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058f67c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05880ae3d0 con 0x7f05740052a0 2023-10-14T17:57:23.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058dc19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058dc19640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39152/0 (socket says 172.21.15.29:39152) 2023-10-14T17:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.270+0000 7f058dc19640 1 -- 172.21.15.29:0/1883212460 learned_addr learned my addr 172.21.15.29:0/1883212460 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058d418640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05740052a0 0x7f058815a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058cc17640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0588159420 0x7f058815b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05740052a0 msgr2=0x7f058815a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05740052a0 0x7f058815a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0588159420 msgr2=0x7f058815b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0588159420 0x7f058815b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 -- 172.21.15.29:0/1883212460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05880ae6b0 con 0x7f05880af190 2023-10-14T17:57:23.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.271+0000 7f058dc19640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f0580000f80 tx=0x7f0580001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.272+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580004d60 con 0x7f05880af190 2023-10-14T17:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.272+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f058002ce90 con 0x7f05880af190 2023-10-14T17:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.272+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05880ae990 con 0x7f05880af190 2023-10-14T17:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.272+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05800072d0 con 0x7f05880af190 2023-10-14T17:57:23.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.272+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058815bba0 con 0x7f05880af190 2023-10-14T17:57:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.273+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05880aeeb0 con 0x7f05880af190 2023-10-14T17:57:23.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.274+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f0580031030 con 0x7f05880af190 2023-10-14T17:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.275+0000 7f057dffb640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 0x7f056c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.276+0000 7f058d418640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 msgr2=0x7f056c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.276+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f05800aac70 con 0x7f05880af190 2023-10-14T17:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.276+0000 7f058d418640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 0x7f056c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:23.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.280+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f05880aeeb0 con 0x7f05880af190 2023-10-14T17:57:23.293 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.292+0000 7f473bada040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:57:23.298 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.296+0000 7f8af2e13040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:57:23.353 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.352+0000 7f8af2e13040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:57:23.359 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.358+0000 7f473bada040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:23.411 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.409+0000 7f8af2e13040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:57:23.420 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.419+0000 7f473bada040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:57:23.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.477+0000 7f058d418640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 msgr2=0x7f056c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:23.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.477+0000 7f058d418640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 0x7f056c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:23.501 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.500+0000 7f473bada040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:57:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.512+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05880aeeb0 con 0x7f05880af190 2023-10-14T17:57:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.513+0000 7f057dffb640 1 -- 172.21.15.29:0/1883212460 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v164) v1 ==== 104+0+11 (secure 0 0 0) 0x7f05800440a0 con 0x7f05880af190 2023-10-14T17:57:23.514 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:57:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 msgr2=0x7f056c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 0x7f056c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 msgr2=0x7f05880c55a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 secure :-1 s=READY pgs=1867 cs=0 l=1 rev1=1 crypto rx=0x7f0580000f80 tx=0x7f0580001d80 comp rx=0 tx=0).stop 2023-10-14T17:57:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 shutdown_connections 2023-10-14T17:57:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05880af190 0x7f05880c55a0 unknown :-1 s=CLOSED pgs=1867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f056c06af90 0x7f056c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0588159420 0x7f058815b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 --2- 172.21.15.29:0/1883212460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05740052a0 0x7f058815a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 >> 172.21.15.29:0/1883212460 conn(0x7f05880d0ab0 msgr2=0x7f058815c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 shutdown_connections 2023-10-14T17:57:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.515+0000 7f058f67c640 1 -- 172.21.15.29:0/1883212460 wait complete. 2023-10-14T17:57:23.534 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:57:23.556 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.555+0000 7f473bada040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:57:23.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.728+0000 7f99fbfff640 1 Processor -- start 2023-10-14T17:57:23.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.728+0000 7f99fbfff640 1 -- start start 2023-10-14T17:57:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99fc0cded0 0x7f99fc0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fbfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99fc15d150 con 0x7f99fc0b8480 2023-10-14T17:57:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fbfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99fc0d1180 con 0x7f99fc0cded0 2023-10-14T17:57:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fbfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0d1460 con 0x7f99fc15cd70 2023-10-14T17:57:23.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99fc0cded0 0x7f99fc0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f99fc0cded0 0x7f99fc0b5c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39162/0 (socket says 172.21.15.29:39162) 2023-10-14T17:57:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99fb7fe640 1 -- 172.21.15.29:0/3338337540 learned_addr learned my addr 172.21.15.29:0/3338337540 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:23.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.730+0000 7f99faffd640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.731+0000 7f99fb7fe640 1 -- 172.21.15.29:0/3338337540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 msgr2=0x7f99fc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.731+0000 7f99fb7fe640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.731+0000 7f99fb7fe640 1 -- 172.21.15.29:0/3338337540 >> v1:172.21.15.186:6789/0 conn(0x7f99fc0b8480 legacy=0x7f99fc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.731+0000 7f99fb7fe640 1 -- 172.21.15.29:0/3338337540 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99fc0d1740 con 0x7f99fc0cded0 2023-10-14T17:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.731+0000 7f99faffd640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:23.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.732+0000 7f99fb7fe640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0cded0 0x7f99fc0b5c90 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009e50 tx=0x7f99f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4abda4fca64039c0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.732+0000 7f99fa7fc640 1 -- 172.21.15.29:0/3338337540 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f402caf0 con 0x7f99fc0cded0 2023-10-14T17:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.732+0000 7f99fa7fc640 1 -- 172.21.15.29:0/3338337540 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f402cc50 con 0x7f99fc0cded0 2023-10-14T17:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.732+0000 7f99fa7fc640 1 -- 172.21.15.29:0/3338337540 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f40343f0 con 0x7f99fc0cded0 2023-10-14T17:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 -- 172.21.15.29:0/3338337540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0cded0 msgr2=0x7f99fc0b5c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0cded0 0x7f99fc0b5c90 secure :-1 s=READY pgs=1868 cs=0 l=1 rev1=1 crypto rx=0x7f99f4009e50 tx=0x7f99f402bf50 comp rx=0 tx=0).stop 2023-10-14T17:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 -- 172.21.15.29:0/3338337540 shutdown_connections 2023-10-14T17:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0cded0 0x7f99fc0b5c90 unknown :-1 s=CLOSED pgs=1868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3338337540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 -- 172.21.15.29:0/3338337540 >> 172.21.15.29:0/3338337540 conn(0x7f99fc0cf570 msgr2=0x7f99fc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:23.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 -- 172.21.15.29:0/3338337540 shutdown_connections 2023-10-14T17:57:23.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.733+0000 7f99fbfff640 1 -- 172.21.15.29:0/3338337540 wait complete. 2023-10-14T17:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.735+0000 7f99fbfff640 1 Processor -- start 2023-10-14T17:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.736+0000 7f99fbfff640 1 -- start start 2023-10-14T17:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99fc0cded0 0x7f99fc0c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0b3ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc1f2160 con 0x7f99fc0b8480 2023-10-14T17:57:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0c7290 con 0x7f99fc0cded0 2023-10-14T17:57:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fbfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99fc0c7570 con 0x7f99fc15cd70 2023-10-14T17:57:23.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39178/0 (socket says 172.21.15.29:39178) 2023-10-14T17:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99faffd640 1 -- 172.21.15.29:0/3022507197 learned_addr learned my addr 172.21.15.29:0/3022507197 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99fb7fe640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0b3ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.737+0000 7f99f3fff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99fc0cded0 0x7f99fc0c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:23.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 -- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 msgr2=0x7f99fc0b3ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0b3ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 -- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99fc0cded0 msgr2=0x7f99fc0c8d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99fc0cded0 0x7f99fc0c8d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 -- 172.21.15.29:0/3022507197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f4009b30 con 0x7f99fc0b8480 2023-10-14T17:57:23.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99faffd640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00e730 tx=0x7f99ec00ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec004230 con 0x7f99fc0b8480 2023-10-14T17:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99ec004390 con 0x7f99fc0b8480 2023-10-14T17:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99ec00c8f0 con 0x7f99fc0b8480 2023-10-14T17:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.738+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99fc0c7850 con 0x7f99fc0b8480 2023-10-14T17:57:23.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.739+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99fc0c7e10 con 0x7f99fc0b8480 2023-10-14T17:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.740+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99fc04e9b0 con 0x7f99fc0b8480 2023-10-14T17:57:23.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.741+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f99ec00ca50 con 0x7f99fc0b8480 2023-10-14T17:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.742+0000 7f99f37fe640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.742+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f99ec089640 con 0x7f99fc0b8480 2023-10-14T17:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.742+0000 7f99f3fff640 1 -- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 msgr2=0x7f99dc06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:23.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.742+0000 7f99f3fff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:23.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.746+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f99ec054350 con 0x7f99fc0b8480 2023-10-14T17:57:23.824 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.823+0000 7f8af2e13040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:57:23.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.939+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f99fc0cfb60 con 0x7f99fc0b8480 2023-10-14T17:57:23.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.940+0000 7f99f37fe640 1 -- 172.21.15.29:0/3022507197 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v164) v1 ==== 100+0+11 (secure 0 0 0) 0x7f99ec057970 con 0x7f99fc0b8480 2023-10-14T17:57:23.941 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:57:23.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 msgr2=0x7f99dc06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 msgr2=0x7f99fc1f4f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 secure :-1 s=READY pgs=1869 cs=0 l=1 rev1=1 crypto rx=0x7f99ec00e730 tx=0x7f99ec00ebf0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 shutdown_connections 2023-10-14T17:57:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f99fc15cd70 0x7f99fc0b3ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f99dc06af90 0x7f99dc06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99fc0cded0 0x7f99fc0c8d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 --2- 172.21.15.29:0/3022507197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99fc0b8480 0x7f99fc1f4f00 unknown :-1 s=CLOSED pgs=1869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:23.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.942+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 >> 172.21.15.29:0/3022507197 conn(0x7f99fc0cf570 msgr2=0x7f99fc0cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:23.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.943+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 shutdown_connections 2023-10-14T17:57:23.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:23.943+0000 7f99fbfff640 1 -- 172.21.15.29:0/3022507197 wait complete. 2023-10-14T17:57:23.961 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T17:57:23.978 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:23.977+0000 7f473bada040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:57:23.983 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:23.981+0000 7f8af2e13040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:57:24.089 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.087+0000 7f8af2e13040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:57:24.138 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.136+0000 7f8af2e13040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:57:24.148 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.147+0000 7f473bada040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:57:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.186+0000 7f760dae6640 1 Processor -- start 2023-10-14T17:57:24.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.186+0000 7f760dae6640 1 -- start start 2023-10-14T17:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760dae6640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f76080b8480 0x7f76080cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760dae6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f760815d150 con 0x7f76080d0be0 2023-10-14T17:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760dae6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76080d1180 con 0x7f760815cd70 2023-10-14T17:57:24.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760dae6640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76080d1460 con 0x7f76080b8480 2023-10-14T17:57:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760d2e5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f76080d0be0 0x7f76080b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34922/0 (socket says 172.21.15.29:34922) 2023-10-14T17:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f760d2e5640 1 -- 172.21.15.29:0/2346814096 learned_addr learned my addr 172.21.15.29:0/2346814096 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.188+0000 7f7607fff640 1 --2- 172.21.15.29:0/2346814096 >> v2:172.21.15.186:3300/0 conn(0x7f76080b8480 0x7f76080cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553832738 0 0) 0x7f760815d150 con 0x7f76080d0be0 2023-10-14T17:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f8003520 con 0x7f76080d0be0 2023-10-14T17:57:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1849917068 0 0) 0x7f75f8003520 con 0x7f76080d0be0 2023-10-14T17:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 >> v2:172.21.15.186:3300/0 conn(0x7f76080b8480 msgr2=0x7f76080cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 --2- 172.21.15.29:0/2346814096 >> v2:172.21.15.186:3300/0 conn(0x7f76080b8480 0x7f76080cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 >> v1:172.21.15.186:6789/0 conn(0x7f760815cd70 legacy=0x7f76080d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76080d1740 con 0x7f76080d0be0 2023-10-14T17:57:24.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f76000019f0 con 0x7f76080d0be0 2023-10-14T17:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 >> v1:172.21.15.29:6789/0 conn(0x7f76080d0be0 legacy=0x7f76080b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 0x7f75f8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75f80047c0 0x7f75f8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80051d0 0x7f75f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080d1460 con 0x7f75f80039c0 2023-10-14T17:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75f800c120 con 0x7f75f80047c0 2023-10-14T17:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.189+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75f800c2e0 con 0x7f75f80051d0 2023-10-14T17:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 0x7f75f8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760d2e5640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80051d0 0x7f75f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f7607fff640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75f80047c0 0x7f75f8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 -- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80051d0 msgr2=0x7f75f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80051d0 0x7f75f800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 -- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75f80047c0 msgr2=0x7f75f8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75f80047c0 0x7f75f8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760cae4640 1 -- 172.21.15.29:0/2346814096 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8003520 con 0x7f75f80039c0 2023-10-14T17:57:24.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f760d2e5640 1 -- 172.21.15.29:0/2346814096 reap_dead start 2023-10-14T17:57:24.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.190+0000 7f7607fff640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75f80047c0 0x7f75f8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.191+0000 7f760cae4640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 0x7f75f8003de0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f75fc001720 tx=0x7f75f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dfda31055c4f176 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.191+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc0052d0 con 0x7f75f80039c0 2023-10-14T17:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.191+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75fc02e050 con 0x7f75f80039c0 2023-10-14T17:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f76077fe640 1 -- 172.21.15.29:0/2346814096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75fc003830 con 0x7f75f80039c0 2023-10-14T17:57:24.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 -- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 msgr2=0x7f75f8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 0x7f75f8003de0 secure :-1 s=READY pgs=1871 cs=0 l=1 rev1=1 crypto rx=0x7f75fc001720 tx=0x7f75f80057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 -- 172.21.15.29:0/2346814096 shutdown_connections 2023-10-14T17:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 --2- 172.21.15.29:0/2346814096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75f80039c0 0x7f75f8003de0 unknown :-1 s=CLOSED pgs=1871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 -- 172.21.15.29:0/2346814096 >> 172.21.15.29:0/2346814096 conn(0x7f76080cf570 msgr2=0x7f76080cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:24.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 -- 172.21.15.29:0/2346814096 shutdown_connections 2023-10-14T17:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.192+0000 7f760dae6640 1 -- 172.21.15.29:0/2346814096 wait complete. 2023-10-14T17:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.194+0000 7f760dae6640 1 Processor -- start 2023-10-14T17:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.194+0000 7f760dae6640 1 -- start start 2023-10-14T17:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80039c0 0x7f760815e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 0x7f76080c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76080b7670 con 0x7f76080b6ca0 2023-10-14T17:57:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76081503e0 con 0x7f760814ff60 2023-10-14T17:57:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760dae6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76080cc960 con 0x7f75f80039c0 2023-10-14T17:57:24.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760d2e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760d2e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39186/0 (socket says 172.21.15.29:39186) 2023-10-14T17:57:24.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.195+0000 7f760d2e5640 1 -- 172.21.15.29:0/3449489479 learned_addr learned my addr 172.21.15.29:0/3449489479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:24.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760cae4640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80039c0 0x7f760815e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f7607fff640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 0x7f76080c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80039c0 msgr2=0x7f760815e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80039c0 0x7f760815e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 msgr2=0x7f76080c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 0x7f76080c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 -- 172.21.15.29:0/3449489479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76081f5b80 con 0x7f76080b6ca0 2023-10-14T17:57:24.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f7607fff640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 0x7f76080c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760d2e5640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f76080cd0f0 tx=0x7f7600007af0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760000ea40 con 0x7f76080b6ca0 2023-10-14T17:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760000ebc0 con 0x7f76080b6ca0 2023-10-14T17:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7600013420 con 0x7f76080b6ca0 2023-10-14T17:57:24.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.196+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76081f5e60 con 0x7f76080b6ca0 2023-10-14T17:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.197+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76081f6420 con 0x7f76080b6ca0 2023-10-14T17:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.198+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f760804e9b0 con 0x7f76080b6ca0 2023-10-14T17:57:24.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.198+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f7600013580 con 0x7f76080b6ca0 2023-10-14T17:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.199+0000 7f76057fa640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 0x7f75ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.199+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f7600089df0 con 0x7f76080b6ca0 2023-10-14T17:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.199+0000 7f760cae4640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 msgr2=0x7f75ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:24.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.199+0000 7f760cae4640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 0x7f75ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:24.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.203+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7600054ac0 con 0x7f76080b6ca0 2023-10-14T17:57:24.246 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.244+0000 7f8af2e13040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:24.266 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.265+0000 7f473bada040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:57:24.319 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.318+0000 7f473bada040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:57:24.352 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.350+0000 7f8af2e13040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:57:24.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.401+0000 7f760cae4640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 msgr2=0x7f75ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:24.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.401+0000 7f760cae4640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 0x7f75ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:24.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.405+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f76081f2590 con 0x7f76080b6ca0 2023-10-14T17:57:24.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.406+0000 7f76057fa640 1 -- 172.21.15.29:0/3449489479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v164) v1 ==== 80+0+467 (secure 0 0 0) 0x7f76000580e0 con 0x7f76080b6ca0 2023-10-14T17:57:24.407 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3251497707 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.407 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/80005070 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.407 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2913425840 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3470978219 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1610794407 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/3121942650 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/3121942650 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.408 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/542195718 2023-10-15T17:01:29.273525+0000 2023-10-14T17:57:24.409 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T17:57:24.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 msgr2=0x7f75ec06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:24.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 0x7f75ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 msgr2=0x7f76080b7130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:24.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 secure :-1 s=READY pgs=1872 cs=0 l=1 rev1=1 crypto rx=0x7f76080cd0f0 tx=0x7f7600007af0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 shutdown_connections 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76080b6ca0 0x7f76080b7130 unknown :-1 s=CLOSED pgs=1872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f75ec06af90 0x7f75ec06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f760814ff60 0x7f76080c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 --2- 172.21.15.29:0/3449489479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75f80039c0 0x7f760815e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 >> 172.21.15.29:0/3449489479 conn(0x7f76080cf570 msgr2=0x7f76080b2850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:24.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.408+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 shutdown_connections 2023-10-14T17:57:24.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.409+0000 7f760dae6640 1 -- 172.21.15.29:0/3449489479 wait complete. 2023-10-14T17:57:24.426 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T17:57:24.427 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:24.427 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T17:57:24.434 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.433+0000 7f473bada040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:24.507 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.505+0000 7f8af2e13040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:24.519 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T17:57:24.520 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:24.520 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T17:57:24.555 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.553+0000 7f473bada040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:57:24.556 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.555+0000 7f8af2e13040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:24.607 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:57:24.608 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:24.663 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:57:24.664 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:24.664 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:24.678 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:24.677+0000 7f8af2e13040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:57:24.720 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.720+0000 7f473bada040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:24.722 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:57:24.723 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:57:24.723 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:24.774 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.773+0000 7f473bada040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:24.778 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:57:24.778 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:24.902 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:24.901+0000 7f473bada040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:57:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.991+0000 7fafee987640 1 Processor -- start 2023-10-14T17:57:24.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.991+0000 7fafee987640 1 -- start start 2023-10-14T17:57:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.992+0000 7fafee987640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafe80ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.992+0000 7fafee987640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fafe8070240 0x7fafe80706a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.992+0000 7fafee987640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafe8070be0 con 0x7fafe815aaa0 2023-10-14T17:57:24.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.992+0000 7fafee987640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafe80b2a60 con 0x7fafe8070240 2023-10-14T17:57:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.992+0000 7fafee987640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80b2d00 con 0x7fafe80b2600 2023-10-14T17:57:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafed985640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fafe815aaa0 0x7fafe80b3da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34930/0 (socket says 172.21.15.29:34930) 2023-10-14T17:57:24.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafed985640 1 -- 172.21.15.29:0/2358003214 learned_addr learned my addr 172.21.15.29:0/2358003214 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:24.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafee186640 1 --2- 172.21.15.29:0/2358003214 >> v2:172.21.15.186:3300/0 conn(0x7fafe8070240 0x7fafe80706a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafed184640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafe80ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918876929 0 0) 0x7fafe8070be0 con 0x7fafe815aaa0 2023-10-14T17:57:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.993+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafd4003520 con 0x7fafe815aaa0 2023-10-14T17:57:24.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 846720626 0 0) 0x7fafd4003520 con 0x7fafe815aaa0 2023-10-14T17:57:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 msgr2=0x7fafe80ca8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafe80ca8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 >> v2:172.21.15.186:3300/0 conn(0x7fafe8070240 msgr2=0x7fafe80706a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 --2- 172.21.15.29:0/2358003214 >> v2:172.21.15.186:3300/0 conn(0x7fafe8070240 0x7fafe80706a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:24.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe80b2fe0 con 0x7fafe815aaa0 2023-10-14T17:57:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fafe4001670 con 0x7fafe815aaa0 2023-10-14T17:57:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 >> v1:172.21.15.29:6789/0 conn(0x7fafe815aaa0 legacy=0x7fafe80b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafd4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafd40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd4004d00 0x7fafd4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80b2a60 con 0x7fafd4003970 2023-10-14T17:57:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe80b2d00 con 0x7fafd4004d00 2023-10-14T17:57:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafd400b9b0 con 0x7fafe80b2600 2023-10-14T17:57:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.994+0000 7fafed985640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafd4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed184640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafd40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafee186640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd4004d00 0x7fafd4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed985640 1 -- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 msgr2=0x7fafd40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed985640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafd40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed985640 1 -- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd4004d00 msgr2=0x7fafd4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:25.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed985640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafd4004d00 0x7fafd4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed985640 1 -- 172.21.15.29:0/2358003214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafd4003520 con 0x7fafd4003970 2023-10-14T17:57:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.995+0000 7fafed184640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe80b2600 0x7fafd40047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.996+0000 7fafed985640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafd4003d90 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c190 tx=0x7fafd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e4eae7be6093ac4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.996+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4005ad0 con 0x7fafd4003970 2023-10-14T17:57:25.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.996+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe4005c50 con 0x7fafd4003970 2023-10-14T17:57:25.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.996+0000 7fafec983640 1 -- 172.21.15.29:0/2358003214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe402c810 con 0x7fafd4003970 2023-10-14T17:57:25.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 -- 172.21.15.29:0/2358003214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 msgr2=0x7fafd4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 --2- 172.21.15.29:0/2358003214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafd4003d90 secure :-1 s=READY pgs=1874 cs=0 l=1 rev1=1 crypto rx=0x7fafe400c190 tx=0x7fafd40057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee186640 1 -- 172.21.15.29:0/2358003214 reap_dead start 2023-10-14T17:57:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 -- 172.21.15.29:0/2358003214 shutdown_connections 2023-10-14T17:57:25.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 -- 172.21.15.29:0/2358003214 >> 172.21.15.29:0/2358003214 conn(0x7fafe80cc230 msgr2=0x7fafe80cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 -- 172.21.15.29:0/2358003214 shutdown_connections 2023-10-14T17:57:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.997+0000 7fafee987640 1 -- 172.21.15.29:0/2358003214 wait complete. 2023-10-14T17:57:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.999+0000 7fafee987640 1 Processor -- start 2023-10-14T17:57:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:24.999+0000 7fafee987640 1 -- start start 2023-10-14T17:57:25.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafe806d980 0x7fafe8153ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe8154410 0x7fafe81548c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8069da0 con 0x7fafd4003970 2023-10-14T17:57:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafe8156510 con 0x7fafe806d980 2023-10-14T17:57:25.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafee987640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafe81567f0 con 0x7fafe8154410 2023-10-14T17:57:25.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafed985640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafed985640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39196/0 (socket says 172.21.15.29:39196) 2023-10-14T17:57:25.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafed985640 1 -- 172.21.15.29:0/3592857701 learned_addr learned my addr 172.21.15.29:0/3592857701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:25.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.000+0000 7fafed184640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafe806d980 0x7fafe8153ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafee186640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe8154410 0x7fafe81548c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe8154410 msgr2=0x7fafe81548c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe8154410 0x7fafe81548c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafe806d980 msgr2=0x7fafe8153ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafe806d980 0x7fafe8153ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 -- 172.21.15.29:0/3592857701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafe8156ad0 con 0x7fafd4003970 2023-10-14T17:57:25.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafed985640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fafe402c3b0 tx=0x7fafe80b58e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe4039070 con 0x7fafd4003970 2023-10-14T17:57:25.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafe402cc20 con 0x7fafd4003970 2023-10-14T17:57:25.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafe403c040 con 0x7fafd4003970 2023-10-14T17:57:25.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafe8156db0 con 0x7fafd4003970 2023-10-14T17:57:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.001+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafe8156150 con 0x7fafd4003970 2023-10-14T17:57:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.003+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafe804e9b0 con 0x7fafd4003970 2023-10-14T17:57:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.004+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fafe4032070 con 0x7fafd4003970 2023-10-14T17:57:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.004+0000 7fafde7fc640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 0x7fafcc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.004+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7fafe40aacb0 con 0x7fafd4003970 2023-10-14T17:57:25.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.005+0000 7fafed184640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 msgr2=0x7fafcc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:25.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.005+0000 7fafed184640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 0x7fafcc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:25.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.009+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fafe4075a10 con 0x7fafd4003970 2023-10-14T17:57:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.206+0000 7fafed184640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 msgr2=0x7fafcc06d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:25.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.206+0000 7fafed184640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 0x7fafcc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.234+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafe8071580 con 0x7fafd4003970 2023-10-14T17:57:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.234+0000 7fafde7fc640 1 -- 172.21.15.29:0/3592857701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fafe4079030 con 0x7fafd4003970 2023-10-14T17:57:25.236 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:25.236 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:57:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:57:25.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 msgr2=0x7fafcc06d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:25.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 0x7fafcc06d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 msgr2=0x7fafe806d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 secure :-1 s=READY pgs=1875 cs=0 l=1 rev1=1 crypto rx=0x7fafe402c3b0 tx=0x7fafe80b58e0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 shutdown_connections 2023-10-14T17:57:25.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fafe8154410 0x7fafe81548c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7fafcc06afc0 0x7fafcc06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fafe806d980 0x7fafe8153ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 --2- 172.21.15.29:0/3592857701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fafd4003970 0x7fafe806d440 unknown :-1 s=CLOSED pgs=1875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 >> 172.21.15.29:0/3592857701 conn(0x7fafe80cc230 msgr2=0x7fafe8152960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:25.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 shutdown_connections 2023-10-14T17:57:25.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.237+0000 7fafee987640 1 -- 172.21.15.29:0/3592857701 wait complete. 2023-10-14T17:57:25.256 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:25.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.460+0000 7f023216a640 1 Processor -- start 2023-10-14T17:57:25.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.460+0000 7f023216a640 1 -- start start 2023-10-14T17:57:25.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f022c15cbe0 0x7f022c0d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 0x7f022c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c0d1090 0x7f022c0b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c0d15a0 con 0x7f022c15cbe0 2023-10-14T17:57:25.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f022c0d1880 con 0x7f022c15cfc0 2023-10-14T17:57:25.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.461+0000 7f023216a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c0d1b60 con 0x7f022c0d1090 2023-10-14T17:57:25.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f022c15cbe0 0x7f022c0d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f022c15cbe0 0x7f022c0d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39208/0 (socket says 172.21.15.29:39208) 2023-10-14T17:57:25.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 -- 172.21.15.29:0/2248162046 learned_addr learned my addr 172.21.15.29:0/2248162046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:25.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0230967640 1 --2- 172.21.15.29:0/2248162046 >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 0x7f022c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 -- 172.21.15.29:0/2248162046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c0d1090 msgr2=0x7f022c0b4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:25.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 --2- 172.21.15.29:0/2248162046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c0d1090 0x7f022c0b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 -- 172.21.15.29:0/2248162046 >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 msgr2=0x7f022c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:25.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 --2- 172.21.15.29:0/2248162046 >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 0x7f022c0b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0231168640 1 -- 172.21.15.29:0/2248162046 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022c0d1e40 con 0x7f022c15cbe0 2023-10-14T17:57:25.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.462+0000 7f0230967640 1 --2- 172.21.15.29:0/2248162046 >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 0x7f022c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:25.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.463+0000 7f0231168640 1 --2- 172.21.15.29:0/2248162046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c15cbe0 0x7f022c0d0b50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f022c0cf4b0 tx=0x7f022802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40512a23a007a155 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.464+0000 7f0223fff640 1 -- 172.21.15.29:0/2248162046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022802cb00 con 0x7f022c15cbe0 2023-10-14T17:57:25.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.464+0000 7f0223fff640 1 -- 172.21.15.29:0/2248162046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022802cc60 con 0x7f022c15cbe0 2023-10-14T17:57:25.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.464+0000 7f0223fff640 1 -- 172.21.15.29:0/2248162046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02280343f0 con 0x7f022c15cbe0 2023-10-14T17:57:25.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.464+0000 7f023216a640 1 -- 172.21.15.29:0/2248162046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c15cbe0 msgr2=0x7f022c0d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.464+0000 7f023216a640 1 --2- 172.21.15.29:0/2248162046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c15cbe0 0x7f022c0d0b50 secure :-1 s=READY pgs=1876 cs=0 l=1 rev1=1 crypto rx=0x7f022c0cf4b0 tx=0x7f022802bf60 comp rx=0 tx=0).stop 2023-10-14T17:57:25.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 -- 172.21.15.29:0/2248162046 shutdown_connections 2023-10-14T17:57:25.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 --2- 172.21.15.29:0/2248162046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c0d1090 0x7f022c0b4ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 --2- 172.21.15.29:0/2248162046 >> v2:172.21.15.186:3300/0 conn(0x7f022c15cfc0 0x7f022c0b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 --2- 172.21.15.29:0/2248162046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c15cbe0 0x7f022c0d0b50 unknown :-1 s=CLOSED pgs=1876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 -- 172.21.15.29:0/2248162046 >> 172.21.15.29:0/2248162046 conn(0x7f022c0cf570 msgr2=0x7f022c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:25.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 -- 172.21.15.29:0/2248162046 shutdown_connections 2023-10-14T17:57:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.465+0000 7f023216a640 1 -- 172.21.15.29:0/2248162046 wait complete. 2023-10-14T17:57:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.466+0000 7f023216a640 1 Processor -- start 2023-10-14T17:57:25.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.467+0000 7f023216a640 1 -- start start 2023-10-14T17:57:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.467+0000 7f023216a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f023216a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 0x7f022c1f2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f023216a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 0x7f022c155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f023216a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c1f6a50 con 0x7f022c0d1090 2023-10-14T17:57:25.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f023216a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f022c144c90 con 0x7f022c15cfc0 2023-10-14T17:57:25.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f023216a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f022c1f29c0 con 0x7f022c15cbe0 2023-10-14T17:57:25.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39222/0 (socket says 172.21.15.29:39222) 2023-10-14T17:57:25.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 -- 172.21.15.29:0/3442984238 learned_addr learned my addr 172.21.15.29:0/3442984238 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231969640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 0x7f022c155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0230967640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 0x7f022c1f2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 msgr2=0x7f022c1f2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 0x7f022c1f2480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 msgr2=0x7f022c155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 0x7f022c155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231168640 1 -- 172.21.15.29:0/3442984238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0228009b30 con 0x7f022c0d1090 2023-10-14T17:57:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0231969640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 0x7f022c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:25.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.468+0000 7f0230967640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 0x7f022c1f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:25.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f0231168640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f0228000f80 tx=0x7f0228004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022803e450 con 0x7f022c0d1090 2023-10-14T17:57:25.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f022803e5f0 con 0x7f022c0d1090 2023-10-14T17:57:25.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f022803e9a0 con 0x7f022c0d1090 2023-10-14T17:57:25.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f022c1f2c20 con 0x7f022c0d1090 2023-10-14T17:57:25.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.469+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f022c1f31e0 con 0x7f022c0d1090 2023-10-14T17:57:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.470+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f022c04e9b0 con 0x7f022c0d1090 2023-10-14T17:57:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.471+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f022803eb00 con 0x7f022c0d1090 2023-10-14T17:57:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.472+0000 7f0221ffb640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 0x7f021006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.472+0000 7f0230967640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 msgr2=0x7f021006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.472+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f02280ac8f0 con 0x7f022c0d1090 2023-10-14T17:57:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.472+0000 7f0230967640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 0x7f021006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:25.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.477+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0228077820 con 0x7f022c0d1090 2023-10-14T17:57:25.498 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:25.497+0000 7f8af2e13040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:57:25.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.672+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f022c1f33a0 con 0x7f022c0d1090 2023-10-14T17:57:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.673+0000 7f0230967640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 msgr2=0x7f021006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:25.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.673+0000 7f0230967640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 0x7f021006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:25.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.674+0000 7f0221ffb640 1 -- 172.21.15.29:0/3442984238 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v164) v1 ==== 74+0+17388 (secure 0 0 0) 0x7f022803edb0 con 0x7f022c0d1090 2023-10-14T17:57:25.676 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:25.676 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":164,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:22.720690+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"164","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":112,"snap_epoch":164,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.33","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3251497707":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/80005070":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/2913425840":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/3470978219":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/1610794407":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6833/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6832/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/542195718":"2023-10-15T17:01:29.273525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":54},{"begin":59,"length":54}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":51,"length":6},{"begin":107,"length":6}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:25.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 msgr2=0x7f021006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 0x7f021006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 msgr2=0x7f022c1f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 secure :-1 s=READY pgs=1877 cs=0 l=1 rev1=1 crypto rx=0x7f0228000f80 tx=0x7f0228004a40 comp rx=0 tx=0).stop 2023-10-14T17:57:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 shutdown_connections 2023-10-14T17:57:25.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f021006af90 0x7f021006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f022c15cfc0 0x7f022c155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f022c15cbe0 0x7f022c1f2480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 --2- 172.21.15.29:0/3442984238 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f022c0d1090 0x7f022c1f6510 unknown :-1 s=CLOSED pgs=1877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 >> 172.21.15.29:0/3442984238 conn(0x7f022c0cf570 msgr2=0x7f022c0b1a30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:25.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 shutdown_connections 2023-10-14T17:57:25.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.677+0000 7f023216a640 1 -- 172.21.15.29:0/3442984238 wait complete. 2023-10-14T17:57:25.696 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 18, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:01:12.106201+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '153', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 19, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:01:13.162539+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '164', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 112, 'snap_epoch': 164, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:57:25.696 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:57:25.757 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:25.756+0000 7f473bada040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:57:25.794 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:25.792+0000 7f8af2e13040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:57:25.849 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:25.848+0000 7f8af2e13040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:57:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.893+0000 7f291cdb0640 1 Processor -- start 2023-10-14T17:57:25.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.894+0000 7f291cdb0640 1 -- start start 2023-10-14T17:57:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.895+0000 7f291cdb0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f29180661c0 0x7f29180ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.895+0000 7f291cdb0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f291806cee0 con 0x7f29181618b0 2023-10-14T17:57:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f291cdb0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29180d0890 con 0x7f291815d330 2023-10-14T17:57:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f291cdb0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f291806b1e0 con 0x7f29180661c0 2023-10-14T17:57:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f290ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f29181618b0 0x7f2918065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34940/0 (socket says 172.21.15.29:34940) 2023-10-14T17:57:25.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f290ffff640 1 -- 172.21.15.29:0/3336740852 learned_addr learned my addr 172.21.15.29:0/3336740852 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f2916d9d640 1 --2- 172.21.15.29:0/3336740852 >> v2:172.21.15.186:3300/0 conn(0x7f29180661c0 0x7f29180ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402819166 0 0) 0x7f291806cee0 con 0x7f29181618b0 2023-10-14T17:57:25.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.896+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2904003520 con 0x7f29181618b0 2023-10-14T17:57:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.897+0000 7f2916d9d640 1 -- 172.21.15.29:0/3336740852 >> v1:172.21.15.186:6789/0 conn(0x7f291815d330 legacy=0x7f291806c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.897+0000 7f2916d9d640 1 -- 172.21.15.29:0/3336740852 >> v1:172.21.15.29:6789/0 conn(0x7f29181618b0 legacy=0x7f2918065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.897+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3226729203 0 0) 0x7f2904003520 con 0x7f29181618b0 2023-10-14T17:57:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.897+0000 7f2916d9d640 1 -- 172.21.15.29:0/3336740852 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29180d05b0 con 0x7f29180661c0 2023-10-14T17:57:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.898+0000 7f2916d9d640 1 --2- 172.21.15.29:0/3336740852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29180661c0 0x7f29180ce9b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2910009d90 tx=0x7f291002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c8f5011f790f9c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.898+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2900001670 con 0x7f29180661c0 2023-10-14T17:57:25.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.898+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291002c8b0 con 0x7f29180661c0 2023-10-14T17:57:25.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.898+0000 7f2915d9b640 1 -- 172.21.15.29:0/3336740852 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f291002cc20 con 0x7f29180661c0 2023-10-14T17:57:25.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 -- 172.21.15.29:0/3336740852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29180661c0 msgr2=0x7f29180ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 --2- 172.21.15.29:0/3336740852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29180661c0 0x7f29180ce9b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f2910009d90 tx=0x7f291002bd80 comp rx=0 tx=0).stop 2023-10-14T17:57:25.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 -- 172.21.15.29:0/3336740852 shutdown_connections 2023-10-14T17:57:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 --2- 172.21.15.29:0/3336740852 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29180661c0 0x7f29180ce9b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 -- 172.21.15.29:0/3336740852 >> 172.21.15.29:0/3336740852 conn(0x7f29180d0ab0 msgr2=0x7f29180d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 -- 172.21.15.29:0/3336740852 shutdown_connections 2023-10-14T17:57:25.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.899+0000 7f291cdb0640 1 -- 172.21.15.29:0/3336740852 wait complete. 2023-10-14T17:57:25.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.901+0000 7f291cdb0640 1 Processor -- start 2023-10-14T17:57:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.901+0000 7f291cdb0640 1 -- start start 2023-10-14T17:57:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f291815d330 0x7f29180acec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29181618b0 0x7f29180aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180b2590 con 0x7f29180661c0 2023-10-14T17:57:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29180b1250 con 0x7f291815d330 2023-10-14T17:57:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291cdb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29180b14f0 con 0x7f29181618b0 2023-10-14T17:57:25.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39224/0 (socket says 172.21.15.29:39224) 2023-10-14T17:57:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f291659c640 1 -- 172.21.15.29:0/622713815 learned_addr learned my addr 172.21.15.29:0/622713815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f290ffff640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f291815d330 0x7f29180acec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.902+0000 7f2916d9d640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29181618b0 0x7f29180aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29181618b0 msgr2=0x7f29180aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29181618b0 0x7f29180aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f291815d330 msgr2=0x7f29180acec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:25.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f291815d330 0x7f29180acec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 -- 172.21.15.29:0/622713815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2910009a70 con 0x7f29180661c0 2023-10-14T17:57:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291659c640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f290800c400 tx=0x7f290800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29080052d0 con 0x7f29180661c0 2023-10-14T17:57:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2908005430 con 0x7f29180661c0 2023-10-14T17:57:25.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29180b17d0 con 0x7f29180661c0 2023-10-14T17:57:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2908003830 con 0x7f29180661c0 2023-10-14T17:57:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.903+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29180b1d90 con 0x7f29180661c0 2023-10-14T17:57:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.904+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f291804e160 con 0x7f29180661c0 2023-10-14T17:57:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.905+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 89) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f2908003990 con 0x7f29180661c0 2023-10-14T17:57:25.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.906+0000 7f290effd640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 0x7f28f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:25.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.906+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(164..164 src has 1..164) v4 ==== 6271+0+0 (secure 0 0 0) 0x7f2908088fe0 con 0x7f29180661c0 2023-10-14T17:57:25.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.906+0000 7f290ffff640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 msgr2=0x7f28f806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:25.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.907+0000 7f290ffff640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 0x7f28f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:25.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:25.911+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2908053c90 con 0x7f29180661c0 2023-10-14T17:57:25.964 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:25.962+0000 7f8af2e13040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:57:26.074 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.072+0000 7f473bada040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:57:26.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.105+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f291806f3a0 con 0x7f29180661c0 2023-10-14T17:57:26.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.106+0000 7f290effd640 1 -- 172.21.15.29:0/622713815 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v164) v1 ==== 93+0+10 (secure 0 0 0) 0x7f29080572b0 con 0x7f29180661c0 2023-10-14T17:57:26.108 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:57:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f290ffff640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 msgr2=0x7f28f806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2309931697 2023-10-14T17:57:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f290ffff640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 0x7f28f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:26.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 msgr2=0x7f28f806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:26.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 0x7f28f806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 msgr2=0x7f29180b0d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 secure :-1 s=READY pgs=1879 cs=0 l=1 rev1=1 crypto rx=0x7f290800c400 tx=0x7f290800c8c0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 shutdown_connections 2023-10-14T17:57:26.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29181618b0 0x7f29180aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:6832/2309931697,v1:172.21.15.29:6833/2309931697] conn(0x7f28f806afc0 0x7f28f806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f291815d330 0x7f29180acec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 --2- 172.21.15.29:0/622713815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29180661c0 0x7f29180b0d10 unknown :-1 s=CLOSED pgs=1879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.108+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 >> 172.21.15.29:0/622713815 conn(0x7f29180d0ab0 msgr2=0x7f291815f720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:26.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.109+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 shutdown_connections 2023-10-14T17:57:26.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.109+0000 7f291cdb0640 1 -- 172.21.15.29:0/622713815 wait complete. 2023-10-14T17:57:26.128 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:57:26.135 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.134+0000 7f473bada040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:57:26.256 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.255+0000 7f473bada040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:57:26.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.350+0000 7fe4f7a90640 1 Processor -- start 2023-10-14T17:57:26.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.350+0000 7fe4f7a90640 1 -- start start 2023-10-14T17:57:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f7a90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00addb0 0x7fe4f00caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f7a90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4f00c9000 0x7fe4f0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f7a90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4f00cb140 con 0x7fe4f006a2e0 2023-10-14T17:57:26.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f7a90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4f0161a50 con 0x7fe4f00c9000 2023-10-14T17:57:26.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f7a90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f006a750 con 0x7fe4f00addb0 2023-10-14T17:57:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.352+0000 7fe4f502b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4f00c9000 0x7fe4f0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4f00c9000 0x7fe4f0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:46394/0 (socket says 172.21.15.29:46394) 2023-10-14T17:57:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 -- 172.21.15.29:0/4257692941 learned_addr learned my addr 172.21.15.29:0/4257692941 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f582c640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00addb0 0x7fe4f00caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 -- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00addb0 msgr2=0x7fe4f00caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:26.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00addb0 0x7fe4f00caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 -- 172.21.15.29:0/4257692941 >> v1:172.21.15.186:6789/0 conn(0x7fe4f006a2e0 legacy=0x7fe4f00cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.353+0000 7fe4f502b640 1 -- 172.21.15.29:0/4257692941 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4f006b030 con 0x7fe4f00c9000 2023-10-14T17:57:26.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.355+0000 7fe4f502b640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00c9000 0x7fe4f0069da0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098a0 tx=0x7fe4e002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa9763abe7f405 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:26.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.355+0000 7fe4f482a640 1 -- 172.21.15.29:0/4257692941 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0030840 con 0x7fe4f00c9000 2023-10-14T17:57:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f482a640 1 -- 172.21.15.29:0/4257692941 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4e00309a0 con 0x7fe4f00c9000 2023-10-14T17:57:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f482a640 1 -- 172.21.15.29:0/4257692941 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4e0030d30 con 0x7fe4f00c9000 2023-10-14T17:57:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 -- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00c9000 msgr2=0x7fe4f0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00c9000 0x7fe4f0069da0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fe4e00098a0 tx=0x7fe4e002b980 comp rx=0 tx=0).stop 2023-10-14T17:57:26.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 -- 172.21.15.29:0/4257692941 shutdown_connections 2023-10-14T17:57:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00c9000 0x7fe4f0069da0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/4257692941 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00addb0 0x7fe4f00caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.356+0000 7fe4f7a90640 1 -- 172.21.15.29:0/4257692941 >> 172.21.15.29:0/4257692941 conn(0x7fe4f00ce9b0 msgr2=0x7fe4f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.357+0000 7fe4f7a90640 1 -- 172.21.15.29:0/4257692941 shutdown_connections 2023-10-14T17:57:26.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.357+0000 7fe4f7a90640 1 -- 172.21.15.29:0/4257692941 wait complete. 2023-10-14T17:57:26.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.358+0000 7fe4f7a90640 1 Processor -- start 2023-10-14T17:57:26.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.358+0000 7fe4f7a90640 1 -- start start 2023-10-14T17:57:26.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00addb0 0x7fe4f00c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00c9000 0x7fe4f00b3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00b15a0 con 0x7fe4f006a2e0 2023-10-14T17:57:26.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00b23c0 con 0x7fe4f00addb0 2023-10-14T17:57:26.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.359+0000 7fe4f7a90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4f00b17b0 con 0x7fe4f00c9000 2023-10-14T17:57:26.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39300/0 (socket says 172.21.15.29:39300) 2023-10-14T17:57:26.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 -- 172.21.15.29:0/3469813591 learned_addr learned my addr 172.21.15.29:0/3469813591 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:26.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f602d640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00c9000 0x7fe4f00b3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f502b640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00addb0 0x7fe4f00c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 -- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00c9000 msgr2=0x7fe4f00b3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00c9000 0x7fe4f00b3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 -- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00addb0 msgr2=0x7fe4f00c8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00addb0 0x7fe4f00c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.360+0000 7fe4f582c640 1 -- 172.21.15.29:0/3469813591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4e00094d0 con 0x7fe4f006a2e0 2023-10-14T17:57:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4f582c640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00e930 tx=0x7fe4ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec009690 con 0x7fe4f006a2e0 2023-10-14T17:57:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4ec00ca00 con 0x7fe4f006a2e0 2023-10-14T17:57:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4ec00cdb0 con 0x7fe4f006a2e0 2023-10-14T17:57:26.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4f00b1a90 con 0x7fe4f006a2e0 2023-10-14T17:57:26.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.361+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4f00b2050 con 0x7fe4f006a2e0 2023-10-14T17:57:26.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.362+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4f004e160 con 0x7fe4f006a2e0 2023-10-14T17:57:26.369 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46bdd63640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.369 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46bdd63640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.369 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46bdd63640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.369 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46bdd63640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46bdd63640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46c156a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46c156a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46c156a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.370 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46c156a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.371 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:26.363+0000 7f46c156a640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:26.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.364+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fe4ec013470 con 0x7fe4f006a2e0 2023-10-14T17:57:26.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.365+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6571+0+0 (secure 0 0 0) 0x7fe4ec089610 con 0x7fe4f006a2e0 2023-10-14T17:57:26.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.370+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe4ec054870 con 0x7fe4f006a2e0 2023-10-14T17:57:26.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.563+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe4f006c5b0 con 0x7fe4f006a2e0 2023-10-14T17:57:26.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.564+0000 7fe4e67fc640 1 -- 172.21.15.29:0/3469813591 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v165) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe4ec057e90 con 0x7fe4f006a2e0 2023-10-14T17:57:26.565 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:57:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 msgr2=0x7fe4f00c5510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 secure :-1 s=READY pgs=1889 cs=0 l=1 rev1=1 crypto rx=0x7fe4ec00e930 tx=0x7fe4ec00edf0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 shutdown_connections 2023-10-14T17:57:26.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe4f00c9000 0x7fe4f00b3130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4f00addb0 0x7fe4f00c8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 --2- 172.21.15.29:0/3469813591 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4f006a2e0 0x7fe4f00c5510 unknown :-1 s=CLOSED pgs=1889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 >> 172.21.15.29:0/3469813591 conn(0x7fe4f00ce9b0 msgr2=0x7fe4f0071040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 shutdown_connections 2023-10-14T17:57:26.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.567+0000 7fe4f7a90640 1 -- 172.21.15.29:0/3469813591 wait complete. 2023-10-14T17:57:26.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:57:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.820+0000 7f2398d35640 1 Processor -- start 2023-10-14T17:57:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.820+0000 7f2398d35640 1 -- start start 2023-10-14T17:57:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.821+0000 7f2398d35640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415c9f0 0x7f23940c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2398d35640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 0x7f23940709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2398d35640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 0x7f23940ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2398d35640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23940cdec0 con 0x7f239415c9f0 2023-10-14T17:57:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2398d35640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2394160a90 con 0x7f2394070f20 2023-10-14T17:57:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2398d35640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23940cd900 con 0x7f23940c9230 2023-10-14T17:57:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415c9f0 0x7f23940c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f239415c9f0 0x7f23940c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39340/0 (socket says 172.21.15.29:39340) 2023-10-14T17:57:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 -- 172.21.15.29:0/1097975153 learned_addr learned my addr 172.21.15.29:0/1097975153 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2393fff640 1 --2- 172.21.15.29:0/1097975153 >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 0x7f23940ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 -- 172.21.15.29:0/1097975153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 msgr2=0x7f23940709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2392ffd640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 0x7f23940709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 0x7f23940709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 -- 172.21.15.29:0/1097975153 >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 msgr2=0x7f23940ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 --2- 172.21.15.29:0/1097975153 >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 0x7f23940ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f23937fe640 1 -- 172.21.15.29:0/1097975153 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23940cdbe0 con 0x7f239415c9f0 2023-10-14T17:57:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.822+0000 7f2393fff640 1 --2- 172.21.15.29:0/1097975153 >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 0x7f23940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.823+0000 7f2392ffd640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 0x7f23940709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f23937fe640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415c9f0 0x7f23940c8cf0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f23940c8c80 tx=0x7f238402b920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66ba2250f15f1008 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f23927fc640 1 -- 172.21.15.29:0/1097975153 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384031880 con 0x7f239415c9f0 2023-10-14T17:57:26.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f23927fc640 1 -- 172.21.15.29:0/1097975153 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23840319e0 con 0x7f239415c9f0 2023-10-14T17:57:26.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f23927fc640 1 -- 172.21.15.29:0/1097975153 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2384031d70 con 0x7f239415c9f0 2023-10-14T17:57:26.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f2398d35640 1 -- 172.21.15.29:0/1097975153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415c9f0 msgr2=0x7f23940c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.824+0000 7f2398d35640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415c9f0 0x7f23940c8cf0 secure :-1 s=READY pgs=1892 cs=0 l=1 rev1=1 crypto rx=0x7f23940c8c80 tx=0x7f238402b920 comp rx=0 tx=0).stop 2023-10-14T17:57:26.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 -- 172.21.15.29:0/1097975153 shutdown_connections 2023-10-14T17:57:26.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 --2- 172.21.15.29:0/1097975153 >> v2:172.21.15.186:3300/0 conn(0x7f2394070f20 0x7f23940ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23940c9230 0x7f23940709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 --2- 172.21.15.29:0/1097975153 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f239415c9f0 0x7f23940c8cf0 unknown :-1 s=CLOSED pgs=1892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 -- 172.21.15.29:0/1097975153 >> 172.21.15.29:0/1097975153 conn(0x7f23940ce6a0 msgr2=0x7f23940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 -- 172.21.15.29:0/1097975153 shutdown_connections 2023-10-14T17:57:26.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.825+0000 7f2398d35640 1 -- 172.21.15.29:0/1097975153 wait complete. 2023-10-14T17:57:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.826+0000 7f2398d35640 1 Processor -- start 2023-10-14T17:57:26.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.826+0000 7f2398d35640 1 -- start start 2023-10-14T17:57:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2394070f20 0x7f2394147bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415c9f0 0x7f239415f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23941482f0 con 0x7f23940c9230 2023-10-14T17:57:26.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2394149150 con 0x7f239415c9f0 2023-10-14T17:57:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.827+0000 7f2398d35640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2394148540 con 0x7f2394070f20 2023-10-14T17:57:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39346/0 (socket says 172.21.15.29:39346) 2023-10-14T17:57:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/905887461 learned_addr learned my addr 172.21.15.29:0/905887461 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:26.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f23937fe640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2394070f20 0x7f2394147bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2393fff640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415c9f0 0x7f239415f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2394070f20 msgr2=0x7f2394147bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2394070f20 0x7f2394147bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415c9f0 msgr2=0x7f239415f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415c9f0 0x7f239415f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.828+0000 7f2392ffd640 1 -- 172.21.15.29:0/905887461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23840094d0 con 0x7f23940c9230 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f2392ffd640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f237c00cf90 tx=0x7f237c00a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c00d840 con 0x7f23940c9230 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f237c00d9a0 con 0x7f23940c9230 2023-10-14T17:57:26.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f237c00dd30 con 0x7f23940c9230 2023-10-14T17:57:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2394148820 con 0x7f23940c9230 2023-10-14T17:57:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.829+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2394148de0 con 0x7f23940c9230 2023-10-14T17:57:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.830+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f239404e9b0 con 0x7f23940c9230 2023-10-14T17:57:26.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.831+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 90) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f237c015440 con 0x7f23940c9230 2023-10-14T17:57:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.833+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f237c089e60 con 0x7f23940c9230 2023-10-14T17:57:26.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:26.837+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f237c054fc0 con 0x7f23940c9230 2023-10-14T17:57:27.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.037+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f23940cab60 con 0x7f23940c9230 2023-10-14T17:57:27.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.038+0000 7f238bfff640 1 -- 172.21.15.29:0/905887461 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v165) v1 ==== 100+0+11 (secure 0 0 0) 0x7f237c01a020 con 0x7f23940c9230 2023-10-14T17:57:27.040 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:57:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 msgr2=0x7f23940c6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 secure :-1 s=READY pgs=1893 cs=0 l=1 rev1=1 crypto rx=0x7f237c00cf90 tx=0x7f237c00a7d0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 shutdown_connections 2023-10-14T17:57:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f239415c9f0 0x7f239415f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23940c9230 0x7f23940c6800 unknown :-1 s=CLOSED pgs=1893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 --2- 172.21.15.29:0/905887461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2394070f20 0x7f2394147bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 >> 172.21.15.29:0/905887461 conn(0x7f23940ce6a0 msgr2=0x7f23940aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 shutdown_connections 2023-10-14T17:57:27.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.041+0000 7f2398d35640 1 -- 172.21.15.29:0/905887461 wait complete. 2023-10-14T17:57:27.060 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:27.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.264+0000 7f2b5e7a5640 1 Processor -- start 2023-10-14T17:57:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.265+0000 7f2b5e7a5640 1 -- start start 2023-10-14T17:57:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.266+0000 7f2b5e7a5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b5806c040 0x7f2b5806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.266+0000 7f2b5e7a5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b580ad970 0x7f2b5815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.266+0000 7f2b5e7a5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5806c960 con 0x7f2b5815ebc0 2023-10-14T17:57:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.266+0000 7f2b5e7a5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5815b9d0 con 0x7f2b5806c040 2023-10-14T17:57:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.266+0000 7f2b5e7a5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5815bcb0 con 0x7f2b580ad970 2023-10-14T17:57:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b5806c040 0x7f2b5806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b5806c040 0x7f2b5806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:46470/0 (socket says 172.21.15.29:46470) 2023-10-14T17:57:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 -- 172.21.15.29:0/2770068602 learned_addr learned my addr 172.21.15.29:0/2770068602 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5cfa2640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b580ad970 0x7f2b5815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 -- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b580ad970 msgr2=0x7f2b5815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b580ad970 0x7f2b5815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 -- 172.21.15.29:0/2770068602 >> v1:172.21.15.186:6789/0 conn(0x7f2b5815ebc0 legacy=0x7f2b580cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.267+0000 7f2b5d7a3640 1 -- 172.21.15.29:0/2770068602 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5815bf90 con 0x7f2b5806c040 2023-10-14T17:57:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.269+0000 7f2b5d7a3640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5806c040 0x7f2b5806c420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2b540098a0 tx=0x7f2b5402b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=41370098ff35c0b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b4ffff640 1 -- 172.21.15.29:0/2770068602 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b54030840 con 0x7f2b5806c040 2023-10-14T17:57:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b4ffff640 1 -- 172.21.15.29:0/2770068602 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b540309a0 con 0x7f2b5806c040 2023-10-14T17:57:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b4ffff640 1 -- 172.21.15.29:0/2770068602 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b54030d30 con 0x7f2b5806c040 2023-10-14T17:57:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5806c040 msgr2=0x7f2b5806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5806c040 0x7f2b5806c420 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7f2b540098a0 tx=0x7f2b5402b980 comp rx=0 tx=0).stop 2023-10-14T17:57:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/2770068602 shutdown_connections 2023-10-14T17:57:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b580ad970 0x7f2b5815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/2770068602 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5806c040 0x7f2b5806c420 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/2770068602 >> 172.21.15.29:0/2770068602 conn(0x7f2b580ce6a0 msgr2=0x7f2b580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.270+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/2770068602 shutdown_connections 2023-10-14T17:57:27.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.271+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/2770068602 wait complete. 2023-10-14T17:57:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.272+0000 7f2b5e7a5640 1 Processor -- start 2023-10-14T17:57:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.272+0000 7f2b5e7a5640 1 -- start start 2023-10-14T17:57:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5806c040 0x7f2b58152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 0x7f2b580aed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580b3620 con 0x7f2b5815ebc0 2023-10-14T17:57:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580c6e50 con 0x7f2b580ad970 2023-10-14T17:57:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.273+0000 7f2b5e7a5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b580b38c0 con 0x7f2b5806c040 2023-10-14T17:57:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39362/0 (socket says 172.21.15.29:39362) 2023-10-14T17:57:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 -- 172.21.15.29:0/3064180227 learned_addr learned my addr 172.21.15.29:0/3064180227 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5d7a3640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5806c040 0x7f2b58152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5cfa2640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 0x7f2b580aed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 -- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5806c040 msgr2=0x7f2b58152b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5806c040 0x7f2b58152b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 -- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 msgr2=0x7f2b580aed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 0x7f2b580aed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5dfa4640 1 -- 172.21.15.29:0/3064180227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b540094d0 con 0x7f2b5815ebc0 2023-10-14T17:57:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.274+0000 7f2b5cfa2640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 0x7f2b580aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.275+0000 7f2b5dfa4640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c7a0 tx=0x7f2b5000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.282+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50003b70 con 0x7f2b5815ebc0 2023-10-14T17:57:27.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.283+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b50003cf0 con 0x7f2b5815ebc0 2023-10-14T17:57:27.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.283+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b50003ea0 con 0x7f2b5815ebc0 2023-10-14T17:57:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.283+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b580b3ba0 con 0x7f2b5815ebc0 2023-10-14T17:57:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.283+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b58153c10 con 0x7f2b5815ebc0 2023-10-14T17:57:27.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.285+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5806b330 con 0x7f2b5815ebc0 2023-10-14T17:57:27.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.287+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 91) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f2b5000aa90 con 0x7f2b5815ebc0 2023-10-14T17:57:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.288+0000 7f2b4dffb640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 0x7f2b3c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.288+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(165..165 src has 1..165) v4 ==== 6571+0+0 (secure 0 0 0) 0x7f2b50089490 con 0x7f2b5815ebc0 2023-10-14T17:57:27.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.288+0000 7f2b5d7a3640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 0x7f2b3c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.289+0000 7f2b5d7a3640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 0x7f2b3c06d1b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b54036000 tx=0x7f2b5403b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.295+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b50054450 con 0x7f2b5815ebc0 2023-10-14T17:57:27.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.542+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b580c7350 con 0x7f2b5815ebc0 2023-10-14T17:57:27.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.543+0000 7f2b4dffb640 1 -- 172.21.15.29:0/3064180227 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f2b580c7350 con 0x7f2b5815ebc0 2023-10-14T17:57:27.544 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:27.545 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:57:27.545 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:57:27.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 msgr2=0x7f2b3c06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 0x7f2b3c06d1b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2b54036000 tx=0x7f2b5403b040 comp rx=0 tx=0).stop 2023-10-14T17:57:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 msgr2=0x7f2b580ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 secure :-1 s=READY pgs=1894 cs=0 l=1 rev1=1 crypto rx=0x7f2b5000c7a0 tx=0x7f2b5000cc60 comp rx=0 tx=0).stop 2023-10-14T17:57:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 shutdown_connections 2023-10-14T17:57:27.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2b3c06acf0 0x7f2b3c06d1b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5815ebc0 0x7f2b580ac890 unknown :-1 s=CLOSED pgs=1894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b580ad970 0x7f2b580aed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 --2- 172.21.15.29:0/3064180227 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5806c040 0x7f2b58152b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.545+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 >> 172.21.15.29:0/3064180227 conn(0x7f2b580ce6a0 msgr2=0x7f2b5815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:27.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.546+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 shutdown_connections 2023-10-14T17:57:27.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.546+0000 7f2b5e7a5640 1 -- 172.21.15.29:0/3064180227 wait complete. 2023-10-14T17:57:27.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.775+0000 7f5ad480e640 1 Processor -- start 2023-10-14T17:57:27.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.776+0000 7f5ad480e640 1 -- start start 2023-10-14T17:57:27.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.777+0000 7f5ad480e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.777+0000 7f5ad480e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad015ae80 0x7f5ad00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.777+0000 7f5ad480e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ad015ed60 con 0x7f5ad00ccaa0 2023-10-14T17:57:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.777+0000 7f5ad480e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ad006b810 con 0x7f5ad015ae80 2023-10-14T17:57:27.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.777+0000 7f5ad480e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad00707e0 con 0x7f5ad015aaa0 2023-10-14T17:57:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad015ae80 0x7f5ad00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5ad015ae80 0x7f5ad00cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39380/0 (socket says 172.21.15.29:39380) 2023-10-14T17:57:27.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 -- 172.21.15.29:0/1152201028 learned_addr learned my addr 172.21.15.29:0/1152201028 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5ace59c640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 -- 172.21.15.29:0/1152201028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 msgr2=0x7f5ad015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:27.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 -- 172.21.15.29:0/1152201028 >> v1:172.21.15.186:6789/0 conn(0x7f5ad00ccaa0 legacy=0x7f5ad0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.778+0000 7f5acdd9b640 1 -- 172.21.15.29:0/1152201028 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad015ef90 con 0x7f5ad015ae80 2023-10-14T17:57:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.779+0000 7f5acdd9b640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00cc560 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009870 tx=0x7f5ab802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b48d2da071de33 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.780+0000 7f5acd59a640 1 -- 172.21.15.29:0/1152201028 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8031640 con 0x7f5ad015ae80 2023-10-14T17:57:27.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.780+0000 7f5acd59a640 1 -- 172.21.15.29:0/1152201028 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ab80317a0 con 0x7f5ad015ae80 2023-10-14T17:57:27.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.780+0000 7f5acd59a640 1 -- 172.21.15.29:0/1152201028 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ab8031b30 con 0x7f5ad015ae80 2023-10-14T17:57:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.780+0000 7f5ad480e640 1 -- 172.21.15.29:0/1152201028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 msgr2=0x7f5ad00cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.780+0000 7f5ad480e640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00cc560 secure :-1 s=READY pgs=1895 cs=0 l=1 rev1=1 crypto rx=0x7f5ab8009870 tx=0x7f5ab802b8f0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 -- 172.21.15.29:0/1152201028 shutdown_connections 2023-10-14T17:57:27.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00cc560 unknown :-1 s=CLOSED pgs=1895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 --2- 172.21.15.29:0/1152201028 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 -- 172.21.15.29:0/1152201028 >> 172.21.15.29:0/1152201028 conn(0x7f5ad00cb410 msgr2=0x7f5ad00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 -- 172.21.15.29:0/1152201028 shutdown_connections 2023-10-14T17:57:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.781+0000 7f5ad480e640 1 -- 172.21.15.29:0/1152201028 wait complete. 2023-10-14T17:57:27.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.783+0000 7f5ad480e640 1 Processor -- start 2023-10-14T17:57:27.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.783+0000 7f5ad480e640 1 -- start start 2023-10-14T17:57:27.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad00ccaa0 0x7f5ad01f6560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f3590 con 0x7f5ad015ae80 2023-10-14T17:57:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f6aa0 con 0x7f5ad00ccaa0 2023-10-14T17:57:27.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ad480e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ad01f2fd0 con 0x7f5ad015aaa0 2023-10-14T17:57:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5aced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5aced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39384/0 (socket says 172.21.15.29:39384) 2023-10-14T17:57:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5aced9d640 1 -- 172.21.15.29:0/943465337 learned_addr learned my addr 172.21.15.29:0/943465337 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5acdd9b640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5ace59c640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad00ccaa0 0x7f5ad01f6560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.784+0000 7f5aced9d640 1 -- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 msgr2=0x7f5ad00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5aced9d640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5aced9d640 1 -- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad00ccaa0 msgr2=0x7f5ad01f6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5aced9d640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad00ccaa0 0x7f5ad01f6560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:27.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5aced9d640 1 -- 172.21.15.29:0/943465337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab80094d0 con 0x7f5ad015ae80 2023-10-14T17:57:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5aced9d640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000b410 tx=0x7f5ac000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0004020 con 0x7f5ad015ae80 2023-10-14T17:57:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ac0004180 con 0x7f5ad015ae80 2023-10-14T17:57:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ac0011630 con 0x7f5ad015ae80 2023-10-14T17:57:27.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ad01f32b0 con 0x7f5ad015ae80 2023-10-14T17:57:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.785+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ad01f6040 con 0x7f5ad015ae80 2023-10-14T17:57:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.787+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ad01f5bc0 con 0x7f5ad015ae80 2023-10-14T17:57:27.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.788+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 91) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5ac0011790 con 0x7f5ad015ae80 2023-10-14T17:57:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.789+0000 7f5ac6ffd640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 0x7f5ab006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.789+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f5ac00893a0 con 0x7f5ad015ae80 2023-10-14T17:57:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.789+0000 7f5ace59c640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 0x7f5ab006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:27.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.790+0000 7f5ace59c640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 0x7f5ab006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ad015db30 tx=0x7f5ac800b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:27.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.793+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5ad01f5bc0 con 0x7f5ad015ae80 2023-10-14T17:57:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.993+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ad01f5bc0 con 0x7f5ad015ae80 2023-10-14T17:57:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.996+0000 7f5ac6ffd640 1 -- 172.21.15.29:0/943465337 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+17890 (secure 0 0 0) 0x7f5ad01f5bc0 con 0x7f5ad015ae80 2023-10-14T17:57:27.998 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:27.998 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":166,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:27.269798+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":166,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.33","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3687245545":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6833/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6832/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3663692422":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3645494831":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1574130967":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/2381053627":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1627586322":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3251497707":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/80005070":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/2913425840":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/3470978219":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/1610794407":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6833/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6832/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/542195718":"2023-10-15T17:01:29.273525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 msgr2=0x7f5ab006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 0x7f5ab006d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5ad015db30 tx=0x7f5ac800b040 comp rx=0 tx=0).stop 2023-10-14T17:57:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 msgr2=0x7f5ad00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 secure :-1 s=READY pgs=1896 cs=0 l=1 rev1=1 crypto rx=0x7f5ac000b410 tx=0x7f5ac000b8d0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 shutdown_connections 2023-10-14T17:57:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f5ab006acc0 0x7f5ab006d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5ad015ae80 0x7f5ad00acdd0 unknown :-1 s=CLOSED pgs=1896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5ad015aaa0 0x7f5ad00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 --2- 172.21.15.29:0/943465337 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5ad00ccaa0 0x7f5ad01f6560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:27.999+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 >> 172.21.15.29:0/943465337 conn(0x7f5ad00cb410 msgr2=0x7f5ad00b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.000+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 shutdown_connections 2023-10-14T17:57:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.000+0000 7f5ad480e640 1 -- 172.21.15.29:0/943465337 wait complete. 2023-10-14T17:57:28.020 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T17:57:28.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:28.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.238+0000 7fa8edd1d640 1 Processor -- start 2023-10-14T17:57:28.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.239+0000 7fa8edd1d640 1 -- start start 2023-10-14T17:57:28.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.240+0000 7fa8edd1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8e806fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.240+0000 7fa8edd1d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e8159790 con 0x7fa8e80cc920 2023-10-14T17:57:28.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.240+0000 7fa8edd1d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8e80c9b20 con 0x7fa8e81593b0 2023-10-14T17:57:28.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.240+0000 7fa8edd1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e80c9e00 con 0x7fa8e80caa10 2023-10-14T17:57:28.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.241+0000 7fa8e7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8e80cc920 0x7fa8e80ccdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34950/0 (socket says 172.21.15.29:34950) 2023-10-14T17:57:28.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.241+0000 7fa8e7fff640 1 -- 172.21.15.29:0/3808653205 learned_addr learned my addr 172.21.15.29:0/3808653205 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.241+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8e806fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.241+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1451941435 0 0) 0x7fa8e8159790 con 0x7fa8e80cc920 2023-10-14T17:57:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.241+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8d4003520 con 0x7fa8e80cc920 2023-10-14T17:57:28.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 711947202 0 0) 0x7fa8d4003520 con 0x7fa8e80cc920 2023-10-14T17:57:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 msgr2=0x7fa8e806fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8e806fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 >> v1:172.21.15.186:6789/0 conn(0x7fa8e81593b0 legacy=0x7fa8e806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e80ca0e0 con 0x7fa8e80cc920 2023-10-14T17:57:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa8e00019f0 con 0x7fa8e80cc920 2023-10-14T17:57:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 >> v1:172.21.15.29:6789/0 conn(0x7fa8e80cc920 legacy=0x7fa8e80ccdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8d40039a0 0x7fa8d4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d40047c0 0x7fa8d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8d400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e80c9e00 con 0x7fa8d40047c0 2023-10-14T17:57:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40054e0 con 0x7fa8d40039a0 2023-10-14T17:57:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d400e0d0 con 0x7fa8e80caa10 2023-10-14T17:57:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d40047c0 0x7fa8d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8d40039a0 0x7fa8d4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8e7fff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8d400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.242+0000 7fa8dffff640 1 -- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 msgr2=0x7fa8d400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.243+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8e80caa10 0x7fa8d400db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.243+0000 7fa8dffff640 1 -- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8d40039a0 msgr2=0x7fa8d4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:28.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.243+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8d40039a0 0x7fa8d4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.243+0000 7fa8dffff640 1 -- 172.21.15.29:0/3808653205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d4003520 con 0x7fa8d40047c0 2023-10-14T17:57:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.243+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8d40039a0 0x7fa8d4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d40047c0 0x7fa8d4004c40 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009ac0 tx=0x7fa8d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26340215281a038e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d002c6b0 con 0x7fa8d40047c0 2023-10-14T17:57:28.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d002c810 con 0x7fa8d40047c0 2023-10-14T17:57:28.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8e6ffd640 1 -- 172.21.15.29:0/3808653205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d002cba0 con 0x7fa8d40047c0 2023-10-14T17:57:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3808653205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d40047c0 msgr2=0x7fa8d4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.244+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3808653205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8d40047c0 0x7fa8d4004c40 secure :-1 s=READY pgs=1898 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0009ac0 tx=0x7fa8d40057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.245+0000 7fa8e7fff640 1 -- 172.21.15.29:0/3808653205 reap_dead start 2023-10-14T17:57:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.245+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3808653205 shutdown_connections 2023-10-14T17:57:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.245+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3808653205 >> 172.21.15.29:0/3808653205 conn(0x7fa8e80cb410 msgr2=0x7fa8e80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.245+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3808653205 shutdown_connections 2023-10-14T17:57:28.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.245+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3808653205 wait complete. 2023-10-14T17:57:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.246+0000 7fa8edd1d640 1 Processor -- start 2023-10-14T17:57:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.247+0000 7fa8edd1d640 1 -- start start 2023-10-14T17:57:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.247+0000 7fa8edd1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 0x7fa8e8156600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.247+0000 7fa8edd1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.247+0000 7fa8edd1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 0x7fa8e81f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.247+0000 7fa8edd1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81f23c0 con 0x7fa8e80b1530 2023-10-14T17:57:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8edd1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81599c0 con 0x7fa8e81f19f0 2023-10-14T17:57:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8edd1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8e81553b0 con 0x7fa8d40047c0 2023-10-14T17:57:28.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39392/0 (socket says 172.21.15.29:39392) 2023-10-14T17:57:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 -- 172.21.15.29:0/3930137495 learned_addr learned my addr 172.21.15.29:0/3930137495 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 0x7fa8e8156600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8e7fff640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 0x7fa8e81f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 -- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 msgr2=0x7fa8e8156600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 0x7fa8e8156600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 -- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 msgr2=0x7fa8e81f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 0x7fa8e81f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8dffff640 1 -- 172.21.15.29:0/3930137495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e8156c90 con 0x7fa8e80b1530 2023-10-14T17:57:28.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 0x7fa8e8156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.248+0000 7fa8e7fff640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 0x7fa8e81f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8dffff640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0037040 tx=0x7fa8d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0003810 con 0x7fa8e80b1530 2023-10-14T17:57:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d0003990 con 0x7fa8e80b1530 2023-10-14T17:57:28.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8d0003d40 con 0x7fa8e80b1530 2023-10-14T17:57:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8e8156f70 con 0x7fa8e80b1530 2023-10-14T17:57:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.249+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8e8157530 con 0x7fa8e80b1530 2023-10-14T17:57:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.250+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8e804e160 con 0x7fa8e80b1530 2023-10-14T17:57:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.251+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 91) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fa8d0040450 con 0x7fa8e80b1530 2023-10-14T17:57:28.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.252+0000 7fa8e4ff9640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 0x7fa8c806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.252+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fa8d00ab550 con 0x7fa8e80b1530 2023-10-14T17:57:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.253+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 0x7fa8c806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.253+0000 7fa8e77fe640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 0x7fa8c806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8157860 tx=0x7fa8d8009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.257+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8d00765b0 con 0x7fa8e80b1530 2023-10-14T17:57:28.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.499+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa8e80b4910 con 0x7fa8e80b1530 2023-10-14T17:57:28.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.500+0000 7fa8e4ff9640 1 -- 172.21.15.29:0/3930137495 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7fa8d0079bd0 con 0x7fa8e80b1530 2023-10-14T17:57:28.501 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:28.502 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:57:28.502 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:57:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.502+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 msgr2=0x7fa8c806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.502+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 0x7fa8c806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa8e8157860 tx=0x7fa8d8009040 comp rx=0 tx=0).stop 2023-10-14T17:57:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.502+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 msgr2=0x7fa8e8154ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.502+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 secure :-1 s=READY pgs=1899 cs=0 l=1 rev1=1 crypto rx=0x7fa8d0037040 tx=0x7fa8d0004730 comp rx=0 tx=0).stop 2023-10-14T17:57:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 shutdown_connections 2023-10-14T17:57:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa8c806acc0 0x7fa8c806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa8e81f19f0 0x7fa8e81f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8e80b1530 0x7fa8e8154ad0 unknown :-1 s=CLOSED pgs=1899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 --2- 172.21.15.29:0/3930137495 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8d40047c0 0x7fa8e8156600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 >> 172.21.15.29:0/3930137495 conn(0x7fa8e80cb410 msgr2=0x7fa8e80b0670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 shutdown_connections 2023-10-14T17:57:28.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.503+0000 7fa8edd1d640 1 -- 172.21.15.29:0/3930137495 wait complete. 2023-10-14T17:57:28.522 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:28.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.724+0000 7f29d4a75640 1 Processor -- start 2023-10-14T17:57:28.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.724+0000 7f29d4a75640 1 -- start start 2023-10-14T17:57:28.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29d4a75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29d4a75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d0070fd0 0x7f29d00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29d4a75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29d0071450 con 0x7f29d00adf30 2023-10-14T17:57:28.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29d4a75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29d00cc600 con 0x7f29d0070fd0 2023-10-14T17:57:28.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29d4a75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d00cc8e0 con 0x7f29d00adb50 2023-10-14T17:57:28.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29cdd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d0070fd0 0x7f29d00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29cdd9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f29d0070fd0 0x7f29d00b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39418/0 (socket says 172.21.15.29:39418) 2023-10-14T17:57:28.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29cdd9b640 1 -- 172.21.15.29:0/258657194 learned_addr learned my addr 172.21.15.29:0/258657194 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:28.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.726+0000 7f29ce59c640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.727+0000 7f29cdd9b640 1 -- 172.21.15.29:0/258657194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 msgr2=0x7f29d006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.727+0000 7f29cdd9b640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.727+0000 7f29cdd9b640 1 -- 172.21.15.29:0/258657194 >> v1:172.21.15.29:6789/0 conn(0x7f29d00adf30 legacy=0x7f29d0164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.727+0000 7f29cdd9b640 1 -- 172.21.15.29:0/258657194 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d00ccbc0 con 0x7f29d0070fd0 2023-10-14T17:57:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.728+0000 7f29cdd9b640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d00b0410 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009870 tx=0x7f29c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8cb8e5994b1bd1d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.728+0000 7f29cd59a640 1 -- 172.21.15.29:0/258657194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4030840 con 0x7f29d0070fd0 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.728+0000 7f29cd59a640 1 -- 172.21.15.29:0/258657194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c40309a0 con 0x7f29d0070fd0 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.728+0000 7f29cd59a640 1 -- 172.21.15.29:0/258657194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c4030d30 con 0x7f29d0070fd0 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 -- 172.21.15.29:0/258657194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 msgr2=0x7f29d00b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d00b0410 secure :-1 s=READY pgs=1903 cs=0 l=1 rev1=1 crypto rx=0x7f29c4009870 tx=0x7f29c402b980 comp rx=0 tx=0).stop 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 -- 172.21.15.29:0/258657194 shutdown_connections 2023-10-14T17:57:28.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d00b0410 unknown :-1 s=CLOSED pgs=1903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 --2- 172.21.15.29:0/258657194 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 -- 172.21.15.29:0/258657194 >> 172.21.15.29:0/258657194 conn(0x7f29d00cb410 msgr2=0x7f29d00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:28.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 -- 172.21.15.29:0/258657194 shutdown_connections 2023-10-14T17:57:28.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.729+0000 7f29d4a75640 1 -- 172.21.15.29:0/258657194 wait complete. 2023-10-14T17:57:28.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.731+0000 7f29d4a75640 1 Processor -- start 2023-10-14T17:57:28.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.731+0000 7f29d4a75640 1 -- start start 2023-10-14T17:57:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d0163d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29d00adf30 0x7f29d00b3b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0162ab0 con 0x7f29d0070fd0 2023-10-14T17:57:28.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0162cc0 con 0x7f29d00adf30 2023-10-14T17:57:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29d4a75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29d0162fa0 con 0x7f29d00adb50 2023-10-14T17:57:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29ce59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39422/0 (socket says 172.21.15.29:39422) 2023-10-14T17:57:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.732+0000 7f29ce59c640 1 -- 172.21.15.29:0/1882916426 learned_addr learned my addr 172.21.15.29:0/1882916426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:28.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29cdd9b640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d0163d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 -- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 msgr2=0x7f29d0163d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d0163d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 -- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29d00adf30 msgr2=0x7f29d00b3b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29d00adf30 0x7f29d00b3b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 -- 172.21.15.29:0/1882916426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c40094d0 con 0x7f29d0070fd0 2023-10-14T17:57:28.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29ce59c640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d800 tx=0x7f29c000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c000b6c0 con 0x7f29d0070fd0 2023-10-14T17:57:28.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29c000b820 con 0x7f29d0070fd0 2023-10-14T17:57:28.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29c000bbb0 con 0x7f29d0070fd0 2023-10-14T17:57:28.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.733+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d0163280 con 0x7f29d0070fd0 2023-10-14T17:57:28.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.734+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d0163840 con 0x7f29d0070fd0 2023-10-14T17:57:28.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.735+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d01639d0 con 0x7f29d0070fd0 2023-10-14T17:57:28.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.736+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 91) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f29c000bd10 con 0x7f29d0070fd0 2023-10-14T17:57:28.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.737+0000 7f29beffd640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 0x7f29b006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:28.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.737+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f29c0089c30 con 0x7f29d0070fd0 2023-10-14T17:57:28.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.737+0000 7f29cdd9b640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 0x7f29b006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:28.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.738+0000 7f29cdd9b640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 0x7f29b006d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29c4036000 tx=0x7f29c403b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:28.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.741+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f29c0054bc0 con 0x7f29d0070fd0 2023-10-14T17:57:28.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.756+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 92) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f29c0054f90 con 0x7f29d0070fd0 2023-10-14T17:57:28.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.979+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f29d004e160 con 0x7f29d0070fd0 2023-10-14T17:57:28.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.980+0000 7f29beffd640 1 -- 172.21.15.29:0/1882916426 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+4918 (secure 0 0 0) 0x7f29c0070ad0 con 0x7f29d0070fd0 2023-10-14T17:57:28.982 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:28.982 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":74,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70}],"filesystems":[{"mdsmap":{"epoch":74,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:01:14.133527+0000","modified":"2023-10-14T17:01:17.203772+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9105},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9105":{"gid":9105,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1190525894","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1190525894},{"type":"v1","addr":"172.21.15.29:6837","nonce":1190525894}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[19],"metadata_pool":18,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":9}]} 2023-10-14T17:57:28.982 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 74 2023-10-14T17:57:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 msgr2=0x7f29b006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 0x7f29b006d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f29c4036000 tx=0x7f29c403b040 comp rx=0 tx=0).stop 2023-10-14T17:57:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 msgr2=0x7f29d015ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 secure :-1 s=READY pgs=1904 cs=0 l=1 rev1=1 crypto rx=0x7f29c000d800 tx=0x7f29c000dcc0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 shutdown_connections 2023-10-14T17:57:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f29b006acc0 0x7f29b006d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f29d00adf30 0x7f29d00b3b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f29d00adb50 0x7f29d0163d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 --2- 172.21.15.29:0/1882916426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f29d0070fd0 0x7f29d015ad50 unknown :-1 s=CLOSED pgs=1904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 >> 172.21.15.29:0/1882916426 conn(0x7f29d00cb410 msgr2=0x7f29d0160ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 shutdown_connections 2023-10-14T17:57:28.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:28.983+0000 7f29d4a75640 1 -- 172.21.15.29:0/1882916426 wait complete. 2023-10-14T17:57:29.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:29.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.218+0000 7f47034b4640 1 Processor -- start 2023-10-14T17:57:29.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.218+0000 7f47034b4640 1 -- start start 2023-10-14T17:57:29.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.220+0000 7f47034b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.220+0000 7f47034b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46fc0c9230 0x7f46fc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.220+0000 7f47034b4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46fc160a90 con 0x7f46fc070f20 2023-10-14T17:57:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.220+0000 7f47034b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0cdec0 con 0x7f46fc15c9f0 2023-10-14T17:57:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.220+0000 7f47034b4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46fc0cd900 con 0x7f46fc0c9230 2023-10-14T17:57:29.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39428/0 (socket says 172.21.15.29:39428) 2023-10-14T17:57:29.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 -- 172.21.15.29:0/1816563794 learned_addr learned my addr 172.21.15.29:0/1816563794 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f4701cb1640 1 --2- 172.21.15.29:0/1816563794 >> v2:172.21.15.186:3300/0 conn(0x7f46fc0c9230 0x7f46fc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 -- 172.21.15.29:0/1816563794 >> v2:172.21.15.186:3300/0 conn(0x7f46fc0c9230 msgr2=0x7f46fc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 --2- 172.21.15.29:0/1816563794 >> v2:172.21.15.186:3300/0 conn(0x7f46fc0c9230 0x7f46fc0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 -- 172.21.15.29:0/1816563794 >> v1:172.21.15.186:6789/0 conn(0x7f46fc070f20 legacy=0x7f46fc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.221+0000 7f47024b2640 1 -- 172.21.15.29:0/1816563794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46fc0cdbe0 con 0x7f46fc15c9f0 2023-10-14T17:57:29.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.222+0000 7f47024b2640 1 --2- 172.21.15.29:0/1816563794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b1260 tx=0x7f46f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=519f4d921cbf7c86 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.223+0000 7f47014b0640 1 -- 172.21.15.29:0/1816563794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8031870 con 0x7f46fc15c9f0 2023-10-14T17:57:29.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.223+0000 7f47014b0640 1 -- 172.21.15.29:0/1816563794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f80319d0 con 0x7f46fc15c9f0 2023-10-14T17:57:29.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.223+0000 7f47014b0640 1 -- 172.21.15.29:0/1816563794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8031d60 con 0x7f46fc15c9f0 2023-10-14T17:57:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 -- 172.21.15.29:0/1816563794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc15c9f0 msgr2=0x7f46fc0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 --2- 172.21.15.29:0/1816563794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 secure :-1 s=READY pgs=1905 cs=0 l=1 rev1=1 crypto rx=0x7f46fc0b1260 tx=0x7f46f802b910 comp rx=0 tx=0).stop 2023-10-14T17:57:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 -- 172.21.15.29:0/1816563794 shutdown_connections 2023-10-14T17:57:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 --2- 172.21.15.29:0/1816563794 >> v2:172.21.15.186:3300/0 conn(0x7f46fc0c9230 0x7f46fc0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 --2- 172.21.15.29:0/1816563794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc15c9f0 0x7f46fc0c8cf0 unknown :-1 s=CLOSED pgs=1905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 -- 172.21.15.29:0/1816563794 >> 172.21.15.29:0/1816563794 conn(0x7f46fc0ce6a0 msgr2=0x7f46fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 -- 172.21.15.29:0/1816563794 shutdown_connections 2023-10-14T17:57:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.224+0000 7f47034b4640 1 -- 172.21.15.29:0/1816563794 wait complete. 2023-10-14T17:57:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.226+0000 7f47034b4640 1 Processor -- start 2023-10-14T17:57:29.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.226+0000 7f47034b4640 1 -- start start 2023-10-14T17:57:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc070f20 0x7f46fc14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 0x7f46fc0af120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc06b750 con 0x7f46fc0c9230 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc06aad0 con 0x7f46fc070f20 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47034b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46fc06adb0 con 0x7f46fc15c9f0 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f4701cb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f4701cb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39438/0 (socket says 172.21.15.29:39438) 2023-10-14T17:57:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f4701cb1640 1 -- 172.21.15.29:0/2737456373 learned_addr learned my addr 172.21.15.29:0/2737456373 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.227+0000 7f47024b2640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc070f20 0x7f46fc14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4702cb3640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 0x7f46fc0af120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 -- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 msgr2=0x7f46fc0af120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 0x7f46fc0af120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 -- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc070f20 msgr2=0x7f46fc14bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc070f20 0x7f46fc14bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 -- 172.21.15.29:0/2737456373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46f80094d0 con 0x7f46fc0c9230 2023-10-14T17:57:29.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4702cb3640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 0x7f46fc0af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f4701cb1640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d800 tx=0x7f46ec00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec00b6c0 con 0x7f46fc0c9230 2023-10-14T17:57:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46ec00b820 con 0x7f46fc0c9230 2023-10-14T17:57:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46fc06b090 con 0x7f46fc0c9230 2023-10-14T17:57:29.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.228+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46ec00bbb0 con 0x7f46fc0c9230 2023-10-14T17:57:29.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.229+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46fc14c050 con 0x7f46fc0c9230 2023-10-14T17:57:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.230+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46fc0668c0 con 0x7f46fc0c9230 2023-10-14T17:57:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.231+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f46ec00bd10 con 0x7f46fc0c9230 2023-10-14T17:57:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.232+0000 7f46f2ffd640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 0x7f46e006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.232+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6615+0+0 (secure 0 0 0) 0x7f46ec089fe0 con 0x7f46fc0c9230 2023-10-14T17:57:29.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.232+0000 7f47024b2640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 0x7f46e006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.233+0000 7f47024b2640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 0x7f46e006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46f8036000 tx=0x7f46f803c040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.237+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46ec0bcfe0 con 0x7f46fc0c9230 2023-10-14T17:57:29.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.437+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46fc161670 con 0x7f46fc0c9230 2023-10-14T17:57:29.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.440+0000 7f46f2ffd640 1 -- 172.21.15.29:0/2737456373 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v166) v1 ==== 74+0+17890 (secure 0 0 0) 0x7f46ec0586e0 con 0x7f46fc0c9230 2023-10-14T17:57:29.442 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:29.442 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":166,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:27.269798+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":18,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:01:12.106201+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"153","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":19,"pool_name":"cephfs_data","create_time":"2023-10-14T17:01:13.162539+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"166","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":113,"snap_epoch":166,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"19.33","mappings":[{"from":2,"to":0}]},{"pgid":"19.3a","mappings":[{"from":2,"to":0}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3687245545":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6833/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6832/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3663692422":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3645494831":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1574130967":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/2381053627":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1627586322":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3251497707":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/80005070":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/2913425840":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/3470978219":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/1610794407":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6833/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6832/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/542195718":"2023-10-15T17:01:29.273525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":3,"length":55},{"begin":59,"length":55}]}],"new_removed_snaps":[{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 msgr2=0x7f46e006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 0x7f46e006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f46f8036000 tx=0x7f46f803c040 comp rx=0 tx=0).stop 2023-10-14T17:57:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 msgr2=0x7f46fc0aebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 secure :-1 s=READY pgs=1906 cs=0 l=1 rev1=1 crypto rx=0x7f46ec00d800 tx=0x7f46ec00dcc0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 shutdown_connections 2023-10-14T17:57:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f46e006af00 0x7f46e006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46fc15c9f0 0x7f46fc0af120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46fc0c9230 0x7f46fc0aebe0 unknown :-1 s=CLOSED pgs=1906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 --2- 172.21.15.29:0/2737456373 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46fc070f20 0x7f46fc14bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.443+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 >> 172.21.15.29:0/2737456373 conn(0x7f46fc0ce6a0 msgr2=0x7f46fc15da00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:29.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.444+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 shutdown_connections 2023-10-14T17:57:29.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.444+0000 7f47034b4640 1 -- 172.21.15.29:0/2737456373 wait complete. 2023-10-14T17:57:29.464 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T17:57:29.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.693+0000 7fd2be3ef640 1 Processor -- start 2023-10-14T17:57:29.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.694+0000 7fd2be3ef640 1 -- start start 2023-10-14T17:57:29.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.695+0000 7fd2be3ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2b80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.695+0000 7fd2be3ef640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2b80ccfb0 0x7fd2b80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.695+0000 7fd2be3ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b815b9d0 con 0x7fd2b806c040 2023-10-14T17:57:29.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.695+0000 7fd2be3ef640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2b815bcb0 con 0x7fd2b80ccfb0 2023-10-14T17:57:29.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.695+0000 7fd2be3ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b815bf90 con 0x7fd2b80ad970 2023-10-14T17:57:29.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2bd3ed640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd2b806c040 0x7fd2b806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:34968/0 (socket says 172.21.15.29:34968) 2023-10-14T17:57:29.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2bd3ed640 1 -- 172.21.15.29:0/4234274731 learned_addr learned my addr 172.21.15.29:0/4234274731 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:29.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2bdbee640 1 --2- 172.21.15.29:0/4234274731 >> v2:172.21.15.186:3300/0 conn(0x7fd2b80ccfb0 0x7fd2b80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2b80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 42255012 0 0) 0x7fd2b815b9d0 con 0x7fd2b806c040 2023-10-14T17:57:29.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.696+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2a4003520 con 0x7fd2b806c040 2023-10-14T17:57:29.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3552077760 0 0) 0x7fd2a4003520 con 0x7fd2b806c040 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 msgr2=0x7fd2b80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2b80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 >> v2:172.21.15.186:3300/0 conn(0x7fd2b80ccfb0 msgr2=0x7fd2b80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 --2- 172.21.15.29:0/4234274731 >> v2:172.21.15.186:3300/0 conn(0x7fd2b80ccfb0 0x7fd2b80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80cc270 con 0x7fd2b806c040 2023-10-14T17:57:29.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd2b4001670 con 0x7fd2b806c040 2023-10-14T17:57:29.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 >> v1:172.21.15.29:6789/0 conn(0x7fd2b806c040 legacy=0x7fd2b806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a4003970 0x7fd2a4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a40047c0 0x7fd2a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b815bcb0 con 0x7fd2a40047c0 2023-10-14T17:57:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b815bf90 con 0x7fd2a4003970 2023-10-14T17:57:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2a400b980 con 0x7fd2b80ad970 2023-10-14T17:57:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a40047c0 0x7fd2a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2bd3ed640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a4003970 0x7fd2a4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.697+0000 7fd2bdbee640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bcbec640 1 -- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 msgr2=0x7fd2a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80ad970 0x7fd2a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bcbec640 1 -- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a4003970 msgr2=0x7fd2a4003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:29.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a4003970 0x7fd2a4003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bcbec640 1 -- 172.21.15.29:0/4234274731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2a4003520 con 0x7fd2a40047c0 2023-10-14T17:57:29.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.698+0000 7fd2bd3ed640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a4003970 0x7fd2a4003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:29.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.699+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a40047c0 0x7fd2a4004c40 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009ac0 tx=0x7fd2a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77e13a44e7edea28 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a802c6b0 con 0x7fd2a40047c0 2023-10-14T17:57:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2a802c810 con 0x7fd2a40047c0 2023-10-14T17:57:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2affff640 1 -- 172.21.15.29:0/4234274731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2a802cba0 con 0x7fd2a40047c0 2023-10-14T17:57:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 -- 172.21.15.29:0/4234274731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a40047c0 msgr2=0x7fd2a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/4234274731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2a40047c0 0x7fd2a4004c40 secure :-1 s=READY pgs=1908 cs=0 l=1 rev1=1 crypto rx=0x7fd2a8009ac0 tx=0x7fd2a40057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2bdbee640 1 -- 172.21.15.29:0/4234274731 reap_dead start 2023-10-14T17:57:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 -- 172.21.15.29:0/4234274731 shutdown_connections 2023-10-14T17:57:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 -- 172.21.15.29:0/4234274731 >> 172.21.15.29:0/4234274731 conn(0x7fd2b80ce6a0 msgr2=0x7fd2b80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 -- 172.21.15.29:0/4234274731 shutdown_connections 2023-10-14T17:57:29.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.700+0000 7fd2be3ef640 1 -- 172.21.15.29:0/4234274731 wait complete. 2023-10-14T17:57:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.702+0000 7fd2be3ef640 1 Processor -- start 2023-10-14T17:57:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.702+0000 7fd2be3ef640 1 -- start start 2023-10-14T17:57:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a40047c0 0x7fd2b80c7200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 0x7fd2b81f2e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80aecf0 con 0x7fd2b81f00b0 2023-10-14T17:57:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80ae100 con 0x7fd2a40047c0 2023-10-14T17:57:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.703+0000 7fd2be3ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2b80ae350 con 0x7fd2b80c6130 2023-10-14T17:57:29.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39450/0 (socket says 172.21.15.29:39450) 2023-10-14T17:57:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 -- 172.21.15.29:0/2914395939 learned_addr learned my addr 172.21.15.29:0/2914395939 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 0x7fd2b81f2e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 -- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 msgr2=0x7fd2b81f2e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 0x7fd2b81f2e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 -- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a40047c0 msgr2=0x7fd2b80c7200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a40047c0 0x7fd2b80c7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bdbee640 1 -- 172.21.15.29:0/2914395939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b80ae630 con 0x7fd2b81f00b0 2023-10-14T17:57:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.704+0000 7fd2bcbec640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 0x7fd2b81f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:29.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2bdbee640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80cd390 tx=0x7fd2b00115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b000e070 con 0x7fd2b81f00b0 2023-10-14T17:57:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2b0011e10 con 0x7fd2b81f00b0 2023-10-14T17:57:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b80ae910 con 0x7fd2b81f00b0 2023-10-14T17:57:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2b0003990 con 0x7fd2b81f00b0 2023-10-14T17:57:29.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.705+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b81f3370 con 0x7fd2b81f00b0 2023-10-14T17:57:29.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.706+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2b806b330 con 0x7fd2b81f00b0 2023-10-14T17:57:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.707+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd2b0003af0 con 0x7fd2b81f00b0 2023-10-14T17:57:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.709+0000 7fd2adffb640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 0x7fd29c06d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.709+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(166..166 src has 1..166) v4 ==== 6615+0+0 (secure 0 0 0) 0x7fd2b008a100 con 0x7fd2b81f00b0 2023-10-14T17:57:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.709+0000 7fd2bd3ed640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 0x7fd29c06d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:29.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.710+0000 7fd2bd3ed640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 0x7fd29c06d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4001200 tx=0x7fd2b4001110 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:29.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.713+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2b0054db0 con 0x7fd2b81f00b0 2023-10-14T17:57:29.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.952+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fd2b80aeaa0 con 0x7fd2b81f00b0 2023-10-14T17:57:29.954 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T17:57:29.953+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T17:57:29.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.957+0000 7fd2adffb640 1 -- 172.21.15.29:0/2914395939 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v75) v1 ==== 161+0+0 (secure 0 0 0) 0x7fd2b00583d0 con 0x7fd2b81f00b0 2023-10-14T17:57:29.958 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T17:57:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.960+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 msgr2=0x7fd29c06d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.960+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 0x7fd29c06d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd2b4001200 tx=0x7fd2b4001110 comp rx=0 tx=0).stop 2023-10-14T17:57:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.960+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 msgr2=0x7fd2b80b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:29.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.960+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 secure :-1 s=READY pgs=1909 cs=0 l=1 rev1=1 crypto rx=0x7fd2b80cd390 tx=0x7fd2b00115a0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 shutdown_connections 2023-10-14T17:57:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd29c06af30 0x7fd29c06d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2b81f00b0 0x7fd2b80b2b10 unknown :-1 s=CLOSED pgs=1909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2b80c6130 0x7fd2b81f2e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 --2- 172.21.15.29:0/2914395939 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2a40047c0 0x7fd2b80c7200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 >> 172.21.15.29:0/2914395939 conn(0x7fd2b80ce6a0 msgr2=0x7fd2b815da80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.961+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 shutdown_connections 2023-10-14T17:57:29.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:29.962+0000 7fd2be3ef640 1 -- 172.21.15.29:0/2914395939 wait complete. 2023-10-14T17:57:29.982 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T17:57:30.061 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T17:57:30.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.180+0000 7f9bcdf67640 1 Processor -- start 2023-10-14T17:57:30.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.181+0000 7f9bcdf67640 1 -- start start 2023-10-14T17:57:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.182+0000 7f9bcdf67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.182+0000 7f9bcdf67640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bc8070450 0x7f9bc80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.182+0000 7f9bcdf67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bc8070df0 con 0x7f9bc806c100 2023-10-14T17:57:30.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.182+0000 7f9bcdf67640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bc815e490 con 0x7f9bc8070450 2023-10-14T17:57:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.182+0000 7f9bcdf67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc815aef0 con 0x7f9bc8159950 2023-10-14T17:57:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bc8070450 0x7f9bc80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bc8070450 0x7f9bc80708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39472/0 (socket says 172.21.15.29:39472) 2023-10-14T17:57:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 -- 172.21.15.29:0/2004488556 learned_addr learned my addr 172.21.15.29:0/2004488556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:30.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bccf65640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 -- 172.21.15.29:0/2004488556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 msgr2=0x7f9bc815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:30.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 -- 172.21.15.29:0/2004488556 >> v1:172.21.15.186:6789/0 conn(0x7f9bc806c100 legacy=0x7f9bc806c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:30.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.183+0000 7f9bc7fff640 1 -- 172.21.15.29:0/2004488556 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc806cc80 con 0x7f9bc8070450 2023-10-14T17:57:30.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.184+0000 7f9bc7fff640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80708b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009870 tx=0x7f9bb002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8c2c79bf4754164 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:30.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bc77fe640 1 -- 172.21.15.29:0/2004488556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb00317c0 con 0x7f9bc8070450 2023-10-14T17:57:30.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bc77fe640 1 -- 172.21.15.29:0/2004488556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb0031920 con 0x7f9bc8070450 2023-10-14T17:57:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bc77fe640 1 -- 172.21.15.29:0/2004488556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0031cb0 con 0x7f9bc8070450 2023-10-14T17:57:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 -- 172.21.15.29:0/2004488556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 msgr2=0x7f9bc80708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80708b0 secure :-1 s=READY pgs=1913 cs=0 l=1 rev1=1 crypto rx=0x7f9bb0009870 tx=0x7f9bb002b860 comp rx=0 tx=0).stop 2023-10-14T17:57:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 -- 172.21.15.29:0/2004488556 shutdown_connections 2023-10-14T17:57:30.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80708b0 unknown :-1 s=CLOSED pgs=1913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/2004488556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 -- 172.21.15.29:0/2004488556 >> 172.21.15.29:0/2004488556 conn(0x7f9bc80ce6a0 msgr2=0x7f9bc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:30.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.185+0000 7f9bcdf67640 1 -- 172.21.15.29:0/2004488556 shutdown_connections 2023-10-14T17:57:30.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.186+0000 7f9bcdf67640 1 -- 172.21.15.29:0/2004488556 wait complete. 2023-10-14T17:57:30.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.187+0000 7f9bcdf67640 1 Processor -- start 2023-10-14T17:57:30.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.187+0000 7f9bcdf67640 1 -- start start 2023-10-14T17:57:30.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 0x7f9bc814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc80c4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80c27e0 con 0x7f9bc8070450 2023-10-14T17:57:30.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80c21b0 con 0x7f9bc806c100 2023-10-14T17:57:30.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.188+0000 7f9bcdf67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc80b34c0 con 0x7f9bc8159950 2023-10-14T17:57:30.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39488/0 (socket says 172.21.15.29:39488) 2023-10-14T17:57:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 -- 172.21.15.29:0/1367053050 learned_addr learned my addr 172.21.15.29:0/1367053050 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bccf65640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 0x7f9bc814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 -- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 msgr2=0x7f9bc80c4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:30.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bcd766640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc80c4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc80c4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 -- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 msgr2=0x7f9bc814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 0x7f9bc814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bc7fff640 1 -- 172.21.15.29:0/1367053050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb00094d0 con 0x7f9bc8070450 2023-10-14T17:57:30.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bccf65640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 0x7f9bc814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:30.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.189+0000 7f9bcd766640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc80c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:30.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bc7fff640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9bb002b8a0 tx=0x7f9bb00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:30.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0032860 con 0x7f9bc8070450 2023-10-14T17:57:30.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bb00329e0 con 0x7f9bc8070450 2023-10-14T17:57:30.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bb0032d90 con 0x7f9bc8070450 2023-10-14T17:57:30.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc80b37a0 con 0x7f9bc8070450 2023-10-14T17:57:30.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.190+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc8150120 con 0x7f9bc8070450 2023-10-14T17:57:30.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.191+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc804e9b0 con 0x7f9bc8070450 2023-10-14T17:57:30.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.192+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 92) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9bb004e450 con 0x7f9bc8070450 2023-10-14T17:57:30.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.193+0000 7f9bc57fa640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 0x7f9ba806d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.193+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6657+0+0 (secure 0 0 0) 0x7f9bb00abe80 con 0x7f9bc8070450 2023-10-14T17:57:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.194+0000 7f9bccf65640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 0x7f9ba806d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.194+0000 7f9bccf65640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 0x7f9ba806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007330 tx=0x7f9bb800b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:30.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.198+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9bb0076c30 con 0x7f9bc8070450 2023-10-14T17:57:30.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.433+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9bc80b4620 con 0x7f9bc8070450 2023-10-14T17:57:30.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.960+0000 7f9bc57fa640 1 -- 172.21.15.29:0/1367053050 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v76) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9bb007a250 con 0x7f9bc8070450 2023-10-14T17:57:30.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 msgr2=0x7f9ba806d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 0x7f9ba806d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8007330 tx=0x7f9bb800b040 comp rx=0 tx=0).stop 2023-10-14T17:57:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 msgr2=0x7f9bc80c43c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 secure :-1 s=READY pgs=1914 cs=0 l=1 rev1=1 crypto rx=0x7f9bb002b8a0 tx=0x7f9bb00046c0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 shutdown_connections 2023-10-14T17:57:30.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f9ba806af00 0x7f9ba806d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bc8159950 0x7f9bc80c4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bc8070450 0x7f9bc80c43c0 unknown :-1 s=CLOSED pgs=1914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 --2- 172.21.15.29:0/1367053050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bc806c100 0x7f9bc814fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 >> 172.21.15.29:0/1367053050 conn(0x7f9bc80ce6a0 msgr2=0x7f9bc8072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:30.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.965+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 shutdown_connections 2023-10-14T17:57:30.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:30.966+0000 7f9bcdf67640 1 -- 172.21.15.29:0/1367053050 wait complete. 2023-10-14T17:57:30.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T17:57:31.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.193+0000 7f7629490640 1 Processor -- start 2023-10-14T17:57:31.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.194+0000 7f7629490640 1 -- start start 2023-10-14T17:57:31.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.195+0000 7f7629490640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7624070b60 0x7f762415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.195+0000 7f7629490640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f762406c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.195+0000 7f7629490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f762415aef0 con 0x7f7624159950 2023-10-14T17:57:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.195+0000 7f7629490640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f762415e490 con 0x7f7624070b60 2023-10-14T17:57:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.195+0000 7f7629490640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f762415d910 con 0x7f762406c100 2023-10-14T17:57:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f7623fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7624159950 0x7f7624070450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42466/0 (socket says 172.21.15.29:42466) 2023-10-14T17:57:31.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f7623fff640 1 -- 172.21.15.29:0/819776532 learned_addr learned my addr 172.21.15.29:0/819776532 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f7628c8f640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f762406c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322835660 0 0) 0x7f7614001110 con 0x7f7624159950 2023-10-14T17:57:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7610003520 con 0x7f7624159950 2023-10-14T17:57:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> v2:172.21.15.186:3300/0 conn(0x7f7624070b60 0x7f762415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4096131865 0 0) 0x7f7610003520 con 0x7f7624159950 2023-10-14T17:57:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 msgr2=0x7f762406c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.196+0000 7f76237fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f762406c510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 >> v2:172.21.15.186:3300/0 conn(0x7f7624070b60 msgr2=0x7f762415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 --2- 172.21.15.29:0/819776532 >> v2:172.21.15.186:3300/0 conn(0x7f7624070b60 0x7f762415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762415dbf0 con 0x7f7624159950 2023-10-14T17:57:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7614001800 con 0x7f7624159950 2023-10-14T17:57:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 >> v1:172.21.15.29:6789/0 conn(0x7f7624159950 legacy=0x7f7624070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7610003970 0x7f7610003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76100047c0 0x7f7610004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f7610005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762415e490 con 0x7f76100047c0 2023-10-14T17:57:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762415d910 con 0x7f7610003970 2023-10-14T17:57:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f761000ba80 con 0x7f762406c100 2023-10-14T17:57:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> v2:172.21.15.186:3300/0 conn(0x7f7624070b60 0x7f762415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:31.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76100047c0 0x7f7610004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f7623fff640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7610003970 0x7f7610003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.197+0000 7f7628c8f640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f7610005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.198+0000 7f761b7fe640 1 -- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 msgr2=0x7f7610005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.198+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f762406c100 0x7f7610005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.198+0000 7f761b7fe640 1 -- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7610003970 msgr2=0x7f7610003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.198+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7610003970 0x7f7610003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.198+0000 7f761b7fe640 1 -- 172.21.15.29:0/819776532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7610003520 con 0x7f76100047c0 2023-10-14T17:57:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.199+0000 7f761b7fe640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76100047c0 0x7f7610004c40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f760c0098a0 tx=0x7f76100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7028b3babfbe6059 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.199+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c030630 con 0x7f76100047c0 2023-10-14T17:57:31.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.199+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f760c030790 con 0x7f76100047c0 2023-10-14T17:57:31.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.199+0000 7f76237fe640 1 -- 172.21.15.29:0/819776532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760c030b00 con 0x7f76100047c0 2023-10-14T17:57:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 -- 172.21.15.29:0/819776532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76100047c0 msgr2=0x7f7610004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 --2- 172.21.15.29:0/819776532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76100047c0 0x7f7610004c40 secure :-1 s=READY pgs=1916 cs=0 l=1 rev1=1 crypto rx=0x7f760c0098a0 tx=0x7f76100057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7628c8f640 1 -- 172.21.15.29:0/819776532 reap_dead start 2023-10-14T17:57:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 -- 172.21.15.29:0/819776532 shutdown_connections 2023-10-14T17:57:31.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 -- 172.21.15.29:0/819776532 >> 172.21.15.29:0/819776532 conn(0x7f76240ce6a0 msgr2=0x7f76240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 -- 172.21.15.29:0/819776532 shutdown_connections 2023-10-14T17:57:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.200+0000 7f7629490640 1 -- 172.21.15.29:0/819776532 wait complete. 2023-10-14T17:57:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.202+0000 7f7629490640 1 Processor -- start 2023-10-14T17:57:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.202+0000 7f7629490640 1 -- start start 2023-10-14T17:57:31.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 0x7f762414da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 0x7f76240b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624150920 con 0x7f76240b2c60 2023-10-14T17:57:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76240b27d0 con 0x7f762414f780 2023-10-14T17:57:31.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7629490640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76240b1bc0 con 0x7f76100047c0 2023-10-14T17:57:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7628c8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7628c8f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60084/0 (socket says 172.21.15.29:60084) 2023-10-14T17:57:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.203+0000 7f7628c8f640 1 -- 172.21.15.29:0/168741374 learned_addr learned my addr 172.21.15.29:0/168741374 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7623fff640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 0x7f762414da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f761b7fe640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 0x7f76240b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 -- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 msgr2=0x7f762414da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 0x7f762414da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 -- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 msgr2=0x7f76240b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 0x7f76240b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 -- 172.21.15.29:0/168741374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76240b1ea0 con 0x7f76240b2c60 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7623fff640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 0x7f762414da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f761b7fe640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 0x7f76240b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7628c8f640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f762406f1c0 tx=0x7f761c0115a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c00e070 con 0x7f76240b2c60 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c011e40 con 0x7f76240b2c60 2023-10-14T17:57:31.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76240b2180 con 0x7f76240b2c60 2023-10-14T17:57:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.204+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c003990 con 0x7f76240b2c60 2023-10-14T17:57:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.205+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f762414dfc0 con 0x7f76240b2c60 2023-10-14T17:57:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.206+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f762404e9b0 con 0x7f76240b2c60 2023-10-14T17:57:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.207+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f761c003af0 con 0x7f76240b2c60 2023-10-14T17:57:31.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.208+0000 7f76217fa640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 0x7f760406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.208+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(167..167 src has 1..167) v4 ==== 6657+0+0 (secure 0 0 0) 0x7f761c08a450 con 0x7f76240b2c60 2023-10-14T17:57:31.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.209+0000 7f7623fff640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 0x7f760406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.209+0000 7f7623fff640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 0x7f760406d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7614000c00 tx=0x7f7614000ef0 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.213+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f761c054e80 con 0x7f76240b2c60 2023-10-14T17:57:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.410+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f76240b2910 con 0x7f76240b2c60 2023-10-14T17:57:31.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.704+0000 7f76217fa640 1 -- 172.21.15.29:0/168741374 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v168) v1 ==== 181+0+0 (secure 0 0 0) 0x7f761c0584a0 con 0x7f76240b2c60 2023-10-14T17:57:31.706 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T17:57:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 msgr2=0x7f760406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 0x7f760406d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7614000c00 tx=0x7f7614000ef0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 msgr2=0x7f762406b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 secure :-1 s=READY pgs=1917 cs=0 l=1 rev1=1 crypto rx=0x7f762406f1c0 tx=0x7f761c0115a0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 shutdown_connections 2023-10-14T17:57:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f760406afc0 0x7f760406d480 secure :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f7614000c00 tx=0x7f7614000ef0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f76240b2c60 0x7f762406b510 unknown :-1 s=CLOSED pgs=1917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f762414f780 0x7f76240b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 --2- 172.21.15.29:0/168741374 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f76100047c0 0x7f762414da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.711+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 >> 172.21.15.29:0/168741374 conn(0x7f76240ce6a0 msgr2=0x7f76240cd490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:31.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.712+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 shutdown_connections 2023-10-14T17:57:31.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.712+0000 7f7629490640 1 -- 172.21.15.29:0/168741374 wait complete. 2023-10-14T17:57:31.735 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T17:57:31.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.962+0000 7f73fef89640 1 Processor -- start 2023-10-14T17:57:31.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.962+0000 7f73fef89640 1 -- start start 2023-10-14T17:57:31.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fef89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fef89640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f80cac20 0x7f73f80b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fef89640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73f80b0d00 con 0x7f73f80b0360 2023-10-14T17:57:31.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fef89640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73f815f800 con 0x7f73f80cac20 2023-10-14T17:57:31.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fef89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f815b310 con 0x7f73f815ac10 2023-10-14T17:57:31.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fd526640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f80cac20 0x7f73f80b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fd526640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73f80cac20 0x7f73f80b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60094/0 (socket says 172.21.15.29:60094) 2023-10-14T17:57:31.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.964+0000 7f73fd526640 1 -- 172.21.15.29:0/525919008 learned_addr learned my addr 172.21.15.29:0/525919008 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:31.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.965+0000 7f73fcd25640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.965+0000 7f73fd526640 1 -- 172.21.15.29:0/525919008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 msgr2=0x7f73f806c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.965+0000 7f73fd526640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f806c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.965+0000 7f73fd526640 1 -- 172.21.15.29:0/525919008 >> v1:172.21.15.186:6789/0 conn(0x7f73f80b0360 legacy=0x7f73f80ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.965+0000 7f73fd526640 1 -- 172.21.15.29:0/525919008 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f806f640 con 0x7f73f80cac20 2023-10-14T17:57:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.966+0000 7f73fd526640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b07c0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009870 tx=0x7f73f002b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4807033ee8b17db1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.966+0000 7f73f77fe640 1 -- 172.21.15.29:0/525919008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f00317c0 con 0x7f73f80cac20 2023-10-14T17:57:31.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73f77fe640 1 -- 172.21.15.29:0/525919008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73f0031920 con 0x7f73f80cac20 2023-10-14T17:57:31.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73f77fe640 1 -- 172.21.15.29:0/525919008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73f0031cb0 con 0x7f73f80cac20 2023-10-14T17:57:31.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 -- 172.21.15.29:0/525919008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 msgr2=0x7f73f80b07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b07c0 secure :-1 s=READY pgs=1918 cs=0 l=1 rev1=1 crypto rx=0x7f73f0009870 tx=0x7f73f002b860 comp rx=0 tx=0).stop 2023-10-14T17:57:31.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 -- 172.21.15.29:0/525919008 shutdown_connections 2023-10-14T17:57:31.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b07c0 unknown :-1 s=CLOSED pgs=1918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 --2- 172.21.15.29:0/525919008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f806c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 -- 172.21.15.29:0/525919008 >> 172.21.15.29:0/525919008 conn(0x7f73f80cb360 msgr2=0x7f73f80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 -- 172.21.15.29:0/525919008 shutdown_connections 2023-10-14T17:57:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.967+0000 7f73fef89640 1 -- 172.21.15.29:0/525919008 wait complete. 2023-10-14T17:57:31.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.969+0000 7f73fef89640 1 Processor -- start 2023-10-14T17:57:31.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.969+0000 7f73fef89640 1 -- start start 2023-10-14T17:57:31.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f80b0360 0x7f73f80b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f814f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80ccd80 con 0x7f73f80cac20 2023-10-14T17:57:31.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73f81f25a0 con 0x7f73f80b0360 2023-10-14T17:57:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.970+0000 7f73fef89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73f80cc4b0 con 0x7f73f815ac10 2023-10-14T17:57:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60106/0 (socket says 172.21.15.29:60106) 2023-10-14T17:57:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 -- 172.21.15.29:0/4197686985 learned_addr learned my addr 172.21.15.29:0/4197686985 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:31.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73fd526640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f814f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 -- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 msgr2=0x7f73f814f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73fcd25640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f80b0360 0x7f73f80b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f814f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 -- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f80b0360 msgr2=0x7f73f80b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f80b0360 0x7f73f80b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73f7fff640 1 -- 172.21.15.29:0/4197686985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73f00094d0 con 0x7f73f80cac20 2023-10-14T17:57:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.971+0000 7f73fd526640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f814f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73f7fff640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00b3f0 tx=0x7f73ec00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec011830 con 0x7f73f80cac20 2023-10-14T17:57:31.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73ec011990 con 0x7f73f80cac20 2023-10-14T17:57:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73ec011d40 con 0x7f73f80cac20 2023-10-14T17:57:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73f80b2420 con 0x7f73f80cac20 2023-10-14T17:57:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.972+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73f80cc6c0 con 0x7f73f80cac20 2023-10-14T17:57:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.973+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73f80ccec0 con 0x7f73f80cac20 2023-10-14T17:57:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.974+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f73ec01a400 con 0x7f73f80cac20 2023-10-14T17:57:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.975+0000 7f73f57fa640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 0x7f73e406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.975+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(168..168 src has 1..168) v4 ==== 6263+0+0 (secure 0 0 0) 0x7f73ec089800 con 0x7f73f80cac20 2023-10-14T17:57:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.976+0000 7f73fcd25640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 0x7f73e406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:31.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.976+0000 7f73fcd25640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 0x7f73e406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009f30 tx=0x7f73e800b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:31.981+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73f80ccec0 con 0x7f73f80cac20 2023-10-14T17:57:32.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.189+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f73f80ccec0 con 0x7f73f80cac20 2023-10-14T17:57:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.785+0000 7f73f57fa640 1 -- 172.21.15.29:0/4197686985 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v169) v1 ==== 169+0+0 (secure 0 0 0) 0x7f73f80ccec0 con 0x7f73f80cac20 2023-10-14T17:57:32.787 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T17:57:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.789+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 msgr2=0x7f73e406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:32.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.789+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 0x7f73e406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f73e8009f30 tx=0x7f73e800b040 comp rx=0 tx=0).stop 2023-10-14T17:57:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.789+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 msgr2=0x7f73f80b1ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:32.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.789+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 secure :-1 s=READY pgs=1919 cs=0 l=1 rev1=1 crypto rx=0x7f73ec00b3f0 tx=0x7f73ec00b8b0 comp rx=0 tx=0).stop 2023-10-14T17:57:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 shutdown_connections 2023-10-14T17:57:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f73e406af90 0x7f73e406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:32.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73f815ac10 0x7f73f814f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73f80cac20 0x7f73f80b1ee0 unknown :-1 s=CLOSED pgs=1919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 --2- 172.21.15.29:0/4197686985 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73f80b0360 0x7f73f80b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 >> 172.21.15.29:0/4197686985 conn(0x7f73f80cb360 msgr2=0x7f73f806c500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:32.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.791+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 shutdown_connections 2023-10-14T17:57:32.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:32.792+0000 7f73fef89640 1 -- 172.21.15.29:0/4197686985 wait complete. 2023-10-14T17:57:32.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T17:57:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.029+0000 7f94aa1b3640 1 Processor -- start 2023-10-14T17:57:33.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.030+0000 7f94aa1b3640 1 -- start start 2023-10-14T17:57:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.031+0000 7f94aa1b3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 0x7f94a40d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.031+0000 7f94aa1b3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94a40b2080 0x7f94a40ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.031+0000 7f94aa1b3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94a40d0970 con 0x7f94a40ce960 2023-10-14T17:57:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.031+0000 7f94aa1b3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94a40d0ea0 con 0x7f94a40b2080 2023-10-14T17:57:33.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.031+0000 7f94aa1b3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40d0bc0 con 0x7f94a40b1ca0 2023-10-14T17:57:33.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94a40b2080 0x7f94a40ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f94a40b2080 0x7f94a40ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60120/0 (socket says 172.21.15.29:60120) 2023-10-14T17:57:33.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 -- 172.21.15.29:0/4124053073 learned_addr learned my addr 172.21.15.29:0/4124053073 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:33.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a91b1640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 0x7f94a40d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 -- 172.21.15.29:0/4124053073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 msgr2=0x7f94a40d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:33.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 0x7f94a40d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 -- 172.21.15.29:0/4124053073 >> v1:172.21.15.29:6789/0 conn(0x7f94a40ce960 legacy=0x7f94a40b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a89b0640 1 -- 172.21.15.29:0/4124053073 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a40cd940 con 0x7f94a40b2080 2023-10-14T17:57:33.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.032+0000 7f94a91b1640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 0x7f94a40d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:33.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.033+0000 7f94a89b0640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a40ce420 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9498009870 tx=0x7f949802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=967ef7c1f97e49fe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94a3fff640 1 -- 172.21.15.29:0/4124053073 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949802f830 con 0x7f94a40b2080 2023-10-14T17:57:33.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94a3fff640 1 -- 172.21.15.29:0/4124053073 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f949802f990 con 0x7f94a40b2080 2023-10-14T17:57:33.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94a3fff640 1 -- 172.21.15.29:0/4124053073 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949802fd20 con 0x7f94a40b2080 2023-10-14T17:57:33.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4124053073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 msgr2=0x7f94a40ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a40ce420 secure :-1 s=READY pgs=1921 cs=0 l=1 rev1=1 crypto rx=0x7f9498009870 tx=0x7f949802ba10 comp rx=0 tx=0).stop 2023-10-14T17:57:33.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4124053073 shutdown_connections 2023-10-14T17:57:33.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a40ce420 unknown :-1 s=CLOSED pgs=1921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4124053073 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40b1ca0 0x7f94a40d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.034+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4124053073 >> 172.21.15.29:0/4124053073 conn(0x7f94a40d2b90 msgr2=0x7f94a40d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.035+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4124053073 shutdown_connections 2023-10-14T17:57:33.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.035+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4124053073 wait complete. 2023-10-14T17:57:33.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.036+0000 7f94aa1b3640 1 Processor -- start 2023-10-14T17:57:33.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.036+0000 7f94aa1b3640 1 -- start start 2023-10-14T17:57:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94a40b1ca0 0x7f94a415e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40ce960 0x7f94a406b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a415d900 con 0x7f94a40b2080 2023-10-14T17:57:33.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a40c7e50 con 0x7f94a40b1ca0 2023-10-14T17:57:33.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.037+0000 7f94aa1b3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a415dba0 con 0x7f94a40ce960 2023-10-14T17:57:33.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60136/0 (socket says 172.21.15.29:60136) 2023-10-14T17:57:33.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 -- 172.21.15.29:0/4141936140 learned_addr learned my addr 172.21.15.29:0/4141936140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:33.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 -- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40ce960 msgr2=0x7f94a406b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40ce960 0x7f94a406b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 -- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94a40b1ca0 msgr2=0x7f94a415e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94a40b1ca0 0x7f94a415e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 -- 172.21.15.29:0/4141936140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94980094d0 con 0x7f94a40b2080 2023-10-14T17:57:33.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.038+0000 7f94a89b0640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f9498000c00 tx=0x7f9498004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.039+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f949802f800 con 0x7f94a40b2080 2023-10-14T17:57:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.039+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9498041400 con 0x7f94a40b2080 2023-10-14T17:57:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.039+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94980417d0 con 0x7f94a40b2080 2023-10-14T17:57:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.039+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94a415de80 con 0x7f94a40b2080 2023-10-14T17:57:33.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.039+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94a415e8e0 con 0x7f94a40b2080 2023-10-14T17:57:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.040+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a415e090 con 0x7f94a40b2080 2023-10-14T17:57:33.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.041+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9498041930 con 0x7f94a40b2080 2023-10-14T17:57:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.042+0000 7f94a1ffb640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 0x7f949006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.042+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f949803c070 con 0x7f94a40b2080 2023-10-14T17:57:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.042+0000 7f94a91b1640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 0x7f949006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.043+0000 7f94a91b1640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 0x7f949006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9494008e00 tx=0x7f9494008330 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.046+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94980775e0 con 0x7f94a40b2080 2023-10-14T17:57:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.241+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f94a404e9b0 con 0x7f94a40b2080 2023-10-14T17:57:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.246+0000 7f94a1ffb640 1 -- 172.21.15.29:0/4141936140 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v55) v1 ==== 115+0+0 (secure 0 0 0) 0x7f949807ac00 con 0x7f94a40b2080 2023-10-14T17:57:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 msgr2=0x7f949006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 0x7f949006d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f9494008e00 tx=0x7f9494008330 comp rx=0 tx=0).stop 2023-10-14T17:57:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 msgr2=0x7f94a415ce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 secure :-1 s=READY pgs=1922 cs=0 l=1 rev1=1 crypto rx=0x7f9498000c00 tx=0x7f9498004730 comp rx=0 tx=0).stop 2023-10-14T17:57:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 shutdown_connections 2023-10-14T17:57:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f949006af90 0x7f949006d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94a40ce960 0x7f94a406b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94a40b2080 0x7f94a415ce80 unknown :-1 s=CLOSED pgs=1922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 --2- 172.21.15.29:0/4141936140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94a40b1ca0 0x7f94a415e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 >> 172.21.15.29:0/4141936140 conn(0x7f94a40d2b90 msgr2=0x7f94a40d2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.249+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 shutdown_connections 2023-10-14T17:57:33.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.250+0000 7f94aa1b3640 1 -- 172.21.15.29:0/4141936140 wait complete. 2023-10-14T17:57:33.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T17:57:33.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.499+0000 7f30c4ea2640 1 Processor -- start 2023-10-14T17:57:33.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.499+0000 7f30c4ea2640 1 -- start start 2023-10-14T17:57:33.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.500+0000 7f30c4ea2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30c015d330 0x7f30c01618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.500+0000 7f30c4ea2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00d0890 con 0x7f30c006c7d0 2023-10-14T17:57:33.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.501+0000 7f30c4ea2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30c00d05b0 con 0x7f30c00661c0 2023-10-14T17:57:33.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.501+0000 7f30c4ea2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30c006b1e0 con 0x7f30c015d330 2023-10-14T17:57:33.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.501+0000 7f30bdd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30c006c7d0 0x7f30c0065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42480/0 (socket says 172.21.15.29:42480) 2023-10-14T17:57:33.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.501+0000 7f30bdd9b640 1 -- 172.21.15.29:0/2637352292 learned_addr learned my addr 172.21.15.29:0/2637352292 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:33.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.501+0000 7f30be59c640 1 --2- 172.21.15.29:0/2637352292 >> v2:172.21.15.186:3300/0 conn(0x7f30c015d330 0x7f30c01618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2704789201 0 0) 0x7f30c00d0890 con 0x7f30c006c7d0 2023-10-14T17:57:33.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30ac003520 con 0x7f30c006c7d0 2023-10-14T17:57:33.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30be59c640 1 -- 172.21.15.29:0/2637352292 >> v1:172.21.15.186:6789/0 conn(0x7f30c00661c0 legacy=0x7f30c00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30be59c640 1 -- 172.21.15.29:0/2637352292 >> v1:172.21.15.29:6789/0 conn(0x7f30c006c7d0 legacy=0x7f30c0065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 601135445 0 0) 0x7f30ac003520 con 0x7f30c006c7d0 2023-10-14T17:57:33.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.502+0000 7f30be59c640 1 -- 172.21.15.29:0/2637352292 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30c0160dd0 con 0x7f30c015d330 2023-10-14T17:57:33.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.503+0000 7f30be59c640 1 --2- 172.21.15.29:0/2637352292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c015d330 0x7f30c01618b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f30b0009d90 tx=0x7f30b002bd80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0f7247e9f21115f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.504+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30a8001670 con 0x7f30c015d330 2023-10-14T17:57:33.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.504+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b002c8b0 con 0x7f30c015d330 2023-10-14T17:57:33.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.504+0000 7f30bd59a640 1 -- 172.21.15.29:0/2637352292 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b002cc40 con 0x7f30c015d330 2023-10-14T17:57:33.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.504+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2637352292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c015d330 msgr2=0x7f30c01618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.504+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2637352292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c015d330 0x7f30c01618b0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f30b0009d90 tx=0x7f30b002bd80 comp rx=0 tx=0).stop 2023-10-14T17:57:33.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.505+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2637352292 shutdown_connections 2023-10-14T17:57:33.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.505+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2637352292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c015d330 0x7f30c01618b0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.505+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2637352292 >> 172.21.15.29:0/2637352292 conn(0x7f30c00d0ab0 msgr2=0x7f30c00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:33.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.505+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2637352292 shutdown_connections 2023-10-14T17:57:33.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.505+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2637352292 wait complete. 2023-10-14T17:57:33.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.506+0000 7f30c4ea2640 1 Processor -- start 2023-10-14T17:57:33.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.507+0000 7f30c4ea2640 1 -- start start 2023-10-14T17:57:33.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.507+0000 7f30c4ea2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c00661c0 0x7f30c0158e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.507+0000 7f30c4ea2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c006c7d0 0x7f30c0156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.507+0000 7f30c4ea2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.507+0000 7f30c4ea2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00ae2b0 con 0x7f30c015d330 2023-10-14T17:57:33.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30c4ea2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c0159370 con 0x7f30c00661c0 2023-10-14T17:57:33.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30c4ea2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30c00c6dc0 con 0x7f30c006c7d0 2023-10-14T17:57:33.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60144/0 (socket says 172.21.15.29:60144) 2023-10-14T17:57:33.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 -- 172.21.15.29:0/2928781739 learned_addr learned my addr 172.21.15.29:0/2928781739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:33.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30be59c640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c00661c0 0x7f30c0158e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bdd9b640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c006c7d0 0x7f30c0156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 -- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c006c7d0 msgr2=0x7f30c0156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c006c7d0 0x7f30c0156f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 -- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c00661c0 msgr2=0x7f30c0158e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c00661c0 0x7f30c0158e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.508+0000 7f30bed9d640 1 -- 172.21.15.29:0/2928781739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b0009a70 con 0x7f30c015d330 2023-10-14T17:57:33.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30bed9d640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f30b8001eb0 tx=0x7f30b800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b80052d0 con 0x7f30c015d330 2023-10-14T17:57:33.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30b8005430 con 0x7f30c015d330 2023-10-14T17:57:33.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30b8003810 con 0x7f30c015d330 2023-10-14T17:57:33.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c00c7060 con 0x7f30c015d330 2023-10-14T17:57:33.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.509+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c00ae500 con 0x7f30c015d330 2023-10-14T17:57:33.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.510+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c004e160 con 0x7f30c015d330 2023-10-14T17:57:33.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.511+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f30b8003970 con 0x7f30c015d330 2023-10-14T17:57:33.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.512+0000 7f30b6ffd640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 0x7f30a006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.512+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f30b8088e10 con 0x7f30c015d330 2023-10-14T17:57:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.513+0000 7f30be59c640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 0x7f30a006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.513+0000 7f30be59c640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 0x7f30a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30b0036000 tx=0x7f30b0030000 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.517+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f30b808c030 con 0x7f30c015d330 2023-10-14T17:57:33.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.715+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f30c006f3a0 con 0x7f30c015d330 2023-10-14T17:57:33.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.722+0000 7f30b6ffd640 1 -- 172.21.15.29:0/2928781739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v56) v1 ==== 120+0+0 (secure 0 0 0) 0x7f30b8053ce0 con 0x7f30c015d330 2023-10-14T17:57:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 msgr2=0x7f30a006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 0x7f30a006d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f30b0036000 tx=0x7f30b0030000 comp rx=0 tx=0).stop 2023-10-14T17:57:33.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 msgr2=0x7f30c00aea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 secure :-1 s=READY pgs=1924 cs=0 l=1 rev1=1 crypto rx=0x7f30b8001eb0 tx=0x7f30b800c890 comp rx=0 tx=0).stop 2023-10-14T17:57:33.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 shutdown_connections 2023-10-14T17:57:33.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f30a006af90 0x7f30a006d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30c015d330 0x7f30c00aea20 unknown :-1 s=CLOSED pgs=1924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30c006c7d0 0x7f30c0156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 --2- 172.21.15.29:0/2928781739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30c00661c0 0x7f30c0158e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 >> 172.21.15.29:0/2928781739 conn(0x7f30c00d0ab0 msgr2=0x7f30c015f9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:33.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 shutdown_connections 2023-10-14T17:57:33.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.725+0000 7f30c4ea2640 1 -- 172.21.15.29:0/2928781739 wait complete. 2023-10-14T17:57:33.747 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T17:57:33.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.965+0000 7fceabfff640 1 Processor -- start 2023-10-14T17:57:33.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.965+0000 7fceabfff640 1 -- start start 2023-10-14T17:57:33.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.967+0000 7fceabfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac070b60 0x7fceac15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.967+0000 7fceabfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fceac06c100 0x7fceac06c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.967+0000 7fceabfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fceac15aef0 con 0x7fceac159950 2023-10-14T17:57:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.967+0000 7fceabfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fceac15e490 con 0x7fceac06c100 2023-10-14T17:57:33.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.967+0000 7fceabfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac15d910 con 0x7fceac070b60 2023-10-14T17:57:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fceac06c100 0x7fceac06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fceac06c100 0x7fceac06c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60146/0 (socket says 172.21.15.29:60146) 2023-10-14T17:57:33.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 -- 172.21.15.29:0/4205277794 learned_addr learned my addr 172.21.15.29:0/4205277794 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:33.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fcea27fc640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac070b60 0x7fceac15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 -- 172.21.15.29:0/4205277794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac070b60 msgr2=0x7fceac15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:33.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac070b60 0x7fceac15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.968+0000 7fceab7fe640 1 -- 172.21.15.29:0/4205277794 >> v1:172.21.15.186:6789/0 conn(0x7fceac159950 legacy=0x7fceac070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:33.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.969+0000 7fceab7fe640 1 -- 172.21.15.29:0/4205277794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fceac15dbf0 con 0x7fceac06c100 2023-10-14T17:57:33.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.970+0000 7fceab7fe640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac06c100 0x7fceac06c550 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fcea400bb10 tx=0x7fcea402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=564c26c1e9dbe67 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.970+0000 7fceaa7fc640 1 -- 172.21.15.29:0/4205277794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea402ee70 con 0x7fceac06c100 2023-10-14T17:57:33.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.970+0000 7fceaa7fc640 1 -- 172.21.15.29:0/4205277794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea4036440 con 0x7fceac06c100 2023-10-14T17:57:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.970+0000 7fceaa7fc640 1 -- 172.21.15.29:0/4205277794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea40367d0 con 0x7fceac06c100 2023-10-14T17:57:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 -- 172.21.15.29:0/4205277794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac06c100 msgr2=0x7fceac06c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac06c100 0x7fceac06c550 secure :-1 s=READY pgs=1925 cs=0 l=1 rev1=1 crypto rx=0x7fcea400bb10 tx=0x7fcea402e0c0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 -- 172.21.15.29:0/4205277794 shutdown_connections 2023-10-14T17:57:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac06c100 0x7fceac06c550 unknown :-1 s=CLOSED pgs=1925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 --2- 172.21.15.29:0/4205277794 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac070b60 0x7fceac15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 -- 172.21.15.29:0/4205277794 >> 172.21.15.29:0/4205277794 conn(0x7fceac0ce6a0 msgr2=0x7fceac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:33.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 -- 172.21.15.29:0/4205277794 shutdown_connections 2023-10-14T17:57:33.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.971+0000 7fceabfff640 1 -- 172.21.15.29:0/4205277794 wait complete. 2023-10-14T17:57:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.973+0000 7fceabfff640 1 Processor -- start 2023-10-14T17:57:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.973+0000 7fceabfff640 1 -- start start 2023-10-14T17:57:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac06c100 0x7fceac14a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 0x7fceac0a87a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac14abf0 con 0x7fceac159950 2023-10-14T17:57:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0c30f0 con 0x7fceac070b60 2023-10-14T17:57:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceabfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fceac0c33d0 con 0x7fceac06c100 2023-10-14T17:57:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60158/0 (socket says 172.21.15.29:60158) 2023-10-14T17:57:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.974+0000 7fceab7fe640 1 -- 172.21.15.29:0/307798882 learned_addr learned my addr 172.21.15.29:0/307798882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fcea27fc640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 0x7fceac0a87a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceaaffd640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac06c100 0x7fceac14a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 -- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac06c100 msgr2=0x7fceac14a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac06c100 0x7fceac14a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 -- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 msgr2=0x7fceac0a87a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 0x7fceac0a87a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 -- 172.21.15.29:0/307798882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcea400b770 con 0x7fceac159950 2023-10-14T17:57:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fcea27fc640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 0x7fceac0a87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:33.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.975+0000 7fceab7fe640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fcea400b2c0 tx=0x7fcea40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.976+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4034850 con 0x7fceac159950 2023-10-14T17:57:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.976+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fceac0c36b0 con 0x7fceac159950 2023-10-14T17:57:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.976+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcea40349b0 con 0x7fceac159950 2023-10-14T17:57:33.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.976+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcea4034d60 con 0x7fceac159950 2023-10-14T17:57:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.976+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fceac0c3c70 con 0x7fceac159950 2023-10-14T17:57:33.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.977+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fceac0c3e60 con 0x7fceac159950 2023-10-14T17:57:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.978+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcea40403e0 con 0x7fceac159950 2023-10-14T17:57:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.979+0000 7fcea3fff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 0x7fce8c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.979+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fcea40abb40 con 0x7fceac159950 2023-10-14T17:57:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.979+0000 7fceaaffd640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 0x7fce8c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:33.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.980+0000 7fceaaffd640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 0x7fce8c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fce9c000f60 tx=0x7fce9c009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:33.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:33.984+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcea4076a20 con 0x7fceac159950 2023-10-14T17:57:34.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.179+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fceac04e9b0 con 0x7fceac159950 2023-10-14T17:57:34.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.187+0000 7fcea3fff640 1 -- 172.21.15.29:0/307798882 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v57) v1 ==== 104+0+0 (secure 0 0 0) 0x7fcea407a040 con 0x7fceac159950 2023-10-14T17:57:34.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 msgr2=0x7fce8c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 0x7fce8c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fce9c000f60 tx=0x7fce9c009040 comp rx=0 tx=0).stop 2023-10-14T17:57:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 msgr2=0x7fceac0c4190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 secure :-1 s=READY pgs=1926 cs=0 l=1 rev1=1 crypto rx=0x7fcea400b2c0 tx=0x7fcea40046c0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 shutdown_connections 2023-10-14T17:57:34.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fce8c06af90 0x7fce8c06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fceac159950 0x7fceac0c4190 unknown :-1 s=CLOSED pgs=1926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fceac070b60 0x7fceac0a87a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 --2- 172.21.15.29:0/307798882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fceac06c100 0x7fceac14a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.190+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 >> 172.21.15.29:0/307798882 conn(0x7fceac0ce6a0 msgr2=0x7fceac070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:34.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.191+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 shutdown_connections 2023-10-14T17:57:34.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.191+0000 7fceabfff640 1 -- 172.21.15.29:0/307798882 wait complete. 2023-10-14T17:57:34.213 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention' 2023-10-14T17:57:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.441+0000 7f18471ef640 1 Processor -- start 2023-10-14T17:57:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.441+0000 7f18471ef640 1 -- start start 2023-10-14T17:57:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.442+0000 7f18471ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f18400afce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.442+0000 7f18471ef640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f184006f900 0x7f18400b2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.442+0000 7f18471ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f184006fd60 con 0x7f18400b28b0 2023-10-14T17:57:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.442+0000 7f18471ef640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18400b2d40 con 0x7f184006f900 2023-10-14T17:57:34.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.442+0000 7f18471ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18400cf6e0 con 0x7f184015abf0 2023-10-14T17:57:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.443+0000 7f18469ee640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f18400b28b0 0x7f18400ce4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42482/0 (socket says 172.21.15.29:42482) 2023-10-14T17:57:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.443+0000 7f18469ee640 1 -- 172.21.15.29:0/2943292088 learned_addr learned my addr 172.21.15.29:0/2943292088 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.443+0000 7f18459ec640 1 --2- 172.21.15.29:0/2943292088 >> v2:172.21.15.186:3300/0 conn(0x7f184006f900 0x7f18400b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.443+0000 7f18461ed640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f18400afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.443+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795697055 0 0) 0x7f1838001110 con 0x7f18400b28b0 2023-10-14T17:57:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1830003520 con 0x7f18400b28b0 2023-10-14T17:57:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73589142 0 0) 0x7f1830003520 con 0x7f18400b28b0 2023-10-14T17:57:34.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 msgr2=0x7f18400afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f18400afce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 >> v2:172.21.15.186:3300/0 conn(0x7f184006f900 msgr2=0x7f18400b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 --2- 172.21.15.29:0/2943292088 >> v2:172.21.15.186:3300/0 conn(0x7f184006f900 0x7f18400b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400cd140 con 0x7f18400b28b0 2023-10-14T17:57:34.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1838001800 con 0x7f18400b28b0 2023-10-14T17:57:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 >> v1:172.21.15.29:6789/0 conn(0x7f18400b28b0 legacy=0x7f18400ce4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1830003970 0x7f1830003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18300047c0 0x7f1830004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f1830005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18400b2d40 con 0x7f1830003970 2023-10-14T17:57:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18400cf6e0 con 0x7f18300047c0 2023-10-14T17:57:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f183000b9d0 con 0x7f184015abf0 2023-10-14T17:57:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.444+0000 7f18461ed640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1830003970 0x7f1830003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18459ec640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18300047c0 0x7f1830004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18469ee640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f1830005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18461ed640 1 -- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 msgr2=0x7f1830005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18461ed640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f184015abf0 0x7f1830005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18461ed640 1 -- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18300047c0 msgr2=0x7f1830004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18461ed640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f18300047c0 0x7f1830004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.445+0000 7f18461ed640 1 -- 172.21.15.29:0/2943292088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830003520 con 0x7f1830003970 2023-10-14T17:57:34.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.446+0000 7f18461ed640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1830003970 0x7f1830003d90 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f183400e170 tx=0x7f18300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12c8465dea1eef2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:34.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.446+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f183402cbf0 con 0x7f1830003970 2023-10-14T17:57:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.446+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f183400f040 con 0x7f1830003970 2023-10-14T17:57:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.446+0000 7f18451eb640 1 -- 172.21.15.29:0/2943292088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1834034440 con 0x7f1830003970 2023-10-14T17:57:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 -- 172.21.15.29:0/2943292088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1830003970 msgr2=0x7f1830003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 --2- 172.21.15.29:0/2943292088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1830003970 0x7f1830003d90 secure :-1 s=READY pgs=1928 cs=0 l=1 rev1=1 crypto rx=0x7f183400e170 tx=0x7f18300057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18469ee640 1 -- 172.21.15.29:0/2943292088 reap_dead start 2023-10-14T17:57:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 -- 172.21.15.29:0/2943292088 shutdown_connections 2023-10-14T17:57:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 -- 172.21.15.29:0/2943292088 >> 172.21.15.29:0/2943292088 conn(0x7f18400cd370 msgr2=0x7f18400cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 -- 172.21.15.29:0/2943292088 shutdown_connections 2023-10-14T17:57:34.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.447+0000 7f18471ef640 1 -- 172.21.15.29:0/2943292088 wait complete. 2023-10-14T17:57:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.449+0000 7f18471ef640 1 Processor -- start 2023-10-14T17:57:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.449+0000 7f18471ef640 1 -- start start 2023-10-14T17:57:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830003970 0x7f184014d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18400af280 0x7f184006c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184006cdf0 con 0x7f184006aef0 2023-10-14T17:57:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f184006b810 con 0x7f1830003970 2023-10-14T17:57:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.450+0000 7f18471ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f184006baf0 con 0x7f18400af280 2023-10-14T17:57:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60176/0 (socket says 172.21.15.29:60176) 2023-10-14T17:57:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 -- 172.21.15.29:0/382455760 learned_addr learned my addr 172.21.15.29:0/382455760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 -- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18400af280 msgr2=0x7f184006c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18400af280 0x7f184006c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 -- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830003970 msgr2=0x7f184014d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830003970 0x7f184014d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 -- 172.21.15.29:0/382455760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f184006bdd0 con 0x7f184006aef0 2023-10-14T17:57:34.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.451+0000 7f18469ee640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1840073b00 tx=0x7f183800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:34.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.452+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18380074e0 con 0x7f184006aef0 2023-10-14T17:57:34.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.452+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1838007660 con 0x7f184006aef0 2023-10-14T17:57:34.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.452+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1838004a20 con 0x7f184006aef0 2023-10-14T17:57:34.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.452+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f184006c0b0 con 0x7f184006aef0 2023-10-14T17:57:34.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.452+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f184006b370 con 0x7f184006aef0 2023-10-14T17:57:34.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.453+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f184006c5d0 con 0x7f184006aef0 2023-10-14T17:57:34.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.454+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1838004b80 con 0x7f184006aef0 2023-10-14T17:57:34.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.455+0000 7f183effd640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 0x7f182406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:34.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.455+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f18380893c0 con 0x7f184006aef0 2023-10-14T17:57:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.455+0000 7f18461ed640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 0x7f182406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.456+0000 7f18461ed640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 0x7f182406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1834036000 tx=0x7f183402e070 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:34.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.459+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f184006c5d0 con 0x7f184006aef0 2023-10-14T17:57:34.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.731+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]} v 0) v1 -- 0x7f184006fd60 con 0x7f184006aef0 2023-10-14T17:57:34.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.791+0000 7f183effd640 1 -- 172.21.15.29:0/382455760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_for_number_of_snaps_retention"]}]=0 v2526) v1 ==== 178+0+0 (secure 0 0 0) 0x7f184006fd60 con 0x7f184006aef0 2023-10-14T17:57:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 msgr2=0x7f182406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 0x7f182406d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1834036000 tx=0x7f183402e070 comp rx=0 tx=0).stop 2023-10-14T17:57:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 msgr2=0x7f18400b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:34.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 secure :-1 s=READY pgs=1929 cs=0 l=1 rev1=1 crypto rx=0x7f1840073b00 tx=0x7f183800d9d0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 shutdown_connections 2023-10-14T17:57:34.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f182406af90 0x7f182406d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f184006aef0 0x7f18400b1870 unknown :-1 s=CLOSED pgs=1929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f18400af280 0x7f184006c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 --2- 172.21.15.29:0/382455760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1830003970 0x7f184014d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 >> 172.21.15.29:0/382455760 conn(0x7f18400cd370 msgr2=0x7f18400cd750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:34.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.794+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 shutdown_connections 2023-10-14T17:57:34.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:34.795+0000 7f18471ef640 1 -- 172.21.15.29:0/382455760 wait complete. 2023-10-14T17:57:34.814 INFO:tasks.cephfs_test_runner:Test that number of snaps retained are as per user spec. ... ok 2023-10-14T17:57:34.815 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test listing snap schedules post removal of a schedule 2023-10-14T17:57:34.815 INFO:tasks.cephfs_test_runner:test_snap_schedule_list_post_schedule_remove (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T17:57:34.816 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:35.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.034+0000 7f2284977640 1 Processor -- start 2023-10-14T17:57:35.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.035+0000 7f2284977640 1 -- start start 2023-10-14T17:57:35.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.036+0000 7f2284977640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2280158e20 0x7f22800cacc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.036+0000 7f2284977640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 0x7f22800ce040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.036+0000 7f2284977640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22800ce580 con 0x7f2280158a40 2023-10-14T17:57:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.036+0000 7f2284977640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22800c9830 con 0x7f2280158e20 2023-10-14T17:57:35.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.036+0000 7f2284977640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22800c9270 con 0x7f22800cdc20 2023-10-14T17:57:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2280158a40 0x7f228015d1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42486/0 (socket says 172.21.15.29:42486) 2023-10-14T17:57:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227ed9d640 1 -- 172.21.15.29:0/3360220010 learned_addr learned my addr 172.21.15.29:0/3360220010 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:35.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227e59c640 1 --2- 172.21.15.29:0/3360220010 >> v2:172.21.15.29:3300/0 conn(0x7f2280158e20 0x7f22800cacc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227e59c640 1 -- 172.21.15.29:0/3360220010 >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 msgr2=0x7f22800ce040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227f59e640 1 --2- 172.21.15.29:0/3360220010 >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 0x7f22800ce040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227e59c640 1 --2- 172.21.15.29:0/3360220010 >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 0x7f22800ce040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227e59c640 1 -- 172.21.15.29:0/3360220010 >> v1:172.21.15.29:6789/0 conn(0x7f2280158a40 legacy=0x7f228015d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227dd9b640 1 -- 172.21.15.29:0/3360220010 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183180293 0 0) 0x7f22800ce580 con 0x7f2280158a40 2023-10-14T17:57:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227e59c640 1 -- 172.21.15.29:0/3360220010 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22800c9550 con 0x7f2280158e20 2023-10-14T17:57:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.037+0000 7f227f59e640 1 --2- 172.21.15.29:0/3360220010 >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 0x7f22800ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.038+0000 7f227e59c640 1 --2- 172.21.15.29:0/3360220010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158e20 0x7f22800cacc0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f2280071b70 tx=0x7f227402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5583e23f3521fcb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f227dd9b640 1 -- 172.21.15.29:0/3360220010 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274031830 con 0x7f2280158e20 2023-10-14T17:57:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f227dd9b640 1 -- 172.21.15.29:0/3360220010 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2274031990 con 0x7f2280158e20 2023-10-14T17:57:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f227dd9b640 1 -- 172.21.15.29:0/3360220010 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274031d20 con 0x7f2280158e20 2023-10-14T17:57:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 -- 172.21.15.29:0/3360220010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158e20 msgr2=0x7f22800cacc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 --2- 172.21.15.29:0/3360220010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158e20 0x7f22800cacc0 secure :-1 s=READY pgs=1931 cs=0 l=1 rev1=1 crypto rx=0x7f2280071b70 tx=0x7f227402b8d0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 -- 172.21.15.29:0/3360220010 shutdown_connections 2023-10-14T17:57:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 --2- 172.21.15.29:0/3360220010 >> v2:172.21.15.186:3300/0 conn(0x7f22800cdc20 0x7f22800ce040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 --2- 172.21.15.29:0/3360220010 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158e20 0x7f22800cacc0 unknown :-1 s=CLOSED pgs=1931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 -- 172.21.15.29:0/3360220010 >> 172.21.15.29:0/3360220010 conn(0x7f22800ceb80 msgr2=0x7f22800cef90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 -- 172.21.15.29:0/3360220010 shutdown_connections 2023-10-14T17:57:35.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.039+0000 7f2284977640 1 -- 172.21.15.29:0/3360220010 wait complete. 2023-10-14T17:57:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.041+0000 7f2284977640 1 Processor -- start 2023-10-14T17:57:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.041+0000 7f2284977640 1 -- start start 2023-10-14T17:57:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22800cdc20 0x7f228014e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 0x7f22800b2f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228014f740 con 0x7f2280158a40 2023-10-14T17:57:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f228014fa20 con 0x7f22800cdc20 2023-10-14T17:57:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.042+0000 7f2284977640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f228014fd00 con 0x7f2280158e20 2023-10-14T17:57:35.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60204/0 (socket says 172.21.15.29:60204) 2023-10-14T17:57:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 -- 172.21.15.29:0/3749924256 learned_addr learned my addr 172.21.15.29:0/3749924256 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:35.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227ed9d640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22800cdc20 0x7f228014e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227f59e640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 0x7f22800b2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 -- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 msgr2=0x7f22800b2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 0x7f22800b2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 -- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22800cdc20 msgr2=0x7f228014e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22800cdc20 0x7f228014e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227e59c640 1 -- 172.21.15.29:0/3749924256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2274009400 con 0x7f2280158a40 2023-10-14T17:57:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.043+0000 7f227f59e640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 0x7f22800b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f227e59c640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f2274000c00 tx=0x7f22740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2274004230 con 0x7f2280158a40 2023-10-14T17:57:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2274004390 con 0x7f2280158a40 2023-10-14T17:57:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f227403a8f0 con 0x7f2280158a40 2023-10-14T17:57:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f228014ffe0 con 0x7f2280158a40 2023-10-14T17:57:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.044+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f228014e630 con 0x7f2280158a40 2023-10-14T17:57:35.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.045+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f228004e9b0 con 0x7f2280158a40 2023-10-14T17:57:35.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.046+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f227403aa50 con 0x7f2280158a40 2023-10-14T17:57:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.048+0000 7f22677fe640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 0x7f226c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.048+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f22740abb00 con 0x7f2280158a40 2023-10-14T17:57:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.048+0000 7f227ed9d640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 0x7f226c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.049+0000 7f227ed9d640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 0x7f226c06d4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f22700012e0 tx=0x7f2270007ef0 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.052+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22740ade60 con 0x7f2280158a40 2023-10-14T17:57:35.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.255+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f228015d370 con 0x7f2280158a40 2023-10-14T17:57:35.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.257+0000 7f22677fe640 1 -- 172.21.15.29:0/3749924256 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v169) v1 ==== 74+0+14171 (secure 0 0 0) 0x7f2274035090 con 0x7f2280158a40 2023-10-14T17:57:35.258 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:35.258 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":169,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:32.774491+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6837/1190525894":"2023-10-15T17:57:29.954283+0000","172.21.15.29:6836/1190525894":"2023-10-15T17:57:29.954283+0000","172.21.15.29:0/3687245545":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6833/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:6832/2309931697":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3663692422":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3645494831":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1574130967":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/2381053627":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/1627586322":"2023-10-15T17:57:26.260574+0000","172.21.15.29:0/3251497707":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/80005070":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/2913425840":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/3470978219":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/1610794407":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6833/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:6832/3121942650":"2023-10-15T17:01:29.273525+0000","172.21.15.29:0/542195718":"2023-10-15T17:01:29.273525+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 msgr2=0x7f226c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 0x7f226c06d4a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f22700012e0 tx=0x7f2270007ef0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 msgr2=0x7f22800c3c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 secure :-1 s=READY pgs=1932 cs=0 l=1 rev1=1 crypto rx=0x7f2274000c00 tx=0x7f22740046c0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 shutdown_connections 2023-10-14T17:57:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f226c06afe0 0x7f226c06d4a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2280158e20 0x7f22800b2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2280158a40 0x7f22800c3c50 unknown :-1 s=CLOSED pgs=1932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.259+0000 7f2284977640 1 --2- 172.21.15.29:0/3749924256 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22800cdc20 0x7f228014e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.260+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 >> 172.21.15.29:0/3749924256 conn(0x7f22800ceb80 msgr2=0x7f22800b4150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.260+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 shutdown_connections 2023-10-14T17:57:35.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.260+0000 7f2284977640 1 -- 172.21.15.29:0/3749924256 wait complete. 2023-10-14T17:57:35.281 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:57:35.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:57:35.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.507+0000 7fd06ae94640 1 Processor -- start 2023-10-14T17:57:35.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.508+0000 7fd06ae94640 1 -- start start 2023-10-14T17:57:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.509+0000 7fd06ae94640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd06406a4b0 0x7fd06416c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.509+0000 7fd06ae94640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0640caef0 con 0x7fd0640addb0 2023-10-14T17:57:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.509+0000 7fd06ae94640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0640cb140 con 0x7fd0640caa80 2023-10-14T17:57:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.509+0000 7fd06ae94640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd064161a50 con 0x7fd06406a4b0 2023-10-14T17:57:35.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.510+0000 7fd068c30640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd0640addb0 0x7fd0640c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42490/0 (socket says 172.21.15.29:42490) 2023-10-14T17:57:35.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.510+0000 7fd068c30640 1 -- 172.21.15.29:0/2084536695 learned_addr learned my addr 172.21.15.29:0/2084536695 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:35.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.510+0000 7fd069431640 1 --2- 172.21.15.29:0/2084536695 >> v2:172.21.15.29:3300/0 conn(0x7fd06406a4b0 0x7fd06416c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.510+0000 7fd0637fe640 1 -- 172.21.15.29:0/2084536695 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269794679 0 0) 0x7fd0640caef0 con 0x7fd0640addb0 2023-10-14T17:57:35.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.510+0000 7fd0637fe640 1 -- 172.21.15.29:0/2084536695 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd050003520 con 0x7fd0640addb0 2023-10-14T17:57:35.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.511+0000 7fd069431640 1 -- 172.21.15.29:0/2084536695 >> v1:172.21.15.186:6789/0 conn(0x7fd0640caa80 legacy=0x7fd064069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.511+0000 7fd069431640 1 -- 172.21.15.29:0/2084536695 >> v1:172.21.15.29:6789/0 conn(0x7fd0640addb0 legacy=0x7fd0640c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.511+0000 7fd069431640 1 -- 172.21.15.29:0/2084536695 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd06406b030 con 0x7fd06406a4b0 2023-10-14T17:57:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.512+0000 7fd069431640 1 --2- 172.21.15.29:0/2084536695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06406a4b0 0x7fd06416c400 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99569f3d2688e91c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd0637fe640 1 -- 172.21.15.29:0/2084536695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c030610 con 0x7fd06406a4b0 2023-10-14T17:57:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd0637fe640 1 -- 172.21.15.29:0/2084536695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd05c030770 con 0x7fd06406a4b0 2023-10-14T17:57:35.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd0637fe640 1 -- 172.21.15.29:0/2084536695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd05c030ae0 con 0x7fd06406a4b0 2023-10-14T17:57:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 -- 172.21.15.29:0/2084536695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06406a4b0 msgr2=0x7fd06416c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 --2- 172.21.15.29:0/2084536695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06406a4b0 0x7fd06416c400 secure :-1 s=READY pgs=1934 cs=0 l=1 rev1=1 crypto rx=0x7fd05c0097a0 tx=0x7fd05c02b960 comp rx=0 tx=0).stop 2023-10-14T17:57:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 -- 172.21.15.29:0/2084536695 shutdown_connections 2023-10-14T17:57:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 --2- 172.21.15.29:0/2084536695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd06406a4b0 0x7fd06416c400 unknown :-1 s=CLOSED pgs=1934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 -- 172.21.15.29:0/2084536695 >> 172.21.15.29:0/2084536695 conn(0x7fd0640ce9b0 msgr2=0x7fd0640cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 -- 172.21.15.29:0/2084536695 shutdown_connections 2023-10-14T17:57:35.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.513+0000 7fd06ae94640 1 -- 172.21.15.29:0/2084536695 wait complete. 2023-10-14T17:57:35.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.515+0000 7fd06ae94640 1 Processor -- start 2023-10-14T17:57:35.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.515+0000 7fd06ae94640 1 -- start start 2023-10-14T17:57:35.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd06406a4b0 0x7fd064164300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0640caa80 0x7fd0640c34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd064163260 con 0x7fd0640addb0 2023-10-14T17:57:35.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd064163540 con 0x7fd0640caa80 2023-10-14T17:57:35.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd06ae94640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd064163820 con 0x7fd06406a4b0 2023-10-14T17:57:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd063fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60226/0 (socket says 172.21.15.29:60226) 2023-10-14T17:57:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.516+0000 7fd063fff640 1 -- 172.21.15.29:0/1066851600 learned_addr learned my addr 172.21.15.29:0/1066851600 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd068c30640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd06406a4b0 0x7fd064164300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd069431640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0640caa80 0x7fd0640c34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 -- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd06406a4b0 msgr2=0x7fd064164300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd06406a4b0 0x7fd064164300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 -- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0640caa80 msgr2=0x7fd0640c34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0640caa80 0x7fd0640c34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 -- 172.21.15.29:0/1066851600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd05c009400 con 0x7fd0640addb0 2023-10-14T17:57:35.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.517+0000 7fd063fff640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00c400 tx=0x7fd04c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.518+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c0052d0 con 0x7fd0640addb0 2023-10-14T17:57:35.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.518+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd04c005430 con 0x7fd0640addb0 2023-10-14T17:57:35.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.518+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd064163b00 con 0x7fd0640addb0 2023-10-14T17:57:35.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.518+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd04c003830 con 0x7fd0640addb0 2023-10-14T17:57:35.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.518+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd064164840 con 0x7fd0640addb0 2023-10-14T17:57:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.519+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd06404e160 con 0x7fd0640addb0 2023-10-14T17:57:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.520+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd04c003990 con 0x7fd0640addb0 2023-10-14T17:57:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.521+0000 7fd0617fa640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 0x7fd04406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.521+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fd04c088d50 con 0x7fd0640addb0 2023-10-14T17:57:35.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.521+0000 7fd068c30640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 0x7fd04406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.522+0000 7fd068c30640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 0x7fd04406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd054001560 tx=0x7fd054009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.525+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd04c053cb0 con 0x7fd0640addb0 2023-10-14T17:57:35.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.719+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd06406c5b0 con 0x7fd0640addb0 2023-10-14T17:57:35.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.720+0000 7fd0617fa640 1 -- 172.21.15.29:0/1066851600 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v169) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd04c0170a0 con 0x7fd0640addb0 2023-10-14T17:57:35.722 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:57:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 msgr2=0x7fd04406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 0x7fd04406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fd054001560 tx=0x7fd054009040 comp rx=0 tx=0).stop 2023-10-14T17:57:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 msgr2=0x7fd0640b3d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 secure :-1 s=READY pgs=1935 cs=0 l=1 rev1=1 crypto rx=0x7fd04c00c400 tx=0x7fd04c00c8c0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 shutdown_connections 2023-10-14T17:57:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd04406af90 0x7fd04406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0640caa80 0x7fd0640c34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0640addb0 0x7fd0640b3d60 unknown :-1 s=CLOSED pgs=1935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 --2- 172.21.15.29:0/1066851600 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd06406a4b0 0x7fd064164300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 >> 172.21.15.29:0/1066851600 conn(0x7fd0640ce9b0 msgr2=0x7fd0640c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 shutdown_connections 2023-10-14T17:57:35.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.723+0000 7fd06ae94640 1 -- 172.21.15.29:0/1066851600 wait complete. 2023-10-14T17:57:35.742 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-14T17:57:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.964+0000 7f76a2567640 1 Processor -- start 2023-10-14T17:57:35.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.964+0000 7f76a2567640 1 -- start start 2023-10-14T17:57:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a2567640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f769c15c9f0 0x7f769c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a2567640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f769c15dde0 0x7f769c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a2567640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f769c160a90 con 0x7f769c0c9230 2023-10-14T17:57:35.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a2567640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c0cdec0 con 0x7f769c15dde0 2023-10-14T17:57:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a2567640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f769c0cd900 con 0x7f769c15c9f0 2023-10-14T17:57:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a1d66640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f769c15dde0 0x7f769c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a1d66640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f769c15dde0 0x7f769c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60238/0 (socket says 172.21.15.29:60238) 2023-10-14T17:57:35.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a1d66640 1 -- 172.21.15.29:0/1855399027 learned_addr learned my addr 172.21.15.29:0/1855399027 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.966+0000 7f76a1565640 1 --2- 172.21.15.29:0/1855399027 >> v2:172.21.15.186:3300/0 conn(0x7f769c15c9f0 0x7f769c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.967+0000 7f76a1d66640 1 -- 172.21.15.29:0/1855399027 >> v2:172.21.15.186:3300/0 conn(0x7f769c15c9f0 msgr2=0x7f769c0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:35.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.967+0000 7f76a1d66640 1 --2- 172.21.15.29:0/1855399027 >> v2:172.21.15.186:3300/0 conn(0x7f769c15c9f0 0x7f769c0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.967+0000 7f76a1d66640 1 -- 172.21.15.29:0/1855399027 >> v1:172.21.15.29:6789/0 conn(0x7f769c0c9230 legacy=0x7f769c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.967+0000 7f76a1d66640 1 -- 172.21.15.29:0/1855399027 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769c0cdbe0 con 0x7f769c15dde0 2023-10-14T17:57:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.968+0000 7f76a1d66640 1 --2- 172.21.15.29:0/1855399027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c15dde0 0x7f769c0ac890 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f769c0ad4e0 tx=0x7f769402bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb20307a303c077 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.968+0000 7f7693fff640 1 -- 172.21.15.29:0/1855399027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769402cb10 con 0x7f769c15dde0 2023-10-14T17:57:35.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.968+0000 7f7693fff640 1 -- 172.21.15.29:0/1855399027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769402cc70 con 0x7f769c15dde0 2023-10-14T17:57:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.968+0000 7f7693fff640 1 -- 172.21.15.29:0/1855399027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76940343f0 con 0x7f769c15dde0 2023-10-14T17:57:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 -- 172.21.15.29:0/1855399027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c15dde0 msgr2=0x7f769c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 --2- 172.21.15.29:0/1855399027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c15dde0 0x7f769c0ac890 secure :-1 s=READY pgs=1937 cs=0 l=1 rev1=1 crypto rx=0x7f769c0ad4e0 tx=0x7f769402bf70 comp rx=0 tx=0).stop 2023-10-14T17:57:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 -- 172.21.15.29:0/1855399027 shutdown_connections 2023-10-14T17:57:35.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 --2- 172.21.15.29:0/1855399027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c15dde0 0x7f769c0ac890 unknown :-1 s=CLOSED pgs=1937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 --2- 172.21.15.29:0/1855399027 >> v2:172.21.15.186:3300/0 conn(0x7f769c15c9f0 0x7f769c0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 -- 172.21.15.29:0/1855399027 >> 172.21.15.29:0/1855399027 conn(0x7f769c0ce6a0 msgr2=0x7f769c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 -- 172.21.15.29:0/1855399027 shutdown_connections 2023-10-14T17:57:35.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.969+0000 7f76a2567640 1 -- 172.21.15.29:0/1855399027 wait complete. 2023-10-14T17:57:35.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.971+0000 7f76a2567640 1 Processor -- start 2023-10-14T17:57:35.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.971+0000 7f76a2567640 1 -- start start 2023-10-14T17:57:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f769c15c9f0 0x7f769c0b05e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f769c15dde0 0x7f769c06c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0b10a0 con 0x7f769c0c9230 2023-10-14T17:57:35.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f769c06a590 con 0x7f769c15c9f0 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a2567640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f769c0b0b20 con 0x7f769c15dde0 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a1565640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a1565640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60244/0 (socket says 172.21.15.29:60244) 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a1565640 1 -- 172.21.15.29:0/2331040916 learned_addr learned my addr 172.21.15.29:0/2331040916 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.972+0000 7f76a0d64640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f769c15c9f0 0x7f769c0b05e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 -- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f769c15dde0 msgr2=0x7f769c06c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f769c15dde0 0x7f769c06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 -- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f769c15c9f0 msgr2=0x7f769c0b05e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f769c15c9f0 0x7f769c0b05e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 -- 172.21.15.29:0/2331040916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7694009b30 con 0x7f769c0c9230 2023-10-14T17:57:35.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a1565640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f769800d800 tx=0x7f769800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800b6c0 con 0x7f769c0c9230 2023-10-14T17:57:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769800b820 con 0x7f769c0c9230 2023-10-14T17:57:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f769800bbb0 con 0x7f769c0c9230 2023-10-14T17:57:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.973+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f769c0b0dc0 con 0x7f769c0c9230 2023-10-14T17:57:35.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.974+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f769c06a220 con 0x7f769c0c9230 2023-10-14T17:57:35.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.975+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f769c06a3e0 con 0x7f769c0c9230 2023-10-14T17:57:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.976+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f769800bd10 con 0x7f769c0c9230 2023-10-14T17:57:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.976+0000 7f7691ffb640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 0x7f768006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.977+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f7698089ce0 con 0x7f769c0c9230 2023-10-14T17:57:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.977+0000 7f76a0d64640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 0x7f768006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:35.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.978+0000 7f76a0d64640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 0x7f768006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f769c15e810 tx=0x7f768c009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:35.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:35.981+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7698054c40 con 0x7f769c0c9230 2023-10-14T17:57:36.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.246+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7f769c069da0 con 0x7f769c0c9230 2023-10-14T17:57:36.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.803+0000 7f7691ffb640 1 -- 172.21.15.29:0/2331040916 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2528) v1 ==== 177+0+0 (secure 0 0 0) 0x7f769c069da0 con 0x7f769c0c9230 2023-10-14T17:57:36.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 msgr2=0x7f768006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 0x7f768006d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f769c15e810 tx=0x7f768c009040 comp rx=0 tx=0).stop 2023-10-14T17:57:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 msgr2=0x7f769c0c7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:36.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 secure :-1 s=READY pgs=1938 cs=0 l=1 rev1=1 crypto rx=0x7f769800d800 tx=0x7f769800dcc0 comp rx=0 tx=0).stop 2023-10-14T17:57:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 shutdown_connections 2023-10-14T17:57:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f768006afc0 0x7f768006d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f769c15dde0 0x7f769c06c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:36.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f769c15c9f0 0x7f769c0b05e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 --2- 172.21.15.29:0/2331040916 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f769c0c9230 0x7f769c0c7b70 unknown :-1 s=CLOSED pgs=1938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 >> 172.21.15.29:0/2331040916 conn(0x7f769c0ce6a0 msgr2=0x7f769c0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 shutdown_connections 2023-10-14T17:57:36.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:36.806+0000 7f76a2567640 1 -- 172.21.15.29:0/2331040916 wait complete. 2023-10-14T17:57:36.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T17:57:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.059+0000 7f040d7a9640 1 Processor -- start 2023-10-14T17:57:37.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.059+0000 7f040d7a9640 1 -- start start 2023-10-14T17:57:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f040d7a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f04080caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f040d7a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f04080c9000 0x7f0408069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f040d7a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04080cb140 con 0x7f040806a2e0 2023-10-14T17:57:37.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f040d7a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0408161a50 con 0x7f04080c9000 2023-10-14T17:57:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f040d7a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040806a750 con 0x7f04080addb0 2023-10-14T17:57:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f04077fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f040806a2e0 0x7f04080cc580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42514/0 (socket says 172.21.15.29:42514) 2023-10-14T17:57:37.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f04077fe640 1 -- 172.21.15.29:0/4137775405 learned_addr learned my addr 172.21.15.29:0/4137775405 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:37.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f04067fc640 1 --2- 172.21.15.29:0/4137775405 >> v2:172.21.15.186:3300/0 conn(0x7f04080c9000 0x7f0408069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391105052 0 0) 0x7f04080cb140 con 0x7f040806a2e0 2023-10-14T17:57:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.061+0000 7f0406ffd640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f04080caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f03f4003520 con 0x7f040806a2e0 2023-10-14T17:57:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4076646122 0 0) 0x7f03f4003520 con 0x7f040806a2e0 2023-10-14T17:57:37.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 msgr2=0x7f04080caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f04080caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 >> v2:172.21.15.186:3300/0 conn(0x7f04080c9000 msgr2=0x7f0408069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 --2- 172.21.15.29:0/4137775405 >> v2:172.21.15.186:3300/0 conn(0x7f04080c9000 0x7f0408069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040806b030 con 0x7f040806a2e0 2023-10-14T17:57:37.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0400001670 con 0x7f040806a2e0 2023-10-14T17:57:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 >> v1:172.21.15.29:6789/0 conn(0x7f040806a2e0 legacy=0x7f04080cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f03f4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f40047c0 0x7f03f4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f03f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0408161a50 con 0x7f03f40039a0 2023-10-14T17:57:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040806a750 con 0x7f03f40047c0 2023-10-14T17:57:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03f400b980 con 0x7f04080addb0 2023-10-14T17:57:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.062+0000 7f0406ffd640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f03f4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f04077fe640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f03f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f04067fc640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f40047c0 0x7f03f4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f0406ffd640 1 -- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 msgr2=0x7f03f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f0406ffd640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f03f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f0406ffd640 1 -- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f40047c0 msgr2=0x7f03f4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:37.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f0406ffd640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03f40047c0 0x7f03f4004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f0406ffd640 1 -- 172.21.15.29:0/4137775405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03f4003520 con 0x7f03f40039a0 2023-10-14T17:57:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.063+0000 7f04077fe640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080addb0 0x7f03f4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.064+0000 7f0406ffd640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f03f4003dc0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009fc0 tx=0x7f03f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfdbc1188958c048 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.064+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f802cc20 con 0x7f03f40039a0 2023-10-14T17:57:37.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.064+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f800f040 con 0x7f03f40039a0 2023-10-14T17:57:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.064+0000 7f0405ffb640 1 -- 172.21.15.29:0/4137775405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8034440 con 0x7f03f40039a0 2023-10-14T17:57:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 -- 172.21.15.29:0/4137775405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 msgr2=0x7f03f4003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 --2- 172.21.15.29:0/4137775405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f03f4003dc0 secure :-1 s=READY pgs=1940 cs=0 l=1 rev1=1 crypto rx=0x7f03f8009fc0 tx=0x7f03f40057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f04077fe640 1 -- 172.21.15.29:0/4137775405 reap_dead start 2023-10-14T17:57:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 -- 172.21.15.29:0/4137775405 shutdown_connections 2023-10-14T17:57:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 -- 172.21.15.29:0/4137775405 >> 172.21.15.29:0/4137775405 conn(0x7f04080ce9b0 msgr2=0x7f04080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 -- 172.21.15.29:0/4137775405 shutdown_connections 2023-10-14T17:57:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.065+0000 7f040d7a9640 1 -- 172.21.15.29:0/4137775405 wait complete. 2023-10-14T17:57:37.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.067+0000 7f040d7a9640 1 Processor -- start 2023-10-14T17:57:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.067+0000 7f040d7a9640 1 -- start start 2023-10-14T17:57:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 0x7f040806f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04080b3450 0x7f040806fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040815f590 con 0x7f03f40039a0 2023-10-14T17:57:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04080b4600 con 0x7f04080b3450 2023-10-14T17:57:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f040d7a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04080b39f0 con 0x7f04080b2fd0 2023-10-14T17:57:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f0406ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f0406ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60256/0 (socket says 172.21.15.29:60256) 2023-10-14T17:57:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.068+0000 7f0406ffd640 1 -- 172.21.15.29:0/3290110677 learned_addr learned my addr 172.21.15.29:0/3290110677 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f04077fe640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04080b3450 0x7f040806fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f04067fc640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 0x7f040806f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 -- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 msgr2=0x7f040806f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 0x7f040806f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 -- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04080b3450 msgr2=0x7f040806fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04080b3450 0x7f040806fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 -- 172.21.15.29:0/3290110677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04080b3cd0 con 0x7f03f40039a0 2023-10-14T17:57:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f04067fc640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 0x7f040806f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:37.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.069+0000 7f0406ffd640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e230 tx=0x7f03f8009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.070+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8035d60 con 0x7f03f40039a0 2023-10-14T17:57:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.070+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03f8003990 con 0x7f03f40039a0 2023-10-14T17:57:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.070+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03f8003d00 con 0x7f03f40039a0 2023-10-14T17:57:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.070+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04080b3fb0 con 0x7f03f40039a0 2023-10-14T17:57:37.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.070+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0408162940 con 0x7f03f40039a0 2023-10-14T17:57:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.071+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0408162b00 con 0x7f03f40039a0 2023-10-14T17:57:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.072+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f03f803b050 con 0x7f03f40039a0 2023-10-14T17:57:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.073+0000 7f03ff7fe640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.073+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f03f80ab9f0 con 0x7f03f40039a0 2023-10-14T17:57:37.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.073+0000 7f04067fc640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.074+0000 7f04067fc640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 0x7f03e806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04080b0860 tx=0x7f03f0008040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:37.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.078+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03f8076a90 con 0x7f03f40039a0 2023-10-14T17:57:37.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.273+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f040804e160 con 0x7f03f40039a0 2023-10-14T17:57:37.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.278+0000 7f03ff7fe640 1 -- 172.21.15.29:0/3290110677 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v58)=0 v58) v1 ==== 122+0+0 (secure 0 0 0) 0x7f03f804d0c0 con 0x7f03f40039a0 2023-10-14T17:57:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 msgr2=0x7f03e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 0x7f03e806d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f04080b0860 tx=0x7f03f0008040 comp rx=0 tx=0).stop 2023-10-14T17:57:37.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 msgr2=0x7f040815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 secure :-1 s=READY pgs=1941 cs=0 l=1 rev1=1 crypto rx=0x7f03f800e230 tx=0x7f03f8009750 comp rx=0 tx=0).stop 2023-10-14T17:57:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 shutdown_connections 2023-10-14T17:57:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04080b3450 0x7f040806fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04080b2fd0 0x7f040806f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 --2- 172.21.15.29:0/3290110677 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03f40039a0 0x7f040815f050 unknown :-1 s=CLOSED pgs=1941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 >> 172.21.15.29:0/3290110677 conn(0x7f04080ce9b0 msgr2=0x7f04080ae980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 shutdown_connections 2023-10-14T17:57:37.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:37.281+0000 7f040d7a9640 1 -- 172.21.15.29:0/3290110677 wait complete. 2023-10-14T17:57:37.300 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:57:37.301 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:57:37.422 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:57:37.548 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:57:37.548 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:37.549 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:37.613 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:57:37.613 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:57:37.614 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:37.674 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:57:37.675 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:37.675 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:37.739 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:57:37.739 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:57:37.740 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:37.800 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:57:37.800 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T17:57:37.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:38.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.037+0000 7f78c957c640 1 Processor -- start 2023-10-14T17:57:38.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.037+0000 7f78c957c640 1 -- start start 2023-10-14T17:57:38.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c957c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78c406c7d0 0x7f78c40b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c957c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f78c40b4c70 0x7f78c40b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c957c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78c40cd630 con 0x7f78c406c3f0 2023-10-14T17:57:38.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c957c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c415e7d0 con 0x7f78c406c7d0 2023-10-14T17:57:38.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c957c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78c40ce1b0 con 0x7f78c40b4c70 2023-10-14T17:57:38.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f78c406c3f0 0x7f78c415e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42526/0 (socket says 172.21.15.29:42526) 2023-10-14T17:57:38.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.038+0000 7f78c2ffd640 1 -- 172.21.15.29:0/3458563697 learned_addr learned my addr 172.21.15.29:0/3458563697 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:38.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c37fe640 1 --2- 172.21.15.29:0/3458563697 >> v2:172.21.15.186:3300/0 conn(0x7f78c40b4c70 0x7f78c40b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c27fc640 1 --2- 172.21.15.29:0/3458563697 >> v2:172.21.15.29:3300/0 conn(0x7f78c406c7d0 0x7f78c40b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c1ffb640 1 -- 172.21.15.29:0/3458563697 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936643970 0 0) 0x7f78c40cd630 con 0x7f78c406c3f0 2023-10-14T17:57:38.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c1ffb640 1 -- 172.21.15.29:0/3458563697 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78b0003520 con 0x7f78c406c3f0 2023-10-14T17:57:38.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c27fc640 1 -- 172.21.15.29:0/3458563697 >> v2:172.21.15.186:3300/0 conn(0x7f78c40b4c70 msgr2=0x7f78c40b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:38.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c27fc640 1 --2- 172.21.15.29:0/3458563697 >> v2:172.21.15.186:3300/0 conn(0x7f78c40b4c70 0x7f78c40b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c27fc640 1 -- 172.21.15.29:0/3458563697 >> v1:172.21.15.29:6789/0 conn(0x7f78c406c3f0 legacy=0x7f78c415e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.039+0000 7f78c27fc640 1 -- 172.21.15.29:0/3458563697 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78c40b50a0 con 0x7f78c406c7d0 2023-10-14T17:57:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.040+0000 7f78c27fc640 1 --2- 172.21.15.29:0/3458563697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c7d0 0x7f78c40b4730 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f78c40d27d0 tx=0x7f78ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dd82a453e20360b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.041+0000 7f78c1ffb640 1 -- 172.21.15.29:0/3458563697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac031830 con 0x7f78c406c7d0 2023-10-14T17:57:38.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.041+0000 7f78c1ffb640 1 -- 172.21.15.29:0/3458563697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78ac031990 con 0x7f78c406c7d0 2023-10-14T17:57:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.041+0000 7f78c1ffb640 1 -- 172.21.15.29:0/3458563697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78ac031d20 con 0x7f78c406c7d0 2023-10-14T17:57:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 -- 172.21.15.29:0/3458563697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c7d0 msgr2=0x7f78c40b4730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 --2- 172.21.15.29:0/3458563697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c7d0 0x7f78c40b4730 secure :-1 s=READY pgs=1943 cs=0 l=1 rev1=1 crypto rx=0x7f78c40d27d0 tx=0x7f78ac02b8d0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 -- 172.21.15.29:0/3458563697 shutdown_connections 2023-10-14T17:57:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 --2- 172.21.15.29:0/3458563697 >> v2:172.21.15.186:3300/0 conn(0x7f78c40b4c70 0x7f78c40b1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 --2- 172.21.15.29:0/3458563697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c7d0 0x7f78c40b4730 unknown :-1 s=CLOSED pgs=1943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 -- 172.21.15.29:0/3458563697 >> 172.21.15.29:0/3458563697 conn(0x7f78c40d2cb0 msgr2=0x7f78c40d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 -- 172.21.15.29:0/3458563697 shutdown_connections 2023-10-14T17:57:38.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.042+0000 7f78c957c640 1 -- 172.21.15.29:0/3458563697 wait complete. 2023-10-14T17:57:38.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.044+0000 7f78c957c640 1 Processor -- start 2023-10-14T17:57:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.044+0000 7f78c957c640 1 -- start start 2023-10-14T17:57:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78c406c7d0 0x7f78c4155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78c40b4c70 0x7f78c4158740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c415c190 con 0x7f78c406c3f0 2023-10-14T17:57:38.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78c40c8de0 con 0x7f78c406c7d0 2023-10-14T17:57:38.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c957c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78c415c430 con 0x7f78c40b4c70 2023-10-14T17:57:38.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60268/0 (socket says 172.21.15.29:60268) 2023-10-14T17:57:38.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 -- 172.21.15.29:0/3112577776 learned_addr learned my addr 172.21.15.29:0/3112577776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:38.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 -- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78c40b4c70 msgr2=0x7f78c4158740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78c40b4c70 0x7f78c4158740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 -- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78c406c7d0 msgr2=0x7f78c4155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78c406c7d0 0x7f78c4155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.045+0000 7f78c2ffd640 1 -- 172.21.15.29:0/3112577776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78ac009400 con 0x7f78c406c3f0 2023-10-14T17:57:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78c2ffd640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f78b400cbc0 tx=0x7f78b400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400ad40 con 0x7f78c406c3f0 2023-10-14T17:57:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b4012400 con 0x7f78c406c3f0 2023-10-14T17:57:38.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b400aea0 con 0x7f78c406c3f0 2023-10-14T17:57:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78c415c710 con 0x7f78c406c3f0 2023-10-14T17:57:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.046+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78c415ccd0 con 0x7f78c406c3f0 2023-10-14T17:57:38.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.047+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78c404e160 con 0x7f78c406c3f0 2023-10-14T17:57:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.048+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f78b4007070 con 0x7f78c406c3f0 2023-10-14T17:57:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.049+0000 7f78bb7fe640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 0x7f78a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.049+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f78b4089840 con 0x7f78c406c3f0 2023-10-14T17:57:38.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.050+0000 7f78c27fc640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 0x7f78a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.050+0000 7f78c27fc640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 0x7f78a406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78ac036000 tx=0x7f78ac030000 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.054+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78b4058240 con 0x7f78c406c3f0 2023-10-14T17:57:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.288+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f78c4145ac0 con 0x7f78c406c3f0 2023-10-14T17:57:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.289+0000 7f78bb7fe640 1 -- 172.21.15.29:0/3112577776 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+3702 (secure 0 0 0) 0x7f78b4017020 con 0x7f78c406c3f0 2023-10-14T17:57:38.291 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:38.291 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":76,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6332,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1819929389","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1819929389},{"type":"v1","addr":"172.21.15.186:6833","nonce":1819929389}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":69},{"gid":6344,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/871775773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":871775773},{"type":"v1","addr":"172.21.15.186:6835","nonce":871775773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9099,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/1688239052","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1688239052},{"type":"v1","addr":"172.21.15.29:6835","nonce":1688239052}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":70},{"gid":9507,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/2506663430","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2506663430},{"type":"v1","addr":"172.21.15.29:6837","nonce":2506663430}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":76}],"filesystems":[]} 2023-10-14T17:57:38.291 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 76 2023-10-14T17:57:38.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 msgr2=0x7f78a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 0x7f78a406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f78ac036000 tx=0x7f78ac030000 comp rx=0 tx=0).stop 2023-10-14T17:57:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 msgr2=0x7f78c40c88a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 secure :-1 s=READY pgs=1944 cs=0 l=1 rev1=1 crypto rx=0x7f78b400cbc0 tx=0x7f78b400a430 comp rx=0 tx=0).stop 2023-10-14T17:57:38.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 shutdown_connections 2023-10-14T17:57:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f78a406af90 0x7f78a406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78c40b4c70 0x7f78c4158740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78c406c7d0 0x7f78c4155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 --2- 172.21.15.29:0/3112577776 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78c406c3f0 0x7f78c40c88a0 unknown :-1 s=CLOSED pgs=1944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 >> 172.21.15.29:0/3112577776 conn(0x7f78c40d2cb0 msgr2=0x7f78c4156450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.292+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 shutdown_connections 2023-10-14T17:57:38.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.293+0000 7f78c957c640 1 -- 172.21.15.29:0/3112577776 wait complete. 2023-10-14T17:57:38.314 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T17:57:38.315 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T17:57:38.315 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T17:57:38.315 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:57:38.316 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T17:57:38.316 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T17:57:38.316 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T17:57:38.316 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:57:38.317 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T17:57:38.317 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T17:57:38.317 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T17:57:38.318 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:57:38.318 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T17:57:38.318 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T17:57:38.319 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T17:57:38.319 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:57:38.341 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T17:57:38.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T17:57:38.347 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T17:57:38.347 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T17:57:38.348 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T17:57:38.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T17:57:38.350 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T17:57:38.350 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T17:57:38.397 INFO:tasks.ceph.mds.a:Started 2023-10-14T17:57:38.398 DEBUG:teuthology.parallel:result is None 2023-10-14T17:57:38.399 INFO:tasks.ceph.mds.d:Started 2023-10-14T17:57:38.399 DEBUG:teuthology.parallel:result is None 2023-10-14T17:57:38.405 INFO:tasks.ceph.mds.c:Started 2023-10-14T17:57:38.405 DEBUG:teuthology.parallel:result is None 2023-10-14T17:57:38.406 INFO:tasks.ceph.mds.b:Started 2023-10-14T17:57:38.407 DEBUG:teuthology.parallel:result is None 2023-10-14T17:57:38.407 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T17:57:38.564 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:57:38.564 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T17:57:38.565 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T17:57:38.565 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T17:57:38.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.602+0000 7f98ff20e640 1 Processor -- start 2023-10-14T17:57:38.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.602+0000 7f98ff20e640 1 -- start start 2023-10-14T17:57:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98ff20e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98ff20e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f98f806a2e0 0x7f98f81650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98ff20e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f80caec0 con 0x7f98f80addb0 2023-10-14T17:57:38.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98ff20e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f8161a50 con 0x7f98f806a2e0 2023-10-14T17:57:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98ff20e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f806a750 con 0x7f98f80caa80 2023-10-14T17:57:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98fcfaa640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f98f80addb0 0x7f98f80c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42554/0 (socket says 172.21.15.29:42554) 2023-10-14T17:57:38.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98fcfaa640 1 -- 172.21.15.29:0/868948487 learned_addr learned my addr 172.21.15.29:0/868948487 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98fd7ab640 1 --2- 172.21.15.29:0/868948487 >> v2:172.21.15.186:3300/0 conn(0x7f98f806a2e0 0x7f98f81650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.604+0000 7f98f7fff640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98f77fe640 1 -- 172.21.15.29:0/868948487 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3330484111 0 0) 0x7f98f80caec0 con 0x7f98f80addb0 2023-10-14T17:57:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98fd7ab640 1 -- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 msgr2=0x7f98f8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98fd7ab640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98fd7ab640 1 -- 172.21.15.29:0/868948487 >> v1:172.21.15.29:6789/0 conn(0x7f98f80addb0 legacy=0x7f98f80c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98fd7ab640 1 -- 172.21.15.29:0/868948487 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f80cb140 con 0x7f98f806a2e0 2023-10-14T17:57:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.605+0000 7f98f7fff640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:57:38.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.606+0000 7f98fd7ab640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f806a2e0 0x7f98f81650d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009dc0 tx=0x7f98f002be80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42a1558d516b2cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.606+0000 7f98f77fe640 1 -- 172.21.15.29:0/868948487 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f002cc30 con 0x7f98f806a2e0 2023-10-14T17:57:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.606+0000 7f98f77fe640 1 -- 172.21.15.29:0/868948487 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98f002cd90 con 0x7f98f806a2e0 2023-10-14T17:57:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.606+0000 7f98f77fe640 1 -- 172.21.15.29:0/868948487 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98f0034660 con 0x7f98f806a2e0 2023-10-14T17:57:38.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 -- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f806a2e0 msgr2=0x7f98f81650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f806a2e0 0x7f98f81650d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f98f0009dc0 tx=0x7f98f002be80 comp rx=0 tx=0).stop 2023-10-14T17:57:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 -- 172.21.15.29:0/868948487 shutdown_connections 2023-10-14T17:57:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f806a2e0 0x7f98f81650d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 --2- 172.21.15.29:0/868948487 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 -- 172.21.15.29:0/868948487 >> 172.21.15.29:0/868948487 conn(0x7f98f80ce9b0 msgr2=0x7f98f80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 -- 172.21.15.29:0/868948487 shutdown_connections 2023-10-14T17:57:38.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.607+0000 7f98ff20e640 1 -- 172.21.15.29:0/868948487 wait complete. 2023-10-14T17:57:38.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.609+0000 7f98ff20e640 1 Processor -- start 2023-10-14T17:57:38.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.609+0000 7f98ff20e640 1 -- start start 2023-10-14T17:57:38.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f80addb0 0x7f98f81fa370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8162470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f8162050 con 0x7f98f806a2e0 2023-10-14T17:57:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98f81fb210 con 0x7f98f80addb0 2023-10-14T17:57:38.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98ff20e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98f81fac50 con 0x7f98f80caa80 2023-10-14T17:57:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98fcfaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98fcfaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60314/0 (socket says 172.21.15.29:60314) 2023-10-14T17:57:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.610+0000 7f98fcfaa640 1 -- 172.21.15.29:0/1530071437 learned_addr learned my addr 172.21.15.29:0/1530071437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fd7ab640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8162470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98f7fff640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f80addb0 0x7f98f81fa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 -- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 msgr2=0x7f98f8162470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8162470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 -- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f80addb0 msgr2=0x7f98f81fa370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f80addb0 0x7f98f81fa370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:38.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 -- 172.21.15.29:0/1530071437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e4009400 con 0x7f98f806a2e0 2023-10-14T17:57:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.611+0000 7f98fcfaa640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00c4d0 tx=0x7f98ec00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.612+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec0074e0 con 0x7f98f806a2e0 2023-10-14T17:57:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.612+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98ec007660 con 0x7f98f806a2e0 2023-10-14T17:57:38.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.612+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98ec004a20 con 0x7f98f806a2e0 2023-10-14T17:57:38.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.612+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f0009a70 con 0x7f98f806a2e0 2023-10-14T17:57:38.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.612+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f8161c70 con 0x7f98f806a2e0 2023-10-14T17:57:38.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.613+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98f804e160 con 0x7f98f806a2e0 2023-10-14T17:57:38.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.614+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f98ec004b80 con 0x7f98f806a2e0 2023-10-14T17:57:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.615+0000 7f98f57fa640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.616+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(169..169 src has 1..169) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f98ec0888b0 con 0x7f98f806a2e0 2023-10-14T17:57:38.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.616+0000 7f98f7fff640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.617+0000 7f98f7fff640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 0x7f98dc06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f98f80cde90 tx=0x7f98e4009190 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:38.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.620+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98ec0538d0 con 0x7f98f806a2e0 2023-10-14T17:57:38.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:38.834+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f98f80b1300 con 0x7f98f806a2e0 2023-10-14T17:57:38.979 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:57:38.989 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:57:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.849+0000 7f98f57fa640 1 -- 172.21.15.29:0/1530071437 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v174) v1 ==== 97+0+0 (secure 0 0 0) 0x7f98ec018020 con 0x7f98f806a2e0 2023-10-14T17:57:39.851 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T17:57:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.852+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 msgr2=0x7f98dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.852+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 0x7f98dc06d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f98f80cde90 tx=0x7f98e4009190 comp rx=0 tx=0).stop 2023-10-14T17:57:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.852+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 msgr2=0x7f98f815df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.852+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 secure :-1 s=READY pgs=1951 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00c4d0 tx=0x7f98ec00c990 comp rx=0 tx=0).stop 2023-10-14T17:57:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 shutdown_connections 2023-10-14T17:57:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f98dc06af90 0x7f98dc06d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98f80caa80 0x7f98f8162470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98f80addb0 0x7f98f81fa370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 --2- 172.21.15.29:0/1530071437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98f806a2e0 0x7f98f815df40 unknown :-1 s=CLOSED pgs=1951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 >> 172.21.15.29:0/1530071437 conn(0x7f98f80ce9b0 msgr2=0x7f98f80cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 shutdown_connections 2023-10-14T17:57:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:39.853+0000 7f98ff20e640 1 -- 172.21.15.29:0/1530071437 wait complete. 2023-10-14T17:57:39.874 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T17:57:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.092+0000 7f8d3274e640 1 Processor -- start 2023-10-14T17:57:40.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.093+0000 7f8d3274e640 1 -- start start 2023-10-14T17:57:40.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.094+0000 7f8d3274e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8d2c071d40 0x7f8d2c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.094+0000 7f8d3274e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c0ae4f0 con 0x7f8d2c15d000 2023-10-14T17:57:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.094+0000 7f8d3274e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c06e6f0 con 0x7f8d2c06b4c0 2023-10-14T17:57:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.094+0000 7f8d3274e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c070f80 con 0x7f8d2c071d40 2023-10-14T17:57:40.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.095+0000 7f8d2bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8d2c15d000 0x7f8d2c06adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:42568/0 (socket says 172.21.15.29:42568) 2023-10-14T17:57:40.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.095+0000 7f8d2bfff640 1 -- 172.21.15.29:0/2932627168 learned_addr learned my addr 172.21.15.29:0/2932627168 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.095+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540938045 0 0) 0x7f8d2c0ae4f0 con 0x7f8d2c15d000 2023-10-14T17:57:40.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.095+0000 7f8d30ceb640 1 --2- 172.21.15.29:0/2932627168 >> v2:172.21.15.186:3300/0 conn(0x7f8d2c071d40 0x7f8d2c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.095+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d18003520 con 0x7f8d2c15d000 2023-10-14T17:57:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 701477427 0 0) 0x7f8d18003520 con 0x7f8d2c15d000 2023-10-14T17:57:40.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 >> v2:172.21.15.186:3300/0 conn(0x7f8d2c071d40 msgr2=0x7f8d2c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 --2- 172.21.15.29:0/2932627168 >> v2:172.21.15.186:3300/0 conn(0x7f8d2c071d40 0x7f8d2c06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 >> v1:172.21.15.186:6789/0 conn(0x7f8d2c06b4c0 legacy=0x7f8d2c06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c071260 con 0x7f8d2c15d000 2023-10-14T17:57:40.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8d1c001670 con 0x7f8d2c15d000 2023-10-14T17:57:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 >> v1:172.21.15.29:6789/0 conn(0x7f8d2c15d000 legacy=0x7f8d2c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 0x7f8d18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180047c0 0x7f8d18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d18005180 0x7f8d1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c070f80 con 0x7f8d180039c0 2023-10-14T17:57:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1800c050 con 0x7f8d18005180 2023-10-14T17:57:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1800c210 con 0x7f8d180047c0 2023-10-14T17:57:40.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 0x7f8d18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d30ceb640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d18005180 0x7f8d1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2bfff640 1 -- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180047c0 msgr2=0x7f8d18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.096+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180047c0 0x7f8d18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.097+0000 7f8d2bfff640 1 -- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d18005180 msgr2=0x7f8d1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.097+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d18005180 0x7f8d1800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.097+0000 7f8d2bfff640 1 -- 172.21.15.29:0/2932627168 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18003520 con 0x7f8d180039c0 2023-10-14T17:57:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.097+0000 7f8d30ceb640 1 -- 172.21.15.29:0/2932627168 reap_dead start 2023-10-14T17:57:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.098+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 0x7f8d18003dc0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00c190 tx=0x7f8d180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa1841713483d1fb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.098+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c005ad0 con 0x7f8d180039c0 2023-10-14T17:57:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.098+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c005c30 con 0x7f8d180039c0 2023-10-14T17:57:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.098+0000 7f8d2b7fe640 1 -- 172.21.15.29:0/2932627168 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1c02c7d0 con 0x7f8d180039c0 2023-10-14T17:57:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 -- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 msgr2=0x7f8d18003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 0x7f8d18003dc0 secure :-1 s=READY pgs=1953 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c00c190 tx=0x7f8d180057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 -- 172.21.15.29:0/2932627168 shutdown_connections 2023-10-14T17:57:40.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 --2- 172.21.15.29:0/2932627168 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d180039c0 0x7f8d18003dc0 unknown :-1 s=CLOSED pgs=1953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 -- 172.21.15.29:0/2932627168 >> 172.21.15.29:0/2932627168 conn(0x7f8d2c0b0c80 msgr2=0x7f8d2c0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:40.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 -- 172.21.15.29:0/2932627168 shutdown_connections 2023-10-14T17:57:40.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.099+0000 7f8d3274e640 1 -- 172.21.15.29:0/2932627168 wait complete. 2023-10-14T17:57:40.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.101+0000 7f8d3274e640 1 Processor -- start 2023-10-14T17:57:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.101+0000 7f8d3274e640 1 -- start start 2023-10-14T17:57:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180039c0 0x7f8d2c1601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c15a000 0x7f8d2c15a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c15a9c0 con 0x7f8d2c154dc0 2023-10-14T17:57:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0c3fd0 con 0x7f8d2c15a000 2023-10-14T17:57:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d3274e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0c4240 con 0x7f8d180039c0 2023-10-14T17:57:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60334/0 (socket says 172.21.15.29:60334) 2023-10-14T17:57:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d237fe640 1 -- 172.21.15.29:0/1381556114 learned_addr learned my addr 172.21.15.29:0/1381556114 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d30ceb640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c15a000 0x7f8d2c15a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.102+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180039c0 0x7f8d2c1601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 -- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180039c0 msgr2=0x7f8d2c1601b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180039c0 0x7f8d2c1601b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 -- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c15a000 msgr2=0x7f8d2c15a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c15a000 0x7f8d2c15a480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 -- 172.21.15.29:0/1381556114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c0c4520 con 0x7f8d2c154dc0 2023-10-14T17:57:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d237fe640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400c3c0 tx=0x7f8d1400c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14011790 con 0x7f8d2c154dc0 2023-10-14T17:57:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d140118f0 con 0x7f8d2c154dc0 2023-10-14T17:57:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c0c4800 con 0x7f8d2c154dc0 2023-10-14T17:57:40.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d14011ca0 con 0x7f8d2c154dc0 2023-10-14T17:57:40.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.103+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c0c5000 con 0x7f8d2c154dc0 2023-10-14T17:57:40.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.105+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c04e160 con 0x7f8d2c154dc0 2023-10-14T17:57:40.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.106+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8d1401c460 con 0x7f8d2c154dc0 2023-10-14T17:57:40.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.107+0000 7f8d297fa640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 0x7f8d0c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.107+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f8d14089470 con 0x7f8d2c154dc0 2023-10-14T17:57:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.107+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 0x7f8d0c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.108+0000 7f8d2bfff640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 0x7f8d0c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c034000 tx=0x7f8d1c009030 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.111+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8d14054650 con 0x7f8d2c154dc0 2023-10-14T17:57:40.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.385+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8d2c0cb9f0 con 0x7f8d2c154dc0 2023-10-14T17:57:40.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.387+0000 7f8d297fa640 1 -- 172.21.15.29:0/1381556114 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v41) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f8d14057c70 con 0x7f8d2c154dc0 2023-10-14T17:57:40.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:40.388 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:57:40.388 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T17:57:40.388 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.389 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T17:57:40.389 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T17:57:40.389 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.389 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.389 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:57:40.390 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:57:40.390 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.390 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.390 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.391 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:57:40.392 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:57:40.392 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.392 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.392 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.393 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.393 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T17:57:40.393 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T17:57:40.393 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.393 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.394 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:57:40.394 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:57:40.394 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.394 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.394 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.395 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.395 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T17:57:40.395 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T17:57:40.395 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.395 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.396 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:57:40.396 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:57:40.396 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.396 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.396 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.397 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.398 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.398 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.398 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.398 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.398 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T17:57:40.399 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T17:57:40.399 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.399 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.399 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.399 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T17:57:40.400 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T17:57:40.401 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.401 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.401 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.401 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.401 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.402 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.402 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.402 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T17:57:40.402 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T17:57:40.402 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.403 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.404 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.404 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T17:57:40.404 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T17:57:40.404 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.404 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.405 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T17:57:40.406 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T17:57:40.406 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.406 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.406 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.406 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.407 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.408 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.408 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.408 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T17:57:40.408 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T17:57:40.408 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.409 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.410 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.410 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T17:57:40.410 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T17:57:40.410 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.410 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:57:40.411 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:57:40.411 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.411 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.411 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.411 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.412 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T17:57:40.412 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T17:57:40.412 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.412 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.412 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T17:57:40.413 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T17:57:40.413 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.413 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.413 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.413 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T17:57:40.414 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T17:57:40.414 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.414 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:57:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:57:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:57:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.415 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.416 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.416 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T17:57:40.416 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T17:57:40.417 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.417 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.417 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:57:40.417 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:57:40.417 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:57:40.418 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.418 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.418 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.418 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T17:57:40.418 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T17:57:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:57:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:57:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:57:40.419 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T17:57:40.420 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.421 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:57:40.421 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:57:40.421 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:57:40.421 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.421 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.422 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.422 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.422 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T17:57:40.422 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T17:57:40.422 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.423 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:57:40.423 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:57:40.423 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.423 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.424 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:57:40.424 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:40.424 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T17:57:40.424 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T17:57:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:57:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:57:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:57:40.425 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:57:40.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.426 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:40.426 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T17:57:40.426 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:57:40.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.389+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 msgr2=0x7f8d0c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.389+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 0x7f8d0c06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8d1c034000 tx=0x7f8d1c009030 comp rx=0 tx=0).stop 2023-10-14T17:57:40.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.389+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 msgr2=0x7f8d2c0c5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.389+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 secure :-1 s=READY pgs=1954 cs=0 l=1 rev1=1 crypto rx=0x7f8d1400c3c0 tx=0x7f8d1400c7c0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 shutdown_connections 2023-10-14T17:57:40.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f8d0c06af90 0x7f8d0c06d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8d2c15a000 0x7f8d2c15a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8d2c154dc0 0x7f8d2c0c5470 unknown :-1 s=CLOSED pgs=1954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 --2- 172.21.15.29:0/1381556114 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8d180039c0 0x7f8d2c1601b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 >> 172.21.15.29:0/1381556114 conn(0x7f8d2c0b0c80 msgr2=0x7f8d2c0aebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:40.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 shutdown_connections 2023-10-14T17:57:40.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.390+0000 7f8d3274e640 1 -- 172.21.15.29:0/1381556114 wait complete. 2023-10-14T17:57:40.431 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:40.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.655+0000 7f04131ff640 1 Processor -- start 2023-10-14T17:57:40.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.655+0000 7f04131ff640 1 -- start start 2023-10-14T17:57:40.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f040c0ad970 0x7f040c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f040c15ebc0 0x7f040c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c06c960 con 0x7f040c15ebc0 2023-10-14T17:57:40.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040c15b9d0 con 0x7f040c0ad970 2023-10-14T17:57:40.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04131ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c15bcb0 con 0x7f040c06c040 2023-10-14T17:57:40.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04129fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f040c15ebc0 0x7f040c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04129fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f040c15ebc0 0x7f040c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60340/0 (socket says 172.21.15.29:60340) 2023-10-14T17:57:40.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.657+0000 7f04129fe640 1 -- 172.21.15.29:0/2547024163 learned_addr learned my addr 172.21.15.29:0/2547024163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:40.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04119fc640 1 --2- 172.21.15.29:0/2547024163 >> v2:172.21.15.186:3300/0 conn(0x7f040c0ad970 0x7f040c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04121fd640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04129fe640 1 -- 172.21.15.29:0/2547024163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 msgr2=0x7f040c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04129fe640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04129fe640 1 -- 172.21.15.29:0/2547024163 >> v2:172.21.15.186:3300/0 conn(0x7f040c0ad970 msgr2=0x7f040c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:40.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04129fe640 1 --2- 172.21.15.29:0/2547024163 >> v2:172.21.15.186:3300/0 conn(0x7f040c0ad970 0x7f040c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.658+0000 7f04129fe640 1 -- 172.21.15.29:0/2547024163 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040c15bf90 con 0x7f040c15ebc0 2023-10-14T17:57:40.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.659+0000 7f04129fe640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c15f050 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0404009e50 tx=0x7f040402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fb72cf466f68b8c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.659+0000 7f04111fb640 1 -- 172.21.15.29:0/2547024163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040402ca80 con 0x7f040c15ebc0 2023-10-14T17:57:40.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.659+0000 7f04111fb640 1 -- 172.21.15.29:0/2547024163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040402cbe0 con 0x7f040c15ebc0 2023-10-14T17:57:40.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.659+0000 7f04111fb640 1 -- 172.21.15.29:0/2547024163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f04040343f0 con 0x7f040c15ebc0 2023-10-14T17:57:40.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 -- 172.21.15.29:0/2547024163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 msgr2=0x7f040c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c15f050 secure :-1 s=READY pgs=1955 cs=0 l=1 rev1=1 crypto rx=0x7f0404009e50 tx=0x7f040402bee0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 -- 172.21.15.29:0/2547024163 shutdown_connections 2023-10-14T17:57:40.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c15f050 unknown :-1 s=CLOSED pgs=1955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 --2- 172.21.15.29:0/2547024163 >> v2:172.21.15.186:3300/0 conn(0x7f040c0ad970 0x7f040c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 --2- 172.21.15.29:0/2547024163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 -- 172.21.15.29:0/2547024163 >> 172.21.15.29:0/2547024163 conn(0x7f040c0ce6a0 msgr2=0x7f040c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:40.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 -- 172.21.15.29:0/2547024163 shutdown_connections 2023-10-14T17:57:40.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.660+0000 7f04131ff640 1 -- 172.21.15.29:0/2547024163 wait complete. 2023-10-14T17:57:40.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.662+0000 7f04131ff640 1 Processor -- start 2023-10-14T17:57:40.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.662+0000 7f04131ff640 1 -- start start 2023-10-14T17:57:40.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c1f4470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0ad970 0x7f040c1efb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c0c7dd0 con 0x7f040c15ebc0 2023-10-14T17:57:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1f3990 con 0x7f040c0ad970 2023-10-14T17:57:40.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04131ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040c1f3c70 con 0x7f040c06c040 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04129fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04129fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:60344/0 (socket says 172.21.15.29:60344) 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.663+0000 7f04129fe640 1 -- 172.21.15.29:0/1693329286 learned_addr learned my addr 172.21.15.29:0/1693329286 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04121fd640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c1f4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04119fc640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0ad970 0x7f040c1efb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 -- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 msgr2=0x7f040c1f4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c1f4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 -- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0ad970 msgr2=0x7f040c1efb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0ad970 0x7f040c1efb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 -- 172.21.15.29:0/1693329286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0404009b30 con 0x7f040c15ebc0 2023-10-14T17:57:40.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f04129fe640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0404009e50 tx=0x7f0404004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404040670 con 0x7f040c15ebc0 2023-10-14T17:57:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.664+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04040407d0 con 0x7f040c15ebc0 2023-10-14T17:57:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.665+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0404040b80 con 0x7f040c15ebc0 2023-10-14T17:57:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.665+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f040c1f3f50 con 0x7f040c15ebc0 2023-10-14T17:57:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.665+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f040c1f00a0 con 0x7f040c15ebc0 2023-10-14T17:57:40.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.666+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f040c06b330 con 0x7f040c15ebc0 2023-10-14T17:57:40.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.667+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f040402cd50 con 0x7f040c15ebc0 2023-10-14T17:57:40.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.668+0000 7f0402ffd640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 0x7f03f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:40.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.668+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f04040ac340 con 0x7f040c15ebc0 2023-10-14T17:57:40.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.669+0000 7f04121fd640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 0x7f03f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:40.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.669+0000 7f04121fd640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 0x7f03f006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0408006fd0 tx=0x7f0408008040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:40.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.673+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0404077520 con 0x7f040c15ebc0 2023-10-14T17:57:40.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.871+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f040c04e9b0 con 0x7f040c15ebc0 2023-10-14T17:57:40.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.873+0000 7f0402ffd640 1 -- 172.21.15.29:0/1693329286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v174) v1 ==== 74+0+13041 (secure 0 0 0) 0x7f040407ab40 con 0x7f040c15ebc0 2023-10-14T17:57:40.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:40.913 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":174,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:39.846642+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":19,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":156,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":152,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 msgr2=0x7f03f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 0x7f03f006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0408006fd0 tx=0x7f0408008040 comp rx=0 tx=0).stop 2023-10-14T17:57:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 msgr2=0x7f040c0c74d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 secure :-1 s=READY pgs=1956 cs=0 l=1 rev1=1 crypto rx=0x7f0404009e50 tx=0x7f0404004a40 comp rx=0 tx=0).stop 2023-10-14T17:57:40.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 shutdown_connections 2023-10-14T17:57:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f03f006af90 0x7f03f006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040c15ebc0 0x7f040c0c74d0 unknown :-1 s=CLOSED pgs=1956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f040c0ad970 0x7f040c1efb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 --2- 172.21.15.29:0/1693329286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f040c06c040 0x7f040c1f4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:40.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 >> 172.21.15.29:0/1693329286 conn(0x7f040c0ce6a0 msgr2=0x7f040c0cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:40.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.876+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 shutdown_connections 2023-10-14T17:57:40.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:40.877+0000 7f04131ff640 1 -- 172.21.15.29:0/1693329286 wait complete. 2023-10-14T17:57:40.917 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:57:40.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:57:41.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.155+0000 7fa2865e9640 1 Processor -- start 2023-10-14T17:57:41.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.155+0000 7fa2865e9640 1 -- start start 2023-10-14T17:57:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa2865e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2800b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa2865e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa2800cfaf0 0x7fa2800b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa2865e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2800d3980 con 0x7fa2800b38e0 2023-10-14T17:57:41.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa2865e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2800d3c60 con 0x7fa2800cfaf0 2023-10-14T17:57:41.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa2865e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800ce630 con 0x7fa2800cf710 2023-10-14T17:57:41.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa284b86640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa2800b38e0 0x7fa2800d0aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54938/0 (socket says 172.21.15.29:54938) 2023-10-14T17:57:41.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.157+0000 7fa284b86640 1 -- 172.21.15.29:0/3709439806 learned_addr learned my addr 172.21.15.29:0/3709439806 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:41.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27ffff640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2800b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27f7fe640 1 --2- 172.21.15.29:0/3709439806 >> v2:172.21.15.186:3300/0 conn(0x7fa2800cfaf0 0x7fa2800b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800250239 0 0) 0x7fa2800d3980 con 0x7fa2800b38e0 2023-10-14T17:57:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa268003520 con 0x7fa2800b38e0 2023-10-14T17:57:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1069804887 0 0) 0x7fa268003520 con 0x7fa2800b38e0 2023-10-14T17:57:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 msgr2=0x7fa2800b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2800b2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 >> v2:172.21.15.186:3300/0 conn(0x7fa2800cfaf0 msgr2=0x7fa2800b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 --2- 172.21.15.29:0/3709439806 >> v2:172.21.15.186:3300/0 conn(0x7fa2800cfaf0 0x7fa2800b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2800ce910 con 0x7fa2800b38e0 2023-10-14T17:57:41.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa2780019f0 con 0x7fa2800b38e0 2023-10-14T17:57:41.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 >> v1:172.21.15.29:6789/0 conn(0x7fa2800b38e0 legacy=0x7fa2800d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268003970 0x7fa268003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2680047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa268004d00 0x7fa268005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800d3c60 con 0x7fa268004d00 2023-10-14T17:57:41.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800ce630 con 0x7fa268003970 2023-10-14T17:57:41.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.158+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26800b980 con 0x7fa2800cf710 2023-10-14T17:57:41.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa268004d00 0x7fa268005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa27ffff640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268003970 0x7fa268003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa27f7fe640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2680047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 -- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 msgr2=0x7fa2680047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800cf710 0x7fa2680047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 -- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268003970 msgr2=0x7fa268003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268003970 0x7fa268003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.159+0000 7fa284b86640 1 -- 172.21.15.29:0/3709439806 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa268003520 con 0x7fa268004d00 2023-10-14T17:57:41.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.160+0000 7fa284b86640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa268004d00 0x7fa268005180 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fa27800a830 tx=0x7fa2680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d67e06e47df8580e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.161+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa278034630 con 0x7fa268004d00 2023-10-14T17:57:41.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.161+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27802f070 con 0x7fa268004d00 2023-10-14T17:57:41.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.161+0000 7fa27effd640 1 -- 172.21.15.29:0/3709439806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2780349f0 con 0x7fa268004d00 2023-10-14T17:57:41.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 -- 172.21.15.29:0/3709439806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa268004d00 msgr2=0x7fa268005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3709439806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa268004d00 0x7fa268005180 secure :-1 s=READY pgs=1958 cs=0 l=1 rev1=1 crypto rx=0x7fa27800a830 tx=0x7fa2680057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa284b86640 1 -- 172.21.15.29:0/3709439806 reap_dead start 2023-10-14T17:57:41.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 -- 172.21.15.29:0/3709439806 shutdown_connections 2023-10-14T17:57:41.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 -- 172.21.15.29:0/3709439806 >> 172.21.15.29:0/3709439806 conn(0x7fa2800d2cc0 msgr2=0x7fa2800d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:41.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 -- 172.21.15.29:0/3709439806 shutdown_connections 2023-10-14T17:57:41.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.162+0000 7fa2865e9640 1 -- 172.21.15.29:0/3709439806 wait complete. 2023-10-14T17:57:41.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.164+0000 7fa2865e9640 1 Processor -- start 2023-10-14T17:57:41.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.164+0000 7fa2865e9640 1 -- start start 2023-10-14T17:57:41.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268004d00 0x7fa28015da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800d0870 0x7fa2800d0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28014a390 con 0x7fa2800cc5a0 2023-10-14T17:57:41.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800d1210 con 0x7fa268004d00 2023-10-14T17:57:41.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa2865e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800c9d90 con 0x7fa2800d0870 2023-10-14T17:57:41.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa27f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40804/0 (socket says 172.21.15.29:40804) 2023-10-14T17:57:41.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa27f7fe640 1 -- 172.21.15.29:0/3765162171 learned_addr learned my addr 172.21.15.29:0/3765162171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:41.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.166+0000 7fa27ffff640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268004d00 0x7fa28015da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa284b86640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800d0870 0x7fa2800d0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 -- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800d0870 msgr2=0x7fa2800d0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800d0870 0x7fa2800d0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 -- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268004d00 msgr2=0x7fa28015da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268004d00 0x7fa28015da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 -- 172.21.15.29:0/3765162171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa28015fea0 con 0x7fa2800cc5a0 2023-10-14T17:57:41.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.167+0000 7fa27f7fe640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa2740116d0 tx=0x7fa274011ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.168+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa274015870 con 0x7fa2800cc5a0 2023-10-14T17:57:41.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.168+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2740159d0 con 0x7fa2800cc5a0 2023-10-14T17:57:41.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.168+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa274015d40 con 0x7fa2800cc5a0 2023-10-14T17:57:41.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.168+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa280160180 con 0x7fa2800cc5a0 2023-10-14T17:57:41.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.168+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa280160740 con 0x7fa2800cc5a0 2023-10-14T17:57:41.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.169+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa28004e160 con 0x7fa2800cc5a0 2023-10-14T17:57:41.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.170+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa27400f460 con 0x7fa2800cc5a0 2023-10-14T17:57:41.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.171+0000 7fa27cff9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 0x7fa26c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.171+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fa2740899f0 con 0x7fa2800cc5a0 2023-10-14T17:57:41.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.171+0000 7fa27ffff640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 0x7fa26c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.172+0000 7fa27ffff640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 0x7fa26c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa270009a40 tx=0x7fa270008040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.175+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa274054bf0 con 0x7fa2800cc5a0 2023-10-14T17:57:41.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.363+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa28006b130 con 0x7fa2800cc5a0 2023-10-14T17:57:41.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.364+0000 7fa27cff9640 1 -- 172.21.15.29:0/3765162171 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v174) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa274058210 con 0x7fa2800cc5a0 2023-10-14T17:57:41.365 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:57:41.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 msgr2=0x7fa26c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 0x7fa26c06d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa270009a40 tx=0x7fa270008040 comp rx=0 tx=0).stop 2023-10-14T17:57:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 msgr2=0x7fa2800cb960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 secure :-1 s=READY pgs=1959 cs=0 l=1 rev1=1 crypto rx=0x7fa2740116d0 tx=0x7fa274011ad0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 shutdown_connections 2023-10-14T17:57:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa26c06af90 0x7fa26c06d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800d0870 0x7fa2800d0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa2800cc5a0 0x7fa2800cb960 unknown :-1 s=CLOSED pgs=1959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 --2- 172.21.15.29:0/3765162171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa268004d00 0x7fa28015da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 >> 172.21.15.29:0/3765162171 conn(0x7fa2800d2cc0 msgr2=0x7fa2800b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 shutdown_connections 2023-10-14T17:57:41.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.367+0000 7fa2865e9640 1 -- 172.21.15.29:0/3765162171 wait complete. 2023-10-14T17:57:41.387 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T17:57:41.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.609+0000 7f625c8dc640 1 Processor -- start 2023-10-14T17:57:41.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.610+0000 7f625c8dc640 1 -- start start 2023-10-14T17:57:41.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.611+0000 7f625c8dc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f62580b4c80 0x7f6258167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.611+0000 7f625c8dc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580d15a0 con 0x7f625815cfc0 2023-10-14T17:57:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.611+0000 7f625c8dc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62580d1880 con 0x7f625815cbe0 2023-10-14T17:57:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.611+0000 7f625c8dc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62580d1b60 con 0x7f62580b4c80 2023-10-14T17:57:41.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f624ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f625815cfc0 0x7f62580b4570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54950/0 (socket says 172.21.15.29:54950) 2023-10-14T17:57:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f624ffff640 1 -- 172.21.15.29:0/3303330962 learned_addr learned my addr 172.21.15.29:0/3303330962 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6257fff640 1 --2- 172.21.15.29:0/3303330962 >> v2:172.21.15.186:3300/0 conn(0x7f62580b4c80 0x7f6258167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1404484159 0 0) 0x7f62580d15a0 con 0x7f625815cfc0 2023-10-14T17:57:41.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6244003520 con 0x7f625815cfc0 2023-10-14T17:57:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498543892 0 0) 0x7f6244003520 con 0x7f625815cfc0 2023-10-14T17:57:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 >> v2:172.21.15.186:3300/0 conn(0x7f62580b4c80 msgr2=0x7f6258167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 --2- 172.21.15.29:0/3303330962 >> v2:172.21.15.186:3300/0 conn(0x7f62580b4c80 0x7f6258167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.612+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 >> v1:172.21.15.186:6789/0 conn(0x7f625815cbe0 legacy=0x7f62580d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62580d1e40 con 0x7f625815cfc0 2023-10-14T17:57:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6240001670 con 0x7f625815cfc0 2023-10-14T17:57:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 >> v1:172.21.15.29:6789/0 conn(0x7f625815cfc0 legacy=0x7f62580b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6244003970 0x7f6244003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6244004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6244005180 0x7f624400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62580d1b60 con 0x7f62440047c0 2023-10-14T17:57:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f624400c0d0 con 0x7f6244005180 2023-10-14T17:57:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f624400c290 con 0x7f6244003970 2023-10-14T17:57:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6257fff640 1 --2- 172.21.15.29:0/3303330962 >> v2:172.21.15.186:3300/0 conn(0x7f62580b4c80 0x7f6258167910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:41.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f624ffff640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6244004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f62577fe640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6244003970 0x7f6244003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f6257fff640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6244005180 0x7f624400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f624ffff640 1 -- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6244003970 msgr2=0x7f6244003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f624ffff640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6244003970 0x7f6244003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f624ffff640 1 -- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6244005180 msgr2=0x7f624400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.613+0000 7f624ffff640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6244005180 0x7f624400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.614+0000 7f6257fff640 1 -- 172.21.15.29:0/3303330962 reap_dead start 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.614+0000 7f624ffff640 1 -- 172.21.15.29:0/3303330962 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6244003520 con 0x7f62440047c0 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.615+0000 7f624ffff640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6244004c40 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f6240007fd0 tx=0x7f62440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15669ff43384b1c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.615+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6240005ad0 con 0x7f62440047c0 2023-10-14T17:57:41.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.615+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6240005c30 con 0x7f62440047c0 2023-10-14T17:57:41.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.615+0000 7f6256ffd640 1 -- 172.21.15.29:0/3303330962 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f624002cab0 con 0x7f62440047c0 2023-10-14T17:57:41.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 -- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 msgr2=0x7f6244004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6244004c40 secure :-1 s=READY pgs=1961 cs=0 l=1 rev1=1 crypto rx=0x7f6240007fd0 tx=0x7f62440057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 -- 172.21.15.29:0/3303330962 shutdown_connections 2023-10-14T17:57:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3303330962 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6244004c40 unknown :-1 s=CLOSED pgs=1961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 -- 172.21.15.29:0/3303330962 >> 172.21.15.29:0/3303330962 conn(0x7f62580cf570 msgr2=0x7f62580cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:41.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 -- 172.21.15.29:0/3303330962 shutdown_connections 2023-10-14T17:57:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.616+0000 7f625c8dc640 1 -- 172.21.15.29:0/3303330962 wait complete. 2023-10-14T17:57:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.618+0000 7f625c8dc640 1 Processor -- start 2023-10-14T17:57:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.618+0000 7f625c8dc640 1 -- start start 2023-10-14T17:57:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6258159e60 0x7f625815a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 0x7f62581f5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f625815a800 con 0x7f62440047c0 2023-10-14T17:57:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6258156040 con 0x7f6258159e60 2023-10-14T17:57:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f625c8dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6258155010 con 0x7f62581f5ad0 2023-10-14T17:57:41.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f62577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f62577fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40824/0 (socket says 172.21.15.29:40824) 2023-10-14T17:57:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.619+0000 7f62577fe640 1 -- 172.21.15.29:0/3799229099 learned_addr learned my addr 172.21.15.29:0/3799229099 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f624ffff640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6258159e60 0x7f625815a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f6257fff640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 0x7f62581f5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 -- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 msgr2=0x7f62581f5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 0x7f62581f5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 -- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6258159e60 msgr2=0x7f625815a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6258159e60 0x7f625815a2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 -- 172.21.15.29:0/3799229099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6258155280 con 0x7f62440047c0 2023-10-14T17:57:41.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f6257fff640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 0x7f62581f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f62577fe640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f624800e3c0 tx=0x7f624800e880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f62480052e0 con 0x7f62440047c0 2023-10-14T17:57:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6248005440 con 0x7f62440047c0 2023-10-14T17:57:41.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6248003810 con 0x7f62440047c0 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.620+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6258155560 con 0x7f62440047c0 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.621+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6258155b20 con 0x7f62440047c0 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.622+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f625804e9b0 con 0x7f62440047c0 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.623+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6248003970 con 0x7f62440047c0 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.624+0000 7f6254ff9640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 0x7f623806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:41.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.624+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f6248088ae0 con 0x7f62440047c0 2023-10-14T17:57:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.624+0000 7f624ffff640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 0x7f623806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:41.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.625+0000 7f624ffff640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 0x7f623806d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6240034000 tx=0x7f6240009030 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:41.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.628+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6248053ea0 con 0x7f62440047c0 2023-10-14T17:57:41.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.827+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f625815c650 con 0x7f62440047c0 2023-10-14T17:57:41.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.828+0000 7f6254ff9640 1 -- 172.21.15.29:0/3799229099 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v174) v1 ==== 84+0+65 (secure 0 0 0) 0x7f6248019020 con 0x7f62440047c0 2023-10-14T17:57:41.829 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:41.829 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:57:41.830 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:41.830 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T17:57:41.830 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T17:57:41.830 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:41.831 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:57:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 msgr2=0x7f623806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 0x7f623806d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6240034000 tx=0x7f6240009030 comp rx=0 tx=0).stop 2023-10-14T17:57:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 msgr2=0x7f6258151de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 secure :-1 s=READY pgs=1962 cs=0 l=1 rev1=1 crypto rx=0x7f624800e3c0 tx=0x7f624800e880 comp rx=0 tx=0).stop 2023-10-14T17:57:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 shutdown_connections 2023-10-14T17:57:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f623806afc0 0x7f623806d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62581f5ad0 0x7f62581f5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6258159e60 0x7f625815a2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 --2- 172.21.15.29:0/3799229099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62440047c0 0x7f6258151de0 unknown :-1 s=CLOSED pgs=1962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.830+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 >> 172.21.15.29:0/3799229099 conn(0x7f62580cf570 msgr2=0x7f62580b5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.831+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 shutdown_connections 2023-10-14T17:57:41.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:41.831+0000 7f625c8dc640 1 -- 172.21.15.29:0/3799229099 wait complete. 2023-10-14T17:57:41.852 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T17:57:41.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T17:57:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.085+0000 7f5341f87640 1 Processor -- start 2023-10-14T17:57:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.086+0000 7f5341f87640 1 -- start start 2023-10-14T17:57:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.087+0000 7f5341f87640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c0cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.087+0000 7f5341f87640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f533c0b4ca0 0x7f533c0d06e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.087+0000 7f5341f87640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f533c0b8430 con 0x7f533c15d100 2023-10-14T17:57:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.087+0000 7f5341f87640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f533c0d2160 con 0x7f533c0b4ca0 2023-10-14T17:57:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.087+0000 7f5341f87640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0ce0b0 con 0x7f533c15cd20 2023-10-14T17:57:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.088+0000 7f5341786640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f533c0b4ca0 0x7f533c0d06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.088+0000 7f5341786640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f533c0b4ca0 0x7f533c0d06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:60618/0 (socket says 172.21.15.29:60618) 2023-10-14T17:57:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.088+0000 7f5341786640 1 -- 172.21.15.29:0/1456259178 learned_addr learned my addr 172.21.15.29:0/1456259178 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.088+0000 7f5340f85640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c0cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.089+0000 7f5341786640 1 -- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 msgr2=0x7f533c0cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.089+0000 7f5341786640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c0cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.089+0000 7f5341786640 1 -- 172.21.15.29:0/1456259178 >> v1:172.21.15.186:6789/0 conn(0x7f533c15d100 legacy=0x7f533c0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.089+0000 7f5341786640 1 -- 172.21.15.29:0/1456259178 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f533c0b8680 con 0x7f533c0b4ca0 2023-10-14T17:57:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.090+0000 7f5341786640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c0d06e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5334009870 tx=0x7f533402ba10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9811b9b7072abb6d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f533b7fe640 1 -- 172.21.15.29:0/1456259178 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533402f830 con 0x7f533c0b4ca0 2023-10-14T17:57:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f533b7fe640 1 -- 172.21.15.29:0/1456259178 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f533402f990 con 0x7f533c0b4ca0 2023-10-14T17:57:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f533b7fe640 1 -- 172.21.15.29:0/1456259178 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f533402fd20 con 0x7f533c0b4ca0 2023-10-14T17:57:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 -- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 msgr2=0x7f533c0d06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c0d06e0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f5334009870 tx=0x7f533402ba10 comp rx=0 tx=0).stop 2023-10-14T17:57:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 -- 172.21.15.29:0/1456259178 shutdown_connections 2023-10-14T17:57:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c0d06e0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 --2- 172.21.15.29:0/1456259178 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c0cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.091+0000 7f5341f87640 1 -- 172.21.15.29:0/1456259178 >> 172.21.15.29:0/1456259178 conn(0x7f533c0cf590 msgr2=0x7f533c0cf9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:42.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.092+0000 7f5341f87640 1 -- 172.21.15.29:0/1456259178 shutdown_connections 2023-10-14T17:57:42.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.092+0000 7f5341f87640 1 -- 172.21.15.29:0/1456259178 wait complete. 2023-10-14T17:57:42.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.093+0000 7f5341f87640 1 Processor -- start 2023-10-14T17:57:42.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.093+0000 7f5341f87640 1 -- start start 2023-10-14T17:57:42.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c14e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c157cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c14ead0 con 0x7f533c15d100 2023-10-14T17:57:42.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f533c0b79a0 con 0x7f533c0b4ca0 2023-10-14T17:57:42.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.094+0000 7f5341f87640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f533c15e7f0 con 0x7f533c15cd20 2023-10-14T17:57:42.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40826/0 (socket says 172.21.15.29:40826) 2023-10-14T17:57:42.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 -- 172.21.15.29:0/1795534751 learned_addr learned my addr 172.21.15.29:0/1795534751 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:42.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5340f85640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c14e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f533bfff640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c157cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 -- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 msgr2=0x7f533c157cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c157cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 -- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 msgr2=0x7f533c14e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c14e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 -- 172.21.15.29:0/1795534751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53340094d0 con 0x7f533c15d100 2023-10-14T17:57:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f533bfff640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5340f85640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c14e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.095+0000 7f5341786640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f5334009870 tx=0x7f5334004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:42.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.096+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334037bc0 con 0x7f533c15d100 2023-10-14T17:57:42.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.096+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5334037d40 con 0x7f533c15d100 2023-10-14T17:57:42.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.096+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5334041640 con 0x7f533c15d100 2023-10-14T17:57:42.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.096+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f533c15ead0 con 0x7f533c15d100 2023-10-14T17:57:42.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.096+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f533c15f090 con 0x7f533c15d100 2023-10-14T17:57:42.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.097+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f533c04e9b0 con 0x7f533c15d100 2023-10-14T17:57:42.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.098+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f53340417a0 con 0x7f533c15d100 2023-10-14T17:57:42.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.099+0000 7f53397fa640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 0x7f532806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:42.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.100+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(174..174 src has 1..174) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f533403c070 con 0x7f533c15d100 2023-10-14T17:57:42.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.100+0000 7f5340f85640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 0x7f532806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:42.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.100+0000 7f5340f85640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 0x7f532806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f532c007520 tx=0x7f532c00b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:42.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.104+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53340773e0 con 0x7f533c15d100 2023-10-14T17:57:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.290+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f533c0b15d0 con 0x7f533c15d100 2023-10-14T17:57:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.871+0000 7f53397fa640 1 -- 172.21.15.29:0/1795534751 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v175) v1 ==== 138+0+0 (secure 0 0 0) 0x7f533407aa00 con 0x7f533c15d100 2023-10-14T17:57:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T17:57:42.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 msgr2=0x7f532806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 0x7f532806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f532c007520 tx=0x7f532c00b040 comp rx=0 tx=0).stop 2023-10-14T17:57:42.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 msgr2=0x7f533c158790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:42.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 secure :-1 s=READY pgs=1963 cs=0 l=1 rev1=1 crypto rx=0x7f5334009870 tx=0x7f5334004730 comp rx=0 tx=0).stop 2023-10-14T17:57:42.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 shutdown_connections 2023-10-14T17:57:42.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f532806af90 0x7f532806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f533c15d100 0x7f533c158790 unknown :-1 s=CLOSED pgs=1963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f533c15cd20 0x7f533c157cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 --2- 172.21.15.29:0/1795534751 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f533c0b4ca0 0x7f533c14e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.875+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 >> 172.21.15.29:0/1795534751 conn(0x7f533c0cf590 msgr2=0x7f533c0b5b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:42.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.876+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 shutdown_connections 2023-10-14T17:57:42.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:42.876+0000 7f5341f87640 1 -- 172.21.15.29:0/1795534751 wait complete. 2023-10-14T17:57:42.904 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T17:57:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.115+0000 7f92d0bff640 1 Processor -- start 2023-10-14T17:57:43.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.115+0000 7f92d0bff640 1 -- start start 2023-10-14T17:57:43.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.116+0000 7f92d0bff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92cc159ac0 0x7f92cc15e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.116+0000 7f92d0bff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 0x7f92cc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.116+0000 7f92d0bff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 0x7f92cc070970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92d0bff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc070eb0 con 0x7f92cc159ac0 2023-10-14T17:57:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92d0bff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92cc15e5c0 con 0x7f92cc0704e0 2023-10-14T17:57:43.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92d0bff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc06b810 con 0x7f92cc0cc270 2023-10-14T17:57:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92cc159ac0 0x7f92cc15e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92cc159ac0 0x7f92cc15e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40832/0 (socket says 172.21.15.29:40832) 2023-10-14T17:57:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cb7fe640 1 -- 172.21.15.29:0/516952330 learned_addr learned my addr 172.21.15.29:0/516952330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cbfff640 1 --2- 172.21.15.29:0/516952330 >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 0x7f92cc070970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92caffd640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 0x7f92cc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cb7fe640 1 -- 172.21.15.29:0/516952330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 msgr2=0x7f92cc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:43.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.117+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 0x7f92cc0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.118+0000 7f92cb7fe640 1 -- 172.21.15.29:0/516952330 >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 msgr2=0x7f92cc070970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.118+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/516952330 >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 0x7f92cc070970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.118+0000 7f92cb7fe640 1 -- 172.21.15.29:0/516952330 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92cc0cc810 con 0x7f92cc159ac0 2023-10-14T17:57:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.118+0000 7f92cbfff640 1 --2- 172.21.15.29:0/516952330 >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 0x7f92cc070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:43.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.118+0000 7f92caffd640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 0x7f92cc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.119+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc15e080 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f92cc06ce30 tx=0x7f92bc02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120e74209d9a8711 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.119+0000 7f92ca7fc640 1 -- 172.21.15.29:0/516952330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc031860 con 0x7f92cc159ac0 2023-10-14T17:57:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.119+0000 7f92ca7fc640 1 -- 172.21.15.29:0/516952330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92bc0319c0 con 0x7f92cc159ac0 2023-10-14T17:57:43.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.119+0000 7f92ca7fc640 1 -- 172.21.15.29:0/516952330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92bc031d50 con 0x7f92cc159ac0 2023-10-14T17:57:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 -- 172.21.15.29:0/516952330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 msgr2=0x7f92cc15e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc15e080 secure :-1 s=READY pgs=1964 cs=0 l=1 rev1=1 crypto rx=0x7f92cc06ce30 tx=0x7f92bc02b900 comp rx=0 tx=0).stop 2023-10-14T17:57:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 -- 172.21.15.29:0/516952330 shutdown_connections 2023-10-14T17:57:43.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 --2- 172.21.15.29:0/516952330 >> v2:172.21.15.186:3300/0 conn(0x7f92cc0704e0 0x7f92cc070970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0cc270 0x7f92cc0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 --2- 172.21.15.29:0/516952330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc15e080 unknown :-1 s=CLOSED pgs=1964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 -- 172.21.15.29:0/516952330 >> 172.21.15.29:0/516952330 conn(0x7f92cc0ce6a0 msgr2=0x7f92cc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 -- 172.21.15.29:0/516952330 shutdown_connections 2023-10-14T17:57:43.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.120+0000 7f92d0bff640 1 -- 172.21.15.29:0/516952330 wait complete. 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.122+0000 7f92d0bff640 1 Processor -- start 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.122+0000 7f92d0bff640 1 -- start start 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0704e0 0x7f92cc155f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92cc0cc270 0x7f92cc15b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc156480 con 0x7f92cc159ac0 2023-10-14T17:57:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc15a530 con 0x7f92cc0cc270 2023-10-14T17:57:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92d0bff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92cc15a810 con 0x7f92cc0704e0 2023-10-14T17:57:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.123+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40838/0 (socket says 172.21.15.29:40838) 2023-10-14T17:57:43.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 -- 172.21.15.29:0/1140916456 learned_addr learned my addr 172.21.15.29:0/1140916456 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0704e0 0x7f92cc155f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92caffd640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92cc0cc270 0x7f92cc15b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 -- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0704e0 msgr2=0x7f92cc155f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0704e0 0x7f92cc155f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 -- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92cc0cc270 msgr2=0x7f92cc15b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92cc0cc270 0x7f92cc15b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 -- 172.21.15.29:0/1140916456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92bc0094d0 con 0x7f92cc159ac0 2023-10-14T17:57:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.124+0000 7f92cbfff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f92c400cd10 tx=0x7f92c400a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.125+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400f680 con 0x7f92cc159ac0 2023-10-14T17:57:43.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.125+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92c400f800 con 0x7f92cc159ac0 2023-10-14T17:57:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.125+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92cc15aaf0 con 0x7f92cc159ac0 2023-10-14T17:57:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.125+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92c400fbb0 con 0x7f92cc159ac0 2023-10-14T17:57:43.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.125+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92cc15b0b0 con 0x7f92cc159ac0 2023-10-14T17:57:43.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.126+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92cc04e9b0 con 0x7f92cc159ac0 2023-10-14T17:57:43.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.128+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f92c400b040 con 0x7f92cc159ac0 2023-10-14T17:57:43.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.129+0000 7f92b3fff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 0x7f92b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.129+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(175..175 src has 1..175) v4 ==== 5343+0+0 (secure 0 0 0) 0x7f92c408a3f0 con 0x7f92cc159ac0 2023-10-14T17:57:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.129+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 0x7f92b806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:43.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.130+0000 7f92cb7fe640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 0x7f92b806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f92bc036000 tx=0x7f92bc03c040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:43.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.133+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92c4055460 con 0x7f92cc159ac0 2023-10-14T17:57:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.329+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f92cc0b4620 con 0x7f92cc159ac0 2023-10-14T17:57:43.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.878+0000 7f92b3fff640 1 -- 172.21.15.29:0/1140916456 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v176) v1 ==== 170+0+0 (secure 0 0 0) 0x7f92c4058a80 con 0x7f92cc159ac0 2023-10-14T17:57:43.880 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T17:57:43.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.883+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 msgr2=0x7f92b806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:43.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.883+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 0x7f92b806d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f92bc036000 tx=0x7f92bc03c040 comp rx=0 tx=0).stop 2023-10-14T17:57:43.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.883+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 msgr2=0x7f92cc0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:43.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.883+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 secure :-1 s=READY pgs=1965 cs=0 l=1 rev1=1 crypto rx=0x7f92c400cd10 tx=0x7f92c400a650 comp rx=0 tx=0).stop 2023-10-14T17:57:43.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 shutdown_connections 2023-10-14T17:57:43.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f92b806af90 0x7f92b806d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92cc159ac0 0x7f92cc0ac890 unknown :-1 s=CLOSED pgs=1965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92cc0cc270 0x7f92cc15b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 --2- 172.21.15.29:0/1140916456 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92cc0704e0 0x7f92cc155f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:43.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 >> 172.21.15.29:0/1140916456 conn(0x7f92cc0ce6a0 msgr2=0x7f92cc15d5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:43.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 shutdown_connections 2023-10-14T17:57:43.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:43.884+0000 7f92d0bff640 1 -- 172.21.15.29:0/1140916456 wait complete. 2023-10-14T17:57:43.920 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T17:57:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.110+0000 7f0f5615a640 1 Processor -- start 2023-10-14T17:57:44.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.110+0000 7f0f5615a640 1 -- start start 2023-10-14T17:57:44.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.111+0000 7f0f5615a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.111+0000 7f0f5615a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f500c9ca0 0x7f0f5006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.111+0000 7f0f5615a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f500cb030 con 0x7f0f500ca3b0 2023-10-14T17:57:44.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.111+0000 7f0f5615a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f500c91c0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.111+0000 7f0f5615a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500c94a0 con 0x7f0f5015ad70 2023-10-14T17:57:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.112+0000 7f0f4f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.112+0000 7f0f4effd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f500c9ca0 0x7f0f5006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.112+0000 7f0f4ffff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f0f500ca3b0 0x7f0f500ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:42244/0 (socket says 172.21.15.29:42244) 2023-10-14T17:57:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.112+0000 7f0f4effd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f500c9ca0 0x7f0f5006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:60654/0 (socket says 172.21.15.29:60654) 2023-10-14T17:57:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.112+0000 7f0f4ffff640 1 -- 172.21.15.29:0/1223421228 learned_addr learned my addr 172.21.15.29:0/1223421228 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.113+0000 7f0f4effd640 1 -- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 msgr2=0x7f0f50069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.113+0000 7f0f4effd640 1 --2- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.113+0000 7f0f4effd640 1 -- 172.21.15.29:0/1223421228 >> v1:172.21.15.186:6789/0 conn(0x7f0f500ca3b0 legacy=0x7f0f500ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.113+0000 7f0f4effd640 1 -- 172.21.15.29:0/1223421228 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f500c9780 con 0x7f0f500c9ca0 2023-10-14T17:57:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.114+0000 7f0f4effd640 1 --2- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500c9ca0 0x7f0f5006a2e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009d90 tx=0x7f0f4402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e5c4ea022ec53b2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f4e7fc640 1 -- 172.21.15.29:0/1223421228 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4402cc50 con 0x7f0f500c9ca0 2023-10-14T17:57:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f4e7fc640 1 -- 172.21.15.29:0/1223421228 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4402cdb0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f4e7fc640 1 -- 172.21.15.29:0/1223421228 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f44034660 con 0x7f0f500c9ca0 2023-10-14T17:57:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f5615a640 1 -- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500c9ca0 msgr2=0x7f0f5006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f5615a640 1 --2- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500c9ca0 0x7f0f5006a2e0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f0f44009d90 tx=0x7f0f4402bea0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.115+0000 7f0f5615a640 1 -- 172.21.15.29:0/1223421228 shutdown_connections 2023-10-14T17:57:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.116+0000 7f0f5615a640 1 --2- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500c9ca0 0x7f0f5006a2e0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.116+0000 7f0f5615a640 1 --2- 172.21.15.29:0/1223421228 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.116+0000 7f0f5615a640 1 -- 172.21.15.29:0/1223421228 >> 172.21.15.29:0/1223421228 conn(0x7f0f500ce9b0 msgr2=0x7f0f500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.116+0000 7f0f5615a640 1 -- 172.21.15.29:0/1223421228 shutdown_connections 2023-10-14T17:57:44.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.116+0000 7f0f5615a640 1 -- 172.21.15.29:0/1223421228 wait complete. 2023-10-14T17:57:44.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.117+0000 7f0f5615a640 1 Processor -- start 2023-10-14T17:57:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 -- start start 2023-10-14T17:57:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500ca3b0 0x7f0f500c6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500c80c0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500c84e0 con 0x7f0f500ca3b0 2023-10-14T17:57:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.118+0000 7f0f5615a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f500c7440 con 0x7f0f5015ad70 2023-10-14T17:57:44.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40852/0 (socket says 172.21.15.29:40852) 2023-10-14T17:57:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 -- 172.21.15.29:0/803636254 learned_addr learned my addr 172.21.15.29:0/803636254 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4effd640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500ca3b0 0x7f0f500c6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4ffff640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 -- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 msgr2=0x7f0f50159b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50159b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.119+0000 7f0f4f7fe640 1 -- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500ca3b0 msgr2=0x7f0f500c6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f4f7fe640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500ca3b0 0x7f0f500c6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f4f7fe640 1 -- 172.21.15.29:0/803636254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f44009a70 con 0x7f0f500c9ca0 2023-10-14T17:57:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f4ffff640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f4f7fe640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000e870 tx=0x7f0f4000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f400094f0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f4000c940 con 0x7f0f500c9ca0 2023-10-14T17:57:44.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f500c7720 con 0x7f0f500c9ca0 2023-10-14T17:57:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f4000ccf0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.120+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f500c7ce0 con 0x7f0f500c9ca0 2023-10-14T17:57:44.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.121+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f500c8330 con 0x7f0f500c9ca0 2023-10-14T17:57:44.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.123+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0f40013470 con 0x7f0f500c9ca0 2023-10-14T17:57:44.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.123+0000 7f0f37fff640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 0x7f0f3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:44.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.124+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(176..176 src has 1..176) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f0f40089820 con 0x7f0f500c9ca0 2023-10-14T17:57:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.124+0000 7f0f4effd640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 0x7f0f3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:44.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.125+0000 7f0f4effd640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 0x7f0f3c06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f44036000 tx=0x7f0f44030000 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:44.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.129+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0f40054850 con 0x7f0f500c9ca0 2023-10-14T17:57:44.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.365+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f0f5004e160 con 0x7f0f500c9ca0 2023-10-14T17:57:44.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.928+0000 7f0f37fff640 1 -- 172.21.15.29:0/803636254 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 20 and data pool 21 v79) v1 ==== 174+0+0 (secure 0 0 0) 0x7f0f40057e70 con 0x7f0f500c9ca0 2023-10-14T17:57:44.932 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 20 and data pool 21 2023-10-14T17:57:44.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.943+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 msgr2=0x7f0f3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.943+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 0x7f0f3c06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0f44036000 tx=0x7f0f44030000 comp rx=0 tx=0).stop 2023-10-14T17:57:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.943+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 msgr2=0x7f0f500b2e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:44.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.943+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 secure :-1 s=READY pgs=1966 cs=0 l=1 rev1=1 crypto rx=0x7f0f4000e870 tx=0x7f0f4000ed30 comp rx=0 tx=0).stop 2023-10-14T17:57:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 shutdown_connections 2023-10-14T17:57:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f0f3c06af90 0x7f0f3c06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f5015ad70 0x7f0f50159b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f500ca3b0 0x7f0f500c6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 --2- 172.21.15.29:0/803636254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f500c9ca0 0x7f0f500b2e80 unknown :-1 s=CLOSED pgs=1966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 >> 172.21.15.29:0/803636254 conn(0x7f0f500ce9b0 msgr2=0x7f0f500ad8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 shutdown_connections 2023-10-14T17:57:44.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:44.944+0000 7f0f5615a640 1 -- 172.21.15.29:0/803636254 wait complete. 2023-10-14T17:57:44.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.158+0000 7f1191cf2640 1 Processor -- start 2023-10-14T17:57:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.158+0000 7f1191cf2640 1 -- start start 2023-10-14T17:57:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.159+0000 7f1191cf2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f118c164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f1191cf2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c06f7f0 con 0x7f118c159f00 2023-10-14T17:57:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f1191cf2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118c06c5e0 con 0x7f118c159b20 2023-10-14T17:57:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f1191cf2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c06c8c0 con 0x7f118c0b0b20 2023-10-14T17:57:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f118affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f118c159f00 0x7f118c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54952/0 (socket says 172.21.15.29:54952) 2023-10-14T17:57:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f118affd640 1 -- 172.21.15.29:0/1818657076 learned_addr learned my addr 172.21.15.29:0/1818657076 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f118bfff640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f118c164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2349306934 0 0) 0x7f118c06f7f0 con 0x7f118c159f00 2023-10-14T17:57:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.160+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003520 con 0x7f118c159f00 2023-10-14T17:57:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4069346607 0 0) 0x7f1174003520 con 0x7f118c159f00 2023-10-14T17:57:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 msgr2=0x7f118c164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f118c164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 >> v1:172.21.15.186:6789/0 conn(0x7f118c159b20 legacy=0x7f118c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c06cba0 con 0x7f118c159f00 2023-10-14T17:57:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1180001670 con 0x7f118c159f00 2023-10-14T17:57:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 >> v1:172.21.15.29:6789/0 conn(0x7f118c159f00 legacy=0x7f118c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11740039a0 0x7f1174003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740047c0 0x7f1174004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f1174005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c06c8c0 con 0x7f11740039a0 2023-10-14T17:57:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117400dc70 con 0x7f11740047c0 2023-10-14T17:57:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117400de30 con 0x7f118c0b0b20 2023-10-14T17:57:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11740039a0 0x7f1174003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118affd640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740047c0 0x7f1174004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.161+0000 7f118bfff640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f1174005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118b7fe640 1 -- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 msgr2=0x7f1174005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f1174005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118b7fe640 1 -- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740047c0 msgr2=0x7f1174004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740047c0 0x7f1174004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118b7fe640 1 -- 172.21.15.29:0/1818657076 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174003520 con 0x7f11740039a0 2023-10-14T17:57:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.162+0000 7f118bfff640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0b0b20 0x7f1174005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11740039a0 0x7f1174003dc0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f117c001720 tx=0x7f11740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e7e690ee06e709 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c0052d0 con 0x7f11740039a0 2023-10-14T17:57:45.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c02e050 con 0x7f11740039a0 2023-10-14T17:57:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f118a7fc640 1 -- 172.21.15.29:0/1818657076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f117c0035f0 con 0x7f11740039a0 2023-10-14T17:57:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f1191cf2640 1 -- 172.21.15.29:0/1818657076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11740039a0 msgr2=0x7f1174003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1818657076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11740039a0 0x7f1174003dc0 secure :-1 s=READY pgs=1968 cs=0 l=1 rev1=1 crypto rx=0x7f117c001720 tx=0x7f11740057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f118bfff640 1 -- 172.21.15.29:0/1818657076 reap_dead start 2023-10-14T17:57:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f1191cf2640 1 -- 172.21.15.29:0/1818657076 shutdown_connections 2023-10-14T17:57:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.163+0000 7f1191cf2640 1 -- 172.21.15.29:0/1818657076 >> 172.21.15.29:0/1818657076 conn(0x7f118c0cb410 msgr2=0x7f118c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.164+0000 7f1191cf2640 1 -- 172.21.15.29:0/1818657076 shutdown_connections 2023-10-14T17:57:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.164+0000 7f1191cf2640 1 -- 172.21.15.29:0/1818657076 wait complete. 2023-10-14T17:57:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.165+0000 7f1191cf2640 1 Processor -- start 2023-10-14T17:57:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.165+0000 7f1191cf2640 1 -- start start 2023-10-14T17:57:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740039a0 0x7f118c0c71f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0c5f90 0x7f118c0c63f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c156270 con 0x7f118c155a40 2023-10-14T17:57:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0c8560 con 0x7f11740039a0 2023-10-14T17:57:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.166+0000 7f1191cf2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118c0c8840 con 0x7f118c0c5f90 2023-10-14T17:57:45.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40870/0 (socket says 172.21.15.29:40870) 2023-10-14T17:57:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 -- 172.21.15.29:0/1206023580 learned_addr learned my addr 172.21.15.29:0/1206023580 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118bfff640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0c5f90 0x7f118c0c63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740039a0 0x7f118c0c71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 -- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0c5f90 msgr2=0x7f118c0c63f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0c5f90 0x7f118c0c63f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 -- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740039a0 msgr2=0x7f118c0c71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740039a0 0x7f118c0c71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 -- 172.21.15.29:0/1206023580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118c0c69f0 con 0x7f118c155a40 2023-10-14T17:57:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f118affd640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f1180001200 tx=0x7f118000ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.167+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118000ee00 con 0x7f118c155a40 2023-10-14T17:57:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.168+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f118000c7f0 con 0x7f118c155a40 2023-10-14T17:57:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.168+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f118000cba0 con 0x7f118c155a40 2023-10-14T17:57:45.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.168+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f118c0c6cd0 con 0x7f118c155a40 2023-10-14T17:57:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.168+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f118c15bde0 con 0x7f118c155a40 2023-10-14T17:57:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.169+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f118c04e160 con 0x7f118c155a40 2023-10-14T17:57:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.170+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f118000cd00 con 0x7f118c155a40 2023-10-14T17:57:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.171+0000 7f1173fff640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 0x7f117806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.171+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f11800896a0 con 0x7f118c155a40 2023-10-14T17:57:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.171+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 0x7f117806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.172+0000 7f118b7fe640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 0x7f117806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f117c033000 tx=0x7f117c009030 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.175+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11800545c0 con 0x7f118c155a40 2023-10-14T17:57:45.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.372+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f118c0b10b0 con 0x7f118c155a40 2023-10-14T17:57:45.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.374+0000 7f1173fff640 1 -- 172.21.15.29:0/1206023580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16766 (secure 0 0 0) 0x7f1180057be0 con 0x7f118c155a40 2023-10-14T17:57:45.376 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:45.377 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":177,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:44.366933+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:45.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 msgr2=0x7f117806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 0x7f117806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f117c033000 tx=0x7f117c009030 comp rx=0 tx=0).stop 2023-10-14T17:57:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 msgr2=0x7f118c15a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 secure :-1 s=READY pgs=1969 cs=0 l=1 rev1=1 crypto rx=0x7f1180001200 tx=0x7f118000ec10 comp rx=0 tx=0).stop 2023-10-14T17:57:45.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 shutdown_connections 2023-10-14T17:57:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f117806af90 0x7f117806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f118c0c5f90 0x7f118c0c63f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f118c155a40 0x7f118c15a310 unknown :-1 s=CLOSED pgs=1969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 --2- 172.21.15.29:0/1206023580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11740039a0 0x7f118c0c71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.377+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 >> 172.21.15.29:0/1206023580 conn(0x7f118c0cb410 msgr2=0x7f118c0cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.378+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 shutdown_connections 2023-10-14T17:57:45.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.378+0000 7f1191cf2640 1 -- 172.21.15.29:0/1206023580 wait complete. 2023-10-14T17:57:45.398 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:45.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.627+0000 7fc0ec8bb640 1 Processor -- start 2023-10-14T17:57:45.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.627+0000 7fc0ec8bb640 1 -- start start 2023-10-14T17:57:45.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.628+0000 7fc0ec8bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0e815b320 0x7fc0e80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.628+0000 7fc0ec8bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e815b9d0 con 0x7fc0e806c040 2023-10-14T17:57:45.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.628+0000 7fc0ec8bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0e815bcb0 con 0x7fc0e80ad970 2023-10-14T17:57:45.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.628+0000 7fc0ec8bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0e815b7d0 con 0x7fc0e815b320 2023-10-14T17:57:45.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc0e806c040 0x7fc0e806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54964/0 (socket says 172.21.15.29:54964) 2023-10-14T17:57:45.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e77fe640 1 -- 172.21.15.29:0/4294524038 learned_addr learned my addr 172.21.15.29:0/4294524038 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:45.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e7fff640 1 --2- 172.21.15.29:0/4294524038 >> v2:172.21.15.29:3300/0 conn(0x7fc0e815b320 0x7fc0e80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e7fff640 1 -- 172.21.15.29:0/4294524038 >> v1:172.21.15.186:6789/0 conn(0x7fc0e80ad970 legacy=0x7fc0e80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e7fff640 1 -- 172.21.15.29:0/4294524038 >> v1:172.21.15.29:6789/0 conn(0x7fc0e806c040 legacy=0x7fc0e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.629+0000 7fc0e7fff640 1 -- 172.21.15.29:0/4294524038 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e815bf90 con 0x7fc0e815b320 2023-10-14T17:57:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0e7fff640 1 --2- 172.21.15.29:0/4294524038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e815b320 0x7fc0e80cd180 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00097d0 tx=0x7fc0e002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243492f15e6788f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0e67fc640 1 -- 172.21.15.29:0/4294524038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e002d040 con 0x7fc0e815b320 2023-10-14T17:57:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0e67fc640 1 -- 172.21.15.29:0/4294524038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e0031810 con 0x7fc0e815b320 2023-10-14T17:57:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0e67fc640 1 -- 172.21.15.29:0/4294524038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0031ba0 con 0x7fc0e815b320 2023-10-14T17:57:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/4294524038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e815b320 msgr2=0x7fc0e80cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.631+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/4294524038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e815b320 0x7fc0e80cd180 secure :-1 s=READY pgs=1971 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00097d0 tx=0x7fc0e002b8b0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.632+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/4294524038 shutdown_connections 2023-10-14T17:57:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.632+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/4294524038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e815b320 0x7fc0e80cd180 unknown :-1 s=CLOSED pgs=1971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.632+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/4294524038 >> 172.21.15.29:0/4294524038 conn(0x7fc0e80ce6a0 msgr2=0x7fc0e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.632+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/4294524038 shutdown_connections 2023-10-14T17:57:45.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.632+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/4294524038 wait complete. 2023-10-14T17:57:45.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.633+0000 7fc0ec8bb640 1 Processor -- start 2023-10-14T17:57:45.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.634+0000 7fc0ec8bb640 1 -- start start 2023-10-14T17:57:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.634+0000 7fc0ec8bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 0x7fc0e8153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.634+0000 7fc0ec8bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0ec8bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e815b320 0x7fc0e80cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0ec8bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80c5f70 con 0x7fc0e80ad970 2023-10-14T17:57:45.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0ec8bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e81f2230 con 0x7fc0e806c040 2023-10-14T17:57:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0ec8bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0e80c61c0 con 0x7fc0e815b320 2023-10-14T17:57:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40892/0 (socket says 172.21.15.29:40892) 2023-10-14T17:57:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 -- 172.21.15.29:0/3142360713 learned_addr learned my addr 172.21.15.29:0/3142360713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:45.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e77fe640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 0x7fc0e8153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 -- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e815b320 msgr2=0x7fc0e80cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e815b320 0x7fc0e80cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 -- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 msgr2=0x7fc0e8153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 0x7fc0e8153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e6ffd640 1 -- 172.21.15.29:0/3142360713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0e0009400 con 0x7fc0e80ad970 2023-10-14T17:57:45.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.635+0000 7fc0e77fe640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 0x7fc0e8153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:45.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0e6ffd640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c400 tx=0x7fc0d000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0011790 con 0x7fc0e80ad970 2023-10-14T17:57:45.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0d0011930 con 0x7fc0e80ad970 2023-10-14T17:57:45.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0011ce0 con 0x7fc0e80ad970 2023-10-14T17:57:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0e80c64a0 con 0x7fc0e80ad970 2023-10-14T17:57:45.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.636+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0e80c6a60 con 0x7fc0e80ad970 2023-10-14T17:57:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.637+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0e806b330 con 0x7fc0e80ad970 2023-10-14T17:57:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.638+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc0d001c460 con 0x7fc0e80ad970 2023-10-14T17:57:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.639+0000 7fc0dffff640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 0x7fc0c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:45.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.639+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(177..177 src has 1..177) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc0d0089830 con 0x7fc0e80ad970 2023-10-14T17:57:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.640+0000 7fc0e77fe640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 0x7fc0c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:45.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.640+0000 7fc0e77fe640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 0x7fc0c806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8005c00 tx=0x7fc0d8009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:45.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.644+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0d00546b0 con 0x7fc0e80ad970 2023-10-14T17:57:45.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.841+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0e80cc550 con 0x7fc0e80ad970 2023-10-14T17:57:45.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.842+0000 7fc0dffff640 1 -- 172.21.15.29:0/3142360713 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v177) v1 ==== 74+0+16766 (secure 0 0 0) 0x7fc0d0057cd0 con 0x7fc0e80ad970 2023-10-14T17:57:45.844 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:45.845 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":177,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:44.366933+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 msgr2=0x7fc0c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 0x7fc0c806d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc0d8005c00 tx=0x7fc0d8009040 comp rx=0 tx=0).stop 2023-10-14T17:57:45.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 msgr2=0x7fc0e80cd230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 secure :-1 s=READY pgs=1972 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000c400 tx=0x7fc0d000c8c0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 shutdown_connections 2023-10-14T17:57:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc0c806af90 0x7fc0c806d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0e815b320 0x7fc0e80cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0e80ad970 0x7fc0e80cd230 unknown :-1 s=CLOSED pgs=1972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 --2- 172.21.15.29:0/3142360713 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0e806c040 0x7fc0e8153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 >> 172.21.15.29:0/3142360713 conn(0x7fc0e80ce6a0 msgr2=0x7fc0e815cb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:45.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 shutdown_connections 2023-10-14T17:57:45.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:45.845+0000 7fc0ec8bb640 1 -- 172.21.15.29:0/3142360713 wait complete. 2023-10-14T17:57:45.867 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T17:57:46.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.096+0000 7f5157221640 1 Processor -- start 2023-10-14T17:57:46.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.096+0000 7f5157221640 1 -- start start 2023-10-14T17:57:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.097+0000 7f5157221640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 0x7f51500b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5157221640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515015d150 con 0x7f51500b8480 2023-10-14T17:57:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5157221640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51500d1180 con 0x7f515015cd70 2023-10-14T17:57:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5157221640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d1460 con 0x7f51500cded0 2023-10-14T17:57:46.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5155a1e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f51500b8480 0x7f51500cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54970/0 (socket says 172.21.15.29:54970) 2023-10-14T17:57:46.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5155a1e640 1 -- 172.21.15.29:0/3129596079 learned_addr learned my addr 172.21.15.29:0/3129596079 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:46.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f5156a20640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 0x7f51500b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254867617 0 0) 0x7f515015d150 con 0x7f51500b8480 2023-10-14T17:57:46.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.098+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f513c003520 con 0x7f51500b8480 2023-10-14T17:57:46.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708415358 0 0) 0x7f513c003520 con 0x7f51500b8480 2023-10-14T17:57:46.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 msgr2=0x7f51500b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 0x7f51500b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 >> v1:172.21.15.186:6789/0 conn(0x7f515015cd70 legacy=0x7f51500d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51500d1740 con 0x7f51500b8480 2023-10-14T17:57:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5140001690 con 0x7f51500b8480 2023-10-14T17:57:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 >> v1:172.21.15.29:6789/0 conn(0x7f51500b8480 legacy=0x7f51500cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f513c0039a0 0x7f513c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 0x7f513c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c004d00 0x7f513c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51500d1460 con 0x7f513c004d00 2023-10-14T17:57:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f513c00bb40 con 0x7f513c0039a0 2023-10-14T17:57:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f513c00bd00 con 0x7f51500cded0 2023-10-14T17:57:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.099+0000 7f5156a20640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c004d00 0x7f513c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f5156a20640 1 -- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 msgr2=0x7f513c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f5156a20640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51500cded0 0x7f513c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f5156a20640 1 -- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f513c0039a0 msgr2=0x7f513c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:46.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f515621f640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f513c0039a0 0x7f513c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f5156a20640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f513c0039a0 0x7f513c003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.100+0000 7f5156a20640 1 -- 172.21.15.29:0/3129596079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513c003520 con 0x7f513c004d00 2023-10-14T17:57:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.101+0000 7f5156a20640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c004d00 0x7f513c005180 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f514c00ba00 tx=0x7f513c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7970db1d5ac133d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.101+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c032640 con 0x7f513c004d00 2023-10-14T17:57:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.101+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c0327a0 con 0x7f513c004d00 2023-10-14T17:57:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.101+0000 7f515521d640 1 -- 172.21.15.29:0/3129596079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c032b30 con 0x7f513c004d00 2023-10-14T17:57:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 -- 172.21.15.29:0/3129596079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c004d00 msgr2=0x7f513c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 --2- 172.21.15.29:0/3129596079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f513c004d00 0x7f513c005180 secure :-1 s=READY pgs=1974 cs=0 l=1 rev1=1 crypto rx=0x7f514c00ba00 tx=0x7f513c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5156a20640 1 -- 172.21.15.29:0/3129596079 reap_dead start 2023-10-14T17:57:46.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 -- 172.21.15.29:0/3129596079 shutdown_connections 2023-10-14T17:57:46.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 -- 172.21.15.29:0/3129596079 >> 172.21.15.29:0/3129596079 conn(0x7f51500cf570 msgr2=0x7f51500cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:46.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 -- 172.21.15.29:0/3129596079 shutdown_connections 2023-10-14T17:57:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.102+0000 7f5157221640 1 -- 172.21.15.29:0/3129596079 wait complete. 2023-10-14T17:57:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.104+0000 7f5157221640 1 Processor -- start 2023-10-14T17:57:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.104+0000 7f5157221640 1 -- start start 2023-10-14T17:57:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f513c004d00 0x7f51500cb330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 0x7f51500a97a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515014c6b0 con 0x7f51500b6d70 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f515014c990 con 0x7f51500b68f0 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.105+0000 7f5157221640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f515014cc70 con 0x7f513c004d00 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40908/0 (socket says 172.21.15.29:40908) 2023-10-14T17:57:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 -- 172.21.15.29:0/2891139401 learned_addr learned my addr 172.21.15.29:0/2891139401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f515621f640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f513c004d00 0x7f51500cb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5155a1e640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 0x7f51500a97a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 -- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f513c004d00 msgr2=0x7f51500cb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f513c004d00 0x7f51500cb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 -- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 msgr2=0x7f51500a97a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 0x7f51500a97a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 -- 172.21.15.29:0/2891139401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515014cf50 con 0x7f51500b6d70 2023-10-14T17:57:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5155a1e640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 0x7f51500a97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.106+0000 7f5156a20640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f514c02bec0 tx=0x7f514c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.107+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c004020 con 0x7f51500b6d70 2023-10-14T17:57:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.107+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c004180 con 0x7f51500b6d70 2023-10-14T17:57:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.107+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f515014d230 con 0x7f51500b6d70 2023-10-14T17:57:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.107+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f514c0044f0 con 0x7f51500b6d70 2023-10-14T17:57:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.107+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51500aa610 con 0x7f51500b6d70 2023-10-14T17:57:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.108+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51501f5800 con 0x7f51500b6d70 2023-10-14T17:57:46.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.109+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f514c03b6f0 con 0x7f51500b6d70 2023-10-14T17:57:46.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.110+0000 7f5146ffd640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 0x7f513406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:46.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.110+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f514c0aba90 con 0x7f51500b6d70 2023-10-14T17:57:46.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.111+0000 7f515621f640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 0x7f513406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:46.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.112+0000 7f515621f640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 0x7f513406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5148007fd0 tx=0x7f5148009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:46.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.115+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51501f5800 con 0x7f51500b6d70 2023-10-14T17:57:46.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.351+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f51501f5800 con 0x7f51500b6d70 2023-10-14T17:57:46.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.939+0000 7f5146ffd640 1 -- 172.21.15.29:0/2891139401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v82) v1 ==== 118+0+0 (secure 0 0 0) 0x7f51501f5800 con 0x7f51500b6d70 2023-10-14T17:57:46.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 msgr2=0x7f513406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 0x7f513406d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5148007fd0 tx=0x7f5148009040 comp rx=0 tx=0).stop 2023-10-14T17:57:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 msgr2=0x7f51500a9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 secure :-1 s=READY pgs=1975 cs=0 l=1 rev1=1 crypto rx=0x7f514c02bec0 tx=0x7f514c004730 comp rx=0 tx=0).stop 2023-10-14T17:57:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 shutdown_connections 2023-10-14T17:57:46.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f513406af90 0x7f513406d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51500b6d70 0x7f51500a9ce0 unknown :-1 s=CLOSED pgs=1975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51500b68f0 0x7f51500a97a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 --2- 172.21.15.29:0/2891139401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f513c004d00 0x7f51500cb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:46.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 >> 172.21.15.29:0/2891139401 conn(0x7f51500cf570 msgr2=0x7f515015ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 shutdown_connections 2023-10-14T17:57:46.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:46.943+0000 7f5157221640 1 -- 172.21.15.29:0/2891139401 wait complete. 2023-10-14T17:57:46.965 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T17:57:46.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T17:57:47.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.194+0000 7fb4f558e640 1 Processor -- start 2023-10-14T17:57:47.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.194+0000 7fb4f558e640 1 -- start start 2023-10-14T17:57:47.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f006c040 0x7fb4f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 0x7fb4f015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4f015ebc0 0x7fb4f015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f006c960 con 0x7fb4f015ebc0 2023-10-14T17:57:47.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4f015b9d0 con 0x7fb4f00ad970 2023-10-14T17:57:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.196+0000 7fb4f558e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f015bcb0 con 0x7fb4f006c040 2023-10-14T17:57:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4f015ebc0 0x7fb4f015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4f015ebc0 0x7fb4f015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40924/0 (socket says 172.21.15.29:40924) 2023-10-14T17:57:47.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 -- 172.21.15.29:0/3799104977 learned_addr learned my addr 172.21.15.29:0/3799104977 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4effff640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f006c040 0x7fb4f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4e77fe640 1 --2- 172.21.15.29:0/3799104977 >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 0x7fb4f015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 -- 172.21.15.29:0/3799104977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f006c040 msgr2=0x7fb4f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:47.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f006c040 0x7fb4f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 -- 172.21.15.29:0/3799104977 >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 msgr2=0x7fb4f015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 --2- 172.21.15.29:0/3799104977 >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 0x7fb4f015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4f4d8d640 1 -- 172.21.15.29:0/3799104977 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f015bf90 con 0x7fb4f015ebc0 2023-10-14T17:57:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.197+0000 7fb4e77fe640 1 --2- 172.21.15.29:0/3799104977 >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 0x7fb4f015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:47.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.198+0000 7fb4f4d8d640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f015ebc0 0x7fb4f015f050 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009870 tx=0x7fb4e802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9fe20198de5c3f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:47.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4ef7fe640 1 -- 172.21.15.29:0/3799104977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8031640 con 0x7fb4f015ebc0 2023-10-14T17:57:47.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4ef7fe640 1 -- 172.21.15.29:0/3799104977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e80317a0 con 0x7fb4f015ebc0 2023-10-14T17:57:47.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4ef7fe640 1 -- 172.21.15.29:0/3799104977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e8031b30 con 0x7fb4f015ebc0 2023-10-14T17:57:47.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 -- 172.21.15.29:0/3799104977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f015ebc0 msgr2=0x7fb4f015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:47.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f015ebc0 0x7fb4f015f050 secure :-1 s=READY pgs=1976 cs=0 l=1 rev1=1 crypto rx=0x7fb4e8009870 tx=0x7fb4e802b8f0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 -- 172.21.15.29:0/3799104977 shutdown_connections 2023-10-14T17:57:47.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f015ebc0 0x7fb4f015f050 unknown :-1 s=CLOSED pgs=1976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 --2- 172.21.15.29:0/3799104977 >> v2:172.21.15.186:3300/0 conn(0x7fb4f00ad970 0x7fb4f015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 --2- 172.21.15.29:0/3799104977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f006c040 0x7fb4f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 -- 172.21.15.29:0/3799104977 >> 172.21.15.29:0/3799104977 conn(0x7fb4f00ce6a0 msgr2=0x7fb4f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.199+0000 7fb4f558e640 1 -- 172.21.15.29:0/3799104977 shutdown_connections 2023-10-14T17:57:47.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.200+0000 7fb4f558e640 1 -- 172.21.15.29:0/3799104977 wait complete. 2023-10-14T17:57:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.201+0000 7fb4f558e640 1 Processor -- start 2023-10-14T17:57:47.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.201+0000 7fb4f558e640 1 -- start start 2023-10-14T17:57:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.202+0000 7fb4f558e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.202+0000 7fb4f558e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4f00ad970 0x7fb4f0152d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.202+0000 7fb4f558e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 0x7fb4f01535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4f558e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00ae730 con 0x7fb4f006c040 2023-10-14T17:57:47.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4f558e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00af090 con 0x7fb4f00ad970 2023-10-14T17:57:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4f558e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4f00ae100 con 0x7fb4f015ebc0 2023-10-14T17:57:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40934/0 (socket says 172.21.15.29:40934) 2023-10-14T17:57:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 -- 172.21.15.29:0/234406739 learned_addr learned my addr 172.21.15.29:0/234406739 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:47.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4f4d8d640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 0x7fb4f01535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 -- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 msgr2=0x7fb4f01535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4e77fe640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4f00ad970 0x7fb4f0152d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 0x7fb4f01535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 -- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4f00ad970 msgr2=0x7fb4f0152d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:47.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4f00ad970 0x7fb4f0152d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4effff640 1 -- 172.21.15.29:0/234406739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4e80094d0 con 0x7fb4f006c040 2023-10-14T17:57:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.203+0000 7fb4f4d8d640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 0x7fb4f01535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4effff640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000d830 tx=0x7fb4e000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e000be10 con 0x7fb4f006c040 2023-10-14T17:57:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e00098b0 con 0x7fb4f006c040 2023-10-14T17:57:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4f00c72a0 con 0x7fb4f006c040 2023-10-14T17:57:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4e0009a40 con 0x7fb4f006c040 2023-10-14T17:57:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.204+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4f00ae350 con 0x7fb4f006c040 2023-10-14T17:57:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.206+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4f00ae9a0 con 0x7fb4f006c040 2023-10-14T17:57:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.207+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb4e0009ba0 con 0x7fb4f006c040 2023-10-14T17:57:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.207+0000 7fb4ed7fa640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 0x7fb4d006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.208+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb4e0089c10 con 0x7fb4f006c040 2023-10-14T17:57:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.208+0000 7fb4e77fe640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 0x7fb4d006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.209+0000 7fb4e77fe640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 0x7fb4d006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4f006ab50 tx=0x7fb4d8008450 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.212+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4e0054b20 con 0x7fb4f006c040 2023-10-14T17:57:47.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.449+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fb4f006b330 con 0x7fb4f006c040 2023-10-14T17:57:47.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.948+0000 7fb4ed7fa640 1 -- 172.21.15.29:0/234406739 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v83) v1 ==== 122+0+0 (secure 0 0 0) 0x7fb4e000b3a0 con 0x7fb4f006c040 2023-10-14T17:57:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 msgr2=0x7fb4d006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:47.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 0x7fb4d006d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb4f006ab50 tx=0x7fb4d8008450 comp rx=0 tx=0).stop 2023-10-14T17:57:47.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 msgr2=0x7fb4f00aeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:47.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 secure :-1 s=READY pgs=1977 cs=0 l=1 rev1=1 crypto rx=0x7fb4e000d830 tx=0x7fb4e000dcf0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 shutdown_connections 2023-10-14T17:57:47.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb4d006af90 0x7fb4d006d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4f015ebc0 0x7fb4f01535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4f00ad970 0x7fb4f0152d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 --2- 172.21.15.29:0/234406739 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4f006c040 0x7fb4f00aeb50 unknown :-1 s=CLOSED pgs=1977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:47.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 >> 172.21.15.29:0/234406739 conn(0x7fb4f00ce6a0 msgr2=0x7fb4f00ca720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:47.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 shutdown_connections 2023-10-14T17:57:47.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:47.952+0000 7fb4f558e640 1 -- 172.21.15.29:0/234406739 wait complete. 2023-10-14T17:57:47.973 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T17:57:47.974 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:48.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.202+0000 7faa0c578640 1 Processor -- start 2023-10-14T17:57:48.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.202+0000 7faa0c578640 1 -- start start 2023-10-14T17:57:48.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0c578640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa04159b20 0x7faa0406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0c578640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa04159f00 0x7faa040b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0c578640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa0406f7f0 con 0x7faa0406c120 2023-10-14T17:57:48.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0c578640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0406c5e0 con 0x7faa04159f00 2023-10-14T17:57:48.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0c578640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa0406c8c0 con 0x7faa04159b20 2023-10-14T17:57:48.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa09b13640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa04159f00 0x7faa040b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0ab15640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa0406c120 0x7faa040cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54984/0 (socket says 172.21.15.29:54984) 2023-10-14T17:57:48.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.204+0000 7faa0ab15640 1 -- 172.21.15.29:0/3899288139 learned_addr learned my addr 172.21.15.29:0/3899288139 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:48.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.205+0000 7faa0a314640 1 --2- 172.21.15.29:0/3899288139 >> v2:172.21.15.186:3300/0 conn(0x7faa04159b20 0x7faa0406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.205+0000 7faa09b13640 1 -- 172.21.15.29:0/3899288139 >> v2:172.21.15.186:3300/0 conn(0x7faa04159b20 msgr2=0x7faa0406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:48.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.205+0000 7faa09b13640 1 --2- 172.21.15.29:0/3899288139 >> v2:172.21.15.186:3300/0 conn(0x7faa04159b20 0x7faa0406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.205+0000 7faa09b13640 1 -- 172.21.15.29:0/3899288139 >> v1:172.21.15.29:6789/0 conn(0x7faa0406c120 legacy=0x7faa040cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.205+0000 7faa09b13640 1 -- 172.21.15.29:0/3899288139 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa0406cba0 con 0x7faa04159f00 2023-10-14T17:57:48.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.206+0000 7faa09b13640 1 --2- 172.21.15.29:0/3899288139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04159f00 0x7faa040b0410 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7faa040c9340 tx=0x7fa9f402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938d2f23e5f16f4d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.206+0000 7faa09312640 1 -- 172.21.15.29:0/3899288139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4031830 con 0x7faa04159f00 2023-10-14T17:57:48.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.206+0000 7faa09312640 1 -- 172.21.15.29:0/3899288139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9f4031990 con 0x7faa04159f00 2023-10-14T17:57:48.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.206+0000 7faa09312640 1 -- 172.21.15.29:0/3899288139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9f4031d20 con 0x7faa04159f00 2023-10-14T17:57:48.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 -- 172.21.15.29:0/3899288139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04159f00 msgr2=0x7faa040b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 --2- 172.21.15.29:0/3899288139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04159f00 0x7faa040b0410 secure :-1 s=READY pgs=1979 cs=0 l=1 rev1=1 crypto rx=0x7faa040c9340 tx=0x7fa9f402b8d0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 -- 172.21.15.29:0/3899288139 shutdown_connections 2023-10-14T17:57:48.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 --2- 172.21.15.29:0/3899288139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa04159f00 0x7faa040b0410 unknown :-1 s=CLOSED pgs=1979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 --2- 172.21.15.29:0/3899288139 >> v2:172.21.15.186:3300/0 conn(0x7faa04159b20 0x7faa0406bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 -- 172.21.15.29:0/3899288139 >> 172.21.15.29:0/3899288139 conn(0x7faa040cb410 msgr2=0x7faa040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:48.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 -- 172.21.15.29:0/3899288139 shutdown_connections 2023-10-14T17:57:48.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.207+0000 7faa0c578640 1 -- 172.21.15.29:0/3899288139 wait complete. 2023-10-14T17:57:48.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.209+0000 7faa0c578640 1 Processor -- start 2023-10-14T17:57:48.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.209+0000 7faa0c578640 1 -- start start 2023-10-14T17:57:48.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa04159b20 0x7faa041585b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa04159f00 0x7faa04154d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa040c2a00 con 0x7faa0406c120 2023-10-14T17:57:48.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa040c2d60 con 0x7faa04159f00 2023-10-14T17:57:48.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.210+0000 7faa0c578640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa040c3040 con 0x7faa04159b20 2023-10-14T17:57:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40952/0 (socket says 172.21.15.29:40952) 2023-10-14T17:57:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 -- 172.21.15.29:0/716294334 learned_addr learned my addr 172.21.15.29:0/716294334 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:48.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 -- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa04159b20 msgr2=0x7faa041585b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa04159b20 0x7faa041585b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 -- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa04159f00 msgr2=0x7faa04154d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa04159f00 0x7faa04154d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 -- 172.21.15.29:0/716294334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9f4009400 con 0x7faa0406c120 2023-10-14T17:57:48.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.211+0000 7faa0a314640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7faa00007ef0 tx=0x7faa0000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.212+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa00014620 con 0x7faa0406c120 2023-10-14T17:57:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.212+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa00014780 con 0x7faa0406c120 2023-10-14T17:57:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.212+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa00014b30 con 0x7faa0406c120 2023-10-14T17:57:48.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.212+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa040c3320 con 0x7faa0406c120 2023-10-14T17:57:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.212+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa040c6a50 con 0x7faa0406c120 2023-10-14T17:57:48.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.213+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa0406ce60 con 0x7faa0406c120 2023-10-14T17:57:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.214+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faa00014c90 con 0x7faa0406c120 2023-10-14T17:57:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.215+0000 7fa9faffd640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.215+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faa0008a2f0 con 0x7faa0406c120 2023-10-14T17:57:48.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.215+0000 7faa09b13640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.216+0000 7faa09b13640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 0x7fa9e806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4036000 tx=0x7fa9f4030000 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.219+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa0406ce60 con 0x7faa0406c120 2023-10-14T17:57:48.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.454+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa0406ce60 con 0x7faa0406c120 2023-10-14T17:57:48.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.456+0000 7fa9faffd640 1 -- 172.21.15.29:0/716294334 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7faa0406ce60 con 0x7faa0406c120 2023-10-14T17:57:48.457 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:48.458 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:57:48.458 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:57:48.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.458+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 msgr2=0x7fa9e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.458+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 0x7fa9e806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fa9f4036000 tx=0x7fa9f4030000 comp rx=0 tx=0).stop 2023-10-14T17:57:48.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 msgr2=0x7faa040c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 secure :-1 s=READY pgs=1980 cs=0 l=1 rev1=1 crypto rx=0x7faa00007ef0 tx=0x7faa0000e4b0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 shutdown_connections 2023-10-14T17:57:48.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fa9e806af90 0x7fa9e806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa04159f00 0x7faa04154d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa04159b20 0x7faa041585b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 --2- 172.21.15.29:0/716294334 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa0406c120 0x7faa040c24c0 unknown :-1 s=CLOSED pgs=1980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 >> 172.21.15.29:0/716294334 conn(0x7faa040cb410 msgr2=0x7faa040ad8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:48.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 shutdown_connections 2023-10-14T17:57:48.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.459+0000 7faa0c578640 1 -- 172.21.15.29:0/716294334 wait complete. 2023-10-14T17:57:48.477 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:57:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.689+0000 7febf759e640 1 Processor -- start 2023-10-14T17:57:48.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.689+0000 7febf759e640 1 -- start start 2023-10-14T17:57:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf815b9d0 con 0x7febf806c040 2023-10-14T17:57:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf815bcb0 con 0x7febf80ccfb0 2023-10-14T17:57:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf815bf90 con 0x7febf80ad970 2023-10-14T17:57:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7febf806c040 0x7febf806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54996/0 (socket says 172.21.15.29:54996) 2023-10-14T17:57:48.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.690+0000 7febf659c640 1 -- 172.21.15.29:0/2080898087 learned_addr learned my addr 172.21.15.29:0/2080898087 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:48.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf5d9b640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf6d9d640 1 --2- 172.21.15.29:0/2080898087 >> v2:172.21.15.186:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930638449 0 0) 0x7febf815b9d0 con 0x7febf806c040 2023-10-14T17:57:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0003520 con 0x7febf806c040 2023-10-14T17:57:48.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1884460867 0 0) 0x7febe0003520 con 0x7febf806c040 2023-10-14T17:57:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 msgr2=0x7febf80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febf80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 >> v2:172.21.15.186:3300/0 conn(0x7febf80ccfb0 msgr2=0x7febf80cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 --2- 172.21.15.29:0/2080898087 >> v2:172.21.15.186:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf80cc270 con 0x7febf806c040 2023-10-14T17:57:48.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7febe8001670 con 0x7febf806c040 2023-10-14T17:57:48.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf6d9d640 1 --2- 172.21.15.29:0/2080898087 >> v2:172.21.15.186:3300/0 conn(0x7febf80ccfb0 0x7febf80cd440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:48.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.691+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 >> v1:172.21.15.29:6789/0 conn(0x7febf806c040 legacy=0x7febf806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0003970 0x7febe0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febe00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0004d00 0x7febe0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf815bcb0 con 0x7febe0003970 2023-10-14T17:57:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf815bf90 con 0x7febe0004d00 2023-10-14T17:57:48.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe000b9d0 con 0x7febf80ad970 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0003970 0x7febe0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf6d9d640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0004d00 0x7febe0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 -- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 msgr2=0x7febe00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf5d9b640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febe00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80ad970 0x7febe00047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 -- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0004d00 msgr2=0x7febe0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0004d00 0x7febe0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf659c640 1 -- 172.21.15.29:0/2080898087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0003520 con 0x7febe0003970 2023-10-14T17:57:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.692+0000 7febf6d9d640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0004d00 0x7febe0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.693+0000 7febf659c640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0003970 0x7febe0003d90 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7febe800c240 tx=0x7febe00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afd463de028239b8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8005ad0 con 0x7febe0003970 2023-10-14T17:57:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8005c30 con 0x7febe0003970 2023-10-14T17:57:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf559a640 1 -- 172.21.15.29:0/2080898087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe802c8b0 con 0x7febe0003970 2023-10-14T17:57:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf759e640 1 -- 172.21.15.29:0/2080898087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0003970 msgr2=0x7febe0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf759e640 1 --2- 172.21.15.29:0/2080898087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febe0003970 0x7febe0003d90 secure :-1 s=READY pgs=1982 cs=0 l=1 rev1=1 crypto rx=0x7febe800c240 tx=0x7febe00057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf6d9d640 1 -- 172.21.15.29:0/2080898087 reap_dead start 2023-10-14T17:57:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf759e640 1 -- 172.21.15.29:0/2080898087 shutdown_connections 2023-10-14T17:57:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf759e640 1 -- 172.21.15.29:0/2080898087 >> 172.21.15.29:0/2080898087 conn(0x7febf80ce6a0 msgr2=0x7febf80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.694+0000 7febf759e640 1 -- 172.21.15.29:0/2080898087 shutdown_connections 2023-10-14T17:57:48.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.695+0000 7febf759e640 1 -- 172.21.15.29:0/2080898087 wait complete. 2023-10-14T17:57:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.696+0000 7febf759e640 1 Processor -- start 2023-10-14T17:57:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.697+0000 7febf759e640 1 -- start start 2023-10-14T17:57:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.697+0000 7febf759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 0x7febf8156fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.697+0000 7febf759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80c9460 0x7febf80c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.697+0000 7febf759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80c7640 con 0x7febf80c6910 2023-10-14T17:57:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf80c9e20 con 0x7febe0003970 2023-10-14T17:57:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf81f32a0 con 0x7febf80c9460 2023-10-14T17:57:48.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40976/0 (socket says 172.21.15.29:40976) 2023-10-14T17:57:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 -- 172.21.15.29:0/367229883 learned_addr learned my addr 172.21.15.29:0/367229883 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf659c640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 0x7febf8156fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf5d9b640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80c9460 0x7febf80c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 -- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80c9460 msgr2=0x7febf80c98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80c9460 0x7febf80c98e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 -- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 msgr2=0x7febf8156fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 0x7febf8156fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf6d9d640 1 -- 172.21.15.29:0/367229883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf81f30f0 con 0x7febf80c6910 2023-10-14T17:57:48.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.698+0000 7febf659c640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 0x7febf8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febf6d9d640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7febf80cd390 tx=0x7febf000f8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0004230 con 0x7febf80c6910 2023-10-14T17:57:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf000c070 con 0x7febf80c6910 2023-10-14T17:57:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febf0003610 con 0x7febf80c6910 2023-10-14T17:57:48.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf81f20e0 con 0x7febf80c6910 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.699+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf81f2330 con 0x7febf80c6910 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.700+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf804e9b0 con 0x7febf80c6910 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.701+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febf0003770 con 0x7febf80c6910 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.703+0000 7febe6ffd640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 0x7febd806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.703+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7febf0089540 con 0x7febf80c6910 2023-10-14T17:57:48.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.703+0000 7febf659c640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 0x7febd806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.704+0000 7febf659c640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 0x7febd806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febe8034000 tx=0x7febe8009030 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:48.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.707+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febf0054350 con 0x7febf80c6910 2023-10-14T17:57:48.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.895+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febf806b2b0 con 0x7febf80c6910 2023-10-14T17:57:48.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.898+0000 7febe6ffd640 1 -- 172.21.15.29:0/367229883 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v178) v1 ==== 74+0+16753 (secure 0 0 0) 0x7febf0057970 con 0x7febf80c6910 2023-10-14T17:57:48.900 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:48.900 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":178,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:45.926253+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:57:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 msgr2=0x7febd806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 0x7febd806d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7febe8034000 tx=0x7febe8009030 comp rx=0 tx=0).stop 2023-10-14T17:57:48.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 msgr2=0x7febf80c6d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:48.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 secure :-1 s=READY pgs=1983 cs=0 l=1 rev1=1 crypto rx=0x7febf80cd390 tx=0x7febf000f8a0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 shutdown_connections 2023-10-14T17:57:48.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7febd806afc0 0x7febd806d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf80c6910 0x7febf80c6d90 unknown :-1 s=CLOSED pgs=1983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf80c9460 0x7febf80c98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 --2- 172.21.15.29:0/367229883 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febe0003970 0x7febf8156fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 >> 172.21.15.29:0/367229883 conn(0x7febf80ce6a0 msgr2=0x7febf80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 shutdown_connections 2023-10-14T17:57:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:48.901+0000 7febf759e640 1 -- 172.21.15.29:0/367229883 wait complete. 2023-10-14T17:57:48.922 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T17:57:48.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:57:49.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.142+0000 7fd35f59e640 1 Processor -- start 2023-10-14T17:57:49.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.142+0000 7fd35f59e640 1 -- start start 2023-10-14T17:57:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd36006c040 0x7fd36006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd36015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36006c960 con 0x7fd3600ad970 2023-10-14T17:57:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36015b9d0 con 0x7fd36006c040 2023-10-14T17:57:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd36015bcb0 con 0x7fd3600cda80 2023-10-14T17:57:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd3600ad970 0x7fd36015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54998/0 (socket says 172.21.15.29:54998) 2023-10-14T17:57:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.144+0000 7fd35dd9b640 1 -- 172.21.15.29:0/3194255783 learned_addr learned my addr 172.21.15.29:0/3194255783 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:49.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35e59c640 1 --2- 172.21.15.29:0/3194255783 >> v2:172.21.15.186:3300/0 conn(0x7fd36006c040 0x7fd36006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd36015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966280523 0 0) 0x7fd36006c960 con 0x7fd3600ad970 2023-10-14T17:57:49.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd348003520 con 0x7fd3600ad970 2023-10-14T17:57:49.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1085837321 0 0) 0x7fd348003520 con 0x7fd3600ad970 2023-10-14T17:57:49.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 msgr2=0x7fd36015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:49.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd36015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 >> v2:172.21.15.186:3300/0 conn(0x7fd36006c040 msgr2=0x7fd36006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 --2- 172.21.15.29:0/3194255783 >> v2:172.21.15.186:3300/0 conn(0x7fd36006c040 0x7fd36006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36015bf90 con 0x7fd3600ad970 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd354001670 con 0x7fd3600ad970 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 >> v1:172.21.15.29:6789/0 conn(0x7fd3600ad970 legacy=0x7fd36015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd348003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348003ef0 0x7fd3480047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd348004d00 0x7fd348005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36015b9d0 con 0x7fd348004d00 2023-10-14T17:57:49.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd36015bcb0 con 0x7fd348003ef0 2023-10-14T17:57:49.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.145+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34800b980 con 0x7fd3600cda80 2023-10-14T17:57:49.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd348004d00 0x7fd348005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35dd9b640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348003ef0 0x7fd3480047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35e59c640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd348003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 -- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 msgr2=0x7fd348003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:49.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3600cda80 0x7fd348003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 -- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348003ef0 msgr2=0x7fd3480047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348003ef0 0x7fd3480047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.146+0000 7fd35ed9d640 1 -- 172.21.15.29:0/3194255783 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd348003520 con 0x7fd348004d00 2023-10-14T17:57:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.147+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd348004d00 0x7fd348005180 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd358009fc0 tx=0x7fd3480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e5a4d00db86b36 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.147+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd35802cc00 con 0x7fd348004d00 2023-10-14T17:57:49.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.147+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd35800f040 con 0x7fd348004d00 2023-10-14T17:57:49.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.147+0000 7fd35d59a640 1 -- 172.21.15.29:0/3194255783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358034440 con 0x7fd348004d00 2023-10-14T17:57:49.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 -- 172.21.15.29:0/3194255783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd348004d00 msgr2=0x7fd348005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 --2- 172.21.15.29:0/3194255783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd348004d00 0x7fd348005180 secure :-1 s=READY pgs=1985 cs=0 l=1 rev1=1 crypto rx=0x7fd358009fc0 tx=0x7fd3480057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35ed9d640 1 -- 172.21.15.29:0/3194255783 reap_dead start 2023-10-14T17:57:49.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 -- 172.21.15.29:0/3194255783 shutdown_connections 2023-10-14T17:57:49.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 -- 172.21.15.29:0/3194255783 >> 172.21.15.29:0/3194255783 conn(0x7fd3600ce6a0 msgr2=0x7fd3600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:49.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 -- 172.21.15.29:0/3194255783 shutdown_connections 2023-10-14T17:57:49.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.148+0000 7fd35f59e640 1 -- 172.21.15.29:0/3194255783 wait complete. 2023-10-14T17:57:49.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.150+0000 7fd35f59e640 1 Processor -- start 2023-10-14T17:57:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.150+0000 7fd35f59e640 1 -- start start 2023-10-14T17:57:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348004d00 0x7fd3600acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd36014f870 0x7fd36014fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600c8310 con 0x7fd3600c3010 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600c72b0 con 0x7fd348004d00 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd360150230 con 0x7fd36014f870 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40986/0 (socket says 172.21.15.29:40986) 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35ed9d640 1 -- 172.21.15.29:0/90667976 learned_addr learned my addr 172.21.15.29:0/90667976 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:49.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.151+0000 7fd35e59c640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348004d00 0x7fd3600acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35dd9b640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd36014f870 0x7fd36014fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 -- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd36014f870 msgr2=0x7fd36014fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd36014f870 0x7fd36014fcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 -- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348004d00 msgr2=0x7fd3600acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348004d00 0x7fd3600acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 -- 172.21.15.29:0/90667976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3600c7550 con 0x7fd3600c3010 2023-10-14T17:57:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35ed9d640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fd358037040 tx=0x7fd358009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358034ad0 con 0x7fd3600c3010 2023-10-14T17:57:49.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd358034c30 con 0x7fd3600c3010 2023-10-14T17:57:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3600c7830 con 0x7fd3600c3010 2023-10-14T17:57:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd358003990 con 0x7fd3600c3010 2023-10-14T17:57:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.152+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3600c7df0 con 0x7fd3600c3010 2023-10-14T17:57:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.154+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3600c8010 con 0x7fd3600c3010 2023-10-14T17:57:49.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.155+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd358047460 con 0x7fd3600c3010 2023-10-14T17:57:49.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.156+0000 7fd34effd640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 0x7fd34006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.156+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd3580ab5f0 con 0x7fd3600c3010 2023-10-14T17:57:49.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.156+0000 7fd35e59c640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 0x7fd34006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.157+0000 7fd35e59c640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 0x7fd34006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd350009a90 tx=0x7fd350008040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.161+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd3580765a0 con 0x7fd3600c3010 2023-10-14T17:57:49.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.350+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 --> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd3600cc550 con 0x7fd34006afc0 2023-10-14T17:57:49.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.381+0000 7fd34effd640 1 -- 172.21.15.29:0/90667976 <== mgr.7201 v2:172.21.15.29:6832/2441704285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286944 (secure 0 0 0) 0x7fd350013a20 con 0x7fd34006afc0 2023-10-14T17:57:49.388 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:57:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 msgr2=0x7fd34006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 0x7fd34006d480 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fd350009a90 tx=0x7fd350008040 comp rx=0 tx=0).stop 2023-10-14T17:57:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 msgr2=0x7fd3600c3490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 secure :-1 s=READY pgs=1986 cs=0 l=1 rev1=1 crypto rx=0x7fd358037040 tx=0x7fd358009750 comp rx=0 tx=0).stop 2023-10-14T17:57:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 shutdown_connections 2023-10-14T17:57:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fd34006afc0 0x7fd34006d480 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3600c3010 0x7fd3600c3490 unknown :-1 s=CLOSED pgs=1986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd36014f870 0x7fd36014fcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 --2- 172.21.15.29:0/90667976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd348004d00 0x7fd3600acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 >> 172.21.15.29:0/90667976 conn(0x7fd3600ce6a0 msgr2=0x7fd3600c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 shutdown_connections 2023-10-14T17:57:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.387+0000 7fd35f59e640 1 -- 172.21.15.29:0/90667976 wait complete. 2023-10-14T17:57:49.408 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T17:57:48.305696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397108,"kb_used_data":18568,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754577612,"statfs":{"total":773094113280,"available":772687474688,"internally_reserved":0,"allocated":19013632,"data_stored":9896820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.432448"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939922+0000","last_change":"2023-10-14T17:57:44.935583+0000","last_active":"2023-10-14T17:57:45.939922+0000","last_peered":"2023-10-14T17:57:45.939922+0000","last_clean":"2023-10-14T17:57:45.939922+0000","last_became_active":"2023-10-14T17:57:44.934176+0000","last_became_peered":"2023-10-14T17:57:44.934176+0000","last_unstale":"2023-10-14T17:57:45.939922+0000","last_undegraded":"2023-10-14T17:57:45.939922+0000","last_fullsized":"2023-10-14T17:57:45.939922+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:29.105298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095805+0000","last_change":"2023-10-14T17:57:44.944153+0000","last_active":"2023-10-14T17:57:46.095805+0000","last_peered":"2023-10-14T17:57:46.095805+0000","last_clean":"2023-10-14T17:57:46.095805+0000","last_became_active":"2023-10-14T17:57:44.943851+0000","last_became_peered":"2023-10-14T17:57:44.943851+0000","last_unstale":"2023-10-14T17:57:46.095805+0000","last_undegraded":"2023-10-14T17:57:46.095805+0000","last_fullsized":"2023-10-14T17:57:46.095805+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:39.388384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943878+0000","last_change":"2023-10-14T17:57:44.941256+0000","last_active":"2023-10-14T17:57:45.943878+0000","last_peered":"2023-10-14T17:57:45.943878+0000","last_clean":"2023-10-14T17:57:45.943878+0000","last_became_active":"2023-10-14T17:57:44.940968+0000","last_became_peered":"2023-10-14T17:57:44.940968+0000","last_unstale":"2023-10-14T17:57:45.943878+0000","last_undegraded":"2023-10-14T17:57:45.943878+0000","last_fullsized":"2023-10-14T17:57:45.943878+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:56:57.242181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944594+0000","last_change":"2023-10-14T17:57:44.944893+0000","last_active":"2023-10-14T17:57:45.944594+0000","last_peered":"2023-10-14T17:57:45.944594+0000","last_clean":"2023-10-14T17:57:45.944594+0000","last_became_active":"2023-10-14T17:57:44.943900+0000","last_became_peered":"2023-10-14T17:57:44.943900+0000","last_unstale":"2023-10-14T17:57:45.944594+0000","last_undegraded":"2023-10-14T17:57:45.944594+0000","last_fullsized":"2023-10-14T17:57:45.944594+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:29.105298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097453+0000","last_change":"2023-10-14T17:57:44.940092+0000","last_active":"2023-10-14T17:57:46.097453+0000","last_peered":"2023-10-14T17:57:46.097453+0000","last_clean":"2023-10-14T17:57:46.097453+0000","last_became_active":"2023-10-14T17:57:44.939718+0000","last_became_peered":"2023-10-14T17:57:44.939718+0000","last_unstale":"2023-10-14T17:57:46.097453+0000","last_undegraded":"2023-10-14T17:57:46.097453+0000","last_fullsized":"2023-10-14T17:57:46.097453+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:42.280083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939017+0000","last_change":"2023-10-14T17:57:44.947077+0000","last_active":"2023-10-14T17:57:45.939017+0000","last_peered":"2023-10-14T17:57:45.939017+0000","last_clean":"2023-10-14T17:57:45.939017+0000","last_became_active":"2023-10-14T17:57:44.946047+0000","last_became_peered":"2023-10-14T17:57:44.946047+0000","last_unstale":"2023-10-14T17:57:45.939017+0000","last_undegraded":"2023-10-14T17:57:45.939017+0000","last_fullsized":"2023-10-14T17:57:45.939017+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:56.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096290+0000","last_change":"2023-10-14T17:57:44.947365+0000","last_active":"2023-10-14T17:57:46.096290+0000","last_peered":"2023-10-14T17:57:46.096290+0000","last_clean":"2023-10-14T17:57:46.096290+0000","last_became_active":"2023-10-14T17:57:44.946888+0000","last_became_peered":"2023-10-14T17:57:44.946888+0000","last_unstale":"2023-10-14T17:57:46.096290+0000","last_undegraded":"2023-10-14T17:57:46.096290+0000","last_fullsized":"2023-10-14T17:57:46.096290+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:17.136325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938864+0000","last_change":"2023-10-14T17:57:44.941285+0000","last_active":"2023-10-14T17:57:45.938864+0000","last_peered":"2023-10-14T17:57:45.938864+0000","last_clean":"2023-10-14T17:57:45.938864+0000","last_became_active":"2023-10-14T17:57:44.940880+0000","last_became_peered":"2023-10-14T17:57:44.940880+0000","last_unstale":"2023-10-14T17:57:45.938864+0000","last_undegraded":"2023-10-14T17:57:45.938864+0000","last_fullsized":"2023-10-14T17:57:45.938864+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:36:22.091999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096320+0000","last_change":"2023-10-14T17:57:44.955693+0000","last_active":"2023-10-14T17:57:46.096320+0000","last_peered":"2023-10-14T17:57:46.096320+0000","last_clean":"2023-10-14T17:57:46.096320+0000","last_became_active":"2023-10-14T17:57:44.955386+0000","last_became_peered":"2023-10-14T17:57:44.955386+0000","last_unstale":"2023-10-14T17:57:46.096320+0000","last_undegraded":"2023-10-14T17:57:46.096320+0000","last_fullsized":"2023-10-14T17:57:46.096320+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:55:57.885140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100750+0000","last_change":"2023-10-14T17:57:44.934667+0000","last_active":"2023-10-14T17:57:46.100750+0000","last_peered":"2023-10-14T17:57:46.100750+0000","last_clean":"2023-10-14T17:57:46.100750+0000","last_became_active":"2023-10-14T17:57:44.934387+0000","last_became_peered":"2023-10-14T17:57:44.934387+0000","last_unstale":"2023-10-14T17:57:46.100750+0000","last_undegraded":"2023-10-14T17:57:46.100750+0000","last_fullsized":"2023-10-14T17:57:46.100750+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:45.389913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939250+0000","last_change":"2023-10-14T17:57:44.949499+0000","last_active":"2023-10-14T17:57:45.939250+0000","last_peered":"2023-10-14T17:57:45.939250+0000","last_clean":"2023-10-14T17:57:45.939250+0000","last_became_active":"2023-10-14T17:57:44.945736+0000","last_became_peered":"2023-10-14T17:57:44.945736+0000","last_unstale":"2023-10-14T17:57:45.939250+0000","last_undegraded":"2023-10-14T17:57:45.939250+0000","last_fullsized":"2023-10-14T17:57:45.939250+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:44:21.658770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100250+0000","last_change":"2023-10-14T17:57:44.933161+0000","last_active":"2023-10-14T17:57:46.100250+0000","last_peered":"2023-10-14T17:57:46.100250+0000","last_clean":"2023-10-14T17:57:46.100250+0000","last_became_active":"2023-10-14T17:57:44.932891+0000","last_became_peered":"2023-10-14T17:57:44.932891+0000","last_unstale":"2023-10-14T17:57:46.100250+0000","last_undegraded":"2023-10-14T17:57:46.100250+0000","last_fullsized":"2023-10-14T17:57:46.100250+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:58:43.303871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098172+0000","last_change":"2023-10-14T17:57:44.937565+0000","last_active":"2023-10-14T17:57:46.098172+0000","last_peered":"2023-10-14T17:57:46.098172+0000","last_clean":"2023-10-14T17:57:46.098172+0000","last_became_active":"2023-10-14T17:57:44.935756+0000","last_became_peered":"2023-10-14T17:57:44.935756+0000","last_unstale":"2023-10-14T17:57:46.098172+0000","last_undegraded":"2023-10-14T17:57:46.098172+0000","last_fullsized":"2023-10-14T17:57:46.098172+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:36.393884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941983+0000","last_change":"2023-10-14T17:57:44.942046+0000","last_active":"2023-10-14T17:57:45.941983+0000","last_peered":"2023-10-14T17:57:45.941983+0000","last_clean":"2023-10-14T17:57:45.941983+0000","last_became_active":"2023-10-14T17:57:44.938232+0000","last_became_peered":"2023-10-14T17:57:44.938232+0000","last_unstale":"2023-10-14T17:57:45.941983+0000","last_undegraded":"2023-10-14T17:57:45.941983+0000","last_fullsized":"2023-10-14T17:57:45.941983+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:23.748401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098555+0000","last_change":"2023-10-14T17:57:44.939570+0000","last_active":"2023-10-14T17:57:46.098555+0000","last_peered":"2023-10-14T17:57:46.098555+0000","last_clean":"2023-10-14T17:57:46.098555+0000","last_became_active":"2023-10-14T17:57:44.939289+0000","last_became_peered":"2023-10-14T17:57:44.939289+0000","last_unstale":"2023-10-14T17:57:46.098555+0000","last_undegraded":"2023-10-14T17:57:46.098555+0000","last_fullsized":"2023-10-14T17:57:46.098555+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:14.103101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095924+0000","last_change":"2023-10-14T17:57:44.938980+0000","last_active":"2023-10-14T17:57:46.095924+0000","last_peered":"2023-10-14T17:57:46.095924+0000","last_clean":"2023-10-14T17:57:46.095924+0000","last_became_active":"2023-10-14T17:57:44.938633+0000","last_became_peered":"2023-10-14T17:57:44.938633+0000","last_unstale":"2023-10-14T17:57:46.095924+0000","last_undegraded":"2023-10-14T17:57:46.095924+0000","last_fullsized":"2023-10-14T17:57:46.095924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:31:38.656878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101234+0000","last_change":"2023-10-14T17:57:44.938646+0000","last_active":"2023-10-14T17:57:46.101234+0000","last_peered":"2023-10-14T17:57:46.101234+0000","last_clean":"2023-10-14T17:57:46.101234+0000","last_became_active":"2023-10-14T17:57:44.938172+0000","last_became_peered":"2023-10-14T17:57:44.938172+0000","last_unstale":"2023-10-14T17:57:46.101234+0000","last_undegraded":"2023-10-14T17:57:46.101234+0000","last_fullsized":"2023-10-14T17:57:46.101234+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:12.657175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944125+0000","last_change":"2023-10-14T17:57:44.948395+0000","last_active":"2023-10-14T17:57:45.944125+0000","last_peered":"2023-10-14T17:57:45.944125+0000","last_clean":"2023-10-14T17:57:45.944125+0000","last_became_active":"2023-10-14T17:57:44.948095+0000","last_became_peered":"2023-10-14T17:57:44.948095+0000","last_unstale":"2023-10-14T17:57:45.944125+0000","last_undegraded":"2023-10-14T17:57:45.944125+0000","last_fullsized":"2023-10-14T17:57:45.944125+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:17.136325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939642+0000","last_change":"2023-10-14T17:57:44.940533+0000","last_active":"2023-10-14T17:57:45.939642+0000","last_peered":"2023-10-14T17:57:45.939642+0000","last_clean":"2023-10-14T17:57:45.939642+0000","last_became_active":"2023-10-14T17:57:44.940140+0000","last_became_peered":"2023-10-14T17:57:44.940140+0000","last_unstale":"2023-10-14T17:57:45.939642+0000","last_undegraded":"2023-10-14T17:57:45.939642+0000","last_fullsized":"2023-10-14T17:57:45.939642+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:58:49.126002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096604+0000","last_change":"2023-10-14T17:57:44.949412+0000","last_active":"2023-10-14T17:57:46.096604+0000","last_peered":"2023-10-14T17:57:46.096604+0000","last_clean":"2023-10-14T17:57:46.096604+0000","last_became_active":"2023-10-14T17:57:44.949129+0000","last_became_peered":"2023-10-14T17:57:44.949129+0000","last_unstale":"2023-10-14T17:57:46.096604+0000","last_undegraded":"2023-10-14T17:57:46.096604+0000","last_fullsized":"2023-10-14T17:57:46.096604+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:16:54.442674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100102+0000","last_change":"2023-10-14T17:57:44.946438+0000","last_active":"2023-10-14T17:57:46.100102+0000","last_peered":"2023-10-14T17:57:46.100102+0000","last_clean":"2023-10-14T17:57:46.100102+0000","last_became_active":"2023-10-14T17:57:44.946026+0000","last_became_peered":"2023-10-14T17:57:44.946026+0000","last_unstale":"2023-10-14T17:57:46.100102+0000","last_undegraded":"2023-10-14T17:57:46.100102+0000","last_fullsized":"2023-10-14T17:57:46.100102+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:45.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942336+0000","last_change":"2023-10-14T17:57:44.944935+0000","last_active":"2023-10-14T17:57:45.942336+0000","last_peered":"2023-10-14T17:57:45.942336+0000","last_clean":"2023-10-14T17:57:45.942336+0000","last_became_active":"2023-10-14T17:57:44.944668+0000","last_became_peered":"2023-10-14T17:57:44.944668+0000","last_unstale":"2023-10-14T17:57:45.942336+0000","last_undegraded":"2023-10-14T17:57:45.942336+0000","last_fullsized":"2023-10-14T17:57:45.942336+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:44:21.658770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943751+0000","last_change":"2023-10-14T17:57:44.941144+0000","last_active":"2023-10-14T17:57:45.943751+0000","last_peered":"2023-10-14T17:57:45.943751+0000","last_clean":"2023-10-14T17:57:45.943751+0000","last_became_active":"2023-10-14T17:57:44.940795+0000","last_became_peered":"2023-10-14T17:57:44.940795+0000","last_unstale":"2023-10-14T17:57:45.943751+0000","last_undegraded":"2023-10-14T17:57:45.943751+0000","last_fullsized":"2023-10-14T17:57:45.943751+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:58:49.126002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097935+0000","last_change":"2023-10-14T17:57:44.935468+0000","last_active":"2023-10-14T17:57:46.097935+0000","last_peered":"2023-10-14T17:57:46.097935+0000","last_clean":"2023-10-14T17:57:46.097935+0000","last_became_active":"2023-10-14T17:57:44.935192+0000","last_became_peered":"2023-10-14T17:57:44.935192+0000","last_unstale":"2023-10-14T17:57:46.097935+0000","last_undegraded":"2023-10-14T17:57:46.097935+0000","last_fullsized":"2023-10-14T17:57:46.097935+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:56.513623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097460+0000","last_change":"2023-10-14T17:57:44.939575+0000","last_active":"2023-10-14T17:57:46.097460+0000","last_peered":"2023-10-14T17:57:46.097460+0000","last_clean":"2023-10-14T17:57:46.097460+0000","last_became_active":"2023-10-14T17:57:44.939284+0000","last_became_peered":"2023-10-14T17:57:44.939284+0000","last_unstale":"2023-10-14T17:57:46.097460+0000","last_undegraded":"2023-10-14T17:57:46.097460+0000","last_fullsized":"2023-10-14T17:57:46.097460+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:37.156538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939444+0000","last_change":"2023-10-14T17:57:44.946848+0000","last_active":"2023-10-14T17:57:45.939444+0000","last_peered":"2023-10-14T17:57:45.939444+0000","last_clean":"2023-10-14T17:57:45.939444+0000","last_became_active":"2023-10-14T17:57:44.946553+0000","last_became_peered":"2023-10-14T17:57:44.946553+0000","last_unstale":"2023-10-14T17:57:45.939444+0000","last_undegraded":"2023-10-14T17:57:45.939444+0000","last_fullsized":"2023-10-14T17:57:45.939444+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:53.587873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096958+0000","last_change":"2023-10-14T17:57:44.931218+0000","last_active":"2023-10-14T17:57:46.096958+0000","last_peered":"2023-10-14T17:57:46.096958+0000","last_clean":"2023-10-14T17:57:46.096958+0000","last_became_active":"2023-10-14T17:57:44.930875+0000","last_became_peered":"2023-10-14T17:57:44.930875+0000","last_unstale":"2023-10-14T17:57:46.096958+0000","last_undegraded":"2023-10-14T17:57:46.096958+0000","last_fullsized":"2023-10-14T17:57:46.096958+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:31.088464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096953+0000","last_change":"2023-10-14T17:57:44.946439+0000","last_active":"2023-10-14T17:57:46.096953+0000","last_peered":"2023-10-14T17:57:46.096953+0000","last_clean":"2023-10-14T17:57:46.096953+0000","last_became_active":"2023-10-14T17:57:44.944623+0000","last_became_peered":"2023-10-14T17:57:44.944623+0000","last_unstale":"2023-10-14T17:57:46.096953+0000","last_undegraded":"2023-10-14T17:57:46.096953+0000","last_fullsized":"2023-10-14T17:57:46.096953+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:45.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939118+0000","last_change":"2023-10-14T17:57:44.934088+0000","last_active":"2023-10-14T17:57:45.939118+0000","last_peered":"2023-10-14T17:57:45.939118+0000","last_clean":"2023-10-14T17:57:45.939118+0000","last_became_active":"2023-10-14T17:57:44.933768+0000","last_became_peered":"2023-10-14T17:57:44.933768+0000","last_unstale":"2023-10-14T17:57:45.939118+0000","last_undegraded":"2023-10-14T17:57:45.939118+0000","last_fullsized":"2023-10-14T17:57:45.939118+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:23.748401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942680+0000","last_change":"2023-10-14T17:57:44.952397+0000","last_active":"2023-10-14T17:57:45.942680+0000","last_peered":"2023-10-14T17:57:45.942680+0000","last_clean":"2023-10-14T17:57:45.942680+0000","last_became_active":"2023-10-14T17:57:44.952101+0000","last_became_peered":"2023-10-14T17:57:44.952101+0000","last_unstale":"2023-10-14T17:57:45.942680+0000","last_undegraded":"2023-10-14T17:57:45.942680+0000","last_fullsized":"2023-10-14T17:57:45.942680+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:53.587873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095276+0000","last_change":"2023-10-14T17:57:44.945879+0000","last_active":"2023-10-14T17:57:46.095276+0000","last_peered":"2023-10-14T17:57:46.095276+0000","last_clean":"2023-10-14T17:57:46.095276+0000","last_became_active":"2023-10-14T17:57:44.945579+0000","last_became_peered":"2023-10-14T17:57:44.945579+0000","last_unstale":"2023-10-14T17:57:46.095276+0000","last_undegraded":"2023-10-14T17:57:46.095276+0000","last_fullsized":"2023-10-14T17:57:46.095276+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:09.356684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096055+0000","last_change":"2023-10-14T17:57:44.935651+0000","last_active":"2023-10-14T17:57:46.096055+0000","last_peered":"2023-10-14T17:57:46.096055+0000","last_clean":"2023-10-14T17:57:46.096055+0000","last_became_active":"2023-10-14T17:57:44.935384+0000","last_became_peered":"2023-10-14T17:57:44.935384+0000","last_unstale":"2023-10-14T17:57:46.096055+0000","last_undegraded":"2023-10-14T17:57:46.096055+0000","last_fullsized":"2023-10-14T17:57:46.096055+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:28.827875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100887+0000","last_change":"2023-10-14T17:57:44.934349+0000","last_active":"2023-10-14T17:57:46.100887+0000","last_peered":"2023-10-14T17:57:46.100887+0000","last_clean":"2023-10-14T17:57:46.100887+0000","last_became_active":"2023-10-14T17:57:44.933103+0000","last_became_peered":"2023-10-14T17:57:44.933103+0000","last_unstale":"2023-10-14T17:57:46.100887+0000","last_undegraded":"2023-10-14T17:57:46.100887+0000","last_fullsized":"2023-10-14T17:57:46.100887+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:05.086466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"177'4","reported_seq":16,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100787+0000","last_change":"2023-10-14T17:57:43.920589+0000","last_active":"2023-10-14T17:57:46.100787+0000","last_peered":"2023-10-14T17:57:46.100787+0000","last_clean":"2023-10-14T17:57:46.100787+0000","last_became_active":"2023-10-14T17:57:43.919468+0000","last_became_peered":"2023-10-14T17:57:43.919468+0000","last_unstale":"2023-10-14T17:57:46.100787+0000","last_undegraded":"2023-10-14T17:57:46.100787+0000","last_fullsized":"2023-10-14T17:57:46.100787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:32:18.321689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096485+0000","last_change":"2023-10-14T17:57:44.947807+0000","last_active":"2023-10-14T17:57:46.096485+0000","last_peered":"2023-10-14T17:57:46.096485+0000","last_clean":"2023-10-14T17:57:46.096485+0000","last_became_active":"2023-10-14T17:57:44.947530+0000","last_became_peered":"2023-10-14T17:57:44.947530+0000","last_unstale":"2023-10-14T17:57:46.096485+0000","last_undegraded":"2023-10-14T17:57:46.096485+0000","last_fullsized":"2023-10-14T17:57:46.096485+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:32.164095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095960+0000","last_change":"2023-10-14T17:57:43.916292+0000","last_active":"2023-10-14T17:57:46.095960+0000","last_peered":"2023-10-14T17:57:46.095960+0000","last_clean":"2023-10-14T17:57:46.095960+0000","last_became_active":"2023-10-14T17:57:43.916020+0000","last_became_peered":"2023-10-14T17:57:43.916020+0000","last_unstale":"2023-10-14T17:57:46.095960+0000","last_undegraded":"2023-10-14T17:57:46.095960+0000","last_fullsized":"2023-10-14T17:57:46.095960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:07:38.917413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938923+0000","last_change":"2023-10-14T17:57:44.935623+0000","last_active":"2023-10-14T17:57:45.938923+0000","last_peered":"2023-10-14T17:57:45.938923+0000","last_clean":"2023-10-14T17:57:45.938923+0000","last_became_active":"2023-10-14T17:57:44.935332+0000","last_became_peered":"2023-10-14T17:57:44.935332+0000","last_unstale":"2023-10-14T17:57:45.938923+0000","last_undegraded":"2023-10-14T17:57:45.938923+0000","last_fullsized":"2023-10-14T17:57:45.938923+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:44.458283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944397+0000","last_change":"2023-10-14T17:57:43.895545+0000","last_active":"2023-10-14T17:57:45.944397+0000","last_peered":"2023-10-14T17:57:45.944397+0000","last_clean":"2023-10-14T17:57:45.944397+0000","last_became_active":"2023-10-14T17:57:43.894949+0000","last_became_peered":"2023-10-14T17:57:43.894949+0000","last_unstale":"2023-10-14T17:57:45.944397+0000","last_undegraded":"2023-10-14T17:57:45.944397+0000","last_fullsized":"2023-10-14T17:57:45.944397+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:53.966578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100649+0000","last_change":"2023-10-14T17:57:44.933332+0000","last_active":"2023-10-14T17:57:46.100649+0000","last_peered":"2023-10-14T17:57:46.100649+0000","last_clean":"2023-10-14T17:57:46.100649+0000","last_became_active":"2023-10-14T17:57:44.933048+0000","last_became_peered":"2023-10-14T17:57:44.933048+0000","last_unstale":"2023-10-14T17:57:46.100649+0000","last_undegraded":"2023-10-14T17:57:46.100649+0000","last_fullsized":"2023-10-14T17:57:46.100649+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:25:31.351576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944484+0000","last_change":"2023-10-14T17:57:43.894372+0000","last_active":"2023-10-14T17:57:45.944484+0000","last_peered":"2023-10-14T17:57:45.944484+0000","last_clean":"2023-10-14T17:57:45.944484+0000","last_became_active":"2023-10-14T17:57:43.893581+0000","last_became_peered":"2023-10-14T17:57:43.893581+0000","last_unstale":"2023-10-14T17:57:45.944484+0000","last_undegraded":"2023-10-14T17:57:45.944484+0000","last_fullsized":"2023-10-14T17:57:45.944484+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:43.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095676+0000","last_change":"2023-10-14T17:57:44.943279+0000","last_active":"2023-10-14T17:57:46.095676+0000","last_peered":"2023-10-14T17:57:46.095676+0000","last_clean":"2023-10-14T17:57:46.095676+0000","last_became_active":"2023-10-14T17:57:44.942986+0000","last_became_peered":"2023-10-14T17:57:44.942986+0000","last_unstale":"2023-10-14T17:57:46.095676+0000","last_undegraded":"2023-10-14T17:57:46.095676+0000","last_fullsized":"2023-10-14T17:57:46.095676+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:07.265177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096452+0000","last_change":"2023-10-14T17:57:43.915673+0000","last_active":"2023-10-14T17:57:46.096452+0000","last_peered":"2023-10-14T17:57:46.096452+0000","last_clean":"2023-10-14T17:57:46.096452+0000","last_became_active":"2023-10-14T17:57:43.915378+0000","last_became_peered":"2023-10-14T17:57:43.915378+0000","last_unstale":"2023-10-14T17:57:46.096452+0000","last_undegraded":"2023-10-14T17:57:46.096452+0000","last_fullsized":"2023-10-14T17:57:46.096452+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:09.046290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942692+0000","last_change":"2023-10-14T17:57:44.941824+0000","last_active":"2023-10-14T17:57:45.942692+0000","last_peered":"2023-10-14T17:57:45.942692+0000","last_clean":"2023-10-14T17:57:45.942692+0000","last_became_active":"2023-10-14T17:57:44.941525+0000","last_became_peered":"2023-10-14T17:57:44.941525+0000","last_unstale":"2023-10-14T17:57:45.942692+0000","last_undegraded":"2023-10-14T17:57:45.942692+0000","last_fullsized":"2023-10-14T17:57:45.942692+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:18.533962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097431+0000","last_change":"2023-10-14T17:57:43.917338+0000","last_active":"2023-10-14T17:57:46.097431+0000","last_peered":"2023-10-14T17:57:46.097431+0000","last_clean":"2023-10-14T17:57:46.097431+0000","last_became_active":"2023-10-14T17:57:43.915771+0000","last_became_peered":"2023-10-14T17:57:43.915771+0000","last_unstale":"2023-10-14T17:57:46.097431+0000","last_undegraded":"2023-10-14T17:57:46.097431+0000","last_fullsized":"2023-10-14T17:57:46.097431+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:13:13.808377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095064+0000","last_change":"2023-10-14T17:57:44.939702+0000","last_active":"2023-10-14T17:57:46.095064+0000","last_peered":"2023-10-14T17:57:46.095064+0000","last_clean":"2023-10-14T17:57:46.095064+0000","last_became_active":"2023-10-14T17:57:44.939415+0000","last_became_peered":"2023-10-14T17:57:44.939415+0000","last_unstale":"2023-10-14T17:57:46.095064+0000","last_undegraded":"2023-10-14T17:57:46.095064+0000","last_fullsized":"2023-10-14T17:57:46.095064+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:45:15.450800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096899+0000","last_change":"2023-10-14T17:57:43.917984+0000","last_active":"2023-10-14T17:57:46.096899+0000","last_peered":"2023-10-14T17:57:46.096899+0000","last_clean":"2023-10-14T17:57:46.096899+0000","last_became_active":"2023-10-14T17:57:43.916743+0000","last_became_peered":"2023-10-14T17:57:43.916743+0000","last_unstale":"2023-10-14T17:57:46.096899+0000","last_undegraded":"2023-10-14T17:57:46.096899+0000","last_fullsized":"2023-10-14T17:57:46.096899+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:41.265113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097468+0000","last_change":"2023-10-14T17:57:44.954518+0000","last_active":"2023-10-14T17:57:46.097468+0000","last_peered":"2023-10-14T17:57:46.097468+0000","last_clean":"2023-10-14T17:57:46.097468+0000","last_became_active":"2023-10-14T17:57:44.954200+0000","last_became_peered":"2023-10-14T17:57:44.954200+0000","last_unstale":"2023-10-14T17:57:46.097468+0000","last_undegraded":"2023-10-14T17:57:46.097468+0000","last_fullsized":"2023-10-14T17:57:46.097468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:10.061259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941551+0000","last_change":"2023-10-14T17:57:43.910673+0000","last_active":"2023-10-14T17:57:45.941551+0000","last_peered":"2023-10-14T17:57:45.941551+0000","last_clean":"2023-10-14T17:57:45.941551+0000","last_became_active":"2023-10-14T17:57:43.910307+0000","last_became_peered":"2023-10-14T17:57:43.910307+0000","last_unstale":"2023-10-14T17:57:45.941551+0000","last_undegraded":"2023-10-14T17:57:45.941551+0000","last_fullsized":"2023-10-14T17:57:45.941551+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:08.304627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098294+0000","last_change":"2023-10-14T17:57:44.942967+0000","last_active":"2023-10-14T17:57:46.098294+0000","last_peered":"2023-10-14T17:57:46.098294+0000","last_clean":"2023-10-14T17:57:46.098294+0000","last_became_active":"2023-10-14T17:57:44.942683+0000","last_became_peered":"2023-10-14T17:57:44.942683+0000","last_unstale":"2023-10-14T17:57:46.098294+0000","last_undegraded":"2023-10-14T17:57:46.098294+0000","last_fullsized":"2023-10-14T17:57:46.098294+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:45.745017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096588+0000","last_change":"2023-10-14T17:57:43.915160+0000","last_active":"2023-10-14T17:57:46.096588+0000","last_peered":"2023-10-14T17:57:46.096588+0000","last_clean":"2023-10-14T17:57:46.096588+0000","last_became_active":"2023-10-14T17:57:43.914869+0000","last_became_peered":"2023-10-14T17:57:43.914869+0000","last_unstale":"2023-10-14T17:57:46.096588+0000","last_undegraded":"2023-10-14T17:57:46.096588+0000","last_fullsized":"2023-10-14T17:57:46.096588+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:44:46.412740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101002+0000","last_change":"2023-10-14T17:57:44.938449+0000","last_active":"2023-10-14T17:57:46.101002+0000","last_peered":"2023-10-14T17:57:46.101002+0000","last_clean":"2023-10-14T17:57:46.101002+0000","last_became_active":"2023-10-14T17:57:44.937826+0000","last_became_peered":"2023-10-14T17:57:44.937826+0000","last_unstale":"2023-10-14T17:57:46.101002+0000","last_undegraded":"2023-10-14T17:57:46.101002+0000","last_fullsized":"2023-10-14T17:57:46.101002+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:12.935374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101298+0000","last_change":"2023-10-14T17:57:43.919092+0000","last_active":"2023-10-14T17:57:46.101298+0000","last_peered":"2023-10-14T17:57:46.101298+0000","last_clean":"2023-10-14T17:57:46.101298+0000","last_became_active":"2023-10-14T17:57:43.918508+0000","last_became_peered":"2023-10-14T17:57:43.918508+0000","last_unstale":"2023-10-14T17:57:46.101298+0000","last_undegraded":"2023-10-14T17:57:46.101298+0000","last_fullsized":"2023-10-14T17:57:46.101298+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:23:54.726700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097091+0000","last_change":"2023-10-14T17:57:44.940629+0000","last_active":"2023-10-14T17:57:46.097091+0000","last_peered":"2023-10-14T17:57:46.097091+0000","last_clean":"2023-10-14T17:57:46.097091+0000","last_became_active":"2023-10-14T17:57:44.940364+0000","last_became_peered":"2023-10-14T17:57:44.940364+0000","last_unstale":"2023-10-14T17:57:46.097091+0000","last_undegraded":"2023-10-14T17:57:46.097091+0000","last_fullsized":"2023-10-14T17:57:46.097091+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:39.211044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"177'6","reported_seq":18,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097061+0000","last_change":"2023-10-14T17:57:43.913792+0000","last_active":"2023-10-14T17:57:46.097061+0000","last_peered":"2023-10-14T17:57:46.097061+0000","last_clean":"2023-10-14T17:57:46.097061+0000","last_became_active":"2023-10-14T17:57:43.913466+0000","last_became_peered":"2023-10-14T17:57:43.913466+0000","last_unstale":"2023-10-14T17:57:46.097061+0000","last_undegraded":"2023-10-14T17:57:46.097061+0000","last_fullsized":"2023-10-14T17:57:46.097061+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:44.730048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942761+0000","last_change":"2023-10-14T17:57:44.937036+0000","last_active":"2023-10-14T17:57:45.942761+0000","last_peered":"2023-10-14T17:57:45.942761+0000","last_clean":"2023-10-14T17:57:45.942761+0000","last_became_active":"2023-10-14T17:57:44.936196+0000","last_became_peered":"2023-10-14T17:57:44.936196+0000","last_unstale":"2023-10-14T17:57:45.942761+0000","last_undegraded":"2023-10-14T17:57:45.942761+0000","last_fullsized":"2023-10-14T17:57:45.942761+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:36.393884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942592+0000","last_change":"2023-10-14T17:57:43.904483+0000","last_active":"2023-10-14T17:57:45.942592+0000","last_peered":"2023-10-14T17:57:45.942592+0000","last_clean":"2023-10-14T17:57:45.942592+0000","last_became_active":"2023-10-14T17:57:43.901542+0000","last_became_peered":"2023-10-14T17:57:43.901542+0000","last_unstale":"2023-10-14T17:57:45.942592+0000","last_undegraded":"2023-10-14T17:57:45.942592+0000","last_fullsized":"2023-10-14T17:57:45.942592+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:39.771385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095712+0000","last_change":"2023-10-14T17:57:44.942380+0000","last_active":"2023-10-14T17:57:46.095712+0000","last_peered":"2023-10-14T17:57:46.095712+0000","last_clean":"2023-10-14T17:57:46.095712+0000","last_became_active":"2023-10-14T17:57:44.942074+0000","last_became_peered":"2023-10-14T17:57:44.942074+0000","last_unstale":"2023-10-14T17:57:46.095712+0000","last_undegraded":"2023-10-14T17:57:46.095712+0000","last_fullsized":"2023-10-14T17:57:46.095712+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:32:19.336658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942738+0000","last_change":"2023-10-14T17:57:43.915705+0000","last_active":"2023-10-14T17:57:45.942738+0000","last_peered":"2023-10-14T17:57:45.942738+0000","last_clean":"2023-10-14T17:57:45.942738+0000","last_became_active":"2023-10-14T17:57:43.915382+0000","last_became_peered":"2023-10-14T17:57:43.915382+0000","last_unstale":"2023-10-14T17:57:45.942738+0000","last_undegraded":"2023-10-14T17:57:45.942738+0000","last_fullsized":"2023-10-14T17:57:45.942738+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:11.422627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097340+0000","last_change":"2023-10-14T17:57:44.940993+0000","last_active":"2023-10-14T17:57:46.097340+0000","last_peered":"2023-10-14T17:57:46.097340+0000","last_clean":"2023-10-14T17:57:46.097340+0000","last_became_active":"2023-10-14T17:57:44.939813+0000","last_became_peered":"2023-10-14T17:57:44.939813+0000","last_unstale":"2023-10-14T17:57:46.097340+0000","last_undegraded":"2023-10-14T17:57:46.097340+0000","last_fullsized":"2023-10-14T17:57:46.097340+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:02.750348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938698+0000","last_change":"2023-10-14T17:57:43.896216+0000","last_active":"2023-10-14T17:57:45.938698+0000","last_peered":"2023-10-14T17:57:45.938698+0000","last_clean":"2023-10-14T17:57:45.938698+0000","last_became_active":"2023-10-14T17:57:43.895897+0000","last_became_peered":"2023-10-14T17:57:43.895897+0000","last_unstale":"2023-10-14T17:57:45.938698+0000","last_undegraded":"2023-10-14T17:57:45.938698+0000","last_fullsized":"2023-10-14T17:57:45.938698+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:28.765717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942949+0000","last_change":"2023-10-14T17:57:44.924605+0000","last_active":"2023-10-14T17:57:45.942949+0000","last_peered":"2023-10-14T17:57:45.942949+0000","last_clean":"2023-10-14T17:57:45.942949+0000","last_became_active":"2023-10-14T17:57:44.923940+0000","last_became_peered":"2023-10-14T17:57:44.923940+0000","last_unstale":"2023-10-14T17:57:45.942949+0000","last_undegraded":"2023-10-14T17:57:45.942949+0000","last_fullsized":"2023-10-14T17:57:45.942949+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:45.745017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"177'8","reported_seq":20,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096145+0000","last_change":"2023-10-14T17:57:43.918085+0000","last_active":"2023-10-14T17:57:46.096145+0000","last_peered":"2023-10-14T17:57:46.096145+0000","last_clean":"2023-10-14T17:57:46.096145+0000","last_became_active":"2023-10-14T17:57:43.917732+0000","last_became_peered":"2023-10-14T17:57:43.917732+0000","last_unstale":"2023-10-14T17:57:46.096145+0000","last_undegraded":"2023-10-14T17:57:46.096145+0000","last_fullsized":"2023-10-14T17:57:46.096145+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:08.057209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101126+0000","last_change":"2023-10-14T17:57:44.938894+0000","last_active":"2023-10-14T17:57:46.101126+0000","last_peered":"2023-10-14T17:57:46.101126+0000","last_clean":"2023-10-14T17:57:46.101126+0000","last_became_active":"2023-10-14T17:57:44.938607+0000","last_became_peered":"2023-10-14T17:57:44.938607+0000","last_unstale":"2023-10-14T17:57:46.101126+0000","last_undegraded":"2023-10-14T17:57:46.101126+0000","last_fullsized":"2023-10-14T17:57:46.101126+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:41.450998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097994+0000","last_change":"2023-10-14T17:57:43.916493+0000","last_active":"2023-10-14T17:57:46.097994+0000","last_peered":"2023-10-14T17:57:46.097994+0000","last_clean":"2023-10-14T17:57:46.097994+0000","last_became_active":"2023-10-14T17:57:43.916218+0000","last_became_peered":"2023-10-14T17:57:43.916218+0000","last_unstale":"2023-10-14T17:57:46.097994+0000","last_undegraded":"2023-10-14T17:57:46.097994+0000","last_fullsized":"2023-10-14T17:57:46.097994+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:10.713743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097629+0000","last_change":"2023-10-14T17:57:44.933787+0000","last_active":"2023-10-14T17:57:46.097629+0000","last_peered":"2023-10-14T17:57:46.097629+0000","last_clean":"2023-10-14T17:57:46.097629+0000","last_became_active":"2023-10-14T17:57:44.930651+0000","last_became_peered":"2023-10-14T17:57:44.930651+0000","last_unstale":"2023-10-14T17:57:46.097629+0000","last_undegraded":"2023-10-14T17:57:46.097629+0000","last_fullsized":"2023-10-14T17:57:46.097629+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:11:56.739105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096001+0000","last_change":"2023-10-14T17:57:43.919216+0000","last_active":"2023-10-14T17:57:46.096001+0000","last_peered":"2023-10-14T17:57:46.096001+0000","last_clean":"2023-10-14T17:57:46.096001+0000","last_became_active":"2023-10-14T17:57:43.918896+0000","last_became_peered":"2023-10-14T17:57:43.918896+0000","last_unstale":"2023-10-14T17:57:46.096001+0000","last_undegraded":"2023-10-14T17:57:46.096001+0000","last_fullsized":"2023-10-14T17:57:46.096001+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:00:50.427057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943668+0000","last_change":"2023-10-14T17:57:44.937920+0000","last_active":"2023-10-14T17:57:45.943668+0000","last_peered":"2023-10-14T17:57:45.943668+0000","last_clean":"2023-10-14T17:57:45.943668+0000","last_became_active":"2023-10-14T17:57:44.936557+0000","last_became_peered":"2023-10-14T17:57:44.936557+0000","last_unstale":"2023-10-14T17:57:45.943668+0000","last_undegraded":"2023-10-14T17:57:45.943668+0000","last_fullsized":"2023-10-14T17:57:45.943668+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:09.072178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097280+0000","last_change":"2023-10-14T17:57:43.918617+0000","last_active":"2023-10-14T17:57:46.097280+0000","last_peered":"2023-10-14T17:57:46.097280+0000","last_clean":"2023-10-14T17:57:46.097280+0000","last_became_active":"2023-10-14T17:57:43.918349+0000","last_became_peered":"2023-10-14T17:57:43.918349+0000","last_unstale":"2023-10-14T17:57:46.097280+0000","last_undegraded":"2023-10-14T17:57:46.097280+0000","last_fullsized":"2023-10-14T17:57:46.097280+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:20.056835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939440+0000","last_change":"2023-10-14T17:57:44.946394+0000","last_active":"2023-10-14T17:57:45.939440+0000","last_peered":"2023-10-14T17:57:45.939440+0000","last_clean":"2023-10-14T17:57:45.939440+0000","last_became_active":"2023-10-14T17:57:44.945236+0000","last_became_peered":"2023-10-14T17:57:44.945236+0000","last_unstale":"2023-10-14T17:57:45.939440+0000","last_undegraded":"2023-10-14T17:57:45.939440+0000","last_fullsized":"2023-10-14T17:57:45.939440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:18.533962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096060+0000","last_change":"2023-10-14T17:57:43.912578+0000","last_active":"2023-10-14T17:57:46.096060+0000","last_peered":"2023-10-14T17:57:46.096060+0000","last_clean":"2023-10-14T17:57:46.096060+0000","last_became_active":"2023-10-14T17:57:43.912293+0000","last_became_peered":"2023-10-14T17:57:43.912293+0000","last_unstale":"2023-10-14T17:57:46.096060+0000","last_undegraded":"2023-10-14T17:57:46.096060+0000","last_fullsized":"2023-10-14T17:57:46.096060+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:16.097447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943021+0000","last_change":"2023-10-14T17:57:44.952980+0000","last_active":"2023-10-14T17:57:45.943021+0000","last_peered":"2023-10-14T17:57:45.943021+0000","last_clean":"2023-10-14T17:57:45.943021+0000","last_became_active":"2023-10-14T17:57:44.952711+0000","last_became_peered":"2023-10-14T17:57:44.952711+0000","last_unstale":"2023-10-14T17:57:45.943021+0000","last_undegraded":"2023-10-14T17:57:45.943021+0000","last_fullsized":"2023-10-14T17:57:45.943021+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:06.615328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097145+0000","last_change":"2023-10-14T17:57:43.904455+0000","last_active":"2023-10-14T17:57:46.097145+0000","last_peered":"2023-10-14T17:57:46.097145+0000","last_clean":"2023-10-14T17:57:46.097145+0000","last_became_active":"2023-10-14T17:57:43.902019+0000","last_became_peered":"2023-10-14T17:57:43.902019+0000","last_unstale":"2023-10-14T17:57:46.097145+0000","last_undegraded":"2023-10-14T17:57:46.097145+0000","last_fullsized":"2023-10-14T17:57:46.097145+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:37.770797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096969+0000","last_change":"2023-10-14T17:57:44.946938+0000","last_active":"2023-10-14T17:57:46.096969+0000","last_peered":"2023-10-14T17:57:46.096969+0000","last_clean":"2023-10-14T17:57:46.096969+0000","last_became_active":"2023-10-14T17:57:44.946394+0000","last_became_peered":"2023-10-14T17:57:44.946394+0000","last_unstale":"2023-10-14T17:57:46.096969+0000","last_undegraded":"2023-10-14T17:57:46.096969+0000","last_fullsized":"2023-10-14T17:57:46.096969+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:21.071804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.099072+0000","last_change":"2023-10-14T17:57:43.919522+0000","last_active":"2023-10-14T17:57:46.099072+0000","last_peered":"2023-10-14T17:57:46.099072+0000","last_clean":"2023-10-14T17:57:46.099072+0000","last_became_active":"2023-10-14T17:57:43.919109+0000","last_became_peered":"2023-10-14T17:57:43.919109+0000","last_unstale":"2023-10-14T17:57:46.099072+0000","last_undegraded":"2023-10-14T17:57:46.099072+0000","last_fullsized":"2023-10-14T17:57:46.099072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:35.804250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941225+0000","last_change":"2023-10-14T17:57:44.939466+0000","last_active":"2023-10-14T17:57:45.941225+0000","last_peered":"2023-10-14T17:57:45.941225+0000","last_clean":"2023-10-14T17:57:45.941225+0000","last_became_active":"2023-10-14T17:57:44.938799+0000","last_became_peered":"2023-10-14T17:57:44.938799+0000","last_unstale":"2023-10-14T17:57:45.941225+0000","last_undegraded":"2023-10-14T17:57:45.941225+0000","last_fullsized":"2023-10-14T17:57:45.941225+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:56.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098901+0000","last_change":"2023-10-14T17:57:43.915810+0000","last_active":"2023-10-14T17:57:46.098901+0000","last_peered":"2023-10-14T17:57:46.098901+0000","last_clean":"2023-10-14T17:57:46.098901+0000","last_became_active":"2023-10-14T17:57:43.915552+0000","last_became_peered":"2023-10-14T17:57:43.915552+0000","last_unstale":"2023-10-14T17:57:46.098901+0000","last_undegraded":"2023-10-14T17:57:46.098901+0000","last_fullsized":"2023-10-14T17:57:46.098901+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:23:10.612958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096152+0000","last_change":"2023-10-14T17:57:44.947183+0000","last_active":"2023-10-14T17:57:46.096152+0000","last_peered":"2023-10-14T17:57:46.096152+0000","last_clean":"2023-10-14T17:57:46.096152+0000","last_became_active":"2023-10-14T17:57:44.946241+0000","last_became_peered":"2023-10-14T17:57:44.946241+0000","last_unstale":"2023-10-14T17:57:46.096152+0000","last_undegraded":"2023-10-14T17:57:46.096152+0000","last_fullsized":"2023-10-14T17:57:46.096152+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:56:57.242181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941940+0000","last_change":"2023-10-14T17:57:43.916802+0000","last_active":"2023-10-14T17:57:45.941940+0000","last_peered":"2023-10-14T17:57:45.941940+0000","last_clean":"2023-10-14T17:57:45.941940+0000","last_became_active":"2023-10-14T17:57:43.914415+0000","last_became_peered":"2023-10-14T17:57:43.914415+0000","last_unstale":"2023-10-14T17:57:45.941940+0000","last_undegraded":"2023-10-14T17:57:45.941940+0000","last_fullsized":"2023-10-14T17:57:45.941940+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:09.071087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098430+0000","last_change":"2023-10-14T17:57:44.940212+0000","last_active":"2023-10-14T17:57:46.098430+0000","last_peered":"2023-10-14T17:57:46.098430+0000","last_clean":"2023-10-14T17:57:46.098430+0000","last_became_active":"2023-10-14T17:57:44.939908+0000","last_became_peered":"2023-10-14T17:57:44.939908+0000","last_unstale":"2023-10-14T17:57:46.098430+0000","last_undegraded":"2023-10-14T17:57:46.098430+0000","last_fullsized":"2023-10-14T17:57:46.098430+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:09.072178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101307+0000","last_change":"2023-10-14T17:57:43.921693+0000","last_active":"2023-10-14T17:57:46.101307+0000","last_peered":"2023-10-14T17:57:46.101307+0000","last_clean":"2023-10-14T17:57:46.101307+0000","last_became_active":"2023-10-14T17:57:43.921265+0000","last_became_peered":"2023-10-14T17:57:43.921265+0000","last_unstale":"2023-10-14T17:57:46.101307+0000","last_undegraded":"2023-10-14T17:57:46.101307+0000","last_fullsized":"2023-10-14T17:57:46.101307+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:22.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944857+0000","last_change":"2023-10-14T17:57:44.937130+0000","last_active":"2023-10-14T17:57:45.944857+0000","last_peered":"2023-10-14T17:57:45.944857+0000","last_clean":"2023-10-14T17:57:45.944857+0000","last_became_active":"2023-10-14T17:57:44.936833+0000","last_became_peered":"2023-10-14T17:57:44.936833+0000","last_unstale":"2023-10-14T17:57:45.944857+0000","last_undegraded":"2023-10-14T17:57:45.944857+0000","last_fullsized":"2023-10-14T17:57:45.944857+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:45.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939389+0000","last_change":"2023-10-14T17:57:43.912468+0000","last_active":"2023-10-14T17:57:45.939389+0000","last_peered":"2023-10-14T17:57:45.939389+0000","last_clean":"2023-10-14T17:57:45.939389+0000","last_became_active":"2023-10-14T17:57:43.912174+0000","last_became_peered":"2023-10-14T17:57:43.912174+0000","last_unstale":"2023-10-14T17:57:45.939389+0000","last_undegraded":"2023-10-14T17:57:45.939389+0000","last_fullsized":"2023-10-14T17:57:45.939389+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:08.304627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095950+0000","last_change":"2023-10-14T17:57:44.949002+0000","last_active":"2023-10-14T17:57:46.095950+0000","last_peered":"2023-10-14T17:57:46.095950+0000","last_clean":"2023-10-14T17:57:46.095950+0000","last_became_active":"2023-10-14T17:57:44.948662+0000","last_became_peered":"2023-10-14T17:57:44.948662+0000","last_unstale":"2023-10-14T17:57:46.095950+0000","last_undegraded":"2023-10-14T17:57:46.095950+0000","last_fullsized":"2023-10-14T17:57:46.095950+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:36.819220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939589+0000","last_change":"2023-10-14T17:57:43.914999+0000","last_active":"2023-10-14T17:57:45.939589+0000","last_peered":"2023-10-14T17:57:45.939589+0000","last_clean":"2023-10-14T17:57:45.939589+0000","last_became_active":"2023-10-14T17:57:43.914698+0000","last_became_peered":"2023-10-14T17:57:43.914698+0000","last_unstale":"2023-10-14T17:57:45.939589+0000","last_undegraded":"2023-10-14T17:57:45.939589+0000","last_fullsized":"2023-10-14T17:57:45.939589+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:10.178327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939468+0000","last_change":"2023-10-14T17:57:44.936258+0000","last_active":"2023-10-14T17:57:45.939468+0000","last_peered":"2023-10-14T17:57:45.939468+0000","last_clean":"2023-10-14T17:57:45.939468+0000","last_became_active":"2023-10-14T17:57:44.935951+0000","last_became_peered":"2023-10-14T17:57:44.935951+0000","last_unstale":"2023-10-14T17:57:45.939468+0000","last_undegraded":"2023-10-14T17:57:45.939468+0000","last_fullsized":"2023-10-14T17:57:45.939468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:06.615328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939058+0000","last_change":"2023-10-14T17:57:43.914406+0000","last_active":"2023-10-14T17:57:45.939058+0000","last_peered":"2023-10-14T17:57:45.939058+0000","last_clean":"2023-10-14T17:57:45.939058+0000","last_became_active":"2023-10-14T17:57:43.914101+0000","last_became_peered":"2023-10-14T17:57:43.914101+0000","last_unstale":"2023-10-14T17:57:45.939058+0000","last_undegraded":"2023-10-14T17:57:45.939058+0000","last_fullsized":"2023-10-14T17:57:45.939058+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:21.136262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096424+0000","last_change":"2023-10-14T17:57:44.937780+0000","last_active":"2023-10-14T17:57:46.096424+0000","last_peered":"2023-10-14T17:57:46.096424+0000","last_clean":"2023-10-14T17:57:46.096424+0000","last_became_active":"2023-10-14T17:57:44.937444+0000","last_became_peered":"2023-10-14T17:57:44.937444+0000","last_unstale":"2023-10-14T17:57:46.096424+0000","last_undegraded":"2023-10-14T17:57:46.096424+0000","last_fullsized":"2023-10-14T17:57:46.096424+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:07:39.409447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097242+0000","last_change":"2023-10-14T17:57:43.905065+0000","last_active":"2023-10-14T17:57:46.097242+0000","last_peered":"2023-10-14T17:57:46.097242+0000","last_clean":"2023-10-14T17:57:46.097242+0000","last_became_active":"2023-10-14T17:57:43.904355+0000","last_became_peered":"2023-10-14T17:57:43.904355+0000","last_unstale":"2023-10-14T17:57:46.097242+0000","last_undegraded":"2023-10-14T17:57:46.097242+0000","last_fullsized":"2023-10-14T17:57:46.097242+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:08.057209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097911+0000","last_change":"2023-10-14T17:57:44.937752+0000","last_active":"2023-10-14T17:57:46.097911+0000","last_peered":"2023-10-14T17:57:46.097911+0000","last_clean":"2023-10-14T17:57:46.097911+0000","last_became_active":"2023-10-14T17:57:44.937291+0000","last_became_peered":"2023-10-14T17:57:44.937291+0000","last_unstale":"2023-10-14T17:57:46.097911+0000","last_undegraded":"2023-10-14T17:57:46.097911+0000","last_fullsized":"2023-10-14T17:57:46.097911+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:38.785766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.099283+0000","last_change":"2023-10-14T17:57:43.917888+0000","last_active":"2023-10-14T17:57:46.099283+0000","last_peered":"2023-10-14T17:57:46.099283+0000","last_clean":"2023-10-14T17:57:46.099283+0000","last_became_active":"2023-10-14T17:57:43.917465+0000","last_became_peered":"2023-10-14T17:57:43.917465+0000","last_unstale":"2023-10-14T17:57:46.099283+0000","last_undegraded":"2023-10-14T17:57:46.099283+0000","last_fullsized":"2023-10-14T17:57:46.099283+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:59.451750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098468+0000","last_change":"2023-10-14T17:57:44.934521+0000","last_active":"2023-10-14T17:57:46.098468+0000","last_peered":"2023-10-14T17:57:46.098468+0000","last_clean":"2023-10-14T17:57:46.098468+0000","last_became_active":"2023-10-14T17:57:44.932476+0000","last_became_peered":"2023-10-14T17:57:44.932476+0000","last_unstale":"2023-10-14T17:57:46.098468+0000","last_undegraded":"2023-10-14T17:57:46.098468+0000","last_fullsized":"2023-10-14T17:57:46.098468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:27:42.459551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942990+0000","last_change":"2023-10-14T17:57:43.894348+0000","last_active":"2023-10-14T17:57:45.942990+0000","last_peered":"2023-10-14T17:57:45.942990+0000","last_clean":"2023-10-14T17:57:45.942990+0000","last_became_active":"2023-10-14T17:57:43.893675+0000","last_became_peered":"2023-10-14T17:57:43.893675+0000","last_unstale":"2023-10-14T17:57:45.942990+0000","last_undegraded":"2023-10-14T17:57:45.942990+0000","last_fullsized":"2023-10-14T17:57:45.942990+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:47.223729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097508+0000","last_change":"2023-10-14T17:57:44.941836+0000","last_active":"2023-10-14T17:57:46.097508+0000","last_peered":"2023-10-14T17:57:46.097508+0000","last_clean":"2023-10-14T17:57:46.097508+0000","last_became_active":"2023-10-14T17:57:44.941563+0000","last_became_peered":"2023-10-14T17:57:44.941563+0000","last_unstale":"2023-10-14T17:57:46.097508+0000","last_undegraded":"2023-10-14T17:57:46.097508+0000","last_fullsized":"2023-10-14T17:57:46.097508+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:12.437596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095020+0000","last_change":"2023-10-14T17:57:43.912407+0000","last_active":"2023-10-14T17:57:46.095020+0000","last_peered":"2023-10-14T17:57:46.095020+0000","last_clean":"2023-10-14T17:57:46.095020+0000","last_became_active":"2023-10-14T17:57:43.912106+0000","last_became_peered":"2023-10-14T17:57:43.912106+0000","last_unstale":"2023-10-14T17:57:46.095020+0000","last_undegraded":"2023-10-14T17:57:46.095020+0000","last_fullsized":"2023-10-14T17:57:46.095020+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:24.440166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096958+0000","last_change":"2023-10-14T17:57:44.947794+0000","last_active":"2023-10-14T17:57:46.096958+0000","last_peered":"2023-10-14T17:57:46.096958+0000","last_clean":"2023-10-14T17:57:46.096958+0000","last_became_active":"2023-10-14T17:57:44.943854+0000","last_became_peered":"2023-10-14T17:57:44.943854+0000","last_unstale":"2023-10-14T17:57:46.096958+0000","last_undegraded":"2023-10-14T17:57:46.096958+0000","last_fullsized":"2023-10-14T17:57:46.096958+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:45.527699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"177'3","reported_seq":15,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944008+0000","last_change":"2023-10-14T17:57:43.915563+0000","last_active":"2023-10-14T17:57:45.944008+0000","last_peered":"2023-10-14T17:57:45.944008+0000","last_clean":"2023-10-14T17:57:45.944008+0000","last_became_active":"2023-10-14T17:57:43.914800+0000","last_became_peered":"2023-10-14T17:57:43.914800+0000","last_unstale":"2023-10-14T17:57:45.944008+0000","last_undegraded":"2023-10-14T17:57:45.944008+0000","last_fullsized":"2023-10-14T17:57:45.944008+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:41.265113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"166'271","reported_seq":883,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-10-14T17:57:44.920676+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:57:44.920676+0000","last_peered":"2023-10-14T17:57:44.920676+0000","last_clean":"2023-10-14T17:57:44.920676+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:57:44.920676+0000","last_undegraded":"2023-10-14T17:57:44.920676+0000","last_fullsized":"2023-10-14T17:57:44.920676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575773,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51984,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319856,"statfs":{"total":96636764160,"available":96583532544,"internally_reserved":0,"allocated":2822144,"data_stored":1679928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1830000000000001,"15min":1.2050000000000001},"min":{"1min":0.77300000000000002,"5min":0.73099999999999998,"15min":0.71199999999999997},"max":{"1min":1.429,"5min":1.9019999999999999,"15min":1.9710000000000001},"last":1.23},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.837},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.347,"15min":1.3100000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.29,"15min":1.306},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5720000000000001,"5min":1.8,"15min":1.804},"last":1.357}]},{"osd":2,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2430000000000001,"15min":1.2509999999999999},"min":{"1min":0.873,"5min":0.82899999999999996,"15min":0.82199999999999995},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.472,"5min":1.393,"15min":1.3779999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.71}]},{"osd":3,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4199999999999999,"15min":1.4059999999999999},"min":{"1min":0.996,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.883,"5min":1.883,"15min":2.234},"last":1.302},{"interface":"front","average":{"1min":1.504,"5min":1.5429999999999999,"15min":1.5329999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.9410000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.5389999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.335,"15min":1.276},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.573},{"interface":"front","average":{"1min":1.512,"5min":1.403,"15min":1.373},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.5510000000000002},"last":1.177}]},{"osd":5,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.464,"15min":1.458},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0539999999999998},"last":1.425},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4930000000000001,"15min":1.474},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.218,"5min":2.218,"15min":2.6619999999999999},"last":1.675}]},{"osd":6,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.6140000000000001,"15min":1.585},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4430000000000001},"last":1.74},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.5900000000000001,"15min":1.538},"min":{"1min":1.3440000000000001,"5min":1.0149999999999999,"15min":0.91600000000000004},"max":{"1min":2.1440000000000001,"5min":2.3100000000000001,"15min":2.4100000000000001},"last":1.474}]}]},{"osd":4,"up_from":13,"seq":55834575773,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55560,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316280,"statfs":{"total":96636764160,"available":96579870720,"internally_reserved":0,"allocated":2220032,"data_stored":1088908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2689999999999999,"15min":1.2270000000000001},"min":{"1min":0.81000000000000005,"5min":0.73699999999999999,"15min":0.52500000000000002},"max":{"1min":1.732,"5min":1.7749999999999999,"15min":1.8859999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.248,"15min":1.272},"min":{"1min":0.97299999999999998,"5min":0.86699999999999999,"15min":0.84599999999999997},"max":{"1min":1.897,"5min":1.897,"15min":1.905},"last":1.3819999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3220000000000001},"min":{"1min":0.89100000000000001,"5min":0.874,"15min":0.83999999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.038},{"interface":"front","average":{"1min":1.224,"5min":1.278,"15min":1.2829999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.7310000000000001,"5min":1.7629999999999999,"15min":2.02},"last":1.4370000000000001}]},{"osd":2,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.29,"15min":1.298},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.63800000000000001},"max":{"1min":1.827,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.488},{"interface":"front","average":{"1min":1.385,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":1.0960000000000001,"5min":1.0209999999999999,"15min":0.84199999999999997},"max":{"1min":1.8680000000000001,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.212}]},{"osd":3,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.347,"15min":1.353},"min":{"1min":0.93000000000000005,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.6439999999999999,"5min":1.752,"15min":1.984},"last":1.325},{"interface":"front","average":{"1min":1.552,"5min":1.492,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":1.0109999999999999,"15min":0.82299999999999995},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.734}]},{"osd":5,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4139999999999999,"15min":1.3640000000000001},"min":{"1min":0.97399999999999998,"5min":0.91500000000000004,"15min":0.82899999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3779999999999999,"15min":1.3740000000000001},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.79700000000000004},"max":{"1min":1.925,"5min":1.925,"15min":2.0499999999999998},"last":1.536}]},{"osd":6,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4219999999999999,"15min":1.3819999999999999},"min":{"1min":1.004,"5min":0.874,"15min":0.57999999999999996},"max":{"1min":1.9239999999999999,"5min":2.117,"15min":2.117},"last":1.8029999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.5600000000000001,"15min":1.5349999999999999},"min":{"1min":1.202,"5min":1.133,"15min":1.006},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.6000000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5609999999999999,"15min":1.506},"min":{"1min":1.0780000000000001,"5min":0.92500000000000004,"15min":0.86799999999999999},"max":{"1min":2.0880000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":2.0190000000000001},{"interface":"front","average":{"1min":1.696,"5min":1.6319999999999999,"15min":1.593},"min":{"1min":1.4410000000000001,"5min":1.179,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":2.141}]}]},{"osd":6,"up_from":13,"seq":55834575772,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41680,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330160,"statfs":{"total":96636764160,"available":96594083840,"internally_reserved":0,"allocated":2232320,"data_stored":1089054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.73999999999999999},"max":{"1min":1.4870000000000001,"5min":1.696,"15min":1.837},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.1850000000000001,"15min":1.2010000000000001},"min":{"1min":0.76500000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.401,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":0.96199999999999997}]},{"osd":1,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.276,"15min":1.244},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.5640000000000001,"5min":1.752,"15min":1.927},"last":0.749},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2989999999999999,"15min":1.3220000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.6040000000000001,"5min":1.7849999999999999,"15min":1.8069999999999999},"last":1.329}]},{"osd":2,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86299999999999999},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8740000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.4450000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":1.9370000000000001},"last":1.4510000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.417},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0409999999999999},"last":1.897},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.393,"15min":1.431},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.764,"5min":1.903,"15min":1.9790000000000001},"last":1.2310000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.361,"15min":1.367},"min":{"1min":1.014,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.736,"5min":1.929,"15min":1.984},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4379999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.889,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.7150000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3879999999999999,"15min":1.361},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.79000000000000004},"max":{"1min":1.629,"5min":1.7749999999999999,"15min":2.085},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.5580000000000001},"min":{"1min":1.214,"5min":1.048,"15min":1.048},"max":{"1min":1.9450000000000001,"5min":2.0310000000000001,"15min":2.105},"last":2.1840000000000002}]},{"osd":7,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5900000000000001,"15min":1.5840000000000001},"min":{"1min":1.165,"5min":1.026,"15min":1.0189999999999999},"max":{"1min":2.0960000000000001,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.9490000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.5640000000000001,"15min":1.5720000000000001},"min":{"1min":1.268,"5min":1.0129999999999999,"15min":0.81000000000000005},"max":{"1min":1.8700000000000001,"5min":1.954,"15min":2.1589999999999998},"last":1.782}]}]},{"osd":5,"up_from":13,"seq":55834575773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":2224128,"data_stored":1088930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.2070000000000001,"15min":1.1870000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.7949999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.2090000000000001},"min":{"1min":0.85299999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6040000000000001,"5min":1.677,"15min":2.0150000000000001},"last":1.131}]},{"osd":1,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.623,"5min":1.774,"15min":1.774},"last":1.637},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.286,"15min":1.286},"min":{"1min":0.85399999999999998,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.78,"5min":1.7949999999999999,"15min":1.8280000000000001},"last":0.996}]},{"osd":2,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.462,"5min":1.6879999999999999,"15min":1.72},"last":1.73},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.331,"15min":1.353},"min":{"1min":0.91700000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.6779999999999999,"5min":1.899,"15min":2.4140000000000001},"last":1.758}]},{"osd":3,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.458,"15min":1.444},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.419},{"interface":"front","average":{"1min":1.333,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.655,"5min":1.8799999999999999,"15min":2.5219999999999998},"last":1.7889999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3220000000000001,"15min":1.3260000000000001},"min":{"1min":0.89800000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.607,"5min":1.9850000000000001,"15min":2.7210000000000001},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.381,"15min":1.393},"min":{"1min":0.96999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.617}]},{"osd":6,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4119999999999999,"15min":1.3819999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.7949999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.577,"15min":1.581},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.86599999999999999},"max":{"1min":1.804,"5min":2.1389999999999998,"15min":2.907},"last":1.8919999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5509999999999999,"15min":1.544},"min":{"1min":1.073,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7889999999999999,"5min":2.1539999999999999,"15min":2.1760000000000002},"last":1.694},{"interface":"front","average":{"1min":1.413,"5min":1.5049999999999999,"15min":1.516},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.873,"5min":2.0009999999999999,"15min":2.9630000000000001},"last":1.6639999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575773,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51400,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320440,"statfs":{"total":96636764160,"available":96584130560,"internally_reserved":0,"allocated":2224128,"data_stored":1088998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.98199999999999998,"15min":0.98099999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.423,"5min":1.7949999999999999,"15min":2.6680000000000001},"last":1.7},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1890000000000001,"15min":1.206},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47999999999999998},"max":{"1min":1.5469999999999999,"5min":1.6240000000000001,"15min":2.8290000000000002},"last":1.2989999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0369999999999999,"15min":1.008},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.843},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.232,"15min":1.21},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7929999999999999,"5min":1.849,"15min":2.1970000000000001},"last":1.103}]},{"osd":2,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.454,"15min":1.4550000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6759999999999999,"5min":1.9299999999999999,"15min":2.1000000000000001},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.427,"15min":1.4019999999999999},"min":{"1min":0.80000000000000004,"5min":0.79500000000000004,"15min":0.55100000000000005},"max":{"1min":1.8979999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.5}]},{"osd":4,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4550000000000001,"15min":1.427},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.92500000000000004},"max":{"1min":1.716,"5min":1.8660000000000001,"15min":2.0489999999999999},"last":1.931},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.48,"15min":1.468},"min":{"1min":0.96599999999999997,"5min":0.96299999999999997,"15min":0.86099999999999999},"max":{"1min":1.9079999999999999,"5min":2.2210000000000001,"15min":2.6459999999999999},"last":1.3899999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4590000000000001,"15min":1.478},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.73399999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.5830000000000002},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.514,"15min":1.52},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7949999999999999,"5min":1.9850000000000001,"15min":2.3719999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.6699999999999999,"15min":1.6419999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.893,"5min":2.6480000000000001,"15min":2.851},"last":1.875},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.669,"15min":1.679},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0289999999999999,"5min":2.7109999999999999,"15min":2.9910000000000001},"last":1.5529999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.643,"15min":1.6200000000000001},"min":{"1min":1.163,"5min":1.163,"15min":0.79500000000000004},"max":{"1min":2.2040000000000002,"5min":2.4140000000000001,"15min":2.9430000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.6679999999999999,"15min":1.6839999999999999},"min":{"1min":1.2330000000000001,"5min":1.2250000000000001,"15min":1.137},"max":{"1min":1.897,"5min":2.0899999999999999,"15min":2.4980000000000002},"last":1.615}]}]},{"osd":2,"up_from":13,"seq":55834575773,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51412,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320428,"statfs":{"total":96636764160,"available":96584118272,"internally_reserved":0,"allocated":2236416,"data_stored":1090590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.95499999999999996,"15min":0.93600000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3520000000000001,"5min":1.6879999999999999,"15min":1.8759999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.089,"15min":1.115},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.7490000000000001,"15min":1.7889999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.03,"15min":1.0249999999999999},"min":{"1min":0.52300000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5720000000000001,"5min":1.6080000000000001,"15min":1.8620000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.764,"5min":1.8129999999999999,"15min":1.8959999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2210000000000001,"15min":1.274},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.5600000000000001,"5min":1.929,"15min":1.929},"last":0.622},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.375,"15min":1.3640000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0289999999999999},"last":0.72799999999999998}]},{"osd":4,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.472,"15min":1.4790000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.153},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3740000000000001,"15min":1.4099999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.917,"5min":2.0640000000000001,"15min":2.2639999999999998},"last":0.92200000000000004}]},{"osd":5,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4319999999999999,"15min":1.462},"min":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.865,"5min":1.9359999999999999,"15min":2.2669999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.611,"5min":1.591,"15min":1.585},"min":{"1min":0.94599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0070000000000001,"5min":2.0880000000000001,"15min":2.3199999999999998},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5169999999999999,"15min":1.52},"min":{"1min":0.98099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0470000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.54,"5min":1.583,"15min":1.6120000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.0169999999999999,"5min":2.161,"15min":2.3610000000000002},"last":0.97099999999999997}]},{"osd":7,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":1.008,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1640000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.645,"15min":1.677},"min":{"1min":1.0660000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.137,"5min":2.177,"15min":2.4350000000000001},"last":1.0880000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575773,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47240,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324600,"statfs":{"total":96636764160,"available":96588390400,"internally_reserved":0,"allocated":2224128,"data_stored":1089490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.97499999999999998,"15min":0.94999999999999996},"min":{"1min":0.68400000000000005,"5min":0.65100000000000002,"15min":0.64800000000000002},"max":{"1min":1.4630000000000001,"5min":1.706,"15min":1.756},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1379999999999999,"15min":1.1559999999999999},"min":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.51600000000000001},"max":{"1min":1.6579999999999999,"5min":1.724,"15min":1.9299999999999999},"last":0.379}]},{"osd":2,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.78800000000000003,"5min":0.77200000000000002,"15min":0.71499999999999997},"max":{"1min":1.7709999999999999,"5min":1.8660000000000001,"15min":2.597},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.2589999999999999,"15min":1.258},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.81699999999999995},"max":{"1min":1.6699999999999999,"5min":1.7470000000000001,"15min":1.9610000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4079999999999999,"15min":1.3720000000000001},"min":{"1min":1.032,"5min":0.94399999999999995,"15min":0.89700000000000002},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.877},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.327,"15min":1.2989999999999999},"min":{"1min":0.94799999999999995,"5min":0.86299999999999999,"15min":0.84999999999999998},"max":{"1min":1.7669999999999999,"5min":1.8160000000000001,"15min":2.2599999999999998},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.464,"15min":1.474},"min":{"1min":1.1799999999999999,"5min":1.1339999999999999,"15min":0.78400000000000003},"max":{"1min":1.7490000000000001,"5min":1.8029999999999999,"15min":2.044},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.427,"15min":1.415},"min":{"1min":1.085,"5min":1.038,"15min":0.995},"max":{"1min":1.7769999999999999,"5min":2.0489999999999999,"15min":2.706},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4710000000000001,"15min":1.4430000000000001},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":0.84699999999999998},"max":{"1min":1.911,"5min":1.9390000000000001,"15min":2.3439999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5840000000000001,"15min":1.5580000000000001},"min":{"1min":1.323,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.379,"5min":2.379,"15min":2.4660000000000002},"last":0.81899999999999995}]},{"osd":6,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.6060000000000001,"15min":1.585},"min":{"1min":1.3580000000000001,"5min":1.1080000000000001,"15min":1.0960000000000001},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.177},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.6100000000000001,"15min":1.5680000000000001},"min":{"1min":1.093,"5min":1.093,"15min":0.998},"max":{"1min":1.984,"5min":2.1299999999999999,"15min":2.4860000000000002},"last":0.88800000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.706,"15min":1.6739999999999999},"min":{"1min":1.417,"5min":1.2430000000000001,"15min":0.97599999999999998},"max":{"1min":2.294,"5min":2.3660000000000001,"15min":2.5379999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.6950000000000001,"15min":1.6779999999999999},"min":{"1min":1.3620000000000001,"5min":1.282,"15min":1.016},"max":{"1min":1.9990000000000001,"5min":2.1019999999999999,"15min":2.2930000000000001},"last":0.91300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60320,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311520,"statfs":{"total":96636764160,"available":96574996480,"internally_reserved":0,"allocated":2830336,"data_stored":1680922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.577,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.665},{"interface":"front","average":{"1min":1.141,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.72699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6830000000000001,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.94999999999999996}]},{"osd":2,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1739999999999999,"15min":1.1499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.446,"5min":3.375,"15min":3.375},"last":1.113},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2569999999999999,"15min":1.2629999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.681},"last":1.0680000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3819999999999999,"15min":1.3660000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.708,"5min":1.823,"15min":2.1899999999999999},"last":1.234},{"interface":"front","average":{"1min":1.383,"5min":1.425,"15min":1.4279999999999999},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.93999999999999995},"max":{"1min":1.792,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.1830000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.3420000000000001},"min":{"1min":1.0029999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.675,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.488},{"interface":"front","average":{"1min":1.534,"5min":1.5309999999999999,"15min":1.498},"min":{"1min":1.002,"5min":1.002,"15min":0.999},"max":{"1min":2.008,"5min":2.9689999999999999,"15min":2.9689999999999999},"last":1.792}]},{"osd":5,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.401,"15min":1.4119999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.873,"5min":3.077,"15min":3.077},"last":1.609},{"interface":"front","average":{"1min":1.611,"5min":1.621,"15min":1.6020000000000001},"min":{"1min":1.194,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":2.028,"5min":2.468,"15min":2.468},"last":1.5760000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.6799999999999999,"15min":1.681},"min":{"1min":1.2769999999999999,"5min":1.194,"15min":1.147},"max":{"1min":2.1349999999999998,"5min":2.3479999999999999,"15min":2.5270000000000001},"last":1.411},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.548,"15min":1.554},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.875,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.292}]},{"osd":7,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.675,"15min":1.655},"min":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"max":{"1min":1.9399999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.361},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6619999999999999,"15min":1.655},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":1.99,"5min":3.645,"15min":3.645},"last":1.744}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:57:49.415 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:57:49.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.677+0000 7fc21e2a3640 1 Processor -- start 2023-10-14T17:57:49.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.677+0000 7fc21e2a3640 1 -- start start 2023-10-14T17:57:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21e2a3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 0x7fc2180c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21e2a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc21815dde0 0x7fc2180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21e2a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc218160a90 con 0x7fc2180c9230 2023-10-14T17:57:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21e2a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2180cdec0 con 0x7fc21815dde0 2023-10-14T17:57:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21e2a3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2180cd900 con 0x7fc21815c9f0 2023-10-14T17:57:49.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21daa2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc21815dde0 0x7fc2180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21daa2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc21815dde0 0x7fc2180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40994/0 (socket says 172.21.15.29:40994) 2023-10-14T17:57:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.679+0000 7fc21daa2640 1 -- 172.21.15.29:0/3091981581 learned_addr learned my addr 172.21.15.29:0/3091981581 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21d2a1640 1 --2- 172.21.15.29:0/3091981581 >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 0x7fc2180c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21daa2640 1 -- 172.21.15.29:0/3091981581 >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 msgr2=0x7fc2180c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:49.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21daa2640 1 --2- 172.21.15.29:0/3091981581 >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 0x7fc2180c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21daa2640 1 -- 172.21.15.29:0/3091981581 >> v1:172.21.15.29:6789/0 conn(0x7fc2180c9230 legacy=0x7fc2180709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21daa2640 1 -- 172.21.15.29:0/3091981581 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2180cdbe0 con 0x7fc21815dde0 2023-10-14T17:57:49.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.680+0000 7fc21d2a1640 1 --2- 172.21.15.29:0/3091981581 >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 0x7fc2180c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:57:49.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.682+0000 7fc21daa2640 1 --2- 172.21.15.29:0/3091981581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21815dde0 0x7fc2180ac890 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fc2180ad4e0 tx=0x7fc21002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7845e20586dde9b4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.682+0000 7fc20ffff640 1 -- 172.21.15.29:0/3091981581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21002cb20 con 0x7fc21815dde0 2023-10-14T17:57:49.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.682+0000 7fc20ffff640 1 -- 172.21.15.29:0/3091981581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21002cc80 con 0x7fc21815dde0 2023-10-14T17:57:49.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.682+0000 7fc20ffff640 1 -- 172.21.15.29:0/3091981581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2100343f0 con 0x7fc21815dde0 2023-10-14T17:57:49.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 -- 172.21.15.29:0/3091981581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21815dde0 msgr2=0x7fc2180ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/3091981581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21815dde0 0x7fc2180ac890 secure :-1 s=READY pgs=1988 cs=0 l=1 rev1=1 crypto rx=0x7fc2180ad4e0 tx=0x7fc21002bf80 comp rx=0 tx=0).stop 2023-10-14T17:57:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 -- 172.21.15.29:0/3091981581 shutdown_connections 2023-10-14T17:57:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/3091981581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc21815dde0 0x7fc2180ac890 unknown :-1 s=CLOSED pgs=1988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/3091981581 >> v2:172.21.15.186:3300/0 conn(0x7fc21815c9f0 0x7fc2180c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 -- 172.21.15.29:0/3091981581 >> 172.21.15.29:0/3091981581 conn(0x7fc2180ce6a0 msgr2=0x7fc2180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 -- 172.21.15.29:0/3091981581 shutdown_connections 2023-10-14T17:57:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.683+0000 7fc21e2a3640 1 -- 172.21.15.29:0/3091981581 wait complete. 2023-10-14T17:57:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.685+0000 7fc21e2a3640 1 Processor -- start 2023-10-14T17:57:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.685+0000 7fc21e2a3640 1 -- start start 2023-10-14T17:57:49.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc21815c9f0 0x7fc2180c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc21815dde0 0x7fc2180c2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc21806b650 con 0x7fc2180c9230 2023-10-14T17:57:49.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180c54b0 con 0x7fc21815c9f0 2023-10-14T17:57:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21e2a3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2180c51d0 con 0x7fc21815dde0 2023-10-14T17:57:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21d2a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21d2a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41006/0 (socket says 172.21.15.29:41006) 2023-10-14T17:57:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.686+0000 7fc21d2a1640 1 -- 172.21.15.29:0/2530254841 learned_addr learned my addr 172.21.15.29:0/2530254841 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21caa0640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc21815c9f0 0x7fc2180c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21daa2640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc21815dde0 0x7fc2180c2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 -- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc21815dde0 msgr2=0x7fc2180c2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc21815dde0 0x7fc2180c2710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 -- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc21815c9f0 msgr2=0x7fc2180c21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc21815c9f0 0x7fc2180c21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 -- 172.21.15.29:0/2530254841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc210009b30 con 0x7fc2180c9230 2023-10-14T17:57:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.687+0000 7fc21d2a1640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fc21400e730 tx=0x7fc21400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.688+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc214004230 con 0x7fc2180c9230 2023-10-14T17:57:49.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.688+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc214004390 con 0x7fc2180c9230 2023-10-14T17:57:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.688+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc21400c8f0 con 0x7fc2180c9230 2023-10-14T17:57:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.688+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc21806af90 con 0x7fc2180c9230 2023-10-14T17:57:49.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.688+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2181f53c0 con 0x7fc2180c9230 2023-10-14T17:57:49.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.689+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc21806adf0 con 0x7fc2180c9230 2023-10-14T17:57:49.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.690+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc21400ca50 con 0x7fc2180c9230 2023-10-14T17:57:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.691+0000 7fc20dffb640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 0x7fc1fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.691+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc2140895e0 con 0x7fc2180c9230 2023-10-14T17:57:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.691+0000 7fc21caa0640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 0x7fc1fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:49.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.692+0000 7fc21caa0640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 0x7fc1fc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2180c3f40 tx=0x7fc208009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:49.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.696+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc21806adf0 con 0x7fc2180c9230 2023-10-14T17:57:49.851 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T17:57:49.849+0000 7f6fd4feb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:57:49.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.889+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 --> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc2180704c0 con 0x7fc1fc06af90 2023-10-14T17:57:49.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.912+0000 7fc20dffb640 1 -- 172.21.15.29:0/2530254841 <== mgr.7201 v2:172.21.15.29:6832/2441704285 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286944 (secure 0 0 0) 0x7fc208001c70 con 0x7fc1fc06af90 2023-10-14T17:57:49.919 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:49.922 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:57:49.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 msgr2=0x7fc1fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 0x7fc1fc06d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc2180c3f40 tx=0x7fc208009040 comp rx=0 tx=0).stop 2023-10-14T17:57:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 msgr2=0x7fc2181f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:49.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 secure :-1 s=READY pgs=1989 cs=0 l=1 rev1=1 crypto rx=0x7fc21400e730 tx=0x7fc21400ebf0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 shutdown_connections 2023-10-14T17:57:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc1fc06af90 0x7fc1fc06d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc21815dde0 0x7fc2180c2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.917+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc21815c9f0 0x7fc2180c21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.918+0000 7fc21e2a3640 1 --2- 172.21.15.29:0/2530254841 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc2180c9230 0x7fc2181f33f0 unknown :-1 s=CLOSED pgs=1989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.918+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 >> 172.21.15.29:0/2530254841 conn(0x7fc2180ce6a0 msgr2=0x7fc2180708e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.918+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 shutdown_connections 2023-10-14T17:57:49.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:49.918+0000 7fc21e2a3640 1 -- 172.21.15.29:0/2530254841 wait complete. 2023-10-14T17:57:49.938 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T17:57:48.305696+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":483,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":309,"ondisk_log_size":309,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":397108,"kb_used_data":18568,"kb_used_omap":12,"kb_used_meta":378419,"kb_avail":754577612,"statfs":{"total":773094113280,"available":772687474688,"internally_reserved":0,"allocated":19013632,"data_stored":9896820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":387501624},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.432448"},"pg_stats":[{"pgid":"21.3f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939922+0000","last_change":"2023-10-14T17:57:44.935583+0000","last_active":"2023-10-14T17:57:45.939922+0000","last_peered":"2023-10-14T17:57:45.939922+0000","last_clean":"2023-10-14T17:57:45.939922+0000","last_became_active":"2023-10-14T17:57:44.934176+0000","last_became_peered":"2023-10-14T17:57:44.934176+0000","last_unstale":"2023-10-14T17:57:45.939922+0000","last_undegraded":"2023-10-14T17:57:45.939922+0000","last_fullsized":"2023-10-14T17:57:45.939922+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:29.105298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.3e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095805+0000","last_change":"2023-10-14T17:57:44.944153+0000","last_active":"2023-10-14T17:57:46.095805+0000","last_peered":"2023-10-14T17:57:46.095805+0000","last_clean":"2023-10-14T17:57:46.095805+0000","last_became_active":"2023-10-14T17:57:44.943851+0000","last_became_peered":"2023-10-14T17:57:44.943851+0000","last_unstale":"2023-10-14T17:57:46.095805+0000","last_undegraded":"2023-10-14T17:57:46.095805+0000","last_fullsized":"2023-10-14T17:57:46.095805+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:39.388384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.3d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943878+0000","last_change":"2023-10-14T17:57:44.941256+0000","last_active":"2023-10-14T17:57:45.943878+0000","last_peered":"2023-10-14T17:57:45.943878+0000","last_clean":"2023-10-14T17:57:45.943878+0000","last_became_active":"2023-10-14T17:57:44.940968+0000","last_became_peered":"2023-10-14T17:57:44.940968+0000","last_unstale":"2023-10-14T17:57:45.943878+0000","last_undegraded":"2023-10-14T17:57:45.943878+0000","last_fullsized":"2023-10-14T17:57:45.943878+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:56:57.242181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.3c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944594+0000","last_change":"2023-10-14T17:57:44.944893+0000","last_active":"2023-10-14T17:57:45.944594+0000","last_peered":"2023-10-14T17:57:45.944594+0000","last_clean":"2023-10-14T17:57:45.944594+0000","last_became_active":"2023-10-14T17:57:44.943900+0000","last_became_peered":"2023-10-14T17:57:44.943900+0000","last_unstale":"2023-10-14T17:57:45.944594+0000","last_undegraded":"2023-10-14T17:57:45.944594+0000","last_fullsized":"2023-10-14T17:57:45.944594+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:53:29.105298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.3b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097453+0000","last_change":"2023-10-14T17:57:44.940092+0000","last_active":"2023-10-14T17:57:46.097453+0000","last_peered":"2023-10-14T17:57:46.097453+0000","last_clean":"2023-10-14T17:57:46.097453+0000","last_became_active":"2023-10-14T17:57:44.939718+0000","last_became_peered":"2023-10-14T17:57:44.939718+0000","last_unstale":"2023-10-14T17:57:46.097453+0000","last_undegraded":"2023-10-14T17:57:46.097453+0000","last_fullsized":"2023-10-14T17:57:46.097453+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:42.280083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.3a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939017+0000","last_change":"2023-10-14T17:57:44.947077+0000","last_active":"2023-10-14T17:57:45.939017+0000","last_peered":"2023-10-14T17:57:45.939017+0000","last_clean":"2023-10-14T17:57:45.939017+0000","last_became_active":"2023-10-14T17:57:44.946047+0000","last_became_peered":"2023-10-14T17:57:44.946047+0000","last_unstale":"2023-10-14T17:57:45.939017+0000","last_undegraded":"2023-10-14T17:57:45.939017+0000","last_fullsized":"2023-10-14T17:57:45.939017+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:56.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.39","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096290+0000","last_change":"2023-10-14T17:57:44.947365+0000","last_active":"2023-10-14T17:57:46.096290+0000","last_peered":"2023-10-14T17:57:46.096290+0000","last_clean":"2023-10-14T17:57:46.096290+0000","last_became_active":"2023-10-14T17:57:44.946888+0000","last_became_peered":"2023-10-14T17:57:44.946888+0000","last_unstale":"2023-10-14T17:57:46.096290+0000","last_undegraded":"2023-10-14T17:57:46.096290+0000","last_fullsized":"2023-10-14T17:57:46.096290+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:17.136325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.38","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938864+0000","last_change":"2023-10-14T17:57:44.941285+0000","last_active":"2023-10-14T17:57:45.938864+0000","last_peered":"2023-10-14T17:57:45.938864+0000","last_clean":"2023-10-14T17:57:45.938864+0000","last_became_active":"2023-10-14T17:57:44.940880+0000","last_became_peered":"2023-10-14T17:57:44.940880+0000","last_unstale":"2023-10-14T17:57:45.938864+0000","last_undegraded":"2023-10-14T17:57:45.938864+0000","last_fullsized":"2023-10-14T17:57:45.938864+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:36:22.091999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.37","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096320+0000","last_change":"2023-10-14T17:57:44.955693+0000","last_active":"2023-10-14T17:57:46.096320+0000","last_peered":"2023-10-14T17:57:46.096320+0000","last_clean":"2023-10-14T17:57:46.096320+0000","last_became_active":"2023-10-14T17:57:44.955386+0000","last_became_peered":"2023-10-14T17:57:44.955386+0000","last_unstale":"2023-10-14T17:57:46.096320+0000","last_undegraded":"2023-10-14T17:57:46.096320+0000","last_fullsized":"2023-10-14T17:57:46.096320+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:55:57.885140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.36","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100750+0000","last_change":"2023-10-14T17:57:44.934667+0000","last_active":"2023-10-14T17:57:46.100750+0000","last_peered":"2023-10-14T17:57:46.100750+0000","last_clean":"2023-10-14T17:57:46.100750+0000","last_became_active":"2023-10-14T17:57:44.934387+0000","last_became_peered":"2023-10-14T17:57:44.934387+0000","last_unstale":"2023-10-14T17:57:46.100750+0000","last_undegraded":"2023-10-14T17:57:46.100750+0000","last_fullsized":"2023-10-14T17:57:46.100750+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:36:45.389913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.35","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939250+0000","last_change":"2023-10-14T17:57:44.949499+0000","last_active":"2023-10-14T17:57:45.939250+0000","last_peered":"2023-10-14T17:57:45.939250+0000","last_clean":"2023-10-14T17:57:45.939250+0000","last_became_active":"2023-10-14T17:57:44.945736+0000","last_became_peered":"2023-10-14T17:57:44.945736+0000","last_unstale":"2023-10-14T17:57:45.939250+0000","last_undegraded":"2023-10-14T17:57:45.939250+0000","last_fullsized":"2023-10-14T17:57:45.939250+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:44:21.658770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.34","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100250+0000","last_change":"2023-10-14T17:57:44.933161+0000","last_active":"2023-10-14T17:57:46.100250+0000","last_peered":"2023-10-14T17:57:46.100250+0000","last_clean":"2023-10-14T17:57:46.100250+0000","last_became_active":"2023-10-14T17:57:44.932891+0000","last_became_peered":"2023-10-14T17:57:44.932891+0000","last_unstale":"2023-10-14T17:57:46.100250+0000","last_undegraded":"2023-10-14T17:57:46.100250+0000","last_fullsized":"2023-10-14T17:57:46.100250+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:58:43.303871+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.33","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098172+0000","last_change":"2023-10-14T17:57:44.937565+0000","last_active":"2023-10-14T17:57:46.098172+0000","last_peered":"2023-10-14T17:57:46.098172+0000","last_clean":"2023-10-14T17:57:46.098172+0000","last_became_active":"2023-10-14T17:57:44.935756+0000","last_became_peered":"2023-10-14T17:57:44.935756+0000","last_unstale":"2023-10-14T17:57:46.098172+0000","last_undegraded":"2023-10-14T17:57:46.098172+0000","last_fullsized":"2023-10-14T17:57:46.098172+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:36.393884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.32","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941983+0000","last_change":"2023-10-14T17:57:44.942046+0000","last_active":"2023-10-14T17:57:45.941983+0000","last_peered":"2023-10-14T17:57:45.941983+0000","last_clean":"2023-10-14T17:57:45.941983+0000","last_became_active":"2023-10-14T17:57:44.938232+0000","last_became_peered":"2023-10-14T17:57:44.938232+0000","last_unstale":"2023-10-14T17:57:45.941983+0000","last_undegraded":"2023-10-14T17:57:45.941983+0000","last_fullsized":"2023-10-14T17:57:45.941983+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:23.748401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.31","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098555+0000","last_change":"2023-10-14T17:57:44.939570+0000","last_active":"2023-10-14T17:57:46.098555+0000","last_peered":"2023-10-14T17:57:46.098555+0000","last_clean":"2023-10-14T17:57:46.098555+0000","last_became_active":"2023-10-14T17:57:44.939289+0000","last_became_peered":"2023-10-14T17:57:44.939289+0000","last_unstale":"2023-10-14T17:57:46.098555+0000","last_undegraded":"2023-10-14T17:57:46.098555+0000","last_fullsized":"2023-10-14T17:57:46.098555+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:45:14.103101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.30","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095924+0000","last_change":"2023-10-14T17:57:44.938980+0000","last_active":"2023-10-14T17:57:46.095924+0000","last_peered":"2023-10-14T17:57:46.095924+0000","last_clean":"2023-10-14T17:57:46.095924+0000","last_became_active":"2023-10-14T17:57:44.938633+0000","last_became_peered":"2023-10-14T17:57:44.938633+0000","last_unstale":"2023-10-14T17:57:46.095924+0000","last_undegraded":"2023-10-14T17:57:46.095924+0000","last_fullsized":"2023-10-14T17:57:46.095924+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:31:38.656878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101234+0000","last_change":"2023-10-14T17:57:44.938646+0000","last_active":"2023-10-14T17:57:46.101234+0000","last_peered":"2023-10-14T17:57:46.101234+0000","last_clean":"2023-10-14T17:57:46.101234+0000","last_became_active":"2023-10-14T17:57:44.938172+0000","last_became_peered":"2023-10-14T17:57:44.938172+0000","last_unstale":"2023-10-14T17:57:46.101234+0000","last_undegraded":"2023-10-14T17:57:46.101234+0000","last_fullsized":"2023-10-14T17:57:46.101234+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:12.657175+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944125+0000","last_change":"2023-10-14T17:57:44.948395+0000","last_active":"2023-10-14T17:57:45.944125+0000","last_peered":"2023-10-14T17:57:45.944125+0000","last_clean":"2023-10-14T17:57:45.944125+0000","last_became_active":"2023-10-14T17:57:44.948095+0000","last_became_peered":"2023-10-14T17:57:44.948095+0000","last_unstale":"2023-10-14T17:57:45.944125+0000","last_undegraded":"2023-10-14T17:57:45.944125+0000","last_fullsized":"2023-10-14T17:57:45.944125+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:15:17.136325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.2d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939642+0000","last_change":"2023-10-14T17:57:44.940533+0000","last_active":"2023-10-14T17:57:45.939642+0000","last_peered":"2023-10-14T17:57:45.939642+0000","last_clean":"2023-10-14T17:57:45.939642+0000","last_became_active":"2023-10-14T17:57:44.940140+0000","last_became_peered":"2023-10-14T17:57:44.940140+0000","last_unstale":"2023-10-14T17:57:45.939642+0000","last_undegraded":"2023-10-14T17:57:45.939642+0000","last_fullsized":"2023-10-14T17:57:45.939642+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:58:49.126002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096604+0000","last_change":"2023-10-14T17:57:44.949412+0000","last_active":"2023-10-14T17:57:46.096604+0000","last_peered":"2023-10-14T17:57:46.096604+0000","last_clean":"2023-10-14T17:57:46.096604+0000","last_became_active":"2023-10-14T17:57:44.949129+0000","last_became_peered":"2023-10-14T17:57:44.949129+0000","last_unstale":"2023-10-14T17:57:46.096604+0000","last_undegraded":"2023-10-14T17:57:46.096604+0000","last_fullsized":"2023-10-14T17:57:46.096604+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:16:54.442674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.2b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100102+0000","last_change":"2023-10-14T17:57:44.946438+0000","last_active":"2023-10-14T17:57:46.100102+0000","last_peered":"2023-10-14T17:57:46.100102+0000","last_clean":"2023-10-14T17:57:46.100102+0000","last_became_active":"2023-10-14T17:57:44.946026+0000","last_became_peered":"2023-10-14T17:57:44.946026+0000","last_unstale":"2023-10-14T17:57:46.100102+0000","last_undegraded":"2023-10-14T17:57:46.100102+0000","last_fullsized":"2023-10-14T17:57:46.100102+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:45.713924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.2a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942336+0000","last_change":"2023-10-14T17:57:44.944935+0000","last_active":"2023-10-14T17:57:45.942336+0000","last_peered":"2023-10-14T17:57:45.942336+0000","last_clean":"2023-10-14T17:57:45.942336+0000","last_became_active":"2023-10-14T17:57:44.944668+0000","last_became_peered":"2023-10-14T17:57:44.944668+0000","last_unstale":"2023-10-14T17:57:45.942336+0000","last_undegraded":"2023-10-14T17:57:45.942336+0000","last_fullsized":"2023-10-14T17:57:45.942336+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:44:21.658770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.29","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943751+0000","last_change":"2023-10-14T17:57:44.941144+0000","last_active":"2023-10-14T17:57:45.943751+0000","last_peered":"2023-10-14T17:57:45.943751+0000","last_clean":"2023-10-14T17:57:45.943751+0000","last_became_active":"2023-10-14T17:57:44.940795+0000","last_became_peered":"2023-10-14T17:57:44.940795+0000","last_unstale":"2023-10-14T17:57:45.943751+0000","last_undegraded":"2023-10-14T17:57:45.943751+0000","last_fullsized":"2023-10-14T17:57:45.943751+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:58:49.126002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.28","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097935+0000","last_change":"2023-10-14T17:57:44.935468+0000","last_active":"2023-10-14T17:57:46.097935+0000","last_peered":"2023-10-14T17:57:46.097935+0000","last_clean":"2023-10-14T17:57:46.097935+0000","last_became_active":"2023-10-14T17:57:44.935192+0000","last_became_peered":"2023-10-14T17:57:44.935192+0000","last_unstale":"2023-10-14T17:57:46.097935+0000","last_undegraded":"2023-10-14T17:57:46.097935+0000","last_fullsized":"2023-10-14T17:57:46.097935+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:56.513623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.27","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097460+0000","last_change":"2023-10-14T17:57:44.939575+0000","last_active":"2023-10-14T17:57:46.097460+0000","last_peered":"2023-10-14T17:57:46.097460+0000","last_clean":"2023-10-14T17:57:46.097460+0000","last_became_active":"2023-10-14T17:57:44.939284+0000","last_became_peered":"2023-10-14T17:57:44.939284+0000","last_unstale":"2023-10-14T17:57:46.097460+0000","last_undegraded":"2023-10-14T17:57:46.097460+0000","last_fullsized":"2023-10-14T17:57:46.097460+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:37.156538+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.26","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939444+0000","last_change":"2023-10-14T17:57:44.946848+0000","last_active":"2023-10-14T17:57:45.939444+0000","last_peered":"2023-10-14T17:57:45.939444+0000","last_clean":"2023-10-14T17:57:45.939444+0000","last_became_active":"2023-10-14T17:57:44.946553+0000","last_became_peered":"2023-10-14T17:57:44.946553+0000","last_unstale":"2023-10-14T17:57:45.939444+0000","last_undegraded":"2023-10-14T17:57:45.939444+0000","last_fullsized":"2023-10-14T17:57:45.939444+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:53.587873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.25","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096958+0000","last_change":"2023-10-14T17:57:44.931218+0000","last_active":"2023-10-14T17:57:46.096958+0000","last_peered":"2023-10-14T17:57:46.096958+0000","last_clean":"2023-10-14T17:57:46.096958+0000","last_became_active":"2023-10-14T17:57:44.930875+0000","last_became_peered":"2023-10-14T17:57:44.930875+0000","last_unstale":"2023-10-14T17:57:46.096958+0000","last_undegraded":"2023-10-14T17:57:46.096958+0000","last_fullsized":"2023-10-14T17:57:46.096958+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:31.088464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.24","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096953+0000","last_change":"2023-10-14T17:57:44.946439+0000","last_active":"2023-10-14T17:57:46.096953+0000","last_peered":"2023-10-14T17:57:46.096953+0000","last_clean":"2023-10-14T17:57:46.096953+0000","last_became_active":"2023-10-14T17:57:44.944623+0000","last_became_peered":"2023-10-14T17:57:44.944623+0000","last_unstale":"2023-10-14T17:57:46.096953+0000","last_undegraded":"2023-10-14T17:57:46.096953+0000","last_fullsized":"2023-10-14T17:57:46.096953+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:45.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.23","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939118+0000","last_change":"2023-10-14T17:57:44.934088+0000","last_active":"2023-10-14T17:57:45.939118+0000","last_peered":"2023-10-14T17:57:45.939118+0000","last_clean":"2023-10-14T17:57:45.939118+0000","last_became_active":"2023-10-14T17:57:44.933768+0000","last_became_peered":"2023-10-14T17:57:44.933768+0000","last_unstale":"2023-10-14T17:57:45.939118+0000","last_undegraded":"2023-10-14T17:57:45.939118+0000","last_fullsized":"2023-10-14T17:57:45.939118+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:23.748401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.22","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942680+0000","last_change":"2023-10-14T17:57:44.952397+0000","last_active":"2023-10-14T17:57:45.942680+0000","last_peered":"2023-10-14T17:57:45.942680+0000","last_clean":"2023-10-14T17:57:45.942680+0000","last_became_active":"2023-10-14T17:57:44.952101+0000","last_became_peered":"2023-10-14T17:57:44.952101+0000","last_unstale":"2023-10-14T17:57:45.942680+0000","last_undegraded":"2023-10-14T17:57:45.942680+0000","last_fullsized":"2023-10-14T17:57:45.942680+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:53.587873+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.21","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095276+0000","last_change":"2023-10-14T17:57:44.945879+0000","last_active":"2023-10-14T17:57:46.095276+0000","last_peered":"2023-10-14T17:57:46.095276+0000","last_clean":"2023-10-14T17:57:46.095276+0000","last_became_active":"2023-10-14T17:57:44.945579+0000","last_became_peered":"2023-10-14T17:57:44.945579+0000","last_unstale":"2023-10-14T17:57:46.095276+0000","last_undegraded":"2023-10-14T17:57:46.095276+0000","last_fullsized":"2023-10-14T17:57:46.095276+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:09.356684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.20","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096055+0000","last_change":"2023-10-14T17:57:44.935651+0000","last_active":"2023-10-14T17:57:46.096055+0000","last_peered":"2023-10-14T17:57:46.096055+0000","last_clean":"2023-10-14T17:57:46.096055+0000","last_became_active":"2023-10-14T17:57:44.935384+0000","last_became_peered":"2023-10-14T17:57:44.935384+0000","last_unstale":"2023-10-14T17:57:46.096055+0000","last_undegraded":"2023-10-14T17:57:46.096055+0000","last_fullsized":"2023-10-14T17:57:46.096055+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:28.827875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100887+0000","last_change":"2023-10-14T17:57:44.934349+0000","last_active":"2023-10-14T17:57:46.100887+0000","last_peered":"2023-10-14T17:57:46.100887+0000","last_clean":"2023-10-14T17:57:46.100887+0000","last_became_active":"2023-10-14T17:57:44.933103+0000","last_became_peered":"2023-10-14T17:57:44.933103+0000","last_unstale":"2023-10-14T17:57:46.100887+0000","last_undegraded":"2023-10-14T17:57:46.100887+0000","last_fullsized":"2023-10-14T17:57:46.100887+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:05.086466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1f","version":"177'4","reported_seq":16,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100787+0000","last_change":"2023-10-14T17:57:43.920589+0000","last_active":"2023-10-14T17:57:46.100787+0000","last_peered":"2023-10-14T17:57:46.100787+0000","last_clean":"2023-10-14T17:57:46.100787+0000","last_became_active":"2023-10-14T17:57:43.919468+0000","last_became_peered":"2023-10-14T17:57:43.919468+0000","last_unstale":"2023-10-14T17:57:46.100787+0000","last_undegraded":"2023-10-14T17:57:46.100787+0000","last_fullsized":"2023-10-14T17:57:46.100787+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:32:18.321689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.1f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096485+0000","last_change":"2023-10-14T17:57:44.947807+0000","last_active":"2023-10-14T17:57:46.096485+0000","last_peered":"2023-10-14T17:57:46.096485+0000","last_clean":"2023-10-14T17:57:46.096485+0000","last_became_active":"2023-10-14T17:57:44.947530+0000","last_became_peered":"2023-10-14T17:57:44.947530+0000","last_unstale":"2023-10-14T17:57:46.096485+0000","last_undegraded":"2023-10-14T17:57:46.096485+0000","last_fullsized":"2023-10-14T17:57:46.096485+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:05:32.164095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1e","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095960+0000","last_change":"2023-10-14T17:57:43.916292+0000","last_active":"2023-10-14T17:57:46.095960+0000","last_peered":"2023-10-14T17:57:46.095960+0000","last_clean":"2023-10-14T17:57:46.095960+0000","last_became_active":"2023-10-14T17:57:43.916020+0000","last_became_peered":"2023-10-14T17:57:43.916020+0000","last_unstale":"2023-10-14T17:57:46.095960+0000","last_undegraded":"2023-10-14T17:57:46.095960+0000","last_fullsized":"2023-10-14T17:57:46.095960+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:07:38.917413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.1c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938923+0000","last_change":"2023-10-14T17:57:44.935623+0000","last_active":"2023-10-14T17:57:45.938923+0000","last_peered":"2023-10-14T17:57:45.938923+0000","last_clean":"2023-10-14T17:57:45.938923+0000","last_became_active":"2023-10-14T17:57:44.935332+0000","last_became_peered":"2023-10-14T17:57:44.935332+0000","last_unstale":"2023-10-14T17:57:45.938923+0000","last_undegraded":"2023-10-14T17:57:45.938923+0000","last_fullsized":"2023-10-14T17:57:45.938923+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:44.458283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.1d","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944397+0000","last_change":"2023-10-14T17:57:43.895545+0000","last_active":"2023-10-14T17:57:45.944397+0000","last_peered":"2023-10-14T17:57:45.944397+0000","last_clean":"2023-10-14T17:57:45.944397+0000","last_became_active":"2023-10-14T17:57:43.894949+0000","last_became_peered":"2023-10-14T17:57:43.894949+0000","last_unstale":"2023-10-14T17:57:45.944397+0000","last_undegraded":"2023-10-14T17:57:45.944397+0000","last_fullsized":"2023-10-14T17:57:45.944397+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:53.966578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.100649+0000","last_change":"2023-10-14T17:57:44.933332+0000","last_active":"2023-10-14T17:57:46.100649+0000","last_peered":"2023-10-14T17:57:46.100649+0000","last_clean":"2023-10-14T17:57:46.100649+0000","last_became_active":"2023-10-14T17:57:44.933048+0000","last_became_peered":"2023-10-14T17:57:44.933048+0000","last_unstale":"2023-10-14T17:57:46.100649+0000","last_undegraded":"2023-10-14T17:57:46.100649+0000","last_fullsized":"2023-10-14T17:57:46.100649+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:25:31.351576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1c","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944484+0000","last_change":"2023-10-14T17:57:43.894372+0000","last_active":"2023-10-14T17:57:45.944484+0000","last_peered":"2023-10-14T17:57:45.944484+0000","last_clean":"2023-10-14T17:57:45.944484+0000","last_became_active":"2023-10-14T17:57:43.893581+0000","last_became_peered":"2023-10-14T17:57:43.893581+0000","last_unstale":"2023-10-14T17:57:45.944484+0000","last_undegraded":"2023-10-14T17:57:45.944484+0000","last_fullsized":"2023-10-14T17:57:45.944484+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:49:43.443314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.1a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095676+0000","last_change":"2023-10-14T17:57:44.943279+0000","last_active":"2023-10-14T17:57:46.095676+0000","last_peered":"2023-10-14T17:57:46.095676+0000","last_clean":"2023-10-14T17:57:46.095676+0000","last_became_active":"2023-10-14T17:57:44.942986+0000","last_became_peered":"2023-10-14T17:57:44.942986+0000","last_unstale":"2023-10-14T17:57:46.095676+0000","last_undegraded":"2023-10-14T17:57:46.095676+0000","last_fullsized":"2023-10-14T17:57:46.095676+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:07.265177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.1b","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096452+0000","last_change":"2023-10-14T17:57:43.915673+0000","last_active":"2023-10-14T17:57:46.096452+0000","last_peered":"2023-10-14T17:57:46.096452+0000","last_clean":"2023-10-14T17:57:46.096452+0000","last_became_active":"2023-10-14T17:57:43.915378+0000","last_became_peered":"2023-10-14T17:57:43.915378+0000","last_unstale":"2023-10-14T17:57:46.096452+0000","last_undegraded":"2023-10-14T17:57:46.096452+0000","last_fullsized":"2023-10-14T17:57:46.096452+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:09.046290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.1b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942692+0000","last_change":"2023-10-14T17:57:44.941824+0000","last_active":"2023-10-14T17:57:45.942692+0000","last_peered":"2023-10-14T17:57:45.942692+0000","last_clean":"2023-10-14T17:57:45.942692+0000","last_became_active":"2023-10-14T17:57:44.941525+0000","last_became_peered":"2023-10-14T17:57:44.941525+0000","last_unstale":"2023-10-14T17:57:45.942692+0000","last_undegraded":"2023-10-14T17:57:45.942692+0000","last_fullsized":"2023-10-14T17:57:45.942692+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:18.533962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.1a","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097431+0000","last_change":"2023-10-14T17:57:43.917338+0000","last_active":"2023-10-14T17:57:46.097431+0000","last_peered":"2023-10-14T17:57:46.097431+0000","last_clean":"2023-10-14T17:57:46.097431+0000","last_became_active":"2023-10-14T17:57:43.915771+0000","last_became_peered":"2023-10-14T17:57:43.915771+0000","last_unstale":"2023-10-14T17:57:46.097431+0000","last_undegraded":"2023-10-14T17:57:46.097431+0000","last_fullsized":"2023-10-14T17:57:46.097431+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:13:13.808377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.18","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095064+0000","last_change":"2023-10-14T17:57:44.939702+0000","last_active":"2023-10-14T17:57:46.095064+0000","last_peered":"2023-10-14T17:57:46.095064+0000","last_clean":"2023-10-14T17:57:46.095064+0000","last_became_active":"2023-10-14T17:57:44.939415+0000","last_became_peered":"2023-10-14T17:57:44.939415+0000","last_unstale":"2023-10-14T17:57:46.095064+0000","last_undegraded":"2023-10-14T17:57:46.095064+0000","last_fullsized":"2023-10-14T17:57:46.095064+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:45:15.450800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.19","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096899+0000","last_change":"2023-10-14T17:57:43.917984+0000","last_active":"2023-10-14T17:57:46.096899+0000","last_peered":"2023-10-14T17:57:46.096899+0000","last_clean":"2023-10-14T17:57:46.096899+0000","last_became_active":"2023-10-14T17:57:43.916743+0000","last_became_peered":"2023-10-14T17:57:43.916743+0000","last_unstale":"2023-10-14T17:57:46.096899+0000","last_undegraded":"2023-10-14T17:57:46.096899+0000","last_fullsized":"2023-10-14T17:57:46.096899+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:41.265113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.19","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097468+0000","last_change":"2023-10-14T17:57:44.954518+0000","last_active":"2023-10-14T17:57:46.097468+0000","last_peered":"2023-10-14T17:57:46.097468+0000","last_clean":"2023-10-14T17:57:46.097468+0000","last_became_active":"2023-10-14T17:57:44.954200+0000","last_became_peered":"2023-10-14T17:57:44.954200+0000","last_unstale":"2023-10-14T17:57:46.097468+0000","last_undegraded":"2023-10-14T17:57:46.097468+0000","last_fullsized":"2023-10-14T17:57:46.097468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:10.061259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.18","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941551+0000","last_change":"2023-10-14T17:57:43.910673+0000","last_active":"2023-10-14T17:57:45.941551+0000","last_peered":"2023-10-14T17:57:45.941551+0000","last_clean":"2023-10-14T17:57:45.941551+0000","last_became_active":"2023-10-14T17:57:43.910307+0000","last_became_peered":"2023-10-14T17:57:43.910307+0000","last_unstale":"2023-10-14T17:57:45.941551+0000","last_undegraded":"2023-10-14T17:57:45.941551+0000","last_fullsized":"2023-10-14T17:57:45.941551+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:08.304627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"21.16","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098294+0000","last_change":"2023-10-14T17:57:44.942967+0000","last_active":"2023-10-14T17:57:46.098294+0000","last_peered":"2023-10-14T17:57:46.098294+0000","last_clean":"2023-10-14T17:57:46.098294+0000","last_became_active":"2023-10-14T17:57:44.942683+0000","last_became_peered":"2023-10-14T17:57:44.942683+0000","last_unstale":"2023-10-14T17:57:46.098294+0000","last_undegraded":"2023-10-14T17:57:46.098294+0000","last_fullsized":"2023-10-14T17:57:46.098294+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:45.745017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.17","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096588+0000","last_change":"2023-10-14T17:57:43.915160+0000","last_active":"2023-10-14T17:57:46.096588+0000","last_peered":"2023-10-14T17:57:46.096588+0000","last_clean":"2023-10-14T17:57:46.096588+0000","last_became_active":"2023-10-14T17:57:43.914869+0000","last_became_peered":"2023-10-14T17:57:43.914869+0000","last_unstale":"2023-10-14T17:57:46.096588+0000","last_undegraded":"2023-10-14T17:57:46.096588+0000","last_fullsized":"2023-10-14T17:57:46.096588+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:44:46.412740+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.17","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101002+0000","last_change":"2023-10-14T17:57:44.938449+0000","last_active":"2023-10-14T17:57:46.101002+0000","last_peered":"2023-10-14T17:57:46.101002+0000","last_clean":"2023-10-14T17:57:46.101002+0000","last_became_active":"2023-10-14T17:57:44.937826+0000","last_became_peered":"2023-10-14T17:57:44.937826+0000","last_unstale":"2023-10-14T17:57:46.101002+0000","last_undegraded":"2023-10-14T17:57:46.101002+0000","last_fullsized":"2023-10-14T17:57:46.101002+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:12.935374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.16","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101298+0000","last_change":"2023-10-14T17:57:43.919092+0000","last_active":"2023-10-14T17:57:46.101298+0000","last_peered":"2023-10-14T17:57:46.101298+0000","last_clean":"2023-10-14T17:57:46.101298+0000","last_became_active":"2023-10-14T17:57:43.918508+0000","last_became_peered":"2023-10-14T17:57:43.918508+0000","last_unstale":"2023-10-14T17:57:46.101298+0000","last_undegraded":"2023-10-14T17:57:46.101298+0000","last_fullsized":"2023-10-14T17:57:46.101298+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:23:54.726700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.15","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097091+0000","last_change":"2023-10-14T17:57:44.940629+0000","last_active":"2023-10-14T17:57:46.097091+0000","last_peered":"2023-10-14T17:57:46.097091+0000","last_clean":"2023-10-14T17:57:46.097091+0000","last_became_active":"2023-10-14T17:57:44.940364+0000","last_became_peered":"2023-10-14T17:57:44.940364+0000","last_unstale":"2023-10-14T17:57:46.097091+0000","last_undegraded":"2023-10-14T17:57:46.097091+0000","last_fullsized":"2023-10-14T17:57:46.097091+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:39.211044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.14","version":"177'6","reported_seq":18,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097061+0000","last_change":"2023-10-14T17:57:43.913792+0000","last_active":"2023-10-14T17:57:46.097061+0000","last_peered":"2023-10-14T17:57:46.097061+0000","last_clean":"2023-10-14T17:57:46.097061+0000","last_became_active":"2023-10-14T17:57:43.913466+0000","last_became_peered":"2023-10-14T17:57:43.913466+0000","last_unstale":"2023-10-14T17:57:46.097061+0000","last_undegraded":"2023-10-14T17:57:46.097061+0000","last_fullsized":"2023-10-14T17:57:46.097061+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:44.730048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.12","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942761+0000","last_change":"2023-10-14T17:57:44.937036+0000","last_active":"2023-10-14T17:57:45.942761+0000","last_peered":"2023-10-14T17:57:45.942761+0000","last_clean":"2023-10-14T17:57:45.942761+0000","last_became_active":"2023-10-14T17:57:44.936196+0000","last_became_peered":"2023-10-14T17:57:44.936196+0000","last_unstale":"2023-10-14T17:57:45.942761+0000","last_undegraded":"2023-10-14T17:57:45.942761+0000","last_fullsized":"2023-10-14T17:57:45.942761+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:36.393884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.13","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942592+0000","last_change":"2023-10-14T17:57:43.904483+0000","last_active":"2023-10-14T17:57:45.942592+0000","last_peered":"2023-10-14T17:57:45.942592+0000","last_clean":"2023-10-14T17:57:45.942592+0000","last_became_active":"2023-10-14T17:57:43.901542+0000","last_became_peered":"2023-10-14T17:57:43.901542+0000","last_unstale":"2023-10-14T17:57:45.942592+0000","last_undegraded":"2023-10-14T17:57:45.942592+0000","last_fullsized":"2023-10-14T17:57:45.942592+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:06:39.771385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.13","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095712+0000","last_change":"2023-10-14T17:57:44.942380+0000","last_active":"2023-10-14T17:57:46.095712+0000","last_peered":"2023-10-14T17:57:46.095712+0000","last_clean":"2023-10-14T17:57:46.095712+0000","last_became_active":"2023-10-14T17:57:44.942074+0000","last_became_peered":"2023-10-14T17:57:44.942074+0000","last_unstale":"2023-10-14T17:57:46.095712+0000","last_undegraded":"2023-10-14T17:57:46.095712+0000","last_fullsized":"2023-10-14T17:57:46.095712+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:32:19.336658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.12","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942738+0000","last_change":"2023-10-14T17:57:43.915705+0000","last_active":"2023-10-14T17:57:45.942738+0000","last_peered":"2023-10-14T17:57:45.942738+0000","last_clean":"2023-10-14T17:57:45.942738+0000","last_became_active":"2023-10-14T17:57:43.915382+0000","last_became_peered":"2023-10-14T17:57:43.915382+0000","last_unstale":"2023-10-14T17:57:45.942738+0000","last_undegraded":"2023-10-14T17:57:45.942738+0000","last_fullsized":"2023-10-14T17:57:45.942738+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:11.422627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.10","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097340+0000","last_change":"2023-10-14T17:57:44.940993+0000","last_active":"2023-10-14T17:57:46.097340+0000","last_peered":"2023-10-14T17:57:46.097340+0000","last_clean":"2023-10-14T17:57:46.097340+0000","last_became_active":"2023-10-14T17:57:44.939813+0000","last_became_peered":"2023-10-14T17:57:44.939813+0000","last_unstale":"2023-10-14T17:57:46.097340+0000","last_undegraded":"2023-10-14T17:57:46.097340+0000","last_fullsized":"2023-10-14T17:57:46.097340+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:02.750348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.11","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.938698+0000","last_change":"2023-10-14T17:57:43.896216+0000","last_active":"2023-10-14T17:57:45.938698+0000","last_peered":"2023-10-14T17:57:45.938698+0000","last_clean":"2023-10-14T17:57:45.938698+0000","last_became_active":"2023-10-14T17:57:43.895897+0000","last_became_peered":"2023-10-14T17:57:43.895897+0000","last_unstale":"2023-10-14T17:57:45.938698+0000","last_undegraded":"2023-10-14T17:57:45.938698+0000","last_fullsized":"2023-10-14T17:57:45.938698+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:28.765717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.11","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942949+0000","last_change":"2023-10-14T17:57:44.924605+0000","last_active":"2023-10-14T17:57:45.942949+0000","last_peered":"2023-10-14T17:57:45.942949+0000","last_clean":"2023-10-14T17:57:45.942949+0000","last_became_active":"2023-10-14T17:57:44.923940+0000","last_became_peered":"2023-10-14T17:57:44.923940+0000","last_unstale":"2023-10-14T17:57:45.942949+0000","last_undegraded":"2023-10-14T17:57:45.942949+0000","last_fullsized":"2023-10-14T17:57:45.942949+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:45.745017+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.10","version":"177'8","reported_seq":20,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096145+0000","last_change":"2023-10-14T17:57:43.918085+0000","last_active":"2023-10-14T17:57:46.096145+0000","last_peered":"2023-10-14T17:57:46.096145+0000","last_clean":"2023-10-14T17:57:46.096145+0000","last_became_active":"2023-10-14T17:57:43.917732+0000","last_became_peered":"2023-10-14T17:57:43.917732+0000","last_unstale":"2023-10-14T17:57:46.096145+0000","last_undegraded":"2023-10-14T17:57:46.096145+0000","last_fullsized":"2023-10-14T17:57:46.096145+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:08.057209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.e","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101126+0000","last_change":"2023-10-14T17:57:44.938894+0000","last_active":"2023-10-14T17:57:46.101126+0000","last_peered":"2023-10-14T17:57:46.101126+0000","last_clean":"2023-10-14T17:57:46.101126+0000","last_became_active":"2023-10-14T17:57:44.938607+0000","last_became_peered":"2023-10-14T17:57:44.938607+0000","last_unstale":"2023-10-14T17:57:46.101126+0000","last_undegraded":"2023-10-14T17:57:46.101126+0000","last_fullsized":"2023-10-14T17:57:46.101126+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:46:41.450998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.f","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097994+0000","last_change":"2023-10-14T17:57:43.916493+0000","last_active":"2023-10-14T17:57:46.097994+0000","last_peered":"2023-10-14T17:57:46.097994+0000","last_clean":"2023-10-14T17:57:46.097994+0000","last_became_active":"2023-10-14T17:57:43.916218+0000","last_became_peered":"2023-10-14T17:57:43.916218+0000","last_unstale":"2023-10-14T17:57:46.097994+0000","last_undegraded":"2023-10-14T17:57:46.097994+0000","last_fullsized":"2023-10-14T17:57:46.097994+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:10.713743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"21.f","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097629+0000","last_change":"2023-10-14T17:57:44.933787+0000","last_active":"2023-10-14T17:57:46.097629+0000","last_peered":"2023-10-14T17:57:46.097629+0000","last_clean":"2023-10-14T17:57:46.097629+0000","last_became_active":"2023-10-14T17:57:44.930651+0000","last_became_peered":"2023-10-14T17:57:44.930651+0000","last_unstale":"2023-10-14T17:57:46.097629+0000","last_undegraded":"2023-10-14T17:57:46.097629+0000","last_fullsized":"2023-10-14T17:57:46.097629+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:11:56.739105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.e","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096001+0000","last_change":"2023-10-14T17:57:43.919216+0000","last_active":"2023-10-14T17:57:46.096001+0000","last_peered":"2023-10-14T17:57:46.096001+0000","last_clean":"2023-10-14T17:57:46.096001+0000","last_became_active":"2023-10-14T17:57:43.918896+0000","last_became_peered":"2023-10-14T17:57:43.918896+0000","last_unstale":"2023-10-14T17:57:46.096001+0000","last_undegraded":"2023-10-14T17:57:46.096001+0000","last_fullsized":"2023-10-14T17:57:46.096001+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:00:50.427057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.c","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943668+0000","last_change":"2023-10-14T17:57:44.937920+0000","last_active":"2023-10-14T17:57:45.943668+0000","last_peered":"2023-10-14T17:57:45.943668+0000","last_clean":"2023-10-14T17:57:45.943668+0000","last_became_active":"2023-10-14T17:57:44.936557+0000","last_became_peered":"2023-10-14T17:57:44.936557+0000","last_unstale":"2023-10-14T17:57:45.943668+0000","last_undegraded":"2023-10-14T17:57:45.943668+0000","last_fullsized":"2023-10-14T17:57:45.943668+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:09.072178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.d","version":"177'1","reported_seq":13,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097280+0000","last_change":"2023-10-14T17:57:43.918617+0000","last_active":"2023-10-14T17:57:46.097280+0000","last_peered":"2023-10-14T17:57:46.097280+0000","last_clean":"2023-10-14T17:57:46.097280+0000","last_became_active":"2023-10-14T17:57:43.918349+0000","last_became_peered":"2023-10-14T17:57:43.918349+0000","last_unstale":"2023-10-14T17:57:46.097280+0000","last_undegraded":"2023-10-14T17:57:46.097280+0000","last_fullsized":"2023-10-14T17:57:46.097280+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:20.056835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"21.d","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939440+0000","last_change":"2023-10-14T17:57:44.946394+0000","last_active":"2023-10-14T17:57:45.939440+0000","last_peered":"2023-10-14T17:57:45.939440+0000","last_clean":"2023-10-14T17:57:45.939440+0000","last_became_active":"2023-10-14T17:57:44.945236+0000","last_became_peered":"2023-10-14T17:57:44.945236+0000","last_unstale":"2023-10-14T17:57:45.939440+0000","last_undegraded":"2023-10-14T17:57:45.939440+0000","last_fullsized":"2023-10-14T17:57:45.939440+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:18.533962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.c","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096060+0000","last_change":"2023-10-14T17:57:43.912578+0000","last_active":"2023-10-14T17:57:46.096060+0000","last_peered":"2023-10-14T17:57:46.096060+0000","last_clean":"2023-10-14T17:57:46.096060+0000","last_became_active":"2023-10-14T17:57:43.912293+0000","last_became_peered":"2023-10-14T17:57:43.912293+0000","last_unstale":"2023-10-14T17:57:46.096060+0000","last_undegraded":"2023-10-14T17:57:46.096060+0000","last_fullsized":"2023-10-14T17:57:46.096060+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:16.097447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.a","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.943021+0000","last_change":"2023-10-14T17:57:44.952980+0000","last_active":"2023-10-14T17:57:45.943021+0000","last_peered":"2023-10-14T17:57:45.943021+0000","last_clean":"2023-10-14T17:57:45.943021+0000","last_became_active":"2023-10-14T17:57:44.952711+0000","last_became_peered":"2023-10-14T17:57:44.952711+0000","last_unstale":"2023-10-14T17:57:45.943021+0000","last_undegraded":"2023-10-14T17:57:45.943021+0000","last_fullsized":"2023-10-14T17:57:45.943021+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:06.615328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.b","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097145+0000","last_change":"2023-10-14T17:57:43.904455+0000","last_active":"2023-10-14T17:57:46.097145+0000","last_peered":"2023-10-14T17:57:46.097145+0000","last_clean":"2023-10-14T17:57:46.097145+0000","last_became_active":"2023-10-14T17:57:43.902019+0000","last_became_peered":"2023-10-14T17:57:43.902019+0000","last_unstale":"2023-10-14T17:57:46.097145+0000","last_undegraded":"2023-10-14T17:57:46.097145+0000","last_fullsized":"2023-10-14T17:57:46.097145+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:37.770797+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.b","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096969+0000","last_change":"2023-10-14T17:57:44.946938+0000","last_active":"2023-10-14T17:57:46.096969+0000","last_peered":"2023-10-14T17:57:46.096969+0000","last_clean":"2023-10-14T17:57:46.096969+0000","last_became_active":"2023-10-14T17:57:44.946394+0000","last_became_peered":"2023-10-14T17:57:44.946394+0000","last_unstale":"2023-10-14T17:57:46.096969+0000","last_undegraded":"2023-10-14T17:57:46.096969+0000","last_fullsized":"2023-10-14T17:57:46.096969+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:52:21.071804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.a","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.099072+0000","last_change":"2023-10-14T17:57:43.919522+0000","last_active":"2023-10-14T17:57:46.099072+0000","last_peered":"2023-10-14T17:57:46.099072+0000","last_clean":"2023-10-14T17:57:46.099072+0000","last_became_active":"2023-10-14T17:57:43.919109+0000","last_became_peered":"2023-10-14T17:57:43.919109+0000","last_unstale":"2023-10-14T17:57:46.099072+0000","last_undegraded":"2023-10-14T17:57:46.099072+0000","last_fullsized":"2023-10-14T17:57:46.099072+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:35.804250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.8","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941225+0000","last_change":"2023-10-14T17:57:44.939466+0000","last_active":"2023-10-14T17:57:45.941225+0000","last_peered":"2023-10-14T17:57:45.941225+0000","last_clean":"2023-10-14T17:57:45.941225+0000","last_became_active":"2023-10-14T17:57:44.938799+0000","last_became_peered":"2023-10-14T17:57:44.938799+0000","last_unstale":"2023-10-14T17:57:45.941225+0000","last_undegraded":"2023-10-14T17:57:45.941225+0000","last_fullsized":"2023-10-14T17:57:45.941225+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:18:56.213498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"20.9","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098901+0000","last_change":"2023-10-14T17:57:43.915810+0000","last_active":"2023-10-14T17:57:46.098901+0000","last_peered":"2023-10-14T17:57:46.098901+0000","last_clean":"2023-10-14T17:57:46.098901+0000","last_became_active":"2023-10-14T17:57:43.915552+0000","last_became_peered":"2023-10-14T17:57:43.915552+0000","last_unstale":"2023-10-14T17:57:46.098901+0000","last_undegraded":"2023-10-14T17:57:46.098901+0000","last_fullsized":"2023-10-14T17:57:46.098901+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:23:10.612958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.9","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096152+0000","last_change":"2023-10-14T17:57:44.947183+0000","last_active":"2023-10-14T17:57:46.096152+0000","last_peered":"2023-10-14T17:57:46.096152+0000","last_clean":"2023-10-14T17:57:46.096152+0000","last_became_active":"2023-10-14T17:57:44.946241+0000","last_became_peered":"2023-10-14T17:57:44.946241+0000","last_unstale":"2023-10-14T17:57:46.096152+0000","last_undegraded":"2023-10-14T17:57:46.096152+0000","last_fullsized":"2023-10-14T17:57:46.096152+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:56:57.242181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.8","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.941940+0000","last_change":"2023-10-14T17:57:43.916802+0000","last_active":"2023-10-14T17:57:45.941940+0000","last_peered":"2023-10-14T17:57:45.941940+0000","last_clean":"2023-10-14T17:57:45.941940+0000","last_became_active":"2023-10-14T17:57:43.914415+0000","last_became_peered":"2023-10-14T17:57:43.914415+0000","last_unstale":"2023-10-14T17:57:45.941940+0000","last_undegraded":"2023-10-14T17:57:45.941940+0000","last_fullsized":"2023-10-14T17:57:45.941940+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:50:09.071087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.6","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098430+0000","last_change":"2023-10-14T17:57:44.940212+0000","last_active":"2023-10-14T17:57:46.098430+0000","last_peered":"2023-10-14T17:57:46.098430+0000","last_clean":"2023-10-14T17:57:46.098430+0000","last_became_active":"2023-10-14T17:57:44.939908+0000","last_became_peered":"2023-10-14T17:57:44.939908+0000","last_unstale":"2023-10-14T17:57:46.098430+0000","last_undegraded":"2023-10-14T17:57:46.098430+0000","last_fullsized":"2023-10-14T17:57:46.098430+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:09.072178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.7","version":"177'2","reported_seq":14,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.101307+0000","last_change":"2023-10-14T17:57:43.921693+0000","last_active":"2023-10-14T17:57:46.101307+0000","last_peered":"2023-10-14T17:57:46.101307+0000","last_clean":"2023-10-14T17:57:46.101307+0000","last_became_active":"2023-10-14T17:57:43.921265+0000","last_became_peered":"2023-10-14T17:57:43.921265+0000","last_unstale":"2023-10-14T17:57:46.101307+0000","last_undegraded":"2023-10-14T17:57:46.101307+0000","last_fullsized":"2023-10-14T17:57:46.101307+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:22.954279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.7","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944857+0000","last_change":"2023-10-14T17:57:44.937130+0000","last_active":"2023-10-14T17:57:45.944857+0000","last_peered":"2023-10-14T17:57:45.944857+0000","last_clean":"2023-10-14T17:57:45.944857+0000","last_became_active":"2023-10-14T17:57:44.936833+0000","last_became_peered":"2023-10-14T17:57:44.936833+0000","last_unstale":"2023-10-14T17:57:45.944857+0000","last_undegraded":"2023-10-14T17:57:45.944857+0000","last_fullsized":"2023-10-14T17:57:45.944857+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:45.103248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"20.6","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939389+0000","last_change":"2023-10-14T17:57:43.912468+0000","last_active":"2023-10-14T17:57:45.939389+0000","last_peered":"2023-10-14T17:57:45.939389+0000","last_clean":"2023-10-14T17:57:45.939389+0000","last_became_active":"2023-10-14T17:57:43.912174+0000","last_became_peered":"2023-10-14T17:57:43.912174+0000","last_unstale":"2023-10-14T17:57:45.939389+0000","last_undegraded":"2023-10-14T17:57:45.939389+0000","last_fullsized":"2023-10-14T17:57:45.939389+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:08.304627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.4","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095950+0000","last_change":"2023-10-14T17:57:44.949002+0000","last_active":"2023-10-14T17:57:46.095950+0000","last_peered":"2023-10-14T17:57:46.095950+0000","last_clean":"2023-10-14T17:57:46.095950+0000","last_became_active":"2023-10-14T17:57:44.948662+0000","last_became_peered":"2023-10-14T17:57:44.948662+0000","last_unstale":"2023-10-14T17:57:46.095950+0000","last_undegraded":"2023-10-14T17:57:46.095950+0000","last_fullsized":"2023-10-14T17:57:46.095950+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:36.819220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"20.5","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939589+0000","last_change":"2023-10-14T17:57:43.914999+0000","last_active":"2023-10-14T17:57:45.939589+0000","last_peered":"2023-10-14T17:57:45.939589+0000","last_clean":"2023-10-14T17:57:45.939589+0000","last_became_active":"2023-10-14T17:57:43.914698+0000","last_became_peered":"2023-10-14T17:57:43.914698+0000","last_unstale":"2023-10-14T17:57:45.939589+0000","last_undegraded":"2023-10-14T17:57:45.939589+0000","last_fullsized":"2023-10-14T17:57:45.939589+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:10.178327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.5","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939468+0000","last_change":"2023-10-14T17:57:44.936258+0000","last_active":"2023-10-14T17:57:45.939468+0000","last_peered":"2023-10-14T17:57:45.939468+0000","last_clean":"2023-10-14T17:57:45.939468+0000","last_became_active":"2023-10-14T17:57:44.935951+0000","last_became_peered":"2023-10-14T17:57:44.935951+0000","last_unstale":"2023-10-14T17:57:45.939468+0000","last_undegraded":"2023-10-14T17:57:45.939468+0000","last_fullsized":"2023-10-14T17:57:45.939468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:06.615328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"20.4","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.939058+0000","last_change":"2023-10-14T17:57:43.914406+0000","last_active":"2023-10-14T17:57:45.939058+0000","last_peered":"2023-10-14T17:57:45.939058+0000","last_clean":"2023-10-14T17:57:45.939058+0000","last_became_active":"2023-10-14T17:57:43.914101+0000","last_became_peered":"2023-10-14T17:57:43.914101+0000","last_unstale":"2023-10-14T17:57:45.939058+0000","last_undegraded":"2023-10-14T17:57:45.939058+0000","last_fullsized":"2023-10-14T17:57:45.939058+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:21.136262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"21.2","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096424+0000","last_change":"2023-10-14T17:57:44.937780+0000","last_active":"2023-10-14T17:57:46.096424+0000","last_peered":"2023-10-14T17:57:46.096424+0000","last_clean":"2023-10-14T17:57:46.096424+0000","last_became_active":"2023-10-14T17:57:44.937444+0000","last_became_peered":"2023-10-14T17:57:44.937444+0000","last_unstale":"2023-10-14T17:57:46.096424+0000","last_undegraded":"2023-10-14T17:57:46.096424+0000","last_fullsized":"2023-10-14T17:57:46.096424+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:07:39.409447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"20.3","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097242+0000","last_change":"2023-10-14T17:57:43.905065+0000","last_active":"2023-10-14T17:57:46.097242+0000","last_peered":"2023-10-14T17:57:46.097242+0000","last_clean":"2023-10-14T17:57:46.097242+0000","last_became_active":"2023-10-14T17:57:43.904355+0000","last_became_peered":"2023-10-14T17:57:43.904355+0000","last_unstale":"2023-10-14T17:57:46.097242+0000","last_undegraded":"2023-10-14T17:57:46.097242+0000","last_fullsized":"2023-10-14T17:57:46.097242+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:57:08.057209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"21.3","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097911+0000","last_change":"2023-10-14T17:57:44.937752+0000","last_active":"2023-10-14T17:57:46.097911+0000","last_peered":"2023-10-14T17:57:46.097911+0000","last_clean":"2023-10-14T17:57:46.097911+0000","last_became_active":"2023-10-14T17:57:44.937291+0000","last_became_peered":"2023-10-14T17:57:44.937291+0000","last_unstale":"2023-10-14T17:57:46.097911+0000","last_undegraded":"2023-10-14T17:57:46.097911+0000","last_fullsized":"2023-10-14T17:57:46.097911+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:38.785766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.2","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.099283+0000","last_change":"2023-10-14T17:57:43.917888+0000","last_active":"2023-10-14T17:57:46.099283+0000","last_peered":"2023-10-14T17:57:46.099283+0000","last_clean":"2023-10-14T17:57:46.099283+0000","last_became_active":"2023-10-14T17:57:43.917465+0000","last_became_peered":"2023-10-14T17:57:43.917465+0000","last_unstale":"2023-10-14T17:57:46.099283+0000","last_undegraded":"2023-10-14T17:57:46.099283+0000","last_fullsized":"2023-10-14T17:57:46.099283+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:59.451750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"21.0","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.098468+0000","last_change":"2023-10-14T17:57:44.934521+0000","last_active":"2023-10-14T17:57:46.098468+0000","last_peered":"2023-10-14T17:57:46.098468+0000","last_clean":"2023-10-14T17:57:46.098468+0000","last_became_active":"2023-10-14T17:57:44.932476+0000","last_became_peered":"2023-10-14T17:57:44.932476+0000","last_unstale":"2023-10-14T17:57:46.098468+0000","last_undegraded":"2023-10-14T17:57:46.098468+0000","last_fullsized":"2023-10-14T17:57:46.098468+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:27:42.459551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"20.1","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.942990+0000","last_change":"2023-10-14T17:57:43.894348+0000","last_active":"2023-10-14T17:57:45.942990+0000","last_peered":"2023-10-14T17:57:45.942990+0000","last_clean":"2023-10-14T17:57:45.942990+0000","last_became_active":"2023-10-14T17:57:43.893675+0000","last_became_peered":"2023-10-14T17:57:43.893675+0000","last_unstale":"2023-10-14T17:57:45.942990+0000","last_undegraded":"2023-10-14T17:57:45.942990+0000","last_fullsized":"2023-10-14T17:57:45.942990+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:31:47.223729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"21.1","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.097508+0000","last_change":"2023-10-14T17:57:44.941836+0000","last_active":"2023-10-14T17:57:46.097508+0000","last_peered":"2023-10-14T17:57:46.097508+0000","last_clean":"2023-10-14T17:57:46.097508+0000","last_became_active":"2023-10-14T17:57:44.941563+0000","last_became_peered":"2023-10-14T17:57:44.941563+0000","last_unstale":"2023-10-14T17:57:46.097508+0000","last_undegraded":"2023-10-14T17:57:46.097508+0000","last_fullsized":"2023-10-14T17:57:46.097508+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:41:12.437596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"20.0","version":"0'0","reported_seq":12,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.095020+0000","last_change":"2023-10-14T17:57:43.912407+0000","last_active":"2023-10-14T17:57:46.095020+0000","last_peered":"2023-10-14T17:57:46.095020+0000","last_clean":"2023-10-14T17:57:46.095020+0000","last_became_active":"2023-10-14T17:57:43.912106+0000","last_became_peered":"2023-10-14T17:57:43.912106+0000","last_unstale":"2023-10-14T17:57:46.095020+0000","last_undegraded":"2023-10-14T17:57:46.095020+0000","last_fullsized":"2023-10-14T17:57:46.095020+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:00:24.440166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"21.14","version":"0'0","reported_seq":11,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:46.096958+0000","last_change":"2023-10-14T17:57:44.947794+0000","last_active":"2023-10-14T17:57:46.096958+0000","last_peered":"2023-10-14T17:57:46.096958+0000","last_clean":"2023-10-14T17:57:46.096958+0000","last_became_active":"2023-10-14T17:57:44.943854+0000","last_became_peered":"2023-10-14T17:57:44.943854+0000","last_unstale":"2023-10-14T17:57:46.096958+0000","last_undegraded":"2023-10-14T17:57:46.096958+0000","last_fullsized":"2023-10-14T17:57:46.096958+0000","mapping_epoch":176,"log_start":"0'0","ondisk_log_start":"0'0","created":176,"last_epoch_clean":177,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:43.871317+0000","last_clean_scrub_stamp":"2023-10-14T17:57:43.871317+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:21:45.527699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"20.15","version":"177'3","reported_seq":15,"reported_epoch":178,"state":"active+clean","last_fresh":"2023-10-14T17:57:45.944008+0000","last_change":"2023-10-14T17:57:43.915563+0000","last_active":"2023-10-14T17:57:45.944008+0000","last_peered":"2023-10-14T17:57:45.944008+0000","last_clean":"2023-10-14T17:57:45.944008+0000","last_became_active":"2023-10-14T17:57:43.914800+0000","last_became_peered":"2023-10-14T17:57:43.914800+0000","last_unstale":"2023-10-14T17:57:45.944008+0000","last_undegraded":"2023-10-14T17:57:45.944008+0000","last_fullsized":"2023-10-14T17:57:45.944008+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":175,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:57:42.856348+0000","last_clean_scrub_stamp":"2023-10-14T17:57:42.856348+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:41.265113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"166'271","reported_seq":883,"reported_epoch":177,"state":"active+clean","last_fresh":"2023-10-14T17:57:44.920676+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:57:44.920676+0000","last_peered":"2023-10-14T17:57:44.920676+0000","last_clean":"2023-10-14T17:57:44.920676+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:57:44.920676+0000","last_undegraded":"2023-10-14T17:57:44.920676+0000","last_fullsized":"2023-10-14T17:57:44.920676+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":271,"log_dups_size":0,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":21,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":20,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1460,"num_read_kb":2984,"num_write":439,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":271,"ondisk_log_size":271,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575773,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51984,"kb_used_data":2756,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319856,"statfs":{"total":96636764160,"available":96583532544,"internally_reserved":0,"allocated":2822144,"data_stored":1679928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1830000000000001,"15min":1.2050000000000001},"min":{"1min":0.77300000000000002,"5min":0.73099999999999998,"15min":0.71199999999999997},"max":{"1min":1.429,"5min":1.9019999999999999,"15min":1.9710000000000001},"last":1.23},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.837},"last":0.96699999999999997}]},{"osd":1,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.347,"15min":1.3100000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.29,"15min":1.306},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5720000000000001,"5min":1.8,"15min":1.804},"last":1.357}]},{"osd":2,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2430000000000001,"15min":1.2509999999999999},"min":{"1min":0.873,"5min":0.82899999999999996,"15min":0.82199999999999995},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.93200000000000005},{"interface":"front","average":{"1min":1.472,"5min":1.393,"15min":1.3779999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.71}]},{"osd":3,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4199999999999999,"15min":1.4059999999999999},"min":{"1min":0.996,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.883,"5min":1.883,"15min":2.234},"last":1.302},{"interface":"front","average":{"1min":1.504,"5min":1.5429999999999999,"15min":1.5329999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.9410000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.5389999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.335,"15min":1.276},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.573},{"interface":"front","average":{"1min":1.512,"5min":1.403,"15min":1.373},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.5510000000000002},"last":1.177}]},{"osd":5,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.464,"15min":1.458},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0539999999999998},"last":1.425},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4930000000000001,"15min":1.474},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.218,"5min":2.218,"15min":2.6619999999999999},"last":1.675}]},{"osd":6,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.6140000000000001,"15min":1.585},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4430000000000001},"last":1.74},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.5900000000000001,"15min":1.538},"min":{"1min":1.3440000000000001,"5min":1.0149999999999999,"15min":0.91600000000000004},"max":{"1min":2.1440000000000001,"5min":2.3100000000000001,"15min":2.4100000000000001},"last":1.474}]}]},{"osd":4,"up_from":13,"seq":55834575773,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55560,"kb_used_data":2168,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316280,"statfs":{"total":96636764160,"available":96579870720,"internally_reserved":0,"allocated":2220032,"data_stored":1088908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2689999999999999,"15min":1.2270000000000001},"min":{"1min":0.81000000000000005,"5min":0.73699999999999999,"15min":0.52500000000000002},"max":{"1min":1.732,"5min":1.7749999999999999,"15min":1.8859999999999999},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.248,"15min":1.272},"min":{"1min":0.97299999999999998,"5min":0.86699999999999999,"15min":0.84599999999999997},"max":{"1min":1.897,"5min":1.897,"15min":1.905},"last":1.3819999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3220000000000001},"min":{"1min":0.89100000000000001,"5min":0.874,"15min":0.83999999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.038},{"interface":"front","average":{"1min":1.224,"5min":1.278,"15min":1.2829999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.7310000000000001,"5min":1.7629999999999999,"15min":2.02},"last":1.4370000000000001}]},{"osd":2,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.29,"15min":1.298},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.63800000000000001},"max":{"1min":1.827,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.488},{"interface":"front","average":{"1min":1.385,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":1.0960000000000001,"5min":1.0209999999999999,"15min":0.84199999999999997},"max":{"1min":1.8680000000000001,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.212}]},{"osd":3,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.347,"15min":1.353},"min":{"1min":0.93000000000000005,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.6439999999999999,"5min":1.752,"15min":1.984},"last":1.325},{"interface":"front","average":{"1min":1.552,"5min":1.492,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":1.0109999999999999,"15min":0.82299999999999995},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.734}]},{"osd":5,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4139999999999999,"15min":1.3640000000000001},"min":{"1min":0.97399999999999998,"5min":0.91500000000000004,"15min":0.82899999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.4690000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3779999999999999,"15min":1.3740000000000001},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.79700000000000004},"max":{"1min":1.925,"5min":1.925,"15min":2.0499999999999998},"last":1.536}]},{"osd":6,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4219999999999999,"15min":1.3819999999999999},"min":{"1min":1.004,"5min":0.874,"15min":0.57999999999999996},"max":{"1min":1.9239999999999999,"5min":2.117,"15min":2.117},"last":1.8029999999999999},{"interface":"front","average":{"1min":1.673,"5min":1.5600000000000001,"15min":1.5349999999999999},"min":{"1min":1.202,"5min":1.133,"15min":1.006},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.6000000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5609999999999999,"15min":1.506},"min":{"1min":1.0780000000000001,"5min":0.92500000000000004,"15min":0.86799999999999999},"max":{"1min":2.0880000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":2.0190000000000001},{"interface":"front","average":{"1min":1.696,"5min":1.6319999999999999,"15min":1.593},"min":{"1min":1.4410000000000001,"5min":1.179,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":2.141}]}]},{"osd":6,"up_from":13,"seq":55834575772,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41680,"kb_used_data":2180,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94330160,"statfs":{"total":96636764160,"available":96594083840,"internally_reserved":0,"allocated":2232320,"data_stored":1089054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.73999999999999999},"max":{"1min":1.4870000000000001,"5min":1.696,"15min":1.837},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.081,"5min":1.1850000000000001,"15min":1.2010000000000001},"min":{"1min":0.76500000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.401,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":0.96199999999999997}]},{"osd":1,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.276,"15min":1.244},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.5640000000000001,"5min":1.752,"15min":1.927},"last":0.749},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2989999999999999,"15min":1.3220000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.6040000000000001,"5min":1.7849999999999999,"15min":1.8069999999999999},"last":1.329}]},{"osd":2,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86299999999999999},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8740000000000001},"last":1.5489999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.4450000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":1.9370000000000001},"last":1.4510000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.417},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0409999999999999},"last":1.897},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.393,"15min":1.431},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.764,"5min":1.903,"15min":1.9790000000000001},"last":1.2310000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.361,"15min":1.367},"min":{"1min":1.014,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.736,"5min":1.929,"15min":1.984},"last":2.0230000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4379999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.889,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.7150000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3879999999999999,"15min":1.361},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.79000000000000004},"max":{"1min":1.629,"5min":1.7749999999999999,"15min":2.085},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.5580000000000001},"min":{"1min":1.214,"5min":1.048,"15min":1.048},"max":{"1min":1.9450000000000001,"5min":2.0310000000000001,"15min":2.105},"last":2.1840000000000002}]},{"osd":7,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5900000000000001,"15min":1.5840000000000001},"min":{"1min":1.165,"5min":1.026,"15min":1.0189999999999999},"max":{"1min":2.0960000000000001,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.9490000000000001},{"interface":"front","average":{"1min":1.629,"5min":1.5640000000000001,"15min":1.5720000000000001},"min":{"1min":1.268,"5min":1.0129999999999999,"15min":0.81000000000000005},"max":{"1min":1.8700000000000001,"5min":1.954,"15min":2.1589999999999998},"last":1.782}]}]},{"osd":5,"up_from":13,"seq":55834575773,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37512,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334328,"statfs":{"total":96636764160,"available":96598351872,"internally_reserved":0,"allocated":2224128,"data_stored":1088930,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.2070000000000001,"15min":1.1870000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.7949999999999999},"last":1.4790000000000001},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.2090000000000001},"min":{"1min":0.85299999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6040000000000001,"5min":1.677,"15min":2.0150000000000001},"last":1.131}]},{"osd":1,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.623,"5min":1.774,"15min":1.774},"last":1.637},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.286,"15min":1.286},"min":{"1min":0.85399999999999998,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.78,"5min":1.7949999999999999,"15min":1.8280000000000001},"last":0.996}]},{"osd":2,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.462,"5min":1.6879999999999999,"15min":1.72},"last":1.73},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.331,"15min":1.353},"min":{"1min":0.91700000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.6779999999999999,"5min":1.899,"15min":2.4140000000000001},"last":1.758}]},{"osd":3,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.458,"15min":1.444},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.419},{"interface":"front","average":{"1min":1.333,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.655,"5min":1.8799999999999999,"15min":2.5219999999999998},"last":1.7889999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3220000000000001,"15min":1.3260000000000001},"min":{"1min":0.89800000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.607,"5min":1.9850000000000001,"15min":2.7210000000000001},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.381,"15min":1.393},"min":{"1min":0.96999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.617}]},{"osd":6,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4119999999999999,"15min":1.3819999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.7949999999999999},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.577,"15min":1.581},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.86599999999999999},"max":{"1min":1.804,"5min":2.1389999999999998,"15min":2.907},"last":1.8919999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5509999999999999,"15min":1.544},"min":{"1min":1.073,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7889999999999999,"5min":2.1539999999999999,"15min":2.1760000000000002},"last":1.694},{"interface":"front","average":{"1min":1.413,"5min":1.5049999999999999,"15min":1.516},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.873,"5min":2.0009999999999999,"15min":2.9630000000000001},"last":1.6639999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575773,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51400,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320440,"statfs":{"total":96636764160,"available":96584130560,"internally_reserved":0,"allocated":2224128,"data_stored":1088998,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.98199999999999998,"15min":0.98099999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.423,"5min":1.7949999999999999,"15min":2.6680000000000001},"last":1.7},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1890000000000001,"15min":1.206},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47999999999999998},"max":{"1min":1.5469999999999999,"5min":1.6240000000000001,"15min":2.8290000000000002},"last":1.2989999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0369999999999999,"15min":1.008},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.843},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.131,"5min":1.232,"15min":1.21},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7929999999999999,"5min":1.849,"15min":2.1970000000000001},"last":1.103}]},{"osd":2,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.454,"15min":1.4550000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6759999999999999,"5min":1.9299999999999999,"15min":2.1000000000000001},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.395,"5min":1.427,"15min":1.4019999999999999},"min":{"1min":0.80000000000000004,"5min":0.79500000000000004,"15min":0.55100000000000005},"max":{"1min":1.8979999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":1.5}]},{"osd":4,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4550000000000001,"15min":1.427},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.92500000000000004},"max":{"1min":1.716,"5min":1.8660000000000001,"15min":2.0489999999999999},"last":1.931},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.48,"15min":1.468},"min":{"1min":0.96599999999999997,"5min":0.96299999999999997,"15min":0.86099999999999999},"max":{"1min":1.9079999999999999,"5min":2.2210000000000001,"15min":2.6459999999999999},"last":1.3899999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4590000000000001,"15min":1.478},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.73399999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.5830000000000002},"last":1.6539999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.514,"15min":1.52},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7949999999999999,"5min":1.9850000000000001,"15min":2.3719999999999999},"last":1.4419999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.6699999999999999,"15min":1.6419999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.893,"5min":2.6480000000000001,"15min":2.851},"last":1.875},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.669,"15min":1.679},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0289999999999999,"5min":2.7109999999999999,"15min":2.9910000000000001},"last":1.5529999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.643,"15min":1.6200000000000001},"min":{"1min":1.163,"5min":1.163,"15min":0.79500000000000004},"max":{"1min":2.2040000000000002,"5min":2.4140000000000001,"15min":2.9430000000000001},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.583,"5min":1.6679999999999999,"15min":1.6839999999999999},"min":{"1min":1.2330000000000001,"5min":1.2250000000000001,"15min":1.137},"max":{"1min":1.897,"5min":2.0899999999999999,"15min":2.4980000000000002},"last":1.615}]}]},{"osd":2,"up_from":13,"seq":55834575773,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51412,"kb_used_data":2184,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320428,"statfs":{"total":96636764160,"available":96584118272,"internally_reserved":0,"allocated":2236416,"data_stored":1090590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.95499999999999996,"15min":0.93600000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3520000000000001,"5min":1.6879999999999999,"15min":1.8759999999999999},"last":0.59699999999999998},{"interface":"front","average":{"1min":1.03,"5min":1.089,"15min":1.115},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.7490000000000001,"15min":1.7889999999999999},"last":0.38600000000000001}]},{"osd":1,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.03,"15min":1.0249999999999999},"min":{"1min":0.52300000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5720000000000001,"5min":1.6080000000000001,"15min":1.8620000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.764,"5min":1.8129999999999999,"15min":1.8959999999999999},"last":0.42699999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2210000000000001,"15min":1.274},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.5600000000000001,"5min":1.929,"15min":1.929},"last":0.622},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.375,"15min":1.3640000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0289999999999999},"last":0.72799999999999998}]},{"osd":4,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.472,"15min":1.4790000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.153},"last":1.1699999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3740000000000001,"15min":1.4099999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.917,"5min":2.0640000000000001,"15min":2.2639999999999998},"last":0.92200000000000004}]},{"osd":5,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4319999999999999,"15min":1.462},"min":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.865,"5min":1.9359999999999999,"15min":2.2669999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.611,"5min":1.591,"15min":1.585},"min":{"1min":0.94599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0070000000000001,"5min":2.0880000000000001,"15min":2.3199999999999998},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5169999999999999,"15min":1.52},"min":{"1min":0.98099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0470000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":0.95899999999999996},{"interface":"front","average":{"1min":1.54,"5min":1.583,"15min":1.6120000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.0169999999999999,"5min":2.161,"15min":2.3610000000000002},"last":0.97099999999999997}]},{"osd":7,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":1.008,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1640000000000001},"last":1.0289999999999999},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.645,"15min":1.677},"min":{"1min":1.0660000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.137,"5min":2.177,"15min":2.4350000000000001},"last":1.0880000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575773,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47240,"kb_used_data":2172,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324600,"statfs":{"total":96636764160,"available":96588390400,"internally_reserved":0,"allocated":2224128,"data_stored":1089490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.97499999999999998,"15min":0.94999999999999996},"min":{"1min":0.68400000000000005,"5min":0.65100000000000002,"15min":0.64800000000000002},"max":{"1min":1.4630000000000001,"5min":1.706,"15min":1.756},"last":0.46100000000000002},{"interface":"front","average":{"1min":1.1499999999999999,"5min":1.1379999999999999,"15min":1.1559999999999999},"min":{"1min":0.77200000000000002,"5min":0.72999999999999998,"15min":0.51600000000000001},"max":{"1min":1.6579999999999999,"5min":1.724,"15min":1.9299999999999999},"last":0.379}]},{"osd":2,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.0229999999999999,"5min":1.0329999999999999,"15min":1.0329999999999999},"min":{"1min":0.78800000000000003,"5min":0.77200000000000002,"15min":0.71499999999999997},"max":{"1min":1.7709999999999999,"5min":1.8660000000000001,"15min":2.597},"last":0.48899999999999999},{"interface":"front","average":{"1min":1.296,"5min":1.2589999999999999,"15min":1.258},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.81699999999999995},"max":{"1min":1.6699999999999999,"5min":1.7470000000000001,"15min":1.9610000000000001},"last":0.52900000000000003}]},{"osd":3,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4099999999999999,"5min":1.4079999999999999,"15min":1.3720000000000001},"min":{"1min":1.032,"5min":0.94399999999999995,"15min":0.89700000000000002},"max":{"1min":1.8380000000000001,"5min":1.8380000000000001,"15min":1.877},"last":0.65600000000000003},{"interface":"front","average":{"1min":1.3779999999999999,"5min":1.327,"15min":1.2989999999999999},"min":{"1min":0.94799999999999995,"5min":0.86299999999999999,"15min":0.84999999999999998},"max":{"1min":1.7669999999999999,"5min":1.8160000000000001,"15min":2.2599999999999998},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.464,"15min":1.474},"min":{"1min":1.1799999999999999,"5min":1.1339999999999999,"15min":0.78400000000000003},"max":{"1min":1.7490000000000001,"5min":1.8029999999999999,"15min":2.044},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.3899999999999999,"5min":1.427,"15min":1.415},"min":{"1min":1.085,"5min":1.038,"15min":0.995},"max":{"1min":1.7769999999999999,"5min":2.0489999999999999,"15min":2.706},"last":0.82899999999999996}]},{"osd":5,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4079999999999999,"5min":1.4710000000000001,"15min":1.4430000000000001},"min":{"1min":1.0549999999999999,"5min":1.0549999999999999,"15min":0.84699999999999998},"max":{"1min":1.911,"5min":1.9390000000000001,"15min":2.3439999999999999},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.5940000000000001,"5min":1.5840000000000001,"15min":1.5580000000000001},"min":{"1min":1.323,"5min":1.0169999999999999,"15min":1.0169999999999999},"max":{"1min":2.379,"5min":2.379,"15min":2.4660000000000002},"last":0.81899999999999995}]},{"osd":6,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6299999999999999,"5min":1.6060000000000001,"15min":1.585},"min":{"1min":1.3580000000000001,"5min":1.1080000000000001,"15min":1.0960000000000001},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.177},"last":1.2070000000000001},{"interface":"front","average":{"1min":1.5369999999999999,"5min":1.6100000000000001,"15min":1.5680000000000001},"min":{"1min":1.093,"5min":1.093,"15min":0.998},"max":{"1min":1.984,"5min":2.1299999999999999,"15min":2.4860000000000002},"last":0.88800000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:10 2023","interfaces":[{"interface":"back","average":{"1min":1.698,"5min":1.706,"15min":1.6739999999999999},"min":{"1min":1.417,"5min":1.2430000000000001,"15min":0.97599999999999998},"max":{"1min":2.294,"5min":2.3660000000000001,"15min":2.5379999999999998},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.6899999999999999,"5min":1.6950000000000001,"15min":1.6779999999999999},"min":{"1min":1.3620000000000001,"5min":1.282,"15min":1.016},"max":{"1min":1.9990000000000001,"5min":2.1019999999999999,"15min":2.2930000000000001},"last":0.91300000000000003}]}]},{"osd":0,"up_from":13,"seq":55834575772,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60320,"kb_used_data":2764,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311520,"statfs":{"total":96636764160,"available":96574996480,"internally_reserved":0,"allocated":2830336,"data_stored":1680922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.577,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":1.665},{"interface":"front","average":{"1min":1.141,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.72699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6830000000000001,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.94999999999999996}]},{"osd":2,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1739999999999999,"15min":1.1499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.446,"5min":3.375,"15min":3.375},"last":1.113},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2569999999999999,"15min":1.2629999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.681},"last":1.0680000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3819999999999999,"15min":1.3660000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.708,"5min":1.823,"15min":2.1899999999999999},"last":1.234},{"interface":"front","average":{"1min":1.383,"5min":1.425,"15min":1.4279999999999999},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.93999999999999995},"max":{"1min":1.792,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.1830000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.3420000000000001},"min":{"1min":1.0029999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.675,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.488},{"interface":"front","average":{"1min":1.534,"5min":1.5309999999999999,"15min":1.498},"min":{"1min":1.002,"5min":1.002,"15min":0.999},"max":{"1min":2.008,"5min":2.9689999999999999,"15min":2.9689999999999999},"last":1.792}]},{"osd":5,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.401,"15min":1.4119999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.873,"5min":3.077,"15min":3.077},"last":1.609},{"interface":"front","average":{"1min":1.611,"5min":1.621,"15min":1.6020000000000001},"min":{"1min":1.194,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":2.028,"5min":2.468,"15min":2.468},"last":1.5760000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.6799999999999999,"15min":1.681},"min":{"1min":1.2769999999999999,"5min":1.194,"15min":1.147},"max":{"1min":2.1349999999999998,"5min":2.3479999999999999,"15min":2.5270000000000001},"last":1.411},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.548,"15min":1.554},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.875,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.292}]},{"osd":7,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.675,"15min":1.655},"min":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"max":{"1min":1.9399999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.361},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6619999999999999,"15min":1.655},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":1.99,"5min":3.645,"15min":3.645},"last":1.744}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":146,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":20,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":21,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:57:49.945 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T17:57:49.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T17:57:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.219+0000 7f09af59e640 1 Processor -- start 2023-10-14T17:57:50.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.220+0000 7f09af59e640 1 -- start start 2023-10-14T17:57:50.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.221+0000 7f09af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f09b00cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.221+0000 7f09af59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00cc850 con 0x7f09b006a4b0 2023-10-14T17:57:50.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.221+0000 7f09af59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00ccb30 con 0x7f09b015ac70 2023-10-14T17:57:50.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.221+0000 7f09af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00cce10 con 0x7f09b00b4160 2023-10-14T17:57:50.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.222+0000 7f09add9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f09b006a4b0 0x7f09b006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55008/0 (socket says 172.21.15.29:55008) 2023-10-14T17:57:50.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.222+0000 7f09add9b640 1 -- 172.21.15.29:0/2321385811 learned_addr learned my addr 172.21.15.29:0/2321385811 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.222+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f09b00cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.222+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674403654 0 0) 0x7f09b00cc850 con 0x7f09b006a4b0 2023-10-14T17:57:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.222+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f099c003520 con 0x7f09b006a4b0 2023-10-14T17:57:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 225559421 0 0) 0x7f099c003520 con 0x7f09b006a4b0 2023-10-14T17:57:50.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 msgr2=0x7f09b00cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f09b00cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 >> v1:172.21.15.186:6789/0 conn(0x7f09b015ac70 legacy=0x7f09b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b00cd0f0 con 0x7f09b006a4b0 2023-10-14T17:57:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0998001670 con 0x7f09b006a4b0 2023-10-14T17:57:50.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 >> v1:172.21.15.29:6789/0 conn(0x7f09b006a4b0 legacy=0x7f09b006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099c0039a0 0x7f099c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f099c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f099c004d00 0x7f099c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00cce10 con 0x7f099c004d00 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099c00ba60 con 0x7f099c0039a0 2023-10-14T17:57:50.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f099c00bc20 con 0x7f09b00b4160 2023-10-14T17:57:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f099c004d00 0x7f099c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09ae59c640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099c0039a0 0x7f099c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.223+0000 7f09add9b640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f099c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09aed9d640 1 -- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 msgr2=0x7f099c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:50.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f099c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09aed9d640 1 -- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099c0039a0 msgr2=0x7f099c003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:50.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f099c0039a0 0x7f099c003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09aed9d640 1 -- 172.21.15.29:0/2321385811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c003520 con 0x7f099c004d00 2023-10-14T17:57:50.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.224+0000 7f09add9b640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09b00b4160 0x7f099c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:50.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.225+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f099c004d00 0x7f099c005180 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009a90 tx=0x7f099c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9ef5239d67bb33 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.225+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802c680 con 0x7f099c004d00 2023-10-14T17:57:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.225+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a802c7e0 con 0x7f099c004d00 2023-10-14T17:57:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.225+0000 7f09ad59a640 1 -- 172.21.15.29:0/2321385811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802cb70 con 0x7f099c004d00 2023-10-14T17:57:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 -- 172.21.15.29:0/2321385811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f099c004d00 msgr2=0x7f099c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 --2- 172.21.15.29:0/2321385811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f099c004d00 0x7f099c005180 secure :-1 s=READY pgs=1991 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009a90 tx=0x7f099c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09aed9d640 1 -- 172.21.15.29:0/2321385811 reap_dead start 2023-10-14T17:57:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 -- 172.21.15.29:0/2321385811 shutdown_connections 2023-10-14T17:57:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 -- 172.21.15.29:0/2321385811 >> 172.21.15.29:0/2321385811 conn(0x7f09b00ce6a0 msgr2=0x7f09b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 -- 172.21.15.29:0/2321385811 shutdown_connections 2023-10-14T17:57:50.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.226+0000 7f09af59e640 1 -- 172.21.15.29:0/2321385811 wait complete. 2023-10-14T17:57:50.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.228+0000 7f09af59e640 1 Processor -- start 2023-10-14T17:57:50.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.228+0000 7f09af59e640 1 -- start start 2023-10-14T17:57:50.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 0x7f09b00c4c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 0x7f09b0149330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b014a310 con 0x7f09b006ab00 2023-10-14T17:57:50.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b01f2110 con 0x7f09b01f1c90 2023-10-14T17:57:50.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.229+0000 7f09af59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0149870 con 0x7f099c004d00 2023-10-14T17:57:50.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41024/0 (socket says 172.21.15.29:41024) 2023-10-14T17:57:50.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 -- 172.21.15.29:0/2991052908 learned_addr learned my addr 172.21.15.29:0/2991052908 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:50.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09ae59c640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 0x7f09b00c4c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09add9b640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 0x7f09b0149330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 -- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 msgr2=0x7f09b00c4c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 0x7f09b00c4c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 -- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 msgr2=0x7f09b0149330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 0x7f09b0149330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09aed9d640 1 -- 172.21.15.29:0/2991052908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b0149a70 con 0x7f09b006ab00 2023-10-14T17:57:50.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09ae59c640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 0x7f09b00c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.230+0000 7f09add9b640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 0x7f09b0149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09aed9d640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009bb0 tx=0x7f09a8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802c680 con 0x7f09b006ab00 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a8003830 con 0x7f09b006ab00 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09a802dea0 con 0x7f09b006ab00 2023-10-14T17:57:50.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0149d50 con 0x7f09b006ab00 2023-10-14T17:57:50.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.231+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0147480 con 0x7f09b006ab00 2023-10-14T17:57:50.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.232+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00668c0 con 0x7f09b006ab00 2023-10-14T17:57:50.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.233+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f09a8003be0 con 0x7f09b006ab00 2023-10-14T17:57:50.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.234+0000 7f09a6ffd640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 0x7f099006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.234+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f09a80aac40 con 0x7f09b006ab00 2023-10-14T17:57:50.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.235+0000 7f09ae59c640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 0x7f099006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.235+0000 7f09ae59c640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 0x7f099006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09a0007fd0 tx=0x7f09a0009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.239+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f09a8075ac0 con 0x7f09b006ab00 2023-10-14T17:57:50.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.503+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f09b01472e0 con 0x7f09b006ab00 2023-10-14T17:57:50.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.508+0000 7f09a6ffd640 1 -- 172.21.15.29:0/2991052908 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v42) v1 ==== 190+0+147 (secure 0 0 0) 0x7f09b01472e0 con 0x7f09b006ab00 2023-10-14T17:57:50.511 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T17:57:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T17:57:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T17:57:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T17:57:50.511 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T17:57:50.512 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T17:57:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 msgr2=0x7f099006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 0x7f099006d480 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f09a0007fd0 tx=0x7f09a0009040 comp rx=0 tx=0).stop 2023-10-14T17:57:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 msgr2=0x7f09b006af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 secure :-1 s=READY pgs=1992 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009bb0 tx=0x7f09a8004730 comp rx=0 tx=0).stop 2023-10-14T17:57:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 shutdown_connections 2023-10-14T17:57:50.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f099006afc0 0x7f099006d480 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09b006ab00 0x7f09b006af80 unknown :-1 s=CLOSED pgs=1992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09b01f1c90 0x7f09b0149330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 --2- 172.21.15.29:0/2991052908 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f099c004d00 0x7f09b00c4c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 >> 172.21.15.29:0/2991052908 conn(0x7f09b00ce6a0 msgr2=0x7f09b00c9d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:50.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 shutdown_connections 2023-10-14T17:57:50.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.511+0000 7f09af59e640 1 -- 172.21.15.29:0/2991052908 wait complete. 2023-10-14T17:57:50.532 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:57:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.733+0000 7f628b085640 1 Processor -- start 2023-10-14T17:57:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.733+0000 7f628b085640 1 -- start start 2023-10-14T17:57:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.734+0000 7f628b085640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f628406c040 0x7f628406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.734+0000 7f628b085640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f62840cda80 0x7f628415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.734+0000 7f628b085640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f628406c960 con 0x7f62840ad970 2023-10-14T17:57:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.734+0000 7f628b085640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628415b9d0 con 0x7f628406c040 2023-10-14T17:57:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.734+0000 7f628b085640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f628415bcb0 con 0x7f62840cda80 2023-10-14T17:57:50.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f628406c040 0x7f628406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f628406c040 0x7f628406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41030/0 (socket says 172.21.15.29:41030) 2023-10-14T17:57:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 -- 172.21.15.29:0/2617295146 learned_addr learned my addr 172.21.15.29:0/2617295146 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 -- 172.21.15.29:0/2617295146 >> v2:172.21.15.186:3300/0 conn(0x7f62840cda80 msgr2=0x7f628415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:50.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 --2- 172.21.15.29:0/2617295146 >> v2:172.21.15.186:3300/0 conn(0x7f62840cda80 0x7f628415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 -- 172.21.15.29:0/2617295146 >> v1:172.21.15.186:6789/0 conn(0x7f62840ad970 legacy=0x7f628415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.735+0000 7f628a083640 1 -- 172.21.15.29:0/2617295146 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f628415bf90 con 0x7f628406c040 2023-10-14T17:57:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.736+0000 7f628a083640 1 --2- 172.21.15.29:0/2617295146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628406c040 0x7f628406c420 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f62840cf300 tx=0x7f628002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8849364edbf63587 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.736+0000 7f6289081640 1 -- 172.21.15.29:0/2617295146 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280031870 con 0x7f628406c040 2023-10-14T17:57:50.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f6289081640 1 -- 172.21.15.29:0/2617295146 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62800319d0 con 0x7f628406c040 2023-10-14T17:57:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f6289081640 1 -- 172.21.15.29:0/2617295146 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6280031d60 con 0x7f628406c040 2023-10-14T17:57:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 -- 172.21.15.29:0/2617295146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628406c040 msgr2=0x7f628406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 --2- 172.21.15.29:0/2617295146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628406c040 0x7f628406c420 secure :-1 s=READY pgs=1993 cs=0 l=1 rev1=1 crypto rx=0x7f62840cf300 tx=0x7f628002b910 comp rx=0 tx=0).stop 2023-10-14T17:57:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 -- 172.21.15.29:0/2617295146 shutdown_connections 2023-10-14T17:57:50.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 --2- 172.21.15.29:0/2617295146 >> v2:172.21.15.186:3300/0 conn(0x7f62840cda80 0x7f628415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 --2- 172.21.15.29:0/2617295146 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f628406c040 0x7f628406c420 unknown :-1 s=CLOSED pgs=1993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 -- 172.21.15.29:0/2617295146 >> 172.21.15.29:0/2617295146 conn(0x7f62840ce6a0 msgr2=0x7f62840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 -- 172.21.15.29:0/2617295146 shutdown_connections 2023-10-14T17:57:50.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.737+0000 7f628b085640 1 -- 172.21.15.29:0/2617295146 wait complete. 2023-10-14T17:57:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.739+0000 7f628b085640 1 Processor -- start 2023-10-14T17:57:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.739+0000 7f628b085640 1 -- start start 2023-10-14T17:57:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628406c040 0x7f6284156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 0x7f62840c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840c7cb0 con 0x7f62840cda80 2023-10-14T17:57:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62840c2d80 con 0x7f628406c040 2023-10-14T17:57:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628b085640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62840c3060 con 0x7f62840ad970 2023-10-14T17:57:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628a884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628a884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41042/0 (socket says 172.21.15.29:41042) 2023-10-14T17:57:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628a884640 1 -- 172.21.15.29:0/1730894513 learned_addr learned my addr 172.21.15.29:0/1730894513 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.740+0000 7f628a083640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628406c040 0x7f6284156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f6289882640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 0x7f62840c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 -- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 msgr2=0x7f62840c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 0x7f62840c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 -- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628406c040 msgr2=0x7f6284156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628406c040 0x7f6284156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 -- 172.21.15.29:0/1730894513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62800094d0 con 0x7f62840cda80 2023-10-14T17:57:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f6289882640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 0x7f62840c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:50.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628a884640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b970 tx=0x7f627c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c00cb40 con 0x7f62840cda80 2023-10-14T17:57:50.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c00ccc0 con 0x7f62840cda80 2023-10-14T17:57:50.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c014680 con 0x7f62840cda80 2023-10-14T17:57:50.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62840c3340 con 0x7f62840cda80 2023-10-14T17:57:50.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.741+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6284156db0 con 0x7f62840cda80 2023-10-14T17:57:50.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.742+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f628406b330 con 0x7f62840cda80 2023-10-14T17:57:50.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.743+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f627c01d3b0 con 0x7f62840cda80 2023-10-14T17:57:50.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.744+0000 7f627affd640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 0x7f626806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:50.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.744+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f627c089830 con 0x7f62840cda80 2023-10-14T17:57:50.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.745+0000 7f628a083640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 0x7f626806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:50.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.745+0000 7f628a083640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 0x7f626806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6280036000 tx=0x7f628003c040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:50.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.749+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f627c0546b0 con 0x7f62840cda80 2023-10-14T17:57:50.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.983+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f62840c3860 con 0x7f62840cda80 2023-10-14T17:57:50.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.984+0000 7f627affd640 1 -- 172.21.15.29:0/1730894513 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f62840c3860 con 0x7f62840cda80 2023-10-14T17:57:50.986 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:57:50.986 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:57:50.987 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:57:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.986+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 msgr2=0x7f626806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.986+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 0x7f626806d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f6280036000 tx=0x7f628003c040 comp rx=0 tx=0).stop 2023-10-14T17:57:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.986+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 msgr2=0x7f62840ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.986+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 secure :-1 s=READY pgs=1994 cs=0 l=1 rev1=1 crypto rx=0x7f627c00b970 tx=0x7f627c00be30 comp rx=0 tx=0).stop 2023-10-14T17:57:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 shutdown_connections 2023-10-14T17:57:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f626806af90 0x7f626806d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62840cda80 0x7f62840ac890 unknown :-1 s=CLOSED pgs=1994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62840ad970 0x7f62840c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 --2- 172.21.15.29:0/1730894513 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f628406c040 0x7f6284156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 >> 172.21.15.29:0/1730894513 conn(0x7f62840ce6a0 msgr2=0x7f628406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 shutdown_connections 2023-10-14T17:57:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:50.987+0000 7f628b085640 1 -- 172.21.15.29:0/1730894513 wait complete. 2023-10-14T17:57:51.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T17:57:44.368179+0000', 'modified': '2023-10-14T17:57:47.451014+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 9615}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_9615': {'gid': 9615, 'name': 'c', 'rank': 0, 'incarnation': 79, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6836/494120094', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 494120094}, {'type': 'v1', 'addr': '172.21.15.29:6836', 'nonce': 494120094}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [21], 'metadata_pool': 20, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T17:57:51.410 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T17:57:51.411 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T17:57:51.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.651+0000 7ff253661640 1 Processor -- start 2023-10-14T17:57:51.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.651+0000 7ff253661640 1 -- start start 2023-10-14T17:57:51.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.653+0000 7ff253661640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff24c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.653+0000 7ff253661640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff24c06c140 0x7ff24c06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.653+0000 7ff253661640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24c070360 con 0x7ff24c159ab0 2023-10-14T17:57:51.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.653+0000 7ff253661640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff24c06cb10 con 0x7ff24c06c140 2023-10-14T17:57:51.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.653+0000 7ff253661640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c15d890 con 0x7ff24c159e90 2023-10-14T17:57:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff2513fd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff24c159ab0 0x7ff24c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59194/0 (socket says 172.21.15.29:59194) 2023-10-14T17:57:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff2513fd640 1 -- 172.21.15.29:0/1031564217 learned_addr learned my addr 172.21.15.29:0/1031564217 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:51.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff251bfe640 1 --2- 172.21.15.29:0/1031564217 >> v2:172.21.15.186:3300/0 conn(0x7ff24c06c140 0x7ff24c06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff250bfc640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff24c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553087311 0 0) 0x7ff24c070360 con 0x7ff24c159ab0 2023-10-14T17:57:51.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.654+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff238003520 con 0x7ff24c159ab0 2023-10-14T17:57:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873925667 0 0) 0x7ff238003520 con 0x7ff24c159ab0 2023-10-14T17:57:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 msgr2=0x7ff24c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff24c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 >> v2:172.21.15.186:3300/0 conn(0x7ff24c06c140 msgr2=0x7ff24c06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 --2- 172.21.15.29:0/1031564217 >> v2:172.21.15.186:3300/0 conn(0x7ff24c06c140 0x7ff24c06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c0cd1f0 con 0x7ff24c159ab0 2023-10-14T17:57:51.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff248001670 con 0x7ff24c159ab0 2023-10-14T17:57:51.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 >> v1:172.21.15.29:6789/0 conn(0x7ff24c159ab0 legacy=0x7ff24c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff238003970 0x7ff238003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2380047c0 0x7ff238004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff238005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c06cb10 con 0x7ff2380047c0 2023-10-14T17:57:51.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c15d890 con 0x7ff238003970 2023-10-14T17:57:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23800ba60 con 0x7ff24c159e90 2023-10-14T17:57:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.655+0000 7ff250bfc640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2380047c0 0x7ff238004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff2513fd640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff238003970 0x7ff238003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff251bfe640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff238005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff250bfc640 1 -- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 msgr2=0x7ff238005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff250bfc640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c159e90 0x7ff238005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff250bfc640 1 -- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff238003970 msgr2=0x7ff238003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff250bfc640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff238003970 0x7ff238003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.656+0000 7ff250bfc640 1 -- 172.21.15.29:0/1031564217 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff238003520 con 0x7ff2380047c0 2023-10-14T17:57:51.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.657+0000 7ff250bfc640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2380047c0 0x7ff238004c40 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00bf90 tx=0x7ff2380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6abb75a64765a9b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.657+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c02e7d0 con 0x7ff2380047c0 2023-10-14T17:57:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.657+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c011040 con 0x7ff2380047c0 2023-10-14T17:57:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.657+0000 7ff243fff640 1 -- 172.21.15.29:0/1031564217 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c02eb70 con 0x7ff2380047c0 2023-10-14T17:57:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 -- 172.21.15.29:0/1031564217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2380047c0 msgr2=0x7ff238004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 --2- 172.21.15.29:0/1031564217 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2380047c0 0x7ff238004c40 secure :-1 s=READY pgs=1996 cs=0 l=1 rev1=1 crypto rx=0x7ff23c00bf90 tx=0x7ff2380057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff251bfe640 1 -- 172.21.15.29:0/1031564217 reap_dead start 2023-10-14T17:57:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 -- 172.21.15.29:0/1031564217 shutdown_connections 2023-10-14T17:57:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 -- 172.21.15.29:0/1031564217 >> 172.21.15.29:0/1031564217 conn(0x7ff24c0ce9b0 msgr2=0x7ff24c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 -- 172.21.15.29:0/1031564217 shutdown_connections 2023-10-14T17:57:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.658+0000 7ff253661640 1 -- 172.21.15.29:0/1031564217 wait complete. 2023-10-14T17:57:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.660+0000 7ff253661640 1 Processor -- start 2023-10-14T17:57:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.660+0000 7ff253661640 1 -- start start 2023-10-14T17:57:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2380047c0 0x7ff24c1e7030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c15cfb0 0x7ff24c0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1e7910 con 0x7ff24c15cb50 2023-10-14T17:57:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1e7bf0 con 0x7ff2380047c0 2023-10-14T17:57:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff253661640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff24c1e7ed0 con 0x7ff24c15cfb0 2023-10-14T17:57:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff250bfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff250bfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54466/0 (socket says 172.21.15.29:54466) 2023-10-14T17:57:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.661+0000 7ff250bfc640 1 -- 172.21.15.29:0/2811984478 learned_addr learned my addr 172.21.15.29:0/2811984478 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2380047c0 0x7ff24c1e7030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c15cfb0 0x7ff24c0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 -- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c15cfb0 msgr2=0x7ff24c0c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c15cfb0 0x7ff24c0c3a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 -- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2380047c0 msgr2=0x7ff24c1e7030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2380047c0 0x7ff24c1e7030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 -- 172.21.15.29:0/2811984478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24c1e81b0 con 0x7ff24c15cb50 2023-10-14T17:57:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7ff23c039040 tx=0x7ff23c00b750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff241ffb640 1 -- 172.21.15.29:0/2811984478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c02ee50 con 0x7ff24c15cb50 2023-10-14T17:57:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff241ffb640 1 -- 172.21.15.29:0/2811984478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c004030 con 0x7ff24c15cb50 2023-10-14T17:57:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff241ffb640 1 -- 172.21.15.29:0/2811984478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c0043e0 con 0x7ff24c15cb50 2023-10-14T17:57:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.662+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff24c1e8490 con 0x7ff24c15cb50 2023-10-14T17:57:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.663+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0c3fb0 con 0x7ff24c15cb50 2023-10-14T17:57:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.666+0000 7ff241ffb640 1 -- 172.21.15.29:0/2811984478 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff23c03d070 con 0x7ff24c15cb50 2023-10-14T17:57:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.667+0000 7ff241ffb640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.667+0000 7ff241ffb640 1 -- 172.21.15.29:0/2811984478 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff23c0aa4a0 con 0x7ff24c15cb50 2023-10-14T17:57:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.667+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.667+0000 7ff2561f2740 1 Processor -- start 2023-10-14T17:57:51.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 -- start start 2023-10-14T17:57:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 0x7ff23006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff248001120 tx=0x7ff248009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9791b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f952b10 0x56050f952f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f8427a0 con 0x56050f95aaf0 2023-10-14T17:57:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f894740 con 0x56050f952b10 2023-10-14T17:57:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.668+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56050f891e30 con 0x56050f95a710 2023-10-14T17:57:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54476/0 (socket says 172.21.15.29:54476) 2023-10-14T17:57:51.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 -- 172.21.15.29:0/248290384 learned_addr learned my addr 172.21.15.29:0/248290384 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 -- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 msgr2=0x56050f9791b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9791b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 -- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f952b10 msgr2=0x56050f952f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f952b10 0x56050f952f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 -- 172.21.15.29:0/248290384 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56050f851910 con 0x56050f95aaf0 2023-10-14T17:57:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.669+0000 7ff250bfc640 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7ff23c02efd0 tx=0x7ff23c00d040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3138286bf7194e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff22effd640 1 -- 172.21.15.29:0/248290384 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c0812e0 con 0x56050f95aaf0 2023-10-14T17:57:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff22effd640 1 -- 172.21.15.29:0/248290384 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c00bc10 con 0x56050f95aaf0 2023-10-14T17:57:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff22effd640 1 -- 172.21.15.29:0/248290384 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff23c0a88a0 con 0x56050f95aaf0 2023-10-14T17:57:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 -- 172.21.15.29:0/248290384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 msgr2=0x56050f95af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 secure :-1 s=READY pgs=1998 cs=0 l=1 rev1=1 crypto rx=0x7ff23c02efd0 tx=0x7ff23c00d040 comp rx=0 tx=0).stop 2023-10-14T17:57:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 -- 172.21.15.29:0/248290384 shutdown_connections 2023-10-14T17:57:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f952b10 0x56050f952f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f95af70 unknown :-1 s=CLOSED pgs=1998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 --2- 172.21.15.29:0/248290384 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9791b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 -- 172.21.15.29:0/248290384 >> 172.21.15.29:0/248290384 conn(0x56050f92c9a0 msgr2=0x56050f978aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.670+0000 7ff2561f2740 1 -- 172.21.15.29:0/248290384 shutdown_connections 2023-10-14T17:57:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.671+0000 7ff2561f2740 1 -- 172.21.15.29:0/248290384 wait complete. 2023-10-14T17:57:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.671+0000 7ff2561f2740 1 -- start start 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.671+0000 7ff2561f2740 1 Processor -- start 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f94db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95aaf0 0x56050f952f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f88d720 con 0x56050f94f640 2023-10-14T17:57:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f82b970 con 0x56050f95a710 2023-10-14T17:57:51.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.672+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56050f82b610 con 0x56050f95aaf0 2023-10-14T17:57:51.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff2561f2740 10 client.0 _ll_register_callbacks cb 0x56050f84d9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:57:51.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54492/0 (socket says 172.21.15.29:54492) 2023-10-14T17:57:51.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 -- 172.21.15.29:0/2194242997 learned_addr learned my addr 172.21.15.29:0/2194242997 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f94db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 -- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95aaf0 msgr2=0x56050f952f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95aaf0 0x56050f952f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 -- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 msgr2=0x56050f94db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f94db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 -- 172.21.15.29:0/2194242997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff24c0c3fb0 con 0x56050f94f640 2023-10-14T17:57:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0c24b0 tx=0x7ff244007d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45c450e9a0282e28 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.673+0000 7ff227fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:57:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400e040 con 0x56050f94f640 2023-10-14T17:57:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff244012b20 con 0x56050f94f640 2023-10-14T17:57:51.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x56050f8b4160 con 0x56050f94f640 2023-10-14T17:57:51.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff244012d20 con 0x56050f94f640 2023-10-14T17:57:51.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 10 client.9741 fetch_fsmap learned FSMap version 83 2023-10-14T17:57:51.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x56050f88bda0 con 0x56050f94f640 2023-10-14T17:57:51.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x56050f8b4160 con 0x56050f94f640 2023-10-14T17:57:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 83) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff244014040 con 0x56050f94f640 2023-10-14T17:57:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 10 client.9741 fetch_fsmap finished waiting for FSMap version 83 2023-10-14T17:57:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 10 client.9741 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:57:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 20 client.9741 populate_metadata read hostname 'smithi029' 2023-10-14T17:57:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96e0f0 0x56050f94e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 4 client.9741 mds_command: new command op to 9615 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T17:57:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.674+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 --> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56050f85fcc0 con 0x56050f96e0f0 2023-10-14T17:57:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.675+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96e0f0 0x56050f94e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.675+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96e0f0 0x56050f94e0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a874c7e6f40fb7c2 server_cookie=2d153836e3e7168 in_seq=0 out_seq=0 2023-10-14T17:57:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.675+0000 7ff227fff640 10 client.9741 ms_handle_connect on v2:172.21.15.29:6834/494120094 2023-10-14T17:57:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.676+0000 7ff227fff640 1 -- 172.21.15.29:0/2194242997 <== mds.0 v2:172.21.15.29:6834/494120094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x56050f85fcc0 con 0x56050f96e0f0 2023-10-14T17:57:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.676+0000 7ff227fff640 10 client.9741 handle_command_reply: tid=0 2023-10-14T17:57:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.676+0000 7ff2561f2740 1 client.9741 shutdown 2023-10-14T17:57:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.676+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 msgr2=0x56050f977a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.676+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 secure :-1 s=READY pgs=1999 cs=0 l=1 rev1=1 crypto rx=0x7ff24c0c24b0 tx=0x7ff244007d60 comp rx=0 tx=0).stop 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 shutdown_connections 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96e0f0 0x56050f94e0e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f977a20 unknown :-1 s=CLOSED pgs=1999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95aaf0 0x56050f952f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2194242997 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f94db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff227fff640 0 client.9741 ms_handle_reset on v2:172.21.15.29:6834/494120094 2023-10-14T17:57:51.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 >> 172.21.15.29:0/2194242997 conn(0x56050f952b10 msgr2=0x56050f9564f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 shutdown_connections 2023-10-14T17:57:51.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 1 -- 172.21.15.29:0/2194242997 wait complete. 2023-10-14T17:57:51.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.677+0000 7ff2561f2740 20 client.9741 trim_cache size 0 max 16384 2023-10-14T17:57:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.694+0000 7ff2561f2740 1 Processor -- start 2023-10-14T17:57:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.694+0000 7ff2561f2740 1 -- start start 2023-10-14T17:57:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f94f640 0x56050f977b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f92a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f5f1f20 con 0x56050f95aaf0 2023-10-14T17:57:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f997a90 con 0x56050f95a710 2023-10-14T17:57:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56050f976030 con 0x56050f94f640 2023-10-14T17:57:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff251bfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff251bfe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54506/0 (socket says 172.21.15.29:54506) 2023-10-14T17:57:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff251bfe640 1 -- 172.21.15.29:0/2981352612 learned_addr learned my addr 172.21.15.29:0/2981352612 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.695+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f92a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f94f640 0x56050f977b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 -- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f94f640 msgr2=0x56050f977b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f94f640 0x56050f977b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 -- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 msgr2=0x56050f92a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f92a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 -- 172.21.15.29:0/2981352612 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56050f82ab80 con 0x56050f95aaf0 2023-10-14T17:57:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7ff244003a10 tx=0x7ff244010000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ea6ce2e17bb812a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff22f7fe640 1 -- 172.21.15.29:0/2981352612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff24400e050 con 0x56050f95aaf0 2023-10-14T17:57:51.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff22f7fe640 1 -- 172.21.15.29:0/2981352612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff244012b20 con 0x56050f95aaf0 2023-10-14T17:57:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.696+0000 7ff22f7fe640 1 -- 172.21.15.29:0/2981352612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff244029450 con 0x56050f95aaf0 2023-10-14T17:57:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 -- 172.21.15.29:0/2981352612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 msgr2=0x56050f9767e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 secure :-1 s=READY pgs=2000 cs=0 l=1 rev1=1 crypto rx=0x7ff244003a10 tx=0x7ff244010000 comp rx=0 tx=0).stop 2023-10-14T17:57:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 -- 172.21.15.29:0/2981352612 shutdown_connections 2023-10-14T17:57:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f95aaf0 0x56050f9767e0 unknown :-1 s=CLOSED pgs=2000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95a710 0x56050f92a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2981352612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f94f640 0x56050f977b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 -- 172.21.15.29:0/2981352612 >> 172.21.15.29:0/2981352612 conn(0x56050f952b10 msgr2=0x56050f92b150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 -- 172.21.15.29:0/2981352612 shutdown_connections 2023-10-14T17:57:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.697+0000 7ff2561f2740 1 -- 172.21.15.29:0/2981352612 wait complete. 2023-10-14T17:57:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.698+0000 7ff2561f2740 1 -- start start 2023-10-14T17:57:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.698+0000 7ff2561f2740 1 Processor -- start 2023-10-14T17:57:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9dc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95aaf0 0x56050f96aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f9de650 con 0x56050f94f640 2023-10-14T17:57:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x56050f94eaa0 con 0x56050f95aaf0 2023-10-14T17:57:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x56050f96e9a0 con 0x56050f95a710 2023-10-14T17:57:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2561f2740 10 client.0 _ll_register_callbacks cb 0x7ff244007560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:57:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2513fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2513fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54516/0 (socket says 172.21.15.29:54516) 2023-10-14T17:57:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.699+0000 7ff2513fd640 1 -- 172.21.15.29:0/2029034031 learned_addr learned my addr 172.21.15.29:0/2029034031 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff251bfe640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95aaf0 0x56050f96aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 -- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 msgr2=0x56050f9dc930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9dc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9dc930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 -- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95aaf0 msgr2=0x56050f96aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95aaf0 0x56050f96aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 -- 172.21.15.29:0/2029034031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56050f88bda0 con 0x56050f94f640 2023-10-14T17:57:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2513fd640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x56050f94e090 tx=0x7ff24c06f7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55dedeb85e662983 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff227fff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:57:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff248014600 con 0x56050f94f640 2023-10-14T17:57:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff248014760 con 0x56050f94f640 2023-10-14T17:57:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.700+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff23c00bc10 con 0x56050f94f640 2023-10-14T17:57:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=83) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff248014960 con 0x56050f94f640 2023-10-14T17:57:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 10 client.9747 fetch_fsmap learned FSMap version 83 2023-10-14T17:57:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=83,monmap=2+}) v3 -- 0x56050f850170 con 0x56050f94f640 2023-10-14T17:57:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23c00bc10 con 0x56050f94f640 2023-10-14T17:57:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 83) v1 ==== 3294+0+0 (secure 0 0 0) 0x7ff248021070 con 0x56050f94f640 2023-10-14T17:57:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 10 client.9747 fetch_fsmap finished waiting for FSMap version 83 2023-10-14T17:57:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 10 client.9747 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:57:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 20 client.9747 populate_metadata read hostname 'smithi029' 2023-10-14T17:57:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96c5b0 0x56050f96c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 4 client.9747 mds_command: new command op to 9615 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T17:57:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 --> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56050f886700 con 0x56050f96c5b0 2023-10-14T17:57:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.701+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96c5b0 0x56050f96c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff250bfc640 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96c5b0 0x56050f96c990 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfeba56bcfc78b65 server_cookie=8cadae290a55aeb2 in_seq=0 out_seq=0 2023-10-14T17:57:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff227fff640 10 client.9747 ms_handle_connect on v2:172.21.15.29:6834/494120094 2023-10-14T17:57:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff227fff640 1 -- 172.21.15.29:0/2029034031 <== mds.0 v2:172.21.15.29:6834/494120094 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56050f886700 con 0x56050f96c5b0 2023-10-14T17:57:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff227fff640 10 client.9747 handle_command_reply: tid=0 2023-10-14T17:57:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 client.9747 shutdown 2023-10-14T17:57:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 msgr2=0x56050f9302a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 secure :-1 s=READY pgs=2001 cs=0 l=1 rev1=1 crypto rx=0x56050f94e090 tx=0x7ff24c06f7c0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 shutdown_connections 2023-10-14T17:57:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:6834/494120094,v1:172.21.15.29:6836/494120094] conn(0x56050f96c5b0 0x56050f96c990 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.702+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x56050f95aaf0 0x56050f96aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x56050f95a710 0x56050f9dc930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 1 --2- 172.21.15.29:0/2029034031 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x56050f94f640 0x56050f9302a0 unknown :-1 s=CLOSED pgs=2001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 >> 172.21.15.29:0/2029034031 conn(0x56050f952b10 msgr2=0x56050f94d000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff227fff640 0 client.9747 ms_handle_reset on v2:172.21.15.29:6834/494120094 2023-10-14T17:57:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 shutdown_connections 2023-10-14T17:57:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 1 -- 172.21.15.29:0/2029034031 wait complete. 2023-10-14T17:57:51.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff2561f2740 20 client.9747 trim_cache size 0 max 16384 2023-10-14T17:57:51.725 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:57:51.726 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T17:57:51.726 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T17:57:51.726 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 9615, 2023-10-14T17:57:51.726 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T17:57:51.726 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.7742418620000002, 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 83, 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 178, 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 178, 2023-10-14T17:57:51.727 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.135891073 2023-10-14T17:57:51.728 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:57:51.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 msgr2=0x7ff23006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 0x7ff23006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff248001120 tx=0x7ff248009040 comp rx=0 tx=0).stop 2023-10-14T17:57:51.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 msgr2=0x7ff24c0c3530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.703+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 secure :-1 s=READY pgs=1997 cs=0 l=1 rev1=1 crypto rx=0x7ff23c039040 tx=0x7ff23c00b750 comp rx=0 tx=0).stop 2023-10-14T17:57:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 shutdown_connections 2023-10-14T17:57:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff23006af90 0x7ff23006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff24c15cfb0 0x7ff24c0c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff24c15cb50 0x7ff24c0c3530 unknown :-1 s=CLOSED pgs=1997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 --2- 172.21.15.29:0/2811984478 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2380047c0 0x7ff24c1e7030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 >> 172.21.15.29:0/2811984478 conn(0x7ff24c0ce9b0 msgr2=0x7ff24c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 shutdown_connections 2023-10-14T17:57:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.704+0000 7ff253661640 1 -- 172.21.15.29:0/2811984478 wait complete. 2023-10-14T17:57:51.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T17:57:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.959+0000 7fcdfd5ac640 1 Processor -- start 2023-10-14T17:57:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.959+0000 7fcdfd5ac640 1 -- start start 2023-10-14T17:57:51.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.960+0000 7fcdfd5ac640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcdf80cd380 0x7fcdf80cd810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.960+0000 7fcdfd5ac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.960+0000 7fcdfd5ac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdf806bab0 con 0x7fcdf81599c0 2023-10-14T17:57:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.960+0000 7fcdfd5ac640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdf80cdd50 con 0x7fcdf80cd380 2023-10-14T17:57:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.960+0000 7fcdfd5ac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf806f300 con 0x7fcdf806f750 2023-10-14T17:57:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.961+0000 7fcdfcdab640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.961+0000 7fcdeffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcdf80cd380 0x7fcdf80cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.961+0000 7fcdfcdab640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:37006/0 (socket says 172.21.15.29:37006) 2023-10-14T17:57:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.961+0000 7fcdfcdab640 1 -- 172.21.15.29:0/983343932 learned_addr learned my addr 172.21.15.29:0/983343932 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.962+0000 7fcdfcdab640 1 -- 172.21.15.29:0/983343932 >> v2:172.21.15.186:3300/0 conn(0x7fcdf80cd380 msgr2=0x7fcdf80cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.962+0000 7fcdfcdab640 1 --2- 172.21.15.29:0/983343932 >> v2:172.21.15.186:3300/0 conn(0x7fcdf80cd380 0x7fcdf80cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.962+0000 7fcdfcdab640 1 -- 172.21.15.29:0/983343932 >> v1:172.21.15.186:6789/0 conn(0x7fcdf81599c0 legacy=0x7fcdf80ca1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.962+0000 7fcdfcdab640 1 -- 172.21.15.29:0/983343932 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf806bd90 con 0x7fcdf806f750 2023-10-14T17:57:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.963+0000 7fcdfcdab640 1 --2- 172.21.15.29:0/983343932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009e50 tx=0x7fcdf002e060 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c558c2f177b9f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.964+0000 7fcdf77fe640 1 -- 172.21.15.29:0/983343932 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf002ec00 con 0x7fcdf806f750 2023-10-14T17:57:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.964+0000 7fcdf77fe640 1 -- 172.21.15.29:0/983343932 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf002ed60 con 0x7fcdf806f750 2023-10-14T17:57:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.964+0000 7fcdf77fe640 1 -- 172.21.15.29:0/983343932 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdf0036610 con 0x7fcdf806f750 2023-10-14T17:57:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.964+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/983343932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 msgr2=0x7fcdf806fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.964+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/983343932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fcdf0009e50 tx=0x7fcdf002e060 comp rx=0 tx=0).stop 2023-10-14T17:57:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/983343932 shutdown_connections 2023-10-14T17:57:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/983343932 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf806fbe0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/983343932 >> v2:172.21.15.186:3300/0 conn(0x7fcdf80cd380 0x7fcdf80cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/983343932 >> 172.21.15.29:0/983343932 conn(0x7fcdf80b0a90 msgr2=0x7fcdf80ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/983343932 shutdown_connections 2023-10-14T17:57:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.965+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/983343932 wait complete. 2023-10-14T17:57:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.966+0000 7fcdfd5ac640 1 Processor -- start 2023-10-14T17:57:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.967+0000 7fcdfd5ac640 1 -- start start 2023-10-14T17:57:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf8157780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf81599c0 0x7fcdf80a8c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80b4140 con 0x7fcdf80cd380 2023-10-14T17:57:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80abd50 con 0x7fcdf81599c0 2023-10-14T17:57:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfd5ac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf80ac030 con 0x7fcdf806f750 2023-10-14T17:57:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54530/0 (socket says 172.21.15.29:54530) 2023-10-14T17:57:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 -- 172.21.15.29:0/2171831021 learned_addr learned my addr 172.21.15.29:0/2171831021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdfcdab640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf81599c0 0x7fcdf80a8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdf7fff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf8157780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 -- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 msgr2=0x7fcdf8157780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf8157780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 -- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf81599c0 msgr2=0x7fcdf80a8c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf81599c0 0x7fcdf80a8c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.968+0000 7fcdeffff640 1 -- 172.21.15.29:0/2171831021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0009b30 con 0x7fcdf80cd380 2023-10-14T17:57:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.969+0000 7fcdeffff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcde000e9b0 tx=0x7fcde000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.969+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde0009670 con 0x7fcdf80cd380 2023-10-14T17:57:51.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.969+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcde000c9a0 con 0x7fcdf80cd380 2023-10-14T17:57:51.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.969+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf80ac310 con 0x7fcdf80cd380 2023-10-14T17:57:51.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.969+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcde000cd50 con 0x7fcdf80cd380 2023-10-14T17:57:51.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.970+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf80ac8d0 con 0x7fcdf80cd380 2023-10-14T17:57:51.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.971+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf80aca90 con 0x7fcdf80cd380 2023-10-14T17:57:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.972+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcde0013470 con 0x7fcdf80cd380 2023-10-14T17:57:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.973+0000 7fcdf57fa640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 0x7fcdd806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.973+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcde0089b00 con 0x7fcdf80cd380 2023-10-14T17:57:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.973+0000 7fcdf7fff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 0x7fcdd806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.974+0000 7fcdf7fff640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 0x7fcdd806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcde8004ea0 tx=0x7fcde8009040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:51.979+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcde008be60 con 0x7fcdf80cd380 2023-10-14T17:57:52.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.216+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fcdf80ca840 con 0x7fcdf80cd380 2023-10-14T17:57:52.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.217+0000 7fcdf57fa640 1 -- 172.21.15.29:0/2171831021 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v83) v1 ==== 53+0+73 (secure 0 0 0) 0x7fcde0017070 con 0x7fcdf80cd380 2023-10-14T17:57:52.219 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T17:57:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.219+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 msgr2=0x7fcdd806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.219+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 0x7fcdd806d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fcde8004ea0 tx=0x7fcde8009040 comp rx=0 tx=0).stop 2023-10-14T17:57:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.219+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 msgr2=0x7fcdf80b3c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.219+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 secure :-1 s=READY pgs=2002 cs=0 l=1 rev1=1 crypto rx=0x7fcde000e9b0 tx=0x7fcde000ee70 comp rx=0 tx=0).stop 2023-10-14T17:57:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 shutdown_connections 2023-10-14T17:57:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fcdd806af90 0x7fcdd806d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdf81599c0 0x7fcdf80a8c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdf80cd380 0x7fcdf80b3c00 unknown :-1 s=CLOSED pgs=2002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 --2- 172.21.15.29:0/2171831021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcdf806f750 0x7fcdf8157780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 >> 172.21.15.29:0/2171831021 conn(0x7fcdf80b0a90 msgr2=0x7fcdf80cac40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 shutdown_connections 2023-10-14T17:57:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:52.220+0000 7fcdfd5ac640 1 -- 172.21.15.29:0/2171831021 wait complete. 2023-10-14T17:57:52.240 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T17:57:52.240 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T17:57:52.240 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T17:57:52.241 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T17:57:52.241 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T17:57:52.242 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T17:57:52.242 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T17:57:52.242 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T17:57:52.243 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:57:52.243 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.243 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T17:57:52.298 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T17:57:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T17:57:52.298 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T17:57:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T17:57:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.299 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:57:52.299 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:57:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T17:57:52.300 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:57:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:57:52.300 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T17:57:52.300 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T17:57:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T17:57:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T17:57:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T17:57:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37704sec preferred_lft 37704sec 2023-10-14T17:57:52.301 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T17:57:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.302 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:57:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:57:52.302 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T17:57:52.302 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T17:57:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:57:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T17:57:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T17:57:52.303 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.304 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:57:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T17:57:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T17:57:52.304 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.304 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:57:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T17:57:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T17:57:52.305 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:57:52.306 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.306 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T17:57:52.403 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T17:57:52.407 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.408 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T17:57:52.463 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T17:57:52.464 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T17:57:52.464 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T17:57:52.464 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T17:57:52.465 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T17:57:52.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.467 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T17:57:52.467 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T17:57:52.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T17:57:52.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T17:57:52.468 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T17:57:52.468 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T17:57:52.667 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.668 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T17:57:52.724 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T17:57:52.724 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T17:57:52.726 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:57:52.726 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.727 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T17:57:52.830 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:57:52.830 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T17:57:52.892 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:57:52.893 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:57:52.894 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:57:52.954 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T17:57:52.955 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:52.955 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:57:53.102 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:57:53.102 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:57:53.102 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:57:53.126 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:57:53.126 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:57:53.126 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:57:53.127 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:57:53.127 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:57:53.153 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:57:53.153 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:57:53.153 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:57:53.153 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:57:53.154 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:57:53.154 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T17:57:53.154 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T17:57:53.154 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T17:57:53.157 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T17:57:53.157 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T17:57:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T17:57:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T17:57:53.158 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T17:57:53.162 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T17:57:53.163 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:53.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:57:53.259 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:53.260 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T17:57:53.331 INFO:teuthology.orchestra.run.smithi029.stdout:client6629 2023-10-14T17:57:53.332 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:57:53.332 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:57:53.388 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T17:57:53.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.610+0000 7fb14cbff640 1 Processor -- start 2023-10-14T17:57:53.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.611+0000 7fb14cbff640 1 -- start start 2023-10-14T17:57:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb14cbff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb14806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb14cbff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14815f4a0 con 0x7fb1480cda80 2023-10-14T17:57:53.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb14cbff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb14806c960 con 0x7fb1480ad970 2023-10-14T17:57:53.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb14cbff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815b9d0 con 0x7fb14806c040 2023-10-14T17:57:53.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb147fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1480cda80 0x7fb14815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59200/0 (socket says 172.21.15.29:59200) 2023-10-14T17:57:53.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.612+0000 7fb147fff640 1 -- 172.21.15.29:0/1705794140 learned_addr learned my addr 172.21.15.29:0/1705794140 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:53.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1477fe640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb14806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088065129 0 0) 0x7fb14815f4a0 con 0x7fb1480cda80 2023-10-14T17:57:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb130003520 con 0x7fb1480cda80 2023-10-14T17:57:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1200963478 0 0) 0x7fb130003520 con 0x7fb1480cda80 2023-10-14T17:57:53.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 msgr2=0x7fb14806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb14806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 >> v1:172.21.15.186:6789/0 conn(0x7fb1480ad970 legacy=0x7fb14815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb14815bcb0 con 0x7fb1480cda80 2023-10-14T17:57:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb1400019f0 con 0x7fb1480cda80 2023-10-14T17:57:53.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 >> v1:172.21.15.29:6789/0 conn(0x7fb1480cda80 legacy=0x7fb14815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1300039a0 0x7fb130003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb1300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb130004d00 0x7fb130005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815b9d0 con 0x7fb130004d00 2023-10-14T17:57:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13000ba30 con 0x7fb1300039a0 2023-10-14T17:57:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.613+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13000bbf0 con 0x7fb14806c040 2023-10-14T17:57:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb130004d00 0x7fb130005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb1477fe640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1300039a0 0x7fb130003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 -- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 msgr2=0x7fb1300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb146ffd640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb1300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb1300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 -- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1300039a0 msgr2=0x7fb130003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1300039a0 0x7fb130003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb147fff640 1 -- 172.21.15.29:0/1705794140 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb130003520 con 0x7fb130004d00 2023-10-14T17:57:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.614+0000 7fb146ffd640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb14806c040 0x7fb1300047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.615+0000 7fb147fff640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb130004d00 0x7fb130005180 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fb14000a800 tx=0x7fb1300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68ac748881b2103 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1400072d0 con 0x7fb130004d00 2023-10-14T17:57:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb14002f070 con 0x7fb130004d00 2023-10-14T17:57:53.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb1467fc640 1 -- 172.21.15.29:0/1705794140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1400076b0 con 0x7fb130004d00 2023-10-14T17:57:53.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb14cbff640 1 -- 172.21.15.29:0/1705794140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb130004d00 msgr2=0x7fb130005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb14cbff640 1 --2- 172.21.15.29:0/1705794140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb130004d00 0x7fb130005180 secure :-1 s=READY pgs=2004 cs=0 l=1 rev1=1 crypto rx=0x7fb14000a800 tx=0x7fb1300057f0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb147fff640 1 -- 172.21.15.29:0/1705794140 reap_dead start 2023-10-14T17:57:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb14cbff640 1 -- 172.21.15.29:0/1705794140 shutdown_connections 2023-10-14T17:57:53.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb14cbff640 1 -- 172.21.15.29:0/1705794140 >> 172.21.15.29:0/1705794140 conn(0x7fb1480ce6a0 msgr2=0x7fb1480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.616+0000 7fb14cbff640 1 -- 172.21.15.29:0/1705794140 shutdown_connections 2023-10-14T17:57:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.617+0000 7fb14cbff640 1 -- 172.21.15.29:0/1705794140 wait complete. 2023-10-14T17:57:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.618+0000 7fb14cbff640 1 Processor -- start 2023-10-14T17:57:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 -- start start 2023-10-14T17:57:53.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb130004d00 0x7fb148155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 0x7fb14815a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1481594e0 con 0x7fb14813dfd0 2023-10-14T17:57:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb14815aca0 con 0x7fb14815a2e0 2023-10-14T17:57:53.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.619+0000 7fb14cbff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb148154890 con 0x7fb130004d00 2023-10-14T17:57:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54538/0 (socket says 172.21.15.29:54538) 2023-10-14T17:57:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 -- 172.21.15.29:0/3143620496 learned_addr learned my addr 172.21.15.29:0/3143620496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb147fff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 0x7fb14815a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb1477fe640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb130004d00 0x7fb148155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 -- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb130004d00 msgr2=0x7fb148155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb130004d00 0x7fb148155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 -- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 msgr2=0x7fb14815a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 0x7fb14815a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb146ffd640 1 -- 172.21.15.29:0/3143620496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb148154b70 con 0x7fb14813dfd0 2023-10-14T17:57:53.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.620+0000 7fb147fff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 0x7fb14815a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:53.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb146ffd640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00d7a0 tx=0x7fb13c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:53.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c017070 con 0x7fb14813dfd0 2023-10-14T17:57:53.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb13c005070 con 0x7fb14813dfd0 2023-10-14T17:57:53.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb13c005420 con 0x7fb14813dfd0 2023-10-14T17:57:53.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb148154e50 con 0x7fb14813dfd0 2023-10-14T17:57:53.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.621+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb148155410 con 0x7fb14813dfd0 2023-10-14T17:57:53.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.622+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1481555d0 con 0x7fb14813dfd0 2023-10-14T17:57:53.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.623+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb13c007030 con 0x7fb14813dfd0 2023-10-14T17:57:53.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.624+0000 7fb12ffff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:53.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.624+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb13c0892a0 con 0x7fb14813dfd0 2023-10-14T17:57:53.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.625+0000 7fb1477fe640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:53.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.625+0000 7fb1477fe640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 0x7fb13406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb138009a10 tx=0x7fb138008040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:53.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.629+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb13c054010 con 0x7fb14813dfd0 2023-10-14T17:57:53.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.852+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 --> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1480cc270 con 0x7fb13406af90 2023-10-14T17:57:53.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.855+0000 7fb12ffff640 1 -- 172.21.15.29:0/3143620496 <== mgr.7201 v2:172.21.15.29:6832/2441704285 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb138013960 con 0x7fb13406af90 2023-10-14T17:57:53.858 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:57:53.858 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:57:53.858 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T17:57:53.858 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:57:53.858 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:57:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.858+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 msgr2=0x7fb13406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.858+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 0x7fb13406d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb138009a10 tx=0x7fb138008040 comp rx=0 tx=0).stop 2023-10-14T17:57:53.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 msgr2=0x7fb1481f54b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 secure :-1 s=READY pgs=2005 cs=0 l=1 rev1=1 crypto rx=0x7fb13c00d7a0 tx=0x7fb13c00dc60 comp rx=0 tx=0).stop 2023-10-14T17:57:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 shutdown_connections 2023-10-14T17:57:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fb13406af90 0x7fb13406d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb14815a2e0 0x7fb14815a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb14813dfd0 0x7fb1481f54b0 unknown :-1 s=CLOSED pgs=2005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 --2- 172.21.15.29:0/3143620496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb130004d00 0x7fb148155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 >> 172.21.15.29:0/3143620496 conn(0x7fb1480ce6a0 msgr2=0x7fb1480ca720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 shutdown_connections 2023-10-14T17:57:53.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:53.859+0000 7fb14cbff640 1 -- 172.21.15.29:0/3143620496 wait complete. 2023-10-14T17:57:53.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T17:57:54.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.118+0000 7fc600d06640 1 Processor -- start 2023-10-14T17:57:54.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.118+0000 7fc600d06640 1 -- start start 2023-10-14T17:57:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc600d06640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc600d06640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc0c9230 0x7fc5fc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc600d06640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5fc160a90 con 0x7fc5fc070f20 2023-10-14T17:57:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc600d06640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5fc0cdec0 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc600d06640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0cd900 con 0x7fc5fc0c9230 2023-10-14T17:57:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc5fb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc5fb7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54540/0 (socket says 172.21.15.29:54540) 2023-10-14T17:57:54.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.120+0000 7fc5fb7fe640 1 -- 172.21.15.29:0/3834003960 learned_addr learned my addr 172.21.15.29:0/3834003960 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.121+0000 7fc5faffd640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc0c9230 0x7fc5fc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.121+0000 7fc5fb7fe640 1 -- 172.21.15.29:0/3834003960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc0c9230 msgr2=0x7fc5fc0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.121+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc0c9230 0x7fc5fc0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.121+0000 7fc5fb7fe640 1 -- 172.21.15.29:0/3834003960 >> v1:172.21.15.186:6789/0 conn(0x7fc5fc070f20 legacy=0x7fc5fc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.121+0000 7fc5fb7fe640 1 -- 172.21.15.29:0/3834003960 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5fc0cdbe0 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.122+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009870 tx=0x7fc5ec02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a19846c3a25404b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc5fa7fc640 1 -- 172.21.15.29:0/3834003960 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec030840 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc5fa7fc640 1 -- 172.21.15.29:0/3834003960 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5ec0309a0 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc5fa7fc640 1 -- 172.21.15.29:0/3834003960 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5ec030d30 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 -- 172.21.15.29:0/3834003960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc15c9f0 msgr2=0x7fc5fc0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:54.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 secure :-1 s=READY pgs=2006 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec009870 tx=0x7fc5ec02b980 comp rx=0 tx=0).stop 2023-10-14T17:57:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 -- 172.21.15.29:0/3834003960 shutdown_connections 2023-10-14T17:57:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc0c9230 0x7fc5fc0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 --2- 172.21.15.29:0/3834003960 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c8cf0 unknown :-1 s=CLOSED pgs=2006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 -- 172.21.15.29:0/3834003960 >> 172.21.15.29:0/3834003960 conn(0x7fc5fc0ce6a0 msgr2=0x7fc5fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:54.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 -- 172.21.15.29:0/3834003960 shutdown_connections 2023-10-14T17:57:54.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.123+0000 7fc600d06640 1 -- 172.21.15.29:0/3834003960 wait complete. 2023-10-14T17:57:54.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.125+0000 7fc600d06640 1 Processor -- start 2023-10-14T17:57:54.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.125+0000 7fc600d06640 1 -- start start 2023-10-14T17:57:54.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 0x7fc5fc159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc15a050 con 0x7fc5fc0c9230 2023-10-14T17:57:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc0c85c0 con 0x7fc5fc070f20 2023-10-14T17:57:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.126+0000 7fc600d06640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5fc15fc20 con 0x7fc5fc15c9f0 2023-10-14T17:57:54.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54542/0 (socket says 172.21.15.29:54542) 2023-10-14T17:57:54.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 -- 172.21.15.29:0/3157685725 learned_addr learned my addr 172.21.15.29:0/3157685725 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:54.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 0x7fc5fc159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 -- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc15c9f0 msgr2=0x7fc5fc0c6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:54.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5fbfff640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c6d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 -- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 msgr2=0x7fc5fc159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 0x7fc5fc159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:54.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5faffd640 1 -- 172.21.15.29:0/3157685725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5ec0094d0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.127+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 0x7fc5fc159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc5faffd640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000d830 tx=0x7fc5f000dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f000b6c0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5f000b820 con 0x7fc5fc0c9230 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5fc0c7ae0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5f000bbb0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.128+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5fc0c80a0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.129+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5fc0c8260 con 0x7fc5fc0c9230 2023-10-14T17:57:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.130+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 93) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc5f001e460 con 0x7fc5fc0c9230 2023-10-14T17:57:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.131+0000 7fc5e3fff640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 0x7fc5e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:54.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.131+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc5f0089fa0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.132+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 0x7fc5e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:54.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.132+0000 7fc5fb7fe640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 0x7fc5e806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03b040 comp rx=0 tx=0).ready entity=mgr.7201 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:54.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.136+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc5f0054eb0 con 0x7fc5fc0c9230 2023-10-14T17:57:54.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.383+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7fc5fc04e9b0 con 0x7fc5fc0c9230 2023-10-14T17:57:55.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:54.999+0000 7fc5e3fff640 1 -- 172.21.15.29:0/3157685725 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v94) v1 ==== 92+0+0 (secure 0 0 0) 0x7fc5f00584d0 con 0x7fc5fc0c9230 2023-10-14T17:57:55.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 msgr2=0x7fc5e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 0x7fc5e806d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5ec036000 tx=0x7fc5ec03b040 comp rx=0 tx=0).stop 2023-10-14T17:57:55.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 msgr2=0x7fc5fc0c17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 secure :-1 s=READY pgs=2007 cs=0 l=1 rev1=1 crypto rx=0x7fc5f000d830 tx=0x7fc5f000dcf0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 shutdown_connections 2023-10-14T17:57:55.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7fc5e806af90 0x7fc5e806d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc5fc15c9f0 0x7fc5fc0c6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc5fc0c9230 0x7fc5fc0c17f0 unknown :-1 s=CLOSED pgs=2007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 --2- 172.21.15.29:0/3157685725 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc5fc070f20 0x7fc5fc159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 >> 172.21.15.29:0/3157685725 conn(0x7fc5fc0ce6a0 msgr2=0x7fc5fc15ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:55.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.003+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 shutdown_connections 2023-10-14T17:57:55.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.004+0000 7fc600d06640 1 -- 172.21.15.29:0/3157685725 wait complete. 2023-10-14T17:57:55.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T17:57:55.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.235+0000 7f2f055f4640 1 Processor -- start 2023-10-14T17:57:55.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.235+0000 7f2f055f4640 1 -- start start 2023-10-14T17:57:55.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f055f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 0x7f2f000c9e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f055f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f0015a9b0 con 0x7f2f000ca850 2023-10-14T17:57:55.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f055f4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f000ca420 con 0x7f2f00157740 2023-10-14T17:57:55.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f055f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f0015a360 con 0x7f2f000cfe90 2023-10-14T17:57:55.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f04df3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2f000ca850 0x7f2f0015d600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59208/0 (socket says 172.21.15.29:59208) 2023-10-14T17:57:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2f04df3640 1 -- 172.21.15.29:0/1069380693 learned_addr learned my addr 172.21.15.29:0/1069380693 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.237+0000 7f2ef77fe640 1 --2- 172.21.15.29:0/1069380693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 0x7f2f000c9e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.238+0000 7f2eff7fe640 1 -- 172.21.15.29:0/1069380693 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296906968 0 0) 0x7f2f0015a9b0 con 0x7f2f000ca850 2023-10-14T17:57:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.238+0000 7f2eff7fe640 1 -- 172.21.15.29:0/1069380693 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eec005730 con 0x7f2f000ca850 2023-10-14T17:57:55.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.238+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1069380693 >> v1:172.21.15.186:6789/0 conn(0x7f2f00157740 legacy=0x7f2f000cf780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.238+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1069380693 >> v1:172.21.15.29:6789/0 conn(0x7f2f000ca850 legacy=0x7f2f0015d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.238+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1069380693 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0015a640 con 0x7f2f000cfe90 2023-10-14T17:57:55.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.239+0000 7f2ef77fe640 1 --2- 172.21.15.29:0/1069380693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 0x7f2f000c9e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097d0 tx=0x7f2ee802b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=705127a7fec22a04 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:55.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.240+0000 7f2eff7fe640 1 -- 172.21.15.29:0/1069380693 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031890 con 0x7f2f000cfe90 2023-10-14T17:57:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.240+0000 7f2eff7fe640 1 -- 172.21.15.29:0/1069380693 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee80319f0 con 0x7f2f000cfe90 2023-10-14T17:57:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.240+0000 7f2eff7fe640 1 -- 172.21.15.29:0/1069380693 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8031d60 con 0x7f2f000cfe90 2023-10-14T17:57:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.240+0000 7f2f055f4640 1 -- 172.21.15.29:0/1069380693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 msgr2=0x7f2f000c9e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.240+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1069380693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 0x7f2f000c9e40 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f2ee80097d0 tx=0x7f2ee802b930 comp rx=0 tx=0).stop 2023-10-14T17:57:55.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.241+0000 7f2f055f4640 1 -- 172.21.15.29:0/1069380693 shutdown_connections 2023-10-14T17:57:55.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.241+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1069380693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000cfe90 0x7f2f000c9e40 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.241+0000 7f2f055f4640 1 -- 172.21.15.29:0/1069380693 >> 172.21.15.29:0/1069380693 conn(0x7f2f0006aeb0 msgr2=0x7f2f0006b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:55.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.241+0000 7f2f055f4640 1 -- 172.21.15.29:0/1069380693 shutdown_connections 2023-10-14T17:57:55.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.241+0000 7f2f055f4640 1 -- 172.21.15.29:0/1069380693 wait complete. 2023-10-14T17:57:55.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.242+0000 7f2f055f4640 1 Processor -- start 2023-10-14T17:57:55.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.243+0000 7f2f055f4640 1 -- start start 2023-10-14T17:57:55.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.243+0000 7f2f055f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000ca850 0x7f2f001f66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f055f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f055f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 0x7f2f000cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f055f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001e7300 con 0x7f2f000cfe90 2023-10-14T17:57:55.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f055f4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001f6be0 con 0x7f2f00157740 2023-10-14T17:57:55.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f055f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f001e63e0 con 0x7f2f000ca850 2023-10-14T17:57:55.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54562/0 (socket says 172.21.15.29:54562) 2023-10-14T17:57:55.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1228795019 learned_addr learned my addr 172.21.15.29:0/1228795019 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:55.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2efffff640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000ca850 0x7f2f001f66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f04df3640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 0x7f2f000cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000ca850 msgr2=0x7f2f001f66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000ca850 0x7f2f001f66a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 msgr2=0x7f2f000cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 0x7f2f000cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2ef77fe640 1 -- 172.21.15.29:0/1228795019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8009400 con 0x7f2f000cfe90 2023-10-14T17:57:55.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.244+0000 7f2f04df3640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 0x7f2f000cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:57:55.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2ef77fe640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009720 tx=0x7f2ee80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:55.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee803cc00 con 0x7f2f000cfe90 2023-10-14T17:57:55.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ee803cd60 con 0x7f2f000cfe90 2023-10-14T17:57:55.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ee8004230 con 0x7f2f000cfe90 2023-10-14T17:57:55.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f001e6680 con 0x7f2f000cfe90 2023-10-14T17:57:55.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.245+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f001e6c40 con 0x7f2f000cfe90 2023-10-14T17:57:55.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.247+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f000b5060 con 0x7f2f000cfe90 2023-10-14T17:57:55.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.248+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f2ee8004390 con 0x7f2f000cfe90 2023-10-14T17:57:55.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.249+0000 7f2efd7fa640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 0x7f2ee006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.249+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2ee80ac9f0 con 0x7f2f000cfe90 2023-10-14T17:57:55.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.249+0000 7f2efffff640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 msgr2=0x7f2ee006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:55.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.249+0000 7f2efffff640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 0x7f2ee006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:55.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.257+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ee8077920 con 0x7f2f000cfe90 2023-10-14T17:57:55.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.448+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f2f0004e9b0 con 0x7f2f000cfe90 2023-10-14T17:57:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.450+0000 7f2efffff640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 msgr2=0x7f2ee006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:55.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.450+0000 7f2efffff640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 0x7f2ee006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:55.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.453+0000 7f2efd7fa640 1 -- 172.21.15.29:0/1228795019 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v59)=0 v59) v1 ==== 138+0+0 (secure 0 0 0) 0x7f2ee8035020 con 0x7f2f000cfe90 2023-10-14T17:57:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 msgr2=0x7f2ee006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 0x7f2ee006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 msgr2=0x7f2f00154780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 secure :-1 s=READY pgs=2012 cs=0 l=1 rev1=1 crypto rx=0x7f2ee8009720 tx=0x7f2ee80046c0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 shutdown_connections 2023-10-14T17:57:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f2ee006afe0 0x7f2ee006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f00157740 0x7f2f000cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f000cfe90 0x7f2f00154780 unknown :-1 s=CLOSED pgs=2012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 --2- 172.21.15.29:0/1228795019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f000ca850 0x7f2f001f66a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.455+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 >> 172.21.15.29:0/1228795019 conn(0x7f2f0006aeb0 msgr2=0x7f2f0006d150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.456+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 shutdown_connections 2023-10-14T17:57:55.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.456+0000 7f2f055f4640 1 -- 172.21.15.29:0/1228795019 wait complete. 2023-10-14T17:57:55.476 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T17:57:55.524 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:55.523+0000 7fd4d6eb6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:57:55.555 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:55.553+0000 7f15f88b6040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:57:55.578 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:55.576+0000 7fd4d6eb6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:57:55.613 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:55.610+0000 7f15f88b6040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:57:55.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.699+0000 7ff686021640 1 Processor -- start 2023-10-14T17:57:55.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.699+0000 7ff686021640 1 -- start start 2023-10-14T17:57:55.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.700+0000 7ff686021640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff680159e90 0x7ff6800ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.700+0000 7ff686021640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.700+0000 7ff686021640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff680070360 con 0x7ff680159ab0 2023-10-14T17:57:55.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.700+0000 7ff686021640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff68006cb10 con 0x7ff680159e90 2023-10-14T17:57:55.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.701+0000 7ff686021640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff68015d890 con 0x7ff68006c140 2023-10-14T17:57:55.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.701+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.701+0000 7ff67f7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff680159ab0 0x7ff68006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:34956/0 (socket says 172.21.15.29:34956) 2023-10-14T17:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.701+0000 7ff67f7fe640 1 -- 172.21.15.29:0/642407460 learned_addr learned my addr 172.21.15.29:0/642407460 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.701+0000 7ff677fff640 1 --2- 172.21.15.29:0/642407460 >> v2:172.21.15.186:3300/0 conn(0x7ff680159e90 0x7ff6800ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff677fff640 1 -- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 msgr2=0x7ff68006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff677fff640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff67effd640 1 -- 172.21.15.29:0/642407460 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1114652665 0 0) 0x7ff680070360 con 0x7ff680159ab0 2023-10-14T17:57:55.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff677fff640 1 -- 172.21.15.29:0/642407460 >> v1:172.21.15.186:6789/0 conn(0x7ff680159ab0 legacy=0x7ff68006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff677fff640 1 -- 172.21.15.29:0/642407460 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6800cd1f0 con 0x7ff680159e90 2023-10-14T17:57:55.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.702+0000 7ff67ffff640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T17:57:55.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.703+0000 7ff677fff640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159e90 0x7ff6800ce370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff6680097a0 tx=0x7ff66802b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d69643086bc0268f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:55.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff67effd640 1 -- 172.21.15.29:0/642407460 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6680316f0 con 0x7ff680159e90 2023-10-14T17:57:55.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff67effd640 1 -- 172.21.15.29:0/642407460 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff668031850 con 0x7ff680159e90 2023-10-14T17:57:55.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff67effd640 1 -- 172.21.15.29:0/642407460 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff668031be0 con 0x7ff680159e90 2023-10-14T17:57:55.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff686021640 1 -- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159e90 msgr2=0x7ff6800ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff686021640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159e90 0x7ff6800ce370 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7ff6680097a0 tx=0x7ff66802b790 comp rx=0 tx=0).stop 2023-10-14T17:57:55.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff686021640 1 -- 172.21.15.29:0/642407460 shutdown_connections 2023-10-14T17:57:55.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff686021640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.704+0000 7ff686021640 1 --2- 172.21.15.29:0/642407460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159e90 0x7ff6800ce370 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.705+0000 7ff686021640 1 -- 172.21.15.29:0/642407460 >> 172.21.15.29:0/642407460 conn(0x7ff6800ce9b0 msgr2=0x7ff6800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:55.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.705+0000 7ff686021640 1 -- 172.21.15.29:0/642407460 shutdown_connections 2023-10-14T17:57:55.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.705+0000 7ff686021640 1 -- 172.21.15.29:0/642407460 wait complete. 2023-10-14T17:57:55.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.706+0000 7ff686021640 1 Processor -- start 2023-10-14T17:57:55.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.706+0000 7ff686021640 1 -- start start 2023-10-14T17:57:55.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68014f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159ab0 0x7ff6801f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68014e430 con 0x7ff680159e90 2023-10-14T17:57:55.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff68015d070 con 0x7ff680159ab0 2023-10-14T17:57:55.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.707+0000 7ff686021640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6800c53b0 con 0x7ff68006c140 2023-10-14T17:57:55.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54576/0 (socket says 172.21.15.29:54576) 2023-10-14T17:57:55.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 -- 172.21.15.29:0/1131674391 learned_addr learned my addr 172.21.15.29:0/1131674391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:55.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff677fff640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159ab0 0x7ff6801f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67f7fe640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68014f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 msgr2=0x7ff68014f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68014f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159ab0 msgr2=0x7ff6801f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159ab0 0x7ff6801f3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 -- 172.21.15.29:0/1131674391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff678009400 con 0x7ff680159e90 2023-10-14T17:57:55.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.708+0000 7ff67ffff640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff67800e8b0 tx=0x7ff67800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:55.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.709+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff678009660 con 0x7ff680159e90 2023-10-14T17:57:55.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.709+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff678004020 con 0x7ff680159e90 2023-10-14T17:57:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.709+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6780043b0 con 0x7ff680159e90 2023-10-14T17:57:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.709+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff668009400 con 0x7ff680159e90 2023-10-14T17:57:55.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.709+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6800c50d0 con 0x7ff680159e90 2023-10-14T17:57:55.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.710+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff68014e5e0 con 0x7ff680159e90 2023-10-14T17:57:55.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.712+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff67800c940 con 0x7ff680159e90 2023-10-14T17:57:55.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.713+0000 7ff67cff9640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 0x7ff66006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:55.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.713+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff678089a80 con 0x7ff680159e90 2023-10-14T17:57:55.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.713+0000 7ff67f7fe640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 msgr2=0x7ff66006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:55.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.713+0000 7ff67f7fe640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 0x7ff66006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:55.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.721+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff68014e5e0 con 0x7ff680159e90 2023-10-14T17:57:55.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.914+0000 7ff67f7fe640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 msgr2=0x7ff66006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:55.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.914+0000 7ff67f7fe640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 0x7ff66006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:55.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.917+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7ff6800cda60 con 0x7ff680159e90 2023-10-14T17:57:55.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.922+0000 7ff67cff9640 1 -- 172.21.15.29:0/1131674391 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v60)=0 v60) v1 ==== 133+0+0 (secure 0 0 0) 0x7ff6780200c0 con 0x7ff680159e90 2023-10-14T17:57:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 msgr2=0x7ff66006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:55.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 0x7ff66006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 msgr2=0x7ff6801f4250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 secure :-1 s=READY pgs=2013 cs=0 l=1 rev1=1 crypto rx=0x7ff67800e8b0 tx=0x7ff67800ed70 comp rx=0 tx=0).stop 2023-10-14T17:57:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 shutdown_connections 2023-10-14T17:57:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7ff66006b010 0x7ff66006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff680159e90 0x7ff6801f4250 unknown :-1 s=CLOSED pgs=2013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff680159ab0 0x7ff6801f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 --2- 172.21.15.29:0/1131674391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff68006c140 0x7ff68014f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 >> 172.21.15.29:0/1131674391 conn(0x7ff6800ce9b0 msgr2=0x7ff6800ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 shutdown_connections 2023-10-14T17:57:55.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:55.925+0000 7ff686021640 1 -- 172.21.15.29:0/1131674391 wait complete. 2023-10-14T17:57:55.945 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:57:55.946 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T17:57:55.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1h 2023-10-14T17:57:56.031 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.030+0000 7fd4d6eb6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:57:56.083 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.081+0000 7fd4d6eb6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:57:56.097 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.095+0000 7f15f88b6040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:57:56.133 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.132+0000 7fd4d6eb6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:57:56.152 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.150+0000 7f15f88b6040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:57:56.193 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.192+0000 7fd4d6eb6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:57:56.205 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.203+0000 7f15f88b6040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:57:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.204+0000 7f6ae7f41640 1 Processor -- start 2023-10-14T17:57:56.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.204+0000 7f6ae7f41640 1 -- start start 2023-10-14T17:57:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.205+0000 7f6ae7f41640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ae006b660 0x7f6ae006ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.205+0000 7f6ae7f41640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ae00c9860 0x7f6ae00ccb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.205+0000 7f6ae7f41640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ae00cc6b0 con 0x7f6ae01428c0 2023-10-14T17:57:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.205+0000 7f6ae7f41640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae01599b0 con 0x7f6ae006b660 2023-10-14T17:57:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.205+0000 7f6ae7f41640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ae006c290 con 0x7f6ae00c9860 2023-10-14T17:57:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ae006b660 0x7f6ae006ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:56.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae54dc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6ae01428c0 0x7f6ae00c9150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59222/0 (socket says 172.21.15.29:59222) 2023-10-14T17:57:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae54dc640 1 -- 172.21.15.29:0/2125394064 learned_addr learned my addr 172.21.15.29:0/2125394064 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:57:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ae006b660 0x7f6ae006ba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54584/0 (socket says 172.21.15.29:54584) 2023-10-14T17:57:56.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae64de640 1 --2- 172.21.15.29:0/2125394064 >> v2:172.21.15.186:3300/0 conn(0x7f6ae00c9860 0x7f6ae00ccb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/2125394064 >> v2:172.21.15.186:3300/0 conn(0x7f6ae00c9860 msgr2=0x7f6ae00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:57:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 --2- 172.21.15.29:0/2125394064 >> v2:172.21.15.186:3300/0 conn(0x7f6ae00c9860 0x7f6ae00ccb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/2125394064 >> v1:172.21.15.29:6789/0 conn(0x7f6ae01428c0 legacy=0x7f6ae00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:56.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/2125394064 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ae006c570 con 0x7f6ae006b660 2023-10-14T17:57:56.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.206+0000 7f6ae4cdb640 1 -- 172.21.15.29:0/2125394064 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657591309 0 0) 0x7f6ae00cc6b0 con 0x7f6ae01428c0 2023-10-14T17:57:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.207+0000 7f6ae5cdd640 1 --2- 172.21.15.29:0/2125394064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae006ba40 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00b0430 tx=0x7f6adc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfb6040f8eb42e77 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae4cdb640 1 -- 172.21.15.29:0/2125394064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc031830 con 0x7f6ae006b660 2023-10-14T17:57:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae4cdb640 1 -- 172.21.15.29:0/2125394064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc031990 con 0x7f6ae006b660 2023-10-14T17:57:56.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae4cdb640 1 -- 172.21.15.29:0/2125394064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc031d20 con 0x7f6ae006b660 2023-10-14T17:57:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 -- 172.21.15.29:0/2125394064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 msgr2=0x7f6ae006ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 --2- 172.21.15.29:0/2125394064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae006ba40 secure :-1 s=READY pgs=2015 cs=0 l=1 rev1=1 crypto rx=0x7f6ae00b0430 tx=0x7f6adc02b8d0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 -- 172.21.15.29:0/2125394064 shutdown_connections 2023-10-14T17:57:56.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 --2- 172.21.15.29:0/2125394064 >> v2:172.21.15.186:3300/0 conn(0x7f6ae00c9860 0x7f6ae00ccb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 --2- 172.21.15.29:0/2125394064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae006ba40 unknown :-1 s=CLOSED pgs=2015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 -- 172.21.15.29:0/2125394064 >> 172.21.15.29:0/2125394064 conn(0x7f6ae00ce9b0 msgr2=0x7f6ae00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:57:56.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.208+0000 7f6ae7f41640 1 -- 172.21.15.29:0/2125394064 shutdown_connections 2023-10-14T17:57:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.209+0000 7f6ae7f41640 1 -- 172.21.15.29:0/2125394064 wait complete. 2023-10-14T17:57:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.210+0000 7f6ae7f41640 1 Processor -- start 2023-10-14T17:57:56.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.210+0000 7f6ae7f41640 1 -- start start 2023-10-14T17:57:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae0167b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ae00c9860 0x7f6ae0152db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ae01428c0 0x7f6ae0158b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00b3d10 con 0x7f6ae006b660 2023-10-14T17:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00b16c0 con 0x7f6ae01428c0 2023-10-14T17:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae7f41640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ae00b19a0 con 0x7f6ae00c9860 2023-10-14T17:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae5cdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae0167b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:56.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.211+0000 7f6ae5cdd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae0167b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54590/0 (socket says 172.21.15.29:54590) 2023-10-14T17:57:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/19156477 learned_addr learned my addr 172.21.15.29:0/19156477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:57:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae64de640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ae01428c0 0x7f6ae0158b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ae00c9860 0x7f6ae0152db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:57:56.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ae00c9860 msgr2=0x7f6ae0152db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ae00c9860 0x7f6ae0152db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ae01428c0 msgr2=0x7f6ae0158b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:57:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ae01428c0 0x7f6ae0158b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6adc009400 con 0x7f6ae006b660 2023-10-14T17:57:56.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.212+0000 7f6ae5cdd640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae0167b20 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:57:56.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.213+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc03aa60 con 0x7f6ae006b660 2023-10-14T17:57:56.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.213+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6adc03abe0 con 0x7f6ae006b660 2023-10-14T17:57:56.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.213+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ae00b1c80 con 0x7f6ae006b660 2023-10-14T17:57:56.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.213+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6adc0443f0 con 0x7f6ae006b660 2023-10-14T17:57:56.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.213+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ae00b2240 con 0x7f6ae006b660 2023-10-14T17:57:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.214+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ae006bf40 con 0x7f6ae006b660 2023-10-14T17:57:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.216+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 94) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6adc04d450 con 0x7f6ae006b660 2023-10-14T17:57:56.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.217+0000 7f6ad67fc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:57:56.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.217+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(178..178 src has 1..178) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6adc0abb30 con 0x7f6ae006b660 2023-10-14T17:57:56.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.217+0000 7f6ae54dc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:56.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.217+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:57:56.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.221+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6adc0769c0 con 0x7f6ae006b660 2023-10-14T17:57:56.243 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.241+0000 7fd4d6eb6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:57:56.272 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.270+0000 7f15f88b6040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:57:56.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.322+0000 7f15f88b6040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:57:56.352 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.350+0000 7fd4d6eb6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:57:56.414 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.413+0000 7fd4d6eb6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.418+0000 7f6ae54dc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:56.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.418+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:57:56.441 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.439+0000 7f15f88b6040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:57:56.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.439+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ae006cb30 con 0x7f6ac406b010 2023-10-14T17:57:56.463 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.462+0000 7fd4d6eb6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:57:56.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.505+0000 7f15f88b6040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:56.520 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.519+0000 7fd4d6eb6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:57:56.559 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.556+0000 7f15f88b6040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:57:56.570 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.569+0000 7fd4d6eb6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:57:56.618 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.615+0000 7f15f88b6040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:57:56.670 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:56.668+0000 7f15f88b6040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:57:56.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.819+0000 7f6ae54dc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:56.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:56.819+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T17:57:56.951 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:56.949+0000 7fd4d6eb6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:57:57.071 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.068+0000 7f15f88b6040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:57:57.110 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.108+0000 7fd4d6eb6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:57:57.215 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.214+0000 7fd4d6eb6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:57:57.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.236+0000 7f15f88b6040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:57:57.265 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.263+0000 7fd4d6eb6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:57:57.351 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.348+0000 7f15f88b6040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:57:57.371 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.370+0000 7fd4d6eb6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:57.403 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.401+0000 7f15f88b6040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:57:57.477 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.476+0000 7fd4d6eb6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:57:57.515 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.513+0000 7f15f88b6040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:57.620+0000 7f6ae54dc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:57.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:57.620+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T17:57:57.626 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.624+0000 7f15f88b6040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:57:57.632 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.631+0000 7fd4d6eb6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:57.681 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.679+0000 7fd4d6eb6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:57.790 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.787+0000 7f15f88b6040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:57:57.799 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:57.798+0000 7fd4d6eb6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:57:57.841 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.839+0000 7f15f88b6040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:57:57.965 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:57.963+0000 7f15f88b6040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:57:58.630 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:58.628+0000 7fd4d6eb6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:57:58.840 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:58.837+0000 7f15f88b6040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:57:58.925 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:58.923+0000 7fd4d6eb6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:57:58.981 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:58.979+0000 7fd4d6eb6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:57:59.094 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:57:59.092+0000 7fd4d6eb6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:57:59.149 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.147+0000 7f15f88b6040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:57:59.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.163+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 95) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f6adc076d90 con 0x7f6ae006b660 2023-10-14T17:57:59.213 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.211+0000 7f15f88b6040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:57:59.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.222+0000 7f6ae54dc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2441704285 2023-10-14T17:57:59.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.222+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T17:57:59.332 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.330+0000 7f15f88b6040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:57:59.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.349+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 96) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f6adc0784a0 con 0x7f6ae006b660 2023-10-14T17:57:59.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.349+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 msgr2=0x7f6ac406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:57:59.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:57:59.349+0000 7f6ad67fc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/2441704285,v1:172.21.15.29:6833/2441704285] conn(0x7f6ac406b010 0x7f6ac406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:57:59.670 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.668+0000 7f1578b3e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.668+0000 7f1578b3e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.668+0000 7f1578b3e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.668+0000 7f1578b3e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.671 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.668+0000 7f1578b3e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.669+0000 7f15742f5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.669+0000 7f15742f5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.669+0000 7f15742f5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.672 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.669+0000 7f15742f5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:57:59.673 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:57:59.669+0000 7f15742f5640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:00.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.362+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 97) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f6adc077660 con 0x7f6ae006b660 2023-10-14T17:58:00.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.363+0000 7f6ad67fc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f6ac4027d50 0x7f6ac4075050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.364+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 --> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1h", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ac400b680 con 0x7f6ac4027d50 2023-10-14T17:58:00.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.375+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f6ac4027d50 0x7f6ac4075050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.386+0000 7f6ae54dc640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f6ac4027d50 0x7f6ac4075050 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0007fd0 tx=0x7f6ad0007dc0 comp rx=0 tx=0).ready entity=mgr.9777 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:00.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.575+0000 7f6ad67fc640 1 -- 172.21.15.29:0/19156477 <== mgr.9777 v2:172.21.15.29:6832/1035847590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6ac400b680 con 0x7f6ac4027d50 2023-10-14T17:58:00.577 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T17:58:00.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.579+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f6ac4027d50 msgr2=0x7f6ac4075050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.579+0000 7f6ae7f41640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f6ac4027d50 0x7f6ac4075050 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f6ad0007fd0 tx=0x7f6ad0007dc0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 msgr2=0x7f6ae0167b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 --2- 172.21.15.29:0/19156477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ae006b660 0x7f6ae0167b20 secure :-1 s=READY pgs=2016 cs=0 l=1 rev1=1 crypto rx=0x7f6adc000c00 tx=0x7f6adc0046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae64de640 1 -- 172.21.15.29:0/19156477 reap_dead start 2023-10-14T17:58:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 shutdown_connections 2023-10-14T17:58:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 >> 172.21.15.29:0/19156477 conn(0x7f6ae00ce9b0 msgr2=0x7f6ae00afc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:00.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 shutdown_connections 2023-10-14T17:58:00.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.580+0000 7f6ae7f41640 1 -- 172.21.15.29:0/19156477 wait complete. 2023-10-14T17:58:00.606 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T17:58:00.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-14T17:58:00.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.840+0000 7fa3e7fff640 1 Processor -- start 2023-10-14T17:58:00.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.840+0000 7fa3e7fff640 1 -- start start 2023-10-14T17:58:00.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3e80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3e7fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e815b9d0 con 0x7fa3e80ad970 2023-10-14T17:58:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3e7fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3e815bcb0 con 0x7fa3e806c040 2023-10-14T17:58:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3e7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e815b7d0 con 0x7fa3e815b320 2023-10-14T17:58:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3de7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa3e80ad970 0x7fa3e80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59234/0 (socket says 172.21.15.29:59234) 2023-10-14T17:58:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3de7fc640 1 -- 172.21.15.29:0/2261202927 learned_addr learned my addr 172.21.15.29:0/2261202927 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.842+0000 7fa3e77fe640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3e80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1235854233 0 0) 0x7fa3e815b9d0 con 0x7fa3e80ad970 2023-10-14T17:58:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3d4003520 con 0x7fa3e80ad970 2023-10-14T17:58:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1534349653 0 0) 0x7fa3d4003520 con 0x7fa3e80ad970 2023-10-14T17:58:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 msgr2=0x7fa3e80cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3e80cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 >> v1:172.21.15.186:6789/0 conn(0x7fa3e806c040 legacy=0x7fa3e806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e815bf90 con 0x7fa3e80ad970 2023-10-14T17:58:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa3d0001670 con 0x7fa3e80ad970 2023-10-14T17:58:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 >> v1:172.21.15.29:6789/0 conn(0x7fa3e80ad970 legacy=0x7fa3e80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3d40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3d4003f20 0x7fa3d40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3d4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e815b7d0 con 0x7fa3d4004d00 2023-10-14T17:58:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d400ba60 con 0x7fa3d4003f20 2023-10-14T17:58:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.843+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d400bc20 con 0x7fa3e815b320 2023-10-14T17:58:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3d4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e6ffd640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3d40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3de7fc640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3d4003f20 0x7fa3d40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 -- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 msgr2=0x7fa3d40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e815b320 0x7fa3d40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 -- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3d4003f20 msgr2=0x7fa3d40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3d4003f20 0x7fa3d40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.844+0000 7fa3e77fe640 1 -- 172.21.15.29:0/2261202927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d4003520 con 0x7fa3d4004d00 2023-10-14T17:58:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.845+0000 7fa3e77fe640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3d4005180 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009ac0 tx=0x7fa3d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95d9c88bc2cca0e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e002c6b0 con 0x7fa3d4004d00 2023-10-14T17:58:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3e002c810 con 0x7fa3d4004d00 2023-10-14T17:58:00.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e67fc640 1 -- 172.21.15.29:0/2261202927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3e002cba0 con 0x7fa3d4004d00 2023-10-14T17:58:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2261202927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 msgr2=0x7fa3d4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2261202927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3d4005180 secure :-1 s=READY pgs=2031 cs=0 l=1 rev1=1 crypto rx=0x7fa3e0009ac0 tx=0x7fa3d40057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e77fe640 1 -- 172.21.15.29:0/2261202927 reap_dead start 2023-10-14T17:58:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2261202927 shutdown_connections 2023-10-14T17:58:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2261202927 >> 172.21.15.29:0/2261202927 conn(0x7fa3e80ce6a0 msgr2=0x7fa3e80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2261202927 shutdown_connections 2023-10-14T17:58:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.846+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2261202927 wait complete. 2023-10-14T17:58:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.849+0000 7fa3e7fff640 1 Processor -- start 2023-10-14T17:58:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.849+0000 7fa3e7fff640 1 -- start start 2023-10-14T17:58:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 0x7fa3e8159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e81efea0 0x7fa3e8155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8159910 con 0x7fa3d4004d00 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8158c90 con 0x7fa3e8152f40 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.850+0000 7fa3e7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8158f70 con 0x7fa3e81efea0 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54738/0 (socket says 172.21.15.29:54738) 2023-10-14T17:58:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 -- 172.21.15.29:0/2831303815 learned_addr learned my addr 172.21.15.29:0/2831303815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 -- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e81efea0 msgr2=0x7fa3e8155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e81efea0 0x7fa3e8155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 -- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 msgr2=0x7fa3e8159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3de7fc640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 0x7fa3e8159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 0x7fa3e8159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 -- 172.21.15.29:0/2831303815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e8159250 con 0x7fa3d4004d00 2023-10-14T17:58:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3de7fc640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 0x7fa3e8159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.851+0000 7fa3e6ffd640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800e840 tx=0x7fa3d800ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:00.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.852+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d8003b80 con 0x7fa3d4004d00 2023-10-14T17:58:00.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.852+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3d8003ce0 con 0x7fa3d4004d00 2023-10-14T17:58:00.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.852+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e8159530 con 0x7fa3d4004d00 2023-10-14T17:58:00.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.852+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3d80052f0 con 0x7fa3d4004d00 2023-10-14T17:58:00.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.852+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e815a270 con 0x7fa3d4004d00 2023-10-14T17:58:00.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.853+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e804e9b0 con 0x7fa3d4004d00 2023-10-14T17:58:00.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.854+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 97) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa3d8005450 con 0x7fa3d4004d00 2023-10-14T17:58:00.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.855+0000 7fa3dffff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 0x7fa3c806d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:00.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.855+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fa3d80895a0 con 0x7fa3d4004d00 2023-10-14T17:58:00.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.855+0000 7fa3de7fc640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 0x7fa3c806d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:00.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.855+0000 7fa3de7fc640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 0x7fa3c806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0001eb0 tx=0x7fa3d0001d80 comp rx=0 tx=0).ready entity=mgr.9777 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:00.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:00.859+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3d80546f0 con 0x7fa3d4004d00 2023-10-14T17:58:01.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.086+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 --> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3e80cc550 con 0x7fa3c806ad40 2023-10-14T17:58:01.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.260+0000 7fa3dffff640 1 -- 172.21.15.29:0/2831303815 <== mgr.9777 v2:172.21.15.29:6832/1035847590 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fa3d000f7c0 con 0x7fa3c806ad40 2023-10-14T17:58:01.261 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-14T17:58:01.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 msgr2=0x7fa3c806d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 0x7fa3c806d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0001eb0 tx=0x7fa3d0001d80 comp rx=0 tx=0).stop 2023-10-14T17:58:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 msgr2=0x7fa3e80a8480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 secure :-1 s=READY pgs=2032 cs=0 l=1 rev1=1 crypto rx=0x7fa3d800e840 tx=0x7fa3d800ed00 comp rx=0 tx=0).stop 2023-10-14T17:58:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 shutdown_connections 2023-10-14T17:58:01.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fa3c806ad40 0x7fa3c806d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3e81efea0 0x7fa3e8155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa3e8152f40 0x7fa3e8159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 --2- 172.21.15.29:0/2831303815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3d4004d00 0x7fa3e80a8480 unknown :-1 s=CLOSED pgs=2032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 >> 172.21.15.29:0/2831303815 conn(0x7fa3e80ce6a0 msgr2=0x7fa3e80c7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:01.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.263+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 shutdown_connections 2023-10-14T17:58:01.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.264+0000 7fa3e7fff640 1 -- 172.21.15.29:0/2831303815 wait complete. 2023-10-14T17:58:01.285 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-14T17:58:01.286 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list snap_test_dir1 2023-10-14T17:58:01.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.528+0000 7f81ef30f640 1 Processor -- start 2023-10-14T17:58:01.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.528+0000 7f81ef30f640 1 -- start start 2023-10-14T17:58:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ef30f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f81e806a4b0 0x7f81e80ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ef30f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e80c9ca0 0x7f81e80ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ef30f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e80cb030 con 0x7f81e815ad70 2023-10-14T17:58:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ef30f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81e80cae30 con 0x7f81e806a4b0 2023-10-14T17:58:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ef30f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e80c91c0 con 0x7f81e80c9ca0 2023-10-14T17:58:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ed0ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f81e815ad70 0x7f81e8069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47110/0 (socket says 172.21.15.29:47110) 2023-10-14T17:58:01.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ed0ab640 1 -- 172.21.15.29:0/1262392830 learned_addr learned my addr 172.21.15.29:0/1262392830 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:01.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ec8aa640 1 --2- 172.21.15.29:0/1262392830 >> v2:172.21.15.29:3300/0 conn(0x7f81e806a4b0 0x7f81e80ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.530+0000 7f81ed8ac640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e80c9ca0 0x7f81e80ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81dffff640 1 -- 172.21.15.29:0/1262392830 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418373728 0 0) 0x7f81e80cb030 con 0x7f81e815ad70 2023-10-14T17:58:01.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81dffff640 1 -- 172.21.15.29:0/1262392830 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81d4003520 con 0x7f81e815ad70 2023-10-14T17:58:01.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81ec8aa640 1 -- 172.21.15.29:0/1262392830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e80c9ca0 msgr2=0x7f81e80ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:01.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81ec8aa640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e80c9ca0 0x7f81e80ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81ec8aa640 1 -- 172.21.15.29:0/1262392830 >> v1:172.21.15.29:6789/0 conn(0x7f81e815ad70 legacy=0x7f81e8069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.531+0000 7f81ec8aa640 1 -- 172.21.15.29:0/1262392830 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e80c94a0 con 0x7f81e806a4b0 2023-10-14T17:58:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.532+0000 7f81ec8aa640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e806a4b0 0x7f81e80ca3b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097a0 tx=0x7f81d802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9412097e08149bd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.532+0000 7f81dffff640 1 -- 172.21.15.29:0/1262392830 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d802d040 con 0x7f81e806a4b0 2023-10-14T17:58:01.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.532+0000 7f81dffff640 1 -- 172.21.15.29:0/1262392830 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81d8031810 con 0x7f81e806a4b0 2023-10-14T17:58:01.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.532+0000 7f81dffff640 1 -- 172.21.15.29:0/1262392830 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81d8031ba0 con 0x7f81e806a4b0 2023-10-14T17:58:01.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 -- 172.21.15.29:0/1262392830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e806a4b0 msgr2=0x7f81e80ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e806a4b0 0x7f81e80ca3b0 secure :-1 s=READY pgs=2036 cs=0 l=1 rev1=1 crypto rx=0x7f81d80097a0 tx=0x7f81d802b8b0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 -- 172.21.15.29:0/1262392830 shutdown_connections 2023-10-14T17:58:01.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e80c9ca0 0x7f81e80ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 --2- 172.21.15.29:0/1262392830 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e806a4b0 0x7f81e80ca3b0 unknown :-1 s=CLOSED pgs=2036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 -- 172.21.15.29:0/1262392830 >> 172.21.15.29:0/1262392830 conn(0x7f81e80ce9b0 msgr2=0x7f81e80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:01.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 -- 172.21.15.29:0/1262392830 shutdown_connections 2023-10-14T17:58:01.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.533+0000 7f81ef30f640 1 -- 172.21.15.29:0/1262392830 wait complete. 2023-10-14T17:58:01.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.535+0000 7f81ef30f640 1 Processor -- start 2023-10-14T17:58:01.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.535+0000 7f81ef30f640 1 -- start start 2023-10-14T17:58:01.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e806a4b0 0x7f81e8158010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81e80c9ca0 0x7f81e8150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e815ad70 0x7f81e80c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8158550 con 0x7f81e815ad70 2023-10-14T17:58:01.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8156f80 con 0x7f81e80c9ca0 2023-10-14T17:58:01.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ef30f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81e8156370 con 0x7f81e806a4b0 2023-10-14T17:58:01.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ed8ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e815ad70 0x7f81e80c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ed8ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e815ad70 0x7f81e80c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34412/0 (socket says 172.21.15.29:34412) 2023-10-14T17:58:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.536+0000 7f81ed8ac640 1 -- 172.21.15.29:0/3730589838 learned_addr learned my addr 172.21.15.29:0/3730589838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ec8aa640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81e80c9ca0 0x7f81e8150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed0ab640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e806a4b0 0x7f81e8158010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 -- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e806a4b0 msgr2=0x7f81e8158010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f81e806a4b0 0x7f81e8158010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 -- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81e80c9ca0 msgr2=0x7f81e8150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f81e80c9ca0 0x7f81e8150d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 -- 172.21.15.29:0/3730589838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81d8009400 con 0x7f81e815ad70 2023-10-14T17:58:01.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ed8ac640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f81e815ad70 0x7f81e80c24a0 secure :-1 s=READY pgs=2037 cs=0 l=1 rev1=1 crypto rx=0x7f81e0007e40 tx=0x7f81e000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0014620 con 0x7f81e815ad70 2023-10-14T17:58:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81e000ee90 con 0x7f81e815ad70 2023-10-14T17:58:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00149e0 con 0x7f81e815ad70 2023-10-14T17:58:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.537+0000 7f81ef30f640 1 -- 172.21.15.29:0/3730589838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e8156650 con 0x7f81e815ad70 2023-10-14T17:58:01.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.538+0000 7f81ef30f640 1 -- 172.21.15.29:0/3730589838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e8156c10 con 0x7f81e815ad70 2023-10-14T17:58:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.539+0000 7f81ef30f640 1 -- 172.21.15.29:0/3730589838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e81570f0 con 0x7f81e815ad70 2023-10-14T17:58:01.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.540+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f81e0014b40 con 0x7f81e815ad70 2023-10-14T17:58:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.540+0000 7f81ddffb640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f81cc06afe0 0x7f81cc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.541+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f81e008a400 con 0x7f81e815ad70 2023-10-14T17:58:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.541+0000 7f81ed0ab640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f81cc06afe0 0x7f81cc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.541+0000 7f81ed0ab640 1 --2- 172.21.15.29:0/3730589838 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f81cc06afe0 0x7f81cc06d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f81e40011e0 tx=0x7f81e4009040 comp rx=0 tx=0).ready entity=mgr.9777 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:01.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.546+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f81e0055140 con 0x7f81e815ad70 2023-10-14T17:58:01.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.767+0000 7f81ef30f640 1 -- 172.21.15.29:0/3730589838 --> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f81e80c9780 con 0x7f81cc06afe0 2023-10-14T17:58:01.783 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:01.781+0000 7f157fb4c640 -1 mgr.server reply reply (2) No such file or directory SnapSchedule for snap_test_dir1 not found 2023-10-14T17:58:01.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:01.781+0000 7f81ddffb640 1 -- 172.21.15.29:0/3730589838 <== mgr.9777 v2:172.21.15.29:6832/1035847590 1 ==== mgr_command_reply(tid 0: -2 SnapSchedule for snap_test_dir1 not found) v1 ==== 49+0+0 (secure 0 0 0) 0x7f81e4001330 con 0x7f81cc06afe0 2023-10-14T17:58:01.785 INFO:teuthology.orchestra.run.smithi029.stderr:Error ENOENT: SnapSchedule for snap_test_dir1 not found 2023-10-14T17:58:01.798 DEBUG:teuthology.orchestra.run:got remote process result: 2 2023-10-14T17:58:01.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:01.800 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T17:58:01.860 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T17:58:02.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.087+0000 7fc401dcb640 1 Processor -- start 2023-10-14T17:58:02.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.087+0000 7fc401dcb640 1 -- start start 2023-10-14T17:58:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.088+0000 7fc401dcb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 0x7fc3fc06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.088+0000 7fc401dcb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3fc0ad970 0x7fc3fc15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.088+0000 7fc401dcb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 0x7fc3fc15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.088+0000 7fc401dcb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3fc06c960 con 0x7fc3fc0ad970 2023-10-14T17:58:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc401dcb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3fc15b9d0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc401dcb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc15bcb0 con 0x7fc3fc06c040 2023-10-14T17:58:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3fc0ad970 0x7fc3fc15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3fc0ad970 0x7fc3fc15e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34420/0 (socket says 172.21.15.29:34420) 2023-10-14T17:58:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 -- 172.21.15.29:0/1868064731 learned_addr learned my addr 172.21.15.29:0/1868064731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc4015ca640 1 --2- 172.21.15.29:0/1868064731 >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 0x7fc3fc15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc400dc9640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 0x7fc3fc06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 -- 172.21.15.29:0/1868064731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 msgr2=0x7fc3fc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 0x7fc3fc06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 -- 172.21.15.29:0/1868064731 >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 msgr2=0x7fc3fc15f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:02.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 --2- 172.21.15.29:0/1868064731 >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 0x7fc3fc15f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc3fbfff640 1 -- 172.21.15.29:0/1868064731 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3fc15bf90 con 0x7fc3fc0ad970 2023-10-14T17:58:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc4015ca640 1 --2- 172.21.15.29:0/1868064731 >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 0x7fc3fc15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.089+0000 7fc400dc9640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 0x7fc3fc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.091+0000 7fc3fbfff640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc0ad970 0x7fc3fc15e680 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009870 tx=0x7fc3e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61350cea3ef32598 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.091+0000 7fc3fb7fe640 1 -- 172.21.15.29:0/1868064731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4031640 con 0x7fc3fc0ad970 2023-10-14T17:58:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.091+0000 7fc3fb7fe640 1 -- 172.21.15.29:0/1868064731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e40317a0 con 0x7fc3fc0ad970 2023-10-14T17:58:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.091+0000 7fc3fb7fe640 1 -- 172.21.15.29:0/1868064731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4031b30 con 0x7fc3fc0ad970 2023-10-14T17:58:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.091+0000 7fc401dcb640 1 -- 172.21.15.29:0/1868064731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc0ad970 msgr2=0x7fc3fc15e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc0ad970 0x7fc3fc15e680 secure :-1 s=READY pgs=2038 cs=0 l=1 rev1=1 crypto rx=0x7fc3e4009870 tx=0x7fc3e402b8f0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 -- 172.21.15.29:0/1868064731 shutdown_connections 2023-10-14T17:58:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 --2- 172.21.15.29:0/1868064731 >> v2:172.21.15.186:3300/0 conn(0x7fc3fc15ebc0 0x7fc3fc15f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc0ad970 0x7fc3fc15e680 unknown :-1 s=CLOSED pgs=2038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 --2- 172.21.15.29:0/1868064731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc06c040 0x7fc3fc06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 -- 172.21.15.29:0/1868064731 >> 172.21.15.29:0/1868064731 conn(0x7fc3fc0ce6a0 msgr2=0x7fc3fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 -- 172.21.15.29:0/1868064731 shutdown_connections 2023-10-14T17:58:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.092+0000 7fc401dcb640 1 -- 172.21.15.29:0/1868064731 wait complete. 2023-10-14T17:58:02.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.094+0000 7fc401dcb640 1 Processor -- start 2023-10-14T17:58:02.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.094+0000 7fc401dcb640 1 -- start start 2023-10-14T17:58:02.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 0x7fc3fc0c7fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc0ad970 0x7fc3fc14ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc0c8fb0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc0c8500 con 0x7fc3fc06c040 2023-10-14T17:58:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc401dcb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3fc14cfd0 con 0x7fc3fc0ad970 2023-10-14T17:58:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc4015ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc4015ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34422/0 (socket says 172.21.15.29:34422) 2023-10-14T17:58:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.095+0000 7fc4015ca640 1 -- 172.21.15.29:0/938928030 learned_addr learned my addr 172.21.15.29:0/938928030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc3fbfff640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc0ad970 0x7fc3fc14ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc400dc9640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 0x7fc3fc0c7fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 -- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc0ad970 msgr2=0x7fc3fc14ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc0ad970 0x7fc3fc14ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 -- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 msgr2=0x7fc3fc0c7fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 0x7fc3fc0c7fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 -- 172.21.15.29:0/938928030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e40094d0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc400dc9640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 0x7fc3fc0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc4015ca640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400cf20 tx=0x7fc3f400a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400d840 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3f400d9a0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3f400dd50 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.096+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3fc0c67c0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.097+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3fc0c7340 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.098+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3fc06b330 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.099+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 98) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fc3f401e3b0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.100+0000 7fc3f97fa640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 0x7fc3dc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.100+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fc3f408a4e0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.100+0000 7fc400dc9640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 0x7fc3dc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.101+0000 7fc400dc9640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 0x7fc3dc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b840 tx=0x7fc3ec008040 comp rx=0 tx=0).ready entity=mgr.9777 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.104+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3f4055210 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.356+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7fc3fc0c74d0 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.724+0000 7fc3f97fa640 1 -- 172.21.15.29:0/938928030 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v99) v1 ==== 93+0+0 (secure 0 0 0) 0x7fc3f4058830 con 0x7fc3fc15ebc0 2023-10-14T17:58:02.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.728+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 msgr2=0x7fc3dc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:02.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.728+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 0x7fc3dc06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc3ec00b840 tx=0x7fc3ec008040 comp rx=0 tx=0).stop 2023-10-14T17:58:02.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.728+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 msgr2=0x7fc3fc0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:02.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.728+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 secure :-1 s=READY pgs=2039 cs=0 l=1 rev1=1 crypto rx=0x7fc3f400cf20 tx=0x7fc3f400a7d0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 shutdown_connections 2023-10-14T17:58:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fc3dc06afe0 0x7fc3dc06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3fc15ebc0 0x7fc3fc0ac890 unknown :-1 s=CLOSED pgs=2039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3fc0ad970 0x7fc3fc14ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 --2- 172.21.15.29:0/938928030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3fc06c040 0x7fc3fc0c7fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:02.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 >> 172.21.15.29:0/938928030 conn(0x7fc3fc0ce6a0 msgr2=0x7fc3fc0b1950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:02.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.729+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 shutdown_connections 2023-10-14T17:58:02.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:02.730+0000 7fc401dcb640 1 -- 172.21.15.29:0/938928030 wait complete. 2023-10-14T17:58:02.755 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:58:02.756 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:58:02.878 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:58:02.968 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:58:02.969 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:02.969 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:03.026 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:03.026 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:03.076 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:03.285 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.284+0000 7f8fe037b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:58:03.303 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.302+0000 7fbec1432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:58:03.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.326+0000 7fb0f9a44640 1 Processor -- start 2023-10-14T17:58:03.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.327+0000 7fb0f9a44640 1 -- start start 2023-10-14T17:58:03.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.328+0000 7fb0f9a44640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb0f415ae80 0x7fb0f415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.328+0000 7fb0f9a44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f415ed60 con 0x7fb0f415aaa0 2023-10-14T17:58:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.328+0000 7fb0f9a44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0f40707e0 con 0x7fb0f40ce390 2023-10-14T17:58:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.328+0000 7fb0f9a44640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0f406b810 con 0x7fb0f415ae80 2023-10-14T17:58:03.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.329+0000 7fb0f2ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb0f415aaa0 0x7fb0f40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47122/0 (socket says 172.21.15.29:47122) 2023-10-14T17:58:03.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.329+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/1479850553 learned_addr learned my addr 172.21.15.29:0/1479850553 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.329+0000 7fb0ebfff640 1 --2- 172.21.15.29:0/1479850553 >> v2:172.21.15.186:3300/0 conn(0x7fb0f415ae80 0x7fb0f415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.329+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244647451 0 0) 0x7fb0f415ed60 con 0x7fb0f415aaa0 2023-10-14T17:58:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.329+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e0003520 con 0x7fb0f415aaa0 2023-10-14T17:58:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1046746063 0 0) 0x7fb0e0003520 con 0x7fb0f415aaa0 2023-10-14T17:58:03.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 >> v2:172.21.15.186:3300/0 conn(0x7fb0f415ae80 msgr2=0x7fb0f415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 --2- 172.21.15.29:0/1479850553 >> v2:172.21.15.186:3300/0 conn(0x7fb0f415ae80 0x7fb0f415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 >> v1:172.21.15.186:6789/0 conn(0x7fb0f40ce390 legacy=0x7fb0f4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f415ef90 con 0x7fb0f415aaa0 2023-10-14T17:58:03.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb0e4001670 con 0x7fb0f415aaa0 2023-10-14T17:58:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 >> v1:172.21.15.29:6789/0 conn(0x7fb0f415aaa0 legacy=0x7fb0f40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 0x7fb0e0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0e00047c0 0x7fb0e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0005180 0x7fb0e000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f406b810 con 0x7fb0e0003970 2023-10-14T17:58:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e000c020 con 0x7fb0e0005180 2023-10-14T17:58:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e000c1e0 con 0x7fb0e00047c0 2023-10-14T17:58:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.330+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 0x7fb0e0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0e00047c0 msgr2=0x7fb0e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:03.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0ebfff640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0e00047c0 0x7fb0e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0e00047c0 0x7fb0e0004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f37fe640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0005180 0x7fb0e000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0005180 msgr2=0x7fb0e000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:03.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0005180 0x7fb0e000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/1479850553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e0003520 con 0x7fb0e0003970 2023-10-14T17:58:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f37fe640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0005180 0x7fb0e000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.331+0000 7fb0f37fe640 1 -- 172.21.15.29:0/1479850553 reap_dead start 2023-10-14T17:58:03.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.332+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 0x7fb0e0003d90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400c190 tx=0x7fb0e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e76b08b82a6a6cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.332+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e4005ad0 con 0x7fb0e0003970 2023-10-14T17:58:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.332+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4005c30 con 0x7fb0e0003970 2023-10-14T17:58:03.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.332+0000 7fb0f27fc640 1 -- 172.21.15.29:0/1479850553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0e402cae0 con 0x7fb0e0003970 2023-10-14T17:58:03.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 -- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 msgr2=0x7fb0e0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 0x7fb0e0003d90 secure :-1 s=READY pgs=2043 cs=0 l=1 rev1=1 crypto rx=0x7fb0e400c190 tx=0x7fb0e00057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 -- 172.21.15.29:0/1479850553 shutdown_connections 2023-10-14T17:58:03.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/1479850553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0e0003970 0x7fb0e0003d90 unknown :-1 s=CLOSED pgs=2043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 -- 172.21.15.29:0/1479850553 >> 172.21.15.29:0/1479850553 conn(0x7fb0f40cb410 msgr2=0x7fb0f40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 -- 172.21.15.29:0/1479850553 shutdown_connections 2023-10-14T17:58:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.333+0000 7fb0f9a44640 1 -- 172.21.15.29:0/1479850553 wait complete. 2023-10-14T17:58:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.335+0000 7fb0f9a44640 1 Processor -- start 2023-10-14T17:58:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.335+0000 7fb0f9a44640 1 -- start start 2023-10-14T17:58:03.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 0x7fb0f40c6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f40c5010 0x7fb0f40c2490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40ae590 con 0x7fb0f40af630 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40ae870 con 0x7fb0e0003970 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f9a44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0f40aeb50 con 0x7fb0f40c5010 2023-10-14T17:58:03.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34466/0 (socket says 172.21.15.29:34466) 2023-10-14T17:58:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0ebfff640 1 -- 172.21.15.29:0/2242491123 learned_addr learned my addr 172.21.15.29:0/2242491123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f37fe640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f40c5010 0x7fb0f40c2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.336+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 0x7fb0f40c6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f40c5010 msgr2=0x7fb0f40c2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f40c5010 0x7fb0f40c2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 msgr2=0x7fb0f40c6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 0x7fb0f40c6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 -- 172.21.15.29:0/2242491123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0f40aee30 con 0x7fb0f40af630 2023-10-14T17:58:03.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 0x7fb0f40c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0ebfff640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc010890 tx=0x7fb0dc010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc009630 con 0x7fb0f40af630 2023-10-14T17:58:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0dc00eb60 con 0x7fb0f40af630 2023-10-14T17:58:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb0dc01c400 con 0x7fb0f40af630 2023-10-14T17:58:03.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.337+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0f40af110 con 0x7fb0f40af630 2023-10-14T17:58:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.338+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0f40afab0 con 0x7fb0f40af630 2023-10-14T17:58:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.339+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0f404e160 con 0x7fb0f40af630 2023-10-14T17:58:03.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.340+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb0dc01c560 con 0x7fb0f40af630 2023-10-14T17:58:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.341+0000 7fb0f0ff9640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.341+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb0dc089f40 con 0x7fb0f40af630 2023-10-14T17:58:03.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.341+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 msgr2=0x7fb0d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.341+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:03.350 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.343+0000 7f8fe037b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:58:03.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.345+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb0dc054d20 con 0x7fb0f40af630 2023-10-14T17:58:03.363 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.362+0000 7fbec1432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:58:03.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.538+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0f40c21c0 con 0x7fb0f40af630 2023-10-14T17:58:03.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.539+0000 7fb0f0ff9640 1 -- 172.21.15.29:0/2242491123 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+17252 (secure 0 0 0) 0x7fb0dc058340 con 0x7fb0f40af630 2023-10-14T17:58:03.541 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:03.541 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":179,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:59.334060+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3415492501":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2674099337":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2596229485":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/4106038121":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6832/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1011592268":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/84821019":"2023-10-15T17:57:59.333970+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 msgr2=0x7fb0d406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f2ffd640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 msgr2=0x7fb0d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f2ffd640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:03.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 msgr2=0x7fb0f40c4ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 secure :-1 s=READY pgs=2044 cs=0 l=1 rev1=1 crypto rx=0x7fb0dc010890 tx=0x7fb0dc010d50 comp rx=0 tx=0).stop 2023-10-14T17:58:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 shutdown_connections 2023-10-14T17:58:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb0d406af90 0x7fb0d406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb0f40c5010 0x7fb0f40c2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb0f40af630 0x7fb0f40c4ad0 unknown :-1 s=CLOSED pgs=2044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 --2- 172.21.15.29:0/2242491123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb0e0003970 0x7fb0f40c6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 >> 172.21.15.29:0/2242491123 conn(0x7fb0f40cb410 msgr2=0x7fb0f40ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 shutdown_connections 2023-10-14T17:58:03.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.542+0000 7fb0f9a44640 1 -- 172.21.15.29:0/2242491123 wait complete. 2023-10-14T17:58:03.562 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:57:42.346345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:57:43.390271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:58:03.563 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:58:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.771+0000 7f87f96a2640 1 Processor -- start 2023-10-14T17:58:03.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.771+0000 7f87f96a2640 1 -- start start 2023-10-14T17:58:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f96a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f96a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87f40ca780 0x7f87f406aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f96a2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87f40c3610 con 0x7f87f406b3f0 2023-10-14T17:58:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f96a2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87f406b880 con 0x7f87f40ca780 2023-10-14T17:58:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f96a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f415a220 con 0x7f87f4159e40 2023-10-14T17:58:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87f40ca780 0x7f87f406aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f27fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87f40ca780 0x7f87f406aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34468/0 (socket says 172.21.15.29:34468) 2023-10-14T17:58:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f27fc640 1 -- 172.21.15.29:0/253899930 learned_addr learned my addr 172.21.15.29:0/253899930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:03.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.773+0000 7f87f2ffd640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.774+0000 7f87f27fc640 1 -- 172.21.15.29:0/253899930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 msgr2=0x7f87f40cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.774+0000 7f87f27fc640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.774+0000 7f87f27fc640 1 -- 172.21.15.29:0/253899930 >> v1:172.21.15.186:6789/0 conn(0x7f87f406b3f0 legacy=0x7f87f40705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.774+0000 7f87f27fc640 1 -- 172.21.15.29:0/253899930 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40cafd0 con 0x7f87f40ca780 2023-10-14T17:58:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.775+0000 7f87f27fc640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f406aeb0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00d150 tx=0x7f87dc02e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5c197c03aa078d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.775+0000 7f87f1ffb640 1 -- 172.21.15.29:0/253899930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc00b0d0 con 0x7f87f40ca780 2023-10-14T17:58:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f1ffb640 1 -- 172.21.15.29:0/253899930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc00b230 con 0x7f87f40ca780 2023-10-14T17:58:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f1ffb640 1 -- 172.21.15.29:0/253899930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc009600 con 0x7f87f40ca780 2023-10-14T17:58:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 -- 172.21.15.29:0/253899930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 msgr2=0x7f87f406aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f406aeb0 secure :-1 s=READY pgs=2045 cs=0 l=1 rev1=1 crypto rx=0x7f87dc00d150 tx=0x7f87dc02e160 comp rx=0 tx=0).stop 2023-10-14T17:58:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 -- 172.21.15.29:0/253899930 shutdown_connections 2023-10-14T17:58:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f406aeb0 unknown :-1 s=CLOSED pgs=2045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 --2- 172.21.15.29:0/253899930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.776+0000 7f87f96a2640 1 -- 172.21.15.29:0/253899930 >> 172.21.15.29:0/253899930 conn(0x7f87f40cfac0 msgr2=0x7f87f40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.777+0000 7f87f96a2640 1 -- 172.21.15.29:0/253899930 shutdown_connections 2023-10-14T17:58:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.777+0000 7f87f96a2640 1 -- 172.21.15.29:0/253899930 wait complete. 2023-10-14T17:58:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.778+0000 7f87f96a2640 1 Processor -- start 2023-10-14T17:58:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.778+0000 7f87f96a2640 1 -- start start 2023-10-14T17:58:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f406b3f0 0x7f87f4157dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40c7ba0 con 0x7f87f40ca780 2023-10-14T17:58:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40c75e0 con 0x7f87f406b3f0 2023-10-14T17:58:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.779+0000 7f87f96a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40c78c0 con 0x7f87f4159e40 2023-10-14T17:58:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34480/0 (socket says 172.21.15.29:34480) 2023-10-14T17:58:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 -- 172.21.15.29:0/2619304442 learned_addr learned my addr 172.21.15.29:0/2619304442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f37fe640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f2ffd640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f406b3f0 0x7f87f4157dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 msgr2=0x7f87f40c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f406b3f0 msgr2=0x7f87f4157dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f406b3f0 0x7f87f4157dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.780+0000 7f87f27fc640 1 -- 172.21.15.29:0/2619304442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87dc00bd40 con 0x7f87f40ca780 2023-10-14T17:58:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87f27fc640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000f80 tx=0x7f87dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc036800 con 0x7f87f40ca780 2023-10-14T17:58:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87dc036980 con 0x7f87f40ca780 2023-10-14T17:58:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87dc036d30 con 0x7f87f40ca780 2023-10-14T17:58:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40c8850 con 0x7f87f40ca780 2023-10-14T17:58:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.781+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f40c7f50 con 0x7f87f40ca780 2023-10-14T17:58:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.782+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f40c80e0 con 0x7f87f40ca780 2023-10-14T17:58:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.783+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f87dc04a460 con 0x7f87f40ca780 2023-10-14T17:58:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.784+0000 7f87eb7fe640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 0x7f87d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.784+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f87dc0acc70 con 0x7f87f40ca780 2023-10-14T17:58:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.784+0000 7f87f2ffd640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 msgr2=0x7f87d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.784+0000 7f87f2ffd640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 0x7f87d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.789+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87dc077b50 con 0x7f87f40ca780 2023-10-14T17:58:03.822 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.821+0000 7fbec1432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:58:03.823 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.821+0000 7f8fe037b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:58:03.874 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.872+0000 7fbec1432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:58:03.877 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.876+0000 7f8fe037b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:58:03.924 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.923+0000 7fbec1432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:58:03.930 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.929+0000 7f8fe037b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:58:03.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.985+0000 7f87f2ffd640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 msgr2=0x7f87d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:03.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.985+0000 7f87f2ffd640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 0x7f87d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:03.987 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:03.985+0000 7fbec1432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:58:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.986+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f87f40c8d70 con 0x7f87f40ca780 2023-10-14T17:58:03.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.987+0000 7f87eb7fe640 1 -- 172.21.15.29:0/2619304442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7f87f40c8d70 con 0x7f87f40ca780 2023-10-14T17:58:03.989 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:58:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.989+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 msgr2=0x7f87d406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:03.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.989+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 0x7f87d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.989+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 msgr2=0x7f87f4154680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.989+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 secure :-1 s=READY pgs=2046 cs=0 l=1 rev1=1 crypto rx=0x7f87dc000f80 tx=0x7f87dc004a40 comp rx=0 tx=0).stop 2023-10-14T17:58:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 shutdown_connections 2023-10-14T17:58:03.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f87d406af90 0x7f87d406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87f4159e40 0x7f87f40c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87f40ca780 0x7f87f4154680 unknown :-1 s=CLOSED pgs=2046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 --2- 172.21.15.29:0/2619304442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87f406b3f0 0x7f87f4157dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:03.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 >> 172.21.15.29:0/2619304442 conn(0x7f87f40cfac0 msgr2=0x7f87f41e8560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 shutdown_connections 2023-10-14T17:58:03.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:03.990+0000 7f87f96a2640 1 -- 172.21.15.29:0/2619304442 wait complete. 2023-10-14T17:58:03.997 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:03.996+0000 7f8fe037b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:58:04.008 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:58:04.037 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.035+0000 7fbec1432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:58:04.053 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.052+0000 7f8fe037b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:58:04.147 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.145+0000 7fbec1432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:58:04.174 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.173+0000 7f8fe037b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:58:04.210 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.208+0000 7fbec1432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.220+0000 7fb1419c9640 1 Processor -- start 2023-10-14T17:58:04.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.220+0000 7fb1419c9640 1 -- start start 2023-10-14T17:58:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.221+0000 7fb1419c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb13c0cf710 0x7fb13c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb1419c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0d18c0 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb1419c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13c0d3980 con 0x7fb13c0cfaf0 2023-10-14T17:58:04.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb1419c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13c0d3c60 con 0x7fb13c0cf710 2023-10-14T17:58:04.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb13b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb13c0b6cd0 0x7fb13c0d11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47126/0 (socket says 172.21.15.29:47126) 2023-10-14T17:58:04.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb13b7fe640 1 -- 172.21.15.29:0/1751315706 learned_addr learned my addr 172.21.15.29:0/1751315706 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:04.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb13affd640 1 --2- 172.21.15.29:0/1751315706 >> v2:172.21.15.186:3300/0 conn(0x7fb13c0cf710 0x7fb13c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.222+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805400277 0 0) 0x7fb13c0d18c0 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.223+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb128003520 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.223+0000 7fb13affd640 1 -- 172.21.15.29:0/1751315706 >> v1:172.21.15.186:6789/0 conn(0x7fb13c0cfaf0 legacy=0x7fb13c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.223+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997703922 0 0) 0x7fb128003520 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.223+0000 7fb13affd640 1 -- 172.21.15.29:0/1751315706 >> v1:172.21.15.29:6789/0 conn(0x7fb13c0b6cd0 legacy=0x7fb13c0d11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.223+0000 7fb13affd640 1 -- 172.21.15.29:0/1751315706 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13c0ce630 con 0x7fb13c0cf710 2023-10-14T17:58:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.224+0000 7fb13affd640 1 --2- 172.21.15.29:0/1751315706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cf710 0x7fb13c0b2d10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009de0 tx=0x7fb12c02bdc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e826364028d22ec3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.224+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb134001670 con 0x7fb13c0cf710 2023-10-14T17:58:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c02c8f0 con 0x7fb13c0cf710 2023-10-14T17:58:04.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb139ffb640 1 -- 172.21.15.29:0/1751315706 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c02cc60 con 0x7fb13c0cf710 2023-10-14T17:58:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 -- 172.21.15.29:0/1751315706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cf710 msgr2=0x7fb13c0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1751315706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cf710 0x7fb13c0b2d10 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7fb12c009de0 tx=0x7fb12c02bdc0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 -- 172.21.15.29:0/1751315706 shutdown_connections 2023-10-14T17:58:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1751315706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cf710 0x7fb13c0b2d10 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 -- 172.21.15.29:0/1751315706 >> 172.21.15.29:0/1751315706 conn(0x7fb13c0d2cc0 msgr2=0x7fb13c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.225+0000 7fb1419c9640 1 -- 172.21.15.29:0/1751315706 shutdown_connections 2023-10-14T17:58:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.226+0000 7fb1419c9640 1 -- 172.21.15.29:0/1751315706 wait complete. 2023-10-14T17:58:04.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.228+0000 7fb1419c9640 1 Processor -- start 2023-10-14T17:58:04.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.228+0000 7fb1419c9640 1 -- start start 2023-10-14T17:58:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13c0cf710 0x7fb13c155410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cfaf0 0x7fb13c160050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c155950 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15efb0 con 0x7fb13c0cfaf0 2023-10-14T17:58:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb1419c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb13c15f290 con 0x7fb13c0cf710 2023-10-14T17:58:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb13affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb13affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34482/0 (socket says 172.21.15.29:34482) 2023-10-14T17:58:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.229+0000 7fb13affd640 1 -- 172.21.15.29:0/1688225892 learned_addr learned my addr 172.21.15.29:0/1688225892 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:04.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13a7fc640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13c0cf710 0x7fb13c155410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13c0cf710 msgr2=0x7fb13c155410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13c0cf710 0x7fb13c155410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cfaf0 msgr2=0x7fb13c160050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13b7fe640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cfaf0 0x7fb13c160050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cfaf0 0x7fb13c160050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 -- 172.21.15.29:0/1688225892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb12c009a70 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.230+0000 7fb13affd640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fb12c037040 tx=0x7fb12c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.231+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c004450 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.231+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb12c0045b0 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.231+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb13c15f570 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.231+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb12c034440 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.231+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb13c15fb30 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.232+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13c04e160 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.233+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb12c03b050 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.234+0000 7fb133fff640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 0x7fb11c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.234+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb12c0abcd0 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.234+0000 7fb13a7fc640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 msgr2=0x7fb11c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:04.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.234+0000 7fb13a7fc640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 0x7fb11c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:04.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.238+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb12c076c50 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.243 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.242+0000 7f8fe037b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:04.261 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.259+0000 7fbec1432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:58:04.295 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.294+0000 7f8fe037b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:58:04.317 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.315+0000 7fbec1432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:58:04.355 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.354+0000 7f8fe037b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:58:04.367 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.366+0000 7fbec1432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:58:04.408 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.406+0000 7f8fe037b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:58:04.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.432+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb13c08c490 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.433+0000 7fb133fff640 1 -- 172.21.15.29:0/1688225892 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v179) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb12c04d020 con 0x7fb13c0b6cd0 2023-10-14T17:58:04.435 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:58:04.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb13a7fc640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 msgr2=0x7fb11c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:04.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb13a7fc640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 0x7fb11c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:04.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 msgr2=0x7fb11c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:04.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 0x7fb11c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 msgr2=0x7fb13c156060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 secure :-1 s=READY pgs=2048 cs=0 l=1 rev1=1 crypto rx=0x7fb12c037040 tx=0x7fb12c004a40 comp rx=0 tx=0).stop 2023-10-14T17:58:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 shutdown_connections 2023-10-14T17:58:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb11c06af90 0x7fb11c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13c0cfaf0 0x7fb13c160050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13c0cf710 0x7fb13c155410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 --2- 172.21.15.29:0/1688225892 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13c0b6cd0 0x7fb13c156060 unknown :-1 s=CLOSED pgs=2048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 >> 172.21.15.29:0/1688225892 conn(0x7fb13c0d2cc0 msgr2=0x7fb13c0b2ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:04.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 shutdown_connections 2023-10-14T17:58:04.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.435+0000 7fb1419c9640 1 -- 172.21.15.29:0/1688225892 wait complete. 2023-10-14T17:58:04.454 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:58:04.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.720+0000 7ff0dbfce640 1 Processor -- start 2023-10-14T17:58:04.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.720+0000 7ff0dbfce640 1 -- start start 2023-10-14T17:58:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0dbfce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0d4160ad0 0x7ff0d40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0dbfce640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40ce800 con 0x7ff0d40ae260 2023-10-14T17:58:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0dbfce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0d40cdec0 con 0x7ff0d40ce350 2023-10-14T17:58:04.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0dbfce640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0d40acdd0 con 0x7ff0d4160ad0 2023-10-14T17:58:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0da56b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff0d40ae260 0x7ff0d406a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47142/0 (socket says 172.21.15.29:47142) 2023-10-14T17:58:04.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0da56b640 1 -- 172.21.15.29:0/494947794 learned_addr learned my addr 172.21.15.29:0/494947794 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.722+0000 7ff0d9d6a640 1 --2- 172.21.15.29:0/494947794 >> v2:172.21.15.29:3300/0 conn(0x7ff0d4160ad0 0x7ff0d40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.723+0000 7ff0d9d6a640 1 -- 172.21.15.29:0/494947794 >> v1:172.21.15.186:6789/0 conn(0x7ff0d40ce350 legacy=0x7ff0d4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.723+0000 7ff0d9d6a640 1 -- 172.21.15.29:0/494947794 >> v1:172.21.15.29:6789/0 conn(0x7ff0d40ae260 legacy=0x7ff0d406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.723+0000 7ff0d9d6a640 1 -- 172.21.15.29:0/494947794 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d40ae740 con 0x7ff0d4160ad0 2023-10-14T17:58:04.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.724+0000 7ff0d9d6a640 1 --2- 172.21.15.29:0/494947794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40ac890 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097a0 tx=0x7ff0d002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39cb1f7d25d5115c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.724+0000 7ff0d8d68640 1 -- 172.21.15.29:0/494947794 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0030610 con 0x7ff0d4160ad0 2023-10-14T17:58:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.724+0000 7ff0d8d68640 1 -- 172.21.15.29:0/494947794 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d0030770 con 0x7ff0d4160ad0 2023-10-14T17:58:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.724+0000 7ff0d8d68640 1 -- 172.21.15.29:0/494947794 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d0030b00 con 0x7ff0d4160ad0 2023-10-14T17:58:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 -- 172.21.15.29:0/494947794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 msgr2=0x7ff0d40ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/494947794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40ac890 secure :-1 s=READY pgs=2050 cs=0 l=1 rev1=1 crypto rx=0x7ff0d00097a0 tx=0x7ff0d002b960 comp rx=0 tx=0).stop 2023-10-14T17:58:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 -- 172.21.15.29:0/494947794 shutdown_connections 2023-10-14T17:58:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/494947794 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40ac890 unknown :-1 s=CLOSED pgs=2050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 -- 172.21.15.29:0/494947794 >> 172.21.15.29:0/494947794 conn(0x7ff0d406bc10 msgr2=0x7ff0d406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.725+0000 7ff0dbfce640 1 -- 172.21.15.29:0/494947794 shutdown_connections 2023-10-14T17:58:04.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.726+0000 7ff0dbfce640 1 -- 172.21.15.29:0/494947794 wait complete. 2023-10-14T17:58:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.727+0000 7ff0dbfce640 1 Processor -- start 2023-10-14T17:58:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.727+0000 7ff0dbfce640 1 -- start start 2023-10-14T17:58:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.728+0000 7ff0dbfce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0d40ae260 0x7ff0d40af390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.728+0000 7ff0dbfce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0d40ce350 0x7ff0d40c7750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.728+0000 7ff0dbfce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0dbfce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f5e30 con 0x7ff0d4160ad0 2023-10-14T17:58:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0dbfce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f2d90 con 0x7ff0d40ae260 2023-10-14T17:58:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0dbfce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d41f3070 con 0x7ff0d40ce350 2023-10-14T17:58:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34502/0 (socket says 172.21.15.29:34502) 2023-10-14T17:58:04.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 -- 172.21.15.29:0/3959702780 learned_addr learned my addr 172.21.15.29:0/3959702780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0d9d6a640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0d40ae260 0x7ff0d40af390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0d9569640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0d40ce350 0x7ff0d40c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:04.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0d40ce350 msgr2=0x7ff0d40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0d40ce350 0x7ff0d40c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0d40ae260 msgr2=0x7ff0d40af390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0d40ae260 0x7ff0d40af390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.729+0000 7ff0da56b640 1 -- 172.21.15.29:0/3959702780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d0009400 con 0x7ff0d4160ad0 2023-10-14T17:58:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.730+0000 7ff0da56b640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00cbc0 tx=0x7ff0cc00a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.731+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00ad40 con 0x7ff0d4160ad0 2023-10-14T17:58:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.731+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0cc012400 con 0x7ff0d4160ad0 2023-10-14T17:58:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.731+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0cc00aea0 con 0x7ff0d4160ad0 2023-10-14T17:58:04.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.731+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d41f3350 con 0x7ff0d4160ad0 2023-10-14T17:58:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.731+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d40af8d0 con 0x7ff0d4160ad0 2023-10-14T17:58:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.732+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d404e160 con 0x7ff0d4160ad0 2023-10-14T17:58:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.733+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7ff0cc007070 con 0x7ff0d4160ad0 2023-10-14T17:58:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.734+0000 7ff0ca7fc640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 0x7ff0b806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.734+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7ff0cc0897c0 con 0x7ff0d4160ad0 2023-10-14T17:58:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.734+0000 7ff0d9d6a640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 msgr2=0x7ff0b806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:04.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.734+0000 7ff0d9d6a640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 0x7ff0b806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.738+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0cc054640 con 0x7ff0d4160ad0 2023-10-14T17:58:04.753 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.751+0000 7fbec1432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:58:04.819 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.818+0000 7f8fe037b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:58:04.911 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:04.909+0000 7fbec1432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:58:04.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.932+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff0d406c950 con 0x7ff0d4160ad0 2023-10-14T17:58:04.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.933+0000 7ff0ca7fc640 1 -- 172.21.15.29:0/3959702780 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v179) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff0cc017020 con 0x7ff0d4160ad0 2023-10-14T17:58:04.934 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:58:04.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0d9d6a640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 msgr2=0x7ff0b806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:04.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0d9d6a640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 0x7ff0b806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 msgr2=0x7ff0b806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 0x7ff0b806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 msgr2=0x7ff0d40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.935+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 secure :-1 s=READY pgs=2051 cs=0 l=1 rev1=1 crypto rx=0x7ff0cc00cbc0 tx=0x7ff0cc00a430 comp rx=0 tx=0).stop 2023-10-14T17:58:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 shutdown_connections 2023-10-14T17:58:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7ff0b806af90 0x7ff0b806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0d4160ad0 0x7ff0d40b2000 unknown :-1 s=CLOSED pgs=2051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0d40ce350 0x7ff0d40c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 --2- 172.21.15.29:0/3959702780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0d40ae260 0x7ff0d40af390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 >> 172.21.15.29:0/3959702780 conn(0x7ff0d406bc10 msgr2=0x7ff0d40ca8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 shutdown_connections 2023-10-14T17:58:04.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:04.936+0000 7ff0dbfce640 1 -- 172.21.15.29:0/3959702780 wait complete. 2023-10-14T17:58:04.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T17:58:04.992 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:04.991+0000 7f8fe037b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:58:05.020 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.019+0000 7fbec1432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:58:05.070 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.069+0000 7fbec1432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:58:05.106 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.105+0000 7f8fe037b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:58:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.159+0000 7f8b93b4e640 1 Processor -- start 2023-10-14T17:58:05.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.159+0000 7f8b93b4e640 1 -- start start 2023-10-14T17:58:05.160 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.159+0000 7f8fe037b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:58:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.160+0000 7f8b93b4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b8c06a4b0 0x7f8b8c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.160+0000 7f8b93b4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0c9ca0 0x7f8b8c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.160+0000 7f8b93b4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c0cb030 con 0x7f8b8c15ad70 2023-10-14T17:58:05.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.160+0000 7f8b93b4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c0cae30 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.160+0000 7f8b93b4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c91c0 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b918ea640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b8c15ad70 0x7f8b8c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47156/0 (socket says 172.21.15.29:47156) 2023-10-14T17:58:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b918ea640 1 -- 172.21.15.29:0/2553925461 learned_addr learned my addr 172.21.15.29:0/2553925461 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b910e9640 1 --2- 172.21.15.29:0/2553925461 >> v2:172.21.15.29:3300/0 conn(0x7f8b8c06a4b0 0x7f8b8c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b920eb640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0c9ca0 0x7f8b8c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b908e8640 1 -- 172.21.15.29:0/2553925461 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 727496904 0 0) 0x7f8b8c0cb030 con 0x7f8b8c15ad70 2023-10-14T17:58:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b908e8640 1 -- 172.21.15.29:0/2553925461 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b74003520 con 0x7f8b8c15ad70 2023-10-14T17:58:05.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b910e9640 1 -- 172.21.15.29:0/2553925461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0c9ca0 msgr2=0x7f8b8c0ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.161+0000 7f8b910e9640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0c9ca0 0x7f8b8c0ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.162+0000 7f8b910e9640 1 -- 172.21.15.29:0/2553925461 >> v1:172.21.15.29:6789/0 conn(0x7f8b8c15ad70 legacy=0x7f8b8c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.162+0000 7f8b910e9640 1 -- 172.21.15.29:0/2553925461 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0c94a0 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.163+0000 7f8b910e9640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0ca3b0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f8b800097a0 tx=0x7f8b8002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=492d9dc2feebb507 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:05.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.163+0000 7f8b908e8640 1 -- 172.21.15.29:0/2553925461 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80031830 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.163+0000 7f8b908e8640 1 -- 172.21.15.29:0/2553925461 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b80031990 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.163+0000 7f8b908e8640 1 -- 172.21.15.29:0/2553925461 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80031d20 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 -- 172.21.15.29:0/2553925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06a4b0 msgr2=0x7f8b8c0ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0ca3b0 secure :-1 s=READY pgs=2053 cs=0 l=1 rev1=1 crypto rx=0x7f8b800097a0 tx=0x7f8b8002b8d0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 -- 172.21.15.29:0/2553925461 shutdown_connections 2023-10-14T17:58:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0c9ca0 0x7f8b8c0ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/2553925461 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0ca3b0 unknown :-1 s=CLOSED pgs=2053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 -- 172.21.15.29:0/2553925461 >> 172.21.15.29:0/2553925461 conn(0x7f8b8c0ce9b0 msgr2=0x7f8b8c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 -- 172.21.15.29:0/2553925461 shutdown_connections 2023-10-14T17:58:05.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.164+0000 7f8b93b4e640 1 -- 172.21.15.29:0/2553925461 wait complete. 2023-10-14T17:58:05.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.166+0000 7f8b93b4e640 1 Processor -- start 2023-10-14T17:58:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.166+0000 7f8b93b4e640 1 -- start start 2023-10-14T17:58:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0c4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c15ad70 0x7f8b8c0ae630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0b3610 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0b3330 con 0x7f8b8c06a4b0 2023-10-14T17:58:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b93b4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c4730 con 0x7f8b8c15ad70 2023-10-14T17:58:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b910e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b910e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34530/0 (socket says 172.21.15.29:34530) 2023-10-14T17:58:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b910e9640 1 -- 172.21.15.29:0/1190109251 learned_addr learned my addr 172.21.15.29:0/1190109251 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:05.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b918ea640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0c4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.167+0000 7f8b920eb640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c15ad70 0x7f8b8c0ae630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c15ad70 msgr2=0x7f8b8c0ae630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c15ad70 0x7f8b8c0ae630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c06a4b0 msgr2=0x7f8b8c0c4f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0c4f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 -- 172.21.15.29:0/1190109251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b80009400 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b910e9640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002bee0 tx=0x7f8b800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8003a630 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b8003a7b0 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8003ab60 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c0c4a10 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.168+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c151570 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.169+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c0c26e0 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.171+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f8b8003acc0 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.171+0000 7f8b8a7fc640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.172+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f8b800ab3b0 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.172+0000 7f8b918ea640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 msgr2=0x7f8b7806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.172+0000 7f8b918ea640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:05.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.176+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b80076070 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.180 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.179+0000 7fbec1432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:05.277 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.275+0000 7f8fe037b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:05.287 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.286+0000 7fbec1432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:58:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.373+0000 7f8b918ea640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 msgr2=0x7f8b7806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:05.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.373+0000 7f8b918ea640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:05.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.382+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f8b8c0c4c30 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.383+0000 7f8b8a7fc640 1 -- 172.21.15.29:0/1190109251 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v179) v1 ==== 80+0+468 (secure 0 0 0) 0x7f8b80079690 con 0x7f8b8c0c9ca0 2023-10-14T17:58:05.385 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3415492501 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.385 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2674099337 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.385 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2596229485 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.385 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4106038121 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.385 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/2441704285 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.386 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/2441704285 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.386 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1011592268 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.386 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/84821019 2023-10-15T17:57:59.333970+0000 2023-10-14T17:58:05.386 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T17:58:05.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 msgr2=0x7f8b7806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 msgr2=0x7f8b8c150660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 secure :-1 s=READY pgs=2054 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002bee0 tx=0x7f8b800046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 shutdown_connections 2023-10-14T17:58:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c15ad70 0x7f8b8c0ae630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0c9ca0 0x7f8b8c150660 unknown :-1 s=CLOSED pgs=2054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 --2- 172.21.15.29:0/1190109251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c06a4b0 0x7f8b8c0c4f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 >> 172.21.15.29:0/1190109251 conn(0x7f8b8c0ce9b0 msgr2=0x7f8b8c069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 shutdown_connections 2023-10-14T17:58:05.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.386+0000 7f8b93b4e640 1 -- 172.21.15.29:0/1190109251 wait complete. 2023-10-14T17:58:05.390 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.390+0000 7f8fe037b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:58:05.405 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T17:58:05.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:05.406 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T17:58:05.444 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.442+0000 7fbec1432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:05.494 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.492+0000 7fbec1432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:05.550 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T17:58:05.550 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:05.551 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T17:58:05.560 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.559+0000 7f8fe037b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:05.597 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:58:05.598 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:05.613 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:05.611+0000 7fbec1432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:58:05.614 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.613+0000 7f8fe037b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:05.653 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:58:05.654 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:05.654 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:05.709 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:58:05.710 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:58:05.710 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:05.741 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:05.739+0000 7f8fe037b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:58:05.766 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:58:05.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.978+0000 7fd909cd1640 1 Processor -- start 2023-10-14T17:58:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.978+0000 7fd909cd1640 1 -- start start 2023-10-14T17:58:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.980+0000 7fd909cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd904159950 0x7fd90415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.980+0000 7fd909cd1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd90406c810 0x7fd9040708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.980+0000 7fd909cd1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd904070df0 con 0x7fd904070450 2023-10-14T17:58:05.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.980+0000 7fd909cd1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90415e490 con 0x7fd90406c810 2023-10-14T17:58:05.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.980+0000 7fd909cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90415aef0 con 0x7fd904159950 2023-10-14T17:58:05.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd90406c810 0x7fd9040708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd90406c810 0x7fd9040708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34542/0 (socket says 172.21.15.29:34542) 2023-10-14T17:58:05.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 -- 172.21.15.29:0/2406041857 learned_addr learned my addr 172.21.15.29:0/2406041857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:05.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd908ccf640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd904159950 0x7fd90415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 -- 172.21.15.29:0/2406041857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd904159950 msgr2=0x7fd90415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd904159950 0x7fd90415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 -- 172.21.15.29:0/2406041857 >> v1:172.21.15.186:6789/0 conn(0x7fd904070450 legacy=0x7fd90406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.981+0000 7fd9094d0640 1 -- 172.21.15.29:0/2406041857 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90406cc80 con 0x7fd90406c810 2023-10-14T17:58:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.982+0000 7fd9094d0640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd90406c810 0x7fd9040708b0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009870 tx=0x7fd8fc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd44dc5d72ae5adc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2406041857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031640 con 0x7fd90406c810 2023-10-14T17:58:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2406041857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8fc0317a0 con 0x7fd90406c810 2023-10-14T17:58:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd8fb7fe640 1 -- 172.21.15.29:0/2406041857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8fc031b30 con 0x7fd90406c810 2023-10-14T17:58:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 -- 172.21.15.29:0/2406041857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd90406c810 msgr2=0x7fd9040708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd90406c810 0x7fd9040708b0 secure :-1 s=READY pgs=2055 cs=0 l=1 rev1=1 crypto rx=0x7fd8fc009870 tx=0x7fd8fc02b8f0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 -- 172.21.15.29:0/2406041857 shutdown_connections 2023-10-14T17:58:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd90406c810 0x7fd9040708b0 unknown :-1 s=CLOSED pgs=2055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 --2- 172.21.15.29:0/2406041857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd904159950 0x7fd90415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.983+0000 7fd909cd1640 1 -- 172.21.15.29:0/2406041857 >> 172.21.15.29:0/2406041857 conn(0x7fd9040ce6a0 msgr2=0x7fd9040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:05.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.984+0000 7fd909cd1640 1 -- 172.21.15.29:0/2406041857 shutdown_connections 2023-10-14T17:58:05.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.984+0000 7fd909cd1640 1 -- 172.21.15.29:0/2406041857 wait complete. 2023-10-14T17:58:05.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.985+0000 7fd909cd1640 1 Processor -- start 2023-10-14T17:58:05.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.985+0000 7fd909cd1640 1 -- start start 2023-10-14T17:58:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c810 0x7fd9041f30a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904159950 0x7fd90415cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040b3e40 con 0x7fd904070450 2023-10-14T17:58:05.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040ae070 con 0x7fd904159950 2023-10-14T17:58:05.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.986+0000 7fd909cd1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90415d020 con 0x7fd90406c810 2023-10-14T17:58:05.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34558/0 (socket says 172.21.15.29:34558) 2023-10-14T17:58:05.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 -- 172.21.15.29:0/400293175 learned_addr learned my addr 172.21.15.29:0/400293175 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c810 msgr2=0x7fd9041f30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c810 0x7fd9041f30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904159950 msgr2=0x7fd90415cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:05.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd9094d0640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904159950 0x7fd90415cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904159950 0x7fd90415cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 -- 172.21.15.29:0/400293175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8fc0094d0 con 0x7fd904070450 2023-10-14T17:58:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.987+0000 7fd8fbfff640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c450 tx=0x7fd8f400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.988+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40052d0 con 0x7fd904070450 2023-10-14T17:58:05.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.988+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4005430 con 0x7fd904070450 2023-10-14T17:58:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.988+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd90415b090 con 0x7fd904070450 2023-10-14T17:58:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.988+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4012870 con 0x7fd904070450 2023-10-14T17:58:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.988+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90415b650 con 0x7fd904070450 2023-10-14T17:58:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.989+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90404e9b0 con 0x7fd904070450 2023-10-14T17:58:05.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.990+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fd8f40129d0 con 0x7fd904070450 2023-10-14T17:58:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.991+0000 7fd8f97fa640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.992+0000 7fd908ccf640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 msgr2=0x7fd8f006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.992+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fd8f40894b0 con 0x7fd904070450 2023-10-14T17:58:05.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.992+0000 7fd908ccf640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:05.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:05.996+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8f4054270 con 0x7fd904070450 2023-10-14T17:58:06.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.193+0000 7fd908ccf640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 msgr2=0x7fd8f006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:06.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.193+0000 7fd908ccf640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:06.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.229+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9040b4620 con 0x7fd904070450 2023-10-14T17:58:06.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.229+0000 7fd8f97fa640 1 -- 172.21.15.29:0/400293175 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fd8f4057890 con 0x7fd904070450 2023-10-14T17:58:06.231 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:06.231 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:58:06.231 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:58:06.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 msgr2=0x7fd8f006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:06.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 msgr2=0x7fd9041557f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 secure :-1 s=READY pgs=2056 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c450 tx=0x7fd8f400c910 comp rx=0 tx=0).stop 2023-10-14T17:58:06.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 shutdown_connections 2023-10-14T17:58:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fd8f006af90 0x7fd8f006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd904159950 0x7fd90415cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd904070450 0x7fd9041557f0 unknown :-1 s=CLOSED pgs=2056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 --2- 172.21.15.29:0/400293175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c810 0x7fd9041f30a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 >> 172.21.15.29:0/400293175 conn(0x7fd9040ce6a0 msgr2=0x7fd9040cedb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:06.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 shutdown_connections 2023-10-14T17:58:06.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.232+0000 7fd909cd1640 1 -- 172.21.15.29:0/400293175 wait complete. 2023-10-14T17:58:06.253 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:06.432 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:06.430+0000 7fbec1432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:58:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.437+0000 7f35a61c9640 1 Processor -- start 2023-10-14T17:58:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.437+0000 7f35a61c9640 1 -- start start 2023-10-14T17:58:06.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.438+0000 7f35a61c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 0x7f35a006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.438+0000 7f35a61c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f35a00caa10 0x7f35a006fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.438+0000 7f35a61c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35a0159790 con 0x7f35a006ffd0 2023-10-14T17:58:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.438+0000 7f35a61c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35a00c9b20 con 0x7f35a00caa10 2023-10-14T17:58:06.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.438+0000 7f35a61c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c9e00 con 0x7f35a01593b0 2023-10-14T17:58:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35a006ffd0 0x7f35a015ceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47164/0 (socket says 172.21.15.29:47164) 2023-10-14T17:58:06.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359ffff640 1 -- 172.21.15.29:0/2240564890 learned_addr learned my addr 172.21.15.29:0/2240564890 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:06.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> v2:172.21.15.186:3300/0 conn(0x7f35a00caa10 0x7f35a006fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698808182 0 0) 0x7f35a0159790 con 0x7f35a006ffd0 2023-10-14T17:58:06.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f358c003520 con 0x7f35a006ffd0 2023-10-14T17:58:06.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359f7fe640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 0x7f35a006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3853381530 0 0) 0x7f358c003520 con 0x7f35a006ffd0 2023-10-14T17:58:06.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.439+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 msgr2=0x7f35a006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 0x7f35a006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 >> v2:172.21.15.186:3300/0 conn(0x7f35a00caa10 msgr2=0x7f35a006fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 --2- 172.21.15.29:0/2240564890 >> v2:172.21.15.186:3300/0 conn(0x7f35a00caa10 0x7f35a006fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00ca0e0 con 0x7f35a006ffd0 2023-10-14T17:58:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f35980019f0 con 0x7f35a006ffd0 2023-10-14T17:58:06.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> v2:172.21.15.186:3300/0 conn(0x7f35a00caa10 0x7f35a006fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:06.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 >> v1:172.21.15.29:6789/0 conn(0x7f35a006ffd0 legacy=0x7f35a015ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358c003970 0x7f358c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f358c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 0x7f358c0051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c9b20 con 0x7f358c0047c0 2023-10-14T17:58:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c9e00 con 0x7f358c003970 2023-10-14T17:58:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f358c00dd20 con 0x7f35a01593b0 2023-10-14T17:58:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f358c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 -- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 msgr2=0x7f358c0051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a01593b0 0x7f358c0051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 -- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358c003970 msgr2=0x7f358c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f358c003970 0x7f358c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.440+0000 7f3597fff640 1 -- 172.21.15.29:0/2240564890 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c003520 con 0x7f358c0047c0 2023-10-14T17:58:06.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f3597fff640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f358c004c40 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f35880098a0 tx=0x7f358c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98672decda55f561 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588004230 con 0x7f358c0047c0 2023-10-14T17:58:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3588004390 con 0x7f358c0047c0 2023-10-14T17:58:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f359effd640 1 -- 172.21.15.29:0/2240564890 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3588003830 con 0x7f358c0047c0 2023-10-14T17:58:06.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f35a61c9640 1 -- 172.21.15.29:0/2240564890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 msgr2=0x7f358c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2240564890 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f358c004c40 secure :-1 s=READY pgs=2058 cs=0 l=1 rev1=1 crypto rx=0x7f35880098a0 tx=0x7f358c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.442+0000 7f359ffff640 1 -- 172.21.15.29:0/2240564890 reap_dead start 2023-10-14T17:58:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.443+0000 7f35a61c9640 1 -- 172.21.15.29:0/2240564890 shutdown_connections 2023-10-14T17:58:06.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.443+0000 7f35a61c9640 1 -- 172.21.15.29:0/2240564890 >> 172.21.15.29:0/2240564890 conn(0x7f35a00cb410 msgr2=0x7f35a00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.443+0000 7f35a61c9640 1 -- 172.21.15.29:0/2240564890 shutdown_connections 2023-10-14T17:58:06.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.443+0000 7f35a61c9640 1 -- 172.21.15.29:0/2240564890 wait complete. 2023-10-14T17:58:06.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.444+0000 7f35a61c9640 1 Processor -- start 2023-10-14T17:58:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.445+0000 7f35a61c9640 1 -- start start 2023-10-14T17:58:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.445+0000 7f35a61c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.445+0000 7f35a61c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a00c7ef0 0x7f35a00c8370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f35a61c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35a00b1530 0x7f35a00b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f35a61c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c6a30 con 0x7f358c0047c0 2023-10-14T17:58:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f35a61c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c7a60 con 0x7f35a00b1530 2023-10-14T17:58:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f35a61c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35a00c6750 con 0x7f35a00c7ef0 2023-10-14T17:58:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34578/0 (socket says 172.21.15.29:34578) 2023-10-14T17:58:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 -- 172.21.15.29:0/2734282158 learned_addr learned my addr 172.21.15.29:0/2734282158 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:06.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a00c7ef0 msgr2=0x7f35a00c8370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a00c7ef0 0x7f35a00c8370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35a00b1530 msgr2=0x7f35a00b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35a00b1530 0x7f35a00b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.446+0000 7f359f7fe640 1 -- 172.21.15.29:0/2734282158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35a00c7410 con 0x7f358c0047c0 2023-10-14T17:58:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f359f7fe640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f359000e860 tx=0x7f359000ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f359000cc40 con 0x7f358c0047c0 2023-10-14T17:58:06.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f359000cda0 con 0x7f358c0047c0 2023-10-14T17:58:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35a00c76f0 con 0x7f358c0047c0 2023-10-14T17:58:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3590011610 con 0x7f358c0047c0 2023-10-14T17:58:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.447+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35a01542e0 con 0x7f358c0047c0 2023-10-14T17:58:06.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.448+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35a00af560 con 0x7f358c0047c0 2023-10-14T17:58:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.449+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7f3590011770 con 0x7f358c0047c0 2023-10-14T17:58:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.450+0000 7f359cff9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 0x7f358006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.450+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3590089660 con 0x7f358c0047c0 2023-10-14T17:58:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.450+0000 7f3597fff640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 msgr2=0x7f358006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:06.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.450+0000 7f3597fff640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 0x7f358006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:06.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.454+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3590054390 con 0x7f358c0047c0 2023-10-14T17:58:06.610 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:06.608+0000 7f8fe037b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:58:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.651+0000 7f3597fff640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 msgr2=0x7f358006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:06.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.651+0000 7f3597fff640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 0x7f358006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:06.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.654+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f35a004e160 con 0x7f358c0047c0 2023-10-14T17:58:06.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.656+0000 7f359cff9640 1 -- 172.21.15.29:0/2734282158 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v179) v1 ==== 74+0+17252 (secure 0 0 0) 0x7f35900579b0 con 0x7f358c0047c0 2023-10-14T17:58:06.658 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:06.658 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":179,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:57:59.334060+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3415492501":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2674099337":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2596229485":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/4106038121":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6832/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1011592268":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/84821019":"2023-10-15T17:57:59.333970+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 msgr2=0x7f358006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:06.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 0x7f358006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 msgr2=0x7f35a015af00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 secure :-1 s=READY pgs=2059 cs=0 l=1 rev1=1 crypto rx=0x7f359000e860 tx=0x7f359000ed20 comp rx=0 tx=0).stop 2023-10-14T17:58:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 shutdown_connections 2023-10-14T17:58:06.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7f358006b010 0x7f358006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f35a00b1530 0x7f35a00b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35a00c7ef0 0x7f35a00c8370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 --2- 172.21.15.29:0/2734282158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f358c0047c0 0x7f35a015af00 unknown :-1 s=CLOSED pgs=2059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 >> 172.21.15.29:0/2734282158 conn(0x7f35a00cb410 msgr2=0x7f35a00cd9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:06.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 shutdown_connections 2023-10-14T17:58:06.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.659+0000 7f35a61c9640 1 -- 172.21.15.29:0/2734282158 wait complete. 2023-10-14T17:58:06.682 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 20, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:57:42.346345+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '177', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 21, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:57:43.390271+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '178', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:58:06.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:58:06.729 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:06.727+0000 7fbec1432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:58:06.788 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:06.786+0000 7fbec1432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:58:06.902 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:06.900+0000 7fbec1432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:58:06.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.901+0000 7fb25dfb3640 1 Processor -- start 2023-10-14T17:58:06.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.901+0000 7fb25dfb3640 1 -- start start 2023-10-14T17:58:06.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.903+0000 7fb25dfb3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb2580cfc40 0x7fb2580d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.903+0000 7fb25dfb3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580d0970 con 0x7fb2580b2080 2023-10-14T17:58:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.903+0000 7fb25dfb3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2580d0ea0 con 0x7fb2580b1ca0 2023-10-14T17:58:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.903+0000 7fb25dfb3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2580d0bc0 con 0x7fb2580cfc40 2023-10-14T17:58:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb257fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2580b2080 0x7fb2580b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47166/0 (socket says 172.21.15.29:47166) 2023-10-14T17:58:06.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb257fff640 1 -- 172.21.15.29:0/4138120330 learned_addr learned my addr 172.21.15.29:0/4138120330 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:06.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb25d7b2640 1 --2- 172.21.15.29:0/4138120330 >> v2:172.21.15.186:3300/0 conn(0x7fb2580cfc40 0x7fb2580d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1949773737 0 0) 0x7fb2580d0970 con 0x7fb2580b2080 2023-10-14T17:58:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb248003520 con 0x7fb2580b2080 2023-10-14T17:58:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717879885 0 0) 0x7fb2580d0ea0 con 0x7fb2580b1ca0 2023-10-14T17:58:06.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2580d0970 con 0x7fb2580b1ca0 2023-10-14T17:58:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1145956706 0 0) 0x7fb248003520 con 0x7fb2580b2080 2023-10-14T17:58:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 >> v2:172.21.15.186:3300/0 conn(0x7fb2580cfc40 msgr2=0x7fb2580d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.904+0000 7fb2577fe640 1 --2- 172.21.15.29:0/4138120330 >> v2:172.21.15.186:3300/0 conn(0x7fb2580cfc40 0x7fb2580d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 >> v1:172.21.15.186:6789/0 conn(0x7fb2580b1ca0 legacy=0x7fb2580ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2580cd940 con 0x7fb2580b2080 2023-10-14T17:58:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb244001670 con 0x7fb2580b2080 2023-10-14T17:58:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 >> v1:172.21.15.29:6789/0 conn(0x7fb2580b2080 legacy=0x7fb2580b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb248003b00 0x7fb248003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2580d0bc0 con 0x7fb2480047c0 2023-10-14T17:58:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800bf70 con 0x7fb248005180 2023-10-14T17:58:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800c130 con 0x7fb248003b00 2023-10-14T17:58:06.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb257fff640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb25d7b2640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.905+0000 7fb25cfb1640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb248003b00 0x7fb248003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb257fff640 1 -- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb248003b00 msgr2=0x7fb248003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb257fff640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb248003b00 0x7fb248003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb257fff640 1 -- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248005180 msgr2=0x7fb24800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb257fff640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb257fff640 1 -- 172.21.15.29:0/4138120330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248003520 con 0x7fb2480047c0 2023-10-14T17:58:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb25d7b2640 1 -- 172.21.15.29:0/4138120330 reap_dead start 2023-10-14T17:58:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.906+0000 7fb25cfb1640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb248003b00 0x7fb248003f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:06.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.907+0000 7fb257fff640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb248004c40 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb244001eb0 tx=0x7fb2480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10218468507211e8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.907+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb244005ad0 con 0x7fb2480047c0 2023-10-14T17:58:06.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.907+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb244005c50 con 0x7fb2480047c0 2023-10-14T17:58:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.907+0000 7fb2577fe640 1 -- 172.21.15.29:0/4138120330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24402c810 con 0x7fb2480047c0 2023-10-14T17:58:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 -- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 msgr2=0x7fb248004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb248004c40 secure :-1 s=READY pgs=2061 cs=0 l=1 rev1=1 crypto rx=0x7fb244001eb0 tx=0x7fb2480057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 -- 172.21.15.29:0/4138120330 shutdown_connections 2023-10-14T17:58:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/4138120330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=CLOSED pgs=2061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 -- 172.21.15.29:0/4138120330 >> 172.21.15.29:0/4138120330 conn(0x7fb2580d2b90 msgr2=0x7fb2580d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 -- 172.21.15.29:0/4138120330 shutdown_connections 2023-10-14T17:58:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.908+0000 7fb25dfb3640 1 -- 172.21.15.29:0/4138120330 wait complete. 2023-10-14T17:58:06.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.909+0000 7fb25dfb3640 1 Processor -- start 2023-10-14T17:58:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.910+0000 7fb25dfb3640 1 -- start start 2023-10-14T17:58:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.910+0000 7fb25dfb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25dfb3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2580c7340 0x7fb2580c8dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25dfb3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2580c9300 0x7fb25815f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25dfb3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25815d110 con 0x7fb2480047c0 2023-10-14T17:58:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25dfb3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb25815d3f0 con 0x7fb2580c7340 2023-10-14T17:58:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25dfb3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb25815d6d0 con 0x7fb2580c9300 2023-10-14T17:58:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34594/0 (socket says 172.21.15.29:34594) 2023-10-14T17:58:06.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 -- 172.21.15.29:0/1402291176 learned_addr learned my addr 172.21.15.29:0/1402291176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb257fff640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2580c7340 0x7fb2580c8dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25d7b2640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2580c9300 0x7fb25815f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 -- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2580c9300 msgr2=0x7fb25815f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2580c9300 0x7fb25815f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 -- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2580c7340 msgr2=0x7fb2580c8dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:06.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2580c7340 0x7fb2580c8dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:06.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.911+0000 7fb25cfb1640 1 -- 172.21.15.29:0/1402291176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb25815d9b0 con 0x7fb2480047c0 2023-10-14T17:58:06.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb25cfb1640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00d560 tx=0x7fb24c00da20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c007220 con 0x7fb2480047c0 2023-10-14T17:58:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb24c007380 con 0x7fb2480047c0 2023-10-14T17:58:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb24c0067d0 con 0x7fb2480047c0 2023-10-14T17:58:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb25815dc90 con 0x7fb2480047c0 2023-10-14T17:58:06.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.912+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb25815faa0 con 0x7fb2480047c0 2023-10-14T17:58:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.914+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2580cdee0 con 0x7fb2480047c0 2023-10-14T17:58:06.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.915+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 99) v1 ==== 89332+0+0 (secure 0 0 0) 0x7fb24c006930 con 0x7fb2480047c0 2023-10-14T17:58:06.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.916+0000 7fb2557fa640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 0x7fb23c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:06.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.916+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(179..179 src has 1..179) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fb24c0897a0 con 0x7fb2480047c0 2023-10-14T17:58:06.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.916+0000 7fb257fff640 1 -- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 msgr2=0x7fb23c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1035847590 2023-10-14T17:58:06.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.916+0000 7fb257fff640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 0x7fb23c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:06.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:06.921+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2580cdee0 con 0x7fb2480047c0 2023-10-14T17:58:06.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:06.939+0000 7f8fe037b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:58:07.000 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:06.998+0000 7f8fe037b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:58:07.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.113+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb2580cdee0 con 0x7fb2480047c0 2023-10-14T17:58:07.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.114+0000 7fb2557fa640 1 -- 172.21.15.29:0/1402291176 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v179) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb2580cdee0 con 0x7fb2480047c0 2023-10-14T17:58:07.115 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:58:07.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.116+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 msgr2=0x7fb23c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:07.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.116+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 0x7fb23c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 msgr2=0x7fb258158860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 secure :-1 s=READY pgs=2062 cs=0 l=1 rev1=1 crypto rx=0x7fb24c00d560 tx=0x7fb24c00da20 comp rx=0 tx=0).stop 2023-10-14T17:58:07.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 shutdown_connections 2023-10-14T17:58:07.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:6832/1035847590,v1:172.21.15.29:6833/1035847590] conn(0x7fb23c06af90 0x7fb23c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2580c9300 0x7fb25815f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2580c7340 0x7fb2580c8dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 --2- 172.21.15.29:0/1402291176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2480047c0 0x7fb258158860 unknown :-1 s=CLOSED pgs=2062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 >> 172.21.15.29:0/1402291176 conn(0x7fb2580d2b90 msgr2=0x7fb2580b2460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 shutdown_connections 2023-10-14T17:58:07.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.117+0000 7fb25dfb3640 1 -- 172.21.15.29:0/1402291176 wait complete. 2023-10-14T17:58:07.121 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.118+0000 7f8fe037b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:58:07.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T17:58:07.207 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.205+0000 7f8f6142e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.205+0000 7f8f6142e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.205+0000 7f8f6142e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.205+0000 7f8f6142e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.208 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.205+0000 7f8f6142e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.206+0000 7f8f64434640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.206+0000 7f8f64434640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.206+0000 7f8f64434640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.209 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.206+0000 7f8f64434640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.210 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:07.206+0000 7f8f64434640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:07.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.327+0000 7f44b0e4d640 1 Processor -- start 2023-10-14T17:58:07.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.327+0000 7f44b0e4d640 1 -- start start 2023-10-14T17:58:07.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.328+0000 7f44b0e4d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f44ac0ca0a0 0x7f44ac06c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.328+0000 7f44b0e4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac15d910 con 0x7f44ac070b60 2023-10-14T17:58:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.328+0000 7f44b0e4d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44ac15dbf0 con 0x7f44ac159950 2023-10-14T17:58:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.328+0000 7f44b0e4d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44ac15e490 con 0x7f44ac0ca0a0 2023-10-14T17:58:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aaffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f44ac070b60 0x7f44ac06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47182/0 (socket says 172.21.15.29:47182) 2023-10-14T17:58:07.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aaffd640 1 -- 172.21.15.29:0/3761244857 learned_addr learned my addr 172.21.15.29:0/3761244857 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966185523 0 0) 0x7f44ac15d910 con 0x7f44ac070b60 2023-10-14T17:58:07.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44abfff640 1 --2- 172.21.15.29:0/3761244857 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0ca0a0 0x7f44ac06c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4494003520 con 0x7f44ac070b60 2023-10-14T17:58:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 216991978 0 0) 0x7f4494003520 con 0x7f44ac070b60 2023-10-14T17:58:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0ca0a0 msgr2=0x7f44ac06c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:07.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 --2- 172.21.15.29:0/3761244857 >> v2:172.21.15.186:3300/0 conn(0x7f44ac0ca0a0 0x7f44ac06c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 >> v1:172.21.15.186:6789/0 conn(0x7f44ac159950 legacy=0x7f44ac070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac15ded0 con 0x7f44ac070b60 2023-10-14T17:58:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f44a0001670 con 0x7f44ac070b60 2023-10-14T17:58:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.329+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 >> v1:172.21.15.29:6789/0 conn(0x7f44ac070b60 legacy=0x7f44ac06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 0x7f4494003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44940047c0 0x7f4494004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44940052a0 0x7f449400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac15e490 con 0x7f4494003970 2023-10-14T17:58:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f449400c150 con 0x7f44940047c0 2023-10-14T17:58:07.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f449400c310 con 0x7f44940052a0 2023-10-14T17:58:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 0x7f4494003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 -- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44940052a0 msgr2=0x7f449400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44940052a0 0x7f449400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 -- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44940047c0 msgr2=0x7f4494004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:07.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44940047c0 0x7f4494004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44ab7fe640 1 -- 172.21.15.29:0/3761244857 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4494003520 con 0x7f4494003970 2023-10-14T17:58:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.330+0000 7f44abfff640 1 -- 172.21.15.29:0/3761244857 reap_dead start 2023-10-14T17:58:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.331+0000 7f44ab7fe640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 0x7f4494003d70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f449c003d80 tx=0x7f44940057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebe9394089ba5244 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c032600 con 0x7f4494003970 2023-10-14T17:58:07.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449c02e050 con 0x7f4494003970 2023-10-14T17:58:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44aa7fc640 1 -- 172.21.15.29:0/3761244857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449c032a20 con 0x7f4494003970 2023-10-14T17:58:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 -- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 msgr2=0x7f4494003d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 0x7f4494003d70 secure :-1 s=READY pgs=2075 cs=0 l=1 rev1=1 crypto rx=0x7f449c003d80 tx=0x7f44940057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 -- 172.21.15.29:0/3761244857 shutdown_connections 2023-10-14T17:58:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/3761244857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4494003970 0x7f4494003d70 unknown :-1 s=CLOSED pgs=2075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 -- 172.21.15.29:0/3761244857 >> 172.21.15.29:0/3761244857 conn(0x7f44ac0ce6a0 msgr2=0x7f44ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 -- 172.21.15.29:0/3761244857 shutdown_connections 2023-10-14T17:58:07.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.332+0000 7f44b0e4d640 1 -- 172.21.15.29:0/3761244857 wait complete. 2023-10-14T17:58:07.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.334+0000 7f44b0e4d640 1 Processor -- start 2023-10-14T17:58:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.334+0000 7f44b0e4d640 1 -- start start 2023-10-14T17:58:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4494003970 0x7f44ac1f1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c7720 0x7f44ac06f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac0b2140 con 0x7f44ac0c72a0 2023-10-14T17:58:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac0b2420 con 0x7f44ac0c7720 2023-10-14T17:58:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44b0e4d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44ac0b2700 con 0x7f4494003970 2023-10-14T17:58:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.335+0000 7f44aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34708/0 (socket says 172.21.15.29:34708) 2023-10-14T17:58:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 -- 172.21.15.29:0/1069481328 learned_addr learned my addr 172.21.15.29:0/1069481328 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44ab7fe640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4494003970 0x7f44ac1f1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44abfff640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c7720 0x7f44ac06f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 -- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4494003970 msgr2=0x7f44ac1f1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4494003970 0x7f44ac1f1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 -- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c7720 msgr2=0x7f44ac06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c7720 0x7f44ac06f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 -- 172.21.15.29:0/1069481328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44ac0b29e0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.336+0000 7f44aaffd640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f44a0001eb0 tx=0x7f44a000ccd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:07.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.337+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a00074e0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.337+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44a0007640 con 0x7f44ac0c72a0 2023-10-14T17:58:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.337+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44a0004a00 con 0x7f44ac0c72a0 2023-10-14T17:58:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.337+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ac0b2cc0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.337+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44ac0b1eb0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.338+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44ac04e9b0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.339+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 100) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f44a0004b60 con 0x7f44ac0c72a0 2023-10-14T17:58:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.340+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f44a0088800 con 0x7f44ac0c72a0 2023-10-14T17:58:07.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.344+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44a00539d0 con 0x7f44ac0c72a0 2023-10-14T17:58:07.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.529+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f44ac0b4620 con 0x7f44ac0c72a0 2023-10-14T17:58:07.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.530+0000 7f4493fff640 1 -- 172.21.15.29:0/1069481328 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v180) v1 ==== 104+0+11 (secure 0 0 0) 0x7f44a0018020 con 0x7f44ac0c72a0 2023-10-14T17:58:07.532 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T17:58:07.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 msgr2=0x7f44ac06f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 secure :-1 s=READY pgs=2076 cs=0 l=1 rev1=1 crypto rx=0x7f44a0001eb0 tx=0x7f44a000ccd0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 shutdown_connections 2023-10-14T17:58:07.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44ac0c7720 0x7f44ac06f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44ac0c72a0 0x7f44ac06f1f0 unknown :-1 s=CLOSED pgs=2076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 --2- 172.21.15.29:0/1069481328 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4494003970 0x7f44ac1f1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 >> 172.21.15.29:0/1069481328 conn(0x7f44ac0ce6a0 msgr2=0x7f44ac15ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:07.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 shutdown_connections 2023-10-14T17:58:07.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.533+0000 7f44b0e4d640 1 -- 172.21.15.29:0/1069481328 wait complete. 2023-10-14T17:58:07.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T17:58:07.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.772+0000 7f0590ab4640 1 Processor -- start 2023-10-14T17:58:07.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.772+0000 7f0590ab4640 1 -- start start 2023-10-14T17:58:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.773+0000 7f0590ab4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f058c06fb50 0x7f058c0cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.773+0000 7f0590ab4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f0590ab4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f058c0ca560 con 0x7f058c0c9060 2023-10-14T17:58:07.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f0590ab4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f058c0c9950 con 0x7f058c06fb50 2023-10-14T17:58:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f0590ab4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0c9c30 con 0x7f058c06c9e0 2023-10-14T17:58:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f058b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f058c0c9060 0x7f058c06f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47194/0 (socket says 172.21.15.29:47194) 2023-10-14T17:58:07.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f058b7fe640 1 -- 172.21.15.29:0/3827081588 learned_addr learned my addr 172.21.15.29:0/3827081588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:07.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f058affd640 1 --2- 172.21.15.29:0/3827081588 >> v2:172.21.15.29:3300/0 conn(0x7f058c06fb50 0x7f058c0cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f058bfff640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.774+0000 7f058affd640 1 -- 172.21.15.29:0/3827081588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 msgr2=0x7f058c0ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.775+0000 7f058affd640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.775+0000 7f058affd640 1 -- 172.21.15.29:0/3827081588 >> v1:172.21.15.29:6789/0 conn(0x7f058c0c9060 legacy=0x7f058c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.775+0000 7f058affd640 1 -- 172.21.15.29:0/3827081588 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f058c0c9f10 con 0x7f058c06fb50 2023-10-14T17:58:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.775+0000 7f058a7fc640 1 -- 172.21.15.29:0/3827081588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440120269 0 0) 0x7f058c0ca560 con 0x7f058c0c9060 2023-10-14T17:58:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.775+0000 7f058bfff640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:07.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f058affd640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c06fb50 0x7f058c0cc8a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f05800097a0 tx=0x7f058002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=626682556654d8bf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f058a7fc640 1 -- 172.21.15.29:0/3827081588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f058002d040 con 0x7f058c06fb50 2023-10-14T17:58:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f058a7fc640 1 -- 172.21.15.29:0/3827081588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0580031810 con 0x7f058c06fb50 2023-10-14T17:58:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f058a7fc640 1 -- 172.21.15.29:0/3827081588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0580031ba0 con 0x7f058c06fb50 2023-10-14T17:58:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f0590ab4640 1 -- 172.21.15.29:0/3827081588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c06fb50 msgr2=0x7f058c0cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.776+0000 7f0590ab4640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c06fb50 0x7f058c0cc8a0 secure :-1 s=READY pgs=2078 cs=0 l=1 rev1=1 crypto rx=0x7f05800097a0 tx=0x7f058002b8b0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 -- 172.21.15.29:0/3827081588 shutdown_connections 2023-10-14T17:58:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 --2- 172.21.15.29:0/3827081588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c06fb50 0x7f058c0cc8a0 unknown :-1 s=CLOSED pgs=2078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 -- 172.21.15.29:0/3827081588 >> 172.21.15.29:0/3827081588 conn(0x7f058c0ce6a0 msgr2=0x7f058c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:07.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 -- 172.21.15.29:0/3827081588 shutdown_connections 2023-10-14T17:58:07.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.777+0000 7f0590ab4640 1 -- 172.21.15.29:0/3827081588 wait complete. 2023-10-14T17:58:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.778+0000 7f0590ab4640 1 Processor -- start 2023-10-14T17:58:07.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.778+0000 7f0590ab4640 1 -- start start 2023-10-14T17:58:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f058c06fb50 0x7f058c163950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0c3130 con 0x7f058c0c9060 2023-10-14T17:58:07.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f058c157120 con 0x7f058c06fb50 2023-10-14T17:58:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.779+0000 7f0590ab4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f058c0c2090 con 0x7f058c06c9e0 2023-10-14T17:58:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34728/0 (socket says 172.21.15.29:34728) 2023-10-14T17:58:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 -- 172.21.15.29:0/2315378553 learned_addr learned my addr 172.21.15.29:0/2315378553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:07.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 -- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 msgr2=0x7f058c0c4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 -- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f058c06fb50 msgr2=0x7f058c163950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f058c06fb50 0x7f058c163950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 -- 172.21.15.29:0/2315378553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0580009400 con 0x7f058c0c9060 2023-10-14T17:58:07.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.780+0000 7f058bfff640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f058400c7f0 tx=0x7f058400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.781+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584007820 con 0x7f058c0c9060 2023-10-14T17:58:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.781+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0584007980 con 0x7f058c0c9060 2023-10-14T17:58:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.781+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0584007d70 con 0x7f058c0c9060 2023-10-14T17:58:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.781+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f058c0c2370 con 0x7f058c0c9060 2023-10-14T17:58:07.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.781+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f058c0c2930 con 0x7f058c0c9060 2023-10-14T17:58:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.782+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f058c04e9b0 con 0x7f058c0c9060 2023-10-14T17:58:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.783+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 100) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f05840113f0 con 0x7f058c0c9060 2023-10-14T17:58:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.784+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f0584089880 con 0x7f058c0c9060 2023-10-14T17:58:07.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.788+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0584054960 con 0x7f058c0c9060 2023-10-14T17:58:07.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.987+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f058c0b4620 con 0x7f058c0c9060 2023-10-14T17:58:07.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.988+0000 7f0573fff640 1 -- 172.21.15.29:0/2315378553 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v180) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0584057f80 con 0x7f058c0c9060 2023-10-14T17:58:07.990 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T17:58:07.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 msgr2=0x7f058c1562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:07.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 secure :-1 s=READY pgs=2079 cs=0 l=1 rev1=1 crypto rx=0x7f058400c7f0 tx=0x7f058400ccb0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 shutdown_connections 2023-10-14T17:58:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f058c0c9060 0x7f058c1562d0 unknown :-1 s=CLOSED pgs=2079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f058c06fb50 0x7f058c163950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 --2- 172.21.15.29:0/2315378553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f058c06c9e0 0x7f058c0c4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 >> 172.21.15.29:0/2315378553 conn(0x7f058c0ce6a0 msgr2=0x7f058c0cda80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 shutdown_connections 2023-10-14T17:58:07.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:07.991+0000 7f0590ab4640 1 -- 172.21.15.29:0/2315378553 wait complete. 2023-10-14T17:58:08.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.242+0000 7fae1759e640 1 Processor -- start 2023-10-14T17:58:08.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.242+0000 7fae1759e640 1 -- start start 2023-10-14T17:58:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.243+0000 7fae1759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 0x7fae1806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae180ad970 0x7fae1815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae1815f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1806c960 con 0x7fae180ad970 2023-10-14T17:58:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae1815b9d0 con 0x7fae1806c040 2023-10-14T17:58:08.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae1815bcb0 con 0x7fae1815ebc0 2023-10-14T17:58:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae180ad970 0x7fae1815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae180ad970 0x7fae1815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34744/0 (socket says 172.21.15.29:34744) 2023-10-14T17:58:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 -- 172.21.15.29:0/2456060874 learned_addr learned my addr 172.21.15.29:0/2456060874 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 -- 172.21.15.29:0/2456060874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 msgr2=0x7fae1815f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:08.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 --2- 172.21.15.29:0/2456060874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae1815f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae15d9b640 1 -- 172.21.15.29:0/2456060874 >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 msgr2=0x7fae1806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.244+0000 7fae1659c640 1 --2- 172.21.15.29:0/2456060874 >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 0x7fae1806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.245+0000 7fae15d9b640 1 --2- 172.21.15.29:0/2456060874 >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 0x7fae1806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.245+0000 7fae15d9b640 1 -- 172.21.15.29:0/2456060874 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae1815bf90 con 0x7fae180ad970 2023-10-14T17:58:08.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.245+0000 7fae1659c640 1 --2- 172.21.15.29:0/2456060874 >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 0x7fae1806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae15d9b640 1 --2- 172.21.15.29:0/2456060874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae1815e680 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fae180cf350 tx=0x7fae0c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e48d8af9ff0bd6d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae1559a640 1 -- 172.21.15.29:0/2456060874 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c031870 con 0x7fae180ad970 2023-10-14T17:58:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae1559a640 1 -- 172.21.15.29:0/2456060874 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c0319d0 con 0x7fae180ad970 2023-10-14T17:58:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae1559a640 1 -- 172.21.15.29:0/2456060874 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c031d60 con 0x7fae180ad970 2023-10-14T17:58:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae1759e640 1 -- 172.21.15.29:0/2456060874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 msgr2=0x7fae1815e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.246+0000 7fae1759e640 1 --2- 172.21.15.29:0/2456060874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae1815e680 secure :-1 s=READY pgs=2080 cs=0 l=1 rev1=1 crypto rx=0x7fae180cf350 tx=0x7fae0c02b910 comp rx=0 tx=0).stop 2023-10-14T17:58:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 -- 172.21.15.29:0/2456060874 shutdown_connections 2023-10-14T17:58:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 --2- 172.21.15.29:0/2456060874 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae1815f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 --2- 172.21.15.29:0/2456060874 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae1815e680 unknown :-1 s=CLOSED pgs=2080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 --2- 172.21.15.29:0/2456060874 >> v2:172.21.15.186:3300/0 conn(0x7fae1806c040 0x7fae1806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 -- 172.21.15.29:0/2456060874 >> 172.21.15.29:0/2456060874 conn(0x7fae180ce6a0 msgr2=0x7fae180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:08.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 -- 172.21.15.29:0/2456060874 shutdown_connections 2023-10-14T17:58:08.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.247+0000 7fae1759e640 1 -- 172.21.15.29:0/2456060874 wait complete. 2023-10-14T17:58:08.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.248+0000 7fae1759e640 1 Processor -- start 2023-10-14T17:58:08.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 -- start start 2023-10-14T17:58:08.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1806c040 0x7fae1815d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae180c7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae18147200 con 0x7fae180ad970 2023-10-14T17:58:08.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c6c30 con 0x7fae1806c040 2023-10-14T17:58:08.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.249+0000 7fae1759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae180c6f10 con 0x7fae1815ebc0 2023-10-14T17:58:08.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34750/0 (socket says 172.21.15.29:34750) 2023-10-14T17:58:08.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 -- 172.21.15.29:0/14954257 learned_addr learned my addr 172.21.15.29:0/14954257 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:08.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae1659c640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1806c040 0x7fae1815d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 -- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 msgr2=0x7fae180c7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae180c7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 -- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1806c040 msgr2=0x7fae1815d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1806c040 0x7fae1815d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.250+0000 7fae15d9b640 1 -- 172.21.15.29:0/14954257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c0094d0 con 0x7fae180ad970 2023-10-14T17:58:08.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae15d9b640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fae0c000c00 tx=0x7fae0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c031800 con 0x7fae180ad970 2023-10-14T17:58:08.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae0c0443f0 con 0x7fae180ad970 2023-10-14T17:58:08.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae0c044780 con 0x7fae180ad970 2023-10-14T17:58:08.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae180c71f0 con 0x7fae180ad970 2023-10-14T17:58:08.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.251+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae180c77b0 con 0x7fae180ad970 2023-10-14T17:58:08.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.252+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae1806b330 con 0x7fae180ad970 2023-10-14T17:58:08.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.253+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fae0c04d450 con 0x7fae180ad970 2023-10-14T17:58:08.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.254+0000 7fae06ffd640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 0x7fadf806d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.254+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fae0c02d040 con 0x7fae180ad970 2023-10-14T17:58:08.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.254+0000 7fae1659c640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 0x7fadf806d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.255+0000 7fae1659c640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 0x7fadf806d1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae08008850 tx=0x7fae08008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.258+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae0c076d60 con 0x7fae180ad970 2023-10-14T17:58:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.491+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae1804e9b0 con 0x7fae180ad970 2023-10-14T17:58:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.492+0000 7fae06ffd640 1 -- 172.21.15.29:0/14954257 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fae0c035020 con 0x7fae180ad970 2023-10-14T17:58:08.494 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:08.494 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:58:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:58:08.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 msgr2=0x7fadf806d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 0x7fadf806d1b0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fae08008850 tx=0x7fae08008040 comp rx=0 tx=0).stop 2023-10-14T17:58:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 msgr2=0x7fae18146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 secure :-1 s=READY pgs=2081 cs=0 l=1 rev1=1 crypto rx=0x7fae0c000c00 tx=0x7fae0c0046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 shutdown_connections 2023-10-14T17:58:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae1815ebc0 0x7fae180c7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae180ad970 0x7fae18146cc0 unknown :-1 s=CLOSED pgs=2081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fadf806acf0 0x7fadf806d1b0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 --2- 172.21.15.29:0/14954257 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae1806c040 0x7fae1815d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 >> 172.21.15.29:0/14954257 conn(0x7fae180ce6a0 msgr2=0x7fae180ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.495+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 shutdown_connections 2023-10-14T17:58:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.496+0000 7fae1759e640 1 -- 172.21.15.29:0/14954257 wait complete. 2023-10-14T17:58:08.516 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:08.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.735+0000 7f2271613640 1 Processor -- start 2023-10-14T17:58:08.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.736+0000 7f2271613640 1 -- start start 2023-10-14T17:58:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f226c06c040 0x7f226c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c0ad970 0x7f226c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f226c15ebc0 0x7f226c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c06c960 con 0x7f226c15ebc0 2023-10-14T17:58:08.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f226c15b9d0 con 0x7f226c06c040 2023-10-14T17:58:08.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.737+0000 7f2271613640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c15bcb0 con 0x7f226c0ad970 2023-10-14T17:58:08.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f226c15ebc0 0x7f226c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f226c15ebc0 0x7f226c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34754/0 (socket says 172.21.15.29:34754) 2023-10-14T17:58:08.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 -- 172.21.15.29:0/952531936 learned_addr learned my addr 172.21.15.29:0/952531936 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:08.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2263fff640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c0ad970 0x7f226c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f226bfff640 1 --2- 172.21.15.29:0/952531936 >> v2:172.21.15.186:3300/0 conn(0x7f226c06c040 0x7f226c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 -- 172.21.15.29:0/952531936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c0ad970 msgr2=0x7f226c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:08.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c0ad970 0x7f226c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 -- 172.21.15.29:0/952531936 >> v2:172.21.15.186:3300/0 conn(0x7f226c06c040 msgr2=0x7f226c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:08.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 --2- 172.21.15.29:0/952531936 >> v2:172.21.15.186:3300/0 conn(0x7f226c06c040 0x7f226c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.738+0000 7f2270e12640 1 -- 172.21.15.29:0/952531936 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f226c15bf90 con 0x7f226c15ebc0 2023-10-14T17:58:08.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.739+0000 7f2270e12640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c15ebc0 0x7f226c15f050 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f2264009e50 tx=0x7f226402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02058d716d308b6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.741+0000 7f226b7fe640 1 -- 172.21.15.29:0/952531936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f226402ca70 con 0x7f226c15ebc0 2023-10-14T17:58:08.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.741+0000 7f226b7fe640 1 -- 172.21.15.29:0/952531936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226402cbd0 con 0x7f226c15ebc0 2023-10-14T17:58:08.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.741+0000 7f226b7fe640 1 -- 172.21.15.29:0/952531936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22640343f0 con 0x7f226c15ebc0 2023-10-14T17:58:08.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 -- 172.21.15.29:0/952531936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c15ebc0 msgr2=0x7f226c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c15ebc0 0x7f226c15f050 secure :-1 s=READY pgs=2082 cs=0 l=1 rev1=1 crypto rx=0x7f2264009e50 tx=0x7f226402bed0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 -- 172.21.15.29:0/952531936 shutdown_connections 2023-10-14T17:58:08.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c15ebc0 0x7f226c15f050 unknown :-1 s=CLOSED pgs=2082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 --2- 172.21.15.29:0/952531936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c0ad970 0x7f226c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 --2- 172.21.15.29:0/952531936 >> v2:172.21.15.186:3300/0 conn(0x7f226c06c040 0x7f226c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 -- 172.21.15.29:0/952531936 >> 172.21.15.29:0/952531936 conn(0x7f226c0ce6a0 msgr2=0x7f226c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:08.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 -- 172.21.15.29:0/952531936 shutdown_connections 2023-10-14T17:58:08.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.742+0000 7f2271613640 1 -- 172.21.15.29:0/952531936 wait complete. 2023-10-14T17:58:08.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.745+0000 7f2271613640 1 Processor -- start 2023-10-14T17:58:08.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.745+0000 7f2271613640 1 -- start start 2023-10-14T17:58:08.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 0x7f226c155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 0x7f226c1efd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c156130 con 0x7f226c06c040 2023-10-14T17:58:08.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f226c1f02d0 con 0x7f226c0ad970 2023-10-14T17:58:08.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f2271613640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f226c156390 con 0x7f226c15ebc0 2023-10-14T17:58:08.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f226bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f226bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34758/0 (socket says 172.21.15.29:34758) 2023-10-14T17:58:08.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.746+0000 7f226bfff640 1 -- 172.21.15.29:0/1388849516 learned_addr learned my addr 172.21.15.29:0/1388849516 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:08.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f2270e12640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 0x7f226c1efd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f2263fff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 0x7f226c155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 -- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 msgr2=0x7f226c1efd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 0x7f226c1efd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 -- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 msgr2=0x7f226c155bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 0x7f226c155bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 -- 172.21.15.29:0/1388849516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2264009b30 con 0x7f226c06c040 2023-10-14T17:58:08.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f2270e12640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 0x7f226c1efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:08.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f2263fff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 0x7f226c155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:08.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f226bfff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f225c00d800 tx=0x7f225c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c0096a0 con 0x7f226c06c040 2023-10-14T17:58:08.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225c009800 con 0x7f226c06c040 2023-10-14T17:58:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.747+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f225c009990 con 0x7f226c06c040 2023-10-14T17:58:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.748+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f226c156670 con 0x7f226c06c040 2023-10-14T17:58:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.748+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f226c154750 con 0x7f226c06c040 2023-10-14T17:58:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.749+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f226c06b330 con 0x7f226c06c040 2023-10-14T17:58:08.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.750+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f225c009af0 con 0x7f226c06c040 2023-10-14T17:58:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.751+0000 7f22697fa640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 0x7f224c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.751+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f225c089960 con 0x7f226c06c040 2023-10-14T17:58:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.751+0000 7f2263fff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 0x7f224c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:08.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.752+0000 7f2263fff640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 0x7f224c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2254006fd0 tx=0x7f2254008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:08.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.756+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f225c054ae0 con 0x7f226c06c040 2023-10-14T17:58:08.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.959+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f226c04e9b0 con 0x7f226c06c040 2023-10-14T17:58:08.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.962+0000 7f22697fa640 1 -- 172.21.15.29:0/1388849516 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17752 (secure 0 0 0) 0x7f225c00b3a0 con 0x7f226c06c040 2023-10-14T17:58:08.964 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:08.964 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":180,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:07.121223+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/38025471":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/4267645350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:6832/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/1183707990":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3006422350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3667940423":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3415492501":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2674099337":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/2596229485":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/4106038121":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6832/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1011592268":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1904979103":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/84821019":"2023-10-15T17:57:59.333970+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 msgr2=0x7f224c06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 0x7f224c06d1b0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2254006fd0 tx=0x7f2254008040 comp rx=0 tx=0).stop 2023-10-14T17:58:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 msgr2=0x7f226c0c81f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 secure :-1 s=READY pgs=2083 cs=0 l=1 rev1=1 crypto rx=0x7f225c00d800 tx=0x7f225c00dcc0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 shutdown_connections 2023-10-14T17:58:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f226c15ebc0 0x7f226c1efd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f224c06acf0 0x7f224c06d1b0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f226c0ad970 0x7f226c155bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 --2- 172.21.15.29:0/1388849516 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f226c06c040 0x7f226c0c81f0 unknown :-1 s=CLOSED pgs=2083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 >> 172.21.15.29:0/1388849516 conn(0x7f226c0ce6a0 msgr2=0x7f226c15de90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 shutdown_connections 2023-10-14T17:58:08.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:08.965+0000 7f2271613640 1 -- 172.21.15.29:0/1388849516 wait complete. 2023-10-14T17:58:08.985 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T17:58:08.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:09.134 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:58:09.145 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:58:09.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.201+0000 7f6c94a7c640 1 Processor -- start 2023-10-14T17:58:09.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.201+0000 7f6c94a7c640 1 -- start start 2023-10-14T17:58:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c94a7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c900b3070 0x7f6c900cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c94a7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 0x7f6c900cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c94a7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c900b3940 con 0x7f6c900b3450 2023-10-14T17:58:09.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c94a7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c900b45e0 con 0x7f6c900b3070 2023-10-14T17:58:09.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c94a7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c900b3b90 con 0x7f6c900b3e70 2023-10-14T17:58:09.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c8e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6c900b3070 0x7f6c900cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c8ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6c900b3450 0x7f6c900c6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47200/0 (socket says 172.21.15.29:47200) 2023-10-14T17:58:09.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c8ed9d640 1 -- 172.21.15.29:0/3512698465 learned_addr learned my addr 172.21.15.29:0/3512698465 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:09.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.204+0000 7f6c8dd9b640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 0x7f6c900cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8e59c640 1 -- 172.21.15.29:0/3512698465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 msgr2=0x7f6c900cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8e59c640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 0x7f6c900cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8e59c640 1 -- 172.21.15.29:0/3512698465 >> v1:172.21.15.29:6789/0 conn(0x7f6c900b3450 legacy=0x7f6c900c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8d59a640 1 -- 172.21.15.29:0/3512698465 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3809000370 0 0) 0x7f6c900b3940 con 0x7f6c900b3450 2023-10-14T17:58:09.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8e59c640 1 -- 172.21.15.29:0/3512698465 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c900d3840 con 0x7f6c900b3070 2023-10-14T17:58:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.205+0000 7f6c8dd9b640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 0x7f6c900cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.206+0000 7f6c8e59c640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3070 0x7f6c900cf500 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009790 tx=0x7f6c8002b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9a43452b0416b33 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.206+0000 7f6c8d59a640 1 -- 172.21.15.29:0/3512698465 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80031800 con 0x7f6c900b3070 2023-10-14T17:58:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.206+0000 7f6c8d59a640 1 -- 172.21.15.29:0/3512698465 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c80031960 con 0x7f6c900b3070 2023-10-14T17:58:09.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.206+0000 7f6c8d59a640 1 -- 172.21.15.29:0/3512698465 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c80031cf0 con 0x7f6c900b3070 2023-10-14T17:58:09.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 -- 172.21.15.29:0/3512698465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3070 msgr2=0x7f6c900cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3070 0x7f6c900cf500 secure :-1 s=READY pgs=2085 cs=0 l=1 rev1=1 crypto rx=0x7f6c80009790 tx=0x7f6c8002b8a0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 -- 172.21.15.29:0/3512698465 shutdown_connections 2023-10-14T17:58:09.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3e70 0x7f6c900cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/3512698465 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3070 0x7f6c900cf500 unknown :-1 s=CLOSED pgs=2085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 -- 172.21.15.29:0/3512698465 >> 172.21.15.29:0/3512698465 conn(0x7f6c900d2d20 msgr2=0x7f6c900d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:09.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 -- 172.21.15.29:0/3512698465 shutdown_connections 2023-10-14T17:58:09.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.207+0000 7f6c94a7c640 1 -- 172.21.15.29:0/3512698465 wait complete. 2023-10-14T17:58:09.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.209+0000 7f6c94a7c640 1 Processor -- start 2023-10-14T17:58:09.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.210+0000 7f6c94a7c640 1 -- start start 2023-10-14T17:58:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3070 0x7f6c9015c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c900b3e70 0x7f6c901f27a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9015c990 con 0x7f6c900b3450 2023-10-14T17:58:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9006bb10 con 0x7f6c900b3e70 2023-10-14T17:58:09.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c94a7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c9006bdf0 con 0x7f6c900b3070 2023-10-14T17:58:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c8dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34782/0 (socket says 172.21.15.29:34782) 2023-10-14T17:58:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c8dd9b640 1 -- 172.21.15.29:0/1904155972 learned_addr learned my addr 172.21.15.29:0/1904155972 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c8e59c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3070 0x7f6c9015c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.211+0000 7f6c8ed9d640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c900b3e70 0x7f6c901f27a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 -- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3070 msgr2=0x7f6c9015c450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3070 0x7f6c9015c450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 -- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c900b3e70 msgr2=0x7f6c901f27a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c900b3e70 0x7f6c901f27a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 -- 172.21.15.29:0/1904155972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c800093a0 con 0x7f6c900b3450 2023-10-14T17:58:09.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.212+0000 7f6c8dd9b640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d7d0 tx=0x7f6c8400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.213+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84004030 con 0x7f6c900b3450 2023-10-14T17:58:09.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.213+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c84004190 con 0x7f6c900b3450 2023-10-14T17:58:09.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.213+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c84004520 con 0x7f6c900b3450 2023-10-14T17:58:09.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.213+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c9006c0d0 con 0x7f6c900b3450 2023-10-14T17:58:09.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.213+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c901f1ed0 con 0x7f6c900b3450 2023-10-14T17:58:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.215+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c9004e160 con 0x7f6c900b3450 2023-10-14T17:58:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.216+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f6c840096a0 con 0x7f6c900b3450 2023-10-14T17:58:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.217+0000 7f6c7effd640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 0x7f6c7006d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.217+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f6c84089640 con 0x7f6c900b3450 2023-10-14T17:58:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.217+0000 7f6c8e59c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 0x7f6c7006d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.218+0000 7f6c8e59c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 0x7f6c7006d250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c80036000 tx=0x7f6c8003c040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.222+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6c84054760 con 0x7f6c900b3450 2023-10-14T17:58:09.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.457+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c9006a7d0 con 0x7f6c900b3450 2023-10-14T17:58:09.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.459+0000 7f6c7effd640 1 -- 172.21.15.29:0/1904155972 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f6c84057d80 con 0x7f6c900b3450 2023-10-14T17:58:09.460 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:09.460 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:58:09.461 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:58:09.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 msgr2=0x7f6c7006d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 0x7f6c7006d250 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6c80036000 tx=0x7f6c8003c040 comp rx=0 tx=0).stop 2023-10-14T17:58:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 msgr2=0x7f6c900b1330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 secure :-1 s=READY pgs=2087 cs=0 l=1 rev1=1 crypto rx=0x7f6c8400d7d0 tx=0x7f6c8400dc90 comp rx=0 tx=0).stop 2023-10-14T17:58:09.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 shutdown_connections 2023-10-14T17:58:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f6c7006ad90 0x7f6c7006d250 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6c900b3e70 0x7f6c901f27a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6c900b3450 0x7f6c900b1330 unknown :-1 s=CLOSED pgs=2087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 --2- 172.21.15.29:0/1904155972 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6c900b3070 0x7f6c9015c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 >> 172.21.15.29:0/1904155972 conn(0x7f6c900d2d20 msgr2=0x7f6c90160760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.461+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 shutdown_connections 2023-10-14T17:58:09.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.462+0000 7f6c94a7c640 1 -- 172.21.15.29:0/1904155972 wait complete. 2023-10-14T17:58:09.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:09.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.709+0000 7f918ae2e640 1 Processor -- start 2023-10-14T17:58:09.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.710+0000 7f918ae2e640 1 -- start start 2023-10-14T17:58:09.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.711+0000 7f918ae2e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f918415cfc0 0x7f91840b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.711+0000 7f918ae2e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91840d15a0 con 0x7f91840d2640 2023-10-14T17:58:09.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.711+0000 7f918ae2e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91840d1880 con 0x7f918415cbe0 2023-10-14T17:58:09.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.711+0000 7f918ae2e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91840d1b60 con 0x7f918415cfc0 2023-10-14T17:58:09.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.711+0000 7f918962b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f918415cfc0 0x7f91840b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f918962b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f918415cfc0 0x7f91840b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34800/0 (socket says 172.21.15.29:34800) 2023-10-14T17:58:09.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f918962b640 1 -- 172.21.15.29:0/3894176111 learned_addr learned my addr 172.21.15.29:0/3894176111 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:09.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f918962b640 1 -- 172.21.15.29:0/3894176111 >> v1:172.21.15.186:6789/0 conn(0x7f918415cbe0 legacy=0x7f91840d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:09.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f918962b640 1 -- 172.21.15.29:0/3894176111 >> v1:172.21.15.29:6789/0 conn(0x7f91840d2640 legacy=0x7f91840b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f9188e2a640 1 -- 172.21.15.29:0/3894176111 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275833776 0 0) 0x7f91840d15a0 con 0x7f91840d2640 2023-10-14T17:58:09.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.712+0000 7f918962b640 1 -- 172.21.15.29:0/3894176111 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91840d1e40 con 0x7f918415cfc0 2023-10-14T17:58:09.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.713+0000 7f918962b640 1 --2- 172.21.15.29:0/3894176111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918415cfc0 0x7f91840b4570 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f9174009870 tx=0x7f917402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d76f3cee33acdf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f9188e2a640 1 -- 172.21.15.29:0/3894176111 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917402f840 con 0x7f918415cfc0 2023-10-14T17:58:09.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f9188e2a640 1 -- 172.21.15.29:0/3894176111 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f917402f9a0 con 0x7f918415cfc0 2023-10-14T17:58:09.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f9188e2a640 1 -- 172.21.15.29:0/3894176111 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f917402fd30 con 0x7f918415cfc0 2023-10-14T17:58:09.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 -- 172.21.15.29:0/3894176111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918415cfc0 msgr2=0x7f91840b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 --2- 172.21.15.29:0/3894176111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918415cfc0 0x7f91840b4570 secure :-1 s=READY pgs=2090 cs=0 l=1 rev1=1 crypto rx=0x7f9174009870 tx=0x7f917402ba20 comp rx=0 tx=0).stop 2023-10-14T17:58:09.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 -- 172.21.15.29:0/3894176111 shutdown_connections 2023-10-14T17:58:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 --2- 172.21.15.29:0/3894176111 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f918415cfc0 0x7f91840b4570 unknown :-1 s=CLOSED pgs=2090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 -- 172.21.15.29:0/3894176111 >> 172.21.15.29:0/3894176111 conn(0x7f91840cf570 msgr2=0x7f91840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 -- 172.21.15.29:0/3894176111 shutdown_connections 2023-10-14T17:58:09.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.714+0000 7f918ae2e640 1 -- 172.21.15.29:0/3894176111 wait complete. 2023-10-14T17:58:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.716+0000 7f918ae2e640 1 Processor -- start 2023-10-14T17:58:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.716+0000 7f918ae2e640 1 -- start start 2023-10-14T17:58:09.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f918415cbe0 0x7f91841587f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f918415cfc0 0x7f9184156770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91840c8d20 con 0x7f91840d2640 2023-10-14T17:58:09.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9184159b50 con 0x7f918415cbe0 2023-10-14T17:58:09.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.717+0000 7f918ae2e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91840b39e0 con 0x7f918415cfc0 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34816/0 (socket says 172.21.15.29:34816) 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 -- 172.21.15.29:0/2129259885 learned_addr learned my addr 172.21.15.29:0/2129259885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 -- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f918415cfc0 msgr2=0x7f9184156770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f918415cfc0 0x7f9184156770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 -- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f918415cbe0 msgr2=0x7f91841587f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f918415cbe0 0x7f91841587f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 -- 172.21.15.29:0/2129259885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91740094d0 con 0x7f91840d2640 2023-10-14T17:58:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.718+0000 7f9189e2c640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f918000c460 tx=0x7f918000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.719+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180012830 con 0x7f91840d2640 2023-10-14T17:58:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.719+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91800129d0 con 0x7f91840d2640 2023-10-14T17:58:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.719+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9180012d80 con 0x7f91840d2640 2023-10-14T17:58:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.719+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91841592b0 con 0x7f91840d2640 2023-10-14T17:58:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.719+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9184159870 con 0x7f91840d2640 2023-10-14T17:58:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.720+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91840b3840 con 0x7f91840d2640 2023-10-14T17:58:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.721+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 101) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f918001b400 con 0x7f91840d2640 2023-10-14T17:58:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.722+0000 7f917a7fc640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 0x7f916806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.722+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f91800895a0 con 0x7f91840d2640 2023-10-14T17:58:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.722+0000 7f918962b640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 0x7f916806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.723+0000 7f918962b640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 0x7f916806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9174036000 tx=0x7f9174039040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:09.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.728+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91840b3840 con 0x7f91840d2640 2023-10-14T17:58:09.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.759+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 102) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f9180054960 con 0x7f91840d2640 2023-10-14T17:58:09.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.969+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91840b3840 con 0x7f91840d2640 2023-10-14T17:58:09.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.970+0000 7f917a7fc640 1 -- 172.21.15.29:0/2129259885 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f91840b3840 con 0x7f91840d2640 2023-10-14T17:58:09.972 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:09.972 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":83,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:57:44.368179+0000","modified":"2023-10-14T17:57:47.451014+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":9615},"failed":[],"damaged":[],"stopped":[],"info":{"gid_9615":{"gid":9615,"name":"c","rank":0,"incarnation":79,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6836/494120094","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":494120094},{"type":"v1","addr":"172.21.15.29:6836","nonce":494120094}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[21],"metadata_pool":20,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":10}]} 2023-10-14T17:58:09.972 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 83 2023-10-14T17:58:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 msgr2=0x7f916806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 0x7f916806d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9174036000 tx=0x7f9174039040 comp rx=0 tx=0).stop 2023-10-14T17:58:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 msgr2=0x7f91840c87e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 secure :-1 s=READY pgs=2091 cs=0 l=1 rev1=1 crypto rx=0x7f918000c460 tx=0x7f918000c920 comp rx=0 tx=0).stop 2023-10-14T17:58:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 shutdown_connections 2023-10-14T17:58:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f918415cfc0 0x7f9184156770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f916806acc0 0x7f916806d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f918415cbe0 0x7f91841587f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 --2- 172.21.15.29:0/2129259885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91840d2640 0x7f91840c87e0 unknown :-1 s=CLOSED pgs=2091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 >> 172.21.15.29:0/2129259885 conn(0x7f91840cf570 msgr2=0x7f91840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 shutdown_connections 2023-10-14T17:58:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:09.973+0000 7f918ae2e640 1 -- 172.21.15.29:0/2129259885 wait complete. 2023-10-14T17:58:09.995 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:10.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.217+0000 7f277d20e640 1 Processor -- start 2023-10-14T17:58:10.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.217+0000 7f277d20e640 1 -- start start 2023-10-14T17:58:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277d20e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27780b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277d20e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2778071450 con 0x7f27780adf30 2023-10-14T17:58:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277d20e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27780cc600 con 0x7f27780adb50 2023-10-14T17:58:10.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277d20e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27780cc8e0 con 0x7f2778070fd0 2023-10-14T17:58:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277759e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f27780adf30 0x7f2778164f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:47216/0 (socket says 172.21.15.29:47216) 2023-10-14T17:58:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.219+0000 7f277759e640 1 -- 172.21.15.29:0/2950639518 learned_addr learned my addr 172.21.15.29:0/2950639518 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f277659c640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27780b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2652148964 0 0) 0x7f2778071450 con 0x7f27780adf30 2023-10-14T17:58:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2760003520 con 0x7f27780adf30 2023-10-14T17:58:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052749196 0 0) 0x7f2760003520 con 0x7f27780adf30 2023-10-14T17:58:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 msgr2=0x7f27780b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27780b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 >> v1:172.21.15.186:6789/0 conn(0x7f27780adb50 legacy=0x7f277806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780ccbc0 con 0x7f27780adf30 2023-10-14T17:58:10.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f27700019f0 con 0x7f27780adf30 2023-10-14T17:58:10.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 >> v1:172.21.15.29:6789/0 conn(0x7f27780adf30 legacy=0x7f2778164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27600039a0 0x7f2760003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760004d00 0x7f2760005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27780cc8e0 con 0x7f2760004d00 2023-10-14T17:58:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f276000ba30 con 0x7f27600039a0 2023-10-14T17:58:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.220+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f276000bbf0 con 0x7f2778070fd0 2023-10-14T17:58:10.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760004d00 0x7f2760005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277659c640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f2776d9d640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27600039a0 0x7f2760003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 -- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 msgr2=0x7f27600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2778070fd0 0x7f27600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 -- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27600039a0 msgr2=0x7f2760003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27600039a0 0x7f2760003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f277759e640 1 -- 172.21.15.29:0/2950639518 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2760003520 con 0x7f2760004d00 2023-10-14T17:58:10.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.221+0000 7f2776d9d640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27600039a0 0x7f2760003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.222+0000 7f277759e640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760004d00 0x7f2760005180 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f277000a800 tx=0x7f27600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=123c2c9d103990bd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700072d0 con 0x7f2760004d00 2023-10-14T17:58:10.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f277002f070 con 0x7f2760004d00 2023-10-14T17:58:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f2775d9b640 1 -- 172.21.15.29:0/2950639518 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27700076b0 con 0x7f2760004d00 2023-10-14T17:58:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 -- 172.21.15.29:0/2950639518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760004d00 msgr2=0x7f2760005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 --2- 172.21.15.29:0/2950639518 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2760004d00 0x7f2760005180 secure :-1 s=READY pgs=2093 cs=0 l=1 rev1=1 crypto rx=0x7f277000a800 tx=0x7f27600057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277759e640 1 -- 172.21.15.29:0/2950639518 reap_dead start 2023-10-14T17:58:10.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 -- 172.21.15.29:0/2950639518 shutdown_connections 2023-10-14T17:58:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 -- 172.21.15.29:0/2950639518 >> 172.21.15.29:0/2950639518 conn(0x7f27780cb410 msgr2=0x7f27780cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 -- 172.21.15.29:0/2950639518 shutdown_connections 2023-10-14T17:58:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.223+0000 7f277d20e640 1 -- 172.21.15.29:0/2950639518 wait complete. 2023-10-14T17:58:10.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.225+0000 7f277d20e640 1 Processor -- start 2023-10-14T17:58:10.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.225+0000 7f277d20e640 1 -- start start 2023-10-14T17:58:10.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 0x7f27780c4200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27780b35b0 0x7f277806cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27780b2510 con 0x7f27780b1fc0 2023-10-14T17:58:10.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27780b27f0 con 0x7f2760004d00 2023-10-14T17:58:10.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.226+0000 7f277d20e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27780b2ad0 con 0x7f27780b35b0 2023-10-14T17:58:10.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34832/0 (socket says 172.21.15.29:34832) 2023-10-14T17:58:10.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 -- 172.21.15.29:0/886849710 learned_addr learned my addr 172.21.15.29:0/886849710 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:10.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 -- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27780b35b0 msgr2=0x7f277806cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27780b35b0 0x7f277806cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 -- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 msgr2=0x7f27780c4200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f2776d9d640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 0x7f27780c4200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 0x7f27780c4200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 -- 172.21.15.29:0/886849710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27780b2db0 con 0x7f27780b1fc0 2023-10-14T17:58:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f2776d9d640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 0x7f27780c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:10.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.227+0000 7f277659c640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f276c011580 tx=0x7f276c011a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.228+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c015870 con 0x7f27780b1fc0 2023-10-14T17:58:10.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.228+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f276c0159d0 con 0x7f27780b1fc0 2023-10-14T17:58:10.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.228+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f276c015d40 con 0x7f27780b1fc0 2023-10-14T17:58:10.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.228+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27780b3090 con 0x7f27780b1fc0 2023-10-14T17:58:10.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.228+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2778155130 con 0x7f27780b1fc0 2023-10-14T17:58:10.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.229+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f277804e160 con 0x7f27780b1fc0 2023-10-14T17:58:10.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.230+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 102) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f276c00f460 con 0x7f27780b1fc0 2023-10-14T17:58:10.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.231+0000 7f275f7fe640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 0x7f276406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.231+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f276c089de0 con 0x7f27780b1fc0 2023-10-14T17:58:10.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.231+0000 7f2776d9d640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 0x7f276406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.232+0000 7f2776d9d640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 0x7f276406d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27680016f0 tx=0x7f2768009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.236+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f276c054af0 con 0x7f27780b1fc0 2023-10-14T17:58:10.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.436+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27780b4910 con 0x7f27780b1fc0 2023-10-14T17:58:10.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.438+0000 7f275f7fe640 1 -- 172.21.15.29:0/886849710 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v180) v1 ==== 74+0+17752 (secure 0 0 0) 0x7f276c058110 con 0x7f27780b1fc0 2023-10-14T17:58:10.440 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:10.440 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":180,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:07.121223+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":20,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:57:42.346345+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"177","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":21,"pool_name":"cephfs_data","create_time":"2023-10-14T17:57:43.390271+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"178","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/38025471":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/4267645350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:6832/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/1183707990":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3006422350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3667940423":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3415492501":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2674099337":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/2596229485":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/4106038121":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6832/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1011592268":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1904979103":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/84821019":"2023-10-15T17:57:59.333970+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:10.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 msgr2=0x7f276406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 0x7f276406d3f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f27680016f0 tx=0x7f2768009040 comp rx=0 tx=0).stop 2023-10-14T17:58:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 msgr2=0x7f2778154bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 secure :-1 s=READY pgs=2094 cs=0 l=1 rev1=1 crypto rx=0x7f276c011580 tx=0x7f276c011a40 comp rx=0 tx=0).stop 2023-10-14T17:58:10.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 shutdown_connections 2023-10-14T17:58:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27780b35b0 0x7f277806cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27780b1fc0 0x7f2778154bf0 unknown :-1 s=CLOSED pgs=2094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f276406af30 0x7f276406d3f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 --2- 172.21.15.29:0/886849710 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2760004d00 0x7f27780c4200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.441+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 >> 172.21.15.29:0/886849710 conn(0x7f27780cb410 msgr2=0x7f27780c9cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:10.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.442+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 shutdown_connections 2023-10-14T17:58:10.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.442+0000 7f277d20e640 1 -- 172.21.15.29:0/886849710 wait complete. 2023-10-14T17:58:10.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T17:58:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.665+0000 7f0b4ffff640 1 Processor -- start 2023-10-14T17:58:10.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.665+0000 7f0b4ffff640 1 -- start start 2023-10-14T17:58:10.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b5006d220 0x7f0b500c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 0x7f0b500cdb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b500acd70 con 0x7f0b5006d600 2023-10-14T17:58:10.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4ffff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b500ac4a0 con 0x7f0b5006d220 2023-10-14T17:58:10.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b50065f40 con 0x7f0b50065ab0 2023-10-14T17:58:10.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.666+0000 7f0b4effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b5006d220 0x7f0b500c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b5006d220 0x7f0b500c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34840/0 (socket says 172.21.15.29:34840) 2023-10-14T17:58:10.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 -- 172.21.15.29:0/681576573 learned_addr learned my addr 172.21.15.29:0/681576573 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:10.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4e7fc640 1 --2- 172.21.15.29:0/681576573 >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 0x7f0b500cdb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 -- 172.21.15.29:0/681576573 >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 msgr2=0x7f0b500cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 --2- 172.21.15.29:0/681576573 >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 0x7f0b500cdb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 -- 172.21.15.29:0/681576573 >> v1:172.21.15.186:6789/0 conn(0x7f0b5006d600 legacy=0x7f0b500ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:10.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4effd640 1 -- 172.21.15.29:0/681576573 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b500ac720 con 0x7f0b5006d220 2023-10-14T17:58:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.667+0000 7f0b4e7fc640 1 --2- 172.21.15.29:0/681576573 >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 0x7f0b500cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.668+0000 7f0b4effd640 1 --2- 172.21.15.29:0/681576573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c95e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0b500cb8f0 tx=0x7f0b4002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663f5f5dfc205c4c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.668+0000 7f0b4dffb640 1 -- 172.21.15.29:0/681576573 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4002ca80 con 0x7f0b5006d220 2023-10-14T17:58:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4dffb640 1 -- 172.21.15.29:0/681576573 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4002cbe0 con 0x7f0b5006d220 2023-10-14T17:58:10.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4dffb640 1 -- 172.21.15.29:0/681576573 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b400343f0 con 0x7f0b5006d220 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 -- 172.21.15.29:0/681576573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 msgr2=0x7f0b500c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/681576573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c95e0 secure :-1 s=READY pgs=2095 cs=0 l=1 rev1=1 crypto rx=0x7f0b500cb8f0 tx=0x7f0b4002bee0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 -- 172.21.15.29:0/681576573 shutdown_connections 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/681576573 >> v2:172.21.15.186:3300/0 conn(0x7f0b50065ab0 0x7f0b500cdb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/681576573 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c95e0 unknown :-1 s=CLOSED pgs=2095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 -- 172.21.15.29:0/681576573 >> 172.21.15.29:0/681576573 conn(0x7f0b500b0a80 msgr2=0x7f0b500b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:10.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 -- 172.21.15.29:0/681576573 shutdown_connections 2023-10-14T17:58:10.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.669+0000 7f0b4ffff640 1 -- 172.21.15.29:0/681576573 wait complete. 2023-10-14T17:58:10.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.671+0000 7f0b4ffff640 1 Processor -- start 2023-10-14T17:58:10.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.671+0000 7f0b4ffff640 1 -- start start 2023-10-14T17:58:10.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5006d600 0x7f0b500c8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b5015eb40 0x7f0b501f2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b500c6dc0 con 0x7f0b5006d220 2023-10-14T17:58:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b501f33e0 con 0x7f0b5006d600 2023-10-14T17:58:10.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b500c9080 con 0x7f0b5015eb40 2023-10-14T17:58:10.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.672+0000 7f0b4effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34844/0 (socket says 172.21.15.29:34844) 2023-10-14T17:58:10.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 -- 172.21.15.29:0/615568166 learned_addr learned my addr 172.21.15.29:0/615568166 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:10.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 -- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b5015eb40 msgr2=0x7f0b501f2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b5015eb40 0x7f0b501f2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 -- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5006d600 msgr2=0x7f0b500c8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5006d600 0x7f0b500c8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 -- 172.21.15.29:0/615568166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b40009b30 con 0x7f0b5006d220 2023-10-14T17:58:10.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.673+0000 7f0b4effd640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0b40000f80 tx=0x7f0b40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.674+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4003e450 con 0x7f0b5006d220 2023-10-14T17:58:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.674+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4003e5f0 con 0x7f0b5006d220 2023-10-14T17:58:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.674+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4003e9a0 con 0x7f0b5006d220 2023-10-14T17:58:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.674+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b500c6fd0 con 0x7f0b5006d220 2023-10-14T17:58:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.674+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b500c7270 con 0x7f0b5006d220 2023-10-14T17:58:10.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.675+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5004e9b0 con 0x7f0b5006d220 2023-10-14T17:58:10.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.676+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 102) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f0b4002cd50 con 0x7f0b5006d220 2023-10-14T17:58:10.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.677+0000 7f0b477fe640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 0x7f0b3006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:10.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.677+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(180..180 src has 1..180) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f0b400ac440 con 0x7f0b5006d220 2023-10-14T17:58:10.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.677+0000 7f0b4e7fc640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 0x7f0b3006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:10.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.678+0000 7f0b4e7fc640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 0x7f0b3006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b38006fd0 tx=0x7f0b38008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:10.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.682+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b400771f0 con 0x7f0b5006d220 2023-10-14T17:58:10.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.914+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f0b500b4f10 con 0x7f0b5006d220 2023-10-14T17:58:10.916 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T17:58:10.915+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T17:58:10.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.918+0000 7f0b477fe640 1 -- 172.21.15.29:0/615568166 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f0b40033e80 con 0x7f0b5006d220 2023-10-14T17:58:10.920 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T17:58:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.921+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 msgr2=0x7f0b3006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.921+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 0x7f0b3006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0b38006fd0 tx=0x7f0b38008040 comp rx=0 tx=0).stop 2023-10-14T17:58:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.921+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 msgr2=0x7f0b500c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:10.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.921+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 secure :-1 s=READY pgs=2096 cs=0 l=1 rev1=1 crypto rx=0x7f0b40000f80 tx=0x7f0b40004a40 comp rx=0 tx=0).stop 2023-10-14T17:58:10.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 shutdown_connections 2023-10-14T17:58:10.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b5015eb40 0x7f0b501f2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0b3006af30 0x7f0b3006d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b5006d600 0x7f0b500c8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 --2- 172.21.15.29:0/615568166 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b5006d220 0x7f0b500c7d50 unknown :-1 s=CLOSED pgs=2096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:10.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 >> 172.21.15.29:0/615568166 conn(0x7f0b500b0a80 msgr2=0x7f0b500ae860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:10.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 shutdown_connections 2023-10-14T17:58:10.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:10.922+0000 7f0b4ffff640 1 -- 172.21.15.29:0/615568166 wait complete. 2023-10-14T17:58:10.930 INFO:tasks.ceph.mds.c.smithi029.stderr: -3810> 2023-10-14T17:57:49.849+0000 7f6fd4feb640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:58:10.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T17:58:10.998 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:58:11.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.142+0000 7feac2f78640 1 Processor -- start 2023-10-14T17:58:11.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.142+0000 7feac2f78640 1 -- start start 2023-10-14T17:58:11.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.143+0000 7feac2f78640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feabc0c9860 0x7feabc1643b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:11.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.143+0000 7feac2f78640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0cc6b0 con 0x7feabc1428c0 2023-10-14T17:58:11.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac2f78640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc1599b0 con 0x7feabc06b660 2023-10-14T17:58:11.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac2f78640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc06c290 con 0x7feabc0c9860 2023-10-14T17:58:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac1515640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feabc0c9860 0x7feabc1643b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feabbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7feabc1428c0 0x7feabc0c9150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33476/0 (socket says 172.21.15.29:33476) 2023-10-14T17:58:11.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feabbfff640 1 -- 172.21.15.29:0/2479064533 learned_addr learned my addr 172.21.15.29:0/2479064533 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:11.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac1515640 1 -- 172.21.15.29:0/2479064533 >> v1:172.21.15.186:6789/0 conn(0x7feabc06b660 legacy=0x7feabc06ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac1515640 1 -- 172.21.15.29:0/2479064533 >> v1:172.21.15.29:6789/0 conn(0x7feabc1428c0 legacy=0x7feabc0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.144+0000 7feac1515640 1 -- 172.21.15.29:0/2479064533 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc06c570 con 0x7feabc0c9860 2023-10-14T17:58:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.145+0000 7feac1515640 1 --2- 172.21.15.29:0/2479064533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc1643b0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7feab4009d90 tx=0x7feab402e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f031c74fe76555a4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feabb7fe640 1 -- 172.21.15.29:0/2479064533 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab402ee70 con 0x7feabc0c9860 2023-10-14T17:58:11.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feabb7fe640 1 -- 172.21.15.29:0/2479064533 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab4036440 con 0x7feabc0c9860 2023-10-14T17:58:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feabb7fe640 1 -- 172.21.15.29:0/2479064533 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feab40367d0 con 0x7feabc0c9860 2023-10-14T17:58:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feac2f78640 1 -- 172.21.15.29:0/2479064533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 msgr2=0x7feabc1643b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feac2f78640 1 --2- 172.21.15.29:0/2479064533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc1643b0 secure :-1 s=READY pgs=2101 cs=0 l=1 rev1=1 crypto rx=0x7feab4009d90 tx=0x7feab402e0c0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feac2f78640 1 -- 172.21.15.29:0/2479064533 shutdown_connections 2023-10-14T17:58:11.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feac2f78640 1 --2- 172.21.15.29:0/2479064533 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc1643b0 unknown :-1 s=CLOSED pgs=2101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.146+0000 7feac2f78640 1 -- 172.21.15.29:0/2479064533 >> 172.21.15.29:0/2479064533 conn(0x7feabc0ce9b0 msgr2=0x7feabc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.147+0000 7feac2f78640 1 -- 172.21.15.29:0/2479064533 shutdown_connections 2023-10-14T17:58:11.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.147+0000 7feac2f78640 1 -- 172.21.15.29:0/2479064533 wait complete. 2023-10-14T17:58:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.148+0000 7feac2f78640 1 Processor -- start 2023-10-14T17:58:11.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.148+0000 7feac2f78640 1 -- start start 2023-10-14T17:58:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feabc06b660 0x7feabc0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feabc1428c0 0x7feabc070b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c8a00 con 0x7feabc0c9860 2023-10-14T17:58:11.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c7df0 con 0x7feabc06b660 2023-10-14T17:58:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.149+0000 7feac2f78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc0c80d0 con 0x7feabc1428c0 2023-10-14T17:58:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45852/0 (socket says 172.21.15.29:45852) 2023-10-14T17:58:11.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 -- 172.21.15.29:0/3041472492 learned_addr learned my addr 172.21.15.29:0/3041472492 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feac0d14640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feabc06b660 0x7feabc0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feac1515640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feabc1428c0 0x7feabc070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 -- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feabc1428c0 msgr2=0x7feabc070b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feabc1428c0 0x7feabc070b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 -- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feabc06b660 msgr2=0x7feabc0b3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feabc06b660 0x7feabc0b3cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.150+0000 7feabbfff640 1 -- 172.21.15.29:0/3041472492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab4009a70 con 0x7feabc0c9860 2023-10-14T17:58:11.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feabbfff640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c480 tx=0x7feaac00c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:11.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac005ce0 con 0x7feabc0c9860 2023-10-14T17:58:11.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac005e40 con 0x7feabc0c9860 2023-10-14T17:58:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc0c83b0 con 0x7feabc0c9860 2023-10-14T17:58:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feaac0072f0 con 0x7feabc0c9860 2023-10-14T17:58:11.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.151+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc153a40 con 0x7feabc0c9860 2023-10-14T17:58:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.152+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc0c8bb0 con 0x7feabc0c9860 2023-10-14T17:58:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.153+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 102) v1 ==== 89275+0+0 (secure 0 0 0) 0x7feaac007450 con 0x7feabc0c9860 2023-10-14T17:58:11.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.154+0000 7feab97fa640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 0x7feaa806d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.154+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6599+0+0 (secure 0 0 0) 0x7feaac088eb0 con 0x7feabc0c9860 2023-10-14T17:58:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.155+0000 7feac0d14640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 0x7feaa806d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.155+0000 7feac0d14640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 0x7feaa806d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feab0004ea0 tx=0x7feab0009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:11.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.159+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feabc0c8bb0 con 0x7feabc0c9860 2023-10-14T17:58:11.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.399+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7feabc0c8bb0 con 0x7feabc0c9860 2023-10-14T17:58:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.922+0000 7feab97fa640 1 -- 172.21.15.29:0/3041472492 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7feabc0c8bb0 con 0x7feabc0c9860 2023-10-14T17:58:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 msgr2=0x7feaa806d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 0x7feaa806d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7feab0004ea0 tx=0x7feab0009040 comp rx=0 tx=0).stop 2023-10-14T17:58:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 msgr2=0x7feabc0705c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 secure :-1 s=READY pgs=2102 cs=0 l=1 rev1=1 crypto rx=0x7feaac00c480 tx=0x7feaac00c940 comp rx=0 tx=0).stop 2023-10-14T17:58:11.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 shutdown_connections 2023-10-14T17:58:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feabc1428c0 0x7feabc070b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feabc0c9860 0x7feabc0705c0 unknown :-1 s=CLOSED pgs=2102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7feaa806af30 0x7feaa806d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 --2- 172.21.15.29:0/3041472492 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feabc06b660 0x7feabc0b3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:11.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.925+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 >> 172.21.15.29:0/3041472492 conn(0x7feabc0ce9b0 msgr2=0x7feabc158a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.926+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 shutdown_connections 2023-10-14T17:58:11.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:11.926+0000 7feac2f78640 1 -- 172.21.15.29:0/3041472492 wait complete. 2023-10-14T17:58:11.952 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T17:58:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.173+0000 7f9743fff640 1 Processor -- start 2023-10-14T17:58:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.173+0000 7f9743fff640 1 -- start start 2023-10-14T17:58:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f9743fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97440b8480 0x7f97440cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f9743fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97440d0be0 0x7f97440cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f9743fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f974415d150 con 0x7f974415cd70 2023-10-14T17:58:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f9743fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97440d1180 con 0x7f97440b8480 2023-10-14T17:58:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f9743fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97440d1460 con 0x7f97440d0be0 2023-10-14T17:58:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f973bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97440b8480 0x7f97440cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f973bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f97440b8480 0x7f97440cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:55364/0 (socket says 172.21.15.29:55364) 2023-10-14T17:58:12.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f973bfff640 1 -- 172.21.15.29:0/3676191505 learned_addr learned my addr 172.21.15.29:0/3676191505 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:12.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.175+0000 7f97437fe640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97440d0be0 0x7f97440cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.176+0000 7f973bfff640 1 -- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97440d0be0 msgr2=0x7f97440cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.176+0000 7f973bfff640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97440d0be0 0x7f97440cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.176+0000 7f973bfff640 1 -- 172.21.15.29:0/3676191505 >> v1:172.21.15.186:6789/0 conn(0x7f974415cd70 legacy=0x7f97440d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.176+0000 7f973bfff640 1 -- 172.21.15.29:0/3676191505 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97440d1740 con 0x7f97440b8480 2023-10-14T17:58:12.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.178+0000 7f973bfff640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97440cd7c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f972c009870 tx=0x7f972c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c88dc582855226 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:12.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.178+0000 7f97427fc640 1 -- 172.21.15.29:0/3676191505 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c030840 con 0x7f97440b8480 2023-10-14T17:58:12.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.178+0000 7f97427fc640 1 -- 172.21.15.29:0/3676191505 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c0309a0 con 0x7f97440b8480 2023-10-14T17:58:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.178+0000 7f97427fc640 1 -- 172.21.15.29:0/3676191505 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c030d30 con 0x7f97440b8480 2023-10-14T17:58:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 -- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 msgr2=0x7f97440cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97440cd7c0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7f972c009870 tx=0x7f972c02b980 comp rx=0 tx=0).stop 2023-10-14T17:58:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 -- 172.21.15.29:0/3676191505 shutdown_connections 2023-10-14T17:58:12.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f97440d0be0 0x7f97440cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 --2- 172.21.15.29:0/3676191505 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97440cd7c0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 -- 172.21.15.29:0/3676191505 >> 172.21.15.29:0/3676191505 conn(0x7f97440cf570 msgr2=0x7f97440cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:12.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 -- 172.21.15.29:0/3676191505 shutdown_connections 2023-10-14T17:58:12.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.179+0000 7f9743fff640 1 -- 172.21.15.29:0/3676191505 wait complete. 2023-10-14T17:58:12.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.182+0000 7f9743fff640 1 Processor -- start 2023-10-14T17:58:12.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.182+0000 7f9743fff640 1 -- start start 2023-10-14T17:58:12.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.183+0000 7f9743fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97441f5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9743fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9743fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f974415cd70 0x7f9744153d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9743fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97440cafb0 con 0x7f97440d0be0 2023-10-14T17:58:12.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9743fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97440cacb0 con 0x7f97440b8480 2023-10-14T17:58:12.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9743fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97441f6080 con 0x7f974415cd70 2023-10-14T17:58:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f973bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f973bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45866/0 (socket says 172.21.15.29:45866) 2023-10-14T17:58:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f973bfff640 1 -- 172.21.15.29:0/685805639 learned_addr learned my addr 172.21.15.29:0/685805639 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f9742ffd640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97441f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.184+0000 7f97437fe640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f974415cd70 0x7f9744153d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 -- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f974415cd70 msgr2=0x7f9744153d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f974415cd70 0x7f9744153d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 -- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 msgr2=0x7f97441f5850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97441f5850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 -- 172.21.15.29:0/685805639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0094d0 con 0x7f97440d0be0 2023-10-14T17:58:12.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.185+0000 7f973bfff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f972c000c00 tx=0x7f972c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:12.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.186+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c039a90 con 0x7f97440d0be0 2023-10-14T17:58:12.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.186+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f972c039bf0 con 0x7f97440d0be0 2023-10-14T17:58:12.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.186+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97441f6360 con 0x7f97440d0be0 2023-10-14T17:58:12.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.186+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972c043410 con 0x7f97440d0be0 2023-10-14T17:58:12.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.186+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97441524c0 con 0x7f97440d0be0 2023-10-14T17:58:12.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.187+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97440cb120 con 0x7f97440d0be0 2023-10-14T17:58:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.188+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f972c04c3b0 con 0x7f97440d0be0 2023-10-14T17:58:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.189+0000 7f973b7fe640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 0x7f972406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.189+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(181..181 src has 1..181) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f972c0abf10 con 0x7f97440d0be0 2023-10-14T17:58:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.189+0000 7f9742ffd640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 0x7f972406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:12.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.190+0000 7f9742ffd640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 0x7f972406d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9734000f60 tx=0x7f9734009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:12.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.194+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f972c076ad0 con 0x7f97440d0be0 2023-10-14T17:58:12.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.395+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f974404e9b0 con 0x7f97440d0be0 2023-10-14T17:58:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.937+0000 7f973b7fe640 1 -- 172.21.15.29:0/685805639 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v182) v1 ==== 181+0+0 (secure 0 0 0) 0x7f972c07a0f0 con 0x7f97440d0be0 2023-10-14T17:58:12.938 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T17:58:12.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.941+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 msgr2=0x7f972406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.941+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 0x7f972406d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f9734000f60 tx=0x7f9734009040 comp rx=0 tx=0).stop 2023-10-14T17:58:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.941+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 msgr2=0x7f9744151f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:12.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.941+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 secure :-1 s=READY pgs=2103 cs=0 l=1 rev1=1 crypto rx=0x7f972c000c00 tx=0x7f972c004730 comp rx=0 tx=0).stop 2023-10-14T17:58:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 shutdown_connections 2023-10-14T17:58:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f974415cd70 0x7f9744153d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97440d0be0 0x7f9744151f80 unknown :-1 s=CLOSED pgs=2103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f972406afc0 0x7f972406d480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 --2- 172.21.15.29:0/685805639 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f97440b8480 0x7f97441f5850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.942+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 >> 172.21.15.29:0/685805639 conn(0x7f97440cf570 msgr2=0x7f97440b7670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.943+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 shutdown_connections 2023-10-14T17:58:12.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:12.943+0000 7f9743fff640 1 -- 172.21.15.29:0/685805639 wait complete. 2023-10-14T17:58:12.965 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T17:58:13.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.192+0000 7f67b3347640 1 Processor -- start 2023-10-14T17:58:13.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.192+0000 7f67b3347640 1 -- start start 2023-10-14T17:58:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b3347640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b3347640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 0x7f67ac0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b3347640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67ac15d150 con 0x7f67ac15cd70 2023-10-14T17:58:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b3347640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac0d1180 con 0x7f67ac0b8480 2023-10-14T17:58:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b3347640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67ac0d1460 con 0x7f67ac0d0be0 2023-10-14T17:58:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b2345640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f67ac15cd70 0x7f67ac0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33486/0 (socket says 172.21.15.29:33486) 2023-10-14T17:58:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b2345640 1 -- 172.21.15.29:0/1210401672 learned_addr learned my addr 172.21.15.29:0/1210401672 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:13.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b1b44640 1 --2- 172.21.15.29:0/1210401672 >> v2:172.21.15.29:3300/0 conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:13.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b1343640 1 -- 172.21.15.29:0/1210401672 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429221357 0 0) 0x7f67ac15d150 con 0x7f67ac15cd70 2023-10-14T17:58:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.194+0000 7f67b1343640 1 -- 172.21.15.29:0/1210401672 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6798003520 con 0x7f67ac15cd70 2023-10-14T17:58:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b1b44640 1 -- 172.21.15.29:0/1210401672 >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 msgr2=0x7f67ac0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b2b46640 1 --2- 172.21.15.29:0/1210401672 >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 0x7f67ac0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b1b44640 1 --2- 172.21.15.29:0/1210401672 >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 0x7f67ac0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b1b44640 1 -- 172.21.15.29:0/1210401672 >> v1:172.21.15.29:6789/0 conn(0x7f67ac15cd70 legacy=0x7f67ac0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b1b44640 1 -- 172.21.15.29:0/1210401672 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ac0d1740 con 0x7f67ac0b8480 2023-10-14T17:58:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.195+0000 7f67b2b46640 1 --2- 172.21.15.29:0/1210401672 >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 0x7f67ac0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.196+0000 7f67b1b44640 1 --2- 172.21.15.29:0/1210401672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0b8910 tx=0x7f679c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0b6cad17ea38db2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.196+0000 7f67b1343640 1 -- 172.21.15.29:0/1210401672 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c02e070 con 0x7f67ac0b8480 2023-10-14T17:58:13.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.196+0000 7f67b1343640 1 -- 172.21.15.29:0/1210401672 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f679c0315a0 con 0x7f67ac0b8480 2023-10-14T17:58:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.196+0000 7f67b1343640 1 -- 172.21.15.29:0/1210401672 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f679c031910 con 0x7f67ac0b8480 2023-10-14T17:58:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 -- 172.21.15.29:0/1210401672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 msgr2=0x7f67ac0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 --2- 172.21.15.29:0/1210401672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 secure :-1 s=READY pgs=2105 cs=0 l=1 rev1=1 crypto rx=0x7f67ac0b8910 tx=0x7f679c02b840 comp rx=0 tx=0).stop 2023-10-14T17:58:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 -- 172.21.15.29:0/1210401672 shutdown_connections 2023-10-14T17:58:13.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 --2- 172.21.15.29:0/1210401672 >> v2:172.21.15.186:3300/0 conn(0x7f67ac0d0be0 0x7f67ac0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 --2- 172.21.15.29:0/1210401672 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac0cd7c0 unknown :-1 s=CLOSED pgs=2105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 -- 172.21.15.29:0/1210401672 >> 172.21.15.29:0/1210401672 conn(0x7f67ac0cf570 msgr2=0x7f67ac0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 -- 172.21.15.29:0/1210401672 shutdown_connections 2023-10-14T17:58:13.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.197+0000 7f67b3347640 1 -- 172.21.15.29:0/1210401672 wait complete. 2023-10-14T17:58:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.199+0000 7f67b3347640 1 Processor -- start 2023-10-14T17:58:13.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.199+0000 7f67b3347640 1 -- start start 2023-10-14T17:58:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67ac0d0be0 0x7f67ac15e390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67ac15cd70 0x7f67ac15e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15ee10 con 0x7f67ac0b8480 2023-10-14T17:58:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15f0a0 con 0x7f67ac0d0be0 2023-10-14T17:58:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b3347640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67ac15f380 con 0x7f67ac15cd70 2023-10-14T17:58:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b2345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b2345640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45884/0 (socket says 172.21.15.29:45884) 2023-10-14T17:58:13.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.200+0000 7f67b2345640 1 -- 172.21.15.29:0/3819283386 learned_addr learned my addr 172.21.15.29:0/3819283386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 -- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67ac15cd70 msgr2=0x7f67ac15e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67ac15cd70 0x7f67ac15e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 -- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67ac0d0be0 msgr2=0x7f67ac15e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67ac0d0be0 0x7f67ac15e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 -- 172.21.15.29:0/3819283386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c009400 con 0x7f67ac0b8480 2023-10-14T17:58:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b2345640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c430 tx=0x7f67a800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a8005ce0 con 0x7f67ac0b8480 2023-10-14T17:58:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8005e40 con 0x7f67ac0b8480 2023-10-14T17:58:13.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ac15f660 con 0x7f67ac0b8480 2023-10-14T17:58:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a80072d0 con 0x7f67ac0b8480 2023-10-14T17:58:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.201+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ac15fc20 con 0x7f67ac0b8480 2023-10-14T17:58:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.202+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ac15fe10 con 0x7f67ac0b8480 2023-10-14T17:58:13.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.203+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f67a8007430 con 0x7f67ac0b8480 2023-10-14T17:58:13.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.204+0000 7f67a2ffd640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 0x7f679006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:13.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.205+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(182..182 src has 1..182) v4 ==== 6161+0+0 (secure 0 0 0) 0x7f67a8088f60 con 0x7f67ac0b8480 2023-10-14T17:58:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.205+0000 7f67b1b44640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 0x7f679006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.205+0000 7f67b1b44640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 0x7f679006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f679c036000 tx=0x7f679c030000 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:13.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.209+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f67a80577d0 con 0x7f67ac0b8480 2023-10-14T17:58:13.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.406+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f67ac04e9b0 con 0x7f67ac0b8480 2023-10-14T17:58:13.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.949+0000 7f67a2ffd640 1 -- 172.21.15.29:0/3819283386 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v183) v1 ==== 169+0+0 (secure 0 0 0) 0x7f67a8017020 con 0x7f67ac0b8480 2023-10-14T17:58:13.951 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T17:58:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 msgr2=0x7f679006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 0x7f679006d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f679c036000 tx=0x7f679c030000 comp rx=0 tx=0).stop 2023-10-14T17:58:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 msgr2=0x7f67ac156a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 secure :-1 s=READY pgs=2106 cs=0 l=1 rev1=1 crypto rx=0x7f67a800c430 tx=0x7f67a800c8f0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 shutdown_connections 2023-10-14T17:58:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67ac15cd70 0x7f67ac15e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f679006af90 0x7f679006d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67ac0d0be0 0x7f67ac15e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 --2- 172.21.15.29:0/3819283386 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f67ac0b8480 0x7f67ac156a30 unknown :-1 s=CLOSED pgs=2106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:13.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.953+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 >> 172.21.15.29:0/3819283386 conn(0x7f67ac0cf570 msgr2=0x7f67ac0b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.954+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 shutdown_connections 2023-10-14T17:58:13.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:13.954+0000 7f67b3347640 1 -- 172.21.15.29:0/3819283386 wait complete. 2023-10-14T17:58:13.976 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T17:58:14.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.203+0000 7f3016672640 1 Processor -- start 2023-10-14T17:58:14.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.203+0000 7f3016672640 1 -- start start 2023-10-14T17:58:14.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.205+0000 7f3016672640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f301006c630 0x7f30100cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.205+0000 7f3016672640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f301006f3a0 con 0x7f30100b0c90 2023-10-14T17:58:14.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.205+0000 7f3016672640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30100c8e70 con 0x7f30100adb30 2023-10-14T17:58:14.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.205+0000 7f3016672640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30100c9150 con 0x7f301006c630 2023-10-14T17:58:14.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3015e71640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30100b0c90 0x7f30100c9950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33502/0 (socket says 172.21.15.29:33502) 2023-10-14T17:58:14.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3015e71640 1 -- 172.21.15.29:0/1589347832 learned_addr learned my addr 172.21.15.29:0/1589347832 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:14.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3015670640 1 --2- 172.21.15.29:0/1589347832 >> v2:172.21.15.186:3300/0 conn(0x7f301006c630 0x7f30100cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3257745232 0 0) 0x7f301006f3a0 con 0x7f30100b0c90 2023-10-14T17:58:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ffc003520 con 0x7f30100b0c90 2023-10-14T17:58:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2732720406 0 0) 0x7f2ffc003520 con 0x7f30100b0c90 2023-10-14T17:58:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 >> v2:172.21.15.186:3300/0 conn(0x7f301006c630 msgr2=0x7f30100cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 --2- 172.21.15.29:0/1589347832 >> v2:172.21.15.186:3300/0 conn(0x7f301006c630 0x7f30100cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 >> v1:172.21.15.186:6789/0 conn(0x7f30100adb30 legacy=0x7f30100cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:14.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.206+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30100c9430 con 0x7f30100b0c90 2023-10-14T17:58:14.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f30080019f0 con 0x7f30100b0c90 2023-10-14T17:58:14.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 >> v1:172.21.15.29:6789/0 conn(0x7f30100b0c90 legacy=0x7f30100c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ffc0039c0 0x7f2ffc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0047c0 0x7f2ffc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 0x7f2ffc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30100c9150 con 0x7f2ffc0051d0 2023-10-14T17:58:14.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc00c100 con 0x7f2ffc0039c0 2023-10-14T17:58:14.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ffc00c2c0 con 0x7f2ffc0047c0 2023-10-14T17:58:14.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3015e71640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 0x7f2ffc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3015670640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ffc0039c0 0x7f2ffc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3015e71640 1 -- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0047c0 msgr2=0x7f2ffc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.207+0000 7f3015e71640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0047c0 0x7f2ffc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.208+0000 7f3015e71640 1 -- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ffc0039c0 msgr2=0x7f2ffc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:14.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.208+0000 7f3015e71640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ffc0039c0 0x7f2ffc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.208+0000 7f3015e71640 1 -- 172.21.15.29:0/1589347832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ffc003520 con 0x7f2ffc0051d0 2023-10-14T17:58:14.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.208+0000 7f3015e71640 1 -- 172.21.15.29:0/1589347832 reap_dead start 2023-10-14T17:58:14.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.209+0000 7f3015e71640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 0x7f2ffc00b980 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f300800a810 tx=0x7f2ffc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29f055ed9b720d78 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.209+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008034630 con 0x7f2ffc0051d0 2023-10-14T17:58:14.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.209+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300802f070 con 0x7f2ffc0051d0 2023-10-14T17:58:14.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.209+0000 7f3007fff640 1 -- 172.21.15.29:0/1589347832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3008034a10 con 0x7f2ffc0051d0 2023-10-14T17:58:14.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 -- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 msgr2=0x7f2ffc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 0x7f2ffc00b980 secure :-1 s=READY pgs=2108 cs=0 l=1 rev1=1 crypto rx=0x7f300800a810 tx=0x7f2ffc0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 -- 172.21.15.29:0/1589347832 shutdown_connections 2023-10-14T17:58:14.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 --2- 172.21.15.29:0/1589347832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ffc0051d0 0x7f2ffc00b980 unknown :-1 s=CLOSED pgs=2108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 -- 172.21.15.29:0/1589347832 >> 172.21.15.29:0/1589347832 conn(0x7f30100b0170 msgr2=0x7f30100b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 -- 172.21.15.29:0/1589347832 shutdown_connections 2023-10-14T17:58:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.210+0000 7f3016672640 1 -- 172.21.15.29:0/1589347832 wait complete. 2023-10-14T17:58:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.212+0000 7f3016672640 1 Processor -- start 2023-10-14T17:58:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.212+0000 7f3016672640 1 -- start start 2023-10-14T17:58:14.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0051d0 0x7f3010154990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 0x7f30100c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30101649b0 con 0x7f3010163210 2023-10-14T17:58:14.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30101fa110 con 0x7f30101f6400 2023-10-14T17:58:14.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3016672640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30101fa3f0 con 0x7f2ffc0051d0 2023-10-14T17:58:14.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3014e6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3014e6f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45908/0 (socket says 172.21.15.29:45908) 2023-10-14T17:58:14.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.213+0000 7f3014e6f640 1 -- 172.21.15.29:0/2455344176 learned_addr learned my addr 172.21.15.29:0/2455344176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:14.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3015e71640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 0x7f30100c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3015670640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0051d0 0x7f3010154990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 -- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0051d0 msgr2=0x7f3010154990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0051d0 0x7f3010154990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 -- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 msgr2=0x7f30100c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 0x7f30100c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 -- 172.21.15.29:0/2455344176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30101fa6d0 con 0x7f3010163210 2023-10-14T17:58:14.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3015e71640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 0x7f30100c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:14.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.214+0000 7f3014e6f640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f300000c480 tx=0x7f300000c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.215+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000012830 con 0x7f3010163210 2023-10-14T17:58:14.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.215+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3000012990 con 0x7f3010163210 2023-10-14T17:58:14.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.215+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000012d80 con 0x7f3010163210 2023-10-14T17:58:14.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.215+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30101fa9b0 con 0x7f3010163210 2023-10-14T17:58:14.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.215+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30101faf70 con 0x7f3010163210 2023-10-14T17:58:14.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.216+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f301004e9b0 con 0x7f3010163210 2023-10-14T17:58:14.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.217+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f300001b400 con 0x7f3010163210 2023-10-14T17:58:14.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.218+0000 7f3005ffb640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 0x7f2ff406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.218+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f3000089710 con 0x7f3010163210 2023-10-14T17:58:14.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.218+0000 7f3015670640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 0x7f2ff406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.219+0000 7f3015670640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 0x7f2ff406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f300c009990 tx=0x7f300c008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.223+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3000054670 con 0x7f3010163210 2023-10-14T17:58:14.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.420+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f30101638a0 con 0x7f3010163210 2023-10-14T17:58:14.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.424+0000 7f3005ffb640 1 -- 172.21.15.29:0/2455344176 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v61) v1 ==== 115+0+0 (secure 0 0 0) 0x7f3000057c90 con 0x7f3010163210 2023-10-14T17:58:14.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.427+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 msgr2=0x7f2ff406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.427+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 0x7f2ff406d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f300c009990 tx=0x7f300c008040 comp rx=0 tx=0).stop 2023-10-14T17:58:14.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 msgr2=0x7f3010164470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 secure :-1 s=READY pgs=2109 cs=0 l=1 rev1=1 crypto rx=0x7f300000c480 tx=0x7f300000c940 comp rx=0 tx=0).stop 2023-10-14T17:58:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 shutdown_connections 2023-10-14T17:58:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2ff406af90 0x7f2ff406d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30101f6400 0x7f30100c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3010163210 0x7f3010164470 unknown :-1 s=CLOSED pgs=2109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 --2- 172.21.15.29:0/2455344176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2ffc0051d0 0x7f3010154990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 >> 172.21.15.29:0/2455344176 conn(0x7f30100b0170 msgr2=0x7f30101609a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:14.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 shutdown_connections 2023-10-14T17:58:14.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.428+0000 7f3016672640 1 -- 172.21.15.29:0/2455344176 wait complete. 2023-10-14T17:58:14.451 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T17:58:14.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.679+0000 7ff55a9ef640 1 Processor -- start 2023-10-14T17:58:14.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.680+0000 7ff55a9ef640 1 -- start start 2023-10-14T17:58:14.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.681+0000 7ff55a9ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff55415cd70 0x7ff5540d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.681+0000 7ff55a9ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55415d150 con 0x7ff5540cded0 2023-10-14T17:58:14.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.681+0000 7ff55a9ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5540d1180 con 0x7ff5540b8480 2023-10-14T17:58:14.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.681+0000 7ff55a9ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5540d1460 con 0x7ff55415cd70 2023-10-14T17:58:14.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff5599ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff55415cd70 0x7ff5540d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff55a1ee640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff5540cded0 0x7ff5540b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33516/0 (socket says 172.21.15.29:33516) 2023-10-14T17:58:14.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff5599ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff55415cd70 0x7ff5540d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45914/0 (socket says 172.21.15.29:45914) 2023-10-14T17:58:14.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff55a1ee640 1 -- 172.21.15.29:0/1866897715 learned_addr learned my addr 172.21.15.29:0/1866897715 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:14.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff5599ed640 1 -- 172.21.15.29:0/1866897715 >> v1:172.21.15.186:6789/0 conn(0x7ff5540b8480 legacy=0x7ff5540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:14.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff5599ed640 1 -- 172.21.15.29:0/1866897715 >> v1:172.21.15.29:6789/0 conn(0x7ff5540cded0 legacy=0x7ff5540b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.682+0000 7ff5599ed640 1 -- 172.21.15.29:0/1866897715 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5540d1740 con 0x7ff55415cd70 2023-10-14T17:58:14.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.684+0000 7ff5599ed640 1 --2- 172.21.15.29:0/1866897715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55415cd70 0x7ff5540d2220 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7ff5500097a0 tx=0x7ff55002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b4762dfea3d9f92 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.684+0000 7ff5589eb640 1 -- 172.21.15.29:0/1866897715 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550031830 con 0x7ff55415cd70 2023-10-14T17:58:14.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.684+0000 7ff5589eb640 1 -- 172.21.15.29:0/1866897715 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff550031990 con 0x7ff55415cd70 2023-10-14T17:58:14.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.684+0000 7ff5589eb640 1 -- 172.21.15.29:0/1866897715 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff550031d20 con 0x7ff55415cd70 2023-10-14T17:58:14.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 -- 172.21.15.29:0/1866897715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55415cd70 msgr2=0x7ff5540d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/1866897715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55415cd70 0x7ff5540d2220 secure :-1 s=READY pgs=2111 cs=0 l=1 rev1=1 crypto rx=0x7ff5500097a0 tx=0x7ff55002b8d0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 -- 172.21.15.29:0/1866897715 shutdown_connections 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/1866897715 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff55415cd70 0x7ff5540d2220 unknown :-1 s=CLOSED pgs=2111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 -- 172.21.15.29:0/1866897715 >> 172.21.15.29:0/1866897715 conn(0x7ff5540cf570 msgr2=0x7ff5540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 -- 172.21.15.29:0/1866897715 shutdown_connections 2023-10-14T17:58:14.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.685+0000 7ff55a9ef640 1 -- 172.21.15.29:0/1866897715 wait complete. 2023-10-14T17:58:14.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.687+0000 7ff55a9ef640 1 Processor -- start 2023-10-14T17:58:14.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.687+0000 7ff55a9ef640 1 -- start start 2023-10-14T17:58:14.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5540b8480 0x7ff55414f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 0x7ff5540cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540b2ca0 con 0x7ff5540cded0 2023-10-14T17:58:14.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540b2f80 con 0x7ff55415cd70 2023-10-14T17:58:14.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a9ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5540b2110 con 0x7ff5540b8480 2023-10-14T17:58:14.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff5591ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff5591ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45920/0 (socket says 172.21.15.29:45920) 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff5591ec640 1 -- 172.21.15.29:0/340395623 learned_addr learned my addr 172.21.15.29:0/340395623 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff55a1ee640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 0x7ff5540cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff5591ec640 1 -- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5540b8480 msgr2=0x7ff55414f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.688+0000 7ff5599ed640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5540b8480 0x7ff55414f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff5591ec640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5540b8480 0x7ff55414f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff5591ec640 1 -- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 msgr2=0x7ff5540cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff5591ec640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 0x7ff5540cbbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff5591ec640 1 -- 172.21.15.29:0/340395623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff550009400 con 0x7ff5540cded0 2023-10-14T17:58:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff55a1ee640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 0x7ff5540cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff5591ec640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c430 tx=0x7ff54400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011790 con 0x7ff5540cded0 2023-10-14T17:58:14.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5440118f0 con 0x7ff5540cded0 2023-10-14T17:58:14.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff544011ce0 con 0x7ff5540cded0 2023-10-14T17:58:14.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5540b23f0 con 0x7ff5540cded0 2023-10-14T17:58:14.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.689+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55414fcf0 con 0x7ff5540cded0 2023-10-14T17:58:14.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.691+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5540b2910 con 0x7ff5540cded0 2023-10-14T17:58:14.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.692+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff54401c460 con 0x7ff5540cded0 2023-10-14T17:58:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.693+0000 7ff54a7fc640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 0x7ff53806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.693+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7ff5440897f0 con 0x7ff5540cded0 2023-10-14T17:58:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.693+0000 7ff5599ed640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 0x7ff53806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:14.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.694+0000 7ff5599ed640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 0x7ff53806d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff55003c040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:14.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.698+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5440546c0 con 0x7ff5540cded0 2023-10-14T17:58:14.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.893+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7ff5540b30f0 con 0x7ff5540cded0 2023-10-14T17:58:14.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.901+0000 7ff54a7fc640 1 -- 172.21.15.29:0/340395623 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v62) v1 ==== 120+0+0 (secure 0 0 0) 0x7ff544057ce0 con 0x7ff5540cded0 2023-10-14T17:58:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 msgr2=0x7ff53806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 0x7ff53806d480 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7ff550036000 tx=0x7ff55003c040 comp rx=0 tx=0).stop 2023-10-14T17:58:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 msgr2=0x7ff5540b1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 secure :-1 s=READY pgs=2112 cs=0 l=1 rev1=1 crypto rx=0x7ff54400c430 tx=0x7ff54400c8f0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 shutdown_connections 2023-10-14T17:58:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff53806afc0 0x7ff53806d480 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff55415cd70 0x7ff5540cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5540cded0 0x7ff5540b1690 unknown :-1 s=CLOSED pgs=2112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 --2- 172.21.15.29:0/340395623 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5540b8480 0x7ff55414f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:14.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.904+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 >> 172.21.15.29:0/340395623 conn(0x7ff5540cf570 msgr2=0x7ff5540b5c50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:14.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.905+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 shutdown_connections 2023-10-14T17:58:14.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:14.905+0000 7ff55a9ef640 1 -- 172.21.15.29:0/340395623 wait complete. 2023-10-14T17:58:14.927 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T17:58:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.158+0000 7fe226a39640 1 Processor -- start 2023-10-14T17:58:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.158+0000 7fe226a39640 1 -- start start 2023-10-14T17:58:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe226a39640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe220159ab0 0x7fe2200ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe226a39640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 0x7fe22006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe226a39640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe220070360 con 0x7fe220159e90 2023-10-14T17:58:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe226a39640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe22006cb10 con 0x7fe220159ab0 2023-10-14T17:58:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe226a39640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe22015d890 con 0x7fe22006c140 2023-10-14T17:58:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe21ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe220159ab0 0x7fe2200ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe21f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe220159e90 0x7fe22006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33530/0 (socket says 172.21.15.29:33530) 2023-10-14T17:58:15.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe21f7fe640 1 -- 172.21.15.29:0/4144109415 learned_addr learned my addr 172.21.15.29:0/4144109415 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:15.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.160+0000 7fe21effd640 1 -- 172.21.15.29:0/4144109415 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204575304 0 0) 0x7fe220070360 con 0x7fe220159e90 2023-10-14T17:58:15.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe224fd6640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 0x7fe22006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe21effd640 1 -- 172.21.15.29:0/4144109415 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe208005730 con 0x7fe220159e90 2023-10-14T17:58:15.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe21ffff640 1 -- 172.21.15.29:0/4144109415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 msgr2=0x7fe22006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:15.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe21ffff640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 0x7fe22006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe21ffff640 1 -- 172.21.15.29:0/4144109415 >> v1:172.21.15.29:6789/0 conn(0x7fe220159e90 legacy=0x7fe22006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe21ffff640 1 -- 172.21.15.29:0/4144109415 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2200cd1f0 con 0x7fe220159ab0 2023-10-14T17:58:15.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.161+0000 7fe224fd6640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 0x7fe22006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:15.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.162+0000 7fe21ffff640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ce370 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe2100097a0 tx=0x7fe21002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f47d85a667b0a28d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.162+0000 7fe21effd640 1 -- 172.21.15.29:0/4144109415 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe2100316f0 con 0x7fe220159ab0 2023-10-14T17:58:15.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.162+0000 7fe21effd640 1 -- 172.21.15.29:0/4144109415 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe210031850 con 0x7fe220159ab0 2023-10-14T17:58:15.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.162+0000 7fe21effd640 1 -- 172.21.15.29:0/4144109415 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe210031be0 con 0x7fe220159ab0 2023-10-14T17:58:15.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 -- 172.21.15.29:0/4144109415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 msgr2=0x7fe2200ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ce370 secure :-1 s=READY pgs=2114 cs=0 l=1 rev1=1 crypto rx=0x7fe2100097a0 tx=0x7fe21002b790 comp rx=0 tx=0).stop 2023-10-14T17:58:15.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 -- 172.21.15.29:0/4144109415 shutdown_connections 2023-10-14T17:58:15.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe22006c140 0x7fe22006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 --2- 172.21.15.29:0/4144109415 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ce370 unknown :-1 s=CLOSED pgs=2114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 -- 172.21.15.29:0/4144109415 >> 172.21.15.29:0/4144109415 conn(0x7fe2200ce9b0 msgr2=0x7fe2200cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:15.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 -- 172.21.15.29:0/4144109415 shutdown_connections 2023-10-14T17:58:15.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.163+0000 7fe226a39640 1 -- 172.21.15.29:0/4144109415 wait complete. 2023-10-14T17:58:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.165+0000 7fe226a39640 1 Processor -- start 2023-10-14T17:58:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.165+0000 7fe226a39640 1 -- start start 2023-10-14T17:58:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 0x7fe2200b4270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe220159e90 0x7fe2201587e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200b2360 con 0x7fe220159ab0 2023-10-14T17:58:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200b1da0 con 0x7fe22006c140 2023-10-14T17:58:15.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe226a39640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe2200b2080 con 0x7fe220159e90 2023-10-14T17:58:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe21f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe21f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45942/0 (socket says 172.21.15.29:45942) 2023-10-14T17:58:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.166+0000 7fe21f7fe640 1 -- 172.21.15.29:0/928232098 learned_addr learned my addr 172.21.15.29:0/928232098 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21ffff640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 0x7fe2200b4270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe224fd6640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe220159e90 0x7fe2201587e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 -- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe220159e90 msgr2=0x7fe2201587e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe220159e90 0x7fe2201587e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 -- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 msgr2=0x7fe2200b4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 0x7fe2200b4270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 -- 172.21.15.29:0/928232098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210009400 con 0x7fe220159ab0 2023-10-14T17:58:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21ffff640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 0x7fe2200b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21f7fe640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fe21400c500 tx=0x7fe21400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214005ce0 con 0x7fe220159ab0 2023-10-14T17:58:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe214005e40 con 0x7fe220159ab0 2023-10-14T17:58:15.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2200b3a70 con 0x7fe220159ab0 2023-10-14T17:58:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.167+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe214007330 con 0x7fe220159ab0 2023-10-14T17:58:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.168+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe220159230 con 0x7fe220159ab0 2023-10-14T17:58:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.169+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2200b38d0 con 0x7fe220159ab0 2023-10-14T17:58:15.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.170+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe214007550 con 0x7fe220159ab0 2023-10-14T17:58:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.171+0000 7fe21cff9640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 0x7fe20c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.171+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fe214088e20 con 0x7fe220159ab0 2023-10-14T17:58:15.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.171+0000 7fe21ffff640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 0x7fe20c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.172+0000 7fe21ffff640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 0x7fe20c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe210036000 tx=0x7fe210008030 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.175+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe2200b38d0 con 0x7fe220159ab0 2023-10-14T17:58:15.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.371+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe2200b38d0 con 0x7fe220159ab0 2023-10-14T17:58:15.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.378+0000 7fe21cff9640 1 -- 172.21.15.29:0/928232098 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v63) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe2200b38d0 con 0x7fe220159ab0 2023-10-14T17:58:15.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 msgr2=0x7fe20c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 0x7fe20c06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe210036000 tx=0x7fe210008030 comp rx=0 tx=0).stop 2023-10-14T17:58:15.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 msgr2=0x7fe2200ace30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 secure :-1 s=READY pgs=2115 cs=0 l=1 rev1=1 crypto rx=0x7fe21400c500 tx=0x7fe21400c9c0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 shutdown_connections 2023-10-14T17:58:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe220159e90 0x7fe2201587e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe220159ab0 0x7fe2200ace30 unknown :-1 s=CLOSED pgs=2115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fe20c06af90 0x7fe20c06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 --2- 172.21.15.29:0/928232098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe22006c140 0x7fe2200b4270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 >> 172.21.15.29:0/928232098 conn(0x7fe2200ce9b0 msgr2=0x7fe2201e4330 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 shutdown_connections 2023-10-14T17:58:15.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.381+0000 7fe226a39640 1 -- 172.21.15.29:0/928232098 wait complete. 2023-10-14T17:58:15.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove' 2023-10-14T17:58:15.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.644+0000 7fa039f3e640 1 Processor -- start 2023-10-14T17:58:15.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.644+0000 7fa039f3e640 1 -- start start 2023-10-14T17:58:15.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa039f3e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa03415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa039f3e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa03415ae80 0x7fa0340cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa039f3e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03415ed60 con 0x7fa0340ccaa0 2023-10-14T17:58:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa039f3e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03406b810 con 0x7fa03415ae80 2023-10-14T17:58:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa039f3e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340707e0 con 0x7fa03415aaa0 2023-10-14T17:58:15.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.645+0000 7fa032ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa03415ae80 0x7fa0340cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa03415ae80 0x7fa0340cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45948/0 (socket says 172.21.15.29:45948) 2023-10-14T17:58:15.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 -- 172.21.15.29:0/342548140 learned_addr learned my addr 172.21.15.29:0/342548140 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:15.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 -- 172.21.15.29:0/342548140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 msgr2=0x7fa03415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:15.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 --2- 172.21.15.29:0/342548140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa03415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 -- 172.21.15.29:0/342548140 >> v1:172.21.15.186:6789/0 conn(0x7fa0340ccaa0 legacy=0x7fa034070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:15.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.646+0000 7fa032ffd640 1 -- 172.21.15.29:0/342548140 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03415ef90 con 0x7fa03415ae80 2023-10-14T17:58:15.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.647+0000 7fa032ffd640 1 --2- 172.21.15.29:0/342548140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa03415ae80 0x7fa0340cc560 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fa028009870 tx=0x7fa02802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f73d9b41fe4576d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.647+0000 7fa0327fc640 1 -- 172.21.15.29:0/342548140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028031640 con 0x7fa03415ae80 2023-10-14T17:58:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.647+0000 7fa0327fc640 1 -- 172.21.15.29:0/342548140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0280317a0 con 0x7fa03415ae80 2023-10-14T17:58:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.647+0000 7fa0327fc640 1 -- 172.21.15.29:0/342548140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa028031b30 con 0x7fa03415ae80 2023-10-14T17:58:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 -- 172.21.15.29:0/342548140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa03415ae80 msgr2=0x7fa0340cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 --2- 172.21.15.29:0/342548140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa03415ae80 0x7fa0340cc560 secure :-1 s=READY pgs=2116 cs=0 l=1 rev1=1 crypto rx=0x7fa028009870 tx=0x7fa02802b8f0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 -- 172.21.15.29:0/342548140 shutdown_connections 2023-10-14T17:58:15.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 --2- 172.21.15.29:0/342548140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa03415ae80 0x7fa0340cc560 unknown :-1 s=CLOSED pgs=2116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 --2- 172.21.15.29:0/342548140 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa03415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 -- 172.21.15.29:0/342548140 >> 172.21.15.29:0/342548140 conn(0x7fa0340cb410 msgr2=0x7fa0340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:15.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 -- 172.21.15.29:0/342548140 shutdown_connections 2023-10-14T17:58:15.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.648+0000 7fa039f3e640 1 -- 172.21.15.29:0/342548140 wait complete. 2023-10-14T17:58:15.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.650+0000 7fa039f3e640 1 Processor -- start 2023-10-14T17:58:15.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.650+0000 7fa039f3e640 1 -- start start 2023-10-14T17:58:15.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa0340acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03415ae80 0x7fa034153ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340c4470 con 0x7fa0340ccaa0 2023-10-14T17:58:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340c4750 con 0x7fa03415ae80 2023-10-14T17:58:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa039f3e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0340c4a30 con 0x7fa03415aaa0 2023-10-14T17:58:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.651+0000 7fa0337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45960/0 (socket says 172.21.15.29:45960) 2023-10-14T17:58:15.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 -- 172.21.15.29:0/1914613648 learned_addr learned my addr 172.21.15.29:0/1914613648 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 -- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 msgr2=0x7fa0340acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa0340acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 -- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03415ae80 msgr2=0x7fa034153ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03415ae80 0x7fa034153ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 -- 172.21.15.29:0/1914613648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0280094d0 con 0x7fa0340ccaa0 2023-10-14T17:58:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.652+0000 7fa0337fe640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00ba50 tx=0x7fa02c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.653+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c00cbe0 con 0x7fa0340ccaa0 2023-10-14T17:58:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.653+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c00cd60 con 0x7fa0340ccaa0 2023-10-14T17:58:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.653+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa02c014680 con 0x7fa0340ccaa0 2023-10-14T17:58:15.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.653+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0340c4d10 con 0x7fa0340ccaa0 2023-10-14T17:58:15.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.653+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0340c2710 con 0x7fa0340ccaa0 2023-10-14T17:58:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.654+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0340c5230 con 0x7fa0340ccaa0 2023-10-14T17:58:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.655+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa02c01d3b0 con 0x7fa0340ccaa0 2023-10-14T17:58:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.656+0000 7fa01bfff640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 0x7fa02006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.656+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fa02c0897c0 con 0x7fa0340ccaa0 2023-10-14T17:58:15.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.656+0000 7fa032ffd640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 0x7fa02006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.657+0000 7fa032ffd640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 0x7fa02006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa028036000 tx=0x7fa028030000 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:15.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.660+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa0340c5230 con 0x7fa0340ccaa0 2023-10-14T17:58:15.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.928+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]} v 0) v1 -- 0x7fa0340c5230 con 0x7fa0340ccaa0 2023-10-14T17:58:15.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.961+0000 7fa01bfff640 1 -- 172.21.15.29:0/1914613648 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_list_post_schedule_remove"]}]=0 v2564) v1 ==== 174+0+0 (secure 0 0 0) 0x7fa02c0190f0 con 0x7fa0340ccaa0 2023-10-14T17:58:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 msgr2=0x7fa02006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 0x7fa02006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fa028036000 tx=0x7fa028030000 comp rx=0 tx=0).stop 2023-10-14T17:58:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 msgr2=0x7fa0340c3390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 secure :-1 s=READY pgs=2117 cs=0 l=1 rev1=1 crypto rx=0x7fa02c00ba50 tx=0x7fa02c00bf10 comp rx=0 tx=0).stop 2023-10-14T17:58:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 shutdown_connections 2023-10-14T17:58:15.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa02006af90 0x7fa02006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa03415ae80 0x7fa034153ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa03415aaa0 0x7fa0340acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 --2- 172.21.15.29:0/1914613648 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa0340ccaa0 0x7fa0340c3390 unknown :-1 s=CLOSED pgs=2117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 >> 172.21.15.29:0/1914613648 conn(0x7fa0340cb410 msgr2=0x7fa03406cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:15.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 shutdown_connections 2023-10-14T17:58:15.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:15.964+0000 7fa039f3e640 1 -- 172.21.15.29:0/1914613648 wait complete. 2023-10-14T17:58:15.987 INFO:tasks.cephfs_test_runner:Test listing snap schedules post removal of a schedule ... ok 2023-10-14T17:58:15.987 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test that snap schedule is resumed after mgr restart 2023-10-14T17:58:15.987 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_mgr_restart (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T17:58:15.988 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.206+0000 7f9cce55e640 1 Processor -- start 2023-10-14T17:58:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.206+0000 7f9cce55e640 1 -- start start 2023-10-14T17:58:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.207+0000 7f9cce55e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc806c040 0x7f9cc806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.207+0000 7f9cce55e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9cc80cda80 0x7f9cc815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.207+0000 7f9cce55e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc806c960 con 0x7f9cc80ad970 2023-10-14T17:58:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.207+0000 7f9cce55e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc815b9d0 con 0x7f9cc806c040 2023-10-14T17:58:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.207+0000 7f9cce55e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc815bcb0 con 0x7f9cc80cda80 2023-10-14T17:58:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc806c040 0x7f9cc806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9cc806c040 0x7f9cc806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45974/0 (socket says 172.21.15.29:45974) 2023-10-14T17:58:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 -- 172.21.15.29:0/1161879175 learned_addr learned my addr 172.21.15.29:0/1161879175 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:16.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 -- 172.21.15.29:0/1161879175 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80cda80 msgr2=0x7f9cc815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccdd5d640 1 --2- 172.21.15.29:0/1161879175 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80cda80 0x7f9cc815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 --2- 172.21.15.29:0/1161879175 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80cda80 0x7f9cc815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 -- 172.21.15.29:0/1161879175 >> v1:172.21.15.186:6789/0 conn(0x7f9cc80ad970 legacy=0x7f9cc815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.208+0000 7f9ccd55c640 1 -- 172.21.15.29:0/1161879175 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc815bf90 con 0x7f9cc806c040 2023-10-14T17:58:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.209+0000 7f9ccd55c640 1 --2- 172.21.15.29:0/1161879175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc806c040 0x7f9cc806c420 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80cf300 tx=0x7f9cc402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab814aaebcbe4ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.210+0000 7f9cbffff640 1 -- 172.21.15.29:0/1161879175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4031870 con 0x7f9cc806c040 2023-10-14T17:58:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.210+0000 7f9cbffff640 1 -- 172.21.15.29:0/1161879175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cc40319d0 con 0x7f9cc806c040 2023-10-14T17:58:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.210+0000 7f9cbffff640 1 -- 172.21.15.29:0/1161879175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cc4031d60 con 0x7f9cc806c040 2023-10-14T17:58:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.210+0000 7f9cce55e640 1 -- 172.21.15.29:0/1161879175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc806c040 msgr2=0x7f9cc806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.210+0000 7f9cce55e640 1 --2- 172.21.15.29:0/1161879175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc806c040 0x7f9cc806c420 secure :-1 s=READY pgs=2118 cs=0 l=1 rev1=1 crypto rx=0x7f9cc80cf300 tx=0x7f9cc402b910 comp rx=0 tx=0).stop 2023-10-14T17:58:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 -- 172.21.15.29:0/1161879175 shutdown_connections 2023-10-14T17:58:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 --2- 172.21.15.29:0/1161879175 >> v2:172.21.15.186:3300/0 conn(0x7f9cc80cda80 0x7f9cc815ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 --2- 172.21.15.29:0/1161879175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc806c040 0x7f9cc806c420 unknown :-1 s=CLOSED pgs=2118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 -- 172.21.15.29:0/1161879175 >> 172.21.15.29:0/1161879175 conn(0x7f9cc80ce6a0 msgr2=0x7f9cc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:16.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 -- 172.21.15.29:0/1161879175 shutdown_connections 2023-10-14T17:58:16.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.211+0000 7f9cce55e640 1 -- 172.21.15.29:0/1161879175 wait complete. 2023-10-14T17:58:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.212+0000 7f9cce55e640 1 Processor -- start 2023-10-14T17:58:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.213+0000 7f9cce55e640 1 -- start start 2023-10-14T17:58:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cc806c040 0x7f9cc81538d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cc80cda80 0x7f9cc80c79f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c6150 con 0x7f9cc80ad970 2023-10-14T17:58:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c6430 con 0x7f9cc80cda80 2023-10-14T17:58:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cce55e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc80c6710 con 0x7f9cc806c040 2023-10-14T17:58:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45978/0 (socket says 172.21.15.29:45978) 2023-10-14T17:58:16.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 -- 172.21.15.29:0/3863537593 learned_addr learned my addr 172.21.15.29:0/3863537593 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:16.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 -- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cc806c040 msgr2=0x7f9cc81538d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cc806c040 0x7f9cc81538d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 -- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cc80cda80 msgr2=0x7f9cc80c79f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cc80cda80 0x7f9cc80c79f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.214+0000 7f9cccd5b640 1 -- 172.21.15.29:0/3863537593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc40094d0 con 0x7f9cc80ad970 2023-10-14T17:58:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cccd5b640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c400 tx=0x7f9cb800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb80035f0 con 0x7f9cc80ad970 2023-10-14T17:58:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb8003750 con 0x7f9cc80ad970 2023-10-14T17:58:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cb8003b40 con 0x7f9cc80ad970 2023-10-14T17:58:16.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc80c69f0 con 0x7f9cc80ad970 2023-10-14T17:58:16.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.215+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc80af990 con 0x7f9cc80ad970 2023-10-14T17:58:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.216+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc806b330 con 0x7f9cc80ad970 2023-10-14T17:58:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.217+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9cb8003ca0 con 0x7f9cc80ad970 2023-10-14T17:58:16.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.218+0000 7f9cbdffb640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 0x7f9cac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.218+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f9cb8088980 con 0x7f9cc80ad970 2023-10-14T17:58:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.219+0000 7f9ccd55c640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 0x7f9cac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.219+0000 7f9ccd55c640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 0x7f9cac06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4036000 tx=0x7f9cc4030000 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.223+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9cb8053990 con 0x7f9cc80ad970 2023-10-14T17:58:16.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.417+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cc80c6f10 con 0x7f9cc80ad970 2023-10-14T17:58:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.419+0000 7f9cbdffb640 1 -- 172.21.15.29:0/3863537593 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v183) v1 ==== 74+0+14168 (secure 0 0 0) 0x7f9cc80c6f10 con 0x7f9cc80ad970 2023-10-14T17:58:16.420 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:16.421 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":183,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:13.938464+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6834/494120094":"2023-10-15T17:58:10.916033+0000","172.21.15.29:0/38025471":"2023-10-15T17:58:07.121151+0000","172.21.15.29:6836/494120094":"2023-10-15T17:58:10.916033+0000","172.21.15.29:0/4267645350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:6832/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/1183707990":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3006422350":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3667940423":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/3415492501":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/2674099337":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/1035847590":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/2596229485":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/4106038121":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6833/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:6832/2441704285":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1011592268":"2023-10-15T17:57:59.333970+0000","172.21.15.29:0/1904979103":"2023-10-15T17:58:07.121151+0000","172.21.15.29:0/84821019":"2023-10-15T17:57:59.333970+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:16.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 msgr2=0x7f9cac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 0x7f9cac06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9cc4036000 tx=0x7f9cc4030000 comp rx=0 tx=0).stop 2023-10-14T17:58:16.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 msgr2=0x7f9cc80c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 secure :-1 s=READY pgs=2119 cs=0 l=1 rev1=1 crypto rx=0x7f9cb800c400 tx=0x7f9cb800c8c0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 shutdown_connections 2023-10-14T17:58:16.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9cac06af90 0x7f9cac06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9cc80cda80 0x7f9cc80c79f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9cc80ad970 0x7f9cc80c5770 unknown :-1 s=CLOSED pgs=2119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 --2- 172.21.15.29:0/3863537593 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9cc806c040 0x7f9cc81538d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 >> 172.21.15.29:0/3863537593 conn(0x7f9cc80ce6a0 msgr2=0x7f9cc815c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:16.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.421+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 shutdown_connections 2023-10-14T17:58:16.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.422+0000 7f9cce55e640 1 -- 172.21.15.29:0/3863537593 wait complete. 2023-10-14T17:58:16.439 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:58:16.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:58:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.669+0000 7f2d0c5f5640 1 Processor -- start 2023-10-14T17:58:16.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.669+0000 7f2d0c5f5640 1 -- start start 2023-10-14T17:58:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.670+0000 7f2d0c5f5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2d040ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.670+0000 7f2d0c5f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040cb030 con 0x7f2d0415ad70 2023-10-14T17:58:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.670+0000 7f2d0c5f5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d040c91c0 con 0x7f2d0406a4b0 2023-10-14T17:58:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.670+0000 7f2d0c5f5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040c94a0 con 0x7f2d040caac0 2023-10-14T17:58:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0a391640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d0415ad70 0x7f2d04069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33538/0 (socket says 172.21.15.29:33538) 2023-10-14T17:58:16.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0a391640 1 -- 172.21.15.29:0/2540365330 learned_addr learned my addr 172.21.15.29:0/2540365330 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1497263981 0 0) 0x7f2d040cb030 con 0x7f2d0415ad70 2023-10-14T17:58:16.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cec003520 con 0x7f2d0415ad70 2023-10-14T17:58:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0ab92640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2d040ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266472236 0 0) 0x7f2cec003520 con 0x7f2d0415ad70 2023-10-14T17:58:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 msgr2=0x7f2d040ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2d040ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.671+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 >> v1:172.21.15.186:6789/0 conn(0x7f2d0406a4b0 legacy=0x7f2d040ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040c9780 con 0x7f2d0415ad70 2023-10-14T17:58:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2cf40019f0 con 0x7f2d0415ad70 2023-10-14T17:58:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 >> v1:172.21.15.29:6789/0 conn(0x7f2d0415ad70 legacy=0x7f2d04069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2cec003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2cec0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec004d00 0x7f2cec005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040c94a0 con 0x7f2cec0039a0 2023-10-14T17:58:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec00ff70 con 0x7f2cec004d00 2023-10-14T17:58:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cec010130 con 0x7f2d040caac0 2023-10-14T17:58:16.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0a391640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2cec003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0ab92640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec004d00 0x7f2cec005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d09b90640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2cec0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0a391640 1 -- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 msgr2=0x7f2cec0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0a391640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040caac0 0x7f2cec0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.672+0000 7f2d0a391640 1 -- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec004d00 msgr2=0x7f2cec005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.673+0000 7f2d0a391640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec004d00 0x7f2cec005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.673+0000 7f2d0a391640 1 -- 172.21.15.29:0/2540365330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cec003520 con 0x7f2cec0039a0 2023-10-14T17:58:16.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.673+0000 7f2d0ab92640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cec004d00 0x7f2cec005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:16.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.674+0000 7f2d0a391640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2cec003dc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400a760 tx=0x7f2cec0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f897d2888ca28616 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.674+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4034630 con 0x7f2cec0039a0 2023-10-14T17:58:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.674+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf402f070 con 0x7f2cec0039a0 2023-10-14T17:58:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.674+0000 7f2d0938f640 1 -- 172.21.15.29:0/2540365330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40349b0 con 0x7f2cec0039a0 2023-10-14T17:58:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/2540365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 msgr2=0x7f2cec003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/2540365330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2cec003dc0 secure :-1 s=READY pgs=2121 cs=0 l=1 rev1=1 crypto rx=0x7f2cf400a760 tx=0x7f2cec0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0ab92640 1 -- 172.21.15.29:0/2540365330 reap_dead start 2023-10-14T17:58:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/2540365330 shutdown_connections 2023-10-14T17:58:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/2540365330 >> 172.21.15.29:0/2540365330 conn(0x7f2d040ce9b0 msgr2=0x7f2d040cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/2540365330 shutdown_connections 2023-10-14T17:58:16.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.675+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/2540365330 wait complete. 2023-10-14T17:58:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.677+0000 7f2d0c5f5640 1 Processor -- start 2023-10-14T17:58:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.677+0000 7f2d0c5f5640 1 -- start start 2023-10-14T17:58:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040c50f0 0x7f2d040c4260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d040b2770 0x7f2d040cd3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040c5890 con 0x7f2cec0039a0 2023-10-14T17:58:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040c3dd0 con 0x7f2d040b2770 2023-10-14T17:58:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0c5f5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d040c34e0 con 0x7f2d040c50f0 2023-10-14T17:58:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.678+0000 7f2d0a391640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45986/0 (socket says 172.21.15.29:45986) 2023-10-14T17:58:16.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 -- 172.21.15.29:0/764683208 learned_addr learned my addr 172.21.15.29:0/764683208 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:16.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0ab92640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d040b2770 0x7f2d040cd3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 -- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040c50f0 msgr2=0x7f2d040c4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040c50f0 0x7f2d040c4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 -- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d040b2770 msgr2=0x7f2d040cd3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d040b2770 0x7f2d040cd3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 -- 172.21.15.29:0/764683208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d040c55b0 con 0x7f2cec0039a0 2023-10-14T17:58:16.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.679+0000 7f2d0a391640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402cab0 tx=0x7f2cf4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.680+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf4001a10 con 0x7f2cec0039a0 2023-10-14T17:58:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.680+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf4034430 con 0x7f2cec0039a0 2023-10-14T17:58:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.680+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cf40347e0 con 0x7f2cec0039a0 2023-10-14T17:58:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.680+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d040c3780 con 0x7f2cec0039a0 2023-10-14T17:58:16.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.680+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d040c47a0 con 0x7f2cec0039a0 2023-10-14T17:58:16.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.681+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0404e160 con 0x7f2cec0039a0 2023-10-14T17:58:16.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.682+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2cf4031030 con 0x7f2cec0039a0 2023-10-14T17:58:16.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.683+0000 7f2d02ffd640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 0x7f2cf006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:16.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.683+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f2cf40ab1c0 con 0x7f2cec0039a0 2023-10-14T17:58:16.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.683+0000 7f2d09b90640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 0x7f2cf006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:16.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.684+0000 7f2d09b90640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 0x7f2cf006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d04157c20 tx=0x7f2cf8009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:16.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.687+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2cf4079a20 con 0x7f2cec0039a0 2023-10-14T17:58:16.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.885+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2d0406b9b0 con 0x7f2cec0039a0 2023-10-14T17:58:16.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.886+0000 7f2d02ffd640 1 -- 172.21.15.29:0/764683208 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v183) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2cf40450a0 con 0x7f2cec0039a0 2023-10-14T17:58:16.887 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:58:16.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 msgr2=0x7f2cf006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 0x7f2cf006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f2d04157c20 tx=0x7f2cf8009040 comp rx=0 tx=0).stop 2023-10-14T17:58:16.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 msgr2=0x7f2d0415cb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:16.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 secure :-1 s=READY pgs=2122 cs=0 l=1 rev1=1 crypto rx=0x7f2cf402cab0 tx=0x7f2cf4001cb0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 shutdown_connections 2023-10-14T17:58:16.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2cf006afc0 0x7f2cf006d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d040b2770 0x7f2d040cd3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2d040c50f0 0x7f2d040c4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 --2- 172.21.15.29:0/764683208 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cec0039a0 0x7f2d0415cb30 unknown :-1 s=CLOSED pgs=2122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 >> 172.21.15.29:0/764683208 conn(0x7f2d040ce9b0 msgr2=0x7f2d04069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 shutdown_connections 2023-10-14T17:58:16.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:16.889+0000 7f2d0c5f5640 1 -- 172.21.15.29:0/764683208 wait complete. 2023-10-14T17:58:16.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-14T17:58:17.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.129+0000 7f07a61e3640 1 Processor -- start 2023-10-14T17:58:17.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.129+0000 7f07a61e3640 1 -- start start 2023-10-14T17:58:17.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.130+0000 7f07a61e3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07a015aaa0 0x7f07a015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.130+0000 7f07a61e3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 0x7f07a00cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.130+0000 7f07a61e3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.130+0000 7f07a61e3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a006b810 con 0x7f07a015aaa0 2023-10-14T17:58:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f07a61e3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a015ed60 con 0x7f07a015ae80 2023-10-14T17:58:17.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f07a61e3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00707e0 con 0x7f07a00ccaa0 2023-10-14T17:58:17.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f079f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07a015aaa0 0x7f07a015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f079f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07a015aaa0 0x7f07a015e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:45992/0 (socket says 172.21.15.29:45992) 2023-10-14T17:58:17.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f079f7fe640 1 -- 172.21.15.29:0/2451182323 learned_addr learned my addr 172.21.15.29:0/2451182323 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:17.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f079ffff640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.131+0000 7f0797fff640 1 --2- 172.21.15.29:0/2451182323 >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 0x7f07a00cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079f7fe640 1 -- 172.21.15.29:0/2451182323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 msgr2=0x7f07a0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079f7fe640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079f7fe640 1 -- 172.21.15.29:0/2451182323 >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 msgr2=0x7f07a00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079f7fe640 1 --2- 172.21.15.29:0/2451182323 >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 0x7f07a00cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079f7fe640 1 -- 172.21.15.29:0/2451182323 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a015ef90 con 0x7f07a015aaa0 2023-10-14T17:58:17.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f079ffff640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:17.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.132+0000 7f0797fff640 1 --2- 172.21.15.29:0/2451182323 >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 0x7f07a00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.134+0000 7f079f7fe640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015aaa0 0x7f07a015e820 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f07a00c59c0 tx=0x7f079002b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70672fa24b35a804 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.134+0000 7f079effd640 1 -- 172.21.15.29:0/2451182323 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790031860 con 0x7f07a015aaa0 2023-10-14T17:58:17.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.134+0000 7f079effd640 1 -- 172.21.15.29:0/2451182323 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07900319c0 con 0x7f07a015aaa0 2023-10-14T17:58:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.134+0000 7f079effd640 1 -- 172.21.15.29:0/2451182323 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0790031d50 con 0x7f07a015aaa0 2023-10-14T17:58:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 -- 172.21.15.29:0/2451182323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015aaa0 msgr2=0x7f07a015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015aaa0 0x7f07a015e820 secure :-1 s=READY pgs=2123 cs=0 l=1 rev1=1 crypto rx=0x7f07a00c59c0 tx=0x7f079002b900 comp rx=0 tx=0).stop 2023-10-14T17:58:17.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 -- 172.21.15.29:0/2451182323 shutdown_connections 2023-10-14T17:58:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 --2- 172.21.15.29:0/2451182323 >> v2:172.21.15.186:3300/0 conn(0x7f07a015ae80 0x7f07a00cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 --2- 172.21.15.29:0/2451182323 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015aaa0 0x7f07a015e820 unknown :-1 s=CLOSED pgs=2123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 -- 172.21.15.29:0/2451182323 >> 172.21.15.29:0/2451182323 conn(0x7f07a00cb410 msgr2=0x7f07a00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:17.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 -- 172.21.15.29:0/2451182323 shutdown_connections 2023-10-14T17:58:17.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.135+0000 7f07a61e3640 1 -- 172.21.15.29:0/2451182323 wait complete. 2023-10-14T17:58:17.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.137+0000 7f07a61e3640 1 Processor -- start 2023-10-14T17:58:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.137+0000 7f07a61e3640 1 -- start start 2023-10-14T17:58:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a015b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07a015aaa0 0x7f07a00acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a015b860 con 0x7f07a015ae80 2023-10-14T17:58:17.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a015ce40 con 0x7f07a015aaa0 2023-10-14T17:58:17.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f07a61e3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a01e5cb0 con 0x7f07a00ccaa0 2023-10-14T17:58:17.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46004/0 (socket says 172.21.15.29:46004) 2023-10-14T17:58:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 -- 172.21.15.29:0/3521760982 learned_addr learned my addr 172.21.15.29:0/3521760982 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f0797fff640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07a015aaa0 0x7f07a00acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 -- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 msgr2=0x7f07a015b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:17.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a015b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 -- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07a015aaa0 msgr2=0x7f07a00acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07a015aaa0 0x7f07a00acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.138+0000 7f079ffff640 1 -- 172.21.15.29:0/3521760982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07900094d0 con 0x7f07a015ae80 2023-10-14T17:58:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f079ffff640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0798007db0 tx=0x7f079800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:17.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798003990 con 0x7f07a015ae80 2023-10-14T17:58:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f079800f040 con 0x7f07a015ae80 2023-10-14T17:58:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0798003d50 con 0x7f07a015ae80 2023-10-14T17:58:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a01e5f90 con 0x7f07a015ae80 2023-10-14T17:58:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.139+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a01e6830 con 0x7f07a015ae80 2023-10-14T17:58:17.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.140+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a004e160 con 0x7f07a015ae80 2023-10-14T17:58:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.141+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0798004030 con 0x7f07a015ae80 2023-10-14T17:58:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.142+0000 7f079cff9640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 0x7f078006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.142+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f0798089b40 con 0x7f07a015ae80 2023-10-14T17:58:17.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.142+0000 7f079f7fe640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 0x7f078006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.143+0000 7f079f7fe640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 0x7f078006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0790036000 tx=0x7f0790030000 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:17.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.147+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0798054990 con 0x7f07a015ae80 2023-10-14T17:58:17.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.411+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f07a01e6150 con 0x7f07a015ae80 2023-10-14T17:58:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.974+0000 7f079cff9640 1 -- 172.21.15.29:0/3521760982 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v2566) v1 ==== 168+0+0 (secure 0 0 0) 0x7f0798057fb0 con 0x7f07a015ae80 2023-10-14T17:58:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 msgr2=0x7f078006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 0x7f078006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f0790036000 tx=0x7f0790030000 comp rx=0 tx=0).stop 2023-10-14T17:58:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 msgr2=0x7f07a01e6d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 secure :-1 s=READY pgs=2124 cs=0 l=1 rev1=1 crypto rx=0x7f0798007db0 tx=0x7f079800d4a0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 shutdown_connections 2023-10-14T17:58:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07a015ae80 0x7f07a01e6d50 unknown :-1 s=CLOSED pgs=2124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f078006af90 0x7f078006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07a015aaa0 0x7f07a00acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 --2- 172.21.15.29:0/3521760982 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07a00ccaa0 0x7f07a015b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.978+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 >> 172.21.15.29:0/3521760982 conn(0x7f07a00cb410 msgr2=0x7f07a006d2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.979+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 shutdown_connections 2023-10-14T17:58:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:17.979+0000 7f07a61e3640 1 -- 172.21.15.29:0/3521760982 wait complete. 2023-10-14T17:58:18.000 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T17:58:18.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.235+0000 7f9e89567640 1 Processor -- start 2023-10-14T17:58:18.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.235+0000 7f9e89567640 1 -- start start 2023-10-14T17:58:18.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 0x7f9e840d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 0x7f9e840cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e840d0be0 0x7f9e840cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e8415d150 con 0x7f9e840d0be0 2023-10-14T17:58:18.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e840d1180 con 0x7f9e840b8480 2023-10-14T17:58:18.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.236+0000 7f9e89567640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e840d1460 con 0x7f9e8415cd70 2023-10-14T17:58:18.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e840d0be0 0x7f9e840cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e840d0be0 0x7f9e840cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46008/0 (socket says 172.21.15.29:46008) 2023-10-14T17:58:18.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 -- 172.21.15.29:0/3474770123 learned_addr learned my addr 172.21.15.29:0/3474770123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:18.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e837fe640 1 --2- 172.21.15.29:0/3474770123 >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 0x7f9e840cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 -- 172.21.15.29:0/3474770123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 msgr2=0x7f9e840d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:18.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 0x7f9e840d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 0x7f9e840d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 -- 172.21.15.29:0/3474770123 >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 msgr2=0x7f9e840cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:18.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3474770123 >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 0x7f9e840cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e88d66640 1 -- 172.21.15.29:0/3474770123 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e840d1740 con 0x7f9e840d0be0 2023-10-14T17:58:18.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e837fe640 1 --2- 172.21.15.29:0/3474770123 >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 0x7f9e840cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:18.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.237+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 0x7f9e840d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:18.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.239+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e840d0be0 0x7f9e840cdd00 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f9e84066ab0 tx=0x7f9e7c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b8729485f33cf69 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:18.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.239+0000 7f9e82ffd640 1 -- 172.21.15.29:0/3474770123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c031860 con 0x7f9e840d0be0 2023-10-14T17:58:18.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.239+0000 7f9e82ffd640 1 -- 172.21.15.29:0/3474770123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c0319c0 con 0x7f9e840d0be0 2023-10-14T17:58:18.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.239+0000 7f9e82ffd640 1 -- 172.21.15.29:0/3474770123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c031d50 con 0x7f9e840d0be0 2023-10-14T17:58:18.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 -- 172.21.15.29:0/3474770123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e840d0be0 msgr2=0x7f9e840cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:18.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e840d0be0 0x7f9e840cdd00 secure :-1 s=READY pgs=2125 cs=0 l=1 rev1=1 crypto rx=0x7f9e84066ab0 tx=0x7f9e7c02b900 comp rx=0 tx=0).stop 2023-10-14T17:58:18.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 -- 172.21.15.29:0/3474770123 shutdown_connections 2023-10-14T17:58:18.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e840d0be0 0x7f9e840cdd00 unknown :-1 s=CLOSED pgs=2125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 --2- 172.21.15.29:0/3474770123 >> v2:172.21.15.186:3300/0 conn(0x7f9e840b8480 0x7f9e840cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 --2- 172.21.15.29:0/3474770123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e8415cd70 0x7f9e840d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 -- 172.21.15.29:0/3474770123 >> 172.21.15.29:0/3474770123 conn(0x7f9e840cf570 msgr2=0x7f9e840cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:18.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 -- 172.21.15.29:0/3474770123 shutdown_connections 2023-10-14T17:58:18.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.240+0000 7f9e89567640 1 -- 172.21.15.29:0/3474770123 wait complete. 2023-10-14T17:58:18.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.242+0000 7f9e89567640 1 Processor -- start 2023-10-14T17:58:18.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.242+0000 7f9e89567640 1 -- start start 2023-10-14T17:58:18.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 0x7f9e840c72e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 0x7f9e8415fc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e84150510 con 0x7f9e8415cd70 2023-10-14T17:58:18.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8415eb60 con 0x7f9e840d0be0 2023-10-14T17:58:18.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.243+0000 7f9e89567640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e8415ee40 con 0x7f9e840b8480 2023-10-14T17:58:18.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46016/0 (socket says 172.21.15.29:46016) 2023-10-14T17:58:18.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 -- 172.21.15.29:0/3933931471 learned_addr learned my addr 172.21.15.29:0/3933931471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:18.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 -- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 msgr2=0x7f9e840c72e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:18.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e837fe640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 0x7f9e8415fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 0x7f9e840c72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 0x7f9e840c72e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 -- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 msgr2=0x7f9e8415fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 0x7f9e8415fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e88d66640 1 -- 172.21.15.29:0/3933931471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c0094d0 con 0x7f9e8415cd70 2023-10-14T17:58:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e837fe640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 0x7f9e8415fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.244+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 0x7f9e840c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e88d66640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c02b940 tx=0x7f9e7c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c03ad30 con 0x7f9e8415cd70 2023-10-14T17:58:18.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e7c0443f0 con 0x7f9e8415cd70 2023-10-14T17:58:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7c0447a0 con 0x7f9e8415cd70 2023-10-14T17:58:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e8415f120 con 0x7f9e8415cd70 2023-10-14T17:58:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.245+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e8415f6e0 con 0x7f9e8415cd70 2023-10-14T17:58:18.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.247+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e8415f8d0 con 0x7f9e8415cd70 2023-10-14T17:58:18.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.248+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9e7c044900 con 0x7f9e8415cd70 2023-10-14T17:58:18.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.249+0000 7f9e80ff9640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 0x7f9e6406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:18.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.249+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f9e7c0ac0a0 con 0x7f9e8415cd70 2023-10-14T17:58:18.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.249+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 0x7f9e6406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:18.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.250+0000 7f9e83fff640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 0x7f9e6406d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e74006fd0 tx=0x7f9e74008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:18.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.254+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e7c077020 con 0x7f9e8415cd70 2023-10-14T17:58:18.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.452+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9e840d1d00 con 0x7f9e8415cd70 2023-10-14T17:58:18.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.460+0000 7f9e80ff9640 1 -- 172.21.15.29:0/3933931471 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v64)=0 v64) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9e7c07a640 con 0x7f9e8415cd70 2023-10-14T17:58:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 msgr2=0x7f9e6406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 0x7f9e6406d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9e74006fd0 tx=0x7f9e74008040 comp rx=0 tx=0).stop 2023-10-14T17:58:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 msgr2=0x7f9e8414ffd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 secure :-1 s=READY pgs=2126 cs=0 l=1 rev1=1 crypto rx=0x7f9e7c02b940 tx=0x7f9e7c0046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 shutdown_connections 2023-10-14T17:58:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e8415cd70 0x7f9e8414ffd0 unknown :-1 s=CLOSED pgs=2126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f9e6406afc0 0x7f9e6406d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e840d0be0 0x7f9e8415fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 --2- 172.21.15.29:0/3933931471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e840b8480 0x7f9e840c72e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 >> 172.21.15.29:0/3933931471 conn(0x7f9e840cf570 msgr2=0x7f9e840b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 shutdown_connections 2023-10-14T17:58:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:18.463+0000 7f9e89567640 1 -- 172.21.15.29:0/3933931471 wait complete. 2023-10-14T17:58:18.485 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T17:58:18.485 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:58:18.606 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T17:58:18.740 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T17:58:18.740 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:18.741 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:18.804 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:58:18.805 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:58:18.805 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:18.864 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:58:18.865 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:18.865 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:18.929 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T17:58:18.929 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T17:58:18.930 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:18.992 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T17:58:18.993 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T17:58:18.993 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.220+0000 7f916cfac640 1 Processor -- start 2023-10-14T17:58:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.220+0000 7f916cfac640 1 -- start start 2023-10-14T17:58:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f916cfac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f916806fb50 0x7f91680cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f916cfac640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680ca560 con 0x7f916806c9e0 2023-10-14T17:58:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f916cfac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91680c9950 con 0x7f91680c9060 2023-10-14T17:58:19.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f916cfac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91680c9c30 con 0x7f916806fb50 2023-10-14T17:58:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f9166ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f916806fb50 0x7f91680cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f9167fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f916806c9e0 0x7f91680ccde0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33540/0 (socket says 172.21.15.29:33540) 2023-10-14T17:58:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.222+0000 7f9167fff640 1 -- 172.21.15.29:0/2433550710 learned_addr learned my addr 172.21.15.29:0/2433550710 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.223+0000 7f9166ffd640 1 -- 172.21.15.29:0/2433550710 >> v1:172.21.15.186:6789/0 conn(0x7f91680c9060 legacy=0x7f916806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.223+0000 7f9166ffd640 1 -- 172.21.15.29:0/2433550710 >> v1:172.21.15.29:6789/0 conn(0x7f916806c9e0 legacy=0x7f91680ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.223+0000 7f9166ffd640 1 -- 172.21.15.29:0/2433550710 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91680c9f10 con 0x7f916806fb50 2023-10-14T17:58:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.224+0000 7f9166ffd640 1 --2- 172.21.15.29:0/2433550710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806fb50 0x7f91680cc8a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097a0 tx=0x7f915c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82930c2c7dbe9bd8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.224+0000 7f91667fc640 1 -- 172.21.15.29:0/2433550710 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c031830 con 0x7f916806fb50 2023-10-14T17:58:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.224+0000 7f91667fc640 1 -- 172.21.15.29:0/2433550710 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f915c031990 con 0x7f916806fb50 2023-10-14T17:58:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.224+0000 7f91667fc640 1 -- 172.21.15.29:0/2433550710 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f915c031d20 con 0x7f916806fb50 2023-10-14T17:58:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 -- 172.21.15.29:0/2433550710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806fb50 msgr2=0x7f91680cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 --2- 172.21.15.29:0/2433550710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806fb50 0x7f91680cc8a0 secure :-1 s=READY pgs=2128 cs=0 l=1 rev1=1 crypto rx=0x7f915c0097a0 tx=0x7f915c02b8d0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 -- 172.21.15.29:0/2433550710 shutdown_connections 2023-10-14T17:58:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 --2- 172.21.15.29:0/2433550710 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806fb50 0x7f91680cc8a0 unknown :-1 s=CLOSED pgs=2128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 -- 172.21.15.29:0/2433550710 >> 172.21.15.29:0/2433550710 conn(0x7f91680ce6a0 msgr2=0x7f91680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 -- 172.21.15.29:0/2433550710 shutdown_connections 2023-10-14T17:58:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.225+0000 7f916cfac640 1 -- 172.21.15.29:0/2433550710 wait complete. 2023-10-14T17:58:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.227+0000 7f916cfac640 1 Processor -- start 2023-10-14T17:58:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.227+0000 7f916cfac640 1 -- start start 2023-10-14T17:58:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f916806fb50 0x7f91681faa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91680c9060 0x7f91680c8180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680c4300 con 0x7f916806c9e0 2023-10-14T17:58:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91680c4c60 con 0x7f91680c9060 2023-10-14T17:58:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f916cfac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91681fafa0 con 0x7f916806fb50 2023-10-14T17:58:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f91677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f91677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46026/0 (socket says 172.21.15.29:46026) 2023-10-14T17:58:19.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.228+0000 7f91677fe640 1 -- 172.21.15.29:0/3809356682 learned_addr learned my addr 172.21.15.29:0/3809356682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:19.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f9166ffd640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f916806fb50 0x7f91681faa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f9167fff640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91680c9060 0x7f91680c8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 -- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f916806fb50 msgr2=0x7f91681faa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f916806fb50 0x7f91681faa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 -- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91680c9060 msgr2=0x7f91680c8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91680c9060 0x7f91680c8180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 -- 172.21.15.29:0/3809356682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f915c009400 con 0x7f916806c9e0 2023-10-14T17:58:19.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f91677fe640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f915800b3f0 tx=0x7f915800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158011830 con 0x7f916806c9e0 2023-10-14T17:58:19.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9158011990 con 0x7f916806c9e0 2023-10-14T17:58:19.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9158011d40 con 0x7f916806c9e0 2023-10-14T17:58:19.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.229+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91680c3680 con 0x7f916806c9e0 2023-10-14T17:58:19.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.230+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91680c3c40 con 0x7f916806c9e0 2023-10-14T17:58:19.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.231+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91680c4440 con 0x7f916806c9e0 2023-10-14T17:58:19.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.232+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f915801a400 con 0x7f916806c9e0 2023-10-14T17:58:19.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.233+0000 7f914ffff640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 0x7f915406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.233+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f9158089650 con 0x7f916806c9e0 2023-10-14T17:58:19.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.233+0000 7f9166ffd640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 0x7f915406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.234+0000 7f9166ffd640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 0x7f915406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f915c036000 tx=0x7f915c008030 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.237+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91680c4440 con 0x7f916806c9e0 2023-10-14T17:58:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.475+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f91680c4440 con 0x7f916806c9e0 2023-10-14T17:58:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.476+0000 7f914ffff640 1 -- 172.21.15.29:0/3809356682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 85 v85) v1 ==== 94+0+3706 (secure 0 0 0) 0x7f9158015e90 con 0x7f916806c9e0 2023-10-14T17:58:19.478 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:19.478 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":85,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6545,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1094015889","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":1094015889},{"type":"v1","addr":"172.21.15.186:6835","nonce":1094015889}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":6551,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3482166745","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3482166745},{"type":"v1","addr":"172.21.15.29:6837","nonce":3482166745}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":78},{"gid":7345,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/4263703518","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":4263703518},{"type":"v1","addr":"172.21.15.186:6834","nonce":4263703518}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":77},{"gid":10023,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3732843294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3732843294},{"type":"v1","addr":"172.21.15.29:6836","nonce":3732843294}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":85}],"filesystems":[]} 2023-10-14T17:58:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 85 2023-10-14T17:58:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 msgr2=0x7f915406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 0x7f915406d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f915c036000 tx=0x7f915c008030 comp rx=0 tx=0).stop 2023-10-14T17:58:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 msgr2=0x7f91680c4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 secure :-1 s=READY pgs=2129 cs=0 l=1 rev1=1 crypto rx=0x7f915800b3f0 tx=0x7f915800b8b0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 shutdown_connections 2023-10-14T17:58:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f915406af90 0x7f915406d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91680c9060 0x7f91680c8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f916806fb50 0x7f91681faa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 --2- 172.21.15.29:0/3809356682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f916806c9e0 0x7f91680c4720 unknown :-1 s=CLOSED pgs=2129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 >> 172.21.15.29:0/3809356682 conn(0x7f91680ce6a0 msgr2=0x7f91680b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 shutdown_connections 2023-10-14T17:58:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.479+0000 7f916cfac640 1 -- 172.21.15.29:0/3809356682 wait complete. 2023-10-14T17:58:19.499 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T17:58:19.499 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T17:58:19.500 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T17:58:19.500 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:58:19.501 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T17:58:19.501 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T17:58:19.501 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T17:58:19.501 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:58:19.502 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T17:58:19.502 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T17:58:19.502 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T17:58:19.503 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:58:19.503 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T17:58:19.503 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T17:58:19.504 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T17:58:19.504 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T17:58:19.524 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T17:58:19.524 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T17:58:19.525 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T17:58:19.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T17:58:19.528 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T17:58:19.528 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T17:58:19.529 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T17:58:19.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T17:58:19.576 INFO:tasks.ceph.mds.a:Started 2023-10-14T17:58:19.577 DEBUG:teuthology.parallel:result is None 2023-10-14T17:58:19.577 INFO:tasks.ceph.mds.d:Started 2023-10-14T17:58:19.578 DEBUG:teuthology.parallel:result is None 2023-10-14T17:58:19.584 INFO:tasks.ceph.mds.c:Started 2023-10-14T17:58:19.584 DEBUG:teuthology.parallel:result is None 2023-10-14T17:58:19.585 INFO:tasks.ceph.mds.b:Started 2023-10-14T17:58:19.585 DEBUG:teuthology.parallel:result is None 2023-10-14T17:58:19.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T17:58:19.740 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T17:58:19.743 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T17:58:19.751 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T17:58:19.754 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T17:58:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.784+0000 7fc65acad640 1 Processor -- start 2023-10-14T17:58:19.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.784+0000 7fc65acad640 1 -- start start 2023-10-14T17:58:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.785+0000 7fc65acad640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc65415b320 0x7fc6540cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.785+0000 7fc65acad640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65415b9d0 con 0x7fc65406c040 2023-10-14T17:58:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.785+0000 7fc65acad640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65415bcb0 con 0x7fc6540ad970 2023-10-14T17:58:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.785+0000 7fc65acad640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc65415b7d0 con 0x7fc65415b320 2023-10-14T17:58:19.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc659cab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc65406c040 0x7fc65406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33556/0 (socket says 172.21.15.29:33556) 2023-10-14T17:58:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc659cab640 1 -- 172.21.15.29:0/294297688 learned_addr learned my addr 172.21.15.29:0/294297688 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:19.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc65a4ac640 1 --2- 172.21.15.29:0/294297688 >> v2:172.21.15.29:3300/0 conn(0x7fc65415b320 0x7fc6540cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418174376 0 0) 0x7fc65415b9d0 con 0x7fc65406c040 2023-10-14T17:58:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc640003520 con 0x7fc65406c040 2023-10-14T17:58:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2750910843 0 0) 0x7fc65415bcb0 con 0x7fc6540ad970 2023-10-14T17:58:19.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc65a4ac640 1 -- 172.21.15.29:0/294297688 >> v1:172.21.15.186:6789/0 conn(0x7fc6540ad970 legacy=0x7fc6540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc65a4ac640 1 -- 172.21.15.29:0/294297688 >> v1:172.21.15.29:6789/0 conn(0x7fc65406c040 legacy=0x7fc65406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.786+0000 7fc65a4ac640 1 -- 172.21.15.29:0/294297688 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc65415bf90 con 0x7fc65415b320 2023-10-14T17:58:19.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.787+0000 7fc65a4ac640 1 --2- 172.21.15.29:0/294297688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc65415b320 0x7fc6540cd180 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009dc0 tx=0x7fc64c02bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8abf37a674c2a37e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c02cce0 con 0x7fc65415b320 2023-10-14T17:58:19.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc64c02ce40 con 0x7fc65415b320 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc658ca9640 1 -- 172.21.15.29:0/294297688 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc64c034640 con 0x7fc65415b320 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 -- 172.21.15.29:0/294297688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc65415b320 msgr2=0x7fc6540cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 --2- 172.21.15.29:0/294297688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc65415b320 0x7fc6540cd180 secure :-1 s=READY pgs=2137 cs=0 l=1 rev1=1 crypto rx=0x7fc64c009dc0 tx=0x7fc64c02bf30 comp rx=0 tx=0).stop 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 -- 172.21.15.29:0/294297688 shutdown_connections 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 --2- 172.21.15.29:0/294297688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc65415b320 0x7fc6540cd180 unknown :-1 s=CLOSED pgs=2137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 -- 172.21.15.29:0/294297688 >> 172.21.15.29:0/294297688 conn(0x7fc6540ce6a0 msgr2=0x7fc6540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:19.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 -- 172.21.15.29:0/294297688 shutdown_connections 2023-10-14T17:58:19.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.788+0000 7fc65acad640 1 -- 172.21.15.29:0/294297688 wait complete. 2023-10-14T17:58:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.790+0000 7fc65acad640 1 Processor -- start 2023-10-14T17:58:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.790+0000 7fc65acad640 1 -- start start 2023-10-14T17:58:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc65406c040 0x7fc65414db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc65415b320 0x7fc6540a85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540af8a0 con 0x7fc6540ad970 2023-10-14T17:58:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540afb80 con 0x7fc65406c040 2023-10-14T17:58:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc65acad640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6540afe60 con 0x7fc65415b320 2023-10-14T17:58:19.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.791+0000 7fc6594aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46070/0 (socket says 172.21.15.29:46070) 2023-10-14T17:58:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 -- 172.21.15.29:0/2304799953 learned_addr learned my addr 172.21.15.29:0/2304799953 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc659cab640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc65406c040 0x7fc65414db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc65a4ac640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc65415b320 0x7fc6540a85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 -- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc65415b320 msgr2=0x7fc6540a85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc65415b320 0x7fc6540a85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 -- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc65406c040 msgr2=0x7fc65414db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc65406c040 0x7fc65414db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:19.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 -- 172.21.15.29:0/2304799953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc64c009a70 con 0x7fc6540ad970 2023-10-14T17:58:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.792+0000 7fc6594aa640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c520 tx=0x7fc64400c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.793+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6440035f0 con 0x7fc6540ad970 2023-10-14T17:58:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.793+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc644003770 con 0x7fc6540ad970 2023-10-14T17:58:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.793+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc644003b20 con 0x7fc6540ad970 2023-10-14T17:58:19.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.793+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6540b0140 con 0x7fc6540ad970 2023-10-14T17:58:19.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.793+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6540b0e80 con 0x7fc6540ad970 2023-10-14T17:58:19.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.794+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6540b0660 con 0x7fc6540ad970 2023-10-14T17:58:19.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.795+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc644003c80 con 0x7fc6540ad970 2023-10-14T17:58:19.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.796+0000 7fc64a7fc640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 0x7fc63806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:19.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.796+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(183..183 src has 1..183) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fc644088de0 con 0x7fc6540ad970 2023-10-14T17:58:19.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.796+0000 7fc659cab640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 0x7fc63806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:19.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.797+0000 7fc659cab640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 0x7fc63806d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc650005880 tx=0x7fc650009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:19.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:19.801+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6540b0660 con 0x7fc6540ad970 2023-10-14T17:58:20.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:20.017+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fc6540b0660 con 0x7fc6540ad970 2023-10-14T17:58:21.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.017+0000 7fc64a7fc640 1 -- 172.21.15.29:0/2304799953 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v188) v1 ==== 97+0+0 (secure 0 0 0) 0x7fc6540b0660 con 0x7fc6540ad970 2023-10-14T17:58:21.019 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T17:58:21.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 msgr2=0x7fc63806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 0x7fc63806d4a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fc650005880 tx=0x7fc650009040 comp rx=0 tx=0).stop 2023-10-14T17:58:21.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 msgr2=0x7fc6540b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 secure :-1 s=READY pgs=2138 cs=0 l=1 rev1=1 crypto rx=0x7fc64400c520 tx=0x7fc64400c9e0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 shutdown_connections 2023-10-14T17:58:21.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc65415b320 0x7fc6540a85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6540ad970 0x7fc6540b0940 unknown :-1 s=CLOSED pgs=2138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc63806afe0 0x7fc63806d4a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 --2- 172.21.15.29:0/2304799953 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc65406c040 0x7fc65414db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 >> 172.21.15.29:0/2304799953 conn(0x7fc6540ce6a0 msgr2=0x7fc6540c7e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:21.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 shutdown_connections 2023-10-14T17:58:21.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.021+0000 7fc65acad640 1 -- 172.21.15.29:0/2304799953 wait complete. 2023-10-14T17:58:21.043 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T17:58:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.263+0000 7fed4a7e9640 1 Processor -- start 2023-10-14T17:58:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.263+0000 7fed4a7e9640 1 -- start start 2023-10-14T17:58:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed4a7e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed4415b320 0x7fed440cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed4a7e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4415b9d0 con 0x7fed4406c040 2023-10-14T17:58:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed4a7e9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed4415bcb0 con 0x7fed440ad970 2023-10-14T17:58:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed4a7e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed4415b7d0 con 0x7fed4415b320 2023-10-14T17:58:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed497e7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fed4406c040 0x7fed4406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45020/0 (socket says 172.21.15.29:45020) 2023-10-14T17:58:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.265+0000 7fed497e7640 1 -- 172.21.15.29:0/3263638859 learned_addr learned my addr 172.21.15.29:0/3263638859 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed49fe8640 1 --2- 172.21.15.29:0/3263638859 >> v2:172.21.15.186:3300/0 conn(0x7fed4415b320 0x7fed440cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035504520 0 0) 0x7fed4415b9d0 con 0x7fed4406c040 2023-10-14T17:58:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed30003520 con 0x7fed4406c040 2023-10-14T17:58:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3633589278 0 0) 0x7fed30003520 con 0x7fed4406c040 2023-10-14T17:58:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 >> v2:172.21.15.186:3300/0 conn(0x7fed4415b320 msgr2=0x7fed440cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 --2- 172.21.15.29:0/3263638859 >> v2:172.21.15.186:3300/0 conn(0x7fed4415b320 0x7fed440cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 >> v1:172.21.15.186:6789/0 conn(0x7fed440ad970 legacy=0x7fed440cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed4415bf90 con 0x7fed4406c040 2023-10-14T17:58:21.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fed40001670 con 0x7fed4406c040 2023-10-14T17:58:21.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 >> v1:172.21.15.29:6789/0 conn(0x7fed4406c040 legacy=0x7fed4406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed300039a0 0x7fed30003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed300047c0 0x7fed30004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 0x7fed3000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4415b7d0 con 0x7fed30005180 2023-10-14T17:58:21.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3000c050 con 0x7fed300039a0 2023-10-14T17:58:21.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.266+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed3000c210 con 0x7fed300047c0 2023-10-14T17:58:21.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 0x7fed3000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed497e7640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed300039a0 0x7fed30003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed48fe6640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed300047c0 0x7fed30004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 -- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed300047c0 msgr2=0x7fed30004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:21.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed300047c0 0x7fed30004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 -- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed300039a0 msgr2=0x7fed30003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:21.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed300039a0 0x7fed30003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed49fe8640 1 -- 172.21.15.29:0/3263638859 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed30003520 con 0x7fed30005180 2023-10-14T17:58:21.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.267+0000 7fed48fe6640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed300047c0 0x7fed30004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:21.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.268+0000 7fed49fe8640 1 -- 172.21.15.29:0/3263638859 reap_dead start 2023-10-14T17:58:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.268+0000 7fed49fe8640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 0x7fed3000b980 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009fc0 tx=0x7fed300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b59eb8aeb88a213 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c02cc60 con 0x7fed30005180 2023-10-14T17:58:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed3c00f040 con 0x7fed30005180 2023-10-14T17:58:21.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed3bfff640 1 -- 172.21.15.29:0/3263638859 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed3c003990 con 0x7fed30005180 2023-10-14T17:58:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 -- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 msgr2=0x7fed3000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 0x7fed3000b980 secure :-1 s=READY pgs=2140 cs=0 l=1 rev1=1 crypto rx=0x7fed3c009fc0 tx=0x7fed300057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 -- 172.21.15.29:0/3263638859 shutdown_connections 2023-10-14T17:58:21.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/3263638859 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed30005180 0x7fed3000b980 unknown :-1 s=CLOSED pgs=2140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 -- 172.21.15.29:0/3263638859 >> 172.21.15.29:0/3263638859 conn(0x7fed440ce6a0 msgr2=0x7fed440ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 -- 172.21.15.29:0/3263638859 shutdown_connections 2023-10-14T17:58:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.269+0000 7fed4a7e9640 1 -- 172.21.15.29:0/3263638859 wait complete. 2023-10-14T17:58:21.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.271+0000 7fed4a7e9640 1 Processor -- start 2023-10-14T17:58:21.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.271+0000 7fed4a7e9640 1 -- start start 2023-10-14T17:58:21.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 0x7fed441479f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed440ade50 0x7fed440b3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed440afa80 con 0x7fed4414d660 2023-10-14T17:58:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed441476e0 con 0x7fed30005180 2023-10-14T17:58:21.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.272+0000 7fed4a7e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed440ae7a0 con 0x7fed440ade50 2023-10-14T17:58:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33904/0 (socket says 172.21.15.29:33904) 2023-10-14T17:58:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 -- 172.21.15.29:0/477025611 learned_addr learned my addr 172.21.15.29:0/477025611 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed497e7640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 0x7fed441479f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed49fe8640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed440ade50 0x7fed440b3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 -- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed440ade50 msgr2=0x7fed440b3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed440ade50 0x7fed440b3460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 -- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 msgr2=0x7fed441479f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 0x7fed441479f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed48fe6640 1 -- 172.21.15.29:0/477025611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed440aea80 con 0x7fed4414d660 2023-10-14T17:58:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.273+0000 7fed497e7640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 0x7fed441479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed48fe6640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fed3400d540 tx=0x7fed3400da00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed34005480 con 0x7fed4414d660 2023-10-14T17:58:21.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed34003610 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed34003a00 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed440aed60 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.274+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed440ae300 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.275+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed4404e9b0 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.276+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed34003b60 con 0x7fed4414d660 2023-10-14T17:58:21.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.277+0000 7fed39ffb640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 0x7fed2806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.277+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fed34089490 con 0x7fed4414d660 2023-10-14T17:58:21.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.277+0000 7fed497e7640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 0x7fed2806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.278+0000 7fed497e7640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 0x7fed2806d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed40001eb0 tx=0x7fed40000f50 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.281+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed340545f0 con 0x7fed4414d660 2023-10-14T17:58:21.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.548+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fed440cc550 con 0x7fed4414d660 2023-10-14T17:58:21.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.549+0000 7fed39ffb640 1 -- 172.21.15.29:0/477025611 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v42) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fed34057c10 con 0x7fed4414d660 2023-10-14T17:58:21.550 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:21.550 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:58:21.551 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T17:58:21.551 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.551 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T17:58:21.551 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T17:58:21.551 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.552 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.552 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:58:21.552 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:58:21.552 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.552 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.553 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.553 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.553 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T17:58:21.553 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T17:58:21.553 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.554 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.554 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:58:21.554 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:58:21.554 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.554 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T17:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T17:58:21.555 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.556 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.556 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:58:21.556 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:58:21.556 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.556 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.557 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.557 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.557 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T17:58:21.557 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T17:58:21.557 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.558 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.558 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:58:21.558 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:58:21.558 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.558 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.559 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.559 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.559 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T17:58:21.559 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T17:58:21.559 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.560 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.560 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.560 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.560 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.560 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.561 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T17:58:21.561 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T17:58:21.561 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.561 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.562 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.562 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.562 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.562 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.562 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.563 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T17:58:21.563 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T17:58:21.563 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.563 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.563 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.564 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.564 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.564 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.564 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.564 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T17:58:21.565 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T17:58:21.565 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.565 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.565 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.565 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.566 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.566 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.566 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.566 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T17:58:21.566 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T17:58:21.567 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.567 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.567 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.567 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.567 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.568 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.569 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.569 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.569 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.569 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.569 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.570 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T17:58:21.570 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T17:58:21.570 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.570 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.570 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.571 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.571 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.571 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.571 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.571 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.572 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T17:58:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T17:58:21.573 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.574 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.574 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T17:58:21.574 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T17:58:21.574 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.574 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.575 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:58:21.576 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:58:21.576 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:58:21.576 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.576 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.576 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.577 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T17:58:21.577 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T17:58:21.577 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.577 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.577 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:58:21.578 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:58:21.578 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:58:21.578 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.578 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.578 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.579 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T17:58:21.579 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T17:58:21.579 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.579 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T17:58:21.579 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T17:58:21.580 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T17:58:21.580 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T17:58:21.580 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.580 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.580 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.581 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T17:58:21.581 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T17:58:21.581 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.581 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:58:21.581 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.582 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T17:58:21.583 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T17:58:21.583 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.583 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:58:21.583 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:58:21.583 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.584 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.584 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T17:58:21.584 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:21.584 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T17:58:21.584 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.585 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:21.586 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T17:58:21.586 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:58:21.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 msgr2=0x7fed2806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 0x7fed2806d480 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fed40001eb0 tx=0x7fed40000f50 comp rx=0 tx=0).stop 2023-10-14T17:58:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 msgr2=0x7fed441471a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 secure :-1 s=READY pgs=2141 cs=0 l=1 rev1=1 crypto rx=0x7fed3400d540 tx=0x7fed3400da00 comp rx=0 tx=0).stop 2023-10-14T17:58:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 shutdown_connections 2023-10-14T17:58:21.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed440ade50 0x7fed440b3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed4414d660 0x7fed441471a0 unknown :-1 s=CLOSED pgs=2141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fed2806afc0 0x7fed2806d480 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 --2- 172.21.15.29:0/477025611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed30005180 0x7fed441479f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.551+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 >> 172.21.15.29:0/477025611 conn(0x7fed440ce6a0 msgr2=0x7fed440cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:21.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.552+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 shutdown_connections 2023-10-14T17:58:21.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.552+0000 7fed4a7e9640 1 -- 172.21.15.29:0/477025611 wait complete. 2023-10-14T17:58:21.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:21.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.825+0000 7fdd6f59e640 1 Processor -- start 2023-10-14T17:58:21.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.825+0000 7fdd6f59e640 1 -- start start 2023-10-14T17:58:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd6f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdd70159950 0x7fdd7015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd6f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd70070450 0x7fdd700708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd6f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd7015aef0 con 0x7fdd7006c100 2023-10-14T17:58:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd6f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd70070e20 con 0x7fdd70070450 2023-10-14T17:58:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd6f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd7015e490 con 0x7fdd70159950 2023-10-14T17:58:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd70070450 0x7fdd700708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd70070450 0x7fdd700708e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33914/0 (socket says 172.21.15.29:33914) 2023-10-14T17:58:21.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.827+0000 7fdd67fff640 1 -- 172.21.15.29:0/4238831880 learned_addr learned my addr 172.21.15.29:0/4238831880 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.828+0000 7fdd6e59c640 1 --2- 172.21.15.29:0/4238831880 >> v2:172.21.15.186:3300/0 conn(0x7fdd70159950 0x7fdd7015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.828+0000 7fdd67fff640 1 -- 172.21.15.29:0/4238831880 >> v2:172.21.15.186:3300/0 conn(0x7fdd70159950 msgr2=0x7fdd7015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.828+0000 7fdd67fff640 1 --2- 172.21.15.29:0/4238831880 >> v2:172.21.15.186:3300/0 conn(0x7fdd70159950 0x7fdd7015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.828+0000 7fdd67fff640 1 -- 172.21.15.29:0/4238831880 >> v1:172.21.15.186:6789/0 conn(0x7fdd7006c100 legacy=0x7fdd7006c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:21.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.828+0000 7fdd67fff640 1 -- 172.21.15.29:0/4238831880 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd7015d910 con 0x7fdd70070450 2023-10-14T17:58:21.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.829+0000 7fdd67fff640 1 --2- 172.21.15.29:0/4238831880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70070450 0x7fdd700708e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fdd700ca950 tx=0x7fdd5802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7a1cce020d3137 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.829+0000 7fdd6dd9b640 1 -- 172.21.15.29:0/4238831880 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd580317e0 con 0x7fdd70070450 2023-10-14T17:58:21.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6dd9b640 1 -- 172.21.15.29:0/4238831880 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd58031940 con 0x7fdd70070450 2023-10-14T17:58:21.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6dd9b640 1 -- 172.21.15.29:0/4238831880 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58031cd0 con 0x7fdd70070450 2023-10-14T17:58:21.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 -- 172.21.15.29:0/4238831880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70070450 msgr2=0x7fdd700708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/4238831880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70070450 0x7fdd700708e0 secure :-1 s=READY pgs=2142 cs=0 l=1 rev1=1 crypto rx=0x7fdd700ca950 tx=0x7fdd5802b880 comp rx=0 tx=0).stop 2023-10-14T17:58:21.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 -- 172.21.15.29:0/4238831880 shutdown_connections 2023-10-14T17:58:21.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/4238831880 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70070450 0x7fdd700708e0 unknown :-1 s=CLOSED pgs=2142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/4238831880 >> v2:172.21.15.186:3300/0 conn(0x7fdd70159950 0x7fdd7015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.830+0000 7fdd6f59e640 1 -- 172.21.15.29:0/4238831880 >> 172.21.15.29:0/4238831880 conn(0x7fdd700ce6a0 msgr2=0x7fdd700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.831+0000 7fdd6f59e640 1 -- 172.21.15.29:0/4238831880 shutdown_connections 2023-10-14T17:58:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.831+0000 7fdd6f59e640 1 -- 172.21.15.29:0/4238831880 wait complete. 2023-10-14T17:58:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.832+0000 7fdd6f59e640 1 Processor -- start 2023-10-14T17:58:21.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.832+0000 7fdd6f59e640 1 -- start start 2023-10-14T17:58:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd7006c100 0x7fdd70148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd70070450 0x7fdd70155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd701494e0 con 0x7fdd70159950 2023-10-14T17:58:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70155b70 con 0x7fdd7006c100 2023-10-14T17:58:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.833+0000 7fdd6f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd70155e50 con 0x7fdd70070450 2023-10-14T17:58:21.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33920/0 (socket says 172.21.15.29:33920) 2023-10-14T17:58:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 -- 172.21.15.29:0/2327654507 learned_addr learned my addr 172.21.15.29:0/2327654507 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6e59c640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd7006c100 0x7fdd70148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd67fff640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd70070450 0x7fdd70155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 -- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd70070450 msgr2=0x7fdd70155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd70070450 0x7fdd70155170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 -- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd7006c100 msgr2=0x7fdd70148fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd7006c100 0x7fdd70148fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:21.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.834+0000 7fdd6ed9d640 1 -- 172.21.15.29:0/2327654507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd580094d0 con 0x7fdd70159950 2023-10-14T17:58:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd6ed9d640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c780 tx=0x7fdd6800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd68004b60 con 0x7fdd70159950 2023-10-14T17:58:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd6800ce90 con 0x7fdd70159950 2023-10-14T17:58:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6800a850 con 0x7fdd70159950 2023-10-14T17:58:21.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd70156130 con 0x7fdd70159950 2023-10-14T17:58:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.835+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd701566f0 con 0x7fdd70159950 2023-10-14T17:58:21.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.836+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7004e9b0 con 0x7fdd70159950 2023-10-14T17:58:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.837+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd6800a9b0 con 0x7fdd70159950 2023-10-14T17:58:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.838+0000 7fdd66ffd640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 0x7fdd5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.838+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fdd680896a0 con 0x7fdd70159950 2023-10-14T17:58:21.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.839+0000 7fdd6e59c640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 0x7fdd5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.839+0000 7fdd6e59c640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 0x7fdd5006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd60006fd0 tx=0x7fdd60008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:21.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:21.843+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd68054910 con 0x7fdd70159950 2023-10-14T17:58:22.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.044+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd700b45f0 con 0x7fdd70159950 2023-10-14T17:58:22.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.046+0000 7fdd66ffd640 1 -- 172.21.15.29:0/2327654507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v188) v1 ==== 74+0+13041 (secure 0 0 0) 0x7fdd68057f30 con 0x7fdd70159950 2023-10-14T17:58:22.048 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:22.048 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":188,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:21.015574+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":21,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":176,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:22.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 msgr2=0x7fdd5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 0x7fdd5006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd60006fd0 tx=0x7fdd60008040 comp rx=0 tx=0).stop 2023-10-14T17:58:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 msgr2=0x7fdd700c9bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 secure :-1 s=READY pgs=2143 cs=0 l=1 rev1=1 crypto rx=0x7fdd6800c780 tx=0x7fdd6800cc40 comp rx=0 tx=0).stop 2023-10-14T17:58:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 shutdown_connections 2023-10-14T17:58:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd70159950 0x7fdd700c9bc0 unknown :-1 s=CLOSED pgs=2143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd70070450 0x7fdd70155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fdd5006af90 0x7fdd5006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 --2- 172.21.15.29:0/2327654507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd7006c100 0x7fdd70148fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 >> 172.21.15.29:0/2327654507 conn(0x7fdd700ce6a0 msgr2=0x7fdd700cd4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 shutdown_connections 2023-10-14T17:58:22.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.049+0000 7fdd6f59e640 1 -- 172.21.15.29:0/2327654507 wait complete. 2023-10-14T17:58:22.071 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T17:58:22.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T17:58:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.289+0000 7f2bb7fff640 1 Processor -- start 2023-10-14T17:58:22.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.289+0000 7f2bb7fff640 1 -- start start 2023-10-14T17:58:22.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.290+0000 7f2bb7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2bb80b1ca0 0x7f2bb80d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.290+0000 7f2bb7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb80b2080 0x7f2bb80ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.290+0000 7f2bb7fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bb80d0970 con 0x7f2bb80ce960 2023-10-14T17:58:22.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.290+0000 7f2bb7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80d0ea0 con 0x7f2bb80b2080 2023-10-14T17:58:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.290+0000 7f2bb7fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bb80d0bc0 con 0x7f2bb80b1ca0 2023-10-14T17:58:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb80b2080 0x7f2bb80ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bb80b2080 0x7f2bb80ce420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33924/0 (socket says 172.21.15.29:33924) 2023-10-14T17:58:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 -- 172.21.15.29:0/3671884545 learned_addr learned my addr 172.21.15.29:0/3671884545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:22.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb6ffd640 1 --2- 172.21.15.29:0/3671884545 >> v2:172.21.15.186:3300/0 conn(0x7f2bb80b1ca0 0x7f2bb80d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 -- 172.21.15.29:0/3671884545 >> v2:172.21.15.186:3300/0 conn(0x7f2bb80b1ca0 msgr2=0x7f2bb80d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 --2- 172.21.15.29:0/3671884545 >> v2:172.21.15.186:3300/0 conn(0x7f2bb80b1ca0 0x7f2bb80d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.291+0000 7f2bb67fc640 1 -- 172.21.15.29:0/3671884545 >> v1:172.21.15.186:6789/0 conn(0x7f2bb80ce960 legacy=0x7f2bb80b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.292+0000 7f2bb67fc640 1 -- 172.21.15.29:0/3671884545 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bb80cd940 con 0x7f2bb80b2080 2023-10-14T17:58:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.293+0000 7f2bb67fc640 1 --2- 172.21.15.29:0/3671884545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb80ce420 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8067210 tx=0x7f2ba002ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ab0dd9bb8894d50 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.293+0000 7f2bb5ffb640 1 -- 172.21.15.29:0/3671884545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba002f850 con 0x7f2bb80b2080 2023-10-14T17:58:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.293+0000 7f2bb5ffb640 1 -- 172.21.15.29:0/3671884545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba002f9b0 con 0x7f2bb80b2080 2023-10-14T17:58:22.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.293+0000 7f2bb5ffb640 1 -- 172.21.15.29:0/3671884545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba002fd40 con 0x7f2bb80b2080 2023-10-14T17:58:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 -- 172.21.15.29:0/3671884545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 msgr2=0x7f2bb80ce420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/3671884545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb80ce420 secure :-1 s=READY pgs=2144 cs=0 l=1 rev1=1 crypto rx=0x7f2bb8067210 tx=0x7f2ba002ba30 comp rx=0 tx=0).stop 2023-10-14T17:58:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 -- 172.21.15.29:0/3671884545 shutdown_connections 2023-10-14T17:58:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/3671884545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb80ce420 unknown :-1 s=CLOSED pgs=2144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/3671884545 >> v2:172.21.15.186:3300/0 conn(0x7f2bb80b1ca0 0x7f2bb80d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 -- 172.21.15.29:0/3671884545 >> 172.21.15.29:0/3671884545 conn(0x7f2bb80d2b90 msgr2=0x7f2bb80d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:22.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 -- 172.21.15.29:0/3671884545 shutdown_connections 2023-10-14T17:58:22.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.294+0000 7f2bb7fff640 1 -- 172.21.15.29:0/3671884545 wait complete. 2023-10-14T17:58:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.296+0000 7f2bb7fff640 1 Processor -- start 2023-10-14T17:58:22.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.296+0000 7f2bb7fff640 1 -- start start 2023-10-14T17:58:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bb80b1ca0 0x7f2bb80c8580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bb80ce960 0x7f2bb815d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb814e890 con 0x7f2bb80b2080 2023-10-14T17:58:22.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80c6b50 con 0x7f2bb80b1ca0 2023-10-14T17:58:22.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bb80c6e30 con 0x7f2bb80ce960 2023-10-14T17:58:22.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33928/0 (socket says 172.21.15.29:33928) 2023-10-14T17:58:22.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.297+0000 7f2bb67fc640 1 -- 172.21.15.29:0/855666426 learned_addr learned my addr 172.21.15.29:0/855666426 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb6ffd640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bb80b1ca0 0x7f2bb80c8580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 -- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bb80ce960 msgr2=0x7f2bb815d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb77fe640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bb80ce960 0x7f2bb815d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bb80ce960 0x7f2bb815d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 -- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bb80b1ca0 msgr2=0x7f2bb80c8580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bb80b1ca0 0x7f2bb80c8580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 -- 172.21.15.29:0/855666426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ba00094d0 con 0x7f2bb80b2080 2023-10-14T17:58:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.298+0000 7f2bb67fc640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2ba002ba70 tx=0x7f2ba0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.299+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0037d10 con 0x7f2bb80b2080 2023-10-14T17:58:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.299+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ba0037e90 con 0x7f2bb80b2080 2023-10-14T17:58:22.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.299+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bb80c7110 con 0x7f2bb80b2080 2023-10-14T17:58:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.299+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ba0041640 con 0x7f2bb80b2080 2023-10-14T17:58:22.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.299+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bb80c76d0 con 0x7f2bb80b2080 2023-10-14T17:58:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.300+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bb804e9b0 con 0x7f2bb80b2080 2023-10-14T17:58:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.301+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2ba00417a0 con 0x7f2bb80b2080 2023-10-14T17:58:22.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.303+0000 7f2baffff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 0x7f2b9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.303+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f2ba003c070 con 0x7f2bb80b2080 2023-10-14T17:58:22.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.303+0000 7f2bb6ffd640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 0x7f2b9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.304+0000 7f2bb6ffd640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 0x7f2b9806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80099c0 tx=0x7f2ba8003990 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.307+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ba0077490 con 0x7f2bb80b2080 2023-10-14T17:58:22.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.502+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f2bb8073a40 con 0x7f2bb80b2080 2023-10-14T17:58:22.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.502+0000 7f2baffff640 1 -- 172.21.15.29:0/855666426 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v188) v1 ==== 93+0+10 (secure 0 0 0) 0x7f2ba007aab0 con 0x7f2bb80b2080 2023-10-14T17:58:22.504 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T17:58:22.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 msgr2=0x7f2b9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 0x7f2b9806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f2ba80099c0 tx=0x7f2ba8003990 comp rx=0 tx=0).stop 2023-10-14T17:58:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 msgr2=0x7f2bb815c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 secure :-1 s=READY pgs=2145 cs=0 l=1 rev1=1 crypto rx=0x7f2ba002ba70 tx=0x7f2ba0004730 comp rx=0 tx=0).stop 2023-10-14T17:58:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 shutdown_connections 2023-10-14T17:58:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bb80ce960 0x7f2bb815d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bb80b2080 0x7f2bb815c8b0 unknown :-1 s=CLOSED pgs=2145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f2b9806af90 0x7f2b9806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 --2- 172.21.15.29:0/855666426 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bb80b1ca0 0x7f2bb80c8580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.505+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 >> 172.21.15.29:0/855666426 conn(0x7f2bb80d2b90 msgr2=0x7f2bb80b12f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:22.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.506+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 shutdown_connections 2023-10-14T17:58:22.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.506+0000 7f2bb7fff640 1 -- 172.21.15.29:0/855666426 wait complete. 2023-10-14T17:58:22.525 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T17:58:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.743+0000 7f4c388c8640 1 Processor -- start 2023-10-14T17:58:22.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.743+0000 7f4c388c8640 1 -- start start 2023-10-14T17:58:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.744+0000 7f4c388c8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c3415c9f0 0x7f4c340c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.744+0000 7f4c388c8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4c3415dde0 0x7f4c340ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.744+0000 7f4c388c8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c34160a90 con 0x7f4c340c9230 2023-10-14T17:58:22.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.744+0000 7f4c388c8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340cdec0 con 0x7f4c3415c9f0 2023-10-14T17:58:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.744+0000 7f4c388c8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c340cd900 con 0x7f4c3415dde0 2023-10-14T17:58:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c3415c9f0 0x7f4c340c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c3415c9f0 0x7f4c340c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33934/0 (socket says 172.21.15.29:33934) 2023-10-14T17:58:22.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 -- 172.21.15.29:0/1754254081 learned_addr learned my addr 172.21.15.29:0/1754254081 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:22.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 -- 172.21.15.29:0/1754254081 >> v2:172.21.15.186:3300/0 conn(0x7f4c3415dde0 msgr2=0x7f4c340ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 --2- 172.21.15.29:0/1754254081 >> v2:172.21.15.186:3300/0 conn(0x7f4c3415dde0 0x7f4c340ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 -- 172.21.15.29:0/1754254081 >> v1:172.21.15.186:6789/0 conn(0x7f4c340c9230 legacy=0x7f4c340709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.745+0000 7f4c32d9d640 1 -- 172.21.15.29:0/1754254081 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c340cdbe0 con 0x7f4c3415c9f0 2023-10-14T17:58:22.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.746+0000 7f4c32d9d640 1 --2- 172.21.15.29:0/1754254081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415c9f0 0x7f4c340c8cf0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4c340b0120 tx=0x7f4c2402b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f20ca0576db6ea36 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.747+0000 7f4c3259c640 1 -- 172.21.15.29:0/1754254081 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24031890 con 0x7f4c3415c9f0 2023-10-14T17:58:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.747+0000 7f4c3259c640 1 -- 172.21.15.29:0/1754254081 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c240319f0 con 0x7f4c3415c9f0 2023-10-14T17:58:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.747+0000 7f4c3259c640 1 -- 172.21.15.29:0/1754254081 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c24031d80 con 0x7f4c3415c9f0 2023-10-14T17:58:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.747+0000 7f4c388c8640 1 -- 172.21.15.29:0/1754254081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415c9f0 msgr2=0x7f4c340c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.747+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1754254081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415c9f0 0x7f4c340c8cf0 secure :-1 s=READY pgs=2146 cs=0 l=1 rev1=1 crypto rx=0x7f4c340b0120 tx=0x7f4c2402b930 comp rx=0 tx=0).stop 2023-10-14T17:58:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 -- 172.21.15.29:0/1754254081 shutdown_connections 2023-10-14T17:58:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1754254081 >> v2:172.21.15.186:3300/0 conn(0x7f4c3415dde0 0x7f4c340ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1754254081 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415c9f0 0x7f4c340c8cf0 unknown :-1 s=CLOSED pgs=2146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 -- 172.21.15.29:0/1754254081 >> 172.21.15.29:0/1754254081 conn(0x7f4c340ce6a0 msgr2=0x7f4c340ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:22.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 -- 172.21.15.29:0/1754254081 shutdown_connections 2023-10-14T17:58:22.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.748+0000 7f4c388c8640 1 -- 172.21.15.29:0/1754254081 wait complete. 2023-10-14T17:58:22.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.749+0000 7f4c388c8640 1 Processor -- start 2023-10-14T17:58:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.750+0000 7f4c388c8640 1 -- start start 2023-10-14T17:58:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c340c9230 0x7f4c341f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c3415c9f0 0x7f4c340c43d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c341f3350 con 0x7f4c3415dde0 2023-10-14T17:58:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3406b090 con 0x7f4c340c9230 2023-10-14T17:58:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c388c8640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c3406ccc0 con 0x7f4c3415c9f0 2023-10-14T17:58:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c3359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c3359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33950/0 (socket says 172.21.15.29:33950) 2023-10-14T17:58:22.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c3359e640 1 -- 172.21.15.29:0/1614643748 learned_addr learned my addr 172.21.15.29:0/1614643748 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c2a59c640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c3415c9f0 0x7f4c340c43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.751+0000 7f4c32d9d640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c340c9230 0x7f4c341f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 -- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c3415c9f0 msgr2=0x7f4c340c43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c3415c9f0 0x7f4c340c43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 -- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c340c9230 msgr2=0x7f4c341f22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c340c9230 0x7f4c341f22f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 -- 172.21.15.29:0/1614643748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240094d0 con 0x7f4c3415dde0 2023-10-14T17:58:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c3359e640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00ba10 tx=0x7f4c2c00bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c00cb20 con 0x7f4c3415dde0 2023-10-14T17:58:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c2c00cc80 con 0x7f4c3415dde0 2023-10-14T17:58:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c2c014440 con 0x7f4c3415dde0 2023-10-14T17:58:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c341f2830 con 0x7f4c3415dde0 2023-10-14T17:58:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.752+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341f2d90 con 0x7f4c3415dde0 2023-10-14T17:58:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.754+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c3406adf0 con 0x7f4c3415dde0 2023-10-14T17:58:22.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.754+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4c2c01d3b0 con 0x7f4c3415dde0 2023-10-14T17:58:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.755+0000 7f4c2bfff640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 0x7f4c1406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.756+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f4c2c089310 con 0x7f4c3415dde0 2023-10-14T17:58:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.756+0000 7f4c32d9d640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 0x7f4c1406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.757+0000 7f4c32d9d640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 0x7f4c1406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c24036000 tx=0x7f4c2403c040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.760+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c3406adf0 con 0x7f4c3415dde0 2023-10-14T17:58:22.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.957+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f4c3406b1d0 con 0x7f4c3415dde0 2023-10-14T17:58:22.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.958+0000 7f4c2bfff640 1 -- 172.21.15.29:0/1614643748 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v188) v1 ==== 84+0+65 (secure 0 0 0) 0x7f4c2c019060 con 0x7f4c3415dde0 2023-10-14T17:58:22.959 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:22.960 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:58:22.960 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:22.960 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T17:58:22.960 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T17:58:22.961 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:22.961 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:58:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 msgr2=0x7f4c1406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 0x7f4c1406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c24036000 tx=0x7f4c2403c040 comp rx=0 tx=0).stop 2023-10-14T17:58:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 msgr2=0x7f4c3406b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 secure :-1 s=READY pgs=2147 cs=0 l=1 rev1=1 crypto rx=0x7f4c2c00ba10 tx=0x7f4c2c00bed0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 shutdown_connections 2023-10-14T17:58:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c3415dde0 0x7f4c3406b4b0 unknown :-1 s=CLOSED pgs=2147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c3415c9f0 0x7f4c340c43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4c1406af90 0x7f4c1406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 --2- 172.21.15.29:0/1614643748 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c340c9230 0x7f4c341f22f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.960+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 >> 172.21.15.29:0/1614643748 conn(0x7f4c340ce6a0 msgr2=0x7f4c340ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:22.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.961+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 shutdown_connections 2023-10-14T17:58:22.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:22.961+0000 7f4c388c8640 1 -- 172.21.15.29:0/1614643748 wait complete. 2023-10-14T17:58:22.979 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T17:58:22.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T17:58:23.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.207+0000 7fa89759e640 1 Processor -- start 2023-10-14T17:58:23.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.207+0000 7fa89759e640 1 -- start start 2023-10-14T17:58:23.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa898071460 0x7fa898165570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 0x7fa8980c8f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8980cae70 con 0x7fa8980adec0 2023-10-14T17:58:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa898165de0 con 0x7fa898071460 2023-10-14T17:58:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa89806b860 con 0x7fa8980c18e0 2023-10-14T17:58:23.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa895d9b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa898071460 0x7fa898165570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa8980adec0 0x7fa898164e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45022/0 (socket says 172.21.15.29:45022) 2023-10-14T17:58:23.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89659c640 1 -- 172.21.15.29:0/2751364563 learned_addr learned my addr 172.21.15.29:0/2751364563 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa895d9b640 1 -- 172.21.15.29:0/2751364563 >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 msgr2=0x7fa8980c8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:23.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa896d9d640 1 --2- 172.21.15.29:0/2751364563 >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 0x7fa8980c8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.209+0000 7fa89559a640 1 -- 172.21.15.29:0/2751364563 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011082673 0 0) 0x7fa8980cae70 con 0x7fa8980adec0 2023-10-14T17:58:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.210+0000 7fa895d9b640 1 --2- 172.21.15.29:0/2751364563 >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 0x7fa8980c8f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:23.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.210+0000 7fa895d9b640 1 -- 172.21.15.29:0/2751364563 >> v1:172.21.15.29:6789/0 conn(0x7fa8980adec0 legacy=0x7fa898164e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.210+0000 7fa895d9b640 1 -- 172.21.15.29:0/2751364563 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa89806b580 con 0x7fa898071460 2023-10-14T17:58:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.210+0000 7fa896d9d640 1 --2- 172.21.15.29:0/2751364563 >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 0x7fa8980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:23.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.211+0000 7fa895d9b640 1 --2- 172.21.15.29:0/2751364563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa898071460 0x7fa898165570 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fa8980c5880 tx=0x7fa88c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f01ecc2dab661617 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.211+0000 7fa89559a640 1 -- 172.21.15.29:0/2751364563 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c030610 con 0x7fa898071460 2023-10-14T17:58:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.211+0000 7fa89559a640 1 -- 172.21.15.29:0/2751364563 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa88c030770 con 0x7fa898071460 2023-10-14T17:58:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.211+0000 7fa89559a640 1 -- 172.21.15.29:0/2751364563 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88c030b00 con 0x7fa898071460 2023-10-14T17:58:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 -- 172.21.15.29:0/2751364563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa898071460 msgr2=0x7fa898165570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:23.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 --2- 172.21.15.29:0/2751364563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa898071460 0x7fa898165570 secure :-1 s=READY pgs=2149 cs=0 l=1 rev1=1 crypto rx=0x7fa8980c5880 tx=0x7fa88c02b960 comp rx=0 tx=0).stop 2023-10-14T17:58:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 -- 172.21.15.29:0/2751364563 shutdown_connections 2023-10-14T17:58:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 --2- 172.21.15.29:0/2751364563 >> v2:172.21.15.186:3300/0 conn(0x7fa8980c18e0 0x7fa8980c8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 --2- 172.21.15.29:0/2751364563 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa898071460 0x7fa898165570 unknown :-1 s=CLOSED pgs=2149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 -- 172.21.15.29:0/2751364563 >> 172.21.15.29:0/2751364563 conn(0x7fa8980ce820 msgr2=0x7fa8980cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:23.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 -- 172.21.15.29:0/2751364563 shutdown_connections 2023-10-14T17:58:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.212+0000 7fa89759e640 1 -- 172.21.15.29:0/2751364563 wait complete. 2023-10-14T17:58:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.214+0000 7fa89759e640 1 Processor -- start 2023-10-14T17:58:23.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.214+0000 7fa89759e640 1 -- start start 2023-10-14T17:58:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa898071460 0x7fa898155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8980adec0 0x7fa898162a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8980c2430 con 0x7fa8980c18e0 2023-10-14T17:58:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa898163380 con 0x7fa898071460 2023-10-14T17:58:23.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa898163660 con 0x7fa8980adec0 2023-10-14T17:58:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa896d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa896d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33968/0 (socket says 172.21.15.29:33968) 2023-10-14T17:58:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa896d9d640 1 -- 172.21.15.29:0/1318296314 learned_addr learned my addr 172.21.15.29:0/1318296314 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.215+0000 7fa89659c640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa898071460 0x7fa898155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa895d9b640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8980adec0 0x7fa898162a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 -- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8980adec0 msgr2=0x7fa898162a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8980adec0 0x7fa898162a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 -- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa898071460 msgr2=0x7fa898155a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa898071460 0x7fa898155a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:23.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 -- 172.21.15.29:0/1318296314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c009400 con 0x7fa8980c18e0 2023-10-14T17:58:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa896d9d640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fa89000cd50 tx=0x7fa89000a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000f680 con 0x7fa8980c18e0 2023-10-14T17:58:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa89000f800 con 0x7fa8980c18e0 2023-10-14T17:58:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa89000fbb0 con 0x7fa8980c18e0 2023-10-14T17:58:23.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa898163940 con 0x7fa8980c18e0 2023-10-14T17:58:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.216+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa898155fd0 con 0x7fa8980c18e0 2023-10-14T17:58:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.217+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8980c25a0 con 0x7fa8980c18e0 2023-10-14T17:58:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.218+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa89000b040 con 0x7fa8980c18e0 2023-10-14T17:58:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.219+0000 7fa886ffd640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 0x7fa87806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:23.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.219+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(188..188 src has 1..188) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fa89008a060 con 0x7fa8980c18e0 2023-10-14T17:58:23.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.220+0000 7fa89659c640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 0x7fa87806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:23.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.220+0000 7fa89659c640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 0x7fa87806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa888000f60 tx=0x7fa888007b50 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:23.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.223+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa8900bcfe0 con 0x7fa8980c18e0 2023-10-14T17:58:23.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:23.411+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fa89804e9b0 con 0x7fa8980c18e0 2023-10-14T17:58:24.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.040+0000 7fa886ffd640 1 -- 172.21.15.29:0/1318296314 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v189) v1 ==== 138+0+0 (secure 0 0 0) 0x7fa89001d0b0 con 0x7fa8980c18e0 2023-10-14T17:58:24.042 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T17:58:24.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 msgr2=0x7fa87806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:24.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 0x7fa87806d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa888000f60 tx=0x7fa888007b50 comp rx=0 tx=0).stop 2023-10-14T17:58:24.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 msgr2=0x7fa89815fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:24.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 secure :-1 s=READY pgs=2150 cs=0 l=1 rev1=1 crypto rx=0x7fa89000cd50 tx=0x7fa89000a650 comp rx=0 tx=0).stop 2023-10-14T17:58:24.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 shutdown_connections 2023-10-14T17:58:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa8980c18e0 0x7fa89815fb50 unknown :-1 s=CLOSED pgs=2150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa8980adec0 0x7fa898162a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa87806af90 0x7fa87806d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 --2- 172.21.15.29:0/1318296314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa898071460 0x7fa898155a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.044+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 >> 172.21.15.29:0/1318296314 conn(0x7fa8980ce820 msgr2=0x7fa8980c5050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.045+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 shutdown_connections 2023-10-14T17:58:24.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.045+0000 7fa89759e640 1 -- 172.21.15.29:0/1318296314 wait complete. 2023-10-14T17:58:24.072 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T17:58:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.293+0000 7fa1793ac640 1 Processor -- start 2023-10-14T17:58:24.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.293+0000 7fa1793ac640 1 -- start start 2023-10-14T17:58:24.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa17415aa60 0x7fa1740af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa1740afd60 0x7fa1740cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cc820 0x7fa1740b2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1740b3730 con 0x7fa17415aa60 2023-10-14T17:58:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1740b3980 con 0x7fa1740afd60 2023-10-14T17:58:24.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa1793ac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740b3c60 con 0x7fa1740cc820 2023-10-14T17:58:24.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa173fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa17415aa60 0x7fa1740af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa173fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa17415aa60 0x7fa1740af820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33970/0 (socket says 172.21.15.29:33970) 2023-10-14T17:58:24.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.294+0000 7fa173fff640 1 -- 172.21.15.29:0/1032581411 learned_addr learned my addr 172.21.15.29:0/1032581411 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:24.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa1737fe640 1 --2- 172.21.15.29:0/1032581411 >> v2:172.21.15.186:3300/0 conn(0x7fa1740afd60 0x7fa1740cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa178bab640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cc820 0x7fa1740b2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa173fff640 1 -- 172.21.15.29:0/1032581411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cc820 msgr2=0x7fa1740b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:24.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa173fff640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cc820 0x7fa1740b2360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa173fff640 1 -- 172.21.15.29:0/1032581411 >> v2:172.21.15.186:3300/0 conn(0x7fa1740afd60 msgr2=0x7fa1740cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa173fff640 1 --2- 172.21.15.29:0/1032581411 >> v2:172.21.15.186:3300/0 conn(0x7fa1740afd60 0x7fa1740cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.295+0000 7fa173fff640 1 -- 172.21.15.29:0/1032581411 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1740b3f40 con 0x7fa17415aa60 2023-10-14T17:58:24.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.296+0000 7fa173fff640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17415aa60 0x7fa1740af820 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa1740d1610 tx=0x7fa16402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=341aff4f6074343a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa172ffd640 1 -- 172.21.15.29:0/1032581411 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16402ca70 con 0x7fa17415aa60 2023-10-14T17:58:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa172ffd640 1 -- 172.21.15.29:0/1032581411 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16402cbd0 con 0x7fa17415aa60 2023-10-14T17:58:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa172ffd640 1 -- 172.21.15.29:0/1032581411 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1640343f0 con 0x7fa17415aa60 2023-10-14T17:58:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 -- 172.21.15.29:0/1032581411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17415aa60 msgr2=0x7fa1740af820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:24.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17415aa60 0x7fa1740af820 secure :-1 s=READY pgs=2151 cs=0 l=1 rev1=1 crypto rx=0x7fa1740d1610 tx=0x7fa16402bed0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 -- 172.21.15.29:0/1032581411 shutdown_connections 2023-10-14T17:58:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa1740cc820 0x7fa1740b2360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 --2- 172.21.15.29:0/1032581411 >> v2:172.21.15.186:3300/0 conn(0x7fa1740afd60 0x7fa1740cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 --2- 172.21.15.29:0/1032581411 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa17415aa60 0x7fa1740af820 unknown :-1 s=CLOSED pgs=2151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.297+0000 7fa1793ac640 1 -- 172.21.15.29:0/1032581411 >> 172.21.15.29:0/1032581411 conn(0x7fa174065ed0 msgr2=0x7fa1740662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.298+0000 7fa1793ac640 1 -- 172.21.15.29:0/1032581411 shutdown_connections 2023-10-14T17:58:24.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.298+0000 7fa1793ac640 1 -- 172.21.15.29:0/1032581411 wait complete. 2023-10-14T17:58:24.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.299+0000 7fa1793ac640 1 Processor -- start 2023-10-14T17:58:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.300+0000 7fa1793ac640 1 -- start start 2023-10-14T17:58:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.300+0000 7fa1793ac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.300+0000 7fa1793ac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1740cc820 0x7fa1740ca150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa1793ac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17415aa60 0x7fa174153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa1793ac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1740c9c40 con 0x7fa1740afd60 2023-10-14T17:58:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa1793ac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1741461a0 con 0x7fa1740cc820 2023-10-14T17:58:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa1793ac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17406c9d0 con 0x7fa17415aa60 2023-10-14T17:58:24.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33980/0 (socket says 172.21.15.29:33980) 2023-10-14T17:58:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 -- 172.21.15.29:0/515495691 learned_addr learned my addr 172.21.15.29:0/515495691 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 -- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17415aa60 msgr2=0x7fa174153450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa178bab640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17415aa60 0x7fa174153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa1737fe640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1740cc820 0x7fa1740ca150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17415aa60 0x7fa174153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 -- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1740cc820 msgr2=0x7fa1740ca150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1740cc820 0x7fa1740ca150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:24.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.301+0000 7fa173fff640 1 -- 172.21.15.29:0/515495691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa164009b30 con 0x7fa1740afd60 2023-10-14T17:58:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa173fff640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fa164000f80 tx=0x7fa164004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa164009510 con 0x7fa1740afd60 2023-10-14T17:58:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16403e450 con 0x7fa1740afd60 2023-10-14T17:58:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa16403e800 con 0x7fa1740afd60 2023-10-14T17:58:24.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17406ccb0 con 0x7fa1740afd60 2023-10-14T17:58:24.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.302+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa174152f30 con 0x7fa1740afd60 2023-10-14T17:58:24.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.303+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1740cb340 con 0x7fa1740afd60 2023-10-14T17:58:24.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.305+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa16403e960 con 0x7fa1740afd60 2023-10-14T17:58:24.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.306+0000 7fa170ff9640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 0x7fa16006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:24.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.306+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(189..189 src has 1..189) v4 ==== 5343+0+0 (secure 0 0 0) 0x7fa1640ac800 con 0x7fa1740afd60 2023-10-14T17:58:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.306+0000 7fa1737fe640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 0x7fa16006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.307+0000 7fa1737fe640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 0x7fa16006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:24.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.310+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa164077a10 con 0x7fa1740afd60 2023-10-14T17:58:24.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:24.505+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fa17404e9b0 con 0x7fa1740afd60 2023-10-14T17:58:25.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.047+0000 7fa170ff9640 1 -- 172.21.15.29:0/515495691 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v190) v1 ==== 170+0+0 (secure 0 0 0) 0x7fa16407b030 con 0x7fa1740afd60 2023-10-14T17:58:25.049 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T17:58:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 msgr2=0x7fa16006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 0x7fa16006d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa168006fd0 tx=0x7fa168008040 comp rx=0 tx=0).stop 2023-10-14T17:58:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 msgr2=0x7fa174145c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 secure :-1 s=READY pgs=2152 cs=0 l=1 rev1=1 crypto rx=0x7fa164000f80 tx=0x7fa164004a40 comp rx=0 tx=0).stop 2023-10-14T17:58:25.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 shutdown_connections 2023-10-14T17:58:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa17415aa60 0x7fa174153450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fa16006af90 0x7fa16006d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa1740cc820 0x7fa1740ca150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 --2- 172.21.15.29:0/515495691 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa1740afd60 0x7fa174145c60 unknown :-1 s=CLOSED pgs=2152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 >> 172.21.15.29:0/515495691 conn(0x7fa174065ed0 msgr2=0x7fa1740cee30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.051+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 shutdown_connections 2023-10-14T17:58:25.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.052+0000 7fa1793ac640 1 -- 172.21.15.29:0/515495691 wait complete. 2023-10-14T17:58:25.084 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T17:58:25.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.276+0000 7f0789590640 1 Processor -- start 2023-10-14T17:58:25.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.276+0000 7f0789590640 1 -- start start 2023-10-14T17:58:25.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.278+0000 7f0789590640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.278+0000 7f0789590640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 0x7f07841650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.278+0000 7f0789590640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07840cb140 con 0x7f07840c9000 2023-10-14T17:58:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.278+0000 7f0789590640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0784161a50 con 0x7f078406a4b0 2023-10-14T17:58:25.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.278+0000 7f0789590640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f078406b030 con 0x7f07840addb0 2023-10-14T17:58:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.279+0000 7f0782ffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.279+0000 7f07837fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 0x7f07841650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.279+0000 7f07837fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 0x7f07841650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:55278/0 (socket says 172.21.15.29:55278) 2023-10-14T17:58:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.279+0000 7f0782ffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:40824/0 (socket says 172.21.15.29:40824) 2023-10-14T17:58:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.279+0000 7f07837fe640 1 -- 172.21.15.29:0/1308237383 learned_addr learned my addr 172.21.15.29:0/1308237383 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.280+0000 7f0782ffd640 1 -- 172.21.15.29:0/1308237383 >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 msgr2=0x7f07841650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:25.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.280+0000 7f0782ffd640 1 --2- 172.21.15.29:0/1308237383 >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 0x7f07841650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.280+0000 7f0782ffd640 1 -- 172.21.15.29:0/1308237383 >> v1:172.21.15.186:6789/0 conn(0x7f07840c9000 legacy=0x7f0784069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.280+0000 7f0782ffd640 1 -- 172.21.15.29:0/1308237383 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078406b310 con 0x7f07840addb0 2023-10-14T17:58:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.281+0000 7f0782ffd640 1 --2- 172.21.15.29:0/1308237383 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3355d7b444326fa2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.282+0000 7f07827fc640 1 -- 172.21.15.29:0/1308237383 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774031830 con 0x7f07840addb0 2023-10-14T17:58:25.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.282+0000 7f07827fc640 1 -- 172.21.15.29:0/1308237383 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0774031990 con 0x7f07840addb0 2023-10-14T17:58:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.282+0000 7f07827fc640 1 -- 172.21.15.29:0/1308237383 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0774031d20 con 0x7f07840addb0 2023-10-14T17:58:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 -- 172.21.15.29:0/1308237383 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 msgr2=0x7f07840caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 --2- 172.21.15.29:0/1308237383 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f07740097a0 tx=0x7f077402b8d0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 -- 172.21.15.29:0/1308237383 shutdown_connections 2023-10-14T17:58:25.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 --2- 172.21.15.29:0/1308237383 >> v2:172.21.15.186:3300/0 conn(0x7f078406a4b0 0x7f07841650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 --2- 172.21.15.29:0/1308237383 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840caa80 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 -- 172.21.15.29:0/1308237383 >> 172.21.15.29:0/1308237383 conn(0x7f07840ce9b0 msgr2=0x7f07840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 -- 172.21.15.29:0/1308237383 shutdown_connections 2023-10-14T17:58:25.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.283+0000 7f0789590640 1 -- 172.21.15.29:0/1308237383 wait complete. 2023-10-14T17:58:25.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.285+0000 7f0789590640 1 Processor -- start 2023-10-14T17:58:25.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.285+0000 7f0789590640 1 -- start start 2023-10-14T17:58:25.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f078406a4b0 0x7f078415bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f078415e150 con 0x7f07840c9000 2023-10-14T17:58:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07840c65f0 con 0x7f078406a4b0 2023-10-14T17:58:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0789590640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07840c68d0 con 0x7f07840addb0 2023-10-14T17:58:25.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f07837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f07837fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33990/0 (socket says 172.21.15.29:33990) 2023-10-14T17:58:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f07837fe640 1 -- 172.21.15.29:0/1622789302 learned_addr learned my addr 172.21.15.29:0/1622789302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.287+0000 7f0782ffd640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f078406a4b0 0x7f078415bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f077bfff640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 -- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 msgr2=0x7f07840c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840c5a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 -- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f078406a4b0 msgr2=0x7f078415bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f078406a4b0 0x7f078415bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:25.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 -- 172.21.15.29:0/1622789302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0774009400 con 0x7f07840c9000 2023-10-14T17:58:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.288+0000 7f07837fe640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f077c00efc0 tx=0x7f077c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.289+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c00cd20 con 0x7f07840c9000 2023-10-14T17:58:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.289+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f077c00f040 con 0x7f07840c9000 2023-10-14T17:58:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.289+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f077c003bd0 con 0x7f07840c9000 2023-10-14T17:58:25.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.289+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07840c6bb0 con 0x7f07840c9000 2023-10-14T17:58:25.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.289+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07840c7170 con 0x7f07840c9000 2023-10-14T17:58:25.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.290+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07840c7360 con 0x7f07840c9000 2023-10-14T17:58:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.291+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f077c003d30 con 0x7f07840c9000 2023-10-14T17:58:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.292+0000 7f077b7fe640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 0x7f076406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.292+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(190..190 src has 1..190) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f077c089f30 con 0x7f07840c9000 2023-10-14T17:58:25.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.293+0000 7f0782ffd640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 0x7f076406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:25.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.294+0000 7f0782ffd640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 0x7f076406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0774036000 tx=0x7f077403c040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:25.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.298+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f077c054dc0 con 0x7f07840c9000 2023-10-14T17:58:25.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:25.532+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f078404e160 con 0x7f07840c9000 2023-10-14T17:58:26.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.096+0000 7f077b7fe640 1 -- 172.21.15.29:0/1622789302 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 22 and data pool 23 v88) v1 ==== 174+0+0 (secure 0 0 0) 0x7f077c0583e0 con 0x7f07840c9000 2023-10-14T17:58:26.101 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 22 and data pool 23 2023-10-14T17:58:26.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.106+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 msgr2=0x7f076406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.106+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 0x7f076406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0774036000 tx=0x7f077403c040 comp rx=0 tx=0).stop 2023-10-14T17:58:26.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 msgr2=0x7f07840c5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 secure :-1 s=READY pgs=2153 cs=0 l=1 rev1=1 crypto rx=0x7f077c00efc0 tx=0x7f077c00c490 comp rx=0 tx=0).stop 2023-10-14T17:58:26.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 shutdown_connections 2023-10-14T17:58:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07840c9000 0x7f07840c5fc0 unknown :-1 s=CLOSED pgs=2153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07840addb0 0x7f07840c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f076406af90 0x7f076406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 --2- 172.21.15.29:0/1622789302 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f078406a4b0 0x7f078415bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.107+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 >> 172.21.15.29:0/1622789302 conn(0x7f07840ce9b0 msgr2=0x7f07840cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:26.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.108+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 shutdown_connections 2023-10-14T17:58:26.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.108+0000 7f0789590640 1 -- 172.21.15.29:0/1622789302 wait complete. 2023-10-14T17:58:26.143 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:26.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.332+0000 7f21d2b6c640 1 Processor -- start 2023-10-14T17:58:26.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.332+0000 7f21d2b6c640 1 -- start start 2023-10-14T17:58:26.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.333+0000 7f21d2b6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21cc0cd0c0 0x7f21cc1615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.333+0000 7f21d2b6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0b0410 0x7f21cc0b0870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.333+0000 7f21d2b6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21cc0b0db0 con 0x7f21cc15d3f0 2023-10-14T17:58:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.333+0000 7f21d2b6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21cc15e3e0 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.333+0000 7f21d2b6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0cd4a0 con 0x7f21cc0b0410 2023-10-14T17:58:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21cc0cd0c0 0x7f21cc1615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f21cc0cd0c0 0x7f21cc1615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33994/0 (socket says 172.21.15.29:33994) 2023-10-14T17:58:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 -- 172.21.15.29:0/788461700 learned_addr learned my addr 172.21.15.29:0/788461700 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:26.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 -- 172.21.15.29:0/788461700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0b0410 msgr2=0x7f21cc0b0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:26.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 --2- 172.21.15.29:0/788461700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0b0410 0x7f21cc0b0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 -- 172.21.15.29:0/788461700 >> v1:172.21.15.29:6789/0 conn(0x7f21cc15d3f0 legacy=0x7f21cc15d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.334+0000 7f21d0908640 1 -- 172.21.15.29:0/788461700 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21cc15e100 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.335+0000 7f21d0908640 1 --2- 172.21.15.29:0/788461700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0cd0c0 0x7f21cc1615f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c26252f88106988 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21cb7fe640 1 -- 172.21.15.29:0/788461700 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc02ca70 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21cb7fe640 1 -- 172.21.15.29:0/788461700 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc02cbd0 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21cb7fe640 1 -- 172.21.15.29:0/788461700 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc0343f0 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 -- 172.21.15.29:0/788461700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0cd0c0 msgr2=0x7f21cc1615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/788461700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0cd0c0 0x7f21cc1615f0 secure :-1 s=READY pgs=2155 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc02bed0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 -- 172.21.15.29:0/788461700 shutdown_connections 2023-10-14T17:58:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/788461700 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0b0410 0x7f21cc0b0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/788461700 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0cd0c0 0x7f21cc1615f0 unknown :-1 s=CLOSED pgs=2155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 -- 172.21.15.29:0/788461700 >> 172.21.15.29:0/788461700 conn(0x7f21cc0cb410 msgr2=0x7f21cc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:26.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 -- 172.21.15.29:0/788461700 shutdown_connections 2023-10-14T17:58:26.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.336+0000 7f21d2b6c640 1 -- 172.21.15.29:0/788461700 wait complete. 2023-10-14T17:58:26.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.338+0000 7f21d2b6c640 1 Processor -- start 2023-10-14T17:58:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.338+0000 7f21d2b6c640 1 -- start start 2023-10-14T17:58:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0cd0c0 0x7f21cc156b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc15d3f0 0x7f21cc157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0b2710 con 0x7f21cc0b0410 2023-10-14T17:58:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0c4fd0 con 0x7f21cc15d3f0 2023-10-14T17:58:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d2b6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21cc0c52b0 con 0x7f21cc0cd0c0 2023-10-14T17:58:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.339+0000 7f21d0908640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34000/0 (socket says 172.21.15.29:34000) 2023-10-14T17:58:26.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 -- 172.21.15.29:0/4121874552 learned_addr learned my addr 172.21.15.29:0/4121874552 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 -- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0cd0c0 msgr2=0x7f21cc156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0cd0c0 0x7f21cc156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 -- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc15d3f0 msgr2=0x7f21cc157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc15d3f0 0x7f21cc157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 -- 172.21.15.29:0/4121874552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21bc009b30 con 0x7f21cc0b0410 2023-10-14T17:58:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.340+0000 7f21d0908640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.341+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc03e450 con 0x7f21cc0b0410 2023-10-14T17:58:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.341+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21bc03e5f0 con 0x7f21cc0b0410 2023-10-14T17:58:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.341+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21bc03e9a0 con 0x7f21cc0b0410 2023-10-14T17:58:26.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.341+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21cc0b20c0 con 0x7f21cc0b0410 2023-10-14T17:58:26.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.341+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21cc06c7d0 con 0x7f21cc0b0410 2023-10-14T17:58:26.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.342+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21cc04e160 con 0x7f21cc0b0410 2023-10-14T17:58:26.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.343+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f21bc03eb00 con 0x7f21cc0b0410 2023-10-14T17:58:26.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.344+0000 7f21c97fa640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 0x7f21ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.344+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f21bc0ac8a0 con 0x7f21cc0b0410 2023-10-14T17:58:26.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.344+0000 7f21cbfff640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 0x7f21ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.345+0000 7f21cbfff640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 0x7f21ac06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0ca1d0 tx=0x7f21b4006bf0 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.350+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f21bc0778d0 con 0x7f21cc0b0410 2023-10-14T17:58:26.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.552+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21cc0b4910 con 0x7f21cc0b0410 2023-10-14T17:58:26.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.554+0000 7f21c97fa640 1 -- 172.21.15.29:0/4121874552 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16758 (secure 0 0 0) 0x7f21bc03edb0 con 0x7f21cc0b0410 2023-10-14T17:58:26.556 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:26.556 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":191,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:25.534454+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:26.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 msgr2=0x7f21ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 0x7f21ac06d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f21cc0ca1d0 tx=0x7f21b4006bf0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 msgr2=0x7f21cc06c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 secure :-1 s=READY pgs=2156 cs=0 l=1 rev1=1 crypto rx=0x7f21bc009e50 tx=0x7f21bc004a40 comp rx=0 tx=0).stop 2023-10-14T17:58:26.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 shutdown_connections 2023-10-14T17:58:26.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f21ac06af90 0x7f21ac06d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f21cc15d3f0 0x7f21cc157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f21cc0cd0c0 0x7f21cc156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 --2- 172.21.15.29:0/4121874552 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f21cc0b0410 0x7f21cc06c290 unknown :-1 s=CLOSED pgs=2156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.557+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 >> 172.21.15.29:0/4121874552 conn(0x7f21cc0cb410 msgr2=0x7f21cc15c700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.558+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 shutdown_connections 2023-10-14T17:58:26.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.558+0000 7f21d2b6c640 1 -- 172.21.15.29:0/4121874552 wait complete. 2023-10-14T17:58:26.577 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.801+0000 7f96523fc640 1 Processor -- start 2023-10-14T17:58:26.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.801+0000 7f96523fc640 1 -- start start 2023-10-14T17:58:26.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f96523fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f964c159b20 0x7f964c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f96523fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f964c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f96523fc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f964c06f7f0 con 0x7f964c159f00 2023-10-14T17:58:26.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f96523fc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f964c06c5e0 con 0x7f964c159b20 2023-10-14T17:58:26.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f96523fc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06c8c0 con 0x7f964c0b0b20 2023-10-14T17:58:26.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f964b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f964c159f00 0x7f964c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45040/0 (socket says 172.21.15.29:45040) 2023-10-14T17:58:26.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f964b7fe640 1 -- 172.21.15.29:0/2000693100 learned_addr learned my addr 172.21.15.29:0/2000693100 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:26.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 246227729 0 0) 0x7f964c06f7f0 con 0x7f964c159f00 2023-10-14T17:58:26.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9634003520 con 0x7f964c159f00 2023-10-14T17:58:26.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.803+0000 7f964bfff640 1 --2- 172.21.15.29:0/2000693100 >> v2:172.21.15.186:3300/0 conn(0x7f964c159b20 0x7f964c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f9650999640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f964c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 688867260 0 0) 0x7f9634003520 con 0x7f964c159f00 2023-10-14T17:58:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 msgr2=0x7f964c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f964c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 >> v2:172.21.15.186:3300/0 conn(0x7f964c159b20 msgr2=0x7f964c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 --2- 172.21.15.29:0/2000693100 >> v2:172.21.15.186:3300/0 conn(0x7f964c159b20 0x7f964c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c06cba0 con 0x7f964c159f00 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9640001670 con 0x7f964c159f00 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 >> v1:172.21.15.29:6789/0 conn(0x7f964c159f00 legacy=0x7f964c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634003970 0x7f9634003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f96340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9634004d00 0x7f9634005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06c5e0 con 0x7f9634004d00 2023-10-14T17:58:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c06c8c0 con 0x7f9634003970 2023-10-14T17:58:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f963400dc70 con 0x7f964c0b0b20 2023-10-14T17:58:26.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.804+0000 7f9650999640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9634004d00 0x7f9634005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f964b7fe640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f96340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f9650999640 1 -- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 msgr2=0x7f96340047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f9650999640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c0b0b20 0x7f96340047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f964bfff640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634003970 0x7f9634003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f9650999640 1 -- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634003970 msgr2=0x7f9634003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f9650999640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634003970 0x7f9634003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.805+0000 7f9650999640 1 -- 172.21.15.29:0/2000693100 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9634003520 con 0x7f9634004d00 2023-10-14T17:58:26.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.806+0000 7f9650999640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9634004d00 0x7f9634005180 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f964400ba00 tx=0x7f96340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec54f5b7ca4fb218 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.806+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644032620 con 0x7f9634004d00 2023-10-14T17:58:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.806+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9644032780 con 0x7f9634004d00 2023-10-14T17:58:26.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.806+0000 7f964affd640 1 -- 172.21.15.29:0/2000693100 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644032b10 con 0x7f9634004d00 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 -- 172.21.15.29:0/2000693100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9634004d00 msgr2=0x7f9634005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 --2- 172.21.15.29:0/2000693100 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9634004d00 0x7f9634005180 secure :-1 s=READY pgs=2158 cs=0 l=1 rev1=1 crypto rx=0x7f964400ba00 tx=0x7f96340057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f9650999640 1 -- 172.21.15.29:0/2000693100 reap_dead start 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 -- 172.21.15.29:0/2000693100 shutdown_connections 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 -- 172.21.15.29:0/2000693100 >> 172.21.15.29:0/2000693100 conn(0x7f964c0cb410 msgr2=0x7f964c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:26.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 -- 172.21.15.29:0/2000693100 shutdown_connections 2023-10-14T17:58:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.807+0000 7f96523fc640 1 -- 172.21.15.29:0/2000693100 wait complete. 2023-10-14T17:58:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.809+0000 7f96523fc640 1 Processor -- start 2023-10-14T17:58:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.809+0000 7f96523fc640 1 -- start start 2023-10-14T17:58:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 0x7f964c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c1f5960 0x7f964c0cd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c0c7010 con 0x7f964c1f2c00 2023-10-14T17:58:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964c1ed050 con 0x7f9634004d00 2023-10-14T17:58:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f96523fc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964c1f3450 con 0x7f964c1f5960 2023-10-14T17:58:26.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.810+0000 7f964b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34016/0 (socket says 172.21.15.29:34016) 2023-10-14T17:58:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 -- 172.21.15.29:0/321354993 learned_addr learned my addr 172.21.15.29:0/321354993 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964bfff640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 0x7f964c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f9650999640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c1f5960 0x7f964c0cd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 -- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c1f5960 msgr2=0x7f964c0cd470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c1f5960 0x7f964c0cd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 -- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 msgr2=0x7f964c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 0x7f964c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 -- 172.21.15.29:0/321354993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f964c1f3730 con 0x7f964c1f2c00 2023-10-14T17:58:26.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964bfff640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 0x7f964c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.811+0000 7f964b7fe640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f964000d820 tx=0x7f964000dce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.812+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400074e0 con 0x7f964c1f2c00 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.812+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9640007640 con 0x7f964c1f2c00 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.812+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96400049e0 con 0x7f964c1f2c00 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.812+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f964c1f3a10 con 0x7f964c1f2c00 2023-10-14T17:58:26.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.812+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f964c0c6be0 con 0x7f964c1f2c00 2023-10-14T17:58:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.813+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f964c0c6e60 con 0x7f964c1f2c00 2023-10-14T17:58:26.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.814+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9640004b40 con 0x7f964c1f2c00 2023-10-14T17:58:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.815+0000 7f9648ff9640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.815+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(191..191 src has 1..191) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f96400893e0 con 0x7f964c1f2c00 2023-10-14T17:58:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.815+0000 7f964bfff640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:26.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.816+0000 7f964bfff640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 0x7f963806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f964c150aa0 tx=0x7f963c00b040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:26.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:26.820+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9640054150 con 0x7f964c1f2c00 2023-10-14T17:58:27.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.016+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f964c04e160 con 0x7f964c1f2c00 2023-10-14T17:58:27.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.019+0000 7f9648ff9640 1 -- 172.21.15.29:0/321354993 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v191) v1 ==== 74+0+16758 (secure 0 0 0) 0x7f9640057770 con 0x7f964c1f2c00 2023-10-14T17:58:27.021 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:27.021 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":191,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:25.534454+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:27.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 msgr2=0x7f963806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 0x7f963806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f964c150aa0 tx=0x7f963c00b040 comp rx=0 tx=0).stop 2023-10-14T17:58:27.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 msgr2=0x7f964c1f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 secure :-1 s=READY pgs=2159 cs=0 l=1 rev1=1 crypto rx=0x7f964000d820 tx=0x7f964000dce0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 shutdown_connections 2023-10-14T17:58:27.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f964c1f5960 0x7f964c0cd470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f964c1f2c00 0x7f964c1f3cf0 unknown :-1 s=CLOSED pgs=2159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 --2- 172.21.15.29:0/321354993 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9634004d00 0x7f964c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 >> 172.21.15.29:0/321354993 conn(0x7f964c0cb410 msgr2=0x7f964c157b00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:27.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 shutdown_connections 2023-10-14T17:58:27.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.022+0000 7f96523fc640 1 -- 172.21.15.29:0/321354993 wait complete. 2023-10-14T17:58:27.044 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T17:58:27.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.262+0000 7f4eb759e640 1 Processor -- start 2023-10-14T17:58:27.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.263+0000 7f4eb759e640 1 -- start start 2023-10-14T17:58:27.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.264+0000 7f4eb759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.264+0000 7f4eb759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 0x7f4eb815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.264+0000 7f4eb759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb806c960 con 0x7f4eb815ebc0 2023-10-14T17:58:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.264+0000 7f4eb759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb815b9d0 con 0x7f4eb80ad970 2023-10-14T17:58:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.264+0000 7f4eb759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb815bcb0 con 0x7f4eb806c040 2023-10-14T17:58:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.265+0000 7f4eb659c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.265+0000 7f4eb5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 0x7f4eb815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.265+0000 7f4eb659c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:40886/0 (socket says 172.21.15.29:40886) 2023-10-14T17:58:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.265+0000 7f4eb5d9b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 0x7f4eb815e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:55316/0 (socket says 172.21.15.29:55316) 2023-10-14T17:58:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.265+0000 7f4eb659c640 1 -- 172.21.15.29:0/3069970506 learned_addr learned my addr 172.21.15.29:0/3069970506 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.266+0000 7f4eb659c640 1 -- 172.21.15.29:0/3069970506 >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 msgr2=0x7f4eb815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.266+0000 7f4eb659c640 1 --2- 172.21.15.29:0/3069970506 >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 0x7f4eb815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.266+0000 7f4eb659c640 1 -- 172.21.15.29:0/3069970506 >> v1:172.21.15.186:6789/0 conn(0x7f4eb815ebc0 legacy=0x7f4eb80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.266+0000 7f4eb659c640 1 -- 172.21.15.29:0/3069970506 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb815bf90 con 0x7f4eb806c040 2023-10-14T17:58:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.268+0000 7f4eb659c640 1 --2- 172.21.15.29:0/3069970506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009e80 tx=0x7f4ea802bf50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7636dad83ddc5e2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb559a640 1 -- 172.21.15.29:0/3069970506 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea802caf0 con 0x7f4eb806c040 2023-10-14T17:58:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb559a640 1 -- 172.21.15.29:0/3069970506 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea802cc50 con 0x7f4eb806c040 2023-10-14T17:58:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb559a640 1 -- 172.21.15.29:0/3069970506 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea80343f0 con 0x7f4eb806c040 2023-10-14T17:58:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 -- 172.21.15.29:0/3069970506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 msgr2=0x7f4eb806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 --2- 172.21.15.29:0/3069970506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8009e80 tx=0x7f4ea802bf50 comp rx=0 tx=0).stop 2023-10-14T17:58:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 -- 172.21.15.29:0/3069970506 shutdown_connections 2023-10-14T17:58:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 --2- 172.21.15.29:0/3069970506 >> v2:172.21.15.186:3300/0 conn(0x7f4eb80ad970 0x7f4eb815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 --2- 172.21.15.29:0/3069970506 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb806c040 0x7f4eb806c420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.269+0000 7f4eb759e640 1 -- 172.21.15.29:0/3069970506 >> 172.21.15.29:0/3069970506 conn(0x7f4eb80ce6a0 msgr2=0x7f4eb80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.270+0000 7f4eb759e640 1 -- 172.21.15.29:0/3069970506 shutdown_connections 2023-10-14T17:58:27.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.270+0000 7f4eb759e640 1 -- 172.21.15.29:0/3069970506 wait complete. 2023-10-14T17:58:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.271+0000 7f4eb759e640 1 Processor -- start 2023-10-14T17:58:27.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.272+0000 7f4eb759e640 1 -- start start 2023-10-14T17:58:27.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.272+0000 7f4eb759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb806c040 0x7f4eb8156540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.272+0000 7f4eb759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.272+0000 7f4eb759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 0x7f4eb80b2e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb80c7c30 con 0x7f4eb80ad970 2023-10-14T17:58:27.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb81f0170 con 0x7f4eb806c040 2023-10-14T17:58:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb80c7180 con 0x7f4eb815ebc0 2023-10-14T17:58:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb5d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34026/0 (socket says 172.21.15.29:34026) 2023-10-14T17:58:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb5d9b640 1 -- 172.21.15.29:0/1479806525 learned_addr learned my addr 172.21.15.29:0/1479806525 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:27.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb659c640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb806c040 0x7f4eb8156540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb6d9d640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 0x7f4eb80b2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb5d9b640 1 -- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 msgr2=0x7f4eb80b2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.273+0000 7f4eb5d9b640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 0x7f4eb80b2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb5d9b640 1 -- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb806c040 msgr2=0x7f4eb8156540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:27.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb5d9b640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb806c040 0x7f4eb8156540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb5d9b640 1 -- 172.21.15.29:0/1479806525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea8009b30 con 0x7f4eb80ad970 2023-10-14T17:58:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb6d9d640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 0x7f4eb80b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:27.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb5d9b640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00e8a0 tx=0x7f4eac00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac0094f0 con 0x7f4eb80ad970 2023-10-14T17:58:27.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4eac00c940 con 0x7f4eb80ad970 2023-10-14T17:58:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb80ae2f0 con 0x7f4eb80ad970 2023-10-14T17:58:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4eac00ccf0 con 0x7f4eb80ad970 2023-10-14T17:58:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.274+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb8156a80 con 0x7f4eb80ad970 2023-10-14T17:58:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.276+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb806b330 con 0x7f4eb80ad970 2023-10-14T17:58:27.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.277+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4eac013470 con 0x7f4eb80ad970 2023-10-14T17:58:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.277+0000 7f4ea6ffd640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 0x7f4e9806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.278+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4eac089930 con 0x7f4eb80ad970 2023-10-14T17:58:27.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.278+0000 7f4eb659c640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 0x7f4e9806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:27.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.279+0000 7f4eb659c640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 0x7f4e9806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8035000 tx=0x7f4ea8030000 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:27.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.283+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4eac0547b0 con 0x7f4eb80ad970 2023-10-14T17:58:27.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:27.519+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f4eb804e9b0 con 0x7f4eb80ad970 2023-10-14T17:58:28.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.121+0000 7f4ea6ffd640 1 -- 172.21.15.29:0/1479806525 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v90) v1 ==== 118+0+0 (secure 0 0 0) 0x7f4eac0210b0 con 0x7f4eb80ad970 2023-10-14T17:58:28.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 msgr2=0x7f4e9806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 0x7f4e9806d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4ea8035000 tx=0x7f4ea8030000 comp rx=0 tx=0).stop 2023-10-14T17:58:28.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 msgr2=0x7f4eb80c7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 secure :-1 s=READY pgs=2160 cs=0 l=1 rev1=1 crypto rx=0x7f4eac00e8a0 tx=0x7f4eac00ed60 comp rx=0 tx=0).stop 2023-10-14T17:58:28.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 shutdown_connections 2023-10-14T17:58:28.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb815ebc0 0x7f4eb80b2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4eb80ad970 0x7f4eb80c7f80 unknown :-1 s=CLOSED pgs=2160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f4e9806af90 0x7f4e9806d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 --2- 172.21.15.29:0/1479806525 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb806c040 0x7f4eb8156540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.125+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 >> 172.21.15.29:0/1479806525 conn(0x7f4eb80ce6a0 msgr2=0x7f4eb8070630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:28.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.126+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 shutdown_connections 2023-10-14T17:58:28.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.126+0000 7f4eb759e640 1 -- 172.21.15.29:0/1479806525 wait complete. 2023-10-14T17:58:28.149 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T17:58:28.149 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T17:58:28.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.377+0000 7fc3c2f47640 1 Processor -- start 2023-10-14T17:58:28.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.377+0000 7fc3c2f47640 1 -- start start 2023-10-14T17:58:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c2f47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15d410 0x7fc3bc0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c2f47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc0b5d90 con 0x7fc3bc15d030 2023-10-14T17:58:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c2f47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3bc160fc0 con 0x7fc3bc0cd260 2023-10-14T17:58:28.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c2f47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3bc0d2b50 con 0x7fc3bc15d410 2023-10-14T17:58:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c0ce3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3bc15d030 0x7fc3bc161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45054/0 (socket says 172.21.15.29:45054) 2023-10-14T17:58:28.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3c0ce3640 1 -- 172.21.15.29:0/1211533588 learned_addr learned my addr 172.21.15.29:0/1211533588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:28.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725676178 0 0) 0x7fc3bc0b5d90 con 0x7fc3bc15d030 2023-10-14T17:58:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3a4003520 con 0x7fc3bc15d030 2023-10-14T17:58:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.379+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1211533588 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15d410 0x7fc3bc0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2916611800 0 0) 0x7fc3a4003520 con 0x7fc3bc15d030 2023-10-14T17:58:28.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15d410 msgr2=0x7fc3bc0d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1211533588 >> v2:172.21.15.186:3300/0 conn(0x7fc3bc15d410 0x7fc3bc0d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 >> v1:172.21.15.186:6789/0 conn(0x7fc3bc0cd260 legacy=0x7fc3bc0b4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:28.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc0d25d0 con 0x7fc3bc15d030 2023-10-14T17:58:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc3ac001670 con 0x7fc3bc15d030 2023-10-14T17:58:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 >> v1:172.21.15.29:6789/0 conn(0x7fc3bc15d030 legacy=0x7fc3bc161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40039c0 0x7fc3a4003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 0x7fc3a4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40051d0 0x7fc3a400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a400c0e0 con 0x7fc3a40047c0 2023-10-14T17:58:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a400c220 con 0x7fc3a40039c0 2023-10-14T17:58:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a400c3e0 con 0x7fc3a40051d0 2023-10-14T17:58:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.380+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 0x7fc3a4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3c0ce3640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40039c0 0x7fc3a4003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40051d0 msgr2=0x7fc3a400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3c14e4640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40051d0 0x7fc3a400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40051d0 0x7fc3a400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40039c0 msgr2=0x7fc3a4003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:28.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40039c0 0x7fc3a4003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3bbfff640 1 -- 172.21.15.29:0/1211533588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a4003520 con 0x7fc3a40047c0 2023-10-14T17:58:28.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3c14e4640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40051d0 0x7fc3a400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:28.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3c0ce3640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3a40039c0 0x7fc3a4003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:28.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.381+0000 7fc3c14e4640 1 -- 172.21.15.29:0/1211533588 reap_dead start 2023-10-14T17:58:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.382+0000 7fc3bbfff640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 0x7fc3a4004c40 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000ba00 tx=0x7fc3a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bc2124f1578ba14 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:28.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.382+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0032620 con 0x7fc3a40047c0 2023-10-14T17:58:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.382+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b0032780 con 0x7fc3a40047c0 2023-10-14T17:58:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.382+0000 7fc3bb7fe640 1 -- 172.21.15.29:0/1211533588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b0032b10 con 0x7fc3a40047c0 2023-10-14T17:58:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 -- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 msgr2=0x7fc3a4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 0x7fc3a4004c40 secure :-1 s=READY pgs=2162 cs=0 l=1 rev1=1 crypto rx=0x7fc3b000ba00 tx=0x7fc3a40057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 -- 172.21.15.29:0/1211533588 shutdown_connections 2023-10-14T17:58:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/1211533588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a40047c0 0x7fc3a4004c40 unknown :-1 s=CLOSED pgs=2162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 -- 172.21.15.29:0/1211533588 >> 172.21.15.29:0/1211533588 conn(0x7fc3bc0cf720 msgr2=0x7fc3bc0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 -- 172.21.15.29:0/1211533588 shutdown_connections 2023-10-14T17:58:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.383+0000 7fc3c2f47640 1 -- 172.21.15.29:0/1211533588 wait complete. 2023-10-14T17:58:28.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.385+0000 7fc3c2f47640 1 Processor -- start 2023-10-14T17:58:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.385+0000 7fc3c2f47640 1 -- start start 2023-10-14T17:58:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40047c0 0x7fc3bc0c6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15d890 0x7fc3bc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc154a10 con 0x7fc3bc153930 2023-10-14T17:58:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc15e9a0 con 0x7fc3bc15d890 2023-10-14T17:58:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.386+0000 7fc3c2f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3bc0ccb40 con 0x7fc3a40047c0 2023-10-14T17:58:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34044/0 (socket says 172.21.15.29:34044) 2023-10-14T17:58:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 -- 172.21.15.29:0/2814055589 learned_addr learned my addr 172.21.15.29:0/2814055589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 -- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40047c0 msgr2=0x7fc3bc0c6f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40047c0 0x7fc3bc0c6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 -- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15d890 msgr2=0x7fc3bc0cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:28.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15d890 0x7fc3bc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 -- 172.21.15.29:0/2814055589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc15dd90 con 0x7fc3bc153930 2023-10-14T17:58:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.387+0000 7fc3c14e4640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0d05d0 tx=0x7fc3b400d950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.388+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4018070 con 0x7fc3bc153930 2023-10-14T17:58:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.388+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3b4006b50 con 0x7fc3bc153930 2023-10-14T17:58:28.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.388+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4012600 con 0x7fc3bc153930 2023-10-14T17:58:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.388+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3bc15e070 con 0x7fc3bc153930 2023-10-14T17:58:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.388+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3bc15e630 con 0x7fc3bc153930 2023-10-14T17:58:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.389+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3bc04e160 con 0x7fc3bc153930 2023-10-14T17:58:28.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.390+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc3b4012760 con 0x7fc3bc153930 2023-10-14T17:58:28.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.391+0000 7fc3b97fa640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 0x7fc3a806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:28.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.391+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc3b4088f80 con 0x7fc3bc153930 2023-10-14T17:58:28.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.391+0000 7fc3c0ce3640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 0x7fc3a806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:28.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.392+0000 7fc3c0ce3640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 0x7fc3a806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac001a90 tx=0x7fc3ac001a00 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:28.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.395+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3b4053d90 con 0x7fc3bc153930 2023-10-14T17:58:28.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:28.636+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fc3bc15eae0 con 0x7fc3bc153930 2023-10-14T17:58:29.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.133+0000 7fc3b97fa640 1 -- 172.21.15.29:0/2814055589 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v91) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc3bc15eae0 con 0x7fc3bc153930 2023-10-14T17:58:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 msgr2=0x7fc3a806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 0x7fc3a806d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac001a90 tx=0x7fc3ac001a00 comp rx=0 tx=0).stop 2023-10-14T17:58:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 msgr2=0x7fc3bc0b7050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 secure :-1 s=READY pgs=2163 cs=0 l=1 rev1=1 crypto rx=0x7fc3bc0d05d0 tx=0x7fc3b400d950 comp rx=0 tx=0).stop 2023-10-14T17:58:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 shutdown_connections 2023-10-14T17:58:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3bc153930 0x7fc3bc0b7050 unknown :-1 s=CLOSED pgs=2163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc3a806af90 0x7fc3a806d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3bc15d890 0x7fc3bc0cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 --2- 172.21.15.29:0/2814055589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40047c0 0x7fc3bc0c6f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.137+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 >> 172.21.15.29:0/2814055589 conn(0x7fc3bc0cf720 msgr2=0x7fc3bc0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.138+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 shutdown_connections 2023-10-14T17:58:29.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.138+0000 7fc3c2f47640 1 -- 172.21.15.29:0/2814055589 wait complete. 2023-10-14T17:58:29.161 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T17:58:29.162 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:29.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.388+0000 7ff2c5ac2640 1 Processor -- start 2023-10-14T17:58:29.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.388+0000 7ff2c5ac2640 1 -- start start 2023-10-14T17:58:29.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.389+0000 7ff2c5ac2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 0x7ff2c015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.389+0000 7ff2c5ac2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2c006c100 0x7ff2c006c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.389+0000 7ff2c5ac2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2c015aef0 con 0x7ff2c0159950 2023-10-14T17:58:29.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.389+0000 7ff2c5ac2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c015e490 con 0x7ff2c006c100 2023-10-14T17:58:29.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.389+0000 7ff2c5ac2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2c015d910 con 0x7ff2c0070b60 2023-10-14T17:58:29.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2c006c100 0x7ff2c006c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff2c006c100 0x7ff2c006c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34050/0 (socket says 172.21.15.29:34050) 2023-10-14T17:58:29.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 -- 172.21.15.29:0/4282059734 learned_addr learned my addr 172.21.15.29:0/4282059734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:29.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2bffff640 1 --2- 172.21.15.29:0/4282059734 >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 0x7ff2c015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 -- 172.21.15.29:0/4282059734 >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 msgr2=0x7ff2c015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:29.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 --2- 172.21.15.29:0/4282059734 >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 0x7ff2c015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 -- 172.21.15.29:0/4282059734 >> v1:172.21.15.186:6789/0 conn(0x7ff2c0159950 legacy=0x7ff2c0070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:29.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2c52c1640 1 -- 172.21.15.29:0/4282059734 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2c015dbf0 con 0x7ff2c006c100 2023-10-14T17:58:29.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.390+0000 7ff2bffff640 1 --2- 172.21.15.29:0/4282059734 >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 0x7ff2c015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:29.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.392+0000 7ff2c52c1640 1 --2- 172.21.15.29:0/4282059734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c006c510 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00716a0 tx=0x7ff2b802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ecbebda80ad3d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.392+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/4282059734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b80317e0 con 0x7ff2c006c100 2023-10-14T17:58:29.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.392+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/4282059734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b8031940 con 0x7ff2c006c100 2023-10-14T17:58:29.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.392+0000 7ff2bf7fe640 1 -- 172.21.15.29:0/4282059734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b8031cd0 con 0x7ff2c006c100 2023-10-14T17:58:29.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4282059734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 msgr2=0x7ff2c006c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4282059734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c006c510 secure :-1 s=READY pgs=2164 cs=0 l=1 rev1=1 crypto rx=0x7ff2c00716a0 tx=0x7ff2b802b880 comp rx=0 tx=0).stop 2023-10-14T17:58:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4282059734 shutdown_connections 2023-10-14T17:58:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4282059734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c006c510 unknown :-1 s=CLOSED pgs=2164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4282059734 >> v2:172.21.15.186:3300/0 conn(0x7ff2c0070b60 0x7ff2c015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4282059734 >> 172.21.15.29:0/4282059734 conn(0x7ff2c00ce6a0 msgr2=0x7ff2c00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4282059734 shutdown_connections 2023-10-14T17:58:29.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.393+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4282059734 wait complete. 2023-10-14T17:58:29.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.395+0000 7ff2c5ac2640 1 Processor -- start 2023-10-14T17:58:29.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.395+0000 7ff2c5ac2640 1 -- start start 2023-10-14T17:58:29.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2c0070b60 0x7ff2c00c2420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2c0159950 0x7ff2c00c2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c00ca560 con 0x7ff2c006c100 2023-10-14T17:58:29.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c014f340 con 0x7ff2c0070b60 2023-10-14T17:58:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c5ac2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2c014f620 con 0x7ff2c0159950 2023-10-14T17:58:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34060/0 (socket says 172.21.15.29:34060) 2023-10-14T17:58:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 -- 172.21.15.29:0/4080614952 learned_addr learned my addr 172.21.15.29:0/4080614952 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:29.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c52c1640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2c0159950 0x7ff2c00c2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2bffff640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2c0070b60 0x7ff2c00c2420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 -- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2c0159950 msgr2=0x7ff2c00c2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2c0159950 0x7ff2c00c2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 -- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2c0070b60 msgr2=0x7ff2c00c2420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.396+0000 7ff2c4ac0640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2c0070b60 0x7ff2c00c2420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2c4ac0640 1 -- 172.21.15.29:0/4080614952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b80094d0 con 0x7ff2c006c100 2023-10-14T17:58:29.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2c4ac0640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000ba50 tx=0x7ff2b000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b000cbe0 con 0x7ff2c006c100 2023-10-14T17:58:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2b000cd80 con 0x7ff2c006c100 2023-10-14T17:58:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2b0014680 con 0x7ff2c006c100 2023-10-14T17:58:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2c014f900 con 0x7ff2c006c100 2023-10-14T17:58:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.397+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2c006cb60 con 0x7ff2c006c100 2023-10-14T17:58:29.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.398+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2c004e9b0 con 0x7ff2c006c100 2023-10-14T17:58:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.399+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff2b00147e0 con 0x7ff2c006c100 2023-10-14T17:58:29.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.401+0000 7ff2bd7fa640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 0x7ff2ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.401+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff2b0089710 con 0x7ff2c006c100 2023-10-14T17:58:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.401+0000 7ff2bffff640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 0x7ff2ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.402+0000 7ff2bffff640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 0x7ff2ac06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2c006a1d0 tx=0x7ff2b4008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.405+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff2b0054640 con 0x7ff2c006c100 2023-10-14T17:58:29.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.644+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2c00ccc30 con 0x7ff2c006c100 2023-10-14T17:58:29.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.645+0000 7ff2bd7fa640 1 -- 172.21.15.29:0/4080614952 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7ff2b0057c60 con 0x7ff2c006c100 2023-10-14T17:58:29.647 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:29.647 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T17:58:29.648 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T17:58:29.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 msgr2=0x7ff2ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 0x7ff2ac06d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff2c006a1d0 tx=0x7ff2b4008040 comp rx=0 tx=0).stop 2023-10-14T17:58:29.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 msgr2=0x7ff2c00c9ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 secure :-1 s=READY pgs=2165 cs=0 l=1 rev1=1 crypto rx=0x7ff2b000ba50 tx=0x7ff2b000bf10 comp rx=0 tx=0).stop 2023-10-14T17:58:29.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 shutdown_connections 2023-10-14T17:58:29.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2c0159950 0x7ff2c00c2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff2ac06af90 0x7ff2ac06d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2c0070b60 0x7ff2c00c2420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 --2- 172.21.15.29:0/4080614952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2c006c100 0x7ff2c00c9ed0 unknown :-1 s=CLOSED pgs=2165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 >> 172.21.15.29:0/4080614952 conn(0x7ff2c00ce6a0 msgr2=0x7ff2c0070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:29.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.648+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 shutdown_connections 2023-10-14T17:58:29.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.649+0000 7ff2c5ac2640 1 -- 172.21.15.29:0/4080614952 wait complete. 2023-10-14T17:58:29.670 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T17:58:29.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.888+0000 7f56f491a640 1 Processor -- start 2023-10-14T17:58:29.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.888+0000 7f56f491a640 1 -- start start 2023-10-14T17:58:29.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56f491a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56f015cd70 0x7f56f00d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56f491a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f00b8480 0x7f56f00cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56f491a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56f015d150 con 0x7f56f00d0be0 2023-10-14T17:58:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56f491a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00d1180 con 0x7f56f00b8480 2023-10-14T17:58:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56f491a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56f00d1460 con 0x7f56f015cd70 2023-10-14T17:58:29.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f00b8480 0x7f56f00cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56f00b8480 0x7f56f00cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34076/0 (socket says 172.21.15.29:34076) 2023-10-14T17:58:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.890+0000 7f56effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56f00d0be0 0x7f56f00b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45068/0 (socket says 172.21.15.29:45068) 2023-10-14T17:58:29.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 -- 172.21.15.29:0/1717104712 learned_addr learned my addr 172.21.15.29:0/1717104712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:29.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56ef7fe640 1 --2- 172.21.15.29:0/1717104712 >> v2:172.21.15.186:3300/0 conn(0x7f56f015cd70 0x7f56f00d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 -- 172.21.15.29:0/1717104712 >> v2:172.21.15.186:3300/0 conn(0x7f56f015cd70 msgr2=0x7f56f00d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 --2- 172.21.15.29:0/1717104712 >> v2:172.21.15.186:3300/0 conn(0x7f56f015cd70 0x7f56f00d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 -- 172.21.15.29:0/1717104712 >> v1:172.21.15.29:6789/0 conn(0x7f56f00d0be0 legacy=0x7f56f00b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56eeffd640 1 -- 172.21.15.29:0/1717104712 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f00d1740 con 0x7f56f00b8480 2023-10-14T17:58:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.891+0000 7f56ee7fc640 1 -- 172.21.15.29:0/1717104712 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542199886 0 0) 0x7f56f015d150 con 0x7f56f00d0be0 2023-10-14T17:58:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.892+0000 7f56eeffd640 1 --2- 172.21.15.29:0/1717104712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f00cd7c0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f56f00b56a0 tx=0x7f56e402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70686a62eca1252a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.892+0000 7f56ee7fc640 1 -- 172.21.15.29:0/1717104712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031870 con 0x7f56f00b8480 2023-10-14T17:58:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.892+0000 7f56ee7fc640 1 -- 172.21.15.29:0/1717104712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e40319d0 con 0x7f56f00b8480 2023-10-14T17:58:29.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.892+0000 7f56ee7fc640 1 -- 172.21.15.29:0/1717104712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e4031d60 con 0x7f56f00b8480 2023-10-14T17:58:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 -- 172.21.15.29:0/1717104712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 msgr2=0x7f56f00cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 --2- 172.21.15.29:0/1717104712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f00cd7c0 secure :-1 s=READY pgs=2167 cs=0 l=1 rev1=1 crypto rx=0x7f56f00b56a0 tx=0x7f56e402b910 comp rx=0 tx=0).stop 2023-10-14T17:58:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 -- 172.21.15.29:0/1717104712 shutdown_connections 2023-10-14T17:58:29.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 --2- 172.21.15.29:0/1717104712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f00cd7c0 unknown :-1 s=CLOSED pgs=2167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 --2- 172.21.15.29:0/1717104712 >> v2:172.21.15.186:3300/0 conn(0x7f56f015cd70 0x7f56f00d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 -- 172.21.15.29:0/1717104712 >> 172.21.15.29:0/1717104712 conn(0x7f56f00cf570 msgr2=0x7f56f00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 -- 172.21.15.29:0/1717104712 shutdown_connections 2023-10-14T17:58:29.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.893+0000 7f56f491a640 1 -- 172.21.15.29:0/1717104712 wait complete. 2023-10-14T17:58:29.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.895+0000 7f56f491a640 1 Processor -- start 2023-10-14T17:58:29.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.895+0000 7f56f491a640 1 -- start start 2023-10-14T17:58:29.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f00d0be0 0x7f56f00cc2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56f015cd70 0x7f56f00cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f0155b90 con 0x7f56f00b8480 2023-10-14T17:58:29.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f01558b0 con 0x7f56f00d0be0 2023-10-14T17:58:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56f491a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56f0145e80 con 0x7f56f015cd70 2023-10-14T17:58:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56ef7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34090/0 (socket says 172.21.15.29:34090) 2023-10-14T17:58:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.896+0000 7f56ef7fe640 1 -- 172.21.15.29:0/3478360454 learned_addr learned my addr 172.21.15.29:0/3478360454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:29.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56eeffd640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f00d0be0 0x7f56f00cc2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56effff640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56f015cd70 0x7f56f00cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 -- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56f015cd70 msgr2=0x7f56f00cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56f015cd70 0x7f56f00cc830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 -- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f00d0be0 msgr2=0x7f56f00cc2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:29.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f00d0be0 0x7f56f00cc2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:29.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 -- 172.21.15.29:0/3478360454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56e40094d0 con 0x7f56f00b8480 2023-10-14T17:58:29.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56ef7fe640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f56e000d800 tx=0x7f56e000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e000b6c0 con 0x7f56f00b8480 2023-10-14T17:58:29.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56e000b820 con 0x7f56f00b8480 2023-10-14T17:58:29.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.897+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e000bbb0 con 0x7f56f00b8480 2023-10-14T17:58:29.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.898+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f0146160 con 0x7f56f00b8480 2023-10-14T17:58:29.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.898+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f0153130 con 0x7f56f00b8480 2023-10-14T17:58:29.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.899+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56f004e9b0 con 0x7f56f00b8480 2023-10-14T17:58:29.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.900+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f56e000bd10 con 0x7f56f00b8480 2023-10-14T17:58:29.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.901+0000 7f56d7fff640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 0x7f56dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:29.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.901+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f56e0089d20 con 0x7f56f00b8480 2023-10-14T17:58:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.901+0000 7f56eeffd640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 0x7f56dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.902+0000 7f56eeffd640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 0x7f56dc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56e4036000 tx=0x7f56e403c040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:29.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:29.905+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56e0054cd0 con 0x7f56f00b8480 2023-10-14T17:58:30.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.101+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56f015b340 con 0x7f56f00b8480 2023-10-14T17:58:30.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.104+0000 7f56d7fff640 1 -- 172.21.15.29:0/3478360454 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v192) v1 ==== 74+0+16745 (secure 0 0 0) 0x7f56e00582f0 con 0x7f56f00b8480 2023-10-14T17:58:30.105 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:30.105 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":192,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T17:58:27.096429+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T17:58:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 msgr2=0x7f56dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 0x7f56dc06d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f56e4036000 tx=0x7f56e403c040 comp rx=0 tx=0).stop 2023-10-14T17:58:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 msgr2=0x7f56f0152bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 secure :-1 s=READY pgs=2168 cs=0 l=1 rev1=1 crypto rx=0x7f56e000d800 tx=0x7f56e000dcc0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 shutdown_connections 2023-10-14T17:58:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56f015cd70 0x7f56f00cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f56dc06af90 0x7f56dc06d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56f00d0be0 0x7f56f00cc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 --2- 172.21.15.29:0/3478360454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56f00b8480 0x7f56f0152bf0 unknown :-1 s=CLOSED pgs=2168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.106+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 >> 172.21.15.29:0/3478360454 conn(0x7f56f00cf570 msgr2=0x7f56f00b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.107+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 shutdown_connections 2023-10-14T17:58:30.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.107+0000 7f56f491a640 1 -- 172.21.15.29:0/3478360454 wait complete. 2023-10-14T17:58:30.127 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T17:58:30.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:58:30.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.351+0000 7faeb6927640 1 Processor -- start 2023-10-14T17:58:30.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.352+0000 7faeb6927640 1 -- start start 2023-10-14T17:58:30.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.353+0000 7faeb6927640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faeb006c040 0x7faeb006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.353+0000 7faeb6927640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faeb00ad970 0x7faeb015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.353+0000 7faeb6927640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb006c960 con 0x7faeb015ebc0 2023-10-14T17:58:30.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.353+0000 7faeb6927640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb015b9d0 con 0x7faeb00ad970 2023-10-14T17:58:30.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.353+0000 7faeb6927640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb015bcb0 con 0x7faeb006c040 2023-10-14T17:58:30.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faeb00ad970 0x7faeb015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faeb00ad970 0x7faeb015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34100/0 (socket says 172.21.15.29:34100) 2023-10-14T17:58:30.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 -- 172.21.15.29:0/1290642243 learned_addr learned my addr 172.21.15.29:0/1290642243 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:30.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5925640 1 --2- 172.21.15.29:0/1290642243 >> v2:172.21.15.186:3300/0 conn(0x7faeb006c040 0x7faeb006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 -- 172.21.15.29:0/1290642243 >> v2:172.21.15.186:3300/0 conn(0x7faeb006c040 msgr2=0x7faeb006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:30.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 --2- 172.21.15.29:0/1290642243 >> v2:172.21.15.186:3300/0 conn(0x7faeb006c040 0x7faeb006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 -- 172.21.15.29:0/1290642243 >> v1:172.21.15.186:6789/0 conn(0x7faeb015ebc0 legacy=0x7faeb00cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:30.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.354+0000 7faeb5124640 1 -- 172.21.15.29:0/1290642243 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb015bf90 con 0x7faeb00ad970 2023-10-14T17:58:30.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.355+0000 7faeb5124640 1 --2- 172.21.15.29:0/1290642243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb015e680 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7faeb00cf350 tx=0x7faea002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c976c2cd02069566 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb4923640 1 -- 172.21.15.29:0/1290642243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0031870 con 0x7faeb00ad970 2023-10-14T17:58:30.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb4923640 1 -- 172.21.15.29:0/1290642243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea00319d0 con 0x7faeb00ad970 2023-10-14T17:58:30.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb4923640 1 -- 172.21.15.29:0/1290642243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea0031d60 con 0x7faeb00ad970 2023-10-14T17:58:30.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 -- 172.21.15.29:0/1290642243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 msgr2=0x7faeb015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 --2- 172.21.15.29:0/1290642243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb015e680 secure :-1 s=READY pgs=2169 cs=0 l=1 rev1=1 crypto rx=0x7faeb00cf350 tx=0x7faea002b910 comp rx=0 tx=0).stop 2023-10-14T17:58:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 -- 172.21.15.29:0/1290642243 shutdown_connections 2023-10-14T17:58:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 --2- 172.21.15.29:0/1290642243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb015e680 unknown :-1 s=CLOSED pgs=2169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 --2- 172.21.15.29:0/1290642243 >> v2:172.21.15.186:3300/0 conn(0x7faeb006c040 0x7faeb006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.356+0000 7faeb6927640 1 -- 172.21.15.29:0/1290642243 >> 172.21.15.29:0/1290642243 conn(0x7faeb00ce6a0 msgr2=0x7faeb00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.357+0000 7faeb6927640 1 -- 172.21.15.29:0/1290642243 shutdown_connections 2023-10-14T17:58:30.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.357+0000 7faeb6927640 1 -- 172.21.15.29:0/1290642243 wait complete. 2023-10-14T17:58:30.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.358+0000 7faeb6927640 1 Processor -- start 2023-10-14T17:58:30.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.358+0000 7faeb6927640 1 -- start start 2023-10-14T17:58:30.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb006c040 0x7faeb00b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 0x7faeb00c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00af210 con 0x7faeb00ad970 2023-10-14T17:58:30.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00b2a40 con 0x7faeb015ebc0 2023-10-14T17:58:30.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.359+0000 7faeb6927640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb00b1e30 con 0x7faeb006c040 2023-10-14T17:58:30.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34108/0 (socket says 172.21.15.29:34108) 2023-10-14T17:58:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 -- 172.21.15.29:0/225335570 learned_addr learned my addr 172.21.15.29:0/225335570 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 -- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb006c040 msgr2=0x7faeb00b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5925640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb006c040 0x7faeb00b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb6126640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 0x7faeb00c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb006c040 0x7faeb00b1750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 -- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 msgr2=0x7faeb00c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 0x7faeb00c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb5124640 1 -- 172.21.15.29:0/225335570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea00094d0 con 0x7faeb00ad970 2023-10-14T17:58:30.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.360+0000 7faeb6126640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 0x7faeb00c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faeb5124640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7faea002b950 tx=0x7faea00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea003ad30 con 0x7faeb00ad970 2023-10-14T17:58:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faea00443f0 con 0x7faeb00ad970 2023-10-14T17:58:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faea00447a0 con 0x7faeb00ad970 2023-10-14T17:58:30.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb00b2110 con 0x7faeb00ad970 2023-10-14T17:58:30.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.361+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb00b26d0 con 0x7faeb00ad970 2023-10-14T17:58:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.362+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb006b330 con 0x7faeb00ad970 2023-10-14T17:58:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.364+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7faea004d450 con 0x7faeb00ad970 2023-10-14T17:58:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.365+0000 7faea67fc640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 0x7fae9406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.365+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7faea002d040 con 0x7faeb00ad970 2023-10-14T17:58:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.365+0000 7faeb5925640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 0x7fae9406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.366+0000 7faeb5925640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 0x7fae9406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faeac006fd0 tx=0x7faeac008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.369+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faea0076ca0 con 0x7faeb00ad970 2023-10-14T17:58:30.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.560+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 --> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7faeb00cc270 con 0x7fae9406af90 2023-10-14T17:58:30.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.571+0000 7faea67fc640 1 -- 172.21.15.29:0/225335570 <== mgr.9861 v2:172.21.15.29:6832/79844030 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287021 (secure 0 0 0) 0x7faeac00e4b0 con 0x7fae9406af90 2023-10-14T17:58:30.577 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:30.579 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:58:30.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 msgr2=0x7fae9406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 0x7fae9406d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7faeac006fd0 tx=0x7faeac008040 comp rx=0 tx=0).stop 2023-10-14T17:58:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 msgr2=0x7faeb00aecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 secure :-1 s=READY pgs=2170 cs=0 l=1 rev1=1 crypto rx=0x7faea002b950 tx=0x7faea00046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 shutdown_connections 2023-10-14T17:58:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fae9406af90 0x7fae9406d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb015ebc0 0x7faeb00c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb00ad970 0x7faeb00aecd0 unknown :-1 s=CLOSED pgs=2170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 --2- 172.21.15.29:0/225335570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb006c040 0x7faeb00b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 >> 172.21.15.29:0/225335570 conn(0x7faeb00ce6a0 msgr2=0x7faeb00c99c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:30.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 shutdown_connections 2023-10-14T17:58:30.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.578+0000 7faeb6927640 1 -- 172.21.15.29:0/225335570 wait complete. 2023-10-14T17:58:30.600 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T17:58:29.156541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402628,"kb_used_data":19916,"kb_used_omap":12,"kb_used_meta":382579,"kb_avail":754572092,"statfs":{"total":773094113280,"available":772681822208,"internally_reserved":0,"allocated":20393984,"data_stored":10577164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":391761464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.114410"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.117131+0000","last_change":"2023-10-14T17:58:26.115017+0000","last_active":"2023-10-14T17:58:26.117131+0000","last_peered":"2023-10-14T17:58:26.117131+0000","last_clean":"2023-10-14T17:58:26.117131+0000","last_became_active":"2023-10-14T17:58:26.114711+0000","last_became_peered":"2023-10-14T17:58:26.114711+0000","last_unstale":"2023-10-14T17:58:26.117131+0000","last_undegraded":"2023-10-14T17:58:26.117131+0000","last_fullsized":"2023-10-14T17:58:26.117131+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:46:51.050670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121127+0000","last_change":"2023-10-14T17:58:26.110514+0000","last_active":"2023-10-14T17:58:27.121127+0000","last_peered":"2023-10-14T17:58:27.121127+0000","last_clean":"2023-10-14T17:58:27.121127+0000","last_became_active":"2023-10-14T17:58:26.109917+0000","last_became_peered":"2023-10-14T17:58:26.109917+0000","last_unstale":"2023-10-14T17:58:27.121127+0000","last_undegraded":"2023-10-14T17:58:27.121127+0000","last_fullsized":"2023-10-14T17:58:27.121127+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:35.001471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117210+0000","last_change":"2023-10-14T17:58:26.115229+0000","last_active":"2023-10-14T17:58:27.117210+0000","last_peered":"2023-10-14T17:58:27.117210+0000","last_clean":"2023-10-14T17:58:27.117210+0000","last_became_active":"2023-10-14T17:58:26.112909+0000","last_became_peered":"2023-10-14T17:58:26.112909+0000","last_unstale":"2023-10-14T17:58:27.117210+0000","last_undegraded":"2023-10-14T17:58:27.117210+0000","last_fullsized":"2023-10-14T17:58:27.117210+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:03.559514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802215+0000","last_change":"2023-10-14T17:58:26.127178+0000","last_active":"2023-10-14T17:58:27.802215+0000","last_peered":"2023-10-14T17:58:27.802215+0000","last_clean":"2023-10-14T17:58:27.802215+0000","last_became_active":"2023-10-14T17:58:26.125959+0000","last_became_peered":"2023-10-14T17:58:26.125959+0000","last_unstale":"2023-10-14T17:58:27.802215+0000","last_undegraded":"2023-10-14T17:58:27.802215+0000","last_fullsized":"2023-10-14T17:58:27.802215+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:54.759847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121032+0000","last_change":"2023-10-14T17:58:26.111144+0000","last_active":"2023-10-14T17:58:27.121032+0000","last_peered":"2023-10-14T17:58:27.121032+0000","last_clean":"2023-10-14T17:58:27.121032+0000","last_became_active":"2023-10-14T17:58:26.110880+0000","last_became_peered":"2023-10-14T17:58:26.110880+0000","last_unstale":"2023-10-14T17:58:27.121032+0000","last_undegraded":"2023-10-14T17:58:27.121032+0000","last_fullsized":"2023-10-14T17:58:27.121032+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:30.295018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730503+0000","last_change":"2023-10-14T17:58:26.112746+0000","last_active":"2023-10-14T17:58:27.730503+0000","last_peered":"2023-10-14T17:58:27.730503+0000","last_clean":"2023-10-14T17:58:27.730503+0000","last_became_active":"2023-10-14T17:58:26.112435+0000","last_became_peered":"2023-10-14T17:58:26.112435+0000","last_unstale":"2023-10-14T17:58:27.730503+0000","last_undegraded":"2023-10-14T17:58:27.730503+0000","last_fullsized":"2023-10-14T17:58:27.730503+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:09:41.991486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122970+0000","last_change":"2023-10-14T17:58:26.117083+0000","last_active":"2023-10-14T17:58:26.122970+0000","last_peered":"2023-10-14T17:58:26.122970+0000","last_clean":"2023-10-14T17:58:26.122970+0000","last_became_active":"2023-10-14T17:58:26.116789+0000","last_became_peered":"2023-10-14T17:58:26.116789+0000","last_unstale":"2023-10-14T17:58:26.122970+0000","last_undegraded":"2023-10-14T17:58:26.122970+0000","last_fullsized":"2023-10-14T17:58:26.122970+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:31.177662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119851+0000","last_change":"2023-10-14T17:58:26.111068+0000","last_active":"2023-10-14T17:58:27.119851+0000","last_peered":"2023-10-14T17:58:27.119851+0000","last_clean":"2023-10-14T17:58:27.119851+0000","last_became_active":"2023-10-14T17:58:26.110092+0000","last_became_peered":"2023-10-14T17:58:26.110092+0000","last_unstale":"2023-10-14T17:58:27.119851+0000","last_undegraded":"2023-10-14T17:58:27.119851+0000","last_fullsized":"2023-10-14T17:58:27.119851+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:04.917416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121313+0000","last_change":"2023-10-14T17:58:26.106129+0000","last_active":"2023-10-14T17:58:27.121313+0000","last_peered":"2023-10-14T17:58:27.121313+0000","last_clean":"2023-10-14T17:58:27.121313+0000","last_became_active":"2023-10-14T17:58:26.105788+0000","last_became_peered":"2023-10-14T17:58:26.105788+0000","last_unstale":"2023-10-14T17:58:27.121313+0000","last_undegraded":"2023-10-14T17:58:27.121313+0000","last_fullsized":"2023-10-14T17:58:27.121313+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:25.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121119+0000","last_change":"2023-10-14T17:58:26.110462+0000","last_active":"2023-10-14T17:58:27.121119+0000","last_peered":"2023-10-14T17:58:27.121119+0000","last_clean":"2023-10-14T17:58:27.121119+0000","last_became_active":"2023-10-14T17:58:26.108900+0000","last_became_peered":"2023-10-14T17:58:26.108900+0000","last_unstale":"2023-10-14T17:58:27.121119+0000","last_undegraded":"2023-10-14T17:58:27.121119+0000","last_fullsized":"2023-10-14T17:58:27.121119+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:10.274314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120667+0000","last_change":"2023-10-14T17:58:26.112457+0000","last_active":"2023-10-14T17:58:27.120667+0000","last_peered":"2023-10-14T17:58:27.120667+0000","last_clean":"2023-10-14T17:58:27.120667+0000","last_became_active":"2023-10-14T17:58:26.112160+0000","last_became_peered":"2023-10-14T17:58:26.112160+0000","last_unstale":"2023-10-14T17:58:27.120667+0000","last_undegraded":"2023-10-14T17:58:27.120667+0000","last_fullsized":"2023-10-14T17:58:27.120667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:47.784344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120541+0000","last_change":"2023-10-14T17:58:26.105882+0000","last_active":"2023-10-14T17:58:27.120541+0000","last_peered":"2023-10-14T17:58:27.120541+0000","last_clean":"2023-10-14T17:58:27.120541+0000","last_became_active":"2023-10-14T17:58:26.105601+0000","last_became_peered":"2023-10-14T17:58:26.105601+0000","last_unstale":"2023-10-14T17:58:27.120541+0000","last_undegraded":"2023-10-14T17:58:27.120541+0000","last_fullsized":"2023-10-14T17:58:27.120541+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:34.756888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115701+0000","last_change":"2023-10-14T17:58:26.108936+0000","last_active":"2023-10-14T17:58:27.115701+0000","last_peered":"2023-10-14T17:58:27.115701+0000","last_clean":"2023-10-14T17:58:27.115701+0000","last_became_active":"2023-10-14T17:58:26.108510+0000","last_became_peered":"2023-10-14T17:58:26.108510+0000","last_unstale":"2023-10-14T17:58:27.115701+0000","last_undegraded":"2023-10-14T17:58:27.115701+0000","last_fullsized":"2023-10-14T17:58:27.115701+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:47.784344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115308+0000","last_change":"2023-10-14T17:58:26.113124+0000","last_active":"2023-10-14T17:58:27.115308+0000","last_peered":"2023-10-14T17:58:27.115308+0000","last_clean":"2023-10-14T17:58:27.115308+0000","last_became_active":"2023-10-14T17:58:26.112161+0000","last_became_peered":"2023-10-14T17:58:26.112161+0000","last_unstale":"2023-10-14T17:58:27.115308+0000","last_undegraded":"2023-10-14T17:58:27.115308+0000","last_fullsized":"2023-10-14T17:58:27.115308+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:45:02.827786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.110747+0000","last_change":"2023-10-14T17:58:26.108433+0000","last_active":"2023-10-14T17:58:26.110747+0000","last_peered":"2023-10-14T17:58:26.110747+0000","last_clean":"2023-10-14T17:58:26.110747+0000","last_became_active":"2023-10-14T17:58:26.107966+0000","last_became_peered":"2023-10-14T17:58:26.107966+0000","last_unstale":"2023-10-14T17:58:26.110747+0000","last_undegraded":"2023-10-14T17:58:26.110747+0000","last_fullsized":"2023-10-14T17:58:26.110747+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:15.701827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115201+0000","last_change":"2023-10-14T17:58:26.111042+0000","last_active":"2023-10-14T17:58:26.115201+0000","last_peered":"2023-10-14T17:58:26.115201+0000","last_clean":"2023-10-14T17:58:26.115201+0000","last_became_active":"2023-10-14T17:58:26.110375+0000","last_became_peered":"2023-10-14T17:58:26.110375+0000","last_unstale":"2023-10-14T17:58:26.115201+0000","last_undegraded":"2023-10-14T17:58:26.115201+0000","last_fullsized":"2023-10-14T17:58:26.115201+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:10.949702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731305+0000","last_change":"2023-10-14T17:58:26.115084+0000","last_active":"2023-10-14T17:58:27.731305+0000","last_peered":"2023-10-14T17:58:27.731305+0000","last_clean":"2023-10-14T17:58:27.731305+0000","last_became_active":"2023-10-14T17:58:26.114809+0000","last_became_peered":"2023-10-14T17:58:26.114809+0000","last_unstale":"2023-10-14T17:58:27.731305+0000","last_undegraded":"2023-10-14T17:58:27.731305+0000","last_fullsized":"2023-10-14T17:58:27.731305+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:13:44.521120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123438+0000","last_change":"2023-10-14T17:58:26.120080+0000","last_active":"2023-10-14T17:58:26.123438+0000","last_peered":"2023-10-14T17:58:26.123438+0000","last_clean":"2023-10-14T17:58:26.123438+0000","last_became_active":"2023-10-14T17:58:26.119804+0000","last_became_peered":"2023-10-14T17:58:26.119804+0000","last_unstale":"2023-10-14T17:58:26.123438+0000","last_undegraded":"2023-10-14T17:58:26.123438+0000","last_fullsized":"2023-10-14T17:58:26.123438+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:22.620014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730693+0000","last_change":"2023-10-14T17:58:26.112261+0000","last_active":"2023-10-14T17:58:27.730693+0000","last_peered":"2023-10-14T17:58:27.730693+0000","last_clean":"2023-10-14T17:58:27.730693+0000","last_became_active":"2023-10-14T17:58:26.111775+0000","last_became_peered":"2023-10-14T17:58:26.111775+0000","last_unstale":"2023-10-14T17:58:27.730693+0000","last_undegraded":"2023-10-14T17:58:27.730693+0000","last_fullsized":"2023-10-14T17:58:27.730693+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:49.042427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115449+0000","last_change":"2023-10-14T17:58:26.107332+0000","last_active":"2023-10-14T17:58:27.115449+0000","last_peered":"2023-10-14T17:58:27.115449+0000","last_clean":"2023-10-14T17:58:27.115449+0000","last_became_active":"2023-10-14T17:58:26.107059+0000","last_became_peered":"2023-10-14T17:58:26.107059+0000","last_unstale":"2023-10-14T17:58:27.115449+0000","last_undegraded":"2023-10-14T17:58:27.115449+0000","last_fullsized":"2023-10-14T17:58:27.115449+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:34.756888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730700+0000","last_change":"2023-10-14T17:58:26.112541+0000","last_active":"2023-10-14T17:58:27.730700+0000","last_peered":"2023-10-14T17:58:27.730700+0000","last_clean":"2023-10-14T17:58:27.730700+0000","last_became_active":"2023-10-14T17:58:26.112190+0000","last_became_peered":"2023-10-14T17:58:26.112190+0000","last_unstale":"2023-10-14T17:58:27.730700+0000","last_undegraded":"2023-10-14T17:58:27.730700+0000","last_fullsized":"2023-10-14T17:58:27.730700+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:51.934965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120589+0000","last_change":"2023-10-14T17:58:26.111473+0000","last_active":"2023-10-14T17:58:27.120589+0000","last_peered":"2023-10-14T17:58:27.120589+0000","last_clean":"2023-10-14T17:58:27.120589+0000","last_became_active":"2023-10-14T17:58:26.110325+0000","last_became_peered":"2023-10-14T17:58:26.110325+0000","last_unstale":"2023-10-14T17:58:27.120589+0000","last_undegraded":"2023-10-14T17:58:27.120589+0000","last_fullsized":"2023-10-14T17:58:27.120589+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:59.702977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730153+0000","last_change":"2023-10-14T17:58:26.119709+0000","last_active":"2023-10-14T17:58:27.730153+0000","last_peered":"2023-10-14T17:58:27.730153+0000","last_clean":"2023-10-14T17:58:27.730153+0000","last_became_active":"2023-10-14T17:58:26.116619+0000","last_became_peered":"2023-10-14T17:58:26.116619+0000","last_unstale":"2023-10-14T17:58:27.730153+0000","last_undegraded":"2023-10-14T17:58:27.730153+0000","last_fullsized":"2023-10-14T17:58:27.730153+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:10:47.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119942+0000","last_change":"2023-10-14T17:58:26.104466+0000","last_active":"2023-10-14T17:58:27.119942+0000","last_peered":"2023-10-14T17:58:27.119942+0000","last_clean":"2023-10-14T17:58:27.119942+0000","last_became_active":"2023-10-14T17:58:26.104162+0000","last_became_peered":"2023-10-14T17:58:26.104162+0000","last_unstale":"2023-10-14T17:58:27.119942+0000","last_undegraded":"2023-10-14T17:58:27.119942+0000","last_fullsized":"2023-10-14T17:58:27.119942+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:45:02.827786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116891+0000","last_change":"2023-10-14T17:58:26.114566+0000","last_active":"2023-10-14T17:58:27.116891+0000","last_peered":"2023-10-14T17:58:27.116891+0000","last_clean":"2023-10-14T17:58:27.116891+0000","last_became_active":"2023-10-14T17:58:26.113446+0000","last_became_peered":"2023-10-14T17:58:26.113446+0000","last_unstale":"2023-10-14T17:58:27.116891+0000","last_undegraded":"2023-10-14T17:58:27.116891+0000","last_fullsized":"2023-10-14T17:58:27.116891+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:25.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111226+0000","last_change":"2023-10-14T17:58:26.105482+0000","last_active":"2023-10-14T17:58:26.111226+0000","last_peered":"2023-10-14T17:58:26.111226+0000","last_clean":"2023-10-14T17:58:26.111226+0000","last_became_active":"2023-10-14T17:58:26.105213+0000","last_became_peered":"2023-10-14T17:58:26.105213+0000","last_unstale":"2023-10-14T17:58:26.111226+0000","last_undegraded":"2023-10-14T17:58:26.111226+0000","last_fullsized":"2023-10-14T17:58:26.111226+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:50.488612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117294+0000","last_change":"2023-10-14T17:58:26.104039+0000","last_active":"2023-10-14T17:58:27.117294+0000","last_peered":"2023-10-14T17:58:27.117294+0000","last_clean":"2023-10-14T17:58:27.117294+0000","last_became_active":"2023-10-14T17:58:26.103764+0000","last_became_peered":"2023-10-14T17:58:26.103764+0000","last_unstale":"2023-10-14T17:58:27.117294+0000","last_undegraded":"2023-10-14T17:58:27.117294+0000","last_fullsized":"2023-10-14T17:58:27.117294+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:10:47.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802339+0000","last_change":"2023-10-14T17:58:26.114324+0000","last_active":"2023-10-14T17:58:27.802339+0000","last_peered":"2023-10-14T17:58:27.802339+0000","last_clean":"2023-10-14T17:58:27.802339+0000","last_became_active":"2023-10-14T17:58:26.114024+0000","last_became_peered":"2023-10-14T17:58:26.114024+0000","last_unstale":"2023-10-14T17:58:27.802339+0000","last_undegraded":"2023-10-14T17:58:27.802339+0000","last_fullsized":"2023-10-14T17:58:27.802339+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:37.273607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116066+0000","last_change":"2023-10-14T17:58:26.108769+0000","last_active":"2023-10-14T17:58:27.116066+0000","last_peered":"2023-10-14T17:58:27.116066+0000","last_clean":"2023-10-14T17:58:27.116066+0000","last_became_active":"2023-10-14T17:58:26.108135+0000","last_became_peered":"2023-10-14T17:58:26.108135+0000","last_unstale":"2023-10-14T17:58:27.116066+0000","last_undegraded":"2023-10-14T17:58:27.116066+0000","last_fullsized":"2023-10-14T17:58:27.116066+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:10.274314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119472+0000","last_change":"2023-10-14T17:58:26.107688+0000","last_active":"2023-10-14T17:58:27.119472+0000","last_peered":"2023-10-14T17:58:27.119472+0000","last_clean":"2023-10-14T17:58:27.119472+0000","last_became_active":"2023-10-14T17:58:26.107103+0000","last_became_peered":"2023-10-14T17:58:26.107103+0000","last_unstale":"2023-10-14T17:58:27.119472+0000","last_undegraded":"2023-10-14T17:58:27.119472+0000","last_fullsized":"2023-10-14T17:58:27.119472+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:50.606108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117569+0000","last_change":"2023-10-14T17:58:26.117028+0000","last_active":"2023-10-14T17:58:27.117569+0000","last_peered":"2023-10-14T17:58:27.117569+0000","last_clean":"2023-10-14T17:58:27.117569+0000","last_became_active":"2023-10-14T17:58:26.116744+0000","last_became_peered":"2023-10-14T17:58:26.116744+0000","last_unstale":"2023-10-14T17:58:27.117569+0000","last_undegraded":"2023-10-14T17:58:27.117569+0000","last_fullsized":"2023-10-14T17:58:27.117569+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:17:48.822796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.130935+0000","last_change":"2023-10-14T17:58:26.125706+0000","last_active":"2023-10-14T17:58:26.130935+0000","last_peered":"2023-10-14T17:58:26.130935+0000","last_clean":"2023-10-14T17:58:26.130935+0000","last_became_active":"2023-10-14T17:58:26.125086+0000","last_became_peered":"2023-10-14T17:58:26.125086+0000","last_unstale":"2023-10-14T17:58:26.130935+0000","last_undegraded":"2023-10-14T17:58:26.130935+0000","last_fullsized":"2023-10-14T17:58:26.130935+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:34:26.176385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111203+0000","last_change":"2023-10-14T17:58:26.104040+0000","last_active":"2023-10-14T17:58:26.111203+0000","last_peered":"2023-10-14T17:58:26.111203+0000","last_clean":"2023-10-14T17:58:26.111203+0000","last_became_active":"2023-10-14T17:58:26.103693+0000","last_became_peered":"2023-10-14T17:58:26.103693+0000","last_unstale":"2023-10-14T17:58:26.111203+0000","last_undegraded":"2023-10-14T17:58:26.111203+0000","last_fullsized":"2023-10-14T17:58:26.111203+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:18:21.922333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"191'4","reported_seq":15,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.127718+0000","last_change":"2023-10-14T17:58:25.082945+0000","last_active":"2023-10-14T17:58:26.127718+0000","last_peered":"2023-10-14T17:58:26.127718+0000","last_clean":"2023-10-14T17:58:26.127718+0000","last_became_active":"2023-10-14T17:58:25.082562+0000","last_became_peered":"2023-10-14T17:58:25.082562+0000","last_unstale":"2023-10-14T17:58:26.127718+0000","last_undegraded":"2023-10-14T17:58:26.127718+0000","last_fullsized":"2023-10-14T17:58:26.127718+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:27.328801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118816+0000","last_change":"2023-10-14T17:58:26.121680+0000","last_active":"2023-10-14T17:58:27.118816+0000","last_peered":"2023-10-14T17:58:27.118816+0000","last_clean":"2023-10-14T17:58:27.118816+0000","last_became_active":"2023-10-14T17:58:26.121389+0000","last_became_peered":"2023-10-14T17:58:26.121389+0000","last_unstale":"2023-10-14T17:58:27.118816+0000","last_undegraded":"2023-10-14T17:58:27.118816+0000","last_fullsized":"2023-10-14T17:58:27.118816+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:00.443453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119235+0000","last_change":"2023-10-14T17:58:25.071198+0000","last_active":"2023-10-14T17:58:27.119235+0000","last_peered":"2023-10-14T17:58:27.119235+0000","last_clean":"2023-10-14T17:58:27.119235+0000","last_became_active":"2023-10-14T17:58:25.070911+0000","last_became_peered":"2023-10-14T17:58:25.070911+0000","last_unstale":"2023-10-14T17:58:27.119235+0000","last_undegraded":"2023-10-14T17:58:27.119235+0000","last_fullsized":"2023-10-14T17:58:27.119235+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:16.269268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730615+0000","last_change":"2023-10-14T17:58:26.113256+0000","last_active":"2023-10-14T17:58:27.730615+0000","last_peered":"2023-10-14T17:58:27.730615+0000","last_clean":"2023-10-14T17:58:27.730615+0000","last_became_active":"2023-10-14T17:58:26.111849+0000","last_became_peered":"2023-10-14T17:58:26.111849+0000","last_unstale":"2023-10-14T17:58:27.730615+0000","last_undegraded":"2023-10-14T17:58:27.730615+0000","last_fullsized":"2023-10-14T17:58:27.730615+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:29.234136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801883+0000","last_change":"2023-10-14T17:58:25.086732+0000","last_active":"2023-10-14T17:58:27.801883+0000","last_peered":"2023-10-14T17:58:27.801883+0000","last_clean":"2023-10-14T17:58:27.801883+0000","last_became_active":"2023-10-14T17:58:25.086427+0000","last_became_peered":"2023-10-14T17:58:25.086427+0000","last_unstale":"2023-10-14T17:58:27.801883+0000","last_undegraded":"2023-10-14T17:58:27.801883+0000","last_fullsized":"2023-10-14T17:58:27.801883+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:31.520308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117563+0000","last_change":"2023-10-14T17:58:26.114058+0000","last_active":"2023-10-14T17:58:27.117563+0000","last_peered":"2023-10-14T17:58:27.117563+0000","last_clean":"2023-10-14T17:58:27.117563+0000","last_became_active":"2023-10-14T17:58:26.113778+0000","last_became_peered":"2023-10-14T17:58:26.113778+0000","last_unstale":"2023-10-14T17:58:27.117563+0000","last_undegraded":"2023-10-14T17:58:27.117563+0000","last_fullsized":"2023-10-14T17:58:27.117563+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:09:41.991486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802518+0000","last_change":"2023-10-14T17:58:25.087826+0000","last_active":"2023-10-14T17:58:27.802518+0000","last_peered":"2023-10-14T17:58:27.802518+0000","last_clean":"2023-10-14T17:58:27.802518+0000","last_became_active":"2023-10-14T17:58:25.087551+0000","last_became_peered":"2023-10-14T17:58:25.087551+0000","last_unstale":"2023-10-14T17:58:27.802518+0000","last_undegraded":"2023-10-14T17:58:27.802518+0000","last_fullsized":"2023-10-14T17:58:27.802518+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:37:26.326645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.124612+0000","last_change":"2023-10-14T17:58:26.121290+0000","last_active":"2023-10-14T17:58:26.124612+0000","last_peered":"2023-10-14T17:58:26.124612+0000","last_clean":"2023-10-14T17:58:26.124612+0000","last_became_active":"2023-10-14T17:58:26.120995+0000","last_became_peered":"2023-10-14T17:58:26.120995+0000","last_unstale":"2023-10-14T17:58:26.124612+0000","last_undegraded":"2023-10-14T17:58:26.124612+0000","last_fullsized":"2023-10-14T17:58:26.124612+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:35:16.035273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116345+0000","last_change":"2023-10-14T17:58:25.086816+0000","last_active":"2023-10-14T17:58:27.116345+0000","last_peered":"2023-10-14T17:58:27.116345+0000","last_clean":"2023-10-14T17:58:27.116345+0000","last_became_active":"2023-10-14T17:58:25.085155+0000","last_became_peered":"2023-10-14T17:58:25.085155+0000","last_unstale":"2023-10-14T17:58:27.116345+0000","last_undegraded":"2023-10-14T17:58:27.116345+0000","last_fullsized":"2023-10-14T17:58:27.116345+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:36.367182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801932+0000","last_change":"2023-10-14T17:58:26.123450+0000","last_active":"2023-10-14T17:58:27.801932+0000","last_peered":"2023-10-14T17:58:27.801932+0000","last_clean":"2023-10-14T17:58:27.801932+0000","last_became_active":"2023-10-14T17:58:26.123168+0000","last_became_peered":"2023-10-14T17:58:26.123168+0000","last_unstale":"2023-10-14T17:58:27.801932+0000","last_undegraded":"2023-10-14T17:58:27.801932+0000","last_fullsized":"2023-10-14T17:58:27.801932+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:41.396701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801206+0000","last_change":"2023-10-14T17:58:25.084285+0000","last_active":"2023-10-14T17:58:27.801206+0000","last_peered":"2023-10-14T17:58:27.801206+0000","last_clean":"2023-10-14T17:58:27.801206+0000","last_became_active":"2023-10-14T17:58:25.083668+0000","last_became_peered":"2023-10-14T17:58:25.083668+0000","last_unstale":"2023-10-14T17:58:27.801206+0000","last_undegraded":"2023-10-14T17:58:27.801206+0000","last_fullsized":"2023-10-14T17:58:27.801206+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:48.033775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801137+0000","last_change":"2023-10-14T17:58:26.108282+0000","last_active":"2023-10-14T17:58:27.801137+0000","last_peered":"2023-10-14T17:58:27.801137+0000","last_clean":"2023-10-14T17:58:27.801137+0000","last_became_active":"2023-10-14T17:58:26.107969+0000","last_became_peered":"2023-10-14T17:58:26.107969+0000","last_unstale":"2023-10-14T17:58:27.801137+0000","last_undegraded":"2023-10-14T17:58:27.801137+0000","last_fullsized":"2023-10-14T17:58:27.801137+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:55:27.865214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120148+0000","last_change":"2023-10-14T17:58:25.085103+0000","last_active":"2023-10-14T17:58:27.120148+0000","last_peered":"2023-10-14T17:58:27.120148+0000","last_clean":"2023-10-14T17:58:27.120148+0000","last_became_active":"2023-10-14T17:58:25.084773+0000","last_became_peered":"2023-10-14T17:58:25.084773+0000","last_unstale":"2023-10-14T17:58:27.120148+0000","last_undegraded":"2023-10-14T17:58:27.120148+0000","last_fullsized":"2023-10-14T17:58:27.120148+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:51.346979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730772+0000","last_change":"2023-10-14T17:58:26.117661+0000","last_active":"2023-10-14T17:58:27.730772+0000","last_peered":"2023-10-14T17:58:27.730772+0000","last_clean":"2023-10-14T17:58:27.730772+0000","last_became_active":"2023-10-14T17:58:26.116188+0000","last_became_peered":"2023-10-14T17:58:26.116188+0000","last_unstale":"2023-10-14T17:58:27.730772+0000","last_undegraded":"2023-10-14T17:58:27.730772+0000","last_fullsized":"2023-10-14T17:58:27.730772+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:25:07.514050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.093052+0000","last_change":"2023-10-14T17:58:25.072573+0000","last_active":"2023-10-14T17:58:26.093052+0000","last_peered":"2023-10-14T17:58:26.093052+0000","last_clean":"2023-10-14T17:58:26.093052+0000","last_became_active":"2023-10-14T17:58:25.072216+0000","last_became_peered":"2023-10-14T17:58:25.072216+0000","last_unstale":"2023-10-14T17:58:26.093052+0000","last_undegraded":"2023-10-14T17:58:26.093052+0000","last_fullsized":"2023-10-14T17:58:26.093052+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:51.346979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131509+0000","last_change":"2023-10-14T17:58:26.128319+0000","last_active":"2023-10-14T17:58:26.131509+0000","last_peered":"2023-10-14T17:58:26.131509+0000","last_clean":"2023-10-14T17:58:26.131509+0000","last_became_active":"2023-10-14T17:58:26.127922+0000","last_became_peered":"2023-10-14T17:58:26.127922+0000","last_unstale":"2023-10-14T17:58:26.131509+0000","last_undegraded":"2023-10-14T17:58:26.131509+0000","last_fullsized":"2023-10-14T17:58:26.131509+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:12.520592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731189+0000","last_change":"2023-10-14T17:58:25.083253+0000","last_active":"2023-10-14T17:58:27.731189+0000","last_peered":"2023-10-14T17:58:27.731189+0000","last_clean":"2023-10-14T17:58:27.731189+0000","last_became_active":"2023-10-14T17:58:25.082917+0000","last_became_peered":"2023-10-14T17:58:25.082917+0000","last_unstale":"2023-10-14T17:58:27.731189+0000","last_undegraded":"2023-10-14T17:58:27.731189+0000","last_fullsized":"2023-10-14T17:58:27.731189+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:02.245682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131171+0000","last_change":"2023-10-14T17:58:26.129382+0000","last_active":"2023-10-14T17:58:26.131171+0000","last_peered":"2023-10-14T17:58:26.131171+0000","last_clean":"2023-10-14T17:58:26.131171+0000","last_became_active":"2023-10-14T17:58:26.129040+0000","last_became_peered":"2023-10-14T17:58:26.129040+0000","last_unstale":"2023-10-14T17:58:26.131171+0000","last_undegraded":"2023-10-14T17:58:26.131171+0000","last_fullsized":"2023-10-14T17:58:26.131171+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:28:27.345489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"191'3","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731012+0000","last_change":"2023-10-14T17:58:25.079049+0000","last_active":"2023-10-14T17:58:27.731012+0000","last_peered":"2023-10-14T17:58:27.731012+0000","last_clean":"2023-10-14T17:58:27.731012+0000","last_became_active":"2023-10-14T17:58:25.078646+0000","last_became_peered":"2023-10-14T17:58:25.078646+0000","last_unstale":"2023-10-14T17:58:27.731012+0000","last_undegraded":"2023-10-14T17:58:27.731012+0000","last_fullsized":"2023-10-14T17:58:27.731012+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:58.687645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119465+0000","last_change":"2023-10-14T17:58:26.105197+0000","last_active":"2023-10-14T17:58:27.119465+0000","last_peered":"2023-10-14T17:58:27.119465+0000","last_clean":"2023-10-14T17:58:27.119465+0000","last_became_active":"2023-10-14T17:58:26.104900+0000","last_became_peered":"2023-10-14T17:58:26.104900+0000","last_unstale":"2023-10-14T17:58:27.119465+0000","last_undegraded":"2023-10-14T17:58:27.119465+0000","last_fullsized":"2023-10-14T17:58:27.119465+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:37:27.341977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"191'6","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118214+0000","last_change":"2023-10-14T17:58:25.076118+0000","last_active":"2023-10-14T17:58:27.118214+0000","last_peered":"2023-10-14T17:58:27.118214+0000","last_clean":"2023-10-14T17:58:27.118214+0000","last_became_active":"2023-10-14T17:58:25.075743+0000","last_became_peered":"2023-10-14T17:58:25.075743+0000","last_unstale":"2023-10-14T17:58:27.118214+0000","last_undegraded":"2023-10-14T17:58:27.118214+0000","last_fullsized":"2023-10-14T17:58:27.118214+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:31.520308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122506+0000","last_change":"2023-10-14T17:58:26.119934+0000","last_active":"2023-10-14T17:58:26.122506+0000","last_peered":"2023-10-14T17:58:26.122506+0000","last_clean":"2023-10-14T17:58:26.122506+0000","last_became_active":"2023-10-14T17:58:26.119609+0000","last_became_peered":"2023-10-14T17:58:26.119609+0000","last_unstale":"2023-10-14T17:58:26.122506+0000","last_undegraded":"2023-10-14T17:58:26.122506+0000","last_fullsized":"2023-10-14T17:58:26.122506+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:52.830856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801715+0000","last_change":"2023-10-14T17:58:25.085283+0000","last_active":"2023-10-14T17:58:27.801715+0000","last_peered":"2023-10-14T17:58:27.801715+0000","last_clean":"2023-10-14T17:58:27.801715+0000","last_became_active":"2023-10-14T17:58:25.085004+0000","last_became_peered":"2023-10-14T17:58:25.085004+0000","last_unstale":"2023-10-14T17:58:27.801715+0000","last_undegraded":"2023-10-14T17:58:27.801715+0000","last_fullsized":"2023-10-14T17:58:27.801715+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:16.800712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123005+0000","last_change":"2023-10-14T17:58:26.115624+0000","last_active":"2023-10-14T17:58:26.123005+0000","last_peered":"2023-10-14T17:58:26.123005+0000","last_clean":"2023-10-14T17:58:26.123005+0000","last_became_active":"2023-10-14T17:58:26.115298+0000","last_became_peered":"2023-10-14T17:58:26.115298+0000","last_unstale":"2023-10-14T17:58:26.123005+0000","last_undegraded":"2023-10-14T17:58:26.123005+0000","last_fullsized":"2023-10-14T17:58:26.123005+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:36.150563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.109577+0000","last_change":"2023-10-14T17:58:25.080153+0000","last_active":"2023-10-14T17:58:26.109577+0000","last_peered":"2023-10-14T17:58:26.109577+0000","last_clean":"2023-10-14T17:58:26.109577+0000","last_became_active":"2023-10-14T17:58:25.079559+0000","last_became_peered":"2023-10-14T17:58:25.079559+0000","last_unstale":"2023-10-14T17:58:26.109577+0000","last_undegraded":"2023-10-14T17:58:26.109577+0000","last_fullsized":"2023-10-14T17:58:26.109577+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:48.554913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801203+0000","last_change":"2023-10-14T17:58:26.105513+0000","last_active":"2023-10-14T17:58:27.801203+0000","last_peered":"2023-10-14T17:58:27.801203+0000","last_clean":"2023-10-14T17:58:27.801203+0000","last_became_active":"2023-10-14T17:58:26.103936+0000","last_became_peered":"2023-10-14T17:58:26.103936+0000","last_unstale":"2023-10-14T17:58:27.801203+0000","last_undegraded":"2023-10-14T17:58:27.801203+0000","last_fullsized":"2023-10-14T17:58:27.801203+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:54.799910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801334+0000","last_change":"2023-10-14T17:58:25.089220+0000","last_active":"2023-10-14T17:58:27.801334+0000","last_peered":"2023-10-14T17:58:27.801334+0000","last_clean":"2023-10-14T17:58:27.801334+0000","last_became_active":"2023-10-14T17:58:25.088965+0000","last_became_peered":"2023-10-14T17:58:25.088965+0000","last_unstale":"2023-10-14T17:58:27.801334+0000","last_undegraded":"2023-10-14T17:58:27.801334+0000","last_fullsized":"2023-10-14T17:58:27.801334+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:54.056721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115735+0000","last_change":"2023-10-14T17:58:26.109808+0000","last_active":"2023-10-14T17:58:27.115735+0000","last_peered":"2023-10-14T17:58:27.115735+0000","last_clean":"2023-10-14T17:58:27.115735+0000","last_became_active":"2023-10-14T17:58:26.108357+0000","last_became_peered":"2023-10-14T17:58:26.108357+0000","last_unstale":"2023-10-14T17:58:27.115735+0000","last_undegraded":"2023-10-14T17:58:27.115735+0000","last_fullsized":"2023-10-14T17:58:27.115735+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:30.295018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"191'8","reported_seq":20,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117846+0000","last_change":"2023-10-14T17:58:25.078724+0000","last_active":"2023-10-14T17:58:27.117846+0000","last_peered":"2023-10-14T17:58:27.117846+0000","last_clean":"2023-10-14T17:58:27.117846+0000","last_became_active":"2023-10-14T17:58:25.076993+0000","last_became_peered":"2023-10-14T17:58:25.076993+0000","last_unstale":"2023-10-14T17:58:27.117846+0000","last_undegraded":"2023-10-14T17:58:27.117846+0000","last_fullsized":"2023-10-14T17:58:27.117846+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:43.862488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802525+0000","last_change":"2023-10-14T17:58:26.119282+0000","last_active":"2023-10-14T17:58:27.802525+0000","last_peered":"2023-10-14T17:58:27.802525+0000","last_clean":"2023-10-14T17:58:27.802525+0000","last_became_active":"2023-10-14T17:58:26.110912+0000","last_became_peered":"2023-10-14T17:58:26.110912+0000","last_unstale":"2023-10-14T17:58:27.802525+0000","last_undegraded":"2023-10-14T17:58:27.802525+0000","last_fullsized":"2023-10-14T17:58:27.802525+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:14.240634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117139+0000","last_change":"2023-10-14T17:58:25.084919+0000","last_active":"2023-10-14T17:58:27.117139+0000","last_peered":"2023-10-14T17:58:27.117139+0000","last_clean":"2023-10-14T17:58:27.117139+0000","last_became_active":"2023-10-14T17:58:25.084622+0000","last_became_peered":"2023-10-14T17:58:25.084622+0000","last_unstale":"2023-10-14T17:58:27.117139+0000","last_undegraded":"2023-10-14T17:58:27.117139+0000","last_fullsized":"2023-10-14T17:58:27.117139+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:48.874818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122345+0000","last_change":"2023-10-14T17:58:26.117393+0000","last_active":"2023-10-14T17:58:26.122345+0000","last_peered":"2023-10-14T17:58:26.122345+0000","last_clean":"2023-10-14T17:58:26.122345+0000","last_became_active":"2023-10-14T17:58:26.113067+0000","last_became_peered":"2023-10-14T17:58:26.113067+0000","last_unstale":"2023-10-14T17:58:26.122345+0000","last_undegraded":"2023-10-14T17:58:26.122345+0000","last_fullsized":"2023-10-14T17:58:26.122345+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:59:24.472886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115783+0000","last_change":"2023-10-14T17:58:25.083610+0000","last_active":"2023-10-14T17:58:27.115783+0000","last_peered":"2023-10-14T17:58:27.115783+0000","last_clean":"2023-10-14T17:58:27.115783+0000","last_became_active":"2023-10-14T17:58:25.083262+0000","last_became_peered":"2023-10-14T17:58:25.083262+0000","last_unstale":"2023-10-14T17:58:27.115783+0000","last_undegraded":"2023-10-14T17:58:27.115783+0000","last_fullsized":"2023-10-14T17:58:27.115783+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:30:02.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802431+0000","last_change":"2023-10-14T17:58:26.127380+0000","last_active":"2023-10-14T17:58:27.802431+0000","last_peered":"2023-10-14T17:58:27.802431+0000","last_clean":"2023-10-14T17:58:27.802431+0000","last_became_active":"2023-10-14T17:58:26.125547+0000","last_became_peered":"2023-10-14T17:58:26.125547+0000","last_unstale":"2023-10-14T17:58:27.802431+0000","last_undegraded":"2023-10-14T17:58:27.802431+0000","last_fullsized":"2023-10-14T17:58:27.802431+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:43.401744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"191'1","reported_seq":12,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.126886+0000","last_change":"2023-10-14T17:58:25.086558+0000","last_active":"2023-10-14T17:58:26.126886+0000","last_peered":"2023-10-14T17:58:26.126886+0000","last_clean":"2023-10-14T17:58:26.126886+0000","last_became_active":"2023-10-14T17:58:25.086220+0000","last_became_peered":"2023-10-14T17:58:25.086220+0000","last_unstale":"2023-10-14T17:58:26.126886+0000","last_undegraded":"2023-10-14T17:58:26.126886+0000","last_fullsized":"2023-10-14T17:58:26.126886+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:19.803554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123099+0000","last_change":"2023-10-14T17:58:26.116810+0000","last_active":"2023-10-14T17:58:26.123099+0000","last_peered":"2023-10-14T17:58:26.123099+0000","last_clean":"2023-10-14T17:58:26.123099+0000","last_became_active":"2023-10-14T17:58:26.116497+0000","last_became_peered":"2023-10-14T17:58:26.116497+0000","last_unstale":"2023-10-14T17:58:26.123099+0000","last_undegraded":"2023-10-14T17:58:26.123099+0000","last_fullsized":"2023-10-14T17:58:26.123099+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:59:57.367583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.102613+0000","last_change":"2023-10-14T17:58:25.083633+0000","last_active":"2023-10-14T17:58:26.102613+0000","last_peered":"2023-10-14T17:58:26.102613+0000","last_clean":"2023-10-14T17:58:26.102613+0000","last_became_active":"2023-10-14T17:58:25.083361+0000","last_became_peered":"2023-10-14T17:58:25.083361+0000","last_unstale":"2023-10-14T17:58:26.102613+0000","last_undegraded":"2023-10-14T17:58:26.102613+0000","last_fullsized":"2023-10-14T17:58:26.102613+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:46:29.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115675+0000","last_change":"2023-10-14T17:58:26.124910+0000","last_active":"2023-10-14T17:58:27.115675+0000","last_peered":"2023-10-14T17:58:27.115675+0000","last_clean":"2023-10-14T17:58:27.115675+0000","last_became_active":"2023-10-14T17:58:26.122262+0000","last_became_peered":"2023-10-14T17:58:26.122262+0000","last_unstale":"2023-10-14T17:58:27.115675+0000","last_undegraded":"2023-10-14T17:58:27.115675+0000","last_fullsized":"2023-10-14T17:58:27.115675+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:59.702977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802090+0000","last_change":"2023-10-14T17:58:25.088861+0000","last_active":"2023-10-14T17:58:27.802090+0000","last_peered":"2023-10-14T17:58:27.802090+0000","last_clean":"2023-10-14T17:58:27.802090+0000","last_became_active":"2023-10-14T17:58:25.088483+0000","last_became_peered":"2023-10-14T17:58:25.088483+0000","last_unstale":"2023-10-14T17:58:27.802090+0000","last_undegraded":"2023-10-14T17:58:27.802090+0000","last_fullsized":"2023-10-14T17:58:27.802090+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:58.833729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115729+0000","last_change":"2023-10-14T17:58:26.106849+0000","last_active":"2023-10-14T17:58:26.115729+0000","last_peered":"2023-10-14T17:58:26.115729+0000","last_clean":"2023-10-14T17:58:26.115729+0000","last_became_active":"2023-10-14T17:58:26.106569+0000","last_became_peered":"2023-10-14T17:58:26.106569+0000","last_unstale":"2023-10-14T17:58:26.115729+0000","last_undegraded":"2023-10-14T17:58:26.115729+0000","last_fullsized":"2023-10-14T17:58:26.115729+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:23:17.626488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.103609+0000","last_change":"2023-10-14T17:58:25.080883+0000","last_active":"2023-10-14T17:58:26.103609+0000","last_peered":"2023-10-14T17:58:26.103609+0000","last_clean":"2023-10-14T17:58:26.103609+0000","last_became_active":"2023-10-14T17:58:25.080146+0000","last_became_peered":"2023-10-14T17:58:25.080146+0000","last_unstale":"2023-10-14T17:58:26.103609+0000","last_undegraded":"2023-10-14T17:58:26.103609+0000","last_fullsized":"2023-10-14T17:58:26.103609+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:28.193611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111385+0000","last_change":"2023-10-14T17:58:26.109913+0000","last_active":"2023-10-14T17:58:26.111385+0000","last_peered":"2023-10-14T17:58:26.111385+0000","last_clean":"2023-10-14T17:58:26.111385+0000","last_became_active":"2023-10-14T17:58:26.109641+0000","last_became_peered":"2023-10-14T17:58:26.109641+0000","last_unstale":"2023-10-14T17:58:26.111385+0000","last_undegraded":"2023-10-14T17:58:26.111385+0000","last_fullsized":"2023-10-14T17:58:26.111385+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:44.265730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801412+0000","last_change":"2023-10-14T17:58:25.085129+0000","last_active":"2023-10-14T17:58:27.801412+0000","last_peered":"2023-10-14T17:58:27.801412+0000","last_clean":"2023-10-14T17:58:27.801412+0000","last_became_active":"2023-10-14T17:58:25.084548+0000","last_became_peered":"2023-10-14T17:58:25.084548+0000","last_unstale":"2023-10-14T17:58:27.801412+0000","last_undegraded":"2023-10-14T17:58:27.801412+0000","last_fullsized":"2023-10-14T17:58:27.801412+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:53.579044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730542+0000","last_change":"2023-10-14T17:58:26.117900+0000","last_active":"2023-10-14T17:58:27.730542+0000","last_peered":"2023-10-14T17:58:27.730542+0000","last_clean":"2023-10-14T17:58:27.730542+0000","last_became_active":"2023-10-14T17:58:26.117146+0000","last_became_peered":"2023-10-14T17:58:26.117146+0000","last_unstale":"2023-10-14T17:58:27.730542+0000","last_undegraded":"2023-10-14T17:58:27.730542+0000","last_fullsized":"2023-10-14T17:58:27.730542+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:25.563550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.101393+0000","last_change":"2023-10-14T17:58:25.085659+0000","last_active":"2023-10-14T17:58:26.101393+0000","last_peered":"2023-10-14T17:58:26.101393+0000","last_clean":"2023-10-14T17:58:26.101393+0000","last_became_active":"2023-10-14T17:58:25.085340+0000","last_became_peered":"2023-10-14T17:58:25.085340+0000","last_unstale":"2023-10-14T17:58:26.101393+0000","last_undegraded":"2023-10-14T17:58:26.101393+0000","last_fullsized":"2023-10-14T17:58:26.101393+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:33:07.933202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.130396+0000","last_change":"2023-10-14T17:58:26.126937+0000","last_active":"2023-10-14T17:58:26.130396+0000","last_peered":"2023-10-14T17:58:26.130396+0000","last_clean":"2023-10-14T17:58:26.130396+0000","last_became_active":"2023-10-14T17:58:26.126640+0000","last_became_peered":"2023-10-14T17:58:26.126640+0000","last_unstale":"2023-10-14T17:58:26.130396+0000","last_undegraded":"2023-10-14T17:58:26.130396+0000","last_fullsized":"2023-10-14T17:58:26.130396+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:46.255482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"191'2","reported_seq":13,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.132312+0000","last_change":"2023-10-14T17:58:25.076934+0000","last_active":"2023-10-14T17:58:26.132312+0000","last_peered":"2023-10-14T17:58:26.132312+0000","last_clean":"2023-10-14T17:58:26.132312+0000","last_became_active":"2023-10-14T17:58:25.076482+0000","last_became_peered":"2023-10-14T17:58:25.076482+0000","last_unstale":"2023-10-14T17:58:26.132312+0000","last_undegraded":"2023-10-14T17:58:26.132312+0000","last_fullsized":"2023-10-14T17:58:26.132312+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:25.543596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111288+0000","last_change":"2023-10-14T17:58:26.105200+0000","last_active":"2023-10-14T17:58:26.111288+0000","last_peered":"2023-10-14T17:58:26.111288+0000","last_clean":"2023-10-14T17:58:26.111288+0000","last_became_active":"2023-10-14T17:58:26.102252+0000","last_became_peered":"2023-10-14T17:58:26.102252+0000","last_unstale":"2023-10-14T17:58:26.111288+0000","last_undegraded":"2023-10-14T17:58:26.111288+0000","last_fullsized":"2023-10-14T17:58:26.111288+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:30:03.043323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115998+0000","last_change":"2023-10-14T17:58:25.081398+0000","last_active":"2023-10-14T17:58:27.115998+0000","last_peered":"2023-10-14T17:58:27.115998+0000","last_clean":"2023-10-14T17:58:27.115998+0000","last_became_active":"2023-10-14T17:58:25.080936+0000","last_became_peered":"2023-10-14T17:58:25.080936+0000","last_unstale":"2023-10-14T17:58:27.115998+0000","last_undegraded":"2023-10-14T17:58:27.115998+0000","last_fullsized":"2023-10-14T17:58:27.115998+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:30.162330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730905+0000","last_change":"2023-10-14T17:58:26.105754+0000","last_active":"2023-10-14T17:58:27.730905+0000","last_peered":"2023-10-14T17:58:27.730905+0000","last_clean":"2023-10-14T17:58:27.730905+0000","last_became_active":"2023-10-14T17:58:26.105466+0000","last_became_peered":"2023-10-14T17:58:26.105466+0000","last_unstale":"2023-10-14T17:58:27.730905+0000","last_undegraded":"2023-10-14T17:58:27.730905+0000","last_fullsized":"2023-10-14T17:58:27.730905+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:57:27.045397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120717+0000","last_change":"2023-10-14T17:58:25.067263+0000","last_active":"2023-10-14T17:58:27.120717+0000","last_peered":"2023-10-14T17:58:27.120717+0000","last_clean":"2023-10-14T17:58:27.120717+0000","last_became_active":"2023-10-14T17:58:25.065774+0000","last_became_peered":"2023-10-14T17:58:25.065774+0000","last_unstale":"2023-10-14T17:58:27.120717+0000","last_undegraded":"2023-10-14T17:58:27.120717+0000","last_fullsized":"2023-10-14T17:58:27.120717+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:35.135231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122477+0000","last_change":"2023-10-14T17:58:26.119542+0000","last_active":"2023-10-14T17:58:26.122477+0000","last_peered":"2023-10-14T17:58:26.122477+0000","last_clean":"2023-10-14T17:58:26.122477+0000","last_became_active":"2023-10-14T17:58:26.117553+0000","last_became_peered":"2023-10-14T17:58:26.117553+0000","last_unstale":"2023-10-14T17:58:26.122477+0000","last_undegraded":"2023-10-14T17:58:26.122477+0000","last_fullsized":"2023-10-14T17:58:26.122477+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:53.826190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.116106+0000","last_change":"2023-10-14T17:58:25.080562+0000","last_active":"2023-10-14T17:58:26.116106+0000","last_peered":"2023-10-14T17:58:26.116106+0000","last_clean":"2023-10-14T17:58:26.116106+0000","last_became_active":"2023-10-14T17:58:25.080254+0000","last_became_peered":"2023-10-14T17:58:25.080254+0000","last_unstale":"2023-10-14T17:58:26.116106+0000","last_undegraded":"2023-10-14T17:58:26.116106+0000","last_fullsized":"2023-10-14T17:58:26.116106+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:59.855683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.127057+0000","last_change":"2023-10-14T17:58:26.125490+0000","last_active":"2023-10-14T17:58:26.127057+0000","last_peered":"2023-10-14T17:58:26.127057+0000","last_clean":"2023-10-14T17:58:26.127057+0000","last_became_active":"2023-10-14T17:58:26.124256+0000","last_became_peered":"2023-10-14T17:58:26.124256+0000","last_unstale":"2023-10-14T17:58:26.127057+0000","last_undegraded":"2023-10-14T17:58:26.127057+0000","last_fullsized":"2023-10-14T17:58:26.127057+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:25.405674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118791+0000","last_change":"2023-10-14T17:58:25.065584+0000","last_active":"2023-10-14T17:58:27.118791+0000","last_peered":"2023-10-14T17:58:27.118791+0000","last_clean":"2023-10-14T17:58:27.118791+0000","last_became_active":"2023-10-14T17:58:25.065056+0000","last_became_peered":"2023-10-14T17:58:25.065056+0000","last_unstale":"2023-10-14T17:58:27.118791+0000","last_undegraded":"2023-10-14T17:58:27.118791+0000","last_fullsized":"2023-10-14T17:58:27.118791+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:48.033775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118558+0000","last_change":"2023-10-14T17:58:26.108381+0000","last_active":"2023-10-14T17:58:27.118558+0000","last_peered":"2023-10-14T17:58:27.118558+0000","last_clean":"2023-10-14T17:58:27.118558+0000","last_became_active":"2023-10-14T17:58:26.104571+0000","last_became_peered":"2023-10-14T17:58:26.104571+0000","last_unstale":"2023-10-14T17:58:27.118558+0000","last_undegraded":"2023-10-14T17:58:27.118558+0000","last_fullsized":"2023-10-14T17:58:27.118558+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:40:27.777182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801489+0000","last_change":"2023-10-14T17:58:25.086607+0000","last_active":"2023-10-14T17:58:27.801489+0000","last_peered":"2023-10-14T17:58:27.801489+0000","last_clean":"2023-10-14T17:58:27.801489+0000","last_became_active":"2023-10-14T17:58:25.086007+0000","last_became_peered":"2023-10-14T17:58:25.086007+0000","last_unstale":"2023-10-14T17:58:27.801489+0000","last_undegraded":"2023-10-14T17:58:27.801489+0000","last_fullsized":"2023-10-14T17:58:27.801489+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:43.862488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115222+0000","last_change":"2023-10-14T17:58:26.111241+0000","last_active":"2023-10-14T17:58:26.115222+0000","last_peered":"2023-10-14T17:58:26.115222+0000","last_clean":"2023-10-14T17:58:26.115222+0000","last_became_active":"2023-10-14T17:58:26.110733+0000","last_became_peered":"2023-10-14T17:58:26.110733+0000","last_unstale":"2023-10-14T17:58:26.115222+0000","last_undegraded":"2023-10-14T17:58:26.115222+0000","last_fullsized":"2023-10-14T17:58:26.115222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:49.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120469+0000","last_change":"2023-10-14T17:58:25.065905+0000","last_active":"2023-10-14T17:58:27.120469+0000","last_peered":"2023-10-14T17:58:27.120469+0000","last_clean":"2023-10-14T17:58:27.120469+0000","last_became_active":"2023-10-14T17:58:25.064248+0000","last_became_peered":"2023-10-14T17:58:25.064248+0000","last_unstale":"2023-10-14T17:58:27.120469+0000","last_undegraded":"2023-10-14T17:58:27.120469+0000","last_fullsized":"2023-10-14T17:58:27.120469+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:02.245682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116101+0000","last_change":"2023-10-14T17:58:26.120747+0000","last_active":"2023-10-14T17:58:27.116101+0000","last_peered":"2023-10-14T17:58:27.116101+0000","last_clean":"2023-10-14T17:58:27.116101+0000","last_became_active":"2023-10-14T17:58:26.120447+0000","last_became_peered":"2023-10-14T17:58:26.120447+0000","last_unstale":"2023-10-14T17:58:27.116101+0000","last_undegraded":"2023-10-14T17:58:27.116101+0000","last_fullsized":"2023-10-14T17:58:27.116101+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:35.001471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117953+0000","last_change":"2023-10-14T17:58:25.069216+0000","last_active":"2023-10-14T17:58:27.117953+0000","last_peered":"2023-10-14T17:58:27.117953+0000","last_clean":"2023-10-14T17:58:27.117953+0000","last_became_active":"2023-10-14T17:58:25.068597+0000","last_became_peered":"2023-10-14T17:58:25.068597+0000","last_unstale":"2023-10-14T17:58:27.117953+0000","last_undegraded":"2023-10-14T17:58:27.117953+0000","last_fullsized":"2023-10-14T17:58:27.117953+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:27.910859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131610+0000","last_change":"2023-10-14T17:58:26.130535+0000","last_active":"2023-10-14T17:58:26.131610+0000","last_peered":"2023-10-14T17:58:26.131610+0000","last_clean":"2023-10-14T17:58:26.131610+0000","last_became_active":"2023-10-14T17:58:26.130063+0000","last_became_peered":"2023-10-14T17:58:26.130063+0000","last_unstale":"2023-10-14T17:58:26.131610+0000","last_undegraded":"2023-10-14T17:58:26.131610+0000","last_fullsized":"2023-10-14T17:58:26.131610+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:54.104390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119028+0000","last_change":"2023-10-14T17:58:25.086748+0000","last_active":"2023-10-14T17:58:27.119028+0000","last_peered":"2023-10-14T17:58:27.119028+0000","last_clean":"2023-10-14T17:58:27.119028+0000","last_became_active":"2023-10-14T17:58:25.086402+0000","last_became_peered":"2023-10-14T17:58:25.086402+0000","last_unstale":"2023-10-14T17:58:27.119028+0000","last_undegraded":"2023-10-14T17:58:27.119028+0000","last_fullsized":"2023-10-14T17:58:27.119028+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:58.833729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"180'293","reported_seq":972,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-14T17:58:25.073378+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:58:25.073378+0000","last_peered":"2023-10-14T17:58:25.073378+0000","last_clean":"2023-10-14T17:58:25.073378+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:58:25.073378+0000","last_undegraded":"2023-10-14T17:58:25.073378+0000","last_fullsized":"2023-10-14T17:58:25.073378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575780,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52164,"kb_used_data":2936,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319676,"statfs":{"total":96636764160,"available":96583348224,"internally_reserved":0,"allocated":3006464,"data_stored":1764267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1830000000000001,"15min":1.2050000000000001},"min":{"1min":0.77300000000000002,"5min":0.73099999999999998,"15min":0.71199999999999997},"max":{"1min":1.429,"5min":1.9019999999999999,"15min":1.9710000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.837},"last":1.008}]},{"osd":1,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.347,"15min":1.3100000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.266,"5min":1.29,"15min":1.306},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5720000000000001,"5min":1.8,"15min":1.804},"last":1.254}]},{"osd":2,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2430000000000001,"15min":1.2509999999999999},"min":{"1min":0.873,"5min":0.82899999999999996,"15min":0.82199999999999995},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.393,"15min":1.3779999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.097}]},{"osd":3,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4199999999999999,"15min":1.4059999999999999},"min":{"1min":0.996,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.883,"5min":1.883,"15min":2.234},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.5429999999999999,"15min":1.5329999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.9410000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.3140000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.335,"15min":1.276},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.403,"15min":1.373},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.5510000000000002},"last":1.524}]},{"osd":5,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.464,"15min":1.458},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0539999999999998},"last":1.474},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4930000000000001,"15min":1.474},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.218,"5min":2.218,"15min":2.6619999999999999},"last":1.7829999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.6140000000000001,"15min":1.585},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4430000000000001},"last":2.117},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.5900000000000001,"15min":1.538},"min":{"1min":1.3440000000000001,"5min":1.0149999999999999,"15min":0.91600000000000004},"max":{"1min":2.1440000000000001,"5min":2.3100000000000001,"15min":2.4100000000000001},"last":1.6739999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":2404352,"data_stored":1177983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.73999999999999999},"max":{"1min":1.4870000000000001,"5min":1.696,"15min":1.837},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.1850000000000001,"15min":1.2010000000000001},"min":{"1min":0.76500000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.401,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.131}]},{"osd":1,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.276,"15min":1.244},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.5640000000000001,"5min":1.752,"15min":1.927},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2989999999999999,"15min":1.3220000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.6040000000000001,"5min":1.7849999999999999,"15min":1.8069999999999999},"last":0.85599999999999998}]},{"osd":2,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86299999999999999},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8740000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.4450000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":1.9370000000000001},"last":1.4059999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.417},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0409999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.393,"15min":1.431},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.764,"5min":1.903,"15min":1.9790000000000001},"last":1.2709999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.361,"15min":1.367},"min":{"1min":1.014,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.736,"5min":1.929,"15min":1.984},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4379999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.889,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6519999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3879999999999999,"15min":1.361},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.79000000000000004},"max":{"1min":1.629,"5min":1.7749999999999999,"15min":2.085},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.5580000000000001},"min":{"1min":1.214,"5min":1.048,"15min":1.048},"max":{"1min":1.9450000000000001,"5min":2.0310000000000001,"15min":2.105},"last":1.6919999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5900000000000001,"15min":1.5840000000000001},"min":{"1min":1.165,"5min":1.026,"15min":1.0189999999999999},"max":{"1min":2.0960000000000001,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.718},{"interface":"front","average":{"1min":1.629,"5min":1.5640000000000001,"15min":1.5720000000000001},"min":{"1min":1.268,"5min":1.0129999999999999,"15min":0.81000000000000005},"max":{"1min":1.8700000000000001,"5min":1.954,"15min":2.1589999999999998},"last":1.585}]}]},{"osd":5,"up_from":13,"seq":55834575780,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37700,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334140,"statfs":{"total":96636764160,"available":96598159360,"internally_reserved":0,"allocated":2408448,"data_stored":1178565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.2070000000000001,"15min":1.1870000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.7949999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.2090000000000001},"min":{"1min":0.85299999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6040000000000001,"5min":1.677,"15min":2.0150000000000001},"last":1.1339999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.623,"5min":1.774,"15min":1.774},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.286,"15min":1.286},"min":{"1min":0.85399999999999998,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.78,"5min":1.7949999999999999,"15min":1.8280000000000001},"last":1.22}]},{"osd":2,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.462,"5min":1.6879999999999999,"15min":1.72},"last":1.044},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.331,"15min":1.353},"min":{"1min":0.91700000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.6779999999999999,"5min":1.899,"15min":2.4140000000000001},"last":1.4390000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.458,"15min":1.444},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.389},{"interface":"front","average":{"1min":1.333,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.655,"5min":1.8799999999999999,"15min":2.5219999999999998},"last":1.595}]},{"osd":4,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3220000000000001,"15min":1.3260000000000001},"min":{"1min":0.89800000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.607,"5min":1.9850000000000001,"15min":2.7210000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.381,"15min":1.393},"min":{"1min":0.96999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.2809999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4119999999999999,"15min":1.3819999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.7949999999999999},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.577,"15min":1.581},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.86599999999999999},"max":{"1min":1.804,"5min":2.1389999999999998,"15min":2.907},"last":1.883}]},{"osd":7,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5509999999999999,"15min":1.544},"min":{"1min":1.073,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7889999999999999,"5min":2.1539999999999999,"15min":2.1760000000000002},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.5049999999999999,"15min":1.516},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.873,"5min":2.0009999999999999,"15min":2.9630000000000001},"last":1.923}]}]},{"osd":4,"up_from":13,"seq":55834575780,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55748,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316092,"statfs":{"total":96636764160,"available":96579678208,"internally_reserved":0,"allocated":2416640,"data_stored":1179643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2689999999999999,"15min":1.2270000000000001},"min":{"1min":0.81000000000000005,"5min":0.73699999999999999,"15min":0.52500000000000002},"max":{"1min":1.732,"5min":1.7749999999999999,"15min":1.8859999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.248,"15min":1.272},"min":{"1min":0.97299999999999998,"5min":0.86699999999999999,"15min":0.84599999999999997},"max":{"1min":1.897,"5min":1.897,"15min":1.905},"last":1.599}]},{"osd":1,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3220000000000001},"min":{"1min":0.89100000000000001,"5min":0.874,"15min":0.83999999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.278,"15min":1.2829999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.7310000000000001,"5min":1.7629999999999999,"15min":2.02},"last":1.03}]},{"osd":2,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.29,"15min":1.298},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.63800000000000001},"max":{"1min":1.827,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.385,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":1.0960000000000001,"5min":1.0209999999999999,"15min":0.84199999999999997},"max":{"1min":1.8680000000000001,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.294}]},{"osd":3,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.347,"15min":1.353},"min":{"1min":0.93000000000000005,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.6439999999999999,"5min":1.752,"15min":1.984},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.492,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":1.0109999999999999,"15min":0.82299999999999995},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.339}]},{"osd":5,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4139999999999999,"15min":1.3640000000000001},"min":{"1min":0.97399999999999998,"5min":0.91500000000000004,"15min":0.82899999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3779999999999999,"15min":1.3740000000000001},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.79700000000000004},"max":{"1min":1.925,"5min":1.925,"15min":2.0499999999999998},"last":1.5249999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4219999999999999,"15min":1.3819999999999999},"min":{"1min":1.004,"5min":0.874,"15min":0.57999999999999996},"max":{"1min":1.9239999999999999,"5min":2.117,"15min":2.117},"last":1.234},{"interface":"front","average":{"1min":1.673,"5min":1.5600000000000001,"15min":1.5349999999999999},"min":{"1min":1.202,"5min":1.133,"15min":1.006},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.6910000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5609999999999999,"15min":1.506},"min":{"1min":1.0780000000000001,"5min":0.92500000000000004,"15min":0.86799999999999999},"max":{"1min":2.0880000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.696,"5min":1.6319999999999999,"15min":1.593},"min":{"1min":1.4410000000000001,"5min":1.179,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.5629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575780,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51568,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320272,"statfs":{"total":96636764160,"available":96583958528,"internally_reserved":0,"allocated":2392064,"data_stored":1171579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.98199999999999998,"15min":0.98099999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.423,"5min":1.7949999999999999,"15min":2.6680000000000001},"last":1.03},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1890000000000001,"15min":1.206},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47999999999999998},"max":{"1min":1.5469999999999999,"5min":1.6240000000000001,"15min":2.8290000000000002},"last":1.3129999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0369999999999999,"15min":1.008},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.843},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.232,"15min":1.21},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7929999999999999,"5min":1.849,"15min":2.1970000000000001},"last":0.95699999999999996}]},{"osd":2,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.454,"15min":1.4550000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6759999999999999,"5min":1.9299999999999999,"15min":2.1000000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.427,"15min":1.4019999999999999},"min":{"1min":0.80000000000000004,"5min":0.79500000000000004,"15min":0.55100000000000005},"max":{"1min":1.8979999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4550000000000001,"15min":1.427},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.92500000000000004},"max":{"1min":1.716,"5min":1.8660000000000001,"15min":2.0489999999999999},"last":1.29},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.48,"15min":1.468},"min":{"1min":0.96599999999999997,"5min":0.96299999999999997,"15min":0.86099999999999999},"max":{"1min":1.9079999999999999,"5min":2.2210000000000001,"15min":2.6459999999999999},"last":1.083}]},{"osd":5,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4590000000000001,"15min":1.478},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.73399999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.5830000000000002},"last":1.004},{"interface":"front","average":{"1min":1.413,"5min":1.514,"15min":1.52},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7949999999999999,"5min":1.9850000000000001,"15min":2.3719999999999999},"last":1.421}]},{"osd":6,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.6699999999999999,"15min":1.6419999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.893,"5min":2.6480000000000001,"15min":2.851},"last":1.274},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.669,"15min":1.679},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0289999999999999,"5min":2.7109999999999999,"15min":2.9910000000000001},"last":1.218}]},{"osd":7,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.643,"15min":1.6200000000000001},"min":{"1min":1.163,"5min":1.163,"15min":0.79500000000000004},"max":{"1min":2.2040000000000002,"5min":2.4140000000000001,"15min":2.9430000000000001},"last":1.246},{"interface":"front","average":{"1min":1.583,"5min":1.6679999999999999,"15min":1.6839999999999999},"min":{"1min":1.2330000000000001,"5min":1.2250000000000001,"15min":1.137},"max":{"1min":1.897,"5min":2.0899999999999999,"15min":2.4980000000000002},"last":1.3580000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575780,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51564,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320276,"statfs":{"total":96636764160,"available":96583962624,"internally_reserved":0,"allocated":2392064,"data_stored":1171635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.95499999999999996,"15min":0.93600000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3520000000000001,"5min":1.6879999999999999,"15min":1.8759999999999999},"last":1.097},{"interface":"front","average":{"1min":1.03,"5min":1.089,"15min":1.115},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.7490000000000001,"15min":1.7889999999999999},"last":1.163}]},{"osd":1,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.03,"15min":1.0249999999999999},"min":{"1min":0.52300000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5720000000000001,"5min":1.6080000000000001,"15min":1.8620000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.764,"5min":1.8129999999999999,"15min":1.8959999999999999},"last":1.28}]},{"osd":3,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2210000000000001,"15min":1.274},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.5600000000000001,"5min":1.929,"15min":1.929},"last":1.135},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.375,"15min":1.3640000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0289999999999999},"last":1.4359999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.472,"15min":1.4790000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.153},"last":1.621},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3740000000000001,"15min":1.4099999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.917,"5min":2.0640000000000001,"15min":2.2639999999999998},"last":1.3740000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4319999999999999,"15min":1.462},"min":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.865,"5min":1.9359999999999999,"15min":2.2669999999999999},"last":1.968},{"interface":"front","average":{"1min":1.611,"5min":1.591,"15min":1.585},"min":{"1min":0.94599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0070000000000001,"5min":2.0880000000000001,"15min":2.3199999999999998},"last":2.1520000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5169999999999999,"15min":1.52},"min":{"1min":0.98099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0470000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.698},{"interface":"front","average":{"1min":1.54,"5min":1.583,"15min":1.6120000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.0169999999999999,"5min":2.161,"15min":2.3610000000000002},"last":2.044}]},{"osd":7,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":1.008,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1640000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.645,"15min":1.677},"min":{"1min":1.0660000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.137,"5min":2.177,"15min":2.4350000000000001},"last":1.847}]}]},{"osd":1,"up_from":13,"seq":55834575780,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47408,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324432,"statfs":{"total":96636764160,"available":96588218368,"internally_reserved":0,"allocated":2392064,"data_stored":1171579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.93400000000000005,"15min":0.93500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.351,"5min":1.706,"15min":1.756},"last":1.274},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.0860000000000001,"15min":1.135},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.379,"5min":1.71,"15min":1.7849999999999999},"last":0.89200000000000002}]},{"osd":2,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.96599999999999997,"15min":1.0229999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3029999999999999,"5min":1.7709999999999999,"15min":2.597},"last":1.151},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.1870000000000001,"15min":1.2290000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5249999999999999,"5min":1.7250000000000001,"15min":1.9610000000000001},"last":1.9590000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.353,"15min":1.3600000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.077},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.242,"15min":1.272},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5029999999999999,"5min":1.7829999999999999,"15min":2.2599999999999998},"last":1.8069999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4370000000000001,"15min":1.462},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.78400000000000003},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":2.044},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.3959999999999999,"15min":1.419},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.706},"last":1.5509999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.429,"15min":1.444},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8959999999999999,"5min":1.9390000000000001,"15min":2.3439999999999999},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.542,"15min":1.542},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1589999999999998,"5min":2.379,"15min":2.379},"last":2.2789999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.5669999999999999,"15min":1.5780000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.0880000000000001,"5min":2.1259999999999999,"15min":2.177},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.5740000000000001,"15min":1.5669999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.4860000000000002},"last":2.1219999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6599999999999999,"15min":1.6579999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":2.3700000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.6399999999999999,"15min":1.6619999999999999},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2839999999999998},"last":2.0329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575779,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64620,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307220,"statfs":{"total":96636764160,"available":96570593280,"internally_reserved":0,"allocated":2981888,"data_stored":1761913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.577,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.141,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.72699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6830000000000001,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1739999999999999,"15min":1.1499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.446,"5min":3.375,"15min":3.375},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2569999999999999,"15min":1.2629999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.681},"last":1.01}]},{"osd":3,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3819999999999999,"15min":1.3660000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.708,"5min":1.823,"15min":2.1899999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.425,"15min":1.4279999999999999},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.93999999999999995},"max":{"1min":1.792,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.875}]},{"osd":4,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.3420000000000001},"min":{"1min":1.0029999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.675,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.5309999999999999,"15min":1.498},"min":{"1min":1.002,"5min":1.002,"15min":0.999},"max":{"1min":2.008,"5min":2.9689999999999999,"15min":2.9689999999999999},"last":1.155}]},{"osd":5,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.401,"15min":1.4119999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.873,"5min":3.077,"15min":3.077},"last":1.395},{"interface":"front","average":{"1min":1.611,"5min":1.621,"15min":1.6020000000000001},"min":{"1min":1.194,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":2.028,"5min":2.468,"15min":2.468},"last":1.4339999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.6799999999999999,"15min":1.681},"min":{"1min":1.2769999999999999,"5min":1.194,"15min":1.147},"max":{"1min":2.1349999999999998,"5min":2.3479999999999999,"15min":2.5270000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.548,"15min":1.554},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.875,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.276}]},{"osd":7,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.675,"15min":1.655},"min":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"max":{"1min":1.9399999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.466},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6619999999999999,"15min":1.655},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":1.99,"5min":3.645,"15min":3.645},"last":1.4950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:58:30.606 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T17:58:30.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.875+0000 7fc141dbf640 1 Processor -- start 2023-10-14T17:58:30.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.875+0000 7fc141dbf640 1 -- start start 2023-10-14T17:58:30.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.877+0000 7fc141dbf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc13c06c9e0 0x7fc13c0ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.877+0000 7fc141dbf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0ca560 con 0x7fc13c06fb50 2023-10-14T17:58:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.877+0000 7fc141dbf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc13c0c9950 con 0x7fc13c0c9060 2023-10-14T17:58:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.877+0000 7fc141dbf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc13c0c9c30 con 0x7fc13c06c9e0 2023-10-14T17:58:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc13c06fb50 0x7fc13c0cc8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45072/0 (socket says 172.21.15.29:45072) 2023-10-14T17:58:30.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13bfff640 1 -- 172.21.15.29:0/1369829621 learned_addr learned my addr 172.21.15.29:0/1369829621 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc1415be640 1 --2- 172.21.15.29:0/1369829621 >> v2:172.21.15.186:3300/0 conn(0x7fc13c06c9e0 0x7fc13c0ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936281064 0 0) 0x7fc13c0ca560 con 0x7fc13c06fb50 2023-10-14T17:58:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc12c003520 con 0x7fc13c06fb50 2023-10-14T17:58:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 984302762 0 0) 0x7fc12c003520 con 0x7fc13c06fb50 2023-10-14T17:58:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 >> v2:172.21.15.186:3300/0 conn(0x7fc13c06c9e0 msgr2=0x7fc13c0ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:30.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 --2- 172.21.15.29:0/1369829621 >> v2:172.21.15.186:3300/0 conn(0x7fc13c06c9e0 0x7fc13c0ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.878+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 >> v1:172.21.15.186:6789/0 conn(0x7fc13c0c9060 legacy=0x7fc13c06f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0c9f10 con 0x7fc13c06fb50 2023-10-14T17:58:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc128001670 con 0x7fc13c06fb50 2023-10-14T17:58:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 >> v1:172.21.15.29:6789/0 conn(0x7fc13c06fb50 legacy=0x7fc13c0cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0039c0 0x7fc12c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 0x7fc12c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c005180 0x7fc12c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0c9c30 con 0x7fc12c0047c0 2023-10-14T17:58:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c00c080 con 0x7fc12c005180 2023-10-14T17:58:30.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c00c240 con 0x7fc12c0039c0 2023-10-14T17:58:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc13bfff640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 0x7fc12c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc140dbd640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0039c0 0x7fc12c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.879+0000 7fc1415be640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c005180 0x7fc12c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc13bfff640 1 -- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0039c0 msgr2=0x7fc12c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:30.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc13bfff640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0039c0 0x7fc12c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc13bfff640 1 -- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c005180 msgr2=0x7fc12c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc13bfff640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c005180 0x7fc12c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc13bfff640 1 -- 172.21.15.29:0/1369829621 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c003520 con 0x7fc12c0047c0 2023-10-14T17:58:30.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.880+0000 7fc1415be640 1 -- 172.21.15.29:0/1369829621 reap_dead start 2023-10-14T17:58:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.881+0000 7fc13bfff640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 0x7fc12c004c40 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c0e0 tx=0x7fc12c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a128140917d91819 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.881+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc128005ad0 con 0x7fc12c0047c0 2023-10-14T17:58:30.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.881+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc128005c30 con 0x7fc12c0047c0 2023-10-14T17:58:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.881+0000 7fc13b7fe640 1 -- 172.21.15.29:0/1369829621 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12802ca50 con 0x7fc12c0047c0 2023-10-14T17:58:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 -- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 msgr2=0x7fc12c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 0x7fc12c004c40 secure :-1 s=READY pgs=2172 cs=0 l=1 rev1=1 crypto rx=0x7fc12800c0e0 tx=0x7fc12c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 -- 172.21.15.29:0/1369829621 shutdown_connections 2023-10-14T17:58:30.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 --2- 172.21.15.29:0/1369829621 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0047c0 0x7fc12c004c40 unknown :-1 s=CLOSED pgs=2172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 -- 172.21.15.29:0/1369829621 >> 172.21.15.29:0/1369829621 conn(0x7fc13c0ce6a0 msgr2=0x7fc13c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 -- 172.21.15.29:0/1369829621 shutdown_connections 2023-10-14T17:58:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.882+0000 7fc141dbf640 1 -- 172.21.15.29:0/1369829621 wait complete. 2023-10-14T17:58:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.884+0000 7fc141dbf640 1 Processor -- start 2023-10-14T17:58:30.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.884+0000 7fc141dbf640 1 -- start start 2023-10-14T17:58:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.885+0000 7fc141dbf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 0x7fc13c0b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.885+0000 7fc141dbf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc141dbf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc13c0b1cd0 0x7fc13c0af010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc141dbf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0c72e0 con 0x7fc13c161510 2023-10-14T17:58:30.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc141dbf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c0ce130 con 0x7fc12c0047c0 2023-10-14T17:58:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc141dbf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc13c15e650 con 0x7fc13c0b1cd0 2023-10-14T17:58:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34118/0 (socket says 172.21.15.29:34118) 2023-10-14T17:58:30.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 -- 172.21.15.29:0/751573418 learned_addr learned my addr 172.21.15.29:0/751573418 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc1415be640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc13c0b1cd0 0x7fc13c0af010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc140dbd640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 0x7fc13c0b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 -- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc13c0b1cd0 msgr2=0x7fc13c0af010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc13c0b1cd0 0x7fc13c0af010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 -- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 msgr2=0x7fc13c0b12c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:30.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 0x7fc13c0b12c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:30.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc13bfff640 1 -- 172.21.15.29:0/751573418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc13c0b1950 con 0x7fc13c161510 2023-10-14T17:58:30.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.886+0000 7fc140dbd640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 0x7fc13c0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:30.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc13bfff640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc12802e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc12803c040 con 0x7fc13c161510 2023-10-14T17:58:30.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc12802cd80 con 0x7fc13c161510 2023-10-14T17:58:30.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc1280072b0 con 0x7fc13c161510 2023-10-14T17:58:30.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc13c0c6940 con 0x7fc13c161510 2023-10-14T17:58:30.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.887+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc13c0c6f00 con 0x7fc13c161510 2023-10-14T17:58:30.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.888+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc13c04e9b0 con 0x7fc13c161510 2023-10-14T17:58:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.889+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc128032070 con 0x7fc13c161510 2023-10-14T17:58:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.891+0000 7fc1397fa640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 0x7fc12006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.891+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fc1280aac20 con 0x7fc13c161510 2023-10-14T17:58:30.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.891+0000 7fc140dbd640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 0x7fc12006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:30.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.892+0000 7fc140dbd640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 0x7fc12006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc130000c00 tx=0x7fc130009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:30.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:30.895+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc128075b30 con 0x7fc13c161510 2023-10-14T17:58:31.022 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T17:58:31.021+0000 7f33a5503640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T17:58:31.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.092+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 --> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc13c0c1f10 con 0x7fc12006afc0 2023-10-14T17:58:31.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.114+0000 7fc1397fa640 1 -- 172.21.15.29:0/751573418 <== mgr.9861 v2:172.21.15.29:6832/79844030 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287021 (secure 0 0 0) 0x7fc13000d050 con 0x7fc12006afc0 2023-10-14T17:58:31.118 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:31.120 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T17:58:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.119+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 msgr2=0x7fc12006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.119+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 0x7fc12006d480 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc130000c00 tx=0x7fc130009040 comp rx=0 tx=0).stop 2023-10-14T17:58:31.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.119+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 msgr2=0x7fc13c15e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.119+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 secure :-1 s=READY pgs=2173 cs=0 l=1 rev1=1 crypto rx=0x7fc128000c00 tx=0x7fc12802e070 comp rx=0 tx=0).stop 2023-10-14T17:58:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.119+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 shutdown_connections 2023-10-14T17:58:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc13c0b1cd0 0x7fc13c0af010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc13c161510 0x7fc13c15e110 unknown :-1 s=CLOSED pgs=2173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fc12006afc0 0x7fc12006d480 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 --2- 172.21.15.29:0/751573418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0047c0 0x7fc13c0b12c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 >> 172.21.15.29:0/751573418 conn(0x7fc13c0ce6a0 msgr2=0x7fc13c06a440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 shutdown_connections 2023-10-14T17:58:31.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.120+0000 7fc141dbf640 1 -- 172.21.15.29:0/751573418 wait complete. 2023-10-14T17:58:31.140 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T17:58:29.156541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":523,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":331,"ondisk_log_size":331,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":148,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":402628,"kb_used_data":19916,"kb_used_omap":12,"kb_used_meta":382579,"kb_avail":754572092,"statfs":{"total":773094113280,"available":772681822208,"internally_reserved":0,"allocated":20393984,"data_stored":10577164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":391761464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.114410"},"pg_stats":[{"pgid":"23.3f","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.117131+0000","last_change":"2023-10-14T17:58:26.115017+0000","last_active":"2023-10-14T17:58:26.117131+0000","last_peered":"2023-10-14T17:58:26.117131+0000","last_clean":"2023-10-14T17:58:26.117131+0000","last_became_active":"2023-10-14T17:58:26.114711+0000","last_became_peered":"2023-10-14T17:58:26.114711+0000","last_unstale":"2023-10-14T17:58:26.117131+0000","last_undegraded":"2023-10-14T17:58:26.117131+0000","last_fullsized":"2023-10-14T17:58:26.117131+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:46:51.050670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.3e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121127+0000","last_change":"2023-10-14T17:58:26.110514+0000","last_active":"2023-10-14T17:58:27.121127+0000","last_peered":"2023-10-14T17:58:27.121127+0000","last_clean":"2023-10-14T17:58:27.121127+0000","last_became_active":"2023-10-14T17:58:26.109917+0000","last_became_peered":"2023-10-14T17:58:26.109917+0000","last_unstale":"2023-10-14T17:58:27.121127+0000","last_undegraded":"2023-10-14T17:58:27.121127+0000","last_fullsized":"2023-10-14T17:58:27.121127+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:35.001471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117210+0000","last_change":"2023-10-14T17:58:26.115229+0000","last_active":"2023-10-14T17:58:27.117210+0000","last_peered":"2023-10-14T17:58:27.117210+0000","last_clean":"2023-10-14T17:58:27.117210+0000","last_became_active":"2023-10-14T17:58:26.112909+0000","last_became_peered":"2023-10-14T17:58:26.112909+0000","last_unstale":"2023-10-14T17:58:27.117210+0000","last_undegraded":"2023-10-14T17:58:27.117210+0000","last_fullsized":"2023-10-14T17:58:27.117210+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:03.559514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.3c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802215+0000","last_change":"2023-10-14T17:58:26.127178+0000","last_active":"2023-10-14T17:58:27.802215+0000","last_peered":"2023-10-14T17:58:27.802215+0000","last_clean":"2023-10-14T17:58:27.802215+0000","last_became_active":"2023-10-14T17:58:26.125959+0000","last_became_peered":"2023-10-14T17:58:26.125959+0000","last_unstale":"2023-10-14T17:58:27.802215+0000","last_undegraded":"2023-10-14T17:58:27.802215+0000","last_fullsized":"2023-10-14T17:58:27.802215+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:54.759847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.3b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121032+0000","last_change":"2023-10-14T17:58:26.111144+0000","last_active":"2023-10-14T17:58:27.121032+0000","last_peered":"2023-10-14T17:58:27.121032+0000","last_clean":"2023-10-14T17:58:27.121032+0000","last_became_active":"2023-10-14T17:58:26.110880+0000","last_became_peered":"2023-10-14T17:58:26.110880+0000","last_unstale":"2023-10-14T17:58:27.121032+0000","last_undegraded":"2023-10-14T17:58:27.121032+0000","last_fullsized":"2023-10-14T17:58:27.121032+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:30.295018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.3a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730503+0000","last_change":"2023-10-14T17:58:26.112746+0000","last_active":"2023-10-14T17:58:27.730503+0000","last_peered":"2023-10-14T17:58:27.730503+0000","last_clean":"2023-10-14T17:58:27.730503+0000","last_became_active":"2023-10-14T17:58:26.112435+0000","last_became_peered":"2023-10-14T17:58:26.112435+0000","last_unstale":"2023-10-14T17:58:27.730503+0000","last_undegraded":"2023-10-14T17:58:27.730503+0000","last_fullsized":"2023-10-14T17:58:27.730503+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:09:41.991486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.39","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122970+0000","last_change":"2023-10-14T17:58:26.117083+0000","last_active":"2023-10-14T17:58:26.122970+0000","last_peered":"2023-10-14T17:58:26.122970+0000","last_clean":"2023-10-14T17:58:26.122970+0000","last_became_active":"2023-10-14T17:58:26.116789+0000","last_became_peered":"2023-10-14T17:58:26.116789+0000","last_unstale":"2023-10-14T17:58:26.122970+0000","last_undegraded":"2023-10-14T17:58:26.122970+0000","last_fullsized":"2023-10-14T17:58:26.122970+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:31.177662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.38","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119851+0000","last_change":"2023-10-14T17:58:26.111068+0000","last_active":"2023-10-14T17:58:27.119851+0000","last_peered":"2023-10-14T17:58:27.119851+0000","last_clean":"2023-10-14T17:58:27.119851+0000","last_became_active":"2023-10-14T17:58:26.110092+0000","last_became_peered":"2023-10-14T17:58:26.110092+0000","last_unstale":"2023-10-14T17:58:27.119851+0000","last_undegraded":"2023-10-14T17:58:27.119851+0000","last_fullsized":"2023-10-14T17:58:27.119851+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:01:04.917416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.37","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121313+0000","last_change":"2023-10-14T17:58:26.106129+0000","last_active":"2023-10-14T17:58:27.121313+0000","last_peered":"2023-10-14T17:58:27.121313+0000","last_clean":"2023-10-14T17:58:27.121313+0000","last_became_active":"2023-10-14T17:58:26.105788+0000","last_became_peered":"2023-10-14T17:58:26.105788+0000","last_unstale":"2023-10-14T17:58:27.121313+0000","last_undegraded":"2023-10-14T17:58:27.121313+0000","last_fullsized":"2023-10-14T17:58:27.121313+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:25.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.36","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.121119+0000","last_change":"2023-10-14T17:58:26.110462+0000","last_active":"2023-10-14T17:58:27.121119+0000","last_peered":"2023-10-14T17:58:27.121119+0000","last_clean":"2023-10-14T17:58:27.121119+0000","last_became_active":"2023-10-14T17:58:26.108900+0000","last_became_peered":"2023-10-14T17:58:26.108900+0000","last_unstale":"2023-10-14T17:58:27.121119+0000","last_undegraded":"2023-10-14T17:58:27.121119+0000","last_fullsized":"2023-10-14T17:58:27.121119+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:10.274314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.35","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120667+0000","last_change":"2023-10-14T17:58:26.112457+0000","last_active":"2023-10-14T17:58:27.120667+0000","last_peered":"2023-10-14T17:58:27.120667+0000","last_clean":"2023-10-14T17:58:27.120667+0000","last_became_active":"2023-10-14T17:58:26.112160+0000","last_became_peered":"2023-10-14T17:58:26.112160+0000","last_unstale":"2023-10-14T17:58:27.120667+0000","last_undegraded":"2023-10-14T17:58:27.120667+0000","last_fullsized":"2023-10-14T17:58:27.120667+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:47.784344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.34","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120541+0000","last_change":"2023-10-14T17:58:26.105882+0000","last_active":"2023-10-14T17:58:27.120541+0000","last_peered":"2023-10-14T17:58:27.120541+0000","last_clean":"2023-10-14T17:58:27.120541+0000","last_became_active":"2023-10-14T17:58:26.105601+0000","last_became_peered":"2023-10-14T17:58:26.105601+0000","last_unstale":"2023-10-14T17:58:27.120541+0000","last_undegraded":"2023-10-14T17:58:27.120541+0000","last_fullsized":"2023-10-14T17:58:27.120541+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:34.756888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.33","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115701+0000","last_change":"2023-10-14T17:58:26.108936+0000","last_active":"2023-10-14T17:58:27.115701+0000","last_peered":"2023-10-14T17:58:27.115701+0000","last_clean":"2023-10-14T17:58:27.115701+0000","last_became_active":"2023-10-14T17:58:26.108510+0000","last_became_peered":"2023-10-14T17:58:26.108510+0000","last_unstale":"2023-10-14T17:58:27.115701+0000","last_undegraded":"2023-10-14T17:58:27.115701+0000","last_fullsized":"2023-10-14T17:58:27.115701+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:47.784344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.32","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115308+0000","last_change":"2023-10-14T17:58:26.113124+0000","last_active":"2023-10-14T17:58:27.115308+0000","last_peered":"2023-10-14T17:58:27.115308+0000","last_clean":"2023-10-14T17:58:27.115308+0000","last_became_active":"2023-10-14T17:58:26.112161+0000","last_became_peered":"2023-10-14T17:58:26.112161+0000","last_unstale":"2023-10-14T17:58:27.115308+0000","last_undegraded":"2023-10-14T17:58:27.115308+0000","last_fullsized":"2023-10-14T17:58:27.115308+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:45:02.827786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.31","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.110747+0000","last_change":"2023-10-14T17:58:26.108433+0000","last_active":"2023-10-14T17:58:26.110747+0000","last_peered":"2023-10-14T17:58:26.110747+0000","last_clean":"2023-10-14T17:58:26.110747+0000","last_became_active":"2023-10-14T17:58:26.107966+0000","last_became_peered":"2023-10-14T17:58:26.107966+0000","last_unstale":"2023-10-14T17:58:26.110747+0000","last_undegraded":"2023-10-14T17:58:26.110747+0000","last_fullsized":"2023-10-14T17:58:26.110747+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:28:15.701827+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.30","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115201+0000","last_change":"2023-10-14T17:58:26.111042+0000","last_active":"2023-10-14T17:58:26.115201+0000","last_peered":"2023-10-14T17:58:26.115201+0000","last_clean":"2023-10-14T17:58:26.115201+0000","last_became_active":"2023-10-14T17:58:26.110375+0000","last_became_peered":"2023-10-14T17:58:26.110375+0000","last_unstale":"2023-10-14T17:58:26.115201+0000","last_undegraded":"2023-10-14T17:58:26.115201+0000","last_fullsized":"2023-10-14T17:58:26.115201+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:10.949702+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.2f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731305+0000","last_change":"2023-10-14T17:58:26.115084+0000","last_active":"2023-10-14T17:58:27.731305+0000","last_peered":"2023-10-14T17:58:27.731305+0000","last_clean":"2023-10-14T17:58:27.731305+0000","last_became_active":"2023-10-14T17:58:26.114809+0000","last_became_peered":"2023-10-14T17:58:26.114809+0000","last_unstale":"2023-10-14T17:58:27.731305+0000","last_undegraded":"2023-10-14T17:58:27.731305+0000","last_fullsized":"2023-10-14T17:58:27.731305+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:13:44.521120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2e","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123438+0000","last_change":"2023-10-14T17:58:26.120080+0000","last_active":"2023-10-14T17:58:26.123438+0000","last_peered":"2023-10-14T17:58:26.123438+0000","last_clean":"2023-10-14T17:58:26.123438+0000","last_became_active":"2023-10-14T17:58:26.119804+0000","last_became_peered":"2023-10-14T17:58:26.119804+0000","last_unstale":"2023-10-14T17:58:26.123438+0000","last_undegraded":"2023-10-14T17:58:26.123438+0000","last_fullsized":"2023-10-14T17:58:26.123438+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:47:22.620014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730693+0000","last_change":"2023-10-14T17:58:26.112261+0000","last_active":"2023-10-14T17:58:27.730693+0000","last_peered":"2023-10-14T17:58:27.730693+0000","last_clean":"2023-10-14T17:58:27.730693+0000","last_became_active":"2023-10-14T17:58:26.111775+0000","last_became_peered":"2023-10-14T17:58:26.111775+0000","last_unstale":"2023-10-14T17:58:27.730693+0000","last_undegraded":"2023-10-14T17:58:27.730693+0000","last_fullsized":"2023-10-14T17:58:27.730693+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:25:49.042427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115449+0000","last_change":"2023-10-14T17:58:26.107332+0000","last_active":"2023-10-14T17:58:27.115449+0000","last_peered":"2023-10-14T17:58:27.115449+0000","last_clean":"2023-10-14T17:58:27.115449+0000","last_became_active":"2023-10-14T17:58:26.107059+0000","last_became_peered":"2023-10-14T17:58:26.107059+0000","last_unstale":"2023-10-14T17:58:27.115449+0000","last_undegraded":"2023-10-14T17:58:27.115449+0000","last_fullsized":"2023-10-14T17:58:27.115449+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:34.756888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.2b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730700+0000","last_change":"2023-10-14T17:58:26.112541+0000","last_active":"2023-10-14T17:58:27.730700+0000","last_peered":"2023-10-14T17:58:27.730700+0000","last_clean":"2023-10-14T17:58:27.730700+0000","last_became_active":"2023-10-14T17:58:26.112190+0000","last_became_peered":"2023-10-14T17:58:26.112190+0000","last_unstale":"2023-10-14T17:58:27.730700+0000","last_undegraded":"2023-10-14T17:58:27.730700+0000","last_fullsized":"2023-10-14T17:58:27.730700+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:51.934965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.2a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120589+0000","last_change":"2023-10-14T17:58:26.111473+0000","last_active":"2023-10-14T17:58:27.120589+0000","last_peered":"2023-10-14T17:58:27.120589+0000","last_clean":"2023-10-14T17:58:27.120589+0000","last_became_active":"2023-10-14T17:58:26.110325+0000","last_became_peered":"2023-10-14T17:58:26.110325+0000","last_unstale":"2023-10-14T17:58:27.120589+0000","last_undegraded":"2023-10-14T17:58:27.120589+0000","last_fullsized":"2023-10-14T17:58:27.120589+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:59.702977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.29","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730153+0000","last_change":"2023-10-14T17:58:26.119709+0000","last_active":"2023-10-14T17:58:27.730153+0000","last_peered":"2023-10-14T17:58:27.730153+0000","last_clean":"2023-10-14T17:58:27.730153+0000","last_became_active":"2023-10-14T17:58:26.116619+0000","last_became_peered":"2023-10-14T17:58:26.116619+0000","last_unstale":"2023-10-14T17:58:27.730153+0000","last_undegraded":"2023-10-14T17:58:27.730153+0000","last_fullsized":"2023-10-14T17:58:27.730153+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:10:47.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.28","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119942+0000","last_change":"2023-10-14T17:58:26.104466+0000","last_active":"2023-10-14T17:58:27.119942+0000","last_peered":"2023-10-14T17:58:27.119942+0000","last_clean":"2023-10-14T17:58:27.119942+0000","last_became_active":"2023-10-14T17:58:26.104162+0000","last_became_peered":"2023-10-14T17:58:26.104162+0000","last_unstale":"2023-10-14T17:58:27.119942+0000","last_undegraded":"2023-10-14T17:58:27.119942+0000","last_fullsized":"2023-10-14T17:58:27.119942+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:45:02.827786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.27","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116891+0000","last_change":"2023-10-14T17:58:26.114566+0000","last_active":"2023-10-14T17:58:27.116891+0000","last_peered":"2023-10-14T17:58:27.116891+0000","last_clean":"2023-10-14T17:58:27.116891+0000","last_became_active":"2023-10-14T17:58:26.113446+0000","last_became_peered":"2023-10-14T17:58:26.113446+0000","last_unstale":"2023-10-14T17:58:27.116891+0000","last_undegraded":"2023-10-14T17:58:27.116891+0000","last_fullsized":"2023-10-14T17:58:27.116891+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:25.634745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.26","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111226+0000","last_change":"2023-10-14T17:58:26.105482+0000","last_active":"2023-10-14T17:58:26.111226+0000","last_peered":"2023-10-14T17:58:26.111226+0000","last_clean":"2023-10-14T17:58:26.111226+0000","last_became_active":"2023-10-14T17:58:26.105213+0000","last_became_peered":"2023-10-14T17:58:26.105213+0000","last_unstale":"2023-10-14T17:58:26.111226+0000","last_undegraded":"2023-10-14T17:58:26.111226+0000","last_fullsized":"2023-10-14T17:58:26.111226+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:01:50.488612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.25","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117294+0000","last_change":"2023-10-14T17:58:26.104039+0000","last_active":"2023-10-14T17:58:27.117294+0000","last_peered":"2023-10-14T17:58:27.117294+0000","last_clean":"2023-10-14T17:58:27.117294+0000","last_became_active":"2023-10-14T17:58:26.103764+0000","last_became_peered":"2023-10-14T17:58:26.103764+0000","last_unstale":"2023-10-14T17:58:27.117294+0000","last_undegraded":"2023-10-14T17:58:27.117294+0000","last_fullsized":"2023-10-14T17:58:27.117294+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:10:47.869259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.24","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802339+0000","last_change":"2023-10-14T17:58:26.114324+0000","last_active":"2023-10-14T17:58:27.802339+0000","last_peered":"2023-10-14T17:58:27.802339+0000","last_clean":"2023-10-14T17:58:27.802339+0000","last_became_active":"2023-10-14T17:58:26.114024+0000","last_became_peered":"2023-10-14T17:58:26.114024+0000","last_unstale":"2023-10-14T17:58:27.802339+0000","last_undegraded":"2023-10-14T17:58:27.802339+0000","last_fullsized":"2023-10-14T17:58:27.802339+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:37.273607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.23","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116066+0000","last_change":"2023-10-14T17:58:26.108769+0000","last_active":"2023-10-14T17:58:27.116066+0000","last_peered":"2023-10-14T17:58:27.116066+0000","last_clean":"2023-10-14T17:58:27.116066+0000","last_became_active":"2023-10-14T17:58:26.108135+0000","last_became_peered":"2023-10-14T17:58:26.108135+0000","last_unstale":"2023-10-14T17:58:27.116066+0000","last_undegraded":"2023-10-14T17:58:27.116066+0000","last_fullsized":"2023-10-14T17:58:27.116066+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:10.274314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.22","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119472+0000","last_change":"2023-10-14T17:58:26.107688+0000","last_active":"2023-10-14T17:58:27.119472+0000","last_peered":"2023-10-14T17:58:27.119472+0000","last_clean":"2023-10-14T17:58:27.119472+0000","last_became_active":"2023-10-14T17:58:26.107103+0000","last_became_peered":"2023-10-14T17:58:26.107103+0000","last_unstale":"2023-10-14T17:58:27.119472+0000","last_undegraded":"2023-10-14T17:58:27.119472+0000","last_fullsized":"2023-10-14T17:58:27.119472+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:50.606108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.21","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117569+0000","last_change":"2023-10-14T17:58:26.117028+0000","last_active":"2023-10-14T17:58:27.117569+0000","last_peered":"2023-10-14T17:58:27.117569+0000","last_clean":"2023-10-14T17:58:27.117569+0000","last_became_active":"2023-10-14T17:58:26.116744+0000","last_became_peered":"2023-10-14T17:58:26.116744+0000","last_unstale":"2023-10-14T17:58:27.117569+0000","last_undegraded":"2023-10-14T17:58:27.117569+0000","last_fullsized":"2023-10-14T17:58:27.117569+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:17:48.822796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.20","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.130935+0000","last_change":"2023-10-14T17:58:26.125706+0000","last_active":"2023-10-14T17:58:26.130935+0000","last_peered":"2023-10-14T17:58:26.130935+0000","last_clean":"2023-10-14T17:58:26.130935+0000","last_became_active":"2023-10-14T17:58:26.125086+0000","last_became_peered":"2023-10-14T17:58:26.125086+0000","last_unstale":"2023-10-14T17:58:26.130935+0000","last_undegraded":"2023-10-14T17:58:26.130935+0000","last_fullsized":"2023-10-14T17:58:26.130935+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:34:26.176385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.1e","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111203+0000","last_change":"2023-10-14T17:58:26.104040+0000","last_active":"2023-10-14T17:58:26.111203+0000","last_peered":"2023-10-14T17:58:26.111203+0000","last_clean":"2023-10-14T17:58:26.111203+0000","last_became_active":"2023-10-14T17:58:26.103693+0000","last_became_peered":"2023-10-14T17:58:26.103693+0000","last_unstale":"2023-10-14T17:58:26.111203+0000","last_undegraded":"2023-10-14T17:58:26.111203+0000","last_fullsized":"2023-10-14T17:58:26.111203+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:18:21.922333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1f","version":"191'4","reported_seq":15,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.127718+0000","last_change":"2023-10-14T17:58:25.082945+0000","last_active":"2023-10-14T17:58:26.127718+0000","last_peered":"2023-10-14T17:58:26.127718+0000","last_clean":"2023-10-14T17:58:26.127718+0000","last_became_active":"2023-10-14T17:58:25.082562+0000","last_became_peered":"2023-10-14T17:58:25.082562+0000","last_unstale":"2023-10-14T17:58:26.127718+0000","last_undegraded":"2023-10-14T17:58:26.127718+0000","last_fullsized":"2023-10-14T17:58:26.127718+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:09:27.328801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.1f","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118816+0000","last_change":"2023-10-14T17:58:26.121680+0000","last_active":"2023-10-14T17:58:27.118816+0000","last_peered":"2023-10-14T17:58:27.118816+0000","last_clean":"2023-10-14T17:58:27.118816+0000","last_became_active":"2023-10-14T17:58:26.121389+0000","last_became_peered":"2023-10-14T17:58:26.121389+0000","last_unstale":"2023-10-14T17:58:27.118816+0000","last_undegraded":"2023-10-14T17:58:27.118816+0000","last_fullsized":"2023-10-14T17:58:27.118816+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:42:00.443453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.1e","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119235+0000","last_change":"2023-10-14T17:58:25.071198+0000","last_active":"2023-10-14T17:58:27.119235+0000","last_peered":"2023-10-14T17:58:27.119235+0000","last_clean":"2023-10-14T17:58:27.119235+0000","last_became_active":"2023-10-14T17:58:25.070911+0000","last_became_peered":"2023-10-14T17:58:25.070911+0000","last_unstale":"2023-10-14T17:58:27.119235+0000","last_undegraded":"2023-10-14T17:58:27.119235+0000","last_fullsized":"2023-10-14T17:58:27.119235+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:16.269268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.1c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730615+0000","last_change":"2023-10-14T17:58:26.113256+0000","last_active":"2023-10-14T17:58:27.730615+0000","last_peered":"2023-10-14T17:58:27.730615+0000","last_clean":"2023-10-14T17:58:27.730615+0000","last_became_active":"2023-10-14T17:58:26.111849+0000","last_became_peered":"2023-10-14T17:58:26.111849+0000","last_unstale":"2023-10-14T17:58:27.730615+0000","last_undegraded":"2023-10-14T17:58:27.730615+0000","last_fullsized":"2023-10-14T17:58:27.730615+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:43:29.234136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.1d","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801883+0000","last_change":"2023-10-14T17:58:25.086732+0000","last_active":"2023-10-14T17:58:27.801883+0000","last_peered":"2023-10-14T17:58:27.801883+0000","last_clean":"2023-10-14T17:58:27.801883+0000","last_became_active":"2023-10-14T17:58:25.086427+0000","last_became_peered":"2023-10-14T17:58:25.086427+0000","last_unstale":"2023-10-14T17:58:27.801883+0000","last_undegraded":"2023-10-14T17:58:27.801883+0000","last_fullsized":"2023-10-14T17:58:27.801883+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:31.520308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1d","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117563+0000","last_change":"2023-10-14T17:58:26.114058+0000","last_active":"2023-10-14T17:58:27.117563+0000","last_peered":"2023-10-14T17:58:27.117563+0000","last_clean":"2023-10-14T17:58:27.117563+0000","last_became_active":"2023-10-14T17:58:26.113778+0000","last_became_peered":"2023-10-14T17:58:26.113778+0000","last_unstale":"2023-10-14T17:58:27.117563+0000","last_undegraded":"2023-10-14T17:58:27.117563+0000","last_fullsized":"2023-10-14T17:58:27.117563+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:09:41.991486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.1c","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802518+0000","last_change":"2023-10-14T17:58:25.087826+0000","last_active":"2023-10-14T17:58:27.802518+0000","last_peered":"2023-10-14T17:58:27.802518+0000","last_clean":"2023-10-14T17:58:27.802518+0000","last_became_active":"2023-10-14T17:58:25.087551+0000","last_became_peered":"2023-10-14T17:58:25.087551+0000","last_unstale":"2023-10-14T17:58:27.802518+0000","last_undegraded":"2023-10-14T17:58:27.802518+0000","last_fullsized":"2023-10-14T17:58:27.802518+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:37:26.326645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.1a","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.124612+0000","last_change":"2023-10-14T17:58:26.121290+0000","last_active":"2023-10-14T17:58:26.124612+0000","last_peered":"2023-10-14T17:58:26.124612+0000","last_clean":"2023-10-14T17:58:26.124612+0000","last_became_active":"2023-10-14T17:58:26.120995+0000","last_became_peered":"2023-10-14T17:58:26.120995+0000","last_unstale":"2023-10-14T17:58:26.124612+0000","last_undegraded":"2023-10-14T17:58:26.124612+0000","last_fullsized":"2023-10-14T17:58:26.124612+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:35:16.035273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1b","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116345+0000","last_change":"2023-10-14T17:58:25.086816+0000","last_active":"2023-10-14T17:58:27.116345+0000","last_peered":"2023-10-14T17:58:27.116345+0000","last_clean":"2023-10-14T17:58:27.116345+0000","last_became_active":"2023-10-14T17:58:25.085155+0000","last_became_peered":"2023-10-14T17:58:25.085155+0000","last_unstale":"2023-10-14T17:58:27.116345+0000","last_undegraded":"2023-10-14T17:58:27.116345+0000","last_fullsized":"2023-10-14T17:58:27.116345+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:19:36.367182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.1b","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801932+0000","last_change":"2023-10-14T17:58:26.123450+0000","last_active":"2023-10-14T17:58:27.801932+0000","last_peered":"2023-10-14T17:58:27.801932+0000","last_clean":"2023-10-14T17:58:27.801932+0000","last_became_active":"2023-10-14T17:58:26.123168+0000","last_became_peered":"2023-10-14T17:58:26.123168+0000","last_unstale":"2023-10-14T17:58:27.801932+0000","last_undegraded":"2023-10-14T17:58:27.801932+0000","last_fullsized":"2023-10-14T17:58:27.801932+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:41.396701+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.1a","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801206+0000","last_change":"2023-10-14T17:58:25.084285+0000","last_active":"2023-10-14T17:58:27.801206+0000","last_peered":"2023-10-14T17:58:27.801206+0000","last_clean":"2023-10-14T17:58:27.801206+0000","last_became_active":"2023-10-14T17:58:25.083668+0000","last_became_peered":"2023-10-14T17:58:25.083668+0000","last_unstale":"2023-10-14T17:58:27.801206+0000","last_undegraded":"2023-10-14T17:58:27.801206+0000","last_fullsized":"2023-10-14T17:58:27.801206+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:48.033775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.18","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801137+0000","last_change":"2023-10-14T17:58:26.108282+0000","last_active":"2023-10-14T17:58:27.801137+0000","last_peered":"2023-10-14T17:58:27.801137+0000","last_clean":"2023-10-14T17:58:27.801137+0000","last_became_active":"2023-10-14T17:58:26.107969+0000","last_became_peered":"2023-10-14T17:58:26.107969+0000","last_unstale":"2023-10-14T17:58:27.801137+0000","last_undegraded":"2023-10-14T17:58:27.801137+0000","last_fullsized":"2023-10-14T17:58:27.801137+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:55:27.865214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.19","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120148+0000","last_change":"2023-10-14T17:58:25.085103+0000","last_active":"2023-10-14T17:58:27.120148+0000","last_peered":"2023-10-14T17:58:27.120148+0000","last_clean":"2023-10-14T17:58:27.120148+0000","last_became_active":"2023-10-14T17:58:25.084773+0000","last_became_peered":"2023-10-14T17:58:25.084773+0000","last_unstale":"2023-10-14T17:58:27.120148+0000","last_undegraded":"2023-10-14T17:58:27.120148+0000","last_fullsized":"2023-10-14T17:58:27.120148+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:51.346979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.19","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730772+0000","last_change":"2023-10-14T17:58:26.117661+0000","last_active":"2023-10-14T17:58:27.730772+0000","last_peered":"2023-10-14T17:58:27.730772+0000","last_clean":"2023-10-14T17:58:27.730772+0000","last_became_active":"2023-10-14T17:58:26.116188+0000","last_became_peered":"2023-10-14T17:58:26.116188+0000","last_unstale":"2023-10-14T17:58:27.730772+0000","last_undegraded":"2023-10-14T17:58:27.730772+0000","last_fullsized":"2023-10-14T17:58:27.730772+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:25:07.514050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.18","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.093052+0000","last_change":"2023-10-14T17:58:25.072573+0000","last_active":"2023-10-14T17:58:26.093052+0000","last_peered":"2023-10-14T17:58:26.093052+0000","last_clean":"2023-10-14T17:58:26.093052+0000","last_became_active":"2023-10-14T17:58:25.072216+0000","last_became_peered":"2023-10-14T17:58:25.072216+0000","last_unstale":"2023-10-14T17:58:26.093052+0000","last_undegraded":"2023-10-14T17:58:26.093052+0000","last_fullsized":"2023-10-14T17:58:26.093052+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:51.346979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"23.17","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131509+0000","last_change":"2023-10-14T17:58:26.128319+0000","last_active":"2023-10-14T17:58:26.131509+0000","last_peered":"2023-10-14T17:58:26.131509+0000","last_clean":"2023-10-14T17:58:26.131509+0000","last_became_active":"2023-10-14T17:58:26.127922+0000","last_became_peered":"2023-10-14T17:58:26.127922+0000","last_unstale":"2023-10-14T17:58:26.131509+0000","last_undegraded":"2023-10-14T17:58:26.131509+0000","last_fullsized":"2023-10-14T17:58:26.131509+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:12.520592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.16","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731189+0000","last_change":"2023-10-14T17:58:25.083253+0000","last_active":"2023-10-14T17:58:27.731189+0000","last_peered":"2023-10-14T17:58:27.731189+0000","last_clean":"2023-10-14T17:58:27.731189+0000","last_became_active":"2023-10-14T17:58:25.082917+0000","last_became_peered":"2023-10-14T17:58:25.082917+0000","last_unstale":"2023-10-14T17:58:27.731189+0000","last_undegraded":"2023-10-14T17:58:27.731189+0000","last_fullsized":"2023-10-14T17:58:27.731189+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:02.245682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.14","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131171+0000","last_change":"2023-10-14T17:58:26.129382+0000","last_active":"2023-10-14T17:58:26.131171+0000","last_peered":"2023-10-14T17:58:26.131171+0000","last_clean":"2023-10-14T17:58:26.131171+0000","last_became_active":"2023-10-14T17:58:26.129040+0000","last_became_peered":"2023-10-14T17:58:26.129040+0000","last_unstale":"2023-10-14T17:58:26.131171+0000","last_undegraded":"2023-10-14T17:58:26.131171+0000","last_fullsized":"2023-10-14T17:58:26.131171+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:28:27.345489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.15","version":"191'3","reported_seq":15,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.731012+0000","last_change":"2023-10-14T17:58:25.079049+0000","last_active":"2023-10-14T17:58:27.731012+0000","last_peered":"2023-10-14T17:58:27.731012+0000","last_clean":"2023-10-14T17:58:27.731012+0000","last_became_active":"2023-10-14T17:58:25.078646+0000","last_became_peered":"2023-10-14T17:58:25.078646+0000","last_unstale":"2023-10-14T17:58:27.731012+0000","last_undegraded":"2023-10-14T17:58:27.731012+0000","last_fullsized":"2023-10-14T17:58:27.731012+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:58.687645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"23.15","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119465+0000","last_change":"2023-10-14T17:58:26.105197+0000","last_active":"2023-10-14T17:58:27.119465+0000","last_peered":"2023-10-14T17:58:27.119465+0000","last_clean":"2023-10-14T17:58:27.119465+0000","last_became_active":"2023-10-14T17:58:26.104900+0000","last_became_peered":"2023-10-14T17:58:26.104900+0000","last_unstale":"2023-10-14T17:58:27.119465+0000","last_undegraded":"2023-10-14T17:58:27.119465+0000","last_fullsized":"2023-10-14T17:58:27.119465+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:37:27.341977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.14","version":"191'6","reported_seq":18,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118214+0000","last_change":"2023-10-14T17:58:25.076118+0000","last_active":"2023-10-14T17:58:27.118214+0000","last_peered":"2023-10-14T17:58:27.118214+0000","last_clean":"2023-10-14T17:58:27.118214+0000","last_became_active":"2023-10-14T17:58:25.075743+0000","last_became_peered":"2023-10-14T17:58:25.075743+0000","last_unstale":"2023-10-14T17:58:27.118214+0000","last_undegraded":"2023-10-14T17:58:27.118214+0000","last_fullsized":"2023-10-14T17:58:27.118214+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:31.520308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.12","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122506+0000","last_change":"2023-10-14T17:58:26.119934+0000","last_active":"2023-10-14T17:58:26.122506+0000","last_peered":"2023-10-14T17:58:26.122506+0000","last_clean":"2023-10-14T17:58:26.122506+0000","last_became_active":"2023-10-14T17:58:26.119609+0000","last_became_peered":"2023-10-14T17:58:26.119609+0000","last_unstale":"2023-10-14T17:58:26.122506+0000","last_undegraded":"2023-10-14T17:58:26.122506+0000","last_fullsized":"2023-10-14T17:58:26.122506+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:52.830856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.13","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801715+0000","last_change":"2023-10-14T17:58:25.085283+0000","last_active":"2023-10-14T17:58:27.801715+0000","last_peered":"2023-10-14T17:58:27.801715+0000","last_clean":"2023-10-14T17:58:27.801715+0000","last_became_active":"2023-10-14T17:58:25.085004+0000","last_became_peered":"2023-10-14T17:58:25.085004+0000","last_unstale":"2023-10-14T17:58:27.801715+0000","last_undegraded":"2023-10-14T17:58:27.801715+0000","last_fullsized":"2023-10-14T17:58:27.801715+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:16.800712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.13","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123005+0000","last_change":"2023-10-14T17:58:26.115624+0000","last_active":"2023-10-14T17:58:26.123005+0000","last_peered":"2023-10-14T17:58:26.123005+0000","last_clean":"2023-10-14T17:58:26.123005+0000","last_became_active":"2023-10-14T17:58:26.115298+0000","last_became_peered":"2023-10-14T17:58:26.115298+0000","last_unstale":"2023-10-14T17:58:26.123005+0000","last_undegraded":"2023-10-14T17:58:26.123005+0000","last_fullsized":"2023-10-14T17:58:26.123005+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:36.150563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.12","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.109577+0000","last_change":"2023-10-14T17:58:25.080153+0000","last_active":"2023-10-14T17:58:26.109577+0000","last_peered":"2023-10-14T17:58:26.109577+0000","last_clean":"2023-10-14T17:58:26.109577+0000","last_became_active":"2023-10-14T17:58:25.079559+0000","last_became_peered":"2023-10-14T17:58:25.079559+0000","last_unstale":"2023-10-14T17:58:26.109577+0000","last_undegraded":"2023-10-14T17:58:26.109577+0000","last_fullsized":"2023-10-14T17:58:26.109577+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:48.554913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.10","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801203+0000","last_change":"2023-10-14T17:58:26.105513+0000","last_active":"2023-10-14T17:58:27.801203+0000","last_peered":"2023-10-14T17:58:27.801203+0000","last_clean":"2023-10-14T17:58:27.801203+0000","last_became_active":"2023-10-14T17:58:26.103936+0000","last_became_peered":"2023-10-14T17:58:26.103936+0000","last_unstale":"2023-10-14T17:58:27.801203+0000","last_undegraded":"2023-10-14T17:58:27.801203+0000","last_fullsized":"2023-10-14T17:58:27.801203+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:54.799910+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.11","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801334+0000","last_change":"2023-10-14T17:58:25.089220+0000","last_active":"2023-10-14T17:58:27.801334+0000","last_peered":"2023-10-14T17:58:27.801334+0000","last_clean":"2023-10-14T17:58:27.801334+0000","last_became_active":"2023-10-14T17:58:25.088965+0000","last_became_peered":"2023-10-14T17:58:25.088965+0000","last_unstale":"2023-10-14T17:58:27.801334+0000","last_undegraded":"2023-10-14T17:58:27.801334+0000","last_fullsized":"2023-10-14T17:58:27.801334+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:33:54.056721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.11","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115735+0000","last_change":"2023-10-14T17:58:26.109808+0000","last_active":"2023-10-14T17:58:27.115735+0000","last_peered":"2023-10-14T17:58:27.115735+0000","last_clean":"2023-10-14T17:58:27.115735+0000","last_became_active":"2023-10-14T17:58:26.108357+0000","last_became_peered":"2023-10-14T17:58:26.108357+0000","last_unstale":"2023-10-14T17:58:27.115735+0000","last_undegraded":"2023-10-14T17:58:27.115735+0000","last_fullsized":"2023-10-14T17:58:27.115735+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T17:59:30.295018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.10","version":"191'8","reported_seq":20,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117846+0000","last_change":"2023-10-14T17:58:25.078724+0000","last_active":"2023-10-14T17:58:27.117846+0000","last_peered":"2023-10-14T17:58:27.117846+0000","last_clean":"2023-10-14T17:58:27.117846+0000","last_became_active":"2023-10-14T17:58:25.076993+0000","last_became_peered":"2023-10-14T17:58:25.076993+0000","last_unstale":"2023-10-14T17:58:27.117846+0000","last_undegraded":"2023-10-14T17:58:27.117846+0000","last_fullsized":"2023-10-14T17:58:27.117846+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:43.862488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.e","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802525+0000","last_change":"2023-10-14T17:58:26.119282+0000","last_active":"2023-10-14T17:58:27.802525+0000","last_peered":"2023-10-14T17:58:27.802525+0000","last_clean":"2023-10-14T17:58:27.802525+0000","last_became_active":"2023-10-14T17:58:26.110912+0000","last_became_peered":"2023-10-14T17:58:26.110912+0000","last_unstale":"2023-10-14T17:58:27.802525+0000","last_undegraded":"2023-10-14T17:58:27.802525+0000","last_fullsized":"2023-10-14T17:58:27.802525+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:01:14.240634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.f","version":"191'1","reported_seq":13,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117139+0000","last_change":"2023-10-14T17:58:25.084919+0000","last_active":"2023-10-14T17:58:27.117139+0000","last_peered":"2023-10-14T17:58:27.117139+0000","last_clean":"2023-10-14T17:58:27.117139+0000","last_became_active":"2023-10-14T17:58:25.084622+0000","last_became_peered":"2023-10-14T17:58:25.084622+0000","last_unstale":"2023-10-14T17:58:27.117139+0000","last_undegraded":"2023-10-14T17:58:27.117139+0000","last_fullsized":"2023-10-14T17:58:27.117139+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:48.874818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.f","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122345+0000","last_change":"2023-10-14T17:58:26.117393+0000","last_active":"2023-10-14T17:58:26.122345+0000","last_peered":"2023-10-14T17:58:26.122345+0000","last_clean":"2023-10-14T17:58:26.122345+0000","last_became_active":"2023-10-14T17:58:26.113067+0000","last_became_peered":"2023-10-14T17:58:26.113067+0000","last_unstale":"2023-10-14T17:58:26.122345+0000","last_undegraded":"2023-10-14T17:58:26.122345+0000","last_fullsized":"2023-10-14T17:58:26.122345+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:59:24.472886+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.e","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115783+0000","last_change":"2023-10-14T17:58:25.083610+0000","last_active":"2023-10-14T17:58:27.115783+0000","last_peered":"2023-10-14T17:58:27.115783+0000","last_clean":"2023-10-14T17:58:27.115783+0000","last_became_active":"2023-10-14T17:58:25.083262+0000","last_became_peered":"2023-10-14T17:58:25.083262+0000","last_unstale":"2023-10-14T17:58:27.115783+0000","last_undegraded":"2023-10-14T17:58:27.115783+0000","last_fullsized":"2023-10-14T17:58:27.115783+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:30:02.027991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.c","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802431+0000","last_change":"2023-10-14T17:58:26.127380+0000","last_active":"2023-10-14T17:58:27.802431+0000","last_peered":"2023-10-14T17:58:27.802431+0000","last_clean":"2023-10-14T17:58:27.802431+0000","last_became_active":"2023-10-14T17:58:26.125547+0000","last_became_peered":"2023-10-14T17:58:26.125547+0000","last_unstale":"2023-10-14T17:58:27.802431+0000","last_undegraded":"2023-10-14T17:58:27.802431+0000","last_fullsized":"2023-10-14T17:58:27.802431+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:43.401744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"22.d","version":"191'1","reported_seq":12,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.126886+0000","last_change":"2023-10-14T17:58:25.086558+0000","last_active":"2023-10-14T17:58:26.126886+0000","last_peered":"2023-10-14T17:58:26.126886+0000","last_clean":"2023-10-14T17:58:26.126886+0000","last_became_active":"2023-10-14T17:58:25.086220+0000","last_became_peered":"2023-10-14T17:58:25.086220+0000","last_unstale":"2023-10-14T17:58:26.126886+0000","last_undegraded":"2023-10-14T17:58:26.126886+0000","last_fullsized":"2023-10-14T17:58:26.126886+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:19.803554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.d","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.123099+0000","last_change":"2023-10-14T17:58:26.116810+0000","last_active":"2023-10-14T17:58:26.123099+0000","last_peered":"2023-10-14T17:58:26.123099+0000","last_clean":"2023-10-14T17:58:26.123099+0000","last_became_active":"2023-10-14T17:58:26.116497+0000","last_became_peered":"2023-10-14T17:58:26.116497+0000","last_unstale":"2023-10-14T17:58:26.123099+0000","last_undegraded":"2023-10-14T17:58:26.123099+0000","last_fullsized":"2023-10-14T17:58:26.123099+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:59:57.367583+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.c","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.102613+0000","last_change":"2023-10-14T17:58:25.083633+0000","last_active":"2023-10-14T17:58:26.102613+0000","last_peered":"2023-10-14T17:58:26.102613+0000","last_clean":"2023-10-14T17:58:26.102613+0000","last_became_active":"2023-10-14T17:58:25.083361+0000","last_became_peered":"2023-10-14T17:58:25.083361+0000","last_unstale":"2023-10-14T17:58:26.102613+0000","last_undegraded":"2023-10-14T17:58:26.102613+0000","last_fullsized":"2023-10-14T17:58:26.102613+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:46:29.605358+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.a","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115675+0000","last_change":"2023-10-14T17:58:26.124910+0000","last_active":"2023-10-14T17:58:27.115675+0000","last_peered":"2023-10-14T17:58:27.115675+0000","last_clean":"2023-10-14T17:58:27.115675+0000","last_became_active":"2023-10-14T17:58:26.122262+0000","last_became_peered":"2023-10-14T17:58:26.122262+0000","last_unstale":"2023-10-14T17:58:27.115675+0000","last_undegraded":"2023-10-14T17:58:27.115675+0000","last_fullsized":"2023-10-14T17:58:27.115675+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:59.702977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.b","version":"191'2","reported_seq":14,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.802090+0000","last_change":"2023-10-14T17:58:25.088861+0000","last_active":"2023-10-14T17:58:27.802090+0000","last_peered":"2023-10-14T17:58:27.802090+0000","last_clean":"2023-10-14T17:58:27.802090+0000","last_became_active":"2023-10-14T17:58:25.088483+0000","last_became_peered":"2023-10-14T17:58:25.088483+0000","last_unstale":"2023-10-14T17:58:27.802090+0000","last_undegraded":"2023-10-14T17:58:27.802090+0000","last_fullsized":"2023-10-14T17:58:27.802090+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:58.833729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.b","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115729+0000","last_change":"2023-10-14T17:58:26.106849+0000","last_active":"2023-10-14T17:58:26.115729+0000","last_peered":"2023-10-14T17:58:26.115729+0000","last_clean":"2023-10-14T17:58:26.115729+0000","last_became_active":"2023-10-14T17:58:26.106569+0000","last_became_peered":"2023-10-14T17:58:26.106569+0000","last_unstale":"2023-10-14T17:58:26.115729+0000","last_undegraded":"2023-10-14T17:58:26.115729+0000","last_fullsized":"2023-10-14T17:58:26.115729+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:23:17.626488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.a","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.103609+0000","last_change":"2023-10-14T17:58:25.080883+0000","last_active":"2023-10-14T17:58:26.103609+0000","last_peered":"2023-10-14T17:58:26.103609+0000","last_clean":"2023-10-14T17:58:26.103609+0000","last_became_active":"2023-10-14T17:58:25.080146+0000","last_became_peered":"2023-10-14T17:58:25.080146+0000","last_unstale":"2023-10-14T17:58:26.103609+0000","last_undegraded":"2023-10-14T17:58:26.103609+0000","last_fullsized":"2023-10-14T17:58:26.103609+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:16:28.193611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.8","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111385+0000","last_change":"2023-10-14T17:58:26.109913+0000","last_active":"2023-10-14T17:58:26.111385+0000","last_peered":"2023-10-14T17:58:26.111385+0000","last_clean":"2023-10-14T17:58:26.111385+0000","last_became_active":"2023-10-14T17:58:26.109641+0000","last_became_peered":"2023-10-14T17:58:26.109641+0000","last_unstale":"2023-10-14T17:58:26.111385+0000","last_undegraded":"2023-10-14T17:58:26.111385+0000","last_fullsized":"2023-10-14T17:58:26.111385+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:44.265730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"22.9","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801412+0000","last_change":"2023-10-14T17:58:25.085129+0000","last_active":"2023-10-14T17:58:27.801412+0000","last_peered":"2023-10-14T17:58:27.801412+0000","last_clean":"2023-10-14T17:58:27.801412+0000","last_became_active":"2023-10-14T17:58:25.084548+0000","last_became_peered":"2023-10-14T17:58:25.084548+0000","last_unstale":"2023-10-14T17:58:27.801412+0000","last_undegraded":"2023-10-14T17:58:27.801412+0000","last_fullsized":"2023-10-14T17:58:27.801412+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:53.579044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.9","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730542+0000","last_change":"2023-10-14T17:58:26.117900+0000","last_active":"2023-10-14T17:58:27.730542+0000","last_peered":"2023-10-14T17:58:27.730542+0000","last_clean":"2023-10-14T17:58:27.730542+0000","last_became_active":"2023-10-14T17:58:26.117146+0000","last_became_peered":"2023-10-14T17:58:26.117146+0000","last_unstale":"2023-10-14T17:58:27.730542+0000","last_undegraded":"2023-10-14T17:58:27.730542+0000","last_fullsized":"2023-10-14T17:58:27.730542+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:25.563550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.8","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.101393+0000","last_change":"2023-10-14T17:58:25.085659+0000","last_active":"2023-10-14T17:58:26.101393+0000","last_peered":"2023-10-14T17:58:26.101393+0000","last_clean":"2023-10-14T17:58:26.101393+0000","last_became_active":"2023-10-14T17:58:25.085340+0000","last_became_peered":"2023-10-14T17:58:25.085340+0000","last_unstale":"2023-10-14T17:58:26.101393+0000","last_undegraded":"2023-10-14T17:58:26.101393+0000","last_fullsized":"2023-10-14T17:58:26.101393+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:33:07.933202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"23.6","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.130396+0000","last_change":"2023-10-14T17:58:26.126937+0000","last_active":"2023-10-14T17:58:26.130396+0000","last_peered":"2023-10-14T17:58:26.130396+0000","last_clean":"2023-10-14T17:58:26.130396+0000","last_became_active":"2023-10-14T17:58:26.126640+0000","last_became_peered":"2023-10-14T17:58:26.126640+0000","last_unstale":"2023-10-14T17:58:26.130396+0000","last_undegraded":"2023-10-14T17:58:26.130396+0000","last_fullsized":"2023-10-14T17:58:26.130396+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:46.255482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.7","version":"191'2","reported_seq":13,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.132312+0000","last_change":"2023-10-14T17:58:25.076934+0000","last_active":"2023-10-14T17:58:26.132312+0000","last_peered":"2023-10-14T17:58:26.132312+0000","last_clean":"2023-10-14T17:58:26.132312+0000","last_became_active":"2023-10-14T17:58:25.076482+0000","last_became_peered":"2023-10-14T17:58:25.076482+0000","last_unstale":"2023-10-14T17:58:26.132312+0000","last_undegraded":"2023-10-14T17:58:26.132312+0000","last_fullsized":"2023-10-14T17:58:26.132312+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:25.543596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.7","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.111288+0000","last_change":"2023-10-14T17:58:26.105200+0000","last_active":"2023-10-14T17:58:26.111288+0000","last_peered":"2023-10-14T17:58:26.111288+0000","last_clean":"2023-10-14T17:58:26.111288+0000","last_became_active":"2023-10-14T17:58:26.102252+0000","last_became_peered":"2023-10-14T17:58:26.102252+0000","last_unstale":"2023-10-14T17:58:26.111288+0000","last_undegraded":"2023-10-14T17:58:26.111288+0000","last_fullsized":"2023-10-14T17:58:26.111288+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:30:03.043323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.6","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.115998+0000","last_change":"2023-10-14T17:58:25.081398+0000","last_active":"2023-10-14T17:58:27.115998+0000","last_peered":"2023-10-14T17:58:27.115998+0000","last_clean":"2023-10-14T17:58:27.115998+0000","last_became_active":"2023-10-14T17:58:25.080936+0000","last_became_peered":"2023-10-14T17:58:25.080936+0000","last_unstale":"2023-10-14T17:58:27.115998+0000","last_undegraded":"2023-10-14T17:58:27.115998+0000","last_fullsized":"2023-10-14T17:58:27.115998+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:56:30.162330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"23.4","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.730905+0000","last_change":"2023-10-14T17:58:26.105754+0000","last_active":"2023-10-14T17:58:27.730905+0000","last_peered":"2023-10-14T17:58:27.730905+0000","last_clean":"2023-10-14T17:58:27.730905+0000","last_became_active":"2023-10-14T17:58:26.105466+0000","last_became_peered":"2023-10-14T17:58:26.105466+0000","last_unstale":"2023-10-14T17:58:27.730905+0000","last_undegraded":"2023-10-14T17:58:27.730905+0000","last_fullsized":"2023-10-14T17:58:27.730905+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:57:27.045397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"22.5","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120717+0000","last_change":"2023-10-14T17:58:25.067263+0000","last_active":"2023-10-14T17:58:27.120717+0000","last_peered":"2023-10-14T17:58:27.120717+0000","last_clean":"2023-10-14T17:58:27.120717+0000","last_became_active":"2023-10-14T17:58:25.065774+0000","last_became_peered":"2023-10-14T17:58:25.065774+0000","last_unstale":"2023-10-14T17:58:27.120717+0000","last_undegraded":"2023-10-14T17:58:27.120717+0000","last_fullsized":"2023-10-14T17:58:27.120717+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:20:35.135231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.5","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.122477+0000","last_change":"2023-10-14T17:58:26.119542+0000","last_active":"2023-10-14T17:58:26.122477+0000","last_peered":"2023-10-14T17:58:26.122477+0000","last_clean":"2023-10-14T17:58:26.122477+0000","last_became_active":"2023-10-14T17:58:26.117553+0000","last_became_peered":"2023-10-14T17:58:26.117553+0000","last_unstale":"2023-10-14T17:58:26.122477+0000","last_undegraded":"2023-10-14T17:58:26.122477+0000","last_fullsized":"2023-10-14T17:58:26.122477+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:59:53.826190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.4","version":"0'0","reported_seq":11,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.116106+0000","last_change":"2023-10-14T17:58:25.080562+0000","last_active":"2023-10-14T17:58:26.116106+0000","last_peered":"2023-10-14T17:58:26.116106+0000","last_clean":"2023-10-14T17:58:26.116106+0000","last_became_active":"2023-10-14T17:58:25.080254+0000","last_became_peered":"2023-10-14T17:58:25.080254+0000","last_unstale":"2023-10-14T17:58:26.116106+0000","last_undegraded":"2023-10-14T17:58:26.116106+0000","last_fullsized":"2023-10-14T17:58:26.116106+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:59.855683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"23.2","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.127057+0000","last_change":"2023-10-14T17:58:26.125490+0000","last_active":"2023-10-14T17:58:26.127057+0000","last_peered":"2023-10-14T17:58:26.127057+0000","last_clean":"2023-10-14T17:58:26.127057+0000","last_became_active":"2023-10-14T17:58:26.124256+0000","last_became_peered":"2023-10-14T17:58:26.124256+0000","last_unstale":"2023-10-14T17:58:26.127057+0000","last_undegraded":"2023-10-14T17:58:26.127057+0000","last_fullsized":"2023-10-14T17:58:26.127057+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:47:25.405674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.3","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118791+0000","last_change":"2023-10-14T17:58:25.065584+0000","last_active":"2023-10-14T17:58:27.118791+0000","last_peered":"2023-10-14T17:58:27.118791+0000","last_clean":"2023-10-14T17:58:27.118791+0000","last_became_active":"2023-10-14T17:58:25.065056+0000","last_became_peered":"2023-10-14T17:58:25.065056+0000","last_unstale":"2023-10-14T17:58:27.118791+0000","last_undegraded":"2023-10-14T17:58:27.118791+0000","last_fullsized":"2023-10-14T17:58:27.118791+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:42:48.033775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.3","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.118558+0000","last_change":"2023-10-14T17:58:26.108381+0000","last_active":"2023-10-14T17:58:27.118558+0000","last_peered":"2023-10-14T17:58:27.118558+0000","last_clean":"2023-10-14T17:58:27.118558+0000","last_became_active":"2023-10-14T17:58:26.104571+0000","last_became_peered":"2023-10-14T17:58:26.104571+0000","last_unstale":"2023-10-14T17:58:27.118558+0000","last_undegraded":"2023-10-14T17:58:27.118558+0000","last_fullsized":"2023-10-14T17:58:27.118558+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:40:27.777182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"22.2","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.801489+0000","last_change":"2023-10-14T17:58:25.086607+0000","last_active":"2023-10-14T17:58:27.801489+0000","last_peered":"2023-10-14T17:58:27.801489+0000","last_clean":"2023-10-14T17:58:27.801489+0000","last_became_active":"2023-10-14T17:58:25.086007+0000","last_became_peered":"2023-10-14T17:58:25.086007+0000","last_unstale":"2023-10-14T17:58:27.801489+0000","last_undegraded":"2023-10-14T17:58:27.801489+0000","last_fullsized":"2023-10-14T17:58:27.801489+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:43.862488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"23.0","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.115222+0000","last_change":"2023-10-14T17:58:26.111241+0000","last_active":"2023-10-14T17:58:26.115222+0000","last_peered":"2023-10-14T17:58:26.115222+0000","last_clean":"2023-10-14T17:58:26.115222+0000","last_became_active":"2023-10-14T17:58:26.110733+0000","last_became_peered":"2023-10-14T17:58:26.110733+0000","last_unstale":"2023-10-14T17:58:26.115222+0000","last_undegraded":"2023-10-14T17:58:26.115222+0000","last_fullsized":"2023-10-14T17:58:26.115222+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:49.890150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"22.1","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.120469+0000","last_change":"2023-10-14T17:58:25.065905+0000","last_active":"2023-10-14T17:58:27.120469+0000","last_peered":"2023-10-14T17:58:27.120469+0000","last_clean":"2023-10-14T17:58:27.120469+0000","last_became_active":"2023-10-14T17:58:25.064248+0000","last_became_peered":"2023-10-14T17:58:25.064248+0000","last_unstale":"2023-10-14T17:58:27.120469+0000","last_undegraded":"2023-10-14T17:58:27.120469+0000","last_fullsized":"2023-10-14T17:58:27.120469+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:02.245682+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"23.1","version":"0'0","reported_seq":11,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.116101+0000","last_change":"2023-10-14T17:58:26.120747+0000","last_active":"2023-10-14T17:58:27.116101+0000","last_peered":"2023-10-14T17:58:27.116101+0000","last_clean":"2023-10-14T17:58:27.116101+0000","last_became_active":"2023-10-14T17:58:26.120447+0000","last_became_peered":"2023-10-14T17:58:26.120447+0000","last_unstale":"2023-10-14T17:58:27.116101+0000","last_undegraded":"2023-10-14T17:58:27.116101+0000","last_fullsized":"2023-10-14T17:58:27.116101+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:56:35.001471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"22.0","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.117953+0000","last_change":"2023-10-14T17:58:25.069216+0000","last_active":"2023-10-14T17:58:27.117953+0000","last_peered":"2023-10-14T17:58:27.117953+0000","last_clean":"2023-10-14T17:58:27.117953+0000","last_became_active":"2023-10-14T17:58:25.068597+0000","last_became_peered":"2023-10-14T17:58:25.068597+0000","last_unstale":"2023-10-14T17:58:27.117953+0000","last_undegraded":"2023-10-14T17:58:27.117953+0000","last_fullsized":"2023-10-14T17:58:27.117953+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:27.910859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"23.16","version":"0'0","reported_seq":9,"reported_epoch":191,"state":"active+clean","last_fresh":"2023-10-14T17:58:26.131610+0000","last_change":"2023-10-14T17:58:26.130535+0000","last_active":"2023-10-14T17:58:26.131610+0000","last_peered":"2023-10-14T17:58:26.131610+0000","last_clean":"2023-10-14T17:58:26.131610+0000","last_became_active":"2023-10-14T17:58:26.130063+0000","last_became_peered":"2023-10-14T17:58:26.130063+0000","last_unstale":"2023-10-14T17:58:26.131610+0000","last_undegraded":"2023-10-14T17:58:26.131610+0000","last_fullsized":"2023-10-14T17:58:26.131610+0000","mapping_epoch":190,"log_start":"0'0","ondisk_log_start":"0'0","created":190,"last_epoch_clean":191,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:25.040333+0000","last_clean_scrub_stamp":"2023-10-14T17:58:25.040333+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:54.104390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"22.17","version":"0'0","reported_seq":12,"reported_epoch":192,"state":"active+clean","last_fresh":"2023-10-14T17:58:27.119028+0000","last_change":"2023-10-14T17:58:25.086748+0000","last_active":"2023-10-14T17:58:27.119028+0000","last_peered":"2023-10-14T17:58:27.119028+0000","last_clean":"2023-10-14T17:58:27.119028+0000","last_became_active":"2023-10-14T17:58:25.086402+0000","last_became_peered":"2023-10-14T17:58:25.086402+0000","last_unstale":"2023-10-14T17:58:27.119028+0000","last_undegraded":"2023-10-14T17:58:27.119028+0000","last_fullsized":"2023-10-14T17:58:27.119028+0000","mapping_epoch":189,"log_start":"0'0","ondisk_log_start":"0'0","created":189,"last_epoch_clean":190,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T17:58:24.025001+0000","last_clean_scrub_stamp":"2023-10-14T17:58:24.025001+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:58.833729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"180'293","reported_seq":972,"reported_epoch":190,"state":"active+clean","last_fresh":"2023-10-14T17:58:25.073378+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T17:58:25.073378+0000","last_peered":"2023-10-14T17:58:25.073378+0000","last_clean":"2023-10-14T17:58:25.073378+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T17:58:25.073378+0000","last_undegraded":"2023-10-14T17:58:25.073378+0000","last_fullsized":"2023-10-14T17:58:25.073378+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":293,"log_dups_size":0,"ondisk_log_size":293,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":23,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":22,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1652,"num_read_kb":3396,"num_write":479,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":293,"ondisk_log_size":293,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575780,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52164,"kb_used_data":2936,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319676,"statfs":{"total":96636764160,"available":96583348224,"internally_reserved":0,"allocated":3006464,"data_stored":1764267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":1.1830000000000001,"15min":1.2050000000000001},"min":{"1min":0.77300000000000002,"5min":0.73099999999999998,"15min":0.71199999999999997},"max":{"1min":1.429,"5min":1.9019999999999999,"15min":1.9710000000000001},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.2030000000000001,"15min":1.2050000000000001},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.8160000000000001,"5min":1.8160000000000001,"15min":1.837},"last":1.008}]},{"osd":1,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.3899999999999999,"5min":1.347,"15min":1.3100000000000001},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.0649999999999999,"5min":2.0649999999999999,"15min":2.0649999999999999},"last":0.96599999999999997},{"interface":"front","average":{"1min":1.266,"5min":1.29,"15min":1.306},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.5720000000000001,"5min":1.8,"15min":1.804},"last":1.254}]},{"osd":2,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.2430000000000001,"15min":1.2509999999999999},"min":{"1min":0.873,"5min":0.82899999999999996,"15min":0.82199999999999995},"max":{"1min":1.7909999999999999,"5min":1.7909999999999999,"15min":1.7909999999999999},"last":0.88200000000000001},{"interface":"front","average":{"1min":1.472,"5min":1.393,"15min":1.3779999999999999},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":1.9179999999999999,"5min":1.9179999999999999,"15min":1.9179999999999999},"last":1.097}]},{"osd":3,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.4630000000000001,"5min":1.4199999999999999,"15min":1.4059999999999999},"min":{"1min":0.996,"5min":0.90400000000000003,"15min":0.90400000000000003},"max":{"1min":1.883,"5min":1.883,"15min":2.234},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.504,"5min":1.5429999999999999,"15min":1.5329999999999999},"min":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.9410000000000001,"5min":2.1600000000000001,"15min":2.1600000000000001},"last":1.3140000000000001}]},{"osd":4,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.335,"15min":1.276},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"max":{"1min":2.1949999999999998,"5min":2.1949999999999998,"15min":2.1949999999999998},"last":1.5720000000000001},{"interface":"front","average":{"1min":1.512,"5min":1.403,"15min":1.373},"min":{"1min":0.90500000000000003,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":2.0459999999999998,"5min":2.0459999999999998,"15min":2.5510000000000002},"last":1.524}]},{"osd":5,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.53,"5min":1.464,"15min":1.458},"min":{"1min":0.63600000000000001,"5min":0.63600000000000001,"15min":0.63600000000000001},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0539999999999998},"last":1.474},{"interface":"front","average":{"1min":1.5389999999999999,"5min":1.4930000000000001,"15min":1.474},"min":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"max":{"1min":2.218,"5min":2.218,"15min":2.6619999999999999},"last":1.7829999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:46 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.6140000000000001,"15min":1.585},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":2.3860000000000001,"5min":2.3860000000000001,"15min":2.4430000000000001},"last":2.117},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.5900000000000001,"15min":1.538},"min":{"1min":1.3440000000000001,"5min":1.0149999999999999,"15min":0.91600000000000004},"max":{"1min":2.1440000000000001,"5min":2.3100000000000001,"15min":2.4100000000000001},"last":1.6739999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575779,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41856,"kb_used_data":2348,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329984,"statfs":{"total":96636764160,"available":96593903616,"internally_reserved":0,"allocated":2404352,"data_stored":1177983,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40434119},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.1950000000000001,"15min":1.1879999999999999},"min":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.73999999999999999},"max":{"1min":1.4870000000000001,"5min":1.696,"15min":1.837},"last":1.5600000000000001},{"interface":"front","average":{"1min":1.081,"5min":1.1850000000000001,"15min":1.2010000000000001},"min":{"1min":0.76500000000000001,"5min":0.72799999999999998,"15min":0.72799999999999998},"max":{"1min":1.401,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.131}]},{"osd":1,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.161,"5min":1.276,"15min":1.244},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"max":{"1min":1.5640000000000001,"5min":1.752,"15min":1.927},"last":0.92000000000000004},{"interface":"front","average":{"1min":1.2450000000000001,"5min":1.2989999999999999,"15min":1.3220000000000001},"min":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.6040000000000001,"5min":1.7849999999999999,"15min":1.8069999999999999},"last":0.85599999999999998}]},{"osd":2,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.3879999999999999,"15min":1.383},"min":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86299999999999999},"max":{"1min":1.8180000000000001,"5min":1.8180000000000001,"15min":1.8740000000000001},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.373,"5min":1.454,"15min":1.4450000000000001},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.7170000000000001,"5min":1.8480000000000001,"15min":1.9370000000000001},"last":1.4059999999999999}]},{"osd":3,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.3700000000000001,"5min":1.415,"15min":1.417},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.8300000000000001,"5min":1.8300000000000001,"15min":2.0409999999999999},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.3560000000000001,"5min":1.393,"15min":1.431},"min":{"1min":0.873,"5min":0.873,"15min":0.873},"max":{"1min":1.764,"5min":1.903,"15min":1.9790000000000001},"last":1.2709999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4710000000000001,"5min":1.361,"15min":1.367},"min":{"1min":1.014,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":1.736,"5min":1.929,"15min":1.984},"last":1.6240000000000001},{"interface":"front","average":{"1min":1.4710000000000001,"5min":1.4379999999999999,"15min":1.4239999999999999},"min":{"1min":1.1799999999999999,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":1.889,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.6519999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.3879999999999999,"15min":1.361},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.79000000000000004},"max":{"1min":1.629,"5min":1.7749999999999999,"15min":2.085},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.575,"5min":1.575,"15min":1.5580000000000001},"min":{"1min":1.214,"5min":1.048,"15min":1.048},"max":{"1min":1.9450000000000001,"5min":2.0310000000000001,"15min":2.105},"last":1.6919999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:57:35 2023","interfaces":[{"interface":"back","average":{"1min":1.6240000000000001,"5min":1.5900000000000001,"15min":1.5840000000000001},"min":{"1min":1.165,"5min":1.026,"15min":1.0189999999999999},"max":{"1min":2.0960000000000001,"5min":2.1829999999999998,"15min":2.1829999999999998},"last":1.718},{"interface":"front","average":{"1min":1.629,"5min":1.5640000000000001,"15min":1.5720000000000001},"min":{"1min":1.268,"5min":1.0129999999999999,"15min":0.81000000000000005},"max":{"1min":1.8700000000000001,"5min":1.954,"15min":2.1589999999999998},"last":1.585}]}]},{"osd":5,"up_from":13,"seq":55834575780,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37700,"kb_used_data":2352,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94334140,"statfs":{"total":96636764160,"available":96598159360,"internally_reserved":0,"allocated":2408448,"data_stored":1178565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.099,"5min":1.2070000000000001,"15min":1.1870000000000001},"min":{"1min":0.47499999999999998,"5min":0.47499999999999998,"15min":0.47499999999999998},"max":{"1min":1.4359999999999999,"5min":1.6870000000000001,"15min":1.7949999999999999},"last":0.97399999999999998},{"interface":"front","average":{"1min":1.1359999999999999,"5min":1.1599999999999999,"15min":1.2090000000000001},"min":{"1min":0.85299999999999998,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6040000000000001,"5min":1.677,"15min":2.0150000000000001},"last":1.1339999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.1830000000000001,"5min":1.2270000000000001,"15min":1.2350000000000001},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.623,"5min":1.774,"15min":1.774},"last":1.3029999999999999},{"interface":"front","average":{"1min":1.2809999999999999,"5min":1.286,"15min":1.286},"min":{"1min":0.85399999999999998,"5min":0.67300000000000004,"15min":0.67300000000000004},"max":{"1min":1.78,"5min":1.7949999999999999,"15min":1.8280000000000001},"last":1.22}]},{"osd":2,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.143,"5min":1.244,"15min":1.2430000000000001},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":1.462,"5min":1.6879999999999999,"15min":1.72},"last":1.044},{"interface":"front","average":{"1min":1.2609999999999999,"5min":1.331,"15min":1.353},"min":{"1min":0.91700000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.6779999999999999,"5min":1.899,"15min":2.4140000000000001},"last":1.4390000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.458,"15min":1.444},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"max":{"1min":2.0350000000000001,"5min":2.0350000000000001,"15min":2.0350000000000001},"last":1.389},{"interface":"front","average":{"1min":1.333,"5min":1.3939999999999999,"15min":1.4319999999999999},"min":{"1min":0.63,"5min":0.63,"15min":0.63},"max":{"1min":1.655,"5min":1.8799999999999999,"15min":2.5219999999999998},"last":1.595}]},{"osd":4,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.3220000000000001,"15min":1.3260000000000001},"min":{"1min":0.89800000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.607,"5min":1.9850000000000001,"15min":2.7210000000000001},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.381,"15min":1.393},"min":{"1min":0.96999999999999997,"5min":0.82899999999999996,"15min":0.82899999999999996},"max":{"1min":2.1669999999999998,"5min":2.1669999999999998,"15min":2.1669999999999998},"last":1.2809999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.476,"5min":1.4119999999999999,"15min":1.3819999999999999},"min":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.7949999999999999},"last":1.8169999999999999},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.577,"15min":1.581},"min":{"1min":0.93200000000000005,"5min":0.93200000000000005,"15min":0.86599999999999999},"max":{"1min":1.804,"5min":2.1389999999999998,"15min":2.907},"last":1.883}]},{"osd":7,"last update":"Sat Oct 14 17:57:31 2023","interfaces":[{"interface":"back","average":{"1min":1.5580000000000001,"5min":1.5509999999999999,"15min":1.544},"min":{"1min":1.073,"5min":0.77300000000000002,"15min":0.77300000000000002},"max":{"1min":1.7889999999999999,"5min":2.1539999999999999,"15min":2.1760000000000002},"last":1.5469999999999999},{"interface":"front","average":{"1min":1.413,"5min":1.5049999999999999,"15min":1.516},"min":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"max":{"1min":1.873,"5min":2.0009999999999999,"15min":2.9630000000000001},"last":1.923}]}]},{"osd":4,"up_from":13,"seq":55834575780,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55748,"kb_used_data":2360,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94316092,"statfs":{"total":96636764160,"available":96579678208,"internally_reserved":0,"allocated":2416640,"data_stored":1179643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.2689999999999999,"15min":1.2270000000000001},"min":{"1min":0.81000000000000005,"5min":0.73699999999999999,"15min":0.52500000000000002},"max":{"1min":1.732,"5min":1.7749999999999999,"15min":1.8859999999999999},"last":1.1339999999999999},{"interface":"front","average":{"1min":1.329,"5min":1.248,"15min":1.272},"min":{"1min":0.97299999999999998,"5min":0.86699999999999999,"15min":0.84599999999999997},"max":{"1min":1.897,"5min":1.897,"15min":1.905},"last":1.599}]},{"osd":1,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3100000000000001,"15min":1.3220000000000001},"min":{"1min":0.89100000000000001,"5min":0.874,"15min":0.83999999999999997},"max":{"1min":1.8260000000000001,"5min":1.8260000000000001,"15min":1.8260000000000001},"last":1.4419999999999999},{"interface":"front","average":{"1min":1.224,"5min":1.278,"15min":1.2829999999999999},"min":{"1min":0.747,"5min":0.747,"15min":0.747},"max":{"1min":1.7310000000000001,"5min":1.7629999999999999,"15min":2.02},"last":1.03}]},{"osd":2,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.29,"15min":1.298},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.63800000000000001},"max":{"1min":1.827,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":0.91100000000000003},{"interface":"front","average":{"1min":1.385,"5min":1.4019999999999999,"15min":1.409},"min":{"1min":1.0960000000000001,"5min":1.0209999999999999,"15min":0.84199999999999997},"max":{"1min":1.8680000000000001,"5min":1.9279999999999999,"15min":1.9279999999999999},"last":1.294}]},{"osd":3,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.347,"15min":1.353},"min":{"1min":0.93000000000000005,"5min":0.92400000000000004,"15min":0.92000000000000004},"max":{"1min":1.6439999999999999,"5min":1.752,"15min":1.984},"last":1.3660000000000001},{"interface":"front","average":{"1min":1.552,"5min":1.492,"15min":1.464},"min":{"1min":1.1930000000000001,"5min":1.0109999999999999,"15min":0.82299999999999995},"max":{"1min":1.964,"5min":1.964,"15min":1.964},"last":1.339}]},{"osd":5,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.492,"5min":1.4139999999999999,"15min":1.3640000000000001},"min":{"1min":0.97399999999999998,"5min":0.91500000000000004,"15min":0.82899999999999996},"max":{"1min":1.976,"5min":1.976,"15min":1.976},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.3779999999999999,"15min":1.3740000000000001},"min":{"1min":0.92100000000000004,"5min":0.92100000000000004,"15min":0.79700000000000004},"max":{"1min":1.925,"5min":1.925,"15min":2.0499999999999998},"last":1.5249999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.502,"5min":1.4219999999999999,"15min":1.3819999999999999},"min":{"1min":1.004,"5min":0.874,"15min":0.57999999999999996},"max":{"1min":1.9239999999999999,"5min":2.117,"15min":2.117},"last":1.234},{"interface":"front","average":{"1min":1.673,"5min":1.5600000000000001,"15min":1.5349999999999999},"min":{"1min":1.202,"5min":1.133,"15min":1.006},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.0790000000000002},"last":1.6910000000000001}]},{"osd":7,"last update":"Sat Oct 14 17:57:34 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.5609999999999999,"15min":1.506},"min":{"1min":1.0780000000000001,"5min":0.92500000000000004,"15min":0.86799999999999999},"max":{"1min":2.0880000000000001,"5min":2.2309999999999999,"15min":2.2309999999999999},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.696,"5min":1.6319999999999999,"15min":1.593},"min":{"1min":1.4410000000000001,"5min":1.179,"15min":0.70999999999999996},"max":{"1min":2.0059999999999998,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.5629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575780,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51568,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320272,"statfs":{"total":96636764160,"available":96583958528,"internally_reserved":0,"allocated":2392064,"data_stored":1171579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.88900000000000001,"5min":0.98199999999999998,"15min":0.98099999999999998},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":1.423,"5min":1.7949999999999999,"15min":2.6680000000000001},"last":1.03},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.1890000000000001,"15min":1.206},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.47999999999999998},"max":{"1min":1.5469999999999999,"5min":1.6240000000000001,"15min":2.8290000000000002},"last":1.3129999999999999}]},{"osd":1,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":0.98899999999999999,"5min":1.0369999999999999,"15min":1.008},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":1.7270000000000001,"5min":1.819,"15min":1.843},"last":0.77500000000000002},{"interface":"front","average":{"1min":1.131,"5min":1.232,"15min":1.21},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":1.7929999999999999,"5min":1.849,"15min":2.1970000000000001},"last":0.95699999999999996}]},{"osd":2,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3129999999999999,"5min":1.454,"15min":1.4550000000000001},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":1.6759999999999999,"5min":1.9299999999999999,"15min":2.1000000000000001},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.395,"5min":1.427,"15min":1.4019999999999999},"min":{"1min":0.80000000000000004,"5min":0.79500000000000004,"15min":0.55100000000000005},"max":{"1min":1.8979999999999999,"5min":2.0419999999999998,"15min":2.0419999999999998},"last":0.79200000000000004}]},{"osd":4,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3979999999999999,"5min":1.4550000000000001,"15min":1.427},"min":{"1min":1.0249999999999999,"5min":1.0249999999999999,"15min":0.92500000000000004},"max":{"1min":1.716,"5min":1.8660000000000001,"15min":2.0489999999999999},"last":1.29},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.48,"15min":1.468},"min":{"1min":0.96599999999999997,"5min":0.96299999999999997,"15min":0.86099999999999999},"max":{"1min":1.9079999999999999,"5min":2.2210000000000001,"15min":2.6459999999999999},"last":1.083}]},{"osd":5,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.4590000000000001,"15min":1.478},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.73399999999999999},"max":{"1min":2.0859999999999999,"5min":2.0859999999999999,"15min":2.5830000000000002},"last":1.004},{"interface":"front","average":{"1min":1.413,"5min":1.514,"15min":1.52},"min":{"1min":0.93000000000000005,"5min":0.93000000000000005,"15min":0.93000000000000005},"max":{"1min":1.7949999999999999,"5min":1.9850000000000001,"15min":2.3719999999999999},"last":1.421}]},{"osd":6,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.6699999999999999,"15min":1.6419999999999999},"min":{"1min":1.0589999999999999,"5min":1.0589999999999999,"15min":1.0509999999999999},"max":{"1min":1.893,"5min":2.6480000000000001,"15min":2.851},"last":1.274},{"interface":"front","average":{"1min":1.5620000000000001,"5min":1.669,"15min":1.679},"min":{"1min":1.048,"5min":1.048,"15min":1.048},"max":{"1min":2.0289999999999999,"5min":2.7109999999999999,"15min":2.9910000000000001},"last":1.218}]},{"osd":7,"last update":"Sat Oct 14 17:57:29 2023","interfaces":[{"interface":"back","average":{"1min":1.5569999999999999,"5min":1.643,"15min":1.6200000000000001},"min":{"1min":1.163,"5min":1.163,"15min":0.79500000000000004},"max":{"1min":2.2040000000000002,"5min":2.4140000000000001,"15min":2.9430000000000001},"last":1.246},{"interface":"front","average":{"1min":1.583,"5min":1.6679999999999999,"15min":1.6839999999999999},"min":{"1min":1.2330000000000001,"5min":1.2250000000000001,"15min":1.137},"max":{"1min":1.897,"5min":2.0899999999999999,"15min":2.4980000000000002},"last":1.3580000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575780,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51564,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320276,"statfs":{"total":96636764160,"available":96583962624,"internally_reserved":0,"allocated":2392064,"data_stored":1171635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":0.91900000000000004,"5min":0.95499999999999996,"15min":0.93600000000000005},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":1.3520000000000001,"5min":1.6879999999999999,"15min":1.8759999999999999},"last":1.097},{"interface":"front","average":{"1min":1.03,"5min":1.089,"15min":1.115},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.6120000000000001,"5min":1.7490000000000001,"15min":1.7889999999999999},"last":1.163}]},{"osd":1,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.0009999999999999,"5min":1.03,"15min":1.0249999999999999},"min":{"1min":0.52300000000000002,"5min":0.47099999999999997,"15min":0.47099999999999997},"max":{"1min":1.5720000000000001,"5min":1.6080000000000001,"15min":1.8620000000000001},"last":1.0600000000000001},{"interface":"front","average":{"1min":1.1799999999999999,"5min":1.1950000000000001,"15min":1.224},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":1.764,"5min":1.8129999999999999,"15min":1.8959999999999999},"last":1.28}]},{"osd":3,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2210000000000001,"15min":1.274},"min":{"1min":0.627,"5min":0.627,"15min":0.627},"max":{"1min":1.5600000000000001,"5min":1.929,"15min":1.929},"last":1.135},{"interface":"front","average":{"1min":1.2969999999999999,"5min":1.375,"15min":1.3640000000000001},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.8089999999999999,"5min":1.95,"15min":2.0289999999999999},"last":1.4359999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.458,"5min":1.472,"15min":1.4790000000000001},"min":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":0.78400000000000003},"max":{"1min":2.0990000000000002,"5min":2.0990000000000002,"15min":2.153},"last":1.621},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.3740000000000001,"15min":1.4099999999999999},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.917,"5min":2.0640000000000001,"15min":2.2639999999999998},"last":1.3740000000000001}]},{"osd":5,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4370000000000001,"5min":1.4319999999999999,"15min":1.462},"min":{"1min":0.88500000000000001,"5min":0.86299999999999999,"15min":0.86299999999999999},"max":{"1min":1.865,"5min":1.9359999999999999,"15min":2.2669999999999999},"last":1.968},{"interface":"front","average":{"1min":1.611,"5min":1.591,"15min":1.585},"min":{"1min":0.94599999999999995,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.0070000000000001,"5min":2.0880000000000001,"15min":2.3199999999999998},"last":2.1520000000000001}]},{"osd":6,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.5169999999999999,"15min":1.52},"min":{"1min":0.98099999999999998,"5min":0.89600000000000002,"15min":0.89600000000000002},"max":{"1min":2.0470000000000002,"5min":2.2360000000000002,"15min":2.2360000000000002},"last":1.698},{"interface":"front","average":{"1min":1.54,"5min":1.583,"15min":1.6120000000000001},"min":{"1min":0.97199999999999998,"5min":0.97199999999999998,"15min":0.97199999999999998},"max":{"1min":2.0169999999999999,"5min":2.161,"15min":2.3610000000000002},"last":2.044}]},{"osd":7,"last update":"Sat Oct 14 17:57:39 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.6040000000000001,"15min":1.5940000000000001},"min":{"1min":1.008,"5min":0.77500000000000002,"15min":0.77500000000000002},"max":{"1min":2.1520000000000001,"5min":2.1520000000000001,"15min":2.1640000000000001},"last":1.4710000000000001},{"interface":"front","average":{"1min":1.6359999999999999,"5min":1.645,"15min":1.677},"min":{"1min":1.0660000000000001,"5min":0.98699999999999999,"15min":0.98699999999999999},"max":{"1min":2.137,"5min":2.177,"15min":2.4350000000000001},"last":1.847}]}]},{"osd":1,"up_from":13,"seq":55834575780,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":47408,"kb_used_data":2336,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94324432,"statfs":{"total":96636764160,"available":96588218368,"internally_reserved":0,"allocated":2392064,"data_stored":1171579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.93400000000000005,"15min":0.93500000000000005},"min":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"max":{"1min":1.351,"5min":1.706,"15min":1.756},"last":1.274},{"interface":"front","average":{"1min":0.84599999999999997,"5min":1.0860000000000001,"15min":1.135},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":1.379,"5min":1.71,"15min":1.7849999999999999},"last":0.89200000000000002}]},{"osd":2,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":0.81499999999999995,"5min":0.96599999999999997,"15min":1.0229999999999999},"min":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"max":{"1min":1.3029999999999999,"5min":1.7709999999999999,"15min":2.597},"last":1.151},{"interface":"front","average":{"1min":0.94499999999999995,"5min":1.1870000000000001,"15min":1.2290000000000001},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.5249999999999999,"5min":1.7250000000000001,"15min":1.9610000000000001},"last":1.9590000000000001}]},{"osd":3,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.179,"5min":1.353,"15min":1.3600000000000001},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.9770000000000001,"5min":1.9770000000000001,"15min":1.9770000000000001},"last":1.077},{"interface":"front","average":{"1min":0.98099999999999998,"5min":1.242,"15min":1.272},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.5029999999999999,"5min":1.7829999999999999,"15min":2.2599999999999998},"last":1.8069999999999999}]},{"osd":4,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3169999999999999,"5min":1.4370000000000001,"15min":1.462},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.78400000000000003},"max":{"1min":1.8460000000000001,"5min":1.8460000000000001,"15min":2.044},"last":1.4259999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.3959999999999999,"15min":1.419},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":2.0539999999999998,"5min":2.0539999999999998,"15min":2.706},"last":1.5509999999999999}]},{"osd":5,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.429,"15min":1.444},"min":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":1.8959999999999999,"5min":1.9390000000000001,"15min":2.3439999999999999},"last":1.8799999999999999},{"interface":"front","average":{"1min":1.399,"5min":1.542,"15min":1.542},"min":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"max":{"1min":2.1589999999999998,"5min":2.379,"15min":2.379},"last":2.2789999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.4299999999999999,"5min":1.5669999999999999,"15min":1.5780000000000001},"min":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"max":{"1min":2.0880000000000001,"5min":2.1259999999999999,"15min":2.177},"last":2.1880000000000002},{"interface":"front","average":{"1min":1.4970000000000001,"5min":1.5740000000000001,"15min":1.5669999999999999},"min":{"1min":0.88800000000000001,"5min":0.88800000000000001,"15min":0.88800000000000001},"max":{"1min":2.1320000000000001,"5min":2.1320000000000001,"15min":2.4860000000000002},"last":2.1219999999999999}]},{"osd":7,"last update":"Sat Oct 14 17:58:10 2023","interfaces":[{"interface":"back","average":{"1min":1.534,"5min":1.6599999999999999,"15min":1.6579999999999999},"min":{"1min":0.95599999999999996,"5min":0.95599999999999996,"15min":0.95599999999999996},"max":{"1min":2.3849999999999998,"5min":2.3849999999999998,"15min":2.3849999999999998},"last":2.3700000000000001},{"interface":"front","average":{"1min":1.4630000000000001,"5min":1.6399999999999999,"15min":1.6619999999999999},"min":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"max":{"1min":2.2639999999999998,"5min":2.2639999999999998,"15min":2.2839999999999998},"last":2.0329999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575779,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64620,"kb_used_data":2912,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307220,"statfs":{"total":96636764160,"available":96570593280,"internally_reserved":0,"allocated":2981888,"data_stored":1761913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.002,"5min":0.97899999999999998,"15min":0.96099999999999997},"min":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"max":{"1min":1.577,"5min":2.3170000000000002,"15min":2.3170000000000002},"last":0.66800000000000004},{"interface":"front","average":{"1min":1.141,"5min":1.1839999999999999,"15min":1.177},"min":{"1min":0.72699999999999998,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.6830000000000001,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.1739999999999999,"15min":1.1499999999999999},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.446,"5min":3.375,"15min":3.375},"last":0.96799999999999997},{"interface":"front","average":{"1min":1.1919999999999999,"5min":1.2569999999999999,"15min":1.2629999999999999},"min":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"max":{"1min":1.9430000000000001,"5min":1.9430000000000001,"15min":2.681},"last":1.01}]},{"osd":3,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.3819999999999999,"15min":1.3660000000000001},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.708,"5min":1.823,"15min":2.1899999999999999},"last":1.0509999999999999},{"interface":"front","average":{"1min":1.383,"5min":1.425,"15min":1.4279999999999999},"min":{"1min":0.98499999999999999,"5min":0.98499999999999999,"15min":0.93999999999999995},"max":{"1min":1.792,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":0.875}]},{"osd":4,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3009999999999999,"5min":1.3480000000000001,"15min":1.3420000000000001},"min":{"1min":1.0029999999999999,"5min":0.92000000000000004,"15min":0.92000000000000004},"max":{"1min":1.675,"5min":2.4079999999999999,"15min":2.4079999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":1.534,"5min":1.5309999999999999,"15min":1.498},"min":{"1min":1.002,"5min":1.002,"15min":0.999},"max":{"1min":2.008,"5min":2.9689999999999999,"15min":2.9689999999999999},"last":1.155}]},{"osd":5,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3560000000000001,"5min":1.401,"15min":1.4119999999999999},"min":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"max":{"1min":1.873,"5min":3.077,"15min":3.077},"last":1.395},{"interface":"front","average":{"1min":1.611,"5min":1.621,"15min":1.6020000000000001},"min":{"1min":1.194,"5min":1.1539999999999999,"15min":1.1539999999999999},"max":{"1min":2.028,"5min":2.468,"15min":2.468},"last":1.4339999999999999}]},{"osd":6,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.659,"5min":1.6799999999999999,"15min":1.681},"min":{"1min":1.2769999999999999,"5min":1.194,"15min":1.147},"max":{"1min":2.1349999999999998,"5min":2.3479999999999999,"15min":2.5270000000000001},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.548,"15min":1.554},"min":{"1min":0.92200000000000004,"5min":0.92200000000000004,"15min":0.92200000000000004},"max":{"1min":1.875,"5min":2.5659999999999998,"15min":2.5659999999999998},"last":1.276}]},{"osd":7,"last update":"Sat Oct 14 17:57:40 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.675,"15min":1.655},"min":{"1min":1.1419999999999999,"5min":1.1419999999999999,"15min":1.1419999999999999},"max":{"1min":1.9399999999999999,"5min":3.5990000000000002,"15min":3.5990000000000002},"last":1.466},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.6619999999999999,"15min":1.655},"min":{"1min":1.036,"5min":1.036,"15min":1.036},"max":{"1min":1.99,"5min":3.645,"15min":3.645},"last":1.4950000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":22,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":23,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T17:58:31.146 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T17:58:31.146 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T17:58:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.425+0000 7ff5e9883640 1 Processor -- start 2023-10-14T17:58:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.425+0000 7ff5e9883640 1 -- start start 2023-10-14T17:58:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e9883640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff5e415ae80 0x7ff5e415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e9883640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5e40ce390 0x7ff5e4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e9883640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff5e40707e0 con 0x7ff5e415aaa0 2023-10-14T17:58:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e9883640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e406b7a0 con 0x7ff5e40ce390 2023-10-14T17:58:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e9883640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff5e415ef20 con 0x7ff5e415ae80 2023-10-14T17:58:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5e40ce390 0x7ff5e4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff5e40ce390 0x7ff5e4070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41236/0 (socket says 172.21.15.29:41236) 2023-10-14T17:58:31.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.427+0000 7ff5e37fe640 1 -- 172.21.15.29:0/3179991350 learned_addr learned my addr 172.21.15.29:0/3179991350 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:31.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.428+0000 7ff5e27fc640 1 --2- 172.21.15.29:0/3179991350 >> v2:172.21.15.186:3300/0 conn(0x7ff5e415ae80 0x7ff5e415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.428+0000 7ff5e37fe640 1 -- 172.21.15.29:0/3179991350 >> v2:172.21.15.186:3300/0 conn(0x7ff5e415ae80 msgr2=0x7ff5e415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:31.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.428+0000 7ff5e37fe640 1 --2- 172.21.15.29:0/3179991350 >> v2:172.21.15.186:3300/0 conn(0x7ff5e415ae80 0x7ff5e415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.428+0000 7ff5e37fe640 1 -- 172.21.15.29:0/3179991350 >> v1:172.21.15.186:6789/0 conn(0x7ff5e415aaa0 legacy=0x7ff5e40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:31.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.428+0000 7ff5e37fe640 1 -- 172.21.15.29:0/3179991350 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5e415f200 con 0x7ff5e40ce390 2023-10-14T17:58:31.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.429+0000 7ff5e37fe640 1 --2- 172.21.15.29:0/3179991350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40ce390 0x7ff5e4070f50 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b45e0 tx=0x7ff5dc02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c95358bfd9aa22 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e1ffb640 1 -- 172.21.15.29:0/3179991350 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc031800 con 0x7ff5e40ce390 2023-10-14T17:58:31.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e1ffb640 1 -- 172.21.15.29:0/3179991350 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc031960 con 0x7ff5e40ce390 2023-10-14T17:58:31.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e1ffb640 1 -- 172.21.15.29:0/3179991350 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc031cf0 con 0x7ff5e40ce390 2023-10-14T17:58:31.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 -- 172.21.15.29:0/3179991350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40ce390 msgr2=0x7ff5e4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 --2- 172.21.15.29:0/3179991350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40ce390 0x7ff5e4070f50 secure :-1 s=READY pgs=2174 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b45e0 tx=0x7ff5dc02b8a0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 -- 172.21.15.29:0/3179991350 shutdown_connections 2023-10-14T17:58:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 --2- 172.21.15.29:0/3179991350 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e40ce390 0x7ff5e4070f50 unknown :-1 s=CLOSED pgs=2174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 --2- 172.21.15.29:0/3179991350 >> v2:172.21.15.186:3300/0 conn(0x7ff5e415ae80 0x7ff5e415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 -- 172.21.15.29:0/3179991350 >> 172.21.15.29:0/3179991350 conn(0x7ff5e40cb410 msgr2=0x7ff5e40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.430+0000 7ff5e9883640 1 -- 172.21.15.29:0/3179991350 shutdown_connections 2023-10-14T17:58:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.431+0000 7ff5e9883640 1 -- 172.21.15.29:0/3179991350 wait complete. 2023-10-14T17:58:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.432+0000 7ff5e9883640 1 Processor -- start 2023-10-14T17:58:31.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.432+0000 7ff5e9883640 1 -- start start 2023-10-14T17:58:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5e40ce390 0x7ff5e40ca460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e415aaa0 0x7ff5e41580e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b0a60 con 0x7ff5e415ae80 2023-10-14T17:58:31.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b2ca0 con 0x7ff5e415aaa0 2023-10-14T17:58:31.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.433+0000 7ff5e9883640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff5e40b0d00 con 0x7ff5e40ce390 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41252/0 (socket says 172.21.15.29:41252) 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 -- 172.21.15.29:0/1083370115 learned_addr learned my addr 172.21.15.29:0/1083370115 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e2ffd640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5e40ce390 0x7ff5e40ca460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e27fc640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e415aaa0 0x7ff5e41580e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 -- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5e40ce390 msgr2=0x7ff5e40ca460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5e40ce390 0x7ff5e40ca460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 -- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e415aaa0 msgr2=0x7ff5e41580e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e415aaa0 0x7ff5e41580e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.434+0000 7ff5e37fe640 1 -- 172.21.15.29:0/1083370115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5dc0094d0 con 0x7ff5e415ae80 2023-10-14T17:58:31.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5e37fe640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc02b8e0 tx=0x7ff5dc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc03abe0 con 0x7ff5e415ae80 2023-10-14T17:58:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff5dc03ad40 con 0x7ff5e415ae80 2023-10-14T17:58:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5dc044630 con 0x7ff5e415ae80 2023-10-14T17:58:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5e40b0fe0 con 0x7ff5e415ae80 2023-10-14T17:58:31.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.435+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5e40b24a0 con 0x7ff5e415ae80 2023-10-14T17:58:31.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.436+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5e404e160 con 0x7ff5e415ae80 2023-10-14T17:58:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.437+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff5dc044790 con 0x7ff5e415ae80 2023-10-14T17:58:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.439+0000 7ff5dbfff640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 0x7ff5c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.439+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff5dc02e070 con 0x7ff5e415ae80 2023-10-14T17:58:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.439+0000 7ff5e2ffd640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 0x7ff5c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.440+0000 7ff5e2ffd640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 0x7ff5c406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b3ad0 tx=0x7ff5d4009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.445+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff5dc076b00 con 0x7ff5e415ae80 2023-10-14T17:58:31.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.720+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff5e40cdc60 con 0x7ff5e415ae80 2023-10-14T17:58:31.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.725+0000 7ff5dbfff640 1 -- 172.21.15.29:0/1083370115 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v43) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff5dc035020 con 0x7ff5e415ae80 2023-10-14T17:58:31.726 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T17:58:31.726 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T17:58:31.727 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T17:58:31.727 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T17:58:31.727 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T17:58:31.727 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T17:58:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.728+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 msgr2=0x7ff5c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.728+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 0x7ff5c406d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff5e40b3ad0 tx=0x7ff5d4009040 comp rx=0 tx=0).stop 2023-10-14T17:58:31.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.728+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 msgr2=0x7ff5e40b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.728+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 secure :-1 s=READY pgs=2175 cs=0 l=1 rev1=1 crypto rx=0x7ff5dc02b8e0 tx=0x7ff5dc0046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 shutdown_connections 2023-10-14T17:58:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff5e415ae80 0x7ff5e40b0520 unknown :-1 s=CLOSED pgs=2175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7ff5c406af90 0x7ff5c406d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff5e415aaa0 0x7ff5e41580e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 --2- 172.21.15.29:0/1083370115 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff5e40ce390 0x7ff5e40ca460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 >> 172.21.15.29:0/1083370115 conn(0x7ff5e40cb410 msgr2=0x7ff5e4158aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 shutdown_connections 2023-10-14T17:58:31.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.729+0000 7ff5e9883640 1 -- 172.21.15.29:0/1083370115 wait complete. 2023-10-14T17:58:31.751 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T17:58:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.976+0000 7f6624a85640 1 Processor -- start 2023-10-14T17:58:31.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.976+0000 7f6624a85640 1 -- start start 2023-10-14T17:58:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.977+0000 7f6624a85640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f662006c040 0x7f662006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f6624a85640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 0x7f662015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f6624a85640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f662006c960 con 0x7f66200ad970 2023-10-14T17:58:31.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f6624a85640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662015b9d0 con 0x7f662006c040 2023-10-14T17:58:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f6624a85640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662015bcb0 con 0x7f66200cda80 2023-10-14T17:58:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f661f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f662006c040 0x7f662006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f661f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f662006c040 0x7f662006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41254/0 (socket says 172.21.15.29:41254) 2023-10-14T17:58:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f661f7fe640 1 -- 172.21.15.29:0/714905875 learned_addr learned my addr 172.21.15.29:0/714905875 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:31.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f6617fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66200ad970 0x7f662015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45484/0 (socket says 172.21.15.29:45484) 2023-10-14T17:58:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f661ffff640 1 --2- 172.21.15.29:0/714905875 >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 0x7f662015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.978+0000 7f661f7fe640 1 -- 172.21.15.29:0/714905875 >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 msgr2=0x7f662015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.979+0000 7f661f7fe640 1 --2- 172.21.15.29:0/714905875 >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 0x7f662015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.979+0000 7f661f7fe640 1 -- 172.21.15.29:0/714905875 >> v1:172.21.15.29:6789/0 conn(0x7f66200ad970 legacy=0x7f662015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.979+0000 7f661f7fe640 1 -- 172.21.15.29:0/714905875 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662015bf90 con 0x7f662006c040 2023-10-14T17:58:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.979+0000 7f661effd640 1 -- 172.21.15.29:0/714905875 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027571128 0 0) 0x7f662006c960 con 0x7f66200ad970 2023-10-14T17:58:31.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.979+0000 7f661ffff640 1 --2- 172.21.15.29:0/714905875 >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 0x7f662015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:31.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f661f7fe640 1 --2- 172.21.15.29:0/714905875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662006c420 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f66200cf300 tx=0x7f661002b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=169188eb7f226f60 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f661effd640 1 -- 172.21.15.29:0/714905875 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610030870 con 0x7f662006c040 2023-10-14T17:58:31.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f661effd640 1 -- 172.21.15.29:0/714905875 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66100309d0 con 0x7f662006c040 2023-10-14T17:58:31.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f661effd640 1 -- 172.21.15.29:0/714905875 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610030d60 con 0x7f662006c040 2023-10-14T17:58:31.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f6624a85640 1 -- 172.21.15.29:0/714905875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 msgr2=0x7f662006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.980+0000 7f6624a85640 1 --2- 172.21.15.29:0/714905875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662006c420 secure :-1 s=READY pgs=2177 cs=0 l=1 rev1=1 crypto rx=0x7f66200cf300 tx=0x7f661002b9b0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 -- 172.21.15.29:0/714905875 shutdown_connections 2023-10-14T17:58:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 --2- 172.21.15.29:0/714905875 >> v2:172.21.15.186:3300/0 conn(0x7f66200cda80 0x7f662015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 --2- 172.21.15.29:0/714905875 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662006c420 unknown :-1 s=CLOSED pgs=2177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 -- 172.21.15.29:0/714905875 >> 172.21.15.29:0/714905875 conn(0x7f66200ce6a0 msgr2=0x7f66200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 -- 172.21.15.29:0/714905875 shutdown_connections 2023-10-14T17:58:31.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.981+0000 7f6624a85640 1 -- 172.21.15.29:0/714905875 wait complete. 2023-10-14T17:58:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.982+0000 7f6624a85640 1 Processor -- start 2023-10-14T17:58:31.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.983+0000 7f6624a85640 1 -- start start 2023-10-14T17:58:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.983+0000 7f6624a85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.983+0000 7f6624a85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 0x7f66200af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6624a85640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66200cda80 0x7f66200c6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6624a85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f662014f110 con 0x7f662006c040 2023-10-14T17:58:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6624a85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66200afbc0 con 0x7f66200ad970 2023-10-14T17:58:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6624a85640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66200afea0 con 0x7f66200cda80 2023-10-14T17:58:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41262/0 (socket says 172.21.15.29:41262) 2023-10-14T17:58:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 -- 172.21.15.29:0/944941424 learned_addr learned my addr 172.21.15.29:0/944941424 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661ffff640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66200cda80 0x7f66200c6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 -- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66200cda80 msgr2=0x7f66200c6c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66200cda80 0x7f66200c6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 -- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 msgr2=0x7f66200af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6617fff640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 0x7f66200af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 0x7f66200af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f661f7fe640 1 -- 172.21.15.29:0/944941424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66100094d0 con 0x7f662006c040 2023-10-14T17:58:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.984+0000 7f6617fff640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 0x7f66200af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f661f7fe640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f6610000c00 tx=0x7f6610004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610039d40 con 0x7f662006c040 2023-10-14T17:58:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6610043400 con 0x7f662006c040 2023-10-14T17:58:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66200b0180 con 0x7f662006c040 2023-10-14T17:58:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6610039ea0 con 0x7f662006c040 2023-10-14T17:58:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.985+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66200b0740 con 0x7f662006c040 2023-10-14T17:58:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.987+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662006b330 con 0x7f662006c040 2023-10-14T17:58:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.987+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f66100437b0 con 0x7f662006c040 2023-10-14T17:58:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.988+0000 7f661cff9640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 0x7f660006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.988+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f661003e070 con 0x7f662006c040 2023-10-14T17:58:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.989+0000 7f6617fff640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 0x7f660006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.989+0000 7f6617fff640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 0x7f660006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f66200af030 tx=0x7f6608009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:31.993+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6610077a70 con 0x7f662006c040 2023-10-14T17:58:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.230+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66200b0930 con 0x7f662006c040 2023-10-14T17:58:32.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.231+0000 7f661cff9640 1 -- 172.21.15.29:0/944941424 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f661007b090 con 0x7f662006c040 2023-10-14T17:58:32.232 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T17:58:32.232 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T17:58:32.233 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T17:58:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 msgr2=0x7f660006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 0x7f660006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f66200af030 tx=0x7f6608009040 comp rx=0 tx=0).stop 2023-10-14T17:58:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 msgr2=0x7f662014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 secure :-1 s=READY pgs=2178 cs=0 l=1 rev1=1 crypto rx=0x7f6610000c00 tx=0x7f6610004730 comp rx=0 tx=0).stop 2023-10-14T17:58:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 shutdown_connections 2023-10-14T17:58:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66200cda80 0x7f66200c6c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f660006af90 0x7f660006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f66200ad970 0x7f66200af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 --2- 172.21.15.29:0/944941424 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f662006c040 0x7f662014ebd0 unknown :-1 s=CLOSED pgs=2178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.233+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 >> 172.21.15.29:0/944941424 conn(0x7f66200ce6a0 msgr2=0x7f66200b3890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.234+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 shutdown_connections 2023-10-14T17:58:32.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.234+0000 7f6624a85640 1 -- 172.21.15.29:0/944941424 wait complete. 2023-10-14T17:58:32.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 91, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T17:58:25.535658+0000', 'modified': '2023-10-14T17:58:28.638035+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 10128}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_10128': {'gid': 10128, 'name': 'c', 'rank': 0, 'incarnation': 88, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/2108287962', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 2108287962}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 2108287962}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [23], 'metadata_pool': 22, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T17:58:32.656 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T17:58:32.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T17:58:32.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.897+0000 7f0c5a7bf640 1 Processor -- start 2023-10-14T17:58:32.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.897+0000 7f0c5a7bf640 1 -- start start 2023-10-14T17:58:32.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.898+0000 7f0c5a7bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c540cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c5a7bf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c5415d150 con 0x7f0c540d0be0 2023-10-14T17:58:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c5a7bf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c540d1180 con 0x7f0c5415cd70 2023-10-14T17:58:32.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c5a7bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540d1460 con 0x7f0c540b8480 2023-10-14T17:58:32.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c59fbe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0c540d0be0 0x7f0c540b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45488/0 (socket says 172.21.15.29:45488) 2023-10-14T17:58:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3525356453 learned_addr learned my addr 172.21.15.29:0/3525356453 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.899+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c540cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484267796 0 0) 0x7f0c5415d150 con 0x7f0c540d0be0 2023-10-14T17:58:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c3c003520 con 0x7f0c540d0be0 2023-10-14T17:58:32.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2952475278 0 0) 0x7f0c540d1180 con 0x7f0c5415cd70 2023-10-14T17:58:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c5415d150 con 0x7f0c5415cd70 2023-10-14T17:58:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3523398201 0 0) 0x7f0c3c003520 con 0x7f0c540d0be0 2023-10-14T17:58:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 msgr2=0x7f0c540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c540cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 >> v1:172.21.15.186:6789/0 conn(0x7f0c5415cd70 legacy=0x7f0c540d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c540d1740 con 0x7f0c540d0be0 2023-10-14T17:58:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0c4c001670 con 0x7f0c540d0be0 2023-10-14T17:58:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 >> v1:172.21.15.29:6789/0 conn(0x7f0c540d0be0 legacy=0x7f0c540b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c3c003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c3c0040e0 0x7f0c3c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c3c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c540d1460 con 0x7f0c3c004d00 2023-10-14T17:58:32.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c00bac0 con 0x7f0c3c0040e0 2023-10-14T17:58:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.900+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c3c00bc80 con 0x7f0c540b8480 2023-10-14T17:58:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c3c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c3c0040e0 0x7f0c3c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c3c003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 msgr2=0x7f0c3c003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540b8480 0x7f0c3c003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c3c0040e0 msgr2=0x7f0c3c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c3c0040e0 0x7f0c3c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.901+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3525356453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c3c003520 con 0x7f0c3c004d00 2023-10-14T17:58:32.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.902+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c3c005180 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007f90 tx=0x7f0c3c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4405c0141af50e27 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.902+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c005ad0 con 0x7f0c3c004d00 2023-10-14T17:58:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c005c30 con 0x7f0c3c004d00 2023-10-14T17:58:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c53fff640 1 -- 172.21.15.29:0/3525356453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c007290 con 0x7f0c3c004d00 2023-10-14T17:58:32.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3525356453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 msgr2=0x7f0c3c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3525356453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c3c005180 secure :-1 s=READY pgs=2180 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c007f90 tx=0x7f0c3c0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3525356453 reap_dead start 2023-10-14T17:58:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3525356453 shutdown_connections 2023-10-14T17:58:32.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3525356453 >> 172.21.15.29:0/3525356453 conn(0x7f0c540cf570 msgr2=0x7f0c540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3525356453 shutdown_connections 2023-10-14T17:58:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.903+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3525356453 wait complete. 2023-10-14T17:58:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.905+0000 7f0c5a7bf640 1 Processor -- start 2023-10-14T17:58:32.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.905+0000 7f0c5a7bf640 1 -- start start 2023-10-14T17:58:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540cc8e0 0x7f0c54157c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c5415d330 0x7f0c5415d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c54156c60 con 0x7f0c3c004d00 2023-10-14T17:58:32.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c5415dd00 con 0x7f0c5415d330 2023-10-14T17:58:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c5a7bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c541584d0 con 0x7f0c540cc8e0 2023-10-14T17:58:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.906+0000 7f0c597bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41284/0 (socket says 172.21.15.29:41284) 2023-10-14T17:58:32.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 -- 172.21.15.29:0/3219041970 learned_addr learned my addr 172.21.15.29:0/3219041970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 -- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540cc8e0 msgr2=0x7f0c54157c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c5415d330 0x7f0c5415d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540cc8e0 0x7f0c54157c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540cc8e0 0x7f0c54157c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 -- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c5415d330 msgr2=0x7f0c5415d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c5415d330 0x7f0c5415d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 -- 172.21.15.29:0/3219041970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c541587b0 con 0x7f0c3c004d00 2023-10-14T17:58:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.907+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400eaf0 tx=0x7f0c440020b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.908+0000 7f0c51ffb640 1 -- 172.21.15.29:0/3219041970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c44003b80 con 0x7f0c3c004d00 2023-10-14T17:58:32.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.908+0000 7f0c51ffb640 1 -- 172.21.15.29:0/3219041970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c44003ce0 con 0x7f0c3c004d00 2023-10-14T17:58:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.908+0000 7f0c51ffb640 1 -- 172.21.15.29:0/3219041970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4400c980 con 0x7f0c3c004d00 2023-10-14T17:58:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.908+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c54158a90 con 0x7f0c3c004d00 2023-10-14T17:58:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.908+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c541581d0 con 0x7f0c3c004d00 2023-10-14T17:58:32.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.910+0000 7f0c51ffb640 1 -- 172.21.15.29:0/3219041970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0c4400cae0 con 0x7f0c3c004d00 2023-10-14T17:58:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.911+0000 7f0c51ffb640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 0x7f0c4006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.911+0000 7f0c51ffb640 1 -- 172.21.15.29:0/3219041970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0c44089b20 con 0x7f0c3c004d00 2023-10-14T17:58:32.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.911+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 0x7f0c4006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.912+0000 7f0c5e5ba740 1 Processor -- start 2023-10-14T17:58:32.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.912+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 0x7f0c4006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009950 tx=0x7f0c48008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.912+0000 7f0c5e5ba740 1 -- start start 2023-10-14T17:58:32.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a23070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50550 0x55d052a52f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0529437a0 con 0x55d052a235b0 2023-10-14T17:58:32.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d052995740 con 0x55d052a50550 2023-10-14T17:58:32.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d052992e30 con 0x55d052a50170 2023-10-14T17:58:32.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41288/0 (socket says 172.21.15.29:41288) 2023-10-14T17:58:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.913+0000 7f0c59fbe640 1 -- 172.21.15.29:0/51185524 learned_addr learned my addr 172.21.15.29:0/51185524 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c597bd640 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a23070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50550 0x55d052a52f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 -- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 msgr2=0x55d052a23070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a23070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 -- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50550 msgr2=0x55d052a52f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50550 0x55d052a52f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 -- 172.21.15.29:0/51185524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d052952910 con 0x55d052a235b0 2023-10-14T17:58:32.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.914+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c1d0 tx=0x7f0c4c007000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c179b3b7b428307d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c3affd640 1 -- 172.21.15.29:0/51185524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c039070 con 0x55d052a235b0 2023-10-14T17:58:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c3affd640 1 -- 172.21.15.29:0/51185524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c001670 con 0x55d052a235b0 2023-10-14T17:58:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c3affd640 1 -- 172.21.15.29:0/51185524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c03c040 con 0x55d052a235b0 2023-10-14T17:58:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/51185524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 msgr2=0x55d052a53440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 secure :-1 s=READY pgs=2182 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c00c1d0 tx=0x7f0c4c007000 comp rx=0 tx=0).stop 2023-10-14T17:58:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/51185524 shutdown_connections 2023-10-14T17:58:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a235b0 0x55d052a53440 unknown :-1 s=CLOSED pgs=2182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50550 0x55d052a52f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/51185524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a23070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/51185524 >> 172.21.15.29:0/51185524 conn(0x55d052a25380 msgr2=0x55d052a6df00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/51185524 shutdown_connections 2023-10-14T17:58:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.915+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/51185524 wait complete. 2023-10-14T17:58:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.916+0000 7f0c5e5ba740 1 -- start start 2023-10-14T17:58:32.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.916+0000 7f0c5e5ba740 1 Processor -- start 2023-10-14T17:58:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a3de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a41080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05298e720 con 0x55d052a71410 2023-10-14T17:58:32.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d05292c970 con 0x55d052a50170 2023-10-14T17:58:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d05292c610 con 0x55d052a50550 2023-10-14T17:58:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.917+0000 7f0c5e5ba740 10 client.0 _ll_register_callbacks cb 0x55d05294e9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:58:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41292/0 (socket says 172.21.15.29:41292) 2023-10-14T17:58:32.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3383822191 learned_addr learned my addr 172.21.15.29:0/3383822191 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:32.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a3de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 msgr2=0x55d052a41080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a41080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 msgr2=0x55d052a3de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a3de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3383822191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c3c003520 con 0x55d052a71410 2023-10-14T17:58:32.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c001eb0 tx=0x7f0c4c02e070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88eaebf74aa9f262 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.918+0000 7f0c2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:58:32.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c039030 con 0x55d052a71410 2023-10-14T17:58:32.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55d0529b5160 con 0x55d052a71410 2023-10-14T17:58:32.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0c4c001470 con 0x55d052a71410 2023-10-14T17:58:32.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c4c007280 con 0x55d052a71410 2023-10-14T17:58:32.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c5e5ba740 10 client.10260 fetch_fsmap learned FSMap version 91 2023-10-14T17:58:32.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x55d05298cda0 con 0x55d052a71410 2023-10-14T17:58:32.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.919+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55d0529b5160 con 0x55d052a71410 2023-10-14T17:58:32.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 91) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c4c03f070 con 0x55d052a71410 2023-10-14T17:58:32.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 10 client.10260 fetch_fsmap finished waiting for FSMap version 91 2023-10-14T17:58:32.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 10 client.10260 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:58:32.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 20 client.10260 populate_metadata read hostname 'smithi029' 2023-10-14T17:58:32.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a558e0 0x55d052a3f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 4 client.10260 mds_command: new command op to 10128 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T17:58:32.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 --> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d052960cc0 con 0x55d052a558e0 2023-10-14T17:58:32.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.920+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a558e0 0x55d052a3f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.921+0000 7f0c2ffff640 10 client.10260 ms_handle_connect on v2:172.21.15.29:6836/2108287962 2023-10-14T17:58:32.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.921+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a558e0 0x55d052a3f480 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61b946080e6602cc server_cookie=3ac04eeae3f9d07e in_seq=0 out_seq=0 2023-10-14T17:58:32.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.921+0000 7f0c2ffff640 1 -- 172.21.15.29:0/3383822191 <== mds.0 v2:172.21.15.29:6836/2108287962 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55d052960cc0 con 0x55d052a558e0 2023-10-14T17:58:32.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.921+0000 7f0c2ffff640 10 client.10260 handle_command_reply: tid=0 2023-10-14T17:58:32.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.921+0000 7f0c5e5ba740 1 client.10260 shutdown 2023-10-14T17:58:32.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 msgr2=0x55d052a71850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 secure :-1 s=READY pgs=2183 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c001eb0 tx=0x7f0c4c02e070 comp rx=0 tx=0).stop 2023-10-14T17:58:32.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 shutdown_connections 2023-10-14T17:58:32.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a71850 unknown :-1 s=CLOSED pgs=2183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a558e0 0x55d052a3f480 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a41080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3383822191 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a3de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c2ffff640 0 client.10260 ms_handle_reset on v2:172.21.15.29:6836/2108287962 2023-10-14T17:58:32.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 >> 172.21.15.29:0/3383822191 conn(0x55d052a235b0 msgr2=0x55d052a26f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 shutdown_connections 2023-10-14T17:58:32.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3383822191 wait complete. 2023-10-14T17:58:32.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.922+0000 7f0c5e5ba740 20 client.10260 trim_cache size 0 max 16384 2023-10-14T17:58:32.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.939+0000 7f0c5e5ba740 1 Processor -- start 2023-10-14T17:58:32.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.939+0000 7f0c5e5ba740 1 -- start start 2023-10-14T17:58:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a530e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a681f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d0526f2f20 con 0x55d052a71410 2023-10-14T17:58:32.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d052a812c0 con 0x55d052a50170 2023-10-14T17:58:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d052a806d0 con 0x55d052a50550 2023-10-14T17:58:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c59fbe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41306/0 (socket says 172.21.15.29:41306) 2023-10-14T17:58:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.940+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3418839094 learned_addr learned my addr 172.21.15.29:0/3418839094 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:32.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c597bd640 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a530e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a681f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 msgr2=0x55d052a681f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a681f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 msgr2=0x55d052a530e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a530e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 -- 172.21.15.29:0/3418839094 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d05292bb80 con 0x55d052a71410 2023-10-14T17:58:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c59fbe640 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c001400 tx=0x7f0c4c047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7669468f198b540a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c3b7fe640 1 -- 172.21.15.29:0/3418839094 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c007280 con 0x55d052a71410 2023-10-14T17:58:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c3b7fe640 1 -- 172.21.15.29:0/3418839094 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c007580 con 0x55d052a71410 2023-10-14T17:58:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.941+0000 7f0c3b7fe640 1 -- 172.21.15.29:0/3418839094 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c4c02c7c0 con 0x55d052a71410 2023-10-14T17:58:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3418839094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 msgr2=0x55d052a68730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 secure :-1 s=READY pgs=2184 cs=0 l=1 rev1=1 crypto rx=0x7f0c4c001400 tx=0x7f0c4c047000 comp rx=0 tx=0).stop 2023-10-14T17:58:32.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3418839094 shutdown_connections 2023-10-14T17:58:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a71410 0x55d052a68730 unknown :-1 s=CLOSED pgs=2184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50550 0x55d052a681f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/3418839094 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a50170 0x55d052a530e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3418839094 >> 172.21.15.29:0/3418839094 conn(0x55d052a235b0 msgr2=0x55d052a717f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3418839094 shutdown_connections 2023-10-14T17:58:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.942+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/3418839094 wait complete. 2023-10-14T17:58:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.943+0000 7f0c5e5ba740 1 -- start start 2023-10-14T17:58:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.943+0000 7f0c5e5ba740 1 Processor -- start 2023-10-14T17:58:32.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a6f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a71410 0x55d052a68160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d052a29080 con 0x55d052a50550 2023-10-14T17:58:32.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d052a22b50 con 0x55d052a71410 2023-10-14T17:58:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55d052a87450 con 0x55d052a50170 2023-10-14T17:58:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c5e5ba740 10 client.0 _ll_register_callbacks cb 0x7f0c3c0036b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T17:58:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c58fbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c58fbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41322/0 (socket says 172.21.15.29:41322) 2023-10-14T17:58:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.944+0000 7f0c58fbc640 1 -- 172.21.15.29:0/188080105 learned_addr learned my addr 172.21.15.29:0/188080105 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c597bd640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a6f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 -- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 msgr2=0x55d052a6f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a6f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 -- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a71410 msgr2=0x55d052a68160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a71410 0x55d052a68160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 -- 172.21.15.29:0/188080105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d05298cda0 con 0x55d052a50550 2023-10-14T17:58:32.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c597bd640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a6f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c58fbc640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0c480156d0 tx=0x7f0c48009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfdf02fd9501a96d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c2ffff640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T17:58:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c48014740 con 0x55d052a50550 2023-10-14T17:58:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0c480148a0 con 0x55d052a50550 2023-10-14T17:58:32.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.945+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c4c001670 con 0x55d052a50550 2023-10-14T17:58:32.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c48014a60 con 0x55d052a50550 2023-10-14T17:58:32.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 10 client.10266 fetch_fsmap learned FSMap version 91 2023-10-14T17:58:32.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=91,monmap=2+}) v3 -- 0x55d052951170 con 0x55d052a50550 2023-10-14T17:58:32.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4c001670 con 0x55d052a50550 2023-10-14T17:58:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 91) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f0c48014bc0 con 0x55d052a50550 2023-10-14T17:58:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 10 client.10266 fetch_fsmap finished waiting for FSMap version 91 2023-10-14T17:58:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 10 client.10266 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T17:58:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 20 client.10266 populate_metadata read hostname 'smithi029' 2023-10-14T17:58:32.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a686a0 0x55d052a3c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 4 client.10266 mds_command: new command op to 10128 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T17:58:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 --> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55d052987700 con 0x55d052a686a0 2023-10-14T17:58:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.946+0000 7f0c597bd640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a686a0 0x55d052a3c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c2ffff640 10 client.10266 ms_handle_connect on v2:172.21.15.29:6836/2108287962 2023-10-14T17:58:32.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c597bd640 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a686a0 0x55d052a3c9e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3739f5109f290437 server_cookie=9921ff7f299481fa in_seq=0 out_seq=0 2023-10-14T17:58:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c2ffff640 1 -- 172.21.15.29:0/188080105 <== mds.0 v2:172.21.15.29:6836/2108287962 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+320 (crc 0 0 0) 0x55d052987700 con 0x55d052a686a0 2023-10-14T17:58:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c2ffff640 10 client.10266 handle_command_reply: tid=0 2023-10-14T17:58:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 client.10266 shutdown 2023-10-14T17:58:32.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 msgr2=0x55d052a220f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 secure :-1 s=READY pgs=2185 cs=0 l=1 rev1=1 crypto rx=0x7f0c480156d0 tx=0x7f0c48009190 comp rx=0 tx=0).stop 2023-10-14T17:58:32.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 shutdown_connections 2023-10-14T17:58:32.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55d052a71410 0x55d052a68160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55d052a50550 0x55d052a220f0 unknown :-1 s=CLOSED pgs=2185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.29:6836/2108287962,v1:172.21.15.29:6837/2108287962] conn(0x55d052a686a0 0x55d052a3c9e0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 --2- 172.21.15.29:0/188080105 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55d052a50170 0x55d052a6f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 >> 172.21.15.29:0/188080105 conn(0x55d052a235b0 msgr2=0x55d052a53390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.947+0000 7f0c2ffff640 0 client.10266 ms_handle_reset on v2:172.21.15.29:6836/2108287962 2023-10-14T17:58:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 shutdown_connections 2023-10-14T17:58:32.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5e5ba740 1 -- 172.21.15.29:0/188080105 wait complete. 2023-10-14T17:58:32.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5e5ba740 20 client.10266 trim_cache size 0 max 16384 2023-10-14T17:58:32.973 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T17:58:32.974 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T17:58:32.974 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T17:58:32.974 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 10128, 2023-10-14T17:58:32.974 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T17:58:32.975 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T17:58:32.975 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T17:58:32.975 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.84572615, 2023-10-14T17:58:32.975 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 91, 2023-10-14T17:58:32.975 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 192, 2023-10-14T17:58:32.976 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 192, 2023-10-14T17:58:32.976 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.191861048 2023-10-14T17:58:32.976 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T17:58:32.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 msgr2=0x7f0c4006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 0x7f0c4006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0c48009950 tx=0x7f0c48008040 comp rx=0 tx=0).stop 2023-10-14T17:58:32.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 msgr2=0x7f0c540cbba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:32.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.948+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 secure :-1 s=READY pgs=2181 cs=0 l=1 rev1=1 crypto rx=0x7f0c4400eaf0 tx=0x7f0c440020b0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 shutdown_connections 2023-10-14T17:58:32.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f0c4006af90 0x7f0c4006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0c5415d330 0x7f0c5415d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0c540cc8e0 0x7f0c54157c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 --2- 172.21.15.29:0/3219041970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0c3c004d00 0x7f0c540cbba0 unknown :-1 s=CLOSED pgs=2181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 >> 172.21.15.29:0/3219041970 conn(0x7f0c540cf570 msgr2=0x7f0c540cfb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 shutdown_connections 2023-10-14T17:58:32.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:32.949+0000 7f0c5a7bf640 1 -- 172.21.15.29:0/3219041970 wait complete. 2023-10-14T17:58:32.981 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T17:58:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.218+0000 7f87f116c640 1 Processor -- start 2023-10-14T17:58:33.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.218+0000 7f87f116c640 1 -- start start 2023-10-14T17:58:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87f116c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87ec15ae80 0x7f87ec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87f116c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec15ed60 con 0x7f87ec15aaa0 2023-10-14T17:58:33.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87f116c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87ec0707e0 con 0x7f87ec0ce390 2023-10-14T17:58:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87f116c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87ec06b810 con 0x7f87ec15ae80 2023-10-14T17:58:33.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87e3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87ec15ae80 0x7f87ec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:33.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87ead9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f87ec15aaa0 0x7f87ec0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45492/0 (socket says 172.21.15.29:45492) 2023-10-14T17:58:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87ead9d640 1 -- 172.21.15.29:0/197372013 learned_addr learned my addr 172.21.15.29:0/197372013 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:33.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.220+0000 7f87e3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f87ec15ae80 0x7f87ec15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41334/0 (socket says 172.21.15.29:41334) 2023-10-14T17:58:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.221+0000 7f87e3fff640 1 -- 172.21.15.29:0/197372013 >> v1:172.21.15.186:6789/0 conn(0x7f87ec0ce390 legacy=0x7f87ec070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.221+0000 7f87e3fff640 1 -- 172.21.15.29:0/197372013 >> v1:172.21.15.29:6789/0 conn(0x7f87ec15aaa0 legacy=0x7f87ec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.221+0000 7f87e3fff640 1 -- 172.21.15.29:0/197372013 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87ec15ef90 con 0x7f87ec15ae80 2023-10-14T17:58:33.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.222+0000 7f87e3fff640 1 --2- 172.21.15.29:0/197372013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec15e820 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f87d40097a0 tx=0x7f87d402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8f1ae670220ab7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.222+0000 7f87ea59c640 1 -- 172.21.15.29:0/197372013 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4031830 con 0x7f87ec15ae80 2023-10-14T17:58:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.222+0000 7f87ea59c640 1 -- 172.21.15.29:0/197372013 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87d4031990 con 0x7f87ec15ae80 2023-10-14T17:58:33.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.222+0000 7f87ea59c640 1 -- 172.21.15.29:0/197372013 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87d4031d20 con 0x7f87ec15ae80 2023-10-14T17:58:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 -- 172.21.15.29:0/197372013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 msgr2=0x7f87ec15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 --2- 172.21.15.29:0/197372013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec15e820 secure :-1 s=READY pgs=2187 cs=0 l=1 rev1=1 crypto rx=0x7f87d40097a0 tx=0x7f87d402b8d0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 -- 172.21.15.29:0/197372013 shutdown_connections 2023-10-14T17:58:33.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 --2- 172.21.15.29:0/197372013 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec15e820 unknown :-1 s=CLOSED pgs=2187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 -- 172.21.15.29:0/197372013 >> 172.21.15.29:0/197372013 conn(0x7f87ec0cb410 msgr2=0x7f87ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 -- 172.21.15.29:0/197372013 shutdown_connections 2023-10-14T17:58:33.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.223+0000 7f87f116c640 1 -- 172.21.15.29:0/197372013 wait complete. 2023-10-14T17:58:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.225+0000 7f87f116c640 1 Processor -- start 2023-10-14T17:58:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.225+0000 7f87f116c640 1 -- start start 2023-10-14T17:58:33.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 0x7f87ec0c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87ec15aaa0 0x7f87ec0c9d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:33.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c4e80 con 0x7f87ec15ae80 2023-10-14T17:58:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c38a0 con 0x7f87ec15aaa0 2023-10-14T17:58:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87f116c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87ec0c3b80 con 0x7f87ec0ce390 2023-10-14T17:58:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:33.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87eb59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41342/0 (socket says 172.21.15.29:41342) 2023-10-14T17:58:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.226+0000 7f87eb59e640 1 -- 172.21.15.29:0/4217521827 learned_addr learned my addr 172.21.15.29:0/4217521827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87e3fff640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87ec15aaa0 0x7f87ec0c9d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87ead9d640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 0x7f87ec0c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:33.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 -- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 msgr2=0x7f87ec0c4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 0x7f87ec0c4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 -- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87ec15aaa0 msgr2=0x7f87ec0c9d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87ec15aaa0 0x7f87ec0c9d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 -- 172.21.15.29:0/4217521827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d4009400 con 0x7f87ec15ae80 2023-10-14T17:58:33.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87ead9d640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 0x7f87ec0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87eb59e640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f87e400c780 tx=0x7f87e400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.227+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e4004b60 con 0x7f87ec15ae80 2023-10-14T17:58:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.228+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87e400ce90 con 0x7f87ec15ae80 2023-10-14T17:58:33.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.228+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87e400a850 con 0x7f87ec15ae80 2023-10-14T17:58:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.228+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87ec0c3e60 con 0x7f87ec15ae80 2023-10-14T17:58:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.228+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87ec0c4420 con 0x7f87ec15ae80 2023-10-14T17:58:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.229+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87ec0c4610 con 0x7f87ec15ae80 2023-10-14T17:58:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.230+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f87e400a9b0 con 0x7f87ec15ae80 2023-10-14T17:58:33.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.231+0000 7f87e37fe640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 0x7f87cc06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.231+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f87e40898b0 con 0x7f87ec15ae80 2023-10-14T17:58:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.231+0000 7f87ead9d640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 0x7f87cc06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:33.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.232+0000 7f87ead9d640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 0x7f87cc06d4a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005880 tx=0x7f87dc009040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:33.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.236+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f87e40548b0 con 0x7f87ec15ae80 2023-10-14T17:58:33.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.473+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f87ec04e160 con 0x7f87ec15ae80 2023-10-14T17:58:33.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.474+0000 7f87e37fe640 1 -- 172.21.15.29:0/4217521827 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v91) v1 ==== 53+0+73 (secure 0 0 0) 0x7f87e4057ed0 con 0x7f87ec15ae80 2023-10-14T17:58:33.475 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T17:58:33.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 msgr2=0x7f87cc06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 0x7f87cc06d4a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f87dc005880 tx=0x7f87dc009040 comp rx=0 tx=0).stop 2023-10-14T17:58:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 msgr2=0x7f87ec1f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 secure :-1 s=READY pgs=2188 cs=0 l=1 rev1=1 crypto rx=0x7f87e400c780 tx=0x7f87e400cc40 comp rx=0 tx=0).stop 2023-10-14T17:58:33.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 shutdown_connections 2023-10-14T17:58:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f87ec15ae80 0x7f87ec1f3760 unknown :-1 s=CLOSED pgs=2188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f87cc06afe0 0x7f87cc06d4a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f87ec15aaa0 0x7f87ec0c9d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 --2- 172.21.15.29:0/4217521827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f87ec0ce390 0x7f87ec0c4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 >> 172.21.15.29:0/4217521827 conn(0x7f87ec0cb410 msgr2=0x7f87ec0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 shutdown_connections 2023-10-14T17:58:33.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:33.477+0000 7f87f116c640 1 -- 172.21.15.29:0/4217521827 wait complete. 2023-10-14T17:58:33.497 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T17:58:33.497 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T17:58:33.498 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T17:58:33.498 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T17:58:33.499 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T17:58:33.499 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T17:58:33.499 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T17:58:33.500 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T17:58:33.500 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:58:33.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.501 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T17:58:33.555 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T17:58:33.555 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T17:58:33.555 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T17:58:33.555 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T17:58:33.556 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:58:33.557 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T17:58:33.557 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T17:58:33.557 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T17:58:33.557 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T17:58:33.557 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T17:58:33.558 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T17:58:33.558 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 37663sec preferred_lft 37663sec 2023-10-14T17:58:33.558 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T17:58:33.558 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.559 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T17:58:33.559 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:58:33.559 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T17:58:33.559 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T17:58:33.560 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T17:58:33.560 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T17:58:33.560 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.560 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T17:58:33.560 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.561 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:58:33.561 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T17:58:33.561 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T17:58:33.561 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.561 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T17:58:33.562 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T17:58:33.562 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T17:58:33.562 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T17:58:33.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.564 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T17:58:33.666 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T17:58:33.670 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.671 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T17:58:33.733 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T17:58:33.734 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T17:58:33.734 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T17:58:33.734 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T17:58:33.735 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T17:58:33.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.736 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T17:58:33.736 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T17:58:33.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T17:58:33.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T17:58:33.737 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T17:58:33.738 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T17:58:33.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.938 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T17:58:33.995 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T17:58:33.995 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T17:58:33.997 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:58:33.997 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:33.997 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T17:58:34.101 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:58:34.102 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T17:58:34.163 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T17:58:34.163 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T17:58:34.164 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:58:34.219 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T17:58:34.219 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:34.220 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:58:34.366 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:58:34.367 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:58:34.367 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:58:34.390 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:58:34.390 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:58:34.390 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:58:34.390 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:58:34.391 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:58:34.416 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T17:58:34.416 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T17:58:34.417 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T17:58:34.417 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T17:58:34.417 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T17:58:34.417 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T17:58:34.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T17:58:34.418 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T17:58:34.419 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T17:58:34.419 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T17:58:34.420 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T17:58:34.420 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T17:58:34.420 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T17:58:34.423 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T17:58:34.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:34.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T17:58:34.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:34.513 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T17:58:34.584 INFO:teuthology.orchestra.run.smithi029.stdout:client10278 2023-10-14T17:58:34.585 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T17:58:34.585 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T17:58:34.646 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T17:58:34.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.875+0000 7f3fe4924640 1 Processor -- start 2023-10-14T17:58:34.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.875+0000 7f3fe4924640 1 -- start start 2023-10-14T17:58:34.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fe4924640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3fe01428c0 0x7f3fe00c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fe4924640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00c9ae0 con 0x7f3fe006b660 2023-10-14T17:58:34.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fe4924640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fe00cc6b0 con 0x7f3fe00c9690 2023-10-14T17:58:34.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fe4924640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fe01599b0 con 0x7f3fe01428c0 2023-10-14T17:58:34.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fde59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3fe006b660 0x7f3fe006ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45502/0 (socket says 172.21.15.29:45502) 2023-10-14T17:58:34.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fde59c640 1 -- 172.21.15.29:0/1349351815 learned_addr learned my addr 172.21.15.29:0/1349351815 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:34.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.877+0000 7f3fd5d9b640 1 --2- 172.21.15.29:0/1349351815 >> v2:172.21.15.186:3300/0 conn(0x7f3fe01428c0 0x7f3fe00c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756723985 0 0) 0x7f3fd0001490 con 0x7f3fe006b660 2023-10-14T17:58:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fcc003520 con 0x7f3fe006b660 2023-10-14T17:58:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540472302 0 0) 0x7f3fcc003520 con 0x7f3fe006b660 2023-10-14T17:58:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 >> v2:172.21.15.186:3300/0 conn(0x7f3fe01428c0 msgr2=0x7f3fe00c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:34.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 --2- 172.21.15.29:0/1349351815 >> v2:172.21.15.186:3300/0 conn(0x7f3fe01428c0 0x7f3fe00c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 >> v1:172.21.15.186:6789/0 conn(0x7f3fe00c9690 legacy=0x7f3fe00ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:34.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe006c290 con 0x7f3fe006b660 2023-10-14T17:58:34.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3fd0001b80 con 0x7f3fe006b660 2023-10-14T17:58:34.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 >> v1:172.21.15.29:6789/0 conn(0x7f3fe006b660 legacy=0x7f3fe006ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:34.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fcc003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fcc0047c0 0x7f3fcc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc0051d0 0x7f3fcc00db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe01599b0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00e330 con 0x7f3fcc0051d0 2023-10-14T17:58:34.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fcc00e4f0 con 0x7f3fcc0047c0 2023-10-14T17:58:34.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.878+0000 7f3fde59c640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fcc003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:34.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fded9d640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc0051d0 0x7f3fcc00db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:34.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fde59c640 1 -- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fcc0047c0 msgr2=0x7f3fcc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:34.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fde59c640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fcc0047c0 0x7f3fcc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fde59c640 1 -- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc0051d0 msgr2=0x7f3fcc00db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:34.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fde59c640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fcc0051d0 0x7f3fcc00db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fded9d640 1 -- 172.21.15.29:0/1349351815 reap_dead start 2023-10-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.879+0000 7f3fde59c640 1 -- 172.21.15.29:0/1349351815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fcc003520 con 0x7f3fcc0039c0 2023-10-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.880+0000 7f3fde59c640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fcc003de0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000a730 tx=0x7f3fcc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8eb06a4705636b6e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.880+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00072d0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.880+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd002f070 con 0x7f3fcc0039c0 2023-10-14T17:58:34.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.880+0000 7f3fddd9b640 1 -- 172.21.15.29:0/1349351815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00076d0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 -- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 msgr2=0x7f3fcc003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:34.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fcc003de0 secure :-1 s=READY pgs=2191 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000a730 tx=0x7f3fcc0057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 -- 172.21.15.29:0/1349351815 shutdown_connections 2023-10-14T17:58:34.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 --2- 172.21.15.29:0/1349351815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fcc003de0 unknown :-1 s=CLOSED pgs=2191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 -- 172.21.15.29:0/1349351815 >> 172.21.15.29:0/1349351815 conn(0x7f3fe00ce9b0 msgr2=0x7f3fe00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:34.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 -- 172.21.15.29:0/1349351815 shutdown_connections 2023-10-14T17:58:34.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.881+0000 7f3fe4924640 1 -- 172.21.15.29:0/1349351815 wait complete. 2023-10-14T17:58:34.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.883+0000 7f3fe4924640 1 Processor -- start 2023-10-14T17:58:34.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.883+0000 7f3fe4924640 1 -- start start 2023-10-14T17:58:34.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fe0157390 0x7f3fe00b2e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fe00c6970 0x7f3fe00b33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00aeea0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c7780 con 0x7f3fe0157390 2023-10-14T17:58:34.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.884+0000 7f3fe4924640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3fe00c6ee0 con 0x7f3fe00c6970 2023-10-14T17:58:34.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:34.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41356/0 (socket says 172.21.15.29:41356) 2023-10-14T17:58:34.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 -- 172.21.15.29:0/2993368731 learned_addr learned my addr 172.21.15.29:0/2993368731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:34.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 -- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fe00c6970 msgr2=0x7f3fe00b33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:34.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fe00c6970 0x7f3fe00b33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 -- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fe0157390 msgr2=0x7f3fe00b2e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:34.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fe0157390 0x7f3fe00b2e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:34.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 -- 172.21.15.29:0/2993368731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fe00c71c0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.885+0000 7f3fde59c640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000a730 tx=0x7f3fd0001e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:34.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.886+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00070d0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.886+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fd0007230 con 0x7f3fcc0039c0 2023-10-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.886+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3fd00075c0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.886+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fe00c74a0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.886+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fe00b15d0 con 0x7f3fcc0039c0 2023-10-14T17:58:34.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.887+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fe004e160 con 0x7f3fcc0039c0 2023-10-14T17:58:34.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.888+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3fd0031030 con 0x7f3fcc0039c0 2023-10-14T17:58:34.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.889+0000 7f3fd77fe640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 0x7f3fc006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:34.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.890+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f3fd00abb40 con 0x7f3fcc0039c0 2023-10-14T17:58:34.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.890+0000 7f3fd5d9b640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 0x7f3fc006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:34.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.890+0000 7f3fd5d9b640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 0x7f3fc006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006e80 tx=0x7f3fc8008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:34.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:34.894+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3fd0076940 con 0x7f3fcc0039c0 2023-10-14T17:58:35.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.116+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 --> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f3fe006c850 con 0x7f3fc006af90 2023-10-14T17:58:35.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.120+0000 7f3fd77fe640 1 -- 172.21.15.29:0/2993368731 <== mgr.9861 v2:172.21.15.29:6832/79844030 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f3fc8009d60 con 0x7f3fc006af90 2023-10-14T17:58:35.121 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T17:58:35.121 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T17:58:35.122 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T17:58:35.122 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T17:58:35.122 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T17:58:35.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 msgr2=0x7f3fc006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 0x7f3fc006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3fc8006e80 tx=0x7f3fc8008040 comp rx=0 tx=0).stop 2023-10-14T17:58:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 msgr2=0x7f3fe00c6430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 secure :-1 s=READY pgs=2192 cs=0 l=1 rev1=1 crypto rx=0x7f3fd000a730 tx=0x7f3fd0001e40 comp rx=0 tx=0).stop 2023-10-14T17:58:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 shutdown_connections 2023-10-14T17:58:35.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3fe00c6970 0x7f3fe00b33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f3fc006af90 0x7f3fc006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3fe0157390 0x7f3fe00b2e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 --2- 172.21.15.29:0/2993368731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3fcc0039c0 0x7f3fe00c6430 unknown :-1 s=CLOSED pgs=2192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 >> 172.21.15.29:0/2993368731 conn(0x7f3fe00ce9b0 msgr2=0x7f3fe00cdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 shutdown_connections 2023-10-14T17:58:35.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.123+0000 7f3fe4924640 1 -- 172.21.15.29:0/2993368731 wait complete. 2023-10-14T17:58:35.144 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T17:58:35.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.369+0000 7f529f59e640 1 Processor -- start 2023-10-14T17:58:35.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.369+0000 7f529f59e640 1 -- start start 2023-10-14T17:58:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52a006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52a00cda80 0x7f52a015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52a006c960 con 0x7f52a00ad970 2023-10-14T17:58:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52a015b9d0 con 0x7f52a00cda80 2023-10-14T17:58:35.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a015bcb0 con 0x7f52a006c040 2023-10-14T17:58:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529dd9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f52a00ad970 0x7f52a015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45514/0 (socket says 172.21.15.29:45514) 2023-10-14T17:58:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.371+0000 7f529dd9b640 1 -- 172.21.15.29:0/3762680453 learned_addr learned my addr 172.21.15.29:0/3762680453 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529ed9d640 1 --2- 172.21.15.29:0/3762680453 >> v2:172.21.15.186:3300/0 conn(0x7f52a00cda80 0x7f52a015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529e59c640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52a006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2069924582 0 0) 0x7f52a006c960 con 0x7f52a00ad970 2023-10-14T17:58:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5288003520 con 0x7f52a00ad970 2023-10-14T17:58:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2574582213 0 0) 0x7f5288003520 con 0x7f52a00ad970 2023-10-14T17:58:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 msgr2=0x7f52a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:35.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52a006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 >> v2:172.21.15.186:3300/0 conn(0x7f52a00cda80 msgr2=0x7f52a015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 --2- 172.21.15.29:0/3762680453 >> v2:172.21.15.186:3300/0 conn(0x7f52a00cda80 0x7f52a015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a015bf90 con 0x7f52a00ad970 2023-10-14T17:58:35.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5294001670 con 0x7f52a00ad970 2023-10-14T17:58:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 >> v1:172.21.15.29:6789/0 conn(0x7f52a00ad970 legacy=0x7f52a015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288003970 0x7f5288003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52880047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5288004d00 0x7f5288005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a015b9d0 con 0x7f5288004d00 2023-10-14T17:58:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a015bcb0 con 0x7f5288003970 2023-10-14T17:58:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.372+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f528800b980 con 0x7f52a006c040 2023-10-14T17:58:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5288004d00 0x7f5288005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529dd9b640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52880047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529e59c640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288003970 0x7f5288003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 -- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 msgr2=0x7f52880047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52a006c040 0x7f52880047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 -- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288003970 msgr2=0x7f5288003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5288003970 0x7f5288003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.373+0000 7f529ed9d640 1 -- 172.21.15.29:0/3762680453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5288003520 con 0x7f5288004d00 2023-10-14T17:58:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.374+0000 7f529ed9d640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5288004d00 0x7f5288005180 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fc0 tx=0x7f52880057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e87b1e21b4059ac9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:35.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f529802cc60 con 0x7f5288004d00 2023-10-14T17:58:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f529800f040 con 0x7f5288004d00 2023-10-14T17:58:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529d59a640 1 -- 172.21.15.29:0/3762680453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298034440 con 0x7f5288004d00 2023-10-14T17:58:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 -- 172.21.15.29:0/3762680453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5288004d00 msgr2=0x7f5288005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 --2- 172.21.15.29:0/3762680453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5288004d00 0x7f5288005180 secure :-1 s=READY pgs=2194 cs=0 l=1 rev1=1 crypto rx=0x7f5298009fc0 tx=0x7f52880057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529ed9d640 1 -- 172.21.15.29:0/3762680453 reap_dead start 2023-10-14T17:58:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 -- 172.21.15.29:0/3762680453 shutdown_connections 2023-10-14T17:58:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 -- 172.21.15.29:0/3762680453 >> 172.21.15.29:0/3762680453 conn(0x7f52a00ce6a0 msgr2=0x7f52a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:35.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 -- 172.21.15.29:0/3762680453 shutdown_connections 2023-10-14T17:58:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.375+0000 7f529f59e640 1 -- 172.21.15.29:0/3762680453 wait complete. 2023-10-14T17:58:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.377+0000 7f529f59e640 1 Processor -- start 2023-10-14T17:58:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.377+0000 7f529f59e640 1 -- start start 2023-10-14T17:58:35.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 0x7f52a014fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52a00c7940 0x7f52a00c7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0157c60 con 0x7f52a00b11f0 2023-10-14T17:58:35.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52a00b1bb0 con 0x7f52a00c7940 2023-10-14T17:58:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.378+0000 7f529f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52a0156d90 con 0x7f5288004d00 2023-10-14T17:58:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41380/0 (socket says 172.21.15.29:41380) 2023-10-14T17:58:35.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 -- 172.21.15.29:0/4259742747 learned_addr learned my addr 172.21.15.29:0/4259742747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 -- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 msgr2=0x7f52a014fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529e59c640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 0x7f52a014fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529dd9b640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52a00c7940 0x7f52a00c7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 0x7f52a014fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 -- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52a00c7940 msgr2=0x7f52a00c7dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52a00c7940 0x7f52a00c7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 -- 172.21.15.29:0/4259742747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52a0156fe0 con 0x7f52a00b11f0 2023-10-14T17:58:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529e59c640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 0x7f52a014fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:35.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.379+0000 7f529ed9d640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f5298002230 tx=0x7f5298009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.380+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298034b50 con 0x7f52a00b11f0 2023-10-14T17:58:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.380+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5298034cb0 con 0x7f52a00b11f0 2023-10-14T17:58:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.380+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52a01572c0 con 0x7f52a00b11f0 2023-10-14T17:58:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.380+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5298003bb0 con 0x7f52a00b11f0 2023-10-14T17:58:35.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.380+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52a0157880 con 0x7f52a00b11f0 2023-10-14T17:58:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.381+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52a004e9b0 con 0x7f52a00b11f0 2023-10-14T17:58:35.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.382+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 103) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f529803e440 con 0x7f52a00b11f0 2023-10-14T17:58:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.384+0000 7f528effd640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 0x7f528006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.384+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f52980ab7f0 con 0x7f52a00b11f0 2023-10-14T17:58:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.384+0000 7f529e59c640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 0x7f528006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.385+0000 7f529e59c640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 0x7f528006d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5290009980 tx=0x7f5290008040 comp rx=0 tx=0).ready entity=mgr.9861 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.389+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5298076820 con 0x7f52a00b11f0 2023-10-14T17:58:35.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:35.634+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f52a006b2b0 con 0x7f52a00b11f0 2023-10-14T17:58:36.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.195+0000 7f528effd640 1 -- 172.21.15.29:0/4259742747 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v104) v1 ==== 92+0+0 (secure 0 0 0) 0x7f5298079e40 con 0x7f52a00b11f0 2023-10-14T17:58:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.198+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 msgr2=0x7f528006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.198+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 0x7f528006d480 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5290009980 tx=0x7f5290008040 comp rx=0 tx=0).stop 2023-10-14T17:58:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.198+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 msgr2=0x7f52a00b1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.198+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 secure :-1 s=READY pgs=2195 cs=0 l=1 rev1=1 crypto rx=0x7f5298002230 tx=0x7f5298009750 comp rx=0 tx=0).stop 2023-10-14T17:58:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 shutdown_connections 2023-10-14T17:58:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52a00b11f0 0x7f52a00b1670 unknown :-1 s=CLOSED pgs=2195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f528006afc0 0x7f528006d480 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52a00c7940 0x7f52a00c7dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 --2- 172.21.15.29:0/4259742747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5288004d00 0x7f52a014fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 >> 172.21.15.29:0/4259742747 conn(0x7f52a00ce6a0 msgr2=0x7f52a00c5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:36.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 shutdown_connections 2023-10-14T17:58:36.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.199+0000 7f529f59e640 1 -- 172.21.15.29:0/4259742747 wait complete. 2023-10-14T17:58:36.229 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T17:58:36.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.433+0000 7f619610f640 1 Processor -- start 2023-10-14T17:58:36.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.433+0000 7f619610f640 1 -- start start 2023-10-14T17:58:36.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.434+0000 7f619610f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f61900ad350 0x7f6190069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.434+0000 7f619610f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f61900c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.434+0000 7f619610f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619006a730 con 0x7f619015d150 2023-10-14T17:58:36.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.434+0000 7f619610f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619015d530 con 0x7f61900ad350 2023-10-14T17:58:36.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.434+0000 7f619610f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61900ac890 con 0x7f619006a2e0 2023-10-14T17:58:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f619015d150 0x7f6190160b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45532/0 (socket says 172.21.15.29:45532) 2023-10-14T17:58:36.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618f7fe640 1 -- 172.21.15.29:0/2511028021 learned_addr learned my addr 172.21.15.29:0/2511028021 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:36.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618effd640 1 --2- 172.21.15.29:0/2511028021 >> v2:172.21.15.186:3300/0 conn(0x7f61900ad350 0x7f6190069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618ffff640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f61900c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916184577 0 0) 0x7f6180001110 con 0x7f619015d150 2023-10-14T17:58:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.435+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6178003520 con 0x7f619015d150 2023-10-14T17:58:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 54590834 0 0) 0x7f6178003520 con 0x7f619015d150 2023-10-14T17:58:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 msgr2=0x7f61900c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f61900c9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 >> v2:172.21.15.186:3300/0 conn(0x7f61900ad350 msgr2=0x7f6190069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 --2- 172.21.15.29:0/2511028021 >> v2:172.21.15.186:3300/0 conn(0x7f61900ad350 0x7f6190069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61900ad7a0 con 0x7f619015d150 2023-10-14T17:58:36.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6180001820 con 0x7f619015d150 2023-10-14T17:58:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 >> v1:172.21.15.29:6789/0 conn(0x7f619015d150 legacy=0x7f6190160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6178003970 0x7f6178003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61780047c0 0x7f6178004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f6178005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619015d530 con 0x7f6178003970 2023-10-14T17:58:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61900ac890 con 0x7f61780047c0 2023-10-14T17:58:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617800b980 con 0x7f619006a2e0 2023-10-14T17:58:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618f7fe640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6178003970 0x7f6178003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.436+0000 7f618ffff640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f6178005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618effd640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61780047c0 0x7f6178004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618f7fe640 1 -- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 msgr2=0x7f6178005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618f7fe640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f619006a2e0 0x7f6178005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618f7fe640 1 -- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61780047c0 msgr2=0x7f6178004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618f7fe640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61780047c0 0x7f6178004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618f7fe640 1 -- 172.21.15.29:0/2511028021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6178003520 con 0x7f6178003970 2023-10-14T17:58:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.437+0000 7f618effd640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61780047c0 0x7f6178004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.438+0000 7f618f7fe640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6178003970 0x7f6178003d90 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f618000c160 tx=0x7f61780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4afa15cfa5af898 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.438+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180005ad0 con 0x7f6178003970 2023-10-14T17:58:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.438+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6180005c30 con 0x7f6178003970 2023-10-14T17:58:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.438+0000 7f618e7fc640 1 -- 172.21.15.29:0/2511028021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618002c810 con 0x7f6178003970 2023-10-14T17:58:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 -- 172.21.15.29:0/2511028021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6178003970 msgr2=0x7f6178003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 --2- 172.21.15.29:0/2511028021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6178003970 0x7f6178003d90 secure :-1 s=READY pgs=2198 cs=0 l=1 rev1=1 crypto rx=0x7f618000c160 tx=0x7f61780057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f618ffff640 1 -- 172.21.15.29:0/2511028021 reap_dead start 2023-10-14T17:58:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 -- 172.21.15.29:0/2511028021 shutdown_connections 2023-10-14T17:58:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 -- 172.21.15.29:0/2511028021 >> 172.21.15.29:0/2511028021 conn(0x7f61900ce9b0 msgr2=0x7f61900cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 -- 172.21.15.29:0/2511028021 shutdown_connections 2023-10-14T17:58:36.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.439+0000 7f619610f640 1 -- 172.21.15.29:0/2511028021 wait complete. 2023-10-14T17:58:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.441+0000 7f619610f640 1 Processor -- start 2023-10-14T17:58:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.441+0000 7f619610f640 1 -- start start 2023-10-14T17:58:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6178003970 0x7f61901f6b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61900b3950 0x7f61900b3dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61900b4310 con 0x7f61900c7980 2023-10-14T17:58:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901f70a0 con 0x7f61900b3950 2023-10-14T17:58:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f619610f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61901554f0 con 0x7f6178003970 2023-10-14T17:58:36.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f618effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f618effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41424/0 (socket says 172.21.15.29:41424) 2023-10-14T17:58:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f618effd640 1 -- 172.21.15.29:0/178983704 learned_addr learned my addr 172.21.15.29:0/178983704 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f618ffff640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61900b3950 0x7f61900b3dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.442+0000 7f618f7fe640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6178003970 0x7f61901f6b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6178003970 msgr2=0x7f61901f6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6178003970 0x7f61901f6b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61900b3950 msgr2=0x7f61900b3dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61900b3950 0x7f61900b3dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 -- 172.21.15.29:0/178983704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61900c7f40 con 0x7f61900c7980 2023-10-14T17:58:36.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f618effd640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f61840109d0 tx=0x7f6184010e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618400ec20 con 0x7f61900c7980 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618400ed80 con 0x7f61900c7980 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618401c620 con 0x7f61900c7980 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.443+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61900ae2e0 con 0x7f61900c7980 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.444+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61900ae8a0 con 0x7f61900c7980 2023-10-14T17:58:36.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.445+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619004e160 con 0x7f61900c7980 2023-10-14T17:58:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.446+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 104) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f618401c780 con 0x7f61900c7980 2023-10-14T17:58:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.447+0000 7f6177fff640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 0x7f617c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.447+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6184089e90 con 0x7f61900c7980 2023-10-14T17:58:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.447+0000 7f618f7fe640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 msgr2=0x7f617c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:36.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.447+0000 7f618f7fe640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 0x7f617c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:36.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.455+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6184054dc0 con 0x7f61900c7980 2023-10-14T17:58:36.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.648+0000 7f618f7fe640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 msgr2=0x7f617c06d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:36.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.648+0000 7f618f7fe640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 0x7f617c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:36.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.652+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f61900acb20 con 0x7f61900c7980 2023-10-14T17:58:36.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.657+0000 7f6177fff640 1 -- 172.21.15.29:0/178983704 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v65)=0 v65) v1 ==== 138+0+0 (secure 0 0 0) 0x7f61840583e0 con 0x7f61900c7980 2023-10-14T17:58:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 msgr2=0x7f617c06d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 0x7f617c06d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 msgr2=0x7f6190154fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 secure :-1 s=READY pgs=2199 cs=0 l=1 rev1=1 crypto rx=0x7f61840109d0 tx=0x7f6184010e90 comp rx=0 tx=0).stop 2023-10-14T17:58:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 shutdown_connections 2023-10-14T17:58:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f617c06afe0 0x7f617c06d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61900b3950 0x7f61900b3dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61900c7980 0x7f6190154fb0 unknown :-1 s=CLOSED pgs=2199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 --2- 172.21.15.29:0/178983704 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6178003970 0x7f61901f6b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.659+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 >> 172.21.15.29:0/178983704 conn(0x7f61900ce9b0 msgr2=0x7f61900b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:36.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.660+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 shutdown_connections 2023-10-14T17:58:36.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.660+0000 7f619610f640 1 -- 172.21.15.29:0/178983704 wait complete. 2023-10-14T17:58:36.679 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T17:58:36.750 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:36.749+0000 7f94d6030040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:58:36.754 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:36.753+0000 7f4de2e32040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T17:58:36.806 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:36.806+0000 7f94d6030040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:58:36.813 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:36.811+0000 7f4de2e32040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T17:58:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.896+0000 7fcbb4a34640 1 Processor -- start 2023-10-14T17:58:36.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.896+0000 7fcbb4a34640 1 -- start start 2023-10-14T17:58:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbb4a34640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbb00cf710 0x7fcbb00b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbb4a34640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cfaf0 0x7fcbb00b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbb4a34640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbb00d3980 con 0x7fcbb00b38e0 2023-10-14T17:58:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbb4a34640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbb00d3c60 con 0x7fcbb00cf710 2023-10-14T17:58:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbb4a34640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00ce630 con 0x7fcbb00cfaf0 2023-10-14T17:58:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbb00cf710 0x7fcbb00b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcbb00cf710 0x7fcbb00b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41428/0 (socket says 172.21.15.29:41428) 2023-10-14T17:58:36.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 -- 172.21.15.29:0/320027233 learned_addr learned my addr 172.21.15.29:0/320027233 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbadd9b640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cfaf0 0x7fcbb00b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 -- 172.21.15.29:0/320027233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cfaf0 msgr2=0x7fcbb00b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cfaf0 0x7fcbb00b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 -- 172.21.15.29:0/320027233 >> v1:172.21.15.186:6789/0 conn(0x7fcbb00b38e0 legacy=0x7fcbb00d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.898+0000 7fcbae59c640 1 -- 172.21.15.29:0/320027233 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb00ce910 con 0x7fcbb00cf710 2023-10-14T17:58:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbae59c640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00cf710 0x7fcbb00b2d10 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009850 tx=0x7fcba002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e00c2180f8f1f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbad59a640 1 -- 172.21.15.29:0/320027233 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba002e070 con 0x7fcbb00cf710 2023-10-14T17:58:36.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbad59a640 1 -- 172.21.15.29:0/320027233 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0004030 con 0x7fcbb00cf710 2023-10-14T17:58:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbad59a640 1 -- 172.21.15.29:0/320027233 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0004420 con 0x7fcbb00cf710 2023-10-14T17:58:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbb4a34640 1 -- 172.21.15.29:0/320027233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00cf710 msgr2=0x7fcbb00b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.900+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00cf710 0x7fcbb00b2d10 secure :-1 s=READY pgs=2200 cs=0 l=1 rev1=1 crypto rx=0x7fcba0009850 tx=0x7fcba002b840 comp rx=0 tx=0).stop 2023-10-14T17:58:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 -- 172.21.15.29:0/320027233 shutdown_connections 2023-10-14T17:58:36.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cfaf0 0x7fcbb00b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/320027233 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00cf710 0x7fcbb00b2d10 unknown :-1 s=CLOSED pgs=2200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 -- 172.21.15.29:0/320027233 >> 172.21.15.29:0/320027233 conn(0x7fcbb00d2cc0 msgr2=0x7fcbb00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 -- 172.21.15.29:0/320027233 shutdown_connections 2023-10-14T17:58:36.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.901+0000 7fcbb4a34640 1 -- 172.21.15.29:0/320027233 wait complete. 2023-10-14T17:58:36.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.902+0000 7fcbb4a34640 1 Processor -- start 2023-10-14T17:58:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 -- start start 2023-10-14T17:58:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cf710 0x7fcbb00cb170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cfaf0 0x7fcbb00cb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00ca040 con 0x7fcbb00b38e0 2023-10-14T17:58:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00cbce0 con 0x7fcbb00cfaf0 2023-10-14T17:58:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.903+0000 7fcbb4a34640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb00cbfc0 con 0x7fcbb00cf710 2023-10-14T17:58:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41442/0 (socket says 172.21.15.29:41442) 2023-10-14T17:58:36.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 -- 172.21.15.29:0/2927727333 learned_addr learned my addr 172.21.15.29:0/2927727333 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbaed9d640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cfaf0 0x7fcbb00cb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbadd9b640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cf710 0x7fcbb00cb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cf710 msgr2=0x7fcbb00cb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cf710 0x7fcbb00cb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cfaf0 msgr2=0x7fcbb00cb6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cfaf0 0x7fcbb00cb6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.904+0000 7fcbae59c640 1 -- 172.21.15.29:0/2927727333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcba0009460 con 0x7fcbb00b38e0 2023-10-14T17:58:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcbae59c640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcba0035040 tx=0x7fcba00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba00038a0 con 0x7fcbb00b38e0 2023-10-14T17:58:36.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0003a00 con 0x7fcbb00b38e0 2023-10-14T17:58:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb00cc2a0 con 0x7fcbb00b38e0 2023-10-14T17:58:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba0003d70 con 0x7fcbb00b38e0 2023-10-14T17:58:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.905+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb00cc860 con 0x7fcbb00b38e0 2023-10-14T17:58:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.906+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbb004e160 con 0x7fcbb00b38e0 2023-10-14T17:58:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.907+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 104) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fcba0006040 con 0x7fcbb00b38e0 2023-10-14T17:58:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.908+0000 7fcb9effd640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 0x7fcb9006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.908+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fcba00ab2a0 con 0x7fcbb00b38e0 2023-10-14T17:58:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.908+0000 7fcbadd9b640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 msgr2=0x7fcb9006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.908+0000 7fcbadd9b640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 0x7fcb9006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:36.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:36.913+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcba00760b0 con 0x7fcbb00b38e0 2023-10-14T17:58:37.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.106+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fcbb00c9ae0 con 0x7fcbb00b38e0 2023-10-14T17:58:37.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.109+0000 7fcbadd9b640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 msgr2=0x7fcb9006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:37.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.109+0000 7fcbadd9b640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 0x7fcb9006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:37.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.112+0000 7fcb9effd640 1 -- 172.21.15.29:0/2927727333 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v66)=0 v66) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcba00796d0 con 0x7fcbb00b38e0 2023-10-14T17:58:37.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.114+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 msgr2=0x7fcb9006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:37.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.114+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 0x7fcb9006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.114+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 msgr2=0x7fcbb0154250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.114+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 secure :-1 s=READY pgs=2201 cs=0 l=1 rev1=1 crypto rx=0x7fcba0035040 tx=0x7fcba00046c0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 shutdown_connections 2023-10-14T17:58:37.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7fcb9006afe0 0x7fcb9006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbb00cfaf0 0x7fcbb00cb6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbb00cf710 0x7fcbb00cb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 --2- 172.21.15.29:0/2927727333 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbb00b38e0 0x7fcbb0154250 unknown :-1 s=CLOSED pgs=2201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 >> 172.21.15.29:0/2927727333 conn(0x7fcbb00d2cc0 msgr2=0x7fcbb00d14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 shutdown_connections 2023-10-14T17:58:37.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.115+0000 7fcbb4a34640 1 -- 172.21.15.29:0/2927727333 wait complete. 2023-10-14T17:58:37.134 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:37.135 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T17:58:37.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T17:58:37.197 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1/test_restart) 2023-10-14T17:58:37.247 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add /snap_test_dir1/test_restart 1m 2023-10-14T17:58:37.271 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.271+0000 7f94d6030040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:58:37.300 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.299+0000 7f4de2e32040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T17:58:37.324 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.324+0000 7f94d6030040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:58:37.355 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.354+0000 7f4de2e32040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T17:58:37.375 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.375+0000 7f94d6030040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:58:37.409 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.408+0000 7f4de2e32040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T17:58:37.437 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.436+0000 7f94d6030040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:58:37.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.446+0000 7f1203778640 1 Processor -- start 2023-10-14T17:58:37.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.446+0000 7f1203778640 1 -- start start 2023-10-14T17:58:37.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.448+0000 7f1203778640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f11fc160530 0x7f11fc1609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1203778640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0ac890 con 0x7f11fc065ab0 2023-10-14T17:58:37.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1203778640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0ae800 con 0x7f11fc069fa0 2023-10-14T17:58:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1203778640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc0ae1f0 con 0x7f11fc160530 2023-10-14T17:58:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1200d13640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f11fc065ab0 0x7f11fc160f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45536/0 (socket says 172.21.15.29:45536) 2023-10-14T17:58:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1200d13640 1 -- 172.21.15.29:0/3426726467 learned_addr learned my addr 172.21.15.29:0/3426726467 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T17:58:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f1201514640 1 --2- 172.21.15.29:0/3426726467 >> v2:172.21.15.186:3300/0 conn(0x7f11fc160530 0x7f11fc1609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2726602572 0 0) 0x7f11fc0ac890 con 0x7f11fc065ab0 2023-10-14T17:58:37.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.449+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11e8003520 con 0x7f11fc065ab0 2023-10-14T17:58:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3833081362 0 0) 0x7f11e8003520 con 0x7f11fc065ab0 2023-10-14T17:58:37.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 >> v2:172.21.15.186:3300/0 conn(0x7f11fc160530 msgr2=0x7f11fc1609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 --2- 172.21.15.29:0/3426726467 >> v2:172.21.15.186:3300/0 conn(0x7f11fc160530 0x7f11fc1609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 >> v1:172.21.15.186:6789/0 conn(0x7f11fc069fa0 legacy=0x7f11fc06a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc0acae0 con 0x7f11fc065ab0 2023-10-14T17:58:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f11ec001670 con 0x7f11fc065ab0 2023-10-14T17:58:37.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 >> v1:172.21.15.29:6789/0 conn(0x7f11fc065ab0 legacy=0x7f11fc160f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11e80047c0 0x7f11e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11e80051d0 0x7f11e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0ae1f0 con 0x7f11e80039c0 2023-10-14T17:58:37.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11e800c120 con 0x7f11e80051d0 2023-10-14T17:58:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11e800c2e0 con 0x7f11e80047c0 2023-10-14T17:58:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.450+0000 7f1201514640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201514640 1 -- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11e80047c0 msgr2=0x7f11e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201514640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11e80047c0 0x7f11e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201514640 1 -- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11e80051d0 msgr2=0x7f11e800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201d15640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11e80051d0 0x7f11e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201514640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11e80051d0 0x7f11e800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201514640 1 -- 172.21.15.29:0/3426726467 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11e8003520 con 0x7f11e80039c0 2023-10-14T17:58:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201d15640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11e80051d0 0x7f11e800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T17:58:37.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.451+0000 7f1201d15640 1 -- 172.21.15.29:0/3426726467 reap_dead start 2023-10-14T17:58:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.452+0000 7f1201514640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11e8003de0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009a90 tx=0x7f11e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=297c58c49f5ee427 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.452+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f802c680 con 0x7f11e80039c0 2023-10-14T17:58:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.452+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f802c7e0 con 0x7f11e80039c0 2023-10-14T17:58:37.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.452+0000 7f11f3fff640 1 -- 172.21.15.29:0/3426726467 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f802cb70 con 0x7f11e80039c0 2023-10-14T17:58:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 -- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 msgr2=0x7f11e8003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11e8003de0 secure :-1 s=READY pgs=2203 cs=0 l=1 rev1=1 crypto rx=0x7f11f8009a90 tx=0x7f11e80057f0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 -- 172.21.15.29:0/3426726467 shutdown_connections 2023-10-14T17:58:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 --2- 172.21.15.29:0/3426726467 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11e8003de0 unknown :-1 s=CLOSED pgs=2203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 -- 172.21.15.29:0/3426726467 >> 172.21.15.29:0/3426726467 conn(0x7f11fc06bc10 msgr2=0x7f11fc06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 -- 172.21.15.29:0/3426726467 shutdown_connections 2023-10-14T17:58:37.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.453+0000 7f1203778640 1 -- 172.21.15.29:0/3426726467 wait complete. 2023-10-14T17:58:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.455+0000 7f1203778640 1 Processor -- start 2023-10-14T17:58:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.455+0000 7f1203778640 1 -- start start 2023-10-14T17:58:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11fc1596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11fc15cbf0 0x7f11fc06d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11fc06dd70 0x7f11fc0c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1f2a10 con 0x7f11e80039c0 2023-10-14T17:58:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc06e1f0 con 0x7f11fc06dd70 2023-10-14T17:58:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.456+0000 7f1203778640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc1f3830 con 0x7f11fc15cbf0 2023-10-14T17:58:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11fc1596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11fc1596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41470/0 (socket says 172.21.15.29:41470) 2023-10-14T17:58:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 -- 172.21.15.29:0/176303556 learned_addr learned my addr 172.21.15.29:0/176303556 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11fc15cbf0 0x7f11fc06d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201d15640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11fc06dd70 0x7f11fc0c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:37.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11fc15cbf0 msgr2=0x7f11fc06d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f11fc15cbf0 0x7f11fc06d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11fc06dd70 msgr2=0x7f11fc0c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f11fc06dd70 0x7f11fc0c3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc1f2c20 con 0x7f11e80039c0 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.457+0000 7f1201514640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11fc1596f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f11f802c220 tx=0x7f11f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:37.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.458+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8003810 con 0x7f11e80039c0 2023-10-14T17:58:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.458+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f8003970 con 0x7f11e80039c0 2023-10-14T17:58:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.458+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11f8003d00 con 0x7f11e80039c0 2023-10-14T17:58:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.458+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc1f2f00 con 0x7f11e80039c0 2023-10-14T17:58:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.458+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc1f34c0 con 0x7f11e80039c0 2023-10-14T17:58:37.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.459+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc04e160 con 0x7f11e80039c0 2023-10-14T17:58:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.460+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 104) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f11f803b050 con 0x7f11e80039c0 2023-10-14T17:58:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.461+0000 7f11f1ffb640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.461+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(192..192 src has 1..192) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f11f80ab780 con 0x7f11e80039c0 2023-10-14T17:58:37.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.461+0000 7f1200d13640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.461+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T17:58:37.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.465+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f11f8076860 con 0x7f11e80039c0 2023-10-14T17:58:37.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.475+0000 7f4de2e32040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T17:58:37.488 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.488+0000 7f94d6030040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:58:37.529 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.527+0000 7f4de2e32040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T17:58:37.601 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.600+0000 7f94d6030040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:58:37.646 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.645+0000 7f4de2e32040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T17:58:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.662+0000 7f1200d13640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:37.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.662+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T17:58:37.664 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.663+0000 7f94d6030040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:37.681+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f11fc0ae490 con 0x7f11e006afe0 2023-10-14T17:58:37.712 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.711+0000 7f4de2e32040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:37.716 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.715+0000 7f94d6030040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:58:37.764 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.763+0000 7f4de2e32040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T17:58:37.773 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.773+0000 7f94d6030040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:58:37.824 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.823+0000 7f4de2e32040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T17:58:37.825 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:37.825+0000 7f94d6030040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:58:37.877 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:37.875+0000 7f4de2e32040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T17:58:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:38.063+0000 7f1200d13640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:38.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:38.063+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T17:58:38.220 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.219+0000 7f94d6030040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:58:38.279 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.277+0000 7f4de2e32040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T17:58:38.381 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.381+0000 7f94d6030040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:58:38.447 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.445+0000 7f4de2e32040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T17:58:38.492 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.492+0000 7f94d6030040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:58:38.543 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.543+0000 7f94d6030040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:58:38.560 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.558+0000 7f4de2e32040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T17:58:38.612 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.611+0000 7f4de2e32040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T17:58:38.652 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.651+0000 7f94d6030040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:38.725 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.724+0000 7f4de2e32040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:38.760 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.760+0000 7f94d6030040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:58:38.838 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:38.837+0000 7f4de2e32040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T17:58:38.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:38.864+0000 7f1200d13640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:38.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:38.864+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T17:58:38.920 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.919+0000 7f94d6030040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:38.970 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:38.969+0000 7f94d6030040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:39.002 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:39.001+0000 7f4de2e32040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T17:58:39.054 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:39.053+0000 7f4de2e32040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T17:58:39.091 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:39.090+0000 7f94d6030040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:58:39.179 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:39.178+0000 7f4de2e32040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T17:58:39.282 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:58:39.292 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:58:39.939 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:39.939+0000 7f94d6030040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:58:40.068 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.067+0000 7f4de2e32040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T17:58:40.245 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:40.245+0000 7f94d6030040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:58:40.304 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:40.303+0000 7f94d6030040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:58:40.379 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.378+0000 7f4de2e32040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T17:58:40.419 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T17:58:40.419+0000 7f94d6030040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:58:40.439 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.438+0000 7f4de2e32040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T17:58:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.466+0000 7f1200d13640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/79844030 2023-10-14T17:58:40.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.466+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T17:58:40.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.488+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 105) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f11f8076c30 con 0x7f11e80039c0 2023-10-14T17:58:40.565 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.564+0000 7f4de2e32040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T17:58:40.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.580+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 106) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f11f807e260 con 0x7f11e80039c0 2023-10-14T17:58:40.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.582+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 msgr2=0x7f11e006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T17:58:40.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:40.582+0000 7f11f1ffb640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/79844030,v1:172.21.15.29:6833/79844030] conn(0x7f11e006afe0 0x7f11e006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:40.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.937+0000 7f4d6097b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.939 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.937+0000 7f4d6097b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.937+0000 7f4d6097b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.937+0000 7f4d6097b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.937+0000 7f4d6097b640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.938+0000 7f4d5e977640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.941 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.938+0000 7f4d5e977640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.941 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.938+0000 7f4d5e977640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.941 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.938+0000 7f4d5e977640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:40.941 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:58:40.938+0000 7f4d5e977640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:58:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.596+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 107) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f11f8067ff0 con 0x7f11e80039c0 2023-10-14T17:58:41.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.597+0000 7f11f1ffb640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f11e0027900 0x7f11e0027ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:41.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.597+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 --> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "/snap_test_dir1/test_restart", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f11e0028220 con 0x7f11e0027900 2023-10-14T17:58:41.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.605+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f11e0027900 0x7f11e0027ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:41.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.613+0000 7f1200d13640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f11e0027900 0x7f11e0027ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0cac80 tx=0x7f11ec001930 comp rx=0 tx=0).ready entity=mgr.10302 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:41.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.811+0000 7f11f1ffb640 1 -- 172.21.15.29:0/176303556 <== mgr.10302 v2:172.21.15.29:6832/196725873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f11ec00c8a0 con 0x7f11e0027900 2023-10-14T17:58:41.812 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path /snap_test_dir1/test_restart 2023-10-14T17:58:41.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f11e0027900 msgr2=0x7f11e0027ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:41.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f11e0027900 0x7f11e0027ce0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f11fc0cac80 tx=0x7f11ec001930 comp rx=0 tx=0).stop 2023-10-14T17:58:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 msgr2=0x7f11fc1596f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 --2- 172.21.15.29:0/176303556 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f11e80039c0 0x7f11fc1596f0 secure :-1 s=READY pgs=2204 cs=0 l=1 rev1=1 crypto rx=0x7f11f802c220 tx=0x7f11f8004730 comp rx=0 tx=0).stop 2023-10-14T17:58:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1201d15640 1 -- 172.21.15.29:0/176303556 reap_dead start 2023-10-14T17:58:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 shutdown_connections 2023-10-14T17:58:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 >> 172.21.15.29:0/176303556 conn(0x7f11fc06bc10 msgr2=0x7f11fc1609d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:41.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 shutdown_connections 2023-10-14T17:58:41.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:41.815+0000 7f1203778640 1 -- 172.21.15.29:0/176303556 wait complete. 2023-10-14T17:58:41.841 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path /snap_test_dir1/test_restart 2023-10-14T17:58:41.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate /snap_test_dir1/test_restart 1m 2023-10-14T17:58:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.074+0000 7f441146e640 1 Processor -- start 2023-10-14T17:58:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.074+0000 7f441146e640 1 -- start start 2023-10-14T17:58:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.075+0000 7f441146e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f440c06c040 0x7f440c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f441146e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 0x7f440c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f441146e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440c06c960 con 0x7f440c0ad970 2023-10-14T17:58:42.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f441146e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440c15b9d0 con 0x7f440c06c040 2023-10-14T17:58:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f441146e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440c15bcb0 con 0x7f440c0cda80 2023-10-14T17:58:42.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f440bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f440c06c040 0x7f440c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f440bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f440c06c040 0x7f440c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57882/0 (socket says 172.21.15.29:57882) 2023-10-14T17:58:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f440b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f440c0ad970 0x7f440c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:54320/0 (socket says 172.21.15.29:54320) 2023-10-14T17:58:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f440bfff640 1 -- 172.21.15.29:0/2919459475 learned_addr learned my addr 172.21.15.29:0/2919459475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.076+0000 7f4410c6d640 1 --2- 172.21.15.29:0/2919459475 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 0x7f440c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.077+0000 7f440bfff640 1 -- 172.21.15.29:0/2919459475 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 msgr2=0x7f440c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T17:58:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.077+0000 7f440bfff640 1 --2- 172.21.15.29:0/2919459475 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 0x7f440c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.077+0000 7f440bfff640 1 -- 172.21.15.29:0/2919459475 >> v1:172.21.15.29:6789/0 conn(0x7f440c0ad970 legacy=0x7f440c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.077+0000 7f440bfff640 1 -- 172.21.15.29:0/2919459475 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c15bf90 con 0x7f440c06c040 2023-10-14T17:58:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.077+0000 7f4410c6d640 1 --2- 172.21.15.29:0/2919459475 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 0x7f440c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T17:58:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.078+0000 7f440bfff640 1 --2- 172.21.15.29:0/2919459475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c06c040 0x7f440c06c420 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f440c0cf300 tx=0x7f43fc02b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53957f65a969202c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.078+0000 7f440affd640 1 -- 172.21.15.29:0/2919459475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc030870 con 0x7f440c06c040 2023-10-14T17:58:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.078+0000 7f440affd640 1 -- 172.21.15.29:0/2919459475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc0309d0 con 0x7f440c06c040 2023-10-14T17:58:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.078+0000 7f440affd640 1 -- 172.21.15.29:0/2919459475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc030d60 con 0x7f440c06c040 2023-10-14T17:58:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 -- 172.21.15.29:0/2919459475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c06c040 msgr2=0x7f440c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 --2- 172.21.15.29:0/2919459475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c06c040 0x7f440c06c420 secure :-1 s=READY pgs=2219 cs=0 l=1 rev1=1 crypto rx=0x7f440c0cf300 tx=0x7f43fc02b9b0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 -- 172.21.15.29:0/2919459475 shutdown_connections 2023-10-14T17:58:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 --2- 172.21.15.29:0/2919459475 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cda80 0x7f440c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 --2- 172.21.15.29:0/2919459475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c06c040 0x7f440c06c420 unknown :-1 s=CLOSED pgs=2219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 -- 172.21.15.29:0/2919459475 >> 172.21.15.29:0/2919459475 conn(0x7f440c0ce6a0 msgr2=0x7f440c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 -- 172.21.15.29:0/2919459475 shutdown_connections 2023-10-14T17:58:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.079+0000 7f441146e640 1 -- 172.21.15.29:0/2919459475 wait complete. 2023-10-14T17:58:42.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.081+0000 7f441146e640 1 Processor -- start 2023-10-14T17:58:42.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.081+0000 7f441146e640 1 -- start start 2023-10-14T17:58:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c06c040 0x7f440c153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c0ad970 0x7f440c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0c77d0 con 0x7f440c0cda80 2023-10-14T17:58:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c153fa0 con 0x7f440c06c040 2023-10-14T17:58:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f441146e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440c158080 con 0x7f440c0ad970 2023-10-14T17:58:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.082+0000 7f4410c6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:57890/0 (socket says 172.21.15.29:57890) 2023-10-14T17:58:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 -- 172.21.15.29:0/77155847 learned_addr learned my addr 172.21.15.29:0/77155847 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T17:58:42.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f440bfff640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c06c040 0x7f440c153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 -- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c0ad970 msgr2=0x7f440c0c7e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c0ad970 0x7f440c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 -- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c06c040 msgr2=0x7f440c153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c06c040 0x7f440c153a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 -- 172.21.15.29:0/77155847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc0094d0 con 0x7f440c0cda80 2023-10-14T17:58:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.083+0000 7f4410c6d640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f4404007e20 tx=0x7f440400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.084+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4404003b90 con 0x7f440c0cda80 2023-10-14T17:58:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.084+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f440400f040 con 0x7f440c0cda80 2023-10-14T17:58:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.084+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f440400de80 con 0x7f440c0cda80 2023-10-14T17:58:42.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.084+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c0c6c50 con 0x7f440c0cda80 2023-10-14T17:58:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.084+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c0c7210 con 0x7f440c0cda80 2023-10-14T17:58:42.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.085+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c06b330 con 0x7f440c0cda80 2023-10-14T17:58:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.086+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 107) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4404004030 con 0x7f440c0cda80 2023-10-14T17:58:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.087+0000 7f4408ff9640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 0x7f43ec06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T17:58:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.087+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f4404089820 con 0x7f440c0cda80 2023-10-14T17:58:42.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.087+0000 7f440bfff640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 0x7f43ec06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T17:58:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.088+0000 7f440bfff640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 0x7f43ec06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f43fc036000 tx=0x7f43fc03b040 comp rx=0 tx=0).ready entity=mgr.10302 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T17:58:42.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.092+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4404054970 con 0x7f440c0cda80 2023-10-14T17:58:42.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.319+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 --> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f440c0cc550 con 0x7f43ec06ad10 2023-10-14T17:58:42.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.468+0000 7f4408ff9640 1 -- 172.21.15.29:0/77155847 <== mgr.10302 v2:172.21.15.29:6832/196725873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f43fc030d30 con 0x7f43ec06ad10 2023-10-14T17:58:42.470 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path /snap_test_dir1/test_restart 2023-10-14T17:58:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.471+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 msgr2=0x7f43ec06d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.471+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 0x7f43ec06d1d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f43fc036000 tx=0x7f43fc03b040 comp rx=0 tx=0).stop 2023-10-14T17:58:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.471+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 msgr2=0x7f440c0c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T17:58:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.471+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 secure :-1 s=READY pgs=2220 cs=0 l=1 rev1=1 crypto rx=0x7f4404007e20 tx=0x7f440400d4a0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 shutdown_connections 2023-10-14T17:58:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f43ec06ad10 0x7f43ec06d1d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cda80 0x7f440c0c9510 unknown :-1 s=CLOSED pgs=2220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c0ad970 0x7f440c0c7e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 --2- 172.21.15.29:0/77155847 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c06c040 0x7f440c153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T17:58:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 >> 172.21.15.29:0/77155847 conn(0x7f440c0ce6a0 msgr2=0x7f440c15d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T17:58:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 shutdown_connections 2023-10-14T17:58:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T17:58:42.472+0000 7f441146e640 1 -- 172.21.15.29:0/77155847 wait complete. 2023-10-14T17:58:42.497 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path /snap_test_dir1/test_restart 2023-10-14T17:59:00.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:59:00.493+0000 7f4d5115c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:59:00.496 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:59:00.493+0000 7f4d5115c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:59:00.496 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:59:00.493+0000 7f4d5115c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:59:00.496 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:59:00.493+0000 7f4d5115c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:59:00.497 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T17:59:00.493+0000 7f4d5115c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T17:59:09.488 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:59:09.498 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:59:39.686 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T17:59:39.696 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:00:09.900 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:00:09.910 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:00:40.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:00:40.058 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:01:10.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:01:10.264 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:01:40.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:01:40.494 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:02:10.689 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:02:10.699 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:02:40.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:02:40.846 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:03:11.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:03:11.045 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:03:41.239 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:03:41.249 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:04:11.453 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:04:11.463 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:04:41.650 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:04:41.660 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:05:11.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:05:11.815 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:05:42.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:05:42.038 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:06:12.234 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:06:12.244 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:06:42.456 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:06:42.466 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:07:12.662 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:07:12.672 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:07:42.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:07:42.894 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:08:13.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:08:13.108 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:08:43.254 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:08:43.264 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:09:13.459 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:09:13.469 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:09:16.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:09:16.499 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-14T18:09:16.568 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_59_00_UTC 2023-10-14T18:09:16.568 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_00_00_UTC 2023-10-14T18:09:16.568 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_01_00_UTC 2023-10-14T18:09:16.569 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_02_00_UTC 2023-10-14T18:09:16.569 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_03_00_UTC 2023-10-14T18:09:16.569 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_04_00_UTC 2023-10-14T18:09:16.569 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_05_00_UTC 2023-10-14T18:09:16.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_06_00_UTC 2023-10-14T18:09:16.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_07_00_UTC 2023-10-14T18:09:16.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_08_00_UTC 2023-10-14T18:09:16.570 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_09_00_UTC 2023-10-14T18:09:16.571 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-17_59_00_UTC', 'scheduled-2023-10-14-18_00_00_UTC', 'scheduled-2023-10-14-18_01_00_UTC', 'scheduled-2023-10-14-18_02_00_UTC', 'scheduled-2023-10-14-18_03_00_UTC', 'scheduled-2023-10-14-18_04_00_UTC', 'scheduled-2023-10-14-18_05_00_UTC', 'scheduled-2023-10-14-18_06_00_UTC', 'scheduled-2023-10-14-18_07_00_UTC', 'scheduled-2023-10-14-18_08_00_UTC', 'scheduled-2023-10-14-18_09_00_UTC'] 2023-10-14T18:09:16.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-14T18:09:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.812+0000 7f132bfff640 1 Processor -- start 2023-10-14T18:09:16.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.812+0000 7f132bfff640 1 -- start start 2023-10-14T18:09:16.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.813+0000 7f132bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132c0c9060 0x7f132c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f132c0c9440 0x7f132c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c0ca560 con 0x7f132c06c9e0 2023-10-14T18:09:16.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132c0c9950 con 0x7f132c0c9060 2023-10-14T18:09:16.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132bfff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f132c0c9c30 con 0x7f132c0c9440 2023-10-14T18:09:16.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132c0c9060 0x7f132c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f132c0c9060 0x7f132c06f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43312/0 (socket says 172.21.15.29:43312) 2023-10-14T18:09:16.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132affd640 1 -- 172.21.15.29:0/2656548204 learned_addr learned my addr 172.21.15.29:0/2656548204 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:09:16.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.814+0000 7f132b7fe640 1 --2- 172.21.15.29:0/2656548204 >> v2:172.21.15.186:3300/0 conn(0x7f132c0c9440 0x7f132c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.815+0000 7f132affd640 1 -- 172.21.15.29:0/2656548204 >> v2:172.21.15.186:3300/0 conn(0x7f132c0c9440 msgr2=0x7f132c06f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:09:16.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.815+0000 7f132affd640 1 --2- 172.21.15.29:0/2656548204 >> v2:172.21.15.186:3300/0 conn(0x7f132c0c9440 0x7f132c06f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:16.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.815+0000 7f132affd640 1 -- 172.21.15.29:0/2656548204 >> v1:172.21.15.186:6789/0 conn(0x7f132c06c9e0 legacy=0x7f132c164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:09:16.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.815+0000 7f132affd640 1 -- 172.21.15.29:0/2656548204 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0c9f10 con 0x7f132c0c9060 2023-10-14T18:09:16.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.816+0000 7f132affd640 1 --2- 172.21.15.29:0/2656548204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9060 0x7f132c06f440 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f132c0ce550 tx=0x7f131c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea8ea091c968a14a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:16.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.816+0000 7f1329ffb640 1 -- 172.21.15.29:0/2656548204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c031870 con 0x7f132c0c9060 2023-10-14T18:09:16.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f1329ffb640 1 -- 172.21.15.29:0/2656548204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f131c0319d0 con 0x7f132c0c9060 2023-10-14T18:09:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f1329ffb640 1 -- 172.21.15.29:0/2656548204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f131c031d60 con 0x7f132c0c9060 2023-10-14T18:09:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 -- 172.21.15.29:0/2656548204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9060 msgr2=0x7f132c06f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 --2- 172.21.15.29:0/2656548204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9060 0x7f132c06f440 secure :-1 s=READY pgs=2225 cs=0 l=1 rev1=1 crypto rx=0x7f132c0ce550 tx=0x7f131c02b910 comp rx=0 tx=0).stop 2023-10-14T18:09:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 -- 172.21.15.29:0/2656548204 shutdown_connections 2023-10-14T18:09:16.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 --2- 172.21.15.29:0/2656548204 >> v2:172.21.15.186:3300/0 conn(0x7f132c0c9440 0x7f132c06f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:16.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 --2- 172.21.15.29:0/2656548204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9060 0x7f132c06f440 unknown :-1 s=CLOSED pgs=2225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:16.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 -- 172.21.15.29:0/2656548204 >> 172.21.15.29:0/2656548204 conn(0x7f132c0ce6a0 msgr2=0x7f132c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:09:16.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 -- 172.21.15.29:0/2656548204 shutdown_connections 2023-10-14T18:09:16.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.817+0000 7f132bfff640 1 -- 172.21.15.29:0/2656548204 wait complete. 2023-10-14T18:09:16.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.819+0000 7f132bfff640 1 Processor -- start 2023-10-14T18:09:16.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.819+0000 7f132bfff640 1 -- start start 2023-10-14T18:09:16.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f132c06c9e0 0x7f132c1fa010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f132c0c9060 0x7f132c1fa550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0c5ab0 con 0x7f132c0c9440 2023-10-14T18:09:16.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f132c1f6860 con 0x7f132c0c9060 2023-10-14T18:09:16.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.820+0000 7f132bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f132c0b31e0 con 0x7f132c06c9e0 2023-10-14T18:09:16.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43322/0 (socket says 172.21.15.29:43322) 2023-10-14T18:09:16.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 -- 172.21.15.29:0/688954850 learned_addr learned my addr 172.21.15.29:0/688954850 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:09:16.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132affd640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f132c06c9e0 0x7f132c1fa010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132a7fc640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f132c0c9060 0x7f132c1fa550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 -- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f132c06c9e0 msgr2=0x7f132c1fa010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f132c06c9e0 0x7f132c1fa010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 -- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f132c0c9060 msgr2=0x7f132c1fa550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:16.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f132c0c9060 0x7f132c1fa550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.821+0000 7f132b7fe640 1 -- 172.21.15.29:0/688954850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f131c0094d0 con 0x7f132c0c9440 2023-10-14T18:09:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f132b7fe640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f1324009fd0 tx=0x7f132400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324014420 con 0x7f132c0c9440 2023-10-14T18:09:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1324014580 con 0x7f132c0c9440 2023-10-14T18:09:16.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132c0b34c0 con 0x7f132c0c9440 2023-10-14T18:09:16.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1324014930 con 0x7f132c0c9440 2023-10-14T18:09:16.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.822+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f132c0b3a80 con 0x7f132c0c9440 2023-10-14T18:09:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.823+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132c0c5c20 con 0x7f132c0c9440 2023-10-14T18:09:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.824+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f1324014a90 con 0x7f132c0c9440 2023-10-14T18:09:16.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.825+0000 7f13137fe640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 0x7f131806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.825+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f132408a360 con 0x7f132c0c9440 2023-10-14T18:09:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.825+0000 7f132affd640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 0x7f131806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:16.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.826+0000 7f132affd640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 0x7f131806d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f131c036000 tx=0x7f131c03c040 comp rx=0 tx=0).ready entity=mgr.10302 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:16.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:16.830+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1324055090 con 0x7f132c0c9440 2023-10-14T18:09:17.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.049+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 --> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f132c0c1f10 con 0x7f131806afe0 2023-10-14T18:09:17.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.067+0000 7f13137fe640 1 -- 172.21.15.29:0/688954850 <== mgr.10302 v2:172.21.15.29:6832/196725873 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+375 (secure 0 0 0) 0x7f131c031bc0 con 0x7f131806afe0 2023-10-14T18:09:17.069 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:09:17.069 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:58:41", "first": "2023-10-14T17:59:00", "last": "2023-10-14T18:09:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-14T18:09:17.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 msgr2=0x7f131806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 0x7f131806d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f131c036000 tx=0x7f131c03c040 comp rx=0 tx=0).stop 2023-10-14T18:09:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 msgr2=0x7f132c1532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 secure :-1 s=READY pgs=2226 cs=0 l=1 rev1=1 crypto rx=0x7f1324009fd0 tx=0x7f132400e4b0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 shutdown_connections 2023-10-14T18:09:17.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f131806afe0 0x7f131806d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f132c0c9440 0x7f132c1532a0 unknown :-1 s=CLOSED pgs=2226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f132c0c9060 0x7f132c1fa550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 --2- 172.21.15.29:0/688954850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f132c06c9e0 0x7f132c1fa010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 >> 172.21.15.29:0/688954850 conn(0x7f132c0ce6a0 msgr2=0x7f132c0cc830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:09:17.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.070+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 shutdown_connections 2023-10-14T18:09:17.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.071+0000 7f132bfff640 1 -- 172.21.15.29:0/688954850 wait complete. 2023-10-14T18:09:17.091 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:58:41", "first": "2023-10-14T17:59:00", "last": "2023-10-14T18:09:00", "last_pruned": null, "created_count": 11, "pruned_count": 0, "active": true}] 2023-10-14T18:09:17.091 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T17:58:41', 'first': '2023-10-14T17:59:00', 'last': '2023-10-14T18:09:00', 'last_pruned': None, 'created_count': 11, 'pruned_count': 0, 'active': True} 2023-10-14T18:09:17.092 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 11 2023-10-14T18:09:17.092 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 11 2023-10-14T18:09:17.093 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-10-14T18:09:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.317+0000 7f359a599640 1 Processor -- start 2023-10-14T18:09:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.317+0000 7f359a599640 1 -- start start 2023-10-14T18:09:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f359a599640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 0x7f359406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f359a599640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35940cda80 0x7f359415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f359a599640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f359406c960 con 0x7f35940ad970 2023-10-14T18:09:17.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f359a599640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f359415b9d0 con 0x7f35940cda80 2023-10-14T18:09:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f359a599640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f359415bcb0 con 0x7f359406c040 2023-10-14T18:09:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f3599d98640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f35940cda80 0x7f359415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f3598d96640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f35940ad970 0x7f359415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:51314/0 (socket says 172.21.15.29:51314) 2023-10-14T18:09:17.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.319+0000 7f3598d96640 1 -- 172.21.15.29:0/2898450570 learned_addr learned my addr 172.21.15.29:0/2898450570 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:09:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599597640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 0x7f359406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599d98640 1 -- 172.21.15.29:0/2898450570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 msgr2=0x7f359406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:09:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599d98640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 0x7f359406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599d98640 1 -- 172.21.15.29:0/2898450570 >> v1:172.21.15.29:6789/0 conn(0x7f35940ad970 legacy=0x7f359415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599d98640 1 -- 172.21.15.29:0/2898450570 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f359415bf90 con 0x7f35940cda80 2023-10-14T18:09:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.320+0000 7f3599597640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 0x7f359406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:09:17.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.321+0000 7f3599d98640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940cda80 0x7f359415ed90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f358c009d90 tx=0x7f358c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32e85af21db4d778 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.321+0000 7f358bfff640 1 -- 172.21.15.29:0/2898450570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c02cc50 con 0x7f35940cda80 2023-10-14T18:09:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.321+0000 7f358bfff640 1 -- 172.21.15.29:0/2898450570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f358c02cdb0 con 0x7f35940cda80 2023-10-14T18:09:17.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.321+0000 7f358bfff640 1 -- 172.21.15.29:0/2898450570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f358c034660 con 0x7f35940cda80 2023-10-14T18:09:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 -- 172.21.15.29:0/2898450570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940cda80 msgr2=0x7f359415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940cda80 0x7f359415ed90 secure :-1 s=READY pgs=2228 cs=0 l=1 rev1=1 crypto rx=0x7f358c009d90 tx=0x7f358c02bea0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 -- 172.21.15.29:0/2898450570 shutdown_connections 2023-10-14T18:09:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940cda80 0x7f359415ed90 unknown :-1 s=CLOSED pgs=2228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 --2- 172.21.15.29:0/2898450570 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f359406c040 0x7f359406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 -- 172.21.15.29:0/2898450570 >> 172.21.15.29:0/2898450570 conn(0x7f35940ce6a0 msgr2=0x7f35940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:09:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 -- 172.21.15.29:0/2898450570 shutdown_connections 2023-10-14T18:09:17.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.322+0000 7f359a599640 1 -- 172.21.15.29:0/2898450570 wait complete. 2023-10-14T18:09:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.324+0000 7f359a599640 1 Processor -- start 2023-10-14T18:09:17.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.324+0000 7f359a599640 1 -- start start 2023-10-14T18:09:17.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406c040 0x7f35941f1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 0x7f35940a9e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35940b26d0 con 0x7f35940ad970 2023-10-14T18:09:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35940ace10 con 0x7f359406c040 2023-10-14T18:09:17.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f359a599640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35940ad0f0 con 0x7f35940cda80 2023-10-14T18:09:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f3598d96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f3598d96640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43340/0 (socket says 172.21.15.29:43340) 2023-10-14T18:09:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.325+0000 7f3598d96640 1 -- 172.21.15.29:0/499833687 learned_addr learned my addr 172.21.15.29:0/499833687 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:09:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3599d98640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 0x7f35940a9e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3599597640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406c040 0x7f35941f1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 -- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 msgr2=0x7f35940a9e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 0x7f35940a9e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 -- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406c040 msgr2=0x7f35941f1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406c040 0x7f35941f1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 -- 172.21.15.29:0/499833687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f358c009a70 con 0x7f35940ad970 2023-10-14T18:09:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3599d98640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 0x7f35940a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:09:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3598d96640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f358400c4a0 tx=0x7f358400c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584005ce0 con 0x7f35940ad970 2023-10-14T18:09:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3584005e40 con 0x7f35940ad970 2023-10-14T18:09:17.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3584007270 con 0x7f35940ad970 2023-10-14T18:09:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.326+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35940ad3d0 con 0x7f35940ad970 2023-10-14T18:09:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.327+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35941f2110 con 0x7f35940ad970 2023-10-14T18:09:17.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.328+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359406b330 con 0x7f35940ad970 2023-10-14T18:09:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.329+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 111) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f3584007490 con 0x7f35940ad970 2023-10-14T18:09:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.329+0000 7f3589ffb640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 0x7f357806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:09:17.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.330+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(193..193 src has 1..193) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f3584088d80 con 0x7f35940ad970 2023-10-14T18:09:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.330+0000 7f3599597640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 0x7f357806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:09:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.331+0000 7f3599597640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 0x7f357806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3590008850 tx=0x7f3590008040 comp rx=0 tx=0).ready entity=mgr.10302 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:09:17.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.334+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3584053c90 con 0x7f35940ad970 2023-10-14T18:09:17.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.578+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f359404e9b0 con 0x7f35940ad970 2023-10-14T18:09:17.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.586+0000 7f3589ffb640 1 -- 172.21.15.29:0/499833687 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v111) v1 ==== 74+0+171233 (secure 0 0 0) 0x7f3584017020 con 0x7f35940ad970 2023-10-14T18:09:17.590 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:09:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 msgr2=0x7f357806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 0x7f357806d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f3590008850 tx=0x7f3590008040 comp rx=0 tx=0).stop 2023-10-14T18:09:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 msgr2=0x7f35940a94d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:09:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 secure :-1 s=READY pgs=2229 cs=0 l=1 rev1=1 crypto rx=0x7f358400c4a0 tx=0x7f358400c960 comp rx=0 tx=0).stop 2023-10-14T18:09:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 shutdown_connections 2023-10-14T18:09:17.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:6832/196725873,v1:172.21.15.29:6833/196725873] conn(0x7f357806afe0 0x7f357806d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f35940cda80 0x7f35940a9e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f35940ad970 0x7f35940a94d0 unknown :-1 s=CLOSED pgs=2229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 --2- 172.21.15.29:0/499833687 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f359406c040 0x7f35941f1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:09:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 >> 172.21.15.29:0/499833687 conn(0x7f35940ce6a0 msgr2=0x7f359415e610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:09:17.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.591+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 shutdown_connections 2023-10-14T18:09:17.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:09:17.592+0000 7f359a599640 1 -- 172.21.15.29:0/499833687 wait complete. 2023-10-14T18:09:17.612 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":111,"active_gid":10302,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6832","nonce":196725873},{"type":"v1","addr":"172.21.15.29:6833","nonce":196725873}]},"active_addr":"172.21.15.29:6833/196725873","active_change":"2023-10-14T17:58:40.568498+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":6836,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful","snap_schedule"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":193,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":1377837433}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":614546910}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":3575832370}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":841868971}]},{"name":"snap_schedule","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":2776961501}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.29:0","nonce":287831498}]}]} 2023-10-14T18:09:17.617 DEBUG:tasks.cephfs.test_snap_schedules:restarting active mgr: y 2023-10-14T18:09:17.617 INFO:tasks.ceph.mgr.y:Restarting daemon 2023-10-14T18:09:17.618 INFO:tasks.ceph.mgr.y:Stopping old one... 2023-10-14T18:09:17.618 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-14T18:09:17.618 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:09:17.714 INFO:tasks.ceph.mgr.y:Stopped 2023-10-14T18:09:17.714 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2023-10-14T18:09:17.720 INFO:tasks.ceph.mgr.y:Started 2023-10-14T18:09:18.342 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:18.340+0000 7f6453a3e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:09:18.398 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:18.397+0000 7f6453a3e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:09:18.868 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:18.866+0000 7f6453a3e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:09:18.922 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:18.921+0000 7f6453a3e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:09:18.974 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:18.973+0000 7f6453a3e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:09:19.039 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.038+0000 7f6453a3e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:09:19.090 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.089+0000 7f6453a3e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:09:19.207 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.206+0000 7f6453a3e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:09:19.272 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.271+0000 7f6453a3e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:09:19.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.323+0000 7f6453a3e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:09:19.383 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.382+0000 7f6453a3e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:09:19.436 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.434+0000 7f6453a3e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:09:19.838 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:19.837+0000 7f6453a3e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:09:20.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.004+0000 7f6453a3e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:09:20.119 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.117+0000 7f6453a3e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:09:20.171 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.170+0000 7f6453a3e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:09:20.284 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.283+0000 7f6453a3e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:09:20.397 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.396+0000 7f6453a3e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:09:20.561 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.560+0000 7f6453a3e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:09:20.613 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.612+0000 7f6453a3e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:09:20.738 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:20.737+0000 7f6453a3e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:09:21.629 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:21.628+0000 7f6453a3e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:09:21.940 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:21.938+0000 7f6453a3e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:09:22.000 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:21.998+0000 7f6453a3e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:09:22.119 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.118+0000 7f6453a3e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:09:22.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.523+0000 7f63cfd18640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63cfd18640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63cfd18640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63cfd18640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63cfd18640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63d2d1e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63d2d1e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63d2d1e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.527 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63d2d1e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:22.527 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:09:22.524+0000 7f63d2d1e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:09:43.673 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:09:43.683 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:10:00.516 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:10:00.514+0000 7f63dad6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:10:00.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:10:00.514+0000 7f63dad6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:10:00.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:10:00.514+0000 7f63dad6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:10:00.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:10:00.514+0000 7f63dad6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:10:00.517 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:10:00.514+0000 7f63dad6e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:10:13.888 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:10:13.898 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:10:44.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:10:44.129 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:11:14.378 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:11:14.388 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:11:44.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:11:44.623 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:12:14.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:12:14.863 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:12:45.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:12:45.077 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:13:15.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:13:15.283 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:13:45.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:13:45.429 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:14:15.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:14:15.677 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:14:17.721 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate /snap_test_dir1/test_restart 1m 2023-10-14T18:14:17.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.967+0000 7f3653fff640 1 Processor -- start 2023-10-14T18:14:17.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.967+0000 7f3653fff640 1 -- start start 2023-10-14T18:14:17.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.968+0000 7f3653fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.968+0000 7f3653fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 0x7f36540cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f3653fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f365415b9d0 con 0x7f365406c040 2023-10-14T18:14:17.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f3653fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f365415bcb0 con 0x7f36540ad970 2023-10-14T18:14:17.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f3653fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f365415bf90 con 0x7f36540ccfb0 2023-10-14T18:14:17.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f36527fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f36537fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 0x7f36540cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f36527fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:42018/0 (socket says 172.21.15.29:42018) 2023-10-14T18:14:17.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f36527fc640 1 -- 172.21.15.29:0/3017577386 learned_addr learned my addr 172.21.15.29:0/3017577386 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:17.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.969+0000 7f3652ffd640 1 --1- 172.21.15.29:0/3017577386 >> v1:172.21.15.186:6789/0 conn(0x7f365406c040 0x7f365406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:49298/0 (socket says 172.21.15.29:49298) 2023-10-14T18:14:17.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.970+0000 7f36537fe640 1 -- 172.21.15.29:0/3017577386 >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 msgr2=0x7f36540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:17.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.970+0000 7f36537fe640 1 --2- 172.21.15.29:0/3017577386 >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.970+0000 7f36537fe640 1 -- 172.21.15.29:0/3017577386 >> v1:172.21.15.186:6789/0 conn(0x7f365406c040 legacy=0x7f365406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:17.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.970+0000 7f36537fe640 1 -- 172.21.15.29:0/3017577386 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36540cc270 con 0x7f36540ccfb0 2023-10-14T18:14:17.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.970+0000 7f36527fc640 1 --2- 172.21.15.29:0/3017577386 >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:14:17.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.972+0000 7f36537fe640 1 --2- 172.21.15.29:0/3017577386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 0x7f36540cd460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097a0 tx=0x7f364c02b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71166e40aae89c63 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.972+0000 7f3651ffb640 1 -- 172.21.15.29:0/3017577386 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c02d040 con 0x7f36540ccfb0 2023-10-14T18:14:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.972+0000 7f3651ffb640 1 -- 172.21.15.29:0/3017577386 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364c031810 con 0x7f36540ccfb0 2023-10-14T18:14:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3651ffb640 1 -- 172.21.15.29:0/3017577386 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c031ba0 con 0x7f36540ccfb0 2023-10-14T18:14:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 -- 172.21.15.29:0/3017577386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 msgr2=0x7f36540cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:17.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 --2- 172.21.15.29:0/3017577386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 0x7f36540cd460 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097a0 tx=0x7f364c02b8b0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 -- 172.21.15.29:0/3017577386 shutdown_connections 2023-10-14T18:14:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 --2- 172.21.15.29:0/3017577386 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ccfb0 0x7f36540cd460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 --2- 172.21.15.29:0/3017577386 >> v2:172.21.15.186:3300/0 conn(0x7f36540ad970 0x7f36540cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.973+0000 7f3653fff640 1 -- 172.21.15.29:0/3017577386 >> 172.21.15.29:0/3017577386 conn(0x7f36540ce6a0 msgr2=0x7f36540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:17.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.974+0000 7f3653fff640 1 -- 172.21.15.29:0/3017577386 shutdown_connections 2023-10-14T18:14:17.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.974+0000 7f3653fff640 1 -- 172.21.15.29:0/3017577386 wait complete. 2023-10-14T18:14:17.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.976+0000 7f3653fff640 1 Processor -- start 2023-10-14T18:14:17.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.976+0000 7f3653fff640 1 -- start start 2023-10-14T18:14:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f365406c040 0x7f36540afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ad970 0x7f36540c8060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36540c21b0 con 0x7f36540ccfb0 2023-10-14T18:14:17.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36540c85a0 con 0x7f365406c040 2023-10-14T18:14:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.977+0000 7f3653fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36540b0470 con 0x7f36540ad970 2023-10-14T18:14:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42706/0 (socket says 172.21.15.29:42706) 2023-10-14T18:14:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 -- 172.21.15.29:0/4240324274 learned_addr learned my addr 172.21.15.29:0/4240324274 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:17.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36527fc640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ad970 0x7f36540c8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f3652ffd640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f365406c040 0x7f36540afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 -- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ad970 msgr2=0x7f36540c8060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ad970 0x7f36540c8060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 -- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f365406c040 msgr2=0x7f36540afe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f365406c040 0x7f36540afe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.978+0000 7f36537fe640 1 -- 172.21.15.29:0/4240324274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36480094d0 con 0x7f36540ccfb0 2023-10-14T18:14:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f36537fe640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097a0 tx=0x7f364c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c006e10 con 0x7f36540ccfb0 2023-10-14T18:14:17.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f364c03ace0 con 0x7f36540ccfb0 2023-10-14T18:14:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f364c044610 con 0x7f36540ccfb0 2023-10-14T18:14:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f364c009400 con 0x7f36540ccfb0 2023-10-14T18:14:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.979+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36540c23c0 con 0x7f36540ccfb0 2023-10-14T18:14:17.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.981+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f365406b330 con 0x7f36540ccfb0 2023-10-14T18:14:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.982+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f364c04d450 con 0x7f36540ccfb0 2023-10-14T18:14:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.983+0000 7f363b7fe640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 0x7f364006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:17.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.983+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f364c0abda0 con 0x7f36540ccfb0 2023-10-14T18:14:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.983+0000 7f3652ffd640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 0x7f364006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:17.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.984+0000 7f3652ffd640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 0x7f364006d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f36440012e0 tx=0x7f3644005b90 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:17.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:17.987+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f364c0768f0 con 0x7f36540ccfb0 2023-10-14T18:14:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.210+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 --> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f36540cc550 con 0x7f364006afe0 2023-10-14T18:14:18.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.390+0000 7f363b7fe640 1 -- 172.21.15.29:0/4240324274 <== mgr.10398 v2:172.21.15.29:6832/1263890494 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f36440071f0 con 0x7f364006afe0 2023-10-14T18:14:18.391 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-14T18:14:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 msgr2=0x7f364006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 0x7f364006d4a0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f36440012e0 tx=0x7f3644005b90 comp rx=0 tx=0).stop 2023-10-14T18:14:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 msgr2=0x7f36541532a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 secure :-1 s=READY pgs=2248 cs=0 l=1 rev1=1 crypto rx=0x7f364c0097a0 tx=0x7f364c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 shutdown_connections 2023-10-14T18:14:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f364006afe0 0x7f364006d4a0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36540ccfb0 0x7f36541532a0 unknown :-1 s=CLOSED pgs=2248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f36540ad970 0x7f36540c8060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 --2- 172.21.15.29:0/4240324274 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f365406c040 0x7f36540afe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.393+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 >> 172.21.15.29:0/4240324274 conn(0x7f36540ce6a0 msgr2=0x7f36540ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.394+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 shutdown_connections 2023-10-14T18:14:18.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.394+0000 7f3653fff640 1 -- 172.21.15.29:0/4240324274 wait complete. 2023-10-14T18:14:18.415 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path /snap_test_dir1/test_restart 2023-10-14T18:14:18.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:18.416 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-14T18:14:18.475 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_59_00_UTC 2023-10-14T18:14:18.475 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_00_00_UTC 2023-10-14T18:14:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_01_00_UTC 2023-10-14T18:14:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_02_00_UTC 2023-10-14T18:14:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_03_00_UTC 2023-10-14T18:14:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_04_00_UTC 2023-10-14T18:14:18.476 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_05_00_UTC 2023-10-14T18:14:18.477 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_06_00_UTC 2023-10-14T18:14:18.477 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_07_00_UTC 2023-10-14T18:14:18.477 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_08_00_UTC 2023-10-14T18:14:18.477 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_09_00_UTC 2023-10-14T18:14:18.477 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_10_00_UTC 2023-10-14T18:14:18.478 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_11_00_UTC 2023-10-14T18:14:18.478 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_12_00_UTC 2023-10-14T18:14:18.478 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_13_00_UTC 2023-10-14T18:14:18.478 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_14_00_UTC 2023-10-14T18:14:18.479 DEBUG:tasks.cephfs.test_snap_schedules:snapshots: ['scheduled-2023-10-14-17_59_00_UTC', 'scheduled-2023-10-14-18_00_00_UTC', 'scheduled-2023-10-14-18_01_00_UTC', 'scheduled-2023-10-14-18_02_00_UTC', 'scheduled-2023-10-14-18_03_00_UTC', 'scheduled-2023-10-14-18_04_00_UTC', 'scheduled-2023-10-14-18_05_00_UTC', 'scheduled-2023-10-14-18_06_00_UTC', 'scheduled-2023-10-14-18_07_00_UTC', 'scheduled-2023-10-14-18_08_00_UTC', 'scheduled-2023-10-14-18_09_00_UTC', 'scheduled-2023-10-14-18_10_00_UTC', 'scheduled-2023-10-14-18_11_00_UTC', 'scheduled-2023-10-14-18_12_00_UTC', 'scheduled-2023-10-14-18_13_00_UTC', 'scheduled-2023-10-14-18_14_00_UTC'] 2023-10-14T18:14:18.480 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status --format json /snap_test_dir1/test_restart 2023-10-14T18:14:18.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.713+0000 7f63c182d640 1 Processor -- start 2023-10-14T18:14:18.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.713+0000 7f63c182d640 1 -- start start 2023-10-14T18:14:18.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63c182d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63bc159e90 0x7f63bc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63c182d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc06c140 0x7f63bc06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63c182d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63bc070360 con 0x7f63bc159ab0 2023-10-14T18:14:18.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63c182d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63bc06cb10 con 0x7f63bc159e90 2023-10-14T18:14:18.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63c182d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc15d890 con 0x7f63bc06c140 2023-10-14T18:14:18.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63ba7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63bc159e90 0x7f63bc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63ba7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f63bc159e90 0x7f63bc0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42720/0 (socket says 172.21.15.29:42720) 2023-10-14T18:14:18.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.715+0000 7f63ba7fc640 1 -- 172.21.15.29:0/2697342294 learned_addr learned my addr 172.21.15.29:0/2697342294 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:18.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.716+0000 7f63bb7fe640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc06c140 0x7f63bc06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.716+0000 7f63ba7fc640 1 -- 172.21.15.29:0/2697342294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc06c140 msgr2=0x7f63bc06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.716+0000 7f63ba7fc640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc06c140 0x7f63bc06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.716+0000 7f63ba7fc640 1 -- 172.21.15.29:0/2697342294 >> v1:172.21.15.186:6789/0 conn(0x7f63bc159ab0 legacy=0x7f63bc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:18.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.716+0000 7f63ba7fc640 1 -- 172.21.15.29:0/2697342294 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc0cd1f0 con 0x7f63bc159e90 2023-10-14T18:14:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.717+0000 7f63ba7fc640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159e90 0x7f63bc0ce370 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5b752b7d67e223 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:18.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63b9ffb640 1 -- 172.21.15.29:0/2697342294 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0031740 con 0x7f63bc159e90 2023-10-14T18:14:18.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63b9ffb640 1 -- 172.21.15.29:0/2697342294 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b00318a0 con 0x7f63bc159e90 2023-10-14T18:14:18.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63b9ffb640 1 -- 172.21.15.29:0/2697342294 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0031c30 con 0x7f63bc159e90 2023-10-14T18:14:18.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 -- 172.21.15.29:0/2697342294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159e90 msgr2=0x7f63bc0ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159e90 0x7f63bc0ce370 secure :-1 s=READY pgs=2249 cs=0 l=1 rev1=1 crypto rx=0x7f63b0009870 tx=0x7f63b002b7e0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 -- 172.21.15.29:0/2697342294 shutdown_connections 2023-10-14T18:14:18.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc06c140 0x7f63bc06c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 --2- 172.21.15.29:0/2697342294 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159e90 0x7f63bc0ce370 unknown :-1 s=CLOSED pgs=2249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 -- 172.21.15.29:0/2697342294 >> 172.21.15.29:0/2697342294 conn(0x7f63bc0ce9b0 msgr2=0x7f63bc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:18.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.718+0000 7f63c182d640 1 -- 172.21.15.29:0/2697342294 shutdown_connections 2023-10-14T18:14:18.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.719+0000 7f63c182d640 1 -- 172.21.15.29:0/2697342294 wait complete. 2023-10-14T18:14:18.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.720+0000 7f63c182d640 1 Processor -- start 2023-10-14T18:14:18.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.720+0000 7f63c182d640 1 -- start start 2023-10-14T18:14:18.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63bc06c140 0x7f63bc158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc159e90 0x7f63bc0c46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0c90d0 con 0x7f63bc159ab0 2023-10-14T18:14:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0af9a0 con 0x7f63bc06c140 2023-10-14T18:14:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63c182d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63bc0c8c40 con 0x7f63bc159e90 2023-10-14T18:14:18.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.721+0000 7f63ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42736/0 (socket says 172.21.15.29:42736) 2023-10-14T18:14:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 -- 172.21.15.29:0/3997205975 learned_addr learned my addr 172.21.15.29:0/3997205975 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63baffd640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63bc06c140 0x7f63bc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63bb7fe640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc159e90 0x7f63bc0c46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 -- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc159e90 msgr2=0x7f63bc0c46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc159e90 0x7f63bc0c46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 -- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63bc06c140 msgr2=0x7f63bc158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63bc06c140 0x7f63bc158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 -- 172.21.15.29:0/3997205975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63b00094d0 con 0x7f63bc159ab0 2023-10-14T18:14:18.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.722+0000 7f63ba7fc640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f63b00090b0 tx=0x7f63b00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.723+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b0033c70 con 0x7f63bc159ab0 2023-10-14T18:14:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.723+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b0033df0 con 0x7f63bc159ab0 2023-10-14T18:14:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.723+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63bc0c5c20 con 0x7f63bc159ab0 2023-10-14T18:14:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.723+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63b003c640 con 0x7f63bc159ab0 2023-10-14T18:14:18.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.723+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63bc0c88d0 con 0x7f63bc159ab0 2023-10-14T18:14:18.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.724+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63bc0c6140 con 0x7f63bc159ab0 2023-10-14T18:14:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.725+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f63b003a040 con 0x7f63bc159ab0 2023-10-14T18:14:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.726+0000 7f63a3fff640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 0x7f63a806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.726+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f63b00ab630 con 0x7f63bc159ab0 2023-10-14T18:14:18.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.726+0000 7f63baffd640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 0x7f63a806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.727+0000 7f63baffd640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 0x7f63a806d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63ac006fd0 tx=0x7f63ac008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:18.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.730+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63b0076220 con 0x7f63bc159ab0 2023-10-14T18:14:18.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.940+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 --> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "/snap_test_dir1/test_restart", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f63bc0cd7b0 con 0x7f63a806afe0 2023-10-14T18:14:18.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.957+0000 7f63a3fff640 1 -- 172.21.15.29:0/3997205975 <== mgr.10398 v2:172.21.15.29:6832/1263890494 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+376 (secure 0 0 0) 0x7f63ac00bda0 con 0x7f63a806afe0 2023-10-14T18:14:18.958 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:18.959 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:58:41", "first": "2023-10-14T17:59:00", "last": "2023-10-14T18:14:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-14T18:14:18.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 msgr2=0x7f63a806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 0x7f63a806d4a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f63ac006fd0 tx=0x7f63ac008040 comp rx=0 tx=0).stop 2023-10-14T18:14:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 msgr2=0x7f63bc1589c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 secure :-1 s=READY pgs=2250 cs=0 l=1 rev1=1 crypto rx=0x7f63b00090b0 tx=0x7f63b00046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 shutdown_connections 2023-10-14T18:14:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f63a806afe0 0x7f63a806d4a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63bc159e90 0x7f63bc0c46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63bc159ab0 0x7f63bc1589c0 unknown :-1 s=CLOSED pgs=2250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 --2- 172.21.15.29:0/3997205975 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63bc06c140 0x7f63bc158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:18.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.960+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 >> 172.21.15.29:0/3997205975 conn(0x7f63bc0ce9b0 msgr2=0x7f63bc0b0ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:18.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.961+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 shutdown_connections 2023-10-14T18:14:18.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:18.961+0000 7f63c182d640 1 -- 172.21.15.29:0/3997205975 wait complete. 2023-10-14T18:14:18.979 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "cephfs", "subvol": null, "group": null, "path": "/snap_test_dir1/test_restart", "rel_path": "/snap_test_dir1/test_restart", "schedule": "1m", "retention": {}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T17:58:41", "first": "2023-10-14T17:59:00", "last": "2023-10-14T18:14:00", "last_pruned": null, "created_count": 16, "pruned_count": 0, "active": false}] 2023-10-14T18:14:18.979 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'fs': 'cephfs', 'subvol': None, 'group': None, 'path': '/snap_test_dir1/test_restart', 'rel_path': '/snap_test_dir1/test_restart', 'schedule': '1m', 'retention': {}, 'start': '2023-10-14T00:00:00', 'created': '2023-10-14T17:58:41', 'first': '2023-10-14T17:59:00', 'last': '2023-10-14T18:14:00', 'last_pruned': None, 'created_count': 16, 'pruned_count': 0, 'active': False} 2023-10-14T18:14:18.980 DEBUG:tasks.cephfs.test_snap_schedules:fs_count: 16 2023-10-14T18:14:18.980 DEBUG:tasks.cephfs.test_snap_schedules:db_count: 16 2023-10-14T18:14:18.980 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove /snap_test_dir1/test_restart 1m 2023-10-14T18:14:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.208+0000 7f5c03901640 1 Processor -- start 2023-10-14T18:14:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.208+0000 7f5c03901640 1 -- start start 2023-10-14T18:14:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c03901640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bfc15d150 0x7f5bfc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c03901640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc0ac890 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c03901640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bfc161650 con 0x7f5bfc160b90 2023-10-14T18:14:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c03901640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bfc15d530 con 0x7f5bfc15d150 2023-10-14T18:14:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c01e9e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5bfc06a4b0 0x7f5bfc0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:52148/0 (socket says 172.21.15.29:52148) 2023-10-14T18:14:19.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c01e9e640 1 -- 172.21.15.29:0/1883856843 learned_addr learned my addr 172.21.15.29:0/1883856843 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.210+0000 7f5c0169d640 1 --2- 172.21.15.29:0/1883856843 >> v2:172.21.15.186:3300/0 conn(0x7f5bfc15d150 0x7f5bfc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281850879 0 0) 0x7f5bfc0ac890 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5be8003520 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798046212 0 0) 0x7f5be8003520 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 >> v2:172.21.15.186:3300/0 conn(0x7f5bfc15d150 msgr2=0x7f5bfc0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 --2- 172.21.15.29:0/1883856843 >> v2:172.21.15.186:3300/0 conn(0x7f5bfc15d150 0x7f5bfc0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 >> v1:172.21.15.186:6789/0 conn(0x7f5bfc160b90 legacy=0x7f5bfc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc161930 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5bf40019f0 con 0x7f5bfc06a4b0 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 >> v1:172.21.15.29:6789/0 conn(0x7f5bfc06a4b0 legacy=0x7f5bfc0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5be80039c0 0x7f5be8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be80047c0 0x7f5be8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 0x7f5be800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc15d530 con 0x7f5be8005180 2023-10-14T18:14:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5be800c080 con 0x7f5be80039c0 2023-10-14T18:14:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.211+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5be800c240 con 0x7f5be80047c0 2023-10-14T18:14:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 0x7f5be800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c0169d640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5be80039c0 0x7f5be8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c00e9c640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be80047c0 0x7f5be8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 -- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be80047c0 msgr2=0x7f5be8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be80047c0 0x7f5be8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 -- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5be80039c0 msgr2=0x7f5be8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5be80039c0 0x7f5be8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.212+0000 7f5c01e9e640 1 -- 172.21.15.29:0/1883856843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5be8003520 con 0x7f5be8005180 2023-10-14T18:14:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.214+0000 7f5c01e9e640 1 -- 172.21.15.29:0/1883856843 reap_dead start 2023-10-14T18:14:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.214+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 0x7f5be800b980 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400a760 tx=0x7f5be80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40d7c979aa7ab2bb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.214+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4034630 con 0x7f5be8005180 2023-10-14T18:14:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.214+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf402f070 con 0x7f5be8005180 2023-10-14T18:14:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.214+0000 7f5bf3fff640 1 -- 172.21.15.29:0/1883856843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4034a10 con 0x7f5be8005180 2023-10-14T18:14:19.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 -- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 msgr2=0x7f5be800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 0x7f5be800b980 secure :-1 s=READY pgs=2252 cs=0 l=1 rev1=1 crypto rx=0x7f5bf400a760 tx=0x7f5be80057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 -- 172.21.15.29:0/1883856843 shutdown_connections 2023-10-14T18:14:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 --2- 172.21.15.29:0/1883856843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5be8005180 0x7f5be800b980 unknown :-1 s=CLOSED pgs=2252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 -- 172.21.15.29:0/1883856843 >> 172.21.15.29:0/1883856843 conn(0x7f5bfc0ce9b0 msgr2=0x7f5bfc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:19.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 -- 172.21.15.29:0/1883856843 shutdown_connections 2023-10-14T18:14:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.215+0000 7f5c03901640 1 -- 172.21.15.29:0/1883856843 wait complete. 2023-10-14T18:14:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.217+0000 7f5c03901640 1 Processor -- start 2023-10-14T18:14:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.217+0000 7f5c03901640 1 -- start start 2023-10-14T18:14:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be8005180 0x7f5bfc0ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bfc0c8220 0x7f5bfc0b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc15e1b0 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc156800 con 0x7f5bfc0c8220 2023-10-14T18:14:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.218+0000 7f5c03901640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bfc0afd80 con 0x7f5be8005180 2023-10-14T18:14:19.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42758/0 (socket says 172.21.15.29:42758) 2023-10-14T18:14:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 -- 172.21.15.29:0/576126716 learned_addr learned my addr 172.21.15.29:0/576126716 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c00e9c640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bfc0c8220 0x7f5bfc0b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c0169d640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be8005180 0x7f5bfc0ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 -- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be8005180 msgr2=0x7f5bfc0ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be8005180 0x7f5bfc0ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 -- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bfc0c8220 msgr2=0x7f5bfc0b0240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bfc0c8220 0x7f5bfc0b0240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 -- 172.21.15.29:0/576126716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bfc0b0780 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.219+0000 7f5c01e9e640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4002230 tx=0x7f5bf4001cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.220+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf40019f0 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.220+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bf40345f0 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.220+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bf4004da0 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.220+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bfc0c86a0 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.220+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bfc0ae820 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.221+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bfc15df10 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.222+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f5bf4031030 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.223+0000 7f5bf1ffb640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 0x7f5be006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.223+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f5bf40ab100 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.223+0000 7f5c0169d640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 0x7f5be006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.224+0000 7f5c0169d640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 0x7f5be006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009990 tx=0x7f5bf8008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:19.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.228+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bfc15df10 con 0x7f5bfc0b33d0 2023-10-14T18:14:19.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.449+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 --> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "/snap_test_dir1/test_restart", "repeat": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bfc0b3d90 con 0x7f5be006afe0 2023-10-14T18:14:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.653+0000 7f5bf1ffb640 1 -- 172.21.15.29:0/576126716 <== mgr.10398 v2:172.21.15.29:6832/1263890494 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7f5bfc0b3d90 con 0x7f5be006afe0 2023-10-14T18:14:19.655 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path /snap_test_dir1/test_restart 2023-10-14T18:14:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 msgr2=0x7f5be006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 0x7f5be006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f5bf8009990 tx=0x7f5bf8008040 comp rx=0 tx=0).stop 2023-10-14T18:14:19.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 msgr2=0x7f5bfc0b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 secure :-1 s=READY pgs=2253 cs=0 l=1 rev1=1 crypto rx=0x7f5bf4002230 tx=0x7f5bf4001cb0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 shutdown_connections 2023-10-14T18:14:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f5be006afe0 0x7f5be006d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bfc0b33d0 0x7f5bfc0b3850 unknown :-1 s=CLOSED pgs=2253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bfc0c8220 0x7f5bfc0b0240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 --2- 172.21.15.29:0/576126716 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5be8005180 0x7f5bfc0ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.657+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 >> 172.21.15.29:0/576126716 conn(0x7f5bfc0ce9b0 msgr2=0x7f5bfc0c8fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.658+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 shutdown_connections 2023-10-14T18:14:19.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:19.658+0000 7f5c03901640 1 -- 172.21.15.29:0/576126716 wait complete. 2023-10-14T18:14:19.678 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path /snap_test_dir1/test_restart 2023-10-14T18:14:19.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:19.679 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/test_restart/.snap) 2023-10-14T18:14:19.735 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-17_59_00_UTC 2023-10-14T18:14:19.735 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_00_00_UTC 2023-10-14T18:14:19.736 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_01_00_UTC 2023-10-14T18:14:19.736 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_02_00_UTC 2023-10-14T18:14:19.736 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_03_00_UTC 2023-10-14T18:14:19.737 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_04_00_UTC 2023-10-14T18:14:19.737 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_05_00_UTC 2023-10-14T18:14:19.737 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_06_00_UTC 2023-10-14T18:14:19.738 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_07_00_UTC 2023-10-14T18:14:19.738 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_08_00_UTC 2023-10-14T18:14:19.741 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_09_00_UTC 2023-10-14T18:14:19.741 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_10_00_UTC 2023-10-14T18:14:19.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_11_00_UTC 2023-10-14T18:14:19.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_12_00_UTC 2023-10-14T18:14:19.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_13_00_UTC 2023-10-14T18:14:19.743 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_14_00_UTC 2023-10-14T18:14:19.745 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-17_59_00_UTC 2023-10-14T18:14:19.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:19.748 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-17_59_00_UTC) 2023-10-14T18:14:19.821 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_00_00_UTC 2023-10-14T18:14:19.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:19.822 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_00_00_UTC) 2023-10-14T18:14:19.899 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_01_00_UTC 2023-10-14T18:14:19.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:19.900 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_01_00_UTC) 2023-10-14T18:14:19.975 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_02_00_UTC 2023-10-14T18:14:19.975 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:19.976 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_02_00_UTC) 2023-10-14T18:14:20.049 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_03_00_UTC 2023-10-14T18:14:20.050 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.050 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_03_00_UTC) 2023-10-14T18:14:20.123 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_04_00_UTC 2023-10-14T18:14:20.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.124 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_04_00_UTC) 2023-10-14T18:14:20.199 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_05_00_UTC 2023-10-14T18:14:20.199 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.200 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_05_00_UTC) 2023-10-14T18:14:20.274 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_06_00_UTC 2023-10-14T18:14:20.275 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.275 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_06_00_UTC) 2023-10-14T18:14:20.349 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_07_00_UTC 2023-10-14T18:14:20.350 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.350 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_07_00_UTC) 2023-10-14T18:14:20.424 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_08_00_UTC 2023-10-14T18:14:20.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.425 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_08_00_UTC) 2023-10-14T18:14:20.502 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_09_00_UTC 2023-10-14T18:14:20.502 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.503 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_09_00_UTC) 2023-10-14T18:14:20.581 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_10_00_UTC 2023-10-14T18:14:20.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.582 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_10_00_UTC) 2023-10-14T18:14:20.659 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_11_00_UTC 2023-10-14T18:14:20.659 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.659 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_11_00_UTC) 2023-10-14T18:14:20.732 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_12_00_UTC 2023-10-14T18:14:20.733 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.733 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_12_00_UTC) 2023-10-14T18:14:20.807 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_13_00_UTC 2023-10-14T18:14:20.808 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.808 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_13_00_UTC) 2023-10-14T18:14:20.885 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_14_00_UTC 2023-10-14T18:14:20.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.886 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart/.snap/scheduled-2023-10-14-18_14_00_UTC) 2023-10-14T18:14:20.960 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:20.960 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/test_restart) 2023-10-14T18:14:21.024 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T18:14:21.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.257+0000 7f01162bb640 1 Processor -- start 2023-10-14T18:14:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.258+0000 7f01162bb640 1 -- start start 2023-10-14T18:14:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.259+0000 7f01162bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01100caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.259+0000 7f01162bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01100c9000 0x7f0110069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.259+0000 7f01162bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 0x7f01100cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f01162bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0110161a50 con 0x7f01100c9000 2023-10-14T18:14:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f01162bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011006a750 con 0x7f011006a2e0 2023-10-14T18:14:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f01162bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01100cb140 con 0x7f01100addb0 2023-10-14T18:14:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01100c9000 0x7f0110069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f01100c9000 0x7f0110069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43348/0 (socket says 172.21.15.29:43348) 2023-10-14T18:14:21.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 -- 172.21.15.29:0/1438514449 learned_addr learned my addr 172.21.15.29:0/1438514449 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0114858640 1 --2- 172.21.15.29:0/1438514449 >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 0x7f01100cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 -- 172.21.15.29:0/1438514449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 msgr2=0x7f01100caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 --2- 172.21.15.29:0/1438514449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01100caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 -- 172.21.15.29:0/1438514449 >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 msgr2=0x7f01100cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 --2- 172.21.15.29:0/1438514449 >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 0x7f01100cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0107fff640 1 -- 172.21.15.29:0/1438514449 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f011006b030 con 0x7f01100c9000 2023-10-14T18:14:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.260+0000 7f0114858640 1 --2- 172.21.15.29:0/1438514449 >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 0x7f01100cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.262+0000 7f0107fff640 1 --2- 172.21.15.29:0/1438514449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110069da0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f01100cf850 tx=0x7f00f802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2785bec45d481c7e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:21.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.262+0000 7f010f7fe640 1 -- 172.21.15.29:0/1438514449 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8031640 con 0x7f01100c9000 2023-10-14T18:14:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.262+0000 7f010f7fe640 1 -- 172.21.15.29:0/1438514449 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f80317a0 con 0x7f01100c9000 2023-10-14T18:14:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.262+0000 7f010f7fe640 1 -- 172.21.15.29:0/1438514449 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00f8031b30 con 0x7f01100c9000 2023-10-14T18:14:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 -- 172.21.15.29:0/1438514449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 msgr2=0x7f0110069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 --2- 172.21.15.29:0/1438514449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110069da0 secure :-1 s=READY pgs=2254 cs=0 l=1 rev1=1 crypto rx=0x7f01100cf850 tx=0x7f00f802b8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 -- 172.21.15.29:0/1438514449 shutdown_connections 2023-10-14T18:14:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 --2- 172.21.15.29:0/1438514449 >> v2:172.21.15.186:3300/0 conn(0x7f011006a2e0 0x7f01100cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 --2- 172.21.15.29:0/1438514449 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110069da0 unknown :-1 s=CLOSED pgs=2254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 --2- 172.21.15.29:0/1438514449 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01100caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 -- 172.21.15.29:0/1438514449 >> 172.21.15.29:0/1438514449 conn(0x7f01100ce9b0 msgr2=0x7f01100cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:21.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 -- 172.21.15.29:0/1438514449 shutdown_connections 2023-10-14T18:14:21.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.263+0000 7f01162bb640 1 -- 172.21.15.29:0/1438514449 wait complete. 2023-10-14T18:14:21.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.265+0000 7f01162bb640 1 Processor -- start 2023-10-14T18:14:21.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.265+0000 7f01162bb640 1 -- start start 2023-10-14T18:14:21.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 0x7f01100b38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01101529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01101627c0 con 0x7f01100c9000 2023-10-14T18:14:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110161bd0 con 0x7f011006a2e0 2023-10-14T18:14:21.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f01162bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0110153450 con 0x7f01100addb0 2023-10-14T18:14:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f0114858640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f0114858640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43362/0 (socket says 172.21.15.29:43362) 2023-10-14T18:14:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f0114858640 1 -- 172.21.15.29:0/922598583 learned_addr learned my addr 172.21.15.29:0/922598583 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.266+0000 7f010ffff640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 0x7f01100b38f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0107fff640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01101529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 -- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 msgr2=0x7f01101529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01101529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 -- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 msgr2=0x7f01100b38f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 0x7f01100b38f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:21.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 -- 172.21.15.29:0/922598583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f80094d0 con 0x7f01100c9000 2023-10-14T18:14:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f010ffff640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 0x7f01100b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f0114858640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f010800cef0 tx=0x7f010800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800d840 con 0x7f01100c9000 2023-10-14T18:14:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010800d9c0 con 0x7f01100c9000 2023-10-14T18:14:21.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0110161e20 con 0x7f01100c9000 2023-10-14T18:14:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800dd70 con 0x7f01100c9000 2023-10-14T18:14:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.267+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01101623e0 con 0x7f01100c9000 2023-10-14T18:14:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.268+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0110162900 con 0x7f01100c9000 2023-10-14T18:14:21.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.269+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 117) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f010801e3b0 con 0x7f01100c9000 2023-10-14T18:14:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.270+0000 7f010d7fa640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 0x7f00f006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.270+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f010808a660 con 0x7f01100c9000 2023-10-14T18:14:21.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.271+0000 7f010ffff640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 0x7f00f006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:21.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.271+0000 7f010ffff640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 0x7f00f006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0100006fd0 tx=0x7f0100008040 comp rx=0 tx=0).ready entity=mgr.10398 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:21.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.275+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0110162900 con 0x7f01100c9000 2023-10-14T18:14:21.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:21.524+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f0110162900 con 0x7f01100c9000 2023-10-14T18:14:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.369+0000 7f010d7fa640 1 -- 172.21.15.29:0/922598583 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v118) v1 ==== 93+0+0 (secure 0 0 0) 0x7f0110162900 con 0x7f01100c9000 2023-10-14T18:14:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.372+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 msgr2=0x7f00f006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.372+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 0x7f00f006d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0100006fd0 tx=0x7f0100008040 comp rx=0 tx=0).stop 2023-10-14T18:14:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 msgr2=0x7f0110152f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 secure :-1 s=READY pgs=2255 cs=0 l=1 rev1=1 crypto rx=0x7f010800cef0 tx=0x7f010800a710 comp rx=0 tx=0).stop 2023-10-14T18:14:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 shutdown_connections 2023-10-14T18:14:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f00f006afe0 0x7f00f006d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f01100c9000 0x7f0110152f10 unknown :-1 s=CLOSED pgs=2255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f01100addb0 0x7f01101529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 --2- 172.21.15.29:0/922598583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f011006a2e0 0x7f01100b38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.373+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 >> 172.21.15.29:0/922598583 conn(0x7f01100ce9b0 msgr2=0x7f01100af120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.374+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 shutdown_connections 2023-10-14T18:14:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.375+0000 7f01162bb640 1 -- 172.21.15.29:0/922598583 wait complete. 2023-10-14T18:14:22.400 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:14:22.401 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:14:22.517 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:14:22.613 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:14:22.614 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:22.614 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:22.669 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:22.669 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:22.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:22.883 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:22.883+0000 7f1334c3b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:14:22.927 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:22.926+0000 7f7b9842b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:14:22.937 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:22.937+0000 7f1334c3b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:14:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.976+0000 7f47ec133640 1 Processor -- start 2023-10-14T18:14:22.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.976+0000 7f47ec133640 1 -- start start 2023-10-14T18:14:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ec133640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4070a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ec133640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47e40ae5e0 0x7f47e40aea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ec133640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e4070f70 con 0x7f47e4159d10 2023-10-14T18:14:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ec133640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47e406e4b0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ec133640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e406e8d0 con 0x7f47e40705d0 2023-10-14T18:14:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ea6d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47e40ae5e0 0x7f47e40aea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ea6d0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47e40ae5e0 0x7f47e40aea50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43384/0 (socket says 172.21.15.29:43384) 2023-10-14T18:14:22.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47ea6d0640 1 -- 172.21.15.29:0/3628598554 learned_addr learned my addr 172.21.15.29:0/3628598554 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.978+0000 7f47e96ce640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4070a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.979+0000 7f47ea6d0640 1 -- 172.21.15.29:0/3628598554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 msgr2=0x7f47e4070a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.979+0000 7f47ea6d0640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4070a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.979+0000 7f47ea6d0640 1 -- 172.21.15.29:0/3628598554 >> v1:172.21.15.186:6789/0 conn(0x7f47e4159d10 legacy=0x7f47e40ce380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:22.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.979+0000 7f47ea6d0640 1 -- 172.21.15.29:0/3628598554 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e406d830 con 0x7f47e40ae5e0 2023-10-14T18:14:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.980+0000 7f47ea6d0640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40aea50 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00d150 tx=0x7f47dc02e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c1e69d8b0f70f69 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.980+0000 7f47e8ecd640 1 -- 172.21.15.29:0/3628598554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc02e9e0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.980+0000 7f47e8ecd640 1 -- 172.21.15.29:0/3628598554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47dc02eb40 con 0x7f47e40ae5e0 2023-10-14T18:14:22.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.980+0000 7f47e8ecd640 1 -- 172.21.15.29:0/3628598554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47dc009600 con 0x7f47e40ae5e0 2023-10-14T18:14:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 -- 172.21.15.29:0/3628598554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 msgr2=0x7f47e40aea50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:22.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40aea50 secure :-1 s=READY pgs=2260 cs=0 l=1 rev1=1 crypto rx=0x7f47dc00d150 tx=0x7f47dc02e040 comp rx=0 tx=0).stop 2023-10-14T18:14:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 -- 172.21.15.29:0/3628598554 shutdown_connections 2023-10-14T18:14:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40aea50 unknown :-1 s=CLOSED pgs=2260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 --2- 172.21.15.29:0/3628598554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4070a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 -- 172.21.15.29:0/3628598554 >> 172.21.15.29:0/3628598554 conn(0x7f47e40b1560 msgr2=0x7f47e40b1970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:22.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 -- 172.21.15.29:0/3628598554 shutdown_connections 2023-10-14T18:14:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.981+0000 7f47ec133640 1 -- 172.21.15.29:0/3628598554 wait complete. 2023-10-14T18:14:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.983+0000 7f47ec133640 1 Processor -- start 2023-10-14T18:14:22.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.983+0000 7f47ec133640 1 -- start start 2023-10-14T18:14:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47e4159d10 0x7f47e40adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e415afe0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47e415b280 con 0x7f47e4159d10 2023-10-14T18:14:22.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f47ec133640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47e415b560 con 0x7f47e40705d0 2023-10-14T18:14:22.988 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:22.984+0000 7f7b9842b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:14:22.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43386/0 (socket says 172.21.15.29:43386) 2023-10-14T18:14:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e9ecf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 -- 172.21.15.29:0/525673306 learned_addr learned my addr 172.21.15.29:0/525673306 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:22.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47ea6d0640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47e4159d10 0x7f47e40adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 msgr2=0x7f47e4159130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4159130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47e4159d10 msgr2=0x7f47e40adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47e4159d10 0x7f47e40adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:22.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 -- 172.21.15.29:0/525673306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47dc00bd40 con 0x7f47e40ae5e0 2023-10-14T18:14:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.985+0000 7f47e96ce640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d5f0 tx=0x7f47d400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.986+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d40125f0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.986+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47d4012750 con 0x7f47e40ae5e0 2023-10-14T18:14:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.986+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47d4012b00 con 0x7f47e40ae5e0 2023-10-14T18:14:22.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.986+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e415b840 con 0x7f47e40ae5e0 2023-10-14T18:14:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.986+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e40c81c0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.987+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e40714d0 con 0x7f47e40ae5e0 2023-10-14T18:14:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.989+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f47d4012c60 con 0x7f47e40ae5e0 2023-10-14T18:14:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.990+0000 7f47da7fc640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 0x7f47c806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.990+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f47d408a960 con 0x7f47e40ae5e0 2023-10-14T18:14:22.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.990+0000 7f47e9ecf640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 msgr2=0x7f47c806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.990+0000 7f47e9ecf640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 0x7f47c806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:22.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:22.993+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47d4055640 con 0x7f47e40ae5e0 2023-10-14T18:14:23.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.189+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47e404e160 con 0x7f47e40ae5e0 2023-10-14T18:14:23.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.191+0000 7f47da7fc640 1 -- 172.21.15.29:0/525673306 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v194) v1 ==== 74+0+17738 (secure 0 0 0) 0x7f47d4058c60 con 0x7f47e40ae5e0 2023-10-14T18:14:23.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:23.193 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":194,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:09:22.121349+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"192","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/614546910":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2776961501":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/287831498":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/841868971":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1377837433":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3575832370":"2023-10-15T18:09:22.121284+0000","172.21.15.29:6833/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2946416628":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/730049499":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6833/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3534626168":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/1930442848":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2824949307":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2461249058":"2023-10-15T17:58:40.567922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.192+0000 7f47e9ecf640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 msgr2=0x7f47c806d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:23.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.192+0000 7f47e9ecf640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 0x7f47c806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 msgr2=0x7f47c806d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 0x7f47c806d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 msgr2=0x7f47e40ad9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 secure :-1 s=READY pgs=2261 cs=0 l=1 rev1=1 crypto rx=0x7f47d400d5f0 tx=0x7f47d400dab0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 shutdown_connections 2023-10-14T18:14:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f47c806afc0 0x7f47c806d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47e4159d10 0x7f47e40adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47e40ae5e0 0x7f47e40ad9c0 unknown :-1 s=CLOSED pgs=2261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 --2- 172.21.15.29:0/525673306 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47e40705d0 0x7f47e4159130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 >> 172.21.15.29:0/525673306 conn(0x7f47e40b1560 msgr2=0x7f47e40af480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 shutdown_connections 2023-10-14T18:14:23.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.194+0000 7f47ec133640 1 -- 172.21.15.29:0/525673306 wait complete. 2023-10-14T18:14:23.215 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:58:23.469364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:58:24.568568+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '192', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:14:23.216 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:14:23.386 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.386+0000 7f1334c3b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:14:23.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.428+0000 7fb350689640 1 Processor -- start 2023-10-14T18:14:23.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.428+0000 7fb350689640 1 -- start start 2023-10-14T18:14:23.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.430+0000 7fb350689640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb3480d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.430+0000 7fb350689640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb3480cd260 0x7fb3480cd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.430+0000 7fb350689640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3480b5d90 con 0x7fb34815d030 2023-10-14T18:14:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.430+0000 7fb350689640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3480d2b50 con 0x7fb3480cd260 2023-10-14T18:14:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.430+0000 7fb350689640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb348160fc0 con 0x7fb34815d410 2023-10-14T18:14:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34e425640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb34815d030 0x7fb348161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40378/0 (socket says 172.21.15.29:40378) 2023-10-14T18:14:23.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34e425640 1 -- 172.21.15.29:0/3736731815 learned_addr learned my addr 172.21.15.29:0/3736731815 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:23.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34dc24640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb3480d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34ec26640 1 --2- 172.21.15.29:0/3736731815 >> v2:172.21.15.186:3300/0 conn(0x7fb3480cd260 0x7fb3480cd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3750840381 0 0) 0x7fb3480b5d90 con 0x7fb34815d030 2023-10-14T18:14:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb334003520 con 0x7fb34815d030 2023-10-14T18:14:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4049245307 0 0) 0x7fb334003520 con 0x7fb34815d030 2023-10-14T18:14:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 msgr2=0x7fb3480d1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:23.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb3480d1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 >> v2:172.21.15.186:3300/0 conn(0x7fb3480cd260 msgr2=0x7fb3480cd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 --2- 172.21.15.29:0/3736731815 >> v2:172.21.15.186:3300/0 conn(0x7fb3480cd260 0x7fb3480cd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.431+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3480d25d0 con 0x7fb34815d030 2023-10-14T18:14:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb344001670 con 0x7fb34815d030 2023-10-14T18:14:23.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 >> v1:172.21.15.29:6789/0 conn(0x7fb34815d030 legacy=0x7fb348161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb334003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334003f40 0x7fb3340047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb334004d00 0x7fb334005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb348160fc0 con 0x7fb334003f40 2023-10-14T18:14:23.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400b980 con 0x7fb334004d00 2023-10-14T18:14:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb33400bb40 con 0x7fb34815d410 2023-10-14T18:14:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334003f40 0x7fb3340047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34e425640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb334003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 -- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 msgr2=0x7fb334003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:23.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb34815d410 0x7fb334003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 -- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb334004d00 msgr2=0x7fb334005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb334004d00 0x7fb334005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.432+0000 7fb34dc24640 1 -- 172.21.15.29:0/3736731815 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334003520 con 0x7fb334003f40 2023-10-14T18:14:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.433+0000 7fb34dc24640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334003f40 0x7fb3340047c0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fb338009ac0 tx=0x7fb3340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77f9606dca6fed1b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:23.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33802c6b0 con 0x7fb334003f40 2023-10-14T18:14:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb33802c810 con 0x7fb334003f40 2023-10-14T18:14:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb34d423640 1 -- 172.21.15.29:0/3736731815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb33802cba0 con 0x7fb334003f40 2023-10-14T18:14:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb350689640 1 -- 172.21.15.29:0/3736731815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334003f40 msgr2=0x7fb3340047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb350689640 1 --2- 172.21.15.29:0/3736731815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb334003f40 0x7fb3340047c0 secure :-1 s=READY pgs=2263 cs=0 l=1 rev1=1 crypto rx=0x7fb338009ac0 tx=0x7fb3340057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb34ec26640 1 -- 172.21.15.29:0/3736731815 reap_dead start 2023-10-14T18:14:23.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb350689640 1 -- 172.21.15.29:0/3736731815 shutdown_connections 2023-10-14T18:14:23.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb350689640 1 -- 172.21.15.29:0/3736731815 >> 172.21.15.29:0/3736731815 conn(0x7fb3480cf720 msgr2=0x7fb3480cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:23.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.434+0000 7fb350689640 1 -- 172.21.15.29:0/3736731815 shutdown_connections 2023-10-14T18:14:23.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.435+0000 7fb350689640 1 -- 172.21.15.29:0/3736731815 wait complete. 2023-10-14T18:14:23.443 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.440+0000 7f1334c3b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:14:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.436+0000 7fb350689640 1 Processor -- start 2023-10-14T18:14:23.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.436+0000 7fb350689640 1 -- start start 2023-10-14T18:14:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 0x7fb34815d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3481597e0 0x7fb348159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb348145ba0 con 0x7fb34815a620 2023-10-14T18:14:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb348160590 con 0x7fb3481597e0 2023-10-14T18:14:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.437+0000 7fb350689640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb348145df0 con 0x7fb334003f40 2023-10-14T18:14:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43410/0 (socket says 172.21.15.29:43410) 2023-10-14T18:14:23.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 -- 172.21.15.29:0/1589715668 learned_addr learned my addr 172.21.15.29:0/1589715668 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34e425640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 0x7fb34815d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34ec26640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3481597e0 0x7fb348159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 msgr2=0x7fb34815d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 0x7fb34815d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3481597e0 msgr2=0x7fb348159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3481597e0 0x7fb348159c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 -- 172.21.15.29:0/1589715668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3481460d0 con 0x7fb34815a620 2023-10-14T18:14:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34e425640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 0x7fb34815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.438+0000 7fb34dc24640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fb33802c250 tx=0x7fb338004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:23.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.439+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338003810 con 0x7fb34815a620 2023-10-14T18:14:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.439+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb338003970 con 0x7fb34815a620 2023-10-14T18:14:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.439+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb338003ce0 con 0x7fb34815a620 2023-10-14T18:14:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.439+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3481463b0 con 0x7fb34815a620 2023-10-14T18:14:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.439+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34815f150 con 0x7fb34815a620 2023-10-14T18:14:23.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.440+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34815ac80 con 0x7fb34815a620 2023-10-14T18:14:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.441+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb338034440 con 0x7fb34815a620 2023-10-14T18:14:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.442+0000 7fb33effd640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 0x7fb32c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.442+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7fb3380abca0 con 0x7fb34815a620 2023-10-14T18:14:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.442+0000 7fb34e425640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 msgr2=0x7fb32c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:23.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.442+0000 7fb34e425640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 0x7fb32c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:23.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.446+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb338076ad0 con 0x7fb34815a620 2023-10-14T18:14:23.467 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.466+0000 7f7b9842b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:14:23.491 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.491+0000 7f1334c3b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:14:23.522 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.521+0000 7f7b9842b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:14:23.551 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.551+0000 7f1334c3b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:14:23.575 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.574+0000 7f7b9842b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:14:23.600 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.599+0000 7f1334c3b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:14:23.638 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.637+0000 7f7b9842b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:14:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.643+0000 7fb34e425640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 msgr2=0x7fb32c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:23.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.643+0000 7fb34e425640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 0x7fb32c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:23.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.644+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb3480cfb00 con 0x7fb34815a620 2023-10-14T18:14:23.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.645+0000 7fb33effd640 1 -- 172.21.15.29:0/1589715668 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v194) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb33804c0a0 con 0x7fb34815a620 2023-10-14T18:14:23.646 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:14:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 msgr2=0x7fb32c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 0x7fb32c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 msgr2=0x7fb3481592a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 secure :-1 s=READY pgs=2264 cs=0 l=1 rev1=1 crypto rx=0x7fb33802c250 tx=0x7fb338004730 comp rx=0 tx=0).stop 2023-10-14T18:14:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 shutdown_connections 2023-10-14T18:14:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fb32c06af90 0x7fb32c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb3481597e0 0x7fb348159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb34815a620 0x7fb3481592a0 unknown :-1 s=CLOSED pgs=2264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 --2- 172.21.15.29:0/1589715668 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb334003f40 0x7fb34815d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 >> 172.21.15.29:0/1589715668 conn(0x7fb3480cf720 msgr2=0x7fb3480b74a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 shutdown_connections 2023-10-14T18:14:23.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.648+0000 7fb350689640 1 -- 172.21.15.29:0/1589715668 wait complete. 2023-10-14T18:14:23.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:14:23.693 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.692+0000 7f7b9842b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:14:23.710 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.709+0000 7f1334c3b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:14:23.772 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.772+0000 7f1334c3b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:23.813 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.813+0000 7f7b9842b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:14:23.820 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.820+0000 7f1334c3b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:14:23.878 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.877+0000 7f1334c3b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:14:23.881 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.881+0000 7f7b9842b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.884+0000 7f440b59e640 1 Processor -- start 2023-10-14T18:14:23.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.884+0000 7f440b59e640 1 -- start start 2023-10-14T18:14:23.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f440c15cd70 0x7f440c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 0x7f440c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f440c15d150 con 0x7f440c0b8480 2023-10-14T18:14:23.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440c0d1180 con 0x7f440c15cd70 2023-10-14T18:14:23.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f440c0d1460 con 0x7f440c0cded0 2023-10-14T18:14:23.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f4409d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f440c0b8480 0x7f440c0cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40394/0 (socket says 172.21.15.29:40394) 2023-10-14T18:14:23.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f4409d9b640 1 -- 172.21.15.29:0/1394662751 learned_addr learned my addr 172.21.15.29:0/1394662751 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:23.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.886+0000 7f440a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f440c15cd70 0x7f440c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440ad9d640 1 --2- 172.21.15.29:0/1394662751 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 0x7f440c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440959a640 1 -- 172.21.15.29:0/1394662751 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3425869647 0 0) 0x7f440c15d150 con 0x7f440c0b8480 2023-10-14T18:14:23.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440959a640 1 -- 172.21.15.29:0/1394662751 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43f4003520 con 0x7f440c0b8480 2023-10-14T18:14:23.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440a59c640 1 -- 172.21.15.29:0/1394662751 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 msgr2=0x7f440c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:23.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440a59c640 1 --2- 172.21.15.29:0/1394662751 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 0x7f440c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440a59c640 1 -- 172.21.15.29:0/1394662751 >> v1:172.21.15.29:6789/0 conn(0x7f440c0b8480 legacy=0x7f440c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440a59c640 1 -- 172.21.15.29:0/1394662751 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f440c0d1740 con 0x7f440c15cd70 2023-10-14T18:14:23.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.887+0000 7f440ad9d640 1 --2- 172.21.15.29:0/1394662751 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 0x7f440c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:23.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.888+0000 7f440a59c640 1 --2- 172.21.15.29:0/1394662751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c15cd70 0x7f440c0d2220 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f440c0b8910 tx=0x7f43fc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10fa40eef7916527 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:23.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440959a640 1 -- 172.21.15.29:0/1394662751 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc031830 con 0x7f440c15cd70 2023-10-14T18:14:23.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440959a640 1 -- 172.21.15.29:0/1394662751 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43fc031990 con 0x7f440c15cd70 2023-10-14T18:14:23.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440959a640 1 -- 172.21.15.29:0/1394662751 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f43fc031d00 con 0x7f440c15cd70 2023-10-14T18:14:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 -- 172.21.15.29:0/1394662751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c15cd70 msgr2=0x7f440c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 --2- 172.21.15.29:0/1394662751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c15cd70 0x7f440c0d2220 secure :-1 s=READY pgs=2266 cs=0 l=1 rev1=1 crypto rx=0x7f440c0b8910 tx=0x7f43fc02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 -- 172.21.15.29:0/1394662751 shutdown_connections 2023-10-14T18:14:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 --2- 172.21.15.29:0/1394662751 >> v2:172.21.15.186:3300/0 conn(0x7f440c0cded0 0x7f440c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 --2- 172.21.15.29:0/1394662751 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c15cd70 0x7f440c0d2220 unknown :-1 s=CLOSED pgs=2266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 -- 172.21.15.29:0/1394662751 >> 172.21.15.29:0/1394662751 conn(0x7f440c0cf570 msgr2=0x7f440c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 -- 172.21.15.29:0/1394662751 shutdown_connections 2023-10-14T18:14:23.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.889+0000 7f440b59e640 1 -- 172.21.15.29:0/1394662751 wait complete. 2023-10-14T18:14:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.891+0000 7f440b59e640 1 Processor -- start 2023-10-14T18:14:23.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.891+0000 7f440b59e640 1 -- start start 2023-10-14T18:14:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.892+0000 7f440b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c0b8480 0x7f440c0cbed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.892+0000 7f440b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c15cd70 0x7f440c15eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c0c9c80 con 0x7f440c0cded0 2023-10-14T18:14:23.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c15f9f0 con 0x7f440c0b8480 2023-10-14T18:14:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f440c15fcd0 con 0x7f440c15cd70 2023-10-14T18:14:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43414/0 (socket says 172.21.15.29:43414) 2023-10-14T18:14:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 -- 172.21.15.29:0/3793987660 learned_addr learned my addr 172.21.15.29:0/3793987660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:23.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440a59c640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c0b8480 0x7f440c0cbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f440ad9d640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c15cd70 0x7f440c15eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:23.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c15cd70 msgr2=0x7f440c15eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c15cd70 0x7f440c15eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c0b8480 msgr2=0x7f440c0cbed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:23.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c0b8480 0x7f440c0cbed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.893+0000 7f4409d9b640 1 -- 172.21.15.29:0/3793987660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc009400 con 0x7f440c0cded0 2023-10-14T18:14:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f4409d9b640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f440000c430 tx=0x7f440000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4400005ce0 con 0x7f440c0cded0 2023-10-14T18:14:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4400005e40 con 0x7f440c0cded0 2023-10-14T18:14:23.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44000072f0 con 0x7f440c0cded0 2023-10-14T18:14:23.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c15ffb0 con 0x7f440c0cded0 2023-10-14T18:14:23.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.894+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c160290 con 0x7f440c0cded0 2023-10-14T18:14:23.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.895+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c0c9e30 con 0x7f440c0cded0 2023-10-14T18:14:23.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.897+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f4400007450 con 0x7f440c0cded0 2023-10-14T18:14:23.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.897+0000 7f43faffd640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 0x7f43ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:23.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.897+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f4400089280 con 0x7f440c0cded0 2023-10-14T18:14:23.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.898+0000 7f440a59c640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 msgr2=0x7f43ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:23.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.898+0000 7f440a59c640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 0x7f43ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:23.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:23.902+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f440c0c9e30 con 0x7f440c0cded0 2023-10-14T18:14:23.929 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:23.929+0000 7f1334c3b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:14:23.936 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.935+0000 7f7b9842b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:14:23.996 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:23.995+0000 7f7b9842b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:14:24.049 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.048+0000 7f7b9842b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:14:24.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.098+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f440c04e9b0 con 0x7f440c0cded0 2023-10-14T18:14:24.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.098+0000 7f43faffd640 1 -- 172.21.15.29:0/3793987660 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v194) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4400017020 con 0x7f440c0cded0 2023-10-14T18:14:24.100 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:14:24.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.099+0000 7f440a59c640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 msgr2=0x7f43ec06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:24.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.099+0000 7f440a59c640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 0x7f43ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 msgr2=0x7f43ec06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 0x7f43ec06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 msgr2=0x7f440c0cc410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 secure :-1 s=READY pgs=2267 cs=0 l=1 rev1=1 crypto rx=0x7f440000c430 tx=0x7f440000c8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 shutdown_connections 2023-10-14T18:14:24.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f43ec06af90 0x7f43ec06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f440c15cd70 0x7f440c15eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f440c0cded0 0x7f440c0cc410 unknown :-1 s=CLOSED pgs=2267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 --2- 172.21.15.29:0/3793987660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f440c0b8480 0x7f440c0cbed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 >> 172.21.15.29:0/3793987660 conn(0x7f440c0cf570 msgr2=0x7f440c0b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 shutdown_connections 2023-10-14T18:14:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.101+0000 7f440b59e640 1 -- 172.21.15.29:0/3793987660 wait complete. 2023-10-14T18:14:24.119 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:14:24.309 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.308+0000 7f1334c3b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:14:24.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.373+0000 7f9df4c0e640 1 Processor -- start 2023-10-14T18:14:24.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.373+0000 7f9df4c0e640 1 -- start start 2023-10-14T18:14:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.374+0000 7f9df4c0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9df015d330 0x7f9df01618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.374+0000 7f9df4c0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00d0890 con 0x7f9df006c7d0 2023-10-14T18:14:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.374+0000 7f9df4c0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9df00d05b0 con 0x7f9df00661c0 2023-10-14T18:14:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.374+0000 7f9df4c0e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9df006b1e0 con 0x7f9df015d330 2023-10-14T18:14:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.375+0000 7f9de7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9df006c7d0 0x7f9df0065ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40410/0 (socket says 172.21.15.29:40410) 2023-10-14T18:14:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.375+0000 7f9de7fff640 1 -- 172.21.15.29:0/1736353683 learned_addr learned my addr 172.21.15.29:0/1736353683 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.375+0000 7f9dee59c640 1 --2- 172.21.15.29:0/1736353683 >> v2:172.21.15.186:3300/0 conn(0x7f9df015d330 0x7f9df01618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.376+0000 7f9dee59c640 1 -- 172.21.15.29:0/1736353683 >> v1:172.21.15.186:6789/0 conn(0x7f9df00661c0 legacy=0x7f9df00ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.376+0000 7f9dee59c640 1 -- 172.21.15.29:0/1736353683 >> v1:172.21.15.29:6789/0 conn(0x7f9df006c7d0 legacy=0x7f9df0065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.376+0000 7f9dee59c640 1 -- 172.21.15.29:0/1736353683 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9df0160dd0 con 0x7f9df015d330 2023-10-14T18:14:24.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.377+0000 7f9dee59c640 1 --2- 172.21.15.29:0/1736353683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df015d330 0x7f9df01618b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5800496795a68f48 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:24.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.382+0000 7f9dedd9b640 1 -- 172.21.15.29:0/1736353683 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0031830 con 0x7f9df015d330 2023-10-14T18:14:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.382+0000 7f9dedd9b640 1 -- 172.21.15.29:0/1736353683 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de0031990 con 0x7f9df015d330 2023-10-14T18:14:24.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.382+0000 7f9dedd9b640 1 -- 172.21.15.29:0/1736353683 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0031d20 con 0x7f9df015d330 2023-10-14T18:14:24.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.382+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1736353683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df015d330 msgr2=0x7f9df01618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.382+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1736353683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df015d330 0x7f9df01618b0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f9de00097a0 tx=0x7f9de002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.383+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1736353683 shutdown_connections 2023-10-14T18:14:24.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.383+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1736353683 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df015d330 0x7f9df01618b0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.383+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1736353683 >> 172.21.15.29:0/1736353683 conn(0x7f9df00d0ab0 msgr2=0x7f9df00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:24.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.383+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1736353683 shutdown_connections 2023-10-14T18:14:24.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.383+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1736353683 wait complete. 2023-10-14T18:14:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.384+0000 7f9df4c0e640 1 Processor -- start 2023-10-14T18:14:24.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 -- start start 2023-10-14T18:14:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df006c7d0 0x7f9df01f6dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df015d330 0x7f9df015fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c7af0 con 0x7f9df00661c0 2023-10-14T18:14:24.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c8b00 con 0x7f9df006c7d0 2023-10-14T18:14:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.385+0000 7f9df4c0e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9df00c7d40 con 0x7f9df015d330 2023-10-14T18:14:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43420/0 (socket says 172.21.15.29:43420) 2023-10-14T18:14:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 -- 172.21.15.29:0/1711159950 learned_addr learned my addr 172.21.15.29:0/1711159950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:24.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9deed9d640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df015d330 0x7f9df015fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df006c7d0 0x7f9df01f6dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df015d330 msgr2=0x7f9df015fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df015d330 0x7f9df015fcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df006c7d0 msgr2=0x7f9df01f6dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df006c7d0 0x7f9df01f6dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.386+0000 7f9dee59c640 1 -- 172.21.15.29:0/1711159950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9de0009400 con 0x7f9df00661c0 2023-10-14T18:14:24.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9dee59c640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9de0034040 tx=0x7f9de00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:24.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de003aa60 con 0x7f9df00661c0 2023-10-14T18:14:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9de003abe0 con 0x7f9df00661c0 2023-10-14T18:14:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9de0044400 con 0x7f9df00661c0 2023-10-14T18:14:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9df00c8020 con 0x7f9df00661c0 2023-10-14T18:14:24.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.387+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9df00c85e0 con 0x7f9df00661c0 2023-10-14T18:14:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.389+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9df00af8c0 con 0x7f9df00661c0 2023-10-14T18:14:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.389+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f9de004d440 con 0x7f9df00661c0 2023-10-14T18:14:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.390+0000 7f9de6ffd640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 0x7f9dd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.390+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f9de00abe80 con 0x7f9df00661c0 2023-10-14T18:14:24.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.390+0000 7f9de7fff640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 msgr2=0x7f9dd006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.390+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 0x7f9dd006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:24.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.394+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9de00769d0 con 0x7f9df00661c0 2023-10-14T18:14:24.469 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.467+0000 7f7b9842b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:14:24.469 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.468+0000 7f1334c3b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:14:24.575 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.575+0000 7f1334c3b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:14:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.591+0000 7f9de7fff640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 msgr2=0x7f9dd006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:24.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.591+0000 7f9de7fff640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 0x7f9dd006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:24.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.594+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9df004e160 con 0x7f9df00661c0 2023-10-14T18:14:24.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.595+0000 7f9de6ffd640 1 -- 172.21.15.29:0/1711159950 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v194) v1 ==== 100+0+11 (secure 0 0 0) 0x7f9de0035020 con 0x7f9df00661c0 2023-10-14T18:14:24.596 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:14:24.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 msgr2=0x7f9dd006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:24.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 0x7f9dd006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 msgr2=0x7f9df01f6890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 secure :-1 s=READY pgs=2269 cs=0 l=1 rev1=1 crypto rx=0x7f9de0034040 tx=0x7f9de00046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 shutdown_connections 2023-10-14T18:14:24.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f9dd006af90 0x7f9dd006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9df015d330 0x7f9df015fcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9df006c7d0 0x7f9df01f6dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 --2- 172.21.15.29:0/1711159950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9df00661c0 0x7f9df01f6890 unknown :-1 s=CLOSED pgs=2269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 >> 172.21.15.29:0/1711159950 conn(0x7f9df00d0ab0 msgr2=0x7f9df00cc740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:24.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 shutdown_connections 2023-10-14T18:14:24.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.599+0000 7f9df4c0e640 1 -- 172.21.15.29:0/1711159950 wait complete. 2023-10-14T18:14:24.625 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.624+0000 7f1334c3b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:14:24.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T18:14:24.651 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.650+0000 7f7b9842b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:14:24.732 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.731+0000 7f1334c3b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:24.767 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.766+0000 7f7b9842b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:14:24.821 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.821+0000 7f7b9842b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:14:24.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.835+0000 7f3f941d5640 1 Processor -- start 2023-10-14T18:14:24.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.835+0000 7f3f941d5640 1 -- start start 2023-10-14T18:14:24.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 0x7f3f8c0d0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8c0d3c60 con 0x7f3f8c0cf710 2023-10-14T18:14:24.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f8c0d3980 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f941d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0ce630 con 0x7f3f8c0cfaf0 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43434/0 (socket says 172.21.15.29:43434) 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 -- 172.21.15.29:0/2759603459 learned_addr learned my addr 172.21.15.29:0/2759603459 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91770640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f92772640 1 --2- 172.21.15.29:0/2759603459 >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 0x7f3f8c0d0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 -- 172.21.15.29:0/2759603459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 msgr2=0x7f3f8c0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:24.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.837+0000 7f3f91f71640 1 -- 172.21.15.29:0/2759603459 >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 msgr2=0x7f3f8c0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:24.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.838+0000 7f3f91f71640 1 --2- 172.21.15.29:0/2759603459 >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 0x7f3f8c0d0aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.838+0000 7f3f91f71640 1 -- 172.21.15.29:0/2759603459 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f8c0ce910 con 0x7f3f8c0cf710 2023-10-14T18:14:24.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.838+0000 7f3f92772640 1 --2- 172.21.15.29:0/2759603459 >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 0x7f3f8c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:24.842 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.840+0000 7f1334c3b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:14:24.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f91f71640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c06cd60 tx=0x7f3f8802b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2d614051e241342 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:24.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f90f6f640 1 -- 172.21.15.29:0/2759603459 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88004230 con 0x7f3f8c0cf710 2023-10-14T18:14:24.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f90f6f640 1 -- 172.21.15.29:0/2759603459 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f88004390 con 0x7f3f8c0cf710 2023-10-14T18:14:24.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f90f6f640 1 -- 172.21.15.29:0/2759603459 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88003810 con 0x7f3f8c0cf710 2023-10-14T18:14:24.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f941d5640 1 -- 172.21.15.29:0/2759603459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0cf710 msgr2=0x7f3f8c0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.839+0000 7f3f941d5640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 secure :-1 s=READY pgs=2270 cs=0 l=1 rev1=1 crypto rx=0x7f3f8c06cd60 tx=0x7f3f8802b860 comp rx=0 tx=0).stop 2023-10-14T18:14:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 -- 172.21.15.29:0/2759603459 shutdown_connections 2023-10-14T18:14:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 --2- 172.21.15.29:0/2759603459 >> v2:172.21.15.186:3300/0 conn(0x7f3f8c0b38e0 0x7f3f8c0d0aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 --2- 172.21.15.29:0/2759603459 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c0b2d10 unknown :-1 s=CLOSED pgs=2270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 -- 172.21.15.29:0/2759603459 >> 172.21.15.29:0/2759603459 conn(0x7f3f8c0d2cc0 msgr2=0x7f3f8c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 -- 172.21.15.29:0/2759603459 shutdown_connections 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.840+0000 7f3f941d5640 1 -- 172.21.15.29:0/2759603459 wait complete. 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.842+0000 7f3f941d5640 1 Processor -- start 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.842+0000 7f3f941d5640 1 -- start start 2023-10-14T18:14:24.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c1f7c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c1fa190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0c9d90 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0afd40 con 0x7f3f8c0cf710 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f941d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f8c0b0020 con 0x7f3f8c0cfaf0 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f91f71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f91f71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43436/0 (socket says 172.21.15.29:43436) 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f91f71640 1 -- 172.21.15.29:0/503780851 learned_addr learned my addr 172.21.15.29:0/503780851 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f92772640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c1fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.843+0000 7f3f91770640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c1f7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 msgr2=0x7f3f8c1fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c1fa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8c0cf710 msgr2=0x7f3f8c1f7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c1f7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:24.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 -- 172.21.15.29:0/503780851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f88009460 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f91f71640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f3f88000c00 tx=0x7f3f880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88003e50 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f88004500 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f8c0ccaa0 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.844+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f88006260 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.845+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f8c0c8a80 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.846+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f8c04e160 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.847+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f3f880063c0 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.847+0000 7f3f827fc640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 0x7f3f7006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:24.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.848+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(194..194 src has 1..194) v4 ==== 6513+0+0 (secure 0 0 0) 0x7f3f880abe00 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.848+0000 7f3f91770640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 msgr2=0x7f3f7006d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.848+0000 7f3f91770640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 0x7f3f7006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:24.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:24.852+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f880769d0 con 0x7f3f8c0b38e0 2023-10-14T18:14:24.938 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:24.937+0000 7f7b9842b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:24.997 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:24.996+0000 7f1334c3b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:25.046 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:25.045+0000 7f1334c3b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.049+0000 7f3f91770640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 msgr2=0x7f3f7006d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:25.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.049+0000 7f3f91770640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 0x7f3f7006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:25.051 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:25.050+0000 7f7b9842b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:14:25.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.059+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f3f8c0b7ef0 con 0x7f3f8c0b38e0 2023-10-14T18:14:25.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.060+0000 7f3f827fc640 1 -- 172.21.15.29:0/503780851 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v194) v1 ==== 81+0+930 (secure 0 0 0) 0x7f3f88037080 con 0x7f3f8c0b38e0 2023-10-14T18:14:25.061 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/614546910 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.061 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2776961501 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.061 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/287831498 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.061 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/841868971 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.062 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1377837433 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.062 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3575832370 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.062 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/196725873 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.062 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2946416628 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/730049499 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/79844030 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/79844030 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.063 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/196725873 2023-10-15T18:09:22.121284+0000 2023-10-14T18:14:25.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3534626168 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1930442848 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2824949307 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.064 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2461249058 2023-10-15T17:58:40.567922+0000 2023-10-14T18:14:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:listed 16 entries 2023-10-14T18:14:25.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 msgr2=0x7f3f7006d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 0x7f3f7006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 msgr2=0x7f3f8c0c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 secure :-1 s=READY pgs=2271 cs=0 l=1 rev1=1 crypto rx=0x7f3f88000c00 tx=0x7f3f880046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 shutdown_connections 2023-10-14T18:14:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f3f7006afc0 0x7f3f7006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f8c0cfaf0 0x7f3f8c1fa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f8c0cf710 0x7f3f8c1f7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 --2- 172.21.15.29:0/503780851 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f8c0b38e0 0x7f3f8c0c92f0 unknown :-1 s=CLOSED pgs=2271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.062+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 >> 172.21.15.29:0/503780851 conn(0x7f3f8c0d2cc0 msgr2=0x7f3f8c1560a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.063+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 shutdown_connections 2023-10-14T18:14:25.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.063+0000 7f3f941d5640 1 -- 172.21.15.29:0/503780851 wait complete. 2023-10-14T18:14:25.082 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T18:14:25.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:25.083 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T18:14:25.167 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:25.166+0000 7f1334c3b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:14:25.226 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:25.224+0000 7f7b9842b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:25.230 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:14:25.230 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:25.231 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T18:14:25.279 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:14:25.280 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:25.281 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:25.279+0000 7f7b9842b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:25.335 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:14:25.336 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:25.336 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:25.394 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:14:25.395 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:14:25.395 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:25.408 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:25.407+0000 7f7b9842b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:14:25.496 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:14:25.497 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:25.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.748+0000 7faa9026f640 1 Processor -- start 2023-10-14T18:14:25.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.748+0000 7faa9026f640 1 -- start start 2023-10-14T18:14:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.749+0000 7faa9026f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa88159b20 0x7faa8806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa9026f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8806f7f0 con 0x7faa88159f00 2023-10-14T18:14:25.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa9026f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa8806c5e0 con 0x7faa880b0b20 2023-10-14T18:14:25.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa9026f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8806c8c0 con 0x7faa88159b20 2023-10-14T18:14:25.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa8d80a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa88159f00 0x7faa880b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40422/0 (socket says 172.21.15.29:40422) 2023-10-14T18:14:25.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa8d80a640 1 -- 172.21.15.29:0/3134820865 learned_addr learned my addr 172.21.15.29:0/3134820865 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:25.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa8e00b640 1 --2- 172.21.15.29:0/3134820865 >> v2:172.21.15.186:3300/0 conn(0x7faa88159b20 0x7faa8806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 839928858 0 0) 0x7faa8806f7f0 con 0x7faa88159f00 2023-10-14T18:14:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.750+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa74003520 con 0x7faa88159f00 2023-10-14T18:14:25.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884408817 0 0) 0x7faa8806c5e0 con 0x7faa880b0b20 2023-10-14T18:14:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa8806f7f0 con 0x7faa880b0b20 2023-10-14T18:14:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4077730704 0 0) 0x7faa74003520 con 0x7faa88159f00 2023-10-14T18:14:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 >> v2:172.21.15.186:3300/0 conn(0x7faa88159b20 msgr2=0x7faa8806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:25.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 --2- 172.21.15.29:0/3134820865 >> v2:172.21.15.186:3300/0 conn(0x7faa88159b20 0x7faa8806bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 >> v1:172.21.15.186:6789/0 conn(0x7faa880b0b20 legacy=0x7faa8806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8806cba0 con 0x7faa88159f00 2023-10-14T18:14:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7faa78001670 con 0x7faa88159f00 2023-10-14T18:14:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 >> v1:172.21.15.29:6789/0 conn(0x7faa88159f00 legacy=0x7faa880b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa74003b50 0x7faa74003f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 0x7faa74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa74005180 0x7faa7400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8806c8c0 con 0x7faa740047c0 2023-10-14T18:14:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400e260 con 0x7faa74005180 2023-10-14T18:14:25.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7400e420 con 0x7faa74003b50 2023-10-14T18:14:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.751+0000 7faa8d80a640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 0x7faa74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8e00b640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa74003b50 0x7faa74003f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8e80c640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa74005180 0x7faa7400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8d80a640 1 -- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa74003b50 msgr2=0x7faa74003f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8d80a640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa74003b50 0x7faa74003f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8d80a640 1 -- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa74005180 msgr2=0x7faa7400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8d80a640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa74005180 0x7faa7400db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8d80a640 1 -- 172.21.15.29:0/3134820865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74003520 con 0x7faa740047c0 2023-10-14T18:14:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.752+0000 7faa8e80c640 1 -- 172.21.15.29:0/3134820865 reap_dead start 2023-10-14T18:14:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.753+0000 7faa8d80a640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 0x7faa74004c40 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7faa78007f70 tx=0x7faa740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1260c30f53c9a57c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.753+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78005ad0 con 0x7faa740047c0 2023-10-14T18:14:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.753+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa78005c50 con 0x7faa740047c0 2023-10-14T18:14:25.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.753+0000 7faa8d009640 1 -- 172.21.15.29:0/3134820865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa78007070 con 0x7faa740047c0 2023-10-14T18:14:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 -- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 msgr2=0x7faa74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 0x7faa74004c40 secure :-1 s=READY pgs=2273 cs=0 l=1 rev1=1 crypto rx=0x7faa78007f70 tx=0x7faa740057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 -- 172.21.15.29:0/3134820865 shutdown_connections 2023-10-14T18:14:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 --2- 172.21.15.29:0/3134820865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa740047c0 0x7faa74004c40 unknown :-1 s=CLOSED pgs=2273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 -- 172.21.15.29:0/3134820865 >> 172.21.15.29:0/3134820865 conn(0x7faa880cb410 msgr2=0x7faa880cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 -- 172.21.15.29:0/3134820865 shutdown_connections 2023-10-14T18:14:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.754+0000 7faa9026f640 1 -- 172.21.15.29:0/3134820865 wait complete. 2023-10-14T18:14:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.756+0000 7faa9026f640 1 Processor -- start 2023-10-14T18:14:25.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.756+0000 7faa9026f640 1 -- start start 2023-10-14T18:14:25.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa740047c0 0x7faa880c6120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa881f2c30 0x7faa8815c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa880af7d0 con 0x7faa8815ca40 2023-10-14T18:14:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8815cec0 con 0x7faa740047c0 2023-10-14T18:14:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa9026f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa88144860 con 0x7faa881f2c30 2023-10-14T18:14:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa8e80c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa8e80c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43466/0 (socket says 172.21.15.29:43466) 2023-10-14T18:14:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa8e80c640 1 -- 172.21.15.29:0/308299743 learned_addr learned my addr 172.21.15.29:0/308299743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa8e00b640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa740047c0 0x7faa880c6120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.757+0000 7faa8d80a640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa881f2c30 0x7faa8815c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa881f2c30 msgr2=0x7faa8815c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa881f2c30 0x7faa8815c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa740047c0 msgr2=0x7faa880c6120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa740047c0 0x7faa880c6120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 -- 172.21.15.29:0/308299743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa88156440 con 0x7faa8815ca40 2023-10-14T18:14:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa8e80c640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7faa880b4b00 tx=0x7faa8000ea20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:25.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80006d60 con 0x7faa8815ca40 2023-10-14T18:14:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80003970 con 0x7faa8815ca40 2023-10-14T18:14:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa880c6660 con 0x7faa8815ca40 2023-10-14T18:14:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.758+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80003d20 con 0x7faa8815ca40 2023-10-14T18:14:25.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.759+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa881f2150 con 0x7faa8815ca40 2023-10-14T18:14:25.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.760+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa881f1fb0 con 0x7faa8815ca40 2023-10-14T18:14:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.761+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7faa80005070 con 0x7faa8815ca40 2023-10-14T18:14:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.762+0000 7faa7effd640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 0x7faa6c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:25.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.762+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6601+0+0 (secure 0 0 0) 0x7faa80089090 con 0x7faa8815ca40 2023-10-14T18:14:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.762+0000 7faa8e00b640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 msgr2=0x7faa6c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.762+0000 7faa8e00b640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 0x7faa6c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:25.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.766+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa881f1fb0 con 0x7faa8815ca40 2023-10-14T18:14:25.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.965+0000 7faa8e00b640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 msgr2=0x7faa6c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:25.966+0000 7faa8e00b640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 0x7faa6c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:26.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.038+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa881f1fb0 con 0x7faa8815ca40 2023-10-14T18:14:26.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.038+0000 7faa7effd640 1 -- 172.21.15.29:0/308299743 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7faa881f1fb0 con 0x7faa8815ca40 2023-10-14T18:14:26.039 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:26.040 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T18:14:26.040 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T18:14:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.040+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 msgr2=0x7faa6c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.040+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 0x7faa6c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.040+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 msgr2=0x7faa880ae910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.040+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 secure :-1 s=READY pgs=2274 cs=0 l=1 rev1=1 crypto rx=0x7faa880b4b00 tx=0x7faa8000ea20 comp rx=0 tx=0).stop 2023-10-14T18:14:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 shutdown_connections 2023-10-14T18:14:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7faa6c06af90 0x7faa6c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa8815ca40 0x7faa880ae910 unknown :-1 s=CLOSED pgs=2274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa881f2c30 0x7faa8815c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 --2- 172.21.15.29:0/308299743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa740047c0 0x7faa880c6120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 >> 172.21.15.29:0/308299743 conn(0x7faa880cb410 msgr2=0x7faa880cdf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 shutdown_connections 2023-10-14T18:14:26.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.041+0000 7faa9026f640 1 -- 172.21.15.29:0/308299743 wait complete. 2023-10-14T18:14:26.059 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:26.107 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:26.106+0000 7f1334c3b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:14:26.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.249+0000 7fce9dbc0640 1 Processor -- start 2023-10-14T18:14:26.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.249+0000 7fce9dbc0640 1 -- start start 2023-10-14T18:14:26.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce9dbc0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce9815aaa0 0x7fce9815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce9dbc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce980cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce9dbc0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce9815ed60 con 0x7fce980ccaa0 2023-10-14T18:14:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce9dbc0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce9806b810 con 0x7fce9815aaa0 2023-10-14T18:14:26.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce9dbc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce980707e0 con 0x7fce9815ae80 2023-10-14T18:14:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce977fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce9815aaa0 0x7fce9815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce977fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fce9815aaa0 0x7fce9815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43472/0 (socket says 172.21.15.29:43472) 2023-10-14T18:14:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce977fe640 1 -- 172.21.15.29:0/2840188452 learned_addr learned my addr 172.21.15.29:0/2840188452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:26.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.251+0000 7fce96ffd640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce980cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.252+0000 7fce977fe640 1 -- 172.21.15.29:0/2840188452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 msgr2=0x7fce980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.252+0000 7fce977fe640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce980cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.252+0000 7fce977fe640 1 -- 172.21.15.29:0/2840188452 >> v1:172.21.15.186:6789/0 conn(0x7fce980ccaa0 legacy=0x7fce98070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.252+0000 7fce977fe640 1 -- 172.21.15.29:0/2840188452 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce9815ef90 con 0x7fce9815aaa0 2023-10-14T18:14:26.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.253+0000 7fce977fe640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce9815e820 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fce88009870 tx=0x7fce8802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2806fa5c24221ef3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.253+0000 7fce967fc640 1 -- 172.21.15.29:0/2840188452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88031640 con 0x7fce9815aaa0 2023-10-14T18:14:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.253+0000 7fce967fc640 1 -- 172.21.15.29:0/2840188452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce880317a0 con 0x7fce9815aaa0 2023-10-14T18:14:26.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.253+0000 7fce967fc640 1 -- 172.21.15.29:0/2840188452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88031b30 con 0x7fce9815aaa0 2023-10-14T18:14:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 -- 172.21.15.29:0/2840188452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 msgr2=0x7fce9815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce9815e820 secure :-1 s=READY pgs=2275 cs=0 l=1 rev1=1 crypto rx=0x7fce88009870 tx=0x7fce8802b8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 -- 172.21.15.29:0/2840188452 shutdown_connections 2023-10-14T18:14:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce980cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/2840188452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce9815e820 unknown :-1 s=CLOSED pgs=2275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 -- 172.21.15.29:0/2840188452 >> 172.21.15.29:0/2840188452 conn(0x7fce980cb410 msgr2=0x7fce980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 -- 172.21.15.29:0/2840188452 shutdown_connections 2023-10-14T18:14:26.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.254+0000 7fce9dbc0640 1 -- 172.21.15.29:0/2840188452 wait complete. 2023-10-14T18:14:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.256+0000 7fce9dbc0640 1 Processor -- start 2023-10-14T18:14:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.256+0000 7fce9dbc0640 1 -- start start 2023-10-14T18:14:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce980ccaa0 0x7fce98155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce981f6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981f5b10 con 0x7fce9815aaa0 2023-10-14T18:14:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce980c37a0 con 0x7fce980ccaa0 2023-10-14T18:14:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce9dbc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce981596b0 con 0x7fce9815ae80 2023-10-14T18:14:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce96ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce96ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43476/0 (socket says 172.21.15.29:43476) 2023-10-14T18:14:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce96ffd640 1 -- 172.21.15.29:0/1229680767 learned_addr learned my addr 172.21.15.29:0/1229680767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce97fff640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce981f6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.257+0000 7fce977fe640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce980ccaa0 0x7fce98155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 msgr2=0x7fce981f6290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce981f6290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce980ccaa0 msgr2=0x7fce98155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce980ccaa0 0x7fce98155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 -- 172.21.15.29:0/1229680767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880094d0 con 0x7fce9815aaa0 2023-10-14T18:14:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce96ffd640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d800 tx=0x7fce8c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:26.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c00b6c0 con 0x7fce9815aaa0 2023-10-14T18:14:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8c00b820 con 0x7fce9815aaa0 2023-10-14T18:14:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8c00bbb0 con 0x7fce9815aaa0 2023-10-14T18:14:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98159990 con 0x7fce9815aaa0 2023-10-14T18:14:26.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.258+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce98142980 con 0x7fce9815aaa0 2023-10-14T18:14:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.260+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce98159bd0 con 0x7fce9815aaa0 2023-10-14T18:14:26.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.261+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fce8c00bd10 con 0x7fce9815aaa0 2023-10-14T18:14:26.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.262+0000 7fce7ffff640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 0x7fce8406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.262+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(195..195 src has 1..195) v4 ==== 6601+0+0 (secure 0 0 0) 0x7fce8c08a020 con 0x7fce9815aaa0 2023-10-14T18:14:26.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.262+0000 7fce977fe640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 msgr2=0x7fce8406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:26.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.262+0000 7fce977fe640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 0x7fce8406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:26.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.267+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce8c054cd0 con 0x7fce9815aaa0 2023-10-14T18:14:26.371 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:26.370+0000 7f7b9842b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:14:26.444 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:26.444+0000 7f1334c3b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:14:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.464+0000 7fce977fe640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 msgr2=0x7fce8406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:26.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.464+0000 7fce977fe640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 0x7fce8406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:26.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.468+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce9815f240 con 0x7fce9815aaa0 2023-10-14T18:14:26.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.471+0000 7fce7ffff640 1 -- 172.21.15.29:0/1229680767 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v196) v1 ==== 74+0+17877 (secure 0 0 0) 0x7fce8c0582f0 con 0x7fce9815aaa0 2023-10-14T18:14:26.472 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:26.472 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":196,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:26.387874+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/614546910":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2776961501":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/287831498":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/841868971":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1377837433":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3575832370":"2023-10-15T18:09:22.121284+0000","172.21.15.29:6833/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2946416628":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/730049499":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6833/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3534626168":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/1930442848":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2824949307":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2461249058":"2023-10-15T17:58:40.567922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[{"pool":23,"snaps":[{"begin":17,"length":1},{"begin":33,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 msgr2=0x7fce8406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 0x7fce8406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 msgr2=0x7fce981f5d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 secure :-1 s=READY pgs=2276 cs=0 l=1 rev1=1 crypto rx=0x7fce8c00d800 tx=0x7fce8c00dcc0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 shutdown_connections 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7fce8406af90 0x7fce8406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce9815ae80 0x7fce981f6290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce9815aaa0 0x7fce981f5d50 unknown :-1 s=CLOSED pgs=2276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 --2- 172.21.15.29:0/1229680767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce980ccaa0 0x7fce98155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 >> 172.21.15.29:0/1229680767 conn(0x7fce980cb410 msgr2=0x7fce980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 shutdown_connections 2023-10-14T18:14:26.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.473+0000 7fce9dbc0640 1 -- 172.21.15.29:0/1229680767 wait complete. 2023-10-14T18:14:26.496 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 22, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T17:58:23.469364+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '191', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 23, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T17:58:24.568568+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '196', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 33, 'snap_epoch': 196, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:14:26.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:14:26.503 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:26.503+0000 7f1334c3b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:14:26.622 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:26.622+0000 7f1334c3b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:14:26.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.702+0000 7f07a1cc6640 1 Processor -- start 2023-10-14T18:14:26.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.702+0000 7f07a1cc6640 1 -- start start 2023-10-14T18:14:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.703+0000 7f07a1cc6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f079c15d150 0x7f079c0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.703+0000 7f07a1cc6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c06a4b0 0x7f079c160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.703+0000 7f07a1cc6640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f079c0ac890 con 0x7f079c160b90 2023-10-14T18:14:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.703+0000 7f07a1cc6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f079c15d530 con 0x7f079c15d150 2023-10-14T18:14:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.703+0000 7f07a1cc6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c161650 con 0x7f079c06a4b0 2023-10-14T18:14:26.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f079c15d150 0x7f079c0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079affd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f079c160b90 0x7f079c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40424/0 (socket says 172.21.15.29:40424) 2023-10-14T18:14:26.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 --2- 172.21.15.29:0/1965451965 >> v2:172.21.15.29:3300/0 conn(0x7f079c15d150 0x7f079c0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43490/0 (socket says 172.21.15.29:43490) 2023-10-14T18:14:26.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079affd640 1 -- 172.21.15.29:0/1965451965 learned_addr learned my addr 172.21.15.29:0/1965451965 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079bfff640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c06a4b0 0x7f079c160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 -- 172.21.15.29:0/1965451965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c06a4b0 msgr2=0x7f079c160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c06a4b0 0x7f079c160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 -- 172.21.15.29:0/1965451965 >> v1:172.21.15.29:6789/0 conn(0x7f079c160b90 legacy=0x7f079c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079b7fe640 1 -- 172.21.15.29:0/1965451965 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c161930 con 0x7f079c15d150 2023-10-14T18:14:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.704+0000 7f079a7fc640 1 -- 172.21.15.29:0/1965451965 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628910216 0 0) 0x7f079c0ac890 con 0x7f079c160b90 2023-10-14T18:14:26.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.705+0000 7f079b7fe640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c15d150 0x7f079c0ad350 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f078c0097a0 tx=0x7f078c02b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d702275340c285ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.706+0000 7f079a7fc640 1 -- 172.21.15.29:0/1965451965 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c031780 con 0x7f079c15d150 2023-10-14T18:14:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.706+0000 7f079a7fc640 1 -- 172.21.15.29:0/1965451965 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c0318e0 con 0x7f079c15d150 2023-10-14T18:14:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.706+0000 7f079a7fc640 1 -- 172.21.15.29:0/1965451965 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c031c70 con 0x7f079c15d150 2023-10-14T18:14:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.706+0000 7f07a1cc6640 1 -- 172.21.15.29:0/1965451965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c15d150 msgr2=0x7f079c0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.706+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c15d150 0x7f079c0ad350 secure :-1 s=READY pgs=2278 cs=0 l=1 rev1=1 crypto rx=0x7f078c0097a0 tx=0x7f078c02b820 comp rx=0 tx=0).stop 2023-10-14T18:14:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 -- 172.21.15.29:0/1965451965 shutdown_connections 2023-10-14T18:14:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c06a4b0 0x7f079c160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/1965451965 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c15d150 0x7f079c0ad350 unknown :-1 s=CLOSED pgs=2278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 -- 172.21.15.29:0/1965451965 >> 172.21.15.29:0/1965451965 conn(0x7f079c0ce9b0 msgr2=0x7f079c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 -- 172.21.15.29:0/1965451965 shutdown_connections 2023-10-14T18:14:26.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.707+0000 7f07a1cc6640 1 -- 172.21.15.29:0/1965451965 wait complete. 2023-10-14T18:14:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.708+0000 7f07a1cc6640 1 Processor -- start 2023-10-14T18:14:26.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.708+0000 7f07a1cc6640 1 -- start start 2023-10-14T18:14:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f079c15d150 0x7f079c14bd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c160b90 0x7f079c0b0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0c6a50 con 0x7f079c06a4b0 2023-10-14T18:14:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0c5e40 con 0x7f079c15d150 2023-10-14T18:14:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.709+0000 7f07a1cc6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079c0c6120 con 0x7f079c160b90 2023-10-14T18:14:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43502/0 (socket says 172.21.15.29:43502) 2023-10-14T18:14:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 -- 172.21.15.29:0/2111940471 learned_addr learned my addr 172.21.15.29:0/2111940471 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079affd640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f079c15d150 0x7f079c14bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c160b90 msgr2=0x7f079c0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c160b90 0x7f079c0b0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f079c15d150 msgr2=0x7f079c14bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f079c15d150 0x7f079c14bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 -- 172.21.15.29:0/2111940471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f078c009400 con 0x7f079c06a4b0 2023-10-14T18:14:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.710+0000 7f079b7fe640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f078c034040 tx=0x7f078c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.711+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c032820 con 0x7f079c06a4b0 2023-10-14T18:14:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.711+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f078c032980 con 0x7f079c06a4b0 2023-10-14T18:14:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.711+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f078c032d30 con 0x7f079c06a4b0 2023-10-14T18:14:26.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.711+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f079c0c6400 con 0x7f079c06a4b0 2023-10-14T18:14:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.711+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f079c0c7420 con 0x7f079c06a4b0 2023-10-14T18:14:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.713+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f079c04e160 con 0x7f079c06a4b0 2023-10-14T18:14:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.713+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 118) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f078c04e440 con 0x7f079c06a4b0 2023-10-14T18:14:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.715+0000 7f0783fff640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 0x7f078806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.715+0000 7f079affd640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 msgr2=0x7f078806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.715+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(196..196 src has 1..196) v4 ==== 6585+0+0 (secure 0 0 0) 0x7f078c0abdf0 con 0x7f079c06a4b0 2023-10-14T18:14:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.715+0000 7f079affd640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 0x7f078806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.719+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f078c076920 con 0x7f079c06a4b0 2023-10-14T18:14:26.732 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:26.732+0000 7f7b9842b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:14:26.797 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:26.796+0000 7f7b9842b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:14:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.916+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f079c0b3c40 con 0x7f079c06a4b0 2023-10-14T18:14:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.916+0000 7f079affd640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 msgr2=0x7f078806d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1263890494 2023-10-14T18:14:26.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.916+0000 7f079affd640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 0x7f078806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:26.918 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:26.916+0000 7f7b9842b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:14:26.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.917+0000 7f0783fff640 1 -- 172.21.15.29:0/2111940471 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v196) v1 ==== 93+0+10 (secure 0 0 0) 0x7f078c079f40 con 0x7f079c06a4b0 2023-10-14T18:14:26.919 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:14:26.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 msgr2=0x7f078806d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 0x7f078806d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 msgr2=0x7f079c0c6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 secure :-1 s=READY pgs=2279 cs=0 l=1 rev1=1 crypto rx=0x7f078c034040 tx=0x7f078c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 shutdown_connections 2023-10-14T18:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:6832/1263890494,v1:172.21.15.29:6833/1263890494] conn(0x7f078806af90 0x7f078806d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f079c160b90 0x7f079c0b0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f079c15d150 0x7f079c14bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 --2- 172.21.15.29:0/2111940471 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f079c06a4b0 0x7f079c0c6ee0 unknown :-1 s=CLOSED pgs=2279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:26.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.919+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 >> 172.21.15.29:0/2111940471 conn(0x7f079c0ce9b0 msgr2=0x7f079c06f3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.920+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 shutdown_connections 2023-10-14T18:14:26.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:26.920+0000 7f07a1cc6640 1 -- 172.21.15.29:0/2111940471 wait complete. 2023-10-14T18:14:26.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:14:27.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.003+0000 7f7b1c813640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1c813640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1c813640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1c813640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.005 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1c813640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1a00e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1a00e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1a00e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1a00e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:27.004+0000 7f7b1a00e640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:14:27.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.133+0000 7fc36024b640 1 Processor -- start 2023-10-14T18:14:27.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.133+0000 7fc36024b640 1 -- start start 2023-10-14T18:14:27.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc36024b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc358069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc36024b640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3580cb030 con 0x7fc3580c9ca0 2023-10-14T18:14:27.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc36024b640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3580c91c0 con 0x7fc3580caac0 2023-10-14T18:14:27.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc36024b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580c94a0 con 0x7fc35815ad70 2023-10-14T18:14:27.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc35d7e6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3580c9ca0 0x7fc3580ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40432/0 (socket says 172.21.15.29:40432) 2023-10-14T18:14:27.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc35d7e6640 1 -- 172.21.15.29:0/476334892 learned_addr learned my addr 172.21.15.29:0/476334892 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:27.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.135+0000 7fc35dfe7640 1 --2- 172.21.15.29:0/476334892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc358069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.136+0000 7fc35dfe7640 1 -- 172.21.15.29:0/476334892 >> v1:172.21.15.186:6789/0 conn(0x7fc3580caac0 legacy=0x7fc3580ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.136+0000 7fc35dfe7640 1 -- 172.21.15.29:0/476334892 >> v1:172.21.15.29:6789/0 conn(0x7fc3580c9ca0 legacy=0x7fc3580ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.136+0000 7fc35dfe7640 1 -- 172.21.15.29:0/476334892 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3580c9780 con 0x7fc35815ad70 2023-10-14T18:14:27.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.138+0000 7fc35dfe7640 1 --2- 172.21.15.29:0/476334892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc358069da0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc3540097d0 tx=0x7fc35402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bee8d7e055ebb80 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:27.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.139+0000 7fc35cfe5640 1 -- 172.21.15.29:0/476334892 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354031890 con 0x7fc35815ad70 2023-10-14T18:14:27.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.139+0000 7fc35cfe5640 1 -- 172.21.15.29:0/476334892 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3540319f0 con 0x7fc35815ad70 2023-10-14T18:14:27.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.139+0000 7fc35cfe5640 1 -- 172.21.15.29:0/476334892 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc354031d80 con 0x7fc35815ad70 2023-10-14T18:14:27.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 -- 172.21.15.29:0/476334892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 msgr2=0x7fc358069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 --2- 172.21.15.29:0/476334892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc358069da0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc3540097d0 tx=0x7fc35402b930 comp rx=0 tx=0).stop 2023-10-14T18:14:27.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 -- 172.21.15.29:0/476334892 shutdown_connections 2023-10-14T18:14:27.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 --2- 172.21.15.29:0/476334892 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc358069da0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 -- 172.21.15.29:0/476334892 >> 172.21.15.29:0/476334892 conn(0x7fc3580ce9b0 msgr2=0x7fc3580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:27.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 -- 172.21.15.29:0/476334892 shutdown_connections 2023-10-14T18:14:27.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.140+0000 7fc36024b640 1 -- 172.21.15.29:0/476334892 wait complete. 2023-10-14T18:14:27.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.142+0000 7fc36024b640 1 Processor -- start 2023-10-14T18:14:27.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.142+0000 7fc36024b640 1 -- start start 2023-10-14T18:14:27.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3580c9ca0 0x7fc358159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc3580c68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3581595c0 con 0x7fc3580caac0 2023-10-14T18:14:27.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3581f27f0 con 0x7fc3580c9ca0 2023-10-14T18:14:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc36024b640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3580b3160 con 0x7fc35815ad70 2023-10-14T18:14:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc35d7e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc35d7e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43616/0 (socket says 172.21.15.29:43616) 2023-10-14T18:14:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc35d7e6640 1 -- 172.21.15.29:0/1689620884 learned_addr learned my addr 172.21.15.29:0/1689620884 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:27.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.143+0000 7fc35dfe7640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3580c9ca0 0x7fc358159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35e7e8640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc3580c68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 -- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 msgr2=0x7fc3580c68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc3580c68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 -- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3580c9ca0 msgr2=0x7fc358159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3580c9ca0 0x7fc358159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 -- 172.21.15.29:0/1689620884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc354009400 con 0x7fc3580caac0 2023-10-14T18:14:27.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35e7e8640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc3580c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:27.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc35d7e6640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c4a0 tx=0x7fc34800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:27.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348005ce0 con 0x7fc3580caac0 2023-10-14T18:14:27.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc348005e40 con 0x7fc3580caac0 2023-10-14T18:14:27.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc348007270 con 0x7fc3580caac0 2023-10-14T18:14:27.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3580c6ee0 con 0x7fc3580caac0 2023-10-14T18:14:27.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.144+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3581f2480 con 0x7fc3580caac0 2023-10-14T18:14:27.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.145+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3581f2980 con 0x7fc3580caac0 2023-10-14T18:14:27.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.146+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 119) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fc3480073d0 con 0x7fc3580caac0 2023-10-14T18:14:27.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.147+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7fc348088a60 con 0x7fc3580caac0 2023-10-14T18:14:27.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.152+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc348053a80 con 0x7fc3580caac0 2023-10-14T18:14:27.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.347+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc35804e160 con 0x7fc3580caac0 2023-10-14T18:14:27.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.348+0000 7fc34e7fc640 1 -- 172.21.15.29:0/1689620884 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v197) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc348017020 con 0x7fc3580caac0 2023-10-14T18:14:27.349 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:14:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 msgr2=0x7fc3580c6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 secure :-1 s=READY pgs=2292 cs=0 l=1 rev1=1 crypto rx=0x7fc34800c4a0 tx=0x7fc34800c960 comp rx=0 tx=0).stop 2023-10-14T18:14:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 shutdown_connections 2023-10-14T18:14:27.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35815ad70 0x7fc3580c68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3580caac0 0x7fc3580c6370 unknown :-1 s=CLOSED pgs=2292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 --2- 172.21.15.29:0/1689620884 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3580c9ca0 0x7fc358159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.351+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 >> 172.21.15.29:0/1689620884 conn(0x7fc3580ce9b0 msgr2=0x7fc3580ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.352+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 shutdown_connections 2023-10-14T18:14:27.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.352+0000 7fc36024b640 1 -- 172.21.15.29:0/1689620884 wait complete. 2023-10-14T18:14:27.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:14:27.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.591+0000 7ff36da52640 1 Processor -- start 2023-10-14T18:14:27.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.592+0000 7ff36da52640 1 -- start start 2023-10-14T18:14:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36da52640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff3680b8480 0x7ff3680cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36da52640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff36815d150 con 0x7ff36815cd70 2023-10-14T18:14:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36da52640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3680d1180 con 0x7ff3680d0be0 2023-10-14T18:14:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36da52640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3680d1460 con 0x7ff3680b8480 2023-10-14T18:14:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36ca50640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff36815cd70 0x7ff3680d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40434/0 (socket says 172.21.15.29:40434) 2023-10-14T18:14:27.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.593+0000 7ff36ca50640 1 -- 172.21.15.29:0/1351763218 learned_addr learned my addr 172.21.15.29:0/1351763218 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:27.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347220176 0 0) 0x7ff36815d150 con 0x7ff36815cd70 2023-10-14T18:14:27.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff358003520 con 0x7ff36815cd70 2023-10-14T18:14:27.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff367fff640 1 --2- 172.21.15.29:0/1351763218 >> v2:172.21.15.186:3300/0 conn(0x7ff3680b8480 0x7ff3680cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2523228020 0 0) 0x7ff358003520 con 0x7ff36815cd70 2023-10-14T18:14:27.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 >> v2:172.21.15.186:3300/0 conn(0x7ff3680b8480 msgr2=0x7ff3680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 --2- 172.21.15.29:0/1351763218 >> v2:172.21.15.186:3300/0 conn(0x7ff3680b8480 0x7ff3680cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 >> v1:172.21.15.186:6789/0 conn(0x7ff3680d0be0 legacy=0x7ff3680b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3680d1740 con 0x7ff36815cd70 2023-10-14T18:14:27.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.594+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff35c001670 con 0x7ff36815cd70 2023-10-14T18:14:27.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 >> v1:172.21.15.29:6789/0 conn(0x7ff36815cd70 legacy=0x7ff3680d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff358003970 0x7ff358003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3580047c0 0x7ff358004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 0x7ff35800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680d1460 con 0x7ff358005180 2023-10-14T18:14:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff35800c120 con 0x7ff358003970 2023-10-14T18:14:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35800c2e0 con 0x7ff3580047c0 2023-10-14T18:14:27.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff36d251640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 0x7ff35800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff36ca50640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff358003970 0x7ff358003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff36d251640 1 -- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3580047c0 msgr2=0x7ff358004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:27.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff367fff640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3580047c0 0x7ff358004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff36d251640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff3580047c0 0x7ff358004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.595+0000 7ff36d251640 1 -- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff358003970 msgr2=0x7ff358003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:27.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.596+0000 7ff36d251640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff358003970 0x7ff358003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.596+0000 7ff36d251640 1 -- 172.21.15.29:0/1351763218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff358003520 con 0x7ff358005180 2023-10-14T18:14:27.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.596+0000 7ff36d251640 1 -- 172.21.15.29:0/1351763218 reap_dead start 2023-10-14T18:14:27.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff36d251640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 0x7ff35800b980 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff360001e50 tx=0x7ff3580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88bb86fc7c4a9249 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:27.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360003b70 con 0x7ff358005180 2023-10-14T18:14:27.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360003d10 con 0x7ff358005180 2023-10-14T18:14:27.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff3677fe640 1 -- 172.21.15.29:0/1351763218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360005310 con 0x7ff358005180 2023-10-14T18:14:27.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff36da52640 1 -- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 msgr2=0x7ff35800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.597+0000 7ff36da52640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 0x7ff35800b980 secure :-1 s=READY pgs=2294 cs=0 l=1 rev1=1 crypto rx=0x7ff360001e50 tx=0x7ff3580057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.598+0000 7ff36da52640 1 -- 172.21.15.29:0/1351763218 shutdown_connections 2023-10-14T18:14:27.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.598+0000 7ff36da52640 1 --2- 172.21.15.29:0/1351763218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff358005180 0x7ff35800b980 unknown :-1 s=CLOSED pgs=2294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.598+0000 7ff36da52640 1 -- 172.21.15.29:0/1351763218 >> 172.21.15.29:0/1351763218 conn(0x7ff3680cf570 msgr2=0x7ff3680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.598+0000 7ff36da52640 1 -- 172.21.15.29:0/1351763218 shutdown_connections 2023-10-14T18:14:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.598+0000 7ff36da52640 1 -- 172.21.15.29:0/1351763218 wait complete. 2023-10-14T18:14:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.599+0000 7ff36da52640 1 Processor -- start 2023-10-14T18:14:27.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 -- start start 2023-10-14T18:14:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff358005180 0x7ff3681e6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3681e5760 0x7ff36814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:27.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3680b3a90 con 0x7ff3680b4b20 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3681e7200 con 0x7ff3681e5760 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.600+0000 7ff36da52640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3681e5c20 con 0x7ff358005180 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43628/0 (socket says 172.21.15.29:43628) 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 -- 172.21.15.29:0/670512508 learned_addr learned my addr 172.21.15.29:0/670512508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:27.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36ca50640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff358005180 0x7ff3681e6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff367fff640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3681e5760 0x7ff36814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 -- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff358005180 msgr2=0x7ff3681e6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff358005180 0x7ff3681e6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 -- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3681e5760 msgr2=0x7ff36814fb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3681e5760 0x7ff36814fb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.601+0000 7ff36d251640 1 -- 172.21.15.29:0/670512508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3681e5f00 con 0x7ff3680b4b20 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff36d251640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7ff360001f70 tx=0x7ff3600014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360006d50 con 0x7ff3680b4b20 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff360007030 con 0x7ff3680b4b20 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff360007420 con 0x7ff3680b4b20 2023-10-14T18:14:27.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3681e61e0 con 0x7ff3680b4b20 2023-10-14T18:14:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.602+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3681e67a0 con 0x7ff3680b4b20 2023-10-14T18:14:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.603+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff36804e9b0 con 0x7ff3680b4b20 2023-10-14T18:14:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.604+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 119) v1 ==== 88940+0+0 (secure 0 0 0) 0x7ff360007580 con 0x7ff3680b4b20 2023-10-14T18:14:27.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.605+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7ff3600ab400 con 0x7ff3680b4b20 2023-10-14T18:14:27.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.610+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff360076370 con 0x7ff3680b4b20 2023-10-14T18:14:27.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.805+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff3680b3a70 con 0x7ff3680b4b20 2023-10-14T18:14:27.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.806+0000 7ff3657fa640 1 -- 172.21.15.29:0/670512508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v197) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff360079990 con 0x7ff3680b4b20 2023-10-14T18:14:27.807 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:14:27.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 msgr2=0x7ff3681f2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 secure :-1 s=READY pgs=2295 cs=0 l=1 rev1=1 crypto rx=0x7ff360001f70 tx=0x7ff3600014a0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 shutdown_connections 2023-10-14T18:14:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff3680b4b20 0x7ff3681f2350 unknown :-1 s=CLOSED pgs=2295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3681e5760 0x7ff36814fb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 --2- 172.21.15.29:0/670512508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff358005180 0x7ff3681e6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 >> 172.21.15.29:0/670512508 conn(0x7ff3680cf570 msgr2=0x7ff3680ce490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 shutdown_connections 2023-10-14T18:14:27.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:27.808+0000 7ff36da52640 1 -- 172.21.15.29:0/670512508 wait complete. 2023-10-14T18:14:27.827 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:28.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.052+0000 7f3cd379e640 1 Processor -- start 2023-10-14T18:14:28.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.052+0000 7f3cd379e640 1 -- start start 2023-10-14T18:14:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd379e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0adb50 0x7f3ccc06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd379e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ccc0b0b20 0x7f3ccc164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd379e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ccc071450 con 0x7f3ccc070fd0 2023-10-14T18:14:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd379e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ccc0cc600 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd379e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0cc8e0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd1d3b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ccc0b0b20 0x7f3ccc164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd0d39640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3ccc070fd0 0x7f3ccc0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40448/0 (socket says 172.21.15.29:40448) 2023-10-14T18:14:28.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.054+0000 7f3cd0d39640 1 -- 172.21.15.29:0/4138776435 learned_addr learned my addr 172.21.15.29:0/4138776435 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cd153a640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0adb50 0x7f3ccc06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cd1d3b640 1 -- 172.21.15.29:0/4138776435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0adb50 msgr2=0x7f3ccc06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:28.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cd1d3b640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0adb50 0x7f3ccc06f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cd1d3b640 1 -- 172.21.15.29:0/4138776435 >> v1:172.21.15.29:6789/0 conn(0x7f3ccc070fd0 legacy=0x7f3ccc0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cd1d3b640 1 -- 172.21.15.29:0/4138776435 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ccc0ccbc0 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.055+0000 7f3cc3fff640 1 -- 172.21.15.29:0/4138776435 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241142372 0 0) 0x7f3ccc071450 con 0x7f3ccc070fd0 2023-10-14T18:14:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.056+0000 7f3cd1d3b640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0b0b20 0x7f3ccc164f60 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009d90 tx=0x7f3cc402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e23fce2e0d005a8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.056+0000 7f3cc3fff640 1 -- 172.21.15.29:0/4138776435 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc402cc50 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.056+0000 7f3cc3fff640 1 -- 172.21.15.29:0/4138776435 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cc402cdb0 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.056+0000 7f3cc3fff640 1 -- 172.21.15.29:0/4138776435 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cc4034660 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 -- 172.21.15.29:0/4138776435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0b0b20 msgr2=0x7f3ccc164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0b0b20 0x7f3ccc164f60 secure :-1 s=READY pgs=2297 cs=0 l=1 rev1=1 crypto rx=0x7f3cc4009d90 tx=0x7f3cc402bea0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 -- 172.21.15.29:0/4138776435 shutdown_connections 2023-10-14T18:14:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0b0b20 0x7f3ccc164f60 unknown :-1 s=CLOSED pgs=2297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 --2- 172.21.15.29:0/4138776435 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0adb50 0x7f3ccc06f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 -- 172.21.15.29:0/4138776435 >> 172.21.15.29:0/4138776435 conn(0x7f3ccc0cb410 msgr2=0x7f3ccc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:28.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 -- 172.21.15.29:0/4138776435 shutdown_connections 2023-10-14T18:14:28.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.057+0000 7f3cd379e640 1 -- 172.21.15.29:0/4138776435 wait complete. 2023-10-14T18:14:28.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.059+0000 7f3cd379e640 1 Processor -- start 2023-10-14T18:14:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.059+0000 7f3cd379e640 1 -- start start 2023-10-14T18:14:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ccc070fd0 0x7f3ccc0c3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0b0b20 0x7f3ccc152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c29d0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c2cb0 con 0x7f3ccc070fd0 2023-10-14T18:14:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd379e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ccc0c2f90 con 0x7f3ccc0b0b20 2023-10-14T18:14:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd0d39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd0d39640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43660/0 (socket says 172.21.15.29:43660) 2023-10-14T18:14:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.060+0000 7f3cd0d39640 1 -- 172.21.15.29:0/2994816336 learned_addr learned my addr 172.21.15.29:0/2994816336 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd153a640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ccc070fd0 0x7f3ccc0c3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd1d3b640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0b0b20 0x7f3ccc152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 -- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0b0b20 msgr2=0x7f3ccc152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0b0b20 0x7f3ccc152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 -- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ccc070fd0 msgr2=0x7f3ccc0c3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ccc070fd0 0x7f3ccc0c3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 -- 172.21.15.29:0/2994816336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cc4009a70 con 0x7f3ccc0adb50 2023-10-14T18:14:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd0d39640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c430 tx=0x7f3cbc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc0074e0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3cbc007660 con 0x7f3ccc0adb50 2023-10-14T18:14:28.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cbc004a20 con 0x7f3ccc0adb50 2023-10-14T18:14:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.061+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ccc0c3270 con 0x7f3ccc0adb50 2023-10-14T18:14:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.062+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ccc1525c0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.063+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ccc0c8860 con 0x7f3ccc0adb50 2023-10-14T18:14:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.064+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 120) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3cbc004b80 con 0x7f3ccc0adb50 2023-10-14T18:14:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.064+0000 7f3cc1ffb640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 0x7f3cb006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.065+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7f3cbc089630 con 0x7f3ccc0adb50 2023-10-14T18:14:28.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.065+0000 7f3cd153a640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 0x7f3cb006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.066+0000 7f3cd153a640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 0x7f3cb006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006fd0 tx=0x7f3cc8008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.069+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3cbc0544b0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.307+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ccc04e160 con 0x7f3ccc0adb50 2023-10-14T18:14:28.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.308+0000 7f3cc1ffb640 1 -- 172.21.15.29:0/2994816336 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f3cbc057ad0 con 0x7f3ccc0adb50 2023-10-14T18:14:28.310 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:28.310 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T18:14:28.310 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T18:14:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 msgr2=0x7f3cb006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 0x7f3cb006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3cc8006fd0 tx=0x7f3cc8008040 comp rx=0 tx=0).stop 2023-10-14T18:14:28.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 msgr2=0x7f3ccc0c22d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 secure :-1 s=READY pgs=2298 cs=0 l=1 rev1=1 crypto rx=0x7f3cbc00c430 tx=0x7f3cbc00c8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 shutdown_connections 2023-10-14T18:14:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3cb006acc0 0x7f3cb006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ccc0b0b20 0x7f3ccc152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ccc0adb50 0x7f3ccc0c22d0 unknown :-1 s=CLOSED pgs=2298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 --2- 172.21.15.29:0/2994816336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ccc070fd0 0x7f3ccc0c3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 >> 172.21.15.29:0/2994816336 conn(0x7f3ccc0cb410 msgr2=0x7f3ccc06cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 shutdown_connections 2023-10-14T18:14:28.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.311+0000 7f3cd379e640 1 -- 172.21.15.29:0/2994816336 wait complete. 2023-10-14T18:14:28.329 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.555+0000 7f52f3fff640 1 Processor -- start 2023-10-14T18:14:28.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.555+0000 7f52f3fff640 1 -- start start 2023-10-14T18:14:28.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f52f4153e20 0x7f52f40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52f4070950 0x7f52f40cd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f3fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52f40cd490 con 0x7f52f40addd0 2023-10-14T18:14:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40aec90 con 0x7f52f4070950 2023-10-14T18:14:28.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f3fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52f40ae3f0 con 0x7f52f4153e20 2023-10-14T18:14:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52f4070950 0x7f52f40cd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f37fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f52f4070950 0x7f52f40cd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43676/0 (socket says 172.21.15.29:43676) 2023-10-14T18:14:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f37fe640 1 -- 172.21.15.29:0/1100780099 learned_addr learned my addr 172.21.15.29:0/1100780099 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.557+0000 7f52f2ffd640 1 --2- 172.21.15.29:0/1100780099 >> v2:172.21.15.186:3300/0 conn(0x7f52f4153e20 0x7f52f40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.558+0000 7f52f37fe640 1 -- 172.21.15.29:0/1100780099 >> v2:172.21.15.186:3300/0 conn(0x7f52f4153e20 msgr2=0x7f52f40ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.558+0000 7f52f37fe640 1 --2- 172.21.15.29:0/1100780099 >> v2:172.21.15.186:3300/0 conn(0x7f52f4153e20 0x7f52f40ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.558+0000 7f52f37fe640 1 -- 172.21.15.29:0/1100780099 >> v1:172.21.15.186:6789/0 conn(0x7f52f40addd0 legacy=0x7f52f4070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:28.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.558+0000 7f52f37fe640 1 -- 172.21.15.29:0/1100780099 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52f40ae6d0 con 0x7f52f4070950 2023-10-14T18:14:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.559+0000 7f52f37fe640 1 --2- 172.21.15.29:0/1100780099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4070950 0x7f52f40cd920 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f52f40acb00 tx=0x7f52ec02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25aca36fbb89f6ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f1ffb640 1 -- 172.21.15.29:0/1100780099 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec031870 con 0x7f52f4070950 2023-10-14T18:14:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f1ffb640 1 -- 172.21.15.29:0/1100780099 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec0319d0 con 0x7f52f4070950 2023-10-14T18:14:28.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f1ffb640 1 -- 172.21.15.29:0/1100780099 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec031d60 con 0x7f52f4070950 2023-10-14T18:14:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 -- 172.21.15.29:0/1100780099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4070950 msgr2=0x7f52f40cd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 --2- 172.21.15.29:0/1100780099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4070950 0x7f52f40cd920 secure :-1 s=READY pgs=2299 cs=0 l=1 rev1=1 crypto rx=0x7f52f40acb00 tx=0x7f52ec02b910 comp rx=0 tx=0).stop 2023-10-14T18:14:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 -- 172.21.15.29:0/1100780099 shutdown_connections 2023-10-14T18:14:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 --2- 172.21.15.29:0/1100780099 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4070950 0x7f52f40cd920 unknown :-1 s=CLOSED pgs=2299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 --2- 172.21.15.29:0/1100780099 >> v2:172.21.15.186:3300/0 conn(0x7f52f4153e20 0x7f52f40ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.560+0000 7f52f3fff640 1 -- 172.21.15.29:0/1100780099 >> 172.21.15.29:0/1100780099 conn(0x7f52f40cc230 msgr2=0x7f52f40cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.561+0000 7f52f3fff640 1 -- 172.21.15.29:0/1100780099 shutdown_connections 2023-10-14T18:14:28.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.561+0000 7f52f3fff640 1 -- 172.21.15.29:0/1100780099 wait complete. 2023-10-14T18:14:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.562+0000 7f52f3fff640 1 Processor -- start 2023-10-14T18:14:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.562+0000 7f52f3fff640 1 -- start start 2023-10-14T18:14:28.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52f4070950 0x7f52f4150810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f40addd0 0x7f52f41f3400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f41f31a0 con 0x7f52f4153e20 2023-10-14T18:14:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40c4950 con 0x7f52f40addd0 2023-10-14T18:14:28.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.563+0000 7f52f3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52f40b3770 con 0x7f52f4070950 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43690/0 (socket says 172.21.15.29:43690) 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 -- 172.21.15.29:0/2274436502 learned_addr learned my addr 172.21.15.29:0/2274436502 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 -- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52f4070950 msgr2=0x7f52f4150810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52f4070950 0x7f52f4150810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 -- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f40addd0 msgr2=0x7f52f41f3400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f40addd0 0x7f52f41f3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.564+0000 7f52f37fe640 1 -- 172.21.15.29:0/2274436502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec0094d0 con 0x7f52f4153e20 2023-10-14T18:14:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52f37fe640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0090b0 tx=0x7f52ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec03ac60 con 0x7f52f4153e20 2023-10-14T18:14:28.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52ec03adc0 con 0x7f52f4153e20 2023-10-14T18:14:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52ec044630 con 0x7f52f4153e20 2023-10-14T18:14:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52f40b3a50 con 0x7f52f4153e20 2023-10-14T18:14:28.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.565+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52f40b4010 con 0x7f52f4153e20 2023-10-14T18:14:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.566+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52f404e6d0 con 0x7f52f4153e20 2023-10-14T18:14:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.567+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 120) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f52ec04d450 con 0x7f52f4153e20 2023-10-14T18:14:28.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.568+0000 7f52eb7fe640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 0x7f52d406d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.568+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7f52ec02d040 con 0x7f52f4153e20 2023-10-14T18:14:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.568+0000 7f52f2ffd640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 0x7f52d406d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:28.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.569+0000 7f52f2ffd640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 0x7f52d406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52e40070b0 tx=0x7f52e4007040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:28.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.573+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f52ec076c00 con 0x7f52f4153e20 2023-10-14T18:14:28.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.773+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52f404e9b0 con 0x7f52f4153e20 2023-10-14T18:14:28.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.776+0000 7f52eb7fe640 1 -- 172.21.15.29:0/2274436502 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18310 (secure 0 0 0) 0x7f52ec035070 con 0x7f52f4153e20 2023-10-14T18:14:28.778 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:28.778 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":197,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:26.920569+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3286846969":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1757938149":"2023-10-15T18:14:26.920448+0000","172.21.15.29:6833/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1560276409":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1378572596":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2299724842":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/614546910":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1466742201":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2776961501":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/287831498":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/841868971":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1377837433":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3575832370":"2023-10-15T18:09:22.121284+0000","172.21.15.29:6833/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2946416628":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/730049499":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6833/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3534626168":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/1930442848":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2824949307":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2461249058":"2023-10-15T17:58:40.567922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:28.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 msgr2=0x7f52d406d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 0x7f52d406d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f52e40070b0 tx=0x7f52e4007040 comp rx=0 tx=0).stop 2023-10-14T18:14:28.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 msgr2=0x7f52f41563d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 secure :-1 s=READY pgs=2300 cs=0 l=1 rev1=1 crypto rx=0x7f52ec0090b0 tx=0x7f52ec0046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 shutdown_connections 2023-10-14T18:14:28.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f52f4153e20 0x7f52f41563d0 unknown :-1 s=CLOSED pgs=2300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f52f40addd0 0x7f52f41f3400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f52d406acc0 0x7f52d406d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 --2- 172.21.15.29:0/2274436502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f52f4070950 0x7f52f4150810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.779+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 >> 172.21.15.29:0/2274436502 conn(0x7f52f40cc230 msgr2=0x7f52f40ce8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:28.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.780+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 shutdown_connections 2023-10-14T18:14:28.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:28.780+0000 7f52f3fff640 1 -- 172.21.15.29:0/2274436502 wait complete. 2023-10-14T18:14:28.799 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:14:28.799 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.022+0000 7f3e22f47640 1 Processor -- start 2023-10-14T18:14:29.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.022+0000 7f3e22f47640 1 -- start start 2023-10-14T18:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3e1c153e20 0x7f3e1c0ad890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e1c070780 0x7f3e1c160850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e1c070c10 con 0x7f3e1c070780 2023-10-14T18:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e1c0cd490 con 0x7f3e1c153e20 2023-10-14T18:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.023+0000 7f3e22f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0aec90 con 0x7f3e1c0addd0 2023-10-14T18:14:29.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e1c070780 0x7f3e1c160850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3e1c070780 0x7f3e1c160850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43694/0 (socket says 172.21.15.29:43694) 2023-10-14T18:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 -- 172.21.15.29:0/868271502 learned_addr learned my addr 172.21.15.29:0/868271502 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e21f45640 1 --2- 172.21.15.29:0/868271502 >> v2:172.21.15.186:3300/0 conn(0x7f3e1c153e20 0x7f3e1c0ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 -- 172.21.15.29:0/868271502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 msgr2=0x7f3e1c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:29.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e21744640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 -- 172.21.15.29:0/868271502 >> v2:172.21.15.186:3300/0 conn(0x7f3e1c153e20 msgr2=0x7f3e1c0ad890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:29.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 --2- 172.21.15.29:0/868271502 >> v2:172.21.15.186:3300/0 conn(0x7f3e1c153e20 0x7f3e1c0ad890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.024+0000 7f3e22746640 1 -- 172.21.15.29:0/868271502 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0ae3f0 con 0x7f3e1c070780 2023-10-14T18:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.025+0000 7f3e22746640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c160850 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0acb00 tx=0x7f3e1402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab212c06f1210bc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.026+0000 7f3e20f43640 1 -- 172.21.15.29:0/868271502 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1402ca70 con 0x7f3e1c070780 2023-10-14T18:14:29.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.026+0000 7f3e20f43640 1 -- 172.21.15.29:0/868271502 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1402cbd0 con 0x7f3e1c070780 2023-10-14T18:14:29.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.026+0000 7f3e20f43640 1 -- 172.21.15.29:0/868271502 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e140343f0 con 0x7f3e1c070780 2023-10-14T18:14:29.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.026+0000 7f3e22f47640 1 -- 172.21.15.29:0/868271502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 msgr2=0x7f3e1c160850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.026+0000 7f3e22f47640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c160850 secure :-1 s=READY pgs=2301 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c0acb00 tx=0x7f3e1402bed0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 -- 172.21.15.29:0/868271502 shutdown_connections 2023-10-14T18:14:29.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c160850 unknown :-1 s=CLOSED pgs=2301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 --2- 172.21.15.29:0/868271502 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 --2- 172.21.15.29:0/868271502 >> v2:172.21.15.186:3300/0 conn(0x7f3e1c153e20 0x7f3e1c0ad890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 -- 172.21.15.29:0/868271502 >> 172.21.15.29:0/868271502 conn(0x7f3e1c0cc230 msgr2=0x7f3e1c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:29.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 -- 172.21.15.29:0/868271502 shutdown_connections 2023-10-14T18:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.027+0000 7f3e22f47640 1 -- 172.21.15.29:0/868271502 wait complete. 2023-10-14T18:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.029+0000 7f3e22f47640 1 Processor -- start 2023-10-14T18:14:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.029+0000 7f3e22f47640 1 -- start start 2023-10-14T18:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c0c3d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e1c153e20 0x7f3e1c142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0b2440 con 0x7f3e1c070780 2023-10-14T18:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0c38a0 con 0x7f3e1c153e20 2023-10-14T18:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c0b5320 con 0x7f3e1c0addd0 2023-10-14T18:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e21f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e21f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43710/0 (socket says 172.21.15.29:43710) 2023-10-14T18:14:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e21f45640 1 -- 172.21.15.29:0/2960779286 learned_addr learned my addr 172.21.15.29:0/2960779286 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.030+0000 7f3e22746640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e1c153e20 0x7f3e1c142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21744640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c0c3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 -- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 msgr2=0x7f3e1c0c3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c0c3d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 -- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e1c153e20 msgr2=0x7f3e1c142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e1c153e20 0x7f3e1c142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 -- 172.21.15.29:0/2960779286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e14009b30 con 0x7f3e1c070780 2023-10-14T18:14:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e21f45640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800d5f0 tx=0x7f3e1800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e180125f0 con 0x7f3e1c070780 2023-10-14T18:14:29.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e18012750 con 0x7f3e1c070780 2023-10-14T18:14:29.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e18012b00 con 0x7f3e1c070780 2023-10-14T18:14:29.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.031+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e1c0c4270 con 0x7f3e1c070780 2023-10-14T18:14:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.032+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e1c143150 con 0x7f3e1c070780 2023-10-14T18:14:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.033+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e1c04e6d0 con 0x7f3e1c070780 2023-10-14T18:14:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.034+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 120) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3e18012c60 con 0x7f3e1c070780 2023-10-14T18:14:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.035+0000 7f3e127fc640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 0x7f3e0006d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.035+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7f3e18089be0 con 0x7f3e1c070780 2023-10-14T18:14:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.035+0000 7f3e21744640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 0x7f3e0006d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.036+0000 7f3e21744640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 0x7f3e0006d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c150e70 tx=0x7f3e0c008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.039+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3e18054a60 con 0x7f3e1c070780 2023-10-14T18:14:29.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.282+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e1c0cf370 con 0x7f3e1c070780 2023-10-14T18:14:29.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.283+0000 7f3e127fc640 1 -- 172.21.15.29:0/2960779286 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7f3e18058080 con 0x7f3e1c070780 2023-10-14T18:14:29.285 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:29.285 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T18:14:29.285 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T18:14:29.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 msgr2=0x7f3e0006d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 0x7f3e0006d1b0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c150e70 tx=0x7f3e0c008040 comp rx=0 tx=0).stop 2023-10-14T18:14:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 msgr2=0x7f3e1c0b4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 secure :-1 s=READY pgs=2302 cs=0 l=1 rev1=1 crypto rx=0x7f3e1800d5f0 tx=0x7f3e1800dab0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 shutdown_connections 2023-10-14T18:14:29.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3e1c153e20 0x7f3e1c142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f3e0006acf0 0x7f3e0006d1b0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3e1c0addd0 0x7f3e1c0c3d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 --2- 172.21.15.29:0/2960779286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3e1c070780 0x7f3e1c0b4de0 unknown :-1 s=CLOSED pgs=2302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.286+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 >> 172.21.15.29:0/2960779286 conn(0x7f3e1c0cc230 msgr2=0x7f3e1c152570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.287+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 shutdown_connections 2023-10-14T18:14:29.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.287+0000 7f3e22f47640 1 -- 172.21.15.29:0/2960779286 wait complete. 2023-10-14T18:14:29.306 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.525+0000 7fddb4e0e640 1 Processor -- start 2023-10-14T18:14:29.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.525+0000 7fddb4e0e640 1 -- start start 2023-10-14T18:14:29.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.526+0000 7fddb4e0e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddb00cd0c0 0x7fddb01615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.526+0000 7fddb4e0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fddb015db00 0x7fddb00b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.526+0000 7fddb4e0e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb015e3e0 con 0x7fddb00b0410 2023-10-14T18:14:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.526+0000 7fddb4e0e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb00b0de0 con 0x7fddb00cd0c0 2023-10-14T18:14:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.526+0000 7fddb4e0e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb00cd4a0 con 0x7fddb015db00 2023-10-14T18:14:29.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddb00cd0c0 0x7fddb01615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddb00cd0c0 0x7fddb01615f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43750/0 (socket says 172.21.15.29:43750) 2023-10-14T18:14:29.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 -- 172.21.15.29:0/65688466 learned_addr learned my addr 172.21.15.29:0/65688466 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddaed9d640 1 --2- 172.21.15.29:0/65688466 >> v2:172.21.15.186:3300/0 conn(0x7fddb015db00 0x7fddb00b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 -- 172.21.15.29:0/65688466 >> v2:172.21.15.186:3300/0 conn(0x7fddb015db00 msgr2=0x7fddb00b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 --2- 172.21.15.29:0/65688466 >> v2:172.21.15.186:3300/0 conn(0x7fddb015db00 0x7fddb00b08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 -- 172.21.15.29:0/65688466 >> v1:172.21.15.29:6789/0 conn(0x7fddb00b0410 legacy=0x7fddb015d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.527+0000 7fddae59c640 1 -- 172.21.15.29:0/65688466 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb015e100 con 0x7fddb00cd0c0 2023-10-14T18:14:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.528+0000 7fddae59c640 1 --2- 172.21.15.29:0/65688466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb00cd0c0 0x7fddb01615f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fddb015afc0 tx=0x7fdda002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18d7ea8dab71973 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.529+0000 7fddad59a640 1 -- 172.21.15.29:0/65688466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0031870 con 0x7fddb00cd0c0 2023-10-14T18:14:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.529+0000 7fddad59a640 1 -- 172.21.15.29:0/65688466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda00319d0 con 0x7fddb00cd0c0 2023-10-14T18:14:29.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.529+0000 7fddad59a640 1 -- 172.21.15.29:0/65688466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda0031d60 con 0x7fddb00cd0c0 2023-10-14T18:14:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.529+0000 7fddb4e0e640 1 -- 172.21.15.29:0/65688466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb00cd0c0 msgr2=0x7fddb01615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.529+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/65688466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb00cd0c0 0x7fddb01615f0 secure :-1 s=READY pgs=2306 cs=0 l=1 rev1=1 crypto rx=0x7fddb015afc0 tx=0x7fdda002b910 comp rx=0 tx=0).stop 2023-10-14T18:14:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 -- 172.21.15.29:0/65688466 shutdown_connections 2023-10-14T18:14:29.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/65688466 >> v2:172.21.15.186:3300/0 conn(0x7fddb015db00 0x7fddb00b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/65688466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb00cd0c0 0x7fddb01615f0 unknown :-1 s=CLOSED pgs=2306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 -- 172.21.15.29:0/65688466 >> 172.21.15.29:0/65688466 conn(0x7fddb00cb410 msgr2=0x7fddb00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:29.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 -- 172.21.15.29:0/65688466 shutdown_connections 2023-10-14T18:14:29.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.530+0000 7fddb4e0e640 1 -- 172.21.15.29:0/65688466 wait complete. 2023-10-14T18:14:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.531+0000 7fddb4e0e640 1 Processor -- start 2023-10-14T18:14:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.532+0000 7fddb4e0e640 1 -- start start 2023-10-14T18:14:29.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.532+0000 7fddb4e0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 0x7fddb0160410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.532+0000 7fddb4e0e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb00cd0c0 0x7fddb015f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.532+0000 7fddb4e0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddb4e0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb015fbc0 con 0x7fddb015db00 2023-10-14T18:14:29.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddb4e0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01e7980 con 0x7fddb00b0410 2023-10-14T18:14:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddb4e0e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb015f940 con 0x7fddb00cd0c0 2023-10-14T18:14:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43766/0 (socket says 172.21.15.29:43766) 2023-10-14T18:14:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 -- 172.21.15.29:0/3636888755 learned_addr learned my addr 172.21.15.29:0/3636888755 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:29.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddae59c640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 0x7fddb0160410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 -- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb00cd0c0 msgr2=0x7fddb015f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddadd9b640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb00cd0c0 0x7fddb015f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb00cd0c0 0x7fddb015f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 -- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 msgr2=0x7fddb0160410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 0x7fddb0160410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddaed9d640 1 -- 172.21.15.29:0/3636888755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda00094d0 con 0x7fddb015db00 2023-10-14T18:14:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.533+0000 7fddae59c640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 0x7fddb0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fddaed9d640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fdda800c820 tx=0x7fdda800cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8004020 con 0x7fddb015db00 2023-10-14T18:14:29.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda8004180 con 0x7fddb015db00 2023-10-14T18:14:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdda8007830 con 0x7fddb015db00 2023-10-14T18:14:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb01e7050 con 0x7fddb015db00 2023-10-14T18:14:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.534+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb01e7610 con 0x7fddb015db00 2023-10-14T18:14:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.535+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb01e7b10 con 0x7fddb015db00 2023-10-14T18:14:29.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.536+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 121) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fdda8007990 con 0x7fddb015db00 2023-10-14T18:14:29.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.537+0000 7fdd9effd640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 0x7fdd9006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.537+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7fdda8089910 con 0x7fddb015db00 2023-10-14T18:14:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.538+0000 7fddae59c640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 0x7fdd9006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.538+0000 7fddae59c640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 0x7fdd9006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdda0036000 tx=0x7fdda003c040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:29.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.542+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdda80543b0 con 0x7fddb015db00 2023-10-14T18:14:29.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.775+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddb004e160 con 0x7fddb015db00 2023-10-14T18:14:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.776+0000 7fdd9effd640 1 -- 172.21.15.29:0/3636888755 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+4924 (secure 0 0 0) 0x7fdda80579d0 con 0x7fddb015db00 2023-10-14T18:14:29.777 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:29.778 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":91,"default_fscid":11,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T17:58:25.535658+0000","modified":"2023-10-14T17:58:28.638035+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":10128},"failed":[],"damaged":[],"stopped":[],"info":{"gid_10128":{"gid":10128,"name":"c","rank":0,"incarnation":88,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/2108287962","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":2108287962},{"type":"v1","addr":"172.21.15.29:6837","nonce":2108287962}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[23],"metadata_pool":22,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":11}]} 2023-10-14T18:14:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 91 2023-10-14T18:14:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 msgr2=0x7fdd9006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 0x7fdd9006d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fdda0036000 tx=0x7fdda003c040 comp rx=0 tx=0).stop 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 msgr2=0x7fddb01e7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 secure :-1 s=READY pgs=2307 cs=0 l=1 rev1=1 crypto rx=0x7fdda800c820 tx=0x7fdda800cce0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 shutdown_connections 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddb015db00 0x7fddb01e7e10 unknown :-1 s=CLOSED pgs=2307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdd9006af00 0x7fdd9006d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddb00cd0c0 0x7fddb015f400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 --2- 172.21.15.29:0/3636888755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddb00b0410 0x7fddb0160410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 >> 172.21.15.29:0/3636888755 conn(0x7fddb00cb410 msgr2=0x7fddb0155c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 shutdown_connections 2023-10-14T18:14:29.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:29.779+0000 7fddb4e0e640 1 -- 172.21.15.29:0/3636888755 wait complete. 2023-10-14T18:14:29.800 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:30.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.030+0000 7fc28411d640 1 Processor -- start 2023-10-14T18:14:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.031+0000 7fc28411d640 1 -- start start 2023-10-14T18:14:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.032+0000 7fc28411d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc27c15adc0 0x7fc27c0cf380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.032+0000 7fc28411d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc27c0ad990 0x7fc27c15f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.032+0000 7fc28411d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc27c0b4380 con 0x7fc27c070c70 2023-10-14T18:14:30.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.032+0000 7fc28411d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27c0b4620 con 0x7fc27c15adc0 2023-10-14T18:14:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.032+0000 7fc28411d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc27c15e9b0 con 0x7fc27c0ad990 2023-10-14T18:14:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc27c15adc0 0x7fc27c0cf380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc27c15adc0 0x7fc27c0cf380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43778/0 (socket says 172.21.15.29:43778) 2023-10-14T18:14:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 -- 172.21.15.29:0/524980349 learned_addr learned my addr 172.21.15.29:0/524980349 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 -- 172.21.15.29:0/524980349 >> v2:172.21.15.186:3300/0 conn(0x7fc27c0ad990 msgr2=0x7fc27c15f490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 --2- 172.21.15.29:0/524980349 >> v2:172.21.15.186:3300/0 conn(0x7fc27c0ad990 0x7fc27c15f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 -- 172.21.15.29:0/524980349 >> v1:172.21.15.186:6789/0 conn(0x7fc27c070c70 legacy=0x7fc27c0710a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.033+0000 7fc281eb9640 1 -- 172.21.15.29:0/524980349 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc27c15ec90 con 0x7fc27c15adc0 2023-10-14T18:14:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.034+0000 7fc281eb9640 1 --2- 172.21.15.29:0/524980349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c15adc0 0x7fc27c0cf380 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fc27c071910 tx=0x7fc27802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1898208f74a332dc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.034+0000 7fc280eb7640 1 -- 172.21.15.29:0/524980349 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278031870 con 0x7fc27c15adc0 2023-10-14T18:14:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.034+0000 7fc280eb7640 1 -- 172.21.15.29:0/524980349 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2780319d0 con 0x7fc27c15adc0 2023-10-14T18:14:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.034+0000 7fc280eb7640 1 -- 172.21.15.29:0/524980349 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc278031d60 con 0x7fc27c15adc0 2023-10-14T18:14:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 -- 172.21.15.29:0/524980349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c15adc0 msgr2=0x7fc27c0cf380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 --2- 172.21.15.29:0/524980349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c15adc0 0x7fc27c0cf380 secure :-1 s=READY pgs=2308 cs=0 l=1 rev1=1 crypto rx=0x7fc27c071910 tx=0x7fc27802b910 comp rx=0 tx=0).stop 2023-10-14T18:14:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 -- 172.21.15.29:0/524980349 shutdown_connections 2023-10-14T18:14:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 --2- 172.21.15.29:0/524980349 >> v2:172.21.15.186:3300/0 conn(0x7fc27c0ad990 0x7fc27c15f490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 --2- 172.21.15.29:0/524980349 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c15adc0 0x7fc27c0cf380 unknown :-1 s=CLOSED pgs=2308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 -- 172.21.15.29:0/524980349 >> 172.21.15.29:0/524980349 conn(0x7fc27c0ce860 msgr2=0x7fc27c0cec70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 -- 172.21.15.29:0/524980349 shutdown_connections 2023-10-14T18:14:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.035+0000 7fc28411d640 1 -- 172.21.15.29:0/524980349 wait complete. 2023-10-14T18:14:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.037+0000 7fc28411d640 1 Processor -- start 2023-10-14T18:14:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.037+0000 7fc28411d640 1 -- start start 2023-10-14T18:14:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 0x7fc27c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc27c15adc0 0x7fc27c0c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c1f1990 con 0x7fc27c0ad990 2023-10-14T18:14:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c15c7c0 con 0x7fc27c15adc0 2023-10-14T18:14:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc28411d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc27c15e130 con 0x7fc27c070c70 2023-10-14T18:14:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc2816b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc2816b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43780/0 (socket says 172.21.15.29:43780) 2023-10-14T18:14:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc2816b8640 1 -- 172.21.15.29:0/517533564 learned_addr learned my addr 172.21.15.29:0/517533564 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.038+0000 7fc281eb9640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 0x7fc27c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2826ba640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc27c15adc0 0x7fc27c0c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 -- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 msgr2=0x7fc27c0b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 0x7fc27c0b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 -- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc27c15adc0 msgr2=0x7fc27c0c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc27c15adc0 0x7fc27c0c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 -- 172.21.15.29:0/517533564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2780094d0 con 0x7fc27c0ad990 2023-10-14T18:14:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc281eb9640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 0x7fc27c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2816b8640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00b4a0 tx=0x7fc26c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c010610 con 0x7fc27c0ad990 2023-10-14T18:14:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc26c010770 con 0x7fc27c0ad990 2023-10-14T18:14:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.039+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26c010b20 con 0x7fc27c0ad990 2023-10-14T18:14:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.040+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc27c15b320 con 0x7fc27c0ad990 2023-10-14T18:14:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.040+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc27c15c3e0 con 0x7fc27c0ad990 2023-10-14T18:14:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.041+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc27c04e160 con 0x7fc27c0ad990 2023-10-14T18:14:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.042+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 121) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fc26c010c80 con 0x7fc27c0ad990 2023-10-14T18:14:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.043+0000 7fc2727fc640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 0x7fc26006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.043+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7fc26c089660 con 0x7fc27c0ad990 2023-10-14T18:14:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.043+0000 7fc281eb9640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 0x7fc26006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.044+0000 7fc281eb9640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 0x7fc26006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc278036000 tx=0x7fc27803c040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.048+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc26c054310 con 0x7fc27c0ad990 2023-10-14T18:14:30.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.247+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc27c06d550 con 0x7fc27c0ad990 2023-10-14T18:14:30.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.250+0000 7fc2727fc640 1 -- 172.21.15.29:0/517533564 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v197) v1 ==== 74+0+18310 (secure 0 0 0) 0x7fc26c05ce60 con 0x7fc27c0ad990 2023-10-14T18:14:30.252 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:30.252 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":197,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:26.920569+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":22,"pool_name":"cephfs_metadata","create_time":"2023-10-14T17:58:23.469364+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"191","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2,"score_stable":2,"optimal_score":1,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":23,"pool_name":"cephfs_data","create_time":"2023-10-14T17:58:24.568568+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"196","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":33,"snap_epoch":196,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/3286846969":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1757938149":"2023-10-15T18:14:26.920448+0000","172.21.15.29:6833/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1560276409":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1378572596":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2299724842":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/614546910":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1466742201":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2776961501":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/287831498":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/841868971":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1377837433":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3575832370":"2023-10-15T18:09:22.121284+0000","172.21.15.29:6833/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2946416628":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/730049499":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6833/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3534626168":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/1930442848":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2824949307":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2461249058":"2023-10-15T17:58:40.567922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":23,"snaps":[{"begin":2,"length":32}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 msgr2=0x7fc26006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 0x7fc26006d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc278036000 tx=0x7fc27803c040 comp rx=0 tx=0).stop 2023-10-14T18:14:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 msgr2=0x7fc27c0c4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 secure :-1 s=READY pgs=2309 cs=0 l=1 rev1=1 crypto rx=0x7fc26c00b4a0 tx=0x7fc26c00b960 comp rx=0 tx=0).stop 2023-10-14T18:14:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 shutdown_connections 2023-10-14T18:14:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc27c15adc0 0x7fc27c0c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc27c0ad990 0x7fc27c0c4b70 unknown :-1 s=CLOSED pgs=2309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fc26006af00 0x7fc26006d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 --2- 172.21.15.29:0/517533564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc27c070c70 0x7fc27c0b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 >> 172.21.15.29:0/517533564 conn(0x7fc27c0ce860 msgr2=0x7fc27c0cec40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:30.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 shutdown_connections 2023-10-14T18:14:30.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.253+0000 7fc28411d640 1 -- 172.21.15.29:0/517533564 wait complete. 2023-10-14T18:14:30.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T18:14:30.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.506+0000 7fddfa47e640 1 Processor -- start 2023-10-14T18:14:30.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.507+0000 7fddfa47e640 1 -- start start 2023-10-14T18:14:30.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddfa47e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddf406c040 0x7fddf406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddfa47e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fddf40ad970 0x7fddf415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddfa47e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf406c960 con 0x7fddf415ebc0 2023-10-14T18:14:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddfa47e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf415b9d0 con 0x7fddf406c040 2023-10-14T18:14:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddfa47e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf415bcb0 con 0x7fddf40ad970 2023-10-14T18:14:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddf947c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddf406c040 0x7fddf406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddf947c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fddf406c040 0x7fddf406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43790/0 (socket says 172.21.15.29:43790) 2023-10-14T18:14:30.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.508+0000 7fddf947c640 1 -- 172.21.15.29:0/2394748399 learned_addr learned my addr 172.21.15.29:0/2394748399 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.509+0000 7fddf947c640 1 -- 172.21.15.29:0/2394748399 >> v2:172.21.15.186:3300/0 conn(0x7fddf40ad970 msgr2=0x7fddf415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.509+0000 7fddf947c640 1 --2- 172.21.15.29:0/2394748399 >> v2:172.21.15.186:3300/0 conn(0x7fddf40ad970 0x7fddf415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.509+0000 7fddf947c640 1 -- 172.21.15.29:0/2394748399 >> v1:172.21.15.186:6789/0 conn(0x7fddf415ebc0 legacy=0x7fddf40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:30.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.509+0000 7fddf947c640 1 -- 172.21.15.29:0/2394748399 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf415bf90 con 0x7fddf406c040 2023-10-14T18:14:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.510+0000 7fddf947c640 1 --2- 172.21.15.29:0/2394748399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf406c040 0x7fddf406c420 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fddf40cb060 tx=0x7fdde402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19c644306834e39c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.510+0000 7fddf3fff640 1 -- 172.21.15.29:0/2394748399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde402ca80 con 0x7fddf406c040 2023-10-14T18:14:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.510+0000 7fddf3fff640 1 -- 172.21.15.29:0/2394748399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde402cbe0 con 0x7fddf406c040 2023-10-14T18:14:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.510+0000 7fddf3fff640 1 -- 172.21.15.29:0/2394748399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdde40343f0 con 0x7fddf406c040 2023-10-14T18:14:30.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 -- 172.21.15.29:0/2394748399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf406c040 msgr2=0x7fddf406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2394748399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf406c040 0x7fddf406c420 secure :-1 s=READY pgs=2310 cs=0 l=1 rev1=1 crypto rx=0x7fddf40cb060 tx=0x7fdde402bee0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 -- 172.21.15.29:0/2394748399 shutdown_connections 2023-10-14T18:14:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2394748399 >> v2:172.21.15.186:3300/0 conn(0x7fddf40ad970 0x7fddf415e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2394748399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf406c040 0x7fddf406c420 unknown :-1 s=CLOSED pgs=2310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 -- 172.21.15.29:0/2394748399 >> 172.21.15.29:0/2394748399 conn(0x7fddf40ce6a0 msgr2=0x7fddf40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 -- 172.21.15.29:0/2394748399 shutdown_connections 2023-10-14T18:14:30.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.511+0000 7fddfa47e640 1 -- 172.21.15.29:0/2394748399 wait complete. 2023-10-14T18:14:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.513+0000 7fddfa47e640 1 Processor -- start 2023-10-14T18:14:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.513+0000 7fddfa47e640 1 -- start start 2023-10-14T18:14:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 0x7fddf4154950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf40ad970 0x7fddf41551a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40b24c0 con 0x7fddf415ebc0 2023-10-14T18:14:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40ca4d0 con 0x7fddf40ad970 2023-10-14T18:14:30.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddfa47e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf40b1e70 con 0x7fddf406c040 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddf9c7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddf9c7d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43798/0 (socket says 172.21.15.29:43798) 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddf9c7d640 1 -- 172.21.15.29:0/2087365992 learned_addr learned my addr 172.21.15.29:0/2087365992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddf8c7b640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf40ad970 0x7fddf41551a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.514+0000 7fddf947c640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 0x7fddf4154950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 -- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 msgr2=0x7fddf4154950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 0x7fddf4154950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 -- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf40ad970 msgr2=0x7fddf41551a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf40ad970 0x7fddf41551a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 -- 172.21.15.29:0/2087365992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4009b30 con 0x7fddf415ebc0 2023-10-14T18:14:30.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf947c640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 0x7fddf4154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf9c7d640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c390 tx=0x7fddec00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec0035f0 con 0x7fddf415ebc0 2023-10-14T18:14:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddec003750 con 0x7fddf415ebc0 2023-10-14T18:14:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf40b2150 con 0x7fddf415ebc0 2023-10-14T18:14:30.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec003b00 con 0x7fddf415ebc0 2023-10-14T18:14:30.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.515+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4154e90 con 0x7fddf415ebc0 2023-10-14T18:14:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.516+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf406b330 con 0x7fddf415ebc0 2023-10-14T18:14:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.517+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 121) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fddec003c60 con 0x7fddf415ebc0 2023-10-14T18:14:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.519+0000 7fddf1ffb640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 0x7fdde006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.519+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(197..197 src has 1..197) v4 ==== 6885+0+0 (secure 0 0 0) 0x7fddec0890f0 con 0x7fddf415ebc0 2023-10-14T18:14:30.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.519+0000 7fddf947c640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 0x7fdde006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.520+0000 7fddf947c640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 0x7fdde006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdde4035000 tx=0x7fdde4030000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.523+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fddec053ca0 con 0x7fddf415ebc0 2023-10-14T18:14:30.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.758+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fddf40b1cd0 con 0x7fddf415ebc0 2023-10-14T18:14:30.760 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:14:30.759+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:14:30.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.763+0000 7fddf1ffb640 1 -- 172.21.15.29:0/2087365992 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v92) v1 ==== 161+0+0 (secure 0 0 0) 0x7fddf40b1cd0 con 0x7fddf415ebc0 2023-10-14T18:14:30.764 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:14:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.766+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 msgr2=0x7fdde006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.766+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 0x7fdde006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fdde4035000 tx=0x7fdde4030000 comp rx=0 tx=0).stop 2023-10-14T18:14:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.766+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 msgr2=0x7fddf40c9800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.766+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 secure :-1 s=READY pgs=2311 cs=0 l=1 rev1=1 crypto rx=0x7fddec00c390 tx=0x7fddec00c850 comp rx=0 tx=0).stop 2023-10-14T18:14:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 shutdown_connections 2023-10-14T18:14:30.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fddf415ebc0 0x7fddf40c9800 unknown :-1 s=CLOSED pgs=2311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fddf40ad970 0x7fddf41551a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdde006af30 0x7fdde006d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 --2- 172.21.15.29:0/2087365992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fddf406c040 0x7fddf4154950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 >> 172.21.15.29:0/2087365992 conn(0x7fddf40ce6a0 msgr2=0x7fddf4159d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:30.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 shutdown_connections 2023-10-14T18:14:30.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.767+0000 7fddfa47e640 1 -- 172.21.15.29:0/2087365992 wait complete. 2023-10-14T18:14:30.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T18:14:30.864 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:14:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.987+0000 7f738376c640 1 Processor -- start 2023-10-14T18:14:30.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.987+0000 7f738376c640 1 -- start start 2023-10-14T18:14:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.988+0000 7f738376c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.988+0000 7f738376c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f737c0c9ca0 0x7f737c0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.988+0000 7f738376c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c0cb030 con 0x7f737c15ad70 2023-10-14T18:14:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f738376c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c0cae30 con 0x7f737c0c9ca0 2023-10-14T18:14:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f738376c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0c91c0 con 0x7f737c06a4b0 2023-10-14T18:14:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381508640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f737c15ad70 0x7f737c069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38132/0 (socket says 172.21.15.29:38132) 2023-10-14T18:14:30.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381508640 1 -- 172.21.15.29:0/3256867327 learned_addr learned my addr 172.21.15.29:0/3256867327 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381d09640 1 --2- 172.21.15.29:0/3256867327 >> v2:172.21.15.29:3300/0 conn(0x7f737c0c9ca0 0x7f737c0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7380d07640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f736ffff640 1 -- 172.21.15.29:0/3256867327 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090042960 0 0) 0x7f737c0cb030 con 0x7f737c15ad70 2023-10-14T18:14:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f736ffff640 1 -- 172.21.15.29:0/3256867327 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7368003520 con 0x7f737c15ad70 2023-10-14T18:14:30.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381d09640 1 -- 172.21.15.29:0/3256867327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 msgr2=0x7f737c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381d09640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381d09640 1 -- 172.21.15.29:0/3256867327 >> v1:172.21.15.29:6789/0 conn(0x7f737c15ad70 legacy=0x7f737c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.989+0000 7f7381d09640 1 -- 172.21.15.29:0/3256867327 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c0c94a0 con 0x7f737c0c9ca0 2023-10-14T18:14:30.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.991+0000 7f7381d09640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c0c9ca0 0x7f737c0ca8f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be0129d784c22ee1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.991+0000 7f736ffff640 1 -- 172.21.15.29:0/3256867327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737402cc50 con 0x7f737c0c9ca0 2023-10-14T18:14:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.991+0000 7f736ffff640 1 -- 172.21.15.29:0/3256867327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737402cdb0 con 0x7f737c0c9ca0 2023-10-14T18:14:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.991+0000 7f736ffff640 1 -- 172.21.15.29:0/3256867327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7374034660 con 0x7f737c0c9ca0 2023-10-14T18:14:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 -- 172.21.15.29:0/3256867327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c0c9ca0 msgr2=0x7f737c0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:30.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c0c9ca0 0x7f737c0ca8f0 secure :-1 s=READY pgs=2316 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f737402bea0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 -- 172.21.15.29:0/3256867327 shutdown_connections 2023-10-14T18:14:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c0c9ca0 0x7f737c0ca8f0 unknown :-1 s=CLOSED pgs=2316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 --2- 172.21.15.29:0/3256867327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 -- 172.21.15.29:0/3256867327 >> 172.21.15.29:0/3256867327 conn(0x7f737c0ce9b0 msgr2=0x7f737c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:30.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 -- 172.21.15.29:0/3256867327 shutdown_connections 2023-10-14T18:14:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.992+0000 7f738376c640 1 -- 172.21.15.29:0/3256867327 wait complete. 2023-10-14T18:14:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.994+0000 7f738376c640 1 Processor -- start 2023-10-14T18:14:30.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.994+0000 7f738376c640 1 -- start start 2023-10-14T18:14:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0a8d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f737c0c9ca0 0x7f737c0c7a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:30.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0ada20 con 0x7f737c15ad70 2023-10-14T18:14:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c14c6d0 con 0x7f737c0c9ca0 2023-10-14T18:14:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f738376c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0c82b0 con 0x7f737c06a4b0 2023-10-14T18:14:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38896/0 (socket says 172.21.15.29:38896) 2023-10-14T18:14:31.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 -- 172.21.15.29:0/1497860553 learned_addr learned my addr 172.21.15.29:0/1497860553 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 -- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 msgr2=0x7f737c0a8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0a8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 -- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f737c0c9ca0 msgr2=0x7f737c0c7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f737c0c9ca0 0x7f737c0c7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.995+0000 7f7381d09640 1 -- 172.21.15.29:0/1497860553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7374009a70 con 0x7f737c15ad70 2023-10-14T18:14:31.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f7381d09640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f7374004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:31.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737403e610 con 0x7f737c15ad70 2023-10-14T18:14:31.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737403e770 con 0x7f737c15ad70 2023-10-14T18:14:31.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c0c8590 con 0x7f737c15ad70 2023-10-14T18:14:31.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f737403eb20 con 0x7f737c15ad70 2023-10-14T18:14:31.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.996+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c0c7fc0 con 0x7f737c15ad70 2023-10-14T18:14:31.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.997+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c06ad20 con 0x7f737c15ad70 2023-10-14T18:14:31.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.998+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 121) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f737403b050 con 0x7f737c15ad70 2023-10-14T18:14:31.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.999+0000 7f736dffb640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 0x7f736006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:31.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.999+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6971+0+0 (secure 0 0 0) 0x7f73740ac490 con 0x7f737c15ad70 2023-10-14T18:14:31.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:30.999+0000 7f7381508640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 0x7f736006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:31.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.000+0000 7f7381508640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 0x7f736006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7378001570 tx=0x7f7378007930 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:31.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.003+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f73740770b0 con 0x7f737c15ad70 2023-10-14T18:14:31.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.231+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f737c04e160 con 0x7f737c15ad70 2023-10-14T18:14:31.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.767+0000 7f736dffb640 1 -- 172.21.15.29:0/1497860553 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7f7374009450 con 0x7f737c15ad70 2023-10-14T18:14:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 msgr2=0x7f736006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 0x7f736006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f7378001570 tx=0x7f7378007930 comp rx=0 tx=0).stop 2023-10-14T18:14:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 msgr2=0x7f737c0ab520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:31.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 secure :-1 s=READY pgs=2317 cs=0 l=1 rev1=1 crypto rx=0x7f7374009d90 tx=0x7f7374004a40 comp rx=0 tx=0).stop 2023-10-14T18:14:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 shutdown_connections 2023-10-14T18:14:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f737c15ad70 0x7f737c0ab520 unknown :-1 s=CLOSED pgs=2317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f737c0c9ca0 0x7f737c0c7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f736006af00 0x7f736006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 --2- 172.21.15.29:0/1497860553 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f737c06a4b0 0x7f737c0a8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 >> 172.21.15.29:0/1497860553 conn(0x7f737c0ce9b0 msgr2=0x7f737c0ca080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.770+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 shutdown_connections 2023-10-14T18:14:31.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:31.771+0000 7f738376c640 1 -- 172.21.15.29:0/1497860553 wait complete. 2023-10-14T18:14:31.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T18:14:32.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.015+0000 7f7104e75640 1 Processor -- start 2023-10-14T18:14:32.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.015+0000 7f7104e75640 1 -- start start 2023-10-14T18:14:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f7104e75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f710006c040 0x7f710006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f7104e75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710015f4a0 con 0x7f71000ad970 2023-10-14T18:14:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f7104e75640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f710006c960 con 0x7f71000cda80 2023-10-14T18:14:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f7104e75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f710015b9d0 con 0x7f710006c040 2023-10-14T18:14:32.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f70ff7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f710006c040 0x7f710006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:32.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f70f6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f71000ad970 0x7f710015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38146/0 (socket says 172.21.15.29:38146) 2023-10-14T18:14:32.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.017+0000 7f70f6ffd640 1 -- 172.21.15.29:0/2512633450 learned_addr learned my addr 172.21.15.29:0/2512633450 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.018+0000 7f70ff7fe640 1 -- 172.21.15.29:0/2512633450 >> v1:172.21.15.186:6789/0 conn(0x7f71000cda80 legacy=0x7f710015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.018+0000 7f70ff7fe640 1 -- 172.21.15.29:0/2512633450 >> v1:172.21.15.29:6789/0 conn(0x7f71000ad970 legacy=0x7f710015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:32.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.018+0000 7f70ff7fe640 1 -- 172.21.15.29:0/2512633450 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f710015bcb0 con 0x7f710006c040 2023-10-14T18:14:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.018+0000 7f70feffd640 1 -- 172.21.15.29:0/2512633450 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184335309 0 0) 0x7f710015f4a0 con 0x7f71000ad970 2023-10-14T18:14:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.019+0000 7f70ff7fe640 1 --2- 172.21.15.29:0/2512633450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f710006c040 0x7f710006c420 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009d90 tx=0x7f70f002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43fda246f15c89dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.019+0000 7f70feffd640 1 -- 172.21.15.29:0/2512633450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f002cc60 con 0x7f710006c040 2023-10-14T18:14:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.019+0000 7f70feffd640 1 -- 172.21.15.29:0/2512633450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f002cdc0 con 0x7f710006c040 2023-10-14T18:14:32.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.019+0000 7f70feffd640 1 -- 172.21.15.29:0/2512633450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f0034660 con 0x7f710006c040 2023-10-14T18:14:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 -- 172.21.15.29:0/2512633450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f710006c040 msgr2=0x7f710006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 --2- 172.21.15.29:0/2512633450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f710006c040 0x7f710006c420 secure :-1 s=READY pgs=2319 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009d90 tx=0x7f70f002beb0 comp rx=0 tx=0).stop 2023-10-14T18:14:32.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 -- 172.21.15.29:0/2512633450 shutdown_connections 2023-10-14T18:14:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 --2- 172.21.15.29:0/2512633450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f710006c040 0x7f710006c420 unknown :-1 s=CLOSED pgs=2319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 -- 172.21.15.29:0/2512633450 >> 172.21.15.29:0/2512633450 conn(0x7f71000ce6a0 msgr2=0x7f71000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 -- 172.21.15.29:0/2512633450 shutdown_connections 2023-10-14T18:14:32.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.020+0000 7f7104e75640 1 -- 172.21.15.29:0/2512633450 wait complete. 2023-10-14T18:14:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.022+0000 7f7104e75640 1 Processor -- start 2023-10-14T18:14:32.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.022+0000 7f7104e75640 1 -- start start 2023-10-14T18:14:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f710006c040 0x7f710014ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71000cda80 0x7f71000ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71000ca8b0 con 0x7f71000ad970 2023-10-14T18:14:32.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71000b34c0 con 0x7f710006c040 2023-10-14T18:14:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f7104e75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f710014f4a0 con 0x7f71000cda80 2023-10-14T18:14:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f70f6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f70f6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38922/0 (socket says 172.21.15.29:38922) 2023-10-14T18:14:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.023+0000 7f70f6ffd640 1 -- 172.21.15.29:0/2172471724 learned_addr learned my addr 172.21.15.29:0/2172471724 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:32.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70ff7fe640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f710006c040 0x7f710014ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 -- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71000cda80 msgr2=0x7f71000ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71000cda80 0x7f71000ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 -- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f710006c040 msgr2=0x7f710014ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f710006c040 0x7f710014ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 -- 172.21.15.29:0/2172471724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70f0009a70 con 0x7f71000ad970 2023-10-14T18:14:32.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70f6ffd640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f70e800c4b0 tx=0x7f70e800c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e80074e0 con 0x7f71000ad970 2023-10-14T18:14:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70e8007660 con 0x7f71000ad970 2023-10-14T18:14:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71000b2b90 con 0x7f71000ad970 2023-10-14T18:14:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.024+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70e8004a20 con 0x7f71000ad970 2023-10-14T18:14:32.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.025+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71000b3150 con 0x7f71000ad970 2023-10-14T18:14:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.026+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710006b330 con 0x7f71000ad970 2023-10-14T18:14:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.027+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f70e8004b80 con 0x7f71000ad970 2023-10-14T18:14:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.028+0000 7f70fcff9640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 0x7f70e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.028+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(198..198 src has 1..198) v4 ==== 6971+0+0 (secure 0 0 0) 0x7f70e80891b0 con 0x7f71000ad970 2023-10-14T18:14:32.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.028+0000 7f70ff7fe640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 0x7f70e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:32.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.029+0000 7f70ff7fe640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 0x7f70e006d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f70f0036000 tx=0x7f70f0030000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:32.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.032+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70e8053d10 con 0x7f71000ad970 2023-10-14T18:14:32.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:32.234+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71000b3650 con 0x7f71000ad970 2023-10-14T18:14:33.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.154+0000 7f70fcff9640 1 -- 172.21.15.29:0/2172471724 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v200) v1 ==== 181+0+0 (secure 0 0 0) 0x7f70e8057330 con 0x7f71000ad970 2023-10-14T18:14:33.156 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T18:14:33.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 msgr2=0x7f70e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 0x7f70e006d450 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f70f0036000 tx=0x7f70f0030000 comp rx=0 tx=0).stop 2023-10-14T18:14:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 msgr2=0x7f71000b3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 secure :-1 s=READY pgs=2320 cs=0 l=1 rev1=1 crypto rx=0x7f70e800c4b0 tx=0x7f70e800c970 comp rx=0 tx=0).stop 2023-10-14T18:14:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 shutdown_connections 2023-10-14T18:14:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f70e006af90 0x7f70e006d450 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f71000cda80 0x7f71000ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f71000ad970 0x7f71000b3950 unknown :-1 s=CLOSED pgs=2320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 --2- 172.21.15.29:0/2172471724 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f710006c040 0x7f710014ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.159+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 >> 172.21.15.29:0/2172471724 conn(0x7f71000ce6a0 msgr2=0x7f710015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.160+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 shutdown_connections 2023-10-14T18:14:33.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.160+0000 7f7104e75640 1 -- 172.21.15.29:0/2172471724 wait complete. 2023-10-14T18:14:33.183 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T18:14:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.418+0000 7f6ddd516640 1 Processor -- start 2023-10-14T18:14:33.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.418+0000 7f6ddd516640 1 -- start start 2023-10-14T18:14:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.420+0000 7f6ddd516640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6dd8159da0 0x7f6dd80c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.420+0000 7f6ddd516640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 0x7f6dd806c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.420+0000 7f6ddd516640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dd815dfb0 con 0x7f6dd81599c0 2023-10-14T18:14:33.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.420+0000 7f6ddd516640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd815e290 con 0x7f6dd8159da0 2023-10-14T18:14:33.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.420+0000 7f6ddd516640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dd815e570 con 0x7f6dd80cce20 2023-10-14T18:14:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd67fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6dd8159da0 0x7f6dd80c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd6ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6dd81599c0 0x7f6dd815d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38162/0 (socket says 172.21.15.29:38162) 2023-10-14T18:14:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd6ffd640 1 -- 172.21.15.29:0/2700832979 learned_addr learned my addr 172.21.15.29:0/2700832979 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd77fe640 1 --2- 172.21.15.29:0/2700832979 >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 0x7f6dd806c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:33.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd67fc640 1 -- 172.21.15.29:0/2700832979 >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 msgr2=0x7f6dd806c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd67fc640 1 --2- 172.21.15.29:0/2700832979 >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 0x7f6dd806c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd67fc640 1 -- 172.21.15.29:0/2700832979 >> v1:172.21.15.29:6789/0 conn(0x7f6dd81599c0 legacy=0x7f6dd815d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd67fc640 1 -- 172.21.15.29:0/2700832979 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dd806fda0 con 0x7f6dd8159da0 2023-10-14T18:14:33.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.421+0000 7f6dd77fe640 1 --2- 172.21.15.29:0/2700832979 >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 0x7f6dd806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.422+0000 7f6dd67fc640 1 --2- 172.21.15.29:0/2700832979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8159da0 0x7f6dd80c9300 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8071940 tx=0x7f6dcc02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cccb0d7f1ac0268 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:33.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6dd5ffb640 1 -- 172.21.15.29:0/2700832979 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc031830 con 0x7f6dd8159da0 2023-10-14T18:14:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6dd5ffb640 1 -- 172.21.15.29:0/2700832979 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dcc031990 con 0x7f6dd8159da0 2023-10-14T18:14:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6dd5ffb640 1 -- 172.21.15.29:0/2700832979 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc031d20 con 0x7f6dd8159da0 2023-10-14T18:14:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 -- 172.21.15.29:0/2700832979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8159da0 msgr2=0x7f6dd80c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 --2- 172.21.15.29:0/2700832979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8159da0 0x7f6dd80c9300 secure :-1 s=READY pgs=2322 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8071940 tx=0x7f6dcc02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 -- 172.21.15.29:0/2700832979 shutdown_connections 2023-10-14T18:14:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 --2- 172.21.15.29:0/2700832979 >> v2:172.21.15.186:3300/0 conn(0x7f6dd80cce20 0x7f6dd806c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 --2- 172.21.15.29:0/2700832979 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd8159da0 0x7f6dd80c9300 unknown :-1 s=CLOSED pgs=2322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 -- 172.21.15.29:0/2700832979 >> 172.21.15.29:0/2700832979 conn(0x7f6dd80cb360 msgr2=0x7f6dd80cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:33.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 -- 172.21.15.29:0/2700832979 shutdown_connections 2023-10-14T18:14:33.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.423+0000 7f6ddd516640 1 -- 172.21.15.29:0/2700832979 wait complete. 2023-10-14T18:14:33.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.425+0000 7f6ddd516640 1 Processor -- start 2023-10-14T18:14:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.425+0000 7f6ddd516640 1 -- start start 2023-10-14T18:14:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd81599c0 0x7f6dd81565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dd8159da0 0x7f6dd80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd815b040 con 0x7f6dd80cce20 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd80b01d0 con 0x7f6dd81599c0 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.426+0000 7f6ddd516640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dd8157540 con 0x7f6dd8159da0 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38946/0 (socket says 172.21.15.29:38946) 2023-10-14T18:14:33.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 -- 172.21.15.29:0/4057757475 learned_addr learned my addr 172.21.15.29:0/4057757475 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 -- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dd8159da0 msgr2=0x7f6dd80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd67fc640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd81599c0 0x7f6dd81565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dd8159da0 0x7f6dd80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 -- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd81599c0 msgr2=0x7f6dd81565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:33.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd81599c0 0x7f6dd81565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.427+0000 7f6dd6ffd640 1 -- 172.21.15.29:0/4057757475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc009400 con 0x7f6dd80cce20 2023-10-14T18:14:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6dd6ffd640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800c4a0 tx=0x7f6dc800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8005ce0 con 0x7f6dd80cce20 2023-10-14T18:14:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dc8005e40 con 0x7f6dd80cce20 2023-10-14T18:14:33.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dc8007270 con 0x7f6dd80cce20 2023-10-14T18:14:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dd8156ca0 con 0x7f6dd80cce20 2023-10-14T18:14:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.428+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dd8157260 con 0x7f6dd80cce20 2023-10-14T18:14:33.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.429+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dd815ada0 con 0x7f6dd80cce20 2023-10-14T18:14:33.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.430+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6dc8007490 con 0x7f6dd80cce20 2023-10-14T18:14:33.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.432+0000 7f6dbf7fe640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 0x7f6dc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:33.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.432+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(200..200 src has 1..200) v4 ==== 6505+0+0 (secure 0 0 0) 0x7f6dc8088f80 con 0x7f6dd80cce20 2023-10-14T18:14:33.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.432+0000 7f6dd67fc640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 0x7f6dc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:33.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.433+0000 7f6dd67fc640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 0x7f6dc406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc036000 tx=0x7f6dcc008030 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:33.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.439+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6dd815ada0 con 0x7f6dd80cce20 2023-10-14T18:14:33.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:33.637+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6dd815b180 con 0x7f6dd80cce20 2023-10-14T18:14:34.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.435+0000 7f6dbf7fe640 1 -- 172.21.15.29:0/4057757475 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v201) v1 ==== 169+0+0 (secure 0 0 0) 0x7f6dd815b180 con 0x7f6dd80cce20 2023-10-14T18:14:34.437 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T18:14:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 msgr2=0x7f6dc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 0x7f6dc406d450 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc036000 tx=0x7f6dcc008030 comp rx=0 tx=0).stop 2023-10-14T18:14:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 msgr2=0x7f6dd8155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 secure :-1 s=READY pgs=2323 cs=0 l=1 rev1=1 crypto rx=0x7f6dc800c4a0 tx=0x7f6dc800c960 comp rx=0 tx=0).stop 2023-10-14T18:14:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 shutdown_connections 2023-10-14T18:14:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dc406af90 0x7f6dc406d450 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dd8159da0 0x7f6dd80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dd81599c0 0x7f6dd81565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 --2- 172.21.15.29:0/4057757475 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dd80cce20 0x7f6dd8155f80 unknown :-1 s=CLOSED pgs=2323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.439+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 >> 172.21.15.29:0/4057757475 conn(0x7f6dd80cb360 msgr2=0x7f6dd80c47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:34.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.441+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 shutdown_connections 2023-10-14T18:14:34.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.442+0000 7f6ddd516640 1 -- 172.21.15.29:0/4057757475 wait complete. 2023-10-14T18:14:34.467 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T18:14:34.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.688+0000 7f6dd197a640 1 Processor -- start 2023-10-14T18:14:34.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.688+0000 7f6dd197a640 1 -- start start 2023-10-14T18:14:34.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.689+0000 7f6dd197a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:34.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.689+0000 7f6dd197a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc0d0970 con 0x7f6dcc0cfc40 2023-10-14T18:14:34.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.689+0000 7f6dd197a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dcc0d0ea0 con 0x7f6dcc0b2080 2023-10-14T18:14:34.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.689+0000 7f6dd197a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0d0bc0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.690+0000 7f6dd1179640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6dcc0cfc40 0x7f6dcc0b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38168/0 (socket says 172.21.15.29:38168) 2023-10-14T18:14:34.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.690+0000 7f6dd1179640 1 -- 172.21.15.29:0/2196684907 learned_addr learned my addr 172.21.15.29:0/2196684907 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:34.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.690+0000 7f6dd0978640 1 --2- 172.21.15.29:0/2196684907 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:34.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.691+0000 7f6dcb7fe640 1 -- 172.21.15.29:0/2196684907 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354711433 0 0) 0x7f6dcc0d0970 con 0x7f6dcc0cfc40 2023-10-14T18:14:34.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.691+0000 7f6dcb7fe640 1 -- 172.21.15.29:0/2196684907 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6db8003520 con 0x7f6dcc0cfc40 2023-10-14T18:14:34.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.691+0000 7f6dd0978640 1 -- 172.21.15.29:0/2196684907 >> v1:172.21.15.186:6789/0 conn(0x7f6dcc0b2080 legacy=0x7f6dcc0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.691+0000 7f6dd0978640 1 -- 172.21.15.29:0/2196684907 >> v1:172.21.15.29:6789/0 conn(0x7f6dcc0cfc40 legacy=0x7f6dcc0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.691+0000 7f6dd0978640 1 -- 172.21.15.29:0/2196684907 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dcc0cd940 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.692+0000 7f6dd0978640 1 --2- 172.21.15.29:0/2196684907 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0d36b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009dc0 tx=0x7f6dbc02bfb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d1d389036645cd3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:34.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.693+0000 7f6dcb7fe640 1 -- 172.21.15.29:0/2196684907 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc02cd60 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.693+0000 7f6dcb7fe640 1 -- 172.21.15.29:0/2196684907 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc034440 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.693+0000 7f6dcb7fe640 1 -- 172.21.15.29:0/2196684907 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0347b0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.693+0000 7f6dd197a640 1 -- 172.21.15.29:0/2196684907 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 msgr2=0x7f6dcc0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.693+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2196684907 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0d36b0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009dc0 tx=0x7f6dbc02bfb0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.694+0000 7f6dd197a640 1 -- 172.21.15.29:0/2196684907 shutdown_connections 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.694+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2196684907 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0d36b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.694+0000 7f6dd197a640 1 -- 172.21.15.29:0/2196684907 >> 172.21.15.29:0/2196684907 conn(0x7f6dcc0d2b90 msgr2=0x7f6dcc0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.694+0000 7f6dd197a640 1 -- 172.21.15.29:0/2196684907 shutdown_connections 2023-10-14T18:14:34.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.694+0000 7f6dd197a640 1 -- 172.21.15.29:0/2196684907 wait complete. 2023-10-14T18:14:34.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.695+0000 7f6dd197a640 1 Processor -- start 2023-10-14T18:14:34.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.696+0000 7f6dd197a640 1 -- start start 2023-10-14T18:14:34.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.696+0000 7f6dd197a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:34.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd197a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b2080 0x7f6dcc0cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:34.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd197a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dcc0cfc40 0x7f6dcc0cb640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:34.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd197a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc0caf10 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd197a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc150a00 con 0x7f6dcc0cfc40 2023-10-14T18:14:34.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd197a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dcc14f960 con 0x7f6dcc0b2080 2023-10-14T18:14:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38950/0 (socket says 172.21.15.29:38950) 2023-10-14T18:14:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 -- 172.21.15.29:0/2795730344 learned_addr learned my addr 172.21.15.29:0/2795730344 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 -- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b2080 msgr2=0x7f6dcc0cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dcbfff640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b2080 0x7f6dcc0cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:34.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd1179640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dcc0cfc40 0x7f6dcc0cb640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:34.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b2080 0x7f6dcc0cb100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 -- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dcc0cfc40 msgr2=0x7f6dcc0cb640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dcc0cfc40 0x7f6dcc0cb640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.697+0000 7f6dd0978640 1 -- 172.21.15.29:0/2795730344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dbc009a70 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dd0978640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009ee0 tx=0x7f6dbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc003b90 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dbc003cf0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc004260 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dcc14fc40 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.698+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dcc150200 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.699+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dcc04e9b0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.701+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6dbc03b050 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.702+0000 7f6dc97fa640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 0x7f6dac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:34.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.702+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f6dbc0abed0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.702+0000 7f6dcbfff640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 0x7f6dac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:34.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.703+0000 7f6dcbfff640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 0x7f6dac06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc06c250 tx=0x7f6db4009040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:34.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.706+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6dbc076ea0 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.906+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f6dcc073a40 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.914+0000 7f6dc97fa640 1 -- 172.21.15.29:0/2795730344 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v67) v1 ==== 115+0+0 (secure 0 0 0) 0x7f6dbc039020 con 0x7f6dcc0b1ca0 2023-10-14T18:14:34.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 msgr2=0x7f6dac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 0x7f6dac06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc06c250 tx=0x7f6db4009040 comp rx=0 tx=0).stop 2023-10-14T18:14:34.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 msgr2=0x7f6dcc0ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:34.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 secure :-1 s=READY pgs=2325 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc009ee0 tx=0x7f6dbc004a40 comp rx=0 tx=0).stop 2023-10-14T18:14:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 shutdown_connections 2023-10-14T18:14:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6dcc0cfc40 0x7f6dcc0cb640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6dac06af90 0x7f6dac06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6dcc0b2080 0x7f6dcc0cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 --2- 172.21.15.29:0/2795730344 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6dcc0b1ca0 0x7f6dcc0ca9d0 unknown :-1 s=CLOSED pgs=2325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 >> 172.21.15.29:0/2795730344 conn(0x7f6dcc0d2b90 msgr2=0x7f6dcc160080 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 shutdown_connections 2023-10-14T18:14:34.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:34.917+0000 7f6dd197a640 1 -- 172.21.15.29:0/2795730344 wait complete. 2023-10-14T18:14:34.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T18:14:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.159+0000 7f63d9c00640 1 Processor -- start 2023-10-14T18:14:35.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.159+0000 7f63d9c00640 1 -- start start 2023-10-14T18:14:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d9c00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f63d4160ad0 0x7f63d40ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d9c00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40ce800 con 0x7f63d40ae260 2023-10-14T18:14:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d9c00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40cdec0 con 0x7f63d40ce350 2023-10-14T18:14:35.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d9c00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d40acdd0 con 0x7f63d4160ad0 2023-10-14T18:14:35.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f63d40ae260 0x7f63d406a780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38180/0 (socket says 172.21.15.29:38180) 2023-10-14T18:14:35.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d3fff640 1 -- 172.21.15.29:0/1625678278 learned_addr learned my addr 172.21.15.29:0/1625678278 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.161+0000 7f63d37fe640 1 --2- 172.21.15.29:0/1625678278 >> v2:172.21.15.186:3300/0 conn(0x7f63d4160ad0 0x7f63d40ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696616029 0 0) 0x7f63d40ce800 con 0x7f63d40ae260 2023-10-14T18:14:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c0003520 con 0x7f63d40ae260 2023-10-14T18:14:35.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683815200 0 0) 0x7f63c0003520 con 0x7f63d40ae260 2023-10-14T18:14:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 >> v2:172.21.15.186:3300/0 conn(0x7f63d4160ad0 msgr2=0x7f63d40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 --2- 172.21.15.29:0/1625678278 >> v2:172.21.15.186:3300/0 conn(0x7f63d4160ad0 0x7f63d40ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 >> v1:172.21.15.186:6789/0 conn(0x7f63d40ce350 legacy=0x7f63d4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d40ae740 con 0x7f63d40ae260 2023-10-14T18:14:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f63cc0019f0 con 0x7f63d40ae260 2023-10-14T18:14:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 >> v1:172.21.15.29:6789/0 conn(0x7f63d40ae260 legacy=0x7f63d406a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00039c0 0x7f63c0003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c0005180 0x7f63c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d40acdd0 con 0x7f63c00047c0 2023-10-14T18:14:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000c080 con 0x7f63c00039c0 2023-10-14T18:14:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.162+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c000c240 con 0x7f63c0005180 2023-10-14T18:14:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63d37fe640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00039c0 0x7f63c0003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63d3fff640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c0005180 0x7f63c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 -- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c0005180 msgr2=0x7f63c000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63c0005180 0x7f63c000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 -- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00039c0 msgr2=0x7f63c0003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:35.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00039c0 0x7f63c0003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63cbfff640 1 -- 172.21.15.29:0/1625678278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0003520 con 0x7f63c00047c0 2023-10-14T18:14:35.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.163+0000 7f63d3fff640 1 -- 172.21.15.29:0/1625678278 reap_dead start 2023-10-14T18:14:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.165+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001750 tx=0x7f63c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f103f022afacfa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.165+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0052d0 con 0x7f63c00047c0 2023-10-14T18:14:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.165+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc02e050 con 0x7f63c00047c0 2023-10-14T18:14:35.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.165+0000 7f63d2ffd640 1 -- 172.21.15.29:0/1625678278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc003830 con 0x7f63c00047c0 2023-10-14T18:14:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 -- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 msgr2=0x7f63c0004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 secure :-1 s=READY pgs=2327 cs=0 l=1 rev1=1 crypto rx=0x7f63bc001750 tx=0x7f63c00057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 -- 172.21.15.29:0/1625678278 shutdown_connections 2023-10-14T18:14:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1625678278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63c00047c0 0x7f63c0004c40 unknown :-1 s=CLOSED pgs=2327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 -- 172.21.15.29:0/1625678278 >> 172.21.15.29:0/1625678278 conn(0x7f63d406bc10 msgr2=0x7f63d406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:35.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 -- 172.21.15.29:0/1625678278 shutdown_connections 2023-10-14T18:14:35.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.166+0000 7f63d9c00640 1 -- 172.21.15.29:0/1625678278 wait complete. 2023-10-14T18:14:35.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.168+0000 7f63d9c00640 1 Processor -- start 2023-10-14T18:14:35.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.168+0000 7f63d9c00640 1 -- start start 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00047c0 0x7f63d40ca820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63d40c7b80 0x7f63d40b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41f3550 con 0x7f63d41f2530 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d4159dc0 con 0x7f63c00047c0 2023-10-14T18:14:35.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63d9c00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63d41f29d0 con 0x7f63d40c7b80 2023-10-14T18:14:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38958/0 (socket says 172.21.15.29:38958) 2023-10-14T18:14:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.169+0000 7f63cbfff640 1 -- 172.21.15.29:0/1121944584 learned_addr learned my addr 172.21.15.29:0/1121944584 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d3fff640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63d40c7b80 0x7f63d40b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d37fe640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00047c0 0x7f63d40ca820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 -- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63d40c7b80 msgr2=0x7f63d40b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63d40c7b80 0x7f63d40b44e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 -- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00047c0 msgr2=0x7f63d40ca820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00047c0 0x7f63d40ca820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 -- 172.21.15.29:0/1121944584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d41f2cb0 con 0x7f63d41f2530 2023-10-14T18:14:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63cbfff640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f63bc005040 tx=0x7f63bc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc0076c0 con 0x7f63d41f2530 2023-10-14T18:14:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc0069e0 con 0x7f63d41f2530 2023-10-14T18:14:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d41f2f90 con 0x7f63d41f2530 2023-10-14T18:14:35.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.170+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f63bc006d90 con 0x7f63d41f2530 2023-10-14T18:14:35.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.171+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d40c8000 con 0x7f63d41f2530 2023-10-14T18:14:35.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.172+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d41f34b0 con 0x7f63d41f2530 2023-10-14T18:14:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.173+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f63bc0035f0 con 0x7f63d41f2530 2023-10-14T18:14:35.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.174+0000 7f63d0ff9640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 0x7f63b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.174+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f63bc0ab900 con 0x7f63d41f2530 2023-10-14T18:14:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.174+0000 7f63d37fe640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 0x7f63b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.175+0000 7f63d37fe640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 0x7f63b406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63d4069740 tx=0x7f63c4008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.178+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f63bc076690 con 0x7f63d41f2530 2023-10-14T18:14:35.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.382+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f63d404e160 con 0x7f63d41f2530 2023-10-14T18:14:35.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.389+0000 7f63d0ff9640 1 -- 172.21.15.29:0/1121944584 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v68) v1 ==== 120+0+0 (secure 0 0 0) 0x7f63bc04b0b0 con 0x7f63d41f2530 2023-10-14T18:14:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 msgr2=0x7f63b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 0x7f63b406d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f63d4069740 tx=0x7f63c4008040 comp rx=0 tx=0).stop 2023-10-14T18:14:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 msgr2=0x7f63d4159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 secure :-1 s=READY pgs=2328 cs=0 l=1 rev1=1 crypto rx=0x7f63bc005040 tx=0x7f63bc001120 comp rx=0 tx=0).stop 2023-10-14T18:14:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 shutdown_connections 2023-10-14T18:14:35.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f63b406af90 0x7f63b406d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f63d40c7b80 0x7f63d40b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f63d41f2530 0x7f63d4159880 unknown :-1 s=CLOSED pgs=2328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 --2- 172.21.15.29:0/1121944584 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f63c00047c0 0x7f63d40ca820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 >> 172.21.15.29:0/1121944584 conn(0x7f63d406bc10 msgr2=0x7f63d40b0090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:35.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 shutdown_connections 2023-10-14T18:14:35.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.392+0000 7f63d9c00640 1 -- 172.21.15.29:0/1121944584 wait complete. 2023-10-14T18:14:35.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T18:14:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.639+0000 7ff210c67640 1 Processor -- start 2023-10-14T18:14:35.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.639+0000 7ff210c67640 1 -- start start 2023-10-14T18:14:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff210c67640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff210c67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff210c67640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff20c06c960 con 0x7ff20c15ebc0 2023-10-14T18:14:35.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff210c67640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff20c15b9d0 con 0x7ff20c06c040 2023-10-14T18:14:35.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff210c67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c15bcb0 con 0x7ff20c0ad970 2023-10-14T18:14:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.641+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38972/0 (socket says 172.21.15.29:38972) 2023-10-14T18:14:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 -- 172.21.15.29:0/194054696 learned_addr learned my addr 172.21.15.29:0/194054696 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:35.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff203fff640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 -- 172.21.15.29:0/194054696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 msgr2=0x7ff20c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 -- 172.21.15.29:0/194054696 >> v1:172.21.15.186:6789/0 conn(0x7ff20c15ebc0 legacy=0x7ff20c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.642+0000 7ff20b7fe640 1 -- 172.21.15.29:0/194054696 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff20c15bf90 con 0x7ff20c06c040 2023-10-14T18:14:35.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.643+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0098a0 tx=0x7ff1fc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c91174f00de88ca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff20affd640 1 -- 172.21.15.29:0/194054696 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc030840 con 0x7ff20c06c040 2023-10-14T18:14:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff20affd640 1 -- 172.21.15.29:0/194054696 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc0309a0 con 0x7ff20c06c040 2023-10-14T18:14:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff20affd640 1 -- 172.21.15.29:0/194054696 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc030d30 con 0x7ff20c06c040 2023-10-14T18:14:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 -- 172.21.15.29:0/194054696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 msgr2=0x7ff20c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 secure :-1 s=READY pgs=2329 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc0098a0 tx=0x7ff1fc02b980 comp rx=0 tx=0).stop 2023-10-14T18:14:35.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 -- 172.21.15.29:0/194054696 shutdown_connections 2023-10-14T18:14:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c15e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 --2- 172.21.15.29:0/194054696 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c06c420 unknown :-1 s=CLOSED pgs=2329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 -- 172.21.15.29:0/194054696 >> 172.21.15.29:0/194054696 conn(0x7ff20c0ce6a0 msgr2=0x7ff20c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 -- 172.21.15.29:0/194054696 shutdown_connections 2023-10-14T18:14:35.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.644+0000 7ff210c67640 1 -- 172.21.15.29:0/194054696 wait complete. 2023-10-14T18:14:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.646+0000 7ff210c67640 1 Processor -- start 2023-10-14T18:14:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.646+0000 7ff210c67640 1 -- start start 2023-10-14T18:14:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c0acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 0x7ff20c1532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c1f6820 con 0x7ff20c06c040 2023-10-14T18:14:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0c9090 con 0x7ff20c15ebc0 2023-10-14T18:14:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.647+0000 7ff210c67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff20c0c9370 con 0x7ff20c0ad970 2023-10-14T18:14:35.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38980/0 (socket says 172.21.15.29:38980) 2023-10-14T18:14:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 -- 172.21.15.29:0/631556327 learned_addr learned my addr 172.21.15.29:0/631556327 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff203fff640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c0acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20bfff640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 0x7ff20c1532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 -- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 msgr2=0x7ff20c0acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c0acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 -- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 msgr2=0x7ff20c1532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 0x7ff20c1532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20b7fe640 1 -- 172.21.15.29:0/631556327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1fc0094d0 con 0x7ff20c06c040 2023-10-14T18:14:35.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.648+0000 7ff20bfff640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 0x7ff20c1532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:35.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff20b7fe640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc000c00 tx=0x7ff1fc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc039a90 con 0x7ff20c06c040 2023-10-14T18:14:35.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1fc039bf0 con 0x7ff20c06c040 2023-10-14T18:14:35.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1fc043410 con 0x7ff20c06c040 2023-10-14T18:14:35.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff20c0c9650 con 0x7ff20c06c040 2023-10-14T18:14:35.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.649+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff20c0c9c10 con 0x7ff20c06c040 2023-10-14T18:14:35.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.651+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff20c06b330 con 0x7ff20c06c040 2023-10-14T18:14:35.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.651+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff1fc02f390 con 0x7ff20c06c040 2023-10-14T18:14:35.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.652+0000 7ff208ff9640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:35.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.653+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7ff1fc0abc30 con 0x7ff20c06c040 2023-10-14T18:14:35.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.653+0000 7ff203fff640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:35.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.654+0000 7ff203fff640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 0x7ff1ec06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4006fd0 tx=0x7ff1f4008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:35.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.657+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff1fc076930 con 0x7ff20c06c040 2023-10-14T18:14:35.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.852+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff20c04e9b0 con 0x7ff20c06c040 2023-10-14T18:14:35.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.859+0000 7ff208ff9640 1 -- 172.21.15.29:0/631556327 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff1fc079f50 con 0x7ff20c06c040 2023-10-14T18:14:35.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 msgr2=0x7ff1ec06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 0x7ff1ec06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff1f4006fd0 tx=0x7ff1f4008040 comp rx=0 tx=0).stop 2023-10-14T18:14:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 msgr2=0x7ff20c1f62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 secure :-1 s=READY pgs=2330 cs=0 l=1 rev1=1 crypto rx=0x7ff1fc000c00 tx=0x7ff1fc004730 comp rx=0 tx=0).stop 2023-10-14T18:14:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 shutdown_connections 2023-10-14T18:14:35.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff20c15ebc0 0x7ff20c1532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7ff1ec06af90 0x7ff1ec06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff20c0ad970 0x7ff20c0acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 --2- 172.21.15.29:0/631556327 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff20c06c040 0x7ff20c1f62e0 unknown :-1 s=CLOSED pgs=2330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.862+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 >> 172.21.15.29:0/631556327 conn(0x7ff20c0ce6a0 msgr2=0x7ff20c15cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.863+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 shutdown_connections 2023-10-14T18:14:35.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:35.863+0000 7ff210c67640 1 -- 172.21.15.29:0/631556327 wait complete. 2023-10-14T18:14:35.884 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart' 2023-10-14T18:14:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.106+0000 7f61a9e61640 1 Processor -- start 2023-10-14T18:14:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.106+0000 7f61a9e61640 1 -- start start 2023-10-14T18:14:36.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a9e61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f61a415ae80 0x7f61a415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a9e61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a415ed60 con 0x7f61a40ce390 2023-10-14T18:14:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a9e61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61a40707e0 con 0x7f61a415aaa0 2023-10-14T18:14:36.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a9e61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61a406b810 con 0x7f61a415ae80 2023-10-14T18:14:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f61a40ce390 0x7f61a4070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38194/0 (socket says 172.21.15.29:38194) 2023-10-14T18:14:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f61a3fff640 1 -- 172.21.15.29:0/3725359952 learned_addr learned my addr 172.21.15.29:0/3725359952 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:36.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.108+0000 7f619affd640 1 --2- 172.21.15.29:0/3725359952 >> v2:172.21.15.29:3300/0 conn(0x7f61a415ae80 0x7f61a415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.109+0000 7f61a2ffd640 1 -- 172.21.15.29:0/3725359952 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483129737 0 0) 0x7f61a415ed60 con 0x7f61a40ce390 2023-10-14T18:14:36.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.109+0000 7f61a2ffd640 1 -- 172.21.15.29:0/3725359952 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6190003520 con 0x7f61a40ce390 2023-10-14T18:14:36.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.109+0000 7f619affd640 1 -- 172.21.15.29:0/3725359952 >> v1:172.21.15.186:6789/0 conn(0x7f61a415aaa0 legacy=0x7f61a40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.109+0000 7f619affd640 1 -- 172.21.15.29:0/3725359952 >> v1:172.21.15.29:6789/0 conn(0x7f61a40ce390 legacy=0x7f61a4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.109+0000 7f619affd640 1 -- 172.21.15.29:0/3725359952 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a415ef90 con 0x7f61a415ae80 2023-10-14T18:14:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.111+0000 7f619affd640 1 --2- 172.21.15.29:0/3725359952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a415ae80 0x7f61a415e820 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f618c0097a0 tx=0x7f618c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=274beeb7371b5181 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.111+0000 7f61a2ffd640 1 -- 172.21.15.29:0/3725359952 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c030610 con 0x7f61a415ae80 2023-10-14T18:14:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.111+0000 7f61a2ffd640 1 -- 172.21.15.29:0/3725359952 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c030770 con 0x7f61a415ae80 2023-10-14T18:14:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a2ffd640 1 -- 172.21.15.29:0/3725359952 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618c030ae0 con 0x7f61a415ae80 2023-10-14T18:14:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 -- 172.21.15.29:0/3725359952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a415ae80 msgr2=0x7f61a415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3725359952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a415ae80 0x7f61a415e820 secure :-1 s=READY pgs=2332 cs=0 l=1 rev1=1 crypto rx=0x7f618c0097a0 tx=0x7f618c02b960 comp rx=0 tx=0).stop 2023-10-14T18:14:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 -- 172.21.15.29:0/3725359952 shutdown_connections 2023-10-14T18:14:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3725359952 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a415ae80 0x7f61a415e820 unknown :-1 s=CLOSED pgs=2332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 -- 172.21.15.29:0/3725359952 >> 172.21.15.29:0/3725359952 conn(0x7f61a40cb410 msgr2=0x7f61a40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 -- 172.21.15.29:0/3725359952 shutdown_connections 2023-10-14T18:14:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.112+0000 7f61a9e61640 1 -- 172.21.15.29:0/3725359952 wait complete. 2023-10-14T18:14:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.114+0000 7f61a9e61640 1 Processor -- start 2023-10-14T18:14:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.114+0000 7f61a9e61640 1 -- start start 2023-10-14T18:14:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61a415aaa0 0x7f61a4142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61a415ae80 0x7f61a41f58d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4158c20 con 0x7f61a40ce390 2023-10-14T18:14:36.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61a41432d0 con 0x7f61a415ae80 2023-10-14T18:14:36.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.115+0000 7f61a9e61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61a4156670 con 0x7f61a415aaa0 2023-10-14T18:14:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38998/0 (socket says 172.21.15.29:38998) 2023-10-14T18:14:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 -- 172.21.15.29:0/3246512046 learned_addr learned my addr 172.21.15.29:0/3246512046 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 -- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61a415aaa0 msgr2=0x7f61a4142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61a415aaa0 0x7f61a4142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 -- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61a415ae80 msgr2=0x7f61a41f58d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61a415ae80 0x7f61a41f58d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 -- 172.21.15.29:0/3246512046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f618c009400 con 0x7f61a40ce390 2023-10-14T18:14:36.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.116+0000 7f61a37fe640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f619400c790 tx=0x7f619400cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.117+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f619400cea0 con 0x7f61a40ce390 2023-10-14T18:14:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.117+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6194006d50 con 0x7f61a40ce390 2023-10-14T18:14:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.117+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6194003bb0 con 0x7f61a40ce390 2023-10-14T18:14:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.117+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61a41582f0 con 0x7f61a40ce390 2023-10-14T18:14:36.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.117+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61a41588b0 con 0x7f61a40ce390 2023-10-14T18:14:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.118+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61a4158a40 con 0x7f61a40ce390 2023-10-14T18:14:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.119+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6194003d10 con 0x7f61a40ce390 2023-10-14T18:14:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.120+0000 7f61a0ff9640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 0x7f618406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.120+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f6194089550 con 0x7f61a40ce390 2023-10-14T18:14:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.120+0000 7f619affd640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 0x7f618406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.121+0000 7f619affd640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 0x7f618406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f618c036000 tx=0x7f618c030000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.124+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6194057df0 con 0x7f61a40ce390 2023-10-14T18:14:36.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.392+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]} v 0) v1 -- 0x7f61a406c620 con 0x7f61a40ce390 2023-10-14T18:14:36.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.440+0000 7f61a0ff9640 1 -- 172.21.15.29:0/3246512046 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_mgr_restart"]}]=0 v3073) v1 ==== 165+0+0 (secure 0 0 0) 0x7f6194057980 con 0x7f61a40ce390 2023-10-14T18:14:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 msgr2=0x7f618406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 0x7f618406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f618c036000 tx=0x7f618c030000 comp rx=0 tx=0).stop 2023-10-14T18:14:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 msgr2=0x7f61a41457d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 secure :-1 s=READY pgs=2333 cs=0 l=1 rev1=1 crypto rx=0x7f619400c790 tx=0x7f619400cc50 comp rx=0 tx=0).stop 2023-10-14T18:14:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 shutdown_connections 2023-10-14T18:14:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61a415ae80 0x7f61a41f58d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f618406afc0 0x7f618406d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61a415aaa0 0x7f61a4142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 --2- 172.21.15.29:0/3246512046 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61a40ce390 0x7f61a41457d0 unknown :-1 s=CLOSED pgs=2333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 >> 172.21.15.29:0/3246512046 conn(0x7f61a40cb410 msgr2=0x7f61a406f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:36.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.443+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 shutdown_connections 2023-10-14T18:14:36.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.444+0000 7f61a9e61640 1 -- 172.21.15.29:0/3246512046 wait complete. 2023-10-14T18:14:36.465 INFO:tasks.cephfs_test_runner:Test that snap schedule is resumed after mgr restart ... ok 2023-10-14T18:14:36.465 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) Test scheduled snapshots along with rentention policy 2023-10-14T18:14:36.466 INFO:tasks.cephfs_test_runner:test_snap_schedule_with_retention (tasks.cephfs.test_snap_schedules.TestSnapSchedules) 2023-10-14T18:14:36.466 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:36.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.695+0000 7f91f24c9640 1 Processor -- start 2023-10-14T18:14:36.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.695+0000 7f91f24c9640 1 -- start start 2023-10-14T18:14:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91f24c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec15aaa0 0x7f91ec15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91f24c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec0ce390 0x7f91ec070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91f24c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91ec15ed60 con 0x7f91ec15ae80 2023-10-14T18:14:36.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91f24c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91ec06b810 con 0x7f91ec15aaa0 2023-10-14T18:14:36.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91f24c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0707e0 con 0x7f91ec0ce390 2023-10-14T18:14:36.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec15aaa0 0x7f91ec15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f91ec15aaa0 0x7f91ec15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39000/0 (socket says 172.21.15.29:39000) 2023-10-14T18:14:36.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.697+0000 7f91ebfff640 1 -- 172.21.15.29:0/2566505067 learned_addr learned my addr 172.21.15.29:0/2566505067 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.698+0000 7f91ebfff640 1 -- 172.21.15.29:0/2566505067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec0ce390 msgr2=0x7f91ec070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.698+0000 7f91ebfff640 1 --2- 172.21.15.29:0/2566505067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec0ce390 0x7f91ec070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.698+0000 7f91ebfff640 1 -- 172.21.15.29:0/2566505067 >> v1:172.21.15.186:6789/0 conn(0x7f91ec15ae80 legacy=0x7f91ec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.698+0000 7f91ebfff640 1 -- 172.21.15.29:0/2566505067 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91ec15ef90 con 0x7f91ec15aaa0 2023-10-14T18:14:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.699+0000 7f91ebfff640 1 --2- 172.21.15.29:0/2566505067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15e820 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0cd990 tx=0x7f91dc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a925e0cb4704414 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.699+0000 7f91eb7fe640 1 -- 172.21.15.29:0/2566505067 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc031640 con 0x7f91ec15aaa0 2023-10-14T18:14:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.699+0000 7f91eb7fe640 1 -- 172.21.15.29:0/2566505067 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91dc0317a0 con 0x7f91ec15aaa0 2023-10-14T18:14:36.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.699+0000 7f91eb7fe640 1 -- 172.21.15.29:0/2566505067 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91dc031b30 con 0x7f91ec15aaa0 2023-10-14T18:14:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 -- 172.21.15.29:0/2566505067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 msgr2=0x7f91ec15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 --2- 172.21.15.29:0/2566505067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15e820 secure :-1 s=READY pgs=2334 cs=0 l=1 rev1=1 crypto rx=0x7f91ec0cd990 tx=0x7f91dc02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 -- 172.21.15.29:0/2566505067 shutdown_connections 2023-10-14T18:14:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 --2- 172.21.15.29:0/2566505067 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec0ce390 0x7f91ec070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 --2- 172.21.15.29:0/2566505067 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15e820 unknown :-1 s=CLOSED pgs=2334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 -- 172.21.15.29:0/2566505067 >> 172.21.15.29:0/2566505067 conn(0x7f91ec0cb410 msgr2=0x7f91ec0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 -- 172.21.15.29:0/2566505067 shutdown_connections 2023-10-14T18:14:36.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.700+0000 7f91f24c9640 1 -- 172.21.15.29:0/2566505067 wait complete. 2023-10-14T18:14:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.702+0000 7f91f24c9640 1 Processor -- start 2023-10-14T18:14:36.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.702+0000 7f91f24c9640 1 -- start start 2023-10-14T18:14:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 0x7f91ec1f5d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec15ae80 0x7f91ec15d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c3c80 con 0x7f91ec15aaa0 2023-10-14T18:14:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c4e40 con 0x7f91ec0ce390 2023-10-14T18:14:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.703+0000 7f91f24c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91ec0c3070 con 0x7f91ec15ae80 2023-10-14T18:14:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39010/0 (socket says 172.21.15.29:39010) 2023-10-14T18:14:36.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 -- 172.21.15.29:0/1715634087 learned_addr learned my addr 172.21.15.29:0/1715634087 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91f0a66640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec15ae80 0x7f91ec15d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91ebfff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 0x7f91ec1f5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 -- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec15ae80 msgr2=0x7f91ec15d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec15ae80 0x7f91ec15d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 -- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 msgr2=0x7f91ec1f5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 0x7f91ec1f5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91e3fff640 1 -- 172.21.15.29:0/1715634087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91dc0094d0 con 0x7f91ec15aaa0 2023-10-14T18:14:36.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.704+0000 7f91ebfff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 0x7f91ec1f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:36.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91e3fff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f91d400b3c0 tx=0x7f91d400b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4011830 con 0x7f91ec15aaa0 2023-10-14T18:14:36.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d4011990 con 0x7f91ec15aaa0 2023-10-14T18:14:36.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91d4011d60 con 0x7f91ec15aaa0 2023-10-14T18:14:36.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91ec0c3350 con 0x7f91ec15aaa0 2023-10-14T18:14:36.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.705+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91ec0c3910 con 0x7f91ec15aaa0 2023-10-14T18:14:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.706+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91ec0705d0 con 0x7f91ec15aaa0 2023-10-14T18:14:36.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.707+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f91d401b460 con 0x7f91ec15aaa0 2023-10-14T18:14:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.708+0000 7f91e97fa640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 0x7f91cc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.709+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f91d4089660 con 0x7f91ec15aaa0 2023-10-14T18:14:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.709+0000 7f91ebfff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 0x7f91cc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:36.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.710+0000 7f91ebfff640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 0x7f91cc06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03c040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:36.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.713+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91d4054480 con 0x7f91ec15aaa0 2023-10-14T18:14:36.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.910+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91ec0cbd20 con 0x7f91ec15aaa0 2023-10-14T18:14:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.912+0000 7f91e97fa640 1 -- 172.21.15.29:0/1715634087 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v201) v1 ==== 74+0+14666 (secure 0 0 0) 0x7f91d4057aa0 con 0x7f91ec15aaa0 2023-10-14T18:14:36.913 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:36.913 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":201,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:34.432031+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6837/2108287962":"2023-10-15T18:14:30.759937+0000","172.21.15.29:6836/2108287962":"2023-10-15T18:14:30.759937+0000","172.21.15.29:0/3286846969":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1757938149":"2023-10-15T18:14:26.920448+0000","172.21.15.29:6833/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1560276409":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/1378572596":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2299724842":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/614546910":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1466742201":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2776961501":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/287831498":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/841868971":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/1377837433":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3575832370":"2023-10-15T18:09:22.121284+0000","172.21.15.29:6833/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/2946416628":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/730049499":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6833/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/79844030":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/196725873":"2023-10-15T18:09:22.121284+0000","172.21.15.29:0/3534626168":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/1930442848":"2023-10-15T17:58:40.567922+0000","172.21.15.29:6832/1263890494":"2023-10-15T18:14:26.920448+0000","172.21.15.29:0/2824949307":"2023-10-15T17:58:40.567922+0000","172.21.15.29:0/2461249058":"2023-10-15T17:58:40.567922+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 msgr2=0x7f91cc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 0x7f91cc06d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f91dc036000 tx=0x7f91dc03c040 comp rx=0 tx=0).stop 2023-10-14T18:14:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 msgr2=0x7f91ec15d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 secure :-1 s=READY pgs=2335 cs=0 l=1 rev1=1 crypto rx=0x7f91d400b3c0 tx=0x7f91d400b880 comp rx=0 tx=0).stop 2023-10-14T18:14:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 shutdown_connections 2023-10-14T18:14:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f91cc06af90 0x7f91cc06d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91ec15ae80 0x7f91ec15d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f91ec15aaa0 0x7f91ec15d290 unknown :-1 s=CLOSED pgs=2335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 --2- 172.21.15.29:0/1715634087 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f91ec0ce390 0x7f91ec1f5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.914+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 >> 172.21.15.29:0/1715634087 conn(0x7f91ec0cb410 msgr2=0x7f91ec0b0240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.915+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 shutdown_connections 2023-10-14T18:14:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:36.915+0000 7f91f24c9640 1 -- 172.21.15.29:0/1715634087 wait complete. 2023-10-14T18:14:36.937 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:14:36.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:14:37.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.163+0000 7feff9aa9640 1 Processor -- start 2023-10-14T18:14:37.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.164+0000 7feff9aa9640 1 -- start start 2023-10-14T18:14:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.165+0000 7feff9aa9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feff40ce350 0x7feff4160eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.165+0000 7feff9aa9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 0x7feff406a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.165+0000 7feff9aa9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff40ce800 con 0x7feff4160ad0 2023-10-14T18:14:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.165+0000 7feff9aa9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff40cdec0 con 0x7feff40ce350 2023-10-14T18:14:37.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.165+0000 7feff9aa9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff41613f0 con 0x7feff40ae260 2023-10-14T18:14:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.166+0000 7feff2ffd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feff40ce350 0x7feff4160eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.166+0000 7feff37fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7feff4160ad0 0x7feff40ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:56548/0 (socket says 172.21.15.29:56548) 2023-10-14T18:14:37.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.166+0000 7feff37fe640 1 -- 172.21.15.29:0/1181592939 learned_addr learned my addr 172.21.15.29:0/1181592939 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:37.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.166+0000 7feff3fff640 1 --2- 172.21.15.29:0/1181592939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 0x7feff406a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.167+0000 7feff3fff640 1 -- 172.21.15.29:0/1181592939 >> v2:172.21.15.186:3300/0 conn(0x7feff40ce350 msgr2=0x7feff4160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:37.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.167+0000 7feff3fff640 1 --2- 172.21.15.29:0/1181592939 >> v2:172.21.15.186:3300/0 conn(0x7feff40ce350 0x7feff4160eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.167+0000 7feff3fff640 1 -- 172.21.15.29:0/1181592939 >> v1:172.21.15.186:6789/0 conn(0x7feff4160ad0 legacy=0x7feff40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.167+0000 7feff3fff640 1 -- 172.21.15.29:0/1181592939 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff40ae740 con 0x7feff40ae260 2023-10-14T18:14:37.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.167+0000 7feff27fc640 1 -- 172.21.15.29:0/1181592939 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1979520989 0 0) 0x7feff40ce800 con 0x7feff4160ad0 2023-10-14T18:14:37.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.168+0000 7feff3fff640 1 --2- 172.21.15.29:0/1181592939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 0x7feff406a780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097a0 tx=0x7fefec02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8b603bb0889964e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.169+0000 7feff27fc640 1 -- 172.21.15.29:0/1181592939 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec031830 con 0x7feff40ae260 2023-10-14T18:14:37.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.169+0000 7feff27fc640 1 -- 172.21.15.29:0/1181592939 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec031990 con 0x7feff40ae260 2023-10-14T18:14:37.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.169+0000 7feff27fc640 1 -- 172.21.15.29:0/1181592939 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefec031d20 con 0x7feff40ae260 2023-10-14T18:14:37.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 -- 172.21.15.29:0/1181592939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 msgr2=0x7feff406a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 --2- 172.21.15.29:0/1181592939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 0x7feff406a780 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fefec0097a0 tx=0x7fefec02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 -- 172.21.15.29:0/1181592939 shutdown_connections 2023-10-14T18:14:37.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 --2- 172.21.15.29:0/1181592939 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ae260 0x7feff406a780 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 --2- 172.21.15.29:0/1181592939 >> v2:172.21.15.186:3300/0 conn(0x7feff40ce350 0x7feff4160eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 -- 172.21.15.29:0/1181592939 >> 172.21.15.29:0/1181592939 conn(0x7feff406bc10 msgr2=0x7feff406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 -- 172.21.15.29:0/1181592939 shutdown_connections 2023-10-14T18:14:37.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.170+0000 7feff9aa9640 1 -- 172.21.15.29:0/1181592939 wait complete. 2023-10-14T18:14:37.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.172+0000 7feff9aa9640 1 Processor -- start 2023-10-14T18:14:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.172+0000 7feff9aa9640 1 -- start start 2023-10-14T18:14:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ce350 0x7feff40c5250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feff4160ad0 0x7feff4158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff40b2d30 con 0x7feff40ae260 2023-10-14T18:14:37.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff4158860 con 0x7feff4160ad0 2023-10-14T18:14:37.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff9aa9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff40ca7d0 con 0x7feff40ce350 2023-10-14T18:14:37.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39024/0 (socket says 172.21.15.29:39024) 2023-10-14T18:14:37.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff37fe640 1 -- 172.21.15.29:0/2286667858 learned_addr learned my addr 172.21.15.29:0/2286667858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:37.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff3fff640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feff4160ad0 0x7feff4158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.173+0000 7feff2ffd640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ce350 0x7feff40c5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 -- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ce350 msgr2=0x7feff40c5250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ce350 0x7feff40c5250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 -- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feff4160ad0 msgr2=0x7feff4158320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feff4160ad0 0x7feff4158320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 -- 172.21.15.29:0/2286667858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec009400 con 0x7feff40ae260 2023-10-14T18:14:37.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff37fe640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c4d0 tx=0x7fefe400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe40074e0 con 0x7feff40ae260 2023-10-14T18:14:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe4007660 con 0x7feff40ae260 2023-10-14T18:14:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe4004a20 con 0x7feff40ae260 2023-10-14T18:14:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff40caab0 con 0x7feff40ae260 2023-10-14T18:14:37.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.174+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff40cb070 con 0x7feff40ae260 2023-10-14T18:14:37.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.176+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff404e160 con 0x7feff40ae260 2023-10-14T18:14:37.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.177+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fefe4004b80 con 0x7feff40ae260 2023-10-14T18:14:37.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.178+0000 7fefebfff640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 0x7fefd406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.178+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fefe4088b00 con 0x7feff40ae260 2023-10-14T18:14:37.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.179+0000 7feff2ffd640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 0x7fefd406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.179+0000 7feff2ffd640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 0x7fefd406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feff40cb450 tx=0x7fefdc00b040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.183+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fefe4053920 con 0x7feff40ae260 2023-10-14T18:14:37.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.379+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7feff41617e0 con 0x7feff40ae260 2023-10-14T18:14:37.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.380+0000 7fefebfff640 1 -- 172.21.15.29:0/2286667858 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v201) v1 ==== 93+0+10 (secure 0 0 0) 0x7fefe4018020 con 0x7feff40ae260 2023-10-14T18:14:37.381 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:14:37.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 msgr2=0x7fefd406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 0x7fefd406d450 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7feff40cb450 tx=0x7fefdc00b040 comp rx=0 tx=0).stop 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 msgr2=0x7feff40ca1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 secure :-1 s=READY pgs=2336 cs=0 l=1 rev1=1 crypto rx=0x7fefe400c4d0 tx=0x7fefe400c990 comp rx=0 tx=0).stop 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 shutdown_connections 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feff4160ad0 0x7feff4158320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fefd406af90 0x7fefd406d450 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feff40ce350 0x7feff40c5250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 --2- 172.21.15.29:0/2286667858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feff40ae260 0x7feff40ca1a0 unknown :-1 s=CLOSED pgs=2336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 >> 172.21.15.29:0/2286667858 conn(0x7feff406bc10 msgr2=0x7feff4069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:37.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 shutdown_connections 2023-10-14T18:14:37.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.383+0000 7feff9aa9640 1 -- 172.21.15.29:0/2286667858 wait complete. 2023-10-14T18:14:37.404 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-14T18:14:37.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.623+0000 7f5c4edf4640 1 Processor -- start 2023-10-14T18:14:37.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.623+0000 7f5c4edf4640 1 -- start start 2023-10-14T18:14:37.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4edf4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5c48070b60 0x7f5c4815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4edf4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c4806c100 0x7f5c4806c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4edf4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c4815aef0 con 0x7f5c48159950 2023-10-14T18:14:37.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4edf4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4815e490 con 0x7f5c4806c100 2023-10-14T18:14:37.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4edf4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c4815d910 con 0x7f5c48070b60 2023-10-14T18:14:37.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4e5f3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c4806c100 0x7f5c4806c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4e5f3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c4806c100 0x7f5c4806c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39034/0 (socket says 172.21.15.29:39034) 2023-10-14T18:14:37.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.625+0000 7f5c4e5f3640 1 -- 172.21.15.29:0/580722558 learned_addr learned my addr 172.21.15.29:0/580722558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:37.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.626+0000 7f5c4d5f1640 1 --2- 172.21.15.29:0/580722558 >> v2:172.21.15.186:3300/0 conn(0x7f5c48070b60 0x7f5c4815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.626+0000 7f5c4e5f3640 1 -- 172.21.15.29:0/580722558 >> v2:172.21.15.186:3300/0 conn(0x7f5c48070b60 msgr2=0x7f5c4815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:37.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.626+0000 7f5c4e5f3640 1 --2- 172.21.15.29:0/580722558 >> v2:172.21.15.186:3300/0 conn(0x7f5c48070b60 0x7f5c4815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.626+0000 7f5c4e5f3640 1 -- 172.21.15.29:0/580722558 >> v1:172.21.15.186:6789/0 conn(0x7f5c48159950 legacy=0x7f5c48070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:37.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.626+0000 7f5c4e5f3640 1 -- 172.21.15.29:0/580722558 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4815dbf0 con 0x7f5c4806c100 2023-10-14T18:14:37.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.627+0000 7f5c4e5f3640 1 --2- 172.21.15.29:0/580722558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4806c100 0x7f5c4806c510 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f5c480716a0 tx=0x7f5c4002e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d36632a5e778ff51 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4cdf0640 1 -- 172.21.15.29:0/580722558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c4002ea00 con 0x7f5c4806c100 2023-10-14T18:14:37.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4cdf0640 1 -- 172.21.15.29:0/580722558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c4002eb60 con 0x7f5c4806c100 2023-10-14T18:14:37.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4cdf0640 1 -- 172.21.15.29:0/580722558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c40009600 con 0x7f5c4806c100 2023-10-14T18:14:37.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 -- 172.21.15.29:0/580722558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4806c100 msgr2=0x7f5c4806c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/580722558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4806c100 0x7f5c4806c510 secure :-1 s=READY pgs=2337 cs=0 l=1 rev1=1 crypto rx=0x7f5c480716a0 tx=0x7f5c4002e060 comp rx=0 tx=0).stop 2023-10-14T18:14:37.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 -- 172.21.15.29:0/580722558 shutdown_connections 2023-10-14T18:14:37.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/580722558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c4806c100 0x7f5c4806c510 unknown :-1 s=CLOSED pgs=2337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/580722558 >> v2:172.21.15.186:3300/0 conn(0x7f5c48070b60 0x7f5c4815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.628+0000 7f5c4edf4640 1 -- 172.21.15.29:0/580722558 >> 172.21.15.29:0/580722558 conn(0x7f5c480ce6a0 msgr2=0x7f5c480ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:37.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.629+0000 7f5c4edf4640 1 -- 172.21.15.29:0/580722558 shutdown_connections 2023-10-14T18:14:37.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.629+0000 7f5c4edf4640 1 -- 172.21.15.29:0/580722558 wait complete. 2023-10-14T18:14:37.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.631+0000 7f5c4edf4640 1 Processor -- start 2023-10-14T18:14:37.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.631+0000 7f5c4edf4640 1 -- start start 2023-10-14T18:14:37.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4806c100 0x7f5c48155510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c48159950 0x7f5c48156c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480c99c0 con 0x7f5c48070b60 2023-10-14T18:14:37.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c48155a50 con 0x7f5c4806c100 2023-10-14T18:14:37.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4edf4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c480c3c90 con 0x7f5c48159950 2023-10-14T18:14:37.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.632+0000 7f5c4d5f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39042/0 (socket says 172.21.15.29:39042) 2023-10-14T18:14:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 -- 172.21.15.29:0/2120555601 learned_addr learned my addr 172.21.15.29:0/2120555601 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4ddf2640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4806c100 0x7f5c48155510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 -- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c48159950 msgr2=0x7f5c48156c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c48159950 0x7f5c48156c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 -- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4806c100 msgr2=0x7f5c48155510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4806c100 0x7f5c48155510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:37.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 -- 172.21.15.29:0/2120555601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c4000bd40 con 0x7f5c48070b60 2023-10-14T18:14:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.633+0000 7f5c4d5f1640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800d860 tx=0x7f5c3800dd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.634+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3800de50 con 0x7f5c48070b60 2023-10-14T18:14:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.634+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c3800b8d0 con 0x7f5c48070b60 2023-10-14T18:14:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.634+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c481567e0 con 0x7f5c48070b60 2023-10-14T18:14:37.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.634+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c3800bc60 con 0x7f5c48070b60 2023-10-14T18:14:37.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.634+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c48155bf0 con 0x7f5c48070b60 2023-10-14T18:14:37.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.635+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c48156970 con 0x7f5c48070b60 2023-10-14T18:14:37.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.636+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5c3801e460 con 0x7f5c48070b60 2023-10-14T18:14:37.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.637+0000 7f5c3e7fc640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 0x7f5c2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:37.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.638+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f5c38089e80 con 0x7f5c48070b60 2023-10-14T18:14:37.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.638+0000 7f5c4ddf2640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 0x7f5c2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:37.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.639+0000 7f5c4ddf2640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 0x7f5c2c06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400a750 tx=0x7f5c44008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:37.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.645+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c38054c90 con 0x7f5c48070b60 2023-10-14T18:14:37.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:37.921+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f5c4804e9b0 con 0x7f5c48070b60 2023-10-14T18:14:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.456+0000 7f5c3e7fc640 1 -- 172.21.15.29:0/2120555601 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3075) v1 ==== 166+0+0 (secure 0 0 0) 0x7f5c380582b0 con 0x7f5c48070b60 2023-10-14T18:14:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 msgr2=0x7f5c2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 0x7f5c2c06d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f5c4400a750 tx=0x7f5c44008040 comp rx=0 tx=0).stop 2023-10-14T18:14:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 msgr2=0x7f5c480c3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 secure :-1 s=READY pgs=2338 cs=0 l=1 rev1=1 crypto rx=0x7f5c3800d860 tx=0x7f5c3800dd20 comp rx=0 tx=0).stop 2023-10-14T18:14:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 shutdown_connections 2023-10-14T18:14:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5c2c06af90 0x7f5c2c06d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.459+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c48159950 0x7f5c48156c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.460+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c48070b60 0x7f5c480c3750 unknown :-1 s=CLOSED pgs=2338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.460+0000 7f5c4edf4640 1 --2- 172.21.15.29:0/2120555601 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c4806c100 0x7f5c48155510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.460+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 >> 172.21.15.29:0/2120555601 conn(0x7f5c480ce6a0 msgr2=0x7f5c48158c20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.460+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 shutdown_connections 2023-10-14T18:14:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.460+0000 7f5c4edf4640 1 -- 172.21.15.29:0/2120555601 wait complete. 2023-10-14T18:14:38.484 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T18:14:38.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.718+0000 7fdae2d16640 1 Processor -- start 2023-10-14T18:14:38.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.718+0000 7fdae2d16640 1 -- start start 2023-10-14T18:14:38.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae2d16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdadc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae2d16640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc0ac890 con 0x7fdadc06a4b0 2023-10-14T18:14:38.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae2d16640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdadc161650 con 0x7fdadc160b90 2023-10-14T18:14:38.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae2d16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc15d530 con 0x7fdadc15d150 2023-10-14T18:14:38.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae12b3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdadc06a4b0 0x7fdadc0c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38198/0 (socket says 172.21.15.29:38198) 2023-10-14T18:14:38.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae12b3640 1 -- 172.21.15.29:0/2373969207 learned_addr learned my addr 172.21.15.29:0/2373969207 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:38.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.720+0000 7fdae0ab2640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdadc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028788147 0 0) 0x7fdadc0ac890 con 0x7fdadc06a4b0 2023-10-14T18:14:38.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdacc003520 con 0x7fdadc06a4b0 2023-10-14T18:14:38.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3279967114 0 0) 0x7fdacc003520 con 0x7fdadc06a4b0 2023-10-14T18:14:38.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 msgr2=0x7fdadc0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:38.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdadc0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 >> v1:172.21.15.186:6789/0 conn(0x7fdadc160b90 legacy=0x7fdadc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc161930 con 0x7fdadc06a4b0 2023-10-14T18:14:38.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdad40019f0 con 0x7fdadc06a4b0 2023-10-14T18:14:38.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 >> v1:172.21.15.29:6789/0 conn(0x7fdadc06a4b0 legacy=0x7fdadc0c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdacc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc003f20 0x7fdacc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc004d00 0x7fdacc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc15d530 con 0x7fdacc003f20 2023-10-14T18:14:38.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00ba60 con 0x7fdacc004d00 2023-10-14T18:14:38.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.721+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdacc00bc20 con 0x7fdadc15d150 2023-10-14T18:14:38.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc003f20 0x7fdacc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdae0ab2640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdacc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdae12b3640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc004d00 0x7fdacc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 -- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 msgr2=0x7fdacc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:38.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc15d150 0x7fdacc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 -- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc004d00 msgr2=0x7fdacc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:38.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc004d00 0x7fdacc005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdadbfff640 1 -- 172.21.15.29:0/2373969207 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdacc003520 con 0x7fdacc003f20 2023-10-14T18:14:38.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.722+0000 7fdae12b3640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc004d00 0x7fdacc005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:38.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.723+0000 7fdadbfff640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc003f20 0x7fdacc0047c0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fdac8001750 tx=0x7fdacc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=802c6acec780da35 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:38.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac80052d0 con 0x7fdacc003f20 2023-10-14T18:14:38.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdac802e050 con 0x7fdacc003f20 2023-10-14T18:14:38.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdadb7fe640 1 -- 172.21.15.29:0/2373969207 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdac8003830 con 0x7fdacc003f20 2023-10-14T18:14:38.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 -- 172.21.15.29:0/2373969207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc003f20 msgr2=0x7fdacc0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 --2- 172.21.15.29:0/2373969207 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdacc003f20 0x7fdacc0047c0 secure :-1 s=READY pgs=2340 cs=0 l=1 rev1=1 crypto rx=0x7fdac8001750 tx=0x7fdacc0057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae12b3640 1 -- 172.21.15.29:0/2373969207 reap_dead start 2023-10-14T18:14:38.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 -- 172.21.15.29:0/2373969207 shutdown_connections 2023-10-14T18:14:38.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 -- 172.21.15.29:0/2373969207 >> 172.21.15.29:0/2373969207 conn(0x7fdadc0ce9b0 msgr2=0x7fdadc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:38.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 -- 172.21.15.29:0/2373969207 shutdown_connections 2023-10-14T18:14:38.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.724+0000 7fdae2d16640 1 -- 172.21.15.29:0/2373969207 wait complete. 2023-10-14T18:14:38.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.726+0000 7fdae2d16640 1 Processor -- start 2023-10-14T18:14:38.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.726+0000 7fdae2d16640 1 -- start start 2023-10-14T18:14:38.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc003f20 0x7fdadc0b3da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc0b27d0 0x7fdadc1f3460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc1f39a0 con 0x7fdadc0c8120 2023-10-14T18:14:38.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0b2d00 con 0x7fdacc003f20 2023-10-14T18:14:38.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.727+0000 7fdae2d16640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc0b2fe0 con 0x7fdadc0b27d0 2023-10-14T18:14:38.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39052/0 (socket says 172.21.15.29:39052) 2023-10-14T18:14:38.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 -- 172.21.15.29:0/530611599 learned_addr learned my addr 172.21.15.29:0/530611599 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:38.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 -- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc0b27d0 msgr2=0x7fdadc1f3460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc0b27d0 0x7fdadc1f3460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 -- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc003f20 msgr2=0x7fdadc0b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc003f20 0x7fdadc0b3da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 -- 172.21.15.29:0/530611599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdadc0b32c0 con 0x7fdadc0c8120 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.728+0000 7fdae12b3640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fdad4000f80 tx=0x7fdad400b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:38.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.729+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4004d60 con 0x7fdadc0c8120 2023-10-14T18:14:38.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.729+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad400bb80 con 0x7fdadc0c8120 2023-10-14T18:14:38.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.729+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4007070 con 0x7fdadc0c8120 2023-10-14T18:14:38.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.729+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc0b35a0 con 0x7fdadc0c8120 2023-10-14T18:14:38.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.729+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc0c85a0 con 0x7fdadc0c8120 2023-10-14T18:14:38.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.730+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc04e160 con 0x7fdadc0c8120 2023-10-14T18:14:38.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.731+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdad40071d0 con 0x7fdadc0c8120 2023-10-14T18:14:38.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.732+0000 7fdad97fa640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 0x7fdac006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:38.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.732+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fdad4089ec0 con 0x7fdadc0c8120 2023-10-14T18:14:38.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.732+0000 7fdae0ab2640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 0x7fdac006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:38.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.733+0000 7fdae0ab2640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 0x7fdac006d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0c5950 tx=0x7fdad0008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:38.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.737+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdad4054b30 con 0x7fdadc0c8120 2023-10-14T18:14:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.933+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdadc0ace30 con 0x7fdadc0c8120 2023-10-14T18:14:38.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.940+0000 7fdad97fa640 1 -- 172.21.15.29:0/530611599 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v70)=0 v70) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdad4058150 con 0x7fdadc0c8120 2023-10-14T18:14:38.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 msgr2=0x7fdac006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 0x7fdac006d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdadc0c5950 tx=0x7fdad0008040 comp rx=0 tx=0).stop 2023-10-14T18:14:38.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 msgr2=0x7fdadc0aed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 secure :-1 s=READY pgs=2341 cs=0 l=1 rev1=1 crypto rx=0x7fdad4000f80 tx=0x7fdad400b4a0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 shutdown_connections 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdadc0c8120 0x7fdadc0aed30 unknown :-1 s=CLOSED pgs=2341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdac006afc0 0x7fdac006d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdadc0b27d0 0x7fdadc1f3460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 --2- 172.21.15.29:0/530611599 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdacc003f20 0x7fdadc0b3da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:38.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 >> 172.21.15.29:0/530611599 conn(0x7fdadc0ce9b0 msgr2=0x7fdadc0cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:38.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 shutdown_connections 2023-10-14T18:14:38.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:38.943+0000 7fdae2d16640 1 -- 172.21.15.29:0/530611599 wait complete. 2023-10-14T18:14:38.963 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:14:38.964 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:14:39.087 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:14:39.221 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:14:39.222 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:39.222 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:39.286 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:14:39.286 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:14:39.287 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:39.346 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:14:39.347 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:39.347 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:39.411 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:14:39.411 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:14:39.412 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:39.473 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:14:39.473 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T18:14:39.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:39.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.707+0000 7f9e30244640 1 Processor -- start 2023-10-14T18:14:39.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.707+0000 7f9e30244640 1 -- start start 2023-10-14T18:14:39.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e30244640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e2815aaa0 0x7f9e2815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e30244640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e2815ed60 con 0x7f9e280ce390 2023-10-14T18:14:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e30244640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280707e0 con 0x7f9e2815ae80 2023-10-14T18:14:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e30244640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2806b810 con 0x7f9e2815aaa0 2023-10-14T18:14:39.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e2dfe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e2815aaa0 0x7f9e2815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:39.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e2dfe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9e2815aaa0 0x7f9e2815e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39058/0 (socket says 172.21.15.29:39058) 2023-10-14T18:14:39.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/550953313 learned_addr learned my addr 172.21.15.29:0/550953313 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/550953313 >> v1:172.21.15.186:6789/0 conn(0x7f9e2815ae80 legacy=0x7f9e280cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.709+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/550953313 >> v1:172.21.15.29:6789/0 conn(0x7f9e280ce390 legacy=0x7f9e28070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.710+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/550953313 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2815ef90 con 0x7f9e2815aaa0 2023-10-14T18:14:39.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.710+0000 7f9e2cfde640 1 -- 172.21.15.29:0/550953313 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971649624 0 0) 0x7f9e2815ed60 con 0x7f9e280ce390 2023-10-14T18:14:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.711+0000 7f9e2dfe0640 1 --2- 172.21.15.29:0/550953313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e2815aaa0 0x7f9e2815e820 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e2402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64a0243208d1bbea server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.711+0000 7f9e2cfde640 1 -- 172.21.15.29:0/550953313 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24030860 con 0x7f9e2815aaa0 2023-10-14T18:14:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.711+0000 7f9e2cfde640 1 -- 172.21.15.29:0/550953313 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e240309c0 con 0x7f9e2815aaa0 2023-10-14T18:14:39.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.711+0000 7f9e2cfde640 1 -- 172.21.15.29:0/550953313 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24030d50 con 0x7f9e2815aaa0 2023-10-14T18:14:39.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 -- 172.21.15.29:0/550953313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e2815aaa0 msgr2=0x7f9e2815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 --2- 172.21.15.29:0/550953313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e2815aaa0 0x7f9e2815e820 secure :-1 s=READY pgs=2343 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e2402b9a0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 -- 172.21.15.29:0/550953313 shutdown_connections 2023-10-14T18:14:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 --2- 172.21.15.29:0/550953313 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e2815aaa0 0x7f9e2815e820 unknown :-1 s=CLOSED pgs=2343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 -- 172.21.15.29:0/550953313 >> 172.21.15.29:0/550953313 conn(0x7f9e280cb410 msgr2=0x7f9e280cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:39.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 -- 172.21.15.29:0/550953313 shutdown_connections 2023-10-14T18:14:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.712+0000 7f9e30244640 1 -- 172.21.15.29:0/550953313 wait complete. 2023-10-14T18:14:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.714+0000 7f9e30244640 1 Processor -- start 2023-10-14T18:14:39.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.714+0000 7f9e30244640 1 -- start start 2023-10-14T18:14:39.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:39.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e2815aaa0 0x7f9e28145330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:39.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e2815ae80 0x7f9e281f1fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:39.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280c3b80 con 0x7f9e280ce390 2023-10-14T18:14:39.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e280c4120 con 0x7f9e2815aaa0 2023-10-14T18:14:39.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e30244640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2815e190 con 0x7f9e2815ae80 2023-10-14T18:14:39.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:39.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39066/0 (socket says 172.21.15.29:39066) 2023-10-14T18:14:39.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/2303338023 learned_addr learned my addr 172.21.15.29:0/2303338023 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:39.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e2815ae80 msgr2=0x7f9e281f1fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e2815ae80 0x7f9e281f1fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e2815aaa0 msgr2=0x7f9e28145330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e2815aaa0 0x7f9e28145330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.715+0000 7f9e2dfe0640 1 -- 172.21.15.29:0/2303338023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e240094d0 con 0x7f9e280ce390 2023-10-14T18:14:39.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e2dfe0640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e24004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:39.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e24039d60 con 0x7f9e280ce390 2023-10-14T18:14:39.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e24043400 con 0x7f9e280ce390 2023-10-14T18:14:39.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e240437b0 con 0x7f9e280ce390 2023-10-14T18:14:39.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e28145870 con 0x7f9e280ce390 2023-10-14T18:14:39.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.716+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2815deb0 con 0x7f9e280ce390 2023-10-14T18:14:39.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.718+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2804e160 con 0x7f9e280ce390 2023-10-14T18:14:39.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.719+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9e24043910 con 0x7f9e280ce390 2023-10-14T18:14:39.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.719+0000 7f9e1e7fc640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 0x7f9e0c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:39.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.720+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7f9e2403e070 con 0x7f9e280ce390 2023-10-14T18:14:39.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.720+0000 7f9e2d7df640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 0x7f9e0c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:39.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.720+0000 7f9e2d7df640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 0x7f9e0c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18006bc0 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:39.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.724+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9e24077af0 con 0x7f9e280ce390 2023-10-14T18:14:39.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.960+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2815e350 con 0x7f9e280ce390 2023-10-14T18:14:39.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.961+0000 7f9e1e7fc640 1 -- 172.21.15.29:0/2303338023 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+3704 (secure 0 0 0) 0x7f9e24034090 con 0x7f9e280ce390 2023-10-14T18:14:39.962 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:39.962 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":93,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":6782,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1596676753","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1596676753},{"type":"v1","addr":"172.21.15.186:6833","nonce":1596676753}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":86},{"gid":6794,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2610200482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2610200482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2610200482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10125,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/137307500","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":137307500},{"type":"v1","addr":"172.21.15.29:6835","nonce":137307500}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":87},{"gid":10629,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3195978329","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":3195978329},{"type":"v1","addr":"172.21.15.29:6837","nonce":3195978329}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":93}],"filesystems":[]} 2023-10-14T18:14:39.963 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 93 2023-10-14T18:14:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 msgr2=0x7f9e0c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 0x7f9e0c06d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9e18006fd0 tx=0x7f9e18006bc0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 msgr2=0x7f9e281428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:39.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 secure :-1 s=READY pgs=2344 cs=0 l=1 rev1=1 crypto rx=0x7f9e24009870 tx=0x7f9e24004730 comp rx=0 tx=0).stop 2023-10-14T18:14:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 shutdown_connections 2023-10-14T18:14:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9e0c06afc0 0x7f9e0c06d480 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9e2815ae80 0x7f9e281f1fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9e2815aaa0 0x7f9e28145330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 --2- 172.21.15.29:0/2303338023 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9e280ce390 0x7f9e281428d0 unknown :-1 s=CLOSED pgs=2344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 >> 172.21.15.29:0/2303338023 conn(0x7f9e280cb410 msgr2=0x7f9e280b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 shutdown_connections 2023-10-14T18:14:39.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:39.964+0000 7f9e30244640 1 -- 172.21.15.29:0/2303338023 wait complete. 2023-10-14T18:14:39.985 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T18:14:39.985 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T18:14:39.986 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T18:14:39.986 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:14:39.986 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T18:14:39.987 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T18:14:39.987 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T18:14:39.987 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:14:39.988 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T18:14:39.988 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T18:14:39.988 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T18:14:39.988 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:14:39.989 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T18:14:39.989 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T18:14:39.989 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T18:14:39.989 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:14:40.006 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T18:14:40.007 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T18:14:40.009 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T18:14:40.009 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T18:14:40.011 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T18:14:40.012 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T18:14:40.014 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T18:14:40.014 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T18:14:40.057 INFO:tasks.ceph.mds.c:Started 2023-10-14T18:14:40.058 DEBUG:teuthology.parallel:result is None 2023-10-14T18:14:40.062 INFO:tasks.ceph.mds.b:Started 2023-10-14T18:14:40.062 DEBUG:teuthology.parallel:result is None 2023-10-14T18:14:40.065 INFO:tasks.ceph.mds.a:Started 2023-10-14T18:14:40.066 DEBUG:teuthology.parallel:result is None 2023-10-14T18:14:40.070 INFO:tasks.ceph.mds.d:Started 2023-10-14T18:14:40.070 DEBUG:teuthology.parallel:result is None 2023-10-14T18:14:40.071 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T18:14:40.224 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T18:14:40.225 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:14:40.225 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T18:14:40.231 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:14:40.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.267+0000 7fe787807640 1 Processor -- start 2023-10-14T18:14:40.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.267+0000 7fe787807640 1 -- start start 2023-10-14T18:14:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe78015ad70 0x7fe780069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7800c9ca0 0x7fe78006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe7800ca3b0 0x7fe7800ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7800cad80 con 0x7fe7800c9ca0 2023-10-14T18:14:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7800cb030 con 0x7fe7800ca3b0 2023-10-14T18:14:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.268+0000 7fe787807640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7800c91c0 con 0x7fe78015ad70 2023-10-14T18:14:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7800c9ca0 0x7fe78006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe7800c9ca0 0x7fe78006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39098/0 (socket says 172.21.15.29:39098) 2023-10-14T18:14:40.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 -- 172.21.15.29:0/2297977881 learned_addr learned my addr 172.21.15.29:0/2297977881 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe7855a3640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe78015ad70 0x7fe780069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe785da4640 1 --2- 172.21.15.29:0/2297977881 >> v2:172.21.15.186:3300/0 conn(0x7fe7800ca3b0 0x7fe7800ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 -- 172.21.15.29:0/2297977881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe78015ad70 msgr2=0x7fe780069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:40.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe78015ad70 0x7fe780069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 -- 172.21.15.29:0/2297977881 >> v2:172.21.15.186:3300/0 conn(0x7fe7800ca3b0 msgr2=0x7fe7800ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 --2- 172.21.15.29:0/2297977881 >> v2:172.21.15.186:3300/0 conn(0x7fe7800ca3b0 0x7fe7800ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.269+0000 7fe784da2640 1 -- 172.21.15.29:0/2297977881 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7800c94a0 con 0x7fe7800c9ca0 2023-10-14T18:14:40.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.270+0000 7fe784da2640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800c9ca0 0x7fe78006a2e0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cf820 tx=0x7fe77002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c65e3b138f8b920 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:40.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.271+0000 7fe777fff640 1 -- 172.21.15.29:0/2297977881 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770031870 con 0x7fe7800c9ca0 2023-10-14T18:14:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.271+0000 7fe777fff640 1 -- 172.21.15.29:0/2297977881 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7700319d0 con 0x7fe7800c9ca0 2023-10-14T18:14:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.271+0000 7fe777fff640 1 -- 172.21.15.29:0/2297977881 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770031d60 con 0x7fe7800c9ca0 2023-10-14T18:14:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.271+0000 7fe787807640 1 -- 172.21.15.29:0/2297977881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800c9ca0 msgr2=0x7fe78006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:40.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.271+0000 7fe787807640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800c9ca0 0x7fe78006a2e0 secure :-1 s=READY pgs=2349 cs=0 l=1 rev1=1 crypto rx=0x7fe7800cf820 tx=0x7fe77002b910 comp rx=0 tx=0).stop 2023-10-14T18:14:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 -- 172.21.15.29:0/2297977881 shutdown_connections 2023-10-14T18:14:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 --2- 172.21.15.29:0/2297977881 >> v2:172.21.15.186:3300/0 conn(0x7fe7800ca3b0 0x7fe7800ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800c9ca0 0x7fe78006a2e0 unknown :-1 s=CLOSED pgs=2349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 --2- 172.21.15.29:0/2297977881 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe78015ad70 0x7fe780069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 -- 172.21.15.29:0/2297977881 >> 172.21.15.29:0/2297977881 conn(0x7fe7800ce9b0 msgr2=0x7fe7800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:40.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 -- 172.21.15.29:0/2297977881 shutdown_connections 2023-10-14T18:14:40.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.272+0000 7fe787807640 1 -- 172.21.15.29:0/2297977881 wait complete. 2023-10-14T18:14:40.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.273+0000 7fe787807640 1 Processor -- start 2023-10-14T18:14:40.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.273+0000 7fe787807640 1 -- start start 2023-10-14T18:14:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7800c9ca0 0x7fe78014f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe78015ad70 0x7fe7800b26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78014ee40 con 0x7fe7800ca3b0 2023-10-14T18:14:40.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe78014f810 con 0x7fe78015ad70 2023-10-14T18:14:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.274+0000 7fe787807640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe78014e230 con 0x7fe7800c9ca0 2023-10-14T18:14:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39102/0 (socket says 172.21.15.29:39102) 2023-10-14T18:14:40.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 -- 172.21.15.29:0/1414878377 learned_addr learned my addr 172.21.15.29:0/1414878377 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe7855a3640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7800c9ca0 0x7fe78014f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe785da4640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe78015ad70 0x7fe7800b26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 -- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7800c9ca0 msgr2=0x7fe78014f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7800c9ca0 0x7fe78014f2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 -- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe78015ad70 msgr2=0x7fe7800b26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe78015ad70 0x7fe7800b26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 -- 172.21.15.29:0/1414878377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7700094d0 con 0x7fe7800ca3b0 2023-10-14T18:14:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.275+0000 7fe784da2640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe7700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.276+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe77003ac60 con 0x7fe7800ca3b0 2023-10-14T18:14:40.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.276+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe77003adc0 con 0x7fe7800ca3b0 2023-10-14T18:14:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.276+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe770044630 con 0x7fe7800ca3b0 2023-10-14T18:14:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.276+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78014e510 con 0x7fe7800ca3b0 2023-10-14T18:14:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.276+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe78014ead0 con 0x7fe7800ca3b0 2023-10-14T18:14:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.277+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe78014efb0 con 0x7fe7800ca3b0 2023-10-14T18:14:40.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.278+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe77004d450 con 0x7fe7800ca3b0 2023-10-14T18:14:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.279+0000 7fe775ffb640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 0x7fe76406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.279+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(201..201 src has 1..201) v4 ==== 6091+0+0 (secure 0 0 0) 0x7fe77002d040 con 0x7fe7800ca3b0 2023-10-14T18:14:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.279+0000 7fe7855a3640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 0x7fe76406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.280+0000 7fe7855a3640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 0x7fe76406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe78006a640 tx=0x7fe77c008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:40.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.283+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe770076cc0 con 0x7fe7800ca3b0 2023-10-14T18:14:40.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:40.499+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fe78004e160 con 0x7fe7800ca3b0 2023-10-14T18:14:41.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.495+0000 7fe775ffb640 1 -- 172.21.15.29:0/1414878377 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v206) v1 ==== 97+0+0 (secure 0 0 0) 0x7fe77007a2e0 con 0x7fe7800ca3b0 2023-10-14T18:14:41.496 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T18:14:41.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.498+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 msgr2=0x7fe76406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.498+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 0x7fe76406d450 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe78006a640 tx=0x7fe77c008040 comp rx=0 tx=0).stop 2023-10-14T18:14:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.498+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 msgr2=0x7fe7800c81c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.498+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 secure :-1 s=READY pgs=2350 cs=0 l=1 rev1=1 crypto rx=0x7fe770000c00 tx=0x7fe7700046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 shutdown_connections 2023-10-14T18:14:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe78015ad70 0x7fe7800b26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe7800ca3b0 0x7fe7800c81c0 unknown :-1 s=CLOSED pgs=2350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fe76406af90 0x7fe76406d450 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 --2- 172.21.15.29:0/1414878377 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe7800c9ca0 0x7fe78014f2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 >> 172.21.15.29:0/1414878377 conn(0x7fe7800ce9b0 msgr2=0x7fe78015b4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 shutdown_connections 2023-10-14T18:14:41.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.499+0000 7fe787807640 1 -- 172.21.15.29:0/1414878377 wait complete. 2023-10-14T18:14:41.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T18:14:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.741+0000 7f6f29cc2640 1 Processor -- start 2023-10-14T18:14:41.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.741+0000 7f6f29cc2640 1 -- start start 2023-10-14T18:14:41.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.743+0000 7f6f29cc2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f240b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.743+0000 7f6f29cc2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6f24144440 0x7f6f2415e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.743+0000 7f6f29cc2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f2406f1d0 con 0x7f6f2415f1b0 2023-10-14T18:14:41.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.743+0000 7f6f29cc2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f240cd430 con 0x7f6f24144440 2023-10-14T18:14:41.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.743+0000 7f6f29cc2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f240cd610 con 0x7f6f24144060 2023-10-14T18:14:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f1bfff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6f24144440 0x7f6f2415e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f28cc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f240b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f294c1640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f6f2415f1b0 0x7f6f2415afb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:43434/0 (socket says 172.21.15.29:43434) 2023-10-14T18:14:41.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f294c1640 1 -- 172.21.15.29:0/1101276142 learned_addr learned my addr 172.21.15.29:0/1101276142 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:41.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f1bfff640 1 -- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 msgr2=0x7f6f240b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:41.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f1bfff640 1 --2- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f240b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f1bfff640 1 -- 172.21.15.29:0/1101276142 >> v1:172.21.15.186:6789/0 conn(0x7f6f2415f1b0 legacy=0x7f6f2415afb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.744+0000 7f6f1bfff640 1 -- 172.21.15.29:0/1101276142 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f240cd8f0 con 0x7f6f24144440 2023-10-14T18:14:41.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.746+0000 7f6f1bfff640 1 --2- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f24144440 0x7f6f2415e5a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6f140097a0 tx=0x7f6f1402b820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=35bcb70f1a3a51d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:41.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.746+0000 7f6f1b7fe640 1 -- 172.21.15.29:0/1101276142 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14031780 con 0x7f6f24144440 2023-10-14T18:14:41.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.746+0000 7f6f1b7fe640 1 -- 172.21.15.29:0/1101276142 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f140318e0 con 0x7f6f24144440 2023-10-14T18:14:41.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.746+0000 7f6f1b7fe640 1 -- 172.21.15.29:0/1101276142 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14031c70 con 0x7f6f24144440 2023-10-14T18:14:41.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 -- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f24144440 msgr2=0x7f6f2415e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f24144440 0x7f6f2415e5a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6f140097a0 tx=0x7f6f1402b820 comp rx=0 tx=0).stop 2023-10-14T18:14:41.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 -- 172.21.15.29:0/1101276142 shutdown_connections 2023-10-14T18:14:41.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f24144440 0x7f6f2415e5a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/1101276142 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f240b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 -- 172.21.15.29:0/1101276142 >> 172.21.15.29:0/1101276142 conn(0x7f6f24069da0 msgr2=0x7f6f2406a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:41.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 -- 172.21.15.29:0/1101276142 shutdown_connections 2023-10-14T18:14:41.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.747+0000 7f6f29cc2640 1 -- 172.21.15.29:0/1101276142 wait complete. 2023-10-14T18:14:41.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.749+0000 7f6f29cc2640 1 Processor -- start 2023-10-14T18:14:41.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.749+0000 7f6f29cc2640 1 -- start start 2023-10-14T18:14:41.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f241f64f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 0x7f6f2413ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f24141d40 con 0x7f6f24144440 2023-10-14T18:14:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2413b380 con 0x7f6f2415f1b0 2023-10-14T18:14:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f29cc2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f2413b8e0 con 0x7f6f24144060 2023-10-14T18:14:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f1bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50048/0 (socket says 172.21.15.29:50048) 2023-10-14T18:14:41.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.750+0000 7f6f1bfff640 1 -- 172.21.15.29:0/3795748928 learned_addr learned my addr 172.21.15.29:0/3795748928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f28cc0640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f241f64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f294c1640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 0x7f6f2413ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 -- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 msgr2=0x7f6f241f64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f241f64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 -- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 msgr2=0x7f6f2413ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 0x7f6f2413ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 -- 172.21.15.29:0/3795748928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f14009400 con 0x7f6f24144440 2023-10-14T18:14:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f28cc0640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f241f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f294c1640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 0x7f6f2413ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:41.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.751+0000 7f6f1bfff640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f6f140098c0 tx=0x7f6f140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:41.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.752+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14032820 con 0x7f6f24144440 2023-10-14T18:14:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.752+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14032980 con 0x7f6f24144440 2023-10-14T18:14:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.752+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f2413b600 con 0x7f6f24144440 2023-10-14T18:14:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.752+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f14032d30 con 0x7f6f24144440 2023-10-14T18:14:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.752+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f241419d0 con 0x7f6f24144440 2023-10-14T18:14:41.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.753+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f240674f0 con 0x7f6f24144440 2023-10-14T18:14:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.754+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6f1404e440 con 0x7f6f24144440 2023-10-14T18:14:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.755+0000 7f6f197fa640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 0x7f6f1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.755+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f6f140ab650 con 0x7f6f24144440 2023-10-14T18:14:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.755+0000 7f6f28cc0640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 0x7f6f1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:41.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.756+0000 7f6f28cc0640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 0x7f6f1006d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f200074c0 tx=0x7f6f2000b040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:41.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:41.759+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f14076900 con 0x7f6f24144440 2023-10-14T18:14:42.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.026+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f6f24071490 con 0x7f6f24144440 2023-10-14T18:14:42.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.027+0000 7f6f197fa640 1 -- 172.21.15.29:0/3795748928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v48) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f6f14079f20 con 0x7f6f24144440 2023-10-14T18:14:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:42.028 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:14:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T18:14:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.028 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:14:42.029 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T18:14:42.030 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T18:14:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:14:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:14:42.031 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T18:14:42.032 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:14:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:14:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.033 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T18:14:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T18:14:42.034 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:14:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:14:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.035 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.036 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.037 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T18:14:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T18:14:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.038 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T18:14:42.039 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.040 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T18:14:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T18:14:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.041 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.042 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.043 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T18:14:42.044 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.045 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T18:14:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T18:14:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.046 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.047 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T18:14:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T18:14:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:14:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:14:42.048 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T18:14:42.049 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T18:14:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T18:14:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.050 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T18:14:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T18:14:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.051 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.052 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T18:14:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T18:14:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.053 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:14:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:14:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:14:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.054 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T18:14:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T18:14:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:14:42.055 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.056 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T18:14:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T18:14:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:14:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:14:42.057 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:14:42.058 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.058 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.058 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.058 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.058 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T18:14:42.059 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T18:14:42.059 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.059 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:14:42.059 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:14:42.059 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T18:14:42.060 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:14:42.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:14:42.061 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:14:42.061 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:14:42.061 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.061 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:42.062 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T18:14:42.062 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:14:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 msgr2=0x7f6f1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 0x7f6f1006d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f6f200074c0 tx=0x7f6f2000b040 comp rx=0 tx=0).stop 2023-10-14T18:14:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 msgr2=0x7f6f240b1680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 secure :-1 s=READY pgs=2351 cs=0 l=1 rev1=1 crypto rx=0x7f6f140098c0 tx=0x7f6f140046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 shutdown_connections 2023-10-14T18:14:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f2415f1b0 0x7f6f2413ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f24144440 0x7f6f240b1680 unknown :-1 s=CLOSED pgs=2351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f6f1006af90 0x7f6f1006d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 --2- 172.21.15.29:0/3795748928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f24144060 0x7f6f241f64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 >> 172.21.15.29:0/3795748928 conn(0x7f6f24069da0 msgr2=0x7f6f240cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:42.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.029+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 shutdown_connections 2023-10-14T18:14:42.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.030+0000 7f6f29cc2640 1 -- 172.21.15.29:0/3795748928 wait complete. 2023-10-14T18:14:42.066 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.289+0000 7fdd16044640 1 Processor -- start 2023-10-14T18:14:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.289+0000 7fdd16044640 1 -- start start 2023-10-14T18:14:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.290+0000 7fdd16044640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd10160420 0x7fdd100ae170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.290+0000 7fdd16044640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdd1006b4c0 0x7fdd100cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.290+0000 7fdd16044640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd1006e170 con 0x7fdd10160800 2023-10-14T18:14:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.290+0000 7fdd16044640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd100cde30 con 0x7fdd1006b4c0 2023-10-14T18:14:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.290+0000 7fdd16044640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd100cd460 con 0x7fdd10160420 2023-10-14T18:14:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd10160800 0x7fdd1006adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38850/0 (socket says 172.21.15.29:38850) 2023-10-14T18:14:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0effd640 1 -- 172.21.15.29:0/2342094638 learned_addr learned my addr 172.21.15.29:0/2342094638 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/2342094638 >> v2:172.21.15.29:3300/0 conn(0x7fdd1006b4c0 0x7fdd100cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0f7fe640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd10160420 0x7fdd100ae170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0ffff640 1 -- 172.21.15.29:0/2342094638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd10160420 msgr2=0x7fdd100ae170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:42.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd10160420 0x7fdd100ae170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0ffff640 1 -- 172.21.15.29:0/2342094638 >> v1:172.21.15.29:6789/0 conn(0x7fdd10160800 legacy=0x7fdd1006adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.291+0000 7fdd0ffff640 1 -- 172.21.15.29:0/2342094638 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd1006e3c0 con 0x7fdd1006b4c0 2023-10-14T18:14:42.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.292+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd1006b4c0 0x7fdd100cd8f0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fdd080097a0 tx=0x7fdd0802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0631a42b319676f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd0e7fc640 1 -- 172.21.15.29:0/2342094638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0802d040 con 0x7fdd1006b4c0 2023-10-14T18:14:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd0e7fc640 1 -- 172.21.15.29:0/2342094638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd08031810 con 0x7fdd1006b4c0 2023-10-14T18:14:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd0e7fc640 1 -- 172.21.15.29:0/2342094638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08031ba0 con 0x7fdd1006b4c0 2023-10-14T18:14:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 -- 172.21.15.29:0/2342094638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd1006b4c0 msgr2=0x7fdd100cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd1006b4c0 0x7fdd100cd8f0 secure :-1 s=READY pgs=2353 cs=0 l=1 rev1=1 crypto rx=0x7fdd080097a0 tx=0x7fdd0802b8b0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 -- 172.21.15.29:0/2342094638 shutdown_connections 2023-10-14T18:14:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd1006b4c0 0x7fdd100cd8f0 unknown :-1 s=CLOSED pgs=2353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 --2- 172.21.15.29:0/2342094638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd10160420 0x7fdd100ae170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.293+0000 7fdd16044640 1 -- 172.21.15.29:0/2342094638 >> 172.21.15.29:0/2342094638 conn(0x7fdd100b0d30 msgr2=0x7fdd100b1140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:42.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.294+0000 7fdd16044640 1 -- 172.21.15.29:0/2342094638 shutdown_connections 2023-10-14T18:14:42.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.294+0000 7fdd16044640 1 -- 172.21.15.29:0/2342094638 wait complete. 2023-10-14T18:14:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.296+0000 7fdd16044640 1 Processor -- start 2023-10-14T18:14:42.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.296+0000 7fdd16044640 1 -- start start 2023-10-14T18:14:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd1006b4c0 0x7fdd1009e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 0x7fdd1009e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1009a080 con 0x7fdd10160800 2023-10-14T18:14:42.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd1009d3b0 con 0x7fdd10160420 2023-10-14T18:14:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.297+0000 7fdd16044640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd10159d30 con 0x7fdd1006b4c0 2023-10-14T18:14:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50066/0 (socket says 172.21.15.29:50066) 2023-10-14T18:14:42.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 -- 172.21.15.29:0/50718807 learned_addr learned my addr 172.21.15.29:0/50718807 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0f7fe640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd1006b4c0 0x7fdd1009e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0effd640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 0x7fdd1009e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 -- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd1006b4c0 msgr2=0x7fdd1009e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd1006b4c0 0x7fdd1009e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 -- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 msgr2=0x7fdd1009e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 0x7fdd1009e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0ffff640 1 -- 172.21.15.29:0/50718807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd08009400 con 0x7fdd10160800 2023-10-14T18:14:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.298+0000 7fdd0effd640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 0x7fdd1009e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:42.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd0ffff640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fdd080098c0 tx=0x7fdd080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd0803ace0 con 0x7fdd10160800 2023-10-14T18:14:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd0803ae60 con 0x7fdd10160800 2023-10-14T18:14:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd08042630 con 0x7fdd10160800 2023-10-14T18:14:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd1009d600 con 0x7fdd10160800 2023-10-14T18:14:42.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.299+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd1009dbc0 con 0x7fdd10160800 2023-10-14T18:14:42.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.300+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd1004e160 con 0x7fdd10160800 2023-10-14T18:14:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.301+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd0804b3b0 con 0x7fdd10160800 2023-10-14T18:14:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.302+0000 7fdd07fff640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.302+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fdd080ab620 con 0x7fdd10160800 2023-10-14T18:14:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.302+0000 7fdd0f7fe640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.303+0000 7fdd0f7fe640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 0x7fdcf006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009a60 tx=0x7fdd00003990 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.308 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.307+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd08076820 con 0x7fdd10160800 2023-10-14T18:14:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.496+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd100b51c0 con 0x7fdd10160800 2023-10-14T18:14:42.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.497+0000 7fdd07fff640 1 -- 172.21.15.29:0/50718807 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v206) v1 ==== 74+0+13041 (secure 0 0 0) 0x7fdd08079e40 con 0x7fdd10160800 2023-10-14T18:14:42.498 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:42.498 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":206,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:41.491893+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":23,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":190,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:42.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 msgr2=0x7fdcf006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 0x7fdcf006d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fdd00009a60 tx=0x7fdd00003990 comp rx=0 tx=0).stop 2023-10-14T18:14:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 msgr2=0x7fdd1009ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 secure :-1 s=READY pgs=2354 cs=0 l=1 rev1=1 crypto rx=0x7fdd080098c0 tx=0x7fdd080046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 shutdown_connections 2023-10-14T18:14:42.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd10160800 0x7fdd1009ee40 unknown :-1 s=CLOSED pgs=2354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd10160420 0x7fdd1009e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fdcf006af90 0x7fdcf006d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 --2- 172.21.15.29:0/50718807 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd1006b4c0 0x7fdd1009e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.499+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 >> 172.21.15.29:0/50718807 conn(0x7fdd100b0d30 msgr2=0x7fdd1015a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.500+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 shutdown_connections 2023-10-14T18:14:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.500+0000 7fdd16044640 1 -- 172.21.15.29:0/50718807 wait complete. 2023-10-14T18:14:42.519 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:14:42.521 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:14:42.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.725+0000 7fbca18bb640 1 Processor -- start 2023-10-14T18:14:42.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.726+0000 7fbca18bb640 1 -- start start 2023-10-14T18:14:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.727+0000 7fbca18bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 0x7fbc9c06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.727+0000 7fbca18bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbc9c06b3e0 0x7fbc9c0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.727+0000 7fbca18bb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc9c0cd260 con 0x7fbc9c159ad0 2023-10-14T18:14:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.727+0000 7fbca18bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c15de30 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.727+0000 7fbca18bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc9c15e110 con 0x7fbc9c070f30 2023-10-14T18:14:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca08b9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbc9c159ad0 0x7fbc9c15d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38864/0 (socket says 172.21.15.29:38864) 2023-10-14T18:14:42.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca08b9640 1 -- 172.21.15.29:0/1880309124 learned_addr learned my addr 172.21.15.29:0/1880309124 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:42.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca10ba640 1 --2- 172.21.15.29:0/1880309124 >> v2:172.21.15.29:3300/0 conn(0x7fbc9c06b3e0 0x7fbc9c0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/1880309124 >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 0x7fbc9c06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca10ba640 1 -- 172.21.15.29:0/1880309124 >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 msgr2=0x7fbc9c06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:42.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca10ba640 1 --2- 172.21.15.29:0/1880309124 >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 0x7fbc9c06aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca10ba640 1 -- 172.21.15.29:0/1880309124 >> v1:172.21.15.29:6789/0 conn(0x7fbc9c159ad0 legacy=0x7fbc9c15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbca10ba640 1 -- 172.21.15.29:0/1880309124 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc9c15e3f0 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbc9b7fe640 1 -- 172.21.15.29:0/1880309124 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 207091125 0 0) 0x7fbc9c0cd260 con 0x7fbc9c159ad0 2023-10-14T18:14:42.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.728+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/1880309124 >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 0x7fbc9c06aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:42.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.729+0000 7fbca10ba640 1 --2- 172.21.15.29:0/1880309124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c0c9de0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c066ab0 tx=0x7fbc9402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=126179f0b193a586 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbc9b7fe640 1 -- 172.21.15.29:0/1880309124 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94031830 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbc9b7fe640 1 -- 172.21.15.29:0/1880309124 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc94031990 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbc9b7fe640 1 -- 172.21.15.29:0/1880309124 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc94031d20 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 -- 172.21.15.29:0/1880309124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c06b3e0 msgr2=0x7fbc9c0c9de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 --2- 172.21.15.29:0/1880309124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c0c9de0 secure :-1 s=READY pgs=2356 cs=0 l=1 rev1=1 crypto rx=0x7fbc9c066ab0 tx=0x7fbc9402b8d0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 -- 172.21.15.29:0/1880309124 shutdown_connections 2023-10-14T18:14:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 --2- 172.21.15.29:0/1880309124 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c0c9de0 unknown :-1 s=CLOSED pgs=2356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 --2- 172.21.15.29:0/1880309124 >> v2:172.21.15.186:3300/0 conn(0x7fbc9c070f30 0x7fbc9c06aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 -- 172.21.15.29:0/1880309124 >> 172.21.15.29:0/1880309124 conn(0x7fbc9c0b0170 msgr2=0x7fbc9c0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:42.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 -- 172.21.15.29:0/1880309124 shutdown_connections 2023-10-14T18:14:42.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.730+0000 7fbca18bb640 1 -- 172.21.15.29:0/1880309124 wait complete. 2023-10-14T18:14:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.732+0000 7fbca18bb640 1 Processor -- start 2023-10-14T18:14:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.732+0000 7fbca18bb640 1 -- start start 2023-10-14T18:14:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c1f1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc9c159ad0 0x7fbc9c06bdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c15a420 con 0x7fbc9c070f30 2023-10-14T18:14:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0b24a0 con 0x7fbc9c06b3e0 2023-10-14T18:14:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbca18bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c0b2780 con 0x7fbc9c159ad0 2023-10-14T18:14:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbc9bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50082/0 (socket says 172.21.15.29:50082) 2023-10-14T18:14:42.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.733+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4198480943 learned_addr learned my addr 172.21.15.29:0/4198480943 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc9c159ad0 msgr2=0x7fbc9c06bdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc9c159ad0 0x7fbc9c06bdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc9c06b3e0 msgr2=0x7fbc9c1f1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbca08b9640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c1f1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c1f1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 -- 172.21.15.29:0/4198480943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc94009400 con 0x7fbc9c070f30 2023-10-14T18:14:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc9bfff640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000cf60 tx=0x7fbc9000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.734+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9000d840 con 0x7fbc9c070f30 2023-10-14T18:14:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.735+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc9000d9a0 con 0x7fbc9c070f30 2023-10-14T18:14:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.735+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc9000dd30 con 0x7fbc9c070f30 2023-10-14T18:14:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.735+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c0b2a60 con 0x7fbc9c070f30 2023-10-14T18:14:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.735+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c0b3020 con 0x7fbc9c070f30 2023-10-14T18:14:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.736+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c0b3240 con 0x7fbc9c070f30 2023-10-14T18:14:42.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.737+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbc90015440 con 0x7fbc9c070f30 2023-10-14T18:14:42.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.738+0000 7fbc997fa640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 0x7fbc8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:42.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.738+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fbc90089ee0 con 0x7fbc9c070f30 2023-10-14T18:14:42.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.738+0000 7fbca08b9640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 0x7fbc8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.738+0000 7fbca08b9640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 0x7fbc8806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c000f60 tx=0x7fbc8c009040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:42.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.742+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbc90055170 con 0x7fbc9c070f30 2023-10-14T18:14:42.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.935+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbc9c04e9b0 con 0x7fbc9c070f30 2023-10-14T18:14:42.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.936+0000 7fbc997fa640 1 -- 172.21.15.29:0/4198480943 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v206) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbc9001a0a0 con 0x7fbc9c070f30 2023-10-14T18:14:42.937 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:14:42.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 msgr2=0x7fbc8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 0x7fbc8806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fbc8c000f60 tx=0x7fbc8c009040 comp rx=0 tx=0).stop 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 msgr2=0x7fbc9c0c7e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 secure :-1 s=READY pgs=2357 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000cf60 tx=0x7fbc9000cf90 comp rx=0 tx=0).stop 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 shutdown_connections 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fbc8806af90 0x7fbc8806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbc9c159ad0 0x7fbc9c06bdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbc9c070f30 0x7fbc9c0c7e60 unknown :-1 s=CLOSED pgs=2357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 --2- 172.21.15.29:0/4198480943 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbc9c06b3e0 0x7fbc9c1f1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.938+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 >> 172.21.15.29:0/4198480943 conn(0x7fbc9c0b0170 msgr2=0x7fbc9c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.939+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 shutdown_connections 2023-10-14T18:14:42.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:42.939+0000 7fbca18bb640 1 -- 172.21.15.29:0/4198480943 wait complete. 2023-10-14T18:14:42.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:14:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.179+0000 7f965a5ba640 1 Processor -- start 2023-10-14T18:14:43.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.179+0000 7f965a5ba640 1 -- start start 2023-10-14T18:14:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f965a5ba640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f965415c100 0x7f96540ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f965a5ba640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96540cd4a0 con 0x7f96540ccb00 2023-10-14T18:14:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f965a5ba640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f965415ac60 con 0x7f9654153740 2023-10-14T18:14:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f965a5ba640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9654070030 con 0x7f965415c100 2023-10-14T18:14:43.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f9658db7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f96540ccb00 0x7f96540ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38870/0 (socket says 172.21.15.29:38870) 2023-10-14T18:14:43.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.181+0000 7f9658db7640 1 -- 172.21.15.29:0/2823193205 learned_addr learned my addr 172.21.15.29:0/2823193205 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f9659db9640 1 --2- 172.21.15.29:0/2823193205 >> v2:172.21.15.186:3300/0 conn(0x7f965415c100 0x7f96540ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 368906247 0 0) 0x7f96540cd4a0 con 0x7f96540ccb00 2023-10-14T18:14:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9640005730 con 0x7f96540ccb00 2023-10-14T18:14:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661594335 0 0) 0x7f9640005730 con 0x7f96540ccb00 2023-10-14T18:14:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 >> v2:172.21.15.186:3300/0 conn(0x7f965415c100 msgr2=0x7f96540ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 --2- 172.21.15.29:0/2823193205 >> v2:172.21.15.186:3300/0 conn(0x7f965415c100 0x7f96540ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 >> v1:172.21.15.186:6789/0 conn(0x7f9654153740 legacy=0x7f965415a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9654070310 con 0x7f96540ccb00 2023-10-14T18:14:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9644001670 con 0x7f96540ccb00 2023-10-14T18:14:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.182+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 >> v1:172.21.15.29:6789/0 conn(0x7f96540ccb00 legacy=0x7f96540ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640005b80 0x7f9640005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9640006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9640007390 0x7f964000db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9654070030 con 0x7f96400069d0 2023-10-14T18:14:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f964000e230 con 0x7f9640005b80 2023-10-14T18:14:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f964000e3f0 con 0x7f9640007390 2023-10-14T18:14:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9640006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f96595b8640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640005b80 0x7f9640005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 -- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9640007390 msgr2=0x7f964000db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9640007390 0x7f964000db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 -- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640005b80 msgr2=0x7f9640005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640005b80 0x7f9640005fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9658db7640 1 -- 172.21.15.29:0/2823193205 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9640005730 con 0x7f96400069d0 2023-10-14T18:14:43.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f96595b8640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9640005b80 0x7f9640005fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:43.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.183+0000 7f9659db9640 1 -- 172.21.15.29:0/2823193205 reap_dead start 2023-10-14T18:14:43.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.184+0000 7f9658db7640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9640006e50 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f964400c240 tx=0x7f9640007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=611a2682f3459511 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9644005ad0 con 0x7f96400069d0 2023-10-14T18:14:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9644005c30 con 0x7f96400069d0 2023-10-14T18:14:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f964bfff640 1 -- 172.21.15.29:0/2823193205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f964402cab0 con 0x7f96400069d0 2023-10-14T18:14:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 -- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 msgr2=0x7f9640006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9640006e50 secure :-1 s=READY pgs=2359 cs=0 l=1 rev1=1 crypto rx=0x7f964400c240 tx=0x7f9640007a00 comp rx=0 tx=0).stop 2023-10-14T18:14:43.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 -- 172.21.15.29:0/2823193205 shutdown_connections 2023-10-14T18:14:43.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 --2- 172.21.15.29:0/2823193205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9640006e50 unknown :-1 s=CLOSED pgs=2359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 -- 172.21.15.29:0/2823193205 >> 172.21.15.29:0/2823193205 conn(0x7f96540ce6a0 msgr2=0x7f96540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:43.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 -- 172.21.15.29:0/2823193205 shutdown_connections 2023-10-14T18:14:43.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.185+0000 7f965a5ba640 1 -- 172.21.15.29:0/2823193205 wait complete. 2023-10-14T18:14:43.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.187+0000 7f965a5ba640 1 Processor -- start 2023-10-14T18:14:43.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.187+0000 7f965a5ba640 1 -- start start 2023-10-14T18:14:43.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 0x7f96540c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96541ee050 0x7f96540ccee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965406b740 con 0x7f96400069d0 2023-10-14T18:14:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f965406ba20 con 0x7f96541525a0 2023-10-14T18:14:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f965a5ba640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96541556e0 con 0x7f96541ee050 2023-10-14T18:14:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f96595b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f96595b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50102/0 (socket says 172.21.15.29:50102) 2023-10-14T18:14:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.188+0000 7f96595b8640 1 -- 172.21.15.29:0/1560791701 learned_addr learned my addr 172.21.15.29:0/1560791701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f9658db7640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 0x7f96540c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f9659db9640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96541ee050 0x7f96540ccee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 -- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96541ee050 msgr2=0x7f96540ccee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96541ee050 0x7f96540ccee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 -- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 msgr2=0x7f96540c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 0x7f96540c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 -- 172.21.15.29:0/1560791701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96541559c0 con 0x7f96400069d0 2023-10-14T18:14:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f9658db7640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 0x7f96540c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:43.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f96595b8640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f965000d480 tx=0x7f965000d940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.189+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650005480 con 0x7f96400069d0 2023-10-14T18:14:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.190+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9650003610 con 0x7f96400069d0 2023-10-14T18:14:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.190+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9650003a00 con 0x7f96400069d0 2023-10-14T18:14:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.190+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9654148f90 con 0x7f96400069d0 2023-10-14T18:14:43.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.190+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96541ee4d0 con 0x7f96400069d0 2023-10-14T18:14:43.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.191+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9654155540 con 0x7f96400069d0 2023-10-14T18:14:43.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.192+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9650003b60 con 0x7f96400069d0 2023-10-14T18:14:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.193+0000 7f9649ffb640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.193+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f9650089400 con 0x7f96400069d0 2023-10-14T18:14:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.193+0000 7f9658db7640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.194+0000 7f9658db7640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 0x7f963806d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9644034000 tx=0x7f9644009030 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.198+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9654155540 con 0x7f96400069d0 2023-10-14T18:14:43.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.397+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f9654155540 con 0x7f96400069d0 2023-10-14T18:14:43.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.398+0000 7f9649ffb640 1 -- 172.21.15.29:0/1560791701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v206) v1 ==== 84+0+65 (secure 0 0 0) 0x7f9654155540 con 0x7f96400069d0 2023-10-14T18:14:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:43.399 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:14:43.399 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:14:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:14:43.400 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:43.400 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:14:43.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.400+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 msgr2=0x7f963806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.400+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 0x7f963806d480 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9644034000 tx=0x7f9644009030 comp rx=0 tx=0).stop 2023-10-14T18:14:43.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 msgr2=0x7f9654148a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 secure :-1 s=READY pgs=2360 cs=0 l=1 rev1=1 crypto rx=0x7f965000d480 tx=0x7f965000d940 comp rx=0 tx=0).stop 2023-10-14T18:14:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 shutdown_connections 2023-10-14T18:14:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f963806afc0 0x7f963806d480 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96541ee050 0x7f96540ccee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96541525a0 0x7f96540c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 --2- 172.21.15.29:0/1560791701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96400069d0 0x7f9654148a50 unknown :-1 s=CLOSED pgs=2360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 >> 172.21.15.29:0/1560791701 conn(0x7f96540ce6a0 msgr2=0x7f96540b3200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:43.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 shutdown_connections 2023-10-14T18:14:43.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.401+0000 7f965a5ba640 1 -- 172.21.15.29:0/1560791701 wait complete. 2023-10-14T18:14:43.420 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T18:14:43.420 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T18:14:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.640+0000 7f0b7f57d640 1 Processor -- start 2023-10-14T18:14:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.641+0000 7f0b7f57d640 1 -- start start 2023-10-14T18:14:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.642+0000 7f0b7f57d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b780c8fa0 0x7f0b78165aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.642+0000 7f0b7f57d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 0x7f0b780b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.642+0000 7f0b7f57d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b78165610 con 0x7f0b780c1d30 2023-10-14T18:14:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.642+0000 7f0b7f57d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b780b0cb0 con 0x7f0b780c8fa0 2023-10-14T18:14:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.642+0000 7f0b7f57d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780ce370 con 0x7f0b780b0360 2023-10-14T18:14:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b7d319640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b780c1d30 0x7f0b780cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38876/0 (socket says 172.21.15.29:38876) 2023-10-14T18:14:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b7d319640 1 -- 172.21.15.29:0/4115745864 learned_addr learned my addr 172.21.15.29:0/4115745864 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b7db1a640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 0x7f0b780b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/4115745864 >> v2:172.21.15.186:3300/0 conn(0x7f0b780c8fa0 0x7f0b78165aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771663187 0 0) 0x7f0b78165610 con 0x7f0b780c1d30 2023-10-14T18:14:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b64003520 con 0x7f0b780c1d30 2023-10-14T18:14:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 848457676 0 0) 0x7f0b64003520 con 0x7f0b780c1d30 2023-10-14T18:14:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 msgr2=0x7f0b780b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 0x7f0b780b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 >> v2:172.21.15.186:3300/0 conn(0x7f0b780c8fa0 msgr2=0x7f0b78165aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/4115745864 >> v2:172.21.15.186:3300/0 conn(0x7f0b780c8fa0 0x7f0b78165aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b780ce650 con 0x7f0b780c1d30 2023-10-14T18:14:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.643+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f0b74001670 con 0x7f0b780c1d30 2023-10-14T18:14:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 >> v1:172.21.15.29:6789/0 conn(0x7f0b780c1d30 legacy=0x7f0b780cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b640047c0 0x7f0b64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 0x7f0b64005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780b0cb0 con 0x7f0b640039e0 2023-10-14T18:14:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b780ce370 con 0x7f0b640047c0 2023-10-14T18:14:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6400ba80 con 0x7f0b780b0360 2023-10-14T18:14:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/4115745864 >> v2:172.21.15.186:3300/0 conn(0x7f0b780c8fa0 0x7f0b78165aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 -- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 msgr2=0x7f0b64005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b780b0360 0x7f0b64005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 -- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b640047c0 msgr2=0x7f0b64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b640047c0 0x7f0b64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b640047c0 0x7f0b64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7d319640 1 -- 172.21.15.29:0/4115745864 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b64003520 con 0x7f0b640039e0 2023-10-14T18:14:43.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.644+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b640047c0 0x7f0b64004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:43.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7d319640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400c110 tx=0x7f0b640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a44326a3d83d15a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b74005ad0 con 0x7f0b640039e0 2023-10-14T18:14:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b74005c50 con 0x7f0b640039e0 2023-10-14T18:14:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b6ffff640 1 -- 172.21.15.29:0/4115745864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7402c7c0 con 0x7f0b640039e0 2023-10-14T18:14:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7f57d640 1 -- 172.21.15.29:0/4115745864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b640039e0 msgr2=0x7f0b64003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/4115745864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b640039e0 0x7f0b64003e00 secure :-1 s=READY pgs=2362 cs=0 l=1 rev1=1 crypto rx=0x7f0b7400c110 tx=0x7f0b640057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7db1a640 1 -- 172.21.15.29:0/4115745864 reap_dead start 2023-10-14T18:14:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7f57d640 1 -- 172.21.15.29:0/4115745864 shutdown_connections 2023-10-14T18:14:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.646+0000 7f0b7f57d640 1 -- 172.21.15.29:0/4115745864 >> 172.21.15.29:0/4115745864 conn(0x7f0b780cb360 msgr2=0x7f0b780cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.647+0000 7f0b7f57d640 1 -- 172.21.15.29:0/4115745864 shutdown_connections 2023-10-14T18:14:43.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.647+0000 7f0b7f57d640 1 -- 172.21.15.29:0/4115745864 wait complete. 2023-10-14T18:14:43.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.648+0000 7f0b7f57d640 1 Processor -- start 2023-10-14T18:14:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.649+0000 7f0b7f57d640 1 -- start start 2023-10-14T18:14:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.649+0000 7f0b7f57d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b640039e0 0x7f0b781ecb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.649+0000 7f0b7f57d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 0x7f0b7814dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7f57d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7f57d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781ebab0 con 0x7f0b781ed090 2023-10-14T18:14:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7f57d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781ebd90 con 0x7f0b780c2ec0 2023-10-14T18:14:43.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7f57d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b781ec070 con 0x7f0b640039e0 2023-10-14T18:14:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50112/0 (socket says 172.21.15.29:50112) 2023-10-14T18:14:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 -- 172.21.15.29:0/3330554405 learned_addr learned my addr 172.21.15.29:0/3330554405 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 0x7f0b7814dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7d319640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b640039e0 0x7f0b781ecb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 -- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b640039e0 msgr2=0x7f0b781ecb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b640039e0 0x7f0b781ecb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 -- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 msgr2=0x7f0b7814dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:43.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 0x7f0b7814dc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7db1a640 1 -- 172.21.15.29:0/3330554405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b781ec350 con 0x7f0b781ed090 2023-10-14T18:14:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.650+0000 7f0b7cb18640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 0x7f0b7814dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b7db1a640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f0b780b13d0 tx=0x7f0b7000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b70010040 con 0x7f0b781ed090 2023-10-14T18:14:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b7000bae0 con 0x7f0b781ed090 2023-10-14T18:14:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b781ec630 con 0x7f0b781ed090 2023-10-14T18:14:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b7000be70 con 0x7f0b781ed090 2023-10-14T18:14:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.651+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b780b15e0 con 0x7f0b781ed090 2023-10-14T18:14:43.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.652+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b781ec850 con 0x7f0b781ed090 2023-10-14T18:14:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.653+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0b7001e410 con 0x7f0b781ed090 2023-10-14T18:14:43.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.654+0000 7f0b6dffb640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 0x7f0b5c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.654+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(206..206 src has 1..206) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f0b70089a20 con 0x7f0b781ed090 2023-10-14T18:14:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.655+0000 7f0b7d319640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 0x7f0b5c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.655+0000 7f0b7d319640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 0x7f0b5c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b74034000 tx=0x7f0b74007000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:43.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.659+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b70054a60 con 0x7f0b781ed090 2023-10-14T18:14:43.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:43.854+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f0b7804e160 con 0x7f0b781ed090 2023-10-14T18:14:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.498+0000 7f0b6dffb640 1 -- 172.21.15.29:0/3330554405 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v207) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0b70058080 con 0x7f0b781ed090 2023-10-14T18:14:44.499 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T18:14:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.501+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 msgr2=0x7f0b5c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.501+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 0x7f0b5c06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f0b74034000 tx=0x7f0b74007000 comp rx=0 tx=0).stop 2023-10-14T18:14:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.501+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 msgr2=0x7f0b7814e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.501+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 secure :-1 s=READY pgs=2363 cs=0 l=1 rev1=1 crypto rx=0x7f0b780b13d0 tx=0x7f0b7000b480 comp rx=0 tx=0).stop 2023-10-14T18:14:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 shutdown_connections 2023-10-14T18:14:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b781ed090 0x7f0b7814e160 unknown :-1 s=CLOSED pgs=2363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b780c2ec0 0x7f0b7814dc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5c06af90 0x7f0b5c06d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 --2- 172.21.15.29:0/3330554405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b640039e0 0x7f0b781ecb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 >> 172.21.15.29:0/3330554405 conn(0x7f0b780cb360 msgr2=0x7f0b7806afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 shutdown_connections 2023-10-14T18:14:44.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.502+0000 7f0b7f57d640 1 -- 172.21.15.29:0/3330554405 wait complete. 2023-10-14T18:14:44.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:14:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.747+0000 7fedd9da0640 1 Processor -- start 2023-10-14T18:14:44.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.747+0000 7fedd9da0640 1 -- start start 2023-10-14T18:14:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd9da0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fedd40b1690 0x7fedd40b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd9da0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40b4f00 con 0x7fedd415cb50 2023-10-14T18:14:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd9da0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd40cdbf0 con 0x7fedd40b4ab0 2023-10-14T18:14:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd9da0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd40cde90 con 0x7fedd40b1690 2023-10-14T18:14:44.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd8d9e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fedd415cb50 0x7fedd4161180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38892/0 (socket says 172.21.15.29:38892) 2023-10-14T18:14:44.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.749+0000 7fedd8d9e640 1 -- 172.21.15.29:0/793884572 learned_addr learned my addr 172.21.15.29:0/793884572 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcbfff640 1 --2- 172.21.15.29:0/793884572 >> v2:172.21.15.186:3300/0 conn(0x7fedd40b1690 0x7fedd40b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2050874167 0 0) 0x7fedd0001110 con 0x7fedd415cb50 2023-10-14T18:14:44.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedbc003520 con 0x7fedd415cb50 2023-10-14T18:14:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1887582325 0 0) 0x7fedbc003520 con 0x7fedd415cb50 2023-10-14T18:14:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 >> v2:172.21.15.186:3300/0 conn(0x7fedd40b1690 msgr2=0x7fedd40b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 --2- 172.21.15.29:0/793884572 >> v2:172.21.15.186:3300/0 conn(0x7fedd40b1690 0x7fedd40b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 >> v1:172.21.15.186:6789/0 conn(0x7fedd40b4ab0 legacy=0x7fedd40d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40ce170 con 0x7fedd415cb50 2023-10-14T18:14:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fedd0001800 con 0x7fedd415cb50 2023-10-14T18:14:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 >> v1:172.21.15.29:6789/0 conn(0x7fedd415cb50 legacy=0x7fedd4161180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc0047c0 0x7fedbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedbc0052a0 0x7fedbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40cde90 con 0x7fedbc003970 2023-10-14T18:14:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc00c170 con 0x7fedbc0052a0 2023-10-14T18:14:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.750+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedbc00c330 con 0x7fedbc0047c0 2023-10-14T18:14:44.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedcbfff640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc0047c0 0x7fedbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd959f640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedbc0052a0 0x7fedbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 -- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc0047c0 msgr2=0x7fedbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc0047c0 0x7fedbc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 -- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedbc0052a0 msgr2=0x7fedbc00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:44.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedbc0052a0 0x7fedbc00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd8d9e640 1 -- 172.21.15.29:0/793884572 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedbc003520 con 0x7fedbc003970 2023-10-14T18:14:44.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd959f640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedbc0052a0 0x7fedbc00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:44.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.751+0000 7fedd959f640 1 -- 172.21.15.29:0/793884572 reap_dead start 2023-10-14T18:14:44.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.752+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 0x7fedbc003d90 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fedd000c210 tx=0x7fedbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9e2a827779441e7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:44.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.752+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd0005ad0 con 0x7fedbc003970 2023-10-14T18:14:44.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.752+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd0005c50 con 0x7fedbc003970 2023-10-14T18:14:44.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.752+0000 7fedcb7fe640 1 -- 172.21.15.29:0/793884572 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedd002c890 con 0x7fedbc003970 2023-10-14T18:14:44.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 -- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 msgr2=0x7fedbc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 0x7fedbc003d90 secure :-1 s=READY pgs=2365 cs=0 l=1 rev1=1 crypto rx=0x7fedd000c210 tx=0x7fedbc0057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 -- 172.21.15.29:0/793884572 shutdown_connections 2023-10-14T18:14:44.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 --2- 172.21.15.29:0/793884572 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedbc003970 0x7fedbc003d90 unknown :-1 s=CLOSED pgs=2365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 -- 172.21.15.29:0/793884572 >> 172.21.15.29:0/793884572 conn(0x7fedd40cf570 msgr2=0x7fedd40cf980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:44.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 -- 172.21.15.29:0/793884572 shutdown_connections 2023-10-14T18:14:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.753+0000 7fedd9da0640 1 -- 172.21.15.29:0/793884572 wait complete. 2023-10-14T18:14:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.755+0000 7fedd9da0640 1 Processor -- start 2023-10-14T18:14:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.755+0000 7fedd9da0640 1 -- start start 2023-10-14T18:14:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc003970 0x7fedd40c9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd41e6e10 0x7fedd41e7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40c97c0 con 0x7fedd40c82e0 2023-10-14T18:14:44.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40ca150 con 0x7fedd41e6e10 2023-10-14T18:14:44.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd9da0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd40ca3f0 con 0x7fedbc003970 2023-10-14T18:14:44.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.756+0000 7fedd959f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50124/0 (socket says 172.21.15.29:50124) 2023-10-14T18:14:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 -- 172.21.15.29:0/72369004 learned_addr learned my addr 172.21.15.29:0/72369004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc003970 0x7fedd40c9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedcbfff640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd41e6e10 0x7fedd41e7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 -- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc003970 msgr2=0x7fedd40c9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc003970 0x7fedd40c9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 -- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd41e6e10 msgr2=0x7fedd41e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd41e6e10 0x7fedd41e7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 -- 172.21.15.29:0/72369004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd40ca6d0 con 0x7fedd40c82e0 2023-10-14T18:14:44.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.757+0000 7fedd959f640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fedd4066a40 tx=0x7fedcc00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.758+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc003bc0 con 0x7fedd40c82e0 2023-10-14T18:14:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.758+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc003d40 con 0x7fedd40c82e0 2023-10-14T18:14:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.758+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedcc0052b0 con 0x7fedd40c82e0 2023-10-14T18:14:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.758+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd40ca9b0 con 0x7fedd40c82e0 2023-10-14T18:14:44.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.758+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd41e77d0 con 0x7fedd40c82e0 2023-10-14T18:14:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.759+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd404e9b0 con 0x7fedd40c82e0 2023-10-14T18:14:44.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.761+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fedcc005410 con 0x7fedd40c82e0 2023-10-14T18:14:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.762+0000 7fedc97fa640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 0x7fedc006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.762+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(207..207 src has 1..207) v4 ==== 5343+0+0 (secure 0 0 0) 0x7fedcc0894b0 con 0x7fedd40c82e0 2023-10-14T18:14:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.762+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 0x7fedc006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:44.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.763+0000 7fedd8d9e640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 0x7fedc006d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fedd0034000 tx=0x7fedd0009030 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:44.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.766+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fedcc054490 con 0x7fedd40c82e0 2023-10-14T18:14:44.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:44.964+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fedd40d3860 con 0x7fedd40c82e0 2023-10-14T18:14:45.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.518+0000 7fedc97fa640 1 -- 172.21.15.29:0/72369004 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v208) v1 ==== 170+0+0 (secure 0 0 0) 0x7fedcc0180d0 con 0x7fedd40c82e0 2023-10-14T18:14:45.520 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T18:14:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 msgr2=0x7fedc006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 0x7fedc006d480 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fedd0034000 tx=0x7fedd0009030 comp rx=0 tx=0).stop 2023-10-14T18:14:45.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 msgr2=0x7fedd40c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 secure :-1 s=READY pgs=2366 cs=0 l=1 rev1=1 crypto rx=0x7fedd4066a40 tx=0x7fedcc00dc00 comp rx=0 tx=0).stop 2023-10-14T18:14:45.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 shutdown_connections 2023-10-14T18:14:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fedd40c82e0 0x7fedd40c8760 unknown :-1 s=CLOSED pgs=2366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fedd41e6e10 0x7fedd41e7290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.522+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fedc006afc0 0x7fedc006d480 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.523+0000 7fedd9da0640 1 --2- 172.21.15.29:0/72369004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fedbc003970 0x7fedd40c9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.523+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 >> 172.21.15.29:0/72369004 conn(0x7fedd40cf570 msgr2=0x7fedd40d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.523+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 shutdown_connections 2023-10-14T18:14:45.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.523+0000 7fedd9da0640 1 -- 172.21.15.29:0/72369004 wait complete. 2023-10-14T18:14:45.559 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T18:14:45.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.751+0000 7f1a1f573640 1 Processor -- start 2023-10-14T18:14:45.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.751+0000 7f1a1f573640 1 -- start start 2023-10-14T18:14:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1f573640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a180ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1f573640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a18070360 con 0x7f1a18159ab0 2023-10-14T18:14:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1f573640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a180cd1f0 con 0x7f1a1806c140 2023-10-14T18:14:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1f573640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1815d890 con 0x7f1a18159e90 2023-10-14T18:14:45.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1d30f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1a18159ab0 0x7f1a1806fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38900/0 (socket says 172.21.15.29:38900) 2023-10-14T18:14:45.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1d30f640 1 -- 172.21.15.29:0/1020496070 learned_addr learned my addr 172.21.15.29:0/1020496070 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.753+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a180ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879908101 0 0) 0x7f1a18070360 con 0x7f1a18159ab0 2023-10-14T18:14:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a04005730 con 0x7f1a18159ab0 2023-10-14T18:14:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 404970517 0 0) 0x7f1a04005730 con 0x7f1a18159ab0 2023-10-14T18:14:45.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 msgr2=0x7f1a180ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a180ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 >> v1:172.21.15.186:6789/0 conn(0x7f1a1806c140 legacy=0x7f1a1815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180cd4d0 con 0x7f1a18159ab0 2023-10-14T18:14:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1a14001670 con 0x7f1a18159ab0 2023-10-14T18:14:45.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 >> v1:172.21.15.29:6789/0 conn(0x7f1a18159ab0 legacy=0x7f1a1806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a04005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a040069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a04006f10 0x7f1a04007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a1815d890 con 0x7f1a04006130 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0400ddd0 con 0x7f1a04006f10 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.754+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a0400df90 con 0x7f1a18159e90 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a040069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1d30f640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a04005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1db10640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a04006f10 0x7f1a04007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 -- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 msgr2=0x7f1a04005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:45.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a18159e90 0x7f1a04005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 -- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a04006f10 msgr2=0x7f1a04007390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:45.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a04006f10 0x7f1a04007390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1cb0e640 1 -- 172.21.15.29:0/1020496070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a04005730 con 0x7f1a04006130 2023-10-14T18:14:45.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.755+0000 7f1a1db10640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a04006f10 0x7f1a04007390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:45.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.756+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a040069d0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009ac0 tx=0x7f1a04007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d4784f3efbcc80 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:45.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0802c6b0 con 0x7f1a04006130 2023-10-14T18:14:45.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a0802c810 con 0x7f1a04006130 2023-10-14T18:14:45.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a0ffff640 1 -- 172.21.15.29:0/1020496070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a0802cba0 con 0x7f1a04006130 2023-10-14T18:14:45.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a1f573640 1 -- 172.21.15.29:0/1020496070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 msgr2=0x7f1a040069d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a1f573640 1 --2- 172.21.15.29:0/1020496070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a040069d0 secure :-1 s=READY pgs=2368 cs=0 l=1 rev1=1 crypto rx=0x7f1a08009ac0 tx=0x7f1a04007a00 comp rx=0 tx=0).stop 2023-10-14T18:14:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a1db10640 1 -- 172.21.15.29:0/1020496070 reap_dead start 2023-10-14T18:14:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a1f573640 1 -- 172.21.15.29:0/1020496070 shutdown_connections 2023-10-14T18:14:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.757+0000 7f1a1f573640 1 -- 172.21.15.29:0/1020496070 >> 172.21.15.29:0/1020496070 conn(0x7f1a180ce9b0 msgr2=0x7f1a180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.758+0000 7f1a1f573640 1 -- 172.21.15.29:0/1020496070 shutdown_connections 2023-10-14T18:14:45.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.758+0000 7f1a1f573640 1 -- 172.21.15.29:0/1020496070 wait complete. 2023-10-14T18:14:45.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.759+0000 7f1a1f573640 1 Processor -- start 2023-10-14T18:14:45.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.760+0000 7f1a1f573640 1 -- start start 2023-10-14T18:14:45.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.760+0000 7f1a1f573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1f573640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a180c4c80 0x7f1a18069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1f573640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a180c50e0 0x7f1a1806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1f573640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180c24e0 con 0x7f1a04006130 2023-10-14T18:14:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1f573640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180c3340 con 0x7f1a180c4c80 2023-10-14T18:14:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1f573640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a180c37d0 con 0x7f1a180c50e0 2023-10-14T18:14:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50130/0 (socket says 172.21.15.29:50130) 2023-10-14T18:14:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 -- 172.21.15.29:0/461842044 learned_addr learned my addr 172.21.15.29:0/461842044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 -- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a180c50e0 msgr2=0x7f1a1806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a180c50e0 0x7f1a1806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 -- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a180c4c80 msgr2=0x7f1a18069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:45.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a180c4c80 0x7f1a18069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.761+0000 7f1a1d30f640 1 -- 172.21.15.29:0/461842044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a180c2730 con 0x7f1a04006130 2023-10-14T18:14:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a1d30f640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1a14001200 tx=0x7f1a1400ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400ee60 con 0x7f1a04006130 2023-10-14T18:14:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a1400c8d0 con 0x7f1a04006130 2023-10-14T18:14:45.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a180c2a10 con 0x7f1a04006130 2023-10-14T18:14:45.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a1400cc80 con 0x7f1a04006130 2023-10-14T18:14:45.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.762+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a180c2fd0 con 0x7f1a04006130 2023-10-14T18:14:45.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.763+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a180c7940 con 0x7f1a04006130 2023-10-14T18:14:45.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.765+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1a1401d460 con 0x7f1a04006130 2023-10-14T18:14:45.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.766+0000 7f1a0dffb640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:45.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.766+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(208..208 src has 1..208) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f1a1408a080 con 0x7f1a04006130 2023-10-14T18:14:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.766+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.767+0000 7f1a1cb0e640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 0x7f19fc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a08036000 tx=0x7f1a0802e000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:45.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:45.770+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1a14055180 con 0x7f1a04006130 2023-10-14T18:14:45.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:14:45.823 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:14:46.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.006+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f1a1804e160 con 0x7f1a04006130 2023-10-14T18:14:46.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.523+0000 7f1a0dffb640 1 -- 172.21.15.29:0/461842044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 25 v95) v1 ==== 174+0+0 (secure 0 0 0) 0x7f1a140587a0 con 0x7f1a04006130 2023-10-14T18:14:46.525 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 24 and data pool 25 2023-10-14T18:14:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 msgr2=0x7f19fc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 0x7f19fc06d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f1a08036000 tx=0x7f1a0802e000 comp rx=0 tx=0).stop 2023-10-14T18:14:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 msgr2=0x7f1a1814d9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 secure :-1 s=READY pgs=2369 cs=0 l=1 rev1=1 crypto rx=0x7f1a14001200 tx=0x7f1a1400ec10 comp rx=0 tx=0).stop 2023-10-14T18:14:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 shutdown_connections 2023-10-14T18:14:46.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f19fc06af90 0x7f19fc06d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1a180c50e0 0x7f1a1806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1a180c4c80 0x7f1a18069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.527+0000 7f1a1f573640 1 --2- 172.21.15.29:0/461842044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1a04006130 0x7f1a1814d9f0 unknown :-1 s=CLOSED pgs=2369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.528+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 >> 172.21.15.29:0/461842044 conn(0x7f1a180ce9b0 msgr2=0x7f1a18158de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:46.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.528+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 shutdown_connections 2023-10-14T18:14:46.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.528+0000 7f1a1f573640 1 -- 172.21.15.29:0/461842044 wait complete. 2023-10-14T18:14:46.564 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:46.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.757+0000 7f480b11d640 1 Processor -- start 2023-10-14T18:14:46.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.758+0000 7f480b11d640 1 -- start start 2023-10-14T18:14:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f480b11d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f48040b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f480b11d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4804071450 con 0x7f48040adb50 2023-10-14T18:14:46.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f480b11d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48040cc600 con 0x7f48040adf30 2023-10-14T18:14:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f480b11d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48040cc8e0 con 0x7f4804070fd0 2023-10-14T18:14:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f4808eb9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f48040adb50 0x7f480406f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38908/0 (socket says 172.21.15.29:38908) 2023-10-14T18:14:46.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.759+0000 7f4808eb9640 1 -- 172.21.15.29:0/3013777784 learned_addr learned my addr 172.21.15.29:0/3013777784 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f4803fff640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f48040b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385557643 0 0) 0x7f4804071450 con 0x7f48040adb50 2023-10-14T18:14:46.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47f4003520 con 0x7f48040adb50 2023-10-14T18:14:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 379887716 0 0) 0x7f47f4003520 con 0x7f48040adb50 2023-10-14T18:14:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 msgr2=0x7f48040b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f48040b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 >> v1:172.21.15.186:6789/0 conn(0x7f48040adf30 legacy=0x7f4804164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040ccbc0 con 0x7f48040adb50 2023-10-14T18:14:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f47f8001670 con 0x7f48040adb50 2023-10-14T18:14:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 >> v1:172.21.15.29:6789/0 conn(0x7f48040adb50 legacy=0x7f480406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.760+0000 7f48037fe640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f47f40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48037fe640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f4003f20 0x7f47f40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48037fe640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4004d00 0x7f47f4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48040cc8e0 con 0x7f47f4003f20 2023-10-14T18:14:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f400ba60 con 0x7f47f4004d00 2023-10-14T18:14:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47f400bc20 con 0x7f4804070fd0 2023-10-14T18:14:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f4003f20 0x7f47f40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4808eb9640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f47f40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48096ba640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4004d00 0x7f47f4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 -- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 msgr2=0x7f47f40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4804070fd0 0x7f47f40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 -- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4004d00 msgr2=0x7f47f4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:46.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4004d00 0x7f47f4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f4803fff640 1 -- 172.21.15.29:0/3013777784 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47f4003520 con 0x7f47f4003f20 2023-10-14T18:14:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.761+0000 7f48096ba640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4004d00 0x7f47f4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:46.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.762+0000 7f4803fff640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f4003f20 0x7f47f40047c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009ac0 tx=0x7f47f40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0094091debac616 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f002c6b0 con 0x7f47f4003f20 2023-10-14T18:14:46.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f002c810 con 0x7f47f4003f20 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f48037fe640 1 -- 172.21.15.29:0/3013777784 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f002cba0 con 0x7f47f4003f20 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f480b11d640 1 -- 172.21.15.29:0/3013777784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f4003f20 msgr2=0x7f47f40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f480b11d640 1 --2- 172.21.15.29:0/3013777784 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47f4003f20 0x7f47f40047c0 secure :-1 s=READY pgs=2371 cs=0 l=1 rev1=1 crypto rx=0x7f47f0009ac0 tx=0x7f47f40057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f48096ba640 1 -- 172.21.15.29:0/3013777784 reap_dead start 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f480b11d640 1 -- 172.21.15.29:0/3013777784 shutdown_connections 2023-10-14T18:14:46.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f480b11d640 1 -- 172.21.15.29:0/3013777784 >> 172.21.15.29:0/3013777784 conn(0x7f48040cb410 msgr2=0x7f48040cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.763+0000 7f480b11d640 1 -- 172.21.15.29:0/3013777784 shutdown_connections 2023-10-14T18:14:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.764+0000 7f480b11d640 1 -- 172.21.15.29:0/3013777784 wait complete. 2023-10-14T18:14:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.765+0000 7f480b11d640 1 Processor -- start 2023-10-14T18:14:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.765+0000 7f480b11d640 1 -- start start 2023-10-14T18:14:46.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4003f20 0x7f48040bf220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48040c9050 0x7f48041624a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041f71c0 con 0x7f48041f75e0 2023-10-14T18:14:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48041f6b90 con 0x7f47f4003f20 2023-10-14T18:14:46.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.766+0000 7f480b11d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48040c1b60 con 0x7f48040c9050 2023-10-14T18:14:46.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50154/0 (socket says 172.21.15.29:50154) 2023-10-14T18:14:46.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 -- 172.21.15.29:0/1033223044 learned_addr learned my addr 172.21.15.29:0/1033223044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:46.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4808eb9640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4003f20 0x7f48040bf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f48096ba640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48040c9050 0x7f48041624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 -- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48040c9050 msgr2=0x7f48041624a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48040c9050 0x7f48041624a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 -- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4003f20 msgr2=0x7f48040bf220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4003f20 0x7f48040bf220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.767+0000 7f4803fff640 1 -- 172.21.15.29:0/1033223044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48040bf890 con 0x7f48041f75e0 2023-10-14T18:14:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f4803fff640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f47f002c250 tx=0x7f47f0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:46.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0003810 con 0x7f48041f75e0 2023-10-14T18:14:46.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47f0003970 con 0x7f48041f75e0 2023-10-14T18:14:46.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47f0003ce0 con 0x7f48041f75e0 2023-10-14T18:14:46.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48040bfb70 con 0x7f48041f75e0 2023-10-14T18:14:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.768+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48041f7a60 con 0x7f48041f75e0 2023-10-14T18:14:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.769+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f480404e160 con 0x7f48041f75e0 2023-10-14T18:14:46.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.770+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f47f0047450 con 0x7f48041f75e0 2023-10-14T18:14:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.771+0000 7f48017fa640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 0x7f47e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:46.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.771+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f47f00abb20 con 0x7f48041f75e0 2023-10-14T18:14:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.772+0000 7f4808eb9640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 0x7f47e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.772+0000 7f4808eb9640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 0x7f47e806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f47f8001c90 tx=0x7f47f8009040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:46.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.776+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47f00ade60 con 0x7f48041f75e0 2023-10-14T18:14:46.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.975+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48040c5750 con 0x7f48041f75e0 2023-10-14T18:14:46.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.977+0000 7f48017fa640 1 -- 172.21.15.29:0/1033223044 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16766 (secure 0 0 0) 0x7f47f0076ae0 con 0x7f48041f75e0 2023-10-14T18:14:46.978 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:46.978 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":209,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:46.007895+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:46.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 msgr2=0x7f47e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 0x7f47e806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f47f8001c90 tx=0x7f47f8009040 comp rx=0 tx=0).stop 2023-10-14T18:14:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 msgr2=0x7f48040c1620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 secure :-1 s=READY pgs=2372 cs=0 l=1 rev1=1 crypto rx=0x7f47f002c250 tx=0x7f47f0004730 comp rx=0 tx=0).stop 2023-10-14T18:14:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 shutdown_connections 2023-10-14T18:14:46.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f47e806af90 0x7f47e806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48040c9050 0x7f48041624a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48041f75e0 0x7f48040c1620 unknown :-1 s=CLOSED pgs=2372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.979+0000 7f480b11d640 1 --2- 172.21.15.29:0/1033223044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47f4003f20 0x7f48040bf220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.980+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 >> 172.21.15.29:0/1033223044 conn(0x7f48040cb410 msgr2=0x7f48040af1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.980+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 shutdown_connections 2023-10-14T18:14:46.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:46.980+0000 7f480b11d640 1 -- 172.21.15.29:0/1033223044 wait complete. 2023-10-14T18:14:46.999 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:47.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.211+0000 7f0f04fa0640 1 Processor -- start 2023-10-14T18:14:47.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.211+0000 7f0f04fa0640 1 -- start start 2023-10-14T18:14:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.212+0000 7f0f04fa0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0f000c9060 0x7f0f0006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.212+0000 7f0f04fa0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.212+0000 7f0f04fa0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f000ca560 con 0x7f0f000c9440 2023-10-14T18:14:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.212+0000 7f0f04fa0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f000c9950 con 0x7f0f000c9060 2023-10-14T18:14:47.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.212+0000 7f0f04fa0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f000c9c30 con 0x7f0f0006c9e0 2023-10-14T18:14:47.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.213+0000 7f0ef7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.213+0000 7f0ef7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:48282/0 (socket says 172.21.15.29:48282) 2023-10-14T18:14:47.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.213+0000 7f0ef7fff640 1 -- 172.21.15.29:0/3878540107 learned_addr learned my addr 172.21.15.29:0/3878540107 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:47.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.213+0000 7f0eff7fe640 1 --2- 172.21.15.29:0/3878540107 >> v2:172.21.15.186:3300/0 conn(0x7f0f000c9060 0x7f0f0006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.214+0000 7f0ef7fff640 1 -- 172.21.15.29:0/3878540107 >> v2:172.21.15.186:3300/0 conn(0x7f0f000c9060 msgr2=0x7f0f0006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:47.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.214+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/3878540107 >> v2:172.21.15.186:3300/0 conn(0x7f0f000c9060 0x7f0f0006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.214+0000 7f0ef7fff640 1 -- 172.21.15.29:0/3878540107 >> v1:172.21.15.186:6789/0 conn(0x7f0f000c9440 legacy=0x7f0f00164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.214+0000 7f0ef7fff640 1 -- 172.21.15.29:0/3878540107 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f000c9f10 con 0x7f0f0006c9e0 2023-10-14T18:14:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.216+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/3878540107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009870 tx=0x7f0ee802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d01bbc525976b88 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.216+0000 7f0efeffd640 1 -- 172.21.15.29:0/3878540107 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8030840 con 0x7f0f0006c9e0 2023-10-14T18:14:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.216+0000 7f0efeffd640 1 -- 172.21.15.29:0/3878540107 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ee80309a0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.216+0000 7f0efeffd640 1 -- 172.21.15.29:0/3878540107 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ee8030d30 con 0x7f0f0006c9e0 2023-10-14T18:14:47.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 -- 172.21.15.29:0/3878540107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 msgr2=0x7f0f0006f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/3878540107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8009870 tx=0x7f0ee802b980 comp rx=0 tx=0).stop 2023-10-14T18:14:47.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 -- 172.21.15.29:0/3878540107 shutdown_connections 2023-10-14T18:14:47.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/3878540107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f0006c9e0 0x7f0f0006f980 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/3878540107 >> v2:172.21.15.186:3300/0 conn(0x7f0f000c9060 0x7f0f0006f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 -- 172.21.15.29:0/3878540107 >> 172.21.15.29:0/3878540107 conn(0x7f0f000ce6a0 msgr2=0x7f0f000ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:47.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 -- 172.21.15.29:0/3878540107 shutdown_connections 2023-10-14T18:14:47.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.217+0000 7f0f04fa0640 1 -- 172.21.15.29:0/3878540107 wait complete. 2023-10-14T18:14:47.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.219+0000 7f0f04fa0640 1 Processor -- start 2023-10-14T18:14:47.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.219+0000 7f0f04fa0640 1 -- start start 2023-10-14T18:14:47.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.220+0000 7f0f04fa0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0f04fa0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 0x7f0f0014a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0f04fa0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f000c9440 0x7f0f0014ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0f04fa0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00161d20 con 0x7f0f0006c9e0 2023-10-14T18:14:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0f04fa0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f00161510 con 0x7f0f000c9440 2023-10-14T18:14:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0f04fa0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f0014d700 con 0x7f0f000c9060 2023-10-14T18:14:47.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0eff7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50170/0 (socket says 172.21.15.29:50170) 2023-10-14T18:14:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0eff7fe640 1 -- 172.21.15.29:0/228510508 learned_addr learned my addr 172.21.15.29:0/228510508 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 0x7f0f0014a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0efffff640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f000c9440 0x7f0f0014ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0eff7fe640 1 -- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 msgr2=0x7f0f0014a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.221+0000 7f0eff7fe640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 0x7f0f0014a800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0eff7fe640 1 -- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f000c9440 msgr2=0x7f0f0014ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0eff7fe640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f000c9440 0x7f0f0014ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0eff7fe640 1 -- 172.21.15.29:0/228510508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ee80094d0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 0x7f0f0014a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:47.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0eff7fe640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000e9b0 tx=0x7f0ef000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef0009670 con 0x7f0f0006c9e0 2023-10-14T18:14:47.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ef000c9a0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f000c3a80 con 0x7f0f0006c9e0 2023-10-14T18:14:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ef000cd50 con 0x7f0f0006c9e0 2023-10-14T18:14:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.222+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f00161a40 con 0x7f0f0006c9e0 2023-10-14T18:14:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.223+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f0004e9b0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.225+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f0ef0013470 con 0x7f0f0006c9e0 2023-10-14T18:14:47.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.225+0000 7f0efcff9640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 0x7f0ee006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.225+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(209..209 src has 1..209) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0ef0089ad0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.226+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 0x7f0ee006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.226+0000 7f0ef7fff640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 0x7f0ee006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8036000 tx=0x7f0ee8008030 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.230+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0ef00549e0 con 0x7f0f0006c9e0 2023-10-14T18:14:47.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.417+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f000b4620 con 0x7f0f0006c9e0 2023-10-14T18:14:47.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.421+0000 7f0efcff9640 1 -- 172.21.15.29:0/228510508 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v209) v1 ==== 74+0+16766 (secure 0 0 0) 0x7f0ef0017090 con 0x7f0f0006c9e0 2023-10-14T18:14:47.422 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:47.422 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":209,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:46.007895+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:47.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 msgr2=0x7f0ee006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 0x7f0ee006d450 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0ee8036000 tx=0x7f0ee8008030 comp rx=0 tx=0).stop 2023-10-14T18:14:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 msgr2=0x7f0f000c4220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 secure :-1 s=READY pgs=2373 cs=0 l=1 rev1=1 crypto rx=0x7f0ef000e9b0 tx=0x7f0ef000ee70 comp rx=0 tx=0).stop 2023-10-14T18:14:47.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 shutdown_connections 2023-10-14T18:14:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0f000c9440 0x7f0f0014ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0ee006af90 0x7f0ee006d450 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0f000c9060 0x7f0f0014a800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 --2- 172.21.15.29:0/228510508 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0f0006c9e0 0x7f0f000c4220 unknown :-1 s=CLOSED pgs=2373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 >> 172.21.15.29:0/228510508 conn(0x7f0f000ce6a0 msgr2=0x7f0f0006af50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 shutdown_connections 2023-10-14T18:14:47.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.423+0000 7f0f04fa0640 1 -- 172.21.15.29:0/228510508 wait complete. 2023-10-14T18:14:47.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T18:14:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.673+0000 7f55df521640 1 Processor -- start 2023-10-14T18:14:47.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.673+0000 7f55df521640 1 -- start start 2023-10-14T18:14:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55df521640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55d815cd70 0x7f55d80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55df521640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55d80cded0 0x7f55d80b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55df521640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55d815d150 con 0x7f55d80b8480 2023-10-14T18:14:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55df521640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d80d1180 con 0x7f55d815cd70 2023-10-14T18:14:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55df521640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55d80d1460 con 0x7f55d80cded0 2023-10-14T18:14:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55de51f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55d815cd70 0x7f55d80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55de51f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f55d815cd70 0x7f55d80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50184/0 (socket says 172.21.15.29:50184) 2023-10-14T18:14:47.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55de51f640 1 -- 172.21.15.29:0/2360662646 learned_addr learned my addr 172.21.15.29:0/2360662646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.675+0000 7f55ddd1e640 1 --1- 172.21.15.29:0/2360662646 >> v1:172.21.15.29:6789/0 conn(0x7f55d80b8480 0x7f55d80cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38910/0 (socket says 172.21.15.29:38910) 2023-10-14T18:14:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55ded20640 1 --2- 172.21.15.29:0/2360662646 >> v2:172.21.15.186:3300/0 conn(0x7f55d80cded0 0x7f55d80b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55de51f640 1 -- 172.21.15.29:0/2360662646 >> v2:172.21.15.186:3300/0 conn(0x7f55d80cded0 msgr2=0x7f55d80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55de51f640 1 --2- 172.21.15.29:0/2360662646 >> v2:172.21.15.186:3300/0 conn(0x7f55d80cded0 0x7f55d80b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55de51f640 1 -- 172.21.15.29:0/2360662646 >> v1:172.21.15.29:6789/0 conn(0x7f55d80b8480 legacy=0x7f55d80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.676+0000 7f55de51f640 1 -- 172.21.15.29:0/2360662646 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d80d1740 con 0x7f55d815cd70 2023-10-14T18:14:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.677+0000 7f55de51f640 1 --2- 172.21.15.29:0/2360662646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d815cd70 0x7f55d80d2220 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f55d80b8910 tx=0x7f55d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdc8066a571dbddb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.677+0000 7f55dd51d640 1 -- 172.21.15.29:0/2360662646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4030860 con 0x7f55d815cd70 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.677+0000 7f55dd51d640 1 -- 172.21.15.29:0/2360662646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d40309c0 con 0x7f55d815cd70 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.677+0000 7f55dd51d640 1 -- 172.21.15.29:0/2360662646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4030d50 con 0x7f55d815cd70 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 -- 172.21.15.29:0/2360662646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d815cd70 msgr2=0x7f55d80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 --2- 172.21.15.29:0/2360662646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d815cd70 0x7f55d80d2220 secure :-1 s=READY pgs=2375 cs=0 l=1 rev1=1 crypto rx=0x7f55d80b8910 tx=0x7f55d402b9a0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 -- 172.21.15.29:0/2360662646 shutdown_connections 2023-10-14T18:14:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 --2- 172.21.15.29:0/2360662646 >> v2:172.21.15.186:3300/0 conn(0x7f55d80cded0 0x7f55d80b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 --2- 172.21.15.29:0/2360662646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d815cd70 0x7f55d80d2220 unknown :-1 s=CLOSED pgs=2375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 -- 172.21.15.29:0/2360662646 >> 172.21.15.29:0/2360662646 conn(0x7f55d80cf570 msgr2=0x7f55d80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 -- 172.21.15.29:0/2360662646 shutdown_connections 2023-10-14T18:14:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.678+0000 7f55df521640 1 -- 172.21.15.29:0/2360662646 wait complete. 2023-10-14T18:14:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.680+0000 7f55df521640 1 Processor -- start 2023-10-14T18:14:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.680+0000 7f55df521640 1 -- start start 2023-10-14T18:14:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55d80cded0 0x7f55d80d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55d815cd70 0x7f55d80d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b25d0 con 0x7f55d80b8480 2023-10-14T18:14:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b28b0 con 0x7f55d815cd70 2023-10-14T18:14:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55df521640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55d80b2b90 con 0x7f55d80cded0 2023-10-14T18:14:47.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55de51f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55de51f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50196/0 (socket says 172.21.15.29:50196) 2023-10-14T18:14:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.681+0000 7f55de51f640 1 -- 172.21.15.29:0/3526492267 learned_addr learned my addr 172.21.15.29:0/3526492267 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55ddd1e640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55d80cded0 0x7f55d80d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 -- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55d80cded0 msgr2=0x7f55d80d2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55ded20640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55d815cd70 0x7f55d80d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55d80cded0 0x7f55d80d2a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 -- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55d815cd70 msgr2=0x7f55d80d2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55d815cd70 0x7f55d80d2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 -- 172.21.15.29:0/3526492267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55d40094d0 con 0x7f55d80b8480 2023-10-14T18:14:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55de51f640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f55d402b9e0 tx=0x7f55d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.682+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4039b60 con 0x7f55d80b8480 2023-10-14T18:14:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.683+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55d4039ce0 con 0x7f55d80b8480 2023-10-14T18:14:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.683+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55d4043650 con 0x7f55d80b8480 2023-10-14T18:14:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.683+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55d80b2e70 con 0x7f55d80b8480 2023-10-14T18:14:47.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.683+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55d814bbd0 con 0x7f55d80b8480 2023-10-14T18:14:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.684+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55d804e9b0 con 0x7f55d80b8480 2023-10-14T18:14:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.685+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f55d402f3b0 con 0x7f55d80b8480 2023-10-14T18:14:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.686+0000 7f55ceffd640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 0x7f55bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.686+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f55d40abbd0 con 0x7f55d80b8480 2023-10-14T18:14:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.686+0000 7f55ddd1e640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 0x7f55bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.687+0000 7f55ddd1e640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 0x7f55bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55c8007fd0 tx=0x7f55c8009040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.690+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55d40769d0 con 0x7f55d80b8480 2023-10-14T18:14:47.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:47.925+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f55d815c4a0 con 0x7f55d80b8480 2023-10-14T18:14:48.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.555+0000 7f55ceffd640 1 -- 172.21.15.29:0/3526492267 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v97) v1 ==== 118+0+0 (secure 0 0 0) 0x7f55d40480b0 con 0x7f55d80b8480 2023-10-14T18:14:48.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 msgr2=0x7f55bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 0x7f55bc06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f55c8007fd0 tx=0x7f55c8009040 comp rx=0 tx=0).stop 2023-10-14T18:14:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 msgr2=0x7f55d80b1f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 secure :-1 s=READY pgs=2376 cs=0 l=1 rev1=1 crypto rx=0x7f55d402b9e0 tx=0x7f55d4004730 comp rx=0 tx=0).stop 2023-10-14T18:14:48.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 shutdown_connections 2023-10-14T18:14:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55d815cd70 0x7f55d80d2fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f55bc06af90 0x7f55bc06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55d80cded0 0x7f55d80d2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 --2- 172.21.15.29:0/3526492267 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55d80b8480 0x7f55d80b1f60 unknown :-1 s=CLOSED pgs=2376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.559+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 >> 172.21.15.29:0/3526492267 conn(0x7f55d80cf570 msgr2=0x7f55d80c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.560+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 shutdown_connections 2023-10-14T18:14:48.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.560+0000 7f55df521640 1 -- 172.21.15.29:0/3526492267 wait complete. 2023-10-14T18:14:48.581 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:14:48.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T18:14:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.811+0000 7f72e8bfb640 1 Processor -- start 2023-10-14T18:14:48.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.811+0000 7f72e8bfb640 1 -- start start 2023-10-14T18:14:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72e8bfb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72e40ad350 0x7f72e4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72e8bfb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f72e406a2e0 0x7f72e40c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72e8bfb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e406a730 con 0x7f72e415d150 2023-10-14T18:14:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72e8bfb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e415d530 con 0x7f72e40ad350 2023-10-14T18:14:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72e8bfb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72e40ac890 con 0x7f72e406a2e0 2023-10-14T18:14:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72e40ad350 0x7f72e4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:48.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.813+0000 7f72dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f72e40ad350 0x7f72e4069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50198/0 (socket says 172.21.15.29:50198) 2023-10-14T18:14:48.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.814+0000 7f72dbfff640 1 -- 172.21.15.29:0/1895368971 learned_addr learned my addr 172.21.15.29:0/1895368971 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.814+0000 7f72dbfff640 1 -- 172.21.15.29:0/1895368971 >> v2:172.21.15.186:3300/0 conn(0x7f72e406a2e0 msgr2=0x7f72e40c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.814+0000 7f72dbfff640 1 --2- 172.21.15.29:0/1895368971 >> v2:172.21.15.186:3300/0 conn(0x7f72e406a2e0 0x7f72e40c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.814+0000 7f72dbfff640 1 -- 172.21.15.29:0/1895368971 >> v1:172.21.15.186:6789/0 conn(0x7f72e415d150 legacy=0x7f72e4160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.814+0000 7f72dbfff640 1 -- 172.21.15.29:0/1895368971 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e40ad7a0 con 0x7f72e40ad350 2023-10-14T18:14:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.815+0000 7f72dbfff640 1 --2- 172.21.15.29:0/1895368971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e40ad350 0x7f72e4069da0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f72e40adde0 tx=0x7f72cc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ccdc05a9ce88a81 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.816+0000 7f72e1d9b640 1 -- 172.21.15.29:0/1895368971 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031870 con 0x7f72e40ad350 2023-10-14T18:14:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.816+0000 7f72e1d9b640 1 -- 172.21.15.29:0/1895368971 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72cc0319d0 con 0x7f72e40ad350 2023-10-14T18:14:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.816+0000 7f72e1d9b640 1 -- 172.21.15.29:0/1895368971 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72cc031d60 con 0x7f72e40ad350 2023-10-14T18:14:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.816+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1895368971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e40ad350 msgr2=0x7f72e4069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.816+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1895368971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e40ad350 0x7f72e4069da0 secure :-1 s=READY pgs=2377 cs=0 l=1 rev1=1 crypto rx=0x7f72e40adde0 tx=0x7f72cc02b910 comp rx=0 tx=0).stop 2023-10-14T18:14:48.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1895368971 shutdown_connections 2023-10-14T18:14:48.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1895368971 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e40ad350 0x7f72e4069da0 unknown :-1 s=CLOSED pgs=2377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1895368971 >> v2:172.21.15.186:3300/0 conn(0x7f72e406a2e0 0x7f72e40c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1895368971 >> 172.21.15.29:0/1895368971 conn(0x7f72e40ce9b0 msgr2=0x7f72e40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1895368971 shutdown_connections 2023-10-14T18:14:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.817+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1895368971 wait complete. 2023-10-14T18:14:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.819+0000 7f72e8bfb640 1 Processor -- start 2023-10-14T18:14:48.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.819+0000 7f72e8bfb640 1 -- start start 2023-10-14T18:14:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e406a2e0 0x7f72e415f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 0x7f72e4155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e415ee20 con 0x7f72e415d150 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72e415fd10 con 0x7f72e40ad350 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e8bfb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72e415e760 con 0x7f72e406a2e0 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50206/0 (socket says 172.21.15.29:50206) 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e2d9d640 1 -- 172.21.15.29:0/1570779611 learned_addr learned my addr 172.21.15.29:0/1570779611 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:48.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72e259c640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e406a2e0 0x7f72e415f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:48.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.820+0000 7f72dbfff640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 0x7f72e4155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:48.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 -- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e406a2e0 msgr2=0x7f72e415f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:48.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e406a2e0 0x7f72e415f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 -- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 msgr2=0x7f72e4155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:48.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 0x7f72e4155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:48.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 -- 172.21.15.29:0/1570779611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72cc0094d0 con 0x7f72e415d150 2023-10-14T18:14:48.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72dbfff640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 0x7f72e4155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:48.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e2d9d640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00b970 tx=0x7f72dc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:48.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc00cb40 con 0x7f72e415d150 2023-10-14T18:14:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72dc00ccc0 con 0x7f72e415d150 2023-10-14T18:14:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72dc014680 con 0x7f72e415d150 2023-10-14T18:14:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.821+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e415ea40 con 0x7f72e415d150 2023-10-14T18:14:48.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.822+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e41558b0 con 0x7f72e415d150 2023-10-14T18:14:48.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.823+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72e415ef60 con 0x7f72e415d150 2023-10-14T18:14:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.824+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f72dc01d3b0 con 0x7f72e415d150 2023-10-14T18:14:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.825+0000 7f72daffd640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 0x7f72c406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:48.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.825+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f72dc089720 con 0x7f72e415d150 2023-10-14T18:14:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.825+0000 7f72e259c640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 0x7f72c406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.826+0000 7f72e259c640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 0x7f72c406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f72d4006fd0 tx=0x7f72d4008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:48.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:48.829+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f72e415ef60 con 0x7f72e415d150 2023-10-14T18:14:49.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.074+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f72e415ef60 con 0x7f72e415d150 2023-10-14T18:14:49.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.576+0000 7f72daffd640 1 -- 172.21.15.29:0/1570779611 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v98) v1 ==== 122+0+0 (secure 0 0 0) 0x7f72e415ef60 con 0x7f72e415d150 2023-10-14T18:14:49.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.581+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 msgr2=0x7f72c406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.581+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 0x7f72c406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f72d4006fd0 tx=0x7f72d4008040 comp rx=0 tx=0).stop 2023-10-14T18:14:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.581+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 msgr2=0x7f72e4155370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.581+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 secure :-1 s=READY pgs=2378 cs=0 l=1 rev1=1 crypto rx=0x7f72dc00b970 tx=0x7f72dc00be30 comp rx=0 tx=0).stop 2023-10-14T18:14:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 shutdown_connections 2023-10-14T18:14:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f72e415d150 0x7f72e4155370 unknown :-1 s=CLOSED pgs=2378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f72e40ad350 0x7f72e4155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f72c406af90 0x7f72c406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 --2- 172.21.15.29:0/1570779611 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f72e406a2e0 0x7f72e415f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 >> 172.21.15.29:0/1570779611 conn(0x7f72e40ce9b0 msgr2=0x7f72e4158d50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:49.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 shutdown_connections 2023-10-14T18:14:49.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.582+0000 7f72e8bfb640 1 -- 172.21.15.29:0/1570779611 wait complete. 2023-10-14T18:14:49.608 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T18:14:49.609 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:49.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.837+0000 7feb46743640 1 Processor -- start 2023-10-14T18:14:49.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.837+0000 7feb46743640 1 -- start start 2023-10-14T18:14:49.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb46743640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006d520 0x7feb400b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb46743640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb4006fa20 0x7feb400ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb46743640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb4006d900 con 0x7feb40065cc0 2023-10-14T18:14:49.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb46743640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb4015b1d0 con 0x7feb4006fa20 2023-10-14T18:14:49.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb46743640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb4006feb0 con 0x7feb4006d520 2023-10-14T18:14:49.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.839+0000 7feb44f40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb4006fa20 0x7feb400ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:49.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feb4006fa20 0x7feb400ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50222/0 (socket says 172.21.15.29:50222) 2023-10-14T18:14:49.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 -- 172.21.15.29:0/3532471321 learned_addr learned my addr 172.21.15.29:0/3532471321 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:49.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 -- 172.21.15.29:0/3532471321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006d520 msgr2=0x7feb400b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:49.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 --2- 172.21.15.29:0/3532471321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006d520 0x7feb400b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 -- 172.21.15.29:0/3532471321 >> v1:172.21.15.186:6789/0 conn(0x7feb40065cc0 legacy=0x7feb40070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:49.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.840+0000 7feb44f40640 1 -- 172.21.15.29:0/3532471321 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb40070190 con 0x7feb4006fa20 2023-10-14T18:14:49.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.841+0000 7feb44f40640 1 --2- 172.21.15.29:0/3532471321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006fa20 0x7feb400ac750 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7feb30009870 tx=0x7feb3002b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5fde27d864fce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:49.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.841+0000 7feb37fff640 1 -- 172.21.15.29:0/3532471321 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30031740 con 0x7feb4006fa20 2023-10-14T18:14:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.841+0000 7feb37fff640 1 -- 172.21.15.29:0/3532471321 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb300318a0 con 0x7feb4006fa20 2023-10-14T18:14:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.841+0000 7feb37fff640 1 -- 172.21.15.29:0/3532471321 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30031c30 con 0x7feb4006fa20 2023-10-14T18:14:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 -- 172.21.15.29:0/3532471321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006fa20 msgr2=0x7feb400ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 --2- 172.21.15.29:0/3532471321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006fa20 0x7feb400ac750 secure :-1 s=READY pgs=2379 cs=0 l=1 rev1=1 crypto rx=0x7feb30009870 tx=0x7feb3002b7e0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 -- 172.21.15.29:0/3532471321 shutdown_connections 2023-10-14T18:14:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 --2- 172.21.15.29:0/3532471321 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006fa20 0x7feb400ac750 unknown :-1 s=CLOSED pgs=2379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 --2- 172.21.15.29:0/3532471321 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006d520 0x7feb400b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 -- 172.21.15.29:0/3532471321 >> 172.21.15.29:0/3532471321 conn(0x7feb400b0a80 msgr2=0x7feb400b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:49.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 -- 172.21.15.29:0/3532471321 shutdown_connections 2023-10-14T18:14:49.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.842+0000 7feb46743640 1 -- 172.21.15.29:0/3532471321 wait complete. 2023-10-14T18:14:49.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.844+0000 7feb46743640 1 Processor -- start 2023-10-14T18:14:49.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.844+0000 7feb46743640 1 -- start start 2023-10-14T18:14:49.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb40065cc0 0x7feb40099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 0x7feb400b4820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40099f50 con 0x7feb4006d520 2023-10-14T18:14:49.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb40158ec0 con 0x7feb40065cc0 2023-10-14T18:14:49.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb46743640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb400b4d60 con 0x7feb4006a780 2023-10-14T18:14:49.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.845+0000 7feb44f40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:49.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50236/0 (socket says 172.21.15.29:50236) 2023-10-14T18:14:49.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 -- 172.21.15.29:0/2823597007 learned_addr learned my addr 172.21.15.29:0/2823597007 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:49.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb45741640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb40065cc0 0x7feb40099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:49.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb45f42640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 0x7feb400b4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:49.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 -- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 msgr2=0x7feb400b4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:49.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 0x7feb400b4820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 -- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb40065cc0 msgr2=0x7feb40099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:49.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb40065cc0 0x7feb40099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:49.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 -- 172.21.15.29:0/2823597007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300094d0 con 0x7feb4006d520 2023-10-14T18:14:49.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb45f42640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 0x7feb400b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:49.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.846+0000 7feb44f40640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7feb30009870 tx=0x7feb300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:49.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.847+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb30033c70 con 0x7feb4006d520 2023-10-14T18:14:49.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.847+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb30033dd0 con 0x7feb4006d520 2023-10-14T18:14:49.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.847+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb3003d6b0 con 0x7feb4006d520 2023-10-14T18:14:49.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.847+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb4006ac00 con 0x7feb4006d520 2023-10-14T18:14:49.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.847+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb4006b480 con 0x7feb4006d520 2023-10-14T18:14:49.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.848+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb4004e9b0 con 0x7feb4006d520 2023-10-14T18:14:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.849+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7feb3003a040 con 0x7feb4006d520 2023-10-14T18:14:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.850+0000 7feb35ffb640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 0x7feb2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.850+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7feb300ab7a0 con 0x7feb4006d520 2023-10-14T18:14:49.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.850+0000 7feb45741640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 0x7feb2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:49.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.851+0000 7feb45741640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 0x7feb2406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009890 tx=0x7feb3c008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:49.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:49.855+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feb30076640 con 0x7feb4006d520 2023-10-14T18:14:50.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.089+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb40070440 con 0x7feb4006d520 2023-10-14T18:14:50.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.090+0000 7feb35ffb640 1 -- 172.21.15.29:0/2823597007 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7feb30079c60 con 0x7feb4006d520 2023-10-14T18:14:50.158 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:50.158 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:14:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:14:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 msgr2=0x7feb2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 0x7feb2406d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7feb3c009890 tx=0x7feb3c008040 comp rx=0 tx=0).stop 2023-10-14T18:14:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 msgr2=0x7feb40158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 secure :-1 s=READY pgs=2380 cs=0 l=1 rev1=1 crypto rx=0x7feb30009870 tx=0x7feb300046c0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 shutdown_connections 2023-10-14T18:14:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7feb2406af90 0x7feb2406d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.093+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feb4006a780 0x7feb400b4820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.094+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feb4006d520 0x7feb40158980 unknown :-1 s=CLOSED pgs=2380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.094+0000 7feb46743640 1 --2- 172.21.15.29:0/2823597007 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feb40065cc0 0x7feb40099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.094+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 >> 172.21.15.29:0/2823597007 conn(0x7feb400b0a80 msgr2=0x7feb400b02a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.094+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 shutdown_connections 2023-10-14T18:14:50.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.094+0000 7feb46743640 1 -- 172.21.15.29:0/2823597007 wait complete. 2023-10-14T18:14:50.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:14:50.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.397+0000 7f75d533c640 1 Processor -- start 2023-10-14T18:14:50.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.398+0000 7f75d533c640 1 -- start start 2023-10-14T18:14:50.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.399+0000 7f75d533c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75d015cbe0 0x7f75d00d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.399+0000 7f75d533c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00b4c80 0x7f75d00b5c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.399+0000 7f75d533c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d00d15a0 con 0x7f75d015cfc0 2023-10-14T18:14:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.399+0000 7f75d533c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75d00d1880 con 0x7f75d015cbe0 2023-10-14T18:14:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.399+0000 7f75d533c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d00d1b60 con 0x7f75d00b4c80 2023-10-14T18:14:50.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75d015cbe0 0x7f75d00d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f75d015cbe0 0x7f75d00d0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50242/0 (socket says 172.21.15.29:50242) 2023-10-14T18:14:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 -- 172.21.15.29:0/4056581361 learned_addr learned my addr 172.21.15.29:0/4056581361 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:50.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75d4b3b640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00b4c80 0x7f75d00b5c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 -- 172.21.15.29:0/4056581361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00b4c80 msgr2=0x7f75d00b5c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:50.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00b4c80 0x7f75d00b5c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 -- 172.21.15.29:0/4056581361 >> v1:172.21.15.29:6789/0 conn(0x7f75d015cfc0 legacy=0x7f75d00b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.400+0000 7f75cffff640 1 -- 172.21.15.29:0/4056581361 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d00d1e40 con 0x7f75d015cbe0 2023-10-14T18:14:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.401+0000 7f75cffff640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d00d0b50 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009870 tx=0x7f75c002ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a4de4da36a1bc5c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75cf7fe640 1 -- 172.21.15.29:0/4056581361 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c002f830 con 0x7f75d015cbe0 2023-10-14T18:14:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75cf7fe640 1 -- 172.21.15.29:0/4056581361 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75c002f990 con 0x7f75d015cbe0 2023-10-14T18:14:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75cf7fe640 1 -- 172.21.15.29:0/4056581361 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c002fd20 con 0x7f75d015cbe0 2023-10-14T18:14:50.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 -- 172.21.15.29:0/4056581361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 msgr2=0x7f75d00d0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d00d0b50 secure :-1 s=READY pgs=2382 cs=0 l=1 rev1=1 crypto rx=0x7f75c0009870 tx=0x7f75c002ba10 comp rx=0 tx=0).stop 2023-10-14T18:14:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 -- 172.21.15.29:0/4056581361 shutdown_connections 2023-10-14T18:14:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d00b4c80 0x7f75d00b5c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 --2- 172.21.15.29:0/4056581361 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d00d0b50 unknown :-1 s=CLOSED pgs=2382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.402+0000 7f75d533c640 1 -- 172.21.15.29:0/4056581361 >> 172.21.15.29:0/4056581361 conn(0x7f75d00cf570 msgr2=0x7f75d00cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:50.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.403+0000 7f75d533c640 1 -- 172.21.15.29:0/4056581361 shutdown_connections 2023-10-14T18:14:50.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.403+0000 7f75d533c640 1 -- 172.21.15.29:0/4056581361 wait complete. 2023-10-14T18:14:50.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.404+0000 7f75d533c640 1 Processor -- start 2023-10-14T18:14:50.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.405+0000 7f75d533c640 1 -- start start 2023-10-14T18:14:50.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d00b4c80 0x7f75d0155700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d015cfc0 0x7f75d00b7690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d01552e0 con 0x7f75d015cbe0 2023-10-14T18:14:50.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75d01527e0 con 0x7f75d00b4c80 2023-10-14T18:14:50.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d533c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75d0154450 con 0x7f75d015cfc0 2023-10-14T18:14:50.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50258/0 (socket says 172.21.15.29:50258) 2023-10-14T18:14:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75c7fff640 1 -- 172.21.15.29:0/427534452 learned_addr learned my addr 172.21.15.29:0/427534452 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75cffff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d00b4c80 0x7f75d0155700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.406+0000 7f75d4b3b640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d015cfc0 0x7f75d00b7690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 -- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d015cfc0 msgr2=0x7f75d00b7690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d015cfc0 0x7f75d00b7690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 -- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d00b4c80 msgr2=0x7f75d0155700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d00b4c80 0x7f75d0155700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 -- 172.21.15.29:0/427534452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75c00094d0 con 0x7f75d015cbe0 2023-10-14T18:14:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75c7fff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f75b800c3c0 tx=0x7f75b800c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b80052d0 con 0x7f75d015cbe0 2023-10-14T18:14:50.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75b8005450 con 0x7f75d015cbe0 2023-10-14T18:14:50.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d0154660 con 0x7f75d015cbe0 2023-10-14T18:14:50.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75b8003830 con 0x7f75d015cbe0 2023-10-14T18:14:50.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.407+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d0154c20 con 0x7f75d015cbe0 2023-10-14T18:14:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.409+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d0155420 con 0x7f75d015cbe0 2023-10-14T18:14:50.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.410+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f75b8003990 con 0x7f75d015cbe0 2023-10-14T18:14:50.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.411+0000 7f75cd7fa640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 0x7f75b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.411+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f75b8088db0 con 0x7f75d015cbe0 2023-10-14T18:14:50.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.411+0000 7f75cffff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 0x7f75b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.412+0000 7f75cffff640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 0x7f75b006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036000 tx=0x7f75c0008030 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.416+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75d0155420 con 0x7f75d015cbe0 2023-10-14T18:14:50.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.617+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75d0155420 con 0x7f75d015cbe0 2023-10-14T18:14:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.621+0000 7f75cd7fa640 1 -- 172.21.15.29:0/427534452 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v210) v1 ==== 74+0+16753 (secure 0 0 0) 0x7f75d0155420 con 0x7f75d015cbe0 2023-10-14T18:14:50.622 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:50.622 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":210,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:14:47.542960+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:14:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 msgr2=0x7f75b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 0x7f75b006d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f75c0036000 tx=0x7f75c0008030 comp rx=0 tx=0).stop 2023-10-14T18:14:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 msgr2=0x7f75d01522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 secure :-1 s=READY pgs=2383 cs=0 l=1 rev1=1 crypto rx=0x7f75b800c3c0 tx=0x7f75b800c880 comp rx=0 tx=0).stop 2023-10-14T18:14:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 shutdown_connections 2023-10-14T18:14:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f75b006af90 0x7f75b006d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75d015cfc0 0x7f75d00b7690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75d015cbe0 0x7f75d01522a0 unknown :-1 s=CLOSED pgs=2383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 --2- 172.21.15.29:0/427534452 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75d00b4c80 0x7f75d0155700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 >> 172.21.15.29:0/427534452 conn(0x7f75d00cf570 msgr2=0x7f75d00d2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:50.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 shutdown_connections 2023-10-14T18:14:50.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.624+0000 7f75d533c640 1 -- 172.21.15.29:0/427534452 wait complete. 2023-10-14T18:14:50.643 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:14:50.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:14:50.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.862+0000 7fdeaadb3640 1 Processor -- start 2023-10-14T18:14:50.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.863+0000 7fdeaadb3640 1 -- start start 2023-10-14T18:14:50.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdeaadb3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea406c040 0x7fdea406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdeaadb3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdea40ad970 0x7fdea415e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdeaadb3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdea406c960 con 0x7fdea415ebc0 2023-10-14T18:14:50.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdeaadb3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdea415b9d0 con 0x7fdea40ad970 2023-10-14T18:14:50.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdeaadb3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea415bcb0 con 0x7fdea406c040 2023-10-14T18:14:50.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdea95b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdea40ad970 0x7fdea415e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdea95b0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdea40ad970 0x7fdea415e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50272/0 (socket says 172.21.15.29:50272) 2023-10-14T18:14:50.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.864+0000 7fdea95b0640 1 -- 172.21.15.29:0/1225740181 learned_addr learned my addr 172.21.15.29:0/1225740181 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:50.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.865+0000 7fdea9db1640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea406c040 0x7fdea406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.865+0000 7fdea95b0640 1 -- 172.21.15.29:0/1225740181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea406c040 msgr2=0x7fdea406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.865+0000 7fdea95b0640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea406c040 0x7fdea406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.865+0000 7fdea95b0640 1 -- 172.21.15.29:0/1225740181 >> v1:172.21.15.29:6789/0 conn(0x7fdea415ebc0 legacy=0x7fdea40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.865+0000 7fdea95b0640 1 -- 172.21.15.29:0/1225740181 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea415bf90 con 0x7fdea40ad970 2023-10-14T18:14:50.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.866+0000 7fdea95b0640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea40ad970 0x7fdea415e680 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fde980098a0 tx=0x7fde9802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47bbfef5efd0e54a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.866+0000 7fdea8daf640 1 -- 172.21.15.29:0/1225740181 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98030840 con 0x7fdea40ad970 2023-10-14T18:14:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.866+0000 7fdea8daf640 1 -- 172.21.15.29:0/1225740181 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde980309a0 con 0x7fdea40ad970 2023-10-14T18:14:50.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.866+0000 7fdea8daf640 1 -- 172.21.15.29:0/1225740181 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde98030d30 con 0x7fdea40ad970 2023-10-14T18:14:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 -- 172.21.15.29:0/1225740181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea40ad970 msgr2=0x7fdea415e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea40ad970 0x7fdea415e680 secure :-1 s=READY pgs=2385 cs=0 l=1 rev1=1 crypto rx=0x7fde980098a0 tx=0x7fde9802b980 comp rx=0 tx=0).stop 2023-10-14T18:14:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 -- 172.21.15.29:0/1225740181 shutdown_connections 2023-10-14T18:14:50.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea40ad970 0x7fdea415e680 unknown :-1 s=CLOSED pgs=2385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/1225740181 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea406c040 0x7fdea406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 -- 172.21.15.29:0/1225740181 >> 172.21.15.29:0/1225740181 conn(0x7fdea40ce6a0 msgr2=0x7fdea40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:50.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 -- 172.21.15.29:0/1225740181 shutdown_connections 2023-10-14T18:14:50.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.867+0000 7fdeaadb3640 1 -- 172.21.15.29:0/1225740181 wait complete. 2023-10-14T18:14:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.869+0000 7fdeaadb3640 1 Processor -- start 2023-10-14T18:14:50.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.869+0000 7fdeaadb3640 1 -- start start 2023-10-14T18:14:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 0x7fdea40b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 0x7fdea415c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f2330 con 0x7fdea406c040 2023-10-14T18:14:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f25f0 con 0x7fdea415ebc0 2023-10-14T18:14:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdeaadb3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdea41f28d0 con 0x7fdea40ad970 2023-10-14T18:14:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdea9db1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdea9db1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50286/0 (socket says 172.21.15.29:50286) 2023-10-14T18:14:50.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.870+0000 7fdea9db1640 1 -- 172.21.15.29:0/2416106753 learned_addr learned my addr 172.21.15.29:0/2416106753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea95b0640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 0x7fdea40b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdeaa5b2640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 0x7fdea415c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 -- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 msgr2=0x7fdea40b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 0x7fdea40b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 -- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 msgr2=0x7fdea415c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 0x7fdea415c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 -- 172.21.15.29:0/2416106753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde980094d0 con 0x7fdea406c040 2023-10-14T18:14:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea95b0640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 0x7fdea40b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdeaa5b2640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 0x7fdea415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:50.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fdea9db1640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fdea000d5f0 tx=0x7fdea000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.871+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea00125f0 con 0x7fdea406c040 2023-10-14T18:14:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.872+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdea0012750 con 0x7fdea406c040 2023-10-14T18:14:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.872+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdea0012b00 con 0x7fdea406c040 2023-10-14T18:14:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.872+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdea41f2bb0 con 0x7fdea406c040 2023-10-14T18:14:50.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.872+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdea40b3a90 con 0x7fdea406c040 2023-10-14T18:14:50.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.873+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdea406b330 con 0x7fdea406c040 2023-10-14T18:14:50.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.874+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdea0012c60 con 0x7fdea406c040 2023-10-14T18:14:50.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.875+0000 7fde967fc640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 0x7fde8806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:50.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.875+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fdea0089cf0 con 0x7fdea406c040 2023-10-14T18:14:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.875+0000 7fdea95b0640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 0x7fde8806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.876+0000 7fdea95b0640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 0x7fde8806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde98036000 tx=0x7fde9803b040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:50.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:50.879+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdea0054c00 con 0x7fdea406c040 2023-10-14T18:14:50.964 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T18:14:50.963+0000 7fdcadeaa640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:14:51.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.079+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 --> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdea40cc550 con 0x7fde8806af90 2023-10-14T18:14:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.095+0000 7fde967fc640 1 -- 172.21.15.29:0/2416106753 <== mgr.10482 v2:172.21.15.29:6832/2339165088 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286657 (secure 0 0 0) 0x7fde98030be0 con 0x7fde8806af90 2023-10-14T18:14:51.099 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:51.102 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:14:51.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.102+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 msgr2=0x7fde8806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.102+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 0x7fde8806d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fde98036000 tx=0x7fde9803b040 comp rx=0 tx=0).stop 2023-10-14T18:14:51.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.102+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 msgr2=0x7fdea41537b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.102+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 secure :-1 s=READY pgs=2386 cs=0 l=1 rev1=1 crypto rx=0x7fdea000d5f0 tx=0x7fdea000dab0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 shutdown_connections 2023-10-14T18:14:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdea415ebc0 0x7fdea415c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fde8806af90 0x7fde8806d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdea40ad970 0x7fdea40b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 --2- 172.21.15.29:0/2416106753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdea406c040 0x7fdea41537b0 unknown :-1 s=CLOSED pgs=2386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 >> 172.21.15.29:0/2416106753 conn(0x7fdea40ce6a0 msgr2=0x7fdea4069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:51.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 shutdown_connections 2023-10-14T18:14:51.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.103+0000 7fdeaadb3640 1 -- 172.21.15.29:0/2416106753 wait complete. 2023-10-14T18:14:51.124 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-14T18:14:50.952143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":387740,"kb_used_data":21672,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754586980,"statfs":{"total":773094113280,"available":772697067520,"internally_reserved":0,"allocated":22192128,"data_stored":11524740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":374722104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001915"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572757+0000","last_change":"2023-10-14T18:14:46.570972+0000","last_active":"2023-10-14T18:14:46.572757+0000","last_peered":"2023-10-14T18:14:46.572757+0000","last_clean":"2023-10-14T18:14:46.572757+0000","last_became_active":"2023-10-14T18:14:46.570627+0000","last_became_peered":"2023-10-14T18:14:46.570627+0000","last_unstale":"2023-10-14T18:14:46.572757+0000","last_undegraded":"2023-10-14T18:14:46.572757+0000","last_fullsized":"2023-10-14T18:14:46.572757+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:15.964586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562978+0000","last_change":"2023-10-14T18:14:46.550819+0000","last_active":"2023-10-14T18:14:47.562978+0000","last_peered":"2023-10-14T18:14:47.562978+0000","last_clean":"2023-10-14T18:14:47.562978+0000","last_became_active":"2023-10-14T18:14:46.549960+0000","last_became_peered":"2023-10-14T18:14:46.549960+0000","last_unstale":"2023-10-14T18:14:47.562978+0000","last_undegraded":"2023-10-14T18:14:47.562978+0000","last_fullsized":"2023-10-14T18:14:47.562978+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568687+0000","last_change":"2023-10-14T18:14:46.562799+0000","last_active":"2023-10-14T18:14:46.568687+0000","last_peered":"2023-10-14T18:14:46.568687+0000","last_clean":"2023-10-14T18:14:46.568687+0000","last_became_active":"2023-10-14T18:14:46.562528+0000","last_became_peered":"2023-10-14T18:14:46.562528+0000","last_unstale":"2023-10-14T18:14:46.568687+0000","last_undegraded":"2023-10-14T18:14:46.568687+0000","last_fullsized":"2023-10-14T18:14:46.568687+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:29:23.117570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561807+0000","last_change":"2023-10-14T18:14:46.548946+0000","last_active":"2023-10-14T18:14:47.561807+0000","last_peered":"2023-10-14T18:14:47.561807+0000","last_clean":"2023-10-14T18:14:47.561807+0000","last_became_active":"2023-10-14T18:14:46.548619+0000","last_became_peered":"2023-10-14T18:14:46.548619+0000","last_unstale":"2023-10-14T18:14:47.561807+0000","last_undegraded":"2023-10-14T18:14:47.561807+0000","last_fullsized":"2023-10-14T18:14:47.561807+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568730+0000","last_change":"2023-10-14T18:14:46.566502+0000","last_active":"2023-10-14T18:14:46.568730+0000","last_peered":"2023-10-14T18:14:46.568730+0000","last_clean":"2023-10-14T18:14:46.568730+0000","last_became_active":"2023-10-14T18:14:46.563380+0000","last_became_peered":"2023-10-14T18:14:46.563380+0000","last_unstale":"2023-10-14T18:14:46.568730+0000","last_undegraded":"2023-10-14T18:14:46.568730+0000","last_fullsized":"2023-10-14T18:14:46.568730+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574190+0000","last_change":"2023-10-14T18:14:46.567070+0000","last_active":"2023-10-14T18:14:46.574190+0000","last_peered":"2023-10-14T18:14:46.574190+0000","last_clean":"2023-10-14T18:14:46.574190+0000","last_became_active":"2023-10-14T18:14:46.566790+0000","last_became_peered":"2023-10-14T18:14:46.566790+0000","last_unstale":"2023-10-14T18:14:46.574190+0000","last_undegraded":"2023-10-14T18:14:46.574190+0000","last_fullsized":"2023-10-14T18:14:46.574190+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:48.712073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.558923+0000","last_change":"2023-10-14T18:14:46.555506+0000","last_active":"2023-10-14T18:14:47.558923+0000","last_peered":"2023-10-14T18:14:47.558923+0000","last_clean":"2023-10-14T18:14:47.558923+0000","last_became_active":"2023-10-14T18:14:46.553933+0000","last_became_peered":"2023-10-14T18:14:46.553933+0000","last_unstale":"2023-10-14T18:14:47.558923+0000","last_undegraded":"2023-10-14T18:14:47.558923+0000","last_fullsized":"2023-10-14T18:14:47.558923+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:36.892049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.558184+0000","last_change":"2023-10-14T18:14:46.551246+0000","last_active":"2023-10-14T18:14:46.558184+0000","last_peered":"2023-10-14T18:14:46.558184+0000","last_clean":"2023-10-14T18:14:46.558184+0000","last_became_active":"2023-10-14T18:14:46.550781+0000","last_became_peered":"2023-10-14T18:14:46.550781+0000","last_unstale":"2023-10-14T18:14:46.558184+0000","last_undegraded":"2023-10-14T18:14:46.558184+0000","last_fullsized":"2023-10-14T18:14:46.558184+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:15.964586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568683+0000","last_change":"2023-10-14T18:14:46.561054+0000","last_active":"2023-10-14T18:14:46.568683+0000","last_peered":"2023-10-14T18:14:46.568683+0000","last_clean":"2023-10-14T18:14:46.568683+0000","last_became_active":"2023-10-14T18:14:46.560753+0000","last_became_peered":"2023-10-14T18:14:46.560753+0000","last_unstale":"2023-10-14T18:14:46.568683+0000","last_undegraded":"2023-10-14T18:14:46.568683+0000","last_fullsized":"2023-10-14T18:14:46.568683+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:32.135916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557196+0000","last_change":"2023-10-14T18:14:46.559226+0000","last_active":"2023-10-14T18:14:47.557196+0000","last_peered":"2023-10-14T18:14:47.557196+0000","last_clean":"2023-10-14T18:14:47.557196+0000","last_became_active":"2023-10-14T18:14:46.551377+0000","last_became_peered":"2023-10-14T18:14:46.551377+0000","last_unstale":"2023-10-14T18:14:47.557196+0000","last_undegraded":"2023-10-14T18:14:47.557196+0000","last_fullsized":"2023-10-14T18:14:47.557196+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:32.135916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560087+0000","last_change":"2023-10-14T18:14:46.570127+0000","last_active":"2023-10-14T18:14:47.560087+0000","last_peered":"2023-10-14T18:14:47.560087+0000","last_clean":"2023-10-14T18:14:47.560087+0000","last_became_active":"2023-10-14T18:14:46.569846+0000","last_became_peered":"2023-10-14T18:14:46.569846+0000","last_unstale":"2023-10-14T18:14:47.560087+0000","last_undegraded":"2023-10-14T18:14:47.560087+0000","last_fullsized":"2023-10-14T18:14:47.560087+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:38.286925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562301+0000","last_change":"2023-10-14T18:14:46.561397+0000","last_active":"2023-10-14T18:14:47.562301+0000","last_peered":"2023-10-14T18:14:47.562301+0000","last_clean":"2023-10-14T18:14:47.562301+0000","last_became_active":"2023-10-14T18:14:46.561053+0000","last_became_peered":"2023-10-14T18:14:46.561053+0000","last_unstale":"2023-10-14T18:14:47.562301+0000","last_undegraded":"2023-10-14T18:14:47.562301+0000","last_fullsized":"2023-10-14T18:14:47.562301+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561682+0000","last_change":"2023-10-14T18:14:46.557710+0000","last_active":"2023-10-14T18:14:47.561682+0000","last_peered":"2023-10-14T18:14:47.561682+0000","last_clean":"2023-10-14T18:14:47.561682+0000","last_became_active":"2023-10-14T18:14:46.557395+0000","last_became_peered":"2023-10-14T18:14:46.557395+0000","last_unstale":"2023-10-14T18:14:47.561682+0000","last_undegraded":"2023-10-14T18:14:47.561682+0000","last_fullsized":"2023-10-14T18:14:47.561682+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570225+0000","last_change":"2023-10-14T18:14:46.564724+0000","last_active":"2023-10-14T18:14:46.570225+0000","last_peered":"2023-10-14T18:14:46.570225+0000","last_clean":"2023-10-14T18:14:46.570225+0000","last_became_active":"2023-10-14T18:14:46.564423+0000","last_became_peered":"2023-10-14T18:14:46.564423+0000","last_unstale":"2023-10-14T18:14:46.570225+0000","last_undegraded":"2023-10-14T18:14:46.570225+0000","last_fullsized":"2023-10-14T18:14:46.570225+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561722+0000","last_change":"2023-10-14T18:14:46.561306+0000","last_active":"2023-10-14T18:14:47.561722+0000","last_peered":"2023-10-14T18:14:47.561722+0000","last_clean":"2023-10-14T18:14:47.561722+0000","last_became_active":"2023-10-14T18:14:46.561000+0000","last_became_peered":"2023-10-14T18:14:46.561000+0000","last_unstale":"2023-10-14T18:14:47.561722+0000","last_undegraded":"2023-10-14T18:14:47.561722+0000","last_fullsized":"2023-10-14T18:14:47.561722+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568513+0000","last_change":"2023-10-14T18:14:46.555284+0000","last_active":"2023-10-14T18:14:46.568513+0000","last_peered":"2023-10-14T18:14:46.568513+0000","last_clean":"2023-10-14T18:14:46.568513+0000","last_became_active":"2023-10-14T18:14:46.553906+0000","last_became_peered":"2023-10-14T18:14:46.553906+0000","last_unstale":"2023-10-14T18:14:46.568513+0000","last_undegraded":"2023-10-14T18:14:46.568513+0000","last_fullsized":"2023-10-14T18:14:46.568513+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:32:18.970031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570430+0000","last_change":"2023-10-14T18:14:46.566962+0000","last_active":"2023-10-14T18:14:46.570430+0000","last_peered":"2023-10-14T18:14:46.570430+0000","last_clean":"2023-10-14T18:14:46.570430+0000","last_became_active":"2023-10-14T18:14:46.563574+0000","last_became_peered":"2023-10-14T18:14:46.563574+0000","last_unstale":"2023-10-14T18:14:46.570430+0000","last_undegraded":"2023-10-14T18:14:46.570430+0000","last_fullsized":"2023-10-14T18:14:46.570430+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:35.547370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572193+0000","last_change":"2023-10-14T18:14:46.567376+0000","last_active":"2023-10-14T18:14:46.572193+0000","last_peered":"2023-10-14T18:14:46.572193+0000","last_clean":"2023-10-14T18:14:46.572193+0000","last_became_active":"2023-10-14T18:14:46.567058+0000","last_became_peered":"2023-10-14T18:14:46.567058+0000","last_unstale":"2023-10-14T18:14:46.572193+0000","last_undegraded":"2023-10-14T18:14:46.572193+0000","last_fullsized":"2023-10-14T18:14:46.572193+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:21.418564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556802+0000","last_change":"2023-10-14T18:14:46.566674+0000","last_active":"2023-10-14T18:14:47.556802+0000","last_peered":"2023-10-14T18:14:47.556802+0000","last_clean":"2023-10-14T18:14:47.556802+0000","last_became_active":"2023-10-14T18:14:46.566337+0000","last_became_peered":"2023-10-14T18:14:46.566337+0000","last_unstale":"2023-10-14T18:14:47.556802+0000","last_undegraded":"2023-10-14T18:14:47.556802+0000","last_fullsized":"2023-10-14T18:14:47.556802+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568430+0000","last_change":"2023-10-14T18:14:46.562615+0000","last_active":"2023-10-14T18:14:46.568430+0000","last_peered":"2023-10-14T18:14:46.568430+0000","last_clean":"2023-10-14T18:14:46.568430+0000","last_became_active":"2023-10-14T18:14:46.562098+0000","last_became_peered":"2023-10-14T18:14:46.562098+0000","last_unstale":"2023-10-14T18:14:46.568430+0000","last_undegraded":"2023-10-14T18:14:46.568430+0000","last_fullsized":"2023-10-14T18:14:46.568430+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:27.757684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571294+0000","last_change":"2023-10-14T18:14:46.567400+0000","last_active":"2023-10-14T18:14:46.571294+0000","last_peered":"2023-10-14T18:14:46.571294+0000","last_clean":"2023-10-14T18:14:46.571294+0000","last_became_active":"2023-10-14T18:14:46.566651+0000","last_became_peered":"2023-10-14T18:14:46.566651+0000","last_unstale":"2023-10-14T18:14:46.571294+0000","last_undegraded":"2023-10-14T18:14:46.571294+0000","last_fullsized":"2023-10-14T18:14:46.571294+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556170+0000","last_change":"2023-10-14T18:14:46.551965+0000","last_active":"2023-10-14T18:14:47.556170+0000","last_peered":"2023-10-14T18:14:47.556170+0000","last_clean":"2023-10-14T18:14:47.556170+0000","last_became_active":"2023-10-14T18:14:46.551660+0000","last_became_peered":"2023-10-14T18:14:46.551660+0000","last_unstale":"2023-10-14T18:14:47.556170+0000","last_undegraded":"2023-10-14T18:14:47.556170+0000","last_fullsized":"2023-10-14T18:14:47.556170+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571674+0000","last_change":"2023-10-14T18:14:46.564391+0000","last_active":"2023-10-14T18:14:46.571674+0000","last_peered":"2023-10-14T18:14:46.571674+0000","last_clean":"2023-10-14T18:14:46.571674+0000","last_became_active":"2023-10-14T18:14:46.564112+0000","last_became_peered":"2023-10-14T18:14:46.564112+0000","last_unstale":"2023-10-14T18:14:46.571674+0000","last_undegraded":"2023-10-14T18:14:46.571674+0000","last_fullsized":"2023-10-14T18:14:46.571674+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:54.019208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.558165+0000","last_change":"2023-10-14T18:14:46.551071+0000","last_active":"2023-10-14T18:14:46.558165+0000","last_peered":"2023-10-14T18:14:46.558165+0000","last_clean":"2023-10-14T18:14:46.558165+0000","last_became_active":"2023-10-14T18:14:46.550400+0000","last_became_peered":"2023-10-14T18:14:46.550400+0000","last_unstale":"2023-10-14T18:14:46.558165+0000","last_undegraded":"2023-10-14T18:14:46.558165+0000","last_fullsized":"2023-10-14T18:14:46.558165+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:35:40.069324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.563047+0000","last_change":"2023-10-14T18:14:46.552964+0000","last_active":"2023-10-14T18:14:47.563047+0000","last_peered":"2023-10-14T18:14:47.563047+0000","last_clean":"2023-10-14T18:14:47.563047+0000","last_became_active":"2023-10-14T18:14:46.552661+0000","last_became_peered":"2023-10-14T18:14:46.552661+0000","last_unstale":"2023-10-14T18:14:47.563047+0000","last_undegraded":"2023-10-14T18:14:47.563047+0000","last_fullsized":"2023-10-14T18:14:47.563047+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560482+0000","last_change":"2023-10-14T18:14:46.558308+0000","last_active":"2023-10-14T18:14:47.560482+0000","last_peered":"2023-10-14T18:14:47.560482+0000","last_clean":"2023-10-14T18:14:47.560482+0000","last_became_active":"2023-10-14T18:14:46.553562+0000","last_became_peered":"2023-10-14T18:14:46.553562+0000","last_unstale":"2023-10-14T18:14:47.560482+0000","last_undegraded":"2023-10-14T18:14:47.560482+0000","last_fullsized":"2023-10-14T18:14:47.560482+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:34.657113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574203+0000","last_change":"2023-10-14T18:14:46.572915+0000","last_active":"2023-10-14T18:14:46.574203+0000","last_peered":"2023-10-14T18:14:46.574203+0000","last_clean":"2023-10-14T18:14:46.574203+0000","last_became_active":"2023-10-14T18:14:46.572618+0000","last_became_peered":"2023-10-14T18:14:46.572618+0000","last_unstale":"2023-10-14T18:14:46.574203+0000","last_undegraded":"2023-10-14T18:14:46.574203+0000","last_fullsized":"2023-10-14T18:14:46.574203+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:20.544825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.554232+0000","last_change":"2023-10-14T18:14:46.550445+0000","last_active":"2023-10-14T18:14:46.554232+0000","last_peered":"2023-10-14T18:14:46.554232+0000","last_clean":"2023-10-14T18:14:46.554232+0000","last_became_active":"2023-10-14T18:14:46.550155+0000","last_became_peered":"2023-10-14T18:14:46.550155+0000","last_unstale":"2023-10-14T18:14:46.554232+0000","last_undegraded":"2023-10-14T18:14:46.554232+0000","last_fullsized":"2023-10-14T18:14:46.554232+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561681+0000","last_change":"2023-10-14T18:14:46.561854+0000","last_active":"2023-10-14T18:14:47.561681+0000","last_peered":"2023-10-14T18:14:47.561681+0000","last_clean":"2023-10-14T18:14:47.561681+0000","last_became_active":"2023-10-14T18:14:46.561579+0000","last_became_peered":"2023-10-14T18:14:46.561579+0000","last_unstale":"2023-10-14T18:14:47.561681+0000","last_undegraded":"2023-10-14T18:14:47.561681+0000","last_fullsized":"2023-10-14T18:14:47.561681+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:15.737925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566153+0000","last_change":"2023-10-14T18:14:46.559624+0000","last_active":"2023-10-14T18:14:46.566153+0000","last_peered":"2023-10-14T18:14:46.566153+0000","last_clean":"2023-10-14T18:14:46.566153+0000","last_became_active":"2023-10-14T18:14:46.559314+0000","last_became_peered":"2023-10-14T18:14:46.559314+0000","last_unstale":"2023-10-14T18:14:46.566153+0000","last_undegraded":"2023-10-14T18:14:46.566153+0000","last_fullsized":"2023-10-14T18:14:46.566153+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.569808+0000","last_change":"2023-10-14T18:14:46.555509+0000","last_active":"2023-10-14T18:14:46.569808+0000","last_peered":"2023-10-14T18:14:46.569808+0000","last_clean":"2023-10-14T18:14:46.569808+0000","last_became_active":"2023-10-14T18:14:46.555204+0000","last_became_peered":"2023-10-14T18:14:46.555204+0000","last_unstale":"2023-10-14T18:14:46.569808+0000","last_undegraded":"2023-10-14T18:14:46.569808+0000","last_fullsized":"2023-10-14T18:14:46.569808+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:38.286925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567143+0000","last_change":"2023-10-14T18:14:46.559267+0000","last_active":"2023-10-14T18:14:46.567143+0000","last_peered":"2023-10-14T18:14:46.567143+0000","last_clean":"2023-10-14T18:14:46.567143+0000","last_became_active":"2023-10-14T18:14:46.558982+0000","last_became_peered":"2023-10-14T18:14:46.558982+0000","last_unstale":"2023-10-14T18:14:46.567143+0000","last_undegraded":"2023-10-14T18:14:46.567143+0000","last_fullsized":"2023-10-14T18:14:46.567143+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:03.175998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573022+0000","last_change":"2023-10-14T18:14:46.563281+0000","last_active":"2023-10-14T18:14:46.573022+0000","last_peered":"2023-10-14T18:14:46.573022+0000","last_clean":"2023-10-14T18:14:46.573022+0000","last_became_active":"2023-10-14T18:14:46.562991+0000","last_became_peered":"2023-10-14T18:14:46.562991+0000","last_unstale":"2023-10-14T18:14:46.573022+0000","last_undegraded":"2023-10-14T18:14:46.573022+0000","last_fullsized":"2023-10-14T18:14:46.573022+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:34.504377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"209'4","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572114+0000","last_change":"2023-10-14T18:14:45.556082+0000","last_active":"2023-10-14T18:14:46.572114+0000","last_peered":"2023-10-14T18:14:46.572114+0000","last_clean":"2023-10-14T18:14:46.572114+0000","last_became_active":"2023-10-14T18:14:45.555619+0000","last_became_peered":"2023-10-14T18:14:45.555619+0000","last_unstale":"2023-10-14T18:14:46.572114+0000","last_undegraded":"2023-10-14T18:14:46.572114+0000","last_fullsized":"2023-10-14T18:14:46.572114+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:30:29.198245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557314+0000","last_change":"2023-10-14T18:14:46.564731+0000","last_active":"2023-10-14T18:14:47.557314+0000","last_peered":"2023-10-14T18:14:47.557314+0000","last_clean":"2023-10-14T18:14:47.557314+0000","last_became_active":"2023-10-14T18:14:46.562496+0000","last_became_peered":"2023-10-14T18:14:46.562496+0000","last_unstale":"2023-10-14T18:14:47.557314+0000","last_undegraded":"2023-10-14T18:14:47.557314+0000","last_fullsized":"2023-10-14T18:14:47.557314+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:35.547370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566476+0000","last_change":"2023-10-14T18:14:45.552350+0000","last_active":"2023-10-14T18:14:46.566476+0000","last_peered":"2023-10-14T18:14:46.566476+0000","last_clean":"2023-10-14T18:14:46.566476+0000","last_became_active":"2023-10-14T18:14:45.551987+0000","last_became_peered":"2023-10-14T18:14:45.551987+0000","last_unstale":"2023-10-14T18:14:46.566476+0000","last_undegraded":"2023-10-14T18:14:46.566476+0000","last_fullsized":"2023-10-14T18:14:46.566476+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:34.529092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559453+0000","last_change":"2023-10-14T18:14:46.556157+0000","last_active":"2023-10-14T18:14:47.559453+0000","last_peered":"2023-10-14T18:14:47.559453+0000","last_clean":"2023-10-14T18:14:47.559453+0000","last_became_active":"2023-10-14T18:14:46.555405+0000","last_became_peered":"2023-10-14T18:14:46.555405+0000","last_unstale":"2023-10-14T18:14:47.559453+0000","last_undegraded":"2023-10-14T18:14:47.559453+0000","last_fullsized":"2023-10-14T18:14:47.559453+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:54.019208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573304+0000","last_change":"2023-10-14T18:14:45.557612+0000","last_active":"2023-10-14T18:14:46.573304+0000","last_peered":"2023-10-14T18:14:46.573304+0000","last_clean":"2023-10-14T18:14:46.573304+0000","last_became_active":"2023-10-14T18:14:45.556806+0000","last_became_peered":"2023-10-14T18:14:45.556806+0000","last_unstale":"2023-10-14T18:14:46.573304+0000","last_undegraded":"2023-10-14T18:14:46.573304+0000","last_fullsized":"2023-10-14T18:14:46.573304+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:12.146857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570103+0000","last_change":"2023-10-14T18:14:46.564242+0000","last_active":"2023-10-14T18:14:46.570103+0000","last_peered":"2023-10-14T18:14:46.570103+0000","last_clean":"2023-10-14T18:14:46.570103+0000","last_became_active":"2023-10-14T18:14:46.561991+0000","last_became_peered":"2023-10-14T18:14:46.561991+0000","last_unstale":"2023-10-14T18:14:46.570103+0000","last_undegraded":"2023-10-14T18:14:46.570103+0000","last_fullsized":"2023-10-14T18:14:46.570103+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"209'2","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562672+0000","last_change":"2023-10-14T18:14:45.545550+0000","last_active":"2023-10-14T18:14:47.562672+0000","last_peered":"2023-10-14T18:14:47.562672+0000","last_clean":"2023-10-14T18:14:47.562672+0000","last_became_active":"2023-10-14T18:14:45.545277+0000","last_became_peered":"2023-10-14T18:14:45.545277+0000","last_unstale":"2023-10-14T18:14:47.562672+0000","last_undegraded":"2023-10-14T18:14:47.562672+0000","last_fullsized":"2023-10-14T18:14:47.562672+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T06:08:00.801720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556334+0000","last_change":"2023-10-14T18:14:46.544726+0000","last_active":"2023-10-14T18:14:47.556334+0000","last_peered":"2023-10-14T18:14:47.556334+0000","last_clean":"2023-10-14T18:14:47.556334+0000","last_became_active":"2023-10-14T18:14:46.544491+0000","last_became_peered":"2023-10-14T18:14:46.544491+0000","last_unstale":"2023-10-14T18:14:47.556334+0000","last_undegraded":"2023-10-14T18:14:47.556334+0000","last_fullsized":"2023-10-14T18:14:47.556334+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567801+0000","last_change":"2023-10-14T18:14:45.554173+0000","last_active":"2023-10-14T18:14:46.567801+0000","last_peered":"2023-10-14T18:14:46.567801+0000","last_clean":"2023-10-14T18:14:46.567801+0000","last_became_active":"2023-10-14T18:14:45.553826+0000","last_became_peered":"2023-10-14T18:14:45.553826+0000","last_unstale":"2023-10-14T18:14:46.567801+0000","last_undegraded":"2023-10-14T18:14:46.567801+0000","last_fullsized":"2023-10-14T18:14:46.567801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:04.793139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570045+0000","last_change":"2023-10-14T18:14:46.562776+0000","last_active":"2023-10-14T18:14:46.570045+0000","last_peered":"2023-10-14T18:14:46.570045+0000","last_clean":"2023-10-14T18:14:46.570045+0000","last_became_active":"2023-10-14T18:14:46.561896+0000","last_became_peered":"2023-10-14T18:14:46.561896+0000","last_unstale":"2023-10-14T18:14:46.570045+0000","last_undegraded":"2023-10-14T18:14:46.570045+0000","last_fullsized":"2023-10-14T18:14:46.570045+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:21.418564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.569889+0000","last_change":"2023-10-14T18:14:45.545539+0000","last_active":"2023-10-14T18:14:46.569889+0000","last_peered":"2023-10-14T18:14:46.569889+0000","last_clean":"2023-10-14T18:14:46.569889+0000","last_became_active":"2023-10-14T18:14:45.545247+0000","last_became_peered":"2023-10-14T18:14:45.545247+0000","last_unstale":"2023-10-14T18:14:46.569889+0000","last_undegraded":"2023-10-14T18:14:46.569889+0000","last_fullsized":"2023-10-14T18:14:46.569889+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:03:00.466585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.558888+0000","last_change":"2023-10-14T18:14:46.566743+0000","last_active":"2023-10-14T18:14:47.558888+0000","last_peered":"2023-10-14T18:14:47.558888+0000","last_clean":"2023-10-14T18:14:47.558888+0000","last_became_active":"2023-10-14T18:14:46.566463+0000","last_became_peered":"2023-10-14T18:14:46.566463+0000","last_unstale":"2023-10-14T18:14:47.558888+0000","last_undegraded":"2023-10-14T18:14:47.558888+0000","last_fullsized":"2023-10-14T18:14:47.558888+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:52:56.131847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.552837+0000","last_change":"2023-10-14T18:14:45.550168+0000","last_active":"2023-10-14T18:14:46.552837+0000","last_peered":"2023-10-14T18:14:46.552837+0000","last_clean":"2023-10-14T18:14:46.552837+0000","last_became_active":"2023-10-14T18:14:45.547775+0000","last_became_peered":"2023-10-14T18:14:45.547775+0000","last_unstale":"2023-10-14T18:14:46.552837+0000","last_undegraded":"2023-10-14T18:14:46.552837+0000","last_fullsized":"2023-10-14T18:14:46.552837+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:57.839057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568723+0000","last_change":"2023-10-14T18:14:46.560399+0000","last_active":"2023-10-14T18:14:46.568723+0000","last_peered":"2023-10-14T18:14:46.568723+0000","last_clean":"2023-10-14T18:14:46.568723+0000","last_became_active":"2023-10-14T18:14:46.558636+0000","last_became_peered":"2023-10-14T18:14:46.558636+0000","last_unstale":"2023-10-14T18:14:46.568723+0000","last_undegraded":"2023-10-14T18:14:46.568723+0000","last_fullsized":"2023-10-14T18:14:46.568723+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:21.293245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560222+0000","last_change":"2023-10-14T18:14:45.555234+0000","last_active":"2023-10-14T18:14:47.560222+0000","last_peered":"2023-10-14T18:14:47.560222+0000","last_clean":"2023-10-14T18:14:47.560222+0000","last_became_active":"2023-10-14T18:14:45.554937+0000","last_became_peered":"2023-10-14T18:14:45.554937+0000","last_unstale":"2023-10-14T18:14:47.560222+0000","last_undegraded":"2023-10-14T18:14:47.560222+0000","last_fullsized":"2023-10-14T18:14:47.560222+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.635610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553485+0000","last_change":"2023-10-14T18:14:46.545838+0000","last_active":"2023-10-14T18:14:46.553485+0000","last_peered":"2023-10-14T18:14:46.553485+0000","last_clean":"2023-10-14T18:14:46.553485+0000","last_became_active":"2023-10-14T18:14:46.545559+0000","last_became_peered":"2023-10-14T18:14:46.545559+0000","last_unstale":"2023-10-14T18:14:46.553485+0000","last_undegraded":"2023-10-14T18:14:46.553485+0000","last_fullsized":"2023-10-14T18:14:46.553485+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:50.178686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.549187+0000","last_change":"2023-10-14T18:14:45.550279+0000","last_active":"2023-10-14T18:14:46.549187+0000","last_peered":"2023-10-14T18:14:46.549187+0000","last_clean":"2023-10-14T18:14:46.549187+0000","last_became_active":"2023-10-14T18:14:45.549992+0000","last_became_peered":"2023-10-14T18:14:45.549992+0000","last_unstale":"2023-10-14T18:14:46.549187+0000","last_undegraded":"2023-10-14T18:14:46.549187+0000","last_fullsized":"2023-10-14T18:14:46.549187+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:49.118266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559523+0000","last_change":"2023-10-14T18:14:46.570086+0000","last_active":"2023-10-14T18:14:47.559523+0000","last_peered":"2023-10-14T18:14:47.559523+0000","last_clean":"2023-10-14T18:14:47.559523+0000","last_became_active":"2023-10-14T18:14:46.567808+0000","last_became_peered":"2023-10-14T18:14:46.567808+0000","last_unstale":"2023-10-14T18:14:47.559523+0000","last_undegraded":"2023-10-14T18:14:47.559523+0000","last_fullsized":"2023-10-14T18:14:47.559523+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:19.158160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"209'3","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572769+0000","last_change":"2023-10-14T18:14:45.547108+0000","last_active":"2023-10-14T18:14:46.572769+0000","last_peered":"2023-10-14T18:14:46.572769+0000","last_clean":"2023-10-14T18:14:46.572769+0000","last_became_active":"2023-10-14T18:14:45.546782+0000","last_became_peered":"2023-10-14T18:14:45.546782+0000","last_unstale":"2023-10-14T18:14:46.572769+0000","last_undegraded":"2023-10-14T18:14:46.572769+0000","last_fullsized":"2023-10-14T18:14:46.572769+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:42.368847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557097+0000","last_change":"2023-10-14T18:14:46.545417+0000","last_active":"2023-10-14T18:14:47.557097+0000","last_peered":"2023-10-14T18:14:47.557097+0000","last_clean":"2023-10-14T18:14:47.557097+0000","last_became_active":"2023-10-14T18:14:46.544355+0000","last_became_peered":"2023-10-14T18:14:46.544355+0000","last_unstale":"2023-10-14T18:14:47.557097+0000","last_undegraded":"2023-10-14T18:14:47.557097+0000","last_fullsized":"2023-10-14T18:14:47.557097+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:27.757684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"209'6","reported_seq":17,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.575877+0000","last_change":"2023-10-14T18:14:45.556361+0000","last_active":"2023-10-14T18:14:46.575877+0000","last_peered":"2023-10-14T18:14:46.575877+0000","last_clean":"2023-10-14T18:14:46.575877+0000","last_became_active":"2023-10-14T18:14:45.556085+0000","last_became_peered":"2023-10-14T18:14:45.556085+0000","last_unstale":"2023-10-14T18:14:46.575877+0000","last_undegraded":"2023-10-14T18:14:46.575877+0000","last_fullsized":"2023-10-14T18:14:46.575877+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:27.286846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573926+0000","last_change":"2023-10-14T18:14:46.565618+0000","last_active":"2023-10-14T18:14:46.573926+0000","last_peered":"2023-10-14T18:14:46.573926+0000","last_clean":"2023-10-14T18:14:46.573926+0000","last_became_active":"2023-10-14T18:14:46.565352+0000","last_became_peered":"2023-10-14T18:14:46.565352+0000","last_unstale":"2023-10-14T18:14:46.573926+0000","last_undegraded":"2023-10-14T18:14:46.573926+0000","last_fullsized":"2023-10-14T18:14:46.573926+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:36.892049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574111+0000","last_change":"2023-10-14T18:14:45.552115+0000","last_active":"2023-10-14T18:14:46.574111+0000","last_peered":"2023-10-14T18:14:46.574111+0000","last_clean":"2023-10-14T18:14:46.574111+0000","last_became_active":"2023-10-14T18:14:45.551248+0000","last_became_peered":"2023-10-14T18:14:45.551248+0000","last_unstale":"2023-10-14T18:14:46.574111+0000","last_undegraded":"2023-10-14T18:14:46.574111+0000","last_fullsized":"2023-10-14T18:14:46.574111+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:32.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559397+0000","last_change":"2023-10-14T18:14:46.560433+0000","last_active":"2023-10-14T18:14:47.559397+0000","last_peered":"2023-10-14T18:14:47.559397+0000","last_clean":"2023-10-14T18:14:47.559397+0000","last_became_active":"2023-10-14T18:14:46.560153+0000","last_became_peered":"2023-10-14T18:14:46.560153+0000","last_unstale":"2023-10-14T18:14:47.559397+0000","last_undegraded":"2023-10-14T18:14:47.559397+0000","last_fullsized":"2023-10-14T18:14:47.559397+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:28.305124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556578+0000","last_change":"2023-10-14T18:14:45.552984+0000","last_active":"2023-10-14T18:14:47.556578+0000","last_peered":"2023-10-14T18:14:47.556578+0000","last_clean":"2023-10-14T18:14:47.556578+0000","last_became_active":"2023-10-14T18:14:45.552616+0000","last_became_peered":"2023-10-14T18:14:45.552616+0000","last_unstale":"2023-10-14T18:14:47.556578+0000","last_undegraded":"2023-10-14T18:14:47.556578+0000","last_fullsized":"2023-10-14T18:14:47.556578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:04.427609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566410+0000","last_change":"2023-10-14T18:14:46.559770+0000","last_active":"2023-10-14T18:14:46.566410+0000","last_peered":"2023-10-14T18:14:46.566410+0000","last_clean":"2023-10-14T18:14:46.566410+0000","last_became_active":"2023-10-14T18:14:46.559410+0000","last_became_peered":"2023-10-14T18:14:46.559410+0000","last_unstale":"2023-10-14T18:14:46.566410+0000","last_undegraded":"2023-10-14T18:14:46.566410+0000","last_fullsized":"2023-10-14T18:14:46.566410+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:15.737925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.529274+0000","last_change":"2023-10-14T18:14:45.543303+0000","last_active":"2023-10-14T18:14:46.529274+0000","last_peered":"2023-10-14T18:14:46.529274+0000","last_clean":"2023-10-14T18:14:46.529274+0000","last_became_active":"2023-10-14T18:14:45.541994+0000","last_became_peered":"2023-10-14T18:14:45.541994+0000","last_unstale":"2023-10-14T18:14:46.529274+0000","last_undegraded":"2023-10-14T18:14:46.529274+0000","last_fullsized":"2023-10-14T18:14:46.529274+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:00.301921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.555732+0000","last_change":"2023-10-14T18:14:46.560392+0000","last_active":"2023-10-14T18:14:47.555732+0000","last_peered":"2023-10-14T18:14:47.555732+0000","last_clean":"2023-10-14T18:14:47.555732+0000","last_became_active":"2023-10-14T18:14:46.560079+0000","last_became_peered":"2023-10-14T18:14:46.560079+0000","last_unstale":"2023-10-14T18:14:47.555732+0000","last_undegraded":"2023-10-14T18:14:47.555732+0000","last_fullsized":"2023-10-14T18:14:47.555732+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"209'8","reported_seq":19,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574806+0000","last_change":"2023-10-14T18:14:45.550864+0000","last_active":"2023-10-14T18:14:46.574806+0000","last_peered":"2023-10-14T18:14:46.574806+0000","last_clean":"2023-10-14T18:14:46.574806+0000","last_became_active":"2023-10-14T18:14:45.550076+0000","last_became_peered":"2023-10-14T18:14:45.550076+0000","last_unstale":"2023-10-14T18:14:46.574806+0000","last_undegraded":"2023-10-14T18:14:46.574806+0000","last_fullsized":"2023-10-14T18:14:46.574806+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:21.237010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557051+0000","last_change":"2023-10-14T18:14:46.561688+0000","last_active":"2023-10-14T18:14:47.557051+0000","last_peered":"2023-10-14T18:14:47.557051+0000","last_clean":"2023-10-14T18:14:47.557051+0000","last_became_active":"2023-10-14T18:14:46.561370+0000","last_became_peered":"2023-10-14T18:14:46.561370+0000","last_unstale":"2023-10-14T18:14:47.557051+0000","last_undegraded":"2023-10-14T18:14:47.557051+0000","last_fullsized":"2023-10-14T18:14:47.557051+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568861+0000","last_change":"2023-10-14T18:14:45.554342+0000","last_active":"2023-10-14T18:14:46.568861+0000","last_peered":"2023-10-14T18:14:46.568861+0000","last_clean":"2023-10-14T18:14:46.568861+0000","last_became_active":"2023-10-14T18:14:45.553690+0000","last_became_peered":"2023-10-14T18:14:45.553690+0000","last_unstale":"2023-10-14T18:14:46.568861+0000","last_undegraded":"2023-10-14T18:14:46.568861+0000","last_fullsized":"2023-10-14T18:14:46.568861+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:00:24.139422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568707+0000","last_change":"2023-10-14T18:14:46.561989+0000","last_active":"2023-10-14T18:14:46.568707+0000","last_peered":"2023-10-14T18:14:46.568707+0000","last_clean":"2023-10-14T18:14:46.568707+0000","last_became_active":"2023-10-14T18:14:46.561702+0000","last_became_peered":"2023-10-14T18:14:46.561702+0000","last_unstale":"2023-10-14T18:14:46.568707+0000","last_undegraded":"2023-10-14T18:14:46.568707+0000","last_fullsized":"2023-10-14T18:14:46.568707+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570589+0000","last_change":"2023-10-14T18:14:45.556184+0000","last_active":"2023-10-14T18:14:46.570589+0000","last_peered":"2023-10-14T18:14:46.570589+0000","last_clean":"2023-10-14T18:14:46.570589+0000","last_became_active":"2023-10-14T18:14:45.554519+0000","last_became_peered":"2023-10-14T18:14:45.554519+0000","last_unstale":"2023-10-14T18:14:46.570589+0000","last_undegraded":"2023-10-14T18:14:46.570589+0000","last_fullsized":"2023-10-14T18:14:46.570589+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:32.456073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570093+0000","last_change":"2023-10-14T18:14:46.565669+0000","last_active":"2023-10-14T18:14:46.570093+0000","last_peered":"2023-10-14T18:14:46.570093+0000","last_clean":"2023-10-14T18:14:46.570093+0000","last_became_active":"2023-10-14T18:14:46.563658+0000","last_became_peered":"2023-10-14T18:14:46.563658+0000","last_unstale":"2023-10-14T18:14:46.570093+0000","last_undegraded":"2023-10-14T18:14:46.570093+0000","last_fullsized":"2023-10-14T18:14:46.570093+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574270+0000","last_change":"2023-10-14T18:14:45.551331+0000","last_active":"2023-10-14T18:14:46.574270+0000","last_peered":"2023-10-14T18:14:46.574270+0000","last_clean":"2023-10-14T18:14:46.574270+0000","last_became_active":"2023-10-14T18:14:45.550056+0000","last_became_peered":"2023-10-14T18:14:45.550056+0000","last_unstale":"2023-10-14T18:14:46.574270+0000","last_undegraded":"2023-10-14T18:14:46.574270+0000","last_fullsized":"2023-10-14T18:14:46.574270+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:32.153972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562755+0000","last_change":"2023-10-14T18:14:46.556117+0000","last_active":"2023-10-14T18:14:47.562755+0000","last_peered":"2023-10-14T18:14:47.562755+0000","last_clean":"2023-10-14T18:14:47.562755+0000","last_became_active":"2023-10-14T18:14:46.554841+0000","last_became_peered":"2023-10-14T18:14:46.554841+0000","last_unstale":"2023-10-14T18:14:47.562755+0000","last_undegraded":"2023-10-14T18:14:47.562755+0000","last_fullsized":"2023-10-14T18:14:47.562755+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559880+0000","last_change":"2023-10-14T18:14:45.554571+0000","last_active":"2023-10-14T18:14:47.559880+0000","last_peered":"2023-10-14T18:14:47.559880+0000","last_clean":"2023-10-14T18:14:47.559880+0000","last_became_active":"2023-10-14T18:14:45.554270+0000","last_became_peered":"2023-10-14T18:14:45.554270+0000","last_unstale":"2023-10-14T18:14:47.559880+0000","last_undegraded":"2023-10-14T18:14:47.559880+0000","last_fullsized":"2023-10-14T18:14:47.559880+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:57.448693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556510+0000","last_change":"2023-10-14T18:14:46.556384+0000","last_active":"2023-10-14T18:14:47.556510+0000","last_peered":"2023-10-14T18:14:47.556510+0000","last_clean":"2023-10-14T18:14:47.556510+0000","last_became_active":"2023-10-14T18:14:46.556139+0000","last_became_peered":"2023-10-14T18:14:46.556139+0000","last_unstale":"2023-10-14T18:14:47.556510+0000","last_undegraded":"2023-10-14T18:14:47.556510+0000","last_fullsized":"2023-10-14T18:14:47.556510+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571896+0000","last_change":"2023-10-14T18:14:45.554829+0000","last_active":"2023-10-14T18:14:46.571896+0000","last_peered":"2023-10-14T18:14:46.571896+0000","last_clean":"2023-10-14T18:14:46.571896+0000","last_became_active":"2023-10-14T18:14:45.553915+0000","last_became_peered":"2023-10-14T18:14:45.553915+0000","last_unstale":"2023-10-14T18:14:46.571896+0000","last_undegraded":"2023-10-14T18:14:46.571896+0000","last_fullsized":"2023-10-14T18:14:46.571896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:18.139882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559601+0000","last_change":"2023-10-14T18:14:46.549534+0000","last_active":"2023-10-14T18:14:47.559601+0000","last_peered":"2023-10-14T18:14:47.559601+0000","last_clean":"2023-10-14T18:14:47.559601+0000","last_became_active":"2023-10-14T18:14:46.549113+0000","last_became_peered":"2023-10-14T18:14:46.549113+0000","last_unstale":"2023-10-14T18:14:47.559601+0000","last_undegraded":"2023-10-14T18:14:47.559601+0000","last_fullsized":"2023-10-14T18:14:47.559601+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:02.805957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.547385+0000","last_change":"2023-10-14T18:14:45.545136+0000","last_active":"2023-10-14T18:14:46.547385+0000","last_peered":"2023-10-14T18:14:46.547385+0000","last_clean":"2023-10-14T18:14:46.547385+0000","last_became_active":"2023-10-14T18:14:45.544821+0000","last_became_peered":"2023-10-14T18:14:45.544821+0000","last_unstale":"2023-10-14T18:14:46.547385+0000","last_undegraded":"2023-10-14T18:14:46.547385+0000","last_fullsized":"2023-10-14T18:14:46.547385+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:31.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560773+0000","last_change":"2023-10-14T18:14:46.562237+0000","last_active":"2023-10-14T18:14:47.560773+0000","last_peered":"2023-10-14T18:14:47.560773+0000","last_clean":"2023-10-14T18:14:47.560773+0000","last_became_active":"2023-10-14T18:14:46.561910+0000","last_became_peered":"2023-10-14T18:14:46.561910+0000","last_unstale":"2023-10-14T18:14:47.560773+0000","last_undegraded":"2023-10-14T18:14:47.560773+0000","last_fullsized":"2023-10-14T18:14:47.560773+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:22.546148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.539093+0000","last_change":"2023-10-14T18:14:45.547848+0000","last_active":"2023-10-14T18:14:46.539093+0000","last_peered":"2023-10-14T18:14:46.539093+0000","last_clean":"2023-10-14T18:14:46.539093+0000","last_became_active":"2023-10-14T18:14:45.547338+0000","last_became_peered":"2023-10-14T18:14:45.547338+0000","last_unstale":"2023-10-14T18:14:46.539093+0000","last_undegraded":"2023-10-14T18:14:46.539093+0000","last_fullsized":"2023-10-14T18:14:46.539093+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:33.638835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.559071+0000","last_change":"2023-10-14T18:14:46.554976+0000","last_active":"2023-10-14T18:14:46.559071+0000","last_peered":"2023-10-14T18:14:46.559071+0000","last_clean":"2023-10-14T18:14:46.559071+0000","last_became_active":"2023-10-14T18:14:46.554646+0000","last_became_peered":"2023-10-14T18:14:46.554646+0000","last_unstale":"2023-10-14T18:14:46.559071+0000","last_undegraded":"2023-10-14T18:14:46.559071+0000","last_fullsized":"2023-10-14T18:14:46.559071+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:12:16.144262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.546434+0000","last_change":"2023-10-14T18:14:45.548468+0000","last_active":"2023-10-14T18:14:46.546434+0000","last_peered":"2023-10-14T18:14:46.546434+0000","last_clean":"2023-10-14T18:14:46.546434+0000","last_became_active":"2023-10-14T18:14:45.545163+0000","last_became_peered":"2023-10-14T18:14:45.545163+0000","last_unstale":"2023-10-14T18:14:46.546434+0000","last_undegraded":"2023-10-14T18:14:46.546434+0000","last_fullsized":"2023-10-14T18:14:46.546434+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:26.739406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567954+0000","last_change":"2023-10-14T18:14:46.559300+0000","last_active":"2023-10-14T18:14:46.567954+0000","last_peered":"2023-10-14T18:14:46.567954+0000","last_clean":"2023-10-14T18:14:46.567954+0000","last_became_active":"2023-10-14T18:14:46.559011+0000","last_became_peered":"2023-10-14T18:14:46.559011+0000","last_unstale":"2023-10-14T18:14:46.567954+0000","last_undegraded":"2023-10-14T18:14:46.567954+0000","last_fullsized":"2023-10-14T18:14:46.567954+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:13.667031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573390+0000","last_change":"2023-10-14T18:14:45.552202+0000","last_active":"2023-10-14T18:14:46.573390+0000","last_peered":"2023-10-14T18:14:46.573390+0000","last_clean":"2023-10-14T18:14:46.573390+0000","last_became_active":"2023-10-14T18:14:45.551946+0000","last_became_peered":"2023-10-14T18:14:45.551946+0000","last_unstale":"2023-10-14T18:14:46.573390+0000","last_undegraded":"2023-10-14T18:14:46.573390+0000","last_fullsized":"2023-10-14T18:14:46.573390+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:32:17.951753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557383+0000","last_change":"2023-10-14T18:14:46.565738+0000","last_active":"2023-10-14T18:14:47.557383+0000","last_peered":"2023-10-14T18:14:47.557383+0000","last_clean":"2023-10-14T18:14:47.557383+0000","last_became_active":"2023-10-14T18:14:46.564733+0000","last_became_peered":"2023-10-14T18:14:46.564733+0000","last_unstale":"2023-10-14T18:14:47.557383+0000","last_undegraded":"2023-10-14T18:14:47.557383+0000","last_fullsized":"2023-10-14T18:14:47.557383+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:13.667031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557142+0000","last_change":"2023-10-14T18:14:45.546245+0000","last_active":"2023-10-14T18:14:47.557142+0000","last_peered":"2023-10-14T18:14:47.557142+0000","last_clean":"2023-10-14T18:14:47.557142+0000","last_became_active":"2023-10-14T18:14:45.545899+0000","last_became_peered":"2023-10-14T18:14:45.545899+0000","last_unstale":"2023-10-14T18:14:47.557142+0000","last_undegraded":"2023-10-14T18:14:47.557142+0000","last_fullsized":"2023-10-14T18:14:47.557142+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:14:42.146401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567195+0000","last_change":"2023-10-14T18:14:46.552911+0000","last_active":"2023-10-14T18:14:46.567195+0000","last_peered":"2023-10-14T18:14:46.567195+0000","last_clean":"2023-10-14T18:14:46.567195+0000","last_became_active":"2023-10-14T18:14:46.552603+0000","last_became_peered":"2023-10-14T18:14:46.552603+0000","last_unstale":"2023-10-14T18:14:46.567195+0000","last_undegraded":"2023-10-14T18:14:46.567195+0000","last_fullsized":"2023-10-14T18:14:46.567195+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:33.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556897+0000","last_change":"2023-10-14T18:14:45.555101+0000","last_active":"2023-10-14T18:14:47.556897+0000","last_peered":"2023-10-14T18:14:47.556897+0000","last_clean":"2023-10-14T18:14:47.556897+0000","last_became_active":"2023-10-14T18:14:45.554828+0000","last_became_peered":"2023-10-14T18:14:45.554828+0000","last_unstale":"2023-10-14T18:14:47.556897+0000","last_undegraded":"2023-10-14T18:14:47.556897+0000","last_fullsized":"2023-10-14T18:14:47.556897+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:13:51.305061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571764+0000","last_change":"2023-10-14T18:14:46.563147+0000","last_active":"2023-10-14T18:14:46.571764+0000","last_peered":"2023-10-14T18:14:46.571764+0000","last_clean":"2023-10-14T18:14:46.571764+0000","last_became_active":"2023-10-14T18:14:46.562863+0000","last_became_peered":"2023-10-14T18:14:46.562863+0000","last_unstale":"2023-10-14T18:14:46.571764+0000","last_undegraded":"2023-10-14T18:14:46.571764+0000","last_fullsized":"2023-10-14T18:14:46.571764+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:44.186849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.529914+0000","last_change":"2023-10-14T18:14:45.551336+0000","last_active":"2023-10-14T18:14:46.529914+0000","last_peered":"2023-10-14T18:14:46.529914+0000","last_clean":"2023-10-14T18:14:46.529914+0000","last_became_active":"2023-10-14T18:14:45.551061+0000","last_became_peered":"2023-10-14T18:14:45.551061+0000","last_unstale":"2023-10-14T18:14:46.529914+0000","last_undegraded":"2023-10-14T18:14:46.529914+0000","last_fullsized":"2023-10-14T18:14:46.529914+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:32.153972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573137+0000","last_change":"2023-10-14T18:14:46.572423+0000","last_active":"2023-10-14T18:14:46.573137+0000","last_peered":"2023-10-14T18:14:46.573137+0000","last_clean":"2023-10-14T18:14:46.573137+0000","last_became_active":"2023-10-14T18:14:46.572010+0000","last_became_peered":"2023-10-14T18:14:46.572010+0000","last_unstale":"2023-10-14T18:14:46.573137+0000","last_undegraded":"2023-10-14T18:14:46.573137+0000","last_fullsized":"2023-10-14T18:14:46.573137+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:03:01.484863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.545296+0000","last_change":"2023-10-14T18:14:45.555363+0000","last_active":"2023-10-14T18:14:46.545296+0000","last_peered":"2023-10-14T18:14:46.545296+0000","last_clean":"2023-10-14T18:14:46.545296+0000","last_became_active":"2023-10-14T18:14:45.555088+0000","last_became_peered":"2023-10-14T18:14:45.555088+0000","last_unstale":"2023-10-14T18:14:46.545296+0000","last_undegraded":"2023-10-14T18:14:46.545296+0000","last_fullsized":"2023-10-14T18:14:46.545296+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:18.982319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570899+0000","last_change":"2023-10-14T18:14:46.562687+0000","last_active":"2023-10-14T18:14:46.570899+0000","last_peered":"2023-10-14T18:14:46.570899+0000","last_clean":"2023-10-14T18:14:46.570899+0000","last_became_active":"2023-10-14T18:14:46.562402+0000","last_became_peered":"2023-10-14T18:14:46.562402+0000","last_unstale":"2023-10-14T18:14:46.570899+0000","last_undegraded":"2023-10-14T18:14:46.570899+0000","last_fullsized":"2023-10-14T18:14:46.570899+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:45:39.629254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.545296+0000","last_change":"2023-10-14T18:14:45.554797+0000","last_active":"2023-10-14T18:14:46.545296+0000","last_peered":"2023-10-14T18:14:46.545296+0000","last_clean":"2023-10-14T18:14:46.545296+0000","last_became_active":"2023-10-14T18:14:45.554501+0000","last_became_peered":"2023-10-14T18:14:45.554501+0000","last_unstale":"2023-10-14T18:14:46.545296+0000","last_undegraded":"2023-10-14T18:14:46.545296+0000","last_fullsized":"2023-10-14T18:14:46.545296+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:26.747637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560916+0000","last_change":"2023-10-14T18:14:46.548676+0000","last_active":"2023-10-14T18:14:47.560916+0000","last_peered":"2023-10-14T18:14:47.560916+0000","last_clean":"2023-10-14T18:14:47.560916+0000","last_became_active":"2023-10-14T18:14:46.548385+0000","last_became_peered":"2023-10-14T18:14:46.548385+0000","last_unstale":"2023-10-14T18:14:47.560916+0000","last_undegraded":"2023-10-14T18:14:47.560916+0000","last_fullsized":"2023-10-14T18:14:47.560916+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:14:43.164679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562088+0000","last_change":"2023-10-14T18:14:45.551245+0000","last_active":"2023-10-14T18:14:47.562088+0000","last_peered":"2023-10-14T18:14:47.562088+0000","last_clean":"2023-10-14T18:14:47.562088+0000","last_became_active":"2023-10-14T18:14:45.550954+0000","last_became_peered":"2023-10-14T18:14:45.550954+0000","last_unstale":"2023-10-14T18:14:47.562088+0000","last_undegraded":"2023-10-14T18:14:47.562088+0000","last_fullsized":"2023-10-14T18:14:47.562088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:04.427609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567704+0000","last_change":"2023-10-14T18:14:46.559511+0000","last_active":"2023-10-14T18:14:46.567704+0000","last_peered":"2023-10-14T18:14:46.567704+0000","last_clean":"2023-10-14T18:14:46.567704+0000","last_became_active":"2023-10-14T18:14:46.559231+0000","last_became_peered":"2023-10-14T18:14:46.559231+0000","last_unstale":"2023-10-14T18:14:46.567704+0000","last_undegraded":"2023-10-14T18:14:46.567704+0000","last_fullsized":"2023-10-14T18:14:46.567704+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.538940+0000","last_change":"2023-10-14T18:14:45.549540+0000","last_active":"2023-10-14T18:14:46.538940+0000","last_peered":"2023-10-14T18:14:46.538940+0000","last_clean":"2023-10-14T18:14:46.538940+0000","last_became_active":"2023-10-14T18:14:45.547454+0000","last_became_peered":"2023-10-14T18:14:45.547454+0000","last_unstale":"2023-10-14T18:14:46.538940+0000","last_undegraded":"2023-10-14T18:14:46.538940+0000","last_fullsized":"2023-10-14T18:14:46.538940+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:01.787679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.555874+0000","last_change":"2023-10-14T18:14:46.557086+0000","last_active":"2023-10-14T18:14:47.555874+0000","last_peered":"2023-10-14T18:14:47.555874+0000","last_clean":"2023-10-14T18:14:47.555874+0000","last_became_active":"2023-10-14T18:14:46.554501+0000","last_became_peered":"2023-10-14T18:14:46.554501+0000","last_unstale":"2023-10-14T18:14:47.555874+0000","last_undegraded":"2023-10-14T18:14:47.555874+0000","last_fullsized":"2023-10-14T18:14:47.555874+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553292+0000","last_change":"2023-10-14T18:14:45.546699+0000","last_active":"2023-10-14T18:14:46.553292+0000","last_peered":"2023-10-14T18:14:46.553292+0000","last_clean":"2023-10-14T18:14:46.553292+0000","last_became_active":"2023-10-14T18:14:45.546426+0000","last_became_peered":"2023-10-14T18:14:45.546426+0000","last_unstale":"2023-10-14T18:14:46.553292+0000","last_undegraded":"2023-10-14T18:14:46.553292+0000","last_fullsized":"2023-10-14T18:14:46.553292+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:00.301921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'326","reported_seq":1115,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553083+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:14:46.553083+0000","last_peered":"2023-10-14T18:14:46.553083+0000","last_clean":"2023-10-14T18:14:46.553083+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:14:46.553083+0000","last_undegraded":"2023-10-14T18:14:46.553083+0000","last_fullsized":"2023-10-14T18:14:46.553083+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575974,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52372,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319468,"statfs":{"total":96636764160,"available":96583135232,"internally_reserved":0,"allocated":3219456,"data_stored":1882870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2430000000000001,"15min":1.216},"min":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.72099999999999997},"max":{"1min":1.6539999999999999,"5min":1.9430000000000001,"15min":2.1669999999999998},"last":1.014},{"interface":"front","average":{"1min":1.236,"5min":1.24,"15min":1.2310000000000001},"min":{"1min":0.84999999999999998,"5min":0.79500000000000004,"15min":0.76500000000000001},"max":{"1min":1.6579999999999999,"5min":1.7310000000000001,"15min":2.2149999999999999},"last":1.5369999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.294,"15min":1.2989999999999999},"min":{"1min":0.85899999999999999,"5min":0.82599999999999996,"15min":0.80200000000000005},"max":{"1min":1.5940000000000001,"5min":2.214,"15min":2.3239999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.304,"15min":1.3160000000000001},"min":{"1min":1.008,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.6259999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.266}]},{"osd":2,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3089999999999999,"15min":1.2869999999999999},"min":{"1min":0.94699999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.671,"5min":2.004,"15min":2.1190000000000002},"last":1.071},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.355,"15min":1.381},"min":{"1min":1.044,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.8260000000000001,"5min":2.069,"15min":2.2189999999999999},"last":1.325}]},{"osd":3,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4330000000000001,"15min":1.4510000000000001},"min":{"1min":0.999,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.768,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5429999999999999,"15min":1.5509999999999999},"min":{"1min":1.0600000000000001,"5min":1.02,"15min":0.93200000000000005},"max":{"1min":1.724,"5min":2.2810000000000001,"15min":2.423},"last":1.1499999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3129999999999999,"15min":1.3240000000000001},"min":{"1min":0.94399999999999995,"5min":0.81999999999999995,"15min":0.79300000000000004},"max":{"1min":1.784,"5min":2.367,"15min":2.367},"last":1.389},{"interface":"front","average":{"1min":1.3,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.86399999999999999},"max":{"1min":1.8380000000000001,"5min":2.661,"15min":2.661},"last":1.2230000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4430000000000001,"15min":1.4630000000000001},"min":{"1min":0.94899999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.637,"5min":2.395,"15min":2.516},"last":1.49},{"interface":"front","average":{"1min":1.48,"5min":1.4950000000000001,"15min":1.516},"min":{"1min":1.044,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":1.8,"5min":2.218,"15min":2.448},"last":1.6000000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.6040000000000001,"15min":1.631},"min":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"max":{"1min":1.744,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.706},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.573,"15min":1.591},"min":{"1min":1.137,"5min":1.038,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":2.8159999999999998,"15min":2.8159999999999998},"last":1.446}]}]},{"osd":6,"up_from":13,"seq":55834575973,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29576,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342264,"statfs":{"total":96636764160,"available":96606478336,"internally_reserved":0,"allocated":2617344,"data_stored":1292398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.204,"15min":1.1870000000000001},"min":{"1min":0.879,"5min":0.78100000000000003,"15min":0.51300000000000001},"max":{"1min":1.6919999999999999,"5min":1.8560000000000001,"15min":1.873},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.236,"15min":1.216},"min":{"1min":0.90800000000000003,"5min":0.85199999999999998,"15min":0.56200000000000006},"max":{"1min":1.5109999999999999,"5min":1.71,"15min":1.944},"last":0.80600000000000005}]},{"osd":1,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2649999999999999,"15min":1.2589999999999999},"min":{"1min":0.96899999999999997,"5min":0.874,"15min":0.83199999999999996},"max":{"1min":1.6220000000000001,"5min":1.776,"15min":2.0840000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.381,"15min":1.353},"min":{"1min":1.1200000000000001,"5min":0.98299999999999998,"15min":0.85799999999999998},"max":{"1min":1.7010000000000001,"5min":1.7769999999999999,"15min":1.9470000000000001},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4179999999999999,"15min":1.3720000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.82599999999999996},"max":{"1min":1.7330000000000001,"5min":1.8779999999999999,"15min":2.4849999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.4399999999999999,"15min":1.429},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":0.98099999999999998},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.9870000000000001},"last":1.0489999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4119999999999999,"15min":1.4219999999999999},"min":{"1min":1.01,"5min":0.83399999999999996,"15min":0.76900000000000002},"max":{"1min":1.599,"5min":1.8560000000000001,"15min":2.2519999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4299999999999999,"15min":1.4430000000000001},"min":{"1min":1.052,"5min":0.98799999999999999,"15min":0.91300000000000003},"max":{"1min":1.8320000000000001,"5min":1.895,"15min":2.218},"last":1.2709999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3620000000000001,"15min":1.383},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.60299999999999998},"max":{"1min":1.784,"5min":1.9750000000000001,"15min":2.0299999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4199999999999999,"15min":1.4319999999999999},"min":{"1min":1.03,"5min":0.96599999999999997,"15min":0.90500000000000003},"max":{"1min":1.8580000000000001,"5min":2.0099999999999998,"15min":2.27},"last":1.411}]},{"osd":5,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4019999999999999,"15min":1.4199999999999999},"min":{"1min":0.97699999999999998,"5min":0.91100000000000003,"15min":0.873},"max":{"1min":1.851,"5min":1.9410000000000001,"15min":2.4940000000000002},"last":1.302},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5669999999999999,"15min":1.5780000000000001},"min":{"1min":1.0840000000000001,"5min":0.995,"15min":0.81399999999999995},"max":{"1min":1.899,"5min":1.996,"15min":2.5609999999999999},"last":1.3660000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5860000000000001,"15min":1.6080000000000001},"min":{"1min":1.2110000000000001,"5min":1.135,"15min":0.95399999999999996},"max":{"1min":1.9079999999999999,"5min":1.9950000000000001,"15min":3.423},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.581,"15min":1.607},"min":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":1.9690000000000001,"5min":2.0190000000000001,"15min":2.8769999999999998},"last":1.5649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42056,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329784,"statfs":{"total":96636764160,"available":96593698816,"internally_reserved":0,"allocated":2617344,"data_stored":1292398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.214,"15min":1.2030000000000001},"min":{"1min":0.84099999999999997,"5min":0.80700000000000005,"15min":0.60399999999999998},"max":{"1min":1.474,"5min":1.9830000000000001,"15min":2.0459999999999998},"last":1.075},{"interface":"front","average":{"1min":1.248,"5min":1.264,"15min":1.2569999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.71799999999999997},"max":{"1min":1.7270000000000001,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.006}]},{"osd":1,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2829999999999999,"15min":1.27},"min":{"1min":0.90900000000000003,"5min":0.81200000000000006,"15min":0.74099999999999999},"max":{"1min":1.645,"5min":1.875,"15min":1.8899999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2929999999999999,"15min":1.304},"min":{"1min":0.99099999999999999,"5min":0.873,"15min":0.63},"max":{"1min":1.5489999999999999,"5min":1.849,"15min":2.0419999999999998},"last":1.153}]},{"osd":2,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2430000000000001,"15min":1.2649999999999999},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.71199999999999997},"max":{"1min":1.589,"5min":1.9339999999999999,"15min":3.7090000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4139999999999999,"15min":1.385},"min":{"1min":0.96899999999999997,"5min":0.91300000000000003,"15min":0.67600000000000005},"max":{"1min":1.5609999999999999,"5min":1.923,"15min":1.923},"last":1.5209999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.496,"15min":1.4710000000000001},"min":{"1min":0.95699999999999996,"5min":0.90900000000000003,"15min":0.81899999999999995},"max":{"1min":1.7929999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":1.256},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4379999999999999,"15min":1.4430000000000001},"min":{"1min":0.98499999999999999,"5min":0.97999999999999998,"15min":0.81499999999999995},"max":{"1min":1.7909999999999999,"5min":2.4569999999999999,"15min":3.8330000000000002},"last":1.5620000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.367,"15min":1.347},"min":{"1min":0.89600000000000002,"5min":0.79600000000000004,"15min":0.77800000000000002},"max":{"1min":1.8129999999999999,"5min":1.9199999999999999,"15min":2.98},"last":1.298},{"interface":"front","average":{"1min":1.397,"5min":1.389,"15min":1.369},"min":{"1min":0.90400000000000003,"5min":0.81899999999999995,"15min":0.57499999999999996},"max":{"1min":1.8220000000000001,"5min":2.0270000000000001,"15min":3.1280000000000001},"last":1.603}]},{"osd":6,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.417,"15min":1.405},"min":{"1min":1.085,"5min":0.82099999999999995,"15min":0.65400000000000003},"max":{"1min":1.849,"5min":1.9670000000000001,"15min":4.0540000000000003},"last":1.361},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.641,"15min":1.6100000000000001},"min":{"1min":1.2569999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7569999999999999,"5min":2.1850000000000001,"15min":3.6360000000000001},"last":1.6299999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.593,"15min":1.556},"min":{"1min":1.24,"5min":0.998,"15min":0.67700000000000005},"max":{"1min":1.8740000000000001,"5min":2.5899999999999999,"15min":3.2759999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.573,"15min":1.5289999999999999},"min":{"1min":1.212,"5min":1.131,"15min":0.69999999999999996},"max":{"1min":2.206,"5min":2.206,"15min":3.9390000000000001},"last":1.669}]}]},{"osd":4,"up_from":13,"seq":55834575974,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60124,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311716,"statfs":{"total":96636764160,"available":96575197184,"internally_reserved":0,"allocated":2633728,"data_stored":1294592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.212,"15min":1.2150000000000001},"min":{"1min":0.83199999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.643,"5min":1.7130000000000001,"15min":1.954},"last":0.628},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.3009999999999999},"min":{"1min":0.877,"5min":0.84399999999999997,"15min":0.76100000000000001},"max":{"1min":1.6879999999999999,"5min":2.1400000000000001,"15min":2.3140000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.339,"15min":1.3169999999999999},"min":{"1min":0.94299999999999995,"5min":0.89000000000000001,"15min":0.59399999999999997},"max":{"1min":1.7310000000000001,"5min":1.8440000000000001,"15min":1.929},"last":1.204},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.91100000000000003,"5min":0.74199999999999999,"15min":0.69299999999999995},"max":{"1min":1.6619999999999999,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":0.91300000000000003}]},{"osd":2,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":1.0640000000000001,"5min":0.80600000000000005,"15min":0.78000000000000003},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":3.524},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4610000000000001,"15min":1.448},"min":{"1min":1.0369999999999999,"5min":1.014,"15min":0.65700000000000003},"max":{"1min":1.9830000000000001,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.3640000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.80200000000000005},"max":{"1min":1.7430000000000001,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":1.034},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4399999999999999,"15min":1.4550000000000001},"min":{"1min":1.1579999999999999,"5min":1.0700000000000001,"15min":0.93700000000000006},"max":{"1min":1.8400000000000001,"5min":2.419,"15min":4.1600000000000001},"last":0.95299999999999996}]},{"osd":5,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4039999999999999,"15min":1.3899999999999999},"min":{"1min":0.95399999999999996,"5min":0.89400000000000002,"15min":0.69299999999999995},"max":{"1min":1.8620000000000001,"5min":2.02,"15min":2.02},"last":1.292},{"interface":"front","average":{"1min":1.288,"5min":1.363,"15min":1.379},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.64500000000000002},"max":{"1min":1.839,"5min":2.1030000000000002,"15min":2.1320000000000001},"last":1.1279999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.45,"15min":1.444},"min":{"1min":0.96499999999999997,"5min":0.94299999999999995,"15min":0.81799999999999995},"max":{"1min":1.823,"5min":2.8010000000000002,"15min":3.3610000000000002},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.5740000000000001,"15min":1.5720000000000001},"min":{"1min":1.2609999999999999,"5min":1.133,"15min":0.82699999999999996},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.3929999999999998},"last":1.3879999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.512,"15min":1.53},"min":{"1min":1.042,"5min":1.042,"15min":0.77600000000000002},"max":{"1min":1.7769999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.6100000000000001,"15min":1.611},"min":{"1min":1.2729999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1000000000000001,"5min":2.6280000000000001,"15min":3.8610000000000002},"last":1.5629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575974,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51796,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320044,"statfs":{"total":96636764160,"available":96583725056,"internally_reserved":0,"allocated":2629632,"data_stored":1292592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.82799999999999996,"5min":0.69699999999999995,"15min":0.53400000000000003},"max":{"1min":1.179,"5min":1.47,"15min":1.855},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.21,"15min":1.212},"min":{"1min":0.97599999999999998,"5min":0.85899999999999999,"15min":0.82699999999999996},"max":{"1min":1.506,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.0569999999999999,"15min":1.0249999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.59299999999999997},"max":{"1min":1.835,"5min":1.8939999999999999,"15min":1.919},"last":0.873},{"interface":"front","average":{"1min":1.266,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.96699999999999997,"5min":0.80600000000000005,"15min":0.46200000000000002},"max":{"1min":1.546,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.115}]},{"osd":2,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4990000000000001,"15min":1.4650000000000001},"min":{"1min":1.1870000000000001,"5min":1.0389999999999999,"15min":0.64700000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":1.042},{"interface":"front","average":{"1min":1.341,"5min":1.423,"15min":1.4019999999999999},"min":{"1min":0.84799999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.78,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.167}]},{"osd":4,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4970000000000001,"15min":1.448},"min":{"1min":1.284,"5min":1.0980000000000001,"15min":1.0209999999999999},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.464,"15min":1.482},"min":{"1min":1.149,"5min":0.998,"15min":0.747},"max":{"1min":1.756,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.897}]},{"osd":5,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.0589999999999999,"5min":1.0429999999999999,"15min":0.90500000000000003},"max":{"1min":1.931,"5min":2.476,"15min":2.476},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.6040000000000001,"15min":1.5740000000000001},"min":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":0.97799999999999998},"max":{"1min":2.0430000000000001,"5min":2.891,"15min":2.891},"last":1.5980000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.635,"15min":1.645},"min":{"1min":1.385,"5min":1.2250000000000001,"15min":0.97099999999999997},"max":{"1min":2.016,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":2.141},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.669,"15min":1.681},"min":{"1min":1.242,"5min":1.129,"15min":0.91000000000000003},"max":{"1min":2.2949999999999999,"5min":2.3069999999999999,"15min":2.585},"last":2.0419999999999998}]},{"osd":7,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.6140000000000001,"15min":1.6120000000000001},"min":{"1min":1.196,"5min":1.196,"15min":0.85499999999999998},"max":{"1min":2.0150000000000001,"5min":2.419,"15min":2.419},"last":1.534},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.732,"15min":1.7010000000000001},"min":{"1min":1.22,"5min":1.22,"15min":1.0780000000000001},"max":{"1min":2.5169999999999999,"5min":2.548,"15min":2.548},"last":1.5489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51804,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320036,"statfs":{"total":96636764160,"available":96583716864,"internally_reserved":0,"allocated":2629632,"data_stored":1293092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.92300000000000004},"min":{"1min":0.69299999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2949999999999999,"5min":1.819,"15min":1.99},"last":0.499},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.137},"min":{"1min":0.85399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4550000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.438}]},{"osd":1,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1,"15min":1.034},"min":{"1min":0.79500000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.323,"5min":1.724,"15min":1.881},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.361,"5min":1.246,"15min":1.2509999999999999},"min":{"1min":0.89600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.982},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.2689999999999999,"15min":1.3129999999999999},"min":{"1min":0.95299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.74,"5min":1.74,"15min":1.9690000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.478,"5min":1.413,"15min":1.4019999999999999},"min":{"1min":1.024,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.897,"5min":2.165,"15min":2.165},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.46,"15min":1.4790000000000001},"min":{"1min":1.224,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.8859999999999999,"5min":2.0640000000000001,"15min":2.0859999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.357,"15min":1.3779999999999999},"min":{"1min":0.98999999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.613,"5min":2.024,"15min":2.024},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4530000000000001,"15min":1.4690000000000001},"min":{"1min":1.0700000000000001,"5min":0.996,"15min":0.996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.661,"5min":1.589,"15min":1.5900000000000001},"min":{"1min":1.216,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.1419999999999999,"5min":2.1549999999999998,"15min":2.1589999999999998},"last":0.79700000000000004}]},{"osd":6,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.536,"15min":1.536},"min":{"1min":1.222,"5min":1.018,"15min":1.018},"max":{"1min":2.0529999999999999,"5min":2.1150000000000002,"15min":2.1190000000000002},"last":0.747},{"interface":"front","average":{"1min":1.599,"5min":1.605,"15min":1.637},"min":{"1min":1.306,"5min":1.1970000000000001,"15min":1.0760000000000001},"max":{"1min":1.952,"5min":2.1469999999999998,"15min":2.3180000000000001},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.6100000000000001,"15min":1.6020000000000001},"min":{"1min":1.165,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.996,"5min":2.0070000000000001,"15min":2.2490000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.702,"5min":1.696,"15min":1.7030000000000001},"min":{"1min":1.145,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.3250000000000002},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51796,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320044,"statfs":{"total":96636764160,"available":96583725056,"internally_reserved":0,"allocated":2629632,"data_stored":1294010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.96899999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.264,"5min":1.647,"15min":2.4929999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.1579999999999999,"15min":1.157},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.7070000000000001,"5min":1.738,"15min":2.5590000000000002},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0700000000000001,"15min":1.0680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6759999999999999,"5min":1.6930000000000001,"15min":2.7360000000000002},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.238,"15min":1.2729999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.569,"5min":1.7549999999999999,"15min":2.6859999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.419,"15min":1.413},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.671,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":1.046},{"interface":"front","average":{"1min":1.272,"5min":1.292,"15min":1.319},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.7849999999999999,"5min":1.9079999999999999,"15min":2.823},"last":1.1339999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.448,"15min":1.4750000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.756,"5min":2.0750000000000002,"15min":2.4430000000000001},"last":1.421},{"interface":"front","average":{"1min":1.427,"5min":1.431,"15min":1.4550000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.0419999999999998,"5min":2.1080000000000001,"15min":2.899},"last":1.0609999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4930000000000001,"15min":1.486},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.9990000000000001,"5min":2.2559999999999998,"15min":3.004},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.585,"15min":1.581},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9399999999999999,"5min":2.3180000000000001,"15min":2.5129999999999999},"last":1.4590000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.613,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.931},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.621},"min":{"1min":1.139,"5min":1.0469999999999999,"15min":1.002},"max":{"1min":2.1240000000000001,"5min":2.444,"15min":3.1190000000000002},"last":1.4470000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.7070000000000001,"15min":1.71},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.0640000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.609},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.7110000000000001,"15min":1.73},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.165,"5min":2.3929999999999998,"15min":2.6739999999999999},"last":1.532}]}]},{"osd":0,"up_from":13,"seq":55834575973,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48216,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323624,"statfs":{"total":96636764160,"available":96587390976,"internally_reserved":0,"allocated":3215360,"data_stored":1882788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.97799999999999998,"15min":0.98199999999999998},"min":{"1min":0.72199999999999998,"5min":0.626,"15min":0.626},"max":{"1min":1.536,"5min":1.5629999999999999,"15min":2.6440000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1579999999999999,"15min":1.157},"min":{"1min":0.77600000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7490000000000001,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.94399999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2090000000000001,"15min":1.1950000000000001},"min":{"1min":0.91500000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.784,"5min":1.8520000000000001,"15min":1.8640000000000001},"last":1.222},{"interface":"front","average":{"1min":1.198,"5min":1.2829999999999999,"15min":1.2729999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6359999999999999,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":0.77500000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4179999999999999,"15min":1.399},"min":{"1min":1.032,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.804,"5min":1.8440000000000001,"15min":2.0720000000000001},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.397,"15min":1.419},"min":{"1min":1.0649999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6539999999999999,"5min":2.069,"15min":2.194},"last":1.2949999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3029999999999999,"15min":1.3260000000000001},"min":{"1min":0.95899999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":1.7190000000000001,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5329999999999999,"15min":1.522},"min":{"1min":1.2989999999999999,"5min":1.008,"15min":1.008},"max":{"1min":2.0670000000000002,"5min":2.0899999999999999,"15min":2.4889999999999999},"last":1.4490000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.401,"15min":1.4019999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98199999999999998},"max":{"1min":1.7729999999999999,"5min":2.0830000000000002,"15min":2.1339999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.581,"15min":1.597},"min":{"1min":1.163,"5min":1.149,"15min":1.149},"max":{"1min":1.883,"5min":2.0489999999999999,"15min":2.2879999999999998},"last":1.1559999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.641,"15min":1.667},"min":{"1min":1.46,"5min":1.21,"15min":1.1379999999999999},"max":{"1min":1.867,"5min":2.0910000000000002,"15min":2.3969999999999998},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.514,"15min":1.524},"min":{"1min":1.1180000000000001,"5min":0.999,"15min":0.999},"max":{"1min":1.853,"5min":1.8979999999999999,"15min":2.2799999999999998},"last":1.4019999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.6719999999999999,"15min":1.671},"min":{"1min":1.3440000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.2120000000000002,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.6479999999999999,"15min":1.647},"min":{"1min":1.2709999999999999,"5min":1.2470000000000001,"15min":1.113},"max":{"1min":1.9610000000000001,"5min":2.1440000000000001,"15min":2.3620000000000001},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:14:51.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:14:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.402+0000 7f5f23e89640 1 Processor -- start 2023-10-14T18:14:51.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.403+0000 7f5f23e89640 1 -- start start 2023-10-14T18:14:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f23e89640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5f1c159b20 0x7f5f1c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f23e89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f1c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f23e89640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f1c06f7f0 con 0x7f5f1c159f00 2023-10-14T18:14:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f23e89640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f1c06c5e0 con 0x7f5f1c159b20 2023-10-14T18:14:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f23e89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c06c8c0 con 0x7f5f1c0b0b20 2023-10-14T18:14:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f21424640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f1c159f00 0x7f5f1c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58114/0 (socket says 172.21.15.29:58114) 2023-10-14T18:14:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.404+0000 7f5f21424640 1 -- 172.21.15.29:0/3203375542 learned_addr learned my addr 172.21.15.29:0/3203375542 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f22426640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f1c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f21c25640 1 --2- 172.21.15.29:0/3203375542 >> v2:172.21.15.186:3300/0 conn(0x7f5f1c159b20 0x7f5f1c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627953627 0 0) 0x7f5f1c06f7f0 con 0x7f5f1c159f00 2023-10-14T18:14:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f08003520 con 0x7f5f1c159f00 2023-10-14T18:14:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1322186039 0 0) 0x7f5f08003520 con 0x7f5f1c159f00 2023-10-14T18:14:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 msgr2=0x7f5f1c06fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f1c06fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 >> v2:172.21.15.186:3300/0 conn(0x7f5f1c159b20 msgr2=0x7f5f1c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 --2- 172.21.15.29:0/3203375542 >> v2:172.21.15.186:3300/0 conn(0x7f5f1c159b20 0x7f5f1c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c06cba0 con 0x7f5f1c159f00 2023-10-14T18:14:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5f0c001670 con 0x7f5f1c159f00 2023-10-14T18:14:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 >> v1:172.21.15.29:6789/0 conn(0x7f5f1c159f00 legacy=0x7f5f1c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f08003970 0x7f5f08003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f080047c0 0x7f5f08004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f08005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c06c5e0 con 0x7f5f080047c0 2023-10-14T18:14:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c06c8c0 con 0x7f5f08003970 2023-10-14T18:14:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.405+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0800b980 con 0x7f5f1c0b0b20 2023-10-14T18:14:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f080047c0 0x7f5f08004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f22426640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f08005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 -- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 msgr2=0x7f5f08005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21c25640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f08003970 0x7f5f08003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f1c0b0b20 0x7f5f08005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 -- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f08003970 msgr2=0x7f5f08003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f08003970 0x7f5f08003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.406+0000 7f5f21424640 1 -- 172.21.15.29:0/3203375542 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f08003520 con 0x7f5f080047c0 2023-10-14T18:14:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.408+0000 7f5f21424640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f080047c0 0x7f5f08004c40 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c210 tx=0x7f5f080057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43e0126c4833e370 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.408+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c005ad0 con 0x7f5f080047c0 2023-10-14T18:14:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.408+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c005c30 con 0x7f5f080047c0 2023-10-14T18:14:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.408+0000 7f5f20c23640 1 -- 172.21.15.29:0/3203375542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c02c8d0 con 0x7f5f080047c0 2023-10-14T18:14:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 -- 172.21.15.29:0/3203375542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f080047c0 msgr2=0x7f5f08004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 --2- 172.21.15.29:0/3203375542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f080047c0 0x7f5f08004c40 secure :-1 s=READY pgs=2388 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c210 tx=0x7f5f080057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f22426640 1 -- 172.21.15.29:0/3203375542 reap_dead start 2023-10-14T18:14:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 -- 172.21.15.29:0/3203375542 shutdown_connections 2023-10-14T18:14:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 -- 172.21.15.29:0/3203375542 >> 172.21.15.29:0/3203375542 conn(0x7f5f1c0cb410 msgr2=0x7f5f1c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 -- 172.21.15.29:0/3203375542 shutdown_connections 2023-10-14T18:14:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.409+0000 7f5f23e89640 1 -- 172.21.15.29:0/3203375542 wait complete. 2023-10-14T18:14:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.411+0000 7f5f23e89640 1 Processor -- start 2023-10-14T18:14:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.411+0000 7f5f23e89640 1 -- start start 2023-10-14T18:14:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 0x7f5f1c14e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f1c0c6030 0x7f5f1c0c64b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c0b3d90 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c0b1ce0 con 0x7f5f1c0c6030 2023-10-14T18:14:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f23e89640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1c0b2ab0 con 0x7f5f080047c0 2023-10-14T18:14:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f21424640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f21424640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36448/0 (socket says 172.21.15.29:36448) 2023-10-14T18:14:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f21424640 1 -- 172.21.15.29:0/2604770214 learned_addr learned my addr 172.21.15.29:0/2604770214 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.412+0000 7f5f21c25640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 0x7f5f1c14e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 -- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 msgr2=0x7f5f1c14e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 0x7f5f1c14e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f22426640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f1c0c6030 0x7f5f1c0c64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 -- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f1c0c6030 msgr2=0x7f5f1c0c64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f1c0c6030 0x7f5f1c0c64b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 -- 172.21.15.29:0/2604770214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c0b1f30 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21c25640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 0x7f5f1c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:51.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f21424640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c210 tx=0x7f5f0c001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c02cce0 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f0c02ce40 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f0c007270 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f1c0b2210 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.413+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f1c0b27d0 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.415+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f1c04e160 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.416+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5f0c032070 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.417+0000 7f5f127fc640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 0x7f5f0006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.417+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5f0c0aa8b0 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.417+0000 7f5f21c25640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 0x7f5f0006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.418+0000 7f5f21c25640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 0x7f5f0006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007460 tx=0x7f5f1800b040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.422+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f0c075870 con 0x7f5f1c0b33d0 2023-10-14T18:14:51.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.619+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 --> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5f1c06ce80 con 0x7f5f0006af90 2023-10-14T18:14:51.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.635+0000 7f5f127fc640 1 -- 172.21.15.29:0/2604770214 <== mgr.10482 v2:172.21.15.29:6832/2339165088 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+286657 (secure 0 0 0) 0x7f5f1800f040 con 0x7f5f0006af90 2023-10-14T18:14:51.638 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:51.641 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:14:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.641+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 msgr2=0x7f5f0006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.641+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 0x7f5f0006d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f5f18007460 tx=0x7f5f1800b040 comp rx=0 tx=0).stop 2023-10-14T18:14:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.641+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 msgr2=0x7f5f1c0b3850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.641+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 secure :-1 s=READY pgs=2389 cs=0 l=1 rev1=1 crypto rx=0x7f5f0c00c210 tx=0x7f5f0c001a00 comp rx=0 tx=0).stop 2023-10-14T18:14:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 shutdown_connections 2023-10-14T18:14:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f1c0c6030 0x7f5f1c0c64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f1c0b33d0 0x7f5f1c0b3850 unknown :-1 s=CLOSED pgs=2389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f5f0006af90 0x7f5f0006d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 --2- 172.21.15.29:0/2604770214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f080047c0 0x7f5f1c14e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 >> 172.21.15.29:0/2604770214 conn(0x7f5f1c0cb410 msgr2=0x7f5f1c0ccfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 shutdown_connections 2023-10-14T18:14:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.642+0000 7f5f23e89640 1 -- 172.21.15.29:0/2604770214 wait complete. 2023-10-14T18:14:51.663 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2023-10-14T18:14:50.952143+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":583,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":364,"ondisk_log_size":364,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":162,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":387740,"kb_used_data":21672,"kb_used_omap":12,"kb_used_meta":365939,"kb_avail":754586980,"statfs":{"total":773094113280,"available":772697067520,"internally_reserved":0,"allocated":22192128,"data_stored":11524740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":374722104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":32,"apply_latency_ms":32,"commit_latency_ns":32000000,"apply_latency_ns":32000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001915"},"pg_stats":[{"pgid":"25.3f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572757+0000","last_change":"2023-10-14T18:14:46.570972+0000","last_active":"2023-10-14T18:14:46.572757+0000","last_peered":"2023-10-14T18:14:46.572757+0000","last_clean":"2023-10-14T18:14:46.572757+0000","last_became_active":"2023-10-14T18:14:46.570627+0000","last_became_peered":"2023-10-14T18:14:46.570627+0000","last_unstale":"2023-10-14T18:14:46.572757+0000","last_undegraded":"2023-10-14T18:14:46.572757+0000","last_fullsized":"2023-10-14T18:14:46.572757+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:15.964586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.3e","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562978+0000","last_change":"2023-10-14T18:14:46.550819+0000","last_active":"2023-10-14T18:14:47.562978+0000","last_peered":"2023-10-14T18:14:47.562978+0000","last_clean":"2023-10-14T18:14:47.562978+0000","last_became_active":"2023-10-14T18:14:46.549960+0000","last_became_peered":"2023-10-14T18:14:46.549960+0000","last_unstale":"2023-10-14T18:14:47.562978+0000","last_undegraded":"2023-10-14T18:14:47.562978+0000","last_fullsized":"2023-10-14T18:14:47.562978+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568687+0000","last_change":"2023-10-14T18:14:46.562799+0000","last_active":"2023-10-14T18:14:46.568687+0000","last_peered":"2023-10-14T18:14:46.568687+0000","last_clean":"2023-10-14T18:14:46.568687+0000","last_became_active":"2023-10-14T18:14:46.562528+0000","last_became_peered":"2023-10-14T18:14:46.562528+0000","last_unstale":"2023-10-14T18:14:46.568687+0000","last_undegraded":"2023-10-14T18:14:46.568687+0000","last_fullsized":"2023-10-14T18:14:46.568687+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:29:23.117570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.3c","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561807+0000","last_change":"2023-10-14T18:14:46.548946+0000","last_active":"2023-10-14T18:14:47.561807+0000","last_peered":"2023-10-14T18:14:47.561807+0000","last_clean":"2023-10-14T18:14:47.561807+0000","last_became_active":"2023-10-14T18:14:46.548619+0000","last_became_peered":"2023-10-14T18:14:46.548619+0000","last_unstale":"2023-10-14T18:14:47.561807+0000","last_undegraded":"2023-10-14T18:14:47.561807+0000","last_fullsized":"2023-10-14T18:14:47.561807+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.3b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568730+0000","last_change":"2023-10-14T18:14:46.566502+0000","last_active":"2023-10-14T18:14:46.568730+0000","last_peered":"2023-10-14T18:14:46.568730+0000","last_clean":"2023-10-14T18:14:46.568730+0000","last_became_active":"2023-10-14T18:14:46.563380+0000","last_became_peered":"2023-10-14T18:14:46.563380+0000","last_unstale":"2023-10-14T18:14:46.568730+0000","last_undegraded":"2023-10-14T18:14:46.568730+0000","last_fullsized":"2023-10-14T18:14:46.568730+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.3a","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574190+0000","last_change":"2023-10-14T18:14:46.567070+0000","last_active":"2023-10-14T18:14:46.574190+0000","last_peered":"2023-10-14T18:14:46.574190+0000","last_clean":"2023-10-14T18:14:46.574190+0000","last_became_active":"2023-10-14T18:14:46.566790+0000","last_became_peered":"2023-10-14T18:14:46.566790+0000","last_unstale":"2023-10-14T18:14:46.574190+0000","last_undegraded":"2023-10-14T18:14:46.574190+0000","last_fullsized":"2023-10-14T18:14:46.574190+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:06:48.712073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.39","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.558923+0000","last_change":"2023-10-14T18:14:46.555506+0000","last_active":"2023-10-14T18:14:47.558923+0000","last_peered":"2023-10-14T18:14:47.558923+0000","last_clean":"2023-10-14T18:14:47.558923+0000","last_became_active":"2023-10-14T18:14:46.553933+0000","last_became_peered":"2023-10-14T18:14:46.553933+0000","last_unstale":"2023-10-14T18:14:47.558923+0000","last_undegraded":"2023-10-14T18:14:47.558923+0000","last_fullsized":"2023-10-14T18:14:47.558923+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:36.892049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.38","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.558184+0000","last_change":"2023-10-14T18:14:46.551246+0000","last_active":"2023-10-14T18:14:46.558184+0000","last_peered":"2023-10-14T18:14:46.558184+0000","last_clean":"2023-10-14T18:14:46.558184+0000","last_became_active":"2023-10-14T18:14:46.550781+0000","last_became_peered":"2023-10-14T18:14:46.550781+0000","last_unstale":"2023-10-14T18:14:46.558184+0000","last_undegraded":"2023-10-14T18:14:46.558184+0000","last_fullsized":"2023-10-14T18:14:46.558184+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:05:15.964586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.37","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568683+0000","last_change":"2023-10-14T18:14:46.561054+0000","last_active":"2023-10-14T18:14:46.568683+0000","last_peered":"2023-10-14T18:14:46.568683+0000","last_clean":"2023-10-14T18:14:46.568683+0000","last_became_active":"2023-10-14T18:14:46.560753+0000","last_became_peered":"2023-10-14T18:14:46.560753+0000","last_unstale":"2023-10-14T18:14:46.568683+0000","last_undegraded":"2023-10-14T18:14:46.568683+0000","last_fullsized":"2023-10-14T18:14:46.568683+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:32.135916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.36","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557196+0000","last_change":"2023-10-14T18:14:46.559226+0000","last_active":"2023-10-14T18:14:47.557196+0000","last_peered":"2023-10-14T18:14:47.557196+0000","last_clean":"2023-10-14T18:14:47.557196+0000","last_became_active":"2023-10-14T18:14:46.551377+0000","last_became_peered":"2023-10-14T18:14:46.551377+0000","last_unstale":"2023-10-14T18:14:47.557196+0000","last_undegraded":"2023-10-14T18:14:47.557196+0000","last_fullsized":"2023-10-14T18:14:47.557196+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:32.135916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.35","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560087+0000","last_change":"2023-10-14T18:14:46.570127+0000","last_active":"2023-10-14T18:14:47.560087+0000","last_peered":"2023-10-14T18:14:47.560087+0000","last_clean":"2023-10-14T18:14:47.560087+0000","last_became_active":"2023-10-14T18:14:46.569846+0000","last_became_peered":"2023-10-14T18:14:46.569846+0000","last_unstale":"2023-10-14T18:14:47.560087+0000","last_undegraded":"2023-10-14T18:14:47.560087+0000","last_fullsized":"2023-10-14T18:14:47.560087+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:38.286925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.34","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562301+0000","last_change":"2023-10-14T18:14:46.561397+0000","last_active":"2023-10-14T18:14:47.562301+0000","last_peered":"2023-10-14T18:14:47.562301+0000","last_clean":"2023-10-14T18:14:47.562301+0000","last_became_active":"2023-10-14T18:14:46.561053+0000","last_became_peered":"2023-10-14T18:14:46.561053+0000","last_unstale":"2023-10-14T18:14:47.562301+0000","last_undegraded":"2023-10-14T18:14:47.562301+0000","last_fullsized":"2023-10-14T18:14:47.562301+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.33","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561682+0000","last_change":"2023-10-14T18:14:46.557710+0000","last_active":"2023-10-14T18:14:47.561682+0000","last_peered":"2023-10-14T18:14:47.561682+0000","last_clean":"2023-10-14T18:14:47.561682+0000","last_became_active":"2023-10-14T18:14:46.557395+0000","last_became_peered":"2023-10-14T18:14:46.557395+0000","last_unstale":"2023-10-14T18:14:47.561682+0000","last_undegraded":"2023-10-14T18:14:47.561682+0000","last_fullsized":"2023-10-14T18:14:47.561682+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.32","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570225+0000","last_change":"2023-10-14T18:14:46.564724+0000","last_active":"2023-10-14T18:14:46.570225+0000","last_peered":"2023-10-14T18:14:46.570225+0000","last_clean":"2023-10-14T18:14:46.570225+0000","last_became_active":"2023-10-14T18:14:46.564423+0000","last_became_peered":"2023-10-14T18:14:46.564423+0000","last_unstale":"2023-10-14T18:14:46.570225+0000","last_undegraded":"2023-10-14T18:14:46.570225+0000","last_fullsized":"2023-10-14T18:14:46.570225+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.31","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561722+0000","last_change":"2023-10-14T18:14:46.561306+0000","last_active":"2023-10-14T18:14:47.561722+0000","last_peered":"2023-10-14T18:14:47.561722+0000","last_clean":"2023-10-14T18:14:47.561722+0000","last_became_active":"2023-10-14T18:14:46.561000+0000","last_became_peered":"2023-10-14T18:14:46.561000+0000","last_unstale":"2023-10-14T18:14:47.561722+0000","last_undegraded":"2023-10-14T18:14:47.561722+0000","last_fullsized":"2023-10-14T18:14:47.561722+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.30","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568513+0000","last_change":"2023-10-14T18:14:46.555284+0000","last_active":"2023-10-14T18:14:46.568513+0000","last_peered":"2023-10-14T18:14:46.568513+0000","last_clean":"2023-10-14T18:14:46.568513+0000","last_became_active":"2023-10-14T18:14:46.553906+0000","last_became_peered":"2023-10-14T18:14:46.553906+0000","last_unstale":"2023-10-14T18:14:46.568513+0000","last_undegraded":"2023-10-14T18:14:46.568513+0000","last_fullsized":"2023-10-14T18:14:46.568513+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:32:18.970031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570430+0000","last_change":"2023-10-14T18:14:46.566962+0000","last_active":"2023-10-14T18:14:46.570430+0000","last_peered":"2023-10-14T18:14:46.570430+0000","last_clean":"2023-10-14T18:14:46.570430+0000","last_became_active":"2023-10-14T18:14:46.563574+0000","last_became_peered":"2023-10-14T18:14:46.563574+0000","last_unstale":"2023-10-14T18:14:46.570430+0000","last_undegraded":"2023-10-14T18:14:46.570430+0000","last_fullsized":"2023-10-14T18:14:46.570430+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:35.547370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2e","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572193+0000","last_change":"2023-10-14T18:14:46.567376+0000","last_active":"2023-10-14T18:14:46.572193+0000","last_peered":"2023-10-14T18:14:46.572193+0000","last_clean":"2023-10-14T18:14:46.572193+0000","last_became_active":"2023-10-14T18:14:46.567058+0000","last_became_peered":"2023-10-14T18:14:46.567058+0000","last_unstale":"2023-10-14T18:14:46.572193+0000","last_undegraded":"2023-10-14T18:14:46.572193+0000","last_fullsized":"2023-10-14T18:14:46.572193+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:21.418564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.2d","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556802+0000","last_change":"2023-10-14T18:14:46.566674+0000","last_active":"2023-10-14T18:14:47.556802+0000","last_peered":"2023-10-14T18:14:47.556802+0000","last_clean":"2023-10-14T18:14:47.556802+0000","last_became_active":"2023-10-14T18:14:46.566337+0000","last_became_peered":"2023-10-14T18:14:46.566337+0000","last_unstale":"2023-10-14T18:14:47.556802+0000","last_undegraded":"2023-10-14T18:14:47.556802+0000","last_fullsized":"2023-10-14T18:14:47.556802+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.2c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568430+0000","last_change":"2023-10-14T18:14:46.562615+0000","last_active":"2023-10-14T18:14:46.568430+0000","last_peered":"2023-10-14T18:14:46.568430+0000","last_clean":"2023-10-14T18:14:46.568430+0000","last_became_active":"2023-10-14T18:14:46.562098+0000","last_became_peered":"2023-10-14T18:14:46.562098+0000","last_unstale":"2023-10-14T18:14:46.568430+0000","last_undegraded":"2023-10-14T18:14:46.568430+0000","last_fullsized":"2023-10-14T18:14:46.568430+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:27.757684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571294+0000","last_change":"2023-10-14T18:14:46.567400+0000","last_active":"2023-10-14T18:14:46.571294+0000","last_peered":"2023-10-14T18:14:46.571294+0000","last_clean":"2023-10-14T18:14:46.571294+0000","last_became_active":"2023-10-14T18:14:46.566651+0000","last_became_peered":"2023-10-14T18:14:46.566651+0000","last_unstale":"2023-10-14T18:14:46.571294+0000","last_undegraded":"2023-10-14T18:14:46.571294+0000","last_fullsized":"2023-10-14T18:14:46.571294+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.2a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556170+0000","last_change":"2023-10-14T18:14:46.551965+0000","last_active":"2023-10-14T18:14:47.556170+0000","last_peered":"2023-10-14T18:14:47.556170+0000","last_clean":"2023-10-14T18:14:47.556170+0000","last_became_active":"2023-10-14T18:14:46.551660+0000","last_became_peered":"2023-10-14T18:14:46.551660+0000","last_unstale":"2023-10-14T18:14:47.556170+0000","last_undegraded":"2023-10-14T18:14:47.556170+0000","last_fullsized":"2023-10-14T18:14:47.556170+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.29","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571674+0000","last_change":"2023-10-14T18:14:46.564391+0000","last_active":"2023-10-14T18:14:46.571674+0000","last_peered":"2023-10-14T18:14:46.571674+0000","last_clean":"2023-10-14T18:14:46.571674+0000","last_became_active":"2023-10-14T18:14:46.564112+0000","last_became_peered":"2023-10-14T18:14:46.564112+0000","last_unstale":"2023-10-14T18:14:46.571674+0000","last_undegraded":"2023-10-14T18:14:46.571674+0000","last_fullsized":"2023-10-14T18:14:46.571674+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:54.019208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.28","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.558165+0000","last_change":"2023-10-14T18:14:46.551071+0000","last_active":"2023-10-14T18:14:46.558165+0000","last_peered":"2023-10-14T18:14:46.558165+0000","last_clean":"2023-10-14T18:14:46.558165+0000","last_became_active":"2023-10-14T18:14:46.550400+0000","last_became_peered":"2023-10-14T18:14:46.550400+0000","last_unstale":"2023-10-14T18:14:46.558165+0000","last_undegraded":"2023-10-14T18:14:46.558165+0000","last_fullsized":"2023-10-14T18:14:46.558165+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:35:40.069324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.27","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.563047+0000","last_change":"2023-10-14T18:14:46.552964+0000","last_active":"2023-10-14T18:14:47.563047+0000","last_peered":"2023-10-14T18:14:47.563047+0000","last_clean":"2023-10-14T18:14:47.563047+0000","last_became_active":"2023-10-14T18:14:46.552661+0000","last_became_peered":"2023-10-14T18:14:46.552661+0000","last_unstale":"2023-10-14T18:14:47.563047+0000","last_undegraded":"2023-10-14T18:14:47.563047+0000","last_fullsized":"2023-10-14T18:14:47.563047+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.26","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560482+0000","last_change":"2023-10-14T18:14:46.558308+0000","last_active":"2023-10-14T18:14:47.560482+0000","last_peered":"2023-10-14T18:14:47.560482+0000","last_clean":"2023-10-14T18:14:47.560482+0000","last_became_active":"2023-10-14T18:14:46.553562+0000","last_became_peered":"2023-10-14T18:14:46.553562+0000","last_unstale":"2023-10-14T18:14:47.560482+0000","last_undegraded":"2023-10-14T18:14:47.560482+0000","last_fullsized":"2023-10-14T18:14:47.560482+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:34.657113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.25","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574203+0000","last_change":"2023-10-14T18:14:46.572915+0000","last_active":"2023-10-14T18:14:46.574203+0000","last_peered":"2023-10-14T18:14:46.574203+0000","last_clean":"2023-10-14T18:14:46.574203+0000","last_became_active":"2023-10-14T18:14:46.572618+0000","last_became_peered":"2023-10-14T18:14:46.572618+0000","last_unstale":"2023-10-14T18:14:46.574203+0000","last_undegraded":"2023-10-14T18:14:46.574203+0000","last_fullsized":"2023-10-14T18:14:46.574203+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:09:20.544825+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.24","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.554232+0000","last_change":"2023-10-14T18:14:46.550445+0000","last_active":"2023-10-14T18:14:46.554232+0000","last_peered":"2023-10-14T18:14:46.554232+0000","last_clean":"2023-10-14T18:14:46.554232+0000","last_became_active":"2023-10-14T18:14:46.550155+0000","last_became_peered":"2023-10-14T18:14:46.550155+0000","last_unstale":"2023-10-14T18:14:46.554232+0000","last_undegraded":"2023-10-14T18:14:46.554232+0000","last_fullsized":"2023-10-14T18:14:46.554232+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.23","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.561681+0000","last_change":"2023-10-14T18:14:46.561854+0000","last_active":"2023-10-14T18:14:47.561681+0000","last_peered":"2023-10-14T18:14:47.561681+0000","last_clean":"2023-10-14T18:14:47.561681+0000","last_became_active":"2023-10-14T18:14:46.561579+0000","last_became_peered":"2023-10-14T18:14:46.561579+0000","last_unstale":"2023-10-14T18:14:47.561681+0000","last_undegraded":"2023-10-14T18:14:47.561681+0000","last_fullsized":"2023-10-14T18:14:47.561681+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:15.737925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.22","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566153+0000","last_change":"2023-10-14T18:14:46.559624+0000","last_active":"2023-10-14T18:14:46.566153+0000","last_peered":"2023-10-14T18:14:46.566153+0000","last_clean":"2023-10-14T18:14:46.566153+0000","last_became_active":"2023-10-14T18:14:46.559314+0000","last_became_peered":"2023-10-14T18:14:46.559314+0000","last_unstale":"2023-10-14T18:14:46.566153+0000","last_undegraded":"2023-10-14T18:14:46.566153+0000","last_fullsized":"2023-10-14T18:14:46.566153+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.21","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.569808+0000","last_change":"2023-10-14T18:14:46.555509+0000","last_active":"2023-10-14T18:14:46.569808+0000","last_peered":"2023-10-14T18:14:46.569808+0000","last_clean":"2023-10-14T18:14:46.569808+0000","last_became_active":"2023-10-14T18:14:46.555204+0000","last_became_peered":"2023-10-14T18:14:46.555204+0000","last_unstale":"2023-10-14T18:14:46.569808+0000","last_undegraded":"2023-10-14T18:14:46.569808+0000","last_fullsized":"2023-10-14T18:14:46.569808+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:38.286925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.20","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567143+0000","last_change":"2023-10-14T18:14:46.559267+0000","last_active":"2023-10-14T18:14:46.567143+0000","last_peered":"2023-10-14T18:14:46.567143+0000","last_clean":"2023-10-14T18:14:46.567143+0000","last_became_active":"2023-10-14T18:14:46.558982+0000","last_became_peered":"2023-10-14T18:14:46.558982+0000","last_unstale":"2023-10-14T18:14:46.567143+0000","last_undegraded":"2023-10-14T18:14:46.567143+0000","last_fullsized":"2023-10-14T18:14:46.567143+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:03.175998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1e","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573022+0000","last_change":"2023-10-14T18:14:46.563281+0000","last_active":"2023-10-14T18:14:46.573022+0000","last_peered":"2023-10-14T18:14:46.573022+0000","last_clean":"2023-10-14T18:14:46.573022+0000","last_became_active":"2023-10-14T18:14:46.562991+0000","last_became_peered":"2023-10-14T18:14:46.562991+0000","last_unstale":"2023-10-14T18:14:46.573022+0000","last_undegraded":"2023-10-14T18:14:46.573022+0000","last_fullsized":"2023-10-14T18:14:46.573022+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:51:34.504377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1f","version":"209'4","reported_seq":15,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572114+0000","last_change":"2023-10-14T18:14:45.556082+0000","last_active":"2023-10-14T18:14:46.572114+0000","last_peered":"2023-10-14T18:14:46.572114+0000","last_clean":"2023-10-14T18:14:46.572114+0000","last_became_active":"2023-10-14T18:14:45.555619+0000","last_became_peered":"2023-10-14T18:14:45.555619+0000","last_unstale":"2023-10-14T18:14:46.572114+0000","last_undegraded":"2023-10-14T18:14:46.572114+0000","last_fullsized":"2023-10-14T18:14:46.572114+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:30:29.198245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1f","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557314+0000","last_change":"2023-10-14T18:14:46.564731+0000","last_active":"2023-10-14T18:14:47.557314+0000","last_peered":"2023-10-14T18:14:47.557314+0000","last_clean":"2023-10-14T18:14:47.557314+0000","last_became_active":"2023-10-14T18:14:46.562496+0000","last_became_peered":"2023-10-14T18:14:46.562496+0000","last_unstale":"2023-10-14T18:14:47.557314+0000","last_undegraded":"2023-10-14T18:14:47.557314+0000","last_fullsized":"2023-10-14T18:14:47.557314+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:35.547370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1e","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566476+0000","last_change":"2023-10-14T18:14:45.552350+0000","last_active":"2023-10-14T18:14:46.566476+0000","last_peered":"2023-10-14T18:14:46.566476+0000","last_clean":"2023-10-14T18:14:46.566476+0000","last_became_active":"2023-10-14T18:14:45.551987+0000","last_became_peered":"2023-10-14T18:14:45.551987+0000","last_unstale":"2023-10-14T18:14:46.566476+0000","last_undegraded":"2023-10-14T18:14:46.566476+0000","last_fullsized":"2023-10-14T18:14:46.566476+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:50:34.529092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.1c","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559453+0000","last_change":"2023-10-14T18:14:46.556157+0000","last_active":"2023-10-14T18:14:47.559453+0000","last_peered":"2023-10-14T18:14:47.559453+0000","last_clean":"2023-10-14T18:14:47.559453+0000","last_became_active":"2023-10-14T18:14:46.555405+0000","last_became_peered":"2023-10-14T18:14:46.555405+0000","last_unstale":"2023-10-14T18:14:47.559453+0000","last_undegraded":"2023-10-14T18:14:47.559453+0000","last_fullsized":"2023-10-14T18:14:47.559453+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:54.019208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.1d","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573304+0000","last_change":"2023-10-14T18:14:45.557612+0000","last_active":"2023-10-14T18:14:46.573304+0000","last_peered":"2023-10-14T18:14:46.573304+0000","last_clean":"2023-10-14T18:14:46.573304+0000","last_became_active":"2023-10-14T18:14:45.556806+0000","last_became_peered":"2023-10-14T18:14:45.556806+0000","last_unstale":"2023-10-14T18:14:46.573304+0000","last_undegraded":"2023-10-14T18:14:46.573304+0000","last_fullsized":"2023-10-14T18:14:46.573304+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:05:12.146857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.1d","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570103+0000","last_change":"2023-10-14T18:14:46.564242+0000","last_active":"2023-10-14T18:14:46.570103+0000","last_peered":"2023-10-14T18:14:46.570103+0000","last_clean":"2023-10-14T18:14:46.570103+0000","last_became_active":"2023-10-14T18:14:46.561991+0000","last_became_peered":"2023-10-14T18:14:46.561991+0000","last_unstale":"2023-10-14T18:14:46.570103+0000","last_undegraded":"2023-10-14T18:14:46.570103+0000","last_fullsized":"2023-10-14T18:14:46.570103+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:20:00.998736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.1c","version":"209'2","reported_seq":14,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562672+0000","last_change":"2023-10-14T18:14:45.545550+0000","last_active":"2023-10-14T18:14:47.562672+0000","last_peered":"2023-10-14T18:14:47.562672+0000","last_clean":"2023-10-14T18:14:47.562672+0000","last_became_active":"2023-10-14T18:14:45.545277+0000","last_became_peered":"2023-10-14T18:14:45.545277+0000","last_unstale":"2023-10-14T18:14:47.562672+0000","last_undegraded":"2023-10-14T18:14:47.562672+0000","last_fullsized":"2023-10-14T18:14:47.562672+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T06:08:00.801720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556334+0000","last_change":"2023-10-14T18:14:46.544726+0000","last_active":"2023-10-14T18:14:47.556334+0000","last_peered":"2023-10-14T18:14:47.556334+0000","last_clean":"2023-10-14T18:14:47.556334+0000","last_became_active":"2023-10-14T18:14:46.544491+0000","last_became_peered":"2023-10-14T18:14:46.544491+0000","last_unstale":"2023-10-14T18:14:47.556334+0000","last_undegraded":"2023-10-14T18:14:47.556334+0000","last_fullsized":"2023-10-14T18:14:47.556334+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:20.000596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.1b","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567801+0000","last_change":"2023-10-14T18:14:45.554173+0000","last_active":"2023-10-14T18:14:46.567801+0000","last_peered":"2023-10-14T18:14:46.567801+0000","last_clean":"2023-10-14T18:14:46.567801+0000","last_became_active":"2023-10-14T18:14:45.553826+0000","last_became_peered":"2023-10-14T18:14:45.553826+0000","last_unstale":"2023-10-14T18:14:46.567801+0000","last_undegraded":"2023-10-14T18:14:46.567801+0000","last_fullsized":"2023-10-14T18:14:46.567801+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:04.793139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.1b","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570045+0000","last_change":"2023-10-14T18:14:46.562776+0000","last_active":"2023-10-14T18:14:46.570045+0000","last_peered":"2023-10-14T18:14:46.570045+0000","last_clean":"2023-10-14T18:14:46.570045+0000","last_became_active":"2023-10-14T18:14:46.561896+0000","last_became_peered":"2023-10-14T18:14:46.561896+0000","last_unstale":"2023-10-14T18:14:46.570045+0000","last_undegraded":"2023-10-14T18:14:46.570045+0000","last_fullsized":"2023-10-14T18:14:46.570045+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:40:21.418564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.1a","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.569889+0000","last_change":"2023-10-14T18:14:45.545539+0000","last_active":"2023-10-14T18:14:46.569889+0000","last_peered":"2023-10-14T18:14:46.569889+0000","last_clean":"2023-10-14T18:14:46.569889+0000","last_became_active":"2023-10-14T18:14:45.545247+0000","last_became_peered":"2023-10-14T18:14:45.545247+0000","last_unstale":"2023-10-14T18:14:46.569889+0000","last_undegraded":"2023-10-14T18:14:46.569889+0000","last_fullsized":"2023-10-14T18:14:46.569889+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:03:00.466585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.19","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.558888+0000","last_change":"2023-10-14T18:14:46.566743+0000","last_active":"2023-10-14T18:14:47.558888+0000","last_peered":"2023-10-14T18:14:47.558888+0000","last_clean":"2023-10-14T18:14:47.558888+0000","last_became_active":"2023-10-14T18:14:46.566463+0000","last_became_peered":"2023-10-14T18:14:46.566463+0000","last_unstale":"2023-10-14T18:14:47.558888+0000","last_undegraded":"2023-10-14T18:14:47.558888+0000","last_fullsized":"2023-10-14T18:14:47.558888+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:52:56.131847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.18","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.552837+0000","last_change":"2023-10-14T18:14:45.550168+0000","last_active":"2023-10-14T18:14:46.552837+0000","last_peered":"2023-10-14T18:14:46.552837+0000","last_clean":"2023-10-14T18:14:46.552837+0000","last_became_active":"2023-10-14T18:14:45.547775+0000","last_became_peered":"2023-10-14T18:14:45.547775+0000","last_unstale":"2023-10-14T18:14:46.552837+0000","last_undegraded":"2023-10-14T18:14:46.552837+0000","last_fullsized":"2023-10-14T18:14:46.552837+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:57.839057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.16","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568723+0000","last_change":"2023-10-14T18:14:46.560399+0000","last_active":"2023-10-14T18:14:46.568723+0000","last_peered":"2023-10-14T18:14:46.568723+0000","last_clean":"2023-10-14T18:14:46.568723+0000","last_became_active":"2023-10-14T18:14:46.558636+0000","last_became_peered":"2023-10-14T18:14:46.558636+0000","last_unstale":"2023-10-14T18:14:46.568723+0000","last_undegraded":"2023-10-14T18:14:46.568723+0000","last_fullsized":"2023-10-14T18:14:46.568723+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:21.293245+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.17","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560222+0000","last_change":"2023-10-14T18:14:45.555234+0000","last_active":"2023-10-14T18:14:47.560222+0000","last_peered":"2023-10-14T18:14:47.560222+0000","last_clean":"2023-10-14T18:14:47.560222+0000","last_became_active":"2023-10-14T18:14:45.554937+0000","last_became_peered":"2023-10-14T18:14:45.554937+0000","last_unstale":"2023-10-14T18:14:47.560222+0000","last_undegraded":"2023-10-14T18:14:47.560222+0000","last_fullsized":"2023-10-14T18:14:47.560222+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.635610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.17","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553485+0000","last_change":"2023-10-14T18:14:46.545838+0000","last_active":"2023-10-14T18:14:46.553485+0000","last_peered":"2023-10-14T18:14:46.553485+0000","last_clean":"2023-10-14T18:14:46.553485+0000","last_became_active":"2023-10-14T18:14:46.545559+0000","last_became_peered":"2023-10-14T18:14:46.545559+0000","last_unstale":"2023-10-14T18:14:46.553485+0000","last_undegraded":"2023-10-14T18:14:46.553485+0000","last_fullsized":"2023-10-14T18:14:46.553485+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:50.178686+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.16","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.549187+0000","last_change":"2023-10-14T18:14:45.550279+0000","last_active":"2023-10-14T18:14:46.549187+0000","last_peered":"2023-10-14T18:14:46.549187+0000","last_clean":"2023-10-14T18:14:46.549187+0000","last_became_active":"2023-10-14T18:14:45.549992+0000","last_became_peered":"2023-10-14T18:14:45.549992+0000","last_unstale":"2023-10-14T18:14:46.549187+0000","last_undegraded":"2023-10-14T18:14:46.549187+0000","last_fullsized":"2023-10-14T18:14:46.549187+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:49.118266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.14","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559523+0000","last_change":"2023-10-14T18:14:46.570086+0000","last_active":"2023-10-14T18:14:47.559523+0000","last_peered":"2023-10-14T18:14:47.559523+0000","last_clean":"2023-10-14T18:14:47.559523+0000","last_became_active":"2023-10-14T18:14:46.567808+0000","last_became_peered":"2023-10-14T18:14:46.567808+0000","last_unstale":"2023-10-14T18:14:47.559523+0000","last_undegraded":"2023-10-14T18:14:47.559523+0000","last_fullsized":"2023-10-14T18:14:47.559523+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:19.158160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.15","version":"209'3","reported_seq":14,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.572769+0000","last_change":"2023-10-14T18:14:45.547108+0000","last_active":"2023-10-14T18:14:46.572769+0000","last_peered":"2023-10-14T18:14:46.572769+0000","last_clean":"2023-10-14T18:14:46.572769+0000","last_became_active":"2023-10-14T18:14:45.546782+0000","last_became_peered":"2023-10-14T18:14:45.546782+0000","last_unstale":"2023-10-14T18:14:46.572769+0000","last_undegraded":"2023-10-14T18:14:46.572769+0000","last_fullsized":"2023-10-14T18:14:46.572769+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:42.368847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.15","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557097+0000","last_change":"2023-10-14T18:14:46.545417+0000","last_active":"2023-10-14T18:14:47.557097+0000","last_peered":"2023-10-14T18:14:47.557097+0000","last_clean":"2023-10-14T18:14:47.557097+0000","last_became_active":"2023-10-14T18:14:46.544355+0000","last_became_peered":"2023-10-14T18:14:46.544355+0000","last_unstale":"2023-10-14T18:14:47.557097+0000","last_undegraded":"2023-10-14T18:14:47.557097+0000","last_fullsized":"2023-10-14T18:14:47.557097+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:27.757684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.14","version":"209'6","reported_seq":17,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.575877+0000","last_change":"2023-10-14T18:14:45.556361+0000","last_active":"2023-10-14T18:14:46.575877+0000","last_peered":"2023-10-14T18:14:46.575877+0000","last_clean":"2023-10-14T18:14:46.575877+0000","last_became_active":"2023-10-14T18:14:45.556085+0000","last_became_peered":"2023-10-14T18:14:45.556085+0000","last_unstale":"2023-10-14T18:14:46.575877+0000","last_undegraded":"2023-10-14T18:14:46.575877+0000","last_fullsized":"2023-10-14T18:14:46.575877+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:27.286846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.12","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573926+0000","last_change":"2023-10-14T18:14:46.565618+0000","last_active":"2023-10-14T18:14:46.573926+0000","last_peered":"2023-10-14T18:14:46.573926+0000","last_clean":"2023-10-14T18:14:46.573926+0000","last_became_active":"2023-10-14T18:14:46.565352+0000","last_became_peered":"2023-10-14T18:14:46.565352+0000","last_unstale":"2023-10-14T18:14:46.573926+0000","last_undegraded":"2023-10-14T18:14:46.573926+0000","last_fullsized":"2023-10-14T18:14:46.573926+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:57:36.892049+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"24.13","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574111+0000","last_change":"2023-10-14T18:14:45.552115+0000","last_active":"2023-10-14T18:14:46.574111+0000","last_peered":"2023-10-14T18:14:46.574111+0000","last_clean":"2023-10-14T18:14:46.574111+0000","last_became_active":"2023-10-14T18:14:45.551248+0000","last_became_peered":"2023-10-14T18:14:45.551248+0000","last_unstale":"2023-10-14T18:14:46.574111+0000","last_undegraded":"2023-10-14T18:14:46.574111+0000","last_fullsized":"2023-10-14T18:14:46.574111+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:32.199977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.13","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559397+0000","last_change":"2023-10-14T18:14:46.560433+0000","last_active":"2023-10-14T18:14:47.559397+0000","last_peered":"2023-10-14T18:14:47.559397+0000","last_clean":"2023-10-14T18:14:47.559397+0000","last_became_active":"2023-10-14T18:14:46.560153+0000","last_became_peered":"2023-10-14T18:14:46.560153+0000","last_unstale":"2023-10-14T18:14:47.559397+0000","last_undegraded":"2023-10-14T18:14:47.559397+0000","last_fullsized":"2023-10-14T18:14:47.559397+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:42:28.305124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.12","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556578+0000","last_change":"2023-10-14T18:14:45.552984+0000","last_active":"2023-10-14T18:14:47.556578+0000","last_peered":"2023-10-14T18:14:47.556578+0000","last_clean":"2023-10-14T18:14:47.556578+0000","last_became_active":"2023-10-14T18:14:45.552616+0000","last_became_peered":"2023-10-14T18:14:45.552616+0000","last_unstale":"2023-10-14T18:14:47.556578+0000","last_undegraded":"2023-10-14T18:14:47.556578+0000","last_fullsized":"2023-10-14T18:14:47.556578+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:04.427609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.10","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.566410+0000","last_change":"2023-10-14T18:14:46.559770+0000","last_active":"2023-10-14T18:14:46.566410+0000","last_peered":"2023-10-14T18:14:46.566410+0000","last_clean":"2023-10-14T18:14:46.566410+0000","last_became_active":"2023-10-14T18:14:46.559410+0000","last_became_peered":"2023-10-14T18:14:46.559410+0000","last_unstale":"2023-10-14T18:14:46.566410+0000","last_undegraded":"2023-10-14T18:14:46.566410+0000","last_fullsized":"2023-10-14T18:14:46.566410+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:01:15.737925+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.11","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.529274+0000","last_change":"2023-10-14T18:14:45.543303+0000","last_active":"2023-10-14T18:14:46.529274+0000","last_peered":"2023-10-14T18:14:46.529274+0000","last_clean":"2023-10-14T18:14:46.529274+0000","last_became_active":"2023-10-14T18:14:45.541994+0000","last_became_peered":"2023-10-14T18:14:45.541994+0000","last_unstale":"2023-10-14T18:14:46.529274+0000","last_undegraded":"2023-10-14T18:14:46.529274+0000","last_fullsized":"2023-10-14T18:14:46.529274+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:00.301921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.11","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.555732+0000","last_change":"2023-10-14T18:14:46.560392+0000","last_active":"2023-10-14T18:14:47.555732+0000","last_peered":"2023-10-14T18:14:47.555732+0000","last_clean":"2023-10-14T18:14:47.555732+0000","last_became_active":"2023-10-14T18:14:46.560079+0000","last_became_peered":"2023-10-14T18:14:46.560079+0000","last_unstale":"2023-10-14T18:14:47.555732+0000","last_undegraded":"2023-10-14T18:14:47.555732+0000","last_fullsized":"2023-10-14T18:14:47.555732+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:27:22.143858+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.10","version":"209'8","reported_seq":19,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574806+0000","last_change":"2023-10-14T18:14:45.550864+0000","last_active":"2023-10-14T18:14:46.574806+0000","last_peered":"2023-10-14T18:14:46.574806+0000","last_clean":"2023-10-14T18:14:46.574806+0000","last_became_active":"2023-10-14T18:14:45.550076+0000","last_became_peered":"2023-10-14T18:14:45.550076+0000","last_unstale":"2023-10-14T18:14:46.574806+0000","last_undegraded":"2023-10-14T18:14:46.574806+0000","last_fullsized":"2023-10-14T18:14:46.574806+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:21:21.237010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.e","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557051+0000","last_change":"2023-10-14T18:14:46.561688+0000","last_active":"2023-10-14T18:14:47.557051+0000","last_peered":"2023-10-14T18:14:47.557051+0000","last_clean":"2023-10-14T18:14:47.557051+0000","last_became_active":"2023-10-14T18:14:46.561370+0000","last_became_peered":"2023-10-14T18:14:46.561370+0000","last_unstale":"2023-10-14T18:14:47.557051+0000","last_undegraded":"2023-10-14T18:14:47.557051+0000","last_fullsized":"2023-10-14T18:14:47.557051+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.f","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568861+0000","last_change":"2023-10-14T18:14:45.554342+0000","last_active":"2023-10-14T18:14:46.568861+0000","last_peered":"2023-10-14T18:14:46.568861+0000","last_clean":"2023-10-14T18:14:46.568861+0000","last_became_active":"2023-10-14T18:14:45.553690+0000","last_became_peered":"2023-10-14T18:14:45.553690+0000","last_unstale":"2023-10-14T18:14:46.568861+0000","last_undegraded":"2023-10-14T18:14:46.568861+0000","last_fullsized":"2023-10-14T18:14:46.568861+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:00:24.139422+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.f","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.568707+0000","last_change":"2023-10-14T18:14:46.561989+0000","last_active":"2023-10-14T18:14:46.568707+0000","last_peered":"2023-10-14T18:14:46.568707+0000","last_clean":"2023-10-14T18:14:46.568707+0000","last_became_active":"2023-10-14T18:14:46.561702+0000","last_became_peered":"2023-10-14T18:14:46.561702+0000","last_unstale":"2023-10-14T18:14:46.568707+0000","last_undegraded":"2023-10-14T18:14:46.568707+0000","last_fullsized":"2023-10-14T18:14:46.568707+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:48:05.811417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.e","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570589+0000","last_change":"2023-10-14T18:14:45.556184+0000","last_active":"2023-10-14T18:14:46.570589+0000","last_peered":"2023-10-14T18:14:46.570589+0000","last_clean":"2023-10-14T18:14:46.570589+0000","last_became_active":"2023-10-14T18:14:45.554519+0000","last_became_peered":"2023-10-14T18:14:45.554519+0000","last_unstale":"2023-10-14T18:14:46.570589+0000","last_undegraded":"2023-10-14T18:14:46.570589+0000","last_fullsized":"2023-10-14T18:14:46.570589+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:25:32.456073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.c","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570093+0000","last_change":"2023-10-14T18:14:46.565669+0000","last_active":"2023-10-14T18:14:46.570093+0000","last_peered":"2023-10-14T18:14:46.570093+0000","last_clean":"2023-10-14T18:14:46.570093+0000","last_became_active":"2023-10-14T18:14:46.563658+0000","last_became_peered":"2023-10-14T18:14:46.563658+0000","last_unstale":"2023-10-14T18:14:46.570093+0000","last_undegraded":"2023-10-14T18:14:46.570093+0000","last_fullsized":"2023-10-14T18:14:46.570093+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"24.d","version":"209'1","reported_seq":12,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.574270+0000","last_change":"2023-10-14T18:14:45.551331+0000","last_active":"2023-10-14T18:14:46.574270+0000","last_peered":"2023-10-14T18:14:46.574270+0000","last_clean":"2023-10-14T18:14:46.574270+0000","last_became_active":"2023-10-14T18:14:45.550056+0000","last_became_peered":"2023-10-14T18:14:45.550056+0000","last_unstale":"2023-10-14T18:14:46.574270+0000","last_undegraded":"2023-10-14T18:14:46.574270+0000","last_fullsized":"2023-10-14T18:14:46.574270+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:32.153972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.d","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562755+0000","last_change":"2023-10-14T18:14:46.556117+0000","last_active":"2023-10-14T18:14:47.562755+0000","last_peered":"2023-10-14T18:14:47.562755+0000","last_clean":"2023-10-14T18:14:47.562755+0000","last_became_active":"2023-10-14T18:14:46.554841+0000","last_became_peered":"2023-10-14T18:14:46.554841+0000","last_unstale":"2023-10-14T18:14:47.562755+0000","last_undegraded":"2023-10-14T18:14:47.562755+0000","last_fullsized":"2023-10-14T18:14:47.562755+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.c","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559880+0000","last_change":"2023-10-14T18:14:45.554571+0000","last_active":"2023-10-14T18:14:47.559880+0000","last_peered":"2023-10-14T18:14:47.559880+0000","last_clean":"2023-10-14T18:14:47.559880+0000","last_became_active":"2023-10-14T18:14:45.554270+0000","last_became_peered":"2023-10-14T18:14:45.554270+0000","last_unstale":"2023-10-14T18:14:47.559880+0000","last_undegraded":"2023-10-14T18:14:47.559880+0000","last_fullsized":"2023-10-14T18:14:47.559880+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:52:57.448693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"25.a","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556510+0000","last_change":"2023-10-14T18:14:46.556384+0000","last_active":"2023-10-14T18:14:47.556510+0000","last_peered":"2023-10-14T18:14:47.556510+0000","last_clean":"2023-10-14T18:14:47.556510+0000","last_became_active":"2023-10-14T18:14:46.556139+0000","last_became_peered":"2023-10-14T18:14:46.556139+0000","last_unstale":"2023-10-14T18:14:47.556510+0000","last_undegraded":"2023-10-14T18:14:47.556510+0000","last_fullsized":"2023-10-14T18:14:47.556510+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:29.767500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.b","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571896+0000","last_change":"2023-10-14T18:14:45.554829+0000","last_active":"2023-10-14T18:14:46.571896+0000","last_peered":"2023-10-14T18:14:46.571896+0000","last_clean":"2023-10-14T18:14:46.571896+0000","last_became_active":"2023-10-14T18:14:45.553915+0000","last_became_peered":"2023-10-14T18:14:45.553915+0000","last_unstale":"2023-10-14T18:14:46.571896+0000","last_undegraded":"2023-10-14T18:14:46.571896+0000","last_fullsized":"2023-10-14T18:14:46.571896+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:18.139882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.b","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.559601+0000","last_change":"2023-10-14T18:14:46.549534+0000","last_active":"2023-10-14T18:14:47.559601+0000","last_peered":"2023-10-14T18:14:47.559601+0000","last_clean":"2023-10-14T18:14:47.559601+0000","last_became_active":"2023-10-14T18:14:46.549113+0000","last_became_peered":"2023-10-14T18:14:46.549113+0000","last_unstale":"2023-10-14T18:14:47.559601+0000","last_undegraded":"2023-10-14T18:14:47.559601+0000","last_fullsized":"2023-10-14T18:14:47.559601+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:02.805957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"24.a","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.547385+0000","last_change":"2023-10-14T18:14:45.545136+0000","last_active":"2023-10-14T18:14:46.547385+0000","last_peered":"2023-10-14T18:14:46.547385+0000","last_clean":"2023-10-14T18:14:46.547385+0000","last_became_active":"2023-10-14T18:14:45.544821+0000","last_became_peered":"2023-10-14T18:14:45.544821+0000","last_unstale":"2023-10-14T18:14:46.547385+0000","last_undegraded":"2023-10-14T18:14:46.547385+0000","last_fullsized":"2023-10-14T18:14:46.547385+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:31.117638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"25.8","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560773+0000","last_change":"2023-10-14T18:14:46.562237+0000","last_active":"2023-10-14T18:14:47.560773+0000","last_peered":"2023-10-14T18:14:47.560773+0000","last_clean":"2023-10-14T18:14:47.560773+0000","last_became_active":"2023-10-14T18:14:46.561910+0000","last_became_peered":"2023-10-14T18:14:46.561910+0000","last_unstale":"2023-10-14T18:14:47.560773+0000","last_undegraded":"2023-10-14T18:14:47.560773+0000","last_fullsized":"2023-10-14T18:14:47.560773+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:22.546148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.9","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.539093+0000","last_change":"2023-10-14T18:14:45.547848+0000","last_active":"2023-10-14T18:14:46.539093+0000","last_peered":"2023-10-14T18:14:46.539093+0000","last_clean":"2023-10-14T18:14:46.539093+0000","last_became_active":"2023-10-14T18:14:45.547338+0000","last_became_peered":"2023-10-14T18:14:45.547338+0000","last_unstale":"2023-10-14T18:14:46.539093+0000","last_undegraded":"2023-10-14T18:14:46.539093+0000","last_fullsized":"2023-10-14T18:14:46.539093+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:33.638835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.9","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.559071+0000","last_change":"2023-10-14T18:14:46.554976+0000","last_active":"2023-10-14T18:14:46.559071+0000","last_peered":"2023-10-14T18:14:46.559071+0000","last_clean":"2023-10-14T18:14:46.559071+0000","last_became_active":"2023-10-14T18:14:46.554646+0000","last_became_peered":"2023-10-14T18:14:46.554646+0000","last_unstale":"2023-10-14T18:14:46.559071+0000","last_undegraded":"2023-10-14T18:14:46.559071+0000","last_fullsized":"2023-10-14T18:14:46.559071+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:12:16.144262+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.8","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.546434+0000","last_change":"2023-10-14T18:14:45.548468+0000","last_active":"2023-10-14T18:14:46.546434+0000","last_peered":"2023-10-14T18:14:46.546434+0000","last_clean":"2023-10-14T18:14:46.546434+0000","last_became_active":"2023-10-14T18:14:45.545163+0000","last_became_peered":"2023-10-14T18:14:45.545163+0000","last_unstale":"2023-10-14T18:14:46.546434+0000","last_undegraded":"2023-10-14T18:14:46.546434+0000","last_fullsized":"2023-10-14T18:14:46.546434+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:27:26.739406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.6","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567954+0000","last_change":"2023-10-14T18:14:46.559300+0000","last_active":"2023-10-14T18:14:46.567954+0000","last_peered":"2023-10-14T18:14:46.567954+0000","last_clean":"2023-10-14T18:14:46.567954+0000","last_became_active":"2023-10-14T18:14:46.559011+0000","last_became_peered":"2023-10-14T18:14:46.559011+0000","last_unstale":"2023-10-14T18:14:46.567954+0000","last_undegraded":"2023-10-14T18:14:46.567954+0000","last_fullsized":"2023-10-14T18:14:46.567954+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:13.667031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.7","version":"209'2","reported_seq":13,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573390+0000","last_change":"2023-10-14T18:14:45.552202+0000","last_active":"2023-10-14T18:14:46.573390+0000","last_peered":"2023-10-14T18:14:46.573390+0000","last_clean":"2023-10-14T18:14:46.573390+0000","last_became_active":"2023-10-14T18:14:45.551946+0000","last_became_peered":"2023-10-14T18:14:45.551946+0000","last_unstale":"2023-10-14T18:14:46.573390+0000","last_undegraded":"2023-10-14T18:14:46.573390+0000","last_fullsized":"2023-10-14T18:14:46.573390+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:32:17.951753+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"25.7","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557383+0000","last_change":"2023-10-14T18:14:46.565738+0000","last_active":"2023-10-14T18:14:47.557383+0000","last_peered":"2023-10-14T18:14:47.557383+0000","last_clean":"2023-10-14T18:14:47.557383+0000","last_became_active":"2023-10-14T18:14:46.564733+0000","last_became_peered":"2023-10-14T18:14:46.564733+0000","last_unstale":"2023-10-14T18:14:47.557383+0000","last_undegraded":"2023-10-14T18:14:47.557383+0000","last_fullsized":"2023-10-14T18:14:47.557383+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:12:13.667031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.6","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.557142+0000","last_change":"2023-10-14T18:14:45.546245+0000","last_active":"2023-10-14T18:14:47.557142+0000","last_peered":"2023-10-14T18:14:47.557142+0000","last_clean":"2023-10-14T18:14:47.557142+0000","last_became_active":"2023-10-14T18:14:45.545899+0000","last_became_peered":"2023-10-14T18:14:45.545899+0000","last_unstale":"2023-10-14T18:14:47.557142+0000","last_undegraded":"2023-10-14T18:14:47.557142+0000","last_fullsized":"2023-10-14T18:14:47.557142+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:14:42.146401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.4","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567195+0000","last_change":"2023-10-14T18:14:46.552911+0000","last_active":"2023-10-14T18:14:46.567195+0000","last_peered":"2023-10-14T18:14:46.567195+0000","last_clean":"2023-10-14T18:14:46.567195+0000","last_became_active":"2023-10-14T18:14:46.552603+0000","last_became_peered":"2023-10-14T18:14:46.552603+0000","last_unstale":"2023-10-14T18:14:46.567195+0000","last_undegraded":"2023-10-14T18:14:46.567195+0000","last_fullsized":"2023-10-14T18:14:46.567195+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:33.218255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.5","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.556897+0000","last_change":"2023-10-14T18:14:45.555101+0000","last_active":"2023-10-14T18:14:47.556897+0000","last_peered":"2023-10-14T18:14:47.556897+0000","last_clean":"2023-10-14T18:14:47.556897+0000","last_became_active":"2023-10-14T18:14:45.554828+0000","last_became_peered":"2023-10-14T18:14:45.554828+0000","last_unstale":"2023-10-14T18:14:47.556897+0000","last_undegraded":"2023-10-14T18:14:47.556897+0000","last_fullsized":"2023-10-14T18:14:47.556897+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:13:51.305061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"25.5","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.571764+0000","last_change":"2023-10-14T18:14:46.563147+0000","last_active":"2023-10-14T18:14:46.571764+0000","last_peered":"2023-10-14T18:14:46.571764+0000","last_clean":"2023-10-14T18:14:46.571764+0000","last_became_active":"2023-10-14T18:14:46.562863+0000","last_became_peered":"2023-10-14T18:14:46.562863+0000","last_unstale":"2023-10-14T18:14:46.571764+0000","last_undegraded":"2023-10-14T18:14:46.571764+0000","last_fullsized":"2023-10-14T18:14:46.571764+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:14:44.186849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"24.4","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.529914+0000","last_change":"2023-10-14T18:14:45.551336+0000","last_active":"2023-10-14T18:14:46.529914+0000","last_peered":"2023-10-14T18:14:46.529914+0000","last_clean":"2023-10-14T18:14:46.529914+0000","last_became_active":"2023-10-14T18:14:45.551061+0000","last_became_peered":"2023-10-14T18:14:45.551061+0000","last_unstale":"2023-10-14T18:14:46.529914+0000","last_undegraded":"2023-10-14T18:14:46.529914+0000","last_fullsized":"2023-10-14T18:14:46.529914+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:32.153972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"25.2","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.573137+0000","last_change":"2023-10-14T18:14:46.572423+0000","last_active":"2023-10-14T18:14:46.573137+0000","last_peered":"2023-10-14T18:14:46.573137+0000","last_clean":"2023-10-14T18:14:46.573137+0000","last_became_active":"2023-10-14T18:14:46.572010+0000","last_became_peered":"2023-10-14T18:14:46.572010+0000","last_unstale":"2023-10-14T18:14:46.573137+0000","last_undegraded":"2023-10-14T18:14:46.573137+0000","last_fullsized":"2023-10-14T18:14:46.573137+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:03:01.484863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.3","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.545296+0000","last_change":"2023-10-14T18:14:45.555363+0000","last_active":"2023-10-14T18:14:46.545296+0000","last_peered":"2023-10-14T18:14:46.545296+0000","last_clean":"2023-10-14T18:14:46.545296+0000","last_became_active":"2023-10-14T18:14:45.555088+0000","last_became_peered":"2023-10-14T18:14:45.555088+0000","last_unstale":"2023-10-14T18:14:46.545296+0000","last_undegraded":"2023-10-14T18:14:46.545296+0000","last_fullsized":"2023-10-14T18:14:46.545296+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:28:18.982319+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.3","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.570899+0000","last_change":"2023-10-14T18:14:46.562687+0000","last_active":"2023-10-14T18:14:46.570899+0000","last_peered":"2023-10-14T18:14:46.570899+0000","last_clean":"2023-10-14T18:14:46.570899+0000","last_became_active":"2023-10-14T18:14:46.562402+0000","last_became_peered":"2023-10-14T18:14:46.562402+0000","last_unstale":"2023-10-14T18:14:46.570899+0000","last_undegraded":"2023-10-14T18:14:46.570899+0000","last_fullsized":"2023-10-14T18:14:46.570899+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:45:39.629254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"24.2","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.545296+0000","last_change":"2023-10-14T18:14:45.554797+0000","last_active":"2023-10-14T18:14:46.545296+0000","last_peered":"2023-10-14T18:14:46.545296+0000","last_clean":"2023-10-14T18:14:46.545296+0000","last_became_active":"2023-10-14T18:14:45.554501+0000","last_became_peered":"2023-10-14T18:14:45.554501+0000","last_unstale":"2023-10-14T18:14:46.545296+0000","last_undegraded":"2023-10-14T18:14:46.545296+0000","last_fullsized":"2023-10-14T18:14:46.545296+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:26.747637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"25.0","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.560916+0000","last_change":"2023-10-14T18:14:46.548676+0000","last_active":"2023-10-14T18:14:47.560916+0000","last_peered":"2023-10-14T18:14:47.560916+0000","last_clean":"2023-10-14T18:14:47.560916+0000","last_became_active":"2023-10-14T18:14:46.548385+0000","last_became_peered":"2023-10-14T18:14:46.548385+0000","last_unstale":"2023-10-14T18:14:47.560916+0000","last_undegraded":"2023-10-14T18:14:47.560916+0000","last_fullsized":"2023-10-14T18:14:47.560916+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:14:43.164679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"24.1","version":"0'0","reported_seq":12,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.562088+0000","last_change":"2023-10-14T18:14:45.551245+0000","last_active":"2023-10-14T18:14:47.562088+0000","last_peered":"2023-10-14T18:14:47.562088+0000","last_clean":"2023-10-14T18:14:47.562088+0000","last_became_active":"2023-10-14T18:14:45.550954+0000","last_became_peered":"2023-10-14T18:14:45.550954+0000","last_unstale":"2023-10-14T18:14:47.562088+0000","last_undegraded":"2023-10-14T18:14:47.562088+0000","last_fullsized":"2023-10-14T18:14:47.562088+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:04.427609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"25.1","version":"0'0","reported_seq":9,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.567704+0000","last_change":"2023-10-14T18:14:46.559511+0000","last_active":"2023-10-14T18:14:46.567704+0000","last_peered":"2023-10-14T18:14:46.567704+0000","last_clean":"2023-10-14T18:14:46.567704+0000","last_became_active":"2023-10-14T18:14:46.559231+0000","last_became_peered":"2023-10-14T18:14:46.559231+0000","last_unstale":"2023-10-14T18:14:46.567704+0000","last_undegraded":"2023-10-14T18:14:46.567704+0000","last_fullsized":"2023-10-14T18:14:46.567704+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:27.765915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"24.0","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.538940+0000","last_change":"2023-10-14T18:14:45.549540+0000","last_active":"2023-10-14T18:14:46.538940+0000","last_peered":"2023-10-14T18:14:46.538940+0000","last_clean":"2023-10-14T18:14:46.538940+0000","last_became_active":"2023-10-14T18:14:45.547454+0000","last_became_peered":"2023-10-14T18:14:45.547454+0000","last_unstale":"2023-10-14T18:14:46.538940+0000","last_undegraded":"2023-10-14T18:14:46.538940+0000","last_fullsized":"2023-10-14T18:14:46.538940+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:56:01.787679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"25.18","version":"0'0","reported_seq":11,"reported_epoch":210,"state":"active+clean","last_fresh":"2023-10-14T18:14:47.555874+0000","last_change":"2023-10-14T18:14:46.557086+0000","last_active":"2023-10-14T18:14:47.555874+0000","last_peered":"2023-10-14T18:14:47.555874+0000","last_clean":"2023-10-14T18:14:47.555874+0000","last_became_active":"2023-10-14T18:14:46.554501+0000","last_became_peered":"2023-10-14T18:14:46.554501+0000","last_unstale":"2023-10-14T18:14:47.555874+0000","last_undegraded":"2023-10-14T18:14:47.555874+0000","last_fullsized":"2023-10-14T18:14:47.555874+0000","mapping_epoch":208,"log_start":"0'0","ondisk_log_start":"0'0","created":208,"last_epoch_clean":209,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:45.499784+0000","last_clean_scrub_stamp":"2023-10-14T18:14:45.499784+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:43.387125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"24.19","version":"0'0","reported_seq":11,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553292+0000","last_change":"2023-10-14T18:14:45.546699+0000","last_active":"2023-10-14T18:14:46.553292+0000","last_peered":"2023-10-14T18:14:46.553292+0000","last_clean":"2023-10-14T18:14:46.553292+0000","last_became_active":"2023-10-14T18:14:45.546426+0000","last_became_peered":"2023-10-14T18:14:45.546426+0000","last_unstale":"2023-10-14T18:14:46.553292+0000","last_undegraded":"2023-10-14T18:14:46.553292+0000","last_fullsized":"2023-10-14T18:14:46.553292+0000","mapping_epoch":207,"log_start":"0'0","ondisk_log_start":"0'0","created":207,"last_epoch_clean":208,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:14:44.481506+0000","last_clean_scrub_stamp":"2023-10-14T18:14:44.481506+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:40:00.301921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"197'326","reported_seq":1115,"reported_epoch":209,"state":"active+clean","last_fresh":"2023-10-14T18:14:46.553083+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:14:46.553083+0000","last_peered":"2023-10-14T18:14:46.553083+0000","last_clean":"2023-10-14T18:14:46.553083+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:14:46.553083+0000","last_undegraded":"2023-10-14T18:14:46.553083+0000","last_fullsized":"2023-10-14T18:14:46.553083+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":326,"log_dups_size":0,"ondisk_log_size":326,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":25,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":24,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1940,"num_read_kb":4014,"num_write":539,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":326,"ondisk_log_size":326,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575974,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52372,"kb_used_data":3144,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319468,"statfs":{"total":96636764160,"available":96583135232,"internally_reserved":0,"allocated":3219456,"data_stored":1882870,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":50395595},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2430000000000001,"15min":1.216},"min":{"1min":0.82699999999999996,"5min":0.79100000000000004,"15min":0.72099999999999997},"max":{"1min":1.6539999999999999,"5min":1.9430000000000001,"15min":2.1669999999999998},"last":1.014},{"interface":"front","average":{"1min":1.236,"5min":1.24,"15min":1.2310000000000001},"min":{"1min":0.84999999999999998,"5min":0.79500000000000004,"15min":0.76500000000000001},"max":{"1min":1.6579999999999999,"5min":1.7310000000000001,"15min":2.2149999999999999},"last":1.5369999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.2689999999999999,"5min":1.294,"15min":1.2989999999999999},"min":{"1min":0.85899999999999999,"5min":0.82599999999999996,"15min":0.80200000000000005},"max":{"1min":1.5940000000000001,"5min":2.214,"15min":2.3239999999999998},"last":1.2430000000000001},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.304,"15min":1.3160000000000001},"min":{"1min":1.008,"5min":0.84099999999999997,"15min":0.84099999999999997},"max":{"1min":1.6259999999999999,"5min":2.0209999999999999,"15min":2.0209999999999999},"last":1.266}]},{"osd":2,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.3089999999999999,"15min":1.2869999999999999},"min":{"1min":0.94699999999999995,"5min":0.83199999999999996,"15min":0.83199999999999996},"max":{"1min":1.671,"5min":2.004,"15min":2.1190000000000002},"last":1.071},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.355,"15min":1.381},"min":{"1min":1.044,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.8260000000000001,"5min":2.069,"15min":2.2189999999999999},"last":1.325}]},{"osd":3,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3380000000000001,"5min":1.4330000000000001,"15min":1.4510000000000001},"min":{"1min":0.999,"5min":0.92300000000000004,"15min":0.92300000000000004},"max":{"1min":1.768,"5min":2.4430000000000001,"15min":2.4430000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.5249999999999999,"5min":1.5429999999999999,"15min":1.5509999999999999},"min":{"1min":1.0600000000000001,"5min":1.02,"15min":0.93200000000000005},"max":{"1min":1.724,"5min":2.2810000000000001,"15min":2.423},"last":1.1499999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.3129999999999999,"15min":1.3240000000000001},"min":{"1min":0.94399999999999995,"5min":0.81999999999999995,"15min":0.79300000000000004},"max":{"1min":1.784,"5min":2.367,"15min":2.367},"last":1.389},{"interface":"front","average":{"1min":1.3,"5min":1.3899999999999999,"15min":1.383},"min":{"1min":0.90900000000000003,"5min":0.90900000000000003,"15min":0.86399999999999999},"max":{"1min":1.8380000000000001,"5min":2.661,"15min":2.661},"last":1.2230000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.347,"5min":1.4430000000000001,"15min":1.4630000000000001},"min":{"1min":0.94899999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"max":{"1min":1.637,"5min":2.395,"15min":2.516},"last":1.49},{"interface":"front","average":{"1min":1.48,"5min":1.4950000000000001,"15min":1.516},"min":{"1min":1.044,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":1.8,"5min":2.218,"15min":2.448},"last":1.6000000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.6040000000000001,"15min":1.631},"min":{"1min":1.1100000000000001,"5min":1.1100000000000001,"15min":1.1100000000000001},"max":{"1min":1.744,"5min":2.4849999999999999,"15min":2.4849999999999999},"last":1.706},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.573,"15min":1.591},"min":{"1min":1.137,"5min":1.038,"15min":0.80500000000000005},"max":{"1min":1.9019999999999999,"5min":2.8159999999999998,"15min":2.8159999999999998},"last":1.446}]}]},{"osd":6,"up_from":13,"seq":55834575973,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29576,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342264,"statfs":{"total":96636764160,"available":96606478336,"internally_reserved":0,"allocated":2617344,"data_stored":1292398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.1719999999999999,"5min":1.204,"15min":1.1870000000000001},"min":{"1min":0.879,"5min":0.78100000000000003,"15min":0.51300000000000001},"max":{"1min":1.6919999999999999,"5min":1.8560000000000001,"15min":1.873},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.2330000000000001,"5min":1.236,"15min":1.216},"min":{"1min":0.90800000000000003,"5min":0.85199999999999998,"15min":0.56200000000000006},"max":{"1min":1.5109999999999999,"5min":1.71,"15min":1.944},"last":0.80600000000000005}]},{"osd":1,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2649999999999999,"15min":1.2589999999999999},"min":{"1min":0.96899999999999997,"5min":0.874,"15min":0.83199999999999996},"max":{"1min":1.6220000000000001,"5min":1.776,"15min":2.0840000000000001},"last":0.93100000000000005},{"interface":"front","average":{"1min":1.3700000000000001,"5min":1.381,"15min":1.353},"min":{"1min":1.1200000000000001,"5min":0.98299999999999998,"15min":0.85799999999999998},"max":{"1min":1.7010000000000001,"5min":1.7769999999999999,"15min":1.9470000000000001},"last":0.73899999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4159999999999999,"5min":1.4179999999999999,"15min":1.3720000000000001},"min":{"1min":0.94599999999999995,"5min":0.94599999999999995,"15min":0.82599999999999996},"max":{"1min":1.7330000000000001,"5min":1.8779999999999999,"15min":2.4849999999999999},"last":1.1539999999999999},{"interface":"front","average":{"1min":1.393,"5min":1.4399999999999999,"15min":1.429},"min":{"1min":1.0780000000000001,"5min":1.0780000000000001,"15min":0.98099999999999998},"max":{"1min":1.8220000000000001,"5min":1.8220000000000001,"15min":1.9870000000000001},"last":1.0489999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.4119999999999999,"15min":1.4219999999999999},"min":{"1min":1.01,"5min":0.83399999999999996,"15min":0.76900000000000002},"max":{"1min":1.599,"5min":1.8560000000000001,"15min":2.2519999999999998},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4299999999999999,"15min":1.4430000000000001},"min":{"1min":1.052,"5min":0.98799999999999999,"15min":0.91300000000000003},"max":{"1min":1.8320000000000001,"5min":1.895,"15min":2.218},"last":1.2709999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.363,"5min":1.3620000000000001,"15min":1.383},"min":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.60299999999999998},"max":{"1min":1.784,"5min":1.9750000000000001,"15min":2.0299999999999998},"last":1.228},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4199999999999999,"15min":1.4319999999999999},"min":{"1min":1.03,"5min":0.96599999999999997,"15min":0.90500000000000003},"max":{"1min":1.8580000000000001,"5min":2.0099999999999998,"15min":2.27},"last":1.411}]},{"osd":5,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3540000000000001,"5min":1.4019999999999999,"15min":1.4199999999999999},"min":{"1min":0.97699999999999998,"5min":0.91100000000000003,"15min":0.873},"max":{"1min":1.851,"5min":1.9410000000000001,"15min":2.4940000000000002},"last":1.302},{"interface":"front","average":{"1min":1.5289999999999999,"5min":1.5669999999999999,"15min":1.5780000000000001},"min":{"1min":1.0840000000000001,"5min":0.995,"15min":0.81399999999999995},"max":{"1min":1.899,"5min":1.996,"15min":2.5609999999999999},"last":1.3660000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.554,"5min":1.5860000000000001,"15min":1.6080000000000001},"min":{"1min":1.2110000000000001,"5min":1.135,"15min":0.95399999999999996},"max":{"1min":1.9079999999999999,"5min":1.9950000000000001,"15min":3.423},"last":1.6679999999999999},{"interface":"front","average":{"1min":1.54,"5min":1.581,"15min":1.607},"min":{"1min":1.0680000000000001,"5min":1.0629999999999999,"15min":1.0629999999999999},"max":{"1min":1.9690000000000001,"5min":2.0190000000000001,"15min":2.8769999999999998},"last":1.5649999999999999}]}]},{"osd":5,"up_from":13,"seq":55834575974,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42056,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329784,"statfs":{"total":96636764160,"available":96593698816,"internally_reserved":0,"allocated":2617344,"data_stored":1292398,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.0920000000000001,"5min":1.214,"15min":1.2030000000000001},"min":{"1min":0.84099999999999997,"5min":0.80700000000000005,"15min":0.60399999999999998},"max":{"1min":1.474,"5min":1.9830000000000001,"15min":2.0459999999999998},"last":1.075},{"interface":"front","average":{"1min":1.248,"5min":1.264,"15min":1.2569999999999999},"min":{"1min":0.77700000000000002,"5min":0.77700000000000002,"15min":0.71799999999999997},"max":{"1min":1.7270000000000001,"5min":2.1070000000000002,"15min":2.1070000000000002},"last":1.006}]},{"osd":1,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.24,"5min":1.2829999999999999,"15min":1.27},"min":{"1min":0.90900000000000003,"5min":0.81200000000000006,"15min":0.74099999999999999},"max":{"1min":1.645,"5min":1.875,"15min":1.8899999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.2689999999999999,"5min":1.2929999999999999,"15min":1.304},"min":{"1min":0.99099999999999999,"5min":0.873,"15min":0.63},"max":{"1min":1.5489999999999999,"5min":1.849,"15min":2.0419999999999998},"last":1.153}]},{"osd":2,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.2430000000000001,"15min":1.2649999999999999},"min":{"1min":0.92400000000000004,"5min":0.92400000000000004,"15min":0.71199999999999997},"max":{"1min":1.589,"5min":1.9339999999999999,"15min":3.7090000000000001},"last":1.2050000000000001},{"interface":"front","average":{"1min":1.3460000000000001,"5min":1.4139999999999999,"15min":1.385},"min":{"1min":0.96899999999999997,"5min":0.91300000000000003,"15min":0.67600000000000005},"max":{"1min":1.5609999999999999,"5min":1.923,"15min":1.923},"last":1.5209999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.496,"15min":1.4710000000000001},"min":{"1min":0.95699999999999996,"5min":0.90900000000000003,"15min":0.81899999999999995},"max":{"1min":1.7929999999999999,"5min":2.4969999999999999,"15min":2.4969999999999999},"last":1.256},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4379999999999999,"15min":1.4430000000000001},"min":{"1min":0.98499999999999999,"5min":0.97999999999999998,"15min":0.81499999999999995},"max":{"1min":1.7909999999999999,"5min":2.4569999999999999,"15min":3.8330000000000002},"last":1.5620000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.3029999999999999,"5min":1.367,"15min":1.347},"min":{"1min":0.89600000000000002,"5min":0.79600000000000004,"15min":0.77800000000000002},"max":{"1min":1.8129999999999999,"5min":1.9199999999999999,"15min":2.98},"last":1.298},{"interface":"front","average":{"1min":1.397,"5min":1.389,"15min":1.369},"min":{"1min":0.90400000000000003,"5min":0.81899999999999995,"15min":0.57499999999999996},"max":{"1min":1.8220000000000001,"5min":2.0270000000000001,"15min":3.1280000000000001},"last":1.603}]},{"osd":6,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.4119999999999999,"5min":1.417,"15min":1.405},"min":{"1min":1.085,"5min":0.82099999999999995,"15min":0.65400000000000003},"max":{"1min":1.849,"5min":1.9670000000000001,"15min":4.0540000000000003},"last":1.361},{"interface":"front","average":{"1min":1.5629999999999999,"5min":1.641,"15min":1.6100000000000001},"min":{"1min":1.2569999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.7569999999999999,"5min":2.1850000000000001,"15min":3.6360000000000001},"last":1.6299999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:13:54 2023","interfaces":[{"interface":"back","average":{"1min":1.5529999999999999,"5min":1.593,"15min":1.556},"min":{"1min":1.24,"5min":0.998,"15min":0.67700000000000005},"max":{"1min":1.8740000000000001,"5min":2.5899999999999999,"15min":3.2759999999999998},"last":1.4730000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.573,"15min":1.5289999999999999},"min":{"1min":1.212,"5min":1.131,"15min":0.69999999999999996},"max":{"1min":2.206,"5min":2.206,"15min":3.9390000000000001},"last":1.669}]}]},{"osd":4,"up_from":13,"seq":55834575974,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60124,"kb_used_data":2572,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":94311716,"statfs":{"total":96636764160,"available":96575197184,"internally_reserved":0,"allocated":2633728,"data_stored":1294592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58915270},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.2629999999999999,"5min":1.212,"15min":1.2150000000000001},"min":{"1min":0.83199999999999996,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.643,"5min":1.7130000000000001,"15min":1.954},"last":0.628},{"interface":"front","average":{"1min":1.302,"5min":1.302,"15min":1.3009999999999999},"min":{"1min":0.877,"5min":0.84399999999999997,"15min":0.76100000000000001},"max":{"1min":1.6879999999999999,"5min":2.1400000000000001,"15min":2.3140000000000001},"last":0.98499999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3340000000000001,"5min":1.339,"15min":1.3169999999999999},"min":{"1min":0.94299999999999995,"5min":0.89000000000000001,"15min":0.59399999999999997},"max":{"1min":1.7310000000000001,"5min":1.8440000000000001,"15min":1.929},"last":1.204},{"interface":"front","average":{"1min":1.2490000000000001,"5min":1.2589999999999999,"15min":1.2969999999999999},"min":{"1min":0.91100000000000003,"5min":0.74199999999999999,"15min":0.69299999999999995},"max":{"1min":1.6619999999999999,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":0.91300000000000003}]},{"osd":2,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":1.2909999999999999,"15min":1.298},"min":{"1min":1.0640000000000001,"5min":0.80600000000000005,"15min":0.78000000000000003},"max":{"1min":1.6699999999999999,"5min":1.7689999999999999,"15min":3.524},"last":1.0720000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4610000000000001,"15min":1.448},"min":{"1min":1.0369999999999999,"5min":1.014,"15min":0.65700000000000003},"max":{"1min":1.9830000000000001,"5min":2.2440000000000002,"15min":2.2440000000000002},"last":1.1000000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.3600000000000001,"15min":1.3640000000000001},"min":{"1min":0.98999999999999999,"5min":0.98999999999999999,"15min":0.80200000000000005},"max":{"1min":1.7430000000000001,"5min":2.5489999999999999,"15min":2.5489999999999999},"last":1.034},{"interface":"front","average":{"1min":1.4910000000000001,"5min":1.4399999999999999,"15min":1.4550000000000001},"min":{"1min":1.1579999999999999,"5min":1.0700000000000001,"15min":0.93700000000000006},"max":{"1min":1.8400000000000001,"5min":2.419,"15min":4.1600000000000001},"last":0.95299999999999996}]},{"osd":5,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4399999999999999,"5min":1.4039999999999999,"15min":1.3899999999999999},"min":{"1min":0.95399999999999996,"5min":0.89400000000000002,"15min":0.69299999999999995},"max":{"1min":1.8620000000000001,"5min":2.02,"15min":2.02},"last":1.292},{"interface":"front","average":{"1min":1.288,"5min":1.363,"15min":1.379},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.64500000000000002},"max":{"1min":1.839,"5min":2.1030000000000002,"15min":2.1320000000000001},"last":1.1279999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.45,"15min":1.444},"min":{"1min":0.96499999999999997,"5min":0.94299999999999995,"15min":0.81799999999999995},"max":{"1min":1.823,"5min":2.8010000000000002,"15min":3.3610000000000002},"last":1.1719999999999999},{"interface":"front","average":{"1min":1.6080000000000001,"5min":1.5740000000000001,"15min":1.5720000000000001},"min":{"1min":1.2609999999999999,"5min":1.133,"15min":0.82699999999999996},"max":{"1min":2.0790000000000002,"5min":2.0790000000000002,"15min":2.3929999999999998},"last":1.3879999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:14:05 2023","interfaces":[{"interface":"back","average":{"1min":1.4510000000000001,"5min":1.512,"15min":1.53},"min":{"1min":1.042,"5min":1.042,"15min":0.77600000000000002},"max":{"1min":1.7769999999999999,"5min":2.7440000000000002,"15min":2.7440000000000002},"last":1.2549999999999999},{"interface":"front","average":{"1min":1.6539999999999999,"5min":1.6100000000000001,"15min":1.611},"min":{"1min":1.2729999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1000000000000001,"5min":2.6280000000000001,"15min":3.8610000000000002},"last":1.5629999999999999}]}]},{"osd":3,"up_from":13,"seq":55834575974,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51796,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320044,"statfs":{"total":96636764160,"available":96583725056,"internally_reserved":0,"allocated":2629632,"data_stored":1292592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":0.96699999999999997,"5min":0.98899999999999999,"15min":0.98999999999999999},"min":{"1min":0.82799999999999996,"5min":0.69699999999999995,"15min":0.53400000000000003},"max":{"1min":1.179,"5min":1.47,"15min":1.855},"last":0.99099999999999999},{"interface":"front","average":{"1min":1.159,"5min":1.21,"15min":1.212},"min":{"1min":0.97599999999999998,"5min":0.85899999999999999,"15min":0.82699999999999996},"max":{"1min":1.506,"5min":1.9970000000000001,"15min":1.9970000000000001},"last":0.90100000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.073,"5min":1.0569999999999999,"15min":1.0249999999999999},"min":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.59299999999999997},"max":{"1min":1.835,"5min":1.8939999999999999,"15min":1.919},"last":0.873},{"interface":"front","average":{"1min":1.266,"5min":1.2150000000000001,"15min":1.2270000000000001},"min":{"1min":0.96699999999999997,"5min":0.80600000000000005,"15min":0.46200000000000002},"max":{"1min":1.546,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.115}]},{"osd":2,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.546,"5min":1.4990000000000001,"15min":1.4650000000000001},"min":{"1min":1.1870000000000001,"5min":1.0389999999999999,"15min":0.64700000000000002},"max":{"1min":2.032,"5min":2.032,"15min":2.032},"last":1.042},{"interface":"front","average":{"1min":1.341,"5min":1.423,"15min":1.4019999999999999},"min":{"1min":0.84799999999999998,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.78,"5min":2.3540000000000001,"15min":2.3540000000000001},"last":1.167}]},{"osd":4,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.52,"5min":1.4970000000000001,"15min":1.448},"min":{"1min":1.284,"5min":1.0980000000000001,"15min":1.0209999999999999},"max":{"1min":2.153,"5min":2.153,"15min":2.153},"last":1.6359999999999999},{"interface":"front","average":{"1min":1.496,"5min":1.464,"15min":1.482},"min":{"1min":1.149,"5min":0.998,"15min":0.747},"max":{"1min":1.756,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.897}]},{"osd":5,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.454,"5min":1.4490000000000001,"15min":1.452},"min":{"1min":1.0589999999999999,"5min":1.0429999999999999,"15min":0.90500000000000003},"max":{"1min":1.931,"5min":2.476,"15min":2.476},"last":1.7410000000000001},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.6040000000000001,"15min":1.5740000000000001},"min":{"1min":1.1719999999999999,"5min":1.1719999999999999,"15min":0.97799999999999998},"max":{"1min":2.0430000000000001,"5min":2.891,"15min":2.891},"last":1.5980000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.635,"15min":1.645},"min":{"1min":1.385,"5min":1.2250000000000001,"15min":0.97099999999999997},"max":{"1min":2.016,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":2.141},{"interface":"front","average":{"1min":1.7170000000000001,"5min":1.669,"15min":1.681},"min":{"1min":1.242,"5min":1.129,"15min":0.91000000000000003},"max":{"1min":2.2949999999999999,"5min":2.3069999999999999,"15min":2.585},"last":2.0419999999999998}]},{"osd":7,"last update":"Sat Oct 14 18:14:01 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.6140000000000001,"15min":1.6120000000000001},"min":{"1min":1.196,"5min":1.196,"15min":0.85499999999999998},"max":{"1min":2.0150000000000001,"5min":2.419,"15min":2.419},"last":1.534},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.732,"15min":1.7010000000000001},"min":{"1min":1.22,"5min":1.22,"15min":1.0780000000000001},"max":{"1min":2.5169999999999999,"5min":2.548,"15min":2.548},"last":1.5489999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575974,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51804,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320036,"statfs":{"total":96636764160,"available":96583716864,"internally_reserved":0,"allocated":2629632,"data_stored":1293092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95699999999999996,"5min":0.93200000000000005,"15min":0.92300000000000004},"min":{"1min":0.69299999999999995,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":1.2949999999999999,"5min":1.819,"15min":1.99},"last":0.499},{"interface":"front","average":{"1min":1.125,"5min":1.143,"15min":1.137},"min":{"1min":0.85399999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"max":{"1min":1.4550000000000001,"5min":1.8660000000000001,"15min":1.8660000000000001},"last":0.438}]},{"osd":1,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":0.95599999999999996,"5min":1,"15min":1.034},"min":{"1min":0.79500000000000004,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.323,"5min":1.724,"15min":1.881},"last":0.39400000000000002},{"interface":"front","average":{"1min":1.361,"5min":1.246,"15min":1.2509999999999999},"min":{"1min":0.89600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.871,"5min":1.871,"15min":1.982},"last":0.55500000000000005}]},{"osd":3,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.2689999999999999,"15min":1.3129999999999999},"min":{"1min":0.95299999999999996,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.74,"5min":1.74,"15min":1.9690000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":1.478,"5min":1.413,"15min":1.4019999999999999},"min":{"1min":1.024,"5min":0.60399999999999998,"15min":0.60399999999999998},"max":{"1min":1.897,"5min":2.165,"15min":2.165},"last":0.53300000000000003}]},{"osd":4,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.569,"5min":1.46,"15min":1.4790000000000001},"min":{"1min":1.224,"5min":0.96199999999999997,"15min":0.96199999999999997},"max":{"1min":1.8859999999999999,"5min":2.0640000000000001,"15min":2.0859999999999999},"last":0.72499999999999998},{"interface":"front","average":{"1min":1.3859999999999999,"5min":1.357,"15min":1.3779999999999999},"min":{"1min":0.98999999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"max":{"1min":1.613,"5min":2.024,"15min":2.024},"last":0.66500000000000004}]},{"osd":5,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.494,"5min":1.4530000000000001,"15min":1.4690000000000001},"min":{"1min":1.0700000000000001,"5min":0.996,"15min":0.996},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":1.661,"5min":1.589,"15min":1.5900000000000001},"min":{"1min":1.216,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.1419999999999999,"5min":2.1549999999999998,"15min":2.1589999999999998},"last":0.79700000000000004}]},{"osd":6,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.5800000000000001,"5min":1.536,"15min":1.536},"min":{"1min":1.222,"5min":1.018,"15min":1.018},"max":{"1min":2.0529999999999999,"5min":2.1150000000000002,"15min":2.1190000000000002},"last":0.747},{"interface":"front","average":{"1min":1.599,"5min":1.605,"15min":1.637},"min":{"1min":1.306,"5min":1.1970000000000001,"15min":1.0760000000000001},"max":{"1min":1.952,"5min":2.1469999999999998,"15min":2.3180000000000001},"last":0.64500000000000002}]},{"osd":7,"last update":"Sat Oct 14 18:14:19 2023","interfaces":[{"interface":"back","average":{"1min":1.641,"5min":1.6100000000000001,"15min":1.6020000000000001},"min":{"1min":1.165,"5min":0.86599999999999999,"15min":0.86599999999999999},"max":{"1min":1.996,"5min":2.0070000000000001,"15min":2.2490000000000001},"last":0.81699999999999995},{"interface":"front","average":{"1min":1.702,"5min":1.696,"15min":1.7030000000000001},"min":{"1min":1.145,"5min":1.0489999999999999,"15min":1.0489999999999999},"max":{"1min":2.1360000000000001,"5min":2.1360000000000001,"15min":2.3250000000000002},"last":0.76600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51796,"kb_used_data":2568,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94320044,"statfs":{"total":96636764160,"available":96583725056,"internally_reserved":0,"allocated":2629632,"data_stored":1294010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":0.89400000000000002,"5min":0.96599999999999997,"15min":0.96899999999999997},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":1.264,"5min":1.647,"15min":2.4929999999999999},"last":0.85799999999999998},{"interface":"front","average":{"1min":1.123,"5min":1.1579999999999999,"15min":1.157},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.7070000000000001,"5min":1.738,"15min":2.5590000000000002},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.0700000000000001,"15min":1.0680000000000001},"min":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.6759999999999999,"5min":1.6930000000000001,"15min":2.7360000000000002},"last":0.92400000000000004},{"interface":"front","average":{"1min":1.1659999999999999,"5min":1.238,"15min":1.2729999999999999},"min":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.569,"5min":1.7549999999999999,"15min":2.6859999999999999},"last":0.96299999999999997}]},{"osd":3,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.383,"5min":1.419,"15min":1.413},"min":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.671,"5min":2.5910000000000002,"15min":2.5910000000000002},"last":1.046},{"interface":"front","average":{"1min":1.272,"5min":1.292,"15min":1.319},"min":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.7849999999999999,"5min":1.9079999999999999,"15min":2.823},"last":1.1339999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.419,"5min":1.448,"15min":1.4750000000000001},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.756,"5min":2.0750000000000002,"15min":2.4430000000000001},"last":1.421},{"interface":"front","average":{"1min":1.427,"5min":1.431,"15min":1.4550000000000001},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.0419999999999998,"5min":2.1080000000000001,"15min":2.899},"last":1.0609999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.4890000000000001,"5min":1.4930000000000001,"15min":1.486},"min":{"1min":0.872,"5min":0.872,"15min":0.872},"max":{"1min":1.9990000000000001,"5min":2.2559999999999998,"15min":3.004},"last":1.0109999999999999},{"interface":"front","average":{"1min":1.538,"5min":1.585,"15min":1.581},"min":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9399999999999999,"5min":2.3180000000000001,"15min":2.5129999999999999},"last":1.4590000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.673,"5min":1.613,"15min":1.6259999999999999},"min":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"max":{"1min":2.2770000000000001,"5min":2.2770000000000001,"15min":2.931},"last":1.6779999999999999},{"interface":"front","average":{"1min":1.6220000000000001,"5min":1.6220000000000001,"15min":1.621},"min":{"1min":1.139,"5min":1.0469999999999999,"15min":1.002},"max":{"1min":2.1240000000000001,"5min":2.444,"15min":3.1190000000000002},"last":1.4470000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:14:33 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.7070000000000001,"15min":1.71},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.0640000000000001,"5min":2.5009999999999999,"15min":2.5009999999999999},"last":1.609},{"interface":"front","average":{"1min":1.6779999999999999,"5min":1.7110000000000001,"15min":1.73},"min":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.165,"5min":2.3929999999999998,"15min":2.6739999999999999},"last":1.532}]}]},{"osd":0,"up_from":13,"seq":55834575973,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48216,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323624,"statfs":{"total":96636764160,"available":96587390976,"internally_reserved":0,"allocated":3215360,"data_stored":1882788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.97799999999999998,"15min":0.98199999999999998},"min":{"1min":0.72199999999999998,"5min":0.626,"15min":0.626},"max":{"1min":1.536,"5min":1.5629999999999999,"15min":2.6440000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.2190000000000001,"5min":1.1579999999999999,"15min":1.157},"min":{"1min":0.77600000000000002,"5min":0.67700000000000005,"15min":0.67700000000000005},"max":{"1min":1.7490000000000001,"5min":2.2480000000000002,"15min":2.2480000000000002},"last":0.94399999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.28,"5min":1.2090000000000001,"15min":1.1950000000000001},"min":{"1min":0.91500000000000004,"5min":0.55800000000000005,"15min":0.55800000000000005},"max":{"1min":1.784,"5min":1.8520000000000001,"15min":1.8640000000000001},"last":1.222},{"interface":"front","average":{"1min":1.198,"5min":1.2829999999999999,"15min":1.2729999999999999},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.6359999999999999,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":0.77500000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4179999999999999,"15min":1.399},"min":{"1min":1.032,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.804,"5min":1.8440000000000001,"15min":2.0720000000000001},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.397,"15min":1.419},"min":{"1min":1.0649999999999999,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.6539999999999999,"5min":2.069,"15min":2.194},"last":1.2949999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.2949999999999999,"5min":1.3029999999999999,"15min":1.3260000000000001},"min":{"1min":0.95899999999999996,"5min":0.90500000000000003,"15min":0.90500000000000003},"max":{"1min":1.7190000000000001,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.5329999999999999,"15min":1.522},"min":{"1min":1.2989999999999999,"5min":1.008,"15min":1.008},"max":{"1min":2.0670000000000002,"5min":2.0899999999999999,"15min":2.4889999999999999},"last":1.4490000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.401,"15min":1.4019999999999999},"min":{"1min":0.98599999999999999,"5min":0.98599999999999999,"15min":0.98199999999999998},"max":{"1min":1.7729999999999999,"5min":2.0830000000000002,"15min":2.1339999999999999},"last":1.3240000000000001},{"interface":"front","average":{"1min":1.4890000000000001,"5min":1.581,"15min":1.597},"min":{"1min":1.163,"5min":1.149,"15min":1.149},"max":{"1min":1.883,"5min":2.0489999999999999,"15min":2.2879999999999998},"last":1.1559999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.649,"5min":1.641,"15min":1.667},"min":{"1min":1.46,"5min":1.21,"15min":1.1379999999999999},"max":{"1min":1.867,"5min":2.0910000000000002,"15min":2.3969999999999998},"last":1.4159999999999999},{"interface":"front","average":{"1min":1.4730000000000001,"5min":1.514,"15min":1.524},"min":{"1min":1.1180000000000001,"5min":0.999,"15min":0.999},"max":{"1min":1.853,"5min":1.8979999999999999,"15min":2.2799999999999998},"last":1.4019999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:14:09 2023","interfaces":[{"interface":"back","average":{"1min":1.6739999999999999,"5min":1.6719999999999999,"15min":1.671},"min":{"1min":1.3440000000000001,"5min":1.0429999999999999,"15min":1.0429999999999999},"max":{"1min":2.2120000000000002,"5min":2.4630000000000001,"15min":2.4630000000000001},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.621,"5min":1.6479999999999999,"15min":1.647},"min":{"1min":1.2709999999999999,"5min":1.2470000000000001,"15min":1.113},"max":{"1min":1.9610000000000001,"5min":2.1440000000000001,"15min":2.3620000000000001},"last":1.476}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":24,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":25,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:14:51.669 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:14:51.669 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T18:14:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.947+0000 7f9d7f59e640 1 Processor -- start 2023-10-14T18:14:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.948+0000 7f9d7f59e640 1 -- start start 2023-10-14T18:14:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d80159ac0 0x7f9d8015e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 0x7f9d800c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d8015e5c0 con 0x7f9d800704e0 2023-10-14T18:14:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d8006b810 con 0x7f9d80159ac0 2023-10-14T18:14:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d800cc810 con 0x7f9d800cc270 2023-10-14T18:14:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d80159ac0 0x7f9d8015e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9d80159ac0 0x7f9d8015e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36456/0 (socket says 172.21.15.29:36456) 2023-10-14T18:14:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.949+0000 7f9d7e59c640 1 -- 172.21.15.29:0/2929407169 learned_addr learned my addr 172.21.15.29:0/2929407169 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7dd9b640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 0x7f9d800c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7e59c640 1 -- 172.21.15.29:0/2929407169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 msgr2=0x7f9d800c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7e59c640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 0x7f9d800c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7e59c640 1 -- 172.21.15.29:0/2929407169 >> v1:172.21.15.186:6789/0 conn(0x7f9d800704e0 legacy=0x7f9d8006bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7e59c640 1 -- 172.21.15.29:0/2929407169 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8006b1c0 con 0x7f9d80159ac0 2023-10-14T18:14:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.950+0000 7f9d7dd9b640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 0x7f9d800c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.951+0000 7f9d7e59c640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d80159ac0 0x7f9d8015e080 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009870 tx=0x7f9d7002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f3c3580d6c9e293 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.951+0000 7f9d7d59a640 1 -- 172.21.15.29:0/2929407169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70031640 con 0x7f9d80159ac0 2023-10-14T18:14:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.951+0000 7f9d7d59a640 1 -- 172.21.15.29:0/2929407169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d700317a0 con 0x7f9d80159ac0 2023-10-14T18:14:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.951+0000 7f9d7d59a640 1 -- 172.21.15.29:0/2929407169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d70031b30 con 0x7f9d80159ac0 2023-10-14T18:14:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 -- 172.21.15.29:0/2929407169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d80159ac0 msgr2=0x7f9d8015e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d80159ac0 0x7f9d8015e080 secure :-1 s=READY pgs=2390 cs=0 l=1 rev1=1 crypto rx=0x7f9d70009870 tx=0x7f9d7002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 -- 172.21.15.29:0/2929407169 shutdown_connections 2023-10-14T18:14:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d800cc270 0x7f9d800c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/2929407169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d80159ac0 0x7f9d8015e080 unknown :-1 s=CLOSED pgs=2390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 -- 172.21.15.29:0/2929407169 >> 172.21.15.29:0/2929407169 conn(0x7f9d800ce6a0 msgr2=0x7f9d800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 -- 172.21.15.29:0/2929407169 shutdown_connections 2023-10-14T18:14:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.952+0000 7f9d7f59e640 1 -- 172.21.15.29:0/2929407169 wait complete. 2023-10-14T18:14:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.953+0000 7f9d7f59e640 1 Processor -- start 2023-10-14T18:14:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.954+0000 7f9d7f59e640 1 -- start start 2023-10-14T18:14:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.954+0000 7f9d7f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d800704e0 0x7f9d800c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.954+0000 7f9d7f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d80159ac0 0x7f9d800ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d801452f0 con 0x7f9d800cc270 2023-10-14T18:14:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80164d80 con 0x7f9d800704e0 2023-10-14T18:14:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d80165060 con 0x7f9d80159ac0 2023-10-14T18:14:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36458/0 (socket says 172.21.15.29:36458) 2023-10-14T18:14:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 -- 172.21.15.29:0/3185680106 learned_addr learned my addr 172.21.15.29:0/3185680106 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 -- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d80159ac0 msgr2=0x7f9d800ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d80159ac0 0x7f9d800ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 -- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d800704e0 msgr2=0x7f9d800c50a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d800704e0 0x7f9d800c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 -- 172.21.15.29:0/3185680106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d700094d0 con 0x7f9d800cc270 2023-10-14T18:14:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.955+0000 7f9d7dd9b640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c7f0 tx=0x7f9d7400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.956+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74007a30 con 0x7f9d800cc270 2023-10-14T18:14:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.956+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d74007b90 con 0x7f9d800cc270 2023-10-14T18:14:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.956+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d7400ce40 con 0x7f9d800cc270 2023-10-14T18:14:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.956+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d80145590 con 0x7f9d800cc270 2023-10-14T18:14:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.956+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d80165340 con 0x7f9d800cc270 2023-10-14T18:14:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.957+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d80145780 con 0x7f9d800cc270 2023-10-14T18:14:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.958+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9d740113f0 con 0x7f9d800cc270 2023-10-14T18:14:51.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.959+0000 7f9d6effd640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 0x7f9d6006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.959+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9d74089ac0 con 0x7f9d800cc270 2023-10-14T18:14:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.959+0000 7f9d7e59c640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 0x7f9d6006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.960+0000 7f9d7e59c640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 0x7f9d6006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036000 tx=0x7f9d70030000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:51.963+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9d740548d0 con 0x7f9d800cc270 2023-10-14T18:14:52.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.218+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7f9d8004e9b0 con 0x7f9d800cc270 2023-10-14T18:14:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.221+0000 7f9d6effd640 1 -- 172.21.15.29:0/3185680106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v49) v1 ==== 190+0+147 (secure 0 0 0) 0x7f9d74089e60 con 0x7f9d800cc270 2023-10-14T18:14:52.222 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T18:14:52.222 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T18:14:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T18:14:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T18:14:52.223 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T18:14:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T18:14:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 msgr2=0x7f9d6006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 0x7f9d6006d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9d70036000 tx=0x7f9d70030000 comp rx=0 tx=0).stop 2023-10-14T18:14:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 msgr2=0x7f9d8014de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 secure :-1 s=READY pgs=2391 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400c7f0 tx=0x7f9d7400cbf0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 shutdown_connections 2023-10-14T18:14:52.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f9d6006af90 0x7f9d6006d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9d80159ac0 0x7f9d800ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9d800cc270 0x7f9d8014de90 unknown :-1 s=CLOSED pgs=2391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 --2- 172.21.15.29:0/3185680106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9d800704e0 0x7f9d800c50a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 >> 172.21.15.29:0/3185680106 conn(0x7f9d800ce6a0 msgr2=0x7f9d801506c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 shutdown_connections 2023-10-14T18:14:52.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.224+0000 7f9d7f59e640 1 -- 172.21.15.29:0/3185680106 wait complete. 2023-10-14T18:14:52.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:14:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.469+0000 7f3814c4d640 1 Processor -- start 2023-10-14T18:14:52.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.469+0000 7f3814c4d640 1 -- start start 2023-10-14T18:14:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3814c4d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f381015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3814c4d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f38100adbe0 0x7f38100ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3814c4d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38100cd4a0 con 0x7f38100ccb00 2023-10-14T18:14:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3814c4d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f381015ac60 con 0x7f38100adbe0 2023-10-14T18:14:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3814c4d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3810070030 con 0x7f3810153740 2023-10-14T18:14:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3807fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f38100ccb00 0x7f38100ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58116/0 (socket says 172.21.15.29:58116) 2023-10-14T18:14:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.471+0000 7f3807fff640 1 -- 172.21.15.29:0/2172520743 learned_addr learned my addr 172.21.15.29:0/2172520743 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380ffff640 1 --2- 172.21.15.29:0/2172520743 >> v2:172.21.15.186:3300/0 conn(0x7f38100adbe0 0x7f38100ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380f7fe640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f381015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370785224 0 0) 0x7f38100cd4a0 con 0x7f38100ccb00 2023-10-14T18:14:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f37fc005730 con 0x7f38100ccb00 2023-10-14T18:14:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1532125355 0 0) 0x7f37fc005730 con 0x7f38100ccb00 2023-10-14T18:14:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 msgr2=0x7f381015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f381015a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 >> v2:172.21.15.186:3300/0 conn(0x7f38100adbe0 msgr2=0x7f38100ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 --2- 172.21.15.29:0/2172520743 >> v2:172.21.15.186:3300/0 conn(0x7f38100adbe0 0x7f38100ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3810070310 con 0x7f38100ccb00 2023-10-14T18:14:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f37f8001670 con 0x7f38100ccb00 2023-10-14T18:14:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 >> v1:172.21.15.29:6789/0 conn(0x7f38100ccb00 legacy=0x7f38100ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f37fc005b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.472+0000 7f380effd640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37fc006150 0x7f37fc0069d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380effd640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc006f10 0x7f37fc007390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381015ac60 con 0x7f37fc006f10 2023-10-14T18:14:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3810070030 con 0x7f37fc006150 2023-10-14T18:14:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37fc00dc40 con 0x7f3810153740 2023-10-14T18:14:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc006f10 0x7f37fc007390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f3807fff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37fc006150 0x7f37fc0069d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380f7fe640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f37fc005b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 -- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 msgr2=0x7f37fc005b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3810153740 0x7f37fc005b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 -- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37fc006150 msgr2=0x7f37fc0069d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:52.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37fc006150 0x7f37fc0069d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f380ffff640 1 -- 172.21.15.29:0/2172520743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc005730 con 0x7f37fc006f10 2023-10-14T18:14:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.473+0000 7f3807fff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f37fc006150 0x7f37fc0069d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.474+0000 7f380ffff640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc006f10 0x7f37fc007390 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f380800acb0 tx=0x7f37fc007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c352e4b8e103dfac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380800f040 con 0x7f37fc006f10 2023-10-14T18:14:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3808030070 con 0x7f37fc006f10 2023-10-14T18:14:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f380effd640 1 -- 172.21.15.29:0/2172520743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f380802ce50 con 0x7f37fc006f10 2023-10-14T18:14:52.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 -- 172.21.15.29:0/2172520743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc006f10 msgr2=0x7f37fc007390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 --2- 172.21.15.29:0/2172520743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37fc006f10 0x7f37fc007390 secure :-1 s=READY pgs=2393 cs=0 l=1 rev1=1 crypto rx=0x7f380800acb0 tx=0x7f37fc007a00 comp rx=0 tx=0).stop 2023-10-14T18:14:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f380ffff640 1 -- 172.21.15.29:0/2172520743 reap_dead start 2023-10-14T18:14:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 -- 172.21.15.29:0/2172520743 shutdown_connections 2023-10-14T18:14:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 -- 172.21.15.29:0/2172520743 >> 172.21.15.29:0/2172520743 conn(0x7f38100ce6a0 msgr2=0x7f38100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 -- 172.21.15.29:0/2172520743 shutdown_connections 2023-10-14T18:14:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.475+0000 7f3814c4d640 1 -- 172.21.15.29:0/2172520743 wait complete. 2023-10-14T18:14:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.477+0000 7f3814c4d640 1 Processor -- start 2023-10-14T18:14:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.477+0000 7f3814c4d640 1 -- start start 2023-10-14T18:14:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37fc006f10 0x7f3810152960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 0x7f381006bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381006aab0 con 0x7f38100c4b60 2023-10-14T18:14:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f381006ad90 con 0x7f381006a540 2023-10-14T18:14:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.478+0000 7f3814c4d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f381006b070 con 0x7f37fc006f10 2023-10-14T18:14:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36476/0 (socket says 172.21.15.29:36476) 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 -- 172.21.15.29:0/3175478354 learned_addr learned my addr 172.21.15.29:0/3175478354 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f3807fff640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 0x7f381006bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 -- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37fc006f10 msgr2=0x7f3810152960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380f7fe640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37fc006f10 0x7f3810152960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37fc006f10 0x7f3810152960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 -- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 msgr2=0x7f381006bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 0x7f381006bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 -- 172.21.15.29:0/3175478354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f381006b350 con 0x7f38100c4b60 2023-10-14T18:14:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f3807fff640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 0x7f381006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.479+0000 7f380ffff640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f3808038040 tx=0x7f380800a7b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:52.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.480+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808046070 con 0x7f38100c4b60 2023-10-14T18:14:52.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.480+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f380800f000 con 0x7f38100c4b60 2023-10-14T18:14:52.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.480+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3808004450 con 0x7f38100c4b60 2023-10-14T18:14:52.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.480+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f381006b630 con 0x7f38100c4b60 2023-10-14T18:14:52.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.480+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3810153060 con 0x7f38100c4b60 2023-10-14T18:14:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.481+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f381006aed0 con 0x7f38100c4b60 2023-10-14T18:14:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.482+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f38080045b0 con 0x7f38100c4b60 2023-10-14T18:14:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.483+0000 7f380cff9640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 0x7f37f006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.483+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f38080ab4c0 con 0x7f38100c4b60 2023-10-14T18:14:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.483+0000 7f380f7fe640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 0x7f37f006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.484+0000 7f380f7fe640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 0x7f37f006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38101f17d0 tx=0x7f3800008040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.488+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f381006aed0 con 0x7f38100c4b60 2023-10-14T18:14:52.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.723+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f381006aed0 con 0x7f38100c4b60 2023-10-14T18:14:52.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.724+0000 7f380cff9640 1 -- 172.21.15.29:0/3175478354 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f381006aed0 con 0x7f38100c4b60 2023-10-14T18:14:52.726 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:14:52.726 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:14:52.726 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:14:52.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 msgr2=0x7f37f006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 0x7f37f006d450 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f38101f17d0 tx=0x7f3800008040 comp rx=0 tx=0).stop 2023-10-14T18:14:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 msgr2=0x7f38100c21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 secure :-1 s=READY pgs=2394 cs=0 l=1 rev1=1 crypto rx=0x7f3808038040 tx=0x7f380800a7b0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 shutdown_connections 2023-10-14T18:14:52.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f38100c4b60 0x7f38100c21d0 unknown :-1 s=CLOSED pgs=2394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f381006a540 0x7f381006bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f37f006af90 0x7f37f006d450 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 --2- 172.21.15.29:0/3175478354 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37fc006f10 0x7f3810152960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:52.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.727+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 >> 172.21.15.29:0/3175478354 conn(0x7f38100ce6a0 msgr2=0x7f38100ae100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:52.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.728+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 shutdown_connections 2023-10-14T18:14:52.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:52.728+0000 7f3814c4d640 1 -- 172.21.15.29:0/3175478354 wait complete. 2023-10-14T18:14:53.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 98, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:14:46.008237+0000', 'modified': '2023-10-14T18:14:49.076146+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7984}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7984': {'gid': 7984, 'name': 'c', 'rank': 0, 'incarnation': 95, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/1451005601', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6835', 'nonce': 1451005601}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 1451005601}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [25], 'metadata_pool': 24, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:14:53.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:14:53.151 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T18:14:53.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.392+0000 7f5065bcc640 1 Processor -- start 2023-10-14T18:14:53.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.392+0000 7f5065bcc640 1 -- start start 2023-10-14T18:14:53.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f5065bcc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 0x7f5060069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f5065bcc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506006a6f0 con 0x7f506006a2e0 2023-10-14T18:14:53.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f5065bcc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50600ac890 con 0x7f506015d150 2023-10-14T18:14:53.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f5065bcc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f506015d530 con 0x7f50600ad350 2023-10-14T18:14:53.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f505ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f506006a2e0 0x7f50600c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58118/0 (socket says 172.21.15.29:58118) 2023-10-14T18:14:53.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.394+0000 7f505ffff640 1 -- 172.21.15.29:0/687374087 learned_addr learned my addr 172.21.15.29:0/687374087 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:53.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.395+0000 7f505effd640 1 --2- 172.21.15.29:0/687374087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 0x7f5060069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.395+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 330791703 0 0) 0x7f506006a6f0 con 0x7f506006a2e0 2023-10-14T18:14:53.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.395+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f504c003520 con 0x7f506006a2e0 2023-10-14T18:14:53.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2577745815 0 0) 0x7f50600ac890 con 0x7f506015d150 2023-10-14T18:14:53.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506006a6f0 con 0x7f506015d150 2023-10-14T18:14:53.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2943981774 0 0) 0x7f504c003520 con 0x7f506006a2e0 2023-10-14T18:14:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505effd640 1 -- 172.21.15.29:0/687374087 >> v1:172.21.15.186:6789/0 conn(0x7f506015d150 legacy=0x7f5060160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505effd640 1 -- 172.21.15.29:0/687374087 >> v1:172.21.15.29:6789/0 conn(0x7f506006a2e0 legacy=0x7f50600c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.396+0000 7f505effd640 1 -- 172.21.15.29:0/687374087 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5060161650 con 0x7f50600ad350 2023-10-14T18:14:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.397+0000 7f505effd640 1 --2- 172.21.15.29:0/687374087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 0x7f5060069da0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5048009dc0 tx=0x7f504802bde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4fa48de6ba2f506b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058001670 con 0x7f50600ad350 2023-10-14T18:14:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504802c910 con 0x7f50600ad350 2023-10-14T18:14:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f505e7fc640 1 -- 172.21.15.29:0/687374087 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504802cca0 con 0x7f50600ad350 2023-10-14T18:14:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 -- 172.21.15.29:0/687374087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 msgr2=0x7f5060069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 --2- 172.21.15.29:0/687374087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 0x7f5060069da0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5048009dc0 tx=0x7f504802bde0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 -- 172.21.15.29:0/687374087 shutdown_connections 2023-10-14T18:14:53.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 --2- 172.21.15.29:0/687374087 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50600ad350 0x7f5060069da0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 -- 172.21.15.29:0/687374087 >> 172.21.15.29:0/687374087 conn(0x7f50600ce9b0 msgr2=0x7f50600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.398+0000 7f5065bcc640 1 -- 172.21.15.29:0/687374087 shutdown_connections 2023-10-14T18:14:53.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.399+0000 7f5065bcc640 1 -- 172.21.15.29:0/687374087 wait complete. 2023-10-14T18:14:53.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.400+0000 7f5065bcc640 1 Processor -- start 2023-10-14T18:14:53.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.400+0000 7f5065bcc640 1 -- start start 2023-10-14T18:14:53.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 0x7f50601f3240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f506015d150 0x7f50600b2ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50601f3780 con 0x7f50600ad350 2023-10-14T18:14:53.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50600c3b60 con 0x7f506006a2e0 2023-10-14T18:14:53.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.401+0000 7f5065bcc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50600c3e40 con 0x7f506015d150 2023-10-14T18:14:53.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36490/0 (socket says 172.21.15.29:36490) 2023-10-14T18:14:53.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 -- 172.21.15.29:0/4168181445 learned_addr learned my addr 172.21.15.29:0/4168181445 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 -- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f506015d150 msgr2=0x7f50600b2ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505ffff640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f506015d150 0x7f50600b2ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505f7fe640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 0x7f50601f3240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f506015d150 0x7f50600b2ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 -- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 msgr2=0x7f50601f3240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 0x7f50601f3240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505effd640 1 -- 172.21.15.29:0/4168181445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5048009a70 con 0x7f50600ad350 2023-10-14T18:14:53.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.402+0000 7f505f7fe640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 0x7f50601f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:53.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f505effd640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f5048037040 tx=0x7f5048004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f5057fff640 1 -- 172.21.15.29:0/4168181445 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048009450 con 0x7f50600ad350 2023-10-14T18:14:53.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f5057fff640 1 -- 172.21.15.29:0/4168181445 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5048003990 con 0x7f50600ad350 2023-10-14T18:14:53.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50600c4120 con 0x7f50600ad350 2023-10-14T18:14:53.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f5057fff640 1 -- 172.21.15.29:0/4168181445 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5048003d40 con 0x7f50600ad350 2023-10-14T18:14:53.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.403+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50600c46e0 con 0x7f50600ad350 2023-10-14T18:14:53.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.405+0000 7f5057fff640 1 -- 172.21.15.29:0/4168181445 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5048004030 con 0x7f50600ad350 2023-10-14T18:14:53.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.406+0000 7f5057fff640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 0x7f504006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.406+0000 7f5057fff640 1 -- 172.21.15.29:0/4168181445 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f50480ac660 con 0x7f50600ad350 2023-10-14T18:14:53.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.406+0000 7f505f7fe640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 0x7f504006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.407+0000 7f5068833740 1 Processor -- start 2023-10-14T18:14:53.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.407+0000 7f5068833740 1 -- start start 2023-10-14T18:14:53.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.407+0000 7f505f7fe640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 0x7f504006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f50500011e0 tx=0x7f5050009040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 0x556a6f100e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f1013c0 0x556a6f131250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f0217a0 con 0x556a6f12e360 2023-10-14T18:14:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f073740 con 0x556a6f1013c0 2023-10-14T18:14:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a6f070e30 con 0x556a6f12df80 2023-10-14T18:14:53.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36498/0 (socket says 172.21.15.29:36498) 2023-10-14T18:14:53.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 -- 172.21.15.29:0/3457223963 learned_addr learned my addr 172.21.15.29:0/3457223963 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 0x556a6f100e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505ffff640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f1013c0 0x556a6f131250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 -- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 msgr2=0x556a6f100e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 0x556a6f100e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 -- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f1013c0 msgr2=0x556a6f131250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.408+0000 7f505effd640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f1013c0 0x556a6f131250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f505effd640 1 -- 172.21.15.29:0/3457223963 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a6f030910 con 0x556a6f12e360 2023-10-14T18:14:53.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 0x556a6f100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:53.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f505effd640 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f50480aa860 tx=0x7f504807ada0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b49ca2bedc918eaa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f50557fa640 1 -- 172.21.15.29:0/3457223963 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504807f0d0 con 0x556a6f12e360 2023-10-14T18:14:53.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f50557fa640 1 -- 172.21.15.29:0/3457223963 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504807b1a0 con 0x556a6f12e360 2023-10-14T18:14:53.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.409+0000 7f50557fa640 1 -- 172.21.15.29:0/3457223963 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504807f2f0 con 0x556a6f12e360 2023-10-14T18:14:53.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 -- 172.21.15.29:0/3457223963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 msgr2=0x556a6f130d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 secure :-1 s=READY pgs=2397 cs=0 l=1 rev1=1 crypto rx=0x7f50480aa860 tx=0x7f504807ada0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 -- 172.21.15.29:0/3457223963 shutdown_connections 2023-10-14T18:14:53.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f1013c0 0x556a6f131250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f12e360 0x556a6f130d10 unknown :-1 s=CLOSED pgs=2397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 --2- 172.21.15.29:0/3457223963 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12df80 0x556a6f100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 -- 172.21.15.29:0/3457223963 >> 172.21.15.29:0/3457223963 conn(0x556a6f103190 msgr2=0x556a6f14bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 -- 172.21.15.29:0/3457223963 shutdown_connections 2023-10-14T18:14:53.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.410+0000 7f5068833740 1 -- 172.21.15.29:0/3457223963 wait complete. 2023-10-14T18:14:53.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.411+0000 7f5068833740 1 -- start start 2023-10-14T18:14:53.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.411+0000 7f5068833740 1 Processor -- start 2023-10-14T18:14:53.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f152780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f12d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f06c720 con 0x556a6f134890 2023-10-14T18:14:53.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f00a970 con 0x556a6f12df80 2023-10-14T18:14:53.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a6f00a610 con 0x556a6f12e360 2023-10-14T18:14:53.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f5068833740 10 client.0 _ll_register_callbacks cb 0x556a6f02c9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:14:53.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36508/0 (socket says 172.21.15.29:36508) 2023-10-14T18:14:53.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f505ffff640 1 -- 172.21.15.29:0/3093629176 learned_addr learned my addr 172.21.15.29:0/3093629176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f152780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.412+0000 7f505effd640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f12d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 -- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 msgr2=0x556a6f12d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f12d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 -- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 msgr2=0x556a6f152780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f152780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 -- 172.21.15.29:0/3093629176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50600c46e0 con 0x556a6f134890 2023-10-14T18:14:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f505ffff640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f5058007fc0 tx=0x7f505800e590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2462d74318ee7b5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f503e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:14:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058005ad0 con 0x556a6f134890 2023-10-14T18:14:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5058005c30 con 0x556a6f134890 2023-10-14T18:14:53.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x556a6f093160 con 0x556a6f134890 2023-10-14T18:14:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5058001670 con 0x556a6f134890 2023-10-14T18:14:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f5068833740 10 client.10857 fetch_fsmap learned FSMap version 98 2023-10-14T18:14:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.413+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x556a6f06ada0 con 0x556a6f134890 2023-10-14T18:14:53.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x556a6f093160 con 0x556a6f134890 2023-10-14T18:14:53.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 98) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f50580070d0 con 0x556a6f134890 2023-10-14T18:14:53.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 10 client.10857 fetch_fsmap finished waiting for FSMap version 98 2023-10-14T18:14:53.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 10 client.10857 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T18:14:53.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 20 client.10857 populate_metadata read hostname 'smithi029' 2023-10-14T18:14:53.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f133590 0x556a6f167890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 4 client.10857 mds_command: new command op to 7984 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:14:53.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 --> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556a6f03ecc0 con 0x556a6f133590 2023-10-14T18:14:53.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.414+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f133590 0x556a6f167890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.415+0000 7f505f7fe640 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f133590 0x556a6f167890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72e81d567e1d7c9d server_cookie=ecb169b1b46226bb in_seq=0 out_seq=0 2023-10-14T18:14:53.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.415+0000 7f503e7fc640 10 client.10857 ms_handle_connect on v2:172.21.15.29:6835/1451005601 2023-10-14T18:14:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f503e7fc640 1 -- 172.21.15.29:0/3093629176 <== mds.0 v2:172.21.15.29:6835/1451005601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x556a6f03ecc0 con 0x556a6f133590 2023-10-14T18:14:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f503e7fc640 10 client.10857 handle_command_reply: tid=0 2023-10-14T18:14:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 client.10857 shutdown 2023-10-14T18:14:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 msgr2=0x556a6f1503e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 secure :-1 s=READY pgs=2398 cs=0 l=1 rev1=1 crypto rx=0x7f5058007fc0 tx=0x7f505800e590 comp rx=0 tx=0).stop 2023-10-14T18:14:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 shutdown_connections 2023-10-14T18:14:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f133590 0x556a6f167890 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f1503e0 unknown :-1 s=CLOSED pgs=2398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f12d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 --2- 172.21.15.29:0/3093629176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f152780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 >> 172.21.15.29:0/3093629176 conn(0x556a6f1013c0 msgr2=0x556a6f104f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f503e7fc640 0 client.10857 ms_handle_reset on v2:172.21.15.29:6835/1451005601 2023-10-14T18:14:53.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 shutdown_connections 2023-10-14T18:14:53.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 1 -- 172.21.15.29:0/3093629176 wait complete. 2023-10-14T18:14:53.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.416+0000 7f5068833740 20 client.10857 trim_cache size 0 max 16384 2023-10-14T18:14:53.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.433+0000 7f5068833740 1 Processor -- start 2023-10-14T18:14:53.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.433+0000 7f5068833740 1 -- start start 2023-10-14T18:14:53.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f153020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f104c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6edd0f20 con 0x556a6f134890 2023-10-14T18:14:53.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f102150 con 0x556a6f12df80 2023-10-14T18:14:53.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a6f105a40 con 0x556a6f12e360 2023-10-14T18:14:53.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36512/0 (socket says 172.21.15.29:36512) 2023-10-14T18:14:53.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.434+0000 7f505ffff640 1 -- 172.21.15.29:0/35689446 learned_addr learned my addr 172.21.15.29:0/35689446 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 -- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 msgr2=0x556a6f104c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f104c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 -- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 msgr2=0x556a6f153020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f153020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 -- 172.21.15.29:0/35689446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556a6f009b80 con 0x556a6f134890 2023-10-14T18:14:53.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f505ffff640 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f5058000c00 tx=0x7f505800c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb36c78e25f667eb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f5055ffb640 1 -- 172.21.15.29:0/35689446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058007590 con 0x556a6f134890 2023-10-14T18:14:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f5055ffb640 1 -- 172.21.15.29:0/35689446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5058027070 con 0x556a6f134890 2023-10-14T18:14:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.435+0000 7f5055ffb640 1 -- 172.21.15.29:0/35689446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058005b10 con 0x556a6f134890 2023-10-14T18:14:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 -- 172.21.15.29:0/35689446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 msgr2=0x556a6f14f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 secure :-1 s=READY pgs=2399 cs=0 l=1 rev1=1 crypto rx=0x7f5058000c00 tx=0x7f505800c000 comp rx=0 tx=0).stop 2023-10-14T18:14:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 -- 172.21.15.29:0/35689446 shutdown_connections 2023-10-14T18:14:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f220 unknown :-1 s=CLOSED pgs=2399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f104c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 --2- 172.21.15.29:0/35689446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f153020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 -- 172.21.15.29:0/35689446 >> 172.21.15.29:0/35689446 conn(0x556a6f1013c0 msgr2=0x556a6f107050 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 -- 172.21.15.29:0/35689446 shutdown_connections 2023-10-14T18:14:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.436+0000 7f5068833740 1 -- 172.21.15.29:0/35689446 wait complete. 2023-10-14T18:14:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.437+0000 7f5068833740 1 -- start start 2023-10-14T18:14:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.437+0000 7f5068833740 1 Processor -- start 2023-10-14T18:14:53.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f1a7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f14f220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f12b280 con 0x556a6f134890 2023-10-14T18:14:53.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x556a6f1ad710 con 0x556a6f12df80 2023-10-14T18:14:53.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x556a6f1ad940 con 0x556a6f12e360 2023-10-14T18:14:53.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f5068833740 10 client.0 _ll_register_callbacks cb 0x7f5058001420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:14:53.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.438+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36520/0 (socket says 172.21.15.29:36520) 2023-10-14T18:14:53.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 -- 172.21.15.29:0/1118673743 learned_addr learned my addr 172.21.15.29:0/1118673743 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 -- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 msgr2=0x556a6f14f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f14f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505f7fe640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f1a7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 -- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 msgr2=0x556a6f1a7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f1a7820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 -- 172.21.15.29:0/1118673743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556a6f06ada0 con 0x556a6f134890 2023-10-14T18:14:53.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f505ffff640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f5058001de0 tx=0x7f50580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d12a8dd53ce4a742 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f503e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:14:53.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5058005da0 con 0x556a6f134890 2023-10-14T18:14:53.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5058029070 con 0x556a6f134890 2023-10-14T18:14:53.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.439+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f504807b1a0 con 0x556a6f134890 2023-10-14T18:14:53.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=98) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5058013440 con 0x556a6f134890 2023-10-14T18:14:53.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 10 client.10863 fetch_fsmap learned FSMap version 98 2023-10-14T18:14:53.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=98,monmap=2+}) v3 -- 0x556a6f02f170 con 0x556a6f134890 2023-10-14T18:14:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f504807b1a0 con 0x556a6f134890 2023-10-14T18:14:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 98) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f50580135a0 con 0x556a6f134890 2023-10-14T18:14:53.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 10 client.10863 fetch_fsmap finished waiting for FSMap version 98 2023-10-14T18:14:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 10 client.10863 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T18:14:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 20 client.10863 populate_metadata read hostname 'smithi029' 2023-10-14T18:14:53.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f1ac9b0 0x556a6f1acd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 4 client.10863 mds_command: new command op to 7984 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:14:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 --> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556a6f065700 con 0x556a6f1ac9b0 2023-10-14T18:14:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.440+0000 7f505f7fe640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f1ac9b0 0x556a6f1acd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f505f7fe640 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f1ac9b0 0x556a6f1acd90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbeaa5728480cc41 server_cookie=fa1344a6a5b71fed in_seq=0 out_seq=0 2023-10-14T18:14:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f503e7fc640 10 client.10863 ms_handle_connect on v2:172.21.15.29:6835/1451005601 2023-10-14T18:14:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f503e7fc640 1 -- 172.21.15.29:0/1118673743 <== mds.0 v2:172.21.15.29:6835/1451005601 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556a6f065700 con 0x556a6f1ac9b0 2023-10-14T18:14:53.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f503e7fc640 10 client.10863 handle_command_reply: tid=0 2023-10-14T18:14:53.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f5068833740 1 client.10863 shutdown 2023-10-14T18:14:53.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 msgr2=0x556a6f14f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.441+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 secure :-1 s=READY pgs=2400 cs=0 l=1 rev1=1 crypto rx=0x7f5058001de0 tx=0x7f50580057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 shutdown_connections 2023-10-14T18:14:53.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:6835/1451005601,v1:172.21.15.29:6837/1451005601] conn(0x556a6f1ac9b0 0x556a6f1acd90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x556a6f134890 0x556a6f14f760 unknown :-1 s=CLOSED pgs=2400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x556a6f12e360 0x556a6f14f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f503e7fc640 0 client.10863 ms_handle_reset on v2:172.21.15.29:6835/1451005601 2023-10-14T18:14:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 --2- 172.21.15.29:0/1118673743 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x556a6f12df80 0x556a6f1a7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 >> 172.21.15.29:0/1118673743 conn(0x556a6f1013c0 msgr2=0x556a6f104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 shutdown_connections 2023-10-14T18:14:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 1 -- 172.21.15.29:0/1118673743 wait complete. 2023-10-14T18:14:53.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5068833740 20 client.10863 trim_cache size 0 max 16384 2023-10-14T18:14:53.465 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:14:53.466 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:14:53.466 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:14:53.466 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7984, 2023-10-14T18:14:53.467 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:14:53.467 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:14:53.467 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T18:14:53.467 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.9149906679999997, 2023-10-14T18:14:53.467 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 98, 2023-10-14T18:14:53.468 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 210, 2023-10-14T18:14:53.468 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 210, 2023-10-14T18:14:53.468 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.212259156 2023-10-14T18:14:53.468 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:14:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 msgr2=0x7f504006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 0x7f504006d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f50500011e0 tx=0x7f5050009040 comp rx=0 tx=0).stop 2023-10-14T18:14:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 msgr2=0x7f50600c2750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.442+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 secure :-1 s=READY pgs=2396 cs=0 l=1 rev1=1 crypto rx=0x7f5048037040 tx=0x7f5048004a40 comp rx=0 tx=0).stop 2023-10-14T18:14:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 shutdown_connections 2023-10-14T18:14:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f504006af90 0x7f504006d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f506015d150 0x7f50600b2ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50600ad350 0x7f50600c2750 unknown :-1 s=CLOSED pgs=2396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 --2- 172.21.15.29:0/4168181445 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f506006a2e0 0x7f50601f3240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 >> 172.21.15.29:0/4168181445 conn(0x7f50600ce9b0 msgr2=0x7f506006f550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 shutdown_connections 2023-10-14T18:14:53.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.443+0000 7f5065bcc640 1 -- 172.21.15.29:0/4168181445 wait complete. 2023-10-14T18:14:53.472 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T18:14:53.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.694+0000 7febc98a2640 1 Processor -- start 2023-10-14T18:14:53.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.694+0000 7febc98a2640 1 -- start start 2023-10-14T18:14:53.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc98a2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7febc40b0b20 0x7febc4164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc98a2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc4071450 con 0x7febc4070fd0 2023-10-14T18:14:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc98a2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febc40cc600 con 0x7febc40adb50 2023-10-14T18:14:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc98a2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febc40cc8e0 con 0x7febc40b0b20 2023-10-14T18:14:53.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc27fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7febc4070fd0 0x7febc40b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58134/0 (socket says 172.21.15.29:58134) 2023-10-14T18:14:53.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc27fc640 1 -- 172.21.15.29:0/410908777 learned_addr learned my addr 172.21.15.29:0/410908777 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:53.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc37fe640 1 --2- 172.21.15.29:0/410908777 >> v2:172.21.15.186:3300/0 conn(0x7febc40b0b20 0x7febc4164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2436925357 0 0) 0x7febc4071450 con 0x7febc4070fd0 2023-10-14T18:14:53.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.696+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac003520 con 0x7febc4070fd0 2023-10-14T18:14:53.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756162434 0 0) 0x7febac003520 con 0x7febc4070fd0 2023-10-14T18:14:53.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 >> v2:172.21.15.186:3300/0 conn(0x7febc40b0b20 msgr2=0x7febc4164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 --2- 172.21.15.29:0/410908777 >> v2:172.21.15.186:3300/0 conn(0x7febc40b0b20 0x7febc4164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 >> v1:172.21.15.186:6789/0 conn(0x7febc40adb50 legacy=0x7febc406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40ccbc0 con 0x7febc4070fd0 2023-10-14T18:14:53.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7febb80019f0 con 0x7febc4070fd0 2023-10-14T18:14:53.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 >> v1:172.21.15.29:6789/0 conn(0x7febc4070fd0 legacy=0x7febc40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febac003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febac0047c0 0x7febac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febac005180 0x7febac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40cc8e0 con 0x7febac003970 2023-10-14T18:14:53.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febac00c020 con 0x7febac0047c0 2023-10-14T18:14:53.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febac00c1e0 con 0x7febac005180 2023-10-14T18:14:53.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.697+0000 7febc2ffd640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febac003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc27fc640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febac0047c0 0x7febac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc37fe640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febac005180 0x7febac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc2ffd640 1 -- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febac005180 msgr2=0x7febac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc2ffd640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febac005180 0x7febac00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc2ffd640 1 -- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febac0047c0 msgr2=0x7febac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:53.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc2ffd640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febac0047c0 0x7febac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc2ffd640 1 -- 172.21.15.29:0/410908777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febac003520 con 0x7febac003970 2023-10-14T18:14:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc37fe640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febac005180 0x7febac00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.698+0000 7febc37fe640 1 -- 172.21.15.29:0/410908777 reap_dead start 2023-10-14T18:14:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.699+0000 7febc2ffd640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febac003d90 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7febb4001750 tx=0x7febac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c67bbc34b116a0c2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.699+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb40052d0 con 0x7febac003970 2023-10-14T18:14:53.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.699+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb402e050 con 0x7febac003970 2023-10-14T18:14:53.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.699+0000 7febc1ffb640 1 -- 172.21.15.29:0/410908777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4003830 con 0x7febac003970 2023-10-14T18:14:53.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 -- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 msgr2=0x7febac003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febac003d90 secure :-1 s=READY pgs=2402 cs=0 l=1 rev1=1 crypto rx=0x7febb4001750 tx=0x7febac0057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 -- 172.21.15.29:0/410908777 shutdown_connections 2023-10-14T18:14:53.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 --2- 172.21.15.29:0/410908777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febac003d90 unknown :-1 s=CLOSED pgs=2402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 -- 172.21.15.29:0/410908777 >> 172.21.15.29:0/410908777 conn(0x7febc40cb410 msgr2=0x7febc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 -- 172.21.15.29:0/410908777 shutdown_connections 2023-10-14T18:14:53.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.700+0000 7febc98a2640 1 -- 172.21.15.29:0/410908777 wait complete. 2023-10-14T18:14:53.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.702+0000 7febc98a2640 1 Processor -- start 2023-10-14T18:14:53.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.702+0000 7febc98a2640 1 -- start start 2023-10-14T18:14:53.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febc41f7a00 0x7febc40c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc40c4bb0 0x7febc41fa5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c5040 con 0x7febac003970 2023-10-14T18:14:53.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c2ad0 con 0x7febc40c4bb0 2023-10-14T18:14:53.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc98a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febc40c2db0 con 0x7febc41f7a00 2023-10-14T18:14:53.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.703+0000 7febc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36536/0 (socket says 172.21.15.29:36536) 2023-10-14T18:14:53.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 -- 172.21.15.29:0/2200588809 learned_addr learned my addr 172.21.15.29:0/2200588809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:53.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc37fe640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc40c4bb0 0x7febc41fa5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 -- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febc41f7a00 msgr2=0x7febc40c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febc41f7a00 0x7febc40c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 -- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc40c4bb0 msgr2=0x7febc41fa5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc40c4bb0 0x7febc41fa5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 -- 172.21.15.29:0/2200588809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc40c3090 con 0x7febac003970 2023-10-14T18:14:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.704+0000 7febc2ffd640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7febb402bc00 tx=0x7febb4001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.705+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb40076c0 con 0x7febac003970 2023-10-14T18:14:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.705+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4007a30 con 0x7febac003970 2023-10-14T18:14:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.705+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febb4006c20 con 0x7febac003970 2023-10-14T18:14:53.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.705+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc40c3370 con 0x7febac003970 2023-10-14T18:14:53.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.705+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc40c27a0 con 0x7febac003970 2023-10-14T18:14:53.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.706+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc404e160 con 0x7febac003970 2023-10-14T18:14:53.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.707+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febb40035f0 con 0x7febac003970 2023-10-14T18:14:53.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.708+0000 7febabfff640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 0x7febb006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:53.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.708+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7febb40ab530 con 0x7febac003970 2023-10-14T18:14:53.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.708+0000 7febc27fc640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 0x7febb006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:53.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.709+0000 7febc27fc640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 0x7febb006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febb8005c30 tx=0x7febb8001270 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:53.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.712+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febb4079dc0 con 0x7febac003970 2023-10-14T18:14:53.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.948+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7febc40c4450 con 0x7febac003970 2023-10-14T18:14:53.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.949+0000 7febabfff640 1 -- 172.21.15.29:0/2200588809 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v98) v1 ==== 53+0+73 (secure 0 0 0) 0x7febb4079950 con 0x7febac003970 2023-10-14T18:14:53.950 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T18:14:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 msgr2=0x7febb006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 0x7febb006d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7febb8005c30 tx=0x7febb8001270 comp rx=0 tx=0).stop 2023-10-14T18:14:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 msgr2=0x7febc41600d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:53.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 secure :-1 s=READY pgs=2403 cs=0 l=1 rev1=1 crypto rx=0x7febb402bc00 tx=0x7febb4001120 comp rx=0 tx=0).stop 2023-10-14T18:14:53.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 shutdown_connections 2023-10-14T18:14:53.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febc40c4bb0 0x7febc41fa5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7febb006af90 0x7febb006d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febc41f7a00 0x7febc40c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 --2- 172.21.15.29:0/2200588809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febac003970 0x7febc41600d0 unknown :-1 s=CLOSED pgs=2403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 >> 172.21.15.29:0/2200588809 conn(0x7febc40cb410 msgr2=0x7febc40b0010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 shutdown_connections 2023-10-14T18:14:53.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:53.952+0000 7febc98a2640 1 -- 172.21.15.29:0/2200588809 wait complete. 2023-10-14T18:14:53.972 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T18:14:53.973 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T18:14:53.973 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T18:14:53.973 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T18:14:53.974 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T18:14:53.974 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T18:14:53.974 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T18:14:53.975 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T18:14:53.975 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:14:53.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:53.976 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T18:14:54.031 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T18:14:54.032 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T18:14:54.032 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T18:14:54.032 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.032 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T18:14:54.033 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.033 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:14:54.033 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:14:54.033 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T18:14:54.034 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:14:54.034 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:14:54.034 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T18:14:54.034 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T18:14:54.034 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36682sec preferred_lft 36682sec 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.035 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:14:54.036 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:14:54.036 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T18:14:54.036 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T18:14:54.036 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:14:54.036 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T18:14:54.037 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.037 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T18:14:54.037 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.037 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:14:54.037 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T18:14:54.038 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T18:14:54.038 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.038 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:14:54.038 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T18:14:54.038 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T18:14:54.039 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:14:54.039 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.040 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T18:14:54.136 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T18:14:54.140 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.140 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T18:14:54.195 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T18:14:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T18:14:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T18:14:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T18:14:54.196 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T18:14:54.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.197 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T18:14:54.197 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T18:14:54.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T18:14:54.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T18:14:54.199 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T18:14:54.199 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T18:14:54.384 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.385 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:14:54.441 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T18:14:54.441 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T18:14:54.442 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:14:54.443 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T18:14:54.552 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:14:54.552 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T18:14:54.610 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:14:54.610 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:14:54.611 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:14:54.665 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T18:14:54.665 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.666 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:14:54.814 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:14:54.814 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:14:54.814 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:14:54.838 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:14:54.838 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:14:54.838 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:14:54.839 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:14:54.839 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:14:54.865 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:14:54.866 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:14:54.866 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:14:54.866 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:14:54.866 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:14:54.866 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T18:14:54.867 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T18:14:54.867 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T18:14:54.868 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T18:14:54.868 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T18:14:54.869 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T18:14:54.869 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T18:14:54.869 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T18:14:54.871 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T18:14:54.872 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:14:54.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:54.961 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T18:14:55.029 INFO:teuthology.orchestra.run.smithi029.stdout:client8068 2023-10-14T18:14:55.030 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:14:55.030 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:14:55.087 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T18:14:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.318+0000 7fb1360de640 1 Processor -- start 2023-10-14T18:14:55.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.318+0000 7fb1360de640 1 -- start start 2023-10-14T18:14:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb1360de640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1300ad350 0x7fb130069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb1360de640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13006a2e0 0x7fb1300c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb1360de640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb13006a730 con 0x7fb13015d150 2023-10-14T18:14:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb1360de640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb13015d530 con 0x7fb1300ad350 2023-10-14T18:14:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb1360de640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300ac890 con 0x7fb13006a2e0 2023-10-14T18:14:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb12effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1300ad350 0x7fb130069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb12effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1300ad350 0x7fb130069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36542/0 (socket says 172.21.15.29:36542) 2023-10-14T18:14:55.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.320+0000 7fb12effd640 1 -- 172.21.15.29:0/3988457922 learned_addr learned my addr 172.21.15.29:0/3988457922 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.321+0000 7fb12ffff640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13006a2e0 0x7fb1300c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.321+0000 7fb12effd640 1 -- 172.21.15.29:0/3988457922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13006a2e0 msgr2=0x7fb1300c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.321+0000 7fb12effd640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13006a2e0 0x7fb1300c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.321+0000 7fb12effd640 1 -- 172.21.15.29:0/3988457922 >> v1:172.21.15.29:6789/0 conn(0x7fb13015d150 legacy=0x7fb130160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.321+0000 7fb12effd640 1 -- 172.21.15.29:0/3988457922 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1300ad7a0 con 0x7fb1300ad350 2023-10-14T18:14:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.322+0000 7fb12effd640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1300ad350 0x7fb130069da0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fb118009870 tx=0x7fb11802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e5ab383903d036 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.322+0000 7fb12e7fc640 1 -- 172.21.15.29:0/3988457922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118030840 con 0x7fb1300ad350 2023-10-14T18:14:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.322+0000 7fb12e7fc640 1 -- 172.21.15.29:0/3988457922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1180309a0 con 0x7fb1300ad350 2023-10-14T18:14:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.322+0000 7fb12e7fc640 1 -- 172.21.15.29:0/3988457922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb118030d30 con 0x7fb1300ad350 2023-10-14T18:14:55.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 -- 172.21.15.29:0/3988457922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1300ad350 msgr2=0x7fb130069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1300ad350 0x7fb130069da0 secure :-1 s=READY pgs=2405 cs=0 l=1 rev1=1 crypto rx=0x7fb118009870 tx=0x7fb11802b980 comp rx=0 tx=0).stop 2023-10-14T18:14:55.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 -- 172.21.15.29:0/3988457922 shutdown_connections 2023-10-14T18:14:55.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb13006a2e0 0x7fb1300c9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 --2- 172.21.15.29:0/3988457922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1300ad350 0x7fb130069da0 unknown :-1 s=CLOSED pgs=2405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.323+0000 7fb1360de640 1 -- 172.21.15.29:0/3988457922 >> 172.21.15.29:0/3988457922 conn(0x7fb1300ce9b0 msgr2=0x7fb1300cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:55.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.324+0000 7fb1360de640 1 -- 172.21.15.29:0/3988457922 shutdown_connections 2023-10-14T18:14:55.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.324+0000 7fb1360de640 1 -- 172.21.15.29:0/3988457922 wait complete. 2023-10-14T18:14:55.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.325+0000 7fb1360de640 1 Processor -- start 2023-10-14T18:14:55.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.326+0000 7fb1360de640 1 -- start start 2023-10-14T18:14:55.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.326+0000 7fb1360de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.326+0000 7fb1360de640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1300ad350 0x7fb1301f6d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb1360de640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 0x7fb1301f72a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb1360de640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300b46c0 con 0x7fb13006a2e0 2023-10-14T18:14:55.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb1360de640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300ae950 con 0x7fb13015d150 2023-10-14T18:14:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb1360de640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1300af1f0 con 0x7fb1300ad350 2023-10-14T18:14:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36556/0 (socket says 172.21.15.29:36556) 2023-10-14T18:14:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 -- 172.21.15.29:0/2034368679 learned_addr learned my addr 172.21.15.29:0/2034368679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:55.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12ffff640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 0x7fb1301f72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 -- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1300ad350 msgr2=0x7fb1301f6d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1300ad350 0x7fb1301f6d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 -- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 msgr2=0x7fb1301f72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 0x7fb1301f72a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12f7fe640 1 -- 172.21.15.29:0/2034368679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1180094d0 con 0x7fb13006a2e0 2023-10-14T18:14:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.327+0000 7fb12ffff640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 0x7fb1301f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb12f7fe640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb12000c460 tx=0x7fb12000c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1200052d0 con 0x7fb13006a2e0 2023-10-14T18:14:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb120005470 con 0x7fb13006a2e0 2023-10-14T18:14:55.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb120012870 con 0x7fb13006a2e0 2023-10-14T18:14:55.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1300c8950 con 0x7fb13006a2e0 2023-10-14T18:14:55.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.328+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1300aee80 con 0x7fb13006a2e0 2023-10-14T18:14:55.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.330+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb13004e160 con 0x7fb13006a2e0 2023-10-14T18:14:55.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.330+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb1200129d0 con 0x7fb13006a2e0 2023-10-14T18:14:55.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.331+0000 7fb127fff640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 0x7fb11006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.331+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb1200890d0 con 0x7fb13006a2e0 2023-10-14T18:14:55.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.332+0000 7fb12effd640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 0x7fb11006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.333+0000 7fb12effd640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 0x7fb11006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb118036000 tx=0x7fb11803b040 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.336+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb120057b10 con 0x7fb13006a2e0 2023-10-14T18:14:55.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.557+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 --> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fb1300af010 con 0x7fb11006af90 2023-10-14T18:14:55.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.560+0000 7fb127fff640 1 -- 172.21.15.29:0/2034368679 <== mgr.10482 v2:172.21.15.29:6832/2339165088 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fb1300af010 con 0x7fb11006af90 2023-10-14T18:14:55.561 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:14:55.561 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:14:55.561 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T18:14:55.562 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:14:55.562 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:14:55.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.562+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 msgr2=0x7fb11006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.562+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 0x7fb11006d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fb118036000 tx=0x7fb11803b040 comp rx=0 tx=0).stop 2023-10-14T18:14:55.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.562+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 msgr2=0x7fb1301f4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.562+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 secure :-1 s=READY pgs=2406 cs=0 l=1 rev1=1 crypto rx=0x7fb12000c460 tx=0x7fb12000c920 comp rx=0 tx=0).stop 2023-10-14T18:14:55.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 shutdown_connections 2023-10-14T18:14:55.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb13015d150 0x7fb1301f72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7fb11006af90 0x7fb11006d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1300ad350 0x7fb1301f6d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 --2- 172.21.15.29:0/2034368679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb13006a2e0 0x7fb1301f4880 unknown :-1 s=CLOSED pgs=2406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 >> 172.21.15.29:0/2034368679 conn(0x7fb1300ce9b0 msgr2=0x7fb1300cab70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:55.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 shutdown_connections 2023-10-14T18:14:55.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.563+0000 7fb1360de640 1 -- 172.21.15.29:0/2034368679 wait complete. 2023-10-14T18:14:55.582 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T18:14:55.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.802+0000 7f58d4a44640 1 Processor -- start 2023-10-14T18:14:55.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.803+0000 7f58d4a44640 1 -- start start 2023-10-14T18:14:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.804+0000 7f58d4a44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58d015d150 0x7f58d00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.804+0000 7f58d4a44640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 0x7f58d0160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.804+0000 7f58d4a44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58d00ac890 con 0x7f58d0160b90 2023-10-14T18:14:55.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58d4a44640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d015d530 con 0x7f58d015d150 2023-10-14T18:14:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58d4a44640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58d0161650 con 0x7f58d006a4b0 2023-10-14T18:14:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58d015d150 0x7f58d00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f58d015d150 0x7f58d00ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36570/0 (socket says 172.21.15.29:36570) 2023-10-14T18:14:55.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 -- 172.21.15.29:0/2520464570 learned_addr learned my addr 172.21.15.29:0/2520464570 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:55.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ced9d640 1 --2- 172.21.15.29:0/2520464570 >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 0x7f58d0160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 -- 172.21.15.29:0/2520464570 >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 msgr2=0x7f58d0160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:55.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 --2- 172.21.15.29:0/2520464570 >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 0x7f58d0160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 -- 172.21.15.29:0/2520464570 >> v1:172.21.15.186:6789/0 conn(0x7f58d0160b90 legacy=0x7f58d0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:55.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ce59c640 1 -- 172.21.15.29:0/2520464570 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58d0161930 con 0x7f58d015d150 2023-10-14T18:14:55.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.805+0000 7f58ced9d640 1 --2- 172.21.15.29:0/2520464570 >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 0x7f58d0160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.806+0000 7f58ce59c640 1 --2- 172.21.15.29:0/2520464570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015d150 0x7f58d00ad350 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f58d00adde0 tx=0x7f58c002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c06e0c787034588f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58cd59a640 1 -- 172.21.15.29:0/2520464570 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c00317e0 con 0x7f58d015d150 2023-10-14T18:14:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58cd59a640 1 -- 172.21.15.29:0/2520464570 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c0031940 con 0x7f58d015d150 2023-10-14T18:14:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58cd59a640 1 -- 172.21.15.29:0/2520464570 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c0031cd0 con 0x7f58d015d150 2023-10-14T18:14:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58d4a44640 1 -- 172.21.15.29:0/2520464570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015d150 msgr2=0x7f58d00ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58d4a44640 1 --2- 172.21.15.29:0/2520464570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015d150 0x7f58d00ad350 secure :-1 s=READY pgs=2407 cs=0 l=1 rev1=1 crypto rx=0x7f58d00adde0 tx=0x7f58c002b880 comp rx=0 tx=0).stop 2023-10-14T18:14:55.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58d4a44640 1 -- 172.21.15.29:0/2520464570 shutdown_connections 2023-10-14T18:14:55.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58d4a44640 1 --2- 172.21.15.29:0/2520464570 >> v2:172.21.15.186:3300/0 conn(0x7f58d006a4b0 0x7f58d0160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.807+0000 7f58d4a44640 1 --2- 172.21.15.29:0/2520464570 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d015d150 0x7f58d00ad350 unknown :-1 s=CLOSED pgs=2407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.808+0000 7f58d4a44640 1 -- 172.21.15.29:0/2520464570 >> 172.21.15.29:0/2520464570 conn(0x7f58d00ce9b0 msgr2=0x7f58d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:55.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.808+0000 7f58d4a44640 1 -- 172.21.15.29:0/2520464570 shutdown_connections 2023-10-14T18:14:55.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.808+0000 7f58d4a44640 1 -- 172.21.15.29:0/2520464570 wait complete. 2023-10-14T18:14:55.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.809+0000 7f58d4a44640 1 Processor -- start 2023-10-14T18:14:55.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.810+0000 7f58d4a44640 1 -- start start 2023-10-14T18:14:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58d006a4b0 0x7f58d015c000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 0x7f58d00c4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d01e7700 con 0x7f58d0160b90 2023-10-14T18:14:55.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d01e79e0 con 0x7f58d015d150 2023-10-14T18:14:55.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58d4a44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58d01e7cc0 con 0x7f58d006a4b0 2023-10-14T18:14:55.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36586/0 (socket says 172.21.15.29:36586) 2023-10-14T18:14:55.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 -- 172.21.15.29:0/4214742565 learned_addr learned my addr 172.21.15.29:0/4214742565 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:55.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 -- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58d006a4b0 msgr2=0x7f58d015c000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58d006a4b0 0x7f58d015c000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58cdd9b640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 0x7f58d00c4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.811+0000 7f58ced9d640 1 -- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 msgr2=0x7f58d00c4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:55.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58ced9d640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 0x7f58d00c4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:55.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58ced9d640 1 -- 172.21.15.29:0/4214742565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c00094d0 con 0x7f58d0160b90 2023-10-14T18:14:55.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58cdd9b640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 0x7f58d00c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:55.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58ced9d640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f58c800cd50 tx=0x7f58c800a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800f680 con 0x7f58d0160b90 2023-10-14T18:14:55.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c800f800 con 0x7f58d0160b90 2023-10-14T18:14:55.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d01e7fa0 con 0x7f58d0160b90 2023-10-14T18:14:55.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c800fbb0 con 0x7f58d0160b90 2023-10-14T18:14:55.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.812+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d00c6390 con 0x7f58d0160b90 2023-10-14T18:14:55.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.814+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58d004e160 con 0x7f58d0160b90 2023-10-14T18:14:55.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.815+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 122) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f58c800b040 con 0x7f58d0160b90 2023-10-14T18:14:55.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.815+0000 7f58beffd640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 0x7f58b006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:55.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.816+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f58c808a270 con 0x7f58d0160b90 2023-10-14T18:14:55.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.816+0000 7f58ce59c640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 0x7f58b006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:55.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.817+0000 7f58ce59c640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 0x7f58b006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c0030000 comp rx=0 tx=0).ready entity=mgr.10482 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:55.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:55.821+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f58c8055110 con 0x7f58d0160b90 2023-10-14T18:14:56.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.068+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f58d00b4610 con 0x7f58d0160b90 2023-10-14T18:14:56.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.244+0000 7f58beffd640 1 -- 172.21.15.29:0/4214742565 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v123) v1 ==== 92+0+0 (secure 0 0 0) 0x7f58c8058730 con 0x7f58d0160b90 2023-10-14T18:14:56.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.247+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 msgr2=0x7f58b006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.247+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 0x7f58b006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f58c0036000 tx=0x7f58c0030000 comp rx=0 tx=0).stop 2023-10-14T18:14:56.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.247+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 msgr2=0x7f58d00c4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.247+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 secure :-1 s=READY pgs=2408 cs=0 l=1 rev1=1 crypto rx=0x7f58c800cd50 tx=0x7f58c800a650 comp rx=0 tx=0).stop 2023-10-14T18:14:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 shutdown_connections 2023-10-14T18:14:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f58d0160b90 0x7f58d00c4da0 unknown :-1 s=CLOSED pgs=2408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f58d015d150 0x7f58d00c4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f58b006af90 0x7f58b006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 --2- 172.21.15.29:0/4214742565 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f58d006a4b0 0x7f58d015c000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 >> 172.21.15.29:0/4214742565 conn(0x7f58d00ce9b0 msgr2=0x7f58d0071130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 shutdown_connections 2023-10-14T18:14:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.248+0000 7f58d4a44640 1 -- 172.21.15.29:0/4214742565 wait complete. 2023-10-14T18:14:56.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T18:14:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.480+0000 7f56d4ca2640 1 Processor -- start 2023-10-14T18:14:56.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.480+0000 7f56d4ca2640 1 -- start start 2023-10-14T18:14:56.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.481+0000 7f56d4ca2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 0x7f56d00b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56d4ca2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f56d00cfaf0 0x7f56d00b6200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56d4ca2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d00d3980 con 0x7f56d00cf710 2023-10-14T18:14:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56d4ca2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56d00d3c60 con 0x7f56d00cfaf0 2023-10-14T18:14:56.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56d4ca2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00ce630 con 0x7f56d00b3420 2023-10-14T18:14:56.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56ce59c640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f56d00cf710 0x7f56d00b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:39744/0 (socket says 172.21.15.29:39744) 2023-10-14T18:14:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56ce59c640 1 -- 172.21.15.29:0/3581511120 learned_addr learned my addr 172.21.15.29:0/3581511120 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56ced9d640 1 --2- 172.21.15.29:0/3581511120 >> v2:172.21.15.186:3300/0 conn(0x7f56d00cfaf0 0x7f56d00b6200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.482+0000 7f56cdd9b640 1 --2- 172.21.15.29:0/3581511120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 0x7f56d00b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cd59a640 1 -- 172.21.15.29:0/3581511120 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368827576 0 0) 0x7f56d00d3980 con 0x7f56d00cf710 2023-10-14T18:14:56.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cd59a640 1 -- 172.21.15.29:0/3581511120 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b8003520 con 0x7f56d00cf710 2023-10-14T18:14:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cdd9b640 1 -- 172.21.15.29:0/3581511120 >> v2:172.21.15.186:3300/0 conn(0x7f56d00cfaf0 msgr2=0x7f56d00b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cdd9b640 1 --2- 172.21.15.29:0/3581511120 >> v2:172.21.15.186:3300/0 conn(0x7f56d00cfaf0 0x7f56d00b6200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cdd9b640 1 -- 172.21.15.29:0/3581511120 >> v1:172.21.15.186:6789/0 conn(0x7f56d00cf710 legacy=0x7f56d00b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.483+0000 7f56cdd9b640 1 -- 172.21.15.29:0/3581511120 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00ce910 con 0x7f56d00b3420 2023-10-14T18:14:56.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.485+0000 7f56cdd9b640 1 --2- 172.21.15.29:0/3581511120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 0x7f56d00b5cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3de533922dceec1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.485+0000 7f56cd59a640 1 -- 172.21.15.29:0/3581511120 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4031780 con 0x7f56d00b3420 2023-10-14T18:14:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.485+0000 7f56cd59a640 1 -- 172.21.15.29:0/3581511120 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c40318e0 con 0x7f56d00b3420 2023-10-14T18:14:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.485+0000 7f56cd59a640 1 -- 172.21.15.29:0/3581511120 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4031c70 con 0x7f56d00b3420 2023-10-14T18:14:56.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 -- 172.21.15.29:0/3581511120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 msgr2=0x7f56d00b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/3581511120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 0x7f56d00b5cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b820 comp rx=0 tx=0).stop 2023-10-14T18:14:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 -- 172.21.15.29:0/3581511120 shutdown_connections 2023-10-14T18:14:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/3581511120 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00b3420 0x7f56d00b5cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/3581511120 >> v2:172.21.15.186:3300/0 conn(0x7f56d00cfaf0 0x7f56d00b6200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 -- 172.21.15.29:0/3581511120 >> 172.21.15.29:0/3581511120 conn(0x7f56d00d2cc0 msgr2=0x7f56d00d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 -- 172.21.15.29:0/3581511120 shutdown_connections 2023-10-14T18:14:56.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.486+0000 7f56d4ca2640 1 -- 172.21.15.29:0/3581511120 wait complete. 2023-10-14T18:14:56.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.488+0000 7f56d4ca2640 1 Processor -- start 2023-10-14T18:14:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.488+0000 7f56d4ca2640 1 -- start start 2023-10-14T18:14:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d00b3420 0x7f56d009c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00cf710 0x7f56d00b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d009cee0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00b0b80 con 0x7f56d00b3420 2023-10-14T18:14:56.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56d4ca2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56d00b0e60 con 0x7f56d00cf710 2023-10-14T18:14:56.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56ced9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36612/0 (socket says 172.21.15.29:36612) 2023-10-14T18:14:56.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56ced9d640 1 -- 172.21.15.29:0/465233211 learned_addr learned my addr 172.21.15.29:0/465233211 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56ce59c640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d00b3420 0x7f56d009c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.489+0000 7f56cdd9b640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00cf710 0x7f56d00b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00cf710 msgr2=0x7f56d00b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00cf710 0x7f56d00b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d00b3420 msgr2=0x7f56d009c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d00b3420 0x7f56d009c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 -- 172.21.15.29:0/465233211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009400 con 0x7f56d00cfaf0 2023-10-14T18:14:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56ced9d640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f56c800efc0 tx=0x7f56c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c8009ba0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c800f040 con 0x7f56d00cfaf0 2023-10-14T18:14:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c80146c0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.490+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d00b1140 con 0x7f56d00cfaf0 2023-10-14T18:14:56.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.491+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d00b1700 con 0x7f56d00cfaf0 2023-10-14T18:14:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.492+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d00b18f0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.493+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 123) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f56c8014820 con 0x7f56d00cfaf0 2023-10-14T18:14:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.494+0000 7f56beffd640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 0x7f56b006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.494+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f56c808a3d0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.494+0000 7f56ce59c640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 msgr2=0x7f56b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:56.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.494+0000 7f56ce59c640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 0x7f56b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:56.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.498+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56c80551e0 con 0x7f56d00cfaf0 2023-10-14T18:14:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.695+0000 7f56ce59c640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 msgr2=0x7f56b006d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:56.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.695+0000 7f56ce59c640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 0x7f56b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:56.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.746+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f56d004e160 con 0x7f56d00cfaf0 2023-10-14T18:14:56.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.752+0000 7f56beffd640 1 -- 172.21.15.29:0/465233211 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v71)=0 v71) v1 ==== 138+0+0 (secure 0 0 0) 0x7f56c8058800 con 0x7f56d00cfaf0 2023-10-14T18:14:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.754+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 msgr2=0x7f56b006d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.754+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 0x7f56b006d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.754+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 msgr2=0x7f56d00b6f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.754+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 secure :-1 s=READY pgs=2414 cs=0 l=1 rev1=1 crypto rx=0x7f56c800efc0 tx=0x7f56c800c490 comp rx=0 tx=0).stop 2023-10-14T18:14:56.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 shutdown_connections 2023-10-14T18:14:56.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56d00cfaf0 0x7f56d00b6f80 unknown :-1 s=CLOSED pgs=2414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f56b006afe0 0x7f56b006d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56d00cf710 0x7f56d00b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 --2- 172.21.15.29:0/465233211 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56d00b3420 0x7f56d009c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 >> 172.21.15.29:0/465233211 conn(0x7f56d00d2cc0 msgr2=0x7f56d00d1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:56.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 shutdown_connections 2023-10-14T18:14:56.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.755+0000 7f56d4ca2640 1 -- 172.21.15.29:0/465233211 wait complete. 2023-10-14T18:14:56.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T18:14:56.776 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:56.775+0000 7f156f44c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:14:56.830 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:56.828+0000 7f156f44c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:14:56.832 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:56.832+0000 7f1e1f212040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:14:56.890 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:56.889+0000 7f1e1f212040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:14:56.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.989+0000 7f5386415640 1 Processor -- start 2023-10-14T18:14:56.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.989+0000 7f5386415640 1 -- start start 2023-10-14T18:14:56.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.990+0000 7f5386415640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f538006a4b0 0x7f53800ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.990+0000 7f5386415640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800cb030 con 0x7f53800c9ca0 2023-10-14T18:14:56.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.990+0000 7f5386415640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53800c91c0 con 0x7f538015ad70 2023-10-14T18:14:56.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.990+0000 7f5386415640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53800c94a0 con 0x7f538006a4b0 2023-10-14T18:14:56.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.991+0000 7f53849b2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f53800c9ca0 0x7f53800ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58162/0 (socket says 172.21.15.29:58162) 2023-10-14T18:14:56.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.991+0000 7f53849b2640 1 -- 172.21.15.29:0/4024654285 learned_addr learned my addr 172.21.15.29:0/4024654285 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:56.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.991+0000 7f53777fe640 1 --2- 172.21.15.29:0/4024654285 >> v2:172.21.15.186:3300/0 conn(0x7f538006a4b0 0x7f53800ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.991+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2350179617 0 0) 0x7f53800cb030 con 0x7f53800c9ca0 2023-10-14T18:14:56.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.991+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f536c003520 con 0x7f53800c9ca0 2023-10-14T18:14:56.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3045776068 0 0) 0x7f536c003520 con 0x7f53800c9ca0 2023-10-14T18:14:56.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 >> v2:172.21.15.186:3300/0 conn(0x7f538006a4b0 msgr2=0x7f53800ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:56.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 --2- 172.21.15.29:0/4024654285 >> v2:172.21.15.186:3300/0 conn(0x7f538006a4b0 0x7f53800ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:56.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 >> v1:172.21.15.186:6789/0 conn(0x7f538015ad70 legacy=0x7f5380069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53800c9780 con 0x7f53800c9ca0 2023-10-14T18:14:56.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5378001670 con 0x7f53800c9ca0 2023-10-14T18:14:56.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 >> v1:172.21.15.29:6789/0 conn(0x7f53800c9ca0 legacy=0x7f53800ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:56.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c0039c0 0x7f536c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0047c0 0x7f536c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 0x7f536c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:56.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53800c94a0 con 0x7f536c005180 2023-10-14T18:14:56.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c00c080 con 0x7f536c0047c0 2023-10-14T18:14:56.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536c00c240 con 0x7f536c0039c0 2023-10-14T18:14:56.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f53777fe640 1 --2- 172.21.15.29:0/4024654285 >> v2:172.21.15.186:3300/0 conn(0x7f538006a4b0 0x7f53800ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:14:56.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f53849b2640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 0x7f536c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f537ffff640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c0039c0 0x7f536c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.992+0000 7f53777fe640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0047c0 0x7f536c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:56.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.993+0000 7f53849b2640 1 -- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c0039c0 msgr2=0x7f536c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:57.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.993+0000 7f53849b2640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c0039c0 0x7f536c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.993+0000 7f53849b2640 1 -- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0047c0 msgr2=0x7f536c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:57.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.993+0000 7f53849b2640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0047c0 0x7f536c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.993+0000 7f53849b2640 1 -- 172.21.15.29:0/4024654285 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c003520 con 0x7f536c005180 2023-10-14T18:14:57.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.994+0000 7f53849b2640 1 -- 172.21.15.29:0/4024654285 reap_dead start 2023-10-14T18:14:57.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.994+0000 7f53849b2640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 0x7f536c00b980 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f537800c0e0 tx=0x7f536c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c35ba746df1ac6c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.994+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378005ad0 con 0x7f536c005180 2023-10-14T18:14:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.994+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5378005c30 con 0x7f536c005180 2023-10-14T18:14:57.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.994+0000 7f537f7fe640 1 -- 172.21.15.29:0/4024654285 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537802c810 con 0x7f536c005180 2023-10-14T18:14:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 -- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 msgr2=0x7f536c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 0x7f536c00b980 secure :-1 s=READY pgs=2416 cs=0 l=1 rev1=1 crypto rx=0x7f537800c0e0 tx=0x7f536c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 -- 172.21.15.29:0/4024654285 shutdown_connections 2023-10-14T18:14:57.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 --2- 172.21.15.29:0/4024654285 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c005180 0x7f536c00b980 unknown :-1 s=CLOSED pgs=2416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 -- 172.21.15.29:0/4024654285 >> 172.21.15.29:0/4024654285 conn(0x7f53800ce9b0 msgr2=0x7f53800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:57.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 -- 172.21.15.29:0/4024654285 shutdown_connections 2023-10-14T18:14:57.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.995+0000 7f5386415640 1 -- 172.21.15.29:0/4024654285 wait complete. 2023-10-14T18:14:57.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.997+0000 7f5386415640 1 Processor -- start 2023-10-14T18:14:57.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.997+0000 7f5386415640 1 -- start start 2023-10-14T18:14:57.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 0x7f5380159ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53800b0e90 0x7f53801f35b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53801f3f70 con 0x7f53801f3af0 2023-10-14T18:14:57.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53800b1f60 con 0x7f536c005180 2023-10-14T18:14:57.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f5386415640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53801f0040 con 0x7f53800b0e90 2023-10-14T18:14:57.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36642/0 (socket says 172.21.15.29:36642) 2023-10-14T18:14:57.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 -- 172.21.15.29:0/3021437029 learned_addr learned my addr 172.21.15.29:0/3021437029 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:57.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53777fe640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53800b0e90 0x7f53801f35b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f537ffff640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 0x7f5380159ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53800b0e90 msgr2=0x7f53801f35b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53800b0e90 0x7f53801f35b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 msgr2=0x7f5380159ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 0x7f5380159ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f53849b2640 1 -- 172.21.15.29:0/3021437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53801f4390 con 0x7f53801f3af0 2023-10-14T18:14:57.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.998+0000 7f537ffff640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 0x7f5380159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f53849b2640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f5378001eb0 tx=0x7f5378001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f537803b070 con 0x7f53801f3af0 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f537802de00 con 0x7f53801f3af0 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53800b1350 con 0x7f53801f3af0 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5378004a20 con 0x7f53801f3af0 2023-10-14T18:14:57.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:56.999+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53800b1910 con 0x7f53801f3af0 2023-10-14T18:14:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.000+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538004e160 con 0x7f53801f3af0 2023-10-14T18:14:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.001+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 123) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f5378032070 con 0x7f53801f3af0 2023-10-14T18:14:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.003+0000 7f537d7fa640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 0x7f536006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.003+0000 7f537ffff640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 msgr2=0x7f536006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:57.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.003+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f53780a9f80 con 0x7f53801f3af0 2023-10-14T18:14:57.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.003+0000 7f537ffff640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 0x7f536006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:57.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.008+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5378074ea0 con 0x7f53801f3af0 2023-10-14T18:14:57.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.202+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f53800b20a0 con 0x7f53801f3af0 2023-10-14T18:14:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.204+0000 7f537ffff640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 msgr2=0x7f536006d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:57.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.204+0000 7f537ffff640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 0x7f536006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:57.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.207+0000 7f537d7fa640 1 -- 172.21.15.29:0/3021437029 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v72)=0 v72) v1 ==== 133+0+0 (secure 0 0 0) 0x7f53780784c0 con 0x7f53801f3af0 2023-10-14T18:14:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 msgr2=0x7f536006d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:14:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 0x7f536006d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 msgr2=0x7f53800b23f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 secure :-1 s=READY pgs=2417 cs=0 l=1 rev1=1 crypto rx=0x7f5378001eb0 tx=0x7f5378001930 comp rx=0 tx=0).stop 2023-10-14T18:14:57.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 shutdown_connections 2023-10-14T18:14:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53801f3af0 0x7f53800b23f0 unknown :-1 s=CLOSED pgs=2417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f536006b010 0x7f536006d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53800b0e90 0x7f53801f35b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 --2- 172.21.15.29:0/3021437029 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c005180 0x7f5380159ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.209+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 >> 172.21.15.29:0/3021437029 conn(0x7f53800ce9b0 msgr2=0x7f53800c24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:57.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.210+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 shutdown_connections 2023-10-14T18:14:57.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.210+0000 7f5386415640 1 -- 172.21.15.29:0/3021437029 wait complete. 2023-10-14T18:14:57.229 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:14:57.230 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T18:14:57.281 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.280+0000 7f156f44c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:14:57.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-14T18:14:57.333 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.332+0000 7f156f44c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:14:57.376 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.375+0000 7f1e1f212040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:14:57.383 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.382+0000 7f156f44c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:14:57.431 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.430+0000 7f1e1f212040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:14:57.444 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.443+0000 7f156f44c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:14:57.485 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.483+0000 7f1e1f212040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:14:57.492 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.491+0000 7f156f44c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:14:57.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.506+0000 7f0b75a14640 1 Processor -- start 2023-10-14T18:14:57.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.506+0000 7f0b75a14640 1 -- start start 2023-10-14T18:14:57.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75a14640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 0x7f0b700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75a14640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0b700d0be0 0x7f0b700cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75a14640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7015d150 con 0x7f0b7015cd70 2023-10-14T18:14:57.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75a14640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700d1180 con 0x7f0b700d0be0 2023-10-14T18:14:57.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75a14640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b700d1460 con 0x7f0b700b8480 2023-10-14T18:14:57.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b74a12640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0b7015cd70 0x7f0b700d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58178/0 (socket says 172.21.15.29:58178) 2023-10-14T18:14:57.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b74a12640 1 -- 172.21.15.29:0/2643144478 learned_addr learned my addr 172.21.15.29:0/2643144478 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.508+0000 7f0b75213640 1 --2- 172.21.15.29:0/2643144478 >> v2:172.21.15.29:3300/0 conn(0x7f0b700d0be0 0x7f0b700cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/2643144478 >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 0x7f0b700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b6f7fe640 1 -- 172.21.15.29:0/2643144478 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2815616779 0 0) 0x7f0b7015d150 con 0x7f0b7015cd70 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b6f7fe640 1 -- 172.21.15.29:0/2643144478 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b60003520 con 0x7f0b7015cd70 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b75213640 1 -- 172.21.15.29:0/2643144478 >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 msgr2=0x7f0b700cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:14:57.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b75213640 1 --2- 172.21.15.29:0/2643144478 >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 0x7f0b700cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b75213640 1 -- 172.21.15.29:0/2643144478 >> v1:172.21.15.29:6789/0 conn(0x7f0b7015cd70 legacy=0x7f0b700d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b75213640 1 -- 172.21.15.29:0/2643144478 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700d1740 con 0x7f0b700d0be0 2023-10-14T18:14:57.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.509+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/2643144478 >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 0x7f0b700cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:14:57.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.510+0000 7f0b75213640 1 --2- 172.21.15.29:0/2643144478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b700d0be0 0x7f0b700cdd00 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f0b70066a00 tx=0x7f0b6802be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aa7db29b9ad4c24 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.511+0000 7f0b6f7fe640 1 -- 172.21.15.29:0/2643144478 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b6802cbe0 con 0x7f0b700d0be0 2023-10-14T18:14:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.511+0000 7f0b6f7fe640 1 -- 172.21.15.29:0/2643144478 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b6802cd40 con 0x7f0b700d0be0 2023-10-14T18:14:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.511+0000 7f0b6f7fe640 1 -- 172.21.15.29:0/2643144478 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68034660 con 0x7f0b700d0be0 2023-10-14T18:14:57.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.511+0000 7f0b75a14640 1 -- 172.21.15.29:0/2643144478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b700d0be0 msgr2=0x7f0b700cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.511+0000 7f0b75a14640 1 --2- 172.21.15.29:0/2643144478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b700d0be0 0x7f0b700cdd00 secure :-1 s=READY pgs=2419 cs=0 l=1 rev1=1 crypto rx=0x7f0b70066a00 tx=0x7f0b6802be30 comp rx=0 tx=0).stop 2023-10-14T18:14:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 -- 172.21.15.29:0/2643144478 shutdown_connections 2023-10-14T18:14:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 --2- 172.21.15.29:0/2643144478 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b700d0be0 0x7f0b700cdd00 unknown :-1 s=CLOSED pgs=2419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 --2- 172.21.15.29:0/2643144478 >> v2:172.21.15.186:3300/0 conn(0x7f0b700b8480 0x7f0b700cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 -- 172.21.15.29:0/2643144478 >> 172.21.15.29:0/2643144478 conn(0x7f0b700cf570 msgr2=0x7f0b700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:14:57.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 -- 172.21.15.29:0/2643144478 shutdown_connections 2023-10-14T18:14:57.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.512+0000 7f0b75a14640 1 -- 172.21.15.29:0/2643144478 wait complete. 2023-10-14T18:14:57.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.513+0000 7f0b75a14640 1 Processor -- start 2023-10-14T18:14:57.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.514+0000 7f0b75a14640 1 -- start start 2023-10-14T18:14:57.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.514+0000 7f0b75a14640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b700b8480 0x7f0b700b6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.514+0000 7f0b75a14640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b700d0be0 0x7f0b7015dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75a14640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 0x7f0b701508b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75a14640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015f230 con 0x7f0b7015cd70 2023-10-14T18:14:57.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75a14640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015f510 con 0x7f0b700b8480 2023-10-14T18:14:57.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75a14640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7015f7f0 con 0x7f0b700d0be0 2023-10-14T18:14:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 0x7f0b701508b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 0x7f0b701508b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36672/0 (socket says 172.21.15.29:36672) 2023-10-14T18:14:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 -- 172.21.15.29:0/1230793925 learned_addr learned my addr 172.21.15.29:0/1230793925 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:14:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b6ffff640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b700d0be0 0x7f0b7015dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b700b8480 0x7f0b700b6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:14:57.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b700d0be0 msgr2=0x7f0b7015dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0b700d0be0 0x7f0b7015dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b700b8480 msgr2=0x7f0b700b6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:14:57.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b700b8480 0x7f0b700b6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:14:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b75213640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b68009a70 con 0x7f0b7015cd70 2023-10-14T18:14:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.515+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0b700b8480 0x7f0b700b6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:14:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b75213640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 0x7f0b701508b0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009e80 tx=0x7f0b68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:14:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68040650 con 0x7f0b7015cd70 2023-10-14T18:14:57.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b680407b0 con 0x7f0b7015cd70 2023-10-14T18:14:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b68040b60 con 0x7f0b7015cd70 2023-10-14T18:14:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7015fad0 con 0x7f0b7015cd70 2023-10-14T18:14:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.516+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b701602d0 con 0x7f0b7015cd70 2023-10-14T18:14:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.517+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7015fff0 con 0x7f0b7015cd70 2023-10-14T18:14:57.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.518+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 123) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0b68040cc0 con 0x7f0b7015cd70 2023-10-14T18:14:57.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.520+0000 7f0b6d7fa640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:14:57.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.520+0000 7f0b74a12640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:57.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.520+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(210..210 src has 1..210) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f0b680ac5a0 con 0x7f0b7015cd70 2023-10-14T18:14:57.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.520+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:14:57.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.524+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0b68077420 con 0x7f0b7015cd70 2023-10-14T18:14:57.551 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.550+0000 7f1e1f212040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:14:57.602 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.602+0000 7f1e1f212040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:14:57.603 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.602+0000 7f156f44c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:14:57.666 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.665+0000 7f156f44c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:57.714 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.713+0000 7f156f44c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:14:57.719 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.718+0000 7f1e1f212040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:14:57.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.721+0000 7f0b74a12640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:57.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.721+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:14:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:57.742+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b700d1d00 con 0x7f0b5406afe0 2023-10-14T18:14:57.771 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.770+0000 7f156f44c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:14:57.785 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.784+0000 7f1e1f212040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:57.822 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:57.820+0000 7f156f44c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:14:57.836 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.835+0000 7f1e1f212040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:14:57.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.895+0000 7f1e1f212040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:14:57.948 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:57.947+0000 7f1e1f212040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:14:58.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:58.122+0000 7f0b74a12640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:58.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:58.122+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T18:14:58.203 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.202+0000 7f156f44c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:14:58.345 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.344+0000 7f1e1f212040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:14:58.362 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.361+0000 7f156f44c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:14:58.470 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.469+0000 7f156f44c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:14:58.513 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.511+0000 7f1e1f212040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:14:58.520 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.519+0000 7f156f44c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:14:58.625 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.624+0000 7f1e1f212040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:14:58.629 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.628+0000 7f156f44c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:58.676 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.676+0000 7f1e1f212040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:14:58.734 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.733+0000 7f156f44c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:14:58.789 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.788+0000 7f1e1f212040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:58.889 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.887+0000 7f156f44c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:58.899 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:58.898+0000 7f1e1f212040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:14:58.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:58.923+0000 7f0b74a12640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:14:58.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:14:58.923+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T18:14:58.938 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:58.936+0000 7f156f44c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:59.056 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:59.054+0000 7f156f44c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:14:59.063 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:59.061+0000 7f1e1f212040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:14:59.114 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:59.113+0000 7f1e1f212040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:14:59.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:14:59.236+0000 7f1e1f212040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:14:59.873 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:14:59.871+0000 7f156f44c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:15:00.105 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.104+0000 7f1e1f212040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:15:00.168 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:15:00.166+0000 7f156f44c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:15:00.225 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:15:00.223+0000 7f156f44c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:15:00.339 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:15:00.338+0000 7f156f44c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:15:00.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.406+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 124) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f0b680777f0 con 0x7f0b7015cd70 2023-10-14T18:15:00.420 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.419+0000 7f1e1f212040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:15:00.480 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.479+0000 7f1e1f212040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:15:00.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.525+0000 7f0b74a12640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2339165088 2023-10-14T18:15:00.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.525+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T18:15:00.598 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.597+0000 7f1e1f212040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:15:00.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.614+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 125) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f0b680aa9a0 con 0x7f0b7015cd70 2023-10-14T18:15:00.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.614+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 msgr2=0x7f0b5406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:15:00.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:00.614+0000 7f0b6d7fa640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/2339165088,v1:172.21.15.29:6833/2339165088] conn(0x7f0b5406afe0 0x7f0b5406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:00.912 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.911+0000 7f1d9b860640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.912 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.911+0000 7f1d9b860640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.911+0000 7f1d9b860640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.911+0000 7f1d9b860640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.911+0000 7f1d9b860640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.912+0000 7f1d9e8a6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.912+0000 7f1d9e8a6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.912+0000 7f1d9e8a6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.914 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.912+0000 7f1d9e8a6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:00.915 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:15:00.912+0000 7f1d9e8a6640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:15:01.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.627+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 126) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f0b680780c0 con 0x7f0b7015cd70 2023-10-14T18:15:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.628+0000 7f0b6d7fa640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f0b54027900 0x7f0b54027ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:01.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.628+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 --> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b54028220 con 0x7f0b54027900 2023-10-14T18:15:01.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.640+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f0b54027900 0x7f0b54027ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:01.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.650+0000 7f0b74a12640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f0b54027900 0x7f0b54027ce0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b640011e0 tx=0x7f0b640079e0 comp rx=0 tx=0).ready entity=mgr.10896 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.833+0000 7f0b6d7fa640 1 -- 172.21.15.29:0/1230793925 <== mgr.10896 v2:172.21.15.29:6832/966124587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0b640070d0 con 0x7f0b54027900 2023-10-14T18:15:01.834 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T18:15:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.836+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f0b54027900 msgr2=0x7f0b54027ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.836+0000 7f0b75a14640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f0b54027900 0x7f0b54027ce0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f0b640011e0 tx=0x7f0b640079e0 comp rx=0 tx=0).stop 2023-10-14T18:15:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.836+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 msgr2=0x7f0b701508b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.836+0000 7f0b75a14640 1 --2- 172.21.15.29:0/1230793925 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0b7015cd70 0x7f0b701508b0 secure :-1 s=READY pgs=2420 cs=0 l=1 rev1=1 crypto rx=0x7f0b68009e80 tx=0x7f0b68004a40 comp rx=0 tx=0).stop 2023-10-14T18:15:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.836+0000 7f0b75213640 1 -- 172.21.15.29:0/1230793925 reap_dead start 2023-10-14T18:15:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.837+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 shutdown_connections 2023-10-14T18:15:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.837+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 >> 172.21.15.29:0/1230793925 conn(0x7f0b700cf570 msgr2=0x7f0b7015bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:15:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.837+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 shutdown_connections 2023-10-14T18:15:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:01.837+0000 7f0b75a14640 1 -- 172.21.15.29:0/1230793925 wait complete. 2023-10-14T18:15:01.862 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T18:15:01.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-14T18:15:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.090+0000 7f7794e59640 1 Processor -- start 2023-10-14T18:15:02.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.090+0000 7f7794e59640 1 -- start start 2023-10-14T18:15:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.091+0000 7f7794e59640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f779006c040 0x7f779006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.091+0000 7f7794e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 0x7f779015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.091+0000 7f7794e59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 0x7f779015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.091+0000 7f7794e59640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779006c960 con 0x7f779006c040 2023-10-14T18:15:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f7794e59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f779015b9d0 con 0x7f779015ebc0 2023-10-14T18:15:02.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f7794e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f779015bcb0 con 0x7f77900ad970 2023-10-14T18:15:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f778f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f779006c040 0x7f779006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f778f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f779006c040 0x7f779006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52984/0 (socket says 172.21.15.29:52984) 2023-10-14T18:15:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f778f7fe640 1 -- 172.21.15.29:0/2729134545 learned_addr learned my addr 172.21.15.29:0/2729134545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:15:02.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f778ffff640 1 --2- 172.21.15.29:0/2729134545 >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 0x7f779015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f778f7fe640 1 -- 172.21.15.29:0/2729134545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 msgr2=0x7f779015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:15:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.092+0000 7f7787fff640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 0x7f779015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f778f7fe640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 0x7f779015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f778f7fe640 1 -- 172.21.15.29:0/2729134545 >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 msgr2=0x7f779015f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:15:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f778f7fe640 1 --2- 172.21.15.29:0/2729134545 >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 0x7f779015f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f778f7fe640 1 -- 172.21.15.29:0/2729134545 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779015bf90 con 0x7f779006c040 2023-10-14T18:15:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f778ffff640 1 --2- 172.21.15.29:0/2729134545 >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 0x7f779015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:15:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.093+0000 7f7787fff640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 0x7f779015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:15:02.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.094+0000 7f778f7fe640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779006c040 0x7f779006c420 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f7780009870 tx=0x7f778002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9c5c989c951602c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.094+0000 7f778effd640 1 -- 172.21.15.29:0/2729134545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780031640 con 0x7f779006c040 2023-10-14T18:15:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.094+0000 7f778effd640 1 -- 172.21.15.29:0/2729134545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77800317a0 con 0x7f779006c040 2023-10-14T18:15:02.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.094+0000 7f778effd640 1 -- 172.21.15.29:0/2729134545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7780031b30 con 0x7f779006c040 2023-10-14T18:15:02.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 -- 172.21.15.29:0/2729134545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779006c040 msgr2=0x7f779006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779006c040 0x7f779006c420 secure :-1 s=READY pgs=2434 cs=0 l=1 rev1=1 crypto rx=0x7f7780009870 tx=0x7f778002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 -- 172.21.15.29:0/2729134545 shutdown_connections 2023-10-14T18:15:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 --2- 172.21.15.29:0/2729134545 >> v2:172.21.15.186:3300/0 conn(0x7f779015ebc0 0x7f779015f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f77900ad970 0x7f779015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 --2- 172.21.15.29:0/2729134545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779006c040 0x7f779006c420 unknown :-1 s=CLOSED pgs=2434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 -- 172.21.15.29:0/2729134545 >> 172.21.15.29:0/2729134545 conn(0x7f77900ce6a0 msgr2=0x7f77900ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:15:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 -- 172.21.15.29:0/2729134545 shutdown_connections 2023-10-14T18:15:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.095+0000 7f7794e59640 1 -- 172.21.15.29:0/2729134545 wait complete. 2023-10-14T18:15:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.097+0000 7f7794e59640 1 Processor -- start 2023-10-14T18:15:02.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.097+0000 7f7794e59640 1 -- start start 2023-10-14T18:15:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f779006c040 0x7f77900ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77900ad970 0x7f77900a8f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77901f02f0 con 0x7f779015ebc0 2023-10-14T18:15:02.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77900ac670 con 0x7f77900ad970 2023-10-14T18:15:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f7794e59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77900aba60 con 0x7f779006c040 2023-10-14T18:15:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f778ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f778ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52998/0 (socket says 172.21.15.29:52998) 2023-10-14T18:15:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.098+0000 7f778ffff640 1 -- 172.21.15.29:0/3858545055 learned_addr learned my addr 172.21.15.29:0/3858545055 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:15:02.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f7787fff640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77900ad970 0x7f77900a8f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778f7fe640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f779006c040 0x7f77900ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 -- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f779006c040 msgr2=0x7f77900ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f779006c040 0x7f77900ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 -- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77900ad970 msgr2=0x7f77900a8f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77900ad970 0x7f77900a8f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 -- 172.21.15.29:0/3858545055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77800094d0 con 0x7f779015ebc0 2023-10-14T18:15:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778ffff640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f778800cef0 tx=0x7f778800a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778800d840 con 0x7f779015ebc0 2023-10-14T18:15:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778800d9c0 con 0x7f779015ebc0 2023-10-14T18:15:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f778800dd70 con 0x7f779015ebc0 2023-10-14T18:15:02.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.099+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77900abd40 con 0x7f779015ebc0 2023-10-14T18:15:02.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.100+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77900ac300 con 0x7f779015ebc0 2023-10-14T18:15:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.101+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f779006b330 con 0x7f779015ebc0 2023-10-14T18:15:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.102+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 126) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f778801f440 con 0x7f779015ebc0 2023-10-14T18:15:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.102+0000 7f778cff9640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 0x7f777006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.103+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6169+0+0 (secure 0 0 0) 0x7f7788089f70 con 0x7f779015ebc0 2023-10-14T18:15:02.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.103+0000 7f778f7fe640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 0x7f777006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.104+0000 7f778f7fe640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 0x7f777006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7780036000 tx=0x7f778003c040 comp rx=0 tx=0).ready entity=mgr.10896 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.108+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7788055180 con 0x7f779015ebc0 2023-10-14T18:15:02.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.327+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 --> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f77900cc550 con 0x7f777006ad40 2023-10-14T18:15:02.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.477+0000 7f778cff9640 1 -- 172.21.15.29:0/3858545055 <== mgr.10896 v2:172.21.15.29:6832/966124587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f7780031b80 con 0x7f777006ad40 2023-10-14T18:15:02.478 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-14T18:15:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.480+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 msgr2=0x7f777006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.480+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 0x7f777006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f7780036000 tx=0x7f778003c040 comp rx=0 tx=0).stop 2023-10-14T18:15:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.480+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 msgr2=0x7f77900acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.480+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 secure :-1 s=READY pgs=2435 cs=0 l=1 rev1=1 crypto rx=0x7f778800cef0 tx=0x7f778800a710 comp rx=0 tx=0).stop 2023-10-14T18:15:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 shutdown_connections 2023-10-14T18:15:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f777006ad40 0x7f777006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f779015ebc0 0x7f77900acb00 unknown :-1 s=CLOSED pgs=2435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f77900ad970 0x7f77900a8f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 --2- 172.21.15.29:0/3858545055 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f779006c040 0x7f77900ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 >> 172.21.15.29:0/3858545055 conn(0x7f77900ce6a0 msgr2=0x7f77900cd610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:15:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 shutdown_connections 2023-10-14T18:15:02.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.481+0000 7f7794e59640 1 -- 172.21.15.29:0/3858545055 wait complete. 2023-10-14T18:15:02.504 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-14T18:15:02.504 DEBUG:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-14-18_16_00 in ~58s... 2023-10-14T18:15:02.505 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-14T18:15:02.505 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-14T18:15:02.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.730+0000 7fc496b27640 1 Processor -- start 2023-10-14T18:15:02.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.731+0000 7fc496b27640 1 -- start start 2023-10-14T18:15:02.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.732+0000 7fc496b27640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc49015cd70 0x7fc4900d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.732+0000 7fc496b27640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc4900cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.732+0000 7fc496b27640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc49015d150 con 0x7fc4900d0be0 2023-10-14T18:15:02.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.732+0000 7fc496b27640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4900d1180 con 0x7fc49015cd70 2023-10-14T18:15:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.732+0000 7fc496b27640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4900d1460 con 0x7fc4900b8480 2023-10-14T18:15:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495b25640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc49015cd70 0x7fc4900d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc496326640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc4900d0be0 0x7fc4900b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55842/0 (socket says 172.21.15.29:55842) 2023-10-14T18:15:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc496326640 1 -- 172.21.15.29:0/759280205 learned_addr learned my addr 172.21.15.29:0/759280205 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:15:02.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495324640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc4900cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495b25640 1 -- 172.21.15.29:0/759280205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 msgr2=0x7fc4900cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:15:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495b25640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc4900cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495b25640 1 -- 172.21.15.29:0/759280205 >> v1:172.21.15.29:6789/0 conn(0x7fc4900d0be0 legacy=0x7fc4900b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495b25640 1 -- 172.21.15.29:0/759280205 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4900d1740 con 0x7fc49015cd70 2023-10-14T18:15:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc494b23640 1 -- 172.21.15.29:0/759280205 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074051184 0 0) 0x7fc49015d150 con 0x7fc4900d0be0 2023-10-14T18:15:02.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.733+0000 7fc495324640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc4900cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:15:02.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.734+0000 7fc495b25640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc4900d2220 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fc48c0097a0 tx=0x7fc48c02b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12e6c5300d9ce4ea server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.735+0000 7fc494b23640 1 -- 172.21.15.29:0/759280205 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c02d040 con 0x7fc49015cd70 2023-10-14T18:15:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.735+0000 7fc494b23640 1 -- 172.21.15.29:0/759280205 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48c031810 con 0x7fc49015cd70 2023-10-14T18:15:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.735+0000 7fc494b23640 1 -- 172.21.15.29:0/759280205 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48c031ba0 con 0x7fc49015cd70 2023-10-14T18:15:02.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.735+0000 7fc496b27640 1 -- 172.21.15.29:0/759280205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 msgr2=0x7fc4900d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.735+0000 7fc496b27640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc4900d2220 secure :-1 s=READY pgs=2439 cs=0 l=1 rev1=1 crypto rx=0x7fc48c0097a0 tx=0x7fc48c02b8b0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 -- 172.21.15.29:0/759280205 shutdown_connections 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc4900cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 --2- 172.21.15.29:0/759280205 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc4900d2220 unknown :-1 s=CLOSED pgs=2439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 -- 172.21.15.29:0/759280205 >> 172.21.15.29:0/759280205 conn(0x7fc4900cf570 msgr2=0x7fc4900cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:15:02.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 -- 172.21.15.29:0/759280205 shutdown_connections 2023-10-14T18:15:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.736+0000 7fc496b27640 1 -- 172.21.15.29:0/759280205 wait complete. 2023-10-14T18:15:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.738+0000 7fc496b27640 1 Processor -- start 2023-10-14T18:15:02.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.738+0000 7fc496b27640 1 -- start start 2023-10-14T18:15:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc490151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4900d0be0 0x7fc4901528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901531f0 con 0x7fc49015cd70 2023-10-14T18:15:02.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901534d0 con 0x7fc4900d0be0 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496b27640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4901537b0 con 0x7fc4900b8480 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496326640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496326640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53048/0 (socket says 172.21.15.29:53048) 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc496326640 1 -- 172.21.15.29:0/3383917391 learned_addr learned my addr 172.21.15.29:0/3383917391 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc495324640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4900d0be0 0x7fc4901528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.739+0000 7fc495b25640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc490151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 -- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 msgr2=0x7fc490151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc490151d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 -- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4900d0be0 msgr2=0x7fc4901528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4900d0be0 0x7fc4901528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 -- 172.21.15.29:0/3383917391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48c009400 con 0x7fc49015cd70 2023-10-14T18:15:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496326640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fc48800c780 tx=0x7fc48800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48800ce30 con 0x7fc49015cd70 2023-10-14T18:15:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc48800a640 con 0x7fc49015cd70 2023-10-14T18:15:02.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc48800aa70 con 0x7fc49015cd70 2023-10-14T18:15:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc490153a90 con 0x7fc49015cd70 2023-10-14T18:15:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.740+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc490152270 con 0x7fc49015cd70 2023-10-14T18:15:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.742+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4900b3740 con 0x7fc49015cd70 2023-10-14T18:15:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.743+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 126) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fc48800abd0 con 0x7fc49015cd70 2023-10-14T18:15:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.744+0000 7fc4867fc640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 0x7fc47406d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:15:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.744+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(211..211 src has 1..211) v4 ==== 6169+0+0 (secure 0 0 0) 0x7fc4880889d0 con 0x7fc49015cd70 2023-10-14T18:15:02.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.744+0000 7fc495b25640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 0x7fc47406d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:15:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.745+0000 7fc495b25640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 0x7fc47406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc48c036000 tx=0x7fc48c008030 comp rx=0 tx=0).ready entity=mgr.10896 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:15:02.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.749+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc488053bf0 con 0x7fc49015cd70 2023-10-14T18:15:02.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.967+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 --> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc4900d1d00 con 0x7fc47406ad10 2023-10-14T18:15:02.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.980+0000 7fc4867fc640 1 -- 172.21.15.29:0/3383917391 <== mgr.10896 v2:172.21.15.29:6832/966124587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fc48c031610 con 0x7fc47406ad10 2023-10-14T18:15:02.981 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:15:02.981 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-14T18:15:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 msgr2=0x7fc47406d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 0x7fc47406d1d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc48c036000 tx=0x7fc48c008030 comp rx=0 tx=0).stop 2023-10-14T18:15:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 msgr2=0x7fc49006bbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:15:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 secure :-1 s=READY pgs=2440 cs=0 l=1 rev1=1 crypto rx=0x7fc48800c780 tx=0x7fc48800cc40 comp rx=0 tx=0).stop 2023-10-14T18:15:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 shutdown_connections 2023-10-14T18:15:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fc47406ad10 0x7fc47406d1d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc49015cd70 0x7fc49006bbd0 unknown :-1 s=CLOSED pgs=2440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4900d0be0 0x7fc4901528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 --2- 172.21.15.29:0/3383917391 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc4900b8480 0x7fc490151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:15:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.982+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 >> 172.21.15.29:0/3383917391 conn(0x7fc4900cf570 msgr2=0x7fc4900b7c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:15:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.983+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 shutdown_connections 2023-10-14T18:15:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:15:02.983+0000 7fc496b27640 1 -- 172.21.15.29:0/3383917391 wait complete. 2023-10-14T18:15:03.002 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-14T18:15:03.003 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-14T18:15:03.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:03.006 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:03.064 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:03.064 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:03.064 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:04.066 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:04.066 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:04.134 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:04.135 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:04.135 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:05.136 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:05.137 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:05.205 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:05.205 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:05.206 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:06.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:06.207 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:06.275 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:06.275 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:06.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:07.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:07.278 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:07.345 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:07.346 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:07.346 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:08.347 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:08.348 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:08.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:08.414 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:08.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:09.416 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:09.417 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:09.486 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:09.486 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:09.487 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:10.488 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:10.489 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:10.559 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:10.559 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:10.559 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:11.561 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:11.561 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:11.630 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:11.630 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:11.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:12.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:12.632 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:12.697 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:12.698 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:12.698 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:13.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:13.700 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:13.767 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:13.768 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:13.768 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:14.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:14.770 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:14.838 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:14.839 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:14.839 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:15.840 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:15.840 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:15.905 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:15.906 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:15.906 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:15.954 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:15:15.963 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:15:16.907 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:16.907 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:16.976 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:16.977 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:16.977 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:17.979 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:17.979 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:18.045 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:18.045 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:18.046 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:19.047 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:19.048 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:19.117 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:19.117 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:19.118 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:20.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:20.120 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:20.187 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:20.188 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:20.188 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:21.190 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:21.191 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:21.257 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:21.257 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:21.258 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:22.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:22.261 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:22.329 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:22.330 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:22.330 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:23.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:23.332 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:23.397 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:23.397 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:23.398 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:24.399 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:24.400 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:24.466 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:24.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:24.467 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:25.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:25.468 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:25.538 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:25.538 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:25.539 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:26.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:26.541 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:26.609 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:26.610 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:26.610 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:27.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:27.612 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:27.679 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:27.679 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:27.679 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:28.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:28.681 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:28.748 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:28.749 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:28.749 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:29.750 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:29.751 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:29.820 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:29.821 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:29.822 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:30.823 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:30.824 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:30.892 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:30.892 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:30.893 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:31.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:31.894 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:31.963 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:31.964 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:31.965 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:32.966 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:32.967 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:33.035 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:33.035 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:33.036 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:34.037 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:34.037 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:34.097 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:34.098 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:34.098 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:35.099 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:35.100 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:35.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:35.167 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:35.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:36.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:36.169 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:36.237 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:36.237 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:36.238 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:37.239 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:37.239 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:37.307 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:37.308 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:37.308 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:38.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:38.309 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:38.375 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:38.376 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:38.377 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:39.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:39.378 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:39.446 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:39.447 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:39.447 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:40.449 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:40.449 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:40.516 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:40.516 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:40.517 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:41.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:41.519 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:41.584 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:41.584 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:41.585 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:42.586 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:42.586 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:42.651 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:42.651 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:42.652 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:43.653 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:43.653 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:43.718 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:43.719 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:43.719 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:44.720 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:44.721 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:44.789 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:44.790 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:44.790 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:45.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:45.792 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:45.858 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:45.858 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:45.859 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:46.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:15:46.176 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:15:46.860 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:46.861 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:46.929 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:46.930 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:46.931 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:47.933 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:47.933 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:48.000 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:48.000 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:48.001 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:49.001 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:49.002 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:49.069 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:49.070 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:49.070 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:50.071 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:50.072 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:50.137 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:50.137 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:50.138 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:51.139 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:51.139 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:51.207 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:51.207 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:51.208 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:52.208 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:52.209 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:52.275 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:52.276 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:52.276 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:53.277 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:53.278 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:53.346 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:53.347 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:53.347 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:54.348 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:54.348 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:54.414 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:54.415 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:54.415 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:55.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:55.417 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:55.485 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:55.485 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:55.485 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:56.487 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:56.488 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:56.554 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:56.554 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:56.555 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:57.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:57.556 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:57.623 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:57.624 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:57.624 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:58.625 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:58.626 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:58.692 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:58.692 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:58.693 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:15:59.694 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:15:59.695 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:15:59.762 INFO:tasks.cephfs.test_snap_schedules:snapshots: set() 2023-10-14T18:15:59.762 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:15:59.763 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:00.502 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:16:00.501+0000 7f1d8d844640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:16:00.503 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:16:00.501+0000 7f1d8d844640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:16:00.503 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:16:00.501+0000 7f1d8d844640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:16:00.503 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:16:00.501+0000 7f1d8d844640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:16:00.504 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:16:00.501+0000 7f1d8d844640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:16:00.763 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:00.764 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:00.827 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:00.829 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:00.829 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:00.829 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:00.830 DEBUG:tasks.cephfs.test_snap_schedules:snapshots added=['scheduled-2023-10-14-18_16_00_UTC'] 2023-10-14T18:16:00.830 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697307302.5048268, now = 1697307360.8306458, timo = 58 2023-10-14T18:16:01.832 DEBUG:tasks.cephfs.test_snap_schedules:expecting removal of snap snap_test_dir1/.snap/scheduled-2023-10-14-18_16_00 in ~118s... 2023-10-14T18:16:01.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:01.833 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:01.898 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:01.900 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:01.900 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:01.900 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:02.901 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:02.902 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:02.969 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:02.971 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:02.971 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:02.972 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:03.973 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:03.974 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:04.040 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:04.042 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:04.042 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:04.042 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:05.044 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:05.044 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:05.112 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:05.113 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:05.114 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:05.114 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:06.115 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:06.115 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:06.182 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:06.183 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:06.184 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:06.184 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:07.185 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:07.186 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:07.254 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:07.255 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:07.256 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:07.256 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:08.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:08.259 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:08.324 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:08.325 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:08.325 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:08.326 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:09.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:09.327 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:09.393 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:09.395 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:09.395 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:09.396 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:10.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:10.398 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:10.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:10.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:10.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:10.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:11.468 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:11.469 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:11.533 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:11.535 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:11.535 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:11.536 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:12.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:12.537 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:12.604 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:12.606 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:12.606 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:12.606 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:13.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:13.609 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:13.673 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:13.675 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:13.675 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:13.676 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:14.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:14.677 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:14.742 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:14.744 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:14.744 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:14.744 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:15.746 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:15.746 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:15.812 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:15.814 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:15.814 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:15.815 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:16.375 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:16:16.384 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:16:16.815 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:16.816 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:16.882 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:16.883 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:16.883 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:16.884 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:17.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:17.885 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:17.951 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:17.952 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:17.953 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:17.953 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:18.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:18.955 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:19.023 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:19.025 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:19.025 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:19.026 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:20.027 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:20.027 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:20.093 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:20.094 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:20.095 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:20.095 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:21.096 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:21.097 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:21.164 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:21.166 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:21.166 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:21.167 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:22.167 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:22.169 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:22.234 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:22.235 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:22.235 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:22.236 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:23.237 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:23.237 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:23.304 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:23.305 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:23.306 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:23.306 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:24.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:24.307 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:24.375 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:24.376 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:24.377 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:24.377 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:25.379 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:25.380 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:25.446 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:25.450 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:25.450 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:25.451 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:26.451 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:26.452 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:26.518 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:26.519 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:26.519 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:26.519 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:27.521 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:27.521 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:27.586 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:27.588 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:27.588 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:27.588 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:28.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:28.590 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:28.657 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:28.658 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:28.659 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:28.659 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:29.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:29.661 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:29.725 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:29.726 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:29.727 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:29.727 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:30.728 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:30.728 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:30.795 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:30.796 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:30.797 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:30.797 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:31.798 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:31.798 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:31.865 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:31.866 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:31.866 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:31.867 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:32.868 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:32.869 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:32.935 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:32.936 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:32.937 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:32.937 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:33.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:33.938 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:34.003 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:34.004 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:34.005 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:34.005 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:35.007 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:35.007 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:35.074 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:35.075 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:35.076 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:35.076 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:36.077 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:36.077 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:36.143 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:36.144 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:36.145 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:36.145 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:37.147 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:37.147 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:37.212 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:37.213 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:37.214 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:37.214 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:38.216 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:38.216 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:38.282 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:38.283 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:38.283 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:38.284 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:39.285 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:39.286 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:39.351 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:39.352 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:39.353 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:39.353 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:40.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:40.355 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:40.421 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:40.423 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:40.423 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:40.423 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:41.425 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:41.425 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:41.490 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:41.491 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:41.492 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:41.492 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:42.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:42.494 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:42.560 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:42.562 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:42.562 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:42.562 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:43.563 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:43.564 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:43.629 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:43.630 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:43.630 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:43.631 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:44.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:44.632 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:44.698 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:44.699 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:44.699 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:44.700 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:45.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:45.701 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:45.762 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:45.763 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:45.763 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:45.764 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:46.535 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:16:46.545 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:16:46.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:46.765 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:46.830 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:46.831 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:46.831 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:46.832 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:47.833 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:47.834 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:47.899 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:47.901 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:47.901 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:47.902 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:48.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:48.904 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:48.972 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:48.973 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:48.973 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:48.974 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:49.974 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:49.975 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:50.038 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:50.039 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:50.040 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:50.040 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:51.041 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:51.042 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:51.108 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:51.110 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:51.110 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:51.110 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:52.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:52.112 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:52.180 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:52.182 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:52.182 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:52.182 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:53.183 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:53.183 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:53.251 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:53.252 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:53.253 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:53.253 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:54.254 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:54.255 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:54.322 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:54.324 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:54.324 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:54.324 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:55.326 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:55.327 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:55.393 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:55.394 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:55.395 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:55.395 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:56.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:56.397 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:56.464 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:56.465 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:56.466 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:56.466 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:57.466 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:57.467 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:57.534 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:57.535 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:57.536 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:57.536 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:58.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:58.538 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:58.606 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:58.607 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:58.607 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:58.608 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:16:59.609 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:16:59.609 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:16:59.676 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:16:59.678 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:16:59.678 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:16:59.678 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:17:00.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:00.681 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:17:00.746 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_16_00_UTC 2023-10-14T18:17:00.747 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:17:00.747 INFO:tasks.cephfs.test_snap_schedules:added: set() 2023-10-14T18:17:00.748 INFO:tasks.cephfs.test_snap_schedules:removed: set() 2023-10-14T18:17:01.537 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:01.536+0000 7f1d84fb3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:01.537 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:01.536+0000 7f1d84fb3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:01.537 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:01.536+0000 7f1d84fb3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:01.538 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:01.536+0000 7f1d84fb3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:01.538 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:01.536+0000 7f1d84fb3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:01.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:01.749 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:17:01.810 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_17_00_UTC 2023-10-14T18:17:01.811 INFO:tasks.cephfs.test_snap_schedules:snapshots: {'scheduled-2023-10-14-18_17_00_UTC'} 2023-10-14T18:17:01.811 INFO:tasks.cephfs.test_snap_schedules:added: {'scheduled-2023-10-14-18_17_00_UTC'} 2023-10-14T18:17:01.812 INFO:tasks.cephfs.test_snap_schedules:removed: {'scheduled-2023-10-14-18_16_00_UTC'} 2023-10-14T18:17:01.812 DEBUG:tasks.cephfs.test_snap_schedules:snapshots removed=['scheduled-2023-10-14-18_16_00_UTC'] 2023-10-14T18:17:01.813 DEBUG:tasks.cephfs.test_snap_schedules:exec=1697307302.5048268, now = 1697307421.812985, timo = 118 2023-10-14T18:17:02.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-14T18:17:03.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.053+0000 7f5f3d81a640 1 Processor -- start 2023-10-14T18:17:03.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.054+0000 7f5f3d81a640 1 -- start start 2023-10-14T18:17:03.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.055+0000 7f5f3d81a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5f3806c040 0x7f5f3806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.055+0000 7f5f3d81a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 0x7f5f3815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.055+0000 7f5f3d81a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f3806c960 con 0x7f5f380ad970 2023-10-14T18:17:03.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.055+0000 7f5f3d81a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3815b9d0 con 0x7f5f3806c040 2023-10-14T18:17:03.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.055+0000 7f5f3d81a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815bcb0 con 0x7f5f380cda80 2023-10-14T18:17:03.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f37fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f380ad970 0x7f5f3815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36274/0 (socket says 172.21.15.29:36274) 2023-10-14T18:17:03.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f37fff640 1 -- 172.21.15.29:0/196783417 learned_addr learned my addr 172.21.15.29:0/196783417 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:03.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f3c818640 1 --2- 172.21.15.29:0/196783417 >> v2:172.21.15.186:3300/0 conn(0x7f5f3806c040 0x7f5f3806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f3d019640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 0x7f5f3815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415107315 0 0) 0x7f5f3806c960 con 0x7f5f380ad970 2023-10-14T18:17:03.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.056+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f28003520 con 0x7f5f380ad970 2023-10-14T18:17:03.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812181997 0 0) 0x7f5f28003520 con 0x7f5f380ad970 2023-10-14T18:17:03.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 msgr2=0x7f5f3815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:03.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 0x7f5f3815ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 >> v2:172.21.15.186:3300/0 conn(0x7f5f3806c040 msgr2=0x7f5f3806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 --2- 172.21.15.29:0/196783417 >> v2:172.21.15.186:3300/0 conn(0x7f5f3806c040 0x7f5f3806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f3815bf90 con 0x7f5f380ad970 2023-10-14T18:17:03.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5f24001670 con 0x7f5f380ad970 2023-10-14T18:17:03.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 >> v1:172.21.15.29:6789/0 conn(0x7f5f380ad970 legacy=0x7f5f3815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f28003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f280047c0 0x7f5f28004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 0x7f5f28005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815b9d0 con 0x7f5f28003970 2023-10-14T18:17:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3815bcb0 con 0x7f5f280047c0 2023-10-14T18:17:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2800b980 con 0x7f5f380cda80 2023-10-14T18:17:03.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.057+0000 7f5f3c818640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f28003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f37fff640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f280047c0 0x7f5f28004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f3c818640 1 -- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 msgr2=0x7f5f28005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:03.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f3c818640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380cda80 0x7f5f28005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f3c818640 1 -- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f280047c0 msgr2=0x7f5f28004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f3c818640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f280047c0 0x7f5f28004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f3c818640 1 -- 172.21.15.29:0/196783417 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f28003520 con 0x7f5f28003970 2023-10-14T18:17:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.058+0000 7f5f37fff640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f280047c0 0x7f5f28004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:03.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.059+0000 7f5f3c818640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f28003d90 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009af0 tx=0x7f5f280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb1ac0b3e65b30db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.059+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02c710 con 0x7f5f28003970 2023-10-14T18:17:03.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.059+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02c870 con 0x7f5f28003970 2023-10-14T18:17:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.059+0000 7f5f377fe640 1 -- 172.21.15.29:0/196783417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c02cc00 con 0x7f5f28003970 2023-10-14T18:17:03.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 -- 172.21.15.29:0/196783417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 msgr2=0x7f5f28003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/196783417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f28003d90 secure :-1 s=READY pgs=2446 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c009af0 tx=0x7f5f280057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d019640 1 -- 172.21.15.29:0/196783417 reap_dead start 2023-10-14T18:17:03.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 -- 172.21.15.29:0/196783417 shutdown_connections 2023-10-14T18:17:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 -- 172.21.15.29:0/196783417 >> 172.21.15.29:0/196783417 conn(0x7f5f380ce6a0 msgr2=0x7f5f380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:03.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 -- 172.21.15.29:0/196783417 shutdown_connections 2023-10-14T18:17:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.060+0000 7f5f3d81a640 1 -- 172.21.15.29:0/196783417 wait complete. 2023-10-14T18:17:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.062+0000 7f5f3d81a640 1 Processor -- start 2023-10-14T18:17:03.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.062+0000 7f5f3d81a640 1 -- start start 2023-10-14T18:17:03.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380c5850 0x7f5f380ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f380c5cd0 0x7f5f380acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f381f6eb0 con 0x7f5f28003970 2023-10-14T18:17:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f381f6330 con 0x7f5f380c5cd0 2023-10-14T18:17:03.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3d81a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f381f6610 con 0x7f5f380c5850 2023-10-14T18:17:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3c818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3c818640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38736/0 (socket says 172.21.15.29:38736) 2023-10-14T18:17:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.063+0000 7f5f3c818640 1 -- 172.21.15.29:0/2396472619 learned_addr learned my addr 172.21.15.29:0/2396472619 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 -- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380c5850 msgr2=0x7f5f380ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380c5850 0x7f5f380ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 -- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f380c5cd0 msgr2=0x7f5f380acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f380c5cd0 0x7f5f380acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 -- 172.21.15.29:0/2396472619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f381f68f0 con 0x7f5f28003970 2023-10-14T18:17:03.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3c818640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c02c2b0 tx=0x7f5f2c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c003810 con 0x7f5f28003970 2023-10-14T18:17:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2c02de90 con 0x7f5f28003970 2023-10-14T18:17:03.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2c003b70 con 0x7f5f28003970 2023-10-14T18:17:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f381f6bd0 con 0x7f5f28003970 2023-10-14T18:17:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.064+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f381f5f90 con 0x7f5f28003970 2023-10-14T18:17:03.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.065+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f3806b330 con 0x7f5f28003970 2023-10-14T18:17:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.066+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 130) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f5f2c003cd0 con 0x7f5f28003970 2023-10-14T18:17:03.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.067+0000 7f5f357fa640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 0x7f5f1c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.067+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f5f2c0ab940 con 0x7f5f28003970 2023-10-14T18:17:03.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.068+0000 7f5f37fff640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 0x7f5f1c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.068+0000 7f5f37fff640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 0x7f5f1c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f24000c00 tx=0x7f5f24009040 comp rx=0 tx=0).ready entity=mgr.10896 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.072+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f2c076750 con 0x7f5f28003970 2023-10-14T18:17:03.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.238+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 131) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f5f2c076b20 con 0x7f5f28003970 2023-10-14T18:17:03.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.296+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 --> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7f5f381f6780 con 0x7f5f1c06afe0 2023-10-14T18:17:03.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.481+0000 7f5f357fa640 1 -- 172.21.15.29:0/2396472619 <== mgr.10896 v2:172.21.15.29:6832/966124587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f5f381f6780 con 0x7f5f1c06afe0 2023-10-14T18:17:03.482 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-14T18:17:03.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 msgr2=0x7f5f1c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 0x7f5f1c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f5f24000c00 tx=0x7f5f24009040 comp rx=0 tx=0).stop 2023-10-14T18:17:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 msgr2=0x7f5f381567e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 secure :-1 s=READY pgs=2447 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c02c2b0 tx=0x7f5f2c004730 comp rx=0 tx=0).stop 2023-10-14T18:17:03.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 shutdown_connections 2023-10-14T18:17:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f5f1c06afe0 0x7f5f1c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f380c5cd0 0x7f5f380acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f380c5850 0x7f5f380ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 --2- 172.21.15.29:0/2396472619 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f28003970 0x7f5f381567e0 unknown :-1 s=CLOSED pgs=2447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.485+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 >> 172.21.15.29:0/2396472619 conn(0x7f5f380ce6a0 msgr2=0x7f5f380c4b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:03.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.486+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 shutdown_connections 2023-10-14T18:17:03.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.486+0000 7f5f3d81a640 1 -- 172.21.15.29:0/2396472619 wait complete. 2023-10-14T18:17:03.508 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-14T18:17:03.508 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:03.509 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:17:03.566 INFO:teuthology.orchestra.run.smithi029.stdout:scheduled-2023-10-14-18_17_00_UTC 2023-10-14T18:17:03.568 DEBUG:tasks.cephfs.test_snap_schedules:removing snapshot: snap_test_dir1/.snap/scheduled-2023-10-14-18_17_00_UTC 2023-10-14T18:17:03.568 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:03.568 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1/.snap/scheduled-2023-10-14-18_17_00_UTC) 2023-10-14T18:17:03.634 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:03.634 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T18:17:03.697 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T18:17:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.926+0000 7f4951285640 1 Processor -- start 2023-10-14T18:17:03.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.926+0000 7f4951285640 1 -- start start 2023-10-14T18:17:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.927+0000 7f4951285640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f494c06c040 0x7f494c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.927+0000 7f4951285640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f494c0cda80 0x7f494c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.927+0000 7f4951285640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f494c06c960 con 0x7f494c0ad970 2023-10-14T18:17:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f4951285640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c15b9d0 con 0x7f494c06c040 2023-10-14T18:17:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f4951285640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f494c15bcb0 con 0x7f494c0cda80 2023-10-14T18:17:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f494c06c040 0x7f494c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f494c06c040 0x7f494c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38752/0 (socket says 172.21.15.29:38752) 2023-10-14T18:17:03.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 -- 172.21.15.29:0/518607499 learned_addr learned my addr 172.21.15.29:0/518607499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f4950a84640 1 --2- 172.21.15.29:0/518607499 >> v2:172.21.15.186:3300/0 conn(0x7f494c0cda80 0x7f494c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 -- 172.21.15.29:0/518607499 >> v2:172.21.15.186:3300/0 conn(0x7f494c0cda80 msgr2=0x7f494c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 --2- 172.21.15.29:0/518607499 >> v2:172.21.15.186:3300/0 conn(0x7f494c0cda80 0x7f494c15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.928+0000 7f494bfff640 1 -- 172.21.15.29:0/518607499 >> v1:172.21.15.29:6789/0 conn(0x7f494c0ad970 legacy=0x7f494c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.929+0000 7f494bfff640 1 -- 172.21.15.29:0/518607499 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f494c15bf90 con 0x7f494c06c040 2023-10-14T18:17:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.929+0000 7f494affd640 1 -- 172.21.15.29:0/518607499 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700681319 0 0) 0x7f494c06c960 con 0x7f494c0ad970 2023-10-14T18:17:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f494bfff640 1 --2- 172.21.15.29:0/518607499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c06c040 0x7f494c06c420 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f494c0cf300 tx=0x7f493c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=182b30ba166b9e35 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f494affd640 1 -- 172.21.15.29:0/518607499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c030860 con 0x7f494c06c040 2023-10-14T18:17:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f494affd640 1 -- 172.21.15.29:0/518607499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f493c0309c0 con 0x7f494c06c040 2023-10-14T18:17:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f494affd640 1 -- 172.21.15.29:0/518607499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f493c030d50 con 0x7f494c06c040 2023-10-14T18:17:03.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f4951285640 1 -- 172.21.15.29:0/518607499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c06c040 msgr2=0x7f494c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.930+0000 7f4951285640 1 --2- 172.21.15.29:0/518607499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c06c040 0x7f494c06c420 secure :-1 s=READY pgs=2449 cs=0 l=1 rev1=1 crypto rx=0x7f494c0cf300 tx=0x7f493c02b9a0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 -- 172.21.15.29:0/518607499 shutdown_connections 2023-10-14T18:17:03.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 --2- 172.21.15.29:0/518607499 >> v2:172.21.15.186:3300/0 conn(0x7f494c0cda80 0x7f494c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 --2- 172.21.15.29:0/518607499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c06c040 0x7f494c06c420 unknown :-1 s=CLOSED pgs=2449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 -- 172.21.15.29:0/518607499 >> 172.21.15.29:0/518607499 conn(0x7f494c0ce6a0 msgr2=0x7f494c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:03.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 -- 172.21.15.29:0/518607499 shutdown_connections 2023-10-14T18:17:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.931+0000 7f4951285640 1 -- 172.21.15.29:0/518607499 wait complete. 2023-10-14T18:17:03.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.932+0000 7f4951285640 1 Processor -- start 2023-10-14T18:17:03.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.933+0000 7f4951285640 1 -- start start 2023-10-14T18:17:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f494c06c040 0x7f494c1f63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f494c0ad970 0x7f494c1f1c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0c70b0 con 0x7f494c0cda80 2023-10-14T18:17:03.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f494c0c91b0 con 0x7f494c06c040 2023-10-14T18:17:03.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4951285640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f494c1534e0 con 0x7f494c0ad970 2023-10-14T18:17:03.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4950a84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4950a84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38762/0 (socket says 172.21.15.29:38762) 2023-10-14T18:17:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f4950a84640 1 -- 172.21.15.29:0/1709785395 learned_addr learned my addr 172.21.15.29:0/1709785395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f494bfff640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f494c06c040 0x7f494c1f63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.934+0000 7f494b7fe640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f494c0ad970 0x7f494c1f1c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 -- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f494c0ad970 msgr2=0x7f494c1f1c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f494c0ad970 0x7f494c1f1c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 -- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f494c06c040 msgr2=0x7f494c1f63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f494c06c040 0x7f494c1f63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 -- 172.21.15.29:0/1709785395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f493c0094d0 con 0x7f494c0cda80 2023-10-14T18:17:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4950a84640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f4944007e90 tx=0x7f494400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944014420 con 0x7f494c0cda80 2023-10-14T18:17:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4944014580 con 0x7f494c0cda80 2023-10-14T18:17:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4944014930 con 0x7f494c0cda80 2023-10-14T18:17:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f494c1f6900 con 0x7f494c0cda80 2023-10-14T18:17:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.935+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f494c0c6af0 con 0x7f494c0cda80 2023-10-14T18:17:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.937+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f494c06b330 con 0x7f494c0cda80 2023-10-14T18:17:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.938+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 131) v1 ==== 89360+0+0 (secure 0 0 0) 0x7f4944014a90 con 0x7f494c0cda80 2023-10-14T18:17:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.939+0000 7f4948ff9640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 0x7f493806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.939+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6227+0+0 (secure 0 0 0) 0x7f494408a2f0 con 0x7f494c0cda80 2023-10-14T18:17:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.939+0000 7f494bfff640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 0x7f493806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.940+0000 7f494bfff640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 0x7f493806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f493c036000 tx=0x7f493c008030 comp rx=0 tx=0).ready entity=mgr.10896 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:03.943+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4944055070 con 0x7f494c0cda80 2023-10-14T18:17:04.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.189+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f494c0c9320 con 0x7f494c0cda80 2023-10-14T18:17:04.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.250+0000 7f4948ff9640 1 -- 172.21.15.29:0/1709785395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v132) v1 ==== 93+0+0 (secure 0 0 0) 0x7f494c0c9320 con 0x7f494c0cda80 2023-10-14T18:17:04.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 msgr2=0x7f493806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 0x7f493806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f493c036000 tx=0x7f493c008030 comp rx=0 tx=0).stop 2023-10-14T18:17:04.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 msgr2=0x7f494c0ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 secure :-1 s=READY pgs=2450 cs=0 l=1 rev1=1 crypto rx=0x7f4944007e90 tx=0x7f494400e4b0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 shutdown_connections 2023-10-14T18:17:04.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f493806afe0 0x7f493806d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f494c0cda80 0x7f494c0ac890 unknown :-1 s=CLOSED pgs=2450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f494c0ad970 0x7f494c1f1c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 --2- 172.21.15.29:0/1709785395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f494c06c040 0x7f494c1f63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.255+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 >> 172.21.15.29:0/1709785395 conn(0x7f494c0ce6a0 msgr2=0x7f494c15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:04.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.256+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 shutdown_connections 2023-10-14T18:17:04.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.256+0000 7f4951285640 1 -- 172.21.15.29:0/1709785395 wait complete. 2023-10-14T18:17:04.278 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:17:04.278 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:17:04.401 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:17:04.501 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:17:04.501 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:04.502 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:04.559 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:04.559 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:04.607 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:04.808 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:04.807+0000 7fa900a11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:17:04.821 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:04.819+0000 7f954542c040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:17:04.867 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:04.866+0000 7fa900a11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:17:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.868+0000 7fb203180640 1 Processor -- start 2023-10-14T18:17:04.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.868+0000 7fb203180640 1 -- start start 2023-10-14T18:17:04.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.869+0000 7fb203180640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ad970 0x7fb1fc0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb203180640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1fc0ccfb0 0x7fb1fc0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb203180640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1fc15b9d0 con 0x7fb1fc06c040 2023-10-14T18:17:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb203180640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1fc15bcb0 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb203180640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc15bf90 con 0x7fb1fc0ad970 2023-10-14T18:17:04.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20217e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1fc06c040 0x7fb1fc06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36290/0 (socket says 172.21.15.29:36290) 2023-10-14T18:17:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20217e640 1 -- 172.21.15.29:0/3646310643 learned_addr learned my addr 172.21.15.29:0/3646310643 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:04.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20297f640 1 --2- 172.21.15.29:0/3646310643 >> v2:172.21.15.29:3300/0 conn(0x7fb1fc0ccfb0 0x7fb1fc0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20197d640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ad970 0x7fb1fc0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20117c640 1 -- 172.21.15.29:0/3646310643 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772046438 0 0) 0x7fb1fc15b9d0 con 0x7fb1fc06c040 2023-10-14T18:17:04.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.870+0000 7fb20117c640 1 -- 172.21.15.29:0/3646310643 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1e8003520 con 0x7fb1fc06c040 2023-10-14T18:17:04.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.871+0000 7fb20297f640 1 -- 172.21.15.29:0/3646310643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ad970 msgr2=0x7fb1fc0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:04.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.871+0000 7fb20297f640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ad970 0x7fb1fc0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.871+0000 7fb20297f640 1 -- 172.21.15.29:0/3646310643 >> v1:172.21.15.29:6789/0 conn(0x7fb1fc06c040 legacy=0x7fb1fc06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.871+0000 7fb20297f640 1 -- 172.21.15.29:0/3646310643 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1fc0cc270 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.879 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:04.874+0000 7f954542c040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:17:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.872+0000 7fb20297f640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0cd440 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009dc0 tx=0x7fb1f402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cca713aea764b3d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:04.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb20117c640 1 -- 172.21.15.29:0/3646310643 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f402cc40 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb20117c640 1 -- 172.21.15.29:0/3646310643 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f402cda0 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb20117c640 1 -- 172.21.15.29:0/3646310643 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f4034660 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 -- 172.21.15.29:0/3646310643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc0ccfb0 msgr2=0x7fb1fc0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0cd440 secure :-1 s=READY pgs=2457 cs=0 l=1 rev1=1 crypto rx=0x7fb1f4009dc0 tx=0x7fb1f402be90 comp rx=0 tx=0).stop 2023-10-14T18:17:04.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 -- 172.21.15.29:0/3646310643 shutdown_connections 2023-10-14T18:17:04.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0cd440 unknown :-1 s=CLOSED pgs=2457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 --2- 172.21.15.29:0/3646310643 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ad970 0x7fb1fc0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 -- 172.21.15.29:0/3646310643 >> 172.21.15.29:0/3646310643 conn(0x7fb1fc0ce6a0 msgr2=0x7fb1fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.873+0000 7fb203180640 1 -- 172.21.15.29:0/3646310643 shutdown_connections 2023-10-14T18:17:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.874+0000 7fb203180640 1 -- 172.21.15.29:0/3646310643 wait complete. 2023-10-14T18:17:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.876+0000 7fb203180640 1 Processor -- start 2023-10-14T18:17:04.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.876+0000 7fb203180640 1 -- start start 2023-10-14T18:17:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1fc0ad970 0x7fb1fc0af050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0aff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc157d40 con 0x7fb1fc06c040 2023-10-14T18:17:04.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0afc40 con 0x7fb1fc0ad970 2023-10-14T18:17:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb203180640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1fc0ad170 con 0x7fb1fc0ccfb0 2023-10-14T18:17:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb20217e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb20217e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38788/0 (socket says 172.21.15.29:38788) 2023-10-14T18:17:04.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.878+0000 7fb20217e640 1 -- 172.21.15.29:0/2539079882 learned_addr learned my addr 172.21.15.29:0/2539079882 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20197d640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1fc0ad970 0x7fb1fc0af050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ccfb0 msgr2=0x7fb1fc0aff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0aff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1fc0ad970 msgr2=0x7fb1fc0af050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1fc0ad970 0x7fb1fc0af050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 -- 172.21.15.29:0/2539079882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1f4009a70 con 0x7fb1fc06c040 2023-10-14T18:17:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb20217e640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800c500 tx=0x7fb1f800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:04.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8005ce0 con 0x7fb1fc06c040 2023-10-14T18:17:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1f8005e40 con 0x7fb1fc06c040 2023-10-14T18:17:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1fc0af680 con 0x7fb1fc06c040 2023-10-14T18:17:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.879+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1f8007330 con 0x7fb1fc06c040 2023-10-14T18:17:04.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.880+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1fc157960 con 0x7fb1fc06c040 2023-10-14T18:17:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.881+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1fc06b330 con 0x7fb1fc06c040 2023-10-14T18:17:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.883+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb1f8007550 con 0x7fb1fc06c040 2023-10-14T18:17:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.884+0000 7fb1f2ffd640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 0x7fb1e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:04.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.884+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(213..213 src has 1..213) v4 ==== 6227+0+0 (secure 0 0 0) 0x7fb1f8088ff0 con 0x7fb1fc06c040 2023-10-14T18:17:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.884+0000 7fb20197d640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 msgr2=0x7fb1e006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:04.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.884+0000 7fb20197d640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 0x7fb1e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:04.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:04.888+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1f8053e90 con 0x7fb1fc06c040 2023-10-14T18:17:05.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.084+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1fc1577c0 con 0x7fb1fc06c040 2023-10-14T18:17:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.085+0000 7fb20197d640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 msgr2=0x7fb1e006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:05.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.085+0000 7fb20197d640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 0x7fb1e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:05.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.088+0000 7fb1f2ffd640 1 -- 172.21.15.29:0/2539079882 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v213) v1 ==== 74+0+17296 (secure 0 0 0) 0x7fb1f80863a0 con 0x7fb1fc06c040 2023-10-14T18:17:05.089 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:05.089 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":213,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:16:02.375322+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"210","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.17","mappings":[{"from":0,"to":2}]},{"pgid":"25.2d","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/4216364220":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6832/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1790177658":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/3670605654":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/2768480413":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/4181160999":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6833/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1466849565":"2023-10-15T18:15:00.600793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:05.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 msgr2=0x7fb1e006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:05.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 0x7fb1e006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 msgr2=0x7fb1fc0aeb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 secure :-1 s=READY pgs=2458 cs=0 l=1 rev1=1 crypto rx=0x7fb1f800c500 tx=0x7fb1f800c9c0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 shutdown_connections 2023-10-14T18:17:05.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb1e006af90 0x7fb1e006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1fc0ccfb0 0x7fb1fc0aff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1fc0ad970 0x7fb1fc0af050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 --2- 172.21.15.29:0/2539079882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1fc06c040 0x7fb1fc0aeb10 unknown :-1 s=CLOSED pgs=2458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 >> 172.21.15.29:0/2539079882 conn(0x7fb1fc0ce6a0 msgr2=0x7fb1fc0b3580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:05.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.090+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 shutdown_connections 2023-10-14T18:17:05.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.091+0000 7fb203180640 1 -- 172.21.15.29:0/2539079882 wait complete. 2023-10-14T18:17:05.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:14:43.916335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:14:45.028887+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '210', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:17:05.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:17:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.329+0000 7efca56bb640 1 Processor -- start 2023-10-14T18:17:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.330+0000 7efca56bb640 1 -- start start 2023-10-14T18:17:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.332+0000 7efca56bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efca00cf5f0 0x7efca006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.332+0000 7efca56bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca00cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.332+0000 7efca56bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efca0161740 con 0x7efca0070830 2023-10-14T18:17:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.332+0000 7efca56bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efca00cdbd0 con 0x7efca00cf5f0 2023-10-14T18:17:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.332+0000 7efca56bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca006b0f0 con 0x7efca0070c10 2023-10-14T18:17:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.333+0000 7efc9e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7efca00cf5f0 0x7efca006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.333+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca00cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.333+0000 7efc9effd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7efca0070830 0x7efca006c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:39432/0 (socket says 172.21.15.29:39432) 2023-10-14T18:17:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.333+0000 7efc9effd640 1 -- 172.21.15.29:0/4087192673 learned_addr learned my addr 172.21.15.29:0/4087192673 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.334+0000 7efc9f7fe640 1 -- 172.21.15.29:0/4087192673 >> v2:172.21.15.186:3300/0 conn(0x7efca00cf5f0 msgr2=0x7efca006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.334+0000 7efc9f7fe640 1 --2- 172.21.15.29:0/4087192673 >> v2:172.21.15.186:3300/0 conn(0x7efca00cf5f0 0x7efca006d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.334+0000 7efc9f7fe640 1 -- 172.21.15.29:0/4087192673 >> v1:172.21.15.186:6789/0 conn(0x7efca0070830 legacy=0x7efca006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.334+0000 7efc9f7fe640 1 -- 172.21.15.29:0/4087192673 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efca01619b0 con 0x7efca0070c10 2023-10-14T18:17:05.340 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.337+0000 7f954542c040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:17:05.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.336+0000 7efc9f7fe640 1 --2- 172.21.15.29:0/4087192673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca00cd690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc98009d90 tx=0x7efc9802bd90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=88f1c557cde129ce server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:05.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.336+0000 7efc9dffb640 1 -- 172.21.15.29:0/4087192673 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9802cb40 con 0x7efca0070c10 2023-10-14T18:17:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.336+0000 7efc9dffb640 1 -- 172.21.15.29:0/4087192673 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9802cca0 con 0x7efca0070c10 2023-10-14T18:17:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.336+0000 7efc9dffb640 1 -- 172.21.15.29:0/4087192673 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc98034660 con 0x7efca0070c10 2023-10-14T18:17:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.337+0000 7efca56bb640 1 -- 172.21.15.29:0/4087192673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 msgr2=0x7efca00cd690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.337+0000 7efca56bb640 1 --2- 172.21.15.29:0/4087192673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca00cd690 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efc98009d90 tx=0x7efc9802bd90 comp rx=0 tx=0).stop 2023-10-14T18:17:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.338+0000 7efca56bb640 1 -- 172.21.15.29:0/4087192673 shutdown_connections 2023-10-14T18:17:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.338+0000 7efca56bb640 1 --2- 172.21.15.29:0/4087192673 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca00cd690 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.338+0000 7efca56bb640 1 --2- 172.21.15.29:0/4087192673 >> v2:172.21.15.186:3300/0 conn(0x7efca00cf5f0 0x7efca006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.338+0000 7efca56bb640 1 -- 172.21.15.29:0/4087192673 >> 172.21.15.29:0/4087192673 conn(0x7efca00cfac0 msgr2=0x7efca00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:05.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.339+0000 7efca56bb640 1 -- 172.21.15.29:0/4087192673 shutdown_connections 2023-10-14T18:17:05.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.339+0000 7efca56bb640 1 -- 172.21.15.29:0/4087192673 wait complete. 2023-10-14T18:17:05.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.344+0000 7efca56bb640 1 Processor -- start 2023-10-14T18:17:05.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.344+0000 7efca56bb640 1 -- start start 2023-10-14T18:17:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efca0070830 0x7efca00c6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca015b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c3b20 con 0x7efca00cf5f0 2023-10-14T18:17:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c3e00 con 0x7efca0070830 2023-10-14T18:17:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efca56bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efca00c40e0 con 0x7efca0070c10 2023-10-14T18:17:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efc9f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38794/0 (socket says 172.21.15.29:38794) 2023-10-14T18:17:05.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efc9f7fe640 1 -- 172.21.15.29:0/2622645008 learned_addr learned my addr 172.21.15.29:0/2622645008 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.346+0000 7efc9effd640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efca0070830 0x7efca00c6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 -- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 msgr2=0x7efca015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 -- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efca0070830 msgr2=0x7efca00c6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efca0070830 0x7efca00c6360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 -- 172.21.15.29:0/2622645008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc98009a70 con 0x7efca00cf5f0 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.347+0000 7efc9f7fe640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7efc98009eb0 tx=0x7efc98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.348+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9803e610 con 0x7efca00cf5f0 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.348+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc9803e770 con 0x7efca00cf5f0 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.348+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efca00c43c0 con 0x7efca00cf5f0 2023-10-14T18:17:05.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.348+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc9803eb60 con 0x7efca00cf5f0 2023-10-14T18:17:05.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.348+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efca00c37a0 con 0x7efca00cf5f0 2023-10-14T18:17:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.349+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efca00c48e0 con 0x7efca00cf5f0 2023-10-14T18:17:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.350+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7efc9803b050 con 0x7efca00cf5f0 2023-10-14T18:17:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.352+0000 7efc877fe640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 0x7efc8c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.352+0000 7efc9effd640 1 -- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 msgr2=0x7efc8c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:05.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.352+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6315+0+0 (secure 0 0 0) 0x7efc980abcf0 con 0x7efca00cf5f0 2023-10-14T18:17:05.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.352+0000 7efc9effd640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 0x7efc8c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:05.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.356+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efca00c48e0 con 0x7efca00cf5f0 2023-10-14T18:17:05.365 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.364+0000 7fa900a11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:17:05.397 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.396+0000 7f954542c040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:17:05.421 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.420+0000 7fa900a11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:17:05.448 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.447+0000 7f954542c040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:17:05.474 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.473+0000 7fa900a11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:17:05.509 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.508+0000 7f954542c040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:17:05.538 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.536+0000 7fa900a11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:17:05.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.549+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7efca00c48e0 con 0x7efca00cf5f0 2023-10-14T18:17:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.549+0000 7efc877fe640 1 -- 172.21.15.29:0/2622645008 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v214) v1 ==== 93+0+10 (secure 0 0 0) 0x7efca00c48e0 con 0x7efca00cf5f0 2023-10-14T18:17:05.551 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:17:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 msgr2=0x7efc8c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 0x7efc8c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 msgr2=0x7efca00c3260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 secure :-1 s=READY pgs=2459 cs=0 l=1 rev1=1 crypto rx=0x7efc98009eb0 tx=0x7efc98004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 shutdown_connections 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7efc8c06af90 0x7efc8c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7efca00cf5f0 0x7efca00c3260 unknown :-1 s=CLOSED pgs=2459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7efca0070c10 0x7efca015b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 --2- 172.21.15.29:0/2622645008 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7efca0070830 0x7efca00c6360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.552+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 >> 172.21.15.29:0/2622645008 conn(0x7efca00cfac0 msgr2=0x7efca00cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.553+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 shutdown_connections 2023-10-14T18:17:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.553+0000 7efca56bb640 1 -- 172.21.15.29:0/2622645008 wait complete. 2023-10-14T18:17:05.558 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.557+0000 7f954542c040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:17:05.571 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:17:05.591 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.590+0000 7fa900a11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:17:05.669 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.668+0000 7f954542c040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:17:05.710 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.709+0000 7fa900a11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:17:05.732 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.730+0000 7f954542c040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.764+0000 7f676586d640 1 Processor -- start 2023-10-14T18:17:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.764+0000 7f676586d640 1 -- start start 2023-10-14T18:17:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676586d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f676015cd70 0x7f67600d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676586d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f67600cded0 0x7f67600b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676586d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676015d150 con 0x7f67600b8480 2023-10-14T18:17:05.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676586d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67600d1180 con 0x7f676015cd70 2023-10-14T18:17:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676586d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67600d1460 con 0x7f67600cded0 2023-10-14T18:17:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676486b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f676015cd70 0x7f67600d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676486b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f676015cd70 0x7f67600d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38804/0 (socket says 172.21.15.29:38804) 2023-10-14T18:17:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f676486b640 1 -- 172.21.15.29:0/3611888196 learned_addr learned my addr 172.21.15.29:0/3611888196 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:05.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.766+0000 7f675ffff640 1 --1- 172.21.15.29:0/3611888196 >> v1:172.21.15.29:6789/0 conn(0x7f67600b8480 0x7f67600cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36300/0 (socket says 172.21.15.29:36300) 2023-10-14T18:17:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.767+0000 7f676506c640 1 --2- 172.21.15.29:0/3611888196 >> v2:172.21.15.186:3300/0 conn(0x7f67600cded0 0x7f67600b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.767+0000 7f676486b640 1 -- 172.21.15.29:0/3611888196 >> v2:172.21.15.186:3300/0 conn(0x7f67600cded0 msgr2=0x7f67600b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.767+0000 7f676486b640 1 --2- 172.21.15.29:0/3611888196 >> v2:172.21.15.186:3300/0 conn(0x7f67600cded0 0x7f67600b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.767+0000 7f676486b640 1 -- 172.21.15.29:0/3611888196 >> v1:172.21.15.29:6789/0 conn(0x7f67600b8480 legacy=0x7f67600cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.767+0000 7f676486b640 1 -- 172.21.15.29:0/3611888196 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67600d1740 con 0x7f676015cd70 2023-10-14T18:17:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.768+0000 7f676486b640 1 --2- 172.21.15.29:0/3611888196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2220 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f67600b8910 tx=0x7f675402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e093a4e07810013e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.768+0000 7f675f7fe640 1 -- 172.21.15.29:0/3611888196 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754030870 con 0x7f676015cd70 2023-10-14T18:17:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.768+0000 7f675f7fe640 1 -- 172.21.15.29:0/3611888196 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67540309d0 con 0x7f676015cd70 2023-10-14T18:17:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.768+0000 7f675f7fe640 1 -- 172.21.15.29:0/3611888196 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6754030d60 con 0x7f676015cd70 2023-10-14T18:17:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 -- 172.21.15.29:0/3611888196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 msgr2=0x7f67600d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 --2- 172.21.15.29:0/3611888196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2220 secure :-1 s=READY pgs=2461 cs=0 l=1 rev1=1 crypto rx=0x7f67600b8910 tx=0x7f675402b9b0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 -- 172.21.15.29:0/3611888196 shutdown_connections 2023-10-14T18:17:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 --2- 172.21.15.29:0/3611888196 >> v2:172.21.15.186:3300/0 conn(0x7f67600cded0 0x7f67600b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 --2- 172.21.15.29:0/3611888196 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2220 unknown :-1 s=CLOSED pgs=2461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 -- 172.21.15.29:0/3611888196 >> 172.21.15.29:0/3611888196 conn(0x7f67600cf570 msgr2=0x7f67600cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 -- 172.21.15.29:0/3611888196 shutdown_connections 2023-10-14T18:17:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.769+0000 7f676586d640 1 -- 172.21.15.29:0/3611888196 wait complete. 2023-10-14T18:17:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.771+0000 7f676586d640 1 Processor -- start 2023-10-14T18:17:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.771+0000 7f676586d640 1 -- start start 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67600b8480 0x7f67600b1f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 0x7f67600d2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600b25d0 con 0x7f676015cd70 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67600b28b0 con 0x7f67600cded0 2023-10-14T18:17:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676586d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67600b2b90 con 0x7f67600b8480 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676506c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676506c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38820/0 (socket says 172.21.15.29:38820) 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.772+0000 7f676506c640 1 -- 172.21.15.29:0/1582561219 learned_addr learned my addr 172.21.15.29:0/1582561219 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676486b640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67600b8480 0x7f67600b1f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f675ffff640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 0x7f67600d2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67600b8480 msgr2=0x7f67600b1f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67600b8480 0x7f67600b1f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 msgr2=0x7f67600d2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 0x7f67600d2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 -- 172.21.15.29:0/1582561219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67540094d0 con 0x7f676015cd70 2023-10-14T18:17:05.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f675ffff640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 0x7f67600d2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:05.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f676506c640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f6758007d90 tx=0x7f675800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.773+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758014420 con 0x7f676015cd70 2023-10-14T18:17:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.774+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67580145a0 con 0x7f676015cd70 2023-10-14T18:17:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.774+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67600b2e70 con 0x7f676015cd70 2023-10-14T18:17:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.774+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6758014950 con 0x7f676015cd70 2023-10-14T18:17:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.774+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676014bbd0 con 0x7f676015cd70 2023-10-14T18:17:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.775+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676004e9b0 con 0x7f676015cd70 2023-10-14T18:17:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.776+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f6758014ab0 con 0x7f676015cd70 2023-10-14T18:17:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.777+0000 7f675d7fa640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 0x7f674406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.777+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(214..214 src has 1..214) v4 ==== 6315+0+0 (secure 0 0 0) 0x7f675808a360 con 0x7f676015cd70 2023-10-14T18:17:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.777+0000 7f676486b640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 msgr2=0x7f674406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.777+0000 7f676486b640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 0x7f674406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:05.783 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.780+0000 7fa900a11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:05.783 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.782+0000 7f954542c040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:17:05.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.785+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6758055090 con 0x7f676015cd70 2023-10-14T18:17:05.835 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.834+0000 7fa900a11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:17:05.840 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.839+0000 7f954542c040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:17:05.890 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:05.889+0000 7f954542c040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:17:05.896 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.895+0000 7fa900a11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:17:05.949 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:05.948+0000 7fa900a11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:17:05.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.978+0000 7f676486b640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 msgr2=0x7f674406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.978+0000 7f676486b640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 0x7f674406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.979+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f676015c4a0 con 0x7f676015cd70 2023-10-14T18:17:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.980+0000 7f675d7fa640 1 -- 172.21.15.29:0/1582561219 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v214) v1 ==== 104+0+11 (secure 0 0 0) 0x7f67580586b0 con 0x7f676015cd70 2023-10-14T18:17:05.980 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:17:05.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 msgr2=0x7f674406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 0x7f674406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 msgr2=0x7f67600d2fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:05.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 secure :-1 s=READY pgs=2462 cs=0 l=1 rev1=1 crypto rx=0x7f6758007d90 tx=0x7f675800e4b0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 shutdown_connections 2023-10-14T18:17:05.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f674406af90 0x7f674406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f676015cd70 0x7f67600d2fc0 unknown :-1 s=CLOSED pgs=2462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f67600cded0 0x7f67600d2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 --2- 172.21.15.29:0/1582561219 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f67600b8480 0x7f67600b1f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 >> 172.21.15.29:0/1582561219 conn(0x7f67600cf570 msgr2=0x7f67600c8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 shutdown_connections 2023-10-14T18:17:05.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:05.982+0000 7f676586d640 1 -- 172.21.15.29:0/1582561219 wait complete. 2023-10-14T18:17:06.003 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:17:06.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.264+0000 7ff8f6c15640 1 Processor -- start 2023-10-14T18:17:06.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.264+0000 7ff8f6c15640 1 -- start start 2023-10-14T18:17:06.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f6c15640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff8f0159ab0 0x7ff8f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f6c15640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f006c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f6c15640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f0070360 con 0x7ff8f0159e90 2023-10-14T18:17:06.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f6c15640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8f006cb10 con 0x7ff8f0159ab0 2023-10-14T18:17:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f6c15640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015d890 con 0x7ff8f006c140 2023-10-14T18:17:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8effff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff8f0159e90 0x7ff8f006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36304/0 (socket says 172.21.15.29:36304) 2023-10-14T18:17:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8effff640 1 -- 172.21.15.29:0/926444369 learned_addr learned my addr 172.21.15.29:0/926444369 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f49b1640 1 --2- 172.21.15.29:0/926444369 >> v2:172.21.15.186:3300/0 conn(0x7ff8f0159ab0 0x7ff8f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.266+0000 7ff8f51b2640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f006c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8ef7fe640 1 -- 172.21.15.29:0/926444369 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171323882 0 0) 0x7ff8f0070360 con 0x7ff8f0159e90 2023-10-14T18:17:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8ef7fe640 1 -- 172.21.15.29:0/926444369 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e0005730 con 0x7ff8f0159e90 2023-10-14T18:17:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8f49b1640 1 -- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 msgr2=0x7ff8f006c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8f49b1640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f006c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8f49b1640 1 -- 172.21.15.29:0/926444369 >> v1:172.21.15.29:6789/0 conn(0x7ff8f0159e90 legacy=0x7ff8f006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.267+0000 7ff8f49b1640 1 -- 172.21.15.29:0/926444369 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f00cd1f0 con 0x7ff8f0159ab0 2023-10-14T18:17:06.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.268+0000 7ff8f49b1640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00ce370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097a0 tx=0x7ff8e402b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=144aa7f311863bb5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:06.273 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.272+0000 7f954542c040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:17:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.276+0000 7ff8ef7fe640 1 -- 172.21.15.29:0/926444369 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e40316f0 con 0x7ff8f0159ab0 2023-10-14T18:17:06.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.277+0000 7ff8ef7fe640 1 -- 172.21.15.29:0/926444369 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e4031850 con 0x7ff8f0159ab0 2023-10-14T18:17:06.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.277+0000 7ff8ef7fe640 1 -- 172.21.15.29:0/926444369 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e4031be0 con 0x7ff8f0159ab0 2023-10-14T18:17:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.277+0000 7ff8f6c15640 1 -- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 msgr2=0x7ff8f00ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.277+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00ce370 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7ff8e40097a0 tx=0x7ff8e402b790 comp rx=0 tx=0).stop 2023-10-14T18:17:06.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.277+0000 7ff8f6c15640 1 -- 172.21.15.29:0/926444369 shutdown_connections 2023-10-14T18:17:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.278+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f006c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.278+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/926444369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00ce370 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.278+0000 7ff8f6c15640 1 -- 172.21.15.29:0/926444369 >> 172.21.15.29:0/926444369 conn(0x7ff8f00ce9b0 msgr2=0x7ff8f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.278+0000 7ff8f6c15640 1 -- 172.21.15.29:0/926444369 shutdown_connections 2023-10-14T18:17:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.278+0000 7ff8f6c15640 1 -- 172.21.15.29:0/926444369 wait complete. 2023-10-14T18:17:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.281+0000 7ff8f6c15640 1 Processor -- start 2023-10-14T18:17:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.282+0000 7ff8f6c15640 1 -- start start 2023-10-14T18:17:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f015a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015b790 con 0x7ff8f0159e90 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015ab80 con 0x7ff8f0159ab0 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.283+0000 7ff8f6c15640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f015ae60 con 0x7ff8f006c140 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38834/0 (socket says 172.21.15.29:38834) 2023-10-14T18:17:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 -- 172.21.15.29:0/811675885 learned_addr learned my addr 172.21.15.29:0/811675885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:06.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 msgr2=0x7ff8f015a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f015a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 msgr2=0x7ff8f00b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 -- 172.21.15.29:0/811675885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8e4009400 con 0x7ff8f0159e90 2023-10-14T18:17:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.284+0000 7ff8f51b2640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800efc0 tx=0x7ff8e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.285+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e8009ba0 con 0x7ff8f0159e90 2023-10-14T18:17:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.285+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8e800f040 con 0x7ff8f0159e90 2023-10-14T18:17:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.285+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e80146c0 con 0x7ff8f0159e90 2023-10-14T18:17:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.285+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f015b140 con 0x7ff8f0159e90 2023-10-14T18:17:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.285+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f015a870 con 0x7ff8f0159e90 2023-10-14T18:17:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.286+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f004e160 con 0x7ff8f0159e90 2023-10-14T18:17:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.288+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7ff8e8014820 con 0x7ff8f0159e90 2023-10-14T18:17:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.289+0000 7ff8ed7fa640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 0x7ff8d406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.289+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6299+0+0 (secure 0 0 0) 0x7ff8e808a6a0 con 0x7ff8f0159e90 2023-10-14T18:17:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.289+0000 7ff8f49b1640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 msgr2=0x7ff8d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.289+0000 7ff8f49b1640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 0x7ff8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:06.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.294+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff8e8055240 con 0x7ff8f0159e90 2023-10-14T18:17:06.364 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.363+0000 7fa900a11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:17:06.445 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.445+0000 7f954542c040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:17:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.490+0000 7ff8f49b1640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 msgr2=0x7ff8d406d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:06.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.490+0000 7ff8f49b1640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 0x7ff8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:06.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.504+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7ff8f00cda60 con 0x7ff8f0159e90 2023-10-14T18:17:06.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.505+0000 7ff8ed7fa640 1 -- 172.21.15.29:0/811675885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v215) v1 ==== 100+0+11 (secure 0 0 0) 0x7ff8e8058860 con 0x7ff8f0159e90 2023-10-14T18:17:06.506 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:17:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 msgr2=0x7ff8d406d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:06.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 0x7ff8d406d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 msgr2=0x7ff8f01f9e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 secure :-1 s=READY pgs=2464 cs=0 l=1 rev1=1 crypto rx=0x7ff8e800efc0 tx=0x7ff8e800c490 comp rx=0 tx=0).stop 2023-10-14T18:17:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 shutdown_connections 2023-10-14T18:17:06.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7ff8d406af90 0x7ff8d406d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff8f0159e90 0x7ff8f01f9e20 unknown :-1 s=CLOSED pgs=2464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff8f0159ab0 0x7ff8f00b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 --2- 172.21.15.29:0/811675885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff8f006c140 0x7ff8f015a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 >> 172.21.15.29:0/811675885 conn(0x7ff8f00ce9b0 msgr2=0x7ff8f0070520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:06.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.508+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 shutdown_connections 2023-10-14T18:17:06.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.509+0000 7ff8f6c15640 1 -- 172.21.15.29:0/811675885 wait complete. 2023-10-14T18:17:06.526 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T18:17:06.547 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.546+0000 7fa900a11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:17:06.554 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.554+0000 7f954542c040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:17:06.605 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.604+0000 7f954542c040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:17:06.662 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.661+0000 7fa900a11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:17:06.712 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.712+0000 7f954542c040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:06.715 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.714+0000 7fa900a11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:17:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.738+0000 7fb1b316a640 1 Processor -- start 2023-10-14T18:17:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.739+0000 7fb1b316a640 1 -- start start 2023-10-14T18:17:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b316a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb1ac0ca830 0x7fb1ac070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b316a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 0x7fb1ac06bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b316a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1ac0ca3a0 con 0x7fb1ac1573e0 2023-10-14T18:17:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b316a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ac070c10 con 0x7fb1ac0ca830 2023-10-14T18:17:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b316a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1ac0c9d70 con 0x7fb1ac070780 2023-10-14T18:17:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b2168640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb1ac1573e0 0x7fb1ac159e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36310/0 (socket says 172.21.15.29:36310) 2023-10-14T18:17:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b2168640 1 -- 172.21.15.29:0/4084275986 learned_addr learned my addr 172.21.15.29:0/4084275986 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.740+0000 7fb1b1967640 1 --2- 172.21.15.29:0/4084275986 >> v2:172.21.15.29:3300/0 conn(0x7fb1ac0ca830 0x7fb1ac070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b2969640 1 --2- 172.21.15.29:0/4084275986 >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 0x7fb1ac06bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1166640 1 -- 172.21.15.29:0/4084275986 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1039641588 0 0) 0x7fb1ac0ca3a0 con 0x7fb1ac1573e0 2023-10-14T18:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1166640 1 -- 172.21.15.29:0/4084275986 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198005730 con 0x7fb1ac1573e0 2023-10-14T18:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1967640 1 -- 172.21.15.29:0/4084275986 >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 msgr2=0x7fb1ac06bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:06.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1967640 1 --2- 172.21.15.29:0/4084275986 >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 0x7fb1ac06bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1967640 1 -- 172.21.15.29:0/4084275986 >> v1:172.21.15.29:6789/0 conn(0x7fb1ac1573e0 legacy=0x7fb1ac159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b1967640 1 -- 172.21.15.29:0/4084275986 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1ac1577c0 con 0x7fb1ac0ca830 2023-10-14T18:17:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.741+0000 7fb1b2969640 1 --2- 172.21.15.29:0/4084275986 >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 0x7fb1ac06bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.742+0000 7fb1b1967640 1 --2- 172.21.15.29:0/4084275986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac070240 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0ca310 tx=0x7fb19c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5dd134c82c7d424a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:06.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.742+0000 7fb1b1166640 1 -- 172.21.15.29:0/4084275986 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c031830 con 0x7fb1ac0ca830 2023-10-14T18:17:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b1166640 1 -- 172.21.15.29:0/4084275986 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c031990 con 0x7fb1ac0ca830 2023-10-14T18:17:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b1166640 1 -- 172.21.15.29:0/4084275986 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c031d20 con 0x7fb1ac0ca830 2023-10-14T18:17:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 -- 172.21.15.29:0/4084275986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 msgr2=0x7fb1ac070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 --2- 172.21.15.29:0/4084275986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac070240 secure :-1 s=READY pgs=2466 cs=0 l=1 rev1=1 crypto rx=0x7fb1ac0ca310 tx=0x7fb19c02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 -- 172.21.15.29:0/4084275986 shutdown_connections 2023-10-14T18:17:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 --2- 172.21.15.29:0/4084275986 >> v2:172.21.15.186:3300/0 conn(0x7fb1ac070780 0x7fb1ac06bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 --2- 172.21.15.29:0/4084275986 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac070240 unknown :-1 s=CLOSED pgs=2466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 -- 172.21.15.29:0/4084275986 >> 172.21.15.29:0/4084275986 conn(0x7fb1ac0cc230 msgr2=0x7fb1ac0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 -- 172.21.15.29:0/4084275986 shutdown_connections 2023-10-14T18:17:06.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.743+0000 7fb1b316a640 1 -- 172.21.15.29:0/4084275986 wait complete. 2023-10-14T18:17:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.745+0000 7fb1b316a640 1 Processor -- start 2023-10-14T18:17:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.745+0000 7fb1b316a640 1 -- start start 2023-10-14T18:17:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ac070780 0x7fb1ac0c7a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ac1573e0 0x7fb1ac0afce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0c7610 con 0x7fb1ac0ca830 2023-10-14T18:17:06.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0c6990 con 0x7fb1ac1573e0 2023-10-14T18:17:06.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b316a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1ac0c6c70 con 0x7fb1ac070780 2023-10-14T18:17:06.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b1967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b1967640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38842/0 (socket says 172.21.15.29:38842) 2023-10-14T18:17:06.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.746+0000 7fb1b1967640 1 -- 172.21.15.29:0/2770521147 learned_addr learned my addr 172.21.15.29:0/2770521147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:06.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b2168640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ac070780 0x7fb1ac0c7a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b2969640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ac1573e0 0x7fb1ac0afce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:06.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ac070780 msgr2=0x7fb1ac0c7a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ac070780 0x7fb1ac0c7a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ac1573e0 msgr2=0x7fb1ac0afce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ac1573e0 0x7fb1ac0afce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 -- 172.21.15.29:0/2770521147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c009400 con 0x7fb1ac0ca830 2023-10-14T18:17:06.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.747+0000 7fb1b1967640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009890 tx=0x7fb19c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.748+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c03a6f0 con 0x7fb1ac0ca830 2023-10-14T18:17:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.748+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c03a870 con 0x7fb1ac0ca830 2023-10-14T18:17:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.748+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb19c03ac20 con 0x7fb1ac0ca830 2023-10-14T18:17:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.748+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1ac0c6f50 con 0x7fb1ac0ca830 2023-10-14T18:17:06.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.748+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1ac0b0220 con 0x7fb1ac0ca830 2023-10-14T18:17:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.749+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1ac0c7750 con 0x7fb1ac0ca830 2023-10-14T18:17:06.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.750+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fb19c04c3b0 con 0x7fb1ac0ca830 2023-10-14T18:17:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.751+0000 7fb1a2ffd640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 0x7fb19006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.751+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fb19c0ab8c0 con 0x7fb1ac0ca830 2023-10-14T18:17:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.751+0000 7fb1b2168640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 msgr2=0x7fb19006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.751+0000 7fb1b2168640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 0x7fb19006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:06.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.755+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb1ac0c7750 con 0x7fb1ac0ca830 2023-10-14T18:17:06.821 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.821+0000 7f954542c040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:17:06.832 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.831+0000 7fa900a11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:06.946 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:06.945+0000 7fa900a11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:17:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.952+0000 7fb1b2168640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 msgr2=0x7fb19006d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:06.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.952+0000 7fb1b2168640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 0x7fb19006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:06.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.961+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fb1ac0c7750 con 0x7fb1ac0ca830 2023-10-14T18:17:06.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.962+0000 7fb1a2ffd640 1 -- 172.21.15.29:0/2770521147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v215) v1 ==== 80+0+470 (secure 0 0 0) 0x7fb1ac0c7750 con 0x7fb1ac0ca830 2023-10-14T18:17:06.963 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4216364220 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.963 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/2339165088 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1790177658 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3670605654 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2768480413 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4181160999 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/2339165088 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.964 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1466849565 2023-10-15T18:15:00.600793+0000 2023-10-14T18:17:06.965 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T18:17:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 msgr2=0x7fb19006d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 0x7fb19006d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 msgr2=0x7fb1ac0c7f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:06.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 secure :-1 s=READY pgs=2467 cs=0 l=1 rev1=1 crypto rx=0x7fb19c009890 tx=0x7fb19c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 shutdown_connections 2023-10-14T18:17:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fb19006af90 0x7fb19006d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb1ac1573e0 0x7fb1ac0afce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb1ac0ca830 0x7fb1ac0c7f70 unknown :-1 s=CLOSED pgs=2467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 --2- 172.21.15.29:0/2770521147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb1ac070780 0x7fb1ac0c7a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 >> 172.21.15.29:0/2770521147 conn(0x7fb1ac0cc230 msgr2=0x7fb1ac158ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 shutdown_connections 2023-10-14T18:17:06.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:06.965+0000 7fb1b316a640 1 -- 172.21.15.29:0/2770521147 wait complete. 2023-10-14T18:17:06.977 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:06.977+0000 7f954542c040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:06.984 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T18:17:06.984 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:06.985 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T18:17:07.027 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:07.027+0000 7f954542c040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:07.114 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:07.113+0000 7fa900a11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:07.127 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:17:07.128 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:07.128 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T18:17:07.156 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:07.156+0000 7f954542c040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:17:07.171 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:07.170+0000 7fa900a11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:07.178 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:17:07.179 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:07.231 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:17:07.232 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:07.232 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:07.287 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:17:07.287 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:17:07.288 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:07.299 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:07.298+0000 7fa900a11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:17:07.341 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:17:07.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:07.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.552+0000 7f224b59e640 1 Processor -- start 2023-10-14T18:17:07.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.552+0000 7f224b59e640 1 -- start start 2023-10-14T18:17:07.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.553+0000 7f224b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f224c167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.553+0000 7f224b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0d15a0 con 0x7f224c15cbe0 2023-10-14T18:17:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.553+0000 7f224b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224c0d1880 con 0x7f224c15cfc0 2023-10-14T18:17:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.553+0000 7f224b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0d1b60 con 0x7f224c0b4c80 2023-10-14T18:17:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f224a59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f224c15cbe0 0x7f224c0d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36322/0 (socket says 172.21.15.29:36322) 2023-10-14T18:17:07.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f224a59c640 1 -- 172.21.15.29:0/3223153580 learned_addr learned my addr 172.21.15.29:0/3223153580 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:07.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f224ad9d640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f224c167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1758565993 0 0) 0x7f224c0d15a0 con 0x7f224c15cbe0 2023-10-14T18:17:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2238003520 con 0x7f224c15cbe0 2023-10-14T18:17:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418081893 0 0) 0x7f2238003520 con 0x7f224c15cbe0 2023-10-14T18:17:07.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 msgr2=0x7f224c167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f224c167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 >> v1:172.21.15.186:6789/0 conn(0x7f224c15cfc0 legacy=0x7f224c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c0d1e40 con 0x7f224c15cbe0 2023-10-14T18:17:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f223c001670 con 0x7f224c15cbe0 2023-10-14T18:17:07.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.554+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 >> v1:172.21.15.29:6789/0 conn(0x7f224c15cbe0 legacy=0x7f224c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22380039a0 0x7f2238003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f22380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2238004d00 0x7f2238005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0d1b60 con 0x7f22380039a0 2023-10-14T18:17:07.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223800baf0 con 0x7f2238004d00 2023-10-14T18:17:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f223800bcb0 con 0x7f224c0b4c80 2023-10-14T18:17:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22380039a0 0x7f2238003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2243fff640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f22380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 -- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 msgr2=0x7f22380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:07.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f22380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 -- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2238004d00 msgr2=0x7f2238005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2238004d00 0x7f2238005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f224a59c640 1 -- 172.21.15.29:0/3223153580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2238003520 con 0x7f22380039a0 2023-10-14T18:17:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.555+0000 7f2243fff640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f224c0b4c80 0x7f22380047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:07.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.556+0000 7f224a59c640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22380039a0 0x7f2238003dc0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f223c001eb0 tx=0x7f22380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c63d8d5acd23c93 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.557+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c005ad0 con 0x7f22380039a0 2023-10-14T18:17:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.557+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f223c005c30 con 0x7f22380039a0 2023-10-14T18:17:07.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.557+0000 7f2249d9b640 1 -- 172.21.15.29:0/3223153580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f223c02c7c0 con 0x7f22380039a0 2023-10-14T18:17:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 -- 172.21.15.29:0/3223153580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22380039a0 msgr2=0x7f2238003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 --2- 172.21.15.29:0/3223153580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22380039a0 0x7f2238003dc0 secure :-1 s=READY pgs=2469 cs=0 l=1 rev1=1 crypto rx=0x7f223c001eb0 tx=0x7f22380057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224ad9d640 1 -- 172.21.15.29:0/3223153580 reap_dead start 2023-10-14T18:17:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 -- 172.21.15.29:0/3223153580 shutdown_connections 2023-10-14T18:17:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 -- 172.21.15.29:0/3223153580 >> 172.21.15.29:0/3223153580 conn(0x7f224c0cf570 msgr2=0x7f224c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 -- 172.21.15.29:0/3223153580 shutdown_connections 2023-10-14T18:17:07.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.558+0000 7f224b59e640 1 -- 172.21.15.29:0/3223153580 wait complete. 2023-10-14T18:17:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.560+0000 7f224b59e640 1 Processor -- start 2023-10-14T18:17:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.560+0000 7f224b59e640 1 -- start start 2023-10-14T18:17:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22380039a0 0x7f224c0cc830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 0x7f224c0c8d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c145e30 con 0x7f224c0c9bc0 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f224c0ca580 con 0x7f224c144e40 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f224c15e650 con 0x7f22380039a0 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38864/0 (socket says 172.21.15.29:38864) 2023-10-14T18:17:07.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 -- 172.21.15.29:0/2090449598 learned_addr learned my addr 172.21.15.29:0/2090449598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224a59c640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22380039a0 0x7f224c0cc830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f2243fff640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 0x7f224c0c8d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22380039a0 msgr2=0x7f224c0cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22380039a0 0x7f224c0cc830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 msgr2=0x7f224c0c8d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 0x7f224c0c8d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f224ad9d640 1 -- 172.21.15.29:0/2090449598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c1452c0 con 0x7f224c0c9bc0 2023-10-14T18:17:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.561+0000 7f2243fff640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 0x7f224c0c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:07.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f224ad9d640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f224c0d3b20 tx=0x7f224400eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2244010040 con 0x7f224c0c9bc0 2023-10-14T18:17:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f224400bab0 con 0x7f224c0c9bc0 2023-10-14T18:17:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f224400be60 con 0x7f224c0c9bc0 2023-10-14T18:17:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f224c145590 con 0x7f224c0c9bc0 2023-10-14T18:17:07.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.562+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f224c145b50 con 0x7f224c0c9bc0 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.563+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f224c15e420 con 0x7f224c0c9bc0 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.564+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f224401d400 con 0x7f224c0c9bc0 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.565+0000 7f2242ffd640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 0x7f222c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.565+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f2244089ff0 con 0x7f224c0c9bc0 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.565+0000 7f224a59c640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 msgr2=0x7f222c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:07.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.565+0000 7f224a59c640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 0x7f222c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:07.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.570+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f224c15e420 con 0x7f224c0c9bc0 2023-10-14T18:17:07.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.766+0000 7f224a59c640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 msgr2=0x7f222c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:07.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.766+0000 7f224a59c640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 0x7f222c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:07.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.804+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f224c0d1240 con 0x7f224c0c9bc0 2023-10-14T18:17:07.805 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.805+0000 7f2242ffd640 1 -- 172.21.15.29:0/2090449598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f224c0d1240 con 0x7f224c0c9bc0 2023-10-14T18:17:07.806 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:07.806 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:17:07.807 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:17:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 msgr2=0x7f222c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 0x7f222c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 msgr2=0x7f224c0ca040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 secure :-1 s=READY pgs=2470 cs=0 l=1 rev1=1 crypto rx=0x7f224c0d3b20 tx=0x7f224400eea0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 shutdown_connections 2023-10-14T18:17:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f222c06af90 0x7f222c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f224c0c9bc0 0x7f224c0ca040 unknown :-1 s=CLOSED pgs=2470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f224c144e40 0x7f224c0c8d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 --2- 172.21.15.29:0/2090449598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22380039a0 0x7f224c0cc830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 >> 172.21.15.29:0/2090449598 conn(0x7f224c0cf570 msgr2=0x7f224c0b5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 shutdown_connections 2023-10-14T18:17:07.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:07.808+0000 7f224b59e640 1 -- 172.21.15.29:0/2090449598 wait complete. 2023-10-14T18:17:07.829 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:07.983 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:07.983+0000 7f954542c040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:17:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.021+0000 7f5636b33640 1 Processor -- start 2023-10-14T18:17:08.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.021+0000 7f5636b33640 1 -- start start 2023-10-14T18:17:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.022+0000 7f5636b33640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f563006c3f0 0x7f563006c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f5636b33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f56300b1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f5636b33640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56300cd630 con 0x7f563015e0c0 2023-10-14T18:17:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f5636b33640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56300ce1b0 con 0x7f563006c3f0 2023-10-14T18:17:08.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f5636b33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f563006cd10 con 0x7f563015e540 2023-10-14T18:17:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f562ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f563015e0c0 0x7f56300b4730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36330/0 (socket says 172.21.15.29:36330) 2023-10-14T18:17:08.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f562ffff640 1 -- 172.21.15.29:0/498732038 learned_addr learned my addr 172.21.15.29:0/498732038 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f56350d0640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f56300b1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f56348cf640 1 --2- 172.21.15.29:0/498732038 >> v2:172.21.15.186:3300/0 conn(0x7f563006c3f0 0x7f563006c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.023+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515765620 0 0) 0x7f56300cd630 con 0x7f563015e0c0 2023-10-14T18:17:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f561c003520 con 0x7f563015e0c0 2023-10-14T18:17:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1852052748 0 0) 0x7f561c003520 con 0x7f563015e0c0 2023-10-14T18:17:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 msgr2=0x7f56300b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f56300b1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 >> v2:172.21.15.186:3300/0 conn(0x7f563006c3f0 msgr2=0x7f563006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 --2- 172.21.15.29:0/498732038 >> v2:172.21.15.186:3300/0 conn(0x7f563006c3f0 0x7f563006c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56300d2170 con 0x7f563015e0c0 2023-10-14T18:17:08.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5618001670 con 0x7f563015e0c0 2023-10-14T18:17:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 >> v1:172.21.15.29:6789/0 conn(0x7f563015e0c0 legacy=0x7f56300b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f561c003990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f561c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f561c004d00 0x7f561c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56300ce1b0 con 0x7f561c003f10 2023-10-14T18:17:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f563006cd10 con 0x7f561c004d00 2023-10-14T18:17:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f561c00b980 con 0x7f563015e540 2023-10-14T18:17:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.024+0000 7f562ffff640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f561c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f56348cf640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f561c003990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f56350d0640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f561c004d00 0x7f561c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f562ffff640 1 -- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 msgr2=0x7f561c003990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f562ffff640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f563015e540 0x7f561c003990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f562ffff640 1 -- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f561c004d00 msgr2=0x7f561c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f562ffff640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f561c004d00 0x7f561c005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.025+0000 7f562ffff640 1 -- 172.21.15.29:0/498732038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f561c003520 con 0x7f561c003f10 2023-10-14T18:17:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.026+0000 7f562ffff640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f561c0047c0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f561800c2c0 tx=0x7f561c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffb332701c9ffe71 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.026+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5618005ad0 con 0x7f561c003f10 2023-10-14T18:17:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.026+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5618005c30 con 0x7f561c003f10 2023-10-14T18:17:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.026+0000 7f562f7fe640 1 -- 172.21.15.29:0/498732038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561802c8b0 con 0x7f561c003f10 2023-10-14T18:17:08.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 -- 172.21.15.29:0/498732038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 msgr2=0x7f561c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 --2- 172.21.15.29:0/498732038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f561c0047c0 secure :-1 s=READY pgs=2472 cs=0 l=1 rev1=1 crypto rx=0x7f561800c2c0 tx=0x7f561c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f56350d0640 1 -- 172.21.15.29:0/498732038 reap_dead start 2023-10-14T18:17:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 -- 172.21.15.29:0/498732038 shutdown_connections 2023-10-14T18:17:08.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 -- 172.21.15.29:0/498732038 >> 172.21.15.29:0/498732038 conn(0x7f56300d2cb0 msgr2=0x7f56300d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 -- 172.21.15.29:0/498732038 shutdown_connections 2023-10-14T18:17:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.027+0000 7f5636b33640 1 -- 172.21.15.29:0/498732038 wait complete. 2023-10-14T18:17:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.029+0000 7f5636b33640 1 Processor -- start 2023-10-14T18:17:08.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.029+0000 7f5636b33640 1 -- start start 2023-10-14T18:17:08.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56301f2810 0x7f56300cab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56301f2c90 0x7f563015bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5630159200 con 0x7f561c003f10 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56301594e0 con 0x7f56301f2c90 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f5636b33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56301597c0 con 0x7f56301f2810 2023-10-14T18:17:08.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f56348cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f56348cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38888/0 (socket says 172.21.15.29:38888) 2023-10-14T18:17:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f56348cf640 1 -- 172.21.15.29:0/991123093 learned_addr learned my addr 172.21.15.29:0/991123093 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f56350d0640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56301f2c90 0x7f563015bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.030+0000 7f562ffff640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56301f2810 0x7f56300cab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56301f2810 msgr2=0x7f56300cab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56301f2810 0x7f56300cab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56301f2c90 msgr2=0x7f563015bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56301f2c90 0x7f563015bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 -- 172.21.15.29:0/991123093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5630159aa0 con 0x7f561c003f10 2023-10-14T18:17:08.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f56348cf640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5620007f30 tx=0x7f5620011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f562000e030 con 0x7f561c003f10 2023-10-14T18:17:08.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5620016470 con 0x7f561c003f10 2023-10-14T18:17:08.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5620016820 con 0x7f561c003f10 2023-10-14T18:17:08.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5630159d80 con 0x7f561c003f10 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.031+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56300b41f0 con 0x7f561c003f10 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.033+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56300ce320 con 0x7f561c003f10 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.034+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7f5620016980 con 0x7f561c003f10 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.034+0000 7f562d7fa640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 0x7f561006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.035+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6299+0+0 (secure 0 0 0) 0x7f562000f070 con 0x7f561c003f10 2023-10-14T18:17:08.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.035+0000 7f562ffff640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 msgr2=0x7f561006d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:08.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.035+0000 7f562ffff640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 0x7f561006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:08.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.043+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5620055740 con 0x7f561c003f10 2023-10-14T18:17:08.179 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.177+0000 7fa900a11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:17:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.236+0000 7f562ffff640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 msgr2=0x7f561006d480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:08.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.236+0000 7f562ffff640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 0x7f561006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:08.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.242+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f563004e160 con 0x7f561c003f10 2023-10-14T18:17:08.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.244+0000 7f562d7fa640 1 -- 172.21.15.29:0/991123093 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v215) v1 ==== 74+0+17431 (secure 0 0 0) 0x7f5620058d60 con 0x7f561c003f10 2023-10-14T18:17:08.246 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:08.246 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":215,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:06.267922+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":215,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.17","mappings":[{"from":0,"to":2}]},{"pgid":"25.2d","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/4216364220":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6832/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1790177658":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/3670605654":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/2768480413":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/4181160999":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6833/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1466849565":"2023-10-15T18:15:00.600793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[{"pool":25,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.247+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 msgr2=0x7f561006d480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.247+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 0x7f561006d480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.247+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 msgr2=0x7f56300b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.247+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 secure :-1 s=READY pgs=2473 cs=0 l=1 rev1=1 crypto rx=0x7f5620007f30 tx=0x7f5620011660 comp rx=0 tx=0).stop 2023-10-14T18:17:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 shutdown_connections 2023-10-14T18:17:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7f561006afc0 0x7f561006d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56301f2c90 0x7f563015bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56301f2810 0x7f56300cab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 --2- 172.21.15.29:0/991123093 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f561c003f10 0x7f56300b3cb0 unknown :-1 s=CLOSED pgs=2473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 >> 172.21.15.29:0/991123093 conn(0x7f56300d2cb0 msgr2=0x7f56300d0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 shutdown_connections 2023-10-14T18:17:08.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.248+0000 7f5636b33640 1 -- 172.21.15.29:0/991123093 wait complete. 2023-10-14T18:17:08.267 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 24, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:14:43.916335+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '209', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 25, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:14:45.028887+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '215', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 215, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.25, 'score_stable': 1.25, 'optimal_score': 1, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:17:08.269 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:17:08.283 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:08.283+0000 7f954542c040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:17:08.342 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:08.342+0000 7f954542c040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:17:08.457 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:08.457+0000 7f954542c040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:17:08.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.490+0000 7fd842461640 1 Processor -- start 2023-10-14T18:17:08.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.490+0000 7fd842461640 1 -- start start 2023-10-14T18:17:08.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.491+0000 7fd842461640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.491+0000 7fd842461640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd83c0cf5f0 0x7fd83c06c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.491+0000 7fd842461640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd83c161740 con 0x7fd83c070c10 2023-10-14T18:17:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.491+0000 7fd842461640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd83c06c680 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.491+0000 7fd842461640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c06b0f0 con 0x7fd83c070830 2023-10-14T18:17:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd83b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd83c0cf5f0 0x7fd83c06c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd8409fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd83c070c10 0x7fd83c0cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36336/0 (socket says 172.21.15.29:36336) 2023-10-14T18:17:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd8409fe640 1 -- 172.21.15.29:0/2496140742 learned_addr learned my addr 172.21.15.29:0/2496140742 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:08.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd83bfff640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd83affd640 1 -- 172.21.15.29:0/2496140742 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4284450603 0 0) 0x7fd83c161740 con 0x7fd83c070c10 2023-10-14T18:17:08.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd83affd640 1 -- 172.21.15.29:0/2496140742 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd828003520 con 0x7fd83c070c10 2023-10-14T18:17:08.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.492+0000 7fd83b7fe640 1 -- 172.21.15.29:0/2496140742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 msgr2=0x7fd83c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.493+0000 7fd83b7fe640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.493+0000 7fd83b7fe640 1 -- 172.21.15.29:0/2496140742 >> v1:172.21.15.29:6789/0 conn(0x7fd83c070c10 legacy=0x7fd83c0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.493+0000 7fd83b7fe640 1 -- 172.21.15.29:0/2496140742 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd83c1619b0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.493+0000 7fd83bfff640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.494+0000 7fd83b7fe640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c06c140 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097a0 tx=0x7fd82402b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f7c5d203d9cfc08 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.494+0000 7fd83affd640 1 -- 172.21.15.29:0/2496140742 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8240316f0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.494+0000 7fd83affd640 1 -- 172.21.15.29:0/2496140742 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd824031850 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.494+0000 7fd83affd640 1 -- 172.21.15.29:0/2496140742 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd824031be0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 -- 172.21.15.29:0/2496140742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 msgr2=0x7fd83c06c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c06c140 secure :-1 s=READY pgs=2475 cs=0 l=1 rev1=1 crypto rx=0x7fd8240097a0 tx=0x7fd82402b790 comp rx=0 tx=0).stop 2023-10-14T18:17:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 -- 172.21.15.29:0/2496140742 shutdown_connections 2023-10-14T18:17:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c06c140 unknown :-1 s=CLOSED pgs=2475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 --2- 172.21.15.29:0/2496140742 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c06d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 -- 172.21.15.29:0/2496140742 >> 172.21.15.29:0/2496140742 conn(0x7fd83c0cfac0 msgr2=0x7fd83c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:08.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 -- 172.21.15.29:0/2496140742 shutdown_connections 2023-10-14T18:17:08.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.495+0000 7fd842461640 1 -- 172.21.15.29:0/2496140742 wait complete. 2023-10-14T18:17:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.498+0000 7fd842461640 1 Processor -- start 2023-10-14T18:17:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.498+0000 7fd842461640 1 -- start start 2023-10-14T18:17:08.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c159180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83c070c10 0x7fd83c1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c0afe30 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c1f61e0 con 0x7fd83c070c10 2023-10-14T18:17:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd842461640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd83c1f36c0 con 0x7fd83c070830 2023-10-14T18:17:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38904/0 (socket says 172.21.15.29:38904) 2023-10-14T18:17:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 -- 172.21.15.29:0/52795460 learned_addr learned my addr 172.21.15.29:0/52795460 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd83bfff640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c159180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd83b7fe640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83c070c10 0x7fd83c1f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 -- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 msgr2=0x7fd83c159180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c159180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 -- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83c070c10 msgr2=0x7fd83c1f3180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83c070c10 0x7fd83c1f3180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.500+0000 7fd8409fe640 1 -- 172.21.15.29:0/52795460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd824009400 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd8409fe640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c460 tx=0x7fd83400c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd834005ce0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd834005e40 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd83c0b0080 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8340072b0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.501+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd83c1f5e70 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.502+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd83c0b05a0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.503+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 132) v1 ==== 89343+0+0 (secure 0 0 0) 0x7fd834007410 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.505+0000 7fd838ff9640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 0x7fd81c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.505+0000 7fd83bfff640 1 -- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 msgr2=0x7fd81c06d450 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/966124587 2023-10-14T18:17:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.505+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(215..215 src has 1..215) v4 ==== 6299+0+0 (secure 0 0 0) 0x7fd834088f90 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.505+0000 7fd83bfff640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 0x7fd81c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:08.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.509+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd83c0b05a0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.525+0000 7fa900a11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:17:08.588 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.587+0000 7fa900a11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:17:08.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.698+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd83c1f5cd0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.699+0000 7fd838ff9640 1 -- 172.21.15.29:0/52795460 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v215) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd83c1f5cd0 con 0x7fd83c0cf5f0 2023-10-14T18:17:08.700 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:17:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 msgr2=0x7fd81c06d450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 0x7fd81c06d450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 msgr2=0x7fd83c1f6670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 secure :-1 s=READY pgs=2476 cs=0 l=1 rev1=1 crypto rx=0x7fd83400c460 tx=0x7fd83400c920 comp rx=0 tx=0).stop 2023-10-14T18:17:08.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 shutdown_connections 2023-10-14T18:17:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:6832/966124587,v1:172.21.15.29:6833/966124587] conn(0x7fd81c06af90 0x7fd81c06d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd83c0cf5f0 0x7fd83c1f6670 unknown :-1 s=CLOSED pgs=2476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd83c070c10 0x7fd83c1f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 --2- 172.21.15.29:0/52795460 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd83c070830 0x7fd83c159180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 >> 172.21.15.29:0/52795460 conn(0x7fd83c0cfac0 msgr2=0x7fd83c072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 shutdown_connections 2023-10-14T18:17:08.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.702+0000 7fd842461640 1 -- 172.21.15.29:0/52795460 wait complete. 2023-10-14T18:17:08.712 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.711+0000 7fa900a11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:17:08.725 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:17:08.801 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.800+0000 7fa8859f7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.801 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.800+0000 7fa8859f7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.802 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.800+0000 7fa8859f7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.802 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.800+0000 7fa8859f7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.802 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.800+0000 7fa8859f7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.801+0000 7fa8829f1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.801+0000 7fa8829f1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.801+0000 7fa8829f1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.803 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.801+0000 7fa8829f1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.804 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:08.801+0000 7fa8829f1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:08.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.921+0000 7f913039e640 1 Processor -- start 2023-10-14T18:17:08.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.921+0000 7f913039e640 1 -- start start 2023-10-14T18:17:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f913039e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f912815d330 0x7f91281618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f913039e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f912806c7d0 0x7f9128065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f913039e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91280d0890 con 0x7f912806cc80 2023-10-14T18:17:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f913039e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912806b1e0 con 0x7f912815d330 2023-10-14T18:17:08.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f913039e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91280d05b0 con 0x7f912806c7d0 2023-10-14T18:17:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e13a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f912815d330 0x7f91281618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e93b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f912806cc80 0x7f91280ce9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36338/0 (socket says 172.21.15.29:36338) 2023-10-14T18:17:08.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e93b640 1 -- 172.21.15.29:0/1202644322 learned_addr learned my addr 172.21.15.29:0/1202644322 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:08.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912d939640 1 --2- 172.21.15.29:0/1202644322 >> v2:172.21.15.186:3300/0 conn(0x7f912806c7d0 0x7f9128065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e13a640 1 -- 172.21.15.29:0/1202644322 >> v2:172.21.15.186:3300/0 conn(0x7f912806c7d0 msgr2=0x7f9128065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:08.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e13a640 1 --2- 172.21.15.29:0/1202644322 >> v2:172.21.15.186:3300/0 conn(0x7f912806c7d0 0x7f9128065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e13a640 1 -- 172.21.15.29:0/1202644322 >> v1:172.21.15.29:6789/0 conn(0x7f912806cc80 legacy=0x7f91280ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.923+0000 7f912e13a640 1 -- 172.21.15.29:0/1202644322 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9128160dd0 con 0x7f912815d330 2023-10-14T18:17:08.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.924+0000 7f912e13a640 1 --2- 172.21.15.29:0/1202644322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91281618b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f91280b6bc0 tx=0x7f911c02be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95cec7951de95547 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f912d138640 1 -- 172.21.15.29:0/1202644322 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c02cbe0 con 0x7f912815d330 2023-10-14T18:17:08.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f912d138640 1 -- 172.21.15.29:0/1202644322 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911c02cd40 con 0x7f912815d330 2023-10-14T18:17:08.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f912d138640 1 -- 172.21.15.29:0/1202644322 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911c034660 con 0x7f912815d330 2023-10-14T18:17:08.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 -- 172.21.15.29:0/1202644322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 msgr2=0x7f91281618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 --2- 172.21.15.29:0/1202644322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91281618b0 secure :-1 s=READY pgs=2489 cs=0 l=1 rev1=1 crypto rx=0x7f91280b6bc0 tx=0x7f911c02be30 comp rx=0 tx=0).stop 2023-10-14T18:17:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 -- 172.21.15.29:0/1202644322 shutdown_connections 2023-10-14T18:17:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 --2- 172.21.15.29:0/1202644322 >> v2:172.21.15.186:3300/0 conn(0x7f912806c7d0 0x7f9128065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 --2- 172.21.15.29:0/1202644322 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91281618b0 unknown :-1 s=CLOSED pgs=2489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.925+0000 7f913039e640 1 -- 172.21.15.29:0/1202644322 >> 172.21.15.29:0/1202644322 conn(0x7f91280d0ab0 msgr2=0x7f91280d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:08.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.926+0000 7f913039e640 1 -- 172.21.15.29:0/1202644322 shutdown_connections 2023-10-14T18:17:08.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.926+0000 7f913039e640 1 -- 172.21.15.29:0/1202644322 wait complete. 2023-10-14T18:17:08.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.927+0000 7f913039e640 1 Processor -- start 2023-10-14T18:17:08.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.927+0000 7f913039e640 1 -- start start 2023-10-14T18:17:08.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f912806c7d0 0x7f912815c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912806cc80 0x7f91280b1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280c81e0 con 0x7f912815d330 2023-10-14T18:17:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91280b0ac0 con 0x7f912806cc80 2023-10-14T18:17:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.928+0000 7f913039e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91280b07e0 con 0x7f912806c7d0 2023-10-14T18:17:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39002/0 (socket says 172.21.15.29:39002) 2023-10-14T18:17:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 -- 172.21.15.29:0/3983118453 learned_addr learned my addr 172.21.15.29:0/3983118453 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912d939640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912806cc80 0x7f91280b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 -- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f912806c7d0 msgr2=0x7f912815c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f912806c7d0 0x7f912815c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 -- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912806cc80 msgr2=0x7f91280b1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912806cc80 0x7f91280b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 -- 172.21.15.29:0/3983118453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f911c009a70 con 0x7f912815d330 2023-10-14T18:17:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f912e93b640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f912000c370 tx=0x7f912000c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.929+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120005ce0 con 0x7f912815d330 2023-10-14T18:17:08.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.930+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9120005e40 con 0x7f912815d330 2023-10-14T18:17:08.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.930+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91280c7b90 con 0x7f912815d330 2023-10-14T18:17:08.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.930+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91200072d0 con 0x7f912815d330 2023-10-14T18:17:08.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.930+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91280b04b0 con 0x7f912815d330 2023-10-14T18:17:08.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.931+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912804e160 con 0x7f912815d330 2023-10-14T18:17:08.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.932+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 133) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f9120007430 con 0x7f912815d330 2023-10-14T18:17:08.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.933+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f91200888c0 con 0x7f912815d330 2023-10-14T18:17:08.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:08.937+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9120053a00 con 0x7f912815d330 2023-10-14T18:17:09.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.126+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f912806f3a0 con 0x7f912815d330 2023-10-14T18:17:09.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.127+0000 7f9126ffd640 1 -- 172.21.15.29:0/3983118453 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v216) v1 ==== 104+0+11 (secure 0 0 0) 0x7f9120057020 con 0x7f912815d330 2023-10-14T18:17:09.128 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:17:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.129+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 msgr2=0x7f91280c9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.129+0000 7f913039e640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 secure :-1 s=READY pgs=2490 cs=0 l=1 rev1=1 crypto rx=0x7f912000c370 tx=0x7f912000c830 comp rx=0 tx=0).stop 2023-10-14T18:17:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 shutdown_connections 2023-10-14T18:17:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912815d330 0x7f91280c9990 unknown :-1 s=CLOSED pgs=2490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912806cc80 0x7f91280b1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 --2- 172.21.15.29:0/3983118453 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f912806c7d0 0x7f912815c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 >> 172.21.15.29:0/3983118453 conn(0x7f91280d0ab0 msgr2=0x7f91280cccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 shutdown_connections 2023-10-14T18:17:09.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.130+0000 7f913039e640 1 -- 172.21.15.29:0/3983118453 wait complete. 2023-10-14T18:17:09.147 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:17:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.371+0000 7fa2880d0640 1 Processor -- start 2023-10-14T18:17:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.371+0000 7fa2880d0640 1 -- start start 2023-10-14T18:17:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa2880d0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa2800ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa2880d0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2800cb030 con 0x7fa28015ad70 2023-10-14T18:17:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa2880d0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2800c91c0 con 0x7fa28006a4b0 2023-10-14T18:17:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa2880d0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800c94a0 con 0x7fa2800caac0 2023-10-14T18:17:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa285e6c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa28015ad70 0x7fa280069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36344/0 (socket says 172.21.15.29:36344) 2023-10-14T18:17:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa285e6c640 1 -- 172.21.15.29:0/3404529940 learned_addr learned my addr 172.21.15.29:0/3404529940 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa28666d640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa2800ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.373+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283698547 0 0) 0x7fa2800cb030 con 0x7fa28015ad70 2023-10-14T18:17:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa26c003520 con 0x7fa28015ad70 2023-10-14T18:17:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 993407925 0 0) 0x7fa26c003520 con 0x7fa28015ad70 2023-10-14T18:17:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 msgr2=0x7fa2800ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa2800ad4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 >> v1:172.21.15.186:6789/0 conn(0x7fa28006a4b0 legacy=0x7fa2800ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2800c9780 con 0x7fa28015ad70 2023-10-14T18:17:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa27c001670 con 0x7fa28015ad70 2023-10-14T18:17:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 >> v1:172.21.15.29:6789/0 conn(0x7fa28015ad70 legacy=0x7fa280069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa26c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa26c003f20 0x7fa26c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c004d00 0x7fa26c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2800c94a0 con 0x7fa26c003f20 2023-10-14T18:17:09.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c00ba60 con 0x7fa26c004d00 2023-10-14T18:17:09.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa26c00bc20 con 0x7fa2800caac0 2023-10-14T18:17:09.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.374+0000 7fa28566b640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa26c003f20 0x7fa26c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa285e6c640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa26c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28666d640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c004d00 0x7fa26c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28566b640 1 -- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 msgr2=0x7fa26c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:09.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28566b640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa2800caac0 0x7fa26c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28566b640 1 -- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c004d00 msgr2=0x7fa26c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:09.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28566b640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c004d00 0x7fa26c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.375+0000 7fa28566b640 1 -- 172.21.15.29:0/3404529940 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa26c003520 con 0x7fa26c003f20 2023-10-14T18:17:09.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.376+0000 7fa28566b640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa26c003f20 0x7fa26c0047c0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fa270001750 tx=0x7fa26c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd6ee41b00cb96e6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:09.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.376+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700052d0 con 0x7fa26c003f20 2023-10-14T18:17:09.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.376+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa27002e050 con 0x7fa26c003f20 2023-10-14T18:17:09.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.376+0000 7fa284e6a640 1 -- 172.21.15.29:0/3404529940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270003830 con 0x7fa26c003f20 2023-10-14T18:17:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 -- 172.21.15.29:0/3404529940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa26c003f20 msgr2=0x7fa26c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 --2- 172.21.15.29:0/3404529940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa26c003f20 0x7fa26c0047c0 secure :-1 s=READY pgs=2492 cs=0 l=1 rev1=1 crypto rx=0x7fa270001750 tx=0x7fa26c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa28666d640 1 -- 172.21.15.29:0/3404529940 reap_dead start 2023-10-14T18:17:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 -- 172.21.15.29:0/3404529940 shutdown_connections 2023-10-14T18:17:09.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 -- 172.21.15.29:0/3404529940 >> 172.21.15.29:0/3404529940 conn(0x7fa2800ce9b0 msgr2=0x7fa2800cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 -- 172.21.15.29:0/3404529940 shutdown_connections 2023-10-14T18:17:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.377+0000 7fa2880d0640 1 -- 172.21.15.29:0/3404529940 wait complete. 2023-10-14T18:17:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.379+0000 7fa2880d0640 1 Processor -- start 2023-10-14T18:17:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.379+0000 7fa2880d0640 1 -- start start 2023-10-14T18:17:09.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 0x7fa2800c49a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa280157220 0x7fa280153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2801549c0 con 0x7fa280156da0 2023-10-14T18:17:09.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa280154ca0 con 0x7fa26c003f20 2023-10-14T18:17:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.380+0000 7fa2880d0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa280154f80 con 0x7fa280157220 2023-10-14T18:17:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39012/0 (socket says 172.21.15.29:39012) 2023-10-14T18:17:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 -- 172.21.15.29:0/4229514522 learned_addr learned my addr 172.21.15.29:0/4229514522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:09.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa285e6c640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 0x7fa2800c49a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 -- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa280157220 msgr2=0x7fa280153b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28666d640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa280157220 0x7fa280153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa280157220 0x7fa280153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 -- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 msgr2=0x7fa2800c49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 0x7fa2800c49a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 -- 172.21.15.29:0/4229514522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa280155260 con 0x7fa280156da0 2023-10-14T18:17:09.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa285e6c640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 0x7fa2800c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:09.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.381+0000 7fa28566b640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa270008fd0 tx=0x7fa270001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:09.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.382+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2700076c0 con 0x7fa280156da0 2023-10-14T18:17:09.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.382+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa270007a30 con 0x7fa280156da0 2023-10-14T18:17:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.382+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa270006c00 con 0x7fa280156da0 2023-10-14T18:17:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.382+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa280155540 con 0x7fa280156da0 2023-10-14T18:17:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.382+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2800c4ee0 con 0x7fa280156da0 2023-10-14T18:17:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.383+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2801558b0 con 0x7fa280156da0 2023-10-14T18:17:09.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.384+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 133) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fa2700035f0 con 0x7fa280156da0 2023-10-14T18:17:09.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.385+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fa2700ab670 con 0x7fa280156da0 2023-10-14T18:17:09.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.389+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa270076650 con 0x7fa280156da0 2023-10-14T18:17:09.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.588+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa2801556d0 con 0x7fa280156da0 2023-10-14T18:17:09.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.589+0000 7fa2767fc640 1 -- 172.21.15.29:0/4229514522 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v216) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa27004b020 con 0x7fa280156da0 2023-10-14T18:17:09.590 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:17:09.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 msgr2=0x7fa280153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 secure :-1 s=READY pgs=2493 cs=0 l=1 rev1=1 crypto rx=0x7fa270008fd0 tx=0x7fa270001120 comp rx=0 tx=0).stop 2023-10-14T18:17:09.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 shutdown_connections 2023-10-14T18:17:09.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa280157220 0x7fa280153b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa280156da0 0x7fa280153630 unknown :-1 s=CLOSED pgs=2493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 --2- 172.21.15.29:0/4229514522 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa26c003f20 0x7fa2800c49a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 >> 172.21.15.29:0/4229514522 conn(0x7fa2800ce9b0 msgr2=0x7fa280069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 shutdown_connections 2023-10-14T18:17:09.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.592+0000 7fa2880d0640 1 -- 172.21.15.29:0/4229514522 wait complete. 2023-10-14T18:17:09.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:09.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.849+0000 7fd90a01a640 1 Processor -- start 2023-10-14T18:17:09.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.849+0000 7fd90a01a640 1 -- start start 2023-10-14T18:17:09.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd90a01a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd90a01a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd9040cda80 0x7fd90415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd90a01a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd90406c960 con 0x7fd9040ad970 2023-10-14T18:17:09.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd90a01a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd90415b9d0 con 0x7fd9040cda80 2023-10-14T18:17:09.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd90a01a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd90415bcb0 con 0x7fd90406c040 2023-10-14T18:17:09.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.851+0000 7fd909018640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:51364/0 (socket says 172.21.15.29:51364) 2023-10-14T18:17:09.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 -- 172.21.15.29:0/910191041 learned_addr learned my addr 172.21.15.29:0/910191041 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:09.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd908817640 1 --1- 172.21.15.29:0/910191041 >> v1:172.21.15.186:6789/0 conn(0x7fd9040ad970 0x7fd90415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:39448/0 (socket says 172.21.15.29:39448) 2023-10-14T18:17:09.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909819640 1 --2- 172.21.15.29:0/910191041 >> v2:172.21.15.186:3300/0 conn(0x7fd9040cda80 0x7fd90415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 -- 172.21.15.29:0/910191041 >> v2:172.21.15.186:3300/0 conn(0x7fd9040cda80 msgr2=0x7fd90415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:09.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 --2- 172.21.15.29:0/910191041 >> v2:172.21.15.186:3300/0 conn(0x7fd9040cda80 0x7fd90415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 -- 172.21.15.29:0/910191041 >> v1:172.21.15.186:6789/0 conn(0x7fd9040ad970 legacy=0x7fd90415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.852+0000 7fd909018640 1 -- 172.21.15.29:0/910191041 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd90415bf90 con 0x7fd90406c040 2023-10-14T18:17:09.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.854+0000 7fd909018640 1 --2- 172.21.15.29:0/910191041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd900009870 tx=0x7fd90002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e828c65a794d65ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:09.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.854+0000 7fd8fbfff640 1 -- 172.21.15.29:0/910191041 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900030840 con 0x7fd90406c040 2023-10-14T18:17:09.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.854+0000 7fd8fbfff640 1 -- 172.21.15.29:0/910191041 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9000309a0 con 0x7fd90406c040 2023-10-14T18:17:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.854+0000 7fd8fbfff640 1 -- 172.21.15.29:0/910191041 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd900030d30 con 0x7fd90406c040 2023-10-14T18:17:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 -- 172.21.15.29:0/910191041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 msgr2=0x7fd90406c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 --2- 172.21.15.29:0/910191041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd900009870 tx=0x7fd90002b980 comp rx=0 tx=0).stop 2023-10-14T18:17:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 -- 172.21.15.29:0/910191041 shutdown_connections 2023-10-14T18:17:09.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 --2- 172.21.15.29:0/910191041 >> v2:172.21.15.186:3300/0 conn(0x7fd9040cda80 0x7fd90415ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 --2- 172.21.15.29:0/910191041 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd90406c420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 -- 172.21.15.29:0/910191041 >> 172.21.15.29:0/910191041 conn(0x7fd9040ce6a0 msgr2=0x7fd9040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 -- 172.21.15.29:0/910191041 shutdown_connections 2023-10-14T18:17:09.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.855+0000 7fd90a01a640 1 -- 172.21.15.29:0/910191041 wait complete. 2023-10-14T18:17:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.857+0000 7fd90a01a640 1 Processor -- start 2023-10-14T18:17:09.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.857+0000 7fd90a01a640 1 -- start start 2023-10-14T18:17:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd9040b0290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9040cda80 0x7fd90406ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040af1f0 con 0x7fd9040ad970 2023-10-14T18:17:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040af4d0 con 0x7fd9040cda80 2023-10-14T18:17:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd90a01a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9040af7b0 con 0x7fd90406c040 2023-10-14T18:17:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd908817640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd908817640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39024/0 (socket says 172.21.15.29:39024) 2023-10-14T18:17:09.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.858+0000 7fd908817640 1 -- 172.21.15.29:0/558787627 learned_addr learned my addr 172.21.15.29:0/558787627 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:09.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd909018640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd9040b0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd909819640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9040cda80 0x7fd90406ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 -- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 msgr2=0x7fd9040b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd9040b0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 -- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9040cda80 msgr2=0x7fd90406ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:09.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9040cda80 0x7fd90406ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:09.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 -- 172.21.15.29:0/558787627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9000094d0 con 0x7fd9040ad970 2023-10-14T18:17:09.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd908817640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c400 tx=0x7fd8f400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:09.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f40074e0 con 0x7fd9040ad970 2023-10-14T18:17:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8f4007640 con 0x7fd9040ad970 2023-10-14T18:17:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.859+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8f4004a00 con 0x7fd9040ad970 2023-10-14T18:17:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.860+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9040afa90 con 0x7fd9040ad970 2023-10-14T18:17:09.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.860+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd90414d2f0 con 0x7fd9040ad970 2023-10-14T18:17:09.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.861+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd90406b330 con 0x7fd9040ad970 2023-10-14T18:17:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.862+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd8f4004b60 con 0x7fd9040ad970 2023-10-14T18:17:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.863+0000 7fd8f9ffb640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 0x7fd8e806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.863+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fd8f4089080 con 0x7fd9040ad970 2023-10-14T18:17:09.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.863+0000 7fd909018640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 0x7fd8e806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.864+0000 7fd909018640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 0x7fd8e806d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd900036000 tx=0x7fd900008030 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:09.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:09.867+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8f40bbfe0 con 0x7fd9040ad970 2023-10-14T18:17:10.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.102+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd9040affb0 con 0x7fd9040ad970 2023-10-14T18:17:10.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.103+0000 7fd8f9ffb640 1 -- 172.21.15.29:0/558787627 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7fd9040affb0 con 0x7fd9040ad970 2023-10-14T18:17:10.105 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:10.106 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:17:10.106 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:17:10.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 msgr2=0x7fd8e806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 0x7fd8e806d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fd900036000 tx=0x7fd900008030 comp rx=0 tx=0).stop 2023-10-14T18:17:10.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 msgr2=0x7fd90414cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 secure :-1 s=READY pgs=2494 cs=0 l=1 rev1=1 crypto rx=0x7fd8f400c400 tx=0x7fd8f400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 shutdown_connections 2023-10-14T18:17:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd8e806acc0 0x7fd8e806d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9040cda80 0x7fd90406ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9040ad970 0x7fd90414cdb0 unknown :-1 s=CLOSED pgs=2494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 --2- 172.21.15.29:0/558787627 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd90406c040 0x7fd9040b0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 >> 172.21.15.29:0/558787627 conn(0x7fd9040ce6a0 msgr2=0x7fd9040ca6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:10.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 shutdown_connections 2023-10-14T18:17:10.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.106+0000 7fd90a01a640 1 -- 172.21.15.29:0/558787627 wait complete. 2023-10-14T18:17:10.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.342+0000 7fd5b6283640 1 Processor -- start 2023-10-14T18:17:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.342+0000 7fd5b6283640 1 -- start start 2023-10-14T18:17:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b6283640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd5b015ae80 0x7fd5b015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b6283640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd5b00ce390 0x7fd5b0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b6283640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5b015ed60 con 0x7fd5b015aaa0 2023-10-14T18:17:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b6283640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b006b810 con 0x7fd5b00ce390 2023-10-14T18:17:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b6283640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5b00707e0 con 0x7fd5b015ae80 2023-10-14T18:17:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5affff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd5b015aaa0 0x7fd5b00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:36352/0 (socket says 172.21.15.29:36352) 2023-10-14T18:17:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5affff640 1 -- 172.21.15.29:0/108770705 learned_addr learned my addr 172.21.15.29:0/108770705 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5b4820640 1 --2- 172.21.15.29:0/108770705 >> v2:172.21.15.29:3300/0 conn(0x7fd5b00ce390 0x7fd5b0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5af7fe640 1 --2- 172.21.15.29:0/108770705 >> v2:172.21.15.186:3300/0 conn(0x7fd5b015ae80 0x7fd5b015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.344+0000 7fd5aeffd640 1 -- 172.21.15.29:0/108770705 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464511980 0 0) 0x7fd5b015ed60 con 0x7fd5b015aaa0 2023-10-14T18:17:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.345+0000 7fd5aeffd640 1 -- 172.21.15.29:0/108770705 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd598003520 con 0x7fd5b015aaa0 2023-10-14T18:17:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.345+0000 7fd5b4820640 1 -- 172.21.15.29:0/108770705 >> v2:172.21.15.186:3300/0 conn(0x7fd5b015ae80 msgr2=0x7fd5b015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.345+0000 7fd5b4820640 1 --2- 172.21.15.29:0/108770705 >> v2:172.21.15.186:3300/0 conn(0x7fd5b015ae80 0x7fd5b015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.345+0000 7fd5b4820640 1 -- 172.21.15.29:0/108770705 >> v1:172.21.15.29:6789/0 conn(0x7fd5b015aaa0 legacy=0x7fd5b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.345+0000 7fd5b4820640 1 -- 172.21.15.29:0/108770705 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b015ef90 con 0x7fd5b00ce390 2023-10-14T18:17:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.346+0000 7fd5b4820640 1 --2- 172.21.15.29:0/108770705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b0070f50 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00c59c0 tx=0x7fd5a802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec21fe1a26de85dd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.346+0000 7fd5aeffd640 1 -- 172.21.15.29:0/108770705 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8031830 con 0x7fd5b00ce390 2023-10-14T18:17:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.346+0000 7fd5aeffd640 1 -- 172.21.15.29:0/108770705 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a8031990 con 0x7fd5b00ce390 2023-10-14T18:17:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.346+0000 7fd5aeffd640 1 -- 172.21.15.29:0/108770705 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a8031d20 con 0x7fd5b00ce390 2023-10-14T18:17:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 -- 172.21.15.29:0/108770705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 msgr2=0x7fd5b0070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 --2- 172.21.15.29:0/108770705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b0070f50 secure :-1 s=READY pgs=2496 cs=0 l=1 rev1=1 crypto rx=0x7fd5b00c59c0 tx=0x7fd5a802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 -- 172.21.15.29:0/108770705 shutdown_connections 2023-10-14T18:17:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 --2- 172.21.15.29:0/108770705 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b0070f50 unknown :-1 s=CLOSED pgs=2496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 --2- 172.21.15.29:0/108770705 >> v2:172.21.15.186:3300/0 conn(0x7fd5b015ae80 0x7fd5b015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 -- 172.21.15.29:0/108770705 >> 172.21.15.29:0/108770705 conn(0x7fd5b00cb410 msgr2=0x7fd5b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 -- 172.21.15.29:0/108770705 shutdown_connections 2023-10-14T18:17:10.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.347+0000 7fd5b6283640 1 -- 172.21.15.29:0/108770705 wait complete. 2023-10-14T18:17:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.349+0000 7fd5b6283640 1 Processor -- start 2023-10-14T18:17:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.349+0000 7fd5b6283640 1 -- start start 2023-10-14T18:17:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5b015aaa0 0x7fd5b01f1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5b015ae80 0x7fd5b0152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b01f45d0 con 0x7fd5b00ce390 2023-10-14T18:17:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b01f48b0 con 0x7fd5b015aaa0 2023-10-14T18:17:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5b6283640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b01f4b90 con 0x7fd5b015ae80 2023-10-14T18:17:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5affff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39030/0 (socket says 172.21.15.29:39030) 2023-10-14T18:17:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.350+0000 7fd5affff640 1 -- 172.21.15.29:0/2331512843 learned_addr learned my addr 172.21.15.29:0/2331512843 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 -- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5b015ae80 msgr2=0x7fd5b0152180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5b015ae80 0x7fd5b0152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 -- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5b015aaa0 msgr2=0x7fd5b01f1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5b015aaa0 0x7fd5b01f1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 -- 172.21.15.29:0/2331512843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a8009400 con 0x7fd5b00ce390 2023-10-14T18:17:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5affff640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000cbc0 tx=0x7fd5a000a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000ad40 con 0x7fd5b00ce390 2023-10-14T18:17:10.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5a0012400 con 0x7fd5b00ce390 2023-10-14T18:17:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b01f0810 con 0x7fd5b00ce390 2023-10-14T18:17:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.351+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5a000aea0 con 0x7fd5b00ce390 2023-10-14T18:17:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.352+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b01f4230 con 0x7fd5b00ce390 2023-10-14T18:17:10.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.353+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b004e160 con 0x7fd5b00ce390 2023-10-14T18:17:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.354+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd5a0007070 con 0x7fd5b00ce390 2023-10-14T18:17:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.355+0000 7fd5acff9640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 0x7fd59c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.355+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7fd5a0089640 con 0x7fd5b00ce390 2023-10-14T18:17:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.355+0000 7fd5af7fe640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 0x7fd59c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.356+0000 7fd5af7fe640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 0x7fd59c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4006fd0 tx=0x7fd5a4008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.359+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5a0054690 con 0x7fd5b00ce390 2023-10-14T18:17:10.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.558+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5b00cdc60 con 0x7fd5b00ce390 2023-10-14T18:17:10.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.561+0000 7fd5acff9640 1 -- 172.21.15.29:0/2331512843 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+17863 (secure 0 0 0) 0x7fd5a0017090 con 0x7fd5b00ce390 2023-10-14T18:17:10.563 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:10.563 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":216,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:08.714341+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":215,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.17","mappings":[{"from":0,"to":2}]},{"pgid":"25.2d","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1567973042":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1483914389":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1310817924":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/3470389371":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1992230882":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/976049980":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6833/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6832/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/4216364220":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6832/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1790177658":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/3670605654":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/2768480413":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/4181160999":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6833/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1466849565":"2023-10-15T18:15:00.600793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 msgr2=0x7fd59c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 0x7fd59c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd5a4006fd0 tx=0x7fd5a4008040 comp rx=0 tx=0).stop 2023-10-14T18:17:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 msgr2=0x7fd5b01f3cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 secure :-1 s=READY pgs=2497 cs=0 l=1 rev1=1 crypto rx=0x7fd5a000cbc0 tx=0x7fd5a000a430 comp rx=0 tx=0).stop 2023-10-14T18:17:10.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 shutdown_connections 2023-10-14T18:17:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd59c06acc0 0x7fd59c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd5b015ae80 0x7fd5b0152180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd5b015aaa0 0x7fd5b01f1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 --2- 172.21.15.29:0/2331512843 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd5b00ce390 0x7fd5b01f3cf0 unknown :-1 s=CLOSED pgs=2497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 >> 172.21.15.29:0/2331512843 conn(0x7fd5b00cb410 msgr2=0x7fd5b00700a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:10.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 shutdown_connections 2023-10-14T18:17:10.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.564+0000 7fd5b6283640 1 -- 172.21.15.29:0/2331512843 wait complete. 2023-10-14T18:17:10.587 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:17:10.588 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:10.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.814+0000 7f1fc759e640 1 Processor -- start 2023-10-14T18:17:10.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.814+0000 7f1fc759e640 1 -- start start 2023-10-14T18:17:10.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.815+0000 7f1fc759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.815+0000 7f1fc759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fc80cda80 0x7f1fc815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.815+0000 7f1fc759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fc806c960 con 0x7f1fc80ad970 2023-10-14T18:17:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.815+0000 7f1fc759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fc815b9d0 con 0x7f1fc80cda80 2023-10-14T18:17:10.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.815+0000 7f1fc759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc815bcb0 con 0x7f1fc806c040 2023-10-14T18:17:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fc80cda80 0x7f1fc815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1fc80cda80 0x7f1fc815ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39042/0 (socket says 172.21.15.29:39042) 2023-10-14T18:17:10.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3152799579 learned_addr learned my addr 172.21.15.29:0/3152799579 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:10.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc659c640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3152799579 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 msgr2=0x7f1fc806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3152799579 >> v1:172.21.15.186:6789/0 conn(0x7f1fc80ad970 legacy=0x7f1fc815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:10.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.816+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3152799579 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc815bf90 con 0x7f1fc80cda80 2023-10-14T18:17:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.817+0000 7f1fc6d9d640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc815ed90 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e50 tx=0x7f1fc002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e994386a602205 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.818+0000 7f1fc559a640 1 -- 172.21.15.29:0/3152799579 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc002ca70 con 0x7f1fc80cda80 2023-10-14T18:17:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.818+0000 7f1fc559a640 1 -- 172.21.15.29:0/3152799579 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc002cbd0 con 0x7f1fc80cda80 2023-10-14T18:17:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.818+0000 7f1fc559a640 1 -- 172.21.15.29:0/3152799579 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc00343f0 con 0x7f1fc80cda80 2023-10-14T18:17:10.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.818+0000 7f1fc759e640 1 -- 172.21.15.29:0/3152799579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 msgr2=0x7f1fc815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.818+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc815ed90 secure :-1 s=READY pgs=2498 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0009e50 tx=0x7f1fc002bed0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 -- 172.21.15.29:0/3152799579 shutdown_connections 2023-10-14T18:17:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc815ed90 unknown :-1 s=CLOSED pgs=2498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3152799579 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 -- 172.21.15.29:0/3152799579 >> 172.21.15.29:0/3152799579 conn(0x7f1fc80ce6a0 msgr2=0x7f1fc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 -- 172.21.15.29:0/3152799579 shutdown_connections 2023-10-14T18:17:10.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.819+0000 7f1fc759e640 1 -- 172.21.15.29:0/3152799579 wait complete. 2023-10-14T18:17:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.820+0000 7f1fc759e640 1 Processor -- start 2023-10-14T18:17:10.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.821+0000 7f1fc759e640 1 -- start start 2023-10-14T18:17:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc8149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 0x7f1fc80c7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8156770 con 0x7f1fc80cda80 2023-10-14T18:17:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8158f40 con 0x7f1fc80ad970 2023-10-14T18:17:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc8159220 con 0x7f1fc806c040 2023-10-14T18:17:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:39052/0 (socket says 172.21.15.29:39052) 2023-10-14T18:17:10.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3568725988 learned_addr learned my addr 172.21.15.29:0/3568725988 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.822+0000 7f1fc659c640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc8149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc5d9b640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 0x7f1fc80c7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 msgr2=0x7f1fc8149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc8149930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 msgr2=0x7f1fc80c7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 0x7f1fc80c7300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 -- 172.21.15.29:0/3568725988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc0009b30 con 0x7f1fc80cda80 2023-10-14T18:17:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc659c640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc8149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc5d9b640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 0x7f1fc80c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:10.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fc6d9d640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0036040 tx=0x7f1fc0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.823+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0040670 con 0x7f1fc80cda80 2023-10-14T18:17:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.824+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fc00407f0 con 0x7f1fc80cda80 2023-10-14T18:17:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.824+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fc0040ba0 con 0x7f1fc80cda80 2023-10-14T18:17:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.824+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc8149e70 con 0x7f1fc80cda80 2023-10-14T18:17:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.824+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc8152920 con 0x7f1fc80cda80 2023-10-14T18:17:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.825+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc806b330 con 0x7f1fc80cda80 2023-10-14T18:17:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.826+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 134) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f1fc0040d00 con 0x7f1fc80cda80 2023-10-14T18:17:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.827+0000 7f1fbeffd640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 0x7f1fa806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.827+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f1fc00ac850 con 0x7f1fc80cda80 2023-10-14T18:17:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.827+0000 7f1fc659c640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 0x7f1fa806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.828+0000 7f1fc659c640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 0x7f1fa806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800b820 tx=0x7f1fb8008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:10.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:10.832+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1fc00777f0 con 0x7f1fc80cda80 2023-10-14T18:17:11.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.074+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1fc804e9b0 con 0x7f1fc80cda80 2023-10-14T18:17:11.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.075+0000 7f1fbeffd640 1 -- 172.21.15.29:0/3568725988 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f1fc007ae10 con 0x7f1fc80cda80 2023-10-14T18:17:11.077 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:11.077 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:17:11.077 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:17:11.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 msgr2=0x7f1fa806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 0x7f1fa806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800b820 tx=0x7f1fb8008040 comp rx=0 tx=0).stop 2023-10-14T18:17:11.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 msgr2=0x7f1fc80c7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 secure :-1 s=READY pgs=2499 cs=0 l=1 rev1=1 crypto rx=0x7f1fc0036040 tx=0x7f1fc0004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 shutdown_connections 2023-10-14T18:17:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1fa806acc0 0x7f1fa806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1fc80cda80 0x7f1fc80c7840 unknown :-1 s=CLOSED pgs=2499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1fc80ad970 0x7f1fc80c7300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 --2- 172.21.15.29:0/3568725988 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1fc806c040 0x7f1fc8149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 >> 172.21.15.29:0/3568725988 conn(0x7f1fc80ce6a0 msgr2=0x7f1fc8159b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:11.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 shutdown_connections 2023-10-14T18:17:11.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.078+0000 7f1fc759e640 1 -- 172.21.15.29:0/3568725988 wait complete. 2023-10-14T18:17:11.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.331+0000 7f501b59e640 1 Processor -- start 2023-10-14T18:17:11.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.331+0000 7f501b59e640 1 -- start start 2023-10-14T18:17:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.333+0000 7f501b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c06a4b0 0x7f501c15e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.333+0000 7f501b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c0b0e10 con 0x7f501c0d0400 2023-10-14T18:17:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.333+0000 7f501b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501c155520 con 0x7f501c155140 2023-10-14T18:17:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.333+0000 7f501b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501c0b1070 con 0x7f501c06a4b0 2023-10-14T18:17:11.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c06a4b0 0x7f501c15e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f501c06a4b0 0x7f501c15e6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33844/0 (socket says 172.21.15.29:33844) 2023-10-14T18:17:11.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 -- 172.21.15.29:0/1267502783 learned_addr learned my addr 172.21.15.29:0/1267502783 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 -- 172.21.15.29:0/1267502783 >> v1:172.21.15.186:6789/0 conn(0x7f501c155140 legacy=0x7f501c0b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 -- 172.21.15.29:0/1267502783 >> v1:172.21.15.29:6789/0 conn(0x7f501c0d0400 legacy=0x7f501c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.334+0000 7f501ad9d640 1 -- 172.21.15.29:0/1267502783 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501c0cf920 con 0x7f501c06a4b0 2023-10-14T18:17:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.335+0000 7f501ad9d640 1 --2- 172.21.15.29:0/1267502783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c15e6f0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f5014009870 tx=0x7f501402ba30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3116ff9cf5a6333 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.336+0000 7f501959a640 1 -- 172.21.15.29:0/1267502783 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501402f850 con 0x7f501c06a4b0 2023-10-14T18:17:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.336+0000 7f501959a640 1 -- 172.21.15.29:0/1267502783 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501402f9b0 con 0x7f501c06a4b0 2023-10-14T18:17:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.336+0000 7f501959a640 1 -- 172.21.15.29:0/1267502783 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f501402fd40 con 0x7f501c06a4b0 2023-10-14T18:17:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.336+0000 7f501b59e640 1 -- 172.21.15.29:0/1267502783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 msgr2=0x7f501c15e6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.336+0000 7f501b59e640 1 --2- 172.21.15.29:0/1267502783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c15e6f0 secure :-1 s=READY pgs=2503 cs=0 l=1 rev1=1 crypto rx=0x7f5014009870 tx=0x7f501402ba30 comp rx=0 tx=0).stop 2023-10-14T18:17:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.337+0000 7f501b59e640 1 -- 172.21.15.29:0/1267502783 shutdown_connections 2023-10-14T18:17:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.337+0000 7f501b59e640 1 --2- 172.21.15.29:0/1267502783 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c15e6f0 unknown :-1 s=CLOSED pgs=2503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.337+0000 7f501b59e640 1 -- 172.21.15.29:0/1267502783 >> 172.21.15.29:0/1267502783 conn(0x7f501c06aeb0 msgr2=0x7f501c06b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:11.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.337+0000 7f501b59e640 1 -- 172.21.15.29:0/1267502783 shutdown_connections 2023-10-14T18:17:11.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.337+0000 7f501b59e640 1 -- 172.21.15.29:0/1267502783 wait complete. 2023-10-14T18:17:11.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.338+0000 7f501b59e640 1 Processor -- start 2023-10-14T18:17:11.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.339+0000 7f501b59e640 1 -- start start 2023-10-14T18:17:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f501c0d0400 0x7f501c1e4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f501c155140 0x7f501c144590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c145d10 con 0x7f501c06a4b0 2023-10-14T18:17:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f501c145100 con 0x7f501c0d0400 2023-10-14T18:17:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f501c1453e0 con 0x7f501c155140 2023-10-14T18:17:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33858/0 (socket says 172.21.15.29:33858) 2023-10-14T18:17:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 -- 172.21.15.29:0/425597658 learned_addr learned my addr 172.21.15.29:0/425597658 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f5019d9b640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f501c0d0400 0x7f501c1e4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501ad9d640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f501c155140 0x7f501c144590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 -- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f501c155140 msgr2=0x7f501c144590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f501c155140 0x7f501c144590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 -- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f501c0d0400 msgr2=0x7f501c1e4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f501c0d0400 0x7f501c1e4d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.340+0000 7f501a59c640 1 -- 172.21.15.29:0/425597658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140094d0 con 0x7f501c06a4b0 2023-10-14T18:17:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f501a59c640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c730 tx=0x7f500c00cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c003b70 con 0x7f501c06a4b0 2023-10-14T18:17:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f500c003cf0 con 0x7f501c06a4b0 2023-10-14T18:17:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f500c003ea0 con 0x7f501c06a4b0 2023-10-14T18:17:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c1456c0 con 0x7f501c06a4b0 2023-10-14T18:17:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.341+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f501c153f40 con 0x7f501c06a4b0 2023-10-14T18:17:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.342+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f501c04e9b0 con 0x7f501c06a4b0 2023-10-14T18:17:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.344+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f500c00aa90 con 0x7f501c06a4b0 2023-10-14T18:17:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.344+0000 7f5012ffd640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 0x7f4ffc06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.345+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f500c089870 con 0x7f501c06a4b0 2023-10-14T18:17:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.345+0000 7f5019d9b640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 0x7f4ffc06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.345+0000 7f5019d9b640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 0x7f4ffc06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f501c0b13f0 tx=0x7f5004009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.349+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f500c054520 con 0x7f501c06a4b0 2023-10-14T18:17:11.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.585+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f501c0ccac0 con 0x7f501c06a4b0 2023-10-14T18:17:11.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.587+0000 7f5012ffd640 1 -- 172.21.15.29:0/425597658 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+4920 (secure 0 0 0) 0x7f500c057b40 con 0x7f501c06a4b0 2023-10-14T18:17:11.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:11.589 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":98,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94}],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:14:46.008237+0000","modified":"2023-10-14T18:14:49.076146+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7984},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7984":{"gid":7984,"name":"c","rank":0,"incarnation":95,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1451005601","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1451005601},{"type":"v1","addr":"172.21.15.29:6837","nonce":1451005601}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[25],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":12}]} 2023-10-14T18:17:11.589 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 98 2023-10-14T18:17:11.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 msgr2=0x7f4ffc06d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 0x7f4ffc06d3c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f501c0b13f0 tx=0x7f5004009040 comp rx=0 tx=0).stop 2023-10-14T18:17:11.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 msgr2=0x7f501c1431c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 secure :-1 s=READY pgs=2504 cs=0 l=1 rev1=1 crypto rx=0x7f500c00c730 tx=0x7f500c00cbf0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 shutdown_connections 2023-10-14T18:17:11.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4ffc06af00 0x7f4ffc06d3c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f501c155140 0x7f501c144590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f501c0d0400 0x7f501c1e4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 --2- 172.21.15.29:0/425597658 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f501c06a4b0 0x7f501c1431c0 unknown :-1 s=CLOSED pgs=2504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.589+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 >> 172.21.15.29:0/425597658 conn(0x7f501c06aeb0 msgr2=0x7f501c06b290 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:11.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.590+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 shutdown_connections 2023-10-14T18:17:11.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.590+0000 7f501b59e640 1 -- 172.21.15.29:0/425597658 wait complete. 2023-10-14T18:17:11.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:11.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.842+0000 7f91259ff640 1 Processor -- start 2023-10-14T18:17:11.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.842+0000 7f91259ff640 1 -- start start 2023-10-14T18:17:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 0x7f912006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f912015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f912015ebc0 0x7f912015f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912006c960 con 0x7f912015ebc0 2023-10-14T18:17:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f912015b9d0 con 0x7f912006c040 2023-10-14T18:17:11.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.844+0000 7f91259ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f912015bcb0 con 0x7f91200ad970 2023-10-14T18:17:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f912015ebc0 0x7f912015f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f912015ebc0 0x7f912015f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33868/0 (socket says 172.21.15.29:33868) 2023-10-14T18:17:11.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 -- 172.21.15.29:0/3691852857 learned_addr learned my addr 172.21.15.29:0/3691852857 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91249fd640 1 --2- 172.21.15.29:0/3691852857 >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 0x7f912006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f911ffff640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f912015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 -- 172.21.15.29:0/3691852857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 msgr2=0x7f912015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:11.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f912015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 -- 172.21.15.29:0/3691852857 >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 msgr2=0x7f912006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 --2- 172.21.15.29:0/3691852857 >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 0x7f912006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91251fe640 1 -- 172.21.15.29:0/3691852857 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912015bf90 con 0x7f912015ebc0 2023-10-14T18:17:11.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.845+0000 7f91249fd640 1 --2- 172.21.15.29:0/3691852857 >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 0x7f912006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.847+0000 7f91251fe640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912015ebc0 0x7f912015f050 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f9118009e50 tx=0x7f911802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2df99fa80716cc3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.847+0000 7f911f7fe640 1 -- 172.21.15.29:0/3691852857 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f911802ca70 con 0x7f912015ebc0 2023-10-14T18:17:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.847+0000 7f911f7fe640 1 -- 172.21.15.29:0/3691852857 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f911802cbd0 con 0x7f912015ebc0 2023-10-14T18:17:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.847+0000 7f911f7fe640 1 -- 172.21.15.29:0/3691852857 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91180343f0 con 0x7f912015ebc0 2023-10-14T18:17:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 -- 172.21.15.29:0/3691852857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912015ebc0 msgr2=0x7f912015f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912015ebc0 0x7f912015f050 secure :-1 s=READY pgs=2505 cs=0 l=1 rev1=1 crypto rx=0x7f9118009e50 tx=0x7f911802bed0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 -- 172.21.15.29:0/3691852857 shutdown_connections 2023-10-14T18:17:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912015ebc0 0x7f912015f050 unknown :-1 s=CLOSED pgs=2505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 --2- 172.21.15.29:0/3691852857 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f912015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 --2- 172.21.15.29:0/3691852857 >> v2:172.21.15.186:3300/0 conn(0x7f912006c040 0x7f912006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 -- 172.21.15.29:0/3691852857 >> 172.21.15.29:0/3691852857 conn(0x7f91200ce6a0 msgr2=0x7f91200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 -- 172.21.15.29:0/3691852857 shutdown_connections 2023-10-14T18:17:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.848+0000 7f91259ff640 1 -- 172.21.15.29:0/3691852857 wait complete. 2023-10-14T18:17:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.850+0000 7f91259ff640 1 Processor -- start 2023-10-14T18:17:11.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.850+0000 7f91259ff640 1 -- start start 2023-10-14T18:17:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f91201594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912015ebc0 0x7f91200b00c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91201590d0 con 0x7f912006c040 2023-10-14T18:17:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91200b0a30 con 0x7f912015ebc0 2023-10-14T18:17:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.851+0000 7f91259ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9120158a10 con 0x7f91200ad970 2023-10-14T18:17:11.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33878/0 (socket says 172.21.15.29:33878) 2023-10-14T18:17:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 -- 172.21.15.29:0/2573717038 learned_addr learned my addr 172.21.15.29:0/2573717038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 -- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 msgr2=0x7f91201594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f91201594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 -- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912015ebc0 msgr2=0x7f91200b00c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912015ebc0 0x7f91200b00c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 -- 172.21.15.29:0/2573717038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9118009b30 con 0x7f912006c040 2023-10-14T18:17:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.852+0000 7f91249fd640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f911400d800 tx=0x7f911400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.853+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114004030 con 0x7f912006c040 2023-10-14T18:17:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.853+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9114004190 con 0x7f912006c040 2023-10-14T18:17:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.853+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9114004520 con 0x7f912006c040 2023-10-14T18:17:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.853+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9120158cf0 con 0x7f912006c040 2023-10-14T18:17:11.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.853+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91200b0d10 con 0x7f912006c040 2023-10-14T18:17:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.854+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f912006b330 con 0x7f912006c040 2023-10-14T18:17:11.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.855+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f91140096a0 con 0x7f912006c040 2023-10-14T18:17:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.856+0000 7f911d7fa640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 0x7f910406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.856+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(216..216 src has 1..216) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f9114089da0 con 0x7f912006c040 2023-10-14T18:17:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.856+0000 7f911ffff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 0x7f910406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:11.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.857+0000 7f911ffff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 0x7f910406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f910c006fd0 tx=0x7f910c008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:11.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:11.860+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f91140549e0 con 0x7f912006c040 2023-10-14T18:17:12.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.059+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9120158870 con 0x7f912006c040 2023-10-14T18:17:12.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.062+0000 7f911d7fa640 1 -- 172.21.15.29:0/2573717038 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v216) v1 ==== 74+0+17863 (secure 0 0 0) 0x7f9120158870 con 0x7f912006c040 2023-10-14T18:17:12.065 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:12.065 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":216,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:08.714341+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":24,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:14:43.916335+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"209","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":25,"pool_name":"cephfs_data","create_time":"2023-10-14T18:14:45.028887+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"215","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":5,"snap_epoch":215,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"25.17","mappings":[{"from":0,"to":2}]},{"pgid":"25.2d","mappings":[{"from":0,"to":1}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/1567973042":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1483914389":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1310817924":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/3470389371":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1992230882":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/976049980":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6833/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6832/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/4216364220":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6832/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1790177658":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/3670605654":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/2768480413":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/4181160999":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6833/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1466849565":"2023-10-15T18:15:00.600793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]},{"pool":25,"snaps":[{"begin":2,"length":4}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:12.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 msgr2=0x7f910406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 0x7f910406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f910c006fd0 tx=0x7f910c008040 comp rx=0 tx=0).stop 2023-10-14T18:17:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 msgr2=0x7f91200af360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 secure :-1 s=READY pgs=2506 cs=0 l=1 rev1=1 crypto rx=0x7f911400d800 tx=0x7f911400dcc0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 shutdown_connections 2023-10-14T18:17:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f910406af00 0x7f910406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f912015ebc0 0x7f91200b00c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f91200ad970 0x7f91201594f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 --2- 172.21.15.29:0/2573717038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f912006c040 0x7f91200af360 unknown :-1 s=CLOSED pgs=2506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 >> 172.21.15.29:0/2573717038 conn(0x7f91200ce6a0 msgr2=0x7f9120072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:12.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 shutdown_connections 2023-10-14T18:17:12.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.065+0000 7f91259ff640 1 -- 172.21.15.29:0/2573717038 wait complete. 2023-10-14T18:17:12.088 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T18:17:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.316+0000 7f4e6ffff640 1 Processor -- start 2023-10-14T18:17:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.316+0000 7f4e6ffff640 1 -- start start 2023-10-14T18:17:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e6ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e700cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e6ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7015b9d0 con 0x7f4e700ad970 2023-10-14T18:17:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e6ffff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e7015bcb0 con 0x7f4e7006c040 2023-10-14T18:17:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e6ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7015b7d0 con 0x7f4e7015b320 2023-10-14T18:17:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e667fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4e700ad970 0x7f4e700cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48040/0 (socket says 172.21.15.29:48040) 2023-10-14T18:17:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e667fc640 1 -- 172.21.15.29:0/197007755 learned_addr learned my addr 172.21.15.29:0/197007755 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.318+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e700cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1652604369 0 0) 0x7f4e7015b9d0 con 0x7f4e700ad970 2023-10-14T18:17:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e5c003520 con 0x7f4e700ad970 2023-10-14T18:17:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258883599 0 0) 0x7f4e5c003520 con 0x7f4e700ad970 2023-10-14T18:17:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 msgr2=0x7f4e700cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e700cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 >> v1:172.21.15.186:6789/0 conn(0x7f4e7006c040 legacy=0x7f4e7006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e7015bf90 con 0x7f4e700ad970 2023-10-14T18:17:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4e58001670 con 0x7f4e700ad970 2023-10-14T18:17:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 >> v1:172.21.15.29:6789/0 conn(0x7f4e700ad970 legacy=0x7f4e700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0039a0 0x7f4e5c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0047c0 0x7f4e5c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e5c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e7015b7d0 con 0x7f4e5c0039a0 2023-10-14T18:17:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c00ba90 con 0x7f4e5c0047c0 2023-10-14T18:17:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e5c00bc50 con 0x7f4e7015b320 2023-10-14T18:17:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.319+0000 7f4e6effd640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0039a0 0x7f4e5c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e667fc640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0047c0 0x7f4e5c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e5c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6effd640 1 -- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 msgr2=0x7f4e5c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6effd640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e5c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6effd640 1 -- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0047c0 msgr2=0x7f4e5c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6effd640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0047c0 0x7f4e5c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6effd640 1 -- 172.21.15.29:0/197007755 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e5c003520 con 0x7f4e5c0039a0 2023-10-14T18:17:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.320+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e7015b320 0x7f4e5c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.321+0000 7f4e6effd640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0039a0 0x7f4e5c003dc0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f4e60001720 tx=0x7f4e5c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b062960d6f6f330 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.321+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600052d0 con 0x7f4e5c0039a0 2023-10-14T18:17:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.321+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e6002e050 con 0x7f4e5c0039a0 2023-10-14T18:17:12.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.321+0000 7f4e6e7fc640 1 -- 172.21.15.29:0/197007755 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e600035f0 con 0x7f4e5c0039a0 2023-10-14T18:17:12.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 -- 172.21.15.29:0/197007755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0039a0 msgr2=0x7f4e5c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/197007755 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e5c0039a0 0x7f4e5c003dc0 secure :-1 s=READY pgs=2508 cs=0 l=1 rev1=1 crypto rx=0x7f4e60001720 tx=0x7f4e5c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6f7fe640 1 -- 172.21.15.29:0/197007755 reap_dead start 2023-10-14T18:17:12.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 -- 172.21.15.29:0/197007755 shutdown_connections 2023-10-14T18:17:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 -- 172.21.15.29:0/197007755 >> 172.21.15.29:0/197007755 conn(0x7f4e700ce6a0 msgr2=0x7f4e700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 -- 172.21.15.29:0/197007755 shutdown_connections 2023-10-14T18:17:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.322+0000 7f4e6ffff640 1 -- 172.21.15.29:0/197007755 wait complete. 2023-10-14T18:17:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.324+0000 7f4e6ffff640 1 Processor -- start 2023-10-14T18:17:12.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.324+0000 7f4e6ffff640 1 -- start start 2023-10-14T18:17:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0039a0 0x7f4e701f3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e700c68d0 0x7f4e700ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e701f49e0 con 0x7f4e7014fee0 2023-10-14T18:17:12.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700c7030 con 0x7f4e5c0039a0 2023-10-14T18:17:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e700c85d0 con 0x7f4e700c68d0 2023-10-14T18:17:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.325+0000 7f4e6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33888/0 (socket says 172.21.15.29:33888) 2023-10-14T18:17:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 -- 172.21.15.29:0/1461329401 learned_addr learned my addr 172.21.15.29:0/1461329401 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:12.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6effd640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0039a0 0x7f4e701f3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e667fc640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e700c68d0 0x7f4e700ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 -- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e700c68d0 msgr2=0x7f4e700ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e700c68d0 0x7f4e700ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 -- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0039a0 msgr2=0x7f4e701f3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0039a0 0x7f4e701f3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 -- 172.21.15.29:0/1461329401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e701f4140 con 0x7f4e7014fee0 2023-10-14T18:17:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.326+0000 7f4e6f7fe640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f4e7015b700 tx=0x7f4e6800f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.327+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68017400 con 0x7f4e7014fee0 2023-10-14T18:17:12.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.327+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e68011040 con 0x7f4e7014fee0 2023-10-14T18:17:12.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.327+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e68017780 con 0x7f4e7014fee0 2023-10-14T18:17:12.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.327+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e701f4420 con 0x7f4e7014fee0 2023-10-14T18:17:12.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.327+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e70150360 con 0x7f4e7014fee0 2023-10-14T18:17:12.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.328+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e700c71a0 con 0x7f4e7014fee0 2023-10-14T18:17:12.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.329+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f4e680203b0 con 0x7f4e7014fee0 2023-10-14T18:17:12.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.330+0000 7f4e6cff9640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 0x7f4e5006d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.330+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(217..217 src has 1..217) v4 ==== 6599+0+0 (secure 0 0 0) 0x7f4e68089de0 con 0x7f4e7014fee0 2023-10-14T18:17:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.330+0000 7f4e6effd640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 0x7f4e5006d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.331+0000 7f4e6effd640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 0x7f4e5006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4e60033000 tx=0x7f4e60009030 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.335+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e680549a0 con 0x7f4e7014fee0 2023-10-14T18:17:12.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.575+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f4e700c6e00 con 0x7f4e7014fee0 2023-10-14T18:17:12.577 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:17:12.576+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:17:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.579+0000 7f4e6cff9640 1 -- 172.21.15.29:0/1461329401 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v99) v1 ==== 161+0+0 (secure 0 0 0) 0x7f4e68057fc0 con 0x7f4e7014fee0 2023-10-14T18:17:12.581 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:17:12.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 msgr2=0x7f4e5006d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 0x7f4e5006d3f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4e60033000 tx=0x7f4e60009030 comp rx=0 tx=0).stop 2023-10-14T18:17:12.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 msgr2=0x7f4e700c2c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 secure :-1 s=READY pgs=2509 cs=0 l=1 rev1=1 crypto rx=0x7f4e7015b700 tx=0x7f4e6800f4a0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 shutdown_connections 2023-10-14T18:17:12.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e5006af30 0x7f4e5006d3f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e7014fee0 0x7f4e700c2c80 unknown :-1 s=CLOSED pgs=2509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e700c68d0 0x7f4e700ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 --2- 172.21.15.29:0/1461329401 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e5c0039a0 0x7f4e701f3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 >> 172.21.15.29:0/1461329401 conn(0x7f4e700ce6a0 msgr2=0x7f4e70159710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:12.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 shutdown_connections 2023-10-14T18:17:12.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.583+0000 7f4e6ffff640 1 -- 172.21.15.29:0/1461329401 wait complete. 2023-10-14T18:17:12.613 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T18:17:12.662 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:17:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.817+0000 7fd4f57ef640 1 Processor -- start 2023-10-14T18:17:12.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.817+0000 7fd4f57ef640 1 -- start start 2023-10-14T18:17:12.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd4f006c040 0x7fd4f006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4f00ad970 0x7fd4f015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f015ebc0 0x7fd4f015f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f006c960 con 0x7fd4f00ad970 2023-10-14T18:17:12.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4f015b9d0 con 0x7fd4f006c040 2023-10-14T18:17:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4f57ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f015bcb0 con 0x7fd4f015ebc0 2023-10-14T18:17:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4f00ad970 0x7fd4f015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd4f00ad970 0x7fd4f015e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33908/0 (socket says 172.21.15.29:33908) 2023-10-14T18:17:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.819+0000 7fd4e7fff640 1 -- 172.21.15.29:0/14615615 learned_addr learned my addr 172.21.15.29:0/14615615 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:12.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4f4fee640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f015ebc0 0x7fd4f015f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4effff640 1 --2- 172.21.15.29:0/14615615 >> v2:172.21.15.186:3300/0 conn(0x7fd4f006c040 0x7fd4f006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4e7fff640 1 -- 172.21.15.29:0/14615615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f015ebc0 msgr2=0x7fd4f015f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:12.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4e7fff640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f015ebc0 0x7fd4f015f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4e7fff640 1 -- 172.21.15.29:0/14615615 >> v2:172.21.15.186:3300/0 conn(0x7fd4f006c040 msgr2=0x7fd4f006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:12.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4e7fff640 1 --2- 172.21.15.29:0/14615615 >> v2:172.21.15.186:3300/0 conn(0x7fd4f006c040 0x7fd4f006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.820+0000 7fd4e7fff640 1 -- 172.21.15.29:0/14615615 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4f015bf90 con 0x7fd4f00ad970 2023-10-14T18:17:12.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.821+0000 7fd4e7fff640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00ad970 0x7fd4f015e680 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00cf350 tx=0x7fd4d802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7ce1f3b7ba940d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.821+0000 7fd4ef7fe640 1 -- 172.21.15.29:0/14615615 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8031860 con 0x7fd4f00ad970 2023-10-14T18:17:12.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.821+0000 7fd4ef7fe640 1 -- 172.21.15.29:0/14615615 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d80319c0 con 0x7fd4f00ad970 2023-10-14T18:17:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.821+0000 7fd4ef7fe640 1 -- 172.21.15.29:0/14615615 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8031d50 con 0x7fd4f00ad970 2023-10-14T18:17:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 -- 172.21.15.29:0/14615615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00ad970 msgr2=0x7fd4f015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00ad970 0x7fd4f015e680 secure :-1 s=READY pgs=2513 cs=0 l=1 rev1=1 crypto rx=0x7fd4f00cf350 tx=0x7fd4d802b900 comp rx=0 tx=0).stop 2023-10-14T18:17:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 -- 172.21.15.29:0/14615615 shutdown_connections 2023-10-14T18:17:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f015ebc0 0x7fd4f015f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/14615615 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00ad970 0x7fd4f015e680 unknown :-1 s=CLOSED pgs=2513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/14615615 >> v2:172.21.15.186:3300/0 conn(0x7fd4f006c040 0x7fd4f006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 -- 172.21.15.29:0/14615615 >> 172.21.15.29:0/14615615 conn(0x7fd4f00ce6a0 msgr2=0x7fd4f00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 -- 172.21.15.29:0/14615615 shutdown_connections 2023-10-14T18:17:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.822+0000 7fd4f57ef640 1 -- 172.21.15.29:0/14615615 wait complete. 2023-10-14T18:17:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.824+0000 7fd4f57ef640 1 Processor -- start 2023-10-14T18:17:12.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.824+0000 7fd4f57ef640 1 -- start start 2023-10-14T18:17:12.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.825+0000 7fd4f57ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 0x7fd4f00c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.825+0000 7fd4f57ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 0x7fd4f00ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f57ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f57ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f0155990 con 0x7fd4f00aea90 2023-10-14T18:17:12.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f57ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00af260 con 0x7fd4f015ebc0 2023-10-14T18:17:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f57ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4f00af540 con 0x7fd4f006c040 2023-10-14T18:17:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33920/0 (socket says 172.21.15.29:33920) 2023-10-14T18:17:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 -- 172.21.15.29:0/1558102987 learned_addr learned my addr 172.21.15.29:0/1558102987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:12.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4effff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 0x7fd4f00c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 -- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 msgr2=0x7fd4f00c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 0x7fd4f00c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4e7fff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 0x7fd4f00ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 -- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 msgr2=0x7fd4f00ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 0x7fd4f00ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.826+0000 7fd4f4fee640 1 -- 172.21.15.29:0/1558102987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d80094d0 con 0x7fd4f00aea90 2023-10-14T18:17:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4effff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 0x7fd4f00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4e7fff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 0x7fd4f00ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:12.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4f4fee640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8007e60 tx=0x7fd4e800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8014420 con 0x7fd4f00aea90 2023-10-14T18:17:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4e80145a0 con 0x7fd4f00aea90 2023-10-14T18:17:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4e8014950 con 0x7fd4f00aea90 2023-10-14T18:17:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4f00af820 con 0x7fd4f00aea90 2023-10-14T18:17:12.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.827+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4f00afde0 con 0x7fd4f00aea90 2023-10-14T18:17:12.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.828+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4f006b330 con 0x7fd4f00aea90 2023-10-14T18:17:12.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.829+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 135) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fd4e8014ab0 con 0x7fd4f00aea90 2023-10-14T18:17:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.831+0000 7fd4ed7fa640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 0x7fd4d006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.831+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6657+0+0 (secure 0 0 0) 0x7fd4e808a440 con 0x7fd4f00aea90 2023-10-14T18:17:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.831+0000 7fd4effff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 0x7fd4d006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:12.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.832+0000 7fd4effff640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 0x7fd4d006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:12.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:12.835+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd4e8055040 con 0x7fd4f00aea90 2023-10-14T18:17:13.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.071+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd4f004e9b0 con 0x7fd4f00aea90 2023-10-14T18:17:13.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.583+0000 7fd4ed7fa640 1 -- 172.21.15.29:0/1558102987 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v100) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd4e8058660 con 0x7fd4f00aea90 2023-10-14T18:17:13.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 msgr2=0x7fd4d006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 0x7fd4d006d3c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fd4e0006fd0 tx=0x7fd4e0008040 comp rx=0 tx=0).stop 2023-10-14T18:17:13.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 msgr2=0x7fd4f00b0300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 secure :-1 s=READY pgs=2514 cs=0 l=1 rev1=1 crypto rx=0x7fd4e8007e60 tx=0x7fd4e800e4b0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 shutdown_connections 2023-10-14T18:17:13.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd4d006af00 0x7fd4d006d3c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd4f00aea90 0x7fd4f00b0300 unknown :-1 s=CLOSED pgs=2514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd4f015ebc0 0x7fd4f00ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 --2- 172.21.15.29:0/1558102987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd4f006c040 0x7fd4f00c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.586+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 >> 172.21.15.29:0/1558102987 conn(0x7fd4f00ce6a0 msgr2=0x7fd4f015a830 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:13.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.587+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 shutdown_connections 2023-10-14T18:17:13.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.587+0000 7fd4f57ef640 1 -- 172.21.15.29:0/1558102987 wait complete. 2023-10-14T18:17:13.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T18:17:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.886+0000 7f12730a2640 1 Processor -- start 2023-10-14T18:17:13.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.886+0000 7f12730a2640 1 -- start start 2023-10-14T18:17:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f12730a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f126c0d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f12730a2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c0d1420 con 0x7f126c0b8a10 2023-10-14T18:17:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f12730a2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f126c15d330 con 0x7f126c0b4750 2023-10-14T18:17:13.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f12730a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0d1680 con 0x7f126c15cf50 2023-10-14T18:17:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f126bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f126c0b8a10 0x7f126c0d0d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48058/0 (socket says 172.21.15.29:48058) 2023-10-14T18:17:13.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f126bfff640 1 -- 172.21.15.29:0/3859175583 learned_addr learned my addr 172.21.15.29:0/3859175583 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.888+0000 7f1270e3e640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f126c0d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 67101482 0 0) 0x7f126c0d1420 con 0x7f126c0b8a10 2023-10-14T18:17:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1258003520 con 0x7f126c0b8a10 2023-10-14T18:17:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 997000710 0 0) 0x7f1258003520 con 0x7f126c0b8a10 2023-10-14T18:17:13.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 msgr2=0x7f126c0d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f126c0d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 >> v1:172.21.15.186:6789/0 conn(0x7f126c0b4750 legacy=0x7f126c0b5e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c0d1960 con 0x7f126c0b8a10 2023-10-14T18:17:13.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f12540019f0 con 0x7f126c0b8a10 2023-10-14T18:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 >> v1:172.21.15.29:6789/0 conn(0x7f126c0b8a10 legacy=0x7f126c0d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f1258003a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1258003f90 0x7f12580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1258004d00 0x7f1258005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0d1680 con 0x7f1258003f90 2023-10-14T18:17:13.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f125800bae0 con 0x7f1258004d00 2023-10-14T18:17:13.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f125800bca0 con 0x7f126c15cf50 2023-10-14T18:17:13.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.889+0000 7f126bfff640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1258003f90 0x7f12580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f1270e3e640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f1258003a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f126bfff640 1 -- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 msgr2=0x7f1258003a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:13.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f127163f640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1258004d00 0x7f1258005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f126bfff640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f1258003a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f126bfff640 1 -- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1258004d00 msgr2=0x7f1258005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:13.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f126bfff640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1258004d00 0x7f1258005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f126bfff640 1 -- 172.21.15.29:0/3859175583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1258003520 con 0x7f1258003f90 2023-10-14T18:17:13.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.890+0000 7f1270e3e640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f126c15cf50 0x7f1258003a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:13.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.891+0000 7f126bfff640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1258003f90 0x7f12580047c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f125400a830 tx=0x7f12580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edccf53f9ec228ea server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.891+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540072d0 con 0x7f1258003f90 2023-10-14T18:17:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.891+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f125402f070 con 0x7f1258003f90 2023-10-14T18:17:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.891+0000 7f126b7fe640 1 -- 172.21.15.29:0/3859175583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12540076f0 con 0x7f1258003f90 2023-10-14T18:17:13.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 -- 172.21.15.29:0/3859175583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1258003f90 msgr2=0x7f12580047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 --2- 172.21.15.29:0/3859175583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1258003f90 0x7f12580047c0 secure :-1 s=READY pgs=2516 cs=0 l=1 rev1=1 crypto rx=0x7f125400a830 tx=0x7f12580057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f127163f640 1 -- 172.21.15.29:0/3859175583 reap_dead start 2023-10-14T18:17:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 -- 172.21.15.29:0/3859175583 shutdown_connections 2023-10-14T18:17:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 -- 172.21.15.29:0/3859175583 >> 172.21.15.29:0/3859175583 conn(0x7f126c0cf750 msgr2=0x7f126c0cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:13.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 -- 172.21.15.29:0/3859175583 shutdown_connections 2023-10-14T18:17:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.892+0000 7f12730a2640 1 -- 172.21.15.29:0/3859175583 wait complete. 2023-10-14T18:17:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.894+0000 7f12730a2640 1 Processor -- start 2023-10-14T18:17:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.894+0000 7f12730a2640 1 -- start start 2023-10-14T18:17:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1258003f90 0x7f126c0c8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c154ff0 0x7f126c0ca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c156160 con 0x7f126c154b70 2023-10-14T18:17:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f126c0c94a0 con 0x7f126c154ff0 2023-10-14T18:17:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f12730a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f126c155550 con 0x7f1258003f90 2023-10-14T18:17:13.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f126bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f126bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33940/0 (socket says 172.21.15.29:33940) 2023-10-14T18:17:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.895+0000 7f126bfff640 1 -- 172.21.15.29:0/619999038 learned_addr learned my addr 172.21.15.29:0/619999038 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f1270e3e640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1258003f90 0x7f126c0c8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f127163f640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c154ff0 0x7f126c0ca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 -- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1258003f90 msgr2=0x7f126c0c8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1258003f90 0x7f126c0c8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 -- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c154ff0 msgr2=0x7f126c0ca2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c154ff0 0x7f126c0ca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:13.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 -- 172.21.15.29:0/619999038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f126c155830 con 0x7f126c154b70 2023-10-14T18:17:13.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f126bfff640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f125402cab0 tx=0x7f1254001d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:13.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254004d60 con 0x7f126c154b70 2023-10-14T18:17:13.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12540070d0 con 0x7f126c154b70 2023-10-14T18:17:13.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1254007440 con 0x7f126c154b70 2023-10-14T18:17:13.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.896+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f126c155b10 con 0x7f126c154b70 2023-10-14T18:17:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.897+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f126c0cca70 con 0x7f126c154b70 2023-10-14T18:17:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.897+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f126c159840 con 0x7f126c154b70 2023-10-14T18:17:13.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.899+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1254031030 con 0x7f126c154b70 2023-10-14T18:17:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.900+0000 7f12697fa640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 0x7f124c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.900+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(218..218 src has 1..218) v4 ==== 6657+0+0 (secure 0 0 0) 0x7f12540abbb0 con 0x7f126c154b70 2023-10-14T18:17:13.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.900+0000 7f1270e3e640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 0x7f124c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.901+0000 7f1270e3e640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 0x7f124c06d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f125c0099c0 tx=0x7f125c008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:13.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:13.905+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1254076710 con 0x7f126c154b70 2023-10-14T18:17:14.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.095+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f126c159840 con 0x7f126c154b70 2023-10-14T18:17:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.597+0000 7f12697fa640 1 -- 172.21.15.29:0/619999038 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v219) v1 ==== 181+0+0 (secure 0 0 0) 0x7f1254079d30 con 0x7f126c154b70 2023-10-14T18:17:14.600 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T18:17:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 msgr2=0x7f124c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:14.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 0x7f124c06d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f125c0099c0 tx=0x7f125c008040 comp rx=0 tx=0).stop 2023-10-14T18:17:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 msgr2=0x7f126c0c9da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 secure :-1 s=READY pgs=2517 cs=0 l=1 rev1=1 crypto rx=0x7f125402cab0 tx=0x7f1254001d80 comp rx=0 tx=0).stop 2023-10-14T18:17:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 shutdown_connections 2023-10-14T18:17:14.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f124c06afc0 0x7f124c06d480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f126c154ff0 0x7f126c0ca2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f126c154b70 0x7f126c0c9da0 unknown :-1 s=CLOSED pgs=2517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 --2- 172.21.15.29:0/619999038 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1258003f90 0x7f126c0c8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.602+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 >> 172.21.15.29:0/619999038 conn(0x7f126c0cf750 msgr2=0x7f126c15be70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:14.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.603+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 shutdown_connections 2023-10-14T18:17:14.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.603+0000 7f12730a2640 1 -- 172.21.15.29:0/619999038 wait complete. 2023-10-14T18:17:14.626 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T18:17:14.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.855+0000 7f5198a94640 1 Processor -- start 2023-10-14T18:17:14.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.856+0000 7f5198a94640 1 -- start start 2023-10-14T18:17:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.857+0000 7f5198a94640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f519406c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.857+0000 7f5198a94640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51940cac20 0x7f51940b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.857+0000 7f5198a94640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51940b0d00 con 0x7f51940b0360 2023-10-14T18:17:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.857+0000 7f5198a94640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f519415f800 con 0x7f51940cac20 2023-10-14T18:17:14.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.857+0000 7f5198a94640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f519415b310 con 0x7f519415ac10 2023-10-14T18:17:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51940cac20 0x7f51940b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51940cac20 0x7f51940b07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33950/0 (socket says 172.21.15.29:33950) 2023-10-14T18:17:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 -- 172.21.15.29:0/2559728522 learned_addr learned my addr 172.21.15.29:0/2559728522 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:14.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f519259c640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f519406c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 -- 172.21.15.29:0/2559728522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 msgr2=0x7f519406c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f519406c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 -- 172.21.15.29:0/2559728522 >> v1:172.21.15.186:6789/0 conn(0x7f51940b0360 legacy=0x7f51940ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:14.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f5192d9d640 1 -- 172.21.15.29:0/2559728522 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f519406f640 con 0x7f51940cac20 2023-10-14T18:17:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.858+0000 7f519259c640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f519406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:14.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.859+0000 7f5192d9d640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f51940b07c0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f518c009870 tx=0x7f518c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b990e73eca4398d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.860+0000 7f519159a640 1 -- 172.21.15.29:0/2559728522 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c0317c0 con 0x7f51940cac20 2023-10-14T18:17:14.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.860+0000 7f519159a640 1 -- 172.21.15.29:0/2559728522 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f518c031920 con 0x7f51940cac20 2023-10-14T18:17:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.860+0000 7f519159a640 1 -- 172.21.15.29:0/2559728522 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f518c031cb0 con 0x7f51940cac20 2023-10-14T18:17:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.860+0000 7f5198a94640 1 -- 172.21.15.29:0/2559728522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 msgr2=0x7f51940b07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.860+0000 7f5198a94640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f51940b07c0 secure :-1 s=READY pgs=2518 cs=0 l=1 rev1=1 crypto rx=0x7f518c009870 tx=0x7f518c02b860 comp rx=0 tx=0).stop 2023-10-14T18:17:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 -- 172.21.15.29:0/2559728522 shutdown_connections 2023-10-14T18:17:14.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f51940b07c0 unknown :-1 s=CLOSED pgs=2518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 --2- 172.21.15.29:0/2559728522 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f519406c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 -- 172.21.15.29:0/2559728522 >> 172.21.15.29:0/2559728522 conn(0x7f51940cb360 msgr2=0x7f51940cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 -- 172.21.15.29:0/2559728522 shutdown_connections 2023-10-14T18:17:14.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.861+0000 7f5198a94640 1 -- 172.21.15.29:0/2559728522 wait complete. 2023-10-14T18:17:14.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.862+0000 7f5198a94640 1 Processor -- start 2023-10-14T18:17:14.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.863+0000 7f5198a94640 1 -- start start 2023-10-14T18:17:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 0x7f51940b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f51940b1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940b3180 con 0x7f51940cac20 2023-10-14T18:17:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51940b2570 con 0x7f51940b0360 2023-10-14T18:17:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5198a94640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51940b2850 con 0x7f519415ac10 2023-10-14T18:17:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33964/0 (socket says 172.21.15.29:33964) 2023-10-14T18:17:14.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 -- 172.21.15.29:0/662260218 learned_addr learned my addr 172.21.15.29:0/662260218 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f519259c640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 0x7f51940b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5192d9d640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f51940b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 -- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 msgr2=0x7f51940b1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:14.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f51940b1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 -- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 msgr2=0x7f51940b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.864+0000 7f5191d9b640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 0x7f51940b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5191d9b640 1 -- 172.21.15.29:0/662260218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f518c0094d0 con 0x7f51940cac20 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f519259c640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 0x7f51940b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5192d9d640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f51940b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:14.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5191d9b640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f518800b3f0 tx=0x7f518800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188011830 con 0x7f51940cac20 2023-10-14T18:17:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5188011990 con 0x7f51940cac20 2023-10-14T18:17:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5188011d40 con 0x7f51940cac20 2023-10-14T18:17:14.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51940b2b30 con 0x7f51940cac20 2023-10-14T18:17:14.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.865+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51940b1d80 con 0x7f51940cac20 2023-10-14T18:17:14.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.866+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f519404e160 con 0x7f51940cac20 2023-10-14T18:17:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.868+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f518801a400 con 0x7f51940cac20 2023-10-14T18:17:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.869+0000 7f5182ffd640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 0x7f517406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.869+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(219..219 src has 1..219) v4 ==== 6219+0+0 (secure 0 0 0) 0x7f5188089880 con 0x7f51940cac20 2023-10-14T18:17:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.869+0000 7f519259c640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 0x7f517406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:14.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.870+0000 7f519259c640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 0x7f517406d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5184007f40 tx=0x7f5184007490 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:14.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:14.874+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5188054580 con 0x7f51940cac20 2023-10-14T18:17:15.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.085+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f51940c9090 con 0x7f51940cac20 2023-10-14T18:17:15.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.601+0000 7f5182ffd640 1 -- 172.21.15.29:0/662260218 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v220) v1 ==== 169+0+0 (secure 0 0 0) 0x7f5188057ba0 con 0x7f51940cac20 2023-10-14T18:17:15.604 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T18:17:15.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 msgr2=0x7f517406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 0x7f517406d480 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5184007f40 tx=0x7f5184007490 comp rx=0 tx=0).stop 2023-10-14T18:17:15.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 msgr2=0x7f5194156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 secure :-1 s=READY pgs=2519 cs=0 l=1 rev1=1 crypto rx=0x7f518800b3f0 tx=0x7f518800b8b0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 shutdown_connections 2023-10-14T18:17:15.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f517406afc0 0x7f517406d480 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f519415ac10 0x7f51940b1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51940cac20 0x7f5194156110 unknown :-1 s=CLOSED pgs=2519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 --2- 172.21.15.29:0/662260218 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51940b0360 0x7f51940b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 >> 172.21.15.29:0/662260218 conn(0x7f51940cb360 msgr2=0x7f519406b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:15.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 shutdown_connections 2023-10-14T18:17:15.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.605+0000 7f5198a94640 1 -- 172.21.15.29:0/662260218 wait complete. 2023-10-14T18:17:15.629 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T18:17:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.855+0000 7f7480cbb640 1 Processor -- start 2023-10-14T18:17:15.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.855+0000 7f7480cbb640 1 -- start start 2023-10-14T18:17:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7480cbb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f747c159f00 0x7f747c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7480cbb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 0x7f747c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7480cbb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f747c06f890 con 0x7f747c159b20 2023-10-14T18:17:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7480cbb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c070d30 con 0x7f747c159f00 2023-10-14T18:17:15.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7480cbb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f747c06c650 con 0x7f747c15d8c0 2023-10-14T18:17:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7473fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f747c159f00 0x7f747c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7473fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f747c159f00 0x7f747c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33968/0 (socket says 172.21.15.29:33968) 2023-10-14T18:17:15.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.857+0000 7f7473fff640 1 -- 172.21.15.29:0/2504971132 learned_addr learned my addr 172.21.15.29:0/2504971132 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f747ad9d640 1 --2- 172.21.15.29:0/2504971132 >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 0x7f747c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f7473fff640 1 -- 172.21.15.29:0/2504971132 >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 msgr2=0x7f747c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f7473fff640 1 --2- 172.21.15.29:0/2504971132 >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 0x7f747c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f7473fff640 1 -- 172.21.15.29:0/2504971132 >> v1:172.21.15.29:6789/0 conn(0x7f747c159b20 legacy=0x7f747c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f7473fff640 1 -- 172.21.15.29:0/2504971132 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c06c930 con 0x7f747c159f00 2023-10-14T18:17:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.858+0000 7f747ad9d640 1 --2- 172.21.15.29:0/2504971132 >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 0x7f747c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.859+0000 7f7473fff640 1 --2- 172.21.15.29:0/2504971132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c0b0410 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f747c15d750 tx=0x7f746402b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4749cb20de47a89d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.859+0000 7f7479d9b640 1 -- 172.21.15.29:0/2504971132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464030870 con 0x7f747c159f00 2023-10-14T18:17:15.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.859+0000 7f7479d9b640 1 -- 172.21.15.29:0/2504971132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74640309d0 con 0x7f747c159f00 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.859+0000 7f7479d9b640 1 -- 172.21.15.29:0/2504971132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464030d60 con 0x7f747c159f00 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 -- 172.21.15.29:0/2504971132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 msgr2=0x7f747c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 --2- 172.21.15.29:0/2504971132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c0b0410 secure :-1 s=READY pgs=2521 cs=0 l=1 rev1=1 crypto rx=0x7f747c15d750 tx=0x7f746402b9b0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 -- 172.21.15.29:0/2504971132 shutdown_connections 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 --2- 172.21.15.29:0/2504971132 >> v2:172.21.15.186:3300/0 conn(0x7f747c15d8c0 0x7f747c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 --2- 172.21.15.29:0/2504971132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c0b0410 unknown :-1 s=CLOSED pgs=2521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 -- 172.21.15.29:0/2504971132 >> 172.21.15.29:0/2504971132 conn(0x7f747c0cb410 msgr2=0x7f747c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 -- 172.21.15.29:0/2504971132 shutdown_connections 2023-10-14T18:17:15.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.860+0000 7f7480cbb640 1 -- 172.21.15.29:0/2504971132 wait complete. 2023-10-14T18:17:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.862+0000 7f7480cbb640 1 Processor -- start 2023-10-14T18:17:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.862+0000 7f7480cbb640 1 -- start start 2023-10-14T18:17:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f747c159b20 0x7f747c1f07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747c15d8c0 0x7f747c0c8490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c15b460 con 0x7f747c159f00 2023-10-14T18:17:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c1f1730 con 0x7f747c15d8c0 2023-10-14T18:17:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7480cbb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f747c0c89d0 con 0x7f747c159b20 2023-10-14T18:17:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.863+0000 7f7473fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:15.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33970/0 (socket says 172.21.15.29:33970) 2023-10-14T18:17:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 -- 172.21.15.29:0/842185143 learned_addr learned my addr 172.21.15.29:0/842185143 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 -- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f747c159b20 msgr2=0x7f747c1f07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f747c159b20 0x7f747c1f07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 -- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747c15d8c0 msgr2=0x7f747c0c8490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:15.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f747ad9d640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747c15d8c0 0x7f747c0c8490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747c15d8c0 0x7f747c0c8490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 -- 172.21.15.29:0/842185143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74640094d0 con 0x7f747c159f00 2023-10-14T18:17:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.864+0000 7f7473fff640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f7464000c00 tx=0x7f7464004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.865+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464030860 con 0x7f747c159f00 2023-10-14T18:17:15.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.865+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7464044480 con 0x7f747c159f00 2023-10-14T18:17:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.865+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c1f0e90 con 0x7f747c159f00 2023-10-14T18:17:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.865+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7464044830 con 0x7f747c159f00 2023-10-14T18:17:15.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.865+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c1f1450 con 0x7f747c159f00 2023-10-14T18:17:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.866+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c04e160 con 0x7f747c159f00 2023-10-14T18:17:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.867+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f746402f3c0 con 0x7f747c159f00 2023-10-14T18:17:15.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.868+0000 7f7472ffd640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 0x7f745c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.868+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f746403e070 con 0x7f747c159f00 2023-10-14T18:17:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.868+0000 7f747a59c640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 0x7f745c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:15.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.869+0000 7f747a59c640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 0x7f745c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f746c000f60 tx=0x7f746c0079b0 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:15.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:15.873+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7464077540 con 0x7f747c159f00 2023-10-14T18:17:16.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.069+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f747c0b10b0 con 0x7f747c159f00 2023-10-14T18:17:16.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.074+0000 7f7472ffd640 1 -- 172.21.15.29:0/842185143 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v73) v1 ==== 115+0+0 (secure 0 0 0) 0x7f746407ab60 con 0x7f747c159f00 2023-10-14T18:17:16.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 msgr2=0x7f745c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 0x7f745c06d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f746c000f60 tx=0x7f746c0079b0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 msgr2=0x7f747c157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 secure :-1 s=READY pgs=2522 cs=0 l=1 rev1=1 crypto rx=0x7f7464000c00 tx=0x7f7464004730 comp rx=0 tx=0).stop 2023-10-14T18:17:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 shutdown_connections 2023-10-14T18:17:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f745c06af90 0x7f745c06d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f747c15d8c0 0x7f747c0c8490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f747c159f00 0x7f747c157a70 unknown :-1 s=CLOSED pgs=2522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 --2- 172.21.15.29:0/842185143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f747c159b20 0x7f747c1f07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 >> 172.21.15.29:0/842185143 conn(0x7f747c0cb410 msgr2=0x7f747c0cdb40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:16.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.077+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 shutdown_connections 2023-10-14T18:17:16.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.078+0000 7f7480cbb640 1 -- 172.21.15.29:0/842185143 wait complete. 2023-10-14T18:17:16.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T18:17:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.316+0000 7f337b3b3640 1 Processor -- start 2023-10-14T18:17:16.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.316+0000 7f337b3b3640 1 -- start start 2023-10-14T18:17:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337b3b3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f337415dde0 0x7f33740709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337b3b3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3374160a90 con 0x7f337415c9f0 2023-10-14T18:17:16.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337b3b3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33740cd900 con 0x7f3374070f20 2023-10-14T18:17:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337b3b3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3374071350 con 0x7f337415dde0 2023-10-14T18:17:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337a3b1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f337415c9f0 0x7f33740c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48066/0 (socket says 172.21.15.29:48066) 2023-10-14T18:17:16.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.318+0000 7f337a3b1640 1 -- 172.21.15.29:0/2037574187 learned_addr learned my addr 172.21.15.29:0/2037574187 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:16.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515009336 0 0) 0x7f3374160a90 con 0x7f337415c9f0 2023-10-14T18:17:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> v2:172.21.15.186:3300/0 conn(0x7f337415dde0 0x7f33740709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3360003520 con 0x7f337415c9f0 2023-10-14T18:17:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882033747 0 0) 0x7f3360003520 con 0x7f337415c9f0 2023-10-14T18:17:16.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 >> v2:172.21.15.186:3300/0 conn(0x7f337415dde0 msgr2=0x7f33740709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 --2- 172.21.15.29:0/2037574187 >> v2:172.21.15.186:3300/0 conn(0x7f337415dde0 0x7f33740709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 >> v1:172.21.15.186:6789/0 conn(0x7f3374070f20 legacy=0x7f33740ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33740cdbe0 con 0x7f337415c9f0 2023-10-14T18:17:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3370001670 con 0x7f337415c9f0 2023-10-14T18:17:16.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> v2:172.21.15.186:3300/0 conn(0x7f337415dde0 0x7f33740709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 >> v1:172.21.15.29:6789/0 conn(0x7f337415c9f0 legacy=0x7f33740c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360003970 0x7f3360003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 0x7f3360004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3360005180 0x7f336000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3374071350 con 0x7f33600047c0 2023-10-14T18:17:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336000c0b0 con 0x7f3360003970 2023-10-14T18:17:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.319+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336000c270 con 0x7f3360005180 2023-10-14T18:17:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 0x7f3360004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f337a3b1640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360003970 0x7f3360003d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f337abb2640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3360005180 0x7f336000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 -- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3360005180 msgr2=0x7f336000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3360005180 0x7f336000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 -- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360003970 msgr2=0x7f3360003d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3360003970 0x7f3360003d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f3379bb0640 1 -- 172.21.15.29:0/2037574187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3360003520 con 0x7f33600047c0 2023-10-14T18:17:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f337abb2640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3360005180 0x7f336000b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.320+0000 7f337abb2640 1 -- 172.21.15.29:0/2037574187 reap_dead start 2023-10-14T18:17:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.321+0000 7f3379bb0640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 0x7f3360004c40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f3364006fd0 tx=0x7f33600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae3ed65105d82656 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.322+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364004230 con 0x7f33600047c0 2023-10-14T18:17:16.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.322+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3364004390 con 0x7f33600047c0 2023-10-14T18:17:16.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.322+0000 7f33793af640 1 -- 172.21.15.29:0/2037574187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3364003830 con 0x7f33600047c0 2023-10-14T18:17:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.322+0000 7f337b3b3640 1 -- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 msgr2=0x7f3360004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.322+0000 7f337b3b3640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 0x7f3360004c40 secure :-1 s=READY pgs=2524 cs=0 l=1 rev1=1 crypto rx=0x7f3364006fd0 tx=0x7f33600057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.323+0000 7f337b3b3640 1 -- 172.21.15.29:0/2037574187 shutdown_connections 2023-10-14T18:17:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.323+0000 7f337b3b3640 1 --2- 172.21.15.29:0/2037574187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33600047c0 0x7f3360004c40 unknown :-1 s=CLOSED pgs=2524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.323+0000 7f337b3b3640 1 -- 172.21.15.29:0/2037574187 >> 172.21.15.29:0/2037574187 conn(0x7f33740ce6a0 msgr2=0x7f33740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.323+0000 7f337b3b3640 1 -- 172.21.15.29:0/2037574187 shutdown_connections 2023-10-14T18:17:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.323+0000 7f337b3b3640 1 -- 172.21.15.29:0/2037574187 wait complete. 2023-10-14T18:17:16.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.324+0000 7f337b3b3640 1 Processor -- start 2023-10-14T18:17:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 -- start start 2023-10-14T18:17:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33600047c0 0x7f337406c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33741f2bf0 0x7f33741505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33740c84f0 con 0x7f33740c61d0 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33741f30a0 con 0x7f33600047c0 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.325+0000 7f337b3b3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33741f3330 con 0x7f33741f2bf0 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33988/0 (socket says 172.21.15.29:33988) 2023-10-14T18:17:16.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 -- 172.21.15.29:0/615020588 learned_addr learned my addr 172.21.15.29:0/615020588 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f337abb2640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33741f2bf0 0x7f33741505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f337a3b1640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33600047c0 0x7f337406c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 -- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33741f2bf0 msgr2=0x7f33741505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33741f2bf0 0x7f33741505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 -- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33600047c0 msgr2=0x7f337406c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33600047c0 0x7f337406c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.326+0000 7f3379bb0640 1 -- 172.21.15.29:0/615020588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33741f3610 con 0x7f33740c61d0 2023-10-14T18:17:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f3379bb0640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f336402be60 tx=0x7f3364004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336403a670 con 0x7f33740c61d0 2023-10-14T18:17:16.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336403a7d0 con 0x7f33740c61d0 2023-10-14T18:17:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3374157eb0 con 0x7f33740c61d0 2023-10-14T18:17:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f336403ab40 con 0x7f33740c61d0 2023-10-14T18:17:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.327+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3374158470 con 0x7f33740c61d0 2023-10-14T18:17:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.328+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f337404e9b0 con 0x7f33740c61d0 2023-10-14T18:17:16.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.330+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f336403aca0 con 0x7f33740c61d0 2023-10-14T18:17:16.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.331+0000 7f336affd640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 0x7f335806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.331+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f33640ac540 con 0x7f33740c61d0 2023-10-14T18:17:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.331+0000 7f337a3b1640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 0x7f335806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.332+0000 7f337a3b1640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 0x7f335806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3370001eb0 tx=0x7f3370001d80 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.336+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3364077310 con 0x7f33740c61d0 2023-10-14T18:17:16.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.534+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f33740adac0 con 0x7f33740c61d0 2023-10-14T18:17:16.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.541+0000 7f336affd640 1 -- 172.21.15.29:0/615020588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v74) v1 ==== 120+0+0 (secure 0 0 0) 0x7f33640aae60 con 0x7f33740c61d0 2023-10-14T18:17:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 msgr2=0x7f335806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 0x7f335806d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3370001eb0 tx=0x7f3370001d80 comp rx=0 tx=0).stop 2023-10-14T18:17:16.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 msgr2=0x7f33741f26b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 secure :-1 s=READY pgs=2525 cs=0 l=1 rev1=1 crypto rx=0x7f336402be60 tx=0x7f3364004730 comp rx=0 tx=0).stop 2023-10-14T18:17:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 shutdown_connections 2023-10-14T18:17:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f335806af90 0x7f335806d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f33741f2bf0 0x7f33741505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33740c61d0 0x7f33741f26b0 unknown :-1 s=CLOSED pgs=2525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 --2- 172.21.15.29:0/615020588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33600047c0 0x7f337406c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 >> 172.21.15.29:0/615020588 conn(0x7f33740ce6a0 msgr2=0x7f33740cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 shutdown_connections 2023-10-14T18:17:16.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.545+0000 7f337b3b3640 1 -- 172.21.15.29:0/615020588 wait complete. 2023-10-14T18:17:16.566 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T18:17:16.736 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:17:16.746 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:17:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.768+0000 7fe3f5e90640 1 Processor -- start 2023-10-14T18:17:16.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.768+0000 7fe3f5e90640 1 -- start start 2023-10-14T18:17:16.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.769+0000 7fe3f5e90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe3f015d410 0x7fe3f00d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.769+0000 7fe3f5e90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 0x7fe3f00cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.769+0000 7fe3f5e90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3f00b5d90 con 0x7fe3f015d030 2023-10-14T18:17:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.769+0000 7fe3f5e90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3f00d2b50 con 0x7fe3f015d410 2023-10-14T18:17:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.769+0000 7fe3f5e90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00cdbf0 con 0x7fe3f00cd260 2023-10-14T18:17:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.770+0000 7fe3ef7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe3f015d030 0x7fe3f0161450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:42752/0 (socket says 172.21.15.29:42752) 2023-10-14T18:17:16.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.770+0000 7fe3ef7fe640 1 -- 172.21.15.29:0/129970812 learned_addr learned my addr 172.21.15.29:0/129970812 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.770+0000 7fe3eeffd640 1 --2- 172.21.15.29:0/129970812 >> v2:172.21.15.186:3300/0 conn(0x7fe3f015d410 0x7fe3f00d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.770+0000 7fe3effff640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 0x7fe3f00cd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3eeffd640 1 -- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 msgr2=0x7fe3f00cd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:16.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3eeffd640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 0x7fe3f00cd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3eeffd640 1 -- 172.21.15.29:0/129970812 >> v1:172.21.15.186:6789/0 conn(0x7fe3f015d030 legacy=0x7fe3f0161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3eeffd640 1 -- 172.21.15.29:0/129970812 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3f0160fc0 con 0x7fe3f015d410 2023-10-14T18:17:16.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3ee7fc640 1 -- 172.21.15.29:0/129970812 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3646790769 0 0) 0x7fe3f00b5d90 con 0x7fe3f015d030 2023-10-14T18:17:16.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.771+0000 7fe3effff640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 0x7fe3f00cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:17:16.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.772+0000 7fe3eeffd640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f015d410 0x7fe3f00d1630 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40097a0 tx=0x7fe3e402b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88540115e4d8e9be server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.772+0000 7fe3ee7fc640 1 -- 172.21.15.29:0/129970812 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4030800 con 0x7fe3f015d410 2023-10-14T18:17:16.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.772+0000 7fe3ee7fc640 1 -- 172.21.15.29:0/129970812 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e4030960 con 0x7fe3f015d410 2023-10-14T18:17:16.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.772+0000 7fe3ee7fc640 1 -- 172.21.15.29:0/129970812 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e4030cf0 con 0x7fe3f015d410 2023-10-14T18:17:16.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 -- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f015d410 msgr2=0x7fe3f00d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f015d410 0x7fe3f00d1630 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fe3e40097a0 tx=0x7fe3e402b940 comp rx=0 tx=0).stop 2023-10-14T18:17:16.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 -- 172.21.15.29:0/129970812 shutdown_connections 2023-10-14T18:17:16.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f00cd260 0x7fe3f00cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/129970812 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f015d410 0x7fe3f00d1630 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 -- 172.21.15.29:0/129970812 >> 172.21.15.29:0/129970812 conn(0x7fe3f00cf720 msgr2=0x7fe3f00cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:16.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 -- 172.21.15.29:0/129970812 shutdown_connections 2023-10-14T18:17:16.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.773+0000 7fe3f5e90640 1 -- 172.21.15.29:0/129970812 wait complete. 2023-10-14T18:17:16.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.775+0000 7fe3f5e90640 1 Processor -- start 2023-10-14T18:17:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.775+0000 7fe3f5e90640 1 -- start start 2023-10-14T18:17:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f00cd260 0x7fe3f01523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f015d030 0x7fe3f00c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00c9480 con 0x7fe3f015d410 2023-10-14T18:17:16.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00c83e0 con 0x7fe3f00cd260 2023-10-14T18:17:16.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.776+0000 7fe3f5e90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3f00c86c0 con 0x7fe3f015d030 2023-10-14T18:17:16.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33992/0 (socket says 172.21.15.29:33992) 2023-10-14T18:17:16.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 -- 172.21.15.29:0/3561711757 learned_addr learned my addr 172.21.15.29:0/3561711757 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:16.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3ef7fe640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f00cd260 0x7fe3f01523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3eeffd640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f015d030 0x7fe3f00c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 -- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f015d030 msgr2=0x7fe3f00c7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f015d030 0x7fe3f00c7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 -- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f00cd260 msgr2=0x7fe3f01523b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f00cd260 0x7fe3f01523b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.777+0000 7fe3effff640 1 -- 172.21.15.29:0/3561711757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e8009a70 con 0x7fe3f015d410 2023-10-14T18:17:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3effff640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800efc0 tx=0x7fe3e800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8009cd0 con 0x7fe3f015d410 2023-10-14T18:17:16.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3e800f040 con 0x7fe3f015d410 2023-10-14T18:17:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8003990 con 0x7fe3f015d410 2023-10-14T18:17:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e4009400 con 0x7fe3f015d410 2023-10-14T18:17:16.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.778+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3f00c8c80 con 0x7fe3f015d410 2023-10-14T18:17:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.780+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3f004e160 con 0x7fe3f015d410 2023-10-14T18:17:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.780+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe3e8014480 con 0x7fe3f015d410 2023-10-14T18:17:16.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.781+0000 7fe3d7fff640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 0x7fe3dc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.781+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fe3e808a430 con 0x7fe3f015d410 2023-10-14T18:17:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.782+0000 7fe3ef7fe640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 0x7fe3dc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:16.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.782+0000 7fe3ef7fe640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 0x7fe3dc06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0000f60 tx=0x7fe3e0009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:16.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.786+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe3e8055320 con 0x7fe3f015d410 2023-10-14T18:17:16.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.982+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fe3f00d1760 con 0x7fe3f015d410 2023-10-14T18:17:16.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.988+0000 7fe3d7fff640 1 -- 172.21.15.29:0/3561711757 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v75) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe3e8058940 con 0x7fe3f015d410 2023-10-14T18:17:16.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 msgr2=0x7fe3dc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 0x7fe3dc06d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe3e0000f60 tx=0x7fe3e0009040 comp rx=0 tx=0).stop 2023-10-14T18:17:16.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 msgr2=0x7fe3f00b7660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:16.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 secure :-1 s=READY pgs=2526 cs=0 l=1 rev1=1 crypto rx=0x7fe3e800efc0 tx=0x7fe3e800c490 comp rx=0 tx=0).stop 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 shutdown_connections 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe3dc06af90 0x7fe3dc06d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe3f015d410 0x7fe3f00b7660 unknown :-1 s=CLOSED pgs=2526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe3f015d030 0x7fe3f00c7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 --2- 172.21.15.29:0/3561711757 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe3f00cd260 0x7fe3f01523b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:16.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 >> 172.21.15.29:0/3561711757 conn(0x7fe3f00cf720 msgr2=0x7fe3f00b36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:16.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 shutdown_connections 2023-10-14T18:17:16.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:16.992+0000 7fe3f5e90640 1 -- 172.21.15.29:0/3561711757 wait complete. 2023-10-14T18:17:17.012 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention' 2023-10-14T18:17:17.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.223+0000 7f4786f5a640 1 Processor -- start 2023-10-14T18:17:17.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.224+0000 7f4786f5a640 1 -- start start 2023-10-14T18:17:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.225+0000 7f4786f5a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47800b0b20 0x7f47801615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.225+0000 7f4786f5a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f478015e3e0 con 0x7f478015d3f0 2023-10-14T18:17:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.225+0000 7f4786f5a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47800cd4a0 con 0x7f47800cd0c0 2023-10-14T18:17:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.225+0000 7f4786f5a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f478015e100 con 0x7f47800b0b20 2023-10-14T18:17:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f477ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47800b0b20 0x7f47801615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f47854f7640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f478015d3f0 0x7f478015d820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48076/0 (socket says 172.21.15.29:48076) 2023-10-14T18:17:17.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f47854f7640 1 -- 172.21.15.29:0/2275059688 learned_addr learned my addr 172.21.15.29:0/2275059688 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:17.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f477ffff640 1 -- 172.21.15.29:0/2275059688 >> v1:172.21.15.186:6789/0 conn(0x7f47800cd0c0 legacy=0x7f47800b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f477ffff640 1 -- 172.21.15.29:0/2275059688 >> v1:172.21.15.29:6789/0 conn(0x7f478015d3f0 legacy=0x7f478015d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f477ffff640 1 -- 172.21.15.29:0/2275059688 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4780160b10 con 0x7f47800b0b20 2023-10-14T18:17:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.226+0000 7f477f7fe640 1 -- 172.21.15.29:0/2275059688 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616638047 0 0) 0x7f478015e3e0 con 0x7f478015d3f0 2023-10-14T18:17:17.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.227+0000 7f477ffff640 1 --2- 172.21.15.29:0/2275059688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800b0b20 0x7f47801615f0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097a0 tx=0x7f476c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2819fdc6e460dcf5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.227+0000 7f477f7fe640 1 -- 172.21.15.29:0/2275059688 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c02e070 con 0x7f47800b0b20 2023-10-14T18:17:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.227+0000 7f477f7fe640 1 -- 172.21.15.29:0/2275059688 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c0315a0 con 0x7f47800b0b20 2023-10-14T18:17:17.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.227+0000 7f477f7fe640 1 -- 172.21.15.29:0/2275059688 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c031930 con 0x7f47800b0b20 2023-10-14T18:17:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 -- 172.21.15.29:0/2275059688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800b0b20 msgr2=0x7f47801615f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 --2- 172.21.15.29:0/2275059688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800b0b20 0x7f47801615f0 secure :-1 s=READY pgs=2528 cs=0 l=1 rev1=1 crypto rx=0x7f476c0097a0 tx=0x7f476c02b840 comp rx=0 tx=0).stop 2023-10-14T18:17:17.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 -- 172.21.15.29:0/2275059688 shutdown_connections 2023-10-14T18:17:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 --2- 172.21.15.29:0/2275059688 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800b0b20 0x7f47801615f0 unknown :-1 s=CLOSED pgs=2528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 -- 172.21.15.29:0/2275059688 >> 172.21.15.29:0/2275059688 conn(0x7f47800cb410 msgr2=0x7f47800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 -- 172.21.15.29:0/2275059688 shutdown_connections 2023-10-14T18:17:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.228+0000 7f4786f5a640 1 -- 172.21.15.29:0/2275059688 wait complete. 2023-10-14T18:17:17.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.230+0000 7f4786f5a640 1 Processor -- start 2023-10-14T18:17:17.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.230+0000 7f4786f5a640 1 -- start start 2023-10-14T18:17:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47800b0b20 0x7f47800af510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f478015d3f0 0x7f478014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800b1f90 con 0x7f47800cd0c0 2023-10-14T18:17:17.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47800afaf0 con 0x7f478015d3f0 2023-10-14T18:17:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f4786f5a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47800afdd0 con 0x7f47800b0b20 2023-10-14T18:17:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f477ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f477ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34000/0 (socket says 172.21.15.29:34000) 2023-10-14T18:17:17.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.231+0000 7f477ffff640 1 -- 172.21.15.29:0/494118675 learned_addr learned my addr 172.21.15.29:0/494118675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f47854f7640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f478015d3f0 0x7f478014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f4784cf6640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47800b0b20 0x7f47800af510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 -- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47800b0b20 msgr2=0x7f47800af510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47800b0b20 0x7f47800af510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 -- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f478015d3f0 msgr2=0x7f478014f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f478015d3f0 0x7f478014f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 -- 172.21.15.29:0/494118675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476c009400 con 0x7f47800cd0c0 2023-10-14T18:17:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.232+0000 7f477ffff640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f476c034040 tx=0x7f476c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.233+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c032900 con 0x7f47800cd0c0 2023-10-14T18:17:17.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.233+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f476c032a60 con 0x7f47800cd0c0 2023-10-14T18:17:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.233+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f476c032e10 con 0x7f47800cd0c0 2023-10-14T18:17:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.233+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47800b00b0 con 0x7f47800cd0c0 2023-10-14T18:17:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.233+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f478014f690 con 0x7f47800cd0c0 2023-10-14T18:17:17.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.234+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47800b20d0 con 0x7f47800cd0c0 2023-10-14T18:17:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.236+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f476c03b3f0 con 0x7f47800cd0c0 2023-10-14T18:17:17.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.237+0000 7f477d7fa640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 0x7f476406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.237+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f476c0ab960 con 0x7f47800cd0c0 2023-10-14T18:17:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.237+0000 7f4784cf6640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 0x7f476406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.238+0000 7f4784cf6640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 0x7f476406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4774003d90 tx=0x7f4774009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.249+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47800b20d0 con 0x7f47800cd0c0 2023-10-14T18:17:17.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.537+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]} v 0) v1 -- 0x7f47800b20d0 con 0x7f47800cd0c0 2023-10-14T18:17:17.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.630+0000 7f477d7fa640 1 -- 172.21.15.29:0/494118675 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedules.test_snap_schedule_with_retention"]}]=0 v3172) v1 ==== 163+0+0 (secure 0 0 0) 0x7f47800b20d0 con 0x7f47800cd0c0 2023-10-14T18:17:17.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 msgr2=0x7f476406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 0x7f476406d480 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4774003d90 tx=0x7f4774009040 comp rx=0 tx=0).stop 2023-10-14T18:17:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 msgr2=0x7f4780151e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 secure :-1 s=READY pgs=2529 cs=0 l=1 rev1=1 crypto rx=0x7f476c034040 tx=0x7f476c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 shutdown_connections 2023-10-14T18:17:17.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f476406afc0 0x7f476406d480 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f478015d3f0 0x7f478014f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47800cd0c0 0x7f4780151e50 unknown :-1 s=CLOSED pgs=2529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 --2- 172.21.15.29:0/494118675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47800b0b20 0x7f47800af510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.633+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 >> 172.21.15.29:0/494118675 conn(0x7f47800cb410 msgr2=0x7f47800ce970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:17.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.634+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 shutdown_connections 2023-10-14T18:17:17.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.634+0000 7f4786f5a640 1 -- 172.21.15.29:0/494118675 wait complete. 2023-10-14T18:17:17.657 INFO:tasks.cephfs_test_runner:Test scheduled snapshots along with rentention policy ... ok 2023-10-14T18:17:17.658 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command succes with --fs argument for non-default fs 2023-10-14T18:17:17.658 INFO:tasks.cephfs_test_runner:test_snap_schedule_for_non_default_fs (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-14T18:17:17.659 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.879+0000 7f7868f27640 1 Processor -- start 2023-10-14T18:17:17.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.879+0000 7f7868f27640 1 -- start start 2023-10-14T18:17:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.880+0000 7f7868f27640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f786415b320 0x7f78640cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.880+0000 7f7868f27640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786415b9d0 con 0x7f78640ad970 2023-10-14T18:17:17.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.880+0000 7f7868f27640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f786415bcb0 con 0x7f786406c040 2023-10-14T18:17:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.880+0000 7f7868f27640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f786415b7d0 con 0x7f786415b320 2023-10-14T18:17:17.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f785bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f78640ad970 0x7f78640cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48092/0 (socket says 172.21.15.29:48092) 2023-10-14T18:17:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f7863fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f786415b320 0x7f78640cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f785bfff640 1 -- 172.21.15.29:0/1742285244 learned_addr learned my addr 172.21.15.29:0/1742285244 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f7863fff640 1 -- 172.21.15.29:0/1742285244 >> v1:172.21.15.186:6789/0 conn(0x7f786406c040 legacy=0x7f786406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f7863fff640 1 -- 172.21.15.29:0/1742285244 >> v1:172.21.15.29:6789/0 conn(0x7f78640ad970 legacy=0x7f78640cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f7863fff640 1 -- 172.21.15.29:0/1742285244 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f786415bf90 con 0x7f786415b320 2023-10-14T18:17:17.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.881+0000 7f7862ffd640 1 -- 172.21.15.29:0/1742285244 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162992410 0 0) 0x7f786415b9d0 con 0x7f78640ad970 2023-10-14T18:17:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7863fff640 1 --2- 172.21.15.29:0/1742285244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640cd180 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f785c009dc0 tx=0x7f785c02bf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67aafaea75d439de server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7862ffd640 1 -- 172.21.15.29:0/1742285244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c02ccd0 con 0x7f786415b320 2023-10-14T18:17:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7862ffd640 1 -- 172.21.15.29:0/1742285244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c02ce30 con 0x7f786415b320 2023-10-14T18:17:17.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7862ffd640 1 -- 172.21.15.29:0/1742285244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c034660 con 0x7f786415b320 2023-10-14T18:17:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7868f27640 1 -- 172.21.15.29:0/1742285244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 msgr2=0x7f78640cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.883+0000 7f7868f27640 1 --2- 172.21.15.29:0/1742285244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640cd180 secure :-1 s=READY pgs=2531 cs=0 l=1 rev1=1 crypto rx=0x7f785c009dc0 tx=0x7f785c02bf20 comp rx=0 tx=0).stop 2023-10-14T18:17:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.884+0000 7f7868f27640 1 -- 172.21.15.29:0/1742285244 shutdown_connections 2023-10-14T18:17:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.884+0000 7f7868f27640 1 --2- 172.21.15.29:0/1742285244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640cd180 unknown :-1 s=CLOSED pgs=2531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.884+0000 7f7868f27640 1 -- 172.21.15.29:0/1742285244 >> 172.21.15.29:0/1742285244 conn(0x7f78640ce6a0 msgr2=0x7f78640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.884+0000 7f7868f27640 1 -- 172.21.15.29:0/1742285244 shutdown_connections 2023-10-14T18:17:17.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.884+0000 7f7868f27640 1 -- 172.21.15.29:0/1742285244 wait complete. 2023-10-14T18:17:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.885+0000 7f7868f27640 1 Processor -- start 2023-10-14T18:17:17.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.885+0000 7f7868f27640 1 -- start start 2023-10-14T18:17:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f786406c040 0x7f78641e4480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78640ad970 0x7f78641560e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641e4d80 con 0x7f786415b320 2023-10-14T18:17:17.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78641e5060 con 0x7f78640ad970 2023-10-14T18:17:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.886+0000 7f7868f27640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78641e5340 con 0x7f786406c040 2023-10-14T18:17:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34022/0 (socket says 172.21.15.29:34022) 2023-10-14T18:17:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 -- 172.21.15.29:0/3503760162 learned_addr learned my addr 172.21.15.29:0/3503760162 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:17.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f785bfff640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78640ad970 0x7f78641560e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f78637fe640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f786406c040 0x7f78641e4480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 -- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f786406c040 msgr2=0x7f78641e4480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f786406c040 0x7f78641e4480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 -- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78640ad970 msgr2=0x7f78641560e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78640ad970 0x7f78641560e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:17.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.887+0000 7f7863fff640 1 -- 172.21.15.29:0/3503760162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785c009a70 con 0x7f786415b320 2023-10-14T18:17:17.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7863fff640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f785c02dfd0 tx=0x7f785c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c03d600 con 0x7f786415b320 2023-10-14T18:17:17.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c03d760 con 0x7f786415b320 2023-10-14T18:17:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c03daf0 con 0x7f786415b320 2023-10-14T18:17:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78640afd20 con 0x7f786415b320 2023-10-14T18:17:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.888+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78641e49c0 con 0x7f786415b320 2023-10-14T18:17:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.889+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78641e4b50 con 0x7f786415b320 2023-10-14T18:17:17.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.890+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f785c03b040 con 0x7f786415b320 2023-10-14T18:17:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.891+0000 7f7860ff9640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 0x7f784406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.891+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f785c0abe80 con 0x7f786415b320 2023-10-14T18:17:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.891+0000 7f78637fe640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 0x7f784406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:17.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.892+0000 7f78637fe640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 0x7f784406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7854000f60 tx=0x7f7854009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:17.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:17.896+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f785c076e80 con 0x7f786415b320 2023-10-14T18:17:18.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.093+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f786404e9b0 con 0x7f786415b320 2023-10-14T18:17:18.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.095+0000 7f7860ff9640 1 -- 172.21.15.29:0/3503760162 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v220) v1 ==== 74+0+14171 (secure 0 0 0) 0x7f785c07a4a0 con 0x7f786415b320 2023-10-14T18:17:18.098 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:18.098 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":220,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:15.599152+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/1451005601":"2023-10-15T18:17:12.577615+0000","172.21.15.29:0/1567973042":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1483914389":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6837/1451005601":"2023-10-15T18:17:12.577615+0000","172.21.15.29:0/1310817924":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/3470389371":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/1992230882":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/976049980":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6833/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:6832/966124587":"2023-10-15T18:17:08.714297+0000","172.21.15.29:0/4216364220":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6832/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1790177658":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/3670605654":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/2768480413":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/4181160999":"2023-10-15T18:15:00.600793+0000","172.21.15.29:6833/2339165088":"2023-10-15T18:15:00.600793+0000","172.21.15.29:0/1466849565":"2023-10-15T18:15:00.600793+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:18.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 msgr2=0x7f784406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 0x7f784406d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f7854000f60 tx=0x7f7854009040 comp rx=0 tx=0).stop 2023-10-14T18:17:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 msgr2=0x7f78640b0520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 secure :-1 s=READY pgs=2532 cs=0 l=1 rev1=1 crypto rx=0x7f785c02dfd0 tx=0x7f785c004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 shutdown_connections 2023-10-14T18:17:18.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f784406af90 0x7f784406d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f786415b320 0x7f78640b0520 unknown :-1 s=CLOSED pgs=2532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78640ad970 0x7f78641560e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 --2- 172.21.15.29:0/3503760162 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f786406c040 0x7f78641e4480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.098+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 >> 172.21.15.29:0/3503760162 conn(0x7f78640ce6a0 msgr2=0x7f786415da70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:18.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.099+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 shutdown_connections 2023-10-14T18:17:18.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.099+0000 7f7868f27640 1 -- 172.21.15.29:0/3503760162 wait complete. 2023-10-14T18:17:18.121 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:17:18.122 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:17:18.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.350+0000 7fcc07fff640 1 Processor -- start 2023-10-14T18:17:18.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.350+0000 7fcc07fff640 1 -- start start 2023-10-14T18:17:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcc080cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcc07fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcc080b0c90 0x7fcc080c9950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcc07fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0806f3a0 con 0x7fcc080adb30 2023-10-14T18:17:18.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcc07fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc080c8e70 con 0x7fcc080b0c90 2023-10-14T18:17:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcc07fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c9150 con 0x7fcc0806c630 2023-10-14T18:17:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcbfffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcc080adb30 0x7fcc080cd7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48102/0 (socket says 172.21.15.29:48102) 2023-10-14T18:17:18.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.352+0000 7fcbfffff640 1 -- 172.21.15.29:0/4047628202 learned_addr learned my addr 172.21.15.29:0/4047628202 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc077fe640 1 --2- 172.21.15.29:0/4047628202 >> v2:172.21.15.186:3300/0 conn(0x7fcc080b0c90 0x7fcc080c9950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcc080cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034370181 0 0) 0x7fcc0806f3a0 con 0x7fcc080adb30 2023-10-14T18:17:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf4003520 con 0x7fcc080adb30 2023-10-14T18:17:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1202926361 0 0) 0x7fcbf4003520 con 0x7fcc080adb30 2023-10-14T18:17:18.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 msgr2=0x7fcc080cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcc080cd260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 >> v2:172.21.15.186:3300/0 conn(0x7fcc080b0c90 msgr2=0x7fcc080c9950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 --2- 172.21.15.29:0/4047628202 >> v2:172.21.15.186:3300/0 conn(0x7fcc080b0c90 0x7fcc080c9950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc080c9430 con 0x7fcc080adb30 2023-10-14T18:17:18.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fcbf0001690 con 0x7fcc080adb30 2023-10-14T18:17:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 >> v1:172.21.15.29:6789/0 conn(0x7fcc080adb30 legacy=0x7fcc080cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf4003970 0x7fcbf4003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf40047c0 0x7fcbf4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.353+0000 7fcc067fc640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcbf4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c8e70 con 0x7fcbf4003970 2023-10-14T18:17:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc080c9150 con 0x7fcbf40047c0 2023-10-14T18:17:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf400b9d0 con 0x7fcc0806c630 2023-10-14T18:17:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf4003970 0x7fcbf4003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcbfffff640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf40047c0 0x7fcbf4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc077fe640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcbf4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 -- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 msgr2=0x7fcbf4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc0806c630 0x7fcbf4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 -- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf40047c0 msgr2=0x7fcbf4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf40047c0 0x7fcbf4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.354+0000 7fcc06ffd640 1 -- 172.21.15.29:0/4047628202 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf4003520 con 0x7fcbf4003970 2023-10-14T18:17:18.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.355+0000 7fcbfffff640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf40047c0 0x7fcbf4004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:18.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.356+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf4003970 0x7fcbf4003d90 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009af0 tx=0x7fcbf40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ab85f11d1f1c4683 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.356+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf802c6b0 con 0x7fcbf4003970 2023-10-14T18:17:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.356+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf802c810 con 0x7fcbf4003970 2023-10-14T18:17:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.356+0000 7fcc067fc640 1 -- 172.21.15.29:0/4047628202 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf802cba0 con 0x7fcbf4003970 2023-10-14T18:17:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 -- 172.21.15.29:0/4047628202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf4003970 msgr2=0x7fcbf4003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 --2- 172.21.15.29:0/4047628202 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf4003970 0x7fcbf4003d90 secure :-1 s=READY pgs=2534 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8009af0 tx=0x7fcbf40057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc077fe640 1 -- 172.21.15.29:0/4047628202 reap_dead start 2023-10-14T18:17:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 -- 172.21.15.29:0/4047628202 shutdown_connections 2023-10-14T18:17:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 -- 172.21.15.29:0/4047628202 >> 172.21.15.29:0/4047628202 conn(0x7fcc080b0170 msgr2=0x7fcc080b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 -- 172.21.15.29:0/4047628202 shutdown_connections 2023-10-14T18:17:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.357+0000 7fcc07fff640 1 -- 172.21.15.29:0/4047628202 wait complete. 2023-10-14T18:17:18.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.359+0000 7fcc07fff640 1 Processor -- start 2023-10-14T18:17:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.359+0000 7fcc07fff640 1 -- start start 2023-10-14T18:17:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc081526f0 0x7fcc08152b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081530b0 0x7fcc081fa130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc08156b10 con 0x7fcc081f6620 2023-10-14T18:17:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc081503e0 con 0x7fcc081530b0 2023-10-14T18:17:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.360+0000 7fcc07fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc08150740 con 0x7fcc081526f0 2023-10-14T18:17:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34034/0 (socket says 172.21.15.29:34034) 2023-10-14T18:17:18.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 -- 172.21.15.29:0/256492278 learned_addr learned my addr 172.21.15.29:0/256492278 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc077fe640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081530b0 0x7fcc081fa130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcbfffff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc081526f0 0x7fcc08152b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 -- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc081526f0 msgr2=0x7fcc08152b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc081526f0 0x7fcc08152b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 -- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081530b0 msgr2=0x7fcc081fa130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081530b0 0x7fcc081fa130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.361+0000 7fcc06ffd640 1 -- 172.21.15.29:0/256492278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc08153560 con 0x7fcc081f6620 2023-10-14T18:17:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcc06ffd640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8037040 tx=0x7fcbf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8003810 con 0x7fcc081f6620 2023-10-14T18:17:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbf8003970 con 0x7fcc081f6620 2023-10-14T18:17:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbf8003d20 con 0x7fcc081f6620 2023-10-14T18:17:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc081f6a00 con 0x7fcc081f6620 2023-10-14T18:17:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.362+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc080b3b40 con 0x7fcc081f6620 2023-10-14T18:17:18.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.363+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0804e9b0 con 0x7fcc081f6620 2023-10-14T18:17:18.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.364+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcbf803e440 con 0x7fcc081f6620 2023-10-14T18:17:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.365+0000 7fcbff7fe640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 0x7fcbe806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.365+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fcbf80ab0f0 con 0x7fcc081f6620 2023-10-14T18:17:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.365+0000 7fcbfffff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 0x7fcbe806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.366+0000 7fcbfffff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 0x7fcbe806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0004840 tx=0x7fcbf0009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.369+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbf8076060 con 0x7fcc081f6620 2023-10-14T18:17:18.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.564+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fcc0806a560 con 0x7fcc081f6620 2023-10-14T18:17:18.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.565+0000 7fcbff7fe640 1 -- 172.21.15.29:0/256492278 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v220) v1 ==== 93+0+10 (secure 0 0 0) 0x7fcbf80044e0 con 0x7fcc081f6620 2023-10-14T18:17:18.567 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:17:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 msgr2=0x7fcbe806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 0x7fcbe806d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbf0004840 tx=0x7fcbf0009040 comp rx=0 tx=0).stop 2023-10-14T18:17:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 msgr2=0x7fcc080b3280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 secure :-1 s=READY pgs=2535 cs=0 l=1 rev1=1 crypto rx=0x7fcbf8037040 tx=0x7fcbf8004730 comp rx=0 tx=0).stop 2023-10-14T18:17:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 shutdown_connections 2023-10-14T18:17:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fcbe806afc0 0x7fcbe806d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcc081530b0 0x7fcc081fa130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcc081526f0 0x7fcc08152b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 --2- 172.21.15.29:0/256492278 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcc081f6620 0x7fcc080b3280 unknown :-1 s=CLOSED pgs=2535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 >> 172.21.15.29:0/256492278 conn(0x7fcc080b0170 msgr2=0x7fcc0806a780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 shutdown_connections 2023-10-14T18:17:18.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.568+0000 7fcc07fff640 1 -- 172.21.15.29:0/256492278 wait complete. 2023-10-14T18:17:18.590 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-14T18:17:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.816+0000 7f763574e640 1 Processor -- start 2023-10-14T18:17:18.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.816+0000 7f763574e640 1 -- start start 2023-10-14T18:17:18.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.817+0000 7f763574e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f763015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.817+0000 7f763574e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f763006c100 0x7f763006c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.817+0000 7f763574e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763015aef0 con 0x7f7630159950 2023-10-14T18:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f763574e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763015e490 con 0x7f763006c100 2023-10-14T18:17:18.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f763574e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763015d910 con 0x7f7630070b60 2023-10-14T18:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f7634f4d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f763006c100 0x7f763006c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f7627fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f763015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f7634f4d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f763006c100 0x7f763006c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:43874/0 (socket says 172.21.15.29:43874) 2023-10-14T18:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.818+0000 7f7634f4d640 1 -- 172.21.15.29:0/3014753466 learned_addr learned my addr 172.21.15.29:0/3014753466 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.819+0000 7f7634f4d640 1 -- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 msgr2=0x7f763015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.819+0000 7f7634f4d640 1 --2- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f763015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.819+0000 7f7634f4d640 1 -- 172.21.15.29:0/3014753466 >> v1:172.21.15.186:6789/0 conn(0x7f7630159950 legacy=0x7f7630070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.819+0000 7f7634f4d640 1 -- 172.21.15.29:0/3014753466 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763015dbf0 con 0x7f763006c100 2023-10-14T18:17:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.821+0000 7f7634f4d640 1 --2- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763006c550 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7628009e50 tx=0x7f762802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=204509bea7f1f0f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f762f7fe640 1 -- 172.21.15.29:0/3014753466 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762802ca80 con 0x7f763006c100 2023-10-14T18:17:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f762f7fe640 1 -- 172.21.15.29:0/3014753466 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762802cbe0 con 0x7f763006c100 2023-10-14T18:17:18.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f762f7fe640 1 -- 172.21.15.29:0/3014753466 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76280343f0 con 0x7f763006c100 2023-10-14T18:17:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 -- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 msgr2=0x7f763006c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 --2- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763006c550 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f7628009e50 tx=0x7f762802bee0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 -- 172.21.15.29:0/3014753466 shutdown_connections 2023-10-14T18:17:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 --2- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f763015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 --2- 172.21.15.29:0/3014753466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763006c550 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 -- 172.21.15.29:0/3014753466 >> 172.21.15.29:0/3014753466 conn(0x7f76300ce6a0 msgr2=0x7f76300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:18.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.822+0000 7f763574e640 1 -- 172.21.15.29:0/3014753466 shutdown_connections 2023-10-14T18:17:18.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.823+0000 7f763574e640 1 -- 172.21.15.29:0/3014753466 wait complete. 2023-10-14T18:17:18.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.824+0000 7f763574e640 1 Processor -- start 2023-10-14T18:17:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.825+0000 7f763574e640 1 -- start start 2023-10-14T18:17:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.825+0000 7f763574e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f763574e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f76301471e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f763574e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f763574e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76300c83d0 con 0x7f7630159950 2023-10-14T18:17:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f763574e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76300c86b0 con 0x7f763006c100 2023-10-14T18:17:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f763574e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7630155810 con 0x7f7630070b60 2023-10-14T18:17:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f7634f4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f7634f4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34050/0 (socket says 172.21.15.29:34050) 2023-10-14T18:17:18.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f7634f4d640 1 -- 172.21.15.29:0/1559687092 learned_addr learned my addr 172.21.15.29:0/1559687092 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f7627fff640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f76301471e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.826+0000 7f762ffff640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 -- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 msgr2=0x7f76301471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f76301471e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 -- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 msgr2=0x7f763014d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763014d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 -- 172.21.15.29:0/1559687092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628009b30 con 0x7f7630159950 2023-10-14T18:17:18.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f762ffff640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763014d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f7634f4d640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7628009e50 tx=0x7f7628004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7628009510 con 0x7f7630159950 2023-10-14T18:17:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f762803e450 con 0x7f7630159950 2023-10-14T18:17:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762803e800 con 0x7f7630159950 2023-10-14T18:17:18.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.827+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7630155af0 con 0x7f7630159950 2023-10-14T18:17:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.828+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7630147b10 con 0x7f7630159950 2023-10-14T18:17:18.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.829+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7630155fb0 con 0x7f7630159950 2023-10-14T18:17:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.830+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f762802cd50 con 0x7f7630159950 2023-10-14T18:17:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.831+0000 7f762d7fa640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 0x7f761006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.831+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f76280ac310 con 0x7f7630159950 2023-10-14T18:17:18.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.832+0000 7f762ffff640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 0x7f761006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:18.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.832+0000 7f762ffff640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 0x7f761006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7620007fd0 tx=0x7f7620009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:18.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:18.836+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f76280771e0 con 0x7f7630159950 2023-10-14T18:17:19.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.102+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f76300c8510 con 0x7f7630159950 2023-10-14T18:17:19.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.647+0000 7f762d7fa640 1 -- 172.21.15.29:0/1559687092 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3174) v1 ==== 189+0+0 (secure 0 0 0) 0x7f76300c8510 con 0x7f7630159950 2023-10-14T18:17:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 msgr2=0x7f761006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 0x7f761006d450 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7620007fd0 tx=0x7f7620009040 comp rx=0 tx=0).stop 2023-10-14T18:17:19.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 msgr2=0x7f7630149db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 secure :-1 s=READY pgs=2536 cs=0 l=1 rev1=1 crypto rx=0x7f7628009e50 tx=0x7f7628004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 shutdown_connections 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f761006af90 0x7f761006d450 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7630159950 0x7f7630149db0 unknown :-1 s=CLOSED pgs=2536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7630070b60 0x7f76301471e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 --2- 172.21.15.29:0/1559687092 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f763006c100 0x7f763014d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 >> 172.21.15.29:0/1559687092 conn(0x7f76300ce6a0 msgr2=0x7f763006c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 shutdown_connections 2023-10-14T18:17:19.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.652+0000 7f763574e640 1 -- 172.21.15.29:0/1559687092 wait complete. 2023-10-14T18:17:19.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T18:17:19.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.911+0000 7fe8a986e640 1 Processor -- start 2023-10-14T18:17:19.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.911+0000 7fe8a986e640 1 -- start start 2023-10-14T18:17:19.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a986e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8a406fa20 0x7fe8a40ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a986e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a4065cc0 0x7fe8a40acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a986e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8a406d900 con 0x7fe8a406d520 2023-10-14T18:17:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a986e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8a415b1d0 con 0x7fe8a406fa20 2023-10-14T18:17:19.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a986e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40b24f0 con 0x7fe8a4065cc0 2023-10-14T18:17:19.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8a406fa20 0x7fe8a40ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe8a406fa20 0x7fe8a40ac750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:43880/0 (socket says 172.21.15.29:43880) 2023-10-14T18:17:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a3fff640 1 -- 172.21.15.29:0/2886173329 learned_addr learned my addr 172.21.15.29:0/2886173329 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a886c640 1 --1- 172.21.15.29:0/2886173329 >> v1:172.21.15.186:6789/0 conn(0x7fe8a406d520 0x7fe8a40b1de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:42786/0 (socket says 172.21.15.29:42786) 2023-10-14T18:17:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.913+0000 7fe8a906d640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a4065cc0 0x7fe8a40acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.914+0000 7fe8a3fff640 1 -- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a4065cc0 msgr2=0x7fe8a40acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:19.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.914+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a4065cc0 0x7fe8a40acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.914+0000 7fe8a3fff640 1 -- 172.21.15.29:0/2886173329 >> v1:172.21.15.186:6789/0 conn(0x7fe8a406d520 legacy=0x7fe8a40b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.914+0000 7fe8a3fff640 1 -- 172.21.15.29:0/2886173329 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8a406feb0 con 0x7fe8a406fa20 2023-10-14T18:17:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.915+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a406fa20 0x7fe8a40ac750 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe898009870 tx=0x7fe89802b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c2c139bb2dfbd9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.916+0000 7fe8a37fe640 1 -- 172.21.15.29:0/2886173329 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8980317c0 con 0x7fe8a406fa20 2023-10-14T18:17:19.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.916+0000 7fe8a37fe640 1 -- 172.21.15.29:0/2886173329 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe898031920 con 0x7fe8a406fa20 2023-10-14T18:17:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.916+0000 7fe8a37fe640 1 -- 172.21.15.29:0/2886173329 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe898031cb0 con 0x7fe8a406fa20 2023-10-14T18:17:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 -- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a406fa20 msgr2=0x7fe8a40ac750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a406fa20 0x7fe8a40ac750 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fe898009870 tx=0x7fe89802b860 comp rx=0 tx=0).stop 2023-10-14T18:17:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 -- 172.21.15.29:0/2886173329 shutdown_connections 2023-10-14T18:17:19.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a4065cc0 0x7fe8a40acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 --2- 172.21.15.29:0/2886173329 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a406fa20 0x7fe8a40ac750 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 -- 172.21.15.29:0/2886173329 >> 172.21.15.29:0/2886173329 conn(0x7fe8a40b0a80 msgr2=0x7fe8a40b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 -- 172.21.15.29:0/2886173329 shutdown_connections 2023-10-14T18:17:19.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.917+0000 7fe8a986e640 1 -- 172.21.15.29:0/2886173329 wait complete. 2023-10-14T18:17:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.920+0000 7fe8a986e640 1 Processor -- start 2023-10-14T18:17:19.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.920+0000 7fe8a986e640 1 -- start start 2023-10-14T18:17:19.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.921+0000 7fe8a986e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a986e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a406d520 0x7fe8a41f32d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a986e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 0x7fe8a415cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a986e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40c9d40 con 0x7fe8a4065cc0 2023-10-14T18:17:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a986e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a415d500 con 0x7fe8a415cb10 2023-10-14T18:17:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a986e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe8a40cb4c0 con 0x7fe8a406d520 2023-10-14T18:17:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34066/0 (socket says 172.21.15.29:34066) 2023-10-14T18:17:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 -- 172.21.15.29:0/1317918050 learned_addr learned my addr 172.21.15.29:0/1317918050 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a406d520 0x7fe8a41f32d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a906d640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 0x7fe8a415cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 -- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a406d520 msgr2=0x7fe8a41f32d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a406d520 0x7fe8a41f32d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 -- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 msgr2=0x7fe8a415cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 0x7fe8a415cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a886c640 1 -- 172.21.15.29:0/1317918050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8980094d0 con 0x7fe8a4065cc0 2023-10-14T18:17:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.922+0000 7fe8a906d640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 0x7fe8a415cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a886c640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c800 tx=0x7fe89400ccc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400a640 con 0x7fe8a4065cc0 2023-10-14T18:17:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe89400a7a0 con 0x7fe8a4065cc0 2023-10-14T18:17:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe89400ab50 con 0x7fe8a4065cc0 2023-10-14T18:17:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8a41f2d40 con 0x7fe8a4065cc0 2023-10-14T18:17:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.923+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8a40c9960 con 0x7fe8a4065cc0 2023-10-14T18:17:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.925+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8a404e9b0 con 0x7fe8a4065cc0 2023-10-14T18:17:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.925+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe89400acb0 con 0x7fe8a4065cc0 2023-10-14T18:17:19.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.927+0000 7fe8a17fa640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 0x7fe89006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:19.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.927+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fe894088f90 con 0x7fe8a4065cc0 2023-10-14T18:17:19.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.927+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 0x7fe89006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:19.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.928+0000 7fe8a3fff640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 0x7fe89006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe898036000 tx=0x7fe89803d040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:19.932+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe894057a30 con 0x7fe8a4065cc0 2023-10-14T18:17:20.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.133+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fe8a40ac530 con 0x7fe8a4065cc0 2023-10-14T18:17:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.140+0000 7fe8a17fa640 1 -- 172.21.15.29:0/1317918050 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v76)=0 v76) v1 ==== 122+0+0 (secure 0 0 0) 0x7fe894017020 con 0x7fe8a4065cc0 2023-10-14T18:17:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 msgr2=0x7fe89006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 0x7fe89006d450 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fe898036000 tx=0x7fe89803d040 comp rx=0 tx=0).stop 2023-10-14T18:17:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 msgr2=0x7fe8a406bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 secure :-1 s=READY pgs=2537 cs=0 l=1 rev1=1 crypto rx=0x7fe89400c800 tx=0x7fe89400ccc0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 shutdown_connections 2023-10-14T18:17:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe89006af90 0x7fe89006d450 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe8a415cb10 0x7fe8a415cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8a406d520 0x7fe8a41f32d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 --2- 172.21.15.29:0/1317918050 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8a4065cc0 0x7fe8a406bb40 unknown :-1 s=CLOSED pgs=2537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 >> 172.21.15.29:0/1317918050 conn(0x7fe8a40b0a80 msgr2=0x7fe8a415b610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 shutdown_connections 2023-10-14T18:17:20.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.143+0000 7fe8a986e640 1 -- 172.21.15.29:0/1317918050 wait complete. 2023-10-14T18:17:20.165 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:17:20.165 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:17:20.287 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:17:20.411 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:17:20.411 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:20.412 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:20.476 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:17:20.476 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:17:20.477 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:20.538 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:17:20.538 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:20.539 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:20.603 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:17:20.603 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:17:20.604 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:20.666 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:17:20.666 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T18:17:20.667 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.904+0000 7f5feed0a640 1 Processor -- start 2023-10-14T18:17:20.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.904+0000 7f5feed0a640 1 -- start start 2023-10-14T18:17:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5feed0a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5fe806c040 0x7f5fe806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5feed0a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe815f4a0 con 0x7f5fe80cda80 2023-10-14T18:17:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5feed0a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fe806c960 con 0x7f5fe80ad970 2023-10-14T18:17:20.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5feed0a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fe815b9d0 con 0x7f5fe806c040 2023-10-14T18:17:20.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fee509640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5fe80cda80 0x7f5fe815ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48116/0 (socket says 172.21.15.29:48116) 2023-10-14T18:17:20.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fee509640 1 -- 172.21.15.29:0/589325292 learned_addr learned my addr 172.21.15.29:0/589325292 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:20.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fedd08640 1 --2- 172.21.15.29:0/589325292 >> v2:172.21.15.186:3300/0 conn(0x7f5fe806c040 0x7f5fe806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572556570 0 0) 0x7f5fe815f4a0 con 0x7f5fe80cda80 2023-10-14T18:17:20.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd4003520 con 0x7f5fe80cda80 2023-10-14T18:17:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2882472262 0 0) 0x7f5fd4003520 con 0x7f5fe80cda80 2023-10-14T18:17:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 >> v2:172.21.15.186:3300/0 conn(0x7f5fe806c040 msgr2=0x7f5fe806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.906+0000 7f5fecd06640 1 --2- 172.21.15.29:0/589325292 >> v2:172.21.15.186:3300/0 conn(0x7f5fe806c040 0x7f5fe806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 >> v1:172.21.15.186:6789/0 conn(0x7f5fe80ad970 legacy=0x7f5fe815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe815bcb0 con 0x7f5fe80cda80 2023-10-14T18:17:20.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5fe00019f0 con 0x7f5fe80cda80 2023-10-14T18:17:20.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 >> v1:172.21.15.29:6789/0 conn(0x7f5fe80cda80 legacy=0x7f5fe815ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40039c0 0x7f5fd4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd40047c0 0x7f5fd4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fd400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe815b9d0 con 0x7f5fd4005180 2023-10-14T18:17:20.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400c080 con 0x7f5fd40047c0 2023-10-14T18:17:20.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd400c240 con 0x7f5fd40039c0 2023-10-14T18:17:20.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fee509640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fd400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fed507640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd40047c0 0x7f5fd4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.907+0000 7f5fedd08640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40039c0 0x7f5fd4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 -- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40039c0 msgr2=0x7f5fd4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fd40039c0 0x7f5fd4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 -- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd40047c0 msgr2=0x7f5fd4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:20.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fd40047c0 0x7f5fd4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 -- 172.21.15.29:0/589325292 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd4003520 con 0x7f5fd4005180 2023-10-14T18:17:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.908+0000 7f5fee509640 1 -- 172.21.15.29:0/589325292 reap_dead start 2023-10-14T18:17:20.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5fee509640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fd400b980 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000a760 tx=0x7f5fd40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b729cd1f5611819 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0034630 con 0x7f5fd4005180 2023-10-14T18:17:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe002f070 con 0x7f5fd4005180 2023-10-14T18:17:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5fecd06640 1 -- 172.21.15.29:0/589325292 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe0034a10 con 0x7f5fd4005180 2023-10-14T18:17:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5feed0a640 1 -- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 msgr2=0x7f5fd400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.909+0000 7f5feed0a640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fd400b980 secure :-1 s=READY pgs=2539 cs=0 l=1 rev1=1 crypto rx=0x7f5fe000a760 tx=0x7f5fd40057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.910+0000 7f5feed0a640 1 -- 172.21.15.29:0/589325292 shutdown_connections 2023-10-14T18:17:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.910+0000 7f5feed0a640 1 --2- 172.21.15.29:0/589325292 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fd400b980 unknown :-1 s=CLOSED pgs=2539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.910+0000 7f5feed0a640 1 -- 172.21.15.29:0/589325292 >> 172.21.15.29:0/589325292 conn(0x7f5fe80ce6a0 msgr2=0x7f5fe80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.910+0000 7f5feed0a640 1 -- 172.21.15.29:0/589325292 shutdown_connections 2023-10-14T18:17:20.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.910+0000 7f5feed0a640 1 -- 172.21.15.29:0/589325292 wait complete. 2023-10-14T18:17:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.911+0000 7f5feed0a640 1 Processor -- start 2023-10-14T18:17:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.912+0000 7f5feed0a640 1 -- start start 2023-10-14T18:17:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fe80b0bf0 0x7f5fe80c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fe80c80c0 0x7f5fe80ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe814a0a0 con 0x7f5fd4005180 2023-10-14T18:17:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80c8540 con 0x7f5fe80c80c0 2023-10-14T18:17:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5feed0a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe80afb50 con 0x7f5fe80b0bf0 2023-10-14T18:17:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34096/0 (socket says 172.21.15.29:34096) 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 -- 172.21.15.29:0/679620273 learned_addr learned my addr 172.21.15.29:0/679620273 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fee509640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fe80c80c0 0x7f5fe80ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fed507640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fe80b0bf0 0x7f5fe80c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 -- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fe80b0bf0 msgr2=0x7f5fe80c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fe80b0bf0 0x7f5fe80c7b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 -- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fe80c80c0 msgr2=0x7f5fe80ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fe80c80c0 0x7f5fe80ac9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.913+0000 7f5fedd08640 1 -- 172.21.15.29:0/679620273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fe80afe30 con 0x7f5fd4005180 2023-10-14T18:17:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5fedd08640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4011540 tx=0x7f5fe4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4015930 con 0x7f5fd4005180 2023-10-14T18:17:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fe4015a90 con 0x7f5fd4005180 2023-10-14T18:17:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fe4015e60 con 0x7f5fd4005180 2023-10-14T18:17:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe80b0110 con 0x7f5fd4005180 2023-10-14T18:17:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.914+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe80b06d0 con 0x7f5fd4005180 2023-10-14T18:17:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.915+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe80b0890 con 0x7f5fd4005180 2023-10-14T18:17:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.916+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5fe400f460 con 0x7f5fd4005180 2023-10-14T18:17:20.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.917+0000 7f5fde7fc640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 0x7f5fcc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.917+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7f5fe408a070 con 0x7f5fd4005180 2023-10-14T18:17:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.918+0000 7f5fed507640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 0x7f5fcc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.918+0000 7f5fed507640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 0x7f5fcc06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8007fd0 tx=0x7f5fd8009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:20.922+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5fe40bcfe0 con 0x7f5fd4005180 2023-10-14T18:17:21.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.155+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5fe806b330 con 0x7f5fd4005180 2023-10-14T18:17:21.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.156+0000 7f5fde7fc640 1 -- 172.21.15.29:0/679620273 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 100 v100) v1 ==== 95+0+3705 (secure 0 0 0) 0x7f5fe4022e90 con 0x7f5fd4005180 2023-10-14T18:17:21.159 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:21.159 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":100,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7040,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/839299447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":839299447},{"type":"v1","addr":"172.21.15.186:6833","nonce":839299447}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7049,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2604286629","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2604286629},{"type":"v1","addr":"172.21.15.186:6835","nonce":2604286629}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":7981,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1933527342","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1933527342},{"type":"v1","addr":"172.21.15.29:6836","nonce":1933527342}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":94},{"gid":11175,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1734295501","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":1734295501},{"type":"v1","addr":"172.21.15.29:6837","nonce":1734295501}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":100}],"filesystems":[]} 2023-10-14T18:17:21.159 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 100 2023-10-14T18:17:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 msgr2=0x7f5fcc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 0x7f5fcc06d450 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5fd8007fd0 tx=0x7f5fd8009040 comp rx=0 tx=0).stop 2023-10-14T18:17:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 msgr2=0x7f5fe8149b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 secure :-1 s=READY pgs=2540 cs=0 l=1 rev1=1 crypto rx=0x7f5fe4011540 tx=0x7f5fe4011a00 comp rx=0 tx=0).stop 2023-10-14T18:17:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 shutdown_connections 2023-10-14T18:17:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f5fcc06af90 0x7f5fcc06d450 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5fe80c80c0 0x7f5fe80ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5fe80b0bf0 0x7f5fe80c7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 --2- 172.21.15.29:0/679620273 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5fd4005180 0x7f5fe8149b60 unknown :-1 s=CLOSED pgs=2540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 >> 172.21.15.29:0/679620273 conn(0x7f5fe80ce6a0 msgr2=0x7f5fe80c5820 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 shutdown_connections 2023-10-14T18:17:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.159+0000 7f5feed0a640 1 -- 172.21.15.29:0/679620273 wait complete. 2023-10-14T18:17:21.182 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T18:17:21.183 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T18:17:21.183 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T18:17:21.183 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:17:21.184 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T18:17:21.184 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T18:17:21.185 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T18:17:21.185 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:17:21.185 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T18:17:21.186 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T18:17:21.186 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T18:17:21.186 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:17:21.187 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T18:17:21.187 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T18:17:21.188 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T18:17:21.188 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:17:21.201 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T18:17:21.201 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T18:17:21.202 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T18:17:21.202 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T18:17:21.207 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T18:17:21.207 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T18:17:21.209 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T18:17:21.210 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T18:17:21.246 INFO:tasks.ceph.mds.c:Started 2023-10-14T18:17:21.246 DEBUG:teuthology.parallel:result is None 2023-10-14T18:17:21.249 INFO:tasks.ceph.mds.a:Started 2023-10-14T18:17:21.250 DEBUG:teuthology.parallel:result is None 2023-10-14T18:17:21.258 INFO:tasks.ceph.mds.d:Started 2023-10-14T18:17:21.259 DEBUG:teuthology.parallel:result is None 2023-10-14T18:17:21.266 INFO:tasks.ceph.mds.b:Started 2023-10-14T18:17:21.267 DEBUG:teuthology.parallel:result is None 2023-10-14T18:17:21.267 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T18:17:21.407 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:17:21.421 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T18:17:21.426 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:17:21.429 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T18:17:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.463+0000 7fabd20e7640 1 Processor -- start 2023-10-14T18:17:21.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.463+0000 7fabd20e7640 1 -- start start 2023-10-14T18:17:21.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.464+0000 7fabd20e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fabcc0caac0 0x7fabcc0ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.464+0000 7fabd20e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0cb030 con 0x7fabcc15ad70 2023-10-14T18:17:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.464+0000 7fabd20e7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabcc0c91c0 con 0x7fabcc06a4b0 2023-10-14T18:17:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.464+0000 7fabd20e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabcc0c94a0 con 0x7fabcc0caac0 2023-10-14T18:17:21.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fabcc15ad70 0x7fabcc069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55896/0 (socket says 172.21.15.29:55896) 2023-10-14T18:17:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcb7fe640 1 -- 172.21.15.29:0/81212239 learned_addr learned my addr 172.21.15.29:0/81212239 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:21.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcbfff640 1 --2- 172.21.15.29:0/81212239 >> v2:172.21.15.186:3300/0 conn(0x7fabcc0caac0 0x7fabcc0ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcaffd640 1 -- 172.21.15.29:0/81212239 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873147982 0 0) 0x7fabcc0cb030 con 0x7fabcc15ad70 2023-10-14T18:17:21.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcbfff640 1 -- 172.21.15.29:0/81212239 >> v1:172.21.15.186:6789/0 conn(0x7fabcc06a4b0 legacy=0x7fabcc0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcbfff640 1 -- 172.21.15.29:0/81212239 >> v1:172.21.15.29:6789/0 conn(0x7fabcc15ad70 legacy=0x7fabcc069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.465+0000 7fabcbfff640 1 -- 172.21.15.29:0/81212239 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabcc0c9780 con 0x7fabcc0caac0 2023-10-14T18:17:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.466+0000 7fabcbfff640 1 --2- 172.21.15.29:0/81212239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc0caac0 0x7fabcc0ad4d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009d90 tx=0x7fabc402bf40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ba517e238f28f74 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:21.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabcaffd640 1 -- 172.21.15.29:0/81212239 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc402ccf0 con 0x7fabcc0caac0 2023-10-14T18:17:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabcaffd640 1 -- 172.21.15.29:0/81212239 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc402ce50 con 0x7fabcc0caac0 2023-10-14T18:17:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabcaffd640 1 -- 172.21.15.29:0/81212239 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabc4034660 con 0x7fabcc0caac0 2023-10-14T18:17:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabd20e7640 1 -- 172.21.15.29:0/81212239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc0caac0 msgr2=0x7fabcc0ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabd20e7640 1 --2- 172.21.15.29:0/81212239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc0caac0 0x7fabcc0ad4d0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fabc4009d90 tx=0x7fabc402bf40 comp rx=0 tx=0).stop 2023-10-14T18:17:21.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabd20e7640 1 -- 172.21.15.29:0/81212239 shutdown_connections 2023-10-14T18:17:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabd20e7640 1 --2- 172.21.15.29:0/81212239 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc0caac0 0x7fabcc0ad4d0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.467+0000 7fabd20e7640 1 -- 172.21.15.29:0/81212239 >> 172.21.15.29:0/81212239 conn(0x7fabcc0ce9b0 msgr2=0x7fabcc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.468+0000 7fabd20e7640 1 -- 172.21.15.29:0/81212239 shutdown_connections 2023-10-14T18:17:21.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.468+0000 7fabd20e7640 1 -- 172.21.15.29:0/81212239 wait complete. 2023-10-14T18:17:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.469+0000 7fabd20e7640 1 Processor -- start 2023-10-14T18:17:21.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.469+0000 7fabd20e7640 1 -- start start 2023-10-14T18:17:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc06a4b0 0x7fabcc0c1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabcc15ad70 0x7fabcc0c8470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0c7f70 con 0x7fabcc0caac0 2023-10-14T18:17:21.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0c4de0 con 0x7fabcc06a4b0 2023-10-14T18:17:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.470+0000 7fabd20e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabcc0c50c0 con 0x7fabcc15ad70 2023-10-14T18:17:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38218/0 (socket says 172.21.15.29:38218) 2023-10-14T18:17:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 -- 172.21.15.29:0/987245106 learned_addr learned my addr 172.21.15.29:0/987245106 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:21.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabcb7fe640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc06a4b0 0x7fabcc0c1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabcbfff640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabcc15ad70 0x7fabcc0c8470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 -- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabcc15ad70 msgr2=0x7fabcc0c8470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabcc15ad70 0x7fabcc0c8470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 -- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc06a4b0 msgr2=0x7fabcc0c1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:21.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc06a4b0 0x7fabcc0c1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.471+0000 7fabc3fff640 1 -- 172.21.15.29:0/987245106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabc4009a70 con 0x7fabcc0caac0 2023-10-14T18:17:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabc3fff640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c450 tx=0x7fabb400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb40052d0 con 0x7fabcc0caac0 2023-10-14T18:17:21.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabb4005430 con 0x7fabcc0caac0 2023-10-14T18:17:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabb4012870 con 0x7fabcc0caac0 2023-10-14T18:17:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabcc0cf200 con 0x7fabcc0caac0 2023-10-14T18:17:21.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.472+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabcc0c5bd0 con 0x7fabcc0caac0 2023-10-14T18:17:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.473+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabcc04e160 con 0x7fabcc0caac0 2023-10-14T18:17:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.474+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fabb40129d0 con 0x7fabcc0caac0 2023-10-14T18:17:21.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.475+0000 7fabc8ff9640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 0x7fabac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.475+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(220..220 src has 1..220) v4 ==== 5747+0+0 (secure 0 0 0) 0x7fabb40891e0 con 0x7fabcc0caac0 2023-10-14T18:17:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.475+0000 7fabcb7fe640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 0x7fabac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.476+0000 7fabcb7fe640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 0x7fabac06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000f60 tx=0x7fabbc007e40 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:21.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.480+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fabb4054160 con 0x7fabcc0caac0 2023-10-14T18:17:21.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:21.696+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7fabcc0c5280 con 0x7fabcc0caac0 2023-10-14T18:17:22.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.695+0000 7fabc8ff9640 1 -- 172.21.15.29:0/987245106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v225) v1 ==== 97+0+0 (secure 0 0 0) 0x7fabb4057780 con 0x7fabcc0caac0 2023-10-14T18:17:22.696 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T18:17:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 msgr2=0x7fabac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 0x7fabac06d480 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fabbc000f60 tx=0x7fabbc007e40 comp rx=0 tx=0).stop 2023-10-14T18:17:22.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 msgr2=0x7fabcc0c24a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 secure :-1 s=READY pgs=2549 cs=0 l=1 rev1=1 crypto rx=0x7fabb400c450 tx=0x7fabb400c910 comp rx=0 tx=0).stop 2023-10-14T18:17:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 shutdown_connections 2023-10-14T18:17:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fabac06afc0 0x7fabac06d480 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fabcc15ad70 0x7fabcc0c8470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fabcc0caac0 0x7fabcc0c24a0 unknown :-1 s=CLOSED pgs=2549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 --2- 172.21.15.29:0/987245106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fabcc06a4b0 0x7fabcc0c1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 >> 172.21.15.29:0/987245106 conn(0x7fabcc0ce9b0 msgr2=0x7fabcc15d460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 shutdown_connections 2023-10-14T18:17:22.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.699+0000 7fabd20e7640 1 -- 172.21.15.29:0/987245106 wait complete. 2023-10-14T18:17:22.720 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T18:17:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.933+0000 7f50e1187640 1 Processor -- start 2023-10-14T18:17:22.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.933+0000 7f50e1187640 1 -- start start 2023-10-14T18:17:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e1187640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 0x7f50dc0b0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e1187640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50dc15ecb0 0x7f50dc06f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e1187640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc06f1d0 con 0x7f50dc144440 2023-10-14T18:17:22.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e1187640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50dc0cd430 con 0x7f50dc15ecb0 2023-10-14T18:17:22.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e1187640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50dc0cd610 con 0x7f50dc144060 2023-10-14T18:17:22.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.935+0000 7f50e0986640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50dc15ecb0 0x7f50dc06f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50dc15ecb0 0x7f50dc06f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38222/0 (socket says 172.21.15.29:38222) 2023-10-14T18:17:22.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 -- 172.21.15.29:0/1977249311 learned_addr learned my addr 172.21.15.29:0/1977249311 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:22.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50dbfff640 1 --2- 172.21.15.29:0/1977249311 >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 0x7f50dc0b0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 -- 172.21.15.29:0/1977249311 >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 msgr2=0x7f50dc0b0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:22.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 --2- 172.21.15.29:0/1977249311 >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 0x7f50dc0b0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 -- 172.21.15.29:0/1977249311 >> v1:172.21.15.186:6789/0 conn(0x7f50dc144440 legacy=0x7f50dc15e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50e0986640 1 -- 172.21.15.29:0/1977249311 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0cd8f0 con 0x7f50dc15ecb0 2023-10-14T18:17:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.936+0000 7f50dbfff640 1 --2- 172.21.15.29:0/1977249311 >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 0x7f50dc0b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.938+0000 7f50e0986640 1 --2- 172.21.15.29:0/1977249311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc15ecb0 0x7f50dc06f6b0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f50dc071420 tx=0x7f50d402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1da1c909daa49af0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:22.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.938+0000 7f50db7fe640 1 -- 172.21.15.29:0/1977249311 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d40317e0 con 0x7f50dc15ecb0 2023-10-14T18:17:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.938+0000 7f50db7fe640 1 -- 172.21.15.29:0/1977249311 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50d4031940 con 0x7f50dc15ecb0 2023-10-14T18:17:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.938+0000 7f50db7fe640 1 -- 172.21.15.29:0/1977249311 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50d4031cd0 con 0x7f50dc15ecb0 2023-10-14T18:17:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 -- 172.21.15.29:0/1977249311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc15ecb0 msgr2=0x7f50dc06f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 --2- 172.21.15.29:0/1977249311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc15ecb0 0x7f50dc06f6b0 secure :-1 s=READY pgs=2550 cs=0 l=1 rev1=1 crypto rx=0x7f50dc071420 tx=0x7f50d402b880 comp rx=0 tx=0).stop 2023-10-14T18:17:22.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 -- 172.21.15.29:0/1977249311 shutdown_connections 2023-10-14T18:17:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 --2- 172.21.15.29:0/1977249311 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc15ecb0 0x7f50dc06f6b0 unknown :-1 s=CLOSED pgs=2550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 --2- 172.21.15.29:0/1977249311 >> v2:172.21.15.186:3300/0 conn(0x7f50dc144060 0x7f50dc0b0a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 -- 172.21.15.29:0/1977249311 >> 172.21.15.29:0/1977249311 conn(0x7f50dc069da0 msgr2=0x7f50dc06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 -- 172.21.15.29:0/1977249311 shutdown_connections 2023-10-14T18:17:22.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.939+0000 7f50e1187640 1 -- 172.21.15.29:0/1977249311 wait complete. 2023-10-14T18:17:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.941+0000 7f50e1187640 1 Processor -- start 2023-10-14T18:17:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.941+0000 7f50e1187640 1 -- start start 2023-10-14T18:17:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50dc144060 0x7f50dc1576b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50dc15ecb0 0x7f50dc134550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc13e420 con 0x7f50dc144440 2023-10-14T18:17:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc13e700 con 0x7f50dc144060 2023-10-14T18:17:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e1187640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50dc13e9e0 con 0x7f50dc15ecb0 2023-10-14T18:17:22.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50d37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50d37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38226/0 (socket says 172.21.15.29:38226) 2023-10-14T18:17:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50d37fe640 1 -- 172.21.15.29:0/304962523 learned_addr learned my addr 172.21.15.29:0/304962523 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50dbfff640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50dc144060 0x7f50dc1576b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50d37fe640 1 -- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50dc15ecb0 msgr2=0x7f50dc134550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:22.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50e0986640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50dc15ecb0 0x7f50dc134550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.943+0000 7f50d37fe640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50dc15ecb0 0x7f50dc134550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d37fe640 1 -- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50dc144060 msgr2=0x7f50dc1576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d37fe640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50dc144060 0x7f50dc1576b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d37fe640 1 -- 172.21.15.29:0/304962523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d40094d0 con 0x7f50dc144440 2023-10-14T18:17:22.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d37fe640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c3f0 tx=0x7f50c400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c4016070 con 0x7f50dc144440 2023-10-14T18:17:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50c4007020 con 0x7f50dc144440 2023-10-14T18:17:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50c40073b0 con 0x7f50dc144440 2023-10-14T18:17:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc13ecc0 con 0x7f50dc144440 2023-10-14T18:17:22.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.944+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc13e0d0 con 0x7f50dc144440 2023-10-14T18:17:22.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.946+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc13f1e0 con 0x7f50dc144440 2023-10-14T18:17:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.947+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f50c4007510 con 0x7f50dc144440 2023-10-14T18:17:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.947+0000 7f50d97fa640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 0x7f50bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.948+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f50c4088de0 con 0x7f50dc144440 2023-10-14T18:17:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.948+0000 7f50dbfff640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 0x7f50bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:22.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.948+0000 7f50dbfff640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 0x7f50bc06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0075b0 tx=0x7f50cc007540 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:22.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:22.952+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50dc13f1e0 con 0x7f50dc144440 2023-10-14T18:17:23.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.221+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f50dc13f1e0 con 0x7f50dc144440 2023-10-14T18:17:23.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.222+0000 7f50d97fa640 1 -- 172.21.15.29:0/304962523 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v50) v1 ==== 80+0+5304 (secure 0 0 0) 0x7f50dc13f1e0 con 0x7f50dc144440 2023-10-14T18:17:23.223 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:23.223 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:17:23.223 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T18:17:23.223 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.224 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T18:17:23.224 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T18:17:23.224 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.224 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.224 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:17:23.225 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:17:23.225 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.225 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.225 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.225 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.226 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T18:17:23.226 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T18:17:23.226 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.226 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.227 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.228 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T18:17:23.228 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T18:17:23.228 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.228 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:17:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:17:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.229 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.230 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T18:17:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T18:17:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:17:23.231 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:17:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.232 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T18:17:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T18:17:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.233 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.234 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T18:17:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T18:17:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.235 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.236 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T18:17:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T18:17:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.237 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T18:17:23.238 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T18:17:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.239 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.240 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.241 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T18:17:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T18:17:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.242 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.243 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T18:17:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T18:17:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.244 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.245 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T18:17:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T18:17:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:17:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:17:23.246 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T18:17:23.247 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T18:17:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T18:17:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.248 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T18:17:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T18:17:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.249 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:17:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:17:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:17:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.250 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T18:17:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T18:17:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.251 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:17:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:17:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:17:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.252 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T18:17:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T18:17:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.253 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.254 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T18:17:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T18:17:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:17:23.255 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:17:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:17:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.256 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T18:17:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T18:17:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:17:23.257 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:17:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:23.258 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T18:17:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T18:17:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:17:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:17:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:17:23.259 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:17:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:23.260 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T18:17:23.260 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:17:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.224+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 msgr2=0x7f50bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.224+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 0x7f50bc06d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f50cc0075b0 tx=0x7f50cc007540 comp rx=0 tx=0).stop 2023-10-14T18:17:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.224+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 msgr2=0x7f50dc141650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.224+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 secure :-1 s=READY pgs=2551 cs=0 l=1 rev1=1 crypto rx=0x7f50c400c3f0 tx=0x7f50c400c8b0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 shutdown_connections 2023-10-14T18:17:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f50bc06af90 0x7f50bc06d450 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50dc15ecb0 0x7f50dc134550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50dc144440 0x7f50dc141650 unknown :-1 s=CLOSED pgs=2551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 --2- 172.21.15.29:0/304962523 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50dc144060 0x7f50dc1576b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 >> 172.21.15.29:0/304962523 conn(0x7f50dc069da0 msgr2=0x7f50dc0ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 shutdown_connections 2023-10-14T18:17:23.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.225+0000 7f50e1187640 1 -- 172.21.15.29:0/304962523 wait complete. 2023-10-14T18:17:23.265 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:23.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.499+0000 7f040ae75640 1 Processor -- start 2023-10-14T18:17:23.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.499+0000 7f040ae75640 1 -- start start 2023-10-14T18:17:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.500+0000 7f040ae75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.500+0000 7f040ae75640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f040406c810 0x7f04040708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.500+0000 7f040ae75640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0404070df0 con 0x7f0404070450 2023-10-14T18:17:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.500+0000 7f040ae75640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f040415e490 con 0x7f040406c810 2023-10-14T18:17:23.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.500+0000 7f040ae75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f040415aef0 con 0x7f0404159950 2023-10-14T18:17:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.501+0000 7f0409e73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.501+0000 7f0409e73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:38450/0 (socket says 172.21.15.29:38450) 2023-10-14T18:17:23.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.501+0000 7f0409e73640 1 -- 172.21.15.29:0/1191228141 learned_addr learned my addr 172.21.15.29:0/1191228141 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.501+0000 7f040a674640 1 --2- 172.21.15.29:0/1191228141 >> v2:172.21.15.186:3300/0 conn(0x7f040406c810 0x7f04040708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.502+0000 7f0409e73640 1 -- 172.21.15.29:0/1191228141 >> v2:172.21.15.186:3300/0 conn(0x7f040406c810 msgr2=0x7f04040708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.502+0000 7f0409e73640 1 --2- 172.21.15.29:0/1191228141 >> v2:172.21.15.186:3300/0 conn(0x7f040406c810 0x7f04040708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.502+0000 7f0409e73640 1 -- 172.21.15.29:0/1191228141 >> v1:172.21.15.186:6789/0 conn(0x7f0404070450 legacy=0x7f040406c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.502+0000 7f0409e73640 1 -- 172.21.15.29:0/1191228141 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f040406cc80 con 0x7f0404159950 2023-10-14T18:17:23.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.502+0000 7f0408e71640 1 -- 172.21.15.29:0/1191228141 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3258264731 0 0) 0x7f0404070df0 con 0x7f0404070450 2023-10-14T18:17:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.503+0000 7f0409e73640 1 --2- 172.21.15.29:0/1191228141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0400009870 tx=0x7f040002b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3fafb94554c2a2c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.504+0000 7f0408e71640 1 -- 172.21.15.29:0/1191228141 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400031640 con 0x7f0404159950 2023-10-14T18:17:23.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.504+0000 7f0408e71640 1 -- 172.21.15.29:0/1191228141 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04000317a0 con 0x7f0404159950 2023-10-14T18:17:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.504+0000 7f0408e71640 1 -- 172.21.15.29:0/1191228141 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0400031b30 con 0x7f0404159950 2023-10-14T18:17:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 -- 172.21.15.29:0/1191228141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 msgr2=0x7f040415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 --2- 172.21.15.29:0/1191228141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0400009870 tx=0x7f040002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 -- 172.21.15.29:0/1191228141 shutdown_connections 2023-10-14T18:17:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 --2- 172.21.15.29:0/1191228141 >> v2:172.21.15.186:3300/0 conn(0x7f040406c810 0x7f04040708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 --2- 172.21.15.29:0/1191228141 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f040415a9b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 -- 172.21.15.29:0/1191228141 >> 172.21.15.29:0/1191228141 conn(0x7f04040ce6a0 msgr2=0x7f04040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 -- 172.21.15.29:0/1191228141 shutdown_connections 2023-10-14T18:17:23.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.505+0000 7f040ae75640 1 -- 172.21.15.29:0/1191228141 wait complete. 2023-10-14T18:17:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.507+0000 7f040ae75640 1 Processor -- start 2023-10-14T18:17:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.507+0000 7f040ae75640 1 -- start start 2023-10-14T18:17:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0404070450 0x7f040414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f04041502c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0404150800 con 0x7f040406c810 2023-10-14T18:17:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0404153ec0 con 0x7f0404070450 2023-10-14T18:17:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f040ae75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04040b14f0 con 0x7f0404159950 2023-10-14T18:17:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f0409e73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f0409e73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38230/0 (socket says 172.21.15.29:38230) 2023-10-14T18:17:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.508+0000 7f0409e73640 1 -- 172.21.15.29:0/45697393 learned_addr learned my addr 172.21.15.29:0/45697393 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409672640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0404070450 0x7f040414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f040a674640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f04041502c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 -- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 msgr2=0x7f04041502c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f04041502c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 -- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0404070450 msgr2=0x7f040414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0404070450 0x7f040414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 -- 172.21.15.29:0/45697393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04000094d0 con 0x7f040406c810 2023-10-14T18:17:23.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.509+0000 7f0409e73640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f04000090b0 tx=0x7f04000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.510+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040003b940 con 0x7f040406c810 2023-10-14T18:17:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.510+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f040003bac0 con 0x7f040406c810 2023-10-14T18:17:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.510+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04040b17d0 con 0x7f040406c810 2023-10-14T18:17:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.510+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f040003be70 con 0x7f040406c810 2023-10-14T18:17:23.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.510+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04040b1d90 con 0x7f040406c810 2023-10-14T18:17:23.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.511+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04040b1f80 con 0x7f040406c810 2023-10-14T18:17:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.512+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f040002f050 con 0x7f040406c810 2023-10-14T18:17:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.513+0000 7f03fa7fc640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.513+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f04000ab700 con 0x7f040406c810 2023-10-14T18:17:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.513+0000 7f0409672640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.514+0000 7f0409672640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 0x7f03e806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04040c8870 tx=0x7f03f4009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.517+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0400076890 con 0x7f040406c810 2023-10-14T18:17:23.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.719+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f040404e9b0 con 0x7f040406c810 2023-10-14T18:17:23.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.720+0000 7f03fa7fc640 1 -- 172.21.15.29:0/45697393 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v225) v1 ==== 74+0+13041 (secure 0 0 0) 0x7f0400079eb0 con 0x7f040406c810 2023-10-14T18:17:23.721 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:23.721 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":225,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:22.692650+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":25,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":212,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":208,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.722+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 msgr2=0x7f03e806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.722+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 0x7f03e806d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f04040c8870 tx=0x7f03f4009040 comp rx=0 tx=0).stop 2023-10-14T18:17:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.722+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 msgr2=0x7f04040c75b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.722+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 secure :-1 s=READY pgs=2552 cs=0 l=1 rev1=1 crypto rx=0x7f04000090b0 tx=0x7f04000046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 shutdown_connections 2023-10-14T18:17:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f03e806af90 0x7f03e806d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0404159950 0x7f04041502c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0404070450 0x7f040414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 --2- 172.21.15.29:0/45697393 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f040406c810 0x7f04040c75b0 unknown :-1 s=CLOSED pgs=2552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 >> 172.21.15.29:0/45697393 conn(0x7f04040ce6a0 msgr2=0x7f04040ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 shutdown_connections 2023-10-14T18:17:23.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.723+0000 7f040ae75640 1 -- 172.21.15.29:0/45697393 wait complete. 2023-10-14T18:17:23.744 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:17:23.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:17:23.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.975+0000 7fce8db01640 1 Processor -- start 2023-10-14T18:17:23.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.975+0000 7fce8db01640 1 -- start start 2023-10-14T18:17:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce8db01640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fce8815ad70 0x7fce88069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce8db01640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce8806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce8db01640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce880cb030 con 0x7fce880c9ca0 2023-10-14T18:17:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce8db01640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce880c91c0 con 0x7fce8815ad70 2023-10-14T18:17:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce8db01640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c94a0 con 0x7fce880caac0 2023-10-14T18:17:23.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce7effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fce880c9ca0 0x7fce880ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55898/0 (socket says 172.21.15.29:55898) 2023-10-14T18:17:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce7effd640 1 -- 172.21.15.29:0/1429907484 learned_addr learned my addr 172.21.15.29:0/1429907484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce877fe640 1 --2- 172.21.15.29:0/1429907484 >> v2:172.21.15.186:3300/0 conn(0x7fce8815ad70 0x7fce88069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.977+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1277200546 0 0) 0x7fce880cb030 con 0x7fce880c9ca0 2023-10-14T18:17:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce87fff640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce8806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce74003520 con 0x7fce880c9ca0 2023-10-14T18:17:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1595224559 0 0) 0x7fce74003520 con 0x7fce880c9ca0 2023-10-14T18:17:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 msgr2=0x7fce8806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce8806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 >> v2:172.21.15.186:3300/0 conn(0x7fce8815ad70 msgr2=0x7fce88069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:23.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 --2- 172.21.15.29:0/1429907484 >> v2:172.21.15.186:3300/0 conn(0x7fce8815ad70 0x7fce88069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880c9780 con 0x7fce880c9ca0 2023-10-14T18:17:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fce70001670 con 0x7fce880c9ca0 2023-10-14T18:17:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 >> v1:172.21.15.29:6789/0 conn(0x7fce880c9ca0 legacy=0x7fce880ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce74003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce74003f40 0x7fce740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c91c0 con 0x7fce74004d00 2023-10-14T18:17:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c94a0 con 0x7fce74003f40 2023-10-14T18:17:23.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.978+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce7400b980 con 0x7fce880caac0 2023-10-14T18:17:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce877fe640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce74003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce7effd640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce74003f40 0x7fce740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 -- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 msgr2=0x7fce74003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:23.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880caac0 0x7fce74003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 -- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce74003f40 msgr2=0x7fce740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce74003f40 0x7fce740047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.979+0000 7fce87fff640 1 -- 172.21.15.29:0/1429907484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce74003520 con 0x7fce74004d00 2023-10-14T18:17:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.980+0000 7fce87fff640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce74005180 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fce80009af0 tx=0x7fce740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26f0a56a666a8d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8002c6b0 con 0x7fce74004d00 2023-10-14T18:17:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce8002c810 con 0x7fce74004d00 2023-10-14T18:17:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce86ffd640 1 -- 172.21.15.29:0/1429907484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8002cba0 con 0x7fce74004d00 2023-10-14T18:17:23.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 -- 172.21.15.29:0/1429907484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 msgr2=0x7fce74005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 --2- 172.21.15.29:0/1429907484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce74005180 secure :-1 s=READY pgs=2554 cs=0 l=1 rev1=1 crypto rx=0x7fce80009af0 tx=0x7fce740057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce87fff640 1 -- 172.21.15.29:0/1429907484 reap_dead start 2023-10-14T18:17:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 -- 172.21.15.29:0/1429907484 shutdown_connections 2023-10-14T18:17:23.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 -- 172.21.15.29:0/1429907484 >> 172.21.15.29:0/1429907484 conn(0x7fce880ce9b0 msgr2=0x7fce880cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 -- 172.21.15.29:0/1429907484 shutdown_connections 2023-10-14T18:17:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.981+0000 7fce8db01640 1 -- 172.21.15.29:0/1429907484 wait complete. 2023-10-14T18:17:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.983+0000 7fce8db01640 1 Processor -- start 2023-10-14T18:17:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.983+0000 7fce8db01640 1 -- start start 2023-10-14T18:17:23.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce880c4600 0x7fce881f0a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880cd700 0x7fce880cdb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c5710 con 0x7fce74004d00 2023-10-14T18:17:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce880ce0c0 con 0x7fce880c4600 2023-10-14T18:17:23.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce8db01640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce880c4b90 con 0x7fce880cd700 2023-10-14T18:17:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce877fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38240/0 (socket says 172.21.15.29:38240) 2023-10-14T18:17:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.984+0000 7fce877fe640 1 -- 172.21.15.29:0/2912978732 learned_addr learned my addr 172.21.15.29:0/2912978732 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:23.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 -- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880cd700 msgr2=0x7fce880cdb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880cd700 0x7fce880cdb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 -- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce880c4600 msgr2=0x7fce881f0a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce880c4600 0x7fce881f0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 -- 172.21.15.29:0/2912978732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce880c4e70 con 0x7fce74004d00 2023-10-14T18:17:23.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.985+0000 7fce877fe640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fce78007f30 tx=0x7fce78011660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.986+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce7800e030 con 0x7fce74004d00 2023-10-14T18:17:23.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.986+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce78016470 con 0x7fce74004d00 2023-10-14T18:17:23.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.986+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce880c5150 con 0x7fce74004d00 2023-10-14T18:17:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.986+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce78016820 con 0x7fce74004d00 2023-10-14T18:17:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.986+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce880c61e0 con 0x7fce74004d00 2023-10-14T18:17:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.987+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce88156de0 con 0x7fce74004d00 2023-10-14T18:17:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.988+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fce78016980 con 0x7fce74004d00 2023-10-14T18:17:23.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.989+0000 7fce84ff9640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 0x7fce6806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.989+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fce7800f070 con 0x7fce74004d00 2023-10-14T18:17:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.989+0000 7fce7effd640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 0x7fce6806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.990+0000 7fce7effd640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 0x7fce6806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fce880af0f0 tx=0x7fce70009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:23.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:23.994+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fce780556a0 con 0x7fce74004d00 2023-10-14T18:17:24.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.190+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fce8804e160 con 0x7fce74004d00 2023-10-14T18:17:24.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.190+0000 7fce84ff9640 1 -- 172.21.15.29:0/2912978732 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v225) v1 ==== 93+0+10 (secure 0 0 0) 0x7fce78058cc0 con 0x7fce74004d00 2023-10-14T18:17:24.192 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:17:24.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 msgr2=0x7fce6806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 0x7fce6806d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fce880af0f0 tx=0x7fce70009040 comp rx=0 tx=0).stop 2023-10-14T18:17:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 msgr2=0x7fce880c81a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 secure :-1 s=READY pgs=2555 cs=0 l=1 rev1=1 crypto rx=0x7fce78007f30 tx=0x7fce78011660 comp rx=0 tx=0).stop 2023-10-14T18:17:24.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 shutdown_connections 2023-10-14T18:17:24.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fce6806af90 0x7fce6806d450 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fce880cd700 0x7fce880cdb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fce880c4600 0x7fce881f0a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 --2- 172.21.15.29:0/2912978732 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fce74004d00 0x7fce880c81a0 unknown :-1 s=CLOSED pgs=2555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.193+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 >> 172.21.15.29:0/2912978732 conn(0x7fce880ce9b0 msgr2=0x7fce88070da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.194+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 shutdown_connections 2023-10-14T18:17:24.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.194+0000 7fce8db01640 1 -- 172.21.15.29:0/2912978732 wait complete. 2023-10-14T18:17:24.214 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:17:24.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.439+0000 7fb69ee02640 1 Processor -- start 2023-10-14T18:17:24.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.439+0000 7fb69ee02640 1 -- start start 2023-10-14T18:17:24.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69ee02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6980b0b20 0x7fb6981615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69ee02640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb69815d3f0 0x7fb69815d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69ee02640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb69815e3e0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69ee02640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6980cd4a0 con 0x7fb69815d3f0 2023-10-14T18:17:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69ee02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb69815e100 con 0x7fb6980b0b20 2023-10-14T18:17:24.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69cb9e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb6980cd0c0 0x7fb6980b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55914/0 (socket says 172.21.15.29:55914) 2023-10-14T18:17:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69cb9e640 1 -- 172.21.15.29:0/3168908585 learned_addr learned my addr 172.21.15.29:0/3168908585 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:24.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.441+0000 7fb69d39f640 1 --2- 172.21.15.29:0/3168908585 >> v2:172.21.15.29:3300/0 conn(0x7fb69815d3f0 0x7fb69815d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.442+0000 7fb697fff640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6980b0b20 0x7fb6981615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.442+0000 7fb69d39f640 1 -- 172.21.15.29:0/3168908585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6980b0b20 msgr2=0x7fb6981615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:24.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.442+0000 7fb69d39f640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6980b0b20 0x7fb6981615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.442+0000 7fb69d39f640 1 -- 172.21.15.29:0/3168908585 >> v1:172.21.15.29:6789/0 conn(0x7fb6980cd0c0 legacy=0x7fb6980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.442+0000 7fb69d39f640 1 -- 172.21.15.29:0/3168908585 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb698160b10 con 0x7fb69815d3f0 2023-10-14T18:17:24.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.443+0000 7fb69d39f640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb69815d3f0 0x7fb69815d860 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69002be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3fc9180e57a2a2a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.444+0000 7fb6977fe640 1 -- 172.21.15.29:0/3168908585 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69002cbd0 con 0x7fb69815d3f0 2023-10-14T18:17:24.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.444+0000 7fb6977fe640 1 -- 172.21.15.29:0/3168908585 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb69002cd30 con 0x7fb69815d3f0 2023-10-14T18:17:24.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.444+0000 7fb6977fe640 1 -- 172.21.15.29:0/3168908585 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb69002cea0 con 0x7fb69815d3f0 2023-10-14T18:17:24.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.444+0000 7fb69ee02640 1 -- 172.21.15.29:0/3168908585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb69815d3f0 msgr2=0x7fb69815d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.444+0000 7fb69ee02640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb69815d3f0 0x7fb69815d860 secure :-1 s=READY pgs=2557 cs=0 l=1 rev1=1 crypto rx=0x7fb690009d90 tx=0x7fb69002be20 comp rx=0 tx=0).stop 2023-10-14T18:17:24.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 -- 172.21.15.29:0/3168908585 shutdown_connections 2023-10-14T18:17:24.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb69815d3f0 0x7fb69815d860 unknown :-1 s=CLOSED pgs=2557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 --2- 172.21.15.29:0/3168908585 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6980b0b20 0x7fb6981615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 -- 172.21.15.29:0/3168908585 >> 172.21.15.29:0/3168908585 conn(0x7fb6980cb410 msgr2=0x7fb6980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:24.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 -- 172.21.15.29:0/3168908585 shutdown_connections 2023-10-14T18:17:24.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.445+0000 7fb69ee02640 1 -- 172.21.15.29:0/3168908585 wait complete. 2023-10-14T18:17:24.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.446+0000 7fb69ee02640 1 Processor -- start 2023-10-14T18:17:24.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.447+0000 7fb69ee02640 1 -- start start 2023-10-14T18:17:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.447+0000 7fb69ee02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6980b0b20 0x7fb6980cf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.447+0000 7fb69ee02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69ee02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb69815d3f0 0x7fb698156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69ee02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6981607f0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69ee02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6980cf5e0 con 0x7fb6980b0b20 2023-10-14T18:17:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69ee02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb698156f90 con 0x7fb69815d3f0 2023-10-14T18:17:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38270/0 (socket says 172.21.15.29:38270) 2023-10-14T18:17:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 -- 172.21.15.29:0/137166856 learned_addr learned my addr 172.21.15.29:0/137166856 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69d39f640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb69815d3f0 0x7fb698156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 -- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb69815d3f0 msgr2=0x7fb698156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb69cb9e640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6980b0b20 0x7fb6980cf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb69815d3f0 0x7fb698156050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 -- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6980b0b20 msgr2=0x7fb6980cf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6980b0b20 0x7fb6980cf0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.448+0000 7fb697fff640 1 -- 172.21.15.29:0/137166856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb690009a70 con 0x7fb6980cd0c0 2023-10-14T18:17:24.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb697fff640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fb68400d5f0 tx=0x7fb68400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6840125f0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb684012750 con 0x7fb6980cd0c0 2023-10-14T18:17:24.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb684012b00 con 0x7fb6980cd0c0 2023-10-14T18:17:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6980c8310 con 0x7fb6980cd0c0 2023-10-14T18:17:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.449+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6980c85b0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.451+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb69804e160 con 0x7fb6980cd0c0 2023-10-14T18:17:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.451+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb684012c60 con 0x7fb6980cd0c0 2023-10-14T18:17:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.452+0000 7fb6957fa640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 0x7fb67c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.452+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fb6840899b0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.453+0000 7fb69cb9e640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 0x7fb67c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.453+0000 7fb69cb9e640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 0x7fb67c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb68c005880 tx=0x7fb68c009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.457+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb684054c20 con 0x7fb6980cd0c0 2023-10-14T18:17:24.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.657+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb6981610d0 con 0x7fb6980cd0c0 2023-10-14T18:17:24.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.658+0000 7fb6957fa640 1 -- 172.21.15.29:0/137166856 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v225) v1 ==== 84+0+65 (secure 0 0 0) 0x7fb684058240 con 0x7fb6980cd0c0 2023-10-14T18:17:24.659 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:24.660 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:17:24.660 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:24.660 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:17:24.660 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:17:24.661 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:24.661 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:17:24.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 msgr2=0x7fb67c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 0x7fb67c06d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb68c005880 tx=0x7fb68c009040 comp rx=0 tx=0).stop 2023-10-14T18:17:24.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 msgr2=0x7fb69815f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 secure :-1 s=READY pgs=2558 cs=0 l=1 rev1=1 crypto rx=0x7fb68400d5f0 tx=0x7fb68400dab0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 shutdown_connections 2023-10-14T18:17:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb67c06af90 0x7fb67c06d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb69815d3f0 0x7fb698156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6980cd0c0 0x7fb69815f850 unknown :-1 s=CLOSED pgs=2558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 --2- 172.21.15.29:0/137166856 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6980b0b20 0x7fb6980cf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.661+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 >> 172.21.15.29:0/137166856 conn(0x7fb6980cb410 msgr2=0x7fb6980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.662+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 shutdown_connections 2023-10-14T18:17:24.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.662+0000 7fb69ee02640 1 -- 172.21.15.29:0/137166856 wait complete. 2023-10-14T18:17:24.683 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T18:17:24.685 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T18:17:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.921+0000 7f0475a59640 1 Processor -- start 2023-10-14T18:17:24.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.921+0000 7f0475a59640 1 -- start start 2023-10-14T18:17:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.922+0000 7f0475a59640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f047006d280 0x7f04700c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f0475a59640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700c8e70 con 0x7f04700cae40 2023-10-14T18:17:24.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f0475a59640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04700b4170 con 0x7f047015ad80 2023-10-14T18:17:24.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f0475a59640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04700adde0 con 0x7f047006d280 2023-10-14T18:17:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f0475258640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f04700cae40 0x7f04700c9840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55924/0 (socket says 172.21.15.29:55924) 2023-10-14T18:17:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f0475258640 1 -- 172.21.15.29:0/1734212676 learned_addr learned my addr 172.21.15.29:0/1734212676 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f046ffff640 1 --2- 172.21.15.29:0/1734212676 >> v2:172.21.15.186:3300/0 conn(0x7f047006d280 0x7f04700c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.923+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690679465 0 0) 0x7f04700c8e70 con 0x7f04700cae40 2023-10-14T18:17:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0460003520 con 0x7f04700cae40 2023-10-14T18:17:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 143900921 0 0) 0x7f0460003520 con 0x7f04700cae40 2023-10-14T18:17:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 >> v2:172.21.15.186:3300/0 conn(0x7f047006d280 msgr2=0x7f04700c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 --2- 172.21.15.29:0/1734212676 >> v2:172.21.15.186:3300/0 conn(0x7f047006d280 0x7f04700c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 >> v1:172.21.15.186:6789/0 conn(0x7f047015ad80 legacy=0x7f047006cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700b4410 con 0x7f04700cae40 2023-10-14T18:17:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f04680019f0 con 0x7f04700cae40 2023-10-14T18:17:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 >> v1:172.21.15.29:6789/0 conn(0x7f04700cae40 legacy=0x7f04700c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0460003a30 0x7f0460003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04600047c0 0x7f0460004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f046000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700adde0 con 0x7f04600051d0 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f046000c100 con 0x7f04600047c0 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.924+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f046000c2c0 con 0x7f0460003a30 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f046000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f046ffff640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04600047c0 0x7f0460004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0474a57640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0460003a30 0x7f0460003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 -- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0460003a30 msgr2=0x7f0460003e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0460003a30 0x7f0460003e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 -- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04600047c0 msgr2=0x7f0460004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:24.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04600047c0 0x7f0460004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0475258640 1 -- 172.21.15.29:0/1734212676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0460003520 con 0x7f04600051d0 2023-10-14T18:17:24.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f046ffff640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04600047c0 0x7f0460004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:24.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.925+0000 7f0474a57640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0460003a30 0x7f0460003e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:24.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.926+0000 7f0475258640 1 -- 172.21.15.29:0/1734212676 reap_dead start 2023-10-14T18:17:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.926+0000 7f0475258640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f046000b980 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f04700cb270 tx=0x7f04600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d128b8cdcc5a3c95 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.926+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468034630 con 0x7f04600051d0 2023-10-14T18:17:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.926+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f046802f070 con 0x7f04600051d0 2023-10-14T18:17:24.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.926+0000 7f046f7fe640 1 -- 172.21.15.29:0/1734212676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0468034a10 con 0x7f04600051d0 2023-10-14T18:17:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 -- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 msgr2=0x7f046000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f046000b980 secure :-1 s=READY pgs=2560 cs=0 l=1 rev1=1 crypto rx=0x7f04700cb270 tx=0x7f04600057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 -- 172.21.15.29:0/1734212676 shutdown_connections 2023-10-14T18:17:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 --2- 172.21.15.29:0/1734212676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f046000b980 unknown :-1 s=CLOSED pgs=2560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 -- 172.21.15.29:0/1734212676 >> 172.21.15.29:0/1734212676 conn(0x7f04700ce820 msgr2=0x7f04700cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 -- 172.21.15.29:0/1734212676 shutdown_connections 2023-10-14T18:17:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.927+0000 7f0475a59640 1 -- 172.21.15.29:0/1734212676 wait complete. 2023-10-14T18:17:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.929+0000 7f0475a59640 1 Processor -- start 2023-10-14T18:17:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.929+0000 7f0475a59640 1 -- start start 2023-10-14T18:17:24.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04700c7830 0x7f04700c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04700c3e20 0x7f04700ae570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700c2dc0 con 0x7f04600051d0 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04700c3060 con 0x7f04700c3e20 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0475a59640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04700c3340 con 0x7f04700c7830 2023-10-14T18:17:24.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0474a57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0474a57640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38278/0 (socket says 172.21.15.29:38278) 2023-10-14T18:17:24.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.930+0000 7f0474a57640 1 -- 172.21.15.29:0/3300413512 learned_addr learned my addr 172.21.15.29:0/3300413512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 -- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04700c7830 msgr2=0x7f04700c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04700c7830 0x7f04700c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 -- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04700c3e20 msgr2=0x7f04700ae570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04700c3e20 0x7f04700ae570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 -- 172.21.15.29:0/3300413512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04700c3620 con 0x7f04600051d0 2023-10-14T18:17:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0474a57640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f046400d770 tx=0x7f046400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0464005270 con 0x7f04600051d0 2023-10-14T18:17:24.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04640053d0 con 0x7f04600051d0 2023-10-14T18:17:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f046400b8f0 con 0x7f04600051d0 2023-10-14T18:17:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.931+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04700c3900 con 0x7f04600051d0 2023-10-14T18:17:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.932+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04700c42a0 con 0x7f04600051d0 2023-10-14T18:17:24.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.933+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04700c44c0 con 0x7f04600051d0 2023-10-14T18:17:24.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.934+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f046400ba50 con 0x7f04600051d0 2023-10-14T18:17:24.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.935+0000 7f046d7fa640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 0x7f045406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:24.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.935+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(225..225 src has 1..225) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f04640895c0 con 0x7f04600051d0 2023-10-14T18:17:24.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.935+0000 7f046ffff640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 0x7f045406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:24.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.936+0000 7f046ffff640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 0x7f045406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f047014c380 tx=0x7f045c008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:24.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:24.939+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0464054690 con 0x7f04600051d0 2023-10-14T18:17:25.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.138+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f047004e9b0 con 0x7f04600051d0 2023-10-14T18:17:25.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.720+0000 7f046d7fa640 1 -- 172.21.15.29:0/3300413512 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v226) v1 ==== 138+0+0 (secure 0 0 0) 0x7f0464019310 con 0x7f04600051d0 2023-10-14T18:17:25.721 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T18:17:25.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.723+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 msgr2=0x7f045406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.723+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 0x7f045406d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f047014c380 tx=0x7f045c008040 comp rx=0 tx=0).stop 2023-10-14T18:17:25.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.723+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 msgr2=0x7f047014bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.723+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 secure :-1 s=READY pgs=2561 cs=0 l=1 rev1=1 crypto rx=0x7f046400d770 tx=0x7f046400dc30 comp rx=0 tx=0).stop 2023-10-14T18:17:25.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 shutdown_connections 2023-10-14T18:17:25.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f045406af90 0x7f045406d450 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f04700c3e20 0x7f04700ae570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f04700c7830 0x7f04700c7f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 --2- 172.21.15.29:0/3300413512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f04600051d0 0x7f047014bd00 unknown :-1 s=CLOSED pgs=2561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 >> 172.21.15.29:0/3300413512 conn(0x7f04700ce820 msgr2=0x7f04700af6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:25.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 shutdown_connections 2023-10-14T18:17:25.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.724+0000 7f0475a59640 1 -- 172.21.15.29:0/3300413512 wait complete. 2023-10-14T18:17:25.752 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:17:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.965+0000 7f7de9f0f640 1 Processor -- start 2023-10-14T18:17:25.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.966+0000 7f7de9f0f640 1 -- start start 2023-10-14T18:17:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.967+0000 7f7de9f0f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7de415aaa0 0x7f7de415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.967+0000 7f7de9f0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.967+0000 7f7de9f0f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de40707e0 con 0x7f7de40ccaa0 2023-10-14T18:17:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.967+0000 7f7de9f0f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de406b7a0 con 0x7f7de415aaa0 2023-10-14T18:17:25.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.967+0000 7f7de9f0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de415ef20 con 0x7f7de415ae80 2023-10-14T18:17:25.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.968+0000 7f7de2ffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.968+0000 7f7de2ffd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:38554/0 (socket says 172.21.15.29:38554) 2023-10-14T18:17:25.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.968+0000 7f7de2ffd640 1 -- 172.21.15.29:0/2696832929 learned_addr learned my addr 172.21.15.29:0/2696832929 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.968+0000 7f7de37fe640 1 --2- 172.21.15.29:0/2696832929 >> v2:172.21.15.186:3300/0 conn(0x7f7de415aaa0 0x7f7de415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.969+0000 7f7de2ffd640 1 -- 172.21.15.29:0/2696832929 >> v2:172.21.15.186:3300/0 conn(0x7f7de415aaa0 msgr2=0x7f7de415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.969+0000 7f7de2ffd640 1 --2- 172.21.15.29:0/2696832929 >> v2:172.21.15.186:3300/0 conn(0x7f7de415aaa0 0x7f7de415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.969+0000 7f7de2ffd640 1 -- 172.21.15.29:0/2696832929 >> v1:172.21.15.186:6789/0 conn(0x7f7de40ccaa0 legacy=0x7f7de4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.969+0000 7f7de2ffd640 1 -- 172.21.15.29:0/2696832929 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de415f200 con 0x7f7de415ae80 2023-10-14T18:17:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.971+0000 7f7de2ffd640 1 --2- 172.21.15.29:0/2696832929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009870 tx=0x7f7dcc02b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6107cc7789e3ecd1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.971+0000 7f7de27fc640 1 -- 172.21.15.29:0/2696832929 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc031640 con 0x7f7de415ae80 2023-10-14T18:17:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.971+0000 7f7de27fc640 1 -- 172.21.15.29:0/2696832929 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc0317a0 con 0x7f7de415ae80 2023-10-14T18:17:25.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.971+0000 7f7de27fc640 1 -- 172.21.15.29:0/2696832929 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc031b30 con 0x7f7de415ae80 2023-10-14T18:17:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 -- 172.21.15.29:0/2696832929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 msgr2=0x7f7de40cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/2696832929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc009870 tx=0x7f7dcc02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 -- 172.21.15.29:0/2696832929 shutdown_connections 2023-10-14T18:17:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/2696832929 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de415ae80 0x7f7de40cc560 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/2696832929 >> v2:172.21.15.186:3300/0 conn(0x7f7de415aaa0 0x7f7de415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 -- 172.21.15.29:0/2696832929 >> 172.21.15.29:0/2696832929 conn(0x7f7de40cb410 msgr2=0x7f7de40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 -- 172.21.15.29:0/2696832929 shutdown_connections 2023-10-14T18:17:25.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.972+0000 7f7de9f0f640 1 -- 172.21.15.29:0/2696832929 wait complete. 2023-10-14T18:17:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.974+0000 7f7de9f0f640 1 Processor -- start 2023-10-14T18:17:25.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.974+0000 7f7de9f0f640 1 -- start start 2023-10-14T18:17:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 0x7f7de40af720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7de415ae80 0x7f7de40ca5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41ef8c0 con 0x7f7de415aaa0 2023-10-14T18:17:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41eec40 con 0x7f7de415ae80 2023-10-14T18:17:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de9f0f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de41eef20 con 0x7f7de40ccaa0 2023-10-14T18:17:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38280/0 (socket says 172.21.15.29:38280) 2023-10-14T18:17:25.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de2ffd640 1 -- 172.21.15.29:0/1763562760 learned_addr learned my addr 172.21.15.29:0/1763562760 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.975+0000 7f7de37fe640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 0x7f7de40af720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de3fff640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7de415ae80 0x7f7de40ca5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 -- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 msgr2=0x7f7de40af720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 0x7f7de40af720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 -- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7de415ae80 msgr2=0x7f7de40ca5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7de415ae80 0x7f7de40ca5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 -- 172.21.15.29:0/1763562760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dcc0094d0 con 0x7f7de415aaa0 2023-10-14T18:17:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de37fe640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 0x7f7de40af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7de2ffd640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc000c00 tx=0x7f7dcc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:25.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.976+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc03a930 con 0x7f7de415aaa0 2023-10-14T18:17:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.977+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc03aa90 con 0x7f7de415aaa0 2023-10-14T18:17:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.977+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de41ef200 con 0x7f7de415aaa0 2023-10-14T18:17:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.977+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc03ae40 con 0x7f7de415aaa0 2023-10-14T18:17:25.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.977+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de4155d20 con 0x7f7de415aaa0 2023-10-14T18:17:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.978+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de404e160 con 0x7f7de415aaa0 2023-10-14T18:17:25.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.980+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7dcc02f050 con 0x7f7de415aaa0 2023-10-14T18:17:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.981+0000 7f7ddbfff640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 0x7f7dc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.982+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(226..226 src has 1..226) v4 ==== 5343+0+0 (secure 0 0 0) 0x7f7dcc0aba50 con 0x7f7de415aaa0 2023-10-14T18:17:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.982+0000 7f7de37fe640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 0x7f7dc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:25.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.982+0000 7f7de37fe640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 0x7f7dc406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40074c0 tx=0x7f7dd400b040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:25.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:25.986+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7dcc076b50 con 0x7f7de415aaa0 2023-10-14T18:17:26.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.187+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f7de40cdc60 con 0x7f7de415aaa0 2023-10-14T18:17:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.727+0000 7f7ddbfff640 1 -- 172.21.15.29:0/1763562760 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v227) v1 ==== 170+0+0 (secure 0 0 0) 0x7f7dcc07a170 con 0x7f7de415aaa0 2023-10-14T18:17:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T18:17:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 msgr2=0x7f7dc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 0x7f7dc406d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40074c0 tx=0x7f7dd400b040 comp rx=0 tx=0).stop 2023-10-14T18:17:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 msgr2=0x7f7de41557e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 secure :-1 s=READY pgs=2562 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc000c00 tx=0x7f7dcc0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 shutdown_connections 2023-10-14T18:17:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f7dc406af90 0x7f7dc406d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7de415ae80 0x7f7de40ca5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7de415aaa0 0x7f7de41557e0 unknown :-1 s=CLOSED pgs=2562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 --2- 172.21.15.29:0/1763562760 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7de40ccaa0 0x7f7de40af720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 >> 172.21.15.29:0/1763562760 conn(0x7f7de40cb410 msgr2=0x7f7de40b27d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:26.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.731+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 shutdown_connections 2023-10-14T18:17:26.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.732+0000 7f7de9f0f640 1 -- 172.21.15.29:0/1763562760 wait complete. 2023-10-14T18:17:26.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T18:17:26.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.960+0000 7fc6a8a90640 1 Processor -- start 2023-10-14T18:17:26.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.960+0000 7fc6a8a90640 1 -- start start 2023-10-14T18:17:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc6a8a90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:26.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc6a8a90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a40cb030 con 0x7fc6a415ad70 2023-10-14T18:17:26.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc6a8a90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6a40c91c0 con 0x7fc6a40c9ca0 2023-10-14T18:17:26.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc6a8a90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6a40c94a0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc69bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc69bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38284/0 (socket says 172.21.15.29:38284) 2023-10-14T18:17:26.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.962+0000 7fc69bfff640 1 -- 172.21.15.29:0/1376283665 learned_addr learned my addr 172.21.15.29:0/1376283665 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.963+0000 7fc69bfff640 1 -- 172.21.15.29:0/1376283665 >> v1:172.21.15.186:6789/0 conn(0x7fc6a40c9ca0 legacy=0x7fc6a40ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.963+0000 7fc69bfff640 1 -- 172.21.15.29:0/1376283665 >> v1:172.21.15.29:6789/0 conn(0x7fc6a415ad70 legacy=0x7fc6a4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.963+0000 7fc69bfff640 1 -- 172.21.15.29:0/1376283665 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a40c9780 con 0x7fc6a406a4b0 2023-10-14T18:17:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.964+0000 7fc69bfff640 1 --2- 172.21.15.29:0/1376283665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009870 tx=0x7fc68c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137e6c5953fd7405 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.964+0000 7fc6a1d9b640 1 -- 172.21.15.29:0/1376283665 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c030860 con 0x7fc6a406a4b0 2023-10-14T18:17:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.964+0000 7fc6a1d9b640 1 -- 172.21.15.29:0/1376283665 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc68c0309c0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.964+0000 7fc6a1d9b640 1 -- 172.21.15.29:0/1376283665 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc68c030d50 con 0x7fc6a406a4b0 2023-10-14T18:17:26.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 -- 172.21.15.29:0/1376283665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 msgr2=0x7fc6a40ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/1376283665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 secure :-1 s=READY pgs=2564 cs=0 l=1 rev1=1 crypto rx=0x7fc68c009870 tx=0x7fc68c02b9a0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 -- 172.21.15.29:0/1376283665 shutdown_connections 2023-10-14T18:17:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/1376283665 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40ca3b0 unknown :-1 s=CLOSED pgs=2564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 -- 172.21.15.29:0/1376283665 >> 172.21.15.29:0/1376283665 conn(0x7fc6a40ce9b0 msgr2=0x7fc6a40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:26.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 -- 172.21.15.29:0/1376283665 shutdown_connections 2023-10-14T18:17:26.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.965+0000 7fc6a8a90640 1 -- 172.21.15.29:0/1376283665 wait complete. 2023-10-14T18:17:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.967+0000 7fc6a8a90640 1 Processor -- start 2023-10-14T18:17:26.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.968+0000 7fc6a8a90640 1 -- start start 2023-10-14T18:17:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a40c9ca0 0x7fc6a40b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6a415ad70 0x7fc6a4151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:26.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a415d570 con 0x7fc6a406a4b0 2023-10-14T18:17:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a40c6d00 con 0x7fc6a40c9ca0 2023-10-14T18:17:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a8a90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a415d290 con 0x7fc6a415ad70 2023-10-14T18:17:26.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38296/0 (socket says 172.21.15.29:38296) 2023-10-14T18:17:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.969+0000 7fc6a259c640 1 -- 172.21.15.29:0/3367325128 learned_addr learned my addr 172.21.15.29:0/3367325128 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc69bfff640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a40c9ca0 0x7fc6a40b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:26.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a2d9d640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6a415ad70 0x7fc6a4151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 -- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6a415ad70 msgr2=0x7fc6a4151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6a415ad70 0x7fc6a4151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 -- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a40c9ca0 msgr2=0x7fc6a40b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a40c9ca0 0x7fc6a40b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:26.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 -- 172.21.15.29:0/3367325128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c0094d0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.970+0000 7fc6a259c640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c400 tx=0x7fc69400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.971+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6940052d0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.971+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc694005430 con 0x7fc6a406a4b0 2023-10-14T18:17:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.971+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a40c6fa0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.971+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc694003830 con 0x7fc6a406a4b0 2023-10-14T18:17:26.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.971+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a40c7560 con 0x7fc6a406a4b0 2023-10-14T18:17:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.972+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a404e160 con 0x7fc6a406a4b0 2023-10-14T18:17:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.973+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc694003990 con 0x7fc6a406a4b0 2023-10-14T18:17:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.974+0000 7fc69affd640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 0x7fc68406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.974+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(227..227 src has 1..227) v4 ==== 5725+0+0 (secure 0 0 0) 0x7fc6940891f0 con 0x7fc6a406a4b0 2023-10-14T18:17:26.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.975+0000 7fc69bfff640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 0x7fc68406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:26.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.975+0000 7fc69bfff640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 0x7fc68406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc68c036000 tx=0x7fc68c008030 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:26.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:26.980+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc694054160 con 0x7fc6a406a4b0 2023-10-14T18:17:27.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.211+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fc6a40715d0 con 0x7fc6a406a4b0 2023-10-14T18:17:27.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.757+0000 7fc69affd640 1 -- 172.21.15.29:0/3367325128 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 26 and data pool 27 v103) v1 ==== 174+0+0 (secure 0 0 0) 0x7fc694057780 con 0x7fc6a406a4b0 2023-10-14T18:17:27.759 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 26 and data pool 27 2023-10-14T18:17:27.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 msgr2=0x7fc68406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 0x7fc68406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc68c036000 tx=0x7fc68c008030 comp rx=0 tx=0).stop 2023-10-14T18:17:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 msgr2=0x7fc6a40b29a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 secure :-1 s=READY pgs=2565 cs=0 l=1 rev1=1 crypto rx=0x7fc69400c400 tx=0x7fc69400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 shutdown_connections 2023-10-14T18:17:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc68406af90 0x7fc68406d450 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6a415ad70 0x7fc6a4151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6a40c9ca0 0x7fc6a40b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 --2- 172.21.15.29:0/3367325128 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6a406a4b0 0x7fc6a40b29a0 unknown :-1 s=CLOSED pgs=2565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 >> 172.21.15.29:0/3367325128 conn(0x7fc6a40ce9b0 msgr2=0x7fc6a40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:27.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 shutdown_connections 2023-10-14T18:17:27.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.761+0000 7fc6a8a90640 1 -- 172.21.15.29:0/3367325128 wait complete. 2023-10-14T18:17:27.788 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.985+0000 7f6298ebb640 1 Processor -- start 2023-10-14T18:17:27.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.985+0000 7f6298ebb640 1 -- start start 2023-10-14T18:17:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f6298ebb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62940c9060 0x7f629406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f6298ebb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f62940c9440 0x7f629406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:27.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f6298ebb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62940ca560 con 0x7f629406c9e0 2023-10-14T18:17:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f6298ebb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940c9950 con 0x7f62940c9060 2023-10-14T18:17:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f6298ebb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62940c9c30 con 0x7f62940c9440 2023-10-14T18:17:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f62937fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62940c9060 0x7f629406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f62937fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f62940c9060 0x7f629406f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38300/0 (socket says 172.21.15.29:38300) 2023-10-14T18:17:27.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.987+0000 7f62937fe640 1 -- 172.21.15.29:0/2475610194 learned_addr learned my addr 172.21.15.29:0/2475610194 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.988+0000 7f62937fe640 1 -- 172.21.15.29:0/2475610194 >> v2:172.21.15.186:3300/0 conn(0x7f62940c9440 msgr2=0x7f629406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.988+0000 7f62937fe640 1 --2- 172.21.15.29:0/2475610194 >> v2:172.21.15.186:3300/0 conn(0x7f62940c9440 0x7f629406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.988+0000 7f62937fe640 1 -- 172.21.15.29:0/2475610194 >> v1:172.21.15.186:6789/0 conn(0x7f629406c9e0 legacy=0x7f6294164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.988+0000 7f62937fe640 1 -- 172.21.15.29:0/2475610194 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62940c9f10 con 0x7f62940c9060 2023-10-14T18:17:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.989+0000 7f62937fe640 1 --2- 172.21.15.29:0/2475610194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f629406f440 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f62940ce550 tx=0x7f628402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5bc12a765e920ce server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.989+0000 7f6292ffd640 1 -- 172.21.15.29:0/2475610194 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284031870 con 0x7f62940c9060 2023-10-14T18:17:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.989+0000 7f6292ffd640 1 -- 172.21.15.29:0/2475610194 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62840319d0 con 0x7f62940c9060 2023-10-14T18:17:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.989+0000 7f6292ffd640 1 -- 172.21.15.29:0/2475610194 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6284031d60 con 0x7f62940c9060 2023-10-14T18:17:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 -- 172.21.15.29:0/2475610194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 msgr2=0x7f629406f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 --2- 172.21.15.29:0/2475610194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f629406f440 secure :-1 s=READY pgs=2566 cs=0 l=1 rev1=1 crypto rx=0x7f62940ce550 tx=0x7f628402b910 comp rx=0 tx=0).stop 2023-10-14T18:17:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 -- 172.21.15.29:0/2475610194 shutdown_connections 2023-10-14T18:17:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 --2- 172.21.15.29:0/2475610194 >> v2:172.21.15.186:3300/0 conn(0x7f62940c9440 0x7f629406f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 --2- 172.21.15.29:0/2475610194 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f629406f440 unknown :-1 s=CLOSED pgs=2566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 -- 172.21.15.29:0/2475610194 >> 172.21.15.29:0/2475610194 conn(0x7f62940ce6a0 msgr2=0x7f62940ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 -- 172.21.15.29:0/2475610194 shutdown_connections 2023-10-14T18:17:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.990+0000 7f6298ebb640 1 -- 172.21.15.29:0/2475610194 wait complete. 2023-10-14T18:17:27.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.993+0000 7f6298ebb640 1 Processor -- start 2023-10-14T18:17:27.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.993+0000 7f6298ebb640 1 -- start start 2023-10-14T18:17:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 0x7f62941636c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62940c9440 0x7f6294161f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294162620 con 0x7f62940c9060 2023-10-14T18:17:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6294162900 con 0x7f629406c9e0 2023-10-14T18:17:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f6298ebb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6294162be0 con 0x7f62940c9440 2023-10-14T18:17:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f628affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f628affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38314/0 (socket says 172.21.15.29:38314) 2023-10-14T18:17:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.994+0000 7f628affd640 1 -- 172.21.15.29:0/4198373850 learned_addr learned my addr 172.21.15.29:0/4198373850 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f62937fe640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 0x7f62941636c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 -- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62940c9440 msgr2=0x7f6294161f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62940c9440 0x7f6294161f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 -- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 msgr2=0x7f62941636c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 0x7f62941636c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 -- 172.21.15.29:0/4198373850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62840094d0 con 0x7f62940c9060 2023-10-14T18:17:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f62937fe640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 0x7f62941636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f628affd640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c400 tx=0x7f627c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c011790 con 0x7f62940c9060 2023-10-14T18:17:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.995+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f627c011930 con 0x7f62940c9060 2023-10-14T18:17:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.996+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f627c011ce0 con 0x7f62940c9060 2023-10-14T18:17:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.996+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6294162ec0 con 0x7f62940c9060 2023-10-14T18:17:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.996+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62940c41c0 con 0x7f62940c9060 2023-10-14T18:17:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.997+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f629404e9b0 con 0x7f62940c9060 2023-10-14T18:17:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.998+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f627c01c460 con 0x7f62940c9060 2023-10-14T18:17:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.999+0000 7f6290ff9640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 0x7f627406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.999+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(228..228 src has 1..228) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f627c089730 con 0x7f62940c9060 2023-10-14T18:17:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:27.999+0000 7f62937fe640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 0x7f627406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.000+0000 7f62937fe640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 0x7f627406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6284036000 tx=0x7f628403c040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.004+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f627c054650 con 0x7f62940c9060 2023-10-14T18:17:28.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.202+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62940b4620 con 0x7f62940c9060 2023-10-14T18:17:28.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.204+0000 7f6290ff9640 1 -- 172.21.15.29:0/4198373850 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v228) v1 ==== 74+0+16770 (secure 0 0 0) 0x7f627c057c70 con 0x7f62940c9060 2023-10-14T18:17:28.205 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:28.206 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":228,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:27.214749+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.206+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 msgr2=0x7f627406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.206+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 0x7f627406d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6284036000 tx=0x7f628403c040 comp rx=0 tx=0).stop 2023-10-14T18:17:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.206+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 msgr2=0x7f62941f62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.206+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 secure :-1 s=READY pgs=2567 cs=0 l=1 rev1=1 crypto rx=0x7f627c00c400 tx=0x7f627c00c8c0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 shutdown_connections 2023-10-14T18:17:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f627406af90 0x7f627406d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f62940c9440 0x7f6294161f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f62940c9060 0x7f62941f62e0 unknown :-1 s=CLOSED pgs=2567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 --2- 172.21.15.29:0/4198373850 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f629406c9e0 0x7f62941636c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 >> 172.21.15.29:0/4198373850 conn(0x7f62940ce6a0 msgr2=0x7f629406b120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 shutdown_connections 2023-10-14T18:17:28.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.207+0000 7f6298ebb640 1 -- 172.21.15.29:0/4198373850 wait complete. 2023-10-14T18:17:28.228 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.451+0000 7fe5571e4640 1 Processor -- start 2023-10-14T18:17:28.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.451+0000 7fe5571e4640 1 -- start start 2023-10-14T18:17:28.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5571e4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe55015b320 0x7fe5500cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5571e4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55015b9d0 con 0x7fe5500ad970 2023-10-14T18:17:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5571e4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55015bcb0 con 0x7fe55006c040 2023-10-14T18:17:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5571e4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55015b7d0 con 0x7fe55015b320 2023-10-14T18:17:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5559e1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe5500ad970 0x7fe5500cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55932/0 (socket says 172.21.15.29:55932) 2023-10-14T18:17:28.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5559e1640 1 -- 172.21.15.29:0/450480948 learned_addr learned my addr 172.21.15.29:0/450480948 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5569e3640 1 --2- 172.21.15.29:0/450480948 >> v2:172.21.15.186:3300/0 conn(0x7fe55015b320 0x7fe5500cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3612461762 0 0) 0x7fe55015b9d0 con 0x7fe5500ad970 2023-10-14T18:17:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.453+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c003520 con 0x7fe5500ad970 2023-10-14T18:17:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3335562663 0 0) 0x7fe53c003520 con 0x7fe5500ad970 2023-10-14T18:17:28.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 >> v2:172.21.15.186:3300/0 conn(0x7fe55015b320 msgr2=0x7fe5500cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 --2- 172.21.15.29:0/450480948 >> v2:172.21.15.186:3300/0 conn(0x7fe55015b320 0x7fe5500cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 >> v1:172.21.15.186:6789/0 conn(0x7fe55006c040 legacy=0x7fe55006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55015bf90 con 0x7fe5500ad970 2023-10-14T18:17:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe540001670 con 0x7fe5500ad970 2023-10-14T18:17:28.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 >> v1:172.21.15.29:6789/0 conn(0x7fe5500ad970 legacy=0x7fe5500cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe53c0039f0 0x7fe53c003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c0047c0 0x7fe53c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 0x7fe53c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe55015b7d0 con 0x7fe53c005180 2023-10-14T18:17:28.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c00c0b0 con 0x7fe53c0047c0 2023-10-14T18:17:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53c00c270 con 0x7fe53c0039f0 2023-10-14T18:17:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.454+0000 7fe5569e3640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 0x7fe53c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.455+0000 7fe5569e3640 1 -- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe53c0039f0 msgr2=0x7fe53c003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:28.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.455+0000 7fe5569e3640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe53c0039f0 0x7fe53c003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.455+0000 7fe5569e3640 1 -- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c0047c0 msgr2=0x7fe53c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.455+0000 7fe5569e3640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c0047c0 0x7fe53c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.455+0000 7fe5569e3640 1 -- 172.21.15.29:0/450480948 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53c003520 con 0x7fe53c005180 2023-10-14T18:17:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.456+0000 7fe5569e3640 1 -- 172.21.15.29:0/450480948 reap_dead start 2023-10-14T18:17:28.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.456+0000 7fe5569e3640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 0x7fe53c00b980 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fe548009fc0 tx=0x7fe53c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c793adc442c4549 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.456+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe54802cc60 con 0x7fe53c005180 2023-10-14T18:17:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.456+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54800f040 con 0x7fe53c005180 2023-10-14T18:17:28.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.456+0000 7fe5551e0640 1 -- 172.21.15.29:0/450480948 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548003990 con 0x7fe53c005180 2023-10-14T18:17:28.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 -- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 msgr2=0x7fe53c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 0x7fe53c00b980 secure :-1 s=READY pgs=2569 cs=0 l=1 rev1=1 crypto rx=0x7fe548009fc0 tx=0x7fe53c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 -- 172.21.15.29:0/450480948 shutdown_connections 2023-10-14T18:17:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 --2- 172.21.15.29:0/450480948 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe53c005180 0x7fe53c00b980 unknown :-1 s=CLOSED pgs=2569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 -- 172.21.15.29:0/450480948 >> 172.21.15.29:0/450480948 conn(0x7fe5500ce6a0 msgr2=0x7fe5500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 -- 172.21.15.29:0/450480948 shutdown_connections 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.457+0000 7fe5571e4640 1 -- 172.21.15.29:0/450480948 wait complete. 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.459+0000 7fe5571e4640 1 Processor -- start 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.459+0000 7fe5571e4640 1 -- start start 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 0x7fe550154d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55015c240 0x7fe5501f5e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5501f6360 con 0x7fe55015db80 2023-10-14T18:17:28.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5501f6590 con 0x7fe53c005180 2023-10-14T18:17:28.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5571e4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5501f6870 con 0x7fe55015c240 2023-10-14T18:17:28.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5569e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5569e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38336/0 (socket says 172.21.15.29:38336) 2023-10-14T18:17:28.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.460+0000 7fe5569e3640 1 -- 172.21.15.29:0/168013546 learned_addr learned my addr 172.21.15.29:0/168013546 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:28.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5559e1640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55015c240 0x7fe5501f5e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 -- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55015c240 msgr2=0x7fe5501f5e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5561e2640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 0x7fe550154d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55015c240 0x7fe5501f5e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 -- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 msgr2=0x7fe550154d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 0x7fe550154d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 -- 172.21.15.29:0/168013546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5501f6b50 con 0x7fe55015db80 2023-10-14T18:17:28.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5561e2640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 0x7fe550154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:28.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5569e3640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fe54802c6d0 tx=0x7fe548009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548003da0 con 0x7fe55015db80 2023-10-14T18:17:28.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe548035950 con 0x7fe55015db80 2023-10-14T18:17:28.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe548035d00 con 0x7fe55015db80 2023-10-14T18:17:28.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.461+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5500c8e90 con 0x7fe55015db80 2023-10-14T18:17:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.462+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55015e000 con 0x7fe55015db80 2023-10-14T18:17:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.463+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55006b330 con 0x7fe55015db80 2023-10-14T18:17:28.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.464+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fe548004030 con 0x7fe55015db80 2023-10-14T18:17:28.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.465+0000 7fe546ffd640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 0x7fe53406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.465+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe5480ac520 con 0x7fe55015db80 2023-10-14T18:17:28.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.465+0000 7fe5561e2640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 0x7fe53406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.466+0000 7fe5561e2640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 0x7fe53406d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe55006ba60 tx=0x7fe54c009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.469+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe548077440 con 0x7fe55015db80 2023-10-14T18:17:28.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.667+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5500c8cf0 con 0x7fe55015db80 2023-10-14T18:17:28.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.669+0000 7fe546ffd640 1 -- 172.21.15.29:0/168013546 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fe5500c8cf0 con 0x7fe55015db80 2023-10-14T18:17:28.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:28.670 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":229,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:28.226229+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:28.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.671+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 msgr2=0x7fe53406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 0x7fe53406d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fe55006ba60 tx=0x7fe54c009040 comp rx=0 tx=0).stop 2023-10-14T18:17:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 msgr2=0x7fe5500ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 secure :-1 s=READY pgs=2570 cs=0 l=1 rev1=1 crypto rx=0x7fe54802c6d0 tx=0x7fe548009750 comp rx=0 tx=0).stop 2023-10-14T18:17:28.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 shutdown_connections 2023-10-14T18:17:28.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fe53406afc0 0x7fe53406d480 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe55015db80 0x7fe5500ac890 unknown :-1 s=CLOSED pgs=2570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe55015c240 0x7fe5501f5e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 --2- 172.21.15.29:0/168013546 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe53c005180 0x7fe550154d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 >> 172.21.15.29:0/168013546 conn(0x7fe5500ce6a0 msgr2=0x7fe5500c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 shutdown_connections 2023-10-14T18:17:28.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.672+0000 7fe5571e4640 1 -- 172.21.15.29:0/168013546 wait complete. 2023-10-14T18:17:28.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T18:17:28.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.930+0000 7f697391d640 1 Processor -- start 2023-10-14T18:17:28.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.931+0000 7f697391d640 1 -- start start 2023-10-14T18:17:28.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.932+0000 7f697391d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f696c15d030 0x7f696c0d1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.932+0000 7f697391d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 0x7f696c161450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.932+0000 7f697391d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f696c0b5d90 con 0x7f696c161990 2023-10-14T18:17:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f697391d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f696c0d2b50 con 0x7f696c15d030 2023-10-14T18:17:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f697391d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c160fc0 con 0x7f696c15d410 2023-10-14T18:17:28.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f69716b9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f696c15d030 0x7f696c0d1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f6971eba640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f696c161990 0x7f696c0cd260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55936/0 (socket says 172.21.15.29:55936) 2023-10-14T18:17:28.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f6971eba640 1 -- 172.21.15.29:0/806474774 learned_addr learned my addr 172.21.15.29:0/806474774 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:28.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f6970eb8640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 0x7f696c161450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f69716b9640 1 -- 172.21.15.29:0/806474774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 msgr2=0x7f696c161450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f69716b9640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 0x7f696c161450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f69716b9640 1 -- 172.21.15.29:0/806474774 >> v1:172.21.15.29:6789/0 conn(0x7f696c161990 legacy=0x7f696c0cd260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f69716b9640 1 -- 172.21.15.29:0/806474774 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f696c0d25d0 con 0x7f696c15d030 2023-10-14T18:17:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f6963fff640 1 -- 172.21.15.29:0/806474774 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651083137 0 0) 0x7f696c0b5d90 con 0x7f696c161990 2023-10-14T18:17:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.933+0000 7f6970eb8640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 0x7f696c161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.935+0000 7f69716b9640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d030 0x7f696c0d1630 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696802bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff40e45dfda1f522 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.935+0000 7f6963fff640 1 -- 172.21.15.29:0/806474774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f696802cce0 con 0x7f696c15d030 2023-10-14T18:17:28.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.935+0000 7f6963fff640 1 -- 172.21.15.29:0/806474774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f696802ce40 con 0x7f696c15d030 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.935+0000 7f6963fff640 1 -- 172.21.15.29:0/806474774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6968034660 con 0x7f696c15d030 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 -- 172.21.15.29:0/806474774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d030 msgr2=0x7f696c0d1630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d030 0x7f696c0d1630 secure :-1 s=READY pgs=2572 cs=0 l=1 rev1=1 crypto rx=0x7f6968009d90 tx=0x7f696802bf30 comp rx=0 tx=0).stop 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 -- 172.21.15.29:0/806474774 shutdown_connections 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d410 0x7f696c161450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 --2- 172.21.15.29:0/806474774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d030 0x7f696c0d1630 unknown :-1 s=CLOSED pgs=2572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 -- 172.21.15.29:0/806474774 >> 172.21.15.29:0/806474774 conn(0x7f696c0cf720 msgr2=0x7f696c0cfb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 -- 172.21.15.29:0/806474774 shutdown_connections 2023-10-14T18:17:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.936+0000 7f697391d640 1 -- 172.21.15.29:0/806474774 wait complete. 2023-10-14T18:17:28.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.938+0000 7f697391d640 1 Processor -- start 2023-10-14T18:17:28.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.938+0000 7f697391d640 1 -- start start 2023-10-14T18:17:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d030 0x7f696c0c6740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 0x7f696c1f61b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c98b0 con 0x7f696c15d410 2023-10-14T18:17:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696c0c6c80 con 0x7f696c161990 2023-10-14T18:17:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f697391d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f696c15fa50 con 0x7f696c15d030 2023-10-14T18:17:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38348/0 (socket says 172.21.15.29:38348) 2023-10-14T18:17:28.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 -- 172.21.15.29:0/298023628 learned_addr learned my addr 172.21.15.29:0/298023628 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f69716b9640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d030 0x7f696c0c6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6971eba640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 0x7f696c1f61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 -- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d030 msgr2=0x7f696c0c6740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d030 0x7f696c0c6740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.939+0000 7f6970eb8640 1 -- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 msgr2=0x7f696c1f61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6970eb8640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 0x7f696c1f61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6970eb8640 1 -- 172.21.15.29:0/298023628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6968009a70 con 0x7f696c15d410 2023-10-14T18:17:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6971eba640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 0x7f696c1f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6970eb8640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c820 tx=0x7f695c00cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c007a30 con 0x7f696c15d410 2023-10-14T18:17:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f695c007b90 con 0x7f696c15d410 2023-10-14T18:17:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f695c00ce70 con 0x7f696c15d410 2023-10-14T18:17:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696c0b7890 con 0x7f696c15d410 2023-10-14T18:17:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.940+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696c0b7e50 con 0x7f696c15d410 2023-10-14T18:17:28.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.942+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f696c04e160 con 0x7f696c15d410 2023-10-14T18:17:28.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.943+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f695c0113f0 con 0x7f696c15d410 2023-10-14T18:17:28.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.944+0000 7f6961ffb640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 0x7f695006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:28.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.944+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f695c089b50 con 0x7f696c15d410 2023-10-14T18:17:28.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.944+0000 7f69716b9640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 0x7f695006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:28.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.945+0000 7f69716b9640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 0x7f695006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6968036000 tx=0x7f6968030000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:28.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:28.949+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f695c0548c0 con 0x7f696c15d410 2023-10-14T18:17:29.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.185+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f696c0c9aa0 con 0x7f696c15d410 2023-10-14T18:17:29.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.772+0000 7f6961ffb640 1 -- 172.21.15.29:0/298023628 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v106) v1 ==== 118+0+0 (secure 0 0 0) 0x7f695c01f0b0 con 0x7f696c15d410 2023-10-14T18:17:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.775+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 msgr2=0x7f695006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 0x7f695006d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6968036000 tx=0x7f6968030000 comp rx=0 tx=0).stop 2023-10-14T18:17:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 msgr2=0x7f696c1f5c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:29.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 secure :-1 s=READY pgs=2573 cs=0 l=1 rev1=1 crypto rx=0x7f695c00c820 tx=0x7f695c00cc20 comp rx=0 tx=0).stop 2023-10-14T18:17:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 shutdown_connections 2023-10-14T18:17:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f695006af90 0x7f695006d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f696c161990 0x7f696c1f61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f696c15d410 0x7f696c1f5c70 unknown :-1 s=CLOSED pgs=2573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:29.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 --2- 172.21.15.29:0/298023628 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f696c15d030 0x7f696c0c6740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 >> 172.21.15.29:0/298023628 conn(0x7f696c0cf720 msgr2=0x7f696c0cd240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 shutdown_connections 2023-10-14T18:17:29.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:29.776+0000 7f697391d640 1 -- 172.21.15.29:0/298023628 wait complete. 2023-10-14T18:17:29.798 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:17:29.798 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T18:17:30.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.030+0000 7f9a0edc0640 1 Processor -- start 2023-10-14T18:17:30.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.030+0000 7f9a0edc0640 1 -- start start 2023-10-14T18:17:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.031+0000 7f9a0edc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f9a08069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.031+0000 7f9a0edc0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a080cc6b0 con 0x7f9a080c9150 2023-10-14T18:17:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.031+0000 7f9a0edc0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a081599b0 con 0x7f9a0806bd70 2023-10-14T18:17:30.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.031+0000 7f9a0edc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0806c290 con 0x7f9a081428c0 2023-10-14T18:17:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.032+0000 7f9a07fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9a080c9150 0x7f9a0806b660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55944/0 (socket says 172.21.15.29:55944) 2023-10-14T18:17:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.032+0000 7f9a07fff640 1 -- 172.21.15.29:0/2165468790 learned_addr learned my addr 172.21.15.29:0/2165468790 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:30.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.032+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f9a08069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.032+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 191435746 0 0) 0x7f9a080cc6b0 con 0x7f9a080c9150 2023-10-14T18:17:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.032+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99f0003520 con 0x7f9a080c9150 2023-10-14T18:17:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2366668775 0 0) 0x7f99f0003520 con 0x7f9a080c9150 2023-10-14T18:17:30.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 msgr2=0x7f9a08069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f9a08069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 >> v1:172.21.15.186:6789/0 conn(0x7f9a0806bd70 legacy=0x7f9a081643b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a0806c570 con 0x7f9a080c9150 2023-10-14T18:17:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f99fc001670 con 0x7f9a080c9150 2023-10-14T18:17:30.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 >> v1:172.21.15.29:6789/0 conn(0x7f9a080c9150 legacy=0x7f9a0806b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f00039a0 0x7f99f0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00047c0 0x7f99f0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f99f0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a0806c290 con 0x7f99f00039a0 2023-10-14T18:17:30.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99f000ba30 con 0x7f99f00047c0 2023-10-14T18:17:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99f000bbf0 con 0x7f9a081428c0 2023-10-14T18:17:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.033+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f00039a0 0x7f99f0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a07fff640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00047c0 0x7f99f0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0d35d640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f99f0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0cb5c640 1 -- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 msgr2=0x7f99f0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a081428c0 0x7f99f0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0cb5c640 1 -- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00047c0 msgr2=0x7f99f0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00047c0 0x7f99f0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a0cb5c640 1 -- 172.21.15.29:0/2165468790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99f0003520 con 0x7f99f00039a0 2023-10-14T18:17:30.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.034+0000 7f9a07fff640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00047c0 0x7f99f0004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.035+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f00039a0 0x7f99f0003dc0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009f90 tx=0x7f99f00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fafb1d978cbee7df server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.035+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f802cbd0 con 0x7f99f00039a0 2023-10-14T18:17:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.035+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99f800f040 con 0x7f99f00039a0 2023-10-14T18:17:30.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.035+0000 7f9a077fe640 1 -- 172.21.15.29:0/2165468790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f99f8034440 con 0x7f99f00039a0 2023-10-14T18:17:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2165468790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f00039a0 msgr2=0x7f99f0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2165468790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f99f00039a0 0x7f99f0003dc0 secure :-1 s=READY pgs=2575 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009f90 tx=0x7f99f00057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0d35d640 1 -- 172.21.15.29:0/2165468790 reap_dead start 2023-10-14T18:17:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2165468790 shutdown_connections 2023-10-14T18:17:30.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2165468790 >> 172.21.15.29:0/2165468790 conn(0x7f9a080ce9b0 msgr2=0x7f9a080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2165468790 shutdown_connections 2023-10-14T18:17:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.036+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2165468790 wait complete. 2023-10-14T18:17:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.038+0000 7f9a0edc0640 1 Processor -- start 2023-10-14T18:17:30.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.038+0000 7f9a0edc0640 1 -- start start 2023-10-14T18:17:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00039a0 0x7f9a08157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a080b2e10 0x7f9a080b3290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a080b4510 con 0x7f9a080ac890 2023-10-14T18:17:30.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a080ad280 con 0x7f99f00039a0 2023-10-14T18:17:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0edc0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9a080b3990 con 0x7f9a080b2e10 2023-10-14T18:17:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0d35d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0d35d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:38364/0 (socket says 172.21.15.29:38364) 2023-10-14T18:17:30.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.039+0000 7f9a0d35d640 1 -- 172.21.15.29:0/2609591048 learned_addr learned my addr 172.21.15.29:0/2609591048 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00039a0 0x7f9a08157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a07fff640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a080b2e10 0x7f9a080b3290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 -- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a080b2e10 msgr2=0x7f9a080b3290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a080b2e10 0x7f9a080b3290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 -- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00039a0 msgr2=0x7f9a08157fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00039a0 0x7f9a08157fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 -- 172.21.15.29:0/2609591048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a080b3c70 con 0x7f9a080ac890 2023-10-14T18:17:30.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0d35d640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9a080ae2c0 tx=0x7f9a0000d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a0000b070 con 0x7f9a080ac890 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a000054e0 con 0x7f9a080ac890 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a080b3f50 con 0x7f9a080ac890 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.040+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a00003810 con 0x7f9a080ac890 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.041+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a081584f0 con 0x7f9a080ac890 2023-10-14T18:17:30.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.042+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a0804e160 con 0x7f9a080ac890 2023-10-14T18:17:30.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.043+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f9a00003970 con 0x7f9a080ac890 2023-10-14T18:17:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.044+0000 7f9a057fa640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 0x7f99f406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.044+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9a00088ee0 con 0x7f9a080ac890 2023-10-14T18:17:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.044+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 0x7f99f406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:30.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.045+0000 7f9a0cb5c640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 0x7f99f406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99f8036000 tx=0x7f99f802e000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:30.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.048+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9a00053cf0 con 0x7f9a080ac890 2023-10-14T18:17:30.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.280+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f9a0806ac60 con 0x7f9a080ac890 2023-10-14T18:17:30.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.785+0000 7f9a057fa640 1 -- 172.21.15.29:0/2609591048 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v107) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9a00018020 con 0x7f9a080ac890 2023-10-14T18:17:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 msgr2=0x7f99f406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 0x7f99f406d450 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f99f8036000 tx=0x7f99f802e000 comp rx=0 tx=0).stop 2023-10-14T18:17:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 msgr2=0x7f9a080acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:30.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 secure :-1 s=READY pgs=2576 cs=0 l=1 rev1=1 crypto rx=0x7f9a080ae2c0 tx=0x7f9a0000d890 comp rx=0 tx=0).stop 2023-10-14T18:17:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 shutdown_connections 2023-10-14T18:17:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f99f406af90 0x7f99f406d450 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9a080ac890 0x7f9a080acd40 unknown :-1 s=CLOSED pgs=2576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9a080b2e10 0x7f9a080b3290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 --2- 172.21.15.29:0/2609591048 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f99f00039a0 0x7f9a08157fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:30.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 >> 172.21.15.29:0/2609591048 conn(0x7f9a080ce9b0 msgr2=0x7f9a0815bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:30.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 shutdown_connections 2023-10-14T18:17:30.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:30.789+0000 7f9a0edc0640 1 -- 172.21.15.29:0/2609591048 wait complete. 2023-10-14T18:17:30.813 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T18:17:30.813 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.041+0000 7f8b917ef640 1 Processor -- start 2023-10-14T18:17:31.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.041+0000 7f8b917ef640 1 -- start start 2023-10-14T18:17:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b917ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b8c0ad970 0x7f8b8c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b917ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b917ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b8c15b9d0 con 0x7f8b8c06c040 2023-10-14T18:17:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b917ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b8c15bcb0 con 0x7f8b8c0ad970 2023-10-14T18:17:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b917ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c15bf90 con 0x7f8b8c0ccfb0 2023-10-14T18:17:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.043+0000 7f8b8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b8c0ad970 0x7f8b8c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b8c0ad970 0x7f8b8c0cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55492/0 (socket says 172.21.15.29:55492) 2023-10-14T18:17:31.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 -- 172.21.15.29:0/2762704752 learned_addr learned my addr 172.21.15.29:0/2762704752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b90fee640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 -- 172.21.15.29:0/2762704752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ccfb0 msgr2=0x7f8b8c0cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 -- 172.21.15.29:0/2762704752 >> v1:172.21.15.186:6789/0 conn(0x7f8b8c06c040 legacy=0x7f8b8c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.044+0000 7f8b8b7fe640 1 -- 172.21.15.29:0/2762704752 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b8c0cc270 con 0x7f8b8c0ad970 2023-10-14T18:17:31.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.046+0000 7f8b8b7fe640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0ad970 0x7f8b8c0cca70 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009870 tx=0x7f8b8002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2eabc967e5b9233c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.046+0000 7f8b8affd640 1 -- 172.21.15.29:0/2762704752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80031640 con 0x7f8b8c0ad970 2023-10-14T18:17:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.046+0000 7f8b8affd640 1 -- 172.21.15.29:0/2762704752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b800317a0 con 0x7f8b8c0ad970 2023-10-14T18:17:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.046+0000 7f8b8affd640 1 -- 172.21.15.29:0/2762704752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80031b30 con 0x7f8b8c0ad970 2023-10-14T18:17:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 -- 172.21.15.29:0/2762704752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0ad970 msgr2=0x7f8b8c0cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0ad970 0x7f8b8c0cca70 secure :-1 s=READY pgs=2577 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009870 tx=0x7f8b8002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 -- 172.21.15.29:0/2762704752 shutdown_connections 2023-10-14T18:17:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2762704752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c0ad970 0x7f8b8c0cca70 unknown :-1 s=CLOSED pgs=2577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 -- 172.21.15.29:0/2762704752 >> 172.21.15.29:0/2762704752 conn(0x7f8b8c0ce6a0 msgr2=0x7f8b8c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 -- 172.21.15.29:0/2762704752 shutdown_connections 2023-10-14T18:17:31.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.047+0000 7f8b917ef640 1 -- 172.21.15.29:0/2762704752 wait complete. 2023-10-14T18:17:31.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.049+0000 7f8b917ef640 1 Processor -- start 2023-10-14T18:17:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.049+0000 7f8b917ef640 1 -- start start 2023-10-14T18:17:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ad970 0x7f8b8c0c6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0b1de0 con 0x7f8b8c06c040 2023-10-14T18:17:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0b1a40 con 0x7f8b8c0ccfb0 2023-10-14T18:17:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.050+0000 7f8b917ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b8c0c7590 con 0x7f8b8c0ad970 2023-10-14T18:17:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55496/0 (socket says 172.21.15.29:55496) 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 -- 172.21.15.29:0/2142959774 learned_addr learned my addr 172.21.15.29:0/2142959774 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8b7fe640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ad970 0x7f8b8c0c6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b90fee640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 -- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ad970 msgr2=0x7f8b8c0c6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ad970 0x7f8b8c0c6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 -- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c0ccfb0 msgr2=0x7f8b8c0b2200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 -- 172.21.15.29:0/2142959774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b800094d0 con 0x7f8b8c06c040 2023-10-14T18:17:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.051+0000 7f8b8bfff640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c430 tx=0x7f8b7c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.052+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c011790 con 0x7f8b8c06c040 2023-10-14T18:17:31.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.052+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b7c0118f0 con 0x7f8b8c06c040 2023-10-14T18:17:31.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.052+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c011ce0 con 0x7f8b8c06c040 2023-10-14T18:17:31.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.052+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b8c0c7870 con 0x7f8b8c06c040 2023-10-14T18:17:31.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.052+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b8c15d990 con 0x7f8b8c06c040 2023-10-14T18:17:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.053+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b8c06b330 con 0x7f8b8c06c040 2023-10-14T18:17:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.054+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b7c01c460 con 0x7f8b8c06c040 2023-10-14T18:17:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.055+0000 7f8b88ff9640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.055+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b7c0897c0 con 0x7f8b8c06c040 2023-10-14T18:17:31.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.055+0000 7f8b8b7fe640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.056+0000 7f8b8b7fe640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 0x7f8b7806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b80036000 tx=0x7f8b8003c040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.059+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8b7c0546d0 con 0x7f8b8c06c040 2023-10-14T18:17:31.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.283+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b8c0b1f20 con 0x7f8b8c06c040 2023-10-14T18:17:31.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.285+0000 7f8b88ff9640 1 -- 172.21.15.29:0/2142959774 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f8b8c0b1f20 con 0x7f8b8c06c040 2023-10-14T18:17:31.286 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:31.286 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11256,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-14T18:17:31.287 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 107 2023-10-14T18:17:31.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 msgr2=0x7f8b7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 0x7f8b7806d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f8b80036000 tx=0x7f8b8003c040 comp rx=0 tx=0).stop 2023-10-14T18:17:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 msgr2=0x7f8b8c15d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 secure :-1 s=READY pgs=2578 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c00c430 tx=0x7f8b7c00c8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 shutdown_connections 2023-10-14T18:17:31.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b7806af90 0x7f8b7806d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b8c0ccfb0 0x7f8b8c0b2200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b8c0ad970 0x7f8b8c0c6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.287+0000 7f8b917ef640 1 --2- 172.21.15.29:0/2142959774 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b8c06c040 0x7f8b8c15d450 unknown :-1 s=CLOSED pgs=2578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.288+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 >> 172.21.15.29:0/2142959774 conn(0x7f8b8c0ce6a0 msgr2=0x7f8b8c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:31.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.288+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 shutdown_connections 2023-10-14T18:17:31.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.288+0000 7f8b917ef640 1 -- 172.21.15.29:0/2142959774 wait complete. 2023-10-14T18:17:31.308 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:31.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.527+0000 7fed10561640 1 Processor -- start 2023-10-14T18:17:31.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.527+0000 7fed10561640 1 -- start start 2023-10-14T18:17:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed10561640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed0815ae80 0x7fed0815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed10561640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed080ce390 0x7fed08070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed10561640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0815ed60 con 0x7fed0815aaa0 2023-10-14T18:17:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed10561640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0806b810 con 0x7fed080ce390 2023-10-14T18:17:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed10561640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed080707e0 con 0x7fed0815ae80 2023-10-14T18:17:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed0eafe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed080ce390 0x7fed08070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed0eafe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed080ce390 0x7fed08070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55508/0 (socket says 172.21.15.29:55508) 2023-10-14T18:17:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.528+0000 7fed0eafe640 1 -- 172.21.15.29:0/3664423481 learned_addr learned my addr 172.21.15.29:0/3664423481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:31.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.529+0000 7fed0dafc640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed0815ae80 0x7fed0815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.529+0000 7fed0eafe640 1 -- 172.21.15.29:0/3664423481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed0815ae80 msgr2=0x7fed0815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.529+0000 7fed0eafe640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed0815ae80 0x7fed0815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.529+0000 7fed0eafe640 1 -- 172.21.15.29:0/3664423481 >> v1:172.21.15.186:6789/0 conn(0x7fed0815aaa0 legacy=0x7fed080cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.529+0000 7fed0eafe640 1 -- 172.21.15.29:0/3664423481 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0815ef90 con 0x7fed080ce390 2023-10-14T18:17:31.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.530+0000 7fed0eafe640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed080ce390 0x7fed08070f50 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35b4ccf60b25db7f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.530+0000 7fed0d2fb640 1 -- 172.21.15.29:0/3664423481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00031640 con 0x7fed080ce390 2023-10-14T18:17:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed0d2fb640 1 -- 172.21.15.29:0/3664423481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed000317a0 con 0x7fed080ce390 2023-10-14T18:17:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed0d2fb640 1 -- 172.21.15.29:0/3664423481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed00031b30 con 0x7fed080ce390 2023-10-14T18:17:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 -- 172.21.15.29:0/3664423481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed080ce390 msgr2=0x7fed08070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed080ce390 0x7fed08070f50 secure :-1 s=READY pgs=2579 cs=0 l=1 rev1=1 crypto rx=0x7fed00009870 tx=0x7fed0002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 -- 172.21.15.29:0/3664423481 shutdown_connections 2023-10-14T18:17:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed080ce390 0x7fed08070f50 unknown :-1 s=CLOSED pgs=2579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 --2- 172.21.15.29:0/3664423481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed0815ae80 0x7fed0815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 -- 172.21.15.29:0/3664423481 >> 172.21.15.29:0/3664423481 conn(0x7fed080cb410 msgr2=0x7fed080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 -- 172.21.15.29:0/3664423481 shutdown_connections 2023-10-14T18:17:31.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.531+0000 7fed10561640 1 -- 172.21.15.29:0/3664423481 wait complete. 2023-10-14T18:17:31.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.533+0000 7fed10561640 1 Processor -- start 2023-10-14T18:17:31.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.533+0000 7fed10561640 1 -- start start 2023-10-14T18:17:31.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 0x7fed080c28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 0x7fed080b2b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08149ae0 con 0x7fed0815aaa0 2023-10-14T18:17:31.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed08149dc0 con 0x7fed0815ae80 2023-10-14T18:17:31.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed10561640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0814a0a0 con 0x7fed080ce390 2023-10-14T18:17:31.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed0dafc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed0dafc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55514/0 (socket says 172.21.15.29:55514) 2023-10-14T18:17:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.534+0000 7fed0dafc640 1 -- 172.21.15.29:0/465804378 learned_addr learned my addr 172.21.15.29:0/465804378 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0e2fd640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 0x7fed080c28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0eafe640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 0x7fed080b2b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 -- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 msgr2=0x7fed080c28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 0x7fed080c28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 -- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 msgr2=0x7fed080b2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 0x7fed080b2b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 -- 172.21.15.29:0/465804378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed000094d0 con 0x7fed0815aaa0 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0e2fd640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 0x7fed080c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0eafe640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 0x7fed080b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:31.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed0dafc640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fecf800d800 tx=0x7fecf800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800b6c0 con 0x7fed0815aaa0 2023-10-14T18:17:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf800b820 con 0x7fed0815aaa0 2023-10-14T18:17:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecf800bbb0 con 0x7fed0815aaa0 2023-10-14T18:17:31.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.535+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0814a380 con 0x7fed0815aaa0 2023-10-14T18:17:31.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.536+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed08149710 con 0x7fed0815aaa0 2023-10-14T18:17:31.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.537+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0804e160 con 0x7fed0815aaa0 2023-10-14T18:17:31.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.538+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fecf800bd10 con 0x7fed0815aaa0 2023-10-14T18:17:31.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.538+0000 7fecfeffd640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.539+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fecf8089de0 con 0x7fed0815aaa0 2023-10-14T18:17:31.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.539+0000 7fed0e2fd640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.539+0000 7fed0e2fd640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 0x7fecec06d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed04004520 tx=0x7fed04009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.543+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fecf8054c60 con 0x7fed0815aaa0 2023-10-14T18:17:31.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.741+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed080cdc60 con 0x7fed0815aaa0 2023-10-14T18:17:31.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.743+0000 7fecfeffd640 1 -- 172.21.15.29:0/465804378 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+16757 (secure 0 0 0) 0x7fecf8058280 con 0x7fed0815aaa0 2023-10-14T18:17:31.745 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:31.745 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":229,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:28.226229+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:31.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 msgr2=0x7fecec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 0x7fecec06d480 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fed04004520 tx=0x7fed04009040 comp rx=0 tx=0).stop 2023-10-14T18:17:31.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 msgr2=0x7fed080b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 secure :-1 s=READY pgs=2580 cs=0 l=1 rev1=1 crypto rx=0x7fecf800d800 tx=0x7fecf800dcc0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 shutdown_connections 2023-10-14T18:17:31.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fecec06afc0 0x7fecec06d480 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed0815ae80 0x7fed080b2b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed0815aaa0 0x7fed080b2620 unknown :-1 s=CLOSED pgs=2580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 --2- 172.21.15.29:0/465804378 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed080ce390 0x7fed080c28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.746+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 >> 172.21.15.29:0/465804378 conn(0x7fed080cb410 msgr2=0x7fed080b14b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:31.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.747+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 shutdown_connections 2023-10-14T18:17:31.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.747+0000 7fed10561640 1 -- 172.21.15.29:0/465804378 wait complete. 2023-10-14T18:17:31.765 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:17:31.766 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:17:31.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.988+0000 7faea331d640 1 Processor -- start 2023-10-14T18:17:31.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.989+0000 7faea331d640 1 -- start start 2023-10-14T18:17:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0adb50 0x7fae9c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae9c070fd0 0x7fae9c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 0x7fae9c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c071450 con 0x7fae9c070fd0 2023-10-14T18:17:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c0cc600 con 0x7fae9c0adf30 2023-10-14T18:17:31.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.990+0000 7faea331d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0cc8e0 con 0x7fae9c0adb50 2023-10-14T18:17:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae9c070fd0 0x7fae9c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae9c070fd0 0x7fae9c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55520/0 (socket says 172.21.15.29:55520) 2023-10-14T18:17:31.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 -- 172.21.15.29:0/1555737833 learned_addr learned my addr 172.21.15.29:0/1555737833 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea10b9640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0adb50 0x7fae9c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 -- 172.21.15.29:0/1555737833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0adb50 msgr2=0x7fae9c06f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea18ba640 1 --2- 172.21.15.29:0/1555737833 >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 0x7fae9c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0adb50 0x7fae9c06f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 -- 172.21.15.29:0/1555737833 >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 msgr2=0x7fae9c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 --2- 172.21.15.29:0/1555737833 >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 0x7fae9c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea08b8640 1 -- 172.21.15.29:0/1555737833 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0ccbc0 con 0x7fae9c070fd0 2023-10-14T18:17:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.991+0000 7faea18ba640 1 --2- 172.21.15.29:0/1555737833 >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 0x7fae9c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:31.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.992+0000 7faea08b8640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c070fd0 0x7fae9c0b0410 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06d840 tx=0x7fae8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=989221b1e17e3b7f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:31.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.993+0000 7fae93fff640 1 -- 172.21.15.29:0/1555737833 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c031870 con 0x7fae9c070fd0 2023-10-14T18:17:31.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.993+0000 7fae93fff640 1 -- 172.21.15.29:0/1555737833 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c0319d0 con 0x7fae9c070fd0 2023-10-14T18:17:31.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.993+0000 7fae93fff640 1 -- 172.21.15.29:0/1555737833 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c031d60 con 0x7fae9c070fd0 2023-10-14T18:17:31.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.993+0000 7faea331d640 1 -- 172.21.15.29:0/1555737833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c070fd0 msgr2=0x7fae9c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:31.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.993+0000 7faea331d640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c070fd0 0x7fae9c0b0410 secure :-1 s=READY pgs=2581 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06d840 tx=0x7fae8c02b910 comp rx=0 tx=0).stop 2023-10-14T18:17:31.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 -- 172.21.15.29:0/1555737833 shutdown_connections 2023-10-14T18:17:31.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 --2- 172.21.15.29:0/1555737833 >> v2:172.21.15.186:3300/0 conn(0x7fae9c0adf30 0x7fae9c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c070fd0 0x7fae9c0b0410 unknown :-1 s=CLOSED pgs=2581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 --2- 172.21.15.29:0/1555737833 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0adb50 0x7fae9c06f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 -- 172.21.15.29:0/1555737833 >> 172.21.15.29:0/1555737833 conn(0x7fae9c0cb410 msgr2=0x7fae9c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 -- 172.21.15.29:0/1555737833 shutdown_connections 2023-10-14T18:17:31.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.994+0000 7faea331d640 1 -- 172.21.15.29:0/1555737833 wait complete. 2023-10-14T18:17:31.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.995+0000 7faea331d640 1 Processor -- start 2023-10-14T18:17:31.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.996+0000 7faea331d640 1 -- start start 2023-10-14T18:17:32.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c070fd0 0x7fae9c0c13e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0adf30 0x7fae9c160a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0bd5e0 con 0x7fae9c0adb50 2023-10-14T18:17:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0c1920 con 0x7fae9c0adf30 2023-10-14T18:17:32.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea331d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c1fa840 con 0x7fae9c070fd0 2023-10-14T18:17:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55528/0 (socket says 172.21.15.29:55528) 2023-10-14T18:17:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 -- 172.21.15.29:0/2193302070 learned_addr learned my addr 172.21.15.29:0/2193302070 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 -- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c070fd0 msgr2=0x7fae9c0c13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c070fd0 0x7fae9c0c13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 -- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0adf30 msgr2=0x7fae9c160a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0adf30 0x7fae9c160a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.997+0000 7faea08b8640 1 -- 172.21.15.29:0/2193302070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8c0094d0 con 0x7fae9c0adb50 2023-10-14T18:17:32.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7faea08b8640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02b950 tx=0x7fae8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:32.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c03ad30 con 0x7fae9c0adb50 2023-10-14T18:17:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c0443f0 con 0x7fae9c0adb50 2023-10-14T18:17:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae8c0447a0 con 0x7fae9c0adb50 2023-10-14T18:17:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c1fab20 con 0x7fae9c0adb50 2023-10-14T18:17:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:31.998+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c1fb0e0 con 0x7fae9c0adb50 2023-10-14T18:17:32.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.000+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c0c20d0 con 0x7fae9c0adb50 2023-10-14T18:17:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.000+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fae8c04d450 con 0x7fae9c0adb50 2023-10-14T18:17:32.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.001+0000 7fae91ffb640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 0x7fae8006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.001+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae8c02d040 con 0x7fae9c0adb50 2023-10-14T18:17:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.002+0000 7faea10b9640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 0x7fae8006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.002+0000 7faea10b9640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 0x7fae8006d4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae980098f0 tx=0x7fae98008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:32.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.006+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae8c076d10 con 0x7fae9c0adb50 2023-10-14T18:17:32.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.197+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fae9c06fa90 con 0x7fae8006afe0 2023-10-14T18:17:32.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.212+0000 7fae91ffb640 1 -- 172.21.15.29:0/2193302070 <== mgr.11010 v2:172.21.15.29:6832/2967217610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287028 (secure 0 0 0) 0x7fae98009e70 con 0x7fae8006afe0 2023-10-14T18:17:32.219 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:32.220 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:17:32.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 msgr2=0x7fae8006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 0x7fae8006d4a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fae980098f0 tx=0x7fae98008040 comp rx=0 tx=0).stop 2023-10-14T18:17:32.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 msgr2=0x7fae9c0c4ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 secure :-1 s=READY pgs=2582 cs=0 l=1 rev1=1 crypto rx=0x7fae8c02b950 tx=0x7fae8c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 shutdown_connections 2023-10-14T18:17:32.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fae8006afe0 0x7fae8006d4a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0adf30 0x7fae9c160a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c0adb50 0x7fae9c0c4ce0 unknown :-1 s=CLOSED pgs=2582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 --2- 172.21.15.29:0/2193302070 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c070fd0 0x7fae9c0c13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 >> 172.21.15.29:0/2193302070 conn(0x7fae9c0cb410 msgr2=0x7fae9c0afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:32.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.218+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 shutdown_connections 2023-10-14T18:17:32.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.219+0000 7faea331d640 1 -- 172.21.15.29:0/2193302070 wait complete. 2023-10-14T18:17:32.239 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T18:17:30.746738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398056,"kb_used_data":23656,"kb_used_omap":12,"kb_used_meta":374259,"kb_avail":754576664,"statfs":{"total":773094113280,"available":772686503936,"internally_reserved":0,"allocated":24223744,"data_stored":12562740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":383241784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.024759"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849697+0000","last_change":"2023-10-14T18:17:27.265344+0000","last_active":"2023-10-14T18:17:28.849697+0000","last_peered":"2023-10-14T18:17:28.849697+0000","last_clean":"2023-10-14T18:17:28.849697+0000","last_became_active":"2023-10-14T18:17:27.265080+0000","last_became_peered":"2023-10-14T18:17:27.265080+0000","last_unstale":"2023-10-14T18:17:28.849697+0000","last_undegraded":"2023-10-14T18:17:28.849697+0000","last_fullsized":"2023-10-14T18:17:28.849697+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846965+0000","last_change":"2023-10-14T18:17:27.261872+0000","last_active":"2023-10-14T18:17:28.846965+0000","last_peered":"2023-10-14T18:17:28.846965+0000","last_clean":"2023-10-14T18:17:28.846965+0000","last_became_active":"2023-10-14T18:17:27.261591+0000","last_became_peered":"2023-10-14T18:17:27.261591+0000","last_unstale":"2023-10-14T18:17:28.846965+0000","last_undegraded":"2023-10-14T18:17:28.846965+0000","last_fullsized":"2023-10-14T18:17:28.846965+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847676+0000","last_change":"2023-10-14T18:17:27.266231+0000","last_active":"2023-10-14T18:17:28.847676+0000","last_peered":"2023-10-14T18:17:28.847676+0000","last_clean":"2023-10-14T18:17:28.847676+0000","last_became_active":"2023-10-14T18:17:27.266037+0000","last_became_peered":"2023-10-14T18:17:27.266037+0000","last_unstale":"2023-10-14T18:17:28.847676+0000","last_undegraded":"2023-10-14T18:17:28.847676+0000","last_fullsized":"2023-10-14T18:17:28.847676+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:55.110847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236970+0000","last_change":"2023-10-14T18:17:27.266341+0000","last_active":"2023-10-14T18:17:28.236970+0000","last_peered":"2023-10-14T18:17:28.236970+0000","last_clean":"2023-10-14T18:17:28.236970+0000","last_became_active":"2023-10-14T18:17:27.265032+0000","last_became_peered":"2023-10-14T18:17:27.265032+0000","last_unstale":"2023-10-14T18:17:28.236970+0000","last_undegraded":"2023-10-14T18:17:28.236970+0000","last_fullsized":"2023-10-14T18:17:28.236970+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:53.877988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845341+0000","last_change":"2023-10-14T18:17:27.255198+0000","last_active":"2023-10-14T18:17:28.845341+0000","last_peered":"2023-10-14T18:17:28.845341+0000","last_clean":"2023-10-14T18:17:28.845341+0000","last_became_active":"2023-10-14T18:17:27.254906+0000","last_became_peered":"2023-10-14T18:17:27.254906+0000","last_unstale":"2023-10-14T18:17:28.845341+0000","last_undegraded":"2023-10-14T18:17:28.845341+0000","last_fullsized":"2023-10-14T18:17:28.845341+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:37.913184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845778+0000","last_change":"2023-10-14T18:17:27.257362+0000","last_active":"2023-10-14T18:17:28.845778+0000","last_peered":"2023-10-14T18:17:28.845778+0000","last_clean":"2023-10-14T18:17:28.845778+0000","last_became_active":"2023-10-14T18:17:27.256429+0000","last_became_peered":"2023-10-14T18:17:27.256429+0000","last_unstale":"2023-10-14T18:17:28.845778+0000","last_undegraded":"2023-10-14T18:17:28.845778+0000","last_fullsized":"2023-10-14T18:17:28.845778+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:13.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847671+0000","last_change":"2023-10-14T18:17:27.260365+0000","last_active":"2023-10-14T18:17:28.847671+0000","last_peered":"2023-10-14T18:17:28.847671+0000","last_clean":"2023-10-14T18:17:28.847671+0000","last_became_active":"2023-10-14T18:17:27.260109+0000","last_became_peered":"2023-10-14T18:17:27.260109+0000","last_unstale":"2023-10-14T18:17:28.847671+0000","last_undegraded":"2023-10-14T18:17:28.847671+0000","last_fullsized":"2023-10-14T18:17:28.847671+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845912+0000","last_change":"2023-10-14T18:17:27.246438+0000","last_active":"2023-10-14T18:17:28.845912+0000","last_peered":"2023-10-14T18:17:28.845912+0000","last_clean":"2023-10-14T18:17:28.845912+0000","last_became_active":"2023-10-14T18:17:27.245859+0000","last_became_peered":"2023-10-14T18:17:27.245859+0000","last_unstale":"2023-10-14T18:17:28.845912+0000","last_undegraded":"2023-10-14T18:17:28.845912+0000","last_fullsized":"2023-10-14T18:17:28.845912+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:20.007218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847952+0000","last_change":"2023-10-14T18:17:27.266296+0000","last_active":"2023-10-14T18:17:28.847952+0000","last_peered":"2023-10-14T18:17:28.847952+0000","last_clean":"2023-10-14T18:17:28.847952+0000","last_became_active":"2023-10-14T18:17:27.265771+0000","last_became_peered":"2023-10-14T18:17:27.265771+0000","last_unstale":"2023-10-14T18:17:28.847952+0000","last_undegraded":"2023-10-14T18:17:28.847952+0000","last_fullsized":"2023-10-14T18:17:28.847952+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234393+0000","last_change":"2023-10-14T18:17:27.254298+0000","last_active":"2023-10-14T18:17:28.234393+0000","last_peered":"2023-10-14T18:17:28.234393+0000","last_clean":"2023-10-14T18:17:28.234393+0000","last_became_active":"2023-10-14T18:17:27.254041+0000","last_became_peered":"2023-10-14T18:17:27.254041+0000","last_unstale":"2023-10-14T18:17:28.234393+0000","last_undegraded":"2023-10-14T18:17:28.234393+0000","last_fullsized":"2023-10-14T18:17:28.234393+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:19:48.453168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847809+0000","last_change":"2023-10-14T18:17:27.249208+0000","last_active":"2023-10-14T18:17:28.847809+0000","last_peered":"2023-10-14T18:17:28.847809+0000","last_clean":"2023-10-14T18:17:28.847809+0000","last_became_active":"2023-10-14T18:17:27.248955+0000","last_became_peered":"2023-10-14T18:17:27.248955+0000","last_unstale":"2023-10-14T18:17:28.847809+0000","last_undegraded":"2023-10-14T18:17:28.847809+0000","last_fullsized":"2023-10-14T18:17:28.847809+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:13:04.646143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238587+0000","last_change":"2023-10-14T18:17:27.253472+0000","last_active":"2023-10-14T18:17:28.238587+0000","last_peered":"2023-10-14T18:17:28.238587+0000","last_clean":"2023-10-14T18:17:28.238587+0000","last_became_active":"2023-10-14T18:17:27.253163+0000","last_became_peered":"2023-10-14T18:17:27.253163+0000","last_unstale":"2023-10-14T18:17:28.238587+0000","last_undegraded":"2023-10-14T18:17:28.238587+0000","last_fullsized":"2023-10-14T18:17:28.238587+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:14.392664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238481+0000","last_change":"2023-10-14T18:17:27.253928+0000","last_active":"2023-10-14T18:17:28.238481+0000","last_peered":"2023-10-14T18:17:28.238481+0000","last_clean":"2023-10-14T18:17:28.238481+0000","last_became_active":"2023-10-14T18:17:27.253612+0000","last_became_peered":"2023-10-14T18:17:27.253612+0000","last_unstale":"2023-10-14T18:17:28.238481+0000","last_undegraded":"2023-10-14T18:17:28.238481+0000","last_fullsized":"2023-10-14T18:17:28.238481+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:02.638978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848497+0000","last_change":"2023-10-14T18:17:27.253619+0000","last_active":"2023-10-14T18:17:28.848497+0000","last_peered":"2023-10-14T18:17:28.848497+0000","last_clean":"2023-10-14T18:17:28.848497+0000","last_became_active":"2023-10-14T18:17:27.253369+0000","last_became_peered":"2023-10-14T18:17:27.253369+0000","last_unstale":"2023-10-14T18:17:28.848497+0000","last_undegraded":"2023-10-14T18:17:28.848497+0000","last_fullsized":"2023-10-14T18:17:28.848497+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849281+0000","last_change":"2023-10-14T18:17:27.259030+0000","last_active":"2023-10-14T18:17:28.849281+0000","last_peered":"2023-10-14T18:17:28.849281+0000","last_clean":"2023-10-14T18:17:28.849281+0000","last_became_active":"2023-10-14T18:17:27.257326+0000","last_became_peered":"2023-10-14T18:17:27.257326+0000","last_unstale":"2023-10-14T18:17:28.849281+0000","last_undegraded":"2023-10-14T18:17:28.849281+0000","last_fullsized":"2023-10-14T18:17:28.849281+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.233794+0000","last_change":"2023-10-14T18:17:27.251446+0000","last_active":"2023-10-14T18:17:28.233794+0000","last_peered":"2023-10-14T18:17:28.233794+0000","last_clean":"2023-10-14T18:17:28.233794+0000","last_became_active":"2023-10-14T18:17:27.251165+0000","last_became_peered":"2023-10-14T18:17:27.251165+0000","last_unstale":"2023-10-14T18:17:28.233794+0000","last_undegraded":"2023-10-14T18:17:28.233794+0000","last_fullsized":"2023-10-14T18:17:28.233794+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:35:33.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849696+0000","last_change":"2023-10-14T18:17:27.265286+0000","last_active":"2023-10-14T18:17:28.849696+0000","last_peered":"2023-10-14T18:17:28.849696+0000","last_clean":"2023-10-14T18:17:28.849696+0000","last_became_active":"2023-10-14T18:17:27.265058+0000","last_became_peered":"2023-10-14T18:17:27.265058+0000","last_unstale":"2023-10-14T18:17:28.849696+0000","last_undegraded":"2023-10-14T18:17:28.849696+0000","last_fullsized":"2023-10-14T18:17:28.849696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234515+0000","last_change":"2023-10-14T18:17:27.251453+0000","last_active":"2023-10-14T18:17:28.234515+0000","last_peered":"2023-10-14T18:17:28.234515+0000","last_clean":"2023-10-14T18:17:28.234515+0000","last_became_active":"2023-10-14T18:17:27.247696+0000","last_became_peered":"2023-10-14T18:17:27.247696+0000","last_unstale":"2023-10-14T18:17:28.234515+0000","last_undegraded":"2023-10-14T18:17:28.234515+0000","last_fullsized":"2023-10-14T18:17:28.234515+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:43:29.058123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234422+0000","last_change":"2023-10-14T18:17:27.255618+0000","last_active":"2023-10-14T18:17:28.234422+0000","last_peered":"2023-10-14T18:17:28.234422+0000","last_clean":"2023-10-14T18:17:28.234422+0000","last_became_active":"2023-10-14T18:17:27.255338+0000","last_became_peered":"2023-10-14T18:17:27.255338+0000","last_unstale":"2023-10-14T18:17:28.234422+0000","last_undegraded":"2023-10-14T18:17:28.234422+0000","last_fullsized":"2023-10-14T18:17:28.234422+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:06.792999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846447+0000","last_change":"2023-10-14T18:17:27.252557+0000","last_active":"2023-10-14T18:17:28.846447+0000","last_peered":"2023-10-14T18:17:28.846447+0000","last_clean":"2023-10-14T18:17:28.846447+0000","last_became_active":"2023-10-14T18:17:27.251949+0000","last_became_peered":"2023-10-14T18:17:27.251949+0000","last_unstale":"2023-10-14T18:17:28.846447+0000","last_undegraded":"2023-10-14T18:17:28.846447+0000","last_fullsized":"2023-10-14T18:17:28.846447+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:45.784134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236216+0000","last_change":"2023-10-14T18:17:27.260919+0000","last_active":"2023-10-14T18:17:28.236216+0000","last_peered":"2023-10-14T18:17:28.236216+0000","last_clean":"2023-10-14T18:17:28.236216+0000","last_became_active":"2023-10-14T18:17:27.260634+0000","last_became_peered":"2023-10-14T18:17:27.260634+0000","last_unstale":"2023-10-14T18:17:28.236216+0000","last_undegraded":"2023-10-14T18:17:28.236216+0000","last_fullsized":"2023-10-14T18:17:28.236216+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:51:46.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848628+0000","last_change":"2023-10-14T18:17:27.256296+0000","last_active":"2023-10-14T18:17:28.848628+0000","last_peered":"2023-10-14T18:17:28.848628+0000","last_clean":"2023-10-14T18:17:28.848628+0000","last_became_active":"2023-10-14T18:17:27.255979+0000","last_became_peered":"2023-10-14T18:17:27.255979+0000","last_unstale":"2023-10-14T18:17:28.848628+0000","last_undegraded":"2023-10-14T18:17:28.848628+0000","last_fullsized":"2023-10-14T18:17:28.848628+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:53.896784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846954+0000","last_change":"2023-10-14T18:17:27.257364+0000","last_active":"2023-10-14T18:17:28.846954+0000","last_peered":"2023-10-14T18:17:28.846954+0000","last_clean":"2023-10-14T18:17:28.846954+0000","last_became_active":"2023-10-14T18:17:27.257009+0000","last_became_peered":"2023-10-14T18:17:27.257009+0000","last_unstale":"2023-10-14T18:17:28.846954+0000","last_undegraded":"2023-10-14T18:17:28.846954+0000","last_fullsized":"2023-10-14T18:17:28.846954+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:24.487542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236069+0000","last_change":"2023-10-14T18:17:27.262932+0000","last_active":"2023-10-14T18:17:28.236069+0000","last_peered":"2023-10-14T18:17:28.236069+0000","last_clean":"2023-10-14T18:17:28.236069+0000","last_became_active":"2023-10-14T18:17:27.262582+0000","last_became_peered":"2023-10-14T18:17:27.262582+0000","last_unstale":"2023-10-14T18:17:28.236069+0000","last_undegraded":"2023-10-14T18:17:28.236069+0000","last_fullsized":"2023-10-14T18:17:28.236069+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:01:50.613186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847641+0000","last_change":"2023-10-14T18:17:27.261126+0000","last_active":"2023-10-14T18:17:28.847641+0000","last_peered":"2023-10-14T18:17:28.847641+0000","last_clean":"2023-10-14T18:17:28.847641+0000","last_became_active":"2023-10-14T18:17:27.260846+0000","last_became_peered":"2023-10-14T18:17:27.260846+0000","last_unstale":"2023-10-14T18:17:28.847641+0000","last_undegraded":"2023-10-14T18:17:28.847641+0000","last_fullsized":"2023-10-14T18:17:28.847641+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849362+0000","last_change":"2023-10-14T18:17:27.262030+0000","last_active":"2023-10-14T18:17:28.849362+0000","last_peered":"2023-10-14T18:17:28.849362+0000","last_clean":"2023-10-14T18:17:28.849362+0000","last_became_active":"2023-10-14T18:17:27.261661+0000","last_became_peered":"2023-10-14T18:17:27.261661+0000","last_unstale":"2023-10-14T18:17:28.849362+0000","last_undegraded":"2023-10-14T18:17:28.849362+0000","last_fullsized":"2023-10-14T18:17:28.849362+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:42.391926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238111+0000","last_change":"2023-10-14T18:17:27.255856+0000","last_active":"2023-10-14T18:17:28.238111+0000","last_peered":"2023-10-14T18:17:28.238111+0000","last_clean":"2023-10-14T18:17:28.238111+0000","last_became_active":"2023-10-14T18:17:27.255581+0000","last_became_peered":"2023-10-14T18:17:27.255581+0000","last_unstale":"2023-10-14T18:17:28.238111+0000","last_undegraded":"2023-10-14T18:17:28.238111+0000","last_fullsized":"2023-10-14T18:17:28.238111+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:42:42.540612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846980+0000","last_change":"2023-10-14T18:17:27.241163+0000","last_active":"2023-10-14T18:17:28.846980+0000","last_peered":"2023-10-14T18:17:28.846980+0000","last_clean":"2023-10-14T18:17:28.846980+0000","last_became_active":"2023-10-14T18:17:27.240705+0000","last_became_peered":"2023-10-14T18:17:27.240705+0000","last_unstale":"2023-10-14T18:17:28.846980+0000","last_undegraded":"2023-10-14T18:17:28.846980+0000","last_fullsized":"2023-10-14T18:17:28.846980+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238032+0000","last_change":"2023-10-14T18:17:27.256462+0000","last_active":"2023-10-14T18:17:28.238032+0000","last_peered":"2023-10-14T18:17:28.238032+0000","last_clean":"2023-10-14T18:17:28.238032+0000","last_became_active":"2023-10-14T18:17:27.256190+0000","last_became_peered":"2023-10-14T18:17:27.256190+0000","last_unstale":"2023-10-14T18:17:28.238032+0000","last_undegraded":"2023-10-14T18:17:28.238032+0000","last_fullsized":"2023-10-14T18:17:28.238032+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:42.705277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846803+0000","last_change":"2023-10-14T18:17:27.260534+0000","last_active":"2023-10-14T18:17:28.846803+0000","last_peered":"2023-10-14T18:17:28.846803+0000","last_clean":"2023-10-14T18:17:28.846803+0000","last_became_active":"2023-10-14T18:17:27.259981+0000","last_became_peered":"2023-10-14T18:17:27.259981+0000","last_unstale":"2023-10-14T18:17:28.846803+0000","last_undegraded":"2023-10-14T18:17:28.846803+0000","last_fullsized":"2023-10-14T18:17:28.846803+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:21.289738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234028+0000","last_change":"2023-10-14T18:17:27.264821+0000","last_active":"2023-10-14T18:17:28.234028+0000","last_peered":"2023-10-14T18:17:28.234028+0000","last_clean":"2023-10-14T18:17:28.234028+0000","last_became_active":"2023-10-14T18:17:27.264504+0000","last_became_peered":"2023-10-14T18:17:27.264504+0000","last_unstale":"2023-10-14T18:17:28.234028+0000","last_undegraded":"2023-10-14T18:17:28.234028+0000","last_fullsized":"2023-10-14T18:17:28.234028+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:44.865727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847902+0000","last_change":"2023-10-14T18:17:27.249693+0000","last_active":"2023-10-14T18:17:28.847902+0000","last_peered":"2023-10-14T18:17:28.847902+0000","last_clean":"2023-10-14T18:17:28.847902+0000","last_became_active":"2023-10-14T18:17:27.249386+0000","last_became_peered":"2023-10-14T18:17:27.249386+0000","last_unstale":"2023-10-14T18:17:28.847902+0000","last_undegraded":"2023-10-14T18:17:28.847902+0000","last_fullsized":"2023-10-14T18:17:28.847902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847372+0000","last_change":"2023-10-14T18:17:27.257929+0000","last_active":"2023-10-14T18:17:28.847372+0000","last_peered":"2023-10-14T18:17:28.847372+0000","last_clean":"2023-10-14T18:17:28.847372+0000","last_became_active":"2023-10-14T18:17:27.257029+0000","last_became_peered":"2023-10-14T18:17:27.257029+0000","last_unstale":"2023-10-14T18:17:28.847372+0000","last_undegraded":"2023-10-14T18:17:28.847372+0000","last_fullsized":"2023-10-14T18:17:28.847372+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"228'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.239297+0000","last_change":"2023-10-14T18:17:26.768176+0000","last_active":"2023-10-14T18:17:28.239297+0000","last_peered":"2023-10-14T18:17:28.239297+0000","last_clean":"2023-10-14T18:17:28.239297+0000","last_became_active":"2023-10-14T18:17:26.767847+0000","last_became_peered":"2023-10-14T18:17:26.767847+0000","last_unstale":"2023-10-14T18:17:28.239297+0000","last_undegraded":"2023-10-14T18:17:28.239297+0000","last_fullsized":"2023-10-14T18:17:28.239297+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:11.998927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848538+0000","last_change":"2023-10-14T18:17:27.251816+0000","last_active":"2023-10-14T18:17:28.848538+0000","last_peered":"2023-10-14T18:17:28.848538+0000","last_clean":"2023-10-14T18:17:28.848538+0000","last_became_active":"2023-10-14T18:17:27.250127+0000","last_became_peered":"2023-10-14T18:17:27.250127+0000","last_unstale":"2023-10-14T18:17:28.848538+0000","last_undegraded":"2023-10-14T18:17:28.848538+0000","last_fullsized":"2023-10-14T18:17:28.848538+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:32.974437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847734+0000","last_change":"2023-10-14T18:17:26.765153+0000","last_active":"2023-10-14T18:17:28.847734+0000","last_peered":"2023-10-14T18:17:28.847734+0000","last_clean":"2023-10-14T18:17:28.847734+0000","last_became_active":"2023-10-14T18:17:26.764789+0000","last_became_peered":"2023-10-14T18:17:26.764789+0000","last_unstale":"2023-10-14T18:17:28.847734+0000","last_undegraded":"2023-10-14T18:17:28.847734+0000","last_fullsized":"2023-10-14T18:17:28.847734+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:01.621072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849508+0000","last_change":"2023-10-14T18:17:27.261385+0000","last_active":"2023-10-14T18:17:28.849508+0000","last_peered":"2023-10-14T18:17:28.849508+0000","last_clean":"2023-10-14T18:17:28.849508+0000","last_became_active":"2023-10-14T18:17:27.261114+0000","last_became_peered":"2023-10-14T18:17:27.261114+0000","last_unstale":"2023-10-14T18:17:28.849508+0000","last_undegraded":"2023-10-14T18:17:28.849508+0000","last_fullsized":"2023-10-14T18:17:28.849508+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:29.463852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848009+0000","last_change":"2023-10-14T18:17:26.760994+0000","last_active":"2023-10-14T18:17:28.848009+0000","last_peered":"2023-10-14T18:17:28.848009+0000","last_clean":"2023-10-14T18:17:28.848009+0000","last_became_active":"2023-10-14T18:17:26.760187+0000","last_became_peered":"2023-10-14T18:17:26.760187+0000","last_unstale":"2023-10-14T18:17:28.848009+0000","last_undegraded":"2023-10-14T18:17:28.848009+0000","last_fullsized":"2023-10-14T18:17:28.848009+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:41.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234485+0000","last_change":"2023-10-14T18:17:27.250058+0000","last_active":"2023-10-14T18:17:28.234485+0000","last_peered":"2023-10-14T18:17:28.234485+0000","last_clean":"2023-10-14T18:17:28.234485+0000","last_became_active":"2023-10-14T18:17:27.249763+0000","last_became_peered":"2023-10-14T18:17:27.249763+0000","last_unstale":"2023-10-14T18:17:28.234485+0000","last_undegraded":"2023-10-14T18:17:28.234485+0000","last_fullsized":"2023-10-14T18:17:28.234485+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:54.713237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237308+0000","last_change":"2023-10-14T18:17:26.763585+0000","last_active":"2023-10-14T18:17:28.237308+0000","last_peered":"2023-10-14T18:17:28.237308+0000","last_clean":"2023-10-14T18:17:28.237308+0000","last_became_active":"2023-10-14T18:17:26.763201+0000","last_became_peered":"2023-10-14T18:17:26.763201+0000","last_unstale":"2023-10-14T18:17:28.237308+0000","last_undegraded":"2023-10-14T18:17:28.237308+0000","last_fullsized":"2023-10-14T18:17:28.237308+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:50:47.243311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236576+0000","last_change":"2023-10-14T18:17:27.265679+0000","last_active":"2023-10-14T18:17:28.236576+0000","last_peered":"2023-10-14T18:17:28.236576+0000","last_clean":"2023-10-14T18:17:28.236576+0000","last_became_active":"2023-10-14T18:17:27.264828+0000","last_became_peered":"2023-10-14T18:17:27.264828+0000","last_unstale":"2023-10-14T18:17:28.236576+0000","last_undegraded":"2023-10-14T18:17:28.236576+0000","last_fullsized":"2023-10-14T18:17:28.236576+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:20:40.110815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845905+0000","last_change":"2023-10-14T18:17:26.758140+0000","last_active":"2023-10-14T18:17:28.845905+0000","last_peered":"2023-10-14T18:17:28.845905+0000","last_clean":"2023-10-14T18:17:28.845905+0000","last_became_active":"2023-10-14T18:17:26.757834+0000","last_became_peered":"2023-10-14T18:17:26.757834+0000","last_unstale":"2023-10-14T18:17:28.845905+0000","last_undegraded":"2023-10-14T18:17:28.845905+0000","last_fullsized":"2023-10-14T18:17:28.845905+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:39.799748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845280+0000","last_change":"2023-10-14T18:17:27.263025+0000","last_active":"2023-10-14T18:17:28.845280+0000","last_peered":"2023-10-14T18:17:28.845280+0000","last_clean":"2023-10-14T18:17:28.845280+0000","last_became_active":"2023-10-14T18:17:27.262769+0000","last_became_peered":"2023-10-14T18:17:27.262769+0000","last_unstale":"2023-10-14T18:17:28.845280+0000","last_undegraded":"2023-10-14T18:17:28.845280+0000","last_fullsized":"2023-10-14T18:17:28.845280+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:32.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.844883+0000","last_change":"2023-10-14T18:17:26.752387+0000","last_active":"2023-10-14T18:17:28.844883+0000","last_peered":"2023-10-14T18:17:28.844883+0000","last_clean":"2023-10-14T18:17:28.844883+0000","last_became_active":"2023-10-14T18:17:26.752097+0000","last_became_peered":"2023-10-14T18:17:26.752097+0000","last_unstale":"2023-10-14T18:17:28.844883+0000","last_undegraded":"2023-10-14T18:17:28.844883+0000","last_fullsized":"2023-10-14T18:17:28.844883+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:47.530809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236441+0000","last_change":"2023-10-14T18:17:27.266633+0000","last_active":"2023-10-14T18:17:28.236441+0000","last_peered":"2023-10-14T18:17:28.236441+0000","last_clean":"2023-10-14T18:17:28.236441+0000","last_became_active":"2023-10-14T18:17:27.265995+0000","last_became_peered":"2023-10-14T18:17:27.265995+0000","last_unstale":"2023-10-14T18:17:28.236441+0000","last_undegraded":"2023-10-14T18:17:28.236441+0000","last_fullsized":"2023-10-14T18:17:28.236441+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:13.429707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846364+0000","last_change":"2023-10-14T18:17:26.757809+0000","last_active":"2023-10-14T18:17:28.846364+0000","last_peered":"2023-10-14T18:17:28.846364+0000","last_clean":"2023-10-14T18:17:28.846364+0000","last_became_active":"2023-10-14T18:17:26.757324+0000","last_became_peered":"2023-10-14T18:17:26.757324+0000","last_unstale":"2023-10-14T18:17:28.846364+0000","last_undegraded":"2023-10-14T18:17:28.846364+0000","last_fullsized":"2023-10-14T18:17:28.846364+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:16.576762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237983+0000","last_change":"2023-10-14T18:17:27.267257+0000","last_active":"2023-10-14T18:17:28.237983+0000","last_peered":"2023-10-14T18:17:28.237983+0000","last_clean":"2023-10-14T18:17:28.237983+0000","last_became_active":"2023-10-14T18:17:27.266927+0000","last_became_peered":"2023-10-14T18:17:27.266927+0000","last_unstale":"2023-10-14T18:17:28.237983+0000","last_undegraded":"2023-10-14T18:17:28.237983+0000","last_fullsized":"2023-10-14T18:17:28.237983+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:43.360869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845084+0000","last_change":"2023-10-14T18:17:26.766951+0000","last_active":"2023-10-14T18:17:28.845084+0000","last_peered":"2023-10-14T18:17:28.845084+0000","last_clean":"2023-10-14T18:17:28.845084+0000","last_became_active":"2023-10-14T18:17:26.766441+0000","last_became_peered":"2023-10-14T18:17:26.766441+0000","last_unstale":"2023-10-14T18:17:28.845084+0000","last_undegraded":"2023-10-14T18:17:28.845084+0000","last_fullsized":"2023-10-14T18:17:28.845084+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:34.151644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848188+0000","last_change":"2023-10-14T18:17:27.251334+0000","last_active":"2023-10-14T18:17:28.848188+0000","last_peered":"2023-10-14T18:17:28.848188+0000","last_clean":"2023-10-14T18:17:28.848188+0000","last_became_active":"2023-10-14T18:17:27.251040+0000","last_became_peered":"2023-10-14T18:17:27.251040+0000","last_unstale":"2023-10-14T18:17:28.848188+0000","last_undegraded":"2023-10-14T18:17:28.848188+0000","last_fullsized":"2023-10-14T18:17:28.848188+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:11.130011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848899+0000","last_change":"2023-10-14T18:17:26.766188+0000","last_active":"2023-10-14T18:17:28.848899+0000","last_peered":"2023-10-14T18:17:28.848899+0000","last_clean":"2023-10-14T18:17:28.848899+0000","last_became_active":"2023-10-14T18:17:26.765885+0000","last_became_peered":"2023-10-14T18:17:26.765885+0000","last_unstale":"2023-10-14T18:17:28.848899+0000","last_undegraded":"2023-10-14T18:17:28.848899+0000","last_fullsized":"2023-10-14T18:17:28.848899+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:00.302980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237175+0000","last_change":"2023-10-14T18:17:27.263006+0000","last_active":"2023-10-14T18:17:28.237175+0000","last_peered":"2023-10-14T18:17:28.237175+0000","last_clean":"2023-10-14T18:17:28.237175+0000","last_became_active":"2023-10-14T18:17:27.262724+0000","last_became_peered":"2023-10-14T18:17:27.262724+0000","last_unstale":"2023-10-14T18:17:28.237175+0000","last_undegraded":"2023-10-14T18:17:28.237175+0000","last_fullsized":"2023-10-14T18:17:28.237175+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:54:28.798619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"228'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234497+0000","last_change":"2023-10-14T18:17:26.757138+0000","last_active":"2023-10-14T18:17:28.234497+0000","last_peered":"2023-10-14T18:17:28.234497+0000","last_clean":"2023-10-14T18:17:28.234497+0000","last_became_active":"2023-10-14T18:17:26.753946+0000","last_became_peered":"2023-10-14T18:17:26.753946+0000","last_unstale":"2023-10-14T18:17:28.234497+0000","last_undegraded":"2023-10-14T18:17:28.234497+0000","last_fullsized":"2023-10-14T18:17:28.234497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:29.218181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849276+0000","last_change":"2023-10-14T18:17:27.257920+0000","last_active":"2023-10-14T18:17:28.849276+0000","last_peered":"2023-10-14T18:17:28.849276+0000","last_clean":"2023-10-14T18:17:28.849276+0000","last_became_active":"2023-10-14T18:17:27.256790+0000","last_became_peered":"2023-10-14T18:17:27.256790+0000","last_unstale":"2023-10-14T18:17:28.849276+0000","last_undegraded":"2023-10-14T18:17:28.849276+0000","last_fullsized":"2023-10-14T18:17:28.849276+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:55.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"228'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237453+0000","last_change":"2023-10-14T18:17:26.761103+0000","last_active":"2023-10-14T18:17:28.237453+0000","last_peered":"2023-10-14T18:17:28.237453+0000","last_clean":"2023-10-14T18:17:28.237453+0000","last_became_active":"2023-10-14T18:17:26.760729+0000","last_became_peered":"2023-10-14T18:17:26.760729+0000","last_unstale":"2023-10-14T18:17:28.237453+0000","last_undegraded":"2023-10-14T18:17:28.237453+0000","last_fullsized":"2023-10-14T18:17:28.237453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:27.269547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845779+0000","last_change":"2023-10-14T18:17:27.253122+0000","last_active":"2023-10-14T18:17:28.845779+0000","last_peered":"2023-10-14T18:17:28.845779+0000","last_clean":"2023-10-14T18:17:28.845779+0000","last_became_active":"2023-10-14T18:17:27.252846+0000","last_became_peered":"2023-10-14T18:17:27.252846+0000","last_unstale":"2023-10-14T18:17:28.845779+0000","last_undegraded":"2023-10-14T18:17:28.845779+0000","last_fullsized":"2023-10-14T18:17:28.845779+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:43.166178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848427+0000","last_change":"2023-10-14T18:17:26.754215+0000","last_active":"2023-10-14T18:17:28.848427+0000","last_peered":"2023-10-14T18:17:28.848427+0000","last_clean":"2023-10-14T18:17:28.848427+0000","last_became_active":"2023-10-14T18:17:26.753857+0000","last_became_peered":"2023-10-14T18:17:26.753857+0000","last_unstale":"2023-10-14T18:17:28.848427+0000","last_undegraded":"2023-10-14T18:17:28.848427+0000","last_fullsized":"2023-10-14T18:17:28.848427+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:10.112105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234207+0000","last_change":"2023-10-14T18:17:27.256523+0000","last_active":"2023-10-14T18:17:28.234207+0000","last_peered":"2023-10-14T18:17:28.234207+0000","last_clean":"2023-10-14T18:17:28.234207+0000","last_became_active":"2023-10-14T18:17:27.255795+0000","last_became_peered":"2023-10-14T18:17:27.255795+0000","last_unstale":"2023-10-14T18:17:28.234207+0000","last_undegraded":"2023-10-14T18:17:28.234207+0000","last_fullsized":"2023-10-14T18:17:28.234207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:52:41.950726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848364+0000","last_change":"2023-10-14T18:17:26.758089+0000","last_active":"2023-10-14T18:17:28.848364+0000","last_peered":"2023-10-14T18:17:28.848364+0000","last_clean":"2023-10-14T18:17:28.848364+0000","last_became_active":"2023-10-14T18:17:26.755775+0000","last_became_peered":"2023-10-14T18:17:26.755775+0000","last_unstale":"2023-10-14T18:17:28.848364+0000","last_undegraded":"2023-10-14T18:17:28.848364+0000","last_fullsized":"2023-10-14T18:17:28.848364+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:57.967793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846691+0000","last_change":"2023-10-14T18:17:27.247857+0000","last_active":"2023-10-14T18:17:28.846691+0000","last_peered":"2023-10-14T18:17:28.846691+0000","last_clean":"2023-10-14T18:17:28.846691+0000","last_became_active":"2023-10-14T18:17:27.247391+0000","last_became_peered":"2023-10-14T18:17:27.247391+0000","last_unstale":"2023-10-14T18:17:28.846691+0000","last_undegraded":"2023-10-14T18:17:28.846691+0000","last_fullsized":"2023-10-14T18:17:28.846691+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.233967+0000","last_change":"2023-10-14T18:17:26.760901+0000","last_active":"2023-10-14T18:17:28.233967+0000","last_peered":"2023-10-14T18:17:28.233967+0000","last_clean":"2023-10-14T18:17:28.233967+0000","last_became_active":"2023-10-14T18:17:26.760610+0000","last_became_peered":"2023-10-14T18:17:26.760610+0000","last_unstale":"2023-10-14T18:17:28.233967+0000","last_undegraded":"2023-10-14T18:17:28.233967+0000","last_fullsized":"2023-10-14T18:17:28.233967+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:03:16.410048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848802+0000","last_change":"2023-10-14T18:17:27.258412+0000","last_active":"2023-10-14T18:17:28.848802+0000","last_peered":"2023-10-14T18:17:28.848802+0000","last_clean":"2023-10-14T18:17:28.848802+0000","last_became_active":"2023-10-14T18:17:27.257835+0000","last_became_peered":"2023-10-14T18:17:27.257835+0000","last_unstale":"2023-10-14T18:17:28.848802+0000","last_undegraded":"2023-10-14T18:17:28.848802+0000","last_fullsized":"2023-10-14T18:17:28.848802+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"228'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846793+0000","last_change":"2023-10-14T18:17:26.758995+0000","last_active":"2023-10-14T18:17:28.846793+0000","last_peered":"2023-10-14T18:17:28.846793+0000","last_clean":"2023-10-14T18:17:28.846793+0000","last_became_active":"2023-10-14T18:17:26.758694+0000","last_became_peered":"2023-10-14T18:17:26.758694+0000","last_unstale":"2023-10-14T18:17:28.846793+0000","last_undegraded":"2023-10-14T18:17:28.846793+0000","last_fullsized":"2023-10-14T18:17:28.846793+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847450+0000","last_change":"2023-10-14T18:17:27.265001+0000","last_active":"2023-10-14T18:17:28.847450+0000","last_peered":"2023-10-14T18:17:28.847450+0000","last_clean":"2023-10-14T18:17:28.847450+0000","last_became_active":"2023-10-14T18:17:27.264106+0000","last_became_peered":"2023-10-14T18:17:27.264106+0000","last_unstale":"2023-10-14T18:17:28.847450+0000","last_undegraded":"2023-10-14T18:17:28.847450+0000","last_fullsized":"2023-10-14T18:17:28.847450+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845558+0000","last_change":"2023-10-14T18:17:26.763127+0000","last_active":"2023-10-14T18:17:28.845558+0000","last_peered":"2023-10-14T18:17:28.845558+0000","last_clean":"2023-10-14T18:17:28.845558+0000","last_became_active":"2023-10-14T18:17:26.762581+0000","last_became_peered":"2023-10-14T18:17:26.762581+0000","last_unstale":"2023-10-14T18:17:28.845558+0000","last_undegraded":"2023-10-14T18:17:28.845558+0000","last_fullsized":"2023-10-14T18:17:28.845558+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:15.476495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847645+0000","last_change":"2023-10-14T18:17:27.260977+0000","last_active":"2023-10-14T18:17:28.847645+0000","last_peered":"2023-10-14T18:17:28.847645+0000","last_clean":"2023-10-14T18:17:28.847645+0000","last_became_active":"2023-10-14T18:17:27.260706+0000","last_became_peered":"2023-10-14T18:17:27.260706+0000","last_unstale":"2023-10-14T18:17:28.847645+0000","last_undegraded":"2023-10-14T18:17:28.847645+0000","last_fullsized":"2023-10-14T18:17:28.847645+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847794+0000","last_change":"2023-10-14T18:17:26.766615+0000","last_active":"2023-10-14T18:17:28.847794+0000","last_peered":"2023-10-14T18:17:28.847794+0000","last_clean":"2023-10-14T18:17:28.847794+0000","last_became_active":"2023-10-14T18:17:26.766167+0000","last_became_peered":"2023-10-14T18:17:26.766167+0000","last_unstale":"2023-10-14T18:17:28.847794+0000","last_undegraded":"2023-10-14T18:17:28.847794+0000","last_fullsized":"2023-10-14T18:17:28.847794+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:13.374759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847118+0000","last_change":"2023-10-14T18:17:27.258724+0000","last_active":"2023-10-14T18:17:28.847118+0000","last_peered":"2023-10-14T18:17:28.847118+0000","last_clean":"2023-10-14T18:17:28.847118+0000","last_became_active":"2023-10-14T18:17:27.255949+0000","last_became_peered":"2023-10-14T18:17:27.255949+0000","last_unstale":"2023-10-14T18:17:28.847118+0000","last_undegraded":"2023-10-14T18:17:28.847118+0000","last_fullsized":"2023-10-14T18:17:28.847118+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:58.985698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234597+0000","last_change":"2023-10-14T18:17:26.760063+0000","last_active":"2023-10-14T18:17:28.234597+0000","last_peered":"2023-10-14T18:17:28.234597+0000","last_clean":"2023-10-14T18:17:28.234597+0000","last_became_active":"2023-10-14T18:17:26.758718+0000","last_became_peered":"2023-10-14T18:17:26.758718+0000","last_unstale":"2023-10-14T18:17:28.234597+0000","last_undegraded":"2023-10-14T18:17:28.234597+0000","last_fullsized":"2023-10-14T18:17:28.234597+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:29.586206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234282+0000","last_change":"2023-10-14T18:17:27.269260+0000","last_active":"2023-10-14T18:17:28.234282+0000","last_peered":"2023-10-14T18:17:28.234282+0000","last_clean":"2023-10-14T18:17:28.234282+0000","last_became_active":"2023-10-14T18:17:27.269043+0000","last_became_peered":"2023-10-14T18:17:27.269043+0000","last_unstale":"2023-10-14T18:17:28.234282+0000","last_undegraded":"2023-10-14T18:17:28.234282+0000","last_fullsized":"2023-10-14T18:17:28.234282+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:53.145962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847208+0000","last_change":"2023-10-14T18:17:26.761052+0000","last_active":"2023-10-14T18:17:28.847208+0000","last_peered":"2023-10-14T18:17:28.847208+0000","last_clean":"2023-10-14T18:17:28.847208+0000","last_became_active":"2023-10-14T18:17:26.760754+0000","last_became_peered":"2023-10-14T18:17:26.760754+0000","last_unstale":"2023-10-14T18:17:28.847208+0000","last_undegraded":"2023-10-14T18:17:28.847208+0000","last_fullsized":"2023-10-14T18:17:28.847208+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:54.469184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234040+0000","last_change":"2023-10-14T18:17:27.248764+0000","last_active":"2023-10-14T18:17:28.234040+0000","last_peered":"2023-10-14T18:17:28.234040+0000","last_clean":"2023-10-14T18:17:28.234040+0000","last_became_active":"2023-10-14T18:17:27.248287+0000","last_became_peered":"2023-10-14T18:17:27.248287+0000","last_unstale":"2023-10-14T18:17:28.234040+0000","last_undegraded":"2023-10-14T18:17:28.234040+0000","last_fullsized":"2023-10-14T18:17:28.234040+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:26.912129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234271+0000","last_change":"2023-10-14T18:17:26.756320+0000","last_active":"2023-10-14T18:17:28.234271+0000","last_peered":"2023-10-14T18:17:28.234271+0000","last_clean":"2023-10-14T18:17:28.234271+0000","last_became_active":"2023-10-14T18:17:26.756040+0000","last_became_peered":"2023-10-14T18:17:26.756040+0000","last_unstale":"2023-10-14T18:17:28.234271+0000","last_undegraded":"2023-10-14T18:17:28.234271+0000","last_fullsized":"2023-10-14T18:17:28.234271+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:10.901473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847675+0000","last_change":"2023-10-14T18:17:27.251912+0000","last_active":"2023-10-14T18:17:28.847675+0000","last_peered":"2023-10-14T18:17:28.847675+0000","last_clean":"2023-10-14T18:17:28.847675+0000","last_became_active":"2023-10-14T18:17:27.251629+0000","last_became_peered":"2023-10-14T18:17:27.251629+0000","last_unstale":"2023-10-14T18:17:28.847675+0000","last_undegraded":"2023-10-14T18:17:28.847675+0000","last_fullsized":"2023-10-14T18:17:28.847675+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:17.594667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848757+0000","last_change":"2023-10-14T18:17:26.767394+0000","last_active":"2023-10-14T18:17:28.848757+0000","last_peered":"2023-10-14T18:17:28.848757+0000","last_clean":"2023-10-14T18:17:28.848757+0000","last_became_active":"2023-10-14T18:17:26.767091+0000","last_became_peered":"2023-10-14T18:17:26.767091+0000","last_unstale":"2023-10-14T18:17:28.848757+0000","last_undegraded":"2023-10-14T18:17:28.848757+0000","last_fullsized":"2023-10-14T18:17:28.848757+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846751+0000","last_change":"2023-10-14T18:17:27.258139+0000","last_active":"2023-10-14T18:17:28.846751+0000","last_peered":"2023-10-14T18:17:28.846751+0000","last_clean":"2023-10-14T18:17:28.846751+0000","last_became_active":"2023-10-14T18:17:27.257907+0000","last_became_peered":"2023-10-14T18:17:27.257907+0000","last_unstale":"2023-10-14T18:17:28.846751+0000","last_undegraded":"2023-10-14T18:17:28.846751+0000","last_fullsized":"2023-10-14T18:17:28.846751+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849164+0000","last_change":"2023-10-14T18:17:26.756432+0000","last_active":"2023-10-14T18:17:28.849164+0000","last_peered":"2023-10-14T18:17:28.849164+0000","last_clean":"2023-10-14T18:17:28.849164+0000","last_became_active":"2023-10-14T18:17:26.756148+0000","last_became_peered":"2023-10-14T18:17:26.756148+0000","last_unstale":"2023-10-14T18:17:28.849164+0000","last_undegraded":"2023-10-14T18:17:28.849164+0000","last_fullsized":"2023-10-14T18:17:28.849164+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:13.420763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847330+0000","last_change":"2023-10-14T18:17:27.254894+0000","last_active":"2023-10-14T18:17:28.847330+0000","last_peered":"2023-10-14T18:17:28.847330+0000","last_clean":"2023-10-14T18:17:28.847330+0000","last_became_active":"2023-10-14T18:17:27.253030+0000","last_became_peered":"2023-10-14T18:17:27.253030+0000","last_unstale":"2023-10-14T18:17:28.847330+0000","last_undegraded":"2023-10-14T18:17:28.847330+0000","last_fullsized":"2023-10-14T18:17:28.847330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:38.076369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845529+0000","last_change":"2023-10-14T18:17:26.758398+0000","last_active":"2023-10-14T18:17:28.845529+0000","last_peered":"2023-10-14T18:17:28.845529+0000","last_clean":"2023-10-14T18:17:28.845529+0000","last_became_active":"2023-10-14T18:17:26.758114+0000","last_became_peered":"2023-10-14T18:17:26.758114+0000","last_unstale":"2023-10-14T18:17:28.845529+0000","last_undegraded":"2023-10-14T18:17:28.845529+0000","last_fullsized":"2023-10-14T18:17:28.845529+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:39.144399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.239136+0000","last_change":"2023-10-14T18:17:27.254575+0000","last_active":"2023-10-14T18:17:28.239136+0000","last_peered":"2023-10-14T18:17:28.239136+0000","last_clean":"2023-10-14T18:17:28.239136+0000","last_became_active":"2023-10-14T18:17:27.254278+0000","last_became_peered":"2023-10-14T18:17:27.254278+0000","last_unstale":"2023-10-14T18:17:28.239136+0000","last_undegraded":"2023-10-14T18:17:28.239136+0000","last_fullsized":"2023-10-14T18:17:28.239136+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848281+0000","last_change":"2023-10-14T18:17:26.768016+0000","last_active":"2023-10-14T18:17:28.848281+0000","last_peered":"2023-10-14T18:17:28.848281+0000","last_clean":"2023-10-14T18:17:28.848281+0000","last_became_active":"2023-10-14T18:17:26.767729+0000","last_became_peered":"2023-10-14T18:17:26.767729+0000","last_unstale":"2023-10-14T18:17:28.848281+0000","last_undegraded":"2023-10-14T18:17:28.848281+0000","last_fullsized":"2023-10-14T18:17:28.848281+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:30.381195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849981+0000","last_change":"2023-10-14T18:17:27.252553+0000","last_active":"2023-10-14T18:17:28.849981+0000","last_peered":"2023-10-14T18:17:28.849981+0000","last_clean":"2023-10-14T18:17:28.849981+0000","last_became_active":"2023-10-14T18:17:27.250542+0000","last_became_peered":"2023-10-14T18:17:27.250542+0000","last_unstale":"2023-10-14T18:17:28.849981+0000","last_undegraded":"2023-10-14T18:17:28.849981+0000","last_fullsized":"2023-10-14T18:17:28.849981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237611+0000","last_change":"2023-10-14T18:17:26.765123+0000","last_active":"2023-10-14T18:17:28.237611+0000","last_peered":"2023-10-14T18:17:28.237611+0000","last_clean":"2023-10-14T18:17:28.237611+0000","last_became_active":"2023-10-14T18:17:26.764801+0000","last_became_peered":"2023-10-14T18:17:26.764801+0000","last_unstale":"2023-10-14T18:17:28.237611+0000","last_undegraded":"2023-10-14T18:17:28.237611+0000","last_fullsized":"2023-10-14T18:17:28.237611+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:03.297149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847046+0000","last_change":"2023-10-14T18:17:27.254416+0000","last_active":"2023-10-14T18:17:28.847046+0000","last_peered":"2023-10-14T18:17:28.847046+0000","last_clean":"2023-10-14T18:17:28.847046+0000","last_became_active":"2023-10-14T18:17:27.254010+0000","last_became_peered":"2023-10-14T18:17:27.254010+0000","last_unstale":"2023-10-14T18:17:28.847046+0000","last_undegraded":"2023-10-14T18:17:28.847046+0000","last_fullsized":"2023-10-14T18:17:28.847046+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845550+0000","last_change":"2023-10-14T18:17:26.754064+0000","last_active":"2023-10-14T18:17:28.845550+0000","last_peered":"2023-10-14T18:17:28.845550+0000","last_clean":"2023-10-14T18:17:28.845550+0000","last_became_active":"2023-10-14T18:17:26.752729+0000","last_became_peered":"2023-10-14T18:17:26.752729+0000","last_unstale":"2023-10-14T18:17:28.845550+0000","last_undegraded":"2023-10-14T18:17:28.845550+0000","last_fullsized":"2023-10-14T18:17:28.845550+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:09.744523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848578+0000","last_change":"2023-10-14T18:17:27.259767+0000","last_active":"2023-10-14T18:17:28.848578+0000","last_peered":"2023-10-14T18:17:28.848578+0000","last_clean":"2023-10-14T18:17:28.848578+0000","last_became_active":"2023-10-14T18:17:27.259468+0000","last_became_peered":"2023-10-14T18:17:27.259468+0000","last_unstale":"2023-10-14T18:17:28.848578+0000","last_undegraded":"2023-10-14T18:17:28.848578+0000","last_fullsized":"2023-10-14T18:17:28.848578+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:40.472642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847300+0000","last_change":"2023-10-14T18:17:26.767200+0000","last_active":"2023-10-14T18:17:28.847300+0000","last_peered":"2023-10-14T18:17:28.847300+0000","last_clean":"2023-10-14T18:17:28.847300+0000","last_became_active":"2023-10-14T18:17:26.766925+0000","last_became_peered":"2023-10-14T18:17:26.766925+0000","last_unstale":"2023-10-14T18:17:28.847300+0000","last_undegraded":"2023-10-14T18:17:28.847300+0000","last_fullsized":"2023-10-14T18:17:28.847300+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:28.445946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848909+0000","last_change":"2023-10-14T18:17:27.261099+0000","last_active":"2023-10-14T18:17:28.848909+0000","last_peered":"2023-10-14T18:17:28.848909+0000","last_clean":"2023-10-14T18:17:28.848909+0000","last_became_active":"2023-10-14T18:17:27.260783+0000","last_became_peered":"2023-10-14T18:17:27.260783+0000","last_unstale":"2023-10-14T18:17:28.848909+0000","last_undegraded":"2023-10-14T18:17:28.848909+0000","last_fullsized":"2023-10-14T18:17:28.848909+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236799+0000","last_change":"2023-10-14T18:17:26.765437+0000","last_active":"2023-10-14T18:17:28.236799+0000","last_peered":"2023-10-14T18:17:28.236799+0000","last_clean":"2023-10-14T18:17:28.236799+0000","last_became_active":"2023-10-14T18:17:26.764932+0000","last_became_peered":"2023-10-14T18:17:26.764932+0000","last_unstale":"2023-10-14T18:17:28.236799+0000","last_undegraded":"2023-10-14T18:17:28.236799+0000","last_fullsized":"2023-10-14T18:17:28.236799+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:00.444875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238448+0000","last_change":"2023-10-14T18:17:27.259073+0000","last_active":"2023-10-14T18:17:28.238448+0000","last_peered":"2023-10-14T18:17:28.238448+0000","last_clean":"2023-10-14T18:17:28.238448+0000","last_became_active":"2023-10-14T18:17:27.258114+0000","last_became_peered":"2023-10-14T18:17:27.258114+0000","last_unstale":"2023-10-14T18:17:28.238448+0000","last_undegraded":"2023-10-14T18:17:28.238448+0000","last_fullsized":"2023-10-14T18:17:28.238448+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:14.438669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846023+0000","last_change":"2023-10-14T18:17:26.762092+0000","last_active":"2023-10-14T18:17:28.846023+0000","last_peered":"2023-10-14T18:17:28.846023+0000","last_clean":"2023-10-14T18:17:28.846023+0000","last_became_active":"2023-10-14T18:17:26.761696+0000","last_became_peered":"2023-10-14T18:17:26.761696+0000","last_unstale":"2023-10-14T18:17:28.846023+0000","last_undegraded":"2023-10-14T18:17:28.846023+0000","last_fullsized":"2023-10-14T18:17:28.846023+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:27.118966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846896+0000","last_change":"2023-10-14T18:17:27.248014+0000","last_active":"2023-10-14T18:17:28.846896+0000","last_peered":"2023-10-14T18:17:28.846896+0000","last_clean":"2023-10-14T18:17:28.846896+0000","last_became_active":"2023-10-14T18:17:27.247706+0000","last_became_peered":"2023-10-14T18:17:27.247706+0000","last_unstale":"2023-10-14T18:17:28.846896+0000","last_undegraded":"2023-10-14T18:17:28.846896+0000","last_fullsized":"2023-10-14T18:17:28.846896+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847679+0000","last_change":"2023-10-14T18:17:26.755699+0000","last_active":"2023-10-14T18:17:28.847679+0000","last_peered":"2023-10-14T18:17:28.847679+0000","last_clean":"2023-10-14T18:17:28.847679+0000","last_became_active":"2023-10-14T18:17:26.755286+0000","last_became_peered":"2023-10-14T18:17:26.755286+0000","last_unstale":"2023-10-14T18:17:28.847679+0000","last_undegraded":"2023-10-14T18:17:28.847679+0000","last_fullsized":"2023-10-14T18:17:28.847679+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:07.796326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846754+0000","last_change":"2023-10-14T18:17:27.258205+0000","last_active":"2023-10-14T18:17:28.846754+0000","last_peered":"2023-10-14T18:17:28.846754+0000","last_clean":"2023-10-14T18:17:28.846754+0000","last_became_active":"2023-10-14T18:17:27.257652+0000","last_became_peered":"2023-10-14T18:17:27.257652+0000","last_unstale":"2023-10-14T18:17:28.846754+0000","last_undegraded":"2023-10-14T18:17:28.846754+0000","last_fullsized":"2023-10-14T18:17:28.846754+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:00.190444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238071+0000","last_change":"2023-10-14T18:17:26.754848+0000","last_active":"2023-10-14T18:17:28.238071+0000","last_peered":"2023-10-14T18:17:28.238071+0000","last_clean":"2023-10-14T18:17:28.238071+0000","last_became_active":"2023-10-14T18:17:26.751997+0000","last_became_peered":"2023-10-14T18:17:26.751997+0000","last_unstale":"2023-10-14T18:17:28.238071+0000","last_undegraded":"2023-10-14T18:17:28.238071+0000","last_fullsized":"2023-10-14T18:17:28.238071+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:31.362757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848653+0000","last_change":"2023-10-14T18:17:27.260210+0000","last_active":"2023-10-14T18:17:28.848653+0000","last_peered":"2023-10-14T18:17:28.848653+0000","last_clean":"2023-10-14T18:17:28.848653+0000","last_became_active":"2023-10-14T18:17:27.259297+0000","last_became_peered":"2023-10-14T18:17:27.259297+0000","last_unstale":"2023-10-14T18:17:28.848653+0000","last_undegraded":"2023-10-14T18:17:28.848653+0000","last_fullsized":"2023-10-14T18:17:28.848653+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238622+0000","last_change":"2023-10-14T18:17:26.757505+0000","last_active":"2023-10-14T18:17:28.238622+0000","last_peered":"2023-10-14T18:17:28.238622+0000","last_clean":"2023-10-14T18:17:28.238622+0000","last_became_active":"2023-10-14T18:17:26.757219+0000","last_became_peered":"2023-10-14T18:17:26.757219+0000","last_unstale":"2023-10-14T18:17:28.238622+0000","last_undegraded":"2023-10-14T18:17:28.238622+0000","last_fullsized":"2023-10-14T18:17:28.238622+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:19.831762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1210,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-14T18:17:27.251502+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:17:27.251502+0000","last_peered":"2023-10-14T18:17:27.251502+0000","last_clean":"2023-10-14T18:17:27.251502+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:17:27.251502+0000","last_undegraded":"2023-10-14T18:17:27.251502+0000","last_fullsized":"2023-10-14T18:17:27.251502+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56784,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315056,"statfs":{"total":96636764160,"available":96578617344,"internally_reserved":0,"allocated":3477504,"data_stored":2014128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.216,"15min":1.2090000000000001},"min":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.573,"5min":1.7030000000000001,"15min":2.1669999999999998},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.232,"15min":1.228},"min":{"1min":0.78300000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6619999999999999,"5min":1.76,"15min":2.2149999999999999},"last":1.78}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.264,"15min":1.276},"min":{"1min":0.67500000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9059999999999999,"5min":2.214,"15min":2.214},"last":1.321},{"interface":"front","average":{"1min":1.341,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.871,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.1659999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.2230000000000001,"15min":1.2669999999999999},"min":{"1min":0.73799999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.546,"5min":1.6799999999999999,"15min":2.004},"last":1.254},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.325,"15min":1.3700000000000001},"min":{"1min":0.85199999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.8260000000000001,"15min":2.2010000000000001},"last":1.028}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3819999999999999,"15min":1.4339999999999999},"min":{"1min":1.0009999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.72,"5min":1.8500000000000001,"15min":2.4430000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5109999999999999,"15min":1.536},"min":{"1min":1.0429999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.875,"5min":1.9079999999999999,"15min":2.2810000000000001},"last":1.7}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.331},"min":{"1min":0.97699999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.8029999999999999,"5min":1.8200000000000001,"15min":2.367},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.8360000000000001,"5min":1.865,"15min":2.661},"last":1.427}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4390000000000001,"15min":1.462},"min":{"1min":1.048,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8740000000000001,"15min":2.516},"last":1.958},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4890000000000001,"15min":1.5229999999999999},"min":{"1min":1.1870000000000001,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.448},"last":2.2559999999999998}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5960000000000001,"15min":1.633},"min":{"1min":1.3169999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.4849999999999999},"last":2.4140000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.5349999999999999,"15min":1.5800000000000001},"min":{"1min":1.115,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":1.948,"5min":1.988,"15min":2.8159999999999998},"last":1.7490000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29828,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342012,"statfs":{"total":96636764160,"available":96606220288,"internally_reserved":0,"allocated":2875392,"data_stored":1422730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.754,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6319999999999999,"5min":2.085,"15min":2.085},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.204,"15min":1.2110000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1910000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2230000000000001,"15min":1.2509999999999999},"min":{"1min":0.746,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.5309999999999999,"5min":1.762,"15min":1.776},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.3580000000000001,"15min":1.359},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.6459999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3819999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.383,"15min":1.3779999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8500000000000001,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.403,"15min":1.4219999999999999},"min":{"1min":1.0600000000000001,"5min":0.99199999999999999,"15min":0.98099999999999998},"max":{"1min":1.6719999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.4670000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3740000000000001,"15min":1.407},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7130000000000001,"5min":2.0379999999999998,"15min":2.2519999999999998},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.90300000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.8220000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.375,"15min":1.381},"min":{"1min":0.85899999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.827,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":2.0030000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3799999999999999,"15min":1.413},"min":{"1min":0.96199999999999997,"5min":0.624,"15min":0.624},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.4940000000000002},"last":1.77},{"interface":"front","average":{"1min":1.516,"5min":1.5469999999999999,"15min":1.579},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.8979999999999999,"5min":2.1600000000000001,"15min":2.5609999999999999},"last":1.6200000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5840000000000001,"15min":1.615},"min":{"1min":0.86799999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6389999999999998,"5min":2.903,"15min":3.423},"last":1.931},{"interface":"front","average":{"1min":1.637,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":1.1870000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.6719999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42316,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329524,"statfs":{"total":96636764160,"available":96593432576,"internally_reserved":0,"allocated":2875392,"data_stored":1422730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.179,"15min":1.2010000000000001},"min":{"1min":0.89700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.716,"5min":1.7569999999999999,"15min":2.0459999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.212,"15min":1.2470000000000001},"min":{"1min":0.878,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.722,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":1.022}]},{"osd":1,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.238,"15min":1.2769999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.274,"15min":1.3080000000000001},"min":{"1min":0.91500000000000004,"5min":0.73599999999999999,"15min":0.63400000000000001},"max":{"1min":1.722,"5min":1.7450000000000001,"15min":2.0419999999999998},"last":1.4810000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.258},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.649,"5min":1.764,"15min":3.7090000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.393,"5min":1.3779999999999999,"15min":1.3959999999999999},"min":{"1min":1.0369999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.768,"5min":1.841,"15min":1.923},"last":1.0860000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4239999999999999,"15min":1.4670000000000001},"min":{"1min":1.071,"5min":0.94199999999999995,"15min":0.81899999999999995},"max":{"1min":1.794,"5min":1.867,"15min":2.4969999999999999},"last":1.734},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.421,"15min":1.4450000000000001},"min":{"1min":1.008,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":1.952,"5min":1.952,"15min":3.8330000000000002},"last":1.151}]},{"osd":4,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3700000000000001,"15min":1.363},"min":{"1min":0.98899999999999999,"5min":0.80100000000000005,"15min":0.78400000000000003},"max":{"1min":1.7889999999999999,"5min":1.954,"15min":2.98},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.3939999999999999,"15min":1.3859999999999999},"min":{"1min":0.90600000000000003,"5min":0.61799999999999999,"15min":0.57499999999999996},"max":{"1min":2.016,"5min":2.016,"15min":3.1280000000000001},"last":1.857}]},{"osd":6,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3859999999999999,"15min":1.417},"min":{"1min":0.97399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.853,"5min":1.8799999999999999,"15min":4.0540000000000003},"last":1.335},{"interface":"front","average":{"1min":1.663,"5min":1.601,"15min":1.627},"min":{"1min":1.3859999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":3.6360000000000001},"last":1.6180000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.516,"15min":1.573},"min":{"1min":1.0409999999999999,"5min":0.68600000000000005,"15min":0.67700000000000005},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.2759999999999998},"last":1.9870000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5600000000000001,"15min":1.556},"min":{"1min":1.1100000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.0459999999999998,"5min":2.206,"15min":3.9390000000000001},"last":1.4430000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64532,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307308,"statfs":{"total":96636764160,"available":96570683392,"internally_reserved":0,"allocated":2891776,"data_stored":1423864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":1.0169999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.341,"5min":1.2729999999999999,"15min":1.288},"min":{"1min":0.80000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.698,"5min":1.887,"15min":2.3140000000000001},"last":0.99199999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.282,"15min":1.323},"min":{"1min":1.0009999999999999,"5min":0.76800000000000002,"15min":0.59399999999999997},"max":{"1min":1.7410000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.107},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2490000000000001,"15min":1.268},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.667,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":1.4379999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.29,"15min":1.2849999999999999},"min":{"1min":0.83699999999999997,"5min":0.81399999999999995,"15min":0.78000000000000003},"max":{"1min":1.659,"5min":1.869,"15min":3.524},"last":1.474},{"interface":"front","average":{"1min":1.379,"5min":1.427,"15min":1.4450000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.278,"15min":1.3460000000000001},"min":{"1min":0.90400000000000003,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.627,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.278},{"interface":"front","average":{"1min":1.425,"5min":1.419,"15min":1.4390000000000001},"min":{"1min":1.0509999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7629999999999999,"5min":2.0569999999999999,"15min":4.1600000000000001},"last":1.5529999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4219999999999999,"15min":1.3959999999999999},"min":{"1min":0.96599999999999997,"5min":0.90200000000000002,"15min":0.69299999999999995},"max":{"1min":1.7949999999999999,"5min":1.9319999999999999,"15min":2.02},"last":1.194},{"interface":"front","average":{"1min":1.407,"5min":1.369,"15min":1.3759999999999999},"min":{"1min":0.98299999999999998,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":1.7589999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.524}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.45,"15min":1.4339999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.81799999999999995},"max":{"1min":1.887,"5min":2.077,"15min":3.3610000000000002},"last":1.609},{"interface":"front","average":{"1min":1.548,"5min":1.575,"15min":1.5700000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.82699999999999996},"max":{"1min":1.823,"5min":2.1080000000000001,"15min":2.3929999999999998},"last":1.3939999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.506,"15min":1.516},"min":{"1min":1.234,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.798,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.581},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.601,"15min":1.595},"min":{"1min":1.246,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":1.9790000000000001,"5min":2.1739999999999999,"15min":3.8610000000000002},"last":1.6779999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52044,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319796,"statfs":{"total":96636764160,"available":96583471104,"internally_reserved":0,"allocated":2883584,"data_stored":1422330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96899999999999997,"15min":0.98399999999999999},"min":{"1min":0.502,"5min":0.377,"15min":0.377},"max":{"1min":1.46,"5min":1.4870000000000001,"15min":1.855},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.179,"15min":1.2050000000000001},"min":{"1min":0.61499999999999999,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4890000000000001,"5min":1.7230000000000001,"15min":1.9970000000000001},"last":1.165}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0089999999999999,"15min":1.0169999999999999},"min":{"1min":0.57499999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.417,"5min":1.8939999999999999,"15min":1.919},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.177,"15min":1.212},"min":{"1min":0.78100000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.228}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.446},"min":{"1min":0.71899999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.9730000000000001,"5min":2.032,"15min":2.032},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.72499999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.7430000000000001,"5min":1.8,"15min":2.3540000000000001},"last":0.84099999999999997}]},{"osd":4,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7350000000000001,"5min":2.153,"15min":2.153},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.484},"min":{"1min":0.85299999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":1.9279999999999999,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.478}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.96499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.8009999999999999,"5min":2.1429999999999998,"15min":2.476},"last":1.325},{"interface":"front","average":{"1min":1.46,"5min":1.518,"15min":1.5649999999999999},"min":{"1min":1.0249999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0739999999999998,"5min":2.246,"15min":2.891},"last":1.1919999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6180000000000001,"15min":1.6319999999999999},"min":{"1min":1.0580000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0390000000000001,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":1.585},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6859999999999999,"15min":1.6870000000000001},"min":{"1min":1.0760000000000001,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.1320000000000001,"5min":2.2949999999999999,"15min":2.585},"last":1.679}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.579,"15min":1.607},"min":{"1min":1.2050000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.859,"5min":2.2370000000000001,"15min":2.419},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.704,"15min":1.696},"min":{"1min":1.0469999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1890000000000001,"5min":2.548,"15min":2.548},"last":1.4299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52040,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319800,"statfs":{"total":96636764160,"available":96583475200,"internally_reserved":0,"allocated":2871296,"data_stored":1422148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":0.97799999999999998,"15min":0.94899999999999995},"min":{"1min":0.65000000000000002,"5min":0.499,"15min":0.499},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.0349999999999999,"15min":1.0960000000000001},"min":{"1min":0.61899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.341,"5min":1.653,"15min":1.8660000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98699999999999999,"15min":1.0169999999999999},"min":{"1min":0.61899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.544,"5min":1.603,"15min":1.881},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.623,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.222}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2509999999999999,"15min":1.2909999999999999},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.56599999999999995},"max":{"1min":1.621,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.407,"15min":1.3999999999999999},"min":{"1min":0.76400000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0089999999999999,"5min":2.1579999999999999,"15min":2.165},"last":1.3320000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.492,"15min":1.478},"min":{"1min":1.077,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3400000000000001,"15min":1.3680000000000001},"min":{"1min":0.92300000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.728,"5min":1.8759999999999999,"15min":2.024},"last":1.944}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.444},"min":{"1min":1.012,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9299999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":1.8999999999999999},{"interface":"front","average":{"1min":1.613,"5min":1.625,"15min":1.5980000000000001},"min":{"1min":1.3460000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":2.0049999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.528,"15min":1.53},"min":{"1min":0.99099999999999999,"5min":0.747,"15min":0.747},"max":{"1min":1.8799999999999999,"5min":2.0529999999999999,"15min":2.1150000000000002},"last":1.794},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.587,"15min":1.617},"min":{"1min":0.85599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":2.1400000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.611,"15min":1.5940000000000001},"min":{"1min":1.208,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.6659999999999999,"15min":1.6859999999999999},"min":{"1min":1.155,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":2.0600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52048,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319792,"statfs":{"total":96636764160,"available":96583467008,"internally_reserved":0,"allocated":2879488,"data_stored":1422204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.96099999999999997,"15min":0.97199999999999998},"min":{"1min":0.755,"5min":0.38,"15min":0.38},"max":{"1min":1.464,"5min":1.647,"15min":2.4929999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.0820000000000001,"15min":1.1299999999999999},"min":{"1min":0.69399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3939999999999999,"5min":1.7070000000000001,"15min":2.5590000000000002},"last":2.847}]},{"osd":2,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.70499999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.593,"5min":1.6930000000000001,"15min":2.7360000000000002},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.23,"15min":1.2669999999999999},"min":{"1min":0.96299999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.732,"5min":1.996,"15min":2.6859999999999999},"last":1.153}]},{"osd":3,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3740000000000001,"15min":1.3939999999999999},"min":{"1min":0.92100000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7290000000000001,"5min":1.952,"15min":2.5910000000000002},"last":2.9940000000000002},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2689999999999999,"15min":1.3120000000000001},"min":{"1min":0.89000000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.774,"5min":1.7849999999999999,"15min":2.823},"last":4.8630000000000004}]},{"osd":4,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.423,"15min":1.4690000000000001},"min":{"1min":1.0920000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.849,"5min":1.857,"15min":2.4430000000000001},"last":4.8090000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.409,"15min":1.444},"min":{"1min":1.0309999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.859,"5min":2.0979999999999999,"15min":2.899},"last":2.218}]},{"osd":5,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.464,"15min":1.476},"min":{"1min":1.0449999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.8979999999999999,"5min":2.0600000000000001,"15min":3.004},"last":2.173},{"interface":"front","average":{"1min":1.607,"5min":1.5760000000000001,"15min":1.575},"min":{"1min":1.1890000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9890000000000001,"5min":2.2280000000000002,"15min":2.5129999999999999},"last":1.1950000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.5840000000000001,"15min":1.6180000000000001},"min":{"1min":1.2350000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8600000000000001,"5min":2.2770000000000001,"15min":2.931},"last":2.8999999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.595,"15min":1.611},"min":{"1min":1.2669999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.0070000000000001,"5min":2.2610000000000001,"15min":3.1190000000000002},"last":10.173999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.6990000000000001,"15min":1.704},"min":{"1min":1.4510000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.0859999999999999,"5min":2.3420000000000001,"15min":2.5009999999999999},"last":3.363},{"interface":"front","average":{"1min":1.657,"5min":1.677,"15min":1.7150000000000001},"min":{"1min":1.3080000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.9570000000000001,"5min":2.1800000000000002,"15min":2.6739999999999999},"last":7.0439999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48464,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323376,"statfs":{"total":96636764160,"available":96587137024,"internally_reserved":0,"allocated":3469312,"data_stored":2012606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.095,"5min":1.129,"15min":1.1499999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.0780000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.25},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.845,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":1.4770000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.7150000000000001,"5min":1.804,"15min":2.0720000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":1.883,"15min":2.194},"last":1.4099999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.321,"15min":1.331},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7629999999999999,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.5129999999999999,"15min":1.512},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.121,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.1319999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.74,"5min":1.8220000000000001,"15min":2.1339999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5409999999999999,"15min":1.581},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.655,"15min":1.661},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.548},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.1080000000000001,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.3759999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.631,"15min":1.661},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0089999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.319},{"interface":"front","average":{"1min":1.639,"5min":1.613,"15min":1.641},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.508}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:17:32.246 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:17:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.514+0000 7f4e49e00640 1 Processor -- start 2023-10-14T18:17:32.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.514+0000 7f4e49e00640 1 -- start start 2023-10-14T18:17:32.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.515+0000 7f4e49e00640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e441593b0 0x7f4e4406f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.515+0000 7f4e49e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 0x7f4e4406fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.515+0000 7f4e49e00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e44159790 con 0x7f4e4406ffd0 2023-10-14T18:17:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.515+0000 7f4e49e00640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e440c9b20 con 0x7f4e441593b0 2023-10-14T18:17:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.515+0000 7f4e49e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e440c9e00 con 0x7f4e440caa10 2023-10-14T18:17:32.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e441593b0 0x7f4e4406f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4e441593b0 0x7f4e4406f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55542/0 (socket says 172.21.15.29:55542) 2023-10-14T18:17:32.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 -- 172.21.15.29:0/3510330815 learned_addr learned my addr 172.21.15.29:0/3510330815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e42ffd640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 0x7f4e4406fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 -- 172.21.15.29:0/3510330815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 msgr2=0x7f4e4406fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:32.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 0x7f4e4406fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 -- 172.21.15.29:0/3510330815 >> v1:172.21.15.186:6789/0 conn(0x7f4e4406ffd0 legacy=0x7f4e4415ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e437fe640 1 -- 172.21.15.29:0/3510330815 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e440ca0e0 con 0x7f4e441593b0 2023-10-14T18:17:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.516+0000 7f4e42ffd640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 0x7f4e4406fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.517+0000 7f4e437fe640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e441593b0 0x7f4e4406f550 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400af60 tx=0x7f4e3402ceb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=375cd72502eedbc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:32.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.518+0000 7f4e427fc640 1 -- 172.21.15.29:0/3510330815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3402d850 con 0x7f4e441593b0 2023-10-14T18:17:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.518+0000 7f4e427fc640 1 -- 172.21.15.29:0/3510330815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e3402d9b0 con 0x7f4e441593b0 2023-10-14T18:17:32.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.518+0000 7f4e427fc640 1 -- 172.21.15.29:0/3510330815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e3402dd40 con 0x7f4e441593b0 2023-10-14T18:17:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.518+0000 7f4e49e00640 1 -- 172.21.15.29:0/3510330815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e441593b0 msgr2=0x7f4e4406f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.518+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e441593b0 0x7f4e4406f550 secure :-1 s=READY pgs=2583 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400af60 tx=0x7f4e3402ceb0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 -- 172.21.15.29:0/3510330815 shutdown_connections 2023-10-14T18:17:32.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e440caa10 0x7f4e4406fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3510330815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e441593b0 0x7f4e4406f550 unknown :-1 s=CLOSED pgs=2583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 -- 172.21.15.29:0/3510330815 >> 172.21.15.29:0/3510330815 conn(0x7f4e440cb410 msgr2=0x7f4e440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 -- 172.21.15.29:0/3510330815 shutdown_connections 2023-10-14T18:17:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.519+0000 7f4e49e00640 1 -- 172.21.15.29:0/3510330815 wait complete. 2023-10-14T18:17:32.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.521+0000 7f4e49e00640 1 Processor -- start 2023-10-14T18:17:32.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.521+0000 7f4e49e00640 1 -- start start 2023-10-14T18:17:32.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e4406ffd0 0x7f4e441f2530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e441593b0 0x7f4e440ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4415c630 con 0x7f4e440caa10 2023-10-14T18:17:32.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4415bd00 con 0x7f4e441593b0 2023-10-14T18:17:32.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e49e00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e4415bfe0 con 0x7f4e4406ffd0 2023-10-14T18:17:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e42ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e42ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55558/0 (socket says 172.21.15.29:55558) 2023-10-14T18:17:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.522+0000 7f4e42ffd640 1 -- 172.21.15.29:0/3391498427 learned_addr learned my addr 172.21.15.29:0/3391498427 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e437fe640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e4406ffd0 0x7f4e441f2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e43fff640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e441593b0 0x7f4e440ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 -- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e4406ffd0 msgr2=0x7f4e441f2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e4406ffd0 0x7f4e441f2530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 -- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e441593b0 msgr2=0x7f4e440ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e441593b0 0x7f4e440ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 -- 172.21.15.29:0/3391498427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e3400ac40 con 0x7f4e440caa10 2023-10-14T18:17:32.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e42ffd640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c7f0 tx=0x7f4e2c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:32.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c007820 con 0x7f4e440caa10 2023-10-14T18:17:32.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.524+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e4415c2c0 con 0x7f4e440caa10 2023-10-14T18:17:32.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.523+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e2c007980 con 0x7f4e440caa10 2023-10-14T18:17:32.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.524+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c007cf0 con 0x7f4e440caa10 2023-10-14T18:17:32.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.524+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e4414e0b0 con 0x7f4e440caa10 2023-10-14T18:17:32.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.525+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e4404e160 con 0x7f4e440caa10 2023-10-14T18:17:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.526+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4e2c0113f0 con 0x7f4e440caa10 2023-10-14T18:17:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.527+0000 7f4e3bfff640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 0x7f4e2406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.527+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4e2c089cd0 con 0x7f4e440caa10 2023-10-14T18:17:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.527+0000 7f4e437fe640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 0x7f4e2406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:32.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.528+0000 7f4e437fe640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 0x7f4e2406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e34036000 tx=0x7f4e34031000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:32.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.532+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4e2c054a40 con 0x7f4e440caa10 2023-10-14T18:17:32.690 INFO:tasks.ceph.mds.a.smithi029.stderr:2023-10-14T18:17:32.689+0000 7f17227a1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:17:32.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.724+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4e440ca3c0 con 0x7f4e2406af90 2023-10-14T18:17:32.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.746+0000 7f4e3bfff640 1 -- 172.21.15.29:0/3391498427 <== mgr.11010 v2:172.21.15.29:6832/2967217610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287028 (secure 0 0 0) 0x7f4e3402dc20 con 0x7f4e2406af90 2023-10-14T18:17:32.751 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:32.753 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:17:32.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 msgr2=0x7f4e2406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 0x7f4e2406d450 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4e34036000 tx=0x7f4e34031000 comp rx=0 tx=0).stop 2023-10-14T18:17:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 msgr2=0x7f4e4414db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 secure :-1 s=READY pgs=2584 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c00c7f0 tx=0x7f4e2c00ccb0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 shutdown_connections 2023-10-14T18:17:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f4e2406af90 0x7f4e2406d450 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e441593b0 0x7f4e440ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e440caa10 0x7f4e4414db70 unknown :-1 s=CLOSED pgs=2584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 --2- 172.21.15.29:0/3391498427 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4e4406ffd0 0x7f4e441f2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:32.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 >> 172.21.15.29:0/3391498427 conn(0x7f4e440cb410 msgr2=0x7f4e440b0680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 shutdown_connections 2023-10-14T18:17:32.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:32.752+0000 7f4e49e00640 1 -- 172.21.15.29:0/3391498427 wait complete. 2023-10-14T18:17:32.773 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2023-10-14T18:17:30.746738+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":623,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":386,"ondisk_log_size":386,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":158,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398056,"kb_used_data":23656,"kb_used_omap":12,"kb_used_meta":374259,"kb_avail":754576664,"statfs":{"total":773094113280,"available":772686503936,"internally_reserved":0,"allocated":24223744,"data_stored":12562740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":383241784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":21,"apply_latency_ms":21,"commit_latency_ns":21000000,"apply_latency_ns":21000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.024759"},"pg_stats":[{"pgid":"27.3f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849697+0000","last_change":"2023-10-14T18:17:27.265344+0000","last_active":"2023-10-14T18:17:28.849697+0000","last_peered":"2023-10-14T18:17:28.849697+0000","last_clean":"2023-10-14T18:17:28.849697+0000","last_became_active":"2023-10-14T18:17:27.265080+0000","last_became_peered":"2023-10-14T18:17:27.265080+0000","last_unstale":"2023-10-14T18:17:28.849697+0000","last_undegraded":"2023-10-14T18:17:28.849697+0000","last_fullsized":"2023-10-14T18:17:28.849697+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846965+0000","last_change":"2023-10-14T18:17:27.261872+0000","last_active":"2023-10-14T18:17:28.846965+0000","last_peered":"2023-10-14T18:17:28.846965+0000","last_clean":"2023-10-14T18:17:28.846965+0000","last_became_active":"2023-10-14T18:17:27.261591+0000","last_became_peered":"2023-10-14T18:17:27.261591+0000","last_unstale":"2023-10-14T18:17:28.846965+0000","last_undegraded":"2023-10-14T18:17:28.846965+0000","last_fullsized":"2023-10-14T18:17:28.846965+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847676+0000","last_change":"2023-10-14T18:17:27.266231+0000","last_active":"2023-10-14T18:17:28.847676+0000","last_peered":"2023-10-14T18:17:28.847676+0000","last_clean":"2023-10-14T18:17:28.847676+0000","last_became_active":"2023-10-14T18:17:27.266037+0000","last_became_peered":"2023-10-14T18:17:27.266037+0000","last_unstale":"2023-10-14T18:17:28.847676+0000","last_undegraded":"2023-10-14T18:17:28.847676+0000","last_fullsized":"2023-10-14T18:17:28.847676+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:55.110847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236970+0000","last_change":"2023-10-14T18:17:27.266341+0000","last_active":"2023-10-14T18:17:28.236970+0000","last_peered":"2023-10-14T18:17:28.236970+0000","last_clean":"2023-10-14T18:17:28.236970+0000","last_became_active":"2023-10-14T18:17:27.265032+0000","last_became_peered":"2023-10-14T18:17:27.265032+0000","last_unstale":"2023-10-14T18:17:28.236970+0000","last_undegraded":"2023-10-14T18:17:28.236970+0000","last_fullsized":"2023-10-14T18:17:28.236970+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:53.877988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845341+0000","last_change":"2023-10-14T18:17:27.255198+0000","last_active":"2023-10-14T18:17:28.845341+0000","last_peered":"2023-10-14T18:17:28.845341+0000","last_clean":"2023-10-14T18:17:28.845341+0000","last_became_active":"2023-10-14T18:17:27.254906+0000","last_became_peered":"2023-10-14T18:17:27.254906+0000","last_unstale":"2023-10-14T18:17:28.845341+0000","last_undegraded":"2023-10-14T18:17:28.845341+0000","last_fullsized":"2023-10-14T18:17:28.845341+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:37.913184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845778+0000","last_change":"2023-10-14T18:17:27.257362+0000","last_active":"2023-10-14T18:17:28.845778+0000","last_peered":"2023-10-14T18:17:28.845778+0000","last_clean":"2023-10-14T18:17:28.845778+0000","last_became_active":"2023-10-14T18:17:27.256429+0000","last_became_peered":"2023-10-14T18:17:27.256429+0000","last_unstale":"2023-10-14T18:17:28.845778+0000","last_undegraded":"2023-10-14T18:17:28.845778+0000","last_fullsized":"2023-10-14T18:17:28.845778+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:13.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847671+0000","last_change":"2023-10-14T18:17:27.260365+0000","last_active":"2023-10-14T18:17:28.847671+0000","last_peered":"2023-10-14T18:17:28.847671+0000","last_clean":"2023-10-14T18:17:28.847671+0000","last_became_active":"2023-10-14T18:17:27.260109+0000","last_became_peered":"2023-10-14T18:17:27.260109+0000","last_unstale":"2023-10-14T18:17:28.847671+0000","last_undegraded":"2023-10-14T18:17:28.847671+0000","last_fullsized":"2023-10-14T18:17:28.847671+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845912+0000","last_change":"2023-10-14T18:17:27.246438+0000","last_active":"2023-10-14T18:17:28.845912+0000","last_peered":"2023-10-14T18:17:28.845912+0000","last_clean":"2023-10-14T18:17:28.845912+0000","last_became_active":"2023-10-14T18:17:27.245859+0000","last_became_peered":"2023-10-14T18:17:27.245859+0000","last_unstale":"2023-10-14T18:17:28.845912+0000","last_undegraded":"2023-10-14T18:17:28.845912+0000","last_fullsized":"2023-10-14T18:17:28.845912+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:20.007218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847952+0000","last_change":"2023-10-14T18:17:27.266296+0000","last_active":"2023-10-14T18:17:28.847952+0000","last_peered":"2023-10-14T18:17:28.847952+0000","last_clean":"2023-10-14T18:17:28.847952+0000","last_became_active":"2023-10-14T18:17:27.265771+0000","last_became_peered":"2023-10-14T18:17:27.265771+0000","last_unstale":"2023-10-14T18:17:28.847952+0000","last_undegraded":"2023-10-14T18:17:28.847952+0000","last_fullsized":"2023-10-14T18:17:28.847952+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234393+0000","last_change":"2023-10-14T18:17:27.254298+0000","last_active":"2023-10-14T18:17:28.234393+0000","last_peered":"2023-10-14T18:17:28.234393+0000","last_clean":"2023-10-14T18:17:28.234393+0000","last_became_active":"2023-10-14T18:17:27.254041+0000","last_became_peered":"2023-10-14T18:17:27.254041+0000","last_unstale":"2023-10-14T18:17:28.234393+0000","last_undegraded":"2023-10-14T18:17:28.234393+0000","last_fullsized":"2023-10-14T18:17:28.234393+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:19:48.453168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847809+0000","last_change":"2023-10-14T18:17:27.249208+0000","last_active":"2023-10-14T18:17:28.847809+0000","last_peered":"2023-10-14T18:17:28.847809+0000","last_clean":"2023-10-14T18:17:28.847809+0000","last_became_active":"2023-10-14T18:17:27.248955+0000","last_became_peered":"2023-10-14T18:17:27.248955+0000","last_unstale":"2023-10-14T18:17:28.847809+0000","last_undegraded":"2023-10-14T18:17:28.847809+0000","last_fullsized":"2023-10-14T18:17:28.847809+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:13:04.646143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238587+0000","last_change":"2023-10-14T18:17:27.253472+0000","last_active":"2023-10-14T18:17:28.238587+0000","last_peered":"2023-10-14T18:17:28.238587+0000","last_clean":"2023-10-14T18:17:28.238587+0000","last_became_active":"2023-10-14T18:17:27.253163+0000","last_became_peered":"2023-10-14T18:17:27.253163+0000","last_unstale":"2023-10-14T18:17:28.238587+0000","last_undegraded":"2023-10-14T18:17:28.238587+0000","last_fullsized":"2023-10-14T18:17:28.238587+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:14.392664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238481+0000","last_change":"2023-10-14T18:17:27.253928+0000","last_active":"2023-10-14T18:17:28.238481+0000","last_peered":"2023-10-14T18:17:28.238481+0000","last_clean":"2023-10-14T18:17:28.238481+0000","last_became_active":"2023-10-14T18:17:27.253612+0000","last_became_peered":"2023-10-14T18:17:27.253612+0000","last_unstale":"2023-10-14T18:17:28.238481+0000","last_undegraded":"2023-10-14T18:17:28.238481+0000","last_fullsized":"2023-10-14T18:17:28.238481+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:02.638978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848497+0000","last_change":"2023-10-14T18:17:27.253619+0000","last_active":"2023-10-14T18:17:28.848497+0000","last_peered":"2023-10-14T18:17:28.848497+0000","last_clean":"2023-10-14T18:17:28.848497+0000","last_became_active":"2023-10-14T18:17:27.253369+0000","last_became_peered":"2023-10-14T18:17:27.253369+0000","last_unstale":"2023-10-14T18:17:28.848497+0000","last_undegraded":"2023-10-14T18:17:28.848497+0000","last_fullsized":"2023-10-14T18:17:28.848497+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849281+0000","last_change":"2023-10-14T18:17:27.259030+0000","last_active":"2023-10-14T18:17:28.849281+0000","last_peered":"2023-10-14T18:17:28.849281+0000","last_clean":"2023-10-14T18:17:28.849281+0000","last_became_active":"2023-10-14T18:17:27.257326+0000","last_became_peered":"2023-10-14T18:17:27.257326+0000","last_unstale":"2023-10-14T18:17:28.849281+0000","last_undegraded":"2023-10-14T18:17:28.849281+0000","last_fullsized":"2023-10-14T18:17:28.849281+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.233794+0000","last_change":"2023-10-14T18:17:27.251446+0000","last_active":"2023-10-14T18:17:28.233794+0000","last_peered":"2023-10-14T18:17:28.233794+0000","last_clean":"2023-10-14T18:17:28.233794+0000","last_became_active":"2023-10-14T18:17:27.251165+0000","last_became_peered":"2023-10-14T18:17:27.251165+0000","last_unstale":"2023-10-14T18:17:28.233794+0000","last_undegraded":"2023-10-14T18:17:28.233794+0000","last_fullsized":"2023-10-14T18:17:28.233794+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:35:33.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849696+0000","last_change":"2023-10-14T18:17:27.265286+0000","last_active":"2023-10-14T18:17:28.849696+0000","last_peered":"2023-10-14T18:17:28.849696+0000","last_clean":"2023-10-14T18:17:28.849696+0000","last_became_active":"2023-10-14T18:17:27.265058+0000","last_became_peered":"2023-10-14T18:17:27.265058+0000","last_unstale":"2023-10-14T18:17:28.849696+0000","last_undegraded":"2023-10-14T18:17:28.849696+0000","last_fullsized":"2023-10-14T18:17:28.849696+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234515+0000","last_change":"2023-10-14T18:17:27.251453+0000","last_active":"2023-10-14T18:17:28.234515+0000","last_peered":"2023-10-14T18:17:28.234515+0000","last_clean":"2023-10-14T18:17:28.234515+0000","last_became_active":"2023-10-14T18:17:27.247696+0000","last_became_peered":"2023-10-14T18:17:27.247696+0000","last_unstale":"2023-10-14T18:17:28.234515+0000","last_undegraded":"2023-10-14T18:17:28.234515+0000","last_fullsized":"2023-10-14T18:17:28.234515+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:43:29.058123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234422+0000","last_change":"2023-10-14T18:17:27.255618+0000","last_active":"2023-10-14T18:17:28.234422+0000","last_peered":"2023-10-14T18:17:28.234422+0000","last_clean":"2023-10-14T18:17:28.234422+0000","last_became_active":"2023-10-14T18:17:27.255338+0000","last_became_peered":"2023-10-14T18:17:27.255338+0000","last_unstale":"2023-10-14T18:17:28.234422+0000","last_undegraded":"2023-10-14T18:17:28.234422+0000","last_fullsized":"2023-10-14T18:17:28.234422+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:06.792999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846447+0000","last_change":"2023-10-14T18:17:27.252557+0000","last_active":"2023-10-14T18:17:28.846447+0000","last_peered":"2023-10-14T18:17:28.846447+0000","last_clean":"2023-10-14T18:17:28.846447+0000","last_became_active":"2023-10-14T18:17:27.251949+0000","last_became_peered":"2023-10-14T18:17:27.251949+0000","last_unstale":"2023-10-14T18:17:28.846447+0000","last_undegraded":"2023-10-14T18:17:28.846447+0000","last_fullsized":"2023-10-14T18:17:28.846447+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:45.784134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236216+0000","last_change":"2023-10-14T18:17:27.260919+0000","last_active":"2023-10-14T18:17:28.236216+0000","last_peered":"2023-10-14T18:17:28.236216+0000","last_clean":"2023-10-14T18:17:28.236216+0000","last_became_active":"2023-10-14T18:17:27.260634+0000","last_became_peered":"2023-10-14T18:17:27.260634+0000","last_unstale":"2023-10-14T18:17:28.236216+0000","last_undegraded":"2023-10-14T18:17:28.236216+0000","last_fullsized":"2023-10-14T18:17:28.236216+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:51:46.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848628+0000","last_change":"2023-10-14T18:17:27.256296+0000","last_active":"2023-10-14T18:17:28.848628+0000","last_peered":"2023-10-14T18:17:28.848628+0000","last_clean":"2023-10-14T18:17:28.848628+0000","last_became_active":"2023-10-14T18:17:27.255979+0000","last_became_peered":"2023-10-14T18:17:27.255979+0000","last_unstale":"2023-10-14T18:17:28.848628+0000","last_undegraded":"2023-10-14T18:17:28.848628+0000","last_fullsized":"2023-10-14T18:17:28.848628+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:53.896784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846954+0000","last_change":"2023-10-14T18:17:27.257364+0000","last_active":"2023-10-14T18:17:28.846954+0000","last_peered":"2023-10-14T18:17:28.846954+0000","last_clean":"2023-10-14T18:17:28.846954+0000","last_became_active":"2023-10-14T18:17:27.257009+0000","last_became_peered":"2023-10-14T18:17:27.257009+0000","last_unstale":"2023-10-14T18:17:28.846954+0000","last_undegraded":"2023-10-14T18:17:28.846954+0000","last_fullsized":"2023-10-14T18:17:28.846954+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:24.487542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236069+0000","last_change":"2023-10-14T18:17:27.262932+0000","last_active":"2023-10-14T18:17:28.236069+0000","last_peered":"2023-10-14T18:17:28.236069+0000","last_clean":"2023-10-14T18:17:28.236069+0000","last_became_active":"2023-10-14T18:17:27.262582+0000","last_became_peered":"2023-10-14T18:17:27.262582+0000","last_unstale":"2023-10-14T18:17:28.236069+0000","last_undegraded":"2023-10-14T18:17:28.236069+0000","last_fullsized":"2023-10-14T18:17:28.236069+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:01:50.613186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847641+0000","last_change":"2023-10-14T18:17:27.261126+0000","last_active":"2023-10-14T18:17:28.847641+0000","last_peered":"2023-10-14T18:17:28.847641+0000","last_clean":"2023-10-14T18:17:28.847641+0000","last_became_active":"2023-10-14T18:17:27.260846+0000","last_became_peered":"2023-10-14T18:17:27.260846+0000","last_unstale":"2023-10-14T18:17:28.847641+0000","last_undegraded":"2023-10-14T18:17:28.847641+0000","last_fullsized":"2023-10-14T18:17:28.847641+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849362+0000","last_change":"2023-10-14T18:17:27.262030+0000","last_active":"2023-10-14T18:17:28.849362+0000","last_peered":"2023-10-14T18:17:28.849362+0000","last_clean":"2023-10-14T18:17:28.849362+0000","last_became_active":"2023-10-14T18:17:27.261661+0000","last_became_peered":"2023-10-14T18:17:27.261661+0000","last_unstale":"2023-10-14T18:17:28.849362+0000","last_undegraded":"2023-10-14T18:17:28.849362+0000","last_fullsized":"2023-10-14T18:17:28.849362+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:42.391926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238111+0000","last_change":"2023-10-14T18:17:27.255856+0000","last_active":"2023-10-14T18:17:28.238111+0000","last_peered":"2023-10-14T18:17:28.238111+0000","last_clean":"2023-10-14T18:17:28.238111+0000","last_became_active":"2023-10-14T18:17:27.255581+0000","last_became_peered":"2023-10-14T18:17:27.255581+0000","last_unstale":"2023-10-14T18:17:28.238111+0000","last_undegraded":"2023-10-14T18:17:28.238111+0000","last_fullsized":"2023-10-14T18:17:28.238111+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:42:42.540612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846980+0000","last_change":"2023-10-14T18:17:27.241163+0000","last_active":"2023-10-14T18:17:28.846980+0000","last_peered":"2023-10-14T18:17:28.846980+0000","last_clean":"2023-10-14T18:17:28.846980+0000","last_became_active":"2023-10-14T18:17:27.240705+0000","last_became_peered":"2023-10-14T18:17:27.240705+0000","last_unstale":"2023-10-14T18:17:28.846980+0000","last_undegraded":"2023-10-14T18:17:28.846980+0000","last_fullsized":"2023-10-14T18:17:28.846980+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238032+0000","last_change":"2023-10-14T18:17:27.256462+0000","last_active":"2023-10-14T18:17:28.238032+0000","last_peered":"2023-10-14T18:17:28.238032+0000","last_clean":"2023-10-14T18:17:28.238032+0000","last_became_active":"2023-10-14T18:17:27.256190+0000","last_became_peered":"2023-10-14T18:17:27.256190+0000","last_unstale":"2023-10-14T18:17:28.238032+0000","last_undegraded":"2023-10-14T18:17:28.238032+0000","last_fullsized":"2023-10-14T18:17:28.238032+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:42.705277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846803+0000","last_change":"2023-10-14T18:17:27.260534+0000","last_active":"2023-10-14T18:17:28.846803+0000","last_peered":"2023-10-14T18:17:28.846803+0000","last_clean":"2023-10-14T18:17:28.846803+0000","last_became_active":"2023-10-14T18:17:27.259981+0000","last_became_peered":"2023-10-14T18:17:27.259981+0000","last_unstale":"2023-10-14T18:17:28.846803+0000","last_undegraded":"2023-10-14T18:17:28.846803+0000","last_fullsized":"2023-10-14T18:17:28.846803+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:21.289738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234028+0000","last_change":"2023-10-14T18:17:27.264821+0000","last_active":"2023-10-14T18:17:28.234028+0000","last_peered":"2023-10-14T18:17:28.234028+0000","last_clean":"2023-10-14T18:17:28.234028+0000","last_became_active":"2023-10-14T18:17:27.264504+0000","last_became_peered":"2023-10-14T18:17:27.264504+0000","last_unstale":"2023-10-14T18:17:28.234028+0000","last_undegraded":"2023-10-14T18:17:28.234028+0000","last_fullsized":"2023-10-14T18:17:28.234028+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:44.865727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847902+0000","last_change":"2023-10-14T18:17:27.249693+0000","last_active":"2023-10-14T18:17:28.847902+0000","last_peered":"2023-10-14T18:17:28.847902+0000","last_clean":"2023-10-14T18:17:28.847902+0000","last_became_active":"2023-10-14T18:17:27.249386+0000","last_became_peered":"2023-10-14T18:17:27.249386+0000","last_unstale":"2023-10-14T18:17:28.847902+0000","last_undegraded":"2023-10-14T18:17:28.847902+0000","last_fullsized":"2023-10-14T18:17:28.847902+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847372+0000","last_change":"2023-10-14T18:17:27.257929+0000","last_active":"2023-10-14T18:17:28.847372+0000","last_peered":"2023-10-14T18:17:28.847372+0000","last_clean":"2023-10-14T18:17:28.847372+0000","last_became_active":"2023-10-14T18:17:27.257029+0000","last_became_peered":"2023-10-14T18:17:27.257029+0000","last_unstale":"2023-10-14T18:17:28.847372+0000","last_undegraded":"2023-10-14T18:17:28.847372+0000","last_fullsized":"2023-10-14T18:17:28.847372+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1f","version":"228'4","reported_seq":16,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.239297+0000","last_change":"2023-10-14T18:17:26.768176+0000","last_active":"2023-10-14T18:17:28.239297+0000","last_peered":"2023-10-14T18:17:28.239297+0000","last_clean":"2023-10-14T18:17:28.239297+0000","last_became_active":"2023-10-14T18:17:26.767847+0000","last_became_peered":"2023-10-14T18:17:26.767847+0000","last_unstale":"2023-10-14T18:17:28.239297+0000","last_undegraded":"2023-10-14T18:17:28.239297+0000","last_fullsized":"2023-10-14T18:17:28.239297+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:11.998927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848538+0000","last_change":"2023-10-14T18:17:27.251816+0000","last_active":"2023-10-14T18:17:28.848538+0000","last_peered":"2023-10-14T18:17:28.848538+0000","last_clean":"2023-10-14T18:17:28.848538+0000","last_became_active":"2023-10-14T18:17:27.250127+0000","last_became_peered":"2023-10-14T18:17:27.250127+0000","last_unstale":"2023-10-14T18:17:28.848538+0000","last_undegraded":"2023-10-14T18:17:28.848538+0000","last_fullsized":"2023-10-14T18:17:28.848538+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:32.974437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847734+0000","last_change":"2023-10-14T18:17:26.765153+0000","last_active":"2023-10-14T18:17:28.847734+0000","last_peered":"2023-10-14T18:17:28.847734+0000","last_clean":"2023-10-14T18:17:28.847734+0000","last_became_active":"2023-10-14T18:17:26.764789+0000","last_became_peered":"2023-10-14T18:17:26.764789+0000","last_unstale":"2023-10-14T18:17:28.847734+0000","last_undegraded":"2023-10-14T18:17:28.847734+0000","last_fullsized":"2023-10-14T18:17:28.847734+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:01.621072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849508+0000","last_change":"2023-10-14T18:17:27.261385+0000","last_active":"2023-10-14T18:17:28.849508+0000","last_peered":"2023-10-14T18:17:28.849508+0000","last_clean":"2023-10-14T18:17:28.849508+0000","last_became_active":"2023-10-14T18:17:27.261114+0000","last_became_peered":"2023-10-14T18:17:27.261114+0000","last_unstale":"2023-10-14T18:17:28.849508+0000","last_undegraded":"2023-10-14T18:17:28.849508+0000","last_fullsized":"2023-10-14T18:17:28.849508+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:29.463852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1d","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848009+0000","last_change":"2023-10-14T18:17:26.760994+0000","last_active":"2023-10-14T18:17:28.848009+0000","last_peered":"2023-10-14T18:17:28.848009+0000","last_clean":"2023-10-14T18:17:28.848009+0000","last_became_active":"2023-10-14T18:17:26.760187+0000","last_became_peered":"2023-10-14T18:17:26.760187+0000","last_unstale":"2023-10-14T18:17:28.848009+0000","last_undegraded":"2023-10-14T18:17:28.848009+0000","last_fullsized":"2023-10-14T18:17:28.848009+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:41.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234485+0000","last_change":"2023-10-14T18:17:27.250058+0000","last_active":"2023-10-14T18:17:28.234485+0000","last_peered":"2023-10-14T18:17:28.234485+0000","last_clean":"2023-10-14T18:17:28.234485+0000","last_became_active":"2023-10-14T18:17:27.249763+0000","last_became_peered":"2023-10-14T18:17:27.249763+0000","last_unstale":"2023-10-14T18:17:28.234485+0000","last_undegraded":"2023-10-14T18:17:28.234485+0000","last_fullsized":"2023-10-14T18:17:28.234485+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:54.713237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1c","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237308+0000","last_change":"2023-10-14T18:17:26.763585+0000","last_active":"2023-10-14T18:17:28.237308+0000","last_peered":"2023-10-14T18:17:28.237308+0000","last_clean":"2023-10-14T18:17:28.237308+0000","last_became_active":"2023-10-14T18:17:26.763201+0000","last_became_peered":"2023-10-14T18:17:26.763201+0000","last_unstale":"2023-10-14T18:17:28.237308+0000","last_undegraded":"2023-10-14T18:17:28.237308+0000","last_fullsized":"2023-10-14T18:17:28.237308+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:50:47.243311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236576+0000","last_change":"2023-10-14T18:17:27.265679+0000","last_active":"2023-10-14T18:17:28.236576+0000","last_peered":"2023-10-14T18:17:28.236576+0000","last_clean":"2023-10-14T18:17:28.236576+0000","last_became_active":"2023-10-14T18:17:27.264828+0000","last_became_peered":"2023-10-14T18:17:27.264828+0000","last_unstale":"2023-10-14T18:17:28.236576+0000","last_undegraded":"2023-10-14T18:17:28.236576+0000","last_fullsized":"2023-10-14T18:17:28.236576+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:20:40.110815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1a","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845905+0000","last_change":"2023-10-14T18:17:26.758140+0000","last_active":"2023-10-14T18:17:28.845905+0000","last_peered":"2023-10-14T18:17:28.845905+0000","last_clean":"2023-10-14T18:17:28.845905+0000","last_became_active":"2023-10-14T18:17:26.757834+0000","last_became_peered":"2023-10-14T18:17:26.757834+0000","last_unstale":"2023-10-14T18:17:28.845905+0000","last_undegraded":"2023-10-14T18:17:28.845905+0000","last_fullsized":"2023-10-14T18:17:28.845905+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:39.799748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845280+0000","last_change":"2023-10-14T18:17:27.263025+0000","last_active":"2023-10-14T18:17:28.845280+0000","last_peered":"2023-10-14T18:17:28.845280+0000","last_clean":"2023-10-14T18:17:28.845280+0000","last_became_active":"2023-10-14T18:17:27.262769+0000","last_became_peered":"2023-10-14T18:17:27.262769+0000","last_unstale":"2023-10-14T18:17:28.845280+0000","last_undegraded":"2023-10-14T18:17:28.845280+0000","last_fullsized":"2023-10-14T18:17:28.845280+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:32.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.844883+0000","last_change":"2023-10-14T18:17:26.752387+0000","last_active":"2023-10-14T18:17:28.844883+0000","last_peered":"2023-10-14T18:17:28.844883+0000","last_clean":"2023-10-14T18:17:28.844883+0000","last_became_active":"2023-10-14T18:17:26.752097+0000","last_became_peered":"2023-10-14T18:17:26.752097+0000","last_unstale":"2023-10-14T18:17:28.844883+0000","last_undegraded":"2023-10-14T18:17:28.844883+0000","last_fullsized":"2023-10-14T18:17:28.844883+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:47.530809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236441+0000","last_change":"2023-10-14T18:17:27.266633+0000","last_active":"2023-10-14T18:17:28.236441+0000","last_peered":"2023-10-14T18:17:28.236441+0000","last_clean":"2023-10-14T18:17:28.236441+0000","last_became_active":"2023-10-14T18:17:27.265995+0000","last_became_peered":"2023-10-14T18:17:27.265995+0000","last_unstale":"2023-10-14T18:17:28.236441+0000","last_undegraded":"2023-10-14T18:17:28.236441+0000","last_fullsized":"2023-10-14T18:17:28.236441+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:13.429707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846364+0000","last_change":"2023-10-14T18:17:26.757809+0000","last_active":"2023-10-14T18:17:28.846364+0000","last_peered":"2023-10-14T18:17:28.846364+0000","last_clean":"2023-10-14T18:17:28.846364+0000","last_became_active":"2023-10-14T18:17:26.757324+0000","last_became_peered":"2023-10-14T18:17:26.757324+0000","last_unstale":"2023-10-14T18:17:28.846364+0000","last_undegraded":"2023-10-14T18:17:28.846364+0000","last_fullsized":"2023-10-14T18:17:28.846364+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:16.576762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237983+0000","last_change":"2023-10-14T18:17:27.267257+0000","last_active":"2023-10-14T18:17:28.237983+0000","last_peered":"2023-10-14T18:17:28.237983+0000","last_clean":"2023-10-14T18:17:28.237983+0000","last_became_active":"2023-10-14T18:17:27.266927+0000","last_became_peered":"2023-10-14T18:17:27.266927+0000","last_unstale":"2023-10-14T18:17:28.237983+0000","last_undegraded":"2023-10-14T18:17:28.237983+0000","last_fullsized":"2023-10-14T18:17:28.237983+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:43.360869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845084+0000","last_change":"2023-10-14T18:17:26.766951+0000","last_active":"2023-10-14T18:17:28.845084+0000","last_peered":"2023-10-14T18:17:28.845084+0000","last_clean":"2023-10-14T18:17:28.845084+0000","last_became_active":"2023-10-14T18:17:26.766441+0000","last_became_peered":"2023-10-14T18:17:26.766441+0000","last_unstale":"2023-10-14T18:17:28.845084+0000","last_undegraded":"2023-10-14T18:17:28.845084+0000","last_fullsized":"2023-10-14T18:17:28.845084+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:34.151644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848188+0000","last_change":"2023-10-14T18:17:27.251334+0000","last_active":"2023-10-14T18:17:28.848188+0000","last_peered":"2023-10-14T18:17:28.848188+0000","last_clean":"2023-10-14T18:17:28.848188+0000","last_became_active":"2023-10-14T18:17:27.251040+0000","last_became_peered":"2023-10-14T18:17:27.251040+0000","last_unstale":"2023-10-14T18:17:28.848188+0000","last_undegraded":"2023-10-14T18:17:28.848188+0000","last_fullsized":"2023-10-14T18:17:28.848188+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:11.130011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848899+0000","last_change":"2023-10-14T18:17:26.766188+0000","last_active":"2023-10-14T18:17:28.848899+0000","last_peered":"2023-10-14T18:17:28.848899+0000","last_clean":"2023-10-14T18:17:28.848899+0000","last_became_active":"2023-10-14T18:17:26.765885+0000","last_became_peered":"2023-10-14T18:17:26.765885+0000","last_unstale":"2023-10-14T18:17:28.848899+0000","last_undegraded":"2023-10-14T18:17:28.848899+0000","last_fullsized":"2023-10-14T18:17:28.848899+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:00.302980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237175+0000","last_change":"2023-10-14T18:17:27.263006+0000","last_active":"2023-10-14T18:17:28.237175+0000","last_peered":"2023-10-14T18:17:28.237175+0000","last_clean":"2023-10-14T18:17:28.237175+0000","last_became_active":"2023-10-14T18:17:27.262724+0000","last_became_peered":"2023-10-14T18:17:27.262724+0000","last_unstale":"2023-10-14T18:17:28.237175+0000","last_undegraded":"2023-10-14T18:17:28.237175+0000","last_fullsized":"2023-10-14T18:17:28.237175+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:54:28.798619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.15","version":"228'3","reported_seq":15,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234497+0000","last_change":"2023-10-14T18:17:26.757138+0000","last_active":"2023-10-14T18:17:28.234497+0000","last_peered":"2023-10-14T18:17:28.234497+0000","last_clean":"2023-10-14T18:17:28.234497+0000","last_became_active":"2023-10-14T18:17:26.753946+0000","last_became_peered":"2023-10-14T18:17:26.753946+0000","last_unstale":"2023-10-14T18:17:28.234497+0000","last_undegraded":"2023-10-14T18:17:28.234497+0000","last_fullsized":"2023-10-14T18:17:28.234497+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:29.218181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849276+0000","last_change":"2023-10-14T18:17:27.257920+0000","last_active":"2023-10-14T18:17:28.849276+0000","last_peered":"2023-10-14T18:17:28.849276+0000","last_clean":"2023-10-14T18:17:28.849276+0000","last_became_active":"2023-10-14T18:17:27.256790+0000","last_became_peered":"2023-10-14T18:17:27.256790+0000","last_unstale":"2023-10-14T18:17:28.849276+0000","last_undegraded":"2023-10-14T18:17:28.849276+0000","last_fullsized":"2023-10-14T18:17:28.849276+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:55.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.14","version":"228'6","reported_seq":18,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237453+0000","last_change":"2023-10-14T18:17:26.761103+0000","last_active":"2023-10-14T18:17:28.237453+0000","last_peered":"2023-10-14T18:17:28.237453+0000","last_clean":"2023-10-14T18:17:28.237453+0000","last_became_active":"2023-10-14T18:17:26.760729+0000","last_became_peered":"2023-10-14T18:17:26.760729+0000","last_unstale":"2023-10-14T18:17:28.237453+0000","last_undegraded":"2023-10-14T18:17:28.237453+0000","last_fullsized":"2023-10-14T18:17:28.237453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:27.269547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845779+0000","last_change":"2023-10-14T18:17:27.253122+0000","last_active":"2023-10-14T18:17:28.845779+0000","last_peered":"2023-10-14T18:17:28.845779+0000","last_clean":"2023-10-14T18:17:28.845779+0000","last_became_active":"2023-10-14T18:17:27.252846+0000","last_became_peered":"2023-10-14T18:17:27.252846+0000","last_unstale":"2023-10-14T18:17:28.845779+0000","last_undegraded":"2023-10-14T18:17:28.845779+0000","last_fullsized":"2023-10-14T18:17:28.845779+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:43.166178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.13","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848427+0000","last_change":"2023-10-14T18:17:26.754215+0000","last_active":"2023-10-14T18:17:28.848427+0000","last_peered":"2023-10-14T18:17:28.848427+0000","last_clean":"2023-10-14T18:17:28.848427+0000","last_became_active":"2023-10-14T18:17:26.753857+0000","last_became_peered":"2023-10-14T18:17:26.753857+0000","last_unstale":"2023-10-14T18:17:28.848427+0000","last_undegraded":"2023-10-14T18:17:28.848427+0000","last_fullsized":"2023-10-14T18:17:28.848427+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:10.112105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234207+0000","last_change":"2023-10-14T18:17:27.256523+0000","last_active":"2023-10-14T18:17:28.234207+0000","last_peered":"2023-10-14T18:17:28.234207+0000","last_clean":"2023-10-14T18:17:28.234207+0000","last_became_active":"2023-10-14T18:17:27.255795+0000","last_became_peered":"2023-10-14T18:17:27.255795+0000","last_unstale":"2023-10-14T18:17:28.234207+0000","last_undegraded":"2023-10-14T18:17:28.234207+0000","last_fullsized":"2023-10-14T18:17:28.234207+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:52:41.950726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848364+0000","last_change":"2023-10-14T18:17:26.758089+0000","last_active":"2023-10-14T18:17:28.848364+0000","last_peered":"2023-10-14T18:17:28.848364+0000","last_clean":"2023-10-14T18:17:28.848364+0000","last_became_active":"2023-10-14T18:17:26.755775+0000","last_became_peered":"2023-10-14T18:17:26.755775+0000","last_unstale":"2023-10-14T18:17:28.848364+0000","last_undegraded":"2023-10-14T18:17:28.848364+0000","last_fullsized":"2023-10-14T18:17:28.848364+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:57.967793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846691+0000","last_change":"2023-10-14T18:17:27.247857+0000","last_active":"2023-10-14T18:17:28.846691+0000","last_peered":"2023-10-14T18:17:28.846691+0000","last_clean":"2023-10-14T18:17:28.846691+0000","last_became_active":"2023-10-14T18:17:27.247391+0000","last_became_peered":"2023-10-14T18:17:27.247391+0000","last_unstale":"2023-10-14T18:17:28.846691+0000","last_undegraded":"2023-10-14T18:17:28.846691+0000","last_fullsized":"2023-10-14T18:17:28.846691+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.233967+0000","last_change":"2023-10-14T18:17:26.760901+0000","last_active":"2023-10-14T18:17:28.233967+0000","last_peered":"2023-10-14T18:17:28.233967+0000","last_clean":"2023-10-14T18:17:28.233967+0000","last_became_active":"2023-10-14T18:17:26.760610+0000","last_became_peered":"2023-10-14T18:17:26.760610+0000","last_unstale":"2023-10-14T18:17:28.233967+0000","last_undegraded":"2023-10-14T18:17:28.233967+0000","last_fullsized":"2023-10-14T18:17:28.233967+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:03:16.410048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848802+0000","last_change":"2023-10-14T18:17:27.258412+0000","last_active":"2023-10-14T18:17:28.848802+0000","last_peered":"2023-10-14T18:17:28.848802+0000","last_clean":"2023-10-14T18:17:28.848802+0000","last_became_active":"2023-10-14T18:17:27.257835+0000","last_became_peered":"2023-10-14T18:17:27.257835+0000","last_unstale":"2023-10-14T18:17:28.848802+0000","last_undegraded":"2023-10-14T18:17:28.848802+0000","last_fullsized":"2023-10-14T18:17:28.848802+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.10","version":"228'8","reported_seq":20,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846793+0000","last_change":"2023-10-14T18:17:26.758995+0000","last_active":"2023-10-14T18:17:28.846793+0000","last_peered":"2023-10-14T18:17:28.846793+0000","last_clean":"2023-10-14T18:17:28.846793+0000","last_became_active":"2023-10-14T18:17:26.758694+0000","last_became_peered":"2023-10-14T18:17:26.758694+0000","last_unstale":"2023-10-14T18:17:28.846793+0000","last_undegraded":"2023-10-14T18:17:28.846793+0000","last_fullsized":"2023-10-14T18:17:28.846793+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847450+0000","last_change":"2023-10-14T18:17:27.265001+0000","last_active":"2023-10-14T18:17:28.847450+0000","last_peered":"2023-10-14T18:17:28.847450+0000","last_clean":"2023-10-14T18:17:28.847450+0000","last_became_active":"2023-10-14T18:17:27.264106+0000","last_became_peered":"2023-10-14T18:17:27.264106+0000","last_unstale":"2023-10-14T18:17:28.847450+0000","last_undegraded":"2023-10-14T18:17:28.847450+0000","last_fullsized":"2023-10-14T18:17:28.847450+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.f","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845558+0000","last_change":"2023-10-14T18:17:26.763127+0000","last_active":"2023-10-14T18:17:28.845558+0000","last_peered":"2023-10-14T18:17:28.845558+0000","last_clean":"2023-10-14T18:17:28.845558+0000","last_became_active":"2023-10-14T18:17:26.762581+0000","last_became_peered":"2023-10-14T18:17:26.762581+0000","last_unstale":"2023-10-14T18:17:28.845558+0000","last_undegraded":"2023-10-14T18:17:28.845558+0000","last_fullsized":"2023-10-14T18:17:28.845558+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:15.476495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847645+0000","last_change":"2023-10-14T18:17:27.260977+0000","last_active":"2023-10-14T18:17:28.847645+0000","last_peered":"2023-10-14T18:17:28.847645+0000","last_clean":"2023-10-14T18:17:28.847645+0000","last_became_active":"2023-10-14T18:17:27.260706+0000","last_became_peered":"2023-10-14T18:17:27.260706+0000","last_unstale":"2023-10-14T18:17:28.847645+0000","last_undegraded":"2023-10-14T18:17:28.847645+0000","last_fullsized":"2023-10-14T18:17:28.847645+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.e","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847794+0000","last_change":"2023-10-14T18:17:26.766615+0000","last_active":"2023-10-14T18:17:28.847794+0000","last_peered":"2023-10-14T18:17:28.847794+0000","last_clean":"2023-10-14T18:17:28.847794+0000","last_became_active":"2023-10-14T18:17:26.766167+0000","last_became_peered":"2023-10-14T18:17:26.766167+0000","last_unstale":"2023-10-14T18:17:28.847794+0000","last_undegraded":"2023-10-14T18:17:28.847794+0000","last_fullsized":"2023-10-14T18:17:28.847794+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:13.374759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847118+0000","last_change":"2023-10-14T18:17:27.258724+0000","last_active":"2023-10-14T18:17:28.847118+0000","last_peered":"2023-10-14T18:17:28.847118+0000","last_clean":"2023-10-14T18:17:28.847118+0000","last_became_active":"2023-10-14T18:17:27.255949+0000","last_became_peered":"2023-10-14T18:17:27.255949+0000","last_unstale":"2023-10-14T18:17:28.847118+0000","last_undegraded":"2023-10-14T18:17:28.847118+0000","last_fullsized":"2023-10-14T18:17:28.847118+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:58.985698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.d","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234597+0000","last_change":"2023-10-14T18:17:26.760063+0000","last_active":"2023-10-14T18:17:28.234597+0000","last_peered":"2023-10-14T18:17:28.234597+0000","last_clean":"2023-10-14T18:17:28.234597+0000","last_became_active":"2023-10-14T18:17:26.758718+0000","last_became_peered":"2023-10-14T18:17:26.758718+0000","last_unstale":"2023-10-14T18:17:28.234597+0000","last_undegraded":"2023-10-14T18:17:28.234597+0000","last_fullsized":"2023-10-14T18:17:28.234597+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:29.586206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234282+0000","last_change":"2023-10-14T18:17:27.269260+0000","last_active":"2023-10-14T18:17:28.234282+0000","last_peered":"2023-10-14T18:17:28.234282+0000","last_clean":"2023-10-14T18:17:28.234282+0000","last_became_active":"2023-10-14T18:17:27.269043+0000","last_became_peered":"2023-10-14T18:17:27.269043+0000","last_unstale":"2023-10-14T18:17:28.234282+0000","last_undegraded":"2023-10-14T18:17:28.234282+0000","last_fullsized":"2023-10-14T18:17:28.234282+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:53.145962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847208+0000","last_change":"2023-10-14T18:17:26.761052+0000","last_active":"2023-10-14T18:17:28.847208+0000","last_peered":"2023-10-14T18:17:28.847208+0000","last_clean":"2023-10-14T18:17:28.847208+0000","last_became_active":"2023-10-14T18:17:26.760754+0000","last_became_peered":"2023-10-14T18:17:26.760754+0000","last_unstale":"2023-10-14T18:17:28.847208+0000","last_undegraded":"2023-10-14T18:17:28.847208+0000","last_fullsized":"2023-10-14T18:17:28.847208+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:54.469184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234040+0000","last_change":"2023-10-14T18:17:27.248764+0000","last_active":"2023-10-14T18:17:28.234040+0000","last_peered":"2023-10-14T18:17:28.234040+0000","last_clean":"2023-10-14T18:17:28.234040+0000","last_became_active":"2023-10-14T18:17:27.248287+0000","last_became_peered":"2023-10-14T18:17:27.248287+0000","last_unstale":"2023-10-14T18:17:28.234040+0000","last_undegraded":"2023-10-14T18:17:28.234040+0000","last_fullsized":"2023-10-14T18:17:28.234040+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:26.912129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.b","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.234271+0000","last_change":"2023-10-14T18:17:26.756320+0000","last_active":"2023-10-14T18:17:28.234271+0000","last_peered":"2023-10-14T18:17:28.234271+0000","last_clean":"2023-10-14T18:17:28.234271+0000","last_became_active":"2023-10-14T18:17:26.756040+0000","last_became_peered":"2023-10-14T18:17:26.756040+0000","last_unstale":"2023-10-14T18:17:28.234271+0000","last_undegraded":"2023-10-14T18:17:28.234271+0000","last_fullsized":"2023-10-14T18:17:28.234271+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:10.901473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847675+0000","last_change":"2023-10-14T18:17:27.251912+0000","last_active":"2023-10-14T18:17:28.847675+0000","last_peered":"2023-10-14T18:17:28.847675+0000","last_clean":"2023-10-14T18:17:28.847675+0000","last_became_active":"2023-10-14T18:17:27.251629+0000","last_became_peered":"2023-10-14T18:17:27.251629+0000","last_unstale":"2023-10-14T18:17:28.847675+0000","last_undegraded":"2023-10-14T18:17:28.847675+0000","last_fullsized":"2023-10-14T18:17:28.847675+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:17.594667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848757+0000","last_change":"2023-10-14T18:17:26.767394+0000","last_active":"2023-10-14T18:17:28.848757+0000","last_peered":"2023-10-14T18:17:28.848757+0000","last_clean":"2023-10-14T18:17:28.848757+0000","last_became_active":"2023-10-14T18:17:26.767091+0000","last_became_peered":"2023-10-14T18:17:26.767091+0000","last_unstale":"2023-10-14T18:17:28.848757+0000","last_undegraded":"2023-10-14T18:17:28.848757+0000","last_fullsized":"2023-10-14T18:17:28.848757+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846751+0000","last_change":"2023-10-14T18:17:27.258139+0000","last_active":"2023-10-14T18:17:28.846751+0000","last_peered":"2023-10-14T18:17:28.846751+0000","last_clean":"2023-10-14T18:17:28.846751+0000","last_became_active":"2023-10-14T18:17:27.257907+0000","last_became_peered":"2023-10-14T18:17:27.257907+0000","last_unstale":"2023-10-14T18:17:28.846751+0000","last_undegraded":"2023-10-14T18:17:28.846751+0000","last_fullsized":"2023-10-14T18:17:28.846751+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849164+0000","last_change":"2023-10-14T18:17:26.756432+0000","last_active":"2023-10-14T18:17:28.849164+0000","last_peered":"2023-10-14T18:17:28.849164+0000","last_clean":"2023-10-14T18:17:28.849164+0000","last_became_active":"2023-10-14T18:17:26.756148+0000","last_became_peered":"2023-10-14T18:17:26.756148+0000","last_unstale":"2023-10-14T18:17:28.849164+0000","last_undegraded":"2023-10-14T18:17:28.849164+0000","last_fullsized":"2023-10-14T18:17:28.849164+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:13.420763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847330+0000","last_change":"2023-10-14T18:17:27.254894+0000","last_active":"2023-10-14T18:17:28.847330+0000","last_peered":"2023-10-14T18:17:28.847330+0000","last_clean":"2023-10-14T18:17:28.847330+0000","last_became_active":"2023-10-14T18:17:27.253030+0000","last_became_peered":"2023-10-14T18:17:27.253030+0000","last_unstale":"2023-10-14T18:17:28.847330+0000","last_undegraded":"2023-10-14T18:17:28.847330+0000","last_fullsized":"2023-10-14T18:17:28.847330+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:38.076369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845529+0000","last_change":"2023-10-14T18:17:26.758398+0000","last_active":"2023-10-14T18:17:28.845529+0000","last_peered":"2023-10-14T18:17:28.845529+0000","last_clean":"2023-10-14T18:17:28.845529+0000","last_became_active":"2023-10-14T18:17:26.758114+0000","last_became_peered":"2023-10-14T18:17:26.758114+0000","last_unstale":"2023-10-14T18:17:28.845529+0000","last_undegraded":"2023-10-14T18:17:28.845529+0000","last_fullsized":"2023-10-14T18:17:28.845529+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:39.144399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.239136+0000","last_change":"2023-10-14T18:17:27.254575+0000","last_active":"2023-10-14T18:17:28.239136+0000","last_peered":"2023-10-14T18:17:28.239136+0000","last_clean":"2023-10-14T18:17:28.239136+0000","last_became_active":"2023-10-14T18:17:27.254278+0000","last_became_peered":"2023-10-14T18:17:27.254278+0000","last_unstale":"2023-10-14T18:17:28.239136+0000","last_undegraded":"2023-10-14T18:17:28.239136+0000","last_fullsized":"2023-10-14T18:17:28.239136+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"228'2","reported_seq":14,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848281+0000","last_change":"2023-10-14T18:17:26.768016+0000","last_active":"2023-10-14T18:17:28.848281+0000","last_peered":"2023-10-14T18:17:28.848281+0000","last_clean":"2023-10-14T18:17:28.848281+0000","last_became_active":"2023-10-14T18:17:26.767729+0000","last_became_peered":"2023-10-14T18:17:26.767729+0000","last_unstale":"2023-10-14T18:17:28.848281+0000","last_undegraded":"2023-10-14T18:17:28.848281+0000","last_fullsized":"2023-10-14T18:17:28.848281+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:30.381195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.849981+0000","last_change":"2023-10-14T18:17:27.252553+0000","last_active":"2023-10-14T18:17:28.849981+0000","last_peered":"2023-10-14T18:17:28.849981+0000","last_clean":"2023-10-14T18:17:28.849981+0000","last_became_active":"2023-10-14T18:17:27.250542+0000","last_became_peered":"2023-10-14T18:17:27.250542+0000","last_unstale":"2023-10-14T18:17:28.849981+0000","last_undegraded":"2023-10-14T18:17:28.849981+0000","last_fullsized":"2023-10-14T18:17:28.849981+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.6","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.237611+0000","last_change":"2023-10-14T18:17:26.765123+0000","last_active":"2023-10-14T18:17:28.237611+0000","last_peered":"2023-10-14T18:17:28.237611+0000","last_clean":"2023-10-14T18:17:28.237611+0000","last_became_active":"2023-10-14T18:17:26.764801+0000","last_became_peered":"2023-10-14T18:17:26.764801+0000","last_unstale":"2023-10-14T18:17:28.237611+0000","last_undegraded":"2023-10-14T18:17:28.237611+0000","last_fullsized":"2023-10-14T18:17:28.237611+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:03.297149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847046+0000","last_change":"2023-10-14T18:17:27.254416+0000","last_active":"2023-10-14T18:17:28.847046+0000","last_peered":"2023-10-14T18:17:28.847046+0000","last_clean":"2023-10-14T18:17:28.847046+0000","last_became_active":"2023-10-14T18:17:27.254010+0000","last_became_peered":"2023-10-14T18:17:27.254010+0000","last_unstale":"2023-10-14T18:17:28.847046+0000","last_undegraded":"2023-10-14T18:17:28.847046+0000","last_fullsized":"2023-10-14T18:17:28.847046+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.845550+0000","last_change":"2023-10-14T18:17:26.754064+0000","last_active":"2023-10-14T18:17:28.845550+0000","last_peered":"2023-10-14T18:17:28.845550+0000","last_clean":"2023-10-14T18:17:28.845550+0000","last_became_active":"2023-10-14T18:17:26.752729+0000","last_became_peered":"2023-10-14T18:17:26.752729+0000","last_unstale":"2023-10-14T18:17:28.845550+0000","last_undegraded":"2023-10-14T18:17:28.845550+0000","last_fullsized":"2023-10-14T18:17:28.845550+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:09.744523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848578+0000","last_change":"2023-10-14T18:17:27.259767+0000","last_active":"2023-10-14T18:17:28.848578+0000","last_peered":"2023-10-14T18:17:28.848578+0000","last_clean":"2023-10-14T18:17:28.848578+0000","last_became_active":"2023-10-14T18:17:27.259468+0000","last_became_peered":"2023-10-14T18:17:27.259468+0000","last_unstale":"2023-10-14T18:17:28.848578+0000","last_undegraded":"2023-10-14T18:17:28.848578+0000","last_fullsized":"2023-10-14T18:17:28.848578+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:40.472642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847300+0000","last_change":"2023-10-14T18:17:26.767200+0000","last_active":"2023-10-14T18:17:28.847300+0000","last_peered":"2023-10-14T18:17:28.847300+0000","last_clean":"2023-10-14T18:17:28.847300+0000","last_became_active":"2023-10-14T18:17:26.766925+0000","last_became_peered":"2023-10-14T18:17:26.766925+0000","last_unstale":"2023-10-14T18:17:28.847300+0000","last_undegraded":"2023-10-14T18:17:28.847300+0000","last_fullsized":"2023-10-14T18:17:28.847300+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:28.445946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848909+0000","last_change":"2023-10-14T18:17:27.261099+0000","last_active":"2023-10-14T18:17:28.848909+0000","last_peered":"2023-10-14T18:17:28.848909+0000","last_clean":"2023-10-14T18:17:28.848909+0000","last_became_active":"2023-10-14T18:17:27.260783+0000","last_became_peered":"2023-10-14T18:17:27.260783+0000","last_unstale":"2023-10-14T18:17:28.848909+0000","last_undegraded":"2023-10-14T18:17:28.848909+0000","last_fullsized":"2023-10-14T18:17:28.848909+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.236799+0000","last_change":"2023-10-14T18:17:26.765437+0000","last_active":"2023-10-14T18:17:28.236799+0000","last_peered":"2023-10-14T18:17:28.236799+0000","last_clean":"2023-10-14T18:17:28.236799+0000","last_became_active":"2023-10-14T18:17:26.764932+0000","last_became_peered":"2023-10-14T18:17:26.764932+0000","last_unstale":"2023-10-14T18:17:28.236799+0000","last_undegraded":"2023-10-14T18:17:28.236799+0000","last_fullsized":"2023-10-14T18:17:28.236799+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:00.444875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238448+0000","last_change":"2023-10-14T18:17:27.259073+0000","last_active":"2023-10-14T18:17:28.238448+0000","last_peered":"2023-10-14T18:17:28.238448+0000","last_clean":"2023-10-14T18:17:28.238448+0000","last_became_active":"2023-10-14T18:17:27.258114+0000","last_became_peered":"2023-10-14T18:17:27.258114+0000","last_unstale":"2023-10-14T18:17:28.238448+0000","last_undegraded":"2023-10-14T18:17:28.238448+0000","last_fullsized":"2023-10-14T18:17:28.238448+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:14.438669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846023+0000","last_change":"2023-10-14T18:17:26.762092+0000","last_active":"2023-10-14T18:17:28.846023+0000","last_peered":"2023-10-14T18:17:28.846023+0000","last_clean":"2023-10-14T18:17:28.846023+0000","last_became_active":"2023-10-14T18:17:26.761696+0000","last_became_peered":"2023-10-14T18:17:26.761696+0000","last_unstale":"2023-10-14T18:17:28.846023+0000","last_undegraded":"2023-10-14T18:17:28.846023+0000","last_fullsized":"2023-10-14T18:17:28.846023+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:27.118966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846896+0000","last_change":"2023-10-14T18:17:27.248014+0000","last_active":"2023-10-14T18:17:28.846896+0000","last_peered":"2023-10-14T18:17:28.846896+0000","last_clean":"2023-10-14T18:17:28.846896+0000","last_became_active":"2023-10-14T18:17:27.247706+0000","last_became_peered":"2023-10-14T18:17:27.247706+0000","last_unstale":"2023-10-14T18:17:28.846896+0000","last_undegraded":"2023-10-14T18:17:28.846896+0000","last_fullsized":"2023-10-14T18:17:28.846896+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.847679+0000","last_change":"2023-10-14T18:17:26.755699+0000","last_active":"2023-10-14T18:17:28.847679+0000","last_peered":"2023-10-14T18:17:28.847679+0000","last_clean":"2023-10-14T18:17:28.847679+0000","last_became_active":"2023-10-14T18:17:26.755286+0000","last_became_peered":"2023-10-14T18:17:26.755286+0000","last_unstale":"2023-10-14T18:17:28.847679+0000","last_undegraded":"2023-10-14T18:17:28.847679+0000","last_fullsized":"2023-10-14T18:17:28.847679+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:07.796326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.846754+0000","last_change":"2023-10-14T18:17:27.258205+0000","last_active":"2023-10-14T18:17:28.846754+0000","last_peered":"2023-10-14T18:17:28.846754+0000","last_clean":"2023-10-14T18:17:28.846754+0000","last_became_active":"2023-10-14T18:17:27.257652+0000","last_became_peered":"2023-10-14T18:17:27.257652+0000","last_unstale":"2023-10-14T18:17:28.846754+0000","last_undegraded":"2023-10-14T18:17:28.846754+0000","last_fullsized":"2023-10-14T18:17:28.846754+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:00.190444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":12,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238071+0000","last_change":"2023-10-14T18:17:26.754848+0000","last_active":"2023-10-14T18:17:28.238071+0000","last_peered":"2023-10-14T18:17:28.238071+0000","last_clean":"2023-10-14T18:17:28.238071+0000","last_became_active":"2023-10-14T18:17:26.751997+0000","last_became_peered":"2023-10-14T18:17:26.751997+0000","last_unstale":"2023-10-14T18:17:28.238071+0000","last_undegraded":"2023-10-14T18:17:28.238071+0000","last_fullsized":"2023-10-14T18:17:28.238071+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:31.362757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":11,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.848653+0000","last_change":"2023-10-14T18:17:27.260210+0000","last_active":"2023-10-14T18:17:28.848653+0000","last_peered":"2023-10-14T18:17:28.848653+0000","last_clean":"2023-10-14T18:17:28.848653+0000","last_became_active":"2023-10-14T18:17:27.259297+0000","last_became_peered":"2023-10-14T18:17:27.259297+0000","last_unstale":"2023-10-14T18:17:28.848653+0000","last_undegraded":"2023-10-14T18:17:28.848653+0000","last_fullsized":"2023-10-14T18:17:28.848653+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.1b","version":"228'1","reported_seq":13,"reported_epoch":229,"state":"active+clean","last_fresh":"2023-10-14T18:17:28.238622+0000","last_change":"2023-10-14T18:17:26.757505+0000","last_active":"2023-10-14T18:17:28.238622+0000","last_peered":"2023-10-14T18:17:28.238622+0000","last_clean":"2023-10-14T18:17:28.238622+0000","last_became_active":"2023-10-14T18:17:26.757219+0000","last_became_peered":"2023-10-14T18:17:26.757219+0000","last_unstale":"2023-10-14T18:17:28.238622+0000","last_undegraded":"2023-10-14T18:17:28.238622+0000","last_fullsized":"2023-10-14T18:17:28.238622+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:19.831762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1210,"reported_epoch":228,"state":"active+clean","last_fresh":"2023-10-14T18:17:27.251502+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:17:27.251502+0000","last_peered":"2023-10-14T18:17:27.251502+0000","last_clean":"2023-10-14T18:17:27.251502+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:17:27.251502+0000","last_undegraded":"2023-10-14T18:17:27.251502+0000","last_fullsized":"2023-10-14T18:17:27.251502+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56784,"kb_used_data":3396,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315056,"statfs":{"total":96636764160,"available":96578617344,"internally_reserved":0,"allocated":3477504,"data_stored":2014128,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.216,"15min":1.2090000000000001},"min":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.573,"5min":1.7030000000000001,"15min":2.1669999999999998},"last":1.5209999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.232,"15min":1.228},"min":{"1min":0.78300000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6619999999999999,"5min":1.76,"15min":2.2149999999999999},"last":1.78}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.264,"15min":1.276},"min":{"1min":0.67500000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9059999999999999,"5min":2.214,"15min":2.214},"last":1.321},{"interface":"front","average":{"1min":1.341,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.871,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.1659999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.2230000000000001,"15min":1.2669999999999999},"min":{"1min":0.73799999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.546,"5min":1.6799999999999999,"15min":2.004},"last":1.254},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.325,"15min":1.3700000000000001},"min":{"1min":0.85199999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.8260000000000001,"15min":2.2010000000000001},"last":1.028}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3819999999999999,"15min":1.4339999999999999},"min":{"1min":1.0009999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.72,"5min":1.8500000000000001,"15min":2.4430000000000001},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5109999999999999,"15min":1.536},"min":{"1min":1.0429999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.875,"5min":1.9079999999999999,"15min":2.2810000000000001},"last":1.7}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.331},"min":{"1min":0.97699999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.8029999999999999,"5min":1.8200000000000001,"15min":2.367},"last":2.0600000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.8360000000000001,"5min":1.865,"15min":2.661},"last":1.427}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4390000000000001,"15min":1.462},"min":{"1min":1.048,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8740000000000001,"15min":2.516},"last":1.958},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4890000000000001,"15min":1.5229999999999999},"min":{"1min":1.1870000000000001,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.448},"last":2.2559999999999998}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5960000000000001,"15min":1.633},"min":{"1min":1.3169999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.4849999999999999},"last":2.4140000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.5349999999999999,"15min":1.5800000000000001},"min":{"1min":1.115,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":1.948,"5min":1.988,"15min":2.8159999999999998},"last":1.7490000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576004,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29828,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94342012,"statfs":{"total":96636764160,"available":96606220288,"internally_reserved":0,"allocated":2875392,"data_stored":1422730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.754,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6319999999999999,"5min":2.085,"15min":2.085},"last":1.0469999999999999},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.204,"15min":1.2110000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1910000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2230000000000001,"15min":1.2509999999999999},"min":{"1min":0.746,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.5309999999999999,"5min":1.762,"15min":1.776},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.256,"5min":1.3580000000000001,"15min":1.359},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.6459999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.3819999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.383,"15min":1.3779999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8500000000000001,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.403,"15min":1.4219999999999999},"min":{"1min":1.0600000000000001,"5min":0.99199999999999999,"15min":0.98099999999999998},"max":{"1min":1.6719999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.4670000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3740000000000001,"15min":1.407},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7130000000000001,"5min":2.0379999999999998,"15min":2.2519999999999998},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.90300000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.8220000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.375,"15min":1.381},"min":{"1min":0.85899999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.827,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.5089999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":2.0030000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3799999999999999,"15min":1.413},"min":{"1min":0.96199999999999997,"5min":0.624,"15min":0.624},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.4940000000000002},"last":1.77},{"interface":"front","average":{"1min":1.516,"5min":1.5469999999999999,"15min":1.579},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.8979999999999999,"5min":2.1600000000000001,"15min":2.5609999999999999},"last":1.6200000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5840000000000001,"15min":1.615},"min":{"1min":0.86799999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6389999999999998,"5min":2.903,"15min":3.423},"last":1.931},{"interface":"front","average":{"1min":1.637,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":1.1870000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.6719999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42316,"kb_used_data":2808,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329524,"statfs":{"total":96636764160,"available":96593432576,"internally_reserved":0,"allocated":2875392,"data_stored":1422730,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.179,"15min":1.2010000000000001},"min":{"1min":0.89700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.716,"5min":1.7569999999999999,"15min":2.0459999999999998},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.212,"15min":1.2470000000000001},"min":{"1min":0.878,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.722,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":1.022}]},{"osd":1,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.238,"15min":1.2769999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.274,"15min":1.3080000000000001},"min":{"1min":0.91500000000000004,"5min":0.73599999999999999,"15min":0.63400000000000001},"max":{"1min":1.722,"5min":1.7450000000000001,"15min":2.0419999999999998},"last":1.4810000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.258},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.649,"5min":1.764,"15min":3.7090000000000001},"last":0.79100000000000004},{"interface":"front","average":{"1min":1.393,"5min":1.3779999999999999,"15min":1.3959999999999999},"min":{"1min":1.0369999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.768,"5min":1.841,"15min":1.923},"last":1.0860000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4239999999999999,"15min":1.4670000000000001},"min":{"1min":1.071,"5min":0.94199999999999995,"15min":0.81899999999999995},"max":{"1min":1.794,"5min":1.867,"15min":2.4969999999999999},"last":1.734},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.421,"15min":1.4450000000000001},"min":{"1min":1.008,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":1.952,"5min":1.952,"15min":3.8330000000000002},"last":1.151}]},{"osd":4,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3700000000000001,"15min":1.363},"min":{"1min":0.98899999999999999,"5min":0.80100000000000005,"15min":0.78400000000000003},"max":{"1min":1.7889999999999999,"5min":1.954,"15min":2.98},"last":1.1970000000000001},{"interface":"front","average":{"1min":1.464,"5min":1.3939999999999999,"15min":1.3859999999999999},"min":{"1min":0.90600000000000003,"5min":0.61799999999999999,"15min":0.57499999999999996},"max":{"1min":2.016,"5min":2.016,"15min":3.1280000000000001},"last":1.857}]},{"osd":6,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3859999999999999,"15min":1.417},"min":{"1min":0.97399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.853,"5min":1.8799999999999999,"15min":4.0540000000000003},"last":1.335},{"interface":"front","average":{"1min":1.663,"5min":1.601,"15min":1.627},"min":{"1min":1.3859999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":3.6360000000000001},"last":1.6180000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.516,"15min":1.573},"min":{"1min":1.0409999999999999,"5min":0.68600000000000005,"15min":0.67700000000000005},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.2759999999999998},"last":1.9870000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5600000000000001,"15min":1.556},"min":{"1min":1.1100000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.0459999999999998,"5min":2.206,"15min":3.9390000000000001},"last":1.4430000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576005,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64532,"kb_used_data":2824,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307308,"statfs":{"total":96636764160,"available":96570683392,"internally_reserved":0,"allocated":2891776,"data_stored":1423864,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":1.0169999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":1.341,"5min":1.2729999999999999,"15min":1.288},"min":{"1min":0.80000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.698,"5min":1.887,"15min":2.3140000000000001},"last":0.99199999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.282,"15min":1.323},"min":{"1min":1.0009999999999999,"5min":0.76800000000000002,"15min":0.59399999999999997},"max":{"1min":1.7410000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.107},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2490000000000001,"15min":1.268},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.667,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":1.4379999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.29,"15min":1.2849999999999999},"min":{"1min":0.83699999999999997,"5min":0.81399999999999995,"15min":0.78000000000000003},"max":{"1min":1.659,"5min":1.869,"15min":3.524},"last":1.474},{"interface":"front","average":{"1min":1.379,"5min":1.427,"15min":1.4450000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":0.83499999999999996}]},{"osd":3,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.278,"15min":1.3460000000000001},"min":{"1min":0.90400000000000003,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.627,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.278},{"interface":"front","average":{"1min":1.425,"5min":1.419,"15min":1.4390000000000001},"min":{"1min":1.0509999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7629999999999999,"5min":2.0569999999999999,"15min":4.1600000000000001},"last":1.5529999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4219999999999999,"15min":1.3959999999999999},"min":{"1min":0.96599999999999997,"5min":0.90200000000000002,"15min":0.69299999999999995},"max":{"1min":1.7949999999999999,"5min":1.9319999999999999,"15min":2.02},"last":1.194},{"interface":"front","average":{"1min":1.407,"5min":1.369,"15min":1.3759999999999999},"min":{"1min":0.98299999999999998,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":1.7589999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.524}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.45,"15min":1.4339999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.81799999999999995},"max":{"1min":1.887,"5min":2.077,"15min":3.3610000000000002},"last":1.609},{"interface":"front","average":{"1min":1.548,"5min":1.575,"15min":1.5700000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.82699999999999996},"max":{"1min":1.823,"5min":2.1080000000000001,"15min":2.3929999999999998},"last":1.3939999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.506,"15min":1.516},"min":{"1min":1.234,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.798,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.581},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.601,"15min":1.595},"min":{"1min":1.246,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":1.9790000000000001,"5min":2.1739999999999999,"15min":3.8610000000000002},"last":1.6779999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576005,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52044,"kb_used_data":2816,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319796,"statfs":{"total":96636764160,"available":96583471104,"internally_reserved":0,"allocated":2883584,"data_stored":1422330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96899999999999997,"15min":0.98399999999999999},"min":{"1min":0.502,"5min":0.377,"15min":0.377},"max":{"1min":1.46,"5min":1.4870000000000001,"15min":1.855},"last":0.92700000000000005},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.179,"15min":1.2050000000000001},"min":{"1min":0.61499999999999999,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4890000000000001,"5min":1.7230000000000001,"15min":1.9970000000000001},"last":1.165}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0089999999999999,"15min":1.0169999999999999},"min":{"1min":0.57499999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.417,"5min":1.8939999999999999,"15min":1.919},"last":0.89700000000000002},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.177,"15min":1.212},"min":{"1min":0.78100000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.228}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.446},"min":{"1min":0.71899999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.9730000000000001,"5min":2.032,"15min":2.032},"last":0.73999999999999999},{"interface":"front","average":{"1min":1.284,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.72499999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.7430000000000001,"5min":1.8,"15min":2.3540000000000001},"last":0.84099999999999997}]},{"osd":4,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7350000000000001,"5min":2.153,"15min":2.153},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.484},"min":{"1min":0.85299999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":1.9279999999999999,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.478}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.96499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.8009999999999999,"5min":2.1429999999999998,"15min":2.476},"last":1.325},{"interface":"front","average":{"1min":1.46,"5min":1.518,"15min":1.5649999999999999},"min":{"1min":1.0249999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0739999999999998,"5min":2.246,"15min":2.891},"last":1.1919999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6180000000000001,"15min":1.6319999999999999},"min":{"1min":1.0580000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0390000000000001,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":1.585},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6859999999999999,"15min":1.6870000000000001},"min":{"1min":1.0760000000000001,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.1320000000000001,"5min":2.2949999999999999,"15min":2.585},"last":1.679}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.579,"15min":1.607},"min":{"1min":1.2050000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.859,"5min":2.2370000000000001,"15min":2.419},"last":1.3939999999999999},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.704,"15min":1.696},"min":{"1min":1.0469999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1890000000000001,"5min":2.548,"15min":2.548},"last":1.4299999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576005,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52040,"kb_used_data":2804,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319800,"statfs":{"total":96636764160,"available":96583475200,"internally_reserved":0,"allocated":2871296,"data_stored":1422148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":0.97799999999999998,"15min":0.94899999999999995},"min":{"1min":0.65000000000000002,"5min":0.499,"15min":0.499},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.0349999999999999,"15min":1.0960000000000001},"min":{"1min":0.61899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.341,"5min":1.653,"15min":1.8660000000000001},"last":0.86399999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98699999999999999,"15min":1.0169999999999999},"min":{"1min":0.61899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.544,"5min":1.603,"15min":1.881},"last":1.1439999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.623,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.222}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2509999999999999,"15min":1.2909999999999999},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.56599999999999995},"max":{"1min":1.621,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.2829999999999999},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.407,"15min":1.3999999999999999},"min":{"1min":0.76400000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0089999999999999,"5min":2.1579999999999999,"15min":2.165},"last":1.3320000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.492,"15min":1.478},"min":{"1min":1.077,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":1.5029999999999999},{"interface":"front","average":{"1min":1.327,"5min":1.3400000000000001,"15min":1.3680000000000001},"min":{"1min":0.92300000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.728,"5min":1.8759999999999999,"15min":2.024},"last":1.944}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.444},"min":{"1min":1.012,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9299999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":1.8999999999999999},{"interface":"front","average":{"1min":1.613,"5min":1.625,"15min":1.5980000000000001},"min":{"1min":1.3460000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":2.0049999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.528,"15min":1.53},"min":{"1min":0.99099999999999999,"5min":0.747,"15min":0.747},"max":{"1min":1.8799999999999999,"5min":2.0529999999999999,"15min":2.1150000000000002},"last":1.794},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.587,"15min":1.617},"min":{"1min":0.85599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":2.1400000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.611,"15min":1.5940000000000001},"min":{"1min":1.208,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":1.7010000000000001},{"interface":"front","average":{"1min":1.631,"5min":1.6659999999999999,"15min":1.6859999999999999},"min":{"1min":1.155,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":2.0600000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576005,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52048,"kb_used_data":2812,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319792,"statfs":{"total":96636764160,"available":96583467008,"internally_reserved":0,"allocated":2879488,"data_stored":1422204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.024,"5min":0.96099999999999997,"15min":0.97199999999999998},"min":{"1min":0.755,"5min":0.38,"15min":0.38},"max":{"1min":1.464,"5min":1.647,"15min":2.4929999999999999},"last":1.2290000000000001},{"interface":"front","average":{"1min":1.014,"5min":1.0820000000000001,"15min":1.1299999999999999},"min":{"1min":0.69399999999999995,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":1.3939999999999999,"5min":1.7070000000000001,"15min":2.5590000000000002},"last":2.847}]},{"osd":2,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.0960000000000001,"5min":1.05,"15min":1.0680000000000001},"min":{"1min":0.70499999999999996,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.593,"5min":1.6930000000000001,"15min":2.7360000000000002},"last":2.1200000000000001},{"interface":"front","average":{"1min":1.323,"5min":1.23,"15min":1.2669999999999999},"min":{"1min":0.96299999999999997,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.732,"5min":1.996,"15min":2.6859999999999999},"last":1.153}]},{"osd":3,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.375,"5min":1.3740000000000001,"15min":1.3939999999999999},"min":{"1min":0.92100000000000004,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7290000000000001,"5min":1.952,"15min":2.5910000000000002},"last":2.9940000000000002},{"interface":"front","average":{"1min":1.3320000000000001,"5min":1.2689999999999999,"15min":1.3120000000000001},"min":{"1min":0.89000000000000001,"5min":0.53500000000000003,"15min":0.53500000000000003},"max":{"1min":1.774,"5min":1.7849999999999999,"15min":2.823},"last":4.8630000000000004}]},{"osd":4,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.407,"5min":1.423,"15min":1.4690000000000001},"min":{"1min":1.0920000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.849,"5min":1.857,"15min":2.4430000000000001},"last":4.8090000000000002},{"interface":"front","average":{"1min":1.4399999999999999,"5min":1.409,"15min":1.444},"min":{"1min":1.0309999999999999,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.859,"5min":2.0979999999999999,"15min":2.899},"last":2.218}]},{"osd":5,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.4750000000000001,"5min":1.464,"15min":1.476},"min":{"1min":1.0449999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":1.8979999999999999,"5min":2.0600000000000001,"15min":3.004},"last":2.173},{"interface":"front","average":{"1min":1.607,"5min":1.5760000000000001,"15min":1.575},"min":{"1min":1.1890000000000001,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9890000000000001,"5min":2.2280000000000002,"15min":2.5129999999999999},"last":1.1950000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.5589999999999999,"5min":1.5840000000000001,"15min":1.6180000000000001},"min":{"1min":1.2350000000000001,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":1.8600000000000001,"5min":2.2770000000000001,"15min":2.931},"last":2.8999999999999999},{"interface":"front","average":{"1min":1.623,"5min":1.595,"15min":1.611},"min":{"1min":1.2669999999999999,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.0070000000000001,"5min":2.2610000000000001,"15min":3.1190000000000002},"last":10.173999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:16:38 2023","interfaces":[{"interface":"back","average":{"1min":1.734,"5min":1.6990000000000001,"15min":1.704},"min":{"1min":1.4510000000000001,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.0859999999999999,"5min":2.3420000000000001,"15min":2.5009999999999999},"last":3.363},{"interface":"front","average":{"1min":1.657,"5min":1.677,"15min":1.7150000000000001},"min":{"1min":1.3080000000000001,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":1.9570000000000001,"5min":2.1800000000000002,"15min":2.6739999999999999},"last":7.0439999999999996}]}]},{"osd":0,"up_from":13,"seq":55834576004,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48464,"kb_used_data":3388,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323376,"statfs":{"total":96636764160,"available":96587137024,"internally_reserved":0,"allocated":3469312,"data_stored":2012606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.69299999999999995},{"interface":"front","average":{"1min":1.095,"5min":1.129,"15min":1.1499999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.0780000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":0.89300000000000002},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.25},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.845,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":1.4770000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.7150000000000001,"5min":1.804,"15min":2.0720000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.452,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":1.883,"15min":2.194},"last":1.4099999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.321,"15min":1.331},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7629999999999999,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":0.94599999999999995},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.5129999999999999,"15min":1.512},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.121,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.1319999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.74,"5min":1.8220000000000001,"15min":2.1339999999999999},"last":1.2030000000000001},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5409999999999999,"15min":1.581},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.2569999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.655,"15min":1.661},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.548},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.1080000000000001,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.3759999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.631,"15min":1.661},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0089999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.319},{"interface":"front","average":{"1min":1.639,"5min":1.613,"15min":1.641},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.508}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:17:32.779 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:17:32.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T18:17:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.056+0000 7ff26d9a9640 1 Processor -- start 2023-10-14T18:17:33.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.056+0000 7ff26d9a9640 1 -- start start 2023-10-14T18:17:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff26d9a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff268070fd0 0x7ff2680cd070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff26d9a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2680adfc0 0x7ff2680b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff26d9a9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2680cc600 con 0x7ff2680adbe0 2023-10-14T18:17:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff26d9a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff268071470 con 0x7ff268070fd0 2023-10-14T18:17:33.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff26d9a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680cc870 con 0x7ff2680adfc0 2023-10-14T18:17:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff266ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff2680adbe0 0x7ff26806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40050/0 (socket says 172.21.15.29:40050) 2023-10-14T18:17:33.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff266ffd640 1 -- 172.21.15.29:0/2809502119 learned_addr learned my addr 172.21.15.29:0/2809502119 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.058+0000 7ff25ffff640 1 --2- 172.21.15.29:0/2809502119 >> v2:172.21.15.29:3300/0 conn(0x7ff268070fd0 0x7ff2680cd070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff2677fe640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2680adfc0 0x7ff2680b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff2667fc640 1 -- 172.21.15.29:0/2809502119 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4041649746 0 0) 0x7ff2680cc600 con 0x7ff2680adbe0 2023-10-14T18:17:33.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff2667fc640 1 -- 172.21.15.29:0/2809502119 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff254003520 con 0x7ff2680adbe0 2023-10-14T18:17:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff25ffff640 1 -- 172.21.15.29:0/2809502119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2680adfc0 msgr2=0x7ff2680b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff25ffff640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2680adfc0 0x7ff2680b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff25ffff640 1 -- 172.21.15.29:0/2809502119 >> v1:172.21.15.29:6789/0 conn(0x7ff2680adbe0 legacy=0x7ff26806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.059+0000 7ff25ffff640 1 -- 172.21.15.29:0/2809502119 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2680ccb50 con 0x7ff268070fd0 2023-10-14T18:17:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.060+0000 7ff25ffff640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff268070fd0 0x7ff2680cd070 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7ff2500097a0 tx=0x7ff25002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18e6ce094e50d4d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff2667fc640 1 -- 172.21.15.29:0/2809502119 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25002d040 con 0x7ff268070fd0 2023-10-14T18:17:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff2667fc640 1 -- 172.21.15.29:0/2809502119 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff250031810 con 0x7ff268070fd0 2023-10-14T18:17:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff2667fc640 1 -- 172.21.15.29:0/2809502119 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff250031ba0 con 0x7ff268070fd0 2023-10-14T18:17:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 -- 172.21.15.29:0/2809502119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff268070fd0 msgr2=0x7ff2680cd070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff268070fd0 0x7ff2680cd070 secure :-1 s=READY pgs=2586 cs=0 l=1 rev1=1 crypto rx=0x7ff2500097a0 tx=0x7ff25002b8b0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 -- 172.21.15.29:0/2809502119 shutdown_connections 2023-10-14T18:17:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff2680adfc0 0x7ff2680b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/2809502119 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff268070fd0 0x7ff2680cd070 unknown :-1 s=CLOSED pgs=2586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.061+0000 7ff26d9a9640 1 -- 172.21.15.29:0/2809502119 >> 172.21.15.29:0/2809502119 conn(0x7ff2680cb410 msgr2=0x7ff2680cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.062+0000 7ff26d9a9640 1 -- 172.21.15.29:0/2809502119 shutdown_connections 2023-10-14T18:17:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.062+0000 7ff26d9a9640 1 -- 172.21.15.29:0/2809502119 wait complete. 2023-10-14T18:17:33.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.063+0000 7ff26d9a9640 1 Processor -- start 2023-10-14T18:17:33.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.063+0000 7ff26d9a9640 1 -- start start 2023-10-14T18:17:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 0x7ff2680b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2680adfc0 0x7ff268156300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c3320 con 0x7ff2680adbe0 2023-10-14T18:17:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2680c3b50 con 0x7ff2680adfc0 2023-10-14T18:17:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.064+0000 7ff26d9a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff268156840 con 0x7ff268070fd0 2023-10-14T18:17:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55566/0 (socket says 172.21.15.29:55566) 2023-10-14T18:17:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 -- 172.21.15.29:0/1219357580 learned_addr learned my addr 172.21.15.29:0/1219357580 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 -- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 msgr2=0x7ff2680b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff266ffd640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 0x7ff2680b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff2677fe640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2680adfc0 0x7ff268156300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 0x7ff2680b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 -- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2680adfc0 msgr2=0x7ff268156300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2680adfc0 0x7ff268156300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff25ffff640 1 -- 172.21.15.29:0/1219357580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff250009400 con 0x7ff2680adbe0 2023-10-14T18:17:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.065+0000 7ff266ffd640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 0x7ff2680b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:33.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff25ffff640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7ff250000c00 tx=0x7ff2500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25003ad60 con 0x7ff2680adbe0 2023-10-14T18:17:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2500443f0 con 0x7ff2680adbe0 2023-10-14T18:17:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff2500445a0 con 0x7ff2680adbe0 2023-10-14T18:17:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2680c9670 con 0x7ff2680adbe0 2023-10-14T18:17:33.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.066+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2680c37e0 con 0x7ff2680adbe0 2023-10-14T18:17:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.067+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2680c3cc0 con 0x7ff2680adbe0 2023-10-14T18:17:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.068+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff25004d450 con 0x7ff2680adbe0 2023-10-14T18:17:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.069+0000 7ff264ff9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 0x7ff24806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.069+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff2500aba30 con 0x7ff2680adbe0 2023-10-14T18:17:33.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.070+0000 7ff266ffd640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 0x7ff24806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.070+0000 7ff266ffd640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 0x7ff24806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff258007fd0 tx=0x7ff258009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.074+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff250076870 con 0x7ff2680adbe0 2023-10-14T18:17:33.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.342+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff26804e160 con 0x7ff2680adbe0 2023-10-14T18:17:33.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.346+0000 7ff264ff9640 1 -- 172.21.15.29:0/1219357580 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v51) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff2500350e0 con 0x7ff2680adbe0 2023-10-14T18:17:33.347 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T18:17:33.348 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T18:17:33.348 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T18:17:33.348 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T18:17:33.348 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T18:17:33.349 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T18:17:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 msgr2=0x7ff24806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 0x7ff24806d450 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff258007fd0 tx=0x7ff258009040 comp rx=0 tx=0).stop 2023-10-14T18:17:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 msgr2=0x7ff268155dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 secure :-1 s=READY pgs=2587 cs=0 l=1 rev1=1 crypto rx=0x7ff250000c00 tx=0x7ff2500046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 shutdown_connections 2023-10-14T18:17:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff24806af90 0x7ff24806d450 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff2680adfc0 0x7ff268156300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff2680adbe0 0x7ff268155dc0 unknown :-1 s=CLOSED pgs=2587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 --2- 172.21.15.29:0/1219357580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff268070fd0 0x7ff2680b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 >> 172.21.15.29:0/1219357580 conn(0x7ff2680cb410 msgr2=0x7ff2680b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 shutdown_connections 2023-10-14T18:17:33.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.349+0000 7ff26d9a9640 1 -- 172.21.15.29:0/1219357580 wait complete. 2023-10-14T18:17:33.371 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.601+0000 7f2d0ee68640 1 Processor -- start 2023-10-14T18:17:33.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.601+0000 7f2d0ee68640 1 -- start start 2023-10-14T18:17:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.603+0000 7f2d0ee68640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2d080b0b20 0x7f2d08164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.603+0000 7f2d0ee68640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d08071450 con 0x7f2d080adb50 2023-10-14T18:17:33.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.603+0000 7f2d0ee68640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d080cc600 con 0x7f2d08070fd0 2023-10-14T18:17:33.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.603+0000 7f2d0ee68640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d080cc8e0 con 0x7f2d080b0b20 2023-10-14T18:17:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.604+0000 7f2d0cc04640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f2d080adb50 0x7f2d0806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40066/0 (socket says 172.21.15.29:40066) 2023-10-14T18:17:33.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.604+0000 7f2d0cc04640 1 -- 172.21.15.29:0/511543405 learned_addr learned my addr 172.21.15.29:0/511543405 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.604+0000 7f2d0d405640 1 --2- 172.21.15.29:0/511543405 >> v2:172.21.15.186:3300/0 conn(0x7f2d080b0b20 0x7f2d08164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.604+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2315006488 0 0) 0x7f2d08071450 con 0x7f2d080adb50 2023-10-14T18:17:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.604+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cf8003520 con 0x7f2d080adb50 2023-10-14T18:17:33.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3132177713 0 0) 0x7f2cf8003520 con 0x7f2d080adb50 2023-10-14T18:17:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 >> v2:172.21.15.186:3300/0 conn(0x7f2d080b0b20 msgr2=0x7f2d08164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 --2- 172.21.15.29:0/511543405 >> v2:172.21.15.186:3300/0 conn(0x7f2d080b0b20 0x7f2d08164f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 >> v1:172.21.15.186:6789/0 conn(0x7f2d08070fd0 legacy=0x7f2d080b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080ccbc0 con 0x7f2d080adb50 2023-10-14T18:17:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f2cfc001670 con 0x7f2d080adb50 2023-10-14T18:17:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 >> v1:172.21.15.29:6789/0 conn(0x7f2d080adb50 legacy=0x7f2d0806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 0x7f2cf8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf8004370 0x7f2cf8004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8005040 0x7f2cf800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d080cc8e0 con 0x7f2cf8003970 2023-10-14T18:17:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf800bfa0 con 0x7f2cf8004370 2023-10-14T18:17:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cf800c160 con 0x7f2cf8005040 2023-10-14T18:17:33.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.605+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 0x7f2cf8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d07fff640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf8004370 0x7f2cf8004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0cc04640 1 -- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8005040 msgr2=0x7f2cf800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8005040 0x7f2cf800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0cc04640 1 -- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf8004370 msgr2=0x7f2cf8004840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf8004370 0x7f2cf8004840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0cc04640 1 -- 172.21.15.29:0/511543405 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf8003520 con 0x7f2cf8003970 2023-10-14T18:17:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d07fff640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2cf8004370 0x7f2cf8004840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.606+0000 7f2d0d405640 1 -- 172.21.15.29:0/511543405 reap_dead start 2023-10-14T18:17:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 0x7f2cf8003d50 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c190 tx=0x7f2cf80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0da46b7914bbe43 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc005ad0 con 0x7f2cf8003970 2023-10-14T18:17:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cfc005c50 con 0x7f2cf8003970 2023-10-14T18:17:33.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d077fe640 1 -- 172.21.15.29:0/511543405 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc02c790 con 0x7f2cf8003970 2023-10-14T18:17:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d0ee68640 1 -- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 msgr2=0x7f2cf8003d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.607+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 0x7f2cf8003d50 secure :-1 s=READY pgs=2589 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c190 tx=0x7f2cf80057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.608+0000 7f2d0ee68640 1 -- 172.21.15.29:0/511543405 shutdown_connections 2023-10-14T18:17:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.608+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/511543405 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2cf8003970 0x7f2cf8003d50 unknown :-1 s=CLOSED pgs=2589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.608+0000 7f2d0ee68640 1 -- 172.21.15.29:0/511543405 >> 172.21.15.29:0/511543405 conn(0x7f2d080cb410 msgr2=0x7f2d080cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.608+0000 7f2d0ee68640 1 -- 172.21.15.29:0/511543405 shutdown_connections 2023-10-14T18:17:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.608+0000 7f2d0ee68640 1 -- 172.21.15.29:0/511543405 wait complete. 2023-10-14T18:17:33.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.610+0000 7f2d0ee68640 1 Processor -- start 2023-10-14T18:17:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.610+0000 7f2d0ee68640 1 -- start start 2023-10-14T18:17:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8003970 0x7f2d080c4170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d08158ba0 0x7f2d081f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08159380 con 0x7f2d081f6db0 2023-10-14T18:17:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08161b00 con 0x7f2d08158ba0 2023-10-14T18:17:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0ee68640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08161de0 con 0x7f2cf8003970 2023-10-14T18:17:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0d405640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0d405640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55580/0 (socket says 172.21.15.29:55580) 2023-10-14T18:17:33.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.611+0000 7f2d0d405640 1 -- 172.21.15.29:0/3856271080 learned_addr learned my addr 172.21.15.29:0/3856271080 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8003970 0x7f2d080c4170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d07fff640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d08158ba0 0x7f2d081f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 -- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8003970 msgr2=0x7f2d080c4170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8003970 0x7f2d080c4170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 -- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d08158ba0 msgr2=0x7f2d081f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d08158ba0 0x7f2d081f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 -- 172.21.15.29:0/3856271080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d081620c0 con 0x7f2d081f6db0 2023-10-14T18:17:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.612+0000 7f2d0d405640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f2d0806d870 tx=0x7f2d0000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.613+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d00010040 con 0x7f2d081f6db0 2023-10-14T18:17:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.613+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d0000bae0 con 0x7f2d081f6db0 2023-10-14T18:17:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.613+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d0000be70 con 0x7f2d081f6db0 2023-10-14T18:17:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.613+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081623a0 con 0x7f2d081f6db0 2023-10-14T18:17:33.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.613+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0806c760 con 0x7f2d081f6db0 2023-10-14T18:17:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.614+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0804e160 con 0x7f2d081f6db0 2023-10-14T18:17:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.615+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2d0001e410 con 0x7f2d081f6db0 2023-10-14T18:17:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.616+0000 7f2d057fa640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 0x7f2cec06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.616+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f2d00089be0 con 0x7f2d081f6db0 2023-10-14T18:17:33.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.616+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 0x7f2cec06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:33.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.617+0000 7f2d0cc04640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 0x7f2cec06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc034000 tx=0x7f2cfc009030 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:33.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.623+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2d00054950 con 0x7f2d081f6db0 2023-10-14T18:17:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.863+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d080b4910 con 0x7f2d081f6db0 2023-10-14T18:17:33.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.864+0000 7f2d057fa640 1 -- 172.21.15.29:0/3856271080 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f2d00057f70 con 0x7f2d081f6db0 2023-10-14T18:17:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:33.866 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":107,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11256,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":101}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13}]} 2023-10-14T18:17:33.866 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 107 2023-10-14T18:17:33.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 msgr2=0x7f2cec06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 0x7f2cec06d480 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc034000 tx=0x7f2cfc009030 comp rx=0 tx=0).stop 2023-10-14T18:17:33.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 msgr2=0x7f2d0806c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:33.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 secure :-1 s=READY pgs=2590 cs=0 l=1 rev1=1 crypto rx=0x7f2d0806d870 tx=0x7f2d0000b480 comp rx=0 tx=0).stop 2023-10-14T18:17:33.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 shutdown_connections 2023-10-14T18:17:33.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2cec06afc0 0x7f2cec06d480 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2d081f6db0 0x7f2d0806c220 unknown :-1 s=CLOSED pgs=2590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2d08158ba0 0x7f2d081f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 --2- 172.21.15.29:0/3856271080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2cf8003970 0x7f2d080c4170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.868+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 >> 172.21.15.29:0/3856271080 conn(0x7f2d080cb410 msgr2=0x7f2d0815f1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.869+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 shutdown_connections 2023-10-14T18:17:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:33.869+0000 7f2d0ee68640 1 -- 172.21.15.29:0/3856271080 wait complete. 2023-10-14T18:17:34.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 107, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:17:27.215719+0000', 'modified': '2023-10-14T18:17:30.282217+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11268': {'gid': 11268, 'name': 'a', 'rank': 0, 'incarnation': 103, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.29:6837/1082656487', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6836', 'nonce': 1082656487}, {'type': 'v1', 'addr': '172.21.15.29:6837', 'nonce': 1082656487}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [27], 'metadata_pool': 26, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:17:34.291 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:17:34.292 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2023-10-14T18:17:34.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.531+0000 7f8b63fff640 1 Processor -- start 2023-10-14T18:17:34.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.531+0000 7f8b63fff640 1 -- start start 2023-10-14T18:17:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.533+0000 7f8b63fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8b6406fb50 0x7f8b640cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b63fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8b6406c9e0 0x7f8b640ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b63fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b640ca560 con 0x7f8b640c9060 2023-10-14T18:17:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b63fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b640c9950 con 0x7f8b6406fb50 2023-10-14T18:17:34.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b63fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b640c9c30 con 0x7f8b6406c9e0 2023-10-14T18:17:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b62ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8b640c9060 0x7f8b6406f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40076/0 (socket says 172.21.15.29:40076) 2023-10-14T18:17:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b62ffd640 1 -- 172.21.15.29:0/2937516484 learned_addr learned my addr 172.21.15.29:0/2937516484 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:34.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2937516484 >> v2:172.21.15.29:3300/0 conn(0x7f8b6406fb50 0x7f8b640cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b627fc640 1 -- 172.21.15.29:0/2937516484 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526774972 0 0) 0x7f8b640ca560 con 0x7f8b640c9060 2023-10-14T18:17:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b627fc640 1 -- 172.21.15.29:0/2937516484 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b50003520 con 0x7f8b640c9060 2023-10-14T18:17:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.534+0000 7f8b637fe640 1 --2- 172.21.15.29:0/2937516484 >> v2:172.21.15.186:3300/0 conn(0x7f8b6406c9e0 0x7f8b640ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.535+0000 7f8b5bfff640 1 -- 172.21.15.29:0/2937516484 >> v2:172.21.15.186:3300/0 conn(0x7f8b6406c9e0 msgr2=0x7f8b640ccde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.535+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2937516484 >> v2:172.21.15.186:3300/0 conn(0x7f8b6406c9e0 0x7f8b640ccde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.535+0000 7f8b5bfff640 1 -- 172.21.15.29:0/2937516484 >> v1:172.21.15.29:6789/0 conn(0x7f8b640c9060 legacy=0x7f8b6406f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.535+0000 7f8b5bfff640 1 -- 172.21.15.29:0/2937516484 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b640c9f10 con 0x7f8b6406fb50 2023-10-14T18:17:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.536+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2937516484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b640cc8a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f8b640ce550 tx=0x7f8b4c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9416612a42343fe server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.536+0000 7f8b627fc640 1 -- 172.21.15.29:0/2937516484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c031830 con 0x7f8b6406fb50 2023-10-14T18:17:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.536+0000 7f8b627fc640 1 -- 172.21.15.29:0/2937516484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c031990 con 0x7f8b6406fb50 2023-10-14T18:17:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.536+0000 7f8b627fc640 1 -- 172.21.15.29:0/2937516484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c031d20 con 0x7f8b6406fb50 2023-10-14T18:17:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 -- 172.21.15.29:0/2937516484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 msgr2=0x7f8b640cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 --2- 172.21.15.29:0/2937516484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b640cc8a0 secure :-1 s=READY pgs=2592 cs=0 l=1 rev1=1 crypto rx=0x7f8b640ce550 tx=0x7f8b4c02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 -- 172.21.15.29:0/2937516484 shutdown_connections 2023-10-14T18:17:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 --2- 172.21.15.29:0/2937516484 >> v2:172.21.15.186:3300/0 conn(0x7f8b6406c9e0 0x7f8b640ccde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 --2- 172.21.15.29:0/2937516484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b640cc8a0 unknown :-1 s=CLOSED pgs=2592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 -- 172.21.15.29:0/2937516484 >> 172.21.15.29:0/2937516484 conn(0x7f8b640ce6a0 msgr2=0x7f8b640ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 -- 172.21.15.29:0/2937516484 shutdown_connections 2023-10-14T18:17:34.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.537+0000 7f8b63fff640 1 -- 172.21.15.29:0/2937516484 wait complete. 2023-10-14T18:17:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.539+0000 7f8b63fff640 1 Processor -- start 2023-10-14T18:17:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.539+0000 7f8b63fff640 1 -- start start 2023-10-14T18:17:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b6406c9e0 0x7f8b641f6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b640c9060 0x7f8b640c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c39a0 con 0x7f8b6406fb50 2023-10-14T18:17:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c60e0 con 0x7f8b6406c9e0 2023-10-14T18:17:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b63fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b640c63c0 con 0x7f8b640c9060 2023-10-14T18:17:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b5bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55600/0 (socket says 172.21.15.29:55600) 2023-10-14T18:17:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.540+0000 7f8b5bfff640 1 -- 172.21.15.29:0/3940268548 learned_addr learned my addr 172.21.15.29:0/3940268548 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b637fe640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b640c9060 0x7f8b640c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b6406c9e0 0x7f8b641f6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 -- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b640c9060 msgr2=0x7f8b640c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b640c9060 0x7f8b640c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 -- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b6406c9e0 msgr2=0x7f8b641f6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b6406c9e0 0x7f8b641f6f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 -- 172.21.15.29:0/3940268548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4c009400 con 0x7f8b6406fb50 2023-10-14T18:17:34.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009890 tx=0x7f8b4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5b7fe640 1 -- 172.21.15.29:0/3940268548 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c03a6f0 con 0x7f8b6406fb50 2023-10-14T18:17:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5b7fe640 1 -- 172.21.15.29:0/3940268548 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b4c03a870 con 0x7f8b6406fb50 2023-10-14T18:17:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b640c66a0 con 0x7f8b6406fb50 2023-10-14T18:17:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.541+0000 7f8b5b7fe640 1 -- 172.21.15.29:0/3940268548 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b4c03ac20 con 0x7f8b6406fb50 2023-10-14T18:17:34.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.542+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b641fa170 con 0x7f8b6406fb50 2023-10-14T18:17:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.544+0000 7f8b5b7fe640 1 -- 172.21.15.29:0/3940268548 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8b4c043410 con 0x7f8b6406fb50 2023-10-14T18:17:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.545+0000 7f8b5b7fe640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 0x7f8b4406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.545+0000 7f8b5b7fe640 1 -- 172.21.15.29:0/3940268548 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b4c0ab650 con 0x7f8b6406fb50 2023-10-14T18:17:34.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.545+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 0x7f8b4406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.546+0000 7f8b6c25c740 1 Processor -- start 2023-10-14T18:17:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.546+0000 7f8b6c25c740 1 -- start start 2023-10-14T18:17:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.546+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 0x7f8b4406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b540011e0 tx=0x7f8b54009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd170 0x55eef91d0070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd550 0x55eef91fff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef90f07a0 con 0x55eef91d05b0 2023-10-14T18:17:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef9142740 con 0x55eef91fd550 2023-10-14T18:17:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eef913fe30 con 0x55eef91fd170 2023-10-14T18:17:34.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55612/0 (socket says 172.21.15.29:55612) 2023-10-14T18:17:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b637fe640 1 -- 172.21.15.29:0/716681080 learned_addr learned my addr 172.21.15.29:0/716681080 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd170 0x55eef91d0070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.547+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd550 0x55eef91fff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 -- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd170 msgr2=0x55eef91d0070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd170 0x55eef91d0070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 -- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd550 msgr2=0x55eef91fff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd550 0x55eef91fff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 -- 172.21.15.29:0/716681080 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eef90ff910 con 0x55eef91d05b0 2023-10-14T18:17:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b637fe640 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007410 tx=0x7f8b5c0078d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=768e131478d6c347 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b58ff9640 1 -- 172.21.15.29:0/716681080 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0147e0 con 0x55eef91d05b0 2023-10-14T18:17:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b58ff9640 1 -- 172.21.15.29:0/716681080 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c00f050 con 0x55eef91d05b0 2023-10-14T18:17:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.548+0000 7f8b58ff9640 1 -- 172.21.15.29:0/716681080 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c014b40 con 0x55eef91d05b0 2023-10-14T18:17:34.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 -- 172.21.15.29:0/716681080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 msgr2=0x55eef9200440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 secure :-1 s=READY pgs=2594 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c007410 tx=0x7f8b5c0078d0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 -- 172.21.15.29:0/716681080 shutdown_connections 2023-10-14T18:17:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91d05b0 0x55eef9200440 unknown :-1 s=CLOSED pgs=2594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd550 0x55eef91fff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/716681080 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd170 0x55eef91d0070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 -- 172.21.15.29:0/716681080 >> 172.21.15.29:0/716681080 conn(0x55eef91d2380 msgr2=0x55eef921af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 -- 172.21.15.29:0/716681080 shutdown_connections 2023-10-14T18:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.549+0000 7f8b6c25c740 1 -- 172.21.15.29:0/716681080 wait complete. 2023-10-14T18:17:34.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.550+0000 7f8b6c25c740 1 -- start start 2023-10-14T18:17:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.550+0000 7f8b6c25c740 1 Processor -- start 2023-10-14T18:17:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91ea560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91fedd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef913b720 con 0x55eef91fdf50 2023-10-14T18:17:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef90d9970 con 0x55eef91fd170 2023-10-14T18:17:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eef90d9610 con 0x55eef91fd550 2023-10-14T18:17:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b6c25c740 10 client.0 _ll_register_callbacks cb 0x55eef90fb9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:17:34.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.551+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55618/0 (socket says 172.21.15.29:55618) 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 -- 172.21.15.29:0/1163109785 learned_addr learned my addr 172.21.15.29:0/1163109785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91ea560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91fedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 -- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 msgr2=0x55eef91fedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91fedd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 -- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 msgr2=0x55eef91ea560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91ea560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 -- 172.21.15.29:0/1163109785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b5c009c90 con 0x55eef91fdf50 2023-10-14T18:17:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:34.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b637fe640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c017040 tx=0x7f8b5c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbfed6fdbfba28d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.552+0000 7f8b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:17:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c015880 con 0x55eef91fdf50 2023-10-14T18:17:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55eef9162160 con 0x55eef91fdf50 2023-10-14T18:17:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b5c025070 con 0x55eef91fdf50 2023-10-14T18:17:34.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b5c009af0 con 0x55eef91fdf50 2023-10-14T18:17:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b6c25c740 10 client.11403 fetch_fsmap learned FSMap version 107 2023-10-14T18:17:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x55eef9139da0 con 0x55eef91fdf50 2023-10-14T18:17:34.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55eef9162160 con 0x55eef91fdf50 2023-10-14T18:17:34.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.553+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8b5c0159e0 con 0x55eef91fdf50 2023-10-14T18:17:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 10 client.11403 fetch_fsmap finished waiting for FSMap version 107 2023-10-14T18:17:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 10 client.11403 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T18:17:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 20 client.11403 populate_metadata read hostname 'smithi029' 2023-10-14T18:17:34.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef91f8870 0x55eef92384a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 4 client.11403 mds_command: new command op to 11268 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:17:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 --> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55eef910dcc0 con 0x55eef91f8870 2023-10-14T18:17:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef91f8870 0x55eef92384a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef91f8870 0x55eef92384a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55848a820b31ede5 server_cookie=88971d1ecf975df7 in_seq=0 out_seq=0 2023-10-14T18:17:34.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.554+0000 7f8b41ffb640 10 client.11403 ms_handle_connect on v2:172.21.15.29:6836/1082656487 2023-10-14T18:17:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.555+0000 7f8b41ffb640 1 -- 172.21.15.29:0/1163109785 <== mds.0 v2:172.21.15.29:6836/1082656487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55eef910dcc0 con 0x55eef91f8870 2023-10-14T18:17:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.555+0000 7f8b41ffb640 10 client.11403 handle_command_reply: tid=0 2023-10-14T18:17:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.555+0000 7f8b6c25c740 1 client.11403 shutdown 2023-10-14T18:17:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.555+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 msgr2=0x55eef91f5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.555+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 secure :-1 s=READY pgs=2595 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c017040 tx=0x7f8b5c004ab0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 shutdown_connections 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef91f8870 0x55eef92384a0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef91f5cc0 unknown :-1 s=CLOSED pgs=2595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91fedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/1163109785 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91ea560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b41ffb640 0 client.11403 ms_handle_reset on v2:172.21.15.29:6836/1082656487 2023-10-14T18:17:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 >> 172.21.15.29:0/1163109785 conn(0x55eef91d05b0 msgr2=0x55eef921e410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 shutdown_connections 2023-10-14T18:17:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 1 -- 172.21.15.29:0/1163109785 wait complete. 2023-10-14T18:17:34.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.556+0000 7f8b6c25c740 20 client.11403 trim_cache size 0 max 16384 2023-10-14T18:17:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.573+0000 7f8b6c25c740 1 Processor -- start 2023-10-14T18:17:34.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.573+0000 7f8b6c25c740 1 -- start start 2023-10-14T18:17:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91cf0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91cf630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef8e9ff20 con 0x55eef91fdf50 2023-10-14T18:17:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef921d6e0 con 0x55eef91fd170 2023-10-14T18:17:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eef9108890 con 0x55eef91fd550 2023-10-14T18:17:34.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55630/0 (socket says 172.21.15.29:55630) 2023-10-14T18:17:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 -- 172.21.15.29:0/2763956645 learned_addr learned my addr 172.21.15.29:0/2763956645 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 -- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 msgr2=0x55eef91cf630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91cf630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 -- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 msgr2=0x55eef91cf0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91cf0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.574+0000 7f8b637fe640 1 -- 172.21.15.29:0/2763956645 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55eef90d8b80 con 0x55eef91fdf50 2023-10-14T18:17:34.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b637fe640 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f80 tx=0x7f8b5c021070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4b8ad173df7d5dc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b597fa640 1 -- 172.21.15.29:0/2763956645 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0148f0 con 0x55eef91fdf50 2023-10-14T18:17:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b597fa640 1 -- 172.21.15.29:0/2763956645 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c023070 con 0x55eef91fdf50 2023-10-14T18:17:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b597fa640 1 -- 172.21.15.29:0/2763956645 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b5c0158a0 con 0x55eef91fdf50 2023-10-14T18:17:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2763956645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 msgr2=0x55eef9203630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.575+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 secure :-1 s=READY pgs=2596 cs=0 l=1 rev1=1 crypto rx=0x7f8b5c000f80 tx=0x7f8b5c021070 comp rx=0 tx=0).stop 2023-10-14T18:17:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2763956645 shutdown_connections 2023-10-14T18:17:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fdf50 0x55eef9203630 unknown :-1 s=CLOSED pgs=2596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef91cf630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2763956645 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fd170 0x55eef91cf0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2763956645 >> 172.21.15.29:0/2763956645 conn(0x55eef91d05b0 msgr2=0x55eef91f4c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2763956645 shutdown_connections 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2763956645 wait complete. 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.576+0000 7f8b6c25c740 1 -- start start 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.577+0000 7f8b6c25c740 1 Processor -- start 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.577+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.577+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef92832f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b6c25c740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fdf50 0x55eef921aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef91f8370 con 0x55eef91fd170 2023-10-14T18:17:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55eef9283830 con 0x55eef91fdf50 2023-10-14T18:17:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b6c25c740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55eef91d5830 con 0x55eef91fd550 2023-10-14T18:17:34.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b6c25c740 10 client.0 _ll_register_callbacks cb 0x7f8b5c009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:17:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55642/0 (socket says 172.21.15.29:55642) 2023-10-14T18:17:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 -- 172.21.15.29:0/2027757660 learned_addr learned my addr 172.21.15.29:0/2027757660 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef92832f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b637fe640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fdf50 0x55eef921aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 -- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 msgr2=0x55eef92832f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef92832f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 -- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fdf50 msgr2=0x55eef921aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fdf50 0x55eef921aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.578+0000 7f8b62ffd640 1 -- 172.21.15.29:0/2027757660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55eef9139da0 con 0x55eef91fd170 2023-10-14T18:17:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b62ffd640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x55eef91f8c50 tx=0x7f8b54000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5231c7bd3f41c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b41ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:17:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b540243c0 con 0x55eef91fd170 2023-10-14T18:17:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f8b54024520 con 0x55eef91fd170 2023-10-14T18:17:34.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b5c00f050 con 0x55eef91fd170 2023-10-14T18:17:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=107) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b540249b0 con 0x55eef91fd170 2023-10-14T18:17:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b6c25c740 10 client.11409 fetch_fsmap learned FSMap version 107 2023-10-14T18:17:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.579+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=107,monmap=2+}) v3 -- 0x55eef90fe170 con 0x55eef91fd170 2023-10-14T18:17:34.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b5c00f050 con 0x55eef91fd170 2023-10-14T18:17:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 107) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f8b54024b10 con 0x55eef91fd170 2023-10-14T18:17:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 10 client.11409 fetch_fsmap finished waiting for FSMap version 107 2023-10-14T18:17:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 10 client.11409 resolve_mds: resolved name 'a' to daemon mds.a 2023-10-14T18:17:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 20 client.11409 populate_metadata read hostname 'smithi029' 2023-10-14T18:17:34.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef9203620 0x55eef9203a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 4 client.11409 mds_command: new command op to 11268 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 --> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55eef9134700 con 0x55eef9203620 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.580+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef9203620 0x55eef9203a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b5bfff640 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef9203620 0x55eef9203a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=667cba0b73435b32 server_cookie=8a2de025eaf89644 in_seq=0 out_seq=0 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b41ffb640 10 client.11409 ms_handle_connect on v2:172.21.15.29:6836/1082656487 2023-10-14T18:17:34.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b41ffb640 1 -- 172.21.15.29:0/2027757660 <== mds.0 v2:172.21.15.29:6836/1082656487 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x55eef9134700 con 0x55eef9203620 2023-10-14T18:17:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b41ffb640 10 client.11409 handle_command_reply: tid=0 2023-10-14T18:17:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 client.11409 shutdown 2023-10-14T18:17:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 msgr2=0x55eef91f7090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 secure :-1 s=READY pgs=2597 cs=0 l=1 rev1=1 crypto rx=0x55eef91f8c50 tx=0x7f8b54000ed0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 shutdown_connections 2023-10-14T18:17:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55eef91fdf50 0x55eef921aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55eef91fd550 0x55eef92832f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:6836/1082656487,v1:172.21.15.29:6837/1082656487] conn(0x55eef9203620 0x55eef9203a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 --2- 172.21.15.29:0/2027757660 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55eef91fd170 0x55eef91f7090 unknown :-1 s=CLOSED pgs=2597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b41ffb640 0 client.11409 ms_handle_reset on v2:172.21.15.29:6836/1082656487 2023-10-14T18:17:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 >> 172.21.15.29:0/2027757660 conn(0x55eef91d05b0 msgr2=0x55eef91d3c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 shutdown_connections 2023-10-14T18:17:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.581+0000 7f8b6c25c740 1 -- 172.21.15.29:0/2027757660 wait complete. 2023-10-14T18:17:34.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b6c25c740 20 client.11409 trim_cache size 0 max 16384 2023-10-14T18:17:34.596 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:17:34.596 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:17:34.596 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:17:34.596 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 11268, 2023-10-14T18:17:34.596 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:17:34.597 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:17:34.597 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T18:17:34.597 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8222433980000003, 2023-10-14T18:17:34.597 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 107, 2023-10-14T18:17:34.597 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 229, 2023-10-14T18:17:34.598 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 229, 2023-10-14T18:17:34.598 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.155949381999999 2023-10-14T18:17:34.598 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:17:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 msgr2=0x7f8b4406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 0x7f8b4406d450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f8b540011e0 tx=0x7f8b54009040 comp rx=0 tx=0).stop 2023-10-14T18:17:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 msgr2=0x7f8b641f9c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 secure :-1 s=READY pgs=2593 cs=0 l=1 rev1=1 crypto rx=0x7f8b4c009890 tx=0x7f8b4c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 shutdown_connections 2023-10-14T18:17:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f8b4406af90 0x7f8b4406d450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8b640c9060 0x7f8b640c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8b6406fb50 0x7f8b641f9c30 unknown :-1 s=CLOSED pgs=2593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 --2- 172.21.15.29:0/3940268548 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8b6406c9e0 0x7f8b641f6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 >> 172.21.15.29:0/3940268548 conn(0x7f8b640ce6a0 msgr2=0x7f8b640cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 shutdown_connections 2023-10-14T18:17:34.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.582+0000 7f8b63fff640 1 -- 172.21.15.29:0/3940268548 wait complete. 2023-10-14T18:17:34.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T18:17:34.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.825+0000 7fbac7ba9640 1 Processor -- start 2023-10-14T18:17:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.825+0000 7fbac7ba9640 1 -- start start 2023-10-14T18:17:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.826+0000 7fbac7ba9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbac00adb50 0x7fbac006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.826+0000 7fbac7ba9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbac0070fd0 0x7fbac00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.826+0000 7fbac7ba9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac0071450 con 0x7fbac00adf30 2023-10-14T18:17:34.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.826+0000 7fbac7ba9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00cc600 con 0x7fbac00adb50 2023-10-14T18:17:34.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.826+0000 7fbac7ba9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac00cc8e0 con 0x7fbac0070fd0 2023-10-14T18:17:34.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbac00adb50 0x7fbac006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbac00adb50 0x7fbac006f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55650/0 (socket says 172.21.15.29:55650) 2023-10-14T18:17:34.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 -- 172.21.15.29:0/2271342212 learned_addr learned my addr 172.21.15.29:0/2271342212 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 -- 172.21.15.29:0/2271342212 >> v2:172.21.15.186:3300/0 conn(0x7fbac0070fd0 msgr2=0x7fbac00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 --2- 172.21.15.29:0/2271342212 >> v2:172.21.15.186:3300/0 conn(0x7fbac0070fd0 0x7fbac00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 -- 172.21.15.29:0/2271342212 >> v1:172.21.15.186:6789/0 conn(0x7fbac00adf30 legacy=0x7fbac0164f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:34.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.827+0000 7fbac5945640 1 -- 172.21.15.29:0/2271342212 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac00ccbc0 con 0x7fbac00adb50 2023-10-14T18:17:34.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.829+0000 7fbac5945640 1 --2- 172.21.15.29:0/2271342212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac006f550 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fbac00cc2b0 tx=0x7fbab402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aea966542c5750a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.829+0000 7fbac4943640 1 -- 172.21.15.29:0/2271342212 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4031870 con 0x7fbac00adb50 2023-10-14T18:17:34.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.829+0000 7fbac4943640 1 -- 172.21.15.29:0/2271342212 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab40319d0 con 0x7fbac00adb50 2023-10-14T18:17:34.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.829+0000 7fbac4943640 1 -- 172.21.15.29:0/2271342212 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbab4031d60 con 0x7fbac00adb50 2023-10-14T18:17:34.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 -- 172.21.15.29:0/2271342212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 msgr2=0x7fbac006f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/2271342212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac006f550 secure :-1 s=READY pgs=2598 cs=0 l=1 rev1=1 crypto rx=0x7fbac00cc2b0 tx=0x7fbab402b910 comp rx=0 tx=0).stop 2023-10-14T18:17:34.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 -- 172.21.15.29:0/2271342212 shutdown_connections 2023-10-14T18:17:34.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/2271342212 >> v2:172.21.15.186:3300/0 conn(0x7fbac0070fd0 0x7fbac00b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/2271342212 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac006f550 unknown :-1 s=CLOSED pgs=2598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 -- 172.21.15.29:0/2271342212 >> 172.21.15.29:0/2271342212 conn(0x7fbac00cb410 msgr2=0x7fbac00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:34.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 -- 172.21.15.29:0/2271342212 shutdown_connections 2023-10-14T18:17:34.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.830+0000 7fbac7ba9640 1 -- 172.21.15.29:0/2271342212 wait complete. 2023-10-14T18:17:34.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.832+0000 7fbac7ba9640 1 Processor -- start 2023-10-14T18:17:34.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.832+0000 7fbac7ba9640 1 -- start start 2023-10-14T18:17:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbac0070fd0 0x7fbac00b2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbac00adf30 0x7fbac01f7460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c4c30 con 0x7fbac00adb50 2023-10-14T18:17:34.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c3fb0 con 0x7fbac0070fd0 2023-10-14T18:17:34.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac7ba9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac00c4290 con 0x7fbac00adf30 2023-10-14T18:17:34.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55666/0 (socket says 172.21.15.29:55666) 2023-10-14T18:17:34.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 -- 172.21.15.29:0/1292314697 learned_addr learned my addr 172.21.15.29:0/1292314697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:34.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 -- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbac00adf30 msgr2=0x7fbac01f7460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbac00adf30 0x7fbac01f7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 -- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbac0070fd0 msgr2=0x7fbac00b2c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:34.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbac0070fd0 0x7fbac00b2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:34.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.833+0000 7fbac5144640 1 -- 172.21.15.29:0/1292314697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab40094d0 con 0x7fbac00adb50 2023-10-14T18:17:34.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbac5144640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00ba50 tx=0x7fbabc00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc00cbe0 con 0x7fbac00adb50 2023-10-14T18:17:34.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbabc00cd60 con 0x7fbac00adb50 2023-10-14T18:17:34.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbabc014680 con 0x7fbac00adb50 2023-10-14T18:17:34.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac00c4570 con 0x7fbac00adb50 2023-10-14T18:17:34.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.834+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac00c5590 con 0x7fbac00adb50 2023-10-14T18:17:34.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.835+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac004e160 con 0x7fbac00adb50 2023-10-14T18:17:34.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.836+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbabc01d3b0 con 0x7fbac00adb50 2023-10-14T18:17:34.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.838+0000 7fbab27fc640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 0x7fbaa406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:34.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.838+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbabc0898a0 con 0x7fbac00adb50 2023-10-14T18:17:34.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.838+0000 7fbac5945640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 0x7fbaa406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:34.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.839+0000 7fbac5945640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 0x7fbaa406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab4030000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:34.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:34.842+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbabc054720 con 0x7fbac00adb50 2023-10-14T18:17:35.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.080+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbac00b4910 con 0x7fbac00adb50 2023-10-14T18:17:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.081+0000 7fbab27fc640 1 -- 172.21.15.29:0/1292314697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v107) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbabc019020 con 0x7fbac00adb50 2023-10-14T18:17:35.082 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T18:17:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 msgr2=0x7fbaa406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 0x7fbaa406d450 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbab4036000 tx=0x7fbab4030000 comp rx=0 tx=0).stop 2023-10-14T18:17:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 msgr2=0x7fbac00c5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 secure :-1 s=READY pgs=2599 cs=0 l=1 rev1=1 crypto rx=0x7fbabc00ba50 tx=0x7fbabc00bf10 comp rx=0 tx=0).stop 2023-10-14T18:17:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 shutdown_connections 2023-10-14T18:17:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbaa406af90 0x7fbaa406d450 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbac00adf30 0x7fbac01f7460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbac00adb50 0x7fbac00c5050 unknown :-1 s=CLOSED pgs=2599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 --2- 172.21.15.29:0/1292314697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbac0070fd0 0x7fbac00b2c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 >> 172.21.15.29:0/1292314697 conn(0x7fbac00cb410 msgr2=0x7fbac00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.084+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 shutdown_connections 2023-10-14T18:17:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:35.085+0000 7fbac7ba9640 1 -- 172.21.15.29:0/1292314697 wait complete. 2023-10-14T18:17:35.104 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T18:17:35.105 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T18:17:35.105 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T18:17:35.106 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T18:17:35.106 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T18:17:35.106 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T18:17:35.107 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T18:17:35.107 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T18:17:35.107 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:17:35.108 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.108 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T18:17:35.165 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T18:17:35.166 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T18:17:35.166 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T18:17:35.166 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.166 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T18:17:35.167 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.167 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:17:35.167 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:17:35.167 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T18:17:35.167 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:17:35.169 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:17:35.169 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T18:17:35.169 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T18:17:35.169 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T18:17:35.170 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T18:17:35.170 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T18:17:35.170 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36521sec preferred_lft 36521sec 2023-10-14T18:17:35.170 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T18:17:35.170 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.171 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:17:35.171 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:17:35.171 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T18:17:35.171 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T18:17:35.171 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:17:35.172 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T18:17:35.172 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.172 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T18:17:35.172 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.172 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:17:35.173 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T18:17:35.173 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T18:17:35.173 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.173 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:17:35.173 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T18:17:35.174 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T18:17:35.174 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:17:35.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.175 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T18:17:35.277 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T18:17:35.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.281 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T18:17:35.333 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T18:17:35.333 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T18:17:35.333 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T18:17:35.333 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T18:17:35.334 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T18:17:35.334 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.335 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T18:17:35.335 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T18:17:35.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T18:17:35.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T18:17:35.336 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T18:17:35.337 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T18:17:35.525 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.525 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:17:35.583 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T18:17:35.584 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T18:17:35.584 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:17:35.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.585 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T18:17:35.690 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:17:35.691 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T18:17:35.750 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:17:35.750 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:17:35.751 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:17:35.800 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T18:17:35.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:35.801 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:17:35.956 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:17:35.957 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:17:35.957 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:17:35.981 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:17:35.981 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:17:35.981 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:17:35.982 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:17:35.982 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:17:36.011 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:17:36.011 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:17:36.011 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:17:36.011 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:17:36.012 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:17:36.012 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T18:17:36.012 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T18:17:36.012 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T18:17:36.013 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T18:17:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T18:17:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T18:17:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T18:17:36.014 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T18:17:36.017 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T18:17:36.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:36.018 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:17:36.111 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:36.112 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T18:17:36.181 INFO:teuthology.orchestra.run.smithi029.stdout:client7334 2023-10-14T18:17:36.182 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:17:36.182 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:17:36.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-14T18:17:36.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.473+0000 7fa622d4e640 1 Processor -- start 2023-10-14T18:17:36.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.474+0000 7fa622d4e640 1 -- start start 2023-10-14T18:17:36.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.475+0000 7fa622d4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa61c15ac10 0x7fa61c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa622d4e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 0x7fa61c0b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa622d4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61c15b310 con 0x7fa61c0b0360 2023-10-14T18:17:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa622d4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c15f800 con 0x7fa61c15ac10 2023-10-14T18:17:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa622d4e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61c0b0d30 con 0x7fa61c0cac20 2023-10-14T18:17:36.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa620aea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa61c15ac10 0x7fa61c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa620aea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa61c15ac10 0x7fa61c06c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55672/0 (socket says 172.21.15.29:55672) 2023-10-14T18:17:36.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa620aea640 1 -- 172.21.15.29:0/1769895498 learned_addr learned my addr 172.21.15.29:0/1769895498 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa620aea640 1 -- 172.21.15.29:0/1769895498 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 msgr2=0x7fa61c0b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.476+0000 7fa6212eb640 1 --2- 172.21.15.29:0/1769895498 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 0x7fa61c0b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.477+0000 7fa620aea640 1 --2- 172.21.15.29:0/1769895498 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 0x7fa61c0b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.477+0000 7fa620aea640 1 -- 172.21.15.29:0/1769895498 >> v1:172.21.15.186:6789/0 conn(0x7fa61c0b0360 legacy=0x7fa61c0ca510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:36.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.477+0000 7fa620aea640 1 -- 172.21.15.29:0/1769895498 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61c06f640 con 0x7fa61c15ac10 2023-10-14T18:17:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.477+0000 7fa6212eb640 1 --2- 172.21.15.29:0/1769895498 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 0x7fa61c0b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:36.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.478+0000 7fa620aea640 1 --2- 172.21.15.29:0/1769895498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c06c730 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa61c06d730 tx=0x7fa61802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40b664ef0a60c310 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.478+0000 7fa6137fe640 1 -- 172.21.15.29:0/1769895498 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6180317e0 con 0x7fa61c15ac10 2023-10-14T18:17:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.478+0000 7fa6137fe640 1 -- 172.21.15.29:0/1769895498 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa618031940 con 0x7fa61c15ac10 2023-10-14T18:17:36.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.478+0000 7fa6137fe640 1 -- 172.21.15.29:0/1769895498 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa618031cd0 con 0x7fa61c15ac10 2023-10-14T18:17:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 -- 172.21.15.29:0/1769895498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 msgr2=0x7fa61c06c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 --2- 172.21.15.29:0/1769895498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c06c730 secure :-1 s=READY pgs=2600 cs=0 l=1 rev1=1 crypto rx=0x7fa61c06d730 tx=0x7fa61802b880 comp rx=0 tx=0).stop 2023-10-14T18:17:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 -- 172.21.15.29:0/1769895498 shutdown_connections 2023-10-14T18:17:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 --2- 172.21.15.29:0/1769895498 >> v2:172.21.15.186:3300/0 conn(0x7fa61c0cac20 0x7fa61c0b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:36.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 --2- 172.21.15.29:0/1769895498 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c06c730 unknown :-1 s=CLOSED pgs=2600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 -- 172.21.15.29:0/1769895498 >> 172.21.15.29:0/1769895498 conn(0x7fa61c0cb360 msgr2=0x7fa61c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 -- 172.21.15.29:0/1769895498 shutdown_connections 2023-10-14T18:17:36.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.479+0000 7fa622d4e640 1 -- 172.21.15.29:0/1769895498 wait complete. 2023-10-14T18:17:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.481+0000 7fa622d4e640 1 Processor -- start 2023-10-14T18:17:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.481+0000 7fa622d4e640 1 -- start start 2023-10-14T18:17:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c0b0360 0x7fa61c0cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0cac20 0x7fa61c0c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0b1500 con 0x7fa61c15ac10 2023-10-14T18:17:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0c7ba0 con 0x7fa61c0b0360 2023-10-14T18:17:36.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa622d4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa61c0c7db0 con 0x7fa61c0cac20 2023-10-14T18:17:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa6212eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa6212eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55684/0 (socket says 172.21.15.29:55684) 2023-10-14T18:17:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.482+0000 7fa6212eb640 1 -- 172.21.15.29:0/3090989777 learned_addr learned my addr 172.21.15.29:0/3090989777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa620aea640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c0b0360 0x7fa61c0cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa613fff640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0cac20 0x7fa61c0c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 -- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0cac20 msgr2=0x7fa61c0c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0cac20 0x7fa61c0c7660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 -- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c0b0360 msgr2=0x7fa61c0cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c0b0360 0x7fa61c0cd810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 -- 172.21.15.29:0/3090989777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6180094d0 con 0x7fa61c15ac10 2023-10-14T18:17:36.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.483+0000 7fa6212eb640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fa61400cef0 tx=0x7fa61400a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.484+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61400d840 con 0x7fa61c15ac10 2023-10-14T18:17:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.484+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa61400d9c0 con 0x7fa61c15ac10 2023-10-14T18:17:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.484+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa61400dd70 con 0x7fa61c15ac10 2023-10-14T18:17:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.484+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61c0c8090 con 0x7fa61c15ac10 2023-10-14T18:17:36.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.484+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61c0c8650 con 0x7fa61c15ac10 2023-10-14T18:17:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.485+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61c0b1670 con 0x7fa61c15ac10 2023-10-14T18:17:36.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.486+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa61401e3b0 con 0x7fa61c15ac10 2023-10-14T18:17:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.487+0000 7fa6117fa640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 0x7fa60806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.488+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa61408a350 con 0x7fa61c15ac10 2023-10-14T18:17:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.488+0000 7fa620aea640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 0x7fa60806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:36.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.489+0000 7fa620aea640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 0x7fa60806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa618036000 tx=0x7fa61803d040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:36.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.492+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa61c0b1670 con 0x7fa61c15ac10 2023-10-14T18:17:36.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:36.731+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fa61c15b860 con 0x7fa61c15ac10 2023-10-14T18:17:37.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.255+0000 7fa6117fa640 1 -- 172.21.15.29:0/3090989777 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v108) v1 ==== 136+0+0 (secure 0 0 0) 0x7fa614055290 con 0x7fa61c15ac10 2023-10-14T18:17:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.259+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 msgr2=0x7fa60806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.259+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 0x7fa60806d450 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa618036000 tx=0x7fa61803d040 comp rx=0 tx=0).stop 2023-10-14T18:17:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.259+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 msgr2=0x7fa61c14fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.259+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 secure :-1 s=READY pgs=2601 cs=0 l=1 rev1=1 crypto rx=0x7fa61400cef0 tx=0x7fa61400a710 comp rx=0 tx=0).stop 2023-10-14T18:17:37.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 shutdown_connections 2023-10-14T18:17:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa60806af90 0x7fa60806d450 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa61c15ac10 0x7fa61c14fbd0 unknown :-1 s=CLOSED pgs=2601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa61c0cac20 0x7fa61c0c7660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 --2- 172.21.15.29:0/3090989777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa61c0b0360 0x7fa61c0cd810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 >> 172.21.15.29:0/3090989777 conn(0x7fa61c0cb360 msgr2=0x7fa61c0ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:37.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 shutdown_connections 2023-10-14T18:17:37.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.260+0000 7fa622d4e640 1 -- 172.21.15.29:0/3090989777 wait complete. 2023-10-14T18:17:37.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.515+0000 7f9551d73640 1 Processor -- start 2023-10-14T18:17:37.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.515+0000 7f9551d73640 1 -- start start 2023-10-14T18:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551d73640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 0x7f954c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551d73640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f954c06c100 0x7f954c06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551d73640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f954c15aef0 con 0x7f954c159950 2023-10-14T18:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551d73640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c15e490 con 0x7f954c06c100 2023-10-14T18:17:37.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551d73640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f954c15d910 con 0x7f954c070b60 2023-10-14T18:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551572640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f954c06c100 0x7f954c06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551572640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f954c06c100 0x7f954c06c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55692/0 (socket says 172.21.15.29:55692) 2023-10-14T18:17:37.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.517+0000 7f9551572640 1 -- 172.21.15.29:0/2746352773 learned_addr learned my addr 172.21.15.29:0/2746352773 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f954bfff640 1 --2- 172.21.15.29:0/2746352773 >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 0x7f954c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f9551572640 1 -- 172.21.15.29:0/2746352773 >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 msgr2=0x7f954c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f9551572640 1 --2- 172.21.15.29:0/2746352773 >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 0x7f954c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f9551572640 1 -- 172.21.15.29:0/2746352773 >> v1:172.21.15.186:6789/0 conn(0x7f954c159950 legacy=0x7f954c070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f9551572640 1 -- 172.21.15.29:0/2746352773 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f954c15dbf0 con 0x7f954c06c100 2023-10-14T18:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.518+0000 7f954bfff640 1 --2- 172.21.15.29:0/2746352773 >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 0x7f954c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.519+0000 7f9551572640 1 --2- 172.21.15.29:0/2746352773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c06c100 0x7f954c06c510 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f954c0716a0 tx=0x7f954402be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284a27b697c28ae1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.519+0000 7f954b7fe640 1 -- 172.21.15.29:0/2746352773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954402c9f0 con 0x7f954c06c100 2023-10-14T18:17:37.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.519+0000 7f954b7fe640 1 -- 172.21.15.29:0/2746352773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954402cb50 con 0x7f954c06c100 2023-10-14T18:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.519+0000 7f954b7fe640 1 -- 172.21.15.29:0/2746352773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95440343f0 con 0x7f954c06c100 2023-10-14T18:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 -- 172.21.15.29:0/2746352773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c06c100 msgr2=0x7f954c06c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 --2- 172.21.15.29:0/2746352773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c06c100 0x7f954c06c510 secure :-1 s=READY pgs=2602 cs=0 l=1 rev1=1 crypto rx=0x7f954c0716a0 tx=0x7f954402be50 comp rx=0 tx=0).stop 2023-10-14T18:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 -- 172.21.15.29:0/2746352773 shutdown_connections 2023-10-14T18:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 --2- 172.21.15.29:0/2746352773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c06c100 0x7f954c06c510 unknown :-1 s=CLOSED pgs=2602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 --2- 172.21.15.29:0/2746352773 >> v2:172.21.15.186:3300/0 conn(0x7f954c070b60 0x7f954c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 -- 172.21.15.29:0/2746352773 >> 172.21.15.29:0/2746352773 conn(0x7f954c0ce6a0 msgr2=0x7f954c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:37.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 -- 172.21.15.29:0/2746352773 shutdown_connections 2023-10-14T18:17:37.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.520+0000 7f9551d73640 1 -- 172.21.15.29:0/2746352773 wait complete. 2023-10-14T18:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.522+0000 7f9551d73640 1 Processor -- start 2023-10-14T18:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.522+0000 7f9551d73640 1 -- start start 2023-10-14T18:17:37.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f954c06c100 0x7f954c1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f954c070b60 0x7f954c0c26d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c1d30 con 0x7f954c159950 2023-10-14T18:17:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c4210 con 0x7f954c06c100 2023-10-14T18:17:37.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551d73640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f954c0c2c10 con 0x7f954c070b60 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551572640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551572640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55694/0 (socket says 172.21.15.29:55694) 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.523+0000 7f9551572640 1 -- 172.21.15.29:0/3467435974 learned_addr learned my addr 172.21.15.29:0/3467435974 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9550d71640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f954c06c100 0x7f954c1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f954bfff640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f954c070b60 0x7f954c0c26d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 -- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f954c070b60 msgr2=0x7f954c0c26d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f954c070b60 0x7f954c0c26d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 -- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f954c06c100 msgr2=0x7f954c1f6420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f954c06c100 0x7f954c1f6420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 -- 172.21.15.29:0/3467435974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9544009b30 con 0x7f954c159950 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.524+0000 7f9551572640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f9544036040 tx=0x7f9544004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.525+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954403e450 con 0x7f954c159950 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.525+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f954403e5d0 con 0x7f954c159950 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.525+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f954c0c1f80 con 0x7f954c159950 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.525+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f954403e980 con 0x7f954c159950 2023-10-14T18:17:37.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.525+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f954c0c4470 con 0x7f954c159950 2023-10-14T18:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.526+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f954c0c24a0 con 0x7f954c159950 2023-10-14T18:17:37.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.527+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f954403eae0 con 0x7f954c159950 2023-10-14T18:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.528+0000 7f95497fa640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 0x7f953006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.529+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f95440ac5d0 con 0x7f954c159950 2023-10-14T18:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.529+0000 7f9550d71640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 0x7f953006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.529+0000 7f9550d71640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 0x7f953006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9540006fd0 tx=0x7f9540008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:37.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.534+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9544077570 con 0x7f954c159950 2023-10-14T18:17:37.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.732+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f954c15aef0 con 0x7f954c159950 2023-10-14T18:17:37.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.735+0000 7f95497fa640 1 -- 172.21.15.29:0/3467435974 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v229) v1 ==== 74+0+16757 (secure 0 0 0) 0x7f954c15aef0 con 0x7f954c159950 2023-10-14T18:17:37.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:37.737 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":229,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:28.226229+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":27,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":227,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:37.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.738+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 msgr2=0x7f953006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.738+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 0x7f953006d450 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9540006fd0 tx=0x7f9540008040 comp rx=0 tx=0).stop 2023-10-14T18:17:37.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.738+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 msgr2=0x7f954c15b090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.738+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 secure :-1 s=READY pgs=2603 cs=0 l=1 rev1=1 crypto rx=0x7f9544036040 tx=0x7f9544004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:37.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 shutdown_connections 2023-10-14T18:17:37.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f953006af90 0x7f953006d450 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f954c159950 0x7f954c15b090 unknown :-1 s=CLOSED pgs=2603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f954c070b60 0x7f954c0c26d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 --2- 172.21.15.29:0/3467435974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f954c06c100 0x7f954c1f6420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 >> 172.21.15.29:0/3467435974 conn(0x7f954c0ce6a0 msgr2=0x7f954c070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:37.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 shutdown_connections 2023-10-14T18:17:37.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.739+0000 7f9551d73640 1 -- 172.21.15.29:0/3467435974 wait complete. 2023-10-14T18:17:37.760 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:17:25.200413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:17:26.251106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:17:37.761 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:17:37.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.992+0000 7fb8bd2ca640 1 Processor -- start 2023-10-14T18:17:37.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.992+0000 7fb8bd2ca640 1 -- start start 2023-10-14T18:17:37.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.993+0000 7fb8bd2ca640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb8b80cb000 0x7fb8b80cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.993+0000 7fb8bd2ca640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b815d0e0 con 0x7fb8b815cd00 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.993+0000 7fb8bd2ca640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8b80cbd80 con 0x7fb8b8070380 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.993+0000 7fb8bd2ca640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8b80cbaa0 con 0x7fb8b80cb000 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb8b815cd00 0x7fb8b80cdfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40078/0 (socket says 172.21.15.29:40078) 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b7fff640 1 -- 172.21.15.29:0/1970052104 learned_addr learned my addr 172.21.15.29:0/1970052104 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:37.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b77fe640 1 --2- 172.21.15.29:0/1970052104 >> v2:172.21.15.29:3300/0 conn(0x7fb8b80cb000 0x7fb8b80cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:37.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b77fe640 1 -- 172.21.15.29:0/1970052104 >> v1:172.21.15.186:6789/0 conn(0x7fb8b8070380 legacy=0x7fb8b80ad920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b77fe640 1 -- 172.21.15.29:0/1970052104 >> v1:172.21.15.29:6789/0 conn(0x7fb8b815cd00 legacy=0x7fb8b80cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b77fe640 1 -- 172.21.15.29:0/1970052104 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b80cd500 con 0x7fb8b80cb000 2023-10-14T18:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.994+0000 7fb8b6ffd640 1 -- 172.21.15.29:0/1970052104 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342688305 0 0) 0x7fb8b815d0e0 con 0x7fb8b815cd00 2023-10-14T18:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.995+0000 7fb8b77fe640 1 --2- 172.21.15.29:0/1970052104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b80cb000 0x7fb8b80cb490 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0097a0 tx=0x7fb8ac02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a50977d58d26604 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:37.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8b6ffd640 1 -- 172.21.15.29:0/1970052104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac031830 con 0x7fb8b80cb000 2023-10-14T18:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8b6ffd640 1 -- 172.21.15.29:0/1970052104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac031990 con 0x7fb8b80cb000 2023-10-14T18:17:37.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8b6ffd640 1 -- 172.21.15.29:0/1970052104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8ac031d20 con 0x7fb8b80cb000 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/1970052104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b80cb000 msgr2=0x7fb8b80cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/1970052104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b80cb000 0x7fb8b80cb490 secure :-1 s=READY pgs=2605 cs=0 l=1 rev1=1 crypto rx=0x7fb8ac0097a0 tx=0x7fb8ac02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/1970052104 shutdown_connections 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/1970052104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b80cb000 0x7fb8b80cb490 unknown :-1 s=CLOSED pgs=2605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.996+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/1970052104 >> 172.21.15.29:0/1970052104 conn(0x7fb8b80cf790 msgr2=0x7fb8b80cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:37.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.997+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/1970052104 shutdown_connections 2023-10-14T18:17:38.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.997+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/1970052104 wait complete. 2023-10-14T18:17:38.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.998+0000 7fb8bd2ca640 1 Processor -- start 2023-10-14T18:17:38.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.998+0000 7fb8bd2ca640 1 -- start start 2023-10-14T18:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8b8070380 0x7fb8b80c3d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b80cb000 0x7fb8b8157c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80c38e0 con 0x7fb8b815cd00 2023-10-14T18:17:38.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80c2fb0 con 0x7fb8b80cb000 2023-10-14T18:17:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:37.999+0000 7fb8bd2ca640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8b80c3290 con 0x7fb8b8070380 2023-10-14T18:17:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55720/0 (socket says 172.21.15.29:55720) 2023-10-14T18:17:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 -- 172.21.15.29:0/2636830507 learned_addr learned my addr 172.21.15.29:0/2636830507 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:38.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8b7fff640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8b8070380 0x7fb8b80c3d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8b77fe640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b80cb000 0x7fb8b8157c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 -- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8b8070380 msgr2=0x7fb8b80c3d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8b8070380 0x7fb8b80c3d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 -- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b80cb000 msgr2=0x7fb8b8157c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b80cb000 0x7fb8b8157c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.000+0000 7fb8bcac9640 1 -- 172.21.15.29:0/2636830507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8ac009400 con 0x7fb8b815cd00 2023-10-14T18:17:38.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8bcac9640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000c410 tx=0x7fb8b000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0011790 con 0x7fb8b815cd00 2023-10-14T18:17:38.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b00118f0 con 0x7fb8b815cd00 2023-10-14T18:17:38.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8b80c3570 con 0x7fb8b815cd00 2023-10-14T18:17:38.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb8b0011ce0 con 0x7fb8b815cd00 2023-10-14T18:17:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.001+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8b80b53e0 con 0x7fb8b815cd00 2023-10-14T18:17:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.002+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8b80c3a90 con 0x7fb8b815cd00 2023-10-14T18:17:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.004+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb8b001c460 con 0x7fb8b815cd00 2023-10-14T18:17:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.005+0000 7fb8b4ff9640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 0x7fb8a406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.005+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb8b0089510 con 0x7fb8b815cd00 2023-10-14T18:17:38.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.005+0000 7fb8b7fff640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 0x7fb8a406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.006+0000 7fb8b7fff640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 0x7fb8a406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80012e0 tx=0x7fb8a8005eb0 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.009+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb8b80c3a90 con 0x7fb8b815cd00 2023-10-14T18:17:38.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.204+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb8b804e9b0 con 0x7fb8b815cd00 2023-10-14T18:17:38.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.205+0000 7fb8b4ff9640 1 -- 172.21.15.29:0/2636830507 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v229) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb8b0054420 con 0x7fb8b815cd00 2023-10-14T18:17:38.206 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:17:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 msgr2=0x7fb8a406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 0x7fb8a406d450 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb8a80012e0 tx=0x7fb8a8005eb0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 msgr2=0x7fb8b80c4630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 secure :-1 s=READY pgs=2606 cs=0 l=1 rev1=1 crypto rx=0x7fb8b000c410 tx=0x7fb8b000c8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 shutdown_connections 2023-10-14T18:17:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb8a406af90 0x7fb8a406d450 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb8b815cd00 0x7fb8b80c4630 unknown :-1 s=CLOSED pgs=2606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb8b80cb000 0x7fb8b8157c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 --2- 172.21.15.29:0/2636830507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb8b8070380 0x7fb8b80c3d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 >> 172.21.15.29:0/2636830507 conn(0x7fb8b80cf790 msgr2=0x7fb8b80cfb70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 shutdown_connections 2023-10-14T18:17:38.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.208+0000 7fb8bd2ca640 1 -- 172.21.15.29:0/2636830507 wait complete. 2023-10-14T18:17:38.230 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:17:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.454+0000 7f892dd8e640 1 Processor -- start 2023-10-14T18:17:38.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.454+0000 7f892dd8e640 1 -- start start 2023-10-14T18:17:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f892dd8e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 0x7f89280d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f892dd8e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89280cded0 0x7f89280b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f892dd8e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f892815d150 con 0x7f89280b8480 2023-10-14T18:17:38.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f892dd8e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f89280d1180 con 0x7f89280cded0 2023-10-14T18:17:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f892dd8e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89280d1460 con 0x7f892815cd70 2023-10-14T18:17:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f8927fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f89280b8480 0x7f89280cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40084/0 (socket says 172.21.15.29:40084) 2023-10-14T18:17:38.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.456+0000 7f8927fff640 1 -- 172.21.15.29:0/919483241 learned_addr learned my addr 172.21.15.29:0/919483241 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:38.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f892cd8c640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 0x7f89280d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f892d58d640 1 --2- 172.21.15.29:0/919483241 >> v2:172.21.15.186:3300/0 conn(0x7f89280cded0 0x7f89280b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1960486243 0 0) 0x7f892815d150 con 0x7f89280b8480 2023-10-14T18:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8918003520 con 0x7f89280b8480 2023-10-14T18:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 387166870 0 0) 0x7f8918003520 con 0x7f89280b8480 2023-10-14T18:17:38.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 msgr2=0x7f89280d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 0x7f89280d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 >> v2:172.21.15.186:3300/0 conn(0x7f89280cded0 msgr2=0x7f89280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 --2- 172.21.15.29:0/919483241 >> v2:172.21.15.186:3300/0 conn(0x7f89280cded0 0x7f89280b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89280d1740 con 0x7f89280b8480 2023-10-14T18:17:38.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8914001670 con 0x7f89280b8480 2023-10-14T18:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 >> v1:172.21.15.29:6789/0 conn(0x7f89280b8480 legacy=0x7f89280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8918003970 0x7f8918003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89180047c0 0x7f8918004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 0x7f8918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280d1180 con 0x7f8918003970 2023-10-14T18:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280d1460 con 0x7f89180047c0 2023-10-14T18:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.457+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891800b980 con 0x7f892815cd70 2023-10-14T18:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8918003970 0x7f8918003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 -- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 msgr2=0x7f8918005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815cd70 0x7f8918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f8927fff640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89180047c0 0x7f8918004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 -- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89180047c0 msgr2=0x7f8918004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f89180047c0 0x7f8918004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.458+0000 7f892cd8c640 1 -- 172.21.15.29:0/919483241 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8918003520 con 0x7f8918003970 2023-10-14T18:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.459+0000 7f892cd8c640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8918003970 0x7f8918003d90 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f891c009ac0 tx=0x7f89180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607e8240213a11b9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c02c710 con 0x7f8918003970 2023-10-14T18:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f891c02c870 con 0x7f8918003970 2023-10-14T18:17:38.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f89277fe640 1 -- 172.21.15.29:0/919483241 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f891c02cc00 con 0x7f8918003970 2023-10-14T18:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 -- 172.21.15.29:0/919483241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8918003970 msgr2=0x7f8918003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 --2- 172.21.15.29:0/919483241 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8918003970 0x7f8918003d90 secure :-1 s=READY pgs=2608 cs=0 l=1 rev1=1 crypto rx=0x7f891c009ac0 tx=0x7f89180057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892d58d640 1 -- 172.21.15.29:0/919483241 reap_dead start 2023-10-14T18:17:38.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 -- 172.21.15.29:0/919483241 shutdown_connections 2023-10-14T18:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 -- 172.21.15.29:0/919483241 >> 172.21.15.29:0/919483241 conn(0x7f89280cf570 msgr2=0x7f89280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 -- 172.21.15.29:0/919483241 shutdown_connections 2023-10-14T18:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.460+0000 7f892dd8e640 1 -- 172.21.15.29:0/919483241 wait complete. 2023-10-14T18:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.462+0000 7f892dd8e640 1 Processor -- start 2023-10-14T18:17:38.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.462+0000 7f892dd8e640 1 -- start start 2023-10-14T18:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8918003970 0x7f892814c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815e4d0 0x7f892815e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280a98b0 con 0x7f89280b7670 2023-10-14T18:17:38.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89280b8030 con 0x7f8918003970 2023-10-14T18:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f892dd8e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f892814c900 con 0x7f892815e4d0 2023-10-14T18:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.463+0000 7f8927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55738/0 (socket says 172.21.15.29:55738) 2023-10-14T18:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 -- 172.21.15.29:0/123539893 learned_addr learned my addr 172.21.15.29:0/123539893 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:38.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 -- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815e4d0 msgr2=0x7f892815e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815e4d0 0x7f892815e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 -- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8918003970 msgr2=0x7f892814c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8918003970 0x7f892814c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 -- 172.21.15.29:0/123539893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892815eec0 con 0x7f89280b7670 2023-10-14T18:17:38.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f8927fff640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f891400d7c0 tx=0x7f891400dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.464+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140074e0 con 0x7f89280b7670 2023-10-14T18:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.465+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8914007640 con 0x7f89280b7670 2023-10-14T18:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.465+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89140049e0 con 0x7f89280b7670 2023-10-14T18:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.465+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f892815f150 con 0x7f89280b7670 2023-10-14T18:17:38.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.465+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f892815f710 con 0x7f89280b7670 2023-10-14T18:17:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.466+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f892804e9b0 con 0x7f89280b7670 2023-10-14T18:17:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.467+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8914004b40 con 0x7f89280b7670 2023-10-14T18:17:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.468+0000 7f89257fa640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 0x7f890c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.468+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f89140891d0 con 0x7f89280b7670 2023-10-14T18:17:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.468+0000 7f892cd8c640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 0x7f890c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.469+0000 7f892cd8c640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 0x7f890c06d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f891c036000 tx=0x7f891c02e000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.472+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8914054090 con 0x7f89280b7670 2023-10-14T18:17:38.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.667+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f89280b6790 con 0x7f89280b7670 2023-10-14T18:17:38.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.668+0000 7f89257fa640 1 -- 172.21.15.29:0/123539893 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v229) v1 ==== 104+0+11 (secure 0 0 0) 0x7f89140576b0 con 0x7f89280b7670 2023-10-14T18:17:38.669 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:17:38.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 msgr2=0x7f890c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 0x7f890c06d450 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f891c036000 tx=0x7f891c02e000 comp rx=0 tx=0).stop 2023-10-14T18:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 msgr2=0x7f89280b7af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 secure :-1 s=READY pgs=2609 cs=0 l=1 rev1=1 crypto rx=0x7f891400d7c0 tx=0x7f891400dc80 comp rx=0 tx=0).stop 2023-10-14T18:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 shutdown_connections 2023-10-14T18:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f890c06af90 0x7f890c06d450 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f892815e4d0 0x7f892815e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f89280b7670 0x7f89280b7af0 unknown :-1 s=CLOSED pgs=2609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 --2- 172.21.15.29:0/123539893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8918003970 0x7f892814c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 >> 172.21.15.29:0/123539893 conn(0x7f89280cf570 msgr2=0x7f89280b4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:38.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.671+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 shutdown_connections 2023-10-14T18:17:38.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.672+0000 7f892dd8e640 1 -- 172.21.15.29:0/123539893 wait complete. 2023-10-14T18:17:38.692 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:17:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.922+0000 7f22a5467640 1 Processor -- start 2023-10-14T18:17:38.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.922+0000 7f22a5467640 1 -- start start 2023-10-14T18:17:38.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.924+0000 7f22a5467640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a00ad970 0x7f22a00cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.924+0000 7f22a5467640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a015b9d0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.924+0000 7f22a5467640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22a015bcb0 con 0x7f22a006c040 2023-10-14T18:17:38.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.924+0000 7f22a5467640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22a015bf90 con 0x7f22a00ad970 2023-10-14T18:17:38.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.924+0000 7f2297fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a00ad970 0x7f22a00cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f2297fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f22a00ad970 0x7f22a00cca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55742/0 (socket says 172.21.15.29:55742) 2023-10-14T18:17:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f2297fff640 1 -- 172.21.15.29:0/3884981713 learned_addr learned my addr 172.21.15.29:0/3884981713 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f2297fff640 1 -- 172.21.15.29:0/3884981713 >> v1:172.21.15.186:6789/0 conn(0x7f22a006c040 legacy=0x7f22a006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f2297fff640 1 -- 172.21.15.29:0/3884981713 >> v1:172.21.15.29:6789/0 conn(0x7f22a00ccfb0 legacy=0x7f22a00cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f2297fff640 1 -- 172.21.15.29:0/3884981713 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22a00cc270 con 0x7f22a00ad970 2023-10-14T18:17:38.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.925+0000 7f229f7fe640 1 -- 172.21.15.29:0/3884981713 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199428081 0 0) 0x7f22a015b9d0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.926+0000 7f2297fff640 1 --2- 172.21.15.29:0/3884981713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ad970 0x7f22a00cca70 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f228802ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f3e193ceedf3cfc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f229f7fe640 1 -- 172.21.15.29:0/3884981713 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228802f830 con 0x7f22a00ad970 2023-10-14T18:17:38.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f229f7fe640 1 -- 172.21.15.29:0/3884981713 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228802f990 con 0x7f22a00ad970 2023-10-14T18:17:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f229f7fe640 1 -- 172.21.15.29:0/3884981713 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f228802fd20 con 0x7f22a00ad970 2023-10-14T18:17:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f22a5467640 1 -- 172.21.15.29:0/3884981713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ad970 msgr2=0x7f22a00cca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f22a5467640 1 --2- 172.21.15.29:0/3884981713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ad970 0x7f22a00cca70 secure :-1 s=READY pgs=2611 cs=0 l=1 rev1=1 crypto rx=0x7f22880098a0 tx=0x7f228802ba10 comp rx=0 tx=0).stop 2023-10-14T18:17:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f22a5467640 1 -- 172.21.15.29:0/3884981713 shutdown_connections 2023-10-14T18:17:38.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f22a5467640 1 --2- 172.21.15.29:0/3884981713 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ad970 0x7f22a00cca70 unknown :-1 s=CLOSED pgs=2611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.927+0000 7f22a5467640 1 -- 172.21.15.29:0/3884981713 >> 172.21.15.29:0/3884981713 conn(0x7f22a00ce6a0 msgr2=0x7f22a00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:38.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.928+0000 7f22a5467640 1 -- 172.21.15.29:0/3884981713 shutdown_connections 2023-10-14T18:17:38.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.928+0000 7f22a5467640 1 -- 172.21.15.29:0/3884981713 wait complete. 2023-10-14T18:17:38.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.929+0000 7f22a5467640 1 Processor -- start 2023-10-14T18:17:38.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.930+0000 7f22a5467640 1 -- start start 2023-10-14T18:17:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.930+0000 7f22a5467640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a006c040 0x7f22a01e4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a5467640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 0x7f22a0141660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a5467640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a5467640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b2d10 con 0x7f22a00ccfb0 2023-10-14T18:17:38.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a5467640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b23b0 con 0x7f22a006c040 2023-10-14T18:17:38.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a5467640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22a00b2650 con 0x7f22a00ad970 2023-10-14T18:17:38.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55752/0 (socket says 172.21.15.29:55752) 2023-10-14T18:17:38.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 -- 172.21.15.29:0/3616233301 learned_addr learned my addr 172.21.15.29:0/3616233301 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:38.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f229ffff640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a006c040 0x7f22a01e4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f2297fff640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 0x7f22a0141660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 -- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 msgr2=0x7f22a0141660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 0x7f22a0141660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 -- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a006c040 msgr2=0x7f22a01e4ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:38.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a006c040 0x7f22a01e4ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:38.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f22a4c66640 1 -- 172.21.15.29:0/3616233301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880094d0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.931+0000 7f2297fff640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 0x7f22a0141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:38.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f22a4c66640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f229800c760 tx=0x7f229800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298003da0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f229800ce70 con 0x7f22a00ccfb0 2023-10-14T18:17:38.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22a00b2930 con 0x7f22a00ccfb0 2023-10-14T18:17:38.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2298005290 con 0x7f22a00ccfb0 2023-10-14T18:17:38.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.932+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22a00c4e80 con 0x7f22a00ccfb0 2023-10-14T18:17:38.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.933+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22a00b2e50 con 0x7f22a00ccfb0 2023-10-14T18:17:38.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.934+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f22980053f0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.935+0000 7f229d7fa640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 0x7f228006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:38.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.935+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f22980896a0 con 0x7f22a00ccfb0 2023-10-14T18:17:38.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.936+0000 7f229ffff640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 0x7f228006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:38.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.936+0000 7f229ffff640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 0x7f228006d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2290000f60 tx=0x7f2290009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:38.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:38.940+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f22980545b0 con 0x7f22a00ccfb0 2023-10-14T18:17:39.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.129+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f22a004e9b0 con 0x7f22a00ccfb0 2023-10-14T18:17:39.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.130+0000 7f229d7fa640 1 -- 172.21.15.29:0/3616233301 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v229) v1 ==== 100+0+11 (secure 0 0 0) 0x7f2298057bd0 con 0x7f22a00ccfb0 2023-10-14T18:17:39.131 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:17:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 msgr2=0x7f228006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 0x7f228006d450 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2290000f60 tx=0x7f2290009040 comp rx=0 tx=0).stop 2023-10-14T18:17:39.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 msgr2=0x7f22a00c4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 secure :-1 s=READY pgs=2612 cs=0 l=1 rev1=1 crypto rx=0x7f229800c760 tx=0x7f229800cc20 comp rx=0 tx=0).stop 2023-10-14T18:17:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 shutdown_connections 2023-10-14T18:17:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f228006af90 0x7f228006d450 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f22a00ccfb0 0x7f22a00c4940 unknown :-1 s=CLOSED pgs=2612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f22a00ad970 0x7f22a0141660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 --2- 172.21.15.29:0/3616233301 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f22a006c040 0x7f22a01e4ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 >> 172.21.15.29:0/3616233301 conn(0x7f22a00ce6a0 msgr2=0x7f22a00c5800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 shutdown_connections 2023-10-14T18:17:39.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.133+0000 7f22a5467640 1 -- 172.21.15.29:0/3616233301 wait complete. 2023-10-14T18:17:39.153 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:17:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.375+0000 7fa31f59e640 1 Processor -- start 2023-10-14T18:17:39.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.375+0000 7fa31f59e640 1 -- start start 2023-10-14T18:17:39.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa32006c040 0x7fa32006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3200ad970 0x7fa32015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa32006c960 con 0x7fa32015ebc0 2023-10-14T18:17:39.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa32015b9d0 con 0x7fa32006c040 2023-10-14T18:17:39.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa32015bcb0 con 0x7fa3200ad970 2023-10-14T18:17:39.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa32006c040 0x7fa32006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa32006c040 0x7fa32006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55764/0 (socket says 172.21.15.29:55764) 2023-10-14T18:17:39.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.377+0000 7fa31e59c640 1 -- 172.21.15.29:0/509273091 learned_addr learned my addr 172.21.15.29:0/509273091 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:39.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.378+0000 7fa31e59c640 1 -- 172.21.15.29:0/509273091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3200ad970 msgr2=0x7fa32015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:39.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.378+0000 7fa31e59c640 1 --2- 172.21.15.29:0/509273091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3200ad970 0x7fa32015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.378+0000 7fa31e59c640 1 -- 172.21.15.29:0/509273091 >> v1:172.21.15.186:6789/0 conn(0x7fa32015ebc0 legacy=0x7fa3200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:39.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.378+0000 7fa31e59c640 1 -- 172.21.15.29:0/509273091 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa32015bf90 con 0x7fa32006c040 2023-10-14T18:17:39.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.379+0000 7fa31e59c640 1 --2- 172.21.15.29:0/509273091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa32006c040 0x7fa32006c420 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fa3100098a0 tx=0x7fa31002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25c0a1fe7c94b917 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.379+0000 7fa31dd9b640 1 -- 172.21.15.29:0/509273091 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310031640 con 0x7fa32006c040 2023-10-14T18:17:39.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.379+0000 7fa31dd9b640 1 -- 172.21.15.29:0/509273091 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3100317a0 con 0x7fa32006c040 2023-10-14T18:17:39.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.379+0000 7fa31dd9b640 1 -- 172.21.15.29:0/509273091 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa310031b30 con 0x7fa32006c040 2023-10-14T18:17:39.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 -- 172.21.15.29:0/509273091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa32006c040 msgr2=0x7fa32006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 --2- 172.21.15.29:0/509273091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa32006c040 0x7fa32006c420 secure :-1 s=READY pgs=2613 cs=0 l=1 rev1=1 crypto rx=0x7fa3100098a0 tx=0x7fa31002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 -- 172.21.15.29:0/509273091 shutdown_connections 2023-10-14T18:17:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 --2- 172.21.15.29:0/509273091 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa3200ad970 0x7fa32015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 --2- 172.21.15.29:0/509273091 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa32006c040 0x7fa32006c420 unknown :-1 s=CLOSED pgs=2613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 -- 172.21.15.29:0/509273091 >> 172.21.15.29:0/509273091 conn(0x7fa3200ce6a0 msgr2=0x7fa3200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 -- 172.21.15.29:0/509273091 shutdown_connections 2023-10-14T18:17:39.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.380+0000 7fa31f59e640 1 -- 172.21.15.29:0/509273091 wait complete. 2023-10-14T18:17:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.382+0000 7fa31f59e640 1 Processor -- start 2023-10-14T18:17:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.382+0000 7fa31f59e640 1 -- start start 2023-10-14T18:17:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa32006c040 0x7fa320153260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 0x7fa3201e8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201e5be0 con 0x7fa3200ad970 2023-10-14T18:17:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3200c8cf0 con 0x7fa32015ebc0 2023-10-14T18:17:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3201e5900 con 0x7fa32006c040 2023-10-14T18:17:39.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55776/0 (socket says 172.21.15.29:55776) 2023-10-14T18:17:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 -- 172.21.15.29:0/2538593149 learned_addr learned my addr 172.21.15.29:0/2538593149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31e59c640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa32006c040 0x7fa320153260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 -- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa32006c040 msgr2=0x7fa320153260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa31ed9d640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 0x7fa3201e8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa32006c040 0x7fa320153260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.383+0000 7fa317fff640 1 -- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 msgr2=0x7fa3201e8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa317fff640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 0x7fa3201e8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa317fff640 1 -- 172.21.15.29:0/2538593149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3100094d0 con 0x7fa3200ad970 2023-10-14T18:17:39.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa31ed9d640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 0x7fa3201e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:39.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa317fff640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b410 tx=0x7fa30800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308011830 con 0x7fa3200ad970 2023-10-14T18:17:39.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3080119b0 con 0x7fa3200ad970 2023-10-14T18:17:39.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa308011d60 con 0x7fa3200ad970 2023-10-14T18:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3200b3b40 con 0x7fa3200ad970 2023-10-14T18:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.384+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3201e5590 con 0x7fa3200ad970 2023-10-14T18:17:39.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.385+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa32006b330 con 0x7fa3200ad970 2023-10-14T18:17:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.386+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa30801a400 con 0x7fa3200ad970 2023-10-14T18:17:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.387+0000 7fa316ffd640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 0x7fa30006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.387+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa3080899a0 con 0x7fa3200ad970 2023-10-14T18:17:39.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.388+0000 7fa31e59c640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 0x7fa30006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.388+0000 7fa31e59c640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 0x7fa30006d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa310036000 tx=0x7fa31003c040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.392+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa3080548c0 con 0x7fa3200ad970 2023-10-14T18:17:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.588+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa3200b39a0 con 0x7fa3200ad970 2023-10-14T18:17:39.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.589+0000 7fa316ffd640 1 -- 172.21.15.29:0/2538593149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v229) v1 ==== 84+0+209 (secure 0 0 0) 0x7fa3200b39a0 con 0x7fa3200ad970 2023-10-14T18:17:39.591 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:39.591 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:17:39.591 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:39.591 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:17:39.592 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:17:39.592 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:39.592 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:39.593 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 26, 2023-10-14T18:17:39.593 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_metadata" 2023-10-14T18:17:39.593 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:39.593 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:39.593 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 27, 2023-10-14T18:17:39.594 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_data" 2023-10-14T18:17:39.594 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:39.594 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:17:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.591+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 msgr2=0x7fa30006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.591+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 0x7fa30006d450 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fa310036000 tx=0x7fa31003c040 comp rx=0 tx=0).stop 2023-10-14T18:17:39.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 msgr2=0x7fa3201e8a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 secure :-1 s=READY pgs=2614 cs=0 l=1 rev1=1 crypto rx=0x7fa30800b410 tx=0x7fa30800b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 shutdown_connections 2023-10-14T18:17:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fa30006af90 0x7fa30006d450 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa32015ebc0 0x7fa3201e8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa3200ad970 0x7fa3201e8a60 unknown :-1 s=CLOSED pgs=2614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 --2- 172.21.15.29:0/2538593149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa32006c040 0x7fa320153260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 >> 172.21.15.29:0/2538593149 conn(0x7fa3200ce6a0 msgr2=0x7fa32015dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:39.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 shutdown_connections 2023-10-14T18:17:39.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.592+0000 7fa31f59e640 1 -- 172.21.15.29:0/2538593149 wait complete. 2023-10-14T18:17:39.611 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-14T18:17:39.612 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-14T18:17:39.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.840+0000 7f3052167640 1 Processor -- start 2023-10-14T18:17:39.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.841+0000 7f3052167640 1 -- start start 2023-10-14T18:17:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.842+0000 7f3052167640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f304c15cd70 0x7f304c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.842+0000 7f3052167640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c0cded0 0x7f304c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.842+0000 7f3052167640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304c15d150 con 0x7f304c0b8480 2023-10-14T18:17:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.842+0000 7f3052167640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304c0d1180 con 0x7f304c15cd70 2023-10-14T18:17:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.842+0000 7f3052167640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c0d1460 con 0x7f304c0cded0 2023-10-14T18:17:39.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f304c15cd70 0x7f304c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f304c15cd70 0x7f304c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55792/0 (socket says 172.21.15.29:55792) 2023-10-14T18:17:39.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 -- 172.21.15.29:0/561576428 learned_addr learned my addr 172.21.15.29:0/561576428 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:39.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051966640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c0cded0 0x7f304c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 -- 172.21.15.29:0/561576428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c0cded0 msgr2=0x7f304c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:39.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c0cded0 0x7f304c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 -- 172.21.15.29:0/561576428 >> v1:172.21.15.186:6789/0 conn(0x7f304c0b8480 legacy=0x7f304c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:39.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.843+0000 7f3051165640 1 -- 172.21.15.29:0/561576428 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304c0d1740 con 0x7f304c15cd70 2023-10-14T18:17:39.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.844+0000 7f3051165640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c15cd70 0x7f304c0d2220 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f304c0b56a0 tx=0x7f304802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6d61e826b8461f5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.845+0000 7f3043fff640 1 -- 172.21.15.29:0/561576428 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048031830 con 0x7f304c15cd70 2023-10-14T18:17:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.845+0000 7f3043fff640 1 -- 172.21.15.29:0/561576428 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3048031990 con 0x7f304c15cd70 2023-10-14T18:17:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.845+0000 7f3043fff640 1 -- 172.21.15.29:0/561576428 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3048031d20 con 0x7f304c15cd70 2023-10-14T18:17:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.845+0000 7f3052167640 1 -- 172.21.15.29:0/561576428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c15cd70 msgr2=0x7f304c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.845+0000 7f3052167640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c15cd70 0x7f304c0d2220 secure :-1 s=READY pgs=2615 cs=0 l=1 rev1=1 crypto rx=0x7f304c0b56a0 tx=0x7f304802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 -- 172.21.15.29:0/561576428 shutdown_connections 2023-10-14T18:17:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c0cded0 0x7f304c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 --2- 172.21.15.29:0/561576428 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c15cd70 0x7f304c0d2220 unknown :-1 s=CLOSED pgs=2615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 -- 172.21.15.29:0/561576428 >> 172.21.15.29:0/561576428 conn(0x7f304c0cf570 msgr2=0x7f304c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 -- 172.21.15.29:0/561576428 shutdown_connections 2023-10-14T18:17:39.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.846+0000 7f3052167640 1 -- 172.21.15.29:0/561576428 wait complete. 2023-10-14T18:17:39.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.848+0000 7f3052167640 1 Processor -- start 2023-10-14T18:17:39.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.848+0000 7f3052167640 1 -- start start 2023-10-14T18:17:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.849+0000 7f3052167640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3052167640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 0x7f304c1f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3052167640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 0x7f304c15f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3052167640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c15d500 con 0x7f304c0b8480 2023-10-14T18:17:39.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3052167640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f304c15d7e0 con 0x7f304c0cded0 2023-10-14T18:17:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3052167640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f304c15dac0 con 0x7f304c15cd70 2023-10-14T18:17:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55798/0 (socket says 172.21.15.29:55798) 2023-10-14T18:17:39.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 -- 172.21.15.29:0/1872041768 learned_addr learned my addr 172.21.15.29:0/1872041768 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3050964640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 0x7f304c1f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051966640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 0x7f304c15f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 -- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 msgr2=0x7f304c15f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 0x7f304c15f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 -- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 msgr2=0x7f304c1f2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 0x7f304c1f2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051165640 1 -- 172.21.15.29:0/1872041768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30480094d0 con 0x7f304c0b8480 2023-10-14T18:17:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3050964640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 0x7f304c1f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.850+0000 7f3051966640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 0x7f304c15f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:39.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3051165640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3048000c00 tx=0x7f30480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30480317d0 con 0x7f304c0b8480 2023-10-14T18:17:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f304803aa60 con 0x7f304c0b8480 2023-10-14T18:17:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304803ae10 con 0x7f304c0b8480 2023-10-14T18:17:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f304c15dda0 con 0x7f304c0b8480 2023-10-14T18:17:39.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.851+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f304c0ab860 con 0x7f304c0b8480 2023-10-14T18:17:39.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.852+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f304c04e9b0 con 0x7f304c0b8480 2023-10-14T18:17:39.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.854+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f304804d450 con 0x7f304c0b8480 2023-10-14T18:17:39.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.855+0000 7f3041ffb640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 0x7f303006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:39.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.855+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(229..229 src has 1..229) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f30480abae0 con 0x7f304c0b8480 2023-10-14T18:17:39.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.855+0000 7f3050964640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 0x7f303006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:39.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.856+0000 7f3050964640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 0x7f303006d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f304c0acf70 tx=0x7f303c008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:39.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:39.859+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3048076970 con 0x7f304c0b8480 2023-10-14T18:17:40.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.055+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f304c0ccbf0 con 0x7f304c0b8480 2023-10-14T18:17:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.287+0000 7f3041ffb640 1 -- 172.21.15.29:0/1872041768 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v230) v1 ==== 144+0+0 (secure 0 0 0) 0x7f3048079f90 con 0x7f304c0b8480 2023-10-14T18:17:40.288 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' created 2023-10-14T18:17:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 msgr2=0x7f303006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 0x7f303006d450 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f304c0acf70 tx=0x7f303c008040 comp rx=0 tx=0).stop 2023-10-14T18:17:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 msgr2=0x7f304c0ab320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 secure :-1 s=READY pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3048000c00 tx=0x7f30480046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 shutdown_connections 2023-10-14T18:17:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f303006af90 0x7f303006d450 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f304c15cd70 0x7f304c15f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f304c0cded0 0x7f304c1f2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 --2- 172.21.15.29:0/1872041768 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f304c0b8480 0x7f304c0ab320 secure :-1 s=CLOSED pgs=2616 cs=0 l=1 rev1=1 crypto rx=0x7f3048000c00 tx=0x7f30480046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.292+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 >> 172.21.15.29:0/1872041768 conn(0x7f304c0cf570 msgr2=0x7f304c15c040 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:40.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.293+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 shutdown_connections 2023-10-14T18:17:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.293+0000 7f3052167640 1 -- 172.21.15.29:0/1872041768 wait complete. 2023-10-14T18:17:40.320 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:17:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.534+0000 7fbe34d48640 1 Processor -- start 2023-10-14T18:17:40.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.534+0000 7fbe34d48640 1 -- start start 2023-10-14T18:17:40.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.535+0000 7fbe34d48640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe3015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.535+0000 7fbe34d48640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe300707e0 con 0x7fbe300ce390 2023-10-14T18:17:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe34d48640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe3006b7a0 con 0x7fbe3015aaa0 2023-10-14T18:17:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe34d48640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015ef20 con 0x7fbe3015ae80 2023-10-14T18:17:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe2ed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe300ce390 0x7fbe30070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40098/0 (socket says 172.21.15.29:40098) 2023-10-14T18:17:40.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe2ed9d640 1 -- 172.21.15.29:0/1843957352 learned_addr learned my addr 172.21.15.29:0/1843957352 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:40.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe3015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.536+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146172833 0 0) 0x7fbe300707e0 con 0x7fbe300ce390 2023-10-14T18:17:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe18003520 con 0x7fbe300ce390 2023-10-14T18:17:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065439562 0 0) 0x7fbe3006b7a0 con 0x7fbe3015aaa0 2023-10-14T18:17:40.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe300707e0 con 0x7fbe3015aaa0 2023-10-14T18:17:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 603551430 0 0) 0x7fbe18003520 con 0x7fbe300ce390 2023-10-14T18:17:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 msgr2=0x7fbe3015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe3015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 >> v1:172.21.15.186:6789/0 conn(0x7fbe3015aaa0 legacy=0x7fbe300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe3015f200 con 0x7fbe300ce390 2023-10-14T18:17:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fbe280019f0 con 0x7fbe300ce390 2023-10-14T18:17:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 >> v1:172.21.15.29:6789/0 conn(0x7fbe300ce390 legacy=0x7fbe30070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe18003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe180047c0 0x7fbe18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe180051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015ef20 con 0x7fbe18003b30 2023-10-14T18:17:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1800bb10 con 0x7fbe180047c0 2023-10-14T18:17:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1800bca0 con 0x7fbe3015ae80 2023-10-14T18:17:40.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.537+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe18003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe180047c0 0x7fbe18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2ed9d640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe180051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2e59c640 1 -- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 msgr2=0x7fbe180051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:40.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe3015ae80 0x7fbe180051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2e59c640 1 -- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe180047c0 msgr2=0x7fbe18004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe180047c0 0x7fbe18004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.538+0000 7fbe2e59c640 1 -- 172.21.15.29:0/1843957352 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18003520 con 0x7fbe18003b30 2023-10-14T18:17:40.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.539+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe18003f30 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fbe20001900 tx=0x7fbe180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8403342dcca2b09b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.539+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe200052d0 con 0x7fbe18003b30 2023-10-14T18:17:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.539+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe20005430 con 0x7fbe18003b30 2023-10-14T18:17:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe2d59a640 1 -- 172.21.15.29:0/1843957352 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20003810 con 0x7fbe18003b30 2023-10-14T18:17:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 -- 172.21.15.29:0/1843957352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 msgr2=0x7fbe18003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 --2- 172.21.15.29:0/1843957352 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe18003f30 secure :-1 s=READY pgs=2618 cs=0 l=1 rev1=1 crypto rx=0x7fbe20001900 tx=0x7fbe180057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe2ed9d640 1 -- 172.21.15.29:0/1843957352 reap_dead start 2023-10-14T18:17:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 -- 172.21.15.29:0/1843957352 shutdown_connections 2023-10-14T18:17:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 -- 172.21.15.29:0/1843957352 >> 172.21.15.29:0/1843957352 conn(0x7fbe300cb410 msgr2=0x7fbe300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 -- 172.21.15.29:0/1843957352 shutdown_connections 2023-10-14T18:17:40.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.540+0000 7fbe34d48640 1 -- 172.21.15.29:0/1843957352 wait complete. 2023-10-14T18:17:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.542+0000 7fbe34d48640 1 Processor -- start 2023-10-14T18:17:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.542+0000 7fbe34d48640 1 -- start start 2023-10-14T18:17:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 0x7fbe300c8250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe300c8790 0x7fbe3015c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3015ddd0 con 0x7fbe18003b30 2023-10-14T18:17:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe300aab70 con 0x7fbe300c8790 2023-10-14T18:17:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe34d48640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe3014c2e0 con 0x7fbe300c7dd0 2023-10-14T18:17:40.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe2e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.543+0000 7fbe2e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:55820/0 (socket says 172.21.15.29:55820) 2023-10-14T18:17:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 -- 172.21.15.29:0/3780004496 learned_addr learned my addr 172.21.15.29:0/3780004496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 0x7fbe300c8250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2ed9d640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe300c8790 0x7fbe3015c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 -- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 msgr2=0x7fbe300c8250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 0x7fbe300c8250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 -- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe300c8790 msgr2=0x7fbe3015c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe300c8790 0x7fbe3015c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2e59c640 1 -- 172.21.15.29:0/3780004496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe300c6ad0 con 0x7fbe18003b30 2023-10-14T18:17:40.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.544+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 0x7fbe300c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe2e59c640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fbe20035040 tx=0x7fbe200012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20007450 con 0x7fbe18003b30 2023-10-14T18:17:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe200075b0 con 0x7fbe18003b30 2023-10-14T18:17:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe300c6db0 con 0x7fbe18003b30 2023-10-14T18:17:40.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe20007720 con 0x7fbe18003b30 2023-10-14T18:17:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.545+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe300c7370 con 0x7fbe18003b30 2023-10-14T18:17:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.546+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe3004e160 con 0x7fbe18003b30 2023-10-14T18:17:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.548+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fbe200037f0 con 0x7fbe18003b30 2023-10-14T18:17:40.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.549+0000 7fbe1effd640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 0x7fbe1006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:40.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.549+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(230..230 src has 1..230) v4 ==== 6198+0+0 (secure 0 0 0) 0x7fbe200ac160 con 0x7fbe18003b30 2023-10-14T18:17:40.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.549+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 0x7fbe1006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:40.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.550+0000 7fbe2dd9b640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 0x7fbe1006d480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe30159270 tx=0x7fbe24008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:40.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.553+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe200770c0 con 0x7fbe18003b30 2023-10-14T18:17:40.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:40.751+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7fbe3015e350 con 0x7fbe18003b30 2023-10-14T18:17:41.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.306+0000 7fbe1effd640 1 -- 172.21.15.29:0/3780004496 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v231) v1 ==== 176+0+0 (secure 0 0 0) 0x7fbe2004b0d0 con 0x7fbe18003b30 2023-10-14T18:17:41.308 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' created 2023-10-14T18:17:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 msgr2=0x7fbe1006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 0x7fbe1006d480 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fbe30159270 tx=0x7fbe24008040 comp rx=0 tx=0).stop 2023-10-14T18:17:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 msgr2=0x7fbe300c7890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 secure :-1 s=READY pgs=2619 cs=0 l=1 rev1=1 crypto rx=0x7fbe20035040 tx=0x7fbe200012d0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 shutdown_connections 2023-10-14T18:17:41.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fbe1006afc0 0x7fbe1006d480 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe300c8790 0x7fbe3015c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe300c7dd0 0x7fbe300c8250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 --2- 172.21.15.29:0/3780004496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe18003b30 0x7fbe300c7890 unknown :-1 s=CLOSED pgs=2619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 >> 172.21.15.29:0/3780004496 conn(0x7fbe300cb410 msgr2=0x7fbe300b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 shutdown_connections 2023-10-14T18:17:41.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.310+0000 7fbe34d48640 1 -- 172.21.15.29:0/3780004496 wait complete. 2023-10-14T18:17:41.343 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-14T18:17:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.538+0000 7fea759a0640 1 Processor -- start 2023-10-14T18:17:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.538+0000 7fea759a0640 1 -- start start 2023-10-14T18:17:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.539+0000 7fea759a0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea700cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.539+0000 7fea759a0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7015b9d0 con 0x7fea7006c040 2023-10-14T18:17:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea759a0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea7015bcb0 con 0x7fea700ad970 2023-10-14T18:17:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea759a0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea7015b7d0 con 0x7fea7015b320 2023-10-14T18:17:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea7499e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fea7006c040 0x7fea7006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33906/0 (socket says 172.21.15.29:33906) 2023-10-14T18:17:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea7499e640 1 -- 172.21.15.29:0/969459144 learned_addr learned my addr 172.21.15.29:0/969459144 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea7519f640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea700cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.540+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2969892456 0 0) 0x7fea7015b9d0 con 0x7fea7006c040 2023-10-14T18:17:41.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea60003520 con 0x7fea7006c040 2023-10-14T18:17:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 26330115 0 0) 0x7fea60003520 con 0x7fea7006c040 2023-10-14T18:17:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 msgr2=0x7fea700cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea700cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 >> v1:172.21.15.186:6789/0 conn(0x7fea700ad970 legacy=0x7fea700cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea7015bf90 con 0x7fea7006c040 2023-10-14T18:17:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fea64001670 con 0x7fea7006c040 2023-10-14T18:17:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 >> v1:172.21.15.29:6789/0 conn(0x7fea7006c040 legacy=0x7fea7006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea600039a0 0x7fea60003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea60004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea60005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea7015b7d0 con 0x7fea60004350 2023-10-14T18:17:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000b9b0 con 0x7fea600039a0 2023-10-14T18:17:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea6000bb70 con 0x7fea7015b320 2023-10-14T18:17:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.541+0000 7fea6ffff640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea60004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea7499e640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea600039a0 0x7fea60003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea7519f640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea60005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea6ffff640 1 -- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 msgr2=0x7fea60005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea6ffff640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea7015b320 0x7fea60005080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea6ffff640 1 -- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea600039a0 msgr2=0x7fea60003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea6ffff640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea600039a0 0x7fea60003d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.542+0000 7fea6ffff640 1 -- 172.21.15.29:0/969459144 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea60003520 con 0x7fea60004350 2023-10-14T18:17:41.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.543+0000 7fea6ffff640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea60004820 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fea5c001830 tx=0x7fea600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30f0616f279531a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c02f050 con 0x7fea60004350 2023-10-14T18:17:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea5c033610 con 0x7fea60004350 2023-10-14T18:17:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea6f7fe640 1 -- 172.21.15.29:0/969459144 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea5c0339c0 con 0x7fea60004350 2023-10-14T18:17:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 -- 172.21.15.29:0/969459144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 msgr2=0x7fea60004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 --2- 172.21.15.29:0/969459144 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea60004820 secure :-1 s=READY pgs=2621 cs=0 l=1 rev1=1 crypto rx=0x7fea5c001830 tx=0x7fea600057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea7519f640 1 -- 172.21.15.29:0/969459144 reap_dead start 2023-10-14T18:17:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 -- 172.21.15.29:0/969459144 shutdown_connections 2023-10-14T18:17:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 -- 172.21.15.29:0/969459144 >> 172.21.15.29:0/969459144 conn(0x7fea700ce6a0 msgr2=0x7fea700ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:41.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 -- 172.21.15.29:0/969459144 shutdown_connections 2023-10-14T18:17:41.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.544+0000 7fea759a0640 1 -- 172.21.15.29:0/969459144 wait complete. 2023-10-14T18:17:41.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.546+0000 7fea759a0640 1 Processor -- start 2023-10-14T18:17:41.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.546+0000 7fea759a0640 1 -- start start 2023-10-14T18:17:41.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea70155f40 0x7fea700c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea700c21d0 0x7fea700c2650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700c2b90 con 0x7fea60004350 2023-10-14T18:17:41.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea700c6220 con 0x7fea70155f40 2023-10-14T18:17:41.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.547+0000 7fea759a0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea700c6500 con 0x7fea700c21d0 2023-10-14T18:17:41.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53402/0 (socket says 172.21.15.29:53402) 2023-10-14T18:17:41.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 -- 172.21.15.29:0/646366581 learned_addr learned my addr 172.21.15.29:0/646366581 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 -- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea700c21d0 msgr2=0x7fea700c2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea700c21d0 0x7fea700c2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 -- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea70155f40 msgr2=0x7fea700c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea70155f40 0x7fea700c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 -- 172.21.15.29:0/646366581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea700c67e0 con 0x7fea60004350 2023-10-14T18:17:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.548+0000 7fea7499e640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fea6400d770 tx=0x7fea6400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.549+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea640072d0 con 0x7fea60004350 2023-10-14T18:17:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.549+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea64007430 con 0x7fea60004350 2023-10-14T18:17:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.549+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea640047e0 con 0x7fea60004350 2023-10-14T18:17:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.549+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea700c6ac0 con 0x7fea60004350 2023-10-14T18:17:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.549+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea700c3510 con 0x7fea60004350 2023-10-14T18:17:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.550+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea7006b330 con 0x7fea60004350 2023-10-14T18:17:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.552+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fea64004940 con 0x7fea60004350 2023-10-14T18:17:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.553+0000 7fea6d7fa640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 0x7fea5406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:41.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.553+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(231..231 src has 1..231) v4 ==== 6583+0+0 (secure 0 0 0) 0x7fea64089830 con 0x7fea60004350 2023-10-14T18:17:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.553+0000 7fea6ffff640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 0x7fea5406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.554+0000 7fea6ffff640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 0x7fea5406d480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fea5c036000 tx=0x7fea5c009030 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:41.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.557+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fea64054330 con 0x7fea60004350 2023-10-14T18:17:41.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:41.789+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7fea7004e9b0 con 0x7fea60004350 2023-10-14T18:17:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.308+0000 7fea6d7fa640 1 -- 172.21.15.29:0/646366581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 28 and data pool 29 v109) v1 ==== 183+0+0 (secure 0 0 0) 0x7fea64057950 con 0x7fea60004350 2023-10-14T18:17:42.309 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 28 and data pool 29 2023-10-14T18:17:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 msgr2=0x7fea5406d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 0x7fea5406d480 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fea5c036000 tx=0x7fea5c009030 comp rx=0 tx=0).stop 2023-10-14T18:17:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 msgr2=0x7fea700c7990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 secure :-1 s=READY pgs=2622 cs=0 l=1 rev1=1 crypto rx=0x7fea6400d770 tx=0x7fea6400dc30 comp rx=0 tx=0).stop 2023-10-14T18:17:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 shutdown_connections 2023-10-14T18:17:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fea5406afc0 0x7fea5406d480 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fea700c21d0 0x7fea700c2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fea70155f40 0x7fea700c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 --2- 172.21.15.29:0/646366581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fea60004350 0x7fea700c7990 unknown :-1 s=CLOSED pgs=2622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.311+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 >> 172.21.15.29:0/646366581 conn(0x7fea700ce6a0 msgr2=0x7fea7015cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:42.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.312+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 shutdown_connections 2023-10-14T18:17:42.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.312+0000 7fea759a0640 1 -- 172.21.15.29:0/646366581 wait complete. 2023-10-14T18:17:42.340 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:42.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.525+0000 7fb264cbb640 1 Processor -- start 2023-10-14T18:17:42.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.525+0000 7fb264cbb640 1 -- start start 2023-10-14T18:17:42.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.526+0000 7fb264cbb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb26006c9e0 0x7fb2600ccfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb264cbb640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2600ca560 con 0x7fb2600c9060 2023-10-14T18:17:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb264cbb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2600c9950 con 0x7fb26006fb50 2023-10-14T18:17:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb264cbb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2600c9c30 con 0x7fb26006c9e0 2023-10-14T18:17:42.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb25f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb2600c9060 0x7fb26006f440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33920/0 (socket says 172.21.15.29:33920) 2023-10-14T18:17:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb25f7fe640 1 -- 172.21.15.29:0/1743879400 learned_addr learned my addr 172.21.15.29:0/1743879400 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693559976 0 0) 0x7fb2600ca560 con 0x7fb2600c9060 2023-10-14T18:17:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb248003520 con 0x7fb2600c9060 2023-10-14T18:17:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.527+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1743879400 >> v2:172.21.15.186:3300/0 conn(0x7fb26006c9e0 0x7fb2600ccfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 552856037 0 0) 0x7fb248003520 con 0x7fb2600c9060 2023-10-14T18:17:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 >> v2:172.21.15.186:3300/0 conn(0x7fb26006c9e0 msgr2=0x7fb2600ccfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 --2- 172.21.15.29:0/1743879400 >> v2:172.21.15.186:3300/0 conn(0x7fb26006c9e0 0x7fb2600ccfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 >> v1:172.21.15.186:6789/0 conn(0x7fb26006fb50 legacy=0x7fb2600cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2600c9f10 con 0x7fb2600c9060 2023-10-14T18:17:42.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fb250001670 con 0x7fb2600c9060 2023-10-14T18:17:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 >> v1:172.21.15.29:6789/0 conn(0x7fb2600c9060 legacy=0x7fb26006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248003970 0x7fb248003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600c9c30 con 0x7fb248005180 2023-10-14T18:17:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800c0d0 con 0x7fb248003970 2023-10-14T18:17:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb24800c290 con 0x7fb2480047c0 2023-10-14T18:17:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.528+0000 7fb25f7fe640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248003970 0x7fb248003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25effd640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25ffff640 1 -- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480047c0 msgr2=0x7fb248004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2480047c0 0x7fb248004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25ffff640 1 -- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248003970 msgr2=0x7fb248003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248003970 0x7fb248003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25ffff640 1 -- 172.21.15.29:0/1743879400 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb248003520 con 0x7fb248005180 2023-10-14T18:17:42.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.529+0000 7fb25f7fe640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb248003970 0x7fb248003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.530+0000 7fb25ffff640 1 -- 172.21.15.29:0/1743879400 reap_dead start 2023-10-14T18:17:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.530+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb24800b980 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fb258007330 tx=0x7fb2480057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=299f261ff884133e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.530+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25802ccd0 con 0x7fb248005180 2023-10-14T18:17:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.530+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25802ce30 con 0x7fb248005180 2023-10-14T18:17:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.530+0000 7fb25e7fc640 1 -- 172.21.15.29:0/1743879400 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb258003bd0 con 0x7fb248005180 2023-10-14T18:17:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 -- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 msgr2=0x7fb24800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb24800b980 secure :-1 s=READY pgs=2624 cs=0 l=1 rev1=1 crypto rx=0x7fb258007330 tx=0x7fb2480057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 -- 172.21.15.29:0/1743879400 shutdown_connections 2023-10-14T18:17:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1743879400 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb24800b980 unknown :-1 s=CLOSED pgs=2624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 -- 172.21.15.29:0/1743879400 >> 172.21.15.29:0/1743879400 conn(0x7fb2600ce6a0 msgr2=0x7fb2600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 -- 172.21.15.29:0/1743879400 shutdown_connections 2023-10-14T18:17:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.531+0000 7fb264cbb640 1 -- 172.21.15.29:0/1743879400 wait complete. 2023-10-14T18:17:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.533+0000 7fb264cbb640 1 Processor -- start 2023-10-14T18:17:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.533+0000 7fb264cbb640 1 -- start start 2023-10-14T18:17:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2600c5770 0x7fb26015e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2600c7e60 0x7fb2600c8310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600c8850 con 0x7fb248005180 2023-10-14T18:17:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600c6810 con 0x7fb2600c7e60 2023-10-14T18:17:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb264cbb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2600c5c00 con 0x7fb2600c5770 2023-10-14T18:17:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb25f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb25f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53414/0 (socket says 172.21.15.29:53414) 2023-10-14T18:17:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.534+0000 7fb25f7fe640 1 -- 172.21.15.29:0/1545521437 learned_addr learned my addr 172.21.15.29:0/1545521437 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25effd640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2600c5770 0x7fb26015e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25ffff640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2600c7e60 0x7fb2600c8310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 -- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2600c5770 msgr2=0x7fb26015e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2600c5770 0x7fb26015e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 -- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2600c7e60 msgr2=0x7fb2600c8310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2600c7e60 0x7fb2600c8310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 -- 172.21.15.29:0/1545521437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2600c5ee0 con 0x7fb248005180 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb25f7fe640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e7f0 tx=0x7fb25000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000c8d0 con 0x7fb248005180 2023-10-14T18:17:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25000ca30 con 0x7fb248005180 2023-10-14T18:17:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2600c61c0 con 0x7fb248005180 2023-10-14T18:17:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.535+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb25000cda0 con 0x7fb248005180 2023-10-14T18:17:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.536+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2600c6ca0 con 0x7fb248005180 2023-10-14T18:17:42.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.537+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26004e9b0 con 0x7fb248005180 2023-10-14T18:17:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.538+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb25001d460 con 0x7fb248005180 2023-10-14T18:17:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.539+0000 7fb247fff640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 0x7fb24c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.539+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fb250089b80 con 0x7fb248005180 2023-10-14T18:17:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.540+0000 7fb25effd640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 0x7fb24c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.540+0000 7fb25effd640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 0x7fb24c06d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb2601f9b50 tx=0x7fb254009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.544+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb250054740 con 0x7fb248005180 2023-10-14T18:17:42.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.732+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2600c15d0 con 0x7fb248005180 2023-10-14T18:17:42.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.735+0000 7fb247fff640 1 -- 172.21.15.29:0/1545521437 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20494 (secure 0 0 0) 0x7fb250057d60 con 0x7fb248005180 2023-10-14T18:17:42.737 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:42.737 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":232,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:41.790736+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 msgr2=0x7fb24c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 0x7fb24c06d450 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb2601f9b50 tx=0x7fb254009040 comp rx=0 tx=0).stop 2023-10-14T18:17:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 msgr2=0x7fb26015ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 secure :-1 s=READY pgs=2625 cs=0 l=1 rev1=1 crypto rx=0x7fb25000e7f0 tx=0x7fb25000ecb0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 shutdown_connections 2023-10-14T18:17:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fb24c06af90 0x7fb24c06d450 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2600c7e60 0x7fb2600c8310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2600c5770 0x7fb26015e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 --2- 172.21.15.29:0/1545521437 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb248005180 0x7fb26015ebc0 unknown :-1 s=CLOSED pgs=2625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.738+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 >> 172.21.15.29:0/1545521437 conn(0x7fb2600ce6a0 msgr2=0x7fb2601609e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.739+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 shutdown_connections 2023-10-14T18:17:42.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.739+0000 7fb264cbb640 1 -- 172.21.15.29:0/1545521437 wait complete. 2023-10-14T18:17:42.757 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:42.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.973+0000 7f1c924af640 1 Processor -- start 2023-10-14T18:17:42.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.973+0000 7f1c924af640 1 -- start start 2023-10-14T18:17:42.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 0x7f1c8c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c8c159f00 0x7f1c8c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c06f7f0 con 0x7f1c8c159f00 2023-10-14T18:17:42.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c8c06c5e0 con 0x7f1c8c159b20 2023-10-14T18:17:42.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c924af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c06c8c0 con 0x7f1c8c06c120 2023-10-14T18:17:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c8c159f00 0x7f1c8c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1c8c159f00 0x7f1c8c0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53418/0 (socket says 172.21.15.29:53418) 2023-10-14T18:17:42.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/747755536 learned_addr learned my addr 172.21.15.29:0/747755536 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c8bfff640 1 --2- 172.21.15.29:0/747755536 >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 0x7f1c8c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.975+0000 7f1c90a4c640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/747755536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 msgr2=0x7f1c8c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:42.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/747755536 >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 msgr2=0x7f1c8c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/747755536 >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 0x7f1c8c06bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/747755536 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c8c06cba0 con 0x7f1c8c159f00 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.976+0000 7f1c8bfff640 1 --2- 172.21.15.29:0/747755536 >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 0x7f1c8c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.977+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159f00 0x7f1c8c0b0410 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c9340 tx=0x7f1c8002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6087bf075dff7848 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.977+0000 7f1c8affd640 1 -- 172.21.15.29:0/747755536 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80031870 con 0x7f1c8c159f00 2023-10-14T18:17:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.977+0000 7f1c8affd640 1 -- 172.21.15.29:0/747755536 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c800319d0 con 0x7f1c8c159f00 2023-10-14T18:17:42.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.977+0000 7f1c8affd640 1 -- 172.21.15.29:0/747755536 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c80031d60 con 0x7f1c8c159f00 2023-10-14T18:17:42.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 -- 172.21.15.29:0/747755536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159f00 msgr2=0x7f1c8c0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159f00 0x7f1c8c0b0410 secure :-1 s=READY pgs=2626 cs=0 l=1 rev1=1 crypto rx=0x7f1c8c0c9340 tx=0x7f1c8002b910 comp rx=0 tx=0).stop 2023-10-14T18:17:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 -- 172.21.15.29:0/747755536 shutdown_connections 2023-10-14T18:17:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 --2- 172.21.15.29:0/747755536 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159f00 0x7f1c8c0b0410 unknown :-1 s=CLOSED pgs=2626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 --2- 172.21.15.29:0/747755536 >> v2:172.21.15.186:3300/0 conn(0x7f1c8c159b20 0x7f1c8c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 -- 172.21.15.29:0/747755536 >> 172.21.15.29:0/747755536 conn(0x7f1c8c0cb410 msgr2=0x7f1c8c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 -- 172.21.15.29:0/747755536 shutdown_connections 2023-10-14T18:17:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.978+0000 7f1c924af640 1 -- 172.21.15.29:0/747755536 wait complete. 2023-10-14T18:17:42.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.980+0000 7f1c924af640 1 Processor -- start 2023-10-14T18:17:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.980+0000 7f1c924af640 1 -- start start 2023-10-14T18:17:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c154460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c8c159f00 0x7f1c8c1f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0b24f0 con 0x7f1c8c159b20 2023-10-14T18:17:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c5e40 con 0x7f1c8c159f00 2023-10-14T18:17:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c924af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c8c0c6120 con 0x7f1c8c06c120 2023-10-14T18:17:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.981+0000 7f1c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53434/0 (socket says 172.21.15.29:53434) 2023-10-14T18:17:42.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/935409102 learned_addr learned my addr 172.21.15.29:0/935409102 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 msgr2=0x7f1c8c154460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c154460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c8c159f00 msgr2=0x7f1c8c1f2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c8c159f00 0x7f1c8c1f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:42.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 -- 172.21.15.29:0/935409102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c800094d0 con 0x7f1c8c159b20 2023-10-14T18:17:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.982+0000 7f1c8b7fe640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f1c8002b950 tx=0x7f1c800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.983+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c8003ad30 con 0x7f1c8c159b20 2023-10-14T18:17:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.983+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c800443f0 con 0x7f1c8c159b20 2023-10-14T18:17:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.983+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1c800447a0 con 0x7f1c8c159b20 2023-10-14T18:17:42.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.983+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c8c0c6400 con 0x7f1c8c159b20 2023-10-14T18:17:42.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.983+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c8c0c5a60 con 0x7f1c8c159b20 2023-10-14T18:17:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.984+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c8c04e160 con 0x7f1c8c159b20 2023-10-14T18:17:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.985+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1c8004d450 con 0x7f1c8c159b20 2023-10-14T18:17:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.986+0000 7f1c88ff9640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 0x7f1c7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:42.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.986+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(232..232 src has 1..232) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f1c8002d040 con 0x7f1c8c159b20 2023-10-14T18:17:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.986+0000 7f1c8bfff640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 0x7f1c7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.987+0000 7f1c8bfff640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 0x7f1c7806d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0096f0 tx=0x7f1c7c008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:42.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:42.990+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1c80076d40 con 0x7f1c8c159b20 2023-10-14T18:17:43.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.178+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c8c0c9610 con 0x7f1c8c159b20 2023-10-14T18:17:43.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.181+0000 7f1c88ff9640 1 -- 172.21.15.29:0/935409102 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v232) v1 ==== 74+0+20494 (secure 0 0 0) 0x7f1c80035090 con 0x7f1c8c159b20 2023-10-14T18:17:43.183 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:43.183 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":232,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:41.790736+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 msgr2=0x7f1c7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 0x7f1c7806d450 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1c7c0096f0 tx=0x7f1c7c008040 comp rx=0 tx=0).stop 2023-10-14T18:17:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 msgr2=0x7f1c8c0b3dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 secure :-1 s=READY pgs=2627 cs=0 l=1 rev1=1 crypto rx=0x7f1c8002b950 tx=0x7f1c800046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 shutdown_connections 2023-10-14T18:17:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f1c7806af90 0x7f1c7806d450 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1c8c159f00 0x7f1c8c1f2800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1c8c159b20 0x7f1c8c0b3dc0 unknown :-1 s=CLOSED pgs=2627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 --2- 172.21.15.29:0/935409102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1c8c06c120 0x7f1c8c154460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 >> 172.21.15.29:0/935409102 conn(0x7f1c8c0cb410 msgr2=0x7f1c8c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 shutdown_connections 2023-10-14T18:17:43.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.184+0000 7f1c924af640 1 -- 172.21.15.29:0/935409102 wait complete. 2023-10-14T18:17:43.203 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-14T18:17:43.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.424+0000 7f3f51916640 1 Processor -- start 2023-10-14T18:17:43.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.424+0000 7f3f51916640 1 -- start start 2023-10-14T18:17:43.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.425+0000 7f3f51916640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3f4c06a4b0 0x7f3f4c0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.425+0000 7f3f51916640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c0cb030 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.425+0000 7f3f51916640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f4c0c91c0 con 0x7f3f4c15ad70 2023-10-14T18:17:43.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.425+0000 7f3f51916640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4c0c94a0 con 0x7f3f4c06a4b0 2023-10-14T18:17:43.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.425+0000 7f3f4b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3f4c0c9ca0 0x7f3f4c0ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33934/0 (socket says 172.21.15.29:33934) 2023-10-14T18:17:43.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3986757418 learned_addr learned my addr 172.21.15.29:0/3986757418 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622946615 0 0) 0x7f3f4c0cb030 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f38003520 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f4a7fc640 1 --2- 172.21.15.29:0/3986757418 >> v2:172.21.15.186:3300/0 conn(0x7f3f4c06a4b0 0x7f3f4c0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3546303014 0 0) 0x7f3f38003520 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 >> v2:172.21.15.186:3300/0 conn(0x7f3f4c06a4b0 msgr2=0x7f3f4c0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 --2- 172.21.15.29:0/3986757418 >> v2:172.21.15.186:3300/0 conn(0x7f3f4c06a4b0 0x7f3f4c0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 >> v1:172.21.15.186:6789/0 conn(0x7f3f4c15ad70 legacy=0x7f3f4c069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0c9780 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3f440019f0 con 0x7f3f4c0c9ca0 2023-10-14T18:17:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 >> v1:172.21.15.29:6789/0 conn(0x7f3f4c0c9ca0 legacy=0x7f3f4c0ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.426+0000 7f3f49ffb640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38003970 0x7f3f38003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f49ffb640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f380047c0 0x7f3f38004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f49ffb640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 0x7f3f3800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0c94a0 con 0x7f3f38005180 2023-10-14T18:17:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3800c0d0 con 0x7f3f380047c0 2023-10-14T18:17:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f3800c290 con 0x7f3f38003970 2023-10-14T18:17:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 0x7f3f3800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4affd640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38003970 0x7f3f38003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4a7fc640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f380047c0 0x7f3f38004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38003970 msgr2=0x7f3f38003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38003970 0x7f3f38003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f380047c0 msgr2=0x7f3f38004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f380047c0 0x7f3f38004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.427+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3986757418 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f38003520 con 0x7f3f38005180 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.428+0000 7f3f4b7fe640 1 -- 172.21.15.29:0/3986757418 reap_dead start 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.428+0000 7f3f4b7fe640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 0x7f3f3800b980 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400a830 tx=0x7f3f380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2470b9a8b7978ce2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440072d0 con 0x7f3f38005180 2023-10-14T18:17:43.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f4402f070 con 0x7f3f38005180 2023-10-14T18:17:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f49ffb640 1 -- 172.21.15.29:0/3986757418 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f440076b0 con 0x7f3f38005180 2023-10-14T18:17:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 -- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 msgr2=0x7f3f3800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 0x7f3f3800b980 secure :-1 s=READY pgs=2629 cs=0 l=1 rev1=1 crypto rx=0x7f3f4400a830 tx=0x7f3f380057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 -- 172.21.15.29:0/3986757418 shutdown_connections 2023-10-14T18:17:43.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 --2- 172.21.15.29:0/3986757418 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f38005180 0x7f3f3800b980 unknown :-1 s=CLOSED pgs=2629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 -- 172.21.15.29:0/3986757418 >> 172.21.15.29:0/3986757418 conn(0x7f3f4c0ce9b0 msgr2=0x7f3f4c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 -- 172.21.15.29:0/3986757418 shutdown_connections 2023-10-14T18:17:43.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.429+0000 7f3f51916640 1 -- 172.21.15.29:0/3986757418 wait complete. 2023-10-14T18:17:43.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.431+0000 7f3f51916640 1 Processor -- start 2023-10-14T18:17:43.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.431+0000 7f3f51916640 1 -- start start 2023-10-14T18:17:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38005180 0x7f3f4c158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c0c5a80 0x7f3f4c0c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0c6440 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c1594c0 con 0x7f3f4c0c5a80 2023-10-14T18:17:43.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f51916640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4c0c7340 con 0x7f3f38005180 2023-10-14T18:17:43.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.432+0000 7f3f4a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53452/0 (socket says 172.21.15.29:53452) 2023-10-14T18:17:43.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 -- 172.21.15.29:0/2061354862 learned_addr learned my addr 172.21.15.29:0/2061354862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:43.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 -- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38005180 msgr2=0x7f3f4c158f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38005180 0x7f3f4c158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 -- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c0c5a80 msgr2=0x7f3f4c0c5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c0c5a80 0x7f3f4c0c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 -- 172.21.15.29:0/2061354862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4c0c7620 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f4a7fc640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400f580 tx=0x7f3f3400fa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.433+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34013810 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.434+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f34013970 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.434+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4c0c7900 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.434+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f34013ce0 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.434+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4c0c7ec0 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.435+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4c04e160 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.436+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3f3401f460 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.437+0000 7f3f43fff640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.437+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f3f34089f20 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.437+0000 7f3f4affd640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.438+0000 7f3f4affd640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 0x7f3f2c06d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c007fd0 tx=0x7f3f3c009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:43.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.441+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3f34054a50 con 0x7f3f4c0b2c10 2023-10-14T18:17:43.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:43.688+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f3f4c15ca90 con 0x7f3f4c0b2c10 2023-10-14T18:17:44.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.336+0000 7f3f43fff640 1 -- 172.21.15.29:0/2061354862 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v112) v1 ==== 121+0+0 (secure 0 0 0) 0x7f3f34058070 con 0x7f3f4c0b2c10 2023-10-14T18:17:44.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 msgr2=0x7f3f2c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:44.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 0x7f3f2c06d450 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3f3c007fd0 tx=0x7f3f3c009040 comp rx=0 tx=0).stop 2023-10-14T18:17:44.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 msgr2=0x7f3f4c0b3090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:44.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 secure :-1 s=READY pgs=2630 cs=0 l=1 rev1=1 crypto rx=0x7f3f3400f580 tx=0x7f3f3400fa40 comp rx=0 tx=0).stop 2023-10-14T18:17:44.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 shutdown_connections 2023-10-14T18:17:44.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3f2c06af90 0x7f3f2c06d450 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3f4c0c5a80 0x7f3f4c0c5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3f4c0b2c10 0x7f3f4c0b3090 unknown :-1 s=CLOSED pgs=2630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 --2- 172.21.15.29:0/2061354862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3f38005180 0x7f3f4c158f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 >> 172.21.15.29:0/2061354862 conn(0x7f3f4c0ce9b0 msgr2=0x7f3f4c0ca150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:44.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.341+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 shutdown_connections 2023-10-14T18:17:44.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.342+0000 7f3f51916640 1 -- 172.21.15.29:0/2061354862 wait complete. 2023-10-14T18:17:44.365 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:17:44.366 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-14T18:17:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.598+0000 7f47d951d640 1 Processor -- start 2023-10-14T18:17:44.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.598+0000 7f47d951d640 1 -- start start 2023-10-14T18:17:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47d951d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 0x7f47d4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47d951d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47d40c9ca0 0x7f47d406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47d951d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47d40cb030 con 0x7f47d40ca3b0 2023-10-14T18:17:44.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47d951d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40c91c0 con 0x7f47d40c9ca0 2023-10-14T18:17:44.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47d951d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47d40c94a0 con 0x7f47d415ad70 2023-10-14T18:17:44.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47ca7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47d40c9ca0 0x7f47d406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47ca7fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47d40c9ca0 0x7f47d406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53460/0 (socket says 172.21.15.29:53460) 2023-10-14T18:17:44.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.600+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1805067027 learned_addr learned my addr 172.21.15.29:0/1805067027 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47d2ffd640 1 --2- 172.21.15.29:0/1805067027 >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 0x7f47d4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1805067027 >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 msgr2=0x7f47d4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47ca7fc640 1 --2- 172.21.15.29:0/1805067027 >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 0x7f47d4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1805067027 >> v1:172.21.15.186:6789/0 conn(0x7f47d40ca3b0 legacy=0x7f47d40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:44.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1805067027 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47d40c9780 con 0x7f47d40c9ca0 2023-10-14T18:17:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.601+0000 7f47d2ffd640 1 --2- 172.21.15.29:0/1805067027 >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 0x7f47d4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.602+0000 7f47ca7fc640 1 --2- 172.21.15.29:0/1805067027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40c9ca0 0x7f47d406a2e0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f47d40cf780 tx=0x7f47bc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=155dc542a90c644a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.603+0000 7f47d27fc640 1 -- 172.21.15.29:0/1805067027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc031870 con 0x7f47d40c9ca0 2023-10-14T18:17:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.603+0000 7f47d27fc640 1 -- 172.21.15.29:0/1805067027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc0319d0 con 0x7f47d40c9ca0 2023-10-14T18:17:44.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.603+0000 7f47d27fc640 1 -- 172.21.15.29:0/1805067027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc031d60 con 0x7f47d40c9ca0 2023-10-14T18:17:44.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.603+0000 7f47d951d640 1 -- 172.21.15.29:0/1805067027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40c9ca0 msgr2=0x7f47d406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:44.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.603+0000 7f47d951d640 1 --2- 172.21.15.29:0/1805067027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40c9ca0 0x7f47d406a2e0 secure :-1 s=READY pgs=2631 cs=0 l=1 rev1=1 crypto rx=0x7f47d40cf780 tx=0x7f47bc02b910 comp rx=0 tx=0).stop 2023-10-14T18:17:44.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 -- 172.21.15.29:0/1805067027 shutdown_connections 2023-10-14T18:17:44.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 --2- 172.21.15.29:0/1805067027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40c9ca0 0x7f47d406a2e0 unknown :-1 s=CLOSED pgs=2631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 --2- 172.21.15.29:0/1805067027 >> v2:172.21.15.186:3300/0 conn(0x7f47d415ad70 0x7f47d4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 -- 172.21.15.29:0/1805067027 >> 172.21.15.29:0/1805067027 conn(0x7f47d40ce9b0 msgr2=0x7f47d40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:44.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 -- 172.21.15.29:0/1805067027 shutdown_connections 2023-10-14T18:17:44.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.604+0000 7f47d951d640 1 -- 172.21.15.29:0/1805067027 wait complete. 2023-10-14T18:17:44.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.605+0000 7f47d951d640 1 Processor -- start 2023-10-14T18:17:44.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.606+0000 7f47d951d640 1 -- start start 2023-10-14T18:17:44.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d40c9ca0 0x7f47d40c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 0x7f47d415d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d41f2580 con 0x7f47d40ca3b0 2023-10-14T18:17:44.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47d40c6e70 con 0x7f47d40c9ca0 2023-10-14T18:17:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d951d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47d40c7150 con 0x7f47d415ad70 2023-10-14T18:17:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53466/0 (socket says 172.21.15.29:53466) 2023-10-14T18:17:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1687536106 learned_addr learned my addr 172.21.15.29:0/1687536106 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:44.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d37fe640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 0x7f47d415d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 msgr2=0x7f47d415d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:44.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 0x7f47d415d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d40c9ca0 msgr2=0x7f47d40c44b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:44.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d2ffd640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d40c9ca0 0x7f47d40c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d40c9ca0 0x7f47d40c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47ca7fc640 1 -- 172.21.15.29:0/1687536106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47bc0094d0 con 0x7f47d40ca3b0 2023-10-14T18:17:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.607+0000 7f47d37fe640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 0x7f47d415d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47ca7fc640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f47bc000c00 tx=0x7f47bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:44.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc031840 con 0x7f47d40ca3b0 2023-10-14T18:17:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47bc0443f0 con 0x7f47d40ca3b0 2023-10-14T18:17:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47bc0447a0 con 0x7f47d40ca3b0 2023-10-14T18:17:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47d40c7430 con 0x7f47d40ca3b0 2023-10-14T18:17:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.608+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47d40c49f0 con 0x7f47d40ca3b0 2023-10-14T18:17:44.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.610+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47d41f2730 con 0x7f47d40ca3b0 2023-10-14T18:17:44.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.611+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f47bc04d450 con 0x7f47d40ca3b0 2023-10-14T18:17:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.611+0000 7f47cbfff640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 0x7f47b406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.612+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f47bc02d040 con 0x7f47d40ca3b0 2023-10-14T18:17:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.612+0000 7f47d2ffd640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 0x7f47b406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.613+0000 7f47d2ffd640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 0x7f47b406d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b840 tx=0x7f47c4008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:44.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.616+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f47d41f2730 con 0x7f47d40ca3b0 2023-10-14T18:17:44.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:44.856+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f47d404e160 con 0x7f47d40ca3b0 2023-10-14T18:17:45.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.340+0000 7f47cbfff640 1 -- 172.21.15.29:0/1687536106 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v113) v1 ==== 125+0+0 (secure 0 0 0) 0x7f47bc076f10 con 0x7f47d40ca3b0 2023-10-14T18:17:45.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 msgr2=0x7f47b406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 0x7f47b406d450 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f47c400b840 tx=0x7f47c4008040 comp rx=0 tx=0).stop 2023-10-14T18:17:45.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 msgr2=0x7f47d40c6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 secure :-1 s=READY pgs=2632 cs=0 l=1 rev1=1 crypto rx=0x7f47bc000c00 tx=0x7f47bc0046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 shutdown_connections 2023-10-14T18:17:45.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f47b406af90 0x7f47b406d450 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f47d415ad70 0x7f47d415d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47d40ca3b0 0x7f47d40c6460 unknown :-1 s=CLOSED pgs=2632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 --2- 172.21.15.29:0/1687536106 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47d40c9ca0 0x7f47d40c44b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 >> 172.21.15.29:0/1687536106 conn(0x7f47d40ce9b0 msgr2=0x7f47d4070d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:45.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 shutdown_connections 2023-10-14T18:17:45.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.345+0000 7f47d951d640 1 -- 172.21.15.29:0/1687536106 wait complete. 2023-10-14T18:17:45.370 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-14T18:17:45.370 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:45.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.598+0000 7fd15e2ff640 1 Processor -- start 2023-10-14T18:17:45.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.598+0000 7fd15e2ff640 1 -- start start 2023-10-14T18:17:45.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15e2ff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd15806c040 0x7fd15806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15e2ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd15815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15e2ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd15806c960 con 0x7fd15815ebc0 2023-10-14T18:17:45.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15e2ff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd15815b9d0 con 0x7fd15806c040 2023-10-14T18:17:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15e2ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd15815bcb0 con 0x7fd1580ad970 2023-10-14T18:17:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15dafe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd15815ebc0 0x7fd1580cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33944/0 (socket says 172.21.15.29:33944) 2023-10-14T18:17:45.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.600+0000 7fd15dafe640 1 -- 172.21.15.29:0/508084769 learned_addr learned my addr 172.21.15.29:0/508084769 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd15cafc640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd15815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd15d2fd640 1 --2- 172.21.15.29:0/508084769 >> v2:172.21.15.186:3300/0 conn(0x7fd15806c040 0x7fd15806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617459348 0 0) 0x7fd15806c960 con 0x7fd15815ebc0 2023-10-14T18:17:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd144003520 con 0x7fd15815ebc0 2023-10-14T18:17:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2859762872 0 0) 0x7fd144003520 con 0x7fd15815ebc0 2023-10-14T18:17:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 msgr2=0x7fd15815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:45.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd15815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 >> v2:172.21.15.186:3300/0 conn(0x7fd15806c040 msgr2=0x7fd15806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 --2- 172.21.15.29:0/508084769 >> v2:172.21.15.186:3300/0 conn(0x7fd15806c040 0x7fd15806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd15815bf90 con 0x7fd15815ebc0 2023-10-14T18:17:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd1500019f0 con 0x7fd15815ebc0 2023-10-14T18:17:45.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 >> v1:172.21.15.29:6789/0 conn(0x7fd15815ebc0 legacy=0x7fd1580cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd144003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1440047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004d00 0x7fd144005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15815b9d0 con 0x7fd144003f40 2023-10-14T18:17:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15815bcb0 con 0x7fd144004d00 2023-10-14T18:17:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.601+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd14400ba30 con 0x7fd1580ad970 2023-10-14T18:17:45.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1440047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15d2fd640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd144003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 -- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 msgr2=0x7fd144003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd1580ad970 0x7fd144003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 -- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004d00 msgr2=0x7fd144005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:45.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd144004d00 0x7fd144005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.602+0000 7fd15cafc640 1 -- 172.21.15.29:0/508084769 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd144003520 con 0x7fd144003f40 2023-10-14T18:17:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.603+0000 7fd15cafc640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1440047c0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fd148009ac0 tx=0x7fd1440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a830ad3186c6d05 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:45.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.603+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14802c6b0 con 0x7fd144003f40 2023-10-14T18:17:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.603+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14802c810 con 0x7fd144003f40 2023-10-14T18:17:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.603+0000 7fd14ffff640 1 -- 172.21.15.29:0/508084769 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd14802cba0 con 0x7fd144003f40 2023-10-14T18:17:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 -- 172.21.15.29:0/508084769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 msgr2=0x7fd1440047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/508084769 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1440047c0 secure :-1 s=READY pgs=2634 cs=0 l=1 rev1=1 crypto rx=0x7fd148009ac0 tx=0x7fd1440057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15dafe640 1 -- 172.21.15.29:0/508084769 reap_dead start 2023-10-14T18:17:45.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 -- 172.21.15.29:0/508084769 shutdown_connections 2023-10-14T18:17:45.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 -- 172.21.15.29:0/508084769 >> 172.21.15.29:0/508084769 conn(0x7fd1580ce6a0 msgr2=0x7fd1580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:45.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 -- 172.21.15.29:0/508084769 shutdown_connections 2023-10-14T18:17:45.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.604+0000 7fd15e2ff640 1 -- 172.21.15.29:0/508084769 wait complete. 2023-10-14T18:17:45.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.606+0000 7fd15e2ff640 1 Processor -- start 2023-10-14T18:17:45.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.606+0000 7fd15e2ff640 1 -- start start 2023-10-14T18:17:45.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd158156fe0 0x7fd1580c7570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd158069da0 0x7fd15806a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd15806a760 con 0x7fd144003f40 2023-10-14T18:17:45.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd158157460 con 0x7fd158156fe0 2023-10-14T18:17:45.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.607+0000 7fd15e2ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1581f5a50 con 0x7fd158069da0 2023-10-14T18:17:45.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53484/0 (socket says 172.21.15.29:53484) 2023-10-14T18:17:45.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 -- 172.21.15.29:0/2300864210 learned_addr learned my addr 172.21.15.29:0/2300864210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 -- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd158069da0 msgr2=0x7fd15806a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd158069da0 0x7fd15806a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 -- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd158156fe0 msgr2=0x7fd1580c7570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd158156fe0 0x7fd1580c7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 -- 172.21.15.29:0/2300864210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1581f5d30 con 0x7fd144003f40 2023-10-14T18:17:45.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd15d2fd640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fd15400f600 tx=0x7fd15400fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:45.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.608+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154014870 con 0x7fd144003f40 2023-10-14T18:17:45.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.609+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1540149d0 con 0x7fd144003f40 2023-10-14T18:17:45.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.609+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154014d40 con 0x7fd144003f40 2023-10-14T18:17:45.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.609+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1581f6010 con 0x7fd144003f40 2023-10-14T18:17:45.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.609+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1581f65d0 con 0x7fd144003f40 2023-10-14T18:17:45.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.610+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd15804e9b0 con 0x7fd144003f40 2023-10-14T18:17:45.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.611+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd15400d490 con 0x7fd144003f40 2023-10-14T18:17:45.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.612+0000 7fd14dffb640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 0x7fd13c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:45.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.612+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fd15408a690 con 0x7fd144003f40 2023-10-14T18:17:45.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.612+0000 7fd15cafc640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 0x7fd13c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:45.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.613+0000 7fd15cafc640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 0x7fd13c06d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd148036000 tx=0x7fd14802e000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:45.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.616+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd1540551b0 con 0x7fd144003f40 2023-10-14T18:17:45.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.851+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd15806b2b0 con 0x7fd144003f40 2023-10-14T18:17:45.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.852+0000 7fd14dffb640 1 -- 172.21.15.29:0/2300864210 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7fd1540587d0 con 0x7fd144003f40 2023-10-14T18:17:45.854 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:45.854 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:17:45.854 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:17:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 msgr2=0x7fd13c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 0x7fd13c06d450 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd148036000 tx=0x7fd14802e000 comp rx=0 tx=0).stop 2023-10-14T18:17:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 msgr2=0x7fd1580c5770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 secure :-1 s=READY pgs=2635 cs=0 l=1 rev1=1 crypto rx=0x7fd15400f600 tx=0x7fd15400fac0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 shutdown_connections 2023-10-14T18:17:45.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd13c06af90 0x7fd13c06d450 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd158069da0 0x7fd15806a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd158156fe0 0x7fd1580c7570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 --2- 172.21.15.29:0/2300864210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd144003f40 0x7fd1580c5770 unknown :-1 s=CLOSED pgs=2635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:45.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.855+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 >> 172.21.15.29:0/2300864210 conn(0x7fd1580ce6a0 msgr2=0x7fd15815dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.856+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 shutdown_connections 2023-10-14T18:17:45.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:45.856+0000 7fd15e2ff640 1 -- 172.21.15.29:0/2300864210 wait complete. 2023-10-14T18:17:45.875 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:17:46.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.100+0000 7f6602447640 1 Processor -- start 2023-10-14T18:17:46.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.100+0000 7f6602447640 1 -- start start 2023-10-14T18:17:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6602447640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65fc15dde0 0x7f65fc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6602447640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6602447640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65fc160a90 con 0x7f65fc15c9f0 2023-10-14T18:17:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6602447640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65fc0cdec0 con 0x7f65fc15dde0 2023-10-14T18:17:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6602447640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0cd900 con 0x7f65fc070f20 2023-10-14T18:17:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6600c44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65fc15dde0 0x7f65fc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6600c44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f65fc15dde0 0x7f65fc0709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53494/0 (socket says 172.21.15.29:53494) 2023-10-14T18:17:46.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6600c44640 1 -- 172.21.15.29:0/2388866684 learned_addr learned my addr 172.21.15.29:0/2388866684 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:46.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.102+0000 7f6601c46640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.103+0000 7f6600c44640 1 -- 172.21.15.29:0/2388866684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 msgr2=0x7f65fc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:46.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.103+0000 7f6600c44640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.103+0000 7f6600c44640 1 -- 172.21.15.29:0/2388866684 >> v1:172.21.15.186:6789/0 conn(0x7f65fc15c9f0 legacy=0x7f65fc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:46.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.103+0000 7f6600c44640 1 -- 172.21.15.29:0/2388866684 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65fc0cdbe0 con 0x7f65fc15dde0 2023-10-14T18:17:46.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.104+0000 7f6600c44640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc0709e0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5168662574741171 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.104+0000 7f65f3fff640 1 -- 172.21.15.29:0/2388866684 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec031640 con 0x7f65fc15dde0 2023-10-14T18:17:46.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.104+0000 7f65f3fff640 1 -- 172.21.15.29:0/2388866684 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65ec0317a0 con 0x7f65fc15dde0 2023-10-14T18:17:46.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.104+0000 7f65f3fff640 1 -- 172.21.15.29:0/2388866684 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65ec031b30 con 0x7f65fc15dde0 2023-10-14T18:17:46.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 -- 172.21.15.29:0/2388866684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 msgr2=0x7f65fc0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc0709e0 secure :-1 s=READY pgs=2636 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009870 tx=0x7f65ec02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 -- 172.21.15.29:0/2388866684 shutdown_connections 2023-10-14T18:17:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 --2- 172.21.15.29:0/2388866684 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc0709e0 unknown :-1 s=CLOSED pgs=2636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 -- 172.21.15.29:0/2388866684 >> 172.21.15.29:0/2388866684 conn(0x7f65fc0ce6a0 msgr2=0x7f65fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 -- 172.21.15.29:0/2388866684 shutdown_connections 2023-10-14T18:17:46.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.105+0000 7f6602447640 1 -- 172.21.15.29:0/2388866684 wait complete. 2023-10-14T18:17:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.107+0000 7f6602447640 1 Processor -- start 2023-10-14T18:17:46.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.107+0000 7f6602447640 1 -- start start 2023-10-14T18:17:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc15c9f0 0x7f65fc0c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0c5800 con 0x7f65fc15dde0 2023-10-14T18:17:46.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc1513c0 con 0x7f65fc15c9f0 2023-10-14T18:17:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6602447640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65fc0c3430 con 0x7f65fc070f20 2023-10-14T18:17:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6601c46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6601c46640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53510/0 (socket says 172.21.15.29:53510) 2023-10-14T18:17:46.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6601c46640 1 -- 172.21.15.29:0/334069802 learned_addr learned my addr 172.21.15.29:0/334069802 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.108+0000 7f6601445640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 -- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 msgr2=0x7f65fc150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 -- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc15c9f0 msgr2=0x7f65fc0c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc15c9f0 0x7f65fc0c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 -- 172.21.15.29:0/334069802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65ec0094d0 con 0x7f65fc15dde0 2023-10-14T18:17:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6601c46640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f65f400d830 tx=0x7f65f400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400b6c0 con 0x7f65fc15dde0 2023-10-14T18:17:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65f400b840 con 0x7f65fc15dde0 2023-10-14T18:17:46.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65f400bbf0 con 0x7f65fc15dde0 2023-10-14T18:17:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.109+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65fc0c3710 con 0x7f65fc15dde0 2023-10-14T18:17:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.110+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65fc0c3cd0 con 0x7f65fc15dde0 2023-10-14T18:17:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.111+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65fc160d70 con 0x7f65fc15dde0 2023-10-14T18:17:46.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.112+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f65f401e460 con 0x7f65fc15dde0 2023-10-14T18:17:46.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.113+0000 7f65f1ffb640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 0x7f65e006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.114+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f65f408a3c0 con 0x7f65fc15dde0 2023-10-14T18:17:46.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.114+0000 7f6601445640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 0x7f65e006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.114+0000 7f6601445640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 0x7f65e006d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0aeb40 tx=0x7f65f8009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.118+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f65f4054ee0 con 0x7f65fc15dde0 2023-10-14T18:17:46.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.316+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65fc04e9b0 con 0x7f65fc15dde0 2023-10-14T18:17:46.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.321+0000 7f65f1ffb640 1 -- 172.21.15.29:0/334069802 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v233) v1 ==== 74+0+20481 (secure 0 0 0) 0x7f65f4058500 con 0x7f65fc15dde0 2023-10-14T18:17:46.322 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:46.323 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":233,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:43.224289+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:17:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 msgr2=0x7f65e006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 0x7f65e006d450 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f65fc0aeb40 tx=0x7f65f8009040 comp rx=0 tx=0).stop 2023-10-14T18:17:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 msgr2=0x7f65fc15f2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 secure :-1 s=READY pgs=2637 cs=0 l=1 rev1=1 crypto rx=0x7f65f400d830 tx=0x7f65f400dcf0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 shutdown_connections 2023-10-14T18:17:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f65e006af90 0x7f65e006d450 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f65fc15dde0 0x7f65fc15f2d0 unknown :-1 s=CLOSED pgs=2637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f65fc15c9f0 0x7f65fc0c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 --2- 172.21.15.29:0/334069802 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f65fc070f20 0x7f65fc150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 >> 172.21.15.29:0/334069802 conn(0x7f65fc0ce6a0 msgr2=0x7f65fc0aef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 shutdown_connections 2023-10-14T18:17:46.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.324+0000 7f6602447640 1 -- 172.21.15.29:0/334069802 wait complete. 2023-10-14T18:17:46.345 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:17:46.346 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:17:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.569+0000 7f3b8f59e640 1 Processor -- start 2023-10-14T18:17:46.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.569+0000 7f3b8f59e640 1 -- start start 2023-10-14T18:17:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.570+0000 7f3b8f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3b9015cc80 0x7f3b900cd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.570+0000 7f3b8f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 0x7f3b90071a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.570+0000 7f3b8f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b900ce1c0 con 0x7f3b90071fa0 2023-10-14T18:17:46.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.570+0000 7f3b8f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b900d0500 con 0x7f3b9015cc80 2023-10-14T18:17:46.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.570+0000 7f3b8f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900d12f0 con 0x7f3b900cdd60 2023-10-14T18:17:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.571+0000 7f3b8ed9d640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f3b90071fa0 0x7f3b901607e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:40412/0 (socket says 172.21.15.29:40412) 2023-10-14T18:17:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.571+0000 7f3b8ed9d640 1 -- 172.21.15.29:0/1507501043 learned_addr learned my addr 172.21.15.29:0/1507501043 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:46.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.571+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1507501043 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 0x7f3b90071a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.571+0000 7f3b8e59c640 1 --2- 172.21.15.29:0/1507501043 >> v2:172.21.15.186:3300/0 conn(0x7f3b9015cc80 0x7f3b900cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.572+0000 7f3b8dd9b640 1 -- 172.21.15.29:0/1507501043 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839582946 0 0) 0x7f3b900ce1c0 con 0x7f3b90071fa0 2023-10-14T18:17:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.572+0000 7f3b87fff640 1 -- 172.21.15.29:0/1507501043 >> v2:172.21.15.186:3300/0 conn(0x7f3b9015cc80 msgr2=0x7f3b900cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:46.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.572+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1507501043 >> v2:172.21.15.186:3300/0 conn(0x7f3b9015cc80 0x7f3b900cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.572+0000 7f3b87fff640 1 -- 172.21.15.29:0/1507501043 >> v1:172.21.15.186:6789/0 conn(0x7f3b90071fa0 legacy=0x7f3b901607e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.572+0000 7f3b87fff640 1 -- 172.21.15.29:0/1507501043 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b90072410 con 0x7f3b900cdd60 2023-10-14T18:17:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.573+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1507501043 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 0x7f3b90071a60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b780097a0 tx=0x7f3b7802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b9b7c358b82ac87 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.574+0000 7f3b8dd9b640 1 -- 172.21.15.29:0/1507501043 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78031830 con 0x7f3b900cdd60 2023-10-14T18:17:46.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.574+0000 7f3b8dd9b640 1 -- 172.21.15.29:0/1507501043 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b78031990 con 0x7f3b900cdd60 2023-10-14T18:17:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.574+0000 7f3b8dd9b640 1 -- 172.21.15.29:0/1507501043 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b78031d20 con 0x7f3b900cdd60 2023-10-14T18:17:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1507501043 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 msgr2=0x7f3b90071a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1507501043 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 0x7f3b90071a60 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b780097a0 tx=0x7f3b7802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1507501043 shutdown_connections 2023-10-14T18:17:46.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1507501043 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b900cdd60 0x7f3b90071a60 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1507501043 >> v2:172.21.15.186:3300/0 conn(0x7f3b9015cc80 0x7f3b900cd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1507501043 >> 172.21.15.29:0/1507501043 conn(0x7f3b90065ed0 msgr2=0x7f3b900662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1507501043 shutdown_connections 2023-10-14T18:17:46.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.575+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1507501043 wait complete. 2023-10-14T18:17:46.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.577+0000 7f3b8f59e640 1 Processor -- start 2023-10-14T18:17:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.577+0000 7f3b8f59e640 1 -- start start 2023-10-14T18:17:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 0x7f3b90151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b9015cc80 0x7f3b900c6820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b90152420 con 0x7f3b900cdd60 2023-10-14T18:17:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900b5e20 con 0x7f3b90071fa0 2023-10-14T18:17:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b8f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b900b4da0 con 0x7f3b9015cc80 2023-10-14T18:17:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.578+0000 7f3b87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53516/0 (socket says 172.21.15.29:53516) 2023-10-14T18:17:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 -- 172.21.15.29:0/1705430576 learned_addr learned my addr 172.21.15.29:0/1705430576 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b8e59c640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 0x7f3b90151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b8ed9d640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b9015cc80 0x7f3b900c6820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 -- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b9015cc80 msgr2=0x7f3b900c6820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b9015cc80 0x7f3b900c6820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 -- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 msgr2=0x7f3b90151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 0x7f3b90151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 -- 172.21.15.29:0/1705430576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b78009400 con 0x7f3b900cdd60 2023-10-14T18:17:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b8e59c640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 0x7f3b90151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:46.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.579+0000 7f3b87fff640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f3b7802bee0 tx=0x7f3b780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.580+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7803a6f0 con 0x7f3b900cdd60 2023-10-14T18:17:46.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.580+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b7803a870 con 0x7f3b900cdd60 2023-10-14T18:17:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.580+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b7803ac20 con 0x7f3b900cdd60 2023-10-14T18:17:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.580+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b900b5080 con 0x7f3b900cdd60 2023-10-14T18:17:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.580+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b900b5640 con 0x7f3b900cdd60 2023-10-14T18:17:46.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.581+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b9004e9b0 con 0x7f3b900cdd60 2023-10-14T18:17:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.582+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3b78030380 con 0x7f3b900cdd60 2023-10-14T18:17:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.583+0000 7f3b86ffd640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 0x7f3b7006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.583+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f3b780ab650 con 0x7f3b900cdd60 2023-10-14T18:17:46.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.583+0000 7f3b8e59c640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 0x7f3b7006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.584+0000 7f3b8e59c640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 0x7f3b7006d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b900c67b0 tx=0x7f3b80008040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:46.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.588+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3b780760f0 con 0x7f3b900cdd60 2023-10-14T18:17:46.681 INFO:tasks.ceph.mds.c.smithi029.stderr:2023-10-14T18:17:46.680+0000 7f87cc129640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:17:46.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.783+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3b900d09c0 con 0x7f3b7006af90 2023-10-14T18:17:46.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.829+0000 7f3b86ffd640 1 -- 172.21.15.29:0/1705430576 <== mgr.11010 v2:172.21.15.29:6832/2967217610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525842 (secure 0 0 0) 0x7f3b80011980 con 0x7f3b7006af90 2023-10-14T18:17:46.832 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:46.841 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:17:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 msgr2=0x7f3b7006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 0x7f3b7006d450 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b900c67b0 tx=0x7f3b80008040 comp rx=0 tx=0).stop 2023-10-14T18:17:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 msgr2=0x7f3b901f2b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 secure :-1 s=READY pgs=2638 cs=0 l=1 rev1=1 crypto rx=0x7f3b7802bee0 tx=0x7f3b780046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 shutdown_connections 2023-10-14T18:17:46.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f3b7006af90 0x7f3b7006d450 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3b9015cc80 0x7f3b900c6820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b900cdd60 0x7f3b901f2b80 unknown :-1 s=CLOSED pgs=2638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 --2- 172.21.15.29:0/1705430576 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b90071fa0 0x7f3b90151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.841+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 >> 172.21.15.29:0/1705430576 conn(0x7f3b90065ed0 msgr2=0x7f3b9015bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:46.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.842+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 shutdown_connections 2023-10-14T18:17:46.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:46.842+0000 7f3b8f59e640 1 -- 172.21.15.29:0/1705430576 wait complete. 2023-10-14T18:17:46.869 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-14T18:17:46.752362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":670,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":324,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398496,"kb_used_data":24104,"kb_used_omap":12,"kb_used_meta":374259,"kb_avail":754576224,"statfs":{"total":773094113280,"available":772686053376,"internally_reserved":0,"allocated":24682496,"data_stored":12802746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":383241784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002245"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457578+0000","last_change":"2023-10-14T18:17:42.260490+0000","last_active":"2023-10-14T18:17:43.457578+0000","last_peered":"2023-10-14T18:17:43.457578+0000","last_clean":"2023-10-14T18:17:43.457578+0000","last_became_active":"2023-10-14T18:17:42.260211+0000","last_became_peered":"2023-10-14T18:17:42.260211+0000","last_unstale":"2023-10-14T18:17:43.457578+0000","last_undegraded":"2023-10-14T18:17:43.457578+0000","last_fullsized":"2023-10-14T18:17:43.457578+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:47.448993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1214,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243545+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:17:42.243545+0000","last_peered":"2023-10-14T18:17:42.243545+0000","last_clean":"2023-10-14T18:17:42.243545+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:17:42.243545+0000","last_undegraded":"2023-10-14T18:17:42.243545+0000","last_fullsized":"2023-10-14T18:17:42.243545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250221+0000","last_change":"2023-10-14T18:17:26.757505+0000","last_active":"2023-10-14T18:17:42.250221+0000","last_peered":"2023-10-14T18:17:42.250221+0000","last_clean":"2023-10-14T18:17:42.250221+0000","last_became_active":"2023-10-14T18:17:26.757219+0000","last_became_peered":"2023-10-14T18:17:26.757219+0000","last_unstale":"2023-10-14T18:17:42.250221+0000","last_undegraded":"2023-10-14T18:17:42.250221+0000","last_fullsized":"2023-10-14T18:17:42.250221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:19.831762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247422+0000","last_change":"2023-10-14T18:17:27.260210+0000","last_active":"2023-10-14T18:17:42.247422+0000","last_peered":"2023-10-14T18:17:42.247422+0000","last_clean":"2023-10-14T18:17:42.247422+0000","last_became_active":"2023-10-14T18:17:27.259297+0000","last_became_peered":"2023-10-14T18:17:27.259297+0000","last_unstale":"2023-10-14T18:17:42.247422+0000","last_undegraded":"2023-10-14T18:17:42.247422+0000","last_fullsized":"2023-10-14T18:17:42.247422+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237656+0000","last_change":"2023-10-14T18:17:41.347319+0000","last_active":"2023-10-14T18:17:43.237656+0000","last_peered":"2023-10-14T18:17:43.237656+0000","last_clean":"2023-10-14T18:17:43.237656+0000","last_became_active":"2023-10-14T18:17:41.345913+0000","last_became_peered":"2023-10-14T18:17:41.345913+0000","last_unstale":"2023-10-14T18:17:43.237656+0000","last_undegraded":"2023-10-14T18:17:43.237656+0000","last_fullsized":"2023-10-14T18:17:43.237656+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:46.523999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241004+0000","last_change":"2023-10-14T18:17:42.270250+0000","last_active":"2023-10-14T18:17:43.241004+0000","last_peered":"2023-10-14T18:17:43.241004+0000","last_clean":"2023-10-14T18:17:43.241004+0000","last_became_active":"2023-10-14T18:17:42.267481+0000","last_became_peered":"2023-10-14T18:17:42.267481+0000","last_unstale":"2023-10-14T18:17:43.241004+0000","last_undegraded":"2023-10-14T18:17:43.241004+0000","last_fullsized":"2023-10-14T18:17:43.241004+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:46.735181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.237019+0000","last_change":"2023-10-14T18:17:26.754848+0000","last_active":"2023-10-14T18:17:42.237019+0000","last_peered":"2023-10-14T18:17:42.237019+0000","last_clean":"2023-10-14T18:17:42.237019+0000","last_became_active":"2023-10-14T18:17:26.751997+0000","last_became_peered":"2023-10-14T18:17:26.751997+0000","last_unstale":"2023-10-14T18:17:42.237019+0000","last_undegraded":"2023-10-14T18:17:42.237019+0000","last_fullsized":"2023-10-14T18:17:42.237019+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:31.362757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246072+0000","last_change":"2023-10-14T18:17:27.258205+0000","last_active":"2023-10-14T18:17:42.246072+0000","last_peered":"2023-10-14T18:17:42.246072+0000","last_clean":"2023-10-14T18:17:42.246072+0000","last_became_active":"2023-10-14T18:17:27.257652+0000","last_became_peered":"2023-10-14T18:17:27.257652+0000","last_unstale":"2023-10-14T18:17:42.246072+0000","last_undegraded":"2023-10-14T18:17:42.246072+0000","last_fullsized":"2023-10-14T18:17:42.246072+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:00.190444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457848+0000","last_change":"2023-10-14T18:17:41.343953+0000","last_active":"2023-10-14T18:17:43.457848+0000","last_peered":"2023-10-14T18:17:43.457848+0000","last_clean":"2023-10-14T18:17:43.457848+0000","last_became_active":"2023-10-14T18:17:41.343682+0000","last_became_peered":"2023-10-14T18:17:41.343682+0000","last_unstale":"2023-10-14T18:17:43.457848+0000","last_undegraded":"2023-10-14T18:17:43.457848+0000","last_fullsized":"2023-10-14T18:17:43.457848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:02.002730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454747+0000","last_change":"2023-10-14T18:17:42.267414+0000","last_active":"2023-10-14T18:17:43.454747+0000","last_peered":"2023-10-14T18:17:43.454747+0000","last_clean":"2023-10-14T18:17:43.454747+0000","last_became_active":"2023-10-14T18:17:42.267042+0000","last_became_peered":"2023-10-14T18:17:42.267042+0000","last_unstale":"2023-10-14T18:17:43.454747+0000","last_undegraded":"2023-10-14T18:17:43.454747+0000","last_fullsized":"2023-10-14T18:17:43.454747+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:13.451953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253477+0000","last_change":"2023-10-14T18:17:26.755699+0000","last_active":"2023-10-14T18:17:42.253477+0000","last_peered":"2023-10-14T18:17:42.253477+0000","last_clean":"2023-10-14T18:17:42.253477+0000","last_became_active":"2023-10-14T18:17:26.755286+0000","last_became_peered":"2023-10-14T18:17:26.755286+0000","last_unstale":"2023-10-14T18:17:42.253477+0000","last_undegraded":"2023-10-14T18:17:42.253477+0000","last_fullsized":"2023-10-14T18:17:42.253477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:07.796326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230947+0000","last_change":"2023-10-14T18:17:27.248014+0000","last_active":"2023-10-14T18:17:42.230947+0000","last_peered":"2023-10-14T18:17:42.230947+0000","last_clean":"2023-10-14T18:17:42.230947+0000","last_became_active":"2023-10-14T18:17:27.247706+0000","last_became_peered":"2023-10-14T18:17:27.247706+0000","last_unstale":"2023-10-14T18:17:42.230947+0000","last_undegraded":"2023-10-14T18:17:42.230947+0000","last_fullsized":"2023-10-14T18:17:42.230947+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454805+0000","last_change":"2023-10-14T18:17:41.349108+0000","last_active":"2023-10-14T18:17:43.454805+0000","last_peered":"2023-10-14T18:17:43.454805+0000","last_clean":"2023-10-14T18:17:43.454805+0000","last_became_active":"2023-10-14T18:17:41.348764+0000","last_became_peered":"2023-10-14T18:17:41.348764+0000","last_unstale":"2023-10-14T18:17:43.454805+0000","last_undegraded":"2023-10-14T18:17:43.454805+0000","last_fullsized":"2023-10-14T18:17:43.454805+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:15.080875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454761+0000","last_change":"2023-10-14T18:17:42.254387+0000","last_active":"2023-10-14T18:17:43.454761+0000","last_peered":"2023-10-14T18:17:43.454761+0000","last_clean":"2023-10-14T18:17:43.454761+0000","last_became_active":"2023-10-14T18:17:42.253776+0000","last_became_peered":"2023-10-14T18:17:42.253776+0000","last_unstale":"2023-10-14T18:17:43.454761+0000","last_undegraded":"2023-10-14T18:17:43.454761+0000","last_fullsized":"2023-10-14T18:17:43.454761+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:04.534193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247526+0000","last_change":"2023-10-14T18:17:26.762092+0000","last_active":"2023-10-14T18:17:42.247526+0000","last_peered":"2023-10-14T18:17:42.247526+0000","last_clean":"2023-10-14T18:17:42.247526+0000","last_became_active":"2023-10-14T18:17:26.761696+0000","last_became_peered":"2023-10-14T18:17:26.761696+0000","last_unstale":"2023-10-14T18:17:42.247526+0000","last_undegraded":"2023-10-14T18:17:42.247526+0000","last_fullsized":"2023-10-14T18:17:42.247526+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:27.118966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250016+0000","last_change":"2023-10-14T18:17:27.259073+0000","last_active":"2023-10-14T18:17:42.250016+0000","last_peered":"2023-10-14T18:17:42.250016+0000","last_clean":"2023-10-14T18:17:42.250016+0000","last_became_active":"2023-10-14T18:17:27.258114+0000","last_became_peered":"2023-10-14T18:17:27.258114+0000","last_unstale":"2023-10-14T18:17:42.250016+0000","last_undegraded":"2023-10-14T18:17:42.250016+0000","last_fullsized":"2023-10-14T18:17:42.250016+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:14.438669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239705+0000","last_change":"2023-10-14T18:17:41.340410+0000","last_active":"2023-10-14T18:17:43.239705+0000","last_peered":"2023-10-14T18:17:43.239705+0000","last_clean":"2023-10-14T18:17:43.239705+0000","last_became_active":"2023-10-14T18:17:41.338979+0000","last_became_peered":"2023-10-14T18:17:41.338979+0000","last_unstale":"2023-10-14T18:17:43.239705+0000","last_undegraded":"2023-10-14T18:17:43.239705+0000","last_fullsized":"2023-10-14T18:17:43.239705+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:49:07.446346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239482+0000","last_change":"2023-10-14T18:17:42.262879+0000","last_active":"2023-10-14T18:17:43.239482+0000","last_peered":"2023-10-14T18:17:43.239482+0000","last_clean":"2023-10-14T18:17:43.239482+0000","last_became_active":"2023-10-14T18:17:42.262477+0000","last_became_peered":"2023-10-14T18:17:42.262477+0000","last_unstale":"2023-10-14T18:17:43.239482+0000","last_undegraded":"2023-10-14T18:17:43.239482+0000","last_fullsized":"2023-10-14T18:17:43.239482+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253414+0000","last_change":"2023-10-14T18:17:26.765437+0000","last_active":"2023-10-14T18:17:42.253414+0000","last_peered":"2023-10-14T18:17:42.253414+0000","last_clean":"2023-10-14T18:17:42.253414+0000","last_became_active":"2023-10-14T18:17:26.764932+0000","last_became_peered":"2023-10-14T18:17:26.764932+0000","last_unstale":"2023-10-14T18:17:42.253414+0000","last_undegraded":"2023-10-14T18:17:42.253414+0000","last_fullsized":"2023-10-14T18:17:42.253414+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:00.444875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247544+0000","last_change":"2023-10-14T18:17:27.261099+0000","last_active":"2023-10-14T18:17:42.247544+0000","last_peered":"2023-10-14T18:17:42.247544+0000","last_clean":"2023-10-14T18:17:42.247544+0000","last_became_active":"2023-10-14T18:17:27.260783+0000","last_became_peered":"2023-10-14T18:17:27.260783+0000","last_unstale":"2023-10-14T18:17:42.247544+0000","last_undegraded":"2023-10-14T18:17:42.247544+0000","last_fullsized":"2023-10-14T18:17:42.247544+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458113+0000","last_change":"2023-10-14T18:17:41.342390+0000","last_active":"2023-10-14T18:17:43.458113+0000","last_peered":"2023-10-14T18:17:43.458113+0000","last_clean":"2023-10-14T18:17:43.458113+0000","last_became_active":"2023-10-14T18:17:41.342111+0000","last_became_peered":"2023-10-14T18:17:41.342111+0000","last_unstale":"2023-10-14T18:17:43.458113+0000","last_undegraded":"2023-10-14T18:17:43.458113+0000","last_fullsized":"2023-10-14T18:17:43.458113+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:08.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240972+0000","last_change":"2023-10-14T18:17:42.256640+0000","last_active":"2023-10-14T18:17:43.240972+0000","last_peered":"2023-10-14T18:17:43.240972+0000","last_clean":"2023-10-14T18:17:43.240972+0000","last_became_active":"2023-10-14T18:17:42.253676+0000","last_became_peered":"2023-10-14T18:17:42.253676+0000","last_unstale":"2023-10-14T18:17:43.240972+0000","last_undegraded":"2023-10-14T18:17:43.240972+0000","last_fullsized":"2023-10-14T18:17:43.240972+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:31.063452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230690+0000","last_change":"2023-10-14T18:17:26.767200+0000","last_active":"2023-10-14T18:17:42.230690+0000","last_peered":"2023-10-14T18:17:42.230690+0000","last_clean":"2023-10-14T18:17:42.230690+0000","last_became_active":"2023-10-14T18:17:26.766925+0000","last_became_peered":"2023-10-14T18:17:26.766925+0000","last_unstale":"2023-10-14T18:17:42.230690+0000","last_undegraded":"2023-10-14T18:17:42.230690+0000","last_fullsized":"2023-10-14T18:17:42.230690+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:28.445946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.254013+0000","last_change":"2023-10-14T18:17:27.259767+0000","last_active":"2023-10-14T18:17:42.254013+0000","last_peered":"2023-10-14T18:17:42.254013+0000","last_clean":"2023-10-14T18:17:42.254013+0000","last_became_active":"2023-10-14T18:17:27.259468+0000","last_became_peered":"2023-10-14T18:17:27.259468+0000","last_unstale":"2023-10-14T18:17:42.254013+0000","last_undegraded":"2023-10-14T18:17:42.254013+0000","last_fullsized":"2023-10-14T18:17:42.254013+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:40.472642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453519+0000","last_change":"2023-10-14T18:17:41.331066+0000","last_active":"2023-10-14T18:17:43.453519+0000","last_peered":"2023-10-14T18:17:43.453519+0000","last_clean":"2023-10-14T18:17:43.453519+0000","last_became_active":"2023-10-14T18:17:41.330643+0000","last_became_peered":"2023-10-14T18:17:41.330643+0000","last_unstale":"2023-10-14T18:17:43.453519+0000","last_undegraded":"2023-10-14T18:17:43.453519+0000","last_fullsized":"2023-10-14T18:17:43.453519+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:36:29.940985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.235126+0000","last_change":"2023-10-14T18:17:42.256632+0000","last_active":"2023-10-14T18:17:43.235126+0000","last_peered":"2023-10-14T18:17:43.235126+0000","last_clean":"2023-10-14T18:17:43.235126+0000","last_became_active":"2023-10-14T18:17:42.256317+0000","last_became_peered":"2023-10-14T18:17:42.256317+0000","last_unstale":"2023-10-14T18:17:43.235126+0000","last_undegraded":"2023-10-14T18:17:43.235126+0000","last_fullsized":"2023-10-14T18:17:43.235126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:18:48.609720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249776+0000","last_change":"2023-10-14T18:17:26.754064+0000","last_active":"2023-10-14T18:17:42.249776+0000","last_peered":"2023-10-14T18:17:42.249776+0000","last_clean":"2023-10-14T18:17:42.249776+0000","last_became_active":"2023-10-14T18:17:26.752729+0000","last_became_peered":"2023-10-14T18:17:26.752729+0000","last_unstale":"2023-10-14T18:17:42.249776+0000","last_undegraded":"2023-10-14T18:17:42.249776+0000","last_fullsized":"2023-10-14T18:17:42.249776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:09.744523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.242939+0000","last_change":"2023-10-14T18:17:27.254416+0000","last_active":"2023-10-14T18:17:42.242939+0000","last_peered":"2023-10-14T18:17:42.242939+0000","last_clean":"2023-10-14T18:17:42.242939+0000","last_became_active":"2023-10-14T18:17:27.254010+0000","last_became_peered":"2023-10-14T18:17:27.254010+0000","last_unstale":"2023-10-14T18:17:42.242939+0000","last_undegraded":"2023-10-14T18:17:42.242939+0000","last_fullsized":"2023-10-14T18:17:42.242939+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237270+0000","last_change":"2023-10-14T18:17:41.340549+0000","last_active":"2023-10-14T18:17:43.237270+0000","last_peered":"2023-10-14T18:17:43.237270+0000","last_clean":"2023-10-14T18:17:43.237270+0000","last_became_active":"2023-10-14T18:17:41.340197+0000","last_became_peered":"2023-10-14T18:17:41.340197+0000","last_unstale":"2023-10-14T18:17:43.237270+0000","last_undegraded":"2023-10-14T18:17:43.237270+0000","last_fullsized":"2023-10-14T18:17:43.237270+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456741+0000","last_change":"2023-10-14T18:17:42.264770+0000","last_active":"2023-10-14T18:17:43.456741+0000","last_peered":"2023-10-14T18:17:43.456741+0000","last_clean":"2023-10-14T18:17:43.456741+0000","last_became_active":"2023-10-14T18:17:42.264512+0000","last_became_peered":"2023-10-14T18:17:42.264512+0000","last_unstale":"2023-10-14T18:17:43.456741+0000","last_undegraded":"2023-10-14T18:17:43.456741+0000","last_fullsized":"2023-10-14T18:17:43.456741+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:52.050475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"229'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255086+0000","last_change":"2023-10-14T18:17:26.765123+0000","last_active":"2023-10-14T18:17:42.255086+0000","last_peered":"2023-10-14T18:17:42.255086+0000","last_clean":"2023-10-14T18:17:42.255086+0000","last_became_active":"2023-10-14T18:17:26.764801+0000","last_became_peered":"2023-10-14T18:17:26.764801+0000","last_unstale":"2023-10-14T18:17:42.255086+0000","last_undegraded":"2023-10-14T18:17:42.255086+0000","last_fullsized":"2023-10-14T18:17:42.255086+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:03.297149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253505+0000","last_change":"2023-10-14T18:17:27.252553+0000","last_active":"2023-10-14T18:17:42.253505+0000","last_peered":"2023-10-14T18:17:42.253505+0000","last_clean":"2023-10-14T18:17:42.253505+0000","last_became_active":"2023-10-14T18:17:27.250542+0000","last_became_peered":"2023-10-14T18:17:27.250542+0000","last_unstale":"2023-10-14T18:17:42.253505+0000","last_undegraded":"2023-10-14T18:17:42.253505+0000","last_fullsized":"2023-10-14T18:17:42.253505+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238273+0000","last_change":"2023-10-14T18:17:41.343809+0000","last_active":"2023-10-14T18:17:43.238273+0000","last_peered":"2023-10-14T18:17:43.238273+0000","last_clean":"2023-10-14T18:17:43.238273+0000","last_became_active":"2023-10-14T18:17:41.343130+0000","last_became_peered":"2023-10-14T18:17:41.343130+0000","last_unstale":"2023-10-14T18:17:43.238273+0000","last_undegraded":"2023-10-14T18:17:43.238273+0000","last_fullsized":"2023-10-14T18:17:43.238273+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237924+0000","last_change":"2023-10-14T18:17:42.267983+0000","last_active":"2023-10-14T18:17:43.237924+0000","last_peered":"2023-10-14T18:17:43.237924+0000","last_clean":"2023-10-14T18:17:43.237924+0000","last_became_active":"2023-10-14T18:17:42.267691+0000","last_became_peered":"2023-10-14T18:17:42.267691+0000","last_unstale":"2023-10-14T18:17:43.237924+0000","last_undegraded":"2023-10-14T18:17:43.237924+0000","last_fullsized":"2023-10-14T18:17:43.237924+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:05.726762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.232837+0000","last_change":"2023-10-14T18:17:26.768016+0000","last_active":"2023-10-14T18:17:42.232837+0000","last_peered":"2023-10-14T18:17:42.232837+0000","last_clean":"2023-10-14T18:17:42.232837+0000","last_became_active":"2023-10-14T18:17:26.767729+0000","last_became_peered":"2023-10-14T18:17:26.767729+0000","last_unstale":"2023-10-14T18:17:42.232837+0000","last_undegraded":"2023-10-14T18:17:42.232837+0000","last_fullsized":"2023-10-14T18:17:42.232837+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:30.381195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240577+0000","last_change":"2023-10-14T18:17:27.254575+0000","last_active":"2023-10-14T18:17:42.240577+0000","last_peered":"2023-10-14T18:17:42.240577+0000","last_clean":"2023-10-14T18:17:42.240577+0000","last_became_active":"2023-10-14T18:17:27.254278+0000","last_became_peered":"2023-10-14T18:17:27.254278+0000","last_unstale":"2023-10-14T18:17:42.240577+0000","last_undegraded":"2023-10-14T18:17:42.240577+0000","last_fullsized":"2023-10-14T18:17:42.240577+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453329+0000","last_change":"2023-10-14T18:17:41.343518+0000","last_active":"2023-10-14T18:17:43.453329+0000","last_peered":"2023-10-14T18:17:43.453329+0000","last_clean":"2023-10-14T18:17:43.453329+0000","last_became_active":"2023-10-14T18:17:41.343148+0000","last_became_peered":"2023-10-14T18:17:41.343148+0000","last_unstale":"2023-10-14T18:17:43.453329+0000","last_undegraded":"2023-10-14T18:17:43.453329+0000","last_fullsized":"2023-10-14T18:17:43.453329+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:28.907656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269865+0000","last_change":"2023-10-14T18:17:42.262505+0000","last_active":"2023-10-14T18:17:43.269865+0000","last_peered":"2023-10-14T18:17:43.269865+0000","last_clean":"2023-10-14T18:17:43.269865+0000","last_became_active":"2023-10-14T18:17:42.262149+0000","last_became_peered":"2023-10-14T18:17:42.262149+0000","last_unstale":"2023-10-14T18:17:43.269865+0000","last_undegraded":"2023-10-14T18:17:43.269865+0000","last_fullsized":"2023-10-14T18:17:43.269865+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:18.253464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240776+0000","last_change":"2023-10-14T18:17:26.758398+0000","last_active":"2023-10-14T18:17:42.240776+0000","last_peered":"2023-10-14T18:17:42.240776+0000","last_clean":"2023-10-14T18:17:42.240776+0000","last_became_active":"2023-10-14T18:17:26.758114+0000","last_became_peered":"2023-10-14T18:17:26.758114+0000","last_unstale":"2023-10-14T18:17:42.240776+0000","last_undegraded":"2023-10-14T18:17:42.240776+0000","last_fullsized":"2023-10-14T18:17:42.240776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:39.144399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253218+0000","last_change":"2023-10-14T18:17:27.254894+0000","last_active":"2023-10-14T18:17:42.253218+0000","last_peered":"2023-10-14T18:17:42.253218+0000","last_clean":"2023-10-14T18:17:42.253218+0000","last_became_active":"2023-10-14T18:17:27.253030+0000","last_became_peered":"2023-10-14T18:17:27.253030+0000","last_unstale":"2023-10-14T18:17:42.253218+0000","last_undegraded":"2023-10-14T18:17:42.253218+0000","last_fullsized":"2023-10-14T18:17:42.253218+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:38.076369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454581+0000","last_change":"2023-10-14T18:17:41.338009+0000","last_active":"2023-10-14T18:17:43.454581+0000","last_peered":"2023-10-14T18:17:43.454581+0000","last_clean":"2023-10-14T18:17:43.454581+0000","last_became_active":"2023-10-14T18:17:41.337716+0000","last_became_peered":"2023-10-14T18:17:41.337716+0000","last_unstale":"2023-10-14T18:17:43.454581+0000","last_undegraded":"2023-10-14T18:17:43.454581+0000","last_fullsized":"2023-10-14T18:17:43.454581+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:53.711866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.258926+0000","last_change":"2023-10-14T18:17:42.259942+0000","last_active":"2023-10-14T18:17:43.258926+0000","last_peered":"2023-10-14T18:17:43.258926+0000","last_clean":"2023-10-14T18:17:43.258926+0000","last_became_active":"2023-10-14T18:17:42.259630+0000","last_became_peered":"2023-10-14T18:17:42.259630+0000","last_unstale":"2023-10-14T18:17:43.258926+0000","last_undegraded":"2023-10-14T18:17:43.258926+0000","last_fullsized":"2023-10-14T18:17:43.258926+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248053+0000","last_change":"2023-10-14T18:17:26.756432+0000","last_active":"2023-10-14T18:17:42.248053+0000","last_peered":"2023-10-14T18:17:42.248053+0000","last_clean":"2023-10-14T18:17:42.248053+0000","last_became_active":"2023-10-14T18:17:26.756148+0000","last_became_peered":"2023-10-14T18:17:26.756148+0000","last_unstale":"2023-10-14T18:17:42.248053+0000","last_undegraded":"2023-10-14T18:17:42.248053+0000","last_fullsized":"2023-10-14T18:17:42.248053+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:13.420763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245606+0000","last_change":"2023-10-14T18:17:27.258139+0000","last_active":"2023-10-14T18:17:42.245606+0000","last_peered":"2023-10-14T18:17:42.245606+0000","last_clean":"2023-10-14T18:17:42.245606+0000","last_became_active":"2023-10-14T18:17:27.257907+0000","last_became_peered":"2023-10-14T18:17:27.257907+0000","last_unstale":"2023-10-14T18:17:42.245606+0000","last_undegraded":"2023-10-14T18:17:42.245606+0000","last_fullsized":"2023-10-14T18:17:42.245606+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241295+0000","last_change":"2023-10-14T18:17:41.347434+0000","last_active":"2023-10-14T18:17:43.241295+0000","last_peered":"2023-10-14T18:17:43.241295+0000","last_clean":"2023-10-14T18:17:43.241295+0000","last_became_active":"2023-10-14T18:17:41.346764+0000","last_became_peered":"2023-10-14T18:17:41.346764+0000","last_unstale":"2023-10-14T18:17:43.241295+0000","last_undegraded":"2023-10-14T18:17:43.241295+0000","last_fullsized":"2023-10-14T18:17:43.241295+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:51.625931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240112+0000","last_change":"2023-10-14T18:17:42.256562+0000","last_active":"2023-10-14T18:17:43.240112+0000","last_peered":"2023-10-14T18:17:43.240112+0000","last_clean":"2023-10-14T18:17:43.240112+0000","last_became_active":"2023-10-14T18:17:42.256238+0000","last_became_peered":"2023-10-14T18:17:42.256238+0000","last_unstale":"2023-10-14T18:17:43.240112+0000","last_undegraded":"2023-10-14T18:17:43.240112+0000","last_fullsized":"2023-10-14T18:17:43.240112+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247684+0000","last_change":"2023-10-14T18:17:26.767394+0000","last_active":"2023-10-14T18:17:42.247684+0000","last_peered":"2023-10-14T18:17:42.247684+0000","last_clean":"2023-10-14T18:17:42.247684+0000","last_became_active":"2023-10-14T18:17:26.767091+0000","last_became_peered":"2023-10-14T18:17:26.767091+0000","last_unstale":"2023-10-14T18:17:42.247684+0000","last_undegraded":"2023-10-14T18:17:42.247684+0000","last_fullsized":"2023-10-14T18:17:42.247684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230388+0000","last_change":"2023-10-14T18:17:27.251912+0000","last_active":"2023-10-14T18:17:42.230388+0000","last_peered":"2023-10-14T18:17:42.230388+0000","last_clean":"2023-10-14T18:17:42.230388+0000","last_became_active":"2023-10-14T18:17:27.251629+0000","last_became_peered":"2023-10-14T18:17:27.251629+0000","last_unstale":"2023-10-14T18:17:42.230388+0000","last_undegraded":"2023-10-14T18:17:42.230388+0000","last_fullsized":"2023-10-14T18:17:42.230388+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:17.594667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269752+0000","last_change":"2023-10-14T18:17:41.341578+0000","last_active":"2023-10-14T18:17:43.269752+0000","last_peered":"2023-10-14T18:17:43.269752+0000","last_clean":"2023-10-14T18:17:43.269752+0000","last_became_active":"2023-10-14T18:17:41.341312+0000","last_became_peered":"2023-10-14T18:17:41.341312+0000","last_unstale":"2023-10-14T18:17:43.269752+0000","last_undegraded":"2023-10-14T18:17:43.269752+0000","last_fullsized":"2023-10-14T18:17:43.269752+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:32:38.030124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239043+0000","last_change":"2023-10-14T18:17:42.254578+0000","last_active":"2023-10-14T18:17:43.239043+0000","last_peered":"2023-10-14T18:17:43.239043+0000","last_clean":"2023-10-14T18:17:43.239043+0000","last_became_active":"2023-10-14T18:17:42.254244+0000","last_became_peered":"2023-10-14T18:17:42.254244+0000","last_unstale":"2023-10-14T18:17:43.239043+0000","last_undegraded":"2023-10-14T18:17:43.239043+0000","last_fullsized":"2023-10-14T18:17:43.239043+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:50.707802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247373+0000","last_change":"2023-10-14T18:17:26.756320+0000","last_active":"2023-10-14T18:17:42.247373+0000","last_peered":"2023-10-14T18:17:42.247373+0000","last_clean":"2023-10-14T18:17:42.247373+0000","last_became_active":"2023-10-14T18:17:26.756040+0000","last_became_peered":"2023-10-14T18:17:26.756040+0000","last_unstale":"2023-10-14T18:17:42.247373+0000","last_undegraded":"2023-10-14T18:17:42.247373+0000","last_fullsized":"2023-10-14T18:17:42.247373+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:10.901473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243996+0000","last_change":"2023-10-14T18:17:27.248764+0000","last_active":"2023-10-14T18:17:42.243996+0000","last_peered":"2023-10-14T18:17:42.243996+0000","last_clean":"2023-10-14T18:17:42.243996+0000","last_became_active":"2023-10-14T18:17:27.248287+0000","last_became_peered":"2023-10-14T18:17:27.248287+0000","last_unstale":"2023-10-14T18:17:42.243996+0000","last_undegraded":"2023-10-14T18:17:42.243996+0000","last_fullsized":"2023-10-14T18:17:42.243996+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:26.912129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457779+0000","last_change":"2023-10-14T18:17:41.348300+0000","last_active":"2023-10-14T18:17:43.457779+0000","last_peered":"2023-10-14T18:17:43.457779+0000","last_clean":"2023-10-14T18:17:43.457779+0000","last_became_active":"2023-10-14T18:17:41.347703+0000","last_became_peered":"2023-10-14T18:17:41.347703+0000","last_unstale":"2023-10-14T18:17:43.457779+0000","last_undegraded":"2023-10-14T18:17:43.457779+0000","last_fullsized":"2023-10-14T18:17:43.457779+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:20.342562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239426+0000","last_change":"2023-10-14T18:17:42.264682+0000","last_active":"2023-10-14T18:17:43.239426+0000","last_peered":"2023-10-14T18:17:43.239426+0000","last_clean":"2023-10-14T18:17:43.239426+0000","last_became_active":"2023-10-14T18:17:42.264407+0000","last_became_peered":"2023-10-14T18:17:42.264407+0000","last_unstale":"2023-10-14T18:17:43.239426+0000","last_undegraded":"2023-10-14T18:17:43.239426+0000","last_fullsized":"2023-10-14T18:17:43.239426+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230453+0000","last_change":"2023-10-14T18:17:26.761052+0000","last_active":"2023-10-14T18:17:42.230453+0000","last_peered":"2023-10-14T18:17:42.230453+0000","last_clean":"2023-10-14T18:17:42.230453+0000","last_became_active":"2023-10-14T18:17:26.760754+0000","last_became_peered":"2023-10-14T18:17:26.760754+0000","last_unstale":"2023-10-14T18:17:42.230453+0000","last_undegraded":"2023-10-14T18:17:42.230453+0000","last_fullsized":"2023-10-14T18:17:42.230453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:54.469184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245367+0000","last_change":"2023-10-14T18:17:27.269260+0000","last_active":"2023-10-14T18:17:42.245367+0000","last_peered":"2023-10-14T18:17:42.245367+0000","last_clean":"2023-10-14T18:17:42.245367+0000","last_became_active":"2023-10-14T18:17:27.269043+0000","last_became_peered":"2023-10-14T18:17:27.269043+0000","last_unstale":"2023-10-14T18:17:42.245367+0000","last_undegraded":"2023-10-14T18:17:42.245367+0000","last_fullsized":"2023-10-14T18:17:42.245367+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:53.145962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237739+0000","last_change":"2023-10-14T18:17:41.343875+0000","last_active":"2023-10-14T18:17:43.237739+0000","last_peered":"2023-10-14T18:17:43.237739+0000","last_clean":"2023-10-14T18:17:43.237739+0000","last_became_active":"2023-10-14T18:17:41.343534+0000","last_became_peered":"2023-10-14T18:17:41.343534+0000","last_unstale":"2023-10-14T18:17:43.237739+0000","last_undegraded":"2023-10-14T18:17:43.237739+0000","last_fullsized":"2023-10-14T18:17:43.237739+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:51.625931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453016+0000","last_change":"2023-10-14T18:17:42.264988+0000","last_active":"2023-10-14T18:17:43.453016+0000","last_peered":"2023-10-14T18:17:43.453016+0000","last_clean":"2023-10-14T18:17:43.453016+0000","last_became_active":"2023-10-14T18:17:42.264671+0000","last_became_peered":"2023-10-14T18:17:42.264671+0000","last_unstale":"2023-10-14T18:17:43.453016+0000","last_undegraded":"2023-10-14T18:17:43.453016+0000","last_fullsized":"2023-10-14T18:17:43.453016+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:16.031832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247413+0000","last_change":"2023-10-14T18:17:26.760063+0000","last_active":"2023-10-14T18:17:42.247413+0000","last_peered":"2023-10-14T18:17:42.247413+0000","last_clean":"2023-10-14T18:17:42.247413+0000","last_became_active":"2023-10-14T18:17:26.758718+0000","last_became_peered":"2023-10-14T18:17:26.758718+0000","last_unstale":"2023-10-14T18:17:42.247413+0000","last_undegraded":"2023-10-14T18:17:42.247413+0000","last_fullsized":"2023-10-14T18:17:42.247413+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:29.586206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230177+0000","last_change":"2023-10-14T18:17:27.258724+0000","last_active":"2023-10-14T18:17:42.230177+0000","last_peered":"2023-10-14T18:17:42.230177+0000","last_clean":"2023-10-14T18:17:42.230177+0000","last_became_active":"2023-10-14T18:17:27.255949+0000","last_became_peered":"2023-10-14T18:17:27.255949+0000","last_unstale":"2023-10-14T18:17:42.230177+0000","last_undegraded":"2023-10-14T18:17:42.230177+0000","last_fullsized":"2023-10-14T18:17:42.230177+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:58.985698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240528+0000","last_change":"2023-10-14T18:17:41.347603+0000","last_active":"2023-10-14T18:17:43.240528+0000","last_peered":"2023-10-14T18:17:43.240528+0000","last_clean":"2023-10-14T18:17:43.240528+0000","last_became_active":"2023-10-14T18:17:41.347226+0000","last_became_peered":"2023-10-14T18:17:41.347226+0000","last_unstale":"2023-10-14T18:17:43.240528+0000","last_undegraded":"2023-10-14T18:17:43.240528+0000","last_fullsized":"2023-10-14T18:17:43.240528+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:55.118793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453241+0000","last_change":"2023-10-14T18:17:42.267743+0000","last_active":"2023-10-14T18:17:43.453241+0000","last_peered":"2023-10-14T18:17:43.453241+0000","last_clean":"2023-10-14T18:17:43.453241+0000","last_became_active":"2023-10-14T18:17:42.267190+0000","last_became_peered":"2023-10-14T18:17:42.267190+0000","last_unstale":"2023-10-14T18:17:43.453241+0000","last_undegraded":"2023-10-14T18:17:43.453241+0000","last_fullsized":"2023-10-14T18:17:43.453241+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:30:49.569484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247607+0000","last_change":"2023-10-14T18:17:26.766615+0000","last_active":"2023-10-14T18:17:42.247607+0000","last_peered":"2023-10-14T18:17:42.247607+0000","last_clean":"2023-10-14T18:17:42.247607+0000","last_became_active":"2023-10-14T18:17:26.766167+0000","last_became_peered":"2023-10-14T18:17:26.766167+0000","last_unstale":"2023-10-14T18:17:42.247607+0000","last_undegraded":"2023-10-14T18:17:42.247607+0000","last_fullsized":"2023-10-14T18:17:42.247607+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:13.374759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249934+0000","last_change":"2023-10-14T18:17:27.260977+0000","last_active":"2023-10-14T18:17:42.249934+0000","last_peered":"2023-10-14T18:17:42.249934+0000","last_clean":"2023-10-14T18:17:42.249934+0000","last_became_active":"2023-10-14T18:17:27.260706+0000","last_became_peered":"2023-10-14T18:17:27.260706+0000","last_unstale":"2023-10-14T18:17:42.249934+0000","last_undegraded":"2023-10-14T18:17:42.249934+0000","last_fullsized":"2023-10-14T18:17:42.249934+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238167+0000","last_change":"2023-10-14T18:17:41.344229+0000","last_active":"2023-10-14T18:17:43.238167+0000","last_peered":"2023-10-14T18:17:43.238167+0000","last_clean":"2023-10-14T18:17:43.238167+0000","last_became_active":"2023-10-14T18:17:41.343528+0000","last_became_peered":"2023-10-14T18:17:41.343528+0000","last_unstale":"2023-10-14T18:17:43.238167+0000","last_undegraded":"2023-10-14T18:17:43.238167+0000","last_fullsized":"2023-10-14T18:17:43.238167+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:31.144229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257422+0000","last_change":"2023-10-14T18:17:42.261265+0000","last_active":"2023-10-14T18:17:43.257422+0000","last_peered":"2023-10-14T18:17:43.257422+0000","last_clean":"2023-10-14T18:17:43.257422+0000","last_became_active":"2023-10-14T18:17:42.261000+0000","last_became_peered":"2023-10-14T18:17:42.261000+0000","last_unstale":"2023-10-14T18:17:43.257422+0000","last_undegraded":"2023-10-14T18:17:43.257422+0000","last_fullsized":"2023-10-14T18:17:43.257422+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243042+0000","last_change":"2023-10-14T18:17:26.763127+0000","last_active":"2023-10-14T18:17:42.243042+0000","last_peered":"2023-10-14T18:17:42.243042+0000","last_clean":"2023-10-14T18:17:42.243042+0000","last_became_active":"2023-10-14T18:17:26.762581+0000","last_became_peered":"2023-10-14T18:17:26.762581+0000","last_unstale":"2023-10-14T18:17:42.243042+0000","last_undegraded":"2023-10-14T18:17:42.243042+0000","last_fullsized":"2023-10-14T18:17:42.243042+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:15.476495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247063+0000","last_change":"2023-10-14T18:17:27.265001+0000","last_active":"2023-10-14T18:17:42.247063+0000","last_peered":"2023-10-14T18:17:42.247063+0000","last_clean":"2023-10-14T18:17:42.247063+0000","last_became_active":"2023-10-14T18:17:27.264106+0000","last_became_peered":"2023-10-14T18:17:27.264106+0000","last_unstale":"2023-10-14T18:17:42.247063+0000","last_undegraded":"2023-10-14T18:17:42.247063+0000","last_fullsized":"2023-10-14T18:17:42.247063+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237357+0000","last_change":"2023-10-14T18:17:41.347521+0000","last_active":"2023-10-14T18:17:43.237357+0000","last_peered":"2023-10-14T18:17:43.237357+0000","last_clean":"2023-10-14T18:17:43.237357+0000","last_became_active":"2023-10-14T18:17:41.347194+0000","last_became_peered":"2023-10-14T18:17:41.347194+0000","last_unstale":"2023-10-14T18:17:43.237357+0000","last_undegraded":"2023-10-14T18:17:43.237357+0000","last_fullsized":"2023-10-14T18:17:43.237357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:49:07.446346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458750+0000","last_change":"2023-10-14T18:17:42.268477+0000","last_active":"2023-10-14T18:17:43.458750+0000","last_peered":"2023-10-14T18:17:43.458750+0000","last_clean":"2023-10-14T18:17:43.458750+0000","last_became_active":"2023-10-14T18:17:42.265942+0000","last_became_peered":"2023-10-14T18:17:42.265942+0000","last_unstale":"2023-10-14T18:17:43.458750+0000","last_undegraded":"2023-10-14T18:17:43.458750+0000","last_fullsized":"2023-10-14T18:17:43.458750+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:12.806244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"228'8","reported_seq":24,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230606+0000","last_change":"2023-10-14T18:17:26.758995+0000","last_active":"2023-10-14T18:17:42.230606+0000","last_peered":"2023-10-14T18:17:42.230606+0000","last_clean":"2023-10-14T18:17:42.230606+0000","last_became_active":"2023-10-14T18:17:26.758694+0000","last_became_peered":"2023-10-14T18:17:26.758694+0000","last_unstale":"2023-10-14T18:17:42.230606+0000","last_undegraded":"2023-10-14T18:17:42.230606+0000","last_fullsized":"2023-10-14T18:17:42.230606+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247914+0000","last_change":"2023-10-14T18:17:27.258412+0000","last_active":"2023-10-14T18:17:42.247914+0000","last_peered":"2023-10-14T18:17:42.247914+0000","last_clean":"2023-10-14T18:17:42.247914+0000","last_became_active":"2023-10-14T18:17:27.257835+0000","last_became_peered":"2023-10-14T18:17:27.257835+0000","last_unstale":"2023-10-14T18:17:42.247914+0000","last_undegraded":"2023-10-14T18:17:42.247914+0000","last_fullsized":"2023-10-14T18:17:42.247914+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454318+0000","last_change":"2023-10-14T18:17:41.338219+0000","last_active":"2023-10-14T18:17:43.454318+0000","last_peered":"2023-10-14T18:17:43.454318+0000","last_clean":"2023-10-14T18:17:43.454318+0000","last_became_active":"2023-10-14T18:17:41.335426+0000","last_became_peered":"2023-10-14T18:17:41.335426+0000","last_unstale":"2023-10-14T18:17:43.454318+0000","last_undegraded":"2023-10-14T18:17:43.454318+0000","last_fullsized":"2023-10-14T18:17:43.454318+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:49.688313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236401+0000","last_change":"2023-10-14T18:17:42.271606+0000","last_active":"2023-10-14T18:17:43.236401+0000","last_peered":"2023-10-14T18:17:43.236401+0000","last_clean":"2023-10-14T18:17:43.236401+0000","last_became_active":"2023-10-14T18:17:42.271301+0000","last_became_peered":"2023-10-14T18:17:42.271301+0000","last_unstale":"2023-10-14T18:17:43.236401+0000","last_undegraded":"2023-10-14T18:17:43.236401+0000","last_fullsized":"2023-10-14T18:17:43.236401+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:02.966918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243585+0000","last_change":"2023-10-14T18:17:26.760901+0000","last_active":"2023-10-14T18:17:42.243585+0000","last_peered":"2023-10-14T18:17:42.243585+0000","last_clean":"2023-10-14T18:17:42.243585+0000","last_became_active":"2023-10-14T18:17:26.760610+0000","last_became_peered":"2023-10-14T18:17:26.760610+0000","last_unstale":"2023-10-14T18:17:42.243585+0000","last_undegraded":"2023-10-14T18:17:42.243585+0000","last_fullsized":"2023-10-14T18:17:42.243585+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:03:16.410048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230239+0000","last_change":"2023-10-14T18:17:27.247857+0000","last_active":"2023-10-14T18:17:42.230239+0000","last_peered":"2023-10-14T18:17:42.230239+0000","last_clean":"2023-10-14T18:17:42.230239+0000","last_became_active":"2023-10-14T18:17:27.247391+0000","last_became_peered":"2023-10-14T18:17:27.247391+0000","last_unstale":"2023-10-14T18:17:42.230239+0000","last_undegraded":"2023-10-14T18:17:42.230239+0000","last_fullsized":"2023-10-14T18:17:42.230239+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458579+0000","last_change":"2023-10-14T18:17:41.345492+0000","last_active":"2023-10-14T18:17:43.458579+0000","last_peered":"2023-10-14T18:17:43.458579+0000","last_clean":"2023-10-14T18:17:43.458579+0000","last_became_active":"2023-10-14T18:17:41.345124+0000","last_became_peered":"2023-10-14T18:17:41.345124+0000","last_unstale":"2023-10-14T18:17:43.458579+0000","last_undegraded":"2023-10-14T18:17:43.458579+0000","last_fullsized":"2023-10-14T18:17:43.458579+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:20.202542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457519+0000","last_change":"2023-10-14T18:17:42.263795+0000","last_active":"2023-10-14T18:17:43.457519+0000","last_peered":"2023-10-14T18:17:43.457519+0000","last_clean":"2023-10-14T18:17:43.457519+0000","last_became_active":"2023-10-14T18:17:42.263533+0000","last_became_peered":"2023-10-14T18:17:42.263533+0000","last_unstale":"2023-10-14T18:17:43.457519+0000","last_undegraded":"2023-10-14T18:17:43.457519+0000","last_fullsized":"2023-10-14T18:17:43.457519+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:34:10.009963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255558+0000","last_change":"2023-10-14T18:17:26.758089+0000","last_active":"2023-10-14T18:17:42.255558+0000","last_peered":"2023-10-14T18:17:42.255558+0000","last_clean":"2023-10-14T18:17:42.255558+0000","last_became_active":"2023-10-14T18:17:26.755775+0000","last_became_peered":"2023-10-14T18:17:26.755775+0000","last_unstale":"2023-10-14T18:17:42.255558+0000","last_undegraded":"2023-10-14T18:17:42.255558+0000","last_fullsized":"2023-10-14T18:17:42.255558+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:57.967793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247196+0000","last_change":"2023-10-14T18:17:27.256523+0000","last_active":"2023-10-14T18:17:42.247196+0000","last_peered":"2023-10-14T18:17:42.247196+0000","last_clean":"2023-10-14T18:17:42.247196+0000","last_became_active":"2023-10-14T18:17:27.255795+0000","last_became_peered":"2023-10-14T18:17:27.255795+0000","last_unstale":"2023-10-14T18:17:42.247196+0000","last_undegraded":"2023-10-14T18:17:42.247196+0000","last_fullsized":"2023-10-14T18:17:42.247196+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:52:41.950726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"232'6","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269712+0000","last_change":"2023-10-14T18:17:41.346722+0000","last_active":"2023-10-14T18:17:43.269712+0000","last_peered":"2023-10-14T18:17:43.269712+0000","last_clean":"2023-10-14T18:17:43.269712+0000","last_became_active":"2023-10-14T18:17:41.346210+0000","last_became_peered":"2023-10-14T18:17:41.346210+0000","last_unstale":"2023-10-14T18:17:43.269712+0000","last_undegraded":"2023-10-14T18:17:43.269712+0000","last_fullsized":"2023-10-14T18:17:43.269712+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:02.494491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238283+0000","last_change":"2023-10-14T18:17:42.257842+0000","last_active":"2023-10-14T18:17:43.238283+0000","last_peered":"2023-10-14T18:17:43.238283+0000","last_clean":"2023-10-14T18:17:43.238283+0000","last_became_active":"2023-10-14T18:17:42.257561+0000","last_became_peered":"2023-10-14T18:17:42.257561+0000","last_unstale":"2023-10-14T18:17:43.238283+0000","last_undegraded":"2023-10-14T18:17:43.238283+0000","last_fullsized":"2023-10-14T18:17:43.238283+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252783+0000","last_change":"2023-10-14T18:17:26.754215+0000","last_active":"2023-10-14T18:17:42.252783+0000","last_peered":"2023-10-14T18:17:42.252783+0000","last_clean":"2023-10-14T18:17:42.252783+0000","last_became_active":"2023-10-14T18:17:26.753857+0000","last_became_peered":"2023-10-14T18:17:26.753857+0000","last_unstale":"2023-10-14T18:17:42.252783+0000","last_undegraded":"2023-10-14T18:17:42.252783+0000","last_fullsized":"2023-10-14T18:17:42.252783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:10.112105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247205+0000","last_change":"2023-10-14T18:17:27.253122+0000","last_active":"2023-10-14T18:17:42.247205+0000","last_peered":"2023-10-14T18:17:42.247205+0000","last_clean":"2023-10-14T18:17:42.247205+0000","last_became_active":"2023-10-14T18:17:27.252846+0000","last_became_peered":"2023-10-14T18:17:27.252846+0000","last_unstale":"2023-10-14T18:17:42.247205+0000","last_undegraded":"2023-10-14T18:17:42.247205+0000","last_fullsized":"2023-10-14T18:17:42.247205+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:43.166178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"232'3","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454931+0000","last_change":"2023-10-14T18:17:41.335623+0000","last_active":"2023-10-14T18:17:43.454931+0000","last_peered":"2023-10-14T18:17:43.454931+0000","last_clean":"2023-10-14T18:17:43.454931+0000","last_became_active":"2023-10-14T18:17:41.335335+0000","last_became_peered":"2023-10-14T18:17:41.335335+0000","last_unstale":"2023-10-14T18:17:43.454931+0000","last_undegraded":"2023-10-14T18:17:43.454931+0000","last_fullsized":"2023-10-14T18:17:43.454931+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:00:43.220121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241292+0000","last_change":"2023-10-14T18:17:42.253738+0000","last_active":"2023-10-14T18:17:43.241292+0000","last_peered":"2023-10-14T18:17:43.241292+0000","last_clean":"2023-10-14T18:17:43.241292+0000","last_became_active":"2023-10-14T18:17:42.253386+0000","last_became_peered":"2023-10-14T18:17:42.253386+0000","last_unstale":"2023-10-14T18:17:43.241292+0000","last_undegraded":"2023-10-14T18:17:43.241292+0000","last_fullsized":"2023-10-14T18:17:43.241292+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:52.482235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"229'8","reported_seq":24,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250690+0000","last_change":"2023-10-14T18:17:26.761103+0000","last_active":"2023-10-14T18:17:42.250690+0000","last_peered":"2023-10-14T18:17:42.250690+0000","last_clean":"2023-10-14T18:17:42.250690+0000","last_became_active":"2023-10-14T18:17:26.760729+0000","last_became_peered":"2023-10-14T18:17:26.760729+0000","last_unstale":"2023-10-14T18:17:42.250690+0000","last_undegraded":"2023-10-14T18:17:42.250690+0000","last_fullsized":"2023-10-14T18:17:42.250690+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:27.269547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252649+0000","last_change":"2023-10-14T18:17:27.257920+0000","last_active":"2023-10-14T18:17:42.252649+0000","last_peered":"2023-10-14T18:17:42.252649+0000","last_clean":"2023-10-14T18:17:42.252649+0000","last_became_active":"2023-10-14T18:17:27.256790+0000","last_became_peered":"2023-10-14T18:17:27.256790+0000","last_unstale":"2023-10-14T18:17:42.252649+0000","last_undegraded":"2023-10-14T18:17:42.252649+0000","last_fullsized":"2023-10-14T18:17:42.252649+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:55.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453855+0000","last_change":"2023-10-14T18:17:41.338781+0000","last_active":"2023-10-14T18:17:43.453855+0000","last_peered":"2023-10-14T18:17:43.453855+0000","last_clean":"2023-10-14T18:17:43.453855+0000","last_became_active":"2023-10-14T18:17:41.338426+0000","last_became_peered":"2023-10-14T18:17:41.338426+0000","last_unstale":"2023-10-14T18:17:43.453855+0000","last_undegraded":"2023-10-14T18:17:43.453855+0000","last_fullsized":"2023-10-14T18:17:43.453855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:03.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453668+0000","last_change":"2023-10-14T18:17:42.255575+0000","last_active":"2023-10-14T18:17:43.453668+0000","last_peered":"2023-10-14T18:17:43.453668+0000","last_clean":"2023-10-14T18:17:43.453668+0000","last_became_active":"2023-10-14T18:17:42.255215+0000","last_became_peered":"2023-10-14T18:17:42.255215+0000","last_unstale":"2023-10-14T18:17:43.453668+0000","last_undegraded":"2023-10-14T18:17:43.453668+0000","last_fullsized":"2023-10-14T18:17:43.453668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"228'3","reported_seq":19,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247161+0000","last_change":"2023-10-14T18:17:26.757138+0000","last_active":"2023-10-14T18:17:42.247161+0000","last_peered":"2023-10-14T18:17:42.247161+0000","last_clean":"2023-10-14T18:17:42.247161+0000","last_became_active":"2023-10-14T18:17:26.753946+0000","last_became_peered":"2023-10-14T18:17:26.753946+0000","last_unstale":"2023-10-14T18:17:42.247161+0000","last_undegraded":"2023-10-14T18:17:42.247161+0000","last_fullsized":"2023-10-14T18:17:42.247161+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:29.218181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249566+0000","last_change":"2023-10-14T18:17:27.263006+0000","last_active":"2023-10-14T18:17:42.249566+0000","last_peered":"2023-10-14T18:17:42.249566+0000","last_clean":"2023-10-14T18:17:42.249566+0000","last_became_active":"2023-10-14T18:17:27.262724+0000","last_became_peered":"2023-10-14T18:17:27.262724+0000","last_unstale":"2023-10-14T18:17:42.249566+0000","last_undegraded":"2023-10-14T18:17:42.249566+0000","last_fullsized":"2023-10-14T18:17:42.249566+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:54:28.798619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257605+0000","last_change":"2023-10-14T18:17:41.344858+0000","last_active":"2023-10-14T18:17:43.257605+0000","last_peered":"2023-10-14T18:17:43.257605+0000","last_clean":"2023-10-14T18:17:43.257605+0000","last_became_active":"2023-10-14T18:17:41.344380+0000","last_became_peered":"2023-10-14T18:17:41.344380+0000","last_unstale":"2023-10-14T18:17:43.257605+0000","last_undegraded":"2023-10-14T18:17:43.257605+0000","last_fullsized":"2023-10-14T18:17:43.257605+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:55.118793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237996+0000","last_change":"2023-10-14T18:17:42.256582+0000","last_active":"2023-10-14T18:17:43.237996+0000","last_peered":"2023-10-14T18:17:43.237996+0000","last_clean":"2023-10-14T18:17:43.237996+0000","last_became_active":"2023-10-14T18:17:42.256309+0000","last_became_peered":"2023-10-14T18:17:42.256309+0000","last_unstale":"2023-10-14T18:17:43.237996+0000","last_undegraded":"2023-10-14T18:17:43.237996+0000","last_fullsized":"2023-10-14T18:17:43.237996+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255378+0000","last_change":"2023-10-14T18:17:26.766188+0000","last_active":"2023-10-14T18:17:42.255378+0000","last_peered":"2023-10-14T18:17:42.255378+0000","last_clean":"2023-10-14T18:17:42.255378+0000","last_became_active":"2023-10-14T18:17:26.765885+0000","last_became_peered":"2023-10-14T18:17:26.765885+0000","last_unstale":"2023-10-14T18:17:42.255378+0000","last_undegraded":"2023-10-14T18:17:42.255378+0000","last_fullsized":"2023-10-14T18:17:42.255378+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:00.302980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.232551+0000","last_change":"2023-10-14T18:17:27.251334+0000","last_active":"2023-10-14T18:17:42.232551+0000","last_peered":"2023-10-14T18:17:42.232551+0000","last_clean":"2023-10-14T18:17:42.232551+0000","last_became_active":"2023-10-14T18:17:27.251040+0000","last_became_peered":"2023-10-14T18:17:27.251040+0000","last_unstale":"2023-10-14T18:17:42.232551+0000","last_undegraded":"2023-10-14T18:17:42.232551+0000","last_fullsized":"2023-10-14T18:17:42.232551+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:11.130011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"232'8","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233615+0000","last_change":"2023-10-14T18:17:41.340395+0000","last_active":"2023-10-14T18:17:43.233615+0000","last_peered":"2023-10-14T18:17:43.233615+0000","last_clean":"2023-10-14T18:17:43.233615+0000","last_became_active":"2023-10-14T18:17:41.339899+0000","last_became_peered":"2023-10-14T18:17:41.339899+0000","last_unstale":"2023-10-14T18:17:43.233615+0000","last_undegraded":"2023-10-14T18:17:43.233615+0000","last_fullsized":"2023-10-14T18:17:43.233615+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:53.078883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238773+0000","last_change":"2023-10-14T18:17:42.265213+0000","last_active":"2023-10-14T18:17:43.238773+0000","last_peered":"2023-10-14T18:17:43.238773+0000","last_clean":"2023-10-14T18:17:43.238773+0000","last_became_active":"2023-10-14T18:17:42.264925+0000","last_became_peered":"2023-10-14T18:17:42.264925+0000","last_unstale":"2023-10-14T18:17:43.238773+0000","last_undegraded":"2023-10-14T18:17:43.238773+0000","last_fullsized":"2023-10-14T18:17:43.238773+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:04.430721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245388+0000","last_change":"2023-10-14T18:17:26.766951+0000","last_active":"2023-10-14T18:17:42.245388+0000","last_peered":"2023-10-14T18:17:42.245388+0000","last_clean":"2023-10-14T18:17:42.245388+0000","last_became_active":"2023-10-14T18:17:26.766441+0000","last_became_peered":"2023-10-14T18:17:26.766441+0000","last_unstale":"2023-10-14T18:17:42.245388+0000","last_undegraded":"2023-10-14T18:17:42.245388+0000","last_fullsized":"2023-10-14T18:17:42.245388+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:34.151644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.254959+0000","last_change":"2023-10-14T18:17:27.267257+0000","last_active":"2023-10-14T18:17:42.254959+0000","last_peered":"2023-10-14T18:17:42.254959+0000","last_clean":"2023-10-14T18:17:42.254959+0000","last_became_active":"2023-10-14T18:17:27.266927+0000","last_became_peered":"2023-10-14T18:17:27.266927+0000","last_unstale":"2023-10-14T18:17:42.254959+0000","last_undegraded":"2023-10-14T18:17:42.254959+0000","last_fullsized":"2023-10-14T18:17:42.254959+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:43.360869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453273+0000","last_change":"2023-10-14T18:17:41.339830+0000","last_active":"2023-10-14T18:17:43.453273+0000","last_peered":"2023-10-14T18:17:43.453273+0000","last_clean":"2023-10-14T18:17:43.453273+0000","last_became_active":"2023-10-14T18:17:41.337933+0000","last_became_peered":"2023-10-14T18:17:41.337933+0000","last_unstale":"2023-10-14T18:17:43.453273+0000","last_undegraded":"2023-10-14T18:17:43.453273+0000","last_fullsized":"2023-10-14T18:17:43.453273+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:53.445917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.259249+0000","last_change":"2023-10-14T18:17:42.254340+0000","last_active":"2023-10-14T18:17:43.259249+0000","last_peered":"2023-10-14T18:17:43.259249+0000","last_clean":"2023-10-14T18:17:43.259249+0000","last_became_active":"2023-10-14T18:17:42.254061+0000","last_became_peered":"2023-10-14T18:17:42.254061+0000","last_unstale":"2023-10-14T18:17:43.259249+0000","last_undegraded":"2023-10-14T18:17:43.259249+0000","last_fullsized":"2023-10-14T18:17:43.259249+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236230+0000","last_change":"2023-10-14T18:17:41.346764+0000","last_active":"2023-10-14T18:17:43.236230+0000","last_peered":"2023-10-14T18:17:43.236230+0000","last_clean":"2023-10-14T18:17:43.236230+0000","last_became_active":"2023-10-14T18:17:41.346452+0000","last_became_peered":"2023-10-14T18:17:41.346452+0000","last_unstale":"2023-10-14T18:17:43.236230+0000","last_undegraded":"2023-10-14T18:17:43.236230+0000","last_fullsized":"2023-10-14T18:17:43.236230+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:51.462746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252678+0000","last_change":"2023-10-14T18:17:26.757809+0000","last_active":"2023-10-14T18:17:42.252678+0000","last_peered":"2023-10-14T18:17:42.252678+0000","last_clean":"2023-10-14T18:17:42.252678+0000","last_became_active":"2023-10-14T18:17:26.757324+0000","last_became_peered":"2023-10-14T18:17:26.757324+0000","last_unstale":"2023-10-14T18:17:42.252678+0000","last_undegraded":"2023-10-14T18:17:42.252678+0000","last_fullsized":"2023-10-14T18:17:42.252678+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:16.576762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245621+0000","last_change":"2023-10-14T18:17:27.266633+0000","last_active":"2023-10-14T18:17:42.245621+0000","last_peered":"2023-10-14T18:17:42.245621+0000","last_clean":"2023-10-14T18:17:42.245621+0000","last_became_active":"2023-10-14T18:17:27.265995+0000","last_became_peered":"2023-10-14T18:17:27.265995+0000","last_unstale":"2023-10-14T18:17:42.245621+0000","last_undegraded":"2023-10-14T18:17:42.245621+0000","last_fullsized":"2023-10-14T18:17:42.245621+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:13.429707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240998+0000","last_change":"2023-10-14T18:17:42.269999+0000","last_active":"2023-10-14T18:17:43.240998+0000","last_peered":"2023-10-14T18:17:43.240998+0000","last_clean":"2023-10-14T18:17:43.240998+0000","last_became_active":"2023-10-14T18:17:42.266497+0000","last_became_peered":"2023-10-14T18:17:42.266497+0000","last_unstale":"2023-10-14T18:17:43.240998+0000","last_undegraded":"2023-10-14T18:17:43.240998+0000","last_fullsized":"2023-10-14T18:17:43.240998+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:54.098372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"232'4","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454775+0000","last_change":"2023-10-14T18:17:41.340721+0000","last_active":"2023-10-14T18:17:43.454775+0000","last_peered":"2023-10-14T18:17:43.454775+0000","last_clean":"2023-10-14T18:17:43.454775+0000","last_became_active":"2023-10-14T18:17:41.340418+0000","last_became_peered":"2023-10-14T18:17:41.340418+0000","last_unstale":"2023-10-14T18:17:43.454775+0000","last_undegraded":"2023-10-14T18:17:43.454775+0000","last_fullsized":"2023-10-14T18:17:43.454775+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:02.098277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248662+0000","last_change":"2023-10-14T18:17:26.752387+0000","last_active":"2023-10-14T18:17:42.248662+0000","last_peered":"2023-10-14T18:17:42.248662+0000","last_clean":"2023-10-14T18:17:42.248662+0000","last_became_active":"2023-10-14T18:17:26.752097+0000","last_became_peered":"2023-10-14T18:17:26.752097+0000","last_unstale":"2023-10-14T18:17:42.248662+0000","last_undegraded":"2023-10-14T18:17:42.248662+0000","last_fullsized":"2023-10-14T18:17:42.248662+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:47.530809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240520+0000","last_change":"2023-10-14T18:17:27.263025+0000","last_active":"2023-10-14T18:17:42.240520+0000","last_peered":"2023-10-14T18:17:42.240520+0000","last_clean":"2023-10-14T18:17:42.240520+0000","last_became_active":"2023-10-14T18:17:27.262769+0000","last_became_peered":"2023-10-14T18:17:27.262769+0000","last_unstale":"2023-10-14T18:17:42.240520+0000","last_undegraded":"2023-10-14T18:17:42.240520+0000","last_fullsized":"2023-10-14T18:17:42.240520+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:32.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237401+0000","last_change":"2023-10-14T18:17:42.259060+0000","last_active":"2023-10-14T18:17:43.237401+0000","last_peered":"2023-10-14T18:17:43.237401+0000","last_clean":"2023-10-14T18:17:43.237401+0000","last_became_active":"2023-10-14T18:17:42.258764+0000","last_became_peered":"2023-10-14T18:17:42.258764+0000","last_unstale":"2023-10-14T18:17:43.237401+0000","last_undegraded":"2023-10-14T18:17:43.237401+0000","last_fullsized":"2023-10-14T18:17:43.237401+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:05.726762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247369+0000","last_change":"2023-10-14T18:17:26.758140+0000","last_active":"2023-10-14T18:17:42.247369+0000","last_peered":"2023-10-14T18:17:42.247369+0000","last_clean":"2023-10-14T18:17:42.247369+0000","last_became_active":"2023-10-14T18:17:26.757834+0000","last_became_peered":"2023-10-14T18:17:26.757834+0000","last_unstale":"2023-10-14T18:17:42.247369+0000","last_undegraded":"2023-10-14T18:17:42.247369+0000","last_fullsized":"2023-10-14T18:17:42.247369+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:39.799748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248853+0000","last_change":"2023-10-14T18:17:27.265679+0000","last_active":"2023-10-14T18:17:42.248853+0000","last_peered":"2023-10-14T18:17:42.248853+0000","last_clean":"2023-10-14T18:17:42.248853+0000","last_became_active":"2023-10-14T18:17:27.264828+0000","last_became_peered":"2023-10-14T18:17:27.264828+0000","last_unstale":"2023-10-14T18:17:42.248853+0000","last_undegraded":"2023-10-14T18:17:42.248853+0000","last_fullsized":"2023-10-14T18:17:42.248853+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:20:40.110815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234963+0000","last_change":"2023-10-14T18:17:41.345853+0000","last_active":"2023-10-14T18:17:43.234963+0000","last_peered":"2023-10-14T18:17:43.234963+0000","last_clean":"2023-10-14T18:17:43.234963+0000","last_became_active":"2023-10-14T18:17:41.345579+0000","last_became_peered":"2023-10-14T18:17:41.345579+0000","last_unstale":"2023-10-14T18:17:43.234963+0000","last_undegraded":"2023-10-14T18:17:43.234963+0000","last_fullsized":"2023-10-14T18:17:43.234963+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:45.715692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453961+0000","last_change":"2023-10-14T18:17:42.258755+0000","last_active":"2023-10-14T18:17:43.453961+0000","last_peered":"2023-10-14T18:17:43.453961+0000","last_clean":"2023-10-14T18:17:43.453961+0000","last_became_active":"2023-10-14T18:17:42.258427+0000","last_became_peered":"2023-10-14T18:17:42.258427+0000","last_unstale":"2023-10-14T18:17:43.453961+0000","last_undegraded":"2023-10-14T18:17:43.453961+0000","last_fullsized":"2023-10-14T18:17:43.453961+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:42.856504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249707+0000","last_change":"2023-10-14T18:17:26.763585+0000","last_active":"2023-10-14T18:17:42.249707+0000","last_peered":"2023-10-14T18:17:42.249707+0000","last_clean":"2023-10-14T18:17:42.249707+0000","last_became_active":"2023-10-14T18:17:26.763201+0000","last_became_peered":"2023-10-14T18:17:26.763201+0000","last_unstale":"2023-10-14T18:17:42.249707+0000","last_undegraded":"2023-10-14T18:17:42.249707+0000","last_fullsized":"2023-10-14T18:17:42.249707+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:50:47.243311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244065+0000","last_change":"2023-10-14T18:17:27.250058+0000","last_active":"2023-10-14T18:17:42.244065+0000","last_peered":"2023-10-14T18:17:42.244065+0000","last_clean":"2023-10-14T18:17:42.244065+0000","last_became_active":"2023-10-14T18:17:27.249763+0000","last_became_peered":"2023-10-14T18:17:27.249763+0000","last_unstale":"2023-10-14T18:17:42.244065+0000","last_undegraded":"2023-10-14T18:17:42.244065+0000","last_fullsized":"2023-10-14T18:17:42.244065+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:54.713237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239293+0000","last_change":"2023-10-14T18:17:41.331457+0000","last_active":"2023-10-14T18:17:43.239293+0000","last_peered":"2023-10-14T18:17:43.239293+0000","last_clean":"2023-10-14T18:17:43.239293+0000","last_became_active":"2023-10-14T18:17:41.331123+0000","last_became_peered":"2023-10-14T18:17:41.331123+0000","last_unstale":"2023-10-14T18:17:43.239293+0000","last_undegraded":"2023-10-14T18:17:43.239293+0000","last_fullsized":"2023-10-14T18:17:43.239293+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:14.870447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456949+0000","last_change":"2023-10-14T18:17:42.271854+0000","last_active":"2023-10-14T18:17:43.456949+0000","last_peered":"2023-10-14T18:17:43.456949+0000","last_clean":"2023-10-14T18:17:43.456949+0000","last_became_active":"2023-10-14T18:17:42.271342+0000","last_became_peered":"2023-10-14T18:17:42.271342+0000","last_unstale":"2023-10-14T18:17:43.456949+0000","last_undegraded":"2023-10-14T18:17:43.456949+0000","last_fullsized":"2023-10-14T18:17:43.456949+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:40:30.808342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231959+0000","last_change":"2023-10-14T18:17:26.760994+0000","last_active":"2023-10-14T18:17:42.231959+0000","last_peered":"2023-10-14T18:17:42.231959+0000","last_clean":"2023-10-14T18:17:42.231959+0000","last_became_active":"2023-10-14T18:17:26.760187+0000","last_became_peered":"2023-10-14T18:17:26.760187+0000","last_unstale":"2023-10-14T18:17:42.231959+0000","last_undegraded":"2023-10-14T18:17:42.231959+0000","last_fullsized":"2023-10-14T18:17:42.231959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:41.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252213+0000","last_change":"2023-10-14T18:17:27.261385+0000","last_active":"2023-10-14T18:17:42.252213+0000","last_peered":"2023-10-14T18:17:42.252213+0000","last_clean":"2023-10-14T18:17:42.252213+0000","last_became_active":"2023-10-14T18:17:27.261114+0000","last_became_peered":"2023-10-14T18:17:27.261114+0000","last_unstale":"2023-10-14T18:17:42.252213+0000","last_undegraded":"2023-10-14T18:17:42.252213+0000","last_fullsized":"2023-10-14T18:17:42.252213+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:29.463852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257464+0000","last_change":"2023-10-14T18:17:41.344653+0000","last_active":"2023-10-14T18:17:43.257464+0000","last_peered":"2023-10-14T18:17:43.257464+0000","last_clean":"2023-10-14T18:17:43.257464+0000","last_became_active":"2023-10-14T18:17:41.344370+0000","last_became_peered":"2023-10-14T18:17:41.344370+0000","last_unstale":"2023-10-14T18:17:43.257464+0000","last_undegraded":"2023-10-14T18:17:43.257464+0000","last_fullsized":"2023-10-14T18:17:43.257464+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233744+0000","last_change":"2023-10-14T18:17:42.265518+0000","last_active":"2023-10-14T18:17:43.233744+0000","last_peered":"2023-10-14T18:17:43.233744+0000","last_clean":"2023-10-14T18:17:43.233744+0000","last_became_active":"2023-10-14T18:17:42.265208+0000","last_became_peered":"2023-10-14T18:17:42.265208+0000","last_unstale":"2023-10-14T18:17:43.233744+0000","last_undegraded":"2023-10-14T18:17:43.233744+0000","last_fullsized":"2023-10-14T18:17:43.233744+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:28.087705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247383+0000","last_change":"2023-10-14T18:17:26.765153+0000","last_active":"2023-10-14T18:17:42.247383+0000","last_peered":"2023-10-14T18:17:42.247383+0000","last_clean":"2023-10-14T18:17:42.247383+0000","last_became_active":"2023-10-14T18:17:26.764789+0000","last_became_peered":"2023-10-14T18:17:26.764789+0000","last_unstale":"2023-10-14T18:17:42.247383+0000","last_undegraded":"2023-10-14T18:17:42.247383+0000","last_fullsized":"2023-10-14T18:17:42.247383+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:01.621072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255635+0000","last_change":"2023-10-14T18:17:27.251816+0000","last_active":"2023-10-14T18:17:42.255635+0000","last_peered":"2023-10-14T18:17:42.255635+0000","last_clean":"2023-10-14T18:17:42.255635+0000","last_became_active":"2023-10-14T18:17:27.250127+0000","last_became_peered":"2023-10-14T18:17:27.250127+0000","last_unstale":"2023-10-14T18:17:42.255635+0000","last_undegraded":"2023-10-14T18:17:42.255635+0000","last_fullsized":"2023-10-14T18:17:42.255635+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:32.974437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269045+0000","last_change":"2023-10-14T18:17:41.343616+0000","last_active":"2023-10-14T18:17:43.269045+0000","last_peered":"2023-10-14T18:17:43.269045+0000","last_clean":"2023-10-14T18:17:43.269045+0000","last_became_active":"2023-10-14T18:17:41.343340+0000","last_became_peered":"2023-10-14T18:17:41.343340+0000","last_unstale":"2023-10-14T18:17:43.269045+0000","last_undegraded":"2023-10-14T18:17:43.269045+0000","last_fullsized":"2023-10-14T18:17:43.269045+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:11.786755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.268914+0000","last_change":"2023-10-14T18:17:42.260145+0000","last_active":"2023-10-14T18:17:43.268914+0000","last_peered":"2023-10-14T18:17:43.268914+0000","last_clean":"2023-10-14T18:17:43.268914+0000","last_became_active":"2023-10-14T18:17:42.259850+0000","last_became_peered":"2023-10-14T18:17:42.259850+0000","last_unstale":"2023-10-14T18:17:43.268914+0000","last_undegraded":"2023-10-14T18:17:43.268914+0000","last_fullsized":"2023-10-14T18:17:43.268914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:12:26.151454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"228'4","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.239578+0000","last_change":"2023-10-14T18:17:26.768176+0000","last_active":"2023-10-14T18:17:42.239578+0000","last_peered":"2023-10-14T18:17:42.239578+0000","last_clean":"2023-10-14T18:17:42.239578+0000","last_became_active":"2023-10-14T18:17:26.767847+0000","last_became_peered":"2023-10-14T18:17:26.767847+0000","last_unstale":"2023-10-14T18:17:42.239578+0000","last_undegraded":"2023-10-14T18:17:42.239578+0000","last_fullsized":"2023-10-14T18:17:42.239578+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:11.998927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246896+0000","last_change":"2023-10-14T18:17:27.257929+0000","last_active":"2023-10-14T18:17:42.246896+0000","last_peered":"2023-10-14T18:17:42.246896+0000","last_clean":"2023-10-14T18:17:42.246896+0000","last_became_active":"2023-10-14T18:17:27.257029+0000","last_became_peered":"2023-10-14T18:17:27.257029+0000","last_unstale":"2023-10-14T18:17:42.246896+0000","last_undegraded":"2023-10-14T18:17:42.246896+0000","last_fullsized":"2023-10-14T18:17:42.246896+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236955+0000","last_change":"2023-10-14T18:17:41.340719+0000","last_active":"2023-10-14T18:17:43.236955+0000","last_peered":"2023-10-14T18:17:43.236955+0000","last_clean":"2023-10-14T18:17:43.236955+0000","last_became_active":"2023-10-14T18:17:41.340408+0000","last_became_peered":"2023-10-14T18:17:41.340408+0000","last_unstale":"2023-10-14T18:17:43.236955+0000","last_undegraded":"2023-10-14T18:17:43.236955+0000","last_fullsized":"2023-10-14T18:17:43.236955+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:54.022204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.258910+0000","last_change":"2023-10-14T18:17:42.257031+0000","last_active":"2023-10-14T18:17:43.258910+0000","last_peered":"2023-10-14T18:17:43.258910+0000","last_clean":"2023-10-14T18:17:43.258910+0000","last_became_active":"2023-10-14T18:17:42.256077+0000","last_became_peered":"2023-10-14T18:17:42.256077+0000","last_unstale":"2023-10-14T18:17:43.258910+0000","last_undegraded":"2023-10-14T18:17:43.258910+0000","last_fullsized":"2023-10-14T18:17:43.258910+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243897+0000","last_change":"2023-10-14T18:17:27.249693+0000","last_active":"2023-10-14T18:17:42.243897+0000","last_peered":"2023-10-14T18:17:42.243897+0000","last_clean":"2023-10-14T18:17:42.243897+0000","last_became_active":"2023-10-14T18:17:27.249386+0000","last_became_peered":"2023-10-14T18:17:27.249386+0000","last_unstale":"2023-10-14T18:17:42.243897+0000","last_undegraded":"2023-10-14T18:17:42.243897+0000","last_fullsized":"2023-10-14T18:17:42.243897+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458363+0000","last_change":"2023-10-14T18:17:42.267891+0000","last_active":"2023-10-14T18:17:43.458363+0000","last_peered":"2023-10-14T18:17:43.458363+0000","last_clean":"2023-10-14T18:17:43.458363+0000","last_became_active":"2023-10-14T18:17:42.267622+0000","last_became_peered":"2023-10-14T18:17:42.267622+0000","last_unstale":"2023-10-14T18:17:43.458363+0000","last_undegraded":"2023-10-14T18:17:43.458363+0000","last_fullsized":"2023-10-14T18:17:43.458363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:57.010924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243320+0000","last_change":"2023-10-14T18:17:27.264821+0000","last_active":"2023-10-14T18:17:42.243320+0000","last_peered":"2023-10-14T18:17:42.243320+0000","last_clean":"2023-10-14T18:17:42.243320+0000","last_became_active":"2023-10-14T18:17:27.264504+0000","last_became_peered":"2023-10-14T18:17:27.264504+0000","last_unstale":"2023-10-14T18:17:42.243320+0000","last_undegraded":"2023-10-14T18:17:42.243320+0000","last_fullsized":"2023-10-14T18:17:42.243320+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:44.865727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269575+0000","last_change":"2023-10-14T18:17:42.261418+0000","last_active":"2023-10-14T18:17:43.269575+0000","last_peered":"2023-10-14T18:17:43.269575+0000","last_clean":"2023-10-14T18:17:43.269575+0000","last_became_active":"2023-10-14T18:17:42.259881+0000","last_became_peered":"2023-10-14T18:17:42.259881+0000","last_unstale":"2023-10-14T18:17:43.269575+0000","last_undegraded":"2023-10-14T18:17:43.269575+0000","last_fullsized":"2023-10-14T18:17:43.269575+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:39:51.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244547+0000","last_change":"2023-10-14T18:17:27.260534+0000","last_active":"2023-10-14T18:17:42.244547+0000","last_peered":"2023-10-14T18:17:42.244547+0000","last_clean":"2023-10-14T18:17:42.244547+0000","last_became_active":"2023-10-14T18:17:27.259981+0000","last_became_peered":"2023-10-14T18:17:27.259981+0000","last_unstale":"2023-10-14T18:17:42.244547+0000","last_undegraded":"2023-10-14T18:17:42.244547+0000","last_fullsized":"2023-10-14T18:17:42.244547+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:21.289738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457765+0000","last_change":"2023-10-14T18:17:42.267709+0000","last_active":"2023-10-14T18:17:43.457765+0000","last_peered":"2023-10-14T18:17:43.457765+0000","last_clean":"2023-10-14T18:17:43.457765+0000","last_became_active":"2023-10-14T18:17:42.267264+0000","last_became_peered":"2023-10-14T18:17:42.267264+0000","last_unstale":"2023-10-14T18:17:43.457765+0000","last_undegraded":"2023-10-14T18:17:43.457765+0000","last_fullsized":"2023-10-14T18:17:43.457765+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:21.177190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249937+0000","last_change":"2023-10-14T18:17:27.256462+0000","last_active":"2023-10-14T18:17:42.249937+0000","last_peered":"2023-10-14T18:17:42.249937+0000","last_clean":"2023-10-14T18:17:42.249937+0000","last_became_active":"2023-10-14T18:17:27.256190+0000","last_became_peered":"2023-10-14T18:17:27.256190+0000","last_unstale":"2023-10-14T18:17:42.249937+0000","last_undegraded":"2023-10-14T18:17:42.249937+0000","last_fullsized":"2023-10-14T18:17:42.249937+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:42.705277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456828+0000","last_change":"2023-10-14T18:17:42.259936+0000","last_active":"2023-10-14T18:17:43.456828+0000","last_peered":"2023-10-14T18:17:43.456828+0000","last_clean":"2023-10-14T18:17:43.456828+0000","last_became_active":"2023-10-14T18:17:42.259636+0000","last_became_peered":"2023-10-14T18:17:42.259636+0000","last_unstale":"2023-10-14T18:17:43.456828+0000","last_undegraded":"2023-10-14T18:17:43.456828+0000","last_fullsized":"2023-10-14T18:17:43.456828+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:07.931521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.229930+0000","last_change":"2023-10-14T18:17:27.241163+0000","last_active":"2023-10-14T18:17:42.229930+0000","last_peered":"2023-10-14T18:17:42.229930+0000","last_clean":"2023-10-14T18:17:42.229930+0000","last_became_active":"2023-10-14T18:17:27.240705+0000","last_became_peered":"2023-10-14T18:17:27.240705+0000","last_unstale":"2023-10-14T18:17:42.229930+0000","last_undegraded":"2023-10-14T18:17:42.229930+0000","last_fullsized":"2023-10-14T18:17:42.229930+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237018+0000","last_change":"2023-10-14T18:17:42.258987+0000","last_active":"2023-10-14T18:17:43.237018+0000","last_peered":"2023-10-14T18:17:43.237018+0000","last_clean":"2023-10-14T18:17:43.237018+0000","last_became_active":"2023-10-14T18:17:42.258689+0000","last_became_peered":"2023-10-14T18:17:42.258689+0000","last_unstale":"2023-10-14T18:17:43.237018+0000","last_undegraded":"2023-10-14T18:17:43.237018+0000","last_fullsized":"2023-10-14T18:17:43.237018+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:56.138282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.238601+0000","last_change":"2023-10-14T18:17:27.255856+0000","last_active":"2023-10-14T18:17:42.238601+0000","last_peered":"2023-10-14T18:17:42.238601+0000","last_clean":"2023-10-14T18:17:42.238601+0000","last_became_active":"2023-10-14T18:17:27.255581+0000","last_became_peered":"2023-10-14T18:17:27.255581+0000","last_unstale":"2023-10-14T18:17:42.238601+0000","last_undegraded":"2023-10-14T18:17:42.238601+0000","last_fullsized":"2023-10-14T18:17:42.238601+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:42:42.540612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453150+0000","last_change":"2023-10-14T18:17:42.265216+0000","last_active":"2023-10-14T18:17:43.453150+0000","last_peered":"2023-10-14T18:17:43.453150+0000","last_clean":"2023-10-14T18:17:43.453150+0000","last_became_active":"2023-10-14T18:17:42.264773+0000","last_became_peered":"2023-10-14T18:17:42.264773+0000","last_unstale":"2023-10-14T18:17:43.453150+0000","last_undegraded":"2023-10-14T18:17:43.453150+0000","last_fullsized":"2023-10-14T18:17:43.453150+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252606+0000","last_change":"2023-10-14T18:17:27.262030+0000","last_active":"2023-10-14T18:17:42.252606+0000","last_peered":"2023-10-14T18:17:42.252606+0000","last_clean":"2023-10-14T18:17:42.252606+0000","last_became_active":"2023-10-14T18:17:27.261661+0000","last_became_peered":"2023-10-14T18:17:27.261661+0000","last_unstale":"2023-10-14T18:17:42.252606+0000","last_undegraded":"2023-10-14T18:17:42.252606+0000","last_fullsized":"2023-10-14T18:17:42.252606+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:42.391926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453157+0000","last_change":"2023-10-14T18:17:42.254749+0000","last_active":"2023-10-14T18:17:43.453157+0000","last_peered":"2023-10-14T18:17:43.453157+0000","last_clean":"2023-10-14T18:17:43.453157+0000","last_became_active":"2023-10-14T18:17:42.254204+0000","last_became_peered":"2023-10-14T18:17:42.254204+0000","last_unstale":"2023-10-14T18:17:43.453157+0000","last_undegraded":"2023-10-14T18:17:43.453157+0000","last_fullsized":"2023-10-14T18:17:43.453157+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:34:49.685522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249844+0000","last_change":"2023-10-14T18:17:27.261126+0000","last_active":"2023-10-14T18:17:42.249844+0000","last_peered":"2023-10-14T18:17:42.249844+0000","last_clean":"2023-10-14T18:17:42.249844+0000","last_became_active":"2023-10-14T18:17:27.260846+0000","last_became_peered":"2023-10-14T18:17:27.260846+0000","last_unstale":"2023-10-14T18:17:42.249844+0000","last_undegraded":"2023-10-14T18:17:42.249844+0000","last_fullsized":"2023-10-14T18:17:42.249844+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269914+0000","last_change":"2023-10-14T18:17:42.263605+0000","last_active":"2023-10-14T18:17:43.269914+0000","last_peered":"2023-10-14T18:17:43.269914+0000","last_clean":"2023-10-14T18:17:43.269914+0000","last_became_active":"2023-10-14T18:17:42.263265+0000","last_became_peered":"2023-10-14T18:17:42.263265+0000","last_unstale":"2023-10-14T18:17:43.269914+0000","last_undegraded":"2023-10-14T18:17:43.269914+0000","last_fullsized":"2023-10-14T18:17:43.269914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:08:51.792734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243069+0000","last_change":"2023-10-14T18:17:27.262932+0000","last_active":"2023-10-14T18:17:42.243069+0000","last_peered":"2023-10-14T18:17:42.243069+0000","last_clean":"2023-10-14T18:17:42.243069+0000","last_became_active":"2023-10-14T18:17:27.262582+0000","last_became_peered":"2023-10-14T18:17:27.262582+0000","last_unstale":"2023-10-14T18:17:42.243069+0000","last_undegraded":"2023-10-14T18:17:42.243069+0000","last_fullsized":"2023-10-14T18:17:42.243069+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:01:50.613186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237399+0000","last_change":"2023-10-14T18:17:42.252701+0000","last_active":"2023-10-14T18:17:43.237399+0000","last_peered":"2023-10-14T18:17:43.237399+0000","last_clean":"2023-10-14T18:17:43.237399+0000","last_became_active":"2023-10-14T18:17:42.251939+0000","last_became_peered":"2023-10-14T18:17:42.251939+0000","last_unstale":"2023-10-14T18:17:43.237399+0000","last_undegraded":"2023-10-14T18:17:43.237399+0000","last_fullsized":"2023-10-14T18:17:43.237399+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:04.430721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252882+0000","last_change":"2023-10-14T18:17:27.257364+0000","last_active":"2023-10-14T18:17:42.252882+0000","last_peered":"2023-10-14T18:17:42.252882+0000","last_clean":"2023-10-14T18:17:42.252882+0000","last_became_active":"2023-10-14T18:17:27.257009+0000","last_became_peered":"2023-10-14T18:17:27.257009+0000","last_unstale":"2023-10-14T18:17:42.252882+0000","last_undegraded":"2023-10-14T18:17:42.252882+0000","last_fullsized":"2023-10-14T18:17:42.252882+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:24.487542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269569+0000","last_change":"2023-10-14T18:17:42.259210+0000","last_active":"2023-10-14T18:17:43.269569+0000","last_peered":"2023-10-14T18:17:43.269569+0000","last_clean":"2023-10-14T18:17:43.269569+0000","last_became_active":"2023-10-14T18:17:42.256587+0000","last_became_peered":"2023-10-14T18:17:42.256587+0000","last_unstale":"2023-10-14T18:17:43.269569+0000","last_undegraded":"2023-10-14T18:17:43.269569+0000","last_fullsized":"2023-10-14T18:17:43.269569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:00.107914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255450+0000","last_change":"2023-10-14T18:17:27.256296+0000","last_active":"2023-10-14T18:17:42.255450+0000","last_peered":"2023-10-14T18:17:42.255450+0000","last_clean":"2023-10-14T18:17:42.255450+0000","last_became_active":"2023-10-14T18:17:27.255979+0000","last_became_peered":"2023-10-14T18:17:27.255979+0000","last_unstale":"2023-10-14T18:17:42.255450+0000","last_undegraded":"2023-10-14T18:17:42.255450+0000","last_fullsized":"2023-10-14T18:17:42.255450+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:53.896784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454798+0000","last_change":"2023-10-14T18:17:42.262666+0000","last_active":"2023-10-14T18:17:43.454798+0000","last_peered":"2023-10-14T18:17:43.454798+0000","last_clean":"2023-10-14T18:17:43.454798+0000","last_became_active":"2023-10-14T18:17:42.262316+0000","last_became_peered":"2023-10-14T18:17:42.262316+0000","last_unstale":"2023-10-14T18:17:43.454798+0000","last_undegraded":"2023-10-14T18:17:43.454798+0000","last_fullsized":"2023-10-14T18:17:43.454798+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:18.884106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248724+0000","last_change":"2023-10-14T18:17:27.260919+0000","last_active":"2023-10-14T18:17:42.248724+0000","last_peered":"2023-10-14T18:17:42.248724+0000","last_clean":"2023-10-14T18:17:42.248724+0000","last_became_active":"2023-10-14T18:17:27.260634+0000","last_became_peered":"2023-10-14T18:17:27.260634+0000","last_unstale":"2023-10-14T18:17:42.248724+0000","last_undegraded":"2023-10-14T18:17:42.248724+0000","last_fullsized":"2023-10-14T18:17:42.248724+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:51:46.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454032+0000","last_change":"2023-10-14T18:17:42.260933+0000","last_active":"2023-10-14T18:17:43.454032+0000","last_peered":"2023-10-14T18:17:43.454032+0000","last_clean":"2023-10-14T18:17:43.454032+0000","last_became_active":"2023-10-14T18:17:42.260634+0000","last_became_peered":"2023-10-14T18:17:42.260634+0000","last_unstale":"2023-10-14T18:17:43.454032+0000","last_undegraded":"2023-10-14T18:17:43.454032+0000","last_fullsized":"2023-10-14T18:17:43.454032+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:31.063452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249841+0000","last_change":"2023-10-14T18:17:27.252557+0000","last_active":"2023-10-14T18:17:42.249841+0000","last_peered":"2023-10-14T18:17:42.249841+0000","last_clean":"2023-10-14T18:17:42.249841+0000","last_became_active":"2023-10-14T18:17:27.251949+0000","last_became_peered":"2023-10-14T18:17:27.251949+0000","last_unstale":"2023-10-14T18:17:42.249841+0000","last_undegraded":"2023-10-14T18:17:42.249841+0000","last_fullsized":"2023-10-14T18:17:42.249841+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:45.784134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453668+0000","last_change":"2023-10-14T18:17:42.270868+0000","last_active":"2023-10-14T18:17:43.453668+0000","last_peered":"2023-10-14T18:17:43.453668+0000","last_clean":"2023-10-14T18:17:43.453668+0000","last_became_active":"2023-10-14T18:17:42.270540+0000","last_became_peered":"2023-10-14T18:17:42.270540+0000","last_unstale":"2023-10-14T18:17:43.453668+0000","last_undegraded":"2023-10-14T18:17:43.453668+0000","last_fullsized":"2023-10-14T18:17:43.453668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:02.830268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246899+0000","last_change":"2023-10-14T18:17:27.255618+0000","last_active":"2023-10-14T18:17:42.246899+0000","last_peered":"2023-10-14T18:17:42.246899+0000","last_clean":"2023-10-14T18:17:42.246899+0000","last_became_active":"2023-10-14T18:17:27.255338+0000","last_became_peered":"2023-10-14T18:17:27.255338+0000","last_unstale":"2023-10-14T18:17:42.246899+0000","last_undegraded":"2023-10-14T18:17:42.246899+0000","last_fullsized":"2023-10-14T18:17:42.246899+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:06.792999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.452905+0000","last_change":"2023-10-14T18:17:42.264873+0000","last_active":"2023-10-14T18:17:43.452905+0000","last_peered":"2023-10-14T18:17:43.452905+0000","last_clean":"2023-10-14T18:17:43.452905+0000","last_became_active":"2023-10-14T18:17:42.263922+0000","last_became_peered":"2023-10-14T18:17:42.263922+0000","last_unstale":"2023-10-14T18:17:43.452905+0000","last_undegraded":"2023-10-14T18:17:43.452905+0000","last_fullsized":"2023-10-14T18:17:43.452905+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:07:38.376498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247417+0000","last_change":"2023-10-14T18:17:27.251453+0000","last_active":"2023-10-14T18:17:42.247417+0000","last_peered":"2023-10-14T18:17:42.247417+0000","last_clean":"2023-10-14T18:17:42.247417+0000","last_became_active":"2023-10-14T18:17:27.247696+0000","last_became_peered":"2023-10-14T18:17:27.247696+0000","last_unstale":"2023-10-14T18:17:42.247417+0000","last_undegraded":"2023-10-14T18:17:42.247417+0000","last_fullsized":"2023-10-14T18:17:42.247417+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:43:29.058123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456508+0000","last_change":"2023-10-14T18:17:42.256913+0000","last_active":"2023-10-14T18:17:43.456508+0000","last_peered":"2023-10-14T18:17:43.456508+0000","last_clean":"2023-10-14T18:17:43.456508+0000","last_became_active":"2023-10-14T18:17:42.253387+0000","last_became_peered":"2023-10-14T18:17:42.253387+0000","last_unstale":"2023-10-14T18:17:43.456508+0000","last_undegraded":"2023-10-14T18:17:43.456508+0000","last_fullsized":"2023-10-14T18:17:43.456508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:37:17.417744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253511+0000","last_change":"2023-10-14T18:17:27.265286+0000","last_active":"2023-10-14T18:17:42.253511+0000","last_peered":"2023-10-14T18:17:42.253511+0000","last_clean":"2023-10-14T18:17:42.253511+0000","last_became_active":"2023-10-14T18:17:27.265058+0000","last_became_peered":"2023-10-14T18:17:27.265058+0000","last_unstale":"2023-10-14T18:17:42.253511+0000","last_undegraded":"2023-10-14T18:17:42.253511+0000","last_fullsized":"2023-10-14T18:17:42.253511+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269714+0000","last_change":"2023-10-14T18:17:42.259485+0000","last_active":"2023-10-14T18:17:43.269714+0000","last_peered":"2023-10-14T18:17:43.269714+0000","last_clean":"2023-10-14T18:17:43.269714+0000","last_became_active":"2023-10-14T18:17:42.258846+0000","last_became_peered":"2023-10-14T18:17:42.258846+0000","last_unstale":"2023-10-14T18:17:43.269714+0000","last_undegraded":"2023-10-14T18:17:43.269714+0000","last_fullsized":"2023-10-14T18:17:43.269714+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:47:14.232149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244517+0000","last_change":"2023-10-14T18:17:27.251446+0000","last_active":"2023-10-14T18:17:42.244517+0000","last_peered":"2023-10-14T18:17:42.244517+0000","last_clean":"2023-10-14T18:17:42.244517+0000","last_became_active":"2023-10-14T18:17:27.251165+0000","last_became_peered":"2023-10-14T18:17:27.251165+0000","last_unstale":"2023-10-14T18:17:42.244517+0000","last_undegraded":"2023-10-14T18:17:42.244517+0000","last_fullsized":"2023-10-14T18:17:42.244517+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:35:33.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241031+0000","last_change":"2023-10-14T18:17:42.262549+0000","last_active":"2023-10-14T18:17:43.241031+0000","last_peered":"2023-10-14T18:17:43.241031+0000","last_clean":"2023-10-14T18:17:43.241031+0000","last_became_active":"2023-10-14T18:17:42.261856+0000","last_became_peered":"2023-10-14T18:17:42.261856+0000","last_unstale":"2023-10-14T18:17:43.241031+0000","last_undegraded":"2023-10-14T18:17:43.241031+0000","last_fullsized":"2023-10-14T18:17:43.241031+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:49.738601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247774+0000","last_change":"2023-10-14T18:17:27.259030+0000","last_active":"2023-10-14T18:17:42.247774+0000","last_peered":"2023-10-14T18:17:42.247774+0000","last_clean":"2023-10-14T18:17:42.247774+0000","last_became_active":"2023-10-14T18:17:27.257326+0000","last_became_peered":"2023-10-14T18:17:27.257326+0000","last_unstale":"2023-10-14T18:17:42.247774+0000","last_undegraded":"2023-10-14T18:17:42.247774+0000","last_fullsized":"2023-10-14T18:17:42.247774+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234715+0000","last_change":"2023-10-14T18:17:42.263054+0000","last_active":"2023-10-14T18:17:43.234715+0000","last_peered":"2023-10-14T18:17:43.234715+0000","last_clean":"2023-10-14T18:17:43.234715+0000","last_became_active":"2023-10-14T18:17:42.262632+0000","last_became_peered":"2023-10-14T18:17:42.262632+0000","last_unstale":"2023-10-14T18:17:43.234715+0000","last_undegraded":"2023-10-14T18:17:43.234715+0000","last_fullsized":"2023-10-14T18:17:43.234715+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:00.353186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244716+0000","last_change":"2023-10-14T18:17:27.253619+0000","last_active":"2023-10-14T18:17:42.244716+0000","last_peered":"2023-10-14T18:17:42.244716+0000","last_clean":"2023-10-14T18:17:42.244716+0000","last_became_active":"2023-10-14T18:17:27.253369+0000","last_became_peered":"2023-10-14T18:17:27.253369+0000","last_unstale":"2023-10-14T18:17:42.244716+0000","last_undegraded":"2023-10-14T18:17:42.244716+0000","last_fullsized":"2023-10-14T18:17:42.244716+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457410+0000","last_change":"2023-10-14T18:17:42.267530+0000","last_active":"2023-10-14T18:17:43.457410+0000","last_peered":"2023-10-14T18:17:43.457410+0000","last_clean":"2023-10-14T18:17:43.457410+0000","last_became_active":"2023-10-14T18:17:42.266908+0000","last_became_peered":"2023-10-14T18:17:42.266908+0000","last_unstale":"2023-10-14T18:17:43.457410+0000","last_undegraded":"2023-10-14T18:17:43.457410+0000","last_fullsized":"2023-10-14T18:17:43.457410+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:19.208285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250116+0000","last_change":"2023-10-14T18:17:27.253928+0000","last_active":"2023-10-14T18:17:42.250116+0000","last_peered":"2023-10-14T18:17:42.250116+0000","last_clean":"2023-10-14T18:17:42.250116+0000","last_became_active":"2023-10-14T18:17:27.253612+0000","last_became_peered":"2023-10-14T18:17:27.253612+0000","last_unstale":"2023-10-14T18:17:42.250116+0000","last_undegraded":"2023-10-14T18:17:42.250116+0000","last_fullsized":"2023-10-14T18:17:42.250116+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:02.638978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237879+0000","last_change":"2023-10-14T18:17:42.258988+0000","last_active":"2023-10-14T18:17:43.237879+0000","last_peered":"2023-10-14T18:17:43.237879+0000","last_clean":"2023-10-14T18:17:43.237879+0000","last_became_active":"2023-10-14T18:17:42.258551+0000","last_became_peered":"2023-10-14T18:17:42.258551+0000","last_unstale":"2023-10-14T18:17:43.237879+0000","last_undegraded":"2023-10-14T18:17:43.237879+0000","last_fullsized":"2023-10-14T18:17:43.237879+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:50.707802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240588+0000","last_change":"2023-10-14T18:17:27.253472+0000","last_active":"2023-10-14T18:17:42.240588+0000","last_peered":"2023-10-14T18:17:42.240588+0000","last_clean":"2023-10-14T18:17:42.240588+0000","last_became_active":"2023-10-14T18:17:27.253163+0000","last_became_peered":"2023-10-14T18:17:27.253163+0000","last_unstale":"2023-10-14T18:17:42.240588+0000","last_undegraded":"2023-10-14T18:17:42.240588+0000","last_fullsized":"2023-10-14T18:17:42.240588+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:14.392664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234096+0000","last_change":"2023-10-14T18:17:42.268853+0000","last_active":"2023-10-14T18:17:43.234096+0000","last_peered":"2023-10-14T18:17:43.234096+0000","last_clean":"2023-10-14T18:17:43.234096+0000","last_became_active":"2023-10-14T18:17:42.266796+0000","last_became_peered":"2023-10-14T18:17:42.266796+0000","last_unstale":"2023-10-14T18:17:43.234096+0000","last_undegraded":"2023-10-14T18:17:43.234096+0000","last_fullsized":"2023-10-14T18:17:43.234096+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:34.576269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231671+0000","last_change":"2023-10-14T18:17:27.249208+0000","last_active":"2023-10-14T18:17:42.231671+0000","last_peered":"2023-10-14T18:17:42.231671+0000","last_clean":"2023-10-14T18:17:42.231671+0000","last_became_active":"2023-10-14T18:17:27.248955+0000","last_became_peered":"2023-10-14T18:17:27.248955+0000","last_unstale":"2023-10-14T18:17:42.231671+0000","last_undegraded":"2023-10-14T18:17:42.231671+0000","last_fullsized":"2023-10-14T18:17:42.231671+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:13:04.646143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233962+0000","last_change":"2023-10-14T18:17:42.263050+0000","last_active":"2023-10-14T18:17:43.233962+0000","last_peered":"2023-10-14T18:17:43.233962+0000","last_clean":"2023-10-14T18:17:43.233962+0000","last_became_active":"2023-10-14T18:17:42.262740+0000","last_became_peered":"2023-10-14T18:17:42.262740+0000","last_unstale":"2023-10-14T18:17:43.233962+0000","last_undegraded":"2023-10-14T18:17:43.233962+0000","last_fullsized":"2023-10-14T18:17:43.233962+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:57.735230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247226+0000","last_change":"2023-10-14T18:17:27.254298+0000","last_active":"2023-10-14T18:17:42.247226+0000","last_peered":"2023-10-14T18:17:42.247226+0000","last_clean":"2023-10-14T18:17:42.247226+0000","last_became_active":"2023-10-14T18:17:27.254041+0000","last_became_peered":"2023-10-14T18:17:27.254041+0000","last_unstale":"2023-10-14T18:17:42.247226+0000","last_undegraded":"2023-10-14T18:17:42.247226+0000","last_fullsized":"2023-10-14T18:17:42.247226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:19:48.453168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.452948+0000","last_change":"2023-10-14T18:17:42.265235+0000","last_active":"2023-10-14T18:17:43.452948+0000","last_peered":"2023-10-14T18:17:43.452948+0000","last_clean":"2023-10-14T18:17:43.452948+0000","last_became_active":"2023-10-14T18:17:42.264946+0000","last_became_peered":"2023-10-14T18:17:42.264946+0000","last_unstale":"2023-10-14T18:17:43.452948+0000","last_undegraded":"2023-10-14T18:17:43.452948+0000","last_fullsized":"2023-10-14T18:17:43.452948+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250075+0000","last_change":"2023-10-14T18:17:27.266296+0000","last_active":"2023-10-14T18:17:42.250075+0000","last_peered":"2023-10-14T18:17:42.250075+0000","last_clean":"2023-10-14T18:17:42.250075+0000","last_became_active":"2023-10-14T18:17:27.265771+0000","last_became_peered":"2023-10-14T18:17:27.265771+0000","last_unstale":"2023-10-14T18:17:42.250075+0000","last_undegraded":"2023-10-14T18:17:42.250075+0000","last_fullsized":"2023-10-14T18:17:42.250075+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236832+0000","last_change":"2023-10-14T18:17:42.271663+0000","last_active":"2023-10-14T18:17:43.236832+0000","last_peered":"2023-10-14T18:17:43.236832+0000","last_clean":"2023-10-14T18:17:43.236832+0000","last_became_active":"2023-10-14T18:17:42.271303+0000","last_became_peered":"2023-10-14T18:17:42.271303+0000","last_unstale":"2023-10-14T18:17:43.236832+0000","last_undegraded":"2023-10-14T18:17:43.236832+0000","last_fullsized":"2023-10-14T18:17:43.236832+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:03:38.279151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240298+0000","last_change":"2023-10-14T18:17:27.246438+0000","last_active":"2023-10-14T18:17:42.240298+0000","last_peered":"2023-10-14T18:17:42.240298+0000","last_clean":"2023-10-14T18:17:42.240298+0000","last_became_active":"2023-10-14T18:17:27.245859+0000","last_became_peered":"2023-10-14T18:17:27.245859+0000","last_unstale":"2023-10-14T18:17:42.240298+0000","last_undegraded":"2023-10-14T18:17:42.240298+0000","last_fullsized":"2023-10-14T18:17:42.240298+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:20.007218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458360+0000","last_change":"2023-10-14T18:17:42.264577+0000","last_active":"2023-10-14T18:17:43.458360+0000","last_peered":"2023-10-14T18:17:43.458360+0000","last_clean":"2023-10-14T18:17:43.458360+0000","last_became_active":"2023-10-14T18:17:42.264261+0000","last_became_peered":"2023-10-14T18:17:42.264261+0000","last_unstale":"2023-10-14T18:17:43.458360+0000","last_undegraded":"2023-10-14T18:17:43.458360+0000","last_fullsized":"2023-10-14T18:17:43.458360+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:21.430028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247572+0000","last_change":"2023-10-14T18:17:27.260365+0000","last_active":"2023-10-14T18:17:42.247572+0000","last_peered":"2023-10-14T18:17:42.247572+0000","last_clean":"2023-10-14T18:17:42.247572+0000","last_became_active":"2023-10-14T18:17:27.260109+0000","last_became_peered":"2023-10-14T18:17:27.260109+0000","last_unstale":"2023-10-14T18:17:42.247572+0000","last_undegraded":"2023-10-14T18:17:42.247572+0000","last_fullsized":"2023-10-14T18:17:42.247572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454388+0000","last_change":"2023-10-14T18:17:42.267579+0000","last_active":"2023-10-14T18:17:43.454388+0000","last_peered":"2023-10-14T18:17:43.454388+0000","last_clean":"2023-10-14T18:17:43.454388+0000","last_became_active":"2023-10-14T18:17:42.267166+0000","last_became_peered":"2023-10-14T18:17:42.267166+0000","last_unstale":"2023-10-14T18:17:43.454388+0000","last_undegraded":"2023-10-14T18:17:43.454388+0000","last_fullsized":"2023-10-14T18:17:43.454388+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:41:21.894224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247062+0000","last_change":"2023-10-14T18:17:27.257362+0000","last_active":"2023-10-14T18:17:42.247062+0000","last_peered":"2023-10-14T18:17:42.247062+0000","last_clean":"2023-10-14T18:17:42.247062+0000","last_became_active":"2023-10-14T18:17:27.256429+0000","last_became_peered":"2023-10-14T18:17:27.256429+0000","last_unstale":"2023-10-14T18:17:42.247062+0000","last_undegraded":"2023-10-14T18:17:42.247062+0000","last_fullsized":"2023-10-14T18:17:42.247062+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:13.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239893+0000","last_change":"2023-10-14T18:17:42.255806+0000","last_active":"2023-10-14T18:17:43.239893+0000","last_peered":"2023-10-14T18:17:43.239893+0000","last_clean":"2023-10-14T18:17:43.239893+0000","last_became_active":"2023-10-14T18:17:42.255525+0000","last_became_peered":"2023-10-14T18:17:42.255525+0000","last_unstale":"2023-10-14T18:17:43.239893+0000","last_undegraded":"2023-10-14T18:17:43.239893+0000","last_fullsized":"2023-10-14T18:17:43.239893+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248951+0000","last_change":"2023-10-14T18:17:27.255198+0000","last_active":"2023-10-14T18:17:42.248951+0000","last_peered":"2023-10-14T18:17:42.248951+0000","last_clean":"2023-10-14T18:17:42.248951+0000","last_became_active":"2023-10-14T18:17:27.254906+0000","last_became_peered":"2023-10-14T18:17:27.254906+0000","last_unstale":"2023-10-14T18:17:42.248951+0000","last_undegraded":"2023-10-14T18:17:42.248951+0000","last_fullsized":"2023-10-14T18:17:42.248951+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:37.913184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269089+0000","last_change":"2023-10-14T18:17:42.262376+0000","last_active":"2023-10-14T18:17:43.269089+0000","last_peered":"2023-10-14T18:17:43.269089+0000","last_clean":"2023-10-14T18:17:43.269089+0000","last_became_active":"2023-10-14T18:17:42.262080+0000","last_became_peered":"2023-10-14T18:17:42.262080+0000","last_unstale":"2023-10-14T18:17:43.269089+0000","last_undegraded":"2023-10-14T18:17:43.269089+0000","last_fullsized":"2023-10-14T18:17:43.269089+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:29:07.397894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249439+0000","last_change":"2023-10-14T18:17:27.266341+0000","last_active":"2023-10-14T18:17:42.249439+0000","last_peered":"2023-10-14T18:17:42.249439+0000","last_clean":"2023-10-14T18:17:42.249439+0000","last_became_active":"2023-10-14T18:17:27.265032+0000","last_became_peered":"2023-10-14T18:17:27.265032+0000","last_unstale":"2023-10-14T18:17:42.249439+0000","last_undegraded":"2023-10-14T18:17:42.249439+0000","last_fullsized":"2023-10-14T18:17:42.249439+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:53.877988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453757+0000","last_change":"2023-10-14T18:17:42.259554+0000","last_active":"2023-10-14T18:17:43.453757+0000","last_peered":"2023-10-14T18:17:43.453757+0000","last_clean":"2023-10-14T18:17:43.453757+0000","last_became_active":"2023-10-14T18:17:42.259239+0000","last_became_peered":"2023-10-14T18:17:42.259239+0000","last_unstale":"2023-10-14T18:17:43.453757+0000","last_undegraded":"2023-10-14T18:17:43.453757+0000","last_fullsized":"2023-10-14T18:17:43.453757+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:06.131503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231667+0000","last_change":"2023-10-14T18:17:27.266231+0000","last_active":"2023-10-14T18:17:42.231667+0000","last_peered":"2023-10-14T18:17:42.231667+0000","last_clean":"2023-10-14T18:17:42.231667+0000","last_became_active":"2023-10-14T18:17:27.266037+0000","last_became_peered":"2023-10-14T18:17:27.266037+0000","last_unstale":"2023-10-14T18:17:42.231667+0000","last_undegraded":"2023-10-14T18:17:42.231667+0000","last_fullsized":"2023-10-14T18:17:42.231667+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:55.110847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257562+0000","last_change":"2023-10-14T18:17:42.253138+0000","last_active":"2023-10-14T18:17:43.257562+0000","last_peered":"2023-10-14T18:17:43.257562+0000","last_clean":"2023-10-14T18:17:43.257562+0000","last_became_active":"2023-10-14T18:17:42.252767+0000","last_became_peered":"2023-10-14T18:17:42.252767+0000","last_unstale":"2023-10-14T18:17:43.257562+0000","last_undegraded":"2023-10-14T18:17:43.257562+0000","last_fullsized":"2023-10-14T18:17:43.257562+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246730+0000","last_change":"2023-10-14T18:17:27.261872+0000","last_active":"2023-10-14T18:17:42.246730+0000","last_peered":"2023-10-14T18:17:42.246730+0000","last_clean":"2023-10-14T18:17:42.246730+0000","last_became_active":"2023-10-14T18:17:27.261591+0000","last_became_peered":"2023-10-14T18:17:27.261591+0000","last_unstale":"2023-10-14T18:17:42.246730+0000","last_undegraded":"2023-10-14T18:17:42.246730+0000","last_fullsized":"2023-10-14T18:17:42.246730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240089+0000","last_change":"2023-10-14T18:17:42.254646+0000","last_active":"2023-10-14T18:17:43.240089+0000","last_peered":"2023-10-14T18:17:43.240089+0000","last_clean":"2023-10-14T18:17:43.240089+0000","last_became_active":"2023-10-14T18:17:42.254349+0000","last_became_peered":"2023-10-14T18:17:42.254349+0000","last_unstale":"2023-10-14T18:17:43.240089+0000","last_undegraded":"2023-10-14T18:17:43.240089+0000","last_fullsized":"2023-10-14T18:17:43.240089+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253383+0000","last_change":"2023-10-14T18:17:27.265344+0000","last_active":"2023-10-14T18:17:42.253383+0000","last_peered":"2023-10-14T18:17:42.253383+0000","last_clean":"2023-10-14T18:17:42.253383+0000","last_became_active":"2023-10-14T18:17:27.265080+0000","last_became_peered":"2023-10-14T18:17:27.265080+0000","last_unstale":"2023-10-14T18:17:42.253383+0000","last_undegraded":"2023-10-14T18:17:42.253383+0000","last_fullsized":"2023-10-14T18:17:42.253383+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56836,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3530752,"data_stored":2044530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.216,"15min":1.2090000000000001},"min":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.573,"5min":1.7030000000000001,"15min":2.1669999999999998},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.232,"15min":1.228},"min":{"1min":0.78300000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6619999999999999,"5min":1.76,"15min":2.2149999999999999},"last":1.361}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.264,"15min":1.276},"min":{"1min":0.67500000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9059999999999999,"5min":2.214,"15min":2.214},"last":1.768},{"interface":"front","average":{"1min":1.341,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.871,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.71}]},{"osd":2,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.2230000000000001,"15min":1.2669999999999999},"min":{"1min":0.73799999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.546,"5min":1.6799999999999999,"15min":2.004},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.325,"15min":1.3700000000000001},"min":{"1min":0.85199999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.8260000000000001,"15min":2.2010000000000001},"last":1.9390000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3819999999999999,"15min":1.4339999999999999},"min":{"1min":1.0009999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.72,"5min":1.8500000000000001,"15min":2.4430000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5109999999999999,"15min":1.536},"min":{"1min":1.0429999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.875,"5min":1.9079999999999999,"15min":2.2810000000000001},"last":1.994}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.331},"min":{"1min":0.97699999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.8029999999999999,"5min":1.8200000000000001,"15min":2.367},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.8360000000000001,"5min":1.865,"15min":2.661},"last":1.659}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4390000000000001,"15min":1.462},"min":{"1min":1.048,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8740000000000001,"15min":2.516},"last":1.873},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4890000000000001,"15min":1.5229999999999999},"min":{"1min":1.1870000000000001,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.448},"last":1.6120000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5960000000000001,"15min":1.633},"min":{"1min":1.3169999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.4849999999999999},"last":1.8180000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.5349999999999999,"15min":1.5800000000000001},"min":{"1min":1.115,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":1.948,"5min":1.988,"15min":2.8159999999999998},"last":1.1859999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29884,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341956,"statfs":{"total":96636764160,"available":96606162944,"internally_reserved":0,"allocated":2932736,"data_stored":1451873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.754,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6319999999999999,"5min":2.085,"15min":2.085},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.204,"15min":1.2110000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.129}]},{"osd":1,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2230000000000001,"15min":1.2509999999999999},"min":{"1min":0.746,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.5309999999999999,"5min":1.762,"15min":1.776},"last":1.212},{"interface":"front","average":{"1min":1.256,"5min":1.3580000000000001,"15min":1.359},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.6459999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.383,"15min":1.3779999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8500000000000001,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.403,"15min":1.4219999999999999},"min":{"1min":1.0600000000000001,"5min":0.99199999999999999,"15min":0.98099999999999998},"max":{"1min":1.6719999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.478}]},{"osd":3,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3740000000000001,"15min":1.407},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7130000000000001,"5min":2.0379999999999998,"15min":2.2519999999999998},"last":1.27},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.90300000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.375,"15min":1.381},"min":{"1min":0.85899999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.827,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.554}]},{"osd":5,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3799999999999999,"15min":1.413},"min":{"1min":0.96199999999999997,"5min":0.624,"15min":0.624},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.4940000000000002},"last":1.024},{"interface":"front","average":{"1min":1.516,"5min":1.5469999999999999,"15min":1.579},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.8979999999999999,"5min":2.1600000000000001,"15min":2.5609999999999999},"last":1.512}]},{"osd":7,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5840000000000001,"15min":1.615},"min":{"1min":0.86799999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6389999999999998,"5min":2.903,"15min":3.423},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":1.1870000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.347}]}]},{"osd":5,"up_from":13,"seq":55834576008,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42364,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329476,"statfs":{"total":96636764160,"available":96593383424,"internally_reserved":0,"allocated":2924544,"data_stored":1451817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.179,"15min":1.2010000000000001},"min":{"1min":0.89700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.716,"5min":1.7569999999999999,"15min":2.0459999999999998},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.212,"15min":1.2470000000000001},"min":{"1min":0.878,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.722,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":1.02}]},{"osd":1,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.238,"15min":1.2769999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":1.462},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.274,"15min":1.3080000000000001},"min":{"1min":0.91500000000000004,"5min":0.73599999999999999,"15min":0.63400000000000001},"max":{"1min":1.722,"5min":1.7450000000000001,"15min":2.0419999999999998},"last":1.319}]},{"osd":2,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.258},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.649,"5min":1.764,"15min":3.7090000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.393,"5min":1.3779999999999999,"15min":1.3959999999999999},"min":{"1min":1.0369999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.768,"5min":1.841,"15min":1.923},"last":1.5469999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4239999999999999,"15min":1.4670000000000001},"min":{"1min":1.071,"5min":0.94199999999999995,"15min":0.81899999999999995},"max":{"1min":1.794,"5min":1.867,"15min":2.4969999999999999},"last":1.413},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.421,"15min":1.4450000000000001},"min":{"1min":1.008,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":1.952,"5min":1.952,"15min":3.8330000000000002},"last":1.367}]},{"osd":4,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3700000000000001,"15min":1.363},"min":{"1min":0.98899999999999999,"5min":0.80100000000000005,"15min":0.78400000000000003},"max":{"1min":1.7889999999999999,"5min":1.954,"15min":2.98},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.3939999999999999,"15min":1.3859999999999999},"min":{"1min":0.90600000000000003,"5min":0.61799999999999999,"15min":0.57499999999999996},"max":{"1min":2.016,"5min":2.016,"15min":3.1280000000000001},"last":0.90500000000000003}]},{"osd":6,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3859999999999999,"15min":1.417},"min":{"1min":0.97399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.853,"5min":1.8799999999999999,"15min":4.0540000000000003},"last":1.49},{"interface":"front","average":{"1min":1.663,"5min":1.601,"15min":1.627},"min":{"1min":1.3859999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":3.6360000000000001},"last":1.609}]},{"osd":7,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.516,"15min":1.573},"min":{"1min":1.0409999999999999,"5min":0.68600000000000005,"15min":0.67700000000000005},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.2759999999999998},"last":1.526},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5600000000000001,"15min":1.556},"min":{"1min":1.1100000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.0459999999999998,"5min":2.206,"15min":3.9390000000000001},"last":1.1879999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64584,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307256,"statfs":{"total":96636764160,"available":96570630144,"internally_reserved":0,"allocated":2945024,"data_stored":1454778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":1.0169999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.341,"5min":1.2729999999999999,"15min":1.288},"min":{"1min":0.80000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.698,"5min":1.887,"15min":2.3140000000000001},"last":1.135}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.282,"15min":1.323},"min":{"1min":1.0009999999999999,"5min":0.76800000000000002,"15min":0.59399999999999997},"max":{"1min":1.7410000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2490000000000001,"15min":1.268},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.667,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":1.234}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.29,"15min":1.2849999999999999},"min":{"1min":0.83699999999999997,"5min":0.81399999999999995,"15min":0.78000000000000003},"max":{"1min":1.659,"5min":1.869,"15min":3.524},"last":1.198},{"interface":"front","average":{"1min":1.379,"5min":1.427,"15min":1.4450000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.597}]},{"osd":3,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.278,"15min":1.3460000000000001},"min":{"1min":0.90400000000000003,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.627,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.474},{"interface":"front","average":{"1min":1.425,"5min":1.419,"15min":1.4390000000000001},"min":{"1min":1.0509999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7629999999999999,"5min":2.0569999999999999,"15min":4.1600000000000001},"last":1.494}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4219999999999999,"15min":1.3959999999999999},"min":{"1min":0.96599999999999997,"5min":0.90200000000000002,"15min":0.69299999999999995},"max":{"1min":1.7949999999999999,"5min":1.9319999999999999,"15min":2.02},"last":1.29},{"interface":"front","average":{"1min":1.407,"5min":1.369,"15min":1.3759999999999999},"min":{"1min":0.98299999999999998,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":1.7589999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.45,"15min":1.4339999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.81799999999999995},"max":{"1min":1.887,"5min":2.077,"15min":3.3610000000000002},"last":1.042},{"interface":"front","average":{"1min":1.548,"5min":1.575,"15min":1.5700000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.82699999999999996},"max":{"1min":1.823,"5min":2.1080000000000001,"15min":2.3929999999999998},"last":1.4359999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.506,"15min":1.516},"min":{"1min":1.234,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.798,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.381},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.601,"15min":1.595},"min":{"1min":1.246,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":1.9790000000000001,"5min":2.1739999999999999,"15min":3.8610000000000002},"last":1.337}]}]},{"osd":3,"up_from":13,"seq":55834576008,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52108,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319732,"statfs":{"total":96636764160,"available":96583405568,"internally_reserved":0,"allocated":2949120,"data_stored":1453099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96899999999999997,"15min":0.98399999999999999},"min":{"1min":0.502,"5min":0.377,"15min":0.377},"max":{"1min":1.46,"5min":1.4870000000000001,"15min":1.855},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.179,"15min":1.2050000000000001},"min":{"1min":0.61499999999999999,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4890000000000001,"5min":1.7230000000000001,"15min":1.9970000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0089999999999999,"15min":1.0169999999999999},"min":{"1min":0.57499999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.417,"5min":1.8939999999999999,"15min":1.919},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.177,"15min":1.212},"min":{"1min":0.78100000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.3400000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.446},"min":{"1min":0.71899999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.9730000000000001,"5min":2.032,"15min":2.032},"last":1.462},{"interface":"front","average":{"1min":1.284,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.72499999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.7430000000000001,"5min":1.8,"15min":2.3540000000000001},"last":1.401}]},{"osd":4,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7350000000000001,"5min":2.153,"15min":2.153},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.484},"min":{"1min":0.85299999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":1.9279999999999999,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.7729999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.96499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.8009999999999999,"5min":2.1429999999999998,"15min":2.476},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.518,"15min":1.5649999999999999},"min":{"1min":1.0249999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0739999999999998,"5min":2.246,"15min":2.891},"last":1.595}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6180000000000001,"15min":1.6319999999999999},"min":{"1min":1.0580000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0390000000000001,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6859999999999999,"15min":1.6870000000000001},"min":{"1min":1.0760000000000001,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.1320000000000001,"5min":2.2949999999999999,"15min":2.585},"last":1.637}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.579,"15min":1.607},"min":{"1min":1.2050000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.859,"5min":2.2370000000000001,"15min":2.419},"last":1.546},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.704,"15min":1.696},"min":{"1min":1.0469999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1890000000000001,"5min":2.548,"15min":2.548},"last":1.7210000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576008,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52104,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319736,"statfs":{"total":96636764160,"available":96583409664,"internally_reserved":0,"allocated":2940928,"data_stored":1452903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":0.97799999999999998,"15min":0.94899999999999995},"min":{"1min":0.65000000000000002,"5min":0.499,"15min":0.499},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.0349999999999999,"15min":1.0960000000000001},"min":{"1min":0.61899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.341,"5min":1.653,"15min":1.8660000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98699999999999999,"15min":1.0169999999999999},"min":{"1min":0.61899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.544,"5min":1.603,"15min":1.881},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.623,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2509999999999999,"15min":1.2909999999999999},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.56599999999999995},"max":{"1min":1.621,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.407,"15min":1.3999999999999999},"min":{"1min":0.76400000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0089999999999999,"5min":2.1579999999999999,"15min":2.165},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.492,"15min":1.478},"min":{"1min":1.077,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.3400000000000001,"15min":1.3680000000000001},"min":{"1min":0.92300000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.728,"5min":1.8759999999999999,"15min":2.024},"last":0.73299999999999998}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.444},"min":{"1min":1.012,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9299999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.613,"5min":1.625,"15min":1.5980000000000001},"min":{"1min":1.3460000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.528,"15min":1.53},"min":{"1min":0.99099999999999999,"5min":0.747,"15min":0.747},"max":{"1min":1.8799999999999999,"5min":2.0529999999999999,"15min":2.1150000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.587,"15min":1.617},"min":{"1min":0.85599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":0.91100000000000003}]},{"osd":7,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.611,"15min":1.5940000000000001},"min":{"1min":1.208,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.631,"5min":1.6659999999999999,"15min":1.6859999999999999},"min":{"1min":1.155,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.97999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52100,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319740,"statfs":{"total":96636764160,"available":96583413760,"internally_reserved":0,"allocated":2932736,"data_stored":1451381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95899999999999996,"15min":0.97199999999999998},"min":{"1min":0.51100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.5169999999999999,"5min":1.647,"15min":2.4929999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.0820000000000001,"15min":1.1279999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":1.079}]},{"osd":2,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.038,"15min":1.0629999999999999},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.895,"5min":1.996,"15min":2.6859999999999999},"last":1.204}]},{"osd":3,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.357,"15min":1.385},"min":{"1min":0.52700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2829999999999999,"15min":1.3169999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.8630000000000004,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.419}]},{"osd":4,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.88200000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.298,"5min":1.385,"15min":1.431},"min":{"1min":0.80400000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.899},"last":1.1439999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.4379999999999999,"15min":1.4710000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.173,"5min":2.173,"15min":3.004},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.55,"15min":1.5660000000000001},"min":{"1min":0.91800000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9359999999999999,"5min":2.093,"15min":2.5129999999999999},"last":1.361}]},{"osd":6,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5780000000000001,"15min":1.6140000000000001},"min":{"1min":0.95499999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.593},{"interface":"front","average":{"1min":1.863,"5min":1.649,"15min":1.627},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.478}]},{"osd":7,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.6679999999999999,"15min":1.6970000000000001},"min":{"1min":0.82299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":1.546},{"interface":"front","average":{"1min":1.839,"5min":1.6970000000000001,"15min":1.722},"min":{"1min":0.93300000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.7490000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48516,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323324,"statfs":{"total":96636764160,"available":96587083776,"internally_reserved":0,"allocated":3526656,"data_stored":2042365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.129,"15min":1.1499999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.1930000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.23},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.25},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.845,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":0.69599999999999995}]},{"osd":3,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.7150000000000001,"5min":1.804,"15min":2.0720000000000001},"last":1.056},{"interface":"front","average":{"1min":1.452,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":1.883,"15min":2.194},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.321,"15min":1.331},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7629999999999999,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.5129999999999999,"15min":1.512},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.121,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.377}]},{"osd":5,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.74,"5min":1.8220000000000001,"15min":2.1339999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5409999999999999,"15min":1.581},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":0.83299999999999996}]},{"osd":6,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.655,"15min":1.661},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.1080000000000001,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.3149999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.631,"15min":1.661},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0089999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.665},{"interface":"front","average":{"1min":1.639,"5min":1.613,"15min":1.641},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.359}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:17:46.917 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:17:46.918 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:17:46.934 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:17:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.131+0000 7fd53b59e640 1 Processor -- start 2023-10-14T18:17:47.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.131+0000 7fd53b59e640 1 -- start start 2023-10-14T18:17:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.132+0000 7fd53b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd53c15cd70 0x7fd53c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.132+0000 7fd53b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 0x7fd53c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.132+0000 7fd53b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd53c15d150 con 0x7fd53c0d0be0 2023-10-14T18:17:47.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.132+0000 7fd53b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0d1180 con 0x7fd53c15cd70 2023-10-14T18:17:47.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.132+0000 7fd53b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd53c0d1460 con 0x7fd53c0b8480 2023-10-14T18:17:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.133+0000 7fd53a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd53c15cd70 0x7fd53c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.133+0000 7fd53a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd53c15cd70 0x7fd53c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53518/0 (socket says 172.21.15.29:53518) 2023-10-14T18:17:47.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.133+0000 7fd53a59c640 1 -- 172.21.15.29:0/3075233838 learned_addr learned my addr 172.21.15.29:0/3075233838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.133+0000 7fd53a59c640 1 -- 172.21.15.29:0/3075233838 >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 msgr2=0x7fd53c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.133+0000 7fd539d9b640 1 --2- 172.21.15.29:0/3075233838 >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 0x7fd53c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.134+0000 7fd53a59c640 1 --2- 172.21.15.29:0/3075233838 >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 0x7fd53c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.134+0000 7fd53a59c640 1 -- 172.21.15.29:0/3075233838 >> v1:172.21.15.186:6789/0 conn(0x7fd53c0d0be0 legacy=0x7fd53c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.134+0000 7fd53a59c640 1 -- 172.21.15.29:0/3075233838 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd53c0d1740 con 0x7fd53c15cd70 2023-10-14T18:17:47.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.134+0000 7fd539d9b640 1 --2- 172.21.15.29:0/3075233838 >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 0x7fd53c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:47.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.135+0000 7fd53a59c640 1 --2- 172.21.15.29:0/3075233838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c0d2220 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd53c066a00 tx=0x7fd52c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d91b19309fbcca5a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.135+0000 7fd53959a640 1 -- 172.21.15.29:0/3075233838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c02ca80 con 0x7fd53c15cd70 2023-10-14T18:17:47.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.135+0000 7fd53959a640 1 -- 172.21.15.29:0/3075233838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c02cbe0 con 0x7fd53c15cd70 2023-10-14T18:17:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.135+0000 7fd53959a640 1 -- 172.21.15.29:0/3075233838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd52c0343f0 con 0x7fd53c15cd70 2023-10-14T18:17:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 -- 172.21.15.29:0/3075233838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 msgr2=0x7fd53c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3075233838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c0d2220 secure :-1 s=READY pgs=2639 cs=0 l=1 rev1=1 crypto rx=0x7fd53c066a00 tx=0x7fd52c02bee0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 -- 172.21.15.29:0/3075233838 shutdown_connections 2023-10-14T18:17:47.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3075233838 >> v2:172.21.15.186:3300/0 conn(0x7fd53c0b8480 0x7fd53c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3075233838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c0d2220 unknown :-1 s=CLOSED pgs=2639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 -- 172.21.15.29:0/3075233838 >> 172.21.15.29:0/3075233838 conn(0x7fd53c0cf570 msgr2=0x7fd53c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 -- 172.21.15.29:0/3075233838 shutdown_connections 2023-10-14T18:17:47.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.136+0000 7fd53b59e640 1 -- 172.21.15.29:0/3075233838 wait complete. 2023-10-14T18:17:47.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.138+0000 7fd53b59e640 1 Processor -- start 2023-10-14T18:17:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.138+0000 7fd53b59e640 1 -- start start 2023-10-14T18:17:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd53c0b8480 0x7fd53c0b2f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd53c0d0be0 0x7fd53c0b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b5350 con 0x7fd53c15cd70 2023-10-14T18:17:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c152430 con 0x7fd53c0d0be0 2023-10-14T18:17:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd53c0b35f0 con 0x7fd53c0b8480 2023-10-14T18:17:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53ad9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53520/0 (socket says 172.21.15.29:53520) 2023-10-14T18:17:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.139+0000 7fd53ad9d640 1 -- 172.21.15.29:0/3694192964 learned_addr learned my addr 172.21.15.29:0/3694192964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd539d9b640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd53c0d0be0 0x7fd53c0b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53a59c640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd53c0b8480 0x7fd53c0b2f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 -- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd53c0b8480 msgr2=0x7fd53c0b2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd53c0b8480 0x7fd53c0b2f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 -- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd53c0d0be0 msgr2=0x7fd53c0b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd53c0d0be0 0x7fd53c0b4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 -- 172.21.15.29:0/3694192964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c009b30 con 0x7fd53c15cd70 2023-10-14T18:17:47.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd53ad9d640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fd53400c3c0 tx=0x7fd53400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5340052d0 con 0x7fd53c15cd70 2023-10-14T18:17:47.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.140+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd534005430 con 0x7fd53c15cd70 2023-10-14T18:17:47.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.141+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd53c0b38d0 con 0x7fd53c15cd70 2023-10-14T18:17:47.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.141+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd534003830 con 0x7fd53c15cd70 2023-10-14T18:17:47.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.141+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53c154460 con 0x7fd53c15cd70 2023-10-14T18:17:47.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.142+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53c04e9b0 con 0x7fd53c15cd70 2023-10-14T18:17:47.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.143+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fd534003990 con 0x7fd53c15cd70 2023-10-14T18:17:47.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.144+0000 7fd532ffd640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 0x7fd51c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.144+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fd534088ff0 con 0x7fd53c15cd70 2023-10-14T18:17:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.144+0000 7fd53a59c640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 0x7fd51c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.145+0000 7fd53a59c640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 0x7fd51c06d480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd52c035000 tx=0x7fd52c030000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.148+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd5340bbfe0 con 0x7fd53c15cd70 2023-10-14T18:17:47.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.343+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd53c0d1d00 con 0x7fd51c06afc0 2023-10-14T18:17:47.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.368+0000 7fd532ffd640 1 -- 172.21.15.29:0/3694192964 <== mgr.11010 v2:172.21.15.29:6832/2967217610 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+525842 (secure 0 0 0) 0x7fd52c034470 con 0x7fd51c06afc0 2023-10-14T18:17:47.377 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:47.381 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:17:47.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.378+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 msgr2=0x7fd51c06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.378+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 0x7fd51c06d480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fd52c035000 tx=0x7fd52c030000 comp rx=0 tx=0).stop 2023-10-14T18:17:47.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.378+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 msgr2=0x7fd53c151ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.378+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 secure :-1 s=READY pgs=2640 cs=0 l=1 rev1=1 crypto rx=0x7fd53400c3c0 tx=0x7fd53400c880 comp rx=0 tx=0).stop 2023-10-14T18:17:47.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 shutdown_connections 2023-10-14T18:17:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fd51c06afc0 0x7fd51c06d480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd53c15cd70 0x7fd53c151ef0 unknown :-1 s=CLOSED pgs=2640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd53c0d0be0 0x7fd53c0b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 --2- 172.21.15.29:0/3694192964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd53c0b8480 0x7fd53c0b2f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 >> 172.21.15.29:0/3694192964 conn(0x7fd53c0cf570 msgr2=0x7fd53c15bf80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:47.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 shutdown_connections 2023-10-14T18:17:47.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.379+0000 7fd53b59e640 1 -- 172.21.15.29:0/3694192964 wait complete. 2023-10-14T18:17:47.402 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2023-10-14T18:17:46.752362+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":670,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":427,"ondisk_log_size":427,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":324,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":398496,"kb_used_data":24104,"kb_used_omap":12,"kb_used_meta":374259,"kb_avail":754576224,"statfs":{"total":773094113280,"available":772686053376,"internally_reserved":0,"allocated":24682496,"data_stored":12802746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":383241784},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":24,"apply_latency_ms":24,"commit_latency_ns":24000000,"apply_latency_ns":24000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002245"},"pg_stats":[{"pgid":"29.1c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457578+0000","last_change":"2023-10-14T18:17:42.260490+0000","last_active":"2023-10-14T18:17:43.457578+0000","last_peered":"2023-10-14T18:17:43.457578+0000","last_clean":"2023-10-14T18:17:43.457578+0000","last_became_active":"2023-10-14T18:17:42.260211+0000","last_became_peered":"2023-10-14T18:17:42.260211+0000","last_unstale":"2023-10-14T18:17:43.457578+0000","last_undegraded":"2023-10-14T18:17:43.457578+0000","last_fullsized":"2023-10-14T18:17:43.457578+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:55:47.448993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"216'348","reported_seq":1214,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243545+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:17:42.243545+0000","last_peered":"2023-10-14T18:17:42.243545+0000","last_clean":"2023-10-14T18:17:42.243545+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:17:42.243545+0000","last_undegraded":"2023-10-14T18:17:42.243545+0000","last_fullsized":"2023-10-14T18:17:42.243545+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1b","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250221+0000","last_change":"2023-10-14T18:17:26.757505+0000","last_active":"2023-10-14T18:17:42.250221+0000","last_peered":"2023-10-14T18:17:42.250221+0000","last_clean":"2023-10-14T18:17:42.250221+0000","last_became_active":"2023-10-14T18:17:26.757219+0000","last_became_peered":"2023-10-14T18:17:26.757219+0000","last_unstale":"2023-10-14T18:17:42.250221+0000","last_undegraded":"2023-10-14T18:17:42.250221+0000","last_fullsized":"2023-10-14T18:17:42.250221+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:48:19.831762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247422+0000","last_change":"2023-10-14T18:17:27.260210+0000","last_active":"2023-10-14T18:17:42.247422+0000","last_peered":"2023-10-14T18:17:42.247422+0000","last_clean":"2023-10-14T18:17:42.247422+0000","last_became_active":"2023-10-14T18:17:27.259297+0000","last_became_peered":"2023-10-14T18:17:27.259297+0000","last_unstale":"2023-10-14T18:17:42.247422+0000","last_undegraded":"2023-10-14T18:17:42.247422+0000","last_fullsized":"2023-10-14T18:17:42.247422+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1d","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237656+0000","last_change":"2023-10-14T18:17:41.347319+0000","last_active":"2023-10-14T18:17:43.237656+0000","last_peered":"2023-10-14T18:17:43.237656+0000","last_clean":"2023-10-14T18:17:43.237656+0000","last_became_active":"2023-10-14T18:17:41.345913+0000","last_became_peered":"2023-10-14T18:17:41.345913+0000","last_unstale":"2023-10-14T18:17:43.237656+0000","last_undegraded":"2023-10-14T18:17:43.237656+0000","last_fullsized":"2023-10-14T18:17:43.237656+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:46.523999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.7","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241004+0000","last_change":"2023-10-14T18:17:42.270250+0000","last_active":"2023-10-14T18:17:43.241004+0000","last_peered":"2023-10-14T18:17:43.241004+0000","last_clean":"2023-10-14T18:17:43.241004+0000","last_became_active":"2023-10-14T18:17:42.267481+0000","last_became_peered":"2023-10-14T18:17:42.267481+0000","last_unstale":"2023-10-14T18:17:43.241004+0000","last_undegraded":"2023-10-14T18:17:43.241004+0000","last_fullsized":"2023-10-14T18:17:43.241004+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:46.735181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.0","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.237019+0000","last_change":"2023-10-14T18:17:26.754848+0000","last_active":"2023-10-14T18:17:42.237019+0000","last_peered":"2023-10-14T18:17:42.237019+0000","last_clean":"2023-10-14T18:17:42.237019+0000","last_became_active":"2023-10-14T18:17:26.751997+0000","last_became_peered":"2023-10-14T18:17:26.751997+0000","last_unstale":"2023-10-14T18:17:42.237019+0000","last_undegraded":"2023-10-14T18:17:42.237019+0000","last_fullsized":"2023-10-14T18:17:42.237019+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:31.362757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246072+0000","last_change":"2023-10-14T18:17:27.258205+0000","last_active":"2023-10-14T18:17:42.246072+0000","last_peered":"2023-10-14T18:17:42.246072+0000","last_clean":"2023-10-14T18:17:42.246072+0000","last_became_active":"2023-10-14T18:17:27.257652+0000","last_became_peered":"2023-10-14T18:17:27.257652+0000","last_unstale":"2023-10-14T18:17:42.246072+0000","last_undegraded":"2023-10-14T18:17:42.246072+0000","last_fullsized":"2023-10-14T18:17:42.246072+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:00.190444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.6","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457848+0000","last_change":"2023-10-14T18:17:41.343953+0000","last_active":"2023-10-14T18:17:43.457848+0000","last_peered":"2023-10-14T18:17:43.457848+0000","last_clean":"2023-10-14T18:17:43.457848+0000","last_became_active":"2023-10-14T18:17:41.343682+0000","last_became_peered":"2023-10-14T18:17:41.343682+0000","last_unstale":"2023-10-14T18:17:43.457848+0000","last_undegraded":"2023-10-14T18:17:43.457848+0000","last_fullsized":"2023-10-14T18:17:43.457848+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:20:02.002730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.6","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454747+0000","last_change":"2023-10-14T18:17:42.267414+0000","last_active":"2023-10-14T18:17:43.454747+0000","last_peered":"2023-10-14T18:17:43.454747+0000","last_clean":"2023-10-14T18:17:43.454747+0000","last_became_active":"2023-10-14T18:17:42.267042+0000","last_became_peered":"2023-10-14T18:17:42.267042+0000","last_unstale":"2023-10-14T18:17:43.454747+0000","last_undegraded":"2023-10-14T18:17:43.454747+0000","last_fullsized":"2023-10-14T18:17:43.454747+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:13.451953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253477+0000","last_change":"2023-10-14T18:17:26.755699+0000","last_active":"2023-10-14T18:17:42.253477+0000","last_peered":"2023-10-14T18:17:42.253477+0000","last_clean":"2023-10-14T18:17:42.253477+0000","last_became_active":"2023-10-14T18:17:26.755286+0000","last_became_peered":"2023-10-14T18:17:26.755286+0000","last_unstale":"2023-10-14T18:17:42.253477+0000","last_undegraded":"2023-10-14T18:17:42.253477+0000","last_fullsized":"2023-10-14T18:17:42.253477+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:07.796326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.0","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230947+0000","last_change":"2023-10-14T18:17:27.248014+0000","last_active":"2023-10-14T18:17:42.230947+0000","last_peered":"2023-10-14T18:17:42.230947+0000","last_clean":"2023-10-14T18:17:42.230947+0000","last_became_active":"2023-10-14T18:17:27.247706+0000","last_became_peered":"2023-10-14T18:17:27.247706+0000","last_unstale":"2023-10-14T18:17:42.230947+0000","last_undegraded":"2023-10-14T18:17:42.230947+0000","last_fullsized":"2023-10-14T18:17:42.230947+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.7","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454805+0000","last_change":"2023-10-14T18:17:41.349108+0000","last_active":"2023-10-14T18:17:43.454805+0000","last_peered":"2023-10-14T18:17:43.454805+0000","last_clean":"2023-10-14T18:17:43.454805+0000","last_became_active":"2023-10-14T18:17:41.348764+0000","last_became_peered":"2023-10-14T18:17:41.348764+0000","last_unstale":"2023-10-14T18:17:43.454805+0000","last_undegraded":"2023-10-14T18:17:43.454805+0000","last_fullsized":"2023-10-14T18:17:43.454805+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:15.080875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.5","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454761+0000","last_change":"2023-10-14T18:17:42.254387+0000","last_active":"2023-10-14T18:17:43.454761+0000","last_peered":"2023-10-14T18:17:43.454761+0000","last_clean":"2023-10-14T18:17:43.454761+0000","last_became_active":"2023-10-14T18:17:42.253776+0000","last_became_peered":"2023-10-14T18:17:42.253776+0000","last_unstale":"2023-10-14T18:17:43.454761+0000","last_undegraded":"2023-10-14T18:17:43.454761+0000","last_fullsized":"2023-10-14T18:17:43.454761+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:04.534193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.2","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247526+0000","last_change":"2023-10-14T18:17:26.762092+0000","last_active":"2023-10-14T18:17:42.247526+0000","last_peered":"2023-10-14T18:17:42.247526+0000","last_clean":"2023-10-14T18:17:42.247526+0000","last_became_active":"2023-10-14T18:17:26.761696+0000","last_became_peered":"2023-10-14T18:17:26.761696+0000","last_unstale":"2023-10-14T18:17:42.247526+0000","last_undegraded":"2023-10-14T18:17:42.247526+0000","last_fullsized":"2023-10-14T18:17:42.247526+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:27.118966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250016+0000","last_change":"2023-10-14T18:17:27.259073+0000","last_active":"2023-10-14T18:17:42.250016+0000","last_peered":"2023-10-14T18:17:42.250016+0000","last_clean":"2023-10-14T18:17:42.250016+0000","last_became_active":"2023-10-14T18:17:27.258114+0000","last_became_peered":"2023-10-14T18:17:27.258114+0000","last_unstale":"2023-10-14T18:17:42.250016+0000","last_undegraded":"2023-10-14T18:17:42.250016+0000","last_fullsized":"2023-10-14T18:17:42.250016+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:14.438669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.4","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239705+0000","last_change":"2023-10-14T18:17:41.340410+0000","last_active":"2023-10-14T18:17:43.239705+0000","last_peered":"2023-10-14T18:17:43.239705+0000","last_clean":"2023-10-14T18:17:43.239705+0000","last_became_active":"2023-10-14T18:17:41.338979+0000","last_became_peered":"2023-10-14T18:17:41.338979+0000","last_unstale":"2023-10-14T18:17:43.239705+0000","last_undegraded":"2023-10-14T18:17:43.239705+0000","last_fullsized":"2023-10-14T18:17:43.239705+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:49:07.446346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.4","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239482+0000","last_change":"2023-10-14T18:17:42.262879+0000","last_active":"2023-10-14T18:17:43.239482+0000","last_peered":"2023-10-14T18:17:43.239482+0000","last_clean":"2023-10-14T18:17:43.239482+0000","last_became_active":"2023-10-14T18:17:42.262477+0000","last_became_peered":"2023-10-14T18:17:42.262477+0000","last_unstale":"2023-10-14T18:17:43.239482+0000","last_undegraded":"2023-10-14T18:17:43.239482+0000","last_fullsized":"2023-10-14T18:17:43.239482+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.3","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253414+0000","last_change":"2023-10-14T18:17:26.765437+0000","last_active":"2023-10-14T18:17:42.253414+0000","last_peered":"2023-10-14T18:17:42.253414+0000","last_clean":"2023-10-14T18:17:42.253414+0000","last_became_active":"2023-10-14T18:17:26.764932+0000","last_became_peered":"2023-10-14T18:17:26.764932+0000","last_unstale":"2023-10-14T18:17:42.253414+0000","last_undegraded":"2023-10-14T18:17:42.253414+0000","last_fullsized":"2023-10-14T18:17:42.253414+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:00.444875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247544+0000","last_change":"2023-10-14T18:17:27.261099+0000","last_active":"2023-10-14T18:17:42.247544+0000","last_peered":"2023-10-14T18:17:42.247544+0000","last_clean":"2023-10-14T18:17:42.247544+0000","last_became_active":"2023-10-14T18:17:27.260783+0000","last_became_peered":"2023-10-14T18:17:27.260783+0000","last_unstale":"2023-10-14T18:17:42.247544+0000","last_undegraded":"2023-10-14T18:17:42.247544+0000","last_fullsized":"2023-10-14T18:17:42.247544+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.5","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458113+0000","last_change":"2023-10-14T18:17:41.342390+0000","last_active":"2023-10-14T18:17:43.458113+0000","last_peered":"2023-10-14T18:17:43.458113+0000","last_clean":"2023-10-14T18:17:43.458113+0000","last_became_active":"2023-10-14T18:17:41.342111+0000","last_became_peered":"2023-10-14T18:17:41.342111+0000","last_unstale":"2023-10-14T18:17:43.458113+0000","last_undegraded":"2023-10-14T18:17:43.458113+0000","last_fullsized":"2023-10-14T18:17:43.458113+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:08.262799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240972+0000","last_change":"2023-10-14T18:17:42.256640+0000","last_active":"2023-10-14T18:17:43.240972+0000","last_peered":"2023-10-14T18:17:43.240972+0000","last_clean":"2023-10-14T18:17:43.240972+0000","last_became_active":"2023-10-14T18:17:42.253676+0000","last_became_peered":"2023-10-14T18:17:42.253676+0000","last_unstale":"2023-10-14T18:17:43.240972+0000","last_undegraded":"2023-10-14T18:17:43.240972+0000","last_fullsized":"2023-10-14T18:17:43.240972+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:31.063452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.4","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230690+0000","last_change":"2023-10-14T18:17:26.767200+0000","last_active":"2023-10-14T18:17:42.230690+0000","last_peered":"2023-10-14T18:17:42.230690+0000","last_clean":"2023-10-14T18:17:42.230690+0000","last_became_active":"2023-10-14T18:17:26.766925+0000","last_became_peered":"2023-10-14T18:17:26.766925+0000","last_unstale":"2023-10-14T18:17:42.230690+0000","last_undegraded":"2023-10-14T18:17:42.230690+0000","last_fullsized":"2023-10-14T18:17:42.230690+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:28.445946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.5","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.254013+0000","last_change":"2023-10-14T18:17:27.259767+0000","last_active":"2023-10-14T18:17:42.254013+0000","last_peered":"2023-10-14T18:17:42.254013+0000","last_clean":"2023-10-14T18:17:42.254013+0000","last_became_active":"2023-10-14T18:17:27.259468+0000","last_became_peered":"2023-10-14T18:17:27.259468+0000","last_unstale":"2023-10-14T18:17:42.254013+0000","last_undegraded":"2023-10-14T18:17:42.254013+0000","last_fullsized":"2023-10-14T18:17:42.254013+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:40.472642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.2","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453519+0000","last_change":"2023-10-14T18:17:41.331066+0000","last_active":"2023-10-14T18:17:43.453519+0000","last_peered":"2023-10-14T18:17:43.453519+0000","last_clean":"2023-10-14T18:17:43.453519+0000","last_became_active":"2023-10-14T18:17:41.330643+0000","last_became_peered":"2023-10-14T18:17:41.330643+0000","last_unstale":"2023-10-14T18:17:43.453519+0000","last_undegraded":"2023-10-14T18:17:43.453519+0000","last_fullsized":"2023-10-14T18:17:43.453519+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:36:29.940985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.2","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.235126+0000","last_change":"2023-10-14T18:17:42.256632+0000","last_active":"2023-10-14T18:17:43.235126+0000","last_peered":"2023-10-14T18:17:43.235126+0000","last_clean":"2023-10-14T18:17:43.235126+0000","last_became_active":"2023-10-14T18:17:42.256317+0000","last_became_peered":"2023-10-14T18:17:42.256317+0000","last_unstale":"2023-10-14T18:17:43.235126+0000","last_undegraded":"2023-10-14T18:17:43.235126+0000","last_fullsized":"2023-10-14T18:17:43.235126+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:18:48.609720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.5","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249776+0000","last_change":"2023-10-14T18:17:26.754064+0000","last_active":"2023-10-14T18:17:42.249776+0000","last_peered":"2023-10-14T18:17:42.249776+0000","last_clean":"2023-10-14T18:17:42.249776+0000","last_became_active":"2023-10-14T18:17:26.752729+0000","last_became_peered":"2023-10-14T18:17:26.752729+0000","last_unstale":"2023-10-14T18:17:42.249776+0000","last_undegraded":"2023-10-14T18:17:42.249776+0000","last_fullsized":"2023-10-14T18:17:42.249776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:09.744523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.4","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.242939+0000","last_change":"2023-10-14T18:17:27.254416+0000","last_active":"2023-10-14T18:17:42.242939+0000","last_peered":"2023-10-14T18:17:42.242939+0000","last_clean":"2023-10-14T18:17:42.242939+0000","last_became_active":"2023-10-14T18:17:27.254010+0000","last_became_peered":"2023-10-14T18:17:27.254010+0000","last_unstale":"2023-10-14T18:17:42.242939+0000","last_undegraded":"2023-10-14T18:17:42.242939+0000","last_fullsized":"2023-10-14T18:17:42.242939+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.3","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237270+0000","last_change":"2023-10-14T18:17:41.340549+0000","last_active":"2023-10-14T18:17:43.237270+0000","last_peered":"2023-10-14T18:17:43.237270+0000","last_clean":"2023-10-14T18:17:43.237270+0000","last_became_active":"2023-10-14T18:17:41.340197+0000","last_became_peered":"2023-10-14T18:17:41.340197+0000","last_unstale":"2023-10-14T18:17:43.237270+0000","last_undegraded":"2023-10-14T18:17:43.237270+0000","last_fullsized":"2023-10-14T18:17:43.237270+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.1","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456741+0000","last_change":"2023-10-14T18:17:42.264770+0000","last_active":"2023-10-14T18:17:43.456741+0000","last_peered":"2023-10-14T18:17:43.456741+0000","last_clean":"2023-10-14T18:17:43.456741+0000","last_became_active":"2023-10-14T18:17:42.264512+0000","last_became_peered":"2023-10-14T18:17:42.264512+0000","last_unstale":"2023-10-14T18:17:43.456741+0000","last_undegraded":"2023-10-14T18:17:43.456741+0000","last_fullsized":"2023-10-14T18:17:43.456741+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:16:52.050475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.6","version":"229'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255086+0000","last_change":"2023-10-14T18:17:26.765123+0000","last_active":"2023-10-14T18:17:42.255086+0000","last_peered":"2023-10-14T18:17:42.255086+0000","last_clean":"2023-10-14T18:17:42.255086+0000","last_became_active":"2023-10-14T18:17:26.764801+0000","last_became_peered":"2023-10-14T18:17:26.764801+0000","last_unstale":"2023-10-14T18:17:42.255086+0000","last_undegraded":"2023-10-14T18:17:42.255086+0000","last_fullsized":"2023-10-14T18:17:42.255086+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:03.297149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.7","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253505+0000","last_change":"2023-10-14T18:17:27.252553+0000","last_active":"2023-10-14T18:17:42.253505+0000","last_peered":"2023-10-14T18:17:42.253505+0000","last_clean":"2023-10-14T18:17:42.253505+0000","last_became_active":"2023-10-14T18:17:27.250542+0000","last_became_peered":"2023-10-14T18:17:27.250542+0000","last_unstale":"2023-10-14T18:17:42.253505+0000","last_undegraded":"2023-10-14T18:17:42.253505+0000","last_fullsized":"2023-10-14T18:17:42.253505+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.0","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238273+0000","last_change":"2023-10-14T18:17:41.343809+0000","last_active":"2023-10-14T18:17:43.238273+0000","last_peered":"2023-10-14T18:17:43.238273+0000","last_clean":"2023-10-14T18:17:43.238273+0000","last_became_active":"2023-10-14T18:17:41.343130+0000","last_became_peered":"2023-10-14T18:17:41.343130+0000","last_unstale":"2023-10-14T18:17:43.238273+0000","last_undegraded":"2023-10-14T18:17:43.238273+0000","last_fullsized":"2023-10-14T18:17:43.238273+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.0","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237924+0000","last_change":"2023-10-14T18:17:42.267983+0000","last_active":"2023-10-14T18:17:43.237924+0000","last_peered":"2023-10-14T18:17:43.237924+0000","last_clean":"2023-10-14T18:17:43.237924+0000","last_became_active":"2023-10-14T18:17:42.267691+0000","last_became_peered":"2023-10-14T18:17:42.267691+0000","last_unstale":"2023-10-14T18:17:43.237924+0000","last_undegraded":"2023-10-14T18:17:43.237924+0000","last_fullsized":"2023-10-14T18:17:43.237924+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:05.726762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.7","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.232837+0000","last_change":"2023-10-14T18:17:26.768016+0000","last_active":"2023-10-14T18:17:42.232837+0000","last_peered":"2023-10-14T18:17:42.232837+0000","last_clean":"2023-10-14T18:17:42.232837+0000","last_became_active":"2023-10-14T18:17:26.767729+0000","last_became_peered":"2023-10-14T18:17:26.767729+0000","last_unstale":"2023-10-14T18:17:42.232837+0000","last_undegraded":"2023-10-14T18:17:42.232837+0000","last_fullsized":"2023-10-14T18:17:42.232837+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:30.381195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.6","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240577+0000","last_change":"2023-10-14T18:17:27.254575+0000","last_active":"2023-10-14T18:17:42.240577+0000","last_peered":"2023-10-14T18:17:42.240577+0000","last_clean":"2023-10-14T18:17:42.240577+0000","last_became_active":"2023-10-14T18:17:27.254278+0000","last_became_peered":"2023-10-14T18:17:27.254278+0000","last_unstale":"2023-10-14T18:17:42.240577+0000","last_undegraded":"2023-10-14T18:17:42.240577+0000","last_fullsized":"2023-10-14T18:17:42.240577+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453329+0000","last_change":"2023-10-14T18:17:41.343518+0000","last_active":"2023-10-14T18:17:43.453329+0000","last_peered":"2023-10-14T18:17:43.453329+0000","last_clean":"2023-10-14T18:17:43.453329+0000","last_became_active":"2023-10-14T18:17:41.343148+0000","last_became_peered":"2023-10-14T18:17:41.343148+0000","last_unstale":"2023-10-14T18:17:43.453329+0000","last_undegraded":"2023-10-14T18:17:43.453329+0000","last_fullsized":"2023-10-14T18:17:43.453329+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:28.907656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269865+0000","last_change":"2023-10-14T18:17:42.262505+0000","last_active":"2023-10-14T18:17:43.269865+0000","last_peered":"2023-10-14T18:17:43.269865+0000","last_clean":"2023-10-14T18:17:43.269865+0000","last_became_active":"2023-10-14T18:17:42.262149+0000","last_became_peered":"2023-10-14T18:17:42.262149+0000","last_unstale":"2023-10-14T18:17:43.269865+0000","last_undegraded":"2023-10-14T18:17:43.269865+0000","last_fullsized":"2023-10-14T18:17:43.269865+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:18.253464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.8","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240776+0000","last_change":"2023-10-14T18:17:26.758398+0000","last_active":"2023-10-14T18:17:42.240776+0000","last_peered":"2023-10-14T18:17:42.240776+0000","last_clean":"2023-10-14T18:17:42.240776+0000","last_became_active":"2023-10-14T18:17:26.758114+0000","last_became_peered":"2023-10-14T18:17:26.758114+0000","last_unstale":"2023-10-14T18:17:42.240776+0000","last_undegraded":"2023-10-14T18:17:42.240776+0000","last_fullsized":"2023-10-14T18:17:42.240776+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:39.144399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.9","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253218+0000","last_change":"2023-10-14T18:17:27.254894+0000","last_active":"2023-10-14T18:17:42.253218+0000","last_peered":"2023-10-14T18:17:42.253218+0000","last_clean":"2023-10-14T18:17:42.253218+0000","last_became_active":"2023-10-14T18:17:27.253030+0000","last_became_peered":"2023-10-14T18:17:27.253030+0000","last_unstale":"2023-10-14T18:17:42.253218+0000","last_undegraded":"2023-10-14T18:17:42.253218+0000","last_fullsized":"2023-10-14T18:17:42.253218+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:38.076369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.e","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454581+0000","last_change":"2023-10-14T18:17:41.338009+0000","last_active":"2023-10-14T18:17:43.454581+0000","last_peered":"2023-10-14T18:17:43.454581+0000","last_clean":"2023-10-14T18:17:43.454581+0000","last_became_active":"2023-10-14T18:17:41.337716+0000","last_became_peered":"2023-10-14T18:17:41.337716+0000","last_unstale":"2023-10-14T18:17:43.454581+0000","last_undegraded":"2023-10-14T18:17:43.454581+0000","last_fullsized":"2023-10-14T18:17:43.454581+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:53.711866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.258926+0000","last_change":"2023-10-14T18:17:42.259942+0000","last_active":"2023-10-14T18:17:43.258926+0000","last_peered":"2023-10-14T18:17:43.258926+0000","last_clean":"2023-10-14T18:17:43.258926+0000","last_became_active":"2023-10-14T18:17:42.259630+0000","last_became_peered":"2023-10-14T18:17:42.259630+0000","last_unstale":"2023-10-14T18:17:43.258926+0000","last_undegraded":"2023-10-14T18:17:43.258926+0000","last_fullsized":"2023-10-14T18:17:43.258926+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.9","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248053+0000","last_change":"2023-10-14T18:17:26.756432+0000","last_active":"2023-10-14T18:17:42.248053+0000","last_peered":"2023-10-14T18:17:42.248053+0000","last_clean":"2023-10-14T18:17:42.248053+0000","last_became_active":"2023-10-14T18:17:26.756148+0000","last_became_peered":"2023-10-14T18:17:26.756148+0000","last_unstale":"2023-10-14T18:17:42.248053+0000","last_undegraded":"2023-10-14T18:17:42.248053+0000","last_fullsized":"2023-10-14T18:17:42.248053+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:19:13.420763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.8","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245606+0000","last_change":"2023-10-14T18:17:27.258139+0000","last_active":"2023-10-14T18:17:42.245606+0000","last_peered":"2023-10-14T18:17:42.245606+0000","last_clean":"2023-10-14T18:17:42.245606+0000","last_became_active":"2023-10-14T18:17:27.257907+0000","last_became_peered":"2023-10-14T18:17:27.257907+0000","last_unstale":"2023-10-14T18:17:42.245606+0000","last_undegraded":"2023-10-14T18:17:42.245606+0000","last_fullsized":"2023-10-14T18:17:42.245606+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:07:57.184999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.f","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241295+0000","last_change":"2023-10-14T18:17:41.347434+0000","last_active":"2023-10-14T18:17:43.241295+0000","last_peered":"2023-10-14T18:17:43.241295+0000","last_clean":"2023-10-14T18:17:43.241295+0000","last_became_active":"2023-10-14T18:17:41.346764+0000","last_became_peered":"2023-10-14T18:17:41.346764+0000","last_unstale":"2023-10-14T18:17:43.241295+0000","last_undegraded":"2023-10-14T18:17:43.241295+0000","last_fullsized":"2023-10-14T18:17:43.241295+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:51.625931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240112+0000","last_change":"2023-10-14T18:17:42.256562+0000","last_active":"2023-10-14T18:17:43.240112+0000","last_peered":"2023-10-14T18:17:43.240112+0000","last_clean":"2023-10-14T18:17:43.240112+0000","last_became_active":"2023-10-14T18:17:42.256238+0000","last_became_peered":"2023-10-14T18:17:42.256238+0000","last_unstale":"2023-10-14T18:17:43.240112+0000","last_undegraded":"2023-10-14T18:17:43.240112+0000","last_fullsized":"2023-10-14T18:17:43.240112+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.a","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247684+0000","last_change":"2023-10-14T18:17:26.767394+0000","last_active":"2023-10-14T18:17:42.247684+0000","last_peered":"2023-10-14T18:17:42.247684+0000","last_clean":"2023-10-14T18:17:42.247684+0000","last_became_active":"2023-10-14T18:17:26.767091+0000","last_became_peered":"2023-10-14T18:17:26.767091+0000","last_unstale":"2023-10-14T18:17:42.247684+0000","last_undegraded":"2023-10-14T18:17:42.247684+0000","last_fullsized":"2023-10-14T18:17:42.247684+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230388+0000","last_change":"2023-10-14T18:17:27.251912+0000","last_active":"2023-10-14T18:17:42.230388+0000","last_peered":"2023-10-14T18:17:42.230388+0000","last_clean":"2023-10-14T18:17:42.230388+0000","last_became_active":"2023-10-14T18:17:27.251629+0000","last_became_peered":"2023-10-14T18:17:27.251629+0000","last_unstale":"2023-10-14T18:17:42.230388+0000","last_undegraded":"2023-10-14T18:17:42.230388+0000","last_fullsized":"2023-10-14T18:17:42.230388+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:17.594667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.c","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269752+0000","last_change":"2023-10-14T18:17:41.341578+0000","last_active":"2023-10-14T18:17:43.269752+0000","last_peered":"2023-10-14T18:17:43.269752+0000","last_clean":"2023-10-14T18:17:43.269752+0000","last_became_active":"2023-10-14T18:17:41.341312+0000","last_became_peered":"2023-10-14T18:17:41.341312+0000","last_unstale":"2023-10-14T18:17:43.269752+0000","last_undegraded":"2023-10-14T18:17:43.269752+0000","last_fullsized":"2023-10-14T18:17:43.269752+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:32:38.030124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239043+0000","last_change":"2023-10-14T18:17:42.254578+0000","last_active":"2023-10-14T18:17:43.239043+0000","last_peered":"2023-10-14T18:17:43.239043+0000","last_clean":"2023-10-14T18:17:43.239043+0000","last_became_active":"2023-10-14T18:17:42.254244+0000","last_became_peered":"2023-10-14T18:17:42.254244+0000","last_unstale":"2023-10-14T18:17:43.239043+0000","last_undegraded":"2023-10-14T18:17:43.239043+0000","last_fullsized":"2023-10-14T18:17:43.239043+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:50.707802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.b","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247373+0000","last_change":"2023-10-14T18:17:26.756320+0000","last_active":"2023-10-14T18:17:42.247373+0000","last_peered":"2023-10-14T18:17:42.247373+0000","last_clean":"2023-10-14T18:17:42.247373+0000","last_became_active":"2023-10-14T18:17:26.756040+0000","last_became_peered":"2023-10-14T18:17:26.756040+0000","last_unstale":"2023-10-14T18:17:42.247373+0000","last_undegraded":"2023-10-14T18:17:42.247373+0000","last_fullsized":"2023-10-14T18:17:42.247373+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:10.901473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243996+0000","last_change":"2023-10-14T18:17:27.248764+0000","last_active":"2023-10-14T18:17:42.243996+0000","last_peered":"2023-10-14T18:17:42.243996+0000","last_clean":"2023-10-14T18:17:42.243996+0000","last_became_active":"2023-10-14T18:17:27.248287+0000","last_became_peered":"2023-10-14T18:17:27.248287+0000","last_unstale":"2023-10-14T18:17:42.243996+0000","last_undegraded":"2023-10-14T18:17:42.243996+0000","last_fullsized":"2023-10-14T18:17:42.243996+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:26.912129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.d","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457779+0000","last_change":"2023-10-14T18:17:41.348300+0000","last_active":"2023-10-14T18:17:43.457779+0000","last_peered":"2023-10-14T18:17:43.457779+0000","last_clean":"2023-10-14T18:17:43.457779+0000","last_became_active":"2023-10-14T18:17:41.347703+0000","last_became_peered":"2023-10-14T18:17:41.347703+0000","last_unstale":"2023-10-14T18:17:43.457779+0000","last_undegraded":"2023-10-14T18:17:43.457779+0000","last_fullsized":"2023-10-14T18:17:43.457779+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:20.342562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239426+0000","last_change":"2023-10-14T18:17:42.264682+0000","last_active":"2023-10-14T18:17:43.239426+0000","last_peered":"2023-10-14T18:17:43.239426+0000","last_clean":"2023-10-14T18:17:43.239426+0000","last_became_active":"2023-10-14T18:17:42.264407+0000","last_became_peered":"2023-10-14T18:17:42.264407+0000","last_unstale":"2023-10-14T18:17:43.239426+0000","last_undegraded":"2023-10-14T18:17:43.239426+0000","last_fullsized":"2023-10-14T18:17:43.239426+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.c","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230453+0000","last_change":"2023-10-14T18:17:26.761052+0000","last_active":"2023-10-14T18:17:42.230453+0000","last_peered":"2023-10-14T18:17:42.230453+0000","last_clean":"2023-10-14T18:17:42.230453+0000","last_became_active":"2023-10-14T18:17:26.760754+0000","last_became_peered":"2023-10-14T18:17:26.760754+0000","last_unstale":"2023-10-14T18:17:42.230453+0000","last_undegraded":"2023-10-14T18:17:42.230453+0000","last_fullsized":"2023-10-14T18:17:42.230453+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:54.469184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245367+0000","last_change":"2023-10-14T18:17:27.269260+0000","last_active":"2023-10-14T18:17:42.245367+0000","last_peered":"2023-10-14T18:17:42.245367+0000","last_clean":"2023-10-14T18:17:42.245367+0000","last_became_active":"2023-10-14T18:17:27.269043+0000","last_became_peered":"2023-10-14T18:17:27.269043+0000","last_unstale":"2023-10-14T18:17:42.245367+0000","last_undegraded":"2023-10-14T18:17:42.245367+0000","last_fullsized":"2023-10-14T18:17:42.245367+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:53.145962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.a","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237739+0000","last_change":"2023-10-14T18:17:41.343875+0000","last_active":"2023-10-14T18:17:43.237739+0000","last_peered":"2023-10-14T18:17:43.237739+0000","last_clean":"2023-10-14T18:17:43.237739+0000","last_became_active":"2023-10-14T18:17:41.343534+0000","last_became_peered":"2023-10-14T18:17:41.343534+0000","last_unstale":"2023-10-14T18:17:43.237739+0000","last_undegraded":"2023-10-14T18:17:43.237739+0000","last_fullsized":"2023-10-14T18:17:43.237739+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:51.625931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453016+0000","last_change":"2023-10-14T18:17:42.264988+0000","last_active":"2023-10-14T18:17:43.453016+0000","last_peered":"2023-10-14T18:17:43.453016+0000","last_clean":"2023-10-14T18:17:43.453016+0000","last_became_active":"2023-10-14T18:17:42.264671+0000","last_became_peered":"2023-10-14T18:17:42.264671+0000","last_unstale":"2023-10-14T18:17:43.453016+0000","last_undegraded":"2023-10-14T18:17:43.453016+0000","last_fullsized":"2023-10-14T18:17:43.453016+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:19:16.031832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.d","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247413+0000","last_change":"2023-10-14T18:17:26.760063+0000","last_active":"2023-10-14T18:17:42.247413+0000","last_peered":"2023-10-14T18:17:42.247413+0000","last_clean":"2023-10-14T18:17:42.247413+0000","last_became_active":"2023-10-14T18:17:26.758718+0000","last_became_peered":"2023-10-14T18:17:26.758718+0000","last_unstale":"2023-10-14T18:17:42.247413+0000","last_undegraded":"2023-10-14T18:17:42.247413+0000","last_fullsized":"2023-10-14T18:17:42.247413+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:29.586206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230177+0000","last_change":"2023-10-14T18:17:27.258724+0000","last_active":"2023-10-14T18:17:42.230177+0000","last_peered":"2023-10-14T18:17:42.230177+0000","last_clean":"2023-10-14T18:17:42.230177+0000","last_became_active":"2023-10-14T18:17:27.255949+0000","last_became_peered":"2023-10-14T18:17:27.255949+0000","last_unstale":"2023-10-14T18:17:42.230177+0000","last_undegraded":"2023-10-14T18:17:42.230177+0000","last_fullsized":"2023-10-14T18:17:42.230177+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:58.985698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.b","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240528+0000","last_change":"2023-10-14T18:17:41.347603+0000","last_active":"2023-10-14T18:17:43.240528+0000","last_peered":"2023-10-14T18:17:43.240528+0000","last_clean":"2023-10-14T18:17:43.240528+0000","last_became_active":"2023-10-14T18:17:41.347226+0000","last_became_peered":"2023-10-14T18:17:41.347226+0000","last_unstale":"2023-10-14T18:17:43.240528+0000","last_undegraded":"2023-10-14T18:17:43.240528+0000","last_fullsized":"2023-10-14T18:17:43.240528+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:55.118793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.9","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453241+0000","last_change":"2023-10-14T18:17:42.267743+0000","last_active":"2023-10-14T18:17:43.453241+0000","last_peered":"2023-10-14T18:17:43.453241+0000","last_clean":"2023-10-14T18:17:43.453241+0000","last_became_active":"2023-10-14T18:17:42.267190+0000","last_became_peered":"2023-10-14T18:17:42.267190+0000","last_unstale":"2023-10-14T18:17:43.453241+0000","last_undegraded":"2023-10-14T18:17:43.453241+0000","last_fullsized":"2023-10-14T18:17:43.453241+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:30:49.569484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.e","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247607+0000","last_change":"2023-10-14T18:17:26.766615+0000","last_active":"2023-10-14T18:17:42.247607+0000","last_peered":"2023-10-14T18:17:42.247607+0000","last_clean":"2023-10-14T18:17:42.247607+0000","last_became_active":"2023-10-14T18:17:26.766167+0000","last_became_peered":"2023-10-14T18:17:26.766167+0000","last_unstale":"2023-10-14T18:17:42.247607+0000","last_undegraded":"2023-10-14T18:17:42.247607+0000","last_fullsized":"2023-10-14T18:17:42.247607+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:13.374759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249934+0000","last_change":"2023-10-14T18:17:27.260977+0000","last_active":"2023-10-14T18:17:42.249934+0000","last_peered":"2023-10-14T18:17:42.249934+0000","last_clean":"2023-10-14T18:17:42.249934+0000","last_became_active":"2023-10-14T18:17:27.260706+0000","last_became_peered":"2023-10-14T18:17:27.260706+0000","last_unstale":"2023-10-14T18:17:42.249934+0000","last_undegraded":"2023-10-14T18:17:42.249934+0000","last_fullsized":"2023-10-14T18:17:42.249934+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:45:02.396945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.8","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238167+0000","last_change":"2023-10-14T18:17:41.344229+0000","last_active":"2023-10-14T18:17:43.238167+0000","last_peered":"2023-10-14T18:17:43.238167+0000","last_clean":"2023-10-14T18:17:43.238167+0000","last_became_active":"2023-10-14T18:17:41.343528+0000","last_became_peered":"2023-10-14T18:17:41.343528+0000","last_unstale":"2023-10-14T18:17:43.238167+0000","last_undegraded":"2023-10-14T18:17:43.238167+0000","last_fullsized":"2023-10-14T18:17:43.238167+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:31.144229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.8","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257422+0000","last_change":"2023-10-14T18:17:42.261265+0000","last_active":"2023-10-14T18:17:43.257422+0000","last_peered":"2023-10-14T18:17:43.257422+0000","last_clean":"2023-10-14T18:17:43.257422+0000","last_became_active":"2023-10-14T18:17:42.261000+0000","last_became_peered":"2023-10-14T18:17:42.261000+0000","last_unstale":"2023-10-14T18:17:43.257422+0000","last_undegraded":"2023-10-14T18:17:43.257422+0000","last_fullsized":"2023-10-14T18:17:43.257422+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"26.f","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243042+0000","last_change":"2023-10-14T18:17:26.763127+0000","last_active":"2023-10-14T18:17:42.243042+0000","last_peered":"2023-10-14T18:17:42.243042+0000","last_clean":"2023-10-14T18:17:42.243042+0000","last_became_active":"2023-10-14T18:17:26.762581+0000","last_became_peered":"2023-10-14T18:17:26.762581+0000","last_unstale":"2023-10-14T18:17:42.243042+0000","last_undegraded":"2023-10-14T18:17:42.243042+0000","last_fullsized":"2023-10-14T18:17:42.243042+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:15.476495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247063+0000","last_change":"2023-10-14T18:17:27.265001+0000","last_active":"2023-10-14T18:17:42.247063+0000","last_peered":"2023-10-14T18:17:42.247063+0000","last_clean":"2023-10-14T18:17:42.247063+0000","last_became_active":"2023-10-14T18:17:27.264106+0000","last_became_peered":"2023-10-14T18:17:27.264106+0000","last_unstale":"2023-10-14T18:17:42.247063+0000","last_undegraded":"2023-10-14T18:17:42.247063+0000","last_fullsized":"2023-10-14T18:17:42.247063+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:28:14.694764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.9","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237357+0000","last_change":"2023-10-14T18:17:41.347521+0000","last_active":"2023-10-14T18:17:43.237357+0000","last_peered":"2023-10-14T18:17:43.237357+0000","last_clean":"2023-10-14T18:17:43.237357+0000","last_became_active":"2023-10-14T18:17:41.347194+0000","last_became_peered":"2023-10-14T18:17:41.347194+0000","last_unstale":"2023-10-14T18:17:43.237357+0000","last_undegraded":"2023-10-14T18:17:43.237357+0000","last_fullsized":"2023-10-14T18:17:43.237357+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:49:07.446346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.17","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458750+0000","last_change":"2023-10-14T18:17:42.268477+0000","last_active":"2023-10-14T18:17:43.458750+0000","last_peered":"2023-10-14T18:17:43.458750+0000","last_clean":"2023-10-14T18:17:43.458750+0000","last_became_active":"2023-10-14T18:17:42.265942+0000","last_became_peered":"2023-10-14T18:17:42.265942+0000","last_unstale":"2023-10-14T18:17:43.458750+0000","last_undegraded":"2023-10-14T18:17:43.458750+0000","last_fullsized":"2023-10-14T18:17:43.458750+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:12.806244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.10","version":"228'8","reported_seq":24,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230606+0000","last_change":"2023-10-14T18:17:26.758995+0000","last_active":"2023-10-14T18:17:42.230606+0000","last_peered":"2023-10-14T18:17:42.230606+0000","last_clean":"2023-10-14T18:17:42.230606+0000","last_became_active":"2023-10-14T18:17:26.758694+0000","last_became_peered":"2023-10-14T18:17:26.758694+0000","last_unstale":"2023-10-14T18:17:42.230606+0000","last_undegraded":"2023-10-14T18:17:42.230606+0000","last_fullsized":"2023-10-14T18:17:42.230606+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:20.271832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.11","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247914+0000","last_change":"2023-10-14T18:17:27.258412+0000","last_active":"2023-10-14T18:17:42.247914+0000","last_peered":"2023-10-14T18:17:42.247914+0000","last_clean":"2023-10-14T18:17:42.247914+0000","last_became_active":"2023-10-14T18:17:27.257835+0000","last_became_peered":"2023-10-14T18:17:27.257835+0000","last_unstale":"2023-10-14T18:17:42.247914+0000","last_undegraded":"2023-10-14T18:17:42.247914+0000","last_fullsized":"2023-10-14T18:17:42.247914+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.16","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454318+0000","last_change":"2023-10-14T18:17:41.338219+0000","last_active":"2023-10-14T18:17:43.454318+0000","last_peered":"2023-10-14T18:17:43.454318+0000","last_clean":"2023-10-14T18:17:43.454318+0000","last_became_active":"2023-10-14T18:17:41.335426+0000","last_became_peered":"2023-10-14T18:17:41.335426+0000","last_unstale":"2023-10-14T18:17:43.454318+0000","last_undegraded":"2023-10-14T18:17:43.454318+0000","last_fullsized":"2023-10-14T18:17:43.454318+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:49.688313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.16","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236401+0000","last_change":"2023-10-14T18:17:42.271606+0000","last_active":"2023-10-14T18:17:43.236401+0000","last_peered":"2023-10-14T18:17:43.236401+0000","last_clean":"2023-10-14T18:17:43.236401+0000","last_became_active":"2023-10-14T18:17:42.271301+0000","last_became_peered":"2023-10-14T18:17:42.271301+0000","last_unstale":"2023-10-14T18:17:43.236401+0000","last_undegraded":"2023-10-14T18:17:43.236401+0000","last_fullsized":"2023-10-14T18:17:43.236401+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:48:02.966918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.11","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243585+0000","last_change":"2023-10-14T18:17:26.760901+0000","last_active":"2023-10-14T18:17:42.243585+0000","last_peered":"2023-10-14T18:17:42.243585+0000","last_clean":"2023-10-14T18:17:42.243585+0000","last_became_active":"2023-10-14T18:17:26.760610+0000","last_became_peered":"2023-10-14T18:17:26.760610+0000","last_unstale":"2023-10-14T18:17:42.243585+0000","last_undegraded":"2023-10-14T18:17:42.243585+0000","last_fullsized":"2023-10-14T18:17:42.243585+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:03:16.410048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.10","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.230239+0000","last_change":"2023-10-14T18:17:27.247857+0000","last_active":"2023-10-14T18:17:42.230239+0000","last_peered":"2023-10-14T18:17:42.230239+0000","last_clean":"2023-10-14T18:17:42.230239+0000","last_became_active":"2023-10-14T18:17:27.247391+0000","last_became_peered":"2023-10-14T18:17:27.247391+0000","last_unstale":"2023-10-14T18:17:42.230239+0000","last_undegraded":"2023-10-14T18:17:42.230239+0000","last_fullsized":"2023-10-14T18:17:42.230239+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.17","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458579+0000","last_change":"2023-10-14T18:17:41.345492+0000","last_active":"2023-10-14T18:17:43.458579+0000","last_peered":"2023-10-14T18:17:43.458579+0000","last_clean":"2023-10-14T18:17:43.458579+0000","last_became_active":"2023-10-14T18:17:41.345124+0000","last_became_peered":"2023-10-14T18:17:41.345124+0000","last_unstale":"2023-10-14T18:17:43.458579+0000","last_undegraded":"2023-10-14T18:17:43.458579+0000","last_fullsized":"2023-10-14T18:17:43.458579+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:20.202542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.15","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457519+0000","last_change":"2023-10-14T18:17:42.263795+0000","last_active":"2023-10-14T18:17:43.457519+0000","last_peered":"2023-10-14T18:17:43.457519+0000","last_clean":"2023-10-14T18:17:43.457519+0000","last_became_active":"2023-10-14T18:17:42.263533+0000","last_became_peered":"2023-10-14T18:17:42.263533+0000","last_unstale":"2023-10-14T18:17:43.457519+0000","last_undegraded":"2023-10-14T18:17:43.457519+0000","last_fullsized":"2023-10-14T18:17:43.457519+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:34:10.009963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.12","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255558+0000","last_change":"2023-10-14T18:17:26.758089+0000","last_active":"2023-10-14T18:17:42.255558+0000","last_peered":"2023-10-14T18:17:42.255558+0000","last_clean":"2023-10-14T18:17:42.255558+0000","last_became_active":"2023-10-14T18:17:26.755775+0000","last_became_peered":"2023-10-14T18:17:26.755775+0000","last_unstale":"2023-10-14T18:17:42.255558+0000","last_undegraded":"2023-10-14T18:17:42.255558+0000","last_fullsized":"2023-10-14T18:17:42.255558+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:35:57.967793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.13","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247196+0000","last_change":"2023-10-14T18:17:27.256523+0000","last_active":"2023-10-14T18:17:42.247196+0000","last_peered":"2023-10-14T18:17:42.247196+0000","last_clean":"2023-10-14T18:17:42.247196+0000","last_became_active":"2023-10-14T18:17:27.255795+0000","last_became_peered":"2023-10-14T18:17:27.255795+0000","last_unstale":"2023-10-14T18:17:42.247196+0000","last_undegraded":"2023-10-14T18:17:42.247196+0000","last_fullsized":"2023-10-14T18:17:42.247196+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:52:41.950726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.14","version":"232'6","reported_seq":18,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269712+0000","last_change":"2023-10-14T18:17:41.346722+0000","last_active":"2023-10-14T18:17:43.269712+0000","last_peered":"2023-10-14T18:17:43.269712+0000","last_clean":"2023-10-14T18:17:43.269712+0000","last_became_active":"2023-10-14T18:17:41.346210+0000","last_became_peered":"2023-10-14T18:17:41.346210+0000","last_unstale":"2023-10-14T18:17:43.269712+0000","last_undegraded":"2023-10-14T18:17:43.269712+0000","last_fullsized":"2023-10-14T18:17:43.269712+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:02.494491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.14","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238283+0000","last_change":"2023-10-14T18:17:42.257842+0000","last_active":"2023-10-14T18:17:43.238283+0000","last_peered":"2023-10-14T18:17:43.238283+0000","last_clean":"2023-10-14T18:17:43.238283+0000","last_became_active":"2023-10-14T18:17:42.257561+0000","last_became_peered":"2023-10-14T18:17:42.257561+0000","last_unstale":"2023-10-14T18:17:43.238283+0000","last_undegraded":"2023-10-14T18:17:43.238283+0000","last_fullsized":"2023-10-14T18:17:43.238283+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.13","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252783+0000","last_change":"2023-10-14T18:17:26.754215+0000","last_active":"2023-10-14T18:17:42.252783+0000","last_peered":"2023-10-14T18:17:42.252783+0000","last_clean":"2023-10-14T18:17:42.252783+0000","last_became_active":"2023-10-14T18:17:26.753857+0000","last_became_peered":"2023-10-14T18:17:26.753857+0000","last_unstale":"2023-10-14T18:17:42.252783+0000","last_undegraded":"2023-10-14T18:17:42.252783+0000","last_fullsized":"2023-10-14T18:17:42.252783+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:10.112105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.12","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247205+0000","last_change":"2023-10-14T18:17:27.253122+0000","last_active":"2023-10-14T18:17:42.247205+0000","last_peered":"2023-10-14T18:17:42.247205+0000","last_clean":"2023-10-14T18:17:42.247205+0000","last_became_active":"2023-10-14T18:17:27.252846+0000","last_became_peered":"2023-10-14T18:17:27.252846+0000","last_unstale":"2023-10-14T18:17:42.247205+0000","last_undegraded":"2023-10-14T18:17:42.247205+0000","last_fullsized":"2023-10-14T18:17:42.247205+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:43.166178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"28.15","version":"232'3","reported_seq":15,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454931+0000","last_change":"2023-10-14T18:17:41.335623+0000","last_active":"2023-10-14T18:17:43.454931+0000","last_peered":"2023-10-14T18:17:43.454931+0000","last_clean":"2023-10-14T18:17:43.454931+0000","last_became_active":"2023-10-14T18:17:41.335335+0000","last_became_peered":"2023-10-14T18:17:41.335335+0000","last_unstale":"2023-10-14T18:17:43.454931+0000","last_undegraded":"2023-10-14T18:17:43.454931+0000","last_fullsized":"2023-10-14T18:17:43.454931+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:00:43.220121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.13","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241292+0000","last_change":"2023-10-14T18:17:42.253738+0000","last_active":"2023-10-14T18:17:43.241292+0000","last_peered":"2023-10-14T18:17:43.241292+0000","last_clean":"2023-10-14T18:17:43.241292+0000","last_became_active":"2023-10-14T18:17:42.253386+0000","last_became_peered":"2023-10-14T18:17:42.253386+0000","last_unstale":"2023-10-14T18:17:43.241292+0000","last_undegraded":"2023-10-14T18:17:43.241292+0000","last_fullsized":"2023-10-14T18:17:43.241292+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:52.482235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.14","version":"229'8","reported_seq":24,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250690+0000","last_change":"2023-10-14T18:17:26.761103+0000","last_active":"2023-10-14T18:17:42.250690+0000","last_peered":"2023-10-14T18:17:42.250690+0000","last_clean":"2023-10-14T18:17:42.250690+0000","last_became_active":"2023-10-14T18:17:26.760729+0000","last_became_peered":"2023-10-14T18:17:26.760729+0000","last_unstale":"2023-10-14T18:17:42.250690+0000","last_undegraded":"2023-10-14T18:17:42.250690+0000","last_fullsized":"2023-10-14T18:17:42.250690+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:27.269547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.15","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252649+0000","last_change":"2023-10-14T18:17:27.257920+0000","last_active":"2023-10-14T18:17:42.252649+0000","last_peered":"2023-10-14T18:17:42.252649+0000","last_clean":"2023-10-14T18:17:42.252649+0000","last_became_active":"2023-10-14T18:17:27.256790+0000","last_became_peered":"2023-10-14T18:17:27.256790+0000","last_unstale":"2023-10-14T18:17:42.252649+0000","last_undegraded":"2023-10-14T18:17:42.252649+0000","last_fullsized":"2023-10-14T18:17:42.252649+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:21:55.487089+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.12","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453855+0000","last_change":"2023-10-14T18:17:41.338781+0000","last_active":"2023-10-14T18:17:43.453855+0000","last_peered":"2023-10-14T18:17:43.453855+0000","last_clean":"2023-10-14T18:17:43.453855+0000","last_became_active":"2023-10-14T18:17:41.338426+0000","last_became_peered":"2023-10-14T18:17:41.338426+0000","last_unstale":"2023-10-14T18:17:43.453855+0000","last_undegraded":"2023-10-14T18:17:43.453855+0000","last_fullsized":"2023-10-14T18:17:43.453855+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:03.411232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.12","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453668+0000","last_change":"2023-10-14T18:17:42.255575+0000","last_active":"2023-10-14T18:17:43.453668+0000","last_peered":"2023-10-14T18:17:43.453668+0000","last_clean":"2023-10-14T18:17:43.453668+0000","last_became_active":"2023-10-14T18:17:42.255215+0000","last_became_peered":"2023-10-14T18:17:42.255215+0000","last_unstale":"2023-10-14T18:17:43.453668+0000","last_undegraded":"2023-10-14T18:17:43.453668+0000","last_fullsized":"2023-10-14T18:17:43.453668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.15","version":"228'3","reported_seq":19,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247161+0000","last_change":"2023-10-14T18:17:26.757138+0000","last_active":"2023-10-14T18:17:42.247161+0000","last_peered":"2023-10-14T18:17:42.247161+0000","last_clean":"2023-10-14T18:17:42.247161+0000","last_became_active":"2023-10-14T18:17:26.753946+0000","last_became_peered":"2023-10-14T18:17:26.753946+0000","last_unstale":"2023-10-14T18:17:42.247161+0000","last_undegraded":"2023-10-14T18:17:42.247161+0000","last_fullsized":"2023-10-14T18:17:42.247161+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:54:29.218181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.14","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249566+0000","last_change":"2023-10-14T18:17:27.263006+0000","last_active":"2023-10-14T18:17:42.249566+0000","last_peered":"2023-10-14T18:17:42.249566+0000","last_clean":"2023-10-14T18:17:42.249566+0000","last_became_active":"2023-10-14T18:17:27.262724+0000","last_became_peered":"2023-10-14T18:17:27.262724+0000","last_unstale":"2023-10-14T18:17:42.249566+0000","last_undegraded":"2023-10-14T18:17:42.249566+0000","last_fullsized":"2023-10-14T18:17:42.249566+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:54:28.798619+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.13","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257605+0000","last_change":"2023-10-14T18:17:41.344858+0000","last_active":"2023-10-14T18:17:43.257605+0000","last_peered":"2023-10-14T18:17:43.257605+0000","last_clean":"2023-10-14T18:17:43.257605+0000","last_became_active":"2023-10-14T18:17:41.344380+0000","last_became_peered":"2023-10-14T18:17:41.344380+0000","last_unstale":"2023-10-14T18:17:43.257605+0000","last_undegraded":"2023-10-14T18:17:43.257605+0000","last_fullsized":"2023-10-14T18:17:43.257605+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:55.118793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.11","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237996+0000","last_change":"2023-10-14T18:17:42.256582+0000","last_active":"2023-10-14T18:17:43.237996+0000","last_peered":"2023-10-14T18:17:43.237996+0000","last_clean":"2023-10-14T18:17:43.237996+0000","last_became_active":"2023-10-14T18:17:42.256309+0000","last_became_peered":"2023-10-14T18:17:42.256309+0000","last_unstale":"2023-10-14T18:17:43.237996+0000","last_undegraded":"2023-10-14T18:17:43.237996+0000","last_fullsized":"2023-10-14T18:17:43.237996+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:54.731356+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.16","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255378+0000","last_change":"2023-10-14T18:17:26.766188+0000","last_active":"2023-10-14T18:17:42.255378+0000","last_peered":"2023-10-14T18:17:42.255378+0000","last_clean":"2023-10-14T18:17:42.255378+0000","last_became_active":"2023-10-14T18:17:26.765885+0000","last_became_peered":"2023-10-14T18:17:26.765885+0000","last_unstale":"2023-10-14T18:17:42.255378+0000","last_undegraded":"2023-10-14T18:17:42.255378+0000","last_fullsized":"2023-10-14T18:17:42.255378+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:00.302980+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.17","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.232551+0000","last_change":"2023-10-14T18:17:27.251334+0000","last_active":"2023-10-14T18:17:42.232551+0000","last_peered":"2023-10-14T18:17:42.232551+0000","last_clean":"2023-10-14T18:17:42.232551+0000","last_became_active":"2023-10-14T18:17:27.251040+0000","last_became_peered":"2023-10-14T18:17:27.251040+0000","last_unstale":"2023-10-14T18:17:42.232551+0000","last_undegraded":"2023-10-14T18:17:42.232551+0000","last_fullsized":"2023-10-14T18:17:42.232551+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:30:11.130011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.10","version":"232'8","reported_seq":20,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233615+0000","last_change":"2023-10-14T18:17:41.340395+0000","last_active":"2023-10-14T18:17:43.233615+0000","last_peered":"2023-10-14T18:17:43.233615+0000","last_clean":"2023-10-14T18:17:43.233615+0000","last_became_active":"2023-10-14T18:17:41.339899+0000","last_became_peered":"2023-10-14T18:17:41.339899+0000","last_unstale":"2023-10-14T18:17:43.233615+0000","last_undegraded":"2023-10-14T18:17:43.233615+0000","last_fullsized":"2023-10-14T18:17:43.233615+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:53.078883+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.10","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.238773+0000","last_change":"2023-10-14T18:17:42.265213+0000","last_active":"2023-10-14T18:17:43.238773+0000","last_peered":"2023-10-14T18:17:43.238773+0000","last_clean":"2023-10-14T18:17:43.238773+0000","last_became_active":"2023-10-14T18:17:42.264925+0000","last_became_peered":"2023-10-14T18:17:42.264925+0000","last_unstale":"2023-10-14T18:17:43.238773+0000","last_undegraded":"2023-10-14T18:17:43.238773+0000","last_fullsized":"2023-10-14T18:17:43.238773+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:04.430721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"26.17","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245388+0000","last_change":"2023-10-14T18:17:26.766951+0000","last_active":"2023-10-14T18:17:42.245388+0000","last_peered":"2023-10-14T18:17:42.245388+0000","last_clean":"2023-10-14T18:17:42.245388+0000","last_became_active":"2023-10-14T18:17:26.766441+0000","last_became_peered":"2023-10-14T18:17:26.766441+0000","last_unstale":"2023-10-14T18:17:42.245388+0000","last_undegraded":"2023-10-14T18:17:42.245388+0000","last_fullsized":"2023-10-14T18:17:42.245388+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:34.151644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.16","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.254959+0000","last_change":"2023-10-14T18:17:27.267257+0000","last_active":"2023-10-14T18:17:42.254959+0000","last_peered":"2023-10-14T18:17:42.254959+0000","last_clean":"2023-10-14T18:17:42.254959+0000","last_became_active":"2023-10-14T18:17:27.266927+0000","last_became_peered":"2023-10-14T18:17:27.266927+0000","last_unstale":"2023-10-14T18:17:42.254959+0000","last_undegraded":"2023-10-14T18:17:42.254959+0000","last_fullsized":"2023-10-14T18:17:42.254959+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:43.360869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.11","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453273+0000","last_change":"2023-10-14T18:17:41.339830+0000","last_active":"2023-10-14T18:17:43.453273+0000","last_peered":"2023-10-14T18:17:43.453273+0000","last_clean":"2023-10-14T18:17:43.453273+0000","last_became_active":"2023-10-14T18:17:41.337933+0000","last_became_peered":"2023-10-14T18:17:41.337933+0000","last_unstale":"2023-10-14T18:17:43.453273+0000","last_undegraded":"2023-10-14T18:17:43.453273+0000","last_fullsized":"2023-10-14T18:17:43.453273+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:06:53.445917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.259249+0000","last_change":"2023-10-14T18:17:42.254340+0000","last_active":"2023-10-14T18:17:43.259249+0000","last_peered":"2023-10-14T18:17:43.259249+0000","last_clean":"2023-10-14T18:17:43.259249+0000","last_became_active":"2023-10-14T18:17:42.254061+0000","last_became_peered":"2023-10-14T18:17:42.254061+0000","last_unstale":"2023-10-14T18:17:43.259249+0000","last_undegraded":"2023-10-14T18:17:43.259249+0000","last_fullsized":"2023-10-14T18:17:43.259249+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:24:25.331480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"28.1e","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236230+0000","last_change":"2023-10-14T18:17:41.346764+0000","last_active":"2023-10-14T18:17:43.236230+0000","last_peered":"2023-10-14T18:17:43.236230+0000","last_clean":"2023-10-14T18:17:43.236230+0000","last_became_active":"2023-10-14T18:17:41.346452+0000","last_became_peered":"2023-10-14T18:17:41.346452+0000","last_unstale":"2023-10-14T18:17:43.236230+0000","last_undegraded":"2023-10-14T18:17:43.236230+0000","last_fullsized":"2023-10-14T18:17:43.236230+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:51.462746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.18","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252678+0000","last_change":"2023-10-14T18:17:26.757809+0000","last_active":"2023-10-14T18:17:42.252678+0000","last_peered":"2023-10-14T18:17:42.252678+0000","last_clean":"2023-10-14T18:17:42.252678+0000","last_became_active":"2023-10-14T18:17:26.757324+0000","last_became_peered":"2023-10-14T18:17:26.757324+0000","last_unstale":"2023-10-14T18:17:42.252678+0000","last_undegraded":"2023-10-14T18:17:42.252678+0000","last_fullsized":"2023-10-14T18:17:42.252678+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:06:16.576762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.19","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.245621+0000","last_change":"2023-10-14T18:17:27.266633+0000","last_active":"2023-10-14T18:17:42.245621+0000","last_peered":"2023-10-14T18:17:42.245621+0000","last_clean":"2023-10-14T18:17:42.245621+0000","last_became_active":"2023-10-14T18:17:27.265995+0000","last_became_peered":"2023-10-14T18:17:27.265995+0000","last_unstale":"2023-10-14T18:17:42.245621+0000","last_undegraded":"2023-10-14T18:17:42.245621+0000","last_fullsized":"2023-10-14T18:17:42.245621+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:13.429707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.1e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240998+0000","last_change":"2023-10-14T18:17:42.269999+0000","last_active":"2023-10-14T18:17:43.240998+0000","last_peered":"2023-10-14T18:17:43.240998+0000","last_clean":"2023-10-14T18:17:43.240998+0000","last_became_active":"2023-10-14T18:17:42.266497+0000","last_became_peered":"2023-10-14T18:17:42.266497+0000","last_unstale":"2023-10-14T18:17:43.240998+0000","last_undegraded":"2023-10-14T18:17:43.240998+0000","last_fullsized":"2023-10-14T18:17:43.240998+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:28:54.098372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"28.1f","version":"232'4","reported_seq":16,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454775+0000","last_change":"2023-10-14T18:17:41.340721+0000","last_active":"2023-10-14T18:17:43.454775+0000","last_peered":"2023-10-14T18:17:43.454775+0000","last_clean":"2023-10-14T18:17:43.454775+0000","last_became_active":"2023-10-14T18:17:41.340418+0000","last_became_peered":"2023-10-14T18:17:41.340418+0000","last_unstale":"2023-10-14T18:17:43.454775+0000","last_undegraded":"2023-10-14T18:17:43.454775+0000","last_fullsized":"2023-10-14T18:17:43.454775+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:02.098277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"26.19","version":"0'0","reported_seq":16,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248662+0000","last_change":"2023-10-14T18:17:26.752387+0000","last_active":"2023-10-14T18:17:42.248662+0000","last_peered":"2023-10-14T18:17:42.248662+0000","last_clean":"2023-10-14T18:17:42.248662+0000","last_became_active":"2023-10-14T18:17:26.752097+0000","last_became_peered":"2023-10-14T18:17:26.752097+0000","last_unstale":"2023-10-14T18:17:42.248662+0000","last_undegraded":"2023-10-14T18:17:42.248662+0000","last_fullsized":"2023-10-14T18:17:42.248662+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:53:47.530809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.18","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240520+0000","last_change":"2023-10-14T18:17:27.263025+0000","last_active":"2023-10-14T18:17:42.240520+0000","last_peered":"2023-10-14T18:17:42.240520+0000","last_clean":"2023-10-14T18:17:42.240520+0000","last_became_active":"2023-10-14T18:17:27.262769+0000","last_became_peered":"2023-10-14T18:17:27.262769+0000","last_unstale":"2023-10-14T18:17:42.240520+0000","last_undegraded":"2023-10-14T18:17:42.240520+0000","last_fullsized":"2023-10-14T18:17:42.240520+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:32.166129+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.1d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237401+0000","last_change":"2023-10-14T18:17:42.259060+0000","last_active":"2023-10-14T18:17:43.237401+0000","last_peered":"2023-10-14T18:17:43.237401+0000","last_clean":"2023-10-14T18:17:43.237401+0000","last_became_active":"2023-10-14T18:17:42.258764+0000","last_became_peered":"2023-10-14T18:17:42.258764+0000","last_unstale":"2023-10-14T18:17:43.237401+0000","last_undegraded":"2023-10-14T18:17:43.237401+0000","last_fullsized":"2023-10-14T18:17:43.237401+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:49:05.726762+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"26.1a","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247369+0000","last_change":"2023-10-14T18:17:26.758140+0000","last_active":"2023-10-14T18:17:42.247369+0000","last_peered":"2023-10-14T18:17:42.247369+0000","last_clean":"2023-10-14T18:17:42.247369+0000","last_became_active":"2023-10-14T18:17:26.757834+0000","last_became_peered":"2023-10-14T18:17:26.757834+0000","last_unstale":"2023-10-14T18:17:42.247369+0000","last_undegraded":"2023-10-14T18:17:42.247369+0000","last_fullsized":"2023-10-14T18:17:42.247369+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:39.799748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.1b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248853+0000","last_change":"2023-10-14T18:17:27.265679+0000","last_active":"2023-10-14T18:17:42.248853+0000","last_peered":"2023-10-14T18:17:42.248853+0000","last_clean":"2023-10-14T18:17:42.248853+0000","last_became_active":"2023-10-14T18:17:27.264828+0000","last_became_peered":"2023-10-14T18:17:27.264828+0000","last_unstale":"2023-10-14T18:17:42.248853+0000","last_undegraded":"2023-10-14T18:17:42.248853+0000","last_fullsized":"2023-10-14T18:17:42.248853+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:20:40.110815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"28.1c","version":"232'2","reported_seq":14,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234963+0000","last_change":"2023-10-14T18:17:41.345853+0000","last_active":"2023-10-14T18:17:43.234963+0000","last_peered":"2023-10-14T18:17:43.234963+0000","last_clean":"2023-10-14T18:17:43.234963+0000","last_became_active":"2023-10-14T18:17:41.345579+0000","last_became_peered":"2023-10-14T18:17:41.345579+0000","last_unstale":"2023-10-14T18:17:43.234963+0000","last_undegraded":"2023-10-14T18:17:43.234963+0000","last_fullsized":"2023-10-14T18:17:43.234963+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:51:45.715692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.1b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453961+0000","last_change":"2023-10-14T18:17:42.258755+0000","last_active":"2023-10-14T18:17:43.453961+0000","last_peered":"2023-10-14T18:17:43.453961+0000","last_clean":"2023-10-14T18:17:43.453961+0000","last_became_active":"2023-10-14T18:17:42.258427+0000","last_became_peered":"2023-10-14T18:17:42.258427+0000","last_unstale":"2023-10-14T18:17:43.453961+0000","last_undegraded":"2023-10-14T18:17:43.453961+0000","last_fullsized":"2023-10-14T18:17:43.453961+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:42.856504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"26.1c","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249707+0000","last_change":"2023-10-14T18:17:26.763585+0000","last_active":"2023-10-14T18:17:42.249707+0000","last_peered":"2023-10-14T18:17:42.249707+0000","last_clean":"2023-10-14T18:17:42.249707+0000","last_became_active":"2023-10-14T18:17:26.763201+0000","last_became_peered":"2023-10-14T18:17:26.763201+0000","last_unstale":"2023-10-14T18:17:42.249707+0000","last_undegraded":"2023-10-14T18:17:42.249707+0000","last_fullsized":"2023-10-14T18:17:42.249707+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:50:47.243311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.1d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244065+0000","last_change":"2023-10-14T18:17:27.250058+0000","last_active":"2023-10-14T18:17:42.244065+0000","last_peered":"2023-10-14T18:17:42.244065+0000","last_clean":"2023-10-14T18:17:42.244065+0000","last_became_active":"2023-10-14T18:17:27.249763+0000","last_became_peered":"2023-10-14T18:17:27.249763+0000","last_unstale":"2023-10-14T18:17:42.244065+0000","last_undegraded":"2023-10-14T18:17:42.244065+0000","last_fullsized":"2023-10-14T18:17:42.244065+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:37:54.713237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"28.1a","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239293+0000","last_change":"2023-10-14T18:17:41.331457+0000","last_active":"2023-10-14T18:17:43.239293+0000","last_peered":"2023-10-14T18:17:43.239293+0000","last_clean":"2023-10-14T18:17:43.239293+0000","last_became_active":"2023-10-14T18:17:41.331123+0000","last_became_peered":"2023-10-14T18:17:41.331123+0000","last_unstale":"2023-10-14T18:17:43.239293+0000","last_undegraded":"2023-10-14T18:17:43.239293+0000","last_fullsized":"2023-10-14T18:17:43.239293+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:17:14.870447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.1a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456949+0000","last_change":"2023-10-14T18:17:42.271854+0000","last_active":"2023-10-14T18:17:43.456949+0000","last_peered":"2023-10-14T18:17:43.456949+0000","last_clean":"2023-10-14T18:17:43.456949+0000","last_became_active":"2023-10-14T18:17:42.271342+0000","last_became_peered":"2023-10-14T18:17:42.271342+0000","last_unstale":"2023-10-14T18:17:43.456949+0000","last_undegraded":"2023-10-14T18:17:43.456949+0000","last_fullsized":"2023-10-14T18:17:43.456949+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:40:30.808342+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"26.1d","version":"228'2","reported_seq":18,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231959+0000","last_change":"2023-10-14T18:17:26.760994+0000","last_active":"2023-10-14T18:17:42.231959+0000","last_peered":"2023-10-14T18:17:42.231959+0000","last_clean":"2023-10-14T18:17:42.231959+0000","last_became_active":"2023-10-14T18:17:26.760187+0000","last_became_peered":"2023-10-14T18:17:26.760187+0000","last_unstale":"2023-10-14T18:17:42.231959+0000","last_undegraded":"2023-10-14T18:17:42.231959+0000","last_fullsized":"2023-10-14T18:17:42.231959+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:41.374021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.1c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252213+0000","last_change":"2023-10-14T18:17:27.261385+0000","last_active":"2023-10-14T18:17:42.252213+0000","last_peered":"2023-10-14T18:17:42.252213+0000","last_clean":"2023-10-14T18:17:42.252213+0000","last_became_active":"2023-10-14T18:17:27.261114+0000","last_became_peered":"2023-10-14T18:17:27.261114+0000","last_unstale":"2023-10-14T18:17:42.252213+0000","last_undegraded":"2023-10-14T18:17:42.252213+0000","last_fullsized":"2023-10-14T18:17:42.252213+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:03:29.463852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"28.1b","version":"232'1","reported_seq":13,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257464+0000","last_change":"2023-10-14T18:17:41.344653+0000","last_active":"2023-10-14T18:17:43.257464+0000","last_peered":"2023-10-14T18:17:43.257464+0000","last_clean":"2023-10-14T18:17:43.257464+0000","last_became_active":"2023-10-14T18:17:41.344370+0000","last_became_peered":"2023-10-14T18:17:41.344370+0000","last_unstale":"2023-10-14T18:17:43.257464+0000","last_undegraded":"2023-10-14T18:17:43.257464+0000","last_fullsized":"2023-10-14T18:17:43.257464+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:34.242471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.19","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233744+0000","last_change":"2023-10-14T18:17:42.265518+0000","last_active":"2023-10-14T18:17:43.233744+0000","last_peered":"2023-10-14T18:17:43.233744+0000","last_clean":"2023-10-14T18:17:43.233744+0000","last_became_active":"2023-10-14T18:17:42.265208+0000","last_became_peered":"2023-10-14T18:17:42.265208+0000","last_unstale":"2023-10-14T18:17:43.233744+0000","last_undegraded":"2023-10-14T18:17:43.233744+0000","last_fullsized":"2023-10-14T18:17:43.233744+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:28.087705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"26.1e","version":"228'1","reported_seq":17,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247383+0000","last_change":"2023-10-14T18:17:26.765153+0000","last_active":"2023-10-14T18:17:42.247383+0000","last_peered":"2023-10-14T18:17:42.247383+0000","last_clean":"2023-10-14T18:17:42.247383+0000","last_became_active":"2023-10-14T18:17:26.764789+0000","last_became_peered":"2023-10-14T18:17:26.764789+0000","last_unstale":"2023-10-14T18:17:42.247383+0000","last_undegraded":"2023-10-14T18:17:42.247383+0000","last_fullsized":"2023-10-14T18:17:42.247383+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:01.621072+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.1f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255635+0000","last_change":"2023-10-14T18:17:27.251816+0000","last_active":"2023-10-14T18:17:42.255635+0000","last_peered":"2023-10-14T18:17:42.255635+0000","last_clean":"2023-10-14T18:17:42.255635+0000","last_became_active":"2023-10-14T18:17:27.250127+0000","last_became_peered":"2023-10-14T18:17:27.250127+0000","last_unstale":"2023-10-14T18:17:42.255635+0000","last_undegraded":"2023-10-14T18:17:42.255635+0000","last_fullsized":"2023-10-14T18:17:42.255635+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:56:32.974437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"28.18","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269045+0000","last_change":"2023-10-14T18:17:41.343616+0000","last_active":"2023-10-14T18:17:43.269045+0000","last_peered":"2023-10-14T18:17:43.269045+0000","last_clean":"2023-10-14T18:17:43.269045+0000","last_became_active":"2023-10-14T18:17:41.343340+0000","last_became_peered":"2023-10-14T18:17:41.343340+0000","last_unstale":"2023-10-14T18:17:43.269045+0000","last_undegraded":"2023-10-14T18:17:43.269045+0000","last_fullsized":"2023-10-14T18:17:43.269045+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:11.786755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.18","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.268914+0000","last_change":"2023-10-14T18:17:42.260145+0000","last_active":"2023-10-14T18:17:43.268914+0000","last_peered":"2023-10-14T18:17:43.268914+0000","last_clean":"2023-10-14T18:17:43.268914+0000","last_became_active":"2023-10-14T18:17:42.259850+0000","last_became_peered":"2023-10-14T18:17:42.259850+0000","last_unstale":"2023-10-14T18:17:43.268914+0000","last_undegraded":"2023-10-14T18:17:43.268914+0000","last_fullsized":"2023-10-14T18:17:43.268914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:12:26.151454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"26.1f","version":"228'4","reported_seq":20,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.239578+0000","last_change":"2023-10-14T18:17:26.768176+0000","last_active":"2023-10-14T18:17:42.239578+0000","last_peered":"2023-10-14T18:17:42.239578+0000","last_clean":"2023-10-14T18:17:42.239578+0000","last_became_active":"2023-10-14T18:17:26.767847+0000","last_became_peered":"2023-10-14T18:17:26.767847+0000","last_unstale":"2023-10-14T18:17:42.239578+0000","last_undegraded":"2023-10-14T18:17:42.239578+0000","last_fullsized":"2023-10-14T18:17:42.239578+0000","mapping_epoch":226,"log_start":"0'0","ondisk_log_start":"0'0","created":226,"last_epoch_clean":227,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:25.702292+0000","last_clean_scrub_stamp":"2023-10-14T18:17:25.702292+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:29:11.998927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.1e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246896+0000","last_change":"2023-10-14T18:17:27.257929+0000","last_active":"2023-10-14T18:17:42.246896+0000","last_peered":"2023-10-14T18:17:42.246896+0000","last_clean":"2023-10-14T18:17:42.246896+0000","last_became_active":"2023-10-14T18:17:27.257029+0000","last_became_peered":"2023-10-14T18:17:27.257029+0000","last_unstale":"2023-10-14T18:17:42.246896+0000","last_undegraded":"2023-10-14T18:17:42.246896+0000","last_fullsized":"2023-10-14T18:17:42.246896+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"28.19","version":"0'0","reported_seq":12,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236955+0000","last_change":"2023-10-14T18:17:41.340719+0000","last_active":"2023-10-14T18:17:43.236955+0000","last_peered":"2023-10-14T18:17:43.236955+0000","last_clean":"2023-10-14T18:17:43.236955+0000","last_became_active":"2023-10-14T18:17:41.340408+0000","last_became_peered":"2023-10-14T18:17:41.340408+0000","last_unstale":"2023-10-14T18:17:43.236955+0000","last_undegraded":"2023-10-14T18:17:43.236955+0000","last_fullsized":"2023-10-14T18:17:43.236955+0000","mapping_epoch":230,"log_start":"0'0","ondisk_log_start":"0'0","created":230,"last_epoch_clean":231,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:40.269759+0000","last_clean_scrub_stamp":"2023-10-14T18:17:40.269759+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:54.022204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.26","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.258910+0000","last_change":"2023-10-14T18:17:42.257031+0000","last_active":"2023-10-14T18:17:43.258910+0000","last_peered":"2023-10-14T18:17:43.258910+0000","last_clean":"2023-10-14T18:17:43.258910+0000","last_became_active":"2023-10-14T18:17:42.256077+0000","last_became_peered":"2023-10-14T18:17:42.256077+0000","last_unstale":"2023-10-14T18:17:43.258910+0000","last_undegraded":"2023-10-14T18:17:43.258910+0000","last_fullsized":"2023-10-14T18:17:43.258910+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.20","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243897+0000","last_change":"2023-10-14T18:17:27.249693+0000","last_active":"2023-10-14T18:17:42.243897+0000","last_peered":"2023-10-14T18:17:42.243897+0000","last_clean":"2023-10-14T18:17:42.243897+0000","last_became_active":"2023-10-14T18:17:27.249386+0000","last_became_peered":"2023-10-14T18:17:27.249386+0000","last_unstale":"2023-10-14T18:17:42.243897+0000","last_undegraded":"2023-10-14T18:17:42.243897+0000","last_fullsized":"2023-10-14T18:17:42.243897+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:17:25.407263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.27","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458363+0000","last_change":"2023-10-14T18:17:42.267891+0000","last_active":"2023-10-14T18:17:43.458363+0000","last_peered":"2023-10-14T18:17:43.458363+0000","last_clean":"2023-10-14T18:17:43.458363+0000","last_became_active":"2023-10-14T18:17:42.267622+0000","last_became_peered":"2023-10-14T18:17:42.267622+0000","last_unstale":"2023-10-14T18:17:43.458363+0000","last_undegraded":"2023-10-14T18:17:43.458363+0000","last_fullsized":"2023-10-14T18:17:43.458363+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:57.010924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.21","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243320+0000","last_change":"2023-10-14T18:17:27.264821+0000","last_active":"2023-10-14T18:17:42.243320+0000","last_peered":"2023-10-14T18:17:42.243320+0000","last_clean":"2023-10-14T18:17:42.243320+0000","last_became_active":"2023-10-14T18:17:27.264504+0000","last_became_peered":"2023-10-14T18:17:27.264504+0000","last_unstale":"2023-10-14T18:17:42.243320+0000","last_undegraded":"2023-10-14T18:17:42.243320+0000","last_fullsized":"2023-10-14T18:17:42.243320+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:50:44.865727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.24","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269575+0000","last_change":"2023-10-14T18:17:42.261418+0000","last_active":"2023-10-14T18:17:43.269575+0000","last_peered":"2023-10-14T18:17:43.269575+0000","last_clean":"2023-10-14T18:17:43.269575+0000","last_became_active":"2023-10-14T18:17:42.259881+0000","last_became_peered":"2023-10-14T18:17:42.259881+0000","last_unstale":"2023-10-14T18:17:43.269575+0000","last_undegraded":"2023-10-14T18:17:43.269575+0000","last_fullsized":"2023-10-14T18:17:43.269575+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:39:51.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.22","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244547+0000","last_change":"2023-10-14T18:17:27.260534+0000","last_active":"2023-10-14T18:17:42.244547+0000","last_peered":"2023-10-14T18:17:42.244547+0000","last_clean":"2023-10-14T18:17:42.244547+0000","last_became_active":"2023-10-14T18:17:27.259981+0000","last_became_peered":"2023-10-14T18:17:27.259981+0000","last_unstale":"2023-10-14T18:17:42.244547+0000","last_undegraded":"2023-10-14T18:17:42.244547+0000","last_fullsized":"2023-10-14T18:17:42.244547+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:38:21.289738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.25","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457765+0000","last_change":"2023-10-14T18:17:42.267709+0000","last_active":"2023-10-14T18:17:43.457765+0000","last_peered":"2023-10-14T18:17:43.457765+0000","last_clean":"2023-10-14T18:17:43.457765+0000","last_became_active":"2023-10-14T18:17:42.267264+0000","last_became_peered":"2023-10-14T18:17:42.267264+0000","last_unstale":"2023-10-14T18:17:43.457765+0000","last_undegraded":"2023-10-14T18:17:43.457765+0000","last_fullsized":"2023-10-14T18:17:43.457765+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:45:21.177190+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.23","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249937+0000","last_change":"2023-10-14T18:17:27.256462+0000","last_active":"2023-10-14T18:17:42.249937+0000","last_peered":"2023-10-14T18:17:42.249937+0000","last_clean":"2023-10-14T18:17:42.249937+0000","last_became_active":"2023-10-14T18:17:27.256190+0000","last_became_peered":"2023-10-14T18:17:27.256190+0000","last_unstale":"2023-10-14T18:17:42.249937+0000","last_undegraded":"2023-10-14T18:17:42.249937+0000","last_fullsized":"2023-10-14T18:17:42.249937+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:05:42.705277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.22","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456828+0000","last_change":"2023-10-14T18:17:42.259936+0000","last_active":"2023-10-14T18:17:43.456828+0000","last_peered":"2023-10-14T18:17:43.456828+0000","last_clean":"2023-10-14T18:17:43.456828+0000","last_became_active":"2023-10-14T18:17:42.259636+0000","last_became_peered":"2023-10-14T18:17:42.259636+0000","last_unstale":"2023-10-14T18:17:43.456828+0000","last_undegraded":"2023-10-14T18:17:43.456828+0000","last_fullsized":"2023-10-14T18:17:43.456828+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:57:07.931521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.24","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.229930+0000","last_change":"2023-10-14T18:17:27.241163+0000","last_active":"2023-10-14T18:17:42.229930+0000","last_peered":"2023-10-14T18:17:42.229930+0000","last_clean":"2023-10-14T18:17:42.229930+0000","last_became_active":"2023-10-14T18:17:27.240705+0000","last_became_peered":"2023-10-14T18:17:27.240705+0000","last_unstale":"2023-10-14T18:17:42.229930+0000","last_undegraded":"2023-10-14T18:17:42.229930+0000","last_fullsized":"2023-10-14T18:17:42.229930+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:15:08.814231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.23","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237018+0000","last_change":"2023-10-14T18:17:42.258987+0000","last_active":"2023-10-14T18:17:43.237018+0000","last_peered":"2023-10-14T18:17:43.237018+0000","last_clean":"2023-10-14T18:17:43.237018+0000","last_became_active":"2023-10-14T18:17:42.258689+0000","last_became_peered":"2023-10-14T18:17:42.258689+0000","last_unstale":"2023-10-14T18:17:43.237018+0000","last_undegraded":"2023-10-14T18:17:43.237018+0000","last_fullsized":"2023-10-14T18:17:43.237018+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:14:56.138282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.25","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.238601+0000","last_change":"2023-10-14T18:17:27.255856+0000","last_active":"2023-10-14T18:17:42.238601+0000","last_peered":"2023-10-14T18:17:42.238601+0000","last_clean":"2023-10-14T18:17:42.238601+0000","last_became_active":"2023-10-14T18:17:27.255581+0000","last_became_peered":"2023-10-14T18:17:27.255581+0000","last_unstale":"2023-10-14T18:17:42.238601+0000","last_undegraded":"2023-10-14T18:17:42.238601+0000","last_fullsized":"2023-10-14T18:17:42.238601+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:42:42.540612+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.20","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453150+0000","last_change":"2023-10-14T18:17:42.265216+0000","last_active":"2023-10-14T18:17:43.453150+0000","last_peered":"2023-10-14T18:17:43.453150+0000","last_clean":"2023-10-14T18:17:43.453150+0000","last_became_active":"2023-10-14T18:17:42.264773+0000","last_became_peered":"2023-10-14T18:17:42.264773+0000","last_unstale":"2023-10-14T18:17:43.453150+0000","last_undegraded":"2023-10-14T18:17:43.453150+0000","last_fullsized":"2023-10-14T18:17:43.453150+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.26","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252606+0000","last_change":"2023-10-14T18:17:27.262030+0000","last_active":"2023-10-14T18:17:42.252606+0000","last_peered":"2023-10-14T18:17:42.252606+0000","last_clean":"2023-10-14T18:17:42.252606+0000","last_became_active":"2023-10-14T18:17:27.261661+0000","last_became_peered":"2023-10-14T18:17:27.261661+0000","last_unstale":"2023-10-14T18:17:42.252606+0000","last_undegraded":"2023-10-14T18:17:42.252606+0000","last_fullsized":"2023-10-14T18:17:42.252606+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:57:42.391926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.21","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453157+0000","last_change":"2023-10-14T18:17:42.254749+0000","last_active":"2023-10-14T18:17:43.453157+0000","last_peered":"2023-10-14T18:17:43.453157+0000","last_clean":"2023-10-14T18:17:43.453157+0000","last_became_active":"2023-10-14T18:17:42.254204+0000","last_became_peered":"2023-10-14T18:17:42.254204+0000","last_unstale":"2023-10-14T18:17:43.453157+0000","last_undegraded":"2023-10-14T18:17:43.453157+0000","last_fullsized":"2023-10-14T18:17:43.453157+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:34:49.685522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.27","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249844+0000","last_change":"2023-10-14T18:17:27.261126+0000","last_active":"2023-10-14T18:17:42.249844+0000","last_peered":"2023-10-14T18:17:42.249844+0000","last_clean":"2023-10-14T18:17:42.249844+0000","last_became_active":"2023-10-14T18:17:27.260846+0000","last_became_peered":"2023-10-14T18:17:27.260846+0000","last_unstale":"2023-10-14T18:17:42.249844+0000","last_undegraded":"2023-10-14T18:17:42.249844+0000","last_fullsized":"2023-10-14T18:17:42.249844+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:48.933891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.2e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269914+0000","last_change":"2023-10-14T18:17:42.263605+0000","last_active":"2023-10-14T18:17:43.269914+0000","last_peered":"2023-10-14T18:17:43.269914+0000","last_clean":"2023-10-14T18:17:43.269914+0000","last_became_active":"2023-10-14T18:17:42.263265+0000","last_became_peered":"2023-10-14T18:17:42.263265+0000","last_unstale":"2023-10-14T18:17:43.269914+0000","last_undegraded":"2023-10-14T18:17:43.269914+0000","last_fullsized":"2023-10-14T18:17:43.269914+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:08:51.792734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.28","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.243069+0000","last_change":"2023-10-14T18:17:27.262932+0000","last_active":"2023-10-14T18:17:42.243069+0000","last_peered":"2023-10-14T18:17:42.243069+0000","last_clean":"2023-10-14T18:17:42.243069+0000","last_became_active":"2023-10-14T18:17:27.262582+0000","last_became_peered":"2023-10-14T18:17:27.262582+0000","last_unstale":"2023-10-14T18:17:42.243069+0000","last_undegraded":"2023-10-14T18:17:42.243069+0000","last_fullsized":"2023-10-14T18:17:42.243069+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:01:50.613186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237399+0000","last_change":"2023-10-14T18:17:42.252701+0000","last_active":"2023-10-14T18:17:43.237399+0000","last_peered":"2023-10-14T18:17:43.237399+0000","last_clean":"2023-10-14T18:17:43.237399+0000","last_became_active":"2023-10-14T18:17:42.251939+0000","last_became_peered":"2023-10-14T18:17:42.251939+0000","last_unstale":"2023-10-14T18:17:43.237399+0000","last_undegraded":"2023-10-14T18:17:43.237399+0000","last_fullsized":"2023-10-14T18:17:43.237399+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:54:04.430721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.29","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.252882+0000","last_change":"2023-10-14T18:17:27.257364+0000","last_active":"2023-10-14T18:17:42.252882+0000","last_peered":"2023-10-14T18:17:42.252882+0000","last_clean":"2023-10-14T18:17:42.252882+0000","last_became_active":"2023-10-14T18:17:27.257009+0000","last_became_peered":"2023-10-14T18:17:27.257009+0000","last_unstale":"2023-10-14T18:17:42.252882+0000","last_undegraded":"2023-10-14T18:17:42.252882+0000","last_fullsized":"2023-10-14T18:17:42.252882+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:24.487542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269569+0000","last_change":"2023-10-14T18:17:42.259210+0000","last_active":"2023-10-14T18:17:43.269569+0000","last_peered":"2023-10-14T18:17:43.269569+0000","last_clean":"2023-10-14T18:17:43.269569+0000","last_became_active":"2023-10-14T18:17:42.256587+0000","last_became_peered":"2023-10-14T18:17:42.256587+0000","last_unstale":"2023-10-14T18:17:43.269569+0000","last_undegraded":"2023-10-14T18:17:43.269569+0000","last_fullsized":"2023-10-14T18:17:43.269569+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:00.107914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.2a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.255450+0000","last_change":"2023-10-14T18:17:27.256296+0000","last_active":"2023-10-14T18:17:42.255450+0000","last_peered":"2023-10-14T18:17:42.255450+0000","last_clean":"2023-10-14T18:17:42.255450+0000","last_became_active":"2023-10-14T18:17:27.255979+0000","last_became_peered":"2023-10-14T18:17:27.255979+0000","last_unstale":"2023-10-14T18:17:42.255450+0000","last_undegraded":"2023-10-14T18:17:42.255450+0000","last_fullsized":"2023-10-14T18:17:42.255450+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:48:53.896784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"29.2d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454798+0000","last_change":"2023-10-14T18:17:42.262666+0000","last_active":"2023-10-14T18:17:43.454798+0000","last_peered":"2023-10-14T18:17:43.454798+0000","last_clean":"2023-10-14T18:17:43.454798+0000","last_became_active":"2023-10-14T18:17:42.262316+0000","last_became_peered":"2023-10-14T18:17:42.262316+0000","last_unstale":"2023-10-14T18:17:43.454798+0000","last_undegraded":"2023-10-14T18:17:43.454798+0000","last_fullsized":"2023-10-14T18:17:43.454798+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:18.884106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248724+0000","last_change":"2023-10-14T18:17:27.260919+0000","last_active":"2023-10-14T18:17:42.248724+0000","last_peered":"2023-10-14T18:17:42.248724+0000","last_clean":"2023-10-14T18:17:42.248724+0000","last_became_active":"2023-10-14T18:17:27.260634+0000","last_became_peered":"2023-10-14T18:17:27.260634+0000","last_unstale":"2023-10-14T18:17:42.248724+0000","last_undegraded":"2023-10-14T18:17:42.248724+0000","last_fullsized":"2023-10-14T18:17:42.248724+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:51:46.243479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.2a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454032+0000","last_change":"2023-10-14T18:17:42.260933+0000","last_active":"2023-10-14T18:17:43.454032+0000","last_peered":"2023-10-14T18:17:43.454032+0000","last_clean":"2023-10-14T18:17:43.454032+0000","last_became_active":"2023-10-14T18:17:42.260634+0000","last_became_peered":"2023-10-14T18:17:42.260634+0000","last_unstale":"2023-10-14T18:17:43.454032+0000","last_undegraded":"2023-10-14T18:17:43.454032+0000","last_fullsized":"2023-10-14T18:17:43.454032+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:31.063452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.2c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249841+0000","last_change":"2023-10-14T18:17:27.252557+0000","last_active":"2023-10-14T18:17:42.249841+0000","last_peered":"2023-10-14T18:17:42.249841+0000","last_clean":"2023-10-14T18:17:42.249841+0000","last_became_active":"2023-10-14T18:17:27.251949+0000","last_became_peered":"2023-10-14T18:17:27.251949+0000","last_unstale":"2023-10-14T18:17:42.249841+0000","last_undegraded":"2023-10-14T18:17:42.249841+0000","last_fullsized":"2023-10-14T18:17:42.249841+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:45.784134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.2b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453668+0000","last_change":"2023-10-14T18:17:42.270868+0000","last_active":"2023-10-14T18:17:43.453668+0000","last_peered":"2023-10-14T18:17:43.453668+0000","last_clean":"2023-10-14T18:17:43.453668+0000","last_became_active":"2023-10-14T18:17:42.270540+0000","last_became_peered":"2023-10-14T18:17:42.270540+0000","last_unstale":"2023-10-14T18:17:43.453668+0000","last_undegraded":"2023-10-14T18:17:43.453668+0000","last_fullsized":"2023-10-14T18:17:43.453668+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:02.830268+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246899+0000","last_change":"2023-10-14T18:17:27.255618+0000","last_active":"2023-10-14T18:17:42.246899+0000","last_peered":"2023-10-14T18:17:42.246899+0000","last_clean":"2023-10-14T18:17:42.246899+0000","last_became_active":"2023-10-14T18:17:27.255338+0000","last_became_peered":"2023-10-14T18:17:27.255338+0000","last_unstale":"2023-10-14T18:17:42.246899+0000","last_undegraded":"2023-10-14T18:17:42.246899+0000","last_fullsized":"2023-10-14T18:17:42.246899+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:22:06.792999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.28","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.452905+0000","last_change":"2023-10-14T18:17:42.264873+0000","last_active":"2023-10-14T18:17:43.452905+0000","last_peered":"2023-10-14T18:17:43.452905+0000","last_clean":"2023-10-14T18:17:43.452905+0000","last_became_active":"2023-10-14T18:17:42.263922+0000","last_became_peered":"2023-10-14T18:17:42.263922+0000","last_unstale":"2023-10-14T18:17:43.452905+0000","last_undegraded":"2023-10-14T18:17:43.452905+0000","last_fullsized":"2023-10-14T18:17:43.452905+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:07:38.376498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.2e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247417+0000","last_change":"2023-10-14T18:17:27.251453+0000","last_active":"2023-10-14T18:17:42.247417+0000","last_peered":"2023-10-14T18:17:42.247417+0000","last_clean":"2023-10-14T18:17:42.247417+0000","last_became_active":"2023-10-14T18:17:27.247696+0000","last_became_peered":"2023-10-14T18:17:27.247696+0000","last_unstale":"2023-10-14T18:17:42.247417+0000","last_undegraded":"2023-10-14T18:17:42.247417+0000","last_fullsized":"2023-10-14T18:17:42.247417+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:43:29.058123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.29","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.456508+0000","last_change":"2023-10-14T18:17:42.256913+0000","last_active":"2023-10-14T18:17:43.456508+0000","last_peered":"2023-10-14T18:17:43.456508+0000","last_clean":"2023-10-14T18:17:43.456508+0000","last_became_active":"2023-10-14T18:17:42.253387+0000","last_became_peered":"2023-10-14T18:17:42.253387+0000","last_unstale":"2023-10-14T18:17:43.456508+0000","last_undegraded":"2023-10-14T18:17:43.456508+0000","last_fullsized":"2023-10-14T18:17:43.456508+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:37:17.417744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.2f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253511+0000","last_change":"2023-10-14T18:17:27.265286+0000","last_active":"2023-10-14T18:17:42.253511+0000","last_peered":"2023-10-14T18:17:42.253511+0000","last_clean":"2023-10-14T18:17:42.253511+0000","last_became_active":"2023-10-14T18:17:27.265058+0000","last_became_peered":"2023-10-14T18:17:27.265058+0000","last_unstale":"2023-10-14T18:17:42.253511+0000","last_undegraded":"2023-10-14T18:17:42.253511+0000","last_fullsized":"2023-10-14T18:17:42.253511+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:55:37.720266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.36","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269714+0000","last_change":"2023-10-14T18:17:42.259485+0000","last_active":"2023-10-14T18:17:43.269714+0000","last_peered":"2023-10-14T18:17:43.269714+0000","last_clean":"2023-10-14T18:17:43.269714+0000","last_became_active":"2023-10-14T18:17:42.258846+0000","last_became_peered":"2023-10-14T18:17:42.258846+0000","last_unstale":"2023-10-14T18:17:43.269714+0000","last_undegraded":"2023-10-14T18:17:43.269714+0000","last_fullsized":"2023-10-14T18:17:43.269714+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:47:14.232149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.30","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244517+0000","last_change":"2023-10-14T18:17:27.251446+0000","last_active":"2023-10-14T18:17:42.244517+0000","last_peered":"2023-10-14T18:17:42.244517+0000","last_clean":"2023-10-14T18:17:42.244517+0000","last_became_active":"2023-10-14T18:17:27.251165+0000","last_became_peered":"2023-10-14T18:17:27.251165+0000","last_unstale":"2023-10-14T18:17:42.244517+0000","last_undegraded":"2023-10-14T18:17:42.244517+0000","last_fullsized":"2023-10-14T18:17:42.244517+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:35:33.554452+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.37","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.241031+0000","last_change":"2023-10-14T18:17:42.262549+0000","last_active":"2023-10-14T18:17:43.241031+0000","last_peered":"2023-10-14T18:17:43.241031+0000","last_clean":"2023-10-14T18:17:43.241031+0000","last_became_active":"2023-10-14T18:17:42.261856+0000","last_became_peered":"2023-10-14T18:17:42.261856+0000","last_unstale":"2023-10-14T18:17:43.241031+0000","last_undegraded":"2023-10-14T18:17:43.241031+0000","last_fullsized":"2023-10-14T18:17:43.241031+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:54:49.738601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.31","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247774+0000","last_change":"2023-10-14T18:17:27.259030+0000","last_active":"2023-10-14T18:17:42.247774+0000","last_peered":"2023-10-14T18:17:42.247774+0000","last_clean":"2023-10-14T18:17:42.247774+0000","last_became_active":"2023-10-14T18:17:27.257326+0000","last_became_peered":"2023-10-14T18:17:27.257326+0000","last_unstale":"2023-10-14T18:17:42.247774+0000","last_undegraded":"2023-10-14T18:17:42.247774+0000","last_fullsized":"2023-10-14T18:17:42.247774+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.34","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234715+0000","last_change":"2023-10-14T18:17:42.263054+0000","last_active":"2023-10-14T18:17:43.234715+0000","last_peered":"2023-10-14T18:17:43.234715+0000","last_clean":"2023-10-14T18:17:43.234715+0000","last_became_active":"2023-10-14T18:17:42.262632+0000","last_became_peered":"2023-10-14T18:17:42.262632+0000","last_unstale":"2023-10-14T18:17:43.234715+0000","last_undegraded":"2023-10-14T18:17:43.234715+0000","last_fullsized":"2023-10-14T18:17:43.234715+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:08:00.353186+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.32","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.244716+0000","last_change":"2023-10-14T18:17:27.253619+0000","last_active":"2023-10-14T18:17:42.244716+0000","last_peered":"2023-10-14T18:17:42.244716+0000","last_clean":"2023-10-14T18:17:42.244716+0000","last_became_active":"2023-10-14T18:17:27.253369+0000","last_became_peered":"2023-10-14T18:17:27.253369+0000","last_unstale":"2023-10-14T18:17:42.244716+0000","last_undegraded":"2023-10-14T18:17:42.244716+0000","last_fullsized":"2023-10-14T18:17:42.244716+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:31.399100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.35","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.457410+0000","last_change":"2023-10-14T18:17:42.267530+0000","last_active":"2023-10-14T18:17:43.457410+0000","last_peered":"2023-10-14T18:17:43.457410+0000","last_clean":"2023-10-14T18:17:43.457410+0000","last_became_active":"2023-10-14T18:17:42.266908+0000","last_became_peered":"2023-10-14T18:17:42.266908+0000","last_unstale":"2023-10-14T18:17:43.457410+0000","last_undegraded":"2023-10-14T18:17:43.457410+0000","last_fullsized":"2023-10-14T18:17:43.457410+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:27:19.208285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.33","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250116+0000","last_change":"2023-10-14T18:17:27.253928+0000","last_active":"2023-10-14T18:17:42.250116+0000","last_peered":"2023-10-14T18:17:42.250116+0000","last_clean":"2023-10-14T18:17:42.250116+0000","last_became_active":"2023-10-14T18:17:27.253612+0000","last_became_peered":"2023-10-14T18:17:27.253612+0000","last_unstale":"2023-10-14T18:17:42.250116+0000","last_undegraded":"2023-10-14T18:17:42.250116+0000","last_fullsized":"2023-10-14T18:17:42.250116+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:43:02.638978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.32","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.237879+0000","last_change":"2023-10-14T18:17:42.258988+0000","last_active":"2023-10-14T18:17:43.237879+0000","last_peered":"2023-10-14T18:17:43.237879+0000","last_clean":"2023-10-14T18:17:43.237879+0000","last_became_active":"2023-10-14T18:17:42.258551+0000","last_became_peered":"2023-10-14T18:17:42.258551+0000","last_unstale":"2023-10-14T18:17:43.237879+0000","last_undegraded":"2023-10-14T18:17:43.237879+0000","last_fullsized":"2023-10-14T18:17:43.237879+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:50.707802+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"27.34","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240588+0000","last_change":"2023-10-14T18:17:27.253472+0000","last_active":"2023-10-14T18:17:42.240588+0000","last_peered":"2023-10-14T18:17:42.240588+0000","last_clean":"2023-10-14T18:17:42.240588+0000","last_became_active":"2023-10-14T18:17:27.253163+0000","last_became_peered":"2023-10-14T18:17:27.253163+0000","last_unstale":"2023-10-14T18:17:42.240588+0000","last_undegraded":"2023-10-14T18:17:42.240588+0000","last_fullsized":"2023-10-14T18:17:42.240588+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:43:14.392664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"29.33","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.234096+0000","last_change":"2023-10-14T18:17:42.268853+0000","last_active":"2023-10-14T18:17:43.234096+0000","last_peered":"2023-10-14T18:17:43.234096+0000","last_clean":"2023-10-14T18:17:43.234096+0000","last_became_active":"2023-10-14T18:17:42.266796+0000","last_became_peered":"2023-10-14T18:17:42.266796+0000","last_unstale":"2023-10-14T18:17:43.234096+0000","last_undegraded":"2023-10-14T18:17:43.234096+0000","last_fullsized":"2023-10-14T18:17:43.234096+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:34.576269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.35","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231671+0000","last_change":"2023-10-14T18:17:27.249208+0000","last_active":"2023-10-14T18:17:42.231671+0000","last_peered":"2023-10-14T18:17:42.231671+0000","last_clean":"2023-10-14T18:17:42.231671+0000","last_became_active":"2023-10-14T18:17:27.248955+0000","last_became_peered":"2023-10-14T18:17:27.248955+0000","last_unstale":"2023-10-14T18:17:42.231671+0000","last_undegraded":"2023-10-14T18:17:42.231671+0000","last_fullsized":"2023-10-14T18:17:42.231671+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:13:04.646143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.30","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.233962+0000","last_change":"2023-10-14T18:17:42.263050+0000","last_active":"2023-10-14T18:17:43.233962+0000","last_peered":"2023-10-14T18:17:43.233962+0000","last_clean":"2023-10-14T18:17:43.233962+0000","last_became_active":"2023-10-14T18:17:42.262740+0000","last_became_peered":"2023-10-14T18:17:42.262740+0000","last_unstale":"2023-10-14T18:17:43.233962+0000","last_undegraded":"2023-10-14T18:17:43.233962+0000","last_fullsized":"2023-10-14T18:17:43.233962+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:29:57.735230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.36","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247226+0000","last_change":"2023-10-14T18:17:27.254298+0000","last_active":"2023-10-14T18:17:42.247226+0000","last_peered":"2023-10-14T18:17:42.247226+0000","last_clean":"2023-10-14T18:17:42.247226+0000","last_became_active":"2023-10-14T18:17:27.254041+0000","last_became_peered":"2023-10-14T18:17:27.254041+0000","last_unstale":"2023-10-14T18:17:42.247226+0000","last_undegraded":"2023-10-14T18:17:42.247226+0000","last_fullsized":"2023-10-14T18:17:42.247226+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:19:48.453168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"29.31","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.452948+0000","last_change":"2023-10-14T18:17:42.265235+0000","last_active":"2023-10-14T18:17:43.452948+0000","last_peered":"2023-10-14T18:17:43.452948+0000","last_clean":"2023-10-14T18:17:43.452948+0000","last_became_active":"2023-10-14T18:17:42.264946+0000","last_became_peered":"2023-10-14T18:17:42.264946+0000","last_unstale":"2023-10-14T18:17:43.452948+0000","last_undegraded":"2023-10-14T18:17:43.452948+0000","last_fullsized":"2023-10-14T18:17:43.452948+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"27.37","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.250075+0000","last_change":"2023-10-14T18:17:27.266296+0000","last_active":"2023-10-14T18:17:42.250075+0000","last_peered":"2023-10-14T18:17:42.250075+0000","last_clean":"2023-10-14T18:17:42.250075+0000","last_became_active":"2023-10-14T18:17:27.265771+0000","last_became_peered":"2023-10-14T18:17:27.265771+0000","last_unstale":"2023-10-14T18:17:42.250075+0000","last_undegraded":"2023-10-14T18:17:42.250075+0000","last_fullsized":"2023-10-14T18:17:42.250075+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:35:33.752896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.3e","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.236832+0000","last_change":"2023-10-14T18:17:42.271663+0000","last_active":"2023-10-14T18:17:43.236832+0000","last_peered":"2023-10-14T18:17:43.236832+0000","last_clean":"2023-10-14T18:17:43.236832+0000","last_became_active":"2023-10-14T18:17:42.271303+0000","last_became_peered":"2023-10-14T18:17:42.271303+0000","last_unstale":"2023-10-14T18:17:43.236832+0000","last_undegraded":"2023-10-14T18:17:43.236832+0000","last_fullsized":"2023-10-14T18:17:43.236832+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:03:38.279151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"27.38","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.240298+0000","last_change":"2023-10-14T18:17:27.246438+0000","last_active":"2023-10-14T18:17:42.240298+0000","last_peered":"2023-10-14T18:17:42.240298+0000","last_clean":"2023-10-14T18:17:42.240298+0000","last_became_active":"2023-10-14T18:17:27.245859+0000","last_became_peered":"2023-10-14T18:17:27.245859+0000","last_unstale":"2023-10-14T18:17:42.240298+0000","last_undegraded":"2023-10-14T18:17:42.240298+0000","last_fullsized":"2023-10-14T18:17:42.240298+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:20.007218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3f","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.458360+0000","last_change":"2023-10-14T18:17:42.264577+0000","last_active":"2023-10-14T18:17:43.458360+0000","last_peered":"2023-10-14T18:17:43.458360+0000","last_clean":"2023-10-14T18:17:43.458360+0000","last_became_active":"2023-10-14T18:17:42.264261+0000","last_became_peered":"2023-10-14T18:17:42.264261+0000","last_unstale":"2023-10-14T18:17:43.458360+0000","last_undegraded":"2023-10-14T18:17:43.458360+0000","last_fullsized":"2023-10-14T18:17:43.458360+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:25:21.430028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"27.39","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247572+0000","last_change":"2023-10-14T18:17:27.260365+0000","last_active":"2023-10-14T18:17:42.247572+0000","last_peered":"2023-10-14T18:17:42.247572+0000","last_clean":"2023-10-14T18:17:42.247572+0000","last_became_active":"2023-10-14T18:17:27.260109+0000","last_became_peered":"2023-10-14T18:17:27.260109+0000","last_unstale":"2023-10-14T18:17:42.247572+0000","last_undegraded":"2023-10-14T18:17:42.247572+0000","last_fullsized":"2023-10-14T18:17:42.247572+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"29.3c","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.454388+0000","last_change":"2023-10-14T18:17:42.267579+0000","last_active":"2023-10-14T18:17:43.454388+0000","last_peered":"2023-10-14T18:17:43.454388+0000","last_clean":"2023-10-14T18:17:43.454388+0000","last_became_active":"2023-10-14T18:17:42.267166+0000","last_became_peered":"2023-10-14T18:17:42.267166+0000","last_unstale":"2023-10-14T18:17:43.454388+0000","last_undegraded":"2023-10-14T18:17:43.454388+0000","last_fullsized":"2023-10-14T18:17:43.454388+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:41:21.894224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3a","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.247062+0000","last_change":"2023-10-14T18:17:27.257362+0000","last_active":"2023-10-14T18:17:42.247062+0000","last_peered":"2023-10-14T18:17:42.247062+0000","last_clean":"2023-10-14T18:17:42.247062+0000","last_became_active":"2023-10-14T18:17:27.256429+0000","last_became_peered":"2023-10-14T18:17:27.256429+0000","last_unstale":"2023-10-14T18:17:42.247062+0000","last_undegraded":"2023-10-14T18:17:42.247062+0000","last_fullsized":"2023-10-14T18:17:42.247062+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:13.518653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3d","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.239893+0000","last_change":"2023-10-14T18:17:42.255806+0000","last_active":"2023-10-14T18:17:43.239893+0000","last_peered":"2023-10-14T18:17:43.239893+0000","last_clean":"2023-10-14T18:17:43.239893+0000","last_became_active":"2023-10-14T18:17:42.255525+0000","last_became_peered":"2023-10-14T18:17:42.255525+0000","last_unstale":"2023-10-14T18:17:43.239893+0000","last_undegraded":"2023-10-14T18:17:43.239893+0000","last_fullsized":"2023-10-14T18:17:43.239893+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:55.386705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3b","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.248951+0000","last_change":"2023-10-14T18:17:27.255198+0000","last_active":"2023-10-14T18:17:42.248951+0000","last_peered":"2023-10-14T18:17:42.248951+0000","last_clean":"2023-10-14T18:17:42.248951+0000","last_became_active":"2023-10-14T18:17:27.254906+0000","last_became_peered":"2023-10-14T18:17:27.254906+0000","last_unstale":"2023-10-14T18:17:42.248951+0000","last_undegraded":"2023-10-14T18:17:42.248951+0000","last_fullsized":"2023-10-14T18:17:42.248951+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:40:37.913184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"29.3a","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.269089+0000","last_change":"2023-10-14T18:17:42.262376+0000","last_active":"2023-10-14T18:17:43.269089+0000","last_peered":"2023-10-14T18:17:43.269089+0000","last_clean":"2023-10-14T18:17:43.269089+0000","last_became_active":"2023-10-14T18:17:42.262080+0000","last_became_peered":"2023-10-14T18:17:42.262080+0000","last_unstale":"2023-10-14T18:17:43.269089+0000","last_undegraded":"2023-10-14T18:17:43.269089+0000","last_fullsized":"2023-10-14T18:17:43.269089+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:29:07.397894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"27.3c","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.249439+0000","last_change":"2023-10-14T18:17:27.266341+0000","last_active":"2023-10-14T18:17:42.249439+0000","last_peered":"2023-10-14T18:17:42.249439+0000","last_clean":"2023-10-14T18:17:42.249439+0000","last_became_active":"2023-10-14T18:17:27.265032+0000","last_became_peered":"2023-10-14T18:17:27.265032+0000","last_unstale":"2023-10-14T18:17:42.249439+0000","last_undegraded":"2023-10-14T18:17:42.249439+0000","last_fullsized":"2023-10-14T18:17:42.249439+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:27:53.877988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"29.3b","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.453757+0000","last_change":"2023-10-14T18:17:42.259554+0000","last_active":"2023-10-14T18:17:43.453757+0000","last_peered":"2023-10-14T18:17:43.453757+0000","last_clean":"2023-10-14T18:17:43.453757+0000","last_became_active":"2023-10-14T18:17:42.259239+0000","last_became_peered":"2023-10-14T18:17:42.259239+0000","last_unstale":"2023-10-14T18:17:43.453757+0000","last_undegraded":"2023-10-14T18:17:43.453757+0000","last_fullsized":"2023-10-14T18:17:43.453757+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:06.131503+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"27.3d","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.231667+0000","last_change":"2023-10-14T18:17:27.266231+0000","last_active":"2023-10-14T18:17:42.231667+0000","last_peered":"2023-10-14T18:17:42.231667+0000","last_clean":"2023-10-14T18:17:42.231667+0000","last_became_active":"2023-10-14T18:17:27.266037+0000","last_became_peered":"2023-10-14T18:17:27.266037+0000","last_unstale":"2023-10-14T18:17:42.231667+0000","last_undegraded":"2023-10-14T18:17:42.231667+0000","last_fullsized":"2023-10-14T18:17:42.231667+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:26:55.110847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"29.38","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.257562+0000","last_change":"2023-10-14T18:17:42.253138+0000","last_active":"2023-10-14T18:17:43.257562+0000","last_peered":"2023-10-14T18:17:43.257562+0000","last_clean":"2023-10-14T18:17:43.257562+0000","last_became_active":"2023-10-14T18:17:42.252767+0000","last_became_peered":"2023-10-14T18:17:42.252767+0000","last_unstale":"2023-10-14T18:17:43.257562+0000","last_undegraded":"2023-10-14T18:17:43.257562+0000","last_fullsized":"2023-10-14T18:17:43.257562+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.117766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"27.3e","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.246730+0000","last_change":"2023-10-14T18:17:27.261872+0000","last_active":"2023-10-14T18:17:42.246730+0000","last_peered":"2023-10-14T18:17:42.246730+0000","last_clean":"2023-10-14T18:17:42.246730+0000","last_became_active":"2023-10-14T18:17:27.261591+0000","last_became_peered":"2023-10-14T18:17:27.261591+0000","last_unstale":"2023-10-14T18:17:42.246730+0000","last_undegraded":"2023-10-14T18:17:42.246730+0000","last_fullsized":"2023-10-14T18:17:42.246730+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:54:15.724791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"29.39","version":"0'0","reported_seq":11,"reported_epoch":233,"state":"active+clean","last_fresh":"2023-10-14T18:17:43.240089+0000","last_change":"2023-10-14T18:17:42.254646+0000","last_active":"2023-10-14T18:17:43.240089+0000","last_peered":"2023-10-14T18:17:43.240089+0000","last_clean":"2023-10-14T18:17:43.240089+0000","last_became_active":"2023-10-14T18:17:42.254349+0000","last_became_peered":"2023-10-14T18:17:42.254349+0000","last_unstale":"2023-10-14T18:17:43.240089+0000","last_undegraded":"2023-10-14T18:17:43.240089+0000","last_fullsized":"2023-10-14T18:17:43.240089+0000","mapping_epoch":231,"log_start":"0'0","ondisk_log_start":"0'0","created":231,"last_epoch_clean":232,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:41.289249+0000","last_clean_scrub_stamp":"2023-10-14T18:17:41.289249+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:36:42.705923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"27.3f","version":"0'0","reported_seq":15,"reported_epoch":232,"state":"active+clean","last_fresh":"2023-10-14T18:17:42.253383+0000","last_change":"2023-10-14T18:17:27.265344+0000","last_active":"2023-10-14T18:17:42.253383+0000","last_peered":"2023-10-14T18:17:42.253383+0000","last_clean":"2023-10-14T18:17:42.253383+0000","last_became_active":"2023-10-14T18:17:27.265080+0000","last_became_peered":"2023-10-14T18:17:27.265080+0000","last_unstale":"2023-10-14T18:17:42.253383+0000","last_undegraded":"2023-10-14T18:17:42.253383+0000","last_fullsized":"2023-10-14T18:17:42.253383+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":227,"last_epoch_clean":228,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:17:26.720197+0000","last_clean_scrub_stamp":"2023-10-14T18:17:26.720197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:00.709530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":29,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":28,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":26,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":27,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2132,"num_read_kb":4426,"num_write":579,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":348,"ondisk_log_size":348,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576008,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56836,"kb_used_data":3448,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315004,"statfs":{"total":96636764160,"available":96578564096,"internally_reserved":0,"allocated":3530752,"data_stored":2044530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.21,"5min":1.216,"15min":1.2090000000000001},"min":{"1min":0.68200000000000005,"5min":0.67400000000000004,"15min":0.67400000000000004},"max":{"1min":1.573,"5min":1.7030000000000001,"15min":2.1669999999999998},"last":2.0489999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.232,"15min":1.228},"min":{"1min":0.78300000000000003,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.6619999999999999,"5min":1.76,"15min":2.2149999999999999},"last":1.361}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.27,"5min":1.264,"15min":1.276},"min":{"1min":0.67500000000000004,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.9059999999999999,"5min":2.214,"15min":2.214},"last":1.768},{"interface":"front","average":{"1min":1.341,"5min":1.3120000000000001,"15min":1.3160000000000001},"min":{"1min":0.871,"5min":0.73499999999999999,"15min":0.73499999999999999},"max":{"1min":2.0510000000000002,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.71}]},{"osd":2,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.151,"5min":1.2230000000000001,"15min":1.2669999999999999},"min":{"1min":0.73799999999999999,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.546,"5min":1.6799999999999999,"15min":2.004},"last":0.96499999999999997},{"interface":"front","average":{"1min":1.2889999999999999,"5min":1.325,"15min":1.3700000000000001},"min":{"1min":0.85199999999999998,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.764,"5min":1.8260000000000001,"15min":2.2010000000000001},"last":1.9390000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.413,"5min":1.3819999999999999,"15min":1.4339999999999999},"min":{"1min":1.0009999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.72,"5min":1.8500000000000001,"15min":2.4430000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.4930000000000001,"5min":1.5109999999999999,"15min":1.536},"min":{"1min":1.0429999999999999,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.875,"5min":1.9079999999999999,"15min":2.2810000000000001},"last":1.994}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3220000000000001,"15min":1.331},"min":{"1min":0.97699999999999998,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.8029999999999999,"5min":1.8200000000000001,"15min":2.367},"last":1.4990000000000001},{"interface":"front","average":{"1min":1.3580000000000001,"5min":1.359,"15min":1.3839999999999999},"min":{"1min":0.94199999999999995,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":1.8360000000000001,"5min":1.865,"15min":2.661},"last":1.659}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.478,"5min":1.4390000000000001,"15min":1.462},"min":{"1min":1.048,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.7729999999999999,"5min":1.8740000000000001,"15min":2.516},"last":1.873},{"interface":"front","average":{"1min":1.5449999999999999,"5min":1.4890000000000001,"15min":1.5229999999999999},"min":{"1min":1.1870000000000001,"5min":1.044,"15min":0.95399999999999996},"max":{"1min":2.1800000000000002,"5min":2.1800000000000002,"15min":2.448},"last":1.6120000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.6499999999999999,"5min":1.5960000000000001,"15min":1.633},"min":{"1min":1.3169999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.2509999999999999,"5min":2.2509999999999999,"15min":2.4849999999999999},"last":1.8180000000000001},{"interface":"front","average":{"1min":1.526,"5min":1.5349999999999999,"15min":1.5800000000000001},"min":{"1min":1.115,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":1.948,"5min":1.988,"15min":2.8159999999999998},"last":1.1859999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576007,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":29884,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94341956,"statfs":{"total":96636764160,"available":96606162944,"internally_reserved":0,"allocated":2932736,"data_stored":1451873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.135,"5min":1.198,"15min":1.1879999999999999},"min":{"1min":0.754,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.6319999999999999,"5min":2.085,"15min":2.085},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.2030000000000001,"5min":1.204,"15min":1.2110000000000001},"min":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"max":{"1min":1.9830000000000001,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.129}]},{"osd":1,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2230000000000001,"15min":1.2509999999999999},"min":{"1min":0.746,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.5309999999999999,"5min":1.762,"15min":1.776},"last":1.212},{"interface":"front","average":{"1min":1.256,"5min":1.3580000000000001,"15min":1.359},"min":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.6459999999999999,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":0.93999999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.383,"15min":1.3779999999999999},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.8500000000000001,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.3160000000000001},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.403,"15min":1.4219999999999999},"min":{"1min":1.0600000000000001,"5min":0.99199999999999999,"15min":0.98099999999999998},"max":{"1min":1.6719999999999999,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.478}]},{"osd":3,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.381,"5min":1.3740000000000001,"15min":1.407},"min":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.7130000000000001,"5min":2.0379999999999998,"15min":2.2519999999999998},"last":1.27},{"interface":"front","average":{"1min":1.4470000000000001,"5min":1.425,"15min":1.4410000000000001},"min":{"1min":0.90300000000000002,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.4279999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.375,"15min":1.381},"min":{"1min":0.85899999999999999,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.827,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.3879999999999999},{"interface":"front","average":{"1min":1.4059999999999999,"5min":1.401,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.5379999999999998,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.554}]},{"osd":5,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.4059999999999999,"5min":1.3799999999999999,"15min":1.413},"min":{"1min":0.96199999999999997,"5min":0.624,"15min":0.624},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.4940000000000002},"last":1.024},{"interface":"front","average":{"1min":1.516,"5min":1.5469999999999999,"15min":1.579},"min":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":1.8979999999999999,"5min":2.1600000000000001,"15min":2.5609999999999999},"last":1.512}]},{"osd":7,"last update":"Sat Oct 14 18:17:13 2023","interfaces":[{"interface":"back","average":{"1min":1.5920000000000001,"5min":1.5840000000000001,"15min":1.615},"min":{"1min":0.86799999999999999,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.6389999999999998,"5min":2.903,"15min":3.423},"last":1.6459999999999999},{"interface":"front","average":{"1min":1.637,"5min":1.5569999999999999,"15min":1.605},"min":{"1min":1.1870000000000001,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.3460000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.347}]}]},{"osd":5,"up_from":13,"seq":55834576008,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42364,"kb_used_data":2856,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329476,"statfs":{"total":96636764160,"available":96593383424,"internally_reserved":0,"allocated":2924544,"data_stored":1451817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2050000000000001,"5min":1.179,"15min":1.2010000000000001},"min":{"1min":0.89700000000000002,"5min":0.71399999999999997,"15min":0.71399999999999997},"max":{"1min":1.716,"5min":1.7569999999999999,"15min":2.0459999999999998},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.2470000000000001,"5min":1.212,"15min":1.2470000000000001},"min":{"1min":0.878,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.722,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":1.02}]},{"osd":1,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.238,"15min":1.2769999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.8049999999999999,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":1.462},{"interface":"front","average":{"1min":1.3720000000000001,"5min":1.274,"15min":1.3080000000000001},"min":{"1min":0.91500000000000004,"5min":0.73599999999999999,"15min":0.63400000000000001},"max":{"1min":1.722,"5min":1.7450000000000001,"15min":2.0419999999999998},"last":1.319}]},{"osd":2,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2310000000000001,"5min":1.21,"15min":1.258},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"max":{"1min":1.649,"5min":1.764,"15min":3.7090000000000001},"last":0.94099999999999995},{"interface":"front","average":{"1min":1.393,"5min":1.3779999999999999,"15min":1.3959999999999999},"min":{"1min":1.0369999999999999,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.768,"5min":1.841,"15min":1.923},"last":1.5469999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4450000000000001,"5min":1.4239999999999999,"15min":1.4670000000000001},"min":{"1min":1.071,"5min":0.94199999999999995,"15min":0.81899999999999995},"max":{"1min":1.794,"5min":1.867,"15min":2.4969999999999999},"last":1.413},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.421,"15min":1.4450000000000001},"min":{"1min":1.008,"5min":0.92600000000000005,"15min":0.92600000000000005},"max":{"1min":1.952,"5min":1.952,"15min":3.8330000000000002},"last":1.367}]},{"osd":4,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3620000000000001,"5min":1.3700000000000001,"15min":1.363},"min":{"1min":0.98899999999999999,"5min":0.80100000000000005,"15min":0.78400000000000003},"max":{"1min":1.7889999999999999,"5min":1.954,"15min":2.98},"last":1.2529999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.3939999999999999,"15min":1.3859999999999999},"min":{"1min":0.90600000000000003,"5min":0.61799999999999999,"15min":0.57499999999999996},"max":{"1min":2.016,"5min":2.016,"15min":3.1280000000000001},"last":0.90500000000000003}]},{"osd":6,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.331,"5min":1.3859999999999999,"15min":1.417},"min":{"1min":0.97399999999999998,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":1.853,"5min":1.8799999999999999,"15min":4.0540000000000003},"last":1.49},{"interface":"front","average":{"1min":1.663,"5min":1.601,"15min":1.627},"min":{"1min":1.3859999999999999,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.4359999999999999,"5min":2.4359999999999999,"15min":3.6360000000000001},"last":1.609}]},{"osd":7,"last update":"Sat Oct 14 18:17:00 2023","interfaces":[{"interface":"back","average":{"1min":1.528,"5min":1.516,"15min":1.573},"min":{"1min":1.0409999999999999,"5min":0.68600000000000005,"15min":0.67700000000000005},"max":{"1min":2.0720000000000001,"5min":2.0720000000000001,"15min":3.2759999999999998},"last":1.526},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.5600000000000001,"15min":1.556},"min":{"1min":1.1100000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.0459999999999998,"5min":2.206,"15min":3.9390000000000001},"last":1.1879999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576008,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":64584,"kb_used_data":2876,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94307256,"statfs":{"total":96636764160,"available":96570630144,"internally_reserved":0,"allocated":2945024,"data_stored":1454778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.341,"5min":1.1930000000000001,"15min":1.21},"min":{"1min":1.0169999999999999,"5min":0.60099999999999998,"15min":0.60099999999999998},"max":{"1min":1.8899999999999999,"5min":1.8899999999999999,"15min":1.8899999999999999},"last":0.91900000000000004},{"interface":"front","average":{"1min":1.341,"5min":1.2729999999999999,"15min":1.288},"min":{"1min":0.80000000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.698,"5min":1.887,"15min":2.3140000000000001},"last":1.135}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.278,"5min":1.282,"15min":1.323},"min":{"1min":1.0009999999999999,"5min":0.76800000000000002,"15min":0.59399999999999997},"max":{"1min":1.7410000000000001,"5min":1.9930000000000001,"15min":1.9930000000000001},"last":1.6990000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.2490000000000001,"15min":1.268},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.667,"5min":1.8089999999999999,"15min":2.0059999999999998},"last":1.234}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.29,"15min":1.2849999999999999},"min":{"1min":0.83699999999999997,"5min":0.81399999999999995,"15min":0.78000000000000003},"max":{"1min":1.659,"5min":1.869,"15min":3.524},"last":1.198},{"interface":"front","average":{"1min":1.379,"5min":1.427,"15min":1.4450000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.65700000000000003},"max":{"1min":1.718,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.597}]},{"osd":3,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.2929999999999999,"5min":1.278,"15min":1.3460000000000001},"min":{"1min":0.90400000000000003,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.627,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.474},{"interface":"front","average":{"1min":1.425,"5min":1.419,"15min":1.4390000000000001},"min":{"1min":1.0509999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.7629999999999999,"5min":2.0569999999999999,"15min":4.1600000000000001},"last":1.494}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4239999999999999,"5min":1.4219999999999999,"15min":1.3959999999999999},"min":{"1min":0.96599999999999997,"5min":0.90200000000000002,"15min":0.69299999999999995},"max":{"1min":1.7949999999999999,"5min":1.9319999999999999,"15min":2.02},"last":1.29},{"interface":"front","average":{"1min":1.407,"5min":1.369,"15min":1.3759999999999999},"min":{"1min":0.98299999999999998,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":1.7589999999999999,"5min":2.2290000000000001,"15min":2.2290000000000001},"last":1.5580000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.417,"5min":1.45,"15min":1.4339999999999999},"min":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.81799999999999995},"max":{"1min":1.887,"5min":2.077,"15min":3.3610000000000002},"last":1.042},{"interface":"front","average":{"1min":1.548,"5min":1.575,"15min":1.5700000000000001},"min":{"1min":0.998,"5min":0.998,"15min":0.82699999999999996},"max":{"1min":1.823,"5min":2.1080000000000001,"15min":2.3929999999999998},"last":1.4359999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.506,"15min":1.516},"min":{"1min":1.234,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.798,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.381},{"interface":"front","average":{"1min":1.5980000000000001,"5min":1.601,"15min":1.595},"min":{"1min":1.246,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":1.9790000000000001,"5min":2.1739999999999999,"15min":3.8610000000000002},"last":1.337}]}]},{"osd":3,"up_from":13,"seq":55834576008,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52108,"kb_used_data":2880,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319732,"statfs":{"total":96636764160,"available":96583405568,"internally_reserved":0,"allocated":2949120,"data_stored":1453099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50395592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.94499999999999995,"5min":0.96899999999999997,"15min":0.98399999999999999},"min":{"1min":0.502,"5min":0.377,"15min":0.377},"max":{"1min":1.46,"5min":1.4870000000000001,"15min":1.855},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.1619999999999999,"5min":1.179,"15min":1.2050000000000001},"min":{"1min":0.61499999999999999,"5min":0.52700000000000002,"15min":0.52700000000000002},"max":{"1min":1.4890000000000001,"5min":1.7230000000000001,"15min":1.9970000000000001},"last":0.96499999999999997}]},{"osd":1,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":0.92200000000000004,"5min":1.0089999999999999,"15min":1.0169999999999999},"min":{"1min":0.57499999999999996,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.417,"5min":1.8939999999999999,"15min":1.919},"last":1.0189999999999999},{"interface":"front","average":{"1min":1.1910000000000001,"5min":1.177,"15min":1.212},"min":{"1min":0.78100000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.613,"5min":1.7150000000000001,"15min":1.8560000000000001},"last":1.3400000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4279999999999999,"5min":1.427,"15min":1.446},"min":{"1min":0.71899999999999997,"5min":0.498,"15min":0.498},"max":{"1min":1.9730000000000001,"5min":2.032,"15min":2.032},"last":1.462},{"interface":"front","average":{"1min":1.284,"5min":1.3480000000000001,"15min":1.3879999999999999},"min":{"1min":0.72499999999999998,"5min":0.57199999999999995,"15min":0.57199999999999995},"max":{"1min":1.7430000000000001,"5min":1.8,"15min":2.3540000000000001},"last":1.401}]},{"osd":4,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.399,"5min":1.4399999999999999,"15min":1.448},"min":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"max":{"1min":1.7350000000000001,"5min":2.153,"15min":2.153},"last":1.2729999999999999},{"interface":"front","average":{"1min":1.4119999999999999,"5min":1.482,"15min":1.484},"min":{"1min":0.85299999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":1.9279999999999999,"5min":2.2109999999999999,"15min":2.3199999999999998},"last":1.7729999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4550000000000001,"5min":1.464,"15min":1.464},"min":{"1min":0.96499999999999997,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":1.8009999999999999,"5min":2.1429999999999998,"15min":2.476},"last":1.2090000000000001},{"interface":"front","average":{"1min":1.46,"5min":1.518,"15min":1.5649999999999999},"min":{"1min":1.0249999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"max":{"1min":2.0739999999999998,"5min":2.246,"15min":2.891},"last":1.595}]},{"osd":6,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.5189999999999999,"5min":1.6180000000000001,"15min":1.6319999999999999},"min":{"1min":1.0580000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.0390000000000001,"5min":2.2850000000000001,"15min":2.4889999999999999},"last":1.5229999999999999},{"interface":"front","average":{"1min":1.6060000000000001,"5min":1.6859999999999999,"15min":1.6870000000000001},"min":{"1min":1.0760000000000001,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.1320000000000001,"5min":2.2949999999999999,"15min":2.585},"last":1.637}]},{"osd":7,"last update":"Sat Oct 14 18:17:07 2023","interfaces":[{"interface":"back","average":{"1min":1.4930000000000001,"5min":1.579,"15min":1.607},"min":{"1min":1.2050000000000001,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":1.859,"5min":2.2370000000000001,"15min":2.419},"last":1.546},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.704,"15min":1.696},"min":{"1min":1.0469999999999999,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.1890000000000001,"5min":2.548,"15min":2.548},"last":1.7210000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576008,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52104,"kb_used_data":2872,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319736,"statfs":{"total":96636764160,"available":96583409664,"internally_reserved":0,"allocated":2940928,"data_stored":1452903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":0.97799999999999998,"15min":0.94899999999999995},"min":{"1min":0.65000000000000002,"5min":0.499,"15min":0.499},"max":{"1min":2.1699999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.95599999999999996,"5min":1.0349999999999999,"15min":1.0960000000000001},"min":{"1min":0.61899999999999999,"5min":0.438,"15min":0.438},"max":{"1min":1.341,"5min":1.653,"15min":1.8660000000000001},"last":0.57499999999999996}]},{"osd":1,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":0.96099999999999997,"5min":0.98699999999999999,"15min":1.0169999999999999},"min":{"1min":0.61899999999999999,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.544,"5min":1.603,"15min":1.881},"last":0.60399999999999998},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.2430000000000001,"15min":1.2310000000000001},"min":{"1min":0.623,"5min":0.55500000000000005,"15min":0.55500000000000005},"max":{"1min":2.3109999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.63}]},{"osd":3,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.244,"5min":1.2509999999999999,"15min":1.2909999999999999},"min":{"1min":0.80900000000000005,"5min":0.70199999999999996,"15min":0.56599999999999995},"max":{"1min":1.621,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.68600000000000005},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.407,"15min":1.3999999999999999},"min":{"1min":0.76400000000000001,"5min":0.53300000000000003,"15min":0.53300000000000003},"max":{"1min":2.0089999999999999,"5min":2.1579999999999999,"15min":2.165},"last":0.70199999999999996}]},{"osd":4,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5209999999999999,"5min":1.492,"15min":1.478},"min":{"1min":1.077,"5min":0.72499999999999998,"15min":0.72499999999999998},"max":{"1min":2.097,"5min":2.097,"15min":2.097},"last":0.83499999999999996},{"interface":"front","average":{"1min":1.327,"5min":1.3400000000000001,"15min":1.3680000000000001},"min":{"1min":0.92300000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.728,"5min":1.8759999999999999,"15min":2.024},"last":0.73299999999999998}]},{"osd":5,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.4019999999999999,"15min":1.444},"min":{"1min":1.012,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9299999999999999,"5min":2.0489999999999999,"15min":2.2839999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":1.613,"5min":1.625,"15min":1.5980000000000001},"min":{"1min":1.3460000000000001,"5min":0.79700000000000004,"15min":0.79700000000000004},"max":{"1min":2.2389999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.86899999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5389999999999999,"5min":1.528,"15min":1.53},"min":{"1min":0.99099999999999999,"5min":0.747,"15min":0.747},"max":{"1min":1.8799999999999999,"5min":2.0529999999999999,"15min":2.1150000000000002},"last":0.80600000000000005},{"interface":"front","average":{"1min":1.5640000000000001,"5min":1.587,"15min":1.617},"min":{"1min":0.85599999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.2349999999999999,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":0.91100000000000003}]},{"osd":7,"last update":"Sat Oct 14 18:17:24 2023","interfaces":[{"interface":"back","average":{"1min":1.5629999999999999,"5min":1.611,"15min":1.5940000000000001},"min":{"1min":1.208,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":1.8500000000000001,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.631,"5min":1.6659999999999999,"15min":1.6859999999999999},"min":{"1min":1.155,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.331,"5min":2.331,"15min":2.331},"last":0.97999999999999998}]}]},{"osd":1,"up_from":13,"seq":55834576008,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52100,"kb_used_data":2864,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319740,"statfs":{"total":96636764160,"available":96583413760,"internally_reserved":0,"allocated":2932736,"data_stored":1451381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95899999999999996,"15min":0.97199999999999998},"min":{"1min":0.51100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.5169999999999999,"5min":1.647,"15min":2.4929999999999999},"last":1.0009999999999999},{"interface":"front","average":{"1min":1.109,"5min":1.0820000000000001,"15min":1.1279999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":1.079}]},{"osd":2,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.038,"15min":1.0629999999999999},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.78000000000000003},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.895,"5min":1.996,"15min":2.6859999999999999},"last":1.204}]},{"osd":3,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.357,"15min":1.385},"min":{"1min":0.52700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.6970000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2829999999999999,"15min":1.3169999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.8630000000000004,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.419}]},{"osd":4,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.88200000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":0.97299999999999998},{"interface":"front","average":{"1min":1.298,"5min":1.385,"15min":1.431},"min":{"1min":0.80400000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.899},"last":1.1439999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.4379999999999999,"15min":1.4710000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.173,"5min":2.173,"15min":3.004},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.55,"15min":1.5660000000000001},"min":{"1min":0.91800000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9359999999999999,"5min":2.093,"15min":2.5129999999999999},"last":1.361}]},{"osd":6,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5780000000000001,"15min":1.6140000000000001},"min":{"1min":0.95499999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.593},{"interface":"front","average":{"1min":1.863,"5min":1.649,"15min":1.627},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.478}]},{"osd":7,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.6679999999999999,"15min":1.6970000000000001},"min":{"1min":0.82299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":1.546},{"interface":"front","average":{"1min":1.839,"5min":1.6970000000000001,"15min":1.722},"min":{"1min":0.93300000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.7490000000000001}]}]},{"osd":0,"up_from":13,"seq":55834576007,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48516,"kb_used_data":3444,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323324,"statfs":{"total":96636764160,"available":96587083776,"internally_reserved":0,"allocated":3526656,"data_stored":2042365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":0.98699999999999999,"5min":0.95999999999999996,"15min":0.97799999999999998},"min":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.6890000000000001,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.98199999999999998},{"interface":"front","average":{"1min":1.095,"5min":1.129,"15min":1.1499999999999999},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":2.0529999999999999,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.1930000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.0660000000000001,"5min":1.175,"15min":1.1850000000000001},"min":{"1min":0.54000000000000004,"5min":0.54000000000000004,"15min":0.54000000000000004},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.23},{"interface":"front","average":{"1min":1.204,"5min":1.2010000000000001,"15min":1.25},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.845,"5min":1.9059999999999999,"15min":2.5779999999999998},"last":0.69599999999999995}]},{"osd":3,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2969999999999999,"5min":1.3759999999999999,"15min":1.391},"min":{"1min":0.501,"5min":0.501,"15min":0.501},"max":{"1min":1.7150000000000001,"5min":1.804,"15min":2.0720000000000001},"last":1.056},{"interface":"front","average":{"1min":1.452,"5min":1.391,"15min":1.4139999999999999},"min":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.819,"5min":1.883,"15min":2.194},"last":0.63600000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.286,"5min":1.321,"15min":1.331},"min":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.7629999999999999,"5min":1.9379999999999999,"15min":2.4670000000000001},"last":1.2649999999999999},{"interface":"front","average":{"1min":1.4450000000000001,"5min":1.5129999999999999,"15min":1.512},"min":{"1min":0.88900000000000001,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":2.121,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.377}]},{"osd":5,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.3580000000000001,"15min":1.3819999999999999},"min":{"1min":0.626,"5min":0.626,"15min":0.626},"max":{"1min":1.74,"5min":1.8220000000000001,"15min":2.1339999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.5409999999999999,"15min":1.581},"min":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.3839999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":0.83299999999999996}]},{"osd":6,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6950000000000001,"5min":1.655,"15min":1.661},"min":{"1min":1.107,"5min":0.88400000000000001,"15min":0.88400000000000001},"max":{"1min":2.3140000000000001,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.5269999999999999,"15min":1.5269999999999999},"min":{"1min":0.91700000000000004,"5min":0.91700000000000004,"15min":0.91700000000000004},"max":{"1min":2.1080000000000001,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.3149999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:17:15 2023","interfaces":[{"interface":"back","average":{"1min":1.6080000000000001,"5min":1.631,"15min":1.661},"min":{"1min":0.93799999999999994,"5min":0.93799999999999994,"15min":0.93799999999999994},"max":{"1min":2.0089999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.665},{"interface":"front","average":{"1min":1.639,"5min":1.613,"15min":1.641},"min":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"max":{"1min":2.1869999999999998,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.359}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2961,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":26,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":2857,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":27,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":28,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":29,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:17:47.414 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:17:47.414 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:17:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.677+0000 7f2f2a935640 1 Processor -- start 2023-10-14T18:17:47.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.677+0000 7f2f2a935640 1 -- start start 2023-10-14T18:17:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.678+0000 7f2f2a935640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2f240ad970 0x7f2f240cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.678+0000 7f2f2a935640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f240ccfb0 0x7f2f240cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.678+0000 7f2f2a935640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f2415b9d0 con 0x7f2f2406c040 2023-10-14T18:17:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.678+0000 7f2f2a935640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2415bcb0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.678+0000 7f2f2a935640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f2415bf90 con 0x7f2f240ad970 2023-10-14T18:17:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f240ccfb0 0x7f2f240cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2f240ccfb0 0x7f2f240cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53522/0 (socket says 172.21.15.29:53522) 2023-10-14T18:17:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 -- 172.21.15.29:0/1605320274 learned_addr learned my addr 172.21.15.29:0/1605320274 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:47.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 -- 172.21.15.29:0/1605320274 >> v2:172.21.15.186:3300/0 conn(0x7f2f240ad970 msgr2=0x7f2f240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 --2- 172.21.15.29:0/1605320274 >> v2:172.21.15.186:3300/0 conn(0x7f2f240ad970 0x7f2f240cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 -- 172.21.15.29:0/1605320274 >> v1:172.21.15.186:6789/0 conn(0x7f2f2406c040 legacy=0x7f2f2406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:47.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.679+0000 7f2f2a134640 1 -- 172.21.15.29:0/1605320274 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f240cc270 con 0x7f2f240ccfb0 2023-10-14T18:17:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.680+0000 7f2f2a134640 1 --2- 172.21.15.29:0/1605320274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240cd460 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009e50 tx=0x7f2f1c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95cbfd892fb70b72 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.681+0000 7f2f28931640 1 -- 172.21.15.29:0/1605320274 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c02ca80 con 0x7f2f240ccfb0 2023-10-14T18:17:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.681+0000 7f2f28931640 1 -- 172.21.15.29:0/1605320274 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c02cbe0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.681+0000 7f2f28931640 1 -- 172.21.15.29:0/1605320274 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c0343f0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.681+0000 7f2f2a935640 1 -- 172.21.15.29:0/1605320274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 msgr2=0x7f2f240cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.681+0000 7f2f2a935640 1 --2- 172.21.15.29:0/1605320274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240cd460 secure :-1 s=READY pgs=2641 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c009e50 tx=0x7f2f1c02bee0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 -- 172.21.15.29:0/1605320274 shutdown_connections 2023-10-14T18:17:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 --2- 172.21.15.29:0/1605320274 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240cd460 unknown :-1 s=CLOSED pgs=2641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 --2- 172.21.15.29:0/1605320274 >> v2:172.21.15.186:3300/0 conn(0x7f2f240ad970 0x7f2f240cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 -- 172.21.15.29:0/1605320274 >> 172.21.15.29:0/1605320274 conn(0x7f2f240ce6a0 msgr2=0x7f2f240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 -- 172.21.15.29:0/1605320274 shutdown_connections 2023-10-14T18:17:47.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.682+0000 7f2f2a935640 1 -- 172.21.15.29:0/1605320274 wait complete. 2023-10-14T18:17:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.683+0000 7f2f2a935640 1 Processor -- start 2023-10-14T18:17:47.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.684+0000 7f2f2a935640 1 -- start start 2023-10-14T18:17:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.684+0000 7f2f2a935640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f2406c040 0x7f2f240ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.684+0000 7f2f2a935640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f240ad970 0x7f2f241f1c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.684+0000 7f2f2a935640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a935640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f241f21a0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a935640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240b3440 con 0x7f2f240ad970 2023-10-14T18:17:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a935640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f240b0fe0 con 0x7f2f2406c040 2023-10-14T18:17:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53538/0 (socket says 172.21.15.29:53538) 2023-10-14T18:17:47.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 -- 172.21.15.29:0/2756326425 learned_addr learned my addr 172.21.15.29:0/2756326425 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 -- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f2406c040 msgr2=0x7f2f240ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f29933640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f2406c040 0x7f2f240ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f29132640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f240ad970 0x7f2f241f1c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f2406c040 0x7f2f240ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 -- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f240ad970 msgr2=0x7f2f241f1c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f240ad970 0x7f2f241f1c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.685+0000 7f2f2a134640 1 -- 172.21.15.29:0/2756326425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c009b30 con 0x7f2f240ccfb0 2023-10-14T18:17:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f2a134640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c036040 tx=0x7f2f1c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c040670 con 0x7f2f240ccfb0 2023-10-14T18:17:47.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f1c0407f0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f1c040ba0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f240b1430 con 0x7f2f240ccfb0 2023-10-14T18:17:47.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.686+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f240b30d0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.687+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f2406b330 con 0x7f2f240ccfb0 2023-10-14T18:17:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.688+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2f1c02cd50 con 0x7f2f240ccfb0 2023-10-14T18:17:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.689+0000 7f2f1a7fc640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 0x7f2f0806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:47.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.689+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f2f1c0acad0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.689+0000 7f2f29933640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 0x7f2f0806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:47.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.690+0000 7f2f29933640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 0x7f2f0806d480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f20003d90 tx=0x7f2f20009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:47.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.694+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2f1c0775f0 con 0x7f2f240ccfb0 2023-10-14T18:17:47.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.929+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f240b1950 con 0x7f2f240ccfb0 2023-10-14T18:17:47.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.930+0000 7f2f1a7fc640 1 -- 172.21.15.29:0/2756326425 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7f2f1c07ac10 con 0x7f2f240ccfb0 2023-10-14T18:17:47.932 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:47.932 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:17:47.932 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:17:47.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 msgr2=0x7f2f0806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 0x7f2f0806d480 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f20003d90 tx=0x7f2f20009040 comp rx=0 tx=0).stop 2023-10-14T18:17:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 msgr2=0x7f2f240b0aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 secure :-1 s=READY pgs=2642 cs=0 l=1 rev1=1 crypto rx=0x7f2f1c036040 tx=0x7f2f1c004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 shutdown_connections 2023-10-14T18:17:47.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f2f0806afc0 0x7f2f0806d480 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2f240ccfb0 0x7f2f240b0aa0 unknown :-1 s=CLOSED pgs=2642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2f240ad970 0x7f2f241f1c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 --2- 172.21.15.29:0/2756326425 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2f2406c040 0x7f2f240ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.933+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 >> 172.21.15.29:0/2756326425 conn(0x7f2f240ce6a0 msgr2=0x7f2f2406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:47.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.934+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 shutdown_connections 2023-10-14T18:17:47.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:47.934+0000 7f2f2a935640 1 -- 172.21.15.29:0/2756326425 wait complete. 2023-10-14T18:17:48.357 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 113, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:17:41.791634+0000', 'modified': '2023-10-14T18:17:44.857465+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 11256}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_11256': {'gid': 11256, 'name': 'c', 'rank': 0, 'incarnation': 109, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.29:6835/2605417960', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.29:6834', 'nonce': 2605417960}, {'type': 'v1', 'addr': '172.21.15.29:6835', 'nonce': 2605417960}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [29], 'metadata_pool': 28, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:17:48.358 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:17:48.358 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2023-10-14T18:17:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.599+0000 7f94deb06640 1 Processor -- start 2023-10-14T18:17:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.599+0000 7f94deb06640 1 -- start start 2023-10-14T18:17:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94deb06640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94d80ad970 0x7f94d80cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94deb06640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d815b9d0 con 0x7f94d806c040 2023-10-14T18:17:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94deb06640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94d815bcb0 con 0x7f94d80ccfb0 2023-10-14T18:17:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94deb06640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94d815bf90 con 0x7f94d80ad970 2023-10-14T18:17:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94ddb04640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f94d806c040 0x7f94d806c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33960/0 (socket says 172.21.15.29:33960) 2023-10-14T18:17:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94ddb04640 1 -- 172.21.15.29:0/1698896286 learned_addr learned my addr 172.21.15.29:0/1698896286 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94dd303640 1 --2- 172.21.15.29:0/1698896286 >> v2:172.21.15.186:3300/0 conn(0x7f94d80ad970 0x7f94d80cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.601+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974519215 0 0) 0x7f94d815b9d0 con 0x7f94d806c040 2023-10-14T18:17:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94c4003520 con 0x7f94d806c040 2023-10-14T18:17:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2800324819 0 0) 0x7f94c4003520 con 0x7f94d806c040 2023-10-14T18:17:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 >> v2:172.21.15.186:3300/0 conn(0x7f94d80ad970 msgr2=0x7f94d80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 --2- 172.21.15.29:0/1698896286 >> v2:172.21.15.186:3300/0 conn(0x7f94d80ad970 0x7f94d80cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 >> v1:172.21.15.186:6789/0 conn(0x7f94d80ccfb0 legacy=0x7f94d80cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d80cc270 con 0x7f94d806c040 2023-10-14T18:17:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f94d4001670 con 0x7f94d806c040 2023-10-14T18:17:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 >> v1:172.21.15.29:6789/0 conn(0x7f94d806c040 legacy=0x7f94d806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94c40039c0 0x7f94c4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94c4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94c4005180 0x7f94c400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d815bf90 con 0x7f94c40047c0 2023-10-14T18:17:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c400c080 con 0x7f94c40039c0 2023-10-14T18:17:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c400c240 con 0x7f94c4005180 2023-10-14T18:17:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.602+0000 7f94dd303640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94c4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94ddb04640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94c40039c0 0x7f94c4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94dd303640 1 -- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94c4005180 msgr2=0x7f94c400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94dd303640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94c4005180 0x7f94c400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94dd303640 1 -- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94c40039c0 msgr2=0x7f94c4003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94dd303640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94c40039c0 0x7f94c4003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94dd303640 1 -- 172.21.15.29:0/1698896286 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4003520 con 0x7f94c40047c0 2023-10-14T18:17:48.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.603+0000 7f94de305640 1 -- 172.21.15.29:0/1698896286 reap_dead start 2023-10-14T18:17:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.604+0000 7f94dd303640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94c4004c40 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009a90 tx=0x7f94c40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94fed09c766ac71d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.604+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c802c6b0 con 0x7f94c40047c0 2023-10-14T18:17:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.604+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c802c810 con 0x7f94c40047c0 2023-10-14T18:17:48.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.604+0000 7f94dcb02640 1 -- 172.21.15.29:0/1698896286 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c802cbc0 con 0x7f94c40047c0 2023-10-14T18:17:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 -- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 msgr2=0x7f94c4004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94c4004c40 secure :-1 s=READY pgs=2644 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009a90 tx=0x7f94c40057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 -- 172.21.15.29:0/1698896286 shutdown_connections 2023-10-14T18:17:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 --2- 172.21.15.29:0/1698896286 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94c4004c40 unknown :-1 s=CLOSED pgs=2644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 -- 172.21.15.29:0/1698896286 >> 172.21.15.29:0/1698896286 conn(0x7f94d80ce6a0 msgr2=0x7f94d80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 -- 172.21.15.29:0/1698896286 shutdown_connections 2023-10-14T18:17:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.605+0000 7f94deb06640 1 -- 172.21.15.29:0/1698896286 wait complete. 2023-10-14T18:17:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.607+0000 7f94deb06640 1 Processor -- start 2023-10-14T18:17:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.607+0000 7f94deb06640 1 -- start start 2023-10-14T18:17:48.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d80b08d0 0x7f94d80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d80acdd0 0x7f94d814bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d80ad250 con 0x7f94c40047c0 2023-10-14T18:17:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d80b3b60 con 0x7f94d80b08d0 2023-10-14T18:17:48.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.608+0000 7f94deb06640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94d81499b0 con 0x7f94d80acdd0 2023-10-14T18:17:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53550/0 (socket says 172.21.15.29:53550) 2023-10-14T18:17:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 -- 172.21.15.29:0/3185304992 learned_addr learned my addr 172.21.15.29:0/3185304992 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94dd303640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d80b08d0 0x7f94d80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94de305640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d80acdd0 0x7f94d814bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 -- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d80acdd0 msgr2=0x7f94d814bce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d80acdd0 0x7f94d814bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 -- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d80b08d0 msgr2=0x7f94d80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d80b08d0 0x7f94d80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.609+0000 7f94ddb04640 1 -- 172.21.15.29:0/3185304992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d814c8c0 con 0x7f94c40047c0 2023-10-14T18:17:48.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94ddb04640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e7b0 tx=0x7f94d400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94ce7fc640 1 -- 172.21.15.29:0/3185304992 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400ee60 con 0x7f94c40047c0 2023-10-14T18:17:48.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94ce7fc640 1 -- 172.21.15.29:0/3185304992 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d400c870 con 0x7f94c40047c0 2023-10-14T18:17:48.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d814cba0 con 0x7f94c40047c0 2023-10-14T18:17:48.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94ce7fc640 1 -- 172.21.15.29:0/3185304992 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d400cc20 con 0x7f94c40047c0 2023-10-14T18:17:48.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.610+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d80b00d0 con 0x7f94c40047c0 2023-10-14T18:17:48.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.613+0000 7f94ce7fc640 1 -- 172.21.15.29:0/3185304992 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f94d401d460 con 0x7f94c40047c0 2023-10-14T18:17:48.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.614+0000 7f94ce7fc640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 0x7f94bc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.614+0000 7f94ce7fc640 1 -- 172.21.15.29:0/3185304992 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f94d4089be0 con 0x7f94c40047c0 2023-10-14T18:17:48.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.614+0000 7f94dd303640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 0x7f94bc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.615+0000 7f94e29d6740 1 Processor -- start 2023-10-14T18:17:48.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.615+0000 7f94e29d6740 1 -- start start 2023-10-14T18:17:48.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.615+0000 7f94dd303640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 0x7f94bc06d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94c8036000 tx=0x7f94c8030000 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f110070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 0x55b59f13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f0307a0 con 0x55b59f1105b0 2023-10-14T18:17:48.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f082740 con 0x55b59f13d550 2023-10-14T18:17:48.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b59f07fe30 con 0x55b59f13d170 2023-10-14T18:17:48.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94de305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94de305640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53562/0 (socket says 172.21.15.29:53562) 2023-10-14T18:17:48.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.616+0000 7f94de305640 1 -- 172.21.15.29:0/387356291 learned_addr learned my addr 172.21.15.29:0/387356291 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94dd303640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 0x55b59f13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94ddb04640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f110070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 -- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 msgr2=0x55b59f110070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f110070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 -- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 msgr2=0x55b59f13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 0x55b59f13ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 -- 172.21.15.29:0/387356291 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b59f03f910 con 0x55b59f1105b0 2023-10-14T18:17:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94dd303640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 0x55b59f13ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94de305640 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f94d80b1260 tx=0x7f94d0001e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46fc440f9bc0ea64 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94bb7fe640 1 -- 172.21.15.29:0/387356291 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d000b070 con 0x55b59f1105b0 2023-10-14T18:17:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94bb7fe640 1 -- 172.21.15.29:0/387356291 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d0002070 con 0x55b59f1105b0 2023-10-14T18:17:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.617+0000 7f94bb7fe640 1 -- 172.21.15.29:0/387356291 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d0003bb0 con 0x55b59f1105b0 2023-10-14T18:17:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 -- 172.21.15.29:0/387356291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 msgr2=0x55b59f140440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 secure :-1 s=READY pgs=2646 cs=0 l=1 rev1=1 crypto rx=0x7f94d80b1260 tx=0x7f94d0001e20 comp rx=0 tx=0).stop 2023-10-14T18:17:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 -- 172.21.15.29:0/387356291 shutdown_connections 2023-10-14T18:17:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f1105b0 0x55b59f140440 unknown :-1 s=CLOSED pgs=2646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d550 0x55b59f13ff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 --2- 172.21.15.29:0/387356291 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 -- 172.21.15.29:0/387356291 >> 172.21.15.29:0/387356291 conn(0x55b59f112380 msgr2=0x55b59f15af00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 -- 172.21.15.29:0/387356291 shutdown_connections 2023-10-14T18:17:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.618+0000 7f94e29d6740 1 -- 172.21.15.29:0/387356291 wait complete. 2023-10-14T18:17:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.619+0000 7f94e29d6740 1 -- start start 2023-10-14T18:17:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.619+0000 7f94e29d6740 1 Processor -- start 2023-10-14T18:17:48.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d550 0x55b59f151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f15e380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f07b720 con 0x55b59f13d170 2023-10-14T18:17:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f019970 con 0x55b59f1524f0 2023-10-14T18:17:48.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b59f019610 con 0x55b59f13d550 2023-10-14T18:17:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94e29d6740 10 client.0 _ll_register_callbacks cb 0x55b59f03b9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:17:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.620+0000 7f94ddb04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53568/0 (socket says 172.21.15.29:53568) 2023-10-14T18:17:48.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 -- 172.21.15.29:0/1189958071 learned_addr learned my addr 172.21.15.29:0/1189958071 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94dd303640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d550 0x55b59f151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 -- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d550 msgr2=0x55b59f151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d550 0x55b59f151fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94de305640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f15e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 -- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 msgr2=0x55b59f15e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f15e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 -- 172.21.15.29:0/1189958071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d815bcb0 con 0x55b59f13d170 2023-10-14T18:17:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94ddb04640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f94d400c4b0 tx=0x7f94d4058480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2db8ffc3e25becdb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.621+0000 7f94b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:17:48.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94d4054ad0 con 0x55b59f13d170 2023-10-14T18:17:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f94d4054c30 con 0x55b59f13d170 2023-10-14T18:17:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b59f0a2160 con 0x55b59f13d170 2023-10-14T18:17:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94d405c0b0 con 0x55b59f13d170 2023-10-14T18:17:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 10 client.11538 fetch_fsmap learned FSMap version 113 2023-10-14T18:17:48.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x55b59f079da0 con 0x55b59f13d170 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55b59f0a2160 con 0x55b59f13d170 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 113) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f94d4060230 con 0x55b59f13d170 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 10 client.11538 fetch_fsmap finished waiting for FSMap version 113 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 10 client.11538 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 20 client.11538 populate_metadata read hostname 'smithi029' 2023-10-14T18:17:48.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f16d860 0x55b59f141f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 4 client.11538 mds_command: new command op to 11256 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:17:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.622+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 --> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b59f04dcc0 con 0x55b59f16d860 2023-10-14T18:17:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.623+0000 7f94dd303640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f16d860 0x55b59f141f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.623+0000 7f94dd303640 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f16d860 0x55b59f141f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce898f36980fad99 server_cookie=290c1c97902277b2 in_seq=0 out_seq=0 2023-10-14T18:17:48.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.623+0000 7f94b8ff9640 10 client.11538 ms_handle_connect on v2:172.21.15.29:6834/2605417960 2023-10-14T18:17:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94b8ff9640 1 -- 172.21.15.29:0/1189958071 <== mds.0 v2:172.21.15.29:6834/2605417960 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55b59f04dcc0 con 0x55b59f16d860 2023-10-14T18:17:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94b8ff9640 10 client.11538 handle_command_reply: tid=0 2023-10-14T18:17:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 client.11538 shutdown 2023-10-14T18:17:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 msgr2=0x55b59f131330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 secure :-1 s=READY pgs=2647 cs=0 l=1 rev1=1 crypto rx=0x7f94d400c4b0 tx=0x7f94d4058480 comp rx=0 tx=0).stop 2023-10-14T18:17:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 shutdown_connections 2023-10-14T18:17:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f15e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f16d860 0x55b59f141f40 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d550 0x55b59f151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.624+0000 7f94e29d6740 1 --2- 172.21.15.29:0/1189958071 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d170 0x55b59f131330 unknown :-1 s=CLOSED pgs=2647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.625+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 >> 172.21.15.29:0/1189958071 conn(0x55b59f1105b0 msgr2=0x55b59f13fef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.625+0000 7f94b8ff9640 0 client.11538 ms_handle_reset on v2:172.21.15.29:6834/2605417960 2023-10-14T18:17:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.625+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 shutdown_connections 2023-10-14T18:17:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.625+0000 7f94e29d6740 1 -- 172.21.15.29:0/1189958071 wait complete. 2023-10-14T18:17:48.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.625+0000 7f94e29d6740 20 client.11538 trim_cache size 0 max 16384 2023-10-14T18:17:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.641+0000 7f94e29d6740 1 Processor -- start 2023-10-14T18:17:48.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 -- start start 2023-10-14T18:17:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d170 0x55b59f140050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f1524f0 0x55b59f10f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59eddff20 con 0x55b59f13d550 2023-10-14T18:17:48.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f144430 con 0x55b59f13d170 2023-10-14T18:17:48.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.642+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b59f16d6c0 con 0x55b59f1524f0 2023-10-14T18:17:48.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53574/0 (socket says 172.21.15.29:53574) 2023-10-14T18:17:48.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 -- 172.21.15.29:0/3991116871 learned_addr learned my addr 172.21.15.29:0/3991116871 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94de305640 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f1524f0 0x55b59f10f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94ddb04640 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d170 0x55b59f140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 -- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f1524f0 msgr2=0x55b59f10f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f1524f0 0x55b59f10f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 -- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d170 msgr2=0x55b59f140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d170 0x55b59f140050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.643+0000 7f94dd303640 1 -- 172.21.15.29:0/3991116871 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b59f018b80 con 0x55b59f13d550 2023-10-14T18:17:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94dd303640 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f94c800f210 tx=0x7f94c8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a0eb844708eaa75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94bbfff640 1 -- 172.21.15.29:0/3991116871 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c804d110 con 0x55b59f13d550 2023-10-14T18:17:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94bbfff640 1 -- 172.21.15.29:0/3991116871 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c804d270 con 0x55b59f13d550 2023-10-14T18:17:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94bbfff640 1 -- 172.21.15.29:0/3991116871 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8034440 con 0x55b59f13d550 2023-10-14T18:17:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94e29d6740 1 -- 172.21.15.29:0/3991116871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 msgr2=0x55b59f16ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94e29d6740 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 secure :-1 s=READY pgs=2648 cs=0 l=1 rev1=1 crypto rx=0x7f94c800f210 tx=0x7f94c8008030 comp rx=0 tx=0).stop 2023-10-14T18:17:48.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.644+0000 7f94e29d6740 1 -- 172.21.15.29:0/3991116871 shutdown_connections 2023-10-14T18:17:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f1524f0 0x55b59f10f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f16ca80 unknown :-1 s=CLOSED pgs=2648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 --2- 172.21.15.29:0/3991116871 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f13d170 0x55b59f140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 -- 172.21.15.29:0/3991116871 >> 172.21.15.29:0/3991116871 conn(0x55b59f1105b0 msgr2=0x55b59f10b810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 -- 172.21.15.29:0/3991116871 shutdown_connections 2023-10-14T18:17:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 -- 172.21.15.29:0/3991116871 wait complete. 2023-10-14T18:17:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 -- start start 2023-10-14T18:17:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.645+0000 7f94e29d6740 1 Processor -- start 2023-10-14T18:17:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f15a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f114e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f115700 con 0x55b59f13d550 2023-10-14T18:17:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b59f13fce0 con 0x55b59f1524f0 2023-10-14T18:17:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55b59f1561d0 con 0x55b59f13d170 2023-10-14T18:17:48.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94e29d6740 10 client.0 _ll_register_callbacks cb 0x55b59f110500 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:17:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53582/0 (socket says 172.21.15.29:53582) 2023-10-14T18:17:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 -- 172.21.15.29:0/4071431512 learned_addr learned my addr 172.21.15.29:0/4071431512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94ddb04640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f15a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 -- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 msgr2=0x55b59f15a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f15a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 -- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 msgr2=0x55b59f114e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f114e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.647+0000 7f94dd303640 1 -- 172.21.15.29:0/4071431512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c4003520 con 0x55b59f13d550 2023-10-14T18:17:48.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94dd303640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f94c800f110 tx=0x7f94c80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7457e5361bb4a331 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94b8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:17:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8034440 con 0x55b59f13d550 2023-10-14T18:17:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f94c80345a0 con 0x55b59f13d550 2023-10-14T18:17:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f94d0002070 con 0x55b59f13d550 2023-10-14T18:17:48.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=113) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94c8034760 con 0x55b59f13d550 2023-10-14T18:17:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94e29d6740 10 client.11544 fetch_fsmap learned FSMap version 113 2023-10-14T18:17:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=113,monmap=2+}) v3 -- 0x55b59f03e170 con 0x55b59f13d550 2023-10-14T18:17:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.648+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94d0002070 con 0x55b59f13d550 2023-10-14T18:17:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 113) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f94c80348c0 con 0x55b59f13d550 2023-10-14T18:17:48.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 10 client.11544 fetch_fsmap finished waiting for FSMap version 113 2023-10-14T18:17:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 10 client.11544 resolve_mds: resolved name 'c' to daemon mds.c 2023-10-14T18:17:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 20 client.11544 populate_metadata read hostname 'smithi029' 2023-10-14T18:17:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f155500 0x55b59f1558e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 4 client.11544 mds_command: new command op to 11256 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:17:48.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 --> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55b59f074700 con 0x55b59f155500 2023-10-14T18:17:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.649+0000 7f94ddb04640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f155500 0x55b59f1558e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94ddb04640 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f155500 0x55b59f1558e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6088607db0f4c2a server_cookie=6e822f728feb0832 in_seq=0 out_seq=0 2023-10-14T18:17:48.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94b8ff9640 10 client.11544 ms_handle_connect on v2:172.21.15.29:6834/2605417960 2023-10-14T18:17:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94b8ff9640 1 -- 172.21.15.29:0/4071431512 <== mds.0 v2:172.21.15.29:6834/2605417960 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+338 (crc 0 0 0) 0x55b59f074700 con 0x55b59f155500 2023-10-14T18:17:48.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94b8ff9640 10 client.11544 handle_command_reply: tid=0 2023-10-14T18:17:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 client.11544 shutdown 2023-10-14T18:17:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 msgr2=0x55b59f114900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 secure :-1 s=READY pgs=2649 cs=0 l=1 rev1=1 crypto rx=0x7f94c800f110 tx=0x7f94c80046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 shutdown_connections 2023-10-14T18:17:48.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55b59f1524f0 0x55b59f114e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55b59f13d550 0x55b59f114900 unknown :-1 s=CLOSED pgs=2649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.29:6834/2605417960,v1:172.21.15.29:6835/2605417960] conn(0x55b59f155500 0x55b59f1558e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 --2- 172.21.15.29:0/4071431512 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55b59f13d170 0x55b59f15a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 >> 172.21.15.29:0/4071431512 conn(0x55b59f1105b0 msgr2=0x55b59f10f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94b8ff9640 0 client.11544 ms_handle_reset on v2:172.21.15.29:6834/2605417960 2023-10-14T18:17:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 shutdown_connections 2023-10-14T18:17:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.650+0000 7f94e29d6740 1 -- 172.21.15.29:0/4071431512 wait complete. 2023-10-14T18:17:48.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94e29d6740 20 client.11544 trim_cache size 0 max 16384 2023-10-14T18:17:48.668 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:17:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:17:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:17:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 11256, 2023-10-14T18:17:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:17:48.669 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:17:48.670 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "backup_fs", 2023-10-14T18:17:48.670 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.3409439819999998, 2023-10-14T18:17:48.670 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 113, 2023-10-14T18:17:48.670 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 233, 2023-10-14T18:17:48.671 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 233, 2023-10-14T18:17:48.671 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 27.239984662000001 2023-10-14T18:17:48.671 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:17:48.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 msgr2=0x7f94bc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 0x7f94bc06d450 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f94c8036000 tx=0x7f94c8030000 comp rx=0 tx=0).stop 2023-10-14T18:17:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 msgr2=0x7f94d8149470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 secure :-1 s=READY pgs=2645 cs=0 l=1 rev1=1 crypto rx=0x7f94d400e7b0 tx=0x7f94d400ec70 comp rx=0 tx=0).stop 2023-10-14T18:17:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 shutdown_connections 2023-10-14T18:17:48.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f94bc06af90 0x7f94bc06d450 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94d80acdd0 0x7f94d814bce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94d80b08d0 0x7f94d80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 --2- 172.21.15.29:0/3185304992 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94c40047c0 0x7f94d8149470 unknown :-1 s=CLOSED pgs=2645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 >> 172.21.15.29:0/3185304992 conn(0x7f94d80ce6a0 msgr2=0x7f94d80c7920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 shutdown_connections 2023-10-14T18:17:48.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.651+0000 7f94deb06640 1 -- 172.21.15.29:0/3185304992 wait complete. 2023-10-14T18:17:48.675 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T18:17:48.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.900+0000 7fc317fff640 1 Processor -- start 2023-10-14T18:17:48.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.901+0000 7fc317fff640 1 -- start start 2023-10-14T18:17:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc317fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc31806fb50 0x7fc3180cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc317fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc31806c9e0 0x7fc3180ccde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc317fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3180ca560 con 0x7fc3180c9060 2023-10-14T18:17:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc317fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180c9950 con 0x7fc31806c9e0 2023-10-14T18:17:48.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc317fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3180c9c30 con 0x7fc31806fb50 2023-10-14T18:17:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc3177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc31806c9e0 0x7fc3180ccde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc3177fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc31806c9e0 0x7fc3180ccde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53596/0 (socket says 172.21.15.29:53596) 2023-10-14T18:17:48.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc3177fe640 1 -- 172.21.15.29:0/1548311845 learned_addr learned my addr 172.21.15.29:0/1548311845 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.903+0000 7fc30ffff640 1 --2- 172.21.15.29:0/1548311845 >> v2:172.21.15.186:3300/0 conn(0x7fc31806fb50 0x7fc3180cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.904+0000 7fc3177fe640 1 -- 172.21.15.29:0/1548311845 >> v2:172.21.15.186:3300/0 conn(0x7fc31806fb50 msgr2=0x7fc3180cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.904+0000 7fc3177fe640 1 --2- 172.21.15.29:0/1548311845 >> v2:172.21.15.186:3300/0 conn(0x7fc31806fb50 0x7fc3180cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.904+0000 7fc3177fe640 1 -- 172.21.15.29:0/1548311845 >> v1:172.21.15.186:6789/0 conn(0x7fc3180c9060 legacy=0x7fc31806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.904+0000 7fc3177fe640 1 -- 172.21.15.29:0/1548311845 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3180c9f10 con 0x7fc31806c9e0 2023-10-14T18:17:48.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.905+0000 7fc3177fe640 1 --2- 172.21.15.29:0/1548311845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806c9e0 0x7fc3180ccde0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fc318164cb0 tx=0x7fc31002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3006855d3849d9a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.905+0000 7fc3167fc640 1 -- 172.21.15.29:0/1548311845 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc31002cb10 con 0x7fc31806c9e0 2023-10-14T18:17:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.905+0000 7fc3167fc640 1 -- 172.21.15.29:0/1548311845 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc31002cc70 con 0x7fc31806c9e0 2023-10-14T18:17:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.905+0000 7fc3167fc640 1 -- 172.21.15.29:0/1548311845 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3100343f0 con 0x7fc31806c9e0 2023-10-14T18:17:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 -- 172.21.15.29:0/1548311845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806c9e0 msgr2=0x7fc3180ccde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 --2- 172.21.15.29:0/1548311845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806c9e0 0x7fc3180ccde0 secure :-1 s=READY pgs=2650 cs=0 l=1 rev1=1 crypto rx=0x7fc318164cb0 tx=0x7fc31002bf70 comp rx=0 tx=0).stop 2023-10-14T18:17:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 -- 172.21.15.29:0/1548311845 shutdown_connections 2023-10-14T18:17:48.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 --2- 172.21.15.29:0/1548311845 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806c9e0 0x7fc3180ccde0 unknown :-1 s=CLOSED pgs=2650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 --2- 172.21.15.29:0/1548311845 >> v2:172.21.15.186:3300/0 conn(0x7fc31806fb50 0x7fc3180cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 -- 172.21.15.29:0/1548311845 >> 172.21.15.29:0/1548311845 conn(0x7fc3180ce6a0 msgr2=0x7fc3180ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 -- 172.21.15.29:0/1548311845 shutdown_connections 2023-10-14T18:17:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.906+0000 7fc317fff640 1 -- 172.21.15.29:0/1548311845 wait complete. 2023-10-14T18:17:48.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.908+0000 7fc317fff640 1 Processor -- start 2023-10-14T18:17:48.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.908+0000 7fc317fff640 1 -- start start 2023-10-14T18:17:48.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 0x7fc3180c4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3180c9060 0x7fc3181f6610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180c3b20 con 0x7fc31806fb50 2023-10-14T18:17:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180c1140 con 0x7fc31806c9e0 2023-10-14T18:17:48.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.909+0000 7fc317fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3180c2f10 con 0x7fc3180c9060 2023-10-14T18:17:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53602/0 (socket says 172.21.15.29:53602) 2023-10-14T18:17:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 -- 172.21.15.29:0/4221908651 learned_addr learned my addr 172.21.15.29:0/4221908651 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:48.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc316ffd640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 0x7fc3180c4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 -- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3180c9060 msgr2=0x7fc3181f6610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3180c9060 0x7fc3181f6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 -- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 msgr2=0x7fc3180c4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 0x7fc3180c4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:48.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 -- 172.21.15.29:0/4221908651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc310009b30 con 0x7fc31806fb50 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc316ffd640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 0x7fc3180c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.910+0000 7fc30ffff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fc30000d800 tx=0x7fc30000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.911+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30000b6c0 con 0x7fc31806fb50 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.911+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30000b820 con 0x7fc31806fb50 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.911+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3180c31f0 con 0x7fc31806fb50 2023-10-14T18:17:48.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.911+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc30000bbb0 con 0x7fc31806fb50 2023-10-14T18:17:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.911+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3180c37b0 con 0x7fc31806fb50 2023-10-14T18:17:48.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.912+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc318161510 con 0x7fc31806fb50 2023-10-14T18:17:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.913+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc30000bd10 con 0x7fc31806fb50 2023-10-14T18:17:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.914+0000 7fc314ff9640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 0x7fc2f806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.914+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fc30008a0b0 con 0x7fc31806fb50 2023-10-14T18:17:48.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.914+0000 7fc316ffd640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 0x7fc2f806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:48.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.915+0000 7fc316ffd640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 0x7fc2f806d480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc308000f60 tx=0x7fc308009040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:48.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:48.918+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc318161510 con 0x7fc31806fb50 2023-10-14T18:17:49.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.138+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3180c1f10 con 0x7fc2f806afc0 2023-10-14T18:17:49.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.141+0000 7fc314ff9640 1 -- 172.21.15.29:0/4221908651 <== mgr.11010 v2:172.21.15.29:6832/2967217610 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7fc308001c90 con 0x7fc2f806afc0 2023-10-14T18:17:49.142 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:17:49.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:49.143 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T18:17:49.143 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:17:49.143 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:17:49.144 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "backup_fs" 2023-10-14T18:17:49.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:17:49.144 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:17:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 msgr2=0x7fc2f806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 0x7fc2f806d480 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc308000f60 tx=0x7fc308009040 comp rx=0 tx=0).stop 2023-10-14T18:17:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 msgr2=0x7fc3180c0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 secure :-1 s=READY pgs=2651 cs=0 l=1 rev1=1 crypto rx=0x7fc30000d800 tx=0x7fc30000dcc0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 shutdown_connections 2023-10-14T18:17:49.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fc2f806afc0 0x7fc2f806d480 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3180c9060 0x7fc3181f6610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc31806fb50 0x7fc3180c0c00 unknown :-1 s=CLOSED pgs=2651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 --2- 172.21.15.29:0/4221908651 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc31806c9e0 0x7fc3180c4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.144+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 >> 172.21.15.29:0/4221908651 conn(0x7fc3180ce6a0 msgr2=0x7fc3180cd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:49.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.145+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 shutdown_connections 2023-10-14T18:17:49.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.145+0000 7fc317fff640 1 -- 172.21.15.29:0/4221908651 wait complete. 2023-10-14T18:17:49.166 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T18:17:49.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.386+0000 7febf5adf640 1 Processor -- start 2023-10-14T18:17:49.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.387+0000 7febf5adf640 1 -- start start 2023-10-14T18:17:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf5adf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febf00b86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf5adf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf00d2890 con 0x7febf00d2420 2023-10-14T18:17:49.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf5adf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf00d1380 con 0x7febf015d1b0 2023-10-14T18:17:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf5adf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf00d1660 con 0x7febf015cdd0 2023-10-14T18:17:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf52de640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7febf00d2420 0x7febf00cd810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33966/0 (socket says 172.21.15.29:33966) 2023-10-14T18:17:49.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.388+0000 7febf52de640 1 -- 172.21.15.29:0/2046822450 learned_addr learned my addr 172.21.15.29:0/2046822450 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:49.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febf4add640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febf00b86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961369113 0 0) 0x7febe8001110 con 0x7febf00d2420 2023-10-14T18:17:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febdc003520 con 0x7febf00d2420 2023-10-14T18:17:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1790899651 0 0) 0x7febdc003520 con 0x7febf00d2420 2023-10-14T18:17:49.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 msgr2=0x7febf00b86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febf00b86b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 >> v1:172.21.15.186:6789/0 conn(0x7febf015d1b0 legacy=0x7febf00d0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf00d1940 con 0x7febf00d2420 2023-10-14T18:17:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7febe8001800 con 0x7febf00d2420 2023-10-14T18:17:49.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 >> v1:172.21.15.29:6789/0 conn(0x7febf00d2420 legacy=0x7febf00cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febdc0039a0 0x7febdc003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.389+0000 7febef7fe640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febdc0047c0 0x7febdc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febef7fe640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febdc0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf00d1660 con 0x7febdc0047c0 2023-10-14T18:17:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febdc00bb40 con 0x7febdc0039a0 2023-10-14T18:17:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febdc00bd00 con 0x7febf015cdd0 2023-10-14T18:17:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febdc0047c0 0x7febdc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febf4add640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febdc0039a0 0x7febdc003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febf52de640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febdc0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 -- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 msgr2=0x7febdc0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febf015cdd0 0x7febdc0051c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 -- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febdc0039a0 msgr2=0x7febdc003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febdc0039a0 0x7febdc003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.390+0000 7febeffff640 1 -- 172.21.15.29:0/2046822450 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febdc003520 con 0x7febdc0047c0 2023-10-14T18:17:49.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.391+0000 7febeffff640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febdc0047c0 0x7febdc004c40 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7febf00d12b0 tx=0x7febdc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95574b6dd6d77c87 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd80052d0 con 0x7febdc0047c0 2023-10-14T18:17:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd802e050 con 0x7febdc0047c0 2023-10-14T18:17:49.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febef7fe640 1 -- 172.21.15.29:0/2046822450 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febd80035f0 con 0x7febdc0047c0 2023-10-14T18:17:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febf5adf640 1 -- 172.21.15.29:0/2046822450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febdc0047c0 msgr2=0x7febdc004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febf5adf640 1 --2- 172.21.15.29:0/2046822450 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febdc0047c0 0x7febdc004c40 secure :-1 s=READY pgs=2653 cs=0 l=1 rev1=1 crypto rx=0x7febf00d12b0 tx=0x7febdc0057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febf52de640 1 -- 172.21.15.29:0/2046822450 reap_dead start 2023-10-14T18:17:49.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febf5adf640 1 -- 172.21.15.29:0/2046822450 shutdown_connections 2023-10-14T18:17:49.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.392+0000 7febf5adf640 1 -- 172.21.15.29:0/2046822450 >> 172.21.15.29:0/2046822450 conn(0x7febf00cf550 msgr2=0x7febf00cf960 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:49.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.393+0000 7febf5adf640 1 -- 172.21.15.29:0/2046822450 shutdown_connections 2023-10-14T18:17:49.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.393+0000 7febf5adf640 1 -- 172.21.15.29:0/2046822450 wait complete. 2023-10-14T18:17:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.394+0000 7febf5adf640 1 Processor -- start 2023-10-14T18:17:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 -- start start 2023-10-14T18:17:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febdc0047c0 0x7febf014e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febf0158090 0x7febf0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf0157330 con 0x7febf0158510 2023-10-14T18:17:49.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf01f6d10 con 0x7febf0158090 2023-10-14T18:17:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.395+0000 7febf5adf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf01575b0 con 0x7febdc0047c0 2023-10-14T18:17:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53612/0 (socket says 172.21.15.29:53612) 2023-10-14T18:17:49.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 -- 172.21.15.29:0/3165814931 learned_addr learned my addr 172.21.15.29:0/3165814931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:49.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febeffff640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febf0158090 0x7febf0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf4add640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febdc0047c0 0x7febf014e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 -- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febdc0047c0 msgr2=0x7febf014e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febdc0047c0 0x7febf014e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 -- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febf0158090 msgr2=0x7febf0154ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febf0158090 0x7febf0154ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.396+0000 7febf52de640 1 -- 172.21.15.29:0/3165814931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf0157890 con 0x7febf0158510 2023-10-14T18:17:49.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febf52de640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7febf008b430 tx=0x7febe800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8007440 con 0x7febf0158510 2023-10-14T18:17:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe80075a0 con 0x7febf0158510 2023-10-14T18:17:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febe8004a20 con 0x7febf0158510 2023-10-14T18:17:49.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf0157b70 con 0x7febf0158510 2023-10-14T18:17:49.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.397+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf0152ce0 con 0x7febf0158510 2023-10-14T18:17:49.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.398+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf00c99f0 con 0x7febf0158510 2023-10-14T18:17:49.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.399+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 136) v1 ==== 89331+0+0 (secure 0 0 0) 0x7febe8004b80 con 0x7febf0158510 2023-10-14T18:17:49.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.400+0000 7febed7fa640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:49.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.400+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7febe80897e0 con 0x7febf0158510 2023-10-14T18:17:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.401+0000 7febf4add640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:49.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.401+0000 7febf4add640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 0x7febd006d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febf0154900 tx=0x7febe000b040 comp rx=0 tx=0).ready entity=mgr.11010 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:49.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.405+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7febe80541f0 con 0x7febf0158510 2023-10-14T18:17:49.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:49.653+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7febf00d2890 con 0x7febf0158510 2023-10-14T18:17:50.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.367+0000 7febed7fa640 1 -- 172.21.15.29:0/3165814931 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v137) v1 ==== 92+0+0 (secure 0 0 0) 0x7febf00d2890 con 0x7febf0158510 2023-10-14T18:17:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.371+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 msgr2=0x7febd006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.371+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 0x7febd006d450 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7febf0154900 tx=0x7febe000b040 comp rx=0 tx=0).stop 2023-10-14T18:17:50.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.371+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 msgr2=0x7febf01551e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.371+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 secure :-1 s=READY pgs=2654 cs=0 l=1 rev1=1 crypto rx=0x7febf008b430 tx=0x7febe800d9d0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 shutdown_connections 2023-10-14T18:17:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7febd006af90 0x7febd006d450 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7febf0158510 0x7febf01551e0 unknown :-1 s=CLOSED pgs=2654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7febf0158090 0x7febf0154ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 --2- 172.21.15.29:0/3165814931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7febdc0047c0 0x7febf014e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 >> 172.21.15.29:0/3165814931 conn(0x7febf00cf550 msgr2=0x7febf015d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.372+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 shutdown_connections 2023-10-14T18:17:50.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.373+0000 7febf5adf640 1 -- 172.21.15.29:0/3165814931 wait complete. 2023-10-14T18:17:50.405 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T18:17:50.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.616+0000 7f3557df0640 1 Processor -- start 2023-10-14T18:17:50.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.617+0000 7f3557df0640 1 -- start start 2023-10-14T18:17:50.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f3557df0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3550159e90 0x7f35500ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f3557df0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f355006c140 0x7f355006c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f3557df0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f355006cad0 con 0x7f3550159ab0 2023-10-14T18:17:50.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f3557df0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f355015d890 con 0x7f3550159e90 2023-10-14T18:17:50.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f3557df0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3550070360 con 0x7f355006c140 2023-10-14T18:17:50.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.618+0000 7f355538b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3550159e90 0x7f35500ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:50.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f3555b8c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3550159ab0 0x7f355006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:33978/0 (socket says 172.21.15.29:33978) 2023-10-14T18:17:50.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f3555b8c640 1 -- 172.21.15.29:0/909332369 learned_addr learned my addr 172.21.15.29:0/909332369 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:50.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f355638d640 1 --2- 172.21.15.29:0/909332369 >> v2:172.21.15.186:3300/0 conn(0x7f355006c140 0x7f355006c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:50.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f355538b640 1 -- 172.21.15.29:0/909332369 >> v2:172.21.15.186:3300/0 conn(0x7f355006c140 msgr2=0x7f355006c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f355538b640 1 --2- 172.21.15.29:0/909332369 >> v2:172.21.15.186:3300/0 conn(0x7f355006c140 0x7f355006c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f355538b640 1 -- 172.21.15.29:0/909332369 >> v1:172.21.15.29:6789/0 conn(0x7f3550159ab0 legacy=0x7f355006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.619+0000 7f355538b640 1 -- 172.21.15.29:0/909332369 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500cd1f0 con 0x7f3550159e90 2023-10-14T18:17:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.620+0000 7f355538b640 1 --2- 172.21.15.29:0/909332369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159e90 0x7f35500ce370 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f35500adcd0 tx=0x7f354402df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edb20c8f51e8d5ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:50.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3554b8a640 1 -- 172.21.15.29:0/909332369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354402ed30 con 0x7f3550159e90 2023-10-14T18:17:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3554b8a640 1 -- 172.21.15.29:0/909332369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354402ee90 con 0x7f3550159e90 2023-10-14T18:17:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3554b8a640 1 -- 172.21.15.29:0/909332369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3544036660 con 0x7f3550159e90 2023-10-14T18:17:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 -- 172.21.15.29:0/909332369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159e90 msgr2=0x7f35500ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 --2- 172.21.15.29:0/909332369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159e90 0x7f35500ce370 secure :-1 s=READY pgs=2657 cs=0 l=1 rev1=1 crypto rx=0x7f35500adcd0 tx=0x7f354402df80 comp rx=0 tx=0).stop 2023-10-14T18:17:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 -- 172.21.15.29:0/909332369 shutdown_connections 2023-10-14T18:17:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 --2- 172.21.15.29:0/909332369 >> v2:172.21.15.186:3300/0 conn(0x7f355006c140 0x7f355006c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 --2- 172.21.15.29:0/909332369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159e90 0x7f35500ce370 unknown :-1 s=CLOSED pgs=2657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 -- 172.21.15.29:0/909332369 >> 172.21.15.29:0/909332369 conn(0x7f35500ce9b0 msgr2=0x7f35500cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:50.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.621+0000 7f3557df0640 1 -- 172.21.15.29:0/909332369 shutdown_connections 2023-10-14T18:17:50.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.622+0000 7f3557df0640 1 -- 172.21.15.29:0/909332369 wait complete. 2023-10-14T18:17:50.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.623+0000 7f3557df0640 1 Processor -- start 2023-10-14T18:17:50.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.623+0000 7f3557df0640 1 -- start start 2023-10-14T18:17:50.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f355006c140 0x7f35500c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3550159e90 0x7f35500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35500c3f00 con 0x7f3550159ab0 2023-10-14T18:17:50.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35501f69d0 con 0x7f3550159e90 2023-10-14T18:17:50.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.624+0000 7f3557df0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35501f6cb0 con 0x7f355006c140 2023-10-14T18:17:50.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:50.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:53650/0 (socket says 172.21.15.29:53650) 2023-10-14T18:17:50.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 -- 172.21.15.29:0/2372034466 learned_addr learned my addr 172.21.15.29:0/2372034466 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:50.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f3555b8c640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f355006c140 0x7f35500c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:50.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355638d640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3550159e90 0x7f35500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:50.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f355006c140 msgr2=0x7f35500c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f355006c140 0x7f35500c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3550159e90 msgr2=0x7f35500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3550159e90 0x7f35500ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 -- 172.21.15.29:0/2372034466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f354400b6a0 con 0x7f3550159ab0 2023-10-14T18:17:50.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.625+0000 7f355538b640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f3544039040 tx=0x7f35440046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:50.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.626+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354403f640 con 0x7f3550159ab0 2023-10-14T18:17:50.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.626+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f354403f7a0 con 0x7f3550159ab0 2023-10-14T18:17:50.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.626+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35500c4110 con 0x7f3550159ab0 2023-10-14T18:17:50.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.626+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f354403fb50 con 0x7f3550159ab0 2023-10-14T18:17:50.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.626+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35501f26e0 con 0x7f3550159ab0 2023-10-14T18:17:50.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.628+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35501f2930 con 0x7f3550159ab0 2023-10-14T18:17:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.628+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89348+0+0 (secure 0 0 0) 0x7f354403d070 con 0x7f3550159ab0 2023-10-14T18:17:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.630+0000 7f35427fc640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 0x7f353406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.630+0000 7f3555b8c640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 msgr2=0x7f353406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:50.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.630+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f35440abfc0 con 0x7f3550159ab0 2023-10-14T18:17:50.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.630+0000 7f3555b8c640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 0x7f353406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:50.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.633+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3544076b70 con 0x7f3550159ab0 2023-10-14T18:17:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.831+0000 7f3555b8c640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 msgr2=0x7f353406d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:50.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.831+0000 7f3555b8c640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 0x7f353406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:50.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.840+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f35500b1660 con 0x7f3550159ab0 2023-10-14T18:17:50.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.846+0000 7f35427fc640 1 -- 172.21.15.29:0/2372034466 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v77)=0 v77) v1 ==== 138+0+0 (secure 0 0 0) 0x7f35440430d0 con 0x7f3550159ab0 2023-10-14T18:17:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 msgr2=0x7f353406d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 0x7f353406d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 msgr2=0x7f35501f21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 secure :-1 s=READY pgs=2658 cs=0 l=1 rev1=1 crypto rx=0x7f3544039040 tx=0x7f35440046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 shutdown_connections 2023-10-14T18:17:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7f353406b010 0x7f353406d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3550159e90 0x7f35500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3550159ab0 0x7f35501f21a0 unknown :-1 s=CLOSED pgs=2658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 --2- 172.21.15.29:0/2372034466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f355006c140 0x7f35500c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 >> 172.21.15.29:0/2372034466 conn(0x7f35500ce9b0 msgr2=0x7f35500b0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 shutdown_connections 2023-10-14T18:17:50.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:50.849+0000 7f3557df0640 1 -- 172.21.15.29:0/2372034466 wait complete. 2023-10-14T18:17:50.868 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T18:17:50.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:50.886+0000 7f3dde61f040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:17:50.941 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:50.940+0000 7f3dde61f040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:17:50.945 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:50.944+0000 7fd8b5a45040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:17:51.003 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.002+0000 7fd8b5a45040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:17:51.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.082+0000 7fdc09f47640 1 Processor -- start 2023-10-14T18:17:51.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.082+0000 7fdc09f47640 1 -- start start 2023-10-14T18:17:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc09f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040caf80 0x7fdc040ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc09f47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc040b4e50 0x7fdc04069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc09f47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc040c9930 con 0x7fdc04065ab0 2023-10-14T18:17:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc09f47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc0415b1e0 con 0x7fdc040b4e50 2023-10-14T18:17:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc09f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc040c9c10 con 0x7fdc040caf80 2023-10-14T18:17:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc03fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc040b4e50 0x7fdc04069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc03fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdc040b4e50 0x7fdc04069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33894/0 (socket says 172.21.15.29:33894) 2023-10-14T18:17:51.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc03fff640 1 -- 172.21.15.29:0/1719474479 learned_addr learned my addr 172.21.15.29:0/1719474479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:51.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.084+0000 7fdc037fe640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040caf80 0x7fdc040ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.085+0000 7fdc03fff640 1 -- 172.21.15.29:0/1719474479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040caf80 msgr2=0x7fdc040ca9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.085+0000 7fdc03fff640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040caf80 0x7fdc040ca9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.085+0000 7fdc03fff640 1 -- 172.21.15.29:0/1719474479 >> v1:172.21.15.29:6789/0 conn(0x7fdc04065ab0 legacy=0x7fdc040cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.085+0000 7fdc03fff640 1 -- 172.21.15.29:0/1719474479 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc040c9ef0 con 0x7fdc040b4e50 2023-10-14T18:17:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.086+0000 7fdc03fff640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc040b4e50 0x7fdc04069da0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009e80 tx=0x7fdbfc02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a225aeee276b0e4f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:51.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.086+0000 7fdc02ffd640 1 -- 172.21.15.29:0/1719474479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc02ca50 con 0x7fdc040b4e50 2023-10-14T18:17:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.086+0000 7fdc02ffd640 1 -- 172.21.15.29:0/1719474479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbfc02cbb0 con 0x7fdc040b4e50 2023-10-14T18:17:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.086+0000 7fdc02ffd640 1 -- 172.21.15.29:0/1719474479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbfc0343f0 con 0x7fdc040b4e50 2023-10-14T18:17:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 -- 172.21.15.29:0/1719474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc040b4e50 msgr2=0x7fdc04069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc040b4e50 0x7fdc04069da0 secure :-1 s=READY pgs=2660 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009e80 tx=0x7fdbfc02beb0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 -- 172.21.15.29:0/1719474479 shutdown_connections 2023-10-14T18:17:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc040b4e50 0x7fdc04069da0 unknown :-1 s=CLOSED pgs=2660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 --2- 172.21.15.29:0/1719474479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040caf80 0x7fdc040ca9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 -- 172.21.15.29:0/1719474479 >> 172.21.15.29:0/1719474479 conn(0x7fdc0406bc10 msgr2=0x7fdc0406c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:51.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 -- 172.21.15.29:0/1719474479 shutdown_connections 2023-10-14T18:17:51.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.087+0000 7fdc09f47640 1 -- 172.21.15.29:0/1719474479 wait complete. 2023-10-14T18:17:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.089+0000 7fdc09f47640 1 Processor -- start 2023-10-14T18:17:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.089+0000 7fdc09f47640 1 -- start start 2023-10-14T18:17:51.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040b4e50 0x7fdc041f26a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 0x7fdc040ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc041f3160 con 0x7fdc04065ab0 2023-10-14T18:17:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc041f3440 con 0x7fdc040caf80 2023-10-14T18:17:51.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc09f47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc041f3a20 con 0x7fdc040b4e50 2023-10-14T18:17:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc037fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33896/0 (socket says 172.21.15.29:33896) 2023-10-14T18:17:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.090+0000 7fdc037fe640 1 -- 172.21.15.29:0/428898134 learned_addr learned my addr 172.21.15.29:0/428898134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc03fff640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 0x7fdc040ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040b4e50 msgr2=0x7fdc041f26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdbfbfff640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040b4e50 0x7fdc041f26a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040b4e50 0x7fdc041f26a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 msgr2=0x7fdc040ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 0x7fdc040ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 -- 172.21.15.29:0/428898134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbfc009b30 con 0x7fdc04065ab0 2023-10-14T18:17:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc03fff640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 0x7fdc040ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc037fe640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e870 tx=0x7fdbf400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf40094f0 con 0x7fdc04065ab0 2023-10-14T18:17:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbf400c940 con 0x7fdc04065ab0 2023-10-14T18:17:51.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf400ccf0 con 0x7fdc04065ab0 2023-10-14T18:17:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.091+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc041f3d00 con 0x7fdc04065ab0 2023-10-14T18:17:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.092+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc040c5070 con 0x7fdc04065ab0 2023-10-14T18:17:51.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.093+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc041f3860 con 0x7fdc04065ab0 2023-10-14T18:17:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.094+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89348+0+0 (secure 0 0 0) 0x7fdbf4013470 con 0x7fdc04065ab0 2023-10-14T18:17:51.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.095+0000 7fdc00ff9640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 0x7fdbe406d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.095+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fdbf4089c80 con 0x7fdc04065ab0 2023-10-14T18:17:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.095+0000 7fdbfbfff640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 msgr2=0x7fdbe406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.095+0000 7fdbfbfff640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 0x7fdbe406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:51.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.099+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbf4054840 con 0x7fdc04065ab0 2023-10-14T18:17:51.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.296+0000 7fdbfbfff640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 msgr2=0x7fdbe406d4a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:51.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.296+0000 7fdbfbfff640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 0x7fdbe406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:51.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.389+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7fdc0404e160 con 0x7fdc04065ab0 2023-10-14T18:17:51.395 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.394+0000 7f3dde61f040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:17:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.396+0000 7fdc00ff9640 1 -- 172.21.15.29:0/428898134 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v78)=0 v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7fdbf40170c0 con 0x7fdc04065ab0 2023-10-14T18:17:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 msgr2=0x7fdbe406d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 0x7fdbe406d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 msgr2=0x7fdc041504c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 secure :-1 s=READY pgs=2661 cs=0 l=1 rev1=1 crypto rx=0x7fdbf400e870 tx=0x7fdbf400ed30 comp rx=0 tx=0).stop 2023-10-14T18:17:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 shutdown_connections 2023-10-14T18:17:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7fdbe406afe0 0x7fdbe406d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdc040caf80 0x7fdc040ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdc040b4e50 0x7fdc041f26a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 --2- 172.21.15.29:0/428898134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdc04065ab0 0x7fdc041504c0 unknown :-1 s=CLOSED pgs=2661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 >> 172.21.15.29:0/428898134 conn(0x7fdc0406bc10 msgr2=0x7fdc040b4010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 shutdown_connections 2023-10-14T18:17:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.399+0000 7fdc09f47640 1 -- 172.21.15.29:0/428898134 wait complete. 2023-10-14T18:17:51.420 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:17:51.420 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-14T18:17:51.448 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.447+0000 7f3dde61f040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:17:51.481 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:17:51.500 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.499+0000 7f3dde61f040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:17:51.561 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.560+0000 7f3dde61f040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:17:51.576 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.575+0000 7fd8b5a45040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:17:51.610 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.608+0000 7f3dde61f040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:17:51.634 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.633+0000 7fd8b5a45040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:17:51.692 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.691+0000 7fd8b5a45040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:17:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.698+0000 7ff0e67c0640 1 Processor -- start 2023-10-14T18:17:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.699+0000 7ff0e67c0640 1 -- start start 2023-10-14T18:17:51.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e67c0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 0x7ff0e00c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e67c0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0e00cce20 0x7ff0e006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e67c0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0e015dfb0 con 0x7ff0e0159da0 2023-10-14T18:17:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e67c0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0e015e290 con 0x7ff0e00cce20 2023-10-14T18:17:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e67c0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e015e570 con 0x7ff0e01599c0 2023-10-14T18:17:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e4d5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0e00cce20 0x7ff0e006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e4d5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0e00cce20 0x7ff0e006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33902/0 (socket says 172.21.15.29:33902) 2023-10-14T18:17:51.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/3347747224 learned_addr learned my addr 172.21.15.29:0/3347747224 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:51.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0df7fe640 1 --1- 172.21.15.29:0/3347747224 >> v1:172.21.15.29:6789/0 conn(0x7ff0e0159da0 0x7ff0e015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46136/0 (socket says 172.21.15.29:46136) 2023-10-14T18:17:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.701+0000 7ff0dffff640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 0x7ff0e00c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.702+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/3347747224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 msgr2=0x7ff0e00c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.702+0000 7ff0e4d5d640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 0x7ff0e00c9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.702+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/3347747224 >> v1:172.21.15.29:6789/0 conn(0x7ff0e0159da0 legacy=0x7ff0e015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.702+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/3347747224 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e006fda0 con 0x7ff0e00cce20 2023-10-14T18:17:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.702+0000 7ff0dffff640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 0x7ff0e00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:51.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.703+0000 7ff0e4d5d640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e00cce20 0x7ff0e006c0d0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009e50 tx=0x7ff0d802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4405d873a774f842 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.703+0000 7ff0deffd640 1 -- 172.21.15.29:0/3347747224 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d802cb00 con 0x7ff0e00cce20 2023-10-14T18:17:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.703+0000 7ff0deffd640 1 -- 172.21.15.29:0/3347747224 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d802cc60 con 0x7ff0e00cce20 2023-10-14T18:17:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.703+0000 7ff0deffd640 1 -- 172.21.15.29:0/3347747224 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d80343f0 con 0x7ff0e00cce20 2023-10-14T18:17:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 -- 172.21.15.29:0/3347747224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e00cce20 msgr2=0x7ff0e006c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e00cce20 0x7ff0e006c0d0 secure :-1 s=READY pgs=2663 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009e50 tx=0x7ff0d802bf60 comp rx=0 tx=0).stop 2023-10-14T18:17:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 -- 172.21.15.29:0/3347747224 shutdown_connections 2023-10-14T18:17:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e00cce20 0x7ff0e006c0d0 unknown :-1 s=CLOSED pgs=2663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 --2- 172.21.15.29:0/3347747224 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e01599c0 0x7ff0e00c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 -- 172.21.15.29:0/3347747224 >> 172.21.15.29:0/3347747224 conn(0x7ff0e00cb360 msgr2=0x7ff0e00cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 -- 172.21.15.29:0/3347747224 shutdown_connections 2023-10-14T18:17:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.704+0000 7ff0e67c0640 1 -- 172.21.15.29:0/3347747224 wait complete. 2023-10-14T18:17:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.707+0000 7ff0e67c0640 1 Processor -- start 2023-10-14T18:17:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.707+0000 7ff0e67c0640 1 -- start start 2023-10-14T18:17:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e00cce20 0x7ff0e014d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e01599c0 0x7ff0e0167460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 0x7ff0e015c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e014c620 con 0x7ff0e0159da0 2023-10-14T18:17:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e014d160 con 0x7ff0e01599c0 2023-10-14T18:17:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e67c0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0e01679a0 con 0x7ff0e00cce20 2023-10-14T18:17:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e4d5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 0x7ff0e015c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e4d5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 0x7ff0e015c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33914/0 (socket says 172.21.15.29:33914) 2023-10-14T18:17:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/1087463237 learned_addr learned my addr 172.21.15.29:0/1087463237 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e00cce20 0x7ff0e014d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0df7fe640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e01599c0 0x7ff0e0167460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.709+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e00cce20 msgr2=0x7ff0e014d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e4d5d640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff0e00cce20 0x7ff0e014d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e01599c0 msgr2=0x7ff0e0167460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e4d5d640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e01599c0 0x7ff0e0167460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d8009b30 con 0x7ff0e0159da0 2023-10-14T18:17:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0df7fe640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0e01599c0 0x7ff0e0167460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e4d5d640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 0x7ff0e015c620 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009e50 tx=0x7ff0d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d8009510 con 0x7ff0e0159da0 2023-10-14T18:17:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0d803e450 con 0x7ff0e0159da0 2023-10-14T18:17:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0d803e800 con 0x7ff0e0159da0 2023-10-14T18:17:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0e014c830 con 0x7ff0e0159da0 2023-10-14T18:17:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.710+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0e014cdf0 con 0x7ff0e0159da0 2023-10-14T18:17:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.712+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0e014d2d0 con 0x7ff0e0159da0 2023-10-14T18:17:51.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.713+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 137) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff0d803e960 con 0x7ff0e0159da0 2023-10-14T18:17:51.723 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.720+0000 7f3dde61f040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:17:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.713+0000 7ff0dcff9640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.713+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(233..233 src has 1..233) v4 ==== 6689+0+0 (secure 0 0 0) 0x7ff0d80ac9e0 con 0x7ff0e0159da0 2023-10-14T18:17:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.714+0000 7ff0dffff640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:51.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.714+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:17:51.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.718+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff0d8077500 con 0x7ff0e0159da0 2023-10-14T18:17:51.766 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.765+0000 7fd8b5a45040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:17:51.783 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.782+0000 7f3dde61f040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:51.822 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.822+0000 7fd8b5a45040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:17:51.833 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.831+0000 7f3dde61f040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:17:51.890 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.889+0000 7f3dde61f040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:17:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.915+0000 7ff0dffff640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.915+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:17:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:51.938+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0e00cd2b0 con 0x7ff0cc06b010 2023-10-14T18:17:51.940 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:51.939+0000 7f3dde61f040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:17:51.943 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:51.942+0000 7fd8b5a45040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:17:52.008 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.008+0000 7fd8b5a45040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:52.060 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.059+0000 7fd8b5a45040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:17:52.119 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.118+0000 7fd8b5a45040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:17:52.172 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.171+0000 7fd8b5a45040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:17:52.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:52.316+0000 7ff0dffff640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:52.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:52.316+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T18:17:52.324 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.323+0000 7f3dde61f040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:17:52.485 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.484+0000 7f3dde61f040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:17:52.573 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.572+0000 7fd8b5a45040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:17:52.592 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.591+0000 7f3dde61f040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:17:52.642 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.641+0000 7f3dde61f040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:17:52.742 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.741+0000 7fd8b5a45040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:17:52.750 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.749+0000 7f3dde61f040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:52.855 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.854+0000 7fd8b5a45040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:17:52.856 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:52.855+0000 7f3dde61f040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:17:52.908 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:52.907+0000 7fd8b5a45040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:17:53.011 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:53.010+0000 7f3dde61f040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:53.022 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:53.021+0000 7fd8b5a45040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:53.061 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:53.059+0000 7f3dde61f040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:53.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:53.117+0000 7ff0dffff640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:53.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:53.117+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T18:17:53.134 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:53.133+0000 7fd8b5a45040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:17:53.178 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:53.177+0000 7f3dde61f040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:17:53.298 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:53.297+0000 7fd8b5a45040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:17:53.349 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:53.349+0000 7fd8b5a45040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:17:53.475 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:53.474+0000 7fd8b5a45040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:17:54.002 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:54.000+0000 7f3dde61f040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:17:54.298 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:54.296+0000 7f3dde61f040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:17:54.356 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:54.353+0000 7f3dde61f040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:17:54.367 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:54.366+0000 7fd8b5a45040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:17:54.469 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:17:54.468+0000 7f3dde61f040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:17:54.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.539+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 138) v1 ==== 89348+0+0 (secure 0 0 0) 0x7ff0d80778d0 con 0x7ff0e0159da0 2023-10-14T18:17:54.694 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:54.693+0000 7fd8b5a45040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:17:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.719+0000 7ff0dffff640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2967217610 2023-10-14T18:17:54.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.719+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T18:17:54.759 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:54.758+0000 7fd8b5a45040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:17:54.883 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:54.882+0000 7fd8b5a45040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:17:54.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.899+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 139) v1 ==== 88957+0+0 (secure 0 0 0) 0x7ff0d803ec10 con 0x7ff0e0159da0 2023-10-14T18:17:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.900+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 msgr2=0x7ff0cc06d4d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:17:54.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:54.900+0000 7ff0dcff9640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2967217610,v1:172.21.15.29:6833/2967217610] conn(0x7ff0cc06b010 0x7ff0cc06d4d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:55.506 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.506 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.507 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.505+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.523+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.523+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.523+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.523+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.524 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.523+0000 7fd834552640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.524+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.525 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.524+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.524+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.524+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.526 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:17:55.524+0000 7fd831d4d640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:17:55.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:55.917+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mon.0 v2:172.21.15.29:3300/0 9 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7ff0d8039d10 con 0x7ff0e0159da0 2023-10-14T18:17:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:55.918+0000 7ff0dcff9640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7ff0cc027990 0x7ff0cc027d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:55.918+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0cc00b680 con 0x7ff0cc027990 2023-10-14T18:17:55.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:55.928+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7ff0cc027990 0x7ff0cc027d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:55.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:55.940+0000 7ff0dffff640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7ff0cc027990 0x7ff0cc027d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0006fd0 tx=0x7ff0d0008040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:56.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.129+0000 7ff0dcff9640 1 -- 172.21.15.29:0/1087463237 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+53 (secure 0 0 0) 0x7ff0cc00b680 con 0x7ff0cc027990 2023-10-14T18:17:56.131 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path mandatory_fs_argument_test_dir 2023-10-14T18:17:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7ff0cc027990 msgr2=0x7ff0cc027d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7ff0cc027990 0x7ff0cc027d70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7ff0d0006fd0 tx=0x7ff0d0008040 comp rx=0 tx=0).stop 2023-10-14T18:17:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 msgr2=0x7ff0e015c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 --2- 172.21.15.29:0/1087463237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0e0159da0 0x7ff0e015c620 secure :-1 s=READY pgs=2664 cs=0 l=1 rev1=1 crypto rx=0x7ff0d8009e50 tx=0x7ff0d8004a40 comp rx=0 tx=0).stop 2023-10-14T18:17:56.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e4d5d640 1 -- 172.21.15.29:0/1087463237 reap_dead start 2023-10-14T18:17:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 shutdown_connections 2023-10-14T18:17:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 >> 172.21.15.29:0/1087463237 conn(0x7ff0e00cb360 msgr2=0x7ff0e006bb20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 shutdown_connections 2023-10-14T18:17:56.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.133+0000 7ff0e67c0640 1 -- 172.21.15.29:0/1087463237 wait complete. 2023-10-14T18:17:56.156 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path mandatory_fs_argument_test_dir 2023-10-14T18:17:56.156 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule activate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:17:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.386+0000 7f95b2bf4640 1 Processor -- start 2023-10-14T18:17:56.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.386+0000 7f95b2bf4640 1 -- start start 2023-10-14T18:17:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b2bf4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f95ac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b2bf4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac15f4a0 con 0x7f95ac0cda80 2023-10-14T18:17:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b2bf4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95ac06c960 con 0x7f95ac0ad970 2023-10-14T18:17:56.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b2bf4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac15b9d0 con 0x7f95ac06c040 2023-10-14T18:17:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b23f3640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f95ac0cda80 0x7f95ac15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46152/0 (socket says 172.21.15.29:46152) 2023-10-14T18:17:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.388+0000 7f95b23f3640 1 -- 172.21.15.29:0/1774521641 learned_addr learned my addr 172.21.15.29:0/1774521641 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b1bf2640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f95ac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 745554186 0 0) 0x7f95ac15f4a0 con 0x7f95ac0cda80 2023-10-14T18:17:56.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9598003520 con 0x7f95ac0cda80 2023-10-14T18:17:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2256596225 0 0) 0x7f9598003520 con 0x7f95ac0cda80 2023-10-14T18:17:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 msgr2=0x7f95ac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f95ac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.389+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 >> v1:172.21.15.186:6789/0 conn(0x7f95ac0ad970 legacy=0x7f95ac15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac15bcb0 con 0x7f95ac0cda80 2023-10-14T18:17:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f95a40019f0 con 0x7f95ac0cda80 2023-10-14T18:17:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 >> v1:172.21.15.29:6789/0 conn(0x7f95ac0cda80 legacy=0x7f95ac15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980039a0 0x7f9598003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95980047c0 0x7f9598004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f9598005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac15b9d0 con 0x7f95980047c0 2023-10-14T18:17:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f959800ba30 con 0x7f95980039a0 2023-10-14T18:17:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f959800bbf0 con 0x7f95ac06c040 2023-10-14T18:17:56.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.390+0000 7f95b13f1640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95980047c0 0x7f9598004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b1bf2640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980039a0 0x7f9598003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b23f3640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f9598005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b13f1640 1 -- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 msgr2=0x7f9598005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b13f1640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f9598005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b13f1640 1 -- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980039a0 msgr2=0x7f9598003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:56.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b13f1640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980039a0 0x7f9598003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b13f1640 1 -- 172.21.15.29:0/1774521641 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9598003520 con 0x7f95980047c0 2023-10-14T18:17:56.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.391+0000 7f95b23f3640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac06c040 0x7f9598005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:17:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b13f1640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95980047c0 0x7f9598004c40 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f959c001720 tx=0x7f95980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0f5ae423f33ddf7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c0052d0 con 0x7f95980047c0 2023-10-14T18:17:56.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959c02e050 con 0x7f95980047c0 2023-10-14T18:17:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b0bf0640 1 -- 172.21.15.29:0/1774521641 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f959c0035f0 con 0x7f95980047c0 2023-10-14T18:17:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b2bf4640 1 -- 172.21.15.29:0/1774521641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95980047c0 msgr2=0x7f9598004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.393+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/1774521641 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95980047c0 0x7f9598004c40 secure :-1 s=READY pgs=2682 cs=0 l=1 rev1=1 crypto rx=0x7f959c001720 tx=0x7f95980057f0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.394+0000 7f95b23f3640 1 -- 172.21.15.29:0/1774521641 reap_dead start 2023-10-14T18:17:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.394+0000 7f95b2bf4640 1 -- 172.21.15.29:0/1774521641 shutdown_connections 2023-10-14T18:17:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.394+0000 7f95b2bf4640 1 -- 172.21.15.29:0/1774521641 >> 172.21.15.29:0/1774521641 conn(0x7f95ac0ce6a0 msgr2=0x7f95ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:56.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.394+0000 7f95b2bf4640 1 -- 172.21.15.29:0/1774521641 shutdown_connections 2023-10-14T18:17:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.394+0000 7f95b2bf4640 1 -- 172.21.15.29:0/1774521641 wait complete. 2023-10-14T18:17:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.396+0000 7f95b2bf4640 1 Processor -- start 2023-10-14T18:17:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.396+0000 7f95b2bf4640 1 -- start start 2023-10-14T18:17:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980047c0 0x7f95ac14b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac0c6ee0 0x7f95ac159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac1f1c40 con 0x7f95ac0c7360 2023-10-14T18:17:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac1f1f20 con 0x7f95980047c0 2023-10-14T18:17:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b2bf4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95ac1f2200 con 0x7f95ac0c6ee0 2023-10-14T18:17:56.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b23f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b23f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34072/0 (socket says 172.21.15.29:34072) 2023-10-14T18:17:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b23f3640 1 -- 172.21.15.29:0/2653930767 learned_addr learned my addr 172.21.15.29:0/2653930767 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b13f1640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac0c6ee0 0x7f95ac159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.397+0000 7f95b1bf2640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980047c0 0x7f95ac14b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 -- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac0c6ee0 msgr2=0x7f95ac159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac0c6ee0 0x7f95ac159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 -- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980047c0 msgr2=0x7f95ac14b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980047c0 0x7f95ac14b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 -- 172.21.15.29:0/2653930767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95ac1f24e0 con 0x7f95ac0c7360 2023-10-14T18:17:56.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b23f3640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0b11c0 tx=0x7f95a400e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a400eb90 con 0x7f95ac0c7360 2023-10-14T18:17:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95a400ecf0 con 0x7f95ac0c7360 2023-10-14T18:17:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95a4013660 con 0x7f95ac0c7360 2023-10-14T18:17:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.398+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95ac14bc00 con 0x7f95ac0c7360 2023-10-14T18:17:56.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.399+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95ac1f19d0 con 0x7f95ac0c7360 2023-10-14T18:17:56.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.400+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95ac06b330 con 0x7f95ac0c7360 2023-10-14T18:17:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.401+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 140) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f95a40137c0 con 0x7f95ac0c7360 2023-10-14T18:17:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.402+0000 7f95a27fc640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 0x7f959006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.403+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f95a4089ce0 con 0x7f95ac0c7360 2023-10-14T18:17:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.403+0000 7f95b1bf2640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 0x7f959006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:56.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.403+0000 7f95b1bf2640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 0x7f959006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009b00 tx=0x7f95a8008040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:56.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.408+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f95a4054ad0 con 0x7f95ac0c7360 2023-10-14T18:17:56.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.632+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule activate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f95ac1f2090 con 0x7f959006ad40 2023-10-14T18:17:56.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.780+0000 7f95a27fc640 1 -- 172.21.15.29:0/2653930767 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f95ac1f2090 con 0x7f959006ad40 2023-10-14T18:17:56.782 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule activated for path mandatory_fs_argument_test_dir 2023-10-14T18:17:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 msgr2=0x7f959006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 0x7f959006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f95a8009b00 tx=0x7f95a8008040 comp rx=0 tx=0).stop 2023-10-14T18:17:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 msgr2=0x7f95ac14e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:56.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 secure :-1 s=READY pgs=2683 cs=0 l=1 rev1=1 crypto rx=0x7f95ac0b11c0 tx=0x7f95a400e530 comp rx=0 tx=0).stop 2023-10-14T18:17:56.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 shutdown_connections 2023-10-14T18:17:56.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.784+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f959006ad40 0x7f959006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f95ac0c7360 0x7f95ac14e9f0 unknown :-1 s=CLOSED pgs=2683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f95ac0c6ee0 0x7f95ac159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 --2- 172.21.15.29:0/2653930767 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f95980047c0 0x7f95ac14b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:56.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 >> 172.21.15.29:0/2653930767 conn(0x7f95ac0ce6a0 msgr2=0x7f95ac15d8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:56.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 shutdown_connections 2023-10-14T18:17:56.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:56.785+0000 7f95b2bf4640 1 -- 172.21.15.29:0/2653930767 wait complete. 2023-10-14T18:17:56.809 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule activated for path mandatory_fs_argument_test_dir 2023-10-14T18:17:56.809 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:17:57.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.045+0000 7fd121bfc640 1 Processor -- start 2023-10-14T18:17:57.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.045+0000 7fd121bfc640 1 -- start start 2023-10-14T18:17:57.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.046+0000 7fd121bfc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd11c0b3420 0x7fd11c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.046+0000 7fd121bfc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c0d3980 con 0x7fd11c0cfaf0 2023-10-14T18:17:57.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.046+0000 7fd121bfc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd11c0ce630 con 0x7fd11c0cf710 2023-10-14T18:17:57.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.046+0000 7fd121bfc640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd11c0d3c60 con 0x7fd11c0b3420 2023-10-14T18:17:57.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd113fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd11c0b3420 0x7fd11c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd11bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd11c0cfaf0 0x7fd11c0b6200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46168/0 (socket says 172.21.15.29:46168) 2023-10-14T18:17:57.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd11bfff640 1 -- 172.21.15.29:0/4012641425 learned_addr learned my addr 172.21.15.29:0/4012641425 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:57.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd113fff640 1 -- 172.21.15.29:0/4012641425 >> v1:172.21.15.186:6789/0 conn(0x7fd11c0cf710 legacy=0x7fd11c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd113fff640 1 -- 172.21.15.29:0/4012641425 >> v1:172.21.15.29:6789/0 conn(0x7fd11c0cfaf0 legacy=0x7fd11c0b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd113fff640 1 -- 172.21.15.29:0/4012641425 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c0ce910 con 0x7fd11c0b3420 2023-10-14T18:17:57.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.047+0000 7fd11affd640 1 -- 172.21.15.29:0/4012641425 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727058446 0 0) 0x7fd11c0d3980 con 0x7fd11c0cfaf0 2023-10-14T18:17:57.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.049+0000 7fd113fff640 1 --2- 172.21.15.29:0/4012641425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c0b5cc0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fd1040097a0 tx=0x7fd10402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0f89c30ffcd8f98 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.049+0000 7fd11affd640 1 -- 172.21.15.29:0/4012641425 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104031830 con 0x7fd11c0b3420 2023-10-14T18:17:57.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.049+0000 7fd11affd640 1 -- 172.21.15.29:0/4012641425 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd104031990 con 0x7fd11c0b3420 2023-10-14T18:17:57.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.049+0000 7fd11affd640 1 -- 172.21.15.29:0/4012641425 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd104031d20 con 0x7fd11c0b3420 2023-10-14T18:17:57.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 -- 172.21.15.29:0/4012641425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 msgr2=0x7fd11c0b5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 --2- 172.21.15.29:0/4012641425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c0b5cc0 secure :-1 s=READY pgs=2687 cs=0 l=1 rev1=1 crypto rx=0x7fd1040097a0 tx=0x7fd10402b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 -- 172.21.15.29:0/4012641425 shutdown_connections 2023-10-14T18:17:57.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 --2- 172.21.15.29:0/4012641425 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c0b5cc0 unknown :-1 s=CLOSED pgs=2687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 -- 172.21.15.29:0/4012641425 >> 172.21.15.29:0/4012641425 conn(0x7fd11c0d2cc0 msgr2=0x7fd11c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:57.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 -- 172.21.15.29:0/4012641425 shutdown_connections 2023-10-14T18:17:57.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.050+0000 7fd121bfc640 1 -- 172.21.15.29:0/4012641425 wait complete. 2023-10-14T18:17:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.052+0000 7fd121bfc640 1 Processor -- start 2023-10-14T18:17:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.052+0000 7fd121bfc640 1 -- start start 2023-10-14T18:17:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd11c0cf710 0x7fd11c0a03b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd11c0cfaf0 0x7fd11c09d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c09ff20 con 0x7fd11c0b3420 2023-10-14T18:17:57.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c09f310 con 0x7fd11c0cf710 2023-10-14T18:17:57.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd121bfc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd11c09f5f0 con 0x7fd11c0cfaf0 2023-10-14T18:17:57.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34114/0 (socket says 172.21.15.29:34114) 2023-10-14T18:17:57.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 -- 172.21.15.29:0/979644308 learned_addr learned my addr 172.21.15.29:0/979644308 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:57.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 -- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd11c0cfaf0 msgr2=0x7fd11c09d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd11c0cfaf0 0x7fd11c09d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.053+0000 7fd11b7fe640 1 -- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd11c0cf710 msgr2=0x7fd11c0a03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd11b7fe640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd11c0cf710 0x7fd11c0a03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd11b7fe640 1 -- 172.21.15.29:0/979644308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd104009400 con 0x7fd11c0b3420 2023-10-14T18:17:57.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd11b7fe640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c450 tx=0x7fd10c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c0052d0 con 0x7fd11c0b3420 2023-10-14T18:17:57.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd10c005450 con 0x7fd11c0b3420 2023-10-14T18:17:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c003880 con 0x7fd11c0b3420 2023-10-14T18:17:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd11c09f8d0 con 0x7fd11c0b3420 2023-10-14T18:17:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.054+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd11c15f8c0 con 0x7fd11c0b3420 2023-10-14T18:17:57.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.055+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd11c04e160 con 0x7fd11c0b3420 2023-10-14T18:17:57.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.056+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 141) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fd10c0039e0 con 0x7fd11c0b3420 2023-10-14T18:17:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.057+0000 7fd118ff9640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 0x7fd0fc06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.057+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fd10c089bc0 con 0x7fd11c0b3420 2023-10-14T18:17:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.058+0000 7fd113fff640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 0x7fd0fc06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.058+0000 7fd113fff640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 0x7fd0fc06d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd104036000 tx=0x7fd104030000 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.062+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd10c0543d0 con 0x7fd11c0b3420 2023-10-14T18:17:57.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.282+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7fd11c0d1d50 con 0x7fd0fc06b100 2023-10-14T18:17:57.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.465+0000 7fd118ff9640 1 -- 172.21.15.29:0/979644308 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+55 (secure 0 0 0) 0x7fd104031b80 con 0x7fd0fc06b100 2023-10-14T18:17:57.467 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path mandatory_fs_argument_test_dir 2023-10-14T18:17:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 msgr2=0x7fd0fc06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 0x7fd0fc06d5c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd104036000 tx=0x7fd104030000 comp rx=0 tx=0).stop 2023-10-14T18:17:57.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 msgr2=0x7fd11c15f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 secure :-1 s=READY pgs=2688 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00c450 tx=0x7fd10c00c910 comp rx=0 tx=0).stop 2023-10-14T18:17:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 shutdown_connections 2023-10-14T18:17:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd0fc06b100 0x7fd0fc06d5c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd11c0cfaf0 0x7fd11c09d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd11c0cf710 0x7fd11c0a03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 --2- 172.21.15.29:0/979644308 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd11c0b3420 0x7fd11c15f380 unknown :-1 s=CLOSED pgs=2688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 >> 172.21.15.29:0/979644308 conn(0x7fd11c0d2cc0 msgr2=0x7fd11c1601b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 shutdown_connections 2023-10-14T18:17:57.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.469+0000 7fd121bfc640 1 -- 172.21.15.29:0/979644308 wait complete. 2023-10-14T18:17:57.491 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path mandatory_fs_argument_test_dir 2023-10-14T18:17:57.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-14T18:17:57.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.725+0000 7fb2e1f9a640 1 Processor -- start 2023-10-14T18:17:57.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.725+0000 7fb2e1f9a640 1 -- start start 2023-10-14T18:17:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2e1f9a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2dc070b60 0x7fb2dc15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2e1f9a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb2dc06c100 0x7fb2dc06c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2e1f9a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2dc15aef0 con 0x7fb2dc159950 2023-10-14T18:17:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2e1f9a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc15e490 con 0x7fb2dc070b60 2023-10-14T18:17:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2e1f9a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2dc15d910 con 0x7fb2dc06c100 2023-10-14T18:17:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2dc070b60 0x7fb2dc15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2dbfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb2dc070b60 0x7fb2dc15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34118/0 (socket says 172.21.15.29:34118) 2023-10-14T18:17:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.727+0000 7fb2dbfff640 1 -- 172.21.15.29:0/2297898653 learned_addr learned my addr 172.21.15.29:0/2297898653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.728+0000 7fb2dbfff640 1 -- 172.21.15.29:0/2297898653 >> v2:172.21.15.186:3300/0 conn(0x7fb2dc06c100 msgr2=0x7fb2dc06c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.728+0000 7fb2dbfff640 1 --2- 172.21.15.29:0/2297898653 >> v2:172.21.15.186:3300/0 conn(0x7fb2dc06c100 0x7fb2dc06c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.728+0000 7fb2dbfff640 1 -- 172.21.15.29:0/2297898653 >> v1:172.21.15.186:6789/0 conn(0x7fb2dc159950 legacy=0x7fb2dc070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:57.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.728+0000 7fb2dbfff640 1 -- 172.21.15.29:0/2297898653 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2dc15dbf0 con 0x7fb2dc070b60 2023-10-14T18:17:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.729+0000 7fb2dbfff640 1 --2- 172.21.15.29:0/2297898653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc070b60 0x7fb2dc15a9b0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0ade70 tx=0x7fb2c802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8358343e9a3d67d6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.729+0000 7fb2db7fe640 1 -- 172.21.15.29:0/2297898653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80317e0 con 0x7fb2dc070b60 2023-10-14T18:17:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.729+0000 7fb2db7fe640 1 -- 172.21.15.29:0/2297898653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8031940 con 0x7fb2dc070b60 2023-10-14T18:17:57.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.729+0000 7fb2db7fe640 1 -- 172.21.15.29:0/2297898653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8031cd0 con 0x7fb2dc070b60 2023-10-14T18:17:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/2297898653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc070b60 msgr2=0x7fb2dc15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/2297898653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc070b60 0x7fb2dc15a9b0 secure :-1 s=READY pgs=2689 cs=0 l=1 rev1=1 crypto rx=0x7fb2dc0ade70 tx=0x7fb2c802b880 comp rx=0 tx=0).stop 2023-10-14T18:17:57.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/2297898653 shutdown_connections 2023-10-14T18:17:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/2297898653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc070b60 0x7fb2dc15a9b0 unknown :-1 s=CLOSED pgs=2689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/2297898653 >> v2:172.21.15.186:3300/0 conn(0x7fb2dc06c100 0x7fb2dc06c510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/2297898653 >> 172.21.15.29:0/2297898653 conn(0x7fb2dc0ce6a0 msgr2=0x7fb2dc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/2297898653 shutdown_connections 2023-10-14T18:17:57.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.730+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/2297898653 wait complete. 2023-10-14T18:17:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.732+0000 7fb2e1f9a640 1 Processor -- start 2023-10-14T18:17:57.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.732+0000 7fb2e1f9a640 1 -- start start 2023-10-14T18:17:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 0x7fb2dc0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 0x7fb2dc150910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0c6860 con 0x7fb2dc159950 2023-10-14T18:17:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0c5c50 con 0x7fb2dc06c100 2023-10-14T18:17:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1f9a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2dc0c5f30 con 0x7fb2dc070b60 2023-10-14T18:17:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.733+0000 7fb2e1799640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34128/0 (socket says 172.21.15.29:34128) 2023-10-14T18:17:57.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 -- 172.21.15.29:0/3875739606 learned_addr learned my addr 172.21.15.29:0/3875739606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2dbfff640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 0x7fb2dc150910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e0f98640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 0x7fb2dc0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 -- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 msgr2=0x7fb2dc150910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 0x7fb2dc150910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 -- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 msgr2=0x7fb2dc0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 0x7fb2dc0b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 -- 172.21.15.29:0/3875739606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2c80094d0 con 0x7fb2dc159950 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e0f98640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 0x7fb2dc0b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2dbfff640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 0x7fb2dc150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.734+0000 7fb2e1799640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400b970 tx=0x7fb2d400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.735+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d400cb40 con 0x7fb2dc159950 2023-10-14T18:17:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.735+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d400ccc0 con 0x7fb2dc159950 2023-10-14T18:17:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.735+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4014680 con 0x7fb2dc159950 2023-10-14T18:17:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.735+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2dc0c6210 con 0x7fb2dc159950 2023-10-14T18:17:57.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.735+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2dc0b06b0 con 0x7fb2dc159950 2023-10-14T18:17:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.736+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2dc0c69d0 con 0x7fb2dc159950 2023-10-14T18:17:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.737+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 141) v1 ==== 89451+0+0 (secure 0 0 0) 0x7fb2d401d3b0 con 0x7fb2dc159950 2023-10-14T18:17:57.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.738+0000 7fb2d97fa640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 0x7fb2c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.738+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fb2d4089fe0 con 0x7fb2dc159950 2023-10-14T18:17:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.738+0000 7fb2e0f98640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 0x7fb2c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:57.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.739+0000 7fb2e0f98640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 0x7fb2c006d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000a810 tx=0x7fb2d0008040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:57.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.743+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb2d40547a0 con 0x7fb2dc159950 2023-10-14T18:17:57.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.961+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb2dc15e1b0 con 0x7fb2c006b100 2023-10-14T18:17:57.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.978+0000 7fb2d97fa640 1 -- 172.21.15.29:0/3875739606 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+88 (secure 0 0 0) 0x7fb2d000ad90 con 0x7fb2c006b100 2023-10-14T18:17:57.979 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:57.980 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-14T18:17:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 msgr2=0x7fb2c006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 0x7fb2c006d5c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000a810 tx=0x7fb2d0008040 comp rx=0 tx=0).stop 2023-10-14T18:17:57.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 msgr2=0x7fb2dc0c6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:57.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 secure :-1 s=READY pgs=2690 cs=0 l=1 rev1=1 crypto rx=0x7fb2d400b970 tx=0x7fb2d400be30 comp rx=0 tx=0).stop 2023-10-14T18:17:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 shutdown_connections 2023-10-14T18:17:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb2c006b100 0x7fb2c006d5c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb2dc159950 0x7fb2dc0c6cf0 unknown :-1 s=CLOSED pgs=2690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb2dc070b60 0x7fb2dc150910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 --2- 172.21.15.29:0/3875739606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb2dc06c100 0x7fb2dc0b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 >> 172.21.15.29:0/3875739606 conn(0x7fb2dc0ce6a0 msgr2=0x7fb2dc070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 shutdown_connections 2023-10-14T18:17:57.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:57.981+0000 7fb2e1f9a640 1 -- 172.21.15.29:0/3875739606 wait complete. 2023-10-14T18:17:58.002 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "mandatory_fs_argument_test_dir", "schedule": ["1M"], "retention": [{"M": 1}]} 2023-10-14T18:17:58.002 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule status mandatory_fs_argument_test_dir --fs backup_fs --format json 2023-10-14T18:17:58.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.226+0000 7f3755747640 1 Processor -- start 2023-10-14T18:17:58.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.227+0000 7f3755747640 1 -- start start 2023-10-14T18:17:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3755747640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f375006c040 0x7f375006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3755747640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37500cda80 0x7f375015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3755747640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f375006c960 con 0x7f37500ad970 2023-10-14T18:17:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3755747640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375015b9d0 con 0x7f37500cda80 2023-10-14T18:17:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3755747640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f375015bcb0 con 0x7f375006c040 2023-10-14T18:17:58.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3754f46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37500cda80 0x7f375015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3754f46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f37500cda80 0x7f375015ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34132/0 (socket says 172.21.15.29:34132) 2023-10-14T18:17:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.228+0000 7f3754f46640 1 -- 172.21.15.29:0/111276589 learned_addr learned my addr 172.21.15.29:0/111276589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:58.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.229+0000 7f374ffff640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f375006c040 0x7f375006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.229+0000 7f3754f46640 1 -- 172.21.15.29:0/111276589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f375006c040 msgr2=0x7f375006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.229+0000 7f3754f46640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f375006c040 0x7f375006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.229+0000 7f3754f46640 1 -- 172.21.15.29:0/111276589 >> v1:172.21.15.186:6789/0 conn(0x7f37500ad970 legacy=0x7f375015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.229+0000 7f3754f46640 1 -- 172.21.15.29:0/111276589 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375015bf90 con 0x7f37500cda80 2023-10-14T18:17:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.230+0000 7f3754f46640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500cda80 0x7f375015ed90 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f3748009e50 tx=0x7f374802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=541e2bc9412221a1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f374effd640 1 -- 172.21.15.29:0/111276589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f374802cb00 con 0x7f37500cda80 2023-10-14T18:17:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f374effd640 1 -- 172.21.15.29:0/111276589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f374802cc60 con 0x7f37500cda80 2023-10-14T18:17:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f374effd640 1 -- 172.21.15.29:0/111276589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37480343f0 con 0x7f37500cda80 2023-10-14T18:17:58.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 -- 172.21.15.29:0/111276589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500cda80 msgr2=0x7f375015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500cda80 0x7f375015ed90 secure :-1 s=READY pgs=2691 cs=0 l=1 rev1=1 crypto rx=0x7f3748009e50 tx=0x7f374802bf60 comp rx=0 tx=0).stop 2023-10-14T18:17:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 -- 172.21.15.29:0/111276589 shutdown_connections 2023-10-14T18:17:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500cda80 0x7f375015ed90 unknown :-1 s=CLOSED pgs=2691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 --2- 172.21.15.29:0/111276589 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f375006c040 0x7f375006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 -- 172.21.15.29:0/111276589 >> 172.21.15.29:0/111276589 conn(0x7f37500ce6a0 msgr2=0x7f37500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.231+0000 7f3755747640 1 -- 172.21.15.29:0/111276589 shutdown_connections 2023-10-14T18:17:58.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.232+0000 7f3755747640 1 -- 172.21.15.29:0/111276589 wait complete. 2023-10-14T18:17:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.233+0000 7f3755747640 1 Processor -- start 2023-10-14T18:17:58.238 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.233+0000 7f3755747640 1 -- start start 2023-10-14T18:17:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f375006c040 0x7f3750153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.239 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37500cda80 0x7f37500a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500ac160 con 0x7f37500ad970 2023-10-14T18:17:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37500abf60 con 0x7f375006c040 2023-10-14T18:17:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.234+0000 7f3755747640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37500a8b80 con 0x7f37500cda80 2023-10-14T18:17:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34136/0 (socket says 172.21.15.29:34136) 2023-10-14T18:17:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 -- 172.21.15.29:0/3116939488 learned_addr learned my addr 172.21.15.29:0/3116939488 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f3754f46640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37500cda80 0x7f37500a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374ffff640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f375006c040 0x7f3750153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.241 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 -- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37500cda80 msgr2=0x7f37500a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37500cda80 0x7f37500a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 -- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f375006c040 msgr2=0x7f3750153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f375006c040 0x7f3750153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.242 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.235+0000 7f374f7fe640 1 -- 172.21.15.29:0/3116939488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3748009b30 con 0x7f37500ad970 2023-10-14T18:17:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f374f7fe640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f374400c400 tx=0x7f374400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37440052d0 con 0x7f37500ad970 2023-10-14T18:17:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744005430 con 0x7f37500ad970 2023-10-14T18:17:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3744003830 con 0x7f37500ad970 2023-10-14T18:17:58.243 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37500a8e60 con 0x7f37500ad970 2023-10-14T18:17:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.236+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37500a9420 con 0x7f37500ad970 2023-10-14T18:17:58.244 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.237+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f375006b330 con 0x7f37500ad970 2023-10-14T18:17:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.239+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 141) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f3744003990 con 0x7f37500ad970 2023-10-14T18:17:58.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.239+0000 7f374cff9640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 0x7f373c06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.240+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f3744089520 con 0x7f37500ad970 2023-10-14T18:17:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.240+0000 7f374ffff640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 0x7f373c06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.241+0000 7f374ffff640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 0x7f373c06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3740006fd0 tx=0x7f3740008040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.244+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3744053d70 con 0x7f37500ad970 2023-10-14T18:17:58.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.466+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f37500cc550 con 0x7f373c06b100 2023-10-14T18:17:58.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.482+0000 7f374cff9640 1 -- 172.21.15.29:0/3116939488 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+353 (secure 0 0 0) 0x7f374000f860 con 0x7f373c06b100 2023-10-14T18:17:58.483 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:17:58.483 INFO:teuthology.orchestra.run.smithi029.stdout:[{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T18:17:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-14T18:17:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 msgr2=0x7f373c06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 0x7f373c06d5c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3740006fd0 tx=0x7f3740008040 comp rx=0 tx=0).stop 2023-10-14T18:17:58.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 msgr2=0x7f3750156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 secure :-1 s=READY pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f374400c400 tx=0x7f374400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 shutdown_connections 2023-10-14T18:17:58.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f373c06b100 0x7f373c06d5c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37500cda80 0x7f37500a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f37500ad970 0x7f3750156c90 secure :-1 s=CLOSED pgs=2692 cs=0 l=1 rev1=1 crypto rx=0x7f374400c400 tx=0x7f374400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 --2- 172.21.15.29:0/3116939488 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f375006c040 0x7f3750153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 >> 172.21.15.29:0/3116939488 conn(0x7f37500ce6a0 msgr2=0x7f375015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:58.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.485+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 shutdown_connections 2023-10-14T18:17:58.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.486+0000 7f3755747640 1 -- 172.21.15.29:0/3116939488 wait complete. 2023-10-14T18:17:58.506 DEBUG:tasks.cephfs.test_snap_schedules:res= [{"fs": "backup_fs", "subvol": null, "group": null, "path": "mandatory_fs_argument_test_dir", "rel_path": "mandatory_fs_argument_test_dir", "schedule": "1M", "retention": {"M": 1}, "start": "2023-10-14T00:00:00", "created": "2023-10-14T18:17:55", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true}] 2023-10-14T18:17:58.507 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:17:58.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.741+0000 7f7510947640 1 Processor -- start 2023-10-14T18:17:58.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.741+0000 7f7510947640 1 -- start start 2023-10-14T18:17:58.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.742+0000 7f7510947640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f750c070a90 0x7f750c0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.742+0000 7f7510947640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c15d0e0 con 0x7f750c15cd00 2023-10-14T18:17:58.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.742+0000 7f7510947640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f750c0cbd80 con 0x7f750c0cb000 2023-10-14T18:17:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.742+0000 7f7510947640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f750c0cbaa0 con 0x7f750c070a90 2023-10-14T18:17:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f750c070a90 0x7f750c0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f750c15cd00 0x7f750c0cdfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46184/0 (socket says 172.21.15.29:46184) 2023-10-14T18:17:58.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750b7fe640 1 -- 172.21.15.29:0/1323234777 learned_addr learned my addr 172.21.15.29:0/1323234777 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:17:58.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750bfff640 1 -- 172.21.15.29:0/1323234777 >> v1:172.21.15.186:6789/0 conn(0x7f750c0cb000 legacy=0x7f750c070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750bfff640 1 -- 172.21.15.29:0/1323234777 >> v1:172.21.15.29:6789/0 conn(0x7f750c15cd00 legacy=0x7f750c0cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750a7fc640 1 -- 172.21.15.29:0/1323234777 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593854266 0 0) 0x7f750c15d0e0 con 0x7f750c15cd00 2023-10-14T18:17:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.743+0000 7f750bfff640 1 -- 172.21.15.29:0/1323234777 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750c0cd500 con 0x7f750c070a90 2023-10-14T18:17:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.744+0000 7f750bfff640 1 --2- 172.21.15.29:0/1323234777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c070a90 0x7f750c0cb490 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f75040097a0 tx=0x7f750402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8052bb5d6dcd0f0d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f750a7fc640 1 -- 172.21.15.29:0/1323234777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504031830 con 0x7f750c070a90 2023-10-14T18:17:58.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f750a7fc640 1 -- 172.21.15.29:0/1323234777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7504031990 con 0x7f750c070a90 2023-10-14T18:17:58.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f750a7fc640 1 -- 172.21.15.29:0/1323234777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7504031d20 con 0x7f750c070a90 2023-10-14T18:17:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 -- 172.21.15.29:0/1323234777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c070a90 msgr2=0x7f750c0cb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 --2- 172.21.15.29:0/1323234777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c070a90 0x7f750c0cb490 secure :-1 s=READY pgs=2694 cs=0 l=1 rev1=1 crypto rx=0x7f75040097a0 tx=0x7f750402b8d0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 -- 172.21.15.29:0/1323234777 shutdown_connections 2023-10-14T18:17:58.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 --2- 172.21.15.29:0/1323234777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c070a90 0x7f750c0cb490 unknown :-1 s=CLOSED pgs=2694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 -- 172.21.15.29:0/1323234777 >> 172.21.15.29:0/1323234777 conn(0x7f750c0cf790 msgr2=0x7f750c0cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 -- 172.21.15.29:0/1323234777 shutdown_connections 2023-10-14T18:17:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.745+0000 7f7510947640 1 -- 172.21.15.29:0/1323234777 wait complete. 2023-10-14T18:17:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.747+0000 7f7510947640 1 Processor -- start 2023-10-14T18:17:58.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.747+0000 7f7510947640 1 -- start start 2023-10-14T18:17:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f750c070a90 0x7f750c0b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f750c0cb000 0x7f750c0b4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c14d120 con 0x7f750c15cd00 2023-10-14T18:17:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b3eb0 con 0x7f750c070a90 2023-10-14T18:17:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f7510947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f750c0b4190 con 0x7f750c0cb000 2023-10-14T18:17:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f750bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.748+0000 7f750bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34152/0 (socket says 172.21.15.29:34152) 2023-10-14T18:17:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 -- 172.21.15.29:0/2655860325 learned_addr learned my addr 172.21.15.29:0/2655860325 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 -- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f750c0cb000 msgr2=0x7f750c0b4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f750c0cb000 0x7f750c0b4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 -- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f750c070a90 msgr2=0x7f750c0b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f750c070a90 0x7f750c0b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 -- 172.21.15.29:0/2655860325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7504009400 con 0x7f750c15cd00 2023-10-14T18:17:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.749+0000 7f750bfff640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f75040098c0 tx=0x7f75040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.750+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f750403b9f0 con 0x7f750c15cd00 2023-10-14T18:17:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.750+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f750403bb50 con 0x7f750c15cd00 2023-10-14T18:17:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.750+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75040453f0 con 0x7f750c15cd00 2023-10-14T18:17:58.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.750+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f750c0b4470 con 0x7f750c15cd00 2023-10-14T18:17:58.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.750+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f750c0b4a30 con 0x7f750c15cd00 2023-10-14T18:17:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.751+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f750c0b4c20 con 0x7f750c15cd00 2023-10-14T18:17:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.752+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 141) v1 ==== 89451+0+0 (secure 0 0 0) 0x7f7504045550 con 0x7f750c15cd00 2023-10-14T18:17:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.753+0000 7f7503fff640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 0x7f74ec06d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.753+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f75040ac5a0 con 0x7f750c15cd00 2023-10-14T18:17:58.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.753+0000 7f750b7fe640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 0x7f74ec06d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:58.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.754+0000 7f750b7fe640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 0x7f74ec06d5c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74fc007fd0 tx=0x7f74fc007ce0 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.758+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7504076d30 con 0x7f750c15cd00 2023-10-14T18:17:58.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.953+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 142) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f7504077100 con 0x7f750c15cd00 2023-10-14T18:17:58.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:58.993+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "retention_spec_or_period": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f750c0cdac0 con 0x7f74ec06b100 2023-10-14T18:17:59.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.171+0000 7f7503fff640 1 -- 172.21.15.29:0/2655860325 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+59 (secure 0 0 0) 0x7f74fc004b60 con 0x7f74ec06b100 2023-10-14T18:17:59.173 INFO:teuthology.orchestra.run.smithi029.stdout:Retention removed from path mandatory_fs_argument_test_dir 2023-10-14T18:17:59.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 msgr2=0x7f74ec06d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 0x7f74ec06d5c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f74fc007fd0 tx=0x7f74fc007ce0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 msgr2=0x7f750c14cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 secure :-1 s=READY pgs=2695 cs=0 l=1 rev1=1 crypto rx=0x7f75040098c0 tx=0x7f75040046c0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 shutdown_connections 2023-10-14T18:17:59.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f74ec06b100 0x7f74ec06d5c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f750c15cd00 0x7f750c14cbe0 unknown :-1 s=CLOSED pgs=2695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f750c0cb000 0x7f750c0b4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 --2- 172.21.15.29:0/2655860325 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f750c070a90 0x7f750c0b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 >> 172.21.15.29:0/2655860325 conn(0x7f750c0cf790 msgr2=0x7f750c070280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 shutdown_connections 2023-10-14T18:17:59.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.175+0000 7f7510947640 1 -- 172.21.15.29:0/2655860325 wait complete. 2023-10-14T18:17:59.196 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention removed from path mandatory_fs_argument_test_dir 2023-10-14T18:17:59.197 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule deactivate mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:17:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.417+0000 7f0539d7c640 1 Processor -- start 2023-10-14T18:17:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.417+0000 7f0539d7c640 1 -- start start 2023-10-14T18:17:59.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.418+0000 7f0539d7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b8a10 0x7f05340d2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.418+0000 7f0539d7c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05340b4750 0x7f05340b4be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.418+0000 7f0539d7c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05340d1420 con 0x7f053415cf50 2023-10-14T18:17:59.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.418+0000 7f0539d7c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05340b5120 con 0x7f05340b4750 2023-10-14T18:17:59.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.418+0000 7f0539d7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f053415d330 con 0x7f05340b8a10 2023-10-14T18:17:59.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05340b4750 0x7f05340b4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:59.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f05340b4750 0x7f05340b4be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34158/0 (socket says 172.21.15.29:34158) 2023-10-14T18:17:59.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 -- 172.21.15.29:0/1113293804 learned_addr learned my addr 172.21.15.29:0/1113293804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:59.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f052affd640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b8a10 0x7f05340d2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:59.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 -- 172.21.15.29:0/1113293804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b8a10 msgr2=0x7f05340d2720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:17:59.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b8a10 0x7f05340d2720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.419+0000 7f0533fff640 1 -- 172.21.15.29:0/1113293804 >> v1:172.21.15.186:6789/0 conn(0x7f053415cf50 legacy=0x7f05340d0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.420+0000 7f0533fff640 1 -- 172.21.15.29:0/1113293804 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05340d1680 con 0x7f05340b4750 2023-10-14T18:17:59.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.421+0000 7f0533fff640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b4750 0x7f05340b4be0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f052c009e50 tx=0x7f052c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14e8c4d4a801fa06 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:59.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.421+0000 7f0532ffd640 1 -- 172.21.15.29:0/1113293804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c02cb00 con 0x7f05340b4750 2023-10-14T18:17:59.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.421+0000 7f0532ffd640 1 -- 172.21.15.29:0/1113293804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f052c02cc60 con 0x7f05340b4750 2023-10-14T18:17:59.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.421+0000 7f0532ffd640 1 -- 172.21.15.29:0/1113293804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c0343f0 con 0x7f05340b4750 2023-10-14T18:17:59.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 -- 172.21.15.29:0/1113293804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b4750 msgr2=0x7f05340b4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b4750 0x7f05340b4be0 secure :-1 s=READY pgs=2696 cs=0 l=1 rev1=1 crypto rx=0x7f052c009e50 tx=0x7f052c02bf60 comp rx=0 tx=0).stop 2023-10-14T18:17:59.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 -- 172.21.15.29:0/1113293804 shutdown_connections 2023-10-14T18:17:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b4750 0x7f05340b4be0 unknown :-1 s=CLOSED pgs=2696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1113293804 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b8a10 0x7f05340d2720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 -- 172.21.15.29:0/1113293804 >> 172.21.15.29:0/1113293804 conn(0x7f05340cf750 msgr2=0x7f05340cfb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 -- 172.21.15.29:0/1113293804 shutdown_connections 2023-10-14T18:17:59.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.422+0000 7f0539d7c640 1 -- 172.21.15.29:0/1113293804 wait complete. 2023-10-14T18:17:59.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.424+0000 7f0539d7c640 1 Processor -- start 2023-10-14T18:17:59.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.424+0000 7f0539d7c640 1 -- start start 2023-10-14T18:17:59.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b4750 0x7f053414a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 0x7f05340b78f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05340c9f40 con 0x7f05340b8a10 2023-10-14T18:17:59.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053415eee0 con 0x7f053415cf50 2023-10-14T18:17:59.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0539d7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05340cacc0 con 0x7f05340b4750 2023-10-14T18:17:59.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f052affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:59.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f052affd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34172/0 (socket says 172.21.15.29:34172) 2023-10-14T18:17:59.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f052affd640 1 -- 172.21.15.29:0/1960869364 learned_addr learned my addr 172.21.15.29:0/1960869364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:17:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f0533fff640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 0x7f05340b78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f052affd640 1 -- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b4750 msgr2=0x7f053414a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.425+0000 7f052affd640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b4750 0x7f053414a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f052affd640 1 -- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 msgr2=0x7f05340b78f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f052affd640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 0x7f05340b78f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f052affd640 1 -- 172.21.15.29:0/1960869364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c009b30 con 0x7f05340b8a10 2023-10-14T18:17:59.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0533fff640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 0x7f05340b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:17:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f052affd640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f051c00d800 tx=0x7f051c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c00b6c0 con 0x7f05340b8a10 2023-10-14T18:17:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f051c00b820 con 0x7f05340b8a10 2023-10-14T18:17:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f051c00bbb0 con 0x7f05340b8a10 2023-10-14T18:17:59.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05340ca1e0 con 0x7f05340b8a10 2023-10-14T18:17:59.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.426+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05340ca7a0 con 0x7f05340b8a10 2023-10-14T18:17:59.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.427+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05340ca930 con 0x7f05340b8a10 2023-10-14T18:17:59.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.428+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f051c00bd10 con 0x7f05340b8a10 2023-10-14T18:17:59.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.429+0000 7f0530ff9640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 0x7f051406d680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:17:59.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.430+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f051c08a530 con 0x7f05340b8a10 2023-10-14T18:17:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.430+0000 7f05337fe640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 0x7f051406d680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:17:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.431+0000 7f05337fe640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 0x7f051406d680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0524000f60 tx=0x7f05240079b0 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:17:59.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.434+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f051c054d60 con 0x7f05340b8a10 2023-10-14T18:17:59.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.654+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule deactivate", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f05340d1960 con 0x7f051406b1c0 2023-10-14T18:17:59.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.828+0000 7f0530ff9640 1 -- 172.21.15.29:0/1960869364 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+61 (secure 0 0 0) 0x7f0524005080 con 0x7f051406b1c0 2023-10-14T18:17:59.830 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-14T18:17:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 msgr2=0x7f051406d680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 0x7f051406d680 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f0524000f60 tx=0x7f05240079b0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 msgr2=0x7f05340b73b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:17:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 secure :-1 s=READY pgs=2697 cs=0 l=1 rev1=1 crypto rx=0x7f051c00d800 tx=0x7f051c00dcc0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 shutdown_connections 2023-10-14T18:17:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f051406b1c0 0x7f051406d680 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f053415cf50 0x7f05340b78f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f05340b8a10 0x7f05340b73b0 unknown :-1 s=CLOSED pgs=2697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 --2- 172.21.15.29:0/1960869364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f05340b4750 0x7f053414a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:17:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 >> 172.21.15.29:0/1960869364 conn(0x7f05340cf750 msgr2=0x7f053415be80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:17:59.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.832+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 shutdown_connections 2023-10-14T18:17:59.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:17:59.833+0000 7f0539d7c640 1 -- 172.21.15.29:0/1960869364 wait complete. 2023-10-14T18:17:59.854 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule deactivated for path mandatory_fs_argument_test_dir 2023-10-14T18:17:59.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove mandatory_fs_argument_test_dir --fs backup_fs 1M 2023-10-14T18:18:00.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.090+0000 7f2597fce640 1 Processor -- start 2023-10-14T18:18:00.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.090+0000 7f2597fce640 1 -- start start 2023-10-14T18:18:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2597fce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25900b0360 0x7f25900b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2597fce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25900ca510 0x7f25900ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2597fce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f259015b310 con 0x7f259015ac10 2023-10-14T18:18:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2597fce640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f259015f800 con 0x7f25900b0360 2023-10-14T18:18:00.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2597fce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25900b0d30 con 0x7f25900ca510 2023-10-14T18:18:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.092+0000 7f2595569640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25900b0360 0x7f25900b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f25900b0360 0x7f25900b07f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34178/0 (socket says 172.21.15.29:34178) 2023-10-14T18:18:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 -- 172.21.15.29:0/1072989956 learned_addr learned my addr 172.21.15.29:0/1072989956 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f259656b640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25900ca510 0x7f25900ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 -- 172.21.15.29:0/1072989956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25900ca510 msgr2=0x7f25900ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:00.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25900ca510 0x7f25900ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 -- 172.21.15.29:0/1072989956 >> v1:172.21.15.186:6789/0 conn(0x7f259015ac10 legacy=0x7f259006c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.093+0000 7f2595569640 1 -- 172.21.15.29:0/1072989956 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f259006f640 con 0x7f25900b0360 2023-10-14T18:18:00.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.094+0000 7f2595569640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900b0360 0x7f25900b07f0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f25800098a0 tx=0x7f258002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec538ccf9138a707 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.095+0000 7f2594d68640 1 -- 172.21.15.29:0/1072989956 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580031640 con 0x7f25900b0360 2023-10-14T18:18:00.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.095+0000 7f2594d68640 1 -- 172.21.15.29:0/1072989956 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25800317a0 con 0x7f25900b0360 2023-10-14T18:18:00.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.095+0000 7f2594d68640 1 -- 172.21.15.29:0/1072989956 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580031b30 con 0x7f25900b0360 2023-10-14T18:18:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.095+0000 7f2597fce640 1 -- 172.21.15.29:0/1072989956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900b0360 msgr2=0x7f25900b07f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.095+0000 7f2597fce640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900b0360 0x7f25900b07f0 secure :-1 s=READY pgs=2698 cs=0 l=1 rev1=1 crypto rx=0x7f25800098a0 tx=0x7f258002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 -- 172.21.15.29:0/1072989956 shutdown_connections 2023-10-14T18:18:00.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25900ca510 0x7f25900ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 --2- 172.21.15.29:0/1072989956 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900b0360 0x7f25900b07f0 unknown :-1 s=CLOSED pgs=2698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 -- 172.21.15.29:0/1072989956 >> 172.21.15.29:0/1072989956 conn(0x7f25900cb360 msgr2=0x7f25900cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:00.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 -- 172.21.15.29:0/1072989956 shutdown_connections 2023-10-14T18:18:00.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.096+0000 7f2597fce640 1 -- 172.21.15.29:0/1072989956 wait complete. 2023-10-14T18:18:00.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.098+0000 7f2597fce640 1 Processor -- start 2023-10-14T18:18:00.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.099+0000 7f2597fce640 1 -- start start 2023-10-14T18:18:00.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25900b0360 0x7f25900c59b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f259015ac10 0x7f25901efe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900c2bb0 con 0x7f25900ca510 2023-10-14T18:18:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25900c2d40 con 0x7f25900b0360 2023-10-14T18:18:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2597fce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25900b2ef0 con 0x7f259015ac10 2023-10-14T18:18:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2595569640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2595569640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34184/0 (socket says 172.21.15.29:34184) 2023-10-14T18:18:00.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.100+0000 7f2595569640 1 -- 172.21.15.29:0/365896176 learned_addr learned my addr 172.21.15.29:0/365896176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:00.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595d6a640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25900b0360 0x7f25900c59b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f259656b640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f259015ac10 0x7f25901efe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 -- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f259015ac10 msgr2=0x7f25901efe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f259015ac10 0x7f25901efe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 -- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25900b0360 msgr2=0x7f25900c59b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25900b0360 0x7f25900c59b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 -- 172.21.15.29:0/365896176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25800094d0 con 0x7f25900ca510 2023-10-14T18:18:00.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.101+0000 7f2595569640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f25800098a0 tx=0x7f25800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.102+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f258003ac60 con 0x7f25900ca510 2023-10-14T18:18:00.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.102+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258003adc0 con 0x7f25900ca510 2023-10-14T18:18:00.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.102+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2580044630 con 0x7f25900ca510 2023-10-14T18:18:00.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.102+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25900c6040 con 0x7f25900ca510 2023-10-14T18:18:00.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.102+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25900c6600 con 0x7f25900ca510 2023-10-14T18:18:00.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.103+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f259004e160 con 0x7f25900ca510 2023-10-14T18:18:00.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.104+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f258002f050 con 0x7f25900ca510 2023-10-14T18:18:00.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.105+0000 7f25867fc640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 0x7f257406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.105+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f25800abf60 con 0x7f25900ca510 2023-10-14T18:18:00.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.105+0000 7f2595d6a640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 0x7f257406d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.106+0000 7f2595d6a640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 0x7f257406d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f258c000f60 tx=0x7f258c009040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.110+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2580076680 con 0x7f25900ca510 2023-10-14T18:18:00.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.333+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 --> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "mandatory_fs_argument_test_dir", "fs": "backup_fs", "repeat": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f259015b890 con 0x7f257406b190 2023-10-14T18:18:00.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.527+0000 7f25867fc640 1 -- 172.21.15.29:0/365896176 <== mgr.11568 v2:172.21.15.29:6832/2804419857 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+57 (secure 0 0 0) 0x7f258c001360 con 0x7f257406b190 2023-10-14T18:18:00.529 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path mandatory_fs_argument_test_dir 2023-10-14T18:18:00.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.530+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 msgr2=0x7f257406d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.530+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 0x7f257406d650 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f258c000f60 tx=0x7f258c009040 comp rx=0 tx=0).stop 2023-10-14T18:18:00.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.530+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 msgr2=0x7f25900b29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.530+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 secure :-1 s=READY pgs=2699 cs=0 l=1 rev1=1 crypto rx=0x7f25800098a0 tx=0x7f25800046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 shutdown_connections 2023-10-14T18:18:00.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f257406b190 0x7f257406d650 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f259015ac10 0x7f25901efe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25900ca510 0x7f25900b29b0 unknown :-1 s=CLOSED pgs=2699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 --2- 172.21.15.29:0/365896176 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25900b0360 0x7f25900c59b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 >> 172.21.15.29:0/365896176 conn(0x7f25900cb360 msgr2=0x7f25900ce850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:00.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 shutdown_connections 2023-10-14T18:18:00.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.531+0000 7f2597fce640 1 -- 172.21.15.29:0/365896176 wait complete. 2023-10-14T18:18:00.555 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path mandatory_fs_argument_test_dir 2023-10-14T18:18:00.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:00.556 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-14T18:18:00.615 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T18:18:00.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.842+0000 7f09a7fff640 1 Processor -- start 2023-10-14T18:18:00.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.842+0000 7f09a7fff640 1 -- start start 2023-10-14T18:18:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f09a8159950 0x7f09a815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f09a8070450 0x7f09a80708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a7fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09a815aef0 con 0x7f09a806c100 2023-10-14T18:18:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a8070e20 con 0x7f09a8070450 2023-10-14T18:18:00.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a7fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09a815e490 con 0x7f09a8159950 2023-10-14T18:18:00.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f09a806c100 0x7f09a806c510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:46194/0 (socket says 172.21.15.29:46194) 2023-10-14T18:18:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.844+0000 7f09a77fe640 1 -- 172.21.15.29:0/1975846340 learned_addr learned my addr 172.21.15.29:0/1975846340 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a6ffd640 1 --2- 172.21.15.29:0/1975846340 >> v2:172.21.15.186:3300/0 conn(0x7f09a8159950 0x7f09a815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a67fc640 1 --2- 172.21.15.29:0/1975846340 >> v2:172.21.15.29:3300/0 conn(0x7f09a8070450 0x7f09a80708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a5ffb640 1 -- 172.21.15.29:0/1975846340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833455029 0 0) 0x7f09a0001490 con 0x7f09a806c100 2023-10-14T18:18:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a5ffb640 1 -- 172.21.15.29:0/1975846340 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0990003520 con 0x7f09a806c100 2023-10-14T18:18:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a67fc640 1 -- 172.21.15.29:0/1975846340 >> v2:172.21.15.186:3300/0 conn(0x7f09a8159950 msgr2=0x7f09a815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a67fc640 1 --2- 172.21.15.29:0/1975846340 >> v2:172.21.15.186:3300/0 conn(0x7f09a8159950 0x7f09a815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a67fc640 1 -- 172.21.15.29:0/1975846340 >> v1:172.21.15.29:6789/0 conn(0x7f09a806c100 legacy=0x7f09a806c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.845+0000 7f09a67fc640 1 -- 172.21.15.29:0/1975846340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a815d910 con 0x7f09a8070450 2023-10-14T18:18:00.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.847+0000 7f09a67fc640 1 --2- 172.21.15.29:0/1975846340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80708e0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f09a80ca950 tx=0x7f099c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd07f17455198c1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.847+0000 7f09a5ffb640 1 -- 172.21.15.29:0/1975846340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c02e070 con 0x7f09a8070450 2023-10-14T18:18:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.847+0000 7f09a5ffb640 1 -- 172.21.15.29:0/1975846340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c0315a0 con 0x7f09a8070450 2023-10-14T18:18:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.847+0000 7f09a5ffb640 1 -- 172.21.15.29:0/1975846340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c031930 con 0x7f09a8070450 2023-10-14T18:18:00.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 -- 172.21.15.29:0/1975846340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 msgr2=0x7f09a80708e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 --2- 172.21.15.29:0/1975846340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80708e0 secure :-1 s=READY pgs=2701 cs=0 l=1 rev1=1 crypto rx=0x7f09a80ca950 tx=0x7f099c02b840 comp rx=0 tx=0).stop 2023-10-14T18:18:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 -- 172.21.15.29:0/1975846340 shutdown_connections 2023-10-14T18:18:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 --2- 172.21.15.29:0/1975846340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80708e0 unknown :-1 s=CLOSED pgs=2701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 --2- 172.21.15.29:0/1975846340 >> v2:172.21.15.186:3300/0 conn(0x7f09a8159950 0x7f09a815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 -- 172.21.15.29:0/1975846340 >> 172.21.15.29:0/1975846340 conn(0x7f09a80ce6a0 msgr2=0x7f09a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 -- 172.21.15.29:0/1975846340 shutdown_connections 2023-10-14T18:18:00.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.848+0000 7f09a7fff640 1 -- 172.21.15.29:0/1975846340 wait complete. 2023-10-14T18:18:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.850+0000 7f09a7fff640 1 Processor -- start 2023-10-14T18:18:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.850+0000 7f09a7fff640 1 -- start start 2023-10-14T18:18:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a806c100 0x7f09a814f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a8159950 0x7f09a806ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a80c21b0 con 0x7f09a8070450 2023-10-14T18:18:00.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8156050 con 0x7f09a806c100 2023-10-14T18:18:00.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09a8156330 con 0x7f09a8159950 2023-10-14T18:18:00.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34206/0 (socket says 172.21.15.29:34206) 2023-10-14T18:18:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 -- 172.21.15.29:0/2264177731 learned_addr learned my addr 172.21.15.29:0/2264177731 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 -- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a8159950 msgr2=0x7f09a806ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a8159950 0x7f09a806ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 -- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a806c100 msgr2=0x7f09a814f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:00.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a6ffd640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a806c100 0x7f09a814f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a806c100 0x7f09a814f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.851+0000 7f09a67fc640 1 -- 172.21.15.29:0/2264177731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f099c009400 con 0x7f09a8070450 2023-10-14T18:18:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f09a67fc640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f099c009890 tx=0x7f099c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c032630 con 0x7f09a8070450 2023-10-14T18:18:00.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f099c032790 con 0x7f09a8070450 2023-10-14T18:18:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f099c032b40 con 0x7f09a8070450 2023-10-14T18:18:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09a8156610 con 0x7f09a8070450 2023-10-14T18:18:00.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.852+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09a814f640 con 0x7f09a8070450 2023-10-14T18:18:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.853+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09a804e9b0 con 0x7f09a8070450 2023-10-14T18:18:00.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.854+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 142) v1 ==== 89507+0+0 (secure 0 0 0) 0x7f099c03b3f0 con 0x7f09a8070450 2023-10-14T18:18:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.855+0000 7f098f7fe640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 0x7f099406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.855+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f099c0ac1f0 con 0x7f09a8070450 2023-10-14T18:18:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.856+0000 7f09a6ffd640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 0x7f099406d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:00.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.857+0000 7f09a6ffd640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 0x7f099406d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0998009970 tx=0x7f0998008040 comp rx=0 tx=0).ready entity=mgr.11568 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:00.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:00.860+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f099c0768d0 con 0x7f09a8070450 2023-10-14T18:18:01.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.114+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f09a80b4620 con 0x7f09a8070450 2023-10-14T18:18:01.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.972+0000 7f098f7fe640 1 -- 172.21.15.29:0/2264177731 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v143) v1 ==== 93+0+0 (secure 0 0 0) 0x7f099c079ef0 con 0x7f09a8070450 2023-10-14T18:18:01.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.975+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 msgr2=0x7f099406d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.975+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 0x7f099406d650 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0998009970 tx=0x7f0998008040 comp rx=0 tx=0).stop 2023-10-14T18:18:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.975+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 msgr2=0x7f09a80c84c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:01.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.975+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 secure :-1 s=READY pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f099c009890 tx=0x7f099c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 shutdown_connections 2023-10-14T18:18:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f099406b190 0x7f099406d650 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f09a8159950 0x7f09a806ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:01.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f09a8070450 0x7f09a80c84c0 secure :-1 s=CLOSED pgs=2702 cs=0 l=1 rev1=1 crypto rx=0x7f099c009890 tx=0x7f099c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 --2- 172.21.15.29:0/2264177731 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f09a806c100 0x7f09a814f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.976+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 >> 172.21.15.29:0/2264177731 conn(0x7f09a80ce6a0 msgr2=0x7f09a8072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:01.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.977+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 shutdown_connections 2023-10-14T18:18:01.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:01.977+0000 7f09a7fff640 1 -- 172.21.15.29:0/2264177731 wait complete. 2023-10-14T18:18:02.005 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:18:02.005 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:18:02.119 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:18:02.216 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:18:02.217 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:02.217 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:02.271 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:02.272 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:02.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:02.524 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:02.524+0000 7fe61bc12040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:18:02.528 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:02.527+0000 7f6620eda040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:18:02.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.528+0000 7fb64f38a640 1 Processor -- start 2023-10-14T18:18:02.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.528+0000 7fb64f38a640 1 -- start start 2023-10-14T18:18:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64f38a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6480cb730 0x7fb6480cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64f38a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480ce7f0 0x7fb6480cec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64f38a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6480cc940 con 0x7fb6480cd510 2023-10-14T18:18:02.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64f38a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6480d1560 con 0x7fb6480cb730 2023-10-14T18:18:02.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64f38a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb6480cde30 con 0x7fb6480ce7f0 2023-10-14T18:18:02.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64c925640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6480cb730 0x7fb6480cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64c925640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb6480cb730 0x7fb6480cd8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33732/0 (socket says 172.21.15.29:33732) 2023-10-14T18:18:02.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64c925640 1 -- 172.21.15.29:0/2277573524 learned_addr learned my addr 172.21.15.29:0/2277573524 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:02.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.530+0000 7fb64d927640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480ce7f0 0x7fb6480cec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.531+0000 7fb64c925640 1 -- 172.21.15.29:0/2277573524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480ce7f0 msgr2=0x7fb6480cec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.531+0000 7fb64c925640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480ce7f0 0x7fb6480cec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.531+0000 7fb64c925640 1 -- 172.21.15.29:0/2277573524 >> v1:172.21.15.186:6789/0 conn(0x7fb6480cd510 legacy=0x7fb64815f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.531+0000 7fb64c925640 1 -- 172.21.15.29:0/2277573524 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6480cbd30 con 0x7fb6480cb730 2023-10-14T18:18:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.532+0000 7fb64c925640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cb730 0x7fb6480cd8f0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fb638009870 tx=0x7fb63802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4df0dac4c7cb49e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:02.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.532+0000 7fb63ffff640 1 -- 172.21.15.29:0/2277573524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638031640 con 0x7fb6480cb730 2023-10-14T18:18:02.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.532+0000 7fb63ffff640 1 -- 172.21.15.29:0/2277573524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6380317a0 con 0x7fb6480cb730 2023-10-14T18:18:02.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.532+0000 7fb63ffff640 1 -- 172.21.15.29:0/2277573524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638031b30 con 0x7fb6480cb730 2023-10-14T18:18:02.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 -- 172.21.15.29:0/2277573524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cb730 msgr2=0x7fb6480cd8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cb730 0x7fb6480cd8f0 secure :-1 s=READY pgs=2706 cs=0 l=1 rev1=1 crypto rx=0x7fb638009870 tx=0x7fb63802b8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 -- 172.21.15.29:0/2277573524 shutdown_connections 2023-10-14T18:18:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480ce7f0 0x7fb6480cec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 --2- 172.21.15.29:0/2277573524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cb730 0x7fb6480cd8f0 unknown :-1 s=CLOSED pgs=2706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 -- 172.21.15.29:0/2277573524 >> 172.21.15.29:0/2277573524 conn(0x7fb648099ea0 msgr2=0x7fb6480d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:02.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.533+0000 7fb64f38a640 1 -- 172.21.15.29:0/2277573524 shutdown_connections 2023-10-14T18:18:02.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.534+0000 7fb64f38a640 1 -- 172.21.15.29:0/2277573524 wait complete. 2023-10-14T18:18:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.535+0000 7fb64f38a640 1 Processor -- start 2023-10-14T18:18:02.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.535+0000 7fb64f38a640 1 -- start start 2023-10-14T18:18:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 0x7fb64815d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 0x7fb64815b6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64815cad0 con 0x7fb6480cd510 2023-10-14T18:18:02.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb64815bc30 con 0x7fb6480ce7f0 2023-10-14T18:18:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.536+0000 7fb64f38a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb64815bec0 con 0x7fb6480cb730 2023-10-14T18:18:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33740/0 (socket says 172.21.15.29:33740) 2023-10-14T18:18:02.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 -- 172.21.15.29:0/4061461331 learned_addr learned my addr 172.21.15.29:0/4061461331 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64d126640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 0x7fb64815d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64d927640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 0x7fb64815b6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 msgr2=0x7fb64815d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 0x7fb64815d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 msgr2=0x7fb64815b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 0x7fb64815b6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 -- 172.21.15.29:0/4061461331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6380094d0 con 0x7fb6480cd510 2023-10-14T18:18:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64d927640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 0x7fb64815b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:02.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64d126640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 0x7fb64815d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.537+0000 7fb64c925640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fb638009870 tx=0x7fb6380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.538+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb63803ac60 con 0x7fb6480cd510 2023-10-14T18:18:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.538+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb63803adc0 con 0x7fb6480cd510 2023-10-14T18:18:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.538+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb638044630 con 0x7fb6480cd510 2023-10-14T18:18:02.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.538+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb64815c1a0 con 0x7fb6480cd510 2023-10-14T18:18:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.538+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb64815c760 con 0x7fb6480cd510 2023-10-14T18:18:02.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.539+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb64804e160 con 0x7fb6480cd510 2023-10-14T18:18:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.540+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fb63802f050 con 0x7fb6480cd510 2023-10-14T18:18:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.541+0000 7fb63dffb640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 0x7fb62c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.541+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fb6380ac020 con 0x7fb6480cd510 2023-10-14T18:18:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.541+0000 7fb64d126640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 msgr2=0x7fb62c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.541+0000 7fb64d126640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 0x7fb62c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:02.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.545+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb638076700 con 0x7fb6480cd510 2023-10-14T18:18:02.582 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:02.582+0000 7fe61bc12040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:18:02.587 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:02.586+0000 7f6620eda040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.742+0000 7fb64d126640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 msgr2=0x7fb62c06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.742+0000 7fb64d126640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 0x7fb62c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:02.745 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.745+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6480d0ab0 con 0x7fb6480cd510 2023-10-14T18:18:02.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.747+0000 7fb63dffb640 1 -- 172.21.15.29:0/4061461331 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v234) v1 ==== 74+0+20981 (secure 0 0 0) 0x7fb638079d20 con 0x7fb6480cd510 2023-10-14T18:18:02.749 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:02.749 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":234,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:17:54.886245+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 msgr2=0x7fb62c06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 0x7fb62c06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 msgr2=0x7fb64815b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 secure :-1 s=READY pgs=2707 cs=0 l=1 rev1=1 crypto rx=0x7fb638009870 tx=0x7fb6380046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 shutdown_connections 2023-10-14T18:18:02.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fb62c06b140 0x7fb62c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb6480ce7f0 0x7fb64815b6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb6480cd510 0x7fb64815b1b0 unknown :-1 s=CLOSED pgs=2707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 --2- 172.21.15.29:0/4061461331 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb6480cb730 0x7fb64815d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.750+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 >> 172.21.15.29:0/4061461331 conn(0x7fb648099ea0 msgr2=0x7fb64806d780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.751+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 shutdown_connections 2023-10-14T18:18:02.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.751+0000 7fb64f38a640 1 -- 172.21.15.29:0/4061461331 wait complete. 2023-10-14T18:18:02.771 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:17:25.200413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:17:26.251106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:17:40.115972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:17:40.810362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:02.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.968+0000 7f149c8ce640 1 Processor -- start 2023-10-14T18:18:02.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.968+0000 7f149c8ce640 1 -- start start 2023-10-14T18:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149c8ce640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f14980b2080 0x7f14980d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149c8ce640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14980cfc40 0x7f14980b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149c8ce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14980d0970 con 0x7f14980b1ca0 2023-10-14T18:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149c8ce640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980d0ea0 con 0x7f14980cfc40 2023-10-14T18:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149c8ce640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14980d0bc0 con 0x7f14980b2080 2023-10-14T18:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.970+0000 7f149759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14980cfc40 0x7f14980b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f14980cfc40 0x7f14980b24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33754/0 (socket says 172.21.15.29:33754) 2023-10-14T18:18:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 -- 172.21.15.29:0/1409354446 learned_addr learned my addr 172.21.15.29:0/1409354446 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:02.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149659c640 1 --2- 172.21.15.29:0/1409354446 >> v2:172.21.15.186:3300/0 conn(0x7f14980b2080 0x7f14980d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 -- 172.21.15.29:0/1409354446 >> v2:172.21.15.186:3300/0 conn(0x7f14980b2080 msgr2=0x7f14980d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 --2- 172.21.15.29:0/1409354446 >> v2:172.21.15.186:3300/0 conn(0x7f14980b2080 0x7f14980d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 -- 172.21.15.29:0/1409354446 >> v1:172.21.15.186:6789/0 conn(0x7f14980b1ca0 legacy=0x7f14980ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.971+0000 7f149759e640 1 -- 172.21.15.29:0/1409354446 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14980cd940 con 0x7f14980cfc40 2023-10-14T18:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.972+0000 7f149759e640 1 --2- 172.21.15.29:0/1409354446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980cfc40 0x7f14980b24c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f14980671e0 tx=0x7f149002c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b17f4c5a72e124eb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:02.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f1495d9b640 1 -- 172.21.15.29:0/1409354446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f149002cbb0 con 0x7f14980cfc40 2023-10-14T18:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f1495d9b640 1 -- 172.21.15.29:0/1409354446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f149002cd10 con 0x7f14980cfc40 2023-10-14T18:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f1495d9b640 1 -- 172.21.15.29:0/1409354446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1490034610 con 0x7f14980cfc40 2023-10-14T18:18:02.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 -- 172.21.15.29:0/1409354446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980cfc40 msgr2=0x7f14980b24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1409354446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980cfc40 0x7f14980b24c0 secure :-1 s=READY pgs=2708 cs=0 l=1 rev1=1 crypto rx=0x7f14980671e0 tx=0x7f149002c010 comp rx=0 tx=0).stop 2023-10-14T18:18:02.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 -- 172.21.15.29:0/1409354446 shutdown_connections 2023-10-14T18:18:02.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1409354446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980cfc40 0x7f14980b24c0 unknown :-1 s=CLOSED pgs=2708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1409354446 >> v2:172.21.15.186:3300/0 conn(0x7f14980b2080 0x7f14980d36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.973+0000 7f149c8ce640 1 -- 172.21.15.29:0/1409354446 >> 172.21.15.29:0/1409354446 conn(0x7f14980d2b90 msgr2=0x7f14980d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:02.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.974+0000 7f149c8ce640 1 -- 172.21.15.29:0/1409354446 shutdown_connections 2023-10-14T18:18:02.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.974+0000 7f149c8ce640 1 -- 172.21.15.29:0/1409354446 wait complete. 2023-10-14T18:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.975+0000 7f149c8ce640 1 Processor -- start 2023-10-14T18:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.976+0000 7f149c8ce640 1 -- start start 2023-10-14T18:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980b2080 0x7f149815d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14980cfc40 0x7f149815e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14980cbec0 con 0x7f14980b1ca0 2023-10-14T18:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f149815d9f0 con 0x7f14980b2080 2023-10-14T18:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149c8ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14980cb5b0 con 0x7f14980cfc40 2023-10-14T18:18:02.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f1496d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f1496d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33768/0 (socket says 172.21.15.29:33768) 2023-10-14T18:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f1496d9d640 1 -- 172.21.15.29:0/1601210868 learned_addr learned my addr 172.21.15.29:0/1601210868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149759e640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14980cfc40 0x7f149815e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.977+0000 7f149659c640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980b2080 0x7f149815d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:02.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14980cfc40 msgr2=0x7f149815e3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14980cfc40 0x7f149815e3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980b2080 msgr2=0x7f149815d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980b2080 0x7f149815d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 -- 172.21.15.29:0/1601210868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1490009b30 con 0x7f14980b1ca0 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f1496d9d640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f148800c400 tx=0x7f148800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:02.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14880052d0 con 0x7f14980b1ca0 2023-10-14T18:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488005430 con 0x7f14980b1ca0 2023-10-14T18:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1488003830 con 0x7f14980b1ca0 2023-10-14T18:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f149815a980 con 0x7f14980b1ca0 2023-10-14T18:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.978+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14980cbae0 con 0x7f14980b1ca0 2023-10-14T18:18:02.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.980+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14980cc000 con 0x7f14980b1ca0 2023-10-14T18:18:02.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.981+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f1488003990 con 0x7f14980b1ca0 2023-10-14T18:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.982+0000 7f147f7fe640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 0x7f148406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.982+0000 7f149659c640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 msgr2=0x7f148406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.982+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f1488089490 con 0x7f14980b1ca0 2023-10-14T18:18:02.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.982+0000 7f149659c640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 0x7f148406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:02.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:02.987+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f14980cc000 con 0x7f14980b1ca0 2023-10-14T18:18:03.039 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.039+0000 7fe61bc12040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:18:03.069 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.068+0000 7f6620eda040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:18:03.095 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.095+0000 7fe61bc12040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:18:03.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.123+0000 7f6620eda040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:18:03.146 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.146+0000 7fe61bc12040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:18:03.176 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.175+0000 7f6620eda040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:18:03.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.182+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f14980cc000 con 0x7f14980b1ca0 2023-10-14T18:18:03.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.183+0000 7f149659c640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 msgr2=0x7f148406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:03.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.183+0000 7f149659c640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 0x7f148406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:03.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.183+0000 7f147f7fe640 1 -- 172.21.15.29:0/1601210868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v234) v1 ==== 93+0+10 (secure 0 0 0) 0x7f14980cc000 con 0x7f14980b1ca0 2023-10-14T18:18:03.185 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 msgr2=0x7f148406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:03.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 0x7f148406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 msgr2=0x7f14981f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 secure :-1 s=READY pgs=2709 cs=0 l=1 rev1=1 crypto rx=0x7f148800c400 tx=0x7f148800c8c0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 shutdown_connections 2023-10-14T18:18:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f148406b140 0x7f148406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f14980cfc40 0x7f149815e3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f14980b2080 0x7f149815d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 --2- 172.21.15.29:0/1601210868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f14980b1ca0 0x7f14981f2580 unknown :-1 s=CLOSED pgs=2709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 >> 172.21.15.29:0/1601210868 conn(0x7f14980d2b90 msgr2=0x7f14980d2070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 shutdown_connections 2023-10-14T18:18:03.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.186+0000 7f149c8ce640 1 -- 172.21.15.29:0/1601210868 wait complete. 2023-10-14T18:18:03.204 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:18:03.206 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.206+0000 7fe61bc12040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:18:03.243 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.242+0000 7f6620eda040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:18:03.254 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.254+0000 7fe61bc12040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:18:03.296 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.295+0000 7f6620eda040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:18:03.364 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.364+0000 7fe61bc12040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:18:03.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.413+0000 7f6620eda040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:18:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.422+0000 7f51f742e640 1 Processor -- start 2023-10-14T18:18:03.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.422+0000 7f51f742e640 1 -- start start 2023-10-14T18:18:03.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f742e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f51f015aa60 0x7f51f00af820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f742e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51f00afd60 0x7f51f00cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f742e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51f00b3730 con 0x7f51f00cc820 2023-10-14T18:18:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f742e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00b3980 con 0x7f51f00afd60 2023-10-14T18:18:03.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f742e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51f00b3c60 con 0x7f51f015aa60 2023-10-14T18:18:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f5c2b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51f00afd60 0x7f51f00cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f5c2b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f51f00afd60 0x7f51f00cc2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33780/0 (socket says 172.21.15.29:33780) 2023-10-14T18:18:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f5c2b640 1 -- 172.21.15.29:0/905983805 learned_addr learned my addr 172.21.15.29:0/905983805 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.424+0000 7f51f6c2d640 1 --1- 172.21.15.29:0/905983805 >> v1:172.21.15.29:6789/0 conn(0x7f51f00cc820 0x7f51f00b2360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50776/0 (socket says 172.21.15.29:50776) 2023-10-14T18:18:03.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.425+0000 7f51f642c640 1 --2- 172.21.15.29:0/905983805 >> v2:172.21.15.186:3300/0 conn(0x7f51f015aa60 0x7f51f00af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.425+0000 7f51f5c2b640 1 -- 172.21.15.29:0/905983805 >> v2:172.21.15.186:3300/0 conn(0x7f51f015aa60 msgr2=0x7f51f00af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.425+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/905983805 >> v2:172.21.15.186:3300/0 conn(0x7f51f015aa60 0x7f51f00af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.425+0000 7f51f5c2b640 1 -- 172.21.15.29:0/905983805 >> v1:172.21.15.29:6789/0 conn(0x7f51f00cc820 legacy=0x7f51f00b2360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.425+0000 7f51f5c2b640 1 -- 172.21.15.29:0/905983805 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51f00b3f40 con 0x7f51f00afd60 2023-10-14T18:18:03.429 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.428+0000 7fe61bc12040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.426+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/905983805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f00cc2e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f51f00ce210 tx=0x7f51e002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37d805d12f6c9fa4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:03.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.426+0000 7f51f542a640 1 -- 172.21.15.29:0/905983805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e0030860 con 0x7f51f00afd60 2023-10-14T18:18:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.426+0000 7f51f542a640 1 -- 172.21.15.29:0/905983805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51e00309c0 con 0x7f51f00afd60 2023-10-14T18:18:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.426+0000 7f51f542a640 1 -- 172.21.15.29:0/905983805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51e0030d50 con 0x7f51f00afd60 2023-10-14T18:18:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 -- 172.21.15.29:0/905983805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 msgr2=0x7f51f00cc2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 --2- 172.21.15.29:0/905983805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f00cc2e0 secure :-1 s=READY pgs=2711 cs=0 l=1 rev1=1 crypto rx=0x7f51f00ce210 tx=0x7f51e002b9a0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 -- 172.21.15.29:0/905983805 shutdown_connections 2023-10-14T18:18:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 --2- 172.21.15.29:0/905983805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f00cc2e0 unknown :-1 s=CLOSED pgs=2711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 --2- 172.21.15.29:0/905983805 >> v2:172.21.15.186:3300/0 conn(0x7f51f015aa60 0x7f51f00af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 -- 172.21.15.29:0/905983805 >> 172.21.15.29:0/905983805 conn(0x7f51f0065ed0 msgr2=0x7f51f00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:03.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 -- 172.21.15.29:0/905983805 shutdown_connections 2023-10-14T18:18:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.427+0000 7f51f742e640 1 -- 172.21.15.29:0/905983805 wait complete. 2023-10-14T18:18:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.429+0000 7f51f742e640 1 Processor -- start 2023-10-14T18:18:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.429+0000 7f51f742e640 1 -- start start 2023-10-14T18:18:03.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 0x7f51f01f2aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 0x7f51f01f55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00c65d0 con 0x7f51f00afd60 2023-10-14T18:18:03.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00c68b0 con 0x7f51f015aa60 2023-10-14T18:18:03.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f742e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51f00c6b90 con 0x7f51f00cc820 2023-10-14T18:18:03.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f642c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f642c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33784/0 (socket says 172.21.15.29:33784) 2023-10-14T18:18:03.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f642c640 1 -- 172.21.15.29:0/2844243173 learned_addr learned my addr 172.21.15.29:0/2844243173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:03.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.430+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 0x7f51f01f2aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f6c2d640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 0x7f51f01f55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 msgr2=0x7f51f01f2aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 0x7f51f01f2aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 msgr2=0x7f51f01f55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 0x7f51f01f55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 -- 172.21.15.29:0/2844243173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e00094d0 con 0x7f51f00afd60 2023-10-14T18:18:03.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 0x7f51f01f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f6c2d640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 0x7f51f01f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f642c640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f51ec007e40 tx=0x7f51ec00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec014620 con 0x7f51f00afd60 2023-10-14T18:18:03.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51ec00ee90 con 0x7f51f00afd60 2023-10-14T18:18:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51f00c6e70 con 0x7f51f00afd60 2023-10-14T18:18:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51ec0149c0 con 0x7f51f00afd60 2023-10-14T18:18:03.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.431+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51f01f2fe0 con 0x7f51f00afd60 2023-10-14T18:18:03.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.433+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51f004e9b0 con 0x7f51f00afd60 2023-10-14T18:18:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.433+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f51ec014b20 con 0x7f51f00afd60 2023-10-14T18:18:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.435+0000 7f51e6ffd640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 0x7f51d406d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.435+0000 7f51f5c2b640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 msgr2=0x7f51d406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:03.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.435+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f51ec08a8b0 con 0x7f51f00afd60 2023-10-14T18:18:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.435+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 0x7f51d406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:03.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.438+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f51ec055120 con 0x7f51f00afd60 2023-10-14T18:18:03.479 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.479+0000 7fe61bc12040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:18:03.508 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.507+0000 7f6620eda040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:03.536 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.535+0000 7fe61bc12040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:18:03.561 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.561+0000 7f6620eda040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:18:03.586 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.586+0000 7fe61bc12040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:18:03.622 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.622+0000 7f6620eda040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:18:03.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.636+0000 7f51f5c2b640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 msgr2=0x7f51d406d650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:03.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.636+0000 7f51f5c2b640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 0x7f51d406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:03.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.657+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f51f00c6710 con 0x7f51f00afd60 2023-10-14T18:18:03.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.658+0000 7f51e6ffd640 1 -- 172.21.15.29:0/2844243173 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v234) v1 ==== 104+0+11 (secure 0 0 0) 0x7f51f00c6710 con 0x7f51f00afd60 2023-10-14T18:18:03.659 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:03.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 msgr2=0x7f51d406d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 0x7f51d406d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 msgr2=0x7f51f01f18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 secure :-1 s=READY pgs=2712 cs=0 l=1 rev1=1 crypto rx=0x7f51ec007e40 tx=0x7f51ec00e4b0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 shutdown_connections 2023-10-14T18:18:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f51d406b190 0x7f51d406d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f51f015aa60 0x7f51f01f55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f51f00cc820 0x7f51f01f2aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 --2- 172.21.15.29:0/2844243173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f51f00afd60 0x7f51f01f18c0 unknown :-1 s=CLOSED pgs=2712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.661+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 >> 172.21.15.29:0/2844243173 conn(0x7f51f0065ed0 msgr2=0x7f51f006dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:03.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.662+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 shutdown_connections 2023-10-14T18:18:03.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.662+0000 7f51f742e640 1 -- 172.21.15.29:0/2844243173 wait complete. 2023-10-14T18:18:03.678 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:03.677+0000 7f6620eda040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:18:03.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:18:03.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.942+0000 7f17e3002640 1 Processor -- start 2023-10-14T18:18:03.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.942+0000 7f17e3002640 1 -- start start 2023-10-14T18:18:03.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17e3002640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17dc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17e3002640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f17dc06c140 0x7f17dc06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17e3002640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17dc070360 con 0x7f17dc159e90 2023-10-14T18:18:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17e3002640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17dc06cb10 con 0x7f17dc06c140 2023-10-14T18:18:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17e3002640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc15d890 con 0x7f17dc159ab0 2023-10-14T18:18:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17dbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f17dc159e90 0x7f17dc06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50786/0 (socket says 172.21.15.29:50786) 2023-10-14T18:18:03.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.944+0000 7f17dbfff640 1 -- 172.21.15.29:0/992118262 learned_addr learned my addr 172.21.15.29:0/992118262 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17e159f640 1 --2- 172.21.15.29:0/992118262 >> v2:172.21.15.186:3300/0 conn(0x7f17dc06c140 0x7f17dc06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17dc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728241365 0 0) 0x7f17dc070360 con 0x7f17dc159e90 2023-10-14T18:18:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f17cc003520 con 0x7f17dc159e90 2023-10-14T18:18:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3674729579 0 0) 0x7f17cc003520 con 0x7f17dc159e90 2023-10-14T18:18:03.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 msgr2=0x7f17dc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17dc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 >> v2:172.21.15.186:3300/0 conn(0x7f17dc06c140 msgr2=0x7f17dc06c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:03.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 --2- 172.21.15.29:0/992118262 >> v2:172.21.15.186:3300/0 conn(0x7f17dc06c140 0x7f17dc06c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0cd1f0 con 0x7f17dc159e90 2023-10-14T18:18:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f17c8001670 con 0x7f17dc159e90 2023-10-14T18:18:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 >> v1:172.21.15.29:6789/0 conn(0x7f17dc159e90 legacy=0x7f17dc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.945+0000 7f17db7fe640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc003970 0x7f17cc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17db7fe640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17cc0047c0 0x7f17cc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17db7fe640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc06cb10 con 0x7f17cc003970 2023-10-14T18:18:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc15d890 con 0x7f17cc0047c0 2023-10-14T18:18:03.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17cc00ba60 con 0x7f17dc159ab0 2023-10-14T18:18:03.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc003970 0x7f17cc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e159f640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 -- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 msgr2=0x7f17cc005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:03.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17dc159ab0 0x7f17cc005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 -- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17cc0047c0 msgr2=0x7f17cc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:03.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17cc0047c0 0x7f17cc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.946+0000 7f17e0d9e640 1 -- 172.21.15.29:0/992118262 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17cc003520 con 0x7f17cc003970 2023-10-14T18:18:03.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.947+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc003970 0x7f17cc003d90 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009a90 tx=0x7f17cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8f65e1de963492d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:03.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.947+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002c680 con 0x7f17cc003970 2023-10-14T18:18:03.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.947+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17d002c7e0 con 0x7f17cc003970 2023-10-14T18:18:03.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.947+0000 7f17db7fe640 1 -- 172.21.15.29:0/992118262 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17d002cb70 con 0x7f17cc003970 2023-10-14T18:18:03.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 -- 172.21.15.29:0/992118262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc003970 msgr2=0x7f17cc003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 --2- 172.21.15.29:0/992118262 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17cc003970 0x7f17cc003d90 secure :-1 s=READY pgs=2714 cs=0 l=1 rev1=1 crypto rx=0x7f17d0009a90 tx=0x7f17cc0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e159f640 1 -- 172.21.15.29:0/992118262 reap_dead start 2023-10-14T18:18:03.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 -- 172.21.15.29:0/992118262 shutdown_connections 2023-10-14T18:18:03.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 -- 172.21.15.29:0/992118262 >> 172.21.15.29:0/992118262 conn(0x7f17dc0ce9b0 msgr2=0x7f17dc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:03.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 -- 172.21.15.29:0/992118262 shutdown_connections 2023-10-14T18:18:03.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.948+0000 7f17e3002640 1 -- 172.21.15.29:0/992118262 wait complete. 2023-10-14T18:18:03.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.950+0000 7f17e3002640 1 Processor -- start 2023-10-14T18:18:03.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.950+0000 7f17e3002640 1 -- start start 2023-10-14T18:18:03.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 0x7f17dc0c46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17dc0c2a20 0x7f17dc1506f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0c3ce0 con 0x7f17dc14f530 2023-10-14T18:18:03.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0c4c30 con 0x7f17dc0c2a20 2023-10-14T18:18:03.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17e3002640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17dc0c2f10 con 0x7f17cc003970 2023-10-14T18:18:03.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17dbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33796/0 (socket says 172.21.15.29:33796) 2023-10-14T18:18:03.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.951+0000 7f17dbfff640 1 -- 172.21.15.29:0/1604751017 learned_addr learned my addr 172.21.15.29:0/1604751017 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 0x7f17dc0c46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17e159f640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17dc0c2a20 0x7f17dc1506f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 msgr2=0x7f17dc0c46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 0x7f17dc0c46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17dc0c2a20 msgr2=0x7f17dc1506f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:03.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17dc0c2a20 0x7f17dc1506f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:03.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 -- 172.21.15.29:0/1604751017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc0c31f0 con 0x7f17dc14f530 2023-10-14T18:18:03.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 0x7f17dc0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:03.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.952+0000 7f17dbfff640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c730 tx=0x7f17c800cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:03.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.953+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c80072d0 con 0x7f17dc14f530 2023-10-14T18:18:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.953+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17c8007430 con 0x7f17dc14f530 2023-10-14T18:18:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.953+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17dc0c3ef0 con 0x7f17dc14f530 2023-10-14T18:18:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.953+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17c8004a20 con 0x7f17dc14f530 2023-10-14T18:18:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.953+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17dc14f9b0 con 0x7f17dc14f530 2023-10-14T18:18:03.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.954+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17dc0c4410 con 0x7f17dc14f530 2023-10-14T18:18:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.956+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f17c8004b80 con 0x7f17dc14f530 2023-10-14T18:18:03.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.957+0000 7f17d97fa640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 0x7f17c006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.958+0000 7f17e0d9e640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 msgr2=0x7f17c006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.958+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f17c8089110 con 0x7f17dc14f530 2023-10-14T18:18:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.958+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 0x7f17c006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:03.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:03.962+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f17dc0c4410 con 0x7f17dc14f530 2023-10-14T18:18:03.973 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:03.973+0000 7fe61bc12040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:18:04.095 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.094+0000 7f6620eda040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:18:04.133 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.133+0000 7fe61bc12040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:18:04.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.156+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17dc0c4410 con 0x7f17dc14f530 2023-10-14T18:18:04.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.157+0000 7f17d97fa640 1 -- 172.21.15.29:0/1604751017 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v234) v1 ==== 100+0+11 (secure 0 0 0) 0x7f17dc0c4410 con 0x7f17dc14f530 2023-10-14T18:18:04.158 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:04.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.159+0000 7f17e0d9e640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 msgr2=0x7f17c006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:04.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.159+0000 7f17e0d9e640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 0x7f17c006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 msgr2=0x7f17c006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 0x7f17c006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 msgr2=0x7f17dc0c24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 secure :-1 s=READY pgs=2715 cs=0 l=1 rev1=1 crypto rx=0x7f17c800c730 tx=0x7f17c800cbf0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 shutdown_connections 2023-10-14T18:18:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f17c006b140 0x7f17c006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17dc0c2a20 0x7f17dc1506f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f17dc14f530 0x7f17dc0c24e0 unknown :-1 s=CLOSED pgs=2715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 --2- 172.21.15.29:0/1604751017 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17cc003970 0x7f17dc0c46f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 >> 172.21.15.29:0/1604751017 conn(0x7f17dc0ce9b0 msgr2=0x7f17dc0ae310 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 shutdown_connections 2023-10-14T18:18:04.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.160+0000 7f17e3002640 1 -- 172.21.15.29:0/1604751017 wait complete. 2023-10-14T18:18:04.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:18:04.242 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.241+0000 7fe61bc12040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:18:04.266 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.265+0000 7f6620eda040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:18:04.292 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.291+0000 7fe61bc12040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:18:04.380 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.380+0000 7f6620eda040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:18:04.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.394+0000 7fd16760e640 1 Processor -- start 2023-10-14T18:18:04.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.394+0000 7fd16760e640 1 -- start start 2023-10-14T18:18:04.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd16760e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd1600661c0 0x7fd1600ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd16760e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd16006cee0 con 0x7fd16015d330 2023-10-14T18:18:04.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd16760e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1600d0890 con 0x7fd1601618b0 2023-10-14T18:18:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd16760e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd16006b1e0 con 0x7fd1600661c0 2023-10-14T18:18:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd1653aa640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd16015d330 0x7fd16006c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50792/0 (socket says 172.21.15.29:50792) 2023-10-14T18:18:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd1653aa640 1 -- 172.21.15.29:0/2574512815 learned_addr learned my addr 172.21.15.29:0/2574512815 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:04.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.396+0000 7fd165bab640 1 --2- 172.21.15.29:0/2574512815 >> v2:172.21.15.29:3300/0 conn(0x7fd1600661c0 0x7fd1600ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.399 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.399+0000 7fe61bc12040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:18:04.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.397+0000 7fd165bab640 1 -- 172.21.15.29:0/2574512815 >> v1:172.21.15.186:6789/0 conn(0x7fd1601618b0 legacy=0x7fd160065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:04.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.397+0000 7fd157fff640 1 -- 172.21.15.29:0/2574512815 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345099379 0 0) 0x7fd16006cee0 con 0x7fd16015d330 2023-10-14T18:18:04.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.397+0000 7fd165bab640 1 -- 172.21.15.29:0/2574512815 >> v1:172.21.15.29:6789/0 conn(0x7fd16015d330 legacy=0x7fd16006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.397+0000 7fd165bab640 1 -- 172.21.15.29:0/2574512815 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1600d05b0 con 0x7fd1600661c0 2023-10-14T18:18:04.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.399+0000 7fd165bab640 1 --2- 172.21.15.29:0/2574512815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd1600ce9b0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fd158009d90 tx=0x7fd15802beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ee9895750cad156 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.399+0000 7fd157fff640 1 -- 172.21.15.29:0/2574512815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15802cc60 con 0x7fd1600661c0 2023-10-14T18:18:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.399+0000 7fd157fff640 1 -- 172.21.15.29:0/2574512815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15802cdc0 con 0x7fd1600661c0 2023-10-14T18:18:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.399+0000 7fd157fff640 1 -- 172.21.15.29:0/2574512815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd158034660 con 0x7fd1600661c0 2023-10-14T18:18:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 -- 172.21.15.29:0/2574512815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 msgr2=0x7fd1600ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 --2- 172.21.15.29:0/2574512815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd1600ce9b0 secure :-1 s=READY pgs=2717 cs=0 l=1 rev1=1 crypto rx=0x7fd158009d90 tx=0x7fd15802beb0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 -- 172.21.15.29:0/2574512815 shutdown_connections 2023-10-14T18:18:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 --2- 172.21.15.29:0/2574512815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd1600ce9b0 unknown :-1 s=CLOSED pgs=2717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 -- 172.21.15.29:0/2574512815 >> 172.21.15.29:0/2574512815 conn(0x7fd1600d0ab0 msgr2=0x7fd1600d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 -- 172.21.15.29:0/2574512815 shutdown_connections 2023-10-14T18:18:04.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.400+0000 7fd16760e640 1 -- 172.21.15.29:0/2574512815 wait complete. 2023-10-14T18:18:04.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.402+0000 7fd16760e640 1 Processor -- start 2023-10-14T18:18:04.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.402+0000 7fd16760e640 1 -- start start 2023-10-14T18:18:04.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.403+0000 7fd16760e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.403+0000 7fd16760e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd16015d330 0x7fd1600b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd16760e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 0x7fd1600ad8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd16760e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16015a160 con 0x7fd1600661c0 2023-10-14T18:18:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd16760e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1600c6ed0 con 0x7fd1601618b0 2023-10-14T18:18:04.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd16760e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1601f6650 con 0x7fd16015d330 2023-10-14T18:18:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd1653aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd1653aa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33818/0 (socket says 172.21.15.29:33818) 2023-10-14T18:18:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd1653aa640 1 -- 172.21.15.29:0/3783132679 learned_addr learned my addr 172.21.15.29:0/3783132679 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd164ba9640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd16015d330 0x7fd1600b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.404+0000 7fd165bab640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 0x7fd1600ad8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 -- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd16015d330 msgr2=0x7fd1600b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd16015d330 0x7fd1600b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 -- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 msgr2=0x7fd1600ad8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 0x7fd1600ad8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 -- 172.21.15.29:0/3783132679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd158009a70 con 0x7fd1600661c0 2023-10-14T18:18:04.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd165bab640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 0x7fd1600ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd1653aa640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c3f0 tx=0x7fd15c00c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c0074e0 con 0x7fd1600661c0 2023-10-14T18:18:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd15c007640 con 0x7fd1600661c0 2023-10-14T18:18:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c004a00 con 0x7fd1600661c0 2023-10-14T18:18:04.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1601f6930 con 0x7fd1600661c0 2023-10-14T18:18:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.405+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1600b0730 con 0x7fd1600661c0 2023-10-14T18:18:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.407+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16004e160 con 0x7fd1600661c0 2023-10-14T18:18:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.408+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fd15c004b60 con 0x7fd1600661c0 2023-10-14T18:18:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.408+0000 7fd155ffb640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 0x7fd14406d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.409+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fd15c089d30 con 0x7fd1600661c0 2023-10-14T18:18:04.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.409+0000 7fd164ba9640 1 -- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 msgr2=0x7fd14406d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:04.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.409+0000 7fd164ba9640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 0x7fd14406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:04.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.413+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd15c08e070 con 0x7fd1600661c0 2023-10-14T18:18:04.437 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.436+0000 7f6620eda040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:18:04.508 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.507+0000 7fe61bc12040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:18:04.551 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.550+0000 7f6620eda040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:18:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.605+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd16006f370 con 0x7fd1600661c0 2023-10-14T18:18:04.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.605+0000 7fd155ffb640 1 -- 172.21.15.29:0/3783132679 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v234) v1 ==== 107+0+11 (secure 0 0 0) 0x7fd15c054560 con 0x7fd1600661c0 2023-10-14T18:18:04.607 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 msgr2=0x7fd14406d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:04.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 0x7fd14406d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 msgr2=0x7fd160159c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 secure :-1 s=READY pgs=2718 cs=0 l=1 rev1=1 crypto rx=0x7fd15c00c3f0 tx=0x7fd15c00c8b0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 shutdown_connections 2023-10-14T18:18:04.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fd14406b140 0x7fd14406d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd1601618b0 0x7fd1600ad8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd16015d330 0x7fd1600b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 --2- 172.21.15.29:0/3783132679 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd1600661c0 0x7fd160159c20 unknown :-1 s=CLOSED pgs=2718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 >> 172.21.15.29:0/3783132679 conn(0x7fd1600d0ab0 msgr2=0x7fd16015bc40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:04.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 shutdown_connections 2023-10-14T18:18:04.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.608+0000 7fd16760e640 1 -- 172.21.15.29:0/3783132679 wait complete. 2023-10-14T18:18:04.627 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:18:04.664 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.663+0000 7fe61bc12040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:18:04.666 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.666+0000 7f6620eda040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:18:04.713 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.712+0000 7fe61bc12040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:04.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.827+0000 7fc3f5861640 1 Processor -- start 2023-10-14T18:18:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.827+0000 7fc3f5861640 1 -- start start 2023-10-14T18:18:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3f5861640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc3f006c850 0x7fc3f00ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3f5861640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f0070360 con 0x7fc3f0159ab0 2023-10-14T18:18:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3f5861640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3f00cd1f0 con 0x7fc3f0159e90 2023-10-14T18:18:04.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3f5861640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3f015d890 con 0x7fc3f006c850 2023-10-14T18:18:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3eeffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3f0159ab0 0x7fc3f006fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50806/0 (socket says 172.21.15.29:50806) 2023-10-14T18:18:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3eeffd640 1 -- 172.21.15.29:0/3045140675 learned_addr learned my addr 172.21.15.29:0/3045140675 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:04.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.829+0000 7fc3ef7fe640 1 --2- 172.21.15.29:0/3045140675 >> v2:172.21.15.29:3300/0 conn(0x7fc3f006c850 0x7fc3f00ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.830+0000 7fc3ef7fe640 1 -- 172.21.15.29:0/3045140675 >> v1:172.21.15.186:6789/0 conn(0x7fc3f0159e90 legacy=0x7fc3f006c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.830+0000 7fc3ef7fe640 1 -- 172.21.15.29:0/3045140675 >> v1:172.21.15.29:6789/0 conn(0x7fc3f0159ab0 legacy=0x7fc3f006fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.830+0000 7fc3ef7fe640 1 -- 172.21.15.29:0/3045140675 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f00cd4d0 con 0x7fc3f006c850 2023-10-14T18:18:04.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.831+0000 7fc3ef7fe640 1 --2- 172.21.15.29:0/3045140675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006c850 0x7fc3f00ce370 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009d90 tx=0x7fc3e802bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd99b5aa36683695 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:04.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.831+0000 7fc3edffb640 1 -- 172.21.15.29:0/3045140675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e802cb40 con 0x7fc3f006c850 2023-10-14T18:18:04.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.831+0000 7fc3edffb640 1 -- 172.21.15.29:0/3045140675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e802cca0 con 0x7fc3f006c850 2023-10-14T18:18:04.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.831+0000 7fc3edffb640 1 -- 172.21.15.29:0/3045140675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e8034660 con 0x7fc3f006c850 2023-10-14T18:18:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 -- 172.21.15.29:0/3045140675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006c850 msgr2=0x7fc3f00ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 --2- 172.21.15.29:0/3045140675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006c850 0x7fc3f00ce370 secure :-1 s=READY pgs=2720 cs=0 l=1 rev1=1 crypto rx=0x7fc3e8009d90 tx=0x7fc3e802bd90 comp rx=0 tx=0).stop 2023-10-14T18:18:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 -- 172.21.15.29:0/3045140675 shutdown_connections 2023-10-14T18:18:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 --2- 172.21.15.29:0/3045140675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f006c850 0x7fc3f00ce370 unknown :-1 s=CLOSED pgs=2720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 -- 172.21.15.29:0/3045140675 >> 172.21.15.29:0/3045140675 conn(0x7fc3f00ce9b0 msgr2=0x7fc3f00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 -- 172.21.15.29:0/3045140675 shutdown_connections 2023-10-14T18:18:04.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.832+0000 7fc3f5861640 1 -- 172.21.15.29:0/3045140675 wait complete. 2023-10-14T18:18:04.841 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:04.831+0000 7fe61bc12040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:18:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.834+0000 7fc3f5861640 1 Processor -- start 2023-10-14T18:18:04.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.834+0000 7fc3f5861640 1 -- start start 2023-10-14T18:18:04.842 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.834+0000 7f6620eda040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:18:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3f006c850 0x7fc3f01f32a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3f0159e90 0x7fc3f01f5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f01f2e80 con 0x7fc3f0159ab0 2023-10-14T18:18:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f015c890 con 0x7fc3f006c850 2023-10-14T18:18:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3f5861640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f01f6880 con 0x7fc3f0159e90 2023-10-14T18:18:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33836/0 (socket says 172.21.15.29:33836) 2023-10-14T18:18:04.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 -- 172.21.15.29:0/2757586885 learned_addr learned my addr 172.21.15.29:0/2757586885 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3eeffd640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3f006c850 0x7fc3f01f32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ef7fe640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3f0159e90 0x7fc3f01f5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3f0159e90 msgr2=0x7fc3f01f5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3f0159e90 0x7fc3f01f5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3f006c850 msgr2=0x7fc3f01f32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3f006c850 0x7fc3f01f32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.835+0000 7fc3ee7fc640 1 -- 172.21.15.29:0/2757586885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e8009a70 con 0x7fc3f0159ab0 2023-10-14T18:18:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3ee7fc640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400b3f0 tx=0x7fc3e400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4011830 con 0x7fc3f0159ab0 2023-10-14T18:18:04.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3e4011990 con 0x7fc3f0159ab0 2023-10-14T18:18:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc3e4011d40 con 0x7fc3f0159ab0 2023-10-14T18:18:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f01f6b60 con 0x7fc3f0159ab0 2023-10-14T18:18:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.836+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f01f2aa0 con 0x7fc3f0159ab0 2023-10-14T18:18:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.837+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f01f2fc0 con 0x7fc3f0159ab0 2023-10-14T18:18:04.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.838+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7fc3e401a400 con 0x7fc3f0159ab0 2023-10-14T18:18:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.839+0000 7fc3d7fff640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 0x7fc3dc06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.839+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7fc3e4089d10 con 0x7fc3f0159ab0 2023-10-14T18:18:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.839+0000 7fc3eeffd640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 msgr2=0x7fc3dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.839+0000 7fc3eeffd640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 0x7fc3dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:04.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:04.844+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc3f01f2fc0 con 0x7fc3f0159ab0 2023-10-14T18:18:04.888 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:04.887+0000 7f6620eda040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:05.016 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:05.014+0000 7f6620eda040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:18:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.040+0000 7fc3eeffd640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 msgr2=0x7fc3dc06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.040+0000 7fc3eeffd640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 0x7fc3dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:05.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.040+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7fc3f01f2fc0 con 0x7fc3f0159ab0 2023-10-14T18:18:05.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.041+0000 7fc3d7fff640 1 -- 172.21.15.29:0/2757586885 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v234) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc3f01f2fc0 con 0x7fc3f0159ab0 2023-10-14T18:18:05.043 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 msgr2=0x7fc3dc06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:05.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 0x7fc3dc06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 msgr2=0x7fc3f015ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 secure :-1 s=READY pgs=2721 cs=0 l=1 rev1=1 crypto rx=0x7fc3e400b3f0 tx=0x7fc3e400b8b0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 shutdown_connections 2023-10-14T18:18:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7fc3dc06b140 0x7fc3dc06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3f0159e90 0x7fc3f01f5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3f0159ab0 0x7fc3f015ceb0 unknown :-1 s=CLOSED pgs=2721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 --2- 172.21.15.29:0/2757586885 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3f006c850 0x7fc3f01f32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 >> 172.21.15.29:0/2757586885 conn(0x7fc3f00ce9b0 msgr2=0x7fc3f006b750 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 shutdown_connections 2023-10-14T18:18:05.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.044+0000 7fc3f5861640 1 -- 172.21.15.29:0/2757586885 wait complete. 2023-10-14T18:18:05.064 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T18:18:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.322+0000 7f53cede2640 1 Processor -- start 2023-10-14T18:18:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.322+0000 7f53cede2640 1 -- start start 2023-10-14T18:18:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cede2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53c815cd70 0x7f53c80d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cede2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c815d150 con 0x7f53c80b8480 2023-10-14T18:18:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cede2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53c80d1180 con 0x7f53c80cded0 2023-10-14T18:18:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cede2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53c80d1460 con 0x7f53c815cd70 2023-10-14T18:18:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cdde0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53c815cd70 0x7f53c80d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cdde0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53c815cd70 0x7f53c80d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33852/0 (socket says 172.21.15.29:33852) 2023-10-14T18:18:05.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.324+0000 7f53cdde0640 1 -- 172.21.15.29:0/3758994538 learned_addr learned my addr 172.21.15.29:0/3758994538 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.325+0000 7f53cdde0640 1 -- 172.21.15.29:0/3758994538 >> v1:172.21.15.186:6789/0 conn(0x7f53c80cded0 legacy=0x7f53c80b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.325+0000 7f53cdde0640 1 -- 172.21.15.29:0/3758994538 >> v1:172.21.15.29:6789/0 conn(0x7f53c80b8480 legacy=0x7f53c80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.325+0000 7f53cdde0640 1 -- 172.21.15.29:0/3758994538 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c80d1740 con 0x7f53c815cd70 2023-10-14T18:18:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.325+0000 7f53ccdde640 1 -- 172.21.15.29:0/3758994538 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045495564 0 0) 0x7f53c815d150 con 0x7f53c80b8480 2023-10-14T18:18:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.326+0000 7f53cdde0640 1 --2- 172.21.15.29:0/3758994538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c815cd70 0x7f53c80d2220 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7cbcf89ac4da89 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.326+0000 7f53ccdde640 1 -- 172.21.15.29:0/3758994538 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402f840 con 0x7f53c815cd70 2023-10-14T18:18:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.326+0000 7f53ccdde640 1 -- 172.21.15.29:0/3758994538 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c402f9a0 con 0x7f53c815cd70 2023-10-14T18:18:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.326+0000 7f53ccdde640 1 -- 172.21.15.29:0/3758994538 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53c402fd30 con 0x7f53c815cd70 2023-10-14T18:18:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 -- 172.21.15.29:0/3758994538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c815cd70 msgr2=0x7f53c80d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 --2- 172.21.15.29:0/3758994538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c815cd70 0x7f53c80d2220 secure :-1 s=READY pgs=2723 cs=0 l=1 rev1=1 crypto rx=0x7f53c4009870 tx=0x7f53c402ba20 comp rx=0 tx=0).stop 2023-10-14T18:18:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 -- 172.21.15.29:0/3758994538 shutdown_connections 2023-10-14T18:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 --2- 172.21.15.29:0/3758994538 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c815cd70 0x7f53c80d2220 unknown :-1 s=CLOSED pgs=2723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 -- 172.21.15.29:0/3758994538 >> 172.21.15.29:0/3758994538 conn(0x7f53c80cf570 msgr2=0x7f53c80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 -- 172.21.15.29:0/3758994538 shutdown_connections 2023-10-14T18:18:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.327+0000 7f53cede2640 1 -- 172.21.15.29:0/3758994538 wait complete. 2023-10-14T18:18:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.329+0000 7f53cede2640 1 Processor -- start 2023-10-14T18:18:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.329+0000 7f53cede2640 1 -- start start 2023-10-14T18:18:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 0x7f53c81f1d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c815cd70 0x7f53c80ab100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80b3040 con 0x7f53c80cded0 2023-10-14T18:18:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80b2430 con 0x7f53c815cd70 2023-10-14T18:18:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cede2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53c80b2710 con 0x7f53c80b8480 2023-10-14T18:18:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.330+0000 7f53cd5df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33854/0 (socket says 172.21.15.29:33854) 2023-10-14T18:18:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 -- 172.21.15.29:0/3609448085 learned_addr learned my addr 172.21.15.29:0/3609448085 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53ce5e1640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c815cd70 0x7f53c80ab100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cdde0640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 0x7f53c81f1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 msgr2=0x7f53c81f1d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 0x7f53c81f1d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c815cd70 msgr2=0x7f53c80ab100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c815cd70 0x7f53c80ab100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cd5df640 1 -- 172.21.15.29:0/3609448085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53c40094d0 con 0x7f53c80cded0 2023-10-14T18:18:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.331+0000 7f53cdde0640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 0x7f53c81f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53cd5df640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f53b800c460 tx=0x7f53b800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b8005480 con 0x7f53c80cded0 2023-10-14T18:18:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53b80035f0 con 0x7f53c80cded0 2023-10-14T18:18:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f53b80039a0 con 0x7f53c80cded0 2023-10-14T18:18:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53c80b29f0 con 0x7f53c80cded0 2023-10-14T18:18:05.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.332+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53c81f2250 con 0x7f53c80cded0 2023-10-14T18:18:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.333+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53c804e9b0 con 0x7f53c80cded0 2023-10-14T18:18:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.334+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f53b8003b00 con 0x7f53c80cded0 2023-10-14T18:18:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.335+0000 7f53be7fc640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 0x7f53ac06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.336+0000 7f53cdde0640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 msgr2=0x7f53ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:05.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.336+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f53b8089ac0 con 0x7f53c80cded0 2023-10-14T18:18:05.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.336+0000 7f53cdde0640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 0x7f53ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:05.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.340+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53b80542f0 con 0x7f53c80cded0 2023-10-14T18:18:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.537+0000 7f53cdde0640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 msgr2=0x7f53ac06d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.537+0000 7f53cdde0640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 0x7f53ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:05.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.549+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f53c80b6210 con 0x7f53c80cded0 2023-10-14T18:18:05.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.550+0000 7f53be7fc640 1 -- 172.21.15.29:0/3609448085 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 8 entries v234) v1 ==== 80+0+469 (secure 0 0 0) 0x7f53b8057910 con 0x7f53c80cded0 2023-10-14T18:18:05.551 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/761044706 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.551 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1993258206 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.552 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2825661941 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.552 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/2967217610 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.552 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/2967217610 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.552 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1476090294 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.552 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2871474293 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.553 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3012006424 2023-10-15T18:17:54.886178+0000 2023-10-14T18:18:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:listed 8 entries 2023-10-14T18:18:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 msgr2=0x7f53ac06d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 0x7f53ac06d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 msgr2=0x7f53c80aabc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 secure :-1 s=READY pgs=2724 cs=0 l=1 rev1=1 crypto rx=0x7f53b800c460 tx=0x7f53b800c920 comp rx=0 tx=0).stop 2023-10-14T18:18:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 shutdown_connections 2023-10-14T18:18:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f53ac06b140 0x7f53ac06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53c815cd70 0x7f53c80ab100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.552+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53c80cded0 0x7f53c80aabc0 unknown :-1 s=CLOSED pgs=2724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.553+0000 7f53cede2640 1 --2- 172.21.15.29:0/3609448085 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f53c80b8480 0x7f53c81f1d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.553+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 >> 172.21.15.29:0/3609448085 conn(0x7f53c80cf570 msgr2=0x7f53c80cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.553+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 shutdown_connections 2023-10-14T18:18:05.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:05.553+0000 7f53cede2640 1 -- 172.21.15.29:0/3609448085 wait complete. 2023-10-14T18:18:05.571 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T18:18:05.572 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:05.572 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T18:18:05.654 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:18:05.655 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:05.655 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T18:18:05.658 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:05.658+0000 7fe61bc12040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:18:05.700 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:18:05.701 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:05.756 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:18:05.757 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:05.757 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:05.813 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:18:05.813 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:18:05.814 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:05.869 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:05.868+0000 7f6620eda040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:18:05.871 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:18:05.871 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:05.962 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:05.962+0000 7fe61bc12040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:18:06.021 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:06.021+0000 7fe61bc12040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:18:06.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.093+0000 7f2bef59e640 1 Processor -- start 2023-10-14T18:18:06.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.094+0000 7f2bef59e640 1 -- start start 2023-10-14T18:18:06.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.095+0000 7f2bef59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2bf00ca830 0x7f2bf0070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.095+0000 7f2bef59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bf0070780 0x7f2bf006bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.095+0000 7f2bef59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2bf00ca3a0 con 0x7f2bf01573e0 2023-10-14T18:18:06.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.095+0000 7f2bef59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf0070c10 con 0x7f2bf0070780 2023-10-14T18:18:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.095+0000 7f2bef59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2bf00c9d70 con 0x7f2bf00ca830 2023-10-14T18:18:06.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bf0070780 0x7f2bf006bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2bf0070780 0x7f2bf006bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33862/0 (socket says 172.21.15.29:33862) 2023-10-14T18:18:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 -- 172.21.15.29:0/2302185557 learned_addr learned my addr 172.21.15.29:0/2302185557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:06.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2be7fff640 1 --2- 172.21.15.29:0/2302185557 >> v2:172.21.15.186:3300/0 conn(0x7f2bf00ca830 0x7f2bf0070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 -- 172.21.15.29:0/2302185557 >> v2:172.21.15.186:3300/0 conn(0x7f2bf00ca830 msgr2=0x7f2bf0070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 --2- 172.21.15.29:0/2302185557 >> v2:172.21.15.186:3300/0 conn(0x7f2bf00ca830 0x7f2bf0070240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 -- 172.21.15.29:0/2302185557 >> v1:172.21.15.186:6789/0 conn(0x7f2bf01573e0 legacy=0x7f2bf0159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.096+0000 7f2beed9d640 1 -- 172.21.15.29:0/2302185557 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2bf01577c0 con 0x7f2bf0070780 2023-10-14T18:18:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.098+0000 7f2beed9d640 1 --2- 172.21.15.29:0/2302185557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf0070780 0x7f2bf006bdc0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cd030 tx=0x7f2be802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c250f5279a0ed8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.098+0000 7f2bedd9b640 1 -- 172.21.15.29:0/2302185557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be802cb20 con 0x7f2bf0070780 2023-10-14T18:18:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.098+0000 7f2bedd9b640 1 -- 172.21.15.29:0/2302185557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2be802cc80 con 0x7f2bf0070780 2023-10-14T18:18:06.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.098+0000 7f2bedd9b640 1 -- 172.21.15.29:0/2302185557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2be80343f0 con 0x7f2bf0070780 2023-10-14T18:18:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 -- 172.21.15.29:0/2302185557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf0070780 msgr2=0x7f2bf006bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 --2- 172.21.15.29:0/2302185557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf0070780 0x7f2bf006bdc0 secure :-1 s=READY pgs=2725 cs=0 l=1 rev1=1 crypto rx=0x7f2bf00cd030 tx=0x7f2be802bf80 comp rx=0 tx=0).stop 2023-10-14T18:18:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 -- 172.21.15.29:0/2302185557 shutdown_connections 2023-10-14T18:18:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 --2- 172.21.15.29:0/2302185557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf0070780 0x7f2bf006bdc0 unknown :-1 s=CLOSED pgs=2725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 --2- 172.21.15.29:0/2302185557 >> v2:172.21.15.186:3300/0 conn(0x7f2bf00ca830 0x7f2bf0070240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 -- 172.21.15.29:0/2302185557 >> 172.21.15.29:0/2302185557 conn(0x7f2bf00cc230 msgr2=0x7f2bf00cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 -- 172.21.15.29:0/2302185557 shutdown_connections 2023-10-14T18:18:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.099+0000 7f2bef59e640 1 -- 172.21.15.29:0/2302185557 wait complete. 2023-10-14T18:18:06.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.101+0000 7f2bef59e640 1 Processor -- start 2023-10-14T18:18:06.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.101+0000 7f2bef59e640 1 -- start start 2023-10-14T18:18:06.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bf0070780 0x7f2bf00c6a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 0x7f2bf01647f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf014d0f0 con 0x7f2bf00ca830 2023-10-14T18:18:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf014d290 con 0x7f2bf01573e0 2023-10-14T18:18:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bef59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2bf0164d30 con 0x7f2bf0070780 2023-10-14T18:18:06.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2be7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2be7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:33872/0 (socket says 172.21.15.29:33872) 2023-10-14T18:18:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2be7fff640 1 -- 172.21.15.29:0/4281403396 learned_addr learned my addr 172.21.15.29:0/4281403396 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.102+0000 7f2bee59c640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bf0070780 0x7f2bf00c6a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bf0070780 msgr2=0x7f2bf00c6a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bf0070780 0x7f2bf00c6a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2beed9d640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 0x7f2bf01647f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 msgr2=0x7f2bf01647f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 0x7f2bf01647f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 -- 172.21.15.29:0/4281403396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2be8009b30 con 0x7f2bf00ca830 2023-10-14T18:18:06.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2beed9d640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 0x7f2bf01647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be7fff640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800e9b0 tx=0x7f2bd800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8009670 con 0x7f2bf00ca830 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2bd800c9a0 con 0x7f2bf00ca830 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2bf00c9a70 con 0x7f2bf00ca830 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2bd800cd50 con 0x7f2bf00ca830 2023-10-14T18:18:06.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.103+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2bf00c7400 con 0x7f2bf00ca830 2023-10-14T18:18:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.105+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2bf004e9b0 con 0x7f2bf00ca830 2023-10-14T18:18:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.105+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 143) v1 ==== 89490+0+0 (secure 0 0 0) 0x7f2bd8013470 con 0x7f2bf00ca830 2023-10-14T18:18:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.107+0000 7f2be6ffd640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 0x7f2bd006d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.107+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(234..234 src has 1..234) v4 ==== 7033+0+0 (secure 0 0 0) 0x7f2bd808a550 con 0x7f2bf00ca830 2023-10-14T18:18:06.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.107+0000 7f2bee59c640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 msgr2=0x7f2bd006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:06.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.107+0000 7f2bee59c640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 0x7f2bd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:06.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.111+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2bd8054dd0 con 0x7f2bf00ca830 2023-10-14T18:18:06.137 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:06.137+0000 7fe61bc12040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:18:06.188 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.187+0000 7f6620eda040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:18:06.248 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.247+0000 7f6620eda040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:18:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.308+0000 7f2bee59c640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 msgr2=0x7f2bd006d600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/2804419857 2023-10-14T18:18:06.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.308+0000 7f2bee59c640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 0x7f2bd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:06.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.348+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2bf00b5730 con 0x7f2bf00ca830 2023-10-14T18:18:06.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.349+0000 7f2be6ffd640 1 -- 172.21.15.29:0/4281403396 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7f2bd80583f0 con 0x7f2bf00ca830 2023-10-14T18:18:06.350 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:06.350 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:06.350 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:18:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.351+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 msgr2=0x7f2bd006d600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.351+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 0x7f2bd006d600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.351+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 msgr2=0x7f2bf014cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.351+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 secure :-1 s=READY pgs=2726 cs=0 l=1 rev1=1 crypto rx=0x7f2bd800e9b0 tx=0x7f2bd800ee70 comp rx=0 tx=0).stop 2023-10-14T18:18:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.351+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 shutdown_connections 2023-10-14T18:18:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:6832/2804419857,v1:172.21.15.29:6833/2804419857] conn(0x7f2bd006b140 0x7f2bd006d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2bf01573e0 0x7f2bf01647f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2bf00ca830 0x7f2bf014cbb0 unknown :-1 s=CLOSED pgs=2726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 --2- 172.21.15.29:0/4281403396 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2bf0070780 0x7f2bf00c6a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 >> 172.21.15.29:0/4281403396 conn(0x7f2bf00cc230 msgr2=0x7f2bf00cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:06.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 shutdown_connections 2023-10-14T18:18:06.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.352+0000 7f2bef59e640 1 -- 172.21.15.29:0/4281403396 wait complete. 2023-10-14T18:18:06.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:06.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.371+0000 7f6620eda040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:18:06.475 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.476 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.477 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.474+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.492+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.493 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.492+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.494 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.492+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.494 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.492+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.494 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.492+0000 7f65a5928640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.493+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.493+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.493+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.493+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:06.493+0000 7f65a2922640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:18:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.568+0000 7f71035e2640 1 Processor -- start 2023-10-14T18:18:06.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.568+0000 7f71035e2640 1 -- start start 2023-10-14T18:18:06.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.569+0000 7f71035e2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f70fc15aaa0 0x7f70fc15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.569+0000 7f71035e2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc15ed60 con 0x7f70fc15ae80 2023-10-14T18:18:06.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.569+0000 7f71035e2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70fc0707e0 con 0x7f70fc0ce390 2023-10-14T18:18:06.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.569+0000 7f71035e2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70fc06b810 con 0x7f70fc15aaa0 2023-10-14T18:18:06.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.569+0000 7f7100b7d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f70fc15ae80 0x7f70fc0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50822/0 (socket says 172.21.15.29:50822) 2023-10-14T18:18:06.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f7100b7d640 1 -- 172.21.15.29:0/2728385416 learned_addr learned my addr 172.21.15.29:0/2728385416 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:06.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f710137e640 1 --2- 172.21.15.29:0/2728385416 >> v2:172.21.15.186:3300/0 conn(0x7f70fc15aaa0 0x7f70fc15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 563117853 0 0) 0x7f70fc15ed60 con 0x7f70fc15ae80 2023-10-14T18:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70e8003520 con 0x7f70fc15ae80 2023-10-14T18:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 722958259 0 0) 0x7f70e8003520 con 0x7f70fc15ae80 2023-10-14T18:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 >> v2:172.21.15.186:3300/0 conn(0x7f70fc15aaa0 msgr2=0x7f70fc15e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 --2- 172.21.15.29:0/2728385416 >> v2:172.21.15.186:3300/0 conn(0x7f70fc15aaa0 0x7f70fc15e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 >> v1:172.21.15.186:6789/0 conn(0x7f70fc0ce390 legacy=0x7f70fc070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc15ef90 con 0x7f70fc15ae80 2023-10-14T18:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f70ec001670 con 0x7f70fc15ae80 2023-10-14T18:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 >> v1:172.21.15.29:6789/0 conn(0x7f70fc15ae80 legacy=0x7f70fc0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70e80039c0 0x7f70e8003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70e80047c0 0x7f70e8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70e800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc06b810 con 0x7f70e8005180 2023-10-14T18:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70e800c080 con 0x7f70e80039c0 2023-10-14T18:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.570+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70e800c240 con 0x7f70e80047c0 2023-10-14T18:18:06.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70e800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f710137e640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70e80039c0 0x7f70e8003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7100b7d640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70e80047c0 0x7f70e8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 -- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70e80047c0 msgr2=0x7f70e8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:06.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70e80047c0 0x7f70e8004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 -- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70e80039c0 msgr2=0x7f70e8003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70e80039c0 0x7f70e8003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.571+0000 7f7101b7f640 1 -- 172.21.15.29:0/2728385416 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70e8003520 con 0x7f70e8005180 2023-10-14T18:18:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.572+0000 7f7101b7f640 1 -- 172.21.15.29:0/2728385416 reap_dead start 2023-10-14T18:18:06.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.572+0000 7f7101b7f640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70e800b980 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f70f4001e50 tx=0x7f70e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd6f00aa0f70404a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:06.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.572+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4003b70 con 0x7f70e8005180 2023-10-14T18:18:06.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.572+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f4003d10 con 0x7f70e8005180 2023-10-14T18:18:06.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.572+0000 7f70f3fff640 1 -- 172.21.15.29:0/2728385416 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f4005310 con 0x7f70e8005180 2023-10-14T18:18:06.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 -- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 msgr2=0x7f70e800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70e800b980 secure :-1 s=READY pgs=2742 cs=0 l=1 rev1=1 crypto rx=0x7f70f4001e50 tx=0x7f70e80057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 -- 172.21.15.29:0/2728385416 shutdown_connections 2023-10-14T18:18:06.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 --2- 172.21.15.29:0/2728385416 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70e800b980 unknown :-1 s=CLOSED pgs=2742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 -- 172.21.15.29:0/2728385416 >> 172.21.15.29:0/2728385416 conn(0x7f70fc0cb410 msgr2=0x7f70fc0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:06.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 -- 172.21.15.29:0/2728385416 shutdown_connections 2023-10-14T18:18:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.573+0000 7f71035e2640 1 -- 172.21.15.29:0/2728385416 wait complete. 2023-10-14T18:18:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.575+0000 7f71035e2640 1 Processor -- start 2023-10-14T18:18:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.575+0000 7f71035e2640 1 -- start start 2023-10-14T18:18:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc0c8580 0x7f70fc1491f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc149b60 0x7f70fc0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc15b9d0 con 0x7f70e8005180 2023-10-14T18:18:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc15b450 con 0x7f70fc0c8580 2023-10-14T18:18:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f71035e2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70fc14cb00 con 0x7f70fc149b60 2023-10-14T18:18:06.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34014/0 (socket says 172.21.15.29:34014) 2023-10-14T18:18:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 -- 172.21.15.29:0/3903479147 learned_addr learned my addr 172.21.15.29:0/3903479147 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 -- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc149b60 msgr2=0x7f70fc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc149b60 0x7f70fc0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 -- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc0c8580 msgr2=0x7f70fc1491f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc0c8580 0x7f70fc1491f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.576+0000 7f710137e640 1 -- 172.21.15.29:0/3903479147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc15b6f0 con 0x7f70e8005180 2023-10-14T18:18:06.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f710137e640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f70f8011540 tx=0x7f70f8011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:06.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8015930 con 0x7f70e8005180 2023-10-14T18:18:06.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70f8015a90 con 0x7f70e8005180 2023-10-14T18:18:06.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70f8015e60 con 0x7f70e8005180 2023-10-14T18:18:06.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70fc15d0b0 con 0x7f70e8005180 2023-10-14T18:18:06.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.577+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70fc15cda0 con 0x7f70e8005180 2023-10-14T18:18:06.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.578+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70fc04e160 con 0x7f70e8005180 2023-10-14T18:18:06.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.579+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 144) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f70f800f460 con 0x7f70e8005180 2023-10-14T18:18:06.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.580+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f70f808a0a0 con 0x7f70e8005180 2023-10-14T18:18:06.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.584+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f70f8054da0 con 0x7f70e8005180 2023-10-14T18:18:06.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.783+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f70fc0cdc60 con 0x7f70e8005180 2023-10-14T18:18:06.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.786+0000 7f70f1ffb640 1 -- 172.21.15.29:0/3903479147 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+21667 (secure 0 0 0) 0x7f70f80583c0 con 0x7f70e8005180 2023-10-14T18:18:06.788 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:06.788 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":235,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:06.375225+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 msgr2=0x7f70fc14c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:06.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 secure :-1 s=READY pgs=2743 cs=0 l=1 rev1=1 crypto rx=0x7f70f8011540 tx=0x7f70f8011a00 comp rx=0 tx=0).stop 2023-10-14T18:18:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 shutdown_connections 2023-10-14T18:18:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f70fc149b60 0x7f70fc0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f70fc0c8580 0x7f70fc1491f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 --2- 172.21.15.29:0/3903479147 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f70e8005180 0x7f70fc14c5c0 unknown :-1 s=CLOSED pgs=2743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:06.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.789+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 >> 172.21.15.29:0/3903479147 conn(0x7f70fc0cb410 msgr2=0x7f70fc15a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.790+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 shutdown_connections 2023-10-14T18:18:06.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:06.790+0000 7f71035e2640 1 -- 172.21.15.29:0/3903479147 wait complete. 2023-10-14T18:18:06.809 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 26, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:17:25.200413+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '228', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 27, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:17:26.251106+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '229', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:17:40.115972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:17:40.810362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:06.810 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:07.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.035+0000 7f1e2ae61640 1 Processor -- start 2023-10-14T18:18:07.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.036+0000 7f1e2ae61640 1 -- start start 2023-10-14T18:18:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.037+0000 7f1e2ae61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1e2406a4b0 0x7f1e240ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.037+0000 7f1e2ae61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 0x7f1e240ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.037+0000 7f1e2ae61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e240cb030 con 0x7f1e2415ad70 2023-10-14T18:18:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.037+0000 7f1e2ae61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e240cae30 con 0x7f1e2406a4b0 2023-10-14T18:18:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.037+0000 7f1e2ae61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e240c91c0 con 0x7f1e240c9ca0 2023-10-14T18:18:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1e2406a4b0 0x7f1e240ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1e2406a4b0 0x7f1e240ca3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34024/0 (socket says 172.21.15.29:34024) 2023-10-14T18:18:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 -- 172.21.15.29:0/3939987957 learned_addr learned my addr 172.21.15.29:0/3939987957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e28bfd640 1 --1- 172.21.15.29:0/3939987957 >> v1:172.21.15.29:6789/0 conn(0x7f1e2415ad70 0x7f1e24069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50830/0 (socket says 172.21.15.29:50830) 2023-10-14T18:18:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e293fe640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 0x7f1e240ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 -- 172.21.15.29:0/3939987957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 msgr2=0x7f1e240ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 0x7f1e240ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 -- 172.21.15.29:0/3939987957 >> v1:172.21.15.29:6789/0 conn(0x7f1e2415ad70 legacy=0x7f1e24069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e1bfff640 1 -- 172.21.15.29:0/3939987957 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e240c94a0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.038+0000 7f1e293fe640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 0x7f1e240ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.039+0000 7f1e1bfff640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240ca3b0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009870 tx=0x7f1e1402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b141b8045e2c41d0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e1b7fe640 1 -- 172.21.15.29:0/3939987957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14030840 con 0x7f1e2406a4b0 2023-10-14T18:18:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e1b7fe640 1 -- 172.21.15.29:0/3939987957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e140309a0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e1b7fe640 1 -- 172.21.15.29:0/3939987957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e14030d30 con 0x7f1e2406a4b0 2023-10-14T18:18:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 -- 172.21.15.29:0/3939987957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 msgr2=0x7f1e240ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240ca3b0 secure :-1 s=READY pgs=2745 cs=0 l=1 rev1=1 crypto rx=0x7f1e14009870 tx=0x7f1e1402b980 comp rx=0 tx=0).stop 2023-10-14T18:18:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 -- 172.21.15.29:0/3939987957 shutdown_connections 2023-10-14T18:18:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e240c9ca0 0x7f1e240ca8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/3939987957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240ca3b0 unknown :-1 s=CLOSED pgs=2745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 -- 172.21.15.29:0/3939987957 >> 172.21.15.29:0/3939987957 conn(0x7f1e240ce9b0 msgr2=0x7f1e240cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:07.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 -- 172.21.15.29:0/3939987957 shutdown_connections 2023-10-14T18:18:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.040+0000 7f1e2ae61640 1 -- 172.21.15.29:0/3939987957 wait complete. 2023-10-14T18:18:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.042+0000 7f1e2ae61640 1 Processor -- start 2023-10-14T18:18:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.042+0000 7f1e2ae61640 1 -- start start 2023-10-14T18:18:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e240c9ca0 0x7f1e240c4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e2415ad70 0x7f1e2415c170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e240c43b0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e240af4f0 con 0x7f1e240c9ca0 2023-10-14T18:18:07.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e2ae61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e2415cce0 con 0x7f1e2415ad70 2023-10-14T18:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.043+0000 7f1e28bfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34040/0 (socket says 172.21.15.29:34040) 2023-10-14T18:18:07.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 -- 172.21.15.29:0/2967558644 learned_addr learned my addr 172.21.15.29:0/2967558644 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 -- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e2415ad70 msgr2=0x7f1e2415c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e293fe640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e2415ad70 0x7f1e2415c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e2415ad70 0x7f1e2415c170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 -- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e240c9ca0 msgr2=0x7f1e240c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e240c9ca0 0x7f1e240c4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 -- 172.21.15.29:0/2967558644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e140094d0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.044+0000 7f1e28bfd640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c3e0 tx=0x7f1e2000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.045+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e200074e0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.045+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e20007640 con 0x7f1e2406a4b0 2023-10-14T18:18:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.045+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e200049e0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.045+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2415cfc0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.045+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e240c48e0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.046+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e2404e160 con 0x7f1e2406a4b0 2023-10-14T18:18:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.047+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 144) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f1e20004b40 con 0x7f1e2406a4b0 2023-10-14T18:18:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.048+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f1e200891a0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.052+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1e20053f90 con 0x7f1e2406a4b0 2023-10-14T18:18:07.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.245+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1e2415e310 con 0x7f1e2406a4b0 2023-10-14T18:18:07.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.246+0000 7f1e197fa640 1 -- 172.21.15.29:0/2967558644 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v235) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1e200575b0 con 0x7f1e2406a4b0 2023-10-14T18:18:07.247 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 msgr2=0x7f1e240aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 secure :-1 s=READY pgs=2746 cs=0 l=1 rev1=1 crypto rx=0x7f1e2000c3e0 tx=0x7f1e2000c8a0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 shutdown_connections 2023-10-14T18:18:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1e2415ad70 0x7f1e2415c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1e240c9ca0 0x7f1e240c4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 --2- 172.21.15.29:0/2967558644 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1e2406a4b0 0x7f1e240aefb0 unknown :-1 s=CLOSED pgs=2746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 >> 172.21.15.29:0/2967558644 conn(0x7f1e240ce9b0 msgr2=0x7f1e240ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:07.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 shutdown_connections 2023-10-14T18:18:07.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.249+0000 7f1e2ae61640 1 -- 172.21.15.29:0/2967558644 wait complete. 2023-10-14T18:18:07.268 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:18:07.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.500+0000 7faa311e4640 1 Processor -- start 2023-10-14T18:18:07.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.500+0000 7faa311e4640 1 -- start start 2023-10-14T18:18:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa311e4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa2c0ad970 0x7faa2c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa311e4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa2c0ccfb0 0x7faa2c0cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa311e4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa2c15b9d0 con 0x7faa2c06c040 2023-10-14T18:18:07.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa311e4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c15bcb0 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa311e4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa2c15bf90 con 0x7faa2c0ad970 2023-10-14T18:18:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa2bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7faa2c06c040 0x7faa2c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50844/0 (socket says 172.21.15.29:50844) 2023-10-14T18:18:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.501+0000 7faa2bfff640 1 -- 172.21.15.29:0/2417795727 learned_addr learned my addr 172.21.15.29:0/2417795727 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa309e3640 1 --2- 172.21.15.29:0/2417795727 >> v2:172.21.15.29:3300/0 conn(0x7faa2c0ccfb0 0x7faa2c0cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa23fff640 1 --2- 172.21.15.29:0/2417795727 >> v2:172.21.15.186:3300/0 conn(0x7faa2c0ad970 0x7faa2c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa2b7fe640 1 -- 172.21.15.29:0/2417795727 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2631535260 0 0) 0x7faa2c15b9d0 con 0x7faa2c06c040 2023-10-14T18:18:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa309e3640 1 -- 172.21.15.29:0/2417795727 >> v2:172.21.15.186:3300/0 conn(0x7faa2c0ad970 msgr2=0x7faa2c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa309e3640 1 --2- 172.21.15.29:0/2417795727 >> v2:172.21.15.186:3300/0 conn(0x7faa2c0ad970 0x7faa2c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa309e3640 1 -- 172.21.15.29:0/2417795727 >> v1:172.21.15.29:6789/0 conn(0x7faa2c06c040 legacy=0x7faa2c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.502+0000 7faa309e3640 1 -- 172.21.15.29:0/2417795727 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa2c0cc270 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.503+0000 7faa309e3640 1 --2- 172.21.15.29:0/2417795727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ccfb0 0x7faa2c0cd460 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7faa240097a0 tx=0x7faa2402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cb3c9e42c495d2e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.503+0000 7faa2b7fe640 1 -- 172.21.15.29:0/2417795727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24031830 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa2b7fe640 1 -- 172.21.15.29:0/2417795727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa24031990 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa2b7fe640 1 -- 172.21.15.29:0/2417795727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa24031d20 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 -- 172.21.15.29:0/2417795727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ccfb0 msgr2=0x7faa2c0cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 --2- 172.21.15.29:0/2417795727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ccfb0 0x7faa2c0cd460 secure :-1 s=READY pgs=2748 cs=0 l=1 rev1=1 crypto rx=0x7faa240097a0 tx=0x7faa2402b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 -- 172.21.15.29:0/2417795727 shutdown_connections 2023-10-14T18:18:07.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 --2- 172.21.15.29:0/2417795727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ccfb0 0x7faa2c0cd460 unknown :-1 s=CLOSED pgs=2748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 --2- 172.21.15.29:0/2417795727 >> v2:172.21.15.186:3300/0 conn(0x7faa2c0ad970 0x7faa2c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 -- 172.21.15.29:0/2417795727 >> 172.21.15.29:0/2417795727 conn(0x7faa2c0ce6a0 msgr2=0x7faa2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:07.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 -- 172.21.15.29:0/2417795727 shutdown_connections 2023-10-14T18:18:07.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.504+0000 7faa311e4640 1 -- 172.21.15.29:0/2417795727 wait complete. 2023-10-14T18:18:07.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.506+0000 7faa311e4640 1 Processor -- start 2023-10-14T18:18:07.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.506+0000 7faa311e4640 1 -- start start 2023-10-14T18:18:07.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 0x7faa2c0b2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa2c0ccfb0 0x7faa2c15c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0b02a0 con 0x7faa2c0ad970 2023-10-14T18:18:07.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0b2ad0 con 0x7faa2c06c040 2023-10-14T18:18:07.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa311e4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa2c0b2da0 con 0x7faa2c0ccfb0 2023-10-14T18:18:07.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34058/0 (socket says 172.21.15.29:34058) 2023-10-14T18:18:07.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.507+0000 7faa23fff640 1 -- 172.21.15.29:0/2222202406 learned_addr learned my addr 172.21.15.29:0/2222202406 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:07.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa309e3640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa2c0ccfb0 0x7faa2c15c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa2bfff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 0x7faa2c0b2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 -- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa2c0ccfb0 msgr2=0x7faa2c15c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa2c0ccfb0 0x7faa2c15c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 -- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 msgr2=0x7faa2c0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 0x7faa2c0b2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 -- 172.21.15.29:0/2222202406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa24009400 con 0x7faa2c0ad970 2023-10-14T18:18:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa2bfff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 0x7faa2c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa23fff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d7d0 tx=0x7faa1400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.508+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa14018070 con 0x7faa2c0ad970 2023-10-14T18:18:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.509+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa14004020 con 0x7faa2c0ad970 2023-10-14T18:18:07.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.509+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa140041b0 con 0x7faa2c0ad970 2023-10-14T18:18:07.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.509+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa2c0b3080 con 0x7faa2c0ad970 2023-10-14T18:18:07.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.509+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa2c0b3640 con 0x7faa2c0ad970 2023-10-14T18:18:07.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.510+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa2c06b330 con 0x7faa2c0ad970 2023-10-14T18:18:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.511+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7faa14004310 con 0x7faa2c0ad970 2023-10-14T18:18:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.512+0000 7faa297fa640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 0x7faa0c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.512+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7faa140898a0 con 0x7faa2c0ad970 2023-10-14T18:18:07.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.512+0000 7faa2bfff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 0x7faa0c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.513+0000 7faa2bfff640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 0x7faa0c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0012e0 tx=0x7faa1c009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.517+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa140544b0 con 0x7faa2c0ad970 2023-10-14T18:18:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.712+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7faa2c04e9b0 con 0x7faa2c0ad970 2023-10-14T18:18:07.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.713+0000 7faa297fa640 1 -- 172.21.15.29:0/2222202406 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v235) v1 ==== 104+0+11 (secure 0 0 0) 0x7faa14057ad0 con 0x7faa2c0ad970 2023-10-14T18:18:07.714 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:07.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 msgr2=0x7faa0c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 0x7faa0c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7faa1c0012e0 tx=0x7faa1c009040 comp rx=0 tx=0).stop 2023-10-14T18:18:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 msgr2=0x7faa2c155b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 secure :-1 s=READY pgs=2749 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d7d0 tx=0x7faa1400dc90 comp rx=0 tx=0).stop 2023-10-14T18:18:07.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 shutdown_connections 2023-10-14T18:18:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7faa0c06acc0 0x7faa0c06d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa2c0ccfb0 0x7faa2c15c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa2c0ad970 0x7faa2c155b40 unknown :-1 s=CLOSED pgs=2749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 --2- 172.21.15.29:0/2222202406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa2c06c040 0x7faa2c0b2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 >> 172.21.15.29:0/2222202406 conn(0x7faa2c0ce6a0 msgr2=0x7faa2c15d8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.715+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 shutdown_connections 2023-10-14T18:18:07.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.716+0000 7faa311e4640 1 -- 172.21.15.29:0/2222202406 wait complete. 2023-10-14T18:18:07.734 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:18:07.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.957+0000 7f852e675640 1 Processor -- start 2023-10-14T18:18:07.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.958+0000 7f852e675640 1 -- start start 2023-10-14T18:18:07.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.959+0000 7f852e675640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f852815cd70 0x7f85280d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.959+0000 7f852e675640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85280cded0 0x7f85280b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.959+0000 7f852e675640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f852815d150 con 0x7f85280b8480 2023-10-14T18:18:07.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.959+0000 7f852e675640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85280d1180 con 0x7f852815cd70 2023-10-14T18:18:07.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.959+0000 7f852e675640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85280d1460 con 0x7f85280cded0 2023-10-14T18:18:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852ce72640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f85280b8480 0x7f85280cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50860/0 (socket says 172.21.15.29:50860) 2023-10-14T18:18:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852ce72640 1 -- 172.21.15.29:0/1458378084 learned_addr learned my addr 172.21.15.29:0/1458378084 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:07.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852d673640 1 --2- 172.21.15.29:0/1458378084 >> v2:172.21.15.29:3300/0 conn(0x7f852815cd70 0x7f85280d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852de74640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85280cded0 0x7f85280b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852d673640 1 -- 172.21.15.29:0/1458378084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85280cded0 msgr2=0x7f85280b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:07.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852d673640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85280cded0 0x7f85280b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.960+0000 7f852d673640 1 -- 172.21.15.29:0/1458378084 >> v1:172.21.15.29:6789/0 conn(0x7f85280b8480 legacy=0x7f85280cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.961+0000 7f852d673640 1 -- 172.21.15.29:0/1458378084 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85280d1740 con 0x7f852815cd70 2023-10-14T18:18:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.962+0000 7f852d673640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f852815cd70 0x7f85280d2220 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f8524009d90 tx=0x7f852402beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0d6d3c42c79d8f3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.962+0000 7f851ffff640 1 -- 172.21.15.29:0/1458378084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f852402cc60 con 0x7f852815cd70 2023-10-14T18:18:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.962+0000 7f851ffff640 1 -- 172.21.15.29:0/1458378084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f852402cdc0 con 0x7f852815cd70 2023-10-14T18:18:07.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.962+0000 7f851ffff640 1 -- 172.21.15.29:0/1458378084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8524034660 con 0x7f852815cd70 2023-10-14T18:18:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 -- 172.21.15.29:0/1458378084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f852815cd70 msgr2=0x7f85280d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f852815cd70 0x7f85280d2220 secure :-1 s=READY pgs=2751 cs=0 l=1 rev1=1 crypto rx=0x7f8524009d90 tx=0x7f852402beb0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 -- 172.21.15.29:0/1458378084 shutdown_connections 2023-10-14T18:18:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f85280cded0 0x7f85280b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 --2- 172.21.15.29:0/1458378084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f852815cd70 0x7f85280d2220 unknown :-1 s=CLOSED pgs=2751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 -- 172.21.15.29:0/1458378084 >> 172.21.15.29:0/1458378084 conn(0x7f85280cf570 msgr2=0x7f85280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 -- 172.21.15.29:0/1458378084 shutdown_connections 2023-10-14T18:18:07.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.963+0000 7f852e675640 1 -- 172.21.15.29:0/1458378084 wait complete. 2023-10-14T18:18:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.965+0000 7f852e675640 1 Processor -- start 2023-10-14T18:18:07.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.965+0000 7f852e675640 1 -- start start 2023-10-14T18:18:07.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85280b8480 0x7f8528148670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f852815cd70 0x7f85280b6f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85281495d0 con 0x7f85280cded0 2023-10-14T18:18:07.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8528148d30 con 0x7f85280b8480 2023-10-14T18:18:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852e675640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8528149010 con 0x7f852815cd70 2023-10-14T18:18:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852ce72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852ce72640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34084/0 (socket says 172.21.15.29:34084) 2023-10-14T18:18:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852ce72640 1 -- 172.21.15.29:0/3289482369 learned_addr learned my addr 172.21.15.29:0/3289482369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:07.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852d673640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85280b8480 0x7f8528148670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.967+0000 7f852de74640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f852815cd70 0x7f85280b6f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 -- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f852815cd70 msgr2=0x7f85280b6f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f852815cd70 0x7f85280b6f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 -- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85280b8480 msgr2=0x7f8528148670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:07.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85280b8480 0x7f8528148670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 -- 172.21.15.29:0/3289482369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8524009a70 con 0x7f85280cded0 2023-10-14T18:18:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852ce72640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f851800c450 tx=0x7f851800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8518005ce0 con 0x7f85280cded0 2023-10-14T18:18:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8518005e40 con 0x7f85280cded0 2023-10-14T18:18:07.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f85180072b0 con 0x7f85280cded0 2023-10-14T18:18:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.968+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85281492f0 con 0x7f85280cded0 2023-10-14T18:18:07.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.969+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85281f6780 con 0x7f85280cded0 2023-10-14T18:18:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.970+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f852804e9b0 con 0x7f85280cded0 2023-10-14T18:18:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.971+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f8518007410 con 0x7f85280cded0 2023-10-14T18:18:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.972+0000 7f851dffb640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 0x7f850c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:07.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.972+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f8518088f50 con 0x7f85280cded0 2023-10-14T18:18:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.972+0000 7f852d673640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 0x7f850c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:07.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.973+0000 7f852d673640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 0x7f850c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8524036000 tx=0x7f8524030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:07.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:07.976+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8518053b60 con 0x7f85280cded0 2023-10-14T18:18:08.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.174+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f85280b4330 con 0x7f85280cded0 2023-10-14T18:18:08.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.175+0000 7f851dffb640 1 -- 172.21.15.29:0/3289482369 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v235) v1 ==== 100+0+11 (secure 0 0 0) 0x7f8518017020 con 0x7f85280cded0 2023-10-14T18:18:08.176 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:08.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 msgr2=0x7f850c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 0x7f850c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8524036000 tx=0x7f8524030000 comp rx=0 tx=0).stop 2023-10-14T18:18:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 msgr2=0x7f85281f6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 secure :-1 s=READY pgs=2752 cs=0 l=1 rev1=1 crypto rx=0x7f851800c450 tx=0x7f851800c910 comp rx=0 tx=0).stop 2023-10-14T18:18:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 shutdown_connections 2023-10-14T18:18:08.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f850c06acc0 0x7f850c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f852815cd70 0x7f85280b6f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.177+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f85280cded0 0x7f85281f6240 unknown :-1 s=CLOSED pgs=2752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.178+0000 7f852e675640 1 --2- 172.21.15.29:0/3289482369 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f85280b8480 0x7f8528148670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.178+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 >> 172.21.15.29:0/3289482369 conn(0x7f85280cf570 msgr2=0x7f85280cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:08.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.178+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 shutdown_connections 2023-10-14T18:18:08.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.178+0000 7f852e675640 1 -- 172.21.15.29:0/3289482369 wait complete. 2023-10-14T18:18:08.198 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:18:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.430+0000 7f9380a2d640 1 Processor -- start 2023-10-14T18:18:08.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.430+0000 7f9380a2d640 1 -- start start 2023-10-14T18:18:08.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9380a2d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 0x7f937c0b63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9380a2d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0d3980 con 0x7f937c0b3420 2023-10-14T18:18:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9380a2d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937c0ce630 con 0x7f937c0cf710 2023-10-14T18:18:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9380a2d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0d3c60 con 0x7f937c0cfaf0 2023-10-14T18:18:08.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9371d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f937c0b3420 0x7f937c0b5cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50868/0 (socket says 172.21.15.29:50868) 2023-10-14T18:18:08.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f9371d9b640 1 -- 172.21.15.29:0/2119549052 learned_addr learned my addr 172.21.15.29:0/2119549052 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:08.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.432+0000 7f937ad9d640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 0x7f937c0b63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3162845940 0 0) 0x7f937c0d3980 con 0x7f937c0b3420 2023-10-14T18:18:08.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9368003520 con 0x7f937c0b3420 2023-10-14T18:18:08.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317491464 0 0) 0x7f937c0ce630 con 0x7f937c0cf710 2023-10-14T18:18:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f937c0d3980 con 0x7f937c0cf710 2023-10-14T18:18:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2241928634 0 0) 0x7f9368003520 con 0x7f937c0b3420 2023-10-14T18:18:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 msgr2=0x7f937c0b63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 0x7f937c0b63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 >> v1:172.21.15.186:6789/0 conn(0x7f937c0cf710 legacy=0x7f937c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0ce910 con 0x7f937c0b3420 2023-10-14T18:18:08.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.433+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9364001670 con 0x7f937c0b3420 2023-10-14T18:18:08.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 >> v1:172.21.15.29:6789/0 conn(0x7f937c0b3420 legacy=0x7f937c0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9368003b30 0x7f9368003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93680047c0 0x7f9368004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 0x7f93680051a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0d3c60 con 0x7f9368003b30 2023-10-14T18:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936800bac0 con 0x7f93680047c0 2023-10-14T18:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f936800bc80 con 0x7f937c0cfaf0 2023-10-14T18:18:08.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.434+0000 7f937a59c640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9368003b30 0x7f9368003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f9371d9b640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93680047c0 0x7f9368004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f937a59c640 1 -- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 msgr2=0x7f93680051a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f937a59c640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0cfaf0 0x7f93680051a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f937a59c640 1 -- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93680047c0 msgr2=0x7f9368004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f937a59c640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f93680047c0 0x7f9368004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.435+0000 7f937a59c640 1 -- 172.21.15.29:0/2119549052 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9368003520 con 0x7f9368003b30 2023-10-14T18:18:08.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.436+0000 7f937a59c640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9368003b30 0x7f9368003f30 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f936c002010 tx=0x7f93680057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51fd51c3d412495b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.436+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c003bc0 con 0x7f9368003b30 2023-10-14T18:18:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.436+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c003d40 con 0x7f9368003b30 2023-10-14T18:18:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.436+0000 7f9379d9b640 1 -- 172.21.15.29:0/2119549052 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c005310 con 0x7f9368003b30 2023-10-14T18:18:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 -- 172.21.15.29:0/2119549052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9368003b30 msgr2=0x7f9368003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 --2- 172.21.15.29:0/2119549052 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9368003b30 0x7f9368003f30 secure :-1 s=READY pgs=2754 cs=0 l=1 rev1=1 crypto rx=0x7f936c002010 tx=0x7f93680057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f937ad9d640 1 -- 172.21.15.29:0/2119549052 reap_dead start 2023-10-14T18:18:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 -- 172.21.15.29:0/2119549052 shutdown_connections 2023-10-14T18:18:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 -- 172.21.15.29:0/2119549052 >> 172.21.15.29:0/2119549052 conn(0x7f937c0d2cc0 msgr2=0x7f937c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 -- 172.21.15.29:0/2119549052 shutdown_connections 2023-10-14T18:18:08.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.437+0000 7f9380a2d640 1 -- 172.21.15.29:0/2119549052 wait complete. 2023-10-14T18:18:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.439+0000 7f9380a2d640 1 Processor -- start 2023-10-14T18:18:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.439+0000 7f9380a2d640 1 -- start start 2023-10-14T18:18:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f937c1465f0 0x7f937c0d0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0d0db0 0x7f937c15a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c152a00 con 0x7f937c145f70 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0d1230 con 0x7f937c1465f0 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.440+0000 7f9380a2d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f937c0b1ea0 con 0x7f937c0d0db0 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34098/0 (socket says 172.21.15.29:34098) 2023-10-14T18:18:08.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 -- 172.21.15.29:0/3792805581 learned_addr learned my addr 172.21.15.29:0/3792805581 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937ad9d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0d0db0 0x7f937c15a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f9371d9b640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f937c1465f0 0x7f937c0d0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 -- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0d0db0 msgr2=0x7f937c15a6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0d0db0 0x7f937c15a6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 -- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f937c1465f0 msgr2=0x7f937c0d0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.441+0000 7f937a59c640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f937c1465f0 0x7f937c0d0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f937a59c640 1 -- 172.21.15.29:0/3792805581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937c0b2180 con 0x7f937c145f70 2023-10-14T18:18:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f937a59c640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f936c002010 tx=0x7f936c0016a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c003bd0 con 0x7f937c145f70 2023-10-14T18:18:08.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c006ba0 con 0x7f937c145f70 2023-10-14T18:18:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f937c0b2460 con 0x7f937c145f70 2023-10-14T18:18:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.442+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f936c02c570 con 0x7f937c145f70 2023-10-14T18:18:08.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.443+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f937c146350 con 0x7f937c145f70 2023-10-14T18:18:08.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.444+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937c146350 con 0x7f937c145f70 2023-10-14T18:18:08.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.445+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f936c02c6d0 con 0x7f937c145f70 2023-10-14T18:18:08.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.446+0000 7f93737fe640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 0x7f935c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.446+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f936c0ab070 con 0x7f937c145f70 2023-10-14T18:18:08.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.446+0000 7f9371d9b640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 0x7f935c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.447+0000 7f9371d9b640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 0x7f935c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9364001200 tx=0x7f9364009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.452+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f936c075c80 con 0x7f937c145f70 2023-10-14T18:18:08.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.654+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f937c146350 con 0x7f937c145f70 2023-10-14T18:18:08.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.655+0000 7f93737fe640 1 -- 172.21.15.29:0/3792805581 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v235) v1 ==== 107+0+11 (secure 0 0 0) 0x7f936c0792a0 con 0x7f937c145f70 2023-10-14T18:18:08.656 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:08.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.657+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 msgr2=0x7f935c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.657+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 0x7f935c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9364001200 tx=0x7f9364009040 comp rx=0 tx=0).stop 2023-10-14T18:18:08.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.657+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 msgr2=0x7f937c0af890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.657+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 secure :-1 s=READY pgs=2755 cs=0 l=1 rev1=1 crypto rx=0x7f936c002010 tx=0x7f936c0016a0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 shutdown_connections 2023-10-14T18:18:08.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f935c06acc0 0x7f935c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f937c0d0db0 0x7f937c15a6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f937c1465f0 0x7f937c0d0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 --2- 172.21.15.29:0/3792805581 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f937c145f70 0x7f937c0af890 unknown :-1 s=CLOSED pgs=2755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 >> 172.21.15.29:0/3792805581 conn(0x7f937c0d2cc0 msgr2=0x7f937c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:08.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 shutdown_connections 2023-10-14T18:18:08.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.658+0000 7f9380a2d640 1 -- 172.21.15.29:0/3792805581 wait complete. 2023-10-14T18:18:08.676 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:18:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.903+0000 7f7f54a85640 1 Processor -- start 2023-10-14T18:18:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.903+0000 7f7f54a85640 1 -- start start 2023-10-14T18:18:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.904+0000 7f7f54a85640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.904+0000 7f7f54a85640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f5006c810 0x7f7f500708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.904+0000 7f7f54a85640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f50070df0 con 0x7f7f50070450 2023-10-14T18:18:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.904+0000 7f7f54a85640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5015e490 con 0x7f7f5006c810 2023-10-14T18:18:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.904+0000 7f7f54a85640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5015aef0 con 0x7f7f50159950 2023-10-14T18:18:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.905+0000 7f7f4f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.905+0000 7f7f4ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f5006c810 0x7f7f500708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.905+0000 7f7f4ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7f5006c810 0x7f7f500708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:52382/0 (socket says 172.21.15.29:52382) 2023-10-14T18:18:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.905+0000 7f7f4f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:34658/0 (socket says 172.21.15.29:34658) 2023-10-14T18:18:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.905+0000 7f7f4ffff640 1 -- 172.21.15.29:0/2972068808 learned_addr learned my addr 172.21.15.29:0/2972068808 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.906+0000 7f7f4ffff640 1 -- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 msgr2=0x7f7f5015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.906+0000 7f7f4ffff640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.906+0000 7f7f4ffff640 1 -- 172.21.15.29:0/2972068808 >> v1:172.21.15.186:6789/0 conn(0x7f7f50070450 legacy=0x7f7f5006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.906+0000 7f7f4ffff640 1 -- 172.21.15.29:0/2972068808 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5006cc80 con 0x7f7f5006c810 2023-10-14T18:18:08.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.906+0000 7f7f4f7fe640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:18:08.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.907+0000 7f7f4ffff640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f5006c810 0x7f7f500708b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009e50 tx=0x7f7f4802bee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87f9b0c793a58730 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f4effd640 1 -- 172.21.15.29:0/2972068808 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4802ca80 con 0x7f7f5006c810 2023-10-14T18:18:08.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f4effd640 1 -- 172.21.15.29:0/2972068808 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4802cbe0 con 0x7f7f5006c810 2023-10-14T18:18:08.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f4effd640 1 -- 172.21.15.29:0/2972068808 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f480343f0 con 0x7f7f5006c810 2023-10-14T18:18:08.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 -- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f5006c810 msgr2=0x7f7f500708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f5006c810 0x7f7f500708b0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009e50 tx=0x7f7f4802bee0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 -- 172.21.15.29:0/2972068808 shutdown_connections 2023-10-14T18:18:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f5006c810 0x7f7f500708b0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 --2- 172.21.15.29:0/2972068808 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f50159950 0x7f7f5015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 -- 172.21.15.29:0/2972068808 >> 172.21.15.29:0/2972068808 conn(0x7f7f500ce6a0 msgr2=0x7f7f500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 -- 172.21.15.29:0/2972068808 shutdown_connections 2023-10-14T18:18:08.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.908+0000 7f7f54a85640 1 -- 172.21.15.29:0/2972068808 wait complete. 2023-10-14T18:18:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.910+0000 7f7f54a85640 1 Processor -- start 2023-10-14T18:18:08.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.910+0000 7f7f54a85640 1 -- start start 2023-10-14T18:18:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f5006c810 0x7f7f500c65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f50070450 0x7f7f5014fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500c9e60 con 0x7f7f50159950 2023-10-14T18:18:08.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500c5810 con 0x7f7f50070450 2023-10-14T18:18:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.911+0000 7f7f54a85640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f500c5af0 con 0x7f7f5006c810 2023-10-14T18:18:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34130/0 (socket says 172.21.15.29:34130) 2023-10-14T18:18:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 -- 172.21.15.29:0/4220688195 learned_addr learned my addr 172.21.15.29:0/4220688195 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:08.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4f7fe640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f5006c810 0x7f7f500c65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f47fff640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f50070450 0x7f7f5014fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 -- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f5006c810 msgr2=0x7f7f500c65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f5006c810 0x7f7f500c65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 -- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f50070450 msgr2=0x7f7f5014fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:08.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f50070450 0x7f7f5014fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.912+0000 7f7f4ffff640 1 -- 172.21.15.29:0/4220688195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f40009400 con 0x7f7f50159950 2023-10-14T18:18:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f4ffff640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009e50 tx=0x7f7f48004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f48009510 con 0x7f7f50159950 2023-10-14T18:18:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4803e450 con 0x7f7f50159950 2023-10-14T18:18:08.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f48009b30 con 0x7f7f50159950 2023-10-14T18:18:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7f4803e800 con 0x7f7f50159950 2023-10-14T18:18:08.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.913+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f500c60b0 con 0x7f7f50159950 2023-10-14T18:18:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.914+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f500c62d0 con 0x7f7f50159950 2023-10-14T18:18:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.915+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 145) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f7f4802cd50 con 0x7f7f50159950 2023-10-14T18:18:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.916+0000 7f7f4cff9640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 0x7f7f3006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:08.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.916+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f7f480ac450 con 0x7f7f50159950 2023-10-14T18:18:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.916+0000 7f7f4f7fe640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 0x7f7f3006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.917+0000 7f7f4f7fe640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 0x7f7f3006d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009c60 tx=0x7f7f40008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:08.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:08.921+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7f48077060 con 0x7f7f50159950 2023-10-14T18:18:09.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.114+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f7f5004e9b0 con 0x7f7f50159950 2023-10-14T18:18:09.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.115+0000 7f7f4cff9640 1 -- 172.21.15.29:0/4220688195 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v235) v1 ==== 103+0+11 (secure 0 0 0) 0x7f7f4807a680 con 0x7f7f50159950 2023-10-14T18:18:09.116 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 msgr2=0x7f7f3006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 0x7f7f3006d180 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7f40009c60 tx=0x7f7f40008040 comp rx=0 tx=0).stop 2023-10-14T18:18:09.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 msgr2=0x7f7f500c94f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 secure :-1 s=READY pgs=2758 cs=0 l=1 rev1=1 crypto rx=0x7f7f48009e50 tx=0x7f7f48004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:09.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 shutdown_connections 2023-10-14T18:18:09.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7f3006acc0 0x7f7f3006d180 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7f50159950 0x7f7f500c94f0 unknown :-1 s=CLOSED pgs=2758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7f50070450 0x7f7f5014fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 --2- 172.21.15.29:0/4220688195 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7f5006c810 0x7f7f500c65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 >> 172.21.15.29:0/4220688195 conn(0x7f7f500ce6a0 msgr2=0x7f7f5015b090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 shutdown_connections 2023-10-14T18:18:09.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.118+0000 7f7f54a85640 1 -- 172.21.15.29:0/4220688195 wait complete. 2023-10-14T18:18:09.138 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.359+0000 7f12b19ef640 1 Processor -- start 2023-10-14T18:18:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.359+0000 7f12b19ef640 1 -- start start 2023-10-14T18:18:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b19ef640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f12ac06c040 0x7f12ac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b19ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ac15f4a0 con 0x7f12ac0cda80 2023-10-14T18:18:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b19ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12ac06c960 con 0x7f12ac0ad970 2023-10-14T18:18:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b19ef640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12ac15b9d0 con 0x7f12ac06c040 2023-10-14T18:18:09.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b11ee640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f12ac0cda80 0x7f12ac15ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50884/0 (socket says 172.21.15.29:50884) 2023-10-14T18:18:09.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.361+0000 7f12b11ee640 1 -- 172.21.15.29:0/1630873132 learned_addr learned my addr 172.21.15.29:0/1630873132 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12b09ed640 1 --2- 172.21.15.29:0/1630873132 >> v2:172.21.15.186:3300/0 conn(0x7f12ac06c040 0x7f12ac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070745010 0 0) 0x7f12ac15f4a0 con 0x7f12ac0cda80 2023-10-14T18:18:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c003520 con 0x7f12ac0cda80 2023-10-14T18:18:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423033146 0 0) 0x7f129c003520 con 0x7f12ac0cda80 2023-10-14T18:18:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 >> v2:172.21.15.186:3300/0 conn(0x7f12ac06c040 msgr2=0x7f12ac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 --2- 172.21.15.29:0/1630873132 >> v2:172.21.15.186:3300/0 conn(0x7f12ac06c040 0x7f12ac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 >> v1:172.21.15.186:6789/0 conn(0x7f12ac0ad970 legacy=0x7f12ac15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac15bcb0 con 0x7f12ac0cda80 2023-10-14T18:18:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f12a40019f0 con 0x7f12ac0cda80 2023-10-14T18:18:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 >> v1:172.21.15.29:6789/0 conn(0x7f12ac0cda80 legacy=0x7f12ac15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f129c0039c0 0x7f129c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f129c0047c0 0x7f129c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f129c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac15b9d0 con 0x7f129c005180 2023-10-14T18:18:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00c080 con 0x7f129c0039c0 2023-10-14T18:18:09.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.362+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c00c240 con 0x7f129c0047c0 2023-10-14T18:18:09.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f129c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b09ed640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f129c0039c0 0x7f129c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12abfff640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f129c0047c0 0x7f129c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 -- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f129c0047c0 msgr2=0x7f129c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:09.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f129c0047c0 0x7f129c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 -- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f129c0039c0 msgr2=0x7f129c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f129c0039c0 0x7f129c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b11ee640 1 -- 172.21.15.29:0/1630873132 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c003520 con 0x7f129c005180 2023-10-14T18:18:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.363+0000 7f12b09ed640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f129c0039c0 0x7f129c003de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.364+0000 7f12b11ee640 1 -- 172.21.15.29:0/1630873132 reap_dead start 2023-10-14T18:18:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.364+0000 7f12b11ee640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f129c00b980 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f12a400a760 tx=0x7f129c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcff920b69badc9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4034630 con 0x7f129c005180 2023-10-14T18:18:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a402f070 con 0x7f129c005180 2023-10-14T18:18:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12ab7fe640 1 -- 172.21.15.29:0/1630873132 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a4034a10 con 0x7f129c005180 2023-10-14T18:18:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 -- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 msgr2=0x7f129c00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f129c00b980 secure :-1 s=READY pgs=2760 cs=0 l=1 rev1=1 crypto rx=0x7f12a400a760 tx=0x7f129c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 -- 172.21.15.29:0/1630873132 shutdown_connections 2023-10-14T18:18:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 --2- 172.21.15.29:0/1630873132 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f129c00b980 unknown :-1 s=CLOSED pgs=2760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 -- 172.21.15.29:0/1630873132 >> 172.21.15.29:0/1630873132 conn(0x7f12ac0ce6a0 msgr2=0x7f12ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.365+0000 7f12b19ef640 1 -- 172.21.15.29:0/1630873132 shutdown_connections 2023-10-14T18:18:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.366+0000 7f12b19ef640 1 -- 172.21.15.29:0/1630873132 wait complete. 2023-10-14T18:18:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.367+0000 7f12b19ef640 1 Processor -- start 2023-10-14T18:18:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.368+0000 7f12b19ef640 1 -- start start 2023-10-14T18:18:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.368+0000 7f12b19ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b19ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac0c9150 0x7f12ac0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b19ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12ac0c95d0 0x7f12ac0acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b19ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac14e070 con 0x7f129c005180 2023-10-14T18:18:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b19ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0c9b30 con 0x7f12ac0c9150 2023-10-14T18:18:09.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b19ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12ac0c9e10 con 0x7f12ac0c95d0 2023-10-14T18:18:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34150/0 (socket says 172.21.15.29:34150) 2023-10-14T18:18:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 -- 172.21.15.29:0/2356362919 learned_addr learned my addr 172.21.15.29:0/2356362919 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 -- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12ac0c95d0 msgr2=0x7f12ac0acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12ac0c95d0 0x7f12ac0acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 -- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac0c9150 msgr2=0x7f12ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac0c9150 0x7f12ac0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.369+0000 7f12b09ed640 1 -- 172.21.15.29:0/2356362919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ac0ca0f0 con 0x7f129c005180 2023-10-14T18:18:09.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12b09ed640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f12a0011540 tx=0x7f12a0011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0015870 con 0x7f129c005180 2023-10-14T18:18:09.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12a00159d0 con 0x7f129c005180 2023-10-14T18:18:09.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12a0015d40 con 0x7f129c005180 2023-10-14T18:18:09.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ac0ca3d0 con 0x7f129c005180 2023-10-14T18:18:09.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.370+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ac0cabd0 con 0x7f129c005180 2023-10-14T18:18:09.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.371+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ac06b330 con 0x7f129c005180 2023-10-14T18:18:09.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.372+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f12a000f460 con 0x7f129c005180 2023-10-14T18:18:09.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.373+0000 7f12a97fa640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 0x7f129006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.373+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f12a008a810 con 0x7f129c005180 2023-10-14T18:18:09.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.374+0000 7f12abfff640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 0x7f129006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.374+0000 7f12abfff640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 0x7f129006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f12ac06b950 tx=0x7f1298009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.378+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f12a0054f20 con 0x7f129c005180 2023-10-14T18:18:09.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.617+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12ac04e9b0 con 0x7f129c005180 2023-10-14T18:18:09.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.618+0000 7f12a97fa640 1 -- 172.21.15.29:0/2356362919 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7f12a0058540 con 0x7f129c005180 2023-10-14T18:18:09.620 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:09.620 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:09.620 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:18:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 msgr2=0x7f129006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 0x7f129006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f12ac06b950 tx=0x7f1298009040 comp rx=0 tx=0).stop 2023-10-14T18:18:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 msgr2=0x7f12ac14db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 secure :-1 s=READY pgs=2761 cs=0 l=1 rev1=1 crypto rx=0x7f12a0011540 tx=0x7f12a0011a00 comp rx=0 tx=0).stop 2023-10-14T18:18:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 shutdown_connections 2023-10-14T18:18:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f129006b020 0x7f129006d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f12ac0c95d0 0x7f12ac0acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f12ac0c9150 0x7f12ac0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 --2- 172.21.15.29:0/2356362919 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f129c005180 0x7f12ac14db30 unknown :-1 s=CLOSED pgs=2761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 >> 172.21.15.29:0/2356362919 conn(0x7f12ac0ce6a0 msgr2=0x7f12ac15dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:09.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 shutdown_connections 2023-10-14T18:18:09.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.621+0000 7f12b19ef640 1 -- 172.21.15.29:0/2356362919 wait complete. 2023-10-14T18:18:09.643 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:09.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.869+0000 7fd07759e640 1 Processor -- start 2023-10-14T18:18:09.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.869+0000 7fd07759e640 1 -- start start 2023-10-14T18:18:09.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd07759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd0780c9060 0x7fd07806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd07759e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0780ca560 con 0x7fd07806c9e0 2023-10-14T18:18:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd07759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd078165b70 con 0x7fd0780c9440 2023-10-14T18:18:09.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd07759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0780c9950 con 0x7fd0780c9060 2023-10-14T18:18:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd06ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd07806c9e0 0x7fd078164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50896/0 (socket says 172.21.15.29:50896) 2023-10-14T18:18:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd06ffff640 1 -- 172.21.15.29:0/609237268 learned_addr learned my addr 172.21.15.29:0/609237268 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:09.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.871+0000 7fd07659c640 1 --2- 172.21.15.29:0/609237268 >> v2:172.21.15.186:3300/0 conn(0x7fd0780c9060 0x7fd07806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.872+0000 7fd075d9b640 1 -- 172.21.15.29:0/609237268 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011654655 0 0) 0x7fd0780ca560 con 0x7fd07806c9e0 2023-10-14T18:18:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.872+0000 7fd075d9b640 1 -- 172.21.15.29:0/609237268 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd064003520 con 0x7fd07806c9e0 2023-10-14T18:18:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.872+0000 7fd07659c640 1 -- 172.21.15.29:0/609237268 >> v1:172.21.15.186:6789/0 conn(0x7fd0780c9440 legacy=0x7fd078165460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.872+0000 7fd07659c640 1 -- 172.21.15.29:0/609237268 >> v1:172.21.15.29:6789/0 conn(0x7fd07806c9e0 legacy=0x7fd078164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.872+0000 7fd07659c640 1 -- 172.21.15.29:0/609237268 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0780c9c30 con 0x7fd0780c9060 2023-10-14T18:18:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.874+0000 7fd07659c640 1 --2- 172.21.15.29:0/609237268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9060 0x7fd07806f440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd0680097a0 tx=0x7fd06802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2319e8086739d19d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.874+0000 7fd075d9b640 1 -- 172.21.15.29:0/609237268 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068030610 con 0x7fd0780c9060 2023-10-14T18:18:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.874+0000 7fd075d9b640 1 -- 172.21.15.29:0/609237268 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd068030770 con 0x7fd0780c9060 2023-10-14T18:18:09.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.874+0000 7fd075d9b640 1 -- 172.21.15.29:0/609237268 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd068030b00 con 0x7fd0780c9060 2023-10-14T18:18:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 -- 172.21.15.29:0/609237268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9060 msgr2=0x7fd07806f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 --2- 172.21.15.29:0/609237268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9060 0x7fd07806f440 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fd0680097a0 tx=0x7fd06802b960 comp rx=0 tx=0).stop 2023-10-14T18:18:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 -- 172.21.15.29:0/609237268 shutdown_connections 2023-10-14T18:18:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 --2- 172.21.15.29:0/609237268 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9060 0x7fd07806f440 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 -- 172.21.15.29:0/609237268 >> 172.21.15.29:0/609237268 conn(0x7fd0780ce6a0 msgr2=0x7fd0780ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 -- 172.21.15.29:0/609237268 shutdown_connections 2023-10-14T18:18:09.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.875+0000 7fd07759e640 1 -- 172.21.15.29:0/609237268 wait complete. 2023-10-14T18:18:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.877+0000 7fd07759e640 1 Processor -- start 2023-10-14T18:18:09.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.877+0000 7fd07759e640 1 -- start start 2023-10-14T18:18:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07806c9e0 0x7fd0780c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9440 0x7fd07806a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd078162da0 con 0x7fd0780c9060 2023-10-14T18:18:09.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0781f7470 con 0x7fd0780c9440 2023-10-14T18:18:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd07759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0781626e0 con 0x7fd07806c9e0 2023-10-14T18:18:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd06ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34160/0 (socket says 172.21.15.29:34160) 2023-10-14T18:18:09.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.878+0000 7fd06ffff640 1 -- 172.21.15.29:0/3536325486 learned_addr learned my addr 172.21.15.29:0/3536325486 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 -- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07806c9e0 msgr2=0x7fd0780c7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd07659c640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07806c9e0 0x7fd0780c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd076d9d640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9440 0x7fd07806a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07806c9e0 0x7fd0780c7eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 -- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9440 msgr2=0x7fd07806a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9440 0x7fd07806a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 -- 172.21.15.29:0/3536325486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd068009400 con 0x7fd0780c9060 2023-10-14T18:18:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06ffff640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fd06000c3e0 tx=0x7fd06000c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd060005ce0 con 0x7fd0780c9060 2023-10-14T18:18:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd060005e40 con 0x7fd0780c9060 2023-10-14T18:18:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.879+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0600072d0 con 0x7fd0780c9060 2023-10-14T18:18:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.880+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0781629c0 con 0x7fd0780c9060 2023-10-14T18:18:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.880+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0781631c0 con 0x7fd0780c9060 2023-10-14T18:18:09.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.881+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd078162ee0 con 0x7fd0780c9060 2023-10-14T18:18:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.882+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fd0600074f0 con 0x7fd0780c9060 2023-10-14T18:18:09.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.883+0000 7fd06effd640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 0x7fd05806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.883+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7fd0600894e0 con 0x7fd0780c9060 2023-10-14T18:18:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.883+0000 7fd07659c640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 0x7fd05806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:09.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.884+0000 7fd07659c640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 0x7fd05806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd068036000 tx=0x7fd06803b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:09.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:09.887+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd078162ee0 con 0x7fd0780c9060 2023-10-14T18:18:10.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.085+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd078162ee0 con 0x7fd0780c9060 2023-10-14T18:18:10.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.089+0000 7fd06effd640 1 -- 172.21.15.29:0/3536325486 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+21667 (secure 0 0 0) 0x7fd078162ee0 con 0x7fd0780c9060 2023-10-14T18:18:10.090 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:10.090 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":235,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:06.375225+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 msgr2=0x7fd05806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 0x7fd05806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fd068036000 tx=0x7fd06803b040 comp rx=0 tx=0).stop 2023-10-14T18:18:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 msgr2=0x7fd078069e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 secure :-1 s=READY pgs=2763 cs=0 l=1 rev1=1 crypto rx=0x7fd06000c3e0 tx=0x7fd06000c8a0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 shutdown_connections 2023-10-14T18:18:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd05806b020 0x7fd05806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd0780c9440 0x7fd07806a3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd0780c9060 0x7fd078069e80 unknown :-1 s=CLOSED pgs=2763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 --2- 172.21.15.29:0/3536325486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07806c9e0 0x7fd0780c7eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.091+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 >> 172.21.15.29:0/3536325486 conn(0x7fd0780ce6a0 msgr2=0x7fd0780ccef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:10.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.092+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 shutdown_connections 2023-10-14T18:18:10.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.092+0000 7fd07759e640 1 -- 172.21.15.29:0/3536325486 wait complete. 2023-10-14T18:18:10.112 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:18:10.113 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:10.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.330+0000 7f2528b47640 1 Processor -- start 2023-10-14T18:18:10.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.331+0000 7f2528b47640 1 -- start start 2023-10-14T18:18:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2528b47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f25240cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2528b47640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252415b9d0 con 0x7f25240ccfb0 2023-10-14T18:18:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2528b47640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f252415bcb0 con 0x7f252406c040 2023-10-14T18:18:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2528b47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f252415bf90 con 0x7f25240ad970 2023-10-14T18:18:10.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2523fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f25240ccfb0 0x7f25240cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50910/0 (socket says 172.21.15.29:50910) 2023-10-14T18:18:10.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.332+0000 7f2523fff640 1 -- 172.21.15.29:0/631701882 learned_addr learned my addr 172.21.15.29:0/631701882 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f2522ffd640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f25240cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741257671 0 0) 0x7f252415b9d0 con 0x7f25240ccfb0 2023-10-14T18:18:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f250c003520 con 0x7f25240ccfb0 2023-10-14T18:18:10.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285639209 0 0) 0x7f250c003520 con 0x7f25240ccfb0 2023-10-14T18:18:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 msgr2=0x7f25240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f25240cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 >> v1:172.21.15.186:6789/0 conn(0x7f252406c040 legacy=0x7f252406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25240cc270 con 0x7f25240ccfb0 2023-10-14T18:18:10.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f251c0019f0 con 0x7f25240ccfb0 2023-10-14T18:18:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 >> v1:172.21.15.29:6789/0 conn(0x7f25240ccfb0 legacy=0x7f25240cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f250c0039a0 0x7f250c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f250c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c004d00 0x7f250c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252415bf90 con 0x7f250c0039a0 2023-10-14T18:18:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00ba60 con 0x7f250c004d00 2023-10-14T18:18:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.333+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f250c00bc20 con 0x7f25240ad970 2023-10-14T18:18:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f250c0039a0 0x7f250c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f2523fff640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c004d00 0x7f250c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f2522ffd640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f250c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 -- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 msgr2=0x7f250c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240ad970 0x7f250c0047c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 -- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c004d00 msgr2=0x7f250c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c004d00 0x7f250c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f25237fe640 1 -- 172.21.15.29:0/631701882 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f250c003520 con 0x7f250c0039a0 2023-10-14T18:18:10.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.334+0000 7f2523fff640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c004d00 0x7f250c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:10.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.335+0000 7f25237fe640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f250c0039a0 0x7f250c003dc0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f2514001750 tx=0x7f250c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c5166f8573a52a0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.335+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140052d0 con 0x7f250c0039a0 2023-10-14T18:18:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.335+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251402e050 con 0x7f250c0039a0 2023-10-14T18:18:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.335+0000 7f25227fc640 1 -- 172.21.15.29:0/631701882 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25140035f0 con 0x7f250c0039a0 2023-10-14T18:18:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 -- 172.21.15.29:0/631701882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f250c0039a0 msgr2=0x7f250c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 --2- 172.21.15.29:0/631701882 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f250c0039a0 0x7f250c003dc0 secure :-1 s=READY pgs=2765 cs=0 l=1 rev1=1 crypto rx=0x7f2514001750 tx=0x7f250c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2523fff640 1 -- 172.21.15.29:0/631701882 reap_dead start 2023-10-14T18:18:10.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 -- 172.21.15.29:0/631701882 shutdown_connections 2023-10-14T18:18:10.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 -- 172.21.15.29:0/631701882 >> 172.21.15.29:0/631701882 conn(0x7f25240ce6a0 msgr2=0x7f25240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:10.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 -- 172.21.15.29:0/631701882 shutdown_connections 2023-10-14T18:18:10.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.336+0000 7f2528b47640 1 -- 172.21.15.29:0/631701882 wait complete. 2023-10-14T18:18:10.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.338+0000 7f2528b47640 1 Processor -- start 2023-10-14T18:18:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.338+0000 7f2528b47640 1 -- start start 2023-10-14T18:18:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c0039a0 0x7f2524153db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240c67c0 0x7f2524156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2524155740 con 0x7f25241570b0 2023-10-14T18:18:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25240ca4b0 con 0x7f250c0039a0 2023-10-14T18:18:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2528b47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25240ad090 con 0x7f25240c67c0 2023-10-14T18:18:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2523fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2523fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34170/0 (socket says 172.21.15.29:34170) 2023-10-14T18:18:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.339+0000 7f2523fff640 1 -- 172.21.15.29:0/1221076848 learned_addr learned my addr 172.21.15.29:0/1221076848 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 -- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240c67c0 msgr2=0x7f2524156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f25237fe640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c0039a0 0x7f2524153db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2522ffd640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240c67c0 0x7f2524156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240c67c0 0x7f2524156b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 -- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c0039a0 msgr2=0x7f2524153db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c0039a0 0x7f2524153db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 -- 172.21.15.29:0/1221076848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25240cdcd0 con 0x7f25241570b0 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2523fff640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f25240b1290 tx=0x7f251c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c004d60 con 0x7f25241570b0 2023-10-14T18:18:10.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c00bb80 con 0x7f25241570b0 2023-10-14T18:18:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c007070 con 0x7f25241570b0 2023-10-14T18:18:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.340+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25240ad370 con 0x7f25241570b0 2023-10-14T18:18:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.341+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25241553d0 con 0x7f25241570b0 2023-10-14T18:18:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.342+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25240c6ba0 con 0x7f25241570b0 2023-10-14T18:18:10.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.343+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 146) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f251c0071d0 con 0x7f25241570b0 2023-10-14T18:18:10.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.344+0000 7f250bfff640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 0x7f251006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.344+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f251c08a350 con 0x7f25241570b0 2023-10-14T18:18:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.344+0000 7f25237fe640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 0x7f251006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.345+0000 7f25237fe640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 0x7f251006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2514033000 tx=0x7f2514009030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.348+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f25240c6ba0 con 0x7f25241570b0 2023-10-14T18:18:10.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.427+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f251c054c90 con 0x7f25241570b0 2023-10-14T18:18:10.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.586+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25240c6ba0 con 0x7f25241570b0 2023-10-14T18:18:10.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.588+0000 7f250bfff640 1 -- 172.21.15.29:0/1221076848 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7f25240c6ba0 con 0x7f25241570b0 2023-10-14T18:18:10.589 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:10.589 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:10.590 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:18:10.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 msgr2=0x7f251006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 0x7f251006d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f2514033000 tx=0x7f2514009030 comp rx=0 tx=0).stop 2023-10-14T18:18:10.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 msgr2=0x7f252415c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 secure :-1 s=READY pgs=2766 cs=0 l=1 rev1=1 crypto rx=0x7f25240b1290 tx=0x7f251c00b4a0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 shutdown_connections 2023-10-14T18:18:10.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f251006b020 0x7f251006d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f25241570b0 0x7f252415c950 unknown :-1 s=CLOSED pgs=2766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25240c67c0 0x7f2524156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 --2- 172.21.15.29:0/1221076848 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f250c0039a0 0x7f2524153db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 >> 172.21.15.29:0/1221076848 conn(0x7f25240ce6a0 msgr2=0x7f25240cc790 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:10.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.590+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 shutdown_connections 2023-10-14T18:18:10.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.591+0000 7f2528b47640 1 -- 172.21.15.29:0/1221076848 wait complete. 2023-10-14T18:18:10.611 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:10.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.830+0000 7f3a22116640 1 Processor -- start 2023-10-14T18:18:10.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.831+0000 7f3a22116640 1 -- start start 2023-10-14T18:18:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.832+0000 7f3a22116640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.832+0000 7f3a22116640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3a1c06c810 0x7f3a1c0708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.832+0000 7f3a22116640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a1c070df0 con 0x7f3a1c070450 2023-10-14T18:18:10.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.832+0000 7f3a22116640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a1c15e490 con 0x7f3a1c06c810 2023-10-14T18:18:10.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.832+0000 7f3a22116640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c15aef0 con 0x7f3a1c159950 2023-10-14T18:18:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a21114640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a21915640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3a1c06c810 0x7f3a1c0708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a21114640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:34734/0 (socket says 172.21.15.29:34734) 2023-10-14T18:18:10.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a21915640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3a1c06c810 0x7f3a1c0708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:52468/0 (socket says 172.21.15.29:52468) 2023-10-14T18:18:10.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a21114640 1 -- 172.21.15.29:0/1471581423 learned_addr learned my addr 172.21.15.29:0/1471581423 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:10.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.833+0000 7f3a20913640 1 --1- 172.21.15.29:0/1471581423 >> v1:172.21.15.186:6789/0 conn(0x7f3a1c070450 0x7f3a1c06c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:54270/0 (socket says 172.21.15.29:54270) 2023-10-14T18:18:10.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.834+0000 7f3a21915640 1 -- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 msgr2=0x7f3a1c15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:10.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.834+0000 7f3a21915640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.834+0000 7f3a21915640 1 -- 172.21.15.29:0/1471581423 >> v1:172.21.15.186:6789/0 conn(0x7f3a1c070450 legacy=0x7f3a1c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.834+0000 7f3a21915640 1 -- 172.21.15.29:0/1471581423 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c06cc80 con 0x7f3a1c06c810 2023-10-14T18:18:10.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.834+0000 7f3a21114640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:18:10.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.835+0000 7f3a21915640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0708b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009d90 tx=0x7f3a1402be10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=285cf9c33eafa811 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.836+0000 7f3a13fff640 1 -- 172.21.15.29:0/1471581423 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1402cbc0 con 0x7f3a1c06c810 2023-10-14T18:18:10.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.836+0000 7f3a13fff640 1 -- 172.21.15.29:0/1471581423 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1402cd20 con 0x7f3a1c06c810 2023-10-14T18:18:10.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.836+0000 7f3a13fff640 1 -- 172.21.15.29:0/1471581423 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14034660 con 0x7f3a1c06c810 2023-10-14T18:18:10.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 -- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c06c810 msgr2=0x7f3a1c0708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0708b0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009d90 tx=0x7f3a1402be10 comp rx=0 tx=0).stop 2023-10-14T18:18:10.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 -- 172.21.15.29:0/1471581423 shutdown_connections 2023-10-14T18:18:10.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0708b0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 --2- 172.21.15.29:0/1471581423 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.837+0000 7f3a22116640 1 -- 172.21.15.29:0/1471581423 >> 172.21.15.29:0/1471581423 conn(0x7f3a1c0ce6a0 msgr2=0x7f3a1c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:10.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.838+0000 7f3a22116640 1 -- 172.21.15.29:0/1471581423 shutdown_connections 2023-10-14T18:18:10.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.838+0000 7f3a22116640 1 -- 172.21.15.29:0/1471581423 wait complete. 2023-10-14T18:18:10.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.839+0000 7f3a22116640 1 Processor -- start 2023-10-14T18:18:10.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.840+0000 7f3a22116640 1 -- start start 2023-10-14T18:18:10.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 0x7f3a1c0c3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c14bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0afd30 con 0x7f3a1c06c810 2023-10-14T18:18:10.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b0010 con 0x7f3a1c070450 2023-10-14T18:18:10.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a22116640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c0b02f0 con 0x7f3a1c159950 2023-10-14T18:18:10.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34182/0 (socket says 172.21.15.29:34182) 2023-10-14T18:18:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 -- 172.21.15.29:0/2137443123 learned_addr learned my addr 172.21.15.29:0/2137443123 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a20913640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 0x7f3a1c0c3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21915640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c14bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 -- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 msgr2=0x7f3a1c14bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c14bc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 -- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 msgr2=0x7f3a1c0c3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:10.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 0x7f3a1c0c3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:10.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21114640 1 -- 172.21.15.29:0/2137443123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a180094d0 con 0x7f3a1c06c810 2023-10-14T18:18:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.841+0000 7f3a21915640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c14bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a20913640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 0x7f3a1c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a21114640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800e9d0 tx=0x7f3a1800ee90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1800ca60 con 0x7f3a1c06c810 2023-10-14T18:18:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a1800cbc0 con 0x7f3a1c06c810 2023-10-14T18:18:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a14009a70 con 0x7f3a1c06c810 2023-10-14T18:18:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a18013470 con 0x7f3a1c06c810 2023-10-14T18:18:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.842+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c0b08b0 con 0x7f3a1c06c810 2023-10-14T18:18:10.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.844+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a1c0b0a70 con 0x7f3a1c06c810 2023-10-14T18:18:10.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.844+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3a180135d0 con 0x7f3a1c06c810 2023-10-14T18:18:10.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.845+0000 7f3a11ffb640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 0x7f3a0006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.845+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f3a1808a400 con 0x7f3a1c06c810 2023-10-14T18:18:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.846+0000 7f3a20913640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 0x7f3a0006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.847+0000 7f3a20913640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 0x7f3a0006d5a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0ad750 tx=0x7f3a0c009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:10.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:10.850+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a18054a50 con 0x7f3a1c06c810 2023-10-14T18:18:11.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.083+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a1c04e9b0 con 0x7f3a1c06c810 2023-10-14T18:18:11.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.084+0000 7f3a11ffb640 1 -- 172.21.15.29:0/2137443123 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+6158 (secure 0 0 0) 0x7f3a18058070 con 0x7f3a1c06c810 2023-10-14T18:18:11.085 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:11.086 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":113,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102}],"filesystems":[{"mdsmap":{"epoch":107,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:27.215719+0000","modified":"2023-10-14T18:17:30.282217+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11268":{"gid":11268,"name":"a","rank":0,"incarnation":103,"state":"up:active","state_seq":3,"addr":"172.21.15.29:6837/1082656487","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1082656487},{"type":"v1","addr":"172.21.15.29:6837","nonce":1082656487}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[27],"metadata_pool":26,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":13},{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:11.086 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 113 2023-10-14T18:18:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 msgr2=0x7f3a0006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 0x7f3a0006d5a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0ad750 tx=0x7f3a0c009040 comp rx=0 tx=0).stop 2023-10-14T18:18:11.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 msgr2=0x7f3a1c0af750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 secure :-1 s=READY pgs=2767 cs=0 l=1 rev1=1 crypto rx=0x7f3a1800e9d0 tx=0x7f3a1800ee90 comp rx=0 tx=0).stop 2023-10-14T18:18:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 shutdown_connections 2023-10-14T18:18:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3a0006b0e0 0x7f3a0006d5a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a1c159950 0x7f3a1c14bc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a1c070450 0x7f3a1c0c3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 --2- 172.21.15.29:0/2137443123 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a1c06c810 0x7f3a1c0af750 unknown :-1 s=CLOSED pgs=2767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 >> 172.21.15.29:0/2137443123 conn(0x7f3a1c0ce6a0 msgr2=0x7f3a1c157da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 shutdown_connections 2023-10-14T18:18:11.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.087+0000 7f3a22116640 1 -- 172.21.15.29:0/2137443123 wait complete. 2023-10-14T18:18:11.108 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.338+0000 7f8fd9fac640 1 Processor -- start 2023-10-14T18:18:11.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.338+0000 7f8fd9fac640 1 -- start start 2023-10-14T18:18:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.339+0000 7f8fd9fac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd406c040 0x7f8fd406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.339+0000 7f8fd9fac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fd40cda80 0x7f8fd415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.339+0000 7f8fd9fac640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fd406c960 con 0x7f8fd40ad970 2023-10-14T18:18:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.339+0000 7f8fd9fac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fd415b9d0 con 0x7f8fd40cda80 2023-10-14T18:18:11.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.339+0000 7f8fd9fac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd415bcb0 con 0x7f8fd406c040 2023-10-14T18:18:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fd40cda80 0x7f8fd415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8fd40cda80 0x7f8fd415ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42638/0 (socket says 172.21.15.29:42638) 2023-10-14T18:18:11.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 -- 172.21.15.29:0/195496065 learned_addr learned my addr 172.21.15.29:0/195496065 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd8faa640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd406c040 0x7f8fd406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 -- 172.21.15.29:0/195496065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd406c040 msgr2=0x7f8fd406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd406c040 0x7f8fd406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 -- 172.21.15.29:0/195496065 >> v1:172.21.15.186:6789/0 conn(0x7f8fd40ad970 legacy=0x7f8fd415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:11.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.340+0000 7f8fd97ab640 1 -- 172.21.15.29:0/195496065 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fd415bf90 con 0x7f8fd40cda80 2023-10-14T18:18:11.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.341+0000 7f8fd97ab640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd40cda80 0x7f8fd415ed90 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cd2e70ac133d696 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.342+0000 7f8fd37fe640 1 -- 172.21.15.29:0/195496065 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc031640 con 0x7f8fd40cda80 2023-10-14T18:18:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.342+0000 7f8fd37fe640 1 -- 172.21.15.29:0/195496065 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fcc0317a0 con 0x7f8fd40cda80 2023-10-14T18:18:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.342+0000 7f8fd37fe640 1 -- 172.21.15.29:0/195496065 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fcc031b30 con 0x7f8fd40cda80 2023-10-14T18:18:11.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.342+0000 7f8fd9fac640 1 -- 172.21.15.29:0/195496065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd40cda80 msgr2=0x7f8fd415ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.342+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd40cda80 0x7f8fd415ed90 secure :-1 s=READY pgs=2768 cs=0 l=1 rev1=1 crypto rx=0x7f8fcc009870 tx=0x7f8fcc02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 -- 172.21.15.29:0/195496065 shutdown_connections 2023-10-14T18:18:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd40cda80 0x7f8fd415ed90 unknown :-1 s=CLOSED pgs=2768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/195496065 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd406c040 0x7f8fd406c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 -- 172.21.15.29:0/195496065 >> 172.21.15.29:0/195496065 conn(0x7f8fd40ce6a0 msgr2=0x7f8fd40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:11.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 -- 172.21.15.29:0/195496065 shutdown_connections 2023-10-14T18:18:11.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.343+0000 7f8fd9fac640 1 -- 172.21.15.29:0/195496065 wait complete. 2023-10-14T18:18:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.344+0000 7f8fd9fac640 1 Processor -- start 2023-10-14T18:18:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.345+0000 7f8fd9fac640 1 -- start start 2023-10-14T18:18:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd40ad970 0x7f8fd4150ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd40cda80 0x7f8fd414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd40c47f0 con 0x7f8fd406c040 2023-10-14T18:18:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4154f80 con 0x7f8fd40ad970 2023-10-14T18:18:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd9fac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fd4151910 con 0x7f8fd40cda80 2023-10-14T18:18:11.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42646/0 (socket says 172.21.15.29:42646) 2023-10-14T18:18:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 -- 172.21.15.29:0/1344610248 learned_addr learned my addr 172.21.15.29:0/1344610248 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 -- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd40cda80 msgr2=0x7f8fd414f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd40cda80 0x7f8fd414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 -- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd40ad970 msgr2=0x7f8fd4150ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd40ad970 0x7f8fd4150ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.346+0000 7f8fd8faa640 1 -- 172.21.15.29:0/1344610248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fcc0094d0 con 0x7f8fd406c040 2023-10-14T18:18:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd8faa640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4007e40 tx=0x7f8fc400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc4014620 con 0x7f8fd406c040 2023-10-14T18:18:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc400ee90 con 0x7f8fd406c040 2023-10-14T18:18:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fd4151bf0 con 0x7f8fd406c040 2023-10-14T18:18:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8fc40149c0 con 0x7f8fd406c040 2023-10-14T18:18:11.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.347+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fd41521b0 con 0x7f8fd406c040 2023-10-14T18:18:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.349+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fd406b330 con 0x7f8fd406c040 2023-10-14T18:18:11.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.350+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8fc4014b20 con 0x7f8fd406c040 2023-10-14T18:18:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.350+0000 7f8fd17fa640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 0x7f8fb406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.351+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f8fc408a930 con 0x7f8fd406c040 2023-10-14T18:18:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.351+0000 7f8fd3fff640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 0x7f8fb406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.352+0000 7f8fd3fff640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 0x7f8fb406d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40c7c40 tx=0x7f8fbc0079b0 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.355+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8fc40550d0 con 0x7f8fd406c040 2023-10-14T18:18:11.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.558+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8fd41523a0 con 0x7f8fd406c040 2023-10-14T18:18:11.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.561+0000 7f8fd17fa640 1 -- 172.21.15.29:0/1344610248 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v235) v1 ==== 74+0+21667 (secure 0 0 0) 0x7f8fc40586f0 con 0x7f8fd406c040 2023-10-14T18:18:11.563 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:11.564 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":235,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:06.375225+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":26,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:17:25.200413+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"228","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":27,"pool_name":"cephfs_data","create_time":"2023-10-14T18:17:26.251106+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"229","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.563+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 msgr2=0x7f8fb406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 0x7f8fb406d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8fd40c7c40 tx=0x7f8fbc0079b0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 msgr2=0x7f8fd40c42b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 secure :-1 s=READY pgs=2769 cs=0 l=1 rev1=1 crypto rx=0x7f8fc4007e40 tx=0x7f8fc400e4b0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 shutdown_connections 2023-10-14T18:18:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8fb406b0b0 0x7f8fb406d570 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8fd40cda80 0x7f8fd414f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8fd40ad970 0x7f8fd4150ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 --2- 172.21.15.29:0/1344610248 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8fd406c040 0x7f8fd40c42b0 unknown :-1 s=CLOSED pgs=2769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 >> 172.21.15.29:0/1344610248 conn(0x7f8fd40ce6a0 msgr2=0x7f8fd415dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 shutdown_connections 2023-10-14T18:18:11.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.564+0000 7f8fd9fac640 1 -- 172.21.15.29:0/1344610248 wait complete. 2023-10-14T18:18:11.587 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T18:18:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.812+0000 7f3c5ed6c640 1 Processor -- start 2023-10-14T18:18:11.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.812+0000 7f3c5ed6c640 1 -- start start 2023-10-14T18:18:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5ed6c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3c5815c9f0 0x7f3c580c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5ed6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5ed6c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c58160a90 con 0x7f3c580c9230 2023-10-14T18:18:11.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5ed6c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c580cdec0 con 0x7f3c5815c9f0 2023-10-14T18:18:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5ed6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580cd900 con 0x7f3c5815dde0 2023-10-14T18:18:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5d569640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3c580c9230 0x7f3c580709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59780/0 (socket says 172.21.15.29:59780) 2023-10-14T18:18:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5d569640 1 -- 172.21.15.29:0/2530898486 learned_addr learned my addr 172.21.15.29:0/2530898486 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.814+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/2530898486 >> v2:172.21.15.29:3300/0 conn(0x7f3c5815c9f0 0x7f3c580c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5cd68640 1 -- 172.21.15.29:0/2530898486 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870494047 0 0) 0x7f3c58160a90 con 0x7f3c580c9230 2023-10-14T18:18:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5cd68640 1 -- 172.21.15.29:0/2530898486 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c44003520 con 0x7f3c580c9230 2023-10-14T18:18:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5e56b640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/2530898486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 msgr2=0x7f3c580ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/2530898486 >> v1:172.21.15.29:6789/0 conn(0x7f3c580c9230 legacy=0x7f3c580709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.815+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/2530898486 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c580cdbe0 con 0x7f3c5815c9f0 2023-10-14T18:18:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.817+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c5815c9f0 0x7f3c580c8cf0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f3c540097a0 tx=0x7f3c5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5b75a43fa4f6db server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.817+0000 7f3c5cd68640 1 -- 172.21.15.29:0/2530898486 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54031830 con 0x7f3c5815c9f0 2023-10-14T18:18:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.817+0000 7f3c5cd68640 1 -- 172.21.15.29:0/2530898486 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c54031990 con 0x7f3c5815c9f0 2023-10-14T18:18:11.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.817+0000 7f3c5cd68640 1 -- 172.21.15.29:0/2530898486 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c54031d20 con 0x7f3c5815c9f0 2023-10-14T18:18:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/2530898486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c5815c9f0 msgr2=0x7f3c580c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c5815c9f0 0x7f3c580c8cf0 secure :-1 s=READY pgs=2771 cs=0 l=1 rev1=1 crypto rx=0x7f3c540097a0 tx=0x7f3c5402b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/2530898486 shutdown_connections 2023-10-14T18:18:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/2530898486 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c5815c9f0 0x7f3c580c8cf0 unknown :-1 s=CLOSED pgs=2771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/2530898486 >> 172.21.15.29:0/2530898486 conn(0x7f3c580ce6a0 msgr2=0x7f3c580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/2530898486 shutdown_connections 2023-10-14T18:18:11.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.818+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/2530898486 wait complete. 2023-10-14T18:18:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.820+0000 7f3c5ed6c640 1 Processor -- start 2023-10-14T18:18:11.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.820+0000 7f3c5ed6c640 1 -- start start 2023-10-14T18:18:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c5815c9f0 0x7f3c581f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580c7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c58150f40 con 0x7f3c580c9230 2023-10-14T18:18:11.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c580c66a0 con 0x7f3c5815c9f0 2023-10-14T18:18:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5ed6c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c581f31e0 con 0x7f3c5815dde0 2023-10-14T18:18:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.821+0000 7f3c5dd6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42668/0 (socket says 172.21.15.29:42668) 2023-10-14T18:18:11.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/1482055408 learned_addr learned my addr 172.21.15.29:0/1482055408 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5d569640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c5815c9f0 0x7f3c581f2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5e56b640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580c7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 msgr2=0x7f3c580c7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580c7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c5815c9f0 msgr2=0x7f3c581f2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c5815c9f0 0x7f3c581f2ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:11.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 -- 172.21.15.29:0/1482055408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54009400 con 0x7f3c580c9230 2023-10-14T18:18:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.822+0000 7f3c5dd6a640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f3c5402bee0 tx=0x7f3c540046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.823+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5403a6f0 con 0x7f3c580c9230 2023-10-14T18:18:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.823+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c5403a870 con 0x7f3c580c9230 2023-10-14T18:18:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.823+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c5403ac20 con 0x7f3c580c9230 2023-10-14T18:18:11.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.823+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5806ae50 con 0x7f3c580c9230 2023-10-14T18:18:11.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.823+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5806b550 con 0x7f3c580c9230 2023-10-14T18:18:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.824+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5815b5c0 con 0x7f3c580c9230 2023-10-14T18:18:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.826+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3c54030380 con 0x7f3c580c9230 2023-10-14T18:18:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.827+0000 7f3c4e7fc640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 0x7f3c3c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:11.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.827+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(235..235 src has 1..235) v4 ==== 7506+0+0 (secure 0 0 0) 0x7f3c540abbd0 con 0x7f3c580c9230 2023-10-14T18:18:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.827+0000 7f3c5d569640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 0x7f3c3c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.828+0000 7f3c5d569640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 0x7f3c3c06d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3c5815feb0 tx=0x7f3c48009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:11.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:11.832+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3c54076100 con 0x7f3c580c9230 2023-10-14T18:18:12.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.065+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f3c580c8aa0 con 0x7f3c580c9230 2023-10-14T18:18:12.068 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:18:12.066+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:18:12.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.070+0000 7f3c4e7fc640 1 -- 172.21.15.29:0/1482055408 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v114) v1 ==== 161+0+0 (secure 0 0 0) 0x7f3c54079720 con 0x7f3c580c9230 2023-10-14T18:18:12.072 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:18:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 msgr2=0x7f3c3c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 0x7f3c3c06d570 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f3c5815feb0 tx=0x7f3c48009040 comp rx=0 tx=0).stop 2023-10-14T18:18:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 msgr2=0x7f3c580c6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 secure :-1 s=READY pgs=2772 cs=0 l=1 rev1=1 crypto rx=0x7f3c5402bee0 tx=0x7f3c540046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 shutdown_connections 2023-10-14T18:18:12.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f3c3c06b0b0 0x7f3c3c06d570 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3c5815dde0 0x7f3c580c7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3c5815c9f0 0x7f3c581f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 --2- 172.21.15.29:0/1482055408 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3c580c9230 0x7f3c580c6160 unknown :-1 s=CLOSED pgs=2772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 >> 172.21.15.29:0/1482055408 conn(0x7f3c580ce6a0 msgr2=0x7f3c580cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.074+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 shutdown_connections 2023-10-14T18:18:12.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.075+0000 7f3c5ed6c640 1 -- 172.21.15.29:0/1482055408 wait complete. 2023-10-14T18:18:12.098 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T18:18:12.098 INFO:tasks.ceph.mds.a.smithi029.stderr: -4668> 2023-10-14T18:17:32.689+0000 7f17227a1640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:18:12.158 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T18:18:12.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.302+0000 7fd978490640 1 Processor -- start 2023-10-14T18:18:12.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.303+0000 7fd978490640 1 -- start start 2023-10-14T18:18:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.304+0000 7fd978490640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd97015ad70 0x7fd970069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.304+0000 7fd978490640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700cb030 con 0x7fd9700caac0 2023-10-14T18:18:12.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.304+0000 7fd978490640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9700c91c0 con 0x7fd9700c9ca0 2023-10-14T18:18:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.304+0000 7fd978490640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9700c94a0 con 0x7fd97015ad70 2023-10-14T18:18:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.304+0000 7fd976a2d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9700caac0 0x7fd9700ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59800/0 (socket says 172.21.15.29:59800) 2023-10-14T18:18:12.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd976a2d640 1 -- 172.21.15.29:0/3666285618 learned_addr learned my addr 172.21.15.29:0/3666285618 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97622c640 1 --2- 172.21.15.29:0/3666285618 >> v2:172.21.15.186:3300/0 conn(0x7fd97015ad70 0x7fd970069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398976667 0 0) 0x7fd9700cb030 con 0x7fd9700caac0 2023-10-14T18:18:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd95c003520 con 0x7fd9700caac0 2023-10-14T18:18:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1506902576 0 0) 0x7fd95c003520 con 0x7fd9700caac0 2023-10-14T18:18:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 >> v2:172.21.15.186:3300/0 conn(0x7fd97015ad70 msgr2=0x7fd970069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 --2- 172.21.15.29:0/3666285618 >> v2:172.21.15.186:3300/0 conn(0x7fd97015ad70 0x7fd970069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 >> v1:172.21.15.186:6789/0 conn(0x7fd9700c9ca0 legacy=0x7fd9700ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700c9780 con 0x7fd9700caac0 2023-10-14T18:18:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.305+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd9680019f0 con 0x7fd9700caac0 2023-10-14T18:18:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 >> v1:172.21.15.29:6789/0 conn(0x7fd9700caac0 legacy=0x7fd9700ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0039c0 0x7fd95c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 0x7fd95c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95c005180 0x7fd95c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700c94a0 con 0x7fd95c0047c0 2023-10-14T18:18:12.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c00c080 con 0x7fd95c0039c0 2023-10-14T18:18:12.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd95c00c240 con 0x7fd95c005180 2023-10-14T18:18:12.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 0x7fd95c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd97622c640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0039c0 0x7fd95c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd976a2d640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95c005180 0x7fd95c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 -- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95c005180 msgr2=0x7fd95c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd95c005180 0x7fd95c00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 -- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0039c0 msgr2=0x7fd95c003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0039c0 0x7fd95c003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.306+0000 7fd975a2b640 1 -- 172.21.15.29:0/3666285618 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd95c003520 con 0x7fd95c0047c0 2023-10-14T18:18:12.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.307+0000 7fd976a2d640 1 -- 172.21.15.29:0/3666285618 reap_dead start 2023-10-14T18:18:12.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd975a2b640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 0x7fd95c004c40 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7fd960001750 tx=0x7fd95c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c04b31388fef7991 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9600052d0 con 0x7fd95c0047c0 2023-10-14T18:18:12.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96002e050 con 0x7fd95c0047c0 2023-10-14T18:18:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd97522a640 1 -- 172.21.15.29:0/3666285618 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd960003830 con 0x7fd95c0047c0 2023-10-14T18:18:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd978490640 1 -- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 msgr2=0x7fd95c004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.308+0000 7fd978490640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 0x7fd95c004c40 secure :-1 s=READY pgs=2777 cs=0 l=1 rev1=1 crypto rx=0x7fd960001750 tx=0x7fd95c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.309+0000 7fd978490640 1 -- 172.21.15.29:0/3666285618 shutdown_connections 2023-10-14T18:18:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.309+0000 7fd978490640 1 --2- 172.21.15.29:0/3666285618 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd95c0047c0 0x7fd95c004c40 unknown :-1 s=CLOSED pgs=2777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.309+0000 7fd978490640 1 -- 172.21.15.29:0/3666285618 >> 172.21.15.29:0/3666285618 conn(0x7fd9700ce9b0 msgr2=0x7fd9700cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.309+0000 7fd978490640 1 -- 172.21.15.29:0/3666285618 shutdown_connections 2023-10-14T18:18:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.309+0000 7fd978490640 1 -- 172.21.15.29:0/3666285618 wait complete. 2023-10-14T18:18:12.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.310+0000 7fd978490640 1 Processor -- start 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.311+0000 7fd978490640 1 -- start start 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0047c0 0x7fd9701f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd970159070 0x7fd9700ad2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700af4a0 con 0x7fd9700ad7e0 2023-10-14T18:18:12.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9701f3f50 con 0x7fd95c0047c0 2023-10-14T18:18:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd978490640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700c2560 con 0x7fd970159070 2023-10-14T18:18:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42706/0 (socket says 172.21.15.29:42706) 2023-10-14T18:18:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 -- 172.21.15.29:0/1343219293 learned_addr learned my addr 172.21.15.29:0/1343219293 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:12.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd975a2b640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd970159070 0x7fd9700ad2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd97622c640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0047c0 0x7fd9701f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 -- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd970159070 msgr2=0x7fd9700ad2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd970159070 0x7fd9700ad2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 -- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0047c0 msgr2=0x7fd9701f3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:12.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0047c0 0x7fd9701f3a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.312+0000 7fd976a2d640 1 -- 172.21.15.29:0/1343219293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9700c2840 con 0x7fd9700ad7e0 2023-10-14T18:18:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd976a2d640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fd968007c40 tx=0x7fd96800b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968004d60 con 0x7fd9700ad7e0 2023-10-14T18:18:12.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd96800bb80 con 0x7fd9700ad7e0 2023-10-14T18:18:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd968007070 con 0x7fd9700ad7e0 2023-10-14T18:18:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9700c2b20 con 0x7fd9700ad7e0 2023-10-14T18:18:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.313+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9701594f0 con 0x7fd9700ad7e0 2023-10-14T18:18:12.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.314+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9700c3040 con 0x7fd9700ad7e0 2023-10-14T18:18:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.315+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd9680071d0 con 0x7fd9700ad7e0 2023-10-14T18:18:12.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.316+0000 7fd966ffd640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 0x7fd95406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.316+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 7592+0+0 (secure 0 0 0) 0x7fd96808a650 con 0x7fd9700ad7e0 2023-10-14T18:18:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.317+0000 7fd97622c640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 0x7fd95406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.317+0000 7fd97622c640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 0x7fd95406d5a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9700afc50 tx=0x7fd96c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:12.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.321+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9700c3040 con 0x7fd9700ad7e0 2023-10-14T18:18:12.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:12.565+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd9700c3040 con 0x7fd9700ad7e0 2023-10-14T18:18:13.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.073+0000 7fd966ffd640 1 -- 172.21.15.29:0/1343219293 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v115) v1 ==== 104+0+0 (secure 0 0 0) 0x7fd9700c3040 con 0x7fd9700ad7e0 2023-10-14T18:18:13.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.077+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 msgr2=0x7fd95406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.077+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 0x7fd95406d5a0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fd9700afc50 tx=0x7fd96c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:13.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.077+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 msgr2=0x7fd9700ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.077+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 secure :-1 s=READY pgs=2778 cs=0 l=1 rev1=1 crypto rx=0x7fd968007c40 tx=0x7fd96800b4a0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 shutdown_connections 2023-10-14T18:18:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd95406b0e0 0x7fd95406d5a0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9700ad7e0 0x7fd9700ae530 unknown :-1 s=CLOSED pgs=2778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd970159070 0x7fd9700ad2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 --2- 172.21.15.29:0/1343219293 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd95c0047c0 0x7fd9701f3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 >> 172.21.15.29:0/1343219293 conn(0x7fd9700ce9b0 msgr2=0x7fd970069da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 shutdown_connections 2023-10-14T18:18:13.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.078+0000 7fd978490640 1 -- 172.21.15.29:0/1343219293 wait complete. 2023-10-14T18:18:13.101 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T18:18:13.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.332+0000 7f60f0a40640 1 Processor -- start 2023-10-14T18:18:13.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.332+0000 7f60f0a40640 1 -- start start 2023-10-14T18:18:13.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.333+0000 7f60f0a40640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f60ec160820 0x7f60ec160c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.333+0000 7f60f0a40640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec069e30 con 0x7f60ec15cdb0 2023-10-14T18:18:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.333+0000 7f60f0a40640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ec1611a0 con 0x7f60ec06a2c0 2023-10-14T18:18:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60f0a40640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ec0ccba0 con 0x7f60ec160820 2023-10-14T18:18:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60eb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f60ec15cdb0 0x7f60ec0ce7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59808/0 (socket says 172.21.15.29:59808) 2023-10-14T18:18:13.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60eb7fe640 1 -- 172.21.15.29:0/817062324 learned_addr learned my addr 172.21.15.29:0/817062324 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60eaffd640 1 --2- 172.21.15.29:0/817062324 >> v2:172.21.15.186:3300/0 conn(0x7f60ec160820 0x7f60ec160c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973425414 0 0) 0x7f60ec069e30 con 0x7f60ec15cdb0 2023-10-14T18:18:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.334+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d4003520 con 0x7f60ec15cdb0 2023-10-14T18:18:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1183954264 0 0) 0x7f60d4003520 con 0x7f60ec15cdb0 2023-10-14T18:18:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 >> v2:172.21.15.186:3300/0 conn(0x7f60ec160820 msgr2=0x7f60ec160c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:13.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 --2- 172.21.15.29:0/817062324 >> v2:172.21.15.186:3300/0 conn(0x7f60ec160820 0x7f60ec160c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 >> v1:172.21.15.186:6789/0 conn(0x7f60ec06a2c0 legacy=0x7f60ec06a770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec1615f0 con 0x7f60ec15cdb0 2023-10-14T18:18:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f60dc0019f0 con 0x7f60ec15cdb0 2023-10-14T18:18:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 >> v1:172.21.15.29:6789/0 conn(0x7f60ec15cdb0 legacy=0x7f60ec0ce7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 0x7f60d4003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60d40047c0 0x7f60d4004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d4005180 0x7f60d400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0ccba0 con 0x7f60d40039c0 2023-10-14T18:18:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d400c080 con 0x7f60d40047c0 2023-10-14T18:18:13.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60d400c240 con 0x7f60d4005180 2023-10-14T18:18:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.335+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 0x7f60d4003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eaffd640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60d40047c0 0x7f60d4004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60ebfff640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d4005180 0x7f60d400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eb7fe640 1 -- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d4005180 msgr2=0x7f60d400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d4005180 0x7f60d400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eb7fe640 1 -- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60d40047c0 msgr2=0x7f60d4004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60d40047c0 0x7f60d4004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60eb7fe640 1 -- 172.21.15.29:0/817062324 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d4003520 con 0x7f60d40039c0 2023-10-14T18:18:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60ebfff640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d4005180 0x7f60d400b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.336+0000 7f60ebfff640 1 -- 172.21.15.29:0/817062324 reap_dead start 2023-10-14T18:18:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.337+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 0x7f60d4003de0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00a760 tx=0x7f60d40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1b9193f9792f1e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.337+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc034630 con 0x7f60d40039c0 2023-10-14T18:18:13.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60dc02f070 con 0x7f60d40039c0 2023-10-14T18:18:13.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60ea7fc640 1 -- 172.21.15.29:0/817062324 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60dc034a50 con 0x7f60d40039c0 2023-10-14T18:18:13.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 -- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 msgr2=0x7f60d4003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 0x7f60d4003de0 secure :-1 s=READY pgs=2780 cs=0 l=1 rev1=1 crypto rx=0x7f60dc00a760 tx=0x7f60d40057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 -- 172.21.15.29:0/817062324 shutdown_connections 2023-10-14T18:18:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 --2- 172.21.15.29:0/817062324 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60d40039c0 0x7f60d4003de0 unknown :-1 s=CLOSED pgs=2780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 -- 172.21.15.29:0/817062324 >> 172.21.15.29:0/817062324 conn(0x7f60ec0cba50 msgr2=0x7f60ec0cbe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 -- 172.21.15.29:0/817062324 shutdown_connections 2023-10-14T18:18:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.338+0000 7f60f0a40640 1 -- 172.21.15.29:0/817062324 wait complete. 2023-10-14T18:18:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.340+0000 7f60f0a40640 1 Processor -- start 2023-10-14T18:18:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.341+0000 7f60f0a40640 1 -- start start 2023-10-14T18:18:13.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d40039c0 0x7f60ec15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60ec15a8c0 0x7f60ec0c60a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0c6a90 con 0x7f60ec0c65e0 2023-10-14T18:18:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec0c7de0 con 0x7f60ec15a8c0 2023-10-14T18:18:13.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60f0a40640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ec15ad40 con 0x7f60d40039c0 2023-10-14T18:18:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42724/0 (socket says 172.21.15.29:42724) 2023-10-14T18:18:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60ebfff640 1 -- 172.21.15.29:0/3772212752 learned_addr learned my addr 172.21.15.29:0/3772212752 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d40039c0 0x7f60ec15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60eaffd640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60ec15a8c0 0x7f60ec0c60a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.342+0000 7f60ebfff640 1 -- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d40039c0 msgr2=0x7f60ec15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60ebfff640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d40039c0 0x7f60ec15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60ebfff640 1 -- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60ec15a8c0 msgr2=0x7f60ec0c60a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60ebfff640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60ec15a8c0 0x7f60ec0c60a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:13.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60ebfff640 1 -- 172.21.15.29:0/3772212752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec15b020 con 0x7f60ec0c65e0 2023-10-14T18:18:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60ebfff640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0cc7e0 tx=0x7f60e400d890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.343+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e400b070 con 0x7f60ec0c65e0 2023-10-14T18:18:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.344+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60e40035f0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.344+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ec15b300 con 0x7f60ec0c65e0 2023-10-14T18:18:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.344+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60e4007ea0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.344+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ec15b8c0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.345+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ec04e9b0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.346+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f60e40039a0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.347+0000 7f60d3fff640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 0x7f60d806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.347+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(236..236 src has 1..236) v4 ==== 7592+0+0 (secure 0 0 0) 0x7f60e4089870 con 0x7f60ec0c65e0 2023-10-14T18:18:13.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.347+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 0x7f60d806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.348+0000 7f60eb7fe640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 0x7f60d806d570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60dc034000 tx=0x7f60dc02f000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:13.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.352+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f60e4053cb0 con 0x7f60ec0c65e0 2023-10-14T18:18:13.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:13.553+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f60ec0cc810 con 0x7f60ec0c65e0 2023-10-14T18:18:14.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.441+0000 7f60d3fff640 1 -- 172.21.15.29:0/3772212752 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v237) v1 ==== 181+0+0 (secure 0 0 0) 0x7f60e40572d0 con 0x7f60ec0c65e0 2023-10-14T18:18:14.444 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T18:18:14.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 msgr2=0x7f60d806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 0x7f60d806d570 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f60dc034000 tx=0x7f60dc02f000 comp rx=0 tx=0).stop 2023-10-14T18:18:14.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 msgr2=0x7f60ec148be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 secure :-1 s=READY pgs=2781 cs=0 l=1 rev1=1 crypto rx=0x7f60ec0cc7e0 tx=0x7f60e400d890 comp rx=0 tx=0).stop 2023-10-14T18:18:14.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 shutdown_connections 2023-10-14T18:18:14.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f60d806b0b0 0x7f60d806d570 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f60ec0c65e0 0x7f60ec148be0 unknown :-1 s=CLOSED pgs=2781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f60ec15a8c0 0x7f60ec0c60a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 --2- 172.21.15.29:0/3772212752 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f60d40039c0 0x7f60ec15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.445+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 >> 172.21.15.29:0/3772212752 conn(0x7f60ec0cba50 msgr2=0x7f60ec06a6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:14.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.446+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 shutdown_connections 2023-10-14T18:18:14.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.446+0000 7f60f0a40640 1 -- 172.21.15.29:0/3772212752 wait complete. 2023-10-14T18:18:14.473 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T18:18:14.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.700+0000 7f9f53fff640 1 Processor -- start 2023-10-14T18:18:14.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.701+0000 7f9f53fff640 1 -- start start 2023-10-14T18:18:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f53fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ad970 0x7f9f540cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f53fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9f540ccfb0 0x7f9f540cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f53fff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f5415b9d0 con 0x7f9f5406c040 2023-10-14T18:18:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f53fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f5415bcb0 con 0x7f9f540ccfb0 2023-10-14T18:18:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f53fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f5415bf90 con 0x7f9f540ad970 2023-10-14T18:18:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f52ffd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9f5406c040 0x7f9f5406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59810/0 (socket says 172.21.15.29:59810) 2023-10-14T18:18:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f52ffd640 1 -- 172.21.15.29:0/3263575030 learned_addr learned my addr 172.21.15.29:0/3263575030 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f537fe640 1 --2- 172.21.15.29:0/3263575030 >> v2:172.21.15.29:3300/0 conn(0x7f9f540ccfb0 0x7f9f540cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.703+0000 7f9f4a7fc640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ad970 0x7f9f540cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f527fc640 1 -- 172.21.15.29:0/3263575030 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161156293 0 0) 0x7f9f5415b9d0 con 0x7f9f5406c040 2023-10-14T18:18:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f527fc640 1 -- 172.21.15.29:0/3263575030 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f40007930 con 0x7f9f5406c040 2023-10-14T18:18:14.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f537fe640 1 -- 172.21.15.29:0/3263575030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ad970 msgr2=0x7f9f540cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f537fe640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ad970 0x7f9f540cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f537fe640 1 -- 172.21.15.29:0/3263575030 >> v1:172.21.15.29:6789/0 conn(0x7f9f5406c040 legacy=0x7f9f5406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.704+0000 7f9f537fe640 1 -- 172.21.15.29:0/3263575030 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f540cc270 con 0x7f9f540ccfb0 2023-10-14T18:18:14.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.705+0000 7f9f537fe640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f540ccfb0 0x7f9f540cd440 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0097d0 tx=0x7f9f4c02b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb3133f4ea07151 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.705+0000 7f9f527fc640 1 -- 172.21.15.29:0/3263575030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c031800 con 0x7f9f540ccfb0 2023-10-14T18:18:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.705+0000 7f9f527fc640 1 -- 172.21.15.29:0/3263575030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f4c031960 con 0x7f9f540ccfb0 2023-10-14T18:18:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.705+0000 7f9f527fc640 1 -- 172.21.15.29:0/3263575030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4c031cf0 con 0x7f9f540ccfb0 2023-10-14T18:18:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 -- 172.21.15.29:0/3263575030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f540ccfb0 msgr2=0x7f9f540cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f540ccfb0 0x7f9f540cd440 secure :-1 s=READY pgs=2783 cs=0 l=1 rev1=1 crypto rx=0x7f9f4c0097d0 tx=0x7f9f4c02b8a0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 -- 172.21.15.29:0/3263575030 shutdown_connections 2023-10-14T18:18:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f540ccfb0 0x7f9f540cd440 unknown :-1 s=CLOSED pgs=2783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3263575030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ad970 0x7f9f540cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 -- 172.21.15.29:0/3263575030 >> 172.21.15.29:0/3263575030 conn(0x7f9f540ce6a0 msgr2=0x7f9f540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 -- 172.21.15.29:0/3263575030 shutdown_connections 2023-10-14T18:18:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.706+0000 7f9f53fff640 1 -- 172.21.15.29:0/3263575030 wait complete. 2023-10-14T18:18:14.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.708+0000 7f9f53fff640 1 Processor -- start 2023-10-14T18:18:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.708+0000 7f9f53fff640 1 -- start start 2023-10-14T18:18:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f540ad970 0x7f9f541595d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ccfb0 0x7f9f54159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f541570c0 con 0x7f9f5406c040 2023-10-14T18:18:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f541573a0 con 0x7f9f540ad970 2023-10-14T18:18:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f53fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f54157680 con 0x7f9f540ccfb0 2023-10-14T18:18:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:14.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f52ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42756/0 (socket says 172.21.15.29:42756) 2023-10-14T18:18:14.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.709+0000 7f9f52ffd640 1 -- 172.21.15.29:0/3306275682 learned_addr learned my addr 172.21.15.29:0/3306275682 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:14.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 -- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ccfb0 msgr2=0x7f9f54159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ccfb0 0x7f9f54159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 -- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f540ad970 msgr2=0x7f9f541595d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:14.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f540ad970 0x7f9f541595d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 -- 172.21.15.29:0/3306275682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f4c009400 con 0x7f9f5406c040 2023-10-14T18:18:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f52ffd640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400cbc0 tx=0x7f9f4400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400ad40 con 0x7f9f5406c040 2023-10-14T18:18:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f44012400 con 0x7f9f5406c040 2023-10-14T18:18:14.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.710+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9f4400aea0 con 0x7f9f5406c040 2023-10-14T18:18:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.711+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f54157960 con 0x7f9f5406c040 2023-10-14T18:18:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.711+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f54156c70 con 0x7f9f5406c040 2023-10-14T18:18:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.712+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f5406b330 con 0x7f9f5406c040 2023-10-14T18:18:14.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.713+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9f44007070 con 0x7f9f5406c040 2023-10-14T18:18:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.714+0000 7f9f50ff9640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 0x7f9f3406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:14.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.714+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(237..237 src has 1..237) v4 ==== 7154+0+0 (secure 0 0 0) 0x7f9f44089ef0 con 0x7f9f5406c040 2023-10-14T18:18:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.714+0000 7f9f4a7fc640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 0x7f9f3406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.715+0000 7f9f4a7fc640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 0x7f9f3406d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f5406dab0 tx=0x7f9f3c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:14.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.719+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9f44058260 con 0x7f9f5406c040 2023-10-14T18:18:14.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:14.918+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9f540c96d0 con 0x7f9f5406c040 2023-10-14T18:18:15.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.453+0000 7f9f50ff9640 1 -- 172.21.15.29:0/3306275682 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v238) v1 ==== 169+0+0 (secure 0 0 0) 0x7f9f44057df0 con 0x7f9f5406c040 2023-10-14T18:18:15.455 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T18:18:15.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 msgr2=0x7f9f3406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 0x7f9f3406d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f9f5406dab0 tx=0x7f9f3c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:15.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 msgr2=0x7f9f540c7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 secure :-1 s=READY pgs=2784 cs=0 l=1 rev1=1 crypto rx=0x7f9f4400cbc0 tx=0x7f9f4400a430 comp rx=0 tx=0).stop 2023-10-14T18:18:15.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 shutdown_connections 2023-10-14T18:18:15.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9f3406b0b0 0x7f9f3406d570 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9f540ccfb0 0x7f9f54159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9f540ad970 0x7f9f541595d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 --2- 172.21.15.29:0/3306275682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9f5406c040 0x7f9f540c7860 unknown :-1 s=CLOSED pgs=2784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.457+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 >> 172.21.15.29:0/3306275682 conn(0x7f9f540ce6a0 msgr2=0x7f9f540c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:15.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.458+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 shutdown_connections 2023-10-14T18:18:15.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.458+0000 7f9f53fff640 1 -- 172.21.15.29:0/3306275682 wait complete. 2023-10-14T18:18:15.482 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:15.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.711+0000 7fe0f99c4640 1 Processor -- start 2023-10-14T18:18:15.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.712+0000 7fe0f99c4640 1 -- start start 2023-10-14T18:18:15.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f99c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0f4167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f99c4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f40d12f0 con 0x7fe0f415cc80 2023-10-14T18:18:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f99c4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0f40d06e0 con 0x7fe0f40cdf30 2023-10-14T18:18:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f99c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40d0500 con 0x7fe0f4072170 2023-10-14T18:18:15.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f89c2640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0f415cc80 0x7fe0f40cd820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59814/0 (socket says 172.21.15.29:59814) 2023-10-14T18:18:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.713+0000 7fe0f89c2640 1 -- 172.21.15.29:0/4181061820 learned_addr learned my addr 172.21.15.29:0/4181061820 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f91c3640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0f4167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020354582 0 0) 0x7fe0f40d12f0 con 0x7fe0f415cc80 2023-10-14T18:18:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0e0003520 con 0x7fe0f415cc80 2023-10-14T18:18:15.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 587233158 0 0) 0x7fe0e0003520 con 0x7fe0f415cc80 2023-10-14T18:18:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 msgr2=0x7fe0f4167b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0f4167b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 >> v1:172.21.15.186:6789/0 conn(0x7fe0f40cdf30 legacy=0x7fe0f4071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40d09c0 con 0x7fe0f415cc80 2023-10-14T18:18:15.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.714+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe0e40019f0 con 0x7fe0f415cc80 2023-10-14T18:18:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 >> v1:172.21.15.29:6789/0 conn(0x7fe0f415cc80 legacy=0x7fe0f40cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0e00039a0 0x7fe0e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00047c0 0x7fe0e0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40d0500 con 0x7fe0e00039a0 2023-10-14T18:18:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e000ba60 con 0x7fe0e00047c0 2023-10-14T18:18:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0e000bc20 con 0x7fe0f4072170 2023-10-14T18:18:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0e00039a0 0x7fe0e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f3fff640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00047c0 0x7fe0e0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f91c3640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 -- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 msgr2=0x7fe0e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f4072170 0x7fe0e0005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 -- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00047c0 msgr2=0x7fe0e0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:15.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00047c0 0x7fe0e0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.715+0000 7fe0f89c2640 1 -- 172.21.15.29:0/4181061820 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e0003520 con 0x7fe0e00039a0 2023-10-14T18:18:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0e00039a0 0x7fe0e0003dc0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400a720 tx=0x7fe0e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8a2ddebb6a7bb65 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4034630 con 0x7fe0e00039a0 2023-10-14T18:18:15.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e402f070 con 0x7fe0e00039a0 2023-10-14T18:18:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f37fe640 1 -- 172.21.15.29:0/4181061820 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e40349d0 con 0x7fe0e00039a0 2023-10-14T18:18:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f99c4640 1 -- 172.21.15.29:0/4181061820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0e00039a0 msgr2=0x7fe0e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/4181061820 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0e00039a0 0x7fe0e0003dc0 secure :-1 s=READY pgs=2786 cs=0 l=1 rev1=1 crypto rx=0x7fe0e400a720 tx=0x7fe0e00057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.717+0000 7fe0f91c3640 1 -- 172.21.15.29:0/4181061820 reap_dead start 2023-10-14T18:18:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.718+0000 7fe0f99c4640 1 -- 172.21.15.29:0/4181061820 shutdown_connections 2023-10-14T18:18:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.718+0000 7fe0f99c4640 1 -- 172.21.15.29:0/4181061820 >> 172.21.15.29:0/4181061820 conn(0x7fe0f4065ed0 msgr2=0x7fe0f40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.718+0000 7fe0f99c4640 1 -- 172.21.15.29:0/4181061820 shutdown_connections 2023-10-14T18:18:15.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.718+0000 7fe0f99c4640 1 -- 172.21.15.29:0/4181061820 wait complete. 2023-10-14T18:18:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.719+0000 7fe0f99c4640 1 Processor -- start 2023-10-14T18:18:15.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.720+0000 7fe0f99c4640 1 -- start start 2023-10-14T18:18:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.720+0000 7fe0f99c4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00039a0 0x7fe0f40ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f99c4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f99c4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f40ace50 0x7fe0f40ad2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f99c4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40b17c0 con 0x7fe0f40b0e00 2023-10-14T18:18:15.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f99c4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40af470 con 0x7fe0e00039a0 2023-10-14T18:18:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f99c4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f40ae860 con 0x7fe0f40ace50 2023-10-14T18:18:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42780/0 (socket says 172.21.15.29:42780) 2023-10-14T18:18:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 -- 172.21.15.29:0/1646305064 learned_addr learned my addr 172.21.15.29:0/1646305064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:15.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f91c3640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f40ace50 0x7fe0f40ad2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00039a0 0x7fe0f40ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 -- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f40ace50 msgr2=0x7fe0f40ad2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f40ace50 0x7fe0f40ad2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 -- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00039a0 msgr2=0x7fe0f40ae160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.721+0000 7fe0f3fff640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00039a0 0x7fe0f40ae160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f3fff640 1 -- 172.21.15.29:0/1646305064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f40aeb40 con 0x7fe0f40b0e00 2023-10-14T18:18:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f3fff640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e4d0 tx=0x7fe0dc00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0053a0 con 0x7fe0f40b0e00 2023-10-14T18:18:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0035f0 con 0x7fe0f40b0e00 2023-10-14T18:18:15.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc003980 con 0x7fe0f40b0e00 2023-10-14T18:18:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f40aee20 con 0x7fe0f40b0e00 2023-10-14T18:18:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.722+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f40ad810 con 0x7fe0f40b0e00 2023-10-14T18:18:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.723+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f404e9b0 con 0x7fe0f40b0e00 2023-10-14T18:18:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.724+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe0dc003ae0 con 0x7fe0f40b0e00 2023-10-14T18:18:15.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.725+0000 7fe0f17fa640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 0x7fe0d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.725+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7fe0dc089c20 con 0x7fe0f40b0e00 2023-10-14T18:18:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.726+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 0x7fe0d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.726+0000 7fe0f89c2640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 0x7fe0d406d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4034000 tx=0x7fe0e402f000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:15.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.730+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0dc054560 con 0x7fe0f40b0e00 2023-10-14T18:18:15.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.929+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0f40b6530 con 0x7fe0f40b0e00 2023-10-14T18:18:15.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.931+0000 7fe0f17fa640 1 -- 172.21.15.29:0/1646305064 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+18081 (secure 0 0 0) 0x7fe0dc057b80 con 0x7fe0f40b0e00 2023-10-14T18:18:15.934 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:15.934 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":238,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:15.443657+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6836/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6837/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 msgr2=0x7fe0d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 0x7fe0d406d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4034000 tx=0x7fe0e402f000 comp rx=0 tx=0).stop 2023-10-14T18:18:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 msgr2=0x7fe0f40b1280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 secure :-1 s=READY pgs=2787 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc00e4d0 tx=0x7fe0dc00e990 comp rx=0 tx=0).stop 2023-10-14T18:18:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 shutdown_connections 2023-10-14T18:18:15.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe0d406b0b0 0x7fe0d406d570 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0f40ace50 0x7fe0f40ad2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0f40b0e00 0x7fe0f40b1280 unknown :-1 s=CLOSED pgs=2787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 --2- 172.21.15.29:0/1646305064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0e00039a0 0x7fe0f40ae160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 >> 172.21.15.29:0/1646305064 conn(0x7fe0f4065ed0 msgr2=0x7fe0f415d240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:15.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.934+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 shutdown_connections 2023-10-14T18:18:15.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:15.935+0000 7fe0f99c4640 1 -- 172.21.15.29:0/1646305064 wait complete. 2023-10-14T18:18:15.955 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 28, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:17:40.115972+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '232', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 29, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:17:40.810362+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '233', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.6299999952316284, 'score_stable': 1.6299999952316284, 'optimal_score': 1, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:15.956 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:16.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.190+0000 7fe0665e7640 1 Processor -- start 2023-10-14T18:18:16.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.191+0000 7fe0665e7640 1 -- start start 2023-10-14T18:18:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.192+0000 7fe0665e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe06015ad70 0x7fe060069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.192+0000 7fe0665e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe06006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.192+0000 7fe0665e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0600cb030 con 0x7fe0600ca3b0 2023-10-14T18:18:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.192+0000 7fe0665e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0600c91c0 con 0x7fe06015ad70 2023-10-14T18:18:16.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.192+0000 7fe0665e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600c94a0 con 0x7fe0600c9ca0 2023-10-14T18:18:16.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe064b84640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe0600ca3b0 0x7fe0600ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59826/0 (socket says 172.21.15.29:59826) 2023-10-14T18:18:16.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe064b84640 1 -- 172.21.15.29:0/397536974 learned_addr learned my addr 172.21.15.29:0/397536974 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:16.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe05ffff640 1 --2- 172.21.15.29:0/397536974 >> v2:172.21.15.186:3300/0 conn(0x7fe06015ad70 0x7fe060069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe0577fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe06006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2787616722 0 0) 0x7fe0600cb030 con 0x7fe0600ca3b0 2023-10-14T18:18:16.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.193+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe0600ca3b0 2023-10-14T18:18:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1566347656 0 0) 0x7fe04c003520 con 0x7fe0600ca3b0 2023-10-14T18:18:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 msgr2=0x7fe06006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe06006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 >> v2:172.21.15.186:3300/0 conn(0x7fe06015ad70 msgr2=0x7fe060069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 --2- 172.21.15.29:0/397536974 >> v2:172.21.15.186:3300/0 conn(0x7fe06015ad70 0x7fe060069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600c9780 con 0x7fe0600ca3b0 2023-10-14T18:18:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe058001670 con 0x7fe0600ca3b0 2023-10-14T18:18:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 >> v1:172.21.15.29:6789/0 conn(0x7fe0600ca3b0 legacy=0x7fe0600ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe04c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c003f40 0x7fe04c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c004d00 0x7fe04c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600c91c0 con 0x7fe04c003f40 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600c94a0 con 0x7fe04c004d00 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c00b980 con 0x7fe0600c9ca0 2023-10-14T18:18:16.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.194+0000 7fe0577fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c003f40 0x7fe04c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe064b84640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c004d00 0x7fe04c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe05ffff640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe04c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe0577fe640 1 -- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 msgr2=0x7fe04c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe0577fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe0600c9ca0 0x7fe04c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe0577fe640 1 -- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c004d00 msgr2=0x7fe04c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe0577fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c004d00 0x7fe04c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe0577fe640 1 -- 172.21.15.29:0/397536974 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c003520 con 0x7fe04c003f40 2023-10-14T18:18:16.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.195+0000 7fe064b84640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c004d00 0x7fe04c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.196+0000 7fe0577fe640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c003f40 0x7fe04c0047c0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fe048009af0 tx=0x7fe04c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf78a704b65afa1a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.196+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04802c6b0 con 0x7fe04c003f40 2023-10-14T18:18:16.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.196+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04802c810 con 0x7fe04c003f40 2023-10-14T18:18:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.196+0000 7fe05f7fe640 1 -- 172.21.15.29:0/397536974 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04802cba0 con 0x7fe04c003f40 2023-10-14T18:18:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 -- 172.21.15.29:0/397536974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c003f40 msgr2=0x7fe04c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 --2- 172.21.15.29:0/397536974 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c003f40 0x7fe04c0047c0 secure :-1 s=READY pgs=2789 cs=0 l=1 rev1=1 crypto rx=0x7fe048009af0 tx=0x7fe04c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe064b84640 1 -- 172.21.15.29:0/397536974 reap_dead start 2023-10-14T18:18:16.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 -- 172.21.15.29:0/397536974 shutdown_connections 2023-10-14T18:18:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 -- 172.21.15.29:0/397536974 >> 172.21.15.29:0/397536974 conn(0x7fe0600ce9b0 msgr2=0x7fe0600cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 -- 172.21.15.29:0/397536974 shutdown_connections 2023-10-14T18:18:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.197+0000 7fe0665e7640 1 -- 172.21.15.29:0/397536974 wait complete. 2023-10-14T18:18:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.199+0000 7fe0665e7640 1 Processor -- start 2023-10-14T18:18:16.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.199+0000 7fe0665e7640 1 -- start start 2023-10-14T18:18:16.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c003f40 0x7fe0600a8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0600a8cd0 0x7fe0601f0100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0601f1010 con 0x7fe0601f0640 2023-10-14T18:18:16.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600a9450 con 0x7fe0600a8cd0 2023-10-14T18:18:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0665e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0600a9730 con 0x7fe04c003f40 2023-10-14T18:18:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe064b84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe064b84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42802/0 (socket says 172.21.15.29:42802) 2023-10-14T18:18:16.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe064b84640 1 -- 172.21.15.29:0/3180851187 learned_addr learned my addr 172.21.15.29:0/3180851187 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe05ffff640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c003f40 0x7fe0600a8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.200+0000 7fe0577fe640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0600a8cd0 0x7fe0601f0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 -- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c003f40 msgr2=0x7fe0600a8790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c003f40 0x7fe0600a8790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 -- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0600a8cd0 msgr2=0x7fe0601f0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0600a8cd0 0x7fe0601f0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 -- 172.21.15.29:0/3180851187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0600a9a10 con 0x7fe0601f0640 2023-10-14T18:18:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe064b84640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe0600c5a30 tx=0x7fe05800d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058007440 con 0x7fe0601f0640 2023-10-14T18:18:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0580075c0 con 0x7fe0601f0640 2023-10-14T18:18:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe058004a20 con 0x7fe0601f0640 2023-10-14T18:18:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.201+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0600a9cf0 con 0x7fe0601f0640 2023-10-14T18:18:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.202+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0600a9150 con 0x7fe0601f0640 2023-10-14T18:18:16.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.203+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe06004e160 con 0x7fe0601f0640 2023-10-14T18:18:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.204+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe058004b80 con 0x7fe0601f0640 2023-10-14T18:18:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.206+0000 7fe05d7fa640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 0x7fe04006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.206+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7fe058089c00 con 0x7fe0601f0640 2023-10-14T18:18:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.206+0000 7fe05ffff640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 0x7fe04006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.207+0000 7fe05ffff640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 0x7fe04006d5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe0600c5240 tx=0x7fe05000b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.210+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe058090080 con 0x7fe0601f0640 2023-10-14T18:18:16.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.414+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe0600b1310 con 0x7fe0601f0640 2023-10-14T18:18:16.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.414+0000 7fe05d7fa640 1 -- 172.21.15.29:0/3180851187 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v238) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe0580544f0 con 0x7fe0601f0640 2023-10-14T18:18:16.417 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:16.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.416+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 msgr2=0x7fe04006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.416+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 0x7fe04006d5a0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe0600c5240 tx=0x7fe05000b040 comp rx=0 tx=0).stop 2023-10-14T18:18:16.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 msgr2=0x7fe0601f0ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 secure :-1 s=READY pgs=2790 cs=0 l=1 rev1=1 crypto rx=0x7fe0600c5a30 tx=0x7fe05800d9d0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 shutdown_connections 2023-10-14T18:18:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe04006b0e0 0x7fe04006d5a0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe0601f0640 0x7fe0601f0ad0 unknown :-1 s=CLOSED pgs=2790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe0600a8cd0 0x7fe0601f0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 --2- 172.21.15.29:0/3180851187 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c003f40 0x7fe0600a8790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 >> 172.21.15.29:0/3180851187 conn(0x7fe0600ce9b0 msgr2=0x7fe0600b35e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:16.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 shutdown_connections 2023-10-14T18:18:16.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.417+0000 7fe0665e7640 1 -- 172.21.15.29:0/3180851187 wait complete. 2023-10-14T18:18:16.438 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:18:16.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.639+0000 7f07b82e7640 1 Processor -- start 2023-10-14T18:18:16.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.640+0000 7f07b82e7640 1 -- start start 2023-10-14T18:18:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07b00addb0 0x7f07b00caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 0x7f07b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f07b006a2e0 0x7f07b00cc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b0161a50 con 0x7f07b00addb0 2023-10-14T18:18:16.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b006a750 con 0x7f07b006a2e0 2023-10-14T18:18:16.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b82e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00cb140 con 0x7f07b00c9000 2023-10-14T18:18:16.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.641+0000 7f07b6083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07b00addb0 0x7f07b00caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f07b00addb0 0x7f07b00caa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42808/0 (socket says 172.21.15.29:42808) 2023-10-14T18:18:16.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 -- 172.21.15.29:0/1787331501 learned_addr learned my addr 172.21.15.29:0/1787331501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:16.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6884640 1 --2- 172.21.15.29:0/1787331501 >> v2:172.21.15.186:3300/0 conn(0x7f07b006a2e0 0x7f07b00cc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b5882640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 0x7f07b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 -- 172.21.15.29:0/1787331501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 msgr2=0x7f07b0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 0x7f07b0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 -- 172.21.15.29:0/1787331501 >> v2:172.21.15.186:3300/0 conn(0x7f07b006a2e0 msgr2=0x7f07b00cc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:16.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 --2- 172.21.15.29:0/1787331501 >> v2:172.21.15.186:3300/0 conn(0x7f07b006a2e0 0x7f07b00cc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b6083640 1 -- 172.21.15.29:0/1787331501 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b006b030 con 0x7f07b00addb0 2023-10-14T18:18:16.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.642+0000 7f07b5882640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 0x7f07b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:16.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b6083640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00addb0 0x7f07b00caa80 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f07b00cf850 tx=0x7f07a402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381fd65703e33704 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b5081640 1 -- 172.21.15.29:0/1787331501 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a402ca70 con 0x7f07b00addb0 2023-10-14T18:18:16.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b5081640 1 -- 172.21.15.29:0/1787331501 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a402cbd0 con 0x7f07b00addb0 2023-10-14T18:18:16.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b5081640 1 -- 172.21.15.29:0/1787331501 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a40343f0 con 0x7f07b00addb0 2023-10-14T18:18:16.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b82e7640 1 -- 172.21.15.29:0/1787331501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00addb0 msgr2=0x7f07b00caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.644+0000 7f07b82e7640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00addb0 0x7f07b00caa80 secure :-1 s=READY pgs=2791 cs=0 l=1 rev1=1 crypto rx=0x7f07b00cf850 tx=0x7f07a402bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 -- 172.21.15.29:0/1787331501 shutdown_connections 2023-10-14T18:18:16.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 --2- 172.21.15.29:0/1787331501 >> v2:172.21.15.186:3300/0 conn(0x7f07b006a2e0 0x7f07b00cc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b00c9000 0x7f07b0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 --2- 172.21.15.29:0/1787331501 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00addb0 0x7f07b00caa80 unknown :-1 s=CLOSED pgs=2791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 -- 172.21.15.29:0/1787331501 >> 172.21.15.29:0/1787331501 conn(0x7f07b00ce9b0 msgr2=0x7f07b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 -- 172.21.15.29:0/1787331501 shutdown_connections 2023-10-14T18:18:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.645+0000 7f07b82e7640 1 -- 172.21.15.29:0/1787331501 wait complete. 2023-10-14T18:18:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.646+0000 7f07b82e7640 1 Processor -- start 2023-10-14T18:18:16.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 -- start start 2023-10-14T18:18:16.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b006a2e0 0x7f07b00b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b00addb0 0x7f07b00c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00b40c0 con 0x7f07b00c9000 2023-10-14T18:18:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00b3720 con 0x7f07b00addb0 2023-10-14T18:18:16.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.647+0000 7f07b82e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b00b3a00 con 0x7f07b006a2e0 2023-10-14T18:18:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42822/0 (socket says 172.21.15.29:42822) 2023-10-14T18:18:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 -- 172.21.15.29:0/271407863 learned_addr learned my addr 172.21.15.29:0/271407863 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6083640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b006a2e0 0x7f07b00b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b5882640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b00addb0 0x7f07b00c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 -- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b006a2e0 msgr2=0x7f07b00b44e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b006a2e0 0x7f07b00b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 -- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b00addb0 msgr2=0x7f07b00c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b00addb0 0x7f07b00c8590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.648+0000 7f07b6884640 1 -- 172.21.15.29:0/271407863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a4009b30 con 0x7f07b00c9000 2023-10-14T18:18:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07b6884640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d800 tx=0x7f07a800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a80096c0 con 0x7f07b00c9000 2023-10-14T18:18:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a8009840 con 0x7f07b00c9000 2023-10-14T18:18:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f07a8009bf0 con 0x7f07b00c9000 2023-10-14T18:18:16.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b00b3ce0 con 0x7f07b00c9000 2023-10-14T18:18:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.649+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b00c8ad0 con 0x7f07b00c9000 2023-10-14T18:18:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.650+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b00b4200 con 0x7f07b00c9000 2023-10-14T18:18:16.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.651+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f07a8009d50 con 0x7f07b00c9000 2023-10-14T18:18:16.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.653+0000 7f07aeffd640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 0x7f079406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:16.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.653+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f07a808a210 con 0x7f07b00c9000 2023-10-14T18:18:16.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.653+0000 7f07b6083640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 0x7f079406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:16.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.653+0000 7f07b6083640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 0x7f079406d5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07a4035000 tx=0x7f07a4030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:16.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.657+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f07b00b4200 con 0x7f07b00c9000 2023-10-14T18:18:16.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.852+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f07b00b4200 con 0x7f07b00c9000 2023-10-14T18:18:16.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.853+0000 7f07aeffd640 1 -- 172.21.15.29:0/271407863 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v238) v1 ==== 107+0+11 (secure 0 0 0) 0x7f07b00b4200 con 0x7f07b00c9000 2023-10-14T18:18:16.855 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.855+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 msgr2=0x7f079406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.855+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 0x7f079406d5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f07a4035000 tx=0x7f07a4030000 comp rx=0 tx=0).stop 2023-10-14T18:18:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 msgr2=0x7f07b01fa0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 secure :-1 s=READY pgs=2792 cs=0 l=1 rev1=1 crypto rx=0x7f07a800d800 tx=0x7f07a800dcc0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 shutdown_connections 2023-10-14T18:18:16.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f079406b0e0 0x7f079406d5a0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f07b00c9000 0x7f07b01fa0e0 unknown :-1 s=CLOSED pgs=2792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f07b00addb0 0x7f07b00c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 --2- 172.21.15.29:0/271407863 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f07b006a2e0 0x7f07b00b44e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:16.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 >> 172.21.15.29:0/271407863 conn(0x7f07b00ce9b0 msgr2=0x7f07b0164870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:16.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 shutdown_connections 2023-10-14T18:18:16.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:16.856+0000 7f07b82e7640 1 -- 172.21.15.29:0/271407863 wait complete. 2023-10-14T18:18:16.878 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:18:17.085 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:18:17.095 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:18:17.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.104+0000 7facaf59e640 1 Processor -- start 2023-10-14T18:18:17.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.105+0000 7facaf59e640 1 -- start start 2023-10-14T18:18:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.106+0000 7facaf59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7facb0159950 0x7facb015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.106+0000 7facaf59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb0070450 0x7facb00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.106+0000 7facaf59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facb0070df0 con 0x7facb006c100 2023-10-14T18:18:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.106+0000 7facaf59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7facb015e490 con 0x7facb0159950 2023-10-14T18:18:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.106+0000 7facaf59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb015aef0 con 0x7facb0070450 2023-10-14T18:18:17.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facae59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7facb0159950 0x7facb015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facaed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7facb006c100 0x7facb006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59834/0 (socket says 172.21.15.29:59834) 2023-10-14T18:18:17.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facaed9d640 1 -- 172.21.15.29:0/1157093118 learned_addr learned my addr 172.21.15.29:0/1157093118 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:17.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facadd9b640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb0070450 0x7facb00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facae59c640 1 -- 172.21.15.29:0/1157093118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb0070450 msgr2=0x7facb00708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:17.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facad59a640 1 -- 172.21.15.29:0/1157093118 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104831623 0 0) 0x7facb0070df0 con 0x7facb006c100 2023-10-14T18:18:17.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facae59c640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb0070450 0x7facb00708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facae59c640 1 -- 172.21.15.29:0/1157093118 >> v1:172.21.15.29:6789/0 conn(0x7facb006c100 legacy=0x7facb006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.107+0000 7facae59c640 1 -- 172.21.15.29:0/1157093118 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facb006cc80 con 0x7facb0159950 2023-10-14T18:18:17.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.108+0000 7facae59c640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb015a9b0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7faca00097a0 tx=0x7faca002b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f91e27f47152694 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facad59a640 1 -- 172.21.15.29:0/1157093118 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0031780 con 0x7facb0159950 2023-10-14T18:18:17.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facad59a640 1 -- 172.21.15.29:0/1157093118 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca00318e0 con 0x7facb0159950 2023-10-14T18:18:17.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facad59a640 1 -- 172.21.15.29:0/1157093118 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca0031c70 con 0x7facb0159950 2023-10-14T18:18:17.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 -- 172.21.15.29:0/1157093118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 msgr2=0x7facb015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb015a9b0 secure :-1 s=READY pgs=2794 cs=0 l=1 rev1=1 crypto rx=0x7faca00097a0 tx=0x7faca002b820 comp rx=0 tx=0).stop 2023-10-14T18:18:17.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 -- 172.21.15.29:0/1157093118 shutdown_connections 2023-10-14T18:18:17.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb0070450 0x7facb00708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 --2- 172.21.15.29:0/1157093118 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb015a9b0 unknown :-1 s=CLOSED pgs=2794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 -- 172.21.15.29:0/1157093118 >> 172.21.15.29:0/1157093118 conn(0x7facb00ce6a0 msgr2=0x7facb00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:17.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.109+0000 7facaf59e640 1 -- 172.21.15.29:0/1157093118 shutdown_connections 2023-10-14T18:18:17.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.110+0000 7facaf59e640 1 -- 172.21.15.29:0/1157093118 wait complete. 2023-10-14T18:18:17.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.111+0000 7facaf59e640 1 Processor -- start 2023-10-14T18:18:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.111+0000 7facaf59e640 1 -- start start 2023-10-14T18:18:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 0x7facb014f3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facb0070450 0x7facb014ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb014e300 con 0x7facb0159950 2023-10-14T18:18:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facb015bf70 con 0x7facb0070450 2023-10-14T18:18:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaf59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7facb015b240 con 0x7facb006c100 2023-10-14T18:18:17.119 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.112+0000 7facaed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42836/0 (socket says 172.21.15.29:42836) 2023-10-14T18:18:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 -- 172.21.15.29:0/1412123169 learned_addr learned my addr 172.21.15.29:0/1412123169 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facae59c640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 0x7facb014f3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 -- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 msgr2=0x7facb014f3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facadd9b640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facb0070450 0x7facb014ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 0x7facb014f3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 -- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facb0070450 msgr2=0x7facb014ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facb0070450 0x7facb014ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 -- 172.21.15.29:0/1412123169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca0009400 con 0x7facb0159950 2023-10-14T18:18:17.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facae59c640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 0x7facb014f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:17.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.113+0000 7facaed9d640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7faca800c7b0 tx=0x7faca800cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.114+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca800a5e0 con 0x7facb0159950 2023-10-14T18:18:17.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.114+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca800a760 con 0x7facb0159950 2023-10-14T18:18:17.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.114+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faca800ab10 con 0x7facb0159950 2023-10-14T18:18:17.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.114+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facb015bae0 con 0x7facb0159950 2023-10-14T18:18:17.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.114+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facb015b490 con 0x7facb0159950 2023-10-14T18:18:17.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.115+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb015bc70 con 0x7facb0159950 2023-10-14T18:18:17.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.116+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faca800ac70 con 0x7facb0159950 2023-10-14T18:18:17.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.117+0000 7fac9effd640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 0x7fac9006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.117+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7faca80892c0 con 0x7facb0159950 2023-10-14T18:18:17.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.117+0000 7facae59c640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 0x7fac9006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.118+0000 7facae59c640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 0x7fac9006d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faca0036000 tx=0x7faca003d040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.122+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faca8053b60 con 0x7facb0159950 2023-10-14T18:18:17.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.311+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7facb004e9b0 con 0x7facb0159950 2023-10-14T18:18:17.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.312+0000 7fac9effd640 1 -- 172.21.15.29:0/1412123169 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v238) v1 ==== 103+0+11 (secure 0 0 0) 0x7faca8057180 con 0x7facb0159950 2023-10-14T18:18:17.314 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 msgr2=0x7fac9006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 0x7fac9006d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7faca0036000 tx=0x7faca003d040 comp rx=0 tx=0).stop 2023-10-14T18:18:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 msgr2=0x7facb00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 secure :-1 s=READY pgs=2795 cs=0 l=1 rev1=1 crypto rx=0x7faca800c7b0 tx=0x7faca800cc70 comp rx=0 tx=0).stop 2023-10-14T18:18:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 shutdown_connections 2023-10-14T18:18:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fac9006b0b0 0x7fac9006d570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7facb0159950 0x7facb00ac890 unknown :-1 s=CLOSED pgs=2795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7facb0070450 0x7facb014ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 --2- 172.21.15.29:0/1412123169 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7facb006c100 0x7facb014f3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 >> 172.21.15.29:0/1412123169 conn(0x7facb00ce6a0 msgr2=0x7facb00c9bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 shutdown_connections 2023-10-14T18:18:17.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.315+0000 7facaf59e640 1 -- 172.21.15.29:0/1412123169 wait complete. 2023-10-14T18:18:17.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:17.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.556+0000 7f50b5b97640 1 Processor -- start 2023-10-14T18:18:17.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.556+0000 7f50b5b97640 1 -- start start 2023-10-14T18:18:17.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50b5b97640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f50b015ae80 0x7f50b015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50b5b97640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f50b00ce390 0x7f50b0070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50b5b97640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50b015ed60 con 0x7f50b015aaa0 2023-10-14T18:18:17.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50b5b97640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b006b810 con 0x7f50b015ae80 2023-10-14T18:18:17.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50b5b97640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50b00707e0 con 0x7f50b00ce390 2023-10-14T18:18:17.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50af7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f50b015aaa0 0x7f50b00cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59842/0 (socket says 172.21.15.29:59842) 2023-10-14T18:18:17.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50af7fe640 1 -- 172.21.15.29:0/3906285084 learned_addr learned my addr 172.21.15.29:0/3906285084 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:17.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.558+0000 7f50a7fff640 1 --2- 172.21.15.29:0/3906285084 >> v2:172.21.15.29:3300/0 conn(0x7f50b015ae80 0x7f50b015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50affff640 1 --2- 172.21.15.29:0/3906285084 >> v2:172.21.15.186:3300/0 conn(0x7f50b00ce390 0x7f50b0070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50aeffd640 1 -- 172.21.15.29:0/3906285084 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3489138023 0 0) 0x7f50b015ed60 con 0x7f50b015aaa0 2023-10-14T18:18:17.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50aeffd640 1 -- 172.21.15.29:0/3906285084 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f509c003520 con 0x7f50b015aaa0 2023-10-14T18:18:17.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50a7fff640 1 -- 172.21.15.29:0/3906285084 >> v2:172.21.15.186:3300/0 conn(0x7f50b00ce390 msgr2=0x7f50b0070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:17.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50a7fff640 1 --2- 172.21.15.29:0/3906285084 >> v2:172.21.15.186:3300/0 conn(0x7f50b00ce390 0x7f50b0070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50a7fff640 1 -- 172.21.15.29:0/3906285084 >> v1:172.21.15.29:6789/0 conn(0x7f50b015aaa0 legacy=0x7f50b00cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.559+0000 7f50a7fff640 1 -- 172.21.15.29:0/3906285084 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50b015ef90 con 0x7f50b015ae80 2023-10-14T18:18:17.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.561+0000 7f50a7fff640 1 --2- 172.21.15.29:0/3906285084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b015ae80 0x7f50b015e820 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f50b00c2490 tx=0x7f509802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e41d18fad48a48 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.561+0000 7f50aeffd640 1 -- 172.21.15.29:0/3906285084 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098031830 con 0x7f50b015ae80 2023-10-14T18:18:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.561+0000 7f50aeffd640 1 -- 172.21.15.29:0/3906285084 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5098031990 con 0x7f50b015ae80 2023-10-14T18:18:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.561+0000 7f50aeffd640 1 -- 172.21.15.29:0/3906285084 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5098031d00 con 0x7f50b015ae80 2023-10-14T18:18:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 -- 172.21.15.29:0/3906285084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b015ae80 msgr2=0x7f50b015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 --2- 172.21.15.29:0/3906285084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b015ae80 0x7f50b015e820 secure :-1 s=READY pgs=2797 cs=0 l=1 rev1=1 crypto rx=0x7f50b00c2490 tx=0x7f509802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 -- 172.21.15.29:0/3906285084 shutdown_connections 2023-10-14T18:18:17.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 --2- 172.21.15.29:0/3906285084 >> v2:172.21.15.186:3300/0 conn(0x7f50b00ce390 0x7f50b0070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 --2- 172.21.15.29:0/3906285084 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b015ae80 0x7f50b015e820 unknown :-1 s=CLOSED pgs=2797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 -- 172.21.15.29:0/3906285084 >> 172.21.15.29:0/3906285084 conn(0x7f50b00cb410 msgr2=0x7f50b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:17.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 -- 172.21.15.29:0/3906285084 shutdown_connections 2023-10-14T18:18:17.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.562+0000 7f50b5b97640 1 -- 172.21.15.29:0/3906285084 wait complete. 2023-10-14T18:18:17.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.564+0000 7f50b5b97640 1 Processor -- start 2023-10-14T18:18:17.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.564+0000 7f50b5b97640 1 -- start start 2023-10-14T18:18:17.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 0x7f50b00caa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 0x7f50b00b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00af280 con 0x7f50b00ce390 2023-10-14T18:18:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00c5fb0 con 0x7f50b015ae80 2023-10-14T18:18:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50b5b97640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50b00c6290 con 0x7f50b015aaa0 2023-10-14T18:18:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.565+0000 7f50af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42856/0 (socket says 172.21.15.29:42856) 2023-10-14T18:18:17.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 -- 172.21.15.29:0/460971269 learned_addr learned my addr 172.21.15.29:0/460971269 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 -- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 msgr2=0x7f50b00caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50affff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 0x7f50b00b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50a7fff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 0x7f50b00caa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 0x7f50b00caa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 -- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 msgr2=0x7f50b00b09d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 0x7f50b00b09d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 -- 172.21.15.29:0/460971269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5098009400 con 0x7f50b00ce390 2023-10-14T18:18:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50affff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 0x7f50b00b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:17.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50a7fff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 0x7f50b00caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.566+0000 7f50af7fe640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c430 tx=0x7f50a000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.567+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a0005ce0 con 0x7f50b00ce390 2023-10-14T18:18:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.567+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50a0005e40 con 0x7f50b00ce390 2023-10-14T18:18:17.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.567+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50b00af540 con 0x7f50b00ce390 2023-10-14T18:18:17.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.567+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a00072f0 con 0x7f50b00ce390 2023-10-14T18:18:17.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.567+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50b00caf80 con 0x7f50b00ce390 2023-10-14T18:18:17.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.568+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50b00afa60 con 0x7f50b00ce390 2023-10-14T18:18:17.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.569+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f50a0007450 con 0x7f50b00ce390 2023-10-14T18:18:17.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.570+0000 7f50acff9640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 0x7f509006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:17.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.570+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f50a0089580 con 0x7f50b00ce390 2023-10-14T18:18:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.570+0000 7f50a7fff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 0x7f509006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.571+0000 7f50a7fff640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 0x7f509006d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5098036000 tx=0x7f509803c040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:17.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.575+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f50b00afa60 con 0x7f50b00ce390 2023-10-14T18:18:17.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.810+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50b00afa60 con 0x7f50b00ce390 2023-10-14T18:18:17.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.811+0000 7f50acff9640 1 -- 172.21.15.29:0/460971269 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f50b00afa60 con 0x7f50b00ce390 2023-10-14T18:18:17.814 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:17.814 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":115,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11910,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1839495929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1839495929},{"type":"v1","addr":"172.21.15.29:6837","nonce":1839495929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:17.814 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 115 2023-10-14T18:18:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 msgr2=0x7f509006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 0x7f509006d570 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5098036000 tx=0x7f509803c040 comp rx=0 tx=0).stop 2023-10-14T18:18:17.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 msgr2=0x7f50b00afd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:17.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 secure :-1 s=READY pgs=2798 cs=0 l=1 rev1=1 crypto rx=0x7f50a000c430 tx=0x7f50a000c8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 shutdown_connections 2023-10-14T18:18:17.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f509006b0b0 0x7f509006d570 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f50b015ae80 0x7f50b00b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f50b015aaa0 0x7f50b00caa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 --2- 172.21.15.29:0/460971269 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f50b00ce390 0x7f50b00afd40 unknown :-1 s=CLOSED pgs=2798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:17.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 >> 172.21.15.29:0/460971269 conn(0x7f50b00cb410 msgr2=0x7f50b006cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:17.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 shutdown_connections 2023-10-14T18:18:17.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:17.814+0000 7f50b5b97640 1 -- 172.21.15.29:0/460971269 wait complete. 2023-10-14T18:18:17.835 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:18.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.046+0000 7f963b8e0640 1 Processor -- start 2023-10-14T18:18:18.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.046+0000 7f963b8e0640 1 -- start start 2023-10-14T18:18:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.048+0000 7f963b8e0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f963415ae80 0x7f963415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.048+0000 7f963b8e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 0x7f9634070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.048+0000 7f963b8e0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f963415ed60 con 0x7f963415aaa0 2023-10-14T18:18:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.048+0000 7f963b8e0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f963406b810 con 0x7f963415ae80 2023-10-14T18:18:18.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.048+0000 7f963b8e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96340707e0 con 0x7f96340ce390 2023-10-14T18:18:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f963415ae80 0x7f963415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9639e7d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 0x7f9634070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f963415ae80 0x7f963415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:34780/0 (socket says 172.21.15.29:34780) 2023-10-14T18:18:18.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 -- 172.21.15.29:0/841624854 learned_addr learned my addr 172.21.15.29:0/841624854 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9639e7d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 0x7f9634070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:42280/0 (socket says 172.21.15.29:42280) 2023-10-14T18:18:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f963967c640 1 --1- 172.21.15.29:0/841624854 >> v1:172.21.15.186:6789/0 conn(0x7f963415aaa0 0x7f96340cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:54752/0 (socket says 172.21.15.29:54752) 2023-10-14T18:18:18.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 -- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 msgr2=0x7f9634070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 --2- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 0x7f9634070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 -- 172.21.15.29:0/841624854 >> v1:172.21.15.186:6789/0 conn(0x7f963415aaa0 legacy=0x7f96340cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.049+0000 7f9638e7b640 1 -- 172.21.15.29:0/841624854 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f963415ef90 con 0x7f963415ae80 2023-10-14T18:18:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.051+0000 7f9638e7b640 1 --2- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963415e820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9630009870 tx=0x7f963002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5e3211fad9d68f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.051+0000 7f962bfff640 1 -- 172.21.15.29:0/841624854 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630030840 con 0x7f963415ae80 2023-10-14T18:18:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f962bfff640 1 -- 172.21.15.29:0/841624854 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300309a0 con 0x7f963415ae80 2023-10-14T18:18:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f962bfff640 1 -- 172.21.15.29:0/841624854 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9630030d30 con 0x7f963415ae80 2023-10-14T18:18:18.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 -- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 msgr2=0x7f963415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 --2- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963415e820 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f9630009870 tx=0x7f963002b980 comp rx=0 tx=0).stop 2023-10-14T18:18:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 -- 172.21.15.29:0/841624854 shutdown_connections 2023-10-14T18:18:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 --2- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96340ce390 0x7f9634070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 --2- 172.21.15.29:0/841624854 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963415e820 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 -- 172.21.15.29:0/841624854 >> 172.21.15.29:0/841624854 conn(0x7f96340cb410 msgr2=0x7f96340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 -- 172.21.15.29:0/841624854 shutdown_connections 2023-10-14T18:18:18.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.052+0000 7f963b8e0640 1 -- 172.21.15.29:0/841624854 wait complete. 2023-10-14T18:18:18.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.054+0000 7f963b8e0640 1 Processor -- start 2023-10-14T18:18:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.054+0000 7f963b8e0640 1 -- start start 2023-10-14T18:18:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.055+0000 7f963b8e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.055+0000 7f963b8e0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f963415aaa0 0x7f963406cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.055+0000 7f963b8e0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963406d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963b8e0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634156ff0 con 0x7f96340ce390 2023-10-14T18:18:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963b8e0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9634157a90 con 0x7f963415ae80 2023-10-14T18:18:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963b8e0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9634156d10 con 0x7f963415aaa0 2023-10-14T18:18:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42866/0 (socket says 172.21.15.29:42866) 2023-10-14T18:18:18.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 -- 172.21.15.29:0/3666188026 learned_addr learned my addr 172.21.15.29:0/3666188026 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f9638e7b640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f963415aaa0 0x7f963406cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f9639e7d640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963406d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 -- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f963415aaa0 msgr2=0x7f963406cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f963415aaa0 0x7f963406cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 -- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 msgr2=0x7f963406d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963406d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.056+0000 7f963967c640 1 -- 172.21.15.29:0/3666188026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96300094d0 con 0x7f96340ce390 2023-10-14T18:18:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f963967c640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f962400c450 tx=0x7f962400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624007480 con 0x7f96340ce390 2023-10-14T18:18:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9624007600 con 0x7f96340ce390 2023-10-14T18:18:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9624004a20 con 0x7f96340ce390 2023-10-14T18:18:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9634157440 con 0x7f96340ce390 2023-10-14T18:18:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.057+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96341569a0 con 0x7f96340ce390 2023-10-14T18:18:18.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.058+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f963404e160 con 0x7f96340ce390 2023-10-14T18:18:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.059+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9624004b80 con 0x7f96340ce390 2023-10-14T18:18:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.060+0000 7f9629ffb640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 0x7f962006d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.061+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f9624088f70 con 0x7f96340ce390 2023-10-14T18:18:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.061+0000 7f9638e7b640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 0x7f962006d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.062+0000 7f9638e7b640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 0x7f962006d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9630036000 tx=0x7f963003b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.065+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9624053810 con 0x7f96340ce390 2023-10-14T18:18:18.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.264+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f96340717e0 con 0x7f96340ce390 2023-10-14T18:18:18.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.267+0000 7f9629ffb640 1 -- 172.21.15.29:0/3666188026 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+18081 (secure 0 0 0) 0x7f9624018090 con 0x7f96340ce390 2023-10-14T18:18:18.270 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:18.270 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":238,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:15.443657+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6836/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6837/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:18.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 msgr2=0x7f962006d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 0x7f962006d540 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f9630036000 tx=0x7f963003b040 comp rx=0 tx=0).stop 2023-10-14T18:18:18.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 msgr2=0x7f9634155820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 secure :-1 s=READY pgs=2799 cs=0 l=1 rev1=1 crypto rx=0x7f962400c450 tx=0x7f962400c910 comp rx=0 tx=0).stop 2023-10-14T18:18:18.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 shutdown_connections 2023-10-14T18:18:18.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f962006b080 0x7f962006d540 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f963415ae80 0x7f963406d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f963415aaa0 0x7f963406cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 --2- 172.21.15.29:0/3666188026 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96340ce390 0x7f9634155820 unknown :-1 s=CLOSED pgs=2799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 >> 172.21.15.29:0/3666188026 conn(0x7f96340cb410 msgr2=0x7f96340b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:18.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 shutdown_connections 2023-10-14T18:18:18.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.270+0000 7f963b8e0640 1 -- 172.21.15.29:0/3666188026 wait complete. 2023-10-14T18:18:18.292 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-14T18:18:18.293 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.509+0000 7fd6b5178640 1 Processor -- start 2023-10-14T18:18:18.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.509+0000 7fd6b5178640 1 -- start start 2023-10-14T18:18:18.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6b5178640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 0x7fd6b00b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6b5178640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b006f7f0 con 0x7fd6b015d880 2023-10-14T18:18:18.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6b5178640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6b006c5e0 con 0x7fd6b0159b20 2023-10-14T18:18:18.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6b5178640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b006c8c0 con 0x7fd6b0159f00 2023-10-14T18:18:18.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6af59e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd6b015d880 0x7fd6b006fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59848/0 (socket says 172.21.15.29:59848) 2023-10-14T18:18:18.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.510+0000 7fd6af59e640 1 -- 172.21.15.29:0/3423596529 learned_addr learned my addr 172.21.15.29:0/3423596529 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6a7fff640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 0x7fd6b00b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649497702 0 0) 0x7fd6b006f7f0 con 0x7fd6b015d880 2023-10-14T18:18:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd69c003520 con 0x7fd6b015d880 2023-10-14T18:18:18.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 70153822 0 0) 0x7fd69c003520 con 0x7fd6b015d880 2023-10-14T18:18:18.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 msgr2=0x7fd6b00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:18.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 0x7fd6b00b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 >> v1:172.21.15.186:6789/0 conn(0x7fd6b0159b20 legacy=0x7fd6b006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b006cba0 con 0x7fd6b015d880 2023-10-14T18:18:18.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fd6a80019f0 con 0x7fd6b015d880 2023-10-14T18:18:18.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 >> v1:172.21.15.29:6789/0 conn(0x7fd6b015d880 legacy=0x7fd6b006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c0039a0 0x7fd69c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.511+0000 7fd6ae59c640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd69c0047c0 0x7fd69c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6ae59c640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 0x7fd69c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b006c8c0 con 0x7fd69c0039a0 2023-10-14T18:18:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c00dc40 con 0x7fd69c0047c0 2023-10-14T18:18:18.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd69c00de00 con 0x7fd6b0159f00 2023-10-14T18:18:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c0039a0 0x7fd69c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 -- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 msgr2=0x7fd69c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd6b0159f00 0x7fd69c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6a7fff640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd69c0047c0 0x7fd69c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 -- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd69c0047c0 msgr2=0x7fd69c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd69c0047c0 0x7fd69c004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.512+0000 7fd6aed9d640 1 -- 172.21.15.29:0/3423596529 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd69c003520 con 0x7fd69c0039a0 2023-10-14T18:18:18.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c0039a0 0x7fd69c003dc0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0001720 tx=0x7fd69c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=878c3aff70bc22f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00052d0 con 0x7fd69c0039a0 2023-10-14T18:18:18.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6a002e050 con 0x7fd69c0039a0 2023-10-14T18:18:18.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6ae59c640 1 -- 172.21.15.29:0/3423596529 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a00035f0 con 0x7fd69c0039a0 2023-10-14T18:18:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6b5178640 1 -- 172.21.15.29:0/3423596529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c0039a0 msgr2=0x7fd69c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6b5178640 1 --2- 172.21.15.29:0/3423596529 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd69c0039a0 0x7fd69c003dc0 secure :-1 s=READY pgs=2801 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0001720 tx=0x7fd69c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.514+0000 7fd6af59e640 1 -- 172.21.15.29:0/3423596529 reap_dead start 2023-10-14T18:18:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.515+0000 7fd6b5178640 1 -- 172.21.15.29:0/3423596529 shutdown_connections 2023-10-14T18:18:18.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.515+0000 7fd6b5178640 1 -- 172.21.15.29:0/3423596529 >> 172.21.15.29:0/3423596529 conn(0x7fd6b00cb410 msgr2=0x7fd6b00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:18.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.515+0000 7fd6b5178640 1 -- 172.21.15.29:0/3423596529 shutdown_connections 2023-10-14T18:18:18.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.515+0000 7fd6b5178640 1 -- 172.21.15.29:0/3423596529 wait complete. 2023-10-14T18:18:18.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.516+0000 7fd6b5178640 1 Processor -- start 2023-10-14T18:18:18.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.517+0000 7fd6b5178640 1 -- start start 2023-10-14T18:18:18.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd69c0039a0 0x7fd6b015c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 0x7fd6b00a8480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015b040 con 0x7fd6b0158cb0 2023-10-14T18:18:18.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015b2e0 con 0x7fd6b014dbf0 2023-10-14T18:18:18.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6b5178640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6b015b5c0 con 0x7fd69c0039a0 2023-10-14T18:18:18.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42878/0 (socket says 172.21.15.29:42878) 2023-10-14T18:18:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 -- 172.21.15.29:0/153770928 learned_addr learned my addr 172.21.15.29:0/153770928 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd69c0039a0 0x7fd6b015c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6af59e640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 0x7fd6b00a8480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 -- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd69c0039a0 msgr2=0x7fd6b015c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd69c0039a0 0x7fd6b015c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 -- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 msgr2=0x7fd6b00a8480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 0x7fd6b00a8480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6a7fff640 1 -- 172.21.15.29:0/153770928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6b015b8a0 con 0x7fd6b0158cb0 2023-10-14T18:18:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.518+0000 7fd6af59e640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 0x7fd6b00a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6a7fff640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fd698010890 tx=0x7fd698010d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980095b0 con 0x7fd6b0158cb0 2023-10-14T18:18:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd69800eb80 con 0x7fd6b0158cb0 2023-10-14T18:18:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69801c400 con 0x7fd6b0158cb0 2023-10-14T18:18:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6b015bb80 con 0x7fd6b0158cb0 2023-10-14T18:18:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.519+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6b014e070 con 0x7fd6b0158cb0 2023-10-14T18:18:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.521+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6b014e290 con 0x7fd6b0158cb0 2023-10-14T18:18:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.521+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd6980143f0 con 0x7fd6b0158cb0 2023-10-14T18:18:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.522+0000 7fd6a77fe640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 0x7fd69006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.522+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7fd698089f20 con 0x7fd6b0158cb0 2023-10-14T18:18:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.523+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 0x7fd69006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.523+0000 7fd6aed9d640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 0x7fd69006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0033000 tx=0x7fd6a0009030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:18.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.527+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd698054850 con 0x7fd6b0158cb0 2023-10-14T18:18:18.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.762+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6b006c270 con 0x7fd6b0158cb0 2023-10-14T18:18:18.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.764+0000 7fd6a77fe640 1 -- 172.21.15.29:0/153770928 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4936 (secure 0 0 0) 0x7fd698057e70 con 0x7fd6b0158cb0 2023-10-14T18:18:18.766 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:18.766 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":115,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11910,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1839495929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1839495929},{"type":"v1","addr":"172.21.15.29:6837","nonce":1839495929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:18.766 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 115 2023-10-14T18:18:18.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.766+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 msgr2=0x7fd69006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.766+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 0x7fd69006d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fd6a0033000 tx=0x7fd6a0009030 comp rx=0 tx=0).stop 2023-10-14T18:18:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.766+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 msgr2=0x7fd6b015ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.766+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 secure :-1 s=READY pgs=2802 cs=0 l=1 rev1=1 crypto rx=0x7fd698010890 tx=0x7fd698010d50 comp rx=0 tx=0).stop 2023-10-14T18:18:18.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 shutdown_connections 2023-10-14T18:18:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd69006b0b0 0x7fd69006d570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd6b014dbf0 0x7fd6b00a8480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd6b0158cb0 0x7fd6b015ab00 unknown :-1 s=CLOSED pgs=2802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 --2- 172.21.15.29:0/153770928 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd69c0039a0 0x7fd6b015c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:18.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 >> 172.21.15.29:0/153770928 conn(0x7fd6b00cb410 msgr2=0x7fd6b00ae3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 shutdown_connections 2023-10-14T18:18:18.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:18.767+0000 7fd6b5178640 1 -- 172.21.15.29:0/153770928 wait complete. 2023-10-14T18:18:18.791 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.014+0000 7f664a790640 1 Processor -- start 2023-10-14T18:18:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.014+0000 7f664a790640 1 -- start start 2023-10-14T18:18:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.015+0000 7f664a790640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f664406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f664a790640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664406f7f0 con 0x7f66440b0b20 2023-10-14T18:18:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f664a790640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664406c5e0 con 0x7f6644159f00 2023-10-14T18:18:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f664a790640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664406c8c0 con 0x7f6644159b20 2023-10-14T18:18:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f6648d2d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f66440b0b20 0x7f664406fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59864/0 (socket says 172.21.15.29:59864) 2023-10-14T18:18:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f6648d2d640 1 -- 172.21.15.29:0/2656977583 learned_addr learned my addr 172.21.15.29:0/2656977583 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f6643fff640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f664406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122495869 0 0) 0x7f664406f7f0 con 0x7f66440b0b20 2023-10-14T18:18:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.016+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6630003520 con 0x7f66440b0b20 2023-10-14T18:18:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2975755595 0 0) 0x7f6630003520 con 0x7f66440b0b20 2023-10-14T18:18:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 msgr2=0x7f664406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f664406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 >> v1:172.21.15.186:6789/0 conn(0x7f6644159f00 legacy=0x7f66440b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664406cba0 con 0x7f66440b0b20 2023-10-14T18:18:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f663c001670 con 0x7f66440b0b20 2023-10-14T18:18:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 >> v1:172.21.15.29:6789/0 conn(0x7f66440b0b20 legacy=0x7f664406fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f66300039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630003f20 0x7f66300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6630004d00 0x7f6630005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f664406c8c0 con 0x7f6630004d00 2023-10-14T18:18:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f663000ba60 con 0x7f6630003f20 2023-10-14T18:18:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f663000de30 con 0x7f6644159b20 2023-10-14T18:18:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.017+0000 7f6648d2d640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6630004d00 0x7f6630005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.024 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6643fff640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f66300039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f663b7fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630003f20 0x7f66300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6648d2d640 1 -- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 msgr2=0x7f66300039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6648d2d640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6644159b20 0x7f66300039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.025 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6648d2d640 1 -- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630003f20 msgr2=0x7f66300047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6648d2d640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630003f20 0x7f66300047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f6648d2d640 1 -- 172.21.15.29:0/2656977583 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6630003520 con 0x7f6630004d00 2023-10-14T18:18:19.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.018+0000 7f663b7fe640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630003f20 0x7f66300047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:19.029 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.019+0000 7f6648d2d640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6630004d00 0x7f6630005180 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f66440c5650 tx=0x7f66300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4402f58d2c836e1e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.019+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c005ad0 con 0x7f6630004d00 2023-10-14T18:18:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.019+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c005c30 con 0x7f6630004d00 2023-10-14T18:18:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.019+0000 7f66437fe640 1 -- 172.21.15.29:0/2656977583 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f663c02ca90 con 0x7f6630004d00 2023-10-14T18:18:19.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 -- 172.21.15.29:0/2656977583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6630004d00 msgr2=0x7f6630005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 --2- 172.21.15.29:0/2656977583 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6630004d00 0x7f6630005180 secure :-1 s=READY pgs=2804 cs=0 l=1 rev1=1 crypto rx=0x7f66440c5650 tx=0x7f66300057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f6648d2d640 1 -- 172.21.15.29:0/2656977583 reap_dead start 2023-10-14T18:18:19.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 -- 172.21.15.29:0/2656977583 shutdown_connections 2023-10-14T18:18:19.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 -- 172.21.15.29:0/2656977583 >> 172.21.15.29:0/2656977583 conn(0x7f66440cb410 msgr2=0x7f66440cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:19.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 -- 172.21.15.29:0/2656977583 shutdown_connections 2023-10-14T18:18:19.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.020+0000 7f664a790640 1 -- 172.21.15.29:0/2656977583 wait complete. 2023-10-14T18:18:19.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.022+0000 7f664a790640 1 Processor -- start 2023-10-14T18:18:19.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.022+0000 7f664a790640 1 -- start start 2023-10-14T18:18:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630004d00 0x7f66441579e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66440c7d60 0x7f66440c8210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66440c3510 con 0x7f66441f2a50 2023-10-14T18:18:19.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66441f3560 con 0x7f6630004d00 2023-10-14T18:18:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f664a790640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66441f3840 con 0x7f66440c7d60 2023-10-14T18:18:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f663b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f663b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42900/0 (socket says 172.21.15.29:42900) 2023-10-14T18:18:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.023+0000 7f663b7fe640 1 -- 172.21.15.29:0/2937254493 learned_addr learned my addr 172.21.15.29:0/2937254493 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:19.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f6643fff640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630004d00 0x7f66441579e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f6648d2d640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66440c7d60 0x7f66440c8210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 -- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66440c7d60 msgr2=0x7f66440c8210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66440c7d60 0x7f66440c8210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 -- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630004d00 msgr2=0x7f66441579e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630004d00 0x7f66441579e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 -- 172.21.15.29:0/2937254493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66440c3720 con 0x7f66441f2a50 2023-10-14T18:18:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.024+0000 7f663b7fe640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f662c00d7a0 tx=0x7f662c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.025+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c017070 con 0x7f66441f2a50 2023-10-14T18:18:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.025+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f662c005070 con 0x7f66441f2a50 2023-10-14T18:18:19.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.025+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f662c005420 con 0x7f66441f2a50 2023-10-14T18:18:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.025+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66440c3a00 con 0x7f66441f2a50 2023-10-14T18:18:19.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.025+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66440c5cf0 con 0x7f66441f2a50 2023-10-14T18:18:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.026+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664406ce60 con 0x7f66441f2a50 2023-10-14T18:18:19.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.027+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f662c007030 con 0x7f66441f2a50 2023-10-14T18:18:19.040 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.028+0000 7f66417fa640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 0x7f662406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.028+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f662c0897e0 con 0x7f66441f2a50 2023-10-14T18:18:19.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.028+0000 7f6643fff640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 0x7f662406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.029+0000 7f6643fff640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 0x7f662406d5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6634007520 tx=0x7f663400b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.033+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f664406ce60 con 0x7f66441f2a50 2023-10-14T18:18:19.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.270+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664406ce60 con 0x7f66441f2a50 2023-10-14T18:18:19.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.272+0000 7f66417fa640 1 -- 172.21.15.29:0/2937254493 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+4936 (secure 0 0 0) 0x7f664406ce60 con 0x7f66441f2a50 2023-10-14T18:18:19.274 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:19.274 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":115,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11910,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1839495929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1839495929},{"type":"v1","addr":"172.21.15.29:6837","nonce":1839495929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115}],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:17:41.791634+0000","modified":"2023-10-14T18:17:44.857465+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":11256},"failed":[],"damaged":[],"stopped":[],"info":{"gid_11256":{"gid":11256,"name":"c","rank":0,"incarnation":109,"state":"up:active","state_seq":7,"addr":"172.21.15.29:6835/2605417960","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2605417960},{"type":"v1","addr":"172.21.15.29:6835","nonce":2605417960}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[29],"metadata_pool":28,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":14}]} 2023-10-14T18:18:19.274 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 115 2023-10-14T18:18:19.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 msgr2=0x7f662406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 0x7f662406d5a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f6634007520 tx=0x7f663400b040 comp rx=0 tx=0).stop 2023-10-14T18:18:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 msgr2=0x7f66441f2ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 secure :-1 s=READY pgs=2805 cs=0 l=1 rev1=1 crypto rx=0x7f662c00d7a0 tx=0x7f662c00dc60 comp rx=0 tx=0).stop 2023-10-14T18:18:19.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 shutdown_connections 2023-10-14T18:18:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.274+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f662406b0e0 0x7f662406d5a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f66440c7d60 0x7f66440c8210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f66441f2a50 0x7f66441f2ed0 unknown :-1 s=CLOSED pgs=2805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 --2- 172.21.15.29:0/2937254493 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6630004d00 0x7f66441579e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 >> 172.21.15.29:0/2937254493 conn(0x7f66440cb410 msgr2=0x7f66440b0400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 shutdown_connections 2023-10-14T18:18:19.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.275+0000 7f664a790640 1 -- 172.21.15.29:0/2937254493 wait complete. 2023-10-14T18:18:19.297 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:19.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.513+0000 7f7dd901a640 1 Processor -- start 2023-10-14T18:18:19.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.514+0000 7f7dd901a640 1 -- start start 2023-10-14T18:18:19.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7dd406d520 0x7f7dd40b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7dd406fa20 0x7f7dd40ac750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40acc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd415b1d0 con 0x7f7dd406d520 2023-10-14T18:18:19.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dd406d900 con 0x7f7dd406fa20 2023-10-14T18:18:19.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.515+0000 7f7dd901a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd406feb0 con 0x7f7dd4065cc0 2023-10-14T18:18:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7dd406d520 0x7f7dd40b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7dd406d520 0x7f7dd40b1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42916/0 (socket says 172.21.15.29:42916) 2023-10-14T18:18:19.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 -- 172.21.15.29:0/2526298855 learned_addr learned my addr 172.21.15.29:0/2526298855 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dcbfff640 1 --2- 172.21.15.29:0/2526298855 >> v2:172.21.15.186:3300/0 conn(0x7f7dd406fa20 0x7f7dd40ac750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd8819640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40acc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 -- 172.21.15.29:0/2526298855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 msgr2=0x7f7dd40acc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40acc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 -- 172.21.15.29:0/2526298855 >> v2:172.21.15.186:3300/0 conn(0x7f7dd406fa20 msgr2=0x7f7dd40ac750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/2526298855 >> v2:172.21.15.186:3300/0 conn(0x7f7dd406fa20 0x7f7dd40ac750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.516+0000 7f7dd3fff640 1 -- 172.21.15.29:0/2526298855 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dd4070190 con 0x7f7dd406d520 2023-10-14T18:18:19.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.517+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd406d520 0x7f7dd40b1de0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40c2610 tx=0x7f7dc402b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de52b509968fe76e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.518+0000 7f7dd37fe640 1 -- 172.21.15.29:0/2526298855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4031750 con 0x7f7dd406d520 2023-10-14T18:18:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.518+0000 7f7dd37fe640 1 -- 172.21.15.29:0/2526298855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc40318b0 con 0x7f7dd406d520 2023-10-14T18:18:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.518+0000 7f7dd37fe640 1 -- 172.21.15.29:0/2526298855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dc4031c40 con 0x7f7dd406d520 2023-10-14T18:18:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 -- 172.21.15.29:0/2526298855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd406d520 msgr2=0x7f7dd40b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd406d520 0x7f7dd40b1de0 secure :-1 s=READY pgs=2806 cs=0 l=1 rev1=1 crypto rx=0x7f7dd40c2610 tx=0x7f7dc402b7f0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 -- 172.21.15.29:0/2526298855 shutdown_connections 2023-10-14T18:18:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40acc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 --2- 172.21.15.29:0/2526298855 >> v2:172.21.15.186:3300/0 conn(0x7f7dd406fa20 0x7f7dd40ac750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 --2- 172.21.15.29:0/2526298855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd406d520 0x7f7dd40b1de0 unknown :-1 s=CLOSED pgs=2806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 -- 172.21.15.29:0/2526298855 >> 172.21.15.29:0/2526298855 conn(0x7f7dd40b0a80 msgr2=0x7f7dd40b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:19.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 -- 172.21.15.29:0/2526298855 shutdown_connections 2023-10-14T18:18:19.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.519+0000 7f7dd901a640 1 -- 172.21.15.29:0/2526298855 wait complete. 2023-10-14T18:18:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.521+0000 7f7dd901a640 1 Processor -- start 2023-10-14T18:18:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.521+0000 7f7dd901a640 1 -- start start 2023-10-14T18:18:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.522+0000 7f7dd901a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd901a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd406d520 0x7f7dd40c8600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd901a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd901a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4154fd0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd901a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd4155ca0 con 0x7f7dd406d520 2023-10-14T18:18:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd901a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd40c9bc0 con 0x7f7dd4065cc0 2023-10-14T18:18:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42924/0 (socket says 172.21.15.29:42924) 2023-10-14T18:18:19.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 -- 172.21.15.29:0/437084606 learned_addr learned my addr 172.21.15.29:0/437084606 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dcbfff640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd406d520 0x7f7dd40c8600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 -- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 msgr2=0x7f7dd40c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 -- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd406d520 msgr2=0x7f7dd40c8600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd406d520 0x7f7dd40c8600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.523+0000 7f7dd8819640 1 -- 172.21.15.29:0/437084606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc40094d0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd8819640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc007e60 tx=0x7f7dcc00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc014620 con 0x7f7dd4157ff0 2023-10-14T18:18:19.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dcc0147a0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7dcc014b50 con 0x7f7dd4157ff0 2023-10-14T18:18:19.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd40c9ea0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.524+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd40ca460 con 0x7f7dd4157ff0 2023-10-14T18:18:19.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.525+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4155ae0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.527+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7dcc014cb0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.528+0000 7f7dd17fa640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 0x7f7db406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.528+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7f7dcc08a6c0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.528+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 0x7f7db406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.529+0000 7f7dd3fff640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 0x7f7db406d5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4035000 tx=0x7f7dc4008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.532+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7dd4155ae0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.735 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.734+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dd4155ae0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.736+0000 7f7dd17fa640 1 -- 172.21.15.29:0/437084606 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v238) v1 ==== 74+0+18081 (secure 0 0 0) 0x7f7dd4155ae0 con 0x7f7dd4157ff0 2023-10-14T18:18:19.738 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:19.738 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":238,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:15.443657+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":28,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:17:40.115972+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"232","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":29,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:17:40.810362+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"233","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.6299999952316284,"score_stable":1.6299999952316284,"optimal_score":1,"raw_score_acting":1.6299999952316284,"raw_score_stable":1.6299999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6836/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6837/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 msgr2=0x7f7db406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 0x7f7db406d5a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7dc4035000 tx=0x7f7dc4008030 comp rx=0 tx=0).stop 2023-10-14T18:18:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 msgr2=0x7f7dd406b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 secure :-1 s=READY pgs=2807 cs=0 l=1 rev1=1 crypto rx=0x7f7dcc007e60 tx=0x7f7dcc00e4b0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 shutdown_connections 2023-10-14T18:18:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f7db406b0e0 0x7f7db406d5a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7dd4157ff0 0x7f7dd406b5a0 unknown :-1 s=CLOSED pgs=2807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7dd406d520 0x7f7dd40c8600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 --2- 172.21.15.29:0/437084606 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7dd4065cc0 0x7f7dd40c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 >> 172.21.15.29:0/437084606 conn(0x7f7dd40b0a80 msgr2=0x7f7dd415d3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:19.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 shutdown_connections 2023-10-14T18:18:19.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.739+0000 7f7dd901a640 1 -- 172.21.15.29:0/437084606 wait complete. 2023-10-14T18:18:19.760 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-14T18:18:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.988+0000 7ff06dd0c640 1 Processor -- start 2023-10-14T18:18:19.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.988+0000 7ff06dd0c640 1 -- start start 2023-10-14T18:18:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff06dd0c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0680ad350 0x7ff068069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff06dd0c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff06806a730 con 0x7ff06806a2e0 2023-10-14T18:18:19.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff06dd0c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0680ac890 con 0x7ff06815d150 2023-10-14T18:18:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff06dd0c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff06815d530 con 0x7ff0680ad350 2023-10-14T18:18:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff066ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff0680ad350 0x7ff068069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff067fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff06806a2e0 0x7ff0680c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:59872/0 (socket says 172.21.15.29:59872) 2023-10-14T18:18:19.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.990+0000 7ff067fff640 1 -- 172.21.15.29:0/2232652635 learned_addr learned my addr 172.21.15.29:0/2232652635 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.991+0000 7ff066ffd640 1 -- 172.21.15.29:0/2232652635 >> v1:172.21.15.186:6789/0 conn(0x7ff06815d150 legacy=0x7ff068160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:19.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.991+0000 7ff066ffd640 1 -- 172.21.15.29:0/2232652635 >> v1:172.21.15.29:6789/0 conn(0x7ff06806a2e0 legacy=0x7ff0680c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.991+0000 7ff066ffd640 1 -- 172.21.15.29:0/2232652635 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0680ad7a0 con 0x7ff0680ad350 2023-10-14T18:18:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.991+0000 7ff0667fc640 1 -- 172.21.15.29:0/2232652635 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377593173 0 0) 0x7ff060001490 con 0x7ff06806a2e0 2023-10-14T18:18:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.992+0000 7ff066ffd640 1 --2- 172.21.15.29:0/2232652635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0680ad350 0x7ff068069da0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0097a0 tx=0x7ff05c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=beeefaf2fb81b448 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.992+0000 7ff0667fc640 1 -- 172.21.15.29:0/2232652635 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c031830 con 0x7ff0680ad350 2023-10-14T18:18:19.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.992+0000 7ff0667fc640 1 -- 172.21.15.29:0/2232652635 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff05c031990 con 0x7ff0680ad350 2023-10-14T18:18:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff0667fc640 1 -- 172.21.15.29:0/2232652635 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff05c031d20 con 0x7ff0680ad350 2023-10-14T18:18:19.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2232652635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0680ad350 msgr2=0x7ff068069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2232652635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0680ad350 0x7ff068069da0 secure :-1 s=READY pgs=2809 cs=0 l=1 rev1=1 crypto rx=0x7ff05c0097a0 tx=0x7ff05c02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2232652635 shutdown_connections 2023-10-14T18:18:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2232652635 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff0680ad350 0x7ff068069da0 unknown :-1 s=CLOSED pgs=2809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.993+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2232652635 >> 172.21.15.29:0/2232652635 conn(0x7ff0680ce9b0 msgr2=0x7ff0680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:19.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.994+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2232652635 shutdown_connections 2023-10-14T18:18:19.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.994+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2232652635 wait complete. 2023-10-14T18:18:19.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.995+0000 7ff06dd0c640 1 Processor -- start 2023-10-14T18:18:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.996+0000 7ff06dd0c640 1 -- start start 2023-10-14T18:18:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 0x7ff0681562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff06815d150 0x7ff0680aecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff068156820 con 0x7ff06806a2e0 2023-10-14T18:18:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680b0220 con 0x7ff0680ad350 2023-10-14T18:18:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff06dd0c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0680af610 con 0x7ff06815d150 2023-10-14T18:18:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff0677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff0677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42940/0 (socket says 172.21.15.29:42940) 2023-10-14T18:18:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.997+0000 7ff0677fe640 1 -- 172.21.15.29:0/2234006650 learned_addr learned my addr 172.21.15.29:0/2234006650 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff066ffd640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 0x7ff0681562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 -- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff06815d150 msgr2=0x7ff0680aecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff067fff640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff06815d150 0x7ff0680aecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff06815d150 0x7ff0680aecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 -- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 msgr2=0x7ff0681562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 0x7ff0681562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 -- 172.21.15.29:0/2234006650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff05c009400 con 0x7ff06806a2e0 2023-10-14T18:18:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff066ffd640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 0x7ff0681562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:20.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff0677fe640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7ff05800b3f0 tx=0x7ff05800b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058011830 con 0x7ff06806a2e0 2023-10-14T18:18:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff058011990 con 0x7ff06806a2e0 2023-10-14T18:18:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff058011d40 con 0x7ff06806a2e0 2023-10-14T18:18:20.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.998+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0680af8f0 con 0x7ff06806a2e0 2023-10-14T18:18:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:19.999+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0680afeb0 con 0x7ff06806a2e0 2023-10-14T18:18:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.000+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff06804e160 con 0x7ff06806a2e0 2023-10-14T18:18:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.001+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff05801a400 con 0x7ff06806a2e0 2023-10-14T18:18:20.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.002+0000 7ff04ffff640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 0x7ff05406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.002+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(238..238 src has 1..238) v4 ==== 6740+0+0 (secure 0 0 0) 0x7ff058089ca0 con 0x7ff06806a2e0 2023-10-14T18:18:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.002+0000 7ff066ffd640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 0x7ff05406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.003+0000 7ff066ffd640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 0x7ff05406d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff05c036000 tx=0x7ff05c008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:20.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.006+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff058054540 con 0x7ff06806a2e0 2023-10-14T18:18:20.245 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.243+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7ff0680c5510 con 0x7ff06806a2e0 2023-10-14T18:18:20.245 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:18:20.244+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:18:20.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.247+0000 7ff04ffff640 1 -- 172.21.15.29:0/2234006650 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v116) v1 ==== 167+0+0 (secure 0 0 0) 0x7ff058015090 con 0x7ff06806a2e0 2023-10-14T18:18:20.250 INFO:teuthology.orchestra.run.smithi029.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:18:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 msgr2=0x7ff05406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 0x7ff05406d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff05c036000 tx=0x7ff05c008030 comp rx=0 tx=0).stop 2023-10-14T18:18:20.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 msgr2=0x7ff068158240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 secure :-1 s=READY pgs=2810 cs=0 l=1 rev1=1 crypto rx=0x7ff05800b3f0 tx=0x7ff05800b8b0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 shutdown_connections 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff05406b0e0 0x7ff05406d5a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff06815d150 0x7ff0680aecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff0680ad350 0x7ff0681562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 --2- 172.21.15.29:0/2234006650 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff06806a2e0 0x7ff068158240 unknown :-1 s=CLOSED pgs=2810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 >> 172.21.15.29:0/2234006650 conn(0x7ff0680ce9b0 msgr2=0x7ff06806f770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 shutdown_connections 2023-10-14T18:18:20.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.252+0000 7ff06dd0c640 1 -- 172.21.15.29:0/2234006650 wait complete. 2023-10-14T18:18:20.269 INFO:tasks.ceph.mds.c.smithi029.stderr: -1992> 2023-10-14T18:17:46.680+0000 7f87cc129640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:18:20.274 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-14T18:18:20.346 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:18:20.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.473+0000 7f78d53c9640 1 Processor -- start 2023-10-14T18:18:20.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.474+0000 7f78d53c9640 1 -- start start 2023-10-14T18:18:20.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.475+0000 7f78d53c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f78d015ad70 0x7f78d0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.475+0000 7f78d53c9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78d00c9ca0 0x7f78d006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.475+0000 7f78d53c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78d00cb030 con 0x7f78d00ca3b0 2023-10-14T18:18:20.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.475+0000 7f78d53c9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00c91c0 con 0x7f78d00c9ca0 2023-10-14T18:18:20.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.475+0000 7f78d53c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78d00c94a0 con 0x7f78d015ad70 2023-10-14T18:18:20.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78d00c9ca0 0x7f78d006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78d00c9ca0 0x7f78d006a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42966/0 (socket says 172.21.15.29:42966) 2023-10-14T18:18:20.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 -- 172.21.15.29:0/4088788076 learned_addr learned my addr 172.21.15.29:0/4088788076 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78ceffd640 1 --2- 172.21.15.29:0/4088788076 >> v2:172.21.15.186:3300/0 conn(0x7f78d015ad70 0x7f78d0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 -- 172.21.15.29:0/4088788076 >> v2:172.21.15.186:3300/0 conn(0x7f78d015ad70 msgr2=0x7f78d0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 --2- 172.21.15.29:0/4088788076 >> v2:172.21.15.186:3300/0 conn(0x7f78d015ad70 0x7f78d0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 -- 172.21.15.29:0/4088788076 >> v1:172.21.15.186:6789/0 conn(0x7f78d00ca3b0 legacy=0x7f78d00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:20.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.476+0000 7f78c7fff640 1 -- 172.21.15.29:0/4088788076 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78d00c9780 con 0x7f78d00c9ca0 2023-10-14T18:18:20.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.477+0000 7f78c7fff640 1 --2- 172.21.15.29:0/4088788076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00c9ca0 0x7f78d006a2e0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f78d00cf780 tx=0x7f78b802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38242f753ebf955a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:20.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78ce7fc640 1 -- 172.21.15.29:0/4088788076 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8031870 con 0x7f78d00c9ca0 2023-10-14T18:18:20.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78ce7fc640 1 -- 172.21.15.29:0/4088788076 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b80319d0 con 0x7f78d00c9ca0 2023-10-14T18:18:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78ce7fc640 1 -- 172.21.15.29:0/4088788076 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8031d60 con 0x7f78d00c9ca0 2023-10-14T18:18:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 -- 172.21.15.29:0/4088788076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00c9ca0 msgr2=0x7f78d006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 --2- 172.21.15.29:0/4088788076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00c9ca0 0x7f78d006a2e0 secure :-1 s=READY pgs=2814 cs=0 l=1 rev1=1 crypto rx=0x7f78d00cf780 tx=0x7f78b802b910 comp rx=0 tx=0).stop 2023-10-14T18:18:20.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 -- 172.21.15.29:0/4088788076 shutdown_connections 2023-10-14T18:18:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 --2- 172.21.15.29:0/4088788076 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00c9ca0 0x7f78d006a2e0 unknown :-1 s=CLOSED pgs=2814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 --2- 172.21.15.29:0/4088788076 >> v2:172.21.15.186:3300/0 conn(0x7f78d015ad70 0x7f78d0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 -- 172.21.15.29:0/4088788076 >> 172.21.15.29:0/4088788076 conn(0x7f78d00ce9b0 msgr2=0x7f78d00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:20.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.478+0000 7f78d53c9640 1 -- 172.21.15.29:0/4088788076 shutdown_connections 2023-10-14T18:18:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.479+0000 7f78d53c9640 1 -- 172.21.15.29:0/4088788076 wait complete. 2023-10-14T18:18:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.480+0000 7f78d53c9640 1 Processor -- start 2023-10-14T18:18:20.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.480+0000 7f78d53c9640 1 -- start start 2023-10-14T18:18:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 0x7f78d00aec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78d015ad70 0x7f78d00ac910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00c7f80 con 0x7f78d00ca3b0 2023-10-14T18:18:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00af270 con 0x7f78d00c9ca0 2023-10-14T18:18:20.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.481+0000 7f78d53c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78d00c75e0 con 0x7f78d015ad70 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:42974/0 (socket says 172.21.15.29:42974) 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 -- 172.21.15.29:0/474876834 learned_addr learned my addr 172.21.15.29:0/474876834 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 -- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78d015ad70 msgr2=0x7f78d00ac910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78ceffd640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 0x7f78d00aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78cf7fe640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78d015ad70 0x7f78d00ac910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78d015ad70 0x7f78d00ac910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 -- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 msgr2=0x7f78d00aec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 0x7f78d00aec90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78c7fff640 1 -- 172.21.15.29:0/474876834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b80094d0 con 0x7f78d00ca3b0 2023-10-14T18:18:20.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.482+0000 7f78ceffd640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 0x7f78d00aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78c7fff640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f78b8000c00 tx=0x7f78b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b803abe0 con 0x7f78d00ca3b0 2023-10-14T18:18:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78b803ad40 con 0x7f78d00ca3b0 2023-10-14T18:18:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78d00c78c0 con 0x7f78d00ca3b0 2023-10-14T18:18:20.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78b8044610 con 0x7f78d00ca3b0 2023-10-14T18:18:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.483+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78d00ae870 con 0x7f78d00ca3b0 2023-10-14T18:18:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.484+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78d00c80c0 con 0x7f78d00ca3b0 2023-10-14T18:18:20.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.485+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f78b804d450 con 0x7f78d00ca3b0 2023-10-14T18:18:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.487+0000 7f78c77fe640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 0x7f78b006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.487+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6826+0+0 (secure 0 0 0) 0x7f78b802d040 con 0x7f78d00ca3b0 2023-10-14T18:18:20.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.487+0000 7f78ceffd640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 0x7f78b006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.488+0000 7f78ceffd640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 0x7f78b006d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:20.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.491+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78d00c80c0 con 0x7f78d00ca3b0 2023-10-14T18:18:20.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:20.726+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f78d00c80c0 con 0x7f78d00ca3b0 2023-10-14T18:18:21.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.251+0000 7f78c77fe640 1 -- 172.21.15.29:0/474876834 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v117) v1 ==== 107+0+0 (secure 0 0 0) 0x7f78d00c80c0 con 0x7f78d00ca3b0 2023-10-14T18:18:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.256+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 msgr2=0x7f78b006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.256+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 0x7f78b006d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f78c0006fd0 tx=0x7f78c0008040 comp rx=0 tx=0).stop 2023-10-14T18:18:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.256+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 msgr2=0x7f78d00ae330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:21.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.256+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 secure :-1 s=READY pgs=2815 cs=0 l=1 rev1=1 crypto rx=0x7f78b8000c00 tx=0x7f78b80046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 shutdown_connections 2023-10-14T18:18:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f78b006b0b0 0x7f78b006d570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78d015ad70 0x7f78d00ac910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78d00ca3b0 0x7f78d00ae330 unknown :-1 s=CLOSED pgs=2815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 --2- 172.21.15.29:0/474876834 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f78d00c9ca0 0x7f78d00aec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 >> 172.21.15.29:0/474876834 conn(0x7f78d00ce9b0 msgr2=0x7f78d00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 shutdown_connections 2023-10-14T18:18:21.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.257+0000 7f78d53c9640 1 -- 172.21.15.29:0/474876834 wait complete. 2023-10-14T18:18:21.281 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-14T18:18:21.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.503+0000 7f0a91675640 1 Processor -- start 2023-10-14T18:18:21.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.503+0000 7f0a91675640 1 -- start start 2023-10-14T18:18:21.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c15ac10 0x7f0a8c06c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0a8c0b0360 0x7f0a8c0b07c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a8c0ca510 0x7f0a8c0ca940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c0b0d00 con 0x7f0a8c0ca510 2023-10-14T18:18:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a8c15b310 con 0x7f0a8c0b0360 2023-10-14T18:18:21.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.504+0000 7f0a91675640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c06f640 con 0x7f0a8c15ac10 2023-10-14T18:18:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a8c0ca510 0x7f0a8c0ca940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0a8c0ca510 0x7f0a8c0ca940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49158/0 (socket says 172.21.15.29:49158) 2023-10-14T18:18:21.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 -- 172.21.15.29:0/2472797253 learned_addr learned my addr 172.21.15.29:0/2472797253 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8affd640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c15ac10 0x7f0a8c06c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8a7fc640 1 --2- 172.21.15.29:0/2472797253 >> v2:172.21.15.186:3300/0 conn(0x7f0a8c0b0360 0x7f0a8c0b07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 -- 172.21.15.29:0/2472797253 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c15ac10 msgr2=0x7f0a8c06c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:21.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c15ac10 0x7f0a8c06c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 -- 172.21.15.29:0/2472797253 >> v2:172.21.15.186:3300/0 conn(0x7f0a8c0b0360 msgr2=0x7f0a8c0b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 --2- 172.21.15.29:0/2472797253 >> v2:172.21.15.186:3300/0 conn(0x7f0a8c0b0360 0x7f0a8c0b07c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.505+0000 7f0a8b7fe640 1 -- 172.21.15.29:0/2472797253 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c15b5b0 con 0x7f0a8c0ca510 2023-10-14T18:18:21.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.507+0000 7f0a8b7fe640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c0ca940 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c06d5c0 tx=0x7f0a8402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b77ce7cff4bd3b87 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.507+0000 7f0a89ffb640 1 -- 172.21.15.29:0/2472797253 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a8402c9e0 con 0x7f0a8c0ca510 2023-10-14T18:18:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.507+0000 7f0a89ffb640 1 -- 172.21.15.29:0/2472797253 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a8402cb40 con 0x7f0a8c0ca510 2023-10-14T18:18:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.507+0000 7f0a89ffb640 1 -- 172.21.15.29:0/2472797253 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a840343f0 con 0x7f0a8c0ca510 2023-10-14T18:18:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 -- 172.21.15.29:0/2472797253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 msgr2=0x7f0a8c0ca940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:21.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c0ca940 secure :-1 s=READY pgs=2816 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c06d5c0 tx=0x7f0a8402be40 comp rx=0 tx=0).stop 2023-10-14T18:18:21.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 -- 172.21.15.29:0/2472797253 shutdown_connections 2023-10-14T18:18:21.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c0ca940 unknown :-1 s=CLOSED pgs=2816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 --2- 172.21.15.29:0/2472797253 >> v2:172.21.15.186:3300/0 conn(0x7f0a8c0b0360 0x7f0a8c0b07c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 --2- 172.21.15.29:0/2472797253 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c15ac10 0x7f0a8c06c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 -- 172.21.15.29:0/2472797253 >> 172.21.15.29:0/2472797253 conn(0x7f0a8c0cb360 msgr2=0x7f0a8c0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:21.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 -- 172.21.15.29:0/2472797253 shutdown_connections 2023-10-14T18:18:21.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.508+0000 7f0a91675640 1 -- 172.21.15.29:0/2472797253 wait complete. 2023-10-14T18:18:21.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.510+0000 7f0a91675640 1 Processor -- start 2023-10-14T18:18:21.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.510+0000 7f0a91675640 1 -- start start 2023-10-14T18:18:21.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c0b0360 0x7f0a8c0cd770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 0x7f0a8c148ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c0b2610 con 0x7f0a8c0ca510 2023-10-14T18:18:21.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c149400 con 0x7f0a8c15ac10 2023-10-14T18:18:21.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a91675640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c0b2330 con 0x7f0a8c0b0360 2023-10-14T18:18:21.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49162/0 (socket says 172.21.15.29:49162) 2023-10-14T18:18:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 -- 172.21.15.29:0/2352947163 learned_addr learned my addr 172.21.15.29:0/2352947163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8affd640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c0b0360 0x7f0a8c0cd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 -- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c0b0360 msgr2=0x7f0a8c0cd770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:21.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c0b0360 0x7f0a8c0cd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8a7fc640 1 -- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 msgr2=0x7f0a8c148ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.511+0000 7f0a8b7fe640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 0x7f0a8c148ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a8a7fc640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 0x7f0a8c148ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:21.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a8a7fc640 1 -- 172.21.15.29:0/2352947163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a84009b30 con 0x7f0a8c0ca510 2023-10-14T18:18:21.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a8b7fe640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 0x7f0a8c148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:21.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a8a7fc640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000d800 tx=0x7f0a8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:21.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a800096a0 con 0x7f0a8c0ca510 2023-10-14T18:18:21.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a80009800 con 0x7f0a8c0ca510 2023-10-14T18:18:21.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80009990 con 0x7f0a8c0ca510 2023-10-14T18:18:21.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c0c5b50 con 0x7f0a8c0ca510 2023-10-14T18:18:21.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.512+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c0c6110 con 0x7f0a8c0ca510 2023-10-14T18:18:21.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.514+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c0c5990 con 0x7f0a8c0ca510 2023-10-14T18:18:21.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.514+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0a80009af0 con 0x7f0a8c0ca510 2023-10-14T18:18:21.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.515+0000 7f0a737fe640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 0x7f0a7806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:21.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.516+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(239..239 src has 1..239) v4 ==== 6826+0+0 (secure 0 0 0) 0x7f0a8008a260 con 0x7f0a8c0ca510 2023-10-14T18:18:21.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.516+0000 7f0a8affd640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 0x7f0a7806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:21.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.516+0000 7f0a8affd640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 0x7f0a7806d570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:21.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.520+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0a80054c10 con 0x7f0a8c0ca510 2023-10-14T18:18:21.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:21.718+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0a8c04e160 con 0x7f0a8c0ca510 2023-10-14T18:18:22.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.499+0000 7f0a737fe640 1 -- 172.21.15.29:0/2352947163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v240) v1 ==== 190+0+0 (secure 0 0 0) 0x7f0a80058230 con 0x7f0a8c0ca510 2023-10-14T18:18:22.501 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' removed 2023-10-14T18:18:22.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 msgr2=0x7f0a7806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 0x7f0a7806d570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c006fd0 tx=0x7f0a7c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:22.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 msgr2=0x7f0a8c148980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 secure :-1 s=READY pgs=2817 cs=0 l=1 rev1=1 crypto rx=0x7f0a8000d800 tx=0x7f0a8000dcc0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 shutdown_connections 2023-10-14T18:18:22.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f0a7806b0b0 0x7f0a7806d570 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0a8c15ac10 0x7f0a8c148ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.503+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0a8c0ca510 0x7f0a8c148980 unknown :-1 s=CLOSED pgs=2817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.504+0000 7f0a91675640 1 --2- 172.21.15.29:0/2352947163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0a8c0b0360 0x7f0a8c0cd770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.504+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 >> 172.21.15.29:0/2352947163 conn(0x7f0a8c0cb360 msgr2=0x7f0a8c0c9ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.504+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 shutdown_connections 2023-10-14T18:18:22.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.504+0000 7f0a91675640 1 -- 172.21.15.29:0/2352947163 wait complete. 2023-10-14T18:18:22.529 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-14T18:18:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.757+0000 7f6dbf052640 1 Processor -- start 2023-10-14T18:18:22.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.757+0000 7f6dbf052640 1 -- start start 2023-10-14T18:18:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbf052640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6db815b320 0x7f6db80cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbf052640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db815b9d0 con 0x7f6db80ad970 2023-10-14T18:18:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbf052640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db815bcb0 con 0x7f6db806c040 2023-10-14T18:18:22.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbf052640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db815b7d0 con 0x7f6db815b320 2023-10-14T18:18:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbd84f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6db80ad970 0x7f6db80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38714/0 (socket says 172.21.15.29:38714) 2023-10-14T18:18:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbd84f640 1 -- 172.21.15.29:0/2943427822 learned_addr learned my addr 172.21.15.29:0/2943427822 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:22.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.759+0000 7f6dbe851640 1 --2- 172.21.15.29:0/2943427822 >> v2:172.21.15.29:3300/0 conn(0x7f6db815b320 0x7f6db80cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:22.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.760+0000 7f6dbd04e640 1 -- 172.21.15.29:0/2943427822 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3236695056 0 0) 0x7f6db815b9d0 con 0x7f6db80ad970 2023-10-14T18:18:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.760+0000 7f6dbd04e640 1 -- 172.21.15.29:0/2943427822 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da4003520 con 0x7f6db80ad970 2023-10-14T18:18:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.760+0000 7f6dbe851640 1 -- 172.21.15.29:0/2943427822 >> v1:172.21.15.186:6789/0 conn(0x7f6db806c040 legacy=0x7f6db806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.760+0000 7f6dbe851640 1 -- 172.21.15.29:0/2943427822 >> v1:172.21.15.29:6789/0 conn(0x7f6db80ad970 legacy=0x7f6db80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.760+0000 7f6dbe851640 1 -- 172.21.15.29:0/2943427822 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db815bf90 con 0x7f6db815b320 2023-10-14T18:18:22.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.761+0000 7f6dbe851640 1 --2- 172.21.15.29:0/2943427822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80cd180 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009dc0 tx=0x7f6db002be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=264b234284c3e34d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:22.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.761+0000 7f6dbd04e640 1 -- 172.21.15.29:0/2943427822 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db002cc40 con 0x7f6db815b320 2023-10-14T18:18:22.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.761+0000 7f6dbd04e640 1 -- 172.21.15.29:0/2943427822 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db002cda0 con 0x7f6db815b320 2023-10-14T18:18:22.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.761+0000 7f6dbd04e640 1 -- 172.21.15.29:0/2943427822 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db0034660 con 0x7f6db815b320 2023-10-14T18:18:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 -- 172.21.15.29:0/2943427822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 msgr2=0x7f6db80cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 --2- 172.21.15.29:0/2943427822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80cd180 secure :-1 s=READY pgs=2819 cs=0 l=1 rev1=1 crypto rx=0x7f6db0009dc0 tx=0x7f6db002be90 comp rx=0 tx=0).stop 2023-10-14T18:18:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 -- 172.21.15.29:0/2943427822 shutdown_connections 2023-10-14T18:18:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 --2- 172.21.15.29:0/2943427822 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80cd180 unknown :-1 s=CLOSED pgs=2819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 -- 172.21.15.29:0/2943427822 >> 172.21.15.29:0/2943427822 conn(0x7f6db80ce6a0 msgr2=0x7f6db80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 -- 172.21.15.29:0/2943427822 shutdown_connections 2023-10-14T18:18:22.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.762+0000 7f6dbf052640 1 -- 172.21.15.29:0/2943427822 wait complete. 2023-10-14T18:18:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.764+0000 7f6dbf052640 1 Processor -- start 2023-10-14T18:18:22.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.764+0000 7f6dbf052640 1 -- start start 2023-10-14T18:18:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db806c040 0x7f6db80b1c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 0x7f6db815ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:22.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80b45b0 con 0x7f6db815b320 2023-10-14T18:18:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80aeea0 con 0x7f6db80ad970 2023-10-14T18:18:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.765+0000 7f6dbf052640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80af180 con 0x7f6db806c040 2023-10-14T18:18:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:22.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49174/0 (socket says 172.21.15.29:49174) 2023-10-14T18:18:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 -- 172.21.15.29:0/3817610937 learned_addr learned my addr 172.21.15.29:0/3817610937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbd84f640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 0x7f6db815ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 -- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db806c040 msgr2=0x7f6db80b1c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe050640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db806c040 0x7f6db80b1c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db806c040 0x7f6db80b1c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 -- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 msgr2=0x7f6db815ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 0x7f6db815ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbe851640 1 -- 172.21.15.29:0/3817610937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db0009a70 con 0x7f6db815b320 2023-10-14T18:18:22.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.766+0000 7f6dbd84f640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 0x7f6db815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6dbe851640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f6db0037040 tx=0x7f6db0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db003e610 con 0x7f6db815b320 2023-10-14T18:18:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db003e770 con 0x7f6db815b320 2023-10-14T18:18:22.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6db003eb00 con 0x7f6db815b320 2023-10-14T18:18:22.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db80af460 con 0x7f6db815b320 2023-10-14T18:18:22.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.767+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db80afa20 con 0x7f6db815b320 2023-10-14T18:18:22.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.768+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db804e9b0 con 0x7f6db815b320 2023-10-14T18:18:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.769+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6db003b050 con 0x7f6db815b320 2023-10-14T18:18:22.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.770+0000 7f6daeffd640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 0x7f6d9c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.771+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(240..240 src has 1..240) v4 ==== 6382+0+0 (secure 0 0 0) 0x7f6db00ac610 con 0x7f6db815b320 2023-10-14T18:18:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.771+0000 7f6dbe050640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 0x7f6d9c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.771+0000 7f6dbe050640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 0x7f6d9c06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006fd0 tx=0x7f6db4008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:22.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.775+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6db00771d0 con 0x7f6db815b320 2023-10-14T18:18:22.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:22.974+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f6db806b2b0 con 0x7f6db815b320 2023-10-14T18:18:23.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.518+0000 7f6daeffd640 1 -- 172.21.15.29:0/3817610937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v241) v1 ==== 178+0+0 (secure 0 0 0) 0x7f6db0009450 con 0x7f6db815b320 2023-10-14T18:18:23.520 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' removed 2023-10-14T18:18:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 msgr2=0x7f6d9c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 0x7f6d9c06d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f6db4006fd0 tx=0x7f6db4008040 comp rx=0 tx=0).stop 2023-10-14T18:18:23.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 msgr2=0x7f6db80c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 secure :-1 s=READY pgs=2820 cs=0 l=1 rev1=1 crypto rx=0x7f6db0037040 tx=0x7f6db0004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 shutdown_connections 2023-10-14T18:18:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f6d9c06b0b0 0x7f6d9c06d570 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6db815b320 0x7f6db80c8cf0 unknown :-1 s=CLOSED pgs=2820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6db80ad970 0x7f6db815ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 --2- 172.21.15.29:0/3817610937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6db806c040 0x7f6db80b1c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 >> 172.21.15.29:0/3817610937 conn(0x7f6db80ce6a0 msgr2=0x7f6db815da50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.522+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 shutdown_connections 2023-10-14T18:18:23.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.523+0000 7f6dbf052640 1 -- 172.21.15.29:0/3817610937 wait complete. 2023-10-14T18:18:23.544 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T18:18:23.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.775+0000 7fe490bd5640 1 Processor -- start 2023-10-14T18:18:23.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.775+0000 7fe490bd5640 1 -- start start 2023-10-14T18:18:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.777+0000 7fe490bd5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe48c0b0b20 0x7fe48c164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:23.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.777+0000 7fe490bd5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c06f890 con 0x7fe48c159f00 2023-10-14T18:18:23.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.777+0000 7fe490bd5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe48c06c650 con 0x7fe48c159b20 2023-10-14T18:18:23.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.777+0000 7fe490bd5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48c070d30 con 0x7fe48c0b0b20 2023-10-14T18:18:23.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe483fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe48c159f00 0x7fe48c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38724/0 (socket says 172.21.15.29:38724) 2023-10-14T18:18:23.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe483fff640 1 -- 172.21.15.29:0/1533692929 learned_addr learned my addr 172.21.15.29:0/1533692929 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:23.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/1533692929 >> v2:172.21.15.29:3300/0 conn(0x7fe48c0b0b20 0x7fe48c164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:23.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe489d9b640 1 -- 172.21.15.29:0/1533692929 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276913200 0 0) 0x7fe48c06f890 con 0x7fe48c159f00 2023-10-14T18:18:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe489d9b640 1 -- 172.21.15.29:0/1533692929 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe478003520 con 0x7fe48c159f00 2023-10-14T18:18:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe48ad9d640 1 -- 172.21.15.29:0/1533692929 >> v1:172.21.15.186:6789/0 conn(0x7fe48c159b20 legacy=0x7fe48c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe48ad9d640 1 -- 172.21.15.29:0/1533692929 >> v1:172.21.15.29:6789/0 conn(0x7fe48c159f00 legacy=0x7fe48c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.778+0000 7fe48ad9d640 1 -- 172.21.15.29:0/1533692929 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48c06c930 con 0x7fe48c0b0b20 2023-10-14T18:18:23.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.779+0000 7fe48ad9d640 1 --2- 172.21.15.29:0/1533692929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c164b40 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fe484009d90 tx=0x7fe48402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e985fabd10319f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.780+0000 7fe489d9b640 1 -- 172.21.15.29:0/1533692929 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe48402ccf0 con 0x7fe48c0b0b20 2023-10-14T18:18:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.780+0000 7fe489d9b640 1 -- 172.21.15.29:0/1533692929 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe48402ce50 con 0x7fe48c0b0b20 2023-10-14T18:18:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.780+0000 7fe489d9b640 1 -- 172.21.15.29:0/1533692929 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe484034640 con 0x7fe48c0b0b20 2023-10-14T18:18:23.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.780+0000 7fe490bd5640 1 -- 172.21.15.29:0/1533692929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 msgr2=0x7fe48c164b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.780+0000 7fe490bd5640 1 --2- 172.21.15.29:0/1533692929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c164b40 secure :-1 s=READY pgs=2822 cs=0 l=1 rev1=1 crypto rx=0x7fe484009d90 tx=0x7fe48402bf40 comp rx=0 tx=0).stop 2023-10-14T18:18:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.781+0000 7fe490bd5640 1 -- 172.21.15.29:0/1533692929 shutdown_connections 2023-10-14T18:18:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.781+0000 7fe490bd5640 1 --2- 172.21.15.29:0/1533692929 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c164b40 unknown :-1 s=CLOSED pgs=2822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.781+0000 7fe490bd5640 1 -- 172.21.15.29:0/1533692929 >> 172.21.15.29:0/1533692929 conn(0x7fe48c0cb410 msgr2=0x7fe48c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:23.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.781+0000 7fe490bd5640 1 -- 172.21.15.29:0/1533692929 shutdown_connections 2023-10-14T18:18:23.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.781+0000 7fe490bd5640 1 -- 172.21.15.29:0/1533692929 wait complete. 2023-10-14T18:18:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.783+0000 7fe490bd5640 1 Processor -- start 2023-10-14T18:18:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.783+0000 7fe490bd5640 1 -- start start 2023-10-14T18:18:23.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c159b20 0x7fe48c0c49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c159f00 0x7fe48c0c4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c89c0 con 0x7fe48c0b0b20 2023-10-14T18:18:23.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0c7b30 con 0x7fe48c159f00 2023-10-14T18:18:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe490bd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48c0b3ca0 con 0x7fe48c159b20 2023-10-14T18:18:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49196/0 (socket says 172.21.15.29:49196) 2023-10-14T18:18:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 -- 172.21.15.29:0/2023831607 learned_addr learned my addr 172.21.15.29:0/2023831607 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:23.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe483fff640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c159b20 0x7fe48c0c49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 -- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c159b20 msgr2=0x7fe48c0c49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c159b20 0x7fe48c0c49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 -- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c159f00 msgr2=0x7fe48c0c4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c159f00 0x7fe48c0c4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:23.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.785+0000 7fe48a59c640 1 -- 172.21.15.29:0/2023831607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484009a70 con 0x7fe48c0b0b20 2023-10-14T18:18:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe48a59c640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c450 tx=0x7fe47c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c0052d0 con 0x7fe48c0b0b20 2023-10-14T18:18:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47c005430 con 0x7fe48c0b0b20 2023-10-14T18:18:23.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c012870 con 0x7fe48c0b0b20 2023-10-14T18:18:23.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48c0c7d40 con 0x7fe48c0b0b20 2023-10-14T18:18:23.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.786+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48c0c8300 con 0x7fe48c0b0b20 2023-10-14T18:18:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.788+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48c0c5440 con 0x7fe48c0b0b20 2023-10-14T18:18:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.789+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe47c0129d0 con 0x7fe48c0b0b20 2023-10-14T18:18:23.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.790+0000 7fe482ffd640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 0x7fe46c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.790+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fe47c0897a0 con 0x7fe48c0b0b20 2023-10-14T18:18:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.790+0000 7fe483fff640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 0x7fe46c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:23.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.791+0000 7fe483fff640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 0x7fe46c06d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0c4780 tx=0x7fe474009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:23.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:23.794+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe47c0542c0 con 0x7fe48c0b0b20 2023-10-14T18:18:24.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.001+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7fe48c04e160 con 0x7fe48c0b0b20 2023-10-14T18:18:24.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.006+0000 7fe482ffd640 1 -- 172.21.15.29:0/2023831607 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v79) v1 ==== 115+0+0 (secure 0 0 0) 0x7fe47c0578e0 con 0x7fe48c0b0b20 2023-10-14T18:18:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 msgr2=0x7fe46c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 0x7fe46c06d570 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0c4780 tx=0x7fe474009040 comp rx=0 tx=0).stop 2023-10-14T18:18:24.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 msgr2=0x7fe48c0b3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 secure :-1 s=READY pgs=2823 cs=0 l=1 rev1=1 crypto rx=0x7fe47c00c450 tx=0x7fe47c00c910 comp rx=0 tx=0).stop 2023-10-14T18:18:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 shutdown_connections 2023-10-14T18:18:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe46c06b0b0 0x7fe46c06d570 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe48c159f00 0x7fe48c0c4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe48c159b20 0x7fe48c0c49c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 --2- 172.21.15.29:0/2023831607 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe48c0b0b20 0x7fe48c0b3760 unknown :-1 s=CLOSED pgs=2823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 >> 172.21.15.29:0/2023831607 conn(0x7fe48c0cb410 msgr2=0x7fe48c0ce870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 shutdown_connections 2023-10-14T18:18:24.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.010+0000 7fe490bd5640 1 -- 172.21.15.29:0/2023831607 wait complete. 2023-10-14T18:18:24.031 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T18:18:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.258+0000 7fa68bb1d640 1 Processor -- start 2023-10-14T18:18:24.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.258+0000 7fa68bb1d640 1 -- start start 2023-10-14T18:18:24.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68bb1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 0x7fa684069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68bb1d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa68406a2e0 0x7fa6840c9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68bb1d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa68406a6f0 con 0x7fa68415d150 2023-10-14T18:18:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68bb1d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa68415d530 con 0x7fa68406a2e0 2023-10-14T18:18:24.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68bb1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6840ac890 con 0x7fa6840ad350 2023-10-14T18:18:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68a0ba640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa68406a2e0 0x7fa6840c9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa6890b8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 0x7fa684069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.260+0000 7fa68a0ba640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa68406a2e0 0x7fa6840c9000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:54882/0 (socket says 172.21.15.29:54882) 2023-10-14T18:18:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa68a0ba640 1 -- 172.21.15.29:0/987356828 learned_addr learned my addr 172.21.15.29:0/987356828 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa68a0ba640 1 -- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 msgr2=0x7fa684069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa68a0ba640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 0x7fa684069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa68a0ba640 1 -- 172.21.15.29:0/987356828 >> v1:172.21.15.186:6789/0 conn(0x7fa68415d150 legacy=0x7fa684160b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa68a0ba640 1 -- 172.21.15.29:0/987356828 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa684161650 con 0x7fa68406a2e0 2023-10-14T18:18:24.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.261+0000 7fa6890b8640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 0x7fa684069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:18:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.263+0000 7fa68a0ba640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa68406a2e0 0x7fa6840c9000 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009e50 tx=0x7fa67c02bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=53b72187e919a5c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.263+0000 7fa6888b7640 1 -- 172.21.15.29:0/987356828 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c02cb00 con 0x7fa68406a2e0 2023-10-14T18:18:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.264+0000 7fa6888b7640 1 -- 172.21.15.29:0/987356828 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa67c02cc60 con 0x7fa68406a2e0 2023-10-14T18:18:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.264+0000 7fa6888b7640 1 -- 172.21.15.29:0/987356828 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa67c0343f0 con 0x7fa68406a2e0 2023-10-14T18:18:24.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.264+0000 7fa68bb1d640 1 -- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa68406a2e0 msgr2=0x7fa6840c9000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.264+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa68406a2e0 0x7fa6840c9000 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa67c009e50 tx=0x7fa67c02bf60 comp rx=0 tx=0).stop 2023-10-14T18:18:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 -- 172.21.15.29:0/987356828 shutdown_connections 2023-10-14T18:18:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa6840ad350 0x7fa684069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/987356828 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa68406a2e0 0x7fa6840c9000 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 -- 172.21.15.29:0/987356828 >> 172.21.15.29:0/987356828 conn(0x7fa6840ce9b0 msgr2=0x7fa6840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 -- 172.21.15.29:0/987356828 shutdown_connections 2023-10-14T18:18:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.265+0000 7fa68bb1d640 1 -- 172.21.15.29:0/987356828 wait complete. 2023-10-14T18:18:24.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.267+0000 7fa68bb1d640 1 Processor -- start 2023-10-14T18:18:24.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.267+0000 7fa68bb1d640 1 -- start start 2023-10-14T18:18:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 0x7fa68414d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa68415d150 0x7fa6841f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6840c3200 con 0x7fa68406a2e0 2023-10-14T18:18:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6840c3aa0 con 0x7fa6840ad350 2023-10-14T18:18:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.268+0000 7fa68bb1d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6840b3df0 con 0x7fa68415d150 2023-10-14T18:18:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49200/0 (socket says 172.21.15.29:49200) 2023-10-14T18:18:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 -- 172.21.15.29:0/686200089 learned_addr learned my addr 172.21.15.29:0/686200089 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6890b8640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 0x7fa68414d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 -- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa68415d150 msgr2=0x7fa6841f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa68415d150 0x7fa6841f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 -- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 msgr2=0x7fa68414d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 0x7fa68414d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6898b9640 1 -- 172.21.15.29:0/686200089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa674009400 con 0x7fa68406a2e0 2023-10-14T18:18:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.269+0000 7fa6890b8640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 0x7fa68414d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:24.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa6898b9640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c400 tx=0x7fa68000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa6800052d0 con 0x7fa68406a2e0 2023-10-14T18:18:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa680005430 con 0x7fa68406a2e0 2023-10-14T18:18:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa67c009b30 con 0x7fa68406a2e0 2023-10-14T18:18:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa680003830 con 0x7fa68406a2e0 2023-10-14T18:18:24.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.270+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6840c3730 con 0x7fa68406a2e0 2023-10-14T18:18:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.271+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68404e160 con 0x7fa68406a2e0 2023-10-14T18:18:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.272+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa680003990 con 0x7fa68406a2e0 2023-10-14T18:18:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.273+0000 7fa67a7fc640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 0x7fa66806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.273+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fa680088f90 con 0x7fa68406a2e0 2023-10-14T18:18:24.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.274+0000 7fa6890b8640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 0x7fa66806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.274+0000 7fa6890b8640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 0x7fa66806d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6840c2e50 tx=0x7fa674008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.278+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa680053ca0 con 0x7fa68406a2e0 2023-10-14T18:18:24.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.476+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7fa6840ade10 con 0x7fa68406a2e0 2023-10-14T18:18:24.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.483+0000 7fa67a7fc640 1 -- 172.21.15.29:0/686200089 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v80) v1 ==== 120+0+0 (secure 0 0 0) 0x7fa6800572c0 con 0x7fa68406a2e0 2023-10-14T18:18:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 msgr2=0x7fa66806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 0x7fa66806d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa6840c2e50 tx=0x7fa674008040 comp rx=0 tx=0).stop 2023-10-14T18:18:24.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 msgr2=0x7fa6840b0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 secure :-1 s=READY pgs=2824 cs=0 l=1 rev1=1 crypto rx=0x7fa68000c400 tx=0x7fa68000c8c0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 shutdown_connections 2023-10-14T18:18:24.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa66806b0e0 0x7fa66806d5a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa68415d150 0x7fa6841f60d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa6840ad350 0x7fa68414d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 --2- 172.21.15.29:0/686200089 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa68406a2e0 0x7fa6840b0dc0 unknown :-1 s=CLOSED pgs=2824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 >> 172.21.15.29:0/686200089 conn(0x7fa6840ce9b0 msgr2=0x7fa6840ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:24.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 shutdown_connections 2023-10-14T18:18:24.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.487+0000 7fa68bb1d640 1 -- 172.21.15.29:0/686200089 wait complete. 2023-10-14T18:18:24.508 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T18:18:24.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.740+0000 7ff34ca00640 1 Processor -- start 2023-10-14T18:18:24.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.740+0000 7ff34ca00640 1 -- start start 2023-10-14T18:18:24.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.741+0000 7ff34ca00640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff34815d130 0x7ff3480cbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34ca00640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480cd390 con 0x7ff34806b8c0 2023-10-14T18:18:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34ca00640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3480cc2f0 con 0x7ff34806b1a0 2023-10-14T18:18:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34ca00640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3480cc5d0 con 0x7ff34815d130 2023-10-14T18:18:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff34806b8c0 0x7ff34806bca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38726/0 (socket says 172.21.15.29:38726) 2023-10-14T18:18:24.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34659c640 1 -- 172.21.15.29:0/642651237 learned_addr learned my addr 172.21.15.29:0/642651237 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff345d9b640 1 --2- 172.21.15.29:0/642651237 >> v2:172.21.15.186:3300/0 conn(0x7ff34815d130 0x7ff3480cbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3688877890 0 0) 0x7ff3480cd390 con 0x7ff34806b8c0 2023-10-14T18:18:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.742+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff334003520 con 0x7ff34806b8c0 2023-10-14T18:18:24.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4118527806 0 0) 0x7ff334003520 con 0x7ff34806b8c0 2023-10-14T18:18:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 >> v2:172.21.15.186:3300/0 conn(0x7ff34815d130 msgr2=0x7ff3480cbaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 --2- 172.21.15.29:0/642651237 >> v2:172.21.15.186:3300/0 conn(0x7ff34815d130 0x7ff3480cbaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 >> v1:172.21.15.186:6789/0 conn(0x7ff34806b1a0 legacy=0x7ff3480ad6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480cc8b0 con 0x7ff34806b8c0 2023-10-14T18:18:24.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ff338005410 con 0x7ff34806b8c0 2023-10-14T18:18:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 >> v1:172.21.15.29:6789/0 conn(0x7ff34806b8c0 legacy=0x7ff34806bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 0x7ff334003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3340047c0 0x7ff334004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff334005180 0x7ff33400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480cc5d0 con 0x7ff334003970 2023-10-14T18:18:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400c120 con 0x7ff3340047c0 2023-10-14T18:18:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff33400c2e0 con 0x7ff334005180 2023-10-14T18:18:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.743+0000 7ff34659c640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 0x7ff334003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff345d9b640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3340047c0 0x7ff334004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff34659c640 1 -- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff334005180 msgr2=0x7ff33400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff34659c640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff334005180 0x7ff33400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff34659c640 1 -- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3340047c0 msgr2=0x7ff334004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff34659c640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3340047c0 0x7ff334004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff34659c640 1 -- 172.21.15.29:0/642651237 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff334003520 con 0x7ff334003970 2023-10-14T18:18:24.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff345d9b640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff3340047c0 0x7ff334004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.744+0000 7ff346d9d640 1 -- 172.21.15.29:0/642651237 reap_dead start 2023-10-14T18:18:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.745+0000 7ff34659c640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 0x7ff334003d90 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7ff33800a820 tx=0x7ff3340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d51acb3e4fcb9d1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.745+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff338034630 con 0x7ff334003970 2023-10-14T18:18:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.745+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33802f070 con 0x7ff334003970 2023-10-14T18:18:24.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.745+0000 7ff34559a640 1 -- 172.21.15.29:0/642651237 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3380349f0 con 0x7ff334003970 2023-10-14T18:18:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 -- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 msgr2=0x7ff334003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 0x7ff334003d90 secure :-1 s=READY pgs=2826 cs=0 l=1 rev1=1 crypto rx=0x7ff33800a820 tx=0x7ff3340057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 -- 172.21.15.29:0/642651237 shutdown_connections 2023-10-14T18:18:24.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 --2- 172.21.15.29:0/642651237 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff334003970 0x7ff334003d90 unknown :-1 s=CLOSED pgs=2826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 -- 172.21.15.29:0/642651237 >> 172.21.15.29:0/642651237 conn(0x7ff3480d0ad0 msgr2=0x7ff3480d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 -- 172.21.15.29:0/642651237 shutdown_connections 2023-10-14T18:18:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.746+0000 7ff34ca00640 1 -- 172.21.15.29:0/642651237 wait complete. 2023-10-14T18:18:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.748+0000 7ff34ca00640 1 Processor -- start 2023-10-14T18:18:24.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.748+0000 7ff34ca00640 1 -- start start 2023-10-14T18:18:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff334003970 0x7ff34815a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff348143c20 0x7ff3480c6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c6ee0 con 0x7ff348144080 2023-10-14T18:18:24.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c7bb0 con 0x7ff334003970 2023-10-14T18:18:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff34ca00640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3480c5f80 con 0x7ff348143c20 2023-10-14T18:18:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff346d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff346d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49204/0 (socket says 172.21.15.29:49204) 2023-10-14T18:18:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.749+0000 7ff346d9d640 1 -- 172.21.15.29:0/985243448 learned_addr learned my addr 172.21.15.29:0/985243448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:24.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff345d9b640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff348143c20 0x7ff3480c6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 -- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff348143c20 msgr2=0x7ff3480c6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff348143c20 0x7ff3480c6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 -- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff334003970 msgr2=0x7ff34815a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff334003970 0x7ff34815a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 -- 172.21.15.29:0/985243448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3480c7e50 con 0x7ff348144080 2023-10-14T18:18:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff346d9d640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7ff34806a7d0 tx=0x7ff34000c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340004220 con 0x7ff348144080 2023-10-14T18:18:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff340004380 con 0x7ff348144080 2023-10-14T18:18:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.750+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff340007310 con 0x7ff348144080 2023-10-14T18:18:24.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.751+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3480c8130 con 0x7ff348144080 2023-10-14T18:18:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.751+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff34815ad60 con 0x7ff348144080 2023-10-14T18:18:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.752+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff34804e160 con 0x7ff348144080 2023-10-14T18:18:24.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.753+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff340007470 con 0x7ff348144080 2023-10-14T18:18:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.754+0000 7ff33effd640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 0x7ff32806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.754+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7ff340088ba0 con 0x7ff348144080 2023-10-14T18:18:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.754+0000 7ff34659c640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 0x7ff32806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:24.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.755+0000 7ff34659c640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 0x7ff32806d5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff338034000 tx=0x7ff338009030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:24.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.759+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff340053760 con 0x7ff348144080 2023-10-14T18:18:24.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.960+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff34806b6a0 con 0x7ff348144080 2023-10-14T18:18:24.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.967+0000 7ff33effd640 1 -- 172.21.15.29:0/985243448 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v81) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff34000cea0 con 0x7ff348144080 2023-10-14T18:18:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 msgr2=0x7ff32806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 0x7ff32806d5a0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff338034000 tx=0x7ff338009030 comp rx=0 tx=0).stop 2023-10-14T18:18:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 msgr2=0x7ff3480c69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:24.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 secure :-1 s=READY pgs=2827 cs=0 l=1 rev1=1 crypto rx=0x7ff34806a7d0 tx=0x7ff34000c820 comp rx=0 tx=0).stop 2023-10-14T18:18:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 shutdown_connections 2023-10-14T18:18:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff32806b0e0 0x7ff32806d5a0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff348144080 0x7ff3480c69a0 unknown :-1 s=CLOSED pgs=2827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff348143c20 0x7ff3480c6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 --2- 172.21.15.29:0/985243448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff334003970 0x7ff34815a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 >> 172.21.15.29:0/985243448 conn(0x7ff3480d0ad0 msgr2=0x7ff34815d540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 shutdown_connections 2023-10-14T18:18:24.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:24.970+0000 7ff34ca00640 1 -- 172.21.15.29:0/985243448 wait complete. 2023-10-14T18:18:24.992 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs' 2023-10-14T18:18:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.217+0000 7f789703f640 1 Processor -- start 2023-10-14T18:18:25.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.217+0000 7f789703f640 1 -- start start 2023-10-14T18:18:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f789703f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f78900adb50 0x7f789006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f789703f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78900b0b20 0x7f7890164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f789703f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7890071450 con 0x7f7890070fd0 2023-10-14T18:18:25.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f789703f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78900cc600 con 0x7f78900b0b20 2023-10-14T18:18:25.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f789703f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78900cc8e0 con 0x7f78900adb50 2023-10-14T18:18:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.219+0000 7f78955dc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78900b0b20 0x7f7890164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f78900b0b20 0x7f7890164f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49216/0 (socket says 172.21.15.29:49216) 2023-10-14T18:18:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 -- 172.21.15.29:0/126430199 learned_addr learned my addr 172.21.15.29:0/126430199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:25.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f7894ddb640 1 --2- 172.21.15.29:0/126430199 >> v2:172.21.15.186:3300/0 conn(0x7f78900adb50 0x7f789006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 -- 172.21.15.29:0/126430199 >> v2:172.21.15.186:3300/0 conn(0x7f78900adb50 msgr2=0x7f789006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 --2- 172.21.15.29:0/126430199 >> v2:172.21.15.186:3300/0 conn(0x7f78900adb50 0x7f789006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 -- 172.21.15.29:0/126430199 >> v1:172.21.15.186:6789/0 conn(0x7f7890070fd0 legacy=0x7f78900b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:25.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.220+0000 7f78955dc640 1 -- 172.21.15.29:0/126430199 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78900ccbc0 con 0x7f78900b0b20 2023-10-14T18:18:25.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.221+0000 7f78955dc640 1 --2- 172.21.15.29:0/126430199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900b0b20 0x7f7890164f60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f7890071a00 tx=0x7f788802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e65377dd4a7a7b46 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f788f7fe640 1 -- 172.21.15.29:0/126430199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f788802ca80 con 0x7f78900b0b20 2023-10-14T18:18:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f788f7fe640 1 -- 172.21.15.29:0/126430199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f788802cbe0 con 0x7f78900b0b20 2023-10-14T18:18:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f788f7fe640 1 -- 172.21.15.29:0/126430199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78880343f0 con 0x7f78900b0b20 2023-10-14T18:18:25.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f789703f640 1 -- 172.21.15.29:0/126430199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900b0b20 msgr2=0x7f7890164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f789703f640 1 --2- 172.21.15.29:0/126430199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900b0b20 0x7f7890164f60 secure :-1 s=READY pgs=2828 cs=0 l=1 rev1=1 crypto rx=0x7f7890071a00 tx=0x7f788802bee0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.222+0000 7f789703f640 1 -- 172.21.15.29:0/126430199 shutdown_connections 2023-10-14T18:18:25.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.223+0000 7f789703f640 1 --2- 172.21.15.29:0/126430199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900b0b20 0x7f7890164f60 unknown :-1 s=CLOSED pgs=2828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.223+0000 7f789703f640 1 --2- 172.21.15.29:0/126430199 >> v2:172.21.15.186:3300/0 conn(0x7f78900adb50 0x7f789006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.223+0000 7f789703f640 1 -- 172.21.15.29:0/126430199 >> 172.21.15.29:0/126430199 conn(0x7f78900cb410 msgr2=0x7f78900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.223+0000 7f789703f640 1 -- 172.21.15.29:0/126430199 shutdown_connections 2023-10-14T18:18:25.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.223+0000 7f789703f640 1 -- 172.21.15.29:0/126430199 wait complete. 2023-10-14T18:18:25.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.225+0000 7f789703f640 1 Processor -- start 2023-10-14T18:18:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.225+0000 7f789703f640 1 -- start start 2023-10-14T18:18:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7890070fd0 0x7f78900c5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.230 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78900b0b20 0x7f7890151d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78900bda90 con 0x7f78900adb50 2023-10-14T18:18:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78900c3550 con 0x7f7890070fd0 2023-10-14T18:18:25.231 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.226+0000 7f789703f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f78900c37f0 con 0x7f78900b0b20 2023-10-14T18:18:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49230/0 (socket says 172.21.15.29:49230) 2023-10-14T18:18:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 -- 172.21.15.29:0/3210427598 learned_addr learned my addr 172.21.15.29:0/3210427598 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f7894ddb640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7890070fd0 0x7f78900c5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.232 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 -- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78900b0b20 msgr2=0x7f7890151d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78900b0b20 0x7f7890151d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 -- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7890070fd0 msgr2=0x7f78900c5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7890070fd0 0x7f78900c5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 -- 172.21.15.29:0/3210427598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7888009b30 con 0x7f78900adb50 2023-10-14T18:18:25.233 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.227+0000 7f788ffff640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b3f0 tx=0x7f787c00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.228+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c011830 con 0x7f78900adb50 2023-10-14T18:18:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.228+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f787c011990 con 0x7f78900adb50 2023-10-14T18:18:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.228+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f787c011d40 con 0x7f78900adb50 2023-10-14T18:18:25.234 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.228+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78900c3ad0 con 0x7f78900adb50 2023-10-14T18:18:25.235 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.228+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78900c57e0 con 0x7f78900adb50 2023-10-14T18:18:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.229+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78900c3ff0 con 0x7f78900adb50 2023-10-14T18:18:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.230+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f787c01a400 con 0x7f78900adb50 2023-10-14T18:18:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.231+0000 7f788d7fa640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 0x7f787406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.236 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.231+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f787c089920 con 0x7f78900adb50 2023-10-14T18:18:25.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.231+0000 7f7894ddb640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 0x7f787406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.232+0000 7f7894ddb640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 0x7f787406d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7884006fd0 tx=0x7f7884008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.237 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.235+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f78900c3ff0 con 0x7f78900adb50 2023-10-14T18:18:25.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.503+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]} v 0) v1 -- 0x7f78900c3ff0 con 0x7f78900adb50 2023-10-14T18:18:25.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.540+0000 7f788d7fa640 1 -- 172.21.15.29:0/3210427598 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_for_non_default_fs"]}]=0 v3235) v1 ==== 186+0+0 (secure 0 0 0) 0x7f78900c3ff0 con 0x7f78900adb50 2023-10-14T18:18:25.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 msgr2=0x7f787406d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 0x7f787406d5c0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f7884006fd0 tx=0x7f7884008040 comp rx=0 tx=0).stop 2023-10-14T18:18:25.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 msgr2=0x7f78900bd550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 secure :-1 s=READY pgs=2829 cs=0 l=1 rev1=1 crypto rx=0x7f787c00b3f0 tx=0x7f787c00b8b0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 shutdown_connections 2023-10-14T18:18:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f787406b100 0x7f787406d5c0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f78900b0b20 0x7f7890151d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f78900adb50 0x7f78900bd550 unknown :-1 s=CLOSED pgs=2829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.543+0000 7f789703f640 1 --2- 172.21.15.29:0/3210427598 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7890070fd0 0x7f78900c5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.544+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 >> 172.21.15.29:0/3210427598 conn(0x7f78900cb410 msgr2=0x7f78900af660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:25.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.544+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 shutdown_connections 2023-10-14T18:18:25.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.544+0000 7f789703f640 1 -- 172.21.15.29:0/3210427598 wait complete. 2023-10-14T18:18:25.566 INFO:tasks.cephfs_test_runner:Test command succes with --fs argument for non-default fs ... ok 2023-10-14T18:18:25.566 INFO:tasks.cephfs_test_runner:Starting test: test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) Test command fails without --fs argument in presence of multiple fs 2023-10-14T18:18:25.567 INFO:tasks.cephfs_test_runner:test_snap_schedule_without_fs_argument (tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument) 2023-10-14T18:18:25.567 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:25.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.795+0000 7f490c51d640 1 Processor -- start 2023-10-14T18:18:25.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.795+0000 7f490c51d640 1 -- start start 2023-10-14T18:18:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490c51d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f49040cfc10 0x7f49040d0b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490c51d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49040b3e70 0x7f49040d1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490c51d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49040b3940 con 0x7f49040b3070 2023-10-14T18:18:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490c51d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49040b45e0 con 0x7f49040b3e70 2023-10-14T18:18:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490c51d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49040c7880 con 0x7f49040cfc10 2023-10-14T18:18:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490aaba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49040b3e70 0x7f49040d1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490aaba640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f49040b3e70 0x7f49040d1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49242/0 (socket says 172.21.15.29:49242) 2023-10-14T18:18:25.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.796+0000 7f490aaba640 1 -- 172.21.15.29:0/773599873 learned_addr learned my addr 172.21.15.29:0/773599873 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.797+0000 7f4909ab8640 1 --2- 172.21.15.29:0/773599873 >> v2:172.21.15.186:3300/0 conn(0x7f49040cfc10 0x7f49040d0b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.797+0000 7f490aaba640 1 -- 172.21.15.29:0/773599873 >> v2:172.21.15.186:3300/0 conn(0x7f49040cfc10 msgr2=0x7f49040d0b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:25.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.797+0000 7f490aaba640 1 --2- 172.21.15.29:0/773599873 >> v2:172.21.15.186:3300/0 conn(0x7f49040cfc10 0x7f49040d0b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.797+0000 7f490aaba640 1 -- 172.21.15.29:0/773599873 >> v1:172.21.15.186:6789/0 conn(0x7f49040b3070 legacy=0x7f49040cf500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.797+0000 7f490aaba640 1 -- 172.21.15.29:0/773599873 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49040b3b90 con 0x7f49040b3e70 2023-10-14T18:18:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.798+0000 7f490aaba640 1 --2- 172.21.15.29:0/773599873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3e70 0x7f49040d1040 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f49040b58e0 tx=0x7f48fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=277eb332c0534fe8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.798+0000 7f49092b7640 1 -- 172.21.15.29:0/773599873 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc031870 con 0x7f49040b3e70 2023-10-14T18:18:25.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.798+0000 7f49092b7640 1 -- 172.21.15.29:0/773599873 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48fc0319d0 con 0x7f49040b3e70 2023-10-14T18:18:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.798+0000 7f49092b7640 1 -- 172.21.15.29:0/773599873 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48fc031d60 con 0x7f49040b3e70 2023-10-14T18:18:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 -- 172.21.15.29:0/773599873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3e70 msgr2=0x7f49040d1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 --2- 172.21.15.29:0/773599873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3e70 0x7f49040d1040 secure :-1 s=READY pgs=2830 cs=0 l=1 rev1=1 crypto rx=0x7f49040b58e0 tx=0x7f48fc02b910 comp rx=0 tx=0).stop 2023-10-14T18:18:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 -- 172.21.15.29:0/773599873 shutdown_connections 2023-10-14T18:18:25.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 --2- 172.21.15.29:0/773599873 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3e70 0x7f49040d1040 unknown :-1 s=CLOSED pgs=2830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 --2- 172.21.15.29:0/773599873 >> v2:172.21.15.186:3300/0 conn(0x7f49040cfc10 0x7f49040d0b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 -- 172.21.15.29:0/773599873 >> 172.21.15.29:0/773599873 conn(0x7f49040d2d20 msgr2=0x7f49040d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 -- 172.21.15.29:0/773599873 shutdown_connections 2023-10-14T18:18:25.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.799+0000 7f490c51d640 1 -- 172.21.15.29:0/773599873 wait complete. 2023-10-14T18:18:25.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.801+0000 7f490c51d640 1 Processor -- start 2023-10-14T18:18:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.801+0000 7f490c51d640 1 -- start start 2023-10-14T18:18:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49040b3e70 0x7f490409e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 0x7f490409e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490409f400 con 0x7f49040b3070 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49040cae40 con 0x7f49040b3e70 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490c51d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49041598e0 con 0x7f49040cfc10 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.802+0000 7f490a2b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49252/0 (socket says 172.21.15.29:49252) 2023-10-14T18:18:25.808 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 -- 172.21.15.29:0/4196444949 learned_addr learned my addr 172.21.15.29:0/4196444949 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f4909ab8640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49040b3e70 0x7f490409e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490aaba640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 0x7f490409e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 -- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 msgr2=0x7f490409e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 0x7f490409e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.809 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 -- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49040b3e70 msgr2=0x7f490409e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49040b3e70 0x7f490409e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 -- 172.21.15.29:0/4196444949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48fc0094d0 con 0x7f49040b3070 2023-10-14T18:18:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490aaba640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 0x7f490409e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.803+0000 7f490a2b9640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f490000c7a0 tx=0x7f490000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.804+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f490000aa50 con 0x7f49040b3070 2023-10-14T18:18:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.804+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490000abb0 con 0x7f49040b3070 2023-10-14T18:18:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.804+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4900012420 con 0x7f49040b3070 2023-10-14T18:18:25.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.804+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490415a6e0 con 0x7f49040b3070 2023-10-14T18:18:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.804+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490415aca0 con 0x7f49040b3070 2023-10-14T18:18:25.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.805+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f490404e160 con 0x7f49040b3070 2023-10-14T18:18:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.806+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4900012580 con 0x7f49040b3070 2023-10-14T18:18:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.807+0000 7f48faffd640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 0x7f48e806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:25.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.807+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f490008a150 con 0x7f49040b3070 2023-10-14T18:18:25.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.808+0000 7f4909ab8640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 0x7f48e806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:25.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.808+0000 7f4909ab8640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 0x7f48e806d570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006fd0 tx=0x7f48f4008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:25.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:25.812+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4900054d00 con 0x7f49040b3070 2023-10-14T18:18:26.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.006+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49040cdab0 con 0x7f49040b3070 2023-10-14T18:18:26.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.007+0000 7f48faffd640 1 -- 172.21.15.29:0/4196444949 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v241) v1 ==== 74+0+14487 (secure 0 0 0) 0x7f4900058320 con 0x7f49040b3070 2023-10-14T18:18:26.009 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:26.009 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":241,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:23.515320+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:6835/2605417960":"2023-10-15T18:18:20.245534+0000","172.21.15.29:6834/2605417960":"2023-10-15T18:18:20.245534+0000","172.21.15.29:6836/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2409880047":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/2254489030":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/4078168242":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/218607793":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/1511626404":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/3503206280":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/242473318":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6833/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2804419857":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6837/1082656487":"2023-10-15T18:18:12.068111+0000","172.21.15.29:0/2977418585":"2023-10-15T18:18:06.375162+0000","172.21.15.29:0/761044706":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1993258206":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2825661941":"2023-10-15T18:17:54.886178+0000","172.21.15.29:6833/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/4157539079":"2023-10-15T18:18:06.375162+0000","172.21.15.29:6832/2967217610":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/1476090294":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/2871474293":"2023-10-15T18:17:54.886178+0000","172.21.15.29:0/3012006424":"2023-10-15T18:17:54.886178+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:26.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 msgr2=0x7f48e806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 0x7f48e806d570 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f48f4006fd0 tx=0x7f48f4008040 comp rx=0 tx=0).stop 2023-10-14T18:18:26.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 msgr2=0x7f49041593a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 secure :-1 s=READY pgs=2831 cs=0 l=1 rev1=1 crypto rx=0x7f490000c7a0 tx=0x7f490000cc60 comp rx=0 tx=0).stop 2023-10-14T18:18:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 shutdown_connections 2023-10-14T18:18:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f48e806b0b0 0x7f48e806d570 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f49040cfc10 0x7f490409e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f49040b3e70 0x7f490409e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 --2- 172.21.15.29:0/4196444949 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f49040b3070 0x7f49041593a0 unknown :-1 s=CLOSED pgs=2831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 >> 172.21.15.29:0/4196444949 conn(0x7f49040d2d20 msgr2=0x7f490415d150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 shutdown_connections 2023-10-14T18:18:26.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.010+0000 7f490c51d640 1 -- 172.21.15.29:0/4196444949 wait complete. 2023-10-14T18:18:26.029 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:26.029 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:26.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.267+0000 7f00aaeec640 1 Processor -- start 2023-10-14T18:18:26.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.267+0000 7f00aaeec640 1 -- start start 2023-10-14T18:18:26.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.268+0000 7f00aaeec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00a415e350 0x7f00a40cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.268+0000 7f00aaeec640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 0x7f00a40ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.268+0000 7f00aaeec640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00a406fbc0 con 0x7f00a415e980 2023-10-14T18:18:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.268+0000 7f00aaeec640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a415e730 con 0x7f00a415e350 2023-10-14T18:18:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.268+0000 7f00aaeec640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00a40c9920 con 0x7f00a4070970 2023-10-14T18:18:26.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a9eea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f00a415e350 0x7f00a40cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00aa6eb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f00a415e980 0x7f00a415ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38736/0 (socket says 172.21.15.29:38736) 2023-10-14T18:18:26.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00aa6eb640 1 -- 172.21.15.29:0/4113442442 learned_addr learned my addr 172.21.15.29:0/4113442442 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:26.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a96e9640 1 --2- 172.21.15.29:0/4113442442 >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 0x7f00a40ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a9eea640 1 -- 172.21.15.29:0/4113442442 >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 msgr2=0x7f00a40ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:26.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a9eea640 1 --2- 172.21.15.29:0/4113442442 >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 0x7f00a40ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a9eea640 1 -- 172.21.15.29:0/4113442442 >> v1:172.21.15.29:6789/0 conn(0x7f00a415e980 legacy=0x7f00a415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a9eea640 1 -- 172.21.15.29:0/4113442442 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a40c9c00 con 0x7f00a415e350 2023-10-14T18:18:26.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a96e9640 1 --2- 172.21.15.29:0/4113442442 >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 0x7f00a40ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.269+0000 7f00a8ee8640 1 -- 172.21.15.29:0/4113442442 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1266032641 0 0) 0x7f00a406fbc0 con 0x7f00a415e980 2023-10-14T18:18:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.270+0000 7f00a9eea640 1 --2- 172.21.15.29:0/4113442442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a40cdb40 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f00a40671f0 tx=0x7f00a002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42ec04721eb433f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00a8ee8640 1 -- 172.21.15.29:0/4113442442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a002e070 con 0x7f00a415e350 2023-10-14T18:18:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00a8ee8640 1 -- 172.21.15.29:0/4113442442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00a00315a0 con 0x7f00a415e350 2023-10-14T18:18:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00a8ee8640 1 -- 172.21.15.29:0/4113442442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00a0031930 con 0x7f00a415e350 2023-10-14T18:18:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 -- 172.21.15.29:0/4113442442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 msgr2=0x7f00a40cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 --2- 172.21.15.29:0/4113442442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a40cdb40 secure :-1 s=READY pgs=2833 cs=0 l=1 rev1=1 crypto rx=0x7f00a40671f0 tx=0x7f00a002b840 comp rx=0 tx=0).stop 2023-10-14T18:18:26.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 -- 172.21.15.29:0/4113442442 shutdown_connections 2023-10-14T18:18:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 --2- 172.21.15.29:0/4113442442 >> v2:172.21.15.186:3300/0 conn(0x7f00a4070970 0x7f00a40ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 --2- 172.21.15.29:0/4113442442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a40cdb40 unknown :-1 s=CLOSED pgs=2833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 -- 172.21.15.29:0/4113442442 >> 172.21.15.29:0/4113442442 conn(0x7f00a40b0a50 msgr2=0x7f00a40b0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.271+0000 7f00aaeec640 1 -- 172.21.15.29:0/4113442442 shutdown_connections 2023-10-14T18:18:26.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.272+0000 7f00aaeec640 1 -- 172.21.15.29:0/4113442442 wait complete. 2023-10-14T18:18:26.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.273+0000 7f00aaeec640 1 Processor -- start 2023-10-14T18:18:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.273+0000 7f00aaeec640 1 -- start start 2023-10-14T18:18:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00a4070970 0x7f00a41573e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 0x7f00a40c7540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40c64a0 con 0x7f00a415e350 2023-10-14T18:18:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40c6780 con 0x7f00a415e980 2023-10-14T18:18:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.274+0000 7f00aaeec640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00a40c6a60 con 0x7f00a4070970 2023-10-14T18:18:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49272/0 (socket says 172.21.15.29:49272) 2023-10-14T18:18:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 -- 172.21.15.29:0/3641515930 learned_addr learned my addr 172.21.15.29:0/3641515930 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 -- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00a4070970 msgr2=0x7f00a41573e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00a4070970 0x7f00a41573e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00aa6eb640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 0x7f00a40c7540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 -- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 msgr2=0x7f00a40c7540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 0x7f00a40c7540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 -- 172.21.15.29:0/3641515930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00a0009400 con 0x7f00a415e350 2023-10-14T18:18:26.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00aa6eb640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 0x7f00a40c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:26.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.275+0000 7f00a96e9640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f009400c7f0 tx=0x7f009400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.276+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094007820 con 0x7f00a415e350 2023-10-14T18:18:26.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.276+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0094007980 con 0x7f00a415e350 2023-10-14T18:18:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.276+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00a40c6d40 con 0x7f00a415e350 2023-10-14T18:18:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.276+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0094007cf0 con 0x7f00a415e350 2023-10-14T18:18:26.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.276+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00a40c6030 con 0x7f00a415e350 2023-10-14T18:18:26.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.277+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a406e500 con 0x7f00a415e350 2023-10-14T18:18:26.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.278+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f00940113f0 con 0x7f00a415e350 2023-10-14T18:18:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.279+0000 7f009a7fc640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 0x7f008806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.279+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f0094089f70 con 0x7f00a415e350 2023-10-14T18:18:26.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.280+0000 7f00a9eea640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 0x7f008806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.280+0000 7f00a9eea640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 0x7f008806d570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f00a0036000 tx=0x7f00a0030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.284+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0094054aa0 con 0x7f00a415e350 2023-10-14T18:18:26.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.473+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f00a40716e0 con 0x7f00a415e350 2023-10-14T18:18:26.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.474+0000 7f009a7fc640 1 -- 172.21.15.29:0/3641515930 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v241) v1 ==== 93+0+10 (secure 0 0 0) 0x7f009401f0a0 con 0x7f00a415e350 2023-10-14T18:18:26.475 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:26.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 msgr2=0x7f008806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 0x7f008806d570 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f00a0036000 tx=0x7f00a0030000 comp rx=0 tx=0).stop 2023-10-14T18:18:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 msgr2=0x7f00a41e8cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 secure :-1 s=READY pgs=2834 cs=0 l=1 rev1=1 crypto rx=0x7f009400c7f0 tx=0x7f009400ccb0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 shutdown_connections 2023-10-14T18:18:26.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f008806b0b0 0x7f008806d570 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f00a415e980 0x7f00a40c7540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f00a415e350 0x7f00a41e8cb0 unknown :-1 s=CLOSED pgs=2834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 --2- 172.21.15.29:0/3641515930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f00a4070970 0x7f00a41573e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 >> 172.21.15.29:0/3641515930 conn(0x7f00a40b0a50 msgr2=0x7f00a40ccb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:26.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 shutdown_connections 2023-10-14T18:18:26.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.477+0000 7f00aaeec640 1 -- 172.21.15.29:0/3641515930 wait complete. 2023-10-14T18:18:26.496 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-14T18:18:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.714+0000 7fd0846e7640 1 Processor -- start 2023-10-14T18:18:26.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.714+0000 7fd0846e7640 1 -- start start 2023-10-14T18:18:26.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd07c15aaa0 0x7fd07c15e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15ae80 0x7fd07c0cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd07c0ccaa0 0x7fd07c070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c06b810 con 0x7fd07c15aaa0 2023-10-14T18:18:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd07c15ed60 con 0x7fd07c0ccaa0 2023-10-14T18:18:26.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.715+0000 7fd0846e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c0707e0 con 0x7fd07c15ae80 2023-10-14T18:18:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd07c15aaa0 0x7fd07c15e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd07c15aaa0 0x7fd07c15e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49274/0 (socket says 172.21.15.29:49274) 2023-10-14T18:18:26.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 -- 172.21.15.29:0/1635110870 learned_addr learned my addr 172.21.15.29:0/1635110870 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 -- 172.21.15.29:0/1635110870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15ae80 msgr2=0x7fd07c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd081c82640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15ae80 0x7fd07c0cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15ae80 0x7fd07c0cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082c84640 1 --2- 172.21.15.29:0/1635110870 >> v2:172.21.15.186:3300/0 conn(0x7fd07c0ccaa0 0x7fd07c070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 -- 172.21.15.29:0/1635110870 >> v2:172.21.15.186:3300/0 conn(0x7fd07c0ccaa0 msgr2=0x7fd07c070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 --2- 172.21.15.29:0/1635110870 >> v2:172.21.15.186:3300/0 conn(0x7fd07c0ccaa0 0x7fd07c070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.716+0000 7fd082483640 1 -- 172.21.15.29:0/1635110870 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c15ef90 con 0x7fd07c15aaa0 2023-10-14T18:18:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.717+0000 7fd082483640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15aaa0 0x7fd07c15e820 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0ae3c0 tx=0x7fd07802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e058fe5563e2bbd4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.718+0000 7fd081481640 1 -- 172.21.15.29:0/1635110870 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07802ca70 con 0x7fd07c15aaa0 2023-10-14T18:18:26.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.718+0000 7fd081481640 1 -- 172.21.15.29:0/1635110870 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07802cbd0 con 0x7fd07c15aaa0 2023-10-14T18:18:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.718+0000 7fd081481640 1 -- 172.21.15.29:0/1635110870 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0780343f0 con 0x7fd07c15aaa0 2023-10-14T18:18:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.718+0000 7fd0846e7640 1 -- 172.21.15.29:0/1635110870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15aaa0 msgr2=0x7fd07c15e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.718+0000 7fd0846e7640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15aaa0 0x7fd07c15e820 secure :-1 s=READY pgs=2835 cs=0 l=1 rev1=1 crypto rx=0x7fd07c0ae3c0 tx=0x7fd07802bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 -- 172.21.15.29:0/1635110870 shutdown_connections 2023-10-14T18:18:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 --2- 172.21.15.29:0/1635110870 >> v2:172.21.15.186:3300/0 conn(0x7fd07c0ccaa0 0x7fd07c070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15ae80 0x7fd07c0cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 --2- 172.21.15.29:0/1635110870 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15aaa0 0x7fd07c15e820 unknown :-1 s=CLOSED pgs=2835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 -- 172.21.15.29:0/1635110870 >> 172.21.15.29:0/1635110870 conn(0x7fd07c0cb410 msgr2=0x7fd07c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:26.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 -- 172.21.15.29:0/1635110870 shutdown_connections 2023-10-14T18:18:26.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.719+0000 7fd0846e7640 1 -- 172.21.15.29:0/1635110870 wait complete. 2023-10-14T18:18:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.721+0000 7fd0846e7640 1 Processor -- start 2023-10-14T18:18:26.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.721+0000 7fd0846e7640 1 -- start start 2023-10-14T18:18:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07c0ccaa0 0x7fd07c1587d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15aaa0 0x7fd07c144e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1f5a80 con 0x7fd07c15ae80 2023-10-14T18:18:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1591d0 con 0x7fd07c0ccaa0 2023-10-14T18:18:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd0846e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd07c1594b0 con 0x7fd07c15aaa0 2023-10-14T18:18:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49282/0 (socket says 172.21.15.29:49282) 2023-10-14T18:18:26.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 -- 172.21.15.29:0/3042707862 learned_addr learned my addr 172.21.15.29:0/3042707862 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 -- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15aaa0 msgr2=0x7fd07c144e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15aaa0 0x7fd07c144e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082483640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07c0ccaa0 0x7fd07c1587d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 -- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07c0ccaa0 msgr2=0x7fd07c1587d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:26.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07c0ccaa0 0x7fd07c1587d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.722+0000 7fd082c84640 1 -- 172.21.15.29:0/3042707862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd078009b30 con 0x7fd07c15ae80 2023-10-14T18:18:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd082c84640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c4c0 tx=0x7fd07400c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074004220 con 0x7fd07c15ae80 2023-10-14T18:18:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd074004380 con 0x7fd07c15ae80 2023-10-14T18:18:26.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd074003810 con 0x7fd07c15ae80 2023-10-14T18:18:26.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd07c159790 con 0x7fd07c15ae80 2023-10-14T18:18:26.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.723+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd07c0c3640 con 0x7fd07c15ae80 2023-10-14T18:18:26.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.724+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd07c0c3860 con 0x7fd07c15ae80 2023-10-14T18:18:26.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.725+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd074003970 con 0x7fd07c15ae80 2023-10-14T18:18:26.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.726+0000 7fd072ffd640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 0x7fd06006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:26.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.726+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7fd074089880 con 0x7fd07c15ae80 2023-10-14T18:18:26.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.726+0000 7fd082483640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 0x7fd06006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.727+0000 7fd082483640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 0x7fd06006d570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd078035000 tx=0x7fd078030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:26.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.731+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd0740543b0 con 0x7fd07c15ae80 2023-10-14T18:18:27.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:26.998+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7fd07c04e160 con 0x7fd07c15ae80 2023-10-14T18:18:27.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.554+0000 7fd072ffd640 1 -- 172.21.15.29:0/3042707862 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3237) v1 ==== 190+0+0 (secure 0 0 0) 0x7fd0740579d0 con 0x7fd07c15ae80 2023-10-14T18:18:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 msgr2=0x7fd06006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 0x7fd06006d570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd078035000 tx=0x7fd078030000 comp rx=0 tx=0).stop 2023-10-14T18:18:27.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 msgr2=0x7fd07c1453d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 secure :-1 s=READY pgs=2836 cs=0 l=1 rev1=1 crypto rx=0x7fd07400c4c0 tx=0x7fd07400c980 comp rx=0 tx=0).stop 2023-10-14T18:18:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 shutdown_connections 2023-10-14T18:18:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06006b0b0 0x7fd06006d570 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd07c15ae80 0x7fd07c1453d0 unknown :-1 s=CLOSED pgs=2836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd07c15aaa0 0x7fd07c144e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 --2- 172.21.15.29:0/3042707862 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd07c0ccaa0 0x7fd07c1587d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 >> 172.21.15.29:0/3042707862 conn(0x7fd07c0cb410 msgr2=0x7fd07c0cead0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.558+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 shutdown_connections 2023-10-14T18:18:27.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.559+0000 7fd0846e7640 1 -- 172.21.15.29:0/3042707862 wait complete. 2023-10-14T18:18:27.583 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T18:18:27.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.815+0000 7f13464a3640 1 Processor -- start 2023-10-14T18:18:27.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.815+0000 7f13464a3640 1 -- start start 2023-10-14T18:18:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.816+0000 7f13464a3640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f134015cd70 0x7f13400d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.816+0000 7f13464a3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400b8480 0x7f13400cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.816+0000 7f13464a3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f134015d150 con 0x7f13400d0be0 2023-10-14T18:18:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.816+0000 7f13464a3640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13400d1180 con 0x7f134015cd70 2023-10-14T18:18:27.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.816+0000 7f13464a3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13400d1460 con 0x7f13400b8480 2023-10-14T18:18:27.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.817+0000 7f13454a1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f134015cd70 0x7f13400d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.817+0000 7f13454a1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f134015cd70 0x7f13400d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:54992/0 (socket says 172.21.15.29:54992) 2023-10-14T18:18:27.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.817+0000 7f13454a1640 1 -- 172.21.15.29:0/4121183310 learned_addr learned my addr 172.21.15.29:0/4121183310 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:27.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.817+0000 7f1344ca0640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400b8480 0x7f13400cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.818+0000 7f13454a1640 1 -- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400b8480 msgr2=0x7f13400cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:27.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.818+0000 7f13454a1640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400b8480 0x7f13400cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.818+0000 7f13454a1640 1 -- 172.21.15.29:0/4121183310 >> v1:172.21.15.186:6789/0 conn(0x7f13400d0be0 legacy=0x7f13400b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.818+0000 7f13454a1640 1 -- 172.21.15.29:0/4121183310 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13400d1740 con 0x7f134015cd70 2023-10-14T18:18:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.819+0000 7f13454a1640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f13400d2220 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f133c009870 tx=0x7f133c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3140a2885006e16 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.820+0000 7f1337fff640 1 -- 172.21.15.29:0/4121183310 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c030840 con 0x7f134015cd70 2023-10-14T18:18:27.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.820+0000 7f1337fff640 1 -- 172.21.15.29:0/4121183310 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c0309a0 con 0x7f134015cd70 2023-10-14T18:18:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.820+0000 7f1337fff640 1 -- 172.21.15.29:0/4121183310 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c030d30 con 0x7f134015cd70 2023-10-14T18:18:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.820+0000 7f13464a3640 1 -- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 msgr2=0x7f13400d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.820+0000 7f13464a3640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f13400d2220 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f133c009870 tx=0x7f133c02b980 comp rx=0 tx=0).stop 2023-10-14T18:18:27.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 -- 172.21.15.29:0/4121183310 shutdown_connections 2023-10-14T18:18:27.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400b8480 0x7f13400cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 --2- 172.21.15.29:0/4121183310 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f13400d2220 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 -- 172.21.15.29:0/4121183310 >> 172.21.15.29:0/4121183310 conn(0x7f13400cf570 msgr2=0x7f13400cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:27.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 -- 172.21.15.29:0/4121183310 shutdown_connections 2023-10-14T18:18:27.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.821+0000 7f13464a3640 1 -- 172.21.15.29:0/4121183310 wait complete. 2023-10-14T18:18:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.823+0000 7f13464a3640 1 Processor -- start 2023-10-14T18:18:27.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.823+0000 7f13464a3640 1 -- start start 2023-10-14T18:18:27.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400d0be0 0x7f13400cac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f1340152b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13400cb150 con 0x7f13400b8480 2023-10-14T18:18:27.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13400cb3f0 con 0x7f134015cd70 2023-10-14T18:18:27.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13464a3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13400cb6d0 con 0x7f13400d0be0 2023-10-14T18:18:27.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13454a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13454a1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49298/0 (socket says 172.21.15.29:49298) 2023-10-14T18:18:27.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.824+0000 7f13454a1640 1 -- 172.21.15.29:0/3720517413 learned_addr learned my addr 172.21.15.29:0/3720517413 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:27.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f1344ca0640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400d0be0 0x7f13400cac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f1345ca2640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f1340152b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 -- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400d0be0 msgr2=0x7f13400cac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400d0be0 0x7f13400cac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 -- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 msgr2=0x7f1340152b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:27.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f1340152b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:27.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 -- 172.21.15.29:0/3720517413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0094d0 con 0x7f13400b8480 2023-10-14T18:18:27.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.825+0000 7f13454a1640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f133c000c00 tx=0x7f133c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:27.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.826+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c039c90 con 0x7f13400b8480 2023-10-14T18:18:27.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.826+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c039e10 con 0x7f13400b8480 2023-10-14T18:18:27.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.826+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f133c043640 con 0x7f13400b8480 2023-10-14T18:18:27.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.826+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13400cb9b0 con 0x7f13400b8480 2023-10-14T18:18:27.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.826+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13400cbf70 con 0x7f13400b8480 2023-10-14T18:18:27.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.827+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f134004e9b0 con 0x7f13400b8480 2023-10-14T18:18:27.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.828+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f133c0437a0 con 0x7f13400b8480 2023-10-14T18:18:27.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.829+0000 7f1335ffb640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 0x7f132406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:27.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.830+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7f133c03e070 con 0x7f13400b8480 2023-10-14T18:18:27.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.830+0000 7f1344ca0640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 0x7f132406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:27.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.831+0000 7f1344ca0640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 0x7f132406d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13400c9600 tx=0x7f1330008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:27.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:27.835+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f133c0777c0 con 0x7f13400b8480 2023-10-14T18:18:28.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.030+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f13400d26e0 con 0x7f13400b8480 2023-10-14T18:18:28.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.037+0000 7f1335ffb640 1 -- 172.21.15.29:0/3720517413 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v82)=0 v82) v1 ==== 122+0+0 (secure 0 0 0) 0x7f133c0340c0 con 0x7f13400b8480 2023-10-14T18:18:28.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.040+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 msgr2=0x7f132406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.040+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 0x7f132406d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f13400c9600 tx=0x7f1330008040 comp rx=0 tx=0).stop 2023-10-14T18:18:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.040+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 msgr2=0x7f1340155b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.040+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 secure :-1 s=READY pgs=2837 cs=0 l=1 rev1=1 crypto rx=0x7f133c000c00 tx=0x7f133c004730 comp rx=0 tx=0).stop 2023-10-14T18:18:28.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 shutdown_connections 2023-10-14T18:18:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f132406b0b0 0x7f132406d570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f134015cd70 0x7f1340152b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13400d0be0 0x7f13400cac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 --2- 172.21.15.29:0/3720517413 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13400b8480 0x7f1340155b70 unknown :-1 s=CLOSED pgs=2837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 >> 172.21.15.29:0/3720517413 conn(0x7f13400cf570 msgr2=0x7f134006bbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 shutdown_connections 2023-10-14T18:18:28.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.041+0000 7f13464a3640 1 -- 172.21.15.29:0/3720517413 wait complete. 2023-10-14T18:18:28.062 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:18:28.063 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:18:28.167 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:18:28.288 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:18:28.289 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:28.289 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:28.354 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:18:28.354 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:18:28.355 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:28.417 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:18:28.418 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:28.419 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:28.483 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:18:28.484 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:18:28.484 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:28.545 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:18:28.545 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T18:18:28.546 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.781+0000 7ff9b25dd640 1 Processor -- start 2023-10-14T18:18:28.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.781+0000 7ff9b25dd640 1 -- start start 2023-10-14T18:18:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b25dd640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff9ac15c9f0 0x7ff9ac0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b25dd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9ac0c9230 0x7ff9ac0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b25dd640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9ac160a90 con 0x7ff9ac070f20 2023-10-14T18:18:28.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b25dd640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0cdec0 con 0x7ff9ac0c9230 2023-10-14T18:18:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b25dd640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9ac0cd900 con 0x7ff9ac15c9f0 2023-10-14T18:18:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b0dda640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff9ac0c9230 0x7ff9ac0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b1ddc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ff9ac070f20 0x7ff9ac0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38738/0 (socket says 172.21.15.29:38738) 2023-10-14T18:18:28.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b1ddc640 1 -- 172.21.15.29:0/3336505301 learned_addr learned my addr 172.21.15.29:0/3336505301 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:28.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.783+0000 7ff9b15db640 1 --2- 172.21.15.29:0/3336505301 >> v2:172.21.15.186:3300/0 conn(0x7ff9ac15c9f0 0x7ff9ac0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:28.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.784+0000 7ff9b0dda640 1 -- 172.21.15.29:0/3336505301 >> v2:172.21.15.186:3300/0 conn(0x7ff9ac15c9f0 msgr2=0x7ff9ac0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:28.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.784+0000 7ff9b0dda640 1 --2- 172.21.15.29:0/3336505301 >> v2:172.21.15.186:3300/0 conn(0x7ff9ac15c9f0 0x7ff9ac0c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.784+0000 7ff9b0dda640 1 -- 172.21.15.29:0/3336505301 >> v1:172.21.15.29:6789/0 conn(0x7ff9ac070f20 legacy=0x7ff9ac0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.784+0000 7ff9abfff640 1 -- 172.21.15.29:0/3336505301 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595693179 0 0) 0x7ff9ac160a90 con 0x7ff9ac070f20 2023-10-14T18:18:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.784+0000 7ff9b0dda640 1 -- 172.21.15.29:0/3336505301 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ac0cdbe0 con 0x7ff9ac0c9230 2023-10-14T18:18:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.785+0000 7ff9b0dda640 1 --2- 172.21.15.29:0/3336505301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac0c9230 0x7ff9ac0709e0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0ad4e0 tx=0x7ff99802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=321d5c5f02c2072a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.785+0000 7ff9abfff640 1 -- 172.21.15.29:0/3336505301 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998031830 con 0x7ff9ac0c9230 2023-10-14T18:18:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.785+0000 7ff9abfff640 1 -- 172.21.15.29:0/3336505301 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff998031990 con 0x7ff9ac0c9230 2023-10-14T18:18:28.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.785+0000 7ff9abfff640 1 -- 172.21.15.29:0/3336505301 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff998031d20 con 0x7ff9ac0c9230 2023-10-14T18:18:28.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 -- 172.21.15.29:0/3336505301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac0c9230 msgr2=0x7ff9ac0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/3336505301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac0c9230 0x7ff9ac0709e0 secure :-1 s=READY pgs=2839 cs=0 l=1 rev1=1 crypto rx=0x7ff9ac0ad4e0 tx=0x7ff99802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 -- 172.21.15.29:0/3336505301 shutdown_connections 2023-10-14T18:18:28.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/3336505301 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac0c9230 0x7ff9ac0709e0 unknown :-1 s=CLOSED pgs=2839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/3336505301 >> v2:172.21.15.186:3300/0 conn(0x7ff9ac15c9f0 0x7ff9ac0c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 -- 172.21.15.29:0/3336505301 >> 172.21.15.29:0/3336505301 conn(0x7ff9ac0ce6a0 msgr2=0x7ff9ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:28.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 -- 172.21.15.29:0/3336505301 shutdown_connections 2023-10-14T18:18:28.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.786+0000 7ff9b25dd640 1 -- 172.21.15.29:0/3336505301 wait complete. 2023-10-14T18:18:28.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.788+0000 7ff9b25dd640 1 Processor -- start 2023-10-14T18:18:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.788+0000 7ff9b25dd640 1 -- start start 2023-10-14T18:18:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9ac0c9230 0x7ff9ac069da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9ac15c9f0 0x7ff9ac0cc4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac1574b0 con 0x7ff9ac070f20 2023-10-14T18:18:28.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac06a6f0 con 0x7ff9ac15c9f0 2023-10-14T18:18:28.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b25dd640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ac06a9d0 con 0x7ff9ac0c9230 2023-10-14T18:18:28.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b15db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:28.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b15db640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49304/0 (socket says 172.21.15.29:49304) 2023-10-14T18:18:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.789+0000 7ff9b15db640 1 -- 172.21.15.29:0/1789540149 learned_addr learned my addr 172.21.15.29:0/1789540149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 -- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9ac0c9230 msgr2=0x7ff9ac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9ac0c9230 0x7ff9ac069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 -- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9ac15c9f0 msgr2=0x7ff9ac0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:28.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9ac15c9f0 0x7ff9ac0cc4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 -- 172.21.15.29:0/1789540149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff998009400 con 0x7ff9ac070f20 2023-10-14T18:18:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b15db640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0007e40 tx=0x7ff9a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a0014620 con 0x7ff9ac070f20 2023-10-14T18:18:28.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9a000ee90 con 0x7ff9ac070f20 2023-10-14T18:18:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9a00149c0 con 0x7ff9ac070f20 2023-10-14T18:18:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.790+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ac06acb0 con 0x7ff9ac070f20 2023-10-14T18:18:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.791+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ac06b270 con 0x7ff9ac070f20 2023-10-14T18:18:28.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.791+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ac06b400 con 0x7ff9ac070f20 2023-10-14T18:18:28.799 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.793+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff9a0014b20 con 0x7ff9ac070f20 2023-10-14T18:18:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.794+0000 7ff9a9ffb640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 0x7ff99006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.794+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7ff9a008a490 con 0x7ff9ac070f20 2023-10-14T18:18:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.794+0000 7ff9b0dda640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 0x7ff99006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:28.800 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.795+0000 7ff9b0dda640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 0x7ff99006d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff998036000 tx=0x7ff998030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:28.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:28.798+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff9a0055110 con 0x7ff9ac070f20 2023-10-14T18:18:29.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.031+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9ac04e9b0 con 0x7ff9ac070f20 2023-10-14T18:18:29.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.032+0000 7ff9a9ffb640 1 -- 172.21.15.29:0/1789540149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+3712 (secure 0 0 0) 0x7ff9a0058730 con 0x7ff9ac070f20 2023-10-14T18:18:29.033 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:29.034 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":117,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7280,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/1085345098","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":1085345098},{"type":"v1","addr":"172.21.15.186:6835","nonce":1085345098}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":8290,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1168588297","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1168588297},{"type":"v1","addr":"172.21.15.186:6833","nonce":1168588297}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":102},{"gid":11910,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/1839495929","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6836","nonce":1839495929},{"type":"v1","addr":"172.21.15.29:6837","nonce":1839495929}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":115},{"gid":12006,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6835/2122230403","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":2122230403},{"type":"v1","addr":"172.21.15.29:6835","nonce":2122230403}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":117}],"filesystems":[]} 2023-10-14T18:18:29.034 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 117 2023-10-14T18:18:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.035+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 msgr2=0x7ff99006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.035+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 0x7ff99006d570 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff998036000 tx=0x7ff998030000 comp rx=0 tx=0).stop 2023-10-14T18:18:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 msgr2=0x7ff9ac156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 secure :-1 s=READY pgs=2840 cs=0 l=1 rev1=1 crypto rx=0x7ff9a0007e40 tx=0x7ff9a000e4b0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 shutdown_connections 2023-10-14T18:18:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff99006b0b0 0x7ff99006d570 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff9ac15c9f0 0x7ff9ac0cc4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff9ac0c9230 0x7ff9ac069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 --2- 172.21.15.29:0/1789540149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff9ac070f20 0x7ff9ac156f70 unknown :-1 s=CLOSED pgs=2840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 >> 172.21.15.29:0/1789540149 conn(0x7ff9ac0ce6a0 msgr2=0x7ff9ac15da90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 shutdown_connections 2023-10-14T18:18:29.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.036+0000 7ff9b25dd640 1 -- 172.21.15.29:0/1789540149 wait complete. 2023-10-14T18:18:29.058 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T18:18:29.058 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T18:18:29.059 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T18:18:29.059 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:18:29.059 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T18:18:29.060 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T18:18:29.060 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T18:18:29.060 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:18:29.061 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T18:18:29.061 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T18:18:29.061 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T18:18:29.061 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:18:29.062 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T18:18:29.062 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T18:18:29.062 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T18:18:29.063 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:18:29.077 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T18:18:29.078 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T18:18:29.079 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T18:18:29.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T18:18:29.081 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T18:18:29.082 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T18:18:29.083 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T18:18:29.083 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T18:18:29.127 INFO:tasks.ceph.mds.c:Started 2023-10-14T18:18:29.128 DEBUG:teuthology.parallel:result is None 2023-10-14T18:18:29.131 INFO:tasks.ceph.mds.b:Started 2023-10-14T18:18:29.131 DEBUG:teuthology.parallel:result is None 2023-10-14T18:18:29.136 INFO:tasks.ceph.mds.a:Started 2023-10-14T18:18:29.136 DEBUG:teuthology.parallel:result is None 2023-10-14T18:18:29.138 INFO:tasks.ceph.mds.d:Started 2023-10-14T18:18:29.139 DEBUG:teuthology.parallel:result is None 2023-10-14T18:18:29.139 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T18:18:29.294 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T18:18:29.295 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:18:29.296 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T18:18:29.304 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:18:29.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.329+0000 7ff69887c640 1 Processor -- start 2023-10-14T18:18:29.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.330+0000 7ff69887c640 1 -- start start 2023-10-14T18:18:29.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69887c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff69415ad70 0x7ff694069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69887c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940c9ca0 0x7ff69406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69887c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6940cb030 con 0x7ff6940ca3b0 2023-10-14T18:18:29.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69887c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6940c91c0 con 0x7ff69415ad70 2023-10-14T18:18:29.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69887c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940c94a0 con 0x7ff6940c9ca0 2023-10-14T18:18:29.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff69415ad70 0x7ff694069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff69415ad70 0x7ff694069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49334/0 (socket says 172.21.15.29:49334) 2023-10-14T18:18:29.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff69259c640 1 -- 172.21.15.29:0/64489714 learned_addr learned my addr 172.21.15.29:0/64489714 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.331+0000 7ff68bfff640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940c9ca0 0x7ff69406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.332+0000 7ff69259c640 1 -- 172.21.15.29:0/64489714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940c9ca0 msgr2=0x7ff69406a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.332+0000 7ff69259c640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940c9ca0 0x7ff69406a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.332+0000 7ff69259c640 1 -- 172.21.15.29:0/64489714 >> v1:172.21.15.29:6789/0 conn(0x7ff6940ca3b0 legacy=0x7ff6940ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.332+0000 7ff69259c640 1 -- 172.21.15.29:0/64489714 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6940c9780 con 0x7ff69415ad70 2023-10-14T18:18:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.333+0000 7ff69259c640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff694069da0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7ff684009870 tx=0x7ff68402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e222a9bc9ae44015 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.333+0000 7ff691d9b640 1 -- 172.21.15.29:0/64489714 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684030840 con 0x7ff69415ad70 2023-10-14T18:18:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.333+0000 7ff691d9b640 1 -- 172.21.15.29:0/64489714 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6840309a0 con 0x7ff69415ad70 2023-10-14T18:18:29.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.333+0000 7ff691d9b640 1 -- 172.21.15.29:0/64489714 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff684030d30 con 0x7ff69415ad70 2023-10-14T18:18:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 -- 172.21.15.29:0/64489714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 msgr2=0x7ff694069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff694069da0 secure :-1 s=READY pgs=2846 cs=0 l=1 rev1=1 crypto rx=0x7ff684009870 tx=0x7ff68402b980 comp rx=0 tx=0).stop 2023-10-14T18:18:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 -- 172.21.15.29:0/64489714 shutdown_connections 2023-10-14T18:18:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940c9ca0 0x7ff69406a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 --2- 172.21.15.29:0/64489714 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff694069da0 unknown :-1 s=CLOSED pgs=2846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 -- 172.21.15.29:0/64489714 >> 172.21.15.29:0/64489714 conn(0x7ff6940ce9b0 msgr2=0x7ff6940cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 -- 172.21.15.29:0/64489714 shutdown_connections 2023-10-14T18:18:29.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.334+0000 7ff69887c640 1 -- 172.21.15.29:0/64489714 wait complete. 2023-10-14T18:18:29.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.336+0000 7ff69887c640 1 Processor -- start 2023-10-14T18:18:29.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.336+0000 7ff69887c640 1 -- start start 2023-10-14T18:18:29.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6940c9ca0 0x7ff69415dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940ca3b0 0x7ff6941f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940c4d70 con 0x7ff69415ad70 2023-10-14T18:18:29.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940c35e0 con 0x7ff6940c9ca0 2023-10-14T18:18:29.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69887c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6940c4130 con 0x7ff6940ca3b0 2023-10-14T18:18:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff692d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49344/0 (socket says 172.21.15.29:49344) 2023-10-14T18:18:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff692d9d640 1 -- 172.21.15.29:0/226440792 learned_addr learned my addr 172.21.15.29:0/226440792 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff68bfff640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940ca3b0 0x7ff6941f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.337+0000 7ff69259c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6940c9ca0 0x7ff69415dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 -- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940ca3b0 msgr2=0x7ff6941f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940ca3b0 0x7ff6941f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 -- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6940c9ca0 msgr2=0x7ff69415dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6940c9ca0 0x7ff69415dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:29.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 -- 172.21.15.29:0/226440792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6840094d0 con 0x7ff69415ad70 2023-10-14T18:18:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff692d9d640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c7d0 tx=0x7ff68c00cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c003b70 con 0x7ff69415ad70 2023-10-14T18:18:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff68c003cf0 con 0x7ff69415ad70 2023-10-14T18:18:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff68c00aa90 con 0x7ff69415ad70 2023-10-14T18:18:29.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6940b35c0 con 0x7ff69415ad70 2023-10-14T18:18:29.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.338+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6940c46b0 con 0x7ff69415ad70 2023-10-14T18:18:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.340+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6940c4eb0 con 0x7ff69415ad70 2023-10-14T18:18:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.341+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff68c00abf0 con 0x7ff69415ad70 2023-10-14T18:18:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.342+0000 7ff68affd640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 0x7ff67406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:29.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.342+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(241..241 src has 1..241) v4 ==== 5962+0+0 (secure 0 0 0) 0x7ff68c089770 con 0x7ff69415ad70 2023-10-14T18:18:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.342+0000 7ff69259c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 0x7ff67406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.343+0000 7ff69259c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 0x7ff67406d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff684036000 tx=0x7ff684008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:29.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.347+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff6940c4eb0 con 0x7ff69415ad70 2023-10-14T18:18:29.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:29.565+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7ff6940c4eb0 con 0x7ff69415ad70 2023-10-14T18:18:30.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.590+0000 7ff68affd640 1 -- 172.21.15.29:0/226440792 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v246) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff6940c4eb0 con 0x7ff69415ad70 2023-10-14T18:18:30.591 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T18:18:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.593+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 msgr2=0x7ff67406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.593+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 0x7ff67406d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff684036000 tx=0x7ff684008030 comp rx=0 tx=0).stop 2023-10-14T18:18:30.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.593+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 msgr2=0x7ff6940b3080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.593+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 secure :-1 s=READY pgs=2847 cs=0 l=1 rev1=1 crypto rx=0x7ff68c00c7d0 tx=0x7ff68c00cc90 comp rx=0 tx=0).stop 2023-10-14T18:18:30.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 shutdown_connections 2023-10-14T18:18:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff67406b0b0 0x7ff67406d570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff69415ad70 0x7ff6940b3080 unknown :-1 s=CLOSED pgs=2847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6940ca3b0 0x7ff6941f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 --2- 172.21.15.29:0/226440792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff6940c9ca0 0x7ff69415dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 >> 172.21.15.29:0/226440792 conn(0x7ff6940ce9b0 msgr2=0x7ff6940ad950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 shutdown_connections 2023-10-14T18:18:30.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.594+0000 7ff69887c640 1 -- 172.21.15.29:0/226440792 wait complete. 2023-10-14T18:18:30.618 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T18:18:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.847+0000 7fc9a435a640 1 Processor -- start 2023-10-14T18:18:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.848+0000 7fc9a435a640 1 -- start start 2023-10-14T18:18:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a435a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159b20 0x7fc99c06bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a435a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc99c0b0b20 0x7fc99c06fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a435a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc99c06f7f0 con 0x7fc99c159f00 2023-10-14T18:18:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a435a640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc99c06c5e0 con 0x7fc99c0b0b20 2023-10-14T18:18:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a435a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c06c8c0 con 0x7fc99c159b20 2023-10-14T18:18:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a28f7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc99c0b0b20 0x7fc99c06fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a28f7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc99c0b0b20 0x7fc99c06fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49356/0 (socket says 172.21.15.29:49356) 2023-10-14T18:18:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a18f5640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc99c159f00 0x7fc99c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38772/0 (socket says 172.21.15.29:38772) 2023-10-14T18:18:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.849+0000 7fc9a28f7640 1 -- 172.21.15.29:0/4259341801 learned_addr learned my addr 172.21.15.29:0/4259341801 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:30.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a20f6640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159b20 0x7fc99c06bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a28f7640 1 -- 172.21.15.29:0/4259341801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159b20 msgr2=0x7fc99c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a28f7640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159b20 0x7fc99c06bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a28f7640 1 -- 172.21.15.29:0/4259341801 >> v1:172.21.15.29:6789/0 conn(0x7fc99c159f00 legacy=0x7fc99c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a28f7640 1 -- 172.21.15.29:0/4259341801 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc99c06cba0 con 0x7fc99c0b0b20 2023-10-14T18:18:30.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.850+0000 7fc9a10f4640 1 -- 172.21.15.29:0/4259341801 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623780768 0 0) 0x7fc99c06f7f0 con 0x7fc99c159f00 2023-10-14T18:18:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.851+0000 7fc9a28f7640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c0b0b20 0x7fc99c06fc80 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40a1f7311050d00f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.851+0000 7fc9a10f4640 1 -- 172.21.15.29:0/4259341801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994030840 con 0x7fc99c0b0b20 2023-10-14T18:18:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.851+0000 7fc9a10f4640 1 -- 172.21.15.29:0/4259341801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9940309a0 con 0x7fc99c0b0b20 2023-10-14T18:18:30.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.851+0000 7fc9a10f4640 1 -- 172.21.15.29:0/4259341801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc994030d30 con 0x7fc99c0b0b20 2023-10-14T18:18:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 -- 172.21.15.29:0/4259341801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c0b0b20 msgr2=0x7fc99c06fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c0b0b20 0x7fc99c06fc80 secure :-1 s=READY pgs=2849 cs=0 l=1 rev1=1 crypto rx=0x7fc994009870 tx=0x7fc99402b980 comp rx=0 tx=0).stop 2023-10-14T18:18:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 -- 172.21.15.29:0/4259341801 shutdown_connections 2023-10-14T18:18:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c0b0b20 0x7fc99c06fc80 unknown :-1 s=CLOSED pgs=2849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 --2- 172.21.15.29:0/4259341801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159b20 0x7fc99c06bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 -- 172.21.15.29:0/4259341801 >> 172.21.15.29:0/4259341801 conn(0x7fc99c0cb410 msgr2=0x7fc99c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 -- 172.21.15.29:0/4259341801 shutdown_connections 2023-10-14T18:18:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.852+0000 7fc9a435a640 1 -- 172.21.15.29:0/4259341801 wait complete. 2023-10-14T18:18:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.854+0000 7fc9a435a640 1 Processor -- start 2023-10-14T18:18:30.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.854+0000 7fc9a435a640 1 -- start start 2023-10-14T18:18:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 0x7fc99c1541a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159f00 0x7fc99c0cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0c5550 con 0x7fc99c159b20 2023-10-14T18:18:30.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c0ae620 con 0x7fc99c0b0b20 2023-10-14T18:18:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a435a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc99c1546e0 con 0x7fc99c159f00 2023-10-14T18:18:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a18f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a18f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49362/0 (socket says 172.21.15.29:49362) 2023-10-14T18:18:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.855+0000 7fc9a18f5640 1 -- 172.21.15.29:0/1324057104 learned_addr learned my addr 172.21.15.29:0/1324057104 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:30.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a20f6640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 0x7fc99c1541a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 -- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159f00 msgr2=0x7fc99c0cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a28f7640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159f00 0x7fc99c0cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159f00 0x7fc99c0cee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 -- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 msgr2=0x7fc99c1541a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:30.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 0x7fc99c1541a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 -- 172.21.15.29:0/1324057104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9940094d0 con 0x7fc99c159b20 2023-10-14T18:18:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a20f6640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 0x7fc99c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a18f5640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c440 tx=0x7fc99000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9900074e0 con 0x7fc99c159b20 2023-10-14T18:18:30.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc990007640 con 0x7fc99c159b20 2023-10-14T18:18:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc990004a20 con 0x7fc99c159b20 2023-10-14T18:18:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.856+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc99c15d3a0 con 0x7fc99c159b20 2023-10-14T18:18:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.857+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc99c0c4cb0 con 0x7fc99c159b20 2023-10-14T18:18:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.858+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc99c04e160 con 0x7fc99c159b20 2023-10-14T18:18:30.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.859+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc990004b80 con 0x7fc99c159b20 2023-10-14T18:18:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.860+0000 7fc99affd640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 0x7fc98806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:30.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.860+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fc990088900 con 0x7fc99c159b20 2023-10-14T18:18:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.861+0000 7fc9a20f6640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 0x7fc98806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.861+0000 7fc9a20f6640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 0x7fc98806d5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc98c007f50 tx=0x7fc98c0074a0 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:30.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:30.865+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc990053930 con 0x7fc99c159b20 2023-10-14T18:18:31.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.132+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc99c0ae760 con 0x7fc99c159b20 2023-10-14T18:18:31.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.133+0000 7fc99affd640 1 -- 172.21.15.29:0/1324057104 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v51) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fc990018070 con 0x7fc99c159b20 2023-10-14T18:18:31.134 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:31.135 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:18:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T18:18:31.135 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T18:18:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T18:18:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.136 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:18:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:18:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.137 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T18:18:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T18:18:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.138 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:18:31.139 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:18:31.139 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.139 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.140 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.140 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.140 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T18:18:31.140 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T18:18:31.141 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:18:31.141 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:18:31.141 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.142 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.142 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.142 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.142 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T18:18:31.142 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T18:18:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:18:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:18:31.143 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.144 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.144 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.144 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.144 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.145 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.146 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.146 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.146 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.146 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T18:18:31.146 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T18:18:31.147 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.147 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.147 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.148 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.148 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.148 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.148 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T18:18:31.148 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T18:18:31.149 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.149 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.149 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.149 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.150 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.150 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.150 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T18:18:31.150 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T18:18:31.151 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.151 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.151 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.151 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.151 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.152 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.152 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.152 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T18:18:31.152 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T18:18:31.152 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.153 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.153 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.153 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.153 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.153 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.154 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.154 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T18:18:31.154 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T18:18:31.154 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.155 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.156 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.156 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T18:18:31.156 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T18:18:31.156 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.156 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.157 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.157 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.157 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.157 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.158 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.158 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T18:18:31.158 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T18:18:31.158 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.158 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:18:31.159 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:18:31.159 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.159 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.159 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.160 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.160 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T18:18:31.160 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T18:18:31.160 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.160 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.161 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T18:18:31.161 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T18:18:31.161 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.161 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.161 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.162 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T18:18:31.162 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T18:18:31.162 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.162 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.163 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:18:31.163 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:18:31.163 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:18:31.163 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T18:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T18:18:31.164 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.165 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.165 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:18:31.165 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:18:31.165 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:18:31.166 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.166 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.166 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.166 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T18:18:31.166 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T18:18:31.167 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.167 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:18:31.167 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:18:31.168 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:18:31.168 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:18:31.168 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.168 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.168 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T18:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T18:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:18:31.169 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:18:31.170 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:18:31.170 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.170 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.170 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.171 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.171 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T18:18:31.171 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T18:18:31.171 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.171 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:18:31.172 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:18:31.172 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.172 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.172 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:31.172 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:31.173 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T18:18:31.173 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T18:18:31.173 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:18:31.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:18:31.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:18:31.174 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:18:31.174 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.174 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:31.174 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T18:18:31.175 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:18:31.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 msgr2=0x7fc98806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 0x7fc98806d5a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc98c007f50 tx=0x7fc98c0074a0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 msgr2=0x7fc99c0c44b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 secure :-1 s=READY pgs=2850 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c440 tx=0x7fc99000c900 comp rx=0 tx=0).stop 2023-10-14T18:18:31.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 shutdown_connections 2023-10-14T18:18:31.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc98806b0e0 0x7fc98806d5a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc99c159f00 0x7fc99c0cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc99c159b20 0x7fc99c0c44b0 unknown :-1 s=CLOSED pgs=2850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 --2- 172.21.15.29:0/1324057104 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc99c0b0b20 0x7fc99c1541a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 >> 172.21.15.29:0/1324057104 conn(0x7fc99c0cb410 msgr2=0x7fc99c0cc560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:31.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 shutdown_connections 2023-10-14T18:18:31.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.136+0000 7fc9a435a640 1 -- 172.21.15.29:0/1324057104 wait complete. 2023-10-14T18:18:31.180 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.408+0000 7f1cc9e02640 1 Processor -- start 2023-10-14T18:18:31.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.408+0000 7f1cc9e02640 1 -- start start 2023-10-14T18:18:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1cc415aaa0 0x7f1cc415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc40ccaa0 0x7f1cc4070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc406b810 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cc415ed60 con 0x7f1cc415aaa0 2023-10-14T18:18:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc9e02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc40707e0 con 0x7f1cc415ae80 2023-10-14T18:18:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc40ccaa0 0x7f1cc4070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1cc40ccaa0 0x7f1cc4070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54538/0 (socket says 172.21.15.29:54538) 2023-10-14T18:18:31.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.410+0000 7f1cc3fff640 1 -- 172.21.15.29:0/2034728137 learned_addr learned my addr 172.21.15.29:0/2034728137 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:31.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc37fe640 1 --2- 172.21.15.29:0/2034728137 >> v2:172.21.15.186:3300/0 conn(0x7f1cc415aaa0 0x7f1cc415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cbbfff640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc3fff640 1 -- 172.21.15.29:0/2034728137 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 msgr2=0x7f1cc40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc3fff640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc3fff640 1 -- 172.21.15.29:0/2034728137 >> v2:172.21.15.186:3300/0 conn(0x7f1cc415aaa0 msgr2=0x7f1cc415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc3fff640 1 --2- 172.21.15.29:0/2034728137 >> v2:172.21.15.186:3300/0 conn(0x7f1cc415aaa0 0x7f1cc415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.411+0000 7f1cc3fff640 1 -- 172.21.15.29:0/2034728137 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cc415ef90 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.412+0000 7f1cc3fff640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc4070f50 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40cd990 tx=0x7f1cbc02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=762c8d11f512beaa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.413+0000 7f1cc2ffd640 1 -- 172.21.15.29:0/2034728137 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc02ca80 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.413+0000 7f1cc2ffd640 1 -- 172.21.15.29:0/2034728137 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cbc02cbe0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.413+0000 7f1cc2ffd640 1 -- 172.21.15.29:0/2034728137 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cbc0343f0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.413+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2034728137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 msgr2=0x7f1cc4070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.413+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc4070f50 secure :-1 s=READY pgs=2851 cs=0 l=1 rev1=1 crypto rx=0x7f1cc40cd990 tx=0x7f1cbc02bee0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2034728137 shutdown_connections 2023-10-14T18:18:31.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc4070f50 unknown :-1 s=CLOSED pgs=2851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2034728137 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2034728137 >> v2:172.21.15.186:3300/0 conn(0x7f1cc415aaa0 0x7f1cc415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2034728137 >> 172.21.15.29:0/2034728137 conn(0x7f1cc40cb410 msgr2=0x7f1cc40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2034728137 shutdown_connections 2023-10-14T18:18:31.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.414+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2034728137 wait complete. 2023-10-14T18:18:31.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.416+0000 7f1cc9e02640 1 Processor -- start 2023-10-14T18:18:31.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.416+0000 7f1cc9e02640 1 -- start start 2023-10-14T18:18:31.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc415aaa0 0x7f1cc40ac9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40acef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc414a250 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc41f36a0 con 0x7f1cc415aaa0 2023-10-14T18:18:31.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc9e02640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc4156aa0 con 0x7f1cc415ae80 2023-10-14T18:18:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc37fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54544/0 (socket says 172.21.15.29:54544) 2023-10-14T18:18:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.417+0000 7f1cc37fe640 1 -- 172.21.15.29:0/2094995832 learned_addr learned my addr 172.21.15.29:0/2094995832 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cbbfff640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc415aaa0 0x7f1cc40ac9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc3fff640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40acef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 -- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 msgr2=0x7f1cc40acef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40acef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 -- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc415aaa0 msgr2=0x7f1cc40ac9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc415aaa0 0x7f1cc40ac9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 -- 172.21.15.29:0/2094995832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc009b30 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc37fe640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400d830 tx=0x7f1cb400dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb400be10 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cb40098b0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.418+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4009a40 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.419+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc41f33c0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.419+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc4149e70 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.420+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc404e160 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.421+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1cb4009ba0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.422+0000 7f1cc0ff9640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 0x7f1ca406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.422+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f1cb4089b60 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.422+0000 7f1cbbfff640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 0x7f1ca406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.423+0000 7f1cbbfff640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 0x7f1ca406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1cac006fd0 tx=0x7f1cac008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.428 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.427+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1cb4054b80 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.622+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1cc40cdc60 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.624+0000 7f1cc0ff9640 1 -- 172.21.15.29:0/2094995832 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v246) v1 ==== 74+0+13041 (secure 0 0 0) 0x7f1cb400b3a0 con 0x7f1cc40ccaa0 2023-10-14T18:18:31.625 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:31.625 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":246,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:30.586762+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":29,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":231,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:31.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 msgr2=0x7f1ca406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 0x7f1ca406d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1cac006fd0 tx=0x7f1cac008040 comp rx=0 tx=0).stop 2023-10-14T18:18:31.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 msgr2=0x7f1cc415dc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 secure :-1 s=READY pgs=2852 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400d830 tx=0x7f1cb400dcf0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 shutdown_connections 2023-10-14T18:18:31.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1ca406b0b0 0x7f1ca406d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1cc415ae80 0x7f1cc40acef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1cc415aaa0 0x7f1cc40ac9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 --2- 172.21.15.29:0/2094995832 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1cc40ccaa0 0x7f1cc415dc70 unknown :-1 s=CLOSED pgs=2852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 >> 172.21.15.29:0/2094995832 conn(0x7f1cc40cb410 msgr2=0x7f1cc40b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:31.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.626+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 shutdown_connections 2023-10-14T18:18:31.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.627+0000 7f1cc9e02640 1 -- 172.21.15.29:0/2094995832 wait complete. 2023-10-14T18:18:31.646 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:31.647 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:31.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.877+0000 7fdf4b59e640 1 Processor -- start 2023-10-14T18:18:31.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.877+0000 7fdf4b59e640 1 -- start start 2023-10-14T18:18:31.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.878+0000 7fdf4b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.878+0000 7fdf4b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fdf4c0b2080 0x7fdf4c0ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf4c0d0970 con 0x7fdf4c0ce960 2023-10-14T18:18:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c0d0ea0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf4c0d0bc0 con 0x7fdf4c0b2080 2023-10-14T18:18:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54554/0 (socket says 172.21.15.29:54554) 2023-10-14T18:18:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1420373766 learned_addr learned my addr 172.21.15.29:0/1420373766 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.879+0000 7fdf41d9b640 1 --2- 172.21.15.29:0/1420373766 >> v2:172.21.15.186:3300/0 conn(0x7fdf4c0b2080 0x7fdf4c0ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.880+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1420373766 >> v2:172.21.15.186:3300/0 conn(0x7fdf4c0b2080 msgr2=0x7fdf4c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:31.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.880+0000 7fdf4a59c640 1 --2- 172.21.15.29:0/1420373766 >> v2:172.21.15.186:3300/0 conn(0x7fdf4c0b2080 0x7fdf4c0ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.880+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1420373766 >> v1:172.21.15.186:6789/0 conn(0x7fdf4c0ce960 legacy=0x7fdf4c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.880+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1420373766 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4c0cd940 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.881+0000 7fdf4a59c640 1 --2- 172.21.15.29:0/1420373766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0671e0 tx=0x7fdf3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90d45fd50a7012ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.881+0000 7fdf49d9b640 1 -- 172.21.15.29:0/1420373766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c030860 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.881+0000 7fdf49d9b640 1 -- 172.21.15.29:0/1420373766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c0309c0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.881+0000 7fdf49d9b640 1 -- 172.21.15.29:0/1420373766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c030d50 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1420373766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 msgr2=0x7fdf4c0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1420373766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 secure :-1 s=READY pgs=2853 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c0671e0 tx=0x7fdf3c02b9a0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1420373766 shutdown_connections 2023-10-14T18:18:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1420373766 >> v2:172.21.15.186:3300/0 conn(0x7fdf4c0b2080 0x7fdf4c0ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1420373766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0d36b0 unknown :-1 s=CLOSED pgs=2853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1420373766 >> 172.21.15.29:0/1420373766 conn(0x7fdf4c0d2b90 msgr2=0x7fdf4c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:31.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1420373766 shutdown_connections 2023-10-14T18:18:31.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.882+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1420373766 wait complete. 2023-10-14T18:18:31.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.884+0000 7fdf4b59e640 1 Processor -- start 2023-10-14T18:18:31.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.884+0000 7fdf4b59e640 1 -- start start 2023-10-14T18:18:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 0x7fdf4c0b80b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 0x7fdf4c1f2590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c15ee30 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0b6e50 con 0x7fdf4c0b2080 2023-10-14T18:18:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf4c0b6240 con 0x7fdf4c0ce960 2023-10-14T18:18:31.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54564/0 (socket says 172.21.15.29:54564) 2023-10-14T18:18:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1255401957 learned_addr learned my addr 172.21.15.29:0/1255401957 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf41d9b640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 0x7fdf4c0b80b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4ad9d640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 0x7fdf4c1f2590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 msgr2=0x7fdf4c1f2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 0x7fdf4c1f2590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 msgr2=0x7fdf4c0b80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:31.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 0x7fdf4c0b80b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:31.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf4a59c640 1 -- 172.21.15.29:0/1255401957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3c0094d0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.885+0000 7fdf41d9b640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 0x7fdf4c0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf4ad9d640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 0x7fdf4c1f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf4a59c640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c02b9e0 tx=0x7fdf3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c039b60 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf3c039ce0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c043650 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf4c0b6520 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.886+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf4c0b6ae0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.888+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf4c04e9b0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.889+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fdf3c02f3b0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.889+0000 7fdf437fe640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 0x7fdf2c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:31.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.890+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fdf3c0abb20 con 0x7fdf4c0b1ca0 2023-10-14T18:18:31.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.890+0000 7fdf41d9b640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 0x7fdf2c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:31.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.890+0000 7fdf41d9b640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 0x7fdf2c06d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf34006fd0 tx=0x7fdf34008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:31.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:31.894+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdf3c076a30 con 0x7fdf4c0b1ca0 2023-10-14T18:18:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.089+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fdf4c0ce720 con 0x7fdf4c0b1ca0 2023-10-14T18:18:32.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.090+0000 7fdf437fe640 1 -- 172.21.15.29:0/1255401957 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v246) v1 ==== 93+0+10 (secure 0 0 0) 0x7fdf3c0480a0 con 0x7fdf4c0b1ca0 2023-10-14T18:18:32.091 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:32.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 msgr2=0x7fdf2c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 0x7fdf2c06d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fdf34006fd0 tx=0x7fdf34008040 comp rx=0 tx=0).stop 2023-10-14T18:18:32.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 msgr2=0x7fdf4c0c8050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 secure :-1 s=READY pgs=2854 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c02b9e0 tx=0x7fdf3c004730 comp rx=0 tx=0).stop 2023-10-14T18:18:32.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 shutdown_connections 2023-10-14T18:18:32.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fdf2c06b0b0 0x7fdf2c06d570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdf4c0ce960 0x7fdf4c1f2590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdf4c0b2080 0x7fdf4c0b80b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 --2- 172.21.15.29:0/1255401957 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdf4c0b1ca0 0x7fdf4c0c8050 unknown :-1 s=CLOSED pgs=2854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.092+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 >> 172.21.15.29:0/1255401957 conn(0x7fdf4c0d2b90 msgr2=0x7fdf4c0d1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:32.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.093+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 shutdown_connections 2023-10-14T18:18:32.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.093+0000 7fdf4b59e640 1 -- 172.21.15.29:0/1255401957 wait complete. 2023-10-14T18:18:32.114 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:18:32.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.330+0000 7fc6b176c640 1 Processor -- start 2023-10-14T18:18:32.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.330+0000 7fc6b176c640 1 -- start start 2023-10-14T18:18:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.332+0000 7fc6b176c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac070b60 0x7fc6ac15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.332+0000 7fc6b176c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6ac06c100 0x7fc6ac06c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.332+0000 7fc6b176c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6ac15aef0 con 0x7fc6ac159950 2023-10-14T18:18:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.332+0000 7fc6b176c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6ac15e490 con 0x7fc6ac06c100 2023-10-14T18:18:32.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.332+0000 7fc6b176c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac15d910 con 0x7fc6ac070b60 2023-10-14T18:18:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.333+0000 7fc6b0f6b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6ac06c100 0x7fc6ac06c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.333+0000 7fc6a3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac070b60 0x7fc6ac15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.333+0000 7fc6b0f6b640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc6ac06c100 0x7fc6ac06c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:34202/0 (socket says 172.21.15.29:34202) 2023-10-14T18:18:32.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.333+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2150682596 learned_addr learned my addr 172.21.15.29:0/2150682596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.334+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac070b60 msgr2=0x7fc6ac15a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.334+0000 7fc6b0f6b640 1 --2- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac070b60 0x7fc6ac15a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.334+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2150682596 >> v1:172.21.15.186:6789/0 conn(0x7fc6ac159950 legacy=0x7fc6ac070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.334+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2150682596 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6ac15dbf0 con 0x7fc6ac06c100 2023-10-14T18:18:32.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.335+0000 7fc6b0f6b640 1 --2- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac06c100 0x7fc6ac06c550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a402bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ba702035450ba4e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.335+0000 7fc6ab7fe640 1 -- 172.21.15.29:0/2150682596 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a402ca70 con 0x7fc6ac06c100 2023-10-14T18:18:32.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6ab7fe640 1 -- 172.21.15.29:0/2150682596 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a402cbd0 con 0x7fc6ac06c100 2023-10-14T18:18:32.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6ab7fe640 1 -- 172.21.15.29:0/2150682596 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a40343f0 con 0x7fc6ac06c100 2023-10-14T18:18:32.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 -- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac06c100 msgr2=0x7fc6ac06c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac06c100 0x7fc6ac06c550 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4009e50 tx=0x7fc6a402bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 -- 172.21.15.29:0/2150682596 shutdown_connections 2023-10-14T18:18:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac070b60 0x7fc6ac15a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2150682596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac06c100 0x7fc6ac06c550 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 -- 172.21.15.29:0/2150682596 >> 172.21.15.29:0/2150682596 conn(0x7fc6ac0ce6a0 msgr2=0x7fc6ac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 -- 172.21.15.29:0/2150682596 shutdown_connections 2023-10-14T18:18:32.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.336+0000 7fc6b176c640 1 -- 172.21.15.29:0/2150682596 wait complete. 2023-10-14T18:18:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.338+0000 7fc6b176c640 1 Processor -- start 2023-10-14T18:18:32.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.338+0000 7fc6b176c640 1 -- start start 2023-10-14T18:18:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.339+0000 7fc6b176c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac06c100 0x7fc6ac158ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.339+0000 7fc6b176c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac070b60 0x7fc6ac1f57f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.339+0000 7fc6b176c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b176c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1f6720 con 0x7fc6ac159950 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b176c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac1f21b0 con 0x7fc6ac070b60 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b176c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6ac158ff0 con 0x7fc6ac06c100 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54580/0 (socket says 172.21.15.29:54580) 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2960563561 learned_addr learned my addr 172.21.15.29:0/2960563561 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:32.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6abfff640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac06c100 0x7fc6ac158ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6a3fff640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac070b60 0x7fc6ac1f57f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac06c100 msgr2=0x7fc6ac158ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac06c100 0x7fc6ac158ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac070b60 msgr2=0x7fc6ac1f57f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac070b60 0x7fc6ac1f57f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.340+0000 7fc6b0f6b640 1 -- 172.21.15.29:0/2960563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6a4009b30 con 0x7fc6ac159950 2023-10-14T18:18:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6b0f6b640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4036040 tx=0x7fc6a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a4009510 con 0x7fc6ac159950 2023-10-14T18:18:32.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6a403e450 con 0x7fc6ac159950 2023-10-14T18:18:32.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6a403e800 con 0x7fc6ac159950 2023-10-14T18:18:32.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6ac1f6160 con 0x7fc6ac159950 2023-10-14T18:18:32.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.341+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6ac1f2410 con 0x7fc6ac159950 2023-10-14T18:18:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.342+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6ac04e9b0 con 0x7fc6ac159950 2023-10-14T18:18:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.343+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc6a403e960 con 0x7fc6ac159950 2023-10-14T18:18:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.344+0000 7fc6a97fa640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 0x7fc68c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.344+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fc6a40ac4f0 con 0x7fc6ac159950 2023-10-14T18:18:32.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.345+0000 7fc6abfff640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 0x7fc68c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.345+0000 7fc6abfff640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 0x7fc68c06d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc69c000f60 tx=0x7fc69c009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.349+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc6a4077480 con 0x7fc6ac159950 2023-10-14T18:18:32.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.556+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fc6ac0b4620 con 0x7fc6ac159950 2023-10-14T18:18:32.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.557+0000 7fc6a97fa640 1 -- 172.21.15.29:0/2960563561 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v246) v1 ==== 84+0+65 (secure 0 0 0) 0x7fc6a407aaa0 con 0x7fc6ac159950 2023-10-14T18:18:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:32.558 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:18:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:18:32.558 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:18:32.559 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:32.559 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:18:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 msgr2=0x7fc68c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 0x7fc68c06d570 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc69c000f60 tx=0x7fc69c009040 comp rx=0 tx=0).stop 2023-10-14T18:18:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 msgr2=0x7fc6ac0ae830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 secure :-1 s=READY pgs=2855 cs=0 l=1 rev1=1 crypto rx=0x7fc6a4036040 tx=0x7fc6a4004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 shutdown_connections 2023-10-14T18:18:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc68c06b0b0 0x7fc68c06d570 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc6ac159950 0x7fc6ac0ae830 unknown :-1 s=CLOSED pgs=2855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc6ac070b60 0x7fc6ac1f57f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 --2- 172.21.15.29:0/2960563561 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc6ac06c100 0x7fc6ac158ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 >> 172.21.15.29:0/2960563561 conn(0x7fc6ac0ce6a0 msgr2=0x7fc6ac0cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 shutdown_connections 2023-10-14T18:18:32.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.560+0000 7fc6b176c640 1 -- 172.21.15.29:0/2960563561 wait complete. 2023-10-14T18:18:32.581 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T18:18:32.581 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T18:18:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.809+0000 7f86e4133640 1 Processor -- start 2023-10-14T18:18:32.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.809+0000 7f86e4133640 1 -- start start 2023-10-14T18:18:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e4133640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 0x7f86dc165aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e4133640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f86dc0b0360 0x7f86dc0b07d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e4133640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86dc165610 con 0x7f86dc0c1d30 2023-10-14T18:18:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e4133640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86dc0b0d10 con 0x7f86dc0b0360 2023-10-14T18:18:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e4133640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0ce370 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e1ecf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f86dc0c1d30 0x7f86dc0cda90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38284/0 (socket says 172.21.15.29:38284) 2023-10-14T18:18:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.811+0000 7f86e1ecf640 1 -- 172.21.15.29:0/1611721921 learned_addr learned my addr 172.21.15.29:0/1611721921 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:32.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1611721921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 0x7f86dc165aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e26d0640 1 --2- 172.21.15.29:0/1611721921 >> v2:172.21.15.186:3300/0 conn(0x7f86dc0b0360 0x7f86dc0b07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009342224 0 0) 0x7f86dc165610 con 0x7f86dc0c1d30 2023-10-14T18:18:32.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86c8003520 con 0x7f86dc0c1d30 2023-10-14T18:18:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e16ce640 1 -- 172.21.15.29:0/1611721921 >> v2:172.21.15.186:3300/0 conn(0x7f86dc0b0360 msgr2=0x7f86dc0b07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1611721921 >> v2:172.21.15.186:3300/0 conn(0x7f86dc0b0360 0x7f86dc0b07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 601517122 0 0) 0x7f86c8003520 con 0x7f86dc0c1d30 2023-10-14T18:18:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e16ce640 1 -- 172.21.15.29:0/1611721921 >> v1:172.21.15.29:6789/0 conn(0x7f86dc0c1d30 legacy=0x7f86dc0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.812+0000 7f86e16ce640 1 -- 172.21.15.29:0/1611721921 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86dc0ce650 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.814+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1611721921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 0x7f86dc165aa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097a0 tx=0x7f86cc02b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=728abe2b9b6cf8cf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.814+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86d8001670 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.814+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc02e050 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.814+0000 7f86e0ecd640 1 -- 172.21.15.29:0/1611721921 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc030730 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 -- 172.21.15.29:0/1611721921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 msgr2=0x7f86dc165aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 --2- 172.21.15.29:0/1611721921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 0x7f86dc165aa0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f86cc0097a0 tx=0x7f86cc02b790 comp rx=0 tx=0).stop 2023-10-14T18:18:32.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 -- 172.21.15.29:0/1611721921 shutdown_connections 2023-10-14T18:18:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 --2- 172.21.15.29:0/1611721921 >> v2:172.21.15.186:3300/0 conn(0x7f86dc0b0360 0x7f86dc0b07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 --2- 172.21.15.29:0/1611721921 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0c8fa0 0x7f86dc165aa0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 -- 172.21.15.29:0/1611721921 >> 172.21.15.29:0/1611721921 conn(0x7f86dc0cb360 msgr2=0x7f86dc0cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:32.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 -- 172.21.15.29:0/1611721921 shutdown_connections 2023-10-14T18:18:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.815+0000 7f86e4133640 1 -- 172.21.15.29:0/1611721921 wait complete. 2023-10-14T18:18:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.817+0000 7f86e4133640 1 Processor -- start 2023-10-14T18:18:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.817+0000 7f86e4133640 1 -- start start 2023-10-14T18:18:32.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0b0360 0x7f86dc0c3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86dc0c8fa0 0x7f86dc1faee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0c4ce0 con 0x7f86dc0c1d30 2023-10-14T18:18:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0c40a0 con 0x7f86dc0c8fa0 2023-10-14T18:18:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.818+0000 7f86e4133640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86dc0c4340 con 0x7f86dc0b0360 2023-10-14T18:18:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54592/0 (socket says 172.21.15.29:54592) 2023-10-14T18:18:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 -- 172.21.15.29:0/1285126778 learned_addr learned my addr 172.21.15.29:0/1285126778 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e1ecf640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0b0360 0x7f86dc0c3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 -- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0b0360 msgr2=0x7f86dc0c3620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0b0360 0x7f86dc0c3620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 -- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86dc0c8fa0 msgr2=0x7f86dc1faee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86dc0c8fa0 0x7f86dc1faee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 -- 172.21.15.29:0/1285126778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86cc009400 con 0x7f86dc0c1d30 2023-10-14T18:18:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.819+0000 7f86e16ce640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f86cc035040 tx=0x7f86cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.820+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc006e10 con 0x7f86dc0c1d30 2023-10-14T18:18:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.820+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86cc03c620 con 0x7f86dc0c1d30 2023-10-14T18:18:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.820+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86cc03c9f0 con 0x7f86dc0c1d30 2023-10-14T18:18:32.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.820+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86dc0c4620 con 0x7f86dc0c1d30 2023-10-14T18:18:32.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.820+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86dc0c3b60 con 0x7f86dc0c1d30 2023-10-14T18:18:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.821+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86dc0c4e20 con 0x7f86dc0c1d30 2023-10-14T18:18:32.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.822+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f86cc039070 con 0x7f86dc0c1d30 2023-10-14T18:18:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.823+0000 7f86d27fc640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 0x7f86c006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.823+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(246..246 src has 1..246) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f86cc0aad80 con 0x7f86dc0c1d30 2023-10-14T18:18:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.824+0000 7f86e1ecf640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 0x7f86c006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:32.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.824+0000 7f86e1ecf640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 0x7f86c006d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f86d8001af0 tx=0x7f86d8009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:32.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:32.828+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f86cc075d60 con 0x7f86dc0c1d30 2023-10-14T18:18:33.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.026+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f86dc04e160 con 0x7f86dc0c1d30 2023-10-14T18:18:33.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.608+0000 7f86d27fc640 1 -- 172.21.15.29:0/1285126778 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v247) v1 ==== 138+0+0 (secure 0 0 0) 0x7f86cc0370b0 con 0x7f86dc0c1d30 2023-10-14T18:18:33.610 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T18:18:33.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 msgr2=0x7f86c006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 0x7f86c006d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f86d8001af0 tx=0x7f86d8009040 comp rx=0 tx=0).stop 2023-10-14T18:18:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 msgr2=0x7f86dc1fa9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 secure :-1 s=READY pgs=2857 cs=0 l=1 rev1=1 crypto rx=0x7f86cc035040 tx=0x7f86cc0046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 shutdown_connections 2023-10-14T18:18:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f86c006b0b0 0x7f86c006d570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86dc0c8fa0 0x7f86dc1faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86dc0c1d30 0x7f86dc1fa9a0 unknown :-1 s=CLOSED pgs=2857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 --2- 172.21.15.29:0/1285126778 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86dc0b0360 0x7f86dc0c3620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.612+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 >> 172.21.15.29:0/1285126778 conn(0x7f86dc0cb360 msgr2=0x7f86dc0ccf50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.613+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 shutdown_connections 2023-10-14T18:18:33.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.613+0000 7f86e4133640 1 -- 172.21.15.29:0/1285126778 wait complete. 2023-10-14T18:18:33.641 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:18:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.863+0000 7f1670b78640 1 Processor -- start 2023-10-14T18:18:33.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.863+0000 7f1670b78640 1 -- start start 2023-10-14T18:18:33.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1670b78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f166c164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1670b78640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c06f890 con 0x7f166c159f00 2023-10-14T18:18:33.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1670b78640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166c06c650 con 0x7f166c159b20 2023-10-14T18:18:33.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1670b78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c070d30 con 0x7f166c0b0b20 2023-10-14T18:18:33.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1669d9b640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f166c159f00 0x7f166c0b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38292/0 (socket says 172.21.15.29:38292) 2023-10-14T18:18:33.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.865+0000 7f1669d9b640 1 -- 172.21.15.29:0/1155730976 learned_addr learned my addr 172.21.15.29:0/1155730976 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166ad9d640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f166c164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177468311 0 0) 0x7f166c06f890 con 0x7f166c159f00 2023-10-14T18:18:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1654003520 con 0x7f166c159f00 2023-10-14T18:18:33.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1338542788 0 0) 0x7f1654003520 con 0x7f166c159f00 2023-10-14T18:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 msgr2=0x7f166c164b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f166c164b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 >> v1:172.21.15.186:6789/0 conn(0x7f166c159b20 legacy=0x7f166c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c06c930 con 0x7f166c159f00 2023-10-14T18:18:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1660001670 con 0x7f166c159f00 2023-10-14T18:18:33.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.866+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 >> v1:172.21.15.29:6789/0 conn(0x7f166c159f00 legacy=0x7f166c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f1654003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f16540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1654004d00 0x7f1654005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165400ba30 con 0x7f16540039a0 2023-10-14T18:18:33.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f165400bb70 con 0x7f1654004d00 2023-10-14T18:18:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f165400df10 con 0x7f166c0b0b20 2023-10-14T18:18:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166a59c640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f1654003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f1669d9b640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f16540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166ad9d640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1654004d00 0x7f1654005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166a59c640 1 -- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 msgr2=0x7f16540047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:33.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.867+0000 7f166a59c640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c0b0b20 0x7f16540047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.868+0000 7f166a59c640 1 -- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1654004d00 msgr2=0x7f1654005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:33.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.868+0000 7f166a59c640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1654004d00 0x7f1654005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.868+0000 7f166a59c640 1 -- 172.21.15.29:0/1155730976 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1654003520 con 0x7f16540039a0 2023-10-14T18:18:33.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.869+0000 7f166a59c640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f1654003d80 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f165c001890 tx=0x7f16540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf742cdc4a8c9ed0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:33.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.869+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c033630 con 0x7f16540039a0 2023-10-14T18:18:33.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.869+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c0337b0 con 0x7f16540039a0 2023-10-14T18:18:33.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.869+0000 7f166959a640 1 -- 172.21.15.29:0/1155730976 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c033b60 con 0x7f16540039a0 2023-10-14T18:18:33.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 -- 172.21.15.29:0/1155730976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 msgr2=0x7f1654003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 --2- 172.21.15.29:0/1155730976 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f1654003d80 secure :-1 s=READY pgs=2859 cs=0 l=1 rev1=1 crypto rx=0x7f165c001890 tx=0x7f16540057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f166ad9d640 1 -- 172.21.15.29:0/1155730976 reap_dead start 2023-10-14T18:18:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 -- 172.21.15.29:0/1155730976 shutdown_connections 2023-10-14T18:18:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 -- 172.21.15.29:0/1155730976 >> 172.21.15.29:0/1155730976 conn(0x7f166c0cb410 msgr2=0x7f166c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 -- 172.21.15.29:0/1155730976 shutdown_connections 2023-10-14T18:18:33.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.870+0000 7f1670b78640 1 -- 172.21.15.29:0/1155730976 wait complete. 2023-10-14T18:18:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.872+0000 7f1670b78640 1 Processor -- start 2023-10-14T18:18:33.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.872+0000 7f1670b78640 1 -- start start 2023-10-14T18:18:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166c1ef210 0x7f166c1efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c153120 0x7f166c0c7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c154af0 con 0x7f16540039a0 2023-10-14T18:18:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f166c1541e0 con 0x7f166c1ef210 2023-10-14T18:18:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1670b78640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f166c1f0170 con 0x7f166c153120 2023-10-14T18:18:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54602/0 (socket says 172.21.15.29:54602) 2023-10-14T18:18:33.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 -- 172.21.15.29:0/3679141157 learned_addr learned my addr 172.21.15.29:0/3679141157 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166ad9d640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c153120 0x7f166c0c7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f1669d9b640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166c1ef210 0x7f166c1efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 -- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c153120 msgr2=0x7f166c0c7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c153120 0x7f166c0c7dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 -- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166c1ef210 msgr2=0x7f166c1efc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166c1ef210 0x7f166c1efc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.873+0000 7f166a59c640 1 -- 172.21.15.29:0/3679141157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c14d1c0 con 0x7f16540039a0 2023-10-14T18:18:33.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f166a59c640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f165c000c00 tx=0x7f165c001230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c007050 con 0x7f16540039a0 2023-10-14T18:18:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f165c0071b0 con 0x7f16540039a0 2023-10-14T18:18:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f165c007540 con 0x7f16540039a0 2023-10-14T18:18:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f166c1ef7b0 con 0x7f16540039a0 2023-10-14T18:18:33.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.874+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f166c154710 con 0x7f16540039a0 2023-10-14T18:18:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.875+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c04e160 con 0x7f16540039a0 2023-10-14T18:18:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.877+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f165c0067d0 con 0x7f16540039a0 2023-10-14T18:18:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.878+0000 7f165affd640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 0x7f164c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:33.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.878+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(247..247 src has 1..247) v4 ==== 5343+0+0 (secure 0 0 0) 0x7f165c0abd70 con 0x7f16540039a0 2023-10-14T18:18:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.878+0000 7f1669d9b640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 0x7f164c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.879+0000 7f1669d9b640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 0x7f164c06d5a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1660001050 tx=0x7f1660001db0 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:33.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:33.882+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f165c076b90 con 0x7f16540039a0 2023-10-14T18:18:34.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.080+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f166c0b10b0 con 0x7f16540039a0 2023-10-14T18:18:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.615+0000 7f165affd640 1 -- 172.21.15.29:0/3679141157 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v248) v1 ==== 170+0+0 (secure 0 0 0) 0x7f165c07a1b0 con 0x7f16540039a0 2023-10-14T18:18:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T18:18:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.619+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 msgr2=0x7f164c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.619+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 0x7f164c06d5a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f1660001050 tx=0x7f1660001db0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.619+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 msgr2=0x7f166c0ad900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.619+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 secure :-1 s=READY pgs=2860 cs=0 l=1 rev1=1 crypto rx=0x7f165c000c00 tx=0x7f165c001230 comp rx=0 tx=0).stop 2023-10-14T18:18:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 shutdown_connections 2023-10-14T18:18:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f164c06b0e0 0x7f164c06d5a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f166c153120 0x7f166c0c7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f166c1ef210 0x7f166c1efc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 --2- 172.21.15.29:0/3679141157 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f16540039a0 0x7f166c0ad900 unknown :-1 s=CLOSED pgs=2860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.620+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 >> 172.21.15.29:0/3679141157 conn(0x7f166c0cb410 msgr2=0x7f166c0c4ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.621+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 shutdown_connections 2023-10-14T18:18:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.621+0000 7f1670b78640 1 -- 172.21.15.29:0/3679141157 wait complete. 2023-10-14T18:18:34.654 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T18:18:34.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.847+0000 7f2547451640 1 Processor -- start 2023-10-14T18:18:34.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.848+0000 7f2547451640 1 -- start start 2023-10-14T18:18:34.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.849+0000 7f2547451640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f254015ad70 0x7f2540069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.849+0000 7f2547451640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400c9ca0 0x7f254006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.849+0000 7f2547451640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25400cb030 con 0x7f25400ca3b0 2023-10-14T18:18:34.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.849+0000 7f2547451640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25400c91c0 con 0x7f254015ad70 2023-10-14T18:18:34.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.849+0000 7f2547451640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25400c94a0 con 0x7f25400c9ca0 2023-10-14T18:18:34.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f254015ad70 0x7f2540069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f254015ad70 0x7f2540069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54616/0 (socket says 172.21.15.29:54616) 2023-10-14T18:18:34.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 -- 172.21.15.29:0/793008612 learned_addr learned my addr 172.21.15.29:0/793008612 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:34.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25449ec640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400c9ca0 0x7f254006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 -- 172.21.15.29:0/793008612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400c9ca0 msgr2=0x7f254006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:34.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400c9ca0 0x7f254006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 -- 172.21.15.29:0/793008612 >> v1:172.21.15.186:6789/0 conn(0x7f25400ca3b0 legacy=0x7f25400ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:34.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.850+0000 7f25451ed640 1 -- 172.21.15.29:0/793008612 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25400c9780 con 0x7f254015ad70 2023-10-14T18:18:34.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.851+0000 7f25451ed640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f2540069da0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f253c009870 tx=0x7f253c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9aeeb9c0f211df41 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:34.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.852+0000 7f2537fff640 1 -- 172.21.15.29:0/793008612 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c031640 con 0x7f254015ad70 2023-10-14T18:18:34.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.852+0000 7f2537fff640 1 -- 172.21.15.29:0/793008612 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c0317a0 con 0x7f254015ad70 2023-10-14T18:18:34.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.852+0000 7f2537fff640 1 -- 172.21.15.29:0/793008612 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253c031b30 con 0x7f254015ad70 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 -- 172.21.15.29:0/793008612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 msgr2=0x7f2540069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f2540069da0 secure :-1 s=READY pgs=2861 cs=0 l=1 rev1=1 crypto rx=0x7f253c009870 tx=0x7f253c02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 -- 172.21.15.29:0/793008612 shutdown_connections 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400c9ca0 0x7f254006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 --2- 172.21.15.29:0/793008612 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f2540069da0 unknown :-1 s=CLOSED pgs=2861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 -- 172.21.15.29:0/793008612 >> 172.21.15.29:0/793008612 conn(0x7f25400ce9b0 msgr2=0x7f25400cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:34.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 -- 172.21.15.29:0/793008612 shutdown_connections 2023-10-14T18:18:34.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.853+0000 7f2547451640 1 -- 172.21.15.29:0/793008612 wait complete. 2023-10-14T18:18:34.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.855+0000 7f2547451640 1 Processor -- start 2023-10-14T18:18:34.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.855+0000 7f2547451640 1 -- start start 2023-10-14T18:18:34.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25400c9ca0 0x7f254015cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400ca3b0 0x7f25401f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f254015d420 con 0x7f254015ad70 2023-10-14T18:18:34.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25401f28c0 con 0x7f25400c9ca0 2023-10-14T18:18:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f2547451640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25401579b0 con 0x7f25400ca3b0 2023-10-14T18:18:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f25459ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f25459ee640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54624/0 (socket says 172.21.15.29:54624) 2023-10-14T18:18:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f25459ee640 1 -- 172.21.15.29:0/2138183937 learned_addr learned my addr 172.21.15.29:0/2138183937 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:34.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f25451ed640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25400c9ca0 0x7f254015cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.856+0000 7f25449ec640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400ca3b0 0x7f25401f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 -- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400ca3b0 msgr2=0x7f25401f2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:34.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400ca3b0 0x7f25401f2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 -- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25400c9ca0 msgr2=0x7f254015cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:34.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25400c9ca0 0x7f254015cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:34.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 -- 172.21.15.29:0/2138183937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253c0094d0 con 0x7f254015ad70 2023-10-14T18:18:34.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f25459ee640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f253800c780 tx=0x7f253800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:34.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2538004b60 con 0x7f254015ad70 2023-10-14T18:18:34.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253800ce90 con 0x7f254015ad70 2023-10-14T18:18:34.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f253800a850 con 0x7f254015ad70 2023-10-14T18:18:34.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25400cc580 con 0x7f254015ad70 2023-10-14T18:18:34.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.857+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25401f2ac0 con 0x7f254015ad70 2023-10-14T18:18:34.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.859+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254004e160 con 0x7f254015ad70 2023-10-14T18:18:34.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.860+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f253800a9b0 con 0x7f254015ad70 2023-10-14T18:18:34.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.861+0000 7f2535ffb640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 0x7f252406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:34.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.861+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(248..248 src has 1..248) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f2538089c20 con 0x7f254015ad70 2023-10-14T18:18:34.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.861+0000 7f25451ed640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 0x7f252406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:34.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.862+0000 7f25451ed640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 0x7f252406d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f253c036000 tx=0x7f253c03c040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:34.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:34.866+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2538054950 con 0x7f254015ad70 2023-10-14T18:18:35.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.101+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f254015cd10 con 0x7f254015ad70 2023-10-14T18:18:35.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.665+0000 7f2535ffb640 1 -- 172.21.15.29:0/2138183937 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 30 and data pool 31 v119) v1 ==== 174+0+0 (secure 0 0 0) 0x7f2538057f70 con 0x7f254015ad70 2023-10-14T18:18:35.668 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 30 and data pool 31 2023-10-14T18:18:35.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.674+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 msgr2=0x7f252406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.674+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 0x7f252406d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f253c036000 tx=0x7f253c03c040 comp rx=0 tx=0).stop 2023-10-14T18:18:35.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.674+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 msgr2=0x7f25400ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.674+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 secure :-1 s=READY pgs=2862 cs=0 l=1 rev1=1 crypto rx=0x7f253800c780 tx=0x7f253800cc40 comp rx=0 tx=0).stop 2023-10-14T18:18:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 shutdown_connections 2023-10-14T18:18:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f252406b0b0 0x7f252406d570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f254015ad70 0x7f25400ac890 unknown :-1 s=CLOSED pgs=2862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f25400ca3b0 0x7f25401f2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 --2- 172.21.15.29:0/2138183937 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f25400c9ca0 0x7f254015cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.675+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 >> 172.21.15.29:0/2138183937 conn(0x7f25400ce9b0 msgr2=0x7f25400ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:35.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.677+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 shutdown_connections 2023-10-14T18:18:35.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.677+0000 7f2547451640 1 -- 172.21.15.29:0/2138183937 wait complete. 2023-10-14T18:18:35.710 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.894+0000 7f34ac9bf640 1 Processor -- start 2023-10-14T18:18:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.894+0000 7f34ac9bf640 1 -- start start 2023-10-14T18:18:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34ac9bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 0x7f34a80ccf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34ac9bf640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a80cd4a0 con 0x7f34a80ad4d0 2023-10-14T18:18:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34ac9bf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34a815ac60 con 0x7f34a8153740 2023-10-14T18:18:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34ac9bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8070030 con 0x7f34a80ccb00 2023-10-14T18:18:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34a7fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f34a80ad4d0 0x7f34a80ad960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38304/0 (socket says 172.21.15.29:38304) 2023-10-14T18:18:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f34a7fff640 1 -- 172.21.15.29:0/4155492296 learned_addr learned my addr 172.21.15.29:0/4155492296 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.896+0000 7f349ffff640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 0x7f34a80ccf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3297135250 0 0) 0x7f34a80cd4a0 con 0x7f34a80ad4d0 2023-10-14T18:18:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3494003440 con 0x7f34a80ad4d0 2023-10-14T18:18:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 721389479 0 0) 0x7f3494003440 con 0x7f34a80ad4d0 2023-10-14T18:18:35.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 msgr2=0x7f34a80ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 0x7f34a80ccf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 >> v1:172.21.15.186:6789/0 conn(0x7f34a8153740 legacy=0x7f34a815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a8070310 con 0x7f34a80ad4d0 2023-10-14T18:18:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f34a00019f0 con 0x7f34a80ad4d0 2023-10-14T18:18:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 >> v1:172.21.15.29:6789/0 conn(0x7f34a80ad4d0 legacy=0x7f34a80ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494003a90 0x7f3494003eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3494008850 0x7f349400ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 0x7f349400d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8070030 con 0x7f3494008850 2023-10-14T18:18:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f349400dcb0 con 0x7f3494003a90 2023-10-14T18:18:35.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.897+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f349400de70 con 0x7f34a80ccb00 2023-10-14T18:18:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3494008850 0x7f349400ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f34a77fe640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494003a90 0x7f3494003eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 -- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 msgr2=0x7f349400d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80ccb00 0x7f349400d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 -- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494003a90 msgr2=0x7f3494003eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494003a90 0x7f3494003eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.898+0000 7f349ffff640 1 -- 172.21.15.29:0/4155492296 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3494003440 con 0x7f3494008850 2023-10-14T18:18:35.909 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.899+0000 7f349ffff640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3494008850 0x7f349400ac60 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f3490009ac0 tx=0x7f34940036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc43e477298bff59 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.899+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349002c710 con 0x7f3494008850 2023-10-14T18:18:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.899+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349002c870 con 0x7f3494008850 2023-10-14T18:18:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.899+0000 7f34a6ffd640 1 -- 172.21.15.29:0/4155492296 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f349002cc00 con 0x7f3494008850 2023-10-14T18:18:35.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4155492296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3494008850 msgr2=0x7f349400ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4155492296 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3494008850 0x7f349400ac60 secure :-1 s=READY pgs=2864 cs=0 l=1 rev1=1 crypto rx=0x7f3490009ac0 tx=0x7f34940036f0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34a7fff640 1 -- 172.21.15.29:0/4155492296 reap_dead start 2023-10-14T18:18:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4155492296 shutdown_connections 2023-10-14T18:18:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4155492296 >> 172.21.15.29:0/4155492296 conn(0x7f34a80ce6a0 msgr2=0x7f34a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:35.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4155492296 shutdown_connections 2023-10-14T18:18:35.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.900+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4155492296 wait complete. 2023-10-14T18:18:35.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.902+0000 7f34ac9bf640 1 Processor -- start 2023-10-14T18:18:35.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.902+0000 7f34ac9bf640 1 -- start start 2023-10-14T18:18:35.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494008850 0x7f34a8150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80b2ec0 0x7f34a80c43a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8156bb0 con 0x7f34a80cacf0 2023-10-14T18:18:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a80b14f0 con 0x7f3494008850 2023-10-14T18:18:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f34ac9bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f34a8157e40 con 0x7f34a80b2ec0 2023-10-14T18:18:35.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f349ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f349ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54644/0 (socket says 172.21.15.29:54644) 2023-10-14T18:18:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.903+0000 7f349ffff640 1 -- 172.21.15.29:0/4051012813 learned_addr learned my addr 172.21.15.29:0/4051012813 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34a77fe640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494008850 0x7f34a8150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34a7fff640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80b2ec0 0x7f34a80c43a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 -- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80b2ec0 msgr2=0x7f34a80c43a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80b2ec0 0x7f34a80c43a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 -- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494008850 msgr2=0x7f34a8150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494008850 0x7f34a8150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:35.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 -- 172.21.15.29:0/4051012813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34a81565f0 con 0x7f34a80cacf0 2023-10-14T18:18:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f349ffff640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f349002c2b0 tx=0x7f3490004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490003810 con 0x7f34a80cacf0 2023-10-14T18:18:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f349002de90 con 0x7f34a80cacf0 2023-10-14T18:18:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3490003bb0 con 0x7f34a80cacf0 2023-10-14T18:18:35.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.904+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a81568d0 con 0x7f34a80cacf0 2023-10-14T18:18:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.905+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a8157380 con 0x7f34a80cacf0 2023-10-14T18:18:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.906+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8156450 con 0x7f34a80cacf0 2023-10-14T18:18:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.907+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3490003d10 con 0x7f34a80cacf0 2023-10-14T18:18:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.907+0000 7f34a4ff9640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 0x7f348806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:35.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.908+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f34900abac0 con 0x7f34a80cacf0 2023-10-14T18:18:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.908+0000 7f34a77fe640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 0x7f348806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.908+0000 7f34a77fe640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 0x7f348806d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3498001710 tx=0x7f3498009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:35.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:35.912+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f34a8156450 con 0x7f34a80cacf0 2023-10-14T18:18:36.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.106+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34a8156450 con 0x7f34a80cacf0 2023-10-14T18:18:36.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.109+0000 7f34a4ff9640 1 -- 172.21.15.29:0/4051012813 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+16706 (secure 0 0 0) 0x7f34a8156450 con 0x7f34a80cacf0 2023-10-14T18:18:36.111 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:36.111 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":249,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:35.103304+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:36.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 msgr2=0x7f348806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 0x7f348806d570 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f3498001710 tx=0x7f3498009040 comp rx=0 tx=0).stop 2023-10-14T18:18:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 msgr2=0x7f34a80b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 secure :-1 s=READY pgs=2865 cs=0 l=1 rev1=1 crypto rx=0x7f349002c2b0 tx=0x7f3490004730 comp rx=0 tx=0).stop 2023-10-14T18:18:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 shutdown_connections 2023-10-14T18:18:36.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f348806b0b0 0x7f348806d570 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f34a80b2ec0 0x7f34a80c43a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f34a80cacf0 0x7f34a80b2980 unknown :-1 s=CLOSED pgs=2865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 --2- 172.21.15.29:0/4051012813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3494008850 0x7f34a8150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.112+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 >> 172.21.15.29:0/4051012813 conn(0x7f34a80ce6a0 msgr2=0x7f34a80af9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:36.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.113+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 shutdown_connections 2023-10-14T18:18:36.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.113+0000 7f34ac9bf640 1 -- 172.21.15.29:0/4051012813 wait complete. 2023-10-14T18:18:36.132 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:36.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.354+0000 7f020f260640 1 Processor -- start 2023-10-14T18:18:36.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.354+0000 7f020f260640 1 -- start start 2023-10-14T18:18:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020f260640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f02080cc2e0 0x7f02080cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020f260640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02080b3730 con 0x7f020815aa60 2023-10-14T18:18:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020f260640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02080b3980 con 0x7f02080b4740 2023-10-14T18:18:36.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020f260640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02080b3c60 con 0x7f02080cc2e0 2023-10-14T18:18:36.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020e25e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f020815aa60 0x7f02080af820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38320/0 (socket says 172.21.15.29:38320) 2023-10-14T18:18:36.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020e25e640 1 -- 172.21.15.29:0/1348783462 learned_addr learned my addr 172.21.15.29:0/1348783462 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.355+0000 7f020da5d640 1 --2- 172.21.15.29:0/1348783462 >> v2:172.21.15.29:3300/0 conn(0x7f02080cc2e0 0x7f02080cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.356+0000 7f020d25c640 1 -- 172.21.15.29:0/1348783462 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601547762 0 0) 0x7f02080b3730 con 0x7f020815aa60 2023-10-14T18:18:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.356+0000 7f020d25c640 1 -- 172.21.15.29:0/1348783462 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01f4003520 con 0x7f020815aa60 2023-10-14T18:18:36.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.356+0000 7f020da5d640 1 -- 172.21.15.29:0/1348783462 >> v1:172.21.15.186:6789/0 conn(0x7f02080b4740 legacy=0x7f02080b4bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:36.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.356+0000 7f020da5d640 1 -- 172.21.15.29:0/1348783462 >> v1:172.21.15.29:6789/0 conn(0x7f020815aa60 legacy=0x7f02080af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.356+0000 7f020da5d640 1 -- 172.21.15.29:0/1348783462 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02080b3f40 con 0x7f02080cc2e0 2023-10-14T18:18:36.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.357+0000 7f020da5d640 1 --2- 172.21.15.29:0/1348783462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080cc2e0 0x7f02080cc750 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097a0 tx=0x7f01f802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffa00df4cb6c402a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.357+0000 7f020d25c640 1 -- 172.21.15.29:0/1348783462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8031830 con 0x7f02080cc2e0 2023-10-14T18:18:36.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020d25c640 1 -- 172.21.15.29:0/1348783462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f01f8031990 con 0x7f02080cc2e0 2023-10-14T18:18:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020d25c640 1 -- 172.21.15.29:0/1348783462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01f8031d20 con 0x7f02080cc2e0 2023-10-14T18:18:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 -- 172.21.15.29:0/1348783462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080cc2e0 msgr2=0x7f02080cc750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 --2- 172.21.15.29:0/1348783462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080cc2e0 0x7f02080cc750 secure :-1 s=READY pgs=2867 cs=0 l=1 rev1=1 crypto rx=0x7f01f80097a0 tx=0x7f01f802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 -- 172.21.15.29:0/1348783462 shutdown_connections 2023-10-14T18:18:36.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 --2- 172.21.15.29:0/1348783462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080cc2e0 0x7f02080cc750 unknown :-1 s=CLOSED pgs=2867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 -- 172.21.15.29:0/1348783462 >> 172.21.15.29:0/1348783462 conn(0x7f0208065ed0 msgr2=0x7f02080662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:36.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 -- 172.21.15.29:0/1348783462 shutdown_connections 2023-10-14T18:18:36.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.358+0000 7f020f260640 1 -- 172.21.15.29:0/1348783462 wait complete. 2023-10-14T18:18:36.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.360+0000 7f020f260640 1 Processor -- start 2023-10-14T18:18:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.360+0000 7f020f260640 1 -- start start 2023-10-14T18:18:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 0x7f02080b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020815aa60 0x7f02080c6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02080c68b0 con 0x7f02080b4740 2023-10-14T18:18:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f020815dfb0 con 0x7f02080cc2e0 2023-10-14T18:18:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020f260640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02080c5c30 con 0x7f020815aa60 2023-10-14T18:18:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020e25e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020e25e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54662/0 (socket says 172.21.15.29:54662) 2023-10-14T18:18:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.361+0000 7f020e25e640 1 -- 172.21.15.29:0/89684336 learned_addr learned my addr 172.21.15.29:0/89684336 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020da5d640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 0x7f02080b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020ea5f640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020815aa60 0x7f02080c6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 -- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020815aa60 msgr2=0x7f02080c6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020815aa60 0x7f02080c6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 -- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 msgr2=0x7f02080b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 0x7f02080b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 -- 172.21.15.29:0/89684336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01f8009400 con 0x7f02080b4740 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020da5d640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 0x7f02080b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020e25e640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f020400c440 tx=0x7f020400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0204005ce0 con 0x7f02080b4740 2023-10-14T18:18:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0204005e40 con 0x7f02080b4740 2023-10-14T18:18:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02080c5f10 con 0x7f02080b4740 2023-10-14T18:18:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f02040072b0 con 0x7f02080b4740 2023-10-14T18:18:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.362+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02081551c0 con 0x7f02080b4740 2023-10-14T18:18:36.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.363+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02080c69f0 con 0x7f02080b4740 2023-10-14T18:18:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.364+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f0204007410 con 0x7f02080b4740 2023-10-14T18:18:36.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.366+0000 7f01feffd640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 0x7f01ec06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.366+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(249..249 src has 1..249) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f02040890a0 con 0x7f02080b4740 2023-10-14T18:18:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.366+0000 7f020da5d640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 0x7f01ec06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.367+0000 7f020da5d640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 0x7f01ec06d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01f8036000 tx=0x7f01f8008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.370+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0204053e20 con 0x7f02080b4740 2023-10-14T18:18:36.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.561+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f020804e9b0 con 0x7f02080b4740 2023-10-14T18:18:36.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.564+0000 7f01feffd640 1 -- 172.21.15.29:0/89684336 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v249) v1 ==== 74+0+16706 (secure 0 0 0) 0x7f0204017090 con 0x7f02080b4740 2023-10-14T18:18:36.565 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:36.565 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":249,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:35.103304+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:36.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.566+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 msgr2=0x7f01ec06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.566+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 0x7f01ec06d570 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01f8036000 tx=0x7f01f8008030 comp rx=0 tx=0).stop 2023-10-14T18:18:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.566+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 msgr2=0x7f020815da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.566+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 secure :-1 s=READY pgs=2868 cs=0 l=1 rev1=1 crypto rx=0x7f020400c440 tx=0x7f020400c900 comp rx=0 tx=0).stop 2023-10-14T18:18:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 shutdown_connections 2023-10-14T18:18:36.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f01ec06b0b0 0x7f01ec06d570 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f020815aa60 0x7f02080c6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f02080cc2e0 0x7f02080b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 --2- 172.21.15.29:0/89684336 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f02080b4740 0x7f020815da70 unknown :-1 s=CLOSED pgs=2868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 >> 172.21.15.29:0/89684336 conn(0x7f0208065ed0 msgr2=0x7f02080662b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 shutdown_connections 2023-10-14T18:18:36.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.567+0000 7f020f260640 1 -- 172.21.15.29:0/89684336 wait complete. 2023-10-14T18:18:36.586 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T18:18:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.820+0000 7fe05b35e640 1 Processor -- start 2023-10-14T18:18:36.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.820+0000 7fe05b35e640 1 -- start start 2023-10-14T18:18:36.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05b35e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 0x7fe05415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05b35e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe05406c100 0x7fe05406c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05b35e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05415aef0 con 0x7fe054159950 2023-10-14T18:18:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05b35e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe05415e490 con 0x7fe05406c100 2023-10-14T18:18:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05b35e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe05415d910 con 0x7fe054070b60 2023-10-14T18:18:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05ab5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe05406c100 0x7fe05406c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05ab5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe05406c100 0x7fe05406c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54676/0 (socket says 172.21.15.29:54676) 2023-10-14T18:18:36.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.822+0000 7fe05ab5d640 1 -- 172.21.15.29:0/867944815 learned_addr learned my addr 172.21.15.29:0/867944815 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:36.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe059b5b640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 0x7fe05415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe05ab5d640 1 -- 172.21.15.29:0/867944815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 msgr2=0x7fe05415a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:36.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe05ab5d640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 0x7fe05415a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe05ab5d640 1 -- 172.21.15.29:0/867944815 >> v1:172.21.15.186:6789/0 conn(0x7fe054159950 legacy=0x7fe054070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:36.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe05ab5d640 1 -- 172.21.15.29:0/867944815 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05415dbf0 con 0x7fe05406c100 2023-10-14T18:18:36.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.823+0000 7fe059b5b640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 0x7fe05415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:36.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.824+0000 7fe05ab5d640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe05406c550 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009e50 tx=0x7fe04c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b66e770742dc55e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.825+0000 7fe05935a640 1 -- 172.21.15.29:0/867944815 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c02ca70 con 0x7fe05406c100 2023-10-14T18:18:36.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.825+0000 7fe05935a640 1 -- 172.21.15.29:0/867944815 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe04c02cbd0 con 0x7fe05406c100 2023-10-14T18:18:36.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.825+0000 7fe05935a640 1 -- 172.21.15.29:0/867944815 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe04c0343f0 con 0x7fe05406c100 2023-10-14T18:18:36.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.825+0000 7fe05b35e640 1 -- 172.21.15.29:0/867944815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 msgr2=0x7fe05406c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.825+0000 7fe05b35e640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe05406c550 secure :-1 s=READY pgs=2869 cs=0 l=1 rev1=1 crypto rx=0x7fe04c009e50 tx=0x7fe04c02bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 -- 172.21.15.29:0/867944815 shutdown_connections 2023-10-14T18:18:36.829 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe05406c550 unknown :-1 s=CLOSED pgs=2869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 --2- 172.21.15.29:0/867944815 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054070b60 0x7fe05415a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 -- 172.21.15.29:0/867944815 >> 172.21.15.29:0/867944815 conn(0x7fe0540ce6a0 msgr2=0x7fe0540ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 -- 172.21.15.29:0/867944815 shutdown_connections 2023-10-14T18:18:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.826+0000 7fe05b35e640 1 -- 172.21.15.29:0/867944815 wait complete. 2023-10-14T18:18:36.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.827+0000 7fe05b35e640 1 Processor -- start 2023-10-14T18:18:36.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.828+0000 7fe05b35e640 1 -- start start 2023-10-14T18:18:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe054070b60 0x7fe0541f4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 0x7fe0541f54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0540c5b80 con 0x7fe05406c100 2023-10-14T18:18:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe054151d90 con 0x7fe054070b60 2023-10-14T18:18:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05b35e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe054154fb0 con 0x7fe054159950 2023-10-14T18:18:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54688/0 (socket says 172.21.15.29:54688) 2023-10-14T18:18:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 -- 172.21.15.29:0/588249394 learned_addr learned my addr 172.21.15.29:0/588249394 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe059b5b640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe054070b60 0x7fe0541f4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05ab5d640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 0x7fe0541f54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 -- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 msgr2=0x7fe0541f54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 0x7fe0541f54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 -- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe054070b60 msgr2=0x7fe0541f4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.829+0000 7fe05a35c640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe054070b60 0x7fe0541f4fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe05a35c640 1 -- 172.21.15.29:0/588249394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c009b30 con 0x7fe05406c100 2023-10-14T18:18:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe05ab5d640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 0x7fe0541f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:36.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe05a35c640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c430 tx=0x7fe05000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0500052d0 con 0x7fe05406c100 2023-10-14T18:18:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe050005430 con 0x7fe05406c100 2023-10-14T18:18:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe050003830 con 0x7fe05406c100 2023-10-14T18:18:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe054155290 con 0x7fe05406c100 2023-10-14T18:18:36.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.830+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe054152960 con 0x7fe05406c100 2023-10-14T18:18:36.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.832+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0541557b0 con 0x7fe05406c100 2023-10-14T18:18:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.833+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe050003990 con 0x7fe05406c100 2023-10-14T18:18:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.833+0000 7fe04affd640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:36.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.834+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe050088e80 con 0x7fe05406c100 2023-10-14T18:18:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.834+0000 7fe059b5b640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.835+0000 7fe059b5b640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 0x7fe03806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe044006fd0 tx=0x7fe044008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:36.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:36.838+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0541557b0 con 0x7fe05406c100 2023-10-14T18:18:37.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.071+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fe05415aef0 con 0x7fe05406c100 2023-10-14T18:18:37.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.678+0000 7fe04affd640 1 -- 172.21.15.29:0/588249394 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v122) v1 ==== 118+0+0 (secure 0 0 0) 0x7fe05415aef0 con 0x7fe05406c100 2023-10-14T18:18:37.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.681+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 msgr2=0x7fe03806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 0x7fe03806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe044006fd0 tx=0x7fe044008040 comp rx=0 tx=0).stop 2023-10-14T18:18:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 msgr2=0x7fe054152420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 secure :-1 s=READY pgs=2870 cs=0 l=1 rev1=1 crypto rx=0x7fe05000c430 tx=0x7fe05000c8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 shutdown_connections 2023-10-14T18:18:37.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe054159950 0x7fe0541f54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe054070b60 0x7fe0541f4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 --2- 172.21.15.29:0/588249394 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe05406c100 0x7fe054152420 unknown :-1 s=CLOSED pgs=2870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 >> 172.21.15.29:0/588249394 conn(0x7fe0540ce6a0 msgr2=0x7fe054070450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 shutdown_connections 2023-10-14T18:18:37.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.682+0000 7fe05b35e640 1 -- 172.21.15.29:0/588249394 wait complete. 2023-10-14T18:18:37.706 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:18:37.706 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T18:18:37.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.941+0000 7f3372cd7640 1 Processor -- start 2023-10-14T18:18:37.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.941+0000 7f3372cd7640 1 -- start start 2023-10-14T18:18:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3372cd7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 0x7f336c0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3372cd7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c070360 con 0x7f336c159ab0 2023-10-14T18:18:37.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3372cd7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f336c0cd1f0 con 0x7f336c159e90 2023-10-14T18:18:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3372cd7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c15d890 con 0x7f336c06c850 2023-10-14T18:18:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3370a73640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f336c159ab0 0x7f336c06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38332/0 (socket says 172.21.15.29:38332) 2023-10-14T18:18:37.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.942+0000 7f3370a73640 1 -- 172.21.15.29:0/4290316192 learned_addr learned my addr 172.21.15.29:0/4290316192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f3371274640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 0x7f336c0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281622944 0 0) 0x7f336c070360 con 0x7f336c159ab0 2023-10-14T18:18:37.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3354003520 con 0x7f336c159ab0 2023-10-14T18:18:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966580706 0 0) 0x7f3354003520 con 0x7f336c159ab0 2023-10-14T18:18:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 msgr2=0x7f336c0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 0x7f336c0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 >> v1:172.21.15.186:6789/0 conn(0x7f336c159e90 legacy=0x7f336c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c0cd4d0 con 0x7f336c159ab0 2023-10-14T18:18:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.943+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f335c0019f0 con 0x7f336c159ab0 2023-10-14T18:18:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 >> v1:172.21.15.29:6789/0 conn(0x7f336c159ab0 legacy=0x7f336c06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33540039a0 0x7f3354003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f3354004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 0x7f3354005250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c15d890 con 0x7f33540047c0 2023-10-14T18:18:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f335400bb70 con 0x7f33540039a0 2023-10-14T18:18:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f335400bd30 con 0x7f336c06c850 2023-10-14T18:18:37.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f3354004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f3370a73640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33540039a0 0x7f3354003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 -- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 msgr2=0x7f3354005250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06c850 0x7f3354005250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 -- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33540039a0 msgr2=0x7f3354003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:37.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f33540039a0 0x7f3354003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.944+0000 7f336bfff640 1 -- 172.21.15.29:0/4290316192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3354003520 con 0x7f33540047c0 2023-10-14T18:18:37.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.945+0000 7f336bfff640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f3354004c40 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f3360001750 tx=0x7f33540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3986e09ca5ee02b1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33600052d0 con 0x7f33540047c0 2023-10-14T18:18:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f336002e050 con 0x7f33540047c0 2023-10-14T18:18:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f336b7fe640 1 -- 172.21.15.29:0/4290316192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33600035f0 con 0x7f33540047c0 2023-10-14T18:18:37.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 -- 172.21.15.29:0/4290316192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 msgr2=0x7f3354004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 --2- 172.21.15.29:0/4290316192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f3354004c40 secure :-1 s=READY pgs=2872 cs=0 l=1 rev1=1 crypto rx=0x7f3360001750 tx=0x7f33540057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3371274640 1 -- 172.21.15.29:0/4290316192 reap_dead start 2023-10-14T18:18:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 -- 172.21.15.29:0/4290316192 shutdown_connections 2023-10-14T18:18:37.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 -- 172.21.15.29:0/4290316192 >> 172.21.15.29:0/4290316192 conn(0x7f336c0ce9b0 msgr2=0x7f336c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 -- 172.21.15.29:0/4290316192 shutdown_connections 2023-10-14T18:18:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.946+0000 7f3372cd7640 1 -- 172.21.15.29:0/4290316192 wait complete. 2023-10-14T18:18:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.948+0000 7f3372cd7640 1 Processor -- start 2023-10-14T18:18:37.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.948+0000 7f3372cd7640 1 -- start start 2023-10-14T18:18:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 0x7f336c06b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06b570 0x7f336c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c14cbb0 con 0x7f33540047c0 2023-10-14T18:18:37.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f336c14ce90 con 0x7f336c148df0 2023-10-14T18:18:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.949+0000 7f3372cd7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f336c1689d0 con 0x7f336c06b570 2023-10-14T18:18:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54712/0 (socket says 172.21.15.29:54712) 2023-10-14T18:18:37.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 -- 172.21.15.29:0/1137920368 learned_addr learned my addr 172.21.15.29:0/1137920368 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f336bfff640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 0x7f336c06b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3371274640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06b570 0x7f336c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 -- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06b570 msgr2=0x7f336c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06b570 0x7f336c0ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 -- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 msgr2=0x7f336c06b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 0x7f336c06b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 -- 172.21.15.29:0/1137920368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f336c06ba10 con 0x7f33540047c0 2023-10-14T18:18:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f336bfff640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 0x7f336c06b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:37.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.950+0000 7f3370a73640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f335c0020c0 tx=0x7f335c00b4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.951+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c004d60 con 0x7f33540047c0 2023-10-14T18:18:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.951+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f335c00be80 con 0x7f33540047c0 2023-10-14T18:18:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.951+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f335c007310 con 0x7f33540047c0 2023-10-14T18:18:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.951+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f336c06bcf0 con 0x7f33540047c0 2023-10-14T18:18:37.963 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.951+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f336c15d140 con 0x7f33540047c0 2023-10-14T18:18:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.952+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f336c04e160 con 0x7f33540047c0 2023-10-14T18:18:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.953+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f335c007470 con 0x7f33540047c0 2023-10-14T18:18:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.954+0000 7f33697fa640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 0x7f335806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:37.964 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.954+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f335c089d10 con 0x7f33540047c0 2023-10-14T18:18:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.954+0000 7f336bfff640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 0x7f335806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.955+0000 7f336bfff640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 0x7f335806d5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3360033000 tx=0x7f3360009030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:37.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:37.960+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f335c054af0 con 0x7f33540047c0 2023-10-14T18:18:38.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.196+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f336c0cefd0 con 0x7f33540047c0 2023-10-14T18:18:38.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.691+0000 7f33697fa640 1 -- 172.21.15.29:0/1137920368 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v123) v1 ==== 122+0+0 (secure 0 0 0) 0x7f335c019020 con 0x7f33540047c0 2023-10-14T18:18:38.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 msgr2=0x7f335806d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:38.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 0x7f335806d5c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3360033000 tx=0x7f3360009030 comp rx=0 tx=0).stop 2023-10-14T18:18:38.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 msgr2=0x7f336c168490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:38.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 secure :-1 s=READY pgs=2873 cs=0 l=1 rev1=1 crypto rx=0x7f335c0020c0 tx=0x7f335c00b4a0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 shutdown_connections 2023-10-14T18:18:38.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f335806b100 0x7f335806d5c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f336c06b570 0x7f336c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f336c148df0 0x7f336c06b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 --2- 172.21.15.29:0/1137920368 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f33540047c0 0x7f336c168490 unknown :-1 s=CLOSED pgs=2873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 >> 172.21.15.29:0/1137920368 conn(0x7f336c0ce9b0 msgr2=0x7f336c167090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:38.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 shutdown_connections 2023-10-14T18:18:38.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.695+0000 7f3372cd7640 1 -- 172.21.15.29:0/1137920368 wait complete. 2023-10-14T18:18:38.719 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T18:18:38.719 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:38.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.943+0000 7fd08664d640 1 Processor -- start 2023-10-14T18:18:38.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.944+0000 7fd08664d640 1 -- start start 2023-10-14T18:18:38.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.945+0000 7fd08664d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd080159950 0x7fd08015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.945+0000 7fd08664d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd0800708e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.945+0000 7fd08664d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd08015aef0 con 0x7fd080070450 2023-10-14T18:18:38.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.945+0000 7fd08664d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd080070e20 con 0x7fd080159950 2023-10-14T18:18:38.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.945+0000 7fd08664d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd08015e490 con 0x7fd08006c810 2023-10-14T18:18:38.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd080159950 0x7fd08015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd080159950 0x7fd08015a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54722/0 (socket says 172.21.15.29:54722) 2023-10-14T18:18:38.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 -- 172.21.15.29:0/814528389 learned_addr learned my addr 172.21.15.29:0/814528389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd085e4c640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd0800708e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 -- 172.21.15.29:0/814528389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 msgr2=0x7fd0800708e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd0800708e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 -- 172.21.15.29:0/814528389 >> v1:172.21.15.186:6789/0 conn(0x7fd080070450 legacy=0x7fd08006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:38.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.946+0000 7fd08564b640 1 -- 172.21.15.29:0/814528389 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd08006cc60 con 0x7fd080159950 2023-10-14T18:18:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.948+0000 7fd08564b640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080159950 0x7fd08015a9b0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009870 tx=0x7fd07c02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e6bc2348fb5176 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.948+0000 7fd077fff640 1 -- 172.21.15.29:0/814528389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c0317c0 con 0x7fd080159950 2023-10-14T18:18:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.948+0000 7fd077fff640 1 -- 172.21.15.29:0/814528389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd07c031920 con 0x7fd080159950 2023-10-14T18:18:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.948+0000 7fd077fff640 1 -- 172.21.15.29:0/814528389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd07c031cb0 con 0x7fd080159950 2023-10-14T18:18:38.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 -- 172.21.15.29:0/814528389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080159950 msgr2=0x7fd08015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080159950 0x7fd08015a9b0 secure :-1 s=READY pgs=2874 cs=0 l=1 rev1=1 crypto rx=0x7fd07c009870 tx=0x7fd07c02b860 comp rx=0 tx=0).stop 2023-10-14T18:18:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 -- 172.21.15.29:0/814528389 shutdown_connections 2023-10-14T18:18:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd0800708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 --2- 172.21.15.29:0/814528389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080159950 0x7fd08015a9b0 unknown :-1 s=CLOSED pgs=2874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 -- 172.21.15.29:0/814528389 >> 172.21.15.29:0/814528389 conn(0x7fd0800ce6a0 msgr2=0x7fd0800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:38.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 -- 172.21.15.29:0/814528389 shutdown_connections 2023-10-14T18:18:38.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.949+0000 7fd08664d640 1 -- 172.21.15.29:0/814528389 wait complete. 2023-10-14T18:18:38.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.951+0000 7fd08664d640 1 Processor -- start 2023-10-14T18:18:38.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.951+0000 7fd08664d640 1 -- start start 2023-10-14T18:18:38.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd08014f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd080159950 0x7fd0800ae800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd080159110 con 0x7fd080070450 2023-10-14T18:18:38.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd080158350 con 0x7fd080159950 2023-10-14T18:18:38.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08664d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd080158630 con 0x7fd08006c810 2023-10-14T18:18:38.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd084e4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd084e4a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54732/0 (socket says 172.21.15.29:54732) 2023-10-14T18:18:38.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd084e4a640 1 -- 172.21.15.29:0/2638982199 learned_addr learned my addr 172.21.15.29:0/2638982199 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:38.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd08564b640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd08014f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.952+0000 7fd085e4c640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd080159950 0x7fd0800ae800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 -- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 msgr2=0x7fd08014f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:38.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd08014f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 -- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd080159950 msgr2=0x7fd0800ae800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:38.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd080159950 0x7fd0800ae800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:38.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 -- 172.21.15.29:0/2638982199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd07c0094d0 con 0x7fd080070450 2023-10-14T18:18:38.958 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd084e4a640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c430 tx=0x7fd07000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:38.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.953+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011790 con 0x7fd080070450 2023-10-14T18:18:38.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.954+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0700118f0 con 0x7fd080070450 2023-10-14T18:18:38.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.954+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd070011ce0 con 0x7fd080070450 2023-10-14T18:18:38.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.954+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd080158910 con 0x7fd080070450 2023-10-14T18:18:38.959 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.954+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08014cc10 con 0x7fd080070450 2023-10-14T18:18:38.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.955+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd080158e30 con 0x7fd080070450 2023-10-14T18:18:38.960 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.956+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd07001c460 con 0x7fd080070450 2023-10-14T18:18:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.957+0000 7fd075ffb640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 0x7fd06406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.957+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd0700897f0 con 0x7fd080070450 2023-10-14T18:18:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.957+0000 7fd08564b640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 0x7fd06406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:38.961 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.958+0000 7fd08564b640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 0x7fd06406d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd07c036000 tx=0x7fd07c03d040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:38.962 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:38.961+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd080158e30 con 0x7fd080070450 2023-10-14T18:18:39.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.195+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd080158e30 con 0x7fd080070450 2023-10-14T18:18:39.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.196+0000 7fd075ffb640 1 -- 172.21.15.29:0/2638982199 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fd080158e30 con 0x7fd080070450 2023-10-14T18:18:39.198 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:39.198 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":123,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-14T18:18:39.198 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 123 2023-10-14T18:18:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 msgr2=0x7fd06406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 0x7fd06406d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd07c036000 tx=0x7fd07c03d040 comp rx=0 tx=0).stop 2023-10-14T18:18:39.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 msgr2=0x7fd08014c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 secure :-1 s=READY pgs=2875 cs=0 l=1 rev1=1 crypto rx=0x7fd07000c430 tx=0x7fd07000c8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 shutdown_connections 2023-10-14T18:18:39.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd06406b0b0 0x7fd06406d570 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd080159950 0x7fd0800ae800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd080070450 0x7fd08014c6d0 unknown :-1 s=CLOSED pgs=2875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 --2- 172.21.15.29:0/2638982199 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd08006c810 0x7fd08014f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 >> 172.21.15.29:0/2638982199 conn(0x7fd0800ce6a0 msgr2=0x7fd0800af910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.199+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 shutdown_connections 2023-10-14T18:18:39.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.200+0000 7fd08664d640 1 -- 172.21.15.29:0/2638982199 wait complete. 2023-10-14T18:18:39.221 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.448+0000 7f9b76978640 1 Processor -- start 2023-10-14T18:18:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.448+0000 7f9b76978640 1 -- start start 2023-10-14T18:18:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b76978640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b700ce390 0x7f9b7015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b76978640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7015ed60 con 0x7f9b7015ae80 2023-10-14T18:18:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b76978640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b700707e0 con 0x7f9b7015aaa0 2023-10-14T18:18:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b76978640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7006b810 con 0x7f9b700ce390 2023-10-14T18:18:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b6f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b7015ae80 0x7f9b70070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38340/0 (socket says 172.21.15.29:38340) 2023-10-14T18:18:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.450+0000 7f9b6f7fe640 1 -- 172.21.15.29:0/2680231221 learned_addr learned my addr 172.21.15.29:0/2680231221 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:39.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b700ce390 0x7f9b7015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006060453 0 0) 0x7f9b7015ed60 con 0x7f9b7015ae80 2023-10-14T18:18:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b58003520 con 0x7f9b7015ae80 2023-10-14T18:18:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1369333606 0 0) 0x7f9b58003520 con 0x7f9b7015ae80 2023-10-14T18:18:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b700ce390 msgr2=0x7f9b7015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b700ce390 0x7f9b7015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 >> v1:172.21.15.186:6789/0 conn(0x7f9b7015aaa0 legacy=0x7f9b700cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7015ef90 con 0x7f9b7015ae80 2023-10-14T18:18:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f9b64001690 con 0x7f9b7015ae80 2023-10-14T18:18:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 >> v1:172.21.15.29:6789/0 conn(0x7f9b7015ae80 legacy=0x7f9b70070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b580039a0 0x7f9b58003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7006b810 con 0x7f9b58005180 2023-10-14T18:18:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5800c050 con 0x7f9b580039a0 2023-10-14T18:18:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b5800c210 con 0x7f9b580047c0 2023-10-14T18:18:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.451+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b700ce390 0x7f9b7015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b6ffff640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b580039a0 0x7f9b58003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b6f7fe640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 -- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b580047c0 msgr2=0x7f9b58004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b580047c0 0x7f9b58004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 -- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b580039a0 msgr2=0x7f9b58003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b580039a0 0x7f9b58003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.452+0000 7f9b74f15640 1 -- 172.21.15.29:0/2680231221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b58003520 con 0x7f9b58005180 2023-10-14T18:18:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.453+0000 7f9b74f15640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f9b700ce770 tx=0x7f9b580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4837a37b58f66aa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6802cc80 con 0x7f9b58005180 2023-10-14T18:18:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6802cde0 con 0x7f9b58005180 2023-10-14T18:18:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b6effd640 1 -- 172.21.15.29:0/2680231221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b68003bb0 con 0x7f9b58005180 2023-10-14T18:18:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 -- 172.21.15.29:0/2680231221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b58005180 msgr2=0x7f9b5800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 --2- 172.21.15.29:0/2680231221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b58005180 0x7f9b5800b980 secure :-1 s=READY pgs=2877 cs=0 l=1 rev1=1 crypto rx=0x7f9b700ce770 tx=0x7f9b580057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b74f15640 1 -- 172.21.15.29:0/2680231221 reap_dead start 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 -- 172.21.15.29:0/2680231221 shutdown_connections 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 -- 172.21.15.29:0/2680231221 >> 172.21.15.29:0/2680231221 conn(0x7f9b700cb410 msgr2=0x7f9b700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 -- 172.21.15.29:0/2680231221 shutdown_connections 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.454+0000 7f9b76978640 1 -- 172.21.15.29:0/2680231221 wait complete. 2023-10-14T18:18:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.456+0000 7f9b76978640 1 Processor -- start 2023-10-14T18:18:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.456+0000 7f9b76978640 1 -- start start 2023-10-14T18:18:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b58005180 0x7f9b700c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b700c2e80 0x7f9b7015c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b700c3330 con 0x7f9b700c24c0 2023-10-14T18:18:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b7015c3a0 con 0x7f9b700c2e80 2023-10-14T18:18:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.457+0000 7f9b76978640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b701f25c0 con 0x7f9b58005180 2023-10-14T18:18:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54742/0 (socket says 172.21.15.29:54742) 2023-10-14T18:18:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 -- 172.21.15.29:0/3795162759 learned_addr learned my addr 172.21.15.29:0/3795162759 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 -- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b58005180 msgr2=0x7f9b700c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b58005180 0x7f9b700c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 -- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b700c2e80 msgr2=0x7f9b7015c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b74f15640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b700c2e80 0x7f9b7015c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b700c2e80 0x7f9b7015c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 -- 172.21.15.29:0/3795162759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7015b790 con 0x7f9b700c24c0 2023-10-14T18:18:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.458+0000 7f9b6f7fe640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400e780 tx=0x7f9b6400ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.459+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6400ee30 con 0x7f9b700c24c0 2023-10-14T18:18:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.459+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b6400c870 con 0x7f9b700c24c0 2023-10-14T18:18:39.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.459+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b6400cc20 con 0x7f9b700c24c0 2023-10-14T18:18:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.459+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b7015ba70 con 0x7f9b700c24c0 2023-10-14T18:18:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.459+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b7015c030 con 0x7f9b700c24c0 2023-10-14T18:18:39.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.460+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7004e160 con 0x7f9b700c24c0 2023-10-14T18:18:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.461+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9b6401d460 con 0x7f9b700c24c0 2023-10-14T18:18:39.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.462+0000 7f9b6cff9640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 0x7f9b5c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.462+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9b64089850 con 0x7f9b700c24c0 2023-10-14T18:18:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.462+0000 7f9b6ffff640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 0x7f9b5c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.463+0000 7f9b6ffff640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 0x7f9b5c06d5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9b60007fd0 tx=0x7f9b60009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.466+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b64054680 con 0x7f9b700c24c0 2023-10-14T18:18:39.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.666+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b7015c4e0 con 0x7f9b700c24c0 2023-10-14T18:18:39.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.669+0000 7f9b6cff9640 1 -- 172.21.15.29:0/3795162759 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+16693 (secure 0 0 0) 0x7f9b64057ca0 con 0x7f9b700c24c0 2023-10-14T18:18:39.670 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:39.670 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":250,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:36.639061+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:39.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.671+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 msgr2=0x7f9b5c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.671+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 0x7f9b5c06d5a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f9b60007fd0 tx=0x7f9b60009040 comp rx=0 tx=0).stop 2023-10-14T18:18:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.671+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 msgr2=0x7f9b700c2940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.671+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 secure :-1 s=READY pgs=2878 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400e780 tx=0x7f9b6400ec40 comp rx=0 tx=0).stop 2023-10-14T18:18:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 shutdown_connections 2023-10-14T18:18:39.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f9b5c06b0e0 0x7f9b5c06d5a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b700c2e80 0x7f9b7015c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b700c24c0 0x7f9b700c2940 unknown :-1 s=CLOSED pgs=2878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 --2- 172.21.15.29:0/3795162759 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b58005180 0x7f9b700c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 >> 172.21.15.29:0/3795162759 conn(0x7f9b700cb410 msgr2=0x7f9b700cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 shutdown_connections 2023-10-14T18:18:39.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.672+0000 7f9b76978640 1 -- 172.21.15.29:0/3795162759 wait complete. 2023-10-14T18:18:39.693 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:18:39.693 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:18:39.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.913+0000 7fa7f6561640 1 Processor -- start 2023-10-14T18:18:39.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.913+0000 7fa7f6561640 1 -- start start 2023-10-14T18:18:39.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7f6561640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7f0159b20 0x7fa7f006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7f6561640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f006f7f0 con 0x7fa7f0159f00 2023-10-14T18:18:39.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7f6561640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f006c5e0 con 0x7fa7f00b0b20 2023-10-14T18:18:39.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7f6561640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f006c8c0 con 0x7fa7f0159b20 2023-10-14T18:18:39.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7effff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa7f0159b20 0x7fa7f006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7e77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa7f0159f00 0x7fa7f00b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:38352/0 (socket says 172.21.15.29:38352) 2023-10-14T18:18:39.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.915+0000 7fa7e77fe640 1 -- 172.21.15.29:0/1900037846 learned_addr learned my addr 172.21.15.29:0/1900037846 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:39.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.916+0000 7fa7effff640 1 -- 172.21.15.29:0/1900037846 >> v1:172.21.15.186:6789/0 conn(0x7fa7f00b0b20 legacy=0x7fa7f006fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:39.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.916+0000 7fa7effff640 1 -- 172.21.15.29:0/1900037846 >> v1:172.21.15.29:6789/0 conn(0x7fa7f0159f00 legacy=0x7fa7f00b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.916+0000 7fa7effff640 1 -- 172.21.15.29:0/1900037846 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f006cba0 con 0x7fa7f0159b20 2023-10-14T18:18:39.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.917+0000 7fa7effff640 1 --2- 172.21.15.29:0/1900037846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159b20 0x7fa7f006bbe0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097a0 tx=0x7fa7e002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f96ec6a649b0719f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.917+0000 7fa7ef7fe640 1 -- 172.21.15.29:0/1900037846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031830 con 0x7fa7f0159b20 2023-10-14T18:18:39.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.917+0000 7fa7ef7fe640 1 -- 172.21.15.29:0/1900037846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e0031990 con 0x7fa7f0159b20 2023-10-14T18:18:39.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.917+0000 7fa7ef7fe640 1 -- 172.21.15.29:0/1900037846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e0031d20 con 0x7fa7f0159b20 2023-10-14T18:18:39.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 -- 172.21.15.29:0/1900037846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159b20 msgr2=0x7fa7f006bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 --2- 172.21.15.29:0/1900037846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159b20 0x7fa7f006bbe0 secure :-1 s=READY pgs=2880 cs=0 l=1 rev1=1 crypto rx=0x7fa7e00097a0 tx=0x7fa7e002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 -- 172.21.15.29:0/1900037846 shutdown_connections 2023-10-14T18:18:39.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 --2- 172.21.15.29:0/1900037846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159b20 0x7fa7f006bbe0 unknown :-1 s=CLOSED pgs=2880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 -- 172.21.15.29:0/1900037846 >> 172.21.15.29:0/1900037846 conn(0x7fa7f00cb410 msgr2=0x7fa7f00cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:39.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 -- 172.21.15.29:0/1900037846 shutdown_connections 2023-10-14T18:18:39.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.918+0000 7fa7f6561640 1 -- 172.21.15.29:0/1900037846 wait complete. 2023-10-14T18:18:39.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.920+0000 7fa7f6561640 1 Processor -- start 2023-10-14T18:18:39.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.920+0000 7fa7f6561640 1 -- start start 2023-10-14T18:18:39.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7f00b0b20 0x7fa7f00c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 0x7fa7f00b3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f00c6f30 con 0x7fa7f0159f00 2023-10-14T18:18:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f00c62b0 con 0x7fa7f0159b20 2023-10-14T18:18:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f6561640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f00c6590 con 0x7fa7f00b0b20 2023-10-14T18:18:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f4afe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f4afe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54756/0 (socket says 172.21.15.29:54756) 2023-10-14T18:18:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.921+0000 7fa7f4afe640 1 -- 172.21.15.29:0/3216776712 learned_addr learned my addr 172.21.15.29:0/3216776712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7e77fe640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 0x7fa7f00b3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7effff640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7f00b0b20 0x7fa7f00c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 -- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7f00b0b20 msgr2=0x7fa7f00c5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7f00b0b20 0x7fa7f00c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 -- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 msgr2=0x7fa7f00b3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 0x7fa7f00b3710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 -- 172.21.15.29:0/3216776712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7e0009400 con 0x7fa7f0159f00 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7e77fe640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 0x7fa7f00b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:39.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.922+0000 7fa7f4afe640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800b410 tx=0x7fa7e800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.923+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8004020 con 0x7fa7f0159f00 2023-10-14T18:18:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.923+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e8004180 con 0x7fa7f0159f00 2023-10-14T18:18:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.923+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7e8011630 con 0x7fa7f0159f00 2023-10-14T18:18:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.923+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f00c6870 con 0x7fa7f0159f00 2023-10-14T18:18:39.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.923+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f00c5fa0 con 0x7fa7f0159f00 2023-10-14T18:18:39.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.924+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f00c7070 con 0x7fa7f0159f00 2023-10-14T18:18:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.925+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa7e8011790 con 0x7fa7f0159f00 2023-10-14T18:18:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.926+0000 7fa7ed7fa640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 0x7fa7d006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.926+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa7e8089780 con 0x7fa7f0159f00 2023-10-14T18:18:39.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.926+0000 7fa7effff640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 0x7fa7d006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.927+0000 7fa7effff640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 0x7fa7d006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e0008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:39.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:39.931+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa7f00c7070 con 0x7fa7f0159f00 2023-10-14T18:18:40.057 INFO:tasks.ceph.mds.b.smithi186.stderr:2023-10-14T18:18:40.056+0000 7f0dec32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:18:40.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.130+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa7f006ce80 con 0x7fa7d006b0b0 2023-10-14T18:18:40.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.159+0000 7fa7ed7fa640 1 -- 172.21.15.29:0/3216776712 <== mgr.7463 v2:172.21.15.29:6832/4275371095 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287236 (secure 0 0 0) 0x7fa7e0045400 con 0x7fa7d006b0b0 2023-10-14T18:18:40.164 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:40.166 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:18:40.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 msgr2=0x7fa7d006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 0x7fa7d006d570 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fa7e0036000 tx=0x7fa7e0008030 comp rx=0 tx=0).stop 2023-10-14T18:18:40.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 msgr2=0x7fa7f015a5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 secure :-1 s=READY pgs=2881 cs=0 l=1 rev1=1 crypto rx=0x7fa7e800b410 tx=0x7fa7e800b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 shutdown_connections 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa7d006b0b0 0x7fa7d006d570 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa7f0159f00 0x7fa7f015a5d0 unknown :-1 s=CLOSED pgs=2881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa7f0159b20 0x7fa7f00b3710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 --2- 172.21.15.29:0/3216776712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa7f00b0b20 0x7fa7f00c5a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 >> 172.21.15.29:0/3216776712 conn(0x7fa7f00cb410 msgr2=0x7fa7f0157b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:40.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.166+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 shutdown_connections 2023-10-14T18:18:40.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.167+0000 7fa7f6561640 1 -- 172.21.15.29:0/3216776712 wait complete. 2023-10-14T18:18:40.187 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-14T18:18:38.424876+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423688,"kb_used_data":25740,"kb_used_omap":12,"kb_used_meta":397811,"kb_avail":754551032,"statfs":{"total":773094113280,"available":772660256768,"internally_reserved":0,"allocated":26357760,"data_stored":13689668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":407359032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.814434"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274249+0000","last_change":"2023-10-14T18:18:35.681689+0000","last_active":"2023-10-14T18:18:37.274249+0000","last_peered":"2023-10-14T18:18:37.274249+0000","last_clean":"2023-10-14T18:18:37.274249+0000","last_became_active":"2023-10-14T18:18:35.681412+0000","last_became_peered":"2023-10-14T18:18:35.681412+0000","last_unstale":"2023-10-14T18:18:37.274249+0000","last_undegraded":"2023-10-14T18:18:37.274249+0000","last_fullsized":"2023-10-14T18:18:37.274249+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:30:00.717606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273882+0000","last_change":"2023-10-14T18:18:35.683822+0000","last_active":"2023-10-14T18:18:37.273882+0000","last_peered":"2023-10-14T18:18:37.273882+0000","last_clean":"2023-10-14T18:18:37.273882+0000","last_became_active":"2023-10-14T18:18:35.683492+0000","last_became_peered":"2023-10-14T18:18:35.683492+0000","last_unstale":"2023-10-14T18:18:37.273882+0000","last_undegraded":"2023-10-14T18:18:37.273882+0000","last_fullsized":"2023-10-14T18:18:37.273882+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:15.955276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374695+0000","last_change":"2023-10-14T18:18:35.668307+0000","last_active":"2023-10-14T18:18:37.374695+0000","last_peered":"2023-10-14T18:18:37.374695+0000","last_clean":"2023-10-14T18:18:37.374695+0000","last_became_active":"2023-10-14T18:18:35.667786+0000","last_became_peered":"2023-10-14T18:18:35.667786+0000","last_unstale":"2023-10-14T18:18:37.374695+0000","last_undegraded":"2023-10-14T18:18:37.374695+0000","last_fullsized":"2023-10-14T18:18:37.374695+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:51.991520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644410+0000","last_change":"2023-10-14T18:18:35.663594+0000","last_active":"2023-10-14T18:18:36.644410+0000","last_peered":"2023-10-14T18:18:36.644410+0000","last_clean":"2023-10-14T18:18:36.644410+0000","last_became_active":"2023-10-14T18:18:35.663236+0000","last_became_peered":"2023-10-14T18:18:35.663236+0000","last_unstale":"2023-10-14T18:18:36.644410+0000","last_undegraded":"2023-10-14T18:18:36.644410+0000","last_fullsized":"2023-10-14T18:18:36.644410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:56.542420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372556+0000","last_change":"2023-10-14T18:18:35.670764+0000","last_active":"2023-10-14T18:18:37.372556+0000","last_peered":"2023-10-14T18:18:37.372556+0000","last_clean":"2023-10-14T18:18:37.372556+0000","last_became_active":"2023-10-14T18:18:35.670354+0000","last_became_peered":"2023-10-14T18:18:35.670354+0000","last_unstale":"2023-10-14T18:18:37.372556+0000","last_undegraded":"2023-10-14T18:18:37.372556+0000","last_fullsized":"2023-10-14T18:18:37.372556+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:43.950584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644608+0000","last_change":"2023-10-14T18:18:35.682936+0000","last_active":"2023-10-14T18:18:36.644608+0000","last_peered":"2023-10-14T18:18:36.644608+0000","last_clean":"2023-10-14T18:18:36.644608+0000","last_became_active":"2023-10-14T18:18:35.682060+0000","last_became_peered":"2023-10-14T18:18:35.682060+0000","last_unstale":"2023-10-14T18:18:36.644608+0000","last_undegraded":"2023-10-14T18:18:36.644608+0000","last_fullsized":"2023-10-14T18:18:36.644608+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:31.696209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651282+0000","last_change":"2023-10-14T18:18:35.676700+0000","last_active":"2023-10-14T18:18:36.651282+0000","last_peered":"2023-10-14T18:18:36.651282+0000","last_clean":"2023-10-14T18:18:36.651282+0000","last_became_active":"2023-10-14T18:18:35.676423+0000","last_became_peered":"2023-10-14T18:18:35.676423+0000","last_unstale":"2023-10-14T18:18:36.651282+0000","last_undegraded":"2023-10-14T18:18:36.651282+0000","last_fullsized":"2023-10-14T18:18:36.651282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649693+0000","last_change":"2023-10-14T18:18:35.664633+0000","last_active":"2023-10-14T18:18:36.649693+0000","last_peered":"2023-10-14T18:18:36.649693+0000","last_clean":"2023-10-14T18:18:36.649693+0000","last_became_active":"2023-10-14T18:18:35.664248+0000","last_became_peered":"2023-10-14T18:18:35.664248+0000","last_unstale":"2023-10-14T18:18:36.649693+0000","last_undegraded":"2023-10-14T18:18:36.649693+0000","last_fullsized":"2023-10-14T18:18:36.649693+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:21.318470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373806+0000","last_change":"2023-10-14T18:18:35.679660+0000","last_active":"2023-10-14T18:18:37.373806+0000","last_peered":"2023-10-14T18:18:37.373806+0000","last_clean":"2023-10-14T18:18:37.373806+0000","last_became_active":"2023-10-14T18:18:35.679374+0000","last_became_peered":"2023-10-14T18:18:35.679374+0000","last_unstale":"2023-10-14T18:18:37.373806+0000","last_undegraded":"2023-10-14T18:18:37.373806+0000","last_fullsized":"2023-10-14T18:18:37.373806+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:57.315103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654092+0000","last_change":"2023-10-14T18:18:35.679829+0000","last_active":"2023-10-14T18:18:36.654092+0000","last_peered":"2023-10-14T18:18:36.654092+0000","last_clean":"2023-10-14T18:18:36.654092+0000","last_became_active":"2023-10-14T18:18:35.679412+0000","last_became_peered":"2023-10-14T18:18:35.679412+0000","last_unstale":"2023-10-14T18:18:36.654092+0000","last_undegraded":"2023-10-14T18:18:36.654092+0000","last_fullsized":"2023-10-14T18:18:36.654092+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375596+0000","last_change":"2023-10-14T18:18:35.689722+0000","last_active":"2023-10-14T18:18:37.375596+0000","last_peered":"2023-10-14T18:18:37.375596+0000","last_clean":"2023-10-14T18:18:37.375596+0000","last_became_active":"2023-10-14T18:18:35.686665+0000","last_became_peered":"2023-10-14T18:18:35.686665+0000","last_unstale":"2023-10-14T18:18:37.375596+0000","last_undegraded":"2023-10-14T18:18:37.375596+0000","last_fullsized":"2023-10-14T18:18:37.375596+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:57.853905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651148+0000","last_change":"2023-10-14T18:18:35.674443+0000","last_active":"2023-10-14T18:18:36.651148+0000","last_peered":"2023-10-14T18:18:36.651148+0000","last_clean":"2023-10-14T18:18:36.651148+0000","last_became_active":"2023-10-14T18:18:35.673655+0000","last_became_peered":"2023-10-14T18:18:35.673655+0000","last_unstale":"2023-10-14T18:18:36.651148+0000","last_undegraded":"2023-10-14T18:18:36.651148+0000","last_fullsized":"2023-10-14T18:18:36.651148+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653672+0000","last_change":"2023-10-14T18:18:35.687221+0000","last_active":"2023-10-14T18:18:36.653672+0000","last_peered":"2023-10-14T18:18:36.653672+0000","last_clean":"2023-10-14T18:18:36.653672+0000","last_became_active":"2023-10-14T18:18:35.686900+0000","last_became_peered":"2023-10-14T18:18:35.686900+0000","last_unstale":"2023-10-14T18:18:36.653672+0000","last_undegraded":"2023-10-14T18:18:36.653672+0000","last_fullsized":"2023-10-14T18:18:36.653672+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273360+0000","last_change":"2023-10-14T18:18:35.671605+0000","last_active":"2023-10-14T18:18:37.273360+0000","last_peered":"2023-10-14T18:18:37.273360+0000","last_clean":"2023-10-14T18:18:37.273360+0000","last_became_active":"2023-10-14T18:18:35.671317+0000","last_became_peered":"2023-10-14T18:18:35.671317+0000","last_unstale":"2023-10-14T18:18:37.273360+0000","last_undegraded":"2023-10-14T18:18:37.273360+0000","last_fullsized":"2023-10-14T18:18:37.273360+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:14.496902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373037+0000","last_change":"2023-10-14T18:18:35.682877+0000","last_active":"2023-10-14T18:18:37.373037+0000","last_peered":"2023-10-14T18:18:37.373037+0000","last_clean":"2023-10-14T18:18:37.373037+0000","last_became_active":"2023-10-14T18:18:35.682389+0000","last_became_peered":"2023-10-14T18:18:35.682389+0000","last_unstale":"2023-10-14T18:18:37.373037+0000","last_undegraded":"2023-10-14T18:18:37.373037+0000","last_fullsized":"2023-10-14T18:18:37.373037+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:25.798127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373367+0000","last_change":"2023-10-14T18:18:35.674408+0000","last_active":"2023-10-14T18:18:37.373367+0000","last_peered":"2023-10-14T18:18:37.373367+0000","last_clean":"2023-10-14T18:18:37.373367+0000","last_became_active":"2023-10-14T18:18:35.674062+0000","last_became_peered":"2023-10-14T18:18:35.674062+0000","last_unstale":"2023-10-14T18:18:37.373367+0000","last_undegraded":"2023-10-14T18:18:37.373367+0000","last_fullsized":"2023-10-14T18:18:37.373367+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:16.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652705+0000","last_change":"2023-10-14T18:18:35.678586+0000","last_active":"2023-10-14T18:18:36.652705+0000","last_peered":"2023-10-14T18:18:36.652705+0000","last_clean":"2023-10-14T18:18:36.652705+0000","last_became_active":"2023-10-14T18:18:35.678265+0000","last_became_peered":"2023-10-14T18:18:35.678265+0000","last_unstale":"2023-10-14T18:18:36.652705+0000","last_undegraded":"2023-10-14T18:18:36.652705+0000","last_fullsized":"2023-10-14T18:18:36.652705+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:25.316716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274015+0000","last_change":"2023-10-14T18:18:35.679271+0000","last_active":"2023-10-14T18:18:37.274015+0000","last_peered":"2023-10-14T18:18:37.274015+0000","last_clean":"2023-10-14T18:18:37.274015+0000","last_became_active":"2023-10-14T18:18:35.678924+0000","last_became_peered":"2023-10-14T18:18:35.678924+0000","last_unstale":"2023-10-14T18:18:37.274015+0000","last_undegraded":"2023-10-14T18:18:37.274015+0000","last_fullsized":"2023-10-14T18:18:37.274015+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:56.833692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651918+0000","last_change":"2023-10-14T18:18:35.683679+0000","last_active":"2023-10-14T18:18:36.651918+0000","last_peered":"2023-10-14T18:18:36.651918+0000","last_clean":"2023-10-14T18:18:36.651918+0000","last_became_active":"2023-10-14T18:18:35.682861+0000","last_became_peered":"2023-10-14T18:18:35.682861+0000","last_unstale":"2023-10-14T18:18:36.651918+0000","last_undegraded":"2023-10-14T18:18:36.651918+0000","last_fullsized":"2023-10-14T18:18:36.651918+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375268+0000","last_change":"2023-10-14T18:18:35.675043+0000","last_active":"2023-10-14T18:18:37.375268+0000","last_peered":"2023-10-14T18:18:37.375268+0000","last_clean":"2023-10-14T18:18:37.375268+0000","last_became_active":"2023-10-14T18:18:35.674231+0000","last_became_peered":"2023-10-14T18:18:35.674231+0000","last_unstale":"2023-10-14T18:18:37.375268+0000","last_undegraded":"2023-10-14T18:18:37.375268+0000","last_fullsized":"2023-10-14T18:18:37.375268+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:11:18.531967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650886+0000","last_change":"2023-10-14T18:18:35.678709+0000","last_active":"2023-10-14T18:18:36.650886+0000","last_peered":"2023-10-14T18:18:36.650886+0000","last_clean":"2023-10-14T18:18:36.650886+0000","last_became_active":"2023-10-14T18:18:35.678445+0000","last_became_peered":"2023-10-14T18:18:35.678445+0000","last_unstale":"2023-10-14T18:18:36.650886+0000","last_undegraded":"2023-10-14T18:18:36.650886+0000","last_fullsized":"2023-10-14T18:18:36.650886+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653036+0000","last_change":"2023-10-14T18:18:35.678715+0000","last_active":"2023-10-14T18:18:36.653036+0000","last_peered":"2023-10-14T18:18:36.653036+0000","last_clean":"2023-10-14T18:18:36.653036+0000","last_became_active":"2023-10-14T18:18:35.678366+0000","last_became_peered":"2023-10-14T18:18:35.678366+0000","last_unstale":"2023-10-14T18:18:36.653036+0000","last_undegraded":"2023-10-14T18:18:36.653036+0000","last_fullsized":"2023-10-14T18:18:36.653036+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649759+0000","last_change":"2023-10-14T18:18:35.674737+0000","last_active":"2023-10-14T18:18:36.649759+0000","last_peered":"2023-10-14T18:18:36.649759+0000","last_clean":"2023-10-14T18:18:36.649759+0000","last_became_active":"2023-10-14T18:18:35.673762+0000","last_became_peered":"2023-10-14T18:18:35.673762+0000","last_unstale":"2023-10-14T18:18:36.649759+0000","last_undegraded":"2023-10-14T18:18:36.649759+0000","last_fullsized":"2023-10-14T18:18:36.649759+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273276+0000","last_change":"2023-10-14T18:18:35.679931+0000","last_active":"2023-10-14T18:18:37.273276+0000","last_peered":"2023-10-14T18:18:37.273276+0000","last_clean":"2023-10-14T18:18:37.273276+0000","last_became_active":"2023-10-14T18:18:35.679636+0000","last_became_peered":"2023-10-14T18:18:35.679636+0000","last_unstale":"2023-10-14T18:18:37.273276+0000","last_undegraded":"2023-10-14T18:18:37.273276+0000","last_fullsized":"2023-10-14T18:18:37.273276+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:40.768705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644927+0000","last_change":"2023-10-14T18:18:35.663686+0000","last_active":"2023-10-14T18:18:36.644927+0000","last_peered":"2023-10-14T18:18:36.644927+0000","last_clean":"2023-10-14T18:18:36.644927+0000","last_became_active":"2023-10-14T18:18:35.663364+0000","last_became_peered":"2023-10-14T18:18:35.663364+0000","last_unstale":"2023-10-14T18:18:36.644927+0000","last_undegraded":"2023-10-14T18:18:36.644927+0000","last_fullsized":"2023-10-14T18:18:36.644927+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:43.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654128+0000","last_change":"2023-10-14T18:18:35.683088+0000","last_active":"2023-10-14T18:18:36.654128+0000","last_peered":"2023-10-14T18:18:36.654128+0000","last_clean":"2023-10-14T18:18:36.654128+0000","last_became_active":"2023-10-14T18:18:35.682753+0000","last_became_peered":"2023-10-14T18:18:35.682753+0000","last_unstale":"2023-10-14T18:18:36.654128+0000","last_undegraded":"2023-10-14T18:18:36.654128+0000","last_fullsized":"2023-10-14T18:18:36.654128+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375974+0000","last_change":"2023-10-14T18:18:35.689155+0000","last_active":"2023-10-14T18:18:37.375974+0000","last_peered":"2023-10-14T18:18:37.375974+0000","last_clean":"2023-10-14T18:18:37.375974+0000","last_became_active":"2023-10-14T18:18:35.686147+0000","last_became_peered":"2023-10-14T18:18:35.686147+0000","last_unstale":"2023-10-14T18:18:37.375974+0000","last_undegraded":"2023-10-14T18:18:37.375974+0000","last_fullsized":"2023-10-14T18:18:37.375974+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:12.203818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273778+0000","last_change":"2023-10-14T18:18:35.673716+0000","last_active":"2023-10-14T18:18:37.273778+0000","last_peered":"2023-10-14T18:18:37.273778+0000","last_clean":"2023-10-14T18:18:37.273778+0000","last_became_active":"2023-10-14T18:18:35.672662+0000","last_became_peered":"2023-10-14T18:18:35.672662+0000","last_unstale":"2023-10-14T18:18:37.273778+0000","last_undegraded":"2023-10-14T18:18:37.273778+0000","last_fullsized":"2023-10-14T18:18:37.273778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:06.125956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374334+0000","last_change":"2023-10-14T18:18:35.682800+0000","last_active":"2023-10-14T18:18:37.374334+0000","last_peered":"2023-10-14T18:18:37.374334+0000","last_clean":"2023-10-14T18:18:37.374334+0000","last_became_active":"2023-10-14T18:18:35.682478+0000","last_became_peered":"2023-10-14T18:18:35.682478+0000","last_unstale":"2023-10-14T18:18:37.374334+0000","last_undegraded":"2023-10-14T18:18:37.374334+0000","last_fullsized":"2023-10-14T18:18:37.374334+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372354+0000","last_change":"2023-10-14T18:18:35.676776+0000","last_active":"2023-10-14T18:18:37.372354+0000","last_peered":"2023-10-14T18:18:37.372354+0000","last_clean":"2023-10-14T18:18:37.372354+0000","last_became_active":"2023-10-14T18:18:35.675446+0000","last_became_peered":"2023-10-14T18:18:35.675446+0000","last_unstale":"2023-10-14T18:18:37.372354+0000","last_undegraded":"2023-10-14T18:18:37.372354+0000","last_fullsized":"2023-10-14T18:18:37.372354+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:09.420076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644229+0000","last_change":"2023-10-14T18:18:35.676664+0000","last_active":"2023-10-14T18:18:36.644229+0000","last_peered":"2023-10-14T18:18:36.644229+0000","last_clean":"2023-10-14T18:18:36.644229+0000","last_became_active":"2023-10-14T18:18:35.676381+0000","last_became_peered":"2023-10-14T18:18:35.676381+0000","last_unstale":"2023-10-14T18:18:36.644229+0000","last_undegraded":"2023-10-14T18:18:36.644229+0000","last_fullsized":"2023-10-14T18:18:36.644229+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:23.246857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274107+0000","last_change":"2023-10-14T18:18:35.670711+0000","last_active":"2023-10-14T18:18:37.274107+0000","last_peered":"2023-10-14T18:18:37.274107+0000","last_clean":"2023-10-14T18:18:37.274107+0000","last_became_active":"2023-10-14T18:18:35.670350+0000","last_became_peered":"2023-10-14T18:18:35.670350+0000","last_unstale":"2023-10-14T18:18:37.274107+0000","last_undegraded":"2023-10-14T18:18:37.274107+0000","last_fullsized":"2023-10-14T18:18:37.274107+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:32.369325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375461+0000","last_change":"2023-10-14T18:18:35.677418+0000","last_active":"2023-10-14T18:18:37.375461+0000","last_peered":"2023-10-14T18:18:37.375461+0000","last_clean":"2023-10-14T18:18:37.375461+0000","last_became_active":"2023-10-14T18:18:35.677059+0000","last_became_peered":"2023-10-14T18:18:35.677059+0000","last_unstale":"2023-10-14T18:18:37.375461+0000","last_undegraded":"2023-10-14T18:18:37.375461+0000","last_fullsized":"2023-10-14T18:18:37.375461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.342329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654400+0000","last_change":"2023-10-14T18:18:34.650335+0000","last_active":"2023-10-14T18:18:36.654400+0000","last_peered":"2023-10-14T18:18:36.654400+0000","last_clean":"2023-10-14T18:18:36.654400+0000","last_became_active":"2023-10-14T18:18:34.648549+0000","last_became_peered":"2023-10-14T18:18:34.648549+0000","last_unstale":"2023-10-14T18:18:36.654400+0000","last_undegraded":"2023-10-14T18:18:36.654400+0000","last_fullsized":"2023-10-14T18:18:36.654400+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374469+0000","last_change":"2023-10-14T18:18:35.684529+0000","last_active":"2023-10-14T18:18:37.374469+0000","last_peered":"2023-10-14T18:18:37.374469+0000","last_clean":"2023-10-14T18:18:37.374469+0000","last_became_active":"2023-10-14T18:18:35.684263+0000","last_became_peered":"2023-10-14T18:18:35.684263+0000","last_unstale":"2023-10-14T18:18:37.374469+0000","last_undegraded":"2023-10-14T18:18:37.374469+0000","last_fullsized":"2023-10-14T18:18:37.374469+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:06.771665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374880+0000","last_change":"2023-10-14T18:18:34.653624+0000","last_active":"2023-10-14T18:18:37.374880+0000","last_peered":"2023-10-14T18:18:37.374880+0000","last_clean":"2023-10-14T18:18:37.374880+0000","last_became_active":"2023-10-14T18:18:34.652877+0000","last_became_peered":"2023-10-14T18:18:34.652877+0000","last_unstale":"2023-10-14T18:18:37.374880+0000","last_undegraded":"2023-10-14T18:18:37.374880+0000","last_fullsized":"2023-10-14T18:18:37.374880+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:13.266689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652477+0000","last_change":"2023-10-14T18:18:35.671984+0000","last_active":"2023-10-14T18:18:36.652477+0000","last_peered":"2023-10-14T18:18:36.652477+0000","last_clean":"2023-10-14T18:18:36.652477+0000","last_became_active":"2023-10-14T18:18:35.668766+0000","last_became_peered":"2023-10-14T18:18:35.668766+0000","last_unstale":"2023-10-14T18:18:36.652477+0000","last_undegraded":"2023-10-14T18:18:36.652477+0000","last_fullsized":"2023-10-14T18:18:36.652477+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375523+0000","last_change":"2023-10-14T18:18:34.649372+0000","last_active":"2023-10-14T18:18:37.375523+0000","last_peered":"2023-10-14T18:18:37.375523+0000","last_clean":"2023-10-14T18:18:37.375523+0000","last_became_active":"2023-10-14T18:18:34.648998+0000","last_became_peered":"2023-10-14T18:18:34.648998+0000","last_unstale":"2023-10-14T18:18:37.375523+0000","last_undegraded":"2023-10-14T18:18:37.375523+0000","last_fullsized":"2023-10-14T18:18:37.375523+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:24.567915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373897+0000","last_change":"2023-10-14T18:18:35.672986+0000","last_active":"2023-10-14T18:18:37.373897+0000","last_peered":"2023-10-14T18:18:37.373897+0000","last_clean":"2023-10-14T18:18:37.373897+0000","last_became_active":"2023-10-14T18:18:35.672683+0000","last_became_peered":"2023-10-14T18:18:35.672683+0000","last_unstale":"2023-10-14T18:18:37.373897+0000","last_undegraded":"2023-10-14T18:18:37.373897+0000","last_fullsized":"2023-10-14T18:18:37.373897+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:59.451214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644245+0000","last_change":"2023-10-14T18:18:34.651378+0000","last_active":"2023-10-14T18:18:36.644245+0000","last_peered":"2023-10-14T18:18:36.644245+0000","last_clean":"2023-10-14T18:18:36.644245+0000","last_became_active":"2023-10-14T18:18:34.651072+0000","last_became_peered":"2023-10-14T18:18:34.651072+0000","last_unstale":"2023-10-14T18:18:36.644245+0000","last_undegraded":"2023-10-14T18:18:36.644245+0000","last_fullsized":"2023-10-14T18:18:36.644245+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:07:46.772700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373278+0000","last_change":"2023-10-14T18:18:35.691610+0000","last_active":"2023-10-14T18:18:37.373278+0000","last_peered":"2023-10-14T18:18:37.373278+0000","last_clean":"2023-10-14T18:18:37.373278+0000","last_became_active":"2023-10-14T18:18:35.691317+0000","last_became_peered":"2023-10-14T18:18:35.691317+0000","last_unstale":"2023-10-14T18:18:37.373278+0000","last_undegraded":"2023-10-14T18:18:37.373278+0000","last_fullsized":"2023-10-14T18:18:37.373278+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:24.619829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273569+0000","last_change":"2023-10-14T18:18:34.652803+0000","last_active":"2023-10-14T18:18:37.273569+0000","last_peered":"2023-10-14T18:18:37.273569+0000","last_clean":"2023-10-14T18:18:37.273569+0000","last_became_active":"2023-10-14T18:18:34.647316+0000","last_became_peered":"2023-10-14T18:18:34.647316+0000","last_unstale":"2023-10-14T18:18:37.273569+0000","last_undegraded":"2023-10-14T18:18:37.273569+0000","last_fullsized":"2023-10-14T18:18:37.273569+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:00.238815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372926+0000","last_change":"2023-10-14T18:18:35.678351+0000","last_active":"2023-10-14T18:18:37.372926+0000","last_peered":"2023-10-14T18:18:37.372926+0000","last_clean":"2023-10-14T18:18:37.372926+0000","last_became_active":"2023-10-14T18:18:35.678058+0000","last_became_peered":"2023-10-14T18:18:35.678058+0000","last_unstale":"2023-10-14T18:18:37.372926+0000","last_undegraded":"2023-10-14T18:18:37.372926+0000","last_fullsized":"2023-10-14T18:18:37.372926+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:50.521263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652298+0000","last_change":"2023-10-14T18:18:34.651512+0000","last_active":"2023-10-14T18:18:36.652298+0000","last_peered":"2023-10-14T18:18:36.652298+0000","last_clean":"2023-10-14T18:18:36.652298+0000","last_became_active":"2023-10-14T18:18:34.651205+0000","last_became_peered":"2023-10-14T18:18:34.651205+0000","last_unstale":"2023-10-14T18:18:36.652298+0000","last_undegraded":"2023-10-14T18:18:36.652298+0000","last_fullsized":"2023-10-14T18:18:36.652298+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:11.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273362+0000","last_change":"2023-10-14T18:18:35.677057+0000","last_active":"2023-10-14T18:18:37.273362+0000","last_peered":"2023-10-14T18:18:37.273362+0000","last_clean":"2023-10-14T18:18:37.273362+0000","last_became_active":"2023-10-14T18:18:35.676751+0000","last_became_peered":"2023-10-14T18:18:35.676751+0000","last_unstale":"2023-10-14T18:18:37.273362+0000","last_undegraded":"2023-10-14T18:18:37.273362+0000","last_fullsized":"2023-10-14T18:18:37.273362+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:14.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373180+0000","last_change":"2023-10-14T18:18:34.645198+0000","last_active":"2023-10-14T18:18:37.373180+0000","last_peered":"2023-10-14T18:18:37.373180+0000","last_clean":"2023-10-14T18:18:37.373180+0000","last_became_active":"2023-10-14T18:18:34.644896+0000","last_became_peered":"2023-10-14T18:18:34.644896+0000","last_unstale":"2023-10-14T18:18:37.373180+0000","last_undegraded":"2023-10-14T18:18:37.373180+0000","last_fullsized":"2023-10-14T18:18:37.373180+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:17.263536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274102+0000","last_change":"2023-10-14T18:18:35.675636+0000","last_active":"2023-10-14T18:18:37.274102+0000","last_peered":"2023-10-14T18:18:37.274102+0000","last_clean":"2023-10-14T18:18:37.274102+0000","last_became_active":"2023-10-14T18:18:35.674527+0000","last_became_peered":"2023-10-14T18:18:35.674527+0000","last_unstale":"2023-10-14T18:18:37.274102+0000","last_undegraded":"2023-10-14T18:18:37.274102+0000","last_fullsized":"2023-10-14T18:18:37.274102+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:19.471165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652927+0000","last_change":"2023-10-14T18:18:34.640892+0000","last_active":"2023-10-14T18:18:36.652927+0000","last_peered":"2023-10-14T18:18:36.652927+0000","last_clean":"2023-10-14T18:18:36.652927+0000","last_became_active":"2023-10-14T18:18:34.639779+0000","last_became_peered":"2023-10-14T18:18:34.639779+0000","last_unstale":"2023-10-14T18:18:36.652927+0000","last_undegraded":"2023-10-14T18:18:36.652927+0000","last_fullsized":"2023-10-14T18:18:36.652927+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.376500+0000","last_change":"2023-10-14T18:18:35.685195+0000","last_active":"2023-10-14T18:18:37.376500+0000","last_peered":"2023-10-14T18:18:37.376500+0000","last_clean":"2023-10-14T18:18:37.376500+0000","last_became_active":"2023-10-14T18:18:35.682690+0000","last_became_peered":"2023-10-14T18:18:35.682690+0000","last_unstale":"2023-10-14T18:18:37.376500+0000","last_undegraded":"2023-10-14T18:18:37.376500+0000","last_fullsized":"2023-10-14T18:18:37.376500+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:54.132242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374719+0000","last_change":"2023-10-14T18:18:34.651854+0000","last_active":"2023-10-14T18:18:37.374719+0000","last_peered":"2023-10-14T18:18:37.374719+0000","last_clean":"2023-10-14T18:18:37.374719+0000","last_became_active":"2023-10-14T18:18:34.651540+0000","last_became_peered":"2023-10-14T18:18:34.651540+0000","last_unstale":"2023-10-14T18:18:37.374719+0000","last_undegraded":"2023-10-14T18:18:37.374719+0000","last_fullsized":"2023-10-14T18:18:37.374719+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.752351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375329+0000","last_change":"2023-10-14T18:18:35.679788+0000","last_active":"2023-10-14T18:18:37.375329+0000","last_peered":"2023-10-14T18:18:37.375329+0000","last_clean":"2023-10-14T18:18:37.375329+0000","last_became_active":"2023-10-14T18:18:35.679488+0000","last_became_peered":"2023-10-14T18:18:35.679488+0000","last_unstale":"2023-10-14T18:18:37.375329+0000","last_undegraded":"2023-10-14T18:18:37.375329+0000","last_fullsized":"2023-10-14T18:18:37.375329+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:39.475602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274265+0000","last_change":"2023-10-14T18:18:34.654465+0000","last_active":"2023-10-14T18:18:37.274265+0000","last_peered":"2023-10-14T18:18:37.274265+0000","last_clean":"2023-10-14T18:18:37.274265+0000","last_became_active":"2023-10-14T18:18:34.654111+0000","last_became_peered":"2023-10-14T18:18:34.654111+0000","last_unstale":"2023-10-14T18:18:37.274265+0000","last_undegraded":"2023-10-14T18:18:37.274265+0000","last_fullsized":"2023-10-14T18:18:37.274265+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:01.589582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653831+0000","last_change":"2023-10-14T18:18:35.683637+0000","last_active":"2023-10-14T18:18:36.653831+0000","last_peered":"2023-10-14T18:18:36.653831+0000","last_clean":"2023-10-14T18:18:36.653831+0000","last_became_active":"2023-10-14T18:18:35.683235+0000","last_became_peered":"2023-10-14T18:18:35.683235+0000","last_unstale":"2023-10-14T18:18:36.653831+0000","last_undegraded":"2023-10-14T18:18:36.653831+0000","last_fullsized":"2023-10-14T18:18:36.653831+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"249'6","reported_seq":18,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273702+0000","last_change":"2023-10-14T18:18:34.652609+0000","last_active":"2023-10-14T18:18:37.273702+0000","last_peered":"2023-10-14T18:18:37.273702+0000","last_clean":"2023-10-14T18:18:37.273702+0000","last_became_active":"2023-10-14T18:18:34.646287+0000","last_became_peered":"2023-10-14T18:18:34.646287+0000","last_unstale":"2023-10-14T18:18:37.273702+0000","last_undegraded":"2023-10-14T18:18:37.273702+0000","last_fullsized":"2023-10-14T18:18:37.273702+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:09.725037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653135+0000","last_change":"2023-10-14T18:18:35.672404+0000","last_active":"2023-10-14T18:18:36.653135+0000","last_peered":"2023-10-14T18:18:36.653135+0000","last_clean":"2023-10-14T18:18:36.653135+0000","last_became_active":"2023-10-14T18:18:35.672109+0000","last_became_peered":"2023-10-14T18:18:35.672109+0000","last_unstale":"2023-10-14T18:18:36.653135+0000","last_undegraded":"2023-10-14T18:18:36.653135+0000","last_fullsized":"2023-10-14T18:18:36.653135+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273692+0000","last_change":"2023-10-14T18:18:34.650921+0000","last_active":"2023-10-14T18:18:37.273692+0000","last_peered":"2023-10-14T18:18:37.273692+0000","last_clean":"2023-10-14T18:18:37.273692+0000","last_became_active":"2023-10-14T18:18:34.650170+0000","last_became_peered":"2023-10-14T18:18:34.650170+0000","last_unstale":"2023-10-14T18:18:37.273692+0000","last_undegraded":"2023-10-14T18:18:37.273692+0000","last_fullsized":"2023-10-14T18:18:37.273692+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:24.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373852+0000","last_change":"2023-10-14T18:18:35.685387+0000","last_active":"2023-10-14T18:18:37.373852+0000","last_peered":"2023-10-14T18:18:37.373852+0000","last_clean":"2023-10-14T18:18:37.373852+0000","last_became_active":"2023-10-14T18:18:35.684624+0000","last_became_peered":"2023-10-14T18:18:35.684624+0000","last_unstale":"2023-10-14T18:18:37.373852+0000","last_undegraded":"2023-10-14T18:18:37.373852+0000","last_fullsized":"2023-10-14T18:18:37.373852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:49.959769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374525+0000","last_change":"2023-10-14T18:18:34.644335+0000","last_active":"2023-10-14T18:18:37.374525+0000","last_peered":"2023-10-14T18:18:37.374525+0000","last_clean":"2023-10-14T18:18:37.374525+0000","last_became_active":"2023-10-14T18:18:34.644052+0000","last_became_peered":"2023-10-14T18:18:34.644052+0000","last_unstale":"2023-10-14T18:18:37.374525+0000","last_undegraded":"2023-10-14T18:18:37.374525+0000","last_fullsized":"2023-10-14T18:18:37.374525+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:00.968233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644478+0000","last_change":"2023-10-14T18:18:35.675619+0000","last_active":"2023-10-14T18:18:36.644478+0000","last_peered":"2023-10-14T18:18:36.644478+0000","last_clean":"2023-10-14T18:18:36.644478+0000","last_became_active":"2023-10-14T18:18:35.673589+0000","last_became_peered":"2023-10-14T18:18:35.673589+0000","last_unstale":"2023-10-14T18:18:36.644478+0000","last_undegraded":"2023-10-14T18:18:36.644478+0000","last_fullsized":"2023-10-14T18:18:36.644478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:37.559321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652661+0000","last_change":"2023-10-14T18:18:34.646943+0000","last_active":"2023-10-14T18:18:36.652661+0000","last_peered":"2023-10-14T18:18:36.652661+0000","last_clean":"2023-10-14T18:18:36.652661+0000","last_became_active":"2023-10-14T18:18:34.646630+0000","last_became_peered":"2023-10-14T18:18:34.646630+0000","last_unstale":"2023-10-14T18:18:36.652661+0000","last_undegraded":"2023-10-14T18:18:36.652661+0000","last_fullsized":"2023-10-14T18:18:36.652661+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:57.489531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651630+0000","last_change":"2023-10-14T18:18:35.678311+0000","last_active":"2023-10-14T18:18:36.651630+0000","last_peered":"2023-10-14T18:18:36.651630+0000","last_clean":"2023-10-14T18:18:36.651630+0000","last_became_active":"2023-10-14T18:18:35.678016+0000","last_became_peered":"2023-10-14T18:18:35.678016+0000","last_unstale":"2023-10-14T18:18:36.651630+0000","last_undegraded":"2023-10-14T18:18:36.651630+0000","last_fullsized":"2023-10-14T18:18:36.651630+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373557+0000","last_change":"2023-10-14T18:18:34.648173+0000","last_active":"2023-10-14T18:18:37.373557+0000","last_peered":"2023-10-14T18:18:37.373557+0000","last_clean":"2023-10-14T18:18:37.373557+0000","last_became_active":"2023-10-14T18:18:34.647904+0000","last_became_peered":"2023-10-14T18:18:34.647904+0000","last_unstale":"2023-10-14T18:18:37.373557+0000","last_undegraded":"2023-10-14T18:18:37.373557+0000","last_fullsized":"2023-10-14T18:18:37.373557+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:49.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644809+0000","last_change":"2023-10-14T18:18:35.672233+0000","last_active":"2023-10-14T18:18:36.644809+0000","last_peered":"2023-10-14T18:18:36.644809+0000","last_clean":"2023-10-14T18:18:36.644809+0000","last_became_active":"2023-10-14T18:18:35.671904+0000","last_became_peered":"2023-10-14T18:18:35.671904+0000","last_unstale":"2023-10-14T18:18:36.644809+0000","last_undegraded":"2023-10-14T18:18:36.644809+0000","last_fullsized":"2023-10-14T18:18:36.644809+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:09.351543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654314+0000","last_change":"2023-10-14T18:18:34.644806+0000","last_active":"2023-10-14T18:18:36.654314+0000","last_peered":"2023-10-14T18:18:36.654314+0000","last_clean":"2023-10-14T18:18:36.654314+0000","last_became_active":"2023-10-14T18:18:34.642485+0000","last_became_peered":"2023-10-14T18:18:34.642485+0000","last_unstale":"2023-10-14T18:18:36.654314+0000","last_undegraded":"2023-10-14T18:18:36.654314+0000","last_fullsized":"2023-10-14T18:18:36.654314+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:22.234439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375522+0000","last_change":"2023-10-14T18:18:35.673621+0000","last_active":"2023-10-14T18:18:37.375522+0000","last_peered":"2023-10-14T18:18:37.375522+0000","last_clean":"2023-10-14T18:18:37.375522+0000","last_became_active":"2023-10-14T18:18:35.673296+0000","last_became_peered":"2023-10-14T18:18:35.673296+0000","last_unstale":"2023-10-14T18:18:37.375522+0000","last_undegraded":"2023-10-14T18:18:37.375522+0000","last_fullsized":"2023-10-14T18:18:37.375522+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:58:27.889006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375020+0000","last_change":"2023-10-14T18:18:34.655553+0000","last_active":"2023-10-14T18:18:37.375020+0000","last_peered":"2023-10-14T18:18:37.375020+0000","last_clean":"2023-10-14T18:18:37.375020+0000","last_became_active":"2023-10-14T18:18:34.655113+0000","last_became_peered":"2023-10-14T18:18:34.655113+0000","last_unstale":"2023-10-14T18:18:37.375020+0000","last_undegraded":"2023-10-14T18:18:37.375020+0000","last_fullsized":"2023-10-14T18:18:37.375020+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:04:45.880556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373317+0000","last_change":"2023-10-14T18:18:35.683896+0000","last_active":"2023-10-14T18:18:37.373317+0000","last_peered":"2023-10-14T18:18:37.373317+0000","last_clean":"2023-10-14T18:18:37.373317+0000","last_became_active":"2023-10-14T18:18:35.683606+0000","last_became_peered":"2023-10-14T18:18:35.683606+0000","last_unstale":"2023-10-14T18:18:37.373317+0000","last_undegraded":"2023-10-14T18:18:37.373317+0000","last_fullsized":"2023-10-14T18:18:37.373317+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:02.619684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373307+0000","last_change":"2023-10-14T18:18:34.647730+0000","last_active":"2023-10-14T18:18:37.373307+0000","last_peered":"2023-10-14T18:18:37.373307+0000","last_clean":"2023-10-14T18:18:37.373307+0000","last_became_active":"2023-10-14T18:18:34.646698+0000","last_became_peered":"2023-10-14T18:18:34.646698+0000","last_unstale":"2023-10-14T18:18:37.373307+0000","last_undegraded":"2023-10-14T18:18:37.373307+0000","last_fullsized":"2023-10-14T18:18:37.373307+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:44.897768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652299+0000","last_change":"2023-10-14T18:18:35.675449+0000","last_active":"2023-10-14T18:18:36.652299+0000","last_peered":"2023-10-14T18:18:36.652299+0000","last_clean":"2023-10-14T18:18:36.652299+0000","last_became_active":"2023-10-14T18:18:35.675129+0000","last_became_peered":"2023-10-14T18:18:35.675129+0000","last_unstale":"2023-10-14T18:18:36.652299+0000","last_undegraded":"2023-10-14T18:18:36.652299+0000","last_fullsized":"2023-10-14T18:18:36.652299+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374720+0000","last_change":"2023-10-14T18:18:34.645342+0000","last_active":"2023-10-14T18:18:37.374720+0000","last_peered":"2023-10-14T18:18:37.374720+0000","last_clean":"2023-10-14T18:18:37.374720+0000","last_became_active":"2023-10-14T18:18:34.643272+0000","last_became_peered":"2023-10-14T18:18:34.643272+0000","last_unstale":"2023-10-14T18:18:37.374720+0000","last_undegraded":"2023-10-14T18:18:37.374720+0000","last_fullsized":"2023-10-14T18:18:37.374720+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:08.339125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650684+0000","last_change":"2023-10-14T18:18:35.675593+0000","last_active":"2023-10-14T18:18:36.650684+0000","last_peered":"2023-10-14T18:18:36.650684+0000","last_clean":"2023-10-14T18:18:36.650684+0000","last_became_active":"2023-10-14T18:18:35.675294+0000","last_became_peered":"2023-10-14T18:18:35.675294+0000","last_unstale":"2023-10-14T18:18:36.650684+0000","last_undegraded":"2023-10-14T18:18:36.650684+0000","last_fullsized":"2023-10-14T18:18:36.650684+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375369+0000","last_change":"2023-10-14T18:18:34.640130+0000","last_active":"2023-10-14T18:18:37.375369+0000","last_peered":"2023-10-14T18:18:37.375369+0000","last_clean":"2023-10-14T18:18:37.375369+0000","last_became_active":"2023-10-14T18:18:34.639677+0000","last_became_peered":"2023-10-14T18:18:34.639677+0000","last_unstale":"2023-10-14T18:18:37.375369+0000","last_undegraded":"2023-10-14T18:18:37.375369+0000","last_fullsized":"2023-10-14T18:18:37.375369+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:09.948727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273394+0000","last_change":"2023-10-14T18:18:35.682891+0000","last_active":"2023-10-14T18:18:37.273394+0000","last_peered":"2023-10-14T18:18:37.273394+0000","last_clean":"2023-10-14T18:18:37.273394+0000","last_became_active":"2023-10-14T18:18:35.681440+0000","last_became_peered":"2023-10-14T18:18:35.681440+0000","last_unstale":"2023-10-14T18:18:37.273394+0000","last_undegraded":"2023-10-14T18:18:37.273394+0000","last_fullsized":"2023-10-14T18:18:37.273394+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:50.330635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373798+0000","last_change":"2023-10-14T18:18:34.647137+0000","last_active":"2023-10-14T18:18:37.373798+0000","last_peered":"2023-10-14T18:18:37.373798+0000","last_clean":"2023-10-14T18:18:37.373798+0000","last_became_active":"2023-10-14T18:18:34.646766+0000","last_became_peered":"2023-10-14T18:18:34.646766+0000","last_unstale":"2023-10-14T18:18:37.373798+0000","last_undegraded":"2023-10-14T18:18:37.373798+0000","last_fullsized":"2023-10-14T18:18:37.373798+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373613+0000","last_change":"2023-10-14T18:18:35.672939+0000","last_active":"2023-10-14T18:18:37.373613+0000","last_peered":"2023-10-14T18:18:37.373613+0000","last_clean":"2023-10-14T18:18:37.373613+0000","last_became_active":"2023-10-14T18:18:35.672630+0000","last_became_peered":"2023-10-14T18:18:35.672630+0000","last_unstale":"2023-10-14T18:18:37.373613+0000","last_undegraded":"2023-10-14T18:18:37.373613+0000","last_fullsized":"2023-10-14T18:18:37.373613+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:56.149979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373307+0000","last_change":"2023-10-14T18:18:34.653097+0000","last_active":"2023-10-14T18:18:37.373307+0000","last_peered":"2023-10-14T18:18:37.373307+0000","last_clean":"2023-10-14T18:18:37.373307+0000","last_became_active":"2023-10-14T18:18:34.652338+0000","last_became_peered":"2023-10-14T18:18:34.652338+0000","last_unstale":"2023-10-14T18:18:37.373307+0000","last_undegraded":"2023-10-14T18:18:37.373307+0000","last_fullsized":"2023-10-14T18:18:37.373307+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:39.725847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650630+0000","last_change":"2023-10-14T18:18:35.668420+0000","last_active":"2023-10-14T18:18:36.650630+0000","last_peered":"2023-10-14T18:18:36.650630+0000","last_clean":"2023-10-14T18:18:36.650630+0000","last_became_active":"2023-10-14T18:18:35.667429+0000","last_became_peered":"2023-10-14T18:18:35.667429+0000","last_unstale":"2023-10-14T18:18:36.650630+0000","last_undegraded":"2023-10-14T18:18:36.650630+0000","last_fullsized":"2023-10-14T18:18:36.650630+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651660+0000","last_change":"2023-10-14T18:18:34.649612+0000","last_active":"2023-10-14T18:18:36.651660+0000","last_peered":"2023-10-14T18:18:36.651660+0000","last_clean":"2023-10-14T18:18:36.651660+0000","last_became_active":"2023-10-14T18:18:34.649307+0000","last_became_peered":"2023-10-14T18:18:34.649307+0000","last_unstale":"2023-10-14T18:18:36.651660+0000","last_undegraded":"2023-10-14T18:18:36.651660+0000","last_fullsized":"2023-10-14T18:18:36.651660+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.438796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654483+0000","last_change":"2023-10-14T18:18:35.666113+0000","last_active":"2023-10-14T18:18:36.654483+0000","last_peered":"2023-10-14T18:18:36.654483+0000","last_clean":"2023-10-14T18:18:36.654483+0000","last_became_active":"2023-10-14T18:18:35.665775+0000","last_became_peered":"2023-10-14T18:18:35.665775+0000","last_unstale":"2023-10-14T18:18:36.654483+0000","last_undegraded":"2023-10-14T18:18:36.654483+0000","last_fullsized":"2023-10-14T18:18:36.654483+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644936+0000","last_change":"2023-10-14T18:18:34.657934+0000","last_active":"2023-10-14T18:18:36.644936+0000","last_peered":"2023-10-14T18:18:36.644936+0000","last_clean":"2023-10-14T18:18:36.644936+0000","last_became_active":"2023-10-14T18:18:34.657614+0000","last_became_peered":"2023-10-14T18:18:34.657614+0000","last_unstale":"2023-10-14T18:18:36.644936+0000","last_undegraded":"2023-10-14T18:18:36.644936+0000","last_fullsized":"2023-10-14T18:18:36.644936+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:11.487790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274290+0000","last_change":"2023-10-14T18:18:35.680237+0000","last_active":"2023-10-14T18:18:37.274290+0000","last_peered":"2023-10-14T18:18:37.274290+0000","last_clean":"2023-10-14T18:18:37.274290+0000","last_became_active":"2023-10-14T18:18:35.679960+0000","last_became_peered":"2023-10-14T18:18:35.679960+0000","last_unstale":"2023-10-14T18:18:37.274290+0000","last_undegraded":"2023-10-14T18:18:37.274290+0000","last_fullsized":"2023-10-14T18:18:37.274290+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:53.427626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373452+0000","last_change":"2023-10-14T18:18:34.651448+0000","last_active":"2023-10-14T18:18:37.373452+0000","last_peered":"2023-10-14T18:18:37.373452+0000","last_clean":"2023-10-14T18:18:37.373452+0000","last_became_active":"2023-10-14T18:18:34.651100+0000","last_became_peered":"2023-10-14T18:18:34.651100+0000","last_unstale":"2023-10-14T18:18:37.373452+0000","last_undegraded":"2023-10-14T18:18:37.373452+0000","last_fullsized":"2023-10-14T18:18:37.373452+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:41.231071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374446+0000","last_change":"2023-10-14T18:18:35.687662+0000","last_active":"2023-10-14T18:18:37.374446+0000","last_peered":"2023-10-14T18:18:37.374446+0000","last_clean":"2023-10-14T18:18:37.374446+0000","last_became_active":"2023-10-14T18:18:35.687396+0000","last_became_peered":"2023-10-14T18:18:35.687396+0000","last_unstale":"2023-10-14T18:18:37.374446+0000","last_undegraded":"2023-10-14T18:18:37.374446+0000","last_fullsized":"2023-10-14T18:18:37.374446+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:15.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373758+0000","last_change":"2023-10-14T18:18:34.649374+0000","last_active":"2023-10-14T18:18:37.373758+0000","last_peered":"2023-10-14T18:18:37.373758+0000","last_clean":"2023-10-14T18:18:37.373758+0000","last_became_active":"2023-10-14T18:18:34.649074+0000","last_became_peered":"2023-10-14T18:18:34.649074+0000","last_unstale":"2023-10-14T18:18:37.373758+0000","last_undegraded":"2023-10-14T18:18:37.373758+0000","last_fullsized":"2023-10-14T18:18:37.373758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:45.885592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.376126+0000","last_change":"2023-10-14T18:18:35.682974+0000","last_active":"2023-10-14T18:18:37.376126+0000","last_peered":"2023-10-14T18:18:37.376126+0000","last_clean":"2023-10-14T18:18:37.376126+0000","last_became_active":"2023-10-14T18:18:35.682647+0000","last_became_peered":"2023-10-14T18:18:35.682647+0000","last_unstale":"2023-10-14T18:18:37.376126+0000","last_undegraded":"2023-10-14T18:18:37.376126+0000","last_fullsized":"2023-10-14T18:18:37.376126+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:58.501949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374987+0000","last_change":"2023-10-14T18:18:34.656380+0000","last_active":"2023-10-14T18:18:37.374987+0000","last_peered":"2023-10-14T18:18:37.374987+0000","last_clean":"2023-10-14T18:18:37.374987+0000","last_became_active":"2023-10-14T18:18:34.656073+0000","last_became_peered":"2023-10-14T18:18:34.656073+0000","last_unstale":"2023-10-14T18:18:37.374987+0000","last_undegraded":"2023-10-14T18:18:37.374987+0000","last_fullsized":"2023-10-14T18:18:37.374987+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:41.992815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653450+0000","last_change":"2023-10-14T18:18:35.666278+0000","last_active":"2023-10-14T18:18:36.653450+0000","last_peered":"2023-10-14T18:18:36.653450+0000","last_clean":"2023-10-14T18:18:36.653450+0000","last_became_active":"2023-10-14T18:18:35.665963+0000","last_became_peered":"2023-10-14T18:18:35.665963+0000","last_unstale":"2023-10-14T18:18:36.653450+0000","last_undegraded":"2023-10-14T18:18:36.653450+0000","last_fullsized":"2023-10-14T18:18:36.653450+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653797+0000","last_change":"2023-10-14T18:18:34.649641+0000","last_active":"2023-10-14T18:18:36.653797+0000","last_peered":"2023-10-14T18:18:36.653797+0000","last_clean":"2023-10-14T18:18:36.653797+0000","last_became_active":"2023-10-14T18:18:34.649340+0000","last_became_peered":"2023-10-14T18:18:34.649340+0000","last_unstale":"2023-10-14T18:18:36.653797+0000","last_undegraded":"2023-10-14T18:18:36.653797+0000","last_fullsized":"2023-10-14T18:18:36.653797+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:05.759247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651274+0000","last_change":"2023-10-14T18:18:35.675054+0000","last_active":"2023-10-14T18:18:36.651274+0000","last_peered":"2023-10-14T18:18:36.651274+0000","last_clean":"2023-10-14T18:18:36.651274+0000","last_became_active":"2023-10-14T18:18:35.674739+0000","last_became_peered":"2023-10-14T18:18:35.674739+0000","last_unstale":"2023-10-14T18:18:36.651274+0000","last_undegraded":"2023-10-14T18:18:36.651274+0000","last_fullsized":"2023-10-14T18:18:36.651274+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644693+0000","last_change":"2023-10-14T18:18:34.652028+0000","last_active":"2023-10-14T18:18:36.644693+0000","last_peered":"2023-10-14T18:18:36.644693+0000","last_clean":"2023-10-14T18:18:36.644693+0000","last_became_active":"2023-10-14T18:18:34.651757+0000","last_became_peered":"2023-10-14T18:18:34.651757+0000","last_unstale":"2023-10-14T18:18:36.644693+0000","last_undegraded":"2023-10-14T18:18:36.644693+0000","last_fullsized":"2023-10-14T18:18:36.644693+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649847+0000","last_change":"2023-10-14T18:18:35.660261+0000","last_active":"2023-10-14T18:18:36.649847+0000","last_peered":"2023-10-14T18:18:36.649847+0000","last_clean":"2023-10-14T18:18:36.649847+0000","last_became_active":"2023-10-14T18:18:35.659486+0000","last_became_peered":"2023-10-14T18:18:35.659486+0000","last_unstale":"2023-10-14T18:18:36.649847+0000","last_undegraded":"2023-10-14T18:18:36.649847+0000","last_fullsized":"2023-10-14T18:18:36.649847+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373010+0000","last_change":"2023-10-14T18:18:34.643288+0000","last_active":"2023-10-14T18:18:37.373010+0000","last_peered":"2023-10-14T18:18:37.373010+0000","last_clean":"2023-10-14T18:18:37.373010+0000","last_became_active":"2023-10-14T18:18:34.642997+0000","last_became_peered":"2023-10-14T18:18:34.642997+0000","last_unstale":"2023-10-14T18:18:37.373010+0000","last_undegraded":"2023-10-14T18:18:37.373010+0000","last_fullsized":"2023-10-14T18:18:37.373010+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:39.884500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273854+0000","last_change":"2023-10-14T18:18:35.674240+0000","last_active":"2023-10-14T18:18:37.273854+0000","last_peered":"2023-10-14T18:18:37.273854+0000","last_clean":"2023-10-14T18:18:37.273854+0000","last_became_active":"2023-10-14T18:18:35.671067+0000","last_became_peered":"2023-10-14T18:18:35.671067+0000","last_unstale":"2023-10-14T18:18:37.273854+0000","last_undegraded":"2023-10-14T18:18:37.273854+0000","last_fullsized":"2023-10-14T18:18:37.273854+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:02:59.948900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373424+0000","last_change":"2023-10-14T18:18:34.643687+0000","last_active":"2023-10-14T18:18:37.373424+0000","last_peered":"2023-10-14T18:18:37.373424+0000","last_clean":"2023-10-14T18:18:37.373424+0000","last_became_active":"2023-10-14T18:18:34.643092+0000","last_became_peered":"2023-10-14T18:18:34.643092+0000","last_unstale":"2023-10-14T18:18:37.373424+0000","last_undegraded":"2023-10-14T18:18:37.373424+0000","last_fullsized":"2023-10-14T18:18:37.373424+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.480279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373227+0000","last_change":"2023-10-14T18:18:35.685028+0000","last_active":"2023-10-14T18:18:37.373227+0000","last_peered":"2023-10-14T18:18:37.373227+0000","last_clean":"2023-10-14T18:18:37.373227+0000","last_became_active":"2023-10-14T18:18:35.684687+0000","last_became_peered":"2023-10-14T18:18:35.684687+0000","last_unstale":"2023-10-14T18:18:37.373227+0000","last_undegraded":"2023-10-14T18:18:37.373227+0000","last_fullsized":"2023-10-14T18:18:37.373227+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:27.530239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375704+0000","last_change":"2023-10-14T18:18:34.640893+0000","last_active":"2023-10-14T18:18:37.375704+0000","last_peered":"2023-10-14T18:18:37.375704+0000","last_clean":"2023-10-14T18:18:37.375704+0000","last_became_active":"2023-10-14T18:18:34.639717+0000","last_became_peered":"2023-10-14T18:18:34.639717+0000","last_unstale":"2023-10-14T18:18:37.375704+0000","last_undegraded":"2023-10-14T18:18:37.375704+0000","last_fullsized":"2023-10-14T18:18:37.375704+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:55.441603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"235'370","reported_seq":1308,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-14T18:18:35.670189+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:18:35.670189+0000","last_peered":"2023-10-14T18:18:35.670189+0000","last_clean":"2023-10-14T18:18:35.670189+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:18:35.670189+0000","last_undegraded":"2023-10-14T18:18:35.670189+0000","last_fullsized":"2023-10-14T18:18:35.670189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576018,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57036,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314804,"statfs":{"total":96636764160,"available":96578359296,"internally_reserved":0,"allocated":3731456,"data_stored":2151046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.177,"15min":1.2030000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.351},{"interface":"front","average":{"1min":1.163,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":2.2149999999999999},"last":0.98299999999999998}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.768,"5min":1.9059999999999999,"15min":2.214},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.2909999999999999,"15min":1.3109999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3919999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.206,"15min":1.26},"min":{"1min":0.83199999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.839,"5min":1.839,"15min":2.004},"last":0.878},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3100000000000001,"15min":1.365},"min":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.2010000000000001},"last":1.2450000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.347,"15min":1.4239999999999999},"min":{"1min":0.94799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.488,"15min":1.528},"min":{"1min":1.0600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.994,"5min":1.994,"15min":2.2810000000000001},"last":1.542}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3680000000000001,"15min":1.343},"min":{"1min":0.90500000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.367},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3759999999999999,"15min":1.3919999999999999},"min":{"1min":1.0529999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.661},"last":1.1499999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4370000000000001,"15min":1.4670000000000001},"min":{"1min":1.0409999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.516},"last":1.667},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.448},"last":1.2150000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.615,"15min":1.639},"min":{"1min":1.103,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.5289999999999999,"15min":1.577},"min":{"1min":1.1399999999999999,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.6339999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":3153920,"data_stored":1567628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.167,"15min":1.1779999999999999},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.669,"5min":2.085,"15min":2.085},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.2070000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.704,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1419999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.208,"15min":1.24},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.494,"5min":1.762,"15min":1.776},"last":1.272},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.81799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.1760000000000002,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.665}]},{"osd":2,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3740000000000001,"15min":1.377},"min":{"1min":0.85999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.819,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.978},{"interface":"front","average":{"1min":1.373,"5min":1.397,"15min":1.419},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.6180000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.347,"15min":1.4059999999999999},"min":{"1min":0.72799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.2519999999999998},"last":1.75},{"interface":"front","average":{"1min":1.357,"5min":1.411,"15min":1.4359999999999999},"min":{"1min":0.91300000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8220000000000001,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.5009999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3899999999999999,"15min":1.3859999999999999},"min":{"1min":0.96499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7949999999999999,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":1.109,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0030000000000001,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.8759999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.411,"15min":1.419},"min":{"1min":1.024,"5min":0.624,"15min":0.624},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":0.88},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5780000000000001,"15min":1.5840000000000001},"min":{"1min":1.2310000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.5609999999999999},"last":0.97899999999999998}]},{"osd":7,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.6140000000000001,"15min":1.6180000000000001},"min":{"1min":1.403,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.194,"5min":2.903,"15min":3.423},"last":2.073},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.583,"15min":1.6040000000000001},"min":{"1min":1.296,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.2349999999999999,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.4199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42560,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329280,"statfs":{"total":96636764160,"available":96593182720,"internally_reserved":0,"allocated":3133440,"data_stored":1560608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1719999999999999,"15min":1.2070000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.1739999999999999,"15min":1.2350000000000001},"min":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.629,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":0.77400000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.234,"15min":1.276},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6850000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.296,"15min":1.3100000000000001},"min":{"1min":0.92100000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.4019999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1799999999999999,"15min":1.2430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.591,"5min":1.764,"15min":3.7090000000000001},"last":1.22},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.377,"15min":1.397},"min":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":1.0609999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4159999999999999,"15min":1.46},"min":{"1min":0.94299999999999995,"5min":0.94199999999999995,"15min":0.88900000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.548},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.4339999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.8330000000000002},"last":1.1140000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3879999999999999,"15min":1.3660000000000001},"min":{"1min":1.0700000000000001,"5min":0.85899999999999999,"15min":0.78400000000000003},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.98},"last":1.171},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4370000000000001,"15min":1.401},"min":{"1min":0.90400000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.516,"5min":2.516,"15min":3.1280000000000001},"last":1.629}]},{"osd":6,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.399,"15min":1.4259999999999999},"min":{"1min":0.92900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":4.0540000000000003},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.6259999999999999,"15min":1.635},"min":{"1min":1.429,"5min":0.93000000000000005,"15min":0.81799999999999995},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":3.6360000000000001},"last":1.484}]},{"osd":7,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5609999999999999,"15min":1.585},"min":{"1min":1.121,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":3.2759999999999998},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5840000000000001,"15min":1.5680000000000001},"min":{"1min":1.1419999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.9390000000000001},"last":1.4490000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66148,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305692,"statfs":{"total":96636764160,"available":96569028608,"internally_reserved":0,"allocated":3166208,"data_stored":1568728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.181,"15min":1.2010000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.266,"15min":1.2829999999999999},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.04}]},{"osd":1,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.282,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.59399999999999997},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.95599999999999996}]},{"osd":2,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.286},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.524},"last":1.524},{"interface":"front","average":{"1min":1.216,"5min":1.389,"15min":1.4299999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.65700000000000003},"max":{"1min":1.597,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.444}]},{"osd":3,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":1.0029999999999999,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.6739999999999999,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":1.0009999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.746}]},{"osd":5,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.397},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.69299999999999995},"max":{"1min":1.923,"5min":1.9319999999999999,"15min":2.02},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.393,"15min":1.3839999999999999},"min":{"1min":0.90000000000000002,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.452,"15min":1.4370000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.577,"15min":1.569},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82699999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.7050000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.5229999999999999,"15min":1.522},"min":{"1min":1.248,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.946,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.492},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6080000000000001,"15min":1.5940000000000001},"min":{"1min":1.2889999999999999,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":2.597,"5min":2.597,"15min":3.8610000000000002},"last":1.867}]}]},{"osd":3,"up_from":13,"seq":55834576018,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56456,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315384,"statfs":{"total":96636764160,"available":96578953216,"internally_reserved":0,"allocated":3137536,"data_stored":1561190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94399999999999995,"15min":0.97599999999999998},"min":{"1min":0.45700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.4239999999999999,"5min":1.4870000000000001,"15min":1.855},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1870000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.504,"5min":1.6399999999999999,"15min":1.9970000000000001},"last":0.92500000000000004}]},{"osd":1,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94799999999999995,"15min":0.996},"min":{"1min":0.47099999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.363,"5min":1.835,"15min":1.919},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.1970000000000001},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.8560000000000001},"last":1.032}]},{"osd":2,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.339,"15min":1.4159999999999999},"min":{"1min":0.54900000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.5620000000000001,"5min":2.032,"15min":2.032},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.04,"5min":1.26,"15min":1.3640000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":0.80200000000000005}]},{"osd":4,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3819999999999999,"15min":1.429},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9099999999999999,"5min":2.153,"15min":2.153},"last":1.218},{"interface":"front","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.478},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.25}]},{"osd":5,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4239999999999999,"15min":1.4550000000000001},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.46,"15min":1.548},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.02,"5min":2.1360000000000001,"15min":2.891},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5700000000000001,"15min":1.621},"min":{"1min":0.97199999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.335},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.6339999999999999,"15min":1.677},"min":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.4139999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.532,"15min":1.599},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":3.153,"5min":3.153,"15min":3.153},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.635,"15min":1.681},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.454}]}]},{"osd":2,"up_from":13,"seq":55834576018,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52304,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319536,"statfs":{"total":96636764160,"available":96583204864,"internally_reserved":0,"allocated":3149824,"data_stored":1567114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3029999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.98099999999999998,"15min":1.081},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.2629999999999999,"5min":1.4550000000000001,"15min":1.8660000000000001},"last":0.439}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":1.0069999999999999},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1439999999999999,"5min":1.603,"15min":1.881},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.1679999999999999,"15min":1.212},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5780000000000001,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.218,"15min":1.2749999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.3959999999999999,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.337,"15min":1.3779999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7490000000000001,"5min":2.1579999999999999,"15min":2.165},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.464},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.736,"5min":2.097,"15min":2.097},"last":0.875},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3300000000000001,"15min":1.365},"min":{"1min":0.72399999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.944,"5min":1.944,"15min":2.024},"last":0.80100000000000005}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.389,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.2839999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.577,"15min":1.583},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.0049999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.52,"15min":1.5289999999999999},"min":{"1min":0.80600000000000005,"5min":0.747,"15min":0.747},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.556,"15min":1.603},"min":{"1min":0.91100000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1400000000000001,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":1.03}]},{"osd":7,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5780000000000001,"15min":1.5900000000000001},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.633,"15min":1.675},"min":{"1min":0.97999999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0600000000000001,"5min":2.331,"15min":2.331},"last":0.98499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576018,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52308,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319532,"statfs":{"total":96636764160,"available":96583200768,"internally_reserved":0,"allocated":3153920,"data_stored":1562344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95899999999999996,"15min":0.97199999999999998},"min":{"1min":0.51100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.5169999999999999,"5min":1.647,"15min":2.4929999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.0820000000000001,"15min":1.1279999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":0.86499999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.038,"15min":1.0629999999999999},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.895,"5min":1.996,"15min":2.6859999999999999},"last":1.5129999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.357,"15min":1.385},"min":{"1min":0.52700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2829999999999999,"15min":1.3169999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.8630000000000004,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.88200000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.385,"15min":1.431},"min":{"1min":0.80400000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.899},"last":1.3879999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.4379999999999999,"15min":1.4710000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.173,"5min":2.173,"15min":3.004},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.55,"15min":1.5660000000000001},"min":{"1min":0.91800000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9359999999999999,"5min":2.093,"15min":2.5129999999999999},"last":1.325}]},{"osd":6,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5780000000000001,"15min":1.6140000000000001},"min":{"1min":0.95499999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.629},{"interface":"front","average":{"1min":1.863,"5min":1.649,"15min":1.627},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.667}]},{"osd":7,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.6679999999999999,"15min":1.6970000000000001},"min":{"1min":0.82299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":1.587},{"interface":"front","average":{"1min":1.839,"5min":1.6970000000000001,"15min":1.722},"min":{"1min":0.93300000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":2.0630000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3731456,"data_stored":2151010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.94599999999999995,"15min":0.97299999999999998},"min":{"1min":0.57699999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.367,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":1.008},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.55500000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3560000000000001,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.143,"15min":1.1759999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6379999999999999,"5min":1.855,"15min":1.855},"last":1.518},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.167,"15min":1.2390000000000001},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.522,"5min":1.845,"15min":2.5779999999999998},"last":0.79100000000000004}]},{"osd":3,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.52100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.5429999999999999,"5min":1.804,"15min":2.0720000000000001},"last":1.202},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3300000000000001,"15min":1.391},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.77,"5min":1.883,"15min":2.194},"last":0.94399999999999995}]},{"osd":4,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.284,"15min":1.3169999999999999},"min":{"1min":0.73999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5549999999999999,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.161},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.492,"15min":1.502},"min":{"1min":0.999,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.984,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.448}]},{"osd":5,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.327,"15min":1.3680000000000001},"min":{"1min":0.80800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.51,"15min":1.569},"min":{"1min":0.78700000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.242,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.4870000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.6259999999999999,"15min":1.651},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.1709999999999998,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.4830000000000001,"15min":1.508},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6899999999999999,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.4139999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.6040000000000001,"15min":1.6479999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0649999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.5589999999999999,"15min":1.6160000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.857,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.4730000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:18:40.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:18:40.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.461+0000 7f947790f640 1 Processor -- start 2023-10-14T18:18:40.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.461+0000 7f947790f640 1 -- start start 2023-10-14T18:18:40.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.462+0000 7f947790f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.462+0000 7f947790f640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f947015ae80 0x7f94700cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.462+0000 7f947790f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f947015ed60 con 0x7f94700ccaa0 2023-10-14T18:18:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.462+0000 7f947790f640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f947006b810 con 0x7f947015ae80 2023-10-14T18:18:40.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.462+0000 7f947790f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700707e0 con 0x7f947015aaa0 2023-10-14T18:18:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.463+0000 7f9474eaa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f947015ae80 0x7f94700cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.463+0000 7f94756ab640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.463+0000 7f9474eaa640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f947015ae80 0x7f94700cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:34352/0 (socket says 172.21.15.29:34352) 2023-10-14T18:18:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.463+0000 7f9474eaa640 1 -- 172.21.15.29:0/3385139712 learned_addr learned my addr 172.21.15.29:0/3385139712 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.464+0000 7f9474eaa640 1 -- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 msgr2=0x7f947015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.464+0000 7f9474eaa640 1 --2- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.464+0000 7f9474eaa640 1 -- 172.21.15.29:0/3385139712 >> v1:172.21.15.186:6789/0 conn(0x7f94700ccaa0 legacy=0x7f9470070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.464+0000 7f9474eaa640 1 -- 172.21.15.29:0/3385139712 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947015ef90 con 0x7f947015ae80 2023-10-14T18:18:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.465+0000 7f9474eaa640 1 --2- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947015ae80 0x7f94700cc560 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f946002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42c82f7fbe3099f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.466+0000 7f9467fff640 1 -- 172.21.15.29:0/3385139712 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460030840 con 0x7f947015ae80 2023-10-14T18:18:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.466+0000 7f9467fff640 1 -- 172.21.15.29:0/3385139712 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94600309a0 con 0x7f947015ae80 2023-10-14T18:18:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.466+0000 7f9467fff640 1 -- 172.21.15.29:0/3385139712 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9460030d30 con 0x7f947015ae80 2023-10-14T18:18:40.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.466+0000 7f947790f640 1 -- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947015ae80 msgr2=0x7f94700cc560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 --2- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947015ae80 0x7f94700cc560 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f9460009870 tx=0x7f946002b980 comp rx=0 tx=0).stop 2023-10-14T18:18:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 -- 172.21.15.29:0/3385139712 shutdown_connections 2023-10-14T18:18:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 --2- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f947015ae80 0x7f94700cc560 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 --2- 172.21.15.29:0/3385139712 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947015e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 -- 172.21.15.29:0/3385139712 >> 172.21.15.29:0/3385139712 conn(0x7f94700cb410 msgr2=0x7f94700cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 -- 172.21.15.29:0/3385139712 shutdown_connections 2023-10-14T18:18:40.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.467+0000 7f947790f640 1 -- 172.21.15.29:0/3385139712 wait complete. 2023-10-14T18:18:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.468+0000 7f947790f640 1 Processor -- start 2023-10-14T18:18:40.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.469+0000 7f947790f640 1 -- start start 2023-10-14T18:18:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94700ccaa0 0x7f94701566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947014b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94700c4180 con 0x7f947015ae80 2023-10-14T18:18:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9470156c20 con 0x7f94700ccaa0 2023-10-14T18:18:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f947790f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94700c3e70 con 0x7f947015aaa0 2023-10-14T18:18:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f9475eac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f9475eac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:54766/0 (socket says 172.21.15.29:54766) 2023-10-14T18:18:40.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f9475eac640 1 -- 172.21.15.29:0/488187582 learned_addr learned my addr 172.21.15.29:0/488187582 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f94756ab640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94700ccaa0 0x7f94701566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.470+0000 7f9474eaa640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947014b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 -- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 msgr2=0x7f947014b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947014b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 -- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94700ccaa0 msgr2=0x7f94701566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94700ccaa0 0x7f94701566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 -- 172.21.15.29:0/488187582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94600094d0 con 0x7f947015ae80 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9474eaa640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9475eac640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f946800c6b0 tx=0x7f946800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468003b70 con 0x7f947015ae80 2023-10-14T18:18:40.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9468003cf0 con 0x7f947015ae80 2023-10-14T18:18:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9468003ea0 con 0x7f947015ae80 2023-10-14T18:18:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9470156e10 con 0x7f947015ae80 2023-10-14T18:18:40.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.471+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9470159860 con 0x7f947015ae80 2023-10-14T18:18:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.473+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94700c42c0 con 0x7f947015ae80 2023-10-14T18:18:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.474+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9468005310 con 0x7f947015ae80 2023-10-14T18:18:40.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.475+0000 7f9465ffb640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 0x7f945406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.475+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f94680899a0 con 0x7f947015ae80 2023-10-14T18:18:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.475+0000 7f94756ab640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 0x7f945406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.476+0000 7f94756ab640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 0x7f945406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f946c009ad0 tx=0x7f946c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:40.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.479+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94700c42c0 con 0x7f947015ae80 2023-10-14T18:18:40.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.670+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f947015f550 con 0x7f945406b0b0 2023-10-14T18:18:40.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.680+0000 7f9465ffb640 1 -- 172.21.15.29:0/488187582 <== mgr.7463 v2:172.21.15.29:6832/4275371095 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287236 (secure 0 0 0) 0x7f946c0119a0 con 0x7f945406b0b0 2023-10-14T18:18:40.684 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:40.687 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:18:40.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.687+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 msgr2=0x7f945406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.687+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 0x7f945406d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f946c009ad0 tx=0x7f946c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:40.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.687+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 msgr2=0x7f94700c45a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:40.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.687+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 secure :-1 s=READY pgs=2882 cs=0 l=1 rev1=1 crypto rx=0x7f946800c6b0 tx=0x7f946800cb70 comp rx=0 tx=0).stop 2023-10-14T18:18:40.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 shutdown_connections 2023-10-14T18:18:40.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f945406b0b0 0x7f945406d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f947015ae80 0x7f94700c45a0 unknown :-1 s=CLOSED pgs=2882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f947015aaa0 0x7f947014b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 --2- 172.21.15.29:0/488187582 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94700ccaa0 0x7f94701566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 >> 172.21.15.29:0/488187582 conn(0x7f94700cb410 msgr2=0x7f94700b1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:40.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 shutdown_connections 2023-10-14T18:18:40.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.688+0000 7f947790f640 1 -- 172.21.15.29:0/488187582 wait complete. 2023-10-14T18:18:40.709 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-14T18:18:40.425795+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":663,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":408,"ondisk_log_size":408,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":423688,"kb_used_data":25740,"kb_used_omap":12,"kb_used_meta":397811,"kb_avail":754551032,"statfs":{"total":773094113280,"available":772660256768,"internally_reserved":0,"allocated":26357760,"data_stored":13689668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":407359032},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":27,"apply_latency_ms":27,"commit_latency_ns":27000000,"apply_latency_ns":27000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002295"},"pg_stats":[{"pgid":"31.3f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274249+0000","last_change":"2023-10-14T18:18:35.681689+0000","last_active":"2023-10-14T18:18:37.274249+0000","last_peered":"2023-10-14T18:18:37.274249+0000","last_clean":"2023-10-14T18:18:37.274249+0000","last_became_active":"2023-10-14T18:18:35.681412+0000","last_became_peered":"2023-10-14T18:18:35.681412+0000","last_unstale":"2023-10-14T18:18:37.274249+0000","last_undegraded":"2023-10-14T18:18:37.274249+0000","last_fullsized":"2023-10-14T18:18:37.274249+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:30:00.717606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273882+0000","last_change":"2023-10-14T18:18:35.683822+0000","last_active":"2023-10-14T18:18:37.273882+0000","last_peered":"2023-10-14T18:18:37.273882+0000","last_clean":"2023-10-14T18:18:37.273882+0000","last_became_active":"2023-10-14T18:18:35.683492+0000","last_became_peered":"2023-10-14T18:18:35.683492+0000","last_unstale":"2023-10-14T18:18:37.273882+0000","last_undegraded":"2023-10-14T18:18:37.273882+0000","last_fullsized":"2023-10-14T18:18:37.273882+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:15.955276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374695+0000","last_change":"2023-10-14T18:18:35.668307+0000","last_active":"2023-10-14T18:18:37.374695+0000","last_peered":"2023-10-14T18:18:37.374695+0000","last_clean":"2023-10-14T18:18:37.374695+0000","last_became_active":"2023-10-14T18:18:35.667786+0000","last_became_peered":"2023-10-14T18:18:35.667786+0000","last_unstale":"2023-10-14T18:18:37.374695+0000","last_undegraded":"2023-10-14T18:18:37.374695+0000","last_fullsized":"2023-10-14T18:18:37.374695+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:51.991520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644410+0000","last_change":"2023-10-14T18:18:35.663594+0000","last_active":"2023-10-14T18:18:36.644410+0000","last_peered":"2023-10-14T18:18:36.644410+0000","last_clean":"2023-10-14T18:18:36.644410+0000","last_became_active":"2023-10-14T18:18:35.663236+0000","last_became_peered":"2023-10-14T18:18:35.663236+0000","last_unstale":"2023-10-14T18:18:36.644410+0000","last_undegraded":"2023-10-14T18:18:36.644410+0000","last_fullsized":"2023-10-14T18:18:36.644410+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:56.542420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372556+0000","last_change":"2023-10-14T18:18:35.670764+0000","last_active":"2023-10-14T18:18:37.372556+0000","last_peered":"2023-10-14T18:18:37.372556+0000","last_clean":"2023-10-14T18:18:37.372556+0000","last_became_active":"2023-10-14T18:18:35.670354+0000","last_became_peered":"2023-10-14T18:18:35.670354+0000","last_unstale":"2023-10-14T18:18:37.372556+0000","last_undegraded":"2023-10-14T18:18:37.372556+0000","last_fullsized":"2023-10-14T18:18:37.372556+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:43.950584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644608+0000","last_change":"2023-10-14T18:18:35.682936+0000","last_active":"2023-10-14T18:18:36.644608+0000","last_peered":"2023-10-14T18:18:36.644608+0000","last_clean":"2023-10-14T18:18:36.644608+0000","last_became_active":"2023-10-14T18:18:35.682060+0000","last_became_peered":"2023-10-14T18:18:35.682060+0000","last_unstale":"2023-10-14T18:18:36.644608+0000","last_undegraded":"2023-10-14T18:18:36.644608+0000","last_fullsized":"2023-10-14T18:18:36.644608+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:31.696209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651282+0000","last_change":"2023-10-14T18:18:35.676700+0000","last_active":"2023-10-14T18:18:36.651282+0000","last_peered":"2023-10-14T18:18:36.651282+0000","last_clean":"2023-10-14T18:18:36.651282+0000","last_became_active":"2023-10-14T18:18:35.676423+0000","last_became_peered":"2023-10-14T18:18:35.676423+0000","last_unstale":"2023-10-14T18:18:36.651282+0000","last_undegraded":"2023-10-14T18:18:36.651282+0000","last_fullsized":"2023-10-14T18:18:36.651282+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649693+0000","last_change":"2023-10-14T18:18:35.664633+0000","last_active":"2023-10-14T18:18:36.649693+0000","last_peered":"2023-10-14T18:18:36.649693+0000","last_clean":"2023-10-14T18:18:36.649693+0000","last_became_active":"2023-10-14T18:18:35.664248+0000","last_became_peered":"2023-10-14T18:18:35.664248+0000","last_unstale":"2023-10-14T18:18:36.649693+0000","last_undegraded":"2023-10-14T18:18:36.649693+0000","last_fullsized":"2023-10-14T18:18:36.649693+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:21.318470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373806+0000","last_change":"2023-10-14T18:18:35.679660+0000","last_active":"2023-10-14T18:18:37.373806+0000","last_peered":"2023-10-14T18:18:37.373806+0000","last_clean":"2023-10-14T18:18:37.373806+0000","last_became_active":"2023-10-14T18:18:35.679374+0000","last_became_peered":"2023-10-14T18:18:35.679374+0000","last_unstale":"2023-10-14T18:18:37.373806+0000","last_undegraded":"2023-10-14T18:18:37.373806+0000","last_fullsized":"2023-10-14T18:18:37.373806+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:57.315103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654092+0000","last_change":"2023-10-14T18:18:35.679829+0000","last_active":"2023-10-14T18:18:36.654092+0000","last_peered":"2023-10-14T18:18:36.654092+0000","last_clean":"2023-10-14T18:18:36.654092+0000","last_became_active":"2023-10-14T18:18:35.679412+0000","last_became_peered":"2023-10-14T18:18:35.679412+0000","last_unstale":"2023-10-14T18:18:36.654092+0000","last_undegraded":"2023-10-14T18:18:36.654092+0000","last_fullsized":"2023-10-14T18:18:36.654092+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375596+0000","last_change":"2023-10-14T18:18:35.689722+0000","last_active":"2023-10-14T18:18:37.375596+0000","last_peered":"2023-10-14T18:18:37.375596+0000","last_clean":"2023-10-14T18:18:37.375596+0000","last_became_active":"2023-10-14T18:18:35.686665+0000","last_became_peered":"2023-10-14T18:18:35.686665+0000","last_unstale":"2023-10-14T18:18:37.375596+0000","last_undegraded":"2023-10-14T18:18:37.375596+0000","last_fullsized":"2023-10-14T18:18:37.375596+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:57.853905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651148+0000","last_change":"2023-10-14T18:18:35.674443+0000","last_active":"2023-10-14T18:18:36.651148+0000","last_peered":"2023-10-14T18:18:36.651148+0000","last_clean":"2023-10-14T18:18:36.651148+0000","last_became_active":"2023-10-14T18:18:35.673655+0000","last_became_peered":"2023-10-14T18:18:35.673655+0000","last_unstale":"2023-10-14T18:18:36.651148+0000","last_undegraded":"2023-10-14T18:18:36.651148+0000","last_fullsized":"2023-10-14T18:18:36.651148+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653672+0000","last_change":"2023-10-14T18:18:35.687221+0000","last_active":"2023-10-14T18:18:36.653672+0000","last_peered":"2023-10-14T18:18:36.653672+0000","last_clean":"2023-10-14T18:18:36.653672+0000","last_became_active":"2023-10-14T18:18:35.686900+0000","last_became_peered":"2023-10-14T18:18:35.686900+0000","last_unstale":"2023-10-14T18:18:36.653672+0000","last_undegraded":"2023-10-14T18:18:36.653672+0000","last_fullsized":"2023-10-14T18:18:36.653672+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273360+0000","last_change":"2023-10-14T18:18:35.671605+0000","last_active":"2023-10-14T18:18:37.273360+0000","last_peered":"2023-10-14T18:18:37.273360+0000","last_clean":"2023-10-14T18:18:37.273360+0000","last_became_active":"2023-10-14T18:18:35.671317+0000","last_became_peered":"2023-10-14T18:18:35.671317+0000","last_unstale":"2023-10-14T18:18:37.273360+0000","last_undegraded":"2023-10-14T18:18:37.273360+0000","last_fullsized":"2023-10-14T18:18:37.273360+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:14.496902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373037+0000","last_change":"2023-10-14T18:18:35.682877+0000","last_active":"2023-10-14T18:18:37.373037+0000","last_peered":"2023-10-14T18:18:37.373037+0000","last_clean":"2023-10-14T18:18:37.373037+0000","last_became_active":"2023-10-14T18:18:35.682389+0000","last_became_peered":"2023-10-14T18:18:35.682389+0000","last_unstale":"2023-10-14T18:18:37.373037+0000","last_undegraded":"2023-10-14T18:18:37.373037+0000","last_fullsized":"2023-10-14T18:18:37.373037+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:25.798127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373367+0000","last_change":"2023-10-14T18:18:35.674408+0000","last_active":"2023-10-14T18:18:37.373367+0000","last_peered":"2023-10-14T18:18:37.373367+0000","last_clean":"2023-10-14T18:18:37.373367+0000","last_became_active":"2023-10-14T18:18:35.674062+0000","last_became_peered":"2023-10-14T18:18:35.674062+0000","last_unstale":"2023-10-14T18:18:37.373367+0000","last_undegraded":"2023-10-14T18:18:37.373367+0000","last_fullsized":"2023-10-14T18:18:37.373367+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:16.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652705+0000","last_change":"2023-10-14T18:18:35.678586+0000","last_active":"2023-10-14T18:18:36.652705+0000","last_peered":"2023-10-14T18:18:36.652705+0000","last_clean":"2023-10-14T18:18:36.652705+0000","last_became_active":"2023-10-14T18:18:35.678265+0000","last_became_peered":"2023-10-14T18:18:35.678265+0000","last_unstale":"2023-10-14T18:18:36.652705+0000","last_undegraded":"2023-10-14T18:18:36.652705+0000","last_fullsized":"2023-10-14T18:18:36.652705+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:25.316716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274015+0000","last_change":"2023-10-14T18:18:35.679271+0000","last_active":"2023-10-14T18:18:37.274015+0000","last_peered":"2023-10-14T18:18:37.274015+0000","last_clean":"2023-10-14T18:18:37.274015+0000","last_became_active":"2023-10-14T18:18:35.678924+0000","last_became_peered":"2023-10-14T18:18:35.678924+0000","last_unstale":"2023-10-14T18:18:37.274015+0000","last_undegraded":"2023-10-14T18:18:37.274015+0000","last_fullsized":"2023-10-14T18:18:37.274015+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:56.833692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651918+0000","last_change":"2023-10-14T18:18:35.683679+0000","last_active":"2023-10-14T18:18:36.651918+0000","last_peered":"2023-10-14T18:18:36.651918+0000","last_clean":"2023-10-14T18:18:36.651918+0000","last_became_active":"2023-10-14T18:18:35.682861+0000","last_became_peered":"2023-10-14T18:18:35.682861+0000","last_unstale":"2023-10-14T18:18:36.651918+0000","last_undegraded":"2023-10-14T18:18:36.651918+0000","last_fullsized":"2023-10-14T18:18:36.651918+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375268+0000","last_change":"2023-10-14T18:18:35.675043+0000","last_active":"2023-10-14T18:18:37.375268+0000","last_peered":"2023-10-14T18:18:37.375268+0000","last_clean":"2023-10-14T18:18:37.375268+0000","last_became_active":"2023-10-14T18:18:35.674231+0000","last_became_peered":"2023-10-14T18:18:35.674231+0000","last_unstale":"2023-10-14T18:18:37.375268+0000","last_undegraded":"2023-10-14T18:18:37.375268+0000","last_fullsized":"2023-10-14T18:18:37.375268+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:11:18.531967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650886+0000","last_change":"2023-10-14T18:18:35.678709+0000","last_active":"2023-10-14T18:18:36.650886+0000","last_peered":"2023-10-14T18:18:36.650886+0000","last_clean":"2023-10-14T18:18:36.650886+0000","last_became_active":"2023-10-14T18:18:35.678445+0000","last_became_peered":"2023-10-14T18:18:35.678445+0000","last_unstale":"2023-10-14T18:18:36.650886+0000","last_undegraded":"2023-10-14T18:18:36.650886+0000","last_fullsized":"2023-10-14T18:18:36.650886+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653036+0000","last_change":"2023-10-14T18:18:35.678715+0000","last_active":"2023-10-14T18:18:36.653036+0000","last_peered":"2023-10-14T18:18:36.653036+0000","last_clean":"2023-10-14T18:18:36.653036+0000","last_became_active":"2023-10-14T18:18:35.678366+0000","last_became_peered":"2023-10-14T18:18:35.678366+0000","last_unstale":"2023-10-14T18:18:36.653036+0000","last_undegraded":"2023-10-14T18:18:36.653036+0000","last_fullsized":"2023-10-14T18:18:36.653036+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649759+0000","last_change":"2023-10-14T18:18:35.674737+0000","last_active":"2023-10-14T18:18:36.649759+0000","last_peered":"2023-10-14T18:18:36.649759+0000","last_clean":"2023-10-14T18:18:36.649759+0000","last_became_active":"2023-10-14T18:18:35.673762+0000","last_became_peered":"2023-10-14T18:18:35.673762+0000","last_unstale":"2023-10-14T18:18:36.649759+0000","last_undegraded":"2023-10-14T18:18:36.649759+0000","last_fullsized":"2023-10-14T18:18:36.649759+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273276+0000","last_change":"2023-10-14T18:18:35.679931+0000","last_active":"2023-10-14T18:18:37.273276+0000","last_peered":"2023-10-14T18:18:37.273276+0000","last_clean":"2023-10-14T18:18:37.273276+0000","last_became_active":"2023-10-14T18:18:35.679636+0000","last_became_peered":"2023-10-14T18:18:35.679636+0000","last_unstale":"2023-10-14T18:18:37.273276+0000","last_undegraded":"2023-10-14T18:18:37.273276+0000","last_fullsized":"2023-10-14T18:18:37.273276+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:40.768705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644927+0000","last_change":"2023-10-14T18:18:35.663686+0000","last_active":"2023-10-14T18:18:36.644927+0000","last_peered":"2023-10-14T18:18:36.644927+0000","last_clean":"2023-10-14T18:18:36.644927+0000","last_became_active":"2023-10-14T18:18:35.663364+0000","last_became_peered":"2023-10-14T18:18:35.663364+0000","last_unstale":"2023-10-14T18:18:36.644927+0000","last_undegraded":"2023-10-14T18:18:36.644927+0000","last_fullsized":"2023-10-14T18:18:36.644927+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:43.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654128+0000","last_change":"2023-10-14T18:18:35.683088+0000","last_active":"2023-10-14T18:18:36.654128+0000","last_peered":"2023-10-14T18:18:36.654128+0000","last_clean":"2023-10-14T18:18:36.654128+0000","last_became_active":"2023-10-14T18:18:35.682753+0000","last_became_peered":"2023-10-14T18:18:35.682753+0000","last_unstale":"2023-10-14T18:18:36.654128+0000","last_undegraded":"2023-10-14T18:18:36.654128+0000","last_fullsized":"2023-10-14T18:18:36.654128+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375974+0000","last_change":"2023-10-14T18:18:35.689155+0000","last_active":"2023-10-14T18:18:37.375974+0000","last_peered":"2023-10-14T18:18:37.375974+0000","last_clean":"2023-10-14T18:18:37.375974+0000","last_became_active":"2023-10-14T18:18:35.686147+0000","last_became_peered":"2023-10-14T18:18:35.686147+0000","last_unstale":"2023-10-14T18:18:37.375974+0000","last_undegraded":"2023-10-14T18:18:37.375974+0000","last_fullsized":"2023-10-14T18:18:37.375974+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:12.203818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273778+0000","last_change":"2023-10-14T18:18:35.673716+0000","last_active":"2023-10-14T18:18:37.273778+0000","last_peered":"2023-10-14T18:18:37.273778+0000","last_clean":"2023-10-14T18:18:37.273778+0000","last_became_active":"2023-10-14T18:18:35.672662+0000","last_became_peered":"2023-10-14T18:18:35.672662+0000","last_unstale":"2023-10-14T18:18:37.273778+0000","last_undegraded":"2023-10-14T18:18:37.273778+0000","last_fullsized":"2023-10-14T18:18:37.273778+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:06.125956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374334+0000","last_change":"2023-10-14T18:18:35.682800+0000","last_active":"2023-10-14T18:18:37.374334+0000","last_peered":"2023-10-14T18:18:37.374334+0000","last_clean":"2023-10-14T18:18:37.374334+0000","last_became_active":"2023-10-14T18:18:35.682478+0000","last_became_peered":"2023-10-14T18:18:35.682478+0000","last_unstale":"2023-10-14T18:18:37.374334+0000","last_undegraded":"2023-10-14T18:18:37.374334+0000","last_fullsized":"2023-10-14T18:18:37.374334+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372354+0000","last_change":"2023-10-14T18:18:35.676776+0000","last_active":"2023-10-14T18:18:37.372354+0000","last_peered":"2023-10-14T18:18:37.372354+0000","last_clean":"2023-10-14T18:18:37.372354+0000","last_became_active":"2023-10-14T18:18:35.675446+0000","last_became_peered":"2023-10-14T18:18:35.675446+0000","last_unstale":"2023-10-14T18:18:37.372354+0000","last_undegraded":"2023-10-14T18:18:37.372354+0000","last_fullsized":"2023-10-14T18:18:37.372354+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:09.420076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644229+0000","last_change":"2023-10-14T18:18:35.676664+0000","last_active":"2023-10-14T18:18:36.644229+0000","last_peered":"2023-10-14T18:18:36.644229+0000","last_clean":"2023-10-14T18:18:36.644229+0000","last_became_active":"2023-10-14T18:18:35.676381+0000","last_became_peered":"2023-10-14T18:18:35.676381+0000","last_unstale":"2023-10-14T18:18:36.644229+0000","last_undegraded":"2023-10-14T18:18:36.644229+0000","last_fullsized":"2023-10-14T18:18:36.644229+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:23.246857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274107+0000","last_change":"2023-10-14T18:18:35.670711+0000","last_active":"2023-10-14T18:18:37.274107+0000","last_peered":"2023-10-14T18:18:37.274107+0000","last_clean":"2023-10-14T18:18:37.274107+0000","last_became_active":"2023-10-14T18:18:35.670350+0000","last_became_peered":"2023-10-14T18:18:35.670350+0000","last_unstale":"2023-10-14T18:18:37.274107+0000","last_undegraded":"2023-10-14T18:18:37.274107+0000","last_fullsized":"2023-10-14T18:18:37.274107+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:32.369325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375461+0000","last_change":"2023-10-14T18:18:35.677418+0000","last_active":"2023-10-14T18:18:37.375461+0000","last_peered":"2023-10-14T18:18:37.375461+0000","last_clean":"2023-10-14T18:18:37.375461+0000","last_became_active":"2023-10-14T18:18:35.677059+0000","last_became_peered":"2023-10-14T18:18:35.677059+0000","last_unstale":"2023-10-14T18:18:37.375461+0000","last_undegraded":"2023-10-14T18:18:37.375461+0000","last_fullsized":"2023-10-14T18:18:37.375461+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.342329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654400+0000","last_change":"2023-10-14T18:18:34.650335+0000","last_active":"2023-10-14T18:18:36.654400+0000","last_peered":"2023-10-14T18:18:36.654400+0000","last_clean":"2023-10-14T18:18:36.654400+0000","last_became_active":"2023-10-14T18:18:34.648549+0000","last_became_peered":"2023-10-14T18:18:34.648549+0000","last_unstale":"2023-10-14T18:18:36.654400+0000","last_undegraded":"2023-10-14T18:18:36.654400+0000","last_fullsized":"2023-10-14T18:18:36.654400+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374469+0000","last_change":"2023-10-14T18:18:35.684529+0000","last_active":"2023-10-14T18:18:37.374469+0000","last_peered":"2023-10-14T18:18:37.374469+0000","last_clean":"2023-10-14T18:18:37.374469+0000","last_became_active":"2023-10-14T18:18:35.684263+0000","last_became_peered":"2023-10-14T18:18:35.684263+0000","last_unstale":"2023-10-14T18:18:37.374469+0000","last_undegraded":"2023-10-14T18:18:37.374469+0000","last_fullsized":"2023-10-14T18:18:37.374469+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:06.771665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374880+0000","last_change":"2023-10-14T18:18:34.653624+0000","last_active":"2023-10-14T18:18:37.374880+0000","last_peered":"2023-10-14T18:18:37.374880+0000","last_clean":"2023-10-14T18:18:37.374880+0000","last_became_active":"2023-10-14T18:18:34.652877+0000","last_became_peered":"2023-10-14T18:18:34.652877+0000","last_unstale":"2023-10-14T18:18:37.374880+0000","last_undegraded":"2023-10-14T18:18:37.374880+0000","last_fullsized":"2023-10-14T18:18:37.374880+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:13.266689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652477+0000","last_change":"2023-10-14T18:18:35.671984+0000","last_active":"2023-10-14T18:18:36.652477+0000","last_peered":"2023-10-14T18:18:36.652477+0000","last_clean":"2023-10-14T18:18:36.652477+0000","last_became_active":"2023-10-14T18:18:35.668766+0000","last_became_peered":"2023-10-14T18:18:35.668766+0000","last_unstale":"2023-10-14T18:18:36.652477+0000","last_undegraded":"2023-10-14T18:18:36.652477+0000","last_fullsized":"2023-10-14T18:18:36.652477+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375523+0000","last_change":"2023-10-14T18:18:34.649372+0000","last_active":"2023-10-14T18:18:37.375523+0000","last_peered":"2023-10-14T18:18:37.375523+0000","last_clean":"2023-10-14T18:18:37.375523+0000","last_became_active":"2023-10-14T18:18:34.648998+0000","last_became_peered":"2023-10-14T18:18:34.648998+0000","last_unstale":"2023-10-14T18:18:37.375523+0000","last_undegraded":"2023-10-14T18:18:37.375523+0000","last_fullsized":"2023-10-14T18:18:37.375523+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:24.567915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373897+0000","last_change":"2023-10-14T18:18:35.672986+0000","last_active":"2023-10-14T18:18:37.373897+0000","last_peered":"2023-10-14T18:18:37.373897+0000","last_clean":"2023-10-14T18:18:37.373897+0000","last_became_active":"2023-10-14T18:18:35.672683+0000","last_became_peered":"2023-10-14T18:18:35.672683+0000","last_unstale":"2023-10-14T18:18:37.373897+0000","last_undegraded":"2023-10-14T18:18:37.373897+0000","last_fullsized":"2023-10-14T18:18:37.373897+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:59.451214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644245+0000","last_change":"2023-10-14T18:18:34.651378+0000","last_active":"2023-10-14T18:18:36.644245+0000","last_peered":"2023-10-14T18:18:36.644245+0000","last_clean":"2023-10-14T18:18:36.644245+0000","last_became_active":"2023-10-14T18:18:34.651072+0000","last_became_peered":"2023-10-14T18:18:34.651072+0000","last_unstale":"2023-10-14T18:18:36.644245+0000","last_undegraded":"2023-10-14T18:18:36.644245+0000","last_fullsized":"2023-10-14T18:18:36.644245+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:07:46.772700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373278+0000","last_change":"2023-10-14T18:18:35.691610+0000","last_active":"2023-10-14T18:18:37.373278+0000","last_peered":"2023-10-14T18:18:37.373278+0000","last_clean":"2023-10-14T18:18:37.373278+0000","last_became_active":"2023-10-14T18:18:35.691317+0000","last_became_peered":"2023-10-14T18:18:35.691317+0000","last_unstale":"2023-10-14T18:18:37.373278+0000","last_undegraded":"2023-10-14T18:18:37.373278+0000","last_fullsized":"2023-10-14T18:18:37.373278+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:24.619829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273569+0000","last_change":"2023-10-14T18:18:34.652803+0000","last_active":"2023-10-14T18:18:37.273569+0000","last_peered":"2023-10-14T18:18:37.273569+0000","last_clean":"2023-10-14T18:18:37.273569+0000","last_became_active":"2023-10-14T18:18:34.647316+0000","last_became_peered":"2023-10-14T18:18:34.647316+0000","last_unstale":"2023-10-14T18:18:37.273569+0000","last_undegraded":"2023-10-14T18:18:37.273569+0000","last_fullsized":"2023-10-14T18:18:37.273569+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:00.238815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.372926+0000","last_change":"2023-10-14T18:18:35.678351+0000","last_active":"2023-10-14T18:18:37.372926+0000","last_peered":"2023-10-14T18:18:37.372926+0000","last_clean":"2023-10-14T18:18:37.372926+0000","last_became_active":"2023-10-14T18:18:35.678058+0000","last_became_peered":"2023-10-14T18:18:35.678058+0000","last_unstale":"2023-10-14T18:18:37.372926+0000","last_undegraded":"2023-10-14T18:18:37.372926+0000","last_fullsized":"2023-10-14T18:18:37.372926+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:50.521263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652298+0000","last_change":"2023-10-14T18:18:34.651512+0000","last_active":"2023-10-14T18:18:36.652298+0000","last_peered":"2023-10-14T18:18:36.652298+0000","last_clean":"2023-10-14T18:18:36.652298+0000","last_became_active":"2023-10-14T18:18:34.651205+0000","last_became_peered":"2023-10-14T18:18:34.651205+0000","last_unstale":"2023-10-14T18:18:36.652298+0000","last_undegraded":"2023-10-14T18:18:36.652298+0000","last_fullsized":"2023-10-14T18:18:36.652298+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:11.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273362+0000","last_change":"2023-10-14T18:18:35.677057+0000","last_active":"2023-10-14T18:18:37.273362+0000","last_peered":"2023-10-14T18:18:37.273362+0000","last_clean":"2023-10-14T18:18:37.273362+0000","last_became_active":"2023-10-14T18:18:35.676751+0000","last_became_peered":"2023-10-14T18:18:35.676751+0000","last_unstale":"2023-10-14T18:18:37.273362+0000","last_undegraded":"2023-10-14T18:18:37.273362+0000","last_fullsized":"2023-10-14T18:18:37.273362+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:14.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373180+0000","last_change":"2023-10-14T18:18:34.645198+0000","last_active":"2023-10-14T18:18:37.373180+0000","last_peered":"2023-10-14T18:18:37.373180+0000","last_clean":"2023-10-14T18:18:37.373180+0000","last_became_active":"2023-10-14T18:18:34.644896+0000","last_became_peered":"2023-10-14T18:18:34.644896+0000","last_unstale":"2023-10-14T18:18:37.373180+0000","last_undegraded":"2023-10-14T18:18:37.373180+0000","last_fullsized":"2023-10-14T18:18:37.373180+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:17.263536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274102+0000","last_change":"2023-10-14T18:18:35.675636+0000","last_active":"2023-10-14T18:18:37.274102+0000","last_peered":"2023-10-14T18:18:37.274102+0000","last_clean":"2023-10-14T18:18:37.274102+0000","last_became_active":"2023-10-14T18:18:35.674527+0000","last_became_peered":"2023-10-14T18:18:35.674527+0000","last_unstale":"2023-10-14T18:18:37.274102+0000","last_undegraded":"2023-10-14T18:18:37.274102+0000","last_fullsized":"2023-10-14T18:18:37.274102+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:19.471165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652927+0000","last_change":"2023-10-14T18:18:34.640892+0000","last_active":"2023-10-14T18:18:36.652927+0000","last_peered":"2023-10-14T18:18:36.652927+0000","last_clean":"2023-10-14T18:18:36.652927+0000","last_became_active":"2023-10-14T18:18:34.639779+0000","last_became_peered":"2023-10-14T18:18:34.639779+0000","last_unstale":"2023-10-14T18:18:36.652927+0000","last_undegraded":"2023-10-14T18:18:36.652927+0000","last_fullsized":"2023-10-14T18:18:36.652927+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.376500+0000","last_change":"2023-10-14T18:18:35.685195+0000","last_active":"2023-10-14T18:18:37.376500+0000","last_peered":"2023-10-14T18:18:37.376500+0000","last_clean":"2023-10-14T18:18:37.376500+0000","last_became_active":"2023-10-14T18:18:35.682690+0000","last_became_peered":"2023-10-14T18:18:35.682690+0000","last_unstale":"2023-10-14T18:18:37.376500+0000","last_undegraded":"2023-10-14T18:18:37.376500+0000","last_fullsized":"2023-10-14T18:18:37.376500+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:54.132242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374719+0000","last_change":"2023-10-14T18:18:34.651854+0000","last_active":"2023-10-14T18:18:37.374719+0000","last_peered":"2023-10-14T18:18:37.374719+0000","last_clean":"2023-10-14T18:18:37.374719+0000","last_became_active":"2023-10-14T18:18:34.651540+0000","last_became_peered":"2023-10-14T18:18:34.651540+0000","last_unstale":"2023-10-14T18:18:37.374719+0000","last_undegraded":"2023-10-14T18:18:37.374719+0000","last_fullsized":"2023-10-14T18:18:37.374719+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.752351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375329+0000","last_change":"2023-10-14T18:18:35.679788+0000","last_active":"2023-10-14T18:18:37.375329+0000","last_peered":"2023-10-14T18:18:37.375329+0000","last_clean":"2023-10-14T18:18:37.375329+0000","last_became_active":"2023-10-14T18:18:35.679488+0000","last_became_peered":"2023-10-14T18:18:35.679488+0000","last_unstale":"2023-10-14T18:18:37.375329+0000","last_undegraded":"2023-10-14T18:18:37.375329+0000","last_fullsized":"2023-10-14T18:18:37.375329+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:39.475602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":15,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274265+0000","last_change":"2023-10-14T18:18:34.654465+0000","last_active":"2023-10-14T18:18:37.274265+0000","last_peered":"2023-10-14T18:18:37.274265+0000","last_clean":"2023-10-14T18:18:37.274265+0000","last_became_active":"2023-10-14T18:18:34.654111+0000","last_became_peered":"2023-10-14T18:18:34.654111+0000","last_unstale":"2023-10-14T18:18:37.274265+0000","last_undegraded":"2023-10-14T18:18:37.274265+0000","last_fullsized":"2023-10-14T18:18:37.274265+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:01.589582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653831+0000","last_change":"2023-10-14T18:18:35.683637+0000","last_active":"2023-10-14T18:18:36.653831+0000","last_peered":"2023-10-14T18:18:36.653831+0000","last_clean":"2023-10-14T18:18:36.653831+0000","last_became_active":"2023-10-14T18:18:35.683235+0000","last_became_peered":"2023-10-14T18:18:35.683235+0000","last_unstale":"2023-10-14T18:18:36.653831+0000","last_undegraded":"2023-10-14T18:18:36.653831+0000","last_fullsized":"2023-10-14T18:18:36.653831+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"249'6","reported_seq":18,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273702+0000","last_change":"2023-10-14T18:18:34.652609+0000","last_active":"2023-10-14T18:18:37.273702+0000","last_peered":"2023-10-14T18:18:37.273702+0000","last_clean":"2023-10-14T18:18:37.273702+0000","last_became_active":"2023-10-14T18:18:34.646287+0000","last_became_peered":"2023-10-14T18:18:34.646287+0000","last_unstale":"2023-10-14T18:18:37.273702+0000","last_undegraded":"2023-10-14T18:18:37.273702+0000","last_fullsized":"2023-10-14T18:18:37.273702+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:09.725037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653135+0000","last_change":"2023-10-14T18:18:35.672404+0000","last_active":"2023-10-14T18:18:36.653135+0000","last_peered":"2023-10-14T18:18:36.653135+0000","last_clean":"2023-10-14T18:18:36.653135+0000","last_became_active":"2023-10-14T18:18:35.672109+0000","last_became_peered":"2023-10-14T18:18:35.672109+0000","last_unstale":"2023-10-14T18:18:36.653135+0000","last_undegraded":"2023-10-14T18:18:36.653135+0000","last_fullsized":"2023-10-14T18:18:36.653135+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273692+0000","last_change":"2023-10-14T18:18:34.650921+0000","last_active":"2023-10-14T18:18:37.273692+0000","last_peered":"2023-10-14T18:18:37.273692+0000","last_clean":"2023-10-14T18:18:37.273692+0000","last_became_active":"2023-10-14T18:18:34.650170+0000","last_became_peered":"2023-10-14T18:18:34.650170+0000","last_unstale":"2023-10-14T18:18:37.273692+0000","last_undegraded":"2023-10-14T18:18:37.273692+0000","last_fullsized":"2023-10-14T18:18:37.273692+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:24.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373852+0000","last_change":"2023-10-14T18:18:35.685387+0000","last_active":"2023-10-14T18:18:37.373852+0000","last_peered":"2023-10-14T18:18:37.373852+0000","last_clean":"2023-10-14T18:18:37.373852+0000","last_became_active":"2023-10-14T18:18:35.684624+0000","last_became_peered":"2023-10-14T18:18:35.684624+0000","last_unstale":"2023-10-14T18:18:37.373852+0000","last_undegraded":"2023-10-14T18:18:37.373852+0000","last_fullsized":"2023-10-14T18:18:37.373852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:49.959769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374525+0000","last_change":"2023-10-14T18:18:34.644335+0000","last_active":"2023-10-14T18:18:37.374525+0000","last_peered":"2023-10-14T18:18:37.374525+0000","last_clean":"2023-10-14T18:18:37.374525+0000","last_became_active":"2023-10-14T18:18:34.644052+0000","last_became_peered":"2023-10-14T18:18:34.644052+0000","last_unstale":"2023-10-14T18:18:37.374525+0000","last_undegraded":"2023-10-14T18:18:37.374525+0000","last_fullsized":"2023-10-14T18:18:37.374525+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:00.968233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644478+0000","last_change":"2023-10-14T18:18:35.675619+0000","last_active":"2023-10-14T18:18:36.644478+0000","last_peered":"2023-10-14T18:18:36.644478+0000","last_clean":"2023-10-14T18:18:36.644478+0000","last_became_active":"2023-10-14T18:18:35.673589+0000","last_became_peered":"2023-10-14T18:18:35.673589+0000","last_unstale":"2023-10-14T18:18:36.644478+0000","last_undegraded":"2023-10-14T18:18:36.644478+0000","last_fullsized":"2023-10-14T18:18:36.644478+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:37.559321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652661+0000","last_change":"2023-10-14T18:18:34.646943+0000","last_active":"2023-10-14T18:18:36.652661+0000","last_peered":"2023-10-14T18:18:36.652661+0000","last_clean":"2023-10-14T18:18:36.652661+0000","last_became_active":"2023-10-14T18:18:34.646630+0000","last_became_peered":"2023-10-14T18:18:34.646630+0000","last_unstale":"2023-10-14T18:18:36.652661+0000","last_undegraded":"2023-10-14T18:18:36.652661+0000","last_fullsized":"2023-10-14T18:18:36.652661+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:57.489531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651630+0000","last_change":"2023-10-14T18:18:35.678311+0000","last_active":"2023-10-14T18:18:36.651630+0000","last_peered":"2023-10-14T18:18:36.651630+0000","last_clean":"2023-10-14T18:18:36.651630+0000","last_became_active":"2023-10-14T18:18:35.678016+0000","last_became_peered":"2023-10-14T18:18:35.678016+0000","last_unstale":"2023-10-14T18:18:36.651630+0000","last_undegraded":"2023-10-14T18:18:36.651630+0000","last_fullsized":"2023-10-14T18:18:36.651630+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":20,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373557+0000","last_change":"2023-10-14T18:18:34.648173+0000","last_active":"2023-10-14T18:18:37.373557+0000","last_peered":"2023-10-14T18:18:37.373557+0000","last_clean":"2023-10-14T18:18:37.373557+0000","last_became_active":"2023-10-14T18:18:34.647904+0000","last_became_peered":"2023-10-14T18:18:34.647904+0000","last_unstale":"2023-10-14T18:18:37.373557+0000","last_undegraded":"2023-10-14T18:18:37.373557+0000","last_fullsized":"2023-10-14T18:18:37.373557+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:49.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644809+0000","last_change":"2023-10-14T18:18:35.672233+0000","last_active":"2023-10-14T18:18:36.644809+0000","last_peered":"2023-10-14T18:18:36.644809+0000","last_clean":"2023-10-14T18:18:36.644809+0000","last_became_active":"2023-10-14T18:18:35.671904+0000","last_became_peered":"2023-10-14T18:18:35.671904+0000","last_unstale":"2023-10-14T18:18:36.644809+0000","last_undegraded":"2023-10-14T18:18:36.644809+0000","last_fullsized":"2023-10-14T18:18:36.644809+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:09.351543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654314+0000","last_change":"2023-10-14T18:18:34.644806+0000","last_active":"2023-10-14T18:18:36.654314+0000","last_peered":"2023-10-14T18:18:36.654314+0000","last_clean":"2023-10-14T18:18:36.654314+0000","last_became_active":"2023-10-14T18:18:34.642485+0000","last_became_peered":"2023-10-14T18:18:34.642485+0000","last_unstale":"2023-10-14T18:18:36.654314+0000","last_undegraded":"2023-10-14T18:18:36.654314+0000","last_fullsized":"2023-10-14T18:18:36.654314+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:22.234439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375522+0000","last_change":"2023-10-14T18:18:35.673621+0000","last_active":"2023-10-14T18:18:37.375522+0000","last_peered":"2023-10-14T18:18:37.375522+0000","last_clean":"2023-10-14T18:18:37.375522+0000","last_became_active":"2023-10-14T18:18:35.673296+0000","last_became_peered":"2023-10-14T18:18:35.673296+0000","last_unstale":"2023-10-14T18:18:37.375522+0000","last_undegraded":"2023-10-14T18:18:37.375522+0000","last_fullsized":"2023-10-14T18:18:37.375522+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:58:27.889006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375020+0000","last_change":"2023-10-14T18:18:34.655553+0000","last_active":"2023-10-14T18:18:37.375020+0000","last_peered":"2023-10-14T18:18:37.375020+0000","last_clean":"2023-10-14T18:18:37.375020+0000","last_became_active":"2023-10-14T18:18:34.655113+0000","last_became_peered":"2023-10-14T18:18:34.655113+0000","last_unstale":"2023-10-14T18:18:37.375020+0000","last_undegraded":"2023-10-14T18:18:37.375020+0000","last_fullsized":"2023-10-14T18:18:37.375020+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:04:45.880556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373317+0000","last_change":"2023-10-14T18:18:35.683896+0000","last_active":"2023-10-14T18:18:37.373317+0000","last_peered":"2023-10-14T18:18:37.373317+0000","last_clean":"2023-10-14T18:18:37.373317+0000","last_became_active":"2023-10-14T18:18:35.683606+0000","last_became_peered":"2023-10-14T18:18:35.683606+0000","last_unstale":"2023-10-14T18:18:37.373317+0000","last_undegraded":"2023-10-14T18:18:37.373317+0000","last_fullsized":"2023-10-14T18:18:37.373317+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:02.619684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":13,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373307+0000","last_change":"2023-10-14T18:18:34.647730+0000","last_active":"2023-10-14T18:18:37.373307+0000","last_peered":"2023-10-14T18:18:37.373307+0000","last_clean":"2023-10-14T18:18:37.373307+0000","last_became_active":"2023-10-14T18:18:34.646698+0000","last_became_peered":"2023-10-14T18:18:34.646698+0000","last_unstale":"2023-10-14T18:18:37.373307+0000","last_undegraded":"2023-10-14T18:18:37.373307+0000","last_fullsized":"2023-10-14T18:18:37.373307+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:44.897768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.652299+0000","last_change":"2023-10-14T18:18:35.675449+0000","last_active":"2023-10-14T18:18:36.652299+0000","last_peered":"2023-10-14T18:18:36.652299+0000","last_clean":"2023-10-14T18:18:36.652299+0000","last_became_active":"2023-10-14T18:18:35.675129+0000","last_became_peered":"2023-10-14T18:18:35.675129+0000","last_unstale":"2023-10-14T18:18:36.652299+0000","last_undegraded":"2023-10-14T18:18:36.652299+0000","last_fullsized":"2023-10-14T18:18:36.652299+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374720+0000","last_change":"2023-10-14T18:18:34.645342+0000","last_active":"2023-10-14T18:18:37.374720+0000","last_peered":"2023-10-14T18:18:37.374720+0000","last_clean":"2023-10-14T18:18:37.374720+0000","last_became_active":"2023-10-14T18:18:34.643272+0000","last_became_peered":"2023-10-14T18:18:34.643272+0000","last_unstale":"2023-10-14T18:18:37.374720+0000","last_undegraded":"2023-10-14T18:18:37.374720+0000","last_fullsized":"2023-10-14T18:18:37.374720+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:08.339125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650684+0000","last_change":"2023-10-14T18:18:35.675593+0000","last_active":"2023-10-14T18:18:36.650684+0000","last_peered":"2023-10-14T18:18:36.650684+0000","last_clean":"2023-10-14T18:18:36.650684+0000","last_became_active":"2023-10-14T18:18:35.675294+0000","last_became_peered":"2023-10-14T18:18:35.675294+0000","last_unstale":"2023-10-14T18:18:36.650684+0000","last_undegraded":"2023-10-14T18:18:36.650684+0000","last_fullsized":"2023-10-14T18:18:36.650684+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375369+0000","last_change":"2023-10-14T18:18:34.640130+0000","last_active":"2023-10-14T18:18:37.375369+0000","last_peered":"2023-10-14T18:18:37.375369+0000","last_clean":"2023-10-14T18:18:37.375369+0000","last_became_active":"2023-10-14T18:18:34.639677+0000","last_became_peered":"2023-10-14T18:18:34.639677+0000","last_unstale":"2023-10-14T18:18:37.375369+0000","last_undegraded":"2023-10-14T18:18:37.375369+0000","last_fullsized":"2023-10-14T18:18:37.375369+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:09.948727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273394+0000","last_change":"2023-10-14T18:18:35.682891+0000","last_active":"2023-10-14T18:18:37.273394+0000","last_peered":"2023-10-14T18:18:37.273394+0000","last_clean":"2023-10-14T18:18:37.273394+0000","last_became_active":"2023-10-14T18:18:35.681440+0000","last_became_peered":"2023-10-14T18:18:35.681440+0000","last_unstale":"2023-10-14T18:18:37.273394+0000","last_undegraded":"2023-10-14T18:18:37.273394+0000","last_fullsized":"2023-10-14T18:18:37.273394+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:50.330635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373798+0000","last_change":"2023-10-14T18:18:34.647137+0000","last_active":"2023-10-14T18:18:37.373798+0000","last_peered":"2023-10-14T18:18:37.373798+0000","last_clean":"2023-10-14T18:18:37.373798+0000","last_became_active":"2023-10-14T18:18:34.646766+0000","last_became_peered":"2023-10-14T18:18:34.646766+0000","last_unstale":"2023-10-14T18:18:37.373798+0000","last_undegraded":"2023-10-14T18:18:37.373798+0000","last_fullsized":"2023-10-14T18:18:37.373798+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373613+0000","last_change":"2023-10-14T18:18:35.672939+0000","last_active":"2023-10-14T18:18:37.373613+0000","last_peered":"2023-10-14T18:18:37.373613+0000","last_clean":"2023-10-14T18:18:37.373613+0000","last_became_active":"2023-10-14T18:18:35.672630+0000","last_became_peered":"2023-10-14T18:18:35.672630+0000","last_unstale":"2023-10-14T18:18:37.373613+0000","last_undegraded":"2023-10-14T18:18:37.373613+0000","last_fullsized":"2023-10-14T18:18:37.373613+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:56.149979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373307+0000","last_change":"2023-10-14T18:18:34.653097+0000","last_active":"2023-10-14T18:18:37.373307+0000","last_peered":"2023-10-14T18:18:37.373307+0000","last_clean":"2023-10-14T18:18:37.373307+0000","last_became_active":"2023-10-14T18:18:34.652338+0000","last_became_peered":"2023-10-14T18:18:34.652338+0000","last_unstale":"2023-10-14T18:18:37.373307+0000","last_undegraded":"2023-10-14T18:18:37.373307+0000","last_fullsized":"2023-10-14T18:18:37.373307+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:39.725847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.650630+0000","last_change":"2023-10-14T18:18:35.668420+0000","last_active":"2023-10-14T18:18:36.650630+0000","last_peered":"2023-10-14T18:18:36.650630+0000","last_clean":"2023-10-14T18:18:36.650630+0000","last_became_active":"2023-10-14T18:18:35.667429+0000","last_became_peered":"2023-10-14T18:18:35.667429+0000","last_unstale":"2023-10-14T18:18:36.650630+0000","last_undegraded":"2023-10-14T18:18:36.650630+0000","last_fullsized":"2023-10-14T18:18:36.650630+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651660+0000","last_change":"2023-10-14T18:18:34.649612+0000","last_active":"2023-10-14T18:18:36.651660+0000","last_peered":"2023-10-14T18:18:36.651660+0000","last_clean":"2023-10-14T18:18:36.651660+0000","last_became_active":"2023-10-14T18:18:34.649307+0000","last_became_peered":"2023-10-14T18:18:34.649307+0000","last_unstale":"2023-10-14T18:18:36.651660+0000","last_undegraded":"2023-10-14T18:18:36.651660+0000","last_fullsized":"2023-10-14T18:18:36.651660+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.438796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.654483+0000","last_change":"2023-10-14T18:18:35.666113+0000","last_active":"2023-10-14T18:18:36.654483+0000","last_peered":"2023-10-14T18:18:36.654483+0000","last_clean":"2023-10-14T18:18:36.654483+0000","last_became_active":"2023-10-14T18:18:35.665775+0000","last_became_peered":"2023-10-14T18:18:35.665775+0000","last_unstale":"2023-10-14T18:18:36.654483+0000","last_undegraded":"2023-10-14T18:18:36.654483+0000","last_fullsized":"2023-10-14T18:18:36.654483+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":14,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644936+0000","last_change":"2023-10-14T18:18:34.657934+0000","last_active":"2023-10-14T18:18:36.644936+0000","last_peered":"2023-10-14T18:18:36.644936+0000","last_clean":"2023-10-14T18:18:36.644936+0000","last_became_active":"2023-10-14T18:18:34.657614+0000","last_became_peered":"2023-10-14T18:18:34.657614+0000","last_unstale":"2023-10-14T18:18:36.644936+0000","last_undegraded":"2023-10-14T18:18:36.644936+0000","last_fullsized":"2023-10-14T18:18:36.644936+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:11.487790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.274290+0000","last_change":"2023-10-14T18:18:35.680237+0000","last_active":"2023-10-14T18:18:37.274290+0000","last_peered":"2023-10-14T18:18:37.274290+0000","last_clean":"2023-10-14T18:18:37.274290+0000","last_became_active":"2023-10-14T18:18:35.679960+0000","last_became_peered":"2023-10-14T18:18:35.679960+0000","last_unstale":"2023-10-14T18:18:37.274290+0000","last_undegraded":"2023-10-14T18:18:37.274290+0000","last_fullsized":"2023-10-14T18:18:37.274290+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:53.427626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373452+0000","last_change":"2023-10-14T18:18:34.651448+0000","last_active":"2023-10-14T18:18:37.373452+0000","last_peered":"2023-10-14T18:18:37.373452+0000","last_clean":"2023-10-14T18:18:37.373452+0000","last_became_active":"2023-10-14T18:18:34.651100+0000","last_became_peered":"2023-10-14T18:18:34.651100+0000","last_unstale":"2023-10-14T18:18:37.373452+0000","last_undegraded":"2023-10-14T18:18:37.373452+0000","last_fullsized":"2023-10-14T18:18:37.373452+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:41.231071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374446+0000","last_change":"2023-10-14T18:18:35.687662+0000","last_active":"2023-10-14T18:18:37.374446+0000","last_peered":"2023-10-14T18:18:37.374446+0000","last_clean":"2023-10-14T18:18:37.374446+0000","last_became_active":"2023-10-14T18:18:35.687396+0000","last_became_peered":"2023-10-14T18:18:35.687396+0000","last_unstale":"2023-10-14T18:18:37.374446+0000","last_undegraded":"2023-10-14T18:18:37.374446+0000","last_fullsized":"2023-10-14T18:18:37.374446+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:15.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373758+0000","last_change":"2023-10-14T18:18:34.649374+0000","last_active":"2023-10-14T18:18:37.373758+0000","last_peered":"2023-10-14T18:18:37.373758+0000","last_clean":"2023-10-14T18:18:37.373758+0000","last_became_active":"2023-10-14T18:18:34.649074+0000","last_became_peered":"2023-10-14T18:18:34.649074+0000","last_unstale":"2023-10-14T18:18:37.373758+0000","last_undegraded":"2023-10-14T18:18:37.373758+0000","last_fullsized":"2023-10-14T18:18:37.373758+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:45.885592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.376126+0000","last_change":"2023-10-14T18:18:35.682974+0000","last_active":"2023-10-14T18:18:37.376126+0000","last_peered":"2023-10-14T18:18:37.376126+0000","last_clean":"2023-10-14T18:18:37.376126+0000","last_became_active":"2023-10-14T18:18:35.682647+0000","last_became_peered":"2023-10-14T18:18:35.682647+0000","last_unstale":"2023-10-14T18:18:37.376126+0000","last_undegraded":"2023-10-14T18:18:37.376126+0000","last_fullsized":"2023-10-14T18:18:37.376126+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:58.501949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.374987+0000","last_change":"2023-10-14T18:18:34.656380+0000","last_active":"2023-10-14T18:18:37.374987+0000","last_peered":"2023-10-14T18:18:37.374987+0000","last_clean":"2023-10-14T18:18:37.374987+0000","last_became_active":"2023-10-14T18:18:34.656073+0000","last_became_peered":"2023-10-14T18:18:34.656073+0000","last_unstale":"2023-10-14T18:18:37.374987+0000","last_undegraded":"2023-10-14T18:18:37.374987+0000","last_fullsized":"2023-10-14T18:18:37.374987+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:41.992815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653450+0000","last_change":"2023-10-14T18:18:35.666278+0000","last_active":"2023-10-14T18:18:36.653450+0000","last_peered":"2023-10-14T18:18:36.653450+0000","last_clean":"2023-10-14T18:18:36.653450+0000","last_became_active":"2023-10-14T18:18:35.665963+0000","last_became_peered":"2023-10-14T18:18:35.665963+0000","last_unstale":"2023-10-14T18:18:36.653450+0000","last_undegraded":"2023-10-14T18:18:36.653450+0000","last_fullsized":"2023-10-14T18:18:36.653450+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.653797+0000","last_change":"2023-10-14T18:18:34.649641+0000","last_active":"2023-10-14T18:18:36.653797+0000","last_peered":"2023-10-14T18:18:36.653797+0000","last_clean":"2023-10-14T18:18:36.653797+0000","last_became_active":"2023-10-14T18:18:34.649340+0000","last_became_peered":"2023-10-14T18:18:34.649340+0000","last_unstale":"2023-10-14T18:18:36.653797+0000","last_undegraded":"2023-10-14T18:18:36.653797+0000","last_fullsized":"2023-10-14T18:18:36.653797+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:05.759247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.651274+0000","last_change":"2023-10-14T18:18:35.675054+0000","last_active":"2023-10-14T18:18:36.651274+0000","last_peered":"2023-10-14T18:18:36.651274+0000","last_clean":"2023-10-14T18:18:36.651274+0000","last_became_active":"2023-10-14T18:18:35.674739+0000","last_became_peered":"2023-10-14T18:18:35.674739+0000","last_unstale":"2023-10-14T18:18:36.651274+0000","last_undegraded":"2023-10-14T18:18:36.651274+0000","last_fullsized":"2023-10-14T18:18:36.651274+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.644693+0000","last_change":"2023-10-14T18:18:34.652028+0000","last_active":"2023-10-14T18:18:36.644693+0000","last_peered":"2023-10-14T18:18:36.644693+0000","last_clean":"2023-10-14T18:18:36.644693+0000","last_became_active":"2023-10-14T18:18:34.651757+0000","last_became_peered":"2023-10-14T18:18:34.651757+0000","last_unstale":"2023-10-14T18:18:36.644693+0000","last_undegraded":"2023-10-14T18:18:36.644693+0000","last_fullsized":"2023-10-14T18:18:36.644693+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:36.649847+0000","last_change":"2023-10-14T18:18:35.660261+0000","last_active":"2023-10-14T18:18:36.649847+0000","last_peered":"2023-10-14T18:18:36.649847+0000","last_clean":"2023-10-14T18:18:36.649847+0000","last_became_active":"2023-10-14T18:18:35.659486+0000","last_became_peered":"2023-10-14T18:18:35.659486+0000","last_unstale":"2023-10-14T18:18:36.649847+0000","last_undegraded":"2023-10-14T18:18:36.649847+0000","last_fullsized":"2023-10-14T18:18:36.649847+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373010+0000","last_change":"2023-10-14T18:18:34.643288+0000","last_active":"2023-10-14T18:18:37.373010+0000","last_peered":"2023-10-14T18:18:37.373010+0000","last_clean":"2023-10-14T18:18:37.373010+0000","last_became_active":"2023-10-14T18:18:34.642997+0000","last_became_peered":"2023-10-14T18:18:34.642997+0000","last_unstale":"2023-10-14T18:18:37.373010+0000","last_undegraded":"2023-10-14T18:18:37.373010+0000","last_fullsized":"2023-10-14T18:18:37.373010+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:39.884500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.273854+0000","last_change":"2023-10-14T18:18:35.674240+0000","last_active":"2023-10-14T18:18:37.273854+0000","last_peered":"2023-10-14T18:18:37.273854+0000","last_clean":"2023-10-14T18:18:37.273854+0000","last_became_active":"2023-10-14T18:18:35.671067+0000","last_became_peered":"2023-10-14T18:18:35.671067+0000","last_unstale":"2023-10-14T18:18:37.273854+0000","last_undegraded":"2023-10-14T18:18:37.273854+0000","last_fullsized":"2023-10-14T18:18:37.273854+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:02:59.948900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":12,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373424+0000","last_change":"2023-10-14T18:18:34.643687+0000","last_active":"2023-10-14T18:18:37.373424+0000","last_peered":"2023-10-14T18:18:37.373424+0000","last_clean":"2023-10-14T18:18:37.373424+0000","last_became_active":"2023-10-14T18:18:34.643092+0000","last_became_peered":"2023-10-14T18:18:34.643092+0000","last_unstale":"2023-10-14T18:18:37.373424+0000","last_undegraded":"2023-10-14T18:18:37.373424+0000","last_fullsized":"2023-10-14T18:18:37.373424+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.480279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":11,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.373227+0000","last_change":"2023-10-14T18:18:35.685028+0000","last_active":"2023-10-14T18:18:37.373227+0000","last_peered":"2023-10-14T18:18:37.373227+0000","last_clean":"2023-10-14T18:18:37.373227+0000","last_became_active":"2023-10-14T18:18:35.684687+0000","last_became_peered":"2023-10-14T18:18:35.684687+0000","last_unstale":"2023-10-14T18:18:37.373227+0000","last_undegraded":"2023-10-14T18:18:37.373227+0000","last_fullsized":"2023-10-14T18:18:37.373227+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:27.530239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":16,"reported_epoch":250,"state":"active+clean","last_fresh":"2023-10-14T18:18:37.375704+0000","last_change":"2023-10-14T18:18:34.640893+0000","last_active":"2023-10-14T18:18:37.375704+0000","last_peered":"2023-10-14T18:18:37.375704+0000","last_clean":"2023-10-14T18:18:37.375704+0000","last_became_active":"2023-10-14T18:18:34.639717+0000","last_became_peered":"2023-10-14T18:18:34.639717+0000","last_unstale":"2023-10-14T18:18:37.375704+0000","last_undegraded":"2023-10-14T18:18:37.375704+0000","last_fullsized":"2023-10-14T18:18:37.375704+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:55.441603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"235'370","reported_seq":1308,"reported_epoch":249,"state":"active+clean","last_fresh":"2023-10-14T18:18:35.670189+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:18:35.670189+0000","last_peered":"2023-10-14T18:18:35.670189+0000","last_clean":"2023-10-14T18:18:35.670189+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:18:35.670189+0000","last_undegraded":"2023-10-14T18:18:35.670189+0000","last_fullsized":"2023-10-14T18:18:35.670189+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576018,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57036,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314804,"statfs":{"total":96636764160,"available":96578359296,"internally_reserved":0,"allocated":3731456,"data_stored":2151046,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.177,"15min":1.2030000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.351},{"interface":"front","average":{"1min":1.163,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":2.2149999999999999},"last":0.98299999999999998}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.768,"5min":1.9059999999999999,"15min":2.214},"last":1.4470000000000001},{"interface":"front","average":{"1min":1.145,"5min":1.2909999999999999,"15min":1.3109999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.3919999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.206,"15min":1.26},"min":{"1min":0.83199999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.839,"5min":1.839,"15min":2.004},"last":0.878},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3100000000000001,"15min":1.365},"min":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.2010000000000001},"last":1.2450000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.347,"15min":1.4239999999999999},"min":{"1min":0.94799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.454,"5min":1.488,"15min":1.528},"min":{"1min":1.0600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.994,"5min":1.994,"15min":2.2810000000000001},"last":1.542}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3680000000000001,"15min":1.343},"min":{"1min":0.90500000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.367},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3759999999999999,"15min":1.3919999999999999},"min":{"1min":1.0529999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.661},"last":1.1499999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4370000000000001,"15min":1.4670000000000001},"min":{"1min":1.0409999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.516},"last":1.667},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.448},"last":1.2150000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.615,"15min":1.639},"min":{"1min":1.103,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.51,"5min":1.5289999999999999,"15min":1.577},"min":{"1min":1.1399999999999999,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.6339999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48156,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323684,"statfs":{"total":96636764160,"available":96587452416,"internally_reserved":0,"allocated":3153920,"data_stored":1567628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.167,"15min":1.1779999999999999},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.669,"5min":2.085,"15min":2.085},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.2070000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.704,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.1419999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.208,"15min":1.24},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.494,"5min":1.762,"15min":1.776},"last":1.272},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.81799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.1760000000000002,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.665}]},{"osd":2,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3740000000000001,"15min":1.377},"min":{"1min":0.85999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.819,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.978},{"interface":"front","average":{"1min":1.373,"5min":1.397,"15min":1.419},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.6180000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.347,"15min":1.4059999999999999},"min":{"1min":0.72799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.2519999999999998},"last":1.75},{"interface":"front","average":{"1min":1.357,"5min":1.411,"15min":1.4359999999999999},"min":{"1min":0.91300000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8220000000000001,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.5009999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3899999999999999,"15min":1.3859999999999999},"min":{"1min":0.96499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7949999999999999,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.5800000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":1.109,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0030000000000001,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.8759999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.411,"15min":1.419},"min":{"1min":1.024,"5min":0.624,"15min":0.624},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":0.88},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5780000000000001,"15min":1.5840000000000001},"min":{"1min":1.2310000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.5609999999999999},"last":0.97899999999999998}]},{"osd":7,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.6140000000000001,"15min":1.6180000000000001},"min":{"1min":1.403,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.194,"5min":2.903,"15min":3.423},"last":2.073},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.583,"15min":1.6040000000000001},"min":{"1min":1.296,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.2349999999999999,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.4199999999999999}]}]},{"osd":5,"up_from":13,"seq":55834576018,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42560,"kb_used_data":3060,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329280,"statfs":{"total":96636764160,"available":96593182720,"internally_reserved":0,"allocated":3133440,"data_stored":1560608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1719999999999999,"15min":1.2070000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.69699999999999995},{"interface":"front","average":{"1min":1.089,"5min":1.1739999999999999,"15min":1.2350000000000001},"min":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.629,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":0.77400000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.234,"15min":1.276},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6850000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.95999999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.296,"15min":1.3100000000000001},"min":{"1min":0.92100000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.4019999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1799999999999999,"15min":1.2430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.591,"5min":1.764,"15min":3.7090000000000001},"last":1.22},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.377,"15min":1.397},"min":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":1.0609999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4159999999999999,"15min":1.46},"min":{"1min":0.94299999999999995,"5min":0.94199999999999995,"15min":0.88900000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.548},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.4339999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.8330000000000002},"last":1.1140000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3879999999999999,"15min":1.3660000000000001},"min":{"1min":1.0700000000000001,"5min":0.85899999999999999,"15min":0.78400000000000003},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.98},"last":1.171},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4370000000000001,"15min":1.401},"min":{"1min":0.90400000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.516,"5min":2.516,"15min":3.1280000000000001},"last":1.629}]},{"osd":6,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.399,"15min":1.4259999999999999},"min":{"1min":0.92900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":4.0540000000000003},"last":1.3540000000000001},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.6259999999999999,"15min":1.635},"min":{"1min":1.429,"5min":0.93000000000000005,"15min":0.81799999999999995},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":3.6360000000000001},"last":1.484}]},{"osd":7,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5609999999999999,"15min":1.585},"min":{"1min":1.121,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":3.2759999999999998},"last":1.6919999999999999},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5840000000000001,"15min":1.5680000000000001},"min":{"1min":1.1419999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.9390000000000001},"last":1.4490000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576018,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66148,"kb_used_data":3092,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305692,"statfs":{"total":96636764160,"available":96569028608,"internally_reserved":0,"allocated":3166208,"data_stored":1568728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.181,"15min":1.2010000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.6619999999999999},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.266,"15min":1.2829999999999999},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.04}]},{"osd":1,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.282,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.59399999999999997},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.5980000000000001},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.95599999999999996}]},{"osd":2,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.286},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.524},"last":1.524},{"interface":"front","average":{"1min":1.216,"5min":1.389,"15min":1.4299999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.65700000000000003},"max":{"1min":1.597,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.444}]},{"osd":3,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":1.0029999999999999,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.6739999999999999,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.1240000000000001},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":1.0009999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.746}]},{"osd":5,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.397},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.69299999999999995},"max":{"1min":1.923,"5min":1.9319999999999999,"15min":2.02},"last":1.2010000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.393,"15min":1.3839999999999999},"min":{"1min":0.90000000000000002,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.452,"15min":1.4370000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.577,"15min":1.569},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82699999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.7050000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.5229999999999999,"15min":1.522},"min":{"1min":1.248,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.946,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.492},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6080000000000001,"15min":1.5940000000000001},"min":{"1min":1.2889999999999999,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":2.597,"5min":2.597,"15min":3.8610000000000002},"last":1.867}]}]},{"osd":3,"up_from":13,"seq":55834576018,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56456,"kb_used_data":3064,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315384,"statfs":{"total":96636764160,"available":96578953216,"internally_reserved":0,"allocated":3137536,"data_stored":1561190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94399999999999995,"15min":0.97599999999999998},"min":{"1min":0.45700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.4239999999999999,"5min":1.4870000000000001,"15min":1.855},"last":1.1719999999999999},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1870000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.504,"5min":1.6399999999999999,"15min":1.9970000000000001},"last":0.92500000000000004}]},{"osd":1,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94799999999999995,"15min":0.996},"min":{"1min":0.47099999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.363,"5min":1.835,"15min":1.919},"last":1.1040000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.1970000000000001},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.8560000000000001},"last":1.032}]},{"osd":2,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.339,"15min":1.4159999999999999},"min":{"1min":0.54900000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.5620000000000001,"5min":2.032,"15min":2.032},"last":0.84199999999999997},{"interface":"front","average":{"1min":1.04,"5min":1.26,"15min":1.3640000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":0.80200000000000005}]},{"osd":4,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3819999999999999,"15min":1.429},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9099999999999999,"5min":2.153,"15min":2.153},"last":1.218},{"interface":"front","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.478},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.25}]},{"osd":5,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4239999999999999,"15min":1.4550000000000001},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.367,"5min":1.46,"15min":1.548},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.02,"5min":2.1360000000000001,"15min":2.891},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5700000000000001,"15min":1.621},"min":{"1min":0.97199999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.335},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.6339999999999999,"15min":1.677},"min":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.4139999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.532,"15min":1.599},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":3.153,"5min":3.153,"15min":3.153},"last":1.5580000000000001},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.635,"15min":1.681},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.454}]}]},{"osd":2,"up_from":13,"seq":55834576018,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52304,"kb_used_data":3076,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319536,"statfs":{"total":96636764160,"available":96583204864,"internally_reserved":0,"allocated":3149824,"data_stored":1567114,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3029999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.98099999999999998,"15min":1.081},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.2629999999999999,"5min":1.4550000000000001,"15min":1.8660000000000001},"last":0.439}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":1.0069999999999999},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1439999999999999,"5min":1.603,"15min":1.881},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.1679999999999999,"15min":1.212},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5780000000000001,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.60299999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.218,"15min":1.2749999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.3959999999999999,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":1.081,"5min":1.337,"15min":1.3779999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7490000000000001,"5min":2.1579999999999999,"15min":2.165},"last":0.54400000000000004}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.464},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.736,"5min":2.097,"15min":2.097},"last":0.875},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3300000000000001,"15min":1.365},"min":{"1min":0.72399999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.944,"5min":1.944,"15min":2.024},"last":0.80100000000000005}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.389,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.2839999999999998},"last":0.84999999999999998},{"interface":"front","average":{"1min":1.339,"5min":1.577,"15min":1.583},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.0049999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":0.755}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.52,"15min":1.5289999999999999},"min":{"1min":0.80600000000000005,"5min":0.747,"15min":0.747},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":1.0129999999999999},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.556,"15min":1.603},"min":{"1min":0.91100000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1400000000000001,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":1.03}]},{"osd":7,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5780000000000001,"15min":1.5900000000000001},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.633,"15min":1.675},"min":{"1min":0.97999999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0600000000000001,"5min":2.331,"15min":2.331},"last":0.98499999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576018,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52308,"kb_used_data":3080,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319532,"statfs":{"total":96636764160,"available":96583200768,"internally_reserved":0,"allocated":3153920,"data_stored":1562344,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.996,"5min":0.95899999999999996,"15min":0.97199999999999998},"min":{"1min":0.51100000000000001,"5min":0.38,"15min":0.38},"max":{"1min":1.5169999999999999,"5min":1.647,"15min":2.4929999999999999},"last":1.1200000000000001},{"interface":"front","average":{"1min":1.109,"5min":1.0820000000000001,"15min":1.1279999999999999},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":2.847,"5min":2.847,"15min":2.847},"last":0.86499999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":1.038,"15min":1.0629999999999999},"min":{"1min":0.51700000000000002,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":2.1200000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.96699999999999997},{"interface":"front","average":{"1min":1.1839999999999999,"5min":1.2170000000000001,"15min":1.2629999999999999},"min":{"1min":0.54500000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.895,"5min":1.996,"15min":2.6859999999999999},"last":1.5129999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.321,"5min":1.357,"15min":1.385},"min":{"1min":0.52700000000000002,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":2.9940000000000002,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.7370000000000001},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.2829999999999999,"15min":1.3169999999999999},"min":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":4.8630000000000004,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.1739999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.518,"5min":1.4350000000000001,"15min":1.4690000000000001},"min":{"1min":0.88200000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":4.8090000000000002,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.7769999999999999},{"interface":"front","average":{"1min":1.298,"5min":1.385,"15min":1.431},"min":{"1min":0.80400000000000005,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.218,"5min":2.218,"15min":2.899},"last":1.3879999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3580000000000001,"5min":1.4379999999999999,"15min":1.4710000000000001},"min":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.173,"5min":2.173,"15min":3.004},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.417,"5min":1.55,"15min":1.5660000000000001},"min":{"1min":0.91800000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":1.9359999999999999,"5min":2.093,"15min":2.5129999999999999},"last":1.325}]},{"osd":6,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.5489999999999999,"5min":1.5780000000000001,"15min":1.6140000000000001},"min":{"1min":0.95499999999999996,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.8999999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.629},{"interface":"front","average":{"1min":1.863,"5min":1.649,"15min":1.627},"min":{"1min":0.871,"5min":0.871,"15min":0.871},"max":{"1min":10.173999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.667}]},{"osd":7,"last update":"Sat Oct 14 18:17:40 2023","interfaces":[{"interface":"back","average":{"1min":1.603,"5min":1.6679999999999999,"15min":1.6970000000000001},"min":{"1min":0.82299999999999995,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":3.363,"5min":3.363,"15min":3.363},"last":1.587},{"interface":"front","average":{"1min":1.839,"5min":1.6970000000000001,"15min":1.722},"min":{"1min":0.93300000000000005,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":7.0439999999999996,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":2.0630000000000002}]}]},{"osd":0,"up_from":13,"seq":55834576017,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48720,"kb_used_data":3644,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323120,"statfs":{"total":96636764160,"available":96586874880,"internally_reserved":0,"allocated":3731456,"data_stored":2151010,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.94599999999999995,"15min":0.97299999999999998},"min":{"1min":0.57699999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.367,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":1.008},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.55500000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3560000000000001,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.70599999999999996}]},{"osd":2,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.143,"15min":1.1759999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6379999999999999,"5min":1.855,"15min":1.855},"last":1.518},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.167,"15min":1.2390000000000001},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.522,"5min":1.845,"15min":2.5779999999999998},"last":0.79100000000000004}]},{"osd":3,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.52100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.5429999999999999,"5min":1.804,"15min":2.0720000000000001},"last":1.202},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3300000000000001,"15min":1.391},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.77,"5min":1.883,"15min":2.194},"last":0.94399999999999995}]},{"osd":4,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.284,"15min":1.3169999999999999},"min":{"1min":0.73999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5549999999999999,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.161},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.492,"15min":1.502},"min":{"1min":0.999,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.984,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.448}]},{"osd":5,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.327,"15min":1.3680000000000001},"min":{"1min":0.80800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.51,"15min":1.569},"min":{"1min":0.78700000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.242,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.4870000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.6259999999999999,"15min":1.651},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.1709999999999998,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.5589999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.4830000000000001,"15min":1.508},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6899999999999999,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.4139999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.6040000000000001,"15min":1.6479999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0649999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.5880000000000001},{"interface":"front","average":{"1min":1.343,"5min":1.5589999999999999,"15min":1.6160000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.857,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.4730000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:18:40.716 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:18:40.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T18:18:40.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.990+0000 7ff15d727640 1 Processor -- start 2023-10-14T18:18:40.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.991+0000 7ff15d727640 1 -- start start 2023-10-14T18:18:40.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff1580cb7e0 0x7ff1580ad830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15815deb0 0x7ff158070240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580ae230 con 0x7ff15815deb0 2023-10-14T18:18:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1580ae510 con 0x7ff1580cb7e0 2023-10-14T18:18:40.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.992+0000 7ff15d727640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580ae7f0 con 0x7ff1580add70 2023-10-14T18:18:40.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15815deb0 0x7ff158070240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ff15815deb0 0x7ff158070240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51972/0 (socket says 172.21.15.29:51972) 2023-10-14T18:18:40.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 -- 172.21.15.29:0/3521968646 learned_addr learned my addr 172.21.15.29:0/3521968646 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:40.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff157fff640 1 --2- 172.21.15.29:0/3521968646 >> v2:172.21.15.186:3300/0 conn(0x7ff1580cb7e0 0x7ff1580ad830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff15cf26640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:40.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 -- 172.21.15.29:0/3521968646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 msgr2=0x7ff158070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:40.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158070780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:40.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 -- 172.21.15.29:0/3521968646 >> v2:172.21.15.186:3300/0 conn(0x7ff1580cb7e0 msgr2=0x7ff1580ad830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:40.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 --2- 172.21.15.29:0/3521968646 >> v2:172.21.15.186:3300/0 conn(0x7ff1580cb7e0 0x7ff1580ad830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff1577fe640 1 -- 172.21.15.29:0/3521968646 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1580aead0 con 0x7ff15815deb0 2023-10-14T18:18:41.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.993+0000 7ff15cf26640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:41.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.995+0000 7ff1577fe640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff158070240 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b23a0 tx=0x7ff14c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b6e94cb444a0e5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.995+0000 7ff156ffd640 1 -- 172.21.15.29:0/3521968646 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c031860 con 0x7ff15815deb0 2023-10-14T18:18:41.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.995+0000 7ff156ffd640 1 -- 172.21.15.29:0/3521968646 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff14c0319c0 con 0x7ff15815deb0 2023-10-14T18:18:41.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.995+0000 7ff156ffd640 1 -- 172.21.15.29:0/3521968646 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff14c031d50 con 0x7ff15815deb0 2023-10-14T18:18:41.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 -- 172.21.15.29:0/3521968646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 msgr2=0x7ff158070240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff158070240 secure :-1 s=READY pgs=2883 cs=0 l=1 rev1=1 crypto rx=0x7ff1580b23a0 tx=0x7ff14c02b900 comp rx=0 tx=0).stop 2023-10-14T18:18:41.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 -- 172.21.15.29:0/3521968646 shutdown_connections 2023-10-14T18:18:41.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158070780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 --2- 172.21.15.29:0/3521968646 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff158070240 unknown :-1 s=CLOSED pgs=2883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 --2- 172.21.15.29:0/3521968646 >> v2:172.21.15.186:3300/0 conn(0x7ff1580cb7e0 0x7ff1580ad830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 -- 172.21.15.29:0/3521968646 >> 172.21.15.29:0/3521968646 conn(0x7ff1580cc230 msgr2=0x7ff1580cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:41.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 -- 172.21.15.29:0/3521968646 shutdown_connections 2023-10-14T18:18:41.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.996+0000 7ff15d727640 1 -- 172.21.15.29:0/3521968646 wait complete. 2023-10-14T18:18:41.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.998+0000 7ff15d727640 1 Processor -- start 2023-10-14T18:18:41.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.998+0000 7ff15d727640 1 -- start start 2023-10-14T18:18:41.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158154590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 0x7ff1580c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580b0300 con 0x7ff15815deb0 2023-10-14T18:18:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff158154ad0 con 0x7ff1580cb7e0 2023-10-14T18:18:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15d727640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1580af6f0 con 0x7ff1580add70 2023-10-14T18:18:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15cf26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15cf26640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51986/0 (socket says 172.21.15.29:51986) 2023-10-14T18:18:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:40.999+0000 7ff15cf26640 1 -- 172.21.15.29:0/1115747198 learned_addr learned my addr 172.21.15.29:0/1115747198 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff1577fe640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 0x7ff1580c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff157fff640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158154590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 -- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 msgr2=0x7ff158154590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158154590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 -- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 msgr2=0x7ff1580c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 0x7ff1580c7430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 -- 172.21.15.29:0/1115747198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff14c0094d0 con 0x7ff15815deb0 2023-10-14T18:18:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff1577fe640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 0x7ff1580c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff157fff640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15cf26640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c7f0 tx=0x7ff15000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150004220 con 0x7ff15815deb0 2023-10-14T18:18:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1500043a0 con 0x7ff15815deb0 2023-10-14T18:18:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1580af9d0 con 0x7ff15815deb0 2023-10-14T18:18:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff150007860 con 0x7ff15815deb0 2023-10-14T18:18:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.000+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1580aff90 con 0x7ff15815deb0 2023-10-14T18:18:41.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.002+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1580b0470 con 0x7ff15815deb0 2023-10-14T18:18:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.003+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ff150007bd0 con 0x7ff15815deb0 2023-10-14T18:18:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.004+0000 7ff154ff9640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 0x7ff14406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.004+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7ff150089be0 con 0x7ff15815deb0 2023-10-14T18:18:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.004+0000 7ff157fff640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 0x7ff14406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.005+0000 7ff157fff640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 0x7ff14406d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff1581f5750 tx=0x7ff14800b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.008+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ff150054790 con 0x7ff15815deb0 2023-10-14T18:18:41.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.275+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7ff15804e9b0 con 0x7ff15815deb0 2023-10-14T18:18:41.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.279+0000 7ff154ff9640 1 -- 172.21.15.29:0/1115747198 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v52) v1 ==== 190+0+147 (secure 0 0 0) 0x7ff15001e020 con 0x7ff15815deb0 2023-10-14T18:18:41.282 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T18:18:41.282 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T18:18:41.282 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T18:18:41.283 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T18:18:41.283 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T18:18:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T18:18:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.282+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 msgr2=0x7ff14406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.282+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 0x7ff14406d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff1581f5750 tx=0x7ff14800b040 comp rx=0 tx=0).stop 2023-10-14T18:18:41.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.282+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 msgr2=0x7ff1580b0790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.282+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 secure :-1 s=READY pgs=2884 cs=0 l=1 rev1=1 crypto rx=0x7ff15000c7f0 tx=0x7ff15000ccb0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 shutdown_connections 2023-10-14T18:18:41.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7ff14406b0b0 0x7ff14406d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff15815deb0 0x7ff1580b0790 unknown :-1 s=CLOSED pgs=2884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff1580cb7e0 0x7ff1580c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 --2- 172.21.15.29:0/1115747198 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff1580add70 0x7ff158154590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 >> 172.21.15.29:0/1115747198 conn(0x7ff1580cc230 msgr2=0x7ff15815ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:41.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 shutdown_connections 2023-10-14T18:18:41.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.283+0000 7ff15d727640 1 -- 172.21.15.29:0/1115747198 wait complete. 2023-10-14T18:18:41.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:41.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.522+0000 7fd345e76640 1 Processor -- start 2023-10-14T18:18:41.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.522+0000 7fd345e76640 1 -- start start 2023-10-14T18:18:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.523+0000 7fd345e76640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3400b0b20 0x7fd3401615f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.523+0000 7fd345e76640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd34015d3f0 0x7fd34015d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.523+0000 7fd345e76640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd34015e3e0 con 0x7fd3400cd0c0 2023-10-14T18:18:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.523+0000 7fd345e76640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3400cd4a0 con 0x7fd34015d3f0 2023-10-14T18:18:41.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.523+0000 7fd345e76640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd34015e100 con 0x7fd3400b0b20 2023-10-14T18:18:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.524+0000 7fd33f7fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fd3400cd0c0 0x7fd3400b0410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:34242/0 (socket says 172.21.15.29:34242) 2023-10-14T18:18:41.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.524+0000 7fd33f7fe640 1 -- 172.21.15.29:0/2982706436 learned_addr learned my addr 172.21.15.29:0/2982706436 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:41.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.524+0000 7fd33ffff640 1 --2- 172.21.15.29:0/2982706436 >> v2:172.21.15.186:3300/0 conn(0x7fd34015d3f0 0x7fd34015d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.524+0000 7fd33effd640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3400b0b20 0x7fd3401615f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.525+0000 7fd33ffff640 1 -- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3400b0b20 msgr2=0x7fd3401615f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:41.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.525+0000 7fd33ffff640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3400b0b20 0x7fd3401615f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.525+0000 7fd33ffff640 1 -- 172.21.15.29:0/2982706436 >> v1:172.21.15.186:6789/0 conn(0x7fd3400cd0c0 legacy=0x7fd3400b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.525+0000 7fd33ffff640 1 -- 172.21.15.29:0/2982706436 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd340160b10 con 0x7fd34015d3f0 2023-10-14T18:18:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.527+0000 7fd33ffff640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd34015d3f0 0x7fd34015d860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd3380097a0 tx=0x7fd33802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c42c5b176b2226 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.527+0000 7fd33e7fc640 1 -- 172.21.15.29:0/2982706436 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33802e070 con 0x7fd34015d3f0 2023-10-14T18:18:41.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.527+0000 7fd33e7fc640 1 -- 172.21.15.29:0/2982706436 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3380315a0 con 0x7fd34015d3f0 2023-10-14T18:18:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.527+0000 7fd33e7fc640 1 -- 172.21.15.29:0/2982706436 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd338031930 con 0x7fd34015d3f0 2023-10-14T18:18:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 -- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd34015d3f0 msgr2=0x7fd34015d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd34015d3f0 0x7fd34015d860 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fd3380097a0 tx=0x7fd33802b840 comp rx=0 tx=0).stop 2023-10-14T18:18:41.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 -- 172.21.15.29:0/2982706436 shutdown_connections 2023-10-14T18:18:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd3400b0b20 0x7fd3401615f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 --2- 172.21.15.29:0/2982706436 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd34015d3f0 0x7fd34015d860 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 -- 172.21.15.29:0/2982706436 >> 172.21.15.29:0/2982706436 conn(0x7fd3400cb410 msgr2=0x7fd3400cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 -- 172.21.15.29:0/2982706436 shutdown_connections 2023-10-14T18:18:41.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.528+0000 7fd345e76640 1 -- 172.21.15.29:0/2982706436 wait complete. 2023-10-14T18:18:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.530+0000 7fd345e76640 1 Processor -- start 2023-10-14T18:18:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.530+0000 7fd345e76640 1 -- start start 2023-10-14T18:18:41.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3400b0b20 0x7fd3400c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd34015d3f0 0x7fd3400c7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400c4380 con 0x7fd3400cd0c0 2023-10-14T18:18:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400c3700 con 0x7fd3400b0b20 2023-10-14T18:18:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd345e76640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3400c39e0 con 0x7fd34015d3f0 2023-10-14T18:18:41.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd33effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd33effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52000/0 (socket says 172.21.15.29:52000) 2023-10-14T18:18:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.531+0000 7fd33effd640 1 -- 172.21.15.29:0/1116080299 learned_addr learned my addr 172.21.15.29:0/1116080299 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:41.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 -- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd34015d3f0 msgr2=0x7fd3400c7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd34015d3f0 0x7fd3400c7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33f7fe640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3400b0b20 0x7fd3400c2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 -- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3400b0b20 msgr2=0x7fd3400c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3400b0b20 0x7fd3400c2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 -- 172.21.15.29:0/1116080299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338009400 con 0x7fd3400cd0c0 2023-10-14T18:18:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd33effd640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fd33400e870 tx=0x7fd33400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.532+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3340094f0 con 0x7fd3400cd0c0 2023-10-14T18:18:41.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.533+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd33400c940 con 0x7fd3400cd0c0 2023-10-14T18:18:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.533+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3400c3cc0 con 0x7fd3400cd0c0 2023-10-14T18:18:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.533+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd33400ccf0 con 0x7fd3400cd0c0 2023-10-14T18:18:41.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.533+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3400c5fa0 con 0x7fd3400cd0c0 2023-10-14T18:18:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.534+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd34004e160 con 0x7fd3400cd0c0 2023-10-14T18:18:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.535+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fd334013470 con 0x7fd3400cd0c0 2023-10-14T18:18:41.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.536+0000 7fd327fff640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 0x7fd32c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.536+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fd334089b80 con 0x7fd3400cd0c0 2023-10-14T18:18:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.536+0000 7fd33f7fe640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 0x7fd32c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.537+0000 7fd33f7fe640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 0x7fd32c06d570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd330005c00 tx=0x7fd330009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:41.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.540+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd334054840 con 0x7fd3400cd0c0 2023-10-14T18:18:41.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.778+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd3400b4910 con 0x7fd3400cd0c0 2023-10-14T18:18:41.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.779+0000 7fd327fff640 1 -- 172.21.15.29:0/1116080299 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+4929 (secure 0 0 0) 0x7fd334057e60 con 0x7fd3400cd0c0 2023-10-14T18:18:41.780 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:41.780 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":123,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7637,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15}]} 2023-10-14T18:18:41.781 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 123 2023-10-14T18:18:41.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 msgr2=0x7fd32c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 0x7fd32c06d570 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fd330005c00 tx=0x7fd330009040 comp rx=0 tx=0).stop 2023-10-14T18:18:41.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 msgr2=0x7fd3400c5a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:41.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 secure :-1 s=READY pgs=2885 cs=0 l=1 rev1=1 crypto rx=0x7fd33400e870 tx=0x7fd33400ed30 comp rx=0 tx=0).stop 2023-10-14T18:18:41.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 shutdown_connections 2023-10-14T18:18:41.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fd32c06b0b0 0x7fd32c06d570 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd34015d3f0 0x7fd3400c7920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd3400cd0c0 0x7fd3400c5a60 unknown :-1 s=CLOSED pgs=2885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 --2- 172.21.15.29:0/1116080299 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd3400b0b20 0x7fd3400c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:41.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 >> 172.21.15.29:0/1116080299 conn(0x7fd3400cb410 msgr2=0x7fd34015cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:41.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 shutdown_connections 2023-10-14T18:18:41.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:41.782+0000 7fd345e76640 1 -- 172.21.15.29:0/1116080299 wait complete. 2023-10-14T18:18:42.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 123, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:18:35.103933+0000', 'modified': '2023-10-14T18:18:38.197589+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8767}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8767': {'gid': 8767, 'name': 'b', 'rank': 0, 'incarnation': 119, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.186:6833/2448419287', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6832', 'nonce': 2448419287}, {'type': 'v1', 'addr': '172.21.15.186:6833', 'nonce': 2448419287}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [31], 'metadata_pool': 30, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:18:42.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:18:42.207 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2023-10-14T18:18:42.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.445+0000 7fa45fb4e640 1 Processor -- start 2023-10-14T18:18:42.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.445+0000 7fa45fb4e640 1 -- start start 2023-10-14T18:18:42.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.446+0000 7fa45fb4e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa458159ab0 0x7fa4580ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.446+0000 7fa45fb4e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa458070360 con 0x7fa45806c140 2023-10-14T18:18:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45fb4e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4580cd1f0 con 0x7fa458159e90 2023-10-14T18:18:42.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45fb4e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45815d890 con 0x7fa458159ab0 2023-10-14T18:18:42.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45e0eb640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa45806c140 0x7fa45815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45010/0 (socket says 172.21.15.29:45010) 2023-10-14T18:18:42.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45e0eb640 1 -- 172.21.15.29:0/295395610 learned_addr learned my addr 172.21.15.29:0/295395610 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/295395610 >> v2:172.21.15.29:3300/0 conn(0x7fa458159ab0 0x7fa4580ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45c8e8640 1 -- 172.21.15.29:0/295395610 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443499759 0 0) 0x7fa458070360 con 0x7fa45806c140 2023-10-14T18:18:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.447+0000 7fa45c8e8640 1 -- 172.21.15.29:0/295395610 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa444003520 con 0x7fa45806c140 2023-10-14T18:18:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.448+0000 7fa45d8ea640 1 -- 172.21.15.29:0/295395610 >> v1:172.21.15.186:6789/0 conn(0x7fa458159e90 legacy=0x7fa45806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.448+0000 7fa45d8ea640 1 -- 172.21.15.29:0/295395610 >> v1:172.21.15.29:6789/0 conn(0x7fa45806c140 legacy=0x7fa45815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.448+0000 7fa45d8ea640 1 -- 172.21.15.29:0/295395610 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4580cd4d0 con 0x7fa458159ab0 2023-10-14T18:18:42.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.449+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/295395610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa458159ab0 0x7fa4580ce370 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fa454009d90 tx=0x7fa45402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8542ae292467a716 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.449+0000 7fa45c8e8640 1 -- 172.21.15.29:0/295395610 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45402cbd0 con 0x7fa458159ab0 2023-10-14T18:18:42.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.449+0000 7fa45c8e8640 1 -- 172.21.15.29:0/295395610 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45402cd30 con 0x7fa458159ab0 2023-10-14T18:18:42.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.449+0000 7fa45c8e8640 1 -- 172.21.15.29:0/295395610 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45402cea0 con 0x7fa458159ab0 2023-10-14T18:18:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 -- 172.21.15.29:0/295395610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa458159ab0 msgr2=0x7fa4580ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/295395610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa458159ab0 0x7fa4580ce370 secure :-1 s=READY pgs=2887 cs=0 l=1 rev1=1 crypto rx=0x7fa454009d90 tx=0x7fa45402be20 comp rx=0 tx=0).stop 2023-10-14T18:18:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 -- 172.21.15.29:0/295395610 shutdown_connections 2023-10-14T18:18:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/295395610 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa458159ab0 0x7fa4580ce370 unknown :-1 s=CLOSED pgs=2887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 -- 172.21.15.29:0/295395610 >> 172.21.15.29:0/295395610 conn(0x7fa4580ce9b0 msgr2=0x7fa4580cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 -- 172.21.15.29:0/295395610 shutdown_connections 2023-10-14T18:18:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.450+0000 7fa45fb4e640 1 -- 172.21.15.29:0/295395610 wait complete. 2023-10-14T18:18:42.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.452+0000 7fa45fb4e640 1 Processor -- start 2023-10-14T18:18:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.452+0000 7fa45fb4e640 1 -- start start 2023-10-14T18:18:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa458159ab0 0x7fa4580c3f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa458159e90 0x7fa4581e4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580c4e70 con 0x7fa45806c140 2023-10-14T18:18:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580c61a0 con 0x7fa458159ab0 2023-10-14T18:18:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45fb4e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4580c48b0 con 0x7fa458159e90 2023-10-14T18:18:42.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52016/0 (socket says 172.21.15.29:52016) 2023-10-14T18:18:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3200983014 learned_addr learned my addr 172.21.15.29:0/3200983014 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa458159ab0 0x7fa4580c3f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa458159e90 msgr2=0x7fa4581e4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa458159e90 0x7fa4581e4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa458159ab0 msgr2=0x7fa4580c3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa458159ab0 0x7fa4580c3f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.453+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3200983014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa454009a70 con 0x7fa45806c140 2023-10-14T18:18:42.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fa454009eb0 tx=0x7fa454004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa44e7fc640 1 -- 172.21.15.29:0/3200983014 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45402cea0 con 0x7fa45806c140 2023-10-14T18:18:42.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa44e7fc640 1 -- 172.21.15.29:0/3200983014 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa45403e610 con 0x7fa45806c140 2023-10-14T18:18:42.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa44e7fc640 1 -- 172.21.15.29:0/3200983014 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa45403e9c0 con 0x7fa45806c140 2023-10-14T18:18:42.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4580c4b90 con 0x7fa45806c140 2023-10-14T18:18:42.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.454+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581f3630 con 0x7fa45806c140 2023-10-14T18:18:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.457+0000 7fa44e7fc640 1 -- 172.21.15.29:0/3200983014 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa45403b050 con 0x7fa45806c140 2023-10-14T18:18:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.458+0000 7fa44e7fc640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 0x7fa43c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.458+0000 7fa44e7fc640 1 -- 172.21.15.29:0/3200983014 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa4540abd00 con 0x7fa45806c140 2023-10-14T18:18:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.459+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 0x7fa43c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.459+0000 7fa4627be740 1 Processor -- start 2023-10-14T18:18:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.459+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 0x7fa43c06d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4580c1b80 tx=0x7fa448009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.460+0000 7fa4627be740 1 -- start start 2023-10-14T18:18:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.460+0000 7fa4627be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.460+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e6ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.460+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e3f3c0 0x55a0f0e6f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa4627be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0d5f7a0 con 0x55a0f0e6bf80 2023-10-14T18:18:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0db1740 con 0x55a0f0e3f3c0 2023-10-14T18:18:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0daee30 con 0x55a0f0e6c360 2023-10-14T18:18:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52024/0 (socket says 172.21.15.29:52024) 2023-10-14T18:18:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3277998216 learned_addr learned my addr 172.21.15.29:0/3277998216 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 msgr2=0x55a0f0e6ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e6ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e3f3c0 msgr2=0x55a0f0e6f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e3f3c0 0x55a0f0e6f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3277998216 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0f0d6e910 con 0x55a0f0e6bf80 2023-10-14T18:18:42.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.461+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fa454076c10 tx=0x7fa45407a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0313bf0bfeabaa6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa43b7fe640 1 -- 172.21.15.29:0/3277998216 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454092070 con 0x55a0f0e6bf80 2023-10-14T18:18:42.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa43b7fe640 1 -- 172.21.15.29:0/3277998216 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454082a10 con 0x55a0f0e6bf80 2023-10-14T18:18:42.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa43b7fe640 1 -- 172.21.15.29:0/3277998216 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454079c00 con 0x55a0f0e6bf80 2023-10-14T18:18:42.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 -- 172.21.15.29:0/3277998216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 msgr2=0x55a0f0e3ee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 secure :-1 s=READY pgs=2889 cs=0 l=1 rev1=1 crypto rx=0x7fa454076c10 tx=0x7fa45407a1a0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 -- 172.21.15.29:0/3277998216 shutdown_connections 2023-10-14T18:18:42.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e3f3c0 0x55a0f0e6f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e6ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 --2- 172.21.15.29:0/3277998216 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e3ee80 unknown :-1 s=CLOSED pgs=2889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.462+0000 7fa4627be740 1 -- 172.21.15.29:0/3277998216 >> 172.21.15.29:0/3277998216 conn(0x55a0f0e41190 msgr2=0x55a0f0e89d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.463+0000 7fa4627be740 1 -- 172.21.15.29:0/3277998216 shutdown_connections 2023-10-14T18:18:42.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.463+0000 7fa4627be740 1 -- 172.21.15.29:0/3277998216 wait complete. 2023-10-14T18:18:42.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.463+0000 7fa4627be740 1 -- start start 2023-10-14T18:18:42.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.463+0000 7fa4627be740 1 Processor -- start 2023-10-14T18:18:42.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e52ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e535a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0daa720 con 0x55a0f0e59640 2023-10-14T18:18:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0d48970 con 0x55a0f0e6bf80 2023-10-14T18:18:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.464+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0d48610 con 0x55a0f0e6c360 2023-10-14T18:18:42.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa4627be740 10 client.0 _ll_register_callbacks cb 0x55a0f0d6a9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:18:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52030/0 (socket says 172.21.15.29:52030) 2023-10-14T18:18:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 -- 172.21.15.29:0/4007605790 learned_addr learned my addr 172.21.15.29:0/4007605790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e52ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 -- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 msgr2=0x55a0f0e535a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e535a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 -- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 msgr2=0x55a0f0e52ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e52ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45e0eb640 1 -- 172.21.15.29:0/4007605790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4581f3630 con 0x55a0f0e59640 2023-10-14T18:18:42.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.465+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e52ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:42.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fa450001d70 tx=0x7fa45000c4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73dacdd3f494ba5d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:18:42.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa4500070d0 con 0x55a0f0e59640 2023-10-14T18:18:42.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55a0f0dd1160 con 0x55a0f0e59640 2023-10-14T18:18:42.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa450015b50 con 0x55a0f0e59640 2023-10-14T18:18:42.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa450015d10 con 0x55a0f0e59640 2023-10-14T18:18:42.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa4627be740 10 client.12231 fetch_fsmap learned FSMap version 123 2023-10-14T18:18:42.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=2+}) v3 -- 0x55a0f0da8da0 con 0x55a0f0e59640 2023-10-14T18:18:42.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.466+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x55a0f0dd1160 con 0x55a0f0e59640 2023-10-14T18:18:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 123) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa450007230 con 0x55a0f0e59640 2023-10-14T18:18:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 10 client.12231 fetch_fsmap finished waiting for FSMap version 123 2023-10-14T18:18:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 10 client.12231 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-14T18:18:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 20 client.12231 populate_metadata read hostname 'smithi029' 2023-10-14T18:18:42.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0e66fd0 0x55a0f0e673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 4 client.12231 mds_command: new command op to 8767 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:18:42.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 --> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0f0d7ccc0 con 0x55a0f0e66fd0 2023-10-14T18:18:42.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.467+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0e66fd0 0x55a0f0e673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.468+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0e66fd0 0x55a0f0e673b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbbbfbf6f64a0ec0 server_cookie=cf130aeb1a1acdd3 in_seq=0 out_seq=0 2023-10-14T18:18:42.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.468+0000 7fa438ff9640 10 client.12231 ms_handle_connect on v2:172.21.15.186:6832/2448419287 2023-10-14T18:18:42.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa438ff9640 1 -- 172.21.15.29:0/4007605790 <== mds.0 v2:172.21.15.186:6832/2448419287 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x55a0f0d7ccc0 con 0x55a0f0e66fd0 2023-10-14T18:18:42.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa438ff9640 10 client.12231 handle_command_reply: tid=0 2023-10-14T18:18:42.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 client.12231 shutdown 2023-10-14T18:18:42.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 msgr2=0x55a0f0e6e210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 secure :-1 s=READY pgs=2890 cs=0 l=1 rev1=1 crypto rx=0x7fa450001d70 tx=0x7fa45000c4b0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 shutdown_connections 2023-10-14T18:18:42.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e6e210 unknown :-1 s=CLOSED pgs=2890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0e66fd0 0x55a0f0e673b0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e535a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 --2- 172.21.15.29:0/4007605790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e52ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa438ff9640 0 client.12231 ms_handle_reset on v2:172.21.15.186:6832/2448419287 2023-10-14T18:18:42.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 >> 172.21.15.29:0/4007605790 conn(0x55a0f0e3f3c0 msgr2=0x55a0f0e42d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 shutdown_connections 2023-10-14T18:18:42.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.471+0000 7fa4627be740 1 -- 172.21.15.29:0/4007605790 wait complete. 2023-10-14T18:18:42.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.472+0000 7fa4627be740 20 client.12231 trim_cache size 0 max 16384 2023-10-14T18:18:42.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.488+0000 7fa4627be740 1 Processor -- start 2023-10-14T18:18:42.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.488+0000 7fa4627be740 1 -- start start 2023-10-14T18:18:42.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 0x55a0f0e66480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 0x55a0f0e42f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0b0ef20 con 0x55a0f0e59640 2023-10-14T18:18:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0e3e480 con 0x55a0f0e6c360 2023-10-14T18:18:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.489+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0e3e120 con 0x55a0f0e6bf80 2023-10-14T18:18:42.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52036/0 (socket says 172.21.15.29:52036) 2023-10-14T18:18:42.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3495195970 learned_addr learned my addr 172.21.15.29:0/3495195970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 0x55a0f0e66480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 0x55a0f0e42f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 msgr2=0x55a0f0e66480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 0x55a0f0e66480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 msgr2=0x55a0f0e42f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 0x55a0f0e42f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d8ea640 1 -- 172.21.15.29:0/3495195970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0f0d47b80 con 0x55a0f0e59640 2023-10-14T18:18:42.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 0x55a0f0e66480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:42.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.490+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 0x55a0f0e42f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fa454076c10 tx=0x7fa45407d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c7b7c772c63e262 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa43bfff640 1 -- 172.21.15.29:0/3495195970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454079c00 con 0x55a0f0e59640 2023-10-14T18:18:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa43bfff640 1 -- 172.21.15.29:0/3495195970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454079d60 con 0x55a0f0e59640 2023-10-14T18:18:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa43bfff640 1 -- 172.21.15.29:0/3495195970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454079ed0 con 0x55a0f0e59640 2023-10-14T18:18:42.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa4627be740 1 -- 172.21.15.29:0/3495195970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 msgr2=0x55a0f0e8cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.491+0000 7fa4627be740 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 secure :-1 s=READY pgs=2891 cs=0 l=1 rev1=1 crypto rx=0x7fa454076c10 tx=0x7fa45407d000 comp rx=0 tx=0).stop 2023-10-14T18:18:42.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 -- 172.21.15.29:0/3495195970 shutdown_connections 2023-10-14T18:18:42.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e6c360 0x55a0f0e42f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6bf80 0x55a0f0e66480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 --2- 172.21.15.29:0/3495195970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e59640 0x55a0f0e8cec0 unknown :-1 s=CLOSED pgs=2891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 -- 172.21.15.29:0/3495195970 >> 172.21.15.29:0/3495195970 conn(0x55a0f0e3f3c0 msgr2=0x55a0f0e65d70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 -- 172.21.15.29:0/3495195970 shutdown_connections 2023-10-14T18:18:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 -- 172.21.15.29:0/3495195970 wait complete. 2023-10-14T18:18:42.498 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.492+0000 7fa4627be740 1 -- start start 2023-10-14T18:18:42.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.493+0000 7fa4627be740 1 Processor -- start 2023-10-14T18:18:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 0x55a0f0ed0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0e8d280 con 0x55a0f0e6bf80 2023-10-14T18:18:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0ee94e0 con 0x55a0f0e59640 2023-10-14T18:18:42.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x55a0f0e8a0d0 con 0x55a0f0e6c360 2023-10-14T18:18:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa4627be740 10 client.0 _ll_register_callbacks cb 0x7fa450007bb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:18:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52038/0 (socket says 172.21.15.29:52038) 2023-10-14T18:18:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 -- 172.21.15.29:0/1530193462 learned_addr learned my addr 172.21.15.29:0/1530193462 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45e0eb640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 0x55a0f0ed0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 -- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 msgr2=0x55a0f0e723b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e723b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 -- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 msgr2=0x55a0f0ed0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.505 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 0x55a0f0ed0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.494+0000 7fa45d0e9640 1 -- 172.21.15.29:0/1530193462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0f0d6e910 con 0x55a0f0e6bf80 2023-10-14T18:18:42.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 0x55a0f0ed0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:42.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa45d0e9640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7fa448001e00 tx=0x7fa448000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee233988dca38302 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa438ff9640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:18:42.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44800dbd0 con 0x55a0f0e6bf80 2023-10-14T18:18:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa44800dd30 con 0x55a0f0e6bf80 2023-10-14T18:18:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa454082a10 con 0x55a0f0e6bf80 2023-10-14T18:18:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=123) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa448021400 con 0x55a0f0e6bf80 2023-10-14T18:18:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa4627be740 10 client.12237 fetch_fsmap learned FSMap version 123 2023-10-14T18:18:42.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.495+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=123,monmap=2+}) v3 -- 0x55a0f0d6d170 con 0x55a0f0e6bf80 2023-10-14T18:18:42.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454082a10 con 0x55a0f0e6bf80 2023-10-14T18:18:42.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 123) v1 ==== 3294+0+0 (secure 0 0 0) 0x7fa448021560 con 0x55a0f0e6bf80 2023-10-14T18:18:42.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 10 client.12237 fetch_fsmap finished waiting for FSMap version 123 2023-10-14T18:18:42.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 10 client.12237 resolve_mds: resolved name 'b' to daemon mds.b 2023-10-14T18:18:42.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 20 client.12237 populate_metadata read hostname 'smithi029' 2023-10-14T18:18:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0ee89d0 0x55a0f0ee8db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 4 client.12237 mds_command: new command op to 8767 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:18:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.496+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 --> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a0f0da3700 con 0x55a0f0ee89d0 2023-10-14T18:18:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.497+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0ee89d0 0x55a0f0ee8db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa45d8ea640 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0ee89d0 0x55a0f0ee8db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14245fb6ef7ea636 server_cookie=84f27f2e8557e5 in_seq=0 out_seq=0 2023-10-14T18:18:42.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa438ff9640 10 client.12237 ms_handle_connect on v2:172.21.15.186:6832/2448419287 2023-10-14T18:18:42.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa438ff9640 1 -- 172.21.15.29:0/1530193462 <== mds.0 v2:172.21.15.186:6832/2448419287 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+328 (crc 0 0 0) 0x55a0f0da3700 con 0x55a0f0ee89d0 2023-10-14T18:18:42.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa438ff9640 10 client.12237 handle_command_reply: tid=0 2023-10-14T18:18:42.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 client.12237 shutdown 2023-10-14T18:18:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 msgr2=0x55a0f0e896f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 secure :-1 s=READY pgs=2892 cs=0 l=1 rev1=1 crypto rx=0x7fa448001e00 tx=0x7fa448000ed0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 shutdown_connections 2023-10-14T18:18:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:6832/2448419287,v1:172.21.15.186:6833/2448419287] conn(0x55a0f0ee89d0 0x55a0f0ee8db0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x55a0f0e6c360 0x55a0f0e723b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x55a0f0e6bf80 0x55a0f0e896f0 unknown :-1 s=CLOSED pgs=2892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 --2- 172.21.15.29:0/1530193462 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x55a0f0e59640 0x55a0f0ed0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa438ff9640 0 client.12237 ms_handle_reset on v2:172.21.15.186:6832/2448419287 2023-10-14T18:18:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.498+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 >> 172.21.15.29:0/1530193462 conn(0x55a0f0e3f3c0 msgr2=0x55a0f0e42ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 shutdown_connections 2023-10-14T18:18:42.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa4627be740 1 -- 172.21.15.29:0/1530193462 wait complete. 2023-10-14T18:18:42.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa4627be740 20 client.12237 trim_cache size 0 max 16384 2023-10-14T18:18:42.513 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:18:42.514 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:18:42.514 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:18:42.514 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 8767, 2023-10-14T18:18:42.514 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:18:42.514 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:18:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T18:18:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8293658900000001, 2023-10-14T18:18:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 123, 2023-10-14T18:18:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 250, 2023-10-14T18:18:42.515 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 250, 2023-10-14T18:18:42.516 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.197354354 2023-10-14T18:18:42.516 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:18:42.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 msgr2=0x7fa43c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 0x7fa43c06d570 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4580c1b80 tx=0x7fa448009040 comp rx=0 tx=0).stop 2023-10-14T18:18:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 msgr2=0x7fa458157e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 secure :-1 s=READY pgs=2888 cs=0 l=1 rev1=1 crypto rx=0x7fa454009eb0 tx=0x7fa454004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 shutdown_connections 2023-10-14T18:18:42.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fa43c06b0b0 0x7fa43c06d570 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa458159e90 0x7fa4581e4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa458159ab0 0x7fa4580c3f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 --2- 172.21.15.29:0/3200983014 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45806c140 0x7fa458157e50 unknown :-1 s=CLOSED pgs=2888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.499+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 >> 172.21.15.29:0/3200983014 conn(0x7fa4580ce9b0 msgr2=0x7fa45806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.500+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 shutdown_connections 2023-10-14T18:18:42.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.500+0000 7fa45fb4e640 1 -- 172.21.15.29:0/3200983014 wait complete. 2023-10-14T18:18:42.520 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T18:18:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.746+0000 7fceda0d1640 1 Processor -- start 2023-10-14T18:18:42.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.746+0000 7fceda0d1640 1 -- start start 2023-10-14T18:18:42.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fceda0d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fced40ce2d0 0x7fced406bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fceda0d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced415f2f0 0x7fced40712e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fceda0d1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fced415ee60 con 0x7fced40b4b70 2023-10-14T18:18:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fceda0d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fced40ce6b0 con 0x7fced40ce2d0 2023-10-14T18:18:42.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fceda0d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced415e600 con 0x7fced415f2f0 2023-10-14T18:18:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fced90cf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fced40ce2d0 0x7fced406bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fced98d0640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fced40b4b70 0x7fced406fa90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45016/0 (socket says 172.21.15.29:45016) 2023-10-14T18:18:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fced98d0640 1 -- 172.21.15.29:0/1058566773 learned_addr learned my addr 172.21.15.29:0/1058566773 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.748+0000 7fced88ce640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced415f2f0 0x7fced40712e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.749+0000 7fced90cf640 1 -- 172.21.15.29:0/1058566773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced415f2f0 msgr2=0x7fced40712e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.749+0000 7fced90cf640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced415f2f0 0x7fced40712e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.749+0000 7fced90cf640 1 -- 172.21.15.29:0/1058566773 >> v1:172.21.15.29:6789/0 conn(0x7fced40b4b70 legacy=0x7fced406fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.749+0000 7fced90cf640 1 -- 172.21.15.29:0/1058566773 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced406e160 con 0x7fced40ce2d0 2023-10-14T18:18:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.749+0000 7fcecbfff640 1 -- 172.21.15.29:0/1058566773 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782927414 0 0) 0x7fced415ee60 con 0x7fced40b4b70 2023-10-14T18:18:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.750+0000 7fced90cf640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40ce2d0 0x7fced406bd90 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fced00097a0 tx=0x7fced002b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feaf311c44c99f1e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.750+0000 7fcecbfff640 1 -- 172.21.15.29:0/1058566773 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced002d040 con 0x7fced40ce2d0 2023-10-14T18:18:42.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.750+0000 7fcecbfff640 1 -- 172.21.15.29:0/1058566773 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced0031810 con 0x7fced40ce2d0 2023-10-14T18:18:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.750+0000 7fcecbfff640 1 -- 172.21.15.29:0/1058566773 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced0031ba0 con 0x7fced40ce2d0 2023-10-14T18:18:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 -- 172.21.15.29:0/1058566773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40ce2d0 msgr2=0x7fced406bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40ce2d0 0x7fced406bd90 secure :-1 s=READY pgs=2894 cs=0 l=1 rev1=1 crypto rx=0x7fced00097a0 tx=0x7fced002b8b0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 -- 172.21.15.29:0/1058566773 shutdown_connections 2023-10-14T18:18:42.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced415f2f0 0x7fced40712e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1058566773 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40ce2d0 0x7fced406bd90 unknown :-1 s=CLOSED pgs=2894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 -- 172.21.15.29:0/1058566773 >> 172.21.15.29:0/1058566773 conn(0x7fced4069da0 msgr2=0x7fced406a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 -- 172.21.15.29:0/1058566773 shutdown_connections 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.751+0000 7fceda0d1640 1 -- 172.21.15.29:0/1058566773 wait complete. 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.753+0000 7fceda0d1640 1 Processor -- start 2023-10-14T18:18:42.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.753+0000 7fceda0d1640 1 -- start start 2023-10-14T18:18:42.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced40ce2d0 0x7fced414cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced415f2f0 0x7fced414d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced40ad400 con 0x7fced40b4b70 2023-10-14T18:18:42.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced414d920 con 0x7fced415f2f0 2023-10-14T18:18:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fceda0d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fced414dc00 con 0x7fced40ce2d0 2023-10-14T18:18:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fced90cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fced90cf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52052/0 (socket says 172.21.15.29:52052) 2023-10-14T18:18:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.754+0000 7fced90cf640 1 -- 172.21.15.29:0/1717117173 learned_addr learned my addr 172.21.15.29:0/1717117173 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:42.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 -- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced40ce2d0 msgr2=0x7fced414cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced40ce2d0 0x7fced414cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 -- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced415f2f0 msgr2=0x7fced414d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced415f2f0 0x7fced414d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 -- 172.21.15.29:0/1717117173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fced0009400 con 0x7fced40b4b70 2023-10-14T18:18:42.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fced90cf640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fced00098c0 tx=0x7fced00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced003ad60 con 0x7fced40b4b70 2023-10-14T18:18:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fced00443f0 con 0x7fced40b4b70 2023-10-14T18:18:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fced00447a0 con 0x7fced40b4b70 2023-10-14T18:18:42.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.755+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced41f3320 con 0x7fced40b4b70 2023-10-14T18:18:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.756+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced41f38e0 con 0x7fced40b4b70 2023-10-14T18:18:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.757+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced404e9b0 con 0x7fced40b4b70 2023-10-14T18:18:42.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.758+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fced004d450 con 0x7fced40b4b70 2023-10-14T18:18:42.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.759+0000 7fcec9ffb640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 0x7fceb806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:42.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.759+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fced00abe70 con 0x7fced40b4b70 2023-10-14T18:18:42.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.759+0000 7fced88ce640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 0x7fceb806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:42.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.760+0000 7fced88ce640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 0x7fceb806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcec4006fd0 tx=0x7fcec4008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:42.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:42.764+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fced0076ac0 con 0x7fced40b4b70 2023-10-14T18:18:43.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.003+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fced406d260 con 0x7fced40b4b70 2023-10-14T18:18:43.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.004+0000 7fcec9ffb640 1 -- 172.21.15.29:0/1717117173 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v123) v1 ==== 53+0+73 (secure 0 0 0) 0x7fced0035020 con 0x7fced40b4b70 2023-10-14T18:18:43.005 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T18:18:43.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 msgr2=0x7fceb806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:43.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 0x7fceb806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fcec4006fd0 tx=0x7fcec4008040 comp rx=0 tx=0).stop 2023-10-14T18:18:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 msgr2=0x7fced40afe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 secure :-1 s=READY pgs=2895 cs=0 l=1 rev1=1 crypto rx=0x7fced00098c0 tx=0x7fced00046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 shutdown_connections 2023-10-14T18:18:43.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fceb806b0b0 0x7fceb806d570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fced415f2f0 0x7fced414d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fced40ce2d0 0x7fced414cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 --2- 172.21.15.29:0/1717117173 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fced40b4b70 0x7fced40afe00 unknown :-1 s=CLOSED pgs=2895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.007+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 >> 172.21.15.29:0/1717117173 conn(0x7fced4069da0 msgr2=0x7fced40713c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:43.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.008+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 shutdown_connections 2023-10-14T18:18:43.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:43.008+0000 7fceda0d1640 1 -- 172.21.15.29:0/1717117173 wait complete. 2023-10-14T18:18:43.030 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T18:18:43.030 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T18:18:43.031 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T18:18:43.031 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T18:18:43.031 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T18:18:43.032 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T18:18:43.032 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T18:18:43.032 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T18:18:43.033 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:18:43.033 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.034 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T18:18:43.092 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T18:18:43.092 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T18:18:43.092 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T18:18:43.093 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.093 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T18:18:43.093 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.093 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:18:43.093 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:18:43.094 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T18:18:43.094 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:18:43.094 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:18:43.094 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T18:18:43.095 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T18:18:43.095 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T18:18:43.095 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T18:18:43.095 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T18:18:43.095 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36453sec preferred_lft 36453sec 2023-10-14T18:18:43.096 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T18:18:43.096 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.096 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:18:43.096 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:18:43.096 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T18:18:43.097 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T18:18:43.097 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:18:43.097 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T18:18:43.097 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.097 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T18:18:43.098 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.098 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:18:43.098 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T18:18:43.098 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T18:18:43.098 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.099 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:18:43.099 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T18:18:43.099 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T18:18:43.099 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:18:43.100 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.101 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T18:18:43.202 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T18:18:43.206 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.207 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T18:18:43.267 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T18:18:43.268 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T18:18:43.268 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T18:18:43.269 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T18:18:43.269 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T18:18:43.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.270 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T18:18:43.271 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T18:18:43.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T18:18:43.272 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T18:18:43.273 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T18:18:43.274 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T18:18:43.469 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.469 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:18:43.526 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T18:18:43.527 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T18:18:43.527 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:18:43.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.528 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T18:18:43.632 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:18:43.633 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T18:18:43.696 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:18:43.696 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:18:43.697 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:18:43.752 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T18:18:43.752 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.753 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:18:43.904 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:18:43.904 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:18:43.904 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:18:43.927 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:18:43.928 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:18:43.928 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:18:43.928 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:18:43.928 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:18:43.957 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:18:43.957 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:18:43.957 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:18:43.958 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:18:43.958 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:18:43.958 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T18:18:43.958 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T18:18:43.958 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T18:18:43.960 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T18:18:43.960 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T18:18:43.960 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T18:18:43.961 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T18:18:43.961 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T18:18:43.966 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T18:18:43.967 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:43.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:18:44.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:44.051 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T18:18:44.131 INFO:teuthology.orchestra.run.smithi029.stdout:client7709 2023-10-14T18:18:44.131 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:18:44.132 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:18:44.194 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs flag set enable_multiple true --yes-i-really-mean-it 2023-10-14T18:18:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.429+0000 7f55db59e640 1 Processor -- start 2023-10-14T18:18:44.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.429+0000 7f55db59e640 1 -- start start 2023-10-14T18:18:44.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55db59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f55dc0cae40 0x7f55dc0c9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55db59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0c8e70 con 0x7f55dc06d280 2023-10-14T18:18:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55db59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55dc0b4170 con 0x7f55dc15ad80 2023-10-14T18:18:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55db59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55dc0adde0 con 0x7f55dc0cae40 2023-10-14T18:18:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55d3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f55dc06d280 0x7f55dc0c9300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45026/0 (socket says 172.21.15.29:45026) 2023-10-14T18:18:44.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.431+0000 7f55d3fff640 1 -- 172.21.15.29:0/1849358197 learned_addr learned my addr 172.21.15.29:0/1849358197 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55dad9d640 1 --2- 172.21.15.29:0/1849358197 >> v2:172.21.15.186:3300/0 conn(0x7f55dc0cae40 0x7f55dc0c9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1439233595 0 0) 0x7f55dc0c8e70 con 0x7f55dc06d280 2023-10-14T18:18:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55c8003520 con 0x7f55dc06d280 2023-10-14T18:18:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3253321222 0 0) 0x7f55c8003520 con 0x7f55dc06d280 2023-10-14T18:18:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 >> v2:172.21.15.186:3300/0 conn(0x7f55dc0cae40 msgr2=0x7f55dc0c9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 --2- 172.21.15.29:0/1849358197 >> v2:172.21.15.186:3300/0 conn(0x7f55dc0cae40 0x7f55dc0c9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 >> v1:172.21.15.186:6789/0 conn(0x7f55dc15ad80 legacy=0x7f55dc06cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0b4410 con 0x7f55dc06d280 2023-10-14T18:18:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f55c4001670 con 0x7f55dc06d280 2023-10-14T18:18:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 >> v1:172.21.15.29:6789/0 conn(0x7f55dc06d280 legacy=0x7f55dc0c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55c8003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55c80051d0 0x7f55c800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0adde0 con 0x7f55c8003a30 2023-10-14T18:18:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55c800c100 con 0x7f55c80047c0 2023-10-14T18:18:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.432+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55c800c2c0 con 0x7f55c80051d0 2023-10-14T18:18:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55c8003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55d3fff640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 -- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55c80051d0 msgr2=0x7f55c800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55c80051d0 0x7f55c800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 -- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55c80047c0 msgr2=0x7f55c8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55da59c640 1 -- 172.21.15.29:0/1849358197 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55c8003520 con 0x7f55c8003a30 2023-10-14T18:18:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55d3fff640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55c80047c0 0x7f55c8004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.433+0000 7f55dad9d640 1 -- 172.21.15.29:0/1849358197 reap_dead start 2023-10-14T18:18:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.434+0000 7f55da59c640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55c8003e50 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f55cc001750 tx=0x7f55c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ae9c50dc76b386f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0052d0 con 0x7f55c8003a30 2023-10-14T18:18:44.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc02e050 con 0x7f55c8003a30 2023-10-14T18:18:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55d9d9b640 1 -- 172.21.15.29:0/1849358197 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc003830 con 0x7f55c8003a30 2023-10-14T18:18:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 -- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 msgr2=0x7f55c8003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55c8003e50 secure :-1 s=READY pgs=2897 cs=0 l=1 rev1=1 crypto rx=0x7f55cc001750 tx=0x7f55c80057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 -- 172.21.15.29:0/1849358197 shutdown_connections 2023-10-14T18:18:44.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 --2- 172.21.15.29:0/1849358197 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55c8003e50 unknown :-1 s=CLOSED pgs=2897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 -- 172.21.15.29:0/1849358197 >> 172.21.15.29:0/1849358197 conn(0x7f55dc0ce820 msgr2=0x7f55dc0cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 -- 172.21.15.29:0/1849358197 shutdown_connections 2023-10-14T18:18:44.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.435+0000 7f55db59e640 1 -- 172.21.15.29:0/1849358197 wait complete. 2023-10-14T18:18:44.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.437+0000 7f55db59e640 1 Processor -- start 2023-10-14T18:18:44.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.437+0000 7f55db59e640 1 -- start start 2023-10-14T18:18:44.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 0x7f55dc150e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55dc0b2790 0x7f55dc0b2c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0b3150 con 0x7f55c8003a30 2023-10-14T18:18:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0c2a80 con 0x7f55dc0c2310 2023-10-14T18:18:44.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.438+0000 7f55db59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55dc0c2d60 con 0x7f55dc0b2790 2023-10-14T18:18:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52068/0 (socket says 172.21.15.29:52068) 2023-10-14T18:18:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 -- 172.21.15.29:0/67419753 learned_addr learned my addr 172.21.15.29:0/67419753 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55d3fff640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 0x7f55dc150e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55dad9d640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55dc0b2790 0x7f55dc0b2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 -- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55dc0b2790 msgr2=0x7f55dc0b2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55dc0b2790 0x7f55dc0b2c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 -- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 msgr2=0x7f55dc150e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 0x7f55dc150e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55da59c640 1 -- 172.21.15.29:0/67419753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55dc0c3040 con 0x7f55c8003a30 2023-10-14T18:18:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.439+0000 7f55d3fff640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 0x7f55dc150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55da59c640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f55cc02bc30 tx=0x7f55cc001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc0076c0 con 0x7f55c8003a30 2023-10-14T18:18:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55cc007a30 con 0x7f55c8003a30 2023-10-14T18:18:44.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f55cc006be0 con 0x7f55c8003a30 2023-10-14T18:18:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55dc0c3320 con 0x7f55c8003a30 2023-10-14T18:18:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.440+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55dc1513b0 con 0x7f55c8003a30 2023-10-14T18:18:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.441+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55dc0c5bd0 con 0x7f55c8003a30 2023-10-14T18:18:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.442+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f55cc006d40 con 0x7f55c8003a30 2023-10-14T18:18:44.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.443+0000 7f55d2ffd640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 0x7f55bc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.443+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f55cc0abbc0 con 0x7f55c8003a30 2023-10-14T18:18:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.444+0000 7f55d3fff640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 0x7f55bc06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.444+0000 7f55d3fff640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 0x7f55bc06d5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55c40017a0 tx=0x7f55c4009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.448+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f55cc0767b0 con 0x7f55c8003a30 2023-10-14T18:18:44.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.682+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f55dc04e9b0 con 0x7f55c8003a30 2023-10-14T18:18:44.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.722+0000 7f55d2ffd640 1 -- 172.21.15.29:0/67419753 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs flag set", "flag_name": "enable_multiple", "val": "true", "yes_i_really_mean_it": true}]=0 v124) v1 ==== 136+0+0 (secure 0 0 0) 0x7f55cc079dd0 con 0x7f55c8003a30 2023-10-14T18:18:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.725+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 msgr2=0x7f55bc06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.725+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 0x7f55bc06d5a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f55c40017a0 tx=0x7f55c4009040 comp rx=0 tx=0).stop 2023-10-14T18:18:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.725+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 msgr2=0x7f55dc1eeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.725+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 secure :-1 s=READY pgs=2898 cs=0 l=1 rev1=1 crypto rx=0x7f55cc02bc30 tx=0x7f55cc001120 comp rx=0 tx=0).stop 2023-10-14T18:18:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.725+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 shutdown_connections 2023-10-14T18:18:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f55bc06b0e0 0x7f55bc06d5a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f55dc0b2790 0x7f55dc0b2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f55dc0c2310 0x7f55dc150e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 --2- 172.21.15.29:0/67419753 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f55c8003a30 0x7f55dc1eeb80 unknown :-1 s=CLOSED pgs=2898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 >> 172.21.15.29:0/67419753 conn(0x7f55dc0ce820 msgr2=0x7f55dc0cec00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 shutdown_connections 2023-10-14T18:18:44.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.726+0000 7f55db59e640 1 -- 172.21.15.29:0/67419753 wait complete. 2023-10-14T18:18:44.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:44.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.972+0000 7fae8562d640 1 Processor -- start 2023-10-14T18:18:44.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.973+0000 7fae8562d640 1 -- start start 2023-10-14T18:18:44.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae8562d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae8015ae80 0x7fae8015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae8562d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae8015ed60 con 0x7fae800ce390 2023-10-14T18:18:44.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae8562d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae800707e0 con 0x7fae8015aaa0 2023-10-14T18:18:44.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae8562d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae8006b810 con 0x7fae8015ae80 2023-10-14T18:18:44.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae77fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae8015ae80 0x7fae8015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae7f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae800ce390 0x7fae80070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45038/0 (socket says 172.21.15.29:45038) 2023-10-14T18:18:44.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.975+0000 7fae7f7fe640 1 -- 172.21.15.29:0/2577190524 learned_addr learned my addr 172.21.15.29:0/2577190524 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.976+0000 7fae77fff640 1 -- 172.21.15.29:0/2577190524 >> v1:172.21.15.186:6789/0 conn(0x7fae8015aaa0 legacy=0x7fae800cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.976+0000 7fae77fff640 1 -- 172.21.15.29:0/2577190524 >> v1:172.21.15.29:6789/0 conn(0x7fae800ce390 legacy=0x7fae80070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.976+0000 7fae77fff640 1 -- 172.21.15.29:0/2577190524 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae8015ef90 con 0x7fae8015ae80 2023-10-14T18:18:44.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.977+0000 7fae77fff640 1 --2- 172.21.15.29:0/2577190524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015ae80 0x7fae8015e820 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fae680097a0 tx=0x7fae6802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f48c82721761d7ef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.977+0000 7fae7e7fc640 1 -- 172.21.15.29:0/2577190524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68031830 con 0x7fae8015ae80 2023-10-14T18:18:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.977+0000 7fae7e7fc640 1 -- 172.21.15.29:0/2577190524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae68031990 con 0x7fae8015ae80 2023-10-14T18:18:44.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.977+0000 7fae7e7fc640 1 -- 172.21.15.29:0/2577190524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae68031d20 con 0x7fae8015ae80 2023-10-14T18:18:44.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 -- 172.21.15.29:0/2577190524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015ae80 msgr2=0x7fae8015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 --2- 172.21.15.29:0/2577190524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015ae80 0x7fae8015e820 secure :-1 s=READY pgs=2900 cs=0 l=1 rev1=1 crypto rx=0x7fae680097a0 tx=0x7fae6802b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 -- 172.21.15.29:0/2577190524 shutdown_connections 2023-10-14T18:18:44.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 --2- 172.21.15.29:0/2577190524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015ae80 0x7fae8015e820 unknown :-1 s=CLOSED pgs=2900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 -- 172.21.15.29:0/2577190524 >> 172.21.15.29:0/2577190524 conn(0x7fae800cb410 msgr2=0x7fae800cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:44.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 -- 172.21.15.29:0/2577190524 shutdown_connections 2023-10-14T18:18:44.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.978+0000 7fae8562d640 1 -- 172.21.15.29:0/2577190524 wait complete. 2023-10-14T18:18:44.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.980+0000 7fae8562d640 1 Processor -- start 2023-10-14T18:18:44.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.980+0000 7fae8562d640 1 -- start start 2023-10-14T18:18:44.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae800ce390 0x7fae80157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 0x7fae801e4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae801e6b00 con 0x7fae8015aaa0 2023-10-14T18:18:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae801e6de0 con 0x7fae8015ae80 2023-10-14T18:18:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae8562d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae801e70c0 con 0x7fae800ce390 2023-10-14T18:18:44.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae77fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52080/0 (socket says 172.21.15.29:52080) 2023-10-14T18:18:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.981+0000 7fae77fff640 1 -- 172.21.15.29:0/1431366578 learned_addr learned my addr 172.21.15.29:0/1431366578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae7f7fe640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 0x7fae801e4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 -- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae800ce390 msgr2=0x7fae80157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae800ce390 0x7fae80157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 -- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 msgr2=0x7fae801e4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:44.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 0x7fae801e4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:44.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 -- 172.21.15.29:0/1431366578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae68009400 con 0x7fae8015aaa0 2023-10-14T18:18:44.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae7f7fe640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 0x7fae801e4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:44.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.982+0000 7fae77fff640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fae68000c00 tx=0x7fae680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.983+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae6803aa60 con 0x7fae8015aaa0 2023-10-14T18:18:44.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.983+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae6803abc0 con 0x7fae8015aaa0 2023-10-14T18:18:44.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.983+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae680443f0 con 0x7fae8015aaa0 2023-10-14T18:18:44.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.983+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae801e67d0 con 0x7fae8015aaa0 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.983+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae80157c60 con 0x7fae8015aaa0 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.984+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae80157e50 con 0x7fae8015aaa0 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.985+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fae6804d450 con 0x7fae8015aaa0 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.986+0000 7fae777fe640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 0x7fae6006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.986+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae680abd80 con 0x7fae8015aaa0 2023-10-14T18:18:44.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.986+0000 7fae7effd640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 0x7fae6006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:44.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.987+0000 7fae7effd640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 0x7fae6006d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae70003cc0 tx=0x7fae700079b0 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:44.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:44.990+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae68076930 con 0x7fae8015aaa0 2023-10-14T18:18:45.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.189+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae800c2ff0 con 0x7fae8015aaa0 2023-10-14T18:18:45.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.192+0000 7fae777fe640 1 -- 172.21.15.29:0/1431366578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v250) v1 ==== 74+0+16693 (secure 0 0 0) 0x7fae800c2ff0 con 0x7fae8015aaa0 2023-10-14T18:18:45.193 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:45.194 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":250,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:36.639061+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":31,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":248,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:45.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 msgr2=0x7fae6006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 0x7fae6006d570 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fae70003cc0 tx=0x7fae700079b0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 msgr2=0x7fae801e4780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 secure :-1 s=READY pgs=2901 cs=0 l=1 rev1=1 crypto rx=0x7fae68000c00 tx=0x7fae680046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 shutdown_connections 2023-10-14T18:18:45.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fae6006b0b0 0x7fae6006d570 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae8015ae80 0x7fae801e4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae8015aaa0 0x7fae801e4780 unknown :-1 s=CLOSED pgs=2901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 --2- 172.21.15.29:0/1431366578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae800ce390 0x7fae80157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 >> 172.21.15.29:0/1431366578 conn(0x7fae800cb410 msgr2=0x7fae8015a020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:45.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 shutdown_connections 2023-10-14T18:18:45.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.195+0000 7fae8562d640 1 -- 172.21.15.29:0/1431366578 wait complete. 2023-10-14T18:18:45.217 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:18:33.087026+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:18:34.140901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:18:45.218 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:18:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.451+0000 7fb48141a640 1 Processor -- start 2023-10-14T18:18:45.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.451+0000 7fb48141a640 1 -- start start 2023-10-14T18:18:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.452+0000 7fb48141a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb47c0b2080 0x7fb47c0d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.452+0000 7fb48141a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c0b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.452+0000 7fb48141a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb47c0d0970 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.452+0000 7fb48141a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb47c0d0ea0 con 0x7fb47c0b2080 2023-10-14T18:18:45.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.452+0000 7fb48141a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c0d0bc0 con 0x7fb47c0cfc40 2023-10-14T18:18:45.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.453+0000 7fb47b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb47c0b2080 0x7fb47c0d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.453+0000 7fb480c19640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c0b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.453+0000 7fb47b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb47c0b2080 0x7fb47c0d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:60830/0 (socket says 172.21.15.29:60830) 2023-10-14T18:18:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.453+0000 7fb47b7fe640 1 -- 172.21.15.29:0/1821752764 learned_addr learned my addr 172.21.15.29:0/1821752764 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.454+0000 7fb47b7fe640 1 -- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 msgr2=0x7fb47c0b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:45.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.454+0000 7fb47b7fe640 1 --2- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c0b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.454+0000 7fb47b7fe640 1 -- 172.21.15.29:0/1821752764 >> v1:172.21.15.186:6789/0 conn(0x7fb47c0b1ca0 legacy=0x7fb47c0ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:45.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.454+0000 7fb47b7fe640 1 -- 172.21.15.29:0/1821752764 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb47c0cd940 con 0x7fb47c0b2080 2023-10-14T18:18:45.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.456+0000 7fb47b7fe640 1 --2- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0d36b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb47002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaa4213750737b1f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.456+0000 7fb47affd640 1 -- 172.21.15.29:0/1821752764 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470030840 con 0x7fb47c0b2080 2023-10-14T18:18:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.456+0000 7fb47affd640 1 -- 172.21.15.29:0/1821752764 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4700309a0 con 0x7fb47c0b2080 2023-10-14T18:18:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.456+0000 7fb47affd640 1 -- 172.21.15.29:0/1821752764 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb470030d30 con 0x7fb47c0b2080 2023-10-14T18:18:45.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 -- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 msgr2=0x7fb47c0d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 --2- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0d36b0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fb470009870 tx=0x7fb47002b980 comp rx=0 tx=0).stop 2023-10-14T18:18:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 -- 172.21.15.29:0/1821752764 shutdown_connections 2023-10-14T18:18:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 --2- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c0b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 --2- 172.21.15.29:0/1821752764 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0d36b0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 -- 172.21.15.29:0/1821752764 >> 172.21.15.29:0/1821752764 conn(0x7fb47c0d2b90 msgr2=0x7fb47c0d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 -- 172.21.15.29:0/1821752764 shutdown_connections 2023-10-14T18:18:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.457+0000 7fb48141a640 1 -- 172.21.15.29:0/1821752764 wait complete. 2023-10-14T18:18:45.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.460+0000 7fb48141a640 1 Processor -- start 2023-10-14T18:18:45.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.460+0000 7fb48141a640 1 -- start start 2023-10-14T18:18:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c1560d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c156610 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c157500 con 0x7fb47c0b2080 2023-10-14T18:18:45.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb48141a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb47c1577e0 con 0x7fb47c0cfc40 2023-10-14T18:18:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.461+0000 7fb47bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52088/0 (socket says 172.21.15.29:52088) 2023-10-14T18:18:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 -- 172.21.15.29:0/2008104149 learned_addr learned my addr 172.21.15.29:0/2008104149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47b7fe640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb480c19640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c1560d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 -- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 msgr2=0x7fb47c1560d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c1560d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 -- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 msgr2=0x7fb47c0c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 -- 172.21.15.29:0/2008104149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4700094d0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.462+0000 7fb47bfff640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00ba50 tx=0x7fb46c00bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.463+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c00cbe0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.463+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb46c00cd80 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.463+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb46c014680 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.463+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47c157ac0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.463+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47c158080 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.464+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb47c158270 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.465+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb46c0147e0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.467+0000 7fb478ff9640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 0x7fb46806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.467+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fb46c089a20 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.467+0000 7fb47b7fe640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 0x7fb46806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.468+0000 7fb47b7fe640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 0x7fb46806d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb470036000 tx=0x7fb470008030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.471+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb46c054700 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.671+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb47c04e9b0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.672+0000 7fb478ff9640 1 -- 172.21.15.29:0/2008104149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v250) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb46c0190a0 con 0x7fb47c0b1ca0 2023-10-14T18:18:45.674 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:18:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.675+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 msgr2=0x7fb46806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.675+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 0x7fb46806d570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb470036000 tx=0x7fb470008030 comp rx=0 tx=0).stop 2023-10-14T18:18:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.675+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 msgr2=0x7fb47c0c6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.675+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 secure :-1 s=READY pgs=2902 cs=0 l=1 rev1=1 crypto rx=0x7fb46c00ba50 tx=0x7fb46c00bf10 comp rx=0 tx=0).stop 2023-10-14T18:18:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 shutdown_connections 2023-10-14T18:18:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb46806b0b0 0x7fb46806d570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb47c0cfc40 0x7fb47c1560d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb47c0b2080 0x7fb47c0c95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 --2- 172.21.15.29:0/2008104149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb47c0b1ca0 0x7fb47c0c6930 unknown :-1 s=CLOSED pgs=2902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 >> 172.21.15.29:0/2008104149 conn(0x7fb47c0d2b90 msgr2=0x7fb47c0b11f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 shutdown_connections 2023-10-14T18:18:45.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.676+0000 7fb48141a640 1 -- 172.21.15.29:0/2008104149 wait complete. 2023-10-14T18:18:45.695 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:18:45.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.927+0000 7fe630573640 1 Processor -- start 2023-10-14T18:18:45.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.927+0000 7fe630573640 1 -- start start 2023-10-14T18:18:45.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.928+0000 7fe630573640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe6280cf730 0x7fe6280af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.928+0000 7fe630573640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe62815b2c0 con 0x7fe6280ae0d0 2023-10-14T18:18:45.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.928+0000 7fe630573640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6280ae5d0 con 0x7fe62815aee0 2023-10-14T18:18:45.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.928+0000 7fe630573640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6280ae8b0 con 0x7fe6280cf730 2023-10-14T18:18:45.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.929+0000 7fe62db0e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe6280ae0d0 0x7fe6280cf020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45052/0 (socket says 172.21.15.29:45052) 2023-10-14T18:18:45.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.929+0000 7fe62db0e640 1 -- 172.21.15.29:0/1750840083 learned_addr learned my addr 172.21.15.29:0/1750840083 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.929+0000 7fe62eb10640 1 --2- 172.21.15.29:0/1750840083 >> v2:172.21.15.186:3300/0 conn(0x7fe6280cf730 0x7fe6280af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.929+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1085730634 0 0) 0x7fe62815b2c0 con 0x7fe6280ae0d0 2023-10-14T18:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.929+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe614003520 con 0x7fe6280ae0d0 2023-10-14T18:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 146376461 0 0) 0x7fe614003520 con 0x7fe6280ae0d0 2023-10-14T18:18:45.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 >> v2:172.21.15.186:3300/0 conn(0x7fe6280cf730 msgr2=0x7fe6280af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 --2- 172.21.15.29:0/1750840083 >> v2:172.21.15.186:3300/0 conn(0x7fe6280cf730 0x7fe6280af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 >> v1:172.21.15.186:6789/0 conn(0x7fe62815aee0 legacy=0x7fe6280b3920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6280aeb90 con 0x7fe6280ae0d0 2023-10-14T18:18:45.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fe618001670 con 0x7fe6280ae0d0 2023-10-14T18:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 >> v1:172.21.15.29:6789/0 conn(0x7fe6280ae0d0 legacy=0x7fe6280cf020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6140039c0 0x7fe614003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe614004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe614005180 0x7fe61400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6280ae8b0 con 0x7fe6140047c0 2023-10-14T18:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe61400c060 con 0x7fe614005180 2023-10-14T18:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe61400c220 con 0x7fe6140039c0 2023-10-14T18:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62db0e640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe614004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.930+0000 7fe62e30f640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6140039c0 0x7fe614003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62eb10640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe614005180 0x7fe61400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62db0e640 1 -- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6140039c0 msgr2=0x7fe614003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62db0e640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe6140039c0 0x7fe614003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62db0e640 1 -- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe614005180 msgr2=0x7fe61400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:45.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62db0e640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe614005180 0x7fe61400b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62db0e640 1 -- 172.21.15.29:0/1750840083 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe614003520 con 0x7fe6140047c0 2023-10-14T18:18:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.931+0000 7fe62eb10640 1 -- 172.21.15.29:0/1750840083 reap_dead start 2023-10-14T18:18:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.932+0000 7fe62db0e640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe614004c40 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c0e0 tx=0x7fe6140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7c5bb47ce00c482 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.932+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe618005ad0 con 0x7fe6140047c0 2023-10-14T18:18:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.932+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe618005c30 con 0x7fe6140047c0 2023-10-14T18:18:45.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.932+0000 7fe62d30d640 1 -- 172.21.15.29:0/1750840083 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe61802ca50 con 0x7fe6140047c0 2023-10-14T18:18:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 -- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 msgr2=0x7fe614004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe614004c40 secure :-1 s=READY pgs=2904 cs=0 l=1 rev1=1 crypto rx=0x7fe61800c0e0 tx=0x7fe6140057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 -- 172.21.15.29:0/1750840083 shutdown_connections 2023-10-14T18:18:45.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 --2- 172.21.15.29:0/1750840083 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe614004c40 unknown :-1 s=CLOSED pgs=2904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 -- 172.21.15.29:0/1750840083 >> 172.21.15.29:0/1750840083 conn(0x7fe6280d0ad0 msgr2=0x7fe6280d0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 -- 172.21.15.29:0/1750840083 shutdown_connections 2023-10-14T18:18:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.933+0000 7fe630573640 1 -- 172.21.15.29:0/1750840083 wait complete. 2023-10-14T18:18:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.935+0000 7fe630573640 1 Processor -- start 2023-10-14T18:18:45.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.935+0000 7fe630573640 1 -- start start 2023-10-14T18:18:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe628150f80 0x7fe6280b5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 0x7fe6281f5810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe628151400 con 0x7fe6140047c0 2023-10-14T18:18:45.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe628151860 con 0x7fe6280b61a0 2023-10-14T18:18:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe630573640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe62815daf0 con 0x7fe628150f80 2023-10-14T18:18:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe62e30f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe62e30f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52098/0 (socket says 172.21.15.29:52098) 2023-10-14T18:18:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe62e30f640 1 -- 172.21.15.29:0/3635934395 learned_addr learned my addr 172.21.15.29:0/3635934395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:45.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe62eb10640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 0x7fe6281f5810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.936+0000 7fe62db0e640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe628150f80 0x7fe6280b5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 -- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe628150f80 msgr2=0x7fe6280b5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe628150f80 0x7fe6280b5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 -- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 msgr2=0x7fe6281f5810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:45.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 0x7fe6281f5810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 -- 172.21.15.29:0/3635934395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe62815ddd0 con 0x7fe6140047c0 2023-10-14T18:18:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62eb10640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 0x7fe6281f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe62e30f640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fe62400e490 tx=0x7fe62400e950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6240052e0 con 0x7fe6140047c0 2023-10-14T18:18:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe624005440 con 0x7fe6140047c0 2023-10-14T18:18:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe62809a7d0 con 0x7fe6140047c0 2023-10-14T18:18:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.937+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe624003880 con 0x7fe6140047c0 2023-10-14T18:18:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.938+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe62809ad90 con 0x7fe6140047c0 2023-10-14T18:18:45.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.939+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe62809af20 con 0x7fe6140047c0 2023-10-14T18:18:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.940+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fe6240039e0 con 0x7fe6140047c0 2023-10-14T18:18:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.941+0000 7fe61effd640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 0x7fe60c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.941+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fe624089950 con 0x7fe6140047c0 2023-10-14T18:18:45.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.941+0000 7fe62db0e640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 0x7fe60c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.942+0000 7fe62db0e640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 0x7fe60c06d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe618034000 tx=0x7fe618009030 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:45.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:45.945+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe624054640 con 0x7fe6140047c0 2023-10-14T18:18:46.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.142+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe62804e160 con 0x7fe6140047c0 2023-10-14T18:18:46.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.143+0000 7fe61effd640 1 -- 172.21.15.29:0/3635934395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v250) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe624057c60 con 0x7fe6140047c0 2023-10-14T18:18:46.144 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:18:46.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 msgr2=0x7fe60c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 0x7fe60c06d570 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fe618034000 tx=0x7fe618009030 comp rx=0 tx=0).stop 2023-10-14T18:18:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 msgr2=0x7fe628150a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 secure :-1 s=READY pgs=2905 cs=0 l=1 rev1=1 crypto rx=0x7fe62400e490 tx=0x7fe62400e950 comp rx=0 tx=0).stop 2023-10-14T18:18:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 shutdown_connections 2023-10-14T18:18:46.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fe60c06b0b0 0x7fe60c06d570 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe6280b61a0 0x7fe6281f5810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe628150f80 0x7fe6280b5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 --2- 172.21.15.29:0/3635934395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe6140047c0 0x7fe628150a40 unknown :-1 s=CLOSED pgs=2905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.145+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 >> 172.21.15.29:0/3635934395 conn(0x7fe6280d0ad0 msgr2=0x7fe62806b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:46.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.146+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 shutdown_connections 2023-10-14T18:18:46.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.146+0000 7fe630573640 1 -- 172.21.15.29:0/3635934395 wait complete. 2023-10-14T18:18:46.167 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:18:46.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.390+0000 7f7923475640 1 Processor -- start 2023-10-14T18:18:46.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.391+0000 7f7923475640 1 -- start start 2023-10-14T18:18:46.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.392+0000 7f7923475640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f791c1428c0 0x7f791c0c9150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.392+0000 7f7923475640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0c9ae0 con 0x7f791c06b660 2023-10-14T18:18:46.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.392+0000 7f7923475640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f791c0cc6b0 con 0x7f791c0c9690 2023-10-14T18:18:46.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.392+0000 7f7923475640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f791c1599b0 con 0x7f791c1428c0 2023-10-14T18:18:46.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.393+0000 7f7921211640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f791c06b660 0x7f791c06ba40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45054/0 (socket says 172.21.15.29:45054) 2023-10-14T18:18:46.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.393+0000 7f7921211640 1 -- 172.21.15.29:0/2558089201 learned_addr learned my addr 172.21.15.29:0/2558089201 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:46.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.393+0000 7f7920a10640 1 --2- 172.21.15.29:0/2558089201 >> v2:172.21.15.186:3300/0 conn(0x7f791c1428c0 0x7f791c0c9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.393+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065995116 0 0) 0x7f7918001110 con 0x7f791c06b660 2023-10-14T18:18:46.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.393+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7908004630 con 0x7f791c06b660 2023-10-14T18:18:46.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3851507267 0 0) 0x7f7908004630 con 0x7f791c06b660 2023-10-14T18:18:46.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 >> v2:172.21.15.186:3300/0 conn(0x7f791c1428c0 msgr2=0x7f791c0c9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 --2- 172.21.15.29:0/2558089201 >> v2:172.21.15.186:3300/0 conn(0x7f791c1428c0 0x7f791c0c9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 >> v1:172.21.15.186:6789/0 conn(0x7f791c0c9690 legacy=0x7f791c0ccb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c06c290 con 0x7f791c06b660 2023-10-14T18:18:46.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7918001800 con 0x7f791c06b660 2023-10-14T18:18:46.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 >> v1:172.21.15.29:6789/0 conn(0x7f791c06b660 legacy=0x7f791c06ba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7908004a80 0x7f7908004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79080058d0 0x7f7908005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f7908003440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c1599b0 con 0x7f7908006290 2023-10-14T18:18:46.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7908003ae0 con 0x7f79080058d0 2023-10-14T18:18:46.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7908003ca0 con 0x7f7908004a80 2023-10-14T18:18:46.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.394+0000 7f7921a12640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f7908003440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921211640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7908004a80 0x7f7908004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7920a10640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79080058d0 0x7f7908005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921a12640 1 -- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7908004a80 msgr2=0x7f7908004ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921a12640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7908004a80 0x7f7908004ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921a12640 1 -- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79080058d0 msgr2=0x7f7908005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921a12640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79080058d0 0x7f7908005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921a12640 1 -- 172.21.15.29:0/2558089201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7908004630 con 0x7f7908006290 2023-10-14T18:18:46.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7920a10640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f79080058d0 0x7f7908005d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:46.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.395+0000 7f7921211640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7908004a80 0x7f7908004ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:46.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.396+0000 7f7921a12640 1 -- 172.21.15.29:0/2558089201 reap_dead start 2023-10-14T18:18:46.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.396+0000 7f7921a12640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f7908003440 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f7914001e50 tx=0x7f7908006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b0e3e66bc0e332f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.396+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914003b70 con 0x7f7908006290 2023-10-14T18:18:46.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.396+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7914003d10 con 0x7f7908006290 2023-10-14T18:18:46.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.396+0000 7f7913fff640 1 -- 172.21.15.29:0/2558089201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7914005310 con 0x7f7908006290 2023-10-14T18:18:46.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 -- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 msgr2=0x7f7908003440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f7908003440 secure :-1 s=READY pgs=2907 cs=0 l=1 rev1=1 crypto rx=0x7f7914001e50 tx=0x7f7908006900 comp rx=0 tx=0).stop 2023-10-14T18:18:46.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 -- 172.21.15.29:0/2558089201 shutdown_connections 2023-10-14T18:18:46.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 --2- 172.21.15.29:0/2558089201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f7908003440 unknown :-1 s=CLOSED pgs=2907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 -- 172.21.15.29:0/2558089201 >> 172.21.15.29:0/2558089201 conn(0x7f791c0ce9b0 msgr2=0x7f791c0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:46.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 -- 172.21.15.29:0/2558089201 shutdown_connections 2023-10-14T18:18:46.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.397+0000 7f7923475640 1 -- 172.21.15.29:0/2558089201 wait complete. 2023-10-14T18:18:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.399+0000 7f7923475640 1 Processor -- start 2023-10-14T18:18:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.399+0000 7f7923475640 1 -- start start 2023-10-14T18:18:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f791c1ee6c0 0x7f791c148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f791c148d70 0x7f791c0acd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c1491f0 con 0x7f7908006290 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f791c14a020 con 0x7f791c1ee6c0 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7923475640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f791c14b690 con 0x7f791c148d70 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.400+0000 7f7921211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52116/0 (socket says 172.21.15.29:52116) 2023-10-14T18:18:46.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 -- 172.21.15.29:0/840972175 learned_addr learned my addr 172.21.15.29:0/840972175 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7920a10640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f791c1ee6c0 0x7f791c148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 -- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f791c148d70 msgr2=0x7f791c0acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f791c148d70 0x7f791c0acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 -- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f791c1ee6c0 msgr2=0x7f791c148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f791c1ee6c0 0x7f791c148830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 -- 172.21.15.29:0/840972175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c149410 con 0x7f7908006290 2023-10-14T18:18:46.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.401+0000 7f7921211640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f791800d7f0 tx=0x7f791800dcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.402+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180074e0 con 0x7f7908006290 2023-10-14T18:18:46.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.402+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7918007640 con 0x7f7908006290 2023-10-14T18:18:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.402+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79180049e0 con 0x7f7908006290 2023-10-14T18:18:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.402+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f791c1496f0 con 0x7f7908006290 2023-10-14T18:18:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.402+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f791c149cb0 con 0x7f7908006290 2023-10-14T18:18:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.403+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f791c04e160 con 0x7f7908006290 2023-10-14T18:18:46.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.404+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7918004b40 con 0x7f7908006290 2023-10-14T18:18:46.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.405+0000 7f7911ffb640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 0x7f790006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.405+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7918089700 con 0x7f7908006290 2023-10-14T18:18:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.405+0000 7f7920a10640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 0x7f790006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.406+0000 7f7920a10640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 0x7f790006d5a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f791c0ad9d0 tx=0x7f790c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.410+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7918054240 con 0x7f7908006290 2023-10-14T18:18:46.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.607+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f791c0c9ae0 con 0x7f7908006290 2023-10-14T18:18:46.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.608+0000 7f7911ffb640 1 -- 172.21.15.29:0/840972175 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v250) v1 ==== 100+0+11 (secure 0 0 0) 0x7f791c0c9ae0 con 0x7f7908006290 2023-10-14T18:18:46.609 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:18:46.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 msgr2=0x7f790006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 0x7f790006d5a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f791c0ad9d0 tx=0x7f790c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 msgr2=0x7f791c0c4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 secure :-1 s=READY pgs=2908 cs=0 l=1 rev1=1 crypto rx=0x7f791800d7f0 tx=0x7f791800dcb0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 shutdown_connections 2023-10-14T18:18:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f790006b0e0 0x7f790006d5a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f791c148d70 0x7f791c0acd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f791c1ee6c0 0x7f791c148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 --2- 172.21.15.29:0/840972175 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7908006290 0x7f791c0c4d10 unknown :-1 s=CLOSED pgs=2908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 >> 172.21.15.29:0/840972175 conn(0x7f791c0ce9b0 msgr2=0x7f791c15a230 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:46.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 shutdown_connections 2023-10-14T18:18:46.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.611+0000 7f7923475640 1 -- 172.21.15.29:0/840972175 wait complete. 2023-10-14T18:18:46.633 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:18:46.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.859+0000 7f1d422d1640 1 Processor -- start 2023-10-14T18:18:46.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.860+0000 7f1d422d1640 1 -- start start 2023-10-14T18:18:46.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.861+0000 7f1d422d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d3c167910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.861+0000 7f1d422d1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0d15a0 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.861+0000 7f1d422d1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d3c0d1880 con 0x7f1d3c15cfc0 2023-10-14T18:18:46.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.861+0000 7f1d422d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0d1b60 con 0x7f1d3c0b4c80 2023-10-14T18:18:46.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.862+0000 7f1d412cf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1d3c15cbe0 0x7f1d3c0d0b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45062/0 (socket says 172.21.15.29:45062) 2023-10-14T18:18:46.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.862+0000 7f1d412cf640 1 -- 172.21.15.29:0/3303254439 learned_addr learned my addr 172.21.15.29:0/3303254439 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:46.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.862+0000 7f1d41ad0640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d3c167910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.862+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269764040 0 0) 0x7f1d3c0d15a0 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.862+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d28003520 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1082746556 0 0) 0x7f1d28003520 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 msgr2=0x7f1d3c167910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d3c167910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 >> v1:172.21.15.186:6789/0 conn(0x7f1d3c15cfc0 legacy=0x7f1d3c0b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0d1e40 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1d380019f0 con 0x7f1d3c15cbe0 2023-10-14T18:18:46.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 >> v1:172.21.15.29:6789/0 conn(0x7f1d3c15cbe0 legacy=0x7f1d3c0d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d280039a0 0x7f1d28003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d28004390 0x7f1d28004860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d280050c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0d1b60 con 0x7f1d28004390 2023-10-14T18:18:46.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800ba40 con 0x7f1d280039a0 2023-10-14T18:18:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d2800bc00 con 0x7f1d3c0b4c80 2023-10-14T18:18:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.863+0000 7f1d40ace640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d28004390 0x7f1d28004860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d41ad0640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d280050c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d412cf640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d280039a0 0x7f1d28003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d40ace640 1 -- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 msgr2=0x7f1d280050c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d40ace640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d280050c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d40ace640 1 -- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d280039a0 msgr2=0x7f1d28003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d40ace640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d280039a0 0x7f1d28003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d40ace640 1 -- 172.21.15.29:0/3303254439 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0d1e40 con 0x7f1d28004390 2023-10-14T18:18:46.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.864+0000 7f1d41ad0640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0b4c80 0x7f1d280050c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:46.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.865+0000 7f1d40ace640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d28004390 0x7f1d28004860 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c001720 tx=0x7f1d280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b37578dafccfc593 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.865+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0052d0 con 0x7f1d28004390 2023-10-14T18:18:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.865+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c02e050 con 0x7f1d28004390 2023-10-14T18:18:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.865+0000 7f1d33fff640 1 -- 172.21.15.29:0/3303254439 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0035f0 con 0x7f1d28004390 2023-10-14T18:18:46.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 -- 172.21.15.29:0/3303254439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d28004390 msgr2=0x7f1d28004860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 --2- 172.21.15.29:0/3303254439 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d28004390 0x7f1d28004860 secure :-1 s=READY pgs=2910 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c001720 tx=0x7f1d280057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d41ad0640 1 -- 172.21.15.29:0/3303254439 reap_dead start 2023-10-14T18:18:46.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 -- 172.21.15.29:0/3303254439 shutdown_connections 2023-10-14T18:18:46.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 -- 172.21.15.29:0/3303254439 >> 172.21.15.29:0/3303254439 conn(0x7f1d3c0cf570 msgr2=0x7f1d3c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:46.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 -- 172.21.15.29:0/3303254439 shutdown_connections 2023-10-14T18:18:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.866+0000 7f1d422d1640 1 -- 172.21.15.29:0/3303254439 wait complete. 2023-10-14T18:18:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.868+0000 7f1d422d1640 1 Processor -- start 2023-10-14T18:18:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.868+0000 7f1d422d1640 1 -- start start 2023-10-14T18:18:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d28004390 0x7f1d3c0c9bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0cc860 0x7f1d3c1f2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0ccd10 con 0x7f1d3c15ae50 2023-10-14T18:18:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0a9fa0 con 0x7f1d28004390 2023-10-14T18:18:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.869+0000 7f1d422d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d3c0a9670 con 0x7f1d3c0cc860 2023-10-14T18:18:46.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52144/0 (socket says 172.21.15.29:52144) 2023-10-14T18:18:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 -- 172.21.15.29:0/4015752280 learned_addr learned my addr 172.21.15.29:0/4015752280 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d412cf640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d28004390 0x7f1d3c0c9bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d41ad0640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0cc860 0x7f1d3c1f2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 -- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0cc860 msgr2=0x7f1d3c1f2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0cc860 0x7f1d3c1f2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 -- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d28004390 msgr2=0x7f1d3c0c9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d28004390 0x7f1d3c0c9bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:46.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.870+0000 7f1d40ace640 1 -- 172.21.15.29:0/4015752280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d3c0a9950 con 0x7f1d3c15ae50 2023-10-14T18:18:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d40ace640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c008fd0 tx=0x7f1d2c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c0076c0 con 0x7f1d3c15ae50 2023-10-14T18:18:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d2c007a30 con 0x7f1d3c15ae50 2023-10-14T18:18:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1d2c006be0 con 0x7f1d3c15ae50 2023-10-14T18:18:46.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d3c0a9c30 con 0x7f1d3c15ae50 2023-10-14T18:18:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.871+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d3c15b2d0 con 0x7f1d3c15ae50 2023-10-14T18:18:46.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.872+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d3c0b85b0 con 0x7f1d3c15ae50 2023-10-14T18:18:46.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.873+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1d2c006d40 con 0x7f1d3c15ae50 2023-10-14T18:18:46.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.874+0000 7f1d31ffb640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 0x7f1d2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:46.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.875+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1d2c0abc20 con 0x7f1d3c15ae50 2023-10-14T18:18:46.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.875+0000 7f1d412cf640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 0x7f1d2006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:46.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.875+0000 7f1d412cf640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 0x7f1d2006d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0ce0e0 tx=0x7f1d38009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:46.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:46.879+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1d3c0b85b0 con 0x7f1d3c15ae50 2023-10-14T18:18:47.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.077+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1d3c0b85b0 con 0x7f1d3c15ae50 2023-10-14T18:18:47.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.078+0000 7f1d31ffb640 1 -- 172.21.15.29:0/4015752280 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v250) v1 ==== 84+0+209 (secure 0 0 0) 0x7f1d3c0b85b0 con 0x7f1d3c15ae50 2023-10-14T18:18:47.079 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:47.079 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:18:47.079 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:47.079 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:18:47.080 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:18:47.080 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:47.080 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:47.080 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 30, 2023-10-14T18:18:47.080 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_metadata" 2023-10-14T18:18:47.081 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:47.081 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:47.081 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 31, 2023-10-14T18:18:47.081 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": "cephfs_data" 2023-10-14T18:18:47.081 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:47.082 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:18:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 msgr2=0x7f1d2006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 0x7f1d2006d570 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1d3c0ce0e0 tx=0x7f1d38009040 comp rx=0 tx=0).stop 2023-10-14T18:18:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 msgr2=0x7f1d3c0cc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 secure :-1 s=READY pgs=2911 cs=0 l=1 rev1=1 crypto rx=0x7f1d2c008fd0 tx=0x7f1d2c001120 comp rx=0 tx=0).stop 2023-10-14T18:18:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 shutdown_connections 2023-10-14T18:18:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f1d2006b0b0 0x7f1d2006d570 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1d3c0cc860 0x7f1d3c1f2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1d3c15ae50 0x7f1d3c0cc320 unknown :-1 s=CLOSED pgs=2911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 --2- 172.21.15.29:0/4015752280 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1d28004390 0x7f1d3c0c9bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 >> 172.21.15.29:0/4015752280 conn(0x7f1d3c0cf570 msgr2=0x7f1d3c0d0420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.081+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 shutdown_connections 2023-10-14T18:18:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.082+0000 7f1d422d1640 1 -- 172.21.15.29:0/4015752280 wait complete. 2023-10-14T18:18:47.101 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'backup_fs' 2023-10-14T18:18:47.102 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_metadata --pg_num_min 64 2023-10-14T18:18:47.338 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:18:47.345 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:18:47.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.341+0000 7f54f3fff640 1 Processor -- start 2023-10-14T18:18:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.342+0000 7f54f3fff640 1 -- start start 2023-10-14T18:18:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f3fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f54f415c9f0 0x7f54f40c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54f40c9230 0x7f54f40709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f3fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f4160a90 con 0x7f54f4070f20 2023-10-14T18:18:47.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f3fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f40cdec0 con 0x7f54f40c9230 2023-10-14T18:18:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f3fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f40cd900 con 0x7f54f415c9f0 2023-10-14T18:18:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54f40c9230 0x7f54f40709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54ebfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f54f40c9230 0x7f54f40709e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52148/0 (socket says 172.21.15.29:52148) 2023-10-14T18:18:47.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54ebfff640 1 -- 172.21.15.29:0/4260050417 learned_addr learned my addr 172.21.15.29:0/4260050417 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:47.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.344+0000 7f54f2ffd640 1 --2- 172.21.15.29:0/4260050417 >> v2:172.21.15.186:3300/0 conn(0x7f54f415c9f0 0x7f54f40c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.345+0000 7f54ebfff640 1 -- 172.21.15.29:0/4260050417 >> v2:172.21.15.186:3300/0 conn(0x7f54f415c9f0 msgr2=0x7f54f40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:47.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.345+0000 7f54ebfff640 1 --2- 172.21.15.29:0/4260050417 >> v2:172.21.15.186:3300/0 conn(0x7f54f415c9f0 0x7f54f40c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.345+0000 7f54ebfff640 1 -- 172.21.15.29:0/4260050417 >> v1:172.21.15.186:6789/0 conn(0x7f54f4070f20 legacy=0x7f54f40ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:47.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.345+0000 7f54ebfff640 1 -- 172.21.15.29:0/4260050417 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f40cdbe0 con 0x7f54f40c9230 2023-10-14T18:18:47.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.347+0000 7f54ebfff640 1 --2- 172.21.15.29:0/4260050417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f40c9230 0x7f54f40709e0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f54f40ad4e0 tx=0x7f54dc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=247545cd8c9efe99 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.347+0000 7f54f27fc640 1 -- 172.21.15.29:0/4260050417 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc031870 con 0x7f54f40c9230 2023-10-14T18:18:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.347+0000 7f54f27fc640 1 -- 172.21.15.29:0/4260050417 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54dc0319d0 con 0x7f54f40c9230 2023-10-14T18:18:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.347+0000 7f54f27fc640 1 -- 172.21.15.29:0/4260050417 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54dc031d60 con 0x7f54f40c9230 2023-10-14T18:18:47.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 -- 172.21.15.29:0/4260050417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f40c9230 msgr2=0x7f54f40709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 --2- 172.21.15.29:0/4260050417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f40c9230 0x7f54f40709e0 secure :-1 s=READY pgs=2912 cs=0 l=1 rev1=1 crypto rx=0x7f54f40ad4e0 tx=0x7f54dc02b910 comp rx=0 tx=0).stop 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 -- 172.21.15.29:0/4260050417 shutdown_connections 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 --2- 172.21.15.29:0/4260050417 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f40c9230 0x7f54f40709e0 unknown :-1 s=CLOSED pgs=2912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 --2- 172.21.15.29:0/4260050417 >> v2:172.21.15.186:3300/0 conn(0x7f54f415c9f0 0x7f54f40c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 -- 172.21.15.29:0/4260050417 >> 172.21.15.29:0/4260050417 conn(0x7f54f40ce6a0 msgr2=0x7f54f40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:47.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 -- 172.21.15.29:0/4260050417 shutdown_connections 2023-10-14T18:18:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.348+0000 7f54f3fff640 1 -- 172.21.15.29:0/4260050417 wait complete. 2023-10-14T18:18:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.351+0000 7f54f3fff640 1 Processor -- start 2023-10-14T18:18:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.351+0000 7f54f3fff640 1 -- start start 2023-10-14T18:18:47.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f40c9230 0x7f54f41f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54f415c9f0 0x7f54f40aee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f41f6920 con 0x7f54f4070f20 2023-10-14T18:18:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54f406b3a0 con 0x7f54f40c9230 2023-10-14T18:18:47.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.352+0000 7f54f3fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f41f28a0 con 0x7f54f415c9f0 2023-10-14T18:18:47.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52162/0 (socket says 172.21.15.29:52162) 2023-10-14T18:18:47.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 -- 172.21.15.29:0/3520824330 learned_addr learned my addr 172.21.15.29:0/3520824330 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:47.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54ebfff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f40c9230 0x7f54f41f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f37fe640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54f415c9f0 0x7f54f40aee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 -- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54f415c9f0 msgr2=0x7f54f40aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:47.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54f415c9f0 0x7f54f40aee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 -- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f40c9230 msgr2=0x7f54f41f63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:47.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f40c9230 0x7f54f41f63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:47.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.353+0000 7f54f2ffd640 1 -- 172.21.15.29:0/3520824330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54dc0094d0 con 0x7f54f4070f20 2023-10-14T18:18:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54f2ffd640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f54e400d5f0 tx=0x7f54e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:47.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e40125f0 con 0x7f54f4070f20 2023-10-14T18:18:47.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54e4012750 con 0x7f54f4070f20 2023-10-14T18:18:47.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54e4012b00 con 0x7f54f4070f20 2023-10-14T18:18:47.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f41f2b80 con 0x7f54f4070f20 2023-10-14T18:18:47.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.354+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f41f3140 con 0x7f54f4070f20 2023-10-14T18:18:47.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.355+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f41f3330 con 0x7f54f4070f20 2023-10-14T18:18:47.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.356+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f54e4012c60 con 0x7f54f4070f20 2023-10-14T18:18:47.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.357+0000 7f54eb7fe640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 0x7f54d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:47.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.357+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(250..250 src has 1..250) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f54e408a0c0 con 0x7f54f4070f20 2023-10-14T18:18:47.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.358+0000 7f54ebfff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 0x7f54d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:47.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.359+0000 7f54ebfff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 0x7f54d406d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f54dc036000 tx=0x7f54dc03c040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:47.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.363+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f54e4054cf0 con 0x7f54f4070f20 2023-10-14T18:18:47.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:47.564+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f54f404e9b0 con 0x7f54f4070f20 2023-10-14T18:18:48.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.257+0000 7f54eb7fe640 1 -- 172.21.15.29:0/3520824330 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_metadata", "pg_num_min": 64}]=0 pool 'backup_fs_metadata' created v251) v1 ==== 144+0+0 (secure 0 0 0) 0x7f54e4058310 con 0x7f54f4070f20 2023-10-14T18:18:48.259 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' created 2023-10-14T18:18:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 msgr2=0x7f54d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 0x7f54d406d570 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f54dc036000 tx=0x7f54dc03c040 comp rx=0 tx=0).stop 2023-10-14T18:18:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 msgr2=0x7f54f406ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:48.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 secure :-1 s=READY pgs=2913 cs=0 l=1 rev1=1 crypto rx=0x7f54e400d5f0 tx=0x7f54e400dab0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 shutdown_connections 2023-10-14T18:18:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f54d406b0b0 0x7f54d406d570 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f54f415c9f0 0x7f54f40aee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f54f40c9230 0x7f54f41f63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 --2- 172.21.15.29:0/3520824330 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f54f4070f20 0x7f54f406ba30 unknown :-1 s=CLOSED pgs=2913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 >> 172.21.15.29:0/3520824330 conn(0x7f54f40ce6a0 msgr2=0x7f54f40ae200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 shutdown_connections 2023-10-14T18:18:48.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.261+0000 7f54f3fff640 1 -- 172.21.15.29:0/3520824330 wait complete. 2023-10-14T18:18:48.288 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create backup_fs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:18:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.509+0000 7f4c6cd47640 1 Processor -- start 2023-10-14T18:18:48.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.509+0000 7f4c6cd47640 1 -- start start 2023-10-14T18:18:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.510+0000 7f4c6cd47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c6815ac70 0x7f4c680cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.510+0000 7f4c6cd47640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4c680b4160 0x7f4c680cd8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.510+0000 7f4c6cd47640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c68069da0 0x7f4c6806a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.510+0000 7f4c6cd47640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c6806a770 con 0x7f4c68069da0 2023-10-14T18:18:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c6cd47640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c680cc850 con 0x7f4c680b4160 2023-10-14T18:18:48.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c6cd47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680ccb30 con 0x7f4c6815ac70 2023-10-14T18:18:48.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c68069da0 0x7f4c6806a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c67fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4c68069da0 0x7f4c6806a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52168/0 (socket says 172.21.15.29:52168) 2023-10-14T18:18:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c67fff640 1 -- 172.21.15.29:0/1354610918 learned_addr learned my addr 172.21.15.29:0/1354610918 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c677fe640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c6815ac70 0x7f4c680cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.511+0000 7f4c66ffd640 1 --2- 172.21.15.29:0/1354610918 >> v2:172.21.15.186:3300/0 conn(0x7f4c680b4160 0x7f4c680cd8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.512+0000 7f4c67fff640 1 -- 172.21.15.29:0/1354610918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c6815ac70 msgr2=0x7f4c680cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.512+0000 7f4c67fff640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c6815ac70 0x7f4c680cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.512+0000 7f4c67fff640 1 -- 172.21.15.29:0/1354610918 >> v2:172.21.15.186:3300/0 conn(0x7f4c680b4160 msgr2=0x7f4c680cd8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.512+0000 7f4c67fff640 1 --2- 172.21.15.29:0/1354610918 >> v2:172.21.15.186:3300/0 conn(0x7f4c680b4160 0x7f4c680cd8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.512+0000 7f4c67fff640 1 -- 172.21.15.29:0/1354610918 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c680cce10 con 0x7f4c68069da0 2023-10-14T18:18:48.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.513+0000 7f4c67fff640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c68069da0 0x7f4c6806a230 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f4c680caff0 tx=0x7f4c6002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdec2273dbe6b53f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.513+0000 7f4c667fc640 1 -- 172.21.15.29:0/1354610918 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c6002ca70 con 0x7f4c68069da0 2023-10-14T18:18:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c667fc640 1 -- 172.21.15.29:0/1354610918 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6002cbd0 con 0x7f4c68069da0 2023-10-14T18:18:48.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c667fc640 1 -- 172.21.15.29:0/1354610918 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c600343f0 con 0x7f4c68069da0 2023-10-14T18:18:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1354610918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c68069da0 msgr2=0x7f4c6806a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c68069da0 0x7f4c6806a230 secure :-1 s=READY pgs=2914 cs=0 l=1 rev1=1 crypto rx=0x7f4c680caff0 tx=0x7f4c6002bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1354610918 shutdown_connections 2023-10-14T18:18:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c68069da0 0x7f4c6806a230 unknown :-1 s=CLOSED pgs=2914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1354610918 >> v2:172.21.15.186:3300/0 conn(0x7f4c680b4160 0x7f4c680cd8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1354610918 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c6815ac70 0x7f4c680cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1354610918 >> 172.21.15.29:0/1354610918 conn(0x7f4c680ce6a0 msgr2=0x7f4c680ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1354610918 shutdown_connections 2023-10-14T18:18:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.514+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1354610918 wait complete. 2023-10-14T18:18:48.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.516+0000 7f4c6cd47640 1 Processor -- start 2023-10-14T18:18:48.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.516+0000 7f4c6cd47640 1 -- start start 2023-10-14T18:18:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c68069da0 0x7f4c680c8110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 0x7f4c6815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c8650 con 0x7f4c680b4160 2023-10-14T18:18:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c5c00 con 0x7f4c6815ac70 2023-10-14T18:18:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.517+0000 7f4c6cd47640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c680c5ee0 con 0x7f4c68069da0 2023-10-14T18:18:48.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52178/0 (socket says 172.21.15.29:52178) 2023-10-14T18:18:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 -- 172.21.15.29:0/1884382057 learned_addr learned my addr 172.21.15.29:0/1884382057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c67fff640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 0x7f4c6815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c677fe640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c68069da0 0x7f4c680c8110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 -- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c68069da0 msgr2=0x7f4c680c8110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c68069da0 0x7f4c680c8110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 -- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 msgr2=0x7f4c6815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 0x7f4c6815b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c66ffd640 1 -- 172.21.15.29:0/1884382057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c60009b30 con 0x7f4c680b4160 2023-10-14T18:18:48.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.518+0000 7f4c67fff640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 0x7f4c6815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c66ffd640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e870 tx=0x7f4c5c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00ee60 con 0x7f4c680b4160 2023-10-14T18:18:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c00c940 con 0x7f4c680b4160 2023-10-14T18:18:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00ccf0 con 0x7f4c680b4160 2023-10-14T18:18:48.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c680c61c0 con 0x7f4c680b4160 2023-10-14T18:18:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.519+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c6814f480 con 0x7f4c680b4160 2023-10-14T18:18:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.520+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c680c63b0 con 0x7f4c680b4160 2023-10-14T18:18:48.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.521+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4c5c013470 con 0x7f4c680b4160 2023-10-14T18:18:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.523+0000 7f4c4ffff640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 0x7f4c5406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.523+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(251..251 src has 1..251) v4 ==== 6198+0+0 (secure 0 0 0) 0x7f4c5c089d00 con 0x7f4c680b4160 2023-10-14T18:18:48.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.523+0000 7f4c677fe640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 0x7f4c5406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:48.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.524+0000 7f4c677fe640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 0x7f4c5406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c6814f740 tx=0x7f4c58008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:48.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.528+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4c5c0547c0 con 0x7f4c680b4160 2023-10-14T18:18:48.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:48.732+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7f4c6804e9b0 con 0x7f4c680b4160 2023-10-14T18:18:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.277+0000 7f4c4ffff640 1 -- 172.21.15.29:0/1884382057 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "backup_fs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'backup_fs_data' created v252) v1 ==== 176+0+0 (secure 0 0 0) 0x7f4c5c0170d0 con 0x7f4c680b4160 2023-10-14T18:18:49.279 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' created 2023-10-14T18:18:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.281+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 msgr2=0x7f4c5406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.281+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 0x7f4c5406d570 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4c6814f740 tx=0x7f4c58008040 comp rx=0 tx=0).stop 2023-10-14T18:18:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 msgr2=0x7f4c68070c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 secure :-1 s=READY pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e870 tx=0x7f4c5c00ed30 comp rx=0 tx=0).stop 2023-10-14T18:18:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 shutdown_connections 2023-10-14T18:18:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4c5406b0b0 0x7f4c5406d570 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4c6815ac70 0x7f4c6815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4c680b4160 0x7f4c68070c10 secure :-1 s=CLOSED pgs=2915 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c00e870 tx=0x7f4c5c00ed30 comp rx=0 tx=0).stop 2023-10-14T18:18:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 --2- 172.21.15.29:0/1884382057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4c68069da0 0x7f4c680c8110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 >> 172.21.15.29:0/1884382057 conn(0x7f4c680ce6a0 msgr2=0x7f4c6806f840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.282+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 shutdown_connections 2023-10-14T18:18:49.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.283+0000 7f4c6cd47640 1 -- 172.21.15.29:0/1884382057 wait complete. 2023-10-14T18:18:49.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new backup_fs backup_fs_metadata backup_fs_data 2023-10-14T18:18:49.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.513+0000 7f92b1844640 1 Processor -- start 2023-10-14T18:18:49.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.513+0000 7f92b1844640 1 -- start start 2023-10-14T18:18:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.515+0000 7f92b1844640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92ac15ad70 0x7f92ac069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.515+0000 7f92b1844640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0c9ca0 0x7f92ac06a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.515+0000 7f92b1844640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92ac0cb030 con 0x7f92ac0ca3b0 2023-10-14T18:18:49.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.515+0000 7f92b1844640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92ac0c91c0 con 0x7f92ac15ad70 2023-10-14T18:18:49.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.515+0000 7f92b1844640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac0c94a0 con 0x7f92ac0c9ca0 2023-10-14T18:18:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92ac15ad70 0x7f92ac069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92ac15ad70 0x7f92ac069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52180/0 (socket says 172.21.15.29:52180) 2023-10-14T18:18:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 -- 172.21.15.29:0/1269268176 learned_addr learned my addr 172.21.15.29:0/1269268176 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:49.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aa7fc640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0c9ca0 0x7f92ac06a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 -- 172.21.15.29:0/1269268176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0c9ca0 msgr2=0x7f92ac06a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0c9ca0 0x7f92ac06a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 -- 172.21.15.29:0/1269268176 >> v1:172.21.15.186:6789/0 conn(0x7f92ac0ca3b0 legacy=0x7f92ac0ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.516+0000 7f92aaffd640 1 -- 172.21.15.29:0/1269268176 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92ac0c9780 con 0x7f92ac15ad70 2023-10-14T18:18:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92aaffd640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac069da0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f929c009870 tx=0x7f929c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5106f7a9fcfa602 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92a9ffb640 1 -- 172.21.15.29:0/1269268176 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c030840 con 0x7f92ac15ad70 2023-10-14T18:18:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92a9ffb640 1 -- 172.21.15.29:0/1269268176 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f929c0309a0 con 0x7f92ac15ad70 2023-10-14T18:18:49.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92a9ffb640 1 -- 172.21.15.29:0/1269268176 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f929c030d30 con 0x7f92ac15ad70 2023-10-14T18:18:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92b1844640 1 -- 172.21.15.29:0/1269268176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 msgr2=0x7f92ac069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.518+0000 7f92b1844640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac069da0 secure :-1 s=READY pgs=2916 cs=0 l=1 rev1=1 crypto rx=0x7f929c009870 tx=0x7f929c02b980 comp rx=0 tx=0).stop 2023-10-14T18:18:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 -- 172.21.15.29:0/1269268176 shutdown_connections 2023-10-14T18:18:49.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0c9ca0 0x7f92ac06a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 --2- 172.21.15.29:0/1269268176 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac069da0 unknown :-1 s=CLOSED pgs=2916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 -- 172.21.15.29:0/1269268176 >> 172.21.15.29:0/1269268176 conn(0x7f92ac0ce9b0 msgr2=0x7f92ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 -- 172.21.15.29:0/1269268176 shutdown_connections 2023-10-14T18:18:49.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.519+0000 7f92b1844640 1 -- 172.21.15.29:0/1269268176 wait complete. 2023-10-14T18:18:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.521+0000 7f92b1844640 1 Processor -- start 2023-10-14T18:18:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.521+0000 7f92b1844640 1 -- start start 2023-10-14T18:18:49.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ac0c9ca0 0x7f92ac15df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 0x7f92ac15bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac14e940 con 0x7f92ac15ad70 2023-10-14T18:18:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac14e380 con 0x7f92ac0c9ca0 2023-10-14T18:18:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92b1844640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92ac14e660 con 0x7f92ac0ca3b0 2023-10-14T18:18:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92ab7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52192/0 (socket says 172.21.15.29:52192) 2023-10-14T18:18:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92ab7fe640 1 -- 172.21.15.29:0/3890308744 learned_addr learned my addr 172.21.15.29:0/3890308744 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92aa7fc640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 0x7f92ac15bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.522+0000 7f92aaffd640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ac0c9ca0 0x7f92ac15df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 -- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 msgr2=0x7f92ac15bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 0x7f92ac15bd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 -- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ac0c9ca0 msgr2=0x7f92ac15df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ac0c9ca0 0x7f92ac15df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:49.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 -- 172.21.15.29:0/3890308744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929c0094d0 con 0x7f92ac15ad70 2023-10-14T18:18:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92aa7fc640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 0x7f92ac15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92ab7fe640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f92a400c7a0 tx=0x7f92a400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4003b70 con 0x7f92ac15ad70 2023-10-14T18:18:49.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92a4003cf0 con 0x7f92ac15ad70 2023-10-14T18:18:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92a4003ea0 con 0x7f92ac15ad70 2023-10-14T18:18:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92ac15b580 con 0x7f92ac15ad70 2023-10-14T18:18:49.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.523+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92ac15c2c0 con 0x7f92ac15ad70 2023-10-14T18:18:49.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.525+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92ac15baa0 con 0x7f92ac15ad70 2023-10-14T18:18:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.526+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f92a400aa90 con 0x7f92ac15ad70 2023-10-14T18:18:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.528+0000 7f9293fff640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 0x7f929806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.528+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(252..252 src has 1..252) v4 ==== 6583+0+0 (secure 0 0 0) 0x7f92a408a420 con 0x7f92ac15ad70 2023-10-14T18:18:49.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.528+0000 7f92aaffd640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 0x7f929806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.529+0000 7f92aaffd640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 0x7f929806d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f929c036000 tx=0x7f929c03b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:49.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.532+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f92a4054f70 con 0x7f92ac15ad70 2023-10-14T18:18:49.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:49.771+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"} v 0) v1 -- 0x7f92ac15baa0 con 0x7f92ac15ad70 2023-10-14T18:18:50.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.282+0000 7f9293fff640 1 -- 172.21.15.29:0/3890308744 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "backup_fs", "metadata": "backup_fs_metadata", "data": "backup_fs_data"}]=0 new fs with metadata pool 32 and data pool 33 v125) v1 ==== 183+0+0 (secure 0 0 0) 0x7f92ac15baa0 con 0x7f92ac15ad70 2023-10-14T18:18:50.284 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 32 and data pool 33 2023-10-14T18:18:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 msgr2=0x7f929806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 0x7f929806d570 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f929c036000 tx=0x7f929c03b040 comp rx=0 tx=0).stop 2023-10-14T18:18:50.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 msgr2=0x7f92ac14da10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 secure :-1 s=READY pgs=2917 cs=0 l=1 rev1=1 crypto rx=0x7f92a400c7a0 tx=0x7f92a400cc60 comp rx=0 tx=0).stop 2023-10-14T18:18:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 shutdown_connections 2023-10-14T18:18:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f929806b0b0 0x7f929806d570 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92ac15ad70 0x7f92ac14da10 unknown :-1 s=CLOSED pgs=2917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92ac0ca3b0 0x7f92ac15bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 --2- 172.21.15.29:0/3890308744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92ac0c9ca0 0x7f92ac15df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.286+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 >> 172.21.15.29:0/3890308744 conn(0x7f92ac0ce9b0 msgr2=0x7f92ac0c4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:50.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.287+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 shutdown_connections 2023-10-14T18:18:50.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.287+0000 7f92b1844640 1 -- 172.21.15.29:0/3890308744 wait complete. 2023-10-14T18:18:50.332 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:50.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.520+0000 7f595a9e9640 1 Processor -- start 2023-10-14T18:18:50.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.520+0000 7f595a9e9640 1 -- start start 2023-10-14T18:18:50.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.521+0000 7f595a9e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5954159b20 0x7f595406bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.521+0000 7f595a9e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f59540b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f595a9e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f595406f7f0 con 0x7f595406c120 2023-10-14T18:18:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f595a9e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f595406c5e0 con 0x7f5954159b20 2023-10-14T18:18:50.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f595a9e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f595406c8c0 con 0x7f5954159f00 2023-10-14T18:18:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f5958f86640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f595406c120 0x7f59540cd670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45070/0 (socket says 172.21.15.29:45070) 2023-10-14T18:18:50.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f5958f86640 1 -- 172.21.15.29:0/1970711255 learned_addr learned my addr 172.21.15.29:0/1970711255 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f5953fff640 1 --2- 172.21.15.29:0/1970711255 >> v2:172.21.15.186:3300/0 conn(0x7f5954159b20 0x7f595406bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f594bfff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f59540b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.522+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2352519005 0 0) 0x7f595406f7f0 con 0x7f595406c120 2023-10-14T18:18:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5940003520 con 0x7f595406c120 2023-10-14T18:18:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3255449463 0 0) 0x7f5940003520 con 0x7f595406c120 2023-10-14T18:18:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 msgr2=0x7f59540b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f59540b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 >> v2:172.21.15.186:3300/0 conn(0x7f5954159b20 msgr2=0x7f595406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 --2- 172.21.15.29:0/1970711255 >> v2:172.21.15.186:3300/0 conn(0x7f5954159b20 0x7f595406bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f595406cba0 con 0x7f595406c120 2023-10-14T18:18:50.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f594c0019f0 con 0x7f595406c120 2023-10-14T18:18:50.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 >> v1:172.21.15.29:6789/0 conn(0x7f595406c120 legacy=0x7f59540cd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5940003970 0x7f5940003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f5940004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f5940005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595406c5e0 con 0x7f59400047c0 2023-10-14T18:18:50.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f595406c8c0 con 0x7f5940003970 2023-10-14T18:18:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f594000b980 con 0x7f5954159f00 2023-10-14T18:18:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.523+0000 7f594bfff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f5940004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f5953fff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5940003970 0x7f5940003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f5958f86640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f5940005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f594bfff640 1 -- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 msgr2=0x7f5940005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f594bfff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5954159f00 0x7f5940005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f594bfff640 1 -- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5940003970 msgr2=0x7f5940003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f594bfff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5940003970 0x7f5940003d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.524+0000 7f594bfff640 1 -- 172.21.15.29:0/1970711255 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5940003520 con 0x7f59400047c0 2023-10-14T18:18:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.525+0000 7f594bfff640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f5940004c40 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f593c009a90 tx=0x7f59400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=76459c274f091b7b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.525+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c02c6b0 con 0x7f59400047c0 2023-10-14T18:18:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f593c02c810 con 0x7f59400047c0 2023-10-14T18:18:50.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f59537fe640 1 -- 172.21.15.29:0/1970711255 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f593c02cba0 con 0x7f59400047c0 2023-10-14T18:18:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 -- 172.21.15.29:0/1970711255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 msgr2=0x7f5940004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 --2- 172.21.15.29:0/1970711255 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f5940004c40 secure :-1 s=READY pgs=2919 cs=0 l=1 rev1=1 crypto rx=0x7f593c009a90 tx=0x7f59400057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f5958f86640 1 -- 172.21.15.29:0/1970711255 reap_dead start 2023-10-14T18:18:50.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 -- 172.21.15.29:0/1970711255 shutdown_connections 2023-10-14T18:18:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 -- 172.21.15.29:0/1970711255 >> 172.21.15.29:0/1970711255 conn(0x7f59540cb410 msgr2=0x7f59540cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 -- 172.21.15.29:0/1970711255 shutdown_connections 2023-10-14T18:18:50.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.526+0000 7f595a9e9640 1 -- 172.21.15.29:0/1970711255 wait complete. 2023-10-14T18:18:50.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.528+0000 7f595a9e9640 1 Processor -- start 2023-10-14T18:18:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.528+0000 7f595a9e9640 1 -- start start 2023-10-14T18:18:50.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59540c7a40 0x7f5954156a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5954156f60 0x7f59540c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59541573e0 con 0x7f59400047c0 2023-10-14T18:18:50.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59540c31f0 con 0x7f5954156f60 2023-10-14T18:18:50.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.529+0000 7f595a9e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59540c6c80 con 0x7f59540c7a40 2023-10-14T18:18:50.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52204/0 (socket says 172.21.15.29:52204) 2023-10-14T18:18:50.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 -- 172.21.15.29:0/2596691653 learned_addr learned my addr 172.21.15.29:0/2596691653 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5958f86640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5954156f60 0x7f59540c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f594bfff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59540c7a40 0x7f5954156a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 -- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59540c7a40 msgr2=0x7f5954156a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59540c7a40 0x7f5954156a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 -- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5954156f60 msgr2=0x7f59540c24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5954156f60 0x7f59540c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.530+0000 7f5953fff640 1 -- 172.21.15.29:0/2596691653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59540c6f60 con 0x7f59400047c0 2023-10-14T18:18:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f5953fff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f59440108e0 tx=0x7f5944010da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:50.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594400eb60 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f594400ecc0 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f594401c400 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59540c7240 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.531+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59540c7ec0 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.532+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f595404e160 con 0x7f59400047c0 2023-10-14T18:18:50.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.533+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f594401c560 con 0x7f59400047c0 2023-10-14T18:18:50.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.535+0000 7f59517fa640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 0x7f593406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.535+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f594408a420 con 0x7f59400047c0 2023-10-14T18:18:50.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.535+0000 7f594bfff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 0x7f593406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.536+0000 7f594bfff640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 0x7f593406d5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:50.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.540+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5944054f00 con 0x7f59400047c0 2023-10-14T18:18:50.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.741+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59540b10b0 con 0x7f59400047c0 2023-10-14T18:18:50.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.745+0000 7f59517fa640 1 -- 172.21.15.29:0/2596691653 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20374 (secure 0 0 0) 0x7f5944058520 con 0x7f59400047c0 2023-10-14T18:18:50.746 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:50.747 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":253,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:49.773100+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 msgr2=0x7f593406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 0x7f593406d5a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f593c036000 tx=0x7f593c030000 comp rx=0 tx=0).stop 2023-10-14T18:18:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 msgr2=0x7f595414fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 secure :-1 s=READY pgs=2920 cs=0 l=1 rev1=1 crypto rx=0x7f59440108e0 tx=0x7f5944010da0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 shutdown_connections 2023-10-14T18:18:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f593406b0e0 0x7f593406d5a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5954156f60 0x7f59540c24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f59540c7a40 0x7f5954156a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 --2- 172.21.15.29:0/2596691653 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f59400047c0 0x7f595414fc80 unknown :-1 s=CLOSED pgs=2920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 >> 172.21.15.29:0/2596691653 conn(0x7f59540cb410 msgr2=0x7f59540b09a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:50.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 shutdown_connections 2023-10-14T18:18:50.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.748+0000 7f595a9e9640 1 -- 172.21.15.29:0/2596691653 wait complete. 2023-10-14T18:18:50.768 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.988+0000 7f40b5947640 1 Processor -- start 2023-10-14T18:18:50.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.988+0000 7f40b5947640 1 -- start start 2023-10-14T18:18:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.990+0000 7f40b5947640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b015c9f0 0x7f40b00c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.990+0000 7f40b5947640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 0x7f40b00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:50.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.990+0000 7f40b5947640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b0160a90 con 0x7f40b0070f20 2023-10-14T18:18:50.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.990+0000 7f40b5947640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00cdec0 con 0x7f40b015c9f0 2023-10-14T18:18:50.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.990+0000 7f40b5947640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b00cd900 con 0x7f40b00c9230 2023-10-14T18:18:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b015c9f0 0x7f40b00c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40b015c9f0 0x7f40b00c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40774/0 (socket says 172.21.15.29:40774) 2023-10-14T18:18:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 -- 172.21.15.29:0/3996379230 learned_addr learned my addr 172.21.15.29:0/3996379230 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40affff640 1 --2- 172.21.15.29:0/3996379230 >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 0x7f40b00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:50.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 -- 172.21.15.29:0/3996379230 >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 msgr2=0x7f40b00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 --2- 172.21.15.29:0/3996379230 >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 0x7f40b00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 -- 172.21.15.29:0/3996379230 >> v1:172.21.15.186:6789/0 conn(0x7f40b0070f20 legacy=0x7f40b00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40b4945640 1 -- 172.21.15.29:0/3996379230 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b00cdbe0 con 0x7f40b015c9f0 2023-10-14T18:18:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.991+0000 7f40affff640 1 --2- 172.21.15.29:0/3996379230 >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 0x7f40b00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:50.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.992+0000 7f40b4945640 1 --2- 172.21.15.29:0/3996379230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015c9f0 0x7f40b00c8cf0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f40b00b1260 tx=0x7f40a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3624b247edd71ba6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:50.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.993+0000 7f40af7fe640 1 -- 172.21.15.29:0/3996379230 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4031870 con 0x7f40b015c9f0 2023-10-14T18:18:50.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.993+0000 7f40af7fe640 1 -- 172.21.15.29:0/3996379230 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a40319d0 con 0x7f40b015c9f0 2023-10-14T18:18:50.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.993+0000 7f40af7fe640 1 -- 172.21.15.29:0/3996379230 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a4031d60 con 0x7f40b015c9f0 2023-10-14T18:18:50.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 -- 172.21.15.29:0/3996379230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015c9f0 msgr2=0x7f40b00c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:50.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 --2- 172.21.15.29:0/3996379230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015c9f0 0x7f40b00c8cf0 secure :-1 s=READY pgs=2921 cs=0 l=1 rev1=1 crypto rx=0x7f40b00b1260 tx=0x7f40a402b910 comp rx=0 tx=0).stop 2023-10-14T18:18:50.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 -- 172.21.15.29:0/3996379230 shutdown_connections 2023-10-14T18:18:50.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 --2- 172.21.15.29:0/3996379230 >> v2:172.21.15.186:3300/0 conn(0x7f40b00c9230 0x7f40b00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 --2- 172.21.15.29:0/3996379230 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b015c9f0 0x7f40b00c8cf0 unknown :-1 s=CLOSED pgs=2921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:50.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 -- 172.21.15.29:0/3996379230 >> 172.21.15.29:0/3996379230 conn(0x7f40b00ce6a0 msgr2=0x7f40b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:50.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 -- 172.21.15.29:0/3996379230 shutdown_connections 2023-10-14T18:18:50.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.994+0000 7f40b5947640 1 -- 172.21.15.29:0/3996379230 wait complete. 2023-10-14T18:18:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.996+0000 7f40b5947640 1 Processor -- start 2023-10-14T18:18:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.996+0000 7f40b5947640 1 -- start start 2023-10-14T18:18:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b00c9230 0x7f40b00c3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b015c9f0 0x7f40b00aebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00c62b0 con 0x7f40b0070f20 2023-10-14T18:18:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b00c4770 con 0x7f40b015c9f0 2023-10-14T18:18:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5947640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b0159e00 con 0x7f40b00c9230 2023-10-14T18:18:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40780/0 (socket says 172.21.15.29:40780) 2023-10-14T18:18:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 -- 172.21.15.29:0/2928327970 learned_addr learned my addr 172.21.15.29:0/2928327970 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40affff640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b00c9230 0x7f40b00c3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b5146640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b015c9f0 0x7f40b00aebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 -- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b00c9230 msgr2=0x7f40b00c3e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b00c9230 0x7f40b00c3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 -- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b015c9f0 msgr2=0x7f40b00aebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.998+0000 7f40b4945640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b015c9f0 0x7f40b00aebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40b4945640 1 -- 172.21.15.29:0/2928327970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a40094d0 con 0x7f40b0070f20 2023-10-14T18:18:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40b4945640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f40a402b950 tx=0x7f40a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:51.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a403ad30 con 0x7f40b0070f20 2023-10-14T18:18:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40a40443f0 con 0x7f40b0070f20 2023-10-14T18:18:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40a40447a0 con 0x7f40b0070f20 2023-10-14T18:18:51.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b00c5910 con 0x7f40b0070f20 2023-10-14T18:18:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:50.999+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b00c5ed0 con 0x7f40b0070f20 2023-10-14T18:18:51.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.001+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b00668c0 con 0x7f40b0070f20 2023-10-14T18:18:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.002+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f40a4044900 con 0x7f40b0070f20 2023-10-14T18:18:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.003+0000 7f40ad7fa640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 0x7f409406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.003+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(253..253 src has 1..253) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f40a402d040 con 0x7f40b0070f20 2023-10-14T18:18:51.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.004+0000 7f40affff640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 0x7f409406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.004+0000 7f40affff640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 0x7f409406d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f409c006fd0 tx=0x7f409c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:51.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.008+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40a4076dc0 con 0x7f40b0070f20 2023-10-14T18:18:51.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.206+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40b0161670 con 0x7f40b0070f20 2023-10-14T18:18:51.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.210+0000 7f40ad7fa640 1 -- 172.21.15.29:0/2928327970 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v253) v1 ==== 74+0+20374 (secure 0 0 0) 0x7f40a4035090 con 0x7f40b0070f20 2023-10-14T18:18:51.211 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:51.212 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":253,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:49.773100+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:51.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 msgr2=0x7f409406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 0x7f409406d570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f409c006fd0 tx=0x7f409c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 msgr2=0x7f40b01598c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 secure :-1 s=READY pgs=2922 cs=0 l=1 rev1=1 crypto rx=0x7f40a402b950 tx=0x7f40a40046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 shutdown_connections 2023-10-14T18:18:51.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f409406b0b0 0x7f409406d570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b015c9f0 0x7f40b00aebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b00c9230 0x7f40b00c3e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 --2- 172.21.15.29:0/2928327970 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b0070f20 0x7f40b01598c0 unknown :-1 s=CLOSED pgs=2922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 >> 172.21.15.29:0/2928327970 conn(0x7f40b00ce6a0 msgr2=0x7f40b015da00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:51.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 shutdown_connections 2023-10-14T18:18:51.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.213+0000 7f40b5947640 1 -- 172.21.15.29:0/2928327970 wait complete. 2023-10-14T18:18:51.236 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs standby_count_wanted 0 2023-10-14T18:18:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.471+0000 7fcf9cd33640 1 Processor -- start 2023-10-14T18:18:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.471+0000 7fcf9cd33640 1 -- start start 2023-10-14T18:18:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf9815ad70 0x7fcf98069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 0x7fcf9806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 0x7fcf980ca840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980cad80 con 0x7fcf9815ad70 2023-10-14T18:18:51.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf980cb030 con 0x7fcf980ca3b0 2023-10-14T18:18:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9cd33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c91c0 con 0x7fcf980c9ca0 2023-10-14T18:18:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf9815ad70 0x7fcf98069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcf9815ad70 0x7fcf98069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40792/0 (socket says 172.21.15.29:40792) 2023-10-14T18:18:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 -- 172.21.15.29:0/850152931 learned_addr learned my addr 172.21.15.29:0/850152931 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf95d9b640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 0x7fcf9806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 -- 172.21.15.29:0/850152931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 msgr2=0x7fcf9806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 0x7fcf9806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf9659c640 1 -- 172.21.15.29:0/850152931 >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 msgr2=0x7fcf980ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.473+0000 7fcf96d9d640 1 --2- 172.21.15.29:0/850152931 >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 0x7fcf980ca840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.474+0000 7fcf9659c640 1 --2- 172.21.15.29:0/850152931 >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 0x7fcf980ca840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.474+0000 7fcf9659c640 1 -- 172.21.15.29:0/850152931 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf980c94a0 con 0x7fcf9815ad70 2023-10-14T18:18:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.474+0000 7fcf95d9b640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 0x7fcf9806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.474+0000 7fcf96d9d640 1 --2- 172.21.15.29:0/850152931 >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 0x7fcf980ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.475+0000 7fcf9659c640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf98069da0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cf820 tx=0x7fcf8802b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a843aef34619a507 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.475+0000 7fcf9559a640 1 -- 172.21.15.29:0/850152931 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88031860 con 0x7fcf9815ad70 2023-10-14T18:18:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.475+0000 7fcf9559a640 1 -- 172.21.15.29:0/850152931 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf880319c0 con 0x7fcf9815ad70 2023-10-14T18:18:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.475+0000 7fcf9559a640 1 -- 172.21.15.29:0/850152931 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf88031d50 con 0x7fcf9815ad70 2023-10-14T18:18:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 -- 172.21.15.29:0/850152931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 msgr2=0x7fcf98069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf98069da0 secure :-1 s=READY pgs=2923 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cf820 tx=0x7fcf8802b900 comp rx=0 tx=0).stop 2023-10-14T18:18:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 -- 172.21.15.29:0/850152931 shutdown_connections 2023-10-14T18:18:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/850152931 >> v2:172.21.15.186:3300/0 conn(0x7fcf980ca3b0 0x7fcf980ca840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980c9ca0 0x7fcf9806a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/850152931 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf98069da0 unknown :-1 s=CLOSED pgs=2923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 -- 172.21.15.29:0/850152931 >> 172.21.15.29:0/850152931 conn(0x7fcf980ce9b0 msgr2=0x7fcf980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 -- 172.21.15.29:0/850152931 shutdown_connections 2023-10-14T18:18:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.476+0000 7fcf9cd33640 1 -- 172.21.15.29:0/850152931 wait complete. 2023-10-14T18:18:51.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.478+0000 7fcf9cd33640 1 Processor -- start 2023-10-14T18:18:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.478+0000 7fcf9cd33640 1 -- start start 2023-10-14T18:18:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf980c9ca0 0x7fcf980b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980ca3b0 0x7fcf980b17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980c43e0 con 0x7fcf9815ad70 2023-10-14T18:18:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980b3740 con 0x7fcf980c9ca0 2023-10-14T18:18:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf9cd33640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf980b1ce0 con 0x7fcf980ca3b0 2023-10-14T18:18:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf96d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf96d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40808/0 (socket says 172.21.15.29:40808) 2023-10-14T18:18:51.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.479+0000 7fcf96d9d640 1 -- 172.21.15.29:0/116113588 learned_addr learned my addr 172.21.15.29:0/116113588 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf95d9b640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980ca3b0 0x7fcf980b17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 -- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980ca3b0 msgr2=0x7fcf980b17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980ca3b0 0x7fcf980b17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 -- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf980c9ca0 msgr2=0x7fcf980b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf980c9ca0 0x7fcf980b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 -- 172.21.15.29:0/116113588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf880094d0 con 0x7fcf9815ad70 2023-10-14T18:18:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf96d9d640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000cf20 tx=0x7fcf9000a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000d840 con 0x7fcf9815ad70 2023-10-14T18:18:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf9000d9a0 con 0x7fcf9815ad70 2023-10-14T18:18:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9000dd50 con 0x7fcf9815ad70 2023-10-14T18:18:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.480+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf9814c3b0 con 0x7fcf9815ad70 2023-10-14T18:18:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.481+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf980b3990 con 0x7fcf9815ad70 2023-10-14T18:18:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.482+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf9804e160 con 0x7fcf9815ad70 2023-10-14T18:18:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.483+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcf9001e3b0 con 0x7fcf9815ad70 2023-10-14T18:18:51.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.484+0000 7fcf86ffd640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 0x7fcf7806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:51.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.484+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fcf9008a900 con 0x7fcf9815ad70 2023-10-14T18:18:51.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.484+0000 7fcf9659c640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 0x7fcf7806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:51.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.485+0000 7fcf9659c640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 0x7fcf7806d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf88036000 tx=0x7fcf88030000 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:51.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.488+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcf90055260 con 0x7fcf9815ad70 2023-10-14T18:18:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:51.731+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fcf980ce550 con 0x7fcf9815ad70 2023-10-14T18:18:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.318+0000 7fcf86ffd640 1 -- 172.21.15.29:0/116113588 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "standby_count_wanted", "val": "0"}]=0 v128) v1 ==== 121+0+0 (secure 0 0 0) 0x7fcf90058880 con 0x7fcf9815ad70 2023-10-14T18:18:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.322+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 msgr2=0x7fcf7806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.322+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 0x7fcf7806d570 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fcf88036000 tx=0x7fcf88030000 comp rx=0 tx=0).stop 2023-10-14T18:18:52.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.322+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 msgr2=0x7fcf9814be70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.322+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 secure :-1 s=READY pgs=2924 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000cf20 tx=0x7fcf9000a7d0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 shutdown_connections 2023-10-14T18:18:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fcf7806b0b0 0x7fcf7806d570 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcf9815ad70 0x7fcf9814be70 unknown :-1 s=CLOSED pgs=2924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcf980ca3b0 0x7fcf980b17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 --2- 172.21.15.29:0/116113588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcf980c9ca0 0x7fcf980b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 >> 172.21.15.29:0/116113588 conn(0x7fcf980ce9b0 msgr2=0x7fcf9815c2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 shutdown_connections 2023-10-14T18:18:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.323+0000 7fcf9cd33640 1 -- 172.21.15.29:0/116113588 wait complete. 2023-10-14T18:18:52.346 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:18:52.347 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set backup_fs allow_standby_replay false 2023-10-14T18:18:52.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.577+0000 7f8df8b01640 1 Processor -- start 2023-10-14T18:18:52.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.577+0000 7f8df8b01640 1 -- start start 2023-10-14T18:18:52.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.578+0000 7f8df8b01640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8df40addb0 0x7f8df40caa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df8b01640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df8b01640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8df40cb140 con 0x7f8df40c9000 2023-10-14T18:18:52.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df8b01640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8df4161a50 con 0x7f8df40addb0 2023-10-14T18:18:52.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df8b01640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df406b030 con 0x7f8df406a4b0 2023-10-14T18:18:52.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df2d9d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df259c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8df40addb0 0x7f8df40caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df2d9d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:32972/0 (socket says 172.21.15.29:32972) 2023-10-14T18:18:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.579+0000 7f8df2d9d640 1 -- 172.21.15.29:0/2900283853 learned_addr learned my addr 172.21.15.29:0/2900283853 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.580+0000 7f8df259c640 1 -- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 msgr2=0x7f8df41650d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.580+0000 7f8df259c640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.580+0000 7f8df259c640 1 -- 172.21.15.29:0/2900283853 >> v1:172.21.15.186:6789/0 conn(0x7f8df40c9000 legacy=0x7f8df4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.580+0000 7f8df159a640 1 -- 172.21.15.29:0/2900283853 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3408844743 0 0) 0x7f8df40cb140 con 0x7f8df40c9000 2023-10-14T18:18:52.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.581+0000 7f8df259c640 1 -- 172.21.15.29:0/2900283853 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df406b310 con 0x7f8df40addb0 2023-10-14T18:18:52.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.581+0000 7f8df2d9d640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:18:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.582+0000 7f8df259c640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40caa80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009d90 tx=0x7f8de402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f98b9a5c125b84b3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.583+0000 7f8df159a640 1 -- 172.21.15.29:0/2900283853 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de402cc60 con 0x7f8df40addb0 2023-10-14T18:18:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.583+0000 7f8df159a640 1 -- 172.21.15.29:0/2900283853 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de402cdc0 con 0x7f8df40addb0 2023-10-14T18:18:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.583+0000 7f8df159a640 1 -- 172.21.15.29:0/2900283853 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4034660 con 0x7f8df40addb0 2023-10-14T18:18:52.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.583+0000 7f8df8b01640 1 -- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 msgr2=0x7f8df40caa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.583+0000 7f8df8b01640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40caa80 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f8de4009d90 tx=0x7f8de402beb0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 -- 172.21.15.29:0/2900283853 shutdown_connections 2023-10-14T18:18:52.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df406a4b0 0x7f8df41650d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 --2- 172.21.15.29:0/2900283853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40caa80 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 -- 172.21.15.29:0/2900283853 >> 172.21.15.29:0/2900283853 conn(0x7f8df40ce9b0 msgr2=0x7f8df40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:52.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 -- 172.21.15.29:0/2900283853 shutdown_connections 2023-10-14T18:18:52.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.584+0000 7f8df8b01640 1 -- 172.21.15.29:0/2900283853 wait complete. 2023-10-14T18:18:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.585+0000 7f8df8b01640 1 Processor -- start 2023-10-14T18:18:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.586+0000 7f8df8b01640 1 -- start start 2023-10-14T18:18:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df40c9000 0x7f8df40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df41f6750 con 0x7f8df406a4b0 2023-10-14T18:18:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40b1ed0 con 0x7f8df40addb0 2023-10-14T18:18:52.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df8b01640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8df40b21b0 con 0x7f8df40c9000 2023-10-14T18:18:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40814/0 (socket says 172.21.15.29:40814) 2023-10-14T18:18:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df259c640 1 -- 172.21.15.29:0/1180377496 learned_addr learned my addr 172.21.15.29:0/1180377496 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df2d9d640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df40c9000 0x7f8df40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df1d9b640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df259c640 1 -- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df40c9000 msgr2=0x7f8df40b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.587+0000 7f8df259c640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df40c9000 0x7f8df40b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df259c640 1 -- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 msgr2=0x7f8df40b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df259c640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40b15d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df259c640 1 -- 172.21.15.29:0/1180377496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dec0094d0 con 0x7f8df406a4b0 2023-10-14T18:18:52.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df259c640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f8de4037040 tx=0x7f8de4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4040650 con 0x7f8df406a4b0 2023-10-14T18:18:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8de40407f0 con 0x7f8df406a4b0 2023-10-14T18:18:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de4040ba0 con 0x7f8df406a4b0 2023-10-14T18:18:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de4009a70 con 0x7f8df406a4b0 2023-10-14T18:18:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.588+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df40b2770 con 0x7f8df406a4b0 2023-10-14T18:18:52.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.590+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df406a2d0 con 0x7f8df406a4b0 2023-10-14T18:18:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.591+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8de4040d00 con 0x7f8df406a4b0 2023-10-14T18:18:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.592+0000 7f8deaffd640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 0x7f8dd406d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.592+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f8de40acba0 con 0x7f8df406a4b0 2023-10-14T18:18:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.592+0000 7f8df1d9b640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 0x7f8dd406d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.593+0000 7f8df1d9b640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 0x7f8dd406d540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8df4152be0 tx=0x7f8ddc009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:52.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.596+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8de40776f0 con 0x7f8df406a4b0 2023-10-14T18:18:52.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:52.833+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7f8df404e160 con 0x7f8df406a4b0 2023-10-14T18:18:53.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.338+0000 7f8deaffd640 1 -- 172.21.15.29:0/1180377496 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "backup_fs", "var": "allow_standby_replay", "val": "false"}]=0 v129) v1 ==== 125+0+0 (secure 0 0 0) 0x7f8de407ad10 con 0x7f8df406a4b0 2023-10-14T18:18:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.341+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 msgr2=0x7f8dd406d540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.341+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 0x7f8dd406d540 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f8df4152be0 tx=0x7f8ddc009040 comp rx=0 tx=0).stop 2023-10-14T18:18:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.341+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 msgr2=0x7f8df41f6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.341+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 secure :-1 s=READY pgs=2925 cs=0 l=1 rev1=1 crypto rx=0x7f8de4037040 tx=0x7f8de4004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 shutdown_connections 2023-10-14T18:18:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f8dd406b080 0x7f8dd406d540 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8df40c9000 0x7f8df40b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8df40addb0 0x7f8df40b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 --2- 172.21.15.29:0/1180377496 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8df406a4b0 0x7f8df41f6210 unknown :-1 s=CLOSED pgs=2925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 >> 172.21.15.29:0/1180377496 conn(0x7f8df40ce9b0 msgr2=0x7f8df40cd440 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:53.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 shutdown_connections 2023-10-14T18:18:53.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.342+0000 7f8df8b01640 1 -- 172.21.15.29:0/1180377496 wait complete. 2023-10-14T18:18:53.363 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'backup_fs' 2023-10-14T18:18:53.364 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:53.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.578+0000 7fc1314bb640 1 Processor -- start 2023-10-14T18:18:53.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.578+0000 7fc1314bb640 1 -- start start 2023-10-14T18:18:53.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc1314bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c06c040 0x7fc12c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc1314bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0cda80 0x7fc12c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc1314bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc12c06c960 con 0x7fc12c0ad970 2023-10-14T18:18:53.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc1314bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc12c15b9d0 con 0x7fc12c06c040 2023-10-14T18:18:53.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc1314bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c15bcb0 con 0x7fc12c0cda80 2023-10-14T18:18:53.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.580+0000 7fc12bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c06c040 0x7fc12c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:53.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc12c06c040 0x7fc12c06c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40818/0 (socket says 172.21.15.29:40818) 2023-10-14T18:18:53.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 -- 172.21.15.29:0/1464615027 learned_addr learned my addr 172.21.15.29:0/1464615027 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:53.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 -- 172.21.15.29:0/1464615027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0cda80 msgr2=0x7fc12c15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:53.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 --2- 172.21.15.29:0/1464615027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0cda80 0x7fc12c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 -- 172.21.15.29:0/1464615027 >> v1:172.21.15.186:6789/0 conn(0x7fc12c0ad970 legacy=0x7fc12c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:53.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.581+0000 7fc12bfff640 1 -- 172.21.15.29:0/1464615027 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c15bf90 con 0x7fc12c06c040 2023-10-14T18:18:53.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.582+0000 7fc12bfff640 1 --2- 172.21.15.29:0/1464615027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c06c040 0x7fc12c06c420 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009870 tx=0x7fc11c02b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff5b1cfb70773d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:53.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.582+0000 7fc12affd640 1 -- 172.21.15.29:0/1464615027 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c031640 con 0x7fc12c06c040 2023-10-14T18:18:53.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc12affd640 1 -- 172.21.15.29:0/1464615027 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc11c0317a0 con 0x7fc12c06c040 2023-10-14T18:18:53.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc12affd640 1 -- 172.21.15.29:0/1464615027 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc11c031b30 con 0x7fc12c06c040 2023-10-14T18:18:53.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 -- 172.21.15.29:0/1464615027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c06c040 msgr2=0x7fc12c06c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 --2- 172.21.15.29:0/1464615027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c06c040 0x7fc12c06c420 secure :-1 s=READY pgs=2926 cs=0 l=1 rev1=1 crypto rx=0x7fc11c009870 tx=0x7fc11c02b8f0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 -- 172.21.15.29:0/1464615027 shutdown_connections 2023-10-14T18:18:53.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 --2- 172.21.15.29:0/1464615027 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c0cda80 0x7fc12c15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 --2- 172.21.15.29:0/1464615027 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c06c040 0x7fc12c06c420 unknown :-1 s=CLOSED pgs=2926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 -- 172.21.15.29:0/1464615027 >> 172.21.15.29:0/1464615027 conn(0x7fc12c0ce6a0 msgr2=0x7fc12c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:53.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 -- 172.21.15.29:0/1464615027 shutdown_connections 2023-10-14T18:18:53.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.583+0000 7fc1314bb640 1 -- 172.21.15.29:0/1464615027 wait complete. 2023-10-14T18:18:53.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.585+0000 7fc1314bb640 1 Processor -- start 2023-10-14T18:18:53.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.585+0000 7fc1314bb640 1 -- start start 2023-10-14T18:18:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c06c040 0x7fc12c148990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0cda80 0x7fc12c0c4850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0c7510 con 0x7fc12c0ad970 2023-10-14T18:18:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0c6900 con 0x7fc12c0cda80 2023-10-14T18:18:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.586+0000 7fc1314bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0c6be0 con 0x7fc12c06c040 2023-10-14T18:18:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40824/0 (socket says 172.21.15.29:40824) 2023-10-14T18:18:53.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2220374524 learned_addr learned my addr 172.21.15.29:0/2220374524 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12bfff640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c06c040 0x7fc12c148990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc130cba640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0cda80 0x7fc12c0c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c06c040 msgr2=0x7fc12c148990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c06c040 0x7fc12c148990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0cda80 msgr2=0x7fc12c0c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0cda80 0x7fc12c0c4850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 -- 172.21.15.29:0/2220374524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c0094d0 con 0x7fc12c0ad970 2023-10-14T18:18:53.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.587+0000 7fc12b7fe640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fc12000c400 tx=0x7fc12000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:53.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.588+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120011790 con 0x7fc12c0ad970 2023-10-14T18:18:53.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.588+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc120011930 con 0x7fc12c0ad970 2023-10-14T18:18:53.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.588+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc120011ce0 con 0x7fc12c0ad970 2023-10-14T18:18:53.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.588+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c0c6ec0 con 0x7fc12c0ad970 2023-10-14T18:18:53.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.588+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c156100 con 0x7fc12c0ad970 2023-10-14T18:18:53.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.589+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c06b330 con 0x7fc12c0ad970 2023-10-14T18:18:53.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.590+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc12001c460 con 0x7fc12c0ad970 2023-10-14T18:18:53.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.591+0000 7fc128ff9640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 0x7fc11806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:53.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.591+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fc120089d70 con 0x7fc12c0ad970 2023-10-14T18:18:53.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.591+0000 7fc12bfff640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 0x7fc11806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:53.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.592+0000 7fc12bfff640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 0x7fc11806d570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc11c036000 tx=0x7fc11c03c040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:53.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.596+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc120054710 con 0x7fc12c0ad970 2023-10-14T18:18:53.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.829+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc12c04e9b0 con 0x7fc12c0ad970 2023-10-14T18:18:53.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.830+0000 7fc128ff9640 1 -- 172.21.15.29:0/2220374524 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7fc120057d30 con 0x7fc12c0ad970 2023-10-14T18:18:53.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:53.833 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:18:53.834 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:18:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 msgr2=0x7fc11806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 0x7fc11806d570 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc11c036000 tx=0x7fc11c03c040 comp rx=0 tx=0).stop 2023-10-14T18:18:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 msgr2=0x7fc12c0ade50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 secure :-1 s=READY pgs=2927 cs=0 l=1 rev1=1 crypto rx=0x7fc12000c400 tx=0x7fc12000c8c0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 shutdown_connections 2023-10-14T18:18:53.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fc11806b0b0 0x7fc11806d570 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc12c0cda80 0x7fc12c0c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc12c0ad970 0x7fc12c0ade50 unknown :-1 s=CLOSED pgs=2927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 --2- 172.21.15.29:0/2220374524 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc12c06c040 0x7fc12c148990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:53.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 >> 172.21.15.29:0/2220374524 conn(0x7fc12c0ce6a0 msgr2=0x7fc12c159c30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:53.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.833+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 shutdown_connections 2023-10-14T18:18:53.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:53.834+0000 7fc1314bb640 1 -- 172.21.15.29:0/2220374524 wait complete. 2023-10-14T18:18:53.854 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:18:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.076+0000 7f88c90b4640 1 Processor -- start 2023-10-14T18:18:54.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.076+0000 7f88c90b4640 1 -- start start 2023-10-14T18:18:54.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c90b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c40c9060 0x7f88c406f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c90b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f88c406c9e0 0x7f88c406f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c90b4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88c40ca560 con 0x7f88c40c9440 2023-10-14T18:18:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c90b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40c9950 con 0x7f88c40c9060 2023-10-14T18:18:54.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c90b4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88c40c9c30 con 0x7f88c406c9e0 2023-10-14T18:18:54.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c40c9060 0x7f88c406f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c3fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f88c40c9060 0x7f88c406f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40834/0 (socket says 172.21.15.29:40834) 2023-10-14T18:18:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.078+0000 7f88c3fff640 1 -- 172.21.15.29:0/2435807105 learned_addr learned my addr 172.21.15.29:0/2435807105 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.079+0000 7f88bbfff640 1 --2- 172.21.15.29:0/2435807105 >> v2:172.21.15.186:3300/0 conn(0x7f88c406c9e0 0x7f88c406f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.079+0000 7f88c3fff640 1 -- 172.21.15.29:0/2435807105 >> v2:172.21.15.186:3300/0 conn(0x7f88c406c9e0 msgr2=0x7f88c406f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.079+0000 7f88c3fff640 1 --2- 172.21.15.29:0/2435807105 >> v2:172.21.15.186:3300/0 conn(0x7f88c406c9e0 0x7f88c406f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.079+0000 7f88c3fff640 1 -- 172.21.15.29:0/2435807105 >> v1:172.21.15.186:6789/0 conn(0x7f88c40c9440 legacy=0x7f88c4164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.079+0000 7f88c3fff640 1 -- 172.21.15.29:0/2435807105 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88c40c9f10 con 0x7f88c40c9060 2023-10-14T18:18:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.080+0000 7f88c3fff640 1 --2- 172.21.15.29:0/2435807105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c40c9060 0x7f88c406f440 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f88c4164cb0 tx=0x7f88b402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64910f4e4cc505f2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c37fe640 1 -- 172.21.15.29:0/2435807105 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4031870 con 0x7f88c40c9060 2023-10-14T18:18:54.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c37fe640 1 -- 172.21.15.29:0/2435807105 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b40319d0 con 0x7f88c40c9060 2023-10-14T18:18:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c37fe640 1 -- 172.21.15.29:0/2435807105 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4031d60 con 0x7f88c40c9060 2023-10-14T18:18:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 -- 172.21.15.29:0/2435807105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c40c9060 msgr2=0x7f88c406f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 --2- 172.21.15.29:0/2435807105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c40c9060 0x7f88c406f440 secure :-1 s=READY pgs=2928 cs=0 l=1 rev1=1 crypto rx=0x7f88c4164cb0 tx=0x7f88b402b910 comp rx=0 tx=0).stop 2023-10-14T18:18:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 -- 172.21.15.29:0/2435807105 shutdown_connections 2023-10-14T18:18:54.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 --2- 172.21.15.29:0/2435807105 >> v2:172.21.15.186:3300/0 conn(0x7f88c406c9e0 0x7f88c406f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 --2- 172.21.15.29:0/2435807105 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c40c9060 0x7f88c406f440 unknown :-1 s=CLOSED pgs=2928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 -- 172.21.15.29:0/2435807105 >> 172.21.15.29:0/2435807105 conn(0x7f88c40ce6a0 msgr2=0x7f88c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 -- 172.21.15.29:0/2435807105 shutdown_connections 2023-10-14T18:18:54.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.081+0000 7f88c90b4640 1 -- 172.21.15.29:0/2435807105 wait complete. 2023-10-14T18:18:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.083+0000 7f88c90b4640 1 Processor -- start 2023-10-14T18:18:54.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.083+0000 7f88c90b4640 1 -- start start 2023-10-14T18:18:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c40c9060 0x7f88c40b2310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c40c9440 0x7f88c41f9fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b3cb0 con 0x7f88c406c9e0 2023-10-14T18:18:54.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88c41f6890 con 0x7f88c40c9060 2023-10-14T18:18:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.084+0000 7f88c90b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88c40b2c10 con 0x7f88c40c9440 2023-10-14T18:18:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40850/0 (socket says 172.21.15.29:40850) 2023-10-14T18:18:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 -- 172.21.15.29:0/3913623398 learned_addr learned my addr 172.21.15.29:0/3913623398 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:54.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88bbfff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c40c9060 0x7f88c40b2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 -- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c40c9440 msgr2=0x7f88c41f9fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c40c9440 0x7f88c41f9fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 -- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c40c9060 msgr2=0x7f88c40b2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c40c9060 0x7f88c40b2310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 -- 172.21.15.29:0/3913623398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b40094d0 con 0x7f88c406c9e0 2023-10-14T18:18:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.085+0000 7f88c3fff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000c00 tx=0x7f88b40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.086+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b403abe0 con 0x7f88c406c9e0 2023-10-14T18:18:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.086+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b403ad40 con 0x7f88c406c9e0 2023-10-14T18:18:54.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.086+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88b4044610 con 0x7f88c406c9e0 2023-10-14T18:18:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.086+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88c40b2ef0 con 0x7f88c406c9e0 2023-10-14T18:18:54.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.086+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88c40b34b0 con 0x7f88c406c9e0 2023-10-14T18:18:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.087+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c404e9b0 con 0x7f88c406c9e0 2023-10-14T18:18:54.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.088+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f88b404d450 con 0x7f88c406c9e0 2023-10-14T18:18:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.089+0000 7f88c17fa640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 0x7f88a406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.089+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f88b402d040 con 0x7f88c406c9e0 2023-10-14T18:18:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.089+0000 7f88bbfff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 0x7f88a406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.090+0000 7f88bbfff640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 0x7f88a406d570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88c40af790 tx=0x7f88ac008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.093+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f88b4076e00 con 0x7f88c406c9e0 2023-10-14T18:18:54.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.293+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c40b4620 con 0x7f88c406c9e0 2023-10-14T18:18:54.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.296+0000 7f88c17fa640 1 -- 172.21.15.29:0/3913623398 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v254) v1 ==== 74+0+20361 (secure 0 0 0) 0x7f88b407a420 con 0x7f88c406c9e0 2023-10-14T18:18:54.297 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:54.297 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":254,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:18:51.282650+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:18:54.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.298+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 msgr2=0x7f88a406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.298+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 0x7f88a406d570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f88c40af790 tx=0x7f88ac008040 comp rx=0 tx=0).stop 2023-10-14T18:18:54.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.298+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 msgr2=0x7f88c41f6350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.298+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 secure :-1 s=READY pgs=2929 cs=0 l=1 rev1=1 crypto rx=0x7f88b4000c00 tx=0x7f88b40046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 shutdown_connections 2023-10-14T18:18:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f88a406b0b0 0x7f88a406d570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f88c40c9440 0x7f88c41f9fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f88c40c9060 0x7f88c40b2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 --2- 172.21.15.29:0/3913623398 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f88c406c9e0 0x7f88c41f6350 unknown :-1 s=CLOSED pgs=2929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 >> 172.21.15.29:0/3913623398 conn(0x7f88c40ce6a0 msgr2=0x7f88c406afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:54.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 shutdown_connections 2023-10-14T18:18:54.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.299+0000 7f88c90b4640 1 -- 172.21.15.29:0/3913623398 wait complete. 2023-10-14T18:18:54.320 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:18:54.321 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:18:54.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.547+0000 7f892ffff640 1 Processor -- start 2023-10-14T18:18:54.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.548+0000 7f892ffff640 1 -- start start 2023-10-14T18:18:54.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f89300cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892ffff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f89300ccfb0 0x7f89300cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892ffff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f893015b9d0 con 0x7f893006c040 2023-10-14T18:18:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892ffff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f893015bcb0 con 0x7f89300ccfb0 2023-10-14T18:18:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893015bf90 con 0x7f89300ad970 2023-10-14T18:18:54.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892effd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f893006c040 0x7f893006c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:40992/0 (socket says 172.21.15.29:40992) 2023-10-14T18:18:54.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892effd640 1 -- 172.21.15.29:0/2696055977 learned_addr learned my addr 172.21.15.29:0/2696055977 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:54.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346481963 0 0) 0x7f893015b9d0 con 0x7f893006c040 2023-10-14T18:18:54.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892f7fe640 1 --2- 172.21.15.29:0/2696055977 >> v2:172.21.15.186:3300/0 conn(0x7f89300ccfb0 0x7f89300cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.550+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f891c003520 con 0x7f893006c040 2023-10-14T18:18:54.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f8927fff640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f89300cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 331940101 0 0) 0x7f891c003520 con 0x7f893006c040 2023-10-14T18:18:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 msgr2=0x7f89300cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f89300cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 >> v2:172.21.15.186:3300/0 conn(0x7f89300ccfb0 msgr2=0x7f89300cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 --2- 172.21.15.29:0/2696055977 >> v2:172.21.15.186:3300/0 conn(0x7f89300ccfb0 0x7f89300cd440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89300cc270 con 0x7f893006c040 2023-10-14T18:18:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8920001670 con 0x7f893006c040 2023-10-14T18:18:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 >> v1:172.21.15.29:6789/0 conn(0x7f893006c040 legacy=0x7f893006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f891c003970 0x7f891c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f891c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f891c004d00 0x7f891c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893015bcb0 con 0x7f891c004d00 2023-10-14T18:18:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893015bf90 con 0x7f891c003970 2023-10-14T18:18:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f891c00ba80 con 0x7f89300ad970 2023-10-14T18:18:54.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.551+0000 7f892f7fe640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f891c004d00 0x7f891c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892effd640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f891c003970 0x7f891c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f8927fff640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f891c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892f7fe640 1 -- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 msgr2=0x7f891c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892f7fe640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f89300ad970 0x7f891c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892f7fe640 1 -- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f891c003970 msgr2=0x7f891c003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892f7fe640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f891c003970 0x7f891c003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.552+0000 7f892f7fe640 1 -- 172.21.15.29:0/2696055977 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f891c003520 con 0x7f891c004d00 2023-10-14T18:18:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.553+0000 7f892f7fe640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f891c004d00 0x7f891c005180 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f8928007360 tx=0x7f891c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebd289d1e76610bc server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.553+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892802ccf0 con 0x7f891c004d00 2023-10-14T18:18:54.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.553+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892802ce50 con 0x7f891c004d00 2023-10-14T18:18:54.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.553+0000 7f892e7fc640 1 -- 172.21.15.29:0/2696055977 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8928034660 con 0x7f891c004d00 2023-10-14T18:18:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 -- 172.21.15.29:0/2696055977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f891c004d00 msgr2=0x7f891c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 --2- 172.21.15.29:0/2696055977 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f891c004d00 0x7f891c005180 secure :-1 s=READY pgs=2931 cs=0 l=1 rev1=1 crypto rx=0x7f8928007360 tx=0x7f891c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892f7fe640 1 -- 172.21.15.29:0/2696055977 reap_dead start 2023-10-14T18:18:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 -- 172.21.15.29:0/2696055977 shutdown_connections 2023-10-14T18:18:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 -- 172.21.15.29:0/2696055977 >> 172.21.15.29:0/2696055977 conn(0x7f89300ce6a0 msgr2=0x7f89300ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 -- 172.21.15.29:0/2696055977 shutdown_connections 2023-10-14T18:18:54.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.554+0000 7f892ffff640 1 -- 172.21.15.29:0/2696055977 wait complete. 2023-10-14T18:18:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.556+0000 7f892ffff640 1 Processor -- start 2023-10-14T18:18:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.556+0000 7f892ffff640 1 -- start start 2023-10-14T18:18:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891c004d00 0x7f89301f6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f893015c6a0 0x7f893014d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893014e6e0 con 0x7f893015cb20 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893014e120 con 0x7f893015c6a0 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892ffff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f893014e400 con 0x7f891c004d00 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40866/0 (socket says 172.21.15.29:40866) 2023-10-14T18:18:54.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.557+0000 7f892f7fe640 1 -- 172.21.15.29:0/678852912 learned_addr learned my addr 172.21.15.29:0/678852912 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892effd640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891c004d00 0x7f89301f6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 -- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891c004d00 msgr2=0x7f89301f6330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f8927fff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f893015c6a0 0x7f893014d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891c004d00 0x7f89301f6330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 -- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f893015c6a0 msgr2=0x7f893014d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f893015c6a0 0x7f893014d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 -- 172.21.15.29:0/678852912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89301f5b30 con 0x7f893015cb20 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.558+0000 7f892f7fe640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f892802dfd0 tx=0x7f8928004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.559+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892803e400 con 0x7f893015cb20 2023-10-14T18:18:54.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.559+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f892803e560 con 0x7f893015cb20 2023-10-14T18:18:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.559+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f892803e8f0 con 0x7f893015cb20 2023-10-14T18:18:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.559+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89301f5e10 con 0x7f893015cb20 2023-10-14T18:18:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.559+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893014dcf0 con 0x7f893015cb20 2023-10-14T18:18:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.560+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893006b330 con 0x7f893015cb20 2023-10-14T18:18:54.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.561+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f892803ea50 con 0x7f893015cb20 2023-10-14T18:18:54.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.562+0000 7f89277fe640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 0x7f891006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.562+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f89280ac3e0 con 0x7f893015cb20 2023-10-14T18:18:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.562+0000 7f892effd640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 0x7f891006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.563+0000 7f892effd640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 0x7f891006d570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8920001050 tx=0x7f8920009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:54.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.567+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8928076f10 con 0x7f893015cb20 2023-10-14T18:18:54.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.765+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f89300cc550 con 0x7f891006b0b0 2023-10-14T18:18:54.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.787+0000 7f89277fe640 1 -- 172.21.15.29:0/678852912 <== mgr.7463 v2:172.21.15.29:6832/4275371095 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526184 (secure 0 0 0) 0x7f89200109a0 con 0x7f891006b0b0 2023-10-14T18:18:54.794 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:54.799 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:18:54.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 msgr2=0x7f891006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 0x7f891006d570 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8920001050 tx=0x7f8920009040 comp rx=0 tx=0).stop 2023-10-14T18:18:54.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 msgr2=0x7f893014aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:54.801 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 secure :-1 s=READY pgs=2932 cs=0 l=1 rev1=1 crypto rx=0x7f892802dfd0 tx=0x7f8928004ab0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 shutdown_connections 2023-10-14T18:18:54.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f891006b0b0 0x7f891006d570 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f893015cb20 0x7f893014aa20 unknown :-1 s=CLOSED pgs=2932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f893015c6a0 0x7f893014d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 --2- 172.21.15.29:0/678852912 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f891c004d00 0x7f89301f6330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:54.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 >> 172.21.15.29:0/678852912 conn(0x7f89300ce6a0 msgr2=0x7f893015dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:54.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 shutdown_connections 2023-10-14T18:18:54.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:54.798+0000 7f892ffff640 1 -- 172.21.15.29:0/678852912 wait complete. 2023-10-14T18:18:54.820 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-14T18:18:54.430278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":360,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428348,"kb_used_data":26248,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546372,"statfs":{"total":773094113280,"available":772655484928,"internally_reserved":0,"allocated":26877952,"data_stored":13961754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001849"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.292498+0000","last_change":"2023-10-14T18:18:50.322788+0000","last_active":"2023-10-14T18:18:51.292498+0000","last_peered":"2023-10-14T18:18:51.292498+0000","last_clean":"2023-10-14T18:18:51.292498+0000","last_became_active":"2023-10-14T18:18:50.322523+0000","last_became_peered":"2023-10-14T18:18:50.322523+0000","last_unstale":"2023-10-14T18:18:51.292498+0000","last_undegraded":"2023-10-14T18:18:51.292498+0000","last_fullsized":"2023-10-14T18:18:51.292498+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:26.854472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380414+0000","last_change":"2023-10-14T18:18:49.317435+0000","last_active":"2023-10-14T18:18:51.380414+0000","last_peered":"2023-10-14T18:18:51.380414+0000","last_clean":"2023-10-14T18:18:51.380414+0000","last_became_active":"2023-10-14T18:18:49.317061+0000","last_became_peered":"2023-10-14T18:18:49.317061+0000","last_unstale":"2023-10-14T18:18:51.380414+0000","last_undegraded":"2023-10-14T18:18:51.380414+0000","last_fullsized":"2023-10-14T18:18:51.380414+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:03.543371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298869+0000","last_change":"2023-10-14T18:18:35.681689+0000","last_active":"2023-10-14T18:18:50.298869+0000","last_peered":"2023-10-14T18:18:50.298869+0000","last_clean":"2023-10-14T18:18:50.298869+0000","last_became_active":"2023-10-14T18:18:35.681412+0000","last_became_peered":"2023-10-14T18:18:35.681412+0000","last_unstale":"2023-10-14T18:18:50.298869+0000","last_undegraded":"2023-10-14T18:18:50.298869+0000","last_fullsized":"2023-10-14T18:18:50.298869+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:30:00.717606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380166+0000","last_change":"2023-10-14T18:18:50.313424+0000","last_active":"2023-10-14T18:18:51.380166+0000","last_peered":"2023-10-14T18:18:51.380166+0000","last_clean":"2023-10-14T18:18:51.380166+0000","last_became_active":"2023-10-14T18:18:50.312904+0000","last_became_peered":"2023-10-14T18:18:50.312904+0000","last_unstale":"2023-10-14T18:18:51.380166+0000","last_undegraded":"2023-10-14T18:18:51.380166+0000","last_fullsized":"2023-10-14T18:18:51.380166+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:15.685379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380776+0000","last_change":"2023-10-14T18:18:49.305084+0000","last_active":"2023-10-14T18:18:51.380776+0000","last_peered":"2023-10-14T18:18:51.380776+0000","last_clean":"2023-10-14T18:18:51.380776+0000","last_became_active":"2023-10-14T18:18:49.304769+0000","last_became_peered":"2023-10-14T18:18:49.304769+0000","last_unstale":"2023-10-14T18:18:51.380776+0000","last_undegraded":"2023-10-14T18:18:51.380776+0000","last_fullsized":"2023-10-14T18:18:51.380776+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:20.673412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298852+0000","last_change":"2023-10-14T18:18:35.683822+0000","last_active":"2023-10-14T18:18:50.298852+0000","last_peered":"2023-10-14T18:18:50.298852+0000","last_clean":"2023-10-14T18:18:50.298852+0000","last_became_active":"2023-10-14T18:18:35.683492+0000","last_became_peered":"2023-10-14T18:18:35.683492+0000","last_unstale":"2023-10-14T18:18:50.298852+0000","last_undegraded":"2023-10-14T18:18:50.298852+0000","last_fullsized":"2023-10-14T18:18:50.298852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:15.955276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380987+0000","last_change":"2023-10-14T18:18:50.306275+0000","last_active":"2023-10-14T18:18:51.380987+0000","last_peered":"2023-10-14T18:18:51.380987+0000","last_clean":"2023-10-14T18:18:51.380987+0000","last_became_active":"2023-10-14T18:18:50.305978+0000","last_became_peered":"2023-10-14T18:18:50.305978+0000","last_unstale":"2023-10-14T18:18:51.380987+0000","last_undegraded":"2023-10-14T18:18:51.380987+0000","last_fullsized":"2023-10-14T18:18:51.380987+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:17.252654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377710+0000","last_change":"2023-10-14T18:18:49.315666+0000","last_active":"2023-10-14T18:18:51.377710+0000","last_peered":"2023-10-14T18:18:51.377710+0000","last_clean":"2023-10-14T18:18:51.377710+0000","last_became_active":"2023-10-14T18:18:49.314033+0000","last_became_peered":"2023-10-14T18:18:49.314033+0000","last_unstale":"2023-10-14T18:18:51.377710+0000","last_undegraded":"2023-10-14T18:18:51.377710+0000","last_fullsized":"2023-10-14T18:18:51.377710+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:39.882042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.323727+0000","last_change":"2023-10-14T18:18:35.668307+0000","last_active":"2023-10-14T18:18:50.323727+0000","last_peered":"2023-10-14T18:18:50.323727+0000","last_clean":"2023-10-14T18:18:50.323727+0000","last_became_active":"2023-10-14T18:18:35.667786+0000","last_became_peered":"2023-10-14T18:18:35.667786+0000","last_unstale":"2023-10-14T18:18:50.323727+0000","last_undegraded":"2023-10-14T18:18:50.323727+0000","last_fullsized":"2023-10-14T18:18:50.323727+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:51.991520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380613+0000","last_change":"2023-10-14T18:18:50.323319+0000","last_active":"2023-10-14T18:18:51.380613+0000","last_peered":"2023-10-14T18:18:51.380613+0000","last_clean":"2023-10-14T18:18:51.380613+0000","last_became_active":"2023-10-14T18:18:50.323062+0000","last_became_peered":"2023-10-14T18:18:50.323062+0000","last_unstale":"2023-10-14T18:18:51.380613+0000","last_undegraded":"2023-10-14T18:18:51.380613+0000","last_fullsized":"2023-10-14T18:18:51.380613+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:23:29.332417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379784+0000","last_change":"2023-10-14T18:18:49.309269+0000","last_active":"2023-10-14T18:18:51.379784+0000","last_peered":"2023-10-14T18:18:51.379784+0000","last_clean":"2023-10-14T18:18:51.379784+0000","last_became_active":"2023-10-14T18:18:49.308989+0000","last_became_peered":"2023-10-14T18:18:49.308989+0000","last_unstale":"2023-10-14T18:18:51.379784+0000","last_undegraded":"2023-10-14T18:18:51.379784+0000","last_fullsized":"2023-10-14T18:18:51.379784+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:20.673412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.302743+0000","last_change":"2023-10-14T18:18:35.663594+0000","last_active":"2023-10-14T18:18:50.302743+0000","last_peered":"2023-10-14T18:18:50.302743+0000","last_clean":"2023-10-14T18:18:50.302743+0000","last_became_active":"2023-10-14T18:18:35.663236+0000","last_became_peered":"2023-10-14T18:18:35.663236+0000","last_unstale":"2023-10-14T18:18:50.302743+0000","last_undegraded":"2023-10-14T18:18:50.302743+0000","last_fullsized":"2023-10-14T18:18:50.302743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:56.542420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299552+0000","last_change":"2023-10-14T18:18:50.316200+0000","last_active":"2023-10-14T18:18:51.299552+0000","last_peered":"2023-10-14T18:18:51.299552+0000","last_clean":"2023-10-14T18:18:51.299552+0000","last_became_active":"2023-10-14T18:18:50.314765+0000","last_became_peered":"2023-10-14T18:18:50.314765+0000","last_unstale":"2023-10-14T18:18:51.299552+0000","last_undegraded":"2023-10-14T18:18:51.299552+0000","last_fullsized":"2023-10-14T18:18:51.299552+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:34:05.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382176+0000","last_change":"2023-10-14T18:18:49.317710+0000","last_active":"2023-10-14T18:18:51.382176+0000","last_peered":"2023-10-14T18:18:51.382176+0000","last_clean":"2023-10-14T18:18:51.382176+0000","last_became_active":"2023-10-14T18:18:49.317378+0000","last_became_peered":"2023-10-14T18:18:49.317378+0000","last_unstale":"2023-10-14T18:18:51.382176+0000","last_undegraded":"2023-10-14T18:18:51.382176+0000","last_fullsized":"2023-10-14T18:18:51.382176+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:44.007835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309836+0000","last_change":"2023-10-14T18:18:35.670764+0000","last_active":"2023-10-14T18:18:50.309836+0000","last_peered":"2023-10-14T18:18:50.309836+0000","last_clean":"2023-10-14T18:18:50.309836+0000","last_became_active":"2023-10-14T18:18:35.670354+0000","last_became_peered":"2023-10-14T18:18:35.670354+0000","last_unstale":"2023-10-14T18:18:50.309836+0000","last_undegraded":"2023-10-14T18:18:50.309836+0000","last_fullsized":"2023-10-14T18:18:50.309836+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:43.950584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.296441+0000","last_change":"2023-10-14T18:18:50.313891+0000","last_active":"2023-10-14T18:18:51.296441+0000","last_peered":"2023-10-14T18:18:51.296441+0000","last_clean":"2023-10-14T18:18:51.296441+0000","last_became_active":"2023-10-14T18:18:50.313433+0000","last_became_peered":"2023-10-14T18:18:50.313433+0000","last_unstale":"2023-10-14T18:18:51.296441+0000","last_undegraded":"2023-10-14T18:18:51.296441+0000","last_fullsized":"2023-10-14T18:18:51.296441+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:35.969124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299167+0000","last_change":"2023-10-14T18:18:49.312323+0000","last_active":"2023-10-14T18:18:51.299167+0000","last_peered":"2023-10-14T18:18:51.299167+0000","last_clean":"2023-10-14T18:18:51.299167+0000","last_became_active":"2023-10-14T18:18:49.311700+0000","last_became_peered":"2023-10-14T18:18:49.311700+0000","last_unstale":"2023-10-14T18:18:51.299167+0000","last_undegraded":"2023-10-14T18:18:51.299167+0000","last_fullsized":"2023-10-14T18:18:51.299167+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:24.381483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300609+0000","last_change":"2023-10-14T18:18:35.682936+0000","last_active":"2023-10-14T18:18:50.300609+0000","last_peered":"2023-10-14T18:18:50.300609+0000","last_clean":"2023-10-14T18:18:50.300609+0000","last_became_active":"2023-10-14T18:18:35.682060+0000","last_became_peered":"2023-10-14T18:18:35.682060+0000","last_unstale":"2023-10-14T18:18:50.300609+0000","last_undegraded":"2023-10-14T18:18:50.300609+0000","last_fullsized":"2023-10-14T18:18:50.300609+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:31.696209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383878+0000","last_change":"2023-10-14T18:18:50.331470+0000","last_active":"2023-10-14T18:18:51.383878+0000","last_peered":"2023-10-14T18:18:51.383878+0000","last_clean":"2023-10-14T18:18:51.383878+0000","last_became_active":"2023-10-14T18:18:50.330696+0000","last_became_peered":"2023-10-14T18:18:50.330696+0000","last_unstale":"2023-10-14T18:18:51.383878+0000","last_undegraded":"2023-10-14T18:18:51.383878+0000","last_fullsized":"2023-10-14T18:18:51.383878+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:26.470083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297285+0000","last_change":"2023-10-14T18:18:49.305206+0000","last_active":"2023-10-14T18:18:51.297285+0000","last_peered":"2023-10-14T18:18:51.297285+0000","last_clean":"2023-10-14T18:18:51.297285+0000","last_became_active":"2023-10-14T18:18:49.304926+0000","last_became_peered":"2023-10-14T18:18:49.304926+0000","last_unstale":"2023-10-14T18:18:51.297285+0000","last_undegraded":"2023-10-14T18:18:51.297285+0000","last_fullsized":"2023-10-14T18:18:51.297285+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:20.415692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305731+0000","last_change":"2023-10-14T18:18:35.676700+0000","last_active":"2023-10-14T18:18:50.305731+0000","last_peered":"2023-10-14T18:18:50.305731+0000","last_clean":"2023-10-14T18:18:50.305731+0000","last_became_active":"2023-10-14T18:18:35.676423+0000","last_became_peered":"2023-10-14T18:18:35.676423+0000","last_unstale":"2023-10-14T18:18:50.305731+0000","last_undegraded":"2023-10-14T18:18:50.305731+0000","last_fullsized":"2023-10-14T18:18:50.305731+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378514+0000","last_change":"2023-10-14T18:18:50.317687+0000","last_active":"2023-10-14T18:18:51.378514+0000","last_peered":"2023-10-14T18:18:51.378514+0000","last_clean":"2023-10-14T18:18:51.378514+0000","last_became_active":"2023-10-14T18:18:50.316949+0000","last_became_peered":"2023-10-14T18:18:50.316949+0000","last_unstale":"2023-10-14T18:18:51.378514+0000","last_undegraded":"2023-10-14T18:18:51.378514+0000","last_fullsized":"2023-10-14T18:18:51.378514+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:14.708984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379514+0000","last_change":"2023-10-14T18:18:49.308266+0000","last_active":"2023-10-14T18:18:51.379514+0000","last_peered":"2023-10-14T18:18:51.379514+0000","last_clean":"2023-10-14T18:18:51.379514+0000","last_became_active":"2023-10-14T18:18:49.306582+0000","last_became_peered":"2023-10-14T18:18:49.306582+0000","last_unstale":"2023-10-14T18:18:51.379514+0000","last_undegraded":"2023-10-14T18:18:51.379514+0000","last_fullsized":"2023-10-14T18:18:51.379514+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:09.311289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293548+0000","last_change":"2023-10-14T18:18:35.664633+0000","last_active":"2023-10-14T18:18:50.293548+0000","last_peered":"2023-10-14T18:18:50.293548+0000","last_clean":"2023-10-14T18:18:50.293548+0000","last_became_active":"2023-10-14T18:18:35.664248+0000","last_became_peered":"2023-10-14T18:18:35.664248+0000","last_unstale":"2023-10-14T18:18:50.293548+0000","last_undegraded":"2023-10-14T18:18:50.293548+0000","last_fullsized":"2023-10-14T18:18:50.293548+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:21.318470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375837+0000","last_change":"2023-10-14T18:18:50.305367+0000","last_active":"2023-10-14T18:18:51.375837+0000","last_peered":"2023-10-14T18:18:51.375837+0000","last_clean":"2023-10-14T18:18:51.375837+0000","last_became_active":"2023-10-14T18:18:50.305092+0000","last_became_peered":"2023-10-14T18:18:50.305092+0000","last_unstale":"2023-10-14T18:18:51.375837+0000","last_undegraded":"2023-10-14T18:18:51.375837+0000","last_fullsized":"2023-10-14T18:18:51.375837+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:10.317916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293635+0000","last_change":"2023-10-14T18:18:49.310346+0000","last_active":"2023-10-14T18:18:51.293635+0000","last_peered":"2023-10-14T18:18:51.293635+0000","last_clean":"2023-10-14T18:18:51.293635+0000","last_became_active":"2023-10-14T18:18:49.310066+0000","last_became_peered":"2023-10-14T18:18:49.310066+0000","last_unstale":"2023-10-14T18:18:51.293635+0000","last_undegraded":"2023-10-14T18:18:51.293635+0000","last_fullsized":"2023-10-14T18:18:51.293635+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:09.985958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299473+0000","last_change":"2023-10-14T18:18:35.679660+0000","last_active":"2023-10-14T18:18:50.299473+0000","last_peered":"2023-10-14T18:18:50.299473+0000","last_clean":"2023-10-14T18:18:50.299473+0000","last_became_active":"2023-10-14T18:18:35.679374+0000","last_became_peered":"2023-10-14T18:18:35.679374+0000","last_unstale":"2023-10-14T18:18:50.299473+0000","last_undegraded":"2023-10-14T18:18:50.299473+0000","last_fullsized":"2023-10-14T18:18:50.299473+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:57.315103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380211+0000","last_change":"2023-10-14T18:18:50.313779+0000","last_active":"2023-10-14T18:18:51.380211+0000","last_peered":"2023-10-14T18:18:51.380211+0000","last_clean":"2023-10-14T18:18:51.380211+0000","last_became_active":"2023-10-14T18:18:50.313502+0000","last_became_peered":"2023-10-14T18:18:50.313502+0000","last_unstale":"2023-10-14T18:18:51.380211+0000","last_undegraded":"2023-10-14T18:18:51.380211+0000","last_fullsized":"2023-10-14T18:18:51.380211+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:41.753050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375761+0000","last_change":"2023-10-14T18:18:49.298802+0000","last_active":"2023-10-14T18:18:51.375761+0000","last_peered":"2023-10-14T18:18:51.375761+0000","last_clean":"2023-10-14T18:18:51.375761+0000","last_became_active":"2023-10-14T18:18:49.298073+0000","last_became_peered":"2023-10-14T18:18:49.298073+0000","last_unstale":"2023-10-14T18:18:51.375761+0000","last_undegraded":"2023-10-14T18:18:51.375761+0000","last_fullsized":"2023-10-14T18:18:51.375761+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:52.783786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309333+0000","last_change":"2023-10-14T18:18:35.679829+0000","last_active":"2023-10-14T18:18:50.309333+0000","last_peered":"2023-10-14T18:18:50.309333+0000","last_clean":"2023-10-14T18:18:50.309333+0000","last_became_active":"2023-10-14T18:18:35.679412+0000","last_became_peered":"2023-10-14T18:18:35.679412+0000","last_unstale":"2023-10-14T18:18:50.309333+0000","last_undegraded":"2023-10-14T18:18:50.309333+0000","last_fullsized":"2023-10-14T18:18:50.309333+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378447+0000","last_change":"2023-10-14T18:18:50.327269+0000","last_active":"2023-10-14T18:18:51.378447+0000","last_peered":"2023-10-14T18:18:51.378447+0000","last_clean":"2023-10-14T18:18:51.378447+0000","last_became_active":"2023-10-14T18:18:50.326946+0000","last_became_peered":"2023-10-14T18:18:50.326946+0000","last_unstale":"2023-10-14T18:18:51.378447+0000","last_undegraded":"2023-10-14T18:18:51.378447+0000","last_fullsized":"2023-10-14T18:18:51.378447+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:43.478859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293746+0000","last_change":"2023-10-14T18:18:49.315838+0000","last_active":"2023-10-14T18:18:51.293746+0000","last_peered":"2023-10-14T18:18:51.293746+0000","last_clean":"2023-10-14T18:18:51.293746+0000","last_became_active":"2023-10-14T18:18:49.315372+0000","last_became_peered":"2023-10-14T18:18:49.315372+0000","last_unstale":"2023-10-14T18:18:51.293746+0000","last_undegraded":"2023-10-14T18:18:51.293746+0000","last_fullsized":"2023-10-14T18:18:51.293746+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.483516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292157+0000","last_change":"2023-10-14T18:18:35.689722+0000","last_active":"2023-10-14T18:18:50.292157+0000","last_peered":"2023-10-14T18:18:50.292157+0000","last_clean":"2023-10-14T18:18:50.292157+0000","last_became_active":"2023-10-14T18:18:35.686665+0000","last_became_peered":"2023-10-14T18:18:35.686665+0000","last_unstale":"2023-10-14T18:18:50.292157+0000","last_undegraded":"2023-10-14T18:18:50.292157+0000","last_fullsized":"2023-10-14T18:18:50.292157+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:57.853905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378778+0000","last_change":"2023-10-14T18:18:50.312161+0000","last_active":"2023-10-14T18:18:51.378778+0000","last_peered":"2023-10-14T18:18:51.378778+0000","last_clean":"2023-10-14T18:18:51.378778+0000","last_became_active":"2023-10-14T18:18:50.310426+0000","last_became_peered":"2023-10-14T18:18:50.310426+0000","last_unstale":"2023-10-14T18:18:51.378778+0000","last_undegraded":"2023-10-14T18:18:51.378778+0000","last_fullsized":"2023-10-14T18:18:51.378778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:00.299094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.295588+0000","last_change":"2023-10-14T18:18:49.308030+0000","last_active":"2023-10-14T18:18:51.295588+0000","last_peered":"2023-10-14T18:18:51.295588+0000","last_clean":"2023-10-14T18:18:51.295588+0000","last_became_active":"2023-10-14T18:18:49.307402+0000","last_became_peered":"2023-10-14T18:18:49.307402+0000","last_unstale":"2023-10-14T18:18:51.295588+0000","last_undegraded":"2023-10-14T18:18:51.295588+0000","last_fullsized":"2023-10-14T18:18:51.295588+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:28.857963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295742+0000","last_change":"2023-10-14T18:18:35.674443+0000","last_active":"2023-10-14T18:18:50.295742+0000","last_peered":"2023-10-14T18:18:50.295742+0000","last_clean":"2023-10-14T18:18:50.295742+0000","last_became_active":"2023-10-14T18:18:35.673655+0000","last_became_peered":"2023-10-14T18:18:35.673655+0000","last_unstale":"2023-10-14T18:18:50.295742+0000","last_undegraded":"2023-10-14T18:18:50.295742+0000","last_fullsized":"2023-10-14T18:18:50.295742+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299308+0000","last_change":"2023-10-14T18:18:50.316952+0000","last_active":"2023-10-14T18:18:51.299308+0000","last_peered":"2023-10-14T18:18:51.299308+0000","last_clean":"2023-10-14T18:18:51.299308+0000","last_became_active":"2023-10-14T18:18:50.316669+0000","last_became_peered":"2023-10-14T18:18:50.316669+0000","last_unstale":"2023-10-14T18:18:51.299308+0000","last_undegraded":"2023-10-14T18:18:51.299308+0000","last_fullsized":"2023-10-14T18:18:51.299308+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:51.338036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385216+0000","last_change":"2023-10-14T18:18:49.313970+0000","last_active":"2023-10-14T18:18:51.385216+0000","last_peered":"2023-10-14T18:18:51.385216+0000","last_clean":"2023-10-14T18:18:51.385216+0000","last_became_active":"2023-10-14T18:18:49.313689+0000","last_became_peered":"2023-10-14T18:18:49.313689+0000","last_unstale":"2023-10-14T18:18:51.385216+0000","last_undegraded":"2023-10-14T18:18:51.385216+0000","last_fullsized":"2023-10-14T18:18:51.385216+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:55:51.768877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295540+0000","last_change":"2023-10-14T18:18:35.687221+0000","last_active":"2023-10-14T18:18:50.295540+0000","last_peered":"2023-10-14T18:18:50.295540+0000","last_clean":"2023-10-14T18:18:50.295540+0000","last_became_active":"2023-10-14T18:18:35.686900+0000","last_became_peered":"2023-10-14T18:18:35.686900+0000","last_unstale":"2023-10-14T18:18:50.295540+0000","last_undegraded":"2023-10-14T18:18:50.295540+0000","last_fullsized":"2023-10-14T18:18:50.295540+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385231+0000","last_change":"2023-10-14T18:18:50.321146+0000","last_active":"2023-10-14T18:18:51.385231+0000","last_peered":"2023-10-14T18:18:51.385231+0000","last_clean":"2023-10-14T18:18:51.385231+0000","last_became_active":"2023-10-14T18:18:50.318691+0000","last_became_peered":"2023-10-14T18:18:50.318691+0000","last_unstale":"2023-10-14T18:18:51.385231+0000","last_undegraded":"2023-10-14T18:18:51.385231+0000","last_fullsized":"2023-10-14T18:18:51.385231+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:25.388110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376614+0000","last_change":"2023-10-14T18:18:49.303213+0000","last_active":"2023-10-14T18:18:51.376614+0000","last_peered":"2023-10-14T18:18:51.376614+0000","last_clean":"2023-10-14T18:18:51.376614+0000","last_became_active":"2023-10-14T18:18:49.302764+0000","last_became_peered":"2023-10-14T18:18:49.302764+0000","last_unstale":"2023-10-14T18:18:51.376614+0000","last_undegraded":"2023-10-14T18:18:51.376614+0000","last_fullsized":"2023-10-14T18:18:51.376614+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:03.287251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299683+0000","last_change":"2023-10-14T18:18:35.671605+0000","last_active":"2023-10-14T18:18:50.299683+0000","last_peered":"2023-10-14T18:18:50.299683+0000","last_clean":"2023-10-14T18:18:50.299683+0000","last_became_active":"2023-10-14T18:18:35.671317+0000","last_became_peered":"2023-10-14T18:18:35.671317+0000","last_unstale":"2023-10-14T18:18:50.299683+0000","last_undegraded":"2023-10-14T18:18:50.299683+0000","last_fullsized":"2023-10-14T18:18:50.299683+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:14.496902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377102+0000","last_change":"2023-10-14T18:18:50.318042+0000","last_active":"2023-10-14T18:18:51.377102+0000","last_peered":"2023-10-14T18:18:51.377102+0000","last_clean":"2023-10-14T18:18:51.377102+0000","last_became_active":"2023-10-14T18:18:50.317689+0000","last_became_peered":"2023-10-14T18:18:50.317689+0000","last_unstale":"2023-10-14T18:18:51.377102+0000","last_undegraded":"2023-10-14T18:18:51.377102+0000","last_fullsized":"2023-10-14T18:18:51.377102+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:36.530334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377653+0000","last_change":"2023-10-14T18:18:49.305425+0000","last_active":"2023-10-14T18:18:51.377653+0000","last_peered":"2023-10-14T18:18:51.377653+0000","last_clean":"2023-10-14T18:18:51.377653+0000","last_became_active":"2023-10-14T18:18:49.305158+0000","last_became_peered":"2023-10-14T18:18:49.305158+0000","last_unstale":"2023-10-14T18:18:51.377653+0000","last_undegraded":"2023-10-14T18:18:51.377653+0000","last_fullsized":"2023-10-14T18:18:51.377653+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:39.442155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298994+0000","last_change":"2023-10-14T18:18:35.682877+0000","last_active":"2023-10-14T18:18:50.298994+0000","last_peered":"2023-10-14T18:18:50.298994+0000","last_clean":"2023-10-14T18:18:50.298994+0000","last_became_active":"2023-10-14T18:18:35.682389+0000","last_became_peered":"2023-10-14T18:18:35.682389+0000","last_unstale":"2023-10-14T18:18:50.298994+0000","last_undegraded":"2023-10-14T18:18:50.298994+0000","last_fullsized":"2023-10-14T18:18:50.298994+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:25.798127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381081+0000","last_change":"2023-10-14T18:18:50.317867+0000","last_active":"2023-10-14T18:18:51.381081+0000","last_peered":"2023-10-14T18:18:51.381081+0000","last_clean":"2023-10-14T18:18:51.381081+0000","last_became_active":"2023-10-14T18:18:50.317418+0000","last_became_peered":"2023-10-14T18:18:50.317418+0000","last_unstale":"2023-10-14T18:18:51.381081+0000","last_undegraded":"2023-10-14T18:18:51.381081+0000","last_fullsized":"2023-10-14T18:18:51.381081+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:44:51.597540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294269+0000","last_change":"2023-10-14T18:18:49.314303+0000","last_active":"2023-10-14T18:18:51.294269+0000","last_peered":"2023-10-14T18:18:51.294269+0000","last_clean":"2023-10-14T18:18:51.294269+0000","last_became_active":"2023-10-14T18:18:49.314025+0000","last_became_peered":"2023-10-14T18:18:49.314025+0000","last_unstale":"2023-10-14T18:18:51.294269+0000","last_undegraded":"2023-10-14T18:18:51.294269+0000","last_fullsized":"2023-10-14T18:18:51.294269+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:39.490952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309959+0000","last_change":"2023-10-14T18:18:35.674408+0000","last_active":"2023-10-14T18:18:50.309959+0000","last_peered":"2023-10-14T18:18:50.309959+0000","last_clean":"2023-10-14T18:18:50.309959+0000","last_became_active":"2023-10-14T18:18:35.674062+0000","last_became_peered":"2023-10-14T18:18:35.674062+0000","last_unstale":"2023-10-14T18:18:50.309959+0000","last_undegraded":"2023-10-14T18:18:50.309959+0000","last_fullsized":"2023-10-14T18:18:50.309959+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:16.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376778+0000","last_change":"2023-10-14T18:18:50.322490+0000","last_active":"2023-10-14T18:18:51.376778+0000","last_peered":"2023-10-14T18:18:51.376778+0000","last_clean":"2023-10-14T18:18:51.376778+0000","last_became_active":"2023-10-14T18:18:50.322222+0000","last_became_peered":"2023-10-14T18:18:50.322222+0000","last_unstale":"2023-10-14T18:18:51.376778+0000","last_undegraded":"2023-10-14T18:18:51.376778+0000","last_fullsized":"2023-10-14T18:18:51.376778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:16:44.047723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"253'8","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377489+0000","last_change":"2023-10-14T18:18:49.311665+0000","last_active":"2023-10-14T18:18:51.377489+0000","last_peered":"2023-10-14T18:18:51.377489+0000","last_clean":"2023-10-14T18:18:51.377489+0000","last_became_active":"2023-10-14T18:18:49.311417+0000","last_became_peered":"2023-10-14T18:18:49.311417+0000","last_unstale":"2023-10-14T18:18:51.377489+0000","last_undegraded":"2023-10-14T18:18:51.377489+0000","last_fullsized":"2023-10-14T18:18:51.377489+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:21.635177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296956+0000","last_change":"2023-10-14T18:18:35.678586+0000","last_active":"2023-10-14T18:18:50.296956+0000","last_peered":"2023-10-14T18:18:50.296956+0000","last_clean":"2023-10-14T18:18:50.296956+0000","last_became_active":"2023-10-14T18:18:35.678265+0000","last_became_peered":"2023-10-14T18:18:35.678265+0000","last_unstale":"2023-10-14T18:18:50.296956+0000","last_undegraded":"2023-10-14T18:18:50.296956+0000","last_fullsized":"2023-10-14T18:18:50.296956+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:25.316716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379803+0000","last_change":"2023-10-14T18:18:50.320388+0000","last_active":"2023-10-14T18:18:51.379803+0000","last_peered":"2023-10-14T18:18:51.379803+0000","last_clean":"2023-10-14T18:18:51.379803+0000","last_became_active":"2023-10-14T18:18:50.319785+0000","last_became_peered":"2023-10-14T18:18:50.319785+0000","last_unstale":"2023-10-14T18:18:51.379803+0000","last_undegraded":"2023-10-14T18:18:51.379803+0000","last_fullsized":"2023-10-14T18:18:51.379803+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:16.781111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376063+0000","last_change":"2023-10-14T18:18:49.305027+0000","last_active":"2023-10-14T18:18:51.376063+0000","last_peered":"2023-10-14T18:18:51.376063+0000","last_clean":"2023-10-14T18:18:51.376063+0000","last_became_active":"2023-10-14T18:18:49.304531+0000","last_became_peered":"2023-10-14T18:18:49.304531+0000","last_unstale":"2023-10-14T18:18:51.376063+0000","last_undegraded":"2023-10-14T18:18:51.376063+0000","last_fullsized":"2023-10-14T18:18:51.376063+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:06:32.235368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298994+0000","last_change":"2023-10-14T18:18:35.679271+0000","last_active":"2023-10-14T18:18:50.298994+0000","last_peered":"2023-10-14T18:18:50.298994+0000","last_clean":"2023-10-14T18:18:50.298994+0000","last_became_active":"2023-10-14T18:18:35.678924+0000","last_became_peered":"2023-10-14T18:18:35.678924+0000","last_unstale":"2023-10-14T18:18:50.298994+0000","last_undegraded":"2023-10-14T18:18:50.298994+0000","last_fullsized":"2023-10-14T18:18:50.298994+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:56.833692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378103+0000","last_change":"2023-10-14T18:18:50.312738+0000","last_active":"2023-10-14T18:18:51.378103+0000","last_peered":"2023-10-14T18:18:51.378103+0000","last_clean":"2023-10-14T18:18:51.378103+0000","last_became_active":"2023-10-14T18:18:50.311095+0000","last_became_peered":"2023-10-14T18:18:50.311095+0000","last_unstale":"2023-10-14T18:18:51.378103+0000","last_undegraded":"2023-10-14T18:18:51.378103+0000","last_fullsized":"2023-10-14T18:18:51.378103+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:34.190468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378683+0000","last_change":"2023-10-14T18:18:49.310720+0000","last_active":"2023-10-14T18:18:51.378683+0000","last_peered":"2023-10-14T18:18:51.378683+0000","last_clean":"2023-10-14T18:18:51.378683+0000","last_became_active":"2023-10-14T18:18:49.310133+0000","last_became_peered":"2023-10-14T18:18:49.310133+0000","last_unstale":"2023-10-14T18:18:51.378683+0000","last_undegraded":"2023-10-14T18:18:51.378683+0000","last_fullsized":"2023-10-14T18:18:51.378683+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:38.960744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296506+0000","last_change":"2023-10-14T18:18:35.683679+0000","last_active":"2023-10-14T18:18:50.296506+0000","last_peered":"2023-10-14T18:18:50.296506+0000","last_clean":"2023-10-14T18:18:50.296506+0000","last_became_active":"2023-10-14T18:18:35.682861+0000","last_became_peered":"2023-10-14T18:18:35.682861+0000","last_unstale":"2023-10-14T18:18:50.296506+0000","last_undegraded":"2023-10-14T18:18:50.296506+0000","last_fullsized":"2023-10-14T18:18:50.296506+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379795+0000","last_change":"2023-10-14T18:18:50.322584+0000","last_active":"2023-10-14T18:18:51.379795+0000","last_peered":"2023-10-14T18:18:51.379795+0000","last_clean":"2023-10-14T18:18:51.379795+0000","last_became_active":"2023-10-14T18:18:50.322310+0000","last_became_peered":"2023-10-14T18:18:50.322310+0000","last_unstale":"2023-10-14T18:18:51.379795+0000","last_undegraded":"2023-10-14T18:18:51.379795+0000","last_fullsized":"2023-10-14T18:18:51.379795+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:35.835633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293169+0000","last_change":"2023-10-14T18:18:49.314312+0000","last_active":"2023-10-14T18:18:51.293169+0000","last_peered":"2023-10-14T18:18:51.293169+0000","last_clean":"2023-10-14T18:18:51.293169+0000","last_became_active":"2023-10-14T18:18:49.313429+0000","last_became_peered":"2023-10-14T18:18:49.313429+0000","last_unstale":"2023-10-14T18:18:51.293169+0000","last_undegraded":"2023-10-14T18:18:51.293169+0000","last_fullsized":"2023-10-14T18:18:51.293169+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:23:28.325790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309809+0000","last_change":"2023-10-14T18:18:35.675043+0000","last_active":"2023-10-14T18:18:50.309809+0000","last_peered":"2023-10-14T18:18:50.309809+0000","last_clean":"2023-10-14T18:18:50.309809+0000","last_became_active":"2023-10-14T18:18:35.674231+0000","last_became_peered":"2023-10-14T18:18:35.674231+0000","last_unstale":"2023-10-14T18:18:50.309809+0000","last_undegraded":"2023-10-14T18:18:50.309809+0000","last_fullsized":"2023-10-14T18:18:50.309809+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:11:18.531967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383396+0000","last_change":"2023-10-14T18:18:50.330890+0000","last_active":"2023-10-14T18:18:51.383396+0000","last_peered":"2023-10-14T18:18:51.383396+0000","last_clean":"2023-10-14T18:18:51.383396+0000","last_became_active":"2023-10-14T18:18:50.330415+0000","last_became_peered":"2023-10-14T18:18:50.330415+0000","last_unstale":"2023-10-14T18:18:51.383396+0000","last_undegraded":"2023-10-14T18:18:51.383396+0000","last_fullsized":"2023-10-14T18:18:51.383396+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:02.625632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"253'6","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381956+0000","last_change":"2023-10-14T18:18:49.311559+0000","last_active":"2023-10-14T18:18:51.381956+0000","last_peered":"2023-10-14T18:18:51.381956+0000","last_clean":"2023-10-14T18:18:51.381956+0000","last_became_active":"2023-10-14T18:18:49.311286+0000","last_became_peered":"2023-10-14T18:18:49.311286+0000","last_unstale":"2023-10-14T18:18:51.381956+0000","last_undegraded":"2023-10-14T18:18:51.381956+0000","last_fullsized":"2023-10-14T18:18:51.381956+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:32:17.160345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293487+0000","last_change":"2023-10-14T18:18:35.678709+0000","last_active":"2023-10-14T18:18:50.293487+0000","last_peered":"2023-10-14T18:18:50.293487+0000","last_clean":"2023-10-14T18:18:50.293487+0000","last_became_active":"2023-10-14T18:18:35.678445+0000","last_became_peered":"2023-10-14T18:18:35.678445+0000","last_unstale":"2023-10-14T18:18:50.293487+0000","last_undegraded":"2023-10-14T18:18:50.293487+0000","last_fullsized":"2023-10-14T18:18:50.293487+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385067+0000","last_change":"2023-10-14T18:18:50.309163+0000","last_active":"2023-10-14T18:18:51.385067+0000","last_peered":"2023-10-14T18:18:51.385067+0000","last_clean":"2023-10-14T18:18:51.385067+0000","last_became_active":"2023-10-14T18:18:50.308887+0000","last_became_peered":"2023-10-14T18:18:50.308887+0000","last_unstale":"2023-10-14T18:18:51.385067+0000","last_undegraded":"2023-10-14T18:18:51.385067+0000","last_fullsized":"2023-10-14T18:18:51.385067+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:40.497579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"253'3","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381833+0000","last_change":"2023-10-14T18:18:49.315656+0000","last_active":"2023-10-14T18:18:51.381833+0000","last_peered":"2023-10-14T18:18:51.381833+0000","last_clean":"2023-10-14T18:18:51.381833+0000","last_became_active":"2023-10-14T18:18:49.314830+0000","last_became_peered":"2023-10-14T18:18:49.314830+0000","last_unstale":"2023-10-14T18:18:51.381833+0000","last_undegraded":"2023-10-14T18:18:51.381833+0000","last_fullsized":"2023-10-14T18:18:51.381833+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:16.263712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295024+0000","last_change":"2023-10-14T18:18:35.678715+0000","last_active":"2023-10-14T18:18:50.295024+0000","last_peered":"2023-10-14T18:18:50.295024+0000","last_clean":"2023-10-14T18:18:50.295024+0000","last_became_active":"2023-10-14T18:18:35.678366+0000","last_became_peered":"2023-10-14T18:18:35.678366+0000","last_unstale":"2023-10-14T18:18:50.295024+0000","last_undegraded":"2023-10-14T18:18:50.295024+0000","last_fullsized":"2023-10-14T18:18:50.295024+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383713+0000","last_change":"2023-10-14T18:18:50.334460+0000","last_active":"2023-10-14T18:18:51.383713+0000","last_peered":"2023-10-14T18:18:51.383713+0000","last_clean":"2023-10-14T18:18:51.383713+0000","last_became_active":"2023-10-14T18:18:50.332701+0000","last_became_peered":"2023-10-14T18:18:50.332701+0000","last_unstale":"2023-10-14T18:18:51.383713+0000","last_undegraded":"2023-10-14T18:18:51.383713+0000","last_fullsized":"2023-10-14T18:18:51.383713+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:29.252321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377401+0000","last_change":"2023-10-14T18:18:49.315616+0000","last_active":"2023-10-14T18:18:51.377401+0000","last_peered":"2023-10-14T18:18:51.377401+0000","last_clean":"2023-10-14T18:18:51.377401+0000","last_became_active":"2023-10-14T18:18:49.315272+0000","last_became_peered":"2023-10-14T18:18:49.315272+0000","last_unstale":"2023-10-14T18:18:51.377401+0000","last_undegraded":"2023-10-14T18:18:51.377401+0000","last_fullsized":"2023-10-14T18:18:51.377401+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:00.542037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297055+0000","last_change":"2023-10-14T18:18:35.674737+0000","last_active":"2023-10-14T18:18:50.297055+0000","last_peered":"2023-10-14T18:18:50.297055+0000","last_clean":"2023-10-14T18:18:50.297055+0000","last_became_active":"2023-10-14T18:18:35.673762+0000","last_became_peered":"2023-10-14T18:18:35.673762+0000","last_unstale":"2023-10-14T18:18:50.297055+0000","last_undegraded":"2023-10-14T18:18:50.297055+0000","last_fullsized":"2023-10-14T18:18:50.297055+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299514+0000","last_change":"2023-10-14T18:18:50.320254+0000","last_active":"2023-10-14T18:18:51.299514+0000","last_peered":"2023-10-14T18:18:51.299514+0000","last_clean":"2023-10-14T18:18:51.299514+0000","last_became_active":"2023-10-14T18:18:50.319744+0000","last_became_peered":"2023-10-14T18:18:50.319744+0000","last_unstale":"2023-10-14T18:18:51.299514+0000","last_undegraded":"2023-10-14T18:18:51.299514+0000","last_fullsized":"2023-10-14T18:18:51.299514+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:55:11.484346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.384170+0000","last_change":"2023-10-14T18:18:49.314981+0000","last_active":"2023-10-14T18:18:51.384170+0000","last_peered":"2023-10-14T18:18:51.384170+0000","last_clean":"2023-10-14T18:18:51.384170+0000","last_became_active":"2023-10-14T18:18:49.314590+0000","last_became_peered":"2023-10-14T18:18:49.314590+0000","last_unstale":"2023-10-14T18:18:51.384170+0000","last_undegraded":"2023-10-14T18:18:51.384170+0000","last_fullsized":"2023-10-14T18:18:51.384170+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:25.658536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296111+0000","last_change":"2023-10-14T18:18:35.679931+0000","last_active":"2023-10-14T18:18:50.296111+0000","last_peered":"2023-10-14T18:18:50.296111+0000","last_clean":"2023-10-14T18:18:50.296111+0000","last_became_active":"2023-10-14T18:18:35.679636+0000","last_became_peered":"2023-10-14T18:18:35.679636+0000","last_unstale":"2023-10-14T18:18:50.296111+0000","last_undegraded":"2023-10-14T18:18:50.296111+0000","last_fullsized":"2023-10-14T18:18:50.296111+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:40.768705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297905+0000","last_change":"2023-10-14T18:18:50.314849+0000","last_active":"2023-10-14T18:18:51.297905+0000","last_peered":"2023-10-14T18:18:51.297905+0000","last_clean":"2023-10-14T18:18:51.297905+0000","last_became_active":"2023-10-14T18:18:50.313929+0000","last_became_peered":"2023-10-14T18:18:50.313929+0000","last_unstale":"2023-10-14T18:18:51.297905+0000","last_undegraded":"2023-10-14T18:18:51.297905+0000","last_fullsized":"2023-10-14T18:18:51.297905+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:29.147556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297471+0000","last_change":"2023-10-14T18:18:49.315528+0000","last_active":"2023-10-14T18:18:51.297471+0000","last_peered":"2023-10-14T18:18:51.297471+0000","last_clean":"2023-10-14T18:18:51.297471+0000","last_became_active":"2023-10-14T18:18:49.315247+0000","last_became_peered":"2023-10-14T18:18:49.315247+0000","last_unstale":"2023-10-14T18:18:51.297471+0000","last_undegraded":"2023-10-14T18:18:51.297471+0000","last_fullsized":"2023-10-14T18:18:51.297471+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:14.678752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306376+0000","last_change":"2023-10-14T18:18:35.663686+0000","last_active":"2023-10-14T18:18:50.306376+0000","last_peered":"2023-10-14T18:18:50.306376+0000","last_clean":"2023-10-14T18:18:50.306376+0000","last_became_active":"2023-10-14T18:18:35.663364+0000","last_became_peered":"2023-10-14T18:18:35.663364+0000","last_unstale":"2023-10-14T18:18:50.306376+0000","last_undegraded":"2023-10-14T18:18:50.306376+0000","last_fullsized":"2023-10-14T18:18:50.306376+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:43.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377248+0000","last_change":"2023-10-14T18:18:50.325391+0000","last_active":"2023-10-14T18:18:51.377248+0000","last_peered":"2023-10-14T18:18:51.377248+0000","last_clean":"2023-10-14T18:18:51.377248+0000","last_became_active":"2023-10-14T18:18:50.325122+0000","last_became_peered":"2023-10-14T18:18:50.325122+0000","last_unstale":"2023-10-14T18:18:51.377248+0000","last_undegraded":"2023-10-14T18:18:51.377248+0000","last_fullsized":"2023-10-14T18:18:51.377248+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:04.610423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294546+0000","last_change":"2023-10-14T18:18:49.300991+0000","last_active":"2023-10-14T18:18:51.294546+0000","last_peered":"2023-10-14T18:18:51.294546+0000","last_clean":"2023-10-14T18:18:51.294546+0000","last_became_active":"2023-10-14T18:18:49.300438+0000","last_became_peered":"2023-10-14T18:18:49.300438+0000","last_unstale":"2023-10-14T18:18:51.294546+0000","last_undegraded":"2023-10-14T18:18:51.294546+0000","last_fullsized":"2023-10-14T18:18:51.294546+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:13.095242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309419+0000","last_change":"2023-10-14T18:18:35.683088+0000","last_active":"2023-10-14T18:18:50.309419+0000","last_peered":"2023-10-14T18:18:50.309419+0000","last_clean":"2023-10-14T18:18:50.309419+0000","last_became_active":"2023-10-14T18:18:35.682753+0000","last_became_peered":"2023-10-14T18:18:35.682753+0000","last_unstale":"2023-10-14T18:18:50.309419+0000","last_undegraded":"2023-10-14T18:18:50.309419+0000","last_fullsized":"2023-10-14T18:18:50.309419+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377699+0000","last_change":"2023-10-14T18:18:50.307192+0000","last_active":"2023-10-14T18:18:51.377699+0000","last_peered":"2023-10-14T18:18:51.377699+0000","last_clean":"2023-10-14T18:18:51.377699+0000","last_became_active":"2023-10-14T18:18:50.306894+0000","last_became_peered":"2023-10-14T18:18:50.306894+0000","last_unstale":"2023-10-14T18:18:51.377699+0000","last_undegraded":"2023-10-14T18:18:51.377699+0000","last_fullsized":"2023-10-14T18:18:51.377699+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:05:58.616654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379194+0000","last_change":"2023-10-14T18:18:49.307502+0000","last_active":"2023-10-14T18:18:51.379194+0000","last_peered":"2023-10-14T18:18:51.379194+0000","last_clean":"2023-10-14T18:18:51.379194+0000","last_became_active":"2023-10-14T18:18:49.307250+0000","last_became_peered":"2023-10-14T18:18:49.307250+0000","last_unstale":"2023-10-14T18:18:51.379194+0000","last_undegraded":"2023-10-14T18:18:51.379194+0000","last_fullsized":"2023-10-14T18:18:51.379194+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:40:40.785284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292297+0000","last_change":"2023-10-14T18:18:35.689155+0000","last_active":"2023-10-14T18:18:50.292297+0000","last_peered":"2023-10-14T18:18:50.292297+0000","last_clean":"2023-10-14T18:18:50.292297+0000","last_became_active":"2023-10-14T18:18:35.686147+0000","last_became_peered":"2023-10-14T18:18:35.686147+0000","last_unstale":"2023-10-14T18:18:50.292297+0000","last_undegraded":"2023-10-14T18:18:50.292297+0000","last_fullsized":"2023-10-14T18:18:50.292297+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:12.203818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379371+0000","last_change":"2023-10-14T18:18:50.324595+0000","last_active":"2023-10-14T18:18:51.379371+0000","last_peered":"2023-10-14T18:18:51.379371+0000","last_clean":"2023-10-14T18:18:51.379371+0000","last_became_active":"2023-10-14T18:18:50.324293+0000","last_became_peered":"2023-10-14T18:18:50.324293+0000","last_unstale":"2023-10-14T18:18:51.379371+0000","last_undegraded":"2023-10-14T18:18:51.379371+0000","last_fullsized":"2023-10-14T18:18:51.379371+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:36:56.093870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.300008+0000","last_change":"2023-10-14T18:18:49.312699+0000","last_active":"2023-10-14T18:18:51.300008+0000","last_peered":"2023-10-14T18:18:51.300008+0000","last_clean":"2023-10-14T18:18:51.300008+0000","last_became_active":"2023-10-14T18:18:49.311196+0000","last_became_peered":"2023-10-14T18:18:49.311196+0000","last_unstale":"2023-10-14T18:18:51.300008+0000","last_undegraded":"2023-10-14T18:18:51.300008+0000","last_fullsized":"2023-10-14T18:18:51.300008+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.483516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297975+0000","last_change":"2023-10-14T18:18:35.673716+0000","last_active":"2023-10-14T18:18:50.297975+0000","last_peered":"2023-10-14T18:18:50.297975+0000","last_clean":"2023-10-14T18:18:50.297975+0000","last_became_active":"2023-10-14T18:18:35.672662+0000","last_became_peered":"2023-10-14T18:18:35.672662+0000","last_unstale":"2023-10-14T18:18:50.297975+0000","last_undegraded":"2023-10-14T18:18:50.297975+0000","last_fullsized":"2023-10-14T18:18:50.297975+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:06.125956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293622+0000","last_change":"2023-10-14T18:18:50.311394+0000","last_active":"2023-10-14T18:18:51.293622+0000","last_peered":"2023-10-14T18:18:51.293622+0000","last_clean":"2023-10-14T18:18:51.293622+0000","last_became_active":"2023-10-14T18:18:50.311090+0000","last_became_peered":"2023-10-14T18:18:50.311090+0000","last_unstale":"2023-10-14T18:18:51.293622+0000","last_undegraded":"2023-10-14T18:18:51.293622+0000","last_fullsized":"2023-10-14T18:18:51.293622+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:25.388110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380528+0000","last_change":"2023-10-14T18:18:49.310283+0000","last_active":"2023-10-14T18:18:51.380528+0000","last_peered":"2023-10-14T18:18:51.380528+0000","last_clean":"2023-10-14T18:18:51.380528+0000","last_became_active":"2023-10-14T18:18:49.310002+0000","last_became_peered":"2023-10-14T18:18:49.310002+0000","last_unstale":"2023-10-14T18:18:51.380528+0000","last_undegraded":"2023-10-14T18:18:51.380528+0000","last_fullsized":"2023-10-14T18:18:51.380528+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:09:58.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289984+0000","last_change":"2023-10-14T18:18:35.682800+0000","last_active":"2023-10-14T18:18:50.289984+0000","last_peered":"2023-10-14T18:18:50.289984+0000","last_clean":"2023-10-14T18:18:50.289984+0000","last_became_active":"2023-10-14T18:18:35.682478+0000","last_became_peered":"2023-10-14T18:18:35.682478+0000","last_unstale":"2023-10-14T18:18:50.289984+0000","last_undegraded":"2023-10-14T18:18:50.289984+0000","last_fullsized":"2023-10-14T18:18:50.289984+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.298778+0000","last_change":"2023-10-14T18:18:50.318794+0000","last_active":"2023-10-14T18:18:51.298778+0000","last_peered":"2023-10-14T18:18:51.298778+0000","last_clean":"2023-10-14T18:18:51.298778+0000","last_became_active":"2023-10-14T18:18:50.312936+0000","last_became_peered":"2023-10-14T18:18:50.312936+0000","last_unstale":"2023-10-14T18:18:51.298778+0000","last_undegraded":"2023-10-14T18:18:51.298778+0000","last_fullsized":"2023-10-14T18:18:51.298778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:29.400393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381700+0000","last_change":"2023-10-14T18:18:49.308149+0000","last_active":"2023-10-14T18:18:51.381700+0000","last_peered":"2023-10-14T18:18:51.381700+0000","last_clean":"2023-10-14T18:18:51.381700+0000","last_became_active":"2023-10-14T18:18:49.307852+0000","last_became_peered":"2023-10-14T18:18:49.307852+0000","last_unstale":"2023-10-14T18:18:51.381700+0000","last_undegraded":"2023-10-14T18:18:51.381700+0000","last_fullsized":"2023-10-14T18:18:51.381700+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:38.263857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.315418+0000","last_change":"2023-10-14T18:18:35.676776+0000","last_active":"2023-10-14T18:18:50.315418+0000","last_peered":"2023-10-14T18:18:50.315418+0000","last_clean":"2023-10-14T18:18:50.315418+0000","last_became_active":"2023-10-14T18:18:35.675446+0000","last_became_peered":"2023-10-14T18:18:35.675446+0000","last_unstale":"2023-10-14T18:18:50.315418+0000","last_undegraded":"2023-10-14T18:18:50.315418+0000","last_fullsized":"2023-10-14T18:18:50.315418+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:09.420076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379092+0000","last_change":"2023-10-14T18:18:50.316786+0000","last_active":"2023-10-14T18:18:51.379092+0000","last_peered":"2023-10-14T18:18:51.379092+0000","last_clean":"2023-10-14T18:18:51.379092+0000","last_became_active":"2023-10-14T18:18:50.316396+0000","last_became_peered":"2023-10-14T18:18:50.316396+0000","last_unstale":"2023-10-14T18:18:51.379092+0000","last_undegraded":"2023-10-14T18:18:51.379092+0000","last_fullsized":"2023-10-14T18:18:51.379092+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:36:56.093870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378765+0000","last_change":"2023-10-14T18:18:49.313758+0000","last_active":"2023-10-14T18:18:51.378765+0000","last_peered":"2023-10-14T18:18:51.378765+0000","last_clean":"2023-10-14T18:18:51.378765+0000","last_became_active":"2023-10-14T18:18:49.312912+0000","last_became_peered":"2023-10-14T18:18:49.312912+0000","last_unstale":"2023-10-14T18:18:51.378765+0000","last_undegraded":"2023-10-14T18:18:51.378765+0000","last_fullsized":"2023-10-14T18:18:51.378765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:52.136902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.307636+0000","last_change":"2023-10-14T18:18:35.676664+0000","last_active":"2023-10-14T18:18:50.307636+0000","last_peered":"2023-10-14T18:18:50.307636+0000","last_clean":"2023-10-14T18:18:50.307636+0000","last_became_active":"2023-10-14T18:18:35.676381+0000","last_became_peered":"2023-10-14T18:18:35.676381+0000","last_unstale":"2023-10-14T18:18:50.307636+0000","last_undegraded":"2023-10-14T18:18:50.307636+0000","last_fullsized":"2023-10-14T18:18:50.307636+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:23.246857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299515+0000","last_change":"2023-10-14T18:18:50.320064+0000","last_active":"2023-10-14T18:18:51.299515+0000","last_peered":"2023-10-14T18:18:51.299515+0000","last_clean":"2023-10-14T18:18:51.299515+0000","last_became_active":"2023-10-14T18:18:50.319454+0000","last_became_peered":"2023-10-14T18:18:50.319454+0000","last_unstale":"2023-10-14T18:18:51.299515+0000","last_undegraded":"2023-10-14T18:18:51.299515+0000","last_fullsized":"2023-10-14T18:18:51.299515+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:47.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"253'4","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383983+0000","last_change":"2023-10-14T18:18:49.307206+0000","last_active":"2023-10-14T18:18:51.383983+0000","last_peered":"2023-10-14T18:18:51.383983+0000","last_clean":"2023-10-14T18:18:51.383983+0000","last_became_active":"2023-10-14T18:18:49.306864+0000","last_became_peered":"2023-10-14T18:18:49.306864+0000","last_unstale":"2023-10-14T18:18:51.383983+0000","last_undegraded":"2023-10-14T18:18:51.383983+0000","last_fullsized":"2023-10-14T18:18:51.383983+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:47.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296256+0000","last_change":"2023-10-14T18:18:35.670711+0000","last_active":"2023-10-14T18:18:50.296256+0000","last_peered":"2023-10-14T18:18:50.296256+0000","last_clean":"2023-10-14T18:18:50.296256+0000","last_became_active":"2023-10-14T18:18:35.670350+0000","last_became_peered":"2023-10-14T18:18:35.670350+0000","last_unstale":"2023-10-14T18:18:50.296256+0000","last_undegraded":"2023-10-14T18:18:50.296256+0000","last_fullsized":"2023-10-14T18:18:50.296256+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:32.369325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294673+0000","last_change":"2023-10-14T18:18:50.312932+0000","last_active":"2023-10-14T18:18:51.294673+0000","last_peered":"2023-10-14T18:18:51.294673+0000","last_clean":"2023-10-14T18:18:51.294673+0000","last_became_active":"2023-10-14T18:18:50.311269+0000","last_became_peered":"2023-10-14T18:18:50.311269+0000","last_unstale":"2023-10-14T18:18:51.294673+0000","last_undegraded":"2023-10-14T18:18:51.294673+0000","last_fullsized":"2023-10-14T18:18:51.294673+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:41:38.778707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309584+0000","last_change":"2023-10-14T18:18:35.677418+0000","last_active":"2023-10-14T18:18:50.309584+0000","last_peered":"2023-10-14T18:18:50.309584+0000","last_clean":"2023-10-14T18:18:50.309584+0000","last_became_active":"2023-10-14T18:18:35.677059+0000","last_became_peered":"2023-10-14T18:18:35.677059+0000","last_unstale":"2023-10-14T18:18:50.309584+0000","last_undegraded":"2023-10-14T18:18:50.309584+0000","last_fullsized":"2023-10-14T18:18:50.309584+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.342329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309595+0000","last_change":"2023-10-14T18:18:34.650335+0000","last_active":"2023-10-14T18:18:50.309595+0000","last_peered":"2023-10-14T18:18:50.309595+0000","last_clean":"2023-10-14T18:18:50.309595+0000","last_became_active":"2023-10-14T18:18:34.648549+0000","last_became_peered":"2023-10-14T18:18:34.648549+0000","last_unstale":"2023-10-14T18:18:50.309595+0000","last_undegraded":"2023-10-14T18:18:50.309595+0000","last_fullsized":"2023-10-14T18:18:50.309595+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380246+0000","last_change":"2023-10-14T18:18:50.311514+0000","last_active":"2023-10-14T18:18:51.380246+0000","last_peered":"2023-10-14T18:18:51.380246+0000","last_clean":"2023-10-14T18:18:51.380246+0000","last_became_active":"2023-10-14T18:18:50.310994+0000","last_became_peered":"2023-10-14T18:18:50.310994+0000","last_unstale":"2023-10-14T18:18:51.380246+0000","last_undegraded":"2023-10-14T18:18:51.380246+0000","last_fullsized":"2023-10-14T18:18:51.380246+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:10.992585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291062+0000","last_change":"2023-10-14T18:18:35.684529+0000","last_active":"2023-10-14T18:18:50.291062+0000","last_peered":"2023-10-14T18:18:50.291062+0000","last_clean":"2023-10-14T18:18:50.291062+0000","last_became_active":"2023-10-14T18:18:35.684263+0000","last_became_peered":"2023-10-14T18:18:35.684263+0000","last_unstale":"2023-10-14T18:18:50.291062+0000","last_undegraded":"2023-10-14T18:18:50.291062+0000","last_fullsized":"2023-10-14T18:18:50.291062+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:06.771665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.324626+0000","last_change":"2023-10-14T18:18:34.653624+0000","last_active":"2023-10-14T18:18:50.324626+0000","last_peered":"2023-10-14T18:18:50.324626+0000","last_clean":"2023-10-14T18:18:50.324626+0000","last_became_active":"2023-10-14T18:18:34.652877+0000","last_became_peered":"2023-10-14T18:18:34.652877+0000","last_unstale":"2023-10-14T18:18:50.324626+0000","last_undegraded":"2023-10-14T18:18:50.324626+0000","last_fullsized":"2023-10-14T18:18:50.324626+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:13.266689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382719+0000","last_change":"2023-10-14T18:18:50.321052+0000","last_active":"2023-10-14T18:18:51.382719+0000","last_peered":"2023-10-14T18:18:51.382719+0000","last_clean":"2023-10-14T18:18:51.382719+0000","last_became_active":"2023-10-14T18:18:50.320652+0000","last_became_peered":"2023-10-14T18:18:50.320652+0000","last_unstale":"2023-10-14T18:18:51.382719+0000","last_undegraded":"2023-10-14T18:18:51.382719+0000","last_fullsized":"2023-10-14T18:18:51.382719+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T06:01:25.633128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296610+0000","last_change":"2023-10-14T18:18:35.671984+0000","last_active":"2023-10-14T18:18:50.296610+0000","last_peered":"2023-10-14T18:18:50.296610+0000","last_clean":"2023-10-14T18:18:50.296610+0000","last_became_active":"2023-10-14T18:18:35.668766+0000","last_became_peered":"2023-10-14T18:18:35.668766+0000","last_unstale":"2023-10-14T18:18:50.296610+0000","last_undegraded":"2023-10-14T18:18:50.296610+0000","last_fullsized":"2023-10-14T18:18:50.296610+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310014+0000","last_change":"2023-10-14T18:18:34.649372+0000","last_active":"2023-10-14T18:18:50.310014+0000","last_peered":"2023-10-14T18:18:50.310014+0000","last_clean":"2023-10-14T18:18:50.310014+0000","last_became_active":"2023-10-14T18:18:34.648998+0000","last_became_peered":"2023-10-14T18:18:34.648998+0000","last_unstale":"2023-10-14T18:18:50.310014+0000","last_undegraded":"2023-10-14T18:18:50.310014+0000","last_fullsized":"2023-10-14T18:18:50.310014+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:24.567915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385161+0000","last_change":"2023-10-14T18:18:50.319043+0000","last_active":"2023-10-14T18:18:51.385161+0000","last_peered":"2023-10-14T18:18:51.385161+0000","last_clean":"2023-10-14T18:18:51.385161+0000","last_became_active":"2023-10-14T18:18:50.316998+0000","last_became_peered":"2023-10-14T18:18:50.316998+0000","last_unstale":"2023-10-14T18:18:51.385161+0000","last_undegraded":"2023-10-14T18:18:51.385161+0000","last_fullsized":"2023-10-14T18:18:51.385161+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:29.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289794+0000","last_change":"2023-10-14T18:18:35.672986+0000","last_active":"2023-10-14T18:18:50.289794+0000","last_peered":"2023-10-14T18:18:50.289794+0000","last_clean":"2023-10-14T18:18:50.289794+0000","last_became_active":"2023-10-14T18:18:35.672683+0000","last_became_peered":"2023-10-14T18:18:35.672683+0000","last_unstale":"2023-10-14T18:18:50.289794+0000","last_undegraded":"2023-10-14T18:18:50.289794+0000","last_fullsized":"2023-10-14T18:18:50.289794+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:59.451214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.302019+0000","last_change":"2023-10-14T18:18:34.651378+0000","last_active":"2023-10-14T18:18:50.302019+0000","last_peered":"2023-10-14T18:18:50.302019+0000","last_clean":"2023-10-14T18:18:50.302019+0000","last_became_active":"2023-10-14T18:18:34.651072+0000","last_became_peered":"2023-10-14T18:18:34.651072+0000","last_unstale":"2023-10-14T18:18:50.302019+0000","last_undegraded":"2023-10-14T18:18:50.302019+0000","last_fullsized":"2023-10-14T18:18:50.302019+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:07:46.772700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299121+0000","last_change":"2023-10-14T18:18:50.319501+0000","last_active":"2023-10-14T18:18:51.299121+0000","last_peered":"2023-10-14T18:18:51.299121+0000","last_clean":"2023-10-14T18:18:51.299121+0000","last_became_active":"2023-10-14T18:18:50.319215+0000","last_became_peered":"2023-10-14T18:18:50.319215+0000","last_unstale":"2023-10-14T18:18:51.299121+0000","last_undegraded":"2023-10-14T18:18:51.299121+0000","last_fullsized":"2023-10-14T18:18:51.299121+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:14.599554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299541+0000","last_change":"2023-10-14T18:18:35.691610+0000","last_active":"2023-10-14T18:18:50.299541+0000","last_peered":"2023-10-14T18:18:50.299541+0000","last_clean":"2023-10-14T18:18:50.299541+0000","last_became_active":"2023-10-14T18:18:35.691317+0000","last_became_peered":"2023-10-14T18:18:35.691317+0000","last_unstale":"2023-10-14T18:18:50.299541+0000","last_undegraded":"2023-10-14T18:18:50.299541+0000","last_fullsized":"2023-10-14T18:18:50.299541+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:24.619829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299275+0000","last_change":"2023-10-14T18:18:34.652803+0000","last_active":"2023-10-14T18:18:50.299275+0000","last_peered":"2023-10-14T18:18:50.299275+0000","last_clean":"2023-10-14T18:18:50.299275+0000","last_became_active":"2023-10-14T18:18:34.647316+0000","last_became_peered":"2023-10-14T18:18:34.647316+0000","last_unstale":"2023-10-14T18:18:50.299275+0000","last_undegraded":"2023-10-14T18:18:50.299275+0000","last_fullsized":"2023-10-14T18:18:50.299275+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:00.238815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297053+0000","last_change":"2023-10-14T18:18:50.325101+0000","last_active":"2023-10-14T18:18:51.297053+0000","last_peered":"2023-10-14T18:18:51.297053+0000","last_clean":"2023-10-14T18:18:51.297053+0000","last_became_active":"2023-10-14T18:18:50.324711+0000","last_became_peered":"2023-10-14T18:18:50.324711+0000","last_unstale":"2023-10-14T18:18:51.297053+0000","last_undegraded":"2023-10-14T18:18:51.297053+0000","last_fullsized":"2023-10-14T18:18:51.297053+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:02.650232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310047+0000","last_change":"2023-10-14T18:18:35.678351+0000","last_active":"2023-10-14T18:18:50.310047+0000","last_peered":"2023-10-14T18:18:50.310047+0000","last_clean":"2023-10-14T18:18:50.310047+0000","last_became_active":"2023-10-14T18:18:35.678058+0000","last_became_peered":"2023-10-14T18:18:35.678058+0000","last_unstale":"2023-10-14T18:18:50.310047+0000","last_undegraded":"2023-10-14T18:18:50.310047+0000","last_fullsized":"2023-10-14T18:18:50.310047+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:50.521263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310329+0000","last_change":"2023-10-14T18:18:34.651512+0000","last_active":"2023-10-14T18:18:50.310329+0000","last_peered":"2023-10-14T18:18:50.310329+0000","last_clean":"2023-10-14T18:18:50.310329+0000","last_became_active":"2023-10-14T18:18:34.651205+0000","last_became_peered":"2023-10-14T18:18:34.651205+0000","last_unstale":"2023-10-14T18:18:50.310329+0000","last_undegraded":"2023-10-14T18:18:50.310329+0000","last_fullsized":"2023-10-14T18:18:50.310329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:11.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379266+0000","last_change":"2023-10-14T18:18:50.311991+0000","last_active":"2023-10-14T18:18:51.379266+0000","last_peered":"2023-10-14T18:18:51.379266+0000","last_clean":"2023-10-14T18:18:51.379266+0000","last_became_active":"2023-10-14T18:18:50.311717+0000","last_became_peered":"2023-10-14T18:18:50.311717+0000","last_unstale":"2023-10-14T18:18:51.379266+0000","last_undegraded":"2023-10-14T18:18:51.379266+0000","last_fullsized":"2023-10-14T18:18:51.379266+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:07.405144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295249+0000","last_change":"2023-10-14T18:18:35.677057+0000","last_active":"2023-10-14T18:18:50.295249+0000","last_peered":"2023-10-14T18:18:50.295249+0000","last_clean":"2023-10-14T18:18:50.295249+0000","last_became_active":"2023-10-14T18:18:35.676751+0000","last_became_peered":"2023-10-14T18:18:35.676751+0000","last_unstale":"2023-10-14T18:18:50.295249+0000","last_undegraded":"2023-10-14T18:18:50.295249+0000","last_fullsized":"2023-10-14T18:18:50.295249+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:14.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310267+0000","last_change":"2023-10-14T18:18:34.645198+0000","last_active":"2023-10-14T18:18:50.310267+0000","last_peered":"2023-10-14T18:18:50.310267+0000","last_clean":"2023-10-14T18:18:50.310267+0000","last_became_active":"2023-10-14T18:18:34.644896+0000","last_became_peered":"2023-10-14T18:18:34.644896+0000","last_unstale":"2023-10-14T18:18:50.310267+0000","last_undegraded":"2023-10-14T18:18:50.310267+0000","last_fullsized":"2023-10-14T18:18:50.310267+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:17.263536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379446+0000","last_change":"2023-10-14T18:18:50.322885+0000","last_active":"2023-10-14T18:18:51.379446+0000","last_peered":"2023-10-14T18:18:51.379446+0000","last_clean":"2023-10-14T18:18:51.379446+0000","last_became_active":"2023-10-14T18:18:50.320103+0000","last_became_peered":"2023-10-14T18:18:50.320103+0000","last_unstale":"2023-10-14T18:18:51.379446+0000","last_undegraded":"2023-10-14T18:18:51.379446+0000","last_fullsized":"2023-10-14T18:18:51.379446+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:48.473004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299115+0000","last_change":"2023-10-14T18:18:35.675636+0000","last_active":"2023-10-14T18:18:50.299115+0000","last_peered":"2023-10-14T18:18:50.299115+0000","last_clean":"2023-10-14T18:18:50.299115+0000","last_became_active":"2023-10-14T18:18:35.674527+0000","last_became_peered":"2023-10-14T18:18:35.674527+0000","last_unstale":"2023-10-14T18:18:50.299115+0000","last_undegraded":"2023-10-14T18:18:50.299115+0000","last_fullsized":"2023-10-14T18:18:50.299115+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:19.471165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297054+0000","last_change":"2023-10-14T18:18:34.640892+0000","last_active":"2023-10-14T18:18:50.297054+0000","last_peered":"2023-10-14T18:18:50.297054+0000","last_clean":"2023-10-14T18:18:50.297054+0000","last_became_active":"2023-10-14T18:18:34.639779+0000","last_became_peered":"2023-10-14T18:18:34.639779+0000","last_unstale":"2023-10-14T18:18:50.297054+0000","last_undegraded":"2023-10-14T18:18:50.297054+0000","last_fullsized":"2023-10-14T18:18:50.297054+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297776+0000","last_change":"2023-10-14T18:18:50.318782+0000","last_active":"2023-10-14T18:18:51.297776+0000","last_peered":"2023-10-14T18:18:51.297776+0000","last_clean":"2023-10-14T18:18:51.297776+0000","last_became_active":"2023-10-14T18:18:50.316788+0000","last_became_peered":"2023-10-14T18:18:50.316788+0000","last_unstale":"2023-10-14T18:18:51.297776+0000","last_undegraded":"2023-10-14T18:18:51.297776+0000","last_fullsized":"2023-10-14T18:18:51.297776+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:55.419359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292510+0000","last_change":"2023-10-14T18:18:35.685195+0000","last_active":"2023-10-14T18:18:50.292510+0000","last_peered":"2023-10-14T18:18:50.292510+0000","last_clean":"2023-10-14T18:18:50.292510+0000","last_became_active":"2023-10-14T18:18:35.682690+0000","last_became_peered":"2023-10-14T18:18:35.682690+0000","last_unstale":"2023-10-14T18:18:50.292510+0000","last_undegraded":"2023-10-14T18:18:50.292510+0000","last_fullsized":"2023-10-14T18:18:50.292510+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:54.132242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.315405+0000","last_change":"2023-10-14T18:18:34.651854+0000","last_active":"2023-10-14T18:18:50.315405+0000","last_peered":"2023-10-14T18:18:50.315405+0000","last_clean":"2023-10-14T18:18:50.315405+0000","last_became_active":"2023-10-14T18:18:34.651540+0000","last_became_peered":"2023-10-14T18:18:34.651540+0000","last_unstale":"2023-10-14T18:18:50.315405+0000","last_undegraded":"2023-10-14T18:18:50.315405+0000","last_fullsized":"2023-10-14T18:18:50.315405+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.752351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378455+0000","last_change":"2023-10-14T18:18:50.304986+0000","last_active":"2023-10-14T18:18:51.378455+0000","last_peered":"2023-10-14T18:18:51.378455+0000","last_clean":"2023-10-14T18:18:51.378455+0000","last_became_active":"2023-10-14T18:18:50.304686+0000","last_became_peered":"2023-10-14T18:18:50.304686+0000","last_unstale":"2023-10-14T18:18:51.378455+0000","last_undegraded":"2023-10-14T18:18:51.378455+0000","last_fullsized":"2023-10-14T18:18:51.378455+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:14.708984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309921+0000","last_change":"2023-10-14T18:18:35.679788+0000","last_active":"2023-10-14T18:18:50.309921+0000","last_peered":"2023-10-14T18:18:50.309921+0000","last_clean":"2023-10-14T18:18:50.309921+0000","last_became_active":"2023-10-14T18:18:35.679488+0000","last_became_peered":"2023-10-14T18:18:35.679488+0000","last_unstale":"2023-10-14T18:18:50.309921+0000","last_undegraded":"2023-10-14T18:18:50.309921+0000","last_fullsized":"2023-10-14T18:18:50.309921+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:39.475602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":19,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299129+0000","last_change":"2023-10-14T18:18:34.654465+0000","last_active":"2023-10-14T18:18:50.299129+0000","last_peered":"2023-10-14T18:18:50.299129+0000","last_clean":"2023-10-14T18:18:50.299129+0000","last_became_active":"2023-10-14T18:18:34.654111+0000","last_became_peered":"2023-10-14T18:18:34.654111+0000","last_unstale":"2023-10-14T18:18:50.299129+0000","last_undegraded":"2023-10-14T18:18:50.299129+0000","last_fullsized":"2023-10-14T18:18:50.299129+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:01.589582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379642+0000","last_change":"2023-10-14T18:18:50.325691+0000","last_active":"2023-10-14T18:18:51.379642+0000","last_peered":"2023-10-14T18:18:51.379642+0000","last_clean":"2023-10-14T18:18:51.379642+0000","last_became_active":"2023-10-14T18:18:50.325428+0000","last_became_peered":"2023-10-14T18:18:50.325428+0000","last_unstale":"2023-10-14T18:18:51.379642+0000","last_undegraded":"2023-10-14T18:18:51.379642+0000","last_fullsized":"2023-10-14T18:18:51.379642+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:07.405144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305345+0000","last_change":"2023-10-14T18:18:35.683637+0000","last_active":"2023-10-14T18:18:50.305345+0000","last_peered":"2023-10-14T18:18:50.305345+0000","last_clean":"2023-10-14T18:18:50.305345+0000","last_became_active":"2023-10-14T18:18:35.683235+0000","last_became_peered":"2023-10-14T18:18:35.683235+0000","last_unstale":"2023-10-14T18:18:50.305345+0000","last_undegraded":"2023-10-14T18:18:50.305345+0000","last_fullsized":"2023-10-14T18:18:50.305345+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"250'8","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298106+0000","last_change":"2023-10-14T18:18:34.652609+0000","last_active":"2023-10-14T18:18:50.298106+0000","last_peered":"2023-10-14T18:18:50.298106+0000","last_clean":"2023-10-14T18:18:50.298106+0000","last_became_active":"2023-10-14T18:18:34.646287+0000","last_became_peered":"2023-10-14T18:18:34.646287+0000","last_unstale":"2023-10-14T18:18:50.298106+0000","last_undegraded":"2023-10-14T18:18:50.298106+0000","last_fullsized":"2023-10-14T18:18:50.298106+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:09.725037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380378+0000","last_change":"2023-10-14T18:18:50.301171+0000","last_active":"2023-10-14T18:18:51.380378+0000","last_peered":"2023-10-14T18:18:51.380378+0000","last_clean":"2023-10-14T18:18:51.380378+0000","last_became_active":"2023-10-14T18:18:50.300424+0000","last_became_peered":"2023-10-14T18:18:50.300424+0000","last_unstale":"2023-10-14T18:18:51.380378+0000","last_undegraded":"2023-10-14T18:18:51.380378+0000","last_fullsized":"2023-10-14T18:18:51.380378+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:15:21.189453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295150+0000","last_change":"2023-10-14T18:18:35.672404+0000","last_active":"2023-10-14T18:18:50.295150+0000","last_peered":"2023-10-14T18:18:50.295150+0000","last_clean":"2023-10-14T18:18:50.295150+0000","last_became_active":"2023-10-14T18:18:35.672109+0000","last_became_peered":"2023-10-14T18:18:35.672109+0000","last_unstale":"2023-10-14T18:18:50.295150+0000","last_undegraded":"2023-10-14T18:18:50.295150+0000","last_fullsized":"2023-10-14T18:18:50.295150+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298850+0000","last_change":"2023-10-14T18:18:34.650921+0000","last_active":"2023-10-14T18:18:50.298850+0000","last_peered":"2023-10-14T18:18:50.298850+0000","last_clean":"2023-10-14T18:18:50.298850+0000","last_became_active":"2023-10-14T18:18:34.650170+0000","last_became_peered":"2023-10-14T18:18:34.650170+0000","last_unstale":"2023-10-14T18:18:50.298850+0000","last_undegraded":"2023-10-14T18:18:50.298850+0000","last_fullsized":"2023-10-14T18:18:50.298850+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:24.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299078+0000","last_change":"2023-10-14T18:18:50.332895+0000","last_active":"2023-10-14T18:18:51.299078+0000","last_peered":"2023-10-14T18:18:51.299078+0000","last_clean":"2023-10-14T18:18:51.299078+0000","last_became_active":"2023-10-14T18:18:50.332553+0000","last_became_peered":"2023-10-14T18:18:50.332553+0000","last_unstale":"2023-10-14T18:18:51.299078+0000","last_undegraded":"2023-10-14T18:18:51.299078+0000","last_fullsized":"2023-10-14T18:18:51.299078+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:20.776610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299636+0000","last_change":"2023-10-14T18:18:35.685387+0000","last_active":"2023-10-14T18:18:50.299636+0000","last_peered":"2023-10-14T18:18:50.299636+0000","last_clean":"2023-10-14T18:18:50.299636+0000","last_became_active":"2023-10-14T18:18:35.684624+0000","last_became_peered":"2023-10-14T18:18:35.684624+0000","last_unstale":"2023-10-14T18:18:50.299636+0000","last_undegraded":"2023-10-14T18:18:50.299636+0000","last_fullsized":"2023-10-14T18:18:50.299636+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:49.959769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.304536+0000","last_change":"2023-10-14T18:18:34.644335+0000","last_active":"2023-10-14T18:18:50.304536+0000","last_peered":"2023-10-14T18:18:50.304536+0000","last_clean":"2023-10-14T18:18:50.304536+0000","last_became_active":"2023-10-14T18:18:34.644052+0000","last_became_peered":"2023-10-14T18:18:34.644052+0000","last_unstale":"2023-10-14T18:18:50.304536+0000","last_undegraded":"2023-10-14T18:18:50.304536+0000","last_fullsized":"2023-10-14T18:18:50.304536+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:00.968233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299739+0000","last_change":"2023-10-14T18:18:50.320011+0000","last_active":"2023-10-14T18:18:51.299739+0000","last_peered":"2023-10-14T18:18:51.299739+0000","last_clean":"2023-10-14T18:18:51.299739+0000","last_became_active":"2023-10-14T18:18:50.319408+0000","last_became_peered":"2023-10-14T18:18:50.319408+0000","last_unstale":"2023-10-14T18:18:51.299739+0000","last_undegraded":"2023-10-14T18:18:51.299739+0000","last_fullsized":"2023-10-14T18:18:51.299739+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:34.121820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297072+0000","last_change":"2023-10-14T18:18:35.675619+0000","last_active":"2023-10-14T18:18:50.297072+0000","last_peered":"2023-10-14T18:18:50.297072+0000","last_clean":"2023-10-14T18:18:50.297072+0000","last_became_active":"2023-10-14T18:18:35.673589+0000","last_became_peered":"2023-10-14T18:18:35.673589+0000","last_unstale":"2023-10-14T18:18:50.297072+0000","last_undegraded":"2023-10-14T18:18:50.297072+0000","last_fullsized":"2023-10-14T18:18:50.297072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:37.559321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306235+0000","last_change":"2023-10-14T18:18:34.646943+0000","last_active":"2023-10-14T18:18:50.306235+0000","last_peered":"2023-10-14T18:18:50.306235+0000","last_clean":"2023-10-14T18:18:50.306235+0000","last_became_active":"2023-10-14T18:18:34.646630+0000","last_became_peered":"2023-10-14T18:18:34.646630+0000","last_unstale":"2023-10-14T18:18:50.306235+0000","last_undegraded":"2023-10-14T18:18:50.306235+0000","last_fullsized":"2023-10-14T18:18:50.306235+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:57.489531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.384519+0000","last_change":"2023-10-14T18:18:50.334156+0000","last_active":"2023-10-14T18:18:51.384519+0000","last_peered":"2023-10-14T18:18:51.384519+0000","last_clean":"2023-10-14T18:18:51.384519+0000","last_became_active":"2023-10-14T18:18:50.331713+0000","last_became_peered":"2023-10-14T18:18:50.331713+0000","last_unstale":"2023-10-14T18:18:51.384519+0000","last_undegraded":"2023-10-14T18:18:51.384519+0000","last_fullsized":"2023-10-14T18:18:51.384519+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:20.400011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305822+0000","last_change":"2023-10-14T18:18:35.678311+0000","last_active":"2023-10-14T18:18:50.305822+0000","last_peered":"2023-10-14T18:18:50.305822+0000","last_clean":"2023-10-14T18:18:50.305822+0000","last_became_active":"2023-10-14T18:18:35.678016+0000","last_became_peered":"2023-10-14T18:18:35.678016+0000","last_unstale":"2023-10-14T18:18:50.305822+0000","last_undegraded":"2023-10-14T18:18:50.305822+0000","last_fullsized":"2023-10-14T18:18:50.305822+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.287997+0000","last_change":"2023-10-14T18:18:34.648173+0000","last_active":"2023-10-14T18:18:50.287997+0000","last_peered":"2023-10-14T18:18:50.287997+0000","last_clean":"2023-10-14T18:18:50.287997+0000","last_became_active":"2023-10-14T18:18:34.647904+0000","last_became_peered":"2023-10-14T18:18:34.647904+0000","last_unstale":"2023-10-14T18:18:50.287997+0000","last_undegraded":"2023-10-14T18:18:50.287997+0000","last_fullsized":"2023-10-14T18:18:50.287997+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:49.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376877+0000","last_change":"2023-10-14T18:18:50.333569+0000","last_active":"2023-10-14T18:18:51.376877+0000","last_peered":"2023-10-14T18:18:51.376877+0000","last_clean":"2023-10-14T18:18:51.376877+0000","last_became_active":"2023-10-14T18:18:50.332171+0000","last_became_peered":"2023-10-14T18:18:50.332171+0000","last_unstale":"2023-10-14T18:18:51.376877+0000","last_undegraded":"2023-10-14T18:18:51.376877+0000","last_fullsized":"2023-10-14T18:18:51.376877+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:11.965757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.303586+0000","last_change":"2023-10-14T18:18:35.672233+0000","last_active":"2023-10-14T18:18:50.303586+0000","last_peered":"2023-10-14T18:18:50.303586+0000","last_clean":"2023-10-14T18:18:50.303586+0000","last_became_active":"2023-10-14T18:18:35.671904+0000","last_became_peered":"2023-10-14T18:18:35.671904+0000","last_unstale":"2023-10-14T18:18:50.303586+0000","last_undegraded":"2023-10-14T18:18:50.303586+0000","last_fullsized":"2023-10-14T18:18:50.303586+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:09.351543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296071+0000","last_change":"2023-10-14T18:18:34.644806+0000","last_active":"2023-10-14T18:18:50.296071+0000","last_peered":"2023-10-14T18:18:50.296071+0000","last_clean":"2023-10-14T18:18:50.296071+0000","last_became_active":"2023-10-14T18:18:34.642485+0000","last_became_peered":"2023-10-14T18:18:34.642485+0000","last_unstale":"2023-10-14T18:18:50.296071+0000","last_undegraded":"2023-10-14T18:18:50.296071+0000","last_fullsized":"2023-10-14T18:18:50.296071+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:22.234439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379990+0000","last_change":"2023-10-14T18:18:50.320259+0000","last_active":"2023-10-14T18:18:51.379990+0000","last_peered":"2023-10-14T18:18:51.379990+0000","last_clean":"2023-10-14T18:18:51.379990+0000","last_became_active":"2023-10-14T18:18:50.317254+0000","last_became_peered":"2023-10-14T18:18:50.317254+0000","last_unstale":"2023-10-14T18:18:51.379990+0000","last_undegraded":"2023-10-14T18:18:51.379990+0000","last_fullsized":"2023-10-14T18:18:51.379990+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:04.490143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309835+0000","last_change":"2023-10-14T18:18:35.673621+0000","last_active":"2023-10-14T18:18:50.309835+0000","last_peered":"2023-10-14T18:18:50.309835+0000","last_clean":"2023-10-14T18:18:50.309835+0000","last_became_active":"2023-10-14T18:18:35.673296+0000","last_became_peered":"2023-10-14T18:18:35.673296+0000","last_unstale":"2023-10-14T18:18:50.309835+0000","last_undegraded":"2023-10-14T18:18:50.309835+0000","last_fullsized":"2023-10-14T18:18:50.309835+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:58:27.889006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.314697+0000","last_change":"2023-10-14T18:18:34.655553+0000","last_active":"2023-10-14T18:18:50.314697+0000","last_peered":"2023-10-14T18:18:50.314697+0000","last_clean":"2023-10-14T18:18:50.314697+0000","last_became_active":"2023-10-14T18:18:34.655113+0000","last_became_peered":"2023-10-14T18:18:34.655113+0000","last_unstale":"2023-10-14T18:18:50.314697+0000","last_undegraded":"2023-10-14T18:18:50.314697+0000","last_fullsized":"2023-10-14T18:18:50.314697+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:04:45.880556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379869+0000","last_change":"2023-10-14T18:18:50.314689+0000","last_active":"2023-10-14T18:18:51.379869+0000","last_peered":"2023-10-14T18:18:51.379869+0000","last_clean":"2023-10-14T18:18:51.379869+0000","last_became_active":"2023-10-14T18:18:50.311321+0000","last_became_peered":"2023-10-14T18:18:50.311321+0000","last_unstale":"2023-10-14T18:18:51.379869+0000","last_undegraded":"2023-10-14T18:18:51.379869+0000","last_fullsized":"2023-10-14T18:18:51.379869+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:49.451546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299966+0000","last_change":"2023-10-14T18:18:35.683896+0000","last_active":"2023-10-14T18:18:50.299966+0000","last_peered":"2023-10-14T18:18:50.299966+0000","last_clean":"2023-10-14T18:18:50.299966+0000","last_became_active":"2023-10-14T18:18:35.683606+0000","last_became_peered":"2023-10-14T18:18:35.683606+0000","last_unstale":"2023-10-14T18:18:50.299966+0000","last_undegraded":"2023-10-14T18:18:50.299966+0000","last_fullsized":"2023-10-14T18:18:50.299966+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:02.619684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300401+0000","last_change":"2023-10-14T18:18:34.647730+0000","last_active":"2023-10-14T18:18:50.300401+0000","last_peered":"2023-10-14T18:18:50.300401+0000","last_clean":"2023-10-14T18:18:50.300401+0000","last_became_active":"2023-10-14T18:18:34.646698+0000","last_became_peered":"2023-10-14T18:18:34.646698+0000","last_unstale":"2023-10-14T18:18:50.300401+0000","last_undegraded":"2023-10-14T18:18:50.300401+0000","last_fullsized":"2023-10-14T18:18:50.300401+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:44.897768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382227+0000","last_change":"2023-10-14T18:18:50.333362+0000","last_active":"2023-10-14T18:18:51.382227+0000","last_peered":"2023-10-14T18:18:51.382227+0000","last_clean":"2023-10-14T18:18:51.382227+0000","last_became_active":"2023-10-14T18:18:50.333070+0000","last_became_peered":"2023-10-14T18:18:50.333070+0000","last_unstale":"2023-10-14T18:18:51.382227+0000","last_undegraded":"2023-10-14T18:18:51.382227+0000","last_fullsized":"2023-10-14T18:18:51.382227+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:54:01.495121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296780+0000","last_change":"2023-10-14T18:18:35.675449+0000","last_active":"2023-10-14T18:18:50.296780+0000","last_peered":"2023-10-14T18:18:50.296780+0000","last_clean":"2023-10-14T18:18:50.296780+0000","last_became_active":"2023-10-14T18:18:35.675129+0000","last_became_peered":"2023-10-14T18:18:35.675129+0000","last_unstale":"2023-10-14T18:18:50.296780+0000","last_undegraded":"2023-10-14T18:18:50.296780+0000","last_fullsized":"2023-10-14T18:18:50.296780+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291324+0000","last_change":"2023-10-14T18:18:34.645342+0000","last_active":"2023-10-14T18:18:50.291324+0000","last_peered":"2023-10-14T18:18:50.291324+0000","last_clean":"2023-10-14T18:18:50.291324+0000","last_became_active":"2023-10-14T18:18:34.643272+0000","last_became_peered":"2023-10-14T18:18:34.643272+0000","last_unstale":"2023-10-14T18:18:50.291324+0000","last_undegraded":"2023-10-14T18:18:50.291324+0000","last_fullsized":"2023-10-14T18:18:50.291324+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:08.339125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294594+0000","last_change":"2023-10-14T18:18:50.315771+0000","last_active":"2023-10-14T18:18:51.294594+0000","last_peered":"2023-10-14T18:18:51.294594+0000","last_clean":"2023-10-14T18:18:51.294594+0000","last_became_active":"2023-10-14T18:18:50.313621+0000","last_became_peered":"2023-10-14T18:18:50.313621+0000","last_unstale":"2023-10-14T18:18:51.294594+0000","last_undegraded":"2023-10-14T18:18:51.294594+0000","last_fullsized":"2023-10-14T18:18:51.294594+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:00.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292883+0000","last_change":"2023-10-14T18:18:35.675593+0000","last_active":"2023-10-14T18:18:50.292883+0000","last_peered":"2023-10-14T18:18:50.292883+0000","last_clean":"2023-10-14T18:18:50.292883+0000","last_became_active":"2023-10-14T18:18:35.675294+0000","last_became_peered":"2023-10-14T18:18:35.675294+0000","last_unstale":"2023-10-14T18:18:50.292883+0000","last_undegraded":"2023-10-14T18:18:50.292883+0000","last_fullsized":"2023-10-14T18:18:50.292883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306058+0000","last_change":"2023-10-14T18:18:34.640130+0000","last_active":"2023-10-14T18:18:50.306058+0000","last_peered":"2023-10-14T18:18:50.306058+0000","last_clean":"2023-10-14T18:18:50.306058+0000","last_became_active":"2023-10-14T18:18:34.639677+0000","last_became_peered":"2023-10-14T18:18:34.639677+0000","last_unstale":"2023-10-14T18:18:50.306058+0000","last_undegraded":"2023-10-14T18:18:50.306058+0000","last_fullsized":"2023-10-14T18:18:50.306058+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:09.948727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.298935+0000","last_change":"2023-10-14T18:18:50.312663+0000","last_active":"2023-10-14T18:18:51.298935+0000","last_peered":"2023-10-14T18:18:51.298935+0000","last_clean":"2023-10-14T18:18:51.298935+0000","last_became_active":"2023-10-14T18:18:50.312393+0000","last_became_peered":"2023-10-14T18:18:50.312393+0000","last_unstale":"2023-10-14T18:18:51.298935+0000","last_undegraded":"2023-10-14T18:18:51.298935+0000","last_fullsized":"2023-10-14T18:18:51.298935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:04.981289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298687+0000","last_change":"2023-10-14T18:18:35.682891+0000","last_active":"2023-10-14T18:18:50.298687+0000","last_peered":"2023-10-14T18:18:50.298687+0000","last_clean":"2023-10-14T18:18:50.298687+0000","last_became_active":"2023-10-14T18:18:35.681440+0000","last_became_peered":"2023-10-14T18:18:35.681440+0000","last_unstale":"2023-10-14T18:18:50.298687+0000","last_undegraded":"2023-10-14T18:18:50.298687+0000","last_fullsized":"2023-10-14T18:18:50.298687+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:50.330635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289955+0000","last_change":"2023-10-14T18:18:34.647137+0000","last_active":"2023-10-14T18:18:50.289955+0000","last_peered":"2023-10-14T18:18:50.289955+0000","last_clean":"2023-10-14T18:18:50.289955+0000","last_became_active":"2023-10-14T18:18:34.646766+0000","last_became_peered":"2023-10-14T18:18:34.646766+0000","last_unstale":"2023-10-14T18:18:50.289955+0000","last_undegraded":"2023-10-14T18:18:50.289955+0000","last_fullsized":"2023-10-14T18:18:50.289955+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299393+0000","last_change":"2023-10-14T18:18:50.327650+0000","last_active":"2023-10-14T18:18:51.299393+0000","last_peered":"2023-10-14T18:18:51.299393+0000","last_clean":"2023-10-14T18:18:51.299393+0000","last_became_active":"2023-10-14T18:18:50.327350+0000","last_became_peered":"2023-10-14T18:18:50.327350+0000","last_unstale":"2023-10-14T18:18:51.299393+0000","last_undegraded":"2023-10-14T18:18:51.299393+0000","last_fullsized":"2023-10-14T18:18:51.299393+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:30.605930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.288082+0000","last_change":"2023-10-14T18:18:35.672939+0000","last_active":"2023-10-14T18:18:50.288082+0000","last_peered":"2023-10-14T18:18:50.288082+0000","last_clean":"2023-10-14T18:18:50.288082+0000","last_became_active":"2023-10-14T18:18:35.672630+0000","last_became_peered":"2023-10-14T18:18:35.672630+0000","last_unstale":"2023-10-14T18:18:50.288082+0000","last_undegraded":"2023-10-14T18:18:50.288082+0000","last_fullsized":"2023-10-14T18:18:50.288082+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:56.149979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299108+0000","last_change":"2023-10-14T18:18:34.653097+0000","last_active":"2023-10-14T18:18:50.299108+0000","last_peered":"2023-10-14T18:18:50.299108+0000","last_clean":"2023-10-14T18:18:50.299108+0000","last_became_active":"2023-10-14T18:18:34.652338+0000","last_became_peered":"2023-10-14T18:18:34.652338+0000","last_unstale":"2023-10-14T18:18:50.299108+0000","last_undegraded":"2023-10-14T18:18:50.299108+0000","last_fullsized":"2023-10-14T18:18:50.299108+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:39.725847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381763+0000","last_change":"2023-10-14T18:18:50.305740+0000","last_active":"2023-10-14T18:18:51.381763+0000","last_peered":"2023-10-14T18:18:51.381763+0000","last_clean":"2023-10-14T18:18:51.381763+0000","last_became_active":"2023-10-14T18:18:50.305421+0000","last_became_peered":"2023-10-14T18:18:50.305421+0000","last_unstale":"2023-10-14T18:18:51.381763+0000","last_undegraded":"2023-10-14T18:18:51.381763+0000","last_fullsized":"2023-10-14T18:18:51.381763+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:49.451546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297163+0000","last_change":"2023-10-14T18:18:35.668420+0000","last_active":"2023-10-14T18:18:50.297163+0000","last_peered":"2023-10-14T18:18:50.297163+0000","last_clean":"2023-10-14T18:18:50.297163+0000","last_became_active":"2023-10-14T18:18:35.667429+0000","last_became_peered":"2023-10-14T18:18:35.667429+0000","last_unstale":"2023-10-14T18:18:50.297163+0000","last_undegraded":"2023-10-14T18:18:50.297163+0000","last_fullsized":"2023-10-14T18:18:50.297163+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295597+0000","last_change":"2023-10-14T18:18:34.649612+0000","last_active":"2023-10-14T18:18:50.295597+0000","last_peered":"2023-10-14T18:18:50.295597+0000","last_clean":"2023-10-14T18:18:50.295597+0000","last_became_active":"2023-10-14T18:18:34.649307+0000","last_became_peered":"2023-10-14T18:18:34.649307+0000","last_unstale":"2023-10-14T18:18:50.295597+0000","last_undegraded":"2023-10-14T18:18:50.295597+0000","last_fullsized":"2023-10-14T18:18:50.295597+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.438796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293382+0000","last_change":"2023-10-14T18:18:50.332902+0000","last_active":"2023-10-14T18:18:51.293382+0000","last_peered":"2023-10-14T18:18:51.293382+0000","last_clean":"2023-10-14T18:18:51.293382+0000","last_became_active":"2023-10-14T18:18:50.332581+0000","last_became_peered":"2023-10-14T18:18:50.332581+0000","last_unstale":"2023-10-14T18:18:51.293382+0000","last_undegraded":"2023-10-14T18:18:51.293382+0000","last_fullsized":"2023-10-14T18:18:51.293382+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:29.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309503+0000","last_change":"2023-10-14T18:18:35.666113+0000","last_active":"2023-10-14T18:18:50.309503+0000","last_peered":"2023-10-14T18:18:50.309503+0000","last_clean":"2023-10-14T18:18:50.309503+0000","last_became_active":"2023-10-14T18:18:35.665775+0000","last_became_peered":"2023-10-14T18:18:35.665775+0000","last_unstale":"2023-10-14T18:18:50.309503+0000","last_undegraded":"2023-10-14T18:18:50.309503+0000","last_fullsized":"2023-10-14T18:18:50.309503+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306460+0000","last_change":"2023-10-14T18:18:34.657934+0000","last_active":"2023-10-14T18:18:50.306460+0000","last_peered":"2023-10-14T18:18:50.306460+0000","last_clean":"2023-10-14T18:18:50.306460+0000","last_became_active":"2023-10-14T18:18:34.657614+0000","last_became_peered":"2023-10-14T18:18:34.657614+0000","last_unstale":"2023-10-14T18:18:50.306460+0000","last_undegraded":"2023-10-14T18:18:50.306460+0000","last_fullsized":"2023-10-14T18:18:50.306460+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:11.487790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297645+0000","last_change":"2023-10-14T18:18:50.315046+0000","last_active":"2023-10-14T18:18:51.297645+0000","last_peered":"2023-10-14T18:18:51.297645+0000","last_clean":"2023-10-14T18:18:51.297645+0000","last_became_active":"2023-10-14T18:18:50.314529+0000","last_became_peered":"2023-10-14T18:18:50.314529+0000","last_unstale":"2023-10-14T18:18:51.297645+0000","last_undegraded":"2023-10-14T18:18:51.297645+0000","last_fullsized":"2023-10-14T18:18:51.297645+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:17.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299198+0000","last_change":"2023-10-14T18:18:35.680237+0000","last_active":"2023-10-14T18:18:50.299198+0000","last_peered":"2023-10-14T18:18:50.299198+0000","last_clean":"2023-10-14T18:18:50.299198+0000","last_became_active":"2023-10-14T18:18:35.679960+0000","last_became_peered":"2023-10-14T18:18:35.679960+0000","last_unstale":"2023-10-14T18:18:50.299198+0000","last_undegraded":"2023-10-14T18:18:50.299198+0000","last_fullsized":"2023-10-14T18:18:50.299198+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:53.427626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310032+0000","last_change":"2023-10-14T18:18:34.651448+0000","last_active":"2023-10-14T18:18:50.310032+0000","last_peered":"2023-10-14T18:18:50.310032+0000","last_clean":"2023-10-14T18:18:50.310032+0000","last_became_active":"2023-10-14T18:18:34.651100+0000","last_became_peered":"2023-10-14T18:18:34.651100+0000","last_unstale":"2023-10-14T18:18:50.310032+0000","last_undegraded":"2023-10-14T18:18:50.310032+0000","last_fullsized":"2023-10-14T18:18:50.310032+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:41.231071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378244+0000","last_change":"2023-10-14T18:18:50.324334+0000","last_active":"2023-10-14T18:18:51.378244+0000","last_peered":"2023-10-14T18:18:51.378244+0000","last_clean":"2023-10-14T18:18:51.378244+0000","last_became_active":"2023-10-14T18:18:50.323321+0000","last_became_peered":"2023-10-14T18:18:50.323321+0000","last_unstale":"2023-10-14T18:18:51.378244+0000","last_undegraded":"2023-10-14T18:18:51.378244+0000","last_fullsized":"2023-10-14T18:18:51.378244+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:02.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291190+0000","last_change":"2023-10-14T18:18:35.687662+0000","last_active":"2023-10-14T18:18:50.291190+0000","last_peered":"2023-10-14T18:18:50.291190+0000","last_clean":"2023-10-14T18:18:50.291190+0000","last_became_active":"2023-10-14T18:18:35.687396+0000","last_became_peered":"2023-10-14T18:18:35.687396+0000","last_unstale":"2023-10-14T18:18:50.291190+0000","last_undegraded":"2023-10-14T18:18:50.291190+0000","last_fullsized":"2023-10-14T18:18:50.291190+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:15.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300488+0000","last_change":"2023-10-14T18:18:34.649374+0000","last_active":"2023-10-14T18:18:50.300488+0000","last_peered":"2023-10-14T18:18:50.300488+0000","last_clean":"2023-10-14T18:18:50.300488+0000","last_became_active":"2023-10-14T18:18:34.649074+0000","last_became_peered":"2023-10-14T18:18:34.649074+0000","last_unstale":"2023-10-14T18:18:50.300488+0000","last_undegraded":"2023-10-14T18:18:50.300488+0000","last_fullsized":"2023-10-14T18:18:50.300488+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:45.885592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.295315+0000","last_change":"2023-10-14T18:18:50.325218+0000","last_active":"2023-10-14T18:18:51.295315+0000","last_peered":"2023-10-14T18:18:51.295315+0000","last_clean":"2023-10-14T18:18:51.295315+0000","last_became_active":"2023-10-14T18:18:50.324903+0000","last_became_peered":"2023-10-14T18:18:50.324903+0000","last_unstale":"2023-10-14T18:18:51.295315+0000","last_undegraded":"2023-10-14T18:18:51.295315+0000","last_fullsized":"2023-10-14T18:18:51.295315+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:03:13.152603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292434+0000","last_change":"2023-10-14T18:18:35.682974+0000","last_active":"2023-10-14T18:18:50.292434+0000","last_peered":"2023-10-14T18:18:50.292434+0000","last_clean":"2023-10-14T18:18:50.292434+0000","last_became_active":"2023-10-14T18:18:35.682647+0000","last_became_peered":"2023-10-14T18:18:35.682647+0000","last_unstale":"2023-10-14T18:18:50.292434+0000","last_undegraded":"2023-10-14T18:18:50.292434+0000","last_fullsized":"2023-10-14T18:18:50.292434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:58.501949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291396+0000","last_change":"2023-10-14T18:18:34.656380+0000","last_active":"2023-10-14T18:18:50.291396+0000","last_peered":"2023-10-14T18:18:50.291396+0000","last_clean":"2023-10-14T18:18:50.291396+0000","last_became_active":"2023-10-14T18:18:34.656073+0000","last_became_peered":"2023-10-14T18:18:34.656073+0000","last_unstale":"2023-10-14T18:18:50.291396+0000","last_undegraded":"2023-10-14T18:18:50.291396+0000","last_fullsized":"2023-10-14T18:18:50.291396+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:41.992815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.296408+0000","last_change":"2023-10-14T18:18:50.322493+0000","last_active":"2023-10-14T18:18:51.296408+0000","last_peered":"2023-10-14T18:18:51.296408+0000","last_clean":"2023-10-14T18:18:51.296408+0000","last_became_active":"2023-10-14T18:18:50.320931+0000","last_became_peered":"2023-10-14T18:18:50.320931+0000","last_unstale":"2023-10-14T18:18:51.296408+0000","last_undegraded":"2023-10-14T18:18:51.296408+0000","last_fullsized":"2023-10-14T18:18:51.296408+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:08.782896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295295+0000","last_change":"2023-10-14T18:18:35.666278+0000","last_active":"2023-10-14T18:18:50.295295+0000","last_peered":"2023-10-14T18:18:50.295295+0000","last_clean":"2023-10-14T18:18:50.295295+0000","last_became_active":"2023-10-14T18:18:35.665963+0000","last_became_peered":"2023-10-14T18:18:35.665963+0000","last_unstale":"2023-10-14T18:18:50.295295+0000","last_undegraded":"2023-10-14T18:18:50.295295+0000","last_fullsized":"2023-10-14T18:18:50.295295+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295713+0000","last_change":"2023-10-14T18:18:34.649641+0000","last_active":"2023-10-14T18:18:50.295713+0000","last_peered":"2023-10-14T18:18:50.295713+0000","last_clean":"2023-10-14T18:18:50.295713+0000","last_became_active":"2023-10-14T18:18:34.649340+0000","last_became_peered":"2023-10-14T18:18:34.649340+0000","last_unstale":"2023-10-14T18:18:50.295713+0000","last_undegraded":"2023-10-14T18:18:50.295713+0000","last_fullsized":"2023-10-14T18:18:50.295713+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:05.759247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299308+0000","last_change":"2023-10-14T18:18:50.323121+0000","last_active":"2023-10-14T18:18:51.299308+0000","last_peered":"2023-10-14T18:18:51.299308+0000","last_clean":"2023-10-14T18:18:51.299308+0000","last_became_active":"2023-10-14T18:18:50.321364+0000","last_became_peered":"2023-10-14T18:18:50.321364+0000","last_unstale":"2023-10-14T18:18:51.299308+0000","last_undegraded":"2023-10-14T18:18:51.299308+0000","last_fullsized":"2023-10-14T18:18:51.299308+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:16.417405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293628+0000","last_change":"2023-10-14T18:18:35.675054+0000","last_active":"2023-10-14T18:18:50.293628+0000","last_peered":"2023-10-14T18:18:50.293628+0000","last_clean":"2023-10-14T18:18:50.293628+0000","last_became_active":"2023-10-14T18:18:35.674739+0000","last_became_peered":"2023-10-14T18:18:35.674739+0000","last_unstale":"2023-10-14T18:18:50.293628+0000","last_undegraded":"2023-10-14T18:18:50.293628+0000","last_fullsized":"2023-10-14T18:18:50.293628+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300733+0000","last_change":"2023-10-14T18:18:34.652028+0000","last_active":"2023-10-14T18:18:50.300733+0000","last_peered":"2023-10-14T18:18:50.300733+0000","last_clean":"2023-10-14T18:18:50.300733+0000","last_became_active":"2023-10-14T18:18:34.651757+0000","last_became_peered":"2023-10-14T18:18:34.651757+0000","last_unstale":"2023-10-14T18:18:50.300733+0000","last_undegraded":"2023-10-14T18:18:50.300733+0000","last_fullsized":"2023-10-14T18:18:50.300733+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377140+0000","last_change":"2023-10-14T18:18:50.321495+0000","last_active":"2023-10-14T18:18:51.377140+0000","last_peered":"2023-10-14T18:18:51.377140+0000","last_clean":"2023-10-14T18:18:51.377140+0000","last_became_active":"2023-10-14T18:18:50.320848+0000","last_became_peered":"2023-10-14T18:18:50.320848+0000","last_unstale":"2023-10-14T18:18:51.377140+0000","last_undegraded":"2023-10-14T18:18:51.377140+0000","last_fullsized":"2023-10-14T18:18:51.377140+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:20:49.661634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293751+0000","last_change":"2023-10-14T18:18:35.660261+0000","last_active":"2023-10-14T18:18:50.293751+0000","last_peered":"2023-10-14T18:18:50.293751+0000","last_clean":"2023-10-14T18:18:50.293751+0000","last_became_active":"2023-10-14T18:18:35.659486+0000","last_became_peered":"2023-10-14T18:18:35.659486+0000","last_unstale":"2023-10-14T18:18:50.293751+0000","last_undegraded":"2023-10-14T18:18:50.293751+0000","last_fullsized":"2023-10-14T18:18:50.293751+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306115+0000","last_change":"2023-10-14T18:18:34.643288+0000","last_active":"2023-10-14T18:18:50.306115+0000","last_peered":"2023-10-14T18:18:50.306115+0000","last_clean":"2023-10-14T18:18:50.306115+0000","last_became_active":"2023-10-14T18:18:34.642997+0000","last_became_peered":"2023-10-14T18:18:34.642997+0000","last_unstale":"2023-10-14T18:18:50.306115+0000","last_undegraded":"2023-10-14T18:18:50.306115+0000","last_fullsized":"2023-10-14T18:18:50.306115+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:39.884500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294652+0000","last_change":"2023-10-14T18:18:50.312703+0000","last_active":"2023-10-14T18:18:51.294652+0000","last_peered":"2023-10-14T18:18:51.294652+0000","last_clean":"2023-10-14T18:18:51.294652+0000","last_became_active":"2023-10-14T18:18:50.312145+0000","last_became_peered":"2023-10-14T18:18:50.312145+0000","last_unstale":"2023-10-14T18:18:51.294652+0000","last_undegraded":"2023-10-14T18:18:51.294652+0000","last_fullsized":"2023-10-14T18:18:51.294652+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:15.901887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295329+0000","last_change":"2023-10-14T18:18:35.674240+0000","last_active":"2023-10-14T18:18:50.295329+0000","last_peered":"2023-10-14T18:18:50.295329+0000","last_clean":"2023-10-14T18:18:50.295329+0000","last_became_active":"2023-10-14T18:18:35.671067+0000","last_became_peered":"2023-10-14T18:18:35.671067+0000","last_unstale":"2023-10-14T18:18:50.295329+0000","last_undegraded":"2023-10-14T18:18:50.295329+0000","last_fullsized":"2023-10-14T18:18:50.295329+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:02:59.948900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310453+0000","last_change":"2023-10-14T18:18:34.643687+0000","last_active":"2023-10-14T18:18:50.310453+0000","last_peered":"2023-10-14T18:18:50.310453+0000","last_clean":"2023-10-14T18:18:50.310453+0000","last_became_active":"2023-10-14T18:18:34.643092+0000","last_became_peered":"2023-10-14T18:18:34.643092+0000","last_unstale":"2023-10-14T18:18:50.310453+0000","last_undegraded":"2023-10-14T18:18:50.310453+0000","last_fullsized":"2023-10-14T18:18:50.310453+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.480279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297169+0000","last_change":"2023-10-14T18:18:50.305214+0000","last_active":"2023-10-14T18:18:51.297169+0000","last_peered":"2023-10-14T18:18:51.297169+0000","last_clean":"2023-10-14T18:18:51.297169+0000","last_became_active":"2023-10-14T18:18:50.304899+0000","last_became_peered":"2023-10-14T18:18:50.304899+0000","last_unstale":"2023-10-14T18:18:51.297169+0000","last_undegraded":"2023-10-14T18:18:51.297169+0000","last_fullsized":"2023-10-14T18:18:51.297169+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:27.178651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309927+0000","last_change":"2023-10-14T18:18:35.685028+0000","last_active":"2023-10-14T18:18:50.309927+0000","last_peered":"2023-10-14T18:18:50.309927+0000","last_clean":"2023-10-14T18:18:50.309927+0000","last_became_active":"2023-10-14T18:18:35.684687+0000","last_became_peered":"2023-10-14T18:18:35.684687+0000","last_unstale":"2023-10-14T18:18:50.309927+0000","last_undegraded":"2023-10-14T18:18:50.309927+0000","last_fullsized":"2023-10-14T18:18:50.309927+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:27.530239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309953+0000","last_change":"2023-10-14T18:18:34.640893+0000","last_active":"2023-10-14T18:18:50.309953+0000","last_peered":"2023-10-14T18:18:50.309953+0000","last_clean":"2023-10-14T18:18:50.309953+0000","last_became_active":"2023-10-14T18:18:34.639717+0000","last_became_peered":"2023-10-14T18:18:34.639717+0000","last_unstale":"2023-10-14T18:18:50.309953+0000","last_undegraded":"2023-10-14T18:18:50.309953+0000","last_fullsized":"2023-10-14T18:18:50.309953+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:55.441603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"235'370","reported_seq":1313,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375631+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:18:51.375631+0000","last_peered":"2023-10-14T18:18:51.375631+0000","last_clean":"2023-10-14T18:18:51.375631+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:18:51.375631+0000","last_undegraded":"2023-10-14T18:18:51.375631+0000","last_fullsized":"2023-10-14T18:18:51.375631+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576021,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57100,"kb_used_data":3708,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314740,"statfs":{"total":96636764160,"available":96578293760,"internally_reserved":0,"allocated":3796992,"data_stored":2186619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.177,"15min":1.2030000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":2.2149999999999999},"last":0.90400000000000003}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.768,"5min":1.9059999999999999,"15min":2.214},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.2909999999999999,"15min":1.3109999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.224}]},{"osd":2,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.206,"15min":1.26},"min":{"1min":0.83199999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.839,"5min":1.839,"15min":2.004},"last":1.294},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3100000000000001,"15min":1.365},"min":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.2010000000000001},"last":0.85099999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.347,"15min":1.4239999999999999},"min":{"1min":0.94799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":1.159},{"interface":"front","average":{"1min":1.454,"5min":1.488,"15min":1.528},"min":{"1min":1.0600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.994,"5min":1.994,"15min":2.2810000000000001},"last":1.03}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3680000000000001,"15min":1.343},"min":{"1min":0.90500000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.367},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3759999999999999,"15min":1.3919999999999999},"min":{"1min":1.0529999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.661},"last":1.403}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4370000000000001,"15min":1.4670000000000001},"min":{"1min":1.0409999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.516},"last":1.129},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.448},"last":1.5089999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.615,"15min":1.639},"min":{"1min":1.103,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.5289999999999999,"15min":1.577},"min":{"1min":1.1399999999999999,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.4690000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48208,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323632,"statfs":{"total":96636764160,"available":96587399168,"internally_reserved":0,"allocated":3211264,"data_stored":1596827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.167,"15min":1.1779999999999999},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.669,"5min":2.085,"15min":2.085},"last":1.901},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.2070000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.704,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.371}]},{"osd":1,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.208,"15min":1.24},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.494,"5min":1.762,"15min":1.776},"last":1.095},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.81799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.1760000000000002,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.659}]},{"osd":2,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3740000000000001,"15min":1.377},"min":{"1min":0.85999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.819,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.9399999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.397,"15min":1.419},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.704}]},{"osd":3,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.347,"15min":1.4059999999999999},"min":{"1min":0.72799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.2519999999999998},"last":1.532},{"interface":"front","average":{"1min":1.357,"5min":1.411,"15min":1.4359999999999999},"min":{"1min":0.91300000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8220000000000001,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.875}]},{"osd":4,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3899999999999999,"15min":1.3859999999999999},"min":{"1min":0.96499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7949999999999999,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":1.109,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0030000000000001,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.4970000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.411,"15min":1.419},"min":{"1min":1.024,"5min":0.624,"15min":0.624},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5780000000000001,"15min":1.5840000000000001},"min":{"1min":1.2310000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.5609999999999999},"last":2.0750000000000002}]},{"osd":7,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.6140000000000001,"15min":1.6180000000000001},"min":{"1min":1.403,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.194,"5min":2.903,"15min":3.423},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.583,"15min":1.6040000000000001},"min":{"1min":1.296,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.2349999999999999,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.972}]}]},{"osd":5,"up_from":13,"seq":55834576021,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42620,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329220,"statfs":{"total":96636764160,"available":96593121280,"internally_reserved":0,"allocated":3194880,"data_stored":1596111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1719999999999999,"15min":1.2070000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.1739999999999999,"15min":1.2350000000000001},"min":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.629,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.234,"15min":1.276},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6850000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.296,"15min":1.3100000000000001},"min":{"1min":0.92100000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1799999999999999,"15min":1.2430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.591,"5min":1.764,"15min":3.7090000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.377,"15min":1.397},"min":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":0.76000000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4159999999999999,"15min":1.46},"min":{"1min":0.94299999999999995,"5min":0.94199999999999995,"15min":0.88900000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.4339999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.8330000000000002},"last":0.88500000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3879999999999999,"15min":1.3660000000000001},"min":{"1min":1.0700000000000001,"5min":0.85899999999999999,"15min":0.78400000000000003},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.98},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4370000000000001,"15min":1.401},"min":{"1min":0.90400000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.516,"5min":2.516,"15min":3.1280000000000001},"last":0.85599999999999998}]},{"osd":6,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.399,"15min":1.4259999999999999},"min":{"1min":0.92900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":4.0540000000000003},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.6259999999999999,"15min":1.635},"min":{"1min":1.429,"5min":0.93000000000000005,"15min":0.81799999999999995},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":3.6360000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5609999999999999,"15min":1.585},"min":{"1min":1.121,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":3.2759999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5840000000000001,"15min":1.5680000000000001},"min":{"1min":1.1419999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.9390000000000001},"last":0.93999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576021,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3235840,"data_stored":1600728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.181,"15min":1.2010000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.266,"15min":1.2829999999999999},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":0.64800000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.282,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.59399999999999997},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.222}]},{"osd":2,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.286},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.524},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.389,"15min":1.4299999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.65700000000000003},"max":{"1min":1.597,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.2909999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":1.0029999999999999,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.6739999999999999,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":1.0009999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.204}]},{"osd":5,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.397},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.69299999999999995},"max":{"1min":1.923,"5min":1.9319999999999999,"15min":2.02},"last":1.389},{"interface":"front","average":{"1min":1.498,"5min":1.393,"15min":1.3839999999999999},"min":{"1min":0.90000000000000002,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.4990000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.452,"15min":1.4370000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.577,"15min":1.569},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82699999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.468}]},{"osd":7,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.5229999999999999,"15min":1.522},"min":{"1min":1.248,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.946,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6080000000000001,"15min":1.5940000000000001},"min":{"1min":1.2889999999999999,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":2.597,"5min":2.597,"15min":3.8610000000000002},"last":1.2689999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576021,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56536,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315304,"statfs":{"total":96636764160,"available":96578871296,"internally_reserved":0,"allocated":3215360,"data_stored":1598887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94399999999999995,"15min":0.97599999999999998},"min":{"1min":0.45700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.4239999999999999,"5min":1.4870000000000001,"15min":1.855},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1870000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.504,"5min":1.6399999999999999,"15min":1.9970000000000001},"last":1.0309999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94799999999999995,"15min":0.996},"min":{"1min":0.47099999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.363,"5min":1.835,"15min":1.919},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.1970000000000001},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.8560000000000001},"last":1.292}]},{"osd":2,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.339,"15min":1.4159999999999999},"min":{"1min":0.54900000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.5620000000000001,"5min":2.032,"15min":2.032},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.26,"15min":1.3640000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.429}]},{"osd":4,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3819999999999999,"15min":1.429},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9099999999999999,"5min":2.153,"15min":2.153},"last":1.579},{"interface":"front","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.478},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.639}]},{"osd":5,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4239999999999999,"15min":1.4550000000000001},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.758},{"interface":"front","average":{"1min":1.367,"5min":1.46,"15min":1.548},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.02,"5min":2.1360000000000001,"15min":2.891},"last":1.4810000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5700000000000001,"15min":1.621},"min":{"1min":0.97199999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.931},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.6339999999999999,"15min":1.677},"min":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.8149999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.532,"15min":1.599},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":3.153,"5min":3.153,"15min":3.153},"last":1.885},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.635,"15min":1.681},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.7110000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56512,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315328,"statfs":{"total":96636764160,"available":96578895872,"internally_reserved":0,"allocated":3198976,"data_stored":1596201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3029999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.98099999999999998,"15min":1.081},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.2629999999999999,"5min":1.4550000000000001,"15min":1.8660000000000001},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":1.0069999999999999},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1439999999999999,"5min":1.603,"15min":1.881},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.1679999999999999,"15min":1.212},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5780000000000001,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.96999999999999997}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.218,"15min":1.2749999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.3959999999999999,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.081,"5min":1.337,"15min":1.3779999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7490000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.0820000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.464},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.736,"5min":2.097,"15min":2.097},"last":1.738},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3300000000000001,"15min":1.365},"min":{"1min":0.72399999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.944,"5min":1.944,"15min":2.024},"last":1.9379999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.389,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.2839999999999998},"last":1.786},{"interface":"front","average":{"1min":1.339,"5min":1.577,"15min":1.583},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.0049999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.52,"15min":1.5289999999999999},"min":{"1min":0.80600000000000005,"5min":0.747,"15min":0.747},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":2.2690000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.556,"15min":1.603},"min":{"1min":0.91100000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1400000000000001,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":2.085}]},{"osd":7,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5780000000000001,"15min":1.5900000000000001},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":2.1309999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.633,"15min":1.675},"min":{"1min":0.97999999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0600000000000001,"5min":2.331,"15min":2.331},"last":2.0110000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576021,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52388,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3231744,"data_stored":1599868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.3400000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0169999999999999,"15min":1.1080000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1659999999999999,"5min":2.847,"15min":2.847},"last":1.238}]},{"osd":2,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.046},"min":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4590000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":1.492},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1799999999999999,"15min":1.254},"min":{"1min":0.67300000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6279999999999999,"5min":1.996,"15min":2.6859999999999999},"last":1.119}]},{"osd":3,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.2769999999999999,"15min":1.3700000000000001},"min":{"1min":0.66300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7370000000000001,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.879},{"interface":"front","average":{"1min":1.056,"5min":1.226,"15min":1.296},"min":{"1min":0.68899999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5609999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.429}]},{"osd":4,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.417,"15min":1.46},"min":{"1min":0.88300000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9259999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3740000000000001,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.1619999999999999,"5min":2.218,"15min":2.899},"last":1.3380000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.425,"15min":1.472},"min":{"1min":0.997,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":3.004},"last":1.173},{"interface":"front","average":{"1min":1.403,"5min":1.496,"15min":1.5569999999999999},"min":{"1min":0.90400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.5129999999999999},"last":1.5669999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.54,"15min":1.601},"min":{"1min":1.171,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0369999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.6220000000000001,"15min":1.623},"min":{"1min":1.1140000000000001,"5min":0.871,"15min":0.871},"max":{"1min":2.0449999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.6240000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.619,"15min":1.6890000000000001},"min":{"1min":1.0029999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.3290000000000002,"5min":3.363,"15min":3.363},"last":1.696},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.651,"15min":1.7090000000000001},"min":{"1min":1.05,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.206,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.964}]}]},{"osd":0,"up_from":13,"seq":55834576020,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48772,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323068,"statfs":{"total":96636764160,"available":96586821632,"internally_reserved":0,"allocated":3792896,"data_stored":2186513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.94599999999999995,"15min":0.97299999999999998},"min":{"1min":0.57699999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.367,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":1.52},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.55500000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3560000000000001,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.143,"15min":1.1759999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6379999999999999,"5min":1.855,"15min":1.855},"last":1.008},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.167,"15min":1.2390000000000001},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.522,"5min":1.845,"15min":2.5779999999999998},"last":1.2290000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.52100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.5429999999999999,"5min":1.804,"15min":2.0720000000000001},"last":1.099},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3300000000000001,"15min":1.391},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.77,"5min":1.883,"15min":2.194},"last":1.619}]},{"osd":4,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.284,"15min":1.3169999999999999},"min":{"1min":0.73999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5549999999999999,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.492,"15min":1.502},"min":{"1min":0.999,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.984,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.3169999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.327,"15min":1.3680000000000001},"min":{"1min":0.80800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.577},{"interface":"front","average":{"1min":1.429,"5min":1.51,"15min":1.569},"min":{"1min":0.78700000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.242,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.464}]},{"osd":6,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.6259999999999999,"15min":1.651},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.1709999999999998,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.4830000000000001,"15min":1.508},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6899999999999999,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.2709999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.6040000000000001,"15min":1.6479999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0649999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.5589999999999999,"15min":1.6160000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.857,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.411}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:18:54.834 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:18:55.056 INFO:tasks.ceph.mds.d.smithi186.stderr:2023-10-14T18:18:55.055+0000 7fdb3fdf5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:18:55.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.090+0000 7f9824850640 1 Processor -- start 2023-10-14T18:18:55.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.090+0000 7f9824850640 1 -- start start 2023-10-14T18:18:55.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f9824850640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f982006c040 0x7f982006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f9824850640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 0x7f982015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f9824850640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f982006c960 con 0x7f98200ad970 2023-10-14T18:18:55.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f9824850640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982015b9d0 con 0x7f982006c040 2023-10-14T18:18:55.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f9824850640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982015bcb0 con 0x7f98200cda80 2023-10-14T18:18:55.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f981ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f982006c040 0x7f982006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f981ed9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f982006c040 0x7f982006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40870/0 (socket says 172.21.15.29:40870) 2023-10-14T18:18:55.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.092+0000 7f981ed9d640 1 -- 172.21.15.29:0/48971210 learned_addr learned my addr 172.21.15.29:0/48971210 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:55.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981f59e640 1 --2- 172.21.15.29:0/48971210 >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 0x7f982015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981ed9d640 1 -- 172.21.15.29:0/48971210 >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 msgr2=0x7f982015ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981ed9d640 1 --2- 172.21.15.29:0/48971210 >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 0x7f982015ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981ed9d640 1 -- 172.21.15.29:0/48971210 >> v1:172.21.15.186:6789/0 conn(0x7f98200ad970 legacy=0x7f982015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981ed9d640 1 -- 172.21.15.29:0/48971210 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982015bf90 con 0x7f982006c040 2023-10-14T18:18:55.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.093+0000 7f981f59e640 1 --2- 172.21.15.29:0/48971210 >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 0x7f982015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.094+0000 7f981ed9d640 1 --2- 172.21.15.29:0/48971210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f982006c420 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f98200cf300 tx=0x7f981002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3cc9e1e6e12ba0f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.094+0000 7f981e59c640 1 -- 172.21.15.29:0/48971210 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810031870 con 0x7f982006c040 2023-10-14T18:18:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.094+0000 7f981e59c640 1 -- 172.21.15.29:0/48971210 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98100319d0 con 0x7f982006c040 2023-10-14T18:18:55.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.094+0000 7f981e59c640 1 -- 172.21.15.29:0/48971210 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810031d60 con 0x7f982006c040 2023-10-14T18:18:55.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 -- 172.21.15.29:0/48971210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 msgr2=0x7f982006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 --2- 172.21.15.29:0/48971210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f982006c420 secure :-1 s=READY pgs=2933 cs=0 l=1 rev1=1 crypto rx=0x7f98200cf300 tx=0x7f981002b910 comp rx=0 tx=0).stop 2023-10-14T18:18:55.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 -- 172.21.15.29:0/48971210 shutdown_connections 2023-10-14T18:18:55.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 --2- 172.21.15.29:0/48971210 >> v2:172.21.15.186:3300/0 conn(0x7f98200cda80 0x7f982015ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 --2- 172.21.15.29:0/48971210 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f982006c420 unknown :-1 s=CLOSED pgs=2933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 -- 172.21.15.29:0/48971210 >> 172.21.15.29:0/48971210 conn(0x7f98200ce6a0 msgr2=0x7f98200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 -- 172.21.15.29:0/48971210 shutdown_connections 2023-10-14T18:18:55.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.095+0000 7f9824850640 1 -- 172.21.15.29:0/48971210 wait complete. 2023-10-14T18:18:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.097+0000 7f9824850640 1 Processor -- start 2023-10-14T18:18:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.097+0000 7f9824850640 1 -- start start 2023-10-14T18:18:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98200ad970 0x7f98200ca530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200cda80 0x7f98200c73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c9070 con 0x7f982006c040 2023-10-14T18:18:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c9350 con 0x7f98200ad970 2023-10-14T18:18:55.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f9824850640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98200c9630 con 0x7f98200cda80 2023-10-14T18:18:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f981ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f981ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40886/0 (socket says 172.21.15.29:40886) 2023-10-14T18:18:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.098+0000 7f981ed9d640 1 -- 172.21.15.29:0/332270082 learned_addr learned my addr 172.21.15.29:0/332270082 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f9817fff640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98200ad970 0x7f98200ca530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981f59e640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200cda80 0x7f98200c73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 -- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200cda80 msgr2=0x7f98200c73d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200cda80 0x7f98200c73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 -- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98200ad970 msgr2=0x7f98200ca530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98200ad970 0x7f98200ca530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 -- 172.21.15.29:0/332270082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100094d0 con 0x7f982006c040 2023-10-14T18:18:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f981ed9d640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f9810000c00 tx=0x7f98100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f981003ad10 con 0x7f982006c040 2023-10-14T18:18:55.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981003ae70 con 0x7f982006c040 2023-10-14T18:18:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810044610 con 0x7f982006c040 2023-10-14T18:18:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.099+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98200c9910 con 0x7f982006c040 2023-10-14T18:18:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.100+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98200c8cf0 con 0x7f982006c040 2023-10-14T18:18:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.101+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982006b330 con 0x7f982006c040 2023-10-14T18:18:55.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.102+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f981004d450 con 0x7f982006c040 2023-10-14T18:18:55.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.103+0000 7f98177fe640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 0x7f980006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.103+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f981002d040 con 0x7f982006c040 2023-10-14T18:18:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.103+0000 7f9817fff640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 0x7f980006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.104+0000 7f9817fff640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 0x7f980006d570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98200ced00 tx=0x7f9808009040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.107+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9810076ea0 con 0x7f982006c040 2023-10-14T18:18:55.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.300+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f98200c94c0 con 0x7f980006b0b0 2023-10-14T18:18:55.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.339+0000 7f98177fe640 1 -- 172.21.15.29:0/332270082 <== mgr.7463 v2:172.21.15.29:6832/4275371095 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+526184 (secure 0 0 0) 0x7f98200c94c0 con 0x7f980006b0b0 2023-10-14T18:18:55.349 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:55.353 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:18:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.350+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 msgr2=0x7f980006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 0x7f980006d570 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f98200ced00 tx=0x7f9808009040 comp rx=0 tx=0).stop 2023-10-14T18:18:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 msgr2=0x7f98200acb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 secure :-1 s=READY pgs=2934 cs=0 l=1 rev1=1 crypto rx=0x7f9810000c00 tx=0x7f98100046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 shutdown_connections 2023-10-14T18:18:55.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f980006b0b0 0x7f980006d570 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98200cda80 0x7f98200c73d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98200ad970 0x7f98200ca530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 --2- 172.21.15.29:0/332270082 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f982006c040 0x7f98200acb00 unknown :-1 s=CLOSED pgs=2934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 >> 172.21.15.29:0/332270082 conn(0x7f98200ce6a0 msgr2=0x7f982015c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 shutdown_connections 2023-10-14T18:18:55.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.351+0000 7f9824850640 1 -- 172.21.15.29:0/332270082 wait complete. 2023-10-14T18:18:55.382 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":46,"stamp":"2023-10-14T18:18:54.430278+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596433,"num_objects":45,"num_object_clones":0,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":710,"num_write_kb":2446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":27,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":449,"ondisk_log_size":449,"up":386,"acting":386,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":360,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":428348,"kb_used_data":26248,"kb_used_omap":12,"kb_used_meta":401971,"kb_avail":754546372,"statfs":{"total":773094113280,"available":772655484928,"internally_reserved":0,"allocated":26877952,"data_stored":13961754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":411618872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":26,"apply_latency_ms":26,"commit_latency_ns":26000000,"apply_latency_ns":26000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001849"},"pg_stats":[{"pgid":"33.1","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.292498+0000","last_change":"2023-10-14T18:18:50.322788+0000","last_active":"2023-10-14T18:18:51.292498+0000","last_peered":"2023-10-14T18:18:51.292498+0000","last_clean":"2023-10-14T18:18:51.292498+0000","last_became_active":"2023-10-14T18:18:50.322523+0000","last_became_peered":"2023-10-14T18:18:50.322523+0000","last_unstale":"2023-10-14T18:18:51.292498+0000","last_undegraded":"2023-10-14T18:18:51.292498+0000","last_fullsized":"2023-10-14T18:18:51.292498+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:26.854472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.0","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380414+0000","last_change":"2023-10-14T18:18:49.317435+0000","last_active":"2023-10-14T18:18:51.380414+0000","last_peered":"2023-10-14T18:18:51.380414+0000","last_clean":"2023-10-14T18:18:51.380414+0000","last_became_active":"2023-10-14T18:18:49.317061+0000","last_became_peered":"2023-10-14T18:18:49.317061+0000","last_unstale":"2023-10-14T18:18:51.380414+0000","last_undegraded":"2023-10-14T18:18:51.380414+0000","last_fullsized":"2023-10-14T18:18:51.380414+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:38:03.543371+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298869+0000","last_change":"2023-10-14T18:18:35.681689+0000","last_active":"2023-10-14T18:18:50.298869+0000","last_peered":"2023-10-14T18:18:50.298869+0000","last_clean":"2023-10-14T18:18:50.298869+0000","last_became_active":"2023-10-14T18:18:35.681412+0000","last_became_peered":"2023-10-14T18:18:35.681412+0000","last_unstale":"2023-10-14T18:18:50.298869+0000","last_undegraded":"2023-10-14T18:18:50.298869+0000","last_fullsized":"2023-10-14T18:18:50.298869+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:30:00.717606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.0","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380166+0000","last_change":"2023-10-14T18:18:50.313424+0000","last_active":"2023-10-14T18:18:51.380166+0000","last_peered":"2023-10-14T18:18:51.380166+0000","last_clean":"2023-10-14T18:18:51.380166+0000","last_became_active":"2023-10-14T18:18:50.312904+0000","last_became_peered":"2023-10-14T18:18:50.312904+0000","last_unstale":"2023-10-14T18:18:51.380166+0000","last_undegraded":"2023-10-14T18:18:51.380166+0000","last_fullsized":"2023-10-14T18:18:51.380166+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:15.685379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380776+0000","last_change":"2023-10-14T18:18:49.305084+0000","last_active":"2023-10-14T18:18:51.380776+0000","last_peered":"2023-10-14T18:18:51.380776+0000","last_clean":"2023-10-14T18:18:51.380776+0000","last_became_active":"2023-10-14T18:18:49.304769+0000","last_became_peered":"2023-10-14T18:18:49.304769+0000","last_unstale":"2023-10-14T18:18:51.380776+0000","last_undegraded":"2023-10-14T18:18:51.380776+0000","last_fullsized":"2023-10-14T18:18:51.380776+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:20.673412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.3e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298852+0000","last_change":"2023-10-14T18:18:35.683822+0000","last_active":"2023-10-14T18:18:50.298852+0000","last_peered":"2023-10-14T18:18:50.298852+0000","last_clean":"2023-10-14T18:18:50.298852+0000","last_became_active":"2023-10-14T18:18:35.683492+0000","last_became_peered":"2023-10-14T18:18:35.683492+0000","last_unstale":"2023-10-14T18:18:50.298852+0000","last_undegraded":"2023-10-14T18:18:50.298852+0000","last_fullsized":"2023-10-14T18:18:50.298852+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:15.955276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.3","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380987+0000","last_change":"2023-10-14T18:18:50.306275+0000","last_active":"2023-10-14T18:18:51.380987+0000","last_peered":"2023-10-14T18:18:51.380987+0000","last_clean":"2023-10-14T18:18:51.380987+0000","last_became_active":"2023-10-14T18:18:50.305978+0000","last_became_peered":"2023-10-14T18:18:50.305978+0000","last_unstale":"2023-10-14T18:18:51.380987+0000","last_undegraded":"2023-10-14T18:18:51.380987+0000","last_fullsized":"2023-10-14T18:18:51.380987+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:17.252654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.2","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377710+0000","last_change":"2023-10-14T18:18:49.315666+0000","last_active":"2023-10-14T18:18:51.377710+0000","last_peered":"2023-10-14T18:18:51.377710+0000","last_clean":"2023-10-14T18:18:51.377710+0000","last_became_active":"2023-10-14T18:18:49.314033+0000","last_became_peered":"2023-10-14T18:18:49.314033+0000","last_unstale":"2023-10-14T18:18:51.377710+0000","last_undegraded":"2023-10-14T18:18:51.377710+0000","last_fullsized":"2023-10-14T18:18:51.377710+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:39.882042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.323727+0000","last_change":"2023-10-14T18:18:35.668307+0000","last_active":"2023-10-14T18:18:50.323727+0000","last_peered":"2023-10-14T18:18:50.323727+0000","last_clean":"2023-10-14T18:18:50.323727+0000","last_became_active":"2023-10-14T18:18:35.667786+0000","last_became_peered":"2023-10-14T18:18:35.667786+0000","last_unstale":"2023-10-14T18:18:50.323727+0000","last_undegraded":"2023-10-14T18:18:50.323727+0000","last_fullsized":"2023-10-14T18:18:50.323727+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:46:51.991520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380613+0000","last_change":"2023-10-14T18:18:50.323319+0000","last_active":"2023-10-14T18:18:51.380613+0000","last_peered":"2023-10-14T18:18:51.380613+0000","last_clean":"2023-10-14T18:18:51.380613+0000","last_became_active":"2023-10-14T18:18:50.323062+0000","last_became_peered":"2023-10-14T18:18:50.323062+0000","last_unstale":"2023-10-14T18:18:51.380613+0000","last_undegraded":"2023-10-14T18:18:51.380613+0000","last_fullsized":"2023-10-14T18:18:51.380613+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:23:29.332417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.3","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379784+0000","last_change":"2023-10-14T18:18:49.309269+0000","last_active":"2023-10-14T18:18:51.379784+0000","last_peered":"2023-10-14T18:18:51.379784+0000","last_clean":"2023-10-14T18:18:51.379784+0000","last_became_active":"2023-10-14T18:18:49.308989+0000","last_became_peered":"2023-10-14T18:18:49.308989+0000","last_unstale":"2023-10-14T18:18:51.379784+0000","last_undegraded":"2023-10-14T18:18:51.379784+0000","last_fullsized":"2023-10-14T18:18:51.379784+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:20.673412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.3c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.302743+0000","last_change":"2023-10-14T18:18:35.663594+0000","last_active":"2023-10-14T18:18:50.302743+0000","last_peered":"2023-10-14T18:18:50.302743+0000","last_clean":"2023-10-14T18:18:50.302743+0000","last_became_active":"2023-10-14T18:18:35.663236+0000","last_became_peered":"2023-10-14T18:18:35.663236+0000","last_unstale":"2023-10-14T18:18:50.302743+0000","last_undegraded":"2023-10-14T18:18:50.302743+0000","last_fullsized":"2023-10-14T18:18:50.302743+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:08:56.542420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.5","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299552+0000","last_change":"2023-10-14T18:18:50.316200+0000","last_active":"2023-10-14T18:18:51.299552+0000","last_peered":"2023-10-14T18:18:51.299552+0000","last_clean":"2023-10-14T18:18:51.299552+0000","last_became_active":"2023-10-14T18:18:50.314765+0000","last_became_peered":"2023-10-14T18:18:50.314765+0000","last_unstale":"2023-10-14T18:18:51.299552+0000","last_undegraded":"2023-10-14T18:18:51.299552+0000","last_fullsized":"2023-10-14T18:18:51.299552+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:34:05.900286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.4","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382176+0000","last_change":"2023-10-14T18:18:49.317710+0000","last_active":"2023-10-14T18:18:51.382176+0000","last_peered":"2023-10-14T18:18:51.382176+0000","last_clean":"2023-10-14T18:18:51.382176+0000","last_became_active":"2023-10-14T18:18:49.317378+0000","last_became_peered":"2023-10-14T18:18:49.317378+0000","last_unstale":"2023-10-14T18:18:51.382176+0000","last_undegraded":"2023-10-14T18:18:51.382176+0000","last_fullsized":"2023-10-14T18:18:51.382176+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:56:44.007835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.3b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309836+0000","last_change":"2023-10-14T18:18:35.670764+0000","last_active":"2023-10-14T18:18:50.309836+0000","last_peered":"2023-10-14T18:18:50.309836+0000","last_clean":"2023-10-14T18:18:50.309836+0000","last_became_active":"2023-10-14T18:18:35.670354+0000","last_became_peered":"2023-10-14T18:18:35.670354+0000","last_unstale":"2023-10-14T18:18:50.309836+0000","last_undegraded":"2023-10-14T18:18:50.309836+0000","last_fullsized":"2023-10-14T18:18:50.309836+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:32:43.950584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.4","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.296441+0000","last_change":"2023-10-14T18:18:50.313891+0000","last_active":"2023-10-14T18:18:51.296441+0000","last_peered":"2023-10-14T18:18:51.296441+0000","last_clean":"2023-10-14T18:18:51.296441+0000","last_became_active":"2023-10-14T18:18:50.313433+0000","last_became_peered":"2023-10-14T18:18:50.313433+0000","last_unstale":"2023-10-14T18:18:51.296441+0000","last_undegraded":"2023-10-14T18:18:51.296441+0000","last_fullsized":"2023-10-14T18:18:51.296441+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:35.969124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.5","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299167+0000","last_change":"2023-10-14T18:18:49.312323+0000","last_active":"2023-10-14T18:18:51.299167+0000","last_peered":"2023-10-14T18:18:51.299167+0000","last_clean":"2023-10-14T18:18:51.299167+0000","last_became_active":"2023-10-14T18:18:49.311700+0000","last_became_peered":"2023-10-14T18:18:49.311700+0000","last_unstale":"2023-10-14T18:18:51.299167+0000","last_undegraded":"2023-10-14T18:18:51.299167+0000","last_fullsized":"2023-10-14T18:18:51.299167+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:24.381483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.3a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300609+0000","last_change":"2023-10-14T18:18:35.682936+0000","last_active":"2023-10-14T18:18:50.300609+0000","last_peered":"2023-10-14T18:18:50.300609+0000","last_clean":"2023-10-14T18:18:50.300609+0000","last_became_active":"2023-10-14T18:18:35.682060+0000","last_became_peered":"2023-10-14T18:18:35.682060+0000","last_unstale":"2023-10-14T18:18:50.300609+0000","last_undegraded":"2023-10-14T18:18:50.300609+0000","last_fullsized":"2023-10-14T18:18:50.300609+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:08:31.696209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.7","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383878+0000","last_change":"2023-10-14T18:18:50.331470+0000","last_active":"2023-10-14T18:18:51.383878+0000","last_peered":"2023-10-14T18:18:51.383878+0000","last_clean":"2023-10-14T18:18:51.383878+0000","last_became_active":"2023-10-14T18:18:50.330696+0000","last_became_peered":"2023-10-14T18:18:50.330696+0000","last_unstale":"2023-10-14T18:18:51.383878+0000","last_undegraded":"2023-10-14T18:18:51.383878+0000","last_fullsized":"2023-10-14T18:18:51.383878+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:26.470083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.6","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297285+0000","last_change":"2023-10-14T18:18:49.305206+0000","last_active":"2023-10-14T18:18:51.297285+0000","last_peered":"2023-10-14T18:18:51.297285+0000","last_clean":"2023-10-14T18:18:51.297285+0000","last_became_active":"2023-10-14T18:18:49.304926+0000","last_became_peered":"2023-10-14T18:18:49.304926+0000","last_unstale":"2023-10-14T18:18:51.297285+0000","last_undegraded":"2023-10-14T18:18:51.297285+0000","last_fullsized":"2023-10-14T18:18:51.297285+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:20.415692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.39","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305731+0000","last_change":"2023-10-14T18:18:35.676700+0000","last_active":"2023-10-14T18:18:50.305731+0000","last_peered":"2023-10-14T18:18:50.305731+0000","last_clean":"2023-10-14T18:18:50.305731+0000","last_became_active":"2023-10-14T18:18:35.676423+0000","last_became_peered":"2023-10-14T18:18:35.676423+0000","last_unstale":"2023-10-14T18:18:50.305731+0000","last_undegraded":"2023-10-14T18:18:50.305731+0000","last_fullsized":"2023-10-14T18:18:50.305731+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.6","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378514+0000","last_change":"2023-10-14T18:18:50.317687+0000","last_active":"2023-10-14T18:18:51.378514+0000","last_peered":"2023-10-14T18:18:51.378514+0000","last_clean":"2023-10-14T18:18:51.378514+0000","last_became_active":"2023-10-14T18:18:50.316949+0000","last_became_peered":"2023-10-14T18:18:50.316949+0000","last_unstale":"2023-10-14T18:18:51.378514+0000","last_undegraded":"2023-10-14T18:18:51.378514+0000","last_fullsized":"2023-10-14T18:18:51.378514+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:14.708984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.7","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379514+0000","last_change":"2023-10-14T18:18:49.308266+0000","last_active":"2023-10-14T18:18:51.379514+0000","last_peered":"2023-10-14T18:18:51.379514+0000","last_clean":"2023-10-14T18:18:51.379514+0000","last_became_active":"2023-10-14T18:18:49.306582+0000","last_became_peered":"2023-10-14T18:18:49.306582+0000","last_unstale":"2023-10-14T18:18:51.379514+0000","last_undegraded":"2023-10-14T18:18:51.379514+0000","last_fullsized":"2023-10-14T18:18:51.379514+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:09.311289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.38","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293548+0000","last_change":"2023-10-14T18:18:35.664633+0000","last_active":"2023-10-14T18:18:50.293548+0000","last_peered":"2023-10-14T18:18:50.293548+0000","last_clean":"2023-10-14T18:18:50.293548+0000","last_became_active":"2023-10-14T18:18:35.664248+0000","last_became_peered":"2023-10-14T18:18:35.664248+0000","last_unstale":"2023-10-14T18:18:50.293548+0000","last_undegraded":"2023-10-14T18:18:50.293548+0000","last_fullsized":"2023-10-14T18:18:50.293548+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:22:21.318470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.9","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375837+0000","last_change":"2023-10-14T18:18:50.305367+0000","last_active":"2023-10-14T18:18:51.375837+0000","last_peered":"2023-10-14T18:18:51.375837+0000","last_clean":"2023-10-14T18:18:51.375837+0000","last_became_active":"2023-10-14T18:18:50.305092+0000","last_became_peered":"2023-10-14T18:18:50.305092+0000","last_unstale":"2023-10-14T18:18:51.375837+0000","last_undegraded":"2023-10-14T18:18:51.375837+0000","last_fullsized":"2023-10-14T18:18:51.375837+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:10.317916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.8","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293635+0000","last_change":"2023-10-14T18:18:49.310346+0000","last_active":"2023-10-14T18:18:51.293635+0000","last_peered":"2023-10-14T18:18:51.293635+0000","last_clean":"2023-10-14T18:18:51.293635+0000","last_became_active":"2023-10-14T18:18:49.310066+0000","last_became_peered":"2023-10-14T18:18:49.310066+0000","last_unstale":"2023-10-14T18:18:51.293635+0000","last_undegraded":"2023-10-14T18:18:51.293635+0000","last_fullsized":"2023-10-14T18:18:51.293635+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:09.985958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.37","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299473+0000","last_change":"2023-10-14T18:18:35.679660+0000","last_active":"2023-10-14T18:18:50.299473+0000","last_peered":"2023-10-14T18:18:50.299473+0000","last_clean":"2023-10-14T18:18:50.299473+0000","last_became_active":"2023-10-14T18:18:35.679374+0000","last_became_peered":"2023-10-14T18:18:35.679374+0000","last_unstale":"2023-10-14T18:18:50.299473+0000","last_undegraded":"2023-10-14T18:18:50.299473+0000","last_fullsized":"2023-10-14T18:18:50.299473+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:57.315103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.8","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380211+0000","last_change":"2023-10-14T18:18:50.313779+0000","last_active":"2023-10-14T18:18:51.380211+0000","last_peered":"2023-10-14T18:18:51.380211+0000","last_clean":"2023-10-14T18:18:51.380211+0000","last_became_active":"2023-10-14T18:18:50.313502+0000","last_became_peered":"2023-10-14T18:18:50.313502+0000","last_unstale":"2023-10-14T18:18:51.380211+0000","last_undegraded":"2023-10-14T18:18:51.380211+0000","last_fullsized":"2023-10-14T18:18:51.380211+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:41.753050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.9","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375761+0000","last_change":"2023-10-14T18:18:49.298802+0000","last_active":"2023-10-14T18:18:51.375761+0000","last_peered":"2023-10-14T18:18:51.375761+0000","last_clean":"2023-10-14T18:18:51.375761+0000","last_became_active":"2023-10-14T18:18:49.298073+0000","last_became_peered":"2023-10-14T18:18:49.298073+0000","last_unstale":"2023-10-14T18:18:51.375761+0000","last_undegraded":"2023-10-14T18:18:51.375761+0000","last_fullsized":"2023-10-14T18:18:51.375761+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:22:52.783786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.36","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309333+0000","last_change":"2023-10-14T18:18:35.679829+0000","last_active":"2023-10-14T18:18:50.309333+0000","last_peered":"2023-10-14T18:18:50.309333+0000","last_clean":"2023-10-14T18:18:50.309333+0000","last_became_active":"2023-10-14T18:18:35.679412+0000","last_became_peered":"2023-10-14T18:18:35.679412+0000","last_unstale":"2023-10-14T18:18:50.309333+0000","last_undegraded":"2023-10-14T18:18:50.309333+0000","last_fullsized":"2023-10-14T18:18:50.309333+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378447+0000","last_change":"2023-10-14T18:18:50.327269+0000","last_active":"2023-10-14T18:18:51.378447+0000","last_peered":"2023-10-14T18:18:51.378447+0000","last_clean":"2023-10-14T18:18:51.378447+0000","last_became_active":"2023-10-14T18:18:50.326946+0000","last_became_peered":"2023-10-14T18:18:50.326946+0000","last_unstale":"2023-10-14T18:18:51.378447+0000","last_undegraded":"2023-10-14T18:18:51.378447+0000","last_fullsized":"2023-10-14T18:18:51.378447+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:00:43.478859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.a","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293746+0000","last_change":"2023-10-14T18:18:49.315838+0000","last_active":"2023-10-14T18:18:51.293746+0000","last_peered":"2023-10-14T18:18:51.293746+0000","last_clean":"2023-10-14T18:18:51.293746+0000","last_became_active":"2023-10-14T18:18:49.315372+0000","last_became_peered":"2023-10-14T18:18:49.315372+0000","last_unstale":"2023-10-14T18:18:51.293746+0000","last_undegraded":"2023-10-14T18:18:51.293746+0000","last_fullsized":"2023-10-14T18:18:51.293746+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.483516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.35","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292157+0000","last_change":"2023-10-14T18:18:35.689722+0000","last_active":"2023-10-14T18:18:50.292157+0000","last_peered":"2023-10-14T18:18:50.292157+0000","last_clean":"2023-10-14T18:18:50.292157+0000","last_became_active":"2023-10-14T18:18:35.686665+0000","last_became_peered":"2023-10-14T18:18:35.686665+0000","last_unstale":"2023-10-14T18:18:50.292157+0000","last_undegraded":"2023-10-14T18:18:50.292157+0000","last_fullsized":"2023-10-14T18:18:50.292157+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:22:57.853905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378778+0000","last_change":"2023-10-14T18:18:50.312161+0000","last_active":"2023-10-14T18:18:51.378778+0000","last_peered":"2023-10-14T18:18:51.378778+0000","last_clean":"2023-10-14T18:18:51.378778+0000","last_became_active":"2023-10-14T18:18:50.310426+0000","last_became_peered":"2023-10-14T18:18:50.310426+0000","last_unstale":"2023-10-14T18:18:51.378778+0000","last_undegraded":"2023-10-14T18:18:51.378778+0000","last_fullsized":"2023-10-14T18:18:51.378778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:37:00.299094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.b","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.295588+0000","last_change":"2023-10-14T18:18:49.308030+0000","last_active":"2023-10-14T18:18:51.295588+0000","last_peered":"2023-10-14T18:18:51.295588+0000","last_clean":"2023-10-14T18:18:51.295588+0000","last_became_active":"2023-10-14T18:18:49.307402+0000","last_became_peered":"2023-10-14T18:18:49.307402+0000","last_unstale":"2023-10-14T18:18:51.295588+0000","last_undegraded":"2023-10-14T18:18:51.295588+0000","last_fullsized":"2023-10-14T18:18:51.295588+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:28.857963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.34","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295742+0000","last_change":"2023-10-14T18:18:35.674443+0000","last_active":"2023-10-14T18:18:50.295742+0000","last_peered":"2023-10-14T18:18:50.295742+0000","last_clean":"2023-10-14T18:18:50.295742+0000","last_became_active":"2023-10-14T18:18:35.673655+0000","last_became_peered":"2023-10-14T18:18:35.673655+0000","last_unstale":"2023-10-14T18:18:50.295742+0000","last_undegraded":"2023-10-14T18:18:50.295742+0000","last_fullsized":"2023-10-14T18:18:50.295742+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299308+0000","last_change":"2023-10-14T18:18:50.316952+0000","last_active":"2023-10-14T18:18:51.299308+0000","last_peered":"2023-10-14T18:18:51.299308+0000","last_clean":"2023-10-14T18:18:51.299308+0000","last_became_active":"2023-10-14T18:18:50.316669+0000","last_became_peered":"2023-10-14T18:18:50.316669+0000","last_unstale":"2023-10-14T18:18:51.299308+0000","last_undegraded":"2023-10-14T18:18:51.299308+0000","last_fullsized":"2023-10-14T18:18:51.299308+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:51.338036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"32.c","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385216+0000","last_change":"2023-10-14T18:18:49.313970+0000","last_active":"2023-10-14T18:18:51.385216+0000","last_peered":"2023-10-14T18:18:51.385216+0000","last_clean":"2023-10-14T18:18:51.385216+0000","last_became_active":"2023-10-14T18:18:49.313689+0000","last_became_peered":"2023-10-14T18:18:49.313689+0000","last_unstale":"2023-10-14T18:18:51.385216+0000","last_undegraded":"2023-10-14T18:18:51.385216+0000","last_fullsized":"2023-10-14T18:18:51.385216+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:55:51.768877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.33","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295540+0000","last_change":"2023-10-14T18:18:35.687221+0000","last_active":"2023-10-14T18:18:50.295540+0000","last_peered":"2023-10-14T18:18:50.295540+0000","last_clean":"2023-10-14T18:18:50.295540+0000","last_became_active":"2023-10-14T18:18:35.686900+0000","last_became_peered":"2023-10-14T18:18:35.686900+0000","last_unstale":"2023-10-14T18:18:50.295540+0000","last_undegraded":"2023-10-14T18:18:50.295540+0000","last_fullsized":"2023-10-14T18:18:50.295540+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:09:46.029025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385231+0000","last_change":"2023-10-14T18:18:50.321146+0000","last_active":"2023-10-14T18:18:51.385231+0000","last_peered":"2023-10-14T18:18:51.385231+0000","last_clean":"2023-10-14T18:18:51.385231+0000","last_became_active":"2023-10-14T18:18:50.318691+0000","last_became_peered":"2023-10-14T18:18:50.318691+0000","last_unstale":"2023-10-14T18:18:51.385231+0000","last_undegraded":"2023-10-14T18:18:51.385231+0000","last_fullsized":"2023-10-14T18:18:51.385231+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:25.388110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.d","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376614+0000","last_change":"2023-10-14T18:18:49.303213+0000","last_active":"2023-10-14T18:18:51.376614+0000","last_peered":"2023-10-14T18:18:51.376614+0000","last_clean":"2023-10-14T18:18:51.376614+0000","last_became_active":"2023-10-14T18:18:49.302764+0000","last_became_peered":"2023-10-14T18:18:49.302764+0000","last_unstale":"2023-10-14T18:18:51.376614+0000","last_undegraded":"2023-10-14T18:18:51.376614+0000","last_fullsized":"2023-10-14T18:18:51.376614+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:03.287251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.32","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299683+0000","last_change":"2023-10-14T18:18:35.671605+0000","last_active":"2023-10-14T18:18:50.299683+0000","last_peered":"2023-10-14T18:18:50.299683+0000","last_clean":"2023-10-14T18:18:50.299683+0000","last_became_active":"2023-10-14T18:18:35.671317+0000","last_became_peered":"2023-10-14T18:18:35.671317+0000","last_unstale":"2023-10-14T18:18:50.299683+0000","last_undegraded":"2023-10-14T18:18:50.299683+0000","last_fullsized":"2023-10-14T18:18:50.299683+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:14.496902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377102+0000","last_change":"2023-10-14T18:18:50.318042+0000","last_active":"2023-10-14T18:18:51.377102+0000","last_peered":"2023-10-14T18:18:51.377102+0000","last_clean":"2023-10-14T18:18:51.377102+0000","last_became_active":"2023-10-14T18:18:50.317689+0000","last_became_peered":"2023-10-14T18:18:50.317689+0000","last_unstale":"2023-10-14T18:18:51.377102+0000","last_undegraded":"2023-10-14T18:18:51.377102+0000","last_fullsized":"2023-10-14T18:18:51.377102+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:07:36.530334+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"32.e","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377653+0000","last_change":"2023-10-14T18:18:49.305425+0000","last_active":"2023-10-14T18:18:51.377653+0000","last_peered":"2023-10-14T18:18:51.377653+0000","last_clean":"2023-10-14T18:18:51.377653+0000","last_became_active":"2023-10-14T18:18:49.305158+0000","last_became_peered":"2023-10-14T18:18:49.305158+0000","last_unstale":"2023-10-14T18:18:51.377653+0000","last_undegraded":"2023-10-14T18:18:51.377653+0000","last_fullsized":"2023-10-14T18:18:51.377653+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:39.442155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.31","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298994+0000","last_change":"2023-10-14T18:18:35.682877+0000","last_active":"2023-10-14T18:18:50.298994+0000","last_peered":"2023-10-14T18:18:50.298994+0000","last_clean":"2023-10-14T18:18:50.298994+0000","last_became_active":"2023-10-14T18:18:35.682389+0000","last_became_peered":"2023-10-14T18:18:35.682389+0000","last_unstale":"2023-10-14T18:18:50.298994+0000","last_undegraded":"2023-10-14T18:18:50.298994+0000","last_fullsized":"2023-10-14T18:18:50.298994+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:58:25.798127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381081+0000","last_change":"2023-10-14T18:18:50.317867+0000","last_active":"2023-10-14T18:18:51.381081+0000","last_peered":"2023-10-14T18:18:51.381081+0000","last_clean":"2023-10-14T18:18:51.381081+0000","last_became_active":"2023-10-14T18:18:50.317418+0000","last_became_peered":"2023-10-14T18:18:50.317418+0000","last_unstale":"2023-10-14T18:18:51.381081+0000","last_undegraded":"2023-10-14T18:18:51.381081+0000","last_fullsized":"2023-10-14T18:18:51.381081+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:44:51.597540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.f","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294269+0000","last_change":"2023-10-14T18:18:49.314303+0000","last_active":"2023-10-14T18:18:51.294269+0000","last_peered":"2023-10-14T18:18:51.294269+0000","last_clean":"2023-10-14T18:18:51.294269+0000","last_became_active":"2023-10-14T18:18:49.314025+0000","last_became_peered":"2023-10-14T18:18:49.314025+0000","last_unstale":"2023-10-14T18:18:51.294269+0000","last_undegraded":"2023-10-14T18:18:51.294269+0000","last_fullsized":"2023-10-14T18:18:51.294269+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:39.490952+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.30","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309959+0000","last_change":"2023-10-14T18:18:35.674408+0000","last_active":"2023-10-14T18:18:50.309959+0000","last_peered":"2023-10-14T18:18:50.309959+0000","last_clean":"2023-10-14T18:18:50.309959+0000","last_became_active":"2023-10-14T18:18:35.674062+0000","last_became_peered":"2023-10-14T18:18:35.674062+0000","last_unstale":"2023-10-14T18:18:50.309959+0000","last_undegraded":"2023-10-14T18:18:50.309959+0000","last_fullsized":"2023-10-14T18:18:50.309959+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:41:16.769480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.11","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376778+0000","last_change":"2023-10-14T18:18:50.322490+0000","last_active":"2023-10-14T18:18:51.376778+0000","last_peered":"2023-10-14T18:18:51.376778+0000","last_clean":"2023-10-14T18:18:51.376778+0000","last_became_active":"2023-10-14T18:18:50.322222+0000","last_became_peered":"2023-10-14T18:18:50.322222+0000","last_unstale":"2023-10-14T18:18:51.376778+0000","last_undegraded":"2023-10-14T18:18:51.376778+0000","last_fullsized":"2023-10-14T18:18:51.376778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:16:44.047723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.10","version":"253'8","reported_seq":20,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377489+0000","last_change":"2023-10-14T18:18:49.311665+0000","last_active":"2023-10-14T18:18:51.377489+0000","last_peered":"2023-10-14T18:18:51.377489+0000","last_clean":"2023-10-14T18:18:51.377489+0000","last_became_active":"2023-10-14T18:18:49.311417+0000","last_became_peered":"2023-10-14T18:18:49.311417+0000","last_unstale":"2023-10-14T18:18:51.377489+0000","last_undegraded":"2023-10-14T18:18:51.377489+0000","last_fullsized":"2023-10-14T18:18:51.377489+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:21:21.635177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296956+0000","last_change":"2023-10-14T18:18:35.678586+0000","last_active":"2023-10-14T18:18:50.296956+0000","last_peered":"2023-10-14T18:18:50.296956+0000","last_clean":"2023-10-14T18:18:50.296956+0000","last_became_active":"2023-10-14T18:18:35.678265+0000","last_became_peered":"2023-10-14T18:18:35.678265+0000","last_unstale":"2023-10-14T18:18:50.296956+0000","last_undegraded":"2023-10-14T18:18:50.296956+0000","last_fullsized":"2023-10-14T18:18:50.296956+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:25.316716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.10","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379803+0000","last_change":"2023-10-14T18:18:50.320388+0000","last_active":"2023-10-14T18:18:51.379803+0000","last_peered":"2023-10-14T18:18:51.379803+0000","last_clean":"2023-10-14T18:18:51.379803+0000","last_became_active":"2023-10-14T18:18:50.319785+0000","last_became_peered":"2023-10-14T18:18:50.319785+0000","last_unstale":"2023-10-14T18:18:51.379803+0000","last_undegraded":"2023-10-14T18:18:51.379803+0000","last_fullsized":"2023-10-14T18:18:51.379803+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:28:16.781111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.11","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376063+0000","last_change":"2023-10-14T18:18:49.305027+0000","last_active":"2023-10-14T18:18:51.376063+0000","last_peered":"2023-10-14T18:18:51.376063+0000","last_clean":"2023-10-14T18:18:51.376063+0000","last_became_active":"2023-10-14T18:18:49.304531+0000","last_became_peered":"2023-10-14T18:18:49.304531+0000","last_unstale":"2023-10-14T18:18:51.376063+0000","last_undegraded":"2023-10-14T18:18:51.376063+0000","last_fullsized":"2023-10-14T18:18:51.376063+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:06:32.235368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.2e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298994+0000","last_change":"2023-10-14T18:18:35.679271+0000","last_active":"2023-10-14T18:18:50.298994+0000","last_peered":"2023-10-14T18:18:50.298994+0000","last_clean":"2023-10-14T18:18:50.298994+0000","last_became_active":"2023-10-14T18:18:35.678924+0000","last_became_peered":"2023-10-14T18:18:35.678924+0000","last_unstale":"2023-10-14T18:18:50.298994+0000","last_undegraded":"2023-10-14T18:18:50.298994+0000","last_fullsized":"2023-10-14T18:18:50.298994+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:54:56.833692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.13","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378103+0000","last_change":"2023-10-14T18:18:50.312738+0000","last_active":"2023-10-14T18:18:51.378103+0000","last_peered":"2023-10-14T18:18:51.378103+0000","last_clean":"2023-10-14T18:18:51.378103+0000","last_became_active":"2023-10-14T18:18:50.311095+0000","last_became_peered":"2023-10-14T18:18:50.311095+0000","last_unstale":"2023-10-14T18:18:51.378103+0000","last_undegraded":"2023-10-14T18:18:51.378103+0000","last_fullsized":"2023-10-14T18:18:51.378103+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:34.190468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.12","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378683+0000","last_change":"2023-10-14T18:18:49.310720+0000","last_active":"2023-10-14T18:18:51.378683+0000","last_peered":"2023-10-14T18:18:51.378683+0000","last_clean":"2023-10-14T18:18:51.378683+0000","last_became_active":"2023-10-14T18:18:49.310133+0000","last_became_peered":"2023-10-14T18:18:49.310133+0000","last_unstale":"2023-10-14T18:18:51.378683+0000","last_undegraded":"2023-10-14T18:18:51.378683+0000","last_fullsized":"2023-10-14T18:18:51.378683+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:04:38.960744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.2d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296506+0000","last_change":"2023-10-14T18:18:35.683679+0000","last_active":"2023-10-14T18:18:50.296506+0000","last_peered":"2023-10-14T18:18:50.296506+0000","last_clean":"2023-10-14T18:18:50.296506+0000","last_became_active":"2023-10-14T18:18:35.682861+0000","last_became_peered":"2023-10-14T18:18:35.682861+0000","last_unstale":"2023-10-14T18:18:50.296506+0000","last_undegraded":"2023-10-14T18:18:50.296506+0000","last_fullsized":"2023-10-14T18:18:50.296506+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:41:26.238015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.12","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379795+0000","last_change":"2023-10-14T18:18:50.322584+0000","last_active":"2023-10-14T18:18:51.379795+0000","last_peered":"2023-10-14T18:18:51.379795+0000","last_clean":"2023-10-14T18:18:51.379795+0000","last_became_active":"2023-10-14T18:18:50.322310+0000","last_became_peered":"2023-10-14T18:18:50.322310+0000","last_unstale":"2023-10-14T18:18:51.379795+0000","last_undegraded":"2023-10-14T18:18:51.379795+0000","last_fullsized":"2023-10-14T18:18:51.379795+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:35.835633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.13","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293169+0000","last_change":"2023-10-14T18:18:49.314312+0000","last_active":"2023-10-14T18:18:51.293169+0000","last_peered":"2023-10-14T18:18:51.293169+0000","last_clean":"2023-10-14T18:18:51.293169+0000","last_became_active":"2023-10-14T18:18:49.313429+0000","last_became_peered":"2023-10-14T18:18:49.313429+0000","last_unstale":"2023-10-14T18:18:51.293169+0000","last_undegraded":"2023-10-14T18:18:51.293169+0000","last_fullsized":"2023-10-14T18:18:51.293169+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:23:28.325790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.2c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309809+0000","last_change":"2023-10-14T18:18:35.675043+0000","last_active":"2023-10-14T18:18:50.309809+0000","last_peered":"2023-10-14T18:18:50.309809+0000","last_clean":"2023-10-14T18:18:50.309809+0000","last_became_active":"2023-10-14T18:18:35.674231+0000","last_became_peered":"2023-10-14T18:18:35.674231+0000","last_unstale":"2023-10-14T18:18:50.309809+0000","last_undegraded":"2023-10-14T18:18:50.309809+0000","last_fullsized":"2023-10-14T18:18:50.309809+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:11:18.531967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.15","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383396+0000","last_change":"2023-10-14T18:18:50.330890+0000","last_active":"2023-10-14T18:18:51.383396+0000","last_peered":"2023-10-14T18:18:51.383396+0000","last_clean":"2023-10-14T18:18:51.383396+0000","last_became_active":"2023-10-14T18:18:50.330415+0000","last_became_peered":"2023-10-14T18:18:50.330415+0000","last_unstale":"2023-10-14T18:18:51.383396+0000","last_undegraded":"2023-10-14T18:18:51.383396+0000","last_fullsized":"2023-10-14T18:18:51.383396+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:38:02.625632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.14","version":"253'6","reported_seq":18,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381956+0000","last_change":"2023-10-14T18:18:49.311559+0000","last_active":"2023-10-14T18:18:51.381956+0000","last_peered":"2023-10-14T18:18:51.381956+0000","last_clean":"2023-10-14T18:18:51.381956+0000","last_became_active":"2023-10-14T18:18:49.311286+0000","last_became_peered":"2023-10-14T18:18:49.311286+0000","last_unstale":"2023-10-14T18:18:51.381956+0000","last_undegraded":"2023-10-14T18:18:51.381956+0000","last_fullsized":"2023-10-14T18:18:51.381956+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:32:17.160345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.2b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293487+0000","last_change":"2023-10-14T18:18:35.678709+0000","last_active":"2023-10-14T18:18:50.293487+0000","last_peered":"2023-10-14T18:18:50.293487+0000","last_clean":"2023-10-14T18:18:50.293487+0000","last_became_active":"2023-10-14T18:18:35.678445+0000","last_became_peered":"2023-10-14T18:18:35.678445+0000","last_unstale":"2023-10-14T18:18:50.293487+0000","last_undegraded":"2023-10-14T18:18:50.293487+0000","last_fullsized":"2023-10-14T18:18:50.293487+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.14","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385067+0000","last_change":"2023-10-14T18:18:50.309163+0000","last_active":"2023-10-14T18:18:51.385067+0000","last_peered":"2023-10-14T18:18:51.385067+0000","last_clean":"2023-10-14T18:18:51.385067+0000","last_became_active":"2023-10-14T18:18:50.308887+0000","last_became_peered":"2023-10-14T18:18:50.308887+0000","last_unstale":"2023-10-14T18:18:51.385067+0000","last_undegraded":"2023-10-14T18:18:51.385067+0000","last_fullsized":"2023-10-14T18:18:51.385067+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:40.497579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.15","version":"253'3","reported_seq":15,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381833+0000","last_change":"2023-10-14T18:18:49.315656+0000","last_active":"2023-10-14T18:18:51.381833+0000","last_peered":"2023-10-14T18:18:51.381833+0000","last_clean":"2023-10-14T18:18:51.381833+0000","last_became_active":"2023-10-14T18:18:49.314830+0000","last_became_peered":"2023-10-14T18:18:49.314830+0000","last_unstale":"2023-10-14T18:18:51.381833+0000","last_undegraded":"2023-10-14T18:18:51.381833+0000","last_fullsized":"2023-10-14T18:18:51.381833+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:16.263712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.2a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295024+0000","last_change":"2023-10-14T18:18:35.678715+0000","last_active":"2023-10-14T18:18:50.295024+0000","last_peered":"2023-10-14T18:18:50.295024+0000","last_clean":"2023-10-14T18:18:50.295024+0000","last_became_active":"2023-10-14T18:18:35.678366+0000","last_became_peered":"2023-10-14T18:18:35.678366+0000","last_unstale":"2023-10-14T18:18:50.295024+0000","last_undegraded":"2023-10-14T18:18:50.295024+0000","last_fullsized":"2023-10-14T18:18:50.295024+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.17","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383713+0000","last_change":"2023-10-14T18:18:50.334460+0000","last_active":"2023-10-14T18:18:51.383713+0000","last_peered":"2023-10-14T18:18:51.383713+0000","last_clean":"2023-10-14T18:18:51.383713+0000","last_became_active":"2023-10-14T18:18:50.332701+0000","last_became_peered":"2023-10-14T18:18:50.332701+0000","last_unstale":"2023-10-14T18:18:51.383713+0000","last_undegraded":"2023-10-14T18:18:51.383713+0000","last_fullsized":"2023-10-14T18:18:51.383713+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:32:29.252321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"32.16","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377401+0000","last_change":"2023-10-14T18:18:49.315616+0000","last_active":"2023-10-14T18:18:51.377401+0000","last_peered":"2023-10-14T18:18:51.377401+0000","last_clean":"2023-10-14T18:18:51.377401+0000","last_became_active":"2023-10-14T18:18:49.315272+0000","last_became_peered":"2023-10-14T18:18:49.315272+0000","last_unstale":"2023-10-14T18:18:51.377401+0000","last_undegraded":"2023-10-14T18:18:51.377401+0000","last_fullsized":"2023-10-14T18:18:51.377401+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:00.542037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.29","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297055+0000","last_change":"2023-10-14T18:18:35.674737+0000","last_active":"2023-10-14T18:18:50.297055+0000","last_peered":"2023-10-14T18:18:50.297055+0000","last_clean":"2023-10-14T18:18:50.297055+0000","last_became_active":"2023-10-14T18:18:35.673762+0000","last_became_peered":"2023-10-14T18:18:35.673762+0000","last_unstale":"2023-10-14T18:18:50.297055+0000","last_undegraded":"2023-10-14T18:18:50.297055+0000","last_fullsized":"2023-10-14T18:18:50.297055+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:21:47.999578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.16","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299514+0000","last_change":"2023-10-14T18:18:50.320254+0000","last_active":"2023-10-14T18:18:51.299514+0000","last_peered":"2023-10-14T18:18:51.299514+0000","last_clean":"2023-10-14T18:18:51.299514+0000","last_became_active":"2023-10-14T18:18:50.319744+0000","last_became_peered":"2023-10-14T18:18:50.319744+0000","last_unstale":"2023-10-14T18:18:51.299514+0000","last_undegraded":"2023-10-14T18:18:51.299514+0000","last_fullsized":"2023-10-14T18:18:51.299514+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:55:11.484346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.17","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.384170+0000","last_change":"2023-10-14T18:18:49.314981+0000","last_active":"2023-10-14T18:18:51.384170+0000","last_peered":"2023-10-14T18:18:51.384170+0000","last_clean":"2023-10-14T18:18:51.384170+0000","last_became_active":"2023-10-14T18:18:49.314590+0000","last_became_peered":"2023-10-14T18:18:49.314590+0000","last_unstale":"2023-10-14T18:18:51.384170+0000","last_undegraded":"2023-10-14T18:18:51.384170+0000","last_fullsized":"2023-10-14T18:18:51.384170+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:48:25.658536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.28","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296111+0000","last_change":"2023-10-14T18:18:35.679931+0000","last_active":"2023-10-14T18:18:50.296111+0000","last_peered":"2023-10-14T18:18:50.296111+0000","last_clean":"2023-10-14T18:18:50.296111+0000","last_became_active":"2023-10-14T18:18:35.679636+0000","last_became_peered":"2023-10-14T18:18:35.679636+0000","last_unstale":"2023-10-14T18:18:50.296111+0000","last_undegraded":"2023-10-14T18:18:50.296111+0000","last_fullsized":"2023-10-14T18:18:50.296111+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:40.768705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.19","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297905+0000","last_change":"2023-10-14T18:18:50.314849+0000","last_active":"2023-10-14T18:18:51.297905+0000","last_peered":"2023-10-14T18:18:51.297905+0000","last_clean":"2023-10-14T18:18:51.297905+0000","last_became_active":"2023-10-14T18:18:50.313929+0000","last_became_peered":"2023-10-14T18:18:50.313929+0000","last_unstale":"2023-10-14T18:18:51.297905+0000","last_undegraded":"2023-10-14T18:18:51.297905+0000","last_fullsized":"2023-10-14T18:18:51.297905+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:46:29.147556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.18","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297471+0000","last_change":"2023-10-14T18:18:49.315528+0000","last_active":"2023-10-14T18:18:51.297471+0000","last_peered":"2023-10-14T18:18:51.297471+0000","last_clean":"2023-10-14T18:18:51.297471+0000","last_became_active":"2023-10-14T18:18:49.315247+0000","last_became_peered":"2023-10-14T18:18:49.315247+0000","last_unstale":"2023-10-14T18:18:51.297471+0000","last_undegraded":"2023-10-14T18:18:51.297471+0000","last_fullsized":"2023-10-14T18:18:51.297471+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:05:14.678752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.27","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306376+0000","last_change":"2023-10-14T18:18:35.663686+0000","last_active":"2023-10-14T18:18:50.306376+0000","last_peered":"2023-10-14T18:18:50.306376+0000","last_clean":"2023-10-14T18:18:50.306376+0000","last_became_active":"2023-10-14T18:18:35.663364+0000","last_became_peered":"2023-10-14T18:18:35.663364+0000","last_unstale":"2023-10-14T18:18:50.306376+0000","last_undegraded":"2023-10-14T18:18:50.306376+0000","last_fullsized":"2023-10-14T18:18:50.306376+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:43.005233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.18","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377248+0000","last_change":"2023-10-14T18:18:50.325391+0000","last_active":"2023-10-14T18:18:51.377248+0000","last_peered":"2023-10-14T18:18:51.377248+0000","last_clean":"2023-10-14T18:18:51.377248+0000","last_became_active":"2023-10-14T18:18:50.325122+0000","last_became_peered":"2023-10-14T18:18:50.325122+0000","last_unstale":"2023-10-14T18:18:51.377248+0000","last_undegraded":"2023-10-14T18:18:51.377248+0000","last_fullsized":"2023-10-14T18:18:51.377248+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:04.610423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.19","version":"0'0","reported_seq":12,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294546+0000","last_change":"2023-10-14T18:18:49.300991+0000","last_active":"2023-10-14T18:18:51.294546+0000","last_peered":"2023-10-14T18:18:51.294546+0000","last_clean":"2023-10-14T18:18:51.294546+0000","last_became_active":"2023-10-14T18:18:49.300438+0000","last_became_peered":"2023-10-14T18:18:49.300438+0000","last_unstale":"2023-10-14T18:18:51.294546+0000","last_undegraded":"2023-10-14T18:18:51.294546+0000","last_fullsized":"2023-10-14T18:18:51.294546+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:13.095242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.26","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309419+0000","last_change":"2023-10-14T18:18:35.683088+0000","last_active":"2023-10-14T18:18:50.309419+0000","last_peered":"2023-10-14T18:18:50.309419+0000","last_clean":"2023-10-14T18:18:50.309419+0000","last_became_active":"2023-10-14T18:18:35.682753+0000","last_became_peered":"2023-10-14T18:18:35.682753+0000","last_unstale":"2023-10-14T18:18:50.309419+0000","last_undegraded":"2023-10-14T18:18:50.309419+0000","last_fullsized":"2023-10-14T18:18:50.309419+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.1b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377699+0000","last_change":"2023-10-14T18:18:50.307192+0000","last_active":"2023-10-14T18:18:51.377699+0000","last_peered":"2023-10-14T18:18:51.377699+0000","last_clean":"2023-10-14T18:18:51.377699+0000","last_became_active":"2023-10-14T18:18:50.306894+0000","last_became_peered":"2023-10-14T18:18:50.306894+0000","last_unstale":"2023-10-14T18:18:51.377699+0000","last_undegraded":"2023-10-14T18:18:51.377699+0000","last_fullsized":"2023-10-14T18:18:51.377699+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:05:58.616654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"32.1a","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379194+0000","last_change":"2023-10-14T18:18:49.307502+0000","last_active":"2023-10-14T18:18:51.379194+0000","last_peered":"2023-10-14T18:18:51.379194+0000","last_clean":"2023-10-14T18:18:51.379194+0000","last_became_active":"2023-10-14T18:18:49.307250+0000","last_became_peered":"2023-10-14T18:18:49.307250+0000","last_unstale":"2023-10-14T18:18:51.379194+0000","last_undegraded":"2023-10-14T18:18:51.379194+0000","last_fullsized":"2023-10-14T18:18:51.379194+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:40:40.785284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.25","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292297+0000","last_change":"2023-10-14T18:18:35.689155+0000","last_active":"2023-10-14T18:18:50.292297+0000","last_peered":"2023-10-14T18:18:50.292297+0000","last_clean":"2023-10-14T18:18:50.292297+0000","last_became_active":"2023-10-14T18:18:35.686147+0000","last_became_peered":"2023-10-14T18:18:35.686147+0000","last_unstale":"2023-10-14T18:18:50.292297+0000","last_undegraded":"2023-10-14T18:18:50.292297+0000","last_fullsized":"2023-10-14T18:18:50.292297+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:12.203818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379371+0000","last_change":"2023-10-14T18:18:50.324595+0000","last_active":"2023-10-14T18:18:51.379371+0000","last_peered":"2023-10-14T18:18:51.379371+0000","last_clean":"2023-10-14T18:18:51.379371+0000","last_became_active":"2023-10-14T18:18:50.324293+0000","last_became_peered":"2023-10-14T18:18:50.324293+0000","last_unstale":"2023-10-14T18:18:51.379371+0000","last_undegraded":"2023-10-14T18:18:51.379371+0000","last_fullsized":"2023-10-14T18:18:51.379371+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:36:56.093870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"32.1b","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.300008+0000","last_change":"2023-10-14T18:18:49.312699+0000","last_active":"2023-10-14T18:18:51.300008+0000","last_peered":"2023-10-14T18:18:51.300008+0000","last_clean":"2023-10-14T18:18:51.300008+0000","last_became_active":"2023-10-14T18:18:49.311196+0000","last_became_peered":"2023-10-14T18:18:49.311196+0000","last_unstale":"2023-10-14T18:18:51.300008+0000","last_undegraded":"2023-10-14T18:18:51.300008+0000","last_fullsized":"2023-10-14T18:18:51.300008+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:03.483516+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.24","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297975+0000","last_change":"2023-10-14T18:18:35.673716+0000","last_active":"2023-10-14T18:18:50.297975+0000","last_peered":"2023-10-14T18:18:50.297975+0000","last_clean":"2023-10-14T18:18:50.297975+0000","last_became_active":"2023-10-14T18:18:35.672662+0000","last_became_peered":"2023-10-14T18:18:35.672662+0000","last_unstale":"2023-10-14T18:18:50.297975+0000","last_undegraded":"2023-10-14T18:18:50.297975+0000","last_fullsized":"2023-10-14T18:18:50.297975+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:06.125956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.1d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293622+0000","last_change":"2023-10-14T18:18:50.311394+0000","last_active":"2023-10-14T18:18:51.293622+0000","last_peered":"2023-10-14T18:18:51.293622+0000","last_clean":"2023-10-14T18:18:51.293622+0000","last_became_active":"2023-10-14T18:18:50.311090+0000","last_became_peered":"2023-10-14T18:18:50.311090+0000","last_unstale":"2023-10-14T18:18:51.293622+0000","last_undegraded":"2023-10-14T18:18:51.293622+0000","last_fullsized":"2023-10-14T18:18:51.293622+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:25.388110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"32.1c","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380528+0000","last_change":"2023-10-14T18:18:49.310283+0000","last_active":"2023-10-14T18:18:51.380528+0000","last_peered":"2023-10-14T18:18:51.380528+0000","last_clean":"2023-10-14T18:18:51.380528+0000","last_became_active":"2023-10-14T18:18:49.310002+0000","last_became_peered":"2023-10-14T18:18:49.310002+0000","last_unstale":"2023-10-14T18:18:51.380528+0000","last_undegraded":"2023-10-14T18:18:51.380528+0000","last_fullsized":"2023-10-14T18:18:51.380528+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:09:58.123098+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.23","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289984+0000","last_change":"2023-10-14T18:18:35.682800+0000","last_active":"2023-10-14T18:18:50.289984+0000","last_peered":"2023-10-14T18:18:50.289984+0000","last_clean":"2023-10-14T18:18:50.289984+0000","last_became_active":"2023-10-14T18:18:35.682478+0000","last_became_peered":"2023-10-14T18:18:35.682478+0000","last_unstale":"2023-10-14T18:18:50.289984+0000","last_undegraded":"2023-10-14T18:18:50.289984+0000","last_fullsized":"2023-10-14T18:18:50.289984+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:52:36.176215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.1c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.298778+0000","last_change":"2023-10-14T18:18:50.318794+0000","last_active":"2023-10-14T18:18:51.298778+0000","last_peered":"2023-10-14T18:18:51.298778+0000","last_clean":"2023-10-14T18:18:51.298778+0000","last_became_active":"2023-10-14T18:18:50.312936+0000","last_became_peered":"2023-10-14T18:18:50.312936+0000","last_unstale":"2023-10-14T18:18:51.298778+0000","last_undegraded":"2023-10-14T18:18:51.298778+0000","last_fullsized":"2023-10-14T18:18:51.298778+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:29.400393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1d","version":"253'2","reported_seq":14,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381700+0000","last_change":"2023-10-14T18:18:49.308149+0000","last_active":"2023-10-14T18:18:51.381700+0000","last_peered":"2023-10-14T18:18:51.381700+0000","last_clean":"2023-10-14T18:18:51.381700+0000","last_became_active":"2023-10-14T18:18:49.307852+0000","last_became_peered":"2023-10-14T18:18:49.307852+0000","last_unstale":"2023-10-14T18:18:51.381700+0000","last_undegraded":"2023-10-14T18:18:51.381700+0000","last_fullsized":"2023-10-14T18:18:51.381700+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:38.263857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.22","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.315418+0000","last_change":"2023-10-14T18:18:35.676776+0000","last_active":"2023-10-14T18:18:50.315418+0000","last_peered":"2023-10-14T18:18:50.315418+0000","last_clean":"2023-10-14T18:18:50.315418+0000","last_became_active":"2023-10-14T18:18:35.675446+0000","last_became_peered":"2023-10-14T18:18:35.675446+0000","last_unstale":"2023-10-14T18:18:50.315418+0000","last_undegraded":"2023-10-14T18:18:50.315418+0000","last_fullsized":"2023-10-14T18:18:50.315418+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:09.420076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.1f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379092+0000","last_change":"2023-10-14T18:18:50.316786+0000","last_active":"2023-10-14T18:18:51.379092+0000","last_peered":"2023-10-14T18:18:51.379092+0000","last_clean":"2023-10-14T18:18:51.379092+0000","last_became_active":"2023-10-14T18:18:50.316396+0000","last_became_peered":"2023-10-14T18:18:50.316396+0000","last_unstale":"2023-10-14T18:18:51.379092+0000","last_undegraded":"2023-10-14T18:18:51.379092+0000","last_fullsized":"2023-10-14T18:18:51.379092+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:36:56.093870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"32.1e","version":"253'1","reported_seq":13,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378765+0000","last_change":"2023-10-14T18:18:49.313758+0000","last_active":"2023-10-14T18:18:51.378765+0000","last_peered":"2023-10-14T18:18:51.378765+0000","last_clean":"2023-10-14T18:18:51.378765+0000","last_became_active":"2023-10-14T18:18:49.312912+0000","last_became_peered":"2023-10-14T18:18:49.312912+0000","last_unstale":"2023-10-14T18:18:51.378765+0000","last_undegraded":"2023-10-14T18:18:51.378765+0000","last_fullsized":"2023-10-14T18:18:51.378765+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:52.136902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.21","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.307636+0000","last_change":"2023-10-14T18:18:35.676664+0000","last_active":"2023-10-14T18:18:50.307636+0000","last_peered":"2023-10-14T18:18:50.307636+0000","last_clean":"2023-10-14T18:18:50.307636+0000","last_became_active":"2023-10-14T18:18:35.676381+0000","last_became_peered":"2023-10-14T18:18:35.676381+0000","last_unstale":"2023-10-14T18:18:50.307636+0000","last_undegraded":"2023-10-14T18:18:50.307636+0000","last_fullsized":"2023-10-14T18:18:50.307636+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:23.246857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.1e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299515+0000","last_change":"2023-10-14T18:18:50.320064+0000","last_active":"2023-10-14T18:18:51.299515+0000","last_peered":"2023-10-14T18:18:51.299515+0000","last_clean":"2023-10-14T18:18:51.299515+0000","last_became_active":"2023-10-14T18:18:50.319454+0000","last_became_peered":"2023-10-14T18:18:50.319454+0000","last_unstale":"2023-10-14T18:18:51.299515+0000","last_undegraded":"2023-10-14T18:18:51.299515+0000","last_fullsized":"2023-10-14T18:18:51.299515+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:47.019979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"32.1f","version":"253'4","reported_seq":16,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.383983+0000","last_change":"2023-10-14T18:18:49.307206+0000","last_active":"2023-10-14T18:18:51.383983+0000","last_peered":"2023-10-14T18:18:51.383983+0000","last_clean":"2023-10-14T18:18:51.383983+0000","last_became_active":"2023-10-14T18:18:49.306864+0000","last_became_peered":"2023-10-14T18:18:49.306864+0000","last_unstale":"2023-10-14T18:18:51.383983+0000","last_undegraded":"2023-10-14T18:18:51.383983+0000","last_fullsized":"2023-10-14T18:18:51.383983+0000","mapping_epoch":251,"log_start":"0'0","ondisk_log_start":"0'0","created":251,"last_epoch_clean":252,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:48.252988+0000","last_clean_scrub_stamp":"2023-10-14T18:18:48.252988+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:45:47.801865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.20","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296256+0000","last_change":"2023-10-14T18:18:35.670711+0000","last_active":"2023-10-14T18:18:50.296256+0000","last_peered":"2023-10-14T18:18:50.296256+0000","last_clean":"2023-10-14T18:18:50.296256+0000","last_became_active":"2023-10-14T18:18:35.670350+0000","last_became_peered":"2023-10-14T18:18:35.670350+0000","last_unstale":"2023-10-14T18:18:50.296256+0000","last_undegraded":"2023-10-14T18:18:50.296256+0000","last_fullsized":"2023-10-14T18:18:50.296256+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:33:32.369325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.21","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294673+0000","last_change":"2023-10-14T18:18:50.312932+0000","last_active":"2023-10-14T18:18:51.294673+0000","last_peered":"2023-10-14T18:18:51.294673+0000","last_clean":"2023-10-14T18:18:51.294673+0000","last_became_active":"2023-10-14T18:18:50.311269+0000","last_became_peered":"2023-10-14T18:18:50.311269+0000","last_unstale":"2023-10-14T18:18:51.294673+0000","last_undegraded":"2023-10-14T18:18:51.294673+0000","last_fullsized":"2023-10-14T18:18:51.294673+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:41:38.778707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309584+0000","last_change":"2023-10-14T18:18:35.677418+0000","last_active":"2023-10-14T18:18:50.309584+0000","last_peered":"2023-10-14T18:18:50.309584+0000","last_clean":"2023-10-14T18:18:50.309584+0000","last_became_active":"2023-10-14T18:18:35.677059+0000","last_became_peered":"2023-10-14T18:18:35.677059+0000","last_unstale":"2023-10-14T18:18:50.309584+0000","last_undegraded":"2023-10-14T18:18:50.309584+0000","last_fullsized":"2023-10-14T18:18:50.309584+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:07.342329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.1e","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309595+0000","last_change":"2023-10-14T18:18:34.650335+0000","last_active":"2023-10-14T18:18:50.309595+0000","last_peered":"2023-10-14T18:18:50.309595+0000","last_clean":"2023-10-14T18:18:50.309595+0000","last_became_active":"2023-10-14T18:18:34.648549+0000","last_became_peered":"2023-10-14T18:18:34.648549+0000","last_unstale":"2023-10-14T18:18:50.309595+0000","last_undegraded":"2023-10-14T18:18:50.309595+0000","last_fullsized":"2023-10-14T18:18:50.309595+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.22","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380246+0000","last_change":"2023-10-14T18:18:50.311514+0000","last_active":"2023-10-14T18:18:51.380246+0000","last_peered":"2023-10-14T18:18:51.380246+0000","last_clean":"2023-10-14T18:18:51.380246+0000","last_became_active":"2023-10-14T18:18:50.310994+0000","last_became_peered":"2023-10-14T18:18:50.310994+0000","last_unstale":"2023-10-14T18:18:51.380246+0000","last_undegraded":"2023-10-14T18:18:51.380246+0000","last_fullsized":"2023-10-14T18:18:51.380246+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:21:10.992585+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291062+0000","last_change":"2023-10-14T18:18:35.684529+0000","last_active":"2023-10-14T18:18:50.291062+0000","last_peered":"2023-10-14T18:18:50.291062+0000","last_clean":"2023-10-14T18:18:50.291062+0000","last_became_active":"2023-10-14T18:18:35.684263+0000","last_became_peered":"2023-10-14T18:18:35.684263+0000","last_unstale":"2023-10-14T18:18:50.291062+0000","last_undegraded":"2023-10-14T18:18:50.291062+0000","last_fullsized":"2023-10-14T18:18:50.291062+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:06.771665+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1d","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.324626+0000","last_change":"2023-10-14T18:18:34.653624+0000","last_active":"2023-10-14T18:18:50.324626+0000","last_peered":"2023-10-14T18:18:50.324626+0000","last_clean":"2023-10-14T18:18:50.324626+0000","last_became_active":"2023-10-14T18:18:34.652877+0000","last_became_peered":"2023-10-14T18:18:34.652877+0000","last_unstale":"2023-10-14T18:18:50.324626+0000","last_undegraded":"2023-10-14T18:18:50.324626+0000","last_fullsized":"2023-10-14T18:18:50.324626+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:13.266689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.23","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382719+0000","last_change":"2023-10-14T18:18:50.321052+0000","last_active":"2023-10-14T18:18:51.382719+0000","last_peered":"2023-10-14T18:18:51.382719+0000","last_clean":"2023-10-14T18:18:51.382719+0000","last_became_active":"2023-10-14T18:18:50.320652+0000","last_became_peered":"2023-10-14T18:18:50.320652+0000","last_unstale":"2023-10-14T18:18:51.382719+0000","last_undegraded":"2023-10-14T18:18:51.382719+0000","last_fullsized":"2023-10-14T18:18:51.382719+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T06:01:25.633128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.1d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296610+0000","last_change":"2023-10-14T18:18:35.671984+0000","last_active":"2023-10-14T18:18:50.296610+0000","last_peered":"2023-10-14T18:18:50.296610+0000","last_clean":"2023-10-14T18:18:50.296610+0000","last_became_active":"2023-10-14T18:18:35.668766+0000","last_became_peered":"2023-10-14T18:18:35.668766+0000","last_unstale":"2023-10-14T18:18:50.296610+0000","last_undegraded":"2023-10-14T18:18:50.296610+0000","last_fullsized":"2023-10-14T18:18:50.296610+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1c","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310014+0000","last_change":"2023-10-14T18:18:34.649372+0000","last_active":"2023-10-14T18:18:50.310014+0000","last_peered":"2023-10-14T18:18:50.310014+0000","last_clean":"2023-10-14T18:18:50.310014+0000","last_became_active":"2023-10-14T18:18:34.648998+0000","last_became_peered":"2023-10-14T18:18:34.648998+0000","last_unstale":"2023-10-14T18:18:50.310014+0000","last_undegraded":"2023-10-14T18:18:50.310014+0000","last_fullsized":"2023-10-14T18:18:50.310014+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:01:24.567915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.24","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.385161+0000","last_change":"2023-10-14T18:18:50.319043+0000","last_active":"2023-10-14T18:18:51.385161+0000","last_peered":"2023-10-14T18:18:51.385161+0000","last_clean":"2023-10-14T18:18:51.385161+0000","last_became_active":"2023-10-14T18:18:50.316998+0000","last_became_peered":"2023-10-14T18:18:50.316998+0000","last_unstale":"2023-10-14T18:18:51.385161+0000","last_undegraded":"2023-10-14T18:18:51.385161+0000","last_fullsized":"2023-10-14T18:18:51.385161+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:29.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.1a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289794+0000","last_change":"2023-10-14T18:18:35.672986+0000","last_active":"2023-10-14T18:18:50.289794+0000","last_peered":"2023-10-14T18:18:50.289794+0000","last_clean":"2023-10-14T18:18:50.289794+0000","last_became_active":"2023-10-14T18:18:35.672683+0000","last_became_peered":"2023-10-14T18:18:35.672683+0000","last_unstale":"2023-10-14T18:18:50.289794+0000","last_undegraded":"2023-10-14T18:18:50.289794+0000","last_fullsized":"2023-10-14T18:18:50.289794+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:59.451214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.1b","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.302019+0000","last_change":"2023-10-14T18:18:34.651378+0000","last_active":"2023-10-14T18:18:50.302019+0000","last_peered":"2023-10-14T18:18:50.302019+0000","last_clean":"2023-10-14T18:18:50.302019+0000","last_became_active":"2023-10-14T18:18:34.651072+0000","last_became_peered":"2023-10-14T18:18:34.651072+0000","last_unstale":"2023-10-14T18:18:50.302019+0000","last_undegraded":"2023-10-14T18:18:50.302019+0000","last_fullsized":"2023-10-14T18:18:50.302019+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:07:46.772700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.25","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299121+0000","last_change":"2023-10-14T18:18:50.319501+0000","last_active":"2023-10-14T18:18:51.299121+0000","last_peered":"2023-10-14T18:18:51.299121+0000","last_clean":"2023-10-14T18:18:51.299121+0000","last_became_active":"2023-10-14T18:18:50.319215+0000","last_became_peered":"2023-10-14T18:18:50.319215+0000","last_unstale":"2023-10-14T18:18:51.299121+0000","last_undegraded":"2023-10-14T18:18:51.299121+0000","last_fullsized":"2023-10-14T18:18:51.299121+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:03:14.599554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299541+0000","last_change":"2023-10-14T18:18:35.691610+0000","last_active":"2023-10-14T18:18:50.299541+0000","last_peered":"2023-10-14T18:18:50.299541+0000","last_clean":"2023-10-14T18:18:50.299541+0000","last_became_active":"2023-10-14T18:18:35.691317+0000","last_became_peered":"2023-10-14T18:18:35.691317+0000","last_unstale":"2023-10-14T18:18:50.299541+0000","last_undegraded":"2023-10-14T18:18:50.299541+0000","last_fullsized":"2023-10-14T18:18:50.299541+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:17:24.619829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.1a","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299275+0000","last_change":"2023-10-14T18:18:34.652803+0000","last_active":"2023-10-14T18:18:50.299275+0000","last_peered":"2023-10-14T18:18:50.299275+0000","last_clean":"2023-10-14T18:18:50.299275+0000","last_became_active":"2023-10-14T18:18:34.647316+0000","last_became_peered":"2023-10-14T18:18:34.647316+0000","last_unstale":"2023-10-14T18:18:50.299275+0000","last_undegraded":"2023-10-14T18:18:50.299275+0000","last_fullsized":"2023-10-14T18:18:50.299275+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:00.238815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.26","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297053+0000","last_change":"2023-10-14T18:18:50.325101+0000","last_active":"2023-10-14T18:18:51.297053+0000","last_peered":"2023-10-14T18:18:51.297053+0000","last_clean":"2023-10-14T18:18:51.297053+0000","last_became_active":"2023-10-14T18:18:50.324711+0000","last_became_peered":"2023-10-14T18:18:50.324711+0000","last_unstale":"2023-10-14T18:18:51.297053+0000","last_undegraded":"2023-10-14T18:18:51.297053+0000","last_fullsized":"2023-10-14T18:18:51.297053+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:02.650232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.18","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310047+0000","last_change":"2023-10-14T18:18:35.678351+0000","last_active":"2023-10-14T18:18:50.310047+0000","last_peered":"2023-10-14T18:18:50.310047+0000","last_clean":"2023-10-14T18:18:50.310047+0000","last_became_active":"2023-10-14T18:18:35.678058+0000","last_became_peered":"2023-10-14T18:18:35.678058+0000","last_unstale":"2023-10-14T18:18:50.310047+0000","last_undegraded":"2023-10-14T18:18:50.310047+0000","last_fullsized":"2023-10-14T18:18:50.310047+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:50.521263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.19","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310329+0000","last_change":"2023-10-14T18:18:34.651512+0000","last_active":"2023-10-14T18:18:50.310329+0000","last_peered":"2023-10-14T18:18:50.310329+0000","last_clean":"2023-10-14T18:18:50.310329+0000","last_became_active":"2023-10-14T18:18:34.651205+0000","last_became_peered":"2023-10-14T18:18:34.651205+0000","last_unstale":"2023-10-14T18:18:50.310329+0000","last_undegraded":"2023-10-14T18:18:50.310329+0000","last_fullsized":"2023-10-14T18:18:50.310329+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:11.191400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.27","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379266+0000","last_change":"2023-10-14T18:18:50.311991+0000","last_active":"2023-10-14T18:18:51.379266+0000","last_peered":"2023-10-14T18:18:51.379266+0000","last_clean":"2023-10-14T18:18:51.379266+0000","last_became_active":"2023-10-14T18:18:50.311717+0000","last_became_peered":"2023-10-14T18:18:50.311717+0000","last_unstale":"2023-10-14T18:18:51.379266+0000","last_undegraded":"2023-10-14T18:18:51.379266+0000","last_fullsized":"2023-10-14T18:18:51.379266+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:07.405144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.19","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295249+0000","last_change":"2023-10-14T18:18:35.677057+0000","last_active":"2023-10-14T18:18:50.295249+0000","last_peered":"2023-10-14T18:18:50.295249+0000","last_clean":"2023-10-14T18:18:50.295249+0000","last_became_active":"2023-10-14T18:18:35.676751+0000","last_became_peered":"2023-10-14T18:18:35.676751+0000","last_unstale":"2023-10-14T18:18:50.295249+0000","last_undegraded":"2023-10-14T18:18:50.295249+0000","last_fullsized":"2023-10-14T18:18:50.295249+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:26:14.749739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.18","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310267+0000","last_change":"2023-10-14T18:18:34.645198+0000","last_active":"2023-10-14T18:18:50.310267+0000","last_peered":"2023-10-14T18:18:50.310267+0000","last_clean":"2023-10-14T18:18:50.310267+0000","last_became_active":"2023-10-14T18:18:34.644896+0000","last_became_peered":"2023-10-14T18:18:34.644896+0000","last_unstale":"2023-10-14T18:18:50.310267+0000","last_undegraded":"2023-10-14T18:18:50.310267+0000","last_fullsized":"2023-10-14T18:18:50.310267+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:17.263536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.28","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379446+0000","last_change":"2023-10-14T18:18:50.322885+0000","last_active":"2023-10-14T18:18:51.379446+0000","last_peered":"2023-10-14T18:18:51.379446+0000","last_clean":"2023-10-14T18:18:51.379446+0000","last_became_active":"2023-10-14T18:18:50.320103+0000","last_became_peered":"2023-10-14T18:18:50.320103+0000","last_unstale":"2023-10-14T18:18:51.379446+0000","last_undegraded":"2023-10-14T18:18:51.379446+0000","last_fullsized":"2023-10-14T18:18:51.379446+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:30:48.473004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.16","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299115+0000","last_change":"2023-10-14T18:18:35.675636+0000","last_active":"2023-10-14T18:18:50.299115+0000","last_peered":"2023-10-14T18:18:50.299115+0000","last_clean":"2023-10-14T18:18:50.299115+0000","last_became_active":"2023-10-14T18:18:35.674527+0000","last_became_peered":"2023-10-14T18:18:35.674527+0000","last_unstale":"2023-10-14T18:18:50.299115+0000","last_undegraded":"2023-10-14T18:18:50.299115+0000","last_fullsized":"2023-10-14T18:18:50.299115+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:19.471165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.17","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297054+0000","last_change":"2023-10-14T18:18:34.640892+0000","last_active":"2023-10-14T18:18:50.297054+0000","last_peered":"2023-10-14T18:18:50.297054+0000","last_clean":"2023-10-14T18:18:50.297054+0000","last_became_active":"2023-10-14T18:18:34.639779+0000","last_became_peered":"2023-10-14T18:18:34.639779+0000","last_unstale":"2023-10-14T18:18:50.297054+0000","last_undegraded":"2023-10-14T18:18:50.297054+0000","last_fullsized":"2023-10-14T18:18:50.297054+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:53.119824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"33.29","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297776+0000","last_change":"2023-10-14T18:18:50.318782+0000","last_active":"2023-10-14T18:18:51.297776+0000","last_peered":"2023-10-14T18:18:51.297776+0000","last_clean":"2023-10-14T18:18:51.297776+0000","last_became_active":"2023-10-14T18:18:50.316788+0000","last_became_peered":"2023-10-14T18:18:50.316788+0000","last_unstale":"2023-10-14T18:18:51.297776+0000","last_undegraded":"2023-10-14T18:18:51.297776+0000","last_fullsized":"2023-10-14T18:18:51.297776+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:56:55.419359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.17","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292510+0000","last_change":"2023-10-14T18:18:35.685195+0000","last_active":"2023-10-14T18:18:50.292510+0000","last_peered":"2023-10-14T18:18:50.292510+0000","last_clean":"2023-10-14T18:18:50.292510+0000","last_became_active":"2023-10-14T18:18:35.682690+0000","last_became_peered":"2023-10-14T18:18:35.682690+0000","last_unstale":"2023-10-14T18:18:50.292510+0000","last_undegraded":"2023-10-14T18:18:50.292510+0000","last_fullsized":"2023-10-14T18:18:50.292510+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:52:54.132242+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.16","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.315405+0000","last_change":"2023-10-14T18:18:34.651854+0000","last_active":"2023-10-14T18:18:50.315405+0000","last_peered":"2023-10-14T18:18:50.315405+0000","last_clean":"2023-10-14T18:18:50.315405+0000","last_became_active":"2023-10-14T18:18:34.651540+0000","last_became_peered":"2023-10-14T18:18:34.651540+0000","last_unstale":"2023-10-14T18:18:50.315405+0000","last_undegraded":"2023-10-14T18:18:50.315405+0000","last_fullsized":"2023-10-14T18:18:50.315405+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:11:05.752351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378455+0000","last_change":"2023-10-14T18:18:50.304986+0000","last_active":"2023-10-14T18:18:51.378455+0000","last_peered":"2023-10-14T18:18:51.378455+0000","last_clean":"2023-10-14T18:18:51.378455+0000","last_became_active":"2023-10-14T18:18:50.304686+0000","last_became_peered":"2023-10-14T18:18:50.304686+0000","last_unstale":"2023-10-14T18:18:51.378455+0000","last_undegraded":"2023-10-14T18:18:51.378455+0000","last_fullsized":"2023-10-14T18:18:51.378455+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:46:14.708984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.14","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309921+0000","last_change":"2023-10-14T18:18:35.679788+0000","last_active":"2023-10-14T18:18:50.309921+0000","last_peered":"2023-10-14T18:18:50.309921+0000","last_clean":"2023-10-14T18:18:50.309921+0000","last_became_active":"2023-10-14T18:18:35.679488+0000","last_became_peered":"2023-10-14T18:18:35.679488+0000","last_unstale":"2023-10-14T18:18:50.309921+0000","last_undegraded":"2023-10-14T18:18:50.309921+0000","last_fullsized":"2023-10-14T18:18:50.309921+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:11:39.475602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.15","version":"249'3","reported_seq":19,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299129+0000","last_change":"2023-10-14T18:18:34.654465+0000","last_active":"2023-10-14T18:18:50.299129+0000","last_peered":"2023-10-14T18:18:50.299129+0000","last_clean":"2023-10-14T18:18:50.299129+0000","last_became_active":"2023-10-14T18:18:34.654111+0000","last_became_peered":"2023-10-14T18:18:34.654111+0000","last_unstale":"2023-10-14T18:18:50.299129+0000","last_undegraded":"2023-10-14T18:18:50.299129+0000","last_fullsized":"2023-10-14T18:18:50.299129+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:39:01.589582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379642+0000","last_change":"2023-10-14T18:18:50.325691+0000","last_active":"2023-10-14T18:18:51.379642+0000","last_peered":"2023-10-14T18:18:51.379642+0000","last_clean":"2023-10-14T18:18:51.379642+0000","last_became_active":"2023-10-14T18:18:50.325428+0000","last_became_peered":"2023-10-14T18:18:50.325428+0000","last_unstale":"2023-10-14T18:18:51.379642+0000","last_undegraded":"2023-10-14T18:18:51.379642+0000","last_fullsized":"2023-10-14T18:18:51.379642+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:52:07.405144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.15","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305345+0000","last_change":"2023-10-14T18:18:35.683637+0000","last_active":"2023-10-14T18:18:50.305345+0000","last_peered":"2023-10-14T18:18:50.305345+0000","last_clean":"2023-10-14T18:18:50.305345+0000","last_became_active":"2023-10-14T18:18:35.683235+0000","last_became_peered":"2023-10-14T18:18:35.683235+0000","last_unstale":"2023-10-14T18:18:50.305345+0000","last_undegraded":"2023-10-14T18:18:50.305345+0000","last_fullsized":"2023-10-14T18:18:50.305345+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.14","version":"250'8","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298106+0000","last_change":"2023-10-14T18:18:34.652609+0000","last_active":"2023-10-14T18:18:50.298106+0000","last_peered":"2023-10-14T18:18:50.298106+0000","last_clean":"2023-10-14T18:18:50.298106+0000","last_became_active":"2023-10-14T18:18:34.646287+0000","last_became_peered":"2023-10-14T18:18:34.646287+0000","last_unstale":"2023-10-14T18:18:50.298106+0000","last_undegraded":"2023-10-14T18:18:50.298106+0000","last_fullsized":"2023-10-14T18:18:50.298106+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:38:09.725037+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2857,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.380378+0000","last_change":"2023-10-14T18:18:50.301171+0000","last_active":"2023-10-14T18:18:51.380378+0000","last_peered":"2023-10-14T18:18:51.380378+0000","last_clean":"2023-10-14T18:18:51.380378+0000","last_became_active":"2023-10-14T18:18:50.300424+0000","last_became_peered":"2023-10-14T18:18:50.300424+0000","last_unstale":"2023-10-14T18:18:51.380378+0000","last_undegraded":"2023-10-14T18:18:51.380378+0000","last_fullsized":"2023-10-14T18:18:51.380378+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:15:21.189453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.12","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295150+0000","last_change":"2023-10-14T18:18:35.672404+0000","last_active":"2023-10-14T18:18:50.295150+0000","last_peered":"2023-10-14T18:18:50.295150+0000","last_clean":"2023-10-14T18:18:50.295150+0000","last_became_active":"2023-10-14T18:18:35.672109+0000","last_became_peered":"2023-10-14T18:18:35.672109+0000","last_unstale":"2023-10-14T18:18:50.295150+0000","last_undegraded":"2023-10-14T18:18:50.295150+0000","last_fullsized":"2023-10-14T18:18:50.295150+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.13","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298850+0000","last_change":"2023-10-14T18:18:34.650921+0000","last_active":"2023-10-14T18:18:50.298850+0000","last_peered":"2023-10-14T18:18:50.298850+0000","last_clean":"2023-10-14T18:18:50.298850+0000","last_became_active":"2023-10-14T18:18:34.650170+0000","last_became_peered":"2023-10-14T18:18:34.650170+0000","last_unstale":"2023-10-14T18:18:50.298850+0000","last_undegraded":"2023-10-14T18:18:50.298850+0000","last_fullsized":"2023-10-14T18:18:50.298850+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:43:24.834507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"33.2d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299078+0000","last_change":"2023-10-14T18:18:50.332895+0000","last_active":"2023-10-14T18:18:51.299078+0000","last_peered":"2023-10-14T18:18:51.299078+0000","last_clean":"2023-10-14T18:18:51.299078+0000","last_became_active":"2023-10-14T18:18:50.332553+0000","last_became_peered":"2023-10-14T18:18:50.332553+0000","last_unstale":"2023-10-14T18:18:51.299078+0000","last_undegraded":"2023-10-14T18:18:51.299078+0000","last_fullsized":"2023-10-14T18:18:51.299078+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:09:20.776610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.13","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299636+0000","last_change":"2023-10-14T18:18:35.685387+0000","last_active":"2023-10-14T18:18:50.299636+0000","last_peered":"2023-10-14T18:18:50.299636+0000","last_clean":"2023-10-14T18:18:50.299636+0000","last_became_active":"2023-10-14T18:18:35.684624+0000","last_became_peered":"2023-10-14T18:18:35.684624+0000","last_unstale":"2023-10-14T18:18:50.299636+0000","last_undegraded":"2023-10-14T18:18:50.299636+0000","last_fullsized":"2023-10-14T18:18:50.299636+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:01:49.959769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.12","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.304536+0000","last_change":"2023-10-14T18:18:34.644335+0000","last_active":"2023-10-14T18:18:50.304536+0000","last_peered":"2023-10-14T18:18:50.304536+0000","last_clean":"2023-10-14T18:18:50.304536+0000","last_became_active":"2023-10-14T18:18:34.644052+0000","last_became_peered":"2023-10-14T18:18:34.644052+0000","last_unstale":"2023-10-14T18:18:50.304536+0000","last_undegraded":"2023-10-14T18:18:50.304536+0000","last_fullsized":"2023-10-14T18:18:50.304536+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:48:00.968233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.2e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299739+0000","last_change":"2023-10-14T18:18:50.320011+0000","last_active":"2023-10-14T18:18:51.299739+0000","last_peered":"2023-10-14T18:18:51.299739+0000","last_clean":"2023-10-14T18:18:51.299739+0000","last_became_active":"2023-10-14T18:18:50.319408+0000","last_became_peered":"2023-10-14T18:18:50.319408+0000","last_unstale":"2023-10-14T18:18:51.299739+0000","last_undegraded":"2023-10-14T18:18:51.299739+0000","last_fullsized":"2023-10-14T18:18:51.299739+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:13:34.121820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.10","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297072+0000","last_change":"2023-10-14T18:18:35.675619+0000","last_active":"2023-10-14T18:18:50.297072+0000","last_peered":"2023-10-14T18:18:50.297072+0000","last_clean":"2023-10-14T18:18:50.297072+0000","last_became_active":"2023-10-14T18:18:35.673589+0000","last_became_peered":"2023-10-14T18:18:35.673589+0000","last_unstale":"2023-10-14T18:18:50.297072+0000","last_undegraded":"2023-10-14T18:18:50.297072+0000","last_fullsized":"2023-10-14T18:18:50.297072+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:01:37.559321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.11","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306235+0000","last_change":"2023-10-14T18:18:34.646943+0000","last_active":"2023-10-14T18:18:50.306235+0000","last_peered":"2023-10-14T18:18:50.306235+0000","last_clean":"2023-10-14T18:18:50.306235+0000","last_became_active":"2023-10-14T18:18:34.646630+0000","last_became_peered":"2023-10-14T18:18:34.646630+0000","last_unstale":"2023-10-14T18:18:50.306235+0000","last_undegraded":"2023-10-14T18:18:50.306235+0000","last_fullsized":"2023-10-14T18:18:50.306235+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:57.489531+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.2f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.384519+0000","last_change":"2023-10-14T18:18:50.334156+0000","last_active":"2023-10-14T18:18:51.384519+0000","last_peered":"2023-10-14T18:18:51.384519+0000","last_clean":"2023-10-14T18:18:51.384519+0000","last_became_active":"2023-10-14T18:18:50.331713+0000","last_became_peered":"2023-10-14T18:18:50.331713+0000","last_unstale":"2023-10-14T18:18:51.384519+0000","last_undegraded":"2023-10-14T18:18:51.384519+0000","last_fullsized":"2023-10-14T18:18:51.384519+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:44:20.400011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"31.11","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.305822+0000","last_change":"2023-10-14T18:18:35.678311+0000","last_active":"2023-10-14T18:18:50.305822+0000","last_peered":"2023-10-14T18:18:50.305822+0000","last_clean":"2023-10-14T18:18:50.305822+0000","last_became_active":"2023-10-14T18:18:35.678016+0000","last_became_peered":"2023-10-14T18:18:35.678016+0000","last_unstale":"2023-10-14T18:18:50.305822+0000","last_undegraded":"2023-10-14T18:18:50.305822+0000","last_fullsized":"2023-10-14T18:18:50.305822+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.10","version":"249'8","reported_seq":24,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.287997+0000","last_change":"2023-10-14T18:18:34.648173+0000","last_active":"2023-10-14T18:18:50.287997+0000","last_peered":"2023-10-14T18:18:50.287997+0000","last_clean":"2023-10-14T18:18:50.287997+0000","last_became_active":"2023-10-14T18:18:34.647904+0000","last_became_peered":"2023-10-14T18:18:34.647904+0000","last_unstale":"2023-10-14T18:18:50.287997+0000","last_undegraded":"2023-10-14T18:18:50.287997+0000","last_fullsized":"2023-10-14T18:18:50.287997+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:25:49.659273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.30","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.376877+0000","last_change":"2023-10-14T18:18:50.333569+0000","last_active":"2023-10-14T18:18:51.376877+0000","last_peered":"2023-10-14T18:18:51.376877+0000","last_clean":"2023-10-14T18:18:51.376877+0000","last_became_active":"2023-10-14T18:18:50.332171+0000","last_became_peered":"2023-10-14T18:18:50.332171+0000","last_unstale":"2023-10-14T18:18:51.376877+0000","last_undegraded":"2023-10-14T18:18:51.376877+0000","last_fullsized":"2023-10-14T18:18:51.376877+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:11.965757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.303586+0000","last_change":"2023-10-14T18:18:35.672233+0000","last_active":"2023-10-14T18:18:50.303586+0000","last_peered":"2023-10-14T18:18:50.303586+0000","last_clean":"2023-10-14T18:18:50.303586+0000","last_became_active":"2023-10-14T18:18:35.671904+0000","last_became_peered":"2023-10-14T18:18:35.671904+0000","last_unstale":"2023-10-14T18:18:50.303586+0000","last_undegraded":"2023-10-14T18:18:50.303586+0000","last_fullsized":"2023-10-14T18:18:50.303586+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:09.351543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"30.f","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296071+0000","last_change":"2023-10-14T18:18:34.644806+0000","last_active":"2023-10-14T18:18:50.296071+0000","last_peered":"2023-10-14T18:18:50.296071+0000","last_clean":"2023-10-14T18:18:50.296071+0000","last_became_active":"2023-10-14T18:18:34.642485+0000","last_became_peered":"2023-10-14T18:18:34.642485+0000","last_unstale":"2023-10-14T18:18:50.296071+0000","last_undegraded":"2023-10-14T18:18:50.296071+0000","last_fullsized":"2023-10-14T18:18:50.296071+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:22.234439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.31","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379990+0000","last_change":"2023-10-14T18:18:50.320259+0000","last_active":"2023-10-14T18:18:51.379990+0000","last_peered":"2023-10-14T18:18:51.379990+0000","last_clean":"2023-10-14T18:18:51.379990+0000","last_became_active":"2023-10-14T18:18:50.317254+0000","last_became_peered":"2023-10-14T18:18:50.317254+0000","last_unstale":"2023-10-14T18:18:51.379990+0000","last_undegraded":"2023-10-14T18:18:51.379990+0000","last_fullsized":"2023-10-14T18:18:51.379990+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:04.490143+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.f","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309835+0000","last_change":"2023-10-14T18:18:35.673621+0000","last_active":"2023-10-14T18:18:50.309835+0000","last_peered":"2023-10-14T18:18:50.309835+0000","last_clean":"2023-10-14T18:18:50.309835+0000","last_became_active":"2023-10-14T18:18:35.673296+0000","last_became_peered":"2023-10-14T18:18:35.673296+0000","last_unstale":"2023-10-14T18:18:50.309835+0000","last_undegraded":"2023-10-14T18:18:50.309835+0000","last_fullsized":"2023-10-14T18:18:50.309835+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:58:27.889006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"30.e","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.314697+0000","last_change":"2023-10-14T18:18:34.655553+0000","last_active":"2023-10-14T18:18:50.314697+0000","last_peered":"2023-10-14T18:18:50.314697+0000","last_clean":"2023-10-14T18:18:50.314697+0000","last_became_active":"2023-10-14T18:18:34.655113+0000","last_became_peered":"2023-10-14T18:18:34.655113+0000","last_unstale":"2023-10-14T18:18:50.314697+0000","last_undegraded":"2023-10-14T18:18:50.314697+0000","last_fullsized":"2023-10-14T18:18:50.314697+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:04:45.880556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.32","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.379869+0000","last_change":"2023-10-14T18:18:50.314689+0000","last_active":"2023-10-14T18:18:51.379869+0000","last_peered":"2023-10-14T18:18:51.379869+0000","last_clean":"2023-10-14T18:18:51.379869+0000","last_became_active":"2023-10-14T18:18:50.311321+0000","last_became_peered":"2023-10-14T18:18:50.311321+0000","last_unstale":"2023-10-14T18:18:51.379869+0000","last_undegraded":"2023-10-14T18:18:51.379869+0000","last_fullsized":"2023-10-14T18:18:51.379869+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:49.451546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.c","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299966+0000","last_change":"2023-10-14T18:18:35.683896+0000","last_active":"2023-10-14T18:18:50.299966+0000","last_peered":"2023-10-14T18:18:50.299966+0000","last_clean":"2023-10-14T18:18:50.299966+0000","last_became_active":"2023-10-14T18:18:35.683606+0000","last_became_peered":"2023-10-14T18:18:35.683606+0000","last_unstale":"2023-10-14T18:18:50.299966+0000","last_undegraded":"2023-10-14T18:18:50.299966+0000","last_fullsized":"2023-10-14T18:18:50.299966+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:48:02.619684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"30.d","version":"249'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300401+0000","last_change":"2023-10-14T18:18:34.647730+0000","last_active":"2023-10-14T18:18:50.300401+0000","last_peered":"2023-10-14T18:18:50.300401+0000","last_clean":"2023-10-14T18:18:50.300401+0000","last_became_active":"2023-10-14T18:18:34.646698+0000","last_became_peered":"2023-10-14T18:18:34.646698+0000","last_unstale":"2023-10-14T18:18:50.300401+0000","last_undegraded":"2023-10-14T18:18:50.300401+0000","last_fullsized":"2023-10-14T18:18:50.300401+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:30:44.897768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.33","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.382227+0000","last_change":"2023-10-14T18:18:50.333362+0000","last_active":"2023-10-14T18:18:51.382227+0000","last_peered":"2023-10-14T18:18:51.382227+0000","last_clean":"2023-10-14T18:18:51.382227+0000","last_became_active":"2023-10-14T18:18:50.333070+0000","last_became_peered":"2023-10-14T18:18:50.333070+0000","last_unstale":"2023-10-14T18:18:51.382227+0000","last_undegraded":"2023-10-14T18:18:51.382227+0000","last_fullsized":"2023-10-14T18:18:51.382227+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:54:01.495121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"31.d","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.296780+0000","last_change":"2023-10-14T18:18:35.675449+0000","last_active":"2023-10-14T18:18:50.296780+0000","last_peered":"2023-10-14T18:18:50.296780+0000","last_clean":"2023-10-14T18:18:50.296780+0000","last_became_active":"2023-10-14T18:18:35.675129+0000","last_became_peered":"2023-10-14T18:18:35.675129+0000","last_unstale":"2023-10-14T18:18:50.296780+0000","last_undegraded":"2023-10-14T18:18:50.296780+0000","last_fullsized":"2023-10-14T18:18:50.296780+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:55:19.808142+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.c","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291324+0000","last_change":"2023-10-14T18:18:34.645342+0000","last_active":"2023-10-14T18:18:50.291324+0000","last_peered":"2023-10-14T18:18:50.291324+0000","last_clean":"2023-10-14T18:18:50.291324+0000","last_became_active":"2023-10-14T18:18:34.643272+0000","last_became_peered":"2023-10-14T18:18:34.643272+0000","last_unstale":"2023-10-14T18:18:50.291324+0000","last_undegraded":"2023-10-14T18:18:50.291324+0000","last_fullsized":"2023-10-14T18:18:50.291324+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:20:08.339125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.34","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294594+0000","last_change":"2023-10-14T18:18:50.315771+0000","last_active":"2023-10-14T18:18:51.294594+0000","last_peered":"2023-10-14T18:18:51.294594+0000","last_clean":"2023-10-14T18:18:51.294594+0000","last_became_active":"2023-10-14T18:18:50.313621+0000","last_became_peered":"2023-10-14T18:18:50.313621+0000","last_unstale":"2023-10-14T18:18:51.294594+0000","last_undegraded":"2023-10-14T18:18:51.294594+0000","last_fullsized":"2023-10-14T18:18:51.294594+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:00.020840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.a","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292883+0000","last_change":"2023-10-14T18:18:35.675593+0000","last_active":"2023-10-14T18:18:50.292883+0000","last_peered":"2023-10-14T18:18:50.292883+0000","last_clean":"2023-10-14T18:18:50.292883+0000","last_became_active":"2023-10-14T18:18:35.675294+0000","last_became_peered":"2023-10-14T18:18:35.675294+0000","last_unstale":"2023-10-14T18:18:50.292883+0000","last_undegraded":"2023-10-14T18:18:50.292883+0000","last_fullsized":"2023-10-14T18:18:50.292883+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:33:51.249632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.b","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306058+0000","last_change":"2023-10-14T18:18:34.640130+0000","last_active":"2023-10-14T18:18:50.306058+0000","last_peered":"2023-10-14T18:18:50.306058+0000","last_clean":"2023-10-14T18:18:50.306058+0000","last_became_active":"2023-10-14T18:18:34.639677+0000","last_became_peered":"2023-10-14T18:18:34.639677+0000","last_unstale":"2023-10-14T18:18:50.306058+0000","last_undegraded":"2023-10-14T18:18:50.306058+0000","last_fullsized":"2023-10-14T18:18:50.306058+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:22:09.948727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.35","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.298935+0000","last_change":"2023-10-14T18:18:50.312663+0000","last_active":"2023-10-14T18:18:51.298935+0000","last_peered":"2023-10-14T18:18:51.298935+0000","last_clean":"2023-10-14T18:18:51.298935+0000","last_became_active":"2023-10-14T18:18:50.312393+0000","last_became_peered":"2023-10-14T18:18:50.312393+0000","last_unstale":"2023-10-14T18:18:51.298935+0000","last_undegraded":"2023-10-14T18:18:51.298935+0000","last_fullsized":"2023-10-14T18:18:51.298935+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:04.981289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.b","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.298687+0000","last_change":"2023-10-14T18:18:35.682891+0000","last_active":"2023-10-14T18:18:50.298687+0000","last_peered":"2023-10-14T18:18:50.298687+0000","last_clean":"2023-10-14T18:18:50.298687+0000","last_became_active":"2023-10-14T18:18:35.681440+0000","last_became_peered":"2023-10-14T18:18:35.681440+0000","last_unstale":"2023-10-14T18:18:50.298687+0000","last_undegraded":"2023-10-14T18:18:50.298687+0000","last_fullsized":"2023-10-14T18:18:50.298687+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:50.330635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.a","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.289955+0000","last_change":"2023-10-14T18:18:34.647137+0000","last_active":"2023-10-14T18:18:50.289955+0000","last_peered":"2023-10-14T18:18:50.289955+0000","last_clean":"2023-10-14T18:18:50.289955+0000","last_became_active":"2023-10-14T18:18:34.646766+0000","last_became_peered":"2023-10-14T18:18:34.646766+0000","last_unstale":"2023-10-14T18:18:50.289955+0000","last_undegraded":"2023-10-14T18:18:50.289955+0000","last_fullsized":"2023-10-14T18:18:50.289955+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.36","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299393+0000","last_change":"2023-10-14T18:18:50.327650+0000","last_active":"2023-10-14T18:18:51.299393+0000","last_peered":"2023-10-14T18:18:51.299393+0000","last_clean":"2023-10-14T18:18:51.299393+0000","last_became_active":"2023-10-14T18:18:50.327350+0000","last_became_peered":"2023-10-14T18:18:50.327350+0000","last_unstale":"2023-10-14T18:18:51.299393+0000","last_undegraded":"2023-10-14T18:18:51.299393+0000","last_fullsized":"2023-10-14T18:18:51.299393+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:36:30.605930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.8","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.288082+0000","last_change":"2023-10-14T18:18:35.672939+0000","last_active":"2023-10-14T18:18:50.288082+0000","last_peered":"2023-10-14T18:18:50.288082+0000","last_clean":"2023-10-14T18:18:50.288082+0000","last_became_active":"2023-10-14T18:18:35.672630+0000","last_became_peered":"2023-10-14T18:18:35.672630+0000","last_unstale":"2023-10-14T18:18:50.288082+0000","last_undegraded":"2023-10-14T18:18:50.288082+0000","last_fullsized":"2023-10-14T18:18:50.288082+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:51:56.149979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.9","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299108+0000","last_change":"2023-10-14T18:18:34.653097+0000","last_active":"2023-10-14T18:18:50.299108+0000","last_peered":"2023-10-14T18:18:50.299108+0000","last_clean":"2023-10-14T18:18:50.299108+0000","last_became_active":"2023-10-14T18:18:34.652338+0000","last_became_peered":"2023-10-14T18:18:34.652338+0000","last_unstale":"2023-10-14T18:18:50.299108+0000","last_undegraded":"2023-10-14T18:18:50.299108+0000","last_fullsized":"2023-10-14T18:18:50.299108+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:39.725847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.37","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.381763+0000","last_change":"2023-10-14T18:18:50.305740+0000","last_active":"2023-10-14T18:18:51.381763+0000","last_peered":"2023-10-14T18:18:51.381763+0000","last_clean":"2023-10-14T18:18:51.381763+0000","last_became_active":"2023-10-14T18:18:50.305421+0000","last_became_peered":"2023-10-14T18:18:50.305421+0000","last_unstale":"2023-10-14T18:18:51.381763+0000","last_undegraded":"2023-10-14T18:18:51.381763+0000","last_fullsized":"2023-10-14T18:18:51.381763+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:49.451546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"31.9","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.297163+0000","last_change":"2023-10-14T18:18:35.668420+0000","last_active":"2023-10-14T18:18:50.297163+0000","last_peered":"2023-10-14T18:18:50.297163+0000","last_clean":"2023-10-14T18:18:50.297163+0000","last_became_active":"2023-10-14T18:18:35.667429+0000","last_became_peered":"2023-10-14T18:18:35.667429+0000","last_unstale":"2023-10-14T18:18:50.297163+0000","last_undegraded":"2023-10-14T18:18:50.297163+0000","last_fullsized":"2023-10-14T18:18:50.297163+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:55:36.687382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.8","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295597+0000","last_change":"2023-10-14T18:18:34.649612+0000","last_active":"2023-10-14T18:18:50.295597+0000","last_peered":"2023-10-14T18:18:50.295597+0000","last_clean":"2023-10-14T18:18:50.295597+0000","last_became_active":"2023-10-14T18:18:34.649307+0000","last_became_peered":"2023-10-14T18:18:34.649307+0000","last_unstale":"2023-10-14T18:18:50.295597+0000","last_undegraded":"2023-10-14T18:18:50.295597+0000","last_fullsized":"2023-10-14T18:18:50.295597+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:47:58.438796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.38","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.293382+0000","last_change":"2023-10-14T18:18:50.332902+0000","last_active":"2023-10-14T18:18:51.293382+0000","last_peered":"2023-10-14T18:18:51.293382+0000","last_clean":"2023-10-14T18:18:51.293382+0000","last_became_active":"2023-10-14T18:18:50.332581+0000","last_became_peered":"2023-10-14T18:18:50.332581+0000","last_unstale":"2023-10-14T18:18:51.293382+0000","last_undegraded":"2023-10-14T18:18:51.293382+0000","last_fullsized":"2023-10-14T18:18:51.293382+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:29.192398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.6","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309503+0000","last_change":"2023-10-14T18:18:35.666113+0000","last_active":"2023-10-14T18:18:50.309503+0000","last_peered":"2023-10-14T18:18:50.309503+0000","last_clean":"2023-10-14T18:18:50.309503+0000","last_became_active":"2023-10-14T18:18:35.665775+0000","last_became_peered":"2023-10-14T18:18:35.665775+0000","last_unstale":"2023-10-14T18:18:50.309503+0000","last_undegraded":"2023-10-14T18:18:50.309503+0000","last_fullsized":"2023-10-14T18:18:50.309503+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:12:00.233622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.7","version":"249'2","reported_seq":18,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306460+0000","last_change":"2023-10-14T18:18:34.657934+0000","last_active":"2023-10-14T18:18:50.306460+0000","last_peered":"2023-10-14T18:18:50.306460+0000","last_clean":"2023-10-14T18:18:50.306460+0000","last_became_active":"2023-10-14T18:18:34.657614+0000","last_became_peered":"2023-10-14T18:18:34.657614+0000","last_unstale":"2023-10-14T18:18:50.306460+0000","last_undegraded":"2023-10-14T18:18:50.306460+0000","last_fullsized":"2023-10-14T18:18:50.306460+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:02:11.487790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.39","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297645+0000","last_change":"2023-10-14T18:18:50.315046+0000","last_active":"2023-10-14T18:18:51.297645+0000","last_peered":"2023-10-14T18:18:51.297645+0000","last_clean":"2023-10-14T18:18:51.297645+0000","last_became_active":"2023-10-14T18:18:50.314529+0000","last_became_peered":"2023-10-14T18:18:50.314529+0000","last_unstale":"2023-10-14T18:18:51.297645+0000","last_undegraded":"2023-10-14T18:18:51.297645+0000","last_fullsized":"2023-10-14T18:18:51.297645+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:35:17.980328+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.7","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.299198+0000","last_change":"2023-10-14T18:18:35.680237+0000","last_active":"2023-10-14T18:18:50.299198+0000","last_peered":"2023-10-14T18:18:50.299198+0000","last_clean":"2023-10-14T18:18:50.299198+0000","last_became_active":"2023-10-14T18:18:35.679960+0000","last_became_peered":"2023-10-14T18:18:35.679960+0000","last_unstale":"2023-10-14T18:18:50.299198+0000","last_undegraded":"2023-10-14T18:18:50.299198+0000","last_fullsized":"2023-10-14T18:18:50.299198+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:16:53.427626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.6","version":"250'1","reported_seq":17,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310032+0000","last_change":"2023-10-14T18:18:34.651448+0000","last_active":"2023-10-14T18:18:50.310032+0000","last_peered":"2023-10-14T18:18:50.310032+0000","last_clean":"2023-10-14T18:18:50.310032+0000","last_became_active":"2023-10-14T18:18:34.651100+0000","last_became_peered":"2023-10-14T18:18:34.651100+0000","last_unstale":"2023-10-14T18:18:50.310032+0000","last_undegraded":"2023-10-14T18:18:50.310032+0000","last_fullsized":"2023-10-14T18:18:50.310032+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:54:41.231071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"33.3a","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.378244+0000","last_change":"2023-10-14T18:18:50.324334+0000","last_active":"2023-10-14T18:18:51.378244+0000","last_peered":"2023-10-14T18:18:51.378244+0000","last_clean":"2023-10-14T18:18:51.378244+0000","last_became_active":"2023-10-14T18:18:50.323321+0000","last_became_peered":"2023-10-14T18:18:50.323321+0000","last_unstale":"2023-10-14T18:18:51.378244+0000","last_undegraded":"2023-10-14T18:18:51.378244+0000","last_fullsized":"2023-10-14T18:18:51.378244+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:02.370808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"31.4","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291190+0000","last_change":"2023-10-14T18:18:35.687662+0000","last_active":"2023-10-14T18:18:50.291190+0000","last_peered":"2023-10-14T18:18:50.291190+0000","last_clean":"2023-10-14T18:18:50.291190+0000","last_became_active":"2023-10-14T18:18:35.687396+0000","last_became_peered":"2023-10-14T18:18:35.687396+0000","last_unstale":"2023-10-14T18:18:50.291190+0000","last_undegraded":"2023-10-14T18:18:50.291190+0000","last_fullsized":"2023-10-14T18:18:50.291190+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:42:15.213936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.5","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300488+0000","last_change":"2023-10-14T18:18:34.649374+0000","last_active":"2023-10-14T18:18:50.300488+0000","last_peered":"2023-10-14T18:18:50.300488+0000","last_clean":"2023-10-14T18:18:50.300488+0000","last_became_active":"2023-10-14T18:18:34.649074+0000","last_became_peered":"2023-10-14T18:18:34.649074+0000","last_unstale":"2023-10-14T18:18:50.300488+0000","last_undegraded":"2023-10-14T18:18:50.300488+0000","last_fullsized":"2023-10-14T18:18:50.300488+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:45.885592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"33.3b","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.295315+0000","last_change":"2023-10-14T18:18:50.325218+0000","last_active":"2023-10-14T18:18:51.295315+0000","last_peered":"2023-10-14T18:18:51.295315+0000","last_clean":"2023-10-14T18:18:51.295315+0000","last_became_active":"2023-10-14T18:18:50.324903+0000","last_became_peered":"2023-10-14T18:18:50.324903+0000","last_unstale":"2023-10-14T18:18:51.295315+0000","last_undegraded":"2023-10-14T18:18:51.295315+0000","last_fullsized":"2023-10-14T18:18:51.295315+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:03:13.152603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.5","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.292434+0000","last_change":"2023-10-14T18:18:35.682974+0000","last_active":"2023-10-14T18:18:50.292434+0000","last_peered":"2023-10-14T18:18:50.292434+0000","last_clean":"2023-10-14T18:18:50.292434+0000","last_became_active":"2023-10-14T18:18:35.682647+0000","last_became_peered":"2023-10-14T18:18:35.682647+0000","last_unstale":"2023-10-14T18:18:50.292434+0000","last_undegraded":"2023-10-14T18:18:50.292434+0000","last_fullsized":"2023-10-14T18:18:50.292434+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:02:58.501949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"30.4","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.291396+0000","last_change":"2023-10-14T18:18:34.656380+0000","last_active":"2023-10-14T18:18:50.291396+0000","last_peered":"2023-10-14T18:18:50.291396+0000","last_clean":"2023-10-14T18:18:50.291396+0000","last_became_active":"2023-10-14T18:18:34.656073+0000","last_became_peered":"2023-10-14T18:18:34.656073+0000","last_unstale":"2023-10-14T18:18:50.291396+0000","last_undegraded":"2023-10-14T18:18:50.291396+0000","last_fullsized":"2023-10-14T18:18:50.291396+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:35:41.992815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"33.3c","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.296408+0000","last_change":"2023-10-14T18:18:50.322493+0000","last_active":"2023-10-14T18:18:51.296408+0000","last_peered":"2023-10-14T18:18:51.296408+0000","last_clean":"2023-10-14T18:18:51.296408+0000","last_became_active":"2023-10-14T18:18:50.320931+0000","last_became_peered":"2023-10-14T18:18:50.320931+0000","last_unstale":"2023-10-14T18:18:51.296408+0000","last_undegraded":"2023-10-14T18:18:51.296408+0000","last_fullsized":"2023-10-14T18:18:51.296408+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:53:08.782896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.2","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295295+0000","last_change":"2023-10-14T18:18:35.666278+0000","last_active":"2023-10-14T18:18:50.295295+0000","last_peered":"2023-10-14T18:18:50.295295+0000","last_clean":"2023-10-14T18:18:50.295295+0000","last_became_active":"2023-10-14T18:18:35.665963+0000","last_became_peered":"2023-10-14T18:18:35.665963+0000","last_unstale":"2023-10-14T18:18:50.295295+0000","last_undegraded":"2023-10-14T18:18:50.295295+0000","last_fullsized":"2023-10-14T18:18:50.295295+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:36:36.577866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"30.3","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295713+0000","last_change":"2023-10-14T18:18:34.649641+0000","last_active":"2023-10-14T18:18:50.295713+0000","last_peered":"2023-10-14T18:18:50.295713+0000","last_clean":"2023-10-14T18:18:50.295713+0000","last_became_active":"2023-10-14T18:18:34.649340+0000","last_became_peered":"2023-10-14T18:18:34.649340+0000","last_unstale":"2023-10-14T18:18:50.295713+0000","last_undegraded":"2023-10-14T18:18:50.295713+0000","last_fullsized":"2023-10-14T18:18:50.295713+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:49:05.759247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"33.3d","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.299308+0000","last_change":"2023-10-14T18:18:50.323121+0000","last_active":"2023-10-14T18:18:51.299308+0000","last_peered":"2023-10-14T18:18:51.299308+0000","last_clean":"2023-10-14T18:18:51.299308+0000","last_became_active":"2023-10-14T18:18:50.321364+0000","last_became_peered":"2023-10-14T18:18:50.321364+0000","last_unstale":"2023-10-14T18:18:51.299308+0000","last_undegraded":"2023-10-14T18:18:51.299308+0000","last_fullsized":"2023-10-14T18:18:51.299308+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:16.417405+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"31.3","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293628+0000","last_change":"2023-10-14T18:18:35.675054+0000","last_active":"2023-10-14T18:18:50.293628+0000","last_peered":"2023-10-14T18:18:50.293628+0000","last_clean":"2023-10-14T18:18:50.293628+0000","last_became_active":"2023-10-14T18:18:35.674739+0000","last_became_peered":"2023-10-14T18:18:35.674739+0000","last_unstale":"2023-10-14T18:18:50.293628+0000","last_undegraded":"2023-10-14T18:18:50.293628+0000","last_fullsized":"2023-10-14T18:18:50.293628+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:04:44.170891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.2","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.300733+0000","last_change":"2023-10-14T18:18:34.652028+0000","last_active":"2023-10-14T18:18:50.300733+0000","last_peered":"2023-10-14T18:18:50.300733+0000","last_clean":"2023-10-14T18:18:50.300733+0000","last_became_active":"2023-10-14T18:18:34.651757+0000","last_became_peered":"2023-10-14T18:18:34.651757+0000","last_unstale":"2023-10-14T18:18:50.300733+0000","last_undegraded":"2023-10-14T18:18:50.300733+0000","last_fullsized":"2023-10-14T18:18:50.300733+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:20.817765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"33.3e","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.377140+0000","last_change":"2023-10-14T18:18:50.321495+0000","last_active":"2023-10-14T18:18:51.377140+0000","last_peered":"2023-10-14T18:18:51.377140+0000","last_clean":"2023-10-14T18:18:51.377140+0000","last_became_active":"2023-10-14T18:18:50.320848+0000","last_became_peered":"2023-10-14T18:18:50.320848+0000","last_unstale":"2023-10-14T18:18:51.377140+0000","last_undegraded":"2023-10-14T18:18:51.377140+0000","last_fullsized":"2023-10-14T18:18:51.377140+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:20:49.661634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"31.0","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.293751+0000","last_change":"2023-10-14T18:18:35.660261+0000","last_active":"2023-10-14T18:18:50.293751+0000","last_peered":"2023-10-14T18:18:50.293751+0000","last_clean":"2023-10-14T18:18:50.293751+0000","last_became_active":"2023-10-14T18:18:35.659486+0000","last_became_peered":"2023-10-14T18:18:35.659486+0000","last_unstale":"2023-10-14T18:18:50.293751+0000","last_undegraded":"2023-10-14T18:18:50.293751+0000","last_fullsized":"2023-10-14T18:18:50.293751+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:29:01.766750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"30.1","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.306115+0000","last_change":"2023-10-14T18:18:34.643288+0000","last_active":"2023-10-14T18:18:50.306115+0000","last_peered":"2023-10-14T18:18:50.306115+0000","last_clean":"2023-10-14T18:18:50.306115+0000","last_became_active":"2023-10-14T18:18:34.642997+0000","last_became_peered":"2023-10-14T18:18:34.642997+0000","last_unstale":"2023-10-14T18:18:50.306115+0000","last_undegraded":"2023-10-14T18:18:50.306115+0000","last_fullsized":"2023-10-14T18:18:50.306115+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:46:39.884500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.3f","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.294652+0000","last_change":"2023-10-14T18:18:50.312703+0000","last_active":"2023-10-14T18:18:51.294652+0000","last_peered":"2023-10-14T18:18:51.294652+0000","last_clean":"2023-10-14T18:18:51.294652+0000","last_became_active":"2023-10-14T18:18:50.312145+0000","last_became_peered":"2023-10-14T18:18:50.312145+0000","last_unstale":"2023-10-14T18:18:51.294652+0000","last_undegraded":"2023-10-14T18:18:51.294652+0000","last_fullsized":"2023-10-14T18:18:51.294652+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:58:15.901887+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"31.1","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.295329+0000","last_change":"2023-10-14T18:18:35.674240+0000","last_active":"2023-10-14T18:18:50.295329+0000","last_peered":"2023-10-14T18:18:50.295329+0000","last_clean":"2023-10-14T18:18:50.295329+0000","last_became_active":"2023-10-14T18:18:35.671067+0000","last_became_peered":"2023-10-14T18:18:35.671067+0000","last_unstale":"2023-10-14T18:18:50.295329+0000","last_undegraded":"2023-10-14T18:18:50.295329+0000","last_fullsized":"2023-10-14T18:18:50.295329+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:02:59.948900+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"30.0","version":"0'0","reported_seq":16,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.310453+0000","last_change":"2023-10-14T18:18:34.643687+0000","last_active":"2023-10-14T18:18:50.310453+0000","last_peered":"2023-10-14T18:18:50.310453+0000","last_clean":"2023-10-14T18:18:50.310453+0000","last_became_active":"2023-10-14T18:18:34.643092+0000","last_became_peered":"2023-10-14T18:18:34.643092+0000","last_unstale":"2023-10-14T18:18:50.310453+0000","last_undegraded":"2023-10-14T18:18:50.310453+0000","last_fullsized":"2023-10-14T18:18:50.310453+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:05:14.480279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"33.20","version":"0'0","reported_seq":11,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.297169+0000","last_change":"2023-10-14T18:18:50.305214+0000","last_active":"2023-10-14T18:18:51.297169+0000","last_peered":"2023-10-14T18:18:51.297169+0000","last_clean":"2023-10-14T18:18:51.297169+0000","last_became_active":"2023-10-14T18:18:50.304899+0000","last_became_peered":"2023-10-14T18:18:50.304899+0000","last_unstale":"2023-10-14T18:18:51.297169+0000","last_undegraded":"2023-10-14T18:18:51.297169+0000","last_fullsized":"2023-10-14T18:18:51.297169+0000","mapping_epoch":252,"log_start":"0'0","ondisk_log_start":"0'0","created":252,"last_epoch_clean":253,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:49.259615+0000","last_clean_scrub_stamp":"2023-10-14T18:18:49.259615+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:28:27.178651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"31.1e","version":"0'0","reported_seq":15,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309927+0000","last_change":"2023-10-14T18:18:35.685028+0000","last_active":"2023-10-14T18:18:50.309927+0000","last_peered":"2023-10-14T18:18:50.309927+0000","last_clean":"2023-10-14T18:18:50.309927+0000","last_became_active":"2023-10-14T18:18:35.684687+0000","last_became_peered":"2023-10-14T18:18:35.684687+0000","last_unstale":"2023-10-14T18:18:50.309927+0000","last_undegraded":"2023-10-14T18:18:50.309927+0000","last_fullsized":"2023-10-14T18:18:50.309927+0000","mapping_epoch":248,"log_start":"0'0","ondisk_log_start":"0'0","created":248,"last_epoch_clean":249,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:34.608960+0000","last_clean_scrub_stamp":"2023-10-14T18:18:34.608960+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:49:27.530239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"30.1f","version":"249'4","reported_seq":20,"reported_epoch":253,"state":"active+clean","last_fresh":"2023-10-14T18:18:50.309953+0000","last_change":"2023-10-14T18:18:34.640893+0000","last_active":"2023-10-14T18:18:50.309953+0000","last_peered":"2023-10-14T18:18:50.309953+0000","last_clean":"2023-10-14T18:18:50.309953+0000","last_became_active":"2023-10-14T18:18:34.639717+0000","last_became_peered":"2023-10-14T18:18:34.639717+0000","last_unstale":"2023-10-14T18:18:50.309953+0000","last_undegraded":"2023-10-14T18:18:50.309953+0000","last_fullsized":"2023-10-14T18:18:50.309953+0000","mapping_epoch":247,"log_start":"0'0","ondisk_log_start":"0'0","created":247,"last_epoch_clean":248,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:18:33.596542+0000","last_clean_scrub_stamp":"2023-10-14T18:18:33.596542+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:26:55.441603+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"235'370","reported_seq":1313,"reported_epoch":254,"state":"active+clean","last_fresh":"2023-10-14T18:18:51.375631+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:18:51.375631+0000","last_peered":"2023-10-14T18:18:51.375631+0000","last_clean":"2023-10-14T18:18:51.375631+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:18:51.375631+0000","last_undegraded":"2023-10-14T18:18:51.375631+0000","last_fullsized":"2023-10-14T18:18:51.375631+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":370,"log_dups_size":0,"ondisk_log_size":370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":33,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":32,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":31,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":30,"num_pg":32,"stat_sum":{"num_bytes":3655,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":47,"num_write_kb":16,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":7310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":41,"ondisk_log_size":41,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2324,"num_read_kb":4838,"num_write":619,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":370,"ondisk_log_size":370,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576021,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57100,"kb_used_data":3708,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314740,"statfs":{"total":96636764160,"available":96578293760,"internally_reserved":0,"allocated":3796992,"data_stored":2186619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1319999999999999,"5min":1.177,"15min":1.2030000000000001},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":2.0489999999999999,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":1.163,"5min":1.2130000000000001,"15min":1.2310000000000001},"min":{"1min":0.76600000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":1.78,"5min":1.78,"15min":2.2149999999999999},"last":0.90400000000000003}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.236,"5min":1.2250000000000001,"15min":1.2749999999999999},"min":{"1min":0.81399999999999995,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.768,"5min":1.9059999999999999,"15min":2.214},"last":1.2669999999999999},{"interface":"front","average":{"1min":1.145,"5min":1.2909999999999999,"15min":1.3109999999999999},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.71,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.224}]},{"osd":2,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.202,"5min":1.206,"15min":1.26},"min":{"1min":0.83199999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.839,"5min":1.839,"15min":2.004},"last":1.294},{"interface":"front","average":{"1min":1.2849999999999999,"5min":1.3100000000000001,"15min":1.365},"min":{"1min":0.81100000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"max":{"1min":1.9390000000000001,"5min":1.9390000000000001,"15min":2.2010000000000001},"last":0.85099999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.272,"5min":1.347,"15min":1.4239999999999999},"min":{"1min":0.94799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.8700000000000001,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":1.159},{"interface":"front","average":{"1min":1.454,"5min":1.488,"15min":1.528},"min":{"1min":1.0600000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.994,"5min":1.994,"15min":2.2810000000000001},"last":1.03}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4790000000000001,"5min":1.3680000000000001,"15min":1.343},"min":{"1min":0.90500000000000003,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.367},"last":1.0649999999999999},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.3759999999999999,"15min":1.3919999999999999},"min":{"1min":1.0529999999999999,"5min":0.90900000000000003,"15min":0.90900000000000003},"max":{"1min":2.2440000000000002,"5min":2.2440000000000002,"15min":2.661},"last":1.403}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4870000000000001,"5min":1.4370000000000001,"15min":1.4670000000000001},"min":{"1min":1.0409999999999999,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.958,"5min":1.958,"15min":2.516},"last":1.129},{"interface":"front","average":{"1min":1.554,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":1.008,"5min":1.008,"15min":1.008},"max":{"1min":2.2559999999999998,"5min":2.2559999999999998,"15min":2.448},"last":1.5089999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.669,"5min":1.615,"15min":1.639},"min":{"1min":1.103,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.4140000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.3640000000000001},{"interface":"front","average":{"1min":1.51,"5min":1.5289999999999999,"15min":1.577},"min":{"1min":1.1399999999999999,"5min":0.96999999999999997,"15min":0.80500000000000005},"max":{"1min":2.1040000000000001,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.4690000000000001}]}]},{"osd":6,"up_from":13,"seq":55834576020,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48208,"kb_used_data":3136,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323632,"statfs":{"total":96636764160,"available":96587399168,"internally_reserved":0,"allocated":3211264,"data_stored":1596827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.0740000000000001,"5min":1.167,"15min":1.1779999999999999},"min":{"1min":0.66600000000000004,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":1.669,"5min":2.085,"15min":2.085},"last":1.901},{"interface":"front","average":{"1min":1.1479999999999999,"5min":1.1779999999999999,"15min":1.2070000000000001},"min":{"1min":0.752,"5min":0.752,"15min":0.752},"max":{"1min":1.704,"5min":1.9830000000000001,"15min":1.9830000000000001},"last":1.371}]},{"osd":1,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.165,"5min":1.208,"15min":1.24},"min":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.494,"5min":1.762,"15min":1.776},"last":1.095},{"interface":"front","average":{"1min":1.3440000000000001,"5min":1.347,"15min":1.3600000000000001},"min":{"1min":0.81799999999999995,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.1760000000000002,"5min":2.2109999999999999,"15min":2.2109999999999999},"last":1.659}]},{"osd":2,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.385,"5min":1.3740000000000001,"15min":1.377},"min":{"1min":0.85999999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.819,"5min":1.9670000000000001,"15min":2.4849999999999999},"last":1.9399999999999999},{"interface":"front","average":{"1min":1.373,"5min":1.397,"15min":1.419},"min":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":2,"5min":2.0289999999999999,"15min":2.0289999999999999},"last":1.704}]},{"osd":3,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.347,"15min":1.4059999999999999},"min":{"1min":0.72799999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.0600000000000001,"5min":2.0600000000000001,"15min":2.2519999999999998},"last":1.532},{"interface":"front","average":{"1min":1.357,"5min":1.411,"15min":1.4359999999999999},"min":{"1min":0.91300000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":1.8220000000000001,"5min":2.4359999999999999,"15min":2.4359999999999999},"last":1.875}]},{"osd":4,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.4319999999999999,"5min":1.3899999999999999,"15min":1.3859999999999999},"min":{"1min":0.96499999999999997,"5min":0.75800000000000001,"15min":0.75800000000000001},"max":{"1min":1.7949999999999999,"5min":1.9950000000000001,"15min":2.0299999999999998},"last":1.8280000000000001},{"interface":"front","average":{"1min":1.5409999999999999,"5min":1.429,"15min":1.4370000000000001},"min":{"1min":1.109,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":2.0030000000000001,"5min":2.5379999999999998,"15min":2.5379999999999998},"last":1.4970000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.522,"5min":1.411,"15min":1.419},"min":{"1min":1.024,"5min":0.624,"15min":0.624},"max":{"1min":2.1389999999999998,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.4570000000000001},{"interface":"front","average":{"1min":1.6930000000000001,"5min":1.5780000000000001,"15min":1.5840000000000001},"min":{"1min":1.2310000000000001,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.2810000000000001,"5min":2.2810000000000001,"15min":2.5609999999999999},"last":2.0750000000000002}]},{"osd":7,"last update":"Sat Oct 14 18:18:16 2023","interfaces":[{"interface":"back","average":{"1min":1.726,"5min":1.6140000000000001,"15min":1.6180000000000001},"min":{"1min":1.403,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":2.194,"5min":2.903,"15min":3.423},"last":1.6120000000000001},{"interface":"front","average":{"1min":1.6479999999999999,"5min":1.583,"15min":1.6040000000000001},"min":{"1min":1.296,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.2349999999999999,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.972}]}]},{"osd":5,"up_from":13,"seq":55834576021,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":42620,"kb_used_data":3120,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":94329220,"statfs":{"total":96636764160,"available":96593121280,"internally_reserved":0,"allocated":3194880,"data_stored":1596111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40434118},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.22,"5min":1.1719999999999999,"15min":1.2070000000000001},"min":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":2.052,"5min":2.052,"15min":2.052},"last":0.64300000000000002},{"interface":"front","average":{"1min":1.089,"5min":1.1739999999999999,"15min":1.2350000000000001},"min":{"1min":0.78800000000000003,"5min":0.70299999999999996,"15min":0.70299999999999996},"max":{"1min":1.629,"5min":1.7270000000000001,"15min":2.1070000000000002},"last":0.59999999999999998}]},{"osd":1,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.234,"15min":1.276},"min":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.6850000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.57599999999999996},{"interface":"front","average":{"1min":1.335,"5min":1.296,"15min":1.3100000000000001},"min":{"1min":0.92100000000000004,"5min":0.73599999999999999,"15min":0.73599999999999999},"max":{"1min":2.3929999999999998,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.69499999999999995}]},{"osd":2,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.0629999999999999,"5min":1.1799999999999999,"15min":1.2430000000000001},"min":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"max":{"1min":1.591,"5min":1.764,"15min":3.7090000000000001},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.3979999999999999,"5min":1.377,"15min":1.397},"min":{"1min":0.90000000000000002,"5min":0.86699999999999999,"15min":0.86599999999999999},"max":{"1min":1.8779999999999999,"5min":1.8779999999999999,"15min":1.923},"last":0.76000000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3939999999999999,"5min":1.4159999999999999,"15min":1.46},"min":{"1min":0.94299999999999995,"5min":0.94199999999999995,"15min":0.88900000000000001},"max":{"1min":2.1930000000000001,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":0.79600000000000004},{"interface":"front","average":{"1min":1.3089999999999999,"5min":1.403,"15min":1.4339999999999999},"min":{"1min":0.85699999999999998,"5min":0.85699999999999998,"15min":0.85699999999999998},"max":{"1min":2.1000000000000001,"5min":2.1000000000000001,"15min":3.8330000000000002},"last":0.88500000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3879999999999999,"15min":1.3660000000000001},"min":{"1min":1.0700000000000001,"5min":0.85899999999999999,"15min":0.78400000000000003},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.98},"last":0.81999999999999995},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.4370000000000001,"15min":1.401},"min":{"1min":0.90400000000000003,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.516,"5min":2.516,"15min":3.1280000000000001},"last":0.85599999999999998}]},{"osd":6,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.4350000000000001,"5min":1.399,"15min":1.4259999999999999},"min":{"1min":0.92900000000000005,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1339999999999999,"5min":2.1339999999999999,"15min":4.0540000000000003},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.7130000000000001,"5min":1.6259999999999999,"15min":1.635},"min":{"1min":1.429,"5min":0.93000000000000005,"15min":0.81799999999999995},"max":{"1min":2.7810000000000001,"5min":2.7810000000000001,"15min":3.6360000000000001},"last":0.91500000000000004}]},{"osd":7,"last update":"Sat Oct 14 18:18:00 2023","interfaces":[{"interface":"back","average":{"1min":1.6719999999999999,"5min":1.5609999999999999,"15min":1.585},"min":{"1min":1.121,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.6640000000000001,"5min":2.6640000000000001,"15min":3.2759999999999998},"last":0.96399999999999997},{"interface":"front","average":{"1min":1.6200000000000001,"5min":1.5840000000000001,"15min":1.5680000000000001},"min":{"1min":1.1419999999999999,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.9390000000000001},"last":0.93999999999999995}]}]},{"osd":4,"up_from":13,"seq":55834576021,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66212,"kb_used_data":3160,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305628,"statfs":{"total":96636764160,"available":96568963072,"internally_reserved":0,"allocated":3235840,"data_stored":1600728,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.1439999999999999,"5min":1.181,"15min":1.2010000000000001},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"max":{"1min":2.2149999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.88400000000000001},{"interface":"front","average":{"1min":1.2869999999999999,"5min":1.266,"15min":1.2829999999999999},"min":{"1min":0.79600000000000004,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.2610000000000001,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":0.64800000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3080000000000001,"5min":1.282,"15min":1.3240000000000001},"min":{"1min":0.751,"5min":0.751,"15min":0.59399999999999997},"max":{"1min":2.0129999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.1659999999999999},{"interface":"front","average":{"1min":1.3280000000000001,"5min":1.248,"15min":1.2689999999999999},"min":{"1min":0.875,"5min":0.66800000000000004,"15min":0.66800000000000004},"max":{"1min":2.1850000000000001,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.222}]},{"osd":2,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3200000000000001,"5min":1.2889999999999999,"15min":1.286},"min":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"max":{"1min":2.3570000000000002,"5min":2.3570000000000002,"15min":3.524},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.216,"5min":1.389,"15min":1.4299999999999999},"min":{"1min":0.83499999999999996,"5min":0.83499999999999996,"15min":0.65700000000000003},"max":{"1min":1.597,"5min":1.9830000000000001,"15min":2.2440000000000002},"last":1.2909999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.3069999999999999,"5min":1.2789999999999999,"15min":1.343},"min":{"1min":1.0029999999999999,"5min":0.879,"15min":0.80200000000000005},"max":{"1min":1.6739999999999999,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.4359999999999999},{"interface":"front","average":{"1min":1.4259999999999999,"5min":1.4199999999999999,"15min":1.4350000000000001},"min":{"1min":1.0009999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.5369999999999999,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.204}]},{"osd":5,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.4019999999999999,"5min":1.4350000000000001,"15min":1.397},"min":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.69299999999999995},"max":{"1min":1.923,"5min":1.9319999999999999,"15min":2.02},"last":1.389},{"interface":"front","average":{"1min":1.498,"5min":1.393,"15min":1.3839999999999999},"min":{"1min":0.90000000000000002,"5min":0.83899999999999997,"15min":0.64500000000000002},"max":{"1min":2.407,"5min":2.407,"15min":2.407},"last":1.4990000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.46,"5min":1.452,"15min":1.4370000000000001},"min":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"max":{"1min":2.4500000000000002,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.3200000000000001},{"interface":"front","average":{"1min":1.5569999999999999,"5min":1.577,"15min":1.569},"min":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.82699999999999996},"max":{"1min":2.1379999999999999,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.468}]},{"osd":7,"last update":"Sat Oct 14 18:18:11 2023","interfaces":[{"interface":"back","average":{"1min":1.6020000000000001,"5min":1.5229999999999999,"15min":1.522},"min":{"1min":1.248,"5min":1.024,"15min":0.77600000000000002},"max":{"1min":1.946,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.3500000000000001},{"interface":"front","average":{"1min":1.6319999999999999,"5min":1.6080000000000001,"15min":1.5940000000000001},"min":{"1min":1.2889999999999999,"5min":1.101,"15min":0.93799999999999994},"max":{"1min":2.597,"5min":2.597,"15min":3.8610000000000002},"last":1.2689999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576021,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56536,"kb_used_data":3140,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315304,"statfs":{"total":96636764160,"available":96578871296,"internally_reserved":0,"allocated":3215360,"data_stored":1598887,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.94399999999999995,"15min":0.97599999999999998},"min":{"1min":0.45700000000000002,"5min":0.377,"15min":0.377},"max":{"1min":1.4239999999999999,"5min":1.4870000000000001,"15min":1.855},"last":0.96299999999999997},{"interface":"front","average":{"1min":0.91100000000000003,"5min":1.1080000000000001,"15min":1.1870000000000001},"min":{"1min":0.47899999999999998,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.504,"5min":1.6399999999999999,"15min":1.9970000000000001},"last":1.0309999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":0.83399999999999996,"5min":0.94799999999999995,"15min":0.996},"min":{"1min":0.47099999999999997,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.363,"5min":1.835,"15min":1.919},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.03,"5min":1.139,"15min":1.1970000000000001},"min":{"1min":0.53800000000000003,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.742,"5min":1.742,"15min":1.8560000000000001},"last":1.292}]},{"osd":2,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.006,"5min":1.339,"15min":1.4159999999999999},"min":{"1min":0.54900000000000004,"5min":0.498,"15min":0.498},"max":{"1min":1.5620000000000001,"5min":2.032,"15min":2.032},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.04,"5min":1.26,"15min":1.3640000000000001},"min":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.9330000000000001,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.429}]},{"osd":4,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.242,"5min":1.3819999999999999,"15min":1.429},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9099999999999999,"5min":2.153,"15min":2.153},"last":1.579},{"interface":"front","average":{"1min":1.403,"5min":1.4490000000000001,"15min":1.478},"min":{"1min":0.85099999999999998,"5min":0.83699999999999997,"15min":0.747},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.639}]},{"osd":5,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.2789999999999999,"5min":1.4239999999999999,"15min":1.4550000000000001},"min":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":3.3170000000000002,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.758},{"interface":"front","average":{"1min":1.367,"5min":1.46,"15min":1.548},"min":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":2.02,"5min":2.1360000000000001,"15min":2.891},"last":1.4810000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.4610000000000001,"5min":1.5700000000000001,"15min":1.621},"min":{"1min":0.97199999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.3319999999999999,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.931},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.6339999999999999,"15min":1.677},"min":{"1min":0.93000000000000005,"5min":0.91900000000000004,"15min":0.91000000000000003},"max":{"1min":2.6840000000000002,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.8149999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:08 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.532,"15min":1.599},"min":{"1min":0.85499999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":3.153,"5min":3.153,"15min":3.153},"last":1.885},{"interface":"front","average":{"1min":1.5489999999999999,"5min":1.635,"15min":1.681},"min":{"1min":0.89900000000000002,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.7829999999999999,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.7110000000000001}]}]},{"osd":2,"up_from":13,"seq":55834576021,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56512,"kb_used_data":3124,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315328,"statfs":{"total":96636764160,"available":96578895872,"internally_reserved":0,"allocated":3198976,"data_stored":1596201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.95799999999999996,"15min":0.94399999999999995},"min":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"max":{"1min":1.3029999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.80000000000000004},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.98099999999999998,"15min":1.081},"min":{"1min":0.56100000000000005,"5min":0.438,"15min":0.438},"max":{"1min":1.2629999999999999,"5min":1.4550000000000001,"15min":1.8660000000000001},"last":0.99099999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":0.82899999999999996,"5min":0.94099999999999995,"15min":1.0069999999999999},"min":{"1min":0.44,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.1439999999999999,"5min":1.603,"15min":1.881},"last":1.3620000000000001},{"interface":"front","average":{"1min":0.95499999999999996,"5min":1.1679999999999999,"15min":1.212},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.5780000000000001,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":0.96999999999999997}]},{"osd":3,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.077,"5min":1.218,"15min":1.2749999999999999},"min":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.3959999999999999,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":0.93600000000000005},{"interface":"front","average":{"1min":1.081,"5min":1.337,"15min":1.3779999999999999},"min":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7490000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.0820000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.256,"5min":1.4650000000000001,"15min":1.464},"min":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.736,"5min":2.097,"15min":2.097},"last":1.738},{"interface":"front","average":{"1min":1.2949999999999999,"5min":1.3300000000000001,"15min":1.365},"min":{"1min":0.72399999999999998,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":1.944,"5min":1.944,"15min":2.024},"last":1.9379999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.3420000000000001,"5min":1.389,"15min":1.4350000000000001},"min":{"1min":0.72099999999999997,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":2.165,"5min":2.165,"15min":2.2839999999999998},"last":1.786},{"interface":"front","average":{"1min":1.339,"5min":1.577,"15min":1.583},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.0049999999999999,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.6080000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.446,"5min":1.52,"15min":1.5289999999999999},"min":{"1min":0.80600000000000005,"5min":0.747,"15min":0.747},"max":{"1min":2.238,"5min":2.238,"15min":2.238},"last":2.2690000000000001},{"interface":"front","average":{"1min":1.4339999999999999,"5min":1.556,"15min":1.603},"min":{"1min":0.91100000000000003,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.1400000000000001,"5min":2.2349999999999999,"15min":2.3180000000000001},"last":2.085}]},{"osd":7,"last update":"Sat Oct 14 18:18:28 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.5780000000000001,"15min":1.5900000000000001},"min":{"1min":0.94699999999999995,"5min":0.81699999999999995,"15min":0.81699999999999995},"max":{"1min":2.0569999999999999,"5min":2.1459999999999999,"15min":2.2370000000000001},"last":2.1309999999999998},{"interface":"front","average":{"1min":1.498,"5min":1.633,"15min":1.675},"min":{"1min":0.97999999999999998,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.0600000000000001,"5min":2.331,"15min":2.331},"last":2.0110000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576021,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52388,"kb_used_data":3156,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319452,"statfs":{"total":96636764160,"available":96583118848,"internally_reserved":0,"allocated":3231744,"data_stored":1599868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.3400000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.81499999999999995},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0169999999999999,"15min":1.1080000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1659999999999999,"5min":2.847,"15min":2.847},"last":1.238}]},{"osd":2,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.046},"min":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4590000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":1.492},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1799999999999999,"15min":1.254},"min":{"1min":0.67300000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6279999999999999,"5min":1.996,"15min":2.6859999999999999},"last":1.119}]},{"osd":3,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.2769999999999999,"15min":1.3700000000000001},"min":{"1min":0.66300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7370000000000001,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.879},{"interface":"front","average":{"1min":1.056,"5min":1.226,"15min":1.296},"min":{"1min":0.68899999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5609999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.429}]},{"osd":4,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.417,"15min":1.46},"min":{"1min":0.88300000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9259999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3740000000000001,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.1619999999999999,"5min":2.218,"15min":2.899},"last":1.3380000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.425,"15min":1.472},"min":{"1min":0.997,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":3.004},"last":1.173},{"interface":"front","average":{"1min":1.403,"5min":1.496,"15min":1.5569999999999999},"min":{"1min":0.90400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.5129999999999999},"last":1.5669999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.54,"15min":1.601},"min":{"1min":1.171,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0369999999999999,"5min":2.8999999999999999,"15min":2.931},"last":1.7669999999999999},{"interface":"front","average":{"1min":1.518,"5min":1.6220000000000001,"15min":1.623},"min":{"1min":1.1140000000000001,"5min":0.871,"15min":0.871},"max":{"1min":2.0449999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.6240000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.619,"15min":1.6890000000000001},"min":{"1min":1.0029999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.3290000000000002,"5min":3.363,"15min":3.363},"last":1.696},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.651,"15min":1.7090000000000001},"min":{"1min":1.05,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.206,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.964}]}]},{"osd":0,"up_from":13,"seq":55834576020,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48772,"kb_used_data":3704,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323068,"statfs":{"total":96636764160,"available":96586821632,"internally_reserved":0,"allocated":3792896,"data_stored":2186513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":0.88800000000000001,"5min":0.94599999999999995,"15min":0.97299999999999998},"min":{"1min":0.57699999999999996,"5min":0.48699999999999999,"15min":0.48699999999999999},"max":{"1min":1.367,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":1.52},{"interface":"front","average":{"1min":0.93799999999999994,"5min":1.0980000000000001,"15min":1.135},"min":{"1min":0.55500000000000005,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.3560000000000001,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.88800000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.034,"5min":1.143,"15min":1.1759999999999999},"min":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.6379999999999999,"5min":1.855,"15min":1.855},"last":1.008},{"interface":"front","average":{"1min":1.1200000000000001,"5min":1.167,"15min":1.2390000000000001},"min":{"1min":0.67300000000000004,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.522,"5min":1.845,"15min":2.5779999999999998},"last":1.2290000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.3100000000000001,"15min":1.3700000000000001},"min":{"1min":0.52100000000000002,"5min":0.501,"15min":0.501},"max":{"1min":1.5429999999999999,"5min":1.804,"15min":2.0720000000000001},"last":1.099},{"interface":"front","average":{"1min":1.1220000000000001,"5min":1.3300000000000001,"15min":1.391},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.56599999999999995},"max":{"1min":1.77,"5min":1.883,"15min":2.194},"last":1.619}]},{"osd":4,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.1379999999999999,"5min":1.284,"15min":1.3169999999999999},"min":{"1min":0.73999999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.5549999999999999,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.157},{"interface":"front","average":{"1min":1.4079999999999999,"5min":1.492,"15min":1.502},"min":{"1min":0.999,"5min":0.88900000000000001,"15min":0.88900000000000001},"max":{"1min":1.984,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.3169999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2190000000000001,"5min":1.327,"15min":1.3680000000000001},"min":{"1min":0.80800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.9359999999999999,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.577},{"interface":"front","average":{"1min":1.429,"5min":1.51,"15min":1.569},"min":{"1min":0.78700000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.242,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.464}]},{"osd":6,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.464,"5min":1.6259999999999999,"15min":1.651},"min":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.1709999999999998,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.276,"5min":1.4830000000000001,"15min":1.508},"min":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":1.6899999999999999,"5min":2.1280000000000001,"15min":2.2799999999999998},"last":1.2709999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4910000000000001,"5min":1.6040000000000001,"15min":1.6479999999999999},"min":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"max":{"1min":2.0649999999999999,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.6519999999999999},{"interface":"front","average":{"1min":1.343,"5min":1.5589999999999999,"15min":1.6160000000000001},"min":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"max":{"1min":1.857,"5min":2.1869999999999998,"15min":2.3620000000000001},"last":1.411}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2981,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2949,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":30,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":31,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2194,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":32,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":33,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:18:55.398 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:18:55.399 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:18:55.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.657+0000 7f536b59e640 1 Processor -- start 2023-10-14T18:18:55.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.657+0000 7f536b59e640 1 -- start start 2023-10-14T18:18:55.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f536c0c9060 0x7f536c06f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f536c0c9440 0x7f536c06f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f536c0ca560 con 0x7f536c06c9e0 2023-10-14T18:18:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c0c9950 con 0x7f536c0c9060 2023-10-14T18:18:55.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f536c0c9c30 con 0x7f536c0c9440 2023-10-14T18:18:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f536c0c9060 0x7f536c06f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f536c0c9060 0x7f536c06f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40890/0 (socket says 172.21.15.29:40890) 2023-10-14T18:18:55.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.659+0000 7f536a59c640 1 -- 172.21.15.29:0/971417766 learned_addr learned my addr 172.21.15.29:0/971417766 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:55.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.660+0000 7f536ad9d640 1 --2- 172.21.15.29:0/971417766 >> v2:172.21.15.186:3300/0 conn(0x7f536c0c9440 0x7f536c06f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.660+0000 7f536a59c640 1 -- 172.21.15.29:0/971417766 >> v2:172.21.15.186:3300/0 conn(0x7f536c0c9440 msgr2=0x7f536c06f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.660+0000 7f536a59c640 1 --2- 172.21.15.29:0/971417766 >> v2:172.21.15.186:3300/0 conn(0x7f536c0c9440 0x7f536c06f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.660+0000 7f536a59c640 1 -- 172.21.15.29:0/971417766 >> v1:172.21.15.186:6789/0 conn(0x7f536c06c9e0 legacy=0x7f536c164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:55.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.660+0000 7f536a59c640 1 -- 172.21.15.29:0/971417766 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f536c0c9f10 con 0x7f536c0c9060 2023-10-14T18:18:55.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.661+0000 7f536a59c640 1 --2- 172.21.15.29:0/971417766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c06f440 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f536c0ce550 tx=0x7f535c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cebe438d1fd3ac server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536959a640 1 -- 172.21.15.29:0/971417766 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c031870 con 0x7f536c0c9060 2023-10-14T18:18:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536959a640 1 -- 172.21.15.29:0/971417766 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f535c0319d0 con 0x7f536c0c9060 2023-10-14T18:18:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536959a640 1 -- 172.21.15.29:0/971417766 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f535c031d60 con 0x7f536c0c9060 2023-10-14T18:18:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 -- 172.21.15.29:0/971417766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 msgr2=0x7f536c06f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 --2- 172.21.15.29:0/971417766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c06f440 secure :-1 s=READY pgs=2935 cs=0 l=1 rev1=1 crypto rx=0x7f536c0ce550 tx=0x7f535c02b910 comp rx=0 tx=0).stop 2023-10-14T18:18:55.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 -- 172.21.15.29:0/971417766 shutdown_connections 2023-10-14T18:18:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 --2- 172.21.15.29:0/971417766 >> v2:172.21.15.186:3300/0 conn(0x7f536c0c9440 0x7f536c06f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 --2- 172.21.15.29:0/971417766 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c06f440 unknown :-1 s=CLOSED pgs=2935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 -- 172.21.15.29:0/971417766 >> 172.21.15.29:0/971417766 conn(0x7f536c0ce6a0 msgr2=0x7f536c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.662+0000 7f536b59e640 1 -- 172.21.15.29:0/971417766 shutdown_connections 2023-10-14T18:18:55.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.663+0000 7f536b59e640 1 -- 172.21.15.29:0/971417766 wait complete. 2023-10-14T18:18:55.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.664+0000 7f536b59e640 1 Processor -- start 2023-10-14T18:18:55.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.664+0000 7f536b59e640 1 -- start start 2023-10-14T18:18:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c06c9e0 0x7f536c0c7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0c9440 0x7f536c163f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c0c6c80 con 0x7f536c0c9060 2023-10-14T18:18:55.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536c0c6f60 con 0x7f536c0c9440 2023-10-14T18:18:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.665+0000 7f536b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f536c0c7240 con 0x7f536c06c9e0 2023-10-14T18:18:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40894/0 (socket says 172.21.15.29:40894) 2023-10-14T18:18:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 -- 172.21.15.29:0/1043537220 learned_addr learned my addr 172.21.15.29:0/1043537220 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:55.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f536a59c640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c06c9e0 0x7f536c0c7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f536ad9d640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0c9440 0x7f536c163f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 -- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c06c9e0 msgr2=0x7f536c0c7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c06c9e0 0x7f536c0c7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 -- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0c9440 msgr2=0x7f536c163f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0c9440 0x7f536c163f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.666+0000 7f5369d9b640 1 -- 172.21.15.29:0/1043537220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f535c0094d0 con 0x7f536c0c9060 2023-10-14T18:18:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f5369d9b640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f536000c400 tx=0x7f536000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360011790 con 0x7f536c0c9060 2023-10-14T18:18:55.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5360011930 con 0x7f536c0c9060 2023-10-14T18:18:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5360011ce0 con 0x7f536c0c9060 2023-10-14T18:18:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f536c0c7520 con 0x7f536c0c9060 2023-10-14T18:18:55.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.667+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f536c0b4360 con 0x7f536c0c9060 2023-10-14T18:18:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.668+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536c0c7a40 con 0x7f536c0c9060 2023-10-14T18:18:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.669+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f536001c460 con 0x7f536c0c9060 2023-10-14T18:18:55.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.670+0000 7f535affd640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 0x7f534c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.670+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f5360089ca0 con 0x7f536c0c9060 2023-10-14T18:18:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.671+0000 7f536a59c640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 0x7f534c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.671+0000 7f536a59c640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 0x7f534c06d5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f535c036000 tx=0x7f535c03c040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:55.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.674+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f536c0c7a40 con 0x7f536c0c9060 2023-10-14T18:18:55.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.914+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f536c0c7a40 con 0x7f536c0c9060 2023-10-14T18:18:55.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.915+0000 7f535affd640 1 -- 172.21.15.29:0/1043537220 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7f536c0c7a40 con 0x7f536c0c9060 2023-10-14T18:18:55.918 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:18:55.919 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:18:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:18:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 msgr2=0x7f534c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 0x7f534c06d5a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f535c036000 tx=0x7f535c03c040 comp rx=0 tx=0).stop 2023-10-14T18:18:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 msgr2=0x7f536c0b3e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:55.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 secure :-1 s=READY pgs=2936 cs=0 l=1 rev1=1 crypto rx=0x7f536000c400 tx=0x7f536000c8c0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 shutdown_connections 2023-10-14T18:18:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f534c06b0e0 0x7f534c06d5a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f536c0c9440 0x7f536c163f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f536c0c9060 0x7f536c0b3e20 unknown :-1 s=CLOSED pgs=2936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 --2- 172.21.15.29:0/1043537220 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f536c06c9e0 0x7f536c0c7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:55.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 >> 172.21.15.29:0/1043537220 conn(0x7f536c0ce6a0 msgr2=0x7f536c06afe0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:55.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 shutdown_connections 2023-10-14T18:18:55.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:55.918+0000 7f536b59e640 1 -- 172.21.15.29:0/1043537220 wait complete. 2023-10-14T18:18:56.340 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 129, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:18:49.773591+0000', 'modified': '2023-10-14T18:18:52.835381+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7637}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7637': {'gid': 7637, 'name': 'd', 'rank': 0, 'incarnation': 125, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.186:6835/2447498482', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6834', 'nonce': 2447498482}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 2447498482}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [33], 'metadata_pool': 32, 'enabled': True, 'fs_name': 'backup_fs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:18:56.341 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:18:56.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T18:18:56.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.578+0000 7f305d4df640 1 Processor -- start 2023-10-14T18:18:56.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.579+0000 7f305d4df640 1 -- start start 2023-10-14T18:18:56.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.580+0000 7f305d4df640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f30580c9060 0x7f305806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.580+0000 7f305d4df640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f305806c9e0 0x7f305806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.580+0000 7f305d4df640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30580ca560 con 0x7f30580c9440 2023-10-14T18:18:56.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.580+0000 7f305d4df640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580c9950 con 0x7f305806c9e0 2023-10-14T18:18:56.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.580+0000 7f305d4df640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30580c9c30 con 0x7f30580c9060 2023-10-14T18:18:56.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f304f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f305806c9e0 0x7f305806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f305ccde640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30580c9440 0x7f3058164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41006/0 (socket says 172.21.15.29:41006) 2023-10-14T18:18:56.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f305ccde640 1 -- 172.21.15.29:0/656753509 learned_addr learned my addr 172.21.15.29:0/656753509 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:56.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f3057fff640 1 --2- 172.21.15.29:0/656753509 >> v2:172.21.15.186:3300/0 conn(0x7f30580c9060 0x7f305806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f304f7fe640 1 -- 172.21.15.29:0/656753509 >> v2:172.21.15.186:3300/0 conn(0x7f30580c9060 msgr2=0x7f305806f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f304f7fe640 1 --2- 172.21.15.29:0/656753509 >> v2:172.21.15.186:3300/0 conn(0x7f30580c9060 0x7f305806f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.581+0000 7f304f7fe640 1 -- 172.21.15.29:0/656753509 >> v1:172.21.15.29:6789/0 conn(0x7f30580c9440 legacy=0x7f3058164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.582+0000 7f304f7fe640 1 -- 172.21.15.29:0/656753509 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30580c9f10 con 0x7f305806c9e0 2023-10-14T18:18:56.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.582+0000 7f30577fe640 1 -- 172.21.15.29:0/656753509 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2297684175 0 0) 0x7f30580ca560 con 0x7f30580c9440 2023-10-14T18:18:56.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.583+0000 7f304f7fe640 1 --2- 172.21.15.29:0/656753509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f305806c9e0 0x7f305806f980 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f305806d490 tx=0x7f304002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=530fa2330286578a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f30577fe640 1 -- 172.21.15.29:0/656753509 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040031830 con 0x7f305806c9e0 2023-10-14T18:18:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f30577fe640 1 -- 172.21.15.29:0/656753509 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040031990 con 0x7f305806c9e0 2023-10-14T18:18:56.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f30577fe640 1 -- 172.21.15.29:0/656753509 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040031d20 con 0x7f305806c9e0 2023-10-14T18:18:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 -- 172.21.15.29:0/656753509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f305806c9e0 msgr2=0x7f305806f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 --2- 172.21.15.29:0/656753509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f305806c9e0 0x7f305806f980 secure :-1 s=READY pgs=2938 cs=0 l=1 rev1=1 crypto rx=0x7f305806d490 tx=0x7f304002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 -- 172.21.15.29:0/656753509 shutdown_connections 2023-10-14T18:18:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 --2- 172.21.15.29:0/656753509 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f305806c9e0 0x7f305806f980 unknown :-1 s=CLOSED pgs=2938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 --2- 172.21.15.29:0/656753509 >> v2:172.21.15.186:3300/0 conn(0x7f30580c9060 0x7f305806f440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 -- 172.21.15.29:0/656753509 >> 172.21.15.29:0/656753509 conn(0x7f30580ce6a0 msgr2=0x7f30580ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.584+0000 7f305d4df640 1 -- 172.21.15.29:0/656753509 shutdown_connections 2023-10-14T18:18:56.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.585+0000 7f305d4df640 1 -- 172.21.15.29:0/656753509 wait complete. 2023-10-14T18:18:56.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.586+0000 7f305d4df640 1 Processor -- start 2023-10-14T18:18:56.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.586+0000 7f305d4df640 1 -- start start 2023-10-14T18:18:56.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f305806c9e0 0x7f3058157490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30580c9440 0x7f3058163790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30581579d0 con 0x7f30580c9060 2023-10-14T18:18:56.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3058157c50 con 0x7f305806c9e0 2023-10-14T18:18:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.587+0000 7f305d4df640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3058157f30 con 0x7f30580c9440 2023-10-14T18:18:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40906/0 (socket says 172.21.15.29:40906) 2023-10-14T18:18:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 -- 172.21.15.29:0/412723057 learned_addr learned my addr 172.21.15.29:0/412723057 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f3057fff640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f305806c9e0 0x7f3058157490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f305ccde640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30580c9440 0x7f3058163790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 -- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30580c9440 msgr2=0x7f3058163790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30580c9440 0x7f3058163790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 -- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f305806c9e0 msgr2=0x7f3058157490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f305806c9e0 0x7f3058157490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.588+0000 7f304f7fe640 1 -- 172.21.15.29:0/412723057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040009400 con 0x7f30580c9060 2023-10-14T18:18:56.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f304f7fe640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f3040009890 tx=0x7f30400046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f30557fa640 1 -- 172.21.15.29:0/412723057 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040004230 con 0x7f30580c9060 2023-10-14T18:18:56.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f30557fa640 1 -- 172.21.15.29:0/412723057 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30400043b0 con 0x7f30580c9060 2023-10-14T18:18:56.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f30557fa640 1 -- 172.21.15.29:0/412723057 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040004520 con 0x7f30580c9060 2023-10-14T18:18:56.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3058158210 con 0x7f30580c9060 2023-10-14T18:18:56.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.589+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581587d0 con 0x7f30580c9060 2023-10-14T18:18:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.592+0000 7f30557fa640 1 -- 172.21.15.29:0/412723057 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f304003a8a0 con 0x7f30580c9060 2023-10-14T18:18:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.593+0000 7f30557fa640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 0x7f303806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.593+0000 7f30557fa640 1 -- 172.21.15.29:0/412723057 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f30400ac590 con 0x7f30580c9060 2023-10-14T18:18:56.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.593+0000 7f3057fff640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 0x7f303806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.593+0000 7f30613af740 1 Processor -- start 2023-10-14T18:18:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.593+0000 7f30613af740 1 -- start start 2023-10-14T18:18:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f3057fff640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 0x7f303806d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30480074a0 tx=0x7f304800b040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 0x5575fbf6c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fbf9bf00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbe8c7a0 con 0x5575fbf6c5b0 2023-10-14T18:18:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbede740 con 0x5575fbf99170 2023-10-14T18:18:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.594+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575fbedbe30 con 0x5575fbf99550 2023-10-14T18:18:56.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40914/0 (socket says 172.21.15.29:40914) 2023-10-14T18:18:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 -- 172.21.15.29:0/2639366389 learned_addr learned my addr 172.21.15.29:0/2639366389 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f3057fff640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 0x5575fbf6c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f304f7fe640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fbf9bf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 -- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 msgr2=0x5575fbf9bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fbf9bf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 -- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 msgr2=0x5575fbf6c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 0x5575fbf6c070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 -- 172.21.15.29:0/2639366389 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575fbe9b910 con 0x5575fbf6c5b0 2023-10-14T18:18:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f3057fff640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 0x5575fbf6c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.595+0000 7f305ccde640 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f3050001600 tx=0x7f3050001ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e247891903d92678 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.596+0000 7f304dffb640 1 -- 172.21.15.29:0/2639366389 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500074e0 con 0x5575fbf6c5b0 2023-10-14T18:18:56.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.596+0000 7f304dffb640 1 -- 172.21.15.29:0/2639366389 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050007640 con 0x5575fbf6c5b0 2023-10-14T18:18:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.596+0000 7f304dffb640 1 -- 172.21.15.29:0/2639366389 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30500049e0 con 0x5575fbf6c5b0 2023-10-14T18:18:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.596+0000 7f30613af740 1 -- 172.21.15.29:0/2639366389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 msgr2=0x5575fbf9c440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.596+0000 7f30613af740 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 secure :-1 s=READY pgs=2940 cs=0 l=1 rev1=1 crypto rx=0x7f3050001600 tx=0x7f3050001ac0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 -- 172.21.15.29:0/2639366389 shutdown_connections 2023-10-14T18:18:56.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf6c5b0 0x5575fbf9c440 unknown :-1 s=CLOSED pgs=2940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fbf9bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 --2- 172.21.15.29:0/2639366389 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99170 0x5575fbf6c070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 -- 172.21.15.29:0/2639366389 >> 172.21.15.29:0/2639366389 conn(0x5575fbf6e380 msgr2=0x5575fbfb6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 -- 172.21.15.29:0/2639366389 shutdown_connections 2023-10-14T18:18:56.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 -- 172.21.15.29:0/2639366389 wait complete. 2023-10-14T18:18:56.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.597+0000 7f30613af740 1 -- start start 2023-10-14T18:18:56.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.598+0000 7f30613af740 1 Processor -- start 2023-10-14T18:18:56.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf9d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 0x5575fbfafca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbed7720 con 0x5575fbf9fb60 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbe75970 con 0x5575fbf99550 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575fbe75610 con 0x5575fbf99170 2023-10-14T18:18:56.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f30613af740 10 client.0 _ll_register_callbacks cb 0x5575fbe979d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:18:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40930/0 (socket says 172.21.15.29:40930) 2023-10-14T18:18:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 -- 172.21.15.29:0/3091147790 learned_addr learned my addr 172.21.15.29:0/3091147790 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f304f7fe640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 0x5575fbfafca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f3057fff640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf9d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 -- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 msgr2=0x5575fbf9d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf9d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 -- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 msgr2=0x5575fbfafca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 0x5575fbfafca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f305ccde640 1 -- 172.21.15.29:0/3091147790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3050001200 con 0x5575fbf9fb60 2023-10-14T18:18:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.599+0000 7f304f7fe640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 0x5575fbfafca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f305ccde640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f30500070a0 tx=0x7f3050007830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0e9dceb4fa9213c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f3036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:18:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3050004da0 con 0x5575fbf9fb60 2023-10-14T18:18:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f305000d730 con 0x5575fbf9fb60 2023-10-14T18:18:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5575fbefe160 con 0x5575fbf9fb60 2023-10-14T18:18:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7f305000d8f0 con 0x5575fbf9fb60 2023-10-14T18:18:56.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f30613af740 10 client.12366 fetch_fsmap learned FSMap version 129 2023-10-14T18:18:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.600+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x5575fbed5da0 con 0x5575fbf9fb60 2023-10-14T18:18:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x5575fbefe160 con 0x5575fbf9fb60 2023-10-14T18:18:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 129) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f305000da50 con 0x5575fbf9fb60 2023-10-14T18:18:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 10 client.12366 fetch_fsmap finished waiting for FSMap version 129 2023-10-14T18:18:56.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 10 client.12366 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T18:18:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 20 client.12366 populate_metadata read hostname 'smithi029' 2023-10-14T18:18:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fbf86fd0 0x5575fbf87e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 4 client.12366 mds_command: new command op to 7637 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:18:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.601+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 --> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5575fbea9cc0 con 0x5575fbf86fd0 2023-10-14T18:18:56.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.602+0000 7f3057fff640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fbf86fd0 0x5575fbf87e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.603+0000 7f3057fff640 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fbf86fd0 0x5575fbf87e40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=860a672b11cf2877 server_cookie=ee53d6aacb195508 in_seq=0 out_seq=0 2023-10-14T18:18:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.603+0000 7f3036ffd640 10 client.12366 ms_handle_connect on v2:172.21.15.186:6834/2447498482 2023-10-14T18:18:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f3036ffd640 1 -- 172.21.15.29:0/3091147790 <== mds.0 v2:172.21.15.186:6834/2447498482 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x5575fbea9cc0 con 0x5575fbf86fd0 2023-10-14T18:18:56.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f3036ffd640 10 client.12366 handle_command_reply: tid=0 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 client.12366 shutdown 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 msgr2=0x5575fbfb01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 secure :-1 s=READY pgs=2941 cs=0 l=1 rev1=1 crypto rx=0x7f30500070a0 tx=0x7f3050007830 comp rx=0 tx=0).stop 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 shutdown_connections 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fbf86fd0 0x5575fbf87e40 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf9fb60 0x5575fbfb01e0 unknown :-1 s=CLOSED pgs=2941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf99550 0x5575fbfafca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 --2- 172.21.15.29:0/3091147790 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf9d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 >> 172.21.15.29:0/3091147790 conn(0x5575fbf6c5b0 msgr2=0x5575fbf9bef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f3036ffd640 0 client.12366 ms_handle_reset on v2:172.21.15.186:6834/2447498482 2023-10-14T18:18:56.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 shutdown_connections 2023-10-14T18:18:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.605+0000 7f30613af740 1 -- 172.21.15.29:0/3091147790 wait complete. 2023-10-14T18:18:56.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.606+0000 7f30613af740 20 client.12366 trim_cache size 0 max 16384 2023-10-14T18:18:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.622+0000 7f30613af740 1 Processor -- start 2023-10-14T18:18:56.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.622+0000 7f30613af740 1 -- start start 2023-10-14T18:18:56.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf732d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb9ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbc3bf20 con 0x5575fbf99550 2023-10-14T18:18:56.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbf97220 con 0x5575fbf9fb60 2023-10-14T18:18:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.623+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575fbf6b460 con 0x5575fbf99170 2023-10-14T18:18:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40940/0 (socket says 172.21.15.29:40940) 2023-10-14T18:18:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 -- 172.21.15.29:0/1178107264 learned_addr learned my addr 172.21.15.29:0/1178107264 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 -- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 msgr2=0x5575fbf732d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf732d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 -- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 msgr2=0x5575fbfb9ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb9ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 -- 172.21.15.29:0/1178107264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5575fbe74b80 con 0x5575fbf99550 2023-10-14T18:18:56.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304f7fe640 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f3040009860 tx=0x7f304007b110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a9a7e313f3d3258 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.624+0000 7f304e7fc640 1 -- 172.21.15.29:0/1178107264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040082d10 con 0x5575fbf99550 2023-10-14T18:18:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f304e7fc640 1 -- 172.21.15.29:0/1178107264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040078db0 con 0x5575fbf99550 2023-10-14T18:18:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f304e7fc640 1 -- 172.21.15.29:0/1178107264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3040079f00 con 0x5575fbf99550 2023-10-14T18:18:56.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 -- 172.21.15.29:0/1178107264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 msgr2=0x5575fbf713c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 secure :-1 s=READY pgs=2942 cs=0 l=1 rev1=1 crypto rx=0x7f3040009860 tx=0x7f304007b110 comp rx=0 tx=0).stop 2023-10-14T18:18:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 -- 172.21.15.29:0/1178107264 shutdown_connections 2023-10-14T18:18:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb9ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99550 0x5575fbf713c0 unknown :-1 s=CLOSED pgs=2942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 --2- 172.21.15.29:0/1178107264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99170 0x5575fbf732d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 -- 172.21.15.29:0/1178107264 >> 172.21.15.29:0/1178107264 conn(0x5575fbf6c5b0 msgr2=0x5575fbfb9200 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 -- 172.21.15.29:0/1178107264 shutdown_connections 2023-10-14T18:18:56.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.625+0000 7f30613af740 1 -- 172.21.15.29:0/1178107264 wait complete. 2023-10-14T18:18:56.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.626+0000 7f30613af740 1 -- start start 2023-10-14T18:18:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.626+0000 7f30613af740 1 Processor -- start 2023-10-14T18:18:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fc015e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb7ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbfb9ea0 con 0x5575fbf99170 2023-10-14T18:18:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x5575fbf95c40 con 0x5575fbf9fb60 2023-10-14T18:18:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x5575fc016810 con 0x5575fbf99550 2023-10-14T18:18:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.627+0000 7f30613af740 10 client.0 _ll_register_callbacks cb 0x7f3050001420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:18:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40952/0 (socket says 172.21.15.29:40952) 2023-10-14T18:18:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 -- 172.21.15.29:0/2708829730 learned_addr learned my addr 172.21.15.29:0/2708829730 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f305ccde640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb7ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 -- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 msgr2=0x5575fc015e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f304f7fe640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fc015e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fc015e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 -- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 msgr2=0x5575fbfb7ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb7ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.628+0000 7f3057fff640 1 -- 172.21.15.29:0/2708829730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30581587d0 con 0x5575fbf99170 2023-10-14T18:18:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f3057fff640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f3048009fd0 tx=0x7f3048009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b26ea97934b8ac31 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f3036ffd640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:18:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f304801d040 con 0x5575fbf99170 2023-10-14T18:18:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3050007640 con 0x5575fbf99170 2023-10-14T18:18:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f3048025cc0 con 0x5575fbf99170 2023-10-14T18:18:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=129) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3048025070 con 0x5575fbf99170 2023-10-14T18:18:56.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f30613af740 10 client.12372 fetch_fsmap learned FSMap version 129 2023-10-14T18:18:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.629+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=129,monmap=2+}) v3 -- 0x5575fbe9a170 con 0x5575fbf99170 2023-10-14T18:18:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3050007640 con 0x5575fbf99170 2023-10-14T18:18:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 129) v1 ==== 3891+0+0 (secure 0 0 0) 0x7f3048014070 con 0x5575fbf99170 2023-10-14T18:18:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 10 client.12372 fetch_fsmap finished waiting for FSMap version 129 2023-10-14T18:18:56.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 10 client.12372 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T18:18:56.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 20 client.12372 populate_metadata read hostname 'smithi029' 2023-10-14T18:18:56.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fc012400 0x5575fbfb8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 4 client.12372 mds_command: new command op to 7637 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:18:56.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 --> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5575fbed0700 con 0x5575fc012400 2023-10-14T18:18:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.630+0000 7f304f7fe640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fc012400 0x5575fbfb8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.631+0000 7f304f7fe640 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fc012400 0x5575fbfb8410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6505f585405db750 server_cookie=b77bd33466e1b413 in_seq=0 out_seq=0 2023-10-14T18:18:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.631+0000 7f3036ffd640 10 client.12372 ms_handle_connect on v2:172.21.15.186:6834/2447498482 2023-10-14T18:18:56.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f3036ffd640 1 -- 172.21.15.29:0/2708829730 <== mds.0 v2:172.21.15.186:6834/2447498482 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+337 (crc 0 0 0) 0x5575fbed0700 con 0x5575fc012400 2023-10-14T18:18:56.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f3036ffd640 10 client.12372 handle_command_reply: tid=0 2023-10-14T18:18:56.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 client.12372 shutdown 2023-10-14T18:18:56.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 msgr2=0x5575fbf96060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 secure :-1 s=READY pgs=2943 cs=0 l=1 rev1=1 crypto rx=0x7f3048009fd0 tx=0x7f3048009750 comp rx=0 tx=0).stop 2023-10-14T18:18:56.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 shutdown_connections 2023-10-14T18:18:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:6834/2447498482,v1:172.21.15.186:6835/2447498482] conn(0x5575fc012400 0x5575fbfb8410 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x5575fbf9fb60 0x5575fbfb7ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x5575fbf99550 0x5575fc015e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 --2- 172.21.15.29:0/2708829730 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x5575fbf99170 0x5575fbf96060 unknown :-1 s=CLOSED pgs=2943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 >> 172.21.15.29:0/2708829730 conn(0x5575fbf6c5b0 msgr2=0x5575fbfd0870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f3036ffd640 0 client.12372 ms_handle_reset on v2:172.21.15.186:6834/2447498482 2023-10-14T18:18:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 shutdown_connections 2023-10-14T18:18:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.632+0000 7f30613af740 1 -- 172.21.15.29:0/2708829730 wait complete. 2023-10-14T18:18:56.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f30613af740 20 client.12372 trim_cache size 0 max 16384 2023-10-14T18:18:56.646 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:18:56.646 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:18:56.646 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:18:56.646 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 7637, 2023-10-14T18:18:56.647 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:18:56.647 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:18:56.647 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "backup_fs", 2023-10-14T18:18:56.647 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.3467437750000002, 2023-10-14T18:18:56.647 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 129, 2023-10-14T18:18:56.648 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 254, 2023-10-14T18:18:56.648 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 254, 2023-10-14T18:18:56.648 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 27.324599854999999 2023-10-14T18:18:56.648 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:18:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 msgr2=0x7f303806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 0x7f303806d570 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f30480074a0 tx=0x7f304800b040 comp rx=0 tx=0).stop 2023-10-14T18:18:56.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 msgr2=0x7f30581f6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 secure :-1 s=READY pgs=2939 cs=0 l=1 rev1=1 crypto rx=0x7f3040009890 tx=0x7f30400046c0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 shutdown_connections 2023-10-14T18:18:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f303806b0b0 0x7f303806d570 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30580c9440 0x7f3058163790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f30580c9060 0x7f30581f6000 unknown :-1 s=CLOSED pgs=2939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 --2- 172.21.15.29:0/412723057 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f305806c9e0 0x7f3058157490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 >> 172.21.15.29:0/412723057 conn(0x7f30580ce6a0 msgr2=0x7f30580cdad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 shutdown_connections 2023-10-14T18:18:56.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.633+0000 7f305d4df640 1 -- 172.21.15.29:0/412723057 wait complete. 2023-10-14T18:18:56.653 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T18:18:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.867+0000 7f48ad93c640 1 Processor -- start 2023-10-14T18:18:56.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.867+0000 7f48ad93c640 1 -- start start 2023-10-14T18:18:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 0x7f48a806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a80ad970 0x7f48a815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48a815ebc0 0x7f48a815f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a806c960 con 0x7f48a815ebc0 2023-10-14T18:18:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a815b9d0 con 0x7f48a806c040 2023-10-14T18:18:56.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad93c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a815bcb0 con 0x7f48a80ad970 2023-10-14T18:18:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.869+0000 7f48ad13b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48a815ebc0 0x7f48a815f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f48a815ebc0 0x7f48a815f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40966/0 (socket says 172.21.15.29:40966) 2023-10-14T18:18:56.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 -- 172.21.15.29:0/4148955004 learned_addr learned my addr 172.21.15.29:0/4148955004 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48a7fff640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a80ad970 0x7f48a815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ac93a640 1 --2- 172.21.15.29:0/4148955004 >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 0x7f48a806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 -- 172.21.15.29:0/4148955004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a80ad970 msgr2=0x7f48a815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a80ad970 0x7f48a815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 -- 172.21.15.29:0/4148955004 >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 msgr2=0x7f48a806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:56.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 --2- 172.21.15.29:0/4148955004 >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 0x7f48a806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ad13b640 1 -- 172.21.15.29:0/4148955004 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a815bf90 con 0x7f48a815ebc0 2023-10-14T18:18:56.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.870+0000 7f48ac93a640 1 --2- 172.21.15.29:0/4148955004 >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 0x7f48a806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:56.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.871+0000 7f48ad13b640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a815ebc0 0x7f48a815f050 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009e50 tx=0x7f48a002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b9581c6baa21f89 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48a77fe640 1 -- 172.21.15.29:0/4148955004 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a002ca70 con 0x7f48a815ebc0 2023-10-14T18:18:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48a77fe640 1 -- 172.21.15.29:0/4148955004 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a002cbd0 con 0x7f48a815ebc0 2023-10-14T18:18:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48a77fe640 1 -- 172.21.15.29:0/4148955004 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48a00343f0 con 0x7f48a815ebc0 2023-10-14T18:18:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 -- 172.21.15.29:0/4148955004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a815ebc0 msgr2=0x7f48a815f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a815ebc0 0x7f48a815f050 secure :-1 s=READY pgs=2944 cs=0 l=1 rev1=1 crypto rx=0x7f48a0009e50 tx=0x7f48a002bed0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.876 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 -- 172.21.15.29:0/4148955004 shutdown_connections 2023-10-14T18:18:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a815ebc0 0x7f48a815f050 unknown :-1 s=CLOSED pgs=2944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 --2- 172.21.15.29:0/4148955004 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a80ad970 0x7f48a815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 --2- 172.21.15.29:0/4148955004 >> v2:172.21.15.186:3300/0 conn(0x7f48a806c040 0x7f48a806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 -- 172.21.15.29:0/4148955004 >> 172.21.15.29:0/4148955004 conn(0x7f48a80ce6a0 msgr2=0x7f48a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:56.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 -- 172.21.15.29:0/4148955004 shutdown_connections 2023-10-14T18:18:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.872+0000 7f48ad93c640 1 -- 172.21.15.29:0/4148955004 wait complete. 2023-10-14T18:18:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.874+0000 7f48ad93c640 1 Processor -- start 2023-10-14T18:18:56.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.874+0000 7f48ad93c640 1 -- start start 2023-10-14T18:18:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 0x7f48a81f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a815ebc0 0x7f48a81f6880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a81f2eb0 con 0x7f48a80ad970 2023-10-14T18:18:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48a81f3190 con 0x7f48a806c040 2023-10-14T18:18:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.875+0000 7f48ad93c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a81f3470 con 0x7f48a815ebc0 2023-10-14T18:18:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.881 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40982/0 (socket says 172.21.15.29:40982) 2023-10-14T18:18:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 -- 172.21.15.29:0/2371250813 learned_addr learned my addr 172.21.15.29:0/2371250813 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48ac93a640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 0x7f48a81f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48ad13b640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a815ebc0 0x7f48a81f6880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 -- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a815ebc0 msgr2=0x7f48a81f6880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a815ebc0 0x7f48a81f6880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 -- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 msgr2=0x7f48a81f28a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 0x7f48a81f28a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48a7fff640 1 -- 172.21.15.29:0/2371250813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a0009b30 con 0x7f48a80ad970 2023-10-14T18:18:56.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.876+0000 7f48ac93a640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 0x7f48a81f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:18:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48a7fff640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f489400e870 tx=0x7f489400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48940094f0 con 0x7f48a80ad970 2023-10-14T18:18:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f489400c940 con 0x7f48a80ad970 2023-10-14T18:18:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f489400ccf0 con 0x7f48a80ad970 2023-10-14T18:18:56.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a80af190 con 0x7f48a80ad970 2023-10-14T18:18:56.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.877+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a81efe70 con 0x7f48a80ad970 2023-10-14T18:18:56.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.878+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a806b330 con 0x7f48a80ad970 2023-10-14T18:18:56.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.879+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4894013470 con 0x7f48a80ad970 2023-10-14T18:18:56.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.880+0000 7f48a57fa640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 0x7f488c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:56.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.881+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f4894089f50 con 0x7f48a80ad970 2023-10-14T18:18:56.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.881+0000 7f48ac93a640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 0x7f488c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:56.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.881+0000 7f48ac93a640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 0x7f488c06d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f48a806d1c0 tx=0x7f489c008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:56.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:56.885+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f48940548b0 con 0x7f48a80ad970 2023-10-14T18:18:57.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.104+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f48a80af010 con 0x7f488c06b0b0 2023-10-14T18:18:57.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.106+0000 7f48a57fa640 1 -- 172.21.15.29:0/2371250813 <== mgr.7463 v2:172.21.15.29:6832/4275371095 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+82 (secure 0 0 0) 0x7f489c00f860 con 0x7f488c06b0b0 2023-10-14T18:18:57.107 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:18:57.107 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:57.107 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T18:18:57.107 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:18:57.108 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:18:57.108 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "backup_fs" 2023-10-14T18:18:57.108 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:18:57.108 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:18:57.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 msgr2=0x7f488c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 0x7f488c06d570 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f48a806d1c0 tx=0x7f489c008040 comp rx=0 tx=0).stop 2023-10-14T18:18:57.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 msgr2=0x7f48a81f3750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 secure :-1 s=READY pgs=2945 cs=0 l=1 rev1=1 crypto rx=0x7f489400e870 tx=0x7f489400ed30 comp rx=0 tx=0).stop 2023-10-14T18:18:57.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 shutdown_connections 2023-10-14T18:18:57.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f488c06b0b0 0x7f488c06d570 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f48a815ebc0 0x7f48a81f6880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f48a80ad970 0x7f48a81f3750 unknown :-1 s=CLOSED pgs=2945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 --2- 172.21.15.29:0/2371250813 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f48a806c040 0x7f48a81f28a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 >> 172.21.15.29:0/2371250813 conn(0x7f48a80ce6a0 msgr2=0x7f48a80cc970 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:57.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 shutdown_connections 2023-10-14T18:18:57.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.108+0000 7f48ad93c640 1 -- 172.21.15.29:0/2371250813 wait complete. 2023-10-14T18:18:57.127 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T18:18:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.345+0000 7f4fe882c640 1 Processor -- start 2023-10-14T18:18:57.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.345+0000 7f4fe882c640 1 -- start start 2023-10-14T18:18:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe882c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4fe4072170 0x7f4fe4167b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe882c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40d12f0 con 0x7f4fe40cdf30 2023-10-14T18:18:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe882c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe40d06e0 con 0x7f4fe415cc80 2023-10-14T18:18:57.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe882c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe40d0500 con 0x7f4fe4072170 2023-10-14T18:18:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4fe4072170 0x7f4fe4167b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4fe4072170 0x7f4fe4167b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40990/0 (socket says 172.21.15.29:40990) 2023-10-14T18:18:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe359e640 1 -- 172.21.15.29:0/2279349728 learned_addr learned my addr 172.21.15.29:0/2279349728 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:57.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.347+0000 7f4fe259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4fe40cdf30 0x7f4fe4071a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41008/0 (socket says 172.21.15.29:41008) 2023-10-14T18:18:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.348+0000 7f4fe359e640 1 -- 172.21.15.29:0/2279349728 >> v1:172.21.15.186:6789/0 conn(0x7f4fe415cc80 legacy=0x7f4fe40cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.348+0000 7f4fe359e640 1 -- 172.21.15.29:0/2279349728 >> v1:172.21.15.29:6789/0 conn(0x7f4fe40cdf30 legacy=0x7f4fe4071a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.348+0000 7f4fe359e640 1 -- 172.21.15.29:0/2279349728 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe40d09c0 con 0x7f4fe4072170 2023-10-14T18:18:57.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.348+0000 7f4fe1d9b640 1 -- 172.21.15.29:0/2279349728 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849566119 0 0) 0x7f4fe40d12f0 con 0x7f4fe40cdf30 2023-10-14T18:18:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.349+0000 7f4fe359e640 1 --2- 172.21.15.29:0/2279349728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe4072170 0x7f4fe4167b10 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009e50 tx=0x7f4fdc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f661c207be8ffd2d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.349+0000 7f4fe1d9b640 1 -- 172.21.15.29:0/2279349728 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc02cb20 con 0x7f4fe4072170 2023-10-14T18:18:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.349+0000 7f4fe1d9b640 1 -- 172.21.15.29:0/2279349728 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc02cc80 con 0x7f4fe4072170 2023-10-14T18:18:57.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.349+0000 7f4fe1d9b640 1 -- 172.21.15.29:0/2279349728 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc0343f0 con 0x7f4fe4072170 2023-10-14T18:18:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 -- 172.21.15.29:0/2279349728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe4072170 msgr2=0x7f4fe4167b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 --2- 172.21.15.29:0/2279349728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe4072170 0x7f4fe4167b10 secure :-1 s=READY pgs=2947 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc009e50 tx=0x7f4fdc02bf80 comp rx=0 tx=0).stop 2023-10-14T18:18:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 -- 172.21.15.29:0/2279349728 shutdown_connections 2023-10-14T18:18:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 --2- 172.21.15.29:0/2279349728 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe4072170 0x7f4fe4167b10 unknown :-1 s=CLOSED pgs=2947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 -- 172.21.15.29:0/2279349728 >> 172.21.15.29:0/2279349728 conn(0x7f4fe4065ed0 msgr2=0x7f4fe40662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 -- 172.21.15.29:0/2279349728 shutdown_connections 2023-10-14T18:18:57.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.350+0000 7f4fe882c640 1 -- 172.21.15.29:0/2279349728 wait complete. 2023-10-14T18:18:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.351+0000 7f4fe882c640 1 Processor -- start 2023-10-14T18:18:57.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.352+0000 7f4fe882c640 1 -- start start 2023-10-14T18:18:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fe4072170 0x7f4fe40c4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 0x7f4fe40c6970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40c4c40 con 0x7f4fe415cc80 2023-10-14T18:18:57.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40c4f20 con 0x7f4fe4072170 2023-10-14T18:18:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe882c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe40c5200 con 0x7f4fe40cdf30 2023-10-14T18:18:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:40992/0 (socket says 172.21.15.29:40992) 2023-10-14T18:18:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 -- 172.21.15.29:0/3913358868 learned_addr learned my addr 172.21.15.29:0/3913358868 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:57.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe2d9d640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fe4072170 0x7f4fe40c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe259c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 0x7f4fe40c6970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 -- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 msgr2=0x7f4fe40c6970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 0x7f4fe40c6970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 -- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fe4072170 msgr2=0x7f4fe40c4410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fe4072170 0x7f4fe40c4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:57.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe359e640 1 -- 172.21.15.29:0/3913358868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc009b30 con 0x7f4fe415cc80 2023-10-14T18:18:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.353+0000 7f4fe259c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 0x7f4fe40c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fe359e640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc036040 tx=0x7f4fdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc009510 con 0x7f4fe415cc80 2023-10-14T18:18:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc007e10 con 0x7f4fe415cc80 2023-10-14T18:18:57.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc03e670 con 0x7f4fe415cc80 2023-10-14T18:18:57.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe40c54e0 con 0x7f4fe415cc80 2023-10-14T18:18:57.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.354+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe40c4950 con 0x7f4fe415cc80 2023-10-14T18:18:57.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.355+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe40c5a00 con 0x7f4fe415cc80 2023-10-14T18:18:57.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.356+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 147) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4fdc03e7d0 con 0x7f4fe415cc80 2023-10-14T18:18:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.357+0000 7f4fcb7fe640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 0x7f4fd006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.357+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f4fdc0acae0 con 0x7f4fe415cc80 2023-10-14T18:18:57.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.358+0000 7f4fe2d9d640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 0x7f4fd006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:57.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.358+0000 7f4fe2d9d640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 0x7f4fd006d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400a750 tx=0x7f4fd4008040 comp rx=0 tx=0).ready entity=mgr.7463 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:57.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.364+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4fdc077440 con 0x7f4fe415cc80 2023-10-14T18:18:57.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:57.614+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4fe404e9b0 con 0x7f4fe415cc80 2023-10-14T18:18:58.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.379+0000 7f4fcb7fe640 1 -- 172.21.15.29:0/3913358868 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v148) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4fdc07aa60 con 0x7f4fe415cc80 2023-10-14T18:18:58.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.384+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 msgr2=0x7f4fd006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.384+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 0x7f4fd006d570 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4fd400a750 tx=0x7f4fd4008040 comp rx=0 tx=0).stop 2023-10-14T18:18:58.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.384+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 msgr2=0x7f4fe41f1770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.384+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 secure :-1 s=READY pgs=2948 cs=0 l=1 rev1=1 crypto rx=0x7f4fdc036040 tx=0x7f4fdc004a40 comp rx=0 tx=0).stop 2023-10-14T18:18:58.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 shutdown_connections 2023-10-14T18:18:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f4fd006b0b0 0x7f4fd006d570 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4fe415cc80 0x7f4fe41f1770 unknown :-1 s=CLOSED pgs=2948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4fe40cdf30 0x7f4fe40c6970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 --2- 172.21.15.29:0/3913358868 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4fe4072170 0x7f4fe40c4410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.387 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.385+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 >> 172.21.15.29:0/3913358868 conn(0x7f4fe4065ed0 msgr2=0x7f4fe406b740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:58.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.386+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 shutdown_connections 2023-10-14T18:18:58.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.386+0000 7f4fe882c640 1 -- 172.21.15.29:0/3913358868 wait complete. 2023-10-14T18:18:58.409 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T18:18:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.605+0000 7fb56d3a9640 1 Processor -- start 2023-10-14T18:18:58.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.605+0000 7fb56d3a9640 1 -- start start 2023-10-14T18:18:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.606+0000 7fb56d3a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb56815ad70 0x7fb568069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.606+0000 7fb56d3a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5680c9ca0 0x7fb56806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.606+0000 7fb56d3a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5680cb030 con 0x7fb5680ca3b0 2023-10-14T18:18:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.606+0000 7fb56d3a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5680c91c0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.606+0000 7fb56d3a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680c94a0 con 0x7fb56815ad70 2023-10-14T18:18:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5680c9ca0 0x7fb56806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb5680c9ca0 0x7fb56806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41010/0 (socket says 172.21.15.29:41010) 2023-10-14T18:18:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 -- 172.21.15.29:0/1063254369 learned_addr learned my addr 172.21.15.29:0/1063254369 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb566ffd640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb56815ad70 0x7fb568069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:58.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 -- 172.21.15.29:0/1063254369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb56815ad70 msgr2=0x7fb568069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:58.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb56815ad70 0x7fb568069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 -- 172.21.15.29:0/1063254369 >> v1:172.21.15.186:6789/0 conn(0x7fb5680ca3b0 legacy=0x7fb5680ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.607+0000 7fb5667fc640 1 -- 172.21.15.29:0/1063254369 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5680c9780 con 0x7fb5680c9ca0 2023-10-14T18:18:58.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.609+0000 7fb5667fc640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb56806a2e0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009870 tx=0x7fb55c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d1e8e748dbb6c58 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.609+0000 7fb565ffb640 1 -- 172.21.15.29:0/1063254369 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c030840 con 0x7fb5680c9ca0 2023-10-14T18:18:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.609+0000 7fb565ffb640 1 -- 172.21.15.29:0/1063254369 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c0309a0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.609+0000 7fb565ffb640 1 -- 172.21.15.29:0/1063254369 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c030d30 con 0x7fb5680c9ca0 2023-10-14T18:18:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1063254369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 msgr2=0x7fb56806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb56806a2e0 secure :-1 s=READY pgs=2952 cs=0 l=1 rev1=1 crypto rx=0x7fb55c009870 tx=0x7fb55c02b980 comp rx=0 tx=0).stop 2023-10-14T18:18:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1063254369 shutdown_connections 2023-10-14T18:18:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb56806a2e0 unknown :-1 s=CLOSED pgs=2952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1063254369 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb56815ad70 0x7fb568069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1063254369 >> 172.21.15.29:0/1063254369 conn(0x7fb5680ce9b0 msgr2=0x7fb5680cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:58.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1063254369 shutdown_connections 2023-10-14T18:18:58.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.610+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1063254369 wait complete. 2023-10-14T18:18:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.612+0000 7fb56d3a9640 1 Processor -- start 2023-10-14T18:18:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.612+0000 7fb56d3a9640 1 -- start start 2023-10-14T18:18:58.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5680ca3b0 0x7fb5680c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 0x7fb568156f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680b34b0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680adeb0 con 0x7fb56815ad70 2023-10-14T18:18:58.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb56d3a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680ad910 con 0x7fb5680ca3b0 2023-10-14T18:18:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb566ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb566ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41012/0 (socket says 172.21.15.29:41012) 2023-10-14T18:18:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.613+0000 7fb566ffd640 1 -- 172.21.15.29:0/1041631019 learned_addr learned my addr 172.21.15.29:0/1041631019 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb5667fc640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5680ca3b0 0x7fb5680c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:58.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5680ca3b0 msgr2=0x7fb5680c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb5677fe640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 0x7fb568156f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5680ca3b0 0x7fb5680c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 msgr2=0x7fb568156f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 0x7fb568156f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 -- 172.21.15.29:0/1041631019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb55c0094d0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb5677fe640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 0x7fb568156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:18:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb566ffd640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7fb55800d5f0 tx=0x7fb55800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5580125f0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558012750 con 0x7fb5680c9ca0 2023-10-14T18:18:58.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb558012b00 con 0x7fb5680c9ca0 2023-10-14T18:18:58.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.614+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5680adbd0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.615+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb568157480 con 0x7fb5680c9ca0 2023-10-14T18:18:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.616+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56804e160 con 0x7fb5680c9ca0 2023-10-14T18:18:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.617+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fb558012c60 con 0x7fb5680c9ca0 2023-10-14T18:18:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.618+0000 7fb54f7fe640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 0x7fb55406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.618+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7fb55808a330 con 0x7fb5680c9ca0 2023-10-14T18:18:58.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.618+0000 7fb5667fc640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 msgr2=0x7fb55406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.618+0000 7fb5667fc640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 0x7fb55406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:58.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.622+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb558054c90 con 0x7fb5680c9ca0 2023-10-14T18:18:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.819+0000 7fb5667fc640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 msgr2=0x7fb55406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:58.820 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.819+0000 7fb5667fc640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 0x7fb55406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:58.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.821+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7fb5680ca840 con 0x7fb5680c9ca0 2023-10-14T18:18:58.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.826+0000 7fb54f7fe640 1 -- 172.21.15.29:0/1041631019 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v83)=0 v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fb5580582b0 con 0x7fb5680c9ca0 2023-10-14T18:18:58.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 msgr2=0x7fb55406d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 0x7fb55406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 msgr2=0x7fb5680c8b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:58.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 secure :-1 s=READY pgs=2953 cs=0 l=1 rev1=1 crypto rx=0x7fb55800d5f0 tx=0x7fb55800dab0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 shutdown_connections 2023-10-14T18:18:58.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7fb55406b100 0x7fb55406d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb56815ad70 0x7fb568156f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb5680ca3b0 0x7fb5680c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 --2- 172.21.15.29:0/1041631019 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb5680c9ca0 0x7fb5680c8b90 unknown :-1 s=CLOSED pgs=2953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:58.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.829+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 >> 172.21.15.29:0/1041631019 conn(0x7fb5680ce9b0 msgr2=0x7fb56806a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.830+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 shutdown_connections 2023-10-14T18:18:58.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:58.830+0000 7fb56d3a9640 1 -- 172.21.15.29:0/1041631019 wait complete. 2023-10-14T18:18:58.849 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T18:18:58.907 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:58.907+0000 7f3513a33040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:18:58.927 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:58.926+0000 7f9c8e21a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:18:58.962 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:58.962+0000 7f3513a33040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:18:58.985 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:58.984+0000 7f9c8e21a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:18:59.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.059+0000 7f5f33f68640 1 Processor -- start 2023-10-14T18:18:59.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.060+0000 7f5f33f68640 1 -- start start 2023-10-14T18:18:59.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f33f68640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 0x7f5f2c06d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f33f68640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c161740 con 0x7f5f2c070c10 2023-10-14T18:18:59.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f33f68640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c06b0f0 con 0x7f5f2c0cf5f0 2023-10-14T18:18:59.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f33f68640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c1619b0 con 0x7f5f2c070830 2023-10-14T18:18:59.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f32505640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5f2c070c10 0x7f5f2c0cd690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:41020/0 (socket says 172.21.15.29:41020) 2023-10-14T18:18:59.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f32505640 1 -- 172.21.15.29:0/307367448 learned_addr learned my addr 172.21.15.29:0/307367448 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:18:59.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.061+0000 7f5f31d04640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 0x7f5f2c06d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933214432 0 0) 0x7f5f2c161740 con 0x7f5f2c070c10 2023-10-14T18:18:59.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f18003520 con 0x7f5f2c070c10 2023-10-14T18:18:59.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2408809672 0 0) 0x7f5f18003520 con 0x7f5f2c070c10 2023-10-14T18:18:59.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 msgr2=0x7f5f2c06d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:59.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 0x7f5f2c06d3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 >> v1:172.21.15.186:6789/0 conn(0x7f5f2c0cf5f0 legacy=0x7f5f2c06c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c06b3d0 con 0x7f5f2c070c10 2023-10-14T18:18:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5f240019f0 con 0x7f5f2c070c10 2023-10-14T18:18:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 >> v1:172.21.15.29:6789/0 conn(0x7f5f2c070c10 legacy=0x7f5f2c0cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180039a0 0x7f5f18003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f18004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 0x7f5f1800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c1619b0 con 0x7f5f180047c0 2023-10-14T18:18:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f180054e0 con 0x7f5f180039a0 2023-10-14T18:18:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.062+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f1800bec0 con 0x7f5f2c070830 2023-10-14T18:18:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f18004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 -- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 msgr2=0x7f5f1800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c070830 0x7f5f1800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 -- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180039a0 msgr2=0x7f5f18003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31d04640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180039a0 0x7f5f18003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f180039a0 0x7f5f18003dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.063+0000 7f5f31503640 1 -- 172.21.15.29:0/307367448 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f18003520 con 0x7f5f180047c0 2023-10-14T18:18:59.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.064+0000 7f5f31503640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f18004c40 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c06c850 tx=0x7f5f180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9edd17688ee5edda server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:59.072 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.064+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0052d0 con 0x7f5f180047c0 2023-10-14T18:18:59.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.064+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c02e050 con 0x7f5f180047c0 2023-10-14T18:18:59.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.064+0000 7f5f30d02640 1 -- 172.21.15.29:0/307367448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f1c0035f0 con 0x7f5f180047c0 2023-10-14T18:18:59.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 -- 172.21.15.29:0/307367448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 msgr2=0x7f5f18004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 --2- 172.21.15.29:0/307367448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f18004c40 secure :-1 s=READY pgs=2955 cs=0 l=1 rev1=1 crypto rx=0x7f5f2c06c850 tx=0x7f5f180057f0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f32505640 1 -- 172.21.15.29:0/307367448 reap_dead start 2023-10-14T18:18:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 -- 172.21.15.29:0/307367448 shutdown_connections 2023-10-14T18:18:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 -- 172.21.15.29:0/307367448 >> 172.21.15.29:0/307367448 conn(0x7f5f2c0cfac0 msgr2=0x7f5f2c0cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 -- 172.21.15.29:0/307367448 shutdown_connections 2023-10-14T18:18:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.065+0000 7f5f33f68640 1 -- 172.21.15.29:0/307367448 wait complete. 2023-10-14T18:18:59.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.068+0000 7f5f33f68640 1 Processor -- start 2023-10-14T18:18:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.068+0000 7f5f33f68640 1 -- start start 2023-10-14T18:18:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c0cb8e0 0x7f5f2c1f22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2c0b4b30 0x7f5f2c0b4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c0b46a0 con 0x7f5f180047c0 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c159b70 con 0x7f5f2c0b4b30 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f33f68640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c0b3a90 con 0x7f5f2c0cb8e0 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41030/0 (socket says 172.21.15.29:41030) 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 -- 172.21.15.29:0/4294603540 learned_addr learned my addr 172.21.15.29:0/4294603540 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:59.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c0cb8e0 msgr2=0x7f5f2c1f22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c0cb8e0 0x7f5f2c1f22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2c0b4b30 msgr2=0x7f5f2c0b4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2c0b4b30 0x7f5f2c0b4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.070+0000 7f5f31d04640 1 -- 172.21.15.29:0/4294603540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0b3d70 con 0x7f5f180047c0 2023-10-14T18:18:59.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f31d04640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f5f28010860 tx=0x7f5f28010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f28009610 con 0x7f5f180047c0 2023-10-14T18:18:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f2800eb60 con 0x7f5f180047c0 2023-10-14T18:18:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f2801c400 con 0x7f5f180047c0 2023-10-14T18:18:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c0b4050 con 0x7f5f180047c0 2023-10-14T18:18:59.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.071+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c1f2810 con 0x7f5f180047c0 2023-10-14T18:18:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.072+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c04e160 con 0x7f5f180047c0 2023-10-14T18:18:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.073+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f5f2801c560 con 0x7f5f180047c0 2023-10-14T18:18:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.074+0000 7f5f227fc640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 0x7f5f1006d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.074+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f5f2808a470 con 0x7f5f180047c0 2023-10-14T18:18:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.074+0000 7f5f31503640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 msgr2=0x7f5f1006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.074+0000 7f5f31503640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 0x7f5f1006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:59.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.079+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5f28054df0 con 0x7f5f180047c0 2023-10-14T18:18:59.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.271+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5f2c0b5a40 con 0x7f5f180047c0 2023-10-14T18:18:59.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.275+0000 7f5f31503640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 msgr2=0x7f5f1006d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:59.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.275+0000 7f5f31503640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 0x7f5f1006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:59.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.277+0000 7f5f227fc640 1 -- 172.21.15.29:0/4294603540 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v84)=0 v84) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5f28058410 con 0x7f5f180047c0 2023-10-14T18:18:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 msgr2=0x7f5f1006d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:18:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 0x7f5f1006d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 msgr2=0x7f5f2c159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 secure :-1 s=READY pgs=2956 cs=0 l=1 rev1=1 crypto rx=0x7f5f28010860 tx=0x7f5f28010d20 comp rx=0 tx=0).stop 2023-10-14T18:18:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 shutdown_connections 2023-10-14T18:18:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f5f1006b130 0x7f5f1006d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5f2c0b4b30 0x7f5f2c0b4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5f2c0cb8e0 0x7f5f2c1f22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 --2- 172.21.15.29:0/4294603540 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5f180047c0 0x7f5f2c159630 unknown :-1 s=CLOSED pgs=2956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 >> 172.21.15.29:0/4294603540 conn(0x7f5f2c0cfac0 msgr2=0x7f5f2c0acb30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 shutdown_connections 2023-10-14T18:18:59.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.280+0000 7f5f33f68640 1 -- 172.21.15.29:0/4294603540 wait complete. 2023-10-14T18:18:59.299 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:18:59.299 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p mandatory_fs_argument_test_dir) 2023-10-14T18:18:59.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add mandatory_fs_argument_test_dir 1M 2023-10-14T18:18:59.423 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.423+0000 7f3513a33040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:18:59.467 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.467+0000 7f9c8e21a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:18:59.477 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.476+0000 7f3513a33040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:18:59.523 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.522+0000 7f9c8e21a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:18:59.528 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.528+0000 7f3513a33040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:18:59.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.560+0000 7f4116083640 1 Processor -- start 2023-10-14T18:18:59.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.560+0000 7f4116083640 1 -- start start 2023-10-14T18:18:59.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.561+0000 7f4116083640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4110070830 0x7f411006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f4116083640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41100cf5f0 0x7f411006c140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f4116083640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4110161740 con 0x7f4110070c10 2023-10-14T18:18:59.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f4116083640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f411006c680 con 0x7f41100cf5f0 2023-10-14T18:18:59.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f4116083640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411006b0f0 con 0x7f4110070830 2023-10-14T18:18:59.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41100cf5f0 0x7f411006c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f41100cf5f0 0x7f411006c140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41040/0 (socket says 172.21.15.29:41040) 2023-10-14T18:18:59.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 -- 172.21.15.29:0/2616968923 learned_addr learned my addr 172.21.15.29:0/2616968923 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:59.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410f7fe640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4110070830 0x7f411006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 -- 172.21.15.29:0/2616968923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4110070830 msgr2=0x7f411006d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:59.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4110070830 0x7f411006d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 -- 172.21.15.29:0/2616968923 >> v1:172.21.15.186:6789/0 conn(0x7f4110070c10 legacy=0x7f41100cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:18:59.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.562+0000 7f410effd640 1 -- 172.21.15.29:0/2616968923 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41101619b0 con 0x7f41100cf5f0 2023-10-14T18:18:59.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.563+0000 7f410effd640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100cf5f0 0x7f411006c140 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f4104009870 tx=0x7f410402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66db6f19054999fa server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:59.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.564+0000 7f410e7fc640 1 -- 172.21.15.29:0/2616968923 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104031740 con 0x7f41100cf5f0 2023-10-14T18:18:59.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.564+0000 7f410e7fc640 1 -- 172.21.15.29:0/2616968923 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41040318a0 con 0x7f41100cf5f0 2023-10-14T18:18:59.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.564+0000 7f410e7fc640 1 -- 172.21.15.29:0/2616968923 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104031c30 con 0x7f41100cf5f0 2023-10-14T18:18:59.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.564+0000 7f4116083640 1 -- 172.21.15.29:0/2616968923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100cf5f0 msgr2=0x7f411006c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.564+0000 7f4116083640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100cf5f0 0x7f411006c140 secure :-1 s=READY pgs=2957 cs=0 l=1 rev1=1 crypto rx=0x7f4104009870 tx=0x7f410402b7e0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 -- 172.21.15.29:0/2616968923 shutdown_connections 2023-10-14T18:18:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100cf5f0 0x7f411006c140 unknown :-1 s=CLOSED pgs=2957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 --2- 172.21.15.29:0/2616968923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4110070830 0x7f411006d3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 -- 172.21.15.29:0/2616968923 >> 172.21.15.29:0/2616968923 conn(0x7f41100cfac0 msgr2=0x7f41100cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:18:59.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 -- 172.21.15.29:0/2616968923 shutdown_connections 2023-10-14T18:18:59.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.565+0000 7f4116083640 1 -- 172.21.15.29:0/2616968923 wait complete. 2023-10-14T18:18:59.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.567+0000 7f4116083640 1 Processor -- start 2023-10-14T18:18:59.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.567+0000 7f4116083640 1 -- start start 2023-10-14T18:18:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110070830 0x7f4110153e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4110070c10 0x7f411015eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100cf5f0 0x7f411015f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411015e4e0 con 0x7f4110070c10 2023-10-14T18:18:59.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41100b4af0 con 0x7f4110070830 2023-10-14T18:18:59.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.568+0000 7f4116083640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41100b4dd0 con 0x7f41100cf5f0 2023-10-14T18:18:59.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4110070c10 0x7f411015eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4110070c10 0x7f411015eb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:41050/0 (socket says 172.21.15.29:41050) 2023-10-14T18:18:59.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 -- 172.21.15.29:0/1988882442 learned_addr learned my addr 172.21.15.29:0/1988882442 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:18:59.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110070830 0x7f4110153e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410ffff640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100cf5f0 0x7f411015f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:18:59.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100cf5f0 msgr2=0x7f411015f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100cf5f0 0x7f411015f040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110070830 msgr2=0x7f4110153e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:18:59.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110070830 0x7f4110153e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:18:59.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.569+0000 7f410effd640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41040094d0 con 0x7f4110070c10 2023-10-14T18:18:59.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f410effd640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4110070c10 0x7f411015eb00 secure :-1 s=READY pgs=2958 cs=0 l=1 rev1=1 crypto rx=0x7f4104009870 tx=0x7f41040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:18:59.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4104033c70 con 0x7f4110070c10 2023-10-14T18:18:59.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4104033dd0 con 0x7f4110070c10 2023-10-14T18:18:59.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410403c640 con 0x7f4110070c10 2023-10-14T18:18:59.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f4116083640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41100b50b0 con 0x7f4110070c10 2023-10-14T18:18:59.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.570+0000 7f4116083640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41101543b0 con 0x7f4110070c10 2023-10-14T18:18:59.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.571+0000 7f4116083640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4110154540 con 0x7f4110070c10 2023-10-14T18:18:59.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.573+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 148) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f410403a040 con 0x7f4110070c10 2023-10-14T18:18:59.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.574+0000 7f40f7fff640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:18:59.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.574+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(254..254 src has 1..254) v4 ==== 6689+0+0 (secure 0 0 0) 0x7f41040abba0 con 0x7f4110070c10 2023-10-14T18:18:59.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.574+0000 7f410f7fe640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:59.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.574+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:18:59.579 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.577+0000 7f9c8e21a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:18:59.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.578+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4110154540 con 0x7f4110070c10 2023-10-14T18:18:59.591 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.591+0000 7f3513a33040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:18:59.640 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.640+0000 7f3513a33040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:18:59.643 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.642+0000 7f9c8e21a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:18:59.700 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.699+0000 7f9c8e21a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:18:59.759 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.759+0000 7f3513a33040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:18:59.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.775+0000 7f410f7fe640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:18:59.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.775+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:18:59.802 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:18:59.801+0000 7f4116083640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f41100cf060 con 0x7f40fc06b130 2023-10-14T18:18:59.817 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.816+0000 7f9c8e21a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:18:59.822 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.822+0000 7f3513a33040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:59.874 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.873+0000 7f3513a33040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:18:59.882 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.882+0000 7f9c8e21a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:18:59.931 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.931+0000 7f3513a33040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:18:59.934 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.933+0000 7f9c8e21a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:18:59.983 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:18:59.983+0000 7f3513a33040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:18:59.993 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:18:59.992+0000 7f9c8e21a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:00.046 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.045+0000 7f9c8e21a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:00.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:00.176+0000 7f410f7fe640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:19:00.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:00.176+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T18:19:00.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.377+0000 7f3513a33040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:00.447 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.446+0000 7f9c8e21a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:00.541 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.541+0000 7f3513a33040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:00.616 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.615+0000 7f9c8e21a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:00.651 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.651+0000 7f3513a33040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:00.702 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.702+0000 7f3513a33040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:00.729 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.728+0000 7f9c8e21a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:00.781 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.780+0000 7f9c8e21a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:00.814 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.814+0000 7f3513a33040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:00.894 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:00.893+0000 7f9c8e21a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:00.923 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:00.923+0000 7f3513a33040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:00.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:00.977+0000 7f410f7fe640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:19:00.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:00.977+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T18:19:01.006 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:01.005+0000 7f9c8e21a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:01.082 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:01.081+0000 7f3513a33040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:01.132 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:01.132+0000 7f3513a33040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:01.170 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:01.169+0000 7f9c8e21a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:01.221 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:01.220+0000 7f9c8e21a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:01.253 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:01.253+0000 7f3513a33040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:01.347 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:01.346+0000 7f9c8e21a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:02.090 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:02.089+0000 7f3513a33040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:02.232 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:02.231+0000 7f9c8e21a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:02.392 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:02.392+0000 7f3513a33040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:02.450 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:02.450+0000 7f3513a33040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:02.544 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:02.543+0000 7f9c8e21a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:02.566 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:02.566+0000 7f3513a33040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:02.578+0000 7f410f7fe640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4275371095 2023-10-14T18:19:02.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:02.578+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T18:19:02.605 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:02.604+0000 7f9c8e21a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:02.723 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:02.722+0000 7f9c8e21a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:02.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:02.742+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 149) v1 ==== 88957+0+0 (secure 0 0 0) 0x7f4104076850 con 0x7f4110070c10 2023-10-14T18:19:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:02.742+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 msgr2=0x7f40fc06d5f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:02.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:02.742+0000 7f40f7fff640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/4275371095,v1:172.21.15.29:6833/4275371095] conn(0x7f40fc06b130 0x7f40fc06d5f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:03.393 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.392+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.392+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.392+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.392+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.392+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.393+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.393+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.393+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.395 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.393+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.396 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.393+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.413 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.414 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0a1d1640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.415 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.413+0000 7f9c0d1d7640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:03.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.753+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 150) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f4104091920 con 0x7f4110070c10 2023-10-14T18:19:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.754+0000 7f40f7fff640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f40fc0356f0 0x7f40fc035ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:03.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.754+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 --> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "mandatory_fs_argument_test_dir", "snap_schedule": "1M", "target": ["mon-mgr", ""]}) v1 -- 0x7f40fc036010 con 0x7f40fc0356f0 2023-10-14T18:19:03.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.759+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f40fc0356f0 0x7f40fc035ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:03.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.770+0000 7f410f7fe640 1 --2- 172.21.15.29:0/1988882442 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f40fc0356f0 0x7f40fc035ad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f4100007040 tx=0x7f4100008040 comp rx=0 tx=0).ready entity=mgr.12399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:03.777 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:03.777+0000 7f9c14225640 -1 mgr.server reply reply (22) Invalid argument filesystem argument is required when there is more than one file system 2023-10-14T18:19:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:03.777+0000 7f40f7fff640 1 -- 172.21.15.29:0/1988882442 <== mgr.12399 v2:172.21.15.29:6832/1194987085 1 ==== mgr_command_reply(tid 0: -22 filesystem argument is required when there is more than one file system) v1 ==== 79+0+0 (secure 0 0 0) 0x7f40fc036010 con 0x7f40fc0356f0 2023-10-14T18:19:03.778 INFO:teuthology.orchestra.run.smithi029.stderr:Error EINVAL: filesystem argument is required when there is more than one file system 2023-10-14T18:19:03.798 DEBUG:teuthology.orchestra.run:got remote process result: 22 2023-10-14T18:19:03.799 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:03.799 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir mandatory_fs_argument_test_dir) 2023-10-14T18:19:03.864 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T18:19:04.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.090+0000 7f24e56d4640 1 Processor -- start 2023-10-14T18:19:04.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.090+0000 7f24e56d4640 1 -- start start 2023-10-14T18:19:04.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24e56d4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f24e00caa80 0x7f24e0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24e56d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24e56d4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e006a730 con 0x7f24e00addb0 2023-10-14T18:19:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24e56d4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e0161a50 con 0x7f24e00caa80 2023-10-14T18:19:04.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24e56d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00caef0 con 0x7f24e006a2e0 2023-10-14T18:19:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24df7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24df7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:43458/0 (socket says 172.21.15.29:43458) 2023-10-14T18:19:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24df7fe640 1 -- 172.21.15.29:0/801873766 learned_addr learned my addr 172.21.15.29:0/801873766 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24deffd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f24e00addb0 0x7f24e00c9000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:35316/0 (socket says 172.21.15.29:35316) 2023-10-14T18:19:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.092+0000 7f24de7fc640 1 --2- 172.21.15.29:0/801873766 >> v2:172.21.15.186:3300/0 conn(0x7f24e00caa80 0x7f24e0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.093+0000 7f24df7fe640 1 -- 172.21.15.29:0/801873766 >> v2:172.21.15.186:3300/0 conn(0x7f24e00caa80 msgr2=0x7f24e0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:04.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.093+0000 7f24df7fe640 1 --2- 172.21.15.29:0/801873766 >> v2:172.21.15.186:3300/0 conn(0x7f24e00caa80 0x7f24e0069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.093+0000 7f24df7fe640 1 -- 172.21.15.29:0/801873766 >> v1:172.21.15.186:6789/0 conn(0x7f24e00addb0 legacy=0x7f24e00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.093+0000 7f24df7fe640 1 -- 172.21.15.29:0/801873766 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e00cb140 con 0x7f24e006a2e0 2023-10-14T18:19:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.094+0000 7f24df7fe640 1 --2- 172.21.15.29:0/801873766 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009870 tx=0x7f24d802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e6103784cd30154 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:04.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.095+0000 7f24ddffb640 1 -- 172.21.15.29:0/801873766 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8030840 con 0x7f24e006a2e0 2023-10-14T18:19:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.095+0000 7f24ddffb640 1 -- 172.21.15.29:0/801873766 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d80309a0 con 0x7f24e006a2e0 2023-10-14T18:19:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.095+0000 7f24ddffb640 1 -- 172.21.15.29:0/801873766 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8030d30 con 0x7f24e006a2e0 2023-10-14T18:19:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 -- 172.21.15.29:0/801873766 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 msgr2=0x7f24e01650d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 --2- 172.21.15.29:0/801873766 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009870 tx=0x7f24d802b980 comp rx=0 tx=0).stop 2023-10-14T18:19:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 -- 172.21.15.29:0/801873766 shutdown_connections 2023-10-14T18:19:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 --2- 172.21.15.29:0/801873766 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e006a2e0 0x7f24e01650d0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 --2- 172.21.15.29:0/801873766 >> v2:172.21.15.186:3300/0 conn(0x7f24e00caa80 0x7f24e0069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 -- 172.21.15.29:0/801873766 >> 172.21.15.29:0/801873766 conn(0x7f24e00ce9b0 msgr2=0x7f24e00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:04.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 -- 172.21.15.29:0/801873766 shutdown_connections 2023-10-14T18:19:04.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.096+0000 7f24e56d4640 1 -- 172.21.15.29:0/801873766 wait complete. 2023-10-14T18:19:04.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.098+0000 7f24e56d4640 1 Processor -- start 2023-10-14T18:19:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.098+0000 7f24e56d4640 1 -- start start 2023-10-14T18:19:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24e006a2e0 0x7f24e014c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e00addb0 0x7f24e00c74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e014ccf0 con 0x7f24e00caa80 2023-10-14T18:19:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00c7ec0 con 0x7f24e006a2e0 2023-10-14T18:19:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24e56d4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e00c81a0 con 0x7f24e00addb0 2023-10-14T18:19:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24df7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49920/0 (socket says 172.21.15.29:49920) 2023-10-14T18:19:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24df7fe640 1 -- 172.21.15.29:0/3784127024 learned_addr learned my addr 172.21.15.29:0/3784127024 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24de7fc640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e00addb0 0x7f24e00c74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.099+0000 7f24deffd640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24e006a2e0 0x7f24e014c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 -- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e00addb0 msgr2=0x7f24e00c74e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e00addb0 0x7f24e00c74e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 -- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24e006a2e0 msgr2=0x7f24e014c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24e006a2e0 0x7f24e014c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 -- 172.21.15.29:0/3784127024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24d80094d0 con 0x7f24e00caa80 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24df7fe640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009870 tx=0x7f24d8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8039c90 con 0x7f24e00caa80 2023-10-14T18:19:04.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d8039e10 con 0x7f24e00caa80 2023-10-14T18:19:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e00c8480 con 0x7f24e00caa80 2023-10-14T18:19:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.100+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d8043640 con 0x7f24e00caa80 2023-10-14T18:19:04.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.101+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e014d250 con 0x7f24e00caa80 2023-10-14T18:19:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.102+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e014d440 con 0x7f24e00caa80 2023-10-14T18:19:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.103+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 150) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f24d80437a0 con 0x7f24e00caa80 2023-10-14T18:19:04.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.104+0000 7f24d77fe640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 0x7f24c006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:04.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.104+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7f24d803e070 con 0x7f24e00caa80 2023-10-14T18:19:04.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.105+0000 7f24deffd640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 0x7f24c006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:04.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.105+0000 7f24deffd640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 0x7f24c006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24d0000f60 tx=0x7f24d0007cc0 comp rx=0 tx=0).ready entity=mgr.12399 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:04.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.109+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f24d80776b0 con 0x7f24e00caa80 2023-10-14T18:19:04.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.356+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f24e004e160 con 0x7f24e00caa80 2023-10-14T18:19:04.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.759+0000 7f24d77fe640 1 -- 172.21.15.29:0/3784127024 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v151) v1 ==== 93+0+0 (secure 0 0 0) 0x7f24d807acd0 con 0x7f24e00caa80 2023-10-14T18:19:04.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 msgr2=0x7f24c006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 0x7f24c006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f24d0000f60 tx=0x7f24d0007cc0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 msgr2=0x7f24e00bd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:04.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 secure :-1 s=READY pgs=2976 cs=0 l=1 rev1=1 crypto rx=0x7f24d8009870 tx=0x7f24d8004730 comp rx=0 tx=0).stop 2023-10-14T18:19:04.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 shutdown_connections 2023-10-14T18:19:04.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f24c006ad40 0x7f24c006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f24e00caa80 0x7f24e00bd570 unknown :-1 s=CLOSED pgs=2976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f24e00addb0 0x7f24e00c74e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 --2- 172.21.15.29:0/3784127024 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f24e006a2e0 0x7f24e014c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:04.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.765+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 >> 172.21.15.29:0/3784127024 conn(0x7f24e00ce9b0 msgr2=0x7f24e00cd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:04.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.766+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 shutdown_connections 2023-10-14T18:19:04.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:04.767+0000 7f24e56d4640 1 -- 172.21.15.29:0/3784127024 wait complete. 2023-10-14T18:19:04.793 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:19:04.794 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:04.893 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:04.993 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:19:04.994 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:04.994 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:05.049 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:05.050 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:05.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:05.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.312+0000 7fc362887640 1 Processor -- start 2023-10-14T18:19:05.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.312+0000 7fc362887640 1 -- start start 2023-10-14T18:19:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.313+0000 7fc362887640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 0x7fc35c06e060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc362887640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c0ae4f0 con 0x7fc35c15d000 2023-10-14T18:19:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc362887640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35c06e6f0 con 0x7fc35c071d40 2023-10-14T18:19:05.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc362887640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c070f80 con 0x7fc35c06b4c0 2023-10-14T18:19:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc35bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc35c15d000 0x7fc35c06adb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50782/0 (socket says 172.21.15.29:50782) 2023-10-14T18:19:05.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc35bfff640 1 -- 172.21.15.29:0/2861499032 learned_addr learned my addr 172.21.15.29:0/2861499032 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.314+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 0x7fc35c06e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2412803135 0 0) 0x7fc35c0ae4f0 con 0x7fc35c15d000 2023-10-14T18:19:05.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc344003520 con 0x7fc35c15d000 2023-10-14T18:19:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2792517017 0 0) 0x7fc344003520 con 0x7fc35c15d000 2023-10-14T18:19:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 msgr2=0x7fc35c06e060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 0x7fc35c06e060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 >> v1:172.21.15.186:6789/0 conn(0x7fc35c071d40 legacy=0x7fc35c06bc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c071260 con 0x7fc35c15d000 2023-10-14T18:19:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc34c001670 con 0x7fc35c15d000 2023-10-14T18:19:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 >> v1:172.21.15.29:6789/0 conn(0x7fc35c15d000 legacy=0x7fc35c06adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc344003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3440047c0 0x7fc344004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 0x7fc344005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c070f80 con 0x7fc3440039a0 2023-10-14T18:19:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400ba30 con 0x7fc3440047c0 2023-10-14T18:19:05.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.315+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc34400bba0 con 0x7fc35c06b4c0 2023-10-14T18:19:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc344003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3440047c0 0x7fc344004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 -- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 msgr2=0x7fc344005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:05.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c06b4c0 0x7fc344005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 -- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3440047c0 msgr2=0x7fc344004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3440047c0 0x7fc344004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.316+0000 7fc35bfff640 1 -- 172.21.15.29:0/2861499032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc344003520 con 0x7fc3440039a0 2023-10-14T18:19:05.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.317+0000 7fc35bfff640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc344003d80 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c160 tx=0x7fc3440057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1730b4a1a083349f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.318+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c005ad0 con 0x7fc3440039a0 2023-10-14T18:19:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.318+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c005c30 con 0x7fc3440039a0 2023-10-14T18:19:05.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.318+0000 7fc35affd640 1 -- 172.21.15.29:0/2861499032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c02c7d0 con 0x7fc3440039a0 2023-10-14T18:19:05.326 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.321+0000 7fbf8d411040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:05.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 -- 172.21.15.29:0/2861499032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 msgr2=0x7fc344003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 --2- 172.21.15.29:0/2861499032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc344003d80 secure :-1 s=READY pgs=2979 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c160 tx=0x7fc3440057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc360e24640 1 -- 172.21.15.29:0/2861499032 reap_dead start 2023-10-14T18:19:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 -- 172.21.15.29:0/2861499032 shutdown_connections 2023-10-14T18:19:05.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 -- 172.21.15.29:0/2861499032 >> 172.21.15.29:0/2861499032 conn(0x7fc35c0b0c80 msgr2=0x7fc35c0b1090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 -- 172.21.15.29:0/2861499032 shutdown_connections 2023-10-14T18:19:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.319+0000 7fc362887640 1 -- 172.21.15.29:0/2861499032 wait complete. 2023-10-14T18:19:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.322+0000 7fc362887640 1 Processor -- start 2023-10-14T18:19:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.322+0000 7fc362887640 1 -- start start 2023-10-14T18:19:05.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 0x7fc35c159bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 0x7fc35c099d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c0c6590 con 0x7fc3440039a0 2023-10-14T18:19:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c15a5b0 con 0x7fc35c159770 2023-10-14T18:19:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc362887640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc35c09e8a0 con 0x7fc35c15a130 2023-10-14T18:19:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49938/0 (socket says 172.21.15.29:49938) 2023-10-14T18:19:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 -- 172.21.15.29:0/3224454788 learned_addr learned my addr 172.21.15.29:0/3224454788 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 0x7fc35c159bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 msgr2=0x7fc35c099d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc360e24640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 0x7fc35c099d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 0x7fc35c099d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 msgr2=0x7fc35c159bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.324+0000 7fc35bfff640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 0x7fc35c159bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc35bfff640 1 -- 172.21.15.29:0/3224454788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc35c09b520 con 0x7fc3440039a0 2023-10-14T18:19:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 0x7fc35c159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc360e24640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 0x7fc35c099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc35bfff640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c280 tx=0x7fc34c001930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c03c040 con 0x7fc3440039a0 2023-10-14T18:19:05.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc34c02cc20 con 0x7fc3440039a0 2023-10-14T18:19:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc34c007070 con 0x7fc3440039a0 2023-10-14T18:19:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc35c160840 con 0x7fc3440039a0 2023-10-14T18:19:05.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.325+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc35c09b240 con 0x7fc3440039a0 2023-10-14T18:19:05.338 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:05.326+0000 7fb12a044040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.327+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc35c04e160 con 0x7fc3440039a0 2023-10-14T18:19:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.327+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fc34c032070 con 0x7fc3440039a0 2023-10-14T18:19:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.329+0000 7fc358ff9640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 0x7fc34806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.329+0000 7fc35b7fe640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 msgr2=0x7fc34806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.329+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7fc34c0ab4f0 con 0x7fc3440039a0 2023-10-14T18:19:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.329+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 0x7fc34806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:05.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.333+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc34c075d60 con 0x7fc3440039a0 2023-10-14T18:19:05.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.377+0000 7fbf8d411040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:05.386 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:05.385+0000 7fb12a044040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:05.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.529+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc35c0c5fd0 con 0x7fc3440039a0 2023-10-14T18:19:05.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.530+0000 7fc35b7fe640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 msgr2=0x7fc34806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:05.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.530+0000 7fc35b7fe640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 0x7fc34806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:05.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.533+0000 7fc358ff9640 1 -- 172.21.15.29:0/3224454788 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v255) v1 ==== 74+0+20983 (secure 0 0 0) 0x7fc34c046090 con 0x7fc3440039a0 2023-10-14T18:19:05.535 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:05.535 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":255,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:02.726649+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 msgr2=0x7fc34806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 0x7fc34806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 msgr2=0x7fc35c160300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 secure :-1 s=READY pgs=2980 cs=0 l=1 rev1=1 crypto rx=0x7fc34c00c280 tx=0x7fc34c001930 comp rx=0 tx=0).stop 2023-10-14T18:19:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 shutdown_connections 2023-10-14T18:19:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fc34806b0b0 0x7fc34806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc35c15a130 0x7fc35c099d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc35c159770 0x7fc35c159bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 --2- 172.21.15.29:0/3224454788 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3440039a0 0x7fc35c160300 unknown :-1 s=CLOSED pgs=2980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 >> 172.21.15.29:0/3224454788 conn(0x7fc35c0b0c80 msgr2=0x7fc35c072f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 shutdown_connections 2023-10-14T18:19:05.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.536+0000 7fc362887640 1 -- 172.21.15.29:0/3224454788 wait complete. 2023-10-14T18:19:05.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:18:33.087026+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:18:34.140901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:18:47.629053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:18:48.796314+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:05.557 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:05.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.758+0000 7fe54759e640 1 Processor -- start 2023-10-14T18:19:05.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.759+0000 7fe54759e640 1 -- start start 2023-10-14T18:19:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.760+0000 7fe54759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5480b3ec0 0x7fe5480cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.760+0000 7fe54759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe5480b0ea0 0x7fe5480cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.760+0000 7fe54759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe54806be30 con 0x7fe5480b43c0 2023-10-14T18:19:05.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.760+0000 7fe54759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe54806ca00 con 0x7fe5480b3ec0 2023-10-14T18:19:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.760+0000 7fe54759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5480ada10 con 0x7fe5480b0ea0 2023-10-14T18:19:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5480b3ec0 0x7fe5480cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.763 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe5480b3ec0 0x7fe5480cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49944/0 (socket says 172.21.15.29:49944) 2023-10-14T18:19:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 -- 172.21.15.29:0/173169534 learned_addr learned my addr 172.21.15.29:0/173169534 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:05.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 -- 172.21.15.29:0/173169534 >> v2:172.21.15.186:3300/0 conn(0x7fe5480b0ea0 msgr2=0x7fe5480cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 --2- 172.21.15.29:0/173169534 >> v2:172.21.15.186:3300/0 conn(0x7fe5480b0ea0 0x7fe5480cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 -- 172.21.15.29:0/173169534 >> v1:172.21.15.186:6789/0 conn(0x7fe5480b43c0 legacy=0x7fe54815a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:05.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.761+0000 7fe54659c640 1 -- 172.21.15.29:0/173169534 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54806c060 con 0x7fe5480b3ec0 2023-10-14T18:19:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.762+0000 7fe54659c640 1 --2- 172.21.15.29:0/173169534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5480cd260 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fe548067810 tx=0x7fe53802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e912106b9da382c8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:05.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.763+0000 7fe545d9b640 1 -- 172.21.15.29:0/173169534 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe5380317e0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.763+0000 7fe545d9b640 1 -- 172.21.15.29:0/173169534 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe538031940 con 0x7fe5480b3ec0 2023-10-14T18:19:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.763+0000 7fe545d9b640 1 -- 172.21.15.29:0/173169534 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe538031cd0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.763+0000 7fe54759e640 1 -- 172.21.15.29:0/173169534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 msgr2=0x7fe5480cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.763+0000 7fe54759e640 1 --2- 172.21.15.29:0/173169534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5480cd260 secure :-1 s=READY pgs=2981 cs=0 l=1 rev1=1 crypto rx=0x7fe548067810 tx=0x7fe53802b880 comp rx=0 tx=0).stop 2023-10-14T18:19:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 -- 172.21.15.29:0/173169534 shutdown_connections 2023-10-14T18:19:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 --2- 172.21.15.29:0/173169534 >> v2:172.21.15.186:3300/0 conn(0x7fe5480b0ea0 0x7fe5480cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 --2- 172.21.15.29:0/173169534 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5480cd260 unknown :-1 s=CLOSED pgs=2981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 -- 172.21.15.29:0/173169534 >> 172.21.15.29:0/173169534 conn(0x7fe5480b0170 msgr2=0x7fe5480b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:05.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 -- 172.21.15.29:0/173169534 shutdown_connections 2023-10-14T18:19:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.764+0000 7fe54759e640 1 -- 172.21.15.29:0/173169534 wait complete. 2023-10-14T18:19:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.765+0000 7fe54759e640 1 Processor -- start 2023-10-14T18:19:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.766+0000 7fe54759e640 1 -- start start 2023-10-14T18:19:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.766+0000 7fe54759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5480b0ea0 0x7fe548157cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.766+0000 7fe54759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe54759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5480b43c0 0x7fe5481f3d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe54759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54814a940 con 0x7fe5480b3ec0 2023-10-14T18:19:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe54759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5480c6960 con 0x7fe5480b43c0 2023-10-14T18:19:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe54759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe54815b040 con 0x7fe5480b0ea0 2023-10-14T18:19:05.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49952/0 (socket says 172.21.15.29:49952) 2023-10-14T18:19:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 -- 172.21.15.29:0/2096761243 learned_addr learned my addr 172.21.15.29:0/2096761243 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe54659c640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5480b0ea0 0x7fe548157cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5480b0ea0 msgr2=0x7fe548157cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.774 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5480b0ea0 0x7fe548157cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5480b43c0 msgr2=0x7fe5481f3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe546d9d640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5480b43c0 0x7fe5481f3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.767+0000 7fe53ffff640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5480b43c0 0x7fe5481f3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe53ffff640 1 -- 172.21.15.29:0/2096761243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5380094d0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe53ffff640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b410 tx=0x7fe53000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530011830 con 0x7fe5480b3ec0 2023-10-14T18:19:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5300119b0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54815a7a0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530011d60 con 0x7fe5480b3ec0 2023-10-14T18:19:05.776 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.768+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54815ad60 con 0x7fe5480b3ec0 2023-10-14T18:19:05.777 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.770+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe54804e9b0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.778 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.770+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe53001a400 con 0x7fe5480b3ec0 2023-10-14T18:19:05.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.771+0000 7fe53effd640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 0x7fe52806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.772+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7fe53008a0d0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.772+0000 7fe54659c640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 msgr2=0x7fe52806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:05.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.772+0000 7fe54659c640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 0x7fe52806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:05.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.780+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe530054930 con 0x7fe5480b3ec0 2023-10-14T18:19:05.834 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.833+0000 7fbf8d411040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:05.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:05.864+0000 7fb12a044040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:05.886 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.886+0000 7fbf8d411040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:05.919 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:05.918+0000 7fb12a044040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:05.937 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.937+0000 7fbf8d411040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:05.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.971+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fe54806abd0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.973 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:05.971+0000 7fb12a044040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:05.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.972+0000 7fe53effd640 1 -- 172.21.15.29:0/2096761243 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v255) v1 ==== 93+0+10 (secure 0 0 0) 0x7fe5300150a0 con 0x7fe5480b3ec0 2023-10-14T18:19:05.973 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.973+0000 7fe54659c640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 msgr2=0x7fe52806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:05.974 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.973+0000 7fe54659c640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 0x7fe52806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 msgr2=0x7fe52806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:05.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 0x7fe52806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 msgr2=0x7fe5481f37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:05.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 secure :-1 s=READY pgs=2982 cs=0 l=1 rev1=1 crypto rx=0x7fe53000b410 tx=0x7fe53000b8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 shutdown_connections 2023-10-14T18:19:05.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe52806b0b0 0x7fe52806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe5480b43c0 0x7fe5481f3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe5480b3ec0 0x7fe5481f37d0 unknown :-1 s=CLOSED pgs=2982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 --2- 172.21.15.29:0/2096761243 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe5480b0ea0 0x7fe548157cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 >> 172.21.15.29:0/2096761243 conn(0x7fe5480b0170 msgr2=0x7fe5480ca320 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:05.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 shutdown_connections 2023-10-14T18:19:05.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:05.975+0000 7fe54759e640 1 -- 172.21.15.29:0/2096761243 wait complete. 2023-10-14T18:19:05.994 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:19:05.998 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:05.998+0000 7fbf8d411040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:06.037 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.036+0000 7fb12a044040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:06.047 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.047+0000 7fbf8d411040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:06.089 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.088+0000 7fb12a044040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:06.158 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.158+0000 7fbf8d411040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:06.206 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.205+0000 7fb12a044040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.205+0000 7fe05172c640 1 Processor -- start 2023-10-14T18:19:06.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.206+0000 7fe05172c640 1 -- start start 2023-10-14T18:19:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe05172c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe05172c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe04c0cded0 0x7fe04c0b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe05172c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c15d150 con 0x7fe04c0b8480 2023-10-14T18:19:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe05172c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe04c0d1180 con 0x7fe04c0cded0 2023-10-14T18:19:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe05172c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0d1460 con 0x7fe04c15cd70 2023-10-14T18:19:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe04bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe04bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:43546/0 (socket says 172.21.15.29:43546) 2023-10-14T18:19:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.207+0000 7fe04bfff640 1 -- 172.21.15.29:0/1052144207 learned_addr learned my addr 172.21.15.29:0/1052144207 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:06.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.208+0000 7fe050f2b640 1 --2- 172.21.15.29:0/1052144207 >> v2:172.21.15.186:3300/0 conn(0x7fe04c0cded0 0x7fe04c0b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.208+0000 7fe04bfff640 1 -- 172.21.15.29:0/1052144207 >> v2:172.21.15.186:3300/0 conn(0x7fe04c0cded0 msgr2=0x7fe04c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.208+0000 7fe04bfff640 1 --2- 172.21.15.29:0/1052144207 >> v2:172.21.15.186:3300/0 conn(0x7fe04c0cded0 0x7fe04c0b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.208+0000 7fe04bfff640 1 -- 172.21.15.29:0/1052144207 >> v1:172.21.15.186:6789/0 conn(0x7fe04c0b8480 legacy=0x7fe04c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.208+0000 7fe04bfff640 1 -- 172.21.15.29:0/1052144207 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0d1740 con 0x7fe04c15cd70 2023-10-14T18:19:06.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.209+0000 7fe04bfff640 1 --2- 172.21.15.29:0/1052144207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3cc09be61fb18c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.210+0000 7fe04affd640 1 -- 172.21.15.29:0/1052144207 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c030840 con 0x7fe04c15cd70 2023-10-14T18:19:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.210+0000 7fe04affd640 1 -- 172.21.15.29:0/1052144207 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe03c0309a0 con 0x7fe04c15cd70 2023-10-14T18:19:06.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.210+0000 7fe04affd640 1 -- 172.21.15.29:0/1052144207 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe03c030d30 con 0x7fe04c15cd70 2023-10-14T18:19:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 -- 172.21.15.29:0/1052144207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 msgr2=0x7fe04c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 --2- 172.21.15.29:0/1052144207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fe03c009870 tx=0x7fe03c02b980 comp rx=0 tx=0).stop 2023-10-14T18:19:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 -- 172.21.15.29:0/1052144207 shutdown_connections 2023-10-14T18:19:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 --2- 172.21.15.29:0/1052144207 >> v2:172.21.15.186:3300/0 conn(0x7fe04c0cded0 0x7fe04c0b5c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 --2- 172.21.15.29:0/1052144207 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c0d2220 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 -- 172.21.15.29:0/1052144207 >> 172.21.15.29:0/1052144207 conn(0x7fe04c0cf570 msgr2=0x7fe04c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:06.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 -- 172.21.15.29:0/1052144207 shutdown_connections 2023-10-14T18:19:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.211+0000 7fe05172c640 1 -- 172.21.15.29:0/1052144207 wait complete. 2023-10-14T18:19:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.213+0000 7fe05172c640 1 Processor -- start 2023-10-14T18:19:06.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.213+0000 7fe05172c640 1 -- start start 2023-10-14T18:19:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 0x7fe04c0cbf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c1f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0b21d0 con 0x7fe04c0cded0 2023-10-14T18:19:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0b24b0 con 0x7fe04c0b8480 2023-10-14T18:19:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe05172c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe04c0b2790 con 0x7fe04c15cd70 2023-10-14T18:19:06.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe04b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe04b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49958/0 (socket says 172.21.15.29:49958) 2023-10-14T18:19:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe04b7fe640 1 -- 172.21.15.29:0/2655654479 learned_addr learned my addr 172.21.15.29:0/2655654479 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe04bfff640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 0x7fe04c0cbf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.214+0000 7fe050f2b640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c1f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 msgr2=0x7fe04c1f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c1f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 msgr2=0x7fe04c0cbf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 0x7fe04c0cbf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 -- 172.21.15.29:0/2655654479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe03c0094d0 con 0x7fe04c0cded0 2023-10-14T18:19:06.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04bfff640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 0x7fe04c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe04b7fe640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c450 tx=0x7fe04000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0400052d0 con 0x7fe04c0cded0 2023-10-14T18:19:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe040005430 con 0x7fe04c0cded0 2023-10-14T18:19:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe040012870 con 0x7fe04c0cded0 2023-10-14T18:19:06.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.215+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe04c0b2a70 con 0x7fe04c0cded0 2023-10-14T18:19:06.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.216+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04c0b1ea0 con 0x7fe04c0cded0 2023-10-14T18:19:06.222 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.222+0000 7fbf8d411040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.217+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04c04e9b0 con 0x7fe04c0cded0 2023-10-14T18:19:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.218+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe0400129d0 con 0x7fe04c0cded0 2023-10-14T18:19:06.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.219+0000 7fe048ff9640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.219+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7fe040089ba0 con 0x7fe04c0cded0 2023-10-14T18:19:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.219+0000 7fe04bfff640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 msgr2=0x7fe03806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:06.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.219+0000 7fe04bfff640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:06.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.227+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe0400542d0 con 0x7fe04c0cded0 2023-10-14T18:19:06.273 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.273+0000 7fbf8d411040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:06.277 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.276+0000 7fb12a044040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:06.334 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.333+0000 7fb12a044040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:06.335 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.334+0000 7fbf8d411040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:06.386 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.386+0000 7fbf8d411040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:06.394 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.393+0000 7fb12a044040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:06.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.420+0000 7fe04bfff640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 msgr2=0x7fe03806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:06.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.420+0000 7fe04bfff640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:06.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.430+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe04c15b5a0 con 0x7fe04c0cded0 2023-10-14T18:19:06.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.431+0000 7fe048ff9640 1 -- 172.21.15.29:0/2655654479 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v255) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe0400578f0 con 0x7fe04c0cded0 2023-10-14T18:19:06.432 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:06.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 msgr2=0x7fe03806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:06.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 msgr2=0x7fe04c0cc4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 secure :-1 s=READY pgs=2983 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c450 tx=0x7fe04000c910 comp rx=0 tx=0).stop 2023-10-14T18:19:06.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 shutdown_connections 2023-10-14T18:19:06.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe03806b0b0 0x7fe03806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe04c15cd70 0x7fe04c1f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.437 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe04c0cded0 0x7fe04c0cc4d0 unknown :-1 s=CLOSED pgs=2983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 --2- 172.21.15.29:0/2655654479 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe04c0b8480 0x7fe04c0cbf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 >> 172.21.15.29:0/2655654479 conn(0x7fe04c0cf570 msgr2=0x7fe04c0d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 shutdown_connections 2023-10-14T18:19:06.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.435+0000 7fe05172c640 1 -- 172.21.15.29:0/2655654479 wait complete. 2023-10-14T18:19:06.456 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.455+0000 7fb12a044040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:06.462 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:19:06.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.724+0000 7fd2f2fb0640 1 Processor -- start 2023-10-14T18:19:06.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.724+0000 7fd2f2fb0640 1 -- start start 2023-10-14T18:19:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f2fb0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f2fb0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fd2ec0b6cd0 0x7fd2ec0b3250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f2fb0640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2ec0d3980 con 0x7fd2ec0cfaf0 2023-10-14T18:19:06.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f2fb0640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0d3c60 con 0x7fd2ec0cf710 2023-10-14T18:19:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f2fb0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2ec0ce630 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f0d4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f0d4c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49964/0 (socket says 172.21.15.29:49964) 2023-10-14T18:19:06.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.726+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/432397855 learned_addr learned my addr 172.21.15.29:0/432397855 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.727+0000 7fd2f154d640 1 --2- 172.21.15.29:0/432397855 >> v2:172.21.15.186:3300/0 conn(0x7fd2ec0b6cd0 0x7fd2ec0b3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.727+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/432397855 >> v2:172.21.15.186:3300/0 conn(0x7fd2ec0b6cd0 msgr2=0x7fd2ec0b3250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:06.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.727+0000 7fd2f0d4c640 1 --2- 172.21.15.29:0/432397855 >> v2:172.21.15.186:3300/0 conn(0x7fd2ec0b6cd0 0x7fd2ec0b3250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.727+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/432397855 >> v1:172.21.15.29:6789/0 conn(0x7fd2ec0cfaf0 legacy=0x7fd2ec0d0aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.727+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/432397855 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2ec0ce910 con 0x7fd2ec0cf710 2023-10-14T18:19:06.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.729+0000 7fd2f0d4c640 1 --2- 172.21.15.29:0/432397855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b5890 tx=0x7fd2e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98228302a306cf77 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:06.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.729+0000 7fd2eb7fe640 1 -- 172.21.15.29:0/432397855 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0004230 con 0x7fd2ec0cf710 2023-10-14T18:19:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.729+0000 7fd2eb7fe640 1 -- 172.21.15.29:0/432397855 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e0004390 con 0x7fd2ec0cf710 2023-10-14T18:19:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.729+0000 7fd2eb7fe640 1 -- 172.21.15.29:0/432397855 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0003810 con 0x7fd2ec0cf710 2023-10-14T18:19:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/432397855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0cf710 msgr2=0x7fd2ec0b2d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/432397855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 secure :-1 s=READY pgs=2985 cs=0 l=1 rev1=1 crypto rx=0x7fd2ec0b5890 tx=0x7fd2e002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/432397855 shutdown_connections 2023-10-14T18:19:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/432397855 >> v2:172.21.15.186:3300/0 conn(0x7fd2ec0b6cd0 0x7fd2ec0b3250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/432397855 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0b2d10 unknown :-1 s=CLOSED pgs=2985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/432397855 >> 172.21.15.29:0/432397855 conn(0x7fd2ec0d2cc0 msgr2=0x7fd2ec0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/432397855 shutdown_connections 2023-10-14T18:19:06.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.730+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/432397855 wait complete. 2023-10-14T18:19:06.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.732+0000 7fd2f2fb0640 1 Processor -- start 2023-10-14T18:19:06.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.732+0000 7fd2f2fb0640 1 -- start start 2023-10-14T18:19:06.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0cc7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2ec0cfaf0 0x7fd2ec148fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b12f0 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b15d0 con 0x7fd2ec0cf710 2023-10-14T18:19:06.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f2fb0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2ec0b18b0 con 0x7fd2ec0cfaf0 2023-10-14T18:19:06.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f0d4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f0d4c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49974/0 (socket says 172.21.15.29:49974) 2023-10-14T18:19:06.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/1692455359 learned_addr learned my addr 172.21.15.29:0/1692455359 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:06.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.733+0000 7fd2ebfff640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0cc7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f154d640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2ec0cfaf0 0x7fd2ec148fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2ec0cfaf0 msgr2=0x7fd2ec148fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2ec0cfaf0 0x7fd2ec148fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2ec0cf710 msgr2=0x7fd2ec0cc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0cc7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 -- 172.21.15.29:0/1692455359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2e0009460 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.734+0000 7fd2f0d4c640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0000c00 tx=0x7fd2e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.735+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0003e50 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.735+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2e0004500 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.735+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd2e0006280 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.735+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2ec0b1b90 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.735+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2ec156fa0 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.736+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2ec0b20b0 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.737+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fd2e002f050 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.738+0000 7fd2e97fa640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 0x7fd2d006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.739+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7fd2e00ac350 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.739+0000 7fd2ebfff640 1 -- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 msgr2=0x7fd2d006d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:06.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.739+0000 7fd2ebfff640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 0x7fd2d006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:06.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.743+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd2e0076970 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.782 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.781+0000 7fbf8d411040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:06.865 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:06.864+0000 7fb12a044040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:06.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.936+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd2ec04e160 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.936+0000 7fd2e97fa640 1 -- 172.21.15.29:0/1692455359 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v255) v1 ==== 100+0+11 (secure 0 0 0) 0x7fd2e0079f90 con 0x7fd2ec0b6cd0 2023-10-14T18:19:06.938 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:06.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 msgr2=0x7fd2d006d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:06.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 0x7fd2d006d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 msgr2=0x7fd2ec156a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 secure :-1 s=READY pgs=2986 cs=0 l=1 rev1=1 crypto rx=0x7fd2e0000c00 tx=0x7fd2e00046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 shutdown_connections 2023-10-14T18:19:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fd2d006b0b0 0x7fd2d006d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd2ec0cfaf0 0x7fd2ec148fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd2ec0cf710 0x7fd2ec0cc7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 --2- 172.21.15.29:0/1692455359 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd2ec0b6cd0 0x7fd2ec156a60 unknown :-1 s=CLOSED pgs=2986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 >> 172.21.15.29:0/1692455359 conn(0x7fd2ec0d2cc0 msgr2=0x7fd2ec0cdb90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 shutdown_connections 2023-10-14T18:19:06.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:06.939+0000 7fd2f2fb0640 1 -- 172.21.15.29:0/1692455359 wait complete. 2023-10-14T18:19:06.943 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:06.942+0000 7fbf8d411040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:06.959 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:19:07.035 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.034+0000 7fb12a044040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:07.050 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.050+0000 7fbf8d411040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:07.101 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.101+0000 7fbf8d411040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:07.149 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.148+0000 7fb12a044040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:07.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.167+0000 7f40ba11d640 1 Processor -- start 2023-10-14T18:19:07.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.167+0000 7f40ba11d640 1 -- start start 2023-10-14T18:19:07.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.168+0000 7f40ba11d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.168+0000 7f40ba11d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f40b4070c10 0x7f40b40cd690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.168+0000 7f40ba11d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40b4161740 con 0x7f40b4070830 2023-10-14T18:19:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.168+0000 7f40ba11d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40b40cdbd0 con 0x7f40b4070c10 2023-10-14T18:19:07.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.168+0000 7f40ba11d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b406b0f0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.169+0000 7f40abfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.169+0000 7f40abfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:43578/0 (socket says 172.21.15.29:43578) 2023-10-14T18:19:07.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.169+0000 7f40abfff640 1 -- 172.21.15.29:0/468566501 learned_addr learned my addr 172.21.15.29:0/468566501 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.169+0000 7f40b3fff640 1 --2- 172.21.15.29:0/468566501 >> v2:172.21.15.186:3300/0 conn(0x7f40b4070c10 0x7f40b40cd690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.170+0000 7f40abfff640 1 -- 172.21.15.29:0/468566501 >> v2:172.21.15.186:3300/0 conn(0x7f40b4070c10 msgr2=0x7f40b40cd690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.170+0000 7f40abfff640 1 --2- 172.21.15.29:0/468566501 >> v2:172.21.15.186:3300/0 conn(0x7f40b4070c10 0x7f40b40cd690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.170+0000 7f40abfff640 1 -- 172.21.15.29:0/468566501 >> v1:172.21.15.186:6789/0 conn(0x7f40b4070830 legacy=0x7f40b406c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.170+0000 7f40abfff640 1 -- 172.21.15.29:0/468566501 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b41619b0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.171+0000 7f40abfff640 1 --2- 172.21.15.29:0/468566501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f409c009870 tx=0x7f409c02b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bb4abfe763e57bb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40b2ffd640 1 -- 172.21.15.29:0/468566501 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c0317d0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40b2ffd640 1 -- 172.21.15.29:0/468566501 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f409c031930 con 0x7f40b40cf5f0 2023-10-14T18:19:07.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40b2ffd640 1 -- 172.21.15.29:0/468566501 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f409c031cc0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 -- 172.21.15.29:0/468566501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 msgr2=0x7f40b406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 --2- 172.21.15.29:0/468566501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f409c009870 tx=0x7f409c02b870 comp rx=0 tx=0).stop 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 -- 172.21.15.29:0/468566501 shutdown_connections 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 --2- 172.21.15.29:0/468566501 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b40cf5f0 0x7f40b406d3c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 --2- 172.21.15.29:0/468566501 >> v2:172.21.15.186:3300/0 conn(0x7f40b4070c10 0x7f40b40cd690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 -- 172.21.15.29:0/468566501 >> 172.21.15.29:0/468566501 conn(0x7f40b40cfac0 msgr2=0x7f40b40cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:07.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.172+0000 7f40ba11d640 1 -- 172.21.15.29:0/468566501 shutdown_connections 2023-10-14T18:19:07.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.173+0000 7f40ba11d640 1 -- 172.21.15.29:0/468566501 wait complete. 2023-10-14T18:19:07.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.174+0000 7f40ba11d640 1 Processor -- start 2023-10-14T18:19:07.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.174+0000 7f40ba11d640 1 -- start start 2023-10-14T18:19:07.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b4070830 0x7f40b40c9820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b4070c10 0x7f40b40c6b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b41597d0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40c73c0 con 0x7f40b4070c10 2023-10-14T18:19:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.175+0000 7f40ba11d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40c76a0 con 0x7f40b4070830 2023-10-14T18:19:07.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:49990/0 (socket says 172.21.15.29:49990) 2023-10-14T18:19:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 -- 172.21.15.29:0/3010452406 learned_addr learned my addr 172.21.15.29:0/3010452406 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b37fe640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b4070830 0x7f40b40c9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40abfff640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b4070c10 0x7f40b40c6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b4070830 msgr2=0x7f40b40c9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b4070830 0x7f40b40c9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b4070c10 msgr2=0x7f40b40c6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b4070c10 0x7f40b40c6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 -- 172.21.15.29:0/3010452406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f409c0094d0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.176+0000 7f40b3fff640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00efc0 tx=0x7f40ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.177+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac014480 con 0x7f40b40cf5f0 2023-10-14T18:19:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.177+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40ac00f040 con 0x7f40b40cf5f0 2023-10-14T18:19:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.177+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40ac014820 con 0x7f40b40cf5f0 2023-10-14T18:19:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.177+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b40c7980 con 0x7f40b40cf5f0 2023-10-14T18:19:07.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.177+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b40c70d0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.178+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b404e160 con 0x7f40b40cf5f0 2023-10-14T18:19:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.179+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f40ac014980 con 0x7f40b40cf5f0 2023-10-14T18:19:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.180+0000 7f40b0ff9640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 0x7f409406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.180+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7f40ac08ac40 con 0x7f40b40cf5f0 2023-10-14T18:19:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.180+0000 7f40b37fe640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 msgr2=0x7f409406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:07.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.180+0000 7f40b37fe640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 0x7f409406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:07.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.184+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f40ac0552f0 con 0x7f40b40cf5f0 2023-10-14T18:19:07.203 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.202+0000 7fb12a044040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:07.210 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.210+0000 7fbf8d411040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:07.318 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.317+0000 7fb12a044040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:07.320 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.320+0000 7fbf8d411040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:07.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.379+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f40b406d080 con 0x7f40b40cf5f0 2023-10-14T18:19:07.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.380+0000 7f40b0ff9640 1 -- 172.21.15.29:0/3010452406 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v255) v1 ==== 107+0+11 (secure 0 0 0) 0x7f40ac058910 con 0x7f40b40cf5f0 2023-10-14T18:19:07.381 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:07.381 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.381+0000 7f40b37fe640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 msgr2=0x7f409406d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:07.382 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.381+0000 7f40b37fe640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 0x7f409406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:07.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.382+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 msgr2=0x7f409406d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.382+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 0x7f409406d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.382+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 msgr2=0x7f40b4157eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.382+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 secure :-1 s=READY pgs=2987 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00efc0 tx=0x7f40ac00c490 comp rx=0 tx=0).stop 2023-10-14T18:19:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 shutdown_connections 2023-10-14T18:19:07.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f409406b0e0 0x7f409406d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40b40cf5f0 0x7f40b4157eb0 unknown :-1 s=CLOSED pgs=2987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f40b4070c10 0x7f40b40c6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 --2- 172.21.15.29:0/3010452406 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40b4070830 0x7f40b40c9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 >> 172.21.15.29:0/3010452406 conn(0x7f40b40cfac0 msgr2=0x7f40b40cfea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:07.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 shutdown_connections 2023-10-14T18:19:07.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.383+0000 7f40ba11d640 1 -- 172.21.15.29:0/3010452406 wait complete. 2023-10-14T18:19:07.403 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:19:07.432 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.431+0000 7fb12a044040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:07.479 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.479+0000 7fbf8d411040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:07.530 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.529+0000 7fbf8d411040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:07.599 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.598+0000 7fb12a044040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.615+0000 7f4cf23f5640 1 Processor -- start 2023-10-14T18:19:07.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.616+0000 7f4cf23f5640 1 -- start start 2023-10-14T18:19:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.617+0000 7f4cf23f5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cec0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.617+0000 7f4cf23f5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0ce800 con 0x7f4cec0ce350 2023-10-14T18:19:07.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.617+0000 7f4cf23f5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cec0cdec0 con 0x7f4cec0ae260 2023-10-14T18:19:07.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.617+0000 7f4cf23f5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0acdd0 con 0x7f4cec160ad0 2023-10-14T18:19:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4ceb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4cec0ce350 0x7f4cec160eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50794/0 (socket says 172.21.15.29:50794) 2023-10-14T18:19:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4ceb7fe640 1 -- 172.21.15.29:0/3597345798 learned_addr learned my addr 172.21.15.29:0/3597345798 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4cebfff640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cec0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224839140 0 0) 0x7f4cec0ce800 con 0x7f4cec0ce350 2023-10-14T18:19:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cd8003520 con 0x7f4cec0ce350 2023-10-14T18:19:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.618+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202890680 0 0) 0x7f4cec0cdec0 con 0x7f4cec0ae260 2023-10-14T18:19:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cec0ce800 con 0x7f4cec0ae260 2023-10-14T18:19:07.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1477606562 0 0) 0x7f4cd8003520 con 0x7f4cec0ce350 2023-10-14T18:19:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 msgr2=0x7f4cec0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cec0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 >> v1:172.21.15.186:6789/0 conn(0x7f4cec0ae260 legacy=0x7f4cec06a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec0ae740 con 0x7f4cec0ce350 2023-10-14T18:19:07.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4cd4001670 con 0x7f4cec0ce350 2023-10-14T18:19:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 >> v1:172.21.15.29:6789/0 conn(0x7f4cec0ce350 legacy=0x7f4cec160eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cd8003b30 0x7f4cd8003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cd8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0acdd0 con 0x7f4cd80047c0 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800ba30 con 0x7f4cd8003b30 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd800bbc0 con 0x7f4cec160ad0 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceb7fe640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4cf0992640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cd8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4cebfff640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cd8003b30 0x7f4cd8003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceb7fe640 1 -- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 msgr2=0x7f4cd8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceb7fe640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cec160ad0 0x7f4cd8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceb7fe640 1 -- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cd8003b30 msgr2=0x7f4cd8003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.619+0000 7f4ceb7fe640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cd8003b30 0x7f4cd8003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.620+0000 7f4ceb7fe640 1 -- 172.21.15.29:0/3597345798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd8003520 con 0x7f4cd80047c0 2023-10-14T18:19:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4ceb7fe640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0ce750 tx=0x7f4cd80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7484090e3ecec974 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd4005ad0 con 0x7f4cd80047c0 2023-10-14T18:19:07.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cd4005c50 con 0x7f4cd80047c0 2023-10-14T18:19:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4ceaffd640 1 -- 172.21.15.29:0/3597345798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd402c790 con 0x7f4cd80047c0 2023-10-14T18:19:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3597345798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd80047c0 msgr2=0x7f4cd8004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3597345798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cd80047c0 0x7f4cd8004c40 secure :-1 s=READY pgs=2989 cs=0 l=1 rev1=1 crypto rx=0x7f4cec0ce750 tx=0x7f4cd80057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.621+0000 7f4cf0992640 1 -- 172.21.15.29:0/3597345798 reap_dead start 2023-10-14T18:19:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.622+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3597345798 shutdown_connections 2023-10-14T18:19:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.622+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3597345798 >> 172.21.15.29:0/3597345798 conn(0x7f4cec06bc10 msgr2=0x7f4cec06c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.622+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3597345798 shutdown_connections 2023-10-14T18:19:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.622+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3597345798 wait complete. 2023-10-14T18:19:07.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.624+0000 7f4cf23f5640 1 Processor -- start 2023-10-14T18:19:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.625+0000 7f4cf23f5640 1 -- start start 2023-10-14T18:19:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cd80047c0 0x7f4cec0c72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cec06aef0 0x7f4cec1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.632 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0c8530 con 0x7f4cec0ccd60 2023-10-14T18:19:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0b2000 con 0x7f4cec06aef0 2023-10-14T18:19:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.626+0000 7f4cf23f5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cec0cd720 con 0x7f4cd80047c0 2023-10-14T18:19:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50008/0 (socket says 172.21.15.29:50008) 2023-10-14T18:19:07.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 -- 172.21.15.29:0/3554200893 learned_addr learned my addr 172.21.15.29:0/3554200893 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cebfff640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cd80047c0 0x7f4cec0c72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cd80047c0 msgr2=0x7f4cec0c72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cd80047c0 0x7f4cec0c72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cec06aef0 msgr2=0x7f4cec1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cec06aef0 0x7f4cec1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 -- 172.21.15.29:0/3554200893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cec156e20 con 0x7f4cec0ccd60 2023-10-14T18:19:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.627+0000 7f4cf0992640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f4cec160a30 tx=0x7f4ce400e820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.628+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40050d0 con 0x7f4cec0ccd60 2023-10-14T18:19:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.628+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ce4005230 con 0x7f4cec0ccd60 2023-10-14T18:19:07.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.628+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ce40069e0 con 0x7f4cec0ccd60 2023-10-14T18:19:07.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.628+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cec0c78b0 con 0x7f4cec0ccd60 2023-10-14T18:19:07.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.628+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cec0c7e70 con 0x7f4cec0ccd60 2023-10-14T18:19:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.629+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cec04e160 con 0x7f4cec0ccd60 2023-10-14T18:19:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.631+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f4ce4006b40 con 0x7f4cec0ccd60 2023-10-14T18:19:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.632+0000 7f4ce8ff9640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 0x7f4ccc06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:07.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.632+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7f4ce4089390 con 0x7f4cec0ccd60 2023-10-14T18:19:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.632+0000 7f4cebfff640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 msgr2=0x7f4ccc06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:07.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.632+0000 7f4cebfff640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 0x7f4ccc06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:07.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.641+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ce40575e0 con 0x7f4cec0ccd60 2023-10-14T18:19:07.651 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:07.651+0000 7fbf8d411040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:07.659 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.658+0000 7fb12a044040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:07.790 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:07.789+0000 7fb12a044040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.833+0000 7f4cebfff640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 msgr2=0x7f4ccc06d5a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:07.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.833+0000 7f4cebfff640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 0x7f4ccc06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.848+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f4cec06c950 con 0x7f4cec0ccd60 2023-10-14T18:19:07.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.849+0000 7f4ce8ff9640 1 -- 172.21.15.29:0/3554200893 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v255) v1 ==== 103+0+11 (secure 0 0 0) 0x7f4ce4007cd0 con 0x7f4cec0ccd60 2023-10-14T18:19:07.850 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 msgr2=0x7f4ccc06d5a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:07.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 0x7f4ccc06d5a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 msgr2=0x7f4cec0cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 secure :-1 s=READY pgs=2990 cs=0 l=1 rev1=1 crypto rx=0x7f4cec160a30 tx=0x7f4ce400e820 comp rx=0 tx=0).stop 2023-10-14T18:19:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 shutdown_connections 2023-10-14T18:19:07.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f4ccc06b0e0 0x7f4ccc06d5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4cec0ccd60 0x7f4cec0cd1e0 unknown :-1 s=CLOSED pgs=2990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4cec06aef0 0x7f4cec1568e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 --2- 172.21.15.29:0/3554200893 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4cd80047c0 0x7f4cec0c72d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:07.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 >> 172.21.15.29:0/3554200893 conn(0x7f4cec06bc10 msgr2=0x7f4cec0aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.851+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 shutdown_connections 2023-10-14T18:19:07.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:07.852+0000 7f4cf23f5640 1 -- 172.21.15.29:0/3554200893 wait complete. 2023-10-14T18:19:07.872 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T18:19:08.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.121+0000 7fe80aaf8640 1 Processor -- start 2023-10-14T18:19:08.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.121+0000 7fe80aaf8640 1 -- start start 2023-10-14T18:19:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80aaf8640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8040b2600 0x7fe8040ca8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80aaf8640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe804070240 0x7fe8040706d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80aaf8640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8040b2a60 con 0x7fe80415aaa0 2023-10-14T18:19:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80aaf8640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe804070c10 con 0x7fe8040b2600 2023-10-14T18:19:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80aaf8640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe8040b2d00 con 0x7fe804070240 2023-10-14T18:19:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe8092f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8040b2600 0x7fe8040ca8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe8092f5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe8040b2600 0x7fe8040ca8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50014/0 (socket says 172.21.15.29:50014) 2023-10-14T18:19:08.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe8092f5640 1 -- 172.21.15.29:0/2708697864 learned_addr learned my addr 172.21.15.29:0/2708697864 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:08.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.123+0000 7fe80a2f7640 1 --2- 172.21.15.29:0/2708697864 >> v2:172.21.15.186:3300/0 conn(0x7fe804070240 0x7fe8040706d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.124+0000 7fe8092f5640 1 -- 172.21.15.29:0/2708697864 >> v2:172.21.15.186:3300/0 conn(0x7fe804070240 msgr2=0x7fe8040706d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.124+0000 7fe8092f5640 1 --2- 172.21.15.29:0/2708697864 >> v2:172.21.15.186:3300/0 conn(0x7fe804070240 0x7fe8040706d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.124+0000 7fe8092f5640 1 -- 172.21.15.29:0/2708697864 >> v1:172.21.15.186:6789/0 conn(0x7fe80415aaa0 legacy=0x7fe8040b3da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.124+0000 7fe8092f5640 1 -- 172.21.15.29:0/2708697864 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8040b2fe0 con 0x7fe8040b2600 2023-10-14T18:19:08.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.125+0000 7fe8092f5640 1 --2- 172.21.15.29:0/2708697864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8040b2600 0x7fe8040ca8a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fe8040af1c0 tx=0x7fe7f402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c0899d34933ca4a9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.126+0000 7fe808af4640 1 -- 172.21.15.29:0/2708697864 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4031870 con 0x7fe8040b2600 2023-10-14T18:19:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.126+0000 7fe808af4640 1 -- 172.21.15.29:0/2708697864 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7f40319d0 con 0x7fe8040b2600 2023-10-14T18:19:08.128 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.126+0000 7fe808af4640 1 -- 172.21.15.29:0/2708697864 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7f4031d60 con 0x7fe8040b2600 2023-10-14T18:19:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 -- 172.21.15.29:0/2708697864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8040b2600 msgr2=0x7fe8040ca8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/2708697864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8040b2600 0x7fe8040ca8a0 secure :-1 s=READY pgs=2991 cs=0 l=1 rev1=1 crypto rx=0x7fe8040af1c0 tx=0x7fe7f402b910 comp rx=0 tx=0).stop 2023-10-14T18:19:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 -- 172.21.15.29:0/2708697864 shutdown_connections 2023-10-14T18:19:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/2708697864 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe8040b2600 0x7fe8040ca8a0 unknown :-1 s=CLOSED pgs=2991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/2708697864 >> v2:172.21.15.186:3300/0 conn(0x7fe804070240 0x7fe8040706d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 -- 172.21.15.29:0/2708697864 >> 172.21.15.29:0/2708697864 conn(0x7fe8040cc230 msgr2=0x7fe8040cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:08.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 -- 172.21.15.29:0/2708697864 shutdown_connections 2023-10-14T18:19:08.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.127+0000 7fe80aaf8640 1 -- 172.21.15.29:0/2708697864 wait complete. 2023-10-14T18:19:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.129+0000 7fe80aaf8640 1 Processor -- start 2023-10-14T18:19:08.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.129+0000 7fe80aaf8640 1 -- start start 2023-10-14T18:19:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.130+0000 7fe80aaf8640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe804070240 0x7fe80406bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.130+0000 7fe80aaf8640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8040b2600 0x7fe8041f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80aaf8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80aaf8640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8041597c0 con 0x7fe80415aaa0 2023-10-14T18:19:08.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80aaf8640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe80406aaa0 con 0x7fe804070240 2023-10-14T18:19:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80aaf8640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe80406ad80 con 0x7fe8040b2600 2023-10-14T18:19:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50018/0 (socket says 172.21.15.29:50018) 2023-10-14T18:19:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 -- 172.21.15.29:0/3748449149 learned_addr learned my addr 172.21.15.29:0/3748449149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:08.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe8092f5640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8040b2600 0x7fe8041f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe809af6640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe804070240 0x7fe80406bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8040b2600 msgr2=0x7fe8041f25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8040b2600 0x7fe8041f25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe804070240 msgr2=0x7fe80406bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe804070240 0x7fe80406bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.131+0000 7fe80a2f7640 1 -- 172.21.15.29:0/3748449149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f40094d0 con 0x7fe80415aaa0 2023-10-14T18:19:08.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe80a2f7640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00efc0 tx=0x7fe7fc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:08.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc009ba0 con 0x7fe80415aaa0 2023-10-14T18:19:08.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe7fc00f040 con 0x7fe80415aaa0 2023-10-14T18:19:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe80406b060 con 0x7fe80415aaa0 2023-10-14T18:19:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7fc0146c0 con 0x7fe80415aaa0 2023-10-14T18:19:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.132+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe80406b620 con 0x7fe80415aaa0 2023-10-14T18:19:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.134+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe80406b810 con 0x7fe80415aaa0 2023-10-14T18:19:08.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.136+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7fe7fc014820 con 0x7fe80415aaa0 2023-10-14T18:19:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.136+0000 7fe7fa7fc640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 0x7fe7e806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.137+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7fe7fc08ab00 con 0x7fe80415aaa0 2023-10-14T18:19:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.137+0000 7fe809af6640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 msgr2=0x7fe7e806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:08.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.137+0000 7fe809af6640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 0x7fe7e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:08.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.142+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe7fc08ce60 con 0x7fe80415aaa0 2023-10-14T18:19:08.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.338+0000 7fe809af6640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 msgr2=0x7fe7e806d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:08.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.338+0000 7fe809af6640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 0x7fe7e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:08.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.350+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fe80404e9b0 con 0x7fe80415aaa0 2023-10-14T18:19:08.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.351+0000 7fe7fa7fc640 1 -- 172.21.15.29:0/3748449149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v255) v1 ==== 81+0+583 (secure 0 0 0) 0x7fe7fc0551e0 con 0x7fe80415aaa0 2023-10-14T18:19:08.352 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/306027610 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.352 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/4275371095 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.352 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/4275371095 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.353 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2052138313 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.353 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3668953662 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.353 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/254602771 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.353 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4136574479 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.353 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3197818126 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.354 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/143268764 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.354 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3241859590 2023-10-15T18:19:02.726508+0000 2023-10-14T18:19:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2023-10-14T18:19:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 msgr2=0x7fe7e806d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:08.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 0x7fe7e806d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 msgr2=0x7fe8041f2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 secure :-1 s=READY pgs=2992 cs=0 l=1 rev1=1 crypto rx=0x7fe7fc00efc0 tx=0x7fe7fc00c490 comp rx=0 tx=0).stop 2023-10-14T18:19:08.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 shutdown_connections 2023-10-14T18:19:08.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7fe7e806b0b0 0x7fe7e806d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe80415aaa0 0x7fe8041f2b10 unknown :-1 s=CLOSED pgs=2992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe8040b2600 0x7fe8041f25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 --2- 172.21.15.29:0/3748449149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe804070240 0x7fe80406bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 >> 172.21.15.29:0/3748449149 conn(0x7fe8040cc230 msgr2=0x7fe8040acdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:08.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.353+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 shutdown_connections 2023-10-14T18:19:08.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.354+0000 7fe80aaf8640 1 -- 172.21.15.29:0/3748449149 wait complete. 2023-10-14T18:19:08.371 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T18:19:08.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:08.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:08.460 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:19:08.461 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:08.461 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T18:19:08.491 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:08.489+0000 7fbf8d411040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:08.506 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:08.507 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:08.564 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:08.564 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:08.565 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:08.622 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:19:08.622 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:08.623 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:08.651 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:08.650+0000 7fb12a044040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:08.679 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:08.680 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:08.804 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:08.803+0000 7fbf8d411040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:08.862 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:08.861+0000 7fbf8d411040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.887+0000 7f539ec61640 1 Processor -- start 2023-10-14T18:19:08.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.887+0000 7f539ec61640 1 -- start start 2023-10-14T18:19:08.889 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.888+0000 7f539ec61640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f539815ae80 0x7f539815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.888+0000 7f539ec61640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f53980ce390 0x7f5398070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.888+0000 7f539ec61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539815ed60 con 0x7f539815aaa0 2023-10-14T18:19:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.888+0000 7f539ec61640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539806b810 con 0x7f53980ce390 2023-10-14T18:19:08.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.888+0000 7f539ec61640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53980707e0 con 0x7f539815ae80 2023-10-14T18:19:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539c9fd640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f539815aaa0 0x7f53980cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50800/0 (socket says 172.21.15.29:50800) 2023-10-14T18:19:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539c9fd640 1 -- 172.21.15.29:0/2682561242 learned_addr learned my addr 172.21.15.29:0/2682561242 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:08.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539d1fe640 1 --2- 172.21.15.29:0/2682561242 >> v2:172.21.15.29:3300/0 conn(0x7f53980ce390 0x7f5398070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f5397fff640 1 --2- 172.21.15.29:0/2682561242 >> v2:172.21.15.186:3300/0 conn(0x7f539815ae80 0x7f539815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f53977fe640 1 -- 172.21.15.29:0/2682561242 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 426350937 0 0) 0x7f539815ed60 con 0x7f539815aaa0 2023-10-14T18:19:08.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f53977fe640 1 -- 172.21.15.29:0/2682561242 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5380003520 con 0x7f539815aaa0 2023-10-14T18:19:08.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539d1fe640 1 -- 172.21.15.29:0/2682561242 >> v2:172.21.15.186:3300/0 conn(0x7f539815ae80 msgr2=0x7f539815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:08.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539d1fe640 1 --2- 172.21.15.29:0/2682561242 >> v2:172.21.15.186:3300/0 conn(0x7f539815ae80 0x7f539815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539d1fe640 1 -- 172.21.15.29:0/2682561242 >> v1:172.21.15.29:6789/0 conn(0x7f539815aaa0 legacy=0x7f53980cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.889+0000 7f539d1fe640 1 -- 172.21.15.29:0/2682561242 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539815ef90 con 0x7f53980ce390 2023-10-14T18:19:08.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.891+0000 7f539d1fe640 1 --2- 172.21.15.29:0/2682561242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53980ce390 0x7f5398070f50 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f53980c59c0 tx=0x7f539002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=671c778edf6a2cb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.891+0000 7f53977fe640 1 -- 172.21.15.29:0/2682561242 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390031830 con 0x7f53980ce390 2023-10-14T18:19:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.891+0000 7f53977fe640 1 -- 172.21.15.29:0/2682561242 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5390031990 con 0x7f53980ce390 2023-10-14T18:19:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.891+0000 7f53977fe640 1 -- 172.21.15.29:0/2682561242 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5390031d20 con 0x7f53980ce390 2023-10-14T18:19:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 -- 172.21.15.29:0/2682561242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53980ce390 msgr2=0x7f5398070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 --2- 172.21.15.29:0/2682561242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53980ce390 0x7f5398070f50 secure :-1 s=READY pgs=2994 cs=0 l=1 rev1=1 crypto rx=0x7f53980c59c0 tx=0x7f539002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 -- 172.21.15.29:0/2682561242 shutdown_connections 2023-10-14T18:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 --2- 172.21.15.29:0/2682561242 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f53980ce390 0x7f5398070f50 unknown :-1 s=CLOSED pgs=2994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 --2- 172.21.15.29:0/2682561242 >> v2:172.21.15.186:3300/0 conn(0x7f539815ae80 0x7f539815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 -- 172.21.15.29:0/2682561242 >> 172.21.15.29:0/2682561242 conn(0x7f53980cb410 msgr2=0x7f53980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:08.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 -- 172.21.15.29:0/2682561242 shutdown_connections 2023-10-14T18:19:08.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.892+0000 7f539ec61640 1 -- 172.21.15.29:0/2682561242 wait complete. 2023-10-14T18:19:08.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.894+0000 7f539ec61640 1 Processor -- start 2023-10-14T18:19:08.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.894+0000 7f539ec61640 1 -- start start 2023-10-14T18:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 0x7f539814cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f539815ae80 0x7f53980b00a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53980aef50 con 0x7f539815aaa0 2023-10-14T18:19:08.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f539814d470 con 0x7f53980ce390 2023-10-14T18:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539ec61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53980af8b0 con 0x7f539815ae80 2023-10-14T18:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f5397fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f5397fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50034/0 (socket says 172.21.15.29:50034) 2023-10-14T18:19:08.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f5397fff640 1 -- 172.21.15.29:0/1126303481 learned_addr learned my addr 172.21.15.29:0/1126303481 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539d1fe640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f539815ae80 0x7f53980b00a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.895+0000 7f539c9fd640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 0x7f539814cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f539815ae80 msgr2=0x7f53980b00a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f539815ae80 0x7f53980b00a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.902 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 msgr2=0x7f539814cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:08.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 0x7f539814cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:08.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 -- 172.21.15.29:0/1126303481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5390009400 con 0x7f539815aaa0 2023-10-14T18:19:08.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f539c9fd640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 0x7f539814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:08.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f5397fff640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f538c00d5c0 tx=0x7f538c00da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:08.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c0125f0 con 0x7f539815aaa0 2023-10-14T18:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c012750 con 0x7f539815aaa0 2023-10-14T18:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f538c012b00 con 0x7f539815aaa0 2023-10-14T18:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53980ae2d0 con 0x7f539815aaa0 2023-10-14T18:19:08.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.896+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53980ae890 con 0x7f539815aaa0 2023-10-14T18:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.897+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53980af090 con 0x7f539815aaa0 2023-10-14T18:19:08.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.899+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 151) v1 ==== 89434+0+0 (secure 0 0 0) 0x7f538c012c60 con 0x7f539815aaa0 2023-10-14T18:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.901+0000 7f53957fa640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 0x7f538406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.901+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(255..255 src has 1..255) v4 ==== 7119+0+0 (secure 0 0 0) 0x7f538c08a5b0 con 0x7f539815aaa0 2023-10-14T18:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.901+0000 7f539c9fd640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 msgr2=0x7f538406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:08.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.901+0000 7f539c9fd640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 0x7f538406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:08.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:08.905+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f53980af090 con 0x7f539815aaa0 2023-10-14T18:19:08.967 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:08.966+0000 7fb12a044040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:08.977 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:08.976+0000 7fbf8d411040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:09.027 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.026+0000 7fb12a044040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:09.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.102+0000 7f539c9fd640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 msgr2=0x7f538406d570 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1194987085 2023-10-14T18:19:09.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.102+0000 7f539c9fd640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 0x7f538406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:09.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.140+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53980af090 con 0x7f539815aaa0 2023-10-14T18:19:09.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.141+0000 7f53957fa640 1 -- 172.21.15.29:0/1126303481 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7f53980af090 con 0x7f539815aaa0 2023-10-14T18:19:09.142 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:09.142 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:09.143 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:19:09.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 msgr2=0x7f538406d570 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:09.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 0x7f538406d570 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 msgr2=0x7f53980af370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 secure :-1 s=READY pgs=2995 cs=0 l=1 rev1=1 crypto rx=0x7f538c00d5c0 tx=0x7f538c00da80 comp rx=0 tx=0).stop 2023-10-14T18:19:09.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 shutdown_connections 2023-10-14T18:19:09.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:6832/1194987085,v1:172.21.15.29:6833/1194987085] conn(0x7f538406b0b0 0x7f538406d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f539815ae80 0x7f53980b00a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f539815aaa0 0x7f53980af370 unknown :-1 s=CLOSED pgs=2995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.143+0000 7f539ec61640 1 --2- 172.21.15.29:0/1126303481 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f53980ce390 0x7f539814cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.144+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 >> 172.21.15.29:0/1126303481 conn(0x7f53980cb410 msgr2=0x7f539815a070 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:09.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.144+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 shutdown_connections 2023-10-14T18:19:09.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.144+0000 7f539ec61640 1 -- 172.21.15.29:0/1126303481 wait complete. 2023-10-14T18:19:09.147 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.144+0000 7fb12a044040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:09.170 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:09.237 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.237 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.238 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.236+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.254+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.254+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.255 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.254+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.256 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.254+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.256 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.254+0000 7fb0aeb91640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.255+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.255+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.255+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.255+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.257 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:09.255+0000 7fb0ac38c640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:09.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.358+0000 7f0da8e5d640 1 Processor -- start 2023-10-14T18:19:09.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.359+0000 7f0da8e5d640 1 -- start start 2023-10-14T18:19:09.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da8e5d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0da40b0b20 0x7f0da4164b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da8e5d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da406f7f0 con 0x7f0da4159b20 2023-10-14T18:19:09.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da8e5d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0da406c5e0 con 0x7f0da4159f00 2023-10-14T18:19:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da8e5d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0da406c8c0 con 0x7f0da40b0b20 2023-10-14T18:19:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da259c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0da4159b20 0x7f0da406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50804/0 (socket says 172.21.15.29:50804) 2023-10-14T18:19:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da259c640 1 -- 172.21.15.29:0/2594824632 learned_addr learned my addr 172.21.15.29:0/2594824632 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:09.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.360+0000 7f0da2d9d640 1 --2- 172.21.15.29:0/2594824632 >> v2:172.21.15.29:3300/0 conn(0x7f0da40b0b20 0x7f0da4164b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.361+0000 7f0da159a640 1 -- 172.21.15.29:0/2594824632 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2858852122 0 0) 0x7f0da406f7f0 con 0x7f0da4159b20 2023-10-14T18:19:09.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.361+0000 7f0da159a640 1 -- 172.21.15.29:0/2594824632 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d90003520 con 0x7f0da4159b20 2023-10-14T18:19:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.361+0000 7f0da2d9d640 1 -- 172.21.15.29:0/2594824632 >> v1:172.21.15.186:6789/0 conn(0x7f0da4159f00 legacy=0x7f0da40b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.361+0000 7f0da2d9d640 1 -- 172.21.15.29:0/2594824632 >> v1:172.21.15.29:6789/0 conn(0x7f0da4159b20 legacy=0x7f0da406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.361+0000 7f0da2d9d640 1 -- 172.21.15.29:0/2594824632 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0da406cba0 con 0x7f0da40b0b20 2023-10-14T18:19:09.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.362+0000 7f0da2d9d640 1 --2- 172.21.15.29:0/2594824632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da40b0b20 0x7f0da4164b40 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009d90 tx=0x7f0d9c02beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b03fc991169838e4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.362+0000 7f0da159a640 1 -- 172.21.15.29:0/2594824632 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c02cc60 con 0x7f0da40b0b20 2023-10-14T18:19:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.362+0000 7f0da159a640 1 -- 172.21.15.29:0/2594824632 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d9c02cdc0 con 0x7f0da40b0b20 2023-10-14T18:19:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.362+0000 7f0da159a640 1 -- 172.21.15.29:0/2594824632 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d9c034640 con 0x7f0da40b0b20 2023-10-14T18:19:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 -- 172.21.15.29:0/2594824632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da40b0b20 msgr2=0x7f0da4164b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/2594824632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da40b0b20 0x7f0da4164b40 secure :-1 s=READY pgs=3012 cs=0 l=1 rev1=1 crypto rx=0x7f0d9c009d90 tx=0x7f0d9c02beb0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 -- 172.21.15.29:0/2594824632 shutdown_connections 2023-10-14T18:19:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/2594824632 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da40b0b20 0x7f0da4164b40 unknown :-1 s=CLOSED pgs=3012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 -- 172.21.15.29:0/2594824632 >> 172.21.15.29:0/2594824632 conn(0x7f0da40cb410 msgr2=0x7f0da40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 -- 172.21.15.29:0/2594824632 shutdown_connections 2023-10-14T18:19:09.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.363+0000 7f0da8e5d640 1 -- 172.21.15.29:0/2594824632 wait complete. 2023-10-14T18:19:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.365+0000 7f0da8e5d640 1 Processor -- start 2023-10-14T18:19:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.365+0000 7f0da8e5d640 1 -- start start 2023-10-14T18:19:09.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 0x7f0da4154d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4159f00 0x7f0da415b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40c3e00 con 0x7f0da4159b20 2023-10-14T18:19:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40c3810 con 0x7f0da4159f00 2023-10-14T18:19:09.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da8e5d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0da40c5a60 con 0x7f0da40b0b20 2023-10-14T18:19:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da1d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50180/0 (socket says 172.21.15.29:50180) 2023-10-14T18:19:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da1d9b640 1 -- 172.21.15.29:0/3319827838 learned_addr learned my addr 172.21.15.29:0/3319827838 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:09.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.366+0000 7f0da2d9d640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4159f00 0x7f0da415b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da259c640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 0x7f0da4154d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 -- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 msgr2=0x7f0da4154d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 0x7f0da4154d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 -- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4159f00 msgr2=0x7f0da415b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4159f00 0x7f0da415b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 -- 172.21.15.29:0/3319827838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c009a70 con 0x7f0da4159b20 2023-10-14T18:19:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da259c640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 0x7f0da4154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da1d9b640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400c400 tx=0x7f0d9400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94011790 con 0x7f0da4159b20 2023-10-14T18:19:09.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d94011930 con 0x7f0da4159b20 2023-10-14T18:19:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d94011ce0 con 0x7f0da4159b20 2023-10-14T18:19:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da41555d0 con 0x7f0da4159b20 2023-10-14T18:19:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.367+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da40c3a20 con 0x7f0da4159b20 2023-10-14T18:19:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.368+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da40c3f40 con 0x7f0da4159b20 2023-10-14T18:19:09.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.369+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f0d9401c460 con 0x7f0da4159b20 2023-10-14T18:19:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.371+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f0d940898a0 con 0x7f0da4159b20 2023-10-14T18:19:09.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.375+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0d940544f0 con 0x7f0da4159b20 2023-10-14T18:19:09.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.577+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0da40c3bb0 con 0x7f0da4159b20 2023-10-14T18:19:09.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.579+0000 7f0d9affd640 1 -- 172.21.15.29:0/3319827838 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+21608 (secure 0 0 0) 0x7f0d94057b10 con 0x7f0da4159b20 2023-10-14T18:19:09.581 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:09.581 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":256,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:09.147522+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:09.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 msgr2=0x7f0da415afc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 secure :-1 s=READY pgs=3013 cs=0 l=1 rev1=1 crypto rx=0x7f0d9400c400 tx=0x7f0d9400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 shutdown_connections 2023-10-14T18:19:09.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0da4159f00 0x7f0da415b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0da4159b20 0x7f0da415afc0 unknown :-1 s=CLOSED pgs=3013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 --2- 172.21.15.29:0/3319827838 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0da40b0b20 0x7f0da4154d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.582+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 >> 172.21.15.29:0/3319827838 conn(0x7f0da40cb410 msgr2=0x7f0da40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:09.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.583+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 shutdown_connections 2023-10-14T18:19:09.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.583+0000 7f0da8e5d640 1 -- 172.21.15.29:0/3319827838 wait complete. 2023-10-14T18:19:09.602 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 30, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:18:33.087026+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '249', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 1, 'raw_score_acting': 2.5, 'raw_score_stable': 2.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 31, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:18:34.140901+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '250', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:18:47.629053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:18:48.796314+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:09.603 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:09.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.829+0000 7fbda6878640 1 Processor -- start 2023-10-14T18:19:09.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.829+0000 7fbda6878640 1 -- start start 2023-10-14T18:19:09.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda015cd00 0x7fbda00cdfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbda00ce520 0x7fbda00cb000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbda00cb540 0x7fbda0070380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda015d0e0 con 0x7fbda00ce520 2023-10-14T18:19:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbda00cbd80 con 0x7fbda00cb540 2023-10-14T18:19:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda6878640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00cbaa0 con 0x7fbda015cd00 2023-10-14T18:19:09.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.831+0000 7fbda5075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbda00ce520 0x7fbda00cb000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbda00ce520 0x7fbda00cb000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50190/0 (socket says 172.21.15.29:50190) 2023-10-14T18:19:09.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 -- 172.21.15.29:0/2523278797 learned_addr learned my addr 172.21.15.29:0/2523278797 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:09.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 -- 172.21.15.29:0/2523278797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda015cd00 msgr2=0x7fbda00cdfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5876640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda015cd00 0x7fbda00cdfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda015cd00 0x7fbda00cdfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 -- 172.21.15.29:0/2523278797 >> v2:172.21.15.186:3300/0 conn(0x7fbda00cb540 msgr2=0x7fbda0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 --2- 172.21.15.29:0/2523278797 >> v2:172.21.15.186:3300/0 conn(0x7fbda00cb540 0x7fbda0070380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.832+0000 7fbda5075640 1 -- 172.21.15.29:0/2523278797 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbda00cd500 con 0x7fbda00ce520 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.833+0000 7fbda5075640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00cb000 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fbda0070140 tx=0x7fbd9002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b11df03f049a7214 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda4874640 1 -- 172.21.15.29:0/2523278797 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90031870 con 0x7fbda00ce520 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda4874640 1 -- 172.21.15.29:0/2523278797 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd900319d0 con 0x7fbda00ce520 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda4874640 1 -- 172.21.15.29:0/2523278797 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90031d60 con 0x7fbda00ce520 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda6878640 1 -- 172.21.15.29:0/2523278797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 msgr2=0x7fbda00cb000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda6878640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00cb000 secure :-1 s=READY pgs=3014 cs=0 l=1 rev1=1 crypto rx=0x7fbda0070140 tx=0x7fbd9002b910 comp rx=0 tx=0).stop 2023-10-14T18:19:09.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.834+0000 7fbda6878640 1 -- 172.21.15.29:0/2523278797 shutdown_connections 2023-10-14T18:19:09.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 --2- 172.21.15.29:0/2523278797 >> v2:172.21.15.186:3300/0 conn(0x7fbda00cb540 0x7fbda0070380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00cb000 unknown :-1 s=CLOSED pgs=3014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 --2- 172.21.15.29:0/2523278797 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda015cd00 0x7fbda00cdfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 -- 172.21.15.29:0/2523278797 >> 172.21.15.29:0/2523278797 conn(0x7fbda00cf790 msgr2=0x7fbda00cfba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:09.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 -- 172.21.15.29:0/2523278797 shutdown_connections 2023-10-14T18:19:09.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.835+0000 7fbda6878640 1 -- 172.21.15.29:0/2523278797 wait complete. 2023-10-14T18:19:09.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.836+0000 7fbda6878640 1 Processor -- start 2023-10-14T18:19:09.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.837+0000 7fbda6878640 1 -- start start 2023-10-14T18:19:09.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda00cb540 0x7fbda006da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 0x7fbda00ca200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:09.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c9de0 con 0x7fbda00ce520 2023-10-14T18:19:09.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c8e90 con 0x7fbda015cd00 2023-10-14T18:19:09.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6878640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbda00c9160 con 0x7fbda00cb540 2023-10-14T18:19:09.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50206/0 (socket says 172.21.15.29:50206) 2023-10-14T18:19:09.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 -- 172.21.15.29:0/1753648747 learned_addr learned my addr 172.21.15.29:0/1753648747 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:09.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5876640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda00cb540 0x7fbda006da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6077640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 0x7fbda00ca200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 -- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda00cb540 msgr2=0x7fbda006da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda00cb540 0x7fbda006da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 -- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 msgr2=0x7fbda00ca200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 0x7fbda00ca200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda5075640 1 -- 172.21.15.29:0/1753648747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd900094d0 con 0x7fbda00ce520 2023-10-14T18:19:09.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.838+0000 7fbda6077640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 0x7fbda00ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:09.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbda5075640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7fbd9002b950 tx=0x7fbd900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:09.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd90031860 con 0x7fbda00ce520 2023-10-14T18:19:09.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd900443f0 con 0x7fbda00ce520 2023-10-14T18:19:09.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbda00c9440 con 0x7fbda00ce520 2023-10-14T18:19:09.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd900447a0 con 0x7fbda00ce520 2023-10-14T18:19:09.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.839+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbda00c9a00 con 0x7fbda00ce520 2023-10-14T18:19:09.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.840+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbda00c9f20 con 0x7fbda00ce520 2023-10-14T18:19:09.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.842+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 152) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fbd9004d450 con 0x7fbda00ce520 2023-10-14T18:19:09.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.843+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7fbd9002d040 con 0x7fbda00ce520 2023-10-14T18:19:09.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:09.848+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbda00c9f20 con 0x7fbda00ce520 2023-10-14T18:19:10.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.044+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbda00c9f20 con 0x7fbda00ce520 2023-10-14T18:19:10.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.045+0000 7fbd967fc640 1 -- 172.21.15.29:0/1753648747 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v256) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbda00c9f20 con 0x7fbda00ce520 2023-10-14T18:19:10.046 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:10.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 msgr2=0x7fbda00c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 secure :-1 s=READY pgs=3015 cs=0 l=1 rev1=1 crypto rx=0x7fbd9002b950 tx=0x7fbd900046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 shutdown_connections 2023-10-14T18:19:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbda015cd00 0x7fbda00ca200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbda00ce520 0x7fbda00c8950 unknown :-1 s=CLOSED pgs=3015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 --2- 172.21.15.29:0/1753648747 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbda00cb540 0x7fbda006da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 >> 172.21.15.29:0/1753648747 conn(0x7fbda00cf790 msgr2=0x7fbda015fa40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 shutdown_connections 2023-10-14T18:19:10.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.048+0000 7fbda6878640 1 -- 172.21.15.29:0/1753648747 wait complete. 2023-10-14T18:19:10.069 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:19:10.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.309+0000 7f4a7f59e640 1 Processor -- start 2023-10-14T18:19:10.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.310+0000 7f4a7f59e640 1 -- start start 2023-10-14T18:19:10.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.311+0000 7f4a7f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 0x7f4a8015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.311+0000 7f4a7f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a8006c810 0x7f4a800708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.311+0000 7f4a7f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a80070df0 con 0x7f4a80070450 2023-10-14T18:19:10.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.311+0000 7f4a7f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4a8015e490 con 0x7f4a8006c810 2023-10-14T18:19:10.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.311+0000 7f4a7f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8015aef0 con 0x7f4a80159950 2023-10-14T18:19:10.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.312+0000 7f4a7dd9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f4a80070450 0x7f4a8006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:35388/0 (socket says 172.21.15.29:35388) 2023-10-14T18:19:10.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.312+0000 7f4a7e59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 0x7f4a8015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.312+0000 7f4a7dd9b640 1 -- 172.21.15.29:0/3788581930 learned_addr learned my addr 172.21.15.29:0/3788581930 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:10.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.312+0000 7f4a7ed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4a8006c810 0x7f4a800708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7d59a640 1 -- 172.21.15.29:0/3788581930 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819592769 0 0) 0x7f4a80070df0 con 0x7f4a80070450 2023-10-14T18:19:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7d59a640 1 -- 172.21.15.29:0/3788581930 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a6c003520 con 0x7f4a80070450 2023-10-14T18:19:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7e59c640 1 -- 172.21.15.29:0/3788581930 >> v2:172.21.15.186:3300/0 conn(0x7f4a8006c810 msgr2=0x7f4a800708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:10.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7e59c640 1 --2- 172.21.15.29:0/3788581930 >> v2:172.21.15.186:3300/0 conn(0x7f4a8006c810 0x7f4a800708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7e59c640 1 -- 172.21.15.29:0/3788581930 >> v1:172.21.15.186:6789/0 conn(0x7f4a80070450 legacy=0x7f4a8006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.313+0000 7f4a7e59c640 1 -- 172.21.15.29:0/3788581930 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a8006cc80 con 0x7f4a80159950 2023-10-14T18:19:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.315+0000 7f4a7e59c640 1 --2- 172.21.15.29:0/3788581930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 0x7f4a8015a9b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4a70009d90 tx=0x7f4a7002be20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21c9b4f77be99c00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.315+0000 7f4a7d59a640 1 -- 172.21.15.29:0/3788581930 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7002cbd0 con 0x7f4a80159950 2023-10-14T18:19:10.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.315+0000 7f4a7d59a640 1 -- 172.21.15.29:0/3788581930 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7002cd30 con 0x7f4a80159950 2023-10-14T18:19:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.315+0000 7f4a7d59a640 1 -- 172.21.15.29:0/3788581930 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7002cea0 con 0x7f4a80159950 2023-10-14T18:19:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3788581930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 msgr2=0x7f4a8015a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3788581930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 0x7f4a8015a9b0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f4a70009d90 tx=0x7f4a7002be20 comp rx=0 tx=0).stop 2023-10-14T18:19:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3788581930 shutdown_connections 2023-10-14T18:19:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3788581930 >> v2:172.21.15.186:3300/0 conn(0x7f4a8006c810 0x7f4a800708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3788581930 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a80159950 0x7f4a8015a9b0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3788581930 >> 172.21.15.29:0/3788581930 conn(0x7f4a800ce6a0 msgr2=0x7f4a800ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:10.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3788581930 shutdown_connections 2023-10-14T18:19:10.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.316+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3788581930 wait complete. 2023-10-14T18:19:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.318+0000 7f4a7f59e640 1 Processor -- start 2023-10-14T18:19:10.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.318+0000 7f4a7f59e640 1 -- start start 2023-10-14T18:19:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a8006c810 0x7f4a8015b090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a80070450 0x7f4a8006ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a800ae8b0 con 0x7f4a80159950 2023-10-14T18:19:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8015bb50 con 0x7f4a80070450 2023-10-14T18:19:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.319+0000 7f4a7f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a8015b5d0 con 0x7f4a8006c810 2023-10-14T18:19:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50214/0 (socket says 172.21.15.29:50214) 2023-10-14T18:19:10.323 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 -- 172.21.15.29:0/3415988434 learned_addr learned my addr 172.21.15.29:0/3415988434 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7e59c640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a8006c810 0x7f4a8015b090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7dd9b640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a80070450 0x7f4a8006ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 -- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a8006c810 msgr2=0x7f4a8015b090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a8006c810 0x7f4a8015b090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 -- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a80070450 msgr2=0x7f4a8006ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a80070450 0x7f4a8006ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.320+0000 7f4a7ed9d640 1 -- 172.21.15.29:0/3415988434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a70009a70 con 0x7f4a80159950 2023-10-14T18:19:10.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a7ed9d640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800e820 tx=0x7f4a7800ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a78009510 con 0x7f4a80159950 2023-10-14T18:19:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a7800ee70 con 0x7f4a80159950 2023-10-14T18:19:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a7800cb80 con 0x7f4a80159950 2023-10-14T18:19:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a8015b870 con 0x7f4a80159950 2023-10-14T18:19:10.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.321+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a800ae4d0 con 0x7f4a80159950 2023-10-14T18:19:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.322+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a800ae9f0 con 0x7f4a80159950 2023-10-14T18:19:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.323+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f4a7800cce0 con 0x7f4a80159950 2023-10-14T18:19:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.324+0000 7f4a76ffd640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 0x7f4a6006d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.324+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f4a78089a70 con 0x7f4a80159950 2023-10-14T18:19:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.325+0000 7f4a7e59c640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 0x7f4a6006d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.325+0000 7f4a7e59c640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 0x7f4a6006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a70036000 tx=0x7f4a70030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.329+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4a800ae9f0 con 0x7f4a80159950 2023-10-14T18:19:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.528+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4a800ae9f0 con 0x7f4a80159950 2023-10-14T18:19:10.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.528+0000 7f4a76ffd640 1 -- 172.21.15.29:0/3415988434 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v256) v1 ==== 104+0+11 (secure 0 0 0) 0x7f4a800ae9f0 con 0x7f4a80159950 2023-10-14T18:19:10.530 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:10.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 msgr2=0x7f4a6006d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 0x7f4a6006d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4a70036000 tx=0x7f4a70030000 comp rx=0 tx=0).stop 2023-10-14T18:19:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 msgr2=0x7f4a8006b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 secure :-1 s=READY pgs=3016 cs=0 l=1 rev1=1 crypto rx=0x7f4a7800e820 tx=0x7f4a7800ec20 comp rx=0 tx=0).stop 2023-10-14T18:19:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 shutdown_connections 2023-10-14T18:19:10.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4a6006acc0 0x7f4a6006d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4a80159950 0x7f4a8006b2e0 unknown :-1 s=CLOSED pgs=3016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4a80070450 0x7f4a8006ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 --2- 172.21.15.29:0/3415988434 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4a8006c810 0x7f4a8015b090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.531+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 >> 172.21.15.29:0/3415988434 conn(0x7f4a800ce6a0 msgr2=0x7f4a800cdb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.532+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 shutdown_connections 2023-10-14T18:19:10.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.532+0000 7f4a7f59e640 1 -- 172.21.15.29:0/3415988434 wait complete. 2023-10-14T18:19:10.551 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:19:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.778+0000 7f5b75ed1640 1 Processor -- start 2023-10-14T18:19:10.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.778+0000 7f5b75ed1640 1 -- start start 2023-10-14T18:19:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b75ed1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b700b8480 0x7f5b700cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b75ed1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5b700d0be0 0x7f5b700cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b75ed1640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7015d150 con 0x7f5b7015cd70 2023-10-14T18:19:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b75ed1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b700d1180 con 0x7f5b700d0be0 2023-10-14T18:19:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b75ed1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b700d1460 con 0x7f5b700b8480 2023-10-14T18:19:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b74ecf640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5b7015cd70 0x7f5b700d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50818/0 (socket says 172.21.15.29:50818) 2023-10-14T18:19:10.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.781+0000 7f5b74ecf640 1 -- 172.21.15.29:0/2454511888 learned_addr learned my addr 172.21.15.29:0/2454511888 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b756d0640 1 --2- 172.21.15.29:0/2454511888 >> v2:172.21.15.29:3300/0 conn(0x7f5b700d0be0 0x7f5b700cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b67fff640 1 --2- 172.21.15.29:0/2454511888 >> v2:172.21.15.186:3300/0 conn(0x7f5b700b8480 0x7f5b700cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b756d0640 1 -- 172.21.15.29:0/2454511888 >> v2:172.21.15.186:3300/0 conn(0x7f5b700b8480 msgr2=0x7f5b700cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b756d0640 1 --2- 172.21.15.29:0/2454511888 >> v2:172.21.15.186:3300/0 conn(0x7f5b700b8480 0x7f5b700cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b756d0640 1 -- 172.21.15.29:0/2454511888 >> v1:172.21.15.29:6789/0 conn(0x7f5b7015cd70 legacy=0x7f5b700d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b756d0640 1 -- 172.21.15.29:0/2454511888 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b700d1740 con 0x7f5b700d0be0 2023-10-14T18:19:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.782+0000 7f5b677fe640 1 -- 172.21.15.29:0/2454511888 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3155222772 0 0) 0x7f5b7015d150 con 0x7f5b7015cd70 2023-10-14T18:19:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.783+0000 7f5b756d0640 1 --2- 172.21.15.29:0/2454511888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b700d0be0 0x7f5b700cdd00 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f5b70066a00 tx=0x7f5b6802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8b3ee0756d31d42 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.784+0000 7f5b677fe640 1 -- 172.21.15.29:0/2454511888 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6802e070 con 0x7f5b700d0be0 2023-10-14T18:19:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.784+0000 7f5b677fe640 1 -- 172.21.15.29:0/2454511888 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b680315a0 con 0x7f5b700d0be0 2023-10-14T18:19:10.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.784+0000 7f5b677fe640 1 -- 172.21.15.29:0/2454511888 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68031930 con 0x7f5b700d0be0 2023-10-14T18:19:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.784+0000 7f5b75ed1640 1 -- 172.21.15.29:0/2454511888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b700d0be0 msgr2=0x7f5b700cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.784+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/2454511888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b700d0be0 0x7f5b700cdd00 secure :-1 s=READY pgs=3018 cs=0 l=1 rev1=1 crypto rx=0x7f5b70066a00 tx=0x7f5b6802b840 comp rx=0 tx=0).stop 2023-10-14T18:19:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 -- 172.21.15.29:0/2454511888 shutdown_connections 2023-10-14T18:19:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/2454511888 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b700d0be0 0x7f5b700cdd00 unknown :-1 s=CLOSED pgs=3018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/2454511888 >> v2:172.21.15.186:3300/0 conn(0x7f5b700b8480 0x7f5b700cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 -- 172.21.15.29:0/2454511888 >> 172.21.15.29:0/2454511888 conn(0x7f5b700cf570 msgr2=0x7f5b700cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 -- 172.21.15.29:0/2454511888 shutdown_connections 2023-10-14T18:19:10.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.785+0000 7f5b75ed1640 1 -- 172.21.15.29:0/2454511888 wait complete. 2023-10-14T18:19:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.786+0000 7f5b75ed1640 1 Processor -- start 2023-10-14T18:19:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.787+0000 7f5b75ed1640 1 -- start start 2023-10-14T18:19:10.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.787+0000 7f5b75ed1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b700b8480 0x7f5b701f5a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.787+0000 7f5b75ed1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b700d0be0 0x7f5b701588d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b75ed1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b75ed1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700cacb0 con 0x7f5b7015cd70 2023-10-14T18:19:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b75ed1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b70159680 con 0x7f5b700b8480 2023-10-14T18:19:10.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b75ed1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b700caf20 con 0x7f5b700d0be0 2023-10-14T18:19:10.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50230/0 (socket says 172.21.15.29:50230) 2023-10-14T18:19:10.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 -- 172.21.15.29:0/1400418697 learned_addr learned my addr 172.21.15.29:0/1400418697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:10.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b67fff640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b700d0be0 0x7f5b701588d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b74ecf640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b700b8480 0x7f5b701f5a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 -- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b700d0be0 msgr2=0x7f5b701588d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b700d0be0 0x7f5b701588d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 -- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b700b8480 msgr2=0x7f5b701f5a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:10.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b700b8480 0x7f5b701f5a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:10.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.788+0000 7f5b756d0640 1 -- 172.21.15.29:0/1400418697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b68009400 con 0x7f5b7015cd70 2023-10-14T18:19:10.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b756d0640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009890 tx=0x7f5b680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68032840 con 0x7f5b7015cd70 2023-10-14T18:19:10.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b680329a0 con 0x7f5b7015cd70 2023-10-14T18:19:10.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b68032d30 con 0x7f5b7015cd70 2023-10-14T18:19:10.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b700cb200 con 0x7f5b7015cd70 2023-10-14T18:19:10.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.789+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b70159960 con 0x7f5b7015cd70 2023-10-14T18:19:10.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.791+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7004e9b0 con 0x7f5b7015cd70 2023-10-14T18:19:10.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.791+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f5b6804e430 con 0x7f5b7015cd70 2023-10-14T18:19:10.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.792+0000 7f5b657fa640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 0x7f5b5c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:10.797 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.792+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f5b680ab800 con 0x7f5b7015cd70 2023-10-14T18:19:10.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.792+0000 7f5b74ecf640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 0x7f5b5c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:10.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.793+0000 7f5b74ecf640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 0x7f5b5c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0012e0 tx=0x7f5b6c009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:10.798 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.797+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5b680764c0 con 0x7f5b7015cd70 2023-10-14T18:19:10.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.996+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5b700b6e70 con 0x7f5b7015cd70 2023-10-14T18:19:10.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.996+0000 7f5b657fa640 1 -- 172.21.15.29:0/1400418697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v256) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5b68079ae0 con 0x7f5b7015cd70 2023-10-14T18:19:10.997 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:11.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 msgr2=0x7f5b5c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 0x7f5b5c06d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f5b6c0012e0 tx=0x7f5b6c009040 comp rx=0 tx=0).stop 2023-10-14T18:19:11.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 msgr2=0x7f5b70158e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 secure :-1 s=READY pgs=3019 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009890 tx=0x7f5b680046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 shutdown_connections 2023-10-14T18:19:11.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b5c06acc0 0x7f5b5c06d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b7015cd70 0x7f5b70158e10 unknown :-1 s=CLOSED pgs=3019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b700d0be0 0x7f5b701588d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 --2- 172.21.15.29:0/1400418697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b700b8480 0x7f5b701f5a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:10.999+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 >> 172.21.15.29:0/1400418697 conn(0x7f5b700cf570 msgr2=0x7f5b700cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:11.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.000+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 shutdown_connections 2023-10-14T18:19:11.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.000+0000 7f5b75ed1640 1 -- 172.21.15.29:0/1400418697 wait complete. 2023-10-14T18:19:11.020 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:19:11.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.252+0000 7f0e96261640 1 Processor -- start 2023-10-14T18:19:11.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.253+0000 7f0e96261640 1 -- start start 2023-10-14T18:19:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.254+0000 7f0e96261640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e901609f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.254+0000 7f0e96261640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0e90065ab0 0x7f0e90160f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.254+0000 7f0e96261640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e900ac890 con 0x7f0e90161470 2023-10-14T18:19:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.254+0000 7f0e96261640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0e900ae800 con 0x7f0e90065ab0 2023-10-14T18:19:11.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.254+0000 7f0e96261640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e900ae1f0 con 0x7f0e90160530 2023-10-14T18:19:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f0e90161470 0x7f0e900ccd80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58420/0 (socket says 172.21.15.29:58420) 2023-10-14T18:19:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8ffff640 1 -- 172.21.15.29:0/3898543030 learned_addr learned my addr 172.21.15.29:0/3898543030 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:11.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8effd640 1 --2- 172.21.15.29:0/3898543030 >> v2:172.21.15.29:3300/0 conn(0x7f0e90065ab0 0x7f0e90160f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8f7fe640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e901609f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8e7fc640 1 -- 172.21.15.29:0/3898543030 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325951398 0 0) 0x7f0e900ac890 con 0x7f0e90161470 2023-10-14T18:19:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8e7fc640 1 -- 172.21.15.29:0/3898543030 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e78003520 con 0x7f0e90161470 2023-10-14T18:19:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8effd640 1 -- 172.21.15.29:0/3898543030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 msgr2=0x7f0e901609f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:11.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8effd640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e901609f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8effd640 1 -- 172.21.15.29:0/3898543030 >> v1:172.21.15.29:6789/0 conn(0x7f0e90161470 legacy=0x7f0e900ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8effd640 1 -- 172.21.15.29:0/3898543030 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e900acae0 con 0x7f0e90065ab0 2023-10-14T18:19:11.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.255+0000 7f0e8f7fe640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e901609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:11.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.257+0000 7f0e8effd640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e90160f30 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7f0e840097a0 tx=0x7f0e8402b820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60d993b4557b799e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.257+0000 7f0e8e7fc640 1 -- 172.21.15.29:0/3898543030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84031780 con 0x7f0e90065ab0 2023-10-14T18:19:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.257+0000 7f0e8e7fc640 1 -- 172.21.15.29:0/3898543030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e840318e0 con 0x7f0e90065ab0 2023-10-14T18:19:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.257+0000 7f0e8e7fc640 1 -- 172.21.15.29:0/3898543030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e84031c50 con 0x7f0e90065ab0 2023-10-14T18:19:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 -- 172.21.15.29:0/3898543030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 msgr2=0x7f0e90160f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e90160f30 secure :-1 s=READY pgs=3021 cs=0 l=1 rev1=1 crypto rx=0x7f0e840097a0 tx=0x7f0e8402b820 comp rx=0 tx=0).stop 2023-10-14T18:19:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 -- 172.21.15.29:0/3898543030 shutdown_connections 2023-10-14T18:19:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e90160f30 unknown :-1 s=CLOSED pgs=3021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 --2- 172.21.15.29:0/3898543030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e901609f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 -- 172.21.15.29:0/3898543030 >> 172.21.15.29:0/3898543030 conn(0x7f0e9006bc10 msgr2=0x7f0e9006c020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 -- 172.21.15.29:0/3898543030 shutdown_connections 2023-10-14T18:19:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.258+0000 7f0e96261640 1 -- 172.21.15.29:0/3898543030 wait complete. 2023-10-14T18:19:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.260+0000 7f0e96261640 1 Processor -- start 2023-10-14T18:19:11.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.260+0000 7f0e96261640 1 -- start start 2023-10-14T18:19:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e900c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e90161470 0x7f0e90070230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9014ae40 con 0x7f0e90065ab0 2023-10-14T18:19:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9015afc0 con 0x7f0e90161470 2023-10-14T18:19:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e96261640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0e9015b2a0 con 0x7f0e90160530 2023-10-14T18:19:11.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36278/0 (socket says 172.21.15.29:36278) 2023-10-14T18:19:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 -- 172.21.15.29:0/2592200086 learned_addr learned my addr 172.21.15.29:0/2592200086 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8ffff640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e90161470 0x7f0e90070230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 -- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 msgr2=0x7f0e900c5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e900c5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 -- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e90161470 msgr2=0x7f0e90070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e90161470 0x7f0e90070230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.261+0000 7f0e8f7fe640 1 -- 172.21.15.29:0/2592200086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e84009400 con 0x7f0e90065ab0 2023-10-14T18:19:11.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e8f7fe640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009fc0 tx=0x7f0e8000d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80003b90 con 0x7f0e90065ab0 2023-10-14T18:19:11.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0e8000f040 con 0x7f0e90065ab0 2023-10-14T18:19:11.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e9015b580 con 0x7f0e90065ab0 2023-10-14T18:19:11.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e80004030 con 0x7f0e90065ab0 2023-10-14T18:19:11.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.262+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e9015bb40 con 0x7f0e90065ab0 2023-10-14T18:19:11.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.264+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e9004e160 con 0x7f0e90065ab0 2023-10-14T18:19:11.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.265+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f0e80004190 con 0x7f0e90065ab0 2023-10-14T18:19:11.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.266+0000 7f0e77fff640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 0x7f0e7c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.266+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f0e80089c60 con 0x7f0e90065ab0 2023-10-14T18:19:11.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.266+0000 7f0e8effd640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 0x7f0e7c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.267+0000 7f0e8effd640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 0x7f0e7c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0e84036000 tx=0x7f0e84030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.270+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0e800549d0 con 0x7f0e90065ab0 2023-10-14T18:19:11.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.468+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f0e900713f0 con 0x7f0e90065ab0 2023-10-14T18:19:11.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.469+0000 7f0e77fff640 1 -- 172.21.15.29:0/2592200086 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v256) v1 ==== 107+0+11 (secure 0 0 0) 0x7f0e8001a020 con 0x7f0e90065ab0 2023-10-14T18:19:11.470 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.471+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 msgr2=0x7f0e7c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.471+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 0x7f0e7c06d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f0e84036000 tx=0x7f0e84030000 comp rx=0 tx=0).stop 2023-10-14T18:19:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.471+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 msgr2=0x7f0e9014a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.471+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 secure :-1 s=READY pgs=3022 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009fc0 tx=0x7f0e8000d560 comp rx=0 tx=0).stop 2023-10-14T18:19:11.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 shutdown_connections 2023-10-14T18:19:11.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f0e7c06acc0 0x7f0e7c06d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0e90161470 0x7f0e90070230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f0e90160530 0x7f0e900c5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 --2- 172.21.15.29:0/2592200086 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0e90065ab0 0x7f0e9014a900 unknown :-1 s=CLOSED pgs=3022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 >> 172.21.15.29:0/2592200086 conn(0x7f0e9006bc10 msgr2=0x7f0e900aeb10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 shutdown_connections 2023-10-14T18:19:11.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.472+0000 7f0e96261640 1 -- 172.21.15.29:0/2592200086 wait complete. 2023-10-14T18:19:11.491 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:19:11.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.704+0000 7f84ee1bf640 1 Processor -- start 2023-10-14T18:19:11.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.704+0000 7f84ee1bf640 1 -- start start 2023-10-14T18:19:11.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.705+0000 7f84ee1bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.705+0000 7f84ee1bf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84e80d0be0 0x7f84e80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ee1bf640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84e815d150 con 0x7f84e815cd70 2023-10-14T18:19:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ee1bf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84e80d1180 con 0x7f84e80d0be0 2023-10-14T18:19:11.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ee1bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80d1460 con 0x7f84e80b8480 2023-10-14T18:19:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ec9bc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ed9be640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84e80d0be0 0x7f84e80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ed1bd640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f84e815cd70 0x7f84e80d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:52672/0 (socket says 172.21.15.29:52672) 2023-10-14T18:19:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.706+0000 7f84ed1bd640 1 -- 172.21.15.29:0/4108136372 learned_addr learned my addr 172.21.15.29:0/4108136372 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.707+0000 7f84ed9be640 1 -- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 msgr2=0x7f84e80cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.707+0000 7f84ed9be640 1 --2- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.707+0000 7f84ed9be640 1 -- 172.21.15.29:0/4108136372 >> v1:172.21.15.186:6789/0 conn(0x7f84e815cd70 legacy=0x7f84e80d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.707+0000 7f84ed9be640 1 -- 172.21.15.29:0/4108136372 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e80d1740 con 0x7f84e80d0be0 2023-10-14T18:19:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.709+0000 7f84ed9be640 1 --2- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e80cdd00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009d90 tx=0x7f84e002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=54e972a88128bbe7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.709+0000 7f84dffff640 1 -- 172.21.15.29:0/4108136372 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e002cc60 con 0x7f84e80d0be0 2023-10-14T18:19:11.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.709+0000 7f84dffff640 1 -- 172.21.15.29:0/4108136372 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e002cdc0 con 0x7f84e80d0be0 2023-10-14T18:19:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.709+0000 7f84dffff640 1 -- 172.21.15.29:0/4108136372 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0034660 con 0x7f84e80d0be0 2023-10-14T18:19:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 -- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 msgr2=0x7f84e80cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e80cdd00 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009d90 tx=0x7f84e002beb0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 -- 172.21.15.29:0/4108136372 shutdown_connections 2023-10-14T18:19:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/4108136372 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e80cdd00 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 -- 172.21.15.29:0/4108136372 >> 172.21.15.29:0/4108136372 conn(0x7f84e80cf570 msgr2=0x7f84e80cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 -- 172.21.15.29:0/4108136372 shutdown_connections 2023-10-14T18:19:11.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.710+0000 7f84ee1bf640 1 -- 172.21.15.29:0/4108136372 wait complete. 2023-10-14T18:19:11.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.713+0000 7f84ee1bf640 1 Processor -- start 2023-10-14T18:19:11.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.713+0000 7f84ee1bf640 1 -- start start 2023-10-14T18:19:11.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80c83c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e815d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80c8ce0 con 0x7f84e815cd70 2023-10-14T18:19:11.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80c7600 con 0x7f84e80d0be0 2023-10-14T18:19:11.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.714+0000 7f84ee1bf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84e80c78e0 con 0x7f84e80b8480 2023-10-14T18:19:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.715+0000 7f84ed9be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.715+0000 7f84ed9be640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36312/0 (socket says 172.21.15.29:36312) 2023-10-14T18:19:11.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.715+0000 7f84ed9be640 1 -- 172.21.15.29:0/3692338107 learned_addr learned my addr 172.21.15.29:0/3692338107 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.715+0000 7f84ed1bd640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80c83c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.715+0000 7f84ec9bc640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e815d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 -- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 msgr2=0x7f84e80c83c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.719 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80c83c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 -- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 msgr2=0x7f84e815d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e815d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 -- 172.21.15.29:0/3692338107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e0009a70 con 0x7f84e815cd70 2023-10-14T18:19:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ec9bc640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:11.720 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ed9be640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009d90 tx=0x7f84e0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0040650 con 0x7f84e815cd70 2023-10-14T18:19:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e00407b0 con 0x7f84e815cd70 2023-10-14T18:19:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84e0040b60 con 0x7f84e815cd70 2023-10-14T18:19:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.716+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e80c7bc0 con 0x7f84e815cd70 2023-10-14T18:19:11.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.717+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e80c8920 con 0x7f84e815cd70 2023-10-14T18:19:11.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.718+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e80c8ab0 con 0x7f84e815cd70 2023-10-14T18:19:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.719+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 153) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f84e0040cc0 con 0x7f84e815cd70 2023-10-14T18:19:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.719+0000 7f84ddffb640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 0x7f84cc06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.720+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f84e00ac620 con 0x7f84e815cd70 2023-10-14T18:19:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.720+0000 7f84ed1bd640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 0x7f84cc06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:11.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.721+0000 7f84ed1bd640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 0x7f84cc06d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f84e4005880 tx=0x7f84e4009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:11.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.725+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84e0077250 con 0x7f84e815cd70 2023-10-14T18:19:11.803 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.802+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 154) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f84e0077620 con 0x7f84e815cd70 2023-10-14T18:19:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.917+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7f84e80c80e0 con 0x7f84e815cd70 2023-10-14T18:19:11.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.918+0000 7f84ddffb640 1 -- 172.21.15.29:0/3692338107 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v256) v1 ==== 103+0+11 (secure 0 0 0) 0x7f84e80c80e0 con 0x7f84e815cd70 2023-10-14T18:19:11.919 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:11.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 msgr2=0x7f84cc06d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 0x7f84cc06d1b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f84e4005880 tx=0x7f84e4009040 comp rx=0 tx=0).stop 2023-10-14T18:19:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 msgr2=0x7f84e814ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:11.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 secure :-1 s=READY pgs=3025 cs=0 l=1 rev1=1 crypto rx=0x7f84e0009d90 tx=0x7f84e0004a40 comp rx=0 tx=0).stop 2023-10-14T18:19:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 shutdown_connections 2023-10-14T18:19:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84cc06acf0 0x7f84cc06d1b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84e815cd70 0x7f84e814ff80 unknown :-1 s=CLOSED pgs=3025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84e80d0be0 0x7f84e815d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 --2- 172.21.15.29:0/3692338107 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84e80b8480 0x7f84e80c83c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 >> 172.21.15.29:0/3692338107 conn(0x7f84e80cf570 msgr2=0x7f84e80cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 shutdown_connections 2023-10-14T18:19:11.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:11.921+0000 7f84ee1bf640 1 -- 172.21.15.29:0/3692338107 wait complete. 2023-10-14T18:19:11.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:12.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.159+0000 7fcbef59e640 1 Processor -- start 2023-10-14T18:19:12.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.160+0000 7fcbef59e640 1 -- start start 2023-10-14T18:19:12.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.161+0000 7fcbef59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.161+0000 7fcbef59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcbf00cda80 0x7fcbf015ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.161+0000 7fcbef59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbf006c960 con 0x7fcbf00ad970 2023-10-14T18:19:12.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.161+0000 7fcbef59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbf015b9d0 con 0x7fcbf00cda80 2023-10-14T18:19:12.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.161+0000 7fcbef59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf015bcb0 con 0x7fcbf006c040 2023-10-14T18:19:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.162+0000 7fcbeed9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcbf00cda80 0x7fcbf015ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.162+0000 7fcbee59c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.162+0000 7fcbedd9b640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fcbf00ad970 0x7fcbf015e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:52682/0 (socket says 172.21.15.29:52682) 2023-10-14T18:19:12.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.162+0000 7fcbedd9b640 1 -- 172.21.15.29:0/191955156 learned_addr learned my addr 172.21.15.29:0/191955156 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:12.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbed59a640 1 -- 172.21.15.29:0/191955156 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775531255 0 0) 0x7fcbf006c960 con 0x7fcbf00ad970 2023-10-14T18:19:12.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbed59a640 1 -- 172.21.15.29:0/191955156 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbdc003520 con 0x7fcbf00ad970 2023-10-14T18:19:12.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbeed9d640 1 -- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 msgr2=0x7fcbf006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:12.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbeed9d640 1 --2- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbeed9d640 1 -- 172.21.15.29:0/191955156 >> v1:172.21.15.186:6789/0 conn(0x7fcbf00ad970 legacy=0x7fcbf015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.163+0000 7fcbeed9d640 1 -- 172.21.15.29:0/191955156 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbf015bf90 con 0x7fcbf00cda80 2023-10-14T18:19:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.164+0000 7fcbeed9d640 1 --2- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00cda80 0x7fcbf015ed90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d8ab2e023a106c3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.165+0000 7fcbed59a640 1 -- 172.21.15.29:0/191955156 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe802cc50 con 0x7fcbf00cda80 2023-10-14T18:19:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.165+0000 7fcbed59a640 1 -- 172.21.15.29:0/191955156 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe802cdb0 con 0x7fcbf00cda80 2023-10-14T18:19:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.165+0000 7fcbed59a640 1 -- 172.21.15.29:0/191955156 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe8034660 con 0x7fcbf00cda80 2023-10-14T18:19:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 -- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00cda80 msgr2=0x7fcbf015ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 --2- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00cda80 0x7fcbf015ed90 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe802bea0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 -- 172.21.15.29:0/191955156 shutdown_connections 2023-10-14T18:19:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 --2- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00cda80 0x7fcbf015ed90 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 --2- 172.21.15.29:0/191955156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 -- 172.21.15.29:0/191955156 >> 172.21.15.29:0/191955156 conn(0x7fcbf00ce6a0 msgr2=0x7fcbf00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:12.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 -- 172.21.15.29:0/191955156 shutdown_connections 2023-10-14T18:19:12.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.166+0000 7fcbef59e640 1 -- 172.21.15.29:0/191955156 wait complete. 2023-10-14T18:19:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.168+0000 7fcbef59e640 1 Processor -- start 2023-10-14T18:19:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.168+0000 7fcbef59e640 1 -- start start 2023-10-14T18:19:12.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf0157f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00ad970 0x7fcbf0158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf00aede0 con 0x7fcbf00cda80 2023-10-14T18:19:12.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0158990 con 0x7fcbf00ad970 2023-10-14T18:19:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbef59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf0155570 con 0x7fcbf006c040 2023-10-14T18:19:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbeed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbeed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36314/0 (socket says 172.21.15.29:36314) 2023-10-14T18:19:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.169+0000 7fcbeed9d640 1 -- 172.21.15.29:0/2592314727 learned_addr learned my addr 172.21.15.29:0/2592314727 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:12.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbedd9b640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00ad970 0x7fcbf0158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbee59c640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf0157f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 -- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 msgr2=0x7fcbf0157f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf0157f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 -- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00ad970 msgr2=0x7fcbf0158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00ad970 0x7fcbf0158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 -- 172.21.15.29:0/2592314727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe8009a70 con 0x7fcbf00cda80 2023-10-14T18:19:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.170+0000 7fcbeed9d640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.171+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe803e610 con 0x7fcbf00cda80 2023-10-14T18:19:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.171+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe803e770 con 0x7fcbf00cda80 2023-10-14T18:19:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.171+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbe803eb60 con 0x7fcbf00cda80 2023-10-14T18:19:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.171+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf01f2b90 con 0x7fcbf00cda80 2023-10-14T18:19:12.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.171+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf01f3150 con 0x7fcbf00cda80 2023-10-14T18:19:12.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.172+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf006b330 con 0x7fcbf00cda80 2023-10-14T18:19:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.173+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fcbe803b050 con 0x7fcbf00cda80 2023-10-14T18:19:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.174+0000 7fcbe6ffd640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 0x7fcbd006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.174+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7fcbe80ac980 con 0x7fcbf00cda80 2023-10-14T18:19:12.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.174+0000 7fcbee59c640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 0x7fcbd006d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.175+0000 7fcbee59c640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 0x7fcbd006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009ad0 tx=0x7fcbe0008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.178+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcbe8077100 con 0x7fcbf00cda80 2023-10-14T18:19:12.419 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.417+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbf01556e0 con 0x7fcbf00cda80 2023-10-14T18:19:12.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.418+0000 7fcbe6ffd640 1 -- 172.21.15.29:0/2592314727 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7fcbf01556e0 con 0x7fcbf00cda80 2023-10-14T18:19:12.421 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:12.421 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:12.421 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:19:12.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 msgr2=0x7fcbd006d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 0x7fcbd006d4e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0009ad0 tx=0x7fcbe0008040 comp rx=0 tx=0).stop 2023-10-14T18:19:12.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 msgr2=0x7fcbf01f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 secure :-1 s=READY pgs=3026 cs=0 l=1 rev1=1 crypto rx=0x7fcbe8009d90 tx=0x7fcbe8004a40 comp rx=0 tx=0).stop 2023-10-14T18:19:12.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 shutdown_connections 2023-10-14T18:19:12.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcbd006b020 0x7fcbd006d4e0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcbf00cda80 0x7fcbf01f5c30 unknown :-1 s=CLOSED pgs=3026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcbf00ad970 0x7fcbf0158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 --2- 172.21.15.29:0/2592314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcbf006c040 0x7fcbf0157f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 >> 172.21.15.29:0/2592314727 conn(0x7fcbf00ce6a0 msgr2=0x7fcbf00ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:12.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.421+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 shutdown_connections 2023-10-14T18:19:12.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.422+0000 7fcbef59e640 1 -- 172.21.15.29:0/2592314727 wait complete. 2023-10-14T18:19:12.441 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:12.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.669+0000 7f400d40d640 1 Processor -- start 2023-10-14T18:19:12.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.669+0000 7f400d40d640 1 -- start start 2023-10-14T18:19:12.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f400d40d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40080c9060 0x7f400806f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f400d40d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f400806c9e0 0x7f400806f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f400d40d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40080ca560 con 0x7f40080c9440 2023-10-14T18:19:12.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f400d40d640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080c9950 con 0x7f40080c9060 2023-10-14T18:19:12.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f400d40d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40080c9c30 con 0x7f400806c9e0 2023-10-14T18:19:12.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.671+0000 7f4007fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40080c9060 0x7f400806f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f40080c9060 0x7f400806f440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36320/0 (socket says 172.21.15.29:36320) 2023-10-14T18:19:12.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 -- 172.21.15.29:0/3905064454 learned_addr learned my addr 172.21.15.29:0/3905064454 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:12.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 -- 172.21.15.29:0/3905064454 >> v2:172.21.15.186:3300/0 conn(0x7f400806c9e0 msgr2=0x7f400806f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:12.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 --2- 172.21.15.29:0/3905064454 >> v2:172.21.15.186:3300/0 conn(0x7f400806c9e0 0x7f400806f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 -- 172.21.15.29:0/3905064454 >> v1:172.21.15.186:6789/0 conn(0x7f40080c9440 legacy=0x7f4008164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:12.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.672+0000 7f4007fff640 1 -- 172.21.15.29:0/3905064454 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40080c9f10 con 0x7f40080c9060 2023-10-14T18:19:12.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.673+0000 7f4007fff640 1 --2- 172.21.15.29:0/3905064454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f400806f440 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f4008164cb0 tx=0x7f3ff802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc1698262e6720e0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.673+0000 7f40077fe640 1 -- 172.21.15.29:0/3905064454 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8031870 con 0x7f40080c9060 2023-10-14T18:19:12.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f40077fe640 1 -- 172.21.15.29:0/3905064454 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff80319d0 con 0x7f40080c9060 2023-10-14T18:19:12.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f40077fe640 1 -- 172.21.15.29:0/3905064454 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff8031d60 con 0x7f40080c9060 2023-10-14T18:19:12.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 -- 172.21.15.29:0/3905064454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 msgr2=0x7f400806f440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 --2- 172.21.15.29:0/3905064454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f400806f440 secure :-1 s=READY pgs=3027 cs=0 l=1 rev1=1 crypto rx=0x7f4008164cb0 tx=0x7f3ff802b910 comp rx=0 tx=0).stop 2023-10-14T18:19:12.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 -- 172.21.15.29:0/3905064454 shutdown_connections 2023-10-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 --2- 172.21.15.29:0/3905064454 >> v2:172.21.15.186:3300/0 conn(0x7f400806c9e0 0x7f400806f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 --2- 172.21.15.29:0/3905064454 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f400806f440 unknown :-1 s=CLOSED pgs=3027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 -- 172.21.15.29:0/3905064454 >> 172.21.15.29:0/3905064454 conn(0x7f40080ce6a0 msgr2=0x7f40080ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 -- 172.21.15.29:0/3905064454 shutdown_connections 2023-10-14T18:19:12.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.674+0000 7f400d40d640 1 -- 172.21.15.29:0/3905064454 wait complete. 2023-10-14T18:19:12.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.676+0000 7f400d40d640 1 Processor -- start 2023-10-14T18:19:12.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.676+0000 7f400d40d640 1 -- start start 2023-10-14T18:19:12.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f400806c9e0 0x7f40080c5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 0x7f40081fa000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c55c0 con 0x7f40080c9060 2023-10-14T18:19:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c8160 con 0x7f400806c9e0 2023-10-14T18:19:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.677+0000 7f400d40d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40080c70c0 con 0x7f40080c9440 2023-10-14T18:19:12.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36336/0 (socket says 172.21.15.29:36336) 2023-10-14T18:19:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 -- 172.21.15.29:0/1337800022 learned_addr learned my addr 172.21.15.29:0/1337800022 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:12.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f4007fff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f400806c9e0 0x7f40080c5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f400cc0c640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 0x7f40081fa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 -- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 msgr2=0x7f40081fa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 0x7f40081fa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 -- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f400806c9e0 msgr2=0x7f40080c5080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f400806c9e0 0x7f40080c5080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f3ffffff640 1 -- 172.21.15.29:0/1337800022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ff80094d0 con 0x7f40080c9060 2023-10-14T18:19:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.678+0000 7f400cc0c640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 0x7f40081fa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f3ffffff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000b450 tx=0x7f3ff000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0011890 con 0x7f40080c9060 2023-10-14T18:19:12.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ff00119f0 con 0x7f40080c9060 2023-10-14T18:19:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ff0011da0 con 0x7f40080c9060 2023-10-14T18:19:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40080c73a0 con 0x7f40080c9060 2023-10-14T18:19:12.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.679+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40080c7960 con 0x7f40080c9060 2023-10-14T18:19:12.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.680+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f400804e9b0 con 0x7f40080c9060 2023-10-14T18:19:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.681+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f3ff001a400 con 0x7f40080c9060 2023-10-14T18:19:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.682+0000 7f40057fa640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.682+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f3ff008a430 con 0x7f40080c9060 2023-10-14T18:19:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.683+0000 7f4007fff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:12.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.684+0000 7f4007fff640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 0x7f3fe806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff803c040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:12.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.687+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ff0054b30 con 0x7f40080c9060 2023-10-14T18:19:12.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.884+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40080b4620 con 0x7f40080c9060 2023-10-14T18:19:12.888 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.887+0000 7f40057fa640 1 -- 172.21.15.29:0/1337800022 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+21608 (secure 0 0 0) 0x7f3ff0009280 con 0x7f40080c9060 2023-10-14T18:19:12.889 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:12.889 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":256,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:09.147522+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.889+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 msgr2=0x7f3fe806d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.889+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 0x7f3fe806d4e0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f3ff8036000 tx=0x7f3ff803c040 comp rx=0 tx=0).stop 2023-10-14T18:19:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.889+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 msgr2=0x7f40080c67c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:12.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.889+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 secure :-1 s=READY pgs=3028 cs=0 l=1 rev1=1 crypto rx=0x7f3ff000b450 tx=0x7f3ff000b910 comp rx=0 tx=0).stop 2023-10-14T18:19:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 shutdown_connections 2023-10-14T18:19:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3fe806b020 0x7f3fe806d4e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f40080c9440 0x7f40081fa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f40080c9060 0x7f40080c67c0 unknown :-1 s=CLOSED pgs=3028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 --2- 172.21.15.29:0/1337800022 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f400806c9e0 0x7f40080c5080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 >> 172.21.15.29:0/1337800022 conn(0x7f40080ce6a0 msgr2=0x7f400806a160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 shutdown_connections 2023-10-14T18:19:12.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:12.890+0000 7f400d40d640 1 -- 172.21.15.29:0/1337800022 wait complete. 2023-10-14T18:19:12.912 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:19:12.912 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:13.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.136+0000 7f064fd90640 1 Processor -- start 2023-10-14T18:19:13.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.137+0000 7f064fd90640 1 -- start start 2023-10-14T18:19:13.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0648159b20 0x7f064806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 0x7f06480b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f06480b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064806f7f0 con 0x7f0648159b20 2023-10-14T18:19:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f064806c5e0 con 0x7f0648159f00 2023-10-14T18:19:13.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.138+0000 7f064fd90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f064806c8c0 con 0x7f064806c120 2023-10-14T18:19:13.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0648159b20 0x7f064806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f0648159b20 0x7f064806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36344/0 (socket says 172.21.15.29:36344) 2023-10-14T18:19:13.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 -- 172.21.15.29:0/2281902915 learned_addr learned my addr 172.21.15.29:0/2281902915 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064d32b640 1 --2- 172.21.15.29:0/2281902915 >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 0x7f06480b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 -- 172.21.15.29:0/2281902915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 msgr2=0x7f06480b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 --2- 172.21.15.29:0/2281902915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f06480b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 -- 172.21.15.29:0/2281902915 >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 msgr2=0x7f06480b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:13.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 --2- 172.21.15.29:0/2281902915 >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 0x7f06480b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064db2c640 1 -- 172.21.15.29:0/2281902915 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f064806cba0 con 0x7f0648159b20 2023-10-14T18:19:13.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.139+0000 7f064d32b640 1 --2- 172.21.15.29:0/2281902915 >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 0x7f06480b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:13.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.140+0000 7f064db2c640 1 --2- 172.21.15.29:0/2281902915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f064806bbe0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f064815d770 tx=0x7f064402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adc735c72c1fb08 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.141+0000 7f064cb2a640 1 -- 172.21.15.29:0/2281902915 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644031870 con 0x7f0648159b20 2023-10-14T18:19:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.141+0000 7f064cb2a640 1 -- 172.21.15.29:0/2281902915 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06440319d0 con 0x7f0648159b20 2023-10-14T18:19:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.141+0000 7f064cb2a640 1 -- 172.21.15.29:0/2281902915 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0644031d60 con 0x7f0648159b20 2023-10-14T18:19:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.141+0000 7f064fd90640 1 -- 172.21.15.29:0/2281902915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 msgr2=0x7f064806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.141+0000 7f064fd90640 1 --2- 172.21.15.29:0/2281902915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f064806bbe0 secure :-1 s=READY pgs=3029 cs=0 l=1 rev1=1 crypto rx=0x7f064815d770 tx=0x7f064402b910 comp rx=0 tx=0).stop 2023-10-14T18:19:13.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 -- 172.21.15.29:0/2281902915 shutdown_connections 2023-10-14T18:19:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 --2- 172.21.15.29:0/2281902915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f06480b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 --2- 172.21.15.29:0/2281902915 >> v2:172.21.15.186:3300/0 conn(0x7f0648159f00 0x7f06480b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 --2- 172.21.15.29:0/2281902915 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f064806bbe0 unknown :-1 s=CLOSED pgs=3029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 -- 172.21.15.29:0/2281902915 >> 172.21.15.29:0/2281902915 conn(0x7f06480cb410 msgr2=0x7f06480cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:13.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 -- 172.21.15.29:0/2281902915 shutdown_connections 2023-10-14T18:19:13.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.142+0000 7f064fd90640 1 -- 172.21.15.29:0/2281902915 wait complete. 2023-10-14T18:19:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.144+0000 7f064fd90640 1 Processor -- start 2023-10-14T18:19:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.144+0000 7f064fd90640 1 -- start start 2023-10-14T18:19:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f064814be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0648159f00 0x7f064815bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480c3900 con 0x7f0648159b20 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06480c2cf0 con 0x7f0648159f00 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064fd90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06480c2fd0 con 0x7f064806c120 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064d32b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064d32b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36358/0 (socket says 172.21.15.29:36358) 2023-10-14T18:19:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.145+0000 7f064d32b640 1 -- 172.21.15.29:0/2914997271 learned_addr learned my addr 172.21.15.29:0/2914997271 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064db2c640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f064814be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064e32d640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0648159f00 0x7f064815bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 -- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 msgr2=0x7f064814be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f064814be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 -- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0648159f00 msgr2=0x7f064815bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0648159f00 0x7f064815bf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 -- 172.21.15.29:0/2914997271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06440094d0 con 0x7f0648159b20 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f064d32b640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f063800c7f0 tx=0x7f063800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.146+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063800cea0 con 0x7f0648159b20 2023-10-14T18:19:13.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.147+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0638007a30 con 0x7f0648159b20 2023-10-14T18:19:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.147+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06480c32b0 con 0x7f0648159b20 2023-10-14T18:19:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.147+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638007dc0 con 0x7f0648159b20 2023-10-14T18:19:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.147+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06480c2a00 con 0x7f0648159b20 2023-10-14T18:19:13.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.148+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f064804e160 con 0x7f0648159b20 2023-10-14T18:19:13.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.149+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89377+0+0 (secure 0 0 0) 0x7f06380113f0 con 0x7f0648159b20 2023-10-14T18:19:13.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.150+0000 7f063e7fc640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 0x7f062c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.150+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f063808a2b0 con 0x7f0648159b20 2023-10-14T18:19:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.151+0000 7f064db2c640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 0x7f062c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.152+0000 7f064db2c640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 0x7f062c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0644036000 tx=0x7f064403c040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.157+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f0638054890 con 0x7f0648159b20 2023-10-14T18:19:13.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.399+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f064806bbe0 con 0x7f0648159b20 2023-10-14T18:19:13.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.400+0000 7f063e7fc640 1 -- 172.21.15.29:0/2914997271 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7f0638057eb0 con 0x7f0648159b20 2023-10-14T18:19:13.403 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:13.403 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:13.403 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:19:13.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 msgr2=0x7f062c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 0x7f062c06d4e0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f0644036000 tx=0x7f064403c040 comp rx=0 tx=0).stop 2023-10-14T18:19:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 msgr2=0x7f06480c24c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 secure :-1 s=READY pgs=3030 cs=0 l=1 rev1=1 crypto rx=0x7f063800c7f0 tx=0x7f063800ccb0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 shutdown_connections 2023-10-14T18:19:13.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f062c06b020 0x7f062c06d4e0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.403+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f0648159f00 0x7f064815bf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.404+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f0648159b20 0x7f06480c24c0 unknown :-1 s=CLOSED pgs=3030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.404+0000 7f064fd90640 1 --2- 172.21.15.29:0/2914997271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f064806c120 0x7f064814be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.404+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 >> 172.21.15.29:0/2914997271 conn(0x7f06480cb410 msgr2=0x7f06480cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:13.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.404+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 shutdown_connections 2023-10-14T18:19:13.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.404+0000 7f064fd90640 1 -- 172.21.15.29:0/2914997271 wait complete. 2023-10-14T18:19:13.424 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:13.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.653+0000 7fa88bfff640 1 Processor -- start 2023-10-14T18:19:13.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.654+0000 7fa88bfff640 1 -- start start 2023-10-14T18:19:13.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88bfff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa88c159950 0x7fa88c15a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88bfff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa88c15aef0 con 0x7fa88c06c810 2023-10-14T18:19:13.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88bfff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa88c15d910 con 0x7fa88c070450 2023-10-14T18:19:13.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88bfff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa88c15e490 con 0x7fa88c159950 2023-10-14T18:19:13.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88affd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa88c159950 0x7fa88c15a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.655+0000 7fa88b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa88c06c810 0x7fa88c0c9090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58424/0 (socket says 172.21.15.29:58424) 2023-10-14T18:19:13.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.656+0000 7fa88b7fe640 1 -- 172.21.15.29:0/3206966360 learned_addr learned my addr 172.21.15.29:0/3206966360 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:13.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.656+0000 7fa88affd640 1 -- 172.21.15.29:0/3206966360 >> v1:172.21.15.186:6789/0 conn(0x7fa88c070450 legacy=0x7fa88c06c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:13.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.656+0000 7fa88affd640 1 -- 172.21.15.29:0/3206966360 >> v1:172.21.15.29:6789/0 conn(0x7fa88c06c810 legacy=0x7fa88c0c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.656+0000 7fa88affd640 1 -- 172.21.15.29:0/3206966360 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88c15dbf0 con 0x7fa88c159950 2023-10-14T18:19:13.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.657+0000 7fa88affd640 1 --2- 172.21.15.29:0/3206966360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c15a9b0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0097a0 tx=0x7fa87c02b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=491178d78fe48657 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.657+0000 7fa88a7fc640 1 -- 172.21.15.29:0/3206966360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c02e070 con 0x7fa88c159950 2023-10-14T18:19:13.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.657+0000 7fa88a7fc640 1 -- 172.21.15.29:0/3206966360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87c0315a0 con 0x7fa88c159950 2023-10-14T18:19:13.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.657+0000 7fa88a7fc640 1 -- 172.21.15.29:0/3206966360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa87c031930 con 0x7fa88c159950 2023-10-14T18:19:13.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 -- 172.21.15.29:0/3206966360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 msgr2=0x7fa88c15a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3206966360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c15a9b0 secure :-1 s=READY pgs=3032 cs=0 l=1 rev1=1 crypto rx=0x7fa87c0097a0 tx=0x7fa87c02b840 comp rx=0 tx=0).stop 2023-10-14T18:19:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 -- 172.21.15.29:0/3206966360 shutdown_connections 2023-10-14T18:19:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3206966360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c15a9b0 unknown :-1 s=CLOSED pgs=3032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 -- 172.21.15.29:0/3206966360 >> 172.21.15.29:0/3206966360 conn(0x7fa88c0ce6a0 msgr2=0x7fa88c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:13.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 -- 172.21.15.29:0/3206966360 shutdown_connections 2023-10-14T18:19:13.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.658+0000 7fa88bfff640 1 -- 172.21.15.29:0/3206966360 wait complete. 2023-10-14T18:19:13.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.660+0000 7fa88bfff640 1 Processor -- start 2023-10-14T18:19:13.666 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.660+0000 7fa88bfff640 1 -- start start 2023-10-14T18:19:13.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa88c06c810 0x7fa88c1f5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.667 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 0x7fa88c0c4370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c1f6690 con 0x7fa88c159950 2023-10-14T18:19:13.668 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c0c3eb0 con 0x7fa88c070450 2023-10-14T18:19:13.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.661+0000 7fa88bfff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88c1f5df0 con 0x7fa88c06c810 2023-10-14T18:19:13.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36382/0 (socket says 172.21.15.29:36382) 2023-10-14T18:19:13.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 -- 172.21.15.29:0/3064879364 learned_addr learned my addr 172.21.15.29:0/3064879364 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:13.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa883fff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 0x7fa88c0c4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 -- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa88c06c810 msgr2=0x7fa88c1f5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa88c06c810 0x7fa88c1f5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 -- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 msgr2=0x7fa88c0c4370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 0x7fa88c0c4370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 -- 172.21.15.29:0/3064879364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c009400 con 0x7fa88c159950 2023-10-14T18:19:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa883fff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 0x7fa88c0c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.662+0000 7fa88b7fe640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fa88400cbc0 tx=0x7fa88400a430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.663+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88400ad40 con 0x7fa88c159950 2023-10-14T18:19:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.663+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa884012400 con 0x7fa88c159950 2023-10-14T18:19:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.663+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa88c1f60d0 con 0x7fa88c159950 2023-10-14T18:19:13.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.663+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa88400aea0 con 0x7fa88c159950 2023-10-14T18:19:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.663+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88c0c2350 con 0x7fa88c159950 2023-10-14T18:19:13.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.664+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa88c0c21b0 con 0x7fa88c159950 2023-10-14T18:19:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.665+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 154) v1 ==== 89377+0+0 (secure 0 0 0) 0x7fa884007070 con 0x7fa88c159950 2023-10-14T18:19:13.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.666+0000 7fa8837fe640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 0x7fa86c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.666+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7fa884089dd0 con 0x7fa88c159950 2023-10-14T18:19:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.666+0000 7fa88affd640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 0x7fa86c06d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:13.678 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.667+0000 7fa88affd640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 0x7fa86c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa87c036000 tx=0x7fa87c030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:13.679 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.670+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa88c0c21b0 con 0x7fa88c159950 2023-10-14T18:19:13.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.821+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa884090080 con 0x7fa88c159950 2023-10-14T18:19:13.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.915+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa88c15deb0 con 0x7fa88c159950 2023-10-14T18:19:13.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.916+0000 7fa8837fe640 1 -- 172.21.15.29:0/3064879364 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+6152 (secure 0 0 0) 0x7fa88c15deb0 con 0x7fa88c159950 2023-10-14T18:19:13.919 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:13.919 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":129,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118}],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:35.103933+0000","modified":"2023-10-14T18:18:38.197589+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":8767},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8767":{"gid":8767,"name":"b","rank":0,"incarnation":119,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6833/2448419287","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":2448419287},{"type":"v1","addr":"172.21.15.186:6833","nonce":2448419287}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[31],"metadata_pool":30,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":15},{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:13.919 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 129 2023-10-14T18:19:13.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 msgr2=0x7fa86c06d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 0x7fa86c06d4e0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fa87c036000 tx=0x7fa87c030000 comp rx=0 tx=0).stop 2023-10-14T18:19:13.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 msgr2=0x7fa88c1f3250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 secure :-1 s=READY pgs=3033 cs=0 l=1 rev1=1 crypto rx=0x7fa88400cbc0 tx=0x7fa88400a430 comp rx=0 tx=0).stop 2023-10-14T18:19:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 shutdown_connections 2023-10-14T18:19:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa86c06b020 0x7fa86c06d4e0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa88c159950 0x7fa88c1f3250 unknown :-1 s=CLOSED pgs=3033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa88c070450 0x7fa88c0c4370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 --2- 172.21.15.29:0/3064879364 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa88c06c810 0x7fa88c1f5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:13.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 >> 172.21.15.29:0/3064879364 conn(0x7fa88c0ce6a0 msgr2=0x7fa88c15a690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:13.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 shutdown_connections 2023-10-14T18:19:13.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:13.919+0000 7fa88bfff640 1 -- 172.21.15.29:0/3064879364 wait complete. 2023-10-14T18:19:13.940 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.154+0000 7fcdea333640 1 Processor -- start 2023-10-14T18:19:14.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.154+0000 7fcdea333640 1 -- start start 2023-10-14T18:19:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcdea333640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcde40caa10 0x7fcde406fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcdea333640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcde40ccdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcdea333640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcde4159790 con 0x7fcde41593b0 2023-10-14T18:19:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcdea333640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcde40c9b20 con 0x7fcde40caa10 2023-10-14T18:19:14.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcdea333640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde40c9e00 con 0x7fcde40cc920 2023-10-14T18:19:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcde3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcde41593b0 0x7fcde406f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58430/0 (socket says 172.21.15.29:58430) 2023-10-14T18:19:14.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcde3fff640 1 -- 172.21.15.29:0/2404605800 learned_addr learned my addr 172.21.15.29:0/2404605800 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcde88d0640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcde40ccdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1681989913 0 0) 0x7fcde4159790 con 0x7fcde41593b0 2023-10-14T18:19:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcddb7fe640 1 --2- 172.21.15.29:0/2404605800 >> v2:172.21.15.186:3300/0 conn(0x7fcde40caa10 0x7fcde406fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.156+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcdd0003520 con 0x7fcde41593b0 2023-10-14T18:19:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3741608075 0 0) 0x7fcdd0003520 con 0x7fcde41593b0 2023-10-14T18:19:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 msgr2=0x7fcde40ccdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcde40ccdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 >> v2:172.21.15.186:3300/0 conn(0x7fcde40caa10 msgr2=0x7fcde406fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 --2- 172.21.15.29:0/2404605800 >> v2:172.21.15.186:3300/0 conn(0x7fcde40caa10 0x7fcde406fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde40ca0e0 con 0x7fcde41593b0 2023-10-14T18:19:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fcdd4001670 con 0x7fcde41593b0 2023-10-14T18:19:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 >> v1:172.21.15.29:6789/0 conn(0x7fcde41593b0 legacy=0x7fcde406f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0003970 0x7fcdd0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcdd00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd0004d00 0x7fcdd0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde40c9b20 con 0x7fcdd0004d00 2023-10-14T18:19:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde40c9e00 con 0x7fcdd0003970 2023-10-14T18:19:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdd000ba30 con 0x7fcde40cc920 2023-10-14T18:19:14.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.157+0000 7fcde88d0640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd0004d00 0x7fcdd0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde3fff640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0003970 0x7fcdd0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcddb7fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcdd00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde88d0640 1 -- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 msgr2=0x7fcdd00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde88d0640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcdd00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde88d0640 1 -- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0003970 msgr2=0x7fcdd0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde88d0640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0003970 0x7fcdd0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde88d0640 1 -- 172.21.15.29:0/2404605800 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd0003520 con 0x7fcdd0004d00 2023-10-14T18:19:14.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcddb7fe640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde40cc920 0x7fcdd00047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:14.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.158+0000 7fcde3fff640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0003970 0x7fcdd0003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:14.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.159+0000 7fcde88d0640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd0004d00 0x7fcdd0005180 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009ac0 tx=0x7fcdd00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a85219e6ab188d3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.159+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc02c6b0 con 0x7fcdd0004d00 2023-10-14T18:19:14.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.159+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc02c810 con 0x7fcdd0004d00 2023-10-14T18:19:14.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.159+0000 7fcde37fe640 1 -- 172.21.15.29:0/2404605800 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc02cba0 con 0x7fcdd0004d00 2023-10-14T18:19:14.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 -- 172.21.15.29:0/2404605800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd0004d00 msgr2=0x7fcdd0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 --2- 172.21.15.29:0/2404605800 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcdd0004d00 0x7fcdd0005180 secure :-1 s=READY pgs=3035 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009ac0 tx=0x7fcdd00057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcde88d0640 1 -- 172.21.15.29:0/2404605800 reap_dead start 2023-10-14T18:19:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 -- 172.21.15.29:0/2404605800 shutdown_connections 2023-10-14T18:19:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 -- 172.21.15.29:0/2404605800 >> 172.21.15.29:0/2404605800 conn(0x7fcde40cb410 msgr2=0x7fcde40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:14.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 -- 172.21.15.29:0/2404605800 shutdown_connections 2023-10-14T18:19:14.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.160+0000 7fcdea333640 1 -- 172.21.15.29:0/2404605800 wait complete. 2023-10-14T18:19:14.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.162+0000 7fcdea333640 1 Processor -- start 2023-10-14T18:19:14.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.162+0000 7fcdea333640 1 -- start start 2023-10-14T18:19:14.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 0x7fcde4158400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde4152240 0x7fcde40c6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde41573f0 con 0x7fcde4151ca0 2023-10-14T18:19:14.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4157640 con 0x7fcdd0004d00 2023-10-14T18:19:14.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcdea333640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcde4157920 con 0x7fcde4152240 2023-10-14T18:19:14.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcde88d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcde88d0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36402/0 (socket says 172.21.15.29:36402) 2023-10-14T18:19:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.163+0000 7fcde88d0640 1 -- 172.21.15.29:0/3757742030 learned_addr learned my addr 172.21.15.29:0/3757742030 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde3fff640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 0x7fcde4158400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcddb7fe640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde4152240 0x7fcde40c6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 -- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde4152240 msgr2=0x7fcde40c6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde4152240 0x7fcde40c6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 -- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 msgr2=0x7fcde4158400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 0x7fcde4158400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 -- 172.21.15.29:0/3757742030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4157c00 con 0x7fcde4151ca0 2023-10-14T18:19:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde3fff640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 0x7fcde4158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:14.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde88d0640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fcddc037040 tx=0x7fcddc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.164+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc02cd90 con 0x7fcde4151ca0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.165+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcddc003610 con 0x7fcde4151ca0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.165+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcddc0039c0 con 0x7fcde4151ca0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.165+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde4157ee0 con 0x7fcde4151ca0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.165+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde4159e60 con 0x7fcde4151ca0 2023-10-14T18:19:14.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.166+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcde4157780 con 0x7fcde4151ca0 2023-10-14T18:19:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.167+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fcddc003b20 con 0x7fcde4151ca0 2023-10-14T18:19:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.168+0000 7fcde17fa640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 0x7fcdc406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.168+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7fcddc0aaf40 con 0x7fcde4151ca0 2023-10-14T18:19:14.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.168+0000 7fcde3fff640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 0x7fcdc406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.169+0000 7fcde3fff640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 0x7fcdc406d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4001120 tx=0x7fcdd4009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.172+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcde4157780 con 0x7fcde4151ca0 2023-10-14T18:19:14.380 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.378+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcde4157780 con 0x7fcde4151ca0 2023-10-14T18:19:14.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.383+0000 7fcde17fa640 1 -- 172.21.15.29:0/3757742030 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v256) v1 ==== 74+0+21608 (secure 0 0 0) 0x7fcde4157780 con 0x7fcde4151ca0 2023-10-14T18:19:14.385 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:14.385 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":256,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:09.147522+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":30,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:18:33.087026+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"249","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":2.5,"score_stable":2.5,"optimal_score":1,"raw_score_acting":2.5,"raw_score_stable":2.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":31,"pool_name":"cephfs_data","create_time":"2023-10-14T18:18:34.140901+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"250","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 msgr2=0x7fcdc406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 0x7fcdc406d570 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fcdd4001120 tx=0x7fcdd4009040 comp rx=0 tx=0).stop 2023-10-14T18:19:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 msgr2=0x7fcde4159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 secure :-1 s=READY pgs=3036 cs=0 l=1 rev1=1 crypto rx=0x7fcddc037040 tx=0x7fcddc004730 comp rx=0 tx=0).stop 2023-10-14T18:19:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 shutdown_connections 2023-10-14T18:19:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fcdc406b0b0 0x7fcdc406d570 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcde4151ca0 0x7fcde4159920 unknown :-1 s=CLOSED pgs=3036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcde4152240 0x7fcde40c6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 --2- 172.21.15.29:0/3757742030 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcdd0004d00 0x7fcde4158400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 >> 172.21.15.29:0/3757742030 conn(0x7fcde40cb410 msgr2=0x7fcde415d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 shutdown_connections 2023-10-14T18:19:14.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.386+0000 7fcdea333640 1 -- 172.21.15.29:0/3757742030 wait complete. 2023-10-14T18:19:14.408 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T18:19:14.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.638+0000 7f854b59e640 1 Processor -- start 2023-10-14T18:19:14.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.638+0000 7f854b59e640 1 -- start start 2023-10-14T18:19:14.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.639+0000 7f854b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f854c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854b59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c15b9d0 con 0x7f854c0ccfb0 2023-10-14T18:19:14.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854b59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c15bcb0 con 0x7f854c06c040 2023-10-14T18:19:14.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c15bf90 con 0x7f854c0ad970 2023-10-14T18:19:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854ad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f854c0ccfb0 0x7f854c0cd440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58442/0 (socket says 172.21.15.29:58442) 2023-10-14T18:19:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854ad9d640 1 -- 172.21.15.29:0/385230515 learned_addr learned my addr 172.21.15.29:0/385230515 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f8549d9b640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f854c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.640+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258926589 0 0) 0x7f854c15b9d0 con 0x7f854c0ccfb0 2023-10-14T18:19:14.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538003520 con 0x7f854c0ccfb0 2023-10-14T18:19:14.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1749496420 0 0) 0x7f8538003520 con 0x7f854c0ccfb0 2023-10-14T18:19:14.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 msgr2=0x7f854c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f854c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 >> v1:172.21.15.186:6789/0 conn(0x7f854c06c040 legacy=0x7f854c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0cc270 con 0x7f854c0ccfb0 2023-10-14T18:19:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8544001670 con 0x7f854c0ccfb0 2023-10-14T18:19:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 >> v1:172.21.15.29:6789/0 conn(0x7f854c0ccfb0 legacy=0x7f854c0cd440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f85380039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8538003f70 0x7f85380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f8538005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c15bf90 con 0x7f8538004d00 2023-10-14T18:19:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f853800bac0 con 0x7f8538003f70 2023-10-14T18:19:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f853800bc80 con 0x7f854c0ad970 2023-10-14T18:19:14.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.641+0000 7f854ad9d640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f8538005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.650 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854a59c640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f85380039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f8549d9b640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8538003f70 0x7f85380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854ad9d640 1 -- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 msgr2=0x7f85380039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854ad9d640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c0ad970 0x7f85380039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854ad9d640 1 -- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8538003f70 msgr2=0x7f85380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:14.651 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854ad9d640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8538003f70 0x7f85380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f854ad9d640 1 -- 172.21.15.29:0/385230515 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8538003520 con 0x7f8538004d00 2023-10-14T18:19:14.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.642+0000 7f8549d9b640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8538003f70 0x7f85380047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:14.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.643+0000 7f854ad9d640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f8538005180 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f854400c0e0 tx=0x7f85380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d2b24ed02606044 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8544005ad0 con 0x7f8538004d00 2023-10-14T18:19:14.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8544005c50 con 0x7f8538004d00 2023-10-14T18:19:14.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854959a640 1 -- 172.21.15.29:0/385230515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f854402c810 con 0x7f8538004d00 2023-10-14T18:19:14.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 -- 172.21.15.29:0/385230515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 msgr2=0x7f8538005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 --2- 172.21.15.29:0/385230515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f8538005180 secure :-1 s=READY pgs=3038 cs=0 l=1 rev1=1 crypto rx=0x7f854400c0e0 tx=0x7f85380057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854ad9d640 1 -- 172.21.15.29:0/385230515 reap_dead start 2023-10-14T18:19:14.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 -- 172.21.15.29:0/385230515 shutdown_connections 2023-10-14T18:19:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 -- 172.21.15.29:0/385230515 >> 172.21.15.29:0/385230515 conn(0x7f854c0ce6a0 msgr2=0x7f854c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 -- 172.21.15.29:0/385230515 shutdown_connections 2023-10-14T18:19:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.644+0000 7f854b59e640 1 -- 172.21.15.29:0/385230515 wait complete. 2023-10-14T18:19:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.646+0000 7f854b59e640 1 Processor -- start 2023-10-14T18:19:14.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.646+0000 7f854b59e640 1 -- start start 2023-10-14T18:19:14.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.647+0000 7f854b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.647+0000 7f854b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 0x7f854c1f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c1553e0 0x7f854c155860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1e4c20 con 0x7f8538004d00 2023-10-14T18:19:14.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c1540b0 con 0x7f854c1f0c60 2023-10-14T18:19:14.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c154340 con 0x7f854c1553e0 2023-10-14T18:19:14.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36424/0 (socket says 172.21.15.29:36424) 2023-10-14T18:19:14.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 -- 172.21.15.29:0/1873253532 learned_addr learned my addr 172.21.15.29:0/1873253532 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:14.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f8549d9b640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 0x7f854c1f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 -- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c1553e0 msgr2=0x7f854c155860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854ad9d640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c1553e0 0x7f854c155860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c1553e0 0x7f854c155860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 -- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 msgr2=0x7f854c1f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 0x7f854c1f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f854a59c640 1 -- 172.21.15.29:0/1873253532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0cdcd0 con 0x7f8538004d00 2023-10-14T18:19:14.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.648+0000 7f8549d9b640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 0x7f854c1f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:14.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f854a59c640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e8c0 tx=0x7f853c00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c003b80 con 0x7f8538004d00 2023-10-14T18:19:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c003ce0 con 0x7f8538004d00 2023-10-14T18:19:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c154620 con 0x7f8538004d00 2023-10-14T18:19:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f853c0052f0 con 0x7f8538004d00 2023-10-14T18:19:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.649+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c154be0 con 0x7f8538004d00 2023-10-14T18:19:14.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.651+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c06b330 con 0x7f8538004d00 2023-10-14T18:19:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.652+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f853c005450 con 0x7f8538004d00 2023-10-14T18:19:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.653+0000 7f8542ffd640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 0x7f852c06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.653+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(256..256 src has 1..256) v4 ==== 7549+0+0 (secure 0 0 0) 0x7f853c08a3a0 con 0x7f8538004d00 2023-10-14T18:19:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.654+0000 7f8549d9b640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 0x7f852c06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:14.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.654+0000 7f8549d9b640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 0x7f852c06d5a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f854c070420 tx=0x7f8534008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:14.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.658+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f853c054920 con 0x7f8538004d00 2023-10-14T18:19:14.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.894+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f854c04e9b0 con 0x7f8538004d00 2023-10-14T18:19:14.897 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:19:14.895+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:19:14.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.899+0000 7f8542ffd640 1 -- 172.21.15.29:0/1873253532 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v130) v1 ==== 161+0+0 (secure 0 0 0) 0x7f853c01a310 con 0x7f8538004d00 2023-10-14T18:19:14.901 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:19:14.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.902+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 msgr2=0x7f852c06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.902+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 0x7f852c06d5a0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f854c070420 tx=0x7f8534008040 comp rx=0 tx=0).stop 2023-10-14T18:19:14.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.902+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 msgr2=0x7f854c153b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:14.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.902+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 secure :-1 s=READY pgs=3039 cs=0 l=1 rev1=1 crypto rx=0x7f853c00e8c0 tx=0x7f853c00ed80 comp rx=0 tx=0).stop 2023-10-14T18:19:14.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 shutdown_connections 2023-10-14T18:19:14.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f852c06b0e0 0x7f852c06d5a0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f854c1553e0 0x7f854c155860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f854c1f0c60 0x7f854c1f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 --2- 172.21.15.29:0/1873253532 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8538004d00 0x7f854c153b70 unknown :-1 s=CLOSED pgs=3039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:14.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 >> 172.21.15.29:0/1873253532 conn(0x7f854c0ce6a0 msgr2=0x7f854c15dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:14.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 shutdown_connections 2023-10-14T18:19:14.908 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:14.903+0000 7f854b59e640 1 -- 172.21.15.29:0/1873253532 wait complete. 2023-10-14T18:19:14.925 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T18:19:14.935 INFO:tasks.ceph.mds.b.smithi186.stderr: -4343> 2023-10-14T18:18:40.056+0000 7f0dec32a640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:19:15.011 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T18:19:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.140+0000 7f96d2a90640 1 Processor -- start 2023-10-14T18:19:15.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.140+0000 7f96d2a90640 1 -- start start 2023-10-14T18:19:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.141+0000 7f96d2a90640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f96cc159ab0 0x7f96cc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.141+0000 7f96d2a90640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc070360 con 0x7f96cc06c140 2023-10-14T18:19:15.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96d2a90640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96cc0cd1f0 con 0x7f96cc159e90 2023-10-14T18:19:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96d2a90640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96cc15d890 con 0x7f96cc159ab0 2023-10-14T18:19:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96d102d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f96cc06c140 0x7f96cc15dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58446/0 (socket says 172.21.15.29:58446) 2023-10-14T18:19:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96d102d640 1 -- 172.21.15.29:0/2707858616 learned_addr learned my addr 172.21.15.29:0/2707858616 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96d082c640 1 --2- 172.21.15.29:0/2707858616 >> v2:172.21.15.186:3300/0 conn(0x7f96cc159ab0 0x7f96cc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290367998 0 0) 0x7f96cc070360 con 0x7f96cc06c140 2023-10-14T18:19:15.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.142+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b4005730 con 0x7f96cc06c140 2023-10-14T18:19:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1846320324 0 0) 0x7f96b4005730 con 0x7f96cc06c140 2023-10-14T18:19:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 >> v2:172.21.15.186:3300/0 conn(0x7f96cc159ab0 msgr2=0x7f96cc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 --2- 172.21.15.29:0/2707858616 >> v2:172.21.15.186:3300/0 conn(0x7f96cc159ab0 0x7f96cc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 >> v1:172.21.15.186:6789/0 conn(0x7f96cc159e90 legacy=0x7f96cc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc0cd4d0 con 0x7f96cc06c140 2023-10-14T18:19:15.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f96c4001670 con 0x7f96cc06c140 2023-10-14T18:19:15.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 >> v1:172.21.15.29:6789/0 conn(0x7f96cc06c140 legacy=0x7f96cc15dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96b4005b80 0x7f96b4005fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 0x7f96b4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40074b0 0x7f96b400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc15d890 con 0x7f96b40069d0 2023-10-14T18:19:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96b400e380 con 0x7f96b4005b80 2023-10-14T18:19:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96b400e540 con 0x7f96b40074b0 2023-10-14T18:19:15.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.143+0000 7f96cbfff640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 0x7f96b4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96cbfff640 1 -- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40074b0 msgr2=0x7f96b400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:15.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96cbfff640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40074b0 0x7f96b400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96cbfff640 1 -- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96b4005b80 msgr2=0x7f96b4005fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:15.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96cbfff640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96b4005b80 0x7f96b4005fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96cbfff640 1 -- 172.21.15.29:0/2707858616 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96b4005730 con 0x7f96b40069d0 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.144+0000 7f96d102d640 1 -- 172.21.15.29:0/2707858616 reap_dead start 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.145+0000 7f96cbfff640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 0x7f96b4006e50 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f96c0001e20 tx=0x7f96b4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a38ff493f1dcae4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.145+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0003b70 con 0x7f96b40069d0 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.145+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c0003cd0 con 0x7f96b40069d0 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.145+0000 7f96cb7fe640 1 -- 172.21.15.29:0/2707858616 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c0005310 con 0x7f96b40069d0 2023-10-14T18:19:15.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 -- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 msgr2=0x7f96b4006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 0x7f96b4006e50 secure :-1 s=READY pgs=3042 cs=0 l=1 rev1=1 crypto rx=0x7f96c0001e20 tx=0x7f96b4007a00 comp rx=0 tx=0).stop 2023-10-14T18:19:15.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 -- 172.21.15.29:0/2707858616 shutdown_connections 2023-10-14T18:19:15.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 --2- 172.21.15.29:0/2707858616 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96b40069d0 0x7f96b4006e50 unknown :-1 s=CLOSED pgs=3042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 -- 172.21.15.29:0/2707858616 >> 172.21.15.29:0/2707858616 conn(0x7f96cc0ce9b0 msgr2=0x7f96cc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:15.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 -- 172.21.15.29:0/2707858616 shutdown_connections 2023-10-14T18:19:15.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.146+0000 7f96d2a90640 1 -- 172.21.15.29:0/2707858616 wait complete. 2023-10-14T18:19:15.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.148+0000 7f96d2a90640 1 Processor -- start 2023-10-14T18:19:15.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.148+0000 7f96d2a90640 1 -- start start 2023-10-14T18:19:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40069d0 0x7f96cc14da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96cc06c590 0x7f96cc06c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc1ef330 con 0x7f96cc0c7e20 2023-10-14T18:19:15.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc06cf30 con 0x7f96cc06c590 2023-10-14T18:19:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d2a90640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96cc1ee990 con 0x7f96b40069d0 2023-10-14T18:19:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d102d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d102d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36446/0 (socket says 172.21.15.29:36446) 2023-10-14T18:19:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d102d640 1 -- 172.21.15.29:0/3676576839 learned_addr learned my addr 172.21.15.29:0/3676576839 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:15.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96d082c640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40069d0 0x7f96cc14da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.149+0000 7f96cbfff640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96cc06c590 0x7f96cc06c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 -- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40069d0 msgr2=0x7f96cc14da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40069d0 0x7f96cc14da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 -- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96cc06c590 msgr2=0x7f96cc06c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96cc06c590 0x7f96cc06c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 -- 172.21.15.29:0/3676576839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96cc1eec70 con 0x7f96cc0c7e20 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d102d640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0b15a0 tx=0x7f96c400c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c40074e0 con 0x7f96cc0c7e20 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96c4007640 con 0x7f96cc0c7e20 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96cc1eef50 con 0x7f96cc0c7e20 2023-10-14T18:19:15.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96c4004a20 con 0x7f96cc0c7e20 2023-10-14T18:19:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.150+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96cc14dfc0 con 0x7f96cc0c7e20 2023-10-14T18:19:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.152+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96cc1ef470 con 0x7f96cc0c7e20 2023-10-14T18:19:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.153+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f96c4004b80 con 0x7f96cc0c7e20 2023-10-14T18:19:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.154+0000 7f96c97fa640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 0x7f96b806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:15.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.154+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7635+0+0 (secure 0 0 0) 0x7f96c4089b20 con 0x7f96cc0c7e20 2023-10-14T18:19:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.154+0000 7f96d082c640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 0x7f96b806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.155+0000 7f96d082c640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 0x7f96b806d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96cc069740 tx=0x7f96bc008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:15.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.158+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f96cc1ef470 con 0x7f96cc0c7e20 2023-10-14T18:19:15.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.395+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f96cc1ef470 con 0x7f96cc0c7e20 2023-10-14T18:19:15.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.910+0000 7f96c97fa640 1 -- 172.21.15.29:0/3676576839 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v131) v1 ==== 104+0+0 (secure 0 0 0) 0x7f96cc1ef470 con 0x7f96cc0c7e20 2023-10-14T18:19:15.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.913+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 msgr2=0x7f96b806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.913+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 0x7f96b806d570 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f96cc069740 tx=0x7f96bc008040 comp rx=0 tx=0).stop 2023-10-14T18:19:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.913+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 msgr2=0x7f96cc0c82a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:15.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.913+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 secure :-1 s=READY pgs=3043 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0b15a0 tx=0x7f96c400c8e0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 shutdown_connections 2023-10-14T18:19:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f96b806b0b0 0x7f96b806d570 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f96cc0c7e20 0x7f96cc0c82a0 unknown :-1 s=CLOSED pgs=3043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f96cc06c590 0x7f96cc06c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 --2- 172.21.15.29:0/3676576839 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f96b40069d0 0x7f96cc14da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 >> 172.21.15.29:0/3676576839 conn(0x7f96cc0ce9b0 msgr2=0x7f96cc0ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 shutdown_connections 2023-10-14T18:19:15.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:15.914+0000 7f96d2a90640 1 -- 172.21.15.29:0/3676576839 wait complete. 2023-10-14T18:19:15.935 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T18:19:16.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.163+0000 7f9310da0640 1 Processor -- start 2023-10-14T18:19:16.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.163+0000 7f9310da0640 1 -- start start 2023-10-14T18:19:16.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.164+0000 7f9310da0640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f930c15cd70 0x7f930c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.164+0000 7f9310da0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.164+0000 7f9310da0640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f930c15d150 con 0x7f930c0d0be0 2023-10-14T18:19:16.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.164+0000 7f9310da0640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f930c0d1180 con 0x7f930c15cd70 2023-10-14T18:19:16.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.164+0000 7f9310da0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c0d1460 con 0x7f930c0b8480 2023-10-14T18:19:16.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.165+0000 7f930affd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:35824/0 (socket says 172.21.15.29:35824) 2023-10-14T18:19:16.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930b7fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f930c15cd70 0x7f930c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 -- 172.21.15.29:0/4160645387 learned_addr learned my addr 172.21.15.29:0/4160645387 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 -- 172.21.15.29:0/4160645387 >> v2:172.21.15.186:3300/0 conn(0x7f930c15cd70 msgr2=0x7f930c0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 --2- 172.21.15.29:0/4160645387 >> v2:172.21.15.186:3300/0 conn(0x7f930c15cd70 0x7f930c0d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 -- 172.21.15.29:0/4160645387 >> v1:172.21.15.186:6789/0 conn(0x7f930c0d0be0 legacy=0x7f930c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.166+0000 7f930affd640 1 -- 172.21.15.29:0/4160645387 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f930c0d1740 con 0x7f930c0b8480 2023-10-14T18:19:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.168+0000 7f930affd640 1 --2- 172.21.15.29:0/4160645387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009870 tx=0x7f92f402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bab7b589fb24be4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:16.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.168+0000 7f930a7fc640 1 -- 172.21.15.29:0/4160645387 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f4030840 con 0x7f930c0b8480 2023-10-14T18:19:16.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.168+0000 7f930a7fc640 1 -- 172.21.15.29:0/4160645387 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92f40309a0 con 0x7f930c0b8480 2023-10-14T18:19:16.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.168+0000 7f930a7fc640 1 -- 172.21.15.29:0/4160645387 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92f4030d30 con 0x7f930c0b8480 2023-10-14T18:19:16.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 -- 172.21.15.29:0/4160645387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 msgr2=0x7f930c0cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 --2- 172.21.15.29:0/4160645387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f92f4009870 tx=0x7f92f402b980 comp rx=0 tx=0).stop 2023-10-14T18:19:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 -- 172.21.15.29:0/4160645387 shutdown_connections 2023-10-14T18:19:16.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 --2- 172.21.15.29:0/4160645387 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c0cd7c0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 --2- 172.21.15.29:0/4160645387 >> v2:172.21.15.186:3300/0 conn(0x7f930c15cd70 0x7f930c0d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 -- 172.21.15.29:0/4160645387 >> 172.21.15.29:0/4160645387 conn(0x7f930c0cf570 msgr2=0x7f930c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 -- 172.21.15.29:0/4160645387 shutdown_connections 2023-10-14T18:19:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.169+0000 7f9310da0640 1 -- 172.21.15.29:0/4160645387 wait complete. 2023-10-14T18:19:16.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.171+0000 7f9310da0640 1 Processor -- start 2023-10-14T18:19:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.171+0000 7f9310da0640 1 -- start start 2023-10-14T18:19:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c15d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f930c0d0be0 0x7f930c15b1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c15e4c0 con 0x7f930c15cd70 2023-10-14T18:19:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f930c1f3680 con 0x7f930c0d0be0 2023-10-14T18:19:16.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.172+0000 7f9310da0640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f930c15d940 con 0x7f930c0b8480 2023-10-14T18:19:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36458/0 (socket says 172.21.15.29:36458) 2023-10-14T18:19:16.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 -- 172.21.15.29:0/201280550 learned_addr learned my addr 172.21.15.29:0/201280550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:16.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930affd640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f930c0d0be0 0x7f930c15b1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930b7fe640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c15d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 -- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 msgr2=0x7f930c15d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c15d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 -- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f930c0d0be0 msgr2=0x7f930c15b1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f930c0d0be0 0x7f930c15b1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 -- 172.21.15.29:0/201280550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92f40094d0 con 0x7f930c15cd70 2023-10-14T18:19:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.173+0000 7f930bfff640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f930400c3c0 tx=0x7f930400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.174+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93040052d0 con 0x7f930c15cd70 2023-10-14T18:19:16.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.174+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9304005430 con 0x7f930c15cd70 2023-10-14T18:19:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.174+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f930c15dc20 con 0x7f930c15cd70 2023-10-14T18:19:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.174+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9304003830 con 0x7f930c15cd70 2023-10-14T18:19:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.174+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f930c15e1e0 con 0x7f930c15cd70 2023-10-14T18:19:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.175+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f930c04e9b0 con 0x7f930c15cd70 2023-10-14T18:19:16.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.176+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9304003990 con 0x7f930c15cd70 2023-10-14T18:19:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.177+0000 7f9303fff640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 0x7f92ec06d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.177+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(257..257 src has 1..257) v4 ==== 7635+0+0 (secure 0 0 0) 0x7f9304089580 con 0x7f930c15cd70 2023-10-14T18:19:16.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.178+0000 7f930b7fe640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 0x7f92ec06d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.178+0000 7f930b7fe640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 0x7f92ec06d5a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0074a0 tx=0x7f92fc00b040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:16.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.182+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9304053c10 con 0x7f930c15cd70 2023-10-14T18:19:16.379 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.377+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f930c06c700 con 0x7f930c15cd70 2023-10-14T18:19:16.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.913+0000 7f9303fff640 1 -- 172.21.15.29:0/201280550 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v258) v1 ==== 181+0+0 (secure 0 0 0) 0x7f9304057230 con 0x7f930c15cd70 2023-10-14T18:19:16.916 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T18:19:16.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.917+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 msgr2=0x7f92ec06d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.917+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 0x7f92ec06d5a0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f92fc0074a0 tx=0x7f92fc00b040 comp rx=0 tx=0).stop 2023-10-14T18:19:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 msgr2=0x7f930c1f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:16.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 secure :-1 s=READY pgs=3044 cs=0 l=1 rev1=1 crypto rx=0x7f930400c3c0 tx=0x7f930400c880 comp rx=0 tx=0).stop 2023-10-14T18:19:16.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 shutdown_connections 2023-10-14T18:19:16.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f92ec06b0e0 0x7f92ec06d5a0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f930c15cd70 0x7f930c1f3140 unknown :-1 s=CLOSED pgs=3044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f930c0d0be0 0x7f930c15b1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 --2- 172.21.15.29:0/201280550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f930c0b8480 0x7f930c15d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:16.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 >> 172.21.15.29:0/201280550 conn(0x7f930c0cf570 msgr2=0x7f930c0b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.918+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 shutdown_connections 2023-10-14T18:19:16.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:16.919+0000 7f9310da0640 1 -- 172.21.15.29:0/201280550 wait complete. 2023-10-14T18:19:16.942 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T18:19:17.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.173+0000 7fbf08820640 1 Processor -- start 2023-10-14T18:19:17.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.174+0000 7fbf08820640 1 -- start start 2023-10-14T18:19:17.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.175+0000 7fbf08820640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 0x7fbf0406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.175+0000 7fbf08820640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbf040cda80 0x7fbf0415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.175+0000 7fbf08820640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf0406c960 con 0x7fbf040ad970 2023-10-14T18:19:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.175+0000 7fbf08820640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf0415b9d0 con 0x7fbf040cda80 2023-10-14T18:19:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.175+0000 7fbf08820640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0415bcb0 con 0x7fbf0406c040 2023-10-14T18:19:17.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbefbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbf040ad970 0x7fbf0415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58456/0 (socket says 172.21.15.29:58456) 2023-10-14T18:19:17.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbefbfff640 1 -- 172.21.15.29:0/3595075102 learned_addr learned my addr 172.21.15.29:0/3595075102 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:17.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbf0359e640 1 --2- 172.21.15.29:0/3595075102 >> v2:172.21.15.186:3300/0 conn(0x7fbf040cda80 0x7fbf0415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 0x7fbf0406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 344822745 0 0) 0x7fbf0406c960 con 0x7fbf040ad970 2023-10-14T18:19:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbef0003520 con 0x7fbf040ad970 2023-10-14T18:19:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.176+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3106112845 0 0) 0x7fbef0003520 con 0x7fbf040ad970 2023-10-14T18:19:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 msgr2=0x7fbf0406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:17.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 0x7fbf0406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 >> v2:172.21.15.186:3300/0 conn(0x7fbf040cda80 msgr2=0x7fbf0415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 --2- 172.21.15.29:0/3595075102 >> v2:172.21.15.186:3300/0 conn(0x7fbf040cda80 0x7fbf0415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf0415bf90 con 0x7fbf040ad970 2023-10-14T18:19:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fbeec001670 con 0x7fbf040ad970 2023-10-14T18:19:17.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 >> v1:172.21.15.29:6789/0 conn(0x7fbf040ad970 legacy=0x7fbf0415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef0003970 0x7fbef0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 0x7fbef00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbef0004d00 0x7fbef0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0415b9d0 con 0x7fbef0003970 2023-10-14T18:19:17.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf0415bcb0 con 0x7fbef0004d00 2023-10-14T18:19:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbef000b980 con 0x7fbf0406c040 2023-10-14T18:19:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef0003970 0x7fbef0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf0359e640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbef0004d00 0x7fbef0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf02d9d640 1 -- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 msgr2=0x7fbef00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:17.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.177+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbf0406c040 0x7fbef00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.178+0000 7fbf02d9d640 1 -- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbef0004d00 msgr2=0x7fbef0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.178+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbef0004d00 0x7fbef0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.178+0000 7fbf02d9d640 1 -- 172.21.15.29:0/3595075102 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbef0003520 con 0x7fbef0003970 2023-10-14T18:19:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.179+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef0003970 0x7fbef0003d90 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009ac0 tx=0x7fbef00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77c73cb613aa71b7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:17.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.179+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef402c710 con 0x7fbef0003970 2023-10-14T18:19:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.179+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbef402c870 con 0x7fbef0003970 2023-10-14T18:19:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.179+0000 7fbf0259c640 1 -- 172.21.15.29:0/3595075102 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbef402cc00 con 0x7fbef0003970 2023-10-14T18:19:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 -- 172.21.15.29:0/3595075102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef0003970 msgr2=0x7fbef0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 --2- 172.21.15.29:0/3595075102 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbef0003970 0x7fbef0003d90 secure :-1 s=READY pgs=3046 cs=0 l=1 rev1=1 crypto rx=0x7fbef4009ac0 tx=0x7fbef00057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf0359e640 1 -- 172.21.15.29:0/3595075102 reap_dead start 2023-10-14T18:19:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 -- 172.21.15.29:0/3595075102 shutdown_connections 2023-10-14T18:19:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 -- 172.21.15.29:0/3595075102 >> 172.21.15.29:0/3595075102 conn(0x7fbf040ce6a0 msgr2=0x7fbf040ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 -- 172.21.15.29:0/3595075102 shutdown_connections 2023-10-14T18:19:17.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.180+0000 7fbf08820640 1 -- 172.21.15.29:0/3595075102 wait complete. 2023-10-14T18:19:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.182+0000 7fbf08820640 1 Processor -- start 2023-10-14T18:19:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.182+0000 7fbf08820640 1 -- start start 2023-10-14T18:19:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbef0003970 0x7fbf040ca790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf040c9270 0x7fbf040ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040cacd0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040c96f0 con 0x7fbf040c9270 2023-10-14T18:19:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf08820640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf040c99d0 con 0x7fbef0003970 2023-10-14T18:19:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbefbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbefbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36466/0 (socket says 172.21.15.29:36466) 2023-10-14T18:19:17.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbefbfff640 1 -- 172.21.15.29:0/995149720 learned_addr learned my addr 172.21.15.29:0/995149720 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf0359e640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf040c9270 0x7fbf040ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.183+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbef0003970 0x7fbf040ca790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 -- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbef0003970 msgr2=0x7fbf040ca790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbef0003970 0x7fbf040ca790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 -- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf040c9270 msgr2=0x7fbf040ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:17.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf040c9270 0x7fbf040ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:17.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 -- 172.21.15.29:0/995149720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf040c9cb0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefbfff640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c750 tx=0x7fbeec00cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:17.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec0074e0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeec007660 con 0x7fbf0414a9d0 2023-10-14T18:19:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf040c9f90 con 0x7fbf0414a9d0 2023-10-14T18:19:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.184+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbeec004a20 con 0x7fbf0414a9d0 2023-10-14T18:19:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.185+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf040c58c0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.185+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0406b330 con 0x7fbf0414a9d0 2023-10-14T18:19:17.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.187+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbeec004b80 con 0x7fbf0414a9d0 2023-10-14T18:19:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.187+0000 7fbefb7fe640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 0x7fbee406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.188+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(258..258 src has 1..258) v4 ==== 7197+0+0 (secure 0 0 0) 0x7fbeec0898a0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.188+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 0x7fbee406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:17.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.189+0000 7fbf02d9d640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 0x7fbee406d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbef4036000 tx=0x7fbef4030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:17.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.192+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbeec057a10 con 0x7fbf0414a9d0 2023-10-14T18:19:17.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:17.392+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fbf040ca4b0 con 0x7fbf0414a9d0 2023-10-14T18:19:17.502 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:19:17.511 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:19:18.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.274+0000 7fbefb7fe640 1 -- 172.21.15.29:0/995149720 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v259) v1 ==== 169+0+0 (secure 0 0 0) 0x7fbf040ca4b0 con 0x7fbf0414a9d0 2023-10-14T18:19:18.278 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T18:19:18.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 msgr2=0x7fbee406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 0x7fbee406d570 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbef4036000 tx=0x7fbef4030000 comp rx=0 tx=0).stop 2023-10-14T18:19:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 msgr2=0x7fbf040c8d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 secure :-1 s=READY pgs=3047 cs=0 l=1 rev1=1 crypto rx=0x7fbeec00c750 tx=0x7fbeec00cc10 comp rx=0 tx=0).stop 2023-10-14T18:19:18.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 shutdown_connections 2023-10-14T18:19:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbee406b0b0 0x7fbee406d570 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbf040c9270 0x7fbf040ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbf0414a9d0 0x7fbf040c8d30 unknown :-1 s=CLOSED pgs=3047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 --2- 172.21.15.29:0/995149720 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbef0003970 0x7fbf040ca790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 >> 172.21.15.29:0/995149720 conn(0x7fbf040ce6a0 msgr2=0x7fbf0406f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 shutdown_connections 2023-10-14T18:19:18.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.278+0000 7fbf08820640 1 -- 172.21.15.29:0/995149720 wait complete. 2023-10-14T18:19:18.302 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:18.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.531+0000 7f2fce316640 1 Processor -- start 2023-10-14T18:19:18.534 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.531+0000 7f2fce316640 1 -- start start 2023-10-14T18:19:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fce316640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815c9f0 0x7f2fc80c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fce316640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fc815dde0 0x7f2fc80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.535 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fce316640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fc8160a90 con 0x7f2fc80c9230 2023-10-14T18:19:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fce316640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2fc80cdec0 con 0x7f2fc815dde0 2023-10-14T18:19:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fce316640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80cd900 con 0x7f2fc815c9f0 2023-10-14T18:19:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fcdb15640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fc815dde0 0x7f2fc80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fcdb15640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2fc815dde0 0x7f2fc80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36480/0 (socket says 172.21.15.29:36480) 2023-10-14T18:19:18.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.533+0000 7f2fcdb15640 1 -- 172.21.15.29:0/1832055134 learned_addr learned my addr 172.21.15.29:0/1832055134 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:18.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.534+0000 7f2fcd314640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815c9f0 0x7f2fc80c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.534+0000 7f2fcdb15640 1 -- 172.21.15.29:0/1832055134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815c9f0 msgr2=0x7f2fc80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.534+0000 7f2fcdb15640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815c9f0 0x7f2fc80c8cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.534+0000 7f2fcdb15640 1 -- 172.21.15.29:0/1832055134 >> v1:172.21.15.186:6789/0 conn(0x7f2fc80c9230 legacy=0x7f2fc80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.534+0000 7f2fcdb15640 1 -- 172.21.15.29:0/1832055134 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc80cdbe0 con 0x7f2fc815dde0 2023-10-14T18:19:18.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.535+0000 7f2fcdb15640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815dde0 0x7f2fc80ac890 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009870 tx=0x7f2fc002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f520a1d58e42d5ca server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.535+0000 7f2fbffff640 1 -- 172.21.15.29:0/1832055134 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031640 con 0x7f2fc815dde0 2023-10-14T18:19:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.535+0000 7f2fbffff640 1 -- 172.21.15.29:0/1832055134 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fc00317a0 con 0x7f2fc815dde0 2023-10-14T18:19:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.535+0000 7f2fbffff640 1 -- 172.21.15.29:0/1832055134 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fc0031b30 con 0x7f2fc815dde0 2023-10-14T18:19:18.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 -- 172.21.15.29:0/1832055134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815dde0 msgr2=0x7f2fc80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815dde0 0x7f2fc80ac890 secure :-1 s=READY pgs=3048 cs=0 l=1 rev1=1 crypto rx=0x7f2fc0009870 tx=0x7f2fc002b8f0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 -- 172.21.15.29:0/1832055134 shutdown_connections 2023-10-14T18:19:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815dde0 0x7f2fc80ac890 unknown :-1 s=CLOSED pgs=3048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 --2- 172.21.15.29:0/1832055134 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815c9f0 0x7f2fc80c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 -- 172.21.15.29:0/1832055134 >> 172.21.15.29:0/1832055134 conn(0x7f2fc80ce6a0 msgr2=0x7f2fc80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 -- 172.21.15.29:0/1832055134 shutdown_connections 2023-10-14T18:19:18.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.536+0000 7f2fce316640 1 -- 172.21.15.29:0/1832055134 wait complete. 2023-10-14T18:19:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.538+0000 7f2fce316640 1 Processor -- start 2023-10-14T18:19:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.538+0000 7f2fce316640 1 -- start start 2023-10-14T18:19:18.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fc80c9230 0x7f2fc81504e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 0x7f2fc806c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc8150a20 con 0x7f2fc815c9f0 2023-10-14T18:19:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc80c4e20 con 0x7f2fc80c9230 2023-10-14T18:19:18.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.539+0000 7f2fce316640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2fc806b220 con 0x7f2fc815dde0 2023-10-14T18:19:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36482/0 (socket says 172.21.15.29:36482) 2023-10-14T18:19:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 -- 172.21.15.29:0/1011582792 learned_addr learned my addr 172.21.15.29:0/1011582792 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fcd314640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fc80c9230 0x7f2fc81504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fcdb15640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 0x7f2fc806c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 -- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 msgr2=0x7f2fc806c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 0x7f2fc806c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 -- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fc80c9230 msgr2=0x7f2fc81504e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fc80c9230 0x7f2fc81504e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fccb13640 1 -- 172.21.15.29:0/1011582792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fc00094d0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.540+0000 7f2fcdb15640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 0x7f2fc806c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fccb13640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800c460 tx=0x7f2fb800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011790 con 0x7f2fc815c9f0 2023-10-14T18:19:18.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2fb8011930 con 0x7f2fc815c9f0 2023-10-14T18:19:18.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8011ce0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fc8069f20 con 0x7f2fc815c9f0 2023-10-14T18:19:18.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.541+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fc806a4e0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.542+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fc804e9b0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.543+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2fb801c460 con 0x7f2fc815c9f0 2023-10-14T18:19:18.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.544+0000 7f2fbdffb640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:18.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.544+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f2fb8089cd0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.544+0000 7f2fcd314640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:18.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.545+0000 7f2fcd314640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 0x7f2fac06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009a60 tx=0x7f2fc4003990 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:18.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.549+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2fb80546b0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.744+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fc80adac0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.747+0000 7f2fbdffb640 1 -- 172.21.15.29:0/1011582792 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+18088 (secure 0 0 0) 0x7f2fb8057cd0 con 0x7f2fc815c9f0 2023-10-14T18:19:18.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:18.750 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":259,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:18.272191+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6833/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.186:6832/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 msgr2=0x7f2fac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 0x7f2fac06d570 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4009a60 tx=0x7f2fc4003990 comp rx=0 tx=0).stop 2023-10-14T18:19:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 msgr2=0x7f2fc806ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 secure :-1 s=READY pgs=3049 cs=0 l=1 rev1=1 crypto rx=0x7f2fb800c460 tx=0x7f2fb800c920 comp rx=0 tx=0).stop 2023-10-14T18:19:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 shutdown_connections 2023-10-14T18:19:18.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fac06b0b0 0x7f2fac06d570 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2fc815dde0 0x7f2fc806c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2fc815c9f0 0x7f2fc806ace0 unknown :-1 s=CLOSED pgs=3049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 --2- 172.21.15.29:0/1011582792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2fc80c9230 0x7f2fc81504e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.750+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 >> 172.21.15.29:0/1011582792 conn(0x7f2fc80ce6a0 msgr2=0x7f2fc815e280 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:18.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.751+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 shutdown_connections 2023-10-14T18:19:18.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:18.751+0000 7f2fce316640 1 -- 172.21.15.29:0/1011582792 wait complete. 2023-10-14T18:19:18.772 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 32, 'pool_name': 'backup_fs_metadata', 'create_time': '2023-10-14T18:18:47.629053+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '253', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'backup_fs'}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 33, 'pool_name': 'backup_fs_data', 'create_time': '2023-10-14T18:18:48.796314+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '254', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'backup_fs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:18.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:19.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.003+0000 7f8be76bb640 1 Processor -- start 2023-10-14T18:19:19.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.003+0000 7f8be76bb640 1 -- start start 2023-10-14T18:19:19.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.004+0000 7f8be76bb640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8be015d150 0x7f8be00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.004+0000 7f8be76bb640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8be0160b90 0x7f8be0160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be76bb640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8be00ac890 con 0x7f8be0069da0 2023-10-14T18:19:19.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be76bb640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be015d530 con 0x7f8be015d150 2023-10-14T18:19:19.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be76bb640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8be0161650 con 0x7f8be0160b90 2023-10-14T18:19:19.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8be015d150 0x7f8be00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8be015d150 0x7f8be00ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36498/0 (socket says 172.21.15.29:36498) 2023-10-14T18:19:19.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 -- 172.21.15.29:0/282102973 learned_addr learned my addr 172.21.15.29:0/282102973 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 -- 172.21.15.29:0/282102973 >> v2:172.21.15.186:3300/0 conn(0x7f8be0160b90 msgr2=0x7f8be0160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 --2- 172.21.15.29:0/282102973 >> v2:172.21.15.186:3300/0 conn(0x7f8be0160b90 0x7f8be0160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 -- 172.21.15.29:0/282102973 >> v1:172.21.15.186:6789/0 conn(0x7f8be0069da0 legacy=0x7f8be00c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.005+0000 7f8be5457640 1 -- 172.21.15.29:0/282102973 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8be0161930 con 0x7f8be015d150 2023-10-14T18:19:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.006+0000 7f8be5457640 1 --2- 172.21.15.29:0/282102973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be015d150 0x7f8be00ad350 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f8be00cf990 tx=0x7f8bdc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fda0cf249cce3b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.007+0000 7f8bd7fff640 1 -- 172.21.15.29:0/282102973 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc0317e0 con 0x7f8be015d150 2023-10-14T18:19:19.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.007+0000 7f8bd7fff640 1 -- 172.21.15.29:0/282102973 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bdc031940 con 0x7f8be015d150 2023-10-14T18:19:19.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.007+0000 7f8bd7fff640 1 -- 172.21.15.29:0/282102973 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bdc031cd0 con 0x7f8be015d150 2023-10-14T18:19:19.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.007+0000 7f8be76bb640 1 -- 172.21.15.29:0/282102973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be015d150 msgr2=0x7f8be00ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.007+0000 7f8be76bb640 1 --2- 172.21.15.29:0/282102973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be015d150 0x7f8be00ad350 secure :-1 s=READY pgs=3050 cs=0 l=1 rev1=1 crypto rx=0x7f8be00cf990 tx=0x7f8bdc02b880 comp rx=0 tx=0).stop 2023-10-14T18:19:19.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 -- 172.21.15.29:0/282102973 shutdown_connections 2023-10-14T18:19:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 --2- 172.21.15.29:0/282102973 >> v2:172.21.15.186:3300/0 conn(0x7f8be0160b90 0x7f8be0160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 --2- 172.21.15.29:0/282102973 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be015d150 0x7f8be00ad350 unknown :-1 s=CLOSED pgs=3050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 -- 172.21.15.29:0/282102973 >> 172.21.15.29:0/282102973 conn(0x7f8be00ce9b0 msgr2=0x7f8be00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 -- 172.21.15.29:0/282102973 shutdown_connections 2023-10-14T18:19:19.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.008+0000 7f8be76bb640 1 -- 172.21.15.29:0/282102973 wait complete. 2023-10-14T18:19:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.010+0000 7f8be76bb640 1 Processor -- start 2023-10-14T18:19:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.010+0000 7f8be76bb640 1 -- start start 2023-10-14T18:19:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 0x7f8be014bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8be015d150 0x7f8be00c6e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be00c5a80 con 0x7f8be0160b90 2023-10-14T18:19:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8be00c5d60 con 0x7f8be015d150 2023-10-14T18:19:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be76bb640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8be00c6040 con 0x7f8be0069da0 2023-10-14T18:19:19.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be5c58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be5c58640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36502/0 (socket says 172.21.15.29:36502) 2023-10-14T18:19:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be5c58640 1 -- 172.21.15.29:0/59732578 learned_addr learned my addr 172.21.15.29:0/59732578 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be4c56640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8be015d150 0x7f8be00c6e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.011+0000 7f8be5457640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 0x7f8be014bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.017 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 -- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 msgr2=0x7f8be014bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 0x7f8be014bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 -- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8be015d150 msgr2=0x7f8be00c6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8be015d150 0x7f8be00c6e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 -- 172.21.15.29:0/59732578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bdc0094d0 con 0x7f8be0160b90 2023-10-14T18:19:19.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5457640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 0x7f8be014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be5c58640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800b990 tx=0x7f8bd800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd800cba0 con 0x7f8be0160b90 2023-10-14T18:19:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bd800cd00 con 0x7f8be0160b90 2023-10-14T18:19:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8be00c6320 con 0x7f8be0160b90 2023-10-14T18:19:19.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8bd8014680 con 0x7f8be0160b90 2023-10-14T18:19:19.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.012+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8be00c68e0 con 0x7f8be0160b90 2023-10-14T18:19:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.014+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8be004e160 con 0x7f8be0160b90 2023-10-14T18:19:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.014+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8bd80147e0 con 0x7f8be0160b90 2023-10-14T18:19:19.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.015+0000 7f8bd5ffb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 0x7f8bc406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.016+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f8bd8089dc0 con 0x7f8be0160b90 2023-10-14T18:19:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.016+0000 7f8be5457640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 0x7f8bc406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.022 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.017+0000 7f8be5457640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 0x7f8bc406d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc036000 tx=0x7f8bdc03d040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.023 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.020+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8bd80546d0 con 0x7f8be0160b90 2023-10-14T18:19:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.213+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f8be00ade10 con 0x7f8be0160b90 2023-10-14T18:19:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.214+0000 7f8bd5ffb640 1 -- 172.21.15.29:0/59732578 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v259) v1 ==== 93+0+10 (secure 0 0 0) 0x7f8bd8019020 con 0x7f8be0160b90 2023-10-14T18:19:19.216 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 msgr2=0x7f8bc406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 0x7f8bc406d570 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f8bdc036000 tx=0x7f8bdc03d040 comp rx=0 tx=0).stop 2023-10-14T18:19:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 msgr2=0x7f8be00b00b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 secure :-1 s=READY pgs=3051 cs=0 l=1 rev1=1 crypto rx=0x7f8bd800b990 tx=0x7f8bd800be50 comp rx=0 tx=0).stop 2023-10-14T18:19:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 shutdown_connections 2023-10-14T18:19:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bc406b0b0 0x7f8bc406d570 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8be0160b90 0x7f8be00b00b0 unknown :-1 s=CLOSED pgs=3051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8be015d150 0x7f8be00c6e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 --2- 172.21.15.29:0/59732578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8be0069da0 0x7f8be014bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 >> 172.21.15.29:0/59732578 conn(0x7f8be00ce9b0 msgr2=0x7f8be00708f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 shutdown_connections 2023-10-14T18:19:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.217+0000 7f8be76bb640 1 -- 172.21.15.29:0/59732578 wait complete. 2023-10-14T18:19:19.237 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_metadata pg_num 2023-10-14T18:19:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.471+0000 7f6ad8ee7640 1 Processor -- start 2023-10-14T18:19:19.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.471+0000 7f6ad8ee7640 1 -- start start 2023-10-14T18:19:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ad415d150 0x7f6ad40ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6ad4160b90 0x7f6ad4160fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad406a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad415d530 con 0x7f6ad415d150 2023-10-14T18:19:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ad406a730 con 0x7f6ad4160b90 2023-10-14T18:19:19.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad8ee7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40ac890 con 0x7f6ad4069da0 2023-10-14T18:19:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ad415d150 0x7f6ad40ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.473+0000 7f6ad259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6ad415d150 0x7f6ad40ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36514/0 (socket says 172.21.15.29:36514) 2023-10-14T18:19:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 -- 172.21.15.29:0/4210901388 learned_addr learned my addr 172.21.15.29:0/4210901388 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad2d9d640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad406a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6acbfff640 1 --2- 172.21.15.29:0/4210901388 >> v2:172.21.15.186:3300/0 conn(0x7f6ad4160b90 0x7f6ad4160fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 -- 172.21.15.29:0/4210901388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 msgr2=0x7f6ad406a1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad406a1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 -- 172.21.15.29:0/4210901388 >> v2:172.21.15.186:3300/0 conn(0x7f6ad4160b90 msgr2=0x7f6ad4160fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 --2- 172.21.15.29:0/4210901388 >> v2:172.21.15.186:3300/0 conn(0x7f6ad4160b90 0x7f6ad4160fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.474+0000 7f6ad259c640 1 -- 172.21.15.29:0/4210901388 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ad4161650 con 0x7f6ad415d150 2023-10-14T18:19:19.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.475+0000 7f6ad259c640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad415d150 0x7f6ad40ad350 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40cf990 tx=0x7f6ac402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf53f4c8a4c452b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad1d9b640 1 -- 172.21.15.29:0/4210901388 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac40317e0 con 0x7f6ad415d150 2023-10-14T18:19:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad1d9b640 1 -- 172.21.15.29:0/4210901388 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ac4031940 con 0x7f6ad415d150 2023-10-14T18:19:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad1d9b640 1 -- 172.21.15.29:0/4210901388 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4031cd0 con 0x7f6ad415d150 2023-10-14T18:19:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/4210901388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad415d150 msgr2=0x7f6ad40ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad415d150 0x7f6ad40ad350 secure :-1 s=READY pgs=3052 cs=0 l=1 rev1=1 crypto rx=0x7f6ad40cf990 tx=0x7f6ac402b880 comp rx=0 tx=0).stop 2023-10-14T18:19:19.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/4210901388 shutdown_connections 2023-10-14T18:19:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad406a1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/4210901388 >> v2:172.21.15.186:3300/0 conn(0x7f6ad4160b90 0x7f6ad4160fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/4210901388 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad415d150 0x7f6ad40ad350 unknown :-1 s=CLOSED pgs=3052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/4210901388 >> 172.21.15.29:0/4210901388 conn(0x7f6ad40ce9b0 msgr2=0x7f6ad40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:19.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/4210901388 shutdown_connections 2023-10-14T18:19:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.476+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/4210901388 wait complete. 2023-10-14T18:19:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.478+0000 7f6ad8ee7640 1 Processor -- start 2023-10-14T18:19:19.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.478+0000 7f6ad8ee7640 1 -- start start 2023-10-14T18:19:19.485 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad4151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ad415d150 0x7f6ad40ae240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad4151ad0 con 0x7f6ad4160b90 2023-10-14T18:19:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40aec90 con 0x7f6ad415d150 2023-10-14T18:19:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.479+0000 7f6ad8ee7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ad40aef70 con 0x7f6ad4069da0 2023-10-14T18:19:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.487 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36530/0 (socket says 172.21.15.29:36530) 2023-10-14T18:19:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 -- 172.21.15.29:0/1502366846 learned_addr learned my addr 172.21.15.29:0/1502366846 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad259c640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad4151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 -- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 msgr2=0x7f6ad4151590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6acbfff640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ad415d150 0x7f6ad40ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.488 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad4151590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 -- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ad415d150 msgr2=0x7f6ad40ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ad415d150 0x7f6ad40ae240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad2d9d640 1 -- 172.21.15.29:0/1502366846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac40094d0 con 0x7f6ad4160b90 2023-10-14T18:19:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.480+0000 7f6ad259c640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad4151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:19.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6ad2d9d640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00eed0 tx=0x7f6acc00b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc010040 con 0x7f6ad4160b90 2023-10-14T18:19:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6acc00bd20 con 0x7f6ad4160b90 2023-10-14T18:19:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6acc014640 con 0x7f6ad4160b90 2023-10-14T18:19:19.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ad40af250 con 0x7f6ad4160b90 2023-10-14T18:19:19.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.481+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ad415e880 con 0x7f6ad4160b90 2023-10-14T18:19:19.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.482+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ad415eaa0 con 0x7f6ad4160b90 2023-10-14T18:19:19.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.483+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f6acc0147a0 con 0x7f6ad4160b90 2023-10-14T18:19:19.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.484+0000 7f6acaffd640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 0x7f6ab406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.484+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f6acc08aa20 con 0x7f6ad4160b90 2023-10-14T18:19:19.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.485+0000 7f6ad259c640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 0x7f6ab406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.485+0000 7f6ad259c640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 0x7f6ab406d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4036000 tx=0x7f6ac403d040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.489+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6acc055190 con 0x7f6ad4160b90 2023-10-14T18:19:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.688+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6ad404e160 con 0x7f6ad4160b90 2023-10-14T18:19:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.689+0000 7f6acaffd640 1 -- 172.21.15.29:0/1502366846 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_metadata", "var": "pg_num"}]=0 v259) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6acc018020 con 0x7f6ad4160b90 2023-10-14T18:19:19.691 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.691+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 msgr2=0x7f6ab406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.691+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 0x7f6ab406d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6ac4036000 tx=0x7f6ac403d040 comp rx=0 tx=0).stop 2023-10-14T18:19:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.691+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 msgr2=0x7f6ad41f61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.691+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 secure :-1 s=READY pgs=3053 cs=0 l=1 rev1=1 crypto rx=0x7f6acc00eed0 tx=0x7f6acc00b480 comp rx=0 tx=0).stop 2023-10-14T18:19:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 shutdown_connections 2023-10-14T18:19:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6ab406b0b0 0x7f6ab406d570 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ad4160b90 0x7f6ad41f61d0 unknown :-1 s=CLOSED pgs=3053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ad415d150 0x7f6ad40ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 --2- 172.21.15.29:0/1502366846 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ad4069da0 0x7f6ad4151590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 >> 172.21.15.29:0/1502366846 conn(0x7f6ad40ce9b0 msgr2=0x7f6ad40c8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 shutdown_connections 2023-10-14T18:19:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.692+0000 7f6ad8ee7640 1 -- 172.21.15.29:0/1502366846 wait complete. 2023-10-14T18:19:19.713 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get backup_fs_data pg_num 2023-10-14T18:19:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.936+0000 7eff12a4d640 1 Processor -- start 2023-10-14T18:19:19.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.936+0000 7eff12a4d640 1 -- start start 2023-10-14T18:19:19.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.938+0000 7eff12a4d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.938+0000 7eff12a4d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7eff0c0b8480 0x7eff0c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.938+0000 7eff12a4d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff0c15d150 con 0x7eff0c0d0be0 2023-10-14T18:19:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.938+0000 7eff12a4d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff0c0d1180 con 0x7eff0c0b8480 2023-10-14T18:19:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.938+0000 7eff12a4d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c0d1460 con 0x7eff0c15cd70 2023-10-14T18:19:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:35924/0 (socket says 172.21.15.29:35924) 2023-10-14T18:19:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 -- 172.21.15.29:0/2885353098 learned_addr learned my addr 172.21.15.29:0/2885353098 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff1124a640 1 --2- 172.21.15.29:0/2885353098 >> v2:172.21.15.186:3300/0 conn(0x7eff0c0b8480 0x7eff0c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 -- 172.21.15.29:0/2885353098 >> v2:172.21.15.186:3300/0 conn(0x7eff0c0b8480 msgr2=0x7eff0c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 --2- 172.21.15.29:0/2885353098 >> v2:172.21.15.186:3300/0 conn(0x7eff0c0b8480 0x7eff0c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 -- 172.21.15.29:0/2885353098 >> v1:172.21.15.186:6789/0 conn(0x7eff0c0d0be0 legacy=0x7eff0c0b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff11a4b640 1 -- 172.21.15.29:0/2885353098 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c0d1740 con 0x7eff0c15cd70 2023-10-14T18:19:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.939+0000 7eff10a49640 1 -- 172.21.15.29:0/2885353098 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471100366 0 0) 0x7eff0c15d150 con 0x7eff0c0d0be0 2023-10-14T18:19:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.941+0000 7eff11a4b640 1 --2- 172.21.15.29:0/2885353098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6b3e24664c986dbb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff10a49640 1 -- 172.21.15.29:0/2885353098 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08030840 con 0x7eff0c15cd70 2023-10-14T18:19:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff10a49640 1 -- 172.21.15.29:0/2885353098 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff080309a0 con 0x7eff0c15cd70 2023-10-14T18:19:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff10a49640 1 -- 172.21.15.29:0/2885353098 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff08030d30 con 0x7eff0c15cd70 2023-10-14T18:19:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 -- 172.21.15.29:0/2885353098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 msgr2=0x7eff0c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2885353098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7eff08009870 tx=0x7eff0802b980 comp rx=0 tx=0).stop 2023-10-14T18:19:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 -- 172.21.15.29:0/2885353098 shutdown_connections 2023-10-14T18:19:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2885353098 >> v2:172.21.15.186:3300/0 conn(0x7eff0c0b8480 0x7eff0c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2885353098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c15cd70 0x7eff0c0d2220 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 -- 172.21.15.29:0/2885353098 >> 172.21.15.29:0/2885353098 conn(0x7eff0c0cf570 msgr2=0x7eff0c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.942+0000 7eff12a4d640 1 -- 172.21.15.29:0/2885353098 shutdown_connections 2023-10-14T18:19:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.943+0000 7eff12a4d640 1 -- 172.21.15.29:0/2885353098 wait complete. 2023-10-14T18:19:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.944+0000 7eff12a4d640 1 Processor -- start 2023-10-14T18:19:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.944+0000 7eff12a4d640 1 -- start start 2023-10-14T18:19:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c0b8480 0x7eff0c150dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 0x7eff0c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c14f9e0 con 0x7eff0c0d0be0 2023-10-14T18:19:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c14fcc0 con 0x7eff0c15cd70 2023-10-14T18:19:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.945+0000 7eff12a4d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0c14ffa0 con 0x7eff0c0b8480 2023-10-14T18:19:19.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36542/0 (socket says 172.21.15.29:36542) 2023-10-14T18:19:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 -- 172.21.15.29:0/2038109597 learned_addr learned my addr 172.21.15.29:0/2038109597 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff11a4b640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c0b8480 0x7eff0c150dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1224c640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 0x7eff0c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 -- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c0b8480 msgr2=0x7eff0c150dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c0b8480 0x7eff0c150dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 -- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 msgr2=0x7eff0c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:19.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 0x7eff0c0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1124a640 1 -- 172.21.15.29:0/2038109597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff080094d0 con 0x7eff0c0d0be0 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.946+0000 7eff1224c640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 0x7eff0c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff1124a640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7efefc00e870 tx=0x7efefc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc0094f0 con 0x7eff0c0d0be0 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efefc00c940 con 0x7eff0c0d0be0 2023-10-14T18:19:19.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efefc00ccf0 con 0x7eff0c0d0be0 2023-10-14T18:19:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff0c150280 con 0x7eff0c0d0be0 2023-10-14T18:19:19.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.947+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff0c151310 con 0x7eff0c0d0be0 2023-10-14T18:19:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.948+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff0c04e9b0 con 0x7eff0c0d0be0 2023-10-14T18:19:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.949+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7efefc013470 con 0x7eff0c0d0be0 2023-10-14T18:19:19.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.951+0000 7eff027fc640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 0x7efef006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.951+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7efefc089f30 con 0x7eff0c0d0be0 2023-10-14T18:19:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.952+0000 7eff11a4b640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 0x7efef006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:19.956 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.952+0000 7eff11a4b640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 0x7efef006d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:19.957 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:19.956+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7efefc054840 con 0x7eff0c0d0be0 2023-10-14T18:19:20.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.152+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"} v 0) v1 -- 0x7eff0c0b6620 con 0x7eff0c0d0be0 2023-10-14T18:19:20.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.153+0000 7eff027fc640 1 -- 172.21.15.29:0/2038109597 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "backup_fs_data", "var": "pg_num"}]=0 v259) v1 ==== 103+0+11 (secure 0 0 0) 0x7efefc0170a0 con 0x7eff0c0d0be0 2023-10-14T18:19:20.155 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 msgr2=0x7efef006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 0x7efef006d570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7eff08036000 tx=0x7eff08008030 comp rx=0 tx=0).stop 2023-10-14T18:19:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 msgr2=0x7eff0c0b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 secure :-1 s=READY pgs=3054 cs=0 l=1 rev1=1 crypto rx=0x7efefc00e870 tx=0x7efefc00ed30 comp rx=0 tx=0).stop 2023-10-14T18:19:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 shutdown_connections 2023-10-14T18:19:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7efef006b0b0 0x7efef006d570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7eff0c15cd70 0x7eff0c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7eff0c0d0be0 0x7eff0c0b3df0 unknown :-1 s=CLOSED pgs=3054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 --2- 172.21.15.29:0/2038109597 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7eff0c0b8480 0x7eff0c150dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 >> 172.21.15.29:0/2038109597 conn(0x7eff0c0cf570 msgr2=0x7eff0c151f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 shutdown_connections 2023-10-14T18:19:20.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.156+0000 7eff12a4d640 1 -- 172.21.15.29:0/2038109597 wait complete. 2023-10-14T18:19:20.179 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.402+0000 7f3bb93e9640 1 Processor -- start 2023-10-14T18:19:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.402+0000 7f3bb93e9640 1 -- start start 2023-10-14T18:19:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb93e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3bb40ad970 0x7f3bb40cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb93e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3bb40cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb93e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb415b9d0 con 0x7f3bb406c040 2023-10-14T18:19:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb93e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb415bcb0 con 0x7f3bb40ad970 2023-10-14T18:19:20.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb93e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb415bf90 con 0x7f3bb40ccfb0 2023-10-14T18:19:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3bb406c040 0x7f3bb406c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:58470/0 (socket says 172.21.15.29:58470) 2023-10-14T18:19:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.404+0000 7f3bb3fff640 1 -- 172.21.15.29:0/228106853 learned_addr learned my addr 172.21.15.29:0/228106853 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/228106853 >> v2:172.21.15.186:3300/0 conn(0x7f3bb40ad970 0x7f3bb40cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb8be8640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3bb40cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368558581 0 0) 0x7f3bb415b9d0 con 0x7f3bb406c040 2023-10-14T18:19:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b9c003520 con 0x7f3bb406c040 2023-10-14T18:19:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1666567254 0 0) 0x7f3b9c003520 con 0x7f3bb406c040 2023-10-14T18:19:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 msgr2=0x7f3bb40cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3bb40cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 >> v2:172.21.15.186:3300/0 conn(0x7f3bb40ad970 msgr2=0x7f3bb40cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/228106853 >> v2:172.21.15.186:3300/0 conn(0x7f3bb40ad970 0x7f3bb40cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb40cc270 con 0x7f3bb406c040 2023-10-14T18:19:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3ba4001670 con 0x7f3bb406c040 2023-10-14T18:19:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 >> v1:172.21.15.29:6789/0 conn(0x7f3bb406c040 legacy=0x7f3bb406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3b9c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3b9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c004d00 0x7f3b9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb415bcb0 con 0x7f3b9c003ef0 2023-10-14T18:19:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb415bf90 con 0x7f3b9c004d00 2023-10-14T18:19:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.405+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b9c00b980 con 0x7f3bb40ccfb0 2023-10-14T18:19:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3b9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb3fff640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3b9c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 -- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 msgr2=0x7f3b9c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40ccfb0 0x7f3b9c003970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 -- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c004d00 msgr2=0x7f3b9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3b9c004d00 0x7f3b9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.406+0000 7f3bb37fe640 1 -- 172.21.15.29:0/228106853 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b9c003520 con 0x7f3b9c003ef0 2023-10-14T18:19:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.407+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3b9c0047c0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009f90 tx=0x7f3b9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7530c5b0d544789e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba802cc00 con 0x7f3b9c003ef0 2023-10-14T18:19:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba800f040 con 0x7f3b9c003ef0 2023-10-14T18:19:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb2ffd640 1 -- 172.21.15.29:0/228106853 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba8034440 con 0x7f3b9c003ef0 2023-10-14T18:19:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 -- 172.21.15.29:0/228106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 msgr2=0x7f3b9c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/228106853 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3b9c0047c0 secure :-1 s=READY pgs=3056 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8009f90 tx=0x7f3b9c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb8be8640 1 -- 172.21.15.29:0/228106853 reap_dead start 2023-10-14T18:19:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 -- 172.21.15.29:0/228106853 shutdown_connections 2023-10-14T18:19:20.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 -- 172.21.15.29:0/228106853 >> 172.21.15.29:0/228106853 conn(0x7f3bb40ce6a0 msgr2=0x7f3bb40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:20.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 -- 172.21.15.29:0/228106853 shutdown_connections 2023-10-14T18:19:20.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.408+0000 7f3bb93e9640 1 -- 172.21.15.29:0/228106853 wait complete. 2023-10-14T18:19:20.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.410+0000 7f3bb93e9640 1 Processor -- start 2023-10-14T18:19:20.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.411+0000 7f3bb93e9640 1 -- start start 2023-10-14T18:19:20.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 0x7f3bb40c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ac890 0x7f3bb40acd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb40ad260 con 0x7f3b9c003ef0 2023-10-14T18:19:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb41524d0 con 0x7f3bb40ac890 2023-10-14T18:19:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb93e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb4158ab0 con 0x7f3bb40c4a50 2023-10-14T18:19:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb3fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36566/0 (socket says 172.21.15.29:36566) 2023-10-14T18:19:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb3fff640 1 -- 172.21.15.29:0/1288290186 learned_addr learned my addr 172.21.15.29:0/1288290186 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 0x7f3bb40c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb3fff640 1 -- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 msgr2=0x7f3bb40c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.412+0000 7f3bb8be8640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ac890 0x7f3bb40acd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb3fff640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 0x7f3bb40c5770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb3fff640 1 -- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ac890 msgr2=0x7f3bb40acd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb3fff640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ac890 0x7f3bb40acd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb3fff640 1 -- 172.21.15.29:0/1288290186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb4158d50 con 0x7f3b9c003ef0 2023-10-14T18:19:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 0x7f3bb40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:20.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb3fff640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d910 tx=0x7f3ba400ddd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba40074e0 con 0x7f3b9c003ef0 2023-10-14T18:19:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba4007640 con 0x7f3b9c003ef0 2023-10-14T18:19:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ba4004a00 con 0x7f3b9c003ef0 2023-10-14T18:19:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb4159030 con 0x7f3b9c003ef0 2023-10-14T18:19:20.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.413+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb41595f0 con 0x7f3b9c003ef0 2023-10-14T18:19:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.414+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb406b330 con 0x7f3b9c003ef0 2023-10-14T18:19:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.416+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3ba4004b60 con 0x7f3b9c003ef0 2023-10-14T18:19:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.417+0000 7f3bb0ff9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 0x7f3ba006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.417+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f3ba4089980 con 0x7f3b9c003ef0 2023-10-14T18:19:20.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.417+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 0x7f3ba006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.418+0000 7f3bb37fe640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 0x7f3ba006d5a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8036000 tx=0x7f3ba802e000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.422+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ba4057c20 con 0x7f3b9c003ef0 2023-10-14T18:19:20.670 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.668+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb404e9b0 con 0x7f3b9c003ef0 2023-10-14T18:19:20.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.669+0000 7f3bb0ff9640 1 -- 172.21.15.29:0/1288290186 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4932 (secure 0 0 0) 0x7f3ba40577b0 con 0x7f3b9c003ef0 2023-10-14T18:19:20.671 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:20.672 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":131,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":9094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1485060674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1485060674},{"type":"v1","addr":"172.21.15.186:6833","nonce":1485060674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:20.672 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 131 2023-10-14T18:19:20.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 msgr2=0x7f3ba006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 0x7f3ba006d5a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f3ba8036000 tx=0x7f3ba802e000 comp rx=0 tx=0).stop 2023-10-14T18:19:20.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 msgr2=0x7f3bb4151f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 secure :-1 s=READY pgs=3057 cs=0 l=1 rev1=1 crypto rx=0x7f3ba400d910 tx=0x7f3ba400ddd0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 shutdown_connections 2023-10-14T18:19:20.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3ba006b0e0 0x7f3ba006d5a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3bb40ac890 0x7f3bb40acd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3bb40c4a50 0x7f3bb40c5770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 --2- 172.21.15.29:0/1288290186 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3b9c003ef0 0x7f3bb4151f90 unknown :-1 s=CLOSED pgs=3057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 >> 172.21.15.29:0/1288290186 conn(0x7f3bb40ce6a0 msgr2=0x7f3bb415da30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:20.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.672+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 shutdown_connections 2023-10-14T18:19:20.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.673+0000 7f3bb93e9640 1 -- 172.21.15.29:0/1288290186 wait complete. 2023-10-14T18:19:20.694 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.920+0000 7f19ee815640 1 Processor -- start 2023-10-14T18:19:20.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.920+0000 7f19ee815640 1 -- start start 2023-10-14T18:19:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19ee815640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19e80b3070 0x7f19e80cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19ee815640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f19e80b3e70 0x7f19e80cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19ee815640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19e80b3940 con 0x7f19e80b3450 2023-10-14T18:19:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19ee815640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e80b45e0 con 0x7f19e80b3070 2023-10-14T18:19:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19ee815640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19e80b3b90 con 0x7f19e80b3e70 2023-10-14T18:19:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19e80b3070 0x7f19e80cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19e7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f19e80b3070 0x7f19e80cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:36578/0 (socket says 172.21.15.29:36578) 2023-10-14T18:19:20.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.922+0000 7f19e7fff640 1 -- 172.21.15.29:0/4233522347 learned_addr learned my addr 172.21.15.29:0/4233522347 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.923+0000 7f19e77fe640 1 --2- 172.21.15.29:0/4233522347 >> v2:172.21.15.186:3300/0 conn(0x7f19e80b3e70 0x7f19e80cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.923+0000 7f19e7fff640 1 -- 172.21.15.29:0/4233522347 >> v2:172.21.15.186:3300/0 conn(0x7f19e80b3e70 msgr2=0x7f19e80cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.923+0000 7f19e7fff640 1 --2- 172.21.15.29:0/4233522347 >> v2:172.21.15.186:3300/0 conn(0x7f19e80b3e70 0x7f19e80cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.923+0000 7f19e7fff640 1 -- 172.21.15.29:0/4233522347 >> v1:172.21.15.186:6789/0 conn(0x7f19e80b3450 legacy=0x7f19e80c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.923+0000 7f19e7fff640 1 -- 172.21.15.29:0/4233522347 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e80d3840 con 0x7f19e80b3070 2023-10-14T18:19:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.924+0000 7f19e7fff640 1 --2- 172.21.15.29:0/4233522347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3070 0x7f19e80cf500 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f19e80b58e0 tx=0x7f19d802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b7e971999eb0de8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19e6ffd640 1 -- 172.21.15.29:0/4233522347 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8004230 con 0x7f19e80b3070 2023-10-14T18:19:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19e6ffd640 1 -- 172.21.15.29:0/4233522347 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19d8004390 con 0x7f19e80b3070 2023-10-14T18:19:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19e6ffd640 1 -- 172.21.15.29:0/4233522347 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19d8030840 con 0x7f19e80b3070 2023-10-14T18:19:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 -- 172.21.15.29:0/4233522347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3070 msgr2=0x7f19e80cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 --2- 172.21.15.29:0/4233522347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3070 0x7f19e80cf500 secure :-1 s=READY pgs=3058 cs=0 l=1 rev1=1 crypto rx=0x7f19e80b58e0 tx=0x7f19d802b980 comp rx=0 tx=0).stop 2023-10-14T18:19:20.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 -- 172.21.15.29:0/4233522347 shutdown_connections 2023-10-14T18:19:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 --2- 172.21.15.29:0/4233522347 >> v2:172.21.15.186:3300/0 conn(0x7f19e80b3e70 0x7f19e80cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 --2- 172.21.15.29:0/4233522347 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3070 0x7f19e80cf500 unknown :-1 s=CLOSED pgs=3058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 -- 172.21.15.29:0/4233522347 >> 172.21.15.29:0/4233522347 conn(0x7f19e80d2d20 msgr2=0x7f19e80d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:20.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.925+0000 7f19ee815640 1 -- 172.21.15.29:0/4233522347 shutdown_connections 2023-10-14T18:19:20.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.926+0000 7f19ee815640 1 -- 172.21.15.29:0/4233522347 wait complete. 2023-10-14T18:19:20.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.927+0000 7f19ee815640 1 Processor -- start 2023-10-14T18:19:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.928+0000 7f19ee815640 1 -- start start 2023-10-14T18:19:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19e80b3070 0x7f19e80cd9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 0x7f19e80cdee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e815e750 con 0x7f19e80b3e70 2023-10-14T18:19:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19e80b87e0 con 0x7f19e80b3070 2023-10-14T18:19:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ee815640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19e814ca30 con 0x7f19e80b3450 2023-10-14T18:19:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ecdb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ecdb2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46988/0 (socket says 172.21.15.29:46988) 2023-10-14T18:19:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ecdb2640 1 -- 172.21.15.29:0/5125987 learned_addr learned my addr 172.21.15.29:0/5125987 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19e77fe640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 0x7f19e80cdee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19e7fff640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19e80b3070 0x7f19e80cd9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.929+0000 7f19ecdb2640 1 -- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 msgr2=0x7f19e80cdee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ecdb2640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 0x7f19e80cdee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ecdb2640 1 -- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19e80b3070 msgr2=0x7f19e80cd9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ecdb2640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19e80b3070 0x7f19e80cd9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ecdb2640 1 -- 172.21.15.29:0/5125987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d8009460 con 0x7f19e80b3e70 2023-10-14T18:19:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19e77fe640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 0x7f19e80cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:20.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ecdb2640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c7d0 tx=0x7f19e000cc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e0003b70 con 0x7f19e80b3e70 2023-10-14T18:19:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e0003cf0 con 0x7f19e80b3e70 2023-10-14T18:19:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19e000aa30 con 0x7f19e80b3e70 2023-10-14T18:19:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19e814f270 con 0x7f19e80b3e70 2023-10-14T18:19:20.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.930+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19e814c3e0 con 0x7f19e80b3e70 2023-10-14T18:19:20.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.931+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19e814cbc0 con 0x7f19e80b3e70 2023-10-14T18:19:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.933+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f19e000ab90 con 0x7f19e80b3e70 2023-10-14T18:19:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.934+0000 7f19e4ff9640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 0x7f19d406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.934+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f19e0089c70 con 0x7f19e80b3e70 2023-10-14T18:19:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.934+0000 7f19e7fff640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 0x7f19d406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:20.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.935+0000 7f19e7fff640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 0x7f19d406d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19d8035000 tx=0x7f19d8008030 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:20.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:20.939+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f19e0054580 con 0x7f19e80b3e70 2023-10-14T18:19:21.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.139+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f19e804e160 con 0x7f19e80b3e70 2023-10-14T18:19:21.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.142+0000 7f19e4ff9640 1 -- 172.21.15.29:0/5125987 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+18088 (secure 0 0 0) 0x7f19e0003e60 con 0x7f19e80b3e70 2023-10-14T18:19:21.145 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:21.145 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":259,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:18.272191+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6833/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.186:6832/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:21.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 msgr2=0x7f19d406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 0x7f19d406d570 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f19d8035000 tx=0x7f19d8008030 comp rx=0 tx=0).stop 2023-10-14T18:19:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 msgr2=0x7f19e80b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 secure :-1 s=READY pgs=3059 cs=0 l=1 rev1=1 crypto rx=0x7f19e000c7d0 tx=0x7f19e000cc90 comp rx=0 tx=0).stop 2023-10-14T18:19:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 shutdown_connections 2023-10-14T18:19:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f19d406b0b0 0x7f19d406d570 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f19e80b3e70 0x7f19e80b0940 unknown :-1 s=CLOSED pgs=3059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f19e80b3450 0x7f19e80cdee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 --2- 172.21.15.29:0/5125987 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f19e80b3070 0x7f19e80cd9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.145+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 >> 172.21.15.29:0/5125987 conn(0x7f19e80d2d20 msgr2=0x7f19e80c6850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.146+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 shutdown_connections 2023-10-14T18:19:21.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.146+0000 7f19ee815640 1 -- 172.21.15.29:0/5125987 wait complete. 2023-10-14T18:19:21.168 INFO:tasks.cephfs.filesystem:Destroying file system backup_fs and related pools 2023-10-14T18:19:21.169 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:21.383 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.381+0000 7f56cfc44640 1 Processor -- start 2023-10-14T18:19:21.384 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.381+0000 7f56cfc44640 1 -- start start 2023-10-14T18:19:21.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cfc44640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f56c815aaa0 0x7f56c815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cfc44640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c815ed60 con 0x7f56c815ae80 2023-10-14T18:19:21.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cfc44640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56c80707e0 con 0x7f56c80ce390 2023-10-14T18:19:21.385 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cfc44640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56c806b810 con 0x7f56c815aaa0 2023-10-14T18:19:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cd1df640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f56c815ae80 0x7f56c80cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48722/0 (socket says 172.21.15.29:48722) 2023-10-14T18:19:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cd1df640 1 -- 172.21.15.29:0/1552575167 learned_addr learned my addr 172.21.15.29:0/1552575167 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:21.386 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.383+0000 7f56cd9e0640 1 --2- 172.21.15.29:0/1552575167 >> v2:172.21.15.29:3300/0 conn(0x7f56c815aaa0 0x7f56c815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.384+0000 7f56cc9de640 1 -- 172.21.15.29:0/1552575167 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41226283 0 0) 0x7f56c815ed60 con 0x7f56c815ae80 2023-10-14T18:19:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.384+0000 7f56cc9de640 1 -- 172.21.15.29:0/1552575167 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b4003520 con 0x7f56c815ae80 2023-10-14T18:19:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.384+0000 7f56cd9e0640 1 -- 172.21.15.29:0/1552575167 >> v1:172.21.15.186:6789/0 conn(0x7f56c80ce390 legacy=0x7f56c8070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.384+0000 7f56cd9e0640 1 -- 172.21.15.29:0/1552575167 >> v1:172.21.15.29:6789/0 conn(0x7f56c815ae80 legacy=0x7f56c80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.388 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.384+0000 7f56cd9e0640 1 -- 172.21.15.29:0/1552575167 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c815ef90 con 0x7f56c815aaa0 2023-10-14T18:19:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.385+0000 7f56cd9e0640 1 --2- 172.21.15.29:0/1552575167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c815aaa0 0x7f56c815e820 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8021bcfbcd77a22c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.385+0000 7f56cc9de640 1 -- 172.21.15.29:0/1552575167 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4030610 con 0x7f56c815aaa0 2023-10-14T18:19:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cc9de640 1 -- 172.21.15.29:0/1552575167 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c4030770 con 0x7f56c815aaa0 2023-10-14T18:19:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cc9de640 1 -- 172.21.15.29:0/1552575167 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4030ae0 con 0x7f56c815aaa0 2023-10-14T18:19:21.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 -- 172.21.15.29:0/1552575167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c815aaa0 msgr2=0x7f56c815e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 --2- 172.21.15.29:0/1552575167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c815aaa0 0x7f56c815e820 secure :-1 s=READY pgs=3061 cs=0 l=1 rev1=1 crypto rx=0x7f56c40097a0 tx=0x7f56c402b960 comp rx=0 tx=0).stop 2023-10-14T18:19:21.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 -- 172.21.15.29:0/1552575167 shutdown_connections 2023-10-14T18:19:21.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 --2- 172.21.15.29:0/1552575167 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c815aaa0 0x7f56c815e820 unknown :-1 s=CLOSED pgs=3061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 -- 172.21.15.29:0/1552575167 >> 172.21.15.29:0/1552575167 conn(0x7f56c80cb410 msgr2=0x7f56c80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 -- 172.21.15.29:0/1552575167 shutdown_connections 2023-10-14T18:19:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.386+0000 7f56cfc44640 1 -- 172.21.15.29:0/1552575167 wait complete. 2023-10-14T18:19:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.388+0000 7f56cfc44640 1 Processor -- start 2023-10-14T18:19:21.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.388+0000 7f56cfc44640 1 -- start start 2023-10-14T18:19:21.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56c815aaa0 0x7f56c815def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56c815ae80 0x7f56c8157d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c815b4c0 con 0x7f56c80ce390 2023-10-14T18:19:21.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c815e430 con 0x7f56c815ae80 2023-10-14T18:19:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.389+0000 7f56cfc44640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56c80c83a0 con 0x7f56c815aaa0 2023-10-14T18:19:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47004/0 (socket says 172.21.15.29:47004) 2023-10-14T18:19:21.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 -- 172.21.15.29:0/4254989693 learned_addr learned my addr 172.21.15.29:0/4254989693 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd1df640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56c815aaa0 0x7f56c815def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56ce1e1640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56c815ae80 0x7f56c8157d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 -- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56c815aaa0 msgr2=0x7f56c815def0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56c815aaa0 0x7f56c815def0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 -- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56c815ae80 msgr2=0x7f56c8157d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56c815ae80 0x7f56c8157d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.390+0000 7f56cd9e0640 1 -- 172.21.15.29:0/4254989693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c4009400 con 0x7f56c80ce390 2023-10-14T18:19:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56cd9e0640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f56c4000c00 tx=0x7f56c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4039c90 con 0x7f56c80ce390 2023-10-14T18:19:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56c403f040 con 0x7f56c80ce390 2023-10-14T18:19:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56c4006e10 con 0x7f56c80ce390 2023-10-14T18:19:21.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c80c8680 con 0x7f56c80ce390 2023-10-14T18:19:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.391+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c80c8c40 con 0x7f56c80ce390 2023-10-14T18:19:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.392+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c804e160 con 0x7f56c80ce390 2023-10-14T18:19:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.393+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f56c4004030 con 0x7f56c80ce390 2023-10-14T18:19:21.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.394+0000 7f56be7fc640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 0x7f56ac06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.394+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f56c40aba10 con 0x7f56c80ce390 2023-10-14T18:19:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.395+0000 7f56cd1df640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 0x7f56ac06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.395+0000 7f56cd1df640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 0x7f56ac06d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56c80ad5a0 tx=0x7f56b8009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.399+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f56c4076480 con 0x7f56c80ce390 2023-10-14T18:19:21.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.633+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f56c80cdc60 con 0x7f56c80ce390 2023-10-14T18:19:21.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.634+0000 7f56be7fc640 1 -- 172.21.15.29:0/4254989693 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4932 (secure 0 0 0) 0x7f56c4079aa0 con 0x7f56c80ce390 2023-10-14T18:19:21.636 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:21.637 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":131,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":9094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1485060674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1485060674},{"type":"v1","addr":"172.21.15.186:6833","nonce":1485060674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:21.637 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 131 2023-10-14T18:19:21.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 msgr2=0x7f56ac06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 0x7f56ac06d570 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56c80ad5a0 tx=0x7f56b8009040 comp rx=0 tx=0).stop 2023-10-14T18:19:21.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 msgr2=0x7f56c80c7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 secure :-1 s=READY pgs=3062 cs=0 l=1 rev1=1 crypto rx=0x7f56c4000c00 tx=0x7f56c40046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 shutdown_connections 2023-10-14T18:19:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f56ac06b0b0 0x7f56ac06d570 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f56c815ae80 0x7f56c8157d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f56c815aaa0 0x7f56c815def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 --2- 172.21.15.29:0/4254989693 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f56c80ce390 0x7f56c80c7cc0 unknown :-1 s=CLOSED pgs=3062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 >> 172.21.15.29:0/4254989693 conn(0x7f56c80cb410 msgr2=0x7f56c815a090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:21.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 shutdown_connections 2023-10-14T18:19:21.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.637+0000 7f56cfc44640 1 -- 172.21.15.29:0/4254989693 wait complete. 2023-10-14T18:19:21.657 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:21.877 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.875+0000 7f3642040640 1 Processor -- start 2023-10-14T18:19:21.878 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.876+0000 7f3642040640 1 -- start start 2023-10-14T18:19:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.877+0000 7f3642040640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c15b320 0x7f363c0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.877+0000 7f3642040640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c15b9d0 con 0x7f363c06c040 2023-10-14T18:19:21.879 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.877+0000 7f3642040640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363c15bcb0 con 0x7f363c0ad970 2023-10-14T18:19:21.880 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.877+0000 7f3642040640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c15b7d0 con 0x7f363c15b320 2023-10-14T18:19:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.878+0000 7f364103e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f363c06c040 0x7f363c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48730/0 (socket says 172.21.15.29:48730) 2023-10-14T18:19:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.878+0000 7f364103e640 1 -- 172.21.15.29:0/353417151 learned_addr learned my addr 172.21.15.29:0/353417151 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.878+0000 7f364183f640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c15b320 0x7f363c0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.882 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.878+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199612187 0 0) 0x7f363c15b9d0 con 0x7f363c06c040 2023-10-14T18:19:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.878+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3628003520 con 0x7f363c06c040 2023-10-14T18:19:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429181229 0 0) 0x7f363c15bcb0 con 0x7f363c0ad970 2023-10-14T18:19:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f363c15b9d0 con 0x7f363c0ad970 2023-10-14T18:19:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1805492220 0 0) 0x7f3628003520 con 0x7f363c06c040 2023-10-14T18:19:21.883 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c15b320 msgr2=0x7f363c0cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c15b320 0x7f363c0cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 >> v1:172.21.15.186:6789/0 conn(0x7f363c0ad970 legacy=0x7f363c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c15bf90 con 0x7f363c06c040 2023-10-14T18:19:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f36380019f0 con 0x7f363c06c040 2023-10-14T18:19:21.884 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 >> v1:172.21.15.29:6789/0 conn(0x7f363c06c040 legacy=0x7f363c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3628003b30 0x7f3628003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36280047c0 0x7f3628004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3628005180 0x7f362800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c15b7d0 con 0x7f36280047c0 2023-10-14T18:19:21.885 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f362800bf70 con 0x7f3628005180 2023-10-14T18:19:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362800c130 con 0x7f3628003b30 2023-10-14T18:19:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.879+0000 7f364083d640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36280047c0 0x7f3628004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364103e640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3628003b30 0x7f3628003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364183f640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3628005180 0x7f362800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.886 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364083d640 1 -- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3628003b30 msgr2=0x7f3628003f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364083d640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3628003b30 0x7f3628003f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364083d640 1 -- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3628005180 msgr2=0x7f362800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364083d640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3628005180 0x7f362800b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.887 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.880+0000 7f364083d640 1 -- 172.21.15.29:0/353417151 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3628003520 con 0x7f36280047c0 2023-10-14T18:19:21.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.881+0000 7f364083d640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36280047c0 0x7f3628004c40 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f362c0018a0 tx=0x7f36280057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f357cc0844d44de server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.890 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.881+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c0052d0 con 0x7f36280047c0 2023-10-14T18:19:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.881+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c005430 con 0x7f36280047c0 2023-10-14T18:19:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.881+0000 7f3633fff640 1 -- 172.21.15.29:0/353417151 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f362c003810 con 0x7f36280047c0 2023-10-14T18:19:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 -- 172.21.15.29:0/353417151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36280047c0 msgr2=0x7f3628004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 --2- 172.21.15.29:0/353417151 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f36280047c0 0x7f3628004c40 secure :-1 s=READY pgs=3064 cs=0 l=1 rev1=1 crypto rx=0x7f362c0018a0 tx=0x7f36280057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f364183f640 1 -- 172.21.15.29:0/353417151 reap_dead start 2023-10-14T18:19:21.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 -- 172.21.15.29:0/353417151 shutdown_connections 2023-10-14T18:19:21.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 -- 172.21.15.29:0/353417151 >> 172.21.15.29:0/353417151 conn(0x7f363c0ce6a0 msgr2=0x7f363c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:21.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 -- 172.21.15.29:0/353417151 shutdown_connections 2023-10-14T18:19:21.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.882+0000 7f3642040640 1 -- 172.21.15.29:0/353417151 wait complete. 2023-10-14T18:19:21.892 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.884+0000 7f3642040640 1 Processor -- start 2023-10-14T18:19:21.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.884+0000 7f3642040640 1 -- start start 2023-10-14T18:19:21.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36280047c0 0x7f363c0b0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c146b10 0x7f363c0c40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.893 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0c4ab0 con 0x7f363c0c4630 2023-10-14T18:19:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0afb90 con 0x7f36280047c0 2023-10-14T18:19:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f3642040640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f363c0aef80 con 0x7f363c146b10 2023-10-14T18:19:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f364183f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f364183f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47028/0 (socket says 172.21.15.29:47028) 2023-10-14T18:19:21.894 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.885+0000 7f364183f640 1 -- 172.21.15.29:0/1575411484 learned_addr learned my addr 172.21.15.29:0/1575411484 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364103e640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36280047c0 0x7f363c0b0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364083d640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c146b10 0x7f363c0c40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 -- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c146b10 msgr2=0x7f363c0c40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c146b10 0x7f363c0c40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 -- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36280047c0 msgr2=0x7f363c0b0020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:21.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36280047c0 0x7f363c0b0020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:21.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 -- 172.21.15.29:0/1575411484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363c0af260 con 0x7f363c0c4630 2023-10-14T18:19:21.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f364183f640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f363c0add50 tx=0x7f3634010c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634004230 con 0x7f363c0c4630 2023-10-14T18:19:21.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3634004390 con 0x7f363c0c4630 2023-10-14T18:19:21.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.886+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3634003850 con 0x7f363c0c4630 2023-10-14T18:19:21.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.887+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f363c0af540 con 0x7f363c0c4630 2023-10-14T18:19:21.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.887+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f363c146f70 con 0x7f363c0c4630 2023-10-14T18:19:21.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.888+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f363c06b330 con 0x7f363c0c4630 2023-10-14T18:19:21.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.889+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f36340039b0 con 0x7f363c0c4630 2023-10-14T18:19:21.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.890+0000 7f3631ffb640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 0x7f362006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:21.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.890+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f3634089700 con 0x7f363c0c4630 2023-10-14T18:19:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.890+0000 7f364103e640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 0x7f362006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.891+0000 7f364103e640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 0x7f362006d570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f36380013d0 tx=0x7f3638009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:21.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:21.894+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3634054050 con 0x7f363c0c4630 2023-10-14T18:19:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.119+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f363c04e9b0 con 0x7f363c0c4630 2023-10-14T18:19:22.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.120+0000 7f3631ffb640 1 -- 172.21.15.29:0/1575411484 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+4932 (secure 0 0 0) 0x7f3634057670 con 0x7f363c0c4630 2023-10-14T18:19:22.122 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:22.122 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":131,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":9094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1485060674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1485060674},{"type":"v1","addr":"172.21.15.186:6833","nonce":1485060674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131}],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:18:49.773591+0000","modified":"2023-10-14T18:18:52.835381+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":7637},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7637":{"gid":7637,"name":"d","rank":0,"incarnation":125,"state":"up:active","state_seq":7,"addr":"172.21.15.186:6835/2447498482","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":2447498482},{"type":"v1","addr":"172.21.15.186:6835","nonce":2447498482}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[33],"metadata_pool":32,"enabled":true,"fs_name":"backup_fs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":16}]} 2023-10-14T18:19:22.123 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 131 2023-10-14T18:19:22.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.122+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 msgr2=0x7f362006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.122+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 0x7f362006d570 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f36380013d0 tx=0x7f3638009040 comp rx=0 tx=0).stop 2023-10-14T18:19:22.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.122+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 msgr2=0x7f363c0ae220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.122+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 secure :-1 s=READY pgs=3065 cs=0 l=1 rev1=1 crypto rx=0x7f363c0add50 tx=0x7f3634010c20 comp rx=0 tx=0).stop 2023-10-14T18:19:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 shutdown_connections 2023-10-14T18:19:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f362006b0b0 0x7f362006d570 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f363c0c4630 0x7f363c0ae220 unknown :-1 s=CLOSED pgs=3065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f363c146b10 0x7f363c0c40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 --2- 172.21.15.29:0/1575411484 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f36280047c0 0x7f363c0b0020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 >> 172.21.15.29:0/1575411484 conn(0x7f363c0ce6a0 msgr2=0x7f363c0c5770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 shutdown_connections 2023-10-14T18:19:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.123+0000 7f3642040640 1 -- 172.21.15.29:0/1575411484 wait complete. 2023-10-14T18:19:22.142 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.358+0000 7f86965a9640 1 Processor -- start 2023-10-14T18:19:22.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.358+0000 7f86965a9640 1 -- start start 2023-10-14T18:19:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.359+0000 7f86965a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900adb50 0x7f869006f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f86965a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86900b0b20 0x7f8690164f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f86965a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8690071450 con 0x7f8690070fd0 2023-10-14T18:19:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f86965a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86900cc600 con 0x7f86900b0b20 2023-10-14T18:19:22.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f86965a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86900cc8e0 con 0x7f86900adb50 2023-10-14T18:19:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f8694b46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86900b0b20 0x7f8690164f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f8694b46640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f86900b0b20 0x7f8690164f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47044/0 (socket says 172.21.15.29:47044) 2023-10-14T18:19:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f8694b46640 1 -- 172.21.15.29:0/2656379214 learned_addr learned my addr 172.21.15.29:0/2656379214 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.360+0000 7f868ffff640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900adb50 0x7f869006f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.361+0000 7f8694b46640 1 -- 172.21.15.29:0/2656379214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900adb50 msgr2=0x7f869006f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.361+0000 7f8694b46640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900adb50 0x7f869006f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.361+0000 7f8694b46640 1 -- 172.21.15.29:0/2656379214 >> v1:172.21.15.186:6789/0 conn(0x7f8690070fd0 legacy=0x7f86900b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.361+0000 7f8694b46640 1 -- 172.21.15.29:0/2656379214 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86900ccbc0 con 0x7f86900b0b20 2023-10-14T18:19:22.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.362+0000 7f8694b46640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86900b0b20 0x7f8690164f60 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f8688009e50 tx=0x7f868802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2307b94f140fa52 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.362+0000 7f868effd640 1 -- 172.21.15.29:0/2656379214 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868802cb00 con 0x7f86900b0b20 2023-10-14T18:19:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.362+0000 7f868effd640 1 -- 172.21.15.29:0/2656379214 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868802cc60 con 0x7f86900b0b20 2023-10-14T18:19:22.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.362+0000 7f868effd640 1 -- 172.21.15.29:0/2656379214 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f86880343f0 con 0x7f86900b0b20 2023-10-14T18:19:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 -- 172.21.15.29:0/2656379214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86900b0b20 msgr2=0x7f8690164f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86900b0b20 0x7f8690164f60 secure :-1 s=READY pgs=3066 cs=0 l=1 rev1=1 crypto rx=0x7f8688009e50 tx=0x7f868802bf60 comp rx=0 tx=0).stop 2023-10-14T18:19:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 -- 172.21.15.29:0/2656379214 shutdown_connections 2023-10-14T18:19:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f86900b0b20 0x7f8690164f60 unknown :-1 s=CLOSED pgs=3066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 --2- 172.21.15.29:0/2656379214 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900adb50 0x7f869006f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 -- 172.21.15.29:0/2656379214 >> 172.21.15.29:0/2656379214 conn(0x7f86900cb410 msgr2=0x7f86900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 -- 172.21.15.29:0/2656379214 shutdown_connections 2023-10-14T18:19:22.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.363+0000 7f86965a9640 1 -- 172.21.15.29:0/2656379214 wait complete. 2023-10-14T18:19:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.365+0000 7f86965a9640 1 Processor -- start 2023-10-14T18:19:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.365+0000 7f86965a9640 1 -- start start 2023-10-14T18:19:22.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86900adb50 0x7f8690150900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900b0b20 0x7f8690163cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f869014e9e0 con 0x7f8690070fd0 2023-10-14T18:19:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86901f6920 con 0x7f86900adb50 2023-10-14T18:19:22.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f86965a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f869014e640 con 0x7f86900b0b20 2023-10-14T18:19:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f868ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f868ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47050/0 (socket says 172.21.15.29:47050) 2023-10-14T18:19:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.366+0000 7f868ffff640 1 -- 172.21.15.29:0/1143705317 learned_addr learned my addr 172.21.15.29:0/1143705317 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 -- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900b0b20 msgr2=0x7f8690163cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900b0b20 0x7f8690163cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 -- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86900adb50 msgr2=0x7f8690150900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86900adb50 0x7f8690150900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 -- 172.21.15.29:0/1143705317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8688009b30 con 0x7f8690070fd0 2023-10-14T18:19:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868ffff640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f868000d800 tx=0x7f868000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868000b6c0 con 0x7f8690070fd0 2023-10-14T18:19:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f868000b840 con 0x7f8690070fd0 2023-10-14T18:19:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86900c3530 con 0x7f8690070fd0 2023-10-14T18:19:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.367+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f868000bbf0 con 0x7f8690070fd0 2023-10-14T18:19:22.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.368+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86900c3af0 con 0x7f8690070fd0 2023-10-14T18:19:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.369+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f869014eb20 con 0x7f8690070fd0 2023-10-14T18:19:22.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.370+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f868000bd50 con 0x7f8690070fd0 2023-10-14T18:19:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.371+0000 7f868cff9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 0x7f867c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.371+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7f868008a4a0 con 0x7f8690070fd0 2023-10-14T18:19:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.371+0000 7f868f7fe640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 0x7f867c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.372+0000 7f868f7fe640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 0x7f867c06d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8684000f60 tx=0x7f86840079b0 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.376+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8680054e60 con 0x7f8690070fd0 2023-10-14T18:19:22.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.572+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f86900c2120 con 0x7f8690070fd0 2023-10-14T18:19:22.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.576+0000 7f868cff9640 1 -- 172.21.15.29:0/1143705317 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v259) v1 ==== 74+0+18088 (secure 0 0 0) 0x7f8680058480 con 0x7f8690070fd0 2023-10-14T18:19:22.578 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:22.578 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":259,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:18.272191+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":32,"pool_name":"backup_fs_metadata","create_time":"2023-10-14T18:18:47.629053+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"253","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"backup_fs"}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":33,"pool_name":"backup_fs_data","create_time":"2023-10-14T18:18:48.796314+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"254","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"backup_fs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6833/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.186:6832/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:22.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 msgr2=0x7f867c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 0x7f867c06d570 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8684000f60 tx=0x7f86840079b0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 msgr2=0x7f86900c2ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 secure :-1 s=READY pgs=3067 cs=0 l=1 rev1=1 crypto rx=0x7f868000d800 tx=0x7f868000dcc0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 shutdown_connections 2023-10-14T18:19:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f867c06b0b0 0x7f867c06d570 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f86900b0b20 0x7f8690163cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f86900adb50 0x7f8690150900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 --2- 172.21.15.29:0/1143705317 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8690070fd0 0x7f86900c2ca0 unknown :-1 s=CLOSED pgs=3067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 >> 172.21.15.29:0/1143705317 conn(0x7f86900cb410 msgr2=0x7f86900cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 shutdown_connections 2023-10-14T18:19:22.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.579+0000 7f86965a9640 1 -- 172.21.15.29:0/1143705317 wait complete. 2023-10-14T18:19:22.602 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail backup_fs 2023-10-14T18:19:22.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 Processor -- start 2023-10-14T18:19:22.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 -- start start 2023-10-14T18:19:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc4600ca0a0 0x7fc46006c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46015d910 con 0x7fc460070b60 2023-10-14T18:19:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc46015dbf0 con 0x7fc460159950 2023-10-14T18:19:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.830+0000 7fc45f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc46015e490 con 0x7fc4600ca0a0 2023-10-14T18:19:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.831+0000 7fc457fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc460070b60 0x7fc46006c100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48736/0 (socket says 172.21.15.29:48736) 2023-10-14T18:19:22.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.831+0000 7fc457fff640 1 -- 172.21.15.29:0/1409750314 learned_addr learned my addr 172.21.15.29:0/1409750314 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.831+0000 7fc45ed9d640 1 --2- 172.21.15.29:0/1409750314 >> v2:172.21.15.186:3300/0 conn(0x7fc4600ca0a0 0x7fc46006c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.831+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 331745754 0 0) 0x7fc46015d910 con 0x7fc460070b60 2023-10-14T18:19:22.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.831+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc44c003520 con 0x7fc460070b60 2023-10-14T18:19:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3869857462 0 0) 0x7fc44c003520 con 0x7fc460070b60 2023-10-14T18:19:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 >> v2:172.21.15.186:3300/0 conn(0x7fc4600ca0a0 msgr2=0x7fc46006c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 --2- 172.21.15.29:0/1409750314 >> v2:172.21.15.186:3300/0 conn(0x7fc4600ca0a0 0x7fc46006c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 >> v1:172.21.15.186:6789/0 conn(0x7fc460159950 legacy=0x7fc460070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc46015ded0 con 0x7fc460070b60 2023-10-14T18:19:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc448001670 con 0x7fc460070b60 2023-10-14T18:19:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 >> v1:172.21.15.29:6789/0 conn(0x7fc460070b60 legacy=0x7fc46006c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc44c003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc44c0047c0 0x7fc44c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc44c0051d0 0x7fc44c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc46015e490 con 0x7fc44c0039c0 2023-10-14T18:19:22.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c120 con 0x7fc44c0051d0 2023-10-14T18:19:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc44c00c2e0 con 0x7fc44c0047c0 2023-10-14T18:19:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.832+0000 7fc45e59c640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc44c003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45e59c640 1 -- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc44c0047c0 msgr2=0x7fc44c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45e59c640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc44c0047c0 0x7fc44c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45e59c640 1 -- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc44c0051d0 msgr2=0x7fc44c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45e59c640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc44c0051d0 0x7fc44c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45e59c640 1 -- 172.21.15.29:0/1409750314 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44c003520 con 0x7fc44c0039c0 2023-10-14T18:19:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.833+0000 7fc45ed9d640 1 -- 172.21.15.29:0/1409750314 reap_dead start 2023-10-14T18:19:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45e59c640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc44c003de0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fc450001720 tx=0x7fc44c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5008dd0daf6c71 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4500052d0 con 0x7fc44c0039c0 2023-10-14T18:19:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc45002e050 con 0x7fc44c0039c0 2023-10-14T18:19:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45dd9b640 1 -- 172.21.15.29:0/1409750314 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450003830 con 0x7fc44c0039c0 2023-10-14T18:19:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45f59e640 1 -- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 msgr2=0x7fc44c003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.834+0000 7fc45f59e640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc44c003de0 secure :-1 s=READY pgs=3069 cs=0 l=1 rev1=1 crypto rx=0x7fc450001720 tx=0x7fc44c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.835+0000 7fc45f59e640 1 -- 172.21.15.29:0/1409750314 shutdown_connections 2023-10-14T18:19:22.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.835+0000 7fc45f59e640 1 --2- 172.21.15.29:0/1409750314 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc44c003de0 unknown :-1 s=CLOSED pgs=3069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.846 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.835+0000 7fc45f59e640 1 -- 172.21.15.29:0/1409750314 >> 172.21.15.29:0/1409750314 conn(0x7fc4600ce6a0 msgr2=0x7fc4600ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:22.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.835+0000 7fc45f59e640 1 -- 172.21.15.29:0/1409750314 shutdown_connections 2023-10-14T18:19:22.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.835+0000 7fc45f59e640 1 -- 172.21.15.29:0/1409750314 wait complete. 2023-10-14T18:19:22.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.837+0000 7fc45f59e640 1 Processor -- start 2023-10-14T18:19:22.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.837+0000 7fc45f59e640 1 -- start start 2023-10-14T18:19:22.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc460156d30 0x7fc4600c2dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4600ae3f0 0x7fc4600ae8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600aede0 con 0x7fc44c0039c0 2023-10-14T18:19:22.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4601571b0 con 0x7fc4600ae3f0 2023-10-14T18:19:22.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4600c4e10 con 0x7fc460156d30 2023-10-14T18:19:22.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47060/0 (socket says 172.21.15.29:47060) 2023-10-14T18:19:22.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 -- 172.21.15.29:0/354729149 learned_addr learned my addr 172.21.15.29:0/354729149 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 -- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc460156d30 msgr2=0x7fc4600c2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc460156d30 0x7fc4600c2dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 -- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4600ae3f0 msgr2=0x7fc4600ae8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4600ae3f0 0x7fc4600ae8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:22.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.838+0000 7fc45e59c640 1 -- 172.21.15.29:0/354729149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4601ef310 con 0x7fc44c0039c0 2023-10-14T18:19:22.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc45e59c640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fc45002bbc0 tx=0x7fc450001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450007a30 con 0x7fc44c0039c0 2023-10-14T18:19:22.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc450007640 con 0x7fc44c0039c0 2023-10-14T18:19:22.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc450006c00 con 0x7fc44c0039c0 2023-10-14T18:19:22.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4601ef5f0 con 0x7fc44c0039c0 2023-10-14T18:19:22.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.839+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4601efbb0 con 0x7fc44c0039c0 2023-10-14T18:19:22.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.840+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc46004e9b0 con 0x7fc44c0039c0 2023-10-14T18:19:22.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.841+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fc4500035f0 con 0x7fc44c0039c0 2023-10-14T18:19:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.842+0000 7fc456ffd640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 0x7fc44006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.842+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(259..259 src has 1..259) v4 ==== 6783+0+0 (secure 0 0 0) 0x7fc4500abc50 con 0x7fc44c0039c0 2023-10-14T18:19:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.843+0000 7fc457fff640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 0x7fc44006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:22.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.843+0000 7fc457fff640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 0x7fc44006d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc448000ff0 tx=0x7fc448001580 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:22.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:22.847+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc4500765b0 con 0x7fc44c0039c0 2023-10-14T18:19:23.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.082+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "backup_fs"} v 0) v1 -- 0x7fc4600b4620 con 0x7fc44c0039c0 2023-10-14T18:19:23.085 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:19:23.083+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:19:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.087+0000 7fc456ffd640 1 -- 172.21.15.29:0/354729149 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "backup_fs"}]=0 backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v132) v1 ==== 167+0+0 (secure 0 0 0) 0x7fc45004b090 con 0x7fc44c0039c0 2023-10-14T18:19:23.089 INFO:teuthology.orchestra.run.smithi029.stderr:backup_fs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:19:23.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 msgr2=0x7fc44006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 0x7fc44006d5a0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc448000ff0 tx=0x7fc448001580 comp rx=0 tx=0).stop 2023-10-14T18:19:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 msgr2=0x7fc4600b12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 secure :-1 s=READY pgs=3070 cs=0 l=1 rev1=1 crypto rx=0x7fc45002bbc0 tx=0x7fc450001120 comp rx=0 tx=0).stop 2023-10-14T18:19:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 shutdown_connections 2023-10-14T18:19:23.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fc44006b0e0 0x7fc44006d5a0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc4600ae3f0 0x7fc4600ae8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc460156d30 0x7fc4600c2dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 --2- 172.21.15.29:0/354729149 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc44c0039c0 0x7fc4600b12c0 unknown :-1 s=CLOSED pgs=3070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.090+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 >> 172.21.15.29:0/354729149 conn(0x7fc4600ce6a0 msgr2=0x7fc4600cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.091+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 shutdown_connections 2023-10-14T18:19:23.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.091+0000 7fc45f59e640 1 -- 172.21.15.29:0/354729149 wait complete. 2023-10-14T18:19:23.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm backup_fs --yes-i-really-mean-it 2023-10-14T18:19:23.115 INFO:tasks.ceph.mds.d.smithi186.stderr: -1729> 2023-10-14T18:18:55.055+0000 7fdb3fdf5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:19:23.178 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:19:23.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.345+0000 7f8c1d494640 1 Processor -- start 2023-10-14T18:19:23.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.345+0000 7f8c1d494640 1 -- start start 2023-10-14T18:19:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.346+0000 7f8c1d494640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c180caac0 0x7f8c180ad4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.346+0000 7f8c1d494640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c180cb030 con 0x7f8c1806a4b0 2023-10-14T18:19:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.346+0000 7f8c1d494640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c180c91c0 con 0x7f8c1815ad70 2023-10-14T18:19:23.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.346+0000 7f8c1d494640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c180c94a0 con 0x7f8c180caac0 2023-10-14T18:19:23.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c0ffff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8c1806a4b0 0x7f8c180ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48748/0 (socket says 172.21.15.29:48748) 2023-10-14T18:19:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c0ffff640 1 -- 172.21.15.29:0/3970912354 learned_addr learned my addr 172.21.15.29:0/3970912354 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:23.350 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c177fe640 1 --2- 172.21.15.29:0/3970912354 >> v2:172.21.15.29:3300/0 conn(0x7f8c180caac0 0x7f8c180ad4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c167fc640 1 -- 172.21.15.29:0/3970912354 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867625063 0 0) 0x7f8c180cb030 con 0x7f8c1806a4b0 2023-10-14T18:19:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c167fc640 1 -- 172.21.15.29:0/3970912354 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c04003520 con 0x7f8c1806a4b0 2023-10-14T18:19:23.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c177fe640 1 -- 172.21.15.29:0/3970912354 >> v1:172.21.15.186:6789/0 conn(0x7f8c1815ad70 legacy=0x7f8c18069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c177fe640 1 -- 172.21.15.29:0/3970912354 >> v1:172.21.15.29:6789/0 conn(0x7f8c1806a4b0 legacy=0x7f8c180ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c177fe640 1 -- 172.21.15.29:0/3970912354 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c180c9780 con 0x7f8c180caac0 2023-10-14T18:19:23.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.348+0000 7f8c177fe640 1 --2- 172.21.15.29:0/3970912354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c180caac0 0x7f8c180ad4d0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009d90 tx=0x7f8c1002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ea64d8c95e41e9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.349+0000 7f8c167fc640 1 -- 172.21.15.29:0/3970912354 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c1002cd00 con 0x7f8c180caac0 2023-10-14T18:19:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.349+0000 7f8c167fc640 1 -- 172.21.15.29:0/3970912354 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c1002ce60 con 0x7f8c180caac0 2023-10-14T18:19:23.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.349+0000 7f8c167fc640 1 -- 172.21.15.29:0/3970912354 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c10034660 con 0x7f8c180caac0 2023-10-14T18:19:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.349+0000 7f8c1d494640 1 -- 172.21.15.29:0/3970912354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c180caac0 msgr2=0x7f8c180ad4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.349+0000 7f8c1d494640 1 --2- 172.21.15.29:0/3970912354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c180caac0 0x7f8c180ad4d0 secure :-1 s=READY pgs=3073 cs=0 l=1 rev1=1 crypto rx=0x7f8c10009d90 tx=0x7f8c1002bf50 comp rx=0 tx=0).stop 2023-10-14T18:19:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.350+0000 7f8c1d494640 1 -- 172.21.15.29:0/3970912354 shutdown_connections 2023-10-14T18:19:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.350+0000 7f8c1d494640 1 --2- 172.21.15.29:0/3970912354 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c180caac0 0x7f8c180ad4d0 unknown :-1 s=CLOSED pgs=3073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.350+0000 7f8c1d494640 1 -- 172.21.15.29:0/3970912354 >> 172.21.15.29:0/3970912354 conn(0x7f8c180ce9b0 msgr2=0x7f8c180cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:23.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.350+0000 7f8c1d494640 1 -- 172.21.15.29:0/3970912354 shutdown_connections 2023-10-14T18:19:23.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.350+0000 7f8c1d494640 1 -- 172.21.15.29:0/3970912354 wait complete. 2023-10-14T18:19:23.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.351+0000 7f8c1d494640 1 Processor -- start 2023-10-14T18:19:23.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.352+0000 7f8c1d494640 1 -- start start 2023-10-14T18:19:23.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.352+0000 7f8c1d494640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:23.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.352+0000 7f8c1d494640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c180caac0 0x7f8c180ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:23.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c1d494640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c1815ad70 0x7f8c180c4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:23.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c1d494640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c18159b50 con 0x7f8c1806a4b0 2023-10-14T18:19:23.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c1d494640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180c8120 con 0x7f8c180caac0 2023-10-14T18:19:23.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c1d494640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c180ad190 con 0x7f8c1815ad70 2023-10-14T18:19:23.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:23.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47066/0 (socket says 172.21.15.29:47066) 2023-10-14T18:19:23.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 -- 172.21.15.29:0/251391066 learned_addr learned my addr 172.21.15.29:0/251391066 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:23.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c0ffff640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c180caac0 0x7f8c180ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:23.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c177fe640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c1815ad70 0x7f8c180c4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:23.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 -- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c1815ad70 msgr2=0x7f8c180c4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c1815ad70 0x7f8c180c4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 -- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c180caac0 msgr2=0x7f8c180ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c180caac0 0x7f8c180ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:23.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.353+0000 7f8c16ffd640 1 -- 172.21.15.29:0/251391066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c10009a70 con 0x7f8c1806a4b0 2023-10-14T18:19:23.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c16ffd640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c400 tx=0x7f8c0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:23.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c080052d0 con 0x7f8c1806a4b0 2023-10-14T18:19:23.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c08005430 con 0x7f8c1806a4b0 2023-10-14T18:19:23.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c08003830 con 0x7f8c1806a4b0 2023-10-14T18:19:23.368 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c180ad470 con 0x7f8c1806a4b0 2023-10-14T18:19:23.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.354+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c180c6950 con 0x7f8c1806a4b0 2023-10-14T18:19:23.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.355+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c1804e160 con 0x7f8c1806a4b0 2023-10-14T18:19:23.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.356+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8c08003990 con 0x7f8c1806a4b0 2023-10-14T18:19:23.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.357+0000 7f8c0f7fe640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 0x7f8bf806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:23.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.357+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6869+0+0 (secure 0 0 0) 0x7f8c08089400 con 0x7f8c1806a4b0 2023-10-14T18:19:23.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.358+0000 7f8c0ffff640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 0x7f8bf806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:23.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.358+0000 7f8c0ffff640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 0x7f8bf806d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c1806f690 tx=0x7f8c00009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:23.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.362+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c08053cf0 con 0x7f8c1806a4b0 2023-10-14T18:19:23.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:23.601+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8c18069ff0 con 0x7f8c1806a4b0 2023-10-14T18:19:24.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.099+0000 7f8c0f7fe640 1 -- 172.21.15.29:0/251391066 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "backup_fs", "yes_i_really_mean_it": true}]=0 v133) v1 ==== 107+0+0 (secure 0 0 0) 0x7f8c08057310 con 0x7f8c1806a4b0 2023-10-14T18:19:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 msgr2=0x7f8bf806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 0x7f8bf806d570 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f8c1806f690 tx=0x7f8c00009040 comp rx=0 tx=0).stop 2023-10-14T18:19:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 msgr2=0x7f8c18159610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 secure :-1 s=READY pgs=3074 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c400 tx=0x7f8c0800c8c0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 shutdown_connections 2023-10-14T18:19:24.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f8bf806b0b0 0x7f8bf806d570 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c1815ad70 0x7f8c180c4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c180caac0 0x7f8c180ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.102+0000 7f8c1d494640 1 --2- 172.21.15.29:0/251391066 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c1806a4b0 0x7f8c18159610 unknown :-1 s=CLOSED pgs=3074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.103+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 >> 172.21.15.29:0/251391066 conn(0x7f8c180ce9b0 msgr2=0x7f8c1815d840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:24.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.103+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 shutdown_connections 2023-10-14T18:19:24.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.103+0000 7f8c1d494640 1 -- 172.21.15.29:0/251391066 wait complete. 2023-10-14T18:19:24.124 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_metadata backup_fs_metadata --yes-i-really-really-mean-it 2023-10-14T18:19:24.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.359+0000 7f28021ef640 1 Processor -- start 2023-10-14T18:19:24.360 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.359+0000 7f28021ef640 1 -- start start 2023-10-14T18:19:24.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.360+0000 7f28021ef640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27fc15dde0 0x7f27fc0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:24.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.360+0000 7f28021ef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc160a90 con 0x7f27fc070f20 2023-10-14T18:19:24.361 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.360+0000 7f28021ef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27fc0cd900 con 0x7f27fc15c9f0 2023-10-14T18:19:24.362 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.360+0000 7f28021ef640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27fc071350 con 0x7f27fc15dde0 2023-10-14T18:19:24.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.361+0000 7f28009ec640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f27fc15dde0 0x7f27fc0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:24.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.361+0000 7f28019ee640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f27fc070f20 0x7f27fc0ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48750/0 (socket says 172.21.15.29:48750) 2023-10-14T18:19:24.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.361+0000 7f28019ee640 1 -- 172.21.15.29:0/3719756900 learned_addr learned my addr 172.21.15.29:0/3719756900 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:24.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.362+0000 7f28009ec640 1 -- 172.21.15.29:0/3719756900 >> v1:172.21.15.186:6789/0 conn(0x7f27fc15c9f0 legacy=0x7f27fc0c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.362+0000 7f28009ec640 1 -- 172.21.15.29:0/3719756900 >> v1:172.21.15.29:6789/0 conn(0x7f27fc070f20 legacy=0x7f27fc0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.362+0000 7f27f3fff640 1 -- 172.21.15.29:0/3719756900 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3474719886 0 0) 0x7f27fc160a90 con 0x7f27fc070f20 2023-10-14T18:19:24.364 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.362+0000 7f28009ec640 1 -- 172.21.15.29:0/3719756900 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27fc0cdbe0 con 0x7f27fc15dde0 2023-10-14T18:19:24.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.363+0000 7f28009ec640 1 --2- 172.21.15.29:0/3719756900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15dde0 0x7f27fc0709e0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceeeea561306e24 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:24.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.363+0000 7f27f3fff640 1 -- 172.21.15.29:0/3719756900 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec030610 con 0x7f27fc15dde0 2023-10-14T18:19:24.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.363+0000 7f27f3fff640 1 -- 172.21.15.29:0/3719756900 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec030770 con 0x7f27fc15dde0 2023-10-14T18:19:24.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.363+0000 7f27f3fff640 1 -- 172.21.15.29:0/3719756900 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec030b00 con 0x7f27fc15dde0 2023-10-14T18:19:24.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 -- 172.21.15.29:0/3719756900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15dde0 msgr2=0x7f27fc0709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 --2- 172.21.15.29:0/3719756900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15dde0 0x7f27fc0709e0 secure :-1 s=READY pgs=3076 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0097a0 tx=0x7f27ec02b960 comp rx=0 tx=0).stop 2023-10-14T18:19:24.366 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 -- 172.21.15.29:0/3719756900 shutdown_connections 2023-10-14T18:19:24.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 --2- 172.21.15.29:0/3719756900 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15dde0 0x7f27fc0709e0 unknown :-1 s=CLOSED pgs=3076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 -- 172.21.15.29:0/3719756900 >> 172.21.15.29:0/3719756900 conn(0x7f27fc0ce6a0 msgr2=0x7f27fc0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:24.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 -- 172.21.15.29:0/3719756900 shutdown_connections 2023-10-14T18:19:24.367 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.364+0000 7f28021ef640 1 -- 172.21.15.29:0/3719756900 wait complete. 2023-10-14T18:19:24.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.366+0000 7f28021ef640 1 Processor -- start 2023-10-14T18:19:24.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.366+0000 7f28021ef640 1 -- start start 2023-10-14T18:19:24.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 0x7f27fc14cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27fc15dde0 0x7f27fc157c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc0c5f90 con 0x7f27fc15c9f0 2023-10-14T18:19:24.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc069f20 con 0x7f27fc15dde0 2023-10-14T18:19:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.367+0000 7f28021ef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27fc06a200 con 0x7f27fc070f20 2023-10-14T18:19:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47092/0 (socket says 172.21.15.29:47092) 2023-10-14T18:19:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 -- 172.21.15.29:0/1564407355 learned_addr learned my addr 172.21.15.29:0/1564407355 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:24.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28019ee640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27fc15dde0 0x7f27fc157c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28011ed640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 0x7f27fc14cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 -- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 msgr2=0x7f27fc14cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 0x7f27fc14cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.372 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 -- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27fc15dde0 msgr2=0x7f27fc157c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27fc15dde0 0x7f27fc157c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28009ec640 1 -- 172.21.15.29:0/1564407355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27ec009400 con 0x7f27fc15c9f0 2023-10-14T18:19:24.373 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.368+0000 7f28011ed640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 0x7f27fc14cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:24.374 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f28009ec640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098c0 tx=0x7f27ec004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec03e040 con 0x7f27fc15c9f0 2023-10-14T18:19:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27ec039cb0 con 0x7f27fc15c9f0 2023-10-14T18:19:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27ec044630 con 0x7f27fc15c9f0 2023-10-14T18:19:24.375 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27fc06a4e0 con 0x7f27fc15c9f0 2023-10-14T18:19:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.369+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27fc14d0b0 con 0x7f27fc15c9f0 2023-10-14T18:19:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.370+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27fc0c5a80 con 0x7f27fc15c9f0 2023-10-14T18:19:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.372+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f27ec043390 con 0x7f27fc15c9f0 2023-10-14T18:19:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.372+0000 7f27f1ffb640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 0x7f27e006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:24.376 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.373+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(260..260 src has 1..260) v4 ==== 6869+0+0 (secure 0 0 0) 0x7f27ec0ac3a0 con 0x7f27fc15c9f0 2023-10-14T18:19:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.373+0000 7f28011ed640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 0x7f27e006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:24.377 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.374+0000 7f28011ed640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 0x7f27e006d570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f60 tx=0x7f27f8009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:24.378 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.377+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f27fc0c5a80 con 0x7f27fc15c9f0 2023-10-14T18:19:24.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:24.573+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f27fc0c5a80 con 0x7f27fc15c9f0 2023-10-14T18:19:25.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.324+0000 7f27f1ffb640 1 -- 172.21.15.29:0/1564407355 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_metadata", "pool2": "backup_fs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_metadata' removed v261) v1 ==== 190+0+0 (secure 0 0 0) 0x7f27fc0c5a80 con 0x7f27fc15c9f0 2023-10-14T18:19:25.326 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_metadata' removed 2023-10-14T18:19:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 msgr2=0x7f27e006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 0x7f27e006d570 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f27f8000f60 tx=0x7f27f8009040 comp rx=0 tx=0).stop 2023-10-14T18:19:25.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 msgr2=0x7f27fc157730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 secure :-1 s=READY pgs=3077 cs=0 l=1 rev1=1 crypto rx=0x7f27ec0098c0 tx=0x7f27ec004730 comp rx=0 tx=0).stop 2023-10-14T18:19:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 shutdown_connections 2023-10-14T18:19:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f27e006b0b0 0x7f27e006d570 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f27fc15dde0 0x7f27fc157c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f27fc15c9f0 0x7f27fc157730 unknown :-1 s=CLOSED pgs=3077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 --2- 172.21.15.29:0/1564407355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f27fc070f20 0x7f27fc14cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 >> 172.21.15.29:0/1564407355 conn(0x7f27fc0ce6a0 msgr2=0x7f27fc0b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 shutdown_connections 2023-10-14T18:19:25.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.328+0000 7f28021ef640 1 -- 172.21.15.29:0/1564407355 wait complete. 2023-10-14T18:19:25.351 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm backup_fs_data backup_fs_data --yes-i-really-really-mean-it 2023-10-14T18:19:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.581+0000 7fda562e5640 1 Processor -- start 2023-10-14T18:19:25.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.581+0000 7fda562e5640 1 -- start start 2023-10-14T18:19:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda562e5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fda5006c040 0x7fda5006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda562e5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda500ad970 0x7fda5015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda562e5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5006c960 con 0x7fda5015ebc0 2023-10-14T18:19:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda562e5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5015b9d0 con 0x7fda500ad970 2023-10-14T18:19:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda562e5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5015bcb0 con 0x7fda5006c040 2023-10-14T18:19:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda54ae2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fda500ad970 0x7fda5015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda55ae4640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fda5015ebc0 0x7fda500cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48760/0 (socket says 172.21.15.29:48760) 2023-10-14T18:19:25.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.583+0000 7fda55ae4640 1 -- 172.21.15.29:0/3133012934 learned_addr learned my addr 172.21.15.29:0/3133012934 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.584+0000 7fda552e3640 1 --2- 172.21.15.29:0/3133012934 >> v2:172.21.15.186:3300/0 conn(0x7fda5006c040 0x7fda5006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.584+0000 7fda54ae2640 1 -- 172.21.15.29:0/3133012934 >> v2:172.21.15.186:3300/0 conn(0x7fda5006c040 msgr2=0x7fda5006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.584+0000 7fda54ae2640 1 --2- 172.21.15.29:0/3133012934 >> v2:172.21.15.186:3300/0 conn(0x7fda5006c040 0x7fda5006c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.584+0000 7fda54ae2640 1 -- 172.21.15.29:0/3133012934 >> v1:172.21.15.29:6789/0 conn(0x7fda5015ebc0 legacy=0x7fda500cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.584+0000 7fda54ae2640 1 -- 172.21.15.29:0/3133012934 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5015bf90 con 0x7fda500ad970 2023-10-14T18:19:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.585+0000 7fda54ae2640 1 --2- 172.21.15.29:0/3133012934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda500ad970 0x7fda5015e680 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7fda500cf350 tx=0x7fda4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=893b7cd19c13a15e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:25.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.585+0000 7fda47fff640 1 -- 172.21.15.29:0/3133012934 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40031830 con 0x7fda500ad970 2023-10-14T18:19:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.585+0000 7fda47fff640 1 -- 172.21.15.29:0/3133012934 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda40031990 con 0x7fda500ad970 2023-10-14T18:19:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.585+0000 7fda47fff640 1 -- 172.21.15.29:0/3133012934 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda40031d20 con 0x7fda500ad970 2023-10-14T18:19:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 -- 172.21.15.29:0/3133012934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda500ad970 msgr2=0x7fda5015e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 --2- 172.21.15.29:0/3133012934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda500ad970 0x7fda5015e680 secure :-1 s=READY pgs=3079 cs=0 l=1 rev1=1 crypto rx=0x7fda500cf350 tx=0x7fda4002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 -- 172.21.15.29:0/3133012934 shutdown_connections 2023-10-14T18:19:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 --2- 172.21.15.29:0/3133012934 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda500ad970 0x7fda5015e680 unknown :-1 s=CLOSED pgs=3079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 --2- 172.21.15.29:0/3133012934 >> v2:172.21.15.186:3300/0 conn(0x7fda5006c040 0x7fda5006c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 -- 172.21.15.29:0/3133012934 >> 172.21.15.29:0/3133012934 conn(0x7fda500ce6a0 msgr2=0x7fda500ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 -- 172.21.15.29:0/3133012934 shutdown_connections 2023-10-14T18:19:25.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.586+0000 7fda562e5640 1 -- 172.21.15.29:0/3133012934 wait complete. 2023-10-14T18:19:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.588+0000 7fda562e5640 1 Processor -- start 2023-10-14T18:19:25.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.588+0000 7fda562e5640 1 -- start start 2023-10-14T18:19:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda500ad970 0x7fda500ae100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda5015ebc0 0x7fda500ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda50150020 con 0x7fda5006c040 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda500ae640 con 0x7fda500ad970 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda562e5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda500ae8e0 con 0x7fda5015ebc0 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47106/0 (socket says 172.21.15.29:47106) 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 -- 172.21.15.29:0/2366949053 learned_addr learned my addr 172.21.15.29:0/2366949053 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:25.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda55ae4640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda5015ebc0 0x7fda500ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda54ae2640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda500ad970 0x7fda500ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 -- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda5015ebc0 msgr2=0x7fda500ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda5015ebc0 0x7fda500ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 -- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda500ad970 msgr2=0x7fda500ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda500ad970 0x7fda500ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.589+0000 7fda552e3640 1 -- 172.21.15.29:0/2366949053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda40009400 con 0x7fda5006c040 2023-10-14T18:19:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda552e3640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d5f0 tx=0x7fda4c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c0125f0 con 0x7fda5006c040 2023-10-14T18:19:25.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c012750 con 0x7fda5006c040 2023-10-14T18:19:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda4c012b00 con 0x7fda5006c040 2023-10-14T18:19:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda500aebc0 con 0x7fda5006c040 2023-10-14T18:19:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.590+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda500af180 con 0x7fda5006c040 2023-10-14T18:19:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.591+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda5006b330 con 0x7fda5006c040 2023-10-14T18:19:25.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.592+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fda4c012c60 con 0x7fda5006c040 2023-10-14T18:19:25.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.594+0000 7fda45ffb640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 0x7fda3406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:25.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.594+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(261..261 src has 1..261) v4 ==== 6425+0+0 (secure 0 0 0) 0x7fda4c08a280 con 0x7fda5006c040 2023-10-14T18:19:25.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.594+0000 7fda54ae2640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 0x7fda3406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:25.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.595+0000 7fda54ae2640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 0x7fda3406d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda4003c040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:25.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.598+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fda4c054c60 con 0x7fda5006c040 2023-10-14T18:19:25.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:25.785+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fda5004e9b0 con 0x7fda5006c040 2023-10-14T18:19:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.328+0000 7fda45ffb640 1 -- 172.21.15.29:0/2366949053 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "backup_fs_data", "pool2": "backup_fs_data", "yes_i_really_really_mean_it": true}]=0 pool 'backup_fs_data' removed v262) v1 ==== 178+0+0 (secure 0 0 0) 0x7fda4c058280 con 0x7fda5006c040 2023-10-14T18:19:26.329 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'backup_fs_data' removed 2023-10-14T18:19:26.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.332+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 msgr2=0x7fda3406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.332+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 0x7fda3406d570 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fda40036000 tx=0x7fda4003c040 comp rx=0 tx=0).stop 2023-10-14T18:19:26.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.332+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 msgr2=0x7fda5014fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.332+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 secure :-1 s=READY pgs=3080 cs=0 l=1 rev1=1 crypto rx=0x7fda4c00d5f0 tx=0x7fda4c00dab0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 shutdown_connections 2023-10-14T18:19:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fda3406b0b0 0x7fda3406d570 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fda5015ebc0 0x7fda500ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fda500ad970 0x7fda500ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 --2- 172.21.15.29:0/2366949053 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fda5006c040 0x7fda5014fae0 unknown :-1 s=CLOSED pgs=3080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.333+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 >> 172.21.15.29:0/2366949053 conn(0x7fda500ce6a0 msgr2=0x7fda5015ddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:26.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.334+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 shutdown_connections 2023-10-14T18:19:26.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.336+0000 7fda562e5640 1 -- 172.21.15.29:0/2366949053 wait complete. 2023-10-14T18:19:26.363 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T18:19:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.570+0000 7f9503111640 1 Processor -- start 2023-10-14T18:19:26.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.570+0000 7f9503111640 1 -- start start 2023-10-14T18:19:26.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f9503111640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f94fc070f30 0x7f94fc06aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f9503111640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94fc0c9de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f9503111640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94fc0cd260 con 0x7f94fc159ad0 2023-10-14T18:19:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f9503111640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94fc15de30 con 0x7f94fc070f30 2023-10-14T18:19:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f9503111640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc15e110 con 0x7f94fc06b3e0 2023-10-14T18:19:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f950210f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f94fc159ad0 0x7f94fc15d650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48770/0 (socket says 172.21.15.29:48770) 2023-10-14T18:19:26.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.572+0000 7f950210f640 1 -- 172.21.15.29:0/3152154497 learned_addr learned my addr 172.21.15.29:0/3152154497 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:26.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f9502910640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94fc0c9de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589436827 0 0) 0x7f94fc0cd260 con 0x7f94fc159ad0 2023-10-14T18:19:26.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94e8003520 con 0x7f94fc159ad0 2023-10-14T18:19:26.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950190e640 1 --2- 172.21.15.29:0/3152154497 >> v2:172.21.15.186:3300/0 conn(0x7f94fc070f30 0x7f94fc06aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2710775939 0 0) 0x7f94e8003520 con 0x7f94fc159ad0 2023-10-14T18:19:26.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 msgr2=0x7f94fc0c9de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:26.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94fc0c9de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 >> v2:172.21.15.186:3300/0 conn(0x7f94fc070f30 msgr2=0x7f94fc06aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 --2- 172.21.15.29:0/3152154497 >> v2:172.21.15.186:3300/0 conn(0x7f94fc070f30 0x7f94fc06aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc15e3f0 con 0x7f94fc159ad0 2023-10-14T18:19:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f94f8001670 con 0x7f94fc159ad0 2023-10-14T18:19:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 >> v1:172.21.15.29:6789/0 conn(0x7f94fc159ad0 legacy=0x7f94fc15d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94e8003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8003ef0 0x7f94e80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8004d00 0x7f94e8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc15de30 con 0x7f94e8004d00 2023-10-14T18:19:26.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc15e110 con 0x7f94e8003ef0 2023-10-14T18:19:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94e800b9d0 con 0x7f94fc06b3e0 2023-10-14T18:19:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.573+0000 7f950190e640 1 --2- 172.21.15.29:0/3152154497 >> v2:172.21.15.186:3300/0 conn(0x7f94fc070f30 0x7f94fc06aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8004d00 0x7f94e8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f950210f640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94e8003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 -- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 msgr2=0x7f94e8003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc06b3e0 0x7f94e8003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 -- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8003ef0 msgr2=0x7f94e80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8003ef0 0x7f94e80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.574+0000 7f9502910640 1 -- 172.21.15.29:0/3152154497 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e8003520 con 0x7f94e8004d00 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.575+0000 7f9502910640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8004d00 0x7f94e8005180 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009fc0 tx=0x7f94e80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e23a6d5f4ae1c6d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:26.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.575+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f402cc20 con 0x7f94e8004d00 2023-10-14T18:19:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94f400f040 con 0x7f94e8004d00 2023-10-14T18:19:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f950110d640 1 -- 172.21.15.29:0/3152154497 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94f4034440 con 0x7f94e8004d00 2023-10-14T18:19:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 -- 172.21.15.29:0/3152154497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8004d00 msgr2=0x7f94e8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 --2- 172.21.15.29:0/3152154497 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94e8004d00 0x7f94e8005180 secure :-1 s=READY pgs=3082 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009fc0 tx=0x7f94e80057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9502910640 1 -- 172.21.15.29:0/3152154497 reap_dead start 2023-10-14T18:19:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 -- 172.21.15.29:0/3152154497 shutdown_connections 2023-10-14T18:19:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 -- 172.21.15.29:0/3152154497 >> 172.21.15.29:0/3152154497 conn(0x7f94fc0b0170 msgr2=0x7f94fc0b0580 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 -- 172.21.15.29:0/3152154497 shutdown_connections 2023-10-14T18:19:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.576+0000 7f9503111640 1 -- 172.21.15.29:0/3152154497 wait complete. 2023-10-14T18:19:26.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.578+0000 7f9503111640 1 Processor -- start 2023-10-14T18:19:26.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.578+0000 7f9503111640 1 -- start start 2023-10-14T18:19:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8004d00 0x7f94fc13bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc152ba0 0x7f94fc153030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc153570 con 0x7f94fc1538f0 2023-10-14T18:19:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc139940 con 0x7f94e8004d00 2023-10-14T18:19:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.579+0000 7f9503111640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94fc14fb80 con 0x7f94fc152ba0 2023-10-14T18:19:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47122/0 (socket says 172.21.15.29:47122) 2023-10-14T18:19:26.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 -- 172.21.15.29:0/2201107697 learned_addr learned my addr 172.21.15.29:0/2201107697 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f9502910640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc152ba0 0x7f94fc153030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950210f640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8004d00 0x7f94fc13bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 -- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc152ba0 msgr2=0x7f94fc153030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc152ba0 0x7f94fc153030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 -- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8004d00 msgr2=0x7f94fc13bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8004d00 0x7f94fc13bb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 -- 172.21.15.29:0/2201107697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94fc153d90 con 0x7f94fc1538f0 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.580+0000 7f950190e640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00e810 tx=0x7f94ec00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.581+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec004030 con 0x7f94fc1538f0 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.581+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ec004190 con 0x7f94fc1538f0 2023-10-14T18:19:26.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.581+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94ec00c6d0 con 0x7f94fc1538f0 2023-10-14T18:19:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.581+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94fc1392f0 con 0x7f94fc1538f0 2023-10-14T18:19:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.581+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94fc154070 con 0x7f94fc1538f0 2023-10-14T18:19:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.582+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94fc04e9b0 con 0x7f94fc1538f0 2023-10-14T18:19:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.583+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f94ec00c830 con 0x7f94fc1538f0 2023-10-14T18:19:26.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.584+0000 7f94f2ffd640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 0x7f94e006d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.584+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f94ec089950 con 0x7f94fc1538f0 2023-10-14T18:19:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.584+0000 7f950210f640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 0x7f94e006d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:26.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.585+0000 7f950210f640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 0x7f94e006d5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94f80058b0 tx=0x7f94f8001d70 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:26.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.589+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f94ec0543c0 con 0x7f94fc1538f0 2023-10-14T18:19:26.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.784+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f94fc139a80 con 0x7f94fc1538f0 2023-10-14T18:19:26.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.790+0000 7f94f2ffd640 1 -- 172.21.15.29:0/2201107697 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v85) v1 ==== 115+0+0 (secure 0 0 0) 0x7f94ec0579e0 con 0x7f94fc1538f0 2023-10-14T18:19:26.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 msgr2=0x7f94e006d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 0x7f94e006d5a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f94f80058b0 tx=0x7f94f8001d70 comp rx=0 tx=0).stop 2023-10-14T18:19:26.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 msgr2=0x7f94fc0c4920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 secure :-1 s=READY pgs=3083 cs=0 l=1 rev1=1 crypto rx=0x7f94ec00e810 tx=0x7f94ec00ecd0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 shutdown_connections 2023-10-14T18:19:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f94e006b0e0 0x7f94e006d5a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f94fc152ba0 0x7f94fc153030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f94fc1538f0 0x7f94fc0c4920 unknown :-1 s=CLOSED pgs=3083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 --2- 172.21.15.29:0/2201107697 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f94e8004d00 0x7f94fc13bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 >> 172.21.15.29:0/2201107697 conn(0x7f94fc0b0170 msgr2=0x7f94fc0aed70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 shutdown_connections 2023-10-14T18:19:26.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:26.793+0000 7f9503111640 1 -- 172.21.15.29:0/2201107697 wait complete. 2023-10-14T18:19:26.815 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T18:19:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.042+0000 7f300cddc640 1 Processor -- start 2023-10-14T18:19:27.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.043+0000 7f300cddc640 1 -- start start 2023-10-14T18:19:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.044+0000 7f300cddc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f300815ad70 0x7f3008069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.044+0000 7f300cddc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f300806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.044+0000 7f300cddc640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30080cb030 con 0x7f30080ca3b0 2023-10-14T18:19:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.044+0000 7f300cddc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30080c91c0 con 0x7f300815ad70 2023-10-14T18:19:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.044+0000 7f300cddc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30080c94a0 con 0x7f30080c9ca0 2023-10-14T18:19:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f3006d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f30080ca3b0 0x7f30080ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48780/0 (socket says 172.21.15.29:48780) 2023-10-14T18:19:27.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f3006d9d640 1 -- 172.21.15.29:0/2744425287 learned_addr learned my addr 172.21.15.29:0/2744425287 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f2ffdd9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f300806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f300659c640 1 --2- 172.21.15.29:0/2744425287 >> v2:172.21.15.186:3300/0 conn(0x7f300815ad70 0x7f3008069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2413820170 0 0) 0x7f30080cb030 con 0x7f30080ca3b0 2023-10-14T18:19:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.045+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ff4003520 con 0x7f30080ca3b0 2023-10-14T18:19:27.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 353694449 0 0) 0x7f2ff4003520 con 0x7f30080ca3b0 2023-10-14T18:19:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 msgr2=0x7f300806a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f300806a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 >> v2:172.21.15.186:3300/0 conn(0x7f300815ad70 msgr2=0x7f3008069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 --2- 172.21.15.29:0/2744425287 >> v2:172.21.15.186:3300/0 conn(0x7f300815ad70 0x7f3008069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080c9780 con 0x7f30080ca3b0 2023-10-14T18:19:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3000001670 con 0x7f30080ca3b0 2023-10-14T18:19:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 >> v1:172.21.15.29:6789/0 conn(0x7f30080ca3b0 legacy=0x7f30080ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f2ff4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff4003ef0 0x7f2ff40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f2ff4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30080c91c0 con 0x7f2ff4004d00 2023-10-14T18:19:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30080c94a0 con 0x7f2ff4003ef0 2023-10-14T18:19:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ff400b980 con 0x7f30080c9ca0 2023-10-14T18:19:27.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f3006d9d640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f2ff4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f300659c640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f2ff4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.046+0000 7f2ffdd9b640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff4003ef0 0x7f2ff40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.047+0000 7f3006d9d640 1 -- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 msgr2=0x7f2ff4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.047+0000 7f3006d9d640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080c9ca0 0x7f2ff4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.047+0000 7f3006d9d640 1 -- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff4003ef0 msgr2=0x7f2ff40047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.047+0000 7f3006d9d640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2ff4003ef0 0x7f2ff40047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.047+0000 7f3006d9d640 1 -- 172.21.15.29:0/2744425287 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ff4003520 con 0x7f2ff4004d00 2023-10-14T18:19:27.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.048+0000 7f3006d9d640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f2ff4005180 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f300000c240 tx=0x7f2ff40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6bfdd9741da8ee8c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.048+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300002cae0 con 0x7f2ff4004d00 2023-10-14T18:19:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.048+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300002cc40 con 0x7f2ff4004d00 2023-10-14T18:19:27.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.048+0000 7f3005d9b640 1 -- 172.21.15.29:0/2744425287 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000005af0 con 0x7f2ff4004d00 2023-10-14T18:19:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 -- 172.21.15.29:0/2744425287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 msgr2=0x7f2ff4005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 --2- 172.21.15.29:0/2744425287 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f2ff4005180 secure :-1 s=READY pgs=3085 cs=0 l=1 rev1=1 crypto rx=0x7f300000c240 tx=0x7f2ff40057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f3006d9d640 1 -- 172.21.15.29:0/2744425287 reap_dead start 2023-10-14T18:19:27.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 -- 172.21.15.29:0/2744425287 shutdown_connections 2023-10-14T18:19:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 -- 172.21.15.29:0/2744425287 >> 172.21.15.29:0/2744425287 conn(0x7f30080ce9b0 msgr2=0x7f30080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 -- 172.21.15.29:0/2744425287 shutdown_connections 2023-10-14T18:19:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.049+0000 7f300cddc640 1 -- 172.21.15.29:0/2744425287 wait complete. 2023-10-14T18:19:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.051+0000 7f300cddc640 1 Processor -- start 2023-10-14T18:19:27.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.051+0000 7f300cddc640 1 -- start start 2023-10-14T18:19:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080b29c0 0x7f30080b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080a8dd0 0x7f30080a9280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30080b3380 con 0x7f2ff4004d00 2023-10-14T18:19:27.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30080aa730 con 0x7f30080a8dd0 2023-10-14T18:19:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300cddc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30080a9b20 con 0x7f30080b29c0 2023-10-14T18:19:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47134/0 (socket says 172.21.15.29:47134) 2023-10-14T18:19:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 -- 172.21.15.29:0/2531859511 learned_addr learned my addr 172.21.15.29:0/2531859511 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:27.061 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f2ffdd9b640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080b29c0 0x7f30080b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 -- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080b29c0 msgr2=0x7f30080b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080b29c0 0x7f30080b2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f300659c640 1 -- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080a8dd0 msgr2=0x7f30080a9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.052+0000 7f3006d9d640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080a8dd0 0x7f30080a9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f300659c640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080a8dd0 0x7f30080a9280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f300659c640 1 -- 172.21.15.29:0/2531859511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30080a9e00 con 0x7f2ff4004d00 2023-10-14T18:19:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f300659c640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8011700 tx=0x7f2ff8011bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800f9c0 con 0x7f2ff4004d00 2023-10-14T18:19:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ff800fb20 con 0x7f2ff4004d00 2023-10-14T18:19:27.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ff800fe90 con 0x7f2ff4004d00 2023-10-14T18:19:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30080aa0e0 con 0x7f2ff4004d00 2023-10-14T18:19:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.053+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30080a97c0 con 0x7f2ff4004d00 2023-10-14T18:19:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.055+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30080acd00 con 0x7f2ff4004d00 2023-10-14T18:19:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.056+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f2ff8015460 con 0x7f2ff4004d00 2023-10-14T18:19:27.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.057+0000 7f2fff7fe640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 0x7f2fe806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.057+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f2ff808a0a0 con 0x7f2ff4004d00 2023-10-14T18:19:27.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.057+0000 7f2ffdd9b640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 0x7f2fe806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.058+0000 7f2ffdd9b640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 0x7f2fe806d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30080c51e0 tx=0x7f2ff000b040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.062+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2ff8054cc0 con 0x7f2ff4004d00 2023-10-14T18:19:27.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.257+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f300804e160 con 0x7f2ff4004d00 2023-10-14T18:19:27.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.265+0000 7f2fff7fe640 1 -- 172.21.15.29:0/2531859511 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v86) v1 ==== 120+0+0 (secure 0 0 0) 0x7f2ff80582e0 con 0x7f2ff4004d00 2023-10-14T18:19:27.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 msgr2=0x7f2fe806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 0x7f2fe806d5a0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f30080c51e0 tx=0x7f2ff000b040 comp rx=0 tx=0).stop 2023-10-14T18:19:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 msgr2=0x7f300814bf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 secure :-1 s=READY pgs=3086 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8011700 tx=0x7f2ff8011bc0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 shutdown_connections 2023-10-14T18:19:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f2fe806b0e0 0x7f2fe806d5a0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f30080a8dd0 0x7f30080a9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f30080b29c0 0x7f30080b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 --2- 172.21.15.29:0/2531859511 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2ff4004d00 0x7f300814bf20 unknown :-1 s=CLOSED pgs=3086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 >> 172.21.15.29:0/2531859511 conn(0x7f30080ce9b0 msgr2=0x7f30080b3910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 shutdown_connections 2023-10-14T18:19:27.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.268+0000 7f300cddc640 1 -- 172.21.15.29:0/2531859511 wait complete. 2023-10-14T18:19:27.290 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T18:19:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.512+0000 7faa83459640 1 Processor -- start 2023-10-14T18:19:27.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.512+0000 7faa83459640 1 -- start start 2023-10-14T18:19:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa83459640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 0x7faa7c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa83459640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa7c0cda80 0x7faa7c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa83459640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7c06c960 con 0x7faa7c0ad970 2023-10-14T18:19:27.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa83459640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c15b9d0 con 0x7faa7c0cda80 2023-10-14T18:19:27.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa83459640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa7c15bcb0 con 0x7faa7c06c040 2023-10-14T18:19:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa82c58640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa7c0cda80 0x7faa7c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.514+0000 7faa82c58640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7faa7c0cda80 0x7faa7c15ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47142/0 (socket says 172.21.15.29:47142) 2023-10-14T18:19:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82c58640 1 -- 172.21.15.29:0/3880416754 learned_addr learned my addr 172.21.15.29:0/3880416754 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:27.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82457640 1 --2- 172.21.15.29:0/3880416754 >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 0x7faa7c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82c58640 1 -- 172.21.15.29:0/3880416754 >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 msgr2=0x7faa7c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82c58640 1 --2- 172.21.15.29:0/3880416754 >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 0x7faa7c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82c58640 1 -- 172.21.15.29:0/3880416754 >> v1:172.21.15.186:6789/0 conn(0x7faa7c0ad970 legacy=0x7faa7c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82c58640 1 -- 172.21.15.29:0/3880416754 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7c15bf90 con 0x7faa7c0cda80 2023-10-14T18:19:27.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.515+0000 7faa82457640 1 --2- 172.21.15.29:0/3880416754 >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 0x7faa7c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:27.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.516+0000 7faa82c58640 1 --2- 172.21.15.29:0/3880416754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0cda80 0x7faa7c15ed90 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0cf350 tx=0x7faa7402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c41b77b8be3fd4e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.516+0000 7faa81455640 1 -- 172.21.15.29:0/3880416754 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7402ca80 con 0x7faa7c0cda80 2023-10-14T18:19:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa81455640 1 -- 172.21.15.29:0/3880416754 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa7402cbe0 con 0x7faa7c0cda80 2023-10-14T18:19:27.523 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa81455640 1 -- 172.21.15.29:0/3880416754 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa740343f0 con 0x7faa7c0cda80 2023-10-14T18:19:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 -- 172.21.15.29:0/3880416754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0cda80 msgr2=0x7faa7c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 --2- 172.21.15.29:0/3880416754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0cda80 0x7faa7c15ed90 secure :-1 s=READY pgs=3087 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0cf350 tx=0x7faa7402bee0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 -- 172.21.15.29:0/3880416754 shutdown_connections 2023-10-14T18:19:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 --2- 172.21.15.29:0/3880416754 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0cda80 0x7faa7c15ed90 unknown :-1 s=CLOSED pgs=3087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.524 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 --2- 172.21.15.29:0/3880416754 >> v2:172.21.15.186:3300/0 conn(0x7faa7c06c040 0x7faa7c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 -- 172.21.15.29:0/3880416754 >> 172.21.15.29:0/3880416754 conn(0x7faa7c0ce6a0 msgr2=0x7faa7c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.517+0000 7faa83459640 1 -- 172.21.15.29:0/3880416754 shutdown_connections 2023-10-14T18:19:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.518+0000 7faa83459640 1 -- 172.21.15.29:0/3880416754 wait complete. 2023-10-14T18:19:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.519+0000 7faa83459640 1 Processor -- start 2023-10-14T18:19:27.525 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.519+0000 7faa83459640 1 -- start start 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa7c06c040 0x7faa7c146b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa7c0cda80 0x7faa7c150fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0c62e0 con 0x7faa7c0ad970 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c15e080 con 0x7faa7c0cda80 2023-10-14T18:19:27.526 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.520+0000 7faa83459640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa7c0c6530 con 0x7faa7c06c040 2023-10-14T18:19:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47150/0 (socket says 172.21.15.29:47150) 2023-10-14T18:19:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 -- 172.21.15.29:0/265306701 learned_addr learned my addr 172.21.15.29:0/265306701 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa82c58640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa7c0cda80 0x7faa7c150fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.527 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa82457640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa7c06c040 0x7faa7c146b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 -- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa7c06c040 msgr2=0x7faa7c146b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa7c06c040 0x7faa7c146b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 -- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa7c0cda80 msgr2=0x7faa7c150fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa7c0cda80 0x7faa7c150fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.521+0000 7faa81c56640 1 -- 172.21.15.29:0/265306701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa74009b30 con 0x7faa7c0ad970 2023-10-14T18:19:27.528 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa81c56640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00c400 tx=0x7faa6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c011790 con 0x7faa7c0ad970 2023-10-14T18:19:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa6c011930 con 0x7faa7c0ad970 2023-10-14T18:19:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa6c011ce0 con 0x7faa7c0ad970 2023-10-14T18:19:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa7c0c6810 con 0x7faa7c0ad970 2023-10-14T18:19:27.529 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.522+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7c147070 con 0x7faa7c0ad970 2023-10-14T18:19:27.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.523+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa7c06b330 con 0x7faa7c0ad970 2023-10-14T18:19:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.524+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faa6c01c460 con 0x7faa7c0ad970 2023-10-14T18:19:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.525+0000 7faa72ffd640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 0x7faa6006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.525+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7faa6c089ad0 con 0x7faa7c0ad970 2023-10-14T18:19:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.526+0000 7faa82457640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 0x7faa6006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.526+0000 7faa82457640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 0x7faa6006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c5940 tx=0x7faa78008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:27.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.530+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faa6c054720 con 0x7faa7c0ad970 2023-10-14T18:19:27.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.741+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7faa7c04e9b0 con 0x7faa7c0ad970 2023-10-14T18:19:27.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.748+0000 7faa72ffd640 1 -- 172.21.15.29:0/265306701 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v87) v1 ==== 104+0+0 (secure 0 0 0) 0x7faa6c057d40 con 0x7faa7c0ad970 2023-10-14T18:19:27.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.751+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 msgr2=0x7faa6006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.751+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 0x7faa6006d570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7faa7c0c5940 tx=0x7faa78008040 comp rx=0 tx=0).stop 2023-10-14T18:19:27.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 msgr2=0x7faa7c150a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 secure :-1 s=READY pgs=3088 cs=0 l=1 rev1=1 crypto rx=0x7faa6c00c400 tx=0x7faa6c00c8c0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 shutdown_connections 2023-10-14T18:19:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faa6006b0b0 0x7faa6006d570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faa7c0cda80 0x7faa7c150fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faa7c0ad970 0x7faa7c150a60 unknown :-1 s=CLOSED pgs=3088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 --2- 172.21.15.29:0/265306701 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faa7c06c040 0x7faa7c146b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 >> 172.21.15.29:0/265306701 conn(0x7faa7c0ce6a0 msgr2=0x7faa7c15e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 shutdown_connections 2023-10-14T18:19:27.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.752+0000 7faa83459640 1 -- 172.21.15.29:0/265306701 wait complete. 2023-10-14T18:19:27.771 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument' 2023-10-14T18:19:27.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.991+0000 7f9b39026640 1 Processor -- start 2023-10-14T18:19:27.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.991+0000 7f9b39026640 1 -- start start 2023-10-14T18:19:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.992+0000 7f9b39026640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159f00 0x7f9b340b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.992+0000 7f9b39026640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b3415d880 0x7f9b340b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:27.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.992+0000 7f9b39026640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b3406f7f0 con 0x7f9b34159b20 2023-10-14T18:19:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.992+0000 7f9b39026640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9b3406c5e0 con 0x7f9b3415d880 2023-10-14T18:19:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.992+0000 7f9b39026640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3406c8c0 con 0x7f9b34159f00 2023-10-14T18:19:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3359e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9b3415d880 0x7f9b340b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b32d9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f9b34159b20 0x7f9b3406bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48796/0 (socket says 172.21.15.29:48796) 2023-10-14T18:19:27.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b32d9d640 1 -- 172.21.15.29:0/3902323254 learned_addr learned my addr 172.21.15.29:0/3902323254 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3259c640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159f00 0x7f9b340b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:27.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b31d9b640 1 -- 172.21.15.29:0/3902323254 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3768288197 0 0) 0x7f9b3406f7f0 con 0x7f9b34159b20 2023-10-14T18:19:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b31d9b640 1 -- 172.21.15.29:0/3902323254 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b1c003520 con 0x7f9b34159b20 2023-10-14T18:19:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3359e640 1 -- 172.21.15.29:0/3902323254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159f00 msgr2=0x7f9b340b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:27.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3359e640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159f00 0x7f9b340b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3359e640 1 -- 172.21.15.29:0/3902323254 >> v1:172.21.15.29:6789/0 conn(0x7f9b34159b20 legacy=0x7f9b3406bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:27.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.993+0000 7f9b3359e640 1 -- 172.21.15.29:0/3902323254 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b3406cba0 con 0x7f9b3415d880 2023-10-14T18:19:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.994+0000 7f9b3359e640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b3415d880 0x7f9b340b0950 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c02bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3f9b07b4b7efc00 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b31d9b640 1 -- 172.21.15.29:0/3902323254 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c02cc50 con 0x7f9b3415d880 2023-10-14T18:19:28.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b31d9b640 1 -- 172.21.15.29:0/3902323254 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2c02cdb0 con 0x7f9b3415d880 2023-10-14T18:19:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b31d9b640 1 -- 172.21.15.29:0/3902323254 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2c034660 con 0x7f9b3415d880 2023-10-14T18:19:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 -- 172.21.15.29:0/3902323254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b3415d880 msgr2=0x7f9b340b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b3415d880 0x7f9b340b0950 secure :-1 s=READY pgs=3090 cs=0 l=1 rev1=1 crypto rx=0x7f9b2c009d90 tx=0x7f9b2c02bea0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 -- 172.21.15.29:0/3902323254 shutdown_connections 2023-10-14T18:19:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b3415d880 0x7f9b340b0950 unknown :-1 s=CLOSED pgs=3090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 --2- 172.21.15.29:0/3902323254 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159f00 0x7f9b340b0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.995+0000 7f9b39026640 1 -- 172.21.15.29:0/3902323254 >> 172.21.15.29:0/3902323254 conn(0x7f9b340cb410 msgr2=0x7f9b340cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.996+0000 7f9b39026640 1 -- 172.21.15.29:0/3902323254 shutdown_connections 2023-10-14T18:19:28.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.996+0000 7f9b39026640 1 -- 172.21.15.29:0/3902323254 wait complete. 2023-10-14T18:19:28.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.997+0000 7f9b39026640 1 Processor -- start 2023-10-14T18:19:28.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.997+0000 7f9b39026640 1 -- start start 2023-10-14T18:19:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 0x7f9b3415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3415d880 0x7f9b3414adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3415d3e0 con 0x7f9b34159f00 2023-10-14T18:19:28.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b340c4700 con 0x7f9b3415d880 2023-10-14T18:19:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.998+0000 7f9b39026640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b3414b330 con 0x7f9b34159b20 2023-10-14T18:19:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47174/0 (socket says 172.21.15.29:47174) 2023-10-14T18:19:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 -- 172.21.15.29:0/2965727586 learned_addr learned my addr 172.21.15.29:0/2965727586 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:28.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b32d9d640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 0x7f9b3415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 -- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 msgr2=0x7f9b3415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3359e640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3415d880 0x7f9b3414adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 0x7f9b3415b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 -- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3415d880 msgr2=0x7f9b3414adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3415d880 0x7f9b3414adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b3259c640 1 -- 172.21.15.29:0/2965727586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c009a70 con 0x7f9b34159f00 2023-10-14T18:19:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:27.999+0000 7f9b32d9d640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 0x7f9b3415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b3259c640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800d800 tx=0x7f9b2800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2800b6c0 con 0x7f9b34159f00 2023-10-14T18:19:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b2800b820 con 0x7f9b34159f00 2023-10-14T18:19:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9b2800bbb0 con 0x7f9b34159f00 2023-10-14T18:19:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b340c3af0 con 0x7f9b34159f00 2023-10-14T18:19:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.000+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b340c40b0 con 0x7f9b34159f00 2023-10-14T18:19:28.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.001+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b340c48b0 con 0x7f9b34159f00 2023-10-14T18:19:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.002+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9b2800bd10 con 0x7f9b34159f00 2023-10-14T18:19:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.003+0000 7f9b1b7fe640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 0x7f9b2006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.003+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f9b2808a010 con 0x7f9b34159f00 2023-10-14T18:19:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.003+0000 7f9b32d9d640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 0x7f9b2006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.004+0000 7f9b32d9d640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 0x7f9b2006d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b240011e0 tx=0x7f9b24009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.008+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9b340c48b0 con 0x7f9b34159f00 2023-10-14T18:19:28.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.274+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]} v 0) v1 -- 0x7f9b340c48b0 con 0x7f9b34159f00 2023-10-14T18:19:28.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.350+0000 7f9b1b7fe640 1 -- 172.21.15.29:0/2965727586 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesMandatoryFSArgument.test_snap_schedule_without_fs_argument"]}]=0 v3292) v1 ==== 187+0+0 (secure 0 0 0) 0x7f9b340c48b0 con 0x7f9b34159f00 2023-10-14T18:19:28.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 msgr2=0x7f9b2006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 0x7f9b2006d570 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f9b240011e0 tx=0x7f9b24009040 comp rx=0 tx=0).stop 2023-10-14T18:19:28.357 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 msgr2=0x7f9b3415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 secure :-1 s=READY pgs=3091 cs=0 l=1 rev1=1 crypto rx=0x7f9b2800d800 tx=0x7f9b2800dcc0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 shutdown_connections 2023-10-14T18:19:28.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9b2006b0b0 0x7f9b2006d570 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9b3415d880 0x7f9b3414adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.358 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9b34159f00 0x7f9b3415cea0 unknown :-1 s=CLOSED pgs=3091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 --2- 172.21.15.29:0/2965727586 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9b34159b20 0x7f9b3415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 >> 172.21.15.29:0/2965727586 conn(0x7f9b340cb410 msgr2=0x7f9b3406d360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:28.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 shutdown_connections 2023-10-14T18:19:28.359 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.356+0000 7f9b39026640 1 -- 172.21.15.29:0/2965727586 wait complete. 2023-10-14T18:19:28.377 INFO:tasks.cephfs_test_runner:Test command fails without --fs argument in presence of multiple fs ... ok 2023-10-14T18:19:28.378 INFO:tasks.cephfs_test_runner:Starting test: test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) Test the correctness of snap directory name 2023-10-14T18:19:28.379 INFO:tasks.cephfs_test_runner:test_snap_dir_name (tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir) 2023-10-14T18:19:28.379 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:28.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.605+0000 7f23c9f99640 1 Processor -- start 2023-10-14T18:19:28.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.606+0000 7f23c9f99640 1 -- start start 2023-10-14T18:19:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.607+0000 7f23c9f99640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23c40acd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.607+0000 7f23c9f99640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c4160a90 con 0x7f23c415dde0 2023-10-14T18:19:28.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.607+0000 7f23c9f99640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23c40cd900 con 0x7f23c415c9f0 2023-10-14T18:19:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.607+0000 7f23c9f99640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c40cdec0 con 0x7f23c40ac890 2023-10-14T18:19:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c3fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f23c415dde0 0x7f23c40709e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48800/0 (socket says 172.21.15.29:48800) 2023-10-14T18:19:28.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c3fff640 1 -- 172.21.15.29:0/695331466 learned_addr learned my addr 172.21.15.29:0/695331466 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c9798640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23c40acd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425534078 0 0) 0x7f23c4160a90 con 0x7f23c415dde0 2023-10-14T18:19:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b4003520 con 0x7f23c415dde0 2023-10-14T18:19:28.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2517239308 0 0) 0x7f23b4003520 con 0x7f23c415dde0 2023-10-14T18:19:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 msgr2=0x7f23c40acd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23c40acd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 >> v1:172.21.15.186:6789/0 conn(0x7f23c415c9f0 legacy=0x7f23c40c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c40cdbe0 con 0x7f23c415dde0 2023-10-14T18:19:28.613 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.608+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f23b0001670 con 0x7f23c415dde0 2023-10-14T18:19:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 >> v1:172.21.15.29:6789/0 conn(0x7f23c415dde0 legacy=0x7f23c40709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23b40039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4003f70 0x7f23b40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.614 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4004d00 0x7f23b4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b400ba90 con 0x7f23b4003f70 2023-10-14T18:19:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23b400bc00 con 0x7f23b4004d00 2023-10-14T18:19:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23b400bdc0 con 0x7f23c40ac890 2023-10-14T18:19:28.615 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4003f70 0x7f23b40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c9798640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4004d00 0x7f23b4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c8f97640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23b40039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 -- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 msgr2=0x7f23b40039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23b40039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 -- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4004d00 msgr2=0x7f23b4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4004d00 0x7f23b4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c3fff640 1 -- 172.21.15.29:0/695331466 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b4003520 con 0x7f23b4003f70 2023-10-14T18:19:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c8f97640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c40ac890 0x7f23b40039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.609+0000 7f23c9798640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4004d00 0x7f23b4005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:28.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.611+0000 7f23c3fff640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4003f70 0x7f23b40047c0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f23b000c0e0 tx=0x7f23b40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5121f911d68206e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.611+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b0005ad0 con 0x7f23b4003f70 2023-10-14T18:19:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.611+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23b0005c30 con 0x7f23b4003f70 2023-10-14T18:19:28.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.611+0000 7f23c37fe640 1 -- 172.21.15.29:0/695331466 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23b00072b0 con 0x7f23b4003f70 2023-10-14T18:19:28.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 -- 172.21.15.29:0/695331466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4003f70 msgr2=0x7f23b40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 --2- 172.21.15.29:0/695331466 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23b4003f70 0x7f23b40047c0 secure :-1 s=READY pgs=3093 cs=0 l=1 rev1=1 crypto rx=0x7f23b000c0e0 tx=0x7f23b40057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9798640 1 -- 172.21.15.29:0/695331466 reap_dead start 2023-10-14T18:19:28.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 -- 172.21.15.29:0/695331466 shutdown_connections 2023-10-14T18:19:28.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 -- 172.21.15.29:0/695331466 >> 172.21.15.29:0/695331466 conn(0x7f23c40ce6a0 msgr2=0x7f23c40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:28.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 -- 172.21.15.29:0/695331466 shutdown_connections 2023-10-14T18:19:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.612+0000 7f23c9f99640 1 -- 172.21.15.29:0/695331466 wait complete. 2023-10-14T18:19:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.614+0000 7f23c9f99640 1 Processor -- start 2023-10-14T18:19:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.614+0000 7f23c9f99640 1 -- start start 2023-10-14T18:19:28.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4003f70 0x7f23c41f1e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c41f35a0 0x7f23c406c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c406cc90 con 0x7f23c406c810 2023-10-14T18:19:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23c406a700 con 0x7f23b4003f70 2023-10-14T18:19:28.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9f99640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23c4069e40 con 0x7f23c41f35a0 2023-10-14T18:19:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9798640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47194/0 (socket says 172.21.15.29:47194) 2023-10-14T18:19:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.615+0000 7f23c9798640 1 -- 172.21.15.29:0/284453171 learned_addr learned my addr 172.21.15.29:0/284453171 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:28.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c8f97640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4003f70 0x7f23c41f1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c3fff640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c41f35a0 0x7f23c406c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 -- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c41f35a0 msgr2=0x7f23c406c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c41f35a0 0x7f23c406c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 -- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4003f70 msgr2=0x7f23c41f1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4003f70 0x7f23c41f1e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 -- 172.21.15.29:0/284453171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c406a0b0 con 0x7f23c406c810 2023-10-14T18:19:28.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c9798640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009fd0 tx=0x7f23bc00f4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc017400 con 0x7f23c406c810 2023-10-14T18:19:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23bc011040 con 0x7f23c406c810 2023-10-14T18:19:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.616+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23bc017780 con 0x7f23c406c810 2023-10-14T18:19:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.617+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23c406a390 con 0x7f23c406c810 2023-10-14T18:19:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.617+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23c4157040 con 0x7f23c406c810 2023-10-14T18:19:28.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.618+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23c404e9b0 con 0x7f23c406c810 2023-10-14T18:19:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.619+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f23bc0203b0 con 0x7f23c406c810 2023-10-14T18:19:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.620+0000 7f23c17fa640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 0x7f23a806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.620+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f23bc089f00 con 0x7f23c406c810 2023-10-14T18:19:28.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.620+0000 7f23c8f97640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 0x7f23a806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.621+0000 7f23c8f97640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 0x7f23a806d5a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23b8001750 tx=0x7f23b8009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:28.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.625+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f23bc054aa0 con 0x7f23c406c810 2023-10-14T18:19:28.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.829+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23c40c1710 con 0x7f23c406c810 2023-10-14T18:19:28.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.831+0000 7f23c17fa640 1 -- 172.21.15.29:0/284453171 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v262) v1 ==== 74+0+14552 (secure 0 0 0) 0x7f23bc0580c0 con 0x7f23c406c810 2023-10-14T18:19:28.833 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:28.833 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":262,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:26.324867+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6833/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.186:6835/2447498482":"2023-10-15T18:19:23.085091+0000","172.21.15.29:0/2636395246":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/3784066686":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6833/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.29:6832/1194987085":"2023-10-15T18:19:09.147468+0000","172.21.15.186:6834/2447498482":"2023-10-15T18:19:23.085091+0000","172.21.15.29:0/3194749708":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2050715181":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/2739964416":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/1100294912":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/306027610":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6833/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:6832/4275371095":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/2052138313":"2023-10-15T18:19:02.726508+0000","172.21.15.186:6832/2448419287":"2023-10-15T18:19:14.897311+0000","172.21.15.29:0/3668953662":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/1653397166":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/254602771":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/4136574479":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3197818126":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/755870186":"2023-10-15T18:19:09.147468+0000","172.21.15.29:0/143268764":"2023-10-15T18:19:02.726508+0000","172.21.15.29:0/3241859590":"2023-10-15T18:19:02.726508+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 msgr2=0x7f23a806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 0x7f23a806d5a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f23b8001750 tx=0x7f23b8009040 comp rx=0 tx=0).stop 2023-10-14T18:19:28.835 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 msgr2=0x7f23c4156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 secure :-1 s=READY pgs=3094 cs=0 l=1 rev1=1 crypto rx=0x7f23bc009fd0 tx=0x7f23bc00f4a0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 shutdown_connections 2023-10-14T18:19:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f23a806b0e0 0x7f23a806d5a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f23c406c810 0x7f23c4156310 unknown :-1 s=CLOSED pgs=3094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.836 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f23c41f35a0 0x7f23c406c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 --2- 172.21.15.29:0/284453171 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f23b4003f70 0x7f23c41f1e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 >> 172.21.15.29:0/284453171 conn(0x7f23c40ce6a0 msgr2=0x7f23c40acd00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 shutdown_connections 2023-10-14T18:19:28.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:28.834+0000 7f23c9f99640 1 -- 172.21.15.29:0/284453171 wait complete. 2023-10-14T18:19:28.854 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:28.855 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:29.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.075+0000 7f412933c640 1 Processor -- start 2023-10-14T18:19:29.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.075+0000 7f412933c640 1 -- start start 2023-10-14T18:19:29.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f412933c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f41240cfc40 0x7f41240d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f412933c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240d0970 con 0x7f41240b2080 2023-10-14T18:19:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f412933c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41240d0ea0 con 0x7f41240b1ca0 2023-10-14T18:19:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f412933c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41240d0bc0 con 0x7f41240cfc40 2023-10-14T18:19:29.078 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f411b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f41240b2080 0x7f41240b24c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:48802/0 (socket says 172.21.15.29:48802) 2023-10-14T18:19:29.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f411b7fe640 1 -- 172.21.15.29:0/1595232192 learned_addr learned my addr 172.21.15.29:0/1595232192 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:29.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.077+0000 7f4128b3b640 1 --2- 172.21.15.29:0/1595232192 >> v2:172.21.15.186:3300/0 conn(0x7f41240cfc40 0x7f41240d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024893553 0 0) 0x7f41240d0970 con 0x7f41240b2080 2023-10-14T18:19:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4110003520 con 0x7f41240b2080 2023-10-14T18:19:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4080852760 0 0) 0x7f4110003520 con 0x7f41240b2080 2023-10-14T18:19:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 >> v2:172.21.15.186:3300/0 conn(0x7f41240cfc40 msgr2=0x7f41240d36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 --2- 172.21.15.29:0/1595232192 >> v2:172.21.15.186:3300/0 conn(0x7f41240cfc40 0x7f41240d36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 >> v1:172.21.15.186:6789/0 conn(0x7f41240b1ca0 legacy=0x7f41240ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240cd940 con 0x7f41240b2080 2023-10-14T18:19:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f410c0019f0 con 0x7f41240b2080 2023-10-14T18:19:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 >> v1:172.21.15.29:6789/0 conn(0x7f41240b2080 legacy=0x7f41240b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100039c0 0x7f4110003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 0x7f4110004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110005180 0x7f411000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41240d0bc0 con 0x7f41100047c0 2023-10-14T18:19:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f411000c060 con 0x7f4110005180 2023-10-14T18:19:29.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.078+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f411000c220 con 0x7f41100039c0 2023-10-14T18:19:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 0x7f4110004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f4128b3b640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110005180 0x7f411000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f4123fff640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100039c0 0x7f4110003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 -- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100039c0 msgr2=0x7f4110003de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:29.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41100039c0 0x7f4110003de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 -- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110005180 msgr2=0x7f411000b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4110005180 0x7f411000b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f4128b3b640 1 -- 172.21.15.29:0/1595232192 reap_dead start 2023-10-14T18:19:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.079+0000 7f411b7fe640 1 -- 172.21.15.29:0/1595232192 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4110003520 con 0x7f41100047c0 2023-10-14T18:19:29.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.080+0000 7f411b7fe640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 0x7f4110004c40 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f410c00a760 tx=0x7f41100057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a20abf66570f38e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c034630 con 0x7f41100047c0 2023-10-14T18:19:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f410c02f070 con 0x7f41100047c0 2023-10-14T18:19:29.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f41237fe640 1 -- 172.21.15.29:0/1595232192 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f410c0349f0 con 0x7f41100047c0 2023-10-14T18:19:29.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 -- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 msgr2=0x7f4110004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 0x7f4110004c40 secure :-1 s=READY pgs=3096 cs=0 l=1 rev1=1 crypto rx=0x7f410c00a760 tx=0x7f41100057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 -- 172.21.15.29:0/1595232192 shutdown_connections 2023-10-14T18:19:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 --2- 172.21.15.29:0/1595232192 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41100047c0 0x7f4110004c40 unknown :-1 s=CLOSED pgs=3096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 -- 172.21.15.29:0/1595232192 >> 172.21.15.29:0/1595232192 conn(0x7f41240d2b90 msgr2=0x7f41240d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:29.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 -- 172.21.15.29:0/1595232192 shutdown_connections 2023-10-14T18:19:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.081+0000 7f412933c640 1 -- 172.21.15.29:0/1595232192 wait complete. 2023-10-14T18:19:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.083+0000 7f412933c640 1 Processor -- start 2023-10-14T18:19:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.083+0000 7f412933c640 1 -- start start 2023-10-14T18:19:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41100047c0 0x7f41241f2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41240c68d0 0x7f412406b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41240cbea0 con 0x7f41240c6d50 2023-10-14T18:19:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41241552b0 con 0x7f41100047c0 2023-10-14T18:19:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.084+0000 7f412933c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4124155590 con 0x7f41240c68d0 2023-10-14T18:19:29.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47198/0 (socket says 172.21.15.29:47198) 2023-10-14T18:19:29.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 -- 172.21.15.29:0/2459369451 learned_addr learned my addr 172.21.15.29:0/2459369451 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:29.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 -- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41240c68d0 msgr2=0x7f412406b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41240c68d0 0x7f412406b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 -- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41100047c0 msgr2=0x7f41241f2300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4123fff640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41100047c0 0x7f41241f2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41100047c0 0x7f41241f2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 -- 172.21.15.29:0/2459369451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41240cbb30 con 0x7f41240c6d50 2023-10-14T18:19:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.085+0000 7f4128b3b640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f41240d0020 tx=0x7f411c011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.086+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c004440 con 0x7f41240c6d50 2023-10-14T18:19:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.086+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411c003610 con 0x7f41240c6d50 2023-10-14T18:19:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.086+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f411c0039c0 con 0x7f41240c6d50 2023-10-14T18:19:29.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.086+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41241f2840 con 0x7f41240c6d50 2023-10-14T18:19:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.086+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41240cc330 con 0x7f41240c6d50 2023-10-14T18:19:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.087+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41240cbcc0 con 0x7f41240c6d50 2023-10-14T18:19:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.088+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f411c003b20 con 0x7f41240c6d50 2023-10-14T18:19:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.089+0000 7f41217fa640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 0x7f410406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.089+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f411c089a20 con 0x7f41240c6d50 2023-10-14T18:19:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.090+0000 7f4123fff640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 0x7f410406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.090+0000 7f4123fff640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 0x7f410406d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4114007fd0 tx=0x7f4114009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.094+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f411c0545a0 con 0x7f41240c6d50 2023-10-14T18:19:29.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.291+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f41240d3a40 con 0x7f41240c6d50 2023-10-14T18:19:29.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.292+0000 7f41217fa640 1 -- 172.21.15.29:0/2459369451 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v262) v1 ==== 93+0+10 (secure 0 0 0) 0x7f411c057bc0 con 0x7f41240c6d50 2023-10-14T18:19:29.293 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:29.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 msgr2=0x7f410406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 0x7f410406d5a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4114007fd0 tx=0x7f4114009040 comp rx=0 tx=0).stop 2023-10-14T18:19:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 msgr2=0x7f412406b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 secure :-1 s=READY pgs=3097 cs=0 l=1 rev1=1 crypto rx=0x7f41240d0020 tx=0x7f411c011920 comp rx=0 tx=0).stop 2023-10-14T18:19:29.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 shutdown_connections 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f410406b0e0 0x7f410406d5a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.294+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f41240c6d50 0x7f412406b7e0 unknown :-1 s=CLOSED pgs=3097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.295+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f41240c68d0 0x7f412406b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.295+0000 7f412933c640 1 --2- 172.21.15.29:0/2459369451 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f41100047c0 0x7f41241f2300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.295+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 >> 172.21.15.29:0/2459369451 conn(0x7f41240d2b90 msgr2=0x7f412415c8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:29.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.295+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 shutdown_connections 2023-10-14T18:19:29.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.295+0000 7f412933c640 1 -- 172.21.15.29:0/2459369451 wait complete. 2023-10-14T18:19:29.316 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-14T18:19:29.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.551+0000 7f376ecff640 1 Processor -- start 2023-10-14T18:19:29.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.551+0000 7f376ecff640 1 -- start start 2023-10-14T18:19:29.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.553+0000 7f376ecff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.553+0000 7f376ecff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.553+0000 7f376ecff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f376806c9c0 con 0x7f376815aad0 2023-10-14T18:19:29.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.553+0000 7f376ecff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37680b4620 con 0x7f37680712c0 2023-10-14T18:19:29.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.553+0000 7f376ecff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37680b4340 con 0x7f3768070d70 2023-10-14T18:19:29.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376d4fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376e4fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376d4fc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:40262/0 (socket says 172.21.15.29:40262) 2023-10-14T18:19:29.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376e4fe640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:46694/0 (socket says 172.21.15.29:46694) 2023-10-14T18:19:29.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376d4fc640 1 -- 172.21.15.29:0/45296045 learned_addr learned my addr 172.21.15.29:0/45296045 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:29.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.554+0000 7f376dcfd640 1 --1- 172.21.15.29:0/45296045 >> v1:172.21.15.186:6789/0 conn(0x7f376815aad0 0x7f37680cc270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:55014/0 (socket says 172.21.15.29:55014) 2023-10-14T18:19:29.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.555+0000 7f376d4fc640 1 -- 172.21.15.29:0/45296045 >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 msgr2=0x7f376806c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:29.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.555+0000 7f376d4fc640 1 --2- 172.21.15.29:0/45296045 >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.555+0000 7f376d4fc640 1 -- 172.21.15.29:0/45296045 >> v1:172.21.15.186:6789/0 conn(0x7f376815aad0 legacy=0x7f37680cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.555+0000 7f376d4fc640 1 -- 172.21.15.29:0/45296045 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f376806f520 con 0x7f3768070d70 2023-10-14T18:19:29.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.555+0000 7f376e4fe640 1 --2- 172.21.15.29:0/45296045 >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:19:29.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.556+0000 7f376d4fc640 1 --2- 172.21.15.29:0/45296045 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3758009870 tx=0x7f375802ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c878979d28748881 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ccfb640 1 -- 172.21.15.29:0/45296045 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375802f830 con 0x7f3768070d70 2023-10-14T18:19:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ccfb640 1 -- 172.21.15.29:0/45296045 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f375802f990 con 0x7f3768070d70 2023-10-14T18:19:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ccfb640 1 -- 172.21.15.29:0/45296045 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f375802fd20 con 0x7f3768070d70 2023-10-14T18:19:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 -- 172.21.15.29:0/45296045 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 msgr2=0x7f376806bf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 --2- 172.21.15.29:0/45296045 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3758009870 tx=0x7f375802ba10 comp rx=0 tx=0).stop 2023-10-14T18:19:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 -- 172.21.15.29:0/45296045 shutdown_connections 2023-10-14T18:19:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 --2- 172.21.15.29:0/45296045 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3768070d70 0x7f376806bf40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 --2- 172.21.15.29:0/45296045 >> v2:172.21.15.186:3300/0 conn(0x7f37680712c0 0x7f376806c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 -- 172.21.15.29:0/45296045 >> 172.21.15.29:0/45296045 conn(0x7f37680b0a90 msgr2=0x7f37680ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:29.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.557+0000 7f376ecff640 1 -- 172.21.15.29:0/45296045 shutdown_connections 2023-10-14T18:19:29.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.558+0000 7f376ecff640 1 -- 172.21.15.29:0/45296045 wait complete. 2023-10-14T18:19:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.559+0000 7f376ecff640 1 Processor -- start 2023-10-14T18:19:29.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.560+0000 7f376ecff640 1 -- start start 2023-10-14T18:19:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37680712c0 0x7f3768153980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376815aad0 0x7f37681581c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376815bea0 con 0x7f3768070d70 2023-10-14T18:19:29.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f376815c390 con 0x7f376815aad0 2023-10-14T18:19:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376ecff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f376815c670 con 0x7f37680712c0 2023-10-14T18:19:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47210/0 (socket says 172.21.15.29:47210) 2023-10-14T18:19:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 -- 172.21.15.29:0/210262734 learned_addr learned my addr 172.21.15.29:0/210262734 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:29.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 -- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37680712c0 msgr2=0x7f3768153980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37680712c0 0x7f3768153980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 -- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376815aad0 msgr2=0x7f37681581c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376815aad0 0x7f37681581c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.561+0000 7f376dcfd640 1 -- 172.21.15.29:0/210262734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3760009400 con 0x7f3768070d70 2023-10-14T18:19:29.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f376dcfd640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f376400c7a0 tx=0x7f376400cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400a640 con 0x7f3768070d70 2023-10-14T18:19:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f376400a7a0 con 0x7f3768070d70 2023-10-14T18:19:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f376400ab10 con 0x7f3768070d70 2023-10-14T18:19:29.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37580094d0 con 0x7f3768070d70 2023-10-14T18:19:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.562+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f376815cc30 con 0x7f3768070d70 2023-10-14T18:19:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.563+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f376804e9b0 con 0x7f3768070d70 2023-10-14T18:19:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.564+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f376400ac70 con 0x7f3768070d70 2023-10-14T18:19:29.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.566+0000 7f375e7fc640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 0x7f374c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.566+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f3764089390 con 0x7f3768070d70 2023-10-14T18:19:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.566+0000 7f376d4fc640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 0x7f374c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:29.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.567+0000 7f376d4fc640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 0x7f374c06d570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758039040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:29.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.570+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3764053f30 con 0x7f3768070d70 2023-10-14T18:19:29.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:29.837+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7f37680cf520 con 0x7f3768070d70 2023-10-14T18:19:30.363 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.362+0000 7f375e7fc640 1 -- 172.21.15.29:0/210262734 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3294) v1 ==== 158+0+0 (secure 0 0 0) 0x7f37640170e0 con 0x7f3768070d70 2023-10-14T18:19:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 msgr2=0x7f374c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 0x7f374c06d570 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3758036000 tx=0x7f3758039040 comp rx=0 tx=0).stop 2023-10-14T18:19:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 msgr2=0x7f376815b8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.369 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 secure :-1 s=READY pgs=3098 cs=0 l=1 rev1=1 crypto rx=0x7f376400c7a0 tx=0x7f376400cc60 comp rx=0 tx=0).stop 2023-10-14T18:19:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 shutdown_connections 2023-10-14T18:19:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f374c06b0b0 0x7f374c06d570 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f376815aad0 0x7f37681581c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f37680712c0 0x7f3768153980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.370 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 --2- 172.21.15.29:0/210262734 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3768070d70 0x7f376815b8c0 unknown :-1 s=CLOSED pgs=3098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.367+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 >> 172.21.15.29:0/210262734 conn(0x7f37680b0a90 msgr2=0x7f37680ae160 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:30.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.368+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 shutdown_connections 2023-10-14T18:19:30.371 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.368+0000 7f376ecff640 1 -- 172.21.15.29:0/210262734 wait complete. 2023-10-14T18:19:30.393 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2023-10-14T18:19:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.627+0000 7f470e6a2640 1 Processor -- start 2023-10-14T18:19:30.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.627+0000 7f470e6a2640 1 -- start start 2023-10-14T18:19:30.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.628+0000 7f470e6a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f4708069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.628+0000 7f470e6a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47080caac0 0x7f470806a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.628+0000 7f470e6a2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47080cb030 con 0x7f47080c9ca0 2023-10-14T18:19:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.628+0000 7f470e6a2640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47080c91c0 con 0x7f47080caac0 2023-10-14T18:19:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.628+0000 7f470e6a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47080c94a0 con 0x7f470815ad70 2023-10-14T18:19:30.630 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47080caac0 0x7f470806a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:30.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f47080caac0 0x7f470806a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47226/0 (socket says 172.21.15.29:47226) 2023-10-14T18:19:30.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 -- 172.21.15.29:0/3158879338 learned_addr learned my addr 172.21.15.29:0/3158879338 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 -- 172.21.15.29:0/3158879338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 msgr2=0x7f4708069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 --2- 172.21.15.29:0/3158879338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f4708069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 -- 172.21.15.29:0/3158879338 >> v1:172.21.15.186:6789/0 conn(0x7f47080c9ca0 legacy=0x7f47080ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:30.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.629+0000 7f470cc3f640 1 -- 172.21.15.29:0/3158879338 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47080c9780 con 0x7f47080caac0 2023-10-14T18:19:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.630+0000 7f470cc3f640 1 --2- 172.21.15.29:0/3158879338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080caac0 0x7f470806a2e0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f4700009e50 tx=0x7f470002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4281fbfa3d850f15 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f4706ffd640 1 -- 172.21.15.29:0/3158879338 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f470002ca70 con 0x7f47080caac0 2023-10-14T18:19:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f4706ffd640 1 -- 172.21.15.29:0/3158879338 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f470002cbd0 con 0x7f47080caac0 2023-10-14T18:19:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f4706ffd640 1 -- 172.21.15.29:0/3158879338 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47000343f0 con 0x7f47080caac0 2023-10-14T18:19:30.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 -- 172.21.15.29:0/3158879338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080caac0 msgr2=0x7f470806a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 --2- 172.21.15.29:0/3158879338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080caac0 0x7f470806a2e0 secure :-1 s=READY pgs=3099 cs=0 l=1 rev1=1 crypto rx=0x7f4700009e50 tx=0x7f470002bed0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 -- 172.21.15.29:0/3158879338 shutdown_connections 2023-10-14T18:19:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 --2- 172.21.15.29:0/3158879338 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080caac0 0x7f470806a2e0 unknown :-1 s=CLOSED pgs=3099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.635 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 --2- 172.21.15.29:0/3158879338 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f4708069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.631+0000 7f470e6a2640 1 -- 172.21.15.29:0/3158879338 >> 172.21.15.29:0/3158879338 conn(0x7f47080ce9b0 msgr2=0x7f47080cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.632+0000 7f470e6a2640 1 -- 172.21.15.29:0/3158879338 shutdown_connections 2023-10-14T18:19:30.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.632+0000 7f470e6a2640 1 -- 172.21.15.29:0/3158879338 wait complete. 2023-10-14T18:19:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.633+0000 7f470e6a2640 1 Processor -- start 2023-10-14T18:19:30.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.633+0000 7f470e6a2640 1 -- start start 2023-10-14T18:19:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 0x7f470815d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f47080c7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4708153c80 con 0x7f47080c9ca0 2023-10-14T18:19:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f470815c4a0 con 0x7f47080caac0 2023-10-14T18:19:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.634+0000 7f470e6a2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f470814dd00 con 0x7f470815ad70 2023-10-14T18:19:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:47242/0 (socket says 172.21.15.29:47242) 2023-10-14T18:19:30.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 -- 172.21.15.29:0/4171061744 learned_addr learned my addr 172.21.15.29:0/4171061744 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f470cc3f640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f47080c7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f47077fe640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 0x7f470815d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 -- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 msgr2=0x7f47080c7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f47080c7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.641 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 -- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 msgr2=0x7f470815d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 0x7f470815d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f4707fff640 1 -- 172.21.15.29:0/4171061744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4700009b30 con 0x7f47080c9ca0 2023-10-14T18:19:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f470cc3f640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f47080c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:30.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.635+0000 7f47077fe640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 0x7f470815d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f4707fff640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f46f800b440 tx=0x7f46f800b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8011890 con 0x7f47080c9ca0 2023-10-14T18:19:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46f80119f0 con 0x7f47080c9ca0 2023-10-14T18:19:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46f8011da0 con 0x7f47080c9ca0 2023-10-14T18:19:30.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f470815c6f0 con 0x7f47080c9ca0 2023-10-14T18:19:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.636+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f470815ccb0 con 0x7f47080c9ca0 2023-10-14T18:19:30.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.637+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f470804e160 con 0x7f47080c9ca0 2023-10-14T18:19:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.638+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f46f801a400 con 0x7f47080c9ca0 2023-10-14T18:19:30.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.639+0000 7f4704ff9640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 0x7f46f406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.639+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f46f8089fa0 con 0x7f47080c9ca0 2023-10-14T18:19:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.640+0000 7f47077fe640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 0x7f46f406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.641+0000 7f47077fe640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 0x7f46f406d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f46fc007fd0 tx=0x7f46fc009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:30.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.644+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46f8054b20 con 0x7f47080c9ca0 2023-10-14T18:19:30.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.838+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f470806a830 con 0x7f47080c9ca0 2023-10-14T18:19:30.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.844+0000 7f4704ff9640 1 -- 172.21.15.29:0/4171061744 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v88)=0 v88) v1 ==== 122+0+0 (secure 0 0 0) 0x7f46f8015020 con 0x7f47080c9ca0 2023-10-14T18:19:30.847 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 msgr2=0x7f46f406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 0x7f46f406d570 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f46fc007fd0 tx=0x7f46fc009040 comp rx=0 tx=0).stop 2023-10-14T18:19:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 msgr2=0x7f470814d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:30.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 secure :-1 s=READY pgs=3100 cs=0 l=1 rev1=1 crypto rx=0x7f46f800b440 tx=0x7f46f800b900 comp rx=0 tx=0).stop 2023-10-14T18:19:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 shutdown_connections 2023-10-14T18:19:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f46f406b0b0 0x7f46f406d570 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.849 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f470815ad70 0x7f47080c7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f47080caac0 0x7f470815d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 --2- 172.21.15.29:0/4171061744 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f47080c9ca0 0x7f470814d7c0 unknown :-1 s=CLOSED pgs=3100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 >> 172.21.15.29:0/4171061744 conn(0x7f47080ce9b0 msgr2=0x7f47080cdd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:30.850 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 shutdown_connections 2023-10-14T18:19:30.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:30.847+0000 7f470e6a2640 1 -- 172.21.15.29:0/4171061744 wait complete. 2023-10-14T18:19:30.869 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:19:30.869 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:30.987 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:31.115 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:19:31.116 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:31.116 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:31.181 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:19:31.182 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:31.182 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:31.241 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:31.241 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:31.242 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:31.306 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:19:31.307 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:31.307 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:31.368 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:31.369 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2023-10-14T18:19:31.369 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:31.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.603+0000 7fab79b7c640 1 Processor -- start 2023-10-14T18:19:31.671 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.604+0000 7fab79b7c640 1 -- start start 2023-10-14T18:19:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab79b7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7415ae80 0x7fab7415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab79b7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7415ed60 con 0x7fab740ce390 2023-10-14T18:19:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab79b7c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab740707e0 con 0x7fab7415aaa0 2023-10-14T18:19:31.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab79b7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7406b810 con 0x7fab7415ae80 2023-10-14T18:19:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab73fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fab740ce390 0x7fab74070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35586/0 (socket says 172.21.15.29:35586) 2023-10-14T18:19:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.605+0000 7fab73fff640 1 -- 172.21.15.29:0/690170801 learned_addr learned my addr 172.21.15.29:0/690170801 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab6affd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7415ae80 0x7fab7415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2568693386 0 0) 0x7fab7415ed60 con 0x7fab740ce390 2023-10-14T18:19:31.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab60003520 con 0x7fab740ce390 2023-10-14T18:19:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3634218428 0 0) 0x7fab60003520 con 0x7fab740ce390 2023-10-14T18:19:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7415ae80 msgr2=0x7fab7415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7415ae80 0x7fab7415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 >> v1:172.21.15.186:6789/0 conn(0x7fab7415aaa0 legacy=0x7fab740cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7415ef90 con 0x7fab740ce390 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fab6c001670 con 0x7fab740ce390 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 >> v1:172.21.15.29:6789/0 conn(0x7fab740ce390 legacy=0x7fab74070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab600039a0 0x7fab60003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab600047c0 0x7fab60004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab6000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7406b810 con 0x7fab60005180 2023-10-14T18:19:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000c050 con 0x7fab600047c0 2023-10-14T18:19:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab6000c210 con 0x7fab600039a0 2023-10-14T18:19:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.606+0000 7fab6affd640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab7415ae80 0x7fab7415e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:31.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab6000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab737fe640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab600039a0 0x7fab60003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 -- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab600039a0 msgr2=0x7fab60003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab600039a0 0x7fab60003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 -- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab600047c0 msgr2=0x7fab60004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:31.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab600047c0 0x7fab60004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab73fff640 1 -- 172.21.15.29:0/690170801 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab60003520 con 0x7fab60005180 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.607+0000 7fab737fe640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab600039a0 0x7fab60003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.608+0000 7fab73fff640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab6000b980 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c240 tx=0x7fab600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5735e26fcc5306 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.608+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c005ad0 con 0x7fab60005180 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.608+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c005c30 con 0x7fab60005180 2023-10-14T18:19:31.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.608+0000 7fab72ffd640 1 -- 172.21.15.29:0/690170801 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c02c8b0 con 0x7fab60005180 2023-10-14T18:19:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 -- 172.21.15.29:0/690170801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 msgr2=0x7fab6000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 --2- 172.21.15.29:0/690170801 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab6000b980 secure :-1 s=READY pgs=3102 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00c240 tx=0x7fab600057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab73fff640 1 -- 172.21.15.29:0/690170801 reap_dead start 2023-10-14T18:19:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 -- 172.21.15.29:0/690170801 shutdown_connections 2023-10-14T18:19:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 -- 172.21.15.29:0/690170801 >> 172.21.15.29:0/690170801 conn(0x7fab740cb410 msgr2=0x7fab740cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 -- 172.21.15.29:0/690170801 shutdown_connections 2023-10-14T18:19:31.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.609+0000 7fab79b7c640 1 -- 172.21.15.29:0/690170801 wait complete. 2023-10-14T18:19:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.611+0000 7fab79b7c640 1 Processor -- start 2023-10-14T18:19:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.611+0000 7fab79b7c640 1 -- start start 2023-10-14T18:19:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 0x7fab7415bec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 0x7fab7415dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab740af3d0 con 0x7fab60005180 2023-10-14T18:19:31.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7415c400 con 0x7fab7415d810 2023-10-14T18:19:31.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab79b7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab740c31b0 con 0x7fab74155fa0 2023-10-14T18:19:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.612+0000 7fab737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:50986/0 (socket says 172.21.15.29:50986) 2023-10-14T18:19:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 -- 172.21.15.29:0/4086996264 learned_addr learned my addr 172.21.15.29:0/4086996264 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:31.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab73fff640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 0x7fab7415dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab6affd640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 0x7fab7415bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 -- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 msgr2=0x7fab7415bec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 0x7fab7415bec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 -- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 msgr2=0x7fab7415dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 0x7fab7415dc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 -- 172.21.15.29:0/4086996264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab74156420 con 0x7fab60005180 2023-10-14T18:19:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab6affd640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 0x7fab7415bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:31.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab73fff640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 0x7fab7415dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.613+0000 7fab737fe640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fab6400db10 tx=0x7fab640020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.614+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab64003da0 con 0x7fab60005180 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.614+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab64006b50 con 0x7fab60005180 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.614+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6400b740 con 0x7fab60005180 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.614+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab740af5e0 con 0x7fab60005180 2023-10-14T18:19:31.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.614+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab740afba0 con 0x7fab60005180 2023-10-14T18:19:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.615+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7404e160 con 0x7fab60005180 2023-10-14T18:19:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.616+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fab6400b8a0 con 0x7fab60005180 2023-10-14T18:19:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.617+0000 7fab70ff9640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 0x7fab5406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.617+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7fab640892a0 con 0x7fab60005180 2023-10-14T18:19:31.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.617+0000 7fab6affd640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 0x7fab5406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.618+0000 7fab6affd640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 0x7fab5406d5a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab740c30a0 tx=0x7fab5c008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:31.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.621+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab64053dc0 con 0x7fab60005180 2023-10-14T18:19:31.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.857+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab740cdc60 con 0x7fab60005180 2023-10-14T18:19:31.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.858+0000 7fab70ff9640 1 -- 172.21.15.29:0/4086996264 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+3710 (secure 0 0 0) 0x7fab64018e90 con 0x7fab60005180 2023-10-14T18:19:31.859 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:31.860 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":133,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/3661121148","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":3661121148},{"type":"v1","addr":"172.21.15.29:6836","nonce":3661121148}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":7631,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3578648145","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3578648145},{"type":"v1","addr":"172.21.15.29:6837","nonce":3578648145}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":118},{"gid":9094,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6833/1485060674","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1485060674},{"type":"v1","addr":"172.21.15.186:6833","nonce":1485060674}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":131},{"gid":9169,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/3524455768","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6834","nonce":3524455768},{"type":"v1","addr":"172.21.15.186:6835","nonce":3524455768}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":133}],"filesystems":[]} 2023-10-14T18:19:31.860 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 133 2023-10-14T18:19:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 msgr2=0x7fab5406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 0x7fab5406d5a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fab740c30a0 tx=0x7fab5c008040 comp rx=0 tx=0).stop 2023-10-14T18:19:31.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 msgr2=0x7fab740aee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 secure :-1 s=READY pgs=3103 cs=0 l=1 rev1=1 crypto rx=0x7fab6400db10 tx=0x7fab640020c0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 shutdown_connections 2023-10-14T18:19:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fab5406b0e0 0x7fab5406d5a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab7415d810 0x7fab7415dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab74155fa0 0x7fab7415bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 --2- 172.21.15.29:0/4086996264 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab60005180 0x7fab740aee90 unknown :-1 s=CLOSED pgs=3103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.861+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 >> 172.21.15.29:0/4086996264 conn(0x7fab740cb410 msgr2=0x7fab740c1d00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:31.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.862+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 shutdown_connections 2023-10-14T18:19:31.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:31.862+0000 7fab79b7c640 1 -- 172.21.15.29:0/4086996264 wait complete. 2023-10-14T18:19:31.882 INFO:tasks.ceph.mds.a:Restarting daemon 2023-10-14T18:19:31.883 INFO:tasks.ceph.mds.a:Stopping old one... 2023-10-14T18:19:31.883 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T18:19:31.883 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:19:31.883 INFO:tasks.ceph.mds.c:Restarting daemon 2023-10-14T18:19:31.884 INFO:tasks.ceph.mds.c:Stopping old one... 2023-10-14T18:19:31.884 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T18:19:31.884 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:19:31.885 INFO:tasks.ceph.mds.b:Restarting daemon 2023-10-14T18:19:31.885 INFO:tasks.ceph.mds.b:Stopping old one... 2023-10-14T18:19:31.885 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T18:19:31.885 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:19:31.886 INFO:tasks.ceph.mds.d:Restarting daemon 2023-10-14T18:19:31.886 INFO:tasks.ceph.mds.d:Stopping old one... 2023-10-14T18:19:31.886 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T18:19:31.887 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:19:31.902 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T18:19:31.903 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2023-10-14T18:19:31.906 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T18:19:31.906 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2023-10-14T18:19:31.912 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T18:19:31.912 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2023-10-14T18:19:31.913 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T18:19:31.913 DEBUG:teuthology.orchestra.run.smithi186:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2023-10-14T18:19:31.954 INFO:tasks.ceph.mds.c:Started 2023-10-14T18:19:31.954 DEBUG:teuthology.parallel:result is None 2023-10-14T18:19:31.961 INFO:tasks.ceph.mds.b:Started 2023-10-14T18:19:31.961 DEBUG:teuthology.parallel:result is None 2023-10-14T18:19:31.962 INFO:tasks.ceph.mds.a:Started 2023-10-14T18:19:31.962 DEBUG:teuthology.parallel:result is None 2023-10-14T18:19:31.967 INFO:tasks.ceph.mds.d:Started 2023-10-14T18:19:31.968 DEBUG:teuthology.parallel:result is None 2023-10-14T18:19:31.968 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2023-10-14T18:19:32.118 INFO:tasks.ceph.mds.c.smithi029.stdout:starting mds.c at 2023-10-14T18:19:32.119 INFO:tasks.ceph.mds.a.smithi029.stdout:starting mds.a at 2023-10-14T18:19:32.124 INFO:tasks.ceph.mds.b.smithi186.stdout:starting mds.b at 2023-10-14T18:19:32.126 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:19:32.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.161+0000 7f926f4e6640 1 Processor -- start 2023-10-14T18:19:32.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.161+0000 7f926f4e6640 1 -- start start 2023-10-14T18:19:32.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.162+0000 7f926f4e6640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.162+0000 7f926f4e6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.162+0000 7f926f4e6640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f926815d150 con 0x7f926815cd70 2023-10-14T18:19:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.162+0000 7f926f4e6640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92680d1180 con 0x7f92680b8480 2023-10-14T18:19:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.162+0000 7f926f4e6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680d1460 con 0x7f92680d0be0 2023-10-14T18:19:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51012/0 (socket says 172.21.15.29:51012) 2023-10-14T18:19:32.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 -- 172.21.15.29:0/1259557550 learned_addr learned my addr 172.21.15.29:0/1259557550 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926ece5640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 -- 172.21.15.29:0/1259557550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92680d0be0 msgr2=0x7f92680cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 -- 172.21.15.29:0/1259557550 >> v1:172.21.15.186:6789/0 conn(0x7f926815cd70 legacy=0x7f92680d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:32.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.163+0000 7f926dce3640 1 -- 172.21.15.29:0/1259557550 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92680d1740 con 0x7f92680b8480 2023-10-14T18:19:32.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.164+0000 7f926dce3640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f92680cd7c0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f9258009870 tx=0x7f925802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf272c0f6ce20ff5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:32.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.164+0000 7f926d4e2640 1 -- 172.21.15.29:0/1259557550 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258030840 con 0x7f92680b8480 2023-10-14T18:19:32.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926d4e2640 1 -- 172.21.15.29:0/1259557550 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f92580309a0 con 0x7f92680b8480 2023-10-14T18:19:32.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926d4e2640 1 -- 172.21.15.29:0/1259557550 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9258030d30 con 0x7f92680b8480 2023-10-14T18:19:32.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 -- 172.21.15.29:0/1259557550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 msgr2=0x7f92680cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:32.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f92680cd7c0 secure :-1 s=READY pgs=3110 cs=0 l=1 rev1=1 crypto rx=0x7f9258009870 tx=0x7f925802b980 comp rx=0 tx=0).stop 2023-10-14T18:19:32.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 -- 172.21.15.29:0/1259557550 shutdown_connections 2023-10-14T18:19:32.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f92680d0be0 0x7f92680cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:32.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1259557550 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f92680cd7c0 unknown :-1 s=CLOSED pgs=3110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:32.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 -- 172.21.15.29:0/1259557550 >> 172.21.15.29:0/1259557550 conn(0x7f92680cf570 msgr2=0x7f92680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:32.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 -- 172.21.15.29:0/1259557550 shutdown_connections 2023-10-14T18:19:32.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.165+0000 7f926f4e6640 1 -- 172.21.15.29:0/1259557550 wait complete. 2023-10-14T18:19:32.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.167+0000 7f926f4e6640 1 Processor -- start 2023-10-14T18:19:32.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.167+0000 7f926f4e6640 1 -- start start 2023-10-14T18:19:32.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92680d0be0 0x7f9268157720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f926815cd70 0x7f92680b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680b32a0 con 0x7f92680b8480 2023-10-14T18:19:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92680b34f0 con 0x7f92680d0be0 2023-10-14T18:19:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926f4e6640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f92680b37d0 con 0x7f926815cd70 2023-10-14T18:19:32.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.168+0000 7f926e4e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51026/0 (socket says 172.21.15.29:51026) 2023-10-14T18:19:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 -- 172.21.15.29:0/1841047917 learned_addr learned my addr 172.21.15.29:0/1841047917 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926dce3640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92680d0be0 0x7f9268157720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926ece5640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f926815cd70 0x7f92680b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 -- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f926815cd70 msgr2=0x7f92680b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f926815cd70 0x7f92680b4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 -- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92680d0be0 msgr2=0x7f9268157720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92680d0be0 0x7f9268157720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 -- 172.21.15.29:0/1841047917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92580094d0 con 0x7f92680b8480 2023-10-14T18:19:32.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.169+0000 7f926e4e4640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7f926400c400 tx=0x7f926400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.170+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92640052d0 con 0x7f92680b8480 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.170+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9264005430 con 0x7f92680b8480 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.170+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92680b3ab0 con 0x7f92680b8480 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.170+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9264003830 con 0x7f92680b8480 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.170+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92680b4070 con 0x7f92680b8480 2023-10-14T18:19:32.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.171+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92680b4260 con 0x7f92680b8480 2023-10-14T18:19:32.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.172+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9264003990 con 0x7f92680b8480 2023-10-14T18:19:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.173+0000 7f925effd640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 0x7f924c06d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.173+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(262..262 src has 1..262) v4 ==== 6005+0+0 (secure 0 0 0) 0x7f9264088fe0 con 0x7f92680b8480 2023-10-14T18:19:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.173+0000 7f926dce3640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 0x7f924c06d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.174+0000 7f926dce3640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 0x7f924c06d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9258036000 tx=0x7f9258008030 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:32.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.178+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9264053d40 con 0x7f92680b8480 2023-10-14T18:19:32.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:32.391+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) v1 -- 0x7f926804e9b0 con 0x7f92680b8480 2023-10-14T18:19:33.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.314+0000 7f925effd640 1 -- 172.21.15.29:0/1841047917 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v267) v1 ==== 97+0+0 (secure 0 0 0) 0x7f9264017080 con 0x7f92680b8480 2023-10-14T18:19:33.315 INFO:teuthology.orchestra.run.smithi029.stderr: removed all blocklist entries 2023-10-14T18:19:33.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 msgr2=0x7f924c06d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 0x7f924c06d570 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f9258036000 tx=0x7f9258008030 comp rx=0 tx=0).stop 2023-10-14T18:19:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 msgr2=0x7f926814cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 secure :-1 s=READY pgs=3111 cs=0 l=1 rev1=1 crypto rx=0x7f926400c400 tx=0x7f926400c8c0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 shutdown_connections 2023-10-14T18:19:33.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f924c06b0b0 0x7f924c06d570 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f926815cd70 0x7f92680b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f92680d0be0 0x7f9268157720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 --2- 172.21.15.29:0/1841047917 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f92680b8480 0x7f926814cea0 unknown :-1 s=CLOSED pgs=3111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.317+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 >> 172.21.15.29:0/1841047917 conn(0x7f92680cf570 msgr2=0x7f92680cd430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:33.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.318+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 shutdown_connections 2023-10-14T18:19:33.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.318+0000 7f926f4e6640 1 -- 172.21.15.29:0/1841047917 wait complete. 2023-10-14T18:19:33.341 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2023-10-14T18:19:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.573+0000 7fb4a7fff640 1 Processor -- start 2023-10-14T18:19:33.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.573+0000 7fb4a7fff640 1 -- start start 2023-10-14T18:19:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fb4a8159950 0x7fb4a815a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4a806c100 0x7fb4a806c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a815e490 con 0x7fb4a806c100 2023-10-14T18:19:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a8070df0 con 0x7fb4a8159950 2023-10-14T18:19:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a815aef0 con 0x7fb4a8070450 2023-10-14T18:19:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4a806c100 0x7fb4a806c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb4a806c100 0x7fb4a806c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51038/0 (socket says 172.21.15.29:51038) 2023-10-14T18:19:33.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.575+0000 7fb4a77fe640 1 -- 172.21.15.29:0/2024724763 learned_addr learned my addr 172.21.15.29:0/2024724763 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:33.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a67fc640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a6ffd640 1 --2- 172.21.15.29:0/2024724763 >> v2:172.21.15.186:3300/0 conn(0x7fb4a8159950 0x7fb4a815a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a77fe640 1 -- 172.21.15.29:0/2024724763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 msgr2=0x7fb4a80708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a77fe640 1 -- 172.21.15.29:0/2024724763 >> v2:172.21.15.186:3300/0 conn(0x7fb4a8159950 msgr2=0x7fb4a815a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:33.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/2024724763 >> v2:172.21.15.186:3300/0 conn(0x7fb4a8159950 0x7fb4a815a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.576+0000 7fb4a77fe640 1 -- 172.21.15.29:0/2024724763 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a815d910 con 0x7fb4a806c100 2023-10-14T18:19:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.577+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a806c100 0x7fb4a806c570 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80ca950 tx=0x7fb4a002be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fecd8abf556df15b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.577+0000 7fb4a5ffb640 1 -- 172.21.15.29:0/2024724763 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a002c9e0 con 0x7fb4a806c100 2023-10-14T18:19:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a5ffb640 1 -- 172.21.15.29:0/2024724763 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a002cb40 con 0x7fb4a806c100 2023-10-14T18:19:33.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a5ffb640 1 -- 172.21.15.29:0/2024724763 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a00343f0 con 0x7fb4a806c100 2023-10-14T18:19:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 -- 172.21.15.29:0/2024724763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a806c100 msgr2=0x7fb4a806c570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a806c100 0x7fb4a806c570 secure :-1 s=READY pgs=3112 cs=0 l=1 rev1=1 crypto rx=0x7fb4a80ca950 tx=0x7fb4a002be40 comp rx=0 tx=0).stop 2023-10-14T18:19:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 -- 172.21.15.29:0/2024724763 shutdown_connections 2023-10-14T18:19:33.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a806c100 0x7fb4a806c570 unknown :-1 s=CLOSED pgs=3112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/2024724763 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/2024724763 >> v2:172.21.15.186:3300/0 conn(0x7fb4a8159950 0x7fb4a815a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 -- 172.21.15.29:0/2024724763 >> 172.21.15.29:0/2024724763 conn(0x7fb4a80ce6a0 msgr2=0x7fb4a80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.578+0000 7fb4a7fff640 1 -- 172.21.15.29:0/2024724763 shutdown_connections 2023-10-14T18:19:33.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.579+0000 7fb4a7fff640 1 -- 172.21.15.29:0/2024724763 wait complete. 2023-10-14T18:19:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.580+0000 7fb4a7fff640 1 Processor -- start 2023-10-14T18:19:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.580+0000 7fb4a7fff640 1 -- start start 2023-10-14T18:19:33.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 0x7fb4a815c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80c7430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a815c190 con 0x7fb4a8159950 2023-10-14T18:19:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a815d350 con 0x7fb4a806c100 2023-10-14T18:19:33.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.581+0000 7fb4a7fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a815d630 con 0x7fb4a8070450 2023-10-14T18:19:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51054/0 (socket says 172.21.15.29:51054) 2023-10-14T18:19:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 -- 172.21.15.29:0/1317781950 learned_addr learned my addr 172.21.15.29:0/1317781950 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:33.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a67fc640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80c7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a6ffd640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 0x7fb4a815c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 -- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 msgr2=0x7fb4a80c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80c7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 -- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 msgr2=0x7fb4a815c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 0x7fb4a815c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a77fe640 1 -- 172.21.15.29:0/1317781950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a0009b30 con 0x7fb4a8159950 2023-10-14T18:19:33.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.582+0000 7fb4a6ffd640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 0x7fb4a815c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:33.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb4a77fe640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009e50 tx=0x7fb4a0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:33.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a0009510 con 0x7fb4a8159950 2023-10-14T18:19:33.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a003e450 con 0x7fb4a8159950 2023-10-14T18:19:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4a003e800 con 0x7fb4a8159950 2023-10-14T18:19:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a815bb40 con 0x7fb4a8159950 2023-10-14T18:19:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.583+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a815ce30 con 0x7fb4a8159950 2023-10-14T18:19:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.584+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a815c340 con 0x7fb4a8159950 2023-10-14T18:19:33.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.585+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fb4a003e960 con 0x7fb4a8159950 2023-10-14T18:19:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.586+0000 7fb48f7fe640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 0x7fb49406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.586+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 4973+0+0 (secure 0 0 0) 0x7fb4a00ac5a0 con 0x7fb4a8159950 2023-10-14T18:19:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.586+0000 7fb4a6ffd640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 0x7fb49406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:33.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.587+0000 7fb4a6ffd640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 0x7fb49406d570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb498006fd0 tx=0x7fb498008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:33.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.590+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb4a0077530 con 0x7fb4a8159950 2023-10-14T18:19:33.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.853+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fb4a804e9b0 con 0x7fb4a8159950 2023-10-14T18:19:33.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.855+0000 7fb48f7fe640 1 -- 172.21.15.29:0/1317781950 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v53) v1 ==== 80+0+5304 (secure 0 0 0) 0x7fb4a007ab50 con 0x7fb4a8159950 2023-10-14T18:19:33.856 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:33.856 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:19:33.857 INFO:teuthology.orchestra.run.smithi029.stdout: "auth_dump": [ 2023-10-14T18:19:33.857 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.857 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.a", 2023-10-14T18:19:33.857 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplDpdYMRAAlCJHP9Y78K1K5t+pGtpMqA==", 2023-10-14T18:19:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.858 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:19:33.859 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:19:33.859 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.859 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.859 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.860 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.860 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.b", 2023-10-14T18:19:33.860 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplGdx+GBAA1ELq98tBfoZbqAEpPcz9Tw==", 2023-10-14T18:19:33.860 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.860 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.861 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:19:33.861 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:19:33.861 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.861 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.862 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.862 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.862 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.c", 2023-10-14T18:19:33.862 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl7jAjBxAAylwLm/BPvrB8ncswg/NOHQ==", 2023-10-14T18:19:33.863 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.863 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.863 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:19:33.863 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:19:33.864 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.864 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.864 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.864 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mds.d", 2023-10-14T18:19:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCpl9SX+KRAA7PZOe9LOHh78d7U7wfBwAw==", 2023-10-14T18:19:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.865 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:19:33.866 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:19:33.866 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.866 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.866 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.866 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.867 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.0", 2023-10-14T18:19:33.867 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA2xCplJTnsIhAAr+Or2Kw/hnHSRmRWWRlglw==", 2023-10-14T18:19:33.867 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.867 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.868 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.868 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.868 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.868 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.868 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.869 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.1", 2023-10-14T18:19:33.869 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA6xCplCP8HARAAzndyP+9/LWMewzxEjvUiWg==", 2023-10-14T18:19:33.869 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.869 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.869 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.870 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.870 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.870 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.870 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.871 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.2", 2023-10-14T18:19:33.871 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA9xCplpN9qDBAA3zW8aEMa4+S1eMvpzWuhnQ==", 2023-10-14T18:19:33.871 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.871 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.872 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.872 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.872 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.872 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.872 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.3", 2023-10-14T18:19:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBAxCploAmnFhAAbrClfHOI3WHxqWCJWlcxeA==", 2023-10-14T18:19:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.873 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.874 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.874 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.874 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.874 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.4", 2023-10-14T18:19:33.875 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBHxCplO0YqIhAAM0GXOcKx0pHAa+ntYWtg+w==", 2023-10-14T18:19:33.875 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.875 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.875 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.876 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.876 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.876 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.876 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.5", 2023-10-14T18:19:33.876 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBKxCpl3ApQOxAAmtewkupkKW9GTjxq4pFlDg==", 2023-10-14T18:19:33.877 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.877 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.877 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.878 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.878 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.878 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.878 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.6", 2023-10-14T18:19:33.878 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBOxCplOdwMGBAAMm1Qfldb3vMWDYkeCpavzg==", 2023-10-14T18:19:33.879 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.879 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.879 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.880 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.880 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.880 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.880 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "osd.7", 2023-10-14T18:19:33.880 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQBRxCplNkPUIRAAC6fPw2FWu0nxzya29VwVZQ==", 2023-10-14T18:19:33.881 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow profile osd", 2023-10-14T18:19:33.881 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile osd", 2023-10-14T18:19:33.881 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.882 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.882 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.882 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.882 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.0", 2023-10-14T18:19:33.882 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA==", 2023-10-14T18:19:33.883 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.883 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow r", 2023-10-14T18:19:33.884 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rw tag cephfs data=cephfs" 2023-10-14T18:19:33.884 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.884 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.884 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.884 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.1", 2023-10-14T18:19:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplXMx/FBAA/GV2E1qj6YJ+56H02wLp8A==", 2023-10-14T18:19:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:19:33.885 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:19:33.886 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:19:33.886 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.886 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.886 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.887 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.2", 2023-10-14T18:19:33.887 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCply6hWCRAApk4fjJv7PkUx0JAdz1qPFg==", 2023-10-14T18:19:33.887 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.887 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.888 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:19:33.888 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:19:33.888 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:19:33.888 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.888 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.889 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.889 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.3", 2023-10-14T18:19:33.889 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQA0xCplFUeMHxAAow2MRKARfGVSsF4yHyou9w==", 2023-10-14T18:19:33.889 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.889 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow", 2023-10-14T18:19:33.890 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow r", 2023-10-14T18:19:33.890 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow rw", 2023-10-14T18:19:33.890 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow rwx" 2023-10-14T18:19:33.890 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "client.admin", 2023-10-14T18:19:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAxxCplM2nFIBAAP8+mZXeqr8XWsMWO3VGzUw==", 2023-10-14T18:19:33.891 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:19:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mgr": "allow *", 2023-10-14T18:19:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow *", 2023-10-14T18:19:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.892 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.x", 2023-10-14T18:19:33.893 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplJ0v1IxAAKb9Y68xpdnr60/m25pqI4g==", 2023-10-14T18:19:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:19:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:19:33.894 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.895 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.895 INFO:teuthology.orchestra.run.smithi029.stdout: }, 2023-10-14T18:19:33.895 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:33.895 INFO:teuthology.orchestra.run.smithi029.stdout: "entity": "mgr.y", 2023-10-14T18:19:33.895 INFO:teuthology.orchestra.run.smithi029.stdout: "key": "AQAyxCplO/m8FhAAs1FYQu6N+QNxotbiXwVg+Q==", 2023-10-14T18:19:33.896 INFO:teuthology.orchestra.run.smithi029.stdout: "caps": { 2023-10-14T18:19:33.896 INFO:teuthology.orchestra.run.smithi029.stdout: "mds": "allow *", 2023-10-14T18:19:33.896 INFO:teuthology.orchestra.run.smithi029.stdout: "mon": "allow profile mgr", 2023-10-14T18:19:33.896 INFO:teuthology.orchestra.run.smithi029.stdout: "osd": "allow *" 2023-10-14T18:19:33.897 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.897 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:33.897 INFO:teuthology.orchestra.run.smithi029.stdout: ] 2023-10-14T18:19:33.897 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:19:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 msgr2=0x7fb49406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 0x7fb49406d570 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb498006fd0 tx=0x7fb498008040 comp rx=0 tx=0).stop 2023-10-14T18:19:33.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 msgr2=0x7fb4a80c7970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 secure :-1 s=READY pgs=3113 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009e50 tx=0x7fb4a0004a40 comp rx=0 tx=0).stop 2023-10-14T18:19:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 shutdown_connections 2023-10-14T18:19:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fb49406b0b0 0x7fb49406d570 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb4a8159950 0x7fb4a80c7970 unknown :-1 s=CLOSED pgs=3113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb4a8070450 0x7fb4a80c7430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 --2- 172.21.15.29:0/1317781950 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb4a806c100 0x7fb4a815c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 >> 172.21.15.29:0/1317781950 conn(0x7fb4a80ce6a0 msgr2=0x7fb4a80ccb00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 shutdown_connections 2023-10-14T18:19:33.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:33.858+0000 7fb4a7fff640 1 -- 172.21.15.29:0/1317781950 wait complete. 2023-10-14T18:19:33.902 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.130+0000 7feec759e640 1 Processor -- start 2023-10-14T18:19:34.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.130+0000 7feec759e640 1 -- start start 2023-10-14T18:19:34.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7feec815dde0 0x7feec80709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec8070f20 0x7feec80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec759e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec8160a90 con 0x7feec815c9f0 2023-10-14T18:19:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec8071350 con 0x7feec8070f20 2023-10-14T18:19:34.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feec80cdec0 con 0x7feec815dde0 2023-10-14T18:19:34.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec8070f20 0x7feec80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec6d9d640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7feec8070f20 0x7feec80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51064/0 (socket says 172.21.15.29:51064) 2023-10-14T18:19:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feec6d9d640 1 -- 172.21.15.29:0/2402323661 learned_addr learned my addr 172.21.15.29:0/2402323661 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.132+0000 7feebffff640 1 --2- 172.21.15.29:0/2402323661 >> v2:172.21.15.186:3300/0 conn(0x7feec815dde0 0x7feec80709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.133+0000 7feec6d9d640 1 -- 172.21.15.29:0/2402323661 >> v2:172.21.15.186:3300/0 conn(0x7feec815dde0 msgr2=0x7feec80709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.133+0000 7feec6d9d640 1 --2- 172.21.15.29:0/2402323661 >> v2:172.21.15.186:3300/0 conn(0x7feec815dde0 0x7feec80709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.133+0000 7feec6d9d640 1 -- 172.21.15.29:0/2402323661 >> v1:172.21.15.186:6789/0 conn(0x7feec815c9f0 legacy=0x7feec80c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.133+0000 7feec6d9d640 1 -- 172.21.15.29:0/2402323661 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80cd900 con 0x7feec8070f20 2023-10-14T18:19:34.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.134+0000 7feec6d9d640 1 --2- 172.21.15.29:0/2402323661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec8070f20 0x7feec80ac890 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7feec80b1260 tx=0x7feec002bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af30fa66344d5460 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.134+0000 7feec5d9b640 1 -- 172.21.15.29:0/2402323661 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec002cb10 con 0x7feec8070f20 2023-10-14T18:19:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.134+0000 7feec5d9b640 1 -- 172.21.15.29:0/2402323661 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec002cc70 con 0x7feec8070f20 2023-10-14T18:19:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.134+0000 7feec5d9b640 1 -- 172.21.15.29:0/2402323661 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec00343f0 con 0x7feec8070f20 2023-10-14T18:19:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 -- 172.21.15.29:0/2402323661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec8070f20 msgr2=0x7feec80ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 --2- 172.21.15.29:0/2402323661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec8070f20 0x7feec80ac890 secure :-1 s=READY pgs=3114 cs=0 l=1 rev1=1 crypto rx=0x7feec80b1260 tx=0x7feec002bf70 comp rx=0 tx=0).stop 2023-10-14T18:19:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 -- 172.21.15.29:0/2402323661 shutdown_connections 2023-10-14T18:19:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 --2- 172.21.15.29:0/2402323661 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec8070f20 0x7feec80ac890 unknown :-1 s=CLOSED pgs=3114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 --2- 172.21.15.29:0/2402323661 >> v2:172.21.15.186:3300/0 conn(0x7feec815dde0 0x7feec80709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 -- 172.21.15.29:0/2402323661 >> 172.21.15.29:0/2402323661 conn(0x7feec80ce6a0 msgr2=0x7feec80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:34.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 -- 172.21.15.29:0/2402323661 shutdown_connections 2023-10-14T18:19:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.135+0000 7feec759e640 1 -- 172.21.15.29:0/2402323661 wait complete. 2023-10-14T18:19:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.137+0000 7feec759e640 1 Processor -- start 2023-10-14T18:19:34.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.137+0000 7feec759e640 1 -- start start 2023-10-14T18:19:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feec8070f20 0x7feec806c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 0x7feec806a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec8150e00 con 0x7feec815dde0 2023-10-14T18:19:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feec806cc90 con 0x7feec815c9f0 2023-10-14T18:19:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feec81f6510 con 0x7feec8070f20 2023-10-14T18:19:34.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.138+0000 7feec6d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51074/0 (socket says 172.21.15.29:51074) 2023-10-14T18:19:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 -- 172.21.15.29:0/3478581662 learned_addr learned my addr 172.21.15.29:0/3478581662 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec659c640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feec8070f20 0x7feec806c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feebffff640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 0x7feec806a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 -- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feec8070f20 msgr2=0x7feec806c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feec8070f20 0x7feec806c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 -- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 msgr2=0x7feec806a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 0x7feec806a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 -- 172.21.15.29:0/3478581662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec0009b30 con 0x7feec815dde0 2023-10-14T18:19:34.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feebffff640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 0x7feec806a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.139+0000 7feec6d9d640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7feec0009f40 tx=0x7feec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.140+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec0009510 con 0x7feec815dde0 2023-10-14T18:19:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.140+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feec003e450 con 0x7feec815dde0 2023-10-14T18:19:34.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.140+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feec003e800 con 0x7feec815dde0 2023-10-14T18:19:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.140+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec81f5750 con 0x7feec815dde0 2023-10-14T18:19:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.140+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feec81f5d10 con 0x7feec815dde0 2023-10-14T18:19:34.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.141+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec804e9b0 con 0x7feec815dde0 2023-10-14T18:19:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.142+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7feec003e960 con 0x7feec815dde0 2023-10-14T18:19:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.143+0000 7feebeffd640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 0x7feea806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.143+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 4973+0+0 (secure 0 0 0) 0x7feec00ac500 con 0x7feec815dde0 2023-10-14T18:19:34.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.143+0000 7feec659c640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 0x7feea806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.144+0000 7feec659c640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 0x7feea806d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feeb8000f60 tx=0x7feeb8009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.147+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7feec0077490 con 0x7feec815dde0 2023-10-14T18:19:34.349 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.348+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec80adac0 con 0x7feec815dde0 2023-10-14T18:19:34.351 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.350+0000 7feebeffd640 1 -- 172.21.15.29:0/3478581662 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v267) v1 ==== 74+0+13041 (secure 0 0 0) 0x7feec007aab0 con 0x7feec815dde0 2023-10-14T18:19:34.351 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:34.351 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":267,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:33.310643+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":33,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":252,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:34.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 msgr2=0x7feea806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 0x7feea806d570 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7feeb8000f60 tx=0x7feeb8009040 comp rx=0 tx=0).stop 2023-10-14T18:19:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 msgr2=0x7feec81508c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 secure :-1 s=READY pgs=3115 cs=0 l=1 rev1=1 crypto rx=0x7feec0009f40 tx=0x7feec0004a40 comp rx=0 tx=0).stop 2023-10-14T18:19:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 shutdown_connections 2023-10-14T18:19:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7feea806b0b0 0x7feea806d570 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7feec815dde0 0x7feec81508c0 unknown :-1 s=CLOSED pgs=3115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7feec815c9f0 0x7feec806a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 --2- 172.21.15.29:0/3478581662 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7feec8070f20 0x7feec806c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 >> 172.21.15.29:0/3478581662 conn(0x7feec80ce6a0 msgr2=0x7feec80cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:34.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 shutdown_connections 2023-10-14T18:19:34.356 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.353+0000 7feec759e640 1 -- 172.21.15.29:0/3478581662 wait complete. 2023-10-14T18:19:34.373 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:34.374 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.605+0000 7f9bf7a0e640 1 Processor -- start 2023-10-14T18:19:34.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.606+0000 7f9bf7a0e640 1 -- start start 2023-10-14T18:19:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.607+0000 7f9bf7a0e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bf015ad70 0x7f9bf0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.607+0000 7f9bf7a0e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 0x7f9bf006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.607+0000 7f9bf7a0e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf00cb030 con 0x7f9bf00c9ca0 2023-10-14T18:19:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.607+0000 7f9bf7a0e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00c91c0 con 0x7f9bf015ad70 2023-10-14T18:19:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.607+0000 7f9bf7a0e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf00c94a0 con 0x7f9bf00caac0 2023-10-14T18:19:34.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bf015ad70 0x7f9bf0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9bf015ad70 0x7f9bf0069da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51090/0 (socket says 172.21.15.29:51090) 2023-10-14T18:19:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 -- 172.21.15.29:0/492561231 learned_addr learned my addr 172.21.15.29:0/492561231 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf5fab640 1 --2- 172.21.15.29:0/492561231 >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 0x7f9bf006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 -- 172.21.15.29:0/492561231 >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 msgr2=0x7f9bf006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/492561231 >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 0x7f9bf006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.608+0000 7f9bf57aa640 1 -- 172.21.15.29:0/492561231 >> v1:172.21.15.186:6789/0 conn(0x7f9bf00c9ca0 legacy=0x7f9bf00ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.609+0000 7f9bf57aa640 1 -- 172.21.15.29:0/492561231 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf00c9780 con 0x7f9bf015ad70 2023-10-14T18:19:34.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.609+0000 7f9bf5fab640 1 --2- 172.21.15.29:0/492561231 >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 0x7f9bf006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:34.616 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.610+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/492561231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf0069da0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f9bf006d090 tx=0x7f9bec02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44bad055769d4650 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.610+0000 7f9be7fff640 1 -- 172.21.15.29:0/492561231 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec02ca80 con 0x7f9bf015ad70 2023-10-14T18:19:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.610+0000 7f9be7fff640 1 -- 172.21.15.29:0/492561231 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec02cbe0 con 0x7f9bf015ad70 2023-10-14T18:19:34.617 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.610+0000 7f9be7fff640 1 -- 172.21.15.29:0/492561231 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bec0343f0 con 0x7f9bf015ad70 2023-10-14T18:19:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/492561231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 msgr2=0x7f9bf0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/492561231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf0069da0 secure :-1 s=READY pgs=3116 cs=0 l=1 rev1=1 crypto rx=0x7f9bf006d090 tx=0x7f9bec02bee0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/492561231 shutdown_connections 2023-10-14T18:19:34.618 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/492561231 >> v2:172.21.15.186:3300/0 conn(0x7f9bf00caac0 0x7f9bf006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/492561231 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf0069da0 unknown :-1 s=CLOSED pgs=3116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/492561231 >> 172.21.15.29:0/492561231 conn(0x7f9bf00ce9b0 msgr2=0x7f9bf00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/492561231 shutdown_connections 2023-10-14T18:19:34.619 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.611+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/492561231 wait complete. 2023-10-14T18:19:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.613+0000 7f9bf7a0e640 1 Processor -- start 2023-10-14T18:19:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.613+0000 7f9bf7a0e640 1 -- start start 2023-10-14T18:19:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 0x7f9bf00c8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.620 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 0x7f9bf00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf015ce70 con 0x7f9bf015ad70 2023-10-14T18:19:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00cc620 con 0x7f9bf00caac0 2023-10-14T18:19:34.621 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf7a0e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf00cc900 con 0x7f9bf00c9ca0 2023-10-14T18:19:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf5fab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf5fab640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51096/0 (socket says 172.21.15.29:51096) 2023-10-14T18:19:34.622 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf5fab640 1 -- 172.21.15.29:0/2446685125 learned_addr learned my addr 172.21.15.29:0/2446685125 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 0x7f9bf00c8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.614+0000 7f9bf4fa9640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 0x7f9bf00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 -- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 msgr2=0x7f9bf00c8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.623 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 0x7f9bf00c8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 -- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 msgr2=0x7f9bf00ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 0x7f9bf00ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 -- 172.21.15.29:0/2446685125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bec009b30 con 0x7f9bf015ad70 2023-10-14T18:19:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf4fa9640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 0x7f9bf00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:34.624 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 0x7f9bf00c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf5fab640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c7f0 tx=0x7f9be800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8007820 con 0x7f9bf015ad70 2023-10-14T18:19:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be8007980 con 0x7f9bf015ad70 2023-10-14T18:19:34.625 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9be8007d70 con 0x7f9bf015ad70 2023-10-14T18:19:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf00ccbe0 con 0x7f9bf015ad70 2023-10-14T18:19:34.626 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.615+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf00cd1a0 con 0x7f9bf015ad70 2023-10-14T18:19:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.617+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf004e160 con 0x7f9bf015ad70 2023-10-14T18:19:34.627 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.618+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f9be80113f0 con 0x7f9bf015ad70 2023-10-14T18:19:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.619+0000 7f9be5ffb640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 0x7f9bd406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.619+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f9be8089bb0 con 0x7f9bf015ad70 2023-10-14T18:19:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.619+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 0x7f9bd406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:34.628 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.620+0000 7f9bf57aa640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 0x7f9bd406d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9bec035000 tx=0x7f9bec030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:34.629 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.624+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9be8054ac0 con 0x7f9bf015ad70 2023-10-14T18:19:34.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.821+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f9bf006fa80 con 0x7f9bf015ad70 2023-10-14T18:19:34.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.822+0000 7f9be5ffb640 1 -- 172.21.15.29:0/2446685125 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v267) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9be801f0a0 con 0x7f9bf015ad70 2023-10-14T18:19:34.823 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 msgr2=0x7f9bd406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 0x7f9bd406d570 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f9bec035000 tx=0x7f9bec030000 comp rx=0 tx=0).stop 2023-10-14T18:19:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 msgr2=0x7f9bf00acdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:34.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 secure :-1 s=READY pgs=3117 cs=0 l=1 rev1=1 crypto rx=0x7f9be800c7f0 tx=0x7f9be800ccb0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 shutdown_connections 2023-10-14T18:19:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9bd406b0b0 0x7f9bd406d570 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9bf015ad70 0x7f9bf00acdd0 unknown :-1 s=CLOSED pgs=3117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9bf00caac0 0x7f9bf00ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 --2- 172.21.15.29:0/2446685125 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9bf00c9ca0 0x7f9bf00c8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:34.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 >> 172.21.15.29:0/2446685125 conn(0x7f9bf00ce9b0 msgr2=0x7f9bf00ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:34.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 shutdown_connections 2023-10-14T18:19:34.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:34.825+0000 7f9bf7a0e640 1 -- 172.21.15.29:0/2446685125 wait complete. 2023-10-14T18:19:34.846 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2023-10-14T18:19:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.078+0000 7f28caa7c640 1 Processor -- start 2023-10-14T18:19:35.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.078+0000 7f28caa7c640 1 -- start start 2023-10-14T18:19:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28caa7c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f28c415aaa0 0x7f28c415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28caa7c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c415ed60 con 0x7f28c415ae80 2023-10-14T18:19:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28caa7c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28c40707e0 con 0x7f28c40ce390 2023-10-14T18:19:35.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28caa7c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28c406b810 con 0x7f28c415aaa0 2023-10-14T18:19:35.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28bbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f28c415ae80 0x7f28c40cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35622/0 (socket says 172.21.15.29:35622) 2023-10-14T18:19:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.079+0000 7f28bbfff640 1 -- 172.21.15.29:0/4086554404 learned_addr learned my addr 172.21.15.29:0/4086554404 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048451142 0 0) 0x7f28c415ed60 con 0x7f28c415ae80 2023-10-14T18:19:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28ac003520 con 0x7f28c415ae80 2023-10-14T18:19:35.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28c8818640 1 --2- 172.21.15.29:0/4086554404 >> v2:172.21.15.186:3300/0 conn(0x7f28c415aaa0 0x7f28c415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1213471161 0 0) 0x7f28ac003520 con 0x7f28c415ae80 2023-10-14T18:19:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 >> v2:172.21.15.186:3300/0 conn(0x7f28c415aaa0 msgr2=0x7f28c415e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 --2- 172.21.15.29:0/4086554404 >> v2:172.21.15.186:3300/0 conn(0x7f28c415aaa0 0x7f28c415e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 >> v1:172.21.15.186:6789/0 conn(0x7f28c40ce390 legacy=0x7f28c4070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c415ef90 con 0x7f28c415ae80 2023-10-14T18:19:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f28b4001670 con 0x7f28c415ae80 2023-10-14T18:19:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.080+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 >> v1:172.21.15.29:6789/0 conn(0x7f28c415ae80 legacy=0x7f28c40cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 0x7f28ac003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac0047c0 0x7f28ac004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac005180 0x7f28ac00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c406b810 con 0x7f28ac0039c0 2023-10-14T18:19:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00c080 con 0x7f28ac0047c0 2023-10-14T18:19:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac00c240 con 0x7f28ac005180 2023-10-14T18:19:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 0x7f28ac003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28bbfff640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac0047c0 0x7f28ac004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c9019640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac005180 0x7f28ac00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 -- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac005180 msgr2=0x7f28ac00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac005180 0x7f28ac00b980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 -- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac0047c0 msgr2=0x7f28ac004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:35.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28ac0047c0 0x7f28ac004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c8818640 1 -- 172.21.15.29:0/4086554404 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28ac003520 con 0x7f28ac0039c0 2023-10-14T18:19:35.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.081+0000 7f28c9019640 1 -- 172.21.15.29:0/4086554404 reap_dead start 2023-10-14T18:19:35.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28c8818640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 0x7f28ac003de0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009ac0 tx=0x7f28ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8c59eb0e9583311 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c002c6b0 con 0x7f28ac0039c0 2023-10-14T18:19:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28c002c810 con 0x7f28ac0039c0 2023-10-14T18:19:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28bb7fe640 1 -- 172.21.15.29:0/4086554404 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28c002cba0 con 0x7f28ac0039c0 2023-10-14T18:19:35.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28caa7c640 1 -- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 msgr2=0x7f28ac003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28caa7c640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 0x7f28ac003de0 secure :-1 s=READY pgs=3119 cs=0 l=1 rev1=1 crypto rx=0x7f28c0009ac0 tx=0x7f28ac0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28caa7c640 1 -- 172.21.15.29:0/4086554404 shutdown_connections 2023-10-14T18:19:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28caa7c640 1 --2- 172.21.15.29:0/4086554404 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28ac0039c0 0x7f28ac003de0 unknown :-1 s=CLOSED pgs=3119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.083+0000 7f28caa7c640 1 -- 172.21.15.29:0/4086554404 >> 172.21.15.29:0/4086554404 conn(0x7f28c40cb410 msgr2=0x7f28c40cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:35.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.084+0000 7f28caa7c640 1 -- 172.21.15.29:0/4086554404 shutdown_connections 2023-10-14T18:19:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.084+0000 7f28caa7c640 1 -- 172.21.15.29:0/4086554404 wait complete. 2023-10-14T18:19:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.085+0000 7f28caa7c640 1 Processor -- start 2023-10-14T18:19:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 -- start start 2023-10-14T18:19:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac0039c0 0x7f28c415b320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 0x7f28c415c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c41564c0 con 0x7f28c4155b00 2023-10-14T18:19:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c415b900 con 0x7f28c40c4da0 2023-10-14T18:19:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.086+0000 7f28caa7c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28c40c4bb0 con 0x7f28ac0039c0 2023-10-14T18:19:35.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51110/0 (socket says 172.21.15.29:51110) 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 -- 172.21.15.29:0/1648992887 learned_addr learned my addr 172.21.15.29:0/1648992887 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28bbfff640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 0x7f28c415c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c8818640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac0039c0 0x7f28c415b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 -- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac0039c0 msgr2=0x7f28c415b320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac0039c0 0x7f28c415b320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 -- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 msgr2=0x7f28c415c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 0x7f28c415c0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28c9019640 1 -- 172.21.15.29:0/1648992887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28c40c4280 con 0x7f28c4155b00 2023-10-14T18:19:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.087+0000 7f28bbfff640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 0x7f28c415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:35.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28c9019640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f28c40b4390 tx=0x7f28bc00d9d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc003d80 con 0x7f28c4155b00 2023-10-14T18:19:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28bc005070 con 0x7f28c4155b00 2023-10-14T18:19:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28bc005400 con 0x7f28c4155b00 2023-10-14T18:19:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c40c4560 con 0x7f28c4155b00 2023-10-14T18:19:35.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.088+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c40c1ef0 con 0x7f28c4155b00 2023-10-14T18:19:35.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.089+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c404e160 con 0x7f28c4155b00 2023-10-14T18:19:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.091+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f28bc005560 con 0x7f28c4155b00 2023-10-14T18:19:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.092+0000 7f28b97fa640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 0x7f28b006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.092+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 4973+0+0 (secure 0 0 0) 0x7f28bc0890e0 con 0x7f28c4155b00 2023-10-14T18:19:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.092+0000 7f28c8818640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 0x7f28b006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.093+0000 7f28c8818640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 0x7f28b006d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f28c0036000 tx=0x7f28c0030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.096+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f28bc054070 con 0x7f28c4155b00 2023-10-14T18:19:35.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.298+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f28c4156b10 con 0x7f28c4155b00 2023-10-14T18:19:35.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.299+0000 7f28b97fa640 1 -- 172.21.15.29:0/1648992887 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v267) v1 ==== 84+0+65 (secure 0 0 0) 0x7f28bc018090 con 0x7f28c4155b00 2023-10-14T18:19:35.300 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:35.300 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:19:35.300 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:35.300 INFO:teuthology.orchestra.run.smithi029.stdout: "poolnum": 1, 2023-10-14T18:19:35.301 INFO:teuthology.orchestra.run.smithi029.stdout: "poolname": ".mgr" 2023-10-14T18:19:35.301 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:35.301 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:19:35.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 msgr2=0x7f28b006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 0x7f28b006d5c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f28c0036000 tx=0x7f28c0030000 comp rx=0 tx=0).stop 2023-10-14T18:19:35.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 msgr2=0x7f28c4155f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 secure :-1 s=READY pgs=3120 cs=0 l=1 rev1=1 crypto rx=0x7f28c40b4390 tx=0x7f28bc00d9d0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 shutdown_connections 2023-10-14T18:19:35.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f28b006b100 0x7f28b006d5c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f28c4155b00 0x7f28c4155f80 unknown :-1 s=CLOSED pgs=3120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f28c40c4da0 0x7f28c415c0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 --2- 172.21.15.29:0/1648992887 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f28ac0039c0 0x7f28c415b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 >> 172.21.15.29:0/1648992887 conn(0x7f28c40cb410 msgr2=0x7f28c415a120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:35.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 shutdown_connections 2023-10-14T18:19:35.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.302+0000 7f28caa7c640 1 -- 172.21.15.29:0/1648992887 wait complete. 2023-10-14T18:19:35.321 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2023-10-14T18:19:35.322 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2023-10-14T18:19:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.542+0000 7faeb359e640 1 Processor -- start 2023-10-14T18:19:35.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.543+0000 7faeb359e640 1 -- start start 2023-10-14T18:19:35.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.544+0000 7faeb359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.544+0000 7faeb359e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faeb40cae40 0x7faeb40c9840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb359e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeb40c8e70 con 0x7faeb415ad80 2023-10-14T18:19:35.546 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb359e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faeb40b4170 con 0x7faeb40cae40 2023-10-14T18:19:35.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb40adde0 con 0x7faeb406d280 2023-10-14T18:19:35.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb1d9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb2d9d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7faeb40cae40 0x7faeb40c9840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb1d9b640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:60468/0 (socket says 172.21.15.29:60468) 2023-10-14T18:19:35.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.545+0000 7faeb1d9b640 1 -- 172.21.15.29:0/784205302 learned_addr learned my addr 172.21.15.29:0/784205302 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:35.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.546+0000 7faeb1d9b640 1 -- 172.21.15.29:0/784205302 >> v2:172.21.15.186:3300/0 conn(0x7faeb40cae40 msgr2=0x7faeb40c9840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:35.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.546+0000 7faeb1d9b640 1 --2- 172.21.15.29:0/784205302 >> v2:172.21.15.186:3300/0 conn(0x7faeb40cae40 0x7faeb40c9840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.546+0000 7faeb1d9b640 1 -- 172.21.15.29:0/784205302 >> v1:172.21.15.186:6789/0 conn(0x7faeb415ad80 legacy=0x7faeb406cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.546+0000 7faeb1d9b640 1 -- 172.21.15.29:0/784205302 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeb40b4410 con 0x7faeb406d280 2023-10-14T18:19:35.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.548+0000 7faeb1d9b640 1 --2- 172.21.15.29:0/784205302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009870 tx=0x7fae9c02b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92ee14b24d6cbc0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.548+0000 7faeb159a640 1 -- 172.21.15.29:0/784205302 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c030840 con 0x7faeb406d280 2023-10-14T18:19:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.548+0000 7faeb159a640 1 -- 172.21.15.29:0/784205302 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c0309a0 con 0x7faeb406d280 2023-10-14T18:19:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.548+0000 7faeb159a640 1 -- 172.21.15.29:0/784205302 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9c030d30 con 0x7faeb406d280 2023-10-14T18:19:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 -- 172.21.15.29:0/784205302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 msgr2=0x7faeb40c9300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 --2- 172.21.15.29:0/784205302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009870 tx=0x7fae9c02b980 comp rx=0 tx=0).stop 2023-10-14T18:19:35.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 -- 172.21.15.29:0/784205302 shutdown_connections 2023-10-14T18:19:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 --2- 172.21.15.29:0/784205302 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb40c9300 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 --2- 172.21.15.29:0/784205302 >> v2:172.21.15.186:3300/0 conn(0x7faeb40cae40 0x7faeb40c9840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 -- 172.21.15.29:0/784205302 >> 172.21.15.29:0/784205302 conn(0x7faeb40ce820 msgr2=0x7faeb40cec30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 -- 172.21.15.29:0/784205302 shutdown_connections 2023-10-14T18:19:35.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.549+0000 7faeb359e640 1 -- 172.21.15.29:0/784205302 wait complete. 2023-10-14T18:19:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.551+0000 7faeb359e640 1 Processor -- start 2023-10-14T18:19:35.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.551+0000 7faeb359e640 1 -- start start 2023-10-14T18:19:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb4155610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb40cae40 0x7faeb40acb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4156620 con 0x7faeb415ad80 2023-10-14T18:19:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4156900 con 0x7faeb40cae40 2023-10-14T18:19:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb359e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faeb4156be0 con 0x7faeb406d280 2023-10-14T18:19:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb2d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51118/0 (socket says 172.21.15.29:51118) 2023-10-14T18:19:35.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb2d9d640 1 -- 172.21.15.29:0/229215376 learned_addr learned my addr 172.21.15.29:0/229215376 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb1d9b640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb40cae40 0x7faeb40acb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.552+0000 7faeb259c640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb4155610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 -- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 msgr2=0x7faeb4155610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb4155610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 -- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb40cae40 msgr2=0x7faeb40acb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb40cae40 0x7faeb40acb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 -- 172.21.15.29:0/229215376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0094d0 con 0x7faeb415ad80 2023-10-14T18:19:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb2d9d640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e930 tx=0x7faeac00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac009690 con 0x7faeb415ad80 2023-10-14T18:19:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faeac00c980 con 0x7faeb415ad80 2023-10-14T18:19:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faeb4156ec0 con 0x7faeb415ad80 2023-10-14T18:19:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faeac00cd30 con 0x7faeb415ad80 2023-10-14T18:19:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.553+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faeb4155d20 con 0x7faeb415ad80 2023-10-14T18:19:35.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.554+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faeb41573e0 con 0x7faeb415ad80 2023-10-14T18:19:35.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.556+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7faeac013470 con 0x7faeb415ad80 2023-10-14T18:19:35.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.557+0000 7faeaaffd640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 0x7fae9406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:35.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.557+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(267..267 src has 1..267) v4 ==== 4973+0+0 (secure 0 0 0) 0x7faeac089970 con 0x7faeb415ad80 2023-10-14T18:19:35.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.557+0000 7faeb259c640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 0x7fae9406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:35.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.558+0000 7faeb259c640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 0x7fae9406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faeb41575a0 tx=0x7faea4009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:35.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.561+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7faeb41573e0 con 0x7faeb415ad80 2023-10-14T18:19:35.758 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.757+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7faeb41573e0 con 0x7faeb415ad80 2023-10-14T18:19:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.890+0000 7faeaaffd640 1 -- 172.21.15.29:0/229215376 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v268) v1 ==== 138+0+0 (secure 0 0 0) 0x7faeb41573e0 con 0x7faeb415ad80 2023-10-14T18:19:35.891 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' created 2023-10-14T18:19:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 msgr2=0x7fae9406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.895 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 0x7fae9406d570 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7faeb41575a0 tx=0x7faea4009040 comp rx=0 tx=0).stop 2023-10-14T18:19:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 msgr2=0x7faeb4154ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 secure :-1 s=READY pgs=3121 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e930 tx=0x7faeac00edf0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 shutdown_connections 2023-10-14T18:19:35.896 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae9406b0b0 0x7fae9406d570 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7faeb415ad80 0x7faeb4154ab0 unknown :-1 s=CLOSED pgs=3121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7faeb40cae40 0x7faeb40acb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 --2- 172.21.15.29:0/229215376 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7faeb406d280 0x7faeb4155610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.894+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 >> 172.21.15.29:0/229215376 conn(0x7faeb40ce820 msgr2=0x7faeb40af470 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:35.897 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.895+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 shutdown_connections 2023-10-14T18:19:35.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:35.895+0000 7faeb359e640 1 -- 172.21.15.29:0/229215376 wait complete. 2023-10-14T18:19:35.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2023-10-14T18:19:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.149+0000 7ffbde47c640 1 Processor -- start 2023-10-14T18:19:36.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.150+0000 7ffbde47c640 1 -- start start 2023-10-14T18:19:36.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbde47c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffbd8159b20 0x7ffbd806bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbde47c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 0x7ffbd806fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbde47c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbd806f890 con 0x7ffbd8159f00 2023-10-14T18:19:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbde47c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbd806c650 con 0x7ffbd8159b20 2023-10-14T18:19:36.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbde47c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd806c930 con 0x7ffbd80b0b20 2023-10-14T18:19:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.151+0000 7ffbd7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffbd8159b20 0x7ffbd806bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7ffbd8159b20 0x7ffbd806bbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51130/0 (socket says 172.21.15.29:51130) 2023-10-14T18:19:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 -- 172.21.15.29:0/3908403096 learned_addr learned my addr 172.21.15.29:0/3908403096 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:36.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbdca19640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 0x7ffbd806fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 -- 172.21.15.29:0/3908403096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 msgr2=0x7ffbd806fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:36.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 0x7ffbd806fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 -- 172.21.15.29:0/3908403096 >> v1:172.21.15.186:6789/0 conn(0x7ffbd8159f00 legacy=0x7ffbd80b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbd7fff640 1 -- 172.21.15.29:0/3908403096 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbd806cc10 con 0x7ffbd8159b20 2023-10-14T18:19:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.152+0000 7ffbdca19640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 0x7ffbd806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbd7fff640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd806bbe0 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800bb10 tx=0x7ffbc802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37141d4f36d4bc1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbd77fe640 1 -- 172.21.15.29:0/3908403096 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8036440 con 0x7ffbd8159b20 2023-10-14T18:19:36.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbd77fe640 1 -- 172.21.15.29:0/3908403096 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc80365a0 con 0x7ffbd8159b20 2023-10-14T18:19:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbd77fe640 1 -- 172.21.15.29:0/3908403096 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8036930 con 0x7ffbd8159b20 2023-10-14T18:19:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbde47c640 1 -- 172.21.15.29:0/3908403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 msgr2=0x7ffbd806bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.154+0000 7ffbde47c640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd806bbe0 secure :-1 s=READY pgs=3122 cs=0 l=1 rev1=1 crypto rx=0x7ffbc800bb10 tx=0x7ffbc802e150 comp rx=0 tx=0).stop 2023-10-14T18:19:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 -- 172.21.15.29:0/3908403096 shutdown_connections 2023-10-14T18:19:36.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd80b0b20 0x7ffbd806fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 --2- 172.21.15.29:0/3908403096 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd806bbe0 unknown :-1 s=CLOSED pgs=3122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 -- 172.21.15.29:0/3908403096 >> 172.21.15.29:0/3908403096 conn(0x7ffbd80cb410 msgr2=0x7ffbd80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 -- 172.21.15.29:0/3908403096 shutdown_connections 2023-10-14T18:19:36.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.155+0000 7ffbde47c640 1 -- 172.21.15.29:0/3908403096 wait complete. 2023-10-14T18:19:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.157+0000 7ffbde47c640 1 Processor -- start 2023-10-14T18:19:36.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.157+0000 7ffbde47c640 1 -- start start 2023-10-14T18:19:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbd80b0b20 0x7ffbd8156f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 0x7ffbd80ac8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80c24c0 con 0x7ffbd8159b20 2023-10-14T18:19:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80b25e0 con 0x7ffbd80b0b20 2023-10-14T18:19:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbde47c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbd80c5ed0 con 0x7ffbd8159f00 2023-10-14T18:19:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbcf7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51136/0 (socket says 172.21.15.29:51136) 2023-10-14T18:19:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbcf7fe640 1 -- 172.21.15.29:0/664617927 learned_addr learned my addr 172.21.15.29:0/664617927 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbd7fff640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbd80b0b20 0x7ffbd8156f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.158+0000 7ffbdca19640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 0x7ffbd80ac8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 -- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 msgr2=0x7ffbd80ac8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 0x7ffbd80ac8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 -- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbd80b0b20 msgr2=0x7ffbd8156f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbd80b0b20 0x7ffbd8156f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 -- 172.21.15.29:0/664617927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbc800b770 con 0x7ffbd8159b20 2023-10-14T18:19:36.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbdca19640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 0x7ffbd80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbcf7fe640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c400 tx=0x7ffbc000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.159+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc00052d0 con 0x7ffbd8159b20 2023-10-14T18:19:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.160+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc0005430 con 0x7ffbd8159b20 2023-10-14T18:19:36.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.160+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc0003830 con 0x7ffbd8159b20 2023-10-14T18:19:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.160+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbd80b2830 con 0x7ffbd8159b20 2023-10-14T18:19:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.160+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbd80b2df0 con 0x7ffbd8159b20 2023-10-14T18:19:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.161+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbd804e160 con 0x7ffbd8159b20 2023-10-14T18:19:36.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.163+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7ffbc0003990 con 0x7ffbd8159b20 2023-10-14T18:19:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.164+0000 7ffbd57fa640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 0x7ffbb806d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:36.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.164+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(268..268 src has 1..268) v4 ==== 5343+0+0 (secure 0 0 0) 0x7ffbc0089180 con 0x7ffbd8159b20 2023-10-14T18:19:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.164+0000 7ffbd7fff640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 0x7ffbb806d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.165+0000 7ffbd7fff640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 0x7ffbb806d5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8038000 tx=0x7ffbc8008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:36.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.168+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffbc0054180 con 0x7ffbd8159b20 2023-10-14T18:19:36.365 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.365+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ffbd80b10b0 con 0x7ffbd8159b20 2023-10-14T18:19:36.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.911+0000 7ffbd57fa640 1 -- 172.21.15.29:0/664617927 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v269) v1 ==== 170+0+0 (secure 0 0 0) 0x7ffbc00577a0 con 0x7ffbd8159b20 2023-10-14T18:19:36.913 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' created 2023-10-14T18:19:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 msgr2=0x7ffbb806d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 0x7ffbb806d5f0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8038000 tx=0x7ffbc8008040 comp rx=0 tx=0).stop 2023-10-14T18:19:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 msgr2=0x7ffbd80c83b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 secure :-1 s=READY pgs=3123 cs=0 l=1 rev1=1 crypto rx=0x7ffbc000c400 tx=0x7ffbc000c8c0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 shutdown_connections 2023-10-14T18:19:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7ffbb806b130 0x7ffbb806d5f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffbd8159f00 0x7ffbd80ac8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffbd8159b20 0x7ffbd80c83b0 unknown :-1 s=CLOSED pgs=3123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 --2- 172.21.15.29:0/664617927 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffbd80b0b20 0x7ffbd8156f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.915+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 >> 172.21.15.29:0/664617927 conn(0x7ffbd80cb410 msgr2=0x7ffbd80cc540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.916+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 shutdown_connections 2023-10-14T18:19:36.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:36.916+0000 7ffbde47c640 1 -- 172.21.15.29:0/664617927 wait complete. 2023-10-14T18:19:36.950 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2023-10-14T18:19:37.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.143+0000 7f3a4d96c640 1 Processor -- start 2023-10-14T18:19:37.144 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.144+0000 7f3a4d96c640 1 -- start start 2023-10-14T18:19:37.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a4d96c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f3a48159ab0 0x7f3a480ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a4d96c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a48159e90 0x7f3a4806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a4d96c640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a480cd1f0 con 0x7f3a4806c140 2023-10-14T18:19:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a4d96c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4815d890 con 0x7f3a48159e90 2023-10-14T18:19:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a4d96c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a480cd4d0 con 0x7f3a48159ab0 2023-10-14T18:19:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a3ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3a48159e90 0x7f3a4806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a477fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3a4806c140 0x7f3a4815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35630/0 (socket says 172.21.15.29:35630) 2023-10-14T18:19:37.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.145+0000 7f3a477fe640 1 -- 172.21.15.29:0/2870256164 learned_addr learned my addr 172.21.15.29:0/2870256164 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.146+0000 7f3a46ffd640 1 --2- 172.21.15.29:0/2870256164 >> v2:172.21.15.186:3300/0 conn(0x7f3a48159ab0 0x7f3a480ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.146+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2870256164 >> v2:172.21.15.186:3300/0 conn(0x7f3a48159ab0 msgr2=0x7f3a480ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:37.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.146+0000 7f3a3ffff640 1 --2- 172.21.15.29:0/2870256164 >> v2:172.21.15.186:3300/0 conn(0x7f3a48159ab0 0x7f3a480ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.146+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2870256164 >> v1:172.21.15.29:6789/0 conn(0x7f3a4806c140 legacy=0x7f3a4815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.146+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2870256164 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a480cd7b0 con 0x7f3a48159e90 2023-10-14T18:19:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.147+0000 7f3a3ffff640 1 --2- 172.21.15.29:0/2870256164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159e90 0x7f3a4806fc50 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cfa00 tx=0x7f3a3002b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5defbbadd457b140 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.147+0000 7f3a467fc640 1 -- 172.21.15.29:0/2870256164 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30031520 con 0x7f3a48159e90 2023-10-14T18:19:37.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a467fc640 1 -- 172.21.15.29:0/2870256164 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a30031680 con 0x7f3a48159e90 2023-10-14T18:19:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a467fc640 1 -- 172.21.15.29:0/2870256164 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30031a10 con 0x7f3a48159e90 2023-10-14T18:19:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2870256164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159e90 msgr2=0x7f3a4806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2870256164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159e90 0x7f3a4806fc50 secure :-1 s=READY pgs=3125 cs=0 l=1 rev1=1 crypto rx=0x7f3a480cfa00 tx=0x7f3a3002b7c0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2870256164 shutdown_connections 2023-10-14T18:19:37.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2870256164 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159e90 0x7f3a4806fc50 unknown :-1 s=CLOSED pgs=3125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2870256164 >> v2:172.21.15.186:3300/0 conn(0x7f3a48159ab0 0x7f3a480ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2870256164 >> 172.21.15.29:0/2870256164 conn(0x7f3a480ce9b0 msgr2=0x7f3a480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2870256164 shutdown_connections 2023-10-14T18:19:37.152 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.148+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2870256164 wait complete. 2023-10-14T18:19:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.150+0000 7f3a4d96c640 1 Processor -- start 2023-10-14T18:19:37.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.150+0000 7f3a4d96c640 1 -- start start 2023-10-14T18:19:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a4806c140 0x7f3a481f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4806b570 0x7f3a481f1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4806ba20 con 0x7f3a48159ab0 2023-10-14T18:19:37.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a480c90e0 con 0x7f3a4806b570 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.151+0000 7f3a4d96c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a480b2350 con 0x7f3a4806c140 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51162/0 (socket says 172.21.15.29:51162) 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2775113395 learned_addr learned my addr 172.21.15.29:0/2775113395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a46ffd640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a4806c140 0x7f3a481f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a477fe640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4806b570 0x7f3a481f1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a4806c140 msgr2=0x7f3a481f6510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a4806c140 0x7f3a481f6510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4806b570 msgr2=0x7f3a481f1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4806b570 0x7f3a481f1cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:37.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.152+0000 7f3a3ffff640 1 -- 172.21.15.29:0/2775113395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a30009400 con 0x7f3a48159ab0 2023-10-14T18:19:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a3ffff640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f3a30036040 tx=0x7f3a300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30031520 con 0x7f3a48159ab0 2023-10-14T18:19:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a30033c70 con 0x7f3a48159ab0 2023-10-14T18:19:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004030 con 0x7f3a48159ab0 2023-10-14T18:19:37.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a480b0780 con 0x7f3a48159ab0 2023-10-14T18:19:37.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.153+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a481f2a20 con 0x7f3a48159ab0 2023-10-14T18:19:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.154+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a480b24e0 con 0x7f3a48159ab0 2023-10-14T18:19:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.156+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f3a3003a040 con 0x7f3a48159ab0 2023-10-14T18:19:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.157+0000 7f3a44ff9640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 0x7f3a2806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.157+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(269..269 src has 1..269) v4 ==== 5725+0+0 (secure 0 0 0) 0x7f3a300ab110 con 0x7f3a48159ab0 2023-10-14T18:19:37.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.157+0000 7f3a46ffd640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 0x7f3a2806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:37.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.158+0000 7f3a46ffd640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 0x7f3a2806d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a38006fd0 tx=0x7f3a38008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:37.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.162+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3a30075ca0 con 0x7f3a48159ab0 2023-10-14T18:19:37.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:37.401+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f3a4804e160 con 0x7f3a48159ab0 2023-10-14T18:19:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.282+0000 7f3a44ff9640 1 -- 172.21.15.29:0/2775113395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 34 and data pool 35 v135) v1 ==== 174+0+0 (secure 0 0 0) 0x7f3a300792c0 con 0x7f3a48159ab0 2023-10-14T18:19:38.284 INFO:teuthology.orchestra.run.smithi029.stderr:new fs with metadata pool 34 and data pool 35 2023-10-14T18:19:38.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 msgr2=0x7f3a2806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 0x7f3a2806d570 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a38006fd0 tx=0x7f3a38008040 comp rx=0 tx=0).stop 2023-10-14T18:19:38.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 msgr2=0x7f3a4806b030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 secure :-1 s=READY pgs=3126 cs=0 l=1 rev1=1 crypto rx=0x7f3a30036040 tx=0x7f3a300046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 shutdown_connections 2023-10-14T18:19:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f3a2806b0b0 0x7f3a2806d570 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3a4806b570 0x7f3a481f1cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3a48159ab0 0x7f3a4806b030 unknown :-1 s=CLOSED pgs=3126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 --2- 172.21.15.29:0/2775113395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3a4806c140 0x7f3a481f6510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 >> 172.21.15.29:0/2775113395 conn(0x7f3a480ce9b0 msgr2=0x7f3a480c47d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 shutdown_connections 2023-10-14T18:19:38.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.287+0000 7f3a4d96c640 1 -- 172.21.15.29:0/2775113395 wait complete. 2023-10-14T18:19:38.313 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:38.530 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.529+0000 7f7d37b61640 1 Processor -- start 2023-10-14T18:19:38.531 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.529+0000 7f7d37b61640 1 -- start start 2023-10-14T18:19:38.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d37b61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d3015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d37b61640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d3015ed60 con 0x7f7d300ce390 2023-10-14T18:19:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d37b61640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d300707e0 con 0x7f7d3015ae80 2023-10-14T18:19:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d37b61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3006b810 con 0x7f7d3015aaa0 2023-10-14T18:19:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d360fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f7d300ce390 0x7f7d30070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:35646/0 (socket says 172.21.15.29:35646) 2023-10-14T18:19:38.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.531+0000 7f7d360fe640 1 -- 172.21.15.29:0/1546101507 learned_addr learned my addr 172.21.15.29:0/1546101507 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:38.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d358fd640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d3015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033480760 0 0) 0x7f7d3015ed60 con 0x7f7d300ce390 2023-10-14T18:19:38.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d1c003520 con 0x7f7d300ce390 2023-10-14T18:19:38.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3069000246 0 0) 0x7f7d1c003520 con 0x7f7d300ce390 2023-10-14T18:19:38.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 msgr2=0x7f7d3015e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:38.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d3015e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 >> v1:172.21.15.186:6789/0 conn(0x7f7d3015ae80 legacy=0x7f7d300cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3015ef90 con 0x7f7d300ce390 2023-10-14T18:19:38.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.532+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f7d280019f0 con 0x7f7d300ce390 2023-10-14T18:19:38.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 >> v1:172.21.15.29:6789/0 conn(0x7f7d300ce390 legacy=0x7f7d30070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d1c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d1c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1c004d00 0x7f7d1c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3006b810 con 0x7f7d1c003f20 2023-10-14T18:19:38.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c00ba60 con 0x7f7d1c004d00 2023-10-14T18:19:38.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d1c00bc20 con 0x7f7d3015aaa0 2023-10-14T18:19:38.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d1c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.540 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d360fe640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1c004d00 0x7f7d1c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d358fd640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d1c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 -- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 msgr2=0x7f7d1c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:38.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015aaa0 0x7f7d1c0039a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 -- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1c004d00 msgr2=0x7f7d1c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:38.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1c004d00 0x7f7d1c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d350fc640 1 -- 172.21.15.29:0/1546101507 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d1c003520 con 0x7f7d1c003f20 2023-10-14T18:19:38.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.533+0000 7f7d360fe640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d1c004d00 0x7f7d1c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:38.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d1c0047c0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f7d20001750 tx=0x7f7d1c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd2cc51337d2347 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:38.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d200052d0 con 0x7f7d1c003f20 2023-10-14T18:19:38.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2002e050 con 0x7f7d1c003f20 2023-10-14T18:19:38.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d348fb640 1 -- 172.21.15.29:0/1546101507 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d20003830 con 0x7f7d1c003f20 2023-10-14T18:19:38.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d37b61640 1 -- 172.21.15.29:0/1546101507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 msgr2=0x7f7d1c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1546101507 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d1c0047c0 secure :-1 s=READY pgs=3128 cs=0 l=1 rev1=1 crypto rx=0x7f7d20001750 tx=0x7f7d1c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.535+0000 7f7d360fe640 1 -- 172.21.15.29:0/1546101507 reap_dead start 2023-10-14T18:19:38.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.536+0000 7f7d37b61640 1 -- 172.21.15.29:0/1546101507 shutdown_connections 2023-10-14T18:19:38.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.536+0000 7f7d37b61640 1 -- 172.21.15.29:0/1546101507 >> 172.21.15.29:0/1546101507 conn(0x7f7d300cb410 msgr2=0x7f7d300cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:38.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.536+0000 7f7d37b61640 1 -- 172.21.15.29:0/1546101507 shutdown_connections 2023-10-14T18:19:38.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.536+0000 7f7d37b61640 1 -- 172.21.15.29:0/1546101507 wait complete. 2023-10-14T18:19:38.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.537+0000 7f7d37b61640 1 Processor -- start 2023-10-14T18:19:38.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.538+0000 7f7d37b61640 1 -- start start 2023-10-14T18:19:38.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 0x7f7d30158e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d300c4690 0x7f7d300c4b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d300c5050 con 0x7f7d1c003f20 2023-10-14T18:19:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015dd10 con 0x7f7d300c4690 2023-10-14T18:19:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d37b61640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d3015c510 con 0x7f7d3015d890 2023-10-14T18:19:38.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51188/0 (socket says 172.21.15.29:51188) 2023-10-14T18:19:38.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 -- 172.21.15.29:0/1213188271 learned_addr learned my addr 172.21.15.29:0/1213188271 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:38.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 -- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 msgr2=0x7f7d30158e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.550 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 0x7f7d30158e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 0x7f7d30158e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 -- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d300c4690 msgr2=0x7f7d300c4b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d300c4690 0x7f7d300c4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d358fd640 1 -- 172.21.15.29:0/1213188271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d3015c7f0 con 0x7f7d1c003f20 2023-10-14T18:19:38.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.539+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 0x7f7d30158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d358fd640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c010860 tx=0x7f7d2c010d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c009610 con 0x7f7d1c003f20 2023-10-14T18:19:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d2c00eb60 con 0x7f7d1c003f20 2023-10-14T18:19:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2c01c400 con 0x7f7d1c003f20 2023-10-14T18:19:38.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d3015cad0 con 0x7f7d1c003f20 2023-10-14T18:19:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.540+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d3015d090 con 0x7f7d1c003f20 2023-10-14T18:19:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.541+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d3004e160 con 0x7f7d1c003f20 2023-10-14T18:19:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.543+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f7d2c01c560 con 0x7f7d1c003f20 2023-10-14T18:19:38.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.543+0000 7f7d267fc640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 0x7f7d1406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.544+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(270..270 src has 1..270) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7d2c089fe0 con 0x7f7d1c003f20 2023-10-14T18:19:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.544+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 0x7f7d1406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.545+0000 7f7d350fc640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 0x7f7d1406d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d20033000 tx=0x7f7d20009030 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:38.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.548+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7d2c054d90 con 0x7f7d1c003f20 2023-10-14T18:19:38.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.746+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d3015f520 con 0x7f7d1c003f20 2023-10-14T18:19:38.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.749+0000 7f7d267fc640 1 -- 172.21.15.29:0/1213188271 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v270) v1 ==== 74+0+16714 (secure 0 0 0) 0x7f7d2c0583b0 con 0x7f7d1c003f20 2023-10-14T18:19:38.750 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:38.751 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":270,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:37.402786+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:38.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 msgr2=0x7f7d1406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 0x7f7d1406d5a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7d20033000 tx=0x7f7d20009030 comp rx=0 tx=0).stop 2023-10-14T18:19:38.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 msgr2=0x7f7d30157a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 secure :-1 s=READY pgs=3129 cs=0 l=1 rev1=1 crypto rx=0x7f7d2c010860 tx=0x7f7d2c010d20 comp rx=0 tx=0).stop 2023-10-14T18:19:38.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 shutdown_connections 2023-10-14T18:19:38.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f7d1406b0e0 0x7f7d1406d5a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7d300c4690 0x7f7d300c4b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7d3015d890 0x7f7d30158e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 --2- 172.21.15.29:0/1213188271 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7d1c003f20 0x7f7d30157a50 unknown :-1 s=CLOSED pgs=3129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 >> 172.21.15.29:0/1213188271 conn(0x7f7d300cb410 msgr2=0x7f7d300cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:38.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 shutdown_connections 2023-10-14T18:19:38.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.752+0000 7f7d37b61640 1 -- 172.21.15.29:0/1213188271 wait complete. 2023-10-14T18:19:38.772 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:38.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.986+0000 7f5d19851640 1 Processor -- start 2023-10-14T18:19:38.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.986+0000 7f5d19851640 1 -- start start 2023-10-14T18:19:38.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.987+0000 7f5d19851640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d1415aaa0 0x7f5d1415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.987+0000 7f5d19851640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5d140ce390 0x7f5d14070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.987+0000 7f5d19851640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d1415ed60 con 0x7f5d1415ae80 2023-10-14T18:19:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.987+0000 7f5d19851640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d1406b810 con 0x7f5d1415aaa0 2023-10-14T18:19:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.987+0000 7f5d19851640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d140707e0 con 0x7f5d140ce390 2023-10-14T18:19:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d1415aaa0 0x7f5d1415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5d1415aaa0 0x7f5d1415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51200/0 (socket says 172.21.15.29:51200) 2023-10-14T18:19:38.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 -- 172.21.15.29:0/3299595375 learned_addr learned my addr 172.21.15.29:0/3299595375 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:38.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 -- 172.21.15.29:0/3299595375 >> v2:172.21.15.186:3300/0 conn(0x7f5d140ce390 msgr2=0x7f5d14070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:38.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/3299595375 >> v2:172.21.15.186:3300/0 conn(0x7f5d140ce390 0x7f5d14070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 -- 172.21.15.29:0/3299595375 >> v1:172.21.15.186:6789/0 conn(0x7f5d1415ae80 legacy=0x7f5d140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:38.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.988+0000 7f5d12ffd640 1 -- 172.21.15.29:0/3299595375 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1415ef90 con 0x7f5d1415aaa0 2023-10-14T18:19:38.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.990+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/3299595375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415aaa0 0x7f5d1415e820 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f5d140c2490 tx=0x7f5d0402bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f2a5b2d6f1489e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:38.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.990+0000 7f5d11ffb640 1 -- 172.21.15.29:0/3299595375 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0402ca80 con 0x7f5d1415aaa0 2023-10-14T18:19:38.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.990+0000 7f5d11ffb640 1 -- 172.21.15.29:0/3299595375 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0402cbe0 con 0x7f5d1415aaa0 2023-10-14T18:19:38.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.990+0000 7f5d11ffb640 1 -- 172.21.15.29:0/3299595375 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d040343f0 con 0x7f5d1415aaa0 2023-10-14T18:19:38.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 -- 172.21.15.29:0/3299595375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415aaa0 msgr2=0x7f5d1415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:38.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 --2- 172.21.15.29:0/3299595375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415aaa0 0x7f5d1415e820 secure :-1 s=READY pgs=3130 cs=0 l=1 rev1=1 crypto rx=0x7f5d140c2490 tx=0x7f5d0402bee0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 -- 172.21.15.29:0/3299595375 shutdown_connections 2023-10-14T18:19:38.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 --2- 172.21.15.29:0/3299595375 >> v2:172.21.15.186:3300/0 conn(0x7f5d140ce390 0x7f5d14070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 --2- 172.21.15.29:0/3299595375 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415aaa0 0x7f5d1415e820 unknown :-1 s=CLOSED pgs=3130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:38.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 -- 172.21.15.29:0/3299595375 >> 172.21.15.29:0/3299595375 conn(0x7f5d140cb410 msgr2=0x7f5d140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:38.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 -- 172.21.15.29:0/3299595375 shutdown_connections 2023-10-14T18:19:38.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.991+0000 7f5d19851640 1 -- 172.21.15.29:0/3299595375 wait complete. 2023-10-14T18:19:38.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.993+0000 7f5d19851640 1 Processor -- start 2023-10-14T18:19:38.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.993+0000 7f5d19851640 1 -- start start 2023-10-14T18:19:38.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 0x7f5d140b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d1415aaa0 0x7f5d1414a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:38.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141e4650 con 0x7f5d1415ae80 2023-10-14T18:19:38.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141e4930 con 0x7f5d140ce390 2023-10-14T18:19:38.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d19851640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d141e4c10 con 0x7f5d1415aaa0 2023-10-14T18:19:39.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d137fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51216/0 (socket says 172.21.15.29:51216) 2023-10-14T18:19:39.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d137fe640 1 -- 172.21.15.29:0/1850362059 learned_addr learned my addr 172.21.15.29:0/1850362059 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:39.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 0x7f5d140b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.994+0000 7f5d127fc640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d1415aaa0 0x7f5d1414a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 -- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d1415aaa0 msgr2=0x7f5d1414a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d1415aaa0 0x7f5d1414a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 -- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 msgr2=0x7f5d140b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 0x7f5d140b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 -- 172.21.15.29:0/1850362059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d04009b30 con 0x7f5d1415ae80 2023-10-14T18:19:39.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 0x7f5d140b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:39.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d137fe640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009fd0 tx=0x7f5d0c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:39.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c011830 con 0x7f5d1415ae80 2023-10-14T18:19:39.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d0c011990 con 0x7f5d1415ae80 2023-10-14T18:19:39.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d0c011d20 con 0x7f5d1415ae80 2023-10-14T18:19:39.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.995+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d141e4ef0 con 0x7f5d1415ae80 2023-10-14T18:19:39.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.996+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d141e56f0 con 0x7f5d1415ae80 2023-10-14T18:19:39.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.997+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d141e5410 con 0x7f5d1415ae80 2023-10-14T18:19:39.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.998+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5d0c01b460 con 0x7f5d1415ae80 2023-10-14T18:19:39.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.999+0000 7f5d0bfff640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 0x7f5cf406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.999+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5d0c0898a0 con 0x7f5d1415ae80 2023-10-14T18:19:39.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:38.999+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 0x7f5cf406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.000+0000 7f5d12ffd640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 0x7f5cf406d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d04035000 tx=0x7f5d04030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:39.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.003+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5d141e5410 con 0x7f5d1415ae80 2023-10-14T18:19:39.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.196+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d141e5410 con 0x7f5d1415ae80 2023-10-14T18:19:39.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.199+0000 7f5d0bfff640 1 -- 172.21.15.29:0/1850362059 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+16701 (secure 0 0 0) 0x7f5d141e5410 con 0x7f5d1415ae80 2023-10-14T18:19:39.201 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:39.201 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":271,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:38.940397+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 msgr2=0x7f5cf406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 0x7f5cf406d5a0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5d04035000 tx=0x7f5d04030000 comp rx=0 tx=0).stop 2023-10-14T18:19:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 msgr2=0x7f5d1414ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 secure :-1 s=READY pgs=3131 cs=0 l=1 rev1=1 crypto rx=0x7f5d0c009fd0 tx=0x7f5d0c00b8e0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 shutdown_connections 2023-10-14T18:19:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5cf406b0e0 0x7f5cf406d5a0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5d1415ae80 0x7f5d1414ae00 unknown :-1 s=CLOSED pgs=3131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5d1415aaa0 0x7f5d1414a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 --2- 172.21.15.29:0/1850362059 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5d140ce390 0x7f5d140b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 >> 172.21.15.29:0/1850362059 conn(0x7f5d140cb410 msgr2=0x7f5d140cc4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:39.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 shutdown_connections 2023-10-14T18:19:39.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.202+0000 7f5d19851640 1 -- 172.21.15.29:0/1850362059 wait complete. 2023-10-14T18:19:39.222 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2023-10-14T18:19:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.446+0000 7f61bc77c640 1 Processor -- start 2023-10-14T18:19:39.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.446+0000 7f61bc77c640 1 -- start start 2023-10-14T18:19:39.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.448+0000 7f61bc77c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f61b415ad70 0x7f61b4069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.448+0000 7f61bc77c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.448+0000 7f61bc77c640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61b40cb030 con 0x7f61b40ca3b0 2023-10-14T18:19:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.448+0000 7f61bc77c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61b40c91c0 con 0x7f61b415ad70 2023-10-14T18:19:39.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.448+0000 7f61bc77c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40c94a0 con 0x7f61b40c9ca0 2023-10-14T18:19:39.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:60542/0 (socket says 172.21.15.29:60542) 2023-10-14T18:19:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61ba518640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f61b415ad70 0x7f61b4069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 -- 172.21.15.29:0/3752669776 learned_addr learned my addr 172.21.15.29:0/3752669776 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 -- 172.21.15.29:0/3752669776 >> v2:172.21.15.186:3300/0 conn(0x7f61b415ad70 msgr2=0x7f61b4069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:39.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 --2- 172.21.15.29:0/3752669776 >> v2:172.21.15.186:3300/0 conn(0x7f61b415ad70 0x7f61b4069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 -- 172.21.15.29:0/3752669776 >> v1:172.21.15.186:6789/0 conn(0x7f61b40ca3b0 legacy=0x7f61b40ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:39.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.449+0000 7f61b9d17640 1 -- 172.21.15.29:0/3752669776 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61b40c9780 con 0x7f61b40c9ca0 2023-10-14T18:19:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.451+0000 7f61b9d17640 1 --2- 172.21.15.29:0/3752669776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009870 tx=0x7f61a402b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af604db2f99bcfd3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.452+0000 7f61b9516640 1 -- 172.21.15.29:0/3752669776 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4031640 con 0x7f61b40c9ca0 2023-10-14T18:19:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.452+0000 7f61b9516640 1 -- 172.21.15.29:0/3752669776 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a40317a0 con 0x7f61b40c9ca0 2023-10-14T18:19:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.452+0000 7f61b9516640 1 -- 172.21.15.29:0/3752669776 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a4031b30 con 0x7f61b40c9ca0 2023-10-14T18:19:39.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.452+0000 7f61bc77c640 1 -- 172.21.15.29:0/3752669776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 msgr2=0x7f61b406a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.452+0000 7f61bc77c640 1 --2- 172.21.15.29:0/3752669776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f61a4009870 tx=0x7f61a402b8f0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 -- 172.21.15.29:0/3752669776 shutdown_connections 2023-10-14T18:19:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 --2- 172.21.15.29:0/3752669776 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b40c9ca0 0x7f61b406a2e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 --2- 172.21.15.29:0/3752669776 >> v2:172.21.15.186:3300/0 conn(0x7f61b415ad70 0x7f61b4069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 -- 172.21.15.29:0/3752669776 >> 172.21.15.29:0/3752669776 conn(0x7f61b40ce9b0 msgr2=0x7f61b40cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 -- 172.21.15.29:0/3752669776 shutdown_connections 2023-10-14T18:19:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.453+0000 7f61bc77c640 1 -- 172.21.15.29:0/3752669776 wait complete. 2023-10-14T18:19:39.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.455+0000 7f61bc77c640 1 Processor -- start 2023-10-14T18:19:39.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.455+0000 7f61bc77c640 1 -- start start 2023-10-14T18:19:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61b40c9ca0 0x7f61b40b11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b415ad70 0x7f61b41593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b1e20 con 0x7f61b40ca3b0 2023-10-14T18:19:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40c4cf0 con 0x7f61b40c9ca0 2023-10-14T18:19:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bc77c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61b40b1860 con 0x7f61b415ad70 2023-10-14T18:19:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61b9d17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61b9d17640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51226/0 (socket says 172.21.15.29:51226) 2023-10-14T18:19:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61b9d17640 1 -- 172.21.15.29:0/4025210346 learned_addr learned my addr 172.21.15.29:0/4025210346 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61ba518640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61b40c9ca0 0x7f61b40b11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.457+0000 7f61bad19640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b415ad70 0x7f61b41593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 -- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b415ad70 msgr2=0x7f61b41593d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b415ad70 0x7f61b41593d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 -- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61b40c9ca0 msgr2=0x7f61b40b11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61b40c9ca0 0x7f61b40b11f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 -- 172.21.15.29:0/4025210346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a40094d0 con 0x7f61b40ca3b0 2023-10-14T18:19:39.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61b9d17640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.458+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a403a930 con 0x7f61b40ca3b0 2023-10-14T18:19:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.459+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b40b1b40 con 0x7f61b40ca3b0 2023-10-14T18:19:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.459+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61a403aa90 con 0x7f61b40ca3b0 2023-10-14T18:19:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.459+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61a403ae40 con 0x7f61b40ca3b0 2023-10-14T18:19:39.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.459+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b40cda10 con 0x7f61b40ca3b0 2023-10-14T18:19:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.460+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b404e160 con 0x7f61b40ca3b0 2023-10-14T18:19:39.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.461+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f61a402f050 con 0x7f61b40ca3b0 2023-10-14T18:19:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.462+0000 7f61aaffd640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 0x7f619806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.462+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f61a40ab9d0 con 0x7f61b40ca3b0 2023-10-14T18:19:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.462+0000 7f61ba518640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 0x7f619806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:39.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.463+0000 7f61ba518640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 0x7f619806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009a30 tx=0x7f61b0008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:39.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.466+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f61a4076580 con 0x7f61b40ca3b0 2023-10-14T18:19:39.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:39.691+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f61b40736e0 con 0x7f61b40ca3b0 2023-10-14T18:19:40.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.295+0000 7f61aaffd640 1 -- 172.21.15.29:0/4025210346 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v138) v1 ==== 118+0+0 (secure 0 0 0) 0x7f61a4079ba0 con 0x7f61b40ca3b0 2023-10-14T18:19:40.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 msgr2=0x7f619806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:40.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 0x7f619806d570 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f61b0009a30 tx=0x7f61b0008040 comp rx=0 tx=0).stop 2023-10-14T18:19:40.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 msgr2=0x7f61b40cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:40.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 secure :-1 s=READY pgs=3132 cs=0 l=1 rev1=1 crypto rx=0x7f61a4000c00 tx=0x7f61a40046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 shutdown_connections 2023-10-14T18:19:40.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f619806b0b0 0x7f619806d570 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f61b415ad70 0x7f61b41593d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f61b40ca3b0 0x7f61b40cdf30 unknown :-1 s=CLOSED pgs=3132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 --2- 172.21.15.29:0/4025210346 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f61b40c9ca0 0x7f61b40b11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.299+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 >> 172.21.15.29:0/4025210346 conn(0x7f61b40ce9b0 msgr2=0x7f61b40ced90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:40.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.300+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 shutdown_connections 2023-10-14T18:19:40.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.300+0000 7f61bc77c640 1 -- 172.21.15.29:0/4025210346 wait complete. 2023-10-14T18:19:40.323 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2023-10-14T18:19:40.323 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2023-10-14T18:19:40.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.555+0000 7fa460cdf640 1 Processor -- start 2023-10-14T18:19:40.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.556+0000 7fa460cdf640 1 -- start start 2023-10-14T18:19:40.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa45c15cd70 0x7fa45c0d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 0x7fa45c0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45c15d150 con 0x7fa45c15cd70 2023-10-14T18:19:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45c0d1180 con 0x7fa45c0b8480 2023-10-14T18:19:40.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.557+0000 7fa460cdf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c0d1460 con 0x7fa45c0d0be0 2023-10-14T18:19:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa45c15cd70 0x7fa45c0d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fa45c15cd70 0x7fa45c0d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51242/0 (socket says 172.21.15.29:51242) 2023-10-14T18:19:40.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 -- 172.21.15.29:0/1792356448 learned_addr learned my addr 172.21.15.29:0/1792356448 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:40.561 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45affd640 1 --2- 172.21.15.29:0/1792356448 >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 0x7fa45c0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 -- 172.21.15.29:0/1792356448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 msgr2=0x7fa45c0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 -- 172.21.15.29:0/1792356448 >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 msgr2=0x7fa45c0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1792356448 >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 0x7fa45c0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45b7fe640 1 -- 172.21.15.29:0/1792356448 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c0d1740 con 0x7fa45c15cd70 2023-10-14T18:19:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.558+0000 7fa45affd640 1 --2- 172.21.15.29:0/1792356448 >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 0x7fa45c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.559+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c0d2220 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fa45c066a00 tx=0x7fa44c02b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec09ba5d35313a22 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:40.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa45a7fc640 1 -- 172.21.15.29:0/1792356448 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c031860 con 0x7fa45c15cd70 2023-10-14T18:19:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa45a7fc640 1 -- 172.21.15.29:0/1792356448 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa44c0319c0 con 0x7fa45c15cd70 2023-10-14T18:19:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa45a7fc640 1 -- 172.21.15.29:0/1792356448 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa44c031d50 con 0x7fa45c15cd70 2023-10-14T18:19:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 -- 172.21.15.29:0/1792356448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 msgr2=0x7fa45c0d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c0d2220 secure :-1 s=READY pgs=3133 cs=0 l=1 rev1=1 crypto rx=0x7fa45c066a00 tx=0x7fa44c02b900 comp rx=0 tx=0).stop 2023-10-14T18:19:40.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 -- 172.21.15.29:0/1792356448 shutdown_connections 2023-10-14T18:19:40.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c0cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1792356448 >> v2:172.21.15.186:3300/0 conn(0x7fa45c0b8480 0x7fa45c0cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1792356448 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c0d2220 unknown :-1 s=CLOSED pgs=3133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.560+0000 7fa460cdf640 1 -- 172.21.15.29:0/1792356448 >> 172.21.15.29:0/1792356448 conn(0x7fa45c0cf570 msgr2=0x7fa45c0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:40.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.561+0000 7fa460cdf640 1 -- 172.21.15.29:0/1792356448 shutdown_connections 2023-10-14T18:19:40.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.561+0000 7fa460cdf640 1 -- 172.21.15.29:0/1792356448 wait complete. 2023-10-14T18:19:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.562+0000 7fa460cdf640 1 Processor -- start 2023-10-14T18:19:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.563+0000 7fa460cdf640 1 -- start start 2023-10-14T18:19:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.563+0000 7fa460cdf640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c0b8480 0x7fa45c150170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.563+0000 7fa460cdf640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c15d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa460cdf640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa460cdf640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c15e580 con 0x7fa45c15cd70 2023-10-14T18:19:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa460cdf640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c15dc10 con 0x7fa45c0b8480 2023-10-14T18:19:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa460cdf640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45c15def0 con 0x7fa45c0d0be0 2023-10-14T18:19:40.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:51258/0 (socket says 172.21.15.29:51258) 2023-10-14T18:19:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 -- 172.21.15.29:0/1195109297 learned_addr learned my addr 172.21.15.29:0/1195109297 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c0b8480 0x7fa45c150170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45affd640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c15d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 -- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 msgr2=0x7fa45c15d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c15d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 -- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c0b8480 msgr2=0x7fa45c150170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c0b8480 0x7fa45c150170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.564+0000 7fa45bfff640 1 -- 172.21.15.29:0/1195109297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa44c0094d0 con 0x7fa45c15cd70 2023-10-14T18:19:40.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa45bfff640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fa454007d90 tx=0x7fa45400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454014420 con 0x7fa45c15cd70 2023-10-14T18:19:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4540145a0 con 0x7fa45c15cd70 2023-10-14T18:19:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa454014950 con 0x7fa45c15cd70 2023-10-14T18:19:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45c15e820 con 0x7fa45c15cd70 2023-10-14T18:19:40.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.565+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45c1506b0 con 0x7fa45c15cd70 2023-10-14T18:19:40.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.566+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa45c15ea10 con 0x7fa45c15cd70 2023-10-14T18:19:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.567+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fa454014ab0 con 0x7fa45c15cd70 2023-10-14T18:19:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.568+0000 7fa443fff640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 0x7fa44806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.568+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa45408a350 con 0x7fa45c15cd70 2023-10-14T18:19:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.569+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 0x7fa44806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:40.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.570+0000 7fa45b7fe640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 0x7fa44806d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa44c036000 tx=0x7fa44c03c040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:40.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.573+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa454055050 con 0x7fa45c15cd70 2023-10-14T18:19:40.807 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:40.806+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) v1 -- 0x7fa45c04e9b0 con 0x7fa45c15cd70 2023-10-14T18:19:41.300 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.299+0000 7fa443fff640 1 -- 172.21.15.29:0/1195109297 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v139) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa454058670 con 0x7fa45c15cd70 2023-10-14T18:19:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 msgr2=0x7fa44806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 0x7fa44806d570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa44c036000 tx=0x7fa44c03c040 comp rx=0 tx=0).stop 2023-10-14T18:19:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 msgr2=0x7fa45c15a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 secure :-1 s=READY pgs=3134 cs=0 l=1 rev1=1 crypto rx=0x7fa454007d90 tx=0x7fa45400e4b0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 shutdown_connections 2023-10-14T18:19:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa44806b0b0 0x7fa44806d570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa45c15cd70 0x7fa45c15a5b0 unknown :-1 s=CLOSED pgs=3134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa45c0d0be0 0x7fa45c15d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.302+0000 7fa460cdf640 1 --2- 172.21.15.29:0/1195109297 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45c0b8480 0x7fa45c150170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.303+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 >> 172.21.15.29:0/1195109297 conn(0x7fa45c0cf570 msgr2=0x7fa45c0ce6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:41.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.303+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 shutdown_connections 2023-10-14T18:19:41.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.303+0000 7fa460cdf640 1 -- 172.21.15.29:0/1195109297 wait complete. 2023-10-14T18:19:41.326 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2023-10-14T18:19:41.327 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:41.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.555+0000 7f8336fef640 1 Processor -- start 2023-10-14T18:19:41.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.556+0000 7f8336fef640 1 -- start start 2023-10-14T18:19:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.557+0000 7f8336fef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f83300b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.557+0000 7f8336fef640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833015d150 con 0x7f83300b8480 2023-10-14T18:19:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.557+0000 7f8336fef640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83300d1180 con 0x7f833015cd70 2023-10-14T18:19:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.557+0000 7f8336fef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83300d1460 con 0x7f83300cded0 2023-10-14T18:19:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.558+0000 7f83357ec640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f83300b8480 0x7f83300cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57858/0 (socket says 172.21.15.29:57858) 2023-10-14T18:19:41.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.558+0000 7f83357ec640 1 -- 172.21.15.29:0/2351010922 learned_addr learned my addr 172.21.15.29:0/2351010922 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.558+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380106719 0 0) 0x7f833015d150 con 0x7f83300b8480 2023-10-14T18:19:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.558+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f831c003520 con 0x7f83300b8480 2023-10-14T18:19:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.558+0000 7f83367ee640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f83300b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879533823 0 0) 0x7f831c003520 con 0x7f83300b8480 2023-10-14T18:19:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 msgr2=0x7f83300b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f83300b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 >> v1:172.21.15.186:6789/0 conn(0x7f833015cd70 legacy=0x7f83300d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83300d1740 con 0x7f83300b8480 2023-10-14T18:19:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f8320001690 con 0x7f83300b8480 2023-10-14T18:19:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 >> v1:172.21.15.29:6789/0 conn(0x7f83300b8480 legacy=0x7f83300cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f831c0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f831c003f70 0x7f831c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300d1460 con 0x7f831c003f70 2023-10-14T18:19:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f831c00bac0 con 0x7f831c004d00 2023-10-14T18:19:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f831c00bc80 con 0x7f83300cded0 2023-10-14T18:19:41.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f83357ec640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f831c003f70 0x7f831c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.559+0000 7f8335fed640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f831c0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83367ee640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83357ec640 1 -- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 msgr2=0x7f831c0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:41.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83357ec640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f831c0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83357ec640 1 -- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831c004d00 msgr2=0x7f831c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83357ec640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83357ec640 1 -- 172.21.15.29:0/2351010922 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f831c003520 con 0x7f831c003f70 2023-10-14T18:19:41.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f83367ee640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f831c004d00 0x7f831c005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:41.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.560+0000 7f8335fed640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83300cded0 0x7f831c0039a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:41.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.561+0000 7f83357ec640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f831c003f70 0x7f831c0047c0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f832000c160 tx=0x7f831c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89ba13d607f296ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:41.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.561+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320005ad0 con 0x7f831c003f70 2023-10-14T18:19:41.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.561+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320005c30 con 0x7f831c003f70 2023-10-14T18:19:41.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.561+0000 7f8334feb640 1 -- 172.21.15.29:0/2351010922 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320007070 con 0x7f831c003f70 2023-10-14T18:19:41.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 -- 172.21.15.29:0/2351010922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f831c003f70 msgr2=0x7f831c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 --2- 172.21.15.29:0/2351010922 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f831c003f70 0x7f831c0047c0 secure :-1 s=READY pgs=3136 cs=0 l=1 rev1=1 crypto rx=0x7f832000c160 tx=0x7f831c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f83367ee640 1 -- 172.21.15.29:0/2351010922 reap_dead start 2023-10-14T18:19:41.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 -- 172.21.15.29:0/2351010922 shutdown_connections 2023-10-14T18:19:41.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 -- 172.21.15.29:0/2351010922 >> 172.21.15.29:0/2351010922 conn(0x7f83300cf570 msgr2=0x7f83300cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:41.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 -- 172.21.15.29:0/2351010922 shutdown_connections 2023-10-14T18:19:41.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.562+0000 7f8336fef640 1 -- 172.21.15.29:0/2351010922 wait complete. 2023-10-14T18:19:41.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.564+0000 7f8336fef640 1 Processor -- start 2023-10-14T18:19:41.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.564+0000 7f8336fef640 1 -- start start 2023-10-14T18:19:41.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 0x7f833015e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83300d2a80 0x7f83300d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83300d3440 con 0x7f833015d340 2023-10-14T18:19:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f833015faf0 con 0x7f83300d2a80 2023-10-14T18:19:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8336fef640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f833015eee0 con 0x7f831c003f70 2023-10-14T18:19:41.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43384/0 (socket says 172.21.15.29:43384) 2023-10-14T18:19:41.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 -- 172.21.15.29:0/1529232735 learned_addr learned my addr 172.21.15.29:0/1529232735 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:41.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8335fed640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 0x7f833015e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 -- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 msgr2=0x7f833015e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 0x7f833015e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 -- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83300d2a80 msgr2=0x7f83300d2f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83300d2a80 0x7f83300d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f83357ec640 1 -- 172.21.15.29:0/1529232735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833015f1c0 con 0x7f833015d340 2023-10-14T18:19:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.565+0000 7f8335fed640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 0x7f833015e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f83357ec640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f832002c2a0 tx=0x7f8320001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8320007480 con 0x7f833015d340 2023-10-14T18:19:41.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8320007600 con 0x7f833015d340 2023-10-14T18:19:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f832002c940 con 0x7f833015d340 2023-10-14T18:19:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f833015f4a0 con 0x7f833015d340 2023-10-14T18:19:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.566+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f833015ac20 con 0x7f833015d340 2023-10-14T18:19:41.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.567+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f833004e9b0 con 0x7f833015d340 2023-10-14T18:19:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.568+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f8320032070 con 0x7f833015d340 2023-10-14T18:19:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.569+0000 7f83267fc640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 0x7f831406d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.569+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f83200ab070 con 0x7f833015d340 2023-10-14T18:19:41.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.570+0000 7f8335fed640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 0x7f831406d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.570+0000 7f8335fed640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 0x7f831406d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f832c001630 tx=0x7f832c009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:41.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.574+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83200ddfe0 con 0x7f833015d340 2023-10-14T18:19:41.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.816+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f833015fc30 con 0x7f833015d340 2023-10-14T18:19:41.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.817+0000 7f83267fc640 1 -- 172.21.15.29:0/1529232735 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f8320075ca0 con 0x7f833015d340 2023-10-14T18:19:41.818 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:41.819 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:19:41.819 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:19:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 msgr2=0x7f831406d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 0x7f831406d570 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f832c001630 tx=0x7f832c009040 comp rx=0 tx=0).stop 2023-10-14T18:19:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 msgr2=0x7f833015a6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:41.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 secure :-1 s=READY pgs=3137 cs=0 l=1 rev1=1 crypto rx=0x7f832002c2a0 tx=0x7f8320001950 comp rx=0 tx=0).stop 2023-10-14T18:19:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 shutdown_connections 2023-10-14T18:19:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f831406b0b0 0x7f831406d570 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83300d2a80 0x7f83300d2f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f833015d340 0x7f833015a6e0 unknown :-1 s=CLOSED pgs=3137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 --2- 172.21.15.29:0/1529232735 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f831c003f70 0x7f833015e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 >> 172.21.15.29:0/1529232735 conn(0x7f83300cf570 msgr2=0x7f833015bfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 shutdown_connections 2023-10-14T18:19:41.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:41.820+0000 7f8336fef640 1 -- 172.21.15.29:0/1529232735 wait complete. 2023-10-14T18:19:41.840 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.066+0000 7f03da221640 1 Processor -- start 2023-10-14T18:19:42.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.067+0000 7f03da221640 1 -- start start 2023-10-14T18:19:42.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.068+0000 7f03da221640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03d40b8480 0x7f03d40cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.068+0000 7f03da221640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f03d40d0be0 0x7f03d40cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.068+0000 7f03da221640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03d415d150 con 0x7f03d415cd70 2023-10-14T18:19:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.068+0000 7f03da221640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40d1180 con 0x7f03d40b8480 2023-10-14T18:19:42.070 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.068+0000 7f03da221640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03d40d1460 con 0x7f03d40d0be0 2023-10-14T18:19:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03d40b8480 0x7f03d40cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f03d40b8480 0x7f03d40cd7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43390/0 (socket says 172.21.15.29:43390) 2023-10-14T18:19:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 -- 172.21.15.29:0/968384512 learned_addr learned my addr 172.21.15.29:0/968384512 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:42.071 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d9a20640 1 --2- 172.21.15.29:0/968384512 >> v2:172.21.15.186:3300/0 conn(0x7f03d40d0be0 0x7f03d40cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 -- 172.21.15.29:0/968384512 >> v2:172.21.15.186:3300/0 conn(0x7f03d40d0be0 msgr2=0x7f03d40cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 --2- 172.21.15.29:0/968384512 >> v2:172.21.15.186:3300/0 conn(0x7f03d40d0be0 0x7f03d40cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 -- 172.21.15.29:0/968384512 >> v1:172.21.15.186:6789/0 conn(0x7f03d415cd70 legacy=0x7f03d40d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.069+0000 7f03d8a1e640 1 -- 172.21.15.29:0/968384512 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d40d1740 con 0x7f03d40b8480 2023-10-14T18:19:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.071+0000 7f03d8a1e640 1 --2- 172.21.15.29:0/968384512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d40b8480 0x7f03d40cd7c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b8910 tx=0x7f03c402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99aeb488d2e4b3c9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.071+0000 7f03cbfff640 1 -- 172.21.15.29:0/968384512 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4031870 con 0x7f03d40b8480 2023-10-14T18:19:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.071+0000 7f03cbfff640 1 -- 172.21.15.29:0/968384512 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03c40319d0 con 0x7f03d40b8480 2023-10-14T18:19:42.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.071+0000 7f03cbfff640 1 -- 172.21.15.29:0/968384512 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c4031d60 con 0x7f03d40b8480 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 -- 172.21.15.29:0/968384512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d40b8480 msgr2=0x7f03d40cd7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 --2- 172.21.15.29:0/968384512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d40b8480 0x7f03d40cd7c0 secure :-1 s=READY pgs=3138 cs=0 l=1 rev1=1 crypto rx=0x7f03d40b8910 tx=0x7f03c402b910 comp rx=0 tx=0).stop 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 -- 172.21.15.29:0/968384512 shutdown_connections 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 --2- 172.21.15.29:0/968384512 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d40b8480 0x7f03d40cd7c0 unknown :-1 s=CLOSED pgs=3138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 --2- 172.21.15.29:0/968384512 >> v2:172.21.15.186:3300/0 conn(0x7f03d40d0be0 0x7f03d40cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 -- 172.21.15.29:0/968384512 >> 172.21.15.29:0/968384512 conn(0x7f03d40cf570 msgr2=0x7f03d40cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 -- 172.21.15.29:0/968384512 shutdown_connections 2023-10-14T18:19:42.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.072+0000 7f03da221640 1 -- 172.21.15.29:0/968384512 wait complete. 2023-10-14T18:19:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.074+0000 7f03da221640 1 Processor -- start 2023-10-14T18:19:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.074+0000 7f03da221640 1 -- start start 2023-10-14T18:19:42.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 0x7f03d40ca5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 0x7f03d414f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d40cb990 con 0x7f03d415cd70 2023-10-14T18:19:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d415db30 con 0x7f03d40d0be0 2023-10-14T18:19:42.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03da221640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03d414f840 con 0x7f03d40b8480 2023-10-14T18:19:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03d9a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03d9a20640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43396/0 (socket says 172.21.15.29:43396) 2023-10-14T18:19:42.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.075+0000 7f03d9a20640 1 -- 172.21.15.29:0/3526998798 learned_addr learned my addr 172.21.15.29:0/3526998798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d8a1e640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 0x7f03d414f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d921f640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 0x7f03d40ca5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 -- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 msgr2=0x7f03d40ca5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 0x7f03d40ca5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 -- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 msgr2=0x7f03d414f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 0x7f03d414f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 -- 172.21.15.29:0/3526998798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03c40094d0 con 0x7f03d415cd70 2023-10-14T18:19:42.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d8a1e640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 0x7f03d414f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d921f640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 0x7f03d40ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.076+0000 7f03d9a20640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00efc0 tx=0x7f03cc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.077+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc009b10 con 0x7f03d415cd70 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.077+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03cc00f040 con 0x7f03d415cd70 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.077+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03cc0146c0 con 0x7f03d415cd70 2023-10-14T18:19:42.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.077+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d41540e0 con 0x7f03d415cd70 2023-10-14T18:19:42.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.077+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d415d4e0 con 0x7f03d415cd70 2023-10-14T18:19:42.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.078+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03d415d340 con 0x7f03d415cd70 2023-10-14T18:19:42.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.079+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f03cc014820 con 0x7f03d415cd70 2023-10-14T18:19:42.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.080+0000 7f03c9ffb640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 0x7f03b806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.080+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f03cc08a9b0 con 0x7f03d415cd70 2023-10-14T18:19:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.080+0000 7f03d921f640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 0x7f03b806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.081+0000 7f03d921f640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 0x7f03b806d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000f60 tx=0x7f03d0009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.085+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f03d415d340 con 0x7f03d415cd70 2023-10-14T18:19:42.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.284+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03d415d340 con 0x7f03d415cd70 2023-10-14T18:19:42.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.286+0000 7f03c9ffb640 1 -- 172.21.15.29:0/3526998798 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v271) v1 ==== 74+0+16701 (secure 0 0 0) 0x7f03d415d340 con 0x7f03d415cd70 2023-10-14T18:19:42.287 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:42.288 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":271,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:38.940397+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:42.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 msgr2=0x7f03b806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 0x7f03b806d570 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f03d0000f60 tx=0x7f03d0009040 comp rx=0 tx=0).stop 2023-10-14T18:19:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 msgr2=0x7f03d4153ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 secure :-1 s=READY pgs=3139 cs=0 l=1 rev1=1 crypto rx=0x7f03cc00efc0 tx=0x7f03cc00c490 comp rx=0 tx=0).stop 2023-10-14T18:19:42.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 shutdown_connections 2023-10-14T18:19:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f03b806b0b0 0x7f03b806d570 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f03d415cd70 0x7f03d4153ba0 unknown :-1 s=CLOSED pgs=3139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f03d40d0be0 0x7f03d414f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 --2- 172.21.15.29:0/3526998798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f03d40b8480 0x7f03d40ca5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.289+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 >> 172.21.15.29:0/3526998798 conn(0x7f03d40cf570 msgr2=0x7f03d40d2a80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.290+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 shutdown_connections 2023-10-14T18:19:42.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.290+0000 7f03da221640 1 -- 172.21.15.29:0/3526998798 wait complete. 2023-10-14T18:19:42.311 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2023-10-14T18:19:42.312 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:19:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.540+0000 7fab0f59e640 1 Processor -- start 2023-10-14T18:19:42.541 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.540+0000 7fab0f59e640 1 -- start start 2023-10-14T18:19:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.541+0000 7fab0f59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab1006c040 0x7fab1006c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.542 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.541+0000 7fab0f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 0x7fab1015e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.541+0000 7fab0f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1006c960 con 0x7fab1015ebc0 2023-10-14T18:19:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.541+0000 7fab0f59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1015b9d0 con 0x7fab1006c040 2023-10-14T18:19:42.543 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.541+0000 7fab0f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1015bcb0 con 0x7fab100ad970 2023-10-14T18:19:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab1006c040 0x7fab1006c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fab1006c040 0x7fab1006c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43400/0 (socket says 172.21.15.29:43400) 2023-10-14T18:19:42.544 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 -- 172.21.15.29:0/3374526835 learned_addr learned my addr 172.21.15.29:0/3374526835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab05d9b640 1 --2- 172.21.15.29:0/3374526835 >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 0x7fab1015e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 -- 172.21.15.29:0/3374526835 >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 msgr2=0x7fab1015e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 --2- 172.21.15.29:0/3374526835 >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 0x7fab1015e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 -- 172.21.15.29:0/3374526835 >> v1:172.21.15.186:6789/0 conn(0x7fab1015ebc0 legacy=0x7fab100cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.542+0000 7fab0e59c640 1 -- 172.21.15.29:0/3374526835 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1015bf90 con 0x7fab1006c040 2023-10-14T18:19:42.545 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.543+0000 7fab05d9b640 1 --2- 172.21.15.29:0/3374526835 >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 0x7fab1015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.544+0000 7fab0e59c640 1 --2- 172.21.15.29:0/3374526835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab1006c420 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7fab100cb060 tx=0x7fab0002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c60983991d1ab0da server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.544+0000 7fab0dd9b640 1 -- 172.21.15.29:0/3374526835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00030860 con 0x7fab1006c040 2023-10-14T18:19:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.544+0000 7fab0dd9b640 1 -- 172.21.15.29:0/3374526835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab000309c0 con 0x7fab1006c040 2023-10-14T18:19:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.544+0000 7fab0dd9b640 1 -- 172.21.15.29:0/3374526835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00030d50 con 0x7fab1006c040 2023-10-14T18:19:42.547 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 -- 172.21.15.29:0/3374526835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 msgr2=0x7fab1006c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 --2- 172.21.15.29:0/3374526835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab1006c420 secure :-1 s=READY pgs=3140 cs=0 l=1 rev1=1 crypto rx=0x7fab100cb060 tx=0x7fab0002b9a0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 -- 172.21.15.29:0/3374526835 shutdown_connections 2023-10-14T18:19:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 --2- 172.21.15.29:0/3374526835 >> v2:172.21.15.186:3300/0 conn(0x7fab100ad970 0x7fab1015e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 --2- 172.21.15.29:0/3374526835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab1006c420 unknown :-1 s=CLOSED pgs=3140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.548 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 -- 172.21.15.29:0/3374526835 >> 172.21.15.29:0/3374526835 conn(0x7fab100ce6a0 msgr2=0x7fab100ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 -- 172.21.15.29:0/3374526835 shutdown_connections 2023-10-14T18:19:42.549 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.545+0000 7fab0f59e640 1 -- 172.21.15.29:0/3374526835 wait complete. 2023-10-14T18:19:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.547+0000 7fab0f59e640 1 Processor -- start 2023-10-14T18:19:42.551 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.547+0000 7fab0f59e640 1 -- start start 2023-10-14T18:19:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab100ad970 0x7fab10158860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1015ebc0 0x7fab10158da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab100b00a0 con 0x7fab1006c040 2023-10-14T18:19:42.552 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab100b0380 con 0x7fab1015ebc0 2023-10-14T18:19:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab100b1440 con 0x7fab100ad970 2023-10-14T18:19:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0e59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43404/0 (socket says 172.21.15.29:43404) 2023-10-14T18:19:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0e59c640 1 -- 172.21.15.29:0/1894786675 learned_addr learned my addr 172.21.15.29:0/1894786675 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:42.553 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.548+0000 7fab0e59c640 1 -- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab100ad970 msgr2=0x7fab10158860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab05d9b640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab100ad970 0x7fab10158860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0ed9d640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1015ebc0 0x7fab10158da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0e59c640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab100ad970 0x7fab10158860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0e59c640 1 -- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1015ebc0 msgr2=0x7fab10158da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.554 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0e59c640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1015ebc0 0x7fab10158da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0e59c640 1 -- 172.21.15.29:0/1894786675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab000094d0 con 0x7fab1006c040 2023-10-14T18:19:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0e59c640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fab0002b9e0 tx=0x7fab00004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00039b60 con 0x7fab1006c040 2023-10-14T18:19:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00039ce0 con 0x7fab1006c040 2023-10-14T18:19:42.555 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab00043650 con 0x7fab1006c040 2023-10-14T18:19:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.549+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab100b1720 con 0x7fab1006c040 2023-10-14T18:19:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.550+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab100af300 con 0x7fab1006c040 2023-10-14T18:19:42.556 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.551+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1006b330 con 0x7fab1006c040 2023-10-14T18:19:42.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.552+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fab0004c3b0 con 0x7fab1006c040 2023-10-14T18:19:42.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.552+0000 7fab077fe640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 0x7faaf006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:42.557 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.553+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fab000ac180 con 0x7fab1006c040 2023-10-14T18:19:42.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.553+0000 7fab05d9b640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 0x7faaf006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:42.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.554+0000 7fab05d9b640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 0x7faaf006d5c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006fd0 tx=0x7faaf8008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:42.558 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.557+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fab00076eb0 con 0x7fab1006c040 2023-10-14T18:19:42.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.751+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 --> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab100b12c0 con 0x7faaf006b100 2023-10-14T18:19:42.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.780+0000 7fab077fe640 1 -- 172.21.15.29:0/1894786675 <== mgr.12471 v2:172.21.15.29:6832/1327833207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287548 (secure 0 0 0) 0x7faaf800e4c0 con 0x7faaf006b100 2023-10-14T18:19:42.785 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:42.787 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:19:42.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 msgr2=0x7faaf006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 0x7faaf006d5c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7faaf8006fd0 tx=0x7faaf8008040 comp rx=0 tx=0).stop 2023-10-14T18:19:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 msgr2=0x7fab10156800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 secure :-1 s=READY pgs=3141 cs=0 l=1 rev1=1 crypto rx=0x7fab0002b9e0 tx=0x7fab00004730 comp rx=0 tx=0).stop 2023-10-14T18:19:42.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 shutdown_connections 2023-10-14T18:19:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7faaf006b100 0x7faaf006d5c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fab1015ebc0 0x7fab10158da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fab100ad970 0x7fab10158860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 --2- 172.21.15.29:0/1894786675 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fab1006c040 0x7fab10156800 unknown :-1 s=CLOSED pgs=3141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 >> 172.21.15.29:0/1894786675 conn(0x7fab100ce6a0 msgr2=0x7fab1015a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 shutdown_connections 2023-10-14T18:19:42.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:42.786+0000 7fab0f59e640 1 -- 172.21.15.29:0/1894786675 wait complete. 2023-10-14T18:19:42.807 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2023-10-14T18:19:41.183121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":456404,"kb_used_data":27908,"kb_used_omap":12,"kb_used_meta":428339,"kb_avail":754518316,"statfs":{"total":773094113280,"available":772626755584,"internally_reserved":0,"allocated":28577792,"data_stored":14866604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":438619704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.290641"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957402+0000","last_change":"2023-10-14T18:19:37.962929+0000","last_active":"2023-10-14T18:19:38.957402+0000","last_peered":"2023-10-14T18:19:38.957402+0000","last_clean":"2023-10-14T18:19:38.957402+0000","last_became_active":"2023-10-14T18:19:37.962473+0000","last_became_peered":"2023-10-14T18:19:37.962473+0000","last_unstale":"2023-10-14T18:19:38.957402+0000","last_undegraded":"2023-10-14T18:19:38.957402+0000","last_fullsized":"2023-10-14T18:19:38.957402+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:43.021524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791381+0000","last_change":"2023-10-14T18:19:37.948637+0000","last_active":"2023-10-14T18:19:39.791381+0000","last_peered":"2023-10-14T18:19:39.791381+0000","last_clean":"2023-10-14T18:19:39.791381+0000","last_became_active":"2023-10-14T18:19:37.947456+0000","last_became_peered":"2023-10-14T18:19:37.947456+0000","last_unstale":"2023-10-14T18:19:39.791381+0000","last_undegraded":"2023-10-14T18:19:39.791381+0000","last_fullsized":"2023-10-14T18:19:39.791381+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:15.520747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791912+0000","last_change":"2023-10-14T18:19:37.955377+0000","last_active":"2023-10-14T18:19:39.791912+0000","last_peered":"2023-10-14T18:19:39.791912+0000","last_clean":"2023-10-14T18:19:39.791912+0000","last_became_active":"2023-10-14T18:19:37.954245+0000","last_became_peered":"2023-10-14T18:19:37.954245+0000","last_unstale":"2023-10-14T18:19:39.791912+0000","last_undegraded":"2023-10-14T18:19:39.791912+0000","last_fullsized":"2023-10-14T18:19:39.791912+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:22.095336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960905+0000","last_change":"2023-10-14T18:19:37.963130+0000","last_active":"2023-10-14T18:19:38.960905+0000","last_peered":"2023-10-14T18:19:38.960905+0000","last_clean":"2023-10-14T18:19:38.960905+0000","last_became_active":"2023-10-14T18:19:37.962846+0000","last_became_peered":"2023-10-14T18:19:37.962846+0000","last_unstale":"2023-10-14T18:19:38.960905+0000","last_undegraded":"2023-10-14T18:19:38.960905+0000","last_fullsized":"2023-10-14T18:19:38.960905+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:52.243028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791504+0000","last_change":"2023-10-14T18:19:37.947461+0000","last_active":"2023-10-14T18:19:39.791504+0000","last_peered":"2023-10-14T18:19:39.791504+0000","last_clean":"2023-10-14T18:19:39.791504+0000","last_became_active":"2023-10-14T18:19:37.947180+0000","last_became_peered":"2023-10-14T18:19:37.947180+0000","last_unstale":"2023-10-14T18:19:39.791504+0000","last_undegraded":"2023-10-14T18:19:39.791504+0000","last_fullsized":"2023-10-14T18:19:39.791504+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:29.926911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790798+0000","last_change":"2023-10-14T18:19:37.958732+0000","last_active":"2023-10-14T18:19:39.790798+0000","last_peered":"2023-10-14T18:19:39.790798+0000","last_clean":"2023-10-14T18:19:39.790798+0000","last_became_active":"2023-10-14T18:19:37.958438+0000","last_became_peered":"2023-10-14T18:19:37.958438+0000","last_unstale":"2023-10-14T18:19:39.790798+0000","last_undegraded":"2023-10-14T18:19:39.790798+0000","last_fullsized":"2023-10-14T18:19:39.790798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:47.792362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959996+0000","last_change":"2023-10-14T18:19:37.949222+0000","last_active":"2023-10-14T18:19:38.959996+0000","last_peered":"2023-10-14T18:19:38.959996+0000","last_clean":"2023-10-14T18:19:38.959996+0000","last_became_active":"2023-10-14T18:19:37.948769+0000","last_became_peered":"2023-10-14T18:19:37.948769+0000","last_unstale":"2023-10-14T18:19:38.959996+0000","last_undegraded":"2023-10-14T18:19:38.959996+0000","last_fullsized":"2023-10-14T18:19:38.959996+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:45.786253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959359+0000","last_change":"2023-10-14T18:19:37.960968+0000","last_active":"2023-10-14T18:19:38.959359+0000","last_peered":"2023-10-14T18:19:38.959359+0000","last_clean":"2023-10-14T18:19:38.959359+0000","last_became_active":"2023-10-14T18:19:37.957973+0000","last_became_peered":"2023-10-14T18:19:37.957973+0000","last_unstale":"2023-10-14T18:19:38.959359+0000","last_undegraded":"2023-10-14T18:19:38.959359+0000","last_fullsized":"2023-10-14T18:19:38.959359+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:08.822058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790464+0000","last_change":"2023-10-14T18:19:37.954101+0000","last_active":"2023-10-14T18:19:39.790464+0000","last_peered":"2023-10-14T18:19:39.790464+0000","last_clean":"2023-10-14T18:19:39.790464+0000","last_became_active":"2023-10-14T18:19:37.951660+0000","last_became_peered":"2023-10-14T18:19:37.951660+0000","last_unstale":"2023-10-14T18:19:39.790464+0000","last_undegraded":"2023-10-14T18:19:39.790464+0000","last_fullsized":"2023-10-14T18:19:39.790464+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:59.513183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960791+0000","last_change":"2023-10-14T18:19:37.951635+0000","last_active":"2023-10-14T18:19:38.960791+0000","last_peered":"2023-10-14T18:19:38.960791+0000","last_clean":"2023-10-14T18:19:38.960791+0000","last_became_active":"2023-10-14T18:19:37.951360+0000","last_became_peered":"2023-10-14T18:19:37.951360+0000","last_unstale":"2023-10-14T18:19:38.960791+0000","last_undegraded":"2023-10-14T18:19:38.960791+0000","last_fullsized":"2023-10-14T18:19:38.960791+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:59.598362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790823+0000","last_change":"2023-10-14T18:19:37.961834+0000","last_active":"2023-10-14T18:19:39.790823+0000","last_peered":"2023-10-14T18:19:39.790823+0000","last_clean":"2023-10-14T18:19:39.790823+0000","last_became_active":"2023-10-14T18:19:37.961547+0000","last_became_peered":"2023-10-14T18:19:37.961547+0000","last_unstale":"2023-10-14T18:19:39.790823+0000","last_undegraded":"2023-10-14T18:19:39.790823+0000","last_fullsized":"2023-10-14T18:19:39.790823+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:40:15.844626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956669+0000","last_change":"2023-10-14T18:19:37.958383+0000","last_active":"2023-10-14T18:19:38.956669+0000","last_peered":"2023-10-14T18:19:38.956669+0000","last_clean":"2023-10-14T18:19:38.956669+0000","last_became_active":"2023-10-14T18:19:37.956054+0000","last_became_peered":"2023-10-14T18:19:37.956054+0000","last_unstale":"2023-10-14T18:19:38.956669+0000","last_undegraded":"2023-10-14T18:19:38.956669+0000","last_fullsized":"2023-10-14T18:19:38.956669+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:59.513183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956976+0000","last_change":"2023-10-14T18:19:37.956359+0000","last_active":"2023-10-14T18:19:38.956976+0000","last_peered":"2023-10-14T18:19:38.956976+0000","last_clean":"2023-10-14T18:19:38.956976+0000","last_became_active":"2023-10-14T18:19:37.955889+0000","last_became_peered":"2023-10-14T18:19:37.955889+0000","last_unstale":"2023-10-14T18:19:38.956976+0000","last_undegraded":"2023-10-14T18:19:38.956976+0000","last_fullsized":"2023-10-14T18:19:38.956976+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:53.502432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960217+0000","last_change":"2023-10-14T18:19:37.949696+0000","last_active":"2023-10-14T18:19:38.960217+0000","last_peered":"2023-10-14T18:19:38.960217+0000","last_clean":"2023-10-14T18:19:38.960217+0000","last_became_active":"2023-10-14T18:19:37.949392+0000","last_became_peered":"2023-10-14T18:19:37.949392+0000","last_unstale":"2023-10-14T18:19:38.960217+0000","last_undegraded":"2023-10-14T18:19:38.960217+0000","last_fullsized":"2023-10-14T18:19:38.960217+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:04.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960350+0000","last_change":"2023-10-14T18:19:37.960510+0000","last_active":"2023-10-14T18:19:38.960350+0000","last_peered":"2023-10-14T18:19:38.960350+0000","last_clean":"2023-10-14T18:19:38.960350+0000","last_became_active":"2023-10-14T18:19:37.960214+0000","last_became_peered":"2023-10-14T18:19:37.960214+0000","last_unstale":"2023-10-14T18:19:38.960350+0000","last_undegraded":"2023-10-14T18:19:38.960350+0000","last_fullsized":"2023-10-14T18:19:38.960350+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:51.926483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959758+0000","last_change":"2023-10-14T18:19:37.946011+0000","last_active":"2023-10-14T18:19:38.959758+0000","last_peered":"2023-10-14T18:19:38.959758+0000","last_clean":"2023-10-14T18:19:38.959758+0000","last_became_active":"2023-10-14T18:19:37.945723+0000","last_became_peered":"2023-10-14T18:19:37.945723+0000","last_unstale":"2023-10-14T18:19:38.959758+0000","last_undegraded":"2023-10-14T18:19:38.959758+0000","last_fullsized":"2023-10-14T18:19:38.959758+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:23:41.423018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790643+0000","last_change":"2023-10-14T18:19:37.957318+0000","last_active":"2023-10-14T18:19:39.790643+0000","last_peered":"2023-10-14T18:19:39.790643+0000","last_clean":"2023-10-14T18:19:39.790643+0000","last_became_active":"2023-10-14T18:19:37.956995+0000","last_became_peered":"2023-10-14T18:19:37.956995+0000","last_unstale":"2023-10-14T18:19:39.790643+0000","last_undegraded":"2023-10-14T18:19:39.790643+0000","last_fullsized":"2023-10-14T18:19:39.790643+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:51.399343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789924+0000","last_change":"2023-10-14T18:19:37.947912+0000","last_active":"2023-10-14T18:19:39.789924+0000","last_peered":"2023-10-14T18:19:39.789924+0000","last_clean":"2023-10-14T18:19:39.789924+0000","last_became_active":"2023-10-14T18:19:37.947174+0000","last_became_peered":"2023-10-14T18:19:37.947174+0000","last_unstale":"2023-10-14T18:19:39.789924+0000","last_undegraded":"2023-10-14T18:19:39.789924+0000","last_fullsized":"2023-10-14T18:19:39.789924+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:18.238535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961713+0000","last_change":"2023-10-14T18:19:37.959011+0000","last_active":"2023-10-14T18:19:38.961713+0000","last_peered":"2023-10-14T18:19:38.961713+0000","last_clean":"2023-10-14T18:19:38.961713+0000","last_became_active":"2023-10-14T18:19:37.951416+0000","last_became_peered":"2023-10-14T18:19:37.951416+0000","last_unstale":"2023-10-14T18:19:38.961713+0000","last_undegraded":"2023-10-14T18:19:38.961713+0000","last_fullsized":"2023-10-14T18:19:38.961713+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:37.294239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961864+0000","last_change":"2023-10-14T18:19:37.956825+0000","last_active":"2023-10-14T18:19:38.961864+0000","last_peered":"2023-10-14T18:19:38.961864+0000","last_clean":"2023-10-14T18:19:38.961864+0000","last_became_active":"2023-10-14T18:19:37.950853+0000","last_became_peered":"2023-10-14T18:19:37.950853+0000","last_unstale":"2023-10-14T18:19:38.961864+0000","last_undegraded":"2023-10-14T18:19:38.961864+0000","last_fullsized":"2023-10-14T18:19:38.961864+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:44:20.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956794+0000","last_change":"2023-10-14T18:19:37.950729+0000","last_active":"2023-10-14T18:19:38.956794+0000","last_peered":"2023-10-14T18:19:38.956794+0000","last_clean":"2023-10-14T18:19:38.956794+0000","last_became_active":"2023-10-14T18:19:37.950072+0000","last_became_peered":"2023-10-14T18:19:37.950072+0000","last_unstale":"2023-10-14T18:19:38.956794+0000","last_undegraded":"2023-10-14T18:19:38.956794+0000","last_fullsized":"2023-10-14T18:19:38.956794+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:51.399343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790936+0000","last_change":"2023-10-14T18:19:37.943458+0000","last_active":"2023-10-14T18:19:39.790936+0000","last_peered":"2023-10-14T18:19:39.790936+0000","last_clean":"2023-10-14T18:19:39.790936+0000","last_became_active":"2023-10-14T18:19:37.941877+0000","last_became_peered":"2023-10-14T18:19:37.941877+0000","last_unstale":"2023-10-14T18:19:39.790936+0000","last_undegraded":"2023-10-14T18:19:39.790936+0000","last_fullsized":"2023-10-14T18:19:39.790936+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:13:47.235856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384624+0000","last_change":"2023-10-14T18:19:37.952373+0000","last_active":"2023-10-14T18:19:39.384624+0000","last_peered":"2023-10-14T18:19:39.384624+0000","last_clean":"2023-10-14T18:19:39.384624+0000","last_became_active":"2023-10-14T18:19:37.951244+0000","last_became_peered":"2023-10-14T18:19:37.951244+0000","last_unstale":"2023-10-14T18:19:39.384624+0000","last_undegraded":"2023-10-14T18:19:39.384624+0000","last_fullsized":"2023-10-14T18:19:39.384624+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:20.815226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789321+0000","last_change":"2023-10-14T18:19:37.945635+0000","last_active":"2023-10-14T18:19:39.789321+0000","last_peered":"2023-10-14T18:19:39.789321+0000","last_clean":"2023-10-14T18:19:39.789321+0000","last_became_active":"2023-10-14T18:19:37.945256+0000","last_became_peered":"2023-10-14T18:19:37.945256+0000","last_unstale":"2023-10-14T18:19:39.789321+0000","last_undegraded":"2023-10-14T18:19:39.789321+0000","last_fullsized":"2023-10-14T18:19:39.789321+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:58:47.232793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385367+0000","last_change":"2023-10-14T18:19:37.957037+0000","last_active":"2023-10-14T18:19:39.385367+0000","last_peered":"2023-10-14T18:19:39.385367+0000","last_clean":"2023-10-14T18:19:39.385367+0000","last_became_active":"2023-10-14T18:19:37.956689+0000","last_became_peered":"2023-10-14T18:19:37.956689+0000","last_unstale":"2023-10-14T18:19:39.385367+0000","last_undegraded":"2023-10-14T18:19:39.385367+0000","last_fullsized":"2023-10-14T18:19:39.385367+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:09.625588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957189+0000","last_change":"2023-10-14T18:19:37.954727+0000","last_active":"2023-10-14T18:19:38.957189+0000","last_peered":"2023-10-14T18:19:38.957189+0000","last_clean":"2023-10-14T18:19:38.957189+0000","last_became_active":"2023-10-14T18:19:37.954251+0000","last_became_peered":"2023-10-14T18:19:37.954251+0000","last_unstale":"2023-10-14T18:19:38.957189+0000","last_undegraded":"2023-10-14T18:19:38.957189+0000","last_fullsized":"2023-10-14T18:19:38.957189+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:10.274409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962035+0000","last_change":"2023-10-14T18:19:37.959240+0000","last_active":"2023-10-14T18:19:38.962035+0000","last_peered":"2023-10-14T18:19:38.962035+0000","last_clean":"2023-10-14T18:19:38.962035+0000","last_became_active":"2023-10-14T18:19:37.951267+0000","last_became_peered":"2023-10-14T18:19:37.951267+0000","last_unstale":"2023-10-14T18:19:38.962035+0000","last_undegraded":"2023-10-14T18:19:38.962035+0000","last_fullsized":"2023-10-14T18:19:38.962035+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:17.775956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961258+0000","last_change":"2023-10-14T18:19:37.954423+0000","last_active":"2023-10-14T18:19:38.961258+0000","last_peered":"2023-10-14T18:19:38.961258+0000","last_clean":"2023-10-14T18:19:38.961258+0000","last_became_active":"2023-10-14T18:19:37.954145+0000","last_became_peered":"2023-10-14T18:19:37.954145+0000","last_unstale":"2023-10-14T18:19:38.961258+0000","last_undegraded":"2023-10-14T18:19:38.961258+0000","last_fullsized":"2023-10-14T18:19:38.961258+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:06:46.249259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961168+0000","last_change":"2023-10-14T18:19:37.944004+0000","last_active":"2023-10-14T18:19:38.961168+0000","last_peered":"2023-10-14T18:19:38.961168+0000","last_clean":"2023-10-14T18:19:38.961168+0000","last_became_active":"2023-10-14T18:19:37.943727+0000","last_became_peered":"2023-10-14T18:19:37.943727+0000","last_unstale":"2023-10-14T18:19:38.961168+0000","last_undegraded":"2023-10-14T18:19:38.961168+0000","last_fullsized":"2023-10-14T18:19:38.961168+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:05:49.176233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962330+0000","last_change":"2023-10-14T18:19:37.943931+0000","last_active":"2023-10-14T18:19:38.962330+0000","last_peered":"2023-10-14T18:19:38.962330+0000","last_clean":"2023-10-14T18:19:38.962330+0000","last_became_active":"2023-10-14T18:19:37.943624+0000","last_became_peered":"2023-10-14T18:19:37.943624+0000","last_unstale":"2023-10-14T18:19:38.962330+0000","last_undegraded":"2023-10-14T18:19:38.962330+0000","last_fullsized":"2023-10-14T18:19:38.962330+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:13.244404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789387+0000","last_change":"2023-10-14T18:19:37.952289+0000","last_active":"2023-10-14T18:19:39.789387+0000","last_peered":"2023-10-14T18:19:39.789387+0000","last_clean":"2023-10-14T18:19:39.789387+0000","last_became_active":"2023-10-14T18:19:37.951590+0000","last_became_peered":"2023-10-14T18:19:37.951590+0000","last_unstale":"2023-10-14T18:19:39.789387+0000","last_undegraded":"2023-10-14T18:19:39.789387+0000","last_fullsized":"2023-10-14T18:19:39.789387+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:05.138450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957381+0000","last_change":"2023-10-14T18:19:37.955225+0000","last_active":"2023-10-14T18:19:38.957381+0000","last_peered":"2023-10-14T18:19:38.957381+0000","last_clean":"2023-10-14T18:19:38.957381+0000","last_became_active":"2023-10-14T18:19:37.954652+0000","last_became_peered":"2023-10-14T18:19:37.954652+0000","last_unstale":"2023-10-14T18:19:38.957381+0000","last_undegraded":"2023-10-14T18:19:38.957381+0000","last_fullsized":"2023-10-14T18:19:38.957381+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:18.041636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"270'4","reported_seq":16,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957417+0000","last_change":"2023-10-14T18:19:36.944847+0000","last_active":"2023-10-14T18:19:38.957417+0000","last_peered":"2023-10-14T18:19:38.957417+0000","last_clean":"2023-10-14T18:19:38.957417+0000","last_became_active":"2023-10-14T18:19:36.944479+0000","last_became_peered":"2023-10-14T18:19:36.944479+0000","last_unstale":"2023-10-14T18:19:38.957417+0000","last_undegraded":"2023-10-14T18:19:38.957417+0000","last_fullsized":"2023-10-14T18:19:38.957417+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:10.448923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791721+0000","last_change":"2023-10-14T18:19:37.962682+0000","last_active":"2023-10-14T18:19:39.791721+0000","last_peered":"2023-10-14T18:19:39.791721+0000","last_clean":"2023-10-14T18:19:39.791721+0000","last_became_active":"2023-10-14T18:19:37.962373+0000","last_became_peered":"2023-10-14T18:19:37.962373+0000","last_unstale":"2023-10-14T18:19:39.791721+0000","last_undegraded":"2023-10-14T18:19:39.791721+0000","last_fullsized":"2023-10-14T18:19:39.791721+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:54.269382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790267+0000","last_change":"2023-10-14T18:19:36.933531+0000","last_active":"2023-10-14T18:19:39.790267+0000","last_peered":"2023-10-14T18:19:39.790267+0000","last_clean":"2023-10-14T18:19:39.790267+0000","last_became_active":"2023-10-14T18:19:36.932553+0000","last_became_peered":"2023-10-14T18:19:36.932553+0000","last_unstale":"2023-10-14T18:19:39.790267+0000","last_undegraded":"2023-10-14T18:19:39.790267+0000","last_fullsized":"2023-10-14T18:19:39.790267+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:39.264522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791532+0000","last_change":"2023-10-14T18:19:37.949859+0000","last_active":"2023-10-14T18:19:39.791532+0000","last_peered":"2023-10-14T18:19:39.791532+0000","last_clean":"2023-10-14T18:19:39.791532+0000","last_became_active":"2023-10-14T18:19:37.949541+0000","last_became_peered":"2023-10-14T18:19:37.949541+0000","last_unstale":"2023-10-14T18:19:39.791532+0000","last_undegraded":"2023-10-14T18:19:39.791532+0000","last_fullsized":"2023-10-14T18:19:39.791532+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:37.633730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790155+0000","last_change":"2023-10-14T18:19:36.934071+0000","last_active":"2023-10-14T18:19:39.790155+0000","last_peered":"2023-10-14T18:19:39.790155+0000","last_clean":"2023-10-14T18:19:39.790155+0000","last_became_active":"2023-10-14T18:19:36.933772+0000","last_became_peered":"2023-10-14T18:19:36.933772+0000","last_unstale":"2023-10-14T18:19:39.790155+0000","last_undegraded":"2023-10-14T18:19:39.790155+0000","last_fullsized":"2023-10-14T18:19:39.790155+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:54.693227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791904+0000","last_change":"2023-10-14T18:19:37.959261+0000","last_active":"2023-10-14T18:19:39.791904+0000","last_peered":"2023-10-14T18:19:39.791904+0000","last_clean":"2023-10-14T18:19:39.791904+0000","last_became_active":"2023-10-14T18:19:37.958940+0000","last_became_peered":"2023-10-14T18:19:37.958940+0000","last_unstale":"2023-10-14T18:19:39.791904+0000","last_undegraded":"2023-10-14T18:19:39.791904+0000","last_fullsized":"2023-10-14T18:19:39.791904+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:21:52.386185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962157+0000","last_change":"2023-10-14T18:19:36.950224+0000","last_active":"2023-10-14T18:19:38.962157+0000","last_peered":"2023-10-14T18:19:38.962157+0000","last_clean":"2023-10-14T18:19:38.962157+0000","last_became_active":"2023-10-14T18:19:36.949907+0000","last_became_peered":"2023-10-14T18:19:36.949907+0000","last_unstale":"2023-10-14T18:19:38.962157+0000","last_undegraded":"2023-10-14T18:19:38.962157+0000","last_fullsized":"2023-10-14T18:19:38.962157+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:48.985755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385009+0000","last_change":"2023-10-14T18:19:37.956104+0000","last_active":"2023-10-14T18:19:39.385009+0000","last_peered":"2023-10-14T18:19:39.385009+0000","last_clean":"2023-10-14T18:19:39.385009+0000","last_became_active":"2023-10-14T18:19:37.955784+0000","last_became_peered":"2023-10-14T18:19:37.955784+0000","last_unstale":"2023-10-14T18:19:39.385009+0000","last_undegraded":"2023-10-14T18:19:39.385009+0000","last_fullsized":"2023-10-14T18:19:39.385009+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:41.758861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791354+0000","last_change":"2023-10-14T18:19:36.942983+0000","last_active":"2023-10-14T18:19:39.791354+0000","last_peered":"2023-10-14T18:19:39.791354+0000","last_clean":"2023-10-14T18:19:39.791354+0000","last_became_active":"2023-10-14T18:19:36.942671+0000","last_became_peered":"2023-10-14T18:19:36.942671+0000","last_unstale":"2023-10-14T18:19:39.791354+0000","last_undegraded":"2023-10-14T18:19:39.791354+0000","last_fullsized":"2023-10-14T18:19:39.791354+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:52:46.834539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961489+0000","last_change":"2023-10-14T18:19:37.958838+0000","last_active":"2023-10-14T18:19:38.961489+0000","last_peered":"2023-10-14T18:19:38.961489+0000","last_clean":"2023-10-14T18:19:38.961489+0000","last_became_active":"2023-10-14T18:19:37.957007+0000","last_became_peered":"2023-10-14T18:19:37.957007+0000","last_unstale":"2023-10-14T18:19:38.961489+0000","last_undegraded":"2023-10-14T18:19:38.961489+0000","last_fullsized":"2023-10-14T18:19:38.961489+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:08:24.162939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789620+0000","last_change":"2023-10-14T18:19:36.951111+0000","last_active":"2023-10-14T18:19:39.789620+0000","last_peered":"2023-10-14T18:19:39.789620+0000","last_clean":"2023-10-14T18:19:39.789620+0000","last_became_active":"2023-10-14T18:19:36.950255+0000","last_became_peered":"2023-10-14T18:19:36.950255+0000","last_unstale":"2023-10-14T18:19:39.789620+0000","last_undegraded":"2023-10-14T18:19:39.789620+0000","last_fullsized":"2023-10-14T18:19:39.789620+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:46.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789572+0000","last_change":"2023-10-14T18:19:37.949676+0000","last_active":"2023-10-14T18:19:39.789572+0000","last_peered":"2023-10-14T18:19:39.789572+0000","last_clean":"2023-10-14T18:19:39.789572+0000","last_became_active":"2023-10-14T18:19:37.949406+0000","last_became_peered":"2023-10-14T18:19:37.949406+0000","last_unstale":"2023-10-14T18:19:39.789572+0000","last_undegraded":"2023-10-14T18:19:39.789572+0000","last_fullsized":"2023-10-14T18:19:39.789572+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:11.466581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956309+0000","last_change":"2023-10-14T18:19:36.937220+0000","last_active":"2023-10-14T18:19:38.956309+0000","last_peered":"2023-10-14T18:19:38.956309+0000","last_clean":"2023-10-14T18:19:38.956309+0000","last_became_active":"2023-10-14T18:19:36.936897+0000","last_became_peered":"2023-10-14T18:19:36.936897+0000","last_unstale":"2023-10-14T18:19:38.956309+0000","last_undegraded":"2023-10-14T18:19:38.956309+0000","last_fullsized":"2023-10-14T18:19:38.956309+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:08.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789100+0000","last_change":"2023-10-14T18:19:37.950508+0000","last_active":"2023-10-14T18:19:39.789100+0000","last_peered":"2023-10-14T18:19:39.789100+0000","last_clean":"2023-10-14T18:19:39.789100+0000","last_became_active":"2023-10-14T18:19:37.947318+0000","last_became_peered":"2023-10-14T18:19:37.947318+0000","last_unstale":"2023-10-14T18:19:39.789100+0000","last_undegraded":"2023-10-14T18:19:39.789100+0000","last_fullsized":"2023-10-14T18:19:39.789100+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:16.780161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.955976+0000","last_change":"2023-10-14T18:19:36.945722+0000","last_active":"2023-10-14T18:19:38.955976+0000","last_peered":"2023-10-14T18:19:38.955976+0000","last_clean":"2023-10-14T18:19:38.955976+0000","last_became_active":"2023-10-14T18:19:36.945465+0000","last_became_peered":"2023-10-14T18:19:36.945465+0000","last_unstale":"2023-10-14T18:19:38.955976+0000","last_undegraded":"2023-10-14T18:19:38.955976+0000","last_fullsized":"2023-10-14T18:19:38.955976+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:46.378047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790284+0000","last_change":"2023-10-14T18:19:37.947828+0000","last_active":"2023-10-14T18:19:39.790284+0000","last_peered":"2023-10-14T18:19:39.790284+0000","last_clean":"2023-10-14T18:19:39.790284+0000","last_became_active":"2023-10-14T18:19:37.947548+0000","last_became_peered":"2023-10-14T18:19:37.947548+0000","last_unstale":"2023-10-14T18:19:39.790284+0000","last_undegraded":"2023-10-14T18:19:39.790284+0000","last_fullsized":"2023-10-14T18:19:39.790284+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:34.652584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961601+0000","last_change":"2023-10-14T18:19:36.933826+0000","last_active":"2023-10-14T18:19:38.961601+0000","last_peered":"2023-10-14T18:19:38.961601+0000","last_clean":"2023-10-14T18:19:38.961601+0000","last_became_active":"2023-10-14T18:19:36.932965+0000","last_became_peered":"2023-10-14T18:19:36.932965+0000","last_unstale":"2023-10-14T18:19:38.961601+0000","last_undegraded":"2023-10-14T18:19:38.961601+0000","last_fullsized":"2023-10-14T18:19:38.961601+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:42.003867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961393+0000","last_change":"2023-10-14T18:19:37.955290+0000","last_active":"2023-10-14T18:19:38.961393+0000","last_peered":"2023-10-14T18:19:38.961393+0000","last_clean":"2023-10-14T18:19:38.961393+0000","last_became_active":"2023-10-14T18:19:37.954984+0000","last_became_peered":"2023-10-14T18:19:37.954984+0000","last_unstale":"2023-10-14T18:19:38.961393+0000","last_undegraded":"2023-10-14T18:19:38.961393+0000","last_fullsized":"2023-10-14T18:19:38.961393+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:26:21.823073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791343+0000","last_change":"2023-10-14T18:19:36.944236+0000","last_active":"2023-10-14T18:19:39.791343+0000","last_peered":"2023-10-14T18:19:39.791343+0000","last_clean":"2023-10-14T18:19:39.791343+0000","last_became_active":"2023-10-14T18:19:36.943907+0000","last_became_peered":"2023-10-14T18:19:36.943907+0000","last_unstale":"2023-10-14T18:19:39.791343+0000","last_undegraded":"2023-10-14T18:19:39.791343+0000","last_fullsized":"2023-10-14T18:19:39.791343+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:02.542955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791536+0000","last_change":"2023-10-14T18:19:37.964487+0000","last_active":"2023-10-14T18:19:39.791536+0000","last_peered":"2023-10-14T18:19:39.791536+0000","last_clean":"2023-10-14T18:19:39.791536+0000","last_became_active":"2023-10-14T18:19:37.964197+0000","last_became_peered":"2023-10-14T18:19:37.964197+0000","last_unstale":"2023-10-14T18:19:39.791536+0000","last_undegraded":"2023-10-14T18:19:39.791536+0000","last_fullsized":"2023-10-14T18:19:39.791536+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:31.227271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"270'3","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790325+0000","last_change":"2023-10-14T18:19:36.943334+0000","last_active":"2023-10-14T18:19:39.790325+0000","last_peered":"2023-10-14T18:19:39.790325+0000","last_clean":"2023-10-14T18:19:39.790325+0000","last_became_active":"2023-10-14T18:19:36.942920+0000","last_became_peered":"2023-10-14T18:19:36.942920+0000","last_unstale":"2023-10-14T18:19:39.790325+0000","last_undegraded":"2023-10-14T18:19:39.790325+0000","last_fullsized":"2023-10-14T18:19:39.790325+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:08.817462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962091+0000","last_change":"2023-10-14T18:19:37.948963+0000","last_active":"2023-10-14T18:19:38.962091+0000","last_peered":"2023-10-14T18:19:38.962091+0000","last_clean":"2023-10-14T18:19:38.962091+0000","last_became_active":"2023-10-14T18:19:37.948601+0000","last_became_peered":"2023-10-14T18:19:37.948601+0000","last_unstale":"2023-10-14T18:19:38.962091+0000","last_undegraded":"2023-10-14T18:19:38.962091+0000","last_fullsized":"2023-10-14T18:19:38.962091+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:50.003412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"270'6","reported_seq":18,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791550+0000","last_change":"2023-10-14T18:19:36.944761+0000","last_active":"2023-10-14T18:19:39.791550+0000","last_peered":"2023-10-14T18:19:39.791550+0000","last_clean":"2023-10-14T18:19:39.791550+0000","last_became_active":"2023-10-14T18:19:36.944297+0000","last_became_peered":"2023-10-14T18:19:36.944297+0000","last_unstale":"2023-10-14T18:19:39.791550+0000","last_undegraded":"2023-10-14T18:19:39.791550+0000","last_fullsized":"2023-10-14T18:19:39.791550+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:55.193023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961042+0000","last_change":"2023-10-14T18:19:37.951816+0000","last_active":"2023-10-14T18:19:38.961042+0000","last_peered":"2023-10-14T18:19:38.961042+0000","last_clean":"2023-10-14T18:19:38.961042+0000","last_became_active":"2023-10-14T18:19:37.950686+0000","last_became_peered":"2023-10-14T18:19:37.950686+0000","last_unstale":"2023-10-14T18:19:38.961042+0000","last_undegraded":"2023-10-14T18:19:38.961042+0000","last_fullsized":"2023-10-14T18:19:38.961042+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:09.048028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385148+0000","last_change":"2023-10-14T18:19:36.939342+0000","last_active":"2023-10-14T18:19:39.385148+0000","last_peered":"2023-10-14T18:19:39.385148+0000","last_clean":"2023-10-14T18:19:39.385148+0000","last_became_active":"2023-10-14T18:19:36.939036+0000","last_became_peered":"2023-10-14T18:19:36.939036+0000","last_unstale":"2023-10-14T18:19:39.385148+0000","last_undegraded":"2023-10-14T18:19:39.385148+0000","last_fullsized":"2023-10-14T18:19:39.385148+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:51.786864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789279+0000","last_change":"2023-10-14T18:19:37.954331+0000","last_active":"2023-10-14T18:19:39.789279+0000","last_peered":"2023-10-14T18:19:39.789279+0000","last_clean":"2023-10-14T18:19:39.789279+0000","last_became_active":"2023-10-14T18:19:37.954038+0000","last_became_peered":"2023-10-14T18:19:37.954038+0000","last_unstale":"2023-10-14T18:19:39.789279+0000","last_undegraded":"2023-10-14T18:19:39.789279+0000","last_fullsized":"2023-10-14T18:19:39.789279+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:27:17.032998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384757+0000","last_change":"2023-10-14T18:19:36.935317+0000","last_active":"2023-10-14T18:19:39.384757+0000","last_peered":"2023-10-14T18:19:39.384757+0000","last_clean":"2023-10-14T18:19:39.384757+0000","last_became_active":"2023-10-14T18:19:36.934957+0000","last_became_peered":"2023-10-14T18:19:36.934957+0000","last_unstale":"2023-10-14T18:19:39.384757+0000","last_undegraded":"2023-10-14T18:19:39.384757+0000","last_fullsized":"2023-10-14T18:19:39.384757+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:18.035081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789773+0000","last_change":"2023-10-14T18:19:37.958897+0000","last_active":"2023-10-14T18:19:39.789773+0000","last_peered":"2023-10-14T18:19:39.789773+0000","last_clean":"2023-10-14T18:19:39.789773+0000","last_became_active":"2023-10-14T18:19:37.957884+0000","last_became_peered":"2023-10-14T18:19:37.957884+0000","last_unstale":"2023-10-14T18:19:39.789773+0000","last_undegraded":"2023-10-14T18:19:39.789773+0000","last_fullsized":"2023-10-14T18:19:39.789773+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:43.956572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960421+0000","last_change":"2023-10-14T18:19:36.946770+0000","last_active":"2023-10-14T18:19:38.960421+0000","last_peered":"2023-10-14T18:19:38.960421+0000","last_clean":"2023-10-14T18:19:38.960421+0000","last_became_active":"2023-10-14T18:19:36.945356+0000","last_became_peered":"2023-10-14T18:19:36.945356+0000","last_unstale":"2023-10-14T18:19:38.960421+0000","last_undegraded":"2023-10-14T18:19:38.960421+0000","last_fullsized":"2023-10-14T18:19:38.960421+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:09.256751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961942+0000","last_change":"2023-10-14T18:19:37.960753+0000","last_active":"2023-10-14T18:19:38.961942+0000","last_peered":"2023-10-14T18:19:38.961942+0000","last_clean":"2023-10-14T18:19:38.961942+0000","last_became_active":"2023-10-14T18:19:37.960057+0000","last_became_peered":"2023-10-14T18:19:37.960057+0000","last_unstale":"2023-10-14T18:19:38.961942+0000","last_undegraded":"2023-10-14T18:19:38.961942+0000","last_fullsized":"2023-10-14T18:19:38.961942+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:27:58.737280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"270'8","reported_seq":20,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384750+0000","last_change":"2023-10-14T18:19:36.945527+0000","last_active":"2023-10-14T18:19:39.384750+0000","last_peered":"2023-10-14T18:19:39.384750+0000","last_clean":"2023-10-14T18:19:39.384750+0000","last_became_active":"2023-10-14T18:19:36.945265+0000","last_became_peered":"2023-10-14T18:19:36.945265+0000","last_unstale":"2023-10-14T18:19:39.384750+0000","last_undegraded":"2023-10-14T18:19:39.384750+0000","last_fullsized":"2023-10-14T18:19:39.384750+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:02:26.845934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385414+0000","last_change":"2023-10-14T18:19:37.954344+0000","last_active":"2023-10-14T18:19:39.385414+0000","last_peered":"2023-10-14T18:19:39.385414+0000","last_clean":"2023-10-14T18:19:39.385414+0000","last_became_active":"2023-10-14T18:19:37.954064+0000","last_became_peered":"2023-10-14T18:19:37.954064+0000","last_unstale":"2023-10-14T18:19:39.385414+0000","last_undegraded":"2023-10-14T18:19:39.385414+0000","last_fullsized":"2023-10-14T18:19:39.385414+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:59:30.172265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961814+0000","last_change":"2023-10-14T18:19:36.941358+0000","last_active":"2023-10-14T18:19:38.961814+0000","last_peered":"2023-10-14T18:19:38.961814+0000","last_clean":"2023-10-14T18:19:38.961814+0000","last_became_active":"2023-10-14T18:19:36.941067+0000","last_became_peered":"2023-10-14T18:19:36.941067+0000","last_unstale":"2023-10-14T18:19:38.961814+0000","last_undegraded":"2023-10-14T18:19:38.961814+0000","last_fullsized":"2023-10-14T18:19:38.961814+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:48.163611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792128+0000","last_change":"2023-10-14T18:19:37.957490+0000","last_active":"2023-10-14T18:19:39.792128+0000","last_peered":"2023-10-14T18:19:39.792128+0000","last_clean":"2023-10-14T18:19:39.792128+0000","last_became_active":"2023-10-14T18:19:37.957199+0000","last_became_peered":"2023-10-14T18:19:37.957199+0000","last_unstale":"2023-10-14T18:19:39.792128+0000","last_undegraded":"2023-10-14T18:19:39.792128+0000","last_fullsized":"2023-10-14T18:19:39.792128+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:27.409684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791994+0000","last_change":"2023-10-14T18:19:36.937845+0000","last_active":"2023-10-14T18:19:39.791994+0000","last_peered":"2023-10-14T18:19:39.791994+0000","last_clean":"2023-10-14T18:19:39.791994+0000","last_became_active":"2023-10-14T18:19:36.937546+0000","last_became_peered":"2023-10-14T18:19:36.937546+0000","last_unstale":"2023-10-14T18:19:39.791994+0000","last_undegraded":"2023-10-14T18:19:39.791994+0000","last_fullsized":"2023-10-14T18:19:39.791994+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:51.575172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789559+0000","last_change":"2023-10-14T18:19:37.948561+0000","last_active":"2023-10-14T18:19:39.789559+0000","last_peered":"2023-10-14T18:19:39.789559+0000","last_clean":"2023-10-14T18:19:39.789559+0000","last_became_active":"2023-10-14T18:19:37.948282+0000","last_became_peered":"2023-10-14T18:19:37.948282+0000","last_unstale":"2023-10-14T18:19:39.789559+0000","last_undegraded":"2023-10-14T18:19:39.789559+0000","last_fullsized":"2023-10-14T18:19:39.789559+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:10:08.409215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962530+0000","last_change":"2023-10-14T18:19:36.948004+0000","last_active":"2023-10-14T18:19:38.962530+0000","last_peered":"2023-10-14T18:19:38.962530+0000","last_clean":"2023-10-14T18:19:38.962530+0000","last_became_active":"2023-10-14T18:19:36.945985+0000","last_became_peered":"2023-10-14T18:19:36.945985+0000","last_unstale":"2023-10-14T18:19:38.962530+0000","last_undegraded":"2023-10-14T18:19:38.962530+0000","last_fullsized":"2023-10-14T18:19:38.962530+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:47.175787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792268+0000","last_change":"2023-10-14T18:19:37.962449+0000","last_active":"2023-10-14T18:19:39.792268+0000","last_peered":"2023-10-14T18:19:39.792268+0000","last_clean":"2023-10-14T18:19:39.792268+0000","last_became_active":"2023-10-14T18:19:37.961855+0000","last_became_peered":"2023-10-14T18:19:37.961855+0000","last_unstale":"2023-10-14T18:19:39.792268+0000","last_undegraded":"2023-10-14T18:19:39.792268+0000","last_fullsized":"2023-10-14T18:19:39.792268+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:51.669310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961493+0000","last_change":"2023-10-14T18:19:36.946620+0000","last_active":"2023-10-14T18:19:38.961493+0000","last_peered":"2023-10-14T18:19:38.961493+0000","last_clean":"2023-10-14T18:19:38.961493+0000","last_became_active":"2023-10-14T18:19:36.946282+0000","last_became_peered":"2023-10-14T18:19:36.946282+0000","last_unstale":"2023-10-14T18:19:38.961493+0000","last_undegraded":"2023-10-14T18:19:38.961493+0000","last_fullsized":"2023-10-14T18:19:38.961493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:45.744672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789424+0000","last_change":"2023-10-14T18:19:37.956151+0000","last_active":"2023-10-14T18:19:39.789424+0000","last_peered":"2023-10-14T18:19:39.789424+0000","last_clean":"2023-10-14T18:19:39.789424+0000","last_became_active":"2023-10-14T18:19:37.953781+0000","last_became_peered":"2023-10-14T18:19:37.953781+0000","last_unstale":"2023-10-14T18:19:39.789424+0000","last_undegraded":"2023-10-14T18:19:39.789424+0000","last_fullsized":"2023-10-14T18:19:39.789424+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:08.275219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956911+0000","last_change":"2023-10-14T18:19:36.939395+0000","last_active":"2023-10-14T18:19:38.956911+0000","last_peered":"2023-10-14T18:19:38.956911+0000","last_clean":"2023-10-14T18:19:38.956911+0000","last_became_active":"2023-10-14T18:19:36.938194+0000","last_became_peered":"2023-10-14T18:19:36.938194+0000","last_unstale":"2023-10-14T18:19:38.956911+0000","last_undegraded":"2023-10-14T18:19:38.956911+0000","last_fullsized":"2023-10-14T18:19:38.956911+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:07.257561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789675+0000","last_change":"2023-10-14T18:19:37.946631+0000","last_active":"2023-10-14T18:19:39.789675+0000","last_peered":"2023-10-14T18:19:39.789675+0000","last_clean":"2023-10-14T18:19:39.789675+0000","last_became_active":"2023-10-14T18:19:37.945811+0000","last_became_peered":"2023-10-14T18:19:37.945811+0000","last_unstale":"2023-10-14T18:19:39.789675+0000","last_undegraded":"2023-10-14T18:19:39.789675+0000","last_fullsized":"2023-10-14T18:19:39.789675+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:02.232159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791165+0000","last_change":"2023-10-14T18:19:36.944301+0000","last_active":"2023-10-14T18:19:39.791165+0000","last_peered":"2023-10-14T18:19:39.791165+0000","last_clean":"2023-10-14T18:19:39.791165+0000","last_became_active":"2023-10-14T18:19:36.943940+0000","last_became_peered":"2023-10-14T18:19:36.943940+0000","last_unstale":"2023-10-14T18:19:39.791165+0000","last_undegraded":"2023-10-14T18:19:39.791165+0000","last_fullsized":"2023-10-14T18:19:39.791165+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:57.172453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790458+0000","last_change":"2023-10-14T18:19:37.955691+0000","last_active":"2023-10-14T18:19:39.790458+0000","last_peered":"2023-10-14T18:19:39.790458+0000","last_clean":"2023-10-14T18:19:39.790458+0000","last_became_active":"2023-10-14T18:19:37.955174+0000","last_became_peered":"2023-10-14T18:19:37.955174+0000","last_unstale":"2023-10-14T18:19:39.790458+0000","last_undegraded":"2023-10-14T18:19:39.790458+0000","last_fullsized":"2023-10-14T18:19:39.790458+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:21.754424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960645+0000","last_change":"2023-10-14T18:19:36.945338+0000","last_active":"2023-10-14T18:19:38.960645+0000","last_peered":"2023-10-14T18:19:38.960645+0000","last_clean":"2023-10-14T18:19:38.960645+0000","last_became_active":"2023-10-14T18:19:36.945017+0000","last_became_peered":"2023-10-14T18:19:36.945017+0000","last_unstale":"2023-10-14T18:19:38.960645+0000","last_undegraded":"2023-10-14T18:19:38.960645+0000","last_fullsized":"2023-10-14T18:19:38.960645+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:17.023978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956291+0000","last_change":"2023-10-14T18:19:37.949995+0000","last_active":"2023-10-14T18:19:38.956291+0000","last_peered":"2023-10-14T18:19:38.956291+0000","last_clean":"2023-10-14T18:19:38.956291+0000","last_became_active":"2023-10-14T18:19:37.948156+0000","last_became_peered":"2023-10-14T18:19:37.948156+0000","last_unstale":"2023-10-14T18:19:38.956291+0000","last_undegraded":"2023-10-14T18:19:38.956291+0000","last_fullsized":"2023-10-14T18:19:38.956291+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:17.373936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956409+0000","last_change":"2023-10-14T18:19:36.942341+0000","last_active":"2023-10-14T18:19:38.956409+0000","last_peered":"2023-10-14T18:19:38.956409+0000","last_clean":"2023-10-14T18:19:38.956409+0000","last_became_active":"2023-10-14T18:19:36.942036+0000","last_became_peered":"2023-10-14T18:19:36.942036+0000","last_unstale":"2023-10-14T18:19:38.956409+0000","last_undegraded":"2023-10-14T18:19:38.956409+0000","last_fullsized":"2023-10-14T18:19:38.956409+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:24.047990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959900+0000","last_change":"2023-10-14T18:19:37.954825+0000","last_active":"2023-10-14T18:19:38.959900+0000","last_peered":"2023-10-14T18:19:38.959900+0000","last_clean":"2023-10-14T18:19:38.959900+0000","last_became_active":"2023-10-14T18:19:37.951869+0000","last_became_peered":"2023-10-14T18:19:37.951869+0000","last_unstale":"2023-10-14T18:19:38.959900+0000","last_undegraded":"2023-10-14T18:19:38.959900+0000","last_fullsized":"2023-10-14T18:19:38.959900+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:20.874599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791757+0000","last_change":"2023-10-14T18:19:36.950250+0000","last_active":"2023-10-14T18:19:39.791757+0000","last_peered":"2023-10-14T18:19:39.791757+0000","last_clean":"2023-10-14T18:19:39.791757+0000","last_became_active":"2023-10-14T18:19:36.949880+0000","last_became_peered":"2023-10-14T18:19:36.949880+0000","last_unstale":"2023-10-14T18:19:39.791757+0000","last_undegraded":"2023-10-14T18:19:39.791757+0000","last_fullsized":"2023-10-14T18:19:39.791757+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:27.904631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790519+0000","last_change":"2023-10-14T18:19:37.949853+0000","last_active":"2023-10-14T18:19:39.790519+0000","last_peered":"2023-10-14T18:19:39.790519+0000","last_clean":"2023-10-14T18:19:39.790519+0000","last_became_active":"2023-10-14T18:19:37.949582+0000","last_became_peered":"2023-10-14T18:19:37.949582+0000","last_unstale":"2023-10-14T18:19:39.790519+0000","last_undegraded":"2023-10-14T18:19:39.790519+0000","last_fullsized":"2023-10-14T18:19:39.790519+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:31:03.000865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790357+0000","last_change":"2023-10-14T18:19:36.943829+0000","last_active":"2023-10-14T18:19:39.790357+0000","last_peered":"2023-10-14T18:19:39.790357+0000","last_clean":"2023-10-14T18:19:39.790357+0000","last_became_active":"2023-10-14T18:19:36.943413+0000","last_became_peered":"2023-10-14T18:19:36.943413+0000","last_unstale":"2023-10-14T18:19:39.790357+0000","last_undegraded":"2023-10-14T18:19:39.790357+0000","last_fullsized":"2023-10-14T18:19:39.790357+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:59:02.516834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962278+0000","last_change":"2023-10-14T18:19:37.960657+0000","last_active":"2023-10-14T18:19:38.962278+0000","last_peered":"2023-10-14T18:19:38.962278+0000","last_clean":"2023-10-14T18:19:38.962278+0000","last_became_active":"2023-10-14T18:19:37.960337+0000","last_became_peered":"2023-10-14T18:19:37.960337+0000","last_unstale":"2023-10-14T18:19:38.962278+0000","last_undegraded":"2023-10-14T18:19:38.962278+0000","last_fullsized":"2023-10-14T18:19:38.962278+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:43.180177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790966+0000","last_change":"2023-10-14T18:19:36.948863+0000","last_active":"2023-10-14T18:19:39.790966+0000","last_peered":"2023-10-14T18:19:39.790966+0000","last_clean":"2023-10-14T18:19:39.790966+0000","last_became_active":"2023-10-14T18:19:36.948404+0000","last_became_peered":"2023-10-14T18:19:36.948404+0000","last_unstale":"2023-10-14T18:19:39.790966+0000","last_undegraded":"2023-10-14T18:19:39.790966+0000","last_fullsized":"2023-10-14T18:19:39.790966+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:25:53.648941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792216+0000","last_change":"2023-10-14T18:19:37.951051+0000","last_active":"2023-10-14T18:19:39.792216+0000","last_peered":"2023-10-14T18:19:39.792216+0000","last_clean":"2023-10-14T18:19:39.792216+0000","last_became_active":"2023-10-14T18:19:37.950740+0000","last_became_peered":"2023-10-14T18:19:37.950740+0000","last_unstale":"2023-10-14T18:19:39.792216+0000","last_undegraded":"2023-10-14T18:19:39.792216+0000","last_fullsized":"2023-10-14T18:19:39.792216+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:52:28.800501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791582+0000","last_change":"2023-10-14T18:19:36.949827+0000","last_active":"2023-10-14T18:19:39.791582+0000","last_peered":"2023-10-14T18:19:39.791582+0000","last_clean":"2023-10-14T18:19:39.791582+0000","last_became_active":"2023-10-14T18:19:36.949453+0000","last_became_peered":"2023-10-14T18:19:36.949453+0000","last_unstale":"2023-10-14T18:19:39.791582+0000","last_undegraded":"2023-10-14T18:19:39.791582+0000","last_fullsized":"2023-10-14T18:19:39.791582+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:23.373832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789777+0000","last_change":"2023-10-14T18:19:37.952673+0000","last_active":"2023-10-14T18:19:39.789777+0000","last_peered":"2023-10-14T18:19:39.789777+0000","last_clean":"2023-10-14T18:19:39.789777+0000","last_became_active":"2023-10-14T18:19:37.951715+0000","last_became_peered":"2023-10-14T18:19:37.951715+0000","last_unstale":"2023-10-14T18:19:39.789777+0000","last_undegraded":"2023-10-14T18:19:39.789777+0000","last_fullsized":"2023-10-14T18:19:39.789777+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:17.373936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961398+0000","last_change":"2023-10-14T18:19:36.946593+0000","last_active":"2023-10-14T18:19:38.961398+0000","last_peered":"2023-10-14T18:19:38.961398+0000","last_clean":"2023-10-14T18:19:38.961398+0000","last_became_active":"2023-10-14T18:19:36.945216+0000","last_became_peered":"2023-10-14T18:19:36.945216+0000","last_unstale":"2023-10-14T18:19:38.961398+0000","last_undegraded":"2023-10-14T18:19:38.961398+0000","last_fullsized":"2023-10-14T18:19:38.961398+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:51.225370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790054+0000","last_change":"2023-10-14T18:19:37.959292+0000","last_active":"2023-10-14T18:19:39.790054+0000","last_peered":"2023-10-14T18:19:39.790054+0000","last_clean":"2023-10-14T18:19:39.790054+0000","last_became_active":"2023-10-14T18:19:37.957642+0000","last_became_peered":"2023-10-14T18:19:37.957642+0000","last_unstale":"2023-10-14T18:19:39.790054+0000","last_undegraded":"2023-10-14T18:19:39.790054+0000","last_fullsized":"2023-10-14T18:19:39.790054+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:55:59.116951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956321+0000","last_change":"2023-10-14T18:19:36.941465+0000","last_active":"2023-10-14T18:19:38.956321+0000","last_peered":"2023-10-14T18:19:38.956321+0000","last_clean":"2023-10-14T18:19:38.956321+0000","last_became_active":"2023-10-14T18:19:36.940751+0000","last_became_peered":"2023-10-14T18:19:36.940751+0000","last_unstale":"2023-10-14T18:19:38.956321+0000","last_undegraded":"2023-10-14T18:19:38.956321+0000","last_fullsized":"2023-10-14T18:19:38.956321+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:39.264522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789370+0000","last_change":"2023-10-14T18:19:37.955484+0000","last_active":"2023-10-14T18:19:39.789370+0000","last_peered":"2023-10-14T18:19:39.789370+0000","last_clean":"2023-10-14T18:19:39.789370+0000","last_became_active":"2023-10-14T18:19:37.954820+0000","last_became_peered":"2023-10-14T18:19:37.954820+0000","last_unstale":"2023-10-14T18:19:39.789370+0000","last_undegraded":"2023-10-14T18:19:39.789370+0000","last_fullsized":"2023-10-14T18:19:39.789370+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:52.613894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956555+0000","last_change":"2023-10-14T18:19:36.937157+0000","last_active":"2023-10-14T18:19:38.956555+0000","last_peered":"2023-10-14T18:19:38.956555+0000","last_clean":"2023-10-14T18:19:38.956555+0000","last_became_active":"2023-10-14T18:19:36.936803+0000","last_became_peered":"2023-10-14T18:19:36.936803+0000","last_unstale":"2023-10-14T18:19:38.956555+0000","last_undegraded":"2023-10-14T18:19:38.956555+0000","last_fullsized":"2023-10-14T18:19:38.956555+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:04.120792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960210+0000","last_change":"2023-10-14T18:19:37.954030+0000","last_active":"2023-10-14T18:19:38.960210+0000","last_peered":"2023-10-14T18:19:38.960210+0000","last_clean":"2023-10-14T18:19:38.960210+0000","last_became_active":"2023-10-14T18:19:37.952956+0000","last_became_peered":"2023-10-14T18:19:37.952956+0000","last_unstale":"2023-10-14T18:19:38.960210+0000","last_undegraded":"2023-10-14T18:19:38.960210+0000","last_fullsized":"2023-10-14T18:19:38.960210+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:26.903088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789209+0000","last_change":"2023-10-14T18:19:36.936461+0000","last_active":"2023-10-14T18:19:39.789209+0000","last_peered":"2023-10-14T18:19:39.789209+0000","last_clean":"2023-10-14T18:19:39.789209+0000","last_became_active":"2023-10-14T18:19:36.936122+0000","last_became_peered":"2023-10-14T18:19:36.936122+0000","last_unstale":"2023-10-14T18:19:39.789209+0000","last_undegraded":"2023-10-14T18:19:39.789209+0000","last_fullsized":"2023-10-14T18:19:39.789209+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:02:23.750313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790629+0000","last_change":"2023-10-14T18:19:37.965034+0000","last_active":"2023-10-14T18:19:39.790629+0000","last_peered":"2023-10-14T18:19:39.790629+0000","last_clean":"2023-10-14T18:19:39.790629+0000","last_became_active":"2023-10-14T18:19:37.964720+0000","last_became_peered":"2023-10-14T18:19:37.964720+0000","last_unstale":"2023-10-14T18:19:39.790629+0000","last_undegraded":"2023-10-14T18:19:39.790629+0000","last_fullsized":"2023-10-14T18:19:39.790629+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:53.867383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"256'381","reported_seq":1367,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-14T18:19:37.941346+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:19:37.941346+0000","last_peered":"2023-10-14T18:19:37.941346+0000","last_clean":"2023-10-14T18:19:37.941346+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:19:37.941346+0000","last_undegraded":"2023-10-14T18:19:37.941346+0000","last_fullsized":"2023-10-14T18:19:37.941346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57300,"kb_used_data":3908,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314540,"statfs":{"total":96636764160,"available":96578088960,"internally_reserved":0,"allocated":4001792,"data_stored":2296489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.056},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2949999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.2390000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.3440000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.405},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.1839999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.46},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.4970000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.274}]}]},{"osd":6,"up_from":13,"seq":55834576029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48416,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323424,"statfs":{"total":96636764160,"available":96587186176,"internally_reserved":0,"allocated":3424256,"data_stored":1712627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.879,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.135},{"interface":"front","average":{"1min":1.988,"5min":1.329,"15min":1.2549999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.583}]},{"osd":1,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.198,"15min":1.2410000000000001},"min":{"1min":0.74299999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3520000000000001,"15min":1.3660000000000001},"min":{"1min":0.92900000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.272}]},{"osd":2,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.397,"15min":1.3859999999999999},"min":{"1min":0.79000000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.9530000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.4179999999999999,"15min":1.427},"min":{"1min":1.028,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.367}]},{"osd":3,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.88800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.486},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5409999999999999,"15min":1.4830000000000001},"min":{"1min":0.95099999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":14.805999999999999,"5min":14.805999999999999,"15min":14.805999999999999},"last":2.081}]},{"osd":4,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.4399999999999999,"15min":1.4039999999999999},"min":{"1min":1.119,"5min":0.85899999999999999,"15min":0.75800000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.4810000000000001,"15min":1.456},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":2.1859999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.423,"15min":1.423},"min":{"1min":0.88,"5min":0.624,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.804},{"interface":"front","average":{"1min":1.655,"5min":1.6040000000000001,"15min":1.5960000000000001},"min":{"1min":0.97899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.8859999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":2.468,"5min":1.796,"15min":1.6779999999999999},"min":{"1min":1.1870000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":16.541,"5min":16.541,"15min":16.541},"last":2.262},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":1.131,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.0059999999999998,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":2.3340000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65064,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306776,"statfs":{"total":96636764160,"available":96570138624,"internally_reserved":0,"allocated":3428352,"data_stored":1708385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.175,"15min":1.2030000000000001},"min":{"1min":0.64300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.952,"5min":2.052,"15min":2.052},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.157,"15min":1.23},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":1.2030000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2210000000000001,"15min":1.268},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7010000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.87},{"interface":"front","average":{"1min":1.131,"5min":1.2689999999999999,"15min":1.294},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5269999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.5349999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1659999999999999,"15min":1.2230000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.5760000000000001,"5min":1.764,"15min":1.9339999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3720000000000001,"15min":1.395},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":1.8779999999999999,"15min":1.923},"last":1.0740000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3819999999999999,"15min":1.448},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.7789999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.02},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.377,"15min":1.407},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7210000000000001,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.613}]},{"osd":4,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4059999999999999,"15min":1.365},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.79600000000000004},"max":{"1min":1.8360000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.45,"15min":1.399},"min":{"1min":0.85599999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.0819999999999999,"5min":2.516,"15min":2.516},"last":1.373}]},{"osd":6,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4099999999999999,"15min":1.415},"min":{"1min":1.0309999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.329},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6319999999999999,"15min":1.629},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.81799999999999995},"max":{"1min":2.206,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.4279999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5600000000000001,"15min":1.5760000000000001},"min":{"1min":0.96399999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.105,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.464},{"interface":"front","average":{"1min":1.53,"5min":1.5760000000000001,"15min":1.556},"min":{"1min":0.90300000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2360000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.694}]}]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66408,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305432,"statfs":{"total":96636764160,"available":96568762368,"internally_reserved":0,"allocated":3428352,"data_stored":1712629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.1060000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.2689999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.1739999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.327},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.4990000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.4570000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56744,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315096,"statfs":{"total":96636764160,"available":96578658304,"internally_reserved":0,"allocated":3432448,"data_stored":1714149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.57399999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.417,"5min":1.4870000000000001,"15min":1.5660000000000001},"last":1.004},{"interface":"front","average":{"1min":1.087,"5min":1.0940000000000001,"15min":1.179},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.9970000000000001},"last":1.327}]},{"osd":1,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.92900000000000005,"15min":0.998},"min":{"1min":0.52400000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.919},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0860000000000001,"15min":1.1819999999999999},"min":{"1min":0.61299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5089999999999999,"5min":1.742,"15min":1.8560000000000001},"last":1.1910000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2609999999999999,"15min":1.401},"min":{"1min":0.50900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9099999999999999,"5min":1.9970000000000001,"15min":2.032},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.25,"15min":1.3560000000000001},"min":{"1min":0.68899999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.915,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.792}]},{"osd":4,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3560000000000001,"15min":1.4299999999999999},"min":{"1min":0.85399999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.105,"5min":2.105,"15min":2.153},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4430000000000001,"15min":1.478},"min":{"1min":1.034,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.9870000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.5069999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":1.0800000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.758,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.071},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9239999999999999,"5min":2.1360000000000001,"15min":2.891},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.5589999999999999,"15min":1.6259999999999999},"min":{"1min":0.88700000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.2120000000000002,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.627},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.2230000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1419999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.7430000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":0.97399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.452,"5min":3.153,"15min":3.153},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.6060000000000001,"15min":1.6759999999999999},"min":{"1min":1.052,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0339999999999998,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.444}]}]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56728,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315112,"statfs":{"total":96636764160,"available":96578674688,"internally_reserved":0,"allocated":3411968,"data_stored":1706211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":1.3169999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.381}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.74},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.7090000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.6339999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.9159999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.883},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.8129999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.9330000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52576,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3424256,"data_stored":1712571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.3400000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":1.246},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0169999999999999,"15min":1.1080000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1659999999999999,"5min":2.847,"15min":2.847},"last":1.3600000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.046},"min":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4590000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":1.306},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1799999999999999,"15min":1.254},"min":{"1min":0.67300000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6279999999999999,"5min":1.996,"15min":2.6859999999999999},"last":1.645}]},{"osd":3,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.2769999999999999,"15min":1.3700000000000001},"min":{"1min":0.66300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7370000000000001,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.47},{"interface":"front","average":{"1min":1.056,"5min":1.226,"15min":1.296},"min":{"1min":0.68899999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5609999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.548}]},{"osd":4,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.417,"15min":1.46},"min":{"1min":0.88300000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9259999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3740000000000001,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.1619999999999999,"5min":2.218,"15min":2.899},"last":1.7529999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.425,"15min":1.472},"min":{"1min":0.997,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":3.004},"last":1.599},{"interface":"front","average":{"1min":1.403,"5min":1.496,"15min":1.5569999999999999},"min":{"1min":0.90400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.5129999999999999},"last":1.9470000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.54,"15min":1.601},"min":{"1min":1.171,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0369999999999999,"5min":2.8999999999999999,"15min":2.931},"last":2.367},{"interface":"front","average":{"1min":1.518,"5min":1.6220000000000001,"15min":1.623},"min":{"1min":1.1140000000000001,"5min":0.871,"15min":0.871},"max":{"1min":2.0449999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":2.2509999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.619,"15min":1.6890000000000001},"min":{"1min":1.0029999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.3290000000000002,"5min":3.363,"15min":3.363},"last":1.847},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.651,"15min":1.7090000000000001},"min":{"1min":1.05,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.206,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":2.1629999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576029,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53168,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318672,"statfs":{"total":96636764160,"available":96582320128,"internally_reserved":0,"allocated":4026368,"data_stored":2303543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.0980000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.4159999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.6839999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.3009999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":2.004}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":2.077},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.363}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.6419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:19:42.814 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:19:42.867 INFO:tasks.ceph.mds.d.smithi186.stderr:2023-10-14T18:19:42.866+0000 7fbe96330640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:19:43.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.084+0000 7f1b78882640 1 Processor -- start 2023-10-14T18:19:43.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.084+0000 7f1b78882640 1 -- start start 2023-10-14T18:19:43.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b78882640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b7415dde0 0x7f1b740709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b78882640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b74160a90 con 0x7f1b74070f20 2023-10-14T18:19:43.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b78882640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b740cd900 con 0x7f1b7415c9f0 2023-10-14T18:19:43.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b78882640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b74071350 con 0x7f1b7415dde0 2023-10-14T18:19:43.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b7259c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1b7415dde0 0x7f1b740709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b7359e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1b74070f20 0x7f1b740ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57870/0 (socket says 172.21.15.29:57870) 2023-10-14T18:19:43.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.086+0000 7f1b7359e640 1 -- 172.21.15.29:0/4292931051 learned_addr learned my addr 172.21.15.29:0/4292931051 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.087+0000 7f1b71d9b640 1 -- 172.21.15.29:0/4292931051 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648785624 0 0) 0x7f1b74160a90 con 0x7f1b74070f20 2023-10-14T18:19:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.087+0000 7f1b71d9b640 1 -- 172.21.15.29:0/4292931051 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b5c003520 con 0x7f1b74070f20 2023-10-14T18:19:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.087+0000 7f1b7259c640 1 -- 172.21.15.29:0/4292931051 >> v1:172.21.15.186:6789/0 conn(0x7f1b7415c9f0 legacy=0x7f1b740c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.087+0000 7f1b7259c640 1 -- 172.21.15.29:0/4292931051 >> v1:172.21.15.29:6789/0 conn(0x7f1b74070f20 legacy=0x7f1b740ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.089 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.087+0000 7f1b7259c640 1 -- 172.21.15.29:0/4292931051 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b740cdbe0 con 0x7f1b7415dde0 2023-10-14T18:19:43.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.088+0000 7f1b7259c640 1 --2- 172.21.15.29:0/4292931051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b7415dde0 0x7f1b740709e0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f1b680097a0 tx=0x7f1b6802b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c73784bcff7370 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.088+0000 7f1b71d9b640 1 -- 172.21.15.29:0/4292931051 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68030610 con 0x7f1b7415dde0 2023-10-14T18:19:43.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.088+0000 7f1b71d9b640 1 -- 172.21.15.29:0/4292931051 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b68030770 con 0x7f1b7415dde0 2023-10-14T18:19:43.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.088+0000 7f1b71d9b640 1 -- 172.21.15.29:0/4292931051 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b68030b00 con 0x7f1b7415dde0 2023-10-14T18:19:43.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 -- 172.21.15.29:0/4292931051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b7415dde0 msgr2=0x7f1b740709e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 --2- 172.21.15.29:0/4292931051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b7415dde0 0x7f1b740709e0 secure :-1 s=READY pgs=3143 cs=0 l=1 rev1=1 crypto rx=0x7f1b680097a0 tx=0x7f1b6802b960 comp rx=0 tx=0).stop 2023-10-14T18:19:43.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 -- 172.21.15.29:0/4292931051 shutdown_connections 2023-10-14T18:19:43.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 --2- 172.21.15.29:0/4292931051 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b7415dde0 0x7f1b740709e0 unknown :-1 s=CLOSED pgs=3143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 -- 172.21.15.29:0/4292931051 >> 172.21.15.29:0/4292931051 conn(0x7f1b740ce6a0 msgr2=0x7f1b740ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:43.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 -- 172.21.15.29:0/4292931051 shutdown_connections 2023-10-14T18:19:43.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.089+0000 7f1b78882640 1 -- 172.21.15.29:0/4292931051 wait complete. 2023-10-14T18:19:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.091+0000 7f1b78882640 1 Processor -- start 2023-10-14T18:19:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.091+0000 7f1b78882640 1 -- start start 2023-10-14T18:19:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 0x7f1b7414a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b7415dde0 0x7f1b741f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7406c430 con 0x7f1b74070f20 2023-10-14T18:19:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7406cc70 con 0x7f1b7415c9f0 2023-10-14T18:19:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.092+0000 7f1b78882640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b741478f0 con 0x7f1b7415dde0 2023-10-14T18:19:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43410/0 (socket says 172.21.15.29:43410) 2023-10-14T18:19:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 -- 172.21.15.29:0/1996078172 learned_addr learned my addr 172.21.15.29:0/1996078172 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b7259c640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 0x7f1b7414a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 -- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b7415dde0 msgr2=0x7f1b741f1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b7415dde0 0x7f1b741f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 -- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 msgr2=0x7f1b7414a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 0x7f1b7414a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b72d9d640 1 -- 172.21.15.29:0/1996078172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b68009400 con 0x7f1b74070f20 2023-10-14T18:19:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.093+0000 7f1b7259c640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 0x7f1b7414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b72d9d640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400c450 tx=0x7f1b6400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b640052d0 con 0x7f1b74070f20 2023-10-14T18:19:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64005430 con 0x7f1b74070f20 2023-10-14T18:19:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b64012870 con 0x7f1b74070f20 2023-10-14T18:19:43.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b7406c6b0 con 0x7f1b74070f20 2023-10-14T18:19:43.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.094+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b7414a540 con 0x7f1b74070f20 2023-10-14T18:19:43.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.095+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b7404e9b0 con 0x7f1b74070f20 2023-10-14T18:19:43.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.096+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f1b640129d0 con 0x7f1b74070f20 2023-10-14T18:19:43.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.097+0000 7f1b5b7fe640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 0x7f1b6006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.097+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f1b64089570 con 0x7f1b74070f20 2023-10-14T18:19:43.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.098+0000 7f1b7259c640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 0x7f1b6006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.099+0000 7f1b7259c640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 0x7f1b6006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b68036000 tx=0x7f1b6803b040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.102+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1b640542d0 con 0x7f1b74070f20 2023-10-14T18:19:43.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.296+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 --> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1b740704c0 con 0x7f1b6006b0b0 2023-10-14T18:19:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.319+0000 7f1b5b7fe640 1 -- 172.21.15.29:0/1996078172 <== mgr.12471 v2:172.21.15.29:6832/1327833207 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+287548 (secure 0 0 0) 0x7f1b680309e0 con 0x7f1b6006b0b0 2023-10-14T18:19:43.324 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:19:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 msgr2=0x7f1b6006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 0x7f1b6006d570 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1b68036000 tx=0x7f1b6803b040 comp rx=0 tx=0).stop 2023-10-14T18:19:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 msgr2=0x7f1b74149ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 secure :-1 s=READY pgs=3144 cs=0 l=1 rev1=1 crypto rx=0x7f1b6400c450 tx=0x7f1b6400c910 comp rx=0 tx=0).stop 2023-10-14T18:19:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 shutdown_connections 2023-10-14T18:19:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f1b6006b0b0 0x7f1b6006d570 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1b7415dde0 0x7f1b741f1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1b7415c9f0 0x7f1b7414a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 --2- 172.21.15.29:0/1996078172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1b74070f20 0x7f1b74149ac0 unknown :-1 s=CLOSED pgs=3144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 >> 172.21.15.29:0/1996078172 conn(0x7f1b740ce6a0 msgr2=0x7f1b7415e7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 shutdown_connections 2023-10-14T18:19:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.325+0000 7f1b78882640 1 -- 172.21.15.29:0/1996078172 wait complete. 2023-10-14T18:19:43.347 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2023-10-14T18:19:43.183973+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":683,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":419,"ondisk_log_size":419,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":145,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":456404,"kb_used_data":27908,"kb_used_omap":12,"kb_used_meta":428339,"kb_avail":754518316,"statfs":{"total":773094113280,"available":772626755584,"internally_reserved":0,"allocated":28577792,"data_stored":14866604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":438619704},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.269994"},"pg_stats":[{"pgid":"35.3f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957402+0000","last_change":"2023-10-14T18:19:37.962929+0000","last_active":"2023-10-14T18:19:38.957402+0000","last_peered":"2023-10-14T18:19:38.957402+0000","last_clean":"2023-10-14T18:19:38.957402+0000","last_became_active":"2023-10-14T18:19:37.962473+0000","last_became_peered":"2023-10-14T18:19:37.962473+0000","last_unstale":"2023-10-14T18:19:38.957402+0000","last_undegraded":"2023-10-14T18:19:38.957402+0000","last_fullsized":"2023-10-14T18:19:38.957402+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:43.021524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.3e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791381+0000","last_change":"2023-10-14T18:19:37.948637+0000","last_active":"2023-10-14T18:19:39.791381+0000","last_peered":"2023-10-14T18:19:39.791381+0000","last_clean":"2023-10-14T18:19:39.791381+0000","last_became_active":"2023-10-14T18:19:37.947456+0000","last_became_peered":"2023-10-14T18:19:37.947456+0000","last_unstale":"2023-10-14T18:19:39.791381+0000","last_undegraded":"2023-10-14T18:19:39.791381+0000","last_fullsized":"2023-10-14T18:19:39.791381+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:18:15.520747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791912+0000","last_change":"2023-10-14T18:19:37.955377+0000","last_active":"2023-10-14T18:19:39.791912+0000","last_peered":"2023-10-14T18:19:39.791912+0000","last_clean":"2023-10-14T18:19:39.791912+0000","last_became_active":"2023-10-14T18:19:37.954245+0000","last_became_peered":"2023-10-14T18:19:37.954245+0000","last_unstale":"2023-10-14T18:19:39.791912+0000","last_undegraded":"2023-10-14T18:19:39.791912+0000","last_fullsized":"2023-10-14T18:19:39.791912+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:23:22.095336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960905+0000","last_change":"2023-10-14T18:19:37.963130+0000","last_active":"2023-10-14T18:19:38.960905+0000","last_peered":"2023-10-14T18:19:38.960905+0000","last_clean":"2023-10-14T18:19:38.960905+0000","last_became_active":"2023-10-14T18:19:37.962846+0000","last_became_peered":"2023-10-14T18:19:37.962846+0000","last_unstale":"2023-10-14T18:19:38.960905+0000","last_undegraded":"2023-10-14T18:19:38.960905+0000","last_fullsized":"2023-10-14T18:19:38.960905+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:52.243028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.3b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791504+0000","last_change":"2023-10-14T18:19:37.947461+0000","last_active":"2023-10-14T18:19:39.791504+0000","last_peered":"2023-10-14T18:19:39.791504+0000","last_clean":"2023-10-14T18:19:39.791504+0000","last_became_active":"2023-10-14T18:19:37.947180+0000","last_became_peered":"2023-10-14T18:19:37.947180+0000","last_unstale":"2023-10-14T18:19:39.791504+0000","last_undegraded":"2023-10-14T18:19:39.791504+0000","last_fullsized":"2023-10-14T18:19:39.791504+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:32:29.926911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.3a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790798+0000","last_change":"2023-10-14T18:19:37.958732+0000","last_active":"2023-10-14T18:19:39.790798+0000","last_peered":"2023-10-14T18:19:39.790798+0000","last_clean":"2023-10-14T18:19:39.790798+0000","last_became_active":"2023-10-14T18:19:37.958438+0000","last_became_peered":"2023-10-14T18:19:37.958438+0000","last_unstale":"2023-10-14T18:19:39.790798+0000","last_undegraded":"2023-10-14T18:19:39.790798+0000","last_fullsized":"2023-10-14T18:19:39.790798+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:41:47.792362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.39","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959996+0000","last_change":"2023-10-14T18:19:37.949222+0000","last_active":"2023-10-14T18:19:38.959996+0000","last_peered":"2023-10-14T18:19:38.959996+0000","last_clean":"2023-10-14T18:19:38.959996+0000","last_became_active":"2023-10-14T18:19:37.948769+0000","last_became_peered":"2023-10-14T18:19:37.948769+0000","last_unstale":"2023-10-14T18:19:38.959996+0000","last_undegraded":"2023-10-14T18:19:38.959996+0000","last_fullsized":"2023-10-14T18:19:38.959996+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:47:45.786253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.38","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959359+0000","last_change":"2023-10-14T18:19:37.960968+0000","last_active":"2023-10-14T18:19:38.959359+0000","last_peered":"2023-10-14T18:19:38.959359+0000","last_clean":"2023-10-14T18:19:38.959359+0000","last_became_active":"2023-10-14T18:19:37.957973+0000","last_became_peered":"2023-10-14T18:19:37.957973+0000","last_unstale":"2023-10-14T18:19:38.959359+0000","last_undegraded":"2023-10-14T18:19:38.959359+0000","last_fullsized":"2023-10-14T18:19:38.959359+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:16:08.822058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.37","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790464+0000","last_change":"2023-10-14T18:19:37.954101+0000","last_active":"2023-10-14T18:19:39.790464+0000","last_peered":"2023-10-14T18:19:39.790464+0000","last_clean":"2023-10-14T18:19:39.790464+0000","last_became_active":"2023-10-14T18:19:37.951660+0000","last_became_peered":"2023-10-14T18:19:37.951660+0000","last_unstale":"2023-10-14T18:19:39.790464+0000","last_undegraded":"2023-10-14T18:19:39.790464+0000","last_fullsized":"2023-10-14T18:19:39.790464+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:59.513183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.36","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960791+0000","last_change":"2023-10-14T18:19:37.951635+0000","last_active":"2023-10-14T18:19:38.960791+0000","last_peered":"2023-10-14T18:19:38.960791+0000","last_clean":"2023-10-14T18:19:38.960791+0000","last_became_active":"2023-10-14T18:19:37.951360+0000","last_became_peered":"2023-10-14T18:19:37.951360+0000","last_unstale":"2023-10-14T18:19:38.960791+0000","last_undegraded":"2023-10-14T18:19:38.960791+0000","last_fullsized":"2023-10-14T18:19:38.960791+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:59.598362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.35","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790823+0000","last_change":"2023-10-14T18:19:37.961834+0000","last_active":"2023-10-14T18:19:39.790823+0000","last_peered":"2023-10-14T18:19:39.790823+0000","last_clean":"2023-10-14T18:19:39.790823+0000","last_became_active":"2023-10-14T18:19:37.961547+0000","last_became_peered":"2023-10-14T18:19:37.961547+0000","last_unstale":"2023-10-14T18:19:39.790823+0000","last_undegraded":"2023-10-14T18:19:39.790823+0000","last_fullsized":"2023-10-14T18:19:39.790823+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:40:15.844626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.34","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956669+0000","last_change":"2023-10-14T18:19:37.958383+0000","last_active":"2023-10-14T18:19:38.956669+0000","last_peered":"2023-10-14T18:19:38.956669+0000","last_clean":"2023-10-14T18:19:38.956669+0000","last_became_active":"2023-10-14T18:19:37.956054+0000","last_became_peered":"2023-10-14T18:19:37.956054+0000","last_unstale":"2023-10-14T18:19:38.956669+0000","last_undegraded":"2023-10-14T18:19:38.956669+0000","last_fullsized":"2023-10-14T18:19:38.956669+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:22:59.513183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.33","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956976+0000","last_change":"2023-10-14T18:19:37.956359+0000","last_active":"2023-10-14T18:19:38.956976+0000","last_peered":"2023-10-14T18:19:38.956976+0000","last_clean":"2023-10-14T18:19:38.956976+0000","last_became_active":"2023-10-14T18:19:37.955889+0000","last_became_peered":"2023-10-14T18:19:37.955889+0000","last_unstale":"2023-10-14T18:19:38.956976+0000","last_undegraded":"2023-10-14T18:19:38.956976+0000","last_fullsized":"2023-10-14T18:19:38.956976+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:23:53.502432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.32","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960217+0000","last_change":"2023-10-14T18:19:37.949696+0000","last_active":"2023-10-14T18:19:38.960217+0000","last_peered":"2023-10-14T18:19:38.960217+0000","last_clean":"2023-10-14T18:19:38.960217+0000","last_became_active":"2023-10-14T18:19:37.949392+0000","last_became_peered":"2023-10-14T18:19:37.949392+0000","last_unstale":"2023-10-14T18:19:38.960217+0000","last_undegraded":"2023-10-14T18:19:38.960217+0000","last_fullsized":"2023-10-14T18:19:38.960217+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:49:04.902943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.31","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960350+0000","last_change":"2023-10-14T18:19:37.960510+0000","last_active":"2023-10-14T18:19:38.960350+0000","last_peered":"2023-10-14T18:19:38.960350+0000","last_clean":"2023-10-14T18:19:38.960350+0000","last_became_active":"2023-10-14T18:19:37.960214+0000","last_became_peered":"2023-10-14T18:19:37.960214+0000","last_unstale":"2023-10-14T18:19:38.960350+0000","last_undegraded":"2023-10-14T18:19:38.960350+0000","last_fullsized":"2023-10-14T18:19:38.960350+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:14:51.926483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.30","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959758+0000","last_change":"2023-10-14T18:19:37.946011+0000","last_active":"2023-10-14T18:19:38.959758+0000","last_peered":"2023-10-14T18:19:38.959758+0000","last_clean":"2023-10-14T18:19:38.959758+0000","last_became_active":"2023-10-14T18:19:37.945723+0000","last_became_peered":"2023-10-14T18:19:37.945723+0000","last_unstale":"2023-10-14T18:19:38.959758+0000","last_undegraded":"2023-10-14T18:19:38.959758+0000","last_fullsized":"2023-10-14T18:19:38.959758+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:23:41.423018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790643+0000","last_change":"2023-10-14T18:19:37.957318+0000","last_active":"2023-10-14T18:19:39.790643+0000","last_peered":"2023-10-14T18:19:39.790643+0000","last_clean":"2023-10-14T18:19:39.790643+0000","last_became_active":"2023-10-14T18:19:37.956995+0000","last_became_peered":"2023-10-14T18:19:37.956995+0000","last_unstale":"2023-10-14T18:19:39.790643+0000","last_undegraded":"2023-10-14T18:19:39.790643+0000","last_fullsized":"2023-10-14T18:19:39.790643+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:51.399343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.2e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789924+0000","last_change":"2023-10-14T18:19:37.947912+0000","last_active":"2023-10-14T18:19:39.789924+0000","last_peered":"2023-10-14T18:19:39.789924+0000","last_clean":"2023-10-14T18:19:39.789924+0000","last_became_active":"2023-10-14T18:19:37.947174+0000","last_became_peered":"2023-10-14T18:19:37.947174+0000","last_unstale":"2023-10-14T18:19:39.789924+0000","last_undegraded":"2023-10-14T18:19:39.789924+0000","last_fullsized":"2023-10-14T18:19:39.789924+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:37:18.238535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.2d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961713+0000","last_change":"2023-10-14T18:19:37.959011+0000","last_active":"2023-10-14T18:19:38.961713+0000","last_peered":"2023-10-14T18:19:38.961713+0000","last_clean":"2023-10-14T18:19:38.961713+0000","last_became_active":"2023-10-14T18:19:37.951416+0000","last_became_peered":"2023-10-14T18:19:37.951416+0000","last_unstale":"2023-10-14T18:19:38.961713+0000","last_undegraded":"2023-10-14T18:19:38.961713+0000","last_fullsized":"2023-10-14T18:19:38.961713+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:21:37.294239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961864+0000","last_change":"2023-10-14T18:19:37.956825+0000","last_active":"2023-10-14T18:19:38.961864+0000","last_peered":"2023-10-14T18:19:38.961864+0000","last_clean":"2023-10-14T18:19:38.961864+0000","last_became_active":"2023-10-14T18:19:37.950853+0000","last_became_peered":"2023-10-14T18:19:37.950853+0000","last_unstale":"2023-10-14T18:19:38.961864+0000","last_undegraded":"2023-10-14T18:19:38.961864+0000","last_fullsized":"2023-10-14T18:19:38.961864+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:44:20.559213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.2b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956794+0000","last_change":"2023-10-14T18:19:37.950729+0000","last_active":"2023-10-14T18:19:38.956794+0000","last_peered":"2023-10-14T18:19:38.956794+0000","last_clean":"2023-10-14T18:19:38.956794+0000","last_became_active":"2023-10-14T18:19:37.950072+0000","last_became_peered":"2023-10-14T18:19:37.950072+0000","last_unstale":"2023-10-14T18:19:38.956794+0000","last_undegraded":"2023-10-14T18:19:38.956794+0000","last_fullsized":"2023-10-14T18:19:38.956794+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:14:51.399343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.2a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790936+0000","last_change":"2023-10-14T18:19:37.943458+0000","last_active":"2023-10-14T18:19:39.790936+0000","last_peered":"2023-10-14T18:19:39.790936+0000","last_clean":"2023-10-14T18:19:39.790936+0000","last_became_active":"2023-10-14T18:19:37.941877+0000","last_became_peered":"2023-10-14T18:19:37.941877+0000","last_unstale":"2023-10-14T18:19:39.790936+0000","last_undegraded":"2023-10-14T18:19:39.790936+0000","last_fullsized":"2023-10-14T18:19:39.790936+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:13:47.235856+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.29","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384624+0000","last_change":"2023-10-14T18:19:37.952373+0000","last_active":"2023-10-14T18:19:39.384624+0000","last_peered":"2023-10-14T18:19:39.384624+0000","last_clean":"2023-10-14T18:19:39.384624+0000","last_became_active":"2023-10-14T18:19:37.951244+0000","last_became_peered":"2023-10-14T18:19:37.951244+0000","last_unstale":"2023-10-14T18:19:39.384624+0000","last_undegraded":"2023-10-14T18:19:39.384624+0000","last_fullsized":"2023-10-14T18:19:39.384624+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:12:20.815226+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.28","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789321+0000","last_change":"2023-10-14T18:19:37.945635+0000","last_active":"2023-10-14T18:19:39.789321+0000","last_peered":"2023-10-14T18:19:39.789321+0000","last_clean":"2023-10-14T18:19:39.789321+0000","last_became_active":"2023-10-14T18:19:37.945256+0000","last_became_peered":"2023-10-14T18:19:37.945256+0000","last_unstale":"2023-10-14T18:19:39.789321+0000","last_undegraded":"2023-10-14T18:19:39.789321+0000","last_fullsized":"2023-10-14T18:19:39.789321+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:58:47.232793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.27","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385367+0000","last_change":"2023-10-14T18:19:37.957037+0000","last_active":"2023-10-14T18:19:39.385367+0000","last_peered":"2023-10-14T18:19:39.385367+0000","last_clean":"2023-10-14T18:19:39.385367+0000","last_became_active":"2023-10-14T18:19:37.956689+0000","last_became_peered":"2023-10-14T18:19:37.956689+0000","last_unstale":"2023-10-14T18:19:39.385367+0000","last_undegraded":"2023-10-14T18:19:39.385367+0000","last_fullsized":"2023-10-14T18:19:39.385367+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:31:09.625588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.26","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957189+0000","last_change":"2023-10-14T18:19:37.954727+0000","last_active":"2023-10-14T18:19:38.957189+0000","last_peered":"2023-10-14T18:19:38.957189+0000","last_clean":"2023-10-14T18:19:38.957189+0000","last_became_active":"2023-10-14T18:19:37.954251+0000","last_became_peered":"2023-10-14T18:19:37.954251+0000","last_unstale":"2023-10-14T18:19:38.957189+0000","last_undegraded":"2023-10-14T18:19:38.957189+0000","last_fullsized":"2023-10-14T18:19:38.957189+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:10.274409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.25","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962035+0000","last_change":"2023-10-14T18:19:37.959240+0000","last_active":"2023-10-14T18:19:38.962035+0000","last_peered":"2023-10-14T18:19:38.962035+0000","last_clean":"2023-10-14T18:19:38.962035+0000","last_became_active":"2023-10-14T18:19:37.951267+0000","last_became_peered":"2023-10-14T18:19:37.951267+0000","last_unstale":"2023-10-14T18:19:38.962035+0000","last_undegraded":"2023-10-14T18:19:38.962035+0000","last_fullsized":"2023-10-14T18:19:38.962035+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:06:17.775956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.24","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961258+0000","last_change":"2023-10-14T18:19:37.954423+0000","last_active":"2023-10-14T18:19:38.961258+0000","last_peered":"2023-10-14T18:19:38.961258+0000","last_clean":"2023-10-14T18:19:38.961258+0000","last_became_active":"2023-10-14T18:19:37.954145+0000","last_became_peered":"2023-10-14T18:19:37.954145+0000","last_unstale":"2023-10-14T18:19:38.961258+0000","last_undegraded":"2023-10-14T18:19:38.961258+0000","last_fullsized":"2023-10-14T18:19:38.961258+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:06:46.249259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.23","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961168+0000","last_change":"2023-10-14T18:19:37.944004+0000","last_active":"2023-10-14T18:19:38.961168+0000","last_peered":"2023-10-14T18:19:38.961168+0000","last_clean":"2023-10-14T18:19:38.961168+0000","last_became_active":"2023-10-14T18:19:37.943727+0000","last_became_peered":"2023-10-14T18:19:37.943727+0000","last_unstale":"2023-10-14T18:19:38.961168+0000","last_undegraded":"2023-10-14T18:19:38.961168+0000","last_fullsized":"2023-10-14T18:19:38.961168+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:05:49.176233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.21","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962330+0000","last_change":"2023-10-14T18:19:37.943931+0000","last_active":"2023-10-14T18:19:38.962330+0000","last_peered":"2023-10-14T18:19:38.962330+0000","last_clean":"2023-10-14T18:19:38.962330+0000","last_became_active":"2023-10-14T18:19:37.943624+0000","last_became_peered":"2023-10-14T18:19:37.943624+0000","last_unstale":"2023-10-14T18:19:38.962330+0000","last_undegraded":"2023-10-14T18:19:38.962330+0000","last_fullsized":"2023-10-14T18:19:38.962330+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:23:13.244404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.20","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789387+0000","last_change":"2023-10-14T18:19:37.952289+0000","last_active":"2023-10-14T18:19:39.789387+0000","last_peered":"2023-10-14T18:19:39.789387+0000","last_clean":"2023-10-14T18:19:39.789387+0000","last_became_active":"2023-10-14T18:19:37.951590+0000","last_became_peered":"2023-10-14T18:19:37.951590+0000","last_unstale":"2023-10-14T18:19:39.789387+0000","last_undegraded":"2023-10-14T18:19:39.789387+0000","last_fullsized":"2023-10-14T18:19:39.789387+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:05.138450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957381+0000","last_change":"2023-10-14T18:19:37.955225+0000","last_active":"2023-10-14T18:19:38.957381+0000","last_peered":"2023-10-14T18:19:38.957381+0000","last_clean":"2023-10-14T18:19:38.957381+0000","last_became_active":"2023-10-14T18:19:37.954652+0000","last_became_peered":"2023-10-14T18:19:37.954652+0000","last_unstale":"2023-10-14T18:19:38.957381+0000","last_undegraded":"2023-10-14T18:19:38.957381+0000","last_fullsized":"2023-10-14T18:19:38.957381+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:18.041636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.1f","version":"270'4","reported_seq":16,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.957417+0000","last_change":"2023-10-14T18:19:36.944847+0000","last_active":"2023-10-14T18:19:38.957417+0000","last_peered":"2023-10-14T18:19:38.957417+0000","last_clean":"2023-10-14T18:19:38.957417+0000","last_became_active":"2023-10-14T18:19:36.944479+0000","last_became_peered":"2023-10-14T18:19:36.944479+0000","last_unstale":"2023-10-14T18:19:38.957417+0000","last_undegraded":"2023-10-14T18:19:38.957417+0000","last_fullsized":"2023-10-14T18:19:38.957417+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:10.448923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791721+0000","last_change":"2023-10-14T18:19:37.962682+0000","last_active":"2023-10-14T18:19:39.791721+0000","last_peered":"2023-10-14T18:19:39.791721+0000","last_clean":"2023-10-14T18:19:39.791721+0000","last_became_active":"2023-10-14T18:19:37.962373+0000","last_became_peered":"2023-10-14T18:19:37.962373+0000","last_unstale":"2023-10-14T18:19:39.791721+0000","last_undegraded":"2023-10-14T18:19:39.791721+0000","last_fullsized":"2023-10-14T18:19:39.791721+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:19:54.269382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.1e","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790267+0000","last_change":"2023-10-14T18:19:36.933531+0000","last_active":"2023-10-14T18:19:39.790267+0000","last_peered":"2023-10-14T18:19:39.790267+0000","last_clean":"2023-10-14T18:19:39.790267+0000","last_became_active":"2023-10-14T18:19:36.932553+0000","last_became_peered":"2023-10-14T18:19:36.932553+0000","last_unstale":"2023-10-14T18:19:39.790267+0000","last_undegraded":"2023-10-14T18:19:39.790267+0000","last_fullsized":"2023-10-14T18:19:39.790267+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:39.264522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791532+0000","last_change":"2023-10-14T18:19:37.949859+0000","last_active":"2023-10-14T18:19:39.791532+0000","last_peered":"2023-10-14T18:19:39.791532+0000","last_clean":"2023-10-14T18:19:39.791532+0000","last_became_active":"2023-10-14T18:19:37.949541+0000","last_became_peered":"2023-10-14T18:19:37.949541+0000","last_unstale":"2023-10-14T18:19:39.791532+0000","last_undegraded":"2023-10-14T18:19:39.791532+0000","last_fullsized":"2023-10-14T18:19:39.791532+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:37.633730+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1d","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790155+0000","last_change":"2023-10-14T18:19:36.934071+0000","last_active":"2023-10-14T18:19:39.790155+0000","last_peered":"2023-10-14T18:19:39.790155+0000","last_clean":"2023-10-14T18:19:39.790155+0000","last_became_active":"2023-10-14T18:19:36.933772+0000","last_became_peered":"2023-10-14T18:19:36.933772+0000","last_unstale":"2023-10-14T18:19:39.790155+0000","last_undegraded":"2023-10-14T18:19:39.790155+0000","last_fullsized":"2023-10-14T18:19:39.790155+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:54.693227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.1d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791904+0000","last_change":"2023-10-14T18:19:37.959261+0000","last_active":"2023-10-14T18:19:39.791904+0000","last_peered":"2023-10-14T18:19:39.791904+0000","last_clean":"2023-10-14T18:19:39.791904+0000","last_became_active":"2023-10-14T18:19:37.958940+0000","last_became_peered":"2023-10-14T18:19:37.958940+0000","last_unstale":"2023-10-14T18:19:39.791904+0000","last_undegraded":"2023-10-14T18:19:39.791904+0000","last_fullsized":"2023-10-14T18:19:39.791904+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:21:52.386185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.1c","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962157+0000","last_change":"2023-10-14T18:19:36.950224+0000","last_active":"2023-10-14T18:19:38.962157+0000","last_peered":"2023-10-14T18:19:38.962157+0000","last_clean":"2023-10-14T18:19:38.962157+0000","last_became_active":"2023-10-14T18:19:36.949907+0000","last_became_peered":"2023-10-14T18:19:36.949907+0000","last_unstale":"2023-10-14T18:19:38.962157+0000","last_undegraded":"2023-10-14T18:19:38.962157+0000","last_fullsized":"2023-10-14T18:19:38.962157+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:48.985755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.1a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385009+0000","last_change":"2023-10-14T18:19:37.956104+0000","last_active":"2023-10-14T18:19:39.385009+0000","last_peered":"2023-10-14T18:19:39.385009+0000","last_clean":"2023-10-14T18:19:39.385009+0000","last_became_active":"2023-10-14T18:19:37.955784+0000","last_became_peered":"2023-10-14T18:19:37.955784+0000","last_unstale":"2023-10-14T18:19:39.385009+0000","last_undegraded":"2023-10-14T18:19:39.385009+0000","last_fullsized":"2023-10-14T18:19:39.385009+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:12:41.758861+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.1b","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791354+0000","last_change":"2023-10-14T18:19:36.942983+0000","last_active":"2023-10-14T18:19:39.791354+0000","last_peered":"2023-10-14T18:19:39.791354+0000","last_clean":"2023-10-14T18:19:39.791354+0000","last_became_active":"2023-10-14T18:19:36.942671+0000","last_became_peered":"2023-10-14T18:19:36.942671+0000","last_unstale":"2023-10-14T18:19:39.791354+0000","last_undegraded":"2023-10-14T18:19:39.791354+0000","last_fullsized":"2023-10-14T18:19:39.791354+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:52:46.834539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.1b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961489+0000","last_change":"2023-10-14T18:19:37.958838+0000","last_active":"2023-10-14T18:19:38.961489+0000","last_peered":"2023-10-14T18:19:38.961489+0000","last_clean":"2023-10-14T18:19:38.961489+0000","last_became_active":"2023-10-14T18:19:37.957007+0000","last_became_peered":"2023-10-14T18:19:37.957007+0000","last_unstale":"2023-10-14T18:19:38.961489+0000","last_undegraded":"2023-10-14T18:19:38.961489+0000","last_fullsized":"2023-10-14T18:19:38.961489+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:08:24.162939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.1a","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789620+0000","last_change":"2023-10-14T18:19:36.951111+0000","last_active":"2023-10-14T18:19:39.789620+0000","last_peered":"2023-10-14T18:19:39.789620+0000","last_clean":"2023-10-14T18:19:39.789620+0000","last_became_active":"2023-10-14T18:19:36.950255+0000","last_became_peered":"2023-10-14T18:19:36.950255+0000","last_unstale":"2023-10-14T18:19:39.789620+0000","last_undegraded":"2023-10-14T18:19:39.789620+0000","last_fullsized":"2023-10-14T18:19:39.789620+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:18:46.635787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.18","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789572+0000","last_change":"2023-10-14T18:19:37.949676+0000","last_active":"2023-10-14T18:19:39.789572+0000","last_peered":"2023-10-14T18:19:39.789572+0000","last_clean":"2023-10-14T18:19:39.789572+0000","last_became_active":"2023-10-14T18:19:37.949406+0000","last_became_peered":"2023-10-14T18:19:37.949406+0000","last_unstale":"2023-10-14T18:19:39.789572+0000","last_undegraded":"2023-10-14T18:19:39.789572+0000","last_fullsized":"2023-10-14T18:19:39.789572+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:39:11.466581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.19","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956309+0000","last_change":"2023-10-14T18:19:36.937220+0000","last_active":"2023-10-14T18:19:38.956309+0000","last_peered":"2023-10-14T18:19:38.956309+0000","last_clean":"2023-10-14T18:19:38.956309+0000","last_became_active":"2023-10-14T18:19:36.936897+0000","last_became_peered":"2023-10-14T18:19:36.936897+0000","last_unstale":"2023-10-14T18:19:38.956309+0000","last_undegraded":"2023-10-14T18:19:38.956309+0000","last_fullsized":"2023-10-14T18:19:38.956309+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:13:08.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.19","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789100+0000","last_change":"2023-10-14T18:19:37.950508+0000","last_active":"2023-10-14T18:19:39.789100+0000","last_peered":"2023-10-14T18:19:39.789100+0000","last_clean":"2023-10-14T18:19:39.789100+0000","last_became_active":"2023-10-14T18:19:37.947318+0000","last_became_peered":"2023-10-14T18:19:37.947318+0000","last_unstale":"2023-10-14T18:19:39.789100+0000","last_undegraded":"2023-10-14T18:19:39.789100+0000","last_fullsized":"2023-10-14T18:19:39.789100+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:47:16.780161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.18","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.955976+0000","last_change":"2023-10-14T18:19:36.945722+0000","last_active":"2023-10-14T18:19:38.955976+0000","last_peered":"2023-10-14T18:19:38.955976+0000","last_clean":"2023-10-14T18:19:38.955976+0000","last_became_active":"2023-10-14T18:19:36.945465+0000","last_became_peered":"2023-10-14T18:19:36.945465+0000","last_unstale":"2023-10-14T18:19:38.955976+0000","last_undegraded":"2023-10-14T18:19:38.955976+0000","last_fullsized":"2023-10-14T18:19:38.955976+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:10:46.378047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.16","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790284+0000","last_change":"2023-10-14T18:19:37.947828+0000","last_active":"2023-10-14T18:19:39.790284+0000","last_peered":"2023-10-14T18:19:39.790284+0000","last_clean":"2023-10-14T18:19:39.790284+0000","last_became_active":"2023-10-14T18:19:37.947548+0000","last_became_peered":"2023-10-14T18:19:37.947548+0000","last_unstale":"2023-10-14T18:19:39.790284+0000","last_undegraded":"2023-10-14T18:19:39.790284+0000","last_fullsized":"2023-10-14T18:19:39.790284+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:34:34.652584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.17","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961601+0000","last_change":"2023-10-14T18:19:36.933826+0000","last_active":"2023-10-14T18:19:38.961601+0000","last_peered":"2023-10-14T18:19:38.961601+0000","last_clean":"2023-10-14T18:19:38.961601+0000","last_became_active":"2023-10-14T18:19:36.932965+0000","last_became_peered":"2023-10-14T18:19:36.932965+0000","last_unstale":"2023-10-14T18:19:38.961601+0000","last_undegraded":"2023-10-14T18:19:38.961601+0000","last_fullsized":"2023-10-14T18:19:38.961601+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:46:42.003867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.17","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961393+0000","last_change":"2023-10-14T18:19:37.955290+0000","last_active":"2023-10-14T18:19:38.961393+0000","last_peered":"2023-10-14T18:19:38.961393+0000","last_clean":"2023-10-14T18:19:38.961393+0000","last_became_active":"2023-10-14T18:19:37.954984+0000","last_became_peered":"2023-10-14T18:19:37.954984+0000","last_unstale":"2023-10-14T18:19:38.961393+0000","last_undegraded":"2023-10-14T18:19:38.961393+0000","last_fullsized":"2023-10-14T18:19:38.961393+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:26:21.823073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.16","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791343+0000","last_change":"2023-10-14T18:19:36.944236+0000","last_active":"2023-10-14T18:19:39.791343+0000","last_peered":"2023-10-14T18:19:39.791343+0000","last_clean":"2023-10-14T18:19:39.791343+0000","last_became_active":"2023-10-14T18:19:36.943907+0000","last_became_peered":"2023-10-14T18:19:36.943907+0000","last_unstale":"2023-10-14T18:19:39.791343+0000","last_undegraded":"2023-10-14T18:19:39.791343+0000","last_fullsized":"2023-10-14T18:19:39.791343+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:54:02.542955+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.14","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791536+0000","last_change":"2023-10-14T18:19:37.964487+0000","last_active":"2023-10-14T18:19:39.791536+0000","last_peered":"2023-10-14T18:19:39.791536+0000","last_clean":"2023-10-14T18:19:39.791536+0000","last_became_active":"2023-10-14T18:19:37.964197+0000","last_became_peered":"2023-10-14T18:19:37.964197+0000","last_unstale":"2023-10-14T18:19:39.791536+0000","last_undegraded":"2023-10-14T18:19:39.791536+0000","last_fullsized":"2023-10-14T18:19:39.791536+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:29:31.227271+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.15","version":"270'3","reported_seq":15,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790325+0000","last_change":"2023-10-14T18:19:36.943334+0000","last_active":"2023-10-14T18:19:39.790325+0000","last_peered":"2023-10-14T18:19:39.790325+0000","last_clean":"2023-10-14T18:19:39.790325+0000","last_became_active":"2023-10-14T18:19:36.942920+0000","last_became_peered":"2023-10-14T18:19:36.942920+0000","last_unstale":"2023-10-14T18:19:39.790325+0000","last_undegraded":"2023-10-14T18:19:39.790325+0000","last_fullsized":"2023-10-14T18:19:39.790325+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:49:08.817462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"35.15","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962091+0000","last_change":"2023-10-14T18:19:37.948963+0000","last_active":"2023-10-14T18:19:38.962091+0000","last_peered":"2023-10-14T18:19:38.962091+0000","last_clean":"2023-10-14T18:19:38.962091+0000","last_became_active":"2023-10-14T18:19:37.948601+0000","last_became_peered":"2023-10-14T18:19:37.948601+0000","last_unstale":"2023-10-14T18:19:38.962091+0000","last_undegraded":"2023-10-14T18:19:38.962091+0000","last_fullsized":"2023-10-14T18:19:38.962091+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:50.003412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.14","version":"270'6","reported_seq":18,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791550+0000","last_change":"2023-10-14T18:19:36.944761+0000","last_active":"2023-10-14T18:19:39.791550+0000","last_peered":"2023-10-14T18:19:39.791550+0000","last_clean":"2023-10-14T18:19:39.791550+0000","last_became_active":"2023-10-14T18:19:36.944297+0000","last_became_peered":"2023-10-14T18:19:36.944297+0000","last_unstale":"2023-10-14T18:19:39.791550+0000","last_undegraded":"2023-10-14T18:19:39.791550+0000","last_fullsized":"2023-10-14T18:19:39.791550+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:16:55.193023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.12","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961042+0000","last_change":"2023-10-14T18:19:37.951816+0000","last_active":"2023-10-14T18:19:38.961042+0000","last_peered":"2023-10-14T18:19:38.961042+0000","last_clean":"2023-10-14T18:19:38.961042+0000","last_became_active":"2023-10-14T18:19:37.950686+0000","last_became_peered":"2023-10-14T18:19:37.950686+0000","last_unstale":"2023-10-14T18:19:38.961042+0000","last_undegraded":"2023-10-14T18:19:38.961042+0000","last_fullsized":"2023-10-14T18:19:38.961042+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:12:09.048028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.13","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385148+0000","last_change":"2023-10-14T18:19:36.939342+0000","last_active":"2023-10-14T18:19:39.385148+0000","last_peered":"2023-10-14T18:19:39.385148+0000","last_clean":"2023-10-14T18:19:39.385148+0000","last_became_active":"2023-10-14T18:19:36.939036+0000","last_became_peered":"2023-10-14T18:19:36.939036+0000","last_unstale":"2023-10-14T18:19:39.385148+0000","last_undegraded":"2023-10-14T18:19:39.385148+0000","last_fullsized":"2023-10-14T18:19:39.385148+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:42:51.786864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.13","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789279+0000","last_change":"2023-10-14T18:19:37.954331+0000","last_active":"2023-10-14T18:19:39.789279+0000","last_peered":"2023-10-14T18:19:39.789279+0000","last_clean":"2023-10-14T18:19:39.789279+0000","last_became_active":"2023-10-14T18:19:37.954038+0000","last_became_peered":"2023-10-14T18:19:37.954038+0000","last_unstale":"2023-10-14T18:19:39.789279+0000","last_undegraded":"2023-10-14T18:19:39.789279+0000","last_fullsized":"2023-10-14T18:19:39.789279+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:27:17.032998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.12","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384757+0000","last_change":"2023-10-14T18:19:36.935317+0000","last_active":"2023-10-14T18:19:39.384757+0000","last_peered":"2023-10-14T18:19:39.384757+0000","last_clean":"2023-10-14T18:19:39.384757+0000","last_became_active":"2023-10-14T18:19:36.934957+0000","last_became_peered":"2023-10-14T18:19:36.934957+0000","last_unstale":"2023-10-14T18:19:39.384757+0000","last_undegraded":"2023-10-14T18:19:39.384757+0000","last_fullsized":"2023-10-14T18:19:39.384757+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:42:18.035081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.10","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789773+0000","last_change":"2023-10-14T18:19:37.958897+0000","last_active":"2023-10-14T18:19:39.789773+0000","last_peered":"2023-10-14T18:19:39.789773+0000","last_clean":"2023-10-14T18:19:39.789773+0000","last_became_active":"2023-10-14T18:19:37.957884+0000","last_became_peered":"2023-10-14T18:19:37.957884+0000","last_unstale":"2023-10-14T18:19:39.789773+0000","last_undegraded":"2023-10-14T18:19:39.789773+0000","last_fullsized":"2023-10-14T18:19:39.789773+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:40:43.956572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"34.11","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960421+0000","last_change":"2023-10-14T18:19:36.946770+0000","last_active":"2023-10-14T18:19:38.960421+0000","last_peered":"2023-10-14T18:19:38.960421+0000","last_clean":"2023-10-14T18:19:38.960421+0000","last_became_active":"2023-10-14T18:19:36.945356+0000","last_became_peered":"2023-10-14T18:19:36.945356+0000","last_unstale":"2023-10-14T18:19:38.960421+0000","last_undegraded":"2023-10-14T18:19:38.960421+0000","last_fullsized":"2023-10-14T18:19:38.960421+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:22:09.256751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.11","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961942+0000","last_change":"2023-10-14T18:19:37.960753+0000","last_active":"2023-10-14T18:19:38.961942+0000","last_peered":"2023-10-14T18:19:38.961942+0000","last_clean":"2023-10-14T18:19:38.961942+0000","last_became_active":"2023-10-14T18:19:37.960057+0000","last_became_peered":"2023-10-14T18:19:37.960057+0000","last_unstale":"2023-10-14T18:19:38.961942+0000","last_undegraded":"2023-10-14T18:19:38.961942+0000","last_fullsized":"2023-10-14T18:19:38.961942+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:27:58.737280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.10","version":"270'8","reported_seq":20,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.384750+0000","last_change":"2023-10-14T18:19:36.945527+0000","last_active":"2023-10-14T18:19:39.384750+0000","last_peered":"2023-10-14T18:19:39.384750+0000","last_clean":"2023-10-14T18:19:39.384750+0000","last_became_active":"2023-10-14T18:19:36.945265+0000","last_became_peered":"2023-10-14T18:19:36.945265+0000","last_unstale":"2023-10-14T18:19:39.384750+0000","last_undegraded":"2023-10-14T18:19:39.384750+0000","last_fullsized":"2023-10-14T18:19:39.384750+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T00:02:26.845934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"35.e","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.385414+0000","last_change":"2023-10-14T18:19:37.954344+0000","last_active":"2023-10-14T18:19:39.385414+0000","last_peered":"2023-10-14T18:19:39.385414+0000","last_clean":"2023-10-14T18:19:39.385414+0000","last_became_active":"2023-10-14T18:19:37.954064+0000","last_became_peered":"2023-10-14T18:19:37.954064+0000","last_unstale":"2023-10-14T18:19:39.385414+0000","last_undegraded":"2023-10-14T18:19:39.385414+0000","last_fullsized":"2023-10-14T18:19:39.385414+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:59:30.172265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"34.f","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961814+0000","last_change":"2023-10-14T18:19:36.941358+0000","last_active":"2023-10-14T18:19:38.961814+0000","last_peered":"2023-10-14T18:19:38.961814+0000","last_clean":"2023-10-14T18:19:38.961814+0000","last_became_active":"2023-10-14T18:19:36.941067+0000","last_became_peered":"2023-10-14T18:19:36.941067+0000","last_unstale":"2023-10-14T18:19:38.961814+0000","last_undegraded":"2023-10-14T18:19:38.961814+0000","last_fullsized":"2023-10-14T18:19:38.961814+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:48.163611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.f","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792128+0000","last_change":"2023-10-14T18:19:37.957490+0000","last_active":"2023-10-14T18:19:39.792128+0000","last_peered":"2023-10-14T18:19:39.792128+0000","last_clean":"2023-10-14T18:19:39.792128+0000","last_became_active":"2023-10-14T18:19:37.957199+0000","last_became_peered":"2023-10-14T18:19:37.957199+0000","last_unstale":"2023-10-14T18:19:39.792128+0000","last_undegraded":"2023-10-14T18:19:39.792128+0000","last_fullsized":"2023-10-14T18:19:39.792128+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:09:27.409684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.e","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791994+0000","last_change":"2023-10-14T18:19:36.937845+0000","last_active":"2023-10-14T18:19:39.791994+0000","last_peered":"2023-10-14T18:19:39.791994+0000","last_clean":"2023-10-14T18:19:39.791994+0000","last_became_active":"2023-10-14T18:19:36.937546+0000","last_became_peered":"2023-10-14T18:19:36.937546+0000","last_unstale":"2023-10-14T18:19:39.791994+0000","last_undegraded":"2023-10-14T18:19:39.791994+0000","last_fullsized":"2023-10-14T18:19:39.791994+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:21:51.575172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.c","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789559+0000","last_change":"2023-10-14T18:19:37.948561+0000","last_active":"2023-10-14T18:19:39.789559+0000","last_peered":"2023-10-14T18:19:39.789559+0000","last_clean":"2023-10-14T18:19:39.789559+0000","last_became_active":"2023-10-14T18:19:37.948282+0000","last_became_peered":"2023-10-14T18:19:37.948282+0000","last_unstale":"2023-10-14T18:19:39.789559+0000","last_undegraded":"2023-10-14T18:19:39.789559+0000","last_fullsized":"2023-10-14T18:19:39.789559+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:10:08.409215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.d","version":"270'1","reported_seq":13,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962530+0000","last_change":"2023-10-14T18:19:36.948004+0000","last_active":"2023-10-14T18:19:38.962530+0000","last_peered":"2023-10-14T18:19:38.962530+0000","last_clean":"2023-10-14T18:19:38.962530+0000","last_became_active":"2023-10-14T18:19:36.945985+0000","last_became_peered":"2023-10-14T18:19:36.945985+0000","last_unstale":"2023-10-14T18:19:38.962530+0000","last_undegraded":"2023-10-14T18:19:38.962530+0000","last_fullsized":"2023-10-14T18:19:38.962530+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:31:47.175787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.d","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792268+0000","last_change":"2023-10-14T18:19:37.962449+0000","last_active":"2023-10-14T18:19:39.792268+0000","last_peered":"2023-10-14T18:19:39.792268+0000","last_clean":"2023-10-14T18:19:39.792268+0000","last_became_active":"2023-10-14T18:19:37.961855+0000","last_became_peered":"2023-10-14T18:19:37.961855+0000","last_unstale":"2023-10-14T18:19:39.792268+0000","last_undegraded":"2023-10-14T18:19:39.792268+0000","last_fullsized":"2023-10-14T18:19:39.792268+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:43:51.669310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.c","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961493+0000","last_change":"2023-10-14T18:19:36.946620+0000","last_active":"2023-10-14T18:19:38.961493+0000","last_peered":"2023-10-14T18:19:38.961493+0000","last_clean":"2023-10-14T18:19:38.961493+0000","last_became_active":"2023-10-14T18:19:36.946282+0000","last_became_peered":"2023-10-14T18:19:36.946282+0000","last_unstale":"2023-10-14T18:19:38.961493+0000","last_undegraded":"2023-10-14T18:19:38.961493+0000","last_fullsized":"2023-10-14T18:19:38.961493+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:10:45.744672+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.a","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789424+0000","last_change":"2023-10-14T18:19:37.956151+0000","last_active":"2023-10-14T18:19:39.789424+0000","last_peered":"2023-10-14T18:19:39.789424+0000","last_clean":"2023-10-14T18:19:39.789424+0000","last_became_active":"2023-10-14T18:19:37.953781+0000","last_became_peered":"2023-10-14T18:19:37.953781+0000","last_unstale":"2023-10-14T18:19:39.789424+0000","last_undegraded":"2023-10-14T18:19:39.789424+0000","last_fullsized":"2023-10-14T18:19:39.789424+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:08.275219+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.b","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956911+0000","last_change":"2023-10-14T18:19:36.939395+0000","last_active":"2023-10-14T18:19:38.956911+0000","last_peered":"2023-10-14T18:19:38.956911+0000","last_clean":"2023-10-14T18:19:38.956911+0000","last_became_active":"2023-10-14T18:19:36.938194+0000","last_became_peered":"2023-10-14T18:19:36.938194+0000","last_unstale":"2023-10-14T18:19:38.956911+0000","last_undegraded":"2023-10-14T18:19:38.956911+0000","last_fullsized":"2023-10-14T18:19:38.956911+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:58:07.257561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.b","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789675+0000","last_change":"2023-10-14T18:19:37.946631+0000","last_active":"2023-10-14T18:19:39.789675+0000","last_peered":"2023-10-14T18:19:39.789675+0000","last_clean":"2023-10-14T18:19:39.789675+0000","last_became_active":"2023-10-14T18:19:37.945811+0000","last_became_peered":"2023-10-14T18:19:37.945811+0000","last_unstale":"2023-10-14T18:19:39.789675+0000","last_undegraded":"2023-10-14T18:19:39.789675+0000","last_fullsized":"2023-10-14T18:19:39.789675+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:04:02.232159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.a","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791165+0000","last_change":"2023-10-14T18:19:36.944301+0000","last_active":"2023-10-14T18:19:39.791165+0000","last_peered":"2023-10-14T18:19:39.791165+0000","last_clean":"2023-10-14T18:19:39.791165+0000","last_became_active":"2023-10-14T18:19:36.943940+0000","last_became_peered":"2023-10-14T18:19:36.943940+0000","last_unstale":"2023-10-14T18:19:39.791165+0000","last_undegraded":"2023-10-14T18:19:39.791165+0000","last_fullsized":"2023-10-14T18:19:39.791165+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:12:57.172453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.8","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790458+0000","last_change":"2023-10-14T18:19:37.955691+0000","last_active":"2023-10-14T18:19:39.790458+0000","last_peered":"2023-10-14T18:19:39.790458+0000","last_clean":"2023-10-14T18:19:39.790458+0000","last_became_active":"2023-10-14T18:19:37.955174+0000","last_became_peered":"2023-10-14T18:19:37.955174+0000","last_unstale":"2023-10-14T18:19:39.790458+0000","last_undegraded":"2023-10-14T18:19:39.790458+0000","last_fullsized":"2023-10-14T18:19:39.790458+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:14:21.754424+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.9","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960645+0000","last_change":"2023-10-14T18:19:36.945338+0000","last_active":"2023-10-14T18:19:38.960645+0000","last_peered":"2023-10-14T18:19:38.960645+0000","last_clean":"2023-10-14T18:19:38.960645+0000","last_became_active":"2023-10-14T18:19:36.945017+0000","last_became_peered":"2023-10-14T18:19:36.945017+0000","last_unstale":"2023-10-14T18:19:38.960645+0000","last_undegraded":"2023-10-14T18:19:38.960645+0000","last_fullsized":"2023-10-14T18:19:38.960645+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:54:17.023978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"35.9","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956291+0000","last_change":"2023-10-14T18:19:37.949995+0000","last_active":"2023-10-14T18:19:38.956291+0000","last_peered":"2023-10-14T18:19:38.956291+0000","last_clean":"2023-10-14T18:19:38.956291+0000","last_became_active":"2023-10-14T18:19:37.948156+0000","last_became_peered":"2023-10-14T18:19:37.948156+0000","last_unstale":"2023-10-14T18:19:38.956291+0000","last_undegraded":"2023-10-14T18:19:38.956291+0000","last_fullsized":"2023-10-14T18:19:38.956291+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:17.373936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"34.8","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956409+0000","last_change":"2023-10-14T18:19:36.942341+0000","last_active":"2023-10-14T18:19:38.956409+0000","last_peered":"2023-10-14T18:19:38.956409+0000","last_clean":"2023-10-14T18:19:38.956409+0000","last_became_active":"2023-10-14T18:19:36.942036+0000","last_became_peered":"2023-10-14T18:19:36.942036+0000","last_unstale":"2023-10-14T18:19:38.956409+0000","last_undegraded":"2023-10-14T18:19:38.956409+0000","last_fullsized":"2023-10-14T18:19:38.956409+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:24:24.047990+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.6","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.959900+0000","last_change":"2023-10-14T18:19:37.954825+0000","last_active":"2023-10-14T18:19:38.959900+0000","last_peered":"2023-10-14T18:19:38.959900+0000","last_clean":"2023-10-14T18:19:38.959900+0000","last_became_active":"2023-10-14T18:19:37.951869+0000","last_became_peered":"2023-10-14T18:19:37.951869+0000","last_unstale":"2023-10-14T18:19:38.959900+0000","last_undegraded":"2023-10-14T18:19:38.959900+0000","last_fullsized":"2023-10-14T18:19:38.959900+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T22:07:20.874599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.7","version":"270'2","reported_seq":14,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791757+0000","last_change":"2023-10-14T18:19:36.950250+0000","last_active":"2023-10-14T18:19:39.791757+0000","last_peered":"2023-10-14T18:19:39.791757+0000","last_clean":"2023-10-14T18:19:39.791757+0000","last_became_active":"2023-10-14T18:19:36.949880+0000","last_became_peered":"2023-10-14T18:19:36.949880+0000","last_unstale":"2023-10-14T18:19:39.791757+0000","last_undegraded":"2023-10-14T18:19:39.791757+0000","last_fullsized":"2023-10-14T18:19:39.791757+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T04:45:27.904631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.7","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790519+0000","last_change":"2023-10-14T18:19:37.949853+0000","last_active":"2023-10-14T18:19:39.790519+0000","last_peered":"2023-10-14T18:19:39.790519+0000","last_clean":"2023-10-14T18:19:39.790519+0000","last_became_active":"2023-10-14T18:19:37.949582+0000","last_became_peered":"2023-10-14T18:19:37.949582+0000","last_unstale":"2023-10-14T18:19:39.790519+0000","last_undegraded":"2023-10-14T18:19:39.790519+0000","last_fullsized":"2023-10-14T18:19:39.790519+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:31:03.000865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.6","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790357+0000","last_change":"2023-10-14T18:19:36.943829+0000","last_active":"2023-10-14T18:19:39.790357+0000","last_peered":"2023-10-14T18:19:39.790357+0000","last_clean":"2023-10-14T18:19:39.790357+0000","last_became_active":"2023-10-14T18:19:36.943413+0000","last_became_peered":"2023-10-14T18:19:36.943413+0000","last_unstale":"2023-10-14T18:19:39.790357+0000","last_undegraded":"2023-10-14T18:19:39.790357+0000","last_fullsized":"2023-10-14T18:19:39.790357+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:59:02.516834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"35.4","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.962278+0000","last_change":"2023-10-14T18:19:37.960657+0000","last_active":"2023-10-14T18:19:38.962278+0000","last_peered":"2023-10-14T18:19:38.962278+0000","last_clean":"2023-10-14T18:19:38.962278+0000","last_became_active":"2023-10-14T18:19:37.960337+0000","last_became_peered":"2023-10-14T18:19:37.960337+0000","last_unstale":"2023-10-14T18:19:38.962278+0000","last_undegraded":"2023-10-14T18:19:38.962278+0000","last_fullsized":"2023-10-14T18:19:38.962278+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:47:43.180177+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"34.5","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790966+0000","last_change":"2023-10-14T18:19:36.948863+0000","last_active":"2023-10-14T18:19:39.790966+0000","last_peered":"2023-10-14T18:19:39.790966+0000","last_clean":"2023-10-14T18:19:39.790966+0000","last_became_active":"2023-10-14T18:19:36.948404+0000","last_became_peered":"2023-10-14T18:19:36.948404+0000","last_unstale":"2023-10-14T18:19:39.790966+0000","last_undegraded":"2023-10-14T18:19:39.790966+0000","last_fullsized":"2023-10-14T18:19:39.790966+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:25:53.648941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.5","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.792216+0000","last_change":"2023-10-14T18:19:37.951051+0000","last_active":"2023-10-14T18:19:39.792216+0000","last_peered":"2023-10-14T18:19:39.792216+0000","last_clean":"2023-10-14T18:19:39.792216+0000","last_became_active":"2023-10-14T18:19:37.950740+0000","last_became_peered":"2023-10-14T18:19:37.950740+0000","last_unstale":"2023-10-14T18:19:39.792216+0000","last_undegraded":"2023-10-14T18:19:39.792216+0000","last_fullsized":"2023-10-14T18:19:39.792216+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:52:28.800501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"34.4","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.791582+0000","last_change":"2023-10-14T18:19:36.949827+0000","last_active":"2023-10-14T18:19:39.791582+0000","last_peered":"2023-10-14T18:19:39.791582+0000","last_clean":"2023-10-14T18:19:39.791582+0000","last_became_active":"2023-10-14T18:19:36.949453+0000","last_became_peered":"2023-10-14T18:19:36.949453+0000","last_unstale":"2023-10-14T18:19:39.791582+0000","last_undegraded":"2023-10-14T18:19:39.791582+0000","last_fullsized":"2023-10-14T18:19:39.791582+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:41:23.373832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"35.2","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789777+0000","last_change":"2023-10-14T18:19:37.952673+0000","last_active":"2023-10-14T18:19:39.789777+0000","last_peered":"2023-10-14T18:19:39.789777+0000","last_clean":"2023-10-14T18:19:39.789777+0000","last_became_active":"2023-10-14T18:19:37.951715+0000","last_became_peered":"2023-10-14T18:19:37.951715+0000","last_unstale":"2023-10-14T18:19:39.789777+0000","last_undegraded":"2023-10-14T18:19:39.789777+0000","last_fullsized":"2023-10-14T18:19:39.789777+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T21:04:17.373936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"34.3","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.961398+0000","last_change":"2023-10-14T18:19:36.946593+0000","last_active":"2023-10-14T18:19:38.961398+0000","last_peered":"2023-10-14T18:19:38.961398+0000","last_clean":"2023-10-14T18:19:38.961398+0000","last_became_active":"2023-10-14T18:19:36.945216+0000","last_became_peered":"2023-10-14T18:19:36.945216+0000","last_unstale":"2023-10-14T18:19:38.961398+0000","last_undegraded":"2023-10-14T18:19:38.961398+0000","last_fullsized":"2023-10-14T18:19:38.961398+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:02:51.225370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"35.3","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790054+0000","last_change":"2023-10-14T18:19:37.959292+0000","last_active":"2023-10-14T18:19:39.790054+0000","last_peered":"2023-10-14T18:19:39.790054+0000","last_clean":"2023-10-14T18:19:39.790054+0000","last_became_active":"2023-10-14T18:19:37.957642+0000","last_became_peered":"2023-10-14T18:19:37.957642+0000","last_unstale":"2023-10-14T18:19:39.790054+0000","last_undegraded":"2023-10-14T18:19:39.790054+0000","last_fullsized":"2023-10-14T18:19:39.790054+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T03:55:59.116951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.2","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956321+0000","last_change":"2023-10-14T18:19:36.941465+0000","last_active":"2023-10-14T18:19:38.956321+0000","last_peered":"2023-10-14T18:19:38.956321+0000","last_clean":"2023-10-14T18:19:38.956321+0000","last_became_active":"2023-10-14T18:19:36.940751+0000","last_became_peered":"2023-10-14T18:19:36.940751+0000","last_unstale":"2023-10-14T18:19:38.956321+0000","last_undegraded":"2023-10-14T18:19:38.956321+0000","last_fullsized":"2023-10-14T18:19:38.956321+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T18:37:39.264522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.0","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789370+0000","last_change":"2023-10-14T18:19:37.955484+0000","last_active":"2023-10-14T18:19:39.789370+0000","last_peered":"2023-10-14T18:19:39.789370+0000","last_clean":"2023-10-14T18:19:39.789370+0000","last_became_active":"2023-10-14T18:19:37.954820+0000","last_became_peered":"2023-10-14T18:19:37.954820+0000","last_unstale":"2023-10-14T18:19:39.789370+0000","last_undegraded":"2023-10-14T18:19:39.789370+0000","last_fullsized":"2023-10-14T18:19:39.789370+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T01:43:52.613894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"34.1","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.956555+0000","last_change":"2023-10-14T18:19:36.937157+0000","last_active":"2023-10-14T18:19:38.956555+0000","last_peered":"2023-10-14T18:19:38.956555+0000","last_clean":"2023-10-14T18:19:38.956555+0000","last_became_active":"2023-10-14T18:19:36.936803+0000","last_became_peered":"2023-10-14T18:19:36.936803+0000","last_unstale":"2023-10-14T18:19:38.956555+0000","last_undegraded":"2023-10-14T18:19:38.956555+0000","last_fullsized":"2023-10-14T18:19:38.956555+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:29:04.120792+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"35.1","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:38.960210+0000","last_change":"2023-10-14T18:19:37.954030+0000","last_active":"2023-10-14T18:19:38.960210+0000","last_peered":"2023-10-14T18:19:38.960210+0000","last_clean":"2023-10-14T18:19:38.960210+0000","last_became_active":"2023-10-14T18:19:37.952956+0000","last_became_peered":"2023-10-14T18:19:37.952956+0000","last_unstale":"2023-10-14T18:19:38.960210+0000","last_undegraded":"2023-10-14T18:19:38.960210+0000","last_fullsized":"2023-10-14T18:19:38.960210+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T19:18:26.903088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"34.0","version":"0'0","reported_seq":12,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.789209+0000","last_change":"2023-10-14T18:19:36.936461+0000","last_active":"2023-10-14T18:19:39.789209+0000","last_peered":"2023-10-14T18:19:39.789209+0000","last_clean":"2023-10-14T18:19:39.789209+0000","last_became_active":"2023-10-14T18:19:36.936122+0000","last_became_peered":"2023-10-14T18:19:36.936122+0000","last_unstale":"2023-10-14T18:19:39.789209+0000","last_undegraded":"2023-10-14T18:19:39.789209+0000","last_fullsized":"2023-10-14T18:19:39.789209+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":268,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:35.874562+0000","last_clean_scrub_stamp":"2023-10-14T18:19:35.874562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T02:02:23.750313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"35.22","version":"0'0","reported_seq":11,"reported_epoch":271,"state":"active+clean","last_fresh":"2023-10-14T18:19:39.790629+0000","last_change":"2023-10-14T18:19:37.965034+0000","last_active":"2023-10-14T18:19:39.790629+0000","last_peered":"2023-10-14T18:19:39.790629+0000","last_clean":"2023-10-14T18:19:39.790629+0000","last_became_active":"2023-10-14T18:19:37.964720+0000","last_became_peered":"2023-10-14T18:19:37.964720+0000","last_unstale":"2023-10-14T18:19:39.790629+0000","last_undegraded":"2023-10-14T18:19:39.790629+0000","last_fullsized":"2023-10-14T18:19:39.790629+0000","mapping_epoch":269,"log_start":"0'0","ondisk_log_start":"0'0","created":269,"last_epoch_clean":270,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T18:19:36.892219+0000","last_clean_scrub_stamp":"2023-10-14T18:19:36.892219+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-16T05:17:53.867383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"256'381","reported_seq":1367,"reported_epoch":270,"state":"active+clean","last_fresh":"2023-10-14T18:19:37.941346+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:19:37.941346+0000","last_peered":"2023-10-14T18:19:37.941346+0000","last_clean":"2023-10-14T18:19:37.941346+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:19:37.941346+0000","last_undegraded":"2023-10-14T18:19:37.941346+0000","last_fullsized":"2023-10-14T18:19:37.941346+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":35,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":34,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2420,"num_read_kb":5044,"num_write":639,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":381,"ondisk_log_size":381,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57300,"kb_used_data":3908,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314540,"statfs":{"total":96636764160,"available":96578088960,"internally_reserved":0,"allocated":4001792,"data_stored":2296489,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.056},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":1.1100000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.2949999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":0.84599999999999997},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.2390000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.3440000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.405},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.1839999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.46},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.4970000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.5649999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.274}]}]},{"osd":6,"up_from":13,"seq":55834576029,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":48416,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323424,"statfs":{"total":96636764160,"available":96587186176,"internally_reserved":0,"allocated":3424256,"data_stored":1712627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.879,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.135},{"interface":"front","average":{"1min":1.988,"5min":1.329,"15min":1.2549999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.583}]},{"osd":1,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.198,"15min":1.2410000000000001},"min":{"1min":0.74299999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3520000000000001,"15min":1.3660000000000001},"min":{"1min":0.92900000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.272}]},{"osd":2,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.397,"15min":1.3859999999999999},"min":{"1min":0.79000000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.9530000000000001},{"interface":"front","average":{"1min":1.498,"5min":1.4179999999999999,"15min":1.427},"min":{"1min":1.028,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.367}]},{"osd":3,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.88800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.486},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5409999999999999,"15min":1.4830000000000001},"min":{"1min":0.95099999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":14.805999999999999,"5min":14.805999999999999,"15min":14.805999999999999},"last":2.081}]},{"osd":4,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.4399999999999999,"15min":1.4039999999999999},"min":{"1min":1.119,"5min":0.85899999999999999,"15min":0.75800000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.7070000000000001},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.4810000000000001,"15min":1.456},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":2.1859999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.423,"15min":1.423},"min":{"1min":0.88,"5min":0.624,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.804},{"interface":"front","average":{"1min":1.655,"5min":1.6040000000000001,"15min":1.5960000000000001},"min":{"1min":0.97899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":1.8859999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":2.468,"5min":1.796,"15min":1.6779999999999999},"min":{"1min":1.1870000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":16.541,"5min":16.541,"15min":16.541},"last":2.262},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":1.131,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.0059999999999998,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":2.3340000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576030,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65064,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":94306776,"statfs":{"total":96636764160,"available":96570138624,"internally_reserved":0,"allocated":3428352,"data_stored":1708385,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":63175110},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.107,"5min":1.175,"15min":1.2030000000000001},"min":{"1min":0.64300000000000002,"5min":0.60999999999999999,"15min":0.60999999999999999},"max":{"1min":1.952,"5min":2.052,"15min":2.052},"last":0.97199999999999998},{"interface":"front","average":{"1min":1.163,"5min":1.157,"15min":1.23},"min":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.9039999999999999,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":1.2030000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.173,"5min":1.2210000000000001,"15min":1.268},"min":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.7010000000000001,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.87},{"interface":"front","average":{"1min":1.131,"5min":1.2689999999999999,"15min":1.294},"min":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.5269999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.5349999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1499999999999999,"5min":1.1659999999999999,"15min":1.2230000000000001},"min":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"max":{"1min":1.5760000000000001,"5min":1.764,"15min":1.9339999999999999},"last":1.4890000000000001},{"interface":"front","average":{"1min":1.3180000000000001,"5min":1.3720000000000001,"15min":1.395},"min":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"max":{"1min":1.8200000000000001,"5min":1.8779999999999999,"15min":1.923},"last":1.0740000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.2729999999999999,"5min":1.3819999999999999,"15min":1.448},"min":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.7789999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.02},{"interface":"front","average":{"1min":1.3120000000000001,"5min":1.377,"15min":1.407},"min":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.7210000000000001,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.613}]},{"osd":4,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.391,"5min":1.4059999999999999,"15min":1.365},"min":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.79600000000000004},"max":{"1min":1.8360000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.2989999999999999},{"interface":"front","average":{"1min":1.464,"5min":1.45,"15min":1.399},"min":{"1min":0.85599999999999998,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":2.0819999999999999,"5min":2.516,"15min":2.516},"last":1.373}]},{"osd":6,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.4099999999999999,"15min":1.415},"min":{"1min":1.0309999999999999,"5min":0.64900000000000002,"15min":0.64900000000000002},"max":{"1min":2.1539999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.329},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.6319999999999999,"15min":1.629},"min":{"1min":0.91500000000000004,"5min":0.91500000000000004,"15min":0.81799999999999995},"max":{"1min":2.206,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.4279999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:00 2023","interfaces":[{"interface":"back","average":{"1min":1.5469999999999999,"5min":1.5600000000000001,"15min":1.5760000000000001},"min":{"1min":0.96399999999999997,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":2.105,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.464},{"interface":"front","average":{"1min":1.53,"5min":1.5760000000000001,"15min":1.556},"min":{"1min":0.90300000000000002,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.2360000000000002,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.694}]}]},{"osd":4,"up_from":13,"seq":55834576030,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66408,"kb_used_data":3348,"kb_used_omap":1,"kb_used_meta":63038,"kb_avail":94305432,"statfs":{"total":96636764160,"available":96568762368,"internally_reserved":0,"allocated":3428352,"data_stored":1712629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64551366},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":0.73199999999999998},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.1060000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.4119999999999999},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":1.2689999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.2190000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.1739999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":0.90800000000000003},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":1.3680000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.327},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.4990000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.5429999999999999},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.4570000000000001}]}]},{"osd":3,"up_from":13,"seq":55834576030,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56744,"kb_used_data":3352,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315096,"statfs":{"total":96636764160,"available":96578658304,"internally_reserved":0,"allocated":3432448,"data_stored":1714149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.57399999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.417,"5min":1.4870000000000001,"15min":1.5660000000000001},"last":1.004},{"interface":"front","average":{"1min":1.087,"5min":1.0940000000000001,"15min":1.179},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.9970000000000001},"last":1.327}]},{"osd":1,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.92900000000000005,"15min":0.998},"min":{"1min":0.52400000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.919},"last":1.1279999999999999},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0860000000000001,"15min":1.1819999999999999},"min":{"1min":0.61299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5089999999999999,"5min":1.742,"15min":1.8560000000000001},"last":1.1910000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2609999999999999,"15min":1.401},"min":{"1min":0.50900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9099999999999999,"5min":1.9970000000000001,"15min":2.032},"last":1.6559999999999999},{"interface":"front","average":{"1min":1.292,"5min":1.25,"15min":1.3560000000000001},"min":{"1min":0.68899999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.915,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.792}]},{"osd":4,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3560000000000001,"15min":1.4299999999999999},"min":{"1min":0.85399999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.105,"5min":2.105,"15min":2.153},"last":1.8600000000000001},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4430000000000001,"15min":1.478},"min":{"1min":1.034,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.9870000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.5069999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":1.0800000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.758,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.071},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9239999999999999,"5min":2.1360000000000001,"15min":2.891},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.5589999999999999,"15min":1.6259999999999999},"min":{"1min":0.88700000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.2120000000000002,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.627},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.2230000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1419999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.7430000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":0.97399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.452,"5min":3.153,"15min":3.153},"last":1.5740000000000001},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.6060000000000001,"15min":1.6759999999999999},"min":{"1min":1.052,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0339999999999998,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.444}]}]},{"osd":2,"up_from":13,"seq":55834576030,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":56728,"kb_used_data":3332,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315112,"statfs":{"total":96636764160,"available":96578674688,"internally_reserved":0,"allocated":3411968,"data_stored":1706211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.93500000000000005},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":1.3169999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":1.6719999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.381}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.74},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.7090000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.8380000000000001},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.6339999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.5069999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.9159999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.883},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.8129999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.7749999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.9330000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576030,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52576,"kb_used_data":3344,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94319264,"statfs":{"total":96636764160,"available":96582926336,"internally_reserved":0,"allocated":3424256,"data_stored":1712571,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.89600000000000002,"5min":0.94499999999999995,"15min":0.97199999999999998},"min":{"1min":0.52800000000000002,"5min":0.38,"15min":0.38},"max":{"1min":1.3400000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":1.246},{"interface":"front","average":{"1min":0.82299999999999995,"5min":1.0169999999999999,"15min":1.1080000000000001},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":1.1659999999999999,"5min":2.847,"15min":2.847},"last":1.3600000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":0.91100000000000003,"5min":0.99199999999999999,"15min":1.046},"min":{"1min":0.53300000000000003,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":1.4590000000000001,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":1.306},{"interface":"front","average":{"1min":1.1040000000000001,"5min":1.1799999999999999,"15min":1.254},"min":{"1min":0.67300000000000004,"5min":0.48399999999999999,"15min":0.48399999999999999},"max":{"1min":1.6279999999999999,"5min":1.996,"15min":2.6859999999999999},"last":1.645}]},{"osd":3,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.091,"5min":1.2769999999999999,"15min":1.3700000000000001},"min":{"1min":0.66300000000000003,"5min":0.44700000000000001,"15min":0.44700000000000001},"max":{"1min":1.7370000000000001,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.47},{"interface":"front","average":{"1min":1.056,"5min":1.226,"15min":1.296},"min":{"1min":0.68899999999999995,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5609999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.548}]},{"osd":4,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.417,"15min":1.46},"min":{"1min":0.88300000000000001,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":1.9259999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.8919999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.3740000000000001,"15min":1.4239999999999999},"min":{"1min":0.94399999999999995,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":2.1619999999999999,"5min":2.218,"15min":2.899},"last":1.7529999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.47,"5min":1.425,"15min":1.472},"min":{"1min":0.997,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":2.2519999999999998,"5min":2.2519999999999998,"15min":3.004},"last":1.599},{"interface":"front","average":{"1min":1.403,"5min":1.496,"15min":1.5569999999999999},"min":{"1min":0.90400000000000003,"5min":0.73199999999999998,"15min":0.73199999999999998},"max":{"1min":2.0329999999999999,"5min":2.0329999999999999,"15min":2.5129999999999999},"last":1.9470000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.4330000000000001,"5min":1.54,"15min":1.601},"min":{"1min":1.171,"5min":0.81799999999999995,"15min":0.81799999999999995},"max":{"1min":2.0369999999999999,"5min":2.8999999999999999,"15min":2.931},"last":2.367},{"interface":"front","average":{"1min":1.518,"5min":1.6220000000000001,"15min":1.623},"min":{"1min":1.1140000000000001,"5min":0.871,"15min":0.871},"max":{"1min":2.0449999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":2.2509999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:18:40 2023","interfaces":[{"interface":"back","average":{"1min":1.55,"5min":1.619,"15min":1.6890000000000001},"min":{"1min":1.0029999999999999,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":2.3290000000000002,"5min":3.363,"15min":3.363},"last":1.847},{"interface":"front","average":{"1min":1.5169999999999999,"5min":1.651,"15min":1.7090000000000001},"min":{"1min":1.05,"5min":0.89200000000000002,"15min":0.89200000000000002},"max":{"1min":2.206,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":2.1629999999999998}]}]},{"osd":0,"up_from":13,"seq":55834576029,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53168,"kb_used_data":3932,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318672,"statfs":{"total":96636764160,"available":96582320128,"internally_reserved":0,"allocated":4026368,"data_stored":2303543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.0980000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.69899999999999995},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.4159999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":0.97899999999999998},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.6839999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.1950000000000001},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.3009999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.5149999999999999},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":2.004}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":2.077},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.363}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.4750000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.6419999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":34,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":2264,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":34,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":35,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:19:43.353 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2023-10-14T18:19:43.354 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2023-10-14T18:19:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.630+0000 7faea3626640 1 Processor -- start 2023-10-14T18:19:43.631 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.630+0000 7faea3626640 1 -- start start 2023-10-14T18:19:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea3626640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae9c160440 0x7fae9c0d1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea3626640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0b1e10 0x7fae9c06bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea3626640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0b0970 con 0x7fae9c06b8c0 2023-10-14T18:19:43.633 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea3626640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c06b440 con 0x7fae9c160440 2023-10-14T18:19:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea3626640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0d0260 con 0x7fae9c0b1e10 2023-10-14T18:19:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea13c2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fae9c160440 0x7fae9c0d1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea0bc1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fae9c06b8c0 0x7fae9c160b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57878/0 (socket says 172.21.15.29:57878) 2023-10-14T18:19:43.634 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea0bc1640 1 -- 172.21.15.29:0/906023682 learned_addr learned my addr 172.21.15.29:0/906023682 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.632+0000 7faea1bc3640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0b1e10 0x7fae9c06bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.633+0000 7faea13c2640 1 -- 172.21.15.29:0/906023682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0b1e10 msgr2=0x7fae9c06bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:43.636 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.633+0000 7faea13c2640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0b1e10 0x7fae9c06bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.633+0000 7faea13c2640 1 -- 172.21.15.29:0/906023682 >> v1:172.21.15.29:6789/0 conn(0x7fae9c06b8c0 legacy=0x7fae9c160b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.633+0000 7faea13c2640 1 -- 172.21.15.29:0/906023682 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c0cfca0 con 0x7fae9c160440 2023-10-14T18:19:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.633+0000 7fae93fff640 1 -- 172.21.15.29:0/906023682 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249411003 0 0) 0x7fae9c0b0970 con 0x7fae9c06b8c0 2023-10-14T18:19:43.637 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.634+0000 7faea13c2640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c160440 0x7fae9c0d1410 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7fae98009de0 tx=0x7fae9802bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f068d5c2a3b41a75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.634+0000 7fae93fff640 1 -- 172.21.15.29:0/906023682 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9802cc90 con 0x7fae9c160440 2023-10-14T18:19:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.634+0000 7fae93fff640 1 -- 172.21.15.29:0/906023682 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9802cdf0 con 0x7fae9c160440 2023-10-14T18:19:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.634+0000 7fae93fff640 1 -- 172.21.15.29:0/906023682 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae98034660 con 0x7fae9c160440 2023-10-14T18:19:43.638 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 -- 172.21.15.29:0/906023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c160440 msgr2=0x7fae9c0d1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c160440 0x7fae9c0d1410 secure :-1 s=READY pgs=3146 cs=0 l=1 rev1=1 crypto rx=0x7fae98009de0 tx=0x7fae9802bee0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 -- 172.21.15.29:0/906023682 shutdown_connections 2023-10-14T18:19:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c0b1e10 0x7fae9c06bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 --2- 172.21.15.29:0/906023682 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c160440 0x7fae9c0d1410 unknown :-1 s=CLOSED pgs=3146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.639 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 -- 172.21.15.29:0/906023682 >> 172.21.15.29:0/906023682 conn(0x7fae9c099ea0 msgr2=0x7fae9c0d0b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 -- 172.21.15.29:0/906023682 shutdown_connections 2023-10-14T18:19:43.640 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.635+0000 7faea3626640 1 -- 172.21.15.29:0/906023682 wait complete. 2023-10-14T18:19:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.637+0000 7faea3626640 1 Processor -- start 2023-10-14T18:19:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.637+0000 7faea3626640 1 -- start start 2023-10-14T18:19:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.642 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 0x7fae9c15e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c160440 0x7fae9c15e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0b4c10 con 0x7fae9c06b8c0 2023-10-14T18:19:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c15ef00 con 0x7fae9c0b1e10 2023-10-14T18:19:43.643 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea3626640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c15f1e0 con 0x7fae9c160440 2023-10-14T18:19:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea13c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea13c2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43424/0 (socket says 172.21.15.29:43424) 2023-10-14T18:19:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.638+0000 7faea13c2640 1 -- 172.21.15.29:0/2198179798 learned_addr learned my addr 172.21.15.29:0/2198179798 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea0bc1640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 0x7fae9c15e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.644 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea1bc3640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c160440 0x7fae9c15e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 -- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c160440 msgr2=0x7fae9c15e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c160440 0x7fae9c15e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 -- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 msgr2=0x7fae9c15e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 0x7fae9c15e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.645 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 -- 172.21.15.29:0/2198179798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae98009a70 con 0x7fae9c06b8c0 2023-10-14T18:19:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea0bc1640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 0x7fae9c15e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7faea13c2640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7fae98009de0 tx=0x7fae98004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9803e610 con 0x7fae9c06b8c0 2023-10-14T18:19:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9803e790 con 0x7fae9c06b8c0 2023-10-14T18:19:43.646 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.639+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fae9803eb40 con 0x7fae9c06b8c0 2023-10-14T18:19:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.640+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c0b4830 con 0x7fae9c06b8c0 2023-10-14T18:19:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.640+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c0c4ef0 con 0x7fae9c06b8c0 2023-10-14T18:19:43.647 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.641+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c04e160 con 0x7fae9c06b8c0 2023-10-14T18:19:43.648 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.642+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fae9803b050 con 0x7fae9c06b8c0 2023-10-14T18:19:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.643+0000 7fae91ffb640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 0x7fae8006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.643+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fae980abee0 con 0x7fae9c06b8c0 2023-10-14T18:19:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.643+0000 7faea0bc1640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 0x7fae8006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.644+0000 7faea0bc1640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 0x7fae8006d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06edc0 tx=0x7fae8c009040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:43.649 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.647+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fae98076b10 con 0x7fae9c06b8c0 2023-10-14T18:19:43.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.915+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) v1 -- 0x7fae9c0ce1a0 con 0x7fae9c06b8c0 2023-10-14T18:19:43.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.918+0000 7fae91ffb640 1 -- 172.21.15.29:0/2198179798 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v54) v1 ==== 190+0+147 (secure 0 0 0) 0x7fae980390e0 con 0x7fae9c06b8c0 2023-10-14T18:19:43.920 INFO:teuthology.orchestra.run.smithi029.stdout:[client.0] 2023-10-14T18:19:43.920 INFO:teuthology.orchestra.run.smithi029.stdout: key = AQAzxCplhSI0ORAAU+iVceZSGjKskHOunb+tuA== 2023-10-14T18:19:43.920 INFO:teuthology.orchestra.run.smithi029.stdout: caps mds = "allow" 2023-10-14T18:19:43.920 INFO:teuthology.orchestra.run.smithi029.stdout: caps mon = "allow r" 2023-10-14T18:19:43.920 INFO:teuthology.orchestra.run.smithi029.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2023-10-14T18:19:43.921 INFO:teuthology.orchestra.run.smithi029.stderr:updated caps for client.0 2023-10-14T18:19:43.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.921+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 msgr2=0x7fae8006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.921+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 0x7fae8006d570 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fae9c06edc0 tx=0x7fae8c009040 comp rx=0 tx=0).stop 2023-10-14T18:19:43.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.921+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 msgr2=0x7fae9c1f1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:43.923 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.921+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 secure :-1 s=READY pgs=3147 cs=0 l=1 rev1=1 crypto rx=0x7fae98009de0 tx=0x7fae98004a40 comp rx=0 tx=0).stop 2023-10-14T18:19:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 shutdown_connections 2023-10-14T18:19:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fae8006b0b0 0x7fae8006d570 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fae9c160440 0x7fae9c15e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fae9c0b1e10 0x7fae9c15e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.924 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 --2- 172.21.15.29:0/2198179798 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fae9c06b8c0 0x7fae9c1f1e10 unknown :-1 s=CLOSED pgs=3147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:43.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 >> 172.21.15.29:0/2198179798 conn(0x7fae9c099ea0 msgr2=0x7fae9c0ad690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:43.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 shutdown_connections 2023-10-14T18:19:43.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:43.922+0000 7faea3626640 1 -- 172.21.15.29:0/2198179798 wait complete. 2023-10-14T18:19:43.943 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:44.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.175+0000 7f84c1232640 1 Processor -- start 2023-10-14T18:19:44.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.175+0000 7f84c1232640 1 -- start start 2023-10-14T18:19:44.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.176+0000 7f84c1232640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84bc0cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.176+0000 7f84c1232640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f84bc0d0be0 0x7f84bc0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.176+0000 7f84c1232640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84bc15d150 con 0x7f84bc15cd70 2023-10-14T18:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.176+0000 7f84c1232640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84bc0d1180 con 0x7f84bc0d0be0 2023-10-14T18:19:44.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.176+0000 7f84c1232640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0d1460 con 0x7f84bc0b8480 2023-10-14T18:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84bbfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f84bc15cd70 0x7f84bc0d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57888/0 (socket says 172.21.15.29:57888) 2023-10-14T18:19:44.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84bbfff640 1 -- 172.21.15.29:0/869160172 learned_addr learned my addr 172.21.15.29:0/869160172 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84c0a31640 1 --2- 172.21.15.29:0/869160172 >> v2:172.21.15.186:3300/0 conn(0x7f84bc0d0be0 0x7f84bc0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84bb7fe640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84bc0cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316824167 0 0) 0x7f84bc15d150 con 0x7f84bc15cd70 2023-10-14T18:19:44.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.177+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84a4003520 con 0x7f84bc15cd70 2023-10-14T18:19:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525919693 0 0) 0x7f84a4003520 con 0x7f84bc15cd70 2023-10-14T18:19:44.183 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 msgr2=0x7f84bc0cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84bc0cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 >> v2:172.21.15.186:3300/0 conn(0x7f84bc0d0be0 msgr2=0x7f84bc0cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:44.184 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 --2- 172.21.15.29:0/869160172 >> v2:172.21.15.186:3300/0 conn(0x7f84bc0d0be0 0x7f84bc0cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc0d1740 con 0x7f84bc15cd70 2023-10-14T18:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f84ac0019f0 con 0x7f84bc15cd70 2023-10-14T18:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 >> v1:172.21.15.29:6789/0 conn(0x7f84bc15cd70 legacy=0x7f84bc0d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84a4003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.185 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a4003f40 0x7f84a40047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0d1180 con 0x7f84a4003f40 2023-10-14T18:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0d1460 con 0x7f84a4004d00 2023-10-14T18:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a400b980 con 0x7f84bc0b8480 2023-10-14T18:19:44.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.178+0000 7f84bb7fe640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a4003f40 0x7f84a40047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bbfff640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84a4003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84c0a31640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bb7fe640 1 -- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 msgr2=0x7f84a4003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bb7fe640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc0b8480 0x7f84a4003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bb7fe640 1 -- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4004d00 msgr2=0x7f84a4005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bb7fe640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4004d00 0x7f84a4005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.179+0000 7f84bb7fe640 1 -- 172.21.15.29:0/869160172 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a4003520 con 0x7f84a4003f40 2023-10-14T18:19:44.188 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.180+0000 7f84bb7fe640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a4003f40 0x7f84a40047c0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009af0 tx=0x7f84a40057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d26faa45f890dcb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.180+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b002c6b0 con 0x7f84a4003f40 2023-10-14T18:19:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.180+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b002c810 con 0x7f84a4003f40 2023-10-14T18:19:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.180+0000 7f84baffd640 1 -- 172.21.15.29:0/869160172 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b002cba0 con 0x7f84a4003f40 2023-10-14T18:19:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 -- 172.21.15.29:0/869160172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a4003f40 msgr2=0x7f84a40047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 --2- 172.21.15.29:0/869160172 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84a4003f40 0x7f84a40047c0 secure :-1 s=READY pgs=3149 cs=0 l=1 rev1=1 crypto rx=0x7f84b0009af0 tx=0x7f84a40057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c0a31640 1 -- 172.21.15.29:0/869160172 reap_dead start 2023-10-14T18:19:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 -- 172.21.15.29:0/869160172 shutdown_connections 2023-10-14T18:19:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 -- 172.21.15.29:0/869160172 >> 172.21.15.29:0/869160172 conn(0x7f84bc0cf570 msgr2=0x7f84bc0cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 -- 172.21.15.29:0/869160172 shutdown_connections 2023-10-14T18:19:44.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.181+0000 7f84c1232640 1 -- 172.21.15.29:0/869160172 wait complete. 2023-10-14T18:19:44.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.183+0000 7f84c1232640 1 Processor -- start 2023-10-14T18:19:44.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.183+0000 7f84c1232640 1 -- start start 2023-10-14T18:19:44.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4003f40 0x7f84bc14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc15d330 0x7f84bc0b3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc15f090 con 0x7f84bc15e6d0 2023-10-14T18:19:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc0c9cd0 con 0x7f84a4003f40 2023-10-14T18:19:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.184+0000 7f84c1232640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc15e440 con 0x7f84bc15d330 2023-10-14T18:19:44.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43440/0 (socket says 172.21.15.29:43440) 2023-10-14T18:19:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 -- 172.21.15.29:0/578786719 learned_addr learned my addr 172.21.15.29:0/578786719 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84bbfff640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4003f40 0x7f84bc14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 -- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc15d330 msgr2=0x7f84bc0b3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc15d330 0x7f84bc0b3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 -- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4003f40 msgr2=0x7f84bc14fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4003f40 0x7f84bc14fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 -- 172.21.15.29:0/578786719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84bc1500e0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.185+0000 7f84c0a31640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0b24f0 tx=0x7f84b4011920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:44.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.186+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b4004440 con 0x7f84bc15e6d0 2023-10-14T18:19:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.186+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84b4003610 con 0x7f84bc15e6d0 2023-10-14T18:19:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.186+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f84b40039e0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.186+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc15d830 con 0x7f84bc15e6d0 2023-10-14T18:19:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.186+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc15ddf0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.187+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc0d28f0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.188+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f84b4003b40 con 0x7f84bc15e6d0 2023-10-14T18:19:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.190+0000 7f84b8ff9640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 0x7f84a806d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.190+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f84b4089ef0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.190+0000 7f84bbfff640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 0x7f84a806d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:44.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.191+0000 7f84bbfff640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 0x7f84a806d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84ac002230 tx=0x7f84ac0012d0 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:44.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.194+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f84b4054b20 con 0x7f84bc15e6d0 2023-10-14T18:19:44.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.433+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84bc04e9b0 con 0x7f84bc15e6d0 2023-10-14T18:19:44.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.434+0000 7f84b8ff9640 1 -- 172.21.15.29:0/578786719 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f84b4058140 con 0x7f84bc15e6d0 2023-10-14T18:19:44.435 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:44.435 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:19:44.436 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 msgr2=0x7f84a806d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 0x7f84a806d5a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f84ac002230 tx=0x7f84ac0012d0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 msgr2=0x7f84bc15eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 secure :-1 s=READY pgs=3150 cs=0 l=1 rev1=1 crypto rx=0x7f84bc0b24f0 tx=0x7f84b4011920 comp rx=0 tx=0).stop 2023-10-14T18:19:44.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 shutdown_connections 2023-10-14T18:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f84a806b0e0 0x7f84a806d5a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f84bc15e6d0 0x7f84bc15eb50 unknown :-1 s=CLOSED pgs=3150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f84bc15d330 0x7f84bc0b3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 --2- 172.21.15.29:0/578786719 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f84a4003f40 0x7f84bc14fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:44.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 >> 172.21.15.29:0/578786719 conn(0x7f84bc0cf570 msgr2=0x7f84bc0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 shutdown_connections 2023-10-14T18:19:44.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:44.437+0000 7f84c1232640 1 -- 172.21.15.29:0/578786719 wait complete. 2023-10-14T18:19:44.861 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 139, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2023-10-14T18:19:37.403465+0000', 'modified': '2023-10-14T18:19:40.808405+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 12843}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_12843': {'gid': 12843, 'name': 'd', 'rank': 0, 'incarnation': 135, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.186:6835/3411376277', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.186:6833', 'nonce': 3411376277}, {'type': 'v1', 'addr': '172.21.15.186:6835', 'nonce': 3411376277}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138322906710015, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [35], 'metadata_pool': 34, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2023-10-14T18:19:44.862 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2023-10-14T18:19:44.862 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2023-10-14T18:19:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.101+0000 7f5b23fff640 1 Processor -- start 2023-10-14T18:19:45.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.101+0000 7f5b23fff640 1 -- start start 2023-10-14T18:19:45.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.103+0000 7f5b23fff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b240ad970 0x7f5b240cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.103+0000 7f5b23fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.103+0000 7f5b23fff640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b2415b9d0 con 0x7f5b2406c040 2023-10-14T18:19:45.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.103+0000 7f5b23fff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b2415bcb0 con 0x7f5b240ad970 2023-10-14T18:19:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.103+0000 7f5b23fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b2415bf90 con 0x7f5b240ccfb0 2023-10-14T18:19:45.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b1a7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5b240ad970 0x7f5b240cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:57584/0 (socket says 172.21.15.29:57584) 2023-10-14T18:19:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 -- 172.21.15.29:0/193476213 learned_addr learned my addr 172.21.15.29:0/193476213 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 -- 172.21.15.29:0/193476213 >> v2:172.21.15.186:3300/0 conn(0x7f5b240ad970 msgr2=0x7f5b240cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 --2- 172.21.15.29:0/193476213 >> v2:172.21.15.186:3300/0 conn(0x7f5b240ad970 0x7f5b240cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.104+0000 7f5b237fe640 1 -- 172.21.15.29:0/193476213 >> v1:172.21.15.186:6789/0 conn(0x7f5b2406c040 legacy=0x7f5b2406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.105+0000 7f5b237fe640 1 -- 172.21.15.29:0/193476213 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b240cc270 con 0x7f5b240ccfb0 2023-10-14T18:19:45.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.106+0000 7f5b237fe640 1 --2- 172.21.15.29:0/193476213 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009e50 tx=0x7f5b1c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7155245fd43b603f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b227fc640 1 -- 172.21.15.29:0/193476213 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c02cb00 con 0x7f5b240ccfb0 2023-10-14T18:19:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b227fc640 1 -- 172.21.15.29:0/193476213 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c02cc60 con 0x7f5b240ccfb0 2023-10-14T18:19:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b227fc640 1 -- 172.21.15.29:0/193476213 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c0343f0 con 0x7f5b240ccfb0 2023-10-14T18:19:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b23fff640 1 -- 172.21.15.29:0/193476213 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 msgr2=0x7f5b240cd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b23fff640 1 --2- 172.21.15.29:0/193476213 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009e50 tx=0x7f5b1c02bf60 comp rx=0 tx=0).stop 2023-10-14T18:19:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b23fff640 1 -- 172.21.15.29:0/193476213 shutdown_connections 2023-10-14T18:19:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b23fff640 1 --2- 172.21.15.29:0/193476213 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b240cd460 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.107+0000 7f5b23fff640 1 --2- 172.21.15.29:0/193476213 >> v2:172.21.15.186:3300/0 conn(0x7f5b240ad970 0x7f5b240cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.110 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.108+0000 7f5b23fff640 1 -- 172.21.15.29:0/193476213 >> 172.21.15.29:0/193476213 conn(0x7f5b240ce6a0 msgr2=0x7f5b240ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.108+0000 7f5b23fff640 1 -- 172.21.15.29:0/193476213 shutdown_connections 2023-10-14T18:19:45.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.108+0000 7f5b23fff640 1 -- 172.21.15.29:0/193476213 wait complete. 2023-10-14T18:19:45.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.109+0000 7f5b23fff640 1 Processor -- start 2023-10-14T18:19:45.113 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.110+0000 7f5b23fff640 1 -- start start 2023-10-14T18:19:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 0x7f5b24155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b24156380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b240c5770 con 0x7f5b2406c040 2023-10-14T18:19:45.114 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241fa4d0 con 0x7f5b240ad970 2023-10-14T18:19:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b23fff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b241fa7b0 con 0x7f5b240ccfb0 2023-10-14T18:19:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b22ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b22ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43446/0 (socket says 172.21.15.29:43446) 2023-10-14T18:19:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b22ffd640 1 -- 172.21.15.29:0/1037048355 learned_addr learned my addr 172.21.15.29:0/1037048355 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 0x7f5b24155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b22ffd640 1 -- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 msgr2=0x7f5b24156380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.111+0000 7f5b237fe640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b24156380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b24156380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b22ffd640 1 -- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 msgr2=0x7f5b24155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 0x7f5b24155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b22ffd640 1 -- 172.21.15.29:0/1037048355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b1c009b30 con 0x7f5b2406c040 2023-10-14T18:19:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 0x7f5b24155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c450 tx=0x7f5b1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b1bfff640 1 -- 172.21.15.29:0/1037048355 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b140052d0 con 0x7f5b2406c040 2023-10-14T18:19:45.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b1bfff640 1 -- 172.21.15.29:0/1037048355 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b14005430 con 0x7f5b2406c040 2023-10-14T18:19:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b241faa90 con 0x7f5b2406c040 2023-10-14T18:19:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b1bfff640 1 -- 172.21.15.29:0/1037048355 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14012870 con 0x7f5b2406c040 2023-10-14T18:19:45.118 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.112+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b241540d0 con 0x7f5b2406c040 2023-10-14T18:19:45.120 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.116+0000 7f5b1bfff640 1 -- 172.21.15.29:0/1037048355 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f5b140129d0 con 0x7f5b2406c040 2023-10-14T18:19:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.117+0000 7f5b1bfff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 0x7f5b0406d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.117+0000 7f5b1bfff640 1 -- 172.21.15.29:0/1037048355 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5b14089640 con 0x7f5b2406c040 2023-10-14T18:19:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.117+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 0x7f5b0406d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.117+0000 7f5b2bfaf740 1 Processor -- start 2023-10-14T18:19:45.121 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.117+0000 7f5b2bfaf740 1 -- start start 2023-10-14T18:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 0x7f5b0406d5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b240ac860 tx=0x7f5b0c008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604b82070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604bb1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.122 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604aa27a0 con 0x555604b825b0 2023-10-14T18:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604af4740 con 0x555604baf550 2023-10-14T18:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.118+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x555604af1e30 con 0x555604baf170 2023-10-14T18:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43450/0 (socket says 172.21.15.29:43450) 2023-10-14T18:19:45.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 -- 172.21.15.29:0/2651022477 learned_addr learned my addr 172.21.15.29:0/2651022477 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604b82070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604bb1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 -- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 msgr2=0x555604b82070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604b82070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.124 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 -- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 msgr2=0x555604bb1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604bb1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.119+0000 7f5b237fe640 1 -- 172.21.15.29:0/2651022477 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555604ab1910 con 0x555604b825b0 2023-10-14T18:19:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b237fe640 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009f70 tx=0x7f5b1c02e000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49fff9d360d80c7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b18ff9640 1 -- 172.21.15.29:0/2651022477 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c02cb00 con 0x555604b825b0 2023-10-14T18:19:45.125 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b18ff9640 1 -- 172.21.15.29:0/2651022477 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c03e430 con 0x555604b825b0 2023-10-14T18:19:45.126 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b18ff9640 1 -- 172.21.15.29:0/2651022477 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b1c03e7a0 con 0x555604b825b0 2023-10-14T18:19:45.129 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2651022477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 msgr2=0x555604bb2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.120+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 secure :-1 s=READY pgs=3152 cs=0 l=1 rev1=1 crypto rx=0x7f5b1c009f70 tx=0x7f5b1c02e000 comp rx=0 tx=0).stop 2023-10-14T18:19:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2651022477 shutdown_connections 2023-10-14T18:19:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604b825b0 0x555604bb2440 unknown :-1 s=CLOSED pgs=3152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604bb1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2651022477 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604b82070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2651022477 >> 172.21.15.29:0/2651022477 conn(0x555604b84380 msgr2=0x555604bccf00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2651022477 shutdown_connections 2023-10-14T18:19:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2651022477 wait complete. 2023-10-14T18:19:45.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.121+0000 7f5b2bfaf740 1 -- start start 2023-10-14T18:19:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.122+0000 7f5b2bfaf740 1 Processor -- start 2023-10-14T18:19:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf170 0x555604bcd590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604bcec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604aed720 con 0x555604baf550 2023-10-14T18:19:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604a8b970 con 0x555604baf170 2023-10-14T18:19:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x555604a8b610 con 0x555604bc4640 2023-10-14T18:19:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b2bfaf740 10 client.0 _ll_register_callbacks cb 0x555604aad9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:19:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b1a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.133 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b1a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43464/0 (socket says 172.21.15.29:43464) 2023-10-14T18:19:45.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/2138140098 learned_addr learned my addr 172.21.15.29:0/2138140098 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf170 0x555604bcd590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.123+0000 7f5b237fe640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604bcec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 msgr2=0x555604bcec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.134 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604bcec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf170 msgr2=0x555604bcd590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf170 0x555604bcd590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/2138140098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b240cc270 con 0x555604baf550 2023-10-14T18:19:45.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f5b240c2160 tx=0x7f5b0c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fbb834560624ddb server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.135 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:19:45.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c016970 con 0x555604baf550 2023-10-14T18:19:45.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x555604b14160 con 0x555604baf550 2023-10-14T18:19:45.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5b0c016ad0 con 0x555604baf550 2023-10-14T18:19:45.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b0c016d20 con 0x555604baf550 2023-10-14T18:19:45.136 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b2bfaf740 10 client.12963 fetch_fsmap learned FSMap version 139 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.124+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x555604aebda0 con 0x555604baf550 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x555604b14160 con 0x555604baf550 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5b0c022420 con 0x555604baf550 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 10 client.12963 fetch_fsmap finished waiting for FSMap version 139 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 10 client.12963 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T18:19:45.137 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 20 client.12963 populate_metadata read hostname 'smithi029' 2023-10-14T18:19:45.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604bb5b90 0x555604bb0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 4 client.12963 mds_command: new command op to 12843 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2023-10-14T18:19:45.138 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.125+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 --> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555604abfcc0 con 0x555604bb5b90 2023-10-14T18:19:45.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.126+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604bb5b90 0x555604bb0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.127+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604bb5b90 0x555604bb0f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=acdd8e9b580789f5 server_cookie=ca19dc9aa76033f4 in_seq=0 out_seq=0 2023-10-14T18:19:45.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.127+0000 7f5b01ffb640 10 client.12963 ms_handle_connect on v2:172.21.15.186:6833/3411376277 2023-10-14T18:19:45.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b01ffb640 1 -- 172.21.15.29:0/2138140098 <== mds.0 v2:172.21.15.186:6833/3411376277 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+18209 (crc 0 0 0) 0x555604abfcc0 con 0x555604bb5b90 2023-10-14T18:19:45.139 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b01ffb640 10 client.12963 handle_command_reply: tid=0 2023-10-14T18:19:45.140 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 client.12963 shutdown 2023-10-14T18:19:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 msgr2=0x555604bc4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 secure :-1 s=READY pgs=3153 cs=0 l=1 rev1=1 crypto rx=0x7f5b240c2160 tx=0x7f5b0c004730 comp rx=0 tx=0).stop 2023-10-14T18:19:45.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 shutdown_connections 2023-10-14T18:19:45.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604bb5b90 0x555604bb0f30 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604bcec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604bc4100 unknown :-1 s=CLOSED pgs=3153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/2138140098 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf170 0x555604bcd590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b01ffb640 0 client.12963 ms_handle_reset on v2:172.21.15.186:6833/3411376277 2023-10-14T18:19:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.129+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 >> 172.21.15.29:0/2138140098 conn(0x555604b825b0 msgr2=0x555604bd0410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.130+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 shutdown_connections 2023-10-14T18:19:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.130+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/2138140098 wait complete. 2023-10-14T18:19:45.143 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.130+0000 7f5b2bfaf740 20 client.12963 trim_cache size 0 max 16384 2023-10-14T18:19:45.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.147+0000 7f5b2bfaf740 1 Processor -- start 2023-10-14T18:19:45.148 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.147+0000 7f5b2bfaf740 1 -- start start 2023-10-14T18:19:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604ba9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604b812e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.149 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604851f20 con 0x555604baf170 2023-10-14T18:19:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604be64c0 con 0x555604baf550 2023-10-14T18:19:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x555604bcd760 con 0x555604bc4640 2023-10-14T18:19:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b22ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.150 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b22ffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43466/0 (socket says 172.21.15.29:43466) 2023-10-14T18:19:45.151 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.148+0000 7f5b22ffd640 1 -- 172.21.15.29:0/3587936858 learned_addr learned my addr 172.21.15.29:0/3587936858 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b237fe640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604b812e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.153 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604ba9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 -- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 msgr2=0x555604b812e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604b812e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 -- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 msgr2=0x555604ba9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604ba9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.154 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 -- 172.21.15.29:0/3587936858 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555604a8ab80 con 0x555604baf170 2023-10-14T18:19:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604ba9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f5b14087900 tx=0x7f5b14005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6cdecb717a839b3a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b197fa640 1 -- 172.21.15.29:0/3587936858 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b140562f0 con 0x555604baf170 2023-10-14T18:19:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b197fa640 1 -- 172.21.15.29:0/3587936858 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1405e0b0 con 0x555604baf170 2023-10-14T18:19:45.155 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.149+0000 7f5b197fa640 1 -- 172.21.15.29:0/3587936858 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b14012c80 con 0x555604baf170 2023-10-14T18:19:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/3587936858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 msgr2=0x555604be6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 secure :-1 s=READY pgs=3154 cs=0 l=1 rev1=1 crypto rx=0x7f5b14087900 tx=0x7f5b14005040 comp rx=0 tx=0).stop 2023-10-14T18:19:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/3587936858 shutdown_connections 2023-10-14T18:19:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604bc4640 0x555604b812e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604baf550 0x555604ba9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/3587936858 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf170 0x555604be6a80 unknown :-1 s=CLOSED pgs=3154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/3587936858 >> 172.21.15.29:0/3587936858 conn(0x555604b825b0 msgr2=0x555604b85c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/3587936858 shutdown_connections 2023-10-14T18:19:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.150+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/3587936858 wait complete. 2023-10-14T18:19:45.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.151+0000 7f5b2bfaf740 1 -- start start 2023-10-14T18:19:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.151+0000 7f5b2bfaf740 1 Processor -- start 2023-10-14T18:19:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604bb2260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 0x555604beb3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604bad040 con 0x555604baf550 2023-10-14T18:19:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x555604b9c770 con 0x555604bc4640 2023-10-14T18:19:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x555604b9c490 con 0x555604baf170 2023-10-14T18:19:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.152+0000 7f5b2bfaf740 10 client.0 _ll_register_callbacks cb 0x7f5b0c0097d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2023-10-14T18:19:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.159 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43480/0 (socket says 172.21.15.29:43480) 2023-10-14T18:19:45.160 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/1201909244 learned_addr learned my addr 172.21.15.29:0/1201909244 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b237fe640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 0x555604beb3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604bb2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 msgr2=0x555604bb2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604bb2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 msgr2=0x555604beb3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 0x555604beb3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 -- 172.21.15.29:0/1201909244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b241540d0 con 0x555604baf550 2023-10-14T18:19:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b237fe640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 0x555604beb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b1a7fc640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c008010 tx=0x7f5b0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7f1bc0e7f1f2546 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.153+0000 7f5b01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.29:3300/0 2023-10-14T18:19:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c022b40 con 0x555604baf550 2023-10-14T18:19:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5b1c03e430 con 0x555604baf550 2023-10-14T18:19:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mon.0 v2:172.21.15.29:3300/0 2 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f5b0c039070 con 0x555604baf550 2023-10-14T18:19:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_get_version_reply(handle=1 version=139) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5b0c022ca0 con 0x555604baf550 2023-10-14T18:19:45.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b2bfaf740 10 client.12969 fetch_fsmap learned FSMap version 139 2023-10-14T18:19:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,fsmap=139,monmap=2+}) v3 -- 0x555604ab0170 con 0x555604baf550 2023-10-14T18:19:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.154+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mon.0 v2:172.21.15.29:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b1c03e430 con 0x555604baf550 2023-10-14T18:19:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mon.0 v2:172.21.15.29:3300/0 5 ==== fsmap(e 139) v1 ==== 3294+0+0 (secure 0 0 0) 0x7f5b0c0167f0 con 0x555604baf550 2023-10-14T18:19:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 10 client.12969 fetch_fsmap finished waiting for FSMap version 139 2023-10-14T18:19:45.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 10 client.12969 resolve_mds: resolved name 'd' to daemon mds.d 2023-10-14T18:19:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 20 client.12969 populate_metadata read hostname 'smithi029' 2023-10-14T18:19:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604c1aa30 0x555604c1ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 4 client.12969 mds_command: new command op to 12843 tid=0 multi_id=0 [{"prefix": "status"}] 2023-10-14T18:19:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 --> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] -- command(tid 0: {"prefix": "status"}) v1 -- 0x555604ae6700 con 0x555604c1aa30 2023-10-14T18:19:45.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.155+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604c1aa30 0x555604c1ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.156+0000 7f5b22ffd640 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604c1aa30 0x555604c1ae10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bbbb772d324a768 server_cookie=8ee1148a7a952da3 in_seq=0 out_seq=0 2023-10-14T18:19:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.156+0000 7f5b01ffb640 10 client.12969 ms_handle_connect on v2:172.21.15.186:6833/3411376277 2023-10-14T18:19:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b01ffb640 1 -- 172.21.15.29:0/1201909244 <== mds.0 v2:172.21.15.186:6833/3411376277 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+335 (crc 0 0 0) 0x555604ae6700 con 0x555604c1aa30 2023-10-14T18:19:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b01ffb640 10 client.12969 handle_command_reply: tid=0 2023-10-14T18:19:45.168 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 client.12969 shutdown 2023-10-14T18:19:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 msgr2=0x555604c19ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 secure :-1 s=READY pgs=3155 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c008010 tx=0x7f5b0c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 shutdown_connections 2023-10-14T18:19:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:6833/3411376277,v1:172.21.15.186:6835/3411376277] conn(0x555604c1aa30 0x555604c1ae10 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.169 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x555604bc4640 0x555604beb3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x555604baf550 0x555604c19ff0 unknown :-1 s=CLOSED pgs=3155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b01ffb640 0 client.12969 ms_handle_reset on v2:172.21.15.186:6833/3411376277 2023-10-14T18:19:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 --2- 172.21.15.29:0/1201909244 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x555604baf170 0x555604bb2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 >> 172.21.15.29:0/1201909244 conn(0x555604b825b0 msgr2=0x555604b7d810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 shutdown_connections 2023-10-14T18:19:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 1 -- 172.21.15.29:0/1201909244 wait complete. 2023-10-14T18:19:45.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.157+0000 7f5b2bfaf740 20 client.12969 trim_cache size 0 max 16384 2023-10-14T18:19:45.171 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:19:45.172 INFO:teuthology.orchestra.run.smithi029.stdout: "cluster_fsid": "f6aee177-7aff-4117-99f6-f2cafd5adc35", 2023-10-14T18:19:45.172 INFO:teuthology.orchestra.run.smithi029.stdout: "whoami": 0, 2023-10-14T18:19:45.172 INFO:teuthology.orchestra.run.smithi029.stdout: "id": 12843, 2023-10-14T18:19:45.172 INFO:teuthology.orchestra.run.smithi029.stdout: "want_state": "up:active", 2023-10-14T18:19:45.173 INFO:teuthology.orchestra.run.smithi029.stdout: "state": "up:active", 2023-10-14T18:19:45.173 INFO:teuthology.orchestra.run.smithi029.stdout: "fs_name": "cephfs", 2023-10-14T18:19:45.173 INFO:teuthology.orchestra.run.smithi029.stdout: "rank_uptime": 6.8723131820000001, 2023-10-14T18:19:45.173 INFO:teuthology.orchestra.run.smithi029.stdout: "mdsmap_epoch": 139, 2023-10-14T18:19:45.173 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch": 271, 2023-10-14T18:19:45.174 INFO:teuthology.orchestra.run.smithi029.stdout: "osdmap_epoch_barrier": 271, 2023-10-14T18:19:45.174 INFO:teuthology.orchestra.run.smithi029.stdout: "uptime": 13.026986534000001 2023-10-14T18:19:45.174 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:19:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 msgr2=0x7f5b0406d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 0x7f5b0406d5a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5b240ac860 tx=0x7f5b0c008040 comp rx=0 tx=0).stop 2023-10-14T18:19:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 msgr2=0x7f5b24153b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 secure :-1 s=READY pgs=3151 cs=0 l=1 rev1=1 crypto rx=0x7f5b1400c450 tx=0x7f5b1400c910 comp rx=0 tx=0).stop 2023-10-14T18:19:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 shutdown_connections 2023-10-14T18:19:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f5b0406b0e0 0x7f5b0406d5a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5b240ccfb0 0x7f5b24156380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5b240ad970 0x7f5b24155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 --2- 172.21.15.29:0/1037048355 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5b2406c040 0x7f5b24153b90 unknown :-1 s=CLOSED pgs=3151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 >> 172.21.15.29:0/1037048355 conn(0x7f5b240ce6a0 msgr2=0x7f5b2406c420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 shutdown_connections 2023-10-14T18:19:45.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.158+0000 7f5b23fff640 1 -- 172.21.15.29:0/1037048355 wait complete. 2023-10-14T18:19:45.178 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2023-10-14T18:19:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.394+0000 7fbe8b3e9640 1 Processor -- start 2023-10-14T18:19:45.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.394+0000 7fbe8b3e9640 1 -- start start 2023-10-14T18:19:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8b3e9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fbe8415aad0 0x7fbe840cc270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8b3e9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fbe840712c0 0x7fbe84070000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8b3e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe840b4620 con 0x7fbe84070d70 2023-10-14T18:19:45.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8b3e9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe840b4340 con 0x7fbe840712c0 2023-10-14T18:19:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8b3e9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe8406f520 con 0x7fbe8415aad0 2023-10-14T18:19:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe89be6640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbe84070d70 0x7fbe8406bf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57890/0 (socket says 172.21.15.29:57890) 2023-10-14T18:19:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe89be6640 1 -- 172.21.15.29:0/1126221061 learned_addr learned my addr 172.21.15.29:0/1126221061 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:45.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8abe8640 1 --2- 172.21.15.29:0/1126221061 >> v2:172.21.15.29:3300/0 conn(0x7fbe840712c0 0x7fbe84070000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.396+0000 7fbe8a3e7640 1 --2- 172.21.15.29:0/1126221061 >> v2:172.21.15.186:3300/0 conn(0x7fbe8415aad0 0x7fbe840cc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe893e5640 1 -- 172.21.15.29:0/1126221061 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083794312 0 0) 0x7fbe840b4620 con 0x7fbe84070d70 2023-10-14T18:19:45.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe893e5640 1 -- 172.21.15.29:0/1126221061 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe70003520 con 0x7fbe84070d70 2023-10-14T18:19:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe8abe8640 1 -- 172.21.15.29:0/1126221061 >> v2:172.21.15.186:3300/0 conn(0x7fbe8415aad0 msgr2=0x7fbe840cc270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe8abe8640 1 --2- 172.21.15.29:0/1126221061 >> v2:172.21.15.186:3300/0 conn(0x7fbe8415aad0 0x7fbe840cc270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe8abe8640 1 -- 172.21.15.29:0/1126221061 >> v1:172.21.15.29:6789/0 conn(0x7fbe84070d70 legacy=0x7fbe8406bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.397+0000 7fbe8abe8640 1 -- 172.21.15.29:0/1126221061 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe8406f800 con 0x7fbe840712c0 2023-10-14T18:19:45.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.398+0000 7fbe8abe8640 1 --2- 172.21.15.29:0/1126221061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe84070000 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fbe840cf4f0 tx=0x7fbe7c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c4a7cbd0b5e8d5e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.398+0000 7fbe893e5640 1 -- 172.21.15.29:0/1126221061 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c02cd10 con 0x7fbe840712c0 2023-10-14T18:19:45.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.398+0000 7fbe893e5640 1 -- 172.21.15.29:0/1126221061 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe7c02ce70 con 0x7fbe840712c0 2023-10-14T18:19:45.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.398+0000 7fbe893e5640 1 -- 172.21.15.29:0/1126221061 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c034640 con 0x7fbe840712c0 2023-10-14T18:19:45.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/1126221061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 msgr2=0x7fbe84070000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/1126221061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe84070000 secure :-1 s=READY pgs=3157 cs=0 l=1 rev1=1 crypto rx=0x7fbe840cf4f0 tx=0x7fbe7c02bf60 comp rx=0 tx=0).stop 2023-10-14T18:19:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/1126221061 shutdown_connections 2023-10-14T18:19:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/1126221061 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe84070000 unknown :-1 s=CLOSED pgs=3157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/1126221061 >> v2:172.21.15.186:3300/0 conn(0x7fbe8415aad0 0x7fbe840cc270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/1126221061 >> 172.21.15.29:0/1126221061 conn(0x7fbe840b0a90 msgr2=0x7fbe840ce6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/1126221061 shutdown_connections 2023-10-14T18:19:45.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.399+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/1126221061 wait complete. 2023-10-14T18:19:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.401+0000 7fbe8b3e9640 1 Processor -- start 2023-10-14T18:19:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.401+0000 7fbe8b3e9640 1 -- start start 2023-10-14T18:19:45.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe84070d70 0x7fbe840c4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe8415aad0 0x7fbe841f20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840c3a90 con 0x7fbe840712c0 2023-10-14T18:19:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe840c2bc0 con 0x7fbe84070d70 2023-10-14T18:19:45.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe8b3e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe8415c310 con 0x7fbe8415aad0 2023-10-14T18:19:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe89be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe89be6640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43502/0 (socket says 172.21.15.29:43502) 2023-10-14T18:19:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.402+0000 7fbe89be6640 1 -- 172.21.15.29:0/2943126163 learned_addr learned my addr 172.21.15.29:0/2943126163 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe8a3e7640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe84070d70 0x7fbe840c4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe8abe8640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe8415aad0 0x7fbe841f20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 -- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe8415aad0 msgr2=0x7fbe841f20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe8415aad0 0x7fbe841f20f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 -- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe84070d70 msgr2=0x7fbe840c4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe84070d70 0x7fbe840c4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 -- 172.21.15.29:0/2943126163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe7c009a70 con 0x7fbe840712c0 2023-10-14T18:19:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe89be6640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400c430 tx=0x7fbe7400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe74005ce0 con 0x7fbe840712c0 2023-10-14T18:19:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe74005e40 con 0x7fbe840712c0 2023-10-14T18:19:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe740072f0 con 0x7fbe840712c0 2023-10-14T18:19:45.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.403+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe840c4fc0 con 0x7fbe840712c0 2023-10-14T18:19:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.404+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe840c30f0 con 0x7fbe840712c0 2023-10-14T18:19:45.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.405+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe840c3bd0 con 0x7fbe840712c0 2023-10-14T18:19:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.406+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7fbe74007450 con 0x7fbe840712c0 2023-10-14T18:19:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.407+0000 7fbe7affd640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 0x7fbe6806d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.407+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fbe74089150 con 0x7fbe840712c0 2023-10-14T18:19:45.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.407+0000 7fbe8a3e7640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 0x7fbe6806d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.408+0000 7fbe8a3e7640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 0x7fbe6806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbe80006fd0 tx=0x7fbe80008040 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:45.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.412+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbe740578a0 con 0x7fbe840712c0 2023-10-14T18:19:45.652 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.651+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbe8404e9b0 con 0x7fbe840712c0 2023-10-14T18:19:45.653 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.652+0000 7fbe7affd640 1 -- 172.21.15.29:0/2943126163 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v139) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbe74057430 con 0x7fbe840712c0 2023-10-14T18:19:45.653 INFO:teuthology.orchestra.run.smithi029.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2023-10-14T18:19:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 msgr2=0x7fbe6806d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 0x7fbe6806d570 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fbe80006fd0 tx=0x7fbe80008040 comp rx=0 tx=0).stop 2023-10-14T18:19:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 msgr2=0x7fbe841f1bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:45.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 secure :-1 s=READY pgs=3158 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400c430 tx=0x7fbe7400c8f0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 shutdown_connections 2023-10-14T18:19:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fbe6806b0b0 0x7fbe6806d570 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbe8415aad0 0x7fbe841f20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbe840712c0 0x7fbe841f1bb0 unknown :-1 s=CLOSED pgs=3158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.657 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 --2- 172.21.15.29:0/2943126163 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbe84070d70 0x7fbe840c4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 >> 172.21.15.29:0/2943126163 conn(0x7fbe840b0a90 msgr2=0x7fbe840ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 shutdown_connections 2023-10-14T18:19:45.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:45.655+0000 7fbe8b3e9640 1 -- 172.21.15.29:0/2943126163 wait complete. 2023-10-14T18:19:45.674 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2023-10-14T18:19:45.675 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2023-10-14T18:19:45.675 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi029.front.sepia.ceph.com 2023-10-14T18:19:45.676 INFO:tasks.cephfs.mount:self.client.name = client.0 2023-10-14T18:19:45.676 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:45.676 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2023-10-14T18:19:45.677 INFO:tasks.cephfs.mount:self.cephfs_mntpt = / 2023-10-14T18:19:45.677 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2023-10-14T18:19:45.677 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:19:45.678 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:45.678 DEBUG:teuthology.orchestra.run.smithi029:> ip addr 2023-10-14T18:19:45.735 INFO:teuthology.orchestra.run.smithi029.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2023-10-14T18:19:45.735 INFO:teuthology.orchestra.run.smithi029.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2023-10-14T18:19:45.735 INFO:teuthology.orchestra.run.smithi029.stdout: inet 127.0.0.1/8 scope host lo 2023-10-14T18:19:45.735 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.736 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 ::1/128 scope host 2023-10-14T18:19:45.736 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.736 INFO:teuthology.orchestra.run.smithi029.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:19:45.736 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a0 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:19:45.736 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f0 2023-10-14T18:19:45.737 INFO:teuthology.orchestra.run.smithi029.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:19:45.737 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:6b:fd:a1 brd ff:ff:ff:ff:ff:ff 2023-10-14T18:19:45.737 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp7s0f1 2023-10-14T18:19:45.737 INFO:teuthology.orchestra.run.smithi029.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2023-10-14T18:19:45.737 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5a brd ff:ff:ff:ff:ff:ff 2023-10-14T18:19:45.738 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f0 2023-10-14T18:19:45.738 INFO:teuthology.orchestra.run.smithi029.stdout: inet 172.21.15.29/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2023-10-14T18:19:45.738 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft 36391sec preferred_lft 36391sec 2023-10-14T18:19:45.738 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::ec4:7aff:febd:125a/64 scope link noprefixroute 2023-10-14T18:19:45.738 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.739 INFO:teuthology.orchestra.run.smithi029.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2023-10-14T18:19:45.739 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 0c:c4:7a:bd:12:5b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:19:45.739 INFO:teuthology.orchestra.run.smithi029.stdout: altname enp1s0f1 2023-10-14T18:19:45.739 INFO:teuthology.orchestra.run.smithi029.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2023-10-14T18:19:45.739 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff 2023-10-14T18:19:45.740 INFO:teuthology.orchestra.run.smithi029.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2023-10-14T18:19:45.740 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.740 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::fc48:6dff:fe78:3335/64 scope link 2023-10-14T18:19:45.740 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.740 INFO:teuthology.orchestra.run.smithi029.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 52:2f:27:b9:b2:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::502f:27ff:feb9:b20d/64 scope link 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout: link/ether 4e:8e:e6:ec:11:3b brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T18:19:45.741 INFO:teuthology.orchestra.run.smithi029.stdout: inet6 fe80::4c8e:e6ff:feec:113b/64 scope link 2023-10-14T18:19:45.742 INFO:teuthology.orchestra.run.smithi029.stdout: valid_lft forever preferred_lft forever 2023-10-14T18:19:45.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:45.743 DEBUG:teuthology.orchestra.run.smithi029:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2023-10-14T18:19:45.843 INFO:teuthology.orchestra.run.smithi029.stdout:1 2023-10-14T18:19:45.847 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:45.848 DEBUG:teuthology.orchestra.run.smithi029:> route 2023-10-14T18:19:45.907 INFO:teuthology.orchestra.run.smithi029.stdout:Kernel IP routing table 2023-10-14T18:19:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2023-10-14T18:19:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2023-10-14T18:19:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2023-10-14T18:19:45.908 INFO:teuthology.orchestra.run.smithi029.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2023-10-14T18:19:45.909 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:45.909 DEBUG:teuthology.orchestra.run.smithi029:> (cd / && exec bash -c ' 2023-10-14T18:19:45.910 DEBUG:teuthology.orchestra.run.smithi029:> set -e 2023-10-14T18:19:45.910 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2023-10-14T18:19:45.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2023-10-14T18:19:45.911 DEBUG:teuthology.orchestra.run.smithi029:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2023-10-14T18:19:45.912 DEBUG:teuthology.orchestra.run.smithi029:> ') 2023-10-14T18:19:46.103 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:46.103 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:19:46.162 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T18:19:46.163 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T18:19:46.163 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:19:46.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:46.164 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link set brx.0 up 2023-10-14T18:19:46.266 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:19:46.266 DEBUG:teuthology.orchestra.run.smithi029:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:46.327 INFO:teuthology.orchestra.run.smithi029.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2023-10-14T18:19:46.328 INFO:teuthology.orchestra.run:Running command with timeout 60 2023-10-14T18:19:46.328 DEBUG:teuthology.orchestra.run.smithi029:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:46.390 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2023-10-14T18:19:46.390 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:46.391 DEBUG:teuthology.orchestra.run.smithi029:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:19:46.545 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:19:46.545 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:19:46.546 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:19:46.569 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:19:46.570 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:19:46.570 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:19:46.570 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:19:46.570 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:19:46.597 INFO:teuthology.orchestra.run.smithi029.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2023-10-14T18:19:46.597 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2023-10-14T18:19:46.597 INFO:teuthology.orchestra.run.smithi029.stdout:invalid new device string format 2023-10-14T18:19:46.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:3300,172.21.15.186:3300,172.21.15.186:3301" 2023-10-14T18:19:46.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: switching to using v1 address with old syntax 2023-10-14T18:19:46.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: resolved to: "172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790" 2023-10-14T18:19:46.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.29:6789,172.21.15.186:6789,172.21.15.186:6790:/ 2023-10-14T18:19:46.598 INFO:teuthology.orchestra.run.smithi029.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=f6aee177-7aff-4117-99f6-f2cafd5adc35" will pass to kernel 2023-10-14T18:19:46.600 INFO:teuthology.orchestra.run.smithi029.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2023-10-14T18:19:46.600 INFO:teuthology.orchestra.run.smithi029.stdout: You just mounted a file system that supports labels which does not 2023-10-14T18:19:46.600 INFO:teuthology.orchestra.run.smithi029.stdout: contain labels, onto an SELinux box. It is likely that confined 2023-10-14T18:19:46.600 INFO:teuthology.orchestra.run.smithi029.stdout: applications will generate AVC messages and not be allowed access to 2023-10-14T18:19:46.601 INFO:teuthology.orchestra.run.smithi029.stdout: this file system. For more details see restorecon(8) and mount(8). 2023-10-14T18:19:46.604 INFO:tasks.cephfs.kernel_mount:mount command passed 2023-10-14T18:19:46.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:46.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:46.702 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:46.702 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2023-10-14T18:19:46.782 INFO:teuthology.orchestra.run.smithi029.stdout:client12981 2023-10-14T18:19:46.783 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:46.783 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:46.844 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs volume ls 2023-10-14T18:19:47.079 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.079+0000 7f73a4be9640 1 Processor -- start 2023-10-14T18:19:47.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.079+0000 7f73a4be9640 1 -- start start 2023-10-14T18:19:47.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.080+0000 7f73a4be9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73a015cba0 0x7f73a0071bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.080+0000 7f73a4be9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a00cd840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.080+0000 7f73a4be9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73a006bb60 con 0x7f73a00cdd80 2023-10-14T18:19:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.080+0000 7f73a4be9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73a00ce210 con 0x7f73a015cba0 2023-10-14T18:19:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.080+0000 7f73a4be9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00716b0 con 0x7f73a00ccdc0 2023-10-14T18:19:47.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73a015cba0 0x7f73a0071bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f73a015cba0 0x7f73a0071bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43516/0 (socket says 172.21.15.29:43516) 2023-10-14T18:19:47.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 -- 172.21.15.29:0/2572387545 learned_addr learned my addr 172.21.15.29:0/2572387545 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 -- 172.21.15.29:0/2572387545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 msgr2=0x7f73a00cd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739effd640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a00cd840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a00cd840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.084 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 -- 172.21.15.29:0/2572387545 >> v1:172.21.15.186:6789/0 conn(0x7f73a00cdd80 legacy=0x7f73a00cbf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739f7fe640 1 -- 172.21.15.29:0/2572387545 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73a00d09e0 con 0x7f73a015cba0 2023-10-14T18:19:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.081+0000 7f739effd640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.083+0000 7f739f7fe640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a0071bd0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f7390009e50 tx=0x7f739002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8de77d8afa2ac848 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.083+0000 7f739e7fc640 1 -- 172.21.15.29:0/2572387545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f739002ca70 con 0x7f73a015cba0 2023-10-14T18:19:47.085 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.083+0000 7f739e7fc640 1 -- 172.21.15.29:0/2572387545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f739002cbd0 con 0x7f73a015cba0 2023-10-14T18:19:47.086 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.083+0000 7f739e7fc640 1 -- 172.21.15.29:0/2572387545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73900343f0 con 0x7f73a015cba0 2023-10-14T18:19:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 -- 172.21.15.29:0/2572387545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 msgr2=0x7f73a0071bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a0071bd0 secure :-1 s=READY pgs=3160 cs=0 l=1 rev1=1 crypto rx=0x7f7390009e50 tx=0x7f739002bed0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 -- 172.21.15.29:0/2572387545 shutdown_connections 2023-10-14T18:19:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a00cd840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.087 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2572387545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a0071bd0 unknown :-1 s=CLOSED pgs=3160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 -- 172.21.15.29:0/2572387545 >> 172.21.15.29:0/2572387545 conn(0x7f73a0065ed0 msgr2=0x7f73a00662e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:47.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 -- 172.21.15.29:0/2572387545 shutdown_connections 2023-10-14T18:19:47.088 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.084+0000 7f73a4be9640 1 -- 172.21.15.29:0/2572387545 wait complete. 2023-10-14T18:19:47.090 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.086+0000 7f73a4be9640 1 Processor -- start 2023-10-14T18:19:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.087+0000 7f73a4be9640 1 -- start start 2023-10-14T18:19:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a015ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73a00cdd80 0x7f73a014cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.091 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a014d2d0 con 0x7f73a015cba0 2023-10-14T18:19:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00ada20 con 0x7f73a00cdd80 2023-10-14T18:19:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f73a4be9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73a00add00 con 0x7f73a00ccdc0 2023-10-14T18:19:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f739ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f739ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43520/0 (socket says 172.21.15.29:43520) 2023-10-14T18:19:47.092 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.088+0000 7f739ffff640 1 -- 172.21.15.29:0/2564444044 learned_addr learned my addr 172.21.15.29:0/2564444044 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739f7fe640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a015ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739effd640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73a00cdd80 0x7f73a014cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 -- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 msgr2=0x7f73a015ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a015ad00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 -- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73a00cdd80 msgr2=0x7f73a014cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73a00cdd80 0x7f73a014cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 -- 172.21.15.29:0/2564444044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390009b30 con 0x7f73a015cba0 2023-10-14T18:19:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f739ffff640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f739800c410 tx=0x7f739800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398011790 con 0x7f73a015cba0 2023-10-14T18:19:47.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73980118f0 con 0x7f73a015cba0 2023-10-14T18:19:47.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.089+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7398011ce0 con 0x7f73a015cba0 2023-10-14T18:19:47.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.090+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73a00adfe0 con 0x7f73a015cba0 2023-10-14T18:19:47.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.090+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73a00c5f00 con 0x7f73a015cba0 2023-10-14T18:19:47.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.091+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73a00cecd0 con 0x7f73a015cba0 2023-10-14T18:19:47.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.092+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f739801c460 con 0x7f73a015cba0 2023-10-14T18:19:47.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.093+0000 7f7397fff640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 0x7f738006d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.093+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f7398089730 con 0x7f73a015cba0 2023-10-14T18:19:47.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.094+0000 7f739f7fe640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 0x7f738006d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.095+0000 7f739f7fe640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 0x7f738006d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7390035000 tx=0x7f7390030000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.098+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f7398054430 con 0x7f73a015cba0 2023-10-14T18:19:47.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.324+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 --> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] -- mgr_command(tid 0: {"prefix": "fs volume ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f73a006b230 con 0x7f738006b0b0 2023-10-14T18:19:47.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.327+0000 7f7397fff640 1 -- 172.21.15.29:0/2564444044 <== mgr.12471 v2:172.21.15.29:6832/1327833207 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7f7390034470 con 0x7f738006b0b0 2023-10-14T18:19:47.328 INFO:teuthology.orchestra.run.smithi029.stdout:[ 2023-10-14T18:19:47.328 INFO:teuthology.orchestra.run.smithi029.stdout: { 2023-10-14T18:19:47.328 INFO:teuthology.orchestra.run.smithi029.stdout: "name": "cephfs" 2023-10-14T18:19:47.329 INFO:teuthology.orchestra.run.smithi029.stdout: } 2023-10-14T18:19:47.329 INFO:teuthology.orchestra.run.smithi029.stdout:] 2023-10-14T18:19:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.329+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 msgr2=0x7f738006d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.329+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 0x7f738006d570 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7390035000 tx=0x7f7390030000 comp rx=0 tx=0).stop 2023-10-14T18:19:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.329+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 msgr2=0x7f73a00af5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.329+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 secure :-1 s=READY pgs=3161 cs=0 l=1 rev1=1 crypto rx=0x7f739800c410 tx=0x7f739800c8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 shutdown_connections 2023-10-14T18:19:47.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f738006b0b0 0x7f738006d570 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f73a015cba0 0x7f73a00af5e0 unknown :-1 s=CLOSED pgs=3161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f73a00cdd80 0x7f73a014cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 --2- 172.21.15.29:0/2564444044 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f73a00ccdc0 0x7f73a015ad00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 >> 172.21.15.29:0/2564444044 conn(0x7f73a0065ed0 msgr2=0x7f73a00cf9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 shutdown_connections 2023-10-14T18:19:47.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.330+0000 7f73a4be9640 1 -- 172.21.15.29:0/2564444044 wait complete. 2023-10-14T18:19:47.349 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module enable snap_schedule 2023-10-14T18:19:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.580+0000 7f4eb52b4640 1 Processor -- start 2023-10-14T18:19:47.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.580+0000 7f4eb52b4640 1 -- start start 2023-10-14T18:19:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4eb52b4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f4eb00c9060 0x7f4eb006f440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4eb52b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4eb006f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4eb52b4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4eb00ca560 con 0x7f4eb006c9e0 2023-10-14T18:19:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4eb52b4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4eb00c9950 con 0x7f4eb00c9060 2023-10-14T18:19:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4eb52b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00c9c30 con 0x7f4eb00c9440 2023-10-14T18:19:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4ea77fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f4eb006c9e0 0x7f4eb0164d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:57894/0 (socket says 172.21.15.29:57894) 2023-10-14T18:19:47.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.582+0000 7f4ea77fe640 1 -- 172.21.15.29:0/2636542703 learned_addr learned my addr 172.21.15.29:0/2636542703 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaffff640 1 --2- 172.21.15.29:0/2636542703 >> v2:172.21.15.186:3300/0 conn(0x7f4eb00c9060 0x7f4eb006f440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eb4ab3640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4eb006f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853992589 0 0) 0x7f4eb00ca560 con 0x7f4eb006c9e0 2023-10-14T18:19:47.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e9c003520 con 0x7f4eb006c9e0 2023-10-14T18:19:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3661913252 0 0) 0x7f4e9c003520 con 0x7f4eb006c9e0 2023-10-14T18:19:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 msgr2=0x7f4eb006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4eb006f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 >> v2:172.21.15.186:3300/0 conn(0x7f4eb00c9060 msgr2=0x7f4eb006f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 --2- 172.21.15.29:0/2636542703 >> v2:172.21.15.186:3300/0 conn(0x7f4eb00c9060 0x7f4eb006f440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb00c9f10 con 0x7f4eb006c9e0 2023-10-14T18:19:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f4e98001670 con 0x7f4eb006c9e0 2023-10-14T18:19:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 >> v1:172.21.15.29:6789/0 conn(0x7f4eb006c9e0 legacy=0x7f4eb0164d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4e9c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4e9c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e9c004d00 0x7f4e9c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00c9950 con 0x7f4e9c003f40 2023-10-14T18:19:47.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb00c9c30 con 0x7f4e9c004d00 2023-10-14T18:19:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.583+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e9c00b980 con 0x7f4eb00c9440 2023-10-14T18:19:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4e9c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4eaffff640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4e9c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4eb4ab3640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e9c004d00 0x7f4e9c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 -- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 msgr2=0x7f4e9c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4e9c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 -- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e9c004d00 msgr2=0x7f4e9c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:47.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4e9c004d00 0x7f4e9c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4ea77fe640 1 -- 172.21.15.29:0/2636542703 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e9c003520 con 0x7f4e9c003f40 2023-10-14T18:19:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.584+0000 7f4eaffff640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c9440 0x7f4e9c003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.585+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4e9c0047c0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800c110 tx=0x7f4e9c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb82d86628cc04e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e98005ad0 con 0x7f4e9c003f40 2023-10-14T18:19:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98005c50 con 0x7f4e9c003f40 2023-10-14T18:19:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eaf7fe640 1 -- 172.21.15.29:0/2636542703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e9802c7e0 con 0x7f4e9c003f40 2023-10-14T18:19:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eb52b4640 1 -- 172.21.15.29:0/2636542703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 msgr2=0x7f4e9c0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/2636542703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4e9c0047c0 secure :-1 s=READY pgs=3163 cs=0 l=1 rev1=1 crypto rx=0x7f4e9800c110 tx=0x7f4e9c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eb4ab3640 1 -- 172.21.15.29:0/2636542703 reap_dead start 2023-10-14T18:19:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eb52b4640 1 -- 172.21.15.29:0/2636542703 shutdown_connections 2023-10-14T18:19:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.586+0000 7f4eb52b4640 1 -- 172.21.15.29:0/2636542703 >> 172.21.15.29:0/2636542703 conn(0x7f4eb00ce6a0 msgr2=0x7f4eb00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.587+0000 7f4eb52b4640 1 -- 172.21.15.29:0/2636542703 shutdown_connections 2023-10-14T18:19:47.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.587+0000 7f4eb52b4640 1 -- 172.21.15.29:0/2636542703 wait complete. 2023-10-14T18:19:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.588+0000 7f4eb52b4640 1 Processor -- start 2023-10-14T18:19:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 -- start start 2023-10-14T18:19:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb00c2aa0 0x7f4eb01633e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c47e0 0x7f4eb00c7790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0162fc0 con 0x7f4e9c003f40 2023-10-14T18:19:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.589+0000 7f4eb52b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0163fd0 con 0x7f4eb00c2aa0 2023-10-14T18:19:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eb52b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb0163920 con 0x7f4eb00c47e0 2023-10-14T18:19:47.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43540/0 (socket says 172.21.15.29:43540) 2023-10-14T18:19:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 -- 172.21.15.29:0/579005088 learned_addr learned my addr 172.21.15.29:0/579005088 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 -- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c47e0 msgr2=0x7f4eb00c7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c47e0 0x7f4eb00c7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 -- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb00c2aa0 msgr2=0x7f4eb01633e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb00c2aa0 0x7f4eb01633e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 -- 172.21.15.29:0/579005088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4eb0164220 con 0x7f4e9c003f40 2023-10-14T18:19:47.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.590+0000 7f4eaffff640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00101c0 tx=0x7f4ea0010680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.591+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea000e050 con 0x7f4e9c003f40 2023-10-14T18:19:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.591+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ea000f040 con 0x7f4e9c003f40 2023-10-14T18:19:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.591+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ea0003bb0 con 0x7f4e9c003f40 2023-10-14T18:19:47.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.591+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb0164500 con 0x7f4e9c003f40 2023-10-14T18:19:47.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.591+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb00b4100 con 0x7f4e9c003f40 2023-10-14T18:19:47.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.592+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb004e9b0 con 0x7f4e9c003f40 2023-10-14T18:19:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.593+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 155) v1 ==== 89433+0+0 (secure 0 0 0) 0x7f4ea0003d10 con 0x7f4e9c003f40 2023-10-14T18:19:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.594+0000 7f4ead7fa640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 0x7f4e9006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.594+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f4ea008a9f0 con 0x7f4e9c003f40 2023-10-14T18:19:47.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.594+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 0x7f4e9006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.595+0000 7f4ea77fe640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 0x7f4e9006d5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e98034000 tx=0x7f4e9802e000 comp rx=0 tx=0).ready entity=mgr.12471 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:47.608 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.599+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f4ea0055620 con 0x7f4e9c003f40 2023-10-14T18:19:47.716 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:19:47.726 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:19:47.848 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:47.847+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f4eb00b4620 con 0x7f4e9c003f40 2023-10-14T18:19:48.348 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.347+0000 7f4ead7fa640 1 -- 172.21.15.29:0/579005088 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v156) v1 ==== 92+0+0 (secure 0 0 0) 0x7f4ea005f030 con 0x7f4e9c003f40 2023-10-14T18:19:48.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 msgr2=0x7f4e9006d5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 0x7f4e9006d5c0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e98034000 tx=0x7f4e9802e000 comp rx=0 tx=0).stop 2023-10-14T18:19:48.352 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 msgr2=0x7f4eb00c23c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 secure :-1 s=READY pgs=3164 cs=0 l=1 rev1=1 crypto rx=0x7f4ea00101c0 tx=0x7f4ea0010680 comp rx=0 tx=0).stop 2023-10-14T18:19:48.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 shutdown_connections 2023-10-14T18:19:48.353 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f4e9006b100 0x7f4e9006d5c0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4eb00c47e0 0x7f4eb00c7790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4eb00c2aa0 0x7f4eb01633e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 --2- 172.21.15.29:0/579005088 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4e9c003f40 0x7f4eb00c23c0 unknown :-1 s=CLOSED pgs=3164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.354 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.351+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 >> 172.21.15.29:0/579005088 conn(0x7f4eb00ce6a0 msgr2=0x7f4eb006fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:48.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.352+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 shutdown_connections 2023-10-14T18:19:48.355 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.352+0000 7f4eb52b4640 1 -- 172.21.15.29:0/579005088 wait complete. 2023-10-14T18:19:48.383 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/allow_m_granularity True 2023-10-14T18:19:48.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.587+0000 7f6b5a214640 1 Processor -- start 2023-10-14T18:19:48.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.587+0000 7f6b5a214640 1 -- start start 2023-10-14T18:19:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b540d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6b540b8480 0x7f6b540cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b540d0be0 0x7f6b540cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5415d150 con 0x7f6b540d0be0 2023-10-14T18:19:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b540d1180 con 0x7f6b540b8480 2023-10-14T18:19:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b5a214640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b540d1460 con 0x7f6b5415cd70 2023-10-14T18:19:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b59a13640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b540d0be0 0x7f6b540cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b59212640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b540d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b59a13640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b540d0be0 0x7f6b540cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43566/0 (socket says 172.21.15.29:43566) 2023-10-14T18:19:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b59a13640 1 -- 172.21.15.29:0/760896554 learned_addr learned my addr 172.21.15.29:0/760896554 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:48.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.589+0000 7f6b58a11640 1 --2- 172.21.15.29:0/760896554 >> v2:172.21.15.186:3300/0 conn(0x7f6b540b8480 0x7f6b540cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.590+0000 7f6b59a13640 1 -- 172.21.15.29:0/760896554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 msgr2=0x7f6b540d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:48.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.590+0000 7f6b59a13640 1 --2- 172.21.15.29:0/760896554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b540d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.590+0000 7f6b59a13640 1 -- 172.21.15.29:0/760896554 >> v2:172.21.15.186:3300/0 conn(0x7f6b540b8480 msgr2=0x7f6b540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:48.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.590+0000 7f6b59a13640 1 --2- 172.21.15.29:0/760896554 >> v2:172.21.15.186:3300/0 conn(0x7f6b540b8480 0x7f6b540cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.590+0000 7f6b59a13640 1 -- 172.21.15.29:0/760896554 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b540d1740 con 0x7f6b540d0be0 2023-10-14T18:19:48.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.592+0000 7f6b59a13640 1 --2- 172.21.15.29:0/760896554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b540cdd00 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f6b54066ab0 tx=0x7f6b4c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10b4110af8c5e0f7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:48.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.592+0000 7f6b4bfff640 1 -- 172.21.15.29:0/760896554 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c031870 con 0x7f6b540d0be0 2023-10-14T18:19:48.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.592+0000 7f6b4bfff640 1 -- 172.21.15.29:0/760896554 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4c0319d0 con 0x7f6b540d0be0 2023-10-14T18:19:48.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.592+0000 7f6b4bfff640 1 -- 172.21.15.29:0/760896554 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4c031d60 con 0x7f6b540d0be0 2023-10-14T18:19:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 -- 172.21.15.29:0/760896554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 msgr2=0x7f6b540cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 --2- 172.21.15.29:0/760896554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b540cdd00 secure :-1 s=READY pgs=3168 cs=0 l=1 rev1=1 crypto rx=0x7f6b54066ab0 tx=0x7f6b4c02b910 comp rx=0 tx=0).stop 2023-10-14T18:19:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 -- 172.21.15.29:0/760896554 shutdown_connections 2023-10-14T18:19:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 --2- 172.21.15.29:0/760896554 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b540cdd00 unknown :-1 s=CLOSED pgs=3168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 --2- 172.21.15.29:0/760896554 >> v2:172.21.15.186:3300/0 conn(0x7f6b540b8480 0x7f6b540cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 --2- 172.21.15.29:0/760896554 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b540d2220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 -- 172.21.15.29:0/760896554 >> 172.21.15.29:0/760896554 conn(0x7f6b540cf570 msgr2=0x7f6b540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:48.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 -- 172.21.15.29:0/760896554 shutdown_connections 2023-10-14T18:19:48.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.593+0000 7f6b5a214640 1 -- 172.21.15.29:0/760896554 wait complete. 2023-10-14T18:19:48.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.595+0000 7f6b5a214640 1 Processor -- start 2023-10-14T18:19:48.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.595+0000 7f6b5a214640 1 -- start start 2023-10-14T18:19:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b540b8480 0x7f6b54154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b5415e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b54155650 con 0x7f6b540d0be0 2023-10-14T18:19:48.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b541558b0 con 0x7f6b540b8480 2023-10-14T18:19:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b5a214640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b540c64f0 con 0x7f6b5415cd70 2023-10-14T18:19:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b58a11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b58a11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43582/0 (socket says 172.21.15.29:43582) 2023-10-14T18:19:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b58a11640 1 -- 172.21.15.29:0/2858407811 learned_addr learned my addr 172.21.15.29:0/2858407811 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:48.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b59a13640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b5415e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.596+0000 7f6b59212640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b540b8480 0x7f6b54154120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 msgr2=0x7f6b5415e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b5415e2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b540b8480 msgr2=0x7f6b54154120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b540b8480 0x7f6b54154120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 -- 172.21.15.29:0/2858407811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b4c0094d0 con 0x7f6b540d0be0 2023-10-14T18:19:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b58a11640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400e9b0 tx=0x7f6b4400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b44009670 con 0x7f6b540d0be0 2023-10-14T18:19:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b4400c9a0 con 0x7f6b540d0be0 2023-10-14T18:19:48.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b540c67d0 con 0x7f6b540d0be0 2023-10-14T18:19:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b4400cd50 con 0x7f6b540d0be0 2023-10-14T18:19:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.597+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b540c6d90 con 0x7f6b540d0be0 2023-10-14T18:19:48.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.599+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5404e9b0 con 0x7f6b540d0be0 2023-10-14T18:19:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.600+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 156) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f6b44013470 con 0x7f6b540d0be0 2023-10-14T18:19:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.601+0000 7f6b49ffb640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 0x7f6b3806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:48.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.601+0000 7f6b59212640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 msgr2=0x7f6b3806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.602+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f6b44089dc0 con 0x7f6b540d0be0 2023-10-14T18:19:48.606 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.602+0000 7f6b59212640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 0x7f6b3806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:48.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.606+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b44054aa0 con 0x7f6b540d0be0 2023-10-14T18:19:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.803+0000 7f6b59212640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 msgr2=0x7f6b3806d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:48.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.803+0000 7f6b59212640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 0x7f6b3806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:48.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.806+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f6b540d1d00 con 0x7f6b540d0be0 2023-10-14T18:19:48.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.813+0000 7f6b49ffb640 1 -- 172.21.15.29:0/2858407811 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v89)=0 v89) v1 ==== 138+0+0 (secure 0 0 0) 0x7f6b440170d0 con 0x7f6b540d0be0 2023-10-14T18:19:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 msgr2=0x7f6b3806d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:48.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 0x7f6b3806d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 msgr2=0x7f6b54155110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 secure :-1 s=READY pgs=3169 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400e9b0 tx=0x7f6b4400ee70 comp rx=0 tx=0).stop 2023-10-14T18:19:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 shutdown_connections 2023-10-14T18:19:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f6b3806b100 0x7f6b3806d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b5415cd70 0x7f6b5415e2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b540d0be0 0x7f6b54155110 unknown :-1 s=CLOSED pgs=3169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 --2- 172.21.15.29:0/2858407811 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b540b8480 0x7f6b54154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 >> 172.21.15.29:0/2858407811 conn(0x7f6b540cf570 msgr2=0x7f6b540b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 shutdown_connections 2023-10-14T18:19:48.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:48.816+0000 7f6b5a214640 1 -- 172.21.15.29:0/2858407811 wait complete. 2023-10-14T18:19:48.836 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr/snap_schedule/dump_on_update True 2023-10-14T18:19:48.925 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:48.924+0000 7f387c22e040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:48.926 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:48.925+0000 7fa81d012040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:48.984 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:48.983+0000 7f387c22e040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:48.985 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:48.984+0000 7fa81d012040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.050+0000 7f9c971a9640 1 Processor -- start 2023-10-14T18:19:49.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.050+0000 7f9c971a9640 1 -- start start 2023-10-14T18:19:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.051+0000 7f9c971a9640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c901618b0 0x7f9c90065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.051+0000 7f9c971a9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 0x7f9c900ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.051+0000 7f9c971a9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c9006cee0 con 0x7f9c9015d330 2023-10-14T18:19:49.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.051+0000 7f9c971a9640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c9006b1e0 con 0x7f9c901618b0 2023-10-14T18:19:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.051+0000 7f9c971a9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c900663d0 con 0x7f9c90065ff0 2023-10-14T18:19:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c901618b0 0x7f9c90065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f9c901618b0 0x7f9c90065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43586/0 (socket says 172.21.15.29:43586) 2023-10-14T18:19:49.054 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 -- 172.21.15.29:0/182185781 learned_addr learned my addr 172.21.15.29:0/182185781 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c95746640 1 --2- 172.21.15.29:0/182185781 >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 0x7f9c900ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 -- 172.21.15.29:0/182185781 >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 msgr2=0x7f9c900ce9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 --2- 172.21.15.29:0/182185781 >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 0x7f9c900ce9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 -- 172.21.15.29:0/182185781 >> v1:172.21.15.186:6789/0 conn(0x7f9c9015d330 legacy=0x7f9c9006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:49.055 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c8ffff640 1 -- 172.21.15.29:0/182185781 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c900d0890 con 0x7f9c901618b0 2023-10-14T18:19:49.056 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.052+0000 7f9c95746640 1 --2- 172.21.15.29:0/182185781 >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 0x7f9c900ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:49.057 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.053+0000 7f9c8ffff640 1 --2- 172.21.15.29:0/182185781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c901618b0 0x7f9c90065ab0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f9c9006f340 tx=0x7f9c7c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=549151c74edc1783 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c8f7fe640 1 -- 172.21.15.29:0/182185781 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0317e0 con 0x7f9c901618b0 2023-10-14T18:19:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c8f7fe640 1 -- 172.21.15.29:0/182185781 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c031940 con 0x7f9c901618b0 2023-10-14T18:19:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c8f7fe640 1 -- 172.21.15.29:0/182185781 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c031cd0 con 0x7f9c901618b0 2023-10-14T18:19:49.058 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 -- 172.21.15.29:0/182185781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c901618b0 msgr2=0x7f9c90065ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 --2- 172.21.15.29:0/182185781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c901618b0 0x7f9c90065ab0 secure :-1 s=READY pgs=3170 cs=0 l=1 rev1=1 crypto rx=0x7f9c9006f340 tx=0x7f9c7c02b880 comp rx=0 tx=0).stop 2023-10-14T18:19:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 -- 172.21.15.29:0/182185781 shutdown_connections 2023-10-14T18:19:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 --2- 172.21.15.29:0/182185781 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c901618b0 0x7f9c90065ab0 unknown :-1 s=CLOSED pgs=3170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 --2- 172.21.15.29:0/182185781 >> v2:172.21.15.186:3300/0 conn(0x7f9c90065ff0 0x7f9c900ce9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.059 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.054+0000 7f9c971a9640 1 -- 172.21.15.29:0/182185781 >> 172.21.15.29:0/182185781 conn(0x7f9c900d0ab0 msgr2=0x7f9c900d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.055+0000 7f9c971a9640 1 -- 172.21.15.29:0/182185781 shutdown_connections 2023-10-14T18:19:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.055+0000 7f9c971a9640 1 -- 172.21.15.29:0/182185781 wait complete. 2023-10-14T18:19:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.056+0000 7f9c971a9640 1 Processor -- start 2023-10-14T18:19:49.060 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.057+0000 7f9c971a9640 1 -- start start 2023-10-14T18:19:49.062 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.057+0000 7f9c971a9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.057+0000 7f9c971a9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c9015d330 0x7f9c9015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c971a9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c901618b0 0x7f9c9015be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c971a9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900b55d0 con 0x7f9c90065ff0 2023-10-14T18:19:49.063 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c971a9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900af6c0 con 0x7f9c901618b0 2023-10-14T18:19:49.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c971a9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c900b57e0 con 0x7f9c9015d330 2023-10-14T18:19:49.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.064 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43596/0 (socket says 172.21.15.29:43596) 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 -- 172.21.15.29:0/4238772283 learned_addr learned my addr 172.21.15.29:0/4238772283 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c8ffff640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c9015d330 0x7f9c9015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c9015d330 msgr2=0x7f9c9015b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c95746640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c901618b0 0x7f9c9015be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c9015d330 0x7f9c9015b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.065 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c901618b0 msgr2=0x7f9c9015be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c901618b0 0x7f9c9015be60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.058+0000 7f9c94f45640 1 -- 172.21.15.29:0/4238772283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c0094d0 con 0x7f9c90065ff0 2023-10-14T18:19:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c94f45640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400b3e0 tx=0x7f9c8400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:49.066 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011830 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c840119b0 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c84011d60 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c900b5ac0 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.059+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c900af350 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.061+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c9004e160 con 0x7f9c90065ff0 2023-10-14T18:19:49.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.061+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 156) v1 ==== 89450+0+0 (secure 0 0 0) 0x7f9c8401a400 con 0x7f9c90065ff0 2023-10-14T18:19:49.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.063+0000 7f9c8d7fa640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 0x7f9c7406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.063+0000 7f9c8ffff640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 msgr2=0x7f9c7406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.063+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7f9c84089ca0 con 0x7f9c90065ff0 2023-10-14T18:19:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.063+0000 7f9c8ffff640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 0x7f9c7406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:49.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.067+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9c84054960 con 0x7f9c90065ff0 2023-10-14T18:19:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.264+0000 7f9c8ffff640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 msgr2=0x7f9c7406d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:49.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.264+0000 7f9c8ffff640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 0x7f9c7406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:49.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.277+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f9c90161080 con 0x7f9c90065ff0 2023-10-14T18:19:49.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.283+0000 7f9c8d7fa640 1 -- 172.21.15.29:0/4238772283 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v90)=0 v90) v1 ==== 133+0+0 (secure 0 0 0) 0x7f9c84015020 con 0x7f9c90065ff0 2023-10-14T18:19:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 msgr2=0x7f9c7406d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:49.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 0x7f9c7406d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 msgr2=0x7f9c9014ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 secure :-1 s=READY pgs=3171 cs=0 l=1 rev1=1 crypto rx=0x7f9c8400b3e0 tx=0x7f9c8400b8a0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 shutdown_connections 2023-10-14T18:19:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7f9c7406b100 0x7f9c7406d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9c901618b0 0x7f9c9015be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9c9015d330 0x7f9c9015b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 --2- 172.21.15.29:0/4238772283 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f9c90065ff0 0x7f9c9014ffb0 unknown :-1 s=CLOSED pgs=3171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 >> 172.21.15.29:0/4238772283 conn(0x7f9c900d0ab0 msgr2=0x7f9c9006c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:49.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 shutdown_connections 2023-10-14T18:19:49.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.286+0000 7f9c971a9640 1 -- 172.21.15.29:0/4238772283 wait complete. 2023-10-14T18:19:49.307 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:49.307 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir -p snap_test_dir1) 2023-10-14T18:19:49.372 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule add snap_test_dir1 1m 2023-10-14T18:19:49.444 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.443+0000 7f387c22e040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:49.491 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.490+0000 7fa81d012040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:49.495 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.494+0000 7f387c22e040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:49.546 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.545+0000 7fa81d012040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:49.547 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.546+0000 7f387c22e040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:49.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.577+0000 7fa4559ce640 1 Processor -- start 2023-10-14T18:19:49.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.577+0000 7fa4559ce640 1 -- start start 2023-10-14T18:19:49.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.578+0000 7fa4559ce640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.578+0000 7fa4559ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.578+0000 7fa4559ce640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa450066420 con 0x7fa45015d330 2023-10-14T18:19:49.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.578+0000 7fa4559ce640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa45006b1e0 con 0x7fa4501618b0 2023-10-14T18:19:49.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.578+0000 7fa4559ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa45006cee0 con 0x7fa450065ff0 2023-10-14T18:19:49.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44e7fc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:39192/0 (socket says 172.21.15.29:39192) 2023-10-14T18:19:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44e7fc640 1 -- 172.21.15.29:0/1076090785 learned_addr learned my addr 172.21.15.29:0/1076090785 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:57722/0 (socket says 172.21.15.29:57722) 2023-10-14T18:19:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.579+0000 7fa44effd640 1 --1- 172.21.15.29:0/1076090785 >> v1:172.21.15.186:6789/0 conn(0x7fa45015d330 0x7fa45006c7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:34182/0 (socket says 172.21.15.29:34182) 2023-10-14T18:19:49.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.580+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1076090785 >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 msgr2=0x7fa450065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.580+0000 7fa44f7fe640 1 --2- 172.21.15.29:0/1076090785 >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.580+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1076090785 >> v1:172.21.15.186:6789/0 conn(0x7fa45015d330 legacy=0x7fa45006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.580+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1076090785 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500d0890 con 0x7fa450065ff0 2023-10-14T18:19:49.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.580+0000 7fa44e7fc640 1 --2- 172.21.15.29:0/1076090785 >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:19:49.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.581+0000 7fa44f7fe640 1 --2- 172.21.15.29:0/1076090785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4480097a0 tx=0x7fa44802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=576bb8566478ecc6 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:49.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.581+0000 7fa44dffb640 1 -- 172.21.15.29:0/1076090785 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448031780 con 0x7fa450065ff0 2023-10-14T18:19:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.581+0000 7fa44dffb640 1 -- 172.21.15.29:0/1076090785 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4480318e0 con 0x7fa450065ff0 2023-10-14T18:19:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.581+0000 7fa44dffb640 1 -- 172.21.15.29:0/1076090785 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448031c70 con 0x7fa450065ff0 2023-10-14T18:19:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 -- 172.21.15.29:0/1076090785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 msgr2=0x7fa4500ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 --2- 172.21.15.29:0/1076090785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa4480097a0 tx=0x7fa44802b820 comp rx=0 tx=0).stop 2023-10-14T18:19:49.591 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 -- 172.21.15.29:0/1076090785 shutdown_connections 2023-10-14T18:19:49.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 --2- 172.21.15.29:0/1076090785 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500ce9b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 --2- 172.21.15.29:0/1076090785 >> v2:172.21.15.186:3300/0 conn(0x7fa4501618b0 0x7fa450065ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 -- 172.21.15.29:0/1076090785 >> 172.21.15.29:0/1076090785 conn(0x7fa4500d0ab0 msgr2=0x7fa4500d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:49.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 -- 172.21.15.29:0/1076090785 shutdown_connections 2023-10-14T18:19:49.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.582+0000 7fa4559ce640 1 -- 172.21.15.29:0/1076090785 wait complete. 2023-10-14T18:19:49.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.584+0000 7fa4559ce640 1 Processor -- start 2023-10-14T18:19:49.594 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.584+0000 7fa4559ce640 1 -- start start 2023-10-14T18:19:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45015d330 0x7fa4500ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 0x7fa45014d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500adbb0 con 0x7fa4501618b0 2023-10-14T18:19:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500c7440 con 0x7fa45015d330 2023-10-14T18:19:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa4559ce640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500b3000 con 0x7fa450065ff0 2023-10-14T18:19:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 0x7fa45014d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa44f7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 0x7fa45014d9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43604/0 (socket says 172.21.15.29:43604) 2023-10-14T18:19:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.585+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1717601640 learned_addr learned my addr 172.21.15.29:0/1717601640 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44e7fc640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45015d330 0x7fa4500ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:49.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 msgr2=0x7fa4500c6f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa450065ff0 0x7fa4500c6f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45015d330 msgr2=0x7fa4500ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45015d330 0x7fa4500ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:49.598 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4380094d0 con 0x7fa4501618b0 2023-10-14T18:19:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44e7fc640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa45015d330 0x7fa4500ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa44f7fe640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 0x7fa45014d9e0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fa448034040 tx=0x7fa4480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448032820 con 0x7fa4501618b0 2023-10-14T18:19:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa448032980 con 0x7fa4501618b0 2023-10-14T18:19:49.599 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.586+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa448009400 con 0x7fa4501618b0 2023-10-14T18:19:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.587+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa448032d30 con 0x7fa4501618b0 2023-10-14T18:19:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.587+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4500ad840 con 0x7fa4501618b0 2023-10-14T18:19:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.588+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4500add20 con 0x7fa4501618b0 2023-10-14T18:19:49.600 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.589+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 156) v1 ==== 89450+0+0 (secure 0 0 0) 0x7fa44804e450 con 0x7fa4501618b0 2023-10-14T18:19:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.590+0000 7fa447fff640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:49.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.590+0000 7fa44effd640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:49.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.590+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(271..271 src has 1..271) v4 ==== 5825+0+0 (secure 0 0 0) 0x7fa4480abb40 con 0x7fa4501618b0 2023-10-14T18:19:49.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.590+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:49.602 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.594+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa4480767c0 con 0x7fa4501618b0 2023-10-14T18:19:49.602 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.600+0000 7fa81d012040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:49.609 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.609+0000 7f387c22e040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:49.659 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.658+0000 7f387c22e040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:49.665 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.664+0000 7fa81d012040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:49.717 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.717+0000 7fa81d012040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:49.772 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.771+0000 7f387c22e040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:49.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.791+0000 7fa44effd640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:49.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.791+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:49.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:49.814+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa450160dd0 con 0x7fa43006b100 2023-10-14T18:19:49.834 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.832+0000 7f387c22e040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:49.835 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.833+0000 7fa81d012040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:49.884 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.883+0000 7f387c22e040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:49.900 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.899+0000 7fa81d012040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:49.941 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.939+0000 7f387c22e040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:49.952 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:49.951+0000 7fa81d012040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:49.991 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:49.989+0000 7f387c22e040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:50.012 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.011+0000 7fa81d012040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:50.064 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.063+0000 7fa81d012040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:50.192+0000 7fa44effd640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:50.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:50.192+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-10-14T18:19:50.370 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.368+0000 7f387c22e040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:50.464 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.463+0000 7fa81d012040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:50.530 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.528+0000 7f387c22e040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:50.632 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.632+0000 7fa81d012040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:50.637 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.636+0000 7f387c22e040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:50.687 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.686+0000 7f387c22e040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:50.746 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.744+0000 7fa81d012040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:50.795 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.794+0000 7f387c22e040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:50.798 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.797+0000 7fa81d012040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:50.902 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:50.901+0000 7f387c22e040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:50.911 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:50.910+0000 7fa81d012040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:50.993+0000 7fa44effd640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:50.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:50.993+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-10-14T18:19:51.024 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:51.023+0000 7fa81d012040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:51.058 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:51.056+0000 7f387c22e040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:51.106 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:51.105+0000 7f387c22e040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:51.188 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:51.186+0000 7fa81d012040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:51.224 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:51.222+0000 7f387c22e040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:51.239 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:51.238+0000 7fa81d012040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:51.364 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:51.362+0000 7fa81d012040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:52.048 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:52.047+0000 7f387c22e040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:52.241 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:52.240+0000 7fa81d012040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:19:52.346 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:52.345+0000 7f387c22e040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:52.402 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:52.401+0000 7f387c22e040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:52.516 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:52.515+0000 7f387c22e040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:52.571 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:52.570+0000 7fa81d012040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:19:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:52.595+0000 7fa44effd640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/1327833207 2023-10-14T18:19:52.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:52.595+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2023-10-14T18:19:52.630 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:52.630+0000 7fa81d012040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:19:52.749 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:52.748+0000 7fa81d012040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:19:52.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:52.771+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 157) v1 ==== 88957+0+0 (secure 0 0 0) 0x7fa448076b90 con 0x7fa4501618b0 2023-10-14T18:19:52.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:52.772+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 msgr2=0x7fa43006d5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:52.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:52.772+0000 7fa447fff640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/1327833207,v1:172.21.15.29:6833/1327833207] conn(0x7fa43006b100 0x7fa43006d5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:53.122 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.121+0000 7fa798eae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.122 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.121+0000 7fa798eae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.122 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.121+0000 7fa798eae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.123 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.121+0000 7fa798eae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.123 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.121+0000 7fa798eae640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.123 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.122+0000 7fa79b6f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.122+0000 7fa79b6f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.122+0000 7fa79b6f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.122+0000 7fa79b6f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.124 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:53.122+0000 7fa79b6f3640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:19:53.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:53.786+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mgrmap(e 158) v1 ==== 89084+0+0 (secure 0 0 0) 0x7fa4480822f0 con 0x7fa4501618b0 2023-10-14T18:19:53.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:53.787+0000 7fa447fff640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fa4300359a0 0x7fa430075140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:53.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:53.787+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 --> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "path": "snap_test_dir1", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7fa430035d80 con 0x7fa4300359a0 2023-10-14T18:19:53.795 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:53.794+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fa4300359a0 0x7fa430075140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:53.804 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:53.803+0000 7fa44effd640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fa4300359a0 0x7fa430075140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa4400012e0 tx=0x7fa440001740 comp rx=0 tx=0).ready entity=mgr.12999 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.008+0000 7fa447fff640 1 -- 172.21.15.29:0/1717601640 <== mgr.12999 v2:172.21.15.29:6832/4074993897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa430035d80 con 0x7fa4300359a0 2023-10-14T18:19:54.009 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule set for path snap_test_dir1 2023-10-14T18:19:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fa4300359a0 msgr2=0x7fa430075140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fa4300359a0 0x7fa430075140 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa4400012e0 tx=0x7fa440001740 comp rx=0 tx=0).stop 2023-10-14T18:19:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 msgr2=0x7fa45014d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 --2- 172.21.15.29:0/1717601640 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa4501618b0 0x7fa45014d9e0 secure :-1 s=READY pgs=3172 cs=0 l=1 rev1=1 crypto rx=0x7fa448034040 tx=0x7fa4480046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa44f7fe640 1 -- 172.21.15.29:0/1717601640 reap_dead start 2023-10-14T18:19:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 shutdown_connections 2023-10-14T18:19:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 >> 172.21.15.29:0/1717601640 conn(0x7fa4500d0ab0 msgr2=0x7fa4500ce9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 shutdown_connections 2023-10-14T18:19:54.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.011+0000 7fa4559ce640 1 -- 172.21.15.29:0/1717601640 wait complete. 2023-10-14T18:19:54.035 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule set for path snap_test_dir1 2023-10-14T18:19:54.035 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule retention add snap_test_dir1 1m 2023-10-14T18:19:54.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.260+0000 7f1af2e1a640 1 Processor -- start 2023-10-14T18:19:54.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.260+0000 7f1af2e1a640 1 -- start start 2023-10-14T18:19:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af2e1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1aec06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af2e1a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f1aec0cda80 0x7f1aec15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af2e1a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec06c960 con 0x7f1aec0ad970 2023-10-14T18:19:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af2e1a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec15b9d0 con 0x7f1aec0cda80 2023-10-14T18:19:54.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af2e1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec15bcb0 con 0x7f1aec06c040 2023-10-14T18:19:54.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af1617640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1aec0ad970 0x7f1aec15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39072/0 (socket says 172.21.15.29:39072) 2023-10-14T18:19:54.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af1617640 1 -- 172.21.15.29:0/712993263 learned_addr learned my addr 172.21.15.29:0/712993263 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:54.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.262+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692682765 0 0) 0x7f1aec06c960 con 0x7f1aec0ad970 2023-10-14T18:19:54.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad8003520 con 0x7f1aec0ad970 2023-10-14T18:19:54.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> v2:172.21.15.186:3300/0 conn(0x7f1aec0cda80 0x7f1aec15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af1e18640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1aec06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4102232096 0 0) 0x7f1ad8003520 con 0x7f1aec0ad970 2023-10-14T18:19:54.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 msgr2=0x7f1aec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:54.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1aec06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 >> v2:172.21.15.186:3300/0 conn(0x7f1aec0cda80 msgr2=0x7f1aec15ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:54.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 --2- 172.21.15.29:0/712993263 >> v2:172.21.15.186:3300/0 conn(0x7f1aec0cda80 0x7f1aec15ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec15bf90 con 0x7f1aec0ad970 2023-10-14T18:19:54.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f1adc001670 con 0x7f1aec0ad970 2023-10-14T18:19:54.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 >> v1:172.21.15.29:6789/0 conn(0x7f1aec0ad970 legacy=0x7f1aec15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad8003970 0x7f1ad8003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1ad80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad8004d00 0x7f1ad8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec15b9d0 con 0x7f1ad8004d00 2023-10-14T18:19:54.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec15bcb0 con 0x7f1ad8003970 2023-10-14T18:19:54.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad800ba30 con 0x7f1aec06c040 2023-10-14T18:19:54.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.263+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> v2:172.21.15.186:3300/0 conn(0x7f1aec0cda80 0x7f1aec15ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:54.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad8004d00 0x7f1ad8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af1617640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1ad80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af1e18640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad8003970 0x7f1ad8003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 -- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 msgr2=0x7f1ad80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:54.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1ad80047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 -- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad8003970 msgr2=0x7f1ad8003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:54.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ad8003970 0x7f1ad8003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af2619640 1 -- 172.21.15.29:0/712993263 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad8003520 con 0x7f1ad8004d00 2023-10-14T18:19:54.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.264+0000 7f1af1617640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1aec06c040 0x7f1ad80047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:54.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.265+0000 7f1af2619640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad8004d00 0x7f1ad8005180 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4007410 tx=0x7f1ad80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=284e95415d18146e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae402cdc0 con 0x7f1ad8004d00 2023-10-14T18:19:54.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae4034440 con 0x7f1ad8004d00 2023-10-14T18:19:54.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af0e16640 1 -- 172.21.15.29:0/712993263 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae40347b0 con 0x7f1ad8004d00 2023-10-14T18:19:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af2e1a640 1 -- 172.21.15.29:0/712993263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad8004d00 msgr2=0x7f1ad8005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/712993263 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ad8004d00 0x7f1ad8005180 secure :-1 s=READY pgs=3187 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4007410 tx=0x7f1ad80057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af2619640 1 -- 172.21.15.29:0/712993263 reap_dead start 2023-10-14T18:19:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af2e1a640 1 -- 172.21.15.29:0/712993263 shutdown_connections 2023-10-14T18:19:54.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.266+0000 7f1af2e1a640 1 -- 172.21.15.29:0/712993263 >> 172.21.15.29:0/712993263 conn(0x7f1aec0ce6a0 msgr2=0x7f1aec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.267+0000 7f1af2e1a640 1 -- 172.21.15.29:0/712993263 shutdown_connections 2023-10-14T18:19:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.267+0000 7f1af2e1a640 1 -- 172.21.15.29:0/712993263 wait complete. 2023-10-14T18:19:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.268+0000 7f1af2e1a640 1 Processor -- start 2023-10-14T18:19:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 -- start start 2023-10-14T18:19:54.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 0x7f1aec0af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aec0b3200 0x7f1aec0b3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0b3bc0 con 0x7f1aec0aed70 2023-10-14T18:19:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.269+0000 7f1af2e1a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0b0fa0 con 0x7f1aec0b3200 2023-10-14T18:19:54.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af2e1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec0afc30 con 0x7f1ad8004d00 2023-10-14T18:19:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52138/0 (socket says 172.21.15.29:52138) 2023-10-14T18:19:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 -- 172.21.15.29:0/3604274558 learned_addr learned my addr 172.21.15.29:0/3604274558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af2619640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aec0b3200 0x7f1aec0b3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1e18640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 0x7f1aec0af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 -- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 msgr2=0x7f1aec0af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 0x7f1aec0af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 -- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aec0b3200 msgr2=0x7f1aec0b3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aec0b3200 0x7f1aec0b3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1617640 1 -- 172.21.15.29:0/3604274558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec0aff00 con 0x7f1aec0aed70 2023-10-14T18:19:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.270+0000 7f1af1e18640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 0x7f1aec0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1af1617640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00e840 tx=0x7f1adc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00c870 con 0x7f1aec0aed70 2023-10-14T18:19:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc00c9d0 con 0x7f1aec0aed70 2023-10-14T18:19:54.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1adc00cd60 con 0x7f1aec0aed70 2023-10-14T18:19:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec0b01e0 con 0x7f1aec0aed70 2023-10-14T18:19:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.271+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0b07a0 con 0x7f1aec0aed70 2023-10-14T18:19:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.272+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aec04e9b0 con 0x7f1aec0aed70 2023-10-14T18:19:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.273+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 158) v1 ==== 89084+0+0 (secure 0 0 0) 0x7f1adc01d460 con 0x7f1aec0aed70 2023-10-14T18:19:54.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.274+0000 7f1ae27fc640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 0x7f1ad006d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.274+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f1adc089670 con 0x7f1aec0aed70 2023-10-14T18:19:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.275+0000 7f1af1e18640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 0x7f1ad006d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.275+0000 7f1af1e18640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 0x7f1ad006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8006fd0 tx=0x7f1ae8008040 comp rx=0 tx=0).ready entity=mgr.12999 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.279+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1adc0546e0 con 0x7f1aec0aed70 2023-10-14T18:19:54.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.502+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 --> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "path": "snap_test_dir1", "retention_spec_or_period": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f1aec0cc550 con 0x7f1ad006ad40 2023-10-14T18:19:54.677 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.676+0000 7f1ae27fc640 1 -- 172.21.15.29:0/3604274558 <== mgr.12999 v2:172.21.15.29:6832/4074993897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+39 (secure 0 0 0) 0x7f1ae8004250 con 0x7f1ad006ad40 2023-10-14T18:19:54.677 INFO:teuthology.orchestra.run.smithi029.stdout:Retention added to path snap_test_dir1 2023-10-14T18:19:54.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.679+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 msgr2=0x7f1ad006d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.679+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 0x7f1ad006d200 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1ae8006fd0 tx=0x7f1ae8008040 comp rx=0 tx=0).stop 2023-10-14T18:19:54.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 msgr2=0x7f1aec151ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 secure :-1 s=READY pgs=3188 cs=0 l=1 rev1=1 crypto rx=0x7f1adc00e840 tx=0x7f1adc00ed00 comp rx=0 tx=0).stop 2023-10-14T18:19:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 shutdown_connections 2023-10-14T18:19:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1aec0b3200 0x7f1aec0b3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f1ad006ad40 0x7f1ad006d200 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1aec0aed70 0x7f1aec151ae0 unknown :-1 s=CLOSED pgs=3188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 --2- 172.21.15.29:0/3604274558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ad8004d00 0x7f1aec0af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 >> 172.21.15.29:0/3604274558 conn(0x7f1aec0ce6a0 msgr2=0x7f1aec15c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 shutdown_connections 2023-10-14T18:19:54.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.680+0000 7f1af2e1a640 1 -- 172.21.15.29:0/3604274558 wait complete. 2023-10-14T18:19:54.701 DEBUG:tasks.cephfs.test_snap_schedules:res=Retention added to path snap_test_dir1 2023-10-14T18:19:54.701 INFO:tasks.cephfs.test_snap_schedules:expecting snap snap_test_dir1/.snap/scheduled-2023-10-14-18_20_00 in ~6s... 2023-10-14T18:19:54.702 DEBUG:tasks.cephfs.test_snap_schedules:expected_schedule: ['1m'], expected_retention: [{'m': 1}] 2023-10-14T18:19:54.702 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule list --format json snap_test_dir1 2023-10-14T18:19:54.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.935+0000 7fd8b4dd5640 1 Processor -- start 2023-10-14T18:19:54.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.935+0000 7fd8b4dd5640 1 -- start start 2023-10-14T18:19:54.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.936+0000 7fd8b4dd5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd8b015ad70 0x7fd8b0069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.936+0000 7fd8b4dd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00c9ca0 0x7fd8b006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8b4dd5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8b00cb030 con 0x7fd8b00ca3b0 2023-10-14T18:19:54.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8b4dd5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8b00c91c0 con 0x7fd8b015ad70 2023-10-14T18:19:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8b4dd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00c94a0 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8aed9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd8b00ca3b0 0x7fd8b00ca840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39086/0 (socket says 172.21.15.29:39086) 2023-10-14T18:19:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8aed9d640 1 -- 172.21.15.29:0/739222340 learned_addr learned my addr 172.21.15.29:0/739222340 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:54.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/739222340 >> v2:172.21.15.29:3300/0 conn(0x7fd8b015ad70 0x7fd8b0069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.937+0000 7fd8add9b640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00c9ca0 0x7fd8b006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ad59a640 1 -- 172.21.15.29:0/739222340 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338653792 0 0) 0x7fd8b00cb030 con 0x7fd8b00ca3b0 2023-10-14T18:19:54.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ad59a640 1 -- 172.21.15.29:0/739222340 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd898003520 con 0x7fd8b00ca3b0 2023-10-14T18:19:54.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ae59c640 1 -- 172.21.15.29:0/739222340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00c9ca0 msgr2=0x7fd8b006a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00c9ca0 0x7fd8b006a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ae59c640 1 -- 172.21.15.29:0/739222340 >> v1:172.21.15.29:6789/0 conn(0x7fd8b00ca3b0 legacy=0x7fd8b00ca840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.938+0000 7fd8ae59c640 1 -- 172.21.15.29:0/739222340 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8b00c9780 con 0x7fd8b015ad70 2023-10-14T18:19:54.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.939+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b015ad70 0x7fd8b0069da0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00097a0 tx=0x7fd8a002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4746bac94c0828cf server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.939+0000 7fd8ad59a640 1 -- 172.21.15.29:0/739222340 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0031830 con 0x7fd8b015ad70 2023-10-14T18:19:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.939+0000 7fd8ad59a640 1 -- 172.21.15.29:0/739222340 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a0031990 con 0x7fd8b015ad70 2023-10-14T18:19:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.939+0000 7fd8ad59a640 1 -- 172.21.15.29:0/739222340 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0031d00 con 0x7fd8b015ad70 2023-10-14T18:19:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/739222340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b015ad70 msgr2=0x7fd8b0069da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b015ad70 0x7fd8b0069da0 secure :-1 s=READY pgs=3191 cs=0 l=1 rev1=1 crypto rx=0x7fd8a00097a0 tx=0x7fd8a002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/739222340 shutdown_connections 2023-10-14T18:19:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00c9ca0 0x7fd8b006a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/739222340 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b015ad70 0x7fd8b0069da0 unknown :-1 s=CLOSED pgs=3191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/739222340 >> 172.21.15.29:0/739222340 conn(0x7fd8b00ce9b0 msgr2=0x7fd8b00cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:54.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/739222340 shutdown_connections 2023-10-14T18:19:54.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.940+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/739222340 wait complete. 2023-10-14T18:19:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.943+0000 7fd8b4dd5640 1 Processor -- start 2023-10-14T18:19:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.943+0000 7fd8b4dd5640 1 -- start start 2023-10-14T18:19:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 0x7fd8b01591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8b015ad70 0x7fd8b01f2660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b015b450 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b0a40 con 0x7fd8b015ad70 2023-10-14T18:19:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8b4dd5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8b00b0d20 con 0x7fd8b00ca3b0 2023-10-14T18:19:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8ae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8ae59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52192/0 (socket says 172.21.15.29:52192) 2023-10-14T18:19:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.944+0000 7fd8ae59c640 1 -- 172.21.15.29:0/4033523520 learned_addr learned my addr 172.21.15.29:0/4033523520 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8add9b640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 0x7fd8b01591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 -- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 msgr2=0x7fd8b01591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8aed9d640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8b015ad70 0x7fd8b01f2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 0x7fd8b01591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 -- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8b015ad70 msgr2=0x7fd8b01f2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8b015ad70 0x7fd8b01f2660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 -- 172.21.15.29:0/4033523520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8a0009400 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8add9b640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 0x7fd8b01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd8ae59c640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7fd8a002b910 tx=0x7fd8a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.945+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a003a830 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.946+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8a003a9b0 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.946+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8a003ad60 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.952 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.946+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8b00b1000 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.946+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8b00b28d0 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.947+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8b015b670 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.948+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 159) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fd8a004d450 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.949+0000 7fd89effd640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 0x7fd89006d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.949+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fd8a00abb90 con 0x7fd8b00c9ca0 2023-10-14T18:19:54.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.949+0000 7fd8add9b640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 0x7fd89006d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.950+0000 7fd8add9b640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 0x7fd89006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00b02b0 tx=0x7fd8a4008040 comp rx=0 tx=0).ready entity=mgr.12999 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:54.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:54.954+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd8a0076800 con 0x7fd8b00c9ca0 2023-10-14T18:19:55.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.170+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 --> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "path": "snap_test_dir1", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd8b00ad4d0 con 0x7fd89006afe0 2023-10-14T18:19:55.187 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.186+0000 7fd89effd640 1 -- 172.21.15.29:0/4033523520 <== mgr.12999 v2:172.21.15.29:6832/4074993897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+72 (secure 0 0 0) 0x7fd8a4009730 con 0x7fd89006afe0 2023-10-14T18:19:55.189 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:55.189 INFO:teuthology.orchestra.run.smithi029.stdout:{"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-14T18:19:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 msgr2=0x7fd89006d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 0x7fd89006d4a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fd8b00b02b0 tx=0x7fd8a4008040 comp rx=0 tx=0).stop 2023-10-14T18:19:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 msgr2=0x7fd8b00b2390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 secure :-1 s=READY pgs=3192 cs=0 l=1 rev1=1 crypto rx=0x7fd8a002b910 tx=0x7fd8a00046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 shutdown_connections 2023-10-14T18:19:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd8b015ad70 0x7fd8b01f2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd8b00ca3b0 0x7fd8b01591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fd89006afe0 0x7fd89006d4a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 --2- 172.21.15.29:0/4033523520 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd8b00c9ca0 0x7fd8b00b2390 unknown :-1 s=CLOSED pgs=3192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 >> 172.21.15.29:0/4033523520 conn(0x7fd8b00ce9b0 msgr2=0x7fd8b00ad930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 shutdown_connections 2023-10-14T18:19:55.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.189+0000 7fd8b4dd5640 1 -- 172.21.15.29:0/4033523520 wait complete. 2023-10-14T18:19:55.208 DEBUG:tasks.cephfs.test_snap_schedules:res= {"path": "snap_test_dir1", "schedule": ["1m"], "retention": [{"m": 1}]} 2023-10-14T18:19:55.209 DEBUG:tasks.cephfs.test_snap_schedules:json_res: {'path': 'snap_test_dir1', 'schedule': ['1m'], 'retention': [{'m': 1}]} 2023-10-14T18:19:55.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs snap-schedule remove snap_test_dir1 2023-10-14T18:19:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.426+0000 7fe9b4e9f640 1 Processor -- start 2023-10-14T18:19:55.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.426+0000 7fe9b4e9f640 1 -- start start 2023-10-14T18:19:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9b4e9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b015a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9b4e9f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9b0070450 0x7fe9b00708b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9b4e9f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9b0070df0 con 0x7fe9b006c100 2023-10-14T18:19:55.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9b4e9f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9b015e490 con 0x7fe9b0070450 2023-10-14T18:19:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9b4e9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b015aef0 con 0x7fe9b0159950 2023-10-14T18:19:55.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9aeffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fe9b0070450 0x7fe9b00708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9affff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fe9b006c100 0x7fe9b006c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39096/0 (socket says 172.21.15.29:39096) 2023-10-14T18:19:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.428+0000 7fe9affff640 1 -- 172.21.15.29:0/3481976936 learned_addr learned my addr 172.21.15.29:0/3481976936 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9af7fe640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b015a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.431 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9aeffd640 1 -- 172.21.15.29:0/3481976936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 msgr2=0x7fe9b015a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9aeffd640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b015a9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9aeffd640 1 -- 172.21.15.29:0/3481976936 >> v1:172.21.15.29:6789/0 conn(0x7fe9b006c100 legacy=0x7fe9b006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9aeffd640 1 -- 172.21.15.29:0/3481976936 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9b006cc80 con 0x7fe9b0070450 2023-10-14T18:19:55.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.429+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/3481976936 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839177230 0 0) 0x7fe9b0070df0 con 0x7fe9b006c100 2023-10-14T18:19:55.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.430+0000 7fe9aeffd640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b00708b0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98239ac532803531 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/3481976936 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a402e070 con 0x7fe9b0070450 2023-10-14T18:19:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/3481976936 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a40315a0 con 0x7fe9b0070450 2023-10-14T18:19:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9ae7fc640 1 -- 172.21.15.29:0/3481976936 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4031930 con 0x7fe9b0070450 2023-10-14T18:19:55.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/3481976936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 msgr2=0x7fe9b00708b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b00708b0 secure :-1 s=READY pgs=3194 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a402b840 comp rx=0 tx=0).stop 2023-10-14T18:19:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/3481976936 shutdown_connections 2023-10-14T18:19:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b00708b0 unknown :-1 s=CLOSED pgs=3194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/3481976936 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b015a9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.435 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.431+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/3481976936 >> 172.21.15.29:0/3481976936 conn(0x7fe9b00ce6a0 msgr2=0x7fe9b00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:55.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.432+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/3481976936 shutdown_connections 2023-10-14T18:19:55.436 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.432+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/3481976936 wait complete. 2023-10-14T18:19:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.433+0000 7fe9b4e9f640 1 Processor -- start 2023-10-14T18:19:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.433+0000 7fe9b4e9f640 1 -- start start 2023-10-14T18:19:55.438 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9b006c100 0x7fe9b00ccd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b014f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b00b12c0 con 0x7fe9b0070450 2023-10-14T18:19:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b015b130 con 0x7fe9b006c100 2023-10-14T18:19:55.439 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.434+0000 7fe9b4e9f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9b015b410 con 0x7fe9b0159950 2023-10-14T18:19:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52200/0 (socket says 172.21.15.29:52200) 2023-10-14T18:19:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 -- 172.21.15.29:0/1100090251 learned_addr learned my addr 172.21.15.29:0/1100090251 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:55.440 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9af7fe640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9b006c100 0x7fe9b00ccd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9affff640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b014f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 -- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 msgr2=0x7fe9b014f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b014f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 -- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9b006c100 msgr2=0x7fe9b00ccd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9b006c100 0x7fe9b00ccd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.435+0000 7fe9aeffd640 1 -- 172.21.15.29:0/1100090251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a4009400 con 0x7fe9b0070450 2023-10-14T18:19:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe9aeffd640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4004230 con 0x7fe9b0070450 2023-10-14T18:19:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe9a4004390 con 0x7fe9b0070450 2023-10-14T18:19:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9b015b6f0 con 0x7fe9b0070450 2023-10-14T18:19:55.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9a4003850 con 0x7fe9b0070450 2023-10-14T18:19:55.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.436+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9b015bcb0 con 0x7fe9b0070450 2023-10-14T18:19:55.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.437+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9b004e9b0 con 0x7fe9b0070450 2023-10-14T18:19:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.438+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 159) v1 ==== 89349+0+0 (secure 0 0 0) 0x7fe9a40039b0 con 0x7fe9b0070450 2023-10-14T18:19:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.439+0000 7fe997fff640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 0x7fe99c06d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.439+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fe9a40abc00 con 0x7fe9b0070450 2023-10-14T18:19:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.440+0000 7fe9af7fe640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 0x7fe99c06d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:55.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.440+0000 7fe9af7fe640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 0x7fe99c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00c9500 tx=0x7fe9a0008040 comp rx=0 tx=0).ready entity=mgr.12999 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:55.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.444+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe9a40767d0 con 0x7fe9b0070450 2023-10-14T18:19:55.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.661+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 --> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] -- mgr_command(tid 0: {"prefix": "fs snap-schedule remove", "path": "snap_test_dir1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe9b015dbf0 con 0x7fe99c06afe0 2023-10-14T18:19:55.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.867+0000 7fe997fff640 1 -- 172.21.15.29:0/1100090251 <== mgr.12999 v2:172.21.15.29:6832/4074993897 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+41 (secure 0 0 0) 0x7fe9a00119a0 con 0x7fe99c06afe0 2023-10-14T18:19:55.869 INFO:teuthology.orchestra.run.smithi029.stdout:Schedule removed for path snap_test_dir1 2023-10-14T18:19:55.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 msgr2=0x7fe99c06d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 0x7fe99c06d4a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe9b00c9500 tx=0x7fe9a0008040 comp rx=0 tx=0).stop 2023-10-14T18:19:55.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 msgr2=0x7fe9b014ebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:55.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 secure :-1 s=READY pgs=3195 cs=0 l=1 rev1=1 crypto rx=0x7fe9a40097a0 tx=0x7fe9a40046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 shutdown_connections 2023-10-14T18:19:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe9b0159950 0x7fe9b014f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fe99c06afe0 0x7fe99c06d4a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe9b0070450 0x7fe9b014ebd0 unknown :-1 s=CLOSED pgs=3195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 --2- 172.21.15.29:0/1100090251 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe9b006c100 0x7fe9b00ccd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:55.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 >> 172.21.15.29:0/1100090251 conn(0x7fe9b00ce6a0 msgr2=0x7fe9b01531d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:55.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 shutdown_connections 2023-10-14T18:19:55.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:55.871+0000 7fe9b4e9f640 1 -- 172.21.15.29:0/1100090251 wait complete. 2023-10-14T18:19:55.892 DEBUG:tasks.cephfs.test_snap_schedules:res=Schedule removed for path snap_test_dir1 2023-10-14T18:19:55.893 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:55.893 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec ls snap_test_dir1/.snap) 2023-10-14T18:19:55.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:55.954 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest/mnt.0 && exec rmdir snap_test_dir1) 2023-10-14T18:19:56.011 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr module disable snap_schedule 2023-10-14T18:19:56.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.246+0000 7f42817f4640 1 Processor -- start 2023-10-14T18:19:56.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.246+0000 7f42817f4640 1 -- start start 2023-10-14T18:19:56.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f42817f4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f427c0b3770 0x7f427c0addc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f42817f4640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f427c0ae300 0x7f427c161370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f42817f4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f427c06fe40 con 0x7f427c160f90 2023-10-14T18:19:56.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f42817f4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c0ae790 con 0x7f427c0b3770 2023-10-14T18:19:56.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f42817f4640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f427c06f270 con 0x7f427c0ae300 2023-10-14T18:19:56.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f427b7fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f427c0b3770 0x7f427c0addc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f427bfff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f427c160f90 0x7f427c0b09c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39112/0 (socket says 172.21.15.29:39112) 2023-10-14T18:19:56.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.248+0000 7f427bfff640 1 -- 172.21.15.29:0/3748987542 learned_addr learned my addr 172.21.15.29:0/3748987542 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f4280ff3640 1 --2- 172.21.15.29:0/3748987542 >> v2:172.21.15.186:3300/0 conn(0x7f427c0ae300 0x7f427c161370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f427b7fe640 1 -- 172.21.15.29:0/3748987542 >> v2:172.21.15.186:3300/0 conn(0x7f427c0ae300 msgr2=0x7f427c161370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:56.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f427b7fe640 1 --2- 172.21.15.29:0/3748987542 >> v2:172.21.15.186:3300/0 conn(0x7f427c0ae300 0x7f427c161370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f427b7fe640 1 -- 172.21.15.29:0/3748987542 >> v1:172.21.15.29:6789/0 conn(0x7f427c160f90 legacy=0x7f427c0b09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f427affd640 1 -- 172.21.15.29:0/3748987542 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1654828310 0 0) 0x7f427c06fe40 con 0x7f427c160f90 2023-10-14T18:19:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.249+0000 7f427b7fe640 1 -- 172.21.15.29:0/3748987542 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c06f510 con 0x7f427c0b3770 2023-10-14T18:19:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.250+0000 7f427b7fe640 1 --2- 172.21.15.29:0/3748987542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c0addc0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f427c0b1ca0 tx=0x7f427002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b326a08d8d55965b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:56.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.250+0000 7f427affd640 1 -- 172.21.15.29:0/3748987542 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270031830 con 0x7f427c0b3770 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.250+0000 7f427affd640 1 -- 172.21.15.29:0/3748987542 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4270031990 con 0x7f427c0b3770 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.250+0000 7f427affd640 1 -- 172.21.15.29:0/3748987542 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270031d20 con 0x7f427c0b3770 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 -- 172.21.15.29:0/3748987542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 msgr2=0x7f427c0addc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 --2- 172.21.15.29:0/3748987542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c0addc0 secure :-1 s=READY pgs=3197 cs=0 l=1 rev1=1 crypto rx=0x7f427c0b1ca0 tx=0x7f427002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 -- 172.21.15.29:0/3748987542 shutdown_connections 2023-10-14T18:19:56.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 --2- 172.21.15.29:0/3748987542 >> v2:172.21.15.186:3300/0 conn(0x7f427c0ae300 0x7f427c161370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 --2- 172.21.15.29:0/3748987542 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c0addc0 unknown :-1 s=CLOSED pgs=3197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 -- 172.21.15.29:0/3748987542 >> 172.21.15.29:0/3748987542 conn(0x7f427c069da0 msgr2=0x7f427c06a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:56.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 -- 172.21.15.29:0/3748987542 shutdown_connections 2023-10-14T18:19:56.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.251+0000 7f42817f4640 1 -- 172.21.15.29:0/3748987542 wait complete. 2023-10-14T18:19:56.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.253+0000 7f42817f4640 1 Processor -- start 2023-10-14T18:19:56.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.253+0000 7f42817f4640 1 -- start start 2023-10-14T18:19:56.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f427c0ae300 0x7f427c0abc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c160f90 0x7f427c163380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c14e1c0 con 0x7f427c0b3770 2023-10-14T18:19:56.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427c14fed0 con 0x7f427c160f90 2023-10-14T18:19:56.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f42817f4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f427c1501b0 con 0x7f427c0ae300 2023-10-14T18:19:56.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f427b7fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52214/0 (socket says 172.21.15.29:52214) 2023-10-14T18:19:56.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.254+0000 7f427b7fe640 1 -- 172.21.15.29:0/527587379 learned_addr learned my addr 172.21.15.29:0/527587379 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:56.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427bfff640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f427c0ae300 0x7f427c0abc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f4280ff3640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c160f90 0x7f427c163380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 -- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f427c0ae300 msgr2=0x7f427c0abc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f427c0ae300 0x7f427c0abc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 -- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c160f90 msgr2=0x7f427c163380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c160f90 0x7f427c163380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 -- 172.21.15.29:0/527587379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4270009400 con 0x7f427c0b3770 2023-10-14T18:19:56.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.255+0000 7f427b7fe640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f4270009890 tx=0x7f42700046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:56.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.256+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270004230 con 0x7f427c0b3770 2023-10-14T18:19:56.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.256+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42700043b0 con 0x7f427c0b3770 2023-10-14T18:19:56.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.256+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4270004520 con 0x7f427c0b3770 2023-10-14T18:19:56.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.256+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427c150490 con 0x7f427c0b3770 2023-10-14T18:19:56.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.256+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427c150a50 con 0x7f427c0b3770 2023-10-14T18:19:56.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.257+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f427c04e9b0 con 0x7f427c0b3770 2023-10-14T18:19:56.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.258+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 159) v1 ==== 89349+0+0 (secure 0 0 0) 0x7f427003a8a0 con 0x7f427c0b3770 2023-10-14T18:19:56.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.259+0000 7f4278ff9640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 0x7f426806d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:56.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.260+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f42700ac5b0 con 0x7f427c0b3770 2023-10-14T18:19:56.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.260+0000 7f427bfff640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 0x7f426806d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:56.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.260+0000 7f427bfff640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 0x7f426806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f426c000f60 tx=0x7f426c007c00 comp rx=0 tx=0).ready entity=mgr.12999 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:56.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.264+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f42700772a0 con 0x7f427c0b3770 2023-10-14T18:19:56.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.517+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "mgr module disable", "module": "snap_schedule"} v 0) v1 -- 0x7f427c0734c0 con 0x7f427c0b3770 2023-10-14T18:19:56.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.811+0000 7f4278ff9640 1 -- 172.21.15.29:0/527587379 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module disable", "module": "snap_schedule"}]=0 v160) v1 ==== 93+0+0 (secure 0 0 0) 0x7f427007a8c0 con 0x7f427c0b3770 2023-10-14T18:19:56.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.815+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 msgr2=0x7f426806d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.815+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 0x7f426806d4a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f426c000f60 tx=0x7f426c007c00 comp rx=0 tx=0).stop 2023-10-14T18:19:56.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.815+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 msgr2=0x7f427c150f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:56.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.815+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 secure :-1 s=READY pgs=3198 cs=0 l=1 rev1=1 crypto rx=0x7f4270009890 tx=0x7f42700046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 shutdown_connections 2023-10-14T18:19:56.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f427c160f90 0x7f427c163380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f426806afe0 0x7f426806d4a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f427c0b3770 0x7f427c150f70 unknown :-1 s=CLOSED pgs=3198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 --2- 172.21.15.29:0/527587379 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f427c0ae300 0x7f427c0abc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:56.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.816+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 >> 172.21.15.29:0/527587379 conn(0x7f427c069da0 msgr2=0x7f427c06c610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.817+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 shutdown_connections 2023-10-14T18:19:56.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:56.818+0000 7f42817f4640 1 -- 172.21.15.29:0/527587379 wait complete. 2023-10-14T18:19:56.841 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2023-10-14T18:19:56.842 DEBUG:teuthology.orchestra.run.smithi029:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:56.935 DEBUG:teuthology.orchestra.run.smithi186:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2023-10-14T18:19:57.024 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2023-10-14T18:19:57.024 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:57.025 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:57.081 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:57.081 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:57.131 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:19:57.332 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.332+0000 7fcc1e21a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:57.366 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:57.365+0000 7ff125c11040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-14T18:19:57.386 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.386+0000 7fcc1e21a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.389+0000 7f9877244640 1 Processor -- start 2023-10-14T18:19:57.390 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.389+0000 7f9877244640 1 -- start start 2023-10-14T18:19:57.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.390+0000 7f9877244640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98700b3070 0x7f98700cf500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.390+0000 7f9877244640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f98700d1b10 0x7f98700cfa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.390+0000 7f9877244640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700b3940 con 0x7f98700b3e70 2023-10-14T18:19:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.390+0000 7f9877244640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700b45e0 con 0x7f98700b3070 2023-10-14T18:19:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.390+0000 7f9877244640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98700b3b90 con 0x7f98700d1b10 2023-10-14T18:19:57.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98700b3070 0x7f98700cf500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f98700b3070 0x7f98700cf500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52234/0 (socket says 172.21.15.29:52234) 2023-10-14T18:19:57.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 -- 172.21.15.29:0/2831820460 learned_addr learned my addr 172.21.15.29:0/2831820460 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:57.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f98757e1640 1 --2- 172.21.15.29:0/2831820460 >> v2:172.21.15.186:3300/0 conn(0x7f98700d1b10 0x7f98700cfa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 -- 172.21.15.29:0/2831820460 >> v2:172.21.15.186:3300/0 conn(0x7f98700d1b10 msgr2=0x7f98700cfa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 --2- 172.21.15.29:0/2831820460 >> v2:172.21.15.186:3300/0 conn(0x7f98700d1b10 0x7f98700cfa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 -- 172.21.15.29:0/2831820460 >> v1:172.21.15.186:6789/0 conn(0x7f98700b3e70 legacy=0x7f98700c6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:57.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.391+0000 7f9874fe0640 1 -- 172.21.15.29:0/2831820460 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700d3840 con 0x7f98700b3070 2023-10-14T18:19:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.392+0000 7f9874fe0640 1 --2- 172.21.15.29:0/2831820460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700b3070 0x7f98700cf500 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f987008c1c0 tx=0x7f986402bf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2af5f62bc8121a70 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.393+0000 7f986f7fe640 1 -- 172.21.15.29:0/2831820460 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f986402cad0 con 0x7f98700b3070 2023-10-14T18:19:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.393+0000 7f986f7fe640 1 -- 172.21.15.29:0/2831820460 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f986402cc30 con 0x7f98700b3070 2023-10-14T18:19:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.393+0000 7f986f7fe640 1 -- 172.21.15.29:0/2831820460 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98640343f0 con 0x7f98700b3070 2023-10-14T18:19:57.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.393+0000 7f9877244640 1 -- 172.21.15.29:0/2831820460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700b3070 msgr2=0x7f98700cf500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.393+0000 7f9877244640 1 --2- 172.21.15.29:0/2831820460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700b3070 0x7f98700cf500 secure :-1 s=READY pgs=3203 cs=0 l=1 rev1=1 crypto rx=0x7f987008c1c0 tx=0x7f986402bf30 comp rx=0 tx=0).stop 2023-10-14T18:19:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 -- 172.21.15.29:0/2831820460 shutdown_connections 2023-10-14T18:19:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 --2- 172.21.15.29:0/2831820460 >> v2:172.21.15.186:3300/0 conn(0x7f98700d1b10 0x7f98700cfa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 --2- 172.21.15.29:0/2831820460 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700b3070 0x7f98700cf500 unknown :-1 s=CLOSED pgs=3203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 -- 172.21.15.29:0/2831820460 >> 172.21.15.29:0/2831820460 conn(0x7f98700d2d20 msgr2=0x7f98700d3130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 -- 172.21.15.29:0/2831820460 shutdown_connections 2023-10-14T18:19:57.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.394+0000 7f9877244640 1 -- 172.21.15.29:0/2831820460 wait complete. 2023-10-14T18:19:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.395+0000 7f9877244640 1 Processor -- start 2023-10-14T18:19:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.396+0000 7f9877244640 1 -- start start 2023-10-14T18:19:57.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.396+0000 7f9877244640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98700b3070 0x7f98700b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.396+0000 7f9877244640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98700b3e70 0x7f9870151070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.396+0000 7f9877244640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f9877244640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700b24f0 con 0x7f98700d1b10 2023-10-14T18:19:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f9877244640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98700b27d0 con 0x7f98700b3e70 2023-10-14T18:19:57.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f9877244640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98700b59b0 con 0x7f98700b3070 2023-10-14T18:19:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52246/0 (socket says 172.21.15.29:52246) 2023-10-14T18:19:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 -- 172.21.15.29:0/2264233562 learned_addr learned my addr 172.21.15.29:0/2264233562 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f9874fe0640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98700b3070 0x7f98700b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f986ffff640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98700b3e70 0x7f9870151070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98700b3070 msgr2=0x7f98700b0370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98700b3070 0x7f98700b0370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98700b3e70 msgr2=0x7f9870151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98700b3e70 0x7f9870151070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.397+0000 7f98757e1640 1 -- 172.21.15.29:0/2264233562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9864009b30 con 0x7f98700d1b10 2023-10-14T18:19:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f98757e1640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f986800d5d0 tx=0x7f986800da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98680125f0 con 0x7f98700d1b10 2023-10-14T18:19:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9868012770 con 0x7f98700d1b10 2023-10-14T18:19:57.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9868012b20 con 0x7f98700d1b10 2023-10-14T18:19:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98700b5c90 con 0x7f98700d1b10 2023-10-14T18:19:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.398+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98701515b0 con 0x7f98700d1b10 2023-10-14T18:19:57.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.399+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98700b61b0 con 0x7f98700d1b10 2023-10-14T18:19:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.400+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f9868012c80 con 0x7f98700d1b10 2023-10-14T18:19:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.401+0000 7f986d7fa640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 0x7f985406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.401+0000 7f9874fe0640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 msgr2=0x7f985406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:57.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.401+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f986808a020 con 0x7f98700d1b10 2023-10-14T18:19:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.401+0000 7f9874fe0640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 0x7f985406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:57.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.406+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f98700b61b0 con 0x7f98700d1b10 2023-10-14T18:19:57.426 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:57.425+0000 7ff125c11040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-14T18:19:57.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.602+0000 7f9874fe0640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 msgr2=0x7f985406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:57.603 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.602+0000 7f9874fe0640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 0x7f985406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:57.604 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.603+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f98700b61b0 con 0x7f98700d1b10 2023-10-14T18:19:57.605 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.605+0000 7f986d7fa640 1 -- 172.21.15.29:0/2264233562 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) v1 ==== 74+0+17326 (secure 0 0 0) 0x7f98700b61b0 con 0x7f98700d1b10 2023-10-14T18:19:57.606 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:19:57.606 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":272,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:52.752557+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2196987879":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3551524051":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3292195983":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/4006438795":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6832/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3279571573":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1082110076":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2709558548":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1343183116":"2023-10-15T18:19:52.752456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:19:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 msgr2=0x7f985406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 0x7f985406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.609 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 msgr2=0x7f98700b1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 secure :-1 s=READY pgs=3204 cs=0 l=1 rev1=1 crypto rx=0x7f986800d5d0 tx=0x7f986800da90 comp rx=0 tx=0).stop 2023-10-14T18:19:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 shutdown_connections 2023-10-14T18:19:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f985406b020 0x7f985406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f98700d1b10 0x7f98700b1bc0 unknown :-1 s=CLOSED pgs=3204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f98700b3e70 0x7f9870151070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 --2- 172.21.15.29:0/2264233562 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f98700b3070 0x7f98700b0370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 >> 172.21.15.29:0/2264233562 conn(0x7f98700d2d20 msgr2=0x7f9870160560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:57.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 shutdown_connections 2023-10-14T18:19:57.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.608+0000 7f9877244640 1 -- 172.21.15.29:0/2264233562 wait complete. 2023-10-14T18:19:57.628 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:19:35.816866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:19:36.425584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:19:57.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:19:57.835 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.835+0000 7fcc1e21a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:57.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.851+0000 7fbb4f7e9640 1 Processor -- start 2023-10-14T18:19:57.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.851+0000 7fbb4f7e9640 1 -- start start 2023-10-14T18:19:57.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4f7e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 0x7fbb480ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4f7e9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb480cb030 con 0x7fbb480c9ca0 2023-10-14T18:19:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4f7e9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb480c91c0 con 0x7fbb4815ad70 2023-10-14T18:19:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4f7e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480c94a0 con 0x7fbb4806a4b0 2023-10-14T18:19:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4dd86640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fbb480c9ca0 0x7fbb480ad4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39136/0 (socket says 172.21.15.29:39136) 2023-10-14T18:19:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.853+0000 7fbb4dd86640 1 -- 172.21.15.29:0/68314727 learned_addr learned my addr 172.21.15.29:0/68314727 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:57.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb4cd84640 1 --2- 172.21.15.29:0/68314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 0x7fbb480ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1515185918 0 0) 0x7fbb480cb030 con 0x7fbb480c9ca0 2023-10-14T18:19:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb34003520 con 0x7fbb480c9ca0 2023-10-14T18:19:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb4cd84640 1 -- 172.21.15.29:0/68314727 >> v1:172.21.15.186:6789/0 conn(0x7fbb4815ad70 legacy=0x7fbb48069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb4cd84640 1 -- 172.21.15.29:0/68314727 >> v1:172.21.15.29:6789/0 conn(0x7fbb480c9ca0 legacy=0x7fbb480ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 856228396 0 0) 0x7fbb34003520 con 0x7fbb480c9ca0 2023-10-14T18:19:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.854+0000 7fbb4cd84640 1 -- 172.21.15.29:0/68314727 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb480c9780 con 0x7fbb4806a4b0 2023-10-14T18:19:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.856+0000 7fbb4cd84640 1 --2- 172.21.15.29:0/68314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 0x7fbb480ca3b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbb380097d0 tx=0x7fbb3802b800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=105564aceb33f36a server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.856+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb400019f0 con 0x7fbb4806a4b0 2023-10-14T18:19:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.856+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb3802e050 con 0x7fbb4806a4b0 2023-10-14T18:19:57.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.856+0000 7fbb3ffff640 1 -- 172.21.15.29:0/68314727 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38030730 con 0x7fbb4806a4b0 2023-10-14T18:19:57.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/68314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 msgr2=0x7fbb480ca3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/68314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 0x7fbb480ca3b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbb380097d0 tx=0x7fbb3802b800 comp rx=0 tx=0).stop 2023-10-14T18:19:57.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/68314727 shutdown_connections 2023-10-14T18:19:57.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/68314727 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4806a4b0 0x7fbb480ca3b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/68314727 >> 172.21.15.29:0/68314727 conn(0x7fbb480ce9b0 msgr2=0x7fbb480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/68314727 shutdown_connections 2023-10-14T18:19:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.857+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/68314727 wait complete. 2023-10-14T18:19:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.859+0000 7fbb4f7e9640 1 Processor -- start 2023-10-14T18:19:57.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.859+0000 7fbb4f7e9640 1 -- start start 2023-10-14T18:19:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4806a4b0 0x7fbb48151b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4815ad70 0x7fbb480ae810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480ad960 con 0x7fbb480c9ca0 2023-10-14T18:19:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb48152040 con 0x7fbb4806a4b0 2023-10-14T18:19:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4f7e9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb480acd50 con 0x7fbb4815ad70 2023-10-14T18:19:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.864 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52260/0 (socket says 172.21.15.29:52260) 2023-10-14T18:19:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 -- 172.21.15.29:0/3232076360 learned_addr learned my addr 172.21.15.29:0/3232076360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4dd86640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4815ad70 0x7fbb480ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4d585640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4806a4b0 0x7fbb48151b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4815ad70 msgr2=0x7fbb480ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.865 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4815ad70 0x7fbb480ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4806a4b0 msgr2=0x7fbb48151b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:57.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4806a4b0 0x7fbb48151b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:57.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.860+0000 7fbb4cd84640 1 -- 172.21.15.29:0/3232076360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb38009400 con 0x7fbb480c9ca0 2023-10-14T18:19:57.866 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb4cd84640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009720 tx=0x7fbb380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:57.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb38004220 con 0x7fbb480c9ca0 2023-10-14T18:19:57.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb38004380 con 0x7fbb480c9ca0 2023-10-14T18:19:57.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb480ad030 con 0x7fbb480c9ca0 2023-10-14T18:19:57.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb3803c8c0 con 0x7fbb480c9ca0 2023-10-14T18:19:57.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.861+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb480ad5f0 con 0x7fbb480c9ca0 2023-10-14T18:19:57.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.863+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb4804e160 con 0x7fbb480c9ca0 2023-10-14T18:19:57.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.864+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7fbb38039040 con 0x7fbb480c9ca0 2023-10-14T18:19:57.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.865+0000 7fbb3dffb640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 0x7fbb2c06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:57.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.865+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7fbb380ab7d0 con 0x7fbb480c9ca0 2023-10-14T18:19:57.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.865+0000 7fbb4d585640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 msgr2=0x7fbb2c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:57.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.865+0000 7fbb4d585640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 0x7fbb2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:57.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:57.873+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fbb38076360 con 0x7fbb480c9ca0 2023-10-14T18:19:57.888 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.888+0000 7fcc1e21a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:57.912 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:57.911+0000 7ff125c11040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-14T18:19:57.938 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.938+0000 7fcc1e21a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:57.968 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:57.967+0000 7ff125c11040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-14T18:19:57.999 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:57.999+0000 7fcc1e21a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:58.021 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.020+0000 7ff125c11040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-14T18:19:58.048 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.048+0000 7fcc1e21a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:58.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.066+0000 7fbb4d585640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 msgr2=0x7fbb2c06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:58.067 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.066+0000 7fbb4d585640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 0x7fbb2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:58.068 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.067+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbb480736e0 con 0x7fbb480c9ca0 2023-10-14T18:19:58.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.068+0000 7fbb3dffb640 1 -- 172.21.15.29:0/3232076360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v272) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbb38037020 con 0x7fbb480c9ca0 2023-10-14T18:19:58.070 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:19:58.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 msgr2=0x7fbb2c06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:58.073 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 0x7fbb2c06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 msgr2=0x7fbb480addf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.074 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 secure :-1 s=READY pgs=3206 cs=0 l=1 rev1=1 crypto rx=0x7fbb38009720 tx=0x7fbb380046c0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 shutdown_connections 2023-10-14T18:19:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fbb4815ad70 0x7fbb480ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.075 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7fbb2c06b020 0x7fbb2c06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fbb480c9ca0 0x7fbb480addf0 unknown :-1 s=CLOSED pgs=3206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 --2- 172.21.15.29:0/3232076360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fbb4806a4b0 0x7fbb48151b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 >> 172.21.15.29:0/3232076360 conn(0x7fbb480ce9b0 msgr2=0x7fbb4806b030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 shutdown_connections 2023-10-14T18:19:58.077 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.071+0000 7fbb4f7e9640 1 -- 172.21.15.29:0/3232076360 wait complete. 2023-10-14T18:19:58.087 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.087+0000 7ff125c11040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-14T18:19:58.089 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:19:58.142 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.141+0000 7ff125c11040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-14T18:19:58.158 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.158+0000 7fcc1e21a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:58.221 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.220+0000 7fcc1e21a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:58.260 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.259+0000 7ff125c11040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-14T18:19:58.269 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.269+0000 7fcc1e21a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:58.298 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.297+0000 7f6b9e141640 1 Processor -- start 2023-10-14T18:19:58.299 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.297+0000 7f6b9e141640 1 -- start start 2023-10-14T18:19:58.301 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b9e141640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6b98159ab0 0x7f6b980ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b9e141640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b98159e90 0x7f6b9806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b9e141640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b980cd1f0 con 0x7f6b9806c140 2023-10-14T18:19:58.302 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b9e141640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9815d890 con 0x7f6b98159e90 2023-10-14T18:19:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b9e141640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b980cd4d0 con 0x7f6b98159ab0 2023-10-14T18:19:58.303 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b96ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b98159e90 0x7f6b9806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b96ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f6b98159e90 0x7f6b9806fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52274/0 (socket says 172.21.15.29:52274) 2023-10-14T18:19:58.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.299+0000 7f6b96ffd640 1 -- 172.21.15.29:0/2763329695 learned_addr learned my addr 172.21.15.29:0/2763329695 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:58.304 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b977fe640 1 --2- 172.21.15.29:0/2763329695 >> v2:172.21.15.186:3300/0 conn(0x7f6b98159ab0 0x7f6b980ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b96ffd640 1 -- 172.21.15.29:0/2763329695 >> v2:172.21.15.186:3300/0 conn(0x7f6b98159ab0 msgr2=0x7f6b980ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:58.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/2763329695 >> v2:172.21.15.186:3300/0 conn(0x7f6b98159ab0 0x7f6b980ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.305 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b96ffd640 1 -- 172.21.15.29:0/2763329695 >> v1:172.21.15.29:6789/0 conn(0x7f6b9806c140 legacy=0x7f6b9815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b96ffd640 1 -- 172.21.15.29:0/2763329695 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b980cd7b0 con 0x7f6b98159e90 2023-10-14T18:19:58.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.300+0000 7f6b967fc640 1 -- 172.21.15.29:0/2763329695 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1985499302 0 0) 0x7f6b980cd1f0 con 0x7f6b9806c140 2023-10-14T18:19:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.301+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/2763329695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b98159e90 0x7f6b9806fc50 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f6b980cfa00 tx=0x7f6b8c02e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be7d45212a98f0c5 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:58.309 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.301+0000 7f6b967fc640 1 -- 172.21.15.29:0/2763329695 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c02ee00 con 0x7f6b98159e90 2023-10-14T18:19:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.301+0000 7f6b967fc640 1 -- 172.21.15.29:0/2763329695 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c036440 con 0x7f6b98159e90 2023-10-14T18:19:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.301+0000 7f6b967fc640 1 -- 172.21.15.29:0/2763329695 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8c0367d0 con 0x7f6b98159e90 2023-10-14T18:19:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 -- 172.21.15.29:0/2763329695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b98159e90 msgr2=0x7f6b9806fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.310 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 --2- 172.21.15.29:0/2763329695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b98159e90 0x7f6b9806fc50 secure :-1 s=READY pgs=3208 cs=0 l=1 rev1=1 crypto rx=0x7f6b980cfa00 tx=0x7f6b8c02e050 comp rx=0 tx=0).stop 2023-10-14T18:19:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 -- 172.21.15.29:0/2763329695 shutdown_connections 2023-10-14T18:19:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 --2- 172.21.15.29:0/2763329695 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b98159e90 0x7f6b9806fc50 unknown :-1 s=CLOSED pgs=3208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 --2- 172.21.15.29:0/2763329695 >> v2:172.21.15.186:3300/0 conn(0x7f6b98159ab0 0x7f6b980ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 -- 172.21.15.29:0/2763329695 >> 172.21.15.29:0/2763329695 conn(0x7f6b980ce9b0 msgr2=0x7f6b980cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:58.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 -- 172.21.15.29:0/2763329695 shutdown_connections 2023-10-14T18:19:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.302+0000 7f6b9e141640 1 -- 172.21.15.29:0/2763329695 wait complete. 2023-10-14T18:19:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.304+0000 7f6b9e141640 1 Processor -- start 2023-10-14T18:19:58.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.304+0000 7f6b9e141640 1 -- start start 2023-10-14T18:19:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 0x7f6b980b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b98159e90 0x7f6b98158f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980b4010 con 0x7f6b9806c140 2023-10-14T18:19:58.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980b1b10 con 0x7f6b98159ab0 2023-10-14T18:19:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b9e141640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b980b25c0 con 0x7f6b98159e90 2023-10-14T18:19:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b977fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52288/0 (socket says 172.21.15.29:52288) 2023-10-14T18:19:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b977fe640 1 -- 172.21.15.29:0/450798395 learned_addr learned my addr 172.21.15.29:0/450798395 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:58.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.305+0000 7f6b97fff640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b98159e90 0x7f6b98158f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 0x7f6b980b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b98159e90 msgr2=0x7f6b98158f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b98159e90 0x7f6b98158f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 msgr2=0x7f6b980b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 0x7f6b980b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 -- 172.21.15.29:0/450798395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b8c00b770 con 0x7f6b9806c140 2023-10-14T18:19:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 0x7f6b980b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b977fe640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800d800 tx=0x7f6b8800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8800b6c0 con 0x7f6b9806c140 2023-10-14T18:19:58.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8800b820 con 0x7f6b9806c140 2023-10-14T18:19:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b980b1d20 con 0x7f6b9806c140 2023-10-14T18:19:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.306+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6b8800bbb0 con 0x7f6b9806c140 2023-10-14T18:19:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.307+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b980b22e0 con 0x7f6b9806c140 2023-10-14T18:19:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.308+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b980ca960 con 0x7f6b9806c140 2023-10-14T18:19:58.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.309+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f6b8800bd10 con 0x7f6b9806c140 2023-10-14T18:19:58.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.309+0000 7f6b7ffff640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 0x7f6b8406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.310+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f6b8808a190 con 0x7f6b9806c140 2023-10-14T18:19:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.310+0000 7f6b96ffd640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 msgr2=0x7f6b8406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.310+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 0x7f6b8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:58.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.314+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6b88054d00 con 0x7f6b9806c140 2023-10-14T18:19:58.327 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.327+0000 7fcc1e21a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:58.329 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.329+0000 7ff125c11040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:58.377 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.377+0000 7fcc1e21a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:58.382 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.381+0000 7ff125c11040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-14T18:19:58.442 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.441+0000 7ff125c11040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-14T18:19:58.495 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.495+0000 7ff125c11040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-14T18:19:58.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.507+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f6b9804e160 con 0x7f6b9806c140 2023-10-14T18:19:58.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.508+0000 7f6b7ffff640 1 -- 172.21.15.29:0/450798395 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v272) v1 ==== 104+0+11 (secure 0 0 0) 0x7f6b88058320 con 0x7f6b9806c140 2023-10-14T18:19:58.510 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:19:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 msgr2=0x7f6b8406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:58.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b96ffd640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 msgr2=0x7f6b8406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b96ffd640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 0x7f6b8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 0x7f6b8406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 msgr2=0x7f6b980b3ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 secure :-1 s=READY pgs=3209 cs=0 l=1 rev1=1 crypto rx=0x7f6b8800d800 tx=0x7f6b8800dcc0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 shutdown_connections 2023-10-14T18:19:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6b98159e90 0x7f6b98158f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6b98159ab0 0x7f6b980b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6b8406b020 0x7f6b8406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 --2- 172.21.15.29:0/450798395 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6b9806c140 0x7f6b980b3ad0 unknown :-1 s=CLOSED pgs=3209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 >> 172.21.15.29:0/450798395 conn(0x7f6b980ce9b0 msgr2=0x7f6b9806b6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 shutdown_connections 2023-10-14T18:19:58.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.511+0000 7f6b9e141640 1 -- 172.21.15.29:0/450798395 wait complete. 2023-10-14T18:19:58.531 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:19:58.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.752+0000 7f456d1ff640 1 Processor -- start 2023-10-14T18:19:58.754 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.753+0000 7f456d1ff640 1 -- start start 2023-10-14T18:19:58.754 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.754+0000 7fcc1e21a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:58.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f456d1ff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f456815cd70 0x7f45680d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f456d1ff640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 0x7f45680cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f456d1ff640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f456815d150 con 0x7f45680d0be0 2023-10-14T18:19:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f456d1ff640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45680d1180 con 0x7f456815cd70 2023-10-14T18:19:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f456d1ff640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45680d1460 con 0x7f45680b8480 2023-10-14T18:19:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.754+0000 7f4567fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f456815cd70 0x7f45680d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f456815cd70 0x7f45680d2220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52294/0 (socket says 172.21.15.29:52294) 2023-10-14T18:19:58.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f456c9fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f45680d0be0 0x7f45680b5c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39154/0 (socket says 172.21.15.29:39154) 2023-10-14T18:19:58.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 -- 172.21.15.29:0/2496693077 learned_addr learned my addr 172.21.15.29:0/2496693077 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f45677fe640 1 --2- 172.21.15.29:0/2496693077 >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 0x7f45680cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 -- 172.21.15.29:0/2496693077 >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 msgr2=0x7f45680cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 --2- 172.21.15.29:0/2496693077 >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 0x7f45680cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 -- 172.21.15.29:0/2496693077 >> v1:172.21.15.29:6789/0 conn(0x7f45680d0be0 legacy=0x7f45680b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.759 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4567fff640 1 -- 172.21.15.29:0/2496693077 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45680d1740 con 0x7f456815cd70 2023-10-14T18:19:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f4566ffd640 1 -- 172.21.15.29:0/2496693077 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757702479 0 0) 0x7f456815d150 con 0x7f45680d0be0 2023-10-14T18:19:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.755+0000 7f45677fe640 1 --2- 172.21.15.29:0/2496693077 >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 0x7f45680cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.756+0000 7f4567fff640 1 --2- 172.21.15.29:0/2496693077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f456815cd70 0x7f45680d2220 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f4568066a00 tx=0x7f455802b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea12fff8e27e2b37 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.756+0000 7f4566ffd640 1 -- 172.21.15.29:0/2496693077 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558031890 con 0x7f456815cd70 2023-10-14T18:19:58.760 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.756+0000 7f4566ffd640 1 -- 172.21.15.29:0/2496693077 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45580319f0 con 0x7f456815cd70 2023-10-14T18:19:58.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.756+0000 7f4566ffd640 1 -- 172.21.15.29:0/2496693077 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4558031d80 con 0x7f456815cd70 2023-10-14T18:19:58.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 -- 172.21.15.29:0/2496693077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f456815cd70 msgr2=0x7f45680d2220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 --2- 172.21.15.29:0/2496693077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f456815cd70 0x7f45680d2220 secure :-1 s=READY pgs=3211 cs=0 l=1 rev1=1 crypto rx=0x7f4568066a00 tx=0x7f455802b930 comp rx=0 tx=0).stop 2023-10-14T18:19:58.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 -- 172.21.15.29:0/2496693077 shutdown_connections 2023-10-14T18:19:58.761 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 --2- 172.21.15.29:0/2496693077 >> v2:172.21.15.186:3300/0 conn(0x7f45680b8480 0x7f45680cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 --2- 172.21.15.29:0/2496693077 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f456815cd70 0x7f45680d2220 unknown :-1 s=CLOSED pgs=3211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 -- 172.21.15.29:0/2496693077 >> 172.21.15.29:0/2496693077 conn(0x7f45680cf570 msgr2=0x7f45680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:58.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 -- 172.21.15.29:0/2496693077 shutdown_connections 2023-10-14T18:19:58.762 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.757+0000 7f456d1ff640 1 -- 172.21.15.29:0/2496693077 wait complete. 2023-10-14T18:19:58.764 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.759+0000 7f456d1ff640 1 Processor -- start 2023-10-14T18:19:58.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.759+0000 7f456d1ff640 1 -- start start 2023-10-14T18:19:58.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45680b8480 0x7f4568156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.765 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f456815cd70 0x7f456815a710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45681565f0 con 0x7f45680d0be0 2023-10-14T18:19:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4568155f70 con 0x7f45680b8480 2023-10-14T18:19:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.760+0000 7f456d1ff640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45680cc760 con 0x7f456815cd70 2023-10-14T18:19:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.766 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52296/0 (socket says 172.21.15.29:52296) 2023-10-14T18:19:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 -- 172.21.15.29:0/1948111733 learned_addr learned my addr 172.21.15.29:0/1948111733 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f456c9fe640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f456815cd70 0x7f456815a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f4567fff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45680b8480 0x7f4568156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f456815cd70 msgr2=0x7f456815a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.767 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f456815cd70 0x7f456815a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45680b8480 msgr2=0x7f4568156a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45680b8480 0x7f4568156a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 -- 172.21.15.29:0/1948111733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45580094d0 con 0x7f45680d0be0 2023-10-14T18:19:58.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.761+0000 7f45677fe640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f455c00c7f0 tx=0x7f455c00ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:58.768 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.762+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c007820 con 0x7f45680d0be0 2023-10-14T18:19:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.762+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455c007980 con 0x7f45680d0be0 2023-10-14T18:19:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.762+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f455c007cf0 con 0x7f45680d0be0 2023-10-14T18:19:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.762+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4568156210 con 0x7f45680d0be0 2023-10-14T18:19:58.769 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.762+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4568158b60 con 0x7f45680d0be0 2023-10-14T18:19:58.770 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.763+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456804e9b0 con 0x7f45680d0be0 2023-10-14T18:19:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.764+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f455c0113f0 con 0x7f45680d0be0 2023-10-14T18:19:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.765+0000 7f4564ff9640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 0x7f455406d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.765+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f455c089ee0 con 0x7f45680d0be0 2023-10-14T18:19:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.765+0000 7f4567fff640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 msgr2=0x7f455406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:58.771 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.765+0000 7f4567fff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 0x7f455406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:58.772 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.769+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f455c054a70 con 0x7f45680d0be0 2023-10-14T18:19:58.913 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:58.912+0000 7ff125c11040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-14T18:19:58.914 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:58.914+0000 7fcc1e21a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:58.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.964+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f45680c8d20 con 0x7f45680d0be0 2023-10-14T18:19:58.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.965+0000 7f4564ff9640 1 -- 172.21.15.29:0/1948111733 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v272) v1 ==== 100+0+11 (secure 0 0 0) 0x7f455c01f0a0 con 0x7f45680d0be0 2023-10-14T18:19:58.966 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:19:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.966+0000 7f4567fff640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 msgr2=0x7f455406d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:58.967 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.966+0000 7f4567fff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 0x7f455406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 msgr2=0x7f455406d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:58.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 0x7f455406d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 msgr2=0x7f4568158620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 secure :-1 s=READY pgs=3212 cs=0 l=1 rev1=1 crypto rx=0x7f455c00c7f0 tx=0x7f455c00ccb0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 shutdown_connections 2023-10-14T18:19:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f456815cd70 0x7f456815a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f455406b020 0x7f455406d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f45680d0be0 0x7f4568158620 unknown :-1 s=CLOSED pgs=3212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 --2- 172.21.15.29:0/1948111733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f45680b8480 0x7f4568156a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 >> 172.21.15.29:0/1948111733 conn(0x7f45680cf570 msgr2=0x7f45680cd400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.967+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 shutdown_connections 2023-10-14T18:19:58.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:58.968+0000 7f456d1ff640 1 -- 172.21.15.29:0/1948111733 wait complete. 2023-10-14T18:19:58.986 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2023-10-14T18:19:59.021 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.021+0000 7fcc1e21a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:59.070 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.070+0000 7fcc1e21a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:59.086 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.085+0000 7ff125c11040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-14T18:19:59.178 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.178+0000 7fcc1e21a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.191+0000 7f9733ca2640 1 Processor -- start 2023-10-14T18:19:59.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.191+0000 7f9733ca2640 1 -- start start 2023-10-14T18:19:59.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9733ca2640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f972c0b3420 0x7f972c0b5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9733ca2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972c0d3980 con 0x7f972c0cf710 2023-10-14T18:19:59.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9733ca2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f972c0ce630 con 0x7f972c0cfaf0 2023-10-14T18:19:59.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9733ca2640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972c0d3c60 con 0x7f972c0b3420 2023-10-14T18:19:59.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9731a3e640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f972c0cf710 0x7f972c0b2d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39168/0 (socket says 172.21.15.29:39168) 2023-10-14T18:19:59.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.193+0000 7f9731a3e640 1 -- 172.21.15.29:0/1353484158 learned_addr learned my addr 172.21.15.29:0/1353484158 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:59.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f973123d640 1 --2- 172.21.15.29:0/1353484158 >> v2:172.21.15.186:3300/0 conn(0x7f972c0b3420 0x7f972c0b5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973905178 0 0) 0x7f972c0d3980 con 0x7f972c0cf710 2023-10-14T18:19:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9718003520 con 0x7f972c0cf710 2023-10-14T18:19:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4269977003 0 0) 0x7f9718003520 con 0x7f972c0cf710 2023-10-14T18:19:59.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 >> v2:172.21.15.186:3300/0 conn(0x7f972c0b3420 msgr2=0x7f972c0b5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 --2- 172.21.15.29:0/1353484158 >> v2:172.21.15.186:3300/0 conn(0x7f972c0b3420 0x7f972c0b5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 >> v1:172.21.15.186:6789/0 conn(0x7f972c0cfaf0 legacy=0x7f972c0b6200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0ce910 con 0x7f972c0cf710 2023-10-14T18:19:59.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f97280017f0 con 0x7f972c0cf710 2023-10-14T18:19:59.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 >> v1:172.21.15.29:6789/0 conn(0x7f972c0cf710 legacy=0x7f972c0b2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9718003970 0x7f9718003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f9718004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9718005180 0x7f971800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0d3c60 con 0x7f97180047c0 2023-10-14T18:19:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971800c120 con 0x7f9718003970 2023-10-14T18:19:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.194+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f971800c2e0 con 0x7f9718005180 2023-10-14T18:19:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f9718004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f9731a3e640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9718003970 0x7f9718003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973223f640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9718005180 0x7f971800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 -- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9718005180 msgr2=0x7f971800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9718005180 0x7f971800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 -- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9718003970 msgr2=0x7f9718003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9718003970 0x7f9718003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973123d640 1 -- 172.21.15.29:0/1353484158 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9718003520 con 0x7f97180047c0 2023-10-14T18:19:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f9731a3e640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f9718003970 0x7f9718003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:19:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973223f640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f9718005180 0x7f971800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.195+0000 7f973223f640 1 -- 172.21.15.29:0/1353484158 reap_dead start 2023-10-14T18:19:59.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.196+0000 7f973123d640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f9718004c40 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f971c009ac0 tx=0x7f97180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2df9bbc815834d7 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c02c6b0 con 0x7f97180047c0 2023-10-14T18:19:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c02c810 con 0x7f97180047c0 2023-10-14T18:19:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9730a3c640 1 -- 172.21.15.29:0/1353484158 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971c02cba0 con 0x7f97180047c0 2023-10-14T18:19:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 -- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 msgr2=0x7f9718004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f9718004c40 secure :-1 s=READY pgs=3214 cs=0 l=1 rev1=1 crypto rx=0x7f971c009ac0 tx=0x7f97180057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 -- 172.21.15.29:0/1353484158 shutdown_connections 2023-10-14T18:19:59.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1353484158 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f9718004c40 unknown :-1 s=CLOSED pgs=3214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 -- 172.21.15.29:0/1353484158 >> 172.21.15.29:0/1353484158 conn(0x7f972c0d2cc0 msgr2=0x7f972c0d30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 -- 172.21.15.29:0/1353484158 shutdown_connections 2023-10-14T18:19:59.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.197+0000 7f9733ca2640 1 -- 172.21.15.29:0/1353484158 wait complete. 2023-10-14T18:19:59.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.199+0000 7f9733ca2640 1 Processor -- start 2023-10-14T18:19:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.199+0000 7f9733ca2640 1 -- start start 2023-10-14T18:19:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f972c0b02a0 0x7f972c0cb750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 0x7f972c0b1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0b0720 con 0x7f97180047c0 2023-10-14T18:19:59.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0b1f40 con 0x7f972c0b1580 2023-10-14T18:19:59.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.200+0000 7f9733ca2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f972c0b0930 con 0x7f972c0b02a0 2023-10-14T18:19:59.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f973123d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f972c0b02a0 0x7f972c0cb750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52306/0 (socket says 172.21.15.29:52306) 2023-10-14T18:19:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 -- 172.21.15.29:0/1675579733 learned_addr learned my addr 172.21.15.29:0/1675579733 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f973223f640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 0x7f972c0b1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f972c0b02a0 msgr2=0x7f972c0cb750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f972c0b02a0 0x7f972c0cb750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 msgr2=0x7f972c0b1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 0x7f972c0b1a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f9731a3e640 1 -- 172.21.15.29:0/1675579733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972c0b0c10 con 0x7f97180047c0 2023-10-14T18:19:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.201+0000 7f973223f640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 0x7f972c0b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:19:59.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f9731a3e640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f972800e750 tx=0x7f972800ec10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f972800ee60 con 0x7f97180047c0 2023-10-14T18:19:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97280072e0 con 0x7f97180047c0 2023-10-14T18:19:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972c0b0ef0 con 0x7f97180047c0 2023-10-14T18:19:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97280057a0 con 0x7f97180047c0 2023-10-14T18:19:59.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.202+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972c0c6a60 con 0x7f97180047c0 2023-10-14T18:19:59.217 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.203+0000 7ff125c11040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-14T18:19:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.203+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972c0c6c20 con 0x7f97180047c0 2023-10-14T18:19:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.204+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f9728005900 con 0x7f97180047c0 2023-10-14T18:19:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.205+0000 7f97227fc640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 0x7f971006d510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.205+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f97280899a0 con 0x7f97180047c0 2023-10-14T18:19:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.205+0000 7f973123d640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 msgr2=0x7f971006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.205+0000 7f973123d640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 0x7f971006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:59.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.209+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f9728054640 con 0x7f97180047c0 2023-10-14T18:19:59.258 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.257+0000 7ff125c11040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-14T18:19:59.287 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.287+0000 7fcc1e21a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:59.373 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.372+0000 7ff125c11040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:59.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.406+0000 7f973123d640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 msgr2=0x7f971006d510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:59.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.406+0000 7f973123d640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 0x7f971006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:19:59.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.416+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f972c04e160 con 0x7f97180047c0 2023-10-14T18:19:59.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.417+0000 7f97227fc640 1 -- 172.21.15.29:0/1675579733 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v272) v1 ==== 81+0+586 (secure 0 0 0) 0x7f9728057c60 con 0x7f97180047c0 2023-10-14T18:19:59.419 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2196987879 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.419 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3551524051 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.419 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3292195983 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.419 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/4006438795 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.420 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6833/1327833207 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.420 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:6832/1327833207 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.420 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/3279571573 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.420 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1082110076 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.420 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/2709558548 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.421 INFO:teuthology.orchestra.run.smithi029.stdout:172.21.15.29:0/1343183116 2023-10-15T18:19:52.752456+0000 2023-10-14T18:19:59.421 INFO:teuthology.orchestra.run.smithi029.stderr:listed 10 entries 2023-10-14T18:19:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 msgr2=0x7f971006d510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:19:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 0x7f971006d510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 msgr2=0x7f972c1522e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.422 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 secure :-1 s=READY pgs=3215 cs=0 l=1 rev1=1 crypto rx=0x7f972800e750 tx=0x7f972800ec10 comp rx=0 tx=0).stop 2023-10-14T18:19:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 shutdown_connections 2023-10-14T18:19:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f972c0b1580 0x7f972c0b1a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f972c0b02a0 0x7f972c0cb750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f971006b050 0x7f971006d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.423 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 --2- 172.21.15.29:0/1675579733 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f97180047c0 0x7f972c1522e0 unknown :-1 s=CLOSED pgs=3215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 >> 172.21.15.29:0/1675579733 conn(0x7f972c0d2cc0 msgr2=0x7f972c06bc00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 shutdown_connections 2023-10-14T18:19:59.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.420+0000 7f9733ca2640 1 -- 172.21.15.29:0/1675579733 wait complete. 2023-10-14T18:19:59.439 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2023-10-14T18:19:59.440 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:59.440 DEBUG:teuthology.orchestra.run.smithi029:> sudo umount /home/ubuntu/cephtest/mnt.0 2023-10-14T18:19:59.442 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.442+0000 7fcc1e21a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:59.490 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.490+0000 7ff125c11040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-14T18:19:59.491 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.491+0000 7fcc1e21a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:59.534 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:19:59.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:19:59.535 DEBUG:teuthology.orchestra.run.smithi029:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2023-10-14T18:19:59.578 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:59.579 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:59.615 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:19:59.615+0000 7fcc1e21a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:59.635 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:59.635 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:19:59.636 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:59.662 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.661+0000 7ff125c11040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-14T18:19:59.692 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:19:59.692 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:19:59.693 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:19:59.714 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.714+0000 7ff125c11040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-14T18:19:59.748 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:19:59.748 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:19:59.843 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:19:59.841+0000 7ff125c11040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-14T18:19:59.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.969+0000 7f6f5aff4640 1 Processor -- start 2023-10-14T18:19:59.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.969+0000 7f6f5aff4640 1 -- start start 2023-10-14T18:19:59.971 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.970+0000 7f6f5aff4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 0x7f6f540d2220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f5aff4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f5415d150 con 0x7f6f540b8480 2023-10-14T18:19:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f5aff4640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f540d1180 con 0x7f6f540cded0 2023-10-14T18:19:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f5aff4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540d1460 con 0x7f6f5415cd70 2023-10-14T18:19:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f597f1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6f540b8480 0x7f6f540cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39172/0 (socket says 172.21.15.29:39172) 2023-10-14T18:19:59.972 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f597f1640 1 -- 172.21.15.29:0/35348964 learned_addr learned my addr 172.21.15.29:0/35348964 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:19:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 0x7f6f540d2220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4280526228 0 0) 0x7f6f5415d150 con 0x7f6f540b8480 2023-10-14T18:19:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.971+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f40003520 con 0x7f6f540b8480 2023-10-14T18:19:59.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3030928854 0 0) 0x7f6f40003520 con 0x7f6f540b8480 2023-10-14T18:19:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 msgr2=0x7f6f540d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 0x7f6f540d2220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 >> v1:172.21.15.186:6789/0 conn(0x7f6f540cded0 legacy=0x7f6f540b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540d1740 con 0x7f6f540b8480 2023-10-14T18:19:59.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6f44001670 con 0x7f6f540b8480 2023-10-14T18:19:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 >> v1:172.21.15.29:6789/0 conn(0x7f6f540b8480 legacy=0x7f6f540cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f400039a0 0x7f6f40003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400047c0 0x7f6f40004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 0x7f6f40005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540d1460 con 0x7f6f400039a0 2023-10-14T18:19:59.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4000ba30 con 0x7f6f400047c0 2023-10-14T18:19:59.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f4000bbf0 con 0x7f6f5415cd70 2023-10-14T18:19:59.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.972+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f400039a0 0x7f6f40003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.973+0000 7f6f59ff2640 1 -- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 msgr2=0x7f6f40005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.973+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f5415cd70 0x7f6f40005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.973+0000 7f6f59ff2640 1 -- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400047c0 msgr2=0x7f6f40004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:19:59.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.973+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400047c0 0x7f6f40004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.973+0000 7f6f59ff2640 1 -- 172.21.15.29:0/35348964 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f40003520 con 0x7f6f400039a0 2023-10-14T18:19:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f400039a0 0x7f6f40003dc0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009a90 tx=0x7f6f400057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c144da8e19c12dc8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5002c6b0 con 0x7f6f400039a0 2023-10-14T18:19:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5002c810 con 0x7f6f400039a0 2023-10-14T18:19:59.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f58ff0640 1 -- 172.21.15.29:0/35348964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f5002cba0 con 0x7f6f400039a0 2023-10-14T18:19:59.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f5aff4640 1 -- 172.21.15.29:0/35348964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f400039a0 msgr2=0x7f6f40003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.974+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/35348964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f400039a0 0x7f6f40003dc0 secure :-1 s=READY pgs=3217 cs=0 l=1 rev1=1 crypto rx=0x7f6f50009a90 tx=0x7f6f400057f0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.975+0000 7f6f5a7f3640 1 -- 172.21.15.29:0/35348964 reap_dead start 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.975+0000 7f6f5aff4640 1 -- 172.21.15.29:0/35348964 shutdown_connections 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.975+0000 7f6f5aff4640 1 -- 172.21.15.29:0/35348964 >> 172.21.15.29:0/35348964 conn(0x7f6f540cf570 msgr2=0x7f6f540cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.975+0000 7f6f5aff4640 1 -- 172.21.15.29:0/35348964 shutdown_connections 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.975+0000 7f6f5aff4640 1 -- 172.21.15.29:0/35348964 wait complete. 2023-10-14T18:19:59.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.976+0000 7f6f5aff4640 1 Processor -- start 2023-10-14T18:19:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.977+0000 7f6f5aff4640 1 -- start start 2023-10-14T18:19:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.977+0000 7f6f5aff4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400039a0 0x7f6f5415a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.977+0000 7f6f5aff4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f541f2170 0x7f6f540a94f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.977+0000 7f6f5aff4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5aff4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540ca340 con 0x7f6f540cc020 2023-10-14T18:19:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5aff4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540ab910 con 0x7f6f400039a0 2023-10-14T18:19:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5aff4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f540c9cf0 con 0x7f6f541f2170 2023-10-14T18:19:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52326/0 (socket says 172.21.15.29:52326) 2023-10-14T18:19:59.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 -- 172.21.15.29:0/1715514517 learned_addr learned my addr 172.21.15.29:0/1715514517 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:19:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f597f1640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f541f2170 0x7f6f540a94f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400039a0 0x7f6f5415a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:19:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f541f2170 msgr2=0x7f6f540a94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f541f2170 0x7f6f540a94f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400039a0 msgr2=0x7f6f5415a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400039a0 0x7f6f5415a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.978+0000 7f6f5a7f3640 1 -- 172.21.15.29:0/1715514517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f540c9fd0 con 0x7f6f540cc020 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f5a7f3640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f6f54066a80 tx=0x7f6f4c00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c005d40 con 0x7f6f540cc020 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f4c00ced0 con 0x7f6f540cc020 2023-10-14T18:19:59.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f540abbf0 con 0x7f6f540cc020 2023-10-14T18:19:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f4c005ea0 con 0x7f6f540cc020 2023-10-14T18:19:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.979+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f5415a7f0 con 0x7f6f540cc020 2023-10-14T18:19:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.980+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f5404e9b0 con 0x7f6f540cc020 2023-10-14T18:19:59.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.982+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f6f4c00a850 con 0x7f6f540cc020 2023-10-14T18:19:59.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.984+0000 7f6f4a7fc640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 0x7f6f3806d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:19:59.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.984+0000 7f6f59ff2640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 msgr2=0x7f6f3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:19:59.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.984+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f6f4c088940 con 0x7f6f540cc020 2023-10-14T18:19:59.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.984+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 0x7f6f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:19:59.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:19:59.988+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6f4c053740 con 0x7f6f540cc020 2023-10-14T18:20:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.185+0000 7f6f59ff2640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 msgr2=0x7f6f3806d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:20:00.186 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.185+0000 7f6f59ff2640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 0x7f6f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:20:00.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.221+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f540ca160 con 0x7f6f540cc020 2023-10-14T18:20:00.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.223+0000 7f6f4a7fc640 1 -- 172.21.15.29:0/1715514517 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f6f4c056d60 con 0x7f6f540cc020 2023-10-14T18:20:00.224 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:00.224 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:20:00.225 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:20:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 msgr2=0x7f6f3806d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:20:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 0x7f6f3806d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 msgr2=0x7f6f540a9a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 secure :-1 s=READY pgs=3218 cs=0 l=1 rev1=1 crypto rx=0x7f6f54066a80 tx=0x7f6f4c00cb90 comp rx=0 tx=0).stop 2023-10-14T18:20:00.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 shutdown_connections 2023-10-14T18:20:00.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f6f3806b020 0x7f6f3806d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.228 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f540cc020 0x7f6f540a9a30 unknown :-1 s=CLOSED pgs=3218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6f541f2170 0x7f6f540a94f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 --2- 172.21.15.29:0/1715514517 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f400039a0 0x7f6f5415a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.225+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 >> 172.21.15.29:0/1715514517 conn(0x7f6f540cf570 msgr2=0x7f6f540b7600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.226+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 shutdown_connections 2023-10-14T18:20:00.229 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.226+0000 7f6f5aff4640 1 -- 172.21.15.29:0/1715514517 wait complete. 2023-10-14T18:20:00.244 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:20:00.441 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.440+0000 7f44e68f2640 1 Processor -- start 2023-10-14T18:20:00.442 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.441+0000 7f44e68f2640 1 -- start start 2023-10-14T18:20:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44e68f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 0x7f44e006ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44e68f2640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e0070660 con 0x7f44e015d430 2023-10-14T18:20:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44e68f2640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e0071530 con 0x7f44e00cd880 2023-10-14T18:20:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44e68f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e015dc60 con 0x7f44e015a970 2023-10-14T18:20:00.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44df7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f44e015d430 0x7f44e0070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:39180/0 (socket says 172.21.15.29:39180) 2023-10-14T18:20:00.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44df7fe640 1 -- 172.21.15.29:0/2440107170 learned_addr learned my addr 172.21.15.29:0/2440107170 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:00.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44dffff640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 0x7f44e006ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.442+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919333260 0 0) 0x7f44e0070660 con 0x7f44e015d430 2023-10-14T18:20:00.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8003520 con 0x7f44e015d430 2023-10-14T18:20:00.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 344048624 0 0) 0x7f44c8003520 con 0x7f44e015d430 2023-10-14T18:20:00.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 msgr2=0x7f44e006ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 0x7f44e006ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 >> v1:172.21.15.186:6789/0 conn(0x7f44e00cd880 legacy=0x7f44e0069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e015df40 con 0x7f44e015d430 2023-10-14T18:20:00.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f44d4001670 con 0x7f44e015d430 2023-10-14T18:20:00.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 >> v1:172.21.15.29:6789/0 conn(0x7f44e015d430 legacy=0x7f44e0070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44c8003da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 0x7f44c80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e015dc60 con 0x7f44c80039a0 2023-10-14T18:20:00.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44c800bab0 con 0x7f44c8004d00 2023-10-14T18:20:00.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44c800bc70 con 0x7f44e015a970 2023-10-14T18:20:00.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.443+0000 7f44dffff640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44c8003da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44e4e8f640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44dffff640 1 -- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 msgr2=0x7f44c80047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44dffff640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e015a970 0x7f44c80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44dffff640 1 -- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8004d00 msgr2=0x7f44c8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44dffff640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44c8004d00 0x7f44c8005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.444+0000 7f44dffff640 1 -- 172.21.15.29:0/2440107170 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44c8003520 con 0x7f44c80039a0 2023-10-14T18:20:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.445+0000 7f44dffff640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44c8003da0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f44d000ba30 tx=0x7f44c80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dda56375b3b795f4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:00.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.445+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0033650 con 0x7f44c80039a0 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.445+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d00337b0 con 0x7f44c80039a0 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.445+0000 7f44deffd640 1 -- 172.21.15.29:0/2440107170 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0033b40 con 0x7f44c80039a0 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 -- 172.21.15.29:0/2440107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 msgr2=0x7f44c8003da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 --2- 172.21.15.29:0/2440107170 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44c8003da0 secure :-1 s=READY pgs=3220 cs=0 l=1 rev1=1 crypto rx=0x7f44d000ba30 tx=0x7f44c80057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e4e8f640 1 -- 172.21.15.29:0/2440107170 reap_dead start 2023-10-14T18:20:00.453 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 -- 172.21.15.29:0/2440107170 shutdown_connections 2023-10-14T18:20:00.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 -- 172.21.15.29:0/2440107170 >> 172.21.15.29:0/2440107170 conn(0x7f44e00cfaa0 msgr2=0x7f44e00cfeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:00.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 -- 172.21.15.29:0/2440107170 shutdown_connections 2023-10-14T18:20:00.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.446+0000 7f44e68f2640 1 -- 172.21.15.29:0/2440107170 wait complete. 2023-10-14T18:20:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.448+0000 7f44e68f2640 1 Processor -- start 2023-10-14T18:20:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.448+0000 7f44e68f2640 1 -- start start 2023-10-14T18:20:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e0158580 0x7f44e00b4320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 0x7f44e00aff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e01595e0 con 0x7f44c80039a0 2023-10-14T18:20:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e00c78b0 con 0x7f44e0158580 2023-10-14T18:20:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e68f2640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e00c7b90 con 0x7f44e00afac0 2023-10-14T18:20:00.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52346/0 (socket says 172.21.15.29:52346) 2023-10-14T18:20:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 -- 172.21.15.29:0/3961310156 learned_addr learned my addr 172.21.15.29:0/3961310156 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44df7fe640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e0158580 0x7f44e00b4320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44e4e8f640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 0x7f44e00aff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 msgr2=0x7f44e00aff40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 0x7f44e00aff40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e0158580 msgr2=0x7f44e00b4320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.449+0000 7f44dffff640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e0158580 0x7f44e00b4320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44dffff640 1 -- 172.21.15.29:0/3961310156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00c7e70 con 0x7f44c80039a0 2023-10-14T18:20:00.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44e4e8f640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 0x7f44e00aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44dffff640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f44d0035040 tx=0x7f44d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0004220 con 0x7f44c80039a0 2023-10-14T18:20:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d0040070 con 0x7f44c80039a0 2023-10-14T18:20:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44d0003610 con 0x7f44c80039a0 2023-10-14T18:20:00.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e00c8150 con 0x7f44c80039a0 2023-10-14T18:20:00.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.450+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e00c7580 con 0x7f44c80039a0 2023-10-14T18:20:00.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.452+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e004e160 con 0x7f44c80039a0 2023-10-14T18:20:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.452+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f44d002f030 con 0x7f44c80039a0 2023-10-14T18:20:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.453+0000 7f44dcff9640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 0x7f44cc06d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.453+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f44d00ab5e0 con 0x7f44c80039a0 2023-10-14T18:20:00.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.453+0000 7f44df7fe640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 msgr2=0x7f44cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:20:00.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.453+0000 7f44df7fe640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 0x7f44cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:20:00.465 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:20:00.464+0000 7fcc1e21a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:20:00.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.458+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44d00761e0 con 0x7f44c80039a0 2023-10-14T18:20:00.654 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.653+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f44e00b5a20 con 0x7f44c80039a0 2023-10-14T18:20:00.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.654+0000 7f44df7fe640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 msgr2=0x7f44cc06d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:20:00.655 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.654+0000 7f44df7fe640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 0x7f44cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:20:00.656 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.655+0000 7f44dcff9640 1 -- 172.21.15.29:0/3961310156 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v272) v1 ==== 74+0+17326 (secure 0 0 0) 0x7f44d002d040 con 0x7f44c80039a0 2023-10-14T18:20:00.657 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:00.657 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":272,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:19:52.752557+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2196987879":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3551524051":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3292195983":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/4006438795":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6832/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3279571573":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1082110076":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2709558548":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1343183116":"2023-10-15T18:19:52.752456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:20:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 msgr2=0x7f44cc06d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:20:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 0x7f44cc06d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 msgr2=0x7f44e00c8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 secure :-1 s=READY pgs=3221 cs=0 l=1 rev1=1 crypto rx=0x7f44d0035040 tx=0x7f44d0004730 comp rx=0 tx=0).stop 2023-10-14T18:20:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 shutdown_connections 2023-10-14T18:20:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44e00afac0 0x7f44e00aff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f44e0158580 0x7f44e00b4320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.660 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f44cc06b020 0x7f44cc06d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 --2- 172.21.15.29:0/3961310156 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f44c80039a0 0x7f44e00c8950 unknown :-1 s=CLOSED pgs=3221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 >> 172.21.15.29:0/3961310156 conn(0x7f44e00cfaa0 msgr2=0x7f44e015c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 shutdown_connections 2023-10-14T18:20:00.661 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.658+0000 7f44e68f2640 1 -- 172.21.15.29:0/3961310156 wait complete. 2023-10-14T18:20:00.680 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-14T16:40:13.373867+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 34, 'pool_name': 'cephfs_metadata', 'create_time': '2023-10-14T18:19:35.816866+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '270', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 35, 'pool_name': 'cephfs_data', 'create_time': '2023-10-14T18:19:36.425584+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '271', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-14T18:20:00.681 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2023-10-14T18:20:00.711 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:00.711+0000 7ff125c11040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-14T18:20:00.759 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:20:00.759+0000 7fcc1e21a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:20:00.815 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:20:00.815+0000 7fcc1e21a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:20:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.897+0000 7f74b3621640 1 Processor -- start 2023-10-14T18:20:00.898 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.897+0000 7f74b3621640 1 -- start start 2023-10-14T18:20:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b3621640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74ac06a4b0 0x7f74ac0ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b3621640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b3621640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74ac0cb030 con 0x7f74ac15ad70 2023-10-14T18:20:00.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b3621640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0cae30 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b3621640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74ac0c91c0 con 0x7f74ac06a4b0 2023-10-14T18:20:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b1bbe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b1bbe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52362/0 (socket says 172.21.15.29:52362) 2023-10-14T18:20:00.901 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b1bbe640 1 -- 172.21.15.29:0/2634316964 learned_addr learned my addr 172.21.15.29:0/2634316964 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:00.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.899+0000 7f74b0bbc640 1 --2- 172.21.15.29:0/2634316964 >> v2:172.21.15.186:3300/0 conn(0x7f74ac06a4b0 0x7f74ac0ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.900+0000 7f74b1bbe640 1 -- 172.21.15.29:0/2634316964 >> v2:172.21.15.186:3300/0 conn(0x7f74ac06a4b0 msgr2=0x7f74ac0ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.900+0000 7f74b1bbe640 1 --2- 172.21.15.29:0/2634316964 >> v2:172.21.15.186:3300/0 conn(0x7f74ac06a4b0 0x7f74ac0ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.900+0000 7f74b1bbe640 1 -- 172.21.15.29:0/2634316964 >> v1:172.21.15.186:6789/0 conn(0x7f74ac15ad70 legacy=0x7f74ac069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:00.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.900+0000 7f74b1bbe640 1 -- 172.21.15.29:0/2634316964 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74ac0c94a0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.901+0000 7f74b1bbe640 1 --2- 172.21.15.29:0/2634316964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0ce870 tx=0x7f74a402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5b501fe265a54ff server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.901+0000 7f74a3fff640 1 -- 172.21.15.29:0/2634316964 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4031870 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.901+0000 7f74a3fff640 1 -- 172.21.15.29:0/2634316964 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a40319d0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.901+0000 7f74a3fff640 1 -- 172.21.15.29:0/2634316964 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74a4031d60 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 -- 172.21.15.29:0/2634316964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 msgr2=0x7f74ac0ca8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 --2- 172.21.15.29:0/2634316964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 secure :-1 s=READY pgs=3222 cs=0 l=1 rev1=1 crypto rx=0x7f74ac0ce870 tx=0x7f74a402b910 comp rx=0 tx=0).stop 2023-10-14T18:20:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 -- 172.21.15.29:0/2634316964 shutdown_connections 2023-10-14T18:20:00.906 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 --2- 172.21.15.29:0/2634316964 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac0ca8f0 unknown :-1 s=CLOSED pgs=3222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 --2- 172.21.15.29:0/2634316964 >> v2:172.21.15.186:3300/0 conn(0x7f74ac06a4b0 0x7f74ac0ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 -- 172.21.15.29:0/2634316964 >> 172.21.15.29:0/2634316964 conn(0x7f74ac0ce9b0 msgr2=0x7f74ac0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 -- 172.21.15.29:0/2634316964 shutdown_connections 2023-10-14T18:20:00.907 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.902+0000 7f74b3621640 1 -- 172.21.15.29:0/2634316964 wait complete. 2023-10-14T18:20:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.904+0000 7f74b3621640 1 Processor -- start 2023-10-14T18:20:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.904+0000 7f74b3621640 1 -- start start 2023-10-14T18:20:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74ac06a4b0 0x7f74ac151e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.910 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ac15ad70 0x7f74ac0cccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac15de70 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac15cee0 con 0x7f74ac06a4b0 2023-10-14T18:20:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b3621640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74ac0cd230 con 0x7f74ac15ad70 2023-10-14T18:20:00.911 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b0bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b0bbc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:52366/0 (socket says 172.21.15.29:52366) 2023-10-14T18:20:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.905+0000 7f74b0bbc640 1 -- 172.21.15.29:0/2978096564 learned_addr learned my addr 172.21.15.29:0/2978096564 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b13bd640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74ac06a4b0 0x7f74ac151e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ac15ad70 msgr2=0x7f74ac0cccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.912 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ac15ad70 0x7f74ac0cccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74ac06a4b0 msgr2=0x7f74ac151e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74ac06a4b0 0x7f74ac151e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 -- 172.21.15.29:0/2978096564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a40094d0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.913 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74b0bbc640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7f749c00d800 tx=0x7f749c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.906+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00b660 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.907+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f749c00b7c0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.907+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f749c00bb50 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.907+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74ac0adb80 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.907+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74ac15d130 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.908+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74ac04e160 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.909+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 160) v1 ==== 89388+0+0 (secure 0 0 0) 0x7f749c00bcb0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.910+0000 7f74a1ffb640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 0x7f749006d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.910+0000 7f74b13bd640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 msgr2=0x7f749006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:20:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.910+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(272..272 src has 1..272) v4 ==== 6255+0+0 (secure 0 0 0) 0x7f749c08a080 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.918 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.910+0000 7f74b13bd640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 0x7f749006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-10-14T18:20:00.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:00.915+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f749c0bcfe0 con 0x7f74ac0c9ca0 2023-10-14T18:20:00.931 INFO:tasks.ceph.mgr.x.smithi186.stderr:2023-10-14T18:20:00.931+0000 7fcc1e21a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:20:01.046 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.045+0000 7ff125c11040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-14T18:20:01.107 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.106+0000 7ff125c11040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-14T18:20:01.111 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.111+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f74ac15a470 con 0x7f74ac0c9ca0 2023-10-14T18:20:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.111+0000 7f74b13bd640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 msgr2=0x7f749006d4e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.29:6832/4074993897 2023-10-14T18:20:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.111+0000 7f74b13bd640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 0x7f749006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-10-14T18:20:01.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.111+0000 7f74a1ffb640 1 -- 172.21.15.29:0/2978096564 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v272) v1 ==== 93+0+10 (secure 0 0 0) 0x7f749c054c80 con 0x7f74ac0c9ca0 2023-10-14T18:20:01.113 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 1 2023-10-14T18:20:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 msgr2=0x7f749006d4e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-10-14T18:20:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 0x7f749006d4e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.115 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 msgr2=0x7f74ac15d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 secure :-1 s=READY pgs=3223 cs=0 l=1 rev1=1 crypto rx=0x7f749c00d800 tx=0x7f749c00dcc0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 shutdown_connections 2023-10-14T18:20:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74ac15ad70 0x7f74ac0cccf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:6832/4074993897,v1:172.21.15.29:6833/4074993897] conn(0x7f749006b020 0x7f749006d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.116 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74ac0c9ca0 0x7f74ac15d930 unknown :-1 s=CLOSED pgs=3223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 --2- 172.21.15.29:0/2978096564 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74ac06a4b0 0x7f74ac151e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 >> 172.21.15.29:0/2978096564 conn(0x7f74ac0ce9b0 msgr2=0x7f74ac06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 shutdown_connections 2023-10-14T18:20:01.117 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.114+0000 7f74b3621640 1 -- 172.21.15.29:0/2978096564 wait complete. 2023-10-14T18:20:01.133 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2023-10-14T18:20:01.228 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.227+0000 7ff125c11040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-14T18:20:01.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.322+0000 7ff0ab69f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.322+0000 7ff0ab69f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.324 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.322+0000 7ff0ab69f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.322+0000 7ff0ab69f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.322+0000 7ff0ab69f640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.323+0000 7ff0a7e98640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.323+0000 7ff0a7e98640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.325 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.323+0000 7ff0a7e98640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.323+0000 7ff0a7e98640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.326 INFO:tasks.ceph.mgr.y.smithi029.stderr:2023-10-14T18:20:01.323+0000 7ff0a7e98640 -1 client.0 error registering admin socket command: (17) File exists 2023-10-14T18:20:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.388+0000 7f83dee75640 1 Processor -- start 2023-10-14T18:20:01.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.388+0000 7f83dee75640 1 -- start start 2023-10-14T18:20:01.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.389+0000 7f83dee75640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83d8159ab0 0x7f83d80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.389+0000 7f83dee75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d806fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.391 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.389+0000 7f83dee75640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d80cd1f0 con 0x7f83d806c140 2023-10-14T18:20:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dee75640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d815d890 con 0x7f83d8159ab0 2023-10-14T18:20:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dee75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d80cd4d0 con 0x7f83d8159e90 2023-10-14T18:20:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83d8159ab0 0x7f83d80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f83d8159ab0 0x7f83d80ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34554/0 (socket says 172.21.15.29:34554) 2023-10-14T18:20:01.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 -- 172.21.15.29:0/2480198189 learned_addr learned my addr 172.21.15.29:0/2480198189 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dd412640 1 --1- 172.21.15.29:0/2480198189 >> v1:172.21.15.29:6789/0 conn(0x7f83d806c140 0x7f83d815dce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44810/0 (socket says 172.21.15.29:44810) 2023-10-14T18:20:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83d7fff640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d806fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 -- 172.21.15.29:0/2480198189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 msgr2=0x7f83d806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:01.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d806fc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 -- 172.21.15.29:0/2480198189 >> v1:172.21.15.29:6789/0 conn(0x7f83d806c140 legacy=0x7f83d815dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83dcc11640 1 -- 172.21.15.29:0/2480198189 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d80cd7b0 con 0x7f83d8159ab0 2023-10-14T18:20:01.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.390+0000 7f83d77fe640 1 -- 172.21.15.29:0/2480198189 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1461940538 0 0) 0x7f83d80cd1f0 con 0x7f83d806c140 2023-10-14T18:20:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.391+0000 7f83dcc11640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d8159ab0 0x7f83d80ce370 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0098a0 tx=0x7f83cc02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23dc6cfd297da037 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83d77fe640 1 -- 172.21.15.29:0/2480198189 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc0317c0 con 0x7f83d8159ab0 2023-10-14T18:20:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83d77fe640 1 -- 172.21.15.29:0/2480198189 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc031920 con 0x7f83d8159ab0 2023-10-14T18:20:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83d77fe640 1 -- 172.21.15.29:0/2480198189 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc031cb0 con 0x7f83d8159ab0 2023-10-14T18:20:01.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 -- 172.21.15.29:0/2480198189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d8159ab0 msgr2=0x7f83d80ce370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d8159ab0 0x7f83d80ce370 secure :-1 s=READY pgs=3236 cs=0 l=1 rev1=1 crypto rx=0x7f83cc0098a0 tx=0x7f83cc02b860 comp rx=0 tx=0).stop 2023-10-14T18:20:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 -- 172.21.15.29:0/2480198189 shutdown_connections 2023-10-14T18:20:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d806fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 --2- 172.21.15.29:0/2480198189 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d8159ab0 0x7f83d80ce370 unknown :-1 s=CLOSED pgs=3236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 -- 172.21.15.29:0/2480198189 >> 172.21.15.29:0/2480198189 conn(0x7f83d80ce9b0 msgr2=0x7f83d80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 -- 172.21.15.29:0/2480198189 shutdown_connections 2023-10-14T18:20:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.392+0000 7f83dee75640 1 -- 172.21.15.29:0/2480198189 wait complete. 2023-10-14T18:20:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.394+0000 7f83dee75640 1 Processor -- start 2023-10-14T18:20:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.394+0000 7f83dee75640 1 -- start start 2023-10-14T18:20:01.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8159ab0 0x7f83d8157160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d815a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d81576a0 con 0x7f83d806c140 2023-10-14T18:20:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83d80b36e0 con 0x7f83d8159ab0 2023-10-14T18:20:01.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dee75640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83d80b3400 con 0x7f83d8159e90 2023-10-14T18:20:01.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dcc11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dcc11640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34562/0 (socket says 172.21.15.29:34562) 2023-10-14T18:20:01.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.395+0000 7f83dcc11640 1 -- 172.21.15.29:0/1325422097 learned_addr learned my addr 172.21.15.29:0/1325422097 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:01.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 -- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 msgr2=0x7f83d815a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d815a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 -- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8159ab0 msgr2=0x7f83d8157160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83d7fff640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8159ab0 0x7f83d8157160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8159ab0 0x7f83d8157160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 -- 172.21.15.29:0/1325422097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc0094d0 con 0x7f83d806c140 2023-10-14T18:20:01.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83dcc11640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:01.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.396+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc032960 con 0x7f83d806c140 2023-10-14T18:20:01.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.397+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83cc032ae0 con 0x7f83d806c140 2023-10-14T18:20:01.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.397+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d80c3a50 con 0x7f83d806c140 2023-10-14T18:20:01.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.397+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f83cc032e90 con 0x7f83d806c140 2023-10-14T18:20:01.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.397+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d80c4010 con 0x7f83d806c140 2023-10-14T18:20:01.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.398+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d804e160 con 0x7f83d806c140 2023-10-14T18:20:01.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.399+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 161) v1 ==== 88940+0+0 (secure 0 0 0) 0x7f83cc04d3f0 con 0x7f83d806c140 2023-10-14T18:20:01.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.400+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f83cc0ab4f0 con 0x7f83d806c140 2023-10-14T18:20:01.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.404+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f83cc076550 con 0x7f83d806c140 2023-10-14T18:20:01.592 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.591+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f83d80ccb60 con 0x7f83d806c140 2023-10-14T18:20:01.593 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.592+0000 7f83d57fa640 1 -- 172.21.15.29:0/1325422097 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v273) v1 ==== 104+0+11 (secure 0 0 0) 0x7f83cc079b70 con 0x7f83d806c140 2023-10-14T18:20:01.593 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 32 2023-10-14T18:20:01.595 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.594+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 msgr2=0x7f83d80b3b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 secure :-1 s=READY pgs=3237 cs=0 l=1 rev1=1 crypto rx=0x7f83cc000c00 tx=0x7f83cc0046c0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 shutdown_connections 2023-10-14T18:20:01.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f83d8159e90 0x7f83d815a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.596 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f83d8159ab0 0x7f83d8157160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 --2- 172.21.15.29:0/1325422097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f83d806c140 0x7f83d80b3b10 unknown :-1 s=CLOSED pgs=3237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 >> 172.21.15.29:0/1325422097 conn(0x7f83d80ce9b0 msgr2=0x7f83d80b0510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:01.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 shutdown_connections 2023-10-14T18:20:01.597 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.595+0000 7f83dee75640 1 -- 172.21.15.29:0/1325422097 wait complete. 2023-10-14T18:20:01.614 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2023-10-14T18:20:01.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.826+0000 7fb9764b4640 1 Processor -- start 2023-10-14T18:20:01.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.827+0000 7fb9764b4640 1 -- start start 2023-10-14T18:20:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.828+0000 7fb9764b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700cda70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.830 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.828+0000 7fb9764b4640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9700ca780 0x7fb97006aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.828+0000 7fb9764b4640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9700c3610 con 0x7fb97006b3f0 2023-10-14T18:20:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.828+0000 7fb9764b4640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb97006b880 con 0x7fb9700ca780 2023-10-14T18:20:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.828+0000 7fb9764b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb97015a220 con 0x7fb970159e40 2023-10-14T18:20:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9700ca780 0x7fb97006aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.831 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb974a51640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb97006b3f0 0x7fb9700705c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44814/0 (socket says 172.21.15.29:44814) 2023-10-14T18:20:01.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb9700ca780 0x7fb97006aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34576/0 (socket says 172.21.15.29:34576) 2023-10-14T18:20:01.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb974a51640 1 -- 172.21.15.29:0/3041785806 learned_addr learned my addr 172.21.15.29:0/3041785806 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:01.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb96ffff640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700cda70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb96f7fe640 1 -- 172.21.15.29:0/3041785806 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 629728161 0 0) 0x7fb9700c3610 con 0x7fb97006b3f0 2023-10-14T18:20:01.832 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb96f7fe640 1 -- 172.21.15.29:0/3041785806 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb95c003520 con 0x7fb97006b3f0 2023-10-14T18:20:01.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 -- 172.21.15.29:0/3041785806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 msgr2=0x7fb9700cda70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:01.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700cda70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.833 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 -- 172.21.15.29:0/3041785806 >> v1:172.21.15.29:6789/0 conn(0x7fb97006b3f0 legacy=0x7fb9700705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb967fff640 1 -- 172.21.15.29:0/3041785806 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9700cafd0 con 0x7fb9700ca780 2023-10-14T18:20:01.834 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.829+0000 7fb96ffff640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.830+0000 7fb967fff640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9700ca780 0x7fb97006aeb0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7fb9580097a0 tx=0x7fb95802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22540ee36feb1d20 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.831+0000 7fb96f7fe640 1 -- 172.21.15.29:0/3041785806 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb95802d040 con 0x7fb9700ca780 2023-10-14T18:20:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.831+0000 7fb96f7fe640 1 -- 172.21.15.29:0/3041785806 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb958031810 con 0x7fb9700ca780 2023-10-14T18:20:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.831+0000 7fb96f7fe640 1 -- 172.21.15.29:0/3041785806 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb958031ba0 con 0x7fb9700ca780 2023-10-14T18:20:01.837 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.831+0000 7fb9764b4640 1 -- 172.21.15.29:0/3041785806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9700ca780 msgr2=0x7fb97006aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.831+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9700ca780 0x7fb97006aeb0 secure :-1 s=READY pgs=3239 cs=0 l=1 rev1=1 crypto rx=0x7fb9580097a0 tx=0x7fb95802b8b0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 -- 172.21.15.29:0/3041785806 shutdown_connections 2023-10-14T18:20:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb9700ca780 0x7fb97006aeb0 unknown :-1 s=CLOSED pgs=3239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.838 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3041785806 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700cda70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 -- 172.21.15.29:0/3041785806 >> 172.21.15.29:0/3041785806 conn(0x7fb9700cfac0 msgr2=0x7fb9700cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 -- 172.21.15.29:0/3041785806 shutdown_connections 2023-10-14T18:20:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.832+0000 7fb9764b4640 1 -- 172.21.15.29:0/3041785806 wait complete. 2023-10-14T18:20:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.833+0000 7fb9764b4640 1 Processor -- start 2023-10-14T18:20:01.839 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.834+0000 7fb9764b4640 1 -- start start 2023-10-14T18:20:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.834+0000 7fb9764b4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb9764b4640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9700ca780 0x7fb9700cd850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb9764b4640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700c4640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb9764b4640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9701541b0 con 0x7fb97006b3f0 2023-10-14T18:20:01.840 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb9764b4640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb970154490 con 0x7fb9700ca780 2023-10-14T18:20:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb9764b4640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb970154770 con 0x7fb970159e40 2023-10-14T18:20:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34584/0 (socket says 172.21.15.29:34584) 2023-10-14T18:20:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 -- 172.21.15.29:0/3177627676 learned_addr learned my addr 172.21.15.29:0/3177627676 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:01.841 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb974a51640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700c4640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 -- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 msgr2=0x7fb9700c4640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700c4640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 -- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9700ca780 msgr2=0x7fb9700cd850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb967fff640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9700ca780 0x7fb9700cd850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:01.842 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9700ca780 0x7fb9700cd850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.835+0000 7fb96ffff640 1 -- 172.21.15.29:0/3177627676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb958009400 con 0x7fb97006b3f0 2023-10-14T18:20:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb96ffff640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7fb96000c7f0 tx=0x7fb96000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb960007a30 con 0x7fb97006b3f0 2023-10-14T18:20:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb960007b90 con 0x7fb97006b3f0 2023-10-14T18:20:01.843 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb96000ce40 con 0x7fb97006b3f0 2023-10-14T18:20:01.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb97015ab60 con 0x7fb97006b3f0 2023-10-14T18:20:01.844 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.836+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb970153d20 con 0x7fb97006b3f0 2023-10-14T18:20:01.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.837+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb970153f70 con 0x7fb97006b3f0 2023-10-14T18:20:01.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.838+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 161) v1 ==== 88940+0+0 (secure 0 0 0) 0x7fb9600113f0 con 0x7fb97006b3f0 2023-10-14T18:20:01.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.839+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fb960089750 con 0x7fb97006b3f0 2023-10-14T18:20:01.845 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:01.843+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb970153f70 con 0x7fb97006b3f0 2023-10-14T18:20:02.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.042+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb97015b080 con 0x7fb97006b3f0 2023-10-14T18:20:02.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.044+0000 7fb96d7fa640 1 -- 172.21.15.29:0/3177627676 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v273) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb97015b080 con 0x7fb97006b3f0 2023-10-14T18:20:02.045 INFO:teuthology.orchestra.run.smithi029.stdout:pg_num: 64 2023-10-14T18:20:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.046+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 msgr2=0x7fb97014f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.046+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 secure :-1 s=READY pgs=3240 cs=0 l=1 rev1=1 crypto rx=0x7fb96000c7f0 tx=0x7fb96000cbf0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 shutdown_connections 2023-10-14T18:20:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb970159e40 0x7fb9700c4640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb9700ca780 0x7fb9700cd850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 --2- 172.21.15.29:0/3177627676 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb97006b3f0 0x7fb97014f380 unknown :-1 s=CLOSED pgs=3240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 >> 172.21.15.29:0/3177627676 conn(0x7fb9700cfac0 msgr2=0x7fb97006a8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 shutdown_connections 2023-10-14T18:20:02.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.047+0000 7fb9764b4640 1 -- 172.21.15.29:0/3177627676 wait complete. 2023-10-14T18:20:02.067 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:20:02.306 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.305+0000 7f759f644640 1 Processor -- start 2023-10-14T18:20:02.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.305+0000 7f759f644640 1 -- start start 2023-10-14T18:20:02.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.306+0000 7f759f644640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 0x7f75980b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759f644640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759806f7f0 con 0x7f759815d880 2023-10-14T18:20:02.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759f644640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759806c5e0 con 0x7f7598159b20 2023-10-14T18:20:02.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759f644640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f759806c8c0 con 0x7f7598159f00 2023-10-14T18:20:02.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759dbe1640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f759815d880 0x7f759806fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44826/0 (socket says 172.21.15.29:44826) 2023-10-14T18:20:02.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759dbe1640 1 -- 172.21.15.29:0/2351486545 learned_addr learned my addr 172.21.15.29:0/2351486545 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:02.332 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f759cbdf640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 0x7f75980b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116802479 0 0) 0x7f759806f7f0 con 0x7f759815d880 2023-10-14T18:20:02.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.307+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7584003520 con 0x7f759815d880 2023-10-14T18:20:02.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 642195948 0 0) 0x7f7584003520 con 0x7f759815d880 2023-10-14T18:20:02.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 msgr2=0x7f75980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:02.333 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 0x7f75980b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 >> v1:172.21.15.186:6789/0 conn(0x7f7598159b20 legacy=0x7f759806bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759806cba0 con 0x7f759815d880 2023-10-14T18:20:02.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f75900019f0 con 0x7f759815d880 2023-10-14T18:20:02.334 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 >> v1:172.21.15.29:6789/0 conn(0x7f759815d880 legacy=0x7f759806fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75840039a0 0x7f7584003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 0x7f7584005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.335 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f759806c8c0 con 0x7f75840039a0 2023-10-14T18:20:02.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f758400ba30 con 0x7f75840047c0 2023-10-14T18:20:02.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f758400bbf0 con 0x7f7598159f00 2023-10-14T18:20:02.336 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.308+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75840039a0 0x7f7584003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759cbdf640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759d3e0640 1 -- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 msgr2=0x7f7584005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7598159f00 0x7f7584005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759d3e0640 1 -- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840047c0 msgr2=0x7f7584004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:02.337 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759d3e0640 1 -- 172.21.15.29:0/2351486545 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7584003520 con 0x7f75840039a0 2023-10-14T18:20:02.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.309+0000 7f759cbdf640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840047c0 0x7f7584004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:02.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.310+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75840039a0 0x7f7584003dc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f7594001720 tx=0x7f75840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe41aa5a992ba63f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.338 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.310+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940052d0 con 0x7f75840039a0 2023-10-14T18:20:02.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.310+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759402e050 con 0x7f75840039a0 2023-10-14T18:20:02.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.310+0000 7f758ffff640 1 -- 172.21.15.29:0/2351486545 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75940035f0 con 0x7f75840039a0 2023-10-14T18:20:02.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 -- 172.21.15.29:0/2351486545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75840039a0 msgr2=0x7f7584003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 --2- 172.21.15.29:0/2351486545 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f75840039a0 0x7f7584003dc0 secure :-1 s=READY pgs=3242 cs=0 l=1 rev1=1 crypto rx=0x7f7594001720 tx=0x7f75840057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.339 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759dbe1640 1 -- 172.21.15.29:0/2351486545 reap_dead start 2023-10-14T18:20:02.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 -- 172.21.15.29:0/2351486545 shutdown_connections 2023-10-14T18:20:02.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 -- 172.21.15.29:0/2351486545 >> 172.21.15.29:0/2351486545 conn(0x7f75980cb410 msgr2=0x7f75980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:02.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 -- 172.21.15.29:0/2351486545 shutdown_connections 2023-10-14T18:20:02.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.311+0000 7f759f644640 1 -- 172.21.15.29:0/2351486545 wait complete. 2023-10-14T18:20:02.340 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.313+0000 7f759f644640 1 Processor -- start 2023-10-14T18:20:02.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.313+0000 7f759f644640 1 -- start start 2023-10-14T18:20:02.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840039a0 0x7f75980b12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75980c4900 0x7f75980b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.341 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980c4d80 con 0x7f7598146050 2023-10-14T18:20:02.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75980b3d50 con 0x7f75840039a0 2023-10-14T18:20:02.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759f644640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75980ae3f0 con 0x7f75980c4900 2023-10-14T18:20:02.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759dbe1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.342 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759dbe1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34594/0 (socket says 172.21.15.29:34594) 2023-10-14T18:20:02.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759dbe1640 1 -- 172.21.15.29:0/2063301980 learned_addr learned my addr 172.21.15.29:0/2063301980 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:02.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840039a0 0x7f75980b12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.314+0000 7f759cbdf640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75980c4900 0x7f75980b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 -- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75980c4900 msgr2=0x7f75980b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.343 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75980c4900 0x7f75980b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 -- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840039a0 msgr2=0x7f75980b12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840039a0 0x7f75980b12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.344 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 -- 172.21.15.29:0/2063301980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75980aec90 con 0x7f7598146050 2023-10-14T18:20:02.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759dbe1640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f75980c8bb0 tx=0x7f759000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000a610 con 0x7f7598146050 2023-10-14T18:20:02.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f759000a770 con 0x7f7598146050 2023-10-14T18:20:02.345 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f759000ab20 con 0x7f7598146050 2023-10-14T18:20:02.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75980b3f80 con 0x7f7598146050 2023-10-14T18:20:02.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.315+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75980ae920 con 0x7f7598146050 2023-10-14T18:20:02.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.316+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f759804e160 con 0x7f7598146050 2023-10-14T18:20:02.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.317+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f759000ac80 con 0x7f7598146050 2023-10-14T18:20:02.346 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.319+0000 7f758dffb640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 0x7f757c06d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.319+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f7590088a70 con 0x7f7598146050 2023-10-14T18:20:02.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.319+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 0x7f757c06d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.320+0000 7f759d3e0640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 0x7f757c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f7594009030 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.347 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.324+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f75900539e0 con 0x7f7598146050 2023-10-14T18:20:02.559 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.558+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75980b10b0 con 0x7f7598146050 2023-10-14T18:20:02.560 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.560+0000 7f758dffb640 1 -- 172.21.15.29:0/2063301980 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f7590057000 con 0x7f7598146050 2023-10-14T18:20:02.561 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:02.561 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:20:02.562 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:20:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 msgr2=0x7f757c06d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 0x7f757c06d180 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f7594033000 tx=0x7f7594009030 comp rx=0 tx=0).stop 2023-10-14T18:20:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 msgr2=0x7f7598149dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 secure :-1 s=READY pgs=3243 cs=0 l=1 rev1=1 crypto rx=0x7f75980c8bb0 tx=0x7f759000ccb0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 shutdown_connections 2023-10-14T18:20:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7598146050 0x7f7598149dd0 unknown :-1 s=CLOSED pgs=3243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f757c06acc0 0x7f757c06d180 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f75980c4900 0x7f75980b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 --2- 172.21.15.29:0/2063301980 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f75840039a0 0x7f75980b12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.562+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 >> 172.21.15.29:0/2063301980 conn(0x7f75980cb410 msgr2=0x7f75980cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:02.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.563+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 shutdown_connections 2023-10-14T18:20:02.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.563+0000 7f759f644640 1 -- 172.21.15.29:0/2063301980 wait complete. 2023-10-14T18:20:02.584 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:20:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.811+0000 7fd9f10a3640 1 Processor -- start 2023-10-14T18:20:02.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.811+0000 7fd9f10a3640 1 -- start start 2023-10-14T18:20:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.812+0000 7fd9f10a3640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9ec15b320 0x7fd9ec0cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.813 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.812+0000 7fd9f10a3640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec15b9d0 con 0x7fd9ec0ad970 2023-10-14T18:20:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.812+0000 7fd9f10a3640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9ec15bcb0 con 0x7fd9ec06c040 2023-10-14T18:20:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.812+0000 7fd9f10a3640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9ec15b7d0 con 0x7fd9ec15b320 2023-10-14T18:20:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.812+0000 7fd9f08a2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fd9ec15b320 0x7fd9ec0cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.814 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9eb7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fd9ec0ad970 0x7fd9ec0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44836/0 (socket says 172.21.15.29:44836) 2023-10-14T18:20:02.815 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9eb7fe640 1 -- 172.21.15.29:0/693424140 learned_addr learned my addr 172.21.15.29:0/693424140 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:02.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9f08a2640 1 -- 172.21.15.29:0/693424140 >> v1:172.21.15.186:6789/0 conn(0x7fd9ec06c040 legacy=0x7fd9ec06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:02.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9f08a2640 1 -- 172.21.15.29:0/693424140 >> v1:172.21.15.29:6789/0 conn(0x7fd9ec0ad970 legacy=0x7fd9ec0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.816 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9f08a2640 1 -- 172.21.15.29:0/693424140 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9ec15bf90 con 0x7fd9ec15b320 2023-10-14T18:20:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.813+0000 7fd9eaffd640 1 -- 172.21.15.29:0/693424140 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123273483 0 0) 0x7fd9ec15b9d0 con 0x7fd9ec0ad970 2023-10-14T18:20:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.814+0000 7fd9f08a2640 1 --2- 172.21.15.29:0/693424140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec15b320 0x7fd9ec0cd180 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009dc0 tx=0x7fd9e402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56fc1cae278ef5c4 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.814+0000 7fd9eaffd640 1 -- 172.21.15.29:0/693424140 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e402cc40 con 0x7fd9ec15b320 2023-10-14T18:20:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9eaffd640 1 -- 172.21.15.29:0/693424140 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9e402cda0 con 0x7fd9ec15b320 2023-10-14T18:20:02.817 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9eaffd640 1 -- 172.21.15.29:0/693424140 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9e4034660 con 0x7fd9ec15b320 2023-10-14T18:20:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 -- 172.21.15.29:0/693424140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec15b320 msgr2=0x7fd9ec0cd180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/693424140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec15b320 0x7fd9ec0cd180 secure :-1 s=READY pgs=3245 cs=0 l=1 rev1=1 crypto rx=0x7fd9e4009dc0 tx=0x7fd9e402be90 comp rx=0 tx=0).stop 2023-10-14T18:20:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 -- 172.21.15.29:0/693424140 shutdown_connections 2023-10-14T18:20:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/693424140 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec15b320 0x7fd9ec0cd180 unknown :-1 s=CLOSED pgs=3245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.818 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 -- 172.21.15.29:0/693424140 >> 172.21.15.29:0/693424140 conn(0x7fd9ec0ce6a0 msgr2=0x7fd9ec0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 -- 172.21.15.29:0/693424140 shutdown_connections 2023-10-14T18:20:02.819 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.815+0000 7fd9f10a3640 1 -- 172.21.15.29:0/693424140 wait complete. 2023-10-14T18:20:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.817+0000 7fd9f10a3640 1 Processor -- start 2023-10-14T18:20:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.817+0000 7fd9f10a3640 1 -- start start 2023-10-14T18:20:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.821 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9ec0ad970 0x7fd9ec0b2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9ec15b320 0x7fd9ec158b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec1f1000 con 0x7fd9ec06c040 2023-10-14T18:20:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0b1910 con 0x7fd9ec0ad970 2023-10-14T18:20:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9f10a3640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9ec0a8f30 con 0x7fd9ec15b320 2023-10-14T18:20:02.822 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9ebfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34610/0 (socket says 172.21.15.29:34610) 2023-10-14T18:20:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.818+0000 7fd9ebfff640 1 -- 172.21.15.29:0/2351573835 learned_addr learned my addr 172.21.15.29:0/2351573835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 -- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9ec15b320 msgr2=0x7fd9ec158b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9ec15b320 0x7fd9ec158b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.823 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 -- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9ec0ad970 msgr2=0x7fd9ec0b2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9ec0ad970 0x7fd9ec0b2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 -- 172.21.15.29:0/2351573835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e4009a70 con 0x7fd9ec06c040 2023-10-14T18:20:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9ebfff640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00b3f0 tx=0x7fd9dc00b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc011830 con 0x7fd9ec06c040 2023-10-14T18:20:02.824 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9dc011990 con 0x7fd9ec06c040 2023-10-14T18:20:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.819+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9dc011d40 con 0x7fd9ec06c040 2023-10-14T18:20:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.820+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9ec0a9210 con 0x7fd9ec06c040 2023-10-14T18:20:02.825 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.820+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9ec0b15a0 con 0x7fd9ec06c040 2023-10-14T18:20:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.821+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9ec06b330 con 0x7fd9ec06c040 2023-10-14T18:20:02.826 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.822+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7fd9dc01a400 con 0x7fd9ec06c040 2023-10-14T18:20:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.823+0000 7fd9e8ff9640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 0x7fd9d806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.823+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fd9dc0894b0 con 0x7fd9ec06c040 2023-10-14T18:20:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.823+0000 7fd9eb7fe640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 0x7fd9d806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:02.827 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.824+0000 7fd9eb7fe640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 0x7fd9d806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0007fd0 tx=0x7fd9e0007920 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:02.828 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:02.827+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fd9dc054420 con 0x7fd9ec06c040 2023-10-14T18:20:03.026 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.025+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9ec0a9400 con 0x7fd9ec06c040 2023-10-14T18:20:03.027 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.026+0000 7fd9e8ff9640 1 -- 172.21.15.29:0/2351573835 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) v1 ==== 74+0+17888 (secure 0 0 0) 0x7fd9dc057a40 con 0x7fd9ec06c040 2023-10-14T18:20:03.028 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:03.028 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":273,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:20:01.230404+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2967850488":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3208700101":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/2196987879":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/754934507":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3551524051":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2464998546":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3292195983":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/4006438795":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1086342911":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/503271084":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/4178572946":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3279571573":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/1082110076":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2709558548":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1343183116":"2023-10-15T18:19:52.752456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:20:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 msgr2=0x7fd9d806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 0x7fd9d806d180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fd9e0007fd0 tx=0x7fd9e0007920 comp rx=0 tx=0).stop 2023-10-14T18:20:03.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 msgr2=0x7fd9ec1f0ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 secure :-1 s=READY pgs=3246 cs=0 l=1 rev1=1 crypto rx=0x7fd9dc00b3f0 tx=0x7fd9dc00b8b0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 shutdown_connections 2023-10-14T18:20:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fd9d806acc0 0x7fd9d806d180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fd9ec15b320 0x7fd9ec158b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fd9ec0ad970 0x7fd9ec0b2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 --2- 172.21.15.29:0/2351573835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fd9ec06c040 0x7fd9ec1f0ac0 unknown :-1 s=CLOSED pgs=3246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 >> 172.21.15.29:0/2351573835 conn(0x7fd9ec0ce6a0 msgr2=0x7fd9ec0cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 shutdown_connections 2023-10-14T18:20:03.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.029+0000 7fd9f10a3640 1 -- 172.21.15.29:0/2351573835 wait complete. 2023-10-14T18:20:03.047 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:20:03.048 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:20:03.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.271+0000 7f268ab35640 1 Processor -- start 2023-10-14T18:20:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.271+0000 7f268ab35640 1 -- start start 2023-10-14T18:20:03.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268ab35640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f268415c9f0 0x7f26840c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268ab35640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2684160a90 con 0x7f268415dde0 2023-10-14T18:20:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268ab35640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26840cd900 con 0x7f26840c9230 2023-10-14T18:20:03.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268ab35640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26840cdec0 con 0x7f268415c9f0 2023-10-14T18:20:03.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268a334640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f268415dde0 0x7f26840ac890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44848/0 (socket says 172.21.15.29:44848) 2023-10-14T18:20:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f268a334640 1 -- 172.21.15.29:0/1734990805 learned_addr learned my addr 172.21.15.29:0/1734990805 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.273+0000 7f2689b33640 1 --2- 172.21.15.29:0/1734990805 >> v2:172.21.15.29:3300/0 conn(0x7f268415c9f0 0x7f26840c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.274+0000 7f2688b31640 1 -- 172.21.15.29:0/1734990805 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871877086 0 0) 0x7f2684160a90 con 0x7f268415dde0 2023-10-14T18:20:03.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.274+0000 7f2688b31640 1 -- 172.21.15.29:0/1734990805 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2670003520 con 0x7f268415dde0 2023-10-14T18:20:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.274+0000 7f2689b33640 1 -- 172.21.15.29:0/1734990805 >> v1:172.21.15.186:6789/0 conn(0x7f26840c9230 legacy=0x7f26840709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.274+0000 7f2689b33640 1 -- 172.21.15.29:0/1734990805 >> v1:172.21.15.29:6789/0 conn(0x7f268415dde0 legacy=0x7f26840ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.274+0000 7f2689b33640 1 -- 172.21.15.29:0/1734990805 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26840cdbe0 con 0x7f268415c9f0 2023-10-14T18:20:03.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.275+0000 7f2689b33640 1 --2- 172.21.15.29:0/1734990805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f26840c8cf0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f26800097a0 tx=0x7f268002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92366c8f9d4b3180 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f2688b31640 1 -- 172.21.15.29:0/1734990805 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680031830 con 0x7f268415c9f0 2023-10-14T18:20:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f2688b31640 1 -- 172.21.15.29:0/1734990805 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2680031990 con 0x7f268415c9f0 2023-10-14T18:20:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f2688b31640 1 -- 172.21.15.29:0/1734990805 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2680031d00 con 0x7f268415c9f0 2023-10-14T18:20:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 -- 172.21.15.29:0/1734990805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 msgr2=0x7f26840c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 --2- 172.21.15.29:0/1734990805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f26840c8cf0 secure :-1 s=READY pgs=3248 cs=0 l=1 rev1=1 crypto rx=0x7f26800097a0 tx=0x7f268002b8d0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 -- 172.21.15.29:0/1734990805 shutdown_connections 2023-10-14T18:20:03.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 --2- 172.21.15.29:0/1734990805 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f26840c8cf0 unknown :-1 s=CLOSED pgs=3248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 -- 172.21.15.29:0/1734990805 >> 172.21.15.29:0/1734990805 conn(0x7f26840ce6a0 msgr2=0x7f26840ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:03.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 -- 172.21.15.29:0/1734990805 shutdown_connections 2023-10-14T18:20:03.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.276+0000 7f268ab35640 1 -- 172.21.15.29:0/1734990805 wait complete. 2023-10-14T18:20:03.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.278+0000 7f268ab35640 1 Processor -- start 2023-10-14T18:20:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.278+0000 7f268ab35640 1 -- start start 2023-10-14T18:20:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26840c9230 0x7f2684153f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268415dde0 0x7f26841f1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26841f68b0 con 0x7f268415c9f0 2023-10-14T18:20:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26841f5950 con 0x7f268415dde0 2023-10-14T18:20:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.279+0000 7f268ab35640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f26841f5c30 con 0x7f26840c9230 2023-10-14T18:20:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34632/0 (socket says 172.21.15.29:34632) 2023-10-14T18:20:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 -- 172.21.15.29:0/1468477527 learned_addr learned my addr 172.21.15.29:0/1468477527 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689b33640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26840c9230 0x7f2684153f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f268a334640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268415dde0 0x7f26841f1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 -- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26840c9230 msgr2=0x7f2684153f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26840c9230 0x7f2684153f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 -- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268415dde0 msgr2=0x7f26841f1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268415dde0 0x7f26841f1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.280+0000 7f2689332640 1 -- 172.21.15.29:0/1468477527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2680009400 con 0x7f268415c9f0 2023-10-14T18:20:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f2689332640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f267400c430 tx=0x7f267400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674011790 con 0x7f268415c9f0 2023-10-14T18:20:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f26740118f0 con 0x7f268415c9f0 2023-10-14T18:20:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2674011ce0 con 0x7f268415c9f0 2023-10-14T18:20:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26841f5f10 con 0x7f268415c9f0 2023-10-14T18:20:03.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.281+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26841f64d0 con 0x7f268415c9f0 2023-10-14T18:20:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.282+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f268404e9b0 con 0x7f268415c9f0 2023-10-14T18:20:03.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.284+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f267401c460 con 0x7f268415c9f0 2023-10-14T18:20:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.285+0000 7f267a7fc640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 0x7f266806d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.285+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f26740895a0 con 0x7f268415c9f0 2023-10-14T18:20:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.285+0000 7f2689b33640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 0x7f266806d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.286+0000 7f2689b33640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 0x7f266806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2680036000 tx=0x7f268003c040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.289+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f26740545e0 con 0x7f268415c9f0 2023-10-14T18:20:03.532 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.532+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26840c1f70 con 0x7f268415c9f0 2023-10-14T18:20:03.533 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.532+0000 7f267a7fc640 1 -- 172.21.15.29:0/1468477527 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f2674057c00 con 0x7f268415c9f0 2023-10-14T18:20:03.534 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:03.534 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:20:03.534 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:20:03.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 msgr2=0x7f266806d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 0x7f266806d180 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f2680036000 tx=0x7f268003c040 comp rx=0 tx=0).stop 2023-10-14T18:20:03.536 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 msgr2=0x7f2684159770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 secure :-1 s=READY pgs=3249 cs=0 l=1 rev1=1 crypto rx=0x7f267400c430 tx=0x7f267400c8f0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 shutdown_connections 2023-10-14T18:20:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f268415dde0 0x7f26841f1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.537 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f268415c9f0 0x7f2684159770 unknown :-1 s=CLOSED pgs=3249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f266806acc0 0x7f266806d180 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 --2- 172.21.15.29:0/1468477527 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f26840c9230 0x7f2684153f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.538 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.535+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 >> 172.21.15.29:0/1468477527 conn(0x7f26840ce6a0 msgr2=0x7f268415a360 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:03.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.536+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 shutdown_connections 2023-10-14T18:20:03.539 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.536+0000 7f268ab35640 1 -- 172.21.15.29:0/1468477527 wait complete. 2023-10-14T18:20:03.555 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:20:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.772+0000 7f3adf615640 1 Processor -- start 2023-10-14T18:20:03.773 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.772+0000 7f3adf615640 1 -- start start 2023-10-14T18:20:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3adf615640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad815ae80 0x7f3ad815e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.779 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3adf615640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ad80ce390 0x7f3ad8070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3adf615640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ad815ed60 con 0x7f3ad815aaa0 2023-10-14T18:20:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3adf615640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ad806b810 con 0x7f3ad80ce390 2023-10-14T18:20:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3adf615640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad80707e0 con 0x7f3ad815ae80 2023-10-14T18:20:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3addbb2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ad80ce390 0x7f3ad8070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.780 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3addbb2640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f3ad80ce390 0x7f3ad8070f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34660/0 (socket says 172.21.15.29:34660) 2023-10-14T18:20:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.774+0000 7f3addbb2640 1 -- 172.21.15.29:0/2577631835 learned_addr learned my addr 172.21.15.29:0/2577631835 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.775+0000 7f3adcbb0640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad815ae80 0x7f3ad815e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.775+0000 7f3addbb2640 1 -- 172.21.15.29:0/2577631835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad815ae80 msgr2=0x7f3ad815e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:03.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.775+0000 7f3addbb2640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad815ae80 0x7f3ad815e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.775+0000 7f3addbb2640 1 -- 172.21.15.29:0/2577631835 >> v1:172.21.15.186:6789/0 conn(0x7f3ad815aaa0 legacy=0x7f3ad80cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.775+0000 7f3addbb2640 1 -- 172.21.15.29:0/2577631835 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad815ef90 con 0x7f3ad80ce390 2023-10-14T18:20:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.777+0000 7f3addbb2640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad80ce390 0x7f3ad8070f50 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009e50 tx=0x7f3ad002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5224b1788570236 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.777+0000 7f3acffff640 1 -- 172.21.15.29:0/2577631835 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad002cb00 con 0x7f3ad80ce390 2023-10-14T18:20:03.782 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.777+0000 7f3acffff640 1 -- 172.21.15.29:0/2577631835 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ad002cc60 con 0x7f3ad80ce390 2023-10-14T18:20:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.777+0000 7f3acffff640 1 -- 172.21.15.29:0/2577631835 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ad00343f0 con 0x7f3ad80ce390 2023-10-14T18:20:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 -- 172.21.15.29:0/2577631835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad80ce390 msgr2=0x7f3ad8070f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad80ce390 0x7f3ad8070f50 secure :-1 s=READY pgs=3252 cs=0 l=1 rev1=1 crypto rx=0x7f3ad0009e50 tx=0x7f3ad002bf60 comp rx=0 tx=0).stop 2023-10-14T18:20:03.783 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 -- 172.21.15.29:0/2577631835 shutdown_connections 2023-10-14T18:20:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad80ce390 0x7f3ad8070f50 unknown :-1 s=CLOSED pgs=3252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 --2- 172.21.15.29:0/2577631835 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad815ae80 0x7f3ad815e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 -- 172.21.15.29:0/2577631835 >> 172.21.15.29:0/2577631835 conn(0x7f3ad80cb410 msgr2=0x7f3ad80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 -- 172.21.15.29:0/2577631835 shutdown_connections 2023-10-14T18:20:03.784 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.778+0000 7f3adf615640 1 -- 172.21.15.29:0/2577631835 wait complete. 2023-10-14T18:20:03.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.780+0000 7f3adf615640 1 Processor -- start 2023-10-14T18:20:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.780+0000 7f3adf615640 1 -- start start 2023-10-14T18:20:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 0x7f3ad80afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 0x7f3ad815c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad81f2b60 con 0x7f3ad815aaa0 2023-10-14T18:20:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad81f2e40 con 0x7f3ad815ae80 2023-10-14T18:20:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adf615640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad81f3120 con 0x7f3ad80ce390 2023-10-14T18:20:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adcbb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.789 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adcbb0640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34676/0 (socket says 172.21.15.29:34676) 2023-10-14T18:20:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3adcbb0640 1 -- 172.21.15.29:0/3440283789 learned_addr learned my addr 172.21.15.29:0/3440283789 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.781+0000 7f3add3b1640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 0x7f3ad80afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3addbb2640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 0x7f3ad815c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 -- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 msgr2=0x7f3ad80afd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.790 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 0x7f3ad80afd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 -- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 msgr2=0x7f3ad815c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 0x7f3ad815c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 -- 172.21.15.29:0/3440283789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ad0009b30 con 0x7f3ad815aaa0 2023-10-14T18:20:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3add3b1640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 0x7f3ad80afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:03.791 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3addbb2640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 0x7f3ad815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adcbb0640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800d800 tx=0x7f3ac800dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800b6c0 con 0x7f3ad815aaa0 2023-10-14T18:20:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ac800b820 con 0x7f3ad815aaa0 2023-10-14T18:20:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3ac800bbb0 con 0x7f3ad815aaa0 2023-10-14T18:20:03.792 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad81f3400 con 0x7f3ad815aaa0 2023-10-14T18:20:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.782+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad81f39c0 con 0x7f3ad815aaa0 2023-10-14T18:20:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.784+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad804e160 con 0x7f3ad815aaa0 2023-10-14T18:20:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.785+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 162) v1 ==== 89067+0+0 (secure 0 0 0) 0x7f3ac800bd10 con 0x7f3ad815aaa0 2023-10-14T18:20:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.786+0000 7f3acdffb640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 0x7f3abc06d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:03.793 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.786+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f3ac8089c50 con 0x7f3ad815aaa0 2023-10-14T18:20:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.786+0000 7f3add3b1640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 0x7f3abc06d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.787+0000 7f3add3b1640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 0x7f3abc06d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8159ec0 tx=0x7f3ad4009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:03.794 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.790+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3ac8054bc0 con 0x7f3ad815aaa0 2023-10-14T18:20:03.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:03.856+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f3ac8054f90 con 0x7f3ad815aaa0 2023-10-14T18:20:04.028 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.028+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ad80b16b0 con 0x7f3ad815aaa0 2023-10-14T18:20:04.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.029+0000 7f3acdffb640 1 -- 172.21.15.29:0/3440283789 <== mon.0 v2:172.21.15.29:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 139 v139) v1 ==== 95+0+4933 (secure 0 0 0) 0x7f3ac8070ad0 con 0x7f3ad815aaa0 2023-10-14T18:20:04.030 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:04.030 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":139,"default_fscid":17,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[{"mdsmap":{"epoch":139,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2023-10-14T18:19:37.403465+0000","modified":"2023-10-14T18:19:40.808405+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":12843},"failed":[],"damaged":[],"stopped":[],"info":{"gid_12843":{"gid":12843,"name":"d","rank":0,"incarnation":135,"state":"up:active","state_seq":3,"addr":"172.21.15.186:6835/3411376277","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":3411376277},{"type":"v1","addr":"172.21.15.186:6835","nonce":3411376277}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[35],"metadata_pool":34,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":17}]} 2023-10-14T18:20:04.031 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 139 2023-10-14T18:20:04.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 msgr2=0x7f3abc06d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 0x7f3abc06d150 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f3ad8159ec0 tx=0x7f3ad4009040 comp rx=0 tx=0).stop 2023-10-14T18:20:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 msgr2=0x7f3ad80acc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 secure :-1 s=READY pgs=3253 cs=0 l=1 rev1=1 crypto rx=0x7f3ac800d800 tx=0x7f3ac800dcc0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 shutdown_connections 2023-10-14T18:20:04.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3ad815ae80 0x7f3ad815c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3ad815aaa0 0x7f3ad80acc60 unknown :-1 s=CLOSED pgs=3253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3abc06ac90 0x7f3abc06d150 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 --2- 172.21.15.29:0/3440283789 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3ad80ce390 0x7f3ad80afd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 >> 172.21.15.29:0/3440283789 conn(0x7f3ad80cb410 msgr2=0x7f3ad806f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:04.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 shutdown_connections 2023-10-14T18:20:04.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.032+0000 7f3adf615640 1 -- 172.21.15.29:0/3440283789 wait complete. 2023-10-14T18:20:04.052 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:20:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.277+0000 7fe43d44d640 1 Processor -- start 2023-10-14T18:20:04.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.277+0000 7fe43d44d640 1 -- start start 2023-10-14T18:20:04.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.279+0000 7fe43d44d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe43806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.279+0000 7fe43d44d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4380cda80 0x7fe43815ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.279+0000 7fe43d44d640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43806c960 con 0x7fe4380ad970 2023-10-14T18:20:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.279+0000 7fe43d44d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43815b9d0 con 0x7fe4380cda80 2023-10-14T18:20:04.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.279+0000 7fe43d44d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43815bcb0 con 0x7fe43806c040 2023-10-14T18:20:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe43cc4c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fe4380cda80 0x7fe43815ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe4377fe640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7fe4380ad970 0x7fe43815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45062/0 (socket says 172.21.15.29:45062) 2023-10-14T18:20:04.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe4377fe640 1 -- 172.21.15.29:0/2974355241 learned_addr learned my addr 172.21.15.29:0/2974355241 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:04.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe437fff640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe43806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe43cc4c640 1 -- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 msgr2=0x7fe43806c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:04.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe43cc4c640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe43806c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe43cc4c640 1 -- 172.21.15.29:0/2974355241 >> v1:172.21.15.186:6789/0 conn(0x7fe4380ad970 legacy=0x7fe43815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.280+0000 7fe43cc4c640 1 -- 172.21.15.29:0/2974355241 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43815bf90 con 0x7fe4380cda80 2023-10-14T18:20:04.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.282+0000 7fe43cc4c640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe43815ed90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe430009d90 tx=0x7fe43002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6bee03e3ff03a55 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.282+0000 7fe436ffd640 1 -- 172.21.15.29:0/2974355241 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe43002cc50 con 0x7fe4380cda80 2023-10-14T18:20:04.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe436ffd640 1 -- 172.21.15.29:0/2974355241 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe43002cdb0 con 0x7fe4380cda80 2023-10-14T18:20:04.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe436ffd640 1 -- 172.21.15.29:0/2974355241 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe430034660 con 0x7fe4380cda80 2023-10-14T18:20:04.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 -- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 msgr2=0x7fe43815ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe43815ed90 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe430009d90 tx=0x7fe43002bea0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 -- 172.21.15.29:0/2974355241 shutdown_connections 2023-10-14T18:20:04.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe43815ed90 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 --2- 172.21.15.29:0/2974355241 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe43806c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.283+0000 7fe43d44d640 1 -- 172.21.15.29:0/2974355241 >> 172.21.15.29:0/2974355241 conn(0x7fe4380ce6a0 msgr2=0x7fe4380ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:04.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.284+0000 7fe43d44d640 1 -- 172.21.15.29:0/2974355241 shutdown_connections 2023-10-14T18:20:04.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.284+0000 7fe43d44d640 1 -- 172.21.15.29:0/2974355241 wait complete. 2023-10-14T18:20:04.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.286+0000 7fe43d44d640 1 Processor -- start 2023-10-14T18:20:04.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.286+0000 7fe43d44d640 1 -- start start 2023-10-14T18:20:04.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.287+0000 7fe43d44d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe4380ca8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.287+0000 7fe43d44d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.287+0000 7fe43d44d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe4381f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe43d44d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438153880 con 0x7fe4380ad970 2023-10-14T18:20:04.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe43d44d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe438155700 con 0x7fe4380cda80 2023-10-14T18:20:04.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe43d44d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4381f71b0 con 0x7fe43806c040 2023-10-14T18:20:04.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34686/0 (socket says 172.21.15.29:34686) 2023-10-14T18:20:04.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 -- 172.21.15.29:0/3478528792 learned_addr learned my addr 172.21.15.29:0/3478528792 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:04.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe43cc4c640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe4381f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe437fff640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe4380ca8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 -- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 msgr2=0x7fe4380ca8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe4380ca8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 -- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 msgr2=0x7fe4381f6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.292 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe4381f6c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe4377fe640 1 -- 172.21.15.29:0/3478528792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe430009a70 con 0x7fe4380ad970 2023-10-14T18:20:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.288+0000 7fe43cc4c640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe4381f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe4377fe640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00c4d0 tx=0x7fe42c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c005ce0 con 0x7fe4380ad970 2023-10-14T18:20:04.293 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c005e40 con 0x7fe4380ad970 2023-10-14T18:20:04.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe42c007270 con 0x7fe4380ad970 2023-10-14T18:20:04.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4380c8eb0 con 0x7fe4380ad970 2023-10-14T18:20:04.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.289+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4381f73d0 con 0x7fe4380ad970 2023-10-14T18:20:04.294 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.290+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43806b330 con 0x7fe4380ad970 2023-10-14T18:20:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.291+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7fe42c0073d0 con 0x7fe4380ad970 2023-10-14T18:20:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.292+0000 7fe434ff9640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 0x7fe42406d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.295 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.292+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7fe42c088e90 con 0x7fe4380ad970 2023-10-14T18:20:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.292+0000 7fe437fff640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 0x7fe42406d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.296 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.293+0000 7fe437fff640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 0x7fe42406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe4381fa220 tx=0x7fe428008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.297 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.297+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fe42c053be0 con 0x7fe4380ad970 2023-10-14T18:20:04.486 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.485+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe43804e9b0 con 0x7fe4380ad970 2023-10-14T18:20:04.489 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.488+0000 7fe434ff9640 1 -- 172.21.15.29:0/3478528792 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v273) v1 ==== 74+0+17888 (secure 0 0 0) 0x7fe42c08a090 con 0x7fe4380ad970 2023-10-14T18:20:04.489 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:04.490 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":273,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:20:01.230404+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":34,"pool_name":"cephfs_metadata","create_time":"2023-10-14T18:19:35.816866+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"270","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":35,"pool_name":"cephfs_data","create_time":"2023-10-14T18:19:36.425584+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"271","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.29:0/2967850488":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3208700101":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/2196987879":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/754934507":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3551524051":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2464998546":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3292195983":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/4006438795":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1086342911":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/503271084":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/4178572946":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3279571573":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/1082110076":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2709558548":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1343183116":"2023-10-15T18:19:52.752456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:20:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 msgr2=0x7fe42406d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 0x7fe42406d3c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe4381fa220 tx=0x7fe428008040 comp rx=0 tx=0).stop 2023-10-14T18:20:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 msgr2=0x7fe438153340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.492 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 secure :-1 s=READY pgs=3254 cs=0 l=1 rev1=1 crypto rx=0x7fe42c00c4d0 tx=0x7fe42c00c990 comp rx=0 tx=0).stop 2023-10-14T18:20:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 shutdown_connections 2023-10-14T18:20:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fe4380cda80 0x7fe4381f6c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fe4380ad970 0x7fe438153340 unknown :-1 s=CLOSED pgs=3254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fe42406af00 0x7fe42406d3c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 --2- 172.21.15.29:0/3478528792 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fe43806c040 0x7fe4380ca8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 >> 172.21.15.29:0/3478528792 conn(0x7fe4380ce6a0 msgr2=0x7fe438159c60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 shutdown_connections 2023-10-14T18:20:04.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.491+0000 7fe43d44d640 1 -- 172.21.15.29:0/3478528792 wait complete. 2023-10-14T18:20:04.512 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2023-10-14T18:20:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.733+0000 7f2261d3c640 1 Processor -- start 2023-10-14T18:20:04.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.733+0000 7f2261d3c640 1 -- start start 2023-10-14T18:20:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.734+0000 7f2261d3c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f225c15c9f0 0x7f225c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2261d3c640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f225c0c9230 0x7f225c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2261d3c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c070f20 0x7f225c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2261d3c640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c0cdec0 con 0x7f225c15c9f0 2023-10-14T18:20:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2261d3c640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f225c160a90 con 0x7f225c0c9230 2023-10-14T18:20:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2261d3c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c0cd900 con 0x7f225c070f20 2023-10-14T18:20:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2260d3a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f225c15c9f0 0x7f225c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2260d3a640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f225c15c9f0 0x7f225c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34692/0 (socket says 172.21.15.29:34692) 2023-10-14T18:20:04.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.735+0000 7f2260d3a640 1 -- 172.21.15.29:0/3043192649 learned_addr learned my addr 172.21.15.29:0/3043192649 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f225bfff640 1 --2- 172.21.15.29:0/3043192649 >> v2:172.21.15.186:3300/0 conn(0x7f225c0c9230 0x7f225c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f226153b640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c070f20 0x7f225c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f2260d3a640 1 -- 172.21.15.29:0/3043192649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c070f20 msgr2=0x7f225c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f2260d3a640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c070f20 0x7f225c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f2260d3a640 1 -- 172.21.15.29:0/3043192649 >> v2:172.21.15.186:3300/0 conn(0x7f225c0c9230 msgr2=0x7f225c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f2260d3a640 1 --2- 172.21.15.29:0/3043192649 >> v2:172.21.15.186:3300/0 conn(0x7f225c0c9230 0x7f225c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.736+0000 7f2260d3a640 1 -- 172.21.15.29:0/3043192649 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f225c0cdbe0 con 0x7f225c15c9f0 2023-10-14T18:20:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.737+0000 7f2260d3a640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c8cf0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f225c0b1260 tx=0x7f225002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8572f23e65321459 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.738+0000 7f225b7fe640 1 -- 172.21.15.29:0/3043192649 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250031870 con 0x7f225c15c9f0 2023-10-14T18:20:04.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.738+0000 7f225b7fe640 1 -- 172.21.15.29:0/3043192649 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22500319d0 con 0x7f225c15c9f0 2023-10-14T18:20:04.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.738+0000 7f225b7fe640 1 -- 172.21.15.29:0/3043192649 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2250031d60 con 0x7f225c15c9f0 2023-10-14T18:20:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.738+0000 7f2261d3c640 1 -- 172.21.15.29:0/3043192649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 msgr2=0x7f225c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.738+0000 7f2261d3c640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c8cf0 secure :-1 s=READY pgs=3255 cs=0 l=1 rev1=1 crypto rx=0x7f225c0b1260 tx=0x7f225002b910 comp rx=0 tx=0).stop 2023-10-14T18:20:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 -- 172.21.15.29:0/3043192649 shutdown_connections 2023-10-14T18:20:04.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c070f20 0x7f225c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 --2- 172.21.15.29:0/3043192649 >> v2:172.21.15.186:3300/0 conn(0x7f225c0c9230 0x7f225c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 --2- 172.21.15.29:0/3043192649 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c8cf0 unknown :-1 s=CLOSED pgs=3255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 -- 172.21.15.29:0/3043192649 >> 172.21.15.29:0/3043192649 conn(0x7f225c0ce6a0 msgr2=0x7f225c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 -- 172.21.15.29:0/3043192649 shutdown_connections 2023-10-14T18:20:04.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.739+0000 7f2261d3c640 1 -- 172.21.15.29:0/3043192649 wait complete. 2023-10-14T18:20:04.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.741+0000 7f2261d3c640 1 Processor -- start 2023-10-14T18:20:04.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.741+0000 7f2261d3c640 1 -- start start 2023-10-14T18:20:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f225c070f20 0x7f225c0c4c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c0c9230 0x7f225c15fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.746 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c15fff0 con 0x7f225c15c9f0 2023-10-14T18:20:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f225c15dc50 con 0x7f225c070f20 2023-10-14T18:20:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2261d3c640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f225c1f2870 con 0x7f225c0c9230 2023-10-14T18:20:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34702/0 (socket says 172.21.15.29:34702) 2023-10-14T18:20:04.747 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 -- 172.21.15.29:0/2108695360 learned_addr learned my addr 172.21.15.29:0/2108695360 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 -- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c0c9230 msgr2=0x7f225c15fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c0c9230 0x7f225c15fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 -- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f225c070f20 msgr2=0x7f225c0c4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f2260d3a640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f225c070f20 0x7f225c0c4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.748 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f225c070f20 0x7f225c0c4c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.742+0000 7f226153b640 1 -- 172.21.15.29:0/2108695360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22500094d0 con 0x7f225c15c9f0 2023-10-14T18:20:04.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f226153b640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f2254007e40 tx=0x7f225400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2254014620 con 0x7f225c15c9f0 2023-10-14T18:20:04.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f225400ee90 con 0x7f225c15c9f0 2023-10-14T18:20:04.749 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22540149e0 con 0x7f225c15c9f0 2023-10-14T18:20:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f225c1f2b50 con 0x7f225c15c9f0 2023-10-14T18:20:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.743+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f225c1f2520 con 0x7f225c15c9f0 2023-10-14T18:20:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.744+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f225c1f26b0 con 0x7f225c15c9f0 2023-10-14T18:20:04.750 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.745+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f2254014b40 con 0x7f225c15c9f0 2023-10-14T18:20:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.746+0000 7f22597fa640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 0x7f224006d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:04.751 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.746+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(273..273 src has 1..273) v4 ==== 6642+0+0 (secure 0 0 0) 0x7f225408a500 con 0x7f225c15c9f0 2023-10-14T18:20:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.746+0000 7f2260d3a640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 0x7f224006d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.747+0000 7f2260d3a640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 0x7f224006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2250036000 tx=0x7f225003c040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:04.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.750+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2254055110 con 0x7f225c15c9f0 2023-10-14T18:20:04.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.986+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f225c1f3010 con 0x7f225c15c9f0 2023-10-14T18:20:04.988 INFO:tasks.ceph.mon.a.smithi029.stderr:2023-10-14T18:20:04.987+0000 7f44e3c12640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2023-10-14T18:20:04.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.991+0000 7f22597fa640 1 -- 172.21.15.29:0/2108695360 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v140) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2254058730 con 0x7f225c15c9f0 2023-10-14T18:20:04.992 INFO:teuthology.orchestra.run.smithi029.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2023-10-14T18:20:04.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.994+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 msgr2=0x7f224006d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.994+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 0x7f224006d3c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f2250036000 tx=0x7f225003c040 comp rx=0 tx=0).stop 2023-10-14T18:20:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.994+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 msgr2=0x7f225c0c7940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.994+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 secure :-1 s=READY pgs=3256 cs=0 l=1 rev1=1 crypto rx=0x7f2254007e40 tx=0x7f225400e4b0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 shutdown_connections 2023-10-14T18:20:04.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f225c15c9f0 0x7f225c0c7940 unknown :-1 s=CLOSED pgs=3256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f224006af00 0x7f224006d3c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f225c0c9230 0x7f225c15fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 --2- 172.21.15.29:0/2108695360 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f225c070f20 0x7f225c0c4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 >> 172.21.15.29:0/2108695360 conn(0x7f225c0ce6a0 msgr2=0x7f225c15b990 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:04.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 shutdown_connections 2023-10-14T18:20:04.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:04.995+0000 7f2261d3c640 1 -- 172.21.15.29:0/2108695360 wait complete. 2023-10-14T18:20:05.016 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2023-10-14T18:20:05.021 INFO:tasks.ceph.mds.d.smithi186.stderr: -3923> 2023-10-14T18:19:42.866+0000 7fbe96330640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2023-10-14T18:20:05.114 INFO:tasks.ceph.mds.d.smithi186.stdout:starting mds.d at 2023-10-14T18:20:05.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.245+0000 7f8e085d5640 1 Processor -- start 2023-10-14T18:20:05.247 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.246+0000 7f8e085d5640 1 -- start start 2023-10-14T18:20:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.247+0000 7f8e085d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e00069da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.247+0000 7f8e085d5640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e000caac0 0x7f8e0006a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.248 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.247+0000 7f8e085d5640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e000cb030 con 0x7f8e000c9ca0 2023-10-14T18:20:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.247+0000 7f8e085d5640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8e000c91c0 con 0x7f8e000caac0 2023-10-14T18:20:05.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.247+0000 7f8e085d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000c94a0 con 0x7f8e0015ad70 2023-10-14T18:20:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.248+0000 7f8e06371640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e00069da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.248+0000 7f8e06b72640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8e000caac0 0x7f8e0006a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.248+0000 7f8e05b70640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f8e000c9ca0 0x7f8e000ca3b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45064/0 (socket says 172.21.15.29:45064) 2023-10-14T18:20:05.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.248+0000 7f8e05b70640 1 -- 172.21.15.29:0/1240408923 learned_addr learned my addr 172.21.15.29:0/1240408923 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:05.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.249+0000 7f8e06b72640 1 -- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 msgr2=0x7f8e00069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:05.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.249+0000 7f8e06b72640 1 --2- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e00069da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.249+0000 7f8e06b72640 1 -- 172.21.15.29:0/1240408923 >> v1:172.21.15.186:6789/0 conn(0x7f8e000c9ca0 legacy=0x7f8e000ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:05.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.249+0000 7f8e06b72640 1 -- 172.21.15.29:0/1240408923 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e000c9780 con 0x7f8e000caac0 2023-10-14T18:20:05.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.251+0000 7f8e06b72640 1 --2- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000caac0 0x7f8e0006a2e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009d90 tx=0x7f8df802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bc580f5b8169e41 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:05.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.251+0000 7f8e0536f640 1 -- 172.21.15.29:0/1240408923 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df802cc50 con 0x7f8e000caac0 2023-10-14T18:20:05.252 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.251+0000 7f8e0536f640 1 -- 172.21.15.29:0/1240408923 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df802cdb0 con 0x7f8e000caac0 2023-10-14T18:20:05.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.251+0000 7f8e0536f640 1 -- 172.21.15.29:0/1240408923 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df8034660 con 0x7f8e000caac0 2023-10-14T18:20:05.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 -- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000caac0 msgr2=0x7f8e0006a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:05.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 --2- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000caac0 0x7f8e0006a2e0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f8df8009d90 tx=0x7f8df802bea0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 -- 172.21.15.29:0/1240408923 shutdown_connections 2023-10-14T18:20:05.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 --2- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000caac0 0x7f8e0006a2e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.254 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 --2- 172.21.15.29:0/1240408923 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e00069da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 -- 172.21.15.29:0/1240408923 >> 172.21.15.29:0/1240408923 conn(0x7f8e000ce9b0 msgr2=0x7f8e000cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:05.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 -- 172.21.15.29:0/1240408923 shutdown_connections 2023-10-14T18:20:05.255 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.252+0000 7f8e085d5640 1 -- 172.21.15.29:0/1240408923 wait complete. 2023-10-14T18:20:05.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.254+0000 7f8e085d5640 1 Processor -- start 2023-10-14T18:20:05.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.255+0000 7f8e085d5640 1 -- start start 2023-10-14T18:20:05.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000c9ca0 0x7f8e0014e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.257 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e000b2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e0014e940 con 0x7f8e000caac0 2023-10-14T18:20:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000c7810 con 0x7f8e000c9ca0 2023-10-14T18:20:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e085d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8e000c7af0 con 0x7f8e0015ad70 2023-10-14T18:20:05.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e05b70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e05b70640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34716/0 (socket says 172.21.15.29:34716) 2023-10-14T18:20:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.256+0000 7f8e05b70640 1 -- 172.21.15.29:0/220288133 learned_addr learned my addr 172.21.15.29:0/220288133 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e06b72640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e000b2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e06371640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000c9ca0 0x7f8e0014e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.260 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 -- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 msgr2=0x7f8e000b2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e000b2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 -- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000c9ca0 msgr2=0x7f8e0014e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000c9ca0 0x7f8e0014e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 -- 172.21.15.29:0/220288133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8df8009a70 con 0x7f8e000caac0 2023-10-14T18:20:05.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.257+0000 7f8e05b70640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f8df000c4a0 tx=0x7f8df000c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.258+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0005ce0 con 0x7f8e000caac0 2023-10-14T18:20:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.258+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8df0005e40 con 0x7f8e000caac0 2023-10-14T18:20:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.258+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8df0007270 con 0x7f8e000caac0 2023-10-14T18:20:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.258+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e000c7dd0 con 0x7f8e000caac0 2023-10-14T18:20:05.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.258+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e000c8390 con 0x7f8e000caac0 2023-10-14T18:20:05.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.259+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e000c85d0 con 0x7f8e000caac0 2023-10-14T18:20:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.260+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 163) v1 ==== 89275+0+0 (secure 0 0 0) 0x7f8df00073d0 con 0x7f8e000caac0 2023-10-14T18:20:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.262+0000 7f8df6ffd640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 0x7f8de406d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.262+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6728+0+0 (secure 0 0 0) 0x7f8df0088f20 con 0x7f8e000caac0 2023-10-14T18:20:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.262+0000 7f8e06371640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 0x7f8de406d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:05.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.263+0000 7f8e06371640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 0x7f8de406d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009a30 tx=0x7f8dfc008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:05.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.267+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8e000c85d0 con 0x7f8e000caac0 2023-10-14T18:20:05.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:05.505+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f8e000c85d0 con 0x7f8e000caac0 2023-10-14T18:20:06.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.002+0000 7f8df6ffd640 1 -- 172.21.15.29:0/220288133 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v141) v1 ==== 104+0+0 (secure 0 0 0) 0x7f8e000c85d0 con 0x7f8e000caac0 2023-10-14T18:20:06.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 msgr2=0x7f8de406d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 0x7f8de406d3f0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f8dfc009a30 tx=0x7f8dfc008040 comp rx=0 tx=0).stop 2023-10-14T18:20:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 msgr2=0x7f8e000b2770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 secure :-1 s=READY pgs=3257 cs=0 l=1 rev1=1 crypto rx=0x7f8df000c4a0 tx=0x7f8df000c960 comp rx=0 tx=0).stop 2023-10-14T18:20:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 shutdown_connections 2023-10-14T18:20:06.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8de406af30 0x7f8de406d3f0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8e0015ad70 0x7f8e000b2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8e000caac0 0x7f8e000b2770 unknown :-1 s=CLOSED pgs=3257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 --2- 172.21.15.29:0/220288133 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8e000c9ca0 0x7f8e0014e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.008+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 >> 172.21.15.29:0/220288133 conn(0x7f8e000ce9b0 msgr2=0x7f8e000ca3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:06.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.009+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 shutdown_connections 2023-10-14T18:20:06.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.009+0000 7f8e085d5640 1 -- 172.21.15.29:0/220288133 wait complete. 2023-10-14T18:20:06.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2023-10-14T18:20:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.269+0000 7fdd24f80640 1 Processor -- start 2023-10-14T18:20:06.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.269+0000 7fdd24f80640 1 -- start start 2023-10-14T18:20:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.270+0000 7fdd24f80640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd200cd180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd24f80640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2015b9d0 con 0x7fdd200ad970 2023-10-14T18:20:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd24f80640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2015bcb0 con 0x7fdd2006c040 2023-10-14T18:20:06.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd24f80640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2015b7d0 con 0x7fdd2015b320 2023-10-14T18:20:06.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd17fff640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd200ad970 0x7fdd200cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44862/0 (socket says 172.21.15.29:44862) 2023-10-14T18:20:06.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd17fff640 1 -- 172.21.15.29:0/3772680613 learned_addr learned my addr 172.21.15.29:0/3772680613 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:06.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd1ffff640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd200cd180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.271+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1584786991 0 0) 0x7fdd2015b9d0 con 0x7fdd200ad970 2023-10-14T18:20:06.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c003520 con 0x7fdd200ad970 2023-10-14T18:20:06.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1189775670 0 0) 0x7fdd0c003520 con 0x7fdd200ad970 2023-10-14T18:20:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 msgr2=0x7fdd200cd180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd200cd180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 >> v1:172.21.15.186:6789/0 conn(0x7fdd2006c040 legacy=0x7fdd2006c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2015bf90 con 0x7fdd200ad970 2023-10-14T18:20:06.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdd08001670 con 0x7fdd200ad970 2023-10-14T18:20:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 >> v1:172.21.15.29:6789/0 conn(0x7fdd200ad970 legacy=0x7fdd200cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0039a0 0x7fdd0c003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd0c0047c0 0x7fdd0c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd0c0051c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2015b7d0 con 0x7fdd0c0039a0 2023-10-14T18:20:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c00bae0 con 0x7fdd0c0047c0 2023-10-14T18:20:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd0c00bca0 con 0x7fdd2015b320 2023-10-14T18:20:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.272+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0039a0 0x7fdd0c003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd17fff640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd0c0047c0 0x7fdd0c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1ffff640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd0c0051c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1f7fe640 1 -- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 msgr2=0x7fdd0c0051c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd2015b320 0x7fdd0c0051c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1f7fe640 1 -- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd0c0047c0 msgr2=0x7fdd0c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd0c0047c0 0x7fdd0c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd1f7fe640 1 -- 172.21.15.29:0/3772680613 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd0c003520 con 0x7fdd0c0039a0 2023-10-14T18:20:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.273+0000 7fdd17fff640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd0c0047c0 0x7fdd0c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:20:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.274+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0039a0 0x7fdd0c003dc0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7fdd10001750 tx=0x7fdd0c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39dfe3f0cc5cf554 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:06.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd100052d0 con 0x7fdd0c0039a0 2023-10-14T18:20:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1002e050 con 0x7fdd0c0039a0 2023-10-14T18:20:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd1effd640 1 -- 172.21.15.29:0/3772680613 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd100035f0 con 0x7fdd0c0039a0 2023-10-14T18:20:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 -- 172.21.15.29:0/3772680613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0039a0 msgr2=0x7fdd0c003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 --2- 172.21.15.29:0/3772680613 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd0c0039a0 0x7fdd0c003dc0 secure :-1 s=READY pgs=3259 cs=0 l=1 rev1=1 crypto rx=0x7fdd10001750 tx=0x7fdd0c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd1ffff640 1 -- 172.21.15.29:0/3772680613 reap_dead start 2023-10-14T18:20:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 -- 172.21.15.29:0/3772680613 shutdown_connections 2023-10-14T18:20:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 -- 172.21.15.29:0/3772680613 >> 172.21.15.29:0/3772680613 conn(0x7fdd200ce6a0 msgr2=0x7fdd200ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 -- 172.21.15.29:0/3772680613 shutdown_connections 2023-10-14T18:20:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.275+0000 7fdd24f80640 1 -- 172.21.15.29:0/3772680613 wait complete. 2023-10-14T18:20:06.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.277+0000 7fdd24f80640 1 Processor -- start 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.277+0000 7fdd24f80640 1 -- start start 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.278+0000 7fdd24f80640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd0c0039a0 0x7fdd2014fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd24f80640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 0x7fdd200b3150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd24f80640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd24f80640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd20150190 con 0x7fdd201e49b0 2023-10-14T18:20:06.284 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd24f80640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd200c6670 con 0x7fdd200b2cd0 2023-10-14T18:20:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd24f80640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd201e5370 con 0x7fdd0c0039a0 2023-10-14T18:20:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34736/0 (socket says 172.21.15.29:34736) 2023-10-14T18:20:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 -- 172.21.15.29:0/219900558 learned_addr learned my addr 172.21.15.29:0/219900558 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:06.285 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd0c0039a0 0x7fdd2014fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd17fff640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 0x7fdd200b3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 -- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd0c0039a0 msgr2=0x7fdd2014fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd0c0039a0 0x7fdd2014fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 -- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 msgr2=0x7fdd200b3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:06.286 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 0x7fdd200b3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:06.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd1ffff640 1 -- 172.21.15.29:0/219900558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd200c6bf0 con 0x7fdd201e49b0 2023-10-14T18:20:06.287 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.279+0000 7fdd17fff640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 0x7fdd200b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd1ffff640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7fdd2015b700 tx=0x7fdd18009d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18007cf0 con 0x7fdd201e49b0 2023-10-14T18:20:06.288 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd1800f050 con 0x7fdd201e49b0 2023-10-14T18:20:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd18014650 con 0x7fdd201e49b0 2023-10-14T18:20:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd200b37e0 con 0x7fdd201e49b0 2023-10-14T18:20:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.280+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd200c6880 con 0x7fdd201e49b0 2023-10-14T18:20:06.289 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.281+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd200c6a40 con 0x7fdd201e49b0 2023-10-14T18:20:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.282+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd180147b0 con 0x7fdd201e49b0 2023-10-14T18:20:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.284+0000 7fdd1cff9640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 0x7fdd0006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.284+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(274..274 src has 1..274) v4 ==== 6728+0+0 (secure 0 0 0) 0x7fdd1808a0f0 con 0x7fdd201e49b0 2023-10-14T18:20:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.284+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 0x7fdd0006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:06.290 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.285+0000 7fdd1f7fe640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 0x7fdd0006d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd10033000 tx=0x7fdd10009030 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:06.291 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.288+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd18054ae0 con 0x7fdd201e49b0 2023-10-14T18:20:06.490 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:06.489+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fdd2004e9b0 con 0x7fdd201e49b0 2023-10-14T18:20:07.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.006+0000 7fdd1cff9640 1 -- 172.21.15.29:0/219900558 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v275) v1 ==== 181+0+0 (secure 0 0 0) 0x7fdd18058100 con 0x7fdd201e49b0 2023-10-14T18:20:07.007 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_metadata' removed 2023-10-14T18:20:07.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.009+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 msgr2=0x7fdd0006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.009+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 0x7fdd0006d480 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fdd10033000 tx=0x7fdd10009030 comp rx=0 tx=0).stop 2023-10-14T18:20:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.009+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 msgr2=0x7fdd201e4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.009+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 secure :-1 s=READY pgs=3260 cs=0 l=1 rev1=1 crypto rx=0x7fdd2015b700 tx=0x7fdd18009d20 comp rx=0 tx=0).stop 2023-10-14T18:20:07.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 shutdown_connections 2023-10-14T18:20:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd201e49b0 0x7fdd201e4e30 unknown :-1 s=CLOSED pgs=3260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd200b2cd0 0x7fdd200b3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd0006afc0 0x7fdd0006d480 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 --2- 172.21.15.29:0/219900558 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd0c0039a0 0x7fdd2014fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 >> 172.21.15.29:0/219900558 conn(0x7fdd200ce6a0 msgr2=0x7fdd2015cd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 shutdown_connections 2023-10-14T18:20:07.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.010+0000 7fdd24f80640 1 -- 172.21.15.29:0/219900558 wait complete. 2023-10-14T18:20:07.032 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2023-10-14T18:20:07.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.263+0000 7fc0d6503640 1 Processor -- start 2023-10-14T18:20:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.264+0000 7fc0d6503640 1 -- start start 2023-10-14T18:20:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d6503640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 0x7fc0d00709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.267 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d6503640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0d0070f20 0x7fc0d00ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d6503640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d0160a90 con 0x7fc0d015c9f0 2023-10-14T18:20:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d6503640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d00cdec0 con 0x7fc0d0070f20 2023-10-14T18:20:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d6503640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d00cd900 con 0x7fc0d015dde0 2023-10-14T18:20:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d5d02640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0d0070f20 0x7fc0d00ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.268 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d5d02640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fc0d0070f20 0x7fc0d00ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34750/0 (socket says 172.21.15.29:34750) 2023-10-14T18:20:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d5d02640 1 -- 172.21.15.29:0/3375250557 learned_addr learned my addr 172.21.15.29:0/3375250557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.265+0000 7fc0d5501640 1 --1- 172.21.15.29:0/3375250557 >> v1:172.21.15.29:6789/0 conn(0x7fc0d015c9f0 0x7fc0d00c8cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44866/0 (socket says 172.21.15.29:44866) 2023-10-14T18:20:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d4d00640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 0x7fc0d00709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d5d02640 1 -- 172.21.15.29:0/3375250557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 msgr2=0x7fc0d00709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:07.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d5d02640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 0x7fc0d00709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d5d02640 1 -- 172.21.15.29:0/3375250557 >> v1:172.21.15.29:6789/0 conn(0x7fc0d015c9f0 legacy=0x7fc0d00c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d5d02640 1 -- 172.21.15.29:0/3375250557 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d00cdbe0 con 0x7fc0d0070f20 2023-10-14T18:20:07.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.266+0000 7fc0d4d00640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 0x7fc0d00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.267+0000 7fc0d5d02640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d0070f20 0x7fc0d00ac890 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009e50 tx=0x7fc0c802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22f89f8855815c7b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:07.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.267+0000 7fc0c7fff640 1 -- 172.21.15.29:0/3375250557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c802cb00 con 0x7fc0d0070f20 2023-10-14T18:20:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.267+0000 7fc0c7fff640 1 -- 172.21.15.29:0/3375250557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c802cc60 con 0x7fc0d0070f20 2023-10-14T18:20:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.267+0000 7fc0c7fff640 1 -- 172.21.15.29:0/3375250557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c80343f0 con 0x7fc0d0070f20 2023-10-14T18:20:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 -- 172.21.15.29:0/3375250557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d0070f20 msgr2=0x7fc0d00ac890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d0070f20 0x7fc0d00ac890 secure :-1 s=READY pgs=3262 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009e50 tx=0x7fc0c802bf60 comp rx=0 tx=0).stop 2023-10-14T18:20:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 -- 172.21.15.29:0/3375250557 shutdown_connections 2023-10-14T18:20:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d0070f20 0x7fc0d00ac890 unknown :-1 s=CLOSED pgs=3262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 --2- 172.21.15.29:0/3375250557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d015dde0 0x7fc0d00709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 -- 172.21.15.29:0/3375250557 >> 172.21.15.29:0/3375250557 conn(0x7fc0d00ce6a0 msgr2=0x7fc0d00ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:07.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 -- 172.21.15.29:0/3375250557 shutdown_connections 2023-10-14T18:20:07.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.268+0000 7fc0d6503640 1 -- 172.21.15.29:0/3375250557 wait complete. 2023-10-14T18:20:07.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.270+0000 7fc0d6503640 1 Processor -- start 2023-10-14T18:20:07.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.270+0000 7fc0d6503640 1 -- start start 2023-10-14T18:20:07.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d0070f20 0x7fc0d0158e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0d015dde0 0x7fc0d015f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d01593b0 con 0x7fc0d015c9f0 2023-10-14T18:20:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d006acc0 con 0x7fc0d015dde0 2023-10-14T18:20:07.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d6503640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d01f5c30 con 0x7fc0d0070f20 2023-10-14T18:20:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d4d00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d4d00640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34766/0 (socket says 172.21.15.29:34766) 2023-10-14T18:20:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.271+0000 7fc0d4d00640 1 -- 172.21.15.29:0/2313523717 learned_addr learned my addr 172.21.15.29:0/2313523717 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:07.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d5501640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d0070f20 0x7fc0d0158e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d5d02640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0d015dde0 0x7fc0d015f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 -- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d0070f20 msgr2=0x7fc0d0158e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d0070f20 0x7fc0d0158e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 -- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0d015dde0 msgr2=0x7fc0d015f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0d015dde0 0x7fc0d015f410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:07.279 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 -- 172.21.15.29:0/2313523717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c8009b30 con 0x7fc0d015c9f0 2023-10-14T18:20:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0d4d00640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000c7c0 tx=0x7fc0c000cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0007820 con 0x7fc0d015c9f0 2023-10-14T18:20:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c0007980 con 0x7fc0d015c9f0 2023-10-14T18:20:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.272+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c0007d10 con 0x7fc0d015c9f0 2023-10-14T18:20:07.280 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.273+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d01f5f10 con 0x7fc0d015c9f0 2023-10-14T18:20:07.281 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.273+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d01f64d0 con 0x7fc0d015c9f0 2023-10-14T18:20:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.274+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d004e9b0 con 0x7fc0d015c9f0 2023-10-14T18:20:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.275+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc0c00113f0 con 0x7fc0d015c9f0 2023-10-14T18:20:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.276+0000 7fc0c5ffb640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 0x7fc0b406d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:07.282 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.276+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(275..275 src has 1..275) v4 ==== 6290+0+0 (secure 0 0 0) 0x7fc0c0089ac0 con 0x7fc0d015c9f0 2023-10-14T18:20:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.276+0000 7fc0d5501640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 0x7fc0b406d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.277+0000 7fc0d5501640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 0x7fc0b406d4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007fd0 tx=0x7fc0cc009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:07.283 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.280+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fc0c0054770 con 0x7fc0d015c9f0 2023-10-14T18:20:07.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:07.481+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc0d00adac0 con 0x7fc0d015c9f0 2023-10-14T18:20:08.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.306+0000 7fc0c5ffb640 1 -- 172.21.15.29:0/2313523717 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v276) v1 ==== 169+0+0 (secure 0 0 0) 0x7fc0c0057d90 con 0x7fc0d015c9f0 2023-10-14T18:20:08.308 INFO:teuthology.orchestra.run.smithi029.stderr:pool 'cephfs_data' removed 2023-10-14T18:20:08.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.313+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 msgr2=0x7fc0b406d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.313+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 0x7fc0b406d4d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc007fd0 tx=0x7fc0cc009040 comp rx=0 tx=0).stop 2023-10-14T18:20:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.313+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 msgr2=0x7fc0d006a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.313+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 secure :-1 s=READY pgs=3263 cs=0 l=1 rev1=1 crypto rx=0x7fc0c000c7c0 tx=0x7fc0c000cc80 comp rx=0 tx=0).stop 2023-10-14T18:20:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 shutdown_connections 2023-10-14T18:20:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc0d015dde0 0x7fc0d015f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc0d015c9f0 0x7fc0d006a780 unknown :-1 s=CLOSED pgs=3263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc0b406b010 0x7fc0b406d4d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 --2- 172.21.15.29:0/2313523717 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc0d0070f20 0x7fc0d0158e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 >> 172.21.15.29:0/2313523717 conn(0x7fc0d00ce6a0 msgr2=0x7fc0d00cea80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.314+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 shutdown_connections 2023-10-14T18:20:08.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.315+0000 7fc0d6503640 1 -- 172.21.15.29:0/2313523717 wait complete. 2023-10-14T18:20:08.342 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/dump_on_update 2023-10-14T18:20:08.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.570+0000 7f8c9759e640 1 Processor -- start 2023-10-14T18:20:08.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.570+0000 7f8c9759e640 1 -- start start 2023-10-14T18:20:08.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c980b1ca0 0x7f8c980d36b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f8c980b2080 0x7f8c980ce420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 0x7f8c980b24c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c980d0970 con 0x7f8c980b1ca0 2023-10-14T18:20:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c980d0ea0 con 0x7f8c980b2080 2023-10-14T18:20:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980d0bc0 con 0x7f8c980ce960 2023-10-14T18:20:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c980b1ca0 0x7f8c980d36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.574 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9659c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f8c980b1ca0 0x7f8c980d36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34782/0 (socket says 172.21.15.29:34782) 2023-10-14T18:20:08.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c9659c640 1 -- 172.21.15.29:0/3083284809 learned_addr learned my addr 172.21.15.29:0/3083284809 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:08.575 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.572+0000 7f8c95d9b640 1 --2- 172.21.15.29:0/3083284809 >> v2:172.21.15.186:3300/0 conn(0x7f8c980b2080 0x7f8c980ce420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c96d9d640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 0x7f8c980b24c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c9659c640 1 -- 172.21.15.29:0/3083284809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 msgr2=0x7f8c980b24c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 0x7f8c980b24c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c9659c640 1 -- 172.21.15.29:0/3083284809 >> v2:172.21.15.186:3300/0 conn(0x7f8c980b2080 msgr2=0x7f8c980ce420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3083284809 >> v2:172.21.15.186:3300/0 conn(0x7f8c980b2080 0x7f8c980ce420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c9659c640 1 -- 172.21.15.29:0/3083284809 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c980cd940 con 0x7f8c980b1ca0 2023-10-14T18:20:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.573+0000 7f8c96d9d640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 0x7f8c980b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.574+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b1ca0 0x7f8c980d36b0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f8c980671e0 tx=0x7f8c8802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f80f76e91e556aef server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.574+0000 7f8c9559a640 1 -- 172.21.15.29:0/3083284809 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8802cb00 con 0x7f8c980b1ca0 2023-10-14T18:20:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.574+0000 7f8c9559a640 1 -- 172.21.15.29:0/3083284809 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8802cc60 con 0x7f8c980b1ca0 2023-10-14T18:20:08.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.574+0000 7f8c9559a640 1 -- 172.21.15.29:0/3083284809 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c880343f0 con 0x7f8c980b1ca0 2023-10-14T18:20:08.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 -- 172.21.15.29:0/3083284809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b1ca0 msgr2=0x7f8c980d36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b1ca0 0x7f8c980d36b0 secure :-1 s=READY pgs=3264 cs=0 l=1 rev1=1 crypto rx=0x7f8c980671e0 tx=0x7f8c8802bf60 comp rx=0 tx=0).stop 2023-10-14T18:20:08.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 -- 172.21.15.29:0/3083284809 shutdown_connections 2023-10-14T18:20:08.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980ce960 0x7f8c980b24c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3083284809 >> v2:172.21.15.186:3300/0 conn(0x7f8c980b2080 0x7f8c980ce420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3083284809 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b1ca0 0x7f8c980d36b0 unknown :-1 s=CLOSED pgs=3264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 -- 172.21.15.29:0/3083284809 >> 172.21.15.29:0/3083284809 conn(0x7f8c980d2b90 msgr2=0x7f8c980d2fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:08.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 -- 172.21.15.29:0/3083284809 shutdown_connections 2023-10-14T18:20:08.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.575+0000 7f8c9759e640 1 -- 172.21.15.29:0/3083284809 wait complete. 2023-10-14T18:20:08.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.577+0000 7f8c9759e640 1 Processor -- start 2023-10-14T18:20:08.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.577+0000 7f8c9759e640 1 -- start start 2023-10-14T18:20:08.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 0x7f8c98150220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c980ce960 0x7f8c980cb180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980b6d30 con 0x7f8c980b2080 2023-10-14T18:20:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8c981f68f0 con 0x7f8c980ce960 2023-10-14T18:20:08.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.578+0000 7f8c9759e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c980b6fb0 con 0x7f8c980b1ca0 2023-10-14T18:20:08.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34796/0 (socket says 172.21.15.29:34796) 2023-10-14T18:20:08.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 -- 172.21.15.29:0/3982141796 learned_addr learned my addr 172.21.15.29:0/3982141796 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:08.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c96d9d640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c980ce960 0x7f8c980cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 0x7f8c98150220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 -- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 msgr2=0x7f8c98150220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 0x7f8c98150220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 -- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c980ce960 msgr2=0x7f8c980cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c980ce960 0x7f8c980cb180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.585 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c95d9b640 1 -- 172.21.15.29:0/3982141796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c88009b30 con 0x7f8c980b2080 2023-10-14T18:20:08.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.579+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 0x7f8c98150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:08.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c95d9b640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000d800 tx=0x7f8c8000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:08.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000b6c0 con 0x7f8c980b2080 2023-10-14T18:20:08.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c8000b820 con 0x7f8c980b2080 2023-10-14T18:20:08.586 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8c8000bbb0 con 0x7f8c980b2080 2023-10-14T18:20:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c980b7290 con 0x7f8c980b2080 2023-10-14T18:20:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.580+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c98150760 con 0x7f8c980b2080 2023-10-14T18:20:08.587 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.581+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c9804e9b0 con 0x7f8c980b2080 2023-10-14T18:20:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.582+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8c8000bd10 con 0x7f8c980b2080 2023-10-14T18:20:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.583+0000 7f8c8effd640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 0x7f8c7806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:08.588 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.583+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f8c80089df0 con 0x7f8c980b2080 2023-10-14T18:20:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.584+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 0x7f8c7806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:08.589 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.584+0000 7f8c9659c640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 0x7f8c7806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c88035000 tx=0x7f8c88030000 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:08.590 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.590+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8c80054cd0 con 0x7f8c980b2080 2023-10-14T18:20:08.796 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.795+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"} v 0) v1 -- 0x7f8c98073a40 con 0x7f8c980b2080 2023-10-14T18:20:08.806 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.805+0000 7f8c8effd640 1 -- 172.21.15.29:0/3982141796 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/dump_on_update"}]=0 v91) v1 ==== 115+0+0 (secure 0 0 0) 0x7f8c800582f0 con 0x7f8c980b2080 2023-10-14T18:20:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.808+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 msgr2=0x7f8c7806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.808+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 0x7f8c7806d450 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f8c88035000 tx=0x7f8c88030000 comp rx=0 tx=0).stop 2023-10-14T18:20:08.810 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.808+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 msgr2=0x7f8c980b7a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.808+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 secure :-1 s=READY pgs=3265 cs=0 l=1 rev1=1 crypto rx=0x7f8c8000d800 tx=0x7f8c8000dcc0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 shutdown_connections 2023-10-14T18:20:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8c980ce960 0x7f8c980cb180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8c980b2080 0x7f8c980b7a90 unknown :-1 s=CLOSED pgs=3265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.811 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8c7806af90 0x7f8c7806d450 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 --2- 172.21.15.29:0/3982141796 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8c980b1ca0 0x7f8c98150220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 >> 172.21.15.29:0/3982141796 conn(0x7f8c980d2b90 msgr2=0x7f8c9815fff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 shutdown_connections 2023-10-14T18:20:08.812 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:08.809+0000 7f8c9759e640 1 -- 172.21.15.29:0/3982141796 wait complete. 2023-10-14T18:20:08.830 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr/snap_schedule/allow_m_granularity 2023-10-14T18:20:09.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.035+0000 7f5c21472640 1 Processor -- start 2023-10-14T18:20:09.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.036+0000 7f5c21472640 1 -- start start 2023-10-14T18:20:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c21472640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c1c070950 0x7f5c1c167b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c21472640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0ae1a0 con 0x7f5c1c0ad7a0 2023-10-14T18:20:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c21472640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c1c0ae480 con 0x7f5c1c160c80 2023-10-14T18:20:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c21472640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c1c0ae760 con 0x7f5c1c070950 2023-10-14T18:20:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c20c71640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f5c1c070950 0x7f5c1c167b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c1b7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5c1c0ad7a0 0x7f5c1c070240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44868/0 (socket says 172.21.15.29:44868) 2023-10-14T18:20:09.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.037+0000 7f5c1b7fe640 1 -- 172.21.15.29:0/4106517699 learned_addr learned my addr 172.21.15.29:0/4106517699 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:09.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.038+0000 7f5c20c71640 1 -- 172.21.15.29:0/4106517699 >> v1:172.21.15.186:6789/0 conn(0x7f5c1c160c80 legacy=0x7f5c1c161060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.038+0000 7f5c20c71640 1 -- 172.21.15.29:0/4106517699 >> v1:172.21.15.29:6789/0 conn(0x7f5c1c0ad7a0 legacy=0x7f5c1c070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.041 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.038+0000 7f5c20c71640 1 -- 172.21.15.29:0/4106517699 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c0aea40 con 0x7f5c1c070950 2023-10-14T18:20:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.039+0000 7f5c20c71640 1 --2- 172.21.15.29:0/4106517699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c070950 0x7f5c1c167b00 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc0e6611ba688be3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.039+0000 7f5c1affd640 1 -- 172.21.15.29:0/4106517699 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14030610 con 0x7f5c1c070950 2023-10-14T18:20:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c1affd640 1 -- 172.21.15.29:0/4106517699 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14030770 con 0x7f5c1c070950 2023-10-14T18:20:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c1affd640 1 -- 172.21.15.29:0/4106517699 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14030b00 con 0x7f5c1c070950 2023-10-14T18:20:09.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 -- 172.21.15.29:0/4106517699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c070950 msgr2=0x7f5c1c167b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 --2- 172.21.15.29:0/4106517699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c070950 0x7f5c1c167b00 secure :-1 s=READY pgs=3267 cs=0 l=1 rev1=1 crypto rx=0x7f5c140097a0 tx=0x7f5c1402b960 comp rx=0 tx=0).stop 2023-10-14T18:20:09.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 -- 172.21.15.29:0/4106517699 shutdown_connections 2023-10-14T18:20:09.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 --2- 172.21.15.29:0/4106517699 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c070950 0x7f5c1c167b00 unknown :-1 s=CLOSED pgs=3267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 -- 172.21.15.29:0/4106517699 >> 172.21.15.29:0/4106517699 conn(0x7f5c1c0cc230 msgr2=0x7f5c1c0cc610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:09.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 -- 172.21.15.29:0/4106517699 shutdown_connections 2023-10-14T18:20:09.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.040+0000 7f5c21472640 1 -- 172.21.15.29:0/4106517699 wait complete. 2023-10-14T18:20:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.042+0000 7f5c21472640 1 Processor -- start 2023-10-14T18:20:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.042+0000 7f5c21472640 1 -- start start 2023-10-14T18:20:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c1c070950 0x7f5c1c1517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 0x7f5c1c06d600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c151390 con 0x7f5c1c160c80 2023-10-14T18:20:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c14ef60 con 0x7f5c1c0ad7a0 2023-10-14T18:20:09.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c21472640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c150710 con 0x7f5c1c070950 2023-10-14T18:20:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c20c71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c20c71640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34814/0 (socket says 172.21.15.29:34814) 2023-10-14T18:20:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.043+0000 7f5c20c71640 1 -- 172.21.15.29:0/1624434865 learned_addr learned my addr 172.21.15.29:0/1624434865 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c1b7fe640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 0x7f5c1c06d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 -- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c1c070950 msgr2=0x7f5c1c1517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c1c070950 0x7f5c1c1517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 -- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 msgr2=0x7f5c1c06d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 0x7f5c1c06d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 -- 172.21.15.29:0/1624434865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c14009400 con 0x7f5c1c160c80 2023-10-14T18:20:09.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c1b7fe640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 0x7f5c1c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.044+0000 7f5c20c71640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f5c1402b9a0 tx=0x7f5c14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.045+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c1403e040 con 0x7f5c1c160c80 2023-10-14T18:20:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.045+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c14039cb0 con 0x7f5c1c160c80 2023-10-14T18:20:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.045+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c14044630 con 0x7f5c1c160c80 2023-10-14T18:20:09.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.045+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c1c1509f0 con 0x7f5c1c160c80 2023-10-14T18:20:09.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.045+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c1c150fb0 con 0x7f5c1c160c80 2023-10-14T18:20:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.046+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c1c1514d0 con 0x7f5c1c160c80 2023-10-14T18:20:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.047+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5c1404d3b0 con 0x7f5c1c160c80 2023-10-14T18:20:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.048+0000 7f5c18ff9640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 0x7f5bfc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.048+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f5c140ac050 con 0x7f5c1c160c80 2023-10-14T18:20:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.048+0000 7f5c1bfff640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 0x7f5bfc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.049+0000 7f5c1bfff640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 0x7f5bfc06d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0012e0 tx=0x7f5c0c007d30 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.053 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.053+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5c14076fe0 con 0x7f5c1c160c80 2023-10-14T18:20:09.253 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.252+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"} v 0) v1 -- 0x7f5c1c04e9b0 con 0x7f5c1c160c80 2023-10-14T18:20:09.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.257+0000 7f5c18ff9640 1 -- 172.21.15.29:0/1624434865 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mgr", "name": "mgr/snap_schedule/allow_m_granularity"}]=0 v92) v1 ==== 120+0+0 (secure 0 0 0) 0x7f5c14006e10 con 0x7f5c1c160c80 2023-10-14T18:20:09.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.260+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 msgr2=0x7f5bfc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.260+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 0x7f5bfc06d450 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c0012e0 tx=0x7f5c0c007d30 comp rx=0 tx=0).stop 2023-10-14T18:20:09.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.260+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 msgr2=0x7f5c1c14fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.260+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 secure :-1 s=READY pgs=3268 cs=0 l=1 rev1=1 crypto rx=0x7f5c1402b9a0 tx=0x7f5c14004730 comp rx=0 tx=0).stop 2023-10-14T18:20:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 shutdown_connections 2023-10-14T18:20:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5c1c160c80 0x7f5c1c14fba0 unknown :-1 s=CLOSED pgs=3268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5c1c0ad7a0 0x7f5c1c06d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bfc06af90 0x7f5bfc06d450 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 --2- 172.21.15.29:0/1624434865 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5c1c070950 0x7f5c1c1517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 >> 172.21.15.29:0/1624434865 conn(0x7f5c1c0cc230 msgr2=0x7f5c1c15c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 shutdown_connections 2023-10-14T18:20:09.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.261+0000 7f5c21472640 1 -- 172.21.15.29:0/1624434865 wait complete. 2023-10-14T18:20:09.283 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2023-10-14T18:20:09.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.503+0000 7f13ed4d1640 1 Processor -- start 2023-10-14T18:20:09.504 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.503+0000 7f13ed4d1640 1 -- start start 2023-10-14T18:20:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.504+0000 7f13ed4d1640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13e8065ab0 0x7f13e80c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13ed4d1640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f13e80ce280 0x7f13e80ad200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13ed4d1640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e80acd70 con 0x7f13e806d220 2023-10-14T18:20:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13ed4d1640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e80ac4a0 con 0x7f13e8065ab0 2023-10-14T18:20:09.506 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13ed4d1640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e8065f40 con 0x7f13e80ce280 2023-10-14T18:20:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13e8065ab0 0x7f13e80c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f13e8065ab0 0x7f13e80c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34826/0 (socket says 172.21.15.29:34826) 2023-10-14T18:20:09.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 -- 172.21.15.29:0/68731476 learned_addr learned my addr 172.21.15.29:0/68731476 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 -- 172.21.15.29:0/68731476 >> v2:172.21.15.186:3300/0 conn(0x7f13e80ce280 msgr2=0x7f13e80ad200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 --2- 172.21.15.29:0/68731476 >> v2:172.21.15.186:3300/0 conn(0x7f13e80ce280 0x7f13e80ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 -- 172.21.15.29:0/68731476 >> v1:172.21.15.186:6789/0 conn(0x7f13e806d220 legacy=0x7f13e80cdb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.505+0000 7f13e77fe640 1 -- 172.21.15.29:0/68731476 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e80ac720 con 0x7f13e8065ab0 2023-10-14T18:20:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.507+0000 7f13e77fe640 1 --2- 172.21.15.29:0/68731476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e8065ab0 0x7f13e80c95e0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f13e8071fd0 tx=0x7f13d002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4af71c469ee7935f server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.507+0000 7f13e6ffd640 1 -- 172.21.15.29:0/68731476 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d00317e0 con 0x7f13e8065ab0 2023-10-14T18:20:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.507+0000 7f13e6ffd640 1 -- 172.21.15.29:0/68731476 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0031940 con 0x7f13e8065ab0 2023-10-14T18:20:09.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.507+0000 7f13e6ffd640 1 -- 172.21.15.29:0/68731476 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0031cd0 con 0x7f13e8065ab0 2023-10-14T18:20:09.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 -- 172.21.15.29:0/68731476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e8065ab0 msgr2=0x7f13e80c95e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/68731476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e8065ab0 0x7f13e80c95e0 secure :-1 s=READY pgs=3269 cs=0 l=1 rev1=1 crypto rx=0x7f13e8071fd0 tx=0x7f13d002b880 comp rx=0 tx=0).stop 2023-10-14T18:20:09.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 -- 172.21.15.29:0/68731476 shutdown_connections 2023-10-14T18:20:09.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/68731476 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e8065ab0 0x7f13e80c95e0 unknown :-1 s=CLOSED pgs=3269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/68731476 >> v2:172.21.15.186:3300/0 conn(0x7f13e80ce280 0x7f13e80ad200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 -- 172.21.15.29:0/68731476 >> 172.21.15.29:0/68731476 conn(0x7f13e80b0a80 msgr2=0x7f13e80b0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 -- 172.21.15.29:0/68731476 shutdown_connections 2023-10-14T18:20:09.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.508+0000 7f13ed4d1640 1 -- 172.21.15.29:0/68731476 wait complete. 2023-10-14T18:20:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.510+0000 7f13ed4d1640 1 Processor -- start 2023-10-14T18:20:09.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.511+0000 7f13ed4d1640 1 -- start start 2023-10-14T18:20:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13e806d220 0x7f13e806c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13e80b3d80 0x7f13e80b4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.515 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e80b4750 con 0x7f13e80ce280 2023-10-14T18:20:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e806bb40 con 0x7f13e806d220 2023-10-14T18:20:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13ed4d1640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e814d730 con 0x7f13e80b3d80 2023-10-14T18:20:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13e77fe640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34842/0 (socket says 172.21.15.29:34842) 2023-10-14T18:20:09.516 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13e77fe640 1 -- 172.21.15.29:0/3277589064 learned_addr learned my addr 172.21.15.29:0/3277589064 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.512+0000 7f13e7fff640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13e806d220 0x7f13e806c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13eccd0640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13e80b3d80 0x7f13e80b4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.517 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 -- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13e80b3d80 msgr2=0x7f13e80b4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13e80b3d80 0x7f13e80b4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 -- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13e806d220 msgr2=0x7f13e806c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.518 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13e806d220 0x7f13e806c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 -- 172.21.15.29:0/3277589064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d00094d0 con 0x7f13e80ce280 2023-10-14T18:20:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e77fe640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f13d00090b0 tx=0x7f13d00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d0032980 con 0x7f13e80ce280 2023-10-14T18:20:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0032b00 con 0x7f13e80ce280 2023-10-14T18:20:09.519 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f13d00453f0 con 0x7f13e80ce280 2023-10-14T18:20:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.513+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e806be00 con 0x7f13e80ce280 2023-10-14T18:20:09.520 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.514+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e806c3c0 con 0x7f13e80ce280 2023-10-14T18:20:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.515+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e8071350 con 0x7f13e80ce280 2023-10-14T18:20:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.516+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f13d004d3f0 con 0x7f13e80ce280 2023-10-14T18:20:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.517+0000 7f13e4ff9640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 0x7f13c806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.521 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.517+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f13d00ab7e0 con 0x7f13e80ce280 2023-10-14T18:20:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.517+0000 7f13e7fff640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 0x7f13c806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.518+0000 7f13e7fff640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 0x7f13c806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13d8001e30 tx=0x7f13d8009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.522 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.521+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f13d0076670 con 0x7f13e80ce280 2023-10-14T18:20:09.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.716+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f13e806c5b0 con 0x7f13e80ce280 2023-10-14T18:20:09.725 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.723+0000 7f13e4ff9640 1 -- 172.21.15.29:0/3277589064 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v93) v1 ==== 104+0+0 (secure 0 0 0) 0x7f13d0079c90 con 0x7f13e80ce280 2023-10-14T18:20:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.725+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 msgr2=0x7f13c806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.725+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 0x7f13c806d450 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f13d8001e30 tx=0x7f13d8009040 comp rx=0 tx=0).stop 2023-10-14T18:20:09.726 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.725+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 msgr2=0x7f13e8149ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.725+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 secure :-1 s=READY pgs=3270 cs=0 l=1 rev1=1 crypto rx=0x7f13d00090b0 tx=0x7f13d00046c0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 shutdown_connections 2023-10-14T18:20:09.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f13c806af90 0x7f13c806d450 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f13e80b3d80 0x7f13e80b4210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f13e80ce280 0x7f13e8149ee0 unknown :-1 s=CLOSED pgs=3270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 --2- 172.21.15.29:0/3277589064 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f13e806d220 0x7f13e806c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 >> 172.21.15.29:0/3277589064 conn(0x7f13e80b0a80 msgr2=0x7f13e8099a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:09.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 shutdown_connections 2023-10-14T18:20:09.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.726+0000 7f13ed4d1640 1 -- 172.21.15.29:0/3277589064 wait complete. 2023-10-14T18:20:09.745 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name' 2023-10-14T18:20:09.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.972+0000 7fcd722df640 1 Processor -- start 2023-10-14T18:20:09.973 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.972+0000 7fcd722df640 1 -- start start 2023-10-14T18:20:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.974+0000 7fcd722df640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fcd6c159a10 0x7fcd6c06d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.974+0000 7fcd722df640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fcd6c06dc70 0x7fcd6c0cdc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.975 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.974+0000 7fcd722df640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd6c06d2a0 con 0x7fcd6c070660 2023-10-14T18:20:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.974+0000 7fcd722df640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c0713e0 con 0x7fcd6c06dc70 2023-10-14T18:20:09.976 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.974+0000 7fcd722df640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd6c06d000 con 0x7fcd6c159a10 2023-10-14T18:20:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd71ade640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fcd6c070660 0x7fcd6c0c95f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:44870/0 (socket says 172.21.15.29:44870) 2023-10-14T18:20:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd71ade640 1 -- 172.21.15.29:0/759978218 learned_addr learned my addr 172.21.15.29:0/759978218 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:09.977 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd70adc640 1 --2- 172.21.15.29:0/759978218 >> v2:172.21.15.29:3300/0 conn(0x7fcd6c06dc70 0x7fcd6c0cdc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd712dd640 1 --2- 172.21.15.29:0/759978218 >> v2:172.21.15.186:3300/0 conn(0x7fcd6c159a10 0x7fcd6c06d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd63fff640 1 -- 172.21.15.29:0/759978218 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448370058 0 0) 0x7fcd6c06d2a0 con 0x7fcd6c070660 2023-10-14T18:20:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd63fff640 1 -- 172.21.15.29:0/759978218 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd58003520 con 0x7fcd6c070660 2023-10-14T18:20:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd70adc640 1 -- 172.21.15.29:0/759978218 >> v2:172.21.15.186:3300/0 conn(0x7fcd6c159a10 msgr2=0x7fcd6c06d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:09.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd70adc640 1 --2- 172.21.15.29:0/759978218 >> v2:172.21.15.186:3300/0 conn(0x7fcd6c159a10 0x7fcd6c06d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd70adc640 1 -- 172.21.15.29:0/759978218 >> v1:172.21.15.29:6789/0 conn(0x7fcd6c070660 legacy=0x7fcd6c0c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.975+0000 7fcd70adc640 1 -- 172.21.15.29:0/759978218 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c070b40 con 0x7fcd6c06dc70 2023-10-14T18:20:09.979 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd70adc640 1 --2- 172.21.15.29:0/759978218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0cdc80 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c072350 tx=0x7fcd5c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f34f180bce2ee32e server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd63fff640 1 -- 172.21.15.29:0/759978218 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c031830 con 0x7fcd6c06dc70 2023-10-14T18:20:09.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd63fff640 1 -- 172.21.15.29:0/759978218 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c031990 con 0x7fcd6c06dc70 2023-10-14T18:20:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd63fff640 1 -- 172.21.15.29:0/759978218 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c031d20 con 0x7fcd6c06dc70 2023-10-14T18:20:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd722df640 1 -- 172.21.15.29:0/759978218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c06dc70 msgr2=0x7fcd6c0cdc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.977+0000 7fcd722df640 1 --2- 172.21.15.29:0/759978218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0cdc80 secure :-1 s=READY pgs=3272 cs=0 l=1 rev1=1 crypto rx=0x7fcd6c072350 tx=0x7fcd5c02b8d0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 -- 172.21.15.29:0/759978218 shutdown_connections 2023-10-14T18:20:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 --2- 172.21.15.29:0/759978218 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0cdc80 unknown :-1 s=CLOSED pgs=3272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 --2- 172.21.15.29:0/759978218 >> v2:172.21.15.186:3300/0 conn(0x7fcd6c159a10 0x7fcd6c06d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 -- 172.21.15.29:0/759978218 >> 172.21.15.29:0/759978218 conn(0x7fcd6c069da0 msgr2=0x7fcd6c06a180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 -- 172.21.15.29:0/759978218 shutdown_connections 2023-10-14T18:20:09.982 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.978+0000 7fcd722df640 1 -- 172.21.15.29:0/759978218 wait complete. 2023-10-14T18:20:09.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.979+0000 7fcd722df640 1 Processor -- start 2023-10-14T18:20:09.983 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 -- start start 2023-10-14T18:20:09.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0c4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd6c159a10 0x7fcd6c0c5920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c141ed0 con 0x7fcd6c070660 2023-10-14T18:20:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c15d1b0 con 0x7fcd6c06dc70 2023-10-14T18:20:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.980+0000 7fcd722df640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd6c141530 con 0x7fcd6c159a10 2023-10-14T18:20:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:34874/0 (socket says 172.21.15.29:34874) 2023-10-14T18:20:09.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 -- 172.21.15.29:0/2659748634 learned_addr learned my addr 172.21.15.29:0/2659748634 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd712dd640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0c4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd71ade640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd6c159a10 0x7fcd6c0c5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 -- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd6c159a10 msgr2=0x7fcd6c0c5920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd6c159a10 0x7fcd6c0c5920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 -- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd6c06dc70 msgr2=0x7fcd6c0c4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0c4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.981+0000 7fcd70adc640 1 -- 172.21.15.29:0/2659748634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd5c009400 con 0x7fcd6c070660 2023-10-14T18:20:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd70adc640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009890 tx=0x7fcd5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c03a8f0 con 0x7fcd6c070660 2023-10-14T18:20:09.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd5c03aa70 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c03ae20 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd6c141810 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.982+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd6c0c4f60 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.983+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd6c142010 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.984+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fcd5c0443f0 con 0x7fcd6c070660 2023-10-14T18:20:09.991 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.985+0000 7fcd61ffb640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 0x7fcd5006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.985+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fcd5c0abae0 con 0x7fcd6c070660 2023-10-14T18:20:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.985+0000 7fcd712dd640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 0x7fcd5006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:09.992 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.986+0000 7fcd712dd640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 0x7fcd5006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006a90 tx=0x7fcd68006a20 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:09.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:09.990+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fcd6c142010 con 0x7fcd6c070660 2023-10-14T18:20:10.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:10.321+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]} v 0) v1 -- 0x7fcd6c142010 con 0x7fcd6c070660 2023-10-14T18:20:11.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.314+0000 7fcd61ffb640 1 -- 172.21.15.29:0/2659748634 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_snap_schedules.TestSnapSchedulesSnapdir.test_snap_dir_name"]}]=0 v3329) v1 ==== 155+0+0 (secure 0 0 0) 0x7fcd6c142010 con 0x7fcd6c070660 2023-10-14T18:20:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 msgr2=0x7fcd5006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 0x7fcd5006d450 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcd68006a90 tx=0x7fcd68006a20 comp rx=0 tx=0).stop 2023-10-14T18:20:11.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 msgr2=0x7fcd6c0c53e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 secure :-1 s=READY pgs=3273 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c009890 tx=0x7fcd5c0046c0 comp rx=0 tx=0).stop 2023-10-14T18:20:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 shutdown_connections 2023-10-14T18:20:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fcd5006af90 0x7fcd5006d450 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fcd6c159a10 0x7fcd6c0c5920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:11.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fcd6c070660 0x7fcd6c0c53e0 unknown :-1 s=CLOSED pgs=3273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 --2- 172.21.15.29:0/2659748634 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fcd6c06dc70 0x7fcd6c0c4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.318+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 >> 172.21.15.29:0/2659748634 conn(0x7fcd6c069da0 msgr2=0x7fcd6c065ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:11.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.319+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 shutdown_connections 2023-10-14T18:20:11.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:11.319+0000 7fcd722df640 1 -- 172.21.15.29:0/2659748634 wait complete. 2023-10-14T18:20:11.341 INFO:tasks.cephfs_test_runner:Test the correctness of snap directory name ... ok 2023-10-14T18:20:11.342 INFO:tasks.cephfs_test_runner: 2023-10-14T18:20:11.342 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2023-10-14T18:20:11.342 INFO:tasks.cephfs_test_runner:Ran 14 tests in 5945.201s 2023-10-14T18:20:11.343 INFO:tasks.cephfs_test_runner: 2023-10-14T18:20:11.343 INFO:tasks.cephfs_test_runner:OK 2023-10-14T18:20:11.343 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2023-10-14T18:20:11.355 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2023-10-14T18:20:11.370 INFO:tasks.kclient:Unmounting kernel clients... 2023-10-14T18:20:11.371 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:20:11.371 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:20:11.434 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:20:11.434 DEBUG:teuthology.orchestra.run.smithi029:> set -ex 2023-10-14T18:20:11.434 DEBUG:teuthology.orchestra.run.smithi029:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:20:11.492 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4055074,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3250028k,nr_inodes=812507,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6500064k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2023-10-14T18:20:11.492 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:20:11.493 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:20:11.556 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:20:11.556 DEBUG:teuthology.orchestra.run.smithi186:> set -ex 2023-10-14T18:20:11.556 DEBUG:teuthology.orchestra.run.smithi186:> dd if=/proc/self/mounts of=/dev/stdout 2023-10-14T18:20:11.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4054301,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=27,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249412k,nr_inodes=812353,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498828k,nr_inodes=819200,mode=755 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2023-10-14T18:20:11.611 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:11.612 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:20:11.675 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2023-10-14T18:20:11.675 INFO:teuthology.orchestra.run.smithi029.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2023-10-14T18:20:11.676 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:11.677 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2023-10-14T18:20:11.780 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:11.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2023-10-14T18:20:11.881 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:11.882 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-14T18:20:12.023 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.023 DEBUG:teuthology.orchestra.run.smithi029:> ip netns list 2023-10-14T18:20:12.080 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.080 DEBUG:teuthology.orchestra.run.smithi029:> sudo ip link delete ceph-brx 2023-10-14T18:20:12.165 INFO:teuthology.orchestra.run.smithi029.stderr:Cannot find device "ceph-brx" 2023-10-14T18:20:12.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:20:12.169 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.169 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T18:20:12.232 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2023-10-14T18:20:12.232 INFO:teuthology.orchestra.run.smithi186.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2023-10-14T18:20:12.233 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.233 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2023-10-14T18:20:12.331 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.332 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2023-10-14T18:20:12.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.418 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2023-10-14T18:20:12.580 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.581 DEBUG:teuthology.orchestra.run.smithi186:> ip netns list 2023-10-14T18:20:12.631 INFO:teuthology.orchestra.run:Running command with timeout 300 2023-10-14T18:20:12.632 DEBUG:teuthology.orchestra.run.smithi186:> sudo ip link delete ceph-brx 2023-10-14T18:20:12.720 INFO:teuthology.orchestra.run.smithi186.stderr:Cannot find device "ceph-brx" 2023-10-14T18:20:12.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:20:12.724 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-10-14T18:20:12.852 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-10-14T18:20:12.853 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:20:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.092+0000 7f44655d5640 1 Processor -- start 2023-10-14T18:20:13.093 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.092+0000 7f44655d5640 1 -- start start 2023-10-14T18:20:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f44655d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f446015aaa0 0x7f446015e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.094 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f44655d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f446015ae80 0x7f44600cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f44655d5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f446015ed60 con 0x7f44600ccaa0 2023-10-14T18:20:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f44655d5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f446006b810 con 0x7f446015aaa0 2023-10-14T18:20:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f44655d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600707e0 con 0x7f446015ae80 2023-10-14T18:20:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f445effd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f446015aaa0 0x7f446015e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.095 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f445f7fe640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f44600ccaa0 0x7f4460070f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55604/0 (socket says 172.21.15.29:55604) 2023-10-14T18:20:13.096 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.093+0000 7f445f7fe640 1 -- 172.21.15.29:0/2236850204 learned_addr learned my addr 172.21.15.29:0/2236850204 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:13.097 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445e7fc640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f446015ae80 0x7f44600cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445effd640 1 -- 172.21.15.29:0/2236850204 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f446015ae80 msgr2=0x7f44600cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445effd640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f446015ae80 0x7f44600cc560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445effd640 1 -- 172.21.15.29:0/2236850204 >> v1:172.21.15.29:6789/0 conn(0x7f44600ccaa0 legacy=0x7f4460070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445effd640 1 -- 172.21.15.29:0/2236850204 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f446015ef90 con 0x7f446015aaa0 2023-10-14T18:20:13.098 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.094+0000 7f445dffb640 1 -- 172.21.15.29:0/2236850204 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 524209732 0 0) 0x7f446015ed60 con 0x7f44600ccaa0 2023-10-14T18:20:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.095+0000 7f445effd640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f446015e820 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445002beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7dc7e0a9a3f2c08 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f445dffb640 1 -- 172.21.15.29:0/2236850204 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f445002cc60 con 0x7f446015aaa0 2023-10-14T18:20:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f445dffb640 1 -- 172.21.15.29:0/2236850204 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f445002cdc0 con 0x7f446015aaa0 2023-10-14T18:20:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f445dffb640 1 -- 172.21.15.29:0/2236850204 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4450034660 con 0x7f446015aaa0 2023-10-14T18:20:13.099 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 -- 172.21.15.29:0/2236850204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 msgr2=0x7f446015e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f446015e820 secure :-1 s=READY pgs=3275 cs=0 l=1 rev1=1 crypto rx=0x7f4450009d90 tx=0x7f445002beb0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 -- 172.21.15.29:0/2236850204 shutdown_connections 2023-10-14T18:20:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f446015ae80 0x7f44600cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 --2- 172.21.15.29:0/2236850204 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f446015e820 unknown :-1 s=CLOSED pgs=3275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.100 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.096+0000 7f44655d5640 1 -- 172.21.15.29:0/2236850204 >> 172.21.15.29:0/2236850204 conn(0x7f44600cb410 msgr2=0x7f44600cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:13.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.097+0000 7f44655d5640 1 -- 172.21.15.29:0/2236850204 shutdown_connections 2023-10-14T18:20:13.101 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.097+0000 7f44655d5640 1 -- 172.21.15.29:0/2236850204 wait complete. 2023-10-14T18:20:13.102 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.098+0000 7f44655d5640 1 Processor -- start 2023-10-14T18:20:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 -- start start 2023-10-14T18:20:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44600ccaa0 0x7f44600c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446015ae80 0x7f446015d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.103 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600c3370 con 0x7f446015aaa0 2023-10-14T18:20:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44600c24a0 con 0x7f446015ae80 2023-10-14T18:20:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.099+0000 7f44655d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44600c26f0 con 0x7f44600ccaa0 2023-10-14T18:20:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43468/0 (socket says 172.21.15.29:43468) 2023-10-14T18:20:13.104 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 -- 172.21.15.29:0/1779469032 learned_addr learned my addr 172.21.15.29:0/1779469032 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:13.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445effd640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44600ccaa0 0x7f44600c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 -- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44600ccaa0 msgr2=0x7f44600c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44600ccaa0 0x7f44600c7da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 -- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446015ae80 msgr2=0x7f446015d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.105 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446015ae80 0x7f446015d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.100+0000 7f445e7fc640 1 -- 172.21.15.29:0/1779469032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4450009a70 con 0x7f446015aaa0 2023-10-14T18:20:13.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f445e7fc640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7f0 tx=0x7f444800ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448004220 con 0x7f446015aaa0 2023-10-14T18:20:13.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44480043a0 con 0x7f446015aaa0 2023-10-14T18:20:13.106 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4448007860 con 0x7f446015aaa0 2023-10-14T18:20:13.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44600c29d0 con 0x7f446015aaa0 2023-10-14T18:20:13.107 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.101+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44600c2f90 con 0x7f446015aaa0 2023-10-14T18:20:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.102+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44601f3be0 con 0x7f446015aaa0 2023-10-14T18:20:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.103+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4448007bd0 con 0x7f446015aaa0 2023-10-14T18:20:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.104+0000 7f44577fe640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 0x7f444006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.108 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.104+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f4448089a20 con 0x7f446015aaa0 2023-10-14T18:20:13.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.104+0000 7f445effd640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 0x7f444006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.109 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.105+0000 7f445effd640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 0x7f444006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f4450030000 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.112 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.111+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f44601f3be0 con 0x7f446015aaa0 2023-10-14T18:20:13.307 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.306+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f446015f550 con 0x7f444006af90 2023-10-14T18:20:13.311 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.310+0000 7f44577fe640 1 -- 172.21.15.29:0/1779469032 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48651 (secure 0 0 0) 0x7f4450034540 con 0x7f444006af90 2023-10-14T18:20:13.312 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:13.313 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:20:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.313+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 msgr2=0x7f444006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.313+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 0x7f444006d450 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4450036000 tx=0x7f4450030000 comp rx=0 tx=0).stop 2023-10-14T18:20:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.313+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 msgr2=0x7f44600c82e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.313+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 secure :-1 s=READY pgs=3276 cs=0 l=1 rev1=1 crypto rx=0x7f444800c7f0 tx=0x7f444800ccb0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.313+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 shutdown_connections 2023-10-14T18:20:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f446015ae80 0x7f446015d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f446015aaa0 0x7f44600c82e0 unknown :-1 s=CLOSED pgs=3276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f444006af90 0x7f444006d450 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 --2- 172.21.15.29:0/1779469032 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f44600ccaa0 0x7f44600c7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 >> 172.21.15.29:0/1779469032 conn(0x7f44600cb410 msgr2=0x7f44600b0b70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:13.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 shutdown_connections 2023-10-14T18:20:13.317 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.314+0000 7f44655d5640 1 -- 172.21.15.29:0/1779469032 wait complete. 2023-10-14T18:20:13.333 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-14T18:20:13.261544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":462456,"kb_used_data":28424,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754512264,"statfs":{"total":773094113280,"available":772620558336,"internally_reserved":0,"allocated":29106176,"data_stored":15157528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-735638,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-321,"num_read_kb":-165,"num_write":-342,"num_write_kb":-2910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003420"},"pg_stats":[{"pgid":"1.0","version":"273'403","reported_seq":1450,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-14T18:20:07.195920+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:20:07.195920+0000","last_peered":"2023-10-14T18:20:07.195920+0000","last_clean":"2023-10-14T18:20:07.195920+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:20:07.195920+0000","last_undegraded":"2023-10-14T18:20:07.195920+0000","last_fullsized":"2023-10-14T18:20:07.195920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57380,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314460,"statfs":{"total":96636764160,"available":96578007040,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.022},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":0.98699999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":1.127},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.236}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.89300000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.4530000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.2}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.302},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.5229999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.879,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.988,"5min":1.329,"15min":1.2549999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.198,"15min":1.2410000000000001},"min":{"1min":0.74299999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.569},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3520000000000001,"15min":1.3660000000000001},"min":{"1min":0.92900000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.1890000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.397,"15min":1.3859999999999999},"min":{"1min":0.79000000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.4179999999999999,"15min":1.427},"min":{"1min":1.028,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.4690000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.88800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.2869999999999999},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5409999999999999,"15min":1.4830000000000001},"min":{"1min":0.95099999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":14.805999999999999,"5min":14.805999999999999,"15min":14.805999999999999},"last":1.4159999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.4399999999999999,"15min":1.4039999999999999},"min":{"1min":1.119,"5min":0.85899999999999999,"15min":0.75800000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.4810000000000001,"15min":1.456},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":1.026}]},{"osd":5,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.423,"15min":1.423},"min":{"1min":0.88,"5min":0.624,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.6040000000000001,"15min":1.5960000000000001},"min":{"1min":0.97899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":0.96299999999999997}]},{"osd":7,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":2.468,"5min":1.796,"15min":1.6779999999999999},"min":{"1min":1.1870000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":16.541,"5min":16.541,"15min":16.541},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":1.131,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.0059999999999998,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.329}]}]},{"osd":5,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.147,"15min":1.1890000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6140000000000001,"5min":2.052,"15min":2.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.214},"min":{"1min":0.70099999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5960000000000001,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":0.88100000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.69899999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.754,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.26,"15min":1.286},"min":{"1min":0.81699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6599999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.99299999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.175,"15min":1.222},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7330000000000001,"5min":1.764,"15min":1.9339999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.389},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.923},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.363,"15min":1.4390000000000001},"min":{"1min":0.91700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8149999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.2},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3999999999999999,"15min":1.4139999999999999},"min":{"1min":1.006,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.74,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.042}]},{"osd":4,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.413,"15min":1.3759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8680000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4610000000000001,"15min":1.4079999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.61799999999999999},"max":{"1min":1.712,"5min":2.516,"15min":2.516},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64900000000000002},"max":{"1min":1.9359999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.6259999999999999,"15min":1.627},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.98,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.381}]},{"osd":7,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5600000000000001,"15min":1.5740000000000001},"min":{"1min":1.1160000000000001,"5min":0.95499999999999996,"15min":0.68600000000000005},"max":{"1min":1.8080000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.504},{"interface":"front","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.5629999999999999},"min":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8460000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.4530000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.181},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.458}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":1.028},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.306}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":1.073},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":1.2210000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.6899999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.345},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.6299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.57399999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.417,"5min":1.4870000000000001,"15min":1.5660000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.0940000000000001,"15min":1.179},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.9970000000000001},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.92900000000000005,"15min":0.998},"min":{"1min":0.52400000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.919},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0860000000000001,"15min":1.1819999999999999},"min":{"1min":0.61299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5089999999999999,"5min":1.742,"15min":1.8560000000000001},"last":1.038}]},{"osd":2,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2609999999999999,"15min":1.401},"min":{"1min":0.50900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9099999999999999,"5min":1.9970000000000001,"15min":2.032},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.25,"15min":1.3560000000000001},"min":{"1min":0.68899999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.915,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3560000000000001,"15min":1.4299999999999999},"min":{"1min":0.85399999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.105,"5min":2.105,"15min":2.153},"last":1.629},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4430000000000001,"15min":1.478},"min":{"1min":1.034,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.9870000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.802}]},{"osd":5,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":1.0800000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.758,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.202},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9239999999999999,"5min":2.1360000000000001,"15min":2.891},"last":1.4670000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.5589999999999999,"15min":1.6259999999999999},"min":{"1min":0.88700000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.2120000000000002,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.2230000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1419999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.8819999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":0.97399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.452,"5min":3.153,"15min":3.153},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.6060000000000001,"15min":1.6759999999999999},"min":{"1min":1.052,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0339999999999998,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.5589999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56800,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":1.0309999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2290000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.4139999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.532},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.496}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.631}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.47}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.7270000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38},"max":{"1min":1.4570000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.0920000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3600000000000001,"5min":2.847,"15min":2.847},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.996,"15min":1.042},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.492,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1659999999999999,"15min":1.2450000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6919999999999999,"5min":1.996,"15min":2.6859999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2350000000000001,"15min":1.3540000000000001},"min":{"1min":0.60199999999999998,"5min":0.52700000000000002,"15min":0.44700000000000001},"max":{"1min":1.879,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1970000000000001,"15min":1.2869999999999999},"min":{"1min":0.54400000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5649999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.0389999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4119999999999999,"15min":1.454},"min":{"1min":0.748,"5min":0.748,"15min":0.73899999999999999},"max":{"1min":2.0329999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4259999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.64800000000000002},"max":{"1min":2.2010000000000001,"5min":2.218,"15min":2.899},"last":0.95099999999999996}]},{"osd":5,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.405,"15min":1.4630000000000001},"min":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.857,"5min":2.2519999999999998,"15min":3.004},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.502,"15min":1.5569999999999999},"min":{"1min":0.94999999999999996,"5min":0.90400000000000003,"15min":0.73199999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.5129999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5069999999999999,"15min":1.597},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.367,"5min":2.8999999999999999,"15min":2.931},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.615,"15min":1.6220000000000001},"min":{"1min":0.90200000000000002,"5min":0.871,"15min":0.871},"max":{"1min":2.2509999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.5289999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.6060000000000001,"15min":1.6830000000000001},"min":{"1min":0.83799999999999997,"5min":0.82299999999999995,"15min":0.57899999999999996},"max":{"1min":1.9830000000000001,"5min":3.363,"15min":3.363},"last":1.488},{"interface":"front","average":{"1min":1.55,"5min":1.6259999999999999,"15min":1.6990000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.1629999999999998,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.6259999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53220,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318620,"statfs":{"total":96636764160,"available":96582266880,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.3899999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.609}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.8129999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.679},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.542}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.304},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.722}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.8979999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.4990000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.242},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.7769999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:20:13.335 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:20:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.561+0000 7fa93442d640 1 Processor -- start 2023-10-14T18:20:13.562 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.561+0000 7fa93442d640 1 -- start start 2023-10-14T18:20:13.563 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.562+0000 7fa93442d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa92c0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.562+0000 7fa93442d640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fa92c15d880 0x7fa92c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.562+0000 7fa93442d640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa92c06f7f0 con 0x7fa92c159b20 2023-10-14T18:20:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.562+0000 7fa93442d640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa92c06c5e0 con 0x7fa92c15d880 2023-10-14T18:20:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.562+0000 7fa93442d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06c8c0 con 0x7fa92c159f00 2023-10-14T18:20:13.564 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9321c9640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fa92c159b20 0x7fa92c06bbe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55614/0 (socket says 172.21.15.29:55614) 2023-10-14T18:20:13.565 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9321c9640 1 -- 172.21.15.29:0/3559208433 learned_addr learned my addr 172.21.15.29:0/3559208433 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9319c8640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa92c0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69121324 0 0) 0x7fa92c06f7f0 con 0x7fa92c159b20 2023-10-14T18:20:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa918003520 con 0x7fa92c159b20 2023-10-14T18:20:13.567 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.563+0000 7fa9329ca640 1 --2- 172.21.15.29:0/3559208433 >> v2:172.21.15.186:3300/0 conn(0x7fa92c15d880 0x7fa92c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2766160612 0 0) 0x7fa918003520 con 0x7fa92c159b20 2023-10-14T18:20:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 msgr2=0x7fa92c0b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa92c0b0410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 >> v2:172.21.15.186:3300/0 conn(0x7fa92c15d880 msgr2=0x7fa92c0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 --2- 172.21.15.29:0/3559208433 >> v2:172.21.15.186:3300/0 conn(0x7fa92c15d880 0x7fa92c0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c06cba0 con 0x7fa92c159b20 2023-10-14T18:20:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fa928001670 con 0x7fa92c159b20 2023-10-14T18:20:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 >> v1:172.21.15.29:6789/0 conn(0x7fa92c159b20 legacy=0x7fa92c06bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.569 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa918003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa918003f40 0x7fa9180047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa918004d00 0x7fa918005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06c5e0 con 0x7fa918003f40 2023-10-14T18:20:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c06c8c0 con 0x7fa918004d00 2023-10-14T18:20:13.570 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa91800ba30 con 0x7fa92c159f00 2023-10-14T18:20:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.564+0000 7fa9319c8640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa918003f40 0x7fa9180047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9319c8640 1 -- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 msgr2=0x7fa918003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9321c9640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa918003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9329ca640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa918004d00 0x7fa918005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9319c8640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa918003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9319c8640 1 -- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa918004d00 msgr2=0x7fa918005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9319c8640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa918004d00 0x7fa918005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9319c8640 1 -- 172.21.15.29:0/3559208433 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa918003520 con 0x7fa918003f40 2023-10-14T18:20:13.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9321c9640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa92c159f00 0x7fa918003970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.565+0000 7fa9329ca640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa918004d00 0x7fa918005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.566+0000 7fa9319c8640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa918003f40 0x7fa9180047c0 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009a90 tx=0x7fa9180057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a5ee2096d1ccb3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.566+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c02c690 con 0x7fa918003f40 2023-10-14T18:20:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.566+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa91c02c7f0 con 0x7fa918003f40 2023-10-14T18:20:13.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.566+0000 7fa9311c7640 1 -- 172.21.15.29:0/3559208433 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa91c02cb80 con 0x7fa918003f40 2023-10-14T18:20:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 -- 172.21.15.29:0/3559208433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa918003f40 msgr2=0x7fa9180047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 --2- 172.21.15.29:0/3559208433 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa918003f40 0x7fa9180047c0 secure :-1 s=READY pgs=3278 cs=0 l=1 rev1=1 crypto rx=0x7fa91c009a90 tx=0x7fa9180057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa9329ca640 1 -- 172.21.15.29:0/3559208433 reap_dead start 2023-10-14T18:20:13.576 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 -- 172.21.15.29:0/3559208433 shutdown_connections 2023-10-14T18:20:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 -- 172.21.15.29:0/3559208433 >> 172.21.15.29:0/3559208433 conn(0x7fa92c0cb410 msgr2=0x7fa92c0cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 -- 172.21.15.29:0/3559208433 shutdown_connections 2023-10-14T18:20:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.567+0000 7fa93442d640 1 -- 172.21.15.29:0/3559208433 wait complete. 2023-10-14T18:20:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.569+0000 7fa93442d640 1 Processor -- start 2023-10-14T18:20:13.577 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.569+0000 7fa93442d640 1 -- start start 2023-10-14T18:20:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa918003f40 0x7fa92c0c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa92c0c68e0 0x7fa92c14fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.578 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c0c6d60 con 0x7fa92c150110 2023-10-14T18:20:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c0c7220 con 0x7fa92c0c68e0 2023-10-14T18:20:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa93442d640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa92c0b05a0 con 0x7fa918003f40 2023-10-14T18:20:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa9329ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa9329ca640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43476/0 (socket says 172.21.15.29:43476) 2023-10-14T18:20:13.579 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.570+0000 7fa9329ca640 1 -- 172.21.15.29:0/569155519 learned_addr learned my addr 172.21.15.29:0/569155519 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9319c8640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa92c0c68e0 0x7fa92c14fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9321c9640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa918003f40 0x7fa92c0c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 -- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa918003f40 msgr2=0x7fa92c0c63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa918003f40 0x7fa92c0c63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.580 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 -- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa92c0c68e0 msgr2=0x7fa92c14fbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa92c0c68e0 0x7fa92c14fbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 -- 172.21.15.29:0/569155519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa92c0b1e10 con 0x7fa92c150110 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa9329ca640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0c8bb0 tx=0x7fa924009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924007cf0 con 0x7fa92c150110 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa924007e70 con 0x7fa92c150110 2023-10-14T18:20:13.581 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa924014620 con 0x7fa92c150110 2023-10-14T18:20:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.571+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa92c0c6f40 con 0x7fa92c150110 2023-10-14T18:20:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.572+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa92c0b0a90 con 0x7fa92c150110 2023-10-14T18:20:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.573+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa92c0b1fa0 con 0x7fa92c150110 2023-10-14T18:20:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.574+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fa92401d3b0 con 0x7fa92c150110 2023-10-14T18:20:13.582 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.575+0000 7fa922ffd640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 0x7fa91006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:13.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.575+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fa924089530 con 0x7fa92c150110 2023-10-14T18:20:13.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.575+0000 7fa9321c9640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 0x7fa91006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:13.583 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.576+0000 7fa9321c9640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 0x7fa91006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0b47e0 tx=0x7fa928009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:13.584 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.579+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fa924054410 con 0x7fa92c150110 2023-10-14T18:20:13.775 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.773+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa92c06ce80 con 0x7fa91006afc0 2023-10-14T18:20:13.781 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.780+0000 7fa922ffd640 1 -- 172.21.15.29:0/569155519 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48651 (secure 0 0 0) 0x7fa92800d040 con 0x7fa91006afc0 2023-10-14T18:20:13.782 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:13.783 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:20:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 msgr2=0x7fa91006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 0x7fa91006d480 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0b47e0 tx=0x7fa928009040 comp rx=0 tx=0).stop 2023-10-14T18:20:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 msgr2=0x7fa92c0b22a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:13.785 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 secure :-1 s=READY pgs=3279 cs=0 l=1 rev1=1 crypto rx=0x7fa92c0c8bb0 tx=0x7fa924009cb0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 shutdown_connections 2023-10-14T18:20:13.786 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fa92c150110 0x7fa92c0b22a0 unknown :-1 s=CLOSED pgs=3279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fa92c0c68e0 0x7fa92c14fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fa91006afc0 0x7fa91006d480 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 --2- 172.21.15.29:0/569155519 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fa918003f40 0x7fa92c0c63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.783+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 >> 172.21.15.29:0/569155519 conn(0x7fa92c0cb410 msgr2=0x7fa92c0ccc10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:13.787 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.784+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 shutdown_connections 2023-10-14T18:20:13.788 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:13.784+0000 7fa93442d640 1 -- 172.21.15.29:0/569155519 wait complete. 2023-10-14T18:20:13.803 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-14T18:20:13.261544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":462456,"kb_used_data":28424,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754512264,"statfs":{"total":773094113280,"available":772620558336,"internally_reserved":0,"allocated":29106176,"data_stored":15157528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-735638,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-321,"num_read_kb":-165,"num_write":-342,"num_write_kb":-2910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003420"},"pg_stats":[{"pgid":"1.0","version":"273'403","reported_seq":1450,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-14T18:20:07.195920+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:20:07.195920+0000","last_peered":"2023-10-14T18:20:07.195920+0000","last_clean":"2023-10-14T18:20:07.195920+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:20:07.195920+0000","last_undegraded":"2023-10-14T18:20:07.195920+0000","last_fullsized":"2023-10-14T18:20:07.195920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57380,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314460,"statfs":{"total":96636764160,"available":96578007040,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.022},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":0.98699999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":1.127},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.236}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.89300000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.4530000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.2}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.302},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.5229999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.879,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.988,"5min":1.329,"15min":1.2549999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.198,"15min":1.2410000000000001},"min":{"1min":0.74299999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.569},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3520000000000001,"15min":1.3660000000000001},"min":{"1min":0.92900000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.1890000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.397,"15min":1.3859999999999999},"min":{"1min":0.79000000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.4179999999999999,"15min":1.427},"min":{"1min":1.028,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.4690000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.88800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.2869999999999999},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5409999999999999,"15min":1.4830000000000001},"min":{"1min":0.95099999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":14.805999999999999,"5min":14.805999999999999,"15min":14.805999999999999},"last":1.4159999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.4399999999999999,"15min":1.4039999999999999},"min":{"1min":1.119,"5min":0.85899999999999999,"15min":0.75800000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.4810000000000001,"15min":1.456},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":1.026}]},{"osd":5,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.423,"15min":1.423},"min":{"1min":0.88,"5min":0.624,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.6040000000000001,"15min":1.5960000000000001},"min":{"1min":0.97899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":0.96299999999999997}]},{"osd":7,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":2.468,"5min":1.796,"15min":1.6779999999999999},"min":{"1min":1.1870000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":16.541,"5min":16.541,"15min":16.541},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":1.131,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.0059999999999998,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.329}]}]},{"osd":5,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.147,"15min":1.1890000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6140000000000001,"5min":2.052,"15min":2.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.214},"min":{"1min":0.70099999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5960000000000001,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":0.88100000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.69899999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.754,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.26,"15min":1.286},"min":{"1min":0.81699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6599999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.99299999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.175,"15min":1.222},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7330000000000001,"5min":1.764,"15min":1.9339999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.389},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.923},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.363,"15min":1.4390000000000001},"min":{"1min":0.91700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8149999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.2},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3999999999999999,"15min":1.4139999999999999},"min":{"1min":1.006,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.74,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.042}]},{"osd":4,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.413,"15min":1.3759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8680000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4610000000000001,"15min":1.4079999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.61799999999999999},"max":{"1min":1.712,"5min":2.516,"15min":2.516},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64900000000000002},"max":{"1min":1.9359999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.6259999999999999,"15min":1.627},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.98,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.381}]},{"osd":7,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5600000000000001,"15min":1.5740000000000001},"min":{"1min":1.1160000000000001,"5min":0.95499999999999996,"15min":0.68600000000000005},"max":{"1min":1.8080000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.504},{"interface":"front","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.5629999999999999},"min":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8460000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.4530000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.181},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.458}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":1.028},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.306}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":1.073},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":1.2210000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.6899999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.345},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.6299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.57399999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.417,"5min":1.4870000000000001,"15min":1.5660000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.0940000000000001,"15min":1.179},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.9970000000000001},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.92900000000000005,"15min":0.998},"min":{"1min":0.52400000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.919},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0860000000000001,"15min":1.1819999999999999},"min":{"1min":0.61299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5089999999999999,"5min":1.742,"15min":1.8560000000000001},"last":1.038}]},{"osd":2,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2609999999999999,"15min":1.401},"min":{"1min":0.50900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9099999999999999,"5min":1.9970000000000001,"15min":2.032},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.25,"15min":1.3560000000000001},"min":{"1min":0.68899999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.915,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3560000000000001,"15min":1.4299999999999999},"min":{"1min":0.85399999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.105,"5min":2.105,"15min":2.153},"last":1.629},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4430000000000001,"15min":1.478},"min":{"1min":1.034,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.9870000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.802}]},{"osd":5,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":1.0800000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.758,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.202},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9239999999999999,"5min":2.1360000000000001,"15min":2.891},"last":1.4670000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.5589999999999999,"15min":1.6259999999999999},"min":{"1min":0.88700000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.2120000000000002,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.2230000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1419999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.8819999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":0.97399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.452,"5min":3.153,"15min":3.153},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.6060000000000001,"15min":1.6759999999999999},"min":{"1min":1.052,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0339999999999998,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.5589999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56800,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":1.0309999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2290000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.4139999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.532},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.496}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.631}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.47}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.7270000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38},"max":{"1min":1.4570000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.0920000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3600000000000001,"5min":2.847,"15min":2.847},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.996,"15min":1.042},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.492,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1659999999999999,"15min":1.2450000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6919999999999999,"5min":1.996,"15min":2.6859999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2350000000000001,"15min":1.3540000000000001},"min":{"1min":0.60199999999999998,"5min":0.52700000000000002,"15min":0.44700000000000001},"max":{"1min":1.879,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1970000000000001,"15min":1.2869999999999999},"min":{"1min":0.54400000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5649999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.0389999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4119999999999999,"15min":1.454},"min":{"1min":0.748,"5min":0.748,"15min":0.73899999999999999},"max":{"1min":2.0329999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4259999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.64800000000000002},"max":{"1min":2.2010000000000001,"5min":2.218,"15min":2.899},"last":0.95099999999999996}]},{"osd":5,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.405,"15min":1.4630000000000001},"min":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.857,"5min":2.2519999999999998,"15min":3.004},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.502,"15min":1.5569999999999999},"min":{"1min":0.94999999999999996,"5min":0.90400000000000003,"15min":0.73199999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.5129999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5069999999999999,"15min":1.597},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.367,"5min":2.8999999999999999,"15min":2.931},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.615,"15min":1.6220000000000001},"min":{"1min":0.90200000000000002,"5min":0.871,"15min":0.871},"max":{"1min":2.2509999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.5289999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.6060000000000001,"15min":1.6830000000000001},"min":{"1min":0.83799999999999997,"5min":0.82299999999999995,"15min":0.57899999999999996},"max":{"1min":1.9830000000000001,"5min":3.363,"15min":3.363},"last":1.488},{"interface":"front","average":{"1min":1.55,"5min":1.6259999999999999,"15min":1.6990000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.1629999999999998,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.6259999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53220,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318620,"statfs":{"total":96636764160,"available":96582266880,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.3899999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.609}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.8129999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.679},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.542}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.304},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.722}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.8979999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.4990000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.242},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.7769999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:20:13.805 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-10-14T18:20:13.805 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:20:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.028+0000 7f3dd0a09640 1 Processor -- start 2023-10-14T18:20:14.030 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.028+0000 7f3dd0a09640 1 -- start start 2023-10-14T18:20:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.029+0000 7f3dd0a09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3dcc0b4730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.029+0000 7f3dd0a09640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc0cd630 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.031 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.029+0000 7f3dd0a09640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dcc0ce1b0 con 0x7f3dcc06c3f0 2023-10-14T18:20:14.032 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.029+0000 7f3dd0a09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc15e7d0 con 0x7f3dcc06c7d0 2023-10-14T18:20:14.033 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.030+0000 7f3dcad9d640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f3dcc0b4c70 0x7f3dcc0b1710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55624/0 (socket says 172.21.15.29:55624) 2023-10-14T18:20:14.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.030+0000 7f3dcad9d640 1 -- 172.21.15.29:0/2758237446 learned_addr learned my addr 172.21.15.29:0/2758237446 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:14.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.030+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3dcc0b4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.034 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.030+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992245425 0 0) 0x7f3dcc0cd630 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.030+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3db8003520 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1589671700 0 0) 0x7f3db8003520 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 msgr2=0x7f3dcc0b4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3dcc0b4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.035 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 >> v1:172.21.15.186:6789/0 conn(0x7f3dcc06c3f0 legacy=0x7f3dcc15e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0d2170 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f3dc4001670 con 0x7f3dcc0b4c70 2023-10-14T18:20:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 >> v1:172.21.15.29:6789/0 conn(0x7f3dcc0b4c70 legacy=0x7f3dcc0b1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3db80039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.036 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3db80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3db8004d00 0x7f3db8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db800ba30 con 0x7f3db8003f20 2023-10-14T18:20:14.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3db800bba0 con 0x7f3db8004d00 2023-10-14T18:20:14.037 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3db800bd60 con 0x7f3dcc06c7d0 2023-10-14T18:20:14.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3db80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dcad9d640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3db8004d00 0x7f3db8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.031+0000 7f3dca59c640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3db80039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.038 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.032+0000 7f3dc9d9b640 1 -- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 msgr2=0x7f3db80039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:14.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.032+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc06c7d0 0x7f3db80039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.032+0000 7f3dc9d9b640 1 -- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3db8004d00 msgr2=0x7f3db8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:14.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.032+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3db8004d00 0x7f3db8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.039 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.032+0000 7f3dc9d9b640 1 -- 172.21.15.29:0/2758237446 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3db8003520 con 0x7f3db8003f20 2023-10-14T18:20:14.042 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.033+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3db80047c0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009ac0 tx=0x7f3db80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2e0d256e0a268e3 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.033+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db402c6b0 con 0x7f3db8003f20 2023-10-14T18:20:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.033+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3db402c810 con 0x7f3db8003f20 2023-10-14T18:20:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.033+0000 7f3dc959a640 1 -- 172.21.15.29:0/2758237446 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3db402cba0 con 0x7f3db8003f20 2023-10-14T18:20:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 -- 172.21.15.29:0/2758237446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 msgr2=0x7f3db80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.043 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/2758237446 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3db80047c0 secure :-1 s=READY pgs=3281 cs=0 l=1 rev1=1 crypto rx=0x7f3db4009ac0 tx=0x7f3db80057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dcad9d640 1 -- 172.21.15.29:0/2758237446 reap_dead start 2023-10-14T18:20:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 -- 172.21.15.29:0/2758237446 shutdown_connections 2023-10-14T18:20:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 -- 172.21.15.29:0/2758237446 >> 172.21.15.29:0/2758237446 conn(0x7f3dcc0d2cb0 msgr2=0x7f3dcc0d30c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 -- 172.21.15.29:0/2758237446 shutdown_connections 2023-10-14T18:20:14.044 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.034+0000 7f3dd0a09640 1 -- 172.21.15.29:0/2758237446 wait complete. 2023-10-14T18:20:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.036+0000 7f3dd0a09640 1 Processor -- start 2023-10-14T18:20:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.036+0000 7f3dd0a09640 1 -- start start 2023-10-14T18:20:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc14c150 0x7f3dcc0c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.045 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 0x7f3dcc0ca6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0cb7b0 con 0x7f3db8003f20 2023-10-14T18:20:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0cbf70 con 0x7f3dcc0ca230 2023-10-14T18:20:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dd0a09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dcc0cc7e0 con 0x7f3dcc14c150 2023-10-14T18:20:14.046 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dca59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43498/0 (socket says 172.21.15.29:43498) 2023-10-14T18:20:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.037+0000 7f3dca59c640 1 -- 172.21.15.29:0/1163084946 learned_addr learned my addr 172.21.15.29:0/1163084946 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc14c150 0x7f3dcc0c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dcad9d640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 0x7f3dcc0ca6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.047 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 -- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc14c150 msgr2=0x7f3dcc0c9cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc14c150 0x7f3dcc0c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 -- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 msgr2=0x7f3dcc0ca6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 0x7f3dcc0ca6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 -- 172.21.15.29:0/1163084946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dcc0ccac0 con 0x7f3db8003f20 2023-10-14T18:20:14.048 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dcad9d640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 0x7f3dcc0ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:14.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dca59c640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00e8c0 tx=0x7f3dbc00ed80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc003b80 con 0x7f3db8003f20 2023-10-14T18:20:14.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dbc003ce0 con 0x7f3db8003f20 2023-10-14T18:20:14.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3dbc0052f0 con 0x7f3db8003f20 2023-10-14T18:20:14.049 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.038+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dcc0cba50 con 0x7f3db8003f20 2023-10-14T18:20:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.039+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dcc0ccda0 con 0x7f3db8003f20 2023-10-14T18:20:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.040+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dcc04e160 con 0x7f3db8003f20 2023-10-14T18:20:14.050 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.041+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3dbc005450 con 0x7f3db8003f20 2023-10-14T18:20:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.042+0000 7f3dc2ffd640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 0x7f3dac06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.042+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f3dbc0899b0 con 0x7f3db8003f20 2023-10-14T18:20:14.051 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.042+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 0x7f3dac06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.043+0000 7f3dc9d9b640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 0x7f3dac06d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db4030000 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.052 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.046+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3dbc0548a0 con 0x7f3db8003f20 2023-10-14T18:20:14.240 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.238+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3dcc0cdb60 con 0x7f3dac06afc0 2023-10-14T18:20:14.246 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.244+0000 7f3dc2ffd640 1 -- 172.21.15.29:0/1163084946 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48651 (secure 0 0 0) 0x7f3db4004140 con 0x7f3dac06afc0 2023-10-14T18:20:14.247 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:14.247 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:20:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 msgr2=0x7f3dac06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 0x7f3dac06d480 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3db4036000 tx=0x7f3db4030000 comp rx=0 tx=0).stop 2023-10-14T18:20:14.249 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 msgr2=0x7f3dcc0a9850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 secure :-1 s=READY pgs=3282 cs=0 l=1 rev1=1 crypto rx=0x7f3dbc00e8c0 tx=0x7f3dbc00ed80 comp rx=0 tx=0).stop 2023-10-14T18:20:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 shutdown_connections 2023-10-14T18:20:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f3dcc0ca230 0x7f3dcc0ca6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f3dac06afc0 0x7f3dac06d480 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.250 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f3dcc14c150 0x7f3dcc0c9cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 --2- 172.21.15.29:0/1163084946 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f3db8003f20 0x7f3dcc0a9850 unknown :-1 s=CLOSED pgs=3282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 >> 172.21.15.29:0/1163084946 conn(0x7f3dcc0d2cb0 msgr2=0x7f3dcc15cb60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 shutdown_connections 2023-10-14T18:20:14.251 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.247+0000 7f3dd0a09640 1 -- 172.21.15.29:0/1163084946 wait complete. 2023-10-14T18:20:14.269 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2023-10-14T18:20:13.261544+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":462456,"kb_used_data":28424,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754512264,"statfs":{"total":773094113280,"available":772620558336,"internally_reserved":0,"allocated":29106176,"data_stored":15157528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-735638,"num_objects":-25,"num_object_clones":0,"num_object_copies":-50,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-25,"num_whiteouts":0,"num_read":-321,"num_read_kb":-165,"num_write":-342,"num_write_kb":-2910,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.003420"},"pg_stats":[{"pgid":"1.0","version":"273'403","reported_seq":1450,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-14T18:20:07.195920+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:20:07.195920+0000","last_peered":"2023-10-14T18:20:07.195920+0000","last_clean":"2023-10-14T18:20:07.195920+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:20:07.195920+0000","last_undegraded":"2023-10-14T18:20:07.195920+0000","last_fullsized":"2023-10-14T18:20:07.195920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57380,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314460,"statfs":{"total":96636764160,"available":96578007040,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.022},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":0.98699999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":1.127},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.236}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":1.0800000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":0.89300000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":0.94499999999999995},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.4530000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.6100000000000001},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.2}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.302},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.5229999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.3580000000000001},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.6459999999999999}]}]},{"osd":6,"up_from":13,"seq":55834576034,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.3109999999999999,"5min":1.1950000000000001,"15min":1.1919999999999999},"min":{"1min":0.879,"5min":0.58899999999999997,"15min":0.58899999999999997},"max":{"1min":3.0409999999999999,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.6950000000000001},{"interface":"front","average":{"1min":1.988,"5min":1.329,"15min":1.2549999999999999},"min":{"1min":0.71599999999999997,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":14.970000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.1719999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.23,"5min":1.198,"15min":1.2410000000000001},"min":{"1min":0.74299999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.855,"5min":1.855,"15min":1.855},"last":1.569},{"interface":"front","average":{"1min":1.3959999999999999,"5min":1.3520000000000001,"15min":1.3660000000000001},"min":{"1min":0.92900000000000005,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":2.5550000000000002,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.1890000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.532,"5min":1.397,"15min":1.3859999999999999},"min":{"1min":0.79000000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":4.4509999999999996,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.5289999999999999},{"interface":"front","average":{"1min":1.498,"5min":1.4179999999999999,"15min":1.427},"min":{"1min":1.028,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":3.3980000000000001,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.4690000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.456,"5min":1.3680000000000001,"15min":1.409},"min":{"1min":0.88800000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":2.6749999999999998,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.2869999999999999},{"interface":"front","average":{"1min":2.1379999999999999,"5min":1.5409999999999999,"15min":1.4830000000000001},"min":{"1min":0.95099999999999996,"5min":0.75900000000000001,"15min":0.75900000000000001},"max":{"1min":14.805999999999999,"5min":14.805999999999999,"15min":14.805999999999999},"last":1.4159999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.6100000000000001,"5min":1.4399999999999999,"15min":1.4039999999999999},"min":{"1min":1.119,"5min":0.85899999999999999,"15min":0.75800000000000001},"max":{"1min":3.1000000000000001,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":0.81399999999999995},{"interface":"front","average":{"1min":1.6699999999999999,"5min":1.4810000000000001,"15min":1.456},"min":{"1min":0.745,"5min":0.72099999999999997,"15min":0.72099999999999997},"max":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":1.026}]},{"osd":5,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.423,"15min":1.423},"min":{"1min":0.88,"5min":0.624,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":0.86799999999999999},{"interface":"front","average":{"1min":1.655,"5min":1.6040000000000001,"15min":1.5960000000000001},"min":{"1min":0.97899999999999998,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.625,"5min":2.625,"15min":2.625},"last":0.96299999999999997}]},{"osd":7,"last update":"Sat Oct 14 18:19:16 2023","interfaces":[{"interface":"back","average":{"1min":2.468,"5min":1.796,"15min":1.6779999999999999},"min":{"1min":1.1870000000000001,"5min":0.71499999999999997,"15min":0.71499999999999997},"max":{"1min":16.541,"5min":16.541,"15min":16.541},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.6499999999999999,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":1.131,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":2.0059999999999998,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.329}]}]},{"osd":5,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.147,"15min":1.1890000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6140000000000001,"5min":2.052,"15min":2.052},"last":0.55200000000000005},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.214},"min":{"1min":0.70099999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5960000000000001,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":0.88100000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.69899999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.754,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.78100000000000003},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.26,"15min":1.286},"min":{"1min":0.81699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6599999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":0.99299999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.175,"15min":1.222},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7330000000000001,"5min":1.764,"15min":1.9339999999999999},"last":0.94999999999999996},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.389},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.923},"last":0.92100000000000004}]},{"osd":3,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.363,"15min":1.4390000000000001},"min":{"1min":0.91700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8149999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.2},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3999999999999999,"15min":1.4139999999999999},"min":{"1min":1.006,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.74,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.042}]},{"osd":4,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.413,"15min":1.3759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8680000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4610000000000001,"15min":1.4079999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.61799999999999999},"max":{"1min":1.712,"5min":2.516,"15min":2.516},"last":1.2609999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64900000000000002},"max":{"1min":1.9359999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.6259999999999999,"15min":1.627},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.98,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.381}]},{"osd":7,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5600000000000001,"15min":1.5740000000000001},"min":{"1min":1.1160000000000001,"5min":0.95499999999999996,"15min":0.68600000000000005},"max":{"1min":1.8080000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.504},{"interface":"front","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.5629999999999999},"min":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8460000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.4530000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.181},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.458}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":0.94699999999999995},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.86599999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":1.028},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.5840000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.306}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":1.073},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":1.2210000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.6899999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.345},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.6299999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.93600000000000005,"15min":0.97499999999999998},"min":{"1min":0.57399999999999995,"5min":0.377,"15min":0.377},"max":{"1min":1.417,"5min":1.4870000000000001,"15min":1.5660000000000001},"last":1.0920000000000001},{"interface":"front","average":{"1min":1.087,"5min":1.0940000000000001,"15min":1.179},"min":{"1min":0.53900000000000003,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.665,"5min":1.665,"15min":1.9970000000000001},"last":0.64500000000000002}]},{"osd":1,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":0.97899999999999998,"5min":0.92900000000000005,"15min":0.998},"min":{"1min":0.52400000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.782,"5min":1.782,"15min":1.919},"last":0.76400000000000001},{"interface":"front","average":{"1min":1.0029999999999999,"5min":1.0860000000000001,"15min":1.1819999999999999},"min":{"1min":0.61299999999999999,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.5089999999999999,"5min":1.742,"15min":1.8560000000000001},"last":1.038}]},{"osd":2,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.1579999999999999,"5min":1.2609999999999999,"15min":1.401},"min":{"1min":0.50900000000000001,"5min":0.498,"15min":0.498},"max":{"1min":1.9099999999999999,"5min":1.9970000000000001,"15min":2.032},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.292,"5min":1.25,"15min":1.3560000000000001},"min":{"1min":0.68899999999999995,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.915,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.2569999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.3919999999999999,"5min":1.3560000000000001,"15min":1.4299999999999999},"min":{"1min":0.85399999999999998,"5min":0.73899999999999999,"15min":0.73899999999999999},"max":{"1min":2.105,"5min":2.105,"15min":2.153},"last":1.629},{"interface":"front","average":{"1min":1.4670000000000001,"5min":1.4430000000000001,"15min":1.478},"min":{"1min":1.034,"5min":0.83699999999999997,"15min":0.83699999999999997},"max":{"1min":1.9870000000000001,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.802}]},{"osd":5,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.415,"5min":1.417,"15min":1.4510000000000001},"min":{"1min":1.0800000000000001,"5min":0.78700000000000003,"15min":0.78700000000000003},"max":{"1min":1.758,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.202},{"interface":"front","average":{"1min":1.4850000000000001,"5min":1.444,"15min":1.542},"min":{"1min":0.81999999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"max":{"1min":1.9239999999999999,"5min":2.1360000000000001,"15min":2.891},"last":1.4670000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.619,"5min":1.5589999999999999,"15min":1.6259999999999999},"min":{"1min":0.88700000000000001,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":2.2120000000000002,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.7529999999999999},{"interface":"front","average":{"1min":1.6830000000000001,"5min":1.6279999999999999,"15min":1.6779999999999999},"min":{"1min":1.2230000000000001,"5min":0.91900000000000004,"15min":0.91900000000000004},"max":{"1min":2.1419999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.8819999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:10 2023","interfaces":[{"interface":"back","average":{"1min":1.6399999999999999,"5min":1.55,"15min":1.6040000000000001},"min":{"1min":0.97399999999999998,"5min":0.77900000000000003,"15min":0.77900000000000003},"max":{"1min":2.452,"5min":3.153,"15min":3.153},"last":1.3169999999999999},{"interface":"front","average":{"1min":1.5700000000000001,"5min":1.6060000000000001,"15min":1.6759999999999999},"min":{"1min":1.052,"5min":0.89900000000000002,"15min":0.89900000000000002},"max":{"1min":2.0339999999999998,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.5589999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56800,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.94499999999999995},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":1.0309999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":0.86199999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2290000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.1619999999999999},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.4139999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.532},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.496}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.3009999999999999},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.631}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.6739999999999999},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.47}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.9139999999999999},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.7270000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38},"max":{"1min":1.4570000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.0920000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3600000000000001,"5min":2.847,"15min":2.847},"last":0.85499999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.996,"15min":1.042},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.492,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.76500000000000001},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1659999999999999,"15min":1.2450000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6919999999999999,"5min":1.996,"15min":2.6859999999999999},"last":0.73099999999999998}]},{"osd":3,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2350000000000001,"15min":1.3540000000000001},"min":{"1min":0.60199999999999998,"5min":0.52700000000000002,"15min":0.44700000000000001},"max":{"1min":1.879,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":0.98699999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1970000000000001,"15min":1.2869999999999999},"min":{"1min":0.54400000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5649999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.0389999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4119999999999999,"15min":1.454},"min":{"1min":0.748,"5min":0.748,"15min":0.73899999999999999},"max":{"1min":2.0329999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.0880000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4259999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.64800000000000002},"max":{"1min":2.2010000000000001,"5min":2.218,"15min":2.899},"last":0.95099999999999996}]},{"osd":5,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.405,"15min":1.4630000000000001},"min":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.857,"5min":2.2519999999999998,"15min":3.004},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.502,"15min":1.5569999999999999},"min":{"1min":0.94999999999999996,"5min":0.90400000000000003,"15min":0.73199999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.5129999999999999},"last":1.2829999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5069999999999999,"15min":1.597},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.367,"5min":2.8999999999999999,"15min":2.931},"last":1.5920000000000001},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.615,"15min":1.6220000000000001},"min":{"1min":0.90200000000000002,"5min":0.871,"15min":0.871},"max":{"1min":2.2509999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.5289999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.6060000000000001,"15min":1.6830000000000001},"min":{"1min":0.83799999999999997,"5min":0.82299999999999995,"15min":0.57899999999999996},"max":{"1min":1.9830000000000001,"5min":3.363,"15min":3.363},"last":1.488},{"interface":"front","average":{"1min":1.55,"5min":1.6259999999999999,"15min":1.6990000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.1629999999999998,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.6259999999999999}]}]},{"osd":0,"up_from":13,"seq":55834576034,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53220,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318620,"statfs":{"total":96636764160,"available":96582266880,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":1.3899999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.609}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.6499999999999999},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.8129999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.679},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.542}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.304},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.722}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.8979999999999999},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.4990000000000001}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.242},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.7769999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:20:14.271 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-10-14T18:20:14.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.491+0000 7fed1ba09640 1 Processor -- start 2023-10-14T18:20:14.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.492+0000 7fed1ba09640 1 -- start start 2023-10-14T18:20:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed1415aaa0 0x7fed1415e820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fed1415ae80 0x7fed140cc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 0x7fed14070f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1406b810 con 0x7fed1415aaa0 2023-10-14T18:20:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed1415ed60 con 0x7fed1415ae80 2023-10-14T18:20:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed1ba09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed140707e0 con 0x7fed140ccaa0 2023-10-14T18:20:14.496 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed197a5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed1415aaa0 0x7fed1415e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.493+0000 7fed197a5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fed1415aaa0 0x7fed1415e820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43510/0 (socket says 172.21.15.29:43510) 2023-10-14T18:20:14.497 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 -- 172.21.15.29:0/2855037944 learned_addr learned my addr 172.21.15.29:0/2855037944 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed18fa4640 1 --2- 172.21.15.29:0/2855037944 >> v2:172.21.15.186:3300/0 conn(0x7fed1415ae80 0x7fed140cc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed19fa6640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 0x7fed14070f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.499 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 -- 172.21.15.29:0/2855037944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 msgr2=0x7fed14070f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 0x7fed14070f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 -- 172.21.15.29:0/2855037944 >> v2:172.21.15.186:3300/0 conn(0x7fed1415ae80 msgr2=0x7fed140cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 --2- 172.21.15.29:0/2855037944 >> v2:172.21.15.186:3300/0 conn(0x7fed1415ae80 0x7fed140cc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed197a5640 1 -- 172.21.15.29:0/2855037944 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed1415ef90 con 0x7fed1415aaa0 2023-10-14T18:20:14.500 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.494+0000 7fed19fa6640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 0x7fed14070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.495+0000 7fed197a5640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed1415e820 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fed140c59c0 tx=0x7fed1002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8d5306ca56ae35b server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed0bfff640 1 -- 172.21.15.29:0/2855037944 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10031870 con 0x7fed1415aaa0 2023-10-14T18:20:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed0bfff640 1 -- 172.21.15.29:0/2855037944 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed100319d0 con 0x7fed1415aaa0 2023-10-14T18:20:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed0bfff640 1 -- 172.21.15.29:0/2855037944 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed10031d60 con 0x7fed1415aaa0 2023-10-14T18:20:14.501 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 -- 172.21.15.29:0/2855037944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 msgr2=0x7fed1415e820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed1415e820 secure :-1 s=READY pgs=3283 cs=0 l=1 rev1=1 crypto rx=0x7fed140c59c0 tx=0x7fed1002b910 comp rx=0 tx=0).stop 2023-10-14T18:20:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 -- 172.21.15.29:0/2855037944 shutdown_connections 2023-10-14T18:20:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed140ccaa0 0x7fed14070f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.502 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2855037944 >> v2:172.21.15.186:3300/0 conn(0x7fed1415ae80 0x7fed140cc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2855037944 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed1415e820 unknown :-1 s=CLOSED pgs=3283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 -- 172.21.15.29:0/2855037944 >> 172.21.15.29:0/2855037944 conn(0x7fed140cb410 msgr2=0x7fed140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 -- 172.21.15.29:0/2855037944 shutdown_connections 2023-10-14T18:20:14.503 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.496+0000 7fed1ba09640 1 -- 172.21.15.29:0/2855037944 wait complete. 2023-10-14T18:20:14.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.498+0000 7fed1ba09640 1 Processor -- start 2023-10-14T18:20:14.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.498+0000 7fed1ba09640 1 -- start start 2023-10-14T18:20:14.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed140ccaa0 0x7fed1415d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.507 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed1415ae80 0x7fed1414d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1415d990 con 0x7fed1415aaa0 2023-10-14T18:20:14.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed1415dc70 con 0x7fed140ccaa0 2023-10-14T18:20:14.508 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.499+0000 7fed1ba09640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed1415df50 con 0x7fed1415ae80 2023-10-14T18:20:14.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43518/0 (socket says 172.21.15.29:43518) 2023-10-14T18:20:14.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 -- 172.21.15.29:0/2730224557 learned_addr learned my addr 172.21.15.29:0/2730224557 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:14.509 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed197a5640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed140ccaa0 0x7fed1415d060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed19fa6640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed1415ae80 0x7fed1414d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 -- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed1415ae80 msgr2=0x7fed1414d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed1415ae80 0x7fed1414d6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.510 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 -- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed140ccaa0 msgr2=0x7fed1415d060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed140ccaa0 0x7fed1415d060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.500+0000 7fed18fa4640 1 -- 172.21.15.29:0/2730224557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed100094d0 con 0x7fed1415aaa0 2023-10-14T18:20:14.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed18fa4640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fed0400d7d0 tx=0x7fed0400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400b6c0 con 0x7fed1415aaa0 2023-10-14T18:20:14.511 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed0400b840 con 0x7fed1415aaa0 2023-10-14T18:20:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fed0400bbf0 con 0x7fed1415aaa0 2023-10-14T18:20:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1414b610 con 0x7fed1415aaa0 2023-10-14T18:20:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.501+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed1415d5a0 con 0x7fed1415aaa0 2023-10-14T18:20:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.502+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed1415d730 con 0x7fed1415aaa0 2023-10-14T18:20:14.512 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.503+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fed0400bd50 con 0x7fed1415aaa0 2023-10-14T18:20:14.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.505+0000 7fed09ffb640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 0x7fecf806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:14.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.505+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fed04089db0 con 0x7fed1415aaa0 2023-10-14T18:20:14.513 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.505+0000 7fed197a5640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 0x7fecf806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:14.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.506+0000 7fed197a5640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 0x7fecf806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed10036000 tx=0x7fed1003c040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:14.514 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.509+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fed0408e070 con 0x7fed1415aaa0 2023-10-14T18:20:14.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.703+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed1404e160 con 0x7fed1415aaa0 2023-10-14T18:20:14.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.705+0000 7fed09ffb640 1 -- 172.21.15.29:0/2730224557 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v276) v1 ==== 74+0+14360 (secure 0 0 0) 0x7fed04058720 con 0x7fed1415aaa0 2023-10-14T18:20:14.707 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:14.707 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":276,"fsid":"f6aee177-7aff-4117-99f6-f2cafd5adc35","created":"2023-10-14T16:39:59.950810+0000","modified":"2023-10-14T18:20:08.303380+0000","last_up_change":"2023-10-14T16:40:13.143770+0000","last_in_change":"2023-10-14T16:40:04.315491+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":35,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-14T16:40:13.373867+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7a6bfcaa-8402-4ea5-913f-e9eab85c856a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6800","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6801","nonce":3107383011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6802","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6803","nonce":3107383011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6806","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6807","nonce":3107383011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6804","nonce":3107383011},{"type":"v1","addr":"172.21.15.29:6805","nonce":3107383011}]},"public_addr":"172.21.15.29:6801/3107383011","cluster_addr":"172.21.15.29:6803/3107383011","heartbeat_back_addr":"172.21.15.29:6807/3107383011","heartbeat_front_addr":"172.21.15.29:6805/3107383011","state":["exists","up"]},{"osd":1,"uuid":"38ce4325-d8a0-4af1-aa17-9e7cf06d521f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6814","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6815","nonce":3723048683}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6817","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6820","nonce":3723048683}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6829","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6830","nonce":3723048683}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6824","nonce":3723048683},{"type":"v1","addr":"172.21.15.29:6826","nonce":3723048683}]},"public_addr":"172.21.15.29:6815/3723048683","cluster_addr":"172.21.15.29:6820/3723048683","heartbeat_back_addr":"172.21.15.29:6830/3723048683","heartbeat_front_addr":"172.21.15.29:6826/3723048683","state":["exists","up"]},{"osd":2,"uuid":"fdb2602a-e449-419e-9eb0-3aa215a44dd6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6808","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6809","nonce":3586046734}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6810","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6812","nonce":3586046734}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6828","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6831","nonce":3586046734}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6818","nonce":3586046734},{"type":"v1","addr":"172.21.15.29:6823","nonce":3586046734}]},"public_addr":"172.21.15.29:6809/3586046734","cluster_addr":"172.21.15.29:6812/3586046734","heartbeat_back_addr":"172.21.15.29:6831/3586046734","heartbeat_front_addr":"172.21.15.29:6823/3586046734","state":["exists","up"]},{"osd":3,"uuid":"e78f6d2c-1521-4d00-9d3b-8988f5aa90ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6811","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6813","nonce":149520408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6816","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6819","nonce":149520408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6825","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6827","nonce":149520408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6821","nonce":149520408},{"type":"v1","addr":"172.21.15.29:6822","nonce":149520408}]},"public_addr":"172.21.15.29:6813/149520408","cluster_addr":"172.21.15.29:6819/149520408","heartbeat_back_addr":"172.21.15.29:6827/149520408","heartbeat_front_addr":"172.21.15.29:6822/149520408","state":["exists","up"]},{"osd":4,"uuid":"c1c0fbc8-08a3-46f0-8377-2acd51c3e048","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6800","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6801","nonce":189625010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6802","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6803","nonce":189625010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6806","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6807","nonce":189625010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6804","nonce":189625010},{"type":"v1","addr":"172.21.15.186:6805","nonce":189625010}]},"public_addr":"172.21.15.186:6801/189625010","cluster_addr":"172.21.15.186:6803/189625010","heartbeat_back_addr":"172.21.15.186:6807/189625010","heartbeat_front_addr":"172.21.15.186:6805/189625010","state":["exists","up"]},{"osd":5,"uuid":"8ba443e1-4c4b-42a6-a349-c1098dcdecb0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6816","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6817","nonce":1666789598}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6818","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6819","nonce":1666789598}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6822","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6823","nonce":1666789598}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6820","nonce":1666789598},{"type":"v1","addr":"172.21.15.186:6821","nonce":1666789598}]},"public_addr":"172.21.15.186:6817/1666789598","cluster_addr":"172.21.15.186:6819/1666789598","heartbeat_back_addr":"172.21.15.186:6823/1666789598","heartbeat_front_addr":"172.21.15.186:6821/1666789598","state":["exists","up"]},{"osd":6,"uuid":"d143f29f-dde3-4f91-b56a-e45c65002e1a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6824","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6825","nonce":3237784830}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6826","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6827","nonce":3237784830}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6830","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6831","nonce":3237784830}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6828","nonce":3237784830},{"type":"v1","addr":"172.21.15.186:6829","nonce":3237784830}]},"public_addr":"172.21.15.186:6825/3237784830","cluster_addr":"172.21.15.186:6827/3237784830","heartbeat_back_addr":"172.21.15.186:6831/3237784830","heartbeat_front_addr":"172.21.15.186:6829/3237784830","state":["exists","up"]},{"osd":7,"uuid":"36e18e36-52bb-4245-9dc0-c5f7a85f4cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":269,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6808","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6809","nonce":4274731638}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6810","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6811","nonce":4274731638}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6814","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6815","nonce":4274731638}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6812","nonce":4274731638},{"type":"v1","addr":"172.21.15.186:6813","nonce":4274731638}]},"public_addr":"172.21.15.186:6809/4274731638","cluster_addr":"172.21.15.186:6811/4274731638","heartbeat_back_addr":"172.21.15.186:6815/4274731638","heartbeat_front_addr":"172.21.15.186:6813/4274731638","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.900795+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.890856+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.614475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.884192+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.623137+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.713974+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.677544+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-14T16:40:09.649880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.186:6835/3411376277":"2023-10-15T18:20:04.987619+0000","172.21.15.29:0/2967850488":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3208700101":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/2196987879":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/754934507":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3551524051":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2464998546":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/3292195983":"2023-10-15T18:19:52.752456+0000","172.21.15.186:6833/3411376277":"2023-10-15T18:20:04.987619+0000","172.21.15.29:0/4006438795":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1086342911":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/503271084":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/4178572946":"2023-10-15T18:20:01.230353+0000","172.21.15.29:6832/1327833207":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/3279571573":"2023-10-15T18:19:52.752456+0000","172.21.15.29:6833/4074993897":"2023-10-15T18:20:01.230353+0000","172.21.15.29:0/1082110076":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/2709558548":"2023-10-15T18:19:52.752456+0000","172.21.15.29:0/1343183116":"2023-10-15T18:19:52.752456+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":5,"snaps":[{"begin":112,"length":1},{"begin":223,"length":1}]},{"pool":15,"snaps":[{"begin":2,"length":2}]},{"pool":19,"snaps":[{"begin":57,"length":1},{"begin":113,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-14T18:20:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 msgr2=0x7fecf806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 0x7fecf806d480 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fed10036000 tx=0x7fed1003c040 comp rx=0 tx=0).stop 2023-10-14T18:20:14.710 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 msgr2=0x7fed141f1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 secure :-1 s=READY pgs=3284 cs=0 l=1 rev1=1 crypto rx=0x7fed0400d7d0 tx=0x7fed0400dc90 comp rx=0 tx=0).stop 2023-10-14T18:20:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 shutdown_connections 2023-10-14T18:20:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fecf806afc0 0x7fecf806d480 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fed1415ae80 0x7fed1414d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.711 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fed1415aaa0 0x7fed141f1d40 unknown :-1 s=CLOSED pgs=3284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 --2- 172.21.15.29:0/2730224557 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fed140ccaa0 0x7fed1415d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 >> 172.21.15.29:0/2730224557 conn(0x7fed140cb410 msgr2=0x7fed140cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 shutdown_connections 2023-10-14T18:20:14.712 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:14.708+0000 7fed1ba09640 1 -- 172.21.15.29:0/2730224557 wait complete. 2023-10-14T18:20:15.732 INFO:tasks.ceph:Scrubbing osd.0 2023-10-14T18:20:15.732 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:15.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.976+0000 7f6fb3180640 1 Processor -- start 2023-10-14T18:20:15.978 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.976+0000 7f6fb3180640 1 -- start start 2023-10-14T18:20:15.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.978+0000 7f6fb3180640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 0x7f6fac06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.978+0000 7f6fb3180640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6fac0ad970 0x7f6fac15e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.980 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.978+0000 7f6fb3180640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fac06c960 con 0x7f6fac15ebc0 2023-10-14T18:20:15.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.978+0000 7f6fb3180640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fac15b9d0 con 0x7f6fac0ad970 2023-10-14T18:20:15.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.978+0000 7f6fb3180640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac15bcb0 con 0x7f6fac06c040 2023-10-14T18:20:15.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb297f640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6fac15ebc0 0x7f6fac0cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55626/0 (socket says 172.21.15.29:55626) 2023-10-14T18:20:15.981 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb297f640 1 -- 172.21.15.29:0/4130570690 learned_addr learned my addr 172.21.15.29:0/4130570690 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb217e640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 0x7f6fac06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb197d640 1 --2- 172.21.15.29:0/4130570690 >> v2:172.21.15.186:3300/0 conn(0x7f6fac0ad970 0x7f6fac15e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862799846 0 0) 0x7f6fac06c960 con 0x7f6fac15ebc0 2023-10-14T18:20:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f98003520 con 0x7f6fac15ebc0 2023-10-14T18:20:15.984 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1817642573 0 0) 0x7f6f98003520 con 0x7f6fac15ebc0 2023-10-14T18:20:15.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 msgr2=0x7f6fac06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:15.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 0x7f6fac06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.985 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 >> v2:172.21.15.186:3300/0 conn(0x7f6fac0ad970 msgr2=0x7f6fac15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 --2- 172.21.15.29:0/4130570690 >> v2:172.21.15.186:3300/0 conn(0x7f6fac0ad970 0x7f6fac15e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac15bf90 con 0x7f6fac15ebc0 2023-10-14T18:20:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.979+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6fa40019f0 con 0x7f6fac15ebc0 2023-10-14T18:20:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 >> v1:172.21.15.29:6789/0 conn(0x7f6fac15ebc0 legacy=0x7f6fac0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:15.986 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6f98003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 0x7f6f980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f98004d00 0x7f6f98005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac15b9d0 con 0x7f6f98003970 2023-10-14T18:20:15.987 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac15bcb0 con 0x7f6f98004d00 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9800b980 con 0x7f6fac06c040 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6f98003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb297f640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f98004d00 0x7f6f98005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 -- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 msgr2=0x7f6f980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac06c040 0x7f6f980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.988 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 -- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f98004d00 msgr2=0x7f6f98005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6f98004d00 0x7f6f98005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.980+0000 7f6fb217e640 1 -- 172.21.15.29:0/4130570690 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f98003520 con 0x7f6f98003970 2023-10-14T18:20:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb217e640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6f98003d90 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009ac0 tx=0x7f6f980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903b94b8ae55c652 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802c710 con 0x7f6f98003970 2023-10-14T18:20:15.989 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802c870 con 0x7f6f98003970 2023-10-14T18:20:15.990 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb117c640 1 -- 172.21.15.29:0/4130570690 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa802cc00 con 0x7f6f98003970 2023-10-14T18:20:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb3180640 1 -- 172.21.15.29:0/4130570690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 msgr2=0x7f6f98003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb3180640 1 --2- 172.21.15.29:0/4130570690 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6f98003d90 secure :-1 s=READY pgs=3286 cs=0 l=1 rev1=1 crypto rx=0x7f6fa8009ac0 tx=0x7f6f980057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.982+0000 7f6fb297f640 1 -- 172.21.15.29:0/4130570690 reap_dead start 2023-10-14T18:20:15.993 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.983+0000 7f6fb3180640 1 -- 172.21.15.29:0/4130570690 shutdown_connections 2023-10-14T18:20:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.983+0000 7f6fb3180640 1 -- 172.21.15.29:0/4130570690 >> 172.21.15.29:0/4130570690 conn(0x7f6fac0ce6a0 msgr2=0x7f6fac0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.983+0000 7f6fb3180640 1 -- 172.21.15.29:0/4130570690 shutdown_connections 2023-10-14T18:20:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.983+0000 7f6fb3180640 1 -- 172.21.15.29:0/4130570690 wait complete. 2023-10-14T18:20:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.985+0000 7f6fb3180640 1 Processor -- start 2023-10-14T18:20:15.994 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.985+0000 7f6fb3180640 1 -- start start 2023-10-14T18:20:15.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6fac1e6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fac1e54b0 0x7f6fac0c4150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac1588a0 0x7f6fac0ae090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:15.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1e6630 con 0x7f6f98003970 2023-10-14T18:20:15.995 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1e59b0 con 0x7f6fac1e54b0 2023-10-14T18:20:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb3180640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fac1e5c90 con 0x7f6fac1588a0 2023-10-14T18:20:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6fac1e6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6fac1e6a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43550/0 (socket says 172.21.15.29:43550) 2023-10-14T18:20:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 -- 172.21.15.29:0/3244327011 learned_addr learned my addr 172.21.15.29:0/3244327011 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:15.996 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb197d640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fac1e54b0 0x7f6fac0c4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:15.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 -- 172.21.15.29:0/3244327011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac1588a0 msgr2=0x7f6fac0ae090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:15.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6fac1588a0 0x7f6fac0ae090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 -- 172.21.15.29:0/3244327011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fac1e54b0 msgr2=0x7f6fac0c4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:15.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6fac1e54b0 0x7f6fac0c4150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:15.997 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.986+0000 7f6fb217e640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fac1e5f70 con 0x7f6f98003970 2023-10-14T18:20:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fb217e640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6fac1e6a50 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f6fa802c2b0 tx=0x7f6fa8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003810 con 0x7f6f98003970 2023-10-14T18:20:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fa802de90 con 0x7f6f98003970 2023-10-14T18:20:15.998 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fa8003b90 con 0x7f6f98003970 2023-10-14T18:20:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fac1e6250 con 0x7f6f98003970 2023-10-14T18:20:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.987+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fac158fb0 con 0x7f6f98003970 2023-10-14T18:20:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.988+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6fac0cc550 con 0x7f6f98003970 2023-10-14T18:20:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.989+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6fa8003cf0 con 0x7f6f98003970 2023-10-14T18:20:15.999 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.990+0000 7f6fa2ffd640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6f9006afc0 0x7f6f9006d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.990+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f6fa80aba90 con 0x7f6f98003970 2023-10-14T18:20:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fb197d640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6f9006afc0 0x7f6f9006d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fa2ffd640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f6f900715c0 0x7f6f900739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6f900740b0 con 0x7f6f900715c0 2023-10-14T18:20:16.000 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6fa80b02a0 con 0x7f6f98003970 2023-10-14T18:20:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fb297f640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f6f900715c0 0x7f6f900739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.991+0000 7f6fb197d640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6f9006afc0 0x7f6f9006d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0aefd0 tx=0x7f6f9c008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.992+0000 7f6fb297f640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f6f900715c0 0x7f6f900739e0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.002 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:15.992+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== osd.0 v2:172.21.15.29:6800/3107383011 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f6f900740b0 con 0x7f6f900715c0 2023-10-14T18:20:16.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.007+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 --> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6fac0cc270 con 0x7f6f900715c0 2023-10-14T18:20:16.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fa2ffd640 1 -- 172.21.15.29:0/3244327011 <== osd.0 v2:172.21.15.29:6800/3107383011 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6fa4001e40 con 0x7f6f900715c0 2023-10-14T18:20:16.010 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:16.010 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:16.010 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f6f900715c0 msgr2=0x7f6f900739e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6800/3107383011,v1:172.21.15.29:6801/3107383011] conn(0x7f6f900715c0 0x7f6f900739e0 crc :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6f9006afc0 msgr2=0x7f6f9006d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6f9006afc0 0x7f6f9006d480 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6fac0aefd0 tx=0x7f6f9c008040 comp rx=0 tx=0).stop 2023-10-14T18:20:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 msgr2=0x7f6fac1e6a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb3180640 1 --2- 172.21.15.29:0/3244327011 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6f98003970 0x7f6fac1e6a50 secure :-1 s=READY pgs=3287 cs=0 l=1 rev1=1 crypto rx=0x7f6fa802c2b0 tx=0x7f6fa8004730 comp rx=0 tx=0).stop 2023-10-14T18:20:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.008+0000 7f6fb297f640 1 -- 172.21.15.29:0/3244327011 reap_dead start 2023-10-14T18:20:16.012 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.009+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 shutdown_connections 2023-10-14T18:20:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.009+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 >> 172.21.15.29:0/3244327011 conn(0x7f6fac0ce6a0 msgr2=0x7f6fac15d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.009+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 shutdown_connections 2023-10-14T18:20:16.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.009+0000 7f6fb3180640 1 -- 172.21.15.29:0/3244327011 wait complete. 2023-10-14T18:20:16.028 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-10-14T18:20:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.254+0000 7ffb7f59e640 1 Processor -- start 2023-10-14T18:20:16.256 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.254+0000 7ffb7f59e640 1 -- start start 2023-10-14T18:20:16.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7f59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ffb800cded0 0x7ffb800b5c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7f59e640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb8015d150 con 0x7ffb8015cd70 2023-10-14T18:20:16.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7f59e640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb800d1180 con 0x7ffb800b8480 2023-10-14T18:20:16.258 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7f59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb800d1460 con 0x7ffb800cded0 2023-10-14T18:20:16.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7e59c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7ffb8015cd70 0x7ffb800d2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55634/0 (socket says 172.21.15.29:55634) 2023-10-14T18:20:16.259 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7e59c640 1 -- 172.21.15.29:0/3754133526 learned_addr learned my addr 172.21.15.29:0/3754133526 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.256+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/3754133526 >> v2:172.21.15.186:3300/0 conn(0x7ffb800cded0 0x7ffb800b5c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773543567 0 0) 0x7ffb8015d150 con 0x7ffb8015cd70 2023-10-14T18:20:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c003520 con 0x7ffb8015cd70 2023-10-14T18:20:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1518171425 0 0) 0x7ffb6c003520 con 0x7ffb8015cd70 2023-10-14T18:20:16.261 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 >> v2:172.21.15.186:3300/0 conn(0x7ffb800cded0 msgr2=0x7ffb800b5c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 --2- 172.21.15.29:0/3754133526 >> v2:172.21.15.186:3300/0 conn(0x7ffb800cded0 0x7ffb800b5c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 >> v1:172.21.15.186:6789/0 conn(0x7ffb800b8480 legacy=0x7ffb800cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb800d1740 con 0x7ffb8015cd70 2023-10-14T18:20:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7ffb700019f0 con 0x7ffb8015cd70 2023-10-14T18:20:16.262 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 >> v1:172.21.15.29:6789/0 conn(0x7ffb8015cd70 legacy=0x7ffb800d2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 0x7ffb6c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb6c0047c0 0x7ffb6c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c005180 0x7ffb6c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800d1460 con 0x7ffb6c003970 2023-10-14T18:20:16.263 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c00c020 con 0x7ffb6c0047c0 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.257+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c00c1e0 con 0x7ffb6c005180 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 0x7ffb6c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c005180 0x7ffb6c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7dd9b640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb6c0047c0 0x7ffb6c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 -- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c005180 msgr2=0x7ffb6c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:16.264 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c005180 0x7ffb6c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 -- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb6c0047c0 msgr2=0x7ffb6c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:16.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb6c0047c0 0x7ffb6c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7e59c640 1 -- 172.21.15.29:0/3754133526 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb6c003520 con 0x7ffb6c003970 2023-10-14T18:20:16.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7dd9b640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb6c0047c0 0x7ffb6c004c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:16.265 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c005180 0x7ffb6c00b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:16.266 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.258+0000 7ffb7ed9d640 1 -- 172.21.15.29:0/3754133526 reap_dead start 2023-10-14T18:20:16.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.259+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 0x7ffb6c003d90 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000a830 tx=0x7ffb6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc23a8b9a1a8c1f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700072d0 con 0x7ffb6c003970 2023-10-14T18:20:16.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7002f070 con 0x7ffb6c003970 2023-10-14T18:20:16.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7d59a640 1 -- 172.21.15.29:0/3754133526 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb700076f0 con 0x7ffb6c003970 2023-10-14T18:20:16.269 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7f59e640 1 -- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 msgr2=0x7ffb6c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 0x7ffb6c003d90 secure :-1 s=READY pgs=3289 cs=0 l=1 rev1=1 crypto rx=0x7ffb7000a830 tx=0x7ffb6c0057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7f59e640 1 -- 172.21.15.29:0/3754133526 shutdown_connections 2023-10-14T18:20:16.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/3754133526 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb6c003970 0x7ffb6c003d90 unknown :-1 s=CLOSED pgs=3289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.260+0000 7ffb7f59e640 1 -- 172.21.15.29:0/3754133526 >> 172.21.15.29:0/3754133526 conn(0x7ffb800cf570 msgr2=0x7ffb800cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:16.270 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.261+0000 7ffb7f59e640 1 -- 172.21.15.29:0/3754133526 shutdown_connections 2023-10-14T18:20:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.261+0000 7ffb7f59e640 1 -- 172.21.15.29:0/3754133526 wait complete. 2023-10-14T18:20:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.262+0000 7ffb7f59e640 1 Processor -- start 2023-10-14T18:20:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 -- start start 2023-10-14T18:20:16.271 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c003970 0x7ffb8014c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 0x7ffb801f1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800d3440 con 0x7ffb800d2a80 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb800a9cd0 con 0x7ffb800a94f0 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.263+0000 7ffb7f59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb80149a70 con 0x7ffb6c003970 2023-10-14T18:20:16.272 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43560/0 (socket says 172.21.15.29:43560) 2023-10-14T18:20:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 -- 172.21.15.29:0/2597575143 learned_addr learned my addr 172.21.15.29:0/2597575143 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7dd9b640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 0x7ffb801f1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c003970 0x7ffb8014c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.273 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 -- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c003970 msgr2=0x7ffb8014c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c003970 0x7ffb8014c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 -- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 msgr2=0x7ffb801f1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 0x7ffb801f1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7ed9d640 1 -- 172.21.15.29:0/2597575143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb80149d50 con 0x7ffb800d2a80 2023-10-14T18:20:16.274 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.264+0000 7ffb7dd9b640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 0x7ffb801f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb7ed9d640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7ffb800cd0f0 tx=0x7ffb780109f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78014800 con 0x7ffb800d2a80 2023-10-14T18:20:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb78014980 con 0x7ffb800d2a80 2023-10-14T18:20:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8014a030 con 0x7ffb800d2a80 2023-10-14T18:20:16.275 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb78014d30 con 0x7ffb800d2a80 2023-10-14T18:20:16.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.265+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb800a9970 con 0x7ffb800d2a80 2023-10-14T18:20:16.276 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.266+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8014a550 con 0x7ffb800d2a80 2023-10-14T18:20:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.267+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ffb7801c400 con 0x7ffb800d2a80 2023-10-14T18:20:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.269+0000 7ffb76ffd640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 0x7ffb6006d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.269+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7ffb78089920 con 0x7ffb800d2a80 2023-10-14T18:20:16.277 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.269+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 0x7ffb6006d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.270+0000 7ffb7e59c640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 0x7ffb6006d4d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb70034000 tx=0x7ffb70007000 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.278 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.273+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7ffb8014a550 con 0x7ffb800d2a80 2023-10-14T18:20:16.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.472+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7ffb800d1d00 con 0x7ffb6006b010 2023-10-14T18:20:16.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.472+0000 7ffb76ffd640 1 -- 172.21.15.29:0/2597575143 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7ffb700019b0 con 0x7ffb6006b010 2023-10-14T18:20:16.475 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 0 to deep-scrub 2023-10-14T18:20:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.475+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 msgr2=0x7ffb6006d4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.475+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 0x7ffb6006d4d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ffb70034000 tx=0x7ffb70007000 comp rx=0 tx=0).stop 2023-10-14T18:20:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.475+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 msgr2=0x7ffb800d2f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.475+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 secure :-1 s=READY pgs=3290 cs=0 l=1 rev1=1 crypto rx=0x7ffb800cd0f0 tx=0x7ffb780109f0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 shutdown_connections 2023-10-14T18:20:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ffb800d2a80 0x7ffb800d2f00 unknown :-1 s=CLOSED pgs=3290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ffb800a94f0 0x7ffb801f1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ffb6006b010 0x7ffb6006d4d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 --2- 172.21.15.29:0/2597575143 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ffb6c003970 0x7ffb8014c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 >> 172.21.15.29:0/2597575143 conn(0x7ffb800cf570 msgr2=0x7ffb800c8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 shutdown_connections 2023-10-14T18:20:16.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.476+0000 7ffb7f59e640 1 -- 172.21.15.29:0/2597575143 wait complete. 2023-10-14T18:20:16.498 INFO:tasks.ceph:Scrubbing osd.1 2023-10-14T18:20:16.498 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.722+0000 7f4f32659640 1 Processor -- start 2023-10-14T18:20:16.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.722+0000 7f4f32659640 1 -- start start 2023-10-14T18:20:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.723+0000 7f4f32659640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.723+0000 7f4f32659640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c15dde0 0x7f4f2c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.723+0000 7f4f32659640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f2c160a90 con 0x7f4f2c0c9230 2023-10-14T18:20:16.727 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.723+0000 7f4f32659640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f2c0cdec0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.723+0000 7f4f32659640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0cd900 con 0x7f4f2c15dde0 2023-10-14T18:20:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43568/0 (socket says 172.21.15.29:43568) 2023-10-14T18:20:16.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 -- 172.21.15.29:0/3482716499 learned_addr learned my addr 172.21.15.29:0/3482716499 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 -- 172.21.15.29:0/3482716499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c15dde0 msgr2=0x7f4f2c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 --2- 172.21.15.29:0/3482716499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c15dde0 0x7f4f2c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 -- 172.21.15.29:0/3482716499 >> v1:172.21.15.29:6789/0 conn(0x7f4f2c0c9230 legacy=0x7f4f2c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.724+0000 7f4f31657640 1 -- 172.21.15.29:0/3482716499 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f2c0cdbe0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.726+0000 7f4f31657640 1 --2- 172.21.15.29:0/3482716499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009870 tx=0x7f4f2802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81d55d8eb68f4769 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.726+0000 7f4f23fff640 1 -- 172.21.15.29:0/3482716499 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28030840 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.726+0000 7f4f23fff640 1 -- 172.21.15.29:0/3482716499 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f280309a0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.726+0000 7f4f23fff640 1 -- 172.21.15.29:0/3482716499 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f28030d30 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 -- 172.21.15.29:0/3482716499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 msgr2=0x7f4f2c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 --2- 172.21.15.29:0/3482716499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 secure :-1 s=READY pgs=3292 cs=0 l=1 rev1=1 crypto rx=0x7f4f28009870 tx=0x7f4f2802b980 comp rx=0 tx=0).stop 2023-10-14T18:20:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 -- 172.21.15.29:0/3482716499 shutdown_connections 2023-10-14T18:20:16.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 --2- 172.21.15.29:0/3482716499 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c15dde0 0x7f4f2c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 --2- 172.21.15.29:0/3482716499 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c0c8cf0 unknown :-1 s=CLOSED pgs=3292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 -- 172.21.15.29:0/3482716499 >> 172.21.15.29:0/3482716499 conn(0x7f4f2c0ce6a0 msgr2=0x7f4f2c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:16.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 -- 172.21.15.29:0/3482716499 shutdown_connections 2023-10-14T18:20:16.733 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.727+0000 7f4f32659640 1 -- 172.21.15.29:0/3482716499 wait complete. 2023-10-14T18:20:16.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.729+0000 7f4f32659640 1 Processor -- start 2023-10-14T18:20:16.734 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.729+0000 7f4f32659640 1 -- start start 2023-10-14T18:20:16.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c0c9230 0x7f4f2c0c5770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.736 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c159aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f2c15dde0 0x7f4f2c0c7c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0aff70 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c87f0 con 0x7f4f2c15dde0 2023-10-14T18:20:16.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f32659640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f2c0c3c50 con 0x7f4f2c0c9230 2023-10-14T18:20:16.737 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f30e56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c159aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f30e56640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c159aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43574/0 (socket says 172.21.15.29:43574) 2023-10-14T18:20:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f30e56640 1 -- 172.21.15.29:0/866016780 learned_addr learned my addr 172.21.15.29:0/866016780 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f31657640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c0c9230 0x7f4f2c0c5770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f31e58640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f2c15dde0 0x7f4f2c0c7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.738 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f30e56640 1 -- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c0c9230 msgr2=0x7f4f2c0c5770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.730+0000 7f4f30e56640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f4f2c0c9230 0x7f4f2c0c5770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f30e56640 1 -- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f2c15dde0 msgr2=0x7f4f2c0c7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f30e56640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f4f2c15dde0 0x7f4f2c0c7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.739 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f30e56640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f280094d0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f30e56640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c159aa0 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c450 tx=0x7f4f1c00c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c0052d0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f1c005430 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c012870 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.740 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f2c0c3f30 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.731+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f2c0c3900 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.732+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f2c0704c0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.733+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f4f1c0129d0 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.734+0000 7f4f21ffb640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f4f1006af90 0x7f4f1006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.741 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.734+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f4f1c089380 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.735+0000 7f4f21ffb640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7f4f10071590 0x7f4f100739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.735+0000 7f4f31657640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f4f1006af90 0x7f4f1006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.735+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f10074080 con 0x7f4f10071590 2023-10-14T18:20:16.742 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.735+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4f1c08b960 con 0x7f4f2c15c9f0 2023-10-14T18:20:16.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.735+0000 7f4f31e58640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7f4f10071590 0x7f4f100739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:16.743 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.736+0000 7f4f31657640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f4f1006af90 0x7f4f1006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f28036000 tx=0x7f4f2803b040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.736+0000 7f4f31e58640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7f4f10071590 0x7f4f100739b0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:16.744 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.736+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== osd.1 v2:172.21.15.29:6814/3723048683 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f4f10074080 con 0x7f4f10071590 2023-10-14T18:20:16.752 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.751+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 --> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f2c0701e0 con 0x7f4f10071590 2023-10-14T18:20:16.753 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.751+0000 7f4f21ffb640 1 -- 172.21.15.29:0/866016780 <== osd.1 v2:172.21.15.29:6814/3723048683 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f4f24007930 con 0x7f4f10071590 2023-10-14T18:20:16.753 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:16.754 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:16.754 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7f4f10071590 msgr2=0x7f4f100739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6814/3723048683,v1:172.21.15.29:6815/3723048683] conn(0x7f4f10071590 0x7f4f100739b0 crc :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:16.755 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f4f1006af90 msgr2=0x7f4f1006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f4f1006af90 0x7f4f1006d450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4f28036000 tx=0x7f4f2803b040 comp rx=0 tx=0).stop 2023-10-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 msgr2=0x7f4f2c159aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 --2- 172.21.15.29:0/866016780 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f4f2c15c9f0 0x7f4f2c159aa0 secure :-1 s=READY pgs=3293 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c00c450 tx=0x7f4f1c00c910 comp rx=0 tx=0).stop 2023-10-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f31e58640 1 -- 172.21.15.29:0/866016780 reap_dead start 2023-10-14T18:20:16.756 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 shutdown_connections 2023-10-14T18:20:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 >> 172.21.15.29:0/866016780 conn(0x7f4f2c0ce6a0 msgr2=0x7f4f2c15a410 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 shutdown_connections 2023-10-14T18:20:16.757 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.752+0000 7f4f32659640 1 -- 172.21.15.29:0/866016780 wait complete. 2023-10-14T18:20:16.773 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-10-14T18:20:17.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.999+0000 7f36a0c1a640 1 Processor -- start 2023-10-14T18:20:17.001 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:16.999+0000 7f36a0c1a640 1 -- start start 2023-10-14T18:20:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.001+0000 7f36a0c1a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f369c06c040 0x7f369c06c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.001+0000 7f36a0c1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c15ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.001+0000 7f36a0c1a640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f369c06c960 con 0x7f369c0ad970 2023-10-14T18:20:17.003 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.001+0000 7f36a0c1a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f369c15b9d0 con 0x7f369c06c040 2023-10-14T18:20:17.004 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.001+0000 7f36a0c1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c15bcb0 con 0x7f369c0cda80 2023-10-14T18:20:17.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.002+0000 7f3693fff640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7f369c0ad970 0x7f369c15e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45870/0 (socket says 172.21.15.29:45870) 2023-10-14T18:20:17.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.002+0000 7f3693fff640 1 -- 172.21.15.29:0/3203697915 learned_addr learned my addr 172.21.15.29:0/3203697915 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:17.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.002+0000 7f369bfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c15ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.002+0000 7f369b7fe640 1 --2- 172.21.15.29:0/3203697915 >> v2:172.21.15.186:3300/0 conn(0x7f369c06c040 0x7f369c06c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.005 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369affd640 1 -- 172.21.15.29:0/3203697915 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935876668 0 0) 0x7f369c06c960 con 0x7f369c0ad970 2023-10-14T18:20:17.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369affd640 1 -- 172.21.15.29:0/3203697915 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3688003520 con 0x7f369c0ad970 2023-10-14T18:20:17.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369bfff640 1 -- 172.21.15.29:0/3203697915 >> v2:172.21.15.186:3300/0 conn(0x7f369c06c040 msgr2=0x7f369c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:17.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369bfff640 1 --2- 172.21.15.29:0/3203697915 >> v2:172.21.15.186:3300/0 conn(0x7f369c06c040 0x7f369c06c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.006 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369bfff640 1 -- 172.21.15.29:0/3203697915 >> v1:172.21.15.186:6789/0 conn(0x7f369c0ad970 legacy=0x7f369c15e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.003+0000 7f369bfff640 1 -- 172.21.15.29:0/3203697915 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c15bf90 con 0x7f369c0cda80 2023-10-14T18:20:17.007 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.005+0000 7f369bfff640 1 --2- 172.21.15.29:0/3203697915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c15ed90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3694009d90 tx=0x7f369402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e630ba28795c4776 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.005+0000 7f369affd640 1 -- 172.21.15.29:0/3203697915 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f369402cc50 con 0x7f369c0cda80 2023-10-14T18:20:17.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.005+0000 7f369affd640 1 -- 172.21.15.29:0/3203697915 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f369402cdb0 con 0x7f369c0cda80 2023-10-14T18:20:17.008 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.005+0000 7f369affd640 1 -- 172.21.15.29:0/3203697915 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3694034660 con 0x7f369c0cda80 2023-10-14T18:20:17.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 -- 172.21.15.29:0/3203697915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 msgr2=0x7f369c15ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/3203697915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c15ed90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3694009d90 tx=0x7f369402bea0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 -- 172.21.15.29:0/3203697915 shutdown_connections 2023-10-14T18:20:17.009 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/3203697915 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c15ed90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/3203697915 >> v2:172.21.15.186:3300/0 conn(0x7f369c06c040 0x7f369c06c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 -- 172.21.15.29:0/3203697915 >> 172.21.15.29:0/3203697915 conn(0x7f369c0ce6a0 msgr2=0x7f369c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 -- 172.21.15.29:0/3203697915 shutdown_connections 2023-10-14T18:20:17.010 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.006+0000 7f36a0c1a640 1 -- 172.21.15.29:0/3203697915 wait complete. 2023-10-14T18:20:17.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.009+0000 7f36a0c1a640 1 Processor -- start 2023-10-14T18:20:17.011 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.009+0000 7f36a0c1a640 1 -- start start 2023-10-14T18:20:17.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f369c06c040 0x7f369c156c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c0c9a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.013 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0c67c0 con 0x7f369c0ad970 2023-10-14T18:20:17.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0b0cc0 con 0x7f369c06c040 2023-10-14T18:20:17.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f36a0c1a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f369c0c69d0 con 0x7f369c0cda80 2023-10-14T18:20:17.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f3693fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.010+0000 7f3693fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43584/0 (socket says 172.21.15.29:43584) 2023-10-14T18:20:17.014 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 -- 172.21.15.29:0/26446706 learned_addr learned my addr 172.21.15.29:0/26446706 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f369b7fe640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f369c06c040 0x7f369c156c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f369bfff640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c0c9a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 -- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 msgr2=0x7f369c0c9a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.015 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c0c9a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 -- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f369c06c040 msgr2=0x7f369c156c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f369c06c040 0x7f369c156c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 -- 172.21.15.29:0/26446706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3694009a70 con 0x7f369c0ad970 2023-10-14T18:20:17.016 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f369bfff640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c0c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:20:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.011+0000 7f3693fff640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7f368400c4d0 tx=0x7f368400c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.012+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684005ce0 con 0x7f369c0ad970 2023-10-14T18:20:17.018 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.012+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3684005e40 con 0x7f369c0ad970 2023-10-14T18:20:17.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.012+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3684007270 con 0x7f369c0ad970 2023-10-14T18:20:17.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.012+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c0c6cb0 con 0x7f369c0ad970 2023-10-14T18:20:17.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.012+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c0c7270 con 0x7f369c0ad970 2023-10-14T18:20:17.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.013+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c06b330 con 0x7f369c0ad970 2023-10-14T18:20:17.019 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.014+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f3684007490 con 0x7f369c0ad970 2023-10-14T18:20:17.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.015+0000 7f3698ff9640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 0x7f367c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.015+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f3684088dc0 con 0x7f369c0ad970 2023-10-14T18:20:17.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.015+0000 7f369b7fe640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 0x7f367c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.020 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.016+0000 7f369b7fe640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 0x7f367c06d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f368c009aa0 tx=0x7f368c009190 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.021 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.019+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f3684053cc0 con 0x7f369c0ad970 2023-10-14T18:20:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.217+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f369c0cc550 con 0x7f367c06af90 2023-10-14T18:20:17.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.218+0000 7f3698ff9640 1 -- 172.21.15.29:0/26446706 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f368c011a00 con 0x7f367c06af90 2023-10-14T18:20:17.220 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 1 to deep-scrub 2023-10-14T18:20:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 msgr2=0x7f367c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 0x7f367c06d450 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f368c009aa0 tx=0x7f368c009190 comp rx=0 tx=0).stop 2023-10-14T18:20:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 msgr2=0x7f369c0c7a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 secure :-1 s=READY pgs=3294 cs=0 l=1 rev1=1 crypto rx=0x7f368400c4d0 tx=0x7f368400c990 comp rx=0 tx=0).stop 2023-10-14T18:20:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 shutdown_connections 2023-10-14T18:20:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f367c06af90 0x7f367c06d450 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f369c0cda80 0x7f369c0c9a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f369c0ad970 0x7f369c0c7a70 unknown :-1 s=CLOSED pgs=3294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 --2- 172.21.15.29:0/26446706 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f369c06c040 0x7f369c156c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 >> 172.21.15.29:0/26446706 conn(0x7f369c0ce6a0 msgr2=0x7f369c06f700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 shutdown_connections 2023-10-14T18:20:17.224 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.220+0000 7f36a0c1a640 1 -- 172.21.15.29:0/26446706 wait complete. 2023-10-14T18:20:17.240 INFO:tasks.ceph:Scrubbing osd.2 2023-10-14T18:20:17.240 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:17.461 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.459+0000 7f746d733640 1 Processor -- start 2023-10-14T18:20:17.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.460+0000 7f746d733640 1 -- start start 2023-10-14T18:20:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.461+0000 7f746d733640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f746815ac10 0x7f746806c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.461+0000 7f746d733640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f74680b0360 0x7f74680b07f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f746d733640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74680ca510 0x7f74680ca920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f746d733640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74680b0d30 con 0x7f746815ac10 2023-10-14T18:20:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f746d733640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f746815b310 con 0x7f74680b0360 2023-10-14T18:20:17.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f746d733640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f746806f640 con 0x7f74680ca510 2023-10-14T18:20:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f7466ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f746815ac10 0x7f746806c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f7466ffd640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f746815ac10 0x7f746806c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43598/0 (socket says 172.21.15.29:43598) 2023-10-14T18:20:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f7466ffd640 1 -- 172.21.15.29:0/1907155464 learned_addr learned my addr 172.21.15.29:0/1907155464 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:17.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f745e7fc640 1 --2- 172.21.15.29:0/1907155464 >> v2:172.21.15.186:3300/0 conn(0x7f74680b0360 0x7f74680b07f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.462+0000 7f74677fe640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74680ca510 0x7f74680ca920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.463+0000 7f7466ffd640 1 -- 172.21.15.29:0/1907155464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74680ca510 msgr2=0x7f74680ca920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.463+0000 7f7466ffd640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74680ca510 0x7f74680ca920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.463+0000 7f7466ffd640 1 -- 172.21.15.29:0/1907155464 >> v2:172.21.15.186:3300/0 conn(0x7f74680b0360 msgr2=0x7f74680b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:17.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.463+0000 7f7466ffd640 1 --2- 172.21.15.29:0/1907155464 >> v2:172.21.15.186:3300/0 conn(0x7f74680b0360 0x7f74680b07f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.463+0000 7f7466ffd640 1 -- 172.21.15.29:0/1907155464 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f746815b5b0 con 0x7f746815ac10 2023-10-14T18:20:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.464+0000 7f7466ffd640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746815ac10 0x7f746806c730 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7f74680ae2b0 tx=0x7f745802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df76e72c70d33d2 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.464+0000 7f74667fc640 1 -- 172.21.15.29:0/1907155464 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74580317e0 con 0x7f746815ac10 2023-10-14T18:20:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.464+0000 7f74667fc640 1 -- 172.21.15.29:0/1907155464 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7458031940 con 0x7f746815ac10 2023-10-14T18:20:17.468 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.464+0000 7f74667fc640 1 -- 172.21.15.29:0/1907155464 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7458031cd0 con 0x7f746815ac10 2023-10-14T18:20:17.469 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 -- 172.21.15.29:0/1907155464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746815ac10 msgr2=0x7f746806c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746815ac10 0x7f746806c730 secure :-1 s=READY pgs=3295 cs=0 l=1 rev1=1 crypto rx=0x7f74680ae2b0 tx=0x7f745802b880 comp rx=0 tx=0).stop 2023-10-14T18:20:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 -- 172.21.15.29:0/1907155464 shutdown_connections 2023-10-14T18:20:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f74680ca510 0x7f74680ca920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 --2- 172.21.15.29:0/1907155464 >> v2:172.21.15.186:3300/0 conn(0x7f74680b0360 0x7f74680b07f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 --2- 172.21.15.29:0/1907155464 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f746815ac10 0x7f746806c730 unknown :-1 s=CLOSED pgs=3295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 -- 172.21.15.29:0/1907155464 >> 172.21.15.29:0/1907155464 conn(0x7f74680cb360 msgr2=0x7f74680cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 -- 172.21.15.29:0/1907155464 shutdown_connections 2023-10-14T18:20:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.465+0000 7f746d733640 1 -- 172.21.15.29:0/1907155464 wait complete. 2023-10-14T18:20:17.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.467+0000 7f746d733640 1 Processor -- start 2023-10-14T18:20:17.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.467+0000 7f746d733640 1 -- start start 2023-10-14T18:20:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74680b0360 0x7f7468153280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 0x7f74681585c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f746815ac10 0x7f74680b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7468159b90 con 0x7f74680ca510 2023-10-14T18:20:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7468158f10 con 0x7f74680b0360 2023-10-14T18:20:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.468+0000 7f746d733640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74681591f0 con 0x7f746815ac10 2023-10-14T18:20:17.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 0x7f74681585c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 0x7f74681585c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43612/0 (socket says 172.21.15.29:43612) 2023-10-14T18:20:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 -- 172.21.15.29:0/2878760777 learned_addr learned my addr 172.21.15.29:0/2878760777 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f7466ffd640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74680b0360 0x7f7468153280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f74677fe640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f746815ac10 0x7f74680b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 -- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f746815ac10 msgr2=0x7f74680b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f746815ac10 0x7f74680b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 -- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74680b0360 msgr2=0x7f7468153280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f74680b0360 0x7f7468153280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74580094d0 con 0x7f74680ca510 2023-10-14T18:20:17.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.469+0000 7f745e7fc640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 0x7f74681585c0 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7f745000d800 tx=0x7f745000dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.470+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74500096a0 con 0x7f74680ca510 2023-10-14T18:20:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.470+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7450009800 con 0x7f74680ca510 2023-10-14T18:20:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.470+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7450009990 con 0x7f74680ca510 2023-10-14T18:20:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.470+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74681594d0 con 0x7f74680ca510 2023-10-14T18:20:17.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.470+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7468159fb0 con 0x7f74680ca510 2023-10-14T18:20:17.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.471+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f746815bb70 con 0x7f74680ca510 2023-10-14T18:20:17.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.472+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7450009af0 con 0x7f74680ca510 2023-10-14T18:20:17.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f745ffff640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f744c06af90 0x7f744c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.481 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f7450089c20 con 0x7f74680ca510 2023-10-14T18:20:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f7466ffd640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f744c06af90 0x7f744c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f745ffff640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f744c071590 0x7f744c0739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f744c074080 con 0x7f744c071590 2023-10-14T18:20:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.473+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f745008e2a0 con 0x7f74680ca510 2023-10-14T18:20:17.482 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.474+0000 7f74677fe640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f744c071590 0x7f744c0739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.474+0000 7f7466ffd640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f744c06af90 0x7f744c06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7458036000 tx=0x7f745803d040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.474+0000 7f74677fe640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f744c071590 0x7f744c0739b0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.483 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.475+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== osd.2 v2:172.21.15.29:6808/3586046734 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f744c074080 con 0x7f744c071590 2023-10-14T18:20:17.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.489+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 --> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7468159360 con 0x7f744c071590 2023-10-14T18:20:17.491 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.490+0000 7f745ffff640 1 -- 172.21.15.29:0/2878760777 <== osd.2 v2:172.21.15.29:6808/3586046734 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7468159360 con 0x7f744c071590 2023-10-14T18:20:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:17.492 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:17.492 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:17.493 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.490+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f744c071590 msgr2=0x7f744c0739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.490+0000 7f746d733640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6808/3586046734,v1:172.21.15.29:6809/3586046734] conn(0x7f744c071590 0x7f744c0739b0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.490+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f744c06af90 msgr2=0x7f744c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.490+0000 7f746d733640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f744c06af90 0x7f744c06d450 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f7458036000 tx=0x7f745803d040 comp rx=0 tx=0).stop 2023-10-14T18:20:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 msgr2=0x7f74681585c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.494 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 --2- 172.21.15.29:0/2878760777 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f74680ca510 0x7f74681585c0 secure :-1 s=READY pgs=3296 cs=0 l=1 rev1=1 crypto rx=0x7f745000d800 tx=0x7f745000dcc0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f74677fe640 1 -- 172.21.15.29:0/2878760777 reap_dead start 2023-10-14T18:20:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 shutdown_connections 2023-10-14T18:20:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 >> 172.21.15.29:0/2878760777 conn(0x7f74680cb360 msgr2=0x7f746806bb80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 shutdown_connections 2023-10-14T18:20:17.495 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.491+0000 7f746d733640 1 -- 172.21.15.29:0/2878760777 wait complete. 2023-10-14T18:20:17.509 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-10-14T18:20:17.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.702+0000 7fdbe850f640 1 Processor -- start 2023-10-14T18:20:17.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.703+0000 7fdbe850f640 1 -- start start 2023-10-14T18:20:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.704+0000 7fdbe850f640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe00661c0 0x7fdbe00ce9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.704+0000 7fdbe850f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe006cee0 con 0x7fdbe01618b0 2023-10-14T18:20:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.704+0000 7fdbe850f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdbe00d0890 con 0x7fdbe015d330 2023-10-14T18:20:17.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.704+0000 7fdbe850f640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdbe006b1e0 con 0x7fdbe00661c0 2023-10-14T18:20:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe00661c0 0x7fdbe00ce9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fdbe00661c0 0x7fdbe00ce9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43614/0 (socket says 172.21.15.29:43614) 2023-10-14T18:20:17.708 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 -- 172.21.15.29:0/2362624804 learned_addr learned my addr 172.21.15.29:0/2362624804 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 -- 172.21.15.29:0/2362624804 >> v1:172.21.15.186:6789/0 conn(0x7fdbe015d330 legacy=0x7fdbe006c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 -- 172.21.15.29:0/2362624804 >> v1:172.21.15.29:6789/0 conn(0x7fdbe01618b0 legacy=0x7fdbe0065ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.709 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.705+0000 7fdbe6aac640 1 -- 172.21.15.29:0/2362624804 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbe00d05b0 con 0x7fdbe00661c0 2023-10-14T18:20:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.706+0000 7fdbe6aac640 1 --2- 172.21.15.29:0/2362624804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe00661c0 0x7fdbe00ce9b0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e50 tx=0x7fdbd802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4914ce7cc5aeb70c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.713 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe52a9640 1 -- 172.21.15.29:0/2362624804 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd802cb10 con 0x7fdbe00661c0 2023-10-14T18:20:17.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe52a9640 1 -- 172.21.15.29:0/2362624804 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd802cc70 con 0x7fdbe00661c0 2023-10-14T18:20:17.714 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe52a9640 1 -- 172.21.15.29:0/2362624804 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd80343f0 con 0x7fdbe00661c0 2023-10-14T18:20:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 -- 172.21.15.29:0/2362624804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe00661c0 msgr2=0x7fdbe00ce9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 --2- 172.21.15.29:0/2362624804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe00661c0 0x7fdbe00ce9b0 secure :-1 s=READY pgs=3298 cs=0 l=1 rev1=1 crypto rx=0x7fdbd8009e50 tx=0x7fdbd802bf70 comp rx=0 tx=0).stop 2023-10-14T18:20:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 -- 172.21.15.29:0/2362624804 shutdown_connections 2023-10-14T18:20:17.715 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 --2- 172.21.15.29:0/2362624804 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe00661c0 0x7fdbe00ce9b0 unknown :-1 s=CLOSED pgs=3298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 -- 172.21.15.29:0/2362624804 >> 172.21.15.29:0/2362624804 conn(0x7fdbe00d0ab0 msgr2=0x7fdbe00d0e90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 -- 172.21.15.29:0/2362624804 shutdown_connections 2023-10-14T18:20:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.707+0000 7fdbe850f640 1 -- 172.21.15.29:0/2362624804 wait complete. 2023-10-14T18:20:17.716 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.709+0000 7fdbe850f640 1 Processor -- start 2023-10-14T18:20:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.709+0000 7fdbe850f640 1 -- start start 2023-10-14T18:20:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe00661c0 0x7fdbe00b2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.717 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbe01618b0 0x7fdbe015b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00adcd0 con 0x7fdbe015d330 2023-10-14T18:20:17.718 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00b3280 con 0x7fdbe00661c0 2023-10-14T18:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.710+0000 7fdbe850f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbe00aef00 con 0x7fdbe01618b0 2023-10-14T18:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.721 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43618/0 (socket says 172.21.15.29:43618) 2023-10-14T18:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 -- 172.21.15.29:0/4167704595 learned_addr learned my addr 172.21.15.29:0/4167704595 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe62ab640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe00661c0 0x7fdbe00b2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.722 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe6aac640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbe01618b0 0x7fdbe015b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 -- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbe01618b0 msgr2=0x7fdbe015b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.723 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbe01618b0 0x7fdbe015b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 -- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe00661c0 msgr2=0x7fdbe00b2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe00661c0 0x7fdbe00b2d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.724 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.711+0000 7fdbe5aaa640 1 -- 172.21.15.29:0/4167704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8009b30 con 0x7fdbe015d330 2023-10-14T18:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbe5aaa640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000c400 tx=0x7fdbd000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd00052d0 con 0x7fdbe015d330 2023-10-14T18:20:17.728 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdbd0005430 con 0x7fdbe015d330 2023-10-14T18:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbd0003830 con 0x7fdbe015d330 2023-10-14T18:20:17.729 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbe00ad710 con 0x7fdbe015d330 2023-10-14T18:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.712+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbe00ae1c0 con 0x7fdbe015d330 2023-10-14T18:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.713+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe004e160 con 0x7fdbe015d330 2023-10-14T18:20:17.730 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.714+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdbd0003990 con 0x7fdbe015d330 2023-10-14T18:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.715+0000 7fdbd6ffd640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 0x7fdbc406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.715+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fdbd0088dc0 con 0x7fdbe015d330 2023-10-14T18:20:17.731 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.716+0000 7fdbe62ab640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 0x7fdbc406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:17.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.716+0000 7fdbe62ab640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 0x7fdbc406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc000f60 tx=0x7fdbdc009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:17.732 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.720+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdbd0053cb0 con 0x7fdbe015d330 2023-10-14T18:20:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.914+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fdbe01610b0 con 0x7fdbc406af90 2023-10-14T18:20:17.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.914+0000 7fdbd6ffd640 1 -- 172.21.15.29:0/4167704595 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdbdc0050d0 con 0x7fdbc406af90 2023-10-14T18:20:17.917 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 2 to deep-scrub 2023-10-14T18:20:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 msgr2=0x7fdbc406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 0x7fdbc406d450 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fdbdc000f60 tx=0x7fdbdc009040 comp rx=0 tx=0).stop 2023-10-14T18:20:17.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 msgr2=0x7fdbe00ae9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 secure :-1 s=READY pgs=3299 cs=0 l=1 rev1=1 crypto rx=0x7fdbd000c400 tx=0x7fdbd000c8c0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 shutdown_connections 2023-10-14T18:20:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdbc406af90 0x7fdbc406d450 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdbe01618b0 0x7fdbe015b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdbe015d330 0x7fdbe00ae9c0 unknown :-1 s=CLOSED pgs=3299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 --2- 172.21.15.29:0/4167704595 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdbe00661c0 0x7fdbe00b2d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 >> 172.21.15.29:0/4167704595 conn(0x7fdbe00d0ab0 msgr2=0x7fdbe015bf30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 shutdown_connections 2023-10-14T18:20:17.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:17.917+0000 7fdbe850f640 1 -- 172.21.15.29:0/4167704595 wait complete. 2023-10-14T18:20:17.922 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:17.928 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:17.937 INFO:tasks.ceph:Scrubbing osd.3 2023-10-14T18:20:17.938 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.189+0000 7ff67c773640 1 Processor -- start 2023-10-14T18:20:18.191 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.189+0000 7ff67c773640 1 -- start start 2023-10-14T18:20:18.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67c773640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 0x7ff6740b03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67c773640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 0x7ff6740c93c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67c773640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff674159740 con 0x7ff67406afc0 2023-10-14T18:20:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67c773640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6740c9900 con 0x7ff6740c8f30 2023-10-14T18:20:18.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67c773640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff67406c6c0 con 0x7ff6741592e0 2023-10-14T18:20:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67ad10640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 0x7ff6740c93c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.195 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff679d0e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 0x7ff6740b03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.191+0000 7ff67a50f640 1 --1- >> v1:172.21.15.186:6789/0 conn(0x7ff67406afc0 0x7ff67406bea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45884/0 (socket says 172.21.15.29:45884) 2023-10-14T18:20:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff67a50f640 1 -- 172.21.15.29:0/3351109578 learned_addr learned my addr 172.21.15.29:0/3351109578 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff679d0e640 1 -- 172.21.15.29:0/3351109578 >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 msgr2=0x7ff6740c93c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:18.196 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff679d0e640 1 --2- 172.21.15.29:0/3351109578 >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 0x7ff6740c93c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff679d0e640 1 -- 172.21.15.29:0/3351109578 >> v1:172.21.15.186:6789/0 conn(0x7ff67406afc0 legacy=0x7ff67406bea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff679d0e640 1 -- 172.21.15.29:0/3351109578 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6740cc860 con 0x7ff6741592e0 2023-10-14T18:20:18.197 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.192+0000 7ff67ad10640 1 --2- 172.21.15.29:0/3351109578 >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 0x7ff6740c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-10-14T18:20:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.194+0000 7ff679d0e640 1 --2- 172.21.15.29:0/3351109578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 0x7ff6740b03d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097a0 tx=0x7ff66402b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ea09c880a87178d8 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.194+0000 7ff67950d640 1 -- 172.21.15.29:0/3351109578 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66402d040 con 0x7ff6741592e0 2023-10-14T18:20:18.198 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.194+0000 7ff67950d640 1 -- 172.21.15.29:0/3351109578 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff664031810 con 0x7ff6741592e0 2023-10-14T18:20:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.194+0000 7ff67950d640 1 -- 172.21.15.29:0/3351109578 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff664031ba0 con 0x7ff6741592e0 2023-10-14T18:20:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 -- 172.21.15.29:0/3351109578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 msgr2=0x7ff6740b03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 --2- 172.21.15.29:0/3351109578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 0x7ff6740b03d0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff6640097a0 tx=0x7ff66402b8b0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.199 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 -- 172.21.15.29:0/3351109578 shutdown_connections 2023-10-14T18:20:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 --2- 172.21.15.29:0/3351109578 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6741592e0 0x7ff6740b03d0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 --2- 172.21.15.29:0/3351109578 >> v2:172.21.15.186:3300/0 conn(0x7ff6740c8f30 0x7ff6740c93c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 -- 172.21.15.29:0/3351109578 >> 172.21.15.29:0/3351109578 conn(0x7ff6740cb360 msgr2=0x7ff6740cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 -- 172.21.15.29:0/3351109578 shutdown_connections 2023-10-14T18:20:18.200 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.195+0000 7ff67c773640 1 -- 172.21.15.29:0/3351109578 wait complete. 2023-10-14T18:20:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.197+0000 7ff67c773640 1 Processor -- start 2023-10-14T18:20:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.198+0000 7ff67c773640 1 -- start start 2023-10-14T18:20:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff67406afc0 0x7ff674154cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6740c8f30 0x7ff674153f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 0x7ff67415b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674156180 con 0x7ff6741592e0 2023-10-14T18:20:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674155570 con 0x7ff67406afc0 2023-10-14T18:20:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67c773640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff674155850 con 0x7ff6740c8f30 2023-10-14T18:20:18.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67ad10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 0x7ff67415b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67ad10640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 0x7ff67415b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43630/0 (socket says 172.21.15.29:43630) 2023-10-14T18:20:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.199+0000 7ff67ad10640 1 -- 172.21.15.29:0/2423941827 learned_addr learned my addr 172.21.15.29:0/2423941827 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:18.205 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67a50f640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff67406afc0 0x7ff674154cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff679d0e640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6740c8f30 0x7ff674153f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 -- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6740c8f30 msgr2=0x7ff674153f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7ff6740c8f30 0x7ff674153f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 -- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff67406afc0 msgr2=0x7ff674154cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7ff67406afc0 0x7ff674154cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff66c009a70 con 0x7ff6741592e0 2023-10-14T18:20:18.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.200+0000 7ff67ad10640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 0x7ff67415b8e0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00efc0 tx=0x7ff66c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.201+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c009d70 con 0x7ff6741592e0 2023-10-14T18:20:18.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.201+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff66c00f040 con 0x7ff6741592e0 2023-10-14T18:20:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.201+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66c003990 con 0x7ff6741592e0 2023-10-14T18:20:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.201+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff664009400 con 0x7ff6741592e0 2023-10-14T18:20:18.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.201+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff674155e10 con 0x7ff6741592e0 2023-10-14T18:20:18.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.202+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff6740cce20 con 0x7ff6741592e0 2023-10-14T18:20:18.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.204+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7ff66c0144a0 con 0x7ff6741592e0 2023-10-14T18:20:18.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff66affd640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ff65806af90 0x7ff65806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7ff66c08a4a0 con 0x7ff6741592e0 2023-10-14T18:20:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff67a50f640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ff65806af90 0x7ff65806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff66affd640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7ff658071590 0x7ff6580739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff658074080 con 0x7ff658071590 2023-10-14T18:20:18.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff66c08cb60 con 0x7ff6741592e0 2023-10-14T18:20:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.205+0000 7ff679d0e640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7ff658071590 0x7ff6580739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.206+0000 7ff67a50f640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ff65806af90 0x7ff65806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff670005880 tx=0x7ff670009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.206+0000 7ff679d0e640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7ff658071590 0x7ff6580739b0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.212 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.207+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== osd.3 v2:172.21.15.29:6811/149520408 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7ff658074080 con 0x7ff658071590 2023-10-14T18:20:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.221+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 --> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7ff6740ccb40 con 0x7ff658071590 2023-10-14T18:20:18.223 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.221+0000 7ff66affd640 1 -- 172.21.15.29:0/2423941827 <== osd.3 v2:172.21.15.29:6811/149520408 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7ff664031a80 con 0x7ff658071590 2023-10-14T18:20:18.224 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:18.224 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:18.224 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7ff658071590 msgr2=0x7ff6580739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.225 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6811/149520408,v1:172.21.15.29:6813/149520408] conn(0x7ff658071590 0x7ff6580739b0 crc :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ff65806af90 msgr2=0x7ff65806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7ff65806af90 0x7ff65806d450 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff670005880 tx=0x7ff670009040 comp rx=0 tx=0).stop 2023-10-14T18:20:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 msgr2=0x7ff67415b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 --2- 172.21.15.29:0/2423941827 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7ff6741592e0 0x7ff67415b8e0 secure :-1 s=READY pgs=3300 cs=0 l=1 rev1=1 crypto rx=0x7ff66c00efc0 tx=0x7ff66c009f90 comp rx=0 tx=0).stop 2023-10-14T18:20:18.226 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67ad10640 1 -- 172.21.15.29:0/2423941827 reap_dead start 2023-10-14T18:20:18.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 shutdown_connections 2023-10-14T18:20:18.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 >> 172.21.15.29:0/2423941827 conn(0x7ff6740cb360 msgr2=0x7ff6740afe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 shutdown_connections 2023-10-14T18:20:18.227 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.222+0000 7ff67c773640 1 -- 172.21.15.29:0/2423941827 wait complete. 2023-10-14T18:20:18.241 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-10-14T18:20:18.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.455+0000 7f6cff2af640 1 Processor -- start 2023-10-14T18:20:18.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.456+0000 7f6cff2af640 1 -- start start 2023-10-14T18:20:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.457+0000 7f6cff2af640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f6cf806c850 0x7f6cf80ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.457+0000 7f6cff2af640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf8070360 con 0x7f6cf8159e90 2023-10-14T18:20:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.457+0000 7f6cff2af640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cf80cd1f0 con 0x7f6cf8159ab0 2023-10-14T18:20:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.457+0000 7f6cff2af640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cf815d890 con 0x7f6cf806c850 2023-10-14T18:20:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6cfc84a640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f6cf8159e90 0x7f6cf806c140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55672/0 (socket says 172.21.15.29:55672) 2023-10-14T18:20:18.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6cfc84a640 1 -- 172.21.15.29:0/3908875358 learned_addr learned my addr 172.21.15.29:0/3908875358 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552186002 0 0) 0x7f6cf8070360 con 0x7f6cf8159e90 2023-10-14T18:20:18.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6cfd84c640 1 --2- 172.21.15.29:0/3908875358 >> v2:172.21.15.186:3300/0 conn(0x7f6cf806c850 0x7f6cf80ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ce4005730 con 0x7f6cf8159e90 2023-10-14T18:20:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2138075698 0 0) 0x7f6ce4005730 con 0x7f6cf8159e90 2023-10-14T18:20:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 >> v2:172.21.15.186:3300/0 conn(0x7f6cf806c850 msgr2=0x7f6cf80ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:18.463 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 --2- 172.21.15.29:0/3908875358 >> v2:172.21.15.186:3300/0 conn(0x7f6cf806c850 0x7f6cf80ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.458+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 >> v1:172.21.15.186:6789/0 conn(0x7f6cf8159ab0 legacy=0x7f6cf806fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf80cd4d0 con 0x7f6cf8159e90 2023-10-14T18:20:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f6ce8001690 con 0x7f6cf8159e90 2023-10-14T18:20:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 >> v1:172.21.15.29:6789/0 conn(0x7f6cf8159e90 legacy=0x7f6cf806c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.464 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce4005bd0 0x7f6ce4005ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 0x7f6ce4006e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce4007590 0x7f6ce400db90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf815d890 con 0x7f6ce40069d0 2023-10-14T18:20:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce400e470 con 0x7f6ce4007590 2023-10-14T18:20:18.465 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce400e630 con 0x7f6ce4005bd0 2023-10-14T18:20:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfd84c640 1 --2- 172.21.15.29:0/3908875358 >> v2:172.21.15.186:3300/0 conn(0x7f6cf806c850 0x7f6cf80ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:20:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 0x7f6ce4006e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfd04b640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce4005bd0 0x7f6ce4005ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfd84c640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce4007590 0x7f6ce400db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.466 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfc84a640 1 -- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce4005bd0 msgr2=0x7f6ce4005ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce4005bd0 0x7f6ce4005ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.459+0000 7f6cfc84a640 1 -- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce4007590 msgr2=0x7f6ce400db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.460+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6ce4007590 0x7f6ce400db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.460+0000 7f6cfc84a640 1 -- 172.21.15.29:0/3908875358 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce4005730 con 0x7f6ce40069d0 2023-10-14T18:20:18.467 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.460+0000 7f6cfd84c640 1 -- 172.21.15.29:0/3908875358 reap_dead start 2023-10-14T18:20:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.461+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 0x7f6ce4006e50 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800c160 tx=0x7f6ce4007a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26fe2fcb57ef5cf9 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.470 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.461+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce8005ad0 con 0x7f6ce40069d0 2023-10-14T18:20:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.461+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8005c30 con 0x7f6ce40069d0 2023-10-14T18:20:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.461+0000 7f6ceffff640 1 -- 172.21.15.29:0/3908875358 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce802c7c0 con 0x7f6ce40069d0 2023-10-14T18:20:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 -- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 msgr2=0x7f6ce4006e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 0x7f6ce4006e50 secure :-1 s=READY pgs=3302 cs=0 l=1 rev1=1 crypto rx=0x7f6ce800c160 tx=0x7f6ce4007a00 comp rx=0 tx=0).stop 2023-10-14T18:20:18.471 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 -- 172.21.15.29:0/3908875358 shutdown_connections 2023-10-14T18:20:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 --2- 172.21.15.29:0/3908875358 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6ce40069d0 0x7f6ce4006e50 unknown :-1 s=CLOSED pgs=3302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 -- 172.21.15.29:0/3908875358 >> 172.21.15.29:0/3908875358 conn(0x7f6cf80ce9b0 msgr2=0x7f6cf80cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 -- 172.21.15.29:0/3908875358 shutdown_connections 2023-10-14T18:20:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.462+0000 7f6cff2af640 1 -- 172.21.15.29:0/3908875358 wait complete. 2023-10-14T18:20:18.472 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.464+0000 7f6cff2af640 1 Processor -- start 2023-10-14T18:20:18.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.464+0000 7f6cff2af640 1 -- start start 2023-10-14T18:20:18.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce40069d0 0x7f6cf815ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf80b2c90 0x7f6cf80b3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.473 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf806a760 con 0x7f6cf8069da0 2023-10-14T18:20:18.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf815a970 con 0x7f6cf80b2c90 2023-10-14T18:20:18.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.465+0000 7f6cff2af640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cf815ac50 con 0x7f6ce40069d0 2023-10-14T18:20:18.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43636/0 (socket says 172.21.15.29:43636) 2023-10-14T18:20:18.474 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 -- 172.21.15.29:0/1886898097 learned_addr learned my addr 172.21.15.29:0/1886898097 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfd04b640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce40069d0 0x7f6cf815ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfd84c640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf80b2c90 0x7f6cf80b3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 -- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce40069d0 msgr2=0x7f6cf815ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce40069d0 0x7f6cf815ba10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.475 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 -- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf80b2c90 msgr2=0x7f6cf80b3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf80b2c90 0x7f6cf80b3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 -- 172.21.15.29:0/1886898097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cf815af30 con 0x7f6cf8069da0 2023-10-14T18:20:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.466+0000 7f6cfc84a640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7f6ce802c360 tx=0x7f6ce8001950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.467+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce803c070 con 0x7f6cf8069da0 2023-10-14T18:20:18.476 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.467+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce8000ed0 con 0x7f6cf8069da0 2023-10-14T18:20:18.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.467+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ce802cc10 con 0x7f6cf8069da0 2023-10-14T18:20:18.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.467+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cf815b210 con 0x7f6cf8069da0 2023-10-14T18:20:18.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.467+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cf815a5d0 con 0x7f6cf8069da0 2023-10-14T18:20:18.477 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.468+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cf804e160 con 0x7f6cf8069da0 2023-10-14T18:20:18.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.469+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f6ce8032070 con 0x7f6cf8069da0 2023-10-14T18:20:18.478 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.470+0000 7f6cedffb640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 0x7f6cdc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.470+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f6ce80aa290 con 0x7f6cf8069da0 2023-10-14T18:20:18.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.470+0000 7f6cfd04b640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 0x7f6cdc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.479 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.471+0000 7f6cfd04b640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 0x7f6cdc06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007fd0 tx=0x7f6cf4009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.480 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.478+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f6ce80750e0 con 0x7f6cf8069da0 2023-10-14T18:20:18.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.692+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f6cf815adc0 con 0x7f6cdc06af90 2023-10-14T18:20:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.693+0000 7f6cedffb640 1 -- 172.21.15.29:0/1886898097 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6cf815adc0 con 0x7f6cdc06af90 2023-10-14T18:20:18.695 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 3 to deep-scrub 2023-10-14T18:20:18.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.695+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 msgr2=0x7f6cdc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.695+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 0x7f6cdc06d450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4007fd0 tx=0x7f6cf4009040 comp rx=0 tx=0).stop 2023-10-14T18:20:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.695+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 msgr2=0x7f6cf806a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.695+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 secure :-1 s=READY pgs=3303 cs=0 l=1 rev1=1 crypto rx=0x7f6ce802c360 tx=0x7f6ce8001950 comp rx=0 tx=0).stop 2023-10-14T18:20:18.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 shutdown_connections 2023-10-14T18:20:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f6cf80b2c90 0x7f6cf80b3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f6cf8069da0 0x7f6cf806a220 unknown :-1 s=CLOSED pgs=3303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f6cdc06af90 0x7f6cdc06d450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 --2- 172.21.15.29:0/1886898097 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f6ce40069d0 0x7f6cf815ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 >> 172.21.15.29:0/1886898097 conn(0x7f6cf80ce9b0 msgr2=0x7f6cf80ac890 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 shutdown_connections 2023-10-14T18:20:18.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.696+0000 7f6cff2af640 1 -- 172.21.15.29:0/1886898097 wait complete. 2023-10-14T18:20:18.715 INFO:tasks.ceph:Scrubbing osd.4 2023-10-14T18:20:18.715 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.933+0000 7f8ed570f640 1 Processor -- start 2023-10-14T18:20:18.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.933+0000 7f8ed570f640 1 -- start start 2023-10-14T18:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ed570f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 0x7f8ed00ccb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ed570f640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed00cb890 con 0x7f8ed00cda30 2023-10-14T18:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ed570f640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ed00cbac0 con 0x7f8ed015af90 2023-10-14T18:20:18.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ed570f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00cef80 con 0x7f8ed00cfae0 2023-10-14T18:20:18.938 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ece7fc640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f8ed00cda30 0x7f8ed00cf3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55684/0 (socket says 172.21.15.29:55684) 2023-10-14T18:20:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.935+0000 7f8ece7fc640 1 -- 172.21.15.29:0/991578895 learned_addr learned my addr 172.21.15.29:0/991578895 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/991578895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 0x7f8ed00ccb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 <== mon.1 v1:172.21.15.186:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1426672634 0 0) 0x7f8ed00cbac0 con 0x7f8ed015af90 2023-10-14T18:20:18.939 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 --> v1:172.21.15.186:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ebc003520 con 0x7f8ed015af90 2023-10-14T18:20:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1618066030 0 0) 0x7f8ed00cb890 con 0x7f8ed00cda30 2023-10-14T18:20:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ed00cbac0 con 0x7f8ed00cda30 2023-10-14T18:20:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/991578895 >> v1:172.21.15.186:6789/0 conn(0x7f8ed015af90 legacy=0x7f8ed00cd320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/991578895 >> v1:172.21.15.29:6789/0 conn(0x7f8ed00cda30 legacy=0x7f8ed00cf3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.936+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/991578895 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed00cbda0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.937+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/991578895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 0x7f8ed00ccb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80097d0 tx=0x7f8ec802b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85dd3793e564dffd server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 <== mon.2 v2:172.21.15.186:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8030800 con 0x7f8ed00cfae0 2023-10-14T18:20:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 <== mon.2 v2:172.21.15.186:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec8030960 con 0x7f8ed00cfae0 2023-10-14T18:20:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ecdffb640 1 -- 172.21.15.29:0/991578895 <== mon.2 v2:172.21.15.186:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8030cd0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ed570f640 1 -- 172.21.15.29:0/991578895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 msgr2=0x7f8ed00ccb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ed570f640 1 --2- 172.21.15.29:0/991578895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 0x7f8ed00ccb60 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8ec80097d0 tx=0x7f8ec802b940 comp rx=0 tx=0).stop 2023-10-14T18:20:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ed570f640 1 -- 172.21.15.29:0/991578895 shutdown_connections 2023-10-14T18:20:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ed570f640 1 --2- 172.21.15.29:0/991578895 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cfae0 0x7f8ed00ccb60 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.938+0000 7f8ed570f640 1 -- 172.21.15.29:0/991578895 >> 172.21.15.29:0/991578895 conn(0x7f8ed0099e30 msgr2=0x7f8ed00d0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.939+0000 7f8ed570f640 1 -- 172.21.15.29:0/991578895 shutdown_connections 2023-10-14T18:20:18.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.939+0000 7f8ed570f640 1 -- 172.21.15.29:0/991578895 wait complete. 2023-10-14T18:20:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.940+0000 7f8ed570f640 1 Processor -- start 2023-10-14T18:20:18.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.940+0000 7f8ed570f640 1 -- start start 2023-10-14T18:20:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cda30 0x7f8ed00c7290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 0x7f8ed009a7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ed015af90 0x7f8ed01570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00c7970 con 0x7f8ed00cfae0 2023-10-14T18:20:18.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00c9640 con 0x7f8ed015af90 2023-10-14T18:20:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.941+0000 7f8ed570f640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00c8d10 con 0x7f8ed00cda30 2023-10-14T18:20:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.942+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 0x7f8ed009a7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.942+0000 7f8ece7fc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 0x7f8ed009a7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43640/0 (socket says 172.21.15.29:43640) 2023-10-14T18:20:18.948 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.942+0000 7f8ece7fc640 1 -- 172.21.15.29:0/271896079 learned_addr learned my addr 172.21.15.29:0/271896079 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.942+0000 7f8eceffd640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cda30 0x7f8ed00c7290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.942+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ed015af90 0x7f8ed01570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 -- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cda30 msgr2=0x7f8ed00c7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f8ed00cda30 0x7f8ed00c7290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.949 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 -- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ed015af90 msgr2=0x7f8ed01570c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f8ed015af90 0x7f8ed01570c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec8009400 con 0x7f8ed00cfae0 2023-10-14T18:20:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ece7fc640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 0x7f8ed009a7a0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c3c0 tx=0x7f8eb800c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.950 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb8005ce0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8eb8005e40 con 0x7f8ed00cfae0 2023-10-14T18:20:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8eb80072d0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.943+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ed00c8ff0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.944+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed00c52d0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.951 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.944+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8ed00cc360 con 0x7f8ed00cfae0 2023-10-14T18:20:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.946+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f8eb8007430 con 0x7f8ed00cfae0 2023-10-14T18:20:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ec77fe640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8eb006af90 0x7f8eb006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f8eb8088d00 con 0x7f8ed00cfae0 2023-10-14T18:20:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8eceffd640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8eb006af90 0x7f8eb006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.953 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ec77fe640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f8eb0071590 0x7f8eb00739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8eb0074080 con 0x7f8eb0071590 2023-10-14T18:20:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8eb808b2e0 con 0x7f8ed00cfae0 2023-10-14T18:20:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.947+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f8eb0071590 0x7f8eb00739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:18.954 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.948+0000 7f8eceffd640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8eb006af90 0x7f8eb006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0001560 tx=0x7f8ec0009040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.948+0000 7f8ecf7fe640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f8eb0071590 0x7f8eb00739b0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:18.955 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.950+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== osd.4 v2:172.21.15.186:6800/189625010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f8eb0074080 con 0x7f8eb0071590 2023-10-14T18:20:18.965 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.964+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 --> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f8ed00cc080 con 0x7f8eb0071590 2023-10-14T18:20:18.966 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ec77fe640 1 -- 172.21.15.29:0/271896079 <== osd.4 v2:172.21.15.186:6800/189625010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f8ec8040040 con 0x7f8eb0071590 2023-10-14T18:20:18.967 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:18.967 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:18.967 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:18.968 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f8eb0071590 msgr2=0x7f8eb00739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.186:6800/189625010,v1:172.21.15.186:6801/189625010] conn(0x7f8eb0071590 0x7f8eb00739b0 crc :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8eb006af90 msgr2=0x7f8eb006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f8eb006af90 0x7f8eb006d450 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f8ec0001560 tx=0x7f8ec0009040 comp rx=0 tx=0).stop 2023-10-14T18:20:18.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 msgr2=0x7f8ed009a7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:18.969 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ed570f640 1 --2- 172.21.15.29:0/271896079 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f8ed00cfae0 0x7f8ed009a7a0 secure :-1 s=READY pgs=3305 cs=0 l=1 rev1=1 crypto rx=0x7f8eb800c3c0 tx=0x7f8eb800c7c0 comp rx=0 tx=0).stop 2023-10-14T18:20:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.965+0000 7f8ecf7fe640 1 -- 172.21.15.29:0/271896079 reap_dead start 2023-10-14T18:20:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.966+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 shutdown_connections 2023-10-14T18:20:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.966+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 >> 172.21.15.29:0/271896079 conn(0x7f8ed0099e30 msgr2=0x7f8ed00b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.966+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 shutdown_connections 2023-10-14T18:20:18.970 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:18.966+0000 7f8ed570f640 1 -- 172.21.15.29:0/271896079 wait complete. 2023-10-14T18:20:18.984 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-10-14T18:20:19.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.199+0000 7f5bd80c9640 1 Processor -- start 2023-10-14T18:20:19.201 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.199+0000 7f5bd80c9640 1 -- start start 2023-10-14T18:20:19.202 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd80c9640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f5bd015aee0 0x7f5bd00aeef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd80c9640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd006c140 con 0x7f5bd015e9b0 2023-10-14T18:20:19.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd80c9640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd015b2c0 con 0x7f5bd00cb810 2023-10-14T18:20:19.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd80c9640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd006b270 con 0x7f5bd015aee0 2023-10-14T18:20:19.203 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd6666640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f5bd015e9b0 0x7f5bd015ee20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55700/0 (socket says 172.21.15.29:55700) 2023-10-14T18:20:19.204 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd6666640 1 -- 172.21.15.29:0/998674638 learned_addr learned my addr 172.21.15.29:0/998674638 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.201+0000 7f5bd5e65640 1 --2- 172.21.15.29:0/998674638 >> v2:172.21.15.186:3300/0 conn(0x7f5bd015aee0 0x7f5bd00aeef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1280075886 0 0) 0x7f5bd006c140 con 0x7f5bd015e9b0 2023-10-14T18:20:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5bbc003520 con 0x7f5bd015e9b0 2023-10-14T18:20:19.206 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389058561 0 0) 0x7f5bbc003520 con 0x7f5bd015e9b0 2023-10-14T18:20:19.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 >> v2:172.21.15.186:3300/0 conn(0x7f5bd015aee0 msgr2=0x7f5bd00aeef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 --2- 172.21.15.29:0/998674638 >> v2:172.21.15.186:3300/0 conn(0x7f5bd015aee0 0x7f5bd00aeef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 >> v1:172.21.15.186:6789/0 conn(0x7f5bd00cb810 legacy=0x7f5bd00cbc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.207 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd006daf0 con 0x7f5bd015e9b0 2023-10-14T18:20:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f5bc80019f0 con 0x7f5bd015e9b0 2023-10-14T18:20:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 >> v1:172.21.15.29:6789/0 conn(0x7f5bd015e9b0 legacy=0x7f5bd015ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc003970 0x7f5bbc003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bbc0047c0 0x7f5bbc004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.208 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 0x7f5bbc00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd006b270 con 0x7f5bbc005180 2023-10-14T18:20:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc00c120 con 0x7f5bbc003970 2023-10-14T18:20:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.202+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bbc00c2e0 con 0x7f5bbc0047c0 2023-10-14T18:20:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 0x7f5bbc00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.209 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd5e65640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc003970 0x7f5bbc003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd5664640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bbc0047c0 0x7f5bbc004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 -- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bbc0047c0 msgr2=0x7f5bbc004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bbc0047c0 0x7f5bbc004c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 -- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc003970 msgr2=0x7f5bbc003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.210 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc003970 0x7f5bbc003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.203+0000 7f5bd6666640 1 -- 172.21.15.29:0/998674638 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bbc003520 con 0x7f5bbc005180 2023-10-14T18:20:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.204+0000 7f5bd6666640 1 -- 172.21.15.29:0/998674638 reap_dead start 2023-10-14T18:20:19.211 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.204+0000 7f5bd6666640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 0x7f5bbc00b980 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800a830 tx=0x7f5bbc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc3a946a22e4d8ad server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.213 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80072d0 con 0x7f5bbc005180 2023-10-14T18:20:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc802f070 con 0x7f5bbc005180 2023-10-14T18:20:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd4e63640 1 -- 172.21.15.29:0/998674638 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc80076b0 con 0x7f5bbc005180 2023-10-14T18:20:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 -- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 msgr2=0x7f5bbc00b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.214 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 0x7f5bbc00b980 secure :-1 s=READY pgs=3307 cs=0 l=1 rev1=1 crypto rx=0x7f5bc800a830 tx=0x7f5bbc0057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 -- 172.21.15.29:0/998674638 shutdown_connections 2023-10-14T18:20:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/998674638 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bbc005180 0x7f5bbc00b980 unknown :-1 s=CLOSED pgs=3307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 -- 172.21.15.29:0/998674638 >> 172.21.15.29:0/998674638 conn(0x7f5bd00cfac0 msgr2=0x7f5bd00cfed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.205+0000 7f5bd80c9640 1 -- 172.21.15.29:0/998674638 shutdown_connections 2023-10-14T18:20:19.215 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.206+0000 7f5bd80c9640 1 -- 172.21.15.29:0/998674638 wait complete. 2023-10-14T18:20:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.207+0000 7f5bd80c9640 1 Processor -- start 2023-10-14T18:20:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.207+0000 7f5bd80c9640 1 -- start start 2023-10-14T18:20:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc005180 0x7f5bd0159860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.216 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd00a8510 0x7f5bd00a8990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd015dcd0 con 0x7f5bd015d310 2023-10-14T18:20:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd0159da0 con 0x7f5bbc005180 2023-10-14T18:20:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.208+0000 7f5bd80c9640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bd00aa370 con 0x7f5bd00a8510 2023-10-14T18:20:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.217 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43654/0 (socket says 172.21.15.29:43654) 2023-10-14T18:20:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 -- 172.21.15.29:0/2223770139 learned_addr learned my addr 172.21.15.29:0/2223770139 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 -- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd00a8510 msgr2=0x7f5bd00a8990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd00a8510 0x7f5bd00a8990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 -- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc005180 msgr2=0x7f5bd0159860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.218 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc005180 0x7f5bd0159860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 -- 172.21.15.29:0/2223770139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd00a92d0 con 0x7f5bd015d310 2023-10-14T18:20:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bd5664640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000d570 tx=0x7f5bc000da30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.209+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0003610 con 0x7f5bd015d310 2023-10-14T18:20:19.219 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.210+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc0003770 con 0x7f5bd015d310 2023-10-14T18:20:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.210+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5bc0003b20 con 0x7f5bd015d310 2023-10-14T18:20:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.210+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd00a95b0 con 0x7f5bd015d310 2023-10-14T18:20:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.210+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd00a9b70 con 0x7f5bd015d310 2023-10-14T18:20:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.211+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd004e160 con 0x7f5bd015d310 2023-10-14T18:20:19.220 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.212+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f5bc0003c80 con 0x7f5bd015d310 2023-10-14T18:20:19.221 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.213+0000 7f5bc67fc640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 0x7f5bb406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.213+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f5bc0089810 con 0x7f5bd015d310 2023-10-14T18:20:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.213+0000 7f5bd5e65640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 0x7f5bb406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.214+0000 7f5bd5e65640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 0x7f5bb406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5bd015df20 tx=0x7f5bcc009320 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.222 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.218+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f5bc0054740 con 0x7f5bd015d310 2023-10-14T18:20:19.420 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.418+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f5bd006b510 con 0x7f5bb406af90 2023-10-14T18:20:19.421 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.419+0000 7f5bc67fc640 1 -- 172.21.15.29:0/2223770139 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5bcc004020 con 0x7f5bb406af90 2023-10-14T18:20:19.421 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 4 to deep-scrub 2023-10-14T18:20:19.424 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 msgr2=0x7f5bb406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 0x7f5bb406d450 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5bd015df20 tx=0x7f5bcc009320 comp rx=0 tx=0).stop 2023-10-14T18:20:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 msgr2=0x7f5bd015d790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 secure :-1 s=READY pgs=3308 cs=0 l=1 rev1=1 crypto rx=0x7f5bc000d570 tx=0x7f5bc000da30 comp rx=0 tx=0).stop 2023-10-14T18:20:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 shutdown_connections 2023-10-14T18:20:19.425 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f5bb406af90 0x7f5bb406d450 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f5bd00a8510 0x7f5bd00a8990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f5bd015d310 0x7f5bd015d790 unknown :-1 s=CLOSED pgs=3308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 --2- 172.21.15.29:0/2223770139 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f5bbc005180 0x7f5bd0159860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 >> 172.21.15.29:0/2223770139 conn(0x7f5bd00cfac0 msgr2=0x7f5bd00ca640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:19.426 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 shutdown_connections 2023-10-14T18:20:19.427 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.422+0000 7f5bd80c9640 1 -- 172.21.15.29:0/2223770139 wait complete. 2023-10-14T18:20:19.442 INFO:tasks.ceph:Scrubbing osd.5 2023-10-14T18:20:19.442 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:19.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.667+0000 7f1efefac640 1 Processor -- start 2023-10-14T18:20:19.669 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.667+0000 7f1efefac640 1 -- start start 2023-10-14T18:20:19.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efefac640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ef806c040 0x7f1ef806c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.672 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efefac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef815e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efefac640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ef806c960 con 0x7f1ef815ebc0 2023-10-14T18:20:19.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efefac640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ef815b9d0 con 0x7f1ef806c040 2023-10-14T18:20:19.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efefac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef815bcb0 con 0x7f1ef80ad970 2023-10-14T18:20:19.673 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efdfaa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ef806c040 0x7f1ef806c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efdfaa640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f1ef806c040 0x7f1ef806c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43660/0 (socket says 172.21.15.29:43660) 2023-10-14T18:20:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efe7ab640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f1ef815ebc0 0x7f1ef80cca70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55706/0 (socket says 172.21.15.29:55706) 2023-10-14T18:20:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.669+0000 7f1efdfaa640 1 -- 172.21.15.29:0/1547897319 learned_addr learned my addr 172.21.15.29:0/1547897319 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:19.674 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.670+0000 7f1efd7a9640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef815e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.670+0000 7f1efdfaa640 1 -- 172.21.15.29:0/1547897319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 msgr2=0x7f1ef815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.675 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.670+0000 7f1efdfaa640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef815e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.670+0000 7f1efdfaa640 1 -- 172.21.15.29:0/1547897319 >> v1:172.21.15.29:6789/0 conn(0x7f1ef815ebc0 legacy=0x7f1ef80cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.676 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.670+0000 7f1efdfaa640 1 -- 172.21.15.29:0/1547897319 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef815bf90 con 0x7f1ef806c040 2023-10-14T18:20:19.680 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.671+0000 7f1efdfaa640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef806c420 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009e80 tx=0x7f1ef402bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18a242f02728c5f1 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.671+0000 7f1efcfa8640 1 -- 172.21.15.29:0/1547897319 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef402cae0 con 0x7f1ef806c040 2023-10-14T18:20:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.671+0000 7f1efcfa8640 1 -- 172.21.15.29:0/1547897319 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef402cc40 con 0x7f1ef806c040 2023-10-14T18:20:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.671+0000 7f1efcfa8640 1 -- 172.21.15.29:0/1547897319 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef40343f0 con 0x7f1ef806c040 2023-10-14T18:20:19.681 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 -- 172.21.15.29:0/1547897319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 msgr2=0x7f1ef806c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef806c420 secure :-1 s=READY pgs=3310 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4009e80 tx=0x7f1ef402bf40 comp rx=0 tx=0).stop 2023-10-14T18:20:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 -- 172.21.15.29:0/1547897319 shutdown_connections 2023-10-14T18:20:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef815e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.682 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 --2- 172.21.15.29:0/1547897319 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef806c420 unknown :-1 s=CLOSED pgs=3310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 -- 172.21.15.29:0/1547897319 >> 172.21.15.29:0/1547897319 conn(0x7f1ef80ce6a0 msgr2=0x7f1ef80ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 -- 172.21.15.29:0/1547897319 shutdown_connections 2023-10-14T18:20:19.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.672+0000 7f1efefac640 1 -- 172.21.15.29:0/1547897319 wait complete. 2023-10-14T18:20:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.674+0000 7f1efefac640 1 Processor -- start 2023-10-14T18:20:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.674+0000 7f1efefac640 1 -- start start 2023-10-14T18:20:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef8156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ef815ebc0 0x7f1ef80acdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c7fb0 con 0x7f1ef806c040 2023-10-14T18:20:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80af7b0 con 0x7f1ef815ebc0 2023-10-14T18:20:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efefac640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ef80c73a0 con 0x7f1ef80ad970 2023-10-14T18:20:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efdfaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef8156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efdfaa640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef8156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43668/0 (socket says 172.21.15.29:43668) 2023-10-14T18:20:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efdfaa640 1 -- 172.21.15.29:0/292411225 learned_addr learned my addr 172.21.15.29:0/292411225 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efe7ab640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ef815ebc0 0x7f1ef80acdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efd7a9640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.675+0000 7f1efdfaa640 1 -- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 msgr2=0x7f1ef80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efdfaa640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef80ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efdfaa640 1 -- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ef815ebc0 msgr2=0x7f1ef80acdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efdfaa640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ef815ebc0 0x7f1ef80acdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efdfaa640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ef4009b30 con 0x7f1ef806c040 2023-10-14T18:20:19.687 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efe7ab640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1ef815ebc0 0x7f1ef80acdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efd7a9640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f1ef80ad970 0x7f1ef80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efdfaa640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef8156ce0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4000f80 tx=0x7f1ef4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef4009510 con 0x7f1ef806c040 2023-10-14T18:20:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ef403e450 con 0x7f1ef806c040 2023-10-14T18:20:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ef403e800 con 0x7f1ef806c040 2023-10-14T18:20:19.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ef80c7680 con 0x7f1ef806c040 2023-10-14T18:20:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.676+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ef80c7c40 con 0x7f1ef806c040 2023-10-14T18:20:19.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.677+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ef80cc550 con 0x7f1ef806c040 2023-10-14T18:20:19.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.678+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1ef403e960 con 0x7f1ef806c040 2023-10-14T18:20:19.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.679+0000 7f1eee7fc640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f1edc06af90 0x7f1edc06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.679+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f1ef40ac690 con 0x7f1ef806c040 2023-10-14T18:20:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1efd7a9640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f1edc06af90 0x7f1edc06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1eee7fc640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f1edc071590 0x7f1edc0739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1edc074080 con 0x7f1edc071590 2023-10-14T18:20:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ef40aec70 con 0x7f1ef806c040 2023-10-14T18:20:19.695 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1efe7ab640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f1edc071590 0x7f1edc0739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.680+0000 7f1efd7a9640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f1edc06af90 0x7f1edc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8006fd0 tx=0x7f1ee8008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.696 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.681+0000 7f1efe7ab640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f1edc071590 0x7f1edc0739b0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.683+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== osd.5 v2:172.21.15.186:6816/1666789598 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f1edc074080 con 0x7f1edc071590 2023-10-14T18:20:19.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.697+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 --> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1ef80cc270 con 0x7f1edc071590 2023-10-14T18:20:19.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.697+0000 7f1eee7fc640 1 -- 172.21.15.29:0/292411225 <== osd.5 v2:172.21.15.186:6816/1666789598 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1ef00017a0 con 0x7f1edc071590 2023-10-14T18:20:19.700 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:19.700 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:19.701 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:19.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f1edc071590 msgr2=0x7f1edc0739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.186:6816/1666789598,v1:172.21.15.186:6817/1666789598] conn(0x7f1edc071590 0x7f1edc0739b0 crc :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f1edc06af90 msgr2=0x7f1edc06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f1edc06af90 0x7f1edc06d450 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f1ee8006fd0 tx=0x7f1ee8008040 comp rx=0 tx=0).stop 2023-10-14T18:20:19.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 msgr2=0x7f1ef8156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 --2- 172.21.15.29:0/292411225 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1ef806c040 0x7f1ef8156ce0 secure :-1 s=READY pgs=3311 cs=0 l=1 rev1=1 crypto rx=0x7f1ef4000f80 tx=0x7f1ef4004a40 comp rx=0 tx=0).stop 2023-10-14T18:20:19.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efe7ab640 1 -- 172.21.15.29:0/292411225 reap_dead start 2023-10-14T18:20:19.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 shutdown_connections 2023-10-14T18:20:19.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 >> 172.21.15.29:0/292411225 conn(0x7f1ef80ce6a0 msgr2=0x7f1ef80b12c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 shutdown_connections 2023-10-14T18:20:19.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.698+0000 7f1efefac640 1 -- 172.21.15.29:0/292411225 wait complete. 2023-10-14T18:20:19.717 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-10-14T18:20:19.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.924+0000 7f2b5b59e640 1 Processor -- start 2023-10-14T18:20:19.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.925+0000 7f2b5b59e640 1 -- start start 2023-10-14T18:20:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c0709e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 0x7f2b5c0ac890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c0cdec0 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c160a90 con 0x7f2b5c070f20 2023-10-14T18:20:19.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.926+0000 7f2b5b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c0cd900 con 0x7f2b5c0c9230 2023-10-14T18:20:19.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43678/0 (socket says 172.21.15.29:43678) 2023-10-14T18:20:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 -- 172.21.15.29:0/2534742940 learned_addr learned my addr 172.21.15.29:0/2534742940 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5ad9d640 1 --2- 172.21.15.29:0/2534742940 >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 0x7f2b5c0ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b53fff640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c0709e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 -- 172.21.15.29:0/2534742940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 msgr2=0x7f2b5c0709e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c0709e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 -- 172.21.15.29:0/2534742940 >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 msgr2=0x7f2b5c0ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/2534742940 >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 0x7f2b5c0ac890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.933 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5a59c640 1 -- 172.21.15.29:0/2534742940 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c0cdbe0 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b53fff640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.927+0000 7f2b5ad9d640 1 --2- 172.21.15.29:0/2534742940 >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 0x7f2b5c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.928+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0c8c80 tx=0x7f2b4c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8bc1df189b6fd2c server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.934 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b59d9b640 1 -- 172.21.15.29:0/2534742940 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c02ca80 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b59d9b640 1 -- 172.21.15.29:0/2534742940 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c02cbe0 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b59d9b640 1 -- 172.21.15.29:0/2534742940 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c0343f0 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 -- 172.21.15.29:0/2534742940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c15c9f0 msgr2=0x7f2b5c0c8cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.935 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 secure :-1 s=READY pgs=3312 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c0c8c80 tx=0x7f2b4c02bee0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 -- 172.21.15.29:0/2534742940 shutdown_connections 2023-10-14T18:20:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/2534742940 >> v2:172.21.15.186:3300/0 conn(0x7f2b5c070f20 0x7f2b5c0ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.936 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c0709e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/2534742940 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c8cf0 unknown :-1 s=CLOSED pgs=3312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.929+0000 7f2b5b59e640 1 -- 172.21.15.29:0/2534742940 >> 172.21.15.29:0/2534742940 conn(0x7f2b5c0ce6a0 msgr2=0x7f2b5c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.930+0000 7f2b5b59e640 1 -- 172.21.15.29:0/2534742940 shutdown_connections 2023-10-14T18:20:19.937 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.930+0000 7f2b5b59e640 1 -- 172.21.15.29:0/2534742940 wait complete. 2023-10-14T18:20:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.932+0000 7f2b5b59e640 1 Processor -- start 2023-10-14T18:20:19.940 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.932+0000 7f2b5b59e640 1 -- start start 2023-10-14T18:20:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c15e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c49e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c158f60 con 0x7f2b5c070f20 2023-10-14T18:20:19.941 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c15f9b0 con 0x7f2b5c15c9f0 2023-10-14T18:20:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5b59e640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c15eda0 con 0x7f2b5c0c9230 2023-10-14T18:20:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5a59c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43682/0 (socket says 172.21.15.29:43682) 2023-10-14T18:20:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5a59c640 1 -- 172.21.15.29:0/3373801021 learned_addr learned my addr 172.21.15.29:0/3373801021 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:19.942 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5ad9d640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b53fff640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c15e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.933+0000 7f2b5a59c640 1 -- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 msgr2=0x7f2b5c15e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c15e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5a59c640 1 -- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 msgr2=0x7f2b5c0c49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:19.943 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5a59c640 1 -- 172.21.15.29:0/3373801021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c009b30 con 0x7f2b5c070f20 2023-10-14T18:20:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b53fff640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5ad9d640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5a59c640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c000f80 tx=0x7f2b4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.944 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c03e450 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c03e5d0 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b4c03e980 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c15f080 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.934+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c15f640 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.936+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c04e9b0 con 0x7f2b5c070f20 2023-10-14T18:20:19.945 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.937+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f2b4c02cd50 con 0x7f2b5c070f20 2023-10-14T18:20:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.938+0000 7f2b52ffd640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 0x7f2b3c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.938+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f2b4c0ac860 con 0x7f2b5c070f20 2023-10-14T18:20:19.946 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.938+0000 7f2b53fff640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 0x7f2b3c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.939+0000 7f2b53fff640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 0x7f2b3c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1f5800 tx=0x7f2b44008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:19.947 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:19.943+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f2b4c0776b0 con 0x7f2b5c070f20 2023-10-14T18:20:20.141 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.139+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f2b5c0704c0 con 0x7f2b3c06af90 2023-10-14T18:20:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.140+0000 7f2b52ffd640 1 -- 172.21.15.29:0/3373801021 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f2b4400c040 con 0x7f2b3c06af90 2023-10-14T18:20:20.142 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 5 to deep-scrub 2023-10-14T18:20:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 msgr2=0x7f2b3c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 0x7f2b3c06d450 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2b5c1f5800 tx=0x7f2b44008040 comp rx=0 tx=0).stop 2023-10-14T18:20:20.145 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 msgr2=0x7f2b5c158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 secure :-1 s=READY pgs=3313 cs=0 l=1 rev1=1 crypto rx=0x7f2b4c000f80 tx=0x7f2b4c004a40 comp rx=0 tx=0).stop 2023-10-14T18:20:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 shutdown_connections 2023-10-14T18:20:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f2b5c15c9f0 0x7f2b5c0c49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f2b3c06af90 0x7f2b3c06d450 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.146 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f2b5c0c9230 0x7f2b5c15e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 --2- 172.21.15.29:0/3373801021 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f2b5c070f20 0x7f2b5c158a20 unknown :-1 s=CLOSED pgs=3313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 >> 172.21.15.29:0/3373801021 conn(0x7f2b5c0ce6a0 msgr2=0x7f2b5c0aeeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 shutdown_connections 2023-10-14T18:20:20.147 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.143+0000 7f2b5b59e640 1 -- 172.21.15.29:0/3373801021 wait complete. 2023-10-14T18:20:20.163 INFO:tasks.ceph:Scrubbing osd.6 2023-10-14T18:20:20.163 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:20.396 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.395+0000 7f7b18275640 1 Processor -- start 2023-10-14T18:20:20.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.395+0000 7f7b18275640 1 -- start start 2023-10-14T18:20:20.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.396+0000 7f7b18275640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f7b10159da0 0x7f7b100c9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.396+0000 7f7b18275640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b100cce20 0x7f7b1006c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.396+0000 7f7b18275640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b1015dfb0 con 0x7f7b101599c0 2023-10-14T18:20:20.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.396+0000 7f7b18275640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1015e290 con 0x7f7b100cce20 2023-10-14T18:20:20.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.396+0000 7f7b18275640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b1015e570 con 0x7f7b10159da0 2023-10-14T18:20:20.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.397+0000 7f7b16812640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b100cce20 0x7f7b1006c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.397+0000 7f7b16812640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f7b100cce20 0x7f7b1006c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43692/0 (socket says 172.21.15.29:43692) 2023-10-14T18:20:20.403 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.397+0000 7f7b16812640 1 -- 172.21.15.29:0/3658539589 learned_addr learned my addr 172.21.15.29:0/3658539589 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.397+0000 7f7b16011640 1 --1- 172.21.15.29:0/3658539589 >> v1:172.21.15.186:6789/0 conn(0x7f7b101599c0 0x7f7b1015d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45918/0 (socket says 172.21.15.29:45918) 2023-10-14T18:20:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.397+0000 7f7b15810640 1 --2- 172.21.15.29:0/3658539589 >> v2:172.21.15.186:3300/0 conn(0x7f7b10159da0 0x7f7b100c9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.398+0000 7f7b16812640 1 -- 172.21.15.29:0/3658539589 >> v2:172.21.15.186:3300/0 conn(0x7f7b10159da0 msgr2=0x7f7b100c9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.398+0000 7f7b16812640 1 --2- 172.21.15.29:0/3658539589 >> v2:172.21.15.186:3300/0 conn(0x7f7b10159da0 0x7f7b100c9300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.398+0000 7f7b16812640 1 -- 172.21.15.29:0/3658539589 >> v1:172.21.15.186:6789/0 conn(0x7f7b101599c0 legacy=0x7f7b1015d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.398+0000 7f7b16812640 1 -- 172.21.15.29:0/3658539589 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b1006fda0 con 0x7f7b100cce20 2023-10-14T18:20:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.399+0000 7f7b16812640 1 --2- 172.21.15.29:0/3658539589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100cce20 0x7f7b1006c0d0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f7b10071260 tx=0x7f7b0802bf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=455933bfb1454196 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.399+0000 7f7b1500f640 1 -- 172.21.15.29:0/3658539589 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0802cb10 con 0x7f7b100cce20 2023-10-14T18:20:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.399+0000 7f7b1500f640 1 -- 172.21.15.29:0/3658539589 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0802cc70 con 0x7f7b100cce20 2023-10-14T18:20:20.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.399+0000 7f7b1500f640 1 -- 172.21.15.29:0/3658539589 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b080343f0 con 0x7f7b100cce20 2023-10-14T18:20:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 -- 172.21.15.29:0/3658539589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100cce20 msgr2=0x7f7b1006c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 --2- 172.21.15.29:0/3658539589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100cce20 0x7f7b1006c0d0 secure :-1 s=READY pgs=3314 cs=0 l=1 rev1=1 crypto rx=0x7f7b10071260 tx=0x7f7b0802bf70 comp rx=0 tx=0).stop 2023-10-14T18:20:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 -- 172.21.15.29:0/3658539589 shutdown_connections 2023-10-14T18:20:20.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 --2- 172.21.15.29:0/3658539589 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b100cce20 0x7f7b1006c0d0 unknown :-1 s=CLOSED pgs=3314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 --2- 172.21.15.29:0/3658539589 >> v2:172.21.15.186:3300/0 conn(0x7f7b10159da0 0x7f7b100c9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 -- 172.21.15.29:0/3658539589 >> 172.21.15.29:0/3658539589 conn(0x7f7b100cb360 msgr2=0x7f7b100cb740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:20.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 -- 172.21.15.29:0/3658539589 shutdown_connections 2023-10-14T18:20:20.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.400+0000 7f7b18275640 1 -- 172.21.15.29:0/3658539589 wait complete. 2023-10-14T18:20:20.409 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.402+0000 7f7b18275640 1 Processor -- start 2023-10-14T18:20:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.402+0000 7f7b18275640 1 -- start start 2023-10-14T18:20:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b100cce20 0x7f7b100c7e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b101599c0 0x7f7b10151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.410 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 0x7f7b100cb740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b101f3830 con 0x7f7b10159da0 2023-10-14T18:20:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b100c86e0 con 0x7f7b101599c0 2023-10-14T18:20:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b18275640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b101f3a80 con 0x7f7b100cce20 2023-10-14T18:20:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.403+0000 7f7b16812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 0x7f7b100cb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.411 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 0x7f7b100cb740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43706/0 (socket says 172.21.15.29:43706) 2023-10-14T18:20:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 -- 172.21.15.29:0/3908300596 learned_addr learned my addr 172.21.15.29:0/3908300596 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b15810640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b101599c0 0x7f7b10151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16011640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b100cce20 0x7f7b100c7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 -- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b100cce20 msgr2=0x7f7b100c7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.412 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f7b100cce20 0x7f7b100c7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 -- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b101599c0 msgr2=0x7f7b10151f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f7b101599c0 0x7f7b10151f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b08009b30 con 0x7f7b10159da0 2023-10-14T18:20:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.404+0000 7f7b16812640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 0x7f7b100cb740 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f7b08036040 tx=0x7f7b08004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.413 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.405+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b08009510 con 0x7f7b10159da0 2023-10-14T18:20:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.405+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b0803e450 con 0x7f7b10159da0 2023-10-14T18:20:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.405+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b0803e800 con 0x7f7b10159da0 2023-10-14T18:20:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.405+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b101f3d60 con 0x7f7b10159da0 2023-10-14T18:20:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.405+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b100c83b0 con 0x7f7b10159da0 2023-10-14T18:20:20.414 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.406+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7b1006fa50 con 0x7f7b10159da0 2023-10-14T18:20:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.407+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f7b0803e960 con 0x7f7b10159da0 2023-10-14T18:20:20.415 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.408+0000 7f7b06ffd640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f7af406af90 0x7f7af406d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.408+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f7b080acbb0 con 0x7f7b10159da0 2023-10-14T18:20:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.408+0000 7f7b06ffd640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7af4071590 0x7f7af40739b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.416 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.408+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7af4074080 con 0x7f7af4071590 2023-10-14T18:20:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.408+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7b080ad000 con 0x7f7b10159da0 2023-10-14T18:20:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.409+0000 7f7b16011640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f7af406af90 0x7f7af406d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.409+0000 7f7b15810640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7af4071590 0x7f7af40739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.417 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.409+0000 7f7b16011640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f7af406af90 0x7f7af406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000f60 tx=0x7f7b0c007c80 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.410+0000 7f7b15810640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7af4071590 0x7f7af40739b0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.418 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.412+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== osd.6 v2:172.21.15.186:6824/3237784830 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7f7af4074080 con 0x7f7af4071590 2023-10-14T18:20:20.429 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.428+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 --> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7b101f4220 con 0x7f7af4071590 2023-10-14T18:20:20.430 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.429+0000 7f7b06ffd640 1 -- 172.21.15.29:0/3908300596 <== osd.6 v2:172.21.15.186:6824/3237784830 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7b101f4220 con 0x7f7af4071590 2023-10-14T18:20:20.431 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:20.431 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:20.431 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.429+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7af4071590 msgr2=0x7f7af40739b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.432 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.429+0000 7f7b18275640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.186:6824/3237784830,v1:172.21.15.186:6825/3237784830] conn(0x7f7af4071590 0x7f7af40739b0 crc :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f7af406af90 msgr2=0x7f7af406d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f7af406af90 0x7f7af406d450 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f7b0c000f60 tx=0x7f7b0c007c80 comp rx=0 tx=0).stop 2023-10-14T18:20:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 msgr2=0x7f7b100cb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 --2- 172.21.15.29:0/3908300596 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f7b10159da0 0x7f7b100cb740 secure :-1 s=READY pgs=3315 cs=0 l=1 rev1=1 crypto rx=0x7f7b08036040 tx=0x7f7b08004a40 comp rx=0 tx=0).stop 2023-10-14T18:20:20.433 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b16812640 1 -- 172.21.15.29:0/3908300596 reap_dead start 2023-10-14T18:20:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 shutdown_connections 2023-10-14T18:20:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 >> 172.21.15.29:0/3908300596 conn(0x7f7b100cb360 msgr2=0x7f7b1006c150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 shutdown_connections 2023-10-14T18:20:20.434 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.430+0000 7f7b18275640 1 -- 172.21.15.29:0/3908300596 wait complete. 2023-10-14T18:20:20.452 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2023-10-14T18:20:20.683 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.681+0000 7fdd4c8e7640 1 Processor -- start 2023-10-14T18:20:20.684 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.682+0000 7fdd4c8e7640 1 -- start start 2023-10-14T18:20:20.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.683+0000 7fdd4c8e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd4806a4b0 0x7fdd480ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.683+0000 7fdd4c8e7640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd480cb030 con 0x7fdd4815ad70 2023-10-14T18:20:20.685 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.683+0000 7fdd4c8e7640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd480c91c0 con 0x7fdd480c9ca0 2023-10-14T18:20:20.686 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.683+0000 7fdd4c8e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480c94a0 con 0x7fdd4806a4b0 2023-10-14T18:20:20.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.684+0000 7fdd4659c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fdd4815ad70 0x7fdd48069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:55714/0 (socket says 172.21.15.29:55714) 2023-10-14T18:20:20.688 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.684+0000 7fdd4659c640 1 -- 172.21.15.29:0/174672062 learned_addr learned my addr 172.21.15.29:0/174672062 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:20.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.684+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd4806a4b0 0x7fdd480ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.684+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343391674 0 0) 0x7fdd480cb030 con 0x7fdd4815ad70 2023-10-14T18:20:20.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.684+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34003520 con 0x7fdd4815ad70 2023-10-14T18:20:20.689 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2502290139 0 0) 0x7fdd34003520 con 0x7fdd4815ad70 2023-10-14T18:20:20.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd4806a4b0 msgr2=0x7fdd480ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:20.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd4806a4b0 0x7fdd480ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 >> v1:172.21.15.186:6789/0 conn(0x7fdd480c9ca0 legacy=0x7fdd480ad4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.690 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480c9780 con 0x7fdd4815ad70 2023-10-14T18:20:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fdd38001670 con 0x7fdd4815ad70 2023-10-14T18:20:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 >> v1:172.21.15.29:6789/0 conn(0x7fdd4815ad70 legacy=0x7fdd48069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340039a0 0x7fdd34003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd340047c0 0x7fdd34004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.691 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd34005180 0x7fdd3400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480c94a0 con 0x7fdd340047c0 2023-10-14T18:20:20.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3400c080 con 0x7fdd34005180 2023-10-14T18:20:20.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3400c240 con 0x7fdd340039a0 2023-10-14T18:20:20.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd4806a4b0 0x7fdd480ca3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:20:20.692 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.685+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd340047c0 0x7fdd34004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd4659c640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340039a0 0x7fdd34003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd46d9d640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd34005180 0x7fdd3400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd3ffff640 1 -- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340039a0 msgr2=0x7fdd34003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:20.693 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340039a0 0x7fdd34003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd3ffff640 1 -- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd34005180 msgr2=0x7fdd3400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:20.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd34005180 0x7fdd3400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.694 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.686+0000 7fdd3ffff640 1 -- 172.21.15.29:0/174672062 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd34003520 con 0x7fdd340047c0 2023-10-14T18:20:20.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.687+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd340047c0 0x7fdd34004c40 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fdd30006fd0 tx=0x7fdd340057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfbc4a78068ef53 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.697 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.687+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30033600 con 0x7fdd340047c0 2023-10-14T18:20:20.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.687+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3002f050 con 0x7fdd340047c0 2023-10-14T18:20:20.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.687+0000 7fdd45d9b640 1 -- 172.21.15.29:0/174672062 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd30033980 con 0x7fdd340047c0 2023-10-14T18:20:20.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/174672062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd340047c0 msgr2=0x7fdd34004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/174672062 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd340047c0 0x7fdd34004c40 secure :-1 s=READY pgs=3317 cs=0 l=1 rev1=1 crypto rx=0x7fdd30006fd0 tx=0x7fdd340057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.698 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd46d9d640 1 -- 172.21.15.29:0/174672062 reap_dead start 2023-10-14T18:20:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/174672062 shutdown_connections 2023-10-14T18:20:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/174672062 >> 172.21.15.29:0/174672062 conn(0x7fdd480ce9b0 msgr2=0x7fdd480cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/174672062 shutdown_connections 2023-10-14T18:20:20.699 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.688+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/174672062 wait complete. 2023-10-14T18:20:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.690+0000 7fdd4c8e7640 1 Processor -- start 2023-10-14T18:20:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.690+0000 7fdd4c8e7640 1 -- start start 2023-10-14T18:20:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 0x7fdd480ae9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.700 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd481f4eb0 0x7fdd480af9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480c64c0 con 0x7fdd480c8730 2023-10-14T18:20:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480af200 con 0x7fdd481f4eb0 2023-10-14T18:20:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd4c8e7640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd480c5b20 con 0x7fdd340047c0 2023-10-14T18:20:20.701 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.691+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:43722/0 (socket says 172.21.15.29:43722) 2023-10-14T18:20:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 -- 172.21.15.29:0/279017399 learned_addr learned my addr 172.21.15.29:0/279017399 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd4659c640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 0x7fdd480ae9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 -- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 msgr2=0x7fdd480ae9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.702 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd46d9d640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd481f4eb0 0x7fdd480af9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 0x7fdd480ae9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 -- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd481f4eb0 msgr2=0x7fdd480af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd481f4eb0 0x7fdd480af9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.703 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 -- 172.21.15.29:0/279017399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd480c5e00 con 0x7fdd480c8730 2023-10-14T18:20:20.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd4659c640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 0x7fdd480ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:20.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.692+0000 7fdd3ffff640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7fdd48071960 tx=0x7fdd300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.693+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3003d850 con 0x7fdd480c8730 2023-10-14T18:20:20.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.693+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3003d9b0 con 0x7fdd480c8730 2023-10-14T18:20:20.704 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.693+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd3003dd40 con 0x7fdd480c8730 2023-10-14T18:20:20.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.693+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd480c60e0 con 0x7fdd480c8730 2023-10-14T18:20:20.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.693+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd481f2740 con 0x7fdd480c8730 2023-10-14T18:20:20.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.694+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd480c6600 con 0x7fdd480c8730 2023-10-14T18:20:20.705 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.695+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fdd30032420 con 0x7fdd480c8730 2023-10-14T18:20:20.706 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.696+0000 7fdd3effd640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 0x7fdd2806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.696+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fdd300ab690 con 0x7fdd480c8730 2023-10-14T18:20:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.696+0000 7fdd4659c640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 0x7fdd2806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.697+0000 7fdd4659c640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 0x7fdd2806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd38001bb0 tx=0x7fdd38000f50 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:20.707 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.700+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fdd480c6600 con 0x7fdd480c8730 2023-10-14T18:20:20.899 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.897+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7fdd480adbe0 con 0x7fdd2806af90 2023-10-14T18:20:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.898+0000 7fdd3effd640 1 -- 172.21.15.29:0/279017399 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fdd38005e60 con 0x7fdd2806af90 2023-10-14T18:20:20.900 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 6 to deep-scrub 2023-10-14T18:20:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 msgr2=0x7fdd2806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.903 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 0x7fdd2806d450 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fdd38001bb0 tx=0x7fdd38000f50 comp rx=0 tx=0).stop 2023-10-14T18:20:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 msgr2=0x7fdd481f2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 secure :-1 s=READY pgs=3318 cs=0 l=1 rev1=1 crypto rx=0x7fdd48071960 tx=0x7fdd300046c0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 shutdown_connections 2023-10-14T18:20:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fdd481f4eb0 0x7fdd480af9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.904 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fdd480c8730 0x7fdd481f2200 unknown :-1 s=CLOSED pgs=3318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fdd2806af90 0x7fdd2806d450 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 --2- 172.21.15.29:0/279017399 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fdd340047c0 0x7fdd480ae9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 >> 172.21.15.29:0/279017399 conn(0x7fdd480ce9b0 msgr2=0x7fdd480ce030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 shutdown_connections 2023-10-14T18:20:20.905 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:20.901+0000 7fdd4c8e7640 1 -- 172.21.15.29:0/279017399 wait complete. 2023-10-14T18:20:20.923 INFO:tasks.ceph:Scrubbing osd.7 2023-10-14T18:20:20.923 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2023-10-14T18:20:21.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.154+0000 7fc3aaf7a640 1 Processor -- start 2023-10-14T18:20:21.156 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.154+0000 7fc3aaf7a640 1 -- start start 2023-10-14T18:20:21.157 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aaf7a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc3a406c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aaf7a640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7fc3a40cda80 0x7fc3a415ed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aaf7a640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3a406c960 con 0x7fc3a40ad970 2023-10-14T18:20:21.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aaf7a640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3a415b9d0 con 0x7fc3a40cda80 2023-10-14T18:20:21.158 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aaf7a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415bcb0 con 0x7fc3a406c040 2023-10-14T18:20:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3a9777640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fc3a40ad970 0x7fc3a415e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50566/0 (socket says 172.21.15.29:50566) 2023-10-14T18:20:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3a9777640 1 -- 172.21.15.29:0/1936000221 learned_addr learned my addr 172.21.15.29:0/1936000221 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc3a406c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.161 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.156+0000 7fc3aa779640 1 --2- 172.21.15.29:0/1936000221 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40cda80 0x7fc3a415ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077006039 0 0) 0x7fc3a406c960 con 0x7fc3a40ad970 2023-10-14T18:20:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc390003520 con 0x7fc3a40ad970 2023-10-14T18:20:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328646469 0 0) 0x7fc390003520 con 0x7fc3a40ad970 2023-10-14T18:20:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 msgr2=0x7fc3a406c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.162 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc3a406c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40cda80 msgr2=0x7fc3a415ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 --2- 172.21.15.29:0/1936000221 >> v2:172.21.15.186:3300/0 conn(0x7fc3a40cda80 0x7fc3a415ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a415bf90 con 0x7fc3a40ad970 2023-10-14T18:20:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7fc394001670 con 0x7fc3a40ad970 2023-10-14T18:20:21.163 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 >> v1:172.21.15.29:6789/0 conn(0x7fc3a40ad970 legacy=0x7fc3a415e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390003970 0x7fc390003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3900047c0 0x7fc390004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc390005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.164 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415b9d0 con 0x7fc3900047c0 2023-10-14T18:20:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a415bcb0 con 0x7fc390003970 2023-10-14T18:20:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.157+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc39000b980 con 0x7fc3a406c040 2023-10-14T18:20:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3900047c0 0x7fc390004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3aa779640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc390005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.165 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390003970 0x7fc390003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 -- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 msgr2=0x7fc390005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a406c040 0x7fc390005180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 -- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390003970 msgr2=0x7fc390003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390003970 0x7fc390003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.166 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9777640 1 -- 172.21.15.29:0/1936000221 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc390003520 con 0x7fc3900047c0 2023-10-14T18:20:21.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.158+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc390003970 0x7fc390003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:21.167 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.159+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3900047c0 0x7fc390004c40 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c110 tx=0x7fc3900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7ca86d6c38ecf75 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.170 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc394005ad0 con 0x7fc3900047c0 2023-10-14T18:20:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc394005c50 con 0x7fc3900047c0 2023-10-14T18:20:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3a8f76640 1 -- 172.21.15.29:0/1936000221 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39402c810 con 0x7fc3900047c0 2023-10-14T18:20:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1936000221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3900047c0 msgr2=0x7fc390004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.171 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 --2- 172.21.15.29:0/1936000221 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3900047c0 0x7fc390004c40 secure :-1 s=READY pgs=3320 cs=0 l=1 rev1=1 crypto rx=0x7fc39400c110 tx=0x7fc3900057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aa779640 1 -- 172.21.15.29:0/1936000221 reap_dead start 2023-10-14T18:20:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1936000221 shutdown_connections 2023-10-14T18:20:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1936000221 >> 172.21.15.29:0/1936000221 conn(0x7fc3a40ce6a0 msgr2=0x7fc3a40ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:21.172 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1936000221 shutdown_connections 2023-10-14T18:20:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.160+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1936000221 wait complete. 2023-10-14T18:20:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.162+0000 7fc3aaf7a640 1 Processor -- start 2023-10-14T18:20:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.162+0000 7fc3aaf7a640 1 -- start start 2023-10-14T18:20:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.163+0000 7fc3aaf7a640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900047c0 0x7fc3a414fce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.173 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.163+0000 7fc3aaf7a640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40c5ed0 0x7fc3a40c6350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aaf7a640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 0x7fc3a4158a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aaf7a640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40b2340 con 0x7fc3a4158590 2023-10-14T18:20:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aaf7a640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a40c6890 con 0x7fc3900047c0 2023-10-14T18:20:21.174 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aaf7a640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3a4159310 con 0x7fc3a40c5ed0 2023-10-14T18:20:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 0x7fc3a4158a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 0x7fc3a4158a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46212/0 (socket says 172.21.15.29:46212) 2023-10-14T18:20:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 -- 172.21.15.29:0/1007434580 learned_addr learned my addr 172.21.15.29:0/1007434580 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40c5ed0 0x7fc3a40c6350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.175 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900047c0 0x7fc3a414fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 -- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40c5ed0 msgr2=0x7fc3a40c6350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fc3a40c5ed0 0x7fc3a40c6350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 -- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900047c0 msgr2=0x7fc3a414fce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900047c0 0x7fc3a414fce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.176 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3aa779640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3a40b19a0 con 0x7fc3a4158590 2023-10-14T18:20:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.164+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fc3900047c0 0x7fc3a414fce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc3aa779640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 0x7fc3a4158a10 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7fc3a406d2e0 tx=0x7fc39c00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c016620 con 0x7fc3a4158590 2023-10-14T18:20:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc39c0167a0 con 0x7fc3a4158590 2023-10-14T18:20:21.177 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc39c016b50 con 0x7fc3a4158590 2023-10-14T18:20:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3a40b1c80 con 0x7fc3a4158590 2023-10-14T18:20:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.165+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3a4150220 con 0x7fc3a4158590 2023-10-14T18:20:21.178 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.166+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc3a40cc550 con 0x7fc3a4158590 2023-10-14T18:20:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.167+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fc39c016cb0 con 0x7fc3a4158590 2023-10-14T18:20:21.179 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.168+0000 7fc39a7fc640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc38806afc0 0x7fc38806d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fc39c08a950 con 0x7fc3a4158590 2023-10-14T18:20:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc39a7fc640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7fc3880715c0 0x7fc3880739e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc3880740b0 con 0x7fc3880715c0 2023-10-14T18:20:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_get_version_reply(handle=1 version=276) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc39c08f2a0 con 0x7fc3a4158590 2023-10-14T18:20:21.180 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc38806afc0 0x7fc38806d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.181 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.169+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7fc3880715c0 0x7fc3880739e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.170+0000 7fc3a9f78640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc38806afc0 0x7fc38806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0006d40 tx=0x7fc3a0008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.170+0000 7fc3a9777640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7fc3880715c0 0x7fc3880739e0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.182 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.173+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== osd.7 v2:172.21.15.186:6808/4274731638 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+25607 (crc 0 0 0) 0x7fc3880740b0 con 0x7fc3880715c0 2023-10-14T18:20:21.189 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.187+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 --> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fc3a40cc270 con 0x7fc3880715c0 2023-10-14T18:20:21.190 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.188+0000 7fc39a7fc640 1 -- 172.21.15.29:0/1007434580 <== osd.7 v2:172.21.15.186:6808/4274731638 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fc3940058f0 con 0x7fc3880715c0 2023-10-14T18:20:21.190 INFO:teuthology.orchestra.run.smithi029.stdout:{ 2023-10-14T18:20:21.191 INFO:teuthology.orchestra.run.smithi029.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-10-14T18:20:21.191 INFO:teuthology.orchestra.run.smithi029.stdout:} 2023-10-14T18:20:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7fc3880715c0 msgr2=0x7fc3880739e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.186:6808/4274731638,v1:172.21.15.186:6809/4274731638] conn(0x7fc3880715c0 0x7fc3880739e0 crc :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc38806afc0 msgr2=0x7fc38806d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.192 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fc38806afc0 0x7fc38806d480 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc3a0006d40 tx=0x7fc3a0008040 comp rx=0 tx=0).stop 2023-10-14T18:20:21.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 msgr2=0x7fc3a4158a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 --2- 172.21.15.29:0/1007434580 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fc3a4158590 0x7fc3a4158a10 secure :-1 s=READY pgs=3321 cs=0 l=1 rev1=1 crypto rx=0x7fc3a406d2e0 tx=0x7fc39c00f520 comp rx=0 tx=0).stop 2023-10-14T18:20:21.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aa779640 1 -- 172.21.15.29:0/1007434580 reap_dead start 2023-10-14T18:20:21.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 shutdown_connections 2023-10-14T18:20:21.193 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 >> 172.21.15.29:0/1007434580 conn(0x7fc3a40ce6a0 msgr2=0x7fc3a4159bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:21.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 shutdown_connections 2023-10-14T18:20:21.194 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.189+0000 7fc3aaf7a640 1 -- 172.21.15.29:0/1007434580 wait complete. 2023-10-14T18:20:21.209 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2023-10-14T18:20:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.441+0000 7f46d23dc640 1 Processor -- start 2023-10-14T18:20:21.443 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.441+0000 7f46d23dc640 1 -- start start 2023-10-14T18:20:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.442+0000 7f46d23dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 0x7f46cc0ce370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.444 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.442+0000 7f46d23dc640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46cc06c140 0x7f46cc06c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.442+0000 7f46d23dc640 1 -- --> v1:172.21.15.186:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46cc070360 con 0x7f46cc159e90 2023-10-14T18:20:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.442+0000 7f46d23dc640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46cc06cb10 con 0x7f46cc06c140 2023-10-14T18:20:21.445 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46d23dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc15d890 con 0x7f46cc159ab0 2023-10-14T18:20:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 0x7f46cc0ce370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46d0979640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46cc06c140 0x7f46cc06c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.446 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46d0979640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f46cc06c140 0x7f46cc06c5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3300/0 says I am v2:172.21.15.29:44232/0 (socket says 172.21.15.29:44232) 2023-10-14T18:20:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46d0979640 1 -- 172.21.15.29:0/508660568 learned_addr learned my addr 172.21.15.29:0/508660568 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 0x7f46cc0ce370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.186:3301/0 says I am v2:172.21.15.29:36352/0 (socket says 172.21.15.29:36352) 2023-10-14T18:20:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.443+0000 7f46c3fff640 1 --1- 172.21.15.29:0/508660568 >> v1:172.21.15.186:6789/0 conn(0x7f46cc159e90 0x7f46cc06fc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.186:6789/0 says I am v1:172.21.15.29:45402/0 (socket says 172.21.15.29:45402) 2023-10-14T18:20:21.447 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.444+0000 7f46d0979640 1 -- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 msgr2=0x7f46cc0ce370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.444+0000 7f46d0979640 1 --2- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 0x7f46cc0ce370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.444+0000 7f46d0979640 1 -- 172.21.15.29:0/508660568 >> v1:172.21.15.186:6789/0 conn(0x7f46cc159e90 legacy=0x7f46cc06fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.448 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.444+0000 7f46d0979640 1 -- 172.21.15.29:0/508660568 --> v2:172.21.15.186:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46cc0cd1f0 con 0x7f46cc06c140 2023-10-14T18:20:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.445+0000 7f46d0979640 1 --2- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc06c140 0x7f46cc06c5d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009e80 tx=0x7f46c402be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d6f1b36fd10d5d server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.449 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.446+0000 7f46cb7fe640 1 -- 172.21.15.29:0/508660568 <== mon.1 v2:172.21.15.186:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c402c9c0 con 0x7f46cc06c140 2023-10-14T18:20:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.446+0000 7f46cb7fe640 1 -- 172.21.15.29:0/508660568 <== mon.1 v2:172.21.15.186:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46c402cb20 con 0x7f46cc06c140 2023-10-14T18:20:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.446+0000 7f46cb7fe640 1 -- 172.21.15.29:0/508660568 <== mon.1 v2:172.21.15.186:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46c40343f0 con 0x7f46cc06c140 2023-10-14T18:20:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 -- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc06c140 msgr2=0x7f46cc06c5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 --2- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc06c140 0x7f46cc06c5d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009e80 tx=0x7f46c402be20 comp rx=0 tx=0).stop 2023-10-14T18:20:21.450 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 -- 172.21.15.29:0/508660568 shutdown_connections 2023-10-14T18:20:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 --2- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc06c140 0x7f46cc06c5d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 --2- 172.21.15.29:0/508660568 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159ab0 0x7f46cc0ce370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 -- 172.21.15.29:0/508660568 >> 172.21.15.29:0/508660568 conn(0x7f46cc0ce9b0 msgr2=0x7f46cc0cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 -- 172.21.15.29:0/508660568 shutdown_connections 2023-10-14T18:20:21.451 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.447+0000 7f46d23dc640 1 -- 172.21.15.29:0/508660568 wait complete. 2023-10-14T18:20:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.449+0000 7f46d23dc640 1 Processor -- start 2023-10-14T18:20:21.452 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.449+0000 7f46d23dc640 1 -- start start 2023-10-14T18:20:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc159ab0 0x7f46cc1e4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159e90 0x7f46cc1e5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.454 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc1e3e50 con 0x7f46cc06c140 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc1e3890 con 0x7f46cc159ab0 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46d23dc640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46cc1e3b70 con 0x7f46cc159e90 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46cbfff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46224/0 (socket says 172.21.15.29:46224) 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46cbfff640 1 -- 172.21.15.29:0/3898746657 learned_addr learned my addr 172.21.15.29:0/3898746657 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:21.455 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.451+0000 7f46c3fff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc159ab0 0x7f46cc1e4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46d0979640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159e90 0x7f46cc1e5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 -- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159e90 msgr2=0x7f46cc1e5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159e90 0x7f46cc1e5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 -- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc159ab0 msgr2=0x7f46cc1e4d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.456 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc159ab0 0x7f46cc1e4d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 -- 172.21.15.29:0/3898746657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c4009b30 con 0x7f46cc06c140 2023-10-14T18:20:21.457 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46cbfff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00e870 tx=0x7f46bc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc0094f0 con 0x7f46cc06c140 2023-10-14T18:20:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46bc00c940 con 0x7f46cc06c140 2023-10-14T18:20:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46bc00ccf0 con 0x7f46cc06c140 2023-10-14T18:20:21.458 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.452+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46cc15a460 con 0x7f46cc06c140 2023-10-14T18:20:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.453+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46cc1573a0 con 0x7f46cc06c140 2023-10-14T18:20:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.454+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46cc04e160 con 0x7f46cc06c140 2023-10-14T18:20:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.455+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f46bc013470 con 0x7f46cc06c140 2023-10-14T18:20:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.456+0000 7f46c97fa640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 0x7f46ac06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.459 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.456+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f46bc0898d0 con 0x7f46cc06c140 2023-10-14T18:20:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.456+0000 7f46c3fff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 0x7f46ac06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.460 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.457+0000 7f46c3fff640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 0x7f46ac06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0c1f00 tx=0x7f46b4007c60 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.462 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.460+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f46bc054850 con 0x7f46cc06c140 2023-10-14T18:20:21.658 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.656+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f46cc0cd7b0 con 0x7f46ac06af90 2023-10-14T18:20:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.657+0000 7f46c97fa640 1 -- 172.21.15.29:0/3898746657 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f46b4005a30 con 0x7f46ac06af90 2023-10-14T18:20:21.659 INFO:teuthology.orchestra.run.smithi029.stderr:instructed osd(s) 7 to deep-scrub 2023-10-14T18:20:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 msgr2=0x7f46ac06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 0x7f46ac06d450 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f46cc0c1f00 tx=0x7f46b4007c60 comp rx=0 tx=0).stop 2023-10-14T18:20:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 msgr2=0x7f46cc156e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.662 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 secure :-1 s=READY pgs=3322 cs=0 l=1 rev1=1 crypto rx=0x7f46bc00e870 tx=0x7f46bc00ed30 comp rx=0 tx=0).stop 2023-10-14T18:20:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 shutdown_connections 2023-10-14T18:20:21.663 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f46ac06af90 0x7f46ac06d450 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f46cc159e90 0x7f46cc1e5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f46cc159ab0 0x7f46cc1e4d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 --2- 172.21.15.29:0/3898746657 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f46cc06c140 0x7f46cc156e60 unknown :-1 s=CLOSED pgs=3322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.664 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.660+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 >> 172.21.15.29:0/3898746657 conn(0x7f46cc0ce9b0 msgr2=0x7f46cc06b8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:21.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.661+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 shutdown_connections 2023-10-14T18:20:21.665 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.661+0000 7f46d23dc640 1 -- 172.21.15.29:0/3898746657 wait complete. 2023-10-14T18:20:21.683 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:20:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.912+0000 7f80ff7d5640 1 Processor -- start 2023-10-14T18:20:21.914 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.912+0000 7f80ff7d5640 1 -- start start 2023-10-14T18:20:21.915 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.913+0000 7f80ff7d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 0x7f80f806fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.913+0000 7f80ff7d5640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f80f80cc920 0x7f80f80ccdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80ff7d5640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f8159790 con 0x7f80f81593b0 2023-10-14T18:20:21.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80ff7d5640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f80c9b20 con 0x7f80f80cc920 2023-10-14T18:20:21.916 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80ff7d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80c9e00 con 0x7f80f80caa10 2023-10-14T18:20:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fd571640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f80f81593b0 0x7f80f806f550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:50580/0 (socket says 172.21.15.29:50580) 2023-10-14T18:20:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fd571640 1 -- 172.21.15.29:0/2046747346 learned_addr learned my addr 172.21.15.29:0/2046747346 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:21.917 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fdd72640 1 --2- 172.21.15.29:0/2046747346 >> v2:172.21.15.29:3300/0 conn(0x7f80f80cc920 0x7f80f80ccdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 0x7f80f806fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fdd72640 1 -- 172.21.15.29:0/2046747346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 msgr2=0x7f80f806fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fdd72640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 0x7f80f806fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.919 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fdd72640 1 -- 172.21.15.29:0/2046747346 >> v1:172.21.15.29:6789/0 conn(0x7f80f81593b0 legacy=0x7f80f806f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fdd72640 1 -- 172.21.15.29:0/2046747346 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f80ca0e0 con 0x7f80f80cc920 2023-10-14T18:20:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.914+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 0x7f80f806fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.916+0000 7f80fdd72640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80cc920 0x7f80f80ccdb0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009d90 tx=0x7f80f002bd90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d2c1d8342045ab server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.916+0000 7f80effff640 1 -- 172.21.15.29:0/2046747346 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f002cb40 con 0x7f80f80cc920 2023-10-14T18:20:21.920 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.916+0000 7f80effff640 1 -- 172.21.15.29:0/2046747346 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f002cca0 con 0x7f80f80cc920 2023-10-14T18:20:21.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.916+0000 7f80effff640 1 -- 172.21.15.29:0/2046747346 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f0034660 con 0x7f80f80cc920 2023-10-14T18:20:21.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2046747346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80cc920 msgr2=0x7f80f80ccdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80cc920 0x7f80f80ccdb0 secure :-1 s=READY pgs=3324 cs=0 l=1 rev1=1 crypto rx=0x7f80f0009d90 tx=0x7f80f002bd90 comp rx=0 tx=0).stop 2023-10-14T18:20:21.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2046747346 shutdown_connections 2023-10-14T18:20:21.921 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80cc920 0x7f80f80ccdb0 unknown :-1 s=CLOSED pgs=3324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2046747346 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f80caa10 0x7f80f806fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2046747346 >> 172.21.15.29:0/2046747346 conn(0x7f80f80cb410 msgr2=0x7f80f80cb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:21.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2046747346 shutdown_connections 2023-10-14T18:20:21.922 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.917+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2046747346 wait complete. 2023-10-14T18:20:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.919+0000 7f80ff7d5640 1 Processor -- start 2023-10-14T18:20:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.919+0000 7f80ff7d5640 1 -- start start 2023-10-14T18:20:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 0x7f80f80c7420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.925 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f81593b0 0x7f80f80ac890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80c6b00 con 0x7f80f80caa10 2023-10-14T18:20:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80c5ef0 con 0x7f80f80cc920 2023-10-14T18:20:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80ff7d5640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f80c61d0 con 0x7f80f81593b0 2023-10-14T18:20:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80fd571640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.926 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80fd571640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:46234/0 (socket says 172.21.15.29:46234) 2023-10-14T18:20:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80fd571640 1 -- 172.21.15.29:0/2727531718 learned_addr learned my addr 172.21.15.29:0/2727531718 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.920+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 0x7f80f80c7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 -- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f81593b0 msgr2=0x7f80f80ac890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fdd72640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f81593b0 0x7f80f80ac890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.927 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f81593b0 0x7f80f80ac890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 -- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 msgr2=0x7f80f80c7420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 0x7f80f80c7420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 -- 172.21.15.29:0/2727531718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0009a70 con 0x7f80f80caa10 2023-10-14T18:20:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 0x7f80f80c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:21.928 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80fd571640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f80f400c480 tx=0x7f80f400c940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f4005ce0 con 0x7f80f80caa10 2023-10-14T18:20:21.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80f4005e40 con 0x7f80f80caa10 2023-10-14T18:20:21.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f80c64b0 con 0x7f80f80caa10 2023-10-14T18:20:21.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80f40072f0 con 0x7f80f80caa10 2023-10-14T18:20:21.929 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.921+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f80c5a60 con 0x7f80f80caa10 2023-10-14T18:20:21.930 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.923+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f804e160 con 0x7f80f80caa10 2023-10-14T18:20:21.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.924+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f80f4007450 con 0x7f80f80caa10 2023-10-14T18:20:21.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.925+0000 7f80edffb640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 0x7f80dc06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:21.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.925+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f80f4088c40 con 0x7f80f80caa10 2023-10-14T18:20:21.931 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.925+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 0x7f80dc06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:21.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.926+0000 7f80fcd70640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 0x7f80dc06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80e8006fd0 tx=0x7f80e8008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:21.932 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:21.929+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f80f4053c70 con 0x7f80f80caa10 2023-10-14T18:20:22.123 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.122+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f80f80c5ce0 con 0x7f80dc06afc0 2023-10-14T18:20:22.127 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.125+0000 7f80edffb640 1 -- 172.21.15.29:0/2727531718 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48339 (secure 0 0 0) 0x7f80f80c5ce0 con 0x7f80dc06afc0 2023-10-14T18:20:22.128 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:22.128 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:20:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 msgr2=0x7f80dc06d480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 0x7f80dc06d480 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f80e8006fd0 tx=0x7f80e8008040 comp rx=0 tx=0).stop 2023-10-14T18:20:22.130 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 msgr2=0x7f80f80c46a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 secure :-1 s=READY pgs=3325 cs=0 l=1 rev1=1 crypto rx=0x7f80f400c480 tx=0x7f80f400c940 comp rx=0 tx=0).stop 2023-10-14T18:20:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 shutdown_connections 2023-10-14T18:20:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f80dc06afc0 0x7f80dc06d480 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f80f81593b0 0x7f80f80ac890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:22.131 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f80f80cc920 0x7f80f80c7420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 --2- 172.21.15.29:0/2727531718 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f80f80caa10 0x7f80f80c46a0 unknown :-1 s=CLOSED pgs=3325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 >> 172.21.15.29:0/2727531718 conn(0x7f80f80cb410 msgr2=0x7f80f80ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.128+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 shutdown_connections 2023-10-14T18:20:22.132 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:22.129+0000 7f80ff7d5640 1 -- 172.21.15.29:0/2727531718 wait complete. 2023-10-14T18:20:22.148 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2023-10-14T18:20:21.263494+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":462456,"kb_used_data":28424,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754512264,"statfs":{"total":773094113280,"available":772620558336,"internally_reserved":0,"allocated":29106176,"data_stored":15157528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004524"},"pg_stats":[{"pgid":"1.0","version":"273'403","reported_seq":1450,"reported_epoch":275,"state":"active+clean","last_fresh":"2023-10-14T18:20:07.195920+0000","last_change":"2023-10-14T16:40:15.622500+0000","last_active":"2023-10-14T18:20:07.195920+0000","last_peered":"2023-10-14T18:20:07.195920+0000","last_clean":"2023-10-14T18:20:07.195920+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:20:07.195920+0000","last_undegraded":"2023-10-14T18:20:07.195920+0000","last_fullsized":"2023-10-14T18:20:07.195920+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-14T16:40:14.147897+0000","last_clean_scrub_stamp":"2023-10-14T16:40:14.147897+0000","objects_scrubbed":0,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T20:04:43.291373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576036,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57380,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314460,"statfs":{"total":96636764160,"available":96578007040,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.0669999999999999,"5min":1.1419999999999999,"15min":1.1930000000000001},"min":{"1min":0.63,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.575,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.639},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.173,"15min":1.2210000000000001},"min":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.575,"5min":1.78,"15min":2.2149999999999999},"last":1.125}]},{"osd":1,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.1970000000000001,"5min":1.21,"15min":1.2729999999999999},"min":{"1min":0.77000000000000002,"5min":0.63100000000000001,"15min":0.63100000000000001},"max":{"1min":1.631,"5min":1.9059999999999999,"15min":2.214},"last":0.89800000000000002},{"interface":"front","average":{"1min":1.1419999999999999,"5min":1.2450000000000001,"15min":1.2989999999999999},"min":{"1min":0.72299999999999998,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.4299999999999999,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.298}]},{"osd":2,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.268,"5min":1.1990000000000001,"15min":1.2589999999999999},"min":{"1min":0.878,"5min":0.70899999999999996,"15min":0.70899999999999996},"max":{"1min":1.835,"5min":1.839,"15min":2.004},"last":1.5860000000000001},{"interface":"front","average":{"1min":1.2629999999999999,"5min":1.294,"15min":1.359},"min":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":2.552,"5min":2.552,"15min":2.552},"last":1.238}]},{"osd":3,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.3149999999999999,"5min":1.3420000000000001,"15min":1.4139999999999999},"min":{"1min":0.81799999999999995,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.75,"5min":1.8700000000000001,"15min":2.4430000000000001},"last":1.764},{"interface":"front","average":{"1min":1.325,"5min":1.448,"15min":1.5129999999999999},"min":{"1min":0.88300000000000001,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.6299999999999999,"5min":1.994,"15min":2.2810000000000001},"last":1.492}]},{"osd":4,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.319,"5min":1.3560000000000001,"15min":1.345},"min":{"1min":0.98999999999999999,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":1.748,"5min":2.0600000000000001,"15min":2.367},"last":1.4319999999999999},{"interface":"front","average":{"1min":1.389,"5min":1.3939999999999999,"15min":1.3939999999999999},"min":{"1min":1.1499999999999999,"5min":0.91300000000000003,"15min":0.90900000000000003},"max":{"1min":1.8169999999999999,"5min":2.2440000000000002,"15min":2.661},"last":1.679}]},{"osd":5,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.468,"15min":1.4730000000000001},"min":{"1min":1.107,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.355,"5min":2.355,"15min":2.516},"last":1.371},{"interface":"front","average":{"1min":1.48,"5min":1.518,"15min":1.5289999999999999},"min":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.1160000000000001,"5min":2.2559999999999998,"15min":2.448},"last":1.716}]},{"osd":6,"last update":"Sat Oct 14 18:19:30 2023","interfaces":[{"interface":"back","average":{"1min":1.5860000000000001,"5min":1.6279999999999999,"15min":1.637},"min":{"1min":1.0389999999999999,"5min":0.99199999999999999,"15min":0.99199999999999999},"max":{"1min":2.0310000000000001,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.5449999999999999},{"interface":"front","average":{"1min":1.5900000000000001,"5min":1.5429999999999999,"15min":1.5780000000000001},"min":{"1min":1.26,"5min":0.96999999999999997,"15min":0.96999999999999997},"max":{"1min":2.0059999999999998,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.865}]}]},{"osd":6,"up_from":13,"seq":55834576035,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2070000000000001,"15min":1.1970000000000001},"min":{"1min":0.70599999999999996,"5min":0.66600000000000004,"15min":0.58899999999999997},"max":{"1min":1.6950000000000001,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.081},{"interface":"front","average":{"1min":1.141,"5min":1.337,"15min":1.252},"min":{"1min":0.81399999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.9570000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":0.85199999999999998}]},{"osd":1,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.208,"15min":1.238},"min":{"1min":0.74399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.696,"5min":1.855,"15min":1.855},"last":0.88800000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.3149999999999999,"15min":1.357},"min":{"1min":0.77200000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5649999999999999,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.524}]},{"osd":2,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.427,"15min":1.3879999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0489999999999999,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.381},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.121,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.6859999999999999,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.161}]},{"osd":3,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3640000000000001,"15min":1.399},"min":{"1min":0.70099999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.831,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.597},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.548,"15min":1.478},"min":{"1min":0.98399999999999999,"5min":0.90300000000000002,"15min":0.75900000000000001},"max":{"1min":2.1389999999999998,"5min":14.805999999999999,"15min":14.805999999999999},"last":1.2050000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4239999999999999,"15min":1.403},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.75800000000000001},"max":{"1min":1.887,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.6599999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.502,"15min":1.4530000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2509999999999999,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":0.99099999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.425,"15min":1.4199999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.052},{"interface":"front","average":{"1min":1.581,"5min":1.599,"15min":1.597},"min":{"1min":0.96299999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.028,"5min":2.625,"15min":2.625},"last":1.72}]},{"osd":7,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.7929999999999999,"15min":1.6739999999999999},"min":{"1min":0.94999999999999996,"5min":0.86799999999999999,"15min":0.71499999999999997},"max":{"1min":2.347,"5min":16.541,"15min":16.541},"last":1.482},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6279999999999999,"15min":1.6100000000000001},"min":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":0.69699999999999995},"max":{"1min":2.3340000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.4410000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.147,"15min":1.1890000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6140000000000001,"5min":2.052,"15min":2.052},"last":1.232},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.214},"min":{"1min":0.70099999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5960000000000001,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":0.92100000000000004}]},{"osd":1,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.69899999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.754,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":0.98599999999999999},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.26,"15min":1.286},"min":{"1min":0.81699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6599999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.0469999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.175,"15min":1.222},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7330000000000001,"5min":1.764,"15min":1.9339999999999999},"last":1.4279999999999999},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.389},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.923},"last":1.3520000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.363,"15min":1.4390000000000001},"min":{"1min":0.91700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8149999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.304},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3999999999999999,"15min":1.4139999999999999},"min":{"1min":1.006,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.74,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.5389999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.413,"15min":1.3759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8680000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.48},{"interface":"front","average":{"1min":1.379,"5min":1.4610000000000001,"15min":1.4079999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.61799999999999999},"max":{"1min":1.712,"5min":2.516,"15min":2.516},"last":1.1990000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64900000000000002},"max":{"1min":1.9359999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.6419999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.6259999999999999,"15min":1.627},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.98,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.593}]},{"osd":7,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5600000000000001,"15min":1.5740000000000001},"min":{"1min":1.1160000000000001,"5min":0.95499999999999996,"15min":0.68600000000000005},"max":{"1min":1.8080000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.3859999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.5629999999999999},"min":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8460000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.7070000000000001}]}]},{"osd":4,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.044,"5min":1.1379999999999999,"15min":1.1899999999999999},"min":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":1.7989999999999999,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.405},{"interface":"front","average":{"1min":1.1870000000000001,"5min":1.2430000000000001,"15min":1.2729999999999999},"min":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"max":{"1min":1.5389999999999999,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":1.0169999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.214,"5min":1.258,"15min":1.321},"min":{"1min":0.76100000000000001,"5min":0.751,"15min":0.751},"max":{"1min":1.8240000000000001,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.3480000000000001},{"interface":"front","average":{"1min":1.163,"5min":1.23,"15min":1.2609999999999999},"min":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.661,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.84799999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2090000000000001,"5min":1.2569999999999999,"15min":1.2849999999999999},"min":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.6499999999999999,"5min":2.3570000000000002,"15min":3.524},"last":1.2789999999999999},{"interface":"front","average":{"1min":1.3140000000000001,"5min":1.3560000000000001,"15min":1.4239999999999999},"min":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.6919999999999999,"5min":1.823,"15min":2.2440000000000002},"last":1.573}]},{"osd":3,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.238,"5min":1.2649999999999999,"15min":1.331},"min":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.554,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":1.3740000000000001,"5min":1.397,"15min":1.4390000000000001},"min":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":1.746,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.532}]},{"osd":5,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.2549999999999999,"5min":1.3979999999999999,"15min":1.3919999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.7090000000000001,"5min":1.9319999999999999,"15min":2.02},"last":1.1910000000000001},{"interface":"front","average":{"1min":1.345,"5min":1.4039999999999999,"15min":1.3859999999999999},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":1.8740000000000001,"5min":2.407,"15min":2.407},"last":0.89800000000000002}]},{"osd":6,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.351,"5min":1.4259999999999999,"15min":1.4399999999999999},"min":{"1min":0.39100000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":1.861,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":0.92500000000000004},{"interface":"front","average":{"1min":1.5129999999999999,"5min":1.5580000000000001,"15min":1.575},"min":{"1min":0.754,"5min":0.754,"15min":0.754},"max":{"1min":1.9690000000000001,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.47}]},{"osd":7,"last update":"Sat Oct 14 18:19:13 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.534,"15min":1.526},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.0680000000000001,"5min":2.1070000000000002,"15min":2.7440000000000002},"last":1.1319999999999999},{"interface":"front","average":{"1min":1.5529999999999999,"5min":1.587,"15min":1.6020000000000001},"min":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.1070000000000002,"5min":2.597,"15min":3.8610000000000002},"last":1.633}]}]},{"osd":3,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.94099999999999995,"15min":0.97599999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0660000000000001,"15min":1.1619999999999999},"min":{"1min":0.54500000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.327,"5min":1.665,"15min":1.9970000000000001},"last":0.435}]},{"osd":1,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.93000000000000005,"15min":0.98999999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2849999999999999,"5min":1.782,"15min":1.919},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.083,"15min":1.161},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.536,"5min":1.742,"15min":1.8560000000000001},"last":0.47099999999999997}]},{"osd":2,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.27,"15min":1.3839999999999999},"min":{"1min":0.59199999999999997,"5min":0.50900000000000001,"15min":0.498},"max":{"1min":1.833,"5min":1.9970000000000001,"15min":2.032},"last":0.55600000000000005},{"interface":"front","average":{"1min":1.149,"5min":1.2370000000000001,"15min":1.3340000000000001},"min":{"1min":0.55500000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.792,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":0.5}]},{"osd":4,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3700000000000001,"15min":1.421},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.129,"5min":2.129,"15min":2.153},"last":1.1000000000000001},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.429,"15min":1.4650000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.915,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.89700000000000002}]},{"osd":5,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3819999999999999,"15min":1.4359999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6180000000000001,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":0.82999999999999996},{"interface":"front","average":{"1min":1.452,"5min":1.4670000000000001,"15min":1.534},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":2.891},"last":0.753}]},{"osd":6,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5649999999999999,"15min":1.611},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.29,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":0.85599999999999998},{"interface":"front","average":{"1min":1.581,"5min":1.627,"15min":1.6639999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.4119999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.036}]},{"osd":7,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.534,"15min":1.5880000000000001},"min":{"1min":0.85699999999999998,"5min":0.85499999999999998,"15min":0.77900000000000003},"max":{"1min":2.173,"5min":3.153,"15min":3.153},"last":0.877},{"interface":"front","average":{"1min":1.542,"5min":1.617,"15min":1.6679999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.589,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.0149999999999999}]}]},{"osd":2,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56800,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.94999999999999996,"15min":0.94399999999999995},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.4670000000000001,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.83499999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.93100000000000005,"15min":1.0600000000000001},"min":{"1min":0.439,"5min":0.438,"15min":0.438},"max":{"1min":1.3979999999999999,"5min":1.448,"15min":1.8660000000000001},"last":0.92100000000000004}]},{"osd":1,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":0.95899999999999996,"5min":0.94199999999999995,"15min":1.002},"min":{"1min":0.46600000000000003,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":1.3620000000000001,"5min":1.603,"15min":1.8680000000000001},"last":0.76100000000000001},{"interface":"front","average":{"1min":1.028,"5min":1.101,"15min":1.1990000000000001},"min":{"1min":0.60299999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.345,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.0600000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.2649999999999999},"min":{"1min":0.65800000000000003,"5min":0.52800000000000002,"15min":0.52800000000000002},"max":{"1min":1.508,"5min":1.8089999999999999,"15min":1.9690000000000001},"last":1.1080000000000001},{"interface":"front","average":{"1min":1.2,"5min":1.2809999999999999,"15min":1.363},"min":{"1min":0.54400000000000004,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7170000000000001,"5min":2.1579999999999999,"15min":2.165},"last":1.3700000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4410000000000001,"15min":1.4610000000000001},"min":{"1min":0.875,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":2.1259999999999999,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.492},{"interface":"front","average":{"1min":1.272,"5min":1.3069999999999999,"15min":1.355},"min":{"1min":0.80100000000000005,"5min":0.66500000000000004,"15min":0.66500000000000004},"max":{"1min":2.2490000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.1419999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.5229999999999999,"5min":1.395,"15min":1.4359999999999999},"min":{"1min":0.84999999999999998,"5min":0.53900000000000003,"15min":0.53900000000000003},"max":{"1min":1.9870000000000001,"5min":2.165,"15min":2.2839999999999998},"last":1.3120000000000001},{"interface":"front","average":{"1min":1.5349999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.198,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.2110000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.4930000000000001,"15min":1.5249999999999999},"min":{"1min":0.98699999999999999,"5min":0.747,"15min":0.747},"max":{"1min":2.2690000000000001,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.014},{"interface":"front","average":{"1min":1.5840000000000001,"5min":1.5529999999999999,"15min":1.601},"min":{"1min":0.98299999999999998,"5min":0.64500000000000002,"15min":0.64500000000000002},"max":{"1min":2.198,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.407}]},{"osd":7,"last update":"Sat Oct 14 18:19:28 2023","interfaces":[{"interface":"back","average":{"1min":1.514,"5min":1.5529999999999999,"15min":1.5860000000000001},"min":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"max":{"1min":2.323,"5min":2.323,"15min":2.323},"last":1.3460000000000001},{"interface":"front","average":{"1min":1.671,"5min":1.6259999999999999,"15min":1.6739999999999999},"min":{"1min":0.98499999999999999,"5min":0.76600000000000001,"15min":0.76600000000000001},"max":{"1min":2.3010000000000002,"5min":2.331,"15min":2.331},"last":1.4610000000000001}]}]},{"osd":1,"up_from":13,"seq":55834576036,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38},"max":{"1min":1.4570000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.77300000000000002},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.0920000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3600000000000001,"5min":2.847,"15min":2.847},"last":1.3240000000000001}]},{"osd":2,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.996,"15min":1.042},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.492,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":0.93300000000000005},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1659999999999999,"15min":1.2450000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6919999999999999,"5min":1.996,"15min":2.6859999999999999},"last":0.90000000000000002}]},{"osd":3,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2350000000000001,"15min":1.3540000000000001},"min":{"1min":0.60199999999999998,"5min":0.52700000000000002,"15min":0.44700000000000001},"max":{"1min":1.879,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.016},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1970000000000001,"15min":1.2869999999999999},"min":{"1min":0.54400000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5649999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.175}]},{"osd":4,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4119999999999999,"15min":1.454},"min":{"1min":0.748,"5min":0.748,"15min":0.73899999999999999},"max":{"1min":2.0329999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.425},{"interface":"front","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4259999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.64800000000000002},"max":{"1min":2.2010000000000001,"5min":2.218,"15min":2.899},"last":1.45}]},{"osd":5,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.405,"15min":1.4630000000000001},"min":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.857,"5min":2.2519999999999998,"15min":3.004},"last":1.274},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.502,"15min":1.5569999999999999},"min":{"1min":0.94999999999999996,"5min":0.90400000000000003,"15min":0.73199999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.5129999999999999},"last":1.5109999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5069999999999999,"15min":1.597},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.367,"5min":2.8999999999999999,"15min":2.931},"last":1.5569999999999999},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.615,"15min":1.6220000000000001},"min":{"1min":0.90200000000000002,"5min":0.871,"15min":0.871},"max":{"1min":2.2509999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.587}]},{"osd":7,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.6060000000000001,"15min":1.6830000000000001},"min":{"1min":0.83799999999999997,"5min":0.82299999999999995,"15min":0.57899999999999996},"max":{"1min":1.9830000000000001,"5min":3.363,"15min":3.363},"last":1.3899999999999999},{"interface":"front","average":{"1min":1.55,"5min":1.6259999999999999,"15min":1.6990000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.1629999999999998,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":1.627}]}]},{"osd":0,"up_from":13,"seq":55834576035,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53220,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318620,"statfs":{"total":96636764160,"available":96582266880,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.91800000000000004,"15min":0.96499999999999997},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.52,"5min":1.6890000000000001,"15min":2.6440000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.91200000000000003,"5min":1.036,"15min":1.1160000000000001},"min":{"1min":0.495,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.488,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.89600000000000002}]},{"osd":2,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.0289999999999999,"5min":1.093,"15min":1.167},"min":{"1min":0.45800000000000002,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":1.702,"5min":1.855,"15min":1.855},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.123,"15min":1.2250000000000001},"min":{"1min":0.47499999999999998,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.7430000000000001,"5min":1.845,"15min":2.5779999999999998},"last":1.3340000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1519999999999999,"5min":1.2509999999999999,"15min":1.3520000000000001},"min":{"1min":0.52400000000000002,"5min":0.501,"15min":0.501},"max":{"1min":2.2850000000000001,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.0680000000000001},{"interface":"front","average":{"1min":1.1439999999999999,"5min":1.2869999999999999,"15min":1.3759999999999999},"min":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":1.8580000000000001,"5min":1.883,"15min":2.194},"last":1.506}]},{"osd":4,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1619999999999999,"5min":1.2569999999999999,"15min":1.3069999999999999},"min":{"1min":0.60899999999999999,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.583,"5min":1.8069999999999999,"15min":2.4670000000000001},"last":1.2889999999999999},{"interface":"front","average":{"1min":1.381,"5min":1.4550000000000001,"15min":1.496},"min":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0329999999999999,"5min":2.2309999999999999,"15min":2.4889999999999999},"last":1.3919999999999999}]},{"osd":5,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.1910000000000001,"5min":1.29,"15min":1.3600000000000001},"min":{"1min":0.67800000000000005,"5min":0.626,"15min":0.626},"max":{"1min":1.629,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.01},{"interface":"front","average":{"1min":1.375,"5min":1.488,"15min":1.5549999999999999},"min":{"1min":0.91000000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":1.857,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.4339999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.4470000000000001,"5min":1.5860000000000001,"15min":1.6359999999999999},"min":{"1min":0.89900000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":1.988,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.6020000000000001},{"interface":"front","average":{"1min":1.3129999999999999,"5min":1.4510000000000001,"15min":1.4970000000000001},"min":{"1min":0.75600000000000001,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.1629999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.48}]},{"osd":7,"last update":"Sat Oct 14 18:19:22 2023","interfaces":[{"interface":"back","average":{"1min":1.51,"5min":1.571,"15min":1.6399999999999999},"min":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.2410000000000001,"5min":2.3620000000000001,"15min":2.4630000000000001},"last":1.2350000000000001},{"interface":"front","average":{"1min":1.4330000000000001,"5min":1.5209999999999999,"15min":1.6060000000000001},"min":{"1min":0.75,"5min":0.75,"15min":0.75},"max":{"1min":2.3250000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.571}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:20:22.152 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-10-14T16:40:14.147897+0000 time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=16, tm_min=40, tm_sec=14, tm_wday=5, tm_yday=287, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=10, tm_mday=14, tm_hour=18, tm_min=20, tm_sec=14, tm_wday=5, tm_yday=287, tm_isdst=0) 2023-10-14T18:20:22.153 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-10-14T18:20:42.154 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-10-14T18:20:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.388+0000 7fb6332ea640 1 Processor -- start 2023-10-14T18:20:42.389 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.388+0000 7fb6332ea640 1 -- start start 2023-10-14T18:20:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6332ea640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c0cca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6332ea640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7fb62c0ccfb0 0x7fb62c0cd440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.392 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6332ea640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb62c15b9d0 con 0x7fb62c06c040 2023-10-14T18:20:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6332ea640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb62c15bcb0 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6332ea640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c15bf90 con 0x7fb62c0ad970 2023-10-14T18:20:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6322e8640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7fb62c06c040 0x7fb62c06c420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45816/0 (socket says 172.21.15.29:45816) 2023-10-14T18:20:42.393 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb6322e8640 1 -- 172.21.15.29:0/534137703 learned_addr learned my addr 172.21.15.29:0/534137703 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.390+0000 7fb632ae9640 1 --2- 172.21.15.29:0/534137703 >> v2:172.21.15.29:3300/0 conn(0x7fb62c0ccfb0 0x7fb62c0cd440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb631ae7640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c0cca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb632ae9640 1 -- 172.21.15.29:0/534137703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 msgr2=0x7fb62c0cca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb632ae9640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c0cca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.394 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb632ae9640 1 -- 172.21.15.29:0/534137703 >> v1:172.21.15.29:6789/0 conn(0x7fb62c06c040 legacy=0x7fb62c06c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb632ae9640 1 -- 172.21.15.29:0/534137703 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb62c0cc270 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.395 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.391+0000 7fb6312e6640 1 -- 172.21.15.29:0/534137703 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414451144 0 0) 0x7fb62c15b9d0 con 0x7fb62c06c040 2023-10-14T18:20:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.392+0000 7fb632ae9640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0cd440 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fb624009dc0 tx=0x7fb62402be90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68f3f0612a86d2f0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.392+0000 7fb6312e6640 1 -- 172.21.15.29:0/534137703 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb62402cc40 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.392+0000 7fb6312e6640 1 -- 172.21.15.29:0/534137703 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb62402cda0 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.397 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.392+0000 7fb6312e6640 1 -- 172.21.15.29:0/534137703 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb624034660 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 -- 172.21.15.29:0/534137703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0ccfb0 msgr2=0x7fb62c0cd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0cd440 secure :-1 s=READY pgs=3327 cs=0 l=1 rev1=1 crypto rx=0x7fb624009dc0 tx=0x7fb62402be90 comp rx=0 tx=0).stop 2023-10-14T18:20:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 -- 172.21.15.29:0/534137703 shutdown_connections 2023-10-14T18:20:42.398 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0cd440 unknown :-1 s=CLOSED pgs=3327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 --2- 172.21.15.29:0/534137703 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c0cca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 -- 172.21.15.29:0/534137703 >> 172.21.15.29:0/534137703 conn(0x7fb62c0ce6a0 msgr2=0x7fb62c0ceab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 -- 172.21.15.29:0/534137703 shutdown_connections 2023-10-14T18:20:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.393+0000 7fb6332ea640 1 -- 172.21.15.29:0/534137703 wait complete. 2023-10-14T18:20:42.399 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.395+0000 7fb6332ea640 1 Processor -- start 2023-10-14T18:20:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.395+0000 7fb6332ea640 1 -- start start 2023-10-14T18:20:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.400 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0afce0 con 0x7fb62c06c040 2023-10-14T18:20:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0afef0 con 0x7fb62c0ccfb0 2023-10-14T18:20:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6332ea640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb62c0b01d0 con 0x7fb62c0ad970 2023-10-14T18:20:42.401 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6322e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6322e8640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59476/0 (socket says 172.21.15.29:59476) 2023-10-14T18:20:42.402 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.396+0000 7fb6322e8640 1 -- 172.21.15.29:0/1418669201 learned_addr learned my addr 172.21.15.29:0/1418669201 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb632ae9640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 -- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 msgr2=0x7fb62c150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb631ae7640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.404 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c150a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 -- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 msgr2=0x7fb62c0ca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0ca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 -- 172.21.15.29:0/1418669201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb624009a70 con 0x7fb62c06c040 2023-10-14T18:20:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb632ae9640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:42.405 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6322e8640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c4d0 tx=0x7fb62800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6280074e0 con 0x7fb62c06c040 2023-10-14T18:20:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb628007660 con 0x7fb62c06c040 2023-10-14T18:20:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb628004a20 con 0x7fb62c06c040 2023-10-14T18:20:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.397+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb62c0b04b0 con 0x7fb62c06c040 2023-10-14T18:20:42.406 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.398+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb62c154b50 con 0x7fb62c06c040 2023-10-14T18:20:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.399+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb62c06b330 con 0x7fb62c06c040 2023-10-14T18:20:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.400+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7fb628004b80 con 0x7fb62c06c040 2023-10-14T18:20:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.401+0000 7fb622ffd640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 0x7fb61006d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.401+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7fb6280889e0 con 0x7fb62c06c040 2023-10-14T18:20:42.407 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.402+0000 7fb631ae7640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 0x7fb61006d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.402+0000 7fb631ae7640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 0x7fb61006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb62c152a40 tx=0x7fb61c008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.408 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.406+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7fb6280538d0 con 0x7fb62c06c040 2023-10-14T18:20:42.601 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.600+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 --> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb62c0b0060 con 0x7fb61006af90 2023-10-14T18:20:42.607 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.606+0000 7fb622ffd640 1 -- 172.21.15.29:0/1418669201 <== mgr.13113 v2:172.21.15.29:6832/748820097 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48301 (secure 0 0 0) 0x7fb62c0b0060 con 0x7fb61006af90 2023-10-14T18:20:42.608 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:42.609 INFO:teuthology.orchestra.run.smithi029.stderr:dumped all 2023-10-14T18:20:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 msgr2=0x7fb61006d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.610 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 0x7fb61006d450 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fb62c152a40 tx=0x7fb61c008040 comp rx=0 tx=0).stop 2023-10-14T18:20:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 msgr2=0x7fb62c154610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 secure :-1 s=READY pgs=3328 cs=0 l=1 rev1=1 crypto rx=0x7fb62800c4d0 tx=0x7fb62800c990 comp rx=0 tx=0).stop 2023-10-14T18:20:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 shutdown_connections 2023-10-14T18:20:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7fb62c0ccfb0 0x7fb62c0ca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.611 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7fb61006af90 0x7fb61006d450 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7fb62c0ad970 0x7fb62c150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 --2- 172.21.15.29:0/1418669201 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7fb62c06c040 0x7fb62c154610 unknown :-1 s=CLOSED pgs=3328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 >> 172.21.15.29:0/1418669201 conn(0x7fb62c0ce6a0 msgr2=0x7fb62c15db00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 shutdown_connections 2023-10-14T18:20:42.612 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.609+0000 7fb6332ea640 1 -- 172.21.15.29:0/1418669201 wait complete. 2023-10-14T18:20:42.627 INFO:teuthology.orchestra.run.smithi029.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2023-10-14T18:20:41.270215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":462440,"kb_used_data":28424,"kb_used_omap":12,"kb_used_meta":433907,"kb_avail":754512280,"statfs":{"total":773094113280,"available":772620574720,"internally_reserved":0,"allocated":29106176,"data_stored":15157528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":444321336},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004335"},"pg_stats":[{"pgid":"1.0","version":"276'404","reported_seq":1460,"reported_epoch":276,"state":"active+clean","last_fresh":"2023-10-14T18:20:22.579666+0000","last_change":"2023-10-14T18:20:22.579784+0000","last_active":"2023-10-14T18:20:22.579666+0000","last_peered":"2023-10-14T18:20:22.579666+0000","last_clean":"2023-10-14T18:20:22.579666+0000","last_became_active":"2023-10-14T16:40:15.620105+0000","last_became_peered":"2023-10-14T16:40:15.620105+0000","last_unstale":"2023-10-14T18:20:22.579666+0000","last_undegraded":"2023-10-14T18:20:22.579666+0000","last_fullsized":"2023-10-14T18:20:22.579666+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'404","last_scrub_stamp":"2023-10-14T18:20:22.579485+0000","last_deep_scrub":"276'404","last_deep_scrub_stamp":"2023-10-14T18:20:22.579485+0000","last_clean_scrub_stamp":"2023-10-14T18:20:22.579485+0000","objects_scrubbed":2,"log_size":404,"log_dups_size":0,"ondisk_log_size":404,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-15T23:18:17.367593+0000","scrub_duration":0.011378556,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2612,"num_read_kb":5456,"num_write":679,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":404,"ondisk_log_size":404,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834576040,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":57372,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94314468,"statfs":{"total":96636764160,"available":96578015232,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54655435},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1799999999999999,"5min":1.163,"15min":1.1919999999999999},"min":{"1min":0.61799999999999999,"5min":0.61699999999999999,"15min":0.61699999999999999},"max":{"1min":1.9379999999999999,"5min":2.0489999999999999,"15min":2.1669999999999998},"last":1.0149999999999999},{"interface":"front","average":{"1min":1.2390000000000001,"5min":1.177,"15min":1.222},"min":{"1min":0.74399999999999999,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.9099999999999999,"5min":1.9099999999999999,"15min":2.2149999999999999},"last":1.6879999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.2130000000000001,"15min":1.264},"min":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"max":{"1min":1.917,"5min":1.917,"15min":2.214},"last":1.4610000000000001},{"interface":"front","average":{"1min":1.1990000000000001,"5min":1.2270000000000001,"15min":1.294},"min":{"1min":0.70699999999999996,"5min":0.66600000000000004,"15min":0.66600000000000004},"max":{"1min":1.675,"5min":2.0510000000000002,"15min":2.0510000000000002},"last":1.5189999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.169,"5min":1.2050000000000001,"15min":1.2529999999999999},"min":{"1min":0.81899999999999995,"5min":0.73799999999999999,"15min":0.70899999999999996},"max":{"1min":1.6080000000000001,"5min":1.839,"15min":2.004},"last":1.5700000000000001},{"interface":"front","average":{"1min":1.282,"5min":1.2969999999999999,"15min":1.351},"min":{"1min":0.89300000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"max":{"1min":1.8919999999999999,"5min":2.552,"15min":2.552},"last":1.6279999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4079999999999999},"min":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.73999999999999999},"max":{"1min":1.873,"5min":1.873,"15min":2.4430000000000001},"last":1.357},{"interface":"front","average":{"1min":1.474,"5min":1.462,"15min":1.512},"min":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.79600000000000004},"max":{"1min":2.1240000000000001,"5min":2.1240000000000001,"15min":2.2810000000000001},"last":1.7310000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.2669999999999999,"5min":1.3420000000000001,"15min":1.349},"min":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"max":{"1min":2.0209999999999999,"5min":2.0600000000000001,"15min":2.367},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.3500000000000001,"5min":1.397,"15min":1.395},"min":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"max":{"1min":1.8240000000000001,"5min":2.2440000000000002,"15min":2.661},"last":1.768}]},{"osd":5,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.373,"5min":1.4550000000000001,"15min":1.47},"min":{"1min":0.97799999999999998,"5min":0.97799999999999998,"15min":0.72899999999999998},"max":{"1min":1.8680000000000001,"5min":2.355,"15min":2.516},"last":1.8080000000000001},{"interface":"front","average":{"1min":1.4830000000000001,"5min":1.5189999999999999,"15min":1.5289999999999999},"min":{"1min":0.93700000000000006,"5min":0.93700000000000006,"15min":0.93700000000000006},"max":{"1min":2.0379999999999998,"5min":2.2559999999999998,"15min":2.448},"last":1.9299999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5329999999999999,"5min":1.615,"15min":1.6359999999999999},"min":{"1min":1.032,"5min":1.032,"15min":0.99199999999999999},"max":{"1min":2.0649999999999999,"5min":2.4140000000000001,"15min":2.4849999999999999},"last":1.825},{"interface":"front","average":{"1min":1.5649999999999999,"5min":1.5509999999999999,"15min":1.5780000000000001},"min":{"1min":1.079,"5min":1.079,"15min":0.96999999999999997},"max":{"1min":2.0899999999999999,"5min":2.1040000000000001,"15min":2.8159999999999998},"last":1.897}]}]},{"osd":6,"up_from":13,"seq":55834576039,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":46135751},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.2270000000000001,"5min":1.2070000000000001,"15min":1.1970000000000001},"min":{"1min":0.70599999999999996,"5min":0.66600000000000004,"15min":0.58899999999999997},"max":{"1min":1.6950000000000001,"5min":3.0409999999999999,"15min":3.0409999999999999},"last":1.1839999999999999},{"interface":"front","average":{"1min":1.141,"5min":1.337,"15min":1.252},"min":{"1min":0.81399999999999995,"5min":0.71599999999999997,"15min":0.71599999999999997},"max":{"1min":1.9570000000000001,"5min":14.970000000000001,"15min":14.970000000000001},"last":1.766}]},{"osd":1,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.232,"5min":1.208,"15min":1.238},"min":{"1min":0.74399999999999999,"5min":0.69999999999999996,"15min":0.69999999999999996},"max":{"1min":1.696,"5min":1.855,"15min":1.855},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.228,"5min":1.3149999999999999,"15min":1.357},"min":{"1min":0.77200000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"max":{"1min":1.5649999999999999,"5min":2.5550000000000002,"15min":2.5550000000000002},"last":1.5269999999999999}]},{"osd":2,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3859999999999999,"5min":1.427,"15min":1.3879999999999999},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":2.0489999999999999,"5min":4.4509999999999996,"15min":4.4509999999999996},"last":1.3260000000000001},{"interface":"front","average":{"1min":1.429,"5min":1.429,"15min":1.4299999999999999},"min":{"1min":1.121,"5min":0.79800000000000004,"15min":0.79800000000000004},"max":{"1min":1.6859999999999999,"5min":3.3980000000000001,"15min":3.3980000000000001},"last":1.7190000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.3260000000000001,"5min":1.3640000000000001,"15min":1.399},"min":{"1min":0.70099999999999996,"5min":0.67200000000000004,"15min":0.67200000000000004},"max":{"1min":1.831,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":1.4099999999999999},{"interface":"front","average":{"1min":1.3799999999999999,"5min":1.548,"15min":1.478},"min":{"1min":0.98399999999999999,"5min":0.90300000000000002,"15min":0.75900000000000001},"max":{"1min":2.1389999999999998,"5min":14.805999999999999,"15min":14.805999999999999},"last":1.5700000000000001}]},{"osd":4,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.379,"5min":1.4239999999999999,"15min":1.403},"min":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.75800000000000001},"max":{"1min":1.887,"5min":3.1000000000000001,"15min":3.1000000000000001},"last":1.0529999999999999},{"interface":"front","average":{"1min":1.4590000000000001,"5min":1.502,"15min":1.4530000000000001},"min":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":2.2509999999999999,"5min":4.5090000000000003,"15min":4.5090000000000003},"last":1.4530000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.4179999999999999,"5min":1.425,"15min":1.4199999999999999},"min":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.624},"max":{"1min":2.1099999999999999,"5min":2.1389999999999998,"15min":2.4940000000000002},"last":1.4850000000000001},{"interface":"front","average":{"1min":1.581,"5min":1.599,"15min":1.597},"min":{"1min":0.96299999999999997,"5min":0.73799999999999999,"15min":0.73799999999999999},"max":{"1min":2.028,"5min":2.625,"15min":2.625},"last":1.599}]},{"osd":7,"last update":"Sat Oct 14 18:20:17 2023","interfaces":[{"interface":"back","average":{"1min":1.589,"5min":1.7929999999999999,"15min":1.6739999999999999},"min":{"1min":0.94999999999999996,"5min":0.86799999999999999,"15min":0.71499999999999997},"max":{"1min":2.347,"5min":16.541,"15min":16.541},"last":1.661},{"interface":"front","average":{"1min":1.6399999999999999,"5min":1.6279999999999999,"15min":1.6100000000000001},"min":{"1min":1.1220000000000001,"5min":1.1220000000000001,"15min":0.69699999999999995},"max":{"1min":2.3340000000000001,"5min":2.3460000000000001,"15min":2.8769999999999998},"last":1.6180000000000001}]}]},{"osd":5,"up_from":13,"seq":55834576040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":48476,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94323364,"statfs":{"total":96636764160,"available":96587124736,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":46135750},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.014,"5min":1.147,"15min":1.1890000000000001},"min":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"max":{"1min":1.6140000000000001,"5min":2.052,"15min":2.052},"last":1.73},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.1399999999999999,"15min":1.214},"min":{"1min":0.70099999999999996,"5min":0.59999999999999998,"15min":0.59999999999999998},"max":{"1min":1.5960000000000001,"5min":1.9039999999999999,"15min":2.1070000000000002},"last":1.6870000000000001}]},{"osd":1,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.171,"5min":1.2190000000000001,"15min":1.26},"min":{"1min":0.69899999999999995,"5min":0.57599999999999996,"15min":0.57599999999999996},"max":{"1min":1.754,"5min":1.8049999999999999,"15min":1.8899999999999999},"last":1.609},{"interface":"front","average":{"1min":1.2070000000000001,"5min":1.26,"15min":1.286},"min":{"1min":0.81699999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"max":{"1min":1.6599999999999999,"5min":2.3929999999999998,"15min":2.3929999999999998},"last":1.52}]},{"osd":2,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.1679999999999999,"5min":1.175,"15min":1.222},"min":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"max":{"1min":1.7330000000000001,"5min":1.764,"15min":1.9339999999999999},"last":1.643},{"interface":"front","average":{"1min":1.3169999999999999,"5min":1.365,"15min":1.389},"min":{"1min":0.623,"5min":0.623,"15min":0.623},"max":{"1min":1.595,"5min":1.8779999999999999,"15min":1.923},"last":1.2549999999999999}]},{"osd":3,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.288,"5min":1.363,"15min":1.4390000000000001},"min":{"1min":0.91700000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"max":{"1min":1.8149999999999999,"5min":2.1930000000000001,"15min":2.4969999999999999},"last":1.3340000000000001},{"interface":"front","average":{"1min":1.4179999999999999,"5min":1.3999999999999999,"15min":1.4139999999999999},"min":{"1min":1.006,"5min":0.80700000000000005,"15min":0.80700000000000005},"max":{"1min":1.74,"5min":2.1000000000000001,"15min":2.4569999999999999},"last":1.5669999999999999}]},{"osd":4,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.413,"15min":1.3759999999999999},"min":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"max":{"1min":1.8680000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.379,"5min":1.4610000000000001,"15min":1.4079999999999999},"min":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.61799999999999999},"max":{"1min":1.712,"5min":2.516,"15min":2.516},"last":1.111}]},{"osd":6,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.3460000000000001,"5min":1.4159999999999999,"15min":1.4099999999999999},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64900000000000002},"max":{"1min":1.9359999999999999,"5min":2.1539999999999999,"15min":2.1539999999999999},"last":1.3919999999999999},{"interface":"front","average":{"1min":1.5329999999999999,"5min":1.6259999999999999,"15min":1.627},"min":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"max":{"1min":1.98,"5min":2.7810000000000001,"15min":2.7810000000000001},"last":1.839}]},{"osd":7,"last update":"Sat Oct 14 18:20:00 2023","interfaces":[{"interface":"back","average":{"1min":1.466,"5min":1.5600000000000001,"15min":1.5740000000000001},"min":{"1min":1.1160000000000001,"5min":0.95499999999999996,"15min":0.68600000000000005},"max":{"1min":1.8080000000000001,"5min":2.6640000000000001,"15min":2.6640000000000001},"last":1.7929999999999999},{"interface":"front","average":{"1min":1.532,"5min":1.5880000000000001,"15min":1.5629999999999999},"min":{"1min":0.90600000000000003,"5min":0.90300000000000002,"15min":0.77900000000000003},"max":{"1min":1.8460000000000001,"5min":2.3570000000000002,"15min":2.3570000000000002},"last":1.7649999999999999}]}]},{"osd":4,"up_from":13,"seq":55834576040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":84512,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":81086,"kb_avail":94287328,"statfs":{"total":96636764160,"available":96550223872,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":83032518},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.1819999999999999,"5min":1.169,"15min":1.1910000000000001},"min":{"1min":0.67900000000000005,"5min":0.53000000000000003,"15min":0.53000000000000003},"max":{"1min":2.1240000000000001,"5min":2.2149999999999999,"15min":2.2149999999999999},"last":1.0309999999999999},{"interface":"front","average":{"1min":1.1930000000000001,"5min":1.2669999999999999,"15min":1.268},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"max":{"1min":1.762,"5min":2.2610000000000001,"15min":2.3140000000000001},"last":0.92600000000000005}]},{"osd":1,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.069,"5min":1.24,"15min":1.3049999999999999},"min":{"1min":0.65400000000000003,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4119999999999999,"5min":2.0129999999999999,"15min":2.0129999999999999},"last":1.302},{"interface":"front","average":{"1min":1.149,"5min":1.2370000000000001,"15min":1.25},"min":{"1min":0.748,"5min":0.49399999999999999,"15min":0.49399999999999999},"max":{"1min":1.6839999999999999,"5min":2.1850000000000001,"15min":2.1850000000000001},"last":0.97199999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.325,"5min":1.2629999999999999,"15min":1.292},"min":{"1min":0.90400000000000003,"5min":0.61299999999999999,"15min":0.61299999999999999},"max":{"1min":1.9419999999999999,"5min":2.3570000000000002,"15min":3.524},"last":1.4810000000000001},{"interface":"front","average":{"1min":1.3100000000000001,"5min":1.3420000000000001,"15min":1.4199999999999999},"min":{"1min":1.0129999999999999,"5min":0.55400000000000005,"15min":0.55400000000000005},"max":{"1min":1.7130000000000001,"5min":1.718,"15min":2.2440000000000002},"last":1.5900000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.3140000000000001,"5min":1.288,"15min":1.3340000000000001},"min":{"1min":0.70299999999999996,"5min":0.64600000000000002,"15min":0.64600000000000002},"max":{"1min":1.7889999999999999,"5min":1.7949999999999999,"15min":2.5489999999999999},"last":1.1020000000000001},{"interface":"front","average":{"1min":1.4099999999999999,"5min":1.4059999999999999,"15min":1.4330000000000001},"min":{"1min":0.82899999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"max":{"1min":2.2410000000000001,"5min":2.5369999999999999,"15min":4.1600000000000001},"last":1.633}]},{"osd":5,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.357,"5min":1.369,"15min":1.3939999999999999},"min":{"1min":0.90800000000000003,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":1.9139999999999999,"5min":1.923,"15min":2.02},"last":1.4330000000000001},{"interface":"front","average":{"1min":1.4379999999999999,"5min":1.3999999999999999,"15min":1.395},"min":{"1min":0.89800000000000002,"5min":0.373,"15min":0.373},"max":{"1min":2.0230000000000001,"5min":2.407,"15min":2.407},"last":1.2090000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5,"5min":1.425,"15min":1.4470000000000001},"min":{"1min":0.88400000000000001,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":2.3079999999999998,"5min":2.4500000000000002,"15min":3.3610000000000002},"last":1.746},{"interface":"front","average":{"1min":1.544,"5min":1.544,"15min":1.577},"min":{"1min":1.276,"5min":0.754,"15min":0.754},"max":{"1min":1.964,"5min":2.1379999999999999,"15min":2.3929999999999998},"last":1.532}]},{"osd":7,"last update":"Sat Oct 14 18:20:18 2023","interfaces":[{"interface":"back","average":{"1min":1.5369999999999999,"5min":1.528,"15min":1.5309999999999999},"min":{"1min":0.84399999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"max":{"1min":1.901,"5min":2.0680000000000001,"15min":2.7440000000000002},"last":1.3620000000000001},{"interface":"front","average":{"1min":1.696,"5min":1.605,"15min":1.6140000000000001},"min":{"1min":0.92000000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"max":{"1min":2.9449999999999998,"5min":2.9449999999999998,"15min":3.8610000000000002},"last":1.8149999999999999}]}]},{"osd":3,"up_from":13,"seq":55834576040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54655432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.94099999999999995,"15min":0.97599999999999998},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":1.7949999999999999,"5min":1.7949999999999999,"15min":1.7949999999999999},"last":1.1679999999999999},{"interface":"front","average":{"1min":0.96599999999999997,"5min":1.0660000000000001,"15min":1.1619999999999999},"min":{"1min":0.54500000000000004,"5min":0.47899999999999998,"15min":0.47899999999999998},"max":{"1min":1.327,"5min":1.665,"15min":1.9970000000000001},"last":1.1259999999999999}]},{"osd":1,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.93000000000000005,"15min":0.98999999999999999},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":1.2849999999999999,"5min":1.782,"15min":1.919},"last":0.79700000000000004},{"interface":"front","average":{"1min":0.98199999999999998,"5min":1.083,"15min":1.161},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.536,"5min":1.742,"15min":1.8560000000000001},"last":0.996}]},{"osd":2,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2210000000000001,"5min":1.27,"15min":1.3839999999999999},"min":{"1min":0.59199999999999997,"5min":0.50900000000000001,"15min":0.498},"max":{"1min":1.833,"5min":1.9970000000000001,"15min":2.032},"last":1.206},{"interface":"front","average":{"1min":1.149,"5min":1.2370000000000001,"15min":1.3340000000000001},"min":{"1min":0.55500000000000005,"5min":0.48999999999999999,"15min":0.48999999999999999},"max":{"1min":1.792,"5min":1.9330000000000001,"15min":2.3540000000000001},"last":1.236}]},{"osd":4,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.3520000000000001,"5min":1.3700000000000001,"15min":1.421},"min":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"max":{"1min":2.129,"5min":2.129,"15min":2.153},"last":1.421},{"interface":"front","average":{"1min":1.3660000000000001,"5min":1.429,"15min":1.4650000000000001},"min":{"1min":0.495,"5min":0.495,"15min":0.495},"max":{"1min":1.915,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":1.381}]},{"osd":5,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.2490000000000001,"5min":1.3819999999999999,"15min":1.4359999999999999},"min":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"max":{"1min":1.6180000000000001,"5min":3.3170000000000002,"15min":3.3170000000000002},"last":1.581},{"interface":"front","average":{"1min":1.452,"5min":1.4670000000000001,"15min":1.534},"min":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"max":{"1min":2.7029999999999998,"5min":2.7029999999999998,"15min":2.891},"last":1.2909999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.5349999999999999,"5min":1.5649999999999999,"15min":1.611},"min":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.61099999999999999},"max":{"1min":2.29,"5min":2.3319999999999999,"15min":2.4889999999999999},"last":1.6160000000000001},{"interface":"front","average":{"1min":1.581,"5min":1.627,"15min":1.6639999999999999},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"max":{"1min":2.4119999999999999,"5min":2.6840000000000002,"15min":2.6840000000000002},"last":1.528}]},{"osd":7,"last update":"Sat Oct 14 18:20:15 2023","interfaces":[{"interface":"back","average":{"1min":1.4139999999999999,"5min":1.534,"15min":1.5880000000000001},"min":{"1min":0.85699999999999998,"5min":0.85499999999999998,"15min":0.77900000000000003},"max":{"1min":2.173,"5min":3.153,"15min":3.153},"last":1.456},{"interface":"front","average":{"1min":1.542,"5min":1.617,"15min":1.6679999999999999},"min":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"max":{"1min":2.589,"5min":2.7829999999999999,"15min":2.7829999999999999},"last":1.353}]}]},{"osd":2,"up_from":13,"seq":55834576040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56800,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315040,"statfs":{"total":96636764160,"available":96578600960,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":0.94999999999999996,"5min":0.96899999999999997,"15min":0.94399999999999995},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.7809999999999999,"5min":2.1699999999999999,"15min":2.1699999999999999},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94799999999999995,"15min":1.044},"min":{"1min":0.55000000000000004,"5min":0.439,"15min":0.438},"max":{"1min":1.343,"5min":1.448,"15min":1.8660000000000001},"last":0.78500000000000003}]},{"osd":1,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":0.97799999999999998,"5min":0.95599999999999996,"15min":0.996},"min":{"1min":0.44600000000000001,"5min":0.44,"15min":0.39400000000000002},"max":{"1min":1.6719999999999999,"5min":1.6719999999999999,"15min":1.8009999999999999},"last":0.90300000000000002},{"interface":"front","average":{"1min":1.103,"5min":1.099,"15min":1.1890000000000001},"min":{"1min":0.48799999999999999,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":1.7729999999999999,"5min":2.3109999999999999,"15min":2.3109999999999999},"last":1.2150000000000001}]},{"osd":3,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.123,"5min":1.1739999999999999,"15min":1.252},"min":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"max":{"1min":1.74,"5min":1.74,"15min":1.9690000000000001},"last":1.4410000000000001},{"interface":"front","average":{"1min":1.204,"5min":1.27,"15min":1.3420000000000001},"min":{"1min":0.61799999999999999,"5min":0.52400000000000002,"15min":0.52400000000000002},"max":{"1min":1.7090000000000001,"5min":2.0089999999999999,"15min":2.165},"last":0.96799999999999997}]},{"osd":4,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.3759999999999999,"5min":1.4159999999999999,"15min":1.454},"min":{"1min":0.72599999999999998,"5min":0.58199999999999996,"15min":0.58199999999999996},"max":{"1min":1.9810000000000001,"5min":2.1259999999999999,"15min":2.1259999999999999},"last":1.365},{"interface":"front","average":{"1min":1.331,"5min":1.2989999999999999,"15min":1.3460000000000001},"min":{"1min":0.74099999999999999,"5min":0.72399999999999998,"15min":0.66500000000000004},"max":{"1min":1.9790000000000001,"5min":2.2490000000000001,"15min":2.2490000000000001},"last":1.512}]},{"osd":5,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.343,"5min":1.389,"15min":1.4299999999999999},"min":{"1min":0.78400000000000003,"5min":0.72099999999999997,"15min":0.53900000000000003},"max":{"1min":1.9319999999999999,"5min":2.165,"15min":2.2839999999999998},"last":1.3100000000000001},{"interface":"front","average":{"1min":1.5109999999999999,"5min":1.5289999999999999,"15min":1.571},"min":{"1min":0.85499999999999998,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":2.1150000000000002,"5min":2.2389999999999999,"15min":2.2389999999999999},"last":1.5740000000000001}]},{"osd":6,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.474,"5min":1.4790000000000001,"15min":1.5149999999999999},"min":{"1min":0.82699999999999996,"5min":0.80600000000000005,"15min":0.747},"max":{"1min":1.883,"5min":2.2690000000000001,"15min":2.2690000000000001},"last":1.26},{"interface":"front","average":{"1min":1.5149999999999999,"5min":1.542,"15min":1.591},"min":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.64500000000000002},"max":{"1min":1.925,"5min":2.2349999999999999,"15min":2.2349999999999999},"last":1.6439999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:20:32 2023","interfaces":[{"interface":"back","average":{"1min":1.5049999999999999,"5min":1.522,"15min":1.575},"min":{"1min":0.621,"5min":0.621,"15min":0.621},"max":{"1min":2.1190000000000002,"5min":2.323,"15min":2.323},"last":1.7050000000000001},{"interface":"front","average":{"1min":1.5920000000000001,"5min":1.613,"15min":1.669},"min":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"max":{"1min":2.0590000000000002,"5min":2.331,"15min":2.331},"last":1.7769999999999999}]}]},{"osd":1,"up_from":13,"seq":55834576040,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":56796,"kb_used_data":3408,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":94315044,"statfs":{"total":96636764160,"available":96578605056,"internally_reserved":0,"allocated":3489792,"data_stored":1747099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54655430},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.94299999999999995,"15min":0.96799999999999997},"min":{"1min":0.45500000000000002,"5min":0.41999999999999998,"15min":0.38},"max":{"1min":1.4570000000000001,"5min":1.5169999999999999,"15min":2.4929999999999999},"last":0.92700000000000005},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.97299999999999998,"15min":1.0920000000000001},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":1.3600000000000001,"5min":2.847,"15min":2.847},"last":1.155}]},{"osd":2,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.0349999999999999,"5min":0.996,"15min":1.042},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.492,"5min":2.1200000000000001,"15min":2.7360000000000002},"last":1.0549999999999999},{"interface":"front","average":{"1min":1.0960000000000001,"5min":1.1659999999999999,"15min":1.2450000000000001},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":1.6919999999999999,"5min":1.996,"15min":2.6859999999999999},"last":1.204}]},{"osd":3,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.1759999999999999,"5min":1.2350000000000001,"15min":1.3540000000000001},"min":{"1min":0.60199999999999998,"5min":0.52700000000000002,"15min":0.44700000000000001},"max":{"1min":1.879,"5min":2.9940000000000002,"15min":2.9940000000000002},"last":1.4059999999999999},{"interface":"front","average":{"1min":1.1259999999999999,"5min":1.1970000000000001,"15min":1.2869999999999999},"min":{"1min":0.54400000000000004,"5min":0.46500000000000002,"15min":0.46500000000000002},"max":{"1min":1.5649999999999999,"5min":4.8630000000000004,"15min":4.8630000000000004},"last":1.369}]},{"osd":4,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.395,"5min":1.4119999999999999,"15min":1.454},"min":{"1min":0.748,"5min":0.748,"15min":0.73899999999999999},"max":{"1min":2.0329999999999999,"5min":4.8090000000000002,"15min":4.8090000000000002},"last":1.4450000000000001},{"interface":"front","average":{"1min":1.397,"5min":1.3680000000000001,"15min":1.4259999999999999},"min":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.64800000000000002},"max":{"1min":2.2010000000000001,"5min":2.218,"15min":2.899},"last":1.262}]},{"osd":5,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.389,"5min":1.405,"15min":1.4630000000000001},"min":{"1min":0.82399999999999995,"5min":0.80100000000000005,"15min":0.80100000000000005},"max":{"1min":1.857,"5min":2.2519999999999998,"15min":3.004},"last":1.4770000000000001},{"interface":"front","average":{"1min":1.5680000000000001,"5min":1.502,"15min":1.5569999999999999},"min":{"1min":0.94999999999999996,"5min":0.90400000000000003,"15min":0.73199999999999998},"max":{"1min":2.1080000000000001,"5min":2.1080000000000001,"15min":2.5129999999999999},"last":1.9219999999999999}]},{"osd":6,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.508,"5min":1.5069999999999999,"15min":1.597},"min":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"max":{"1min":2.367,"5min":2.8999999999999999,"15min":2.931},"last":1.589},{"interface":"front","average":{"1min":1.5860000000000001,"5min":1.615,"15min":1.6220000000000001},"min":{"1min":0.90200000000000002,"5min":0.871,"15min":0.871},"max":{"1min":2.2509999999999999,"5min":10.173999999999999,"15min":10.173999999999999},"last":1.5609999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:19:44 2023","interfaces":[{"interface":"back","average":{"1min":1.585,"5min":1.6060000000000001,"15min":1.6830000000000001},"min":{"1min":0.83799999999999997,"5min":0.82299999999999995,"15min":0.57899999999999996},"max":{"1min":1.9830000000000001,"5min":3.363,"15min":3.363},"last":2.0720000000000001},{"interface":"front","average":{"1min":1.55,"5min":1.6259999999999999,"15min":1.6990000000000001},"min":{"1min":0.84999999999999998,"5min":0.84999999999999998,"15min":0.84999999999999998},"max":{"1min":2.1629999999999998,"5min":7.0439999999999996,"15min":7.0439999999999996},"last":2.028}]}]},{"osd":0,"up_from":13,"seq":55834576039,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":53212,"kb_used_data":3988,"kb_used_omap":1,"kb_used_meta":49214,"kb_avail":94318628,"statfs":{"total":96636764160,"available":96582275072,"internally_reserved":0,"allocated":4083712,"data_stored":2337467,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50395590},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.90900000000000003,"15min":0.96199999999999997},"min":{"1min":0.42199999999999999,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.728,"5min":1.728,"15min":2.6440000000000001},"last":1.1779999999999999},{"interface":"front","average":{"1min":1.028,"5min":1.0229999999999999,"15min":1.113},"min":{"1min":0.50800000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":1.954,"5min":2.0529999999999999,"15min":2.2480000000000002},"last":0.85099999999999998}]},{"osd":2,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.081,"5min":1.0740000000000001,"15min":1.1619999999999999},"min":{"1min":0.59899999999999998,"5min":0.42799999999999999,"15min":0.42799999999999999},"max":{"1min":2.1509999999999998,"5min":2.1509999999999998,"15min":2.1509999999999998},"last":1.05},{"interface":"front","average":{"1min":1.1020000000000001,"5min":1.1200000000000001,"15min":1.2210000000000001},"min":{"1min":0.51900000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":1.792,"5min":1.845,"15min":2.5779999999999998},"last":1.635}]},{"osd":3,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2110000000000001,"5min":1.228,"15min":1.339},"min":{"1min":0.76300000000000001,"5min":0.501,"15min":0.501},"max":{"1min":1.929,"5min":2.2850000000000001,"15min":2.2850000000000001},"last":1.5780000000000001},{"interface":"front","average":{"1min":1.266,"5min":1.276,"15min":1.3720000000000001},"min":{"1min":0.63100000000000001,"5min":0.52900000000000003,"15min":0.52900000000000003},"max":{"1min":2.0880000000000001,"5min":2.0880000000000001,"15min":2.194},"last":1.768}]},{"osd":4,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.2589999999999999,"5min":1.248,"15min":1.306},"min":{"1min":0.66100000000000003,"5min":0.51600000000000001,"15min":0.51600000000000001},"max":{"1min":1.944,"5min":1.944,"15min":2.4670000000000001},"last":1.258},{"interface":"front","average":{"1min":1.4570000000000001,"5min":1.429,"15min":1.4910000000000001},"min":{"1min":1.0980000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"max":{"1min":2.0179999999999998,"5min":2.121,"15min":2.4889999999999999},"last":1.7230000000000001}]},{"osd":5,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.248,"5min":1.2689999999999999,"15min":1.3560000000000001},"min":{"1min":0.877,"5min":0.626,"15min":0.626},"max":{"1min":1.8400000000000001,"5min":1.9359999999999999,"15min":2.1339999999999999},"last":1.1120000000000001},{"interface":"front","average":{"1min":1.4810000000000001,"5min":1.4690000000000001,"15min":1.552},"min":{"1min":0.78200000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"max":{"1min":2.0489999999999999,"5min":2.3839999999999999,"15min":2.3839999999999999},"last":1.492}]},{"osd":6,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.5169999999999999,"5min":1.5469999999999999,"15min":1.6299999999999999},"min":{"1min":0.91600000000000004,"5min":0.85899999999999999,"15min":0.85899999999999999},"max":{"1min":2.238,"5min":2.3140000000000001,"15min":2.3969999999999998},"last":1.833},{"interface":"front","average":{"1min":1.4039999999999999,"5min":1.4259999999999999,"15min":1.4910000000000001},"min":{"1min":0.85999999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"max":{"1min":2.0139999999999998,"5min":2.1629999999999998,"15min":2.2799999999999998},"last":1.5449999999999999}]},{"osd":7,"last update":"Sat Oct 14 18:20:25 2023","interfaces":[{"interface":"back","average":{"1min":1.506,"5min":1.5369999999999999,"15min":1.6279999999999999},"min":{"1min":0.97999999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"max":{"1min":2.3039999999999998,"5min":2.3039999999999998,"15min":2.4630000000000001},"last":1.3140000000000001},{"interface":"front","average":{"1min":1.4990000000000001,"5min":1.504,"15min":1.603},"min":{"1min":0.93600000000000005,"5min":0.75,"15min":0.75},"max":{"1min":2.2240000000000002,"5min":2.3250000000000002,"15min":2.3620000000000001},"last":1.359}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-14T18:20:42.630 DEBUG:teuthology.orchestra.run.smithi029:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-10-14T18:20:42.851 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.851+0000 7f178b590640 1 Processor -- start 2023-10-14T18:20:42.852 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.851+0000 7f178b590640 1 -- start start 2023-10-14T18:20:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.852+0000 7f178b590640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f178406a4b0 0x7f17840ca3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.853 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.852+0000 7f178b590640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f17840ca8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.852+0000 7f178b590640 1 -- --> v1:172.21.15.29:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17840cb030 con 0x7f178415ad70 2023-10-14T18:20:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.852+0000 7f178b590640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17840cae30 con 0x7f178406a4b0 2023-10-14T18:20:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.852+0000 7f178b590640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17840c91c0 con 0x7f17840c9ca0 2023-10-14T18:20:42.854 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f178932c640 1 --1- >> v1:172.21.15.29:6789/0 conn(0x7f178415ad70 0x7f1784069da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.29:6789/0 says I am v1:172.21.15.29:45832/0 (socket says 172.21.15.29:45832) 2023-10-14T18:20:42.855 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f178932c640 1 -- 172.21.15.29:0/3658967588 learned_addr learned my addr 172.21.15.29:0/3658967588 (peer_addr_for_me v1:172.21.15.29:0/0) 2023-10-14T18:20:42.856 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f1788b2b640 1 --2- 172.21.15.29:0/3658967588 >> v2:172.21.15.186:3300/0 conn(0x7f178406a4b0 0x7f17840ca3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f17840ca8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v1:172.21.15.29:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377195034 0 0) 0x7f17840cb030 con 0x7f178415ad70 2023-10-14T18:20:42.857 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.853+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 --> v1:172.21.15.29:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1774003520 con 0x7f178415ad70 2023-10-14T18:20:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v1:172.21.15.29:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1552533155 0 0) 0x7f1774003520 con 0x7f178415ad70 2023-10-14T18:20:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 msgr2=0x7f17840ca8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f17840ca8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.858 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 >> v2:172.21.15.186:3300/0 conn(0x7f178406a4b0 msgr2=0x7f17840ca3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 --2- 172.21.15.29:0/3658967588 >> v2:172.21.15.186:3300/0 conn(0x7f178406a4b0 0x7f17840ca3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 --> v1:172.21.15.29:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17840c94a0 con 0x7f178415ad70 2023-10-14T18:20:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v1:172.21.15.29:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3722149650 0 0) 0x7f17780019f0 con 0x7f178415ad70 2023-10-14T18:20:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 >> v1:172.21.15.29:6789/0 conn(0x7f178415ad70 legacy=0x7f1784069da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.859 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f1774003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1774003f40 0x7f17740047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f1774005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840cae30 con 0x7f1774004d00 2023-10-14T18:20:42.860 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840c91c0 con 0x7f1774003f40 2023-10-14T18:20:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f177400b980 con 0x7f17840c9ca0 2023-10-14T18:20:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f1774005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.861 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f1788b2b640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1774003f40 0x7f17740047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.854+0000 7f178932c640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f1774003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.855+0000 7f1789b2d640 1 -- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 msgr2=0x7f1774003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.855+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c9ca0 0x7f1774003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.862 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.855+0000 7f1789b2d640 1 -- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1774003f40 msgr2=0x7f17740047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.855+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f1774003f40 0x7f17740047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.855+0000 7f1789b2d640 1 -- 172.21.15.29:0/3658967588 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1774003520 con 0x7f1774004d00 2023-10-14T18:20:42.863 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.856+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f1774005180 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f177c009af0 tx=0x7f17740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648518bcaa234b57 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.856+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c02c6b0 con 0x7f1774004d00 2023-10-14T18:20:42.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.856+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f177c02c810 con 0x7f1774004d00 2023-10-14T18:20:42.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.856+0000 7f1783fff640 1 -- 172.21.15.29:0/3658967588 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177c02cba0 con 0x7f1774004d00 2023-10-14T18:20:42.867 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 -- 172.21.15.29:0/3658967588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 msgr2=0x7f1774005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 --2- 172.21.15.29:0/3658967588 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f1774005180 secure :-1 s=READY pgs=3330 cs=0 l=1 rev1=1 crypto rx=0x7f177c009af0 tx=0x7f17740057f0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f1789b2d640 1 -- 172.21.15.29:0/3658967588 reap_dead start 2023-10-14T18:20:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 -- 172.21.15.29:0/3658967588 shutdown_connections 2023-10-14T18:20:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 -- 172.21.15.29:0/3658967588 >> 172.21.15.29:0/3658967588 conn(0x7f17840ce9b0 msgr2=0x7f17840cedc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:42.868 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 -- 172.21.15.29:0/3658967588 shutdown_connections 2023-10-14T18:20:42.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.857+0000 7f178b590640 1 -- 172.21.15.29:0/3658967588 wait complete. 2023-10-14T18:20:42.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.859+0000 7f178b590640 1 Processor -- start 2023-10-14T18:20:42.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.859+0000 7f178b590640 1 -- start start 2023-10-14T18:20:42.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.869 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c8220 0x7f17840cd7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17840c86a0 0x7f17840c5a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840cc720 con 0x7f1774004d00 2023-10-14T18:20:42.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17840cca00 con 0x7f17840c86a0 2023-10-14T18:20:42.870 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178b590640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17840ccce0 con 0x7f17840c8220 2023-10-14T18:20:42.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.860+0000 7f178932c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59490/0 (socket says 172.21.15.29:59490) 2023-10-14T18:20:42.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 -- 172.21.15.29:0/3538880430 learned_addr learned my addr 172.21.15.29:0/3538880430 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:42.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f1788b2b640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c8220 0x7f17840cd7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.871 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f1789b2d640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17840c86a0 0x7f17840c5a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 -- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c8220 msgr2=0x7f17840cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c8220 0x7f17840cd7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 -- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17840c86a0 msgr2=0x7f17840c5a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17840c86a0 0x7f17840c5a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:42.872 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 -- 172.21.15.29:0/3538880430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17840ccfc0 con 0x7f1774004d00 2023-10-14T18:20:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.861+0000 7f178932c640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f1778007b00 tx=0x7f177800e530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.862+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f177800ee90 con 0x7f1774004d00 2023-10-14T18:20:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.862+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1778013420 con 0x7f1774004d00 2023-10-14T18:20:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.862+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17780137d0 con 0x7f1774004d00 2023-10-14T18:20:42.873 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.862+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17840cd2a0 con 0x7f1774004d00 2023-10-14T18:20:42.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.862+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17840c7ea0 con 0x7f1774004d00 2023-10-14T18:20:42.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.863+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17840cd490 con 0x7f1774004d00 2023-10-14T18:20:42.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.864+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f1778013930 con 0x7f1774004d00 2023-10-14T18:20:42.874 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.865+0000 7f1781ffb640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 0x7f176806d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.865+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f177808a200 con 0x7f1774004d00 2023-10-14T18:20:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.865+0000 7f1788b2b640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 0x7f176806d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.866+0000 7f1788b2b640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 0x7f176806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17840c7860 tx=0x7f1770008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:42.875 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:42.870+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f1778055050 con 0x7f1774004d00 2023-10-14T18:20:43.069 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.068+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7f178404e160 con 0x7f1774004d00 2023-10-14T18:20:43.076 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.075+0000 7f1781ffb640 1 -- 172.21.15.29:0/3538880430 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v94)=0 v94) v1 ==== 123+0+0 (secure 0 0 0) 0x7f1778058670 con 0x7f1774004d00 2023-10-14T18:20:43.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.078+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 msgr2=0x7f176806d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.080 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.078+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 0x7f176806d450 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f17840c7860 tx=0x7f1770008040 comp rx=0 tx=0).stop 2023-10-14T18:20:43.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.078+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 msgr2=0x7f178414a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.078+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 secure :-1 s=READY pgs=3331 cs=0 l=1 rev1=1 crypto rx=0x7f1778007b00 tx=0x7f177800e530 comp rx=0 tx=0).stop 2023-10-14T18:20:43.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 shutdown_connections 2023-10-14T18:20:43.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f17840c86a0 0x7f17840c5a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.081 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f176806af90 0x7f176806d450 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f17840c8220 0x7f17840cd7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 --2- 172.21.15.29:0/3538880430 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f1774004d00 0x7f178414a250 unknown :-1 s=CLOSED pgs=3331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 >> 172.21.15.29:0/3538880430 conn(0x7f17840ce9b0 msgr2=0x7f17840b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:43.082 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.079+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 shutdown_connections 2023-10-14T18:20:43.083 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.080+0000 7f178b590640 1 -- 172.21.15.29:0/3538880430 wait complete. 2023-10-14T18:20:43.099 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2023-10-14T18:20:43.099 DEBUG:teuthology.orchestra.run.smithi029:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2023-10-14T18:20:43.312 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.312+0000 7f826dc67640 1 Processor -- start 2023-10-14T18:20:43.313 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.312+0000 7f826dc67640 1 -- start start 2023-10-14T18:20:43.314 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 --2- >> v2:172.21.15.186:3300/0 conn(0x7f826815cbe0 0x7f82680d0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 0x7f82680b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82680d1090 0x7f82680b4ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 -- --> v2:172.21.15.29:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82680d15a0 con 0x7f82680d1090 2023-10-14T18:20:43.315 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 -- --> v2:172.21.15.186:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82680d1880 con 0x7f826815cbe0 2023-10-14T18:20:43.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.313+0000 7f826dc67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82680d1b60 con 0x7f826815cfc0 2023-10-14T18:20:43.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.314+0000 7f826d466640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82680d1090 0x7f82680b4ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.314+0000 7f826d466640 1 --2- >> v2:172.21.15.29:3300/0 conn(0x7f82680d1090 0x7f82680b4ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59500/0 (socket says 172.21.15.29:59500) 2023-10-14T18:20:43.316 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.314+0000 7f826d466640 1 -- 172.21.15.29:0/1637800515 learned_addr learned my addr 172.21.15.29:0/1637800515 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:43.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.314+0000 7f8267fff640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 0x7f82680b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.314+0000 7f826cc65640 1 --2- 172.21.15.29:0/1637800515 >> v2:172.21.15.186:3300/0 conn(0x7f826815cbe0 0x7f82680d0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.318 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f826d466640 1 -- 172.21.15.29:0/1637800515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 msgr2=0x7f82680b4570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:43.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f826d466640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 0x7f82680b4570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f826d466640 1 -- 172.21.15.29:0/1637800515 >> v2:172.21.15.186:3300/0 conn(0x7f826815cbe0 msgr2=0x7f82680d0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-10-14T18:20:43.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f826d466640 1 --2- 172.21.15.29:0/1637800515 >> v2:172.21.15.186:3300/0 conn(0x7f826815cbe0 0x7f82680d0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f826d466640 1 -- 172.21.15.29:0/1637800515 --> v2:172.21.15.29:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82680d1e40 con 0x7f82680d1090 2023-10-14T18:20:43.319 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.315+0000 7f8267fff640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 0x7f82680b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-10-14T18:20:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.316+0000 7f826d466640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82680d1090 0x7f82680b4ab0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f82680b53f0 tx=0x7f826002bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=faf38ae9aa7fb931 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.316+0000 7f82677fe640 1 -- 172.21.15.29:0/1637800515 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826002cb00 con 0x7f82680d1090 2023-10-14T18:20:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.316+0000 7f82677fe640 1 -- 172.21.15.29:0/1637800515 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f826002cc60 con 0x7f82680d1090 2023-10-14T18:20:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.316+0000 7f82677fe640 1 -- 172.21.15.29:0/1637800515 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82600343f0 con 0x7f82680d1090 2023-10-14T18:20:43.320 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 -- 172.21.15.29:0/1637800515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82680d1090 msgr2=0x7f82680b4ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82680d1090 0x7f82680b4ab0 secure :-1 s=READY pgs=3332 cs=0 l=1 rev1=1 crypto rx=0x7f82680b53f0 tx=0x7f826002bf60 comp rx=0 tx=0).stop 2023-10-14T18:20:43.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 -- 172.21.15.29:0/1637800515 shutdown_connections 2023-10-14T18:20:43.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f82680d1090 0x7f82680b4ab0 unknown :-1 s=CLOSED pgs=3332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 --2- 172.21.15.29:0/1637800515 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f826815cfc0 0x7f82680b4570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.321 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 --2- 172.21.15.29:0/1637800515 >> v2:172.21.15.186:3300/0 conn(0x7f826815cbe0 0x7f82680d0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 -- 172.21.15.29:0/1637800515 >> 172.21.15.29:0/1637800515 conn(0x7f82680cf570 msgr2=0x7f82680cf950 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 -- 172.21.15.29:0/1637800515 shutdown_connections 2023-10-14T18:20:43.322 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.317+0000 7f826dc67640 1 -- 172.21.15.29:0/1637800515 wait complete. 2023-10-14T18:20:43.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.319+0000 7f826dc67640 1 Processor -- start 2023-10-14T18:20:43.324 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.319+0000 7f826dc67640 1 -- start start 2023-10-14T18:20:43.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 --2- >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 0x7f82680b68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 --2- >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f826815cfc0 0x7f82680c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 -- --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82680b6e20 con 0x7f826815cbe0 2023-10-14T18:20:43.325 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 -- --> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82680c72b0 con 0x7f826815cfc0 2023-10-14T18:20:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.320+0000 7f826dc67640 1 -- --> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82680c6730 con 0x7f82680d1090 2023-10-14T18:20:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 --2- >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.29:3300/0 says I am v2:172.21.15.29:59508/0 (socket says 172.21.15.29:59508) 2023-10-14T18:20:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 -- 172.21.15.29:0/2730983160 learned_addr learned my addr 172.21.15.29:0/2730983160 (peer_addr_for_me v2:172.21.15.29:0/0) 2023-10-14T18:20:43.326 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f826cc65640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 0x7f82680b68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 -- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 msgr2=0x7f82680b68e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f826d466640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f826815cfc0 0x7f82680c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 0x7f82680b68e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 -- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f826815cfc0 msgr2=0x7f82680c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.327 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f826815cfc0 0x7f82680c9090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f8267fff640 1 -- 172.21.15.29:0/2730983160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260009b30 con 0x7f826815cbe0 2023-10-14T18:20:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.321+0000 7f826cc65640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 0x7f82680b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-10-14T18:20:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f8267fff640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f825400d7d0 tx=0x7f825400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8254004030 con 0x7f826815cbe0 2023-10-14T18:20:43.328 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 2 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7f8254004190 con 0x7f826815cbe0 2023-10-14T18:20:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82680c6a10 con 0x7f826815cbe0 2023-10-14T18:20:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f825400b6c0 con 0x7f826815cbe0 2023-10-14T18:20:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.322+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82680c6fd0 con 0x7f826815cbe0 2023-10-14T18:20:43.329 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.323+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826804e9b0 con 0x7f826815cbe0 2023-10-14T18:20:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.324+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 4 ==== mgrmap(e 164) v1 ==== 89331+0+0 (secure 0 0 0) 0x7f825400b820 con 0x7f826815cbe0 2023-10-14T18:20:43.330 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.325+0000 7f82657fa640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 0x7f824c06d450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-10-14T18:20:43.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.325+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 5 ==== osd_map(276..276 src has 1..276) v4 ==== 5876+0+0 (secure 0 0 0) 0x7f8254089a00 con 0x7f826815cbe0 2023-10-14T18:20:43.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.325+0000 7f826cc65640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 0x7f824c06d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-10-14T18:20:43.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.326+0000 7f826cc65640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 0x7f824c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f825c006fd0 tx=0x7f825c008040 comp rx=0 tx=0).ready entity=mgr.13113 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-10-14T18:20:43.331 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.329+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+180165 (secure 0 0 0) 0x7f8254054850 con 0x7f826815cbe0 2023-10-14T18:20:43.566 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.566+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 --> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82680b2690 con 0x7f826815cbe0 2023-10-14T18:20:43.568 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.567+0000 7f82657fa640 1 -- 172.21.15.29:0/2730983160 <== mon.0 v2:172.21.15.29:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+3711 (secure 0 0 0) 0x7f8254057e70 con 0x7f826815cbe0 2023-10-14T18:20:43.568 INFO:teuthology.orchestra.run.smithi029.stdout: 2023-10-14T18:20:43.568 INFO:teuthology.orchestra.run.smithi029.stdout:{"epoch":141,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":7955,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6837/3908610446","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6835","nonce":3908610446},{"type":"v1","addr":"172.21.15.29:6837","nonce":3908610446}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":7958,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6834/1897542773","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6832","nonce":1897542773},{"type":"v1","addr":"172.21.15.186:6834","nonce":1897542773}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134},{"gid":9490,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.186:6835/2235342280","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.186:6833","nonce":2235342280},{"type":"v1","addr":"172.21.15.186:6835","nonce":2235342280}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":141},{"gid":12834,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.29:6836/1047212662","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.29:6834","nonce":1047212662},{"type":"v1","addr":"172.21.15.29:6836","nonce":1047212662}]},"join_fscid":-1,"export_targets":[],"features":4540138322906710015,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":134}],"filesystems":[]} 2023-10-14T18:20:43.569 INFO:teuthology.orchestra.run.smithi029.stderr:dumped fsmap epoch 141 2023-10-14T18:20:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 msgr2=0x7f824c06d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 0x7f824c06d450 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f825c006fd0 tx=0x7f825c008040 comp rx=0 tx=0).stop 2023-10-14T18:20:43.571 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 msgr2=0x7f82680c8b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-10-14T18:20:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 secure :-1 s=READY pgs=3333 cs=0 l=1 rev1=1 crypto rx=0x7f825400d7d0 tx=0x7f825400dc90 comp rx=0 tx=0).stop 2023-10-14T18:20:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 shutdown_connections 2023-10-14T18:20:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3300/0,v1:172.21.15.186:6789/0] conn(0x7f826815cfc0 0x7f82680c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.572 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:3300/0,v1:172.21.15.29:6789/0] conn(0x7f826815cbe0 0x7f82680c8b50 unknown :-1 s=CLOSED pgs=3333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.29:6832/748820097,v1:172.21.15.29:6833/748820097] conn(0x7f824c06af90 0x7f824c06d450 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 --2- 172.21.15.29:0/2730983160 >> [v2:172.21.15.186:3301/0,v1:172.21.15.186:6790/0] conn(0x7f82680d1090 0x7f82680b68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-10-14T18:20:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 >> 172.21.15.29:0/2730983160 conn(0x7f82680cf570 msgr2=0x7f82680b7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-10-14T18:20:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 shutdown_connections 2023-10-14T18:20:43.573 INFO:teuthology.orchestra.run.smithi029.stderr:2023-10-14T18:20:43.570+0000 7f826dc67640 1 -- 172.21.15.29:0/2730983160 wait complete. 2023-10-14T18:20:43.591 DEBUG:tasks.cephfs.filesystem:already dead... 2023-10-14T18:20:43.591 INFO:teuthology.misc:Shutting down mds daemons... 2023-10-14T18:20:43.592 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2023-10-14T18:20:43.592 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.610 INFO:tasks.ceph.mds.a:Stopped 2023-10-14T18:20:43.611 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2023-10-14T18:20:43.611 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.681 INFO:tasks.ceph.mds.c:Stopped 2023-10-14T18:20:43.681 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2023-10-14T18:20:43.682 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.703 INFO:tasks.ceph.mds.b:Stopped 2023-10-14T18:20:43.704 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2023-10-14T18:20:43.704 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.768 INFO:tasks.ceph.mds.d:Stopped 2023-10-14T18:20:43.769 INFO:teuthology.misc:Shutting down osd daemons... 2023-10-14T18:20:43.769 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-10-14T18:20:43.769 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.837 INFO:tasks.ceph.osd.0:Stopped 2023-10-14T18:20:43.838 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-10-14T18:20:43.838 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:43.931 INFO:tasks.ceph.osd.1:Stopped 2023-10-14T18:20:43.931 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-10-14T18:20:43.932 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.008 INFO:tasks.ceph.osd.2:Stopped 2023-10-14T18:20:44.009 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-10-14T18:20:44.009 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.079 INFO:tasks.ceph.osd.3:Stopped 2023-10-14T18:20:44.080 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-10-14T18:20:44.080 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.138 INFO:tasks.ceph.osd.4:Stopped 2023-10-14T18:20:44.138 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-10-14T18:20:44.139 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.231 INFO:tasks.ceph.osd.5:Stopped 2023-10-14T18:20:44.232 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2023-10-14T18:20:44.232 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.306 INFO:tasks.ceph.osd.6:Stopped 2023-10-14T18:20:44.306 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2023-10-14T18:20:44.307 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.373 INFO:tasks.ceph.osd.7:Stopped 2023-10-14T18:20:44.373 INFO:teuthology.misc:Shutting down mgr daemons... 2023-10-14T18:20:44.374 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2023-10-14T18:20:44.374 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.465 INFO:tasks.ceph.mgr.y:Stopped 2023-10-14T18:20:44.466 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-10-14T18:20:44.466 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.566 INFO:tasks.ceph.mgr.x:Stopped 2023-10-14T18:20:44.567 INFO:teuthology.misc:Shutting down mon daemons... 2023-10-14T18:20:44.567 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-10-14T18:20:44.568 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.652 INFO:tasks.ceph.mon.a:Stopped 2023-10-14T18:20:44.653 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-10-14T18:20:44.653 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.719 INFO:tasks.ceph.mon.b:Stopped 2023-10-14T18:20:44.720 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-10-14T18:20:44.720 INFO:teuthology.orchestra.run:waiting for 300 2023-10-14T18:20:44.779 INFO:tasks.ceph.mon.c:Stopped 2023-10-14T18:20:44.779 INFO:tasks.ceph:Checking cluster log for badness... 2023-10-14T18:20:44.780 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-14T18:20:44.944 INFO:teuthology.orchestra.run.smithi029.stdout:2023-10-14T16:55:33.247598+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-10-14T18:20:44.945 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-10-14T18:20:44.945 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-14T18:20:45.037 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-14T18:20:45.125 DEBUG:teuthology.orchestra.run.smithi029:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-10-14T18:20:45.212 INFO:teuthology.orchestra.run.smithi029.stdout:2023-10-14T16:55:33.247598+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2023-10-14T18:20:45.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:20:45.214 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-10-14T18:20:45.481 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:20:45.481 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-10-14T18:20:45.667 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:20:45.667 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-10-14T18:20:45.832 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:20:45.832 DEBUG:teuthology.orchestra.run.smithi029:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-10-14T18:20:46.015 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T18:20:46.015 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-10-14T18:20:46.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T18:20:46.212 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-10-14T18:20:46.333 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T18:20:46.333 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2023-10-14T18:20:46.478 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T18:20:46.478 DEBUG:teuthology.orchestra.run.smithi186:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2023-10-14T18:20:46.636 INFO:tasks.ceph:Archiving mon data... 2023-10-14T18:20:46.638 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/data/mon.a.tgz 2023-10-14T18:20:46.639 DEBUG:teuthology.orchestra.run.smithi029:> mktemp 2023-10-14T18:20:46.702 INFO:teuthology.orchestra.run.smithi029.stdout:/tmp/tmp.alF3ZunlQA 2023-10-14T18:20:46.703 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.alF3ZunlQA 2023-10-14T18:20:47.469 DEBUG:teuthology.orchestra.run.smithi029:> sudo chmod 0666 /tmp/tmp.alF3ZunlQA 2023-10-14T18:20:47.630 DEBUG:teuthology.orchestra.remote:smithi029:/tmp/tmp.alF3ZunlQA is 4MB 2023-10-14T18:20:47.805 DEBUG:teuthology.orchestra.run.smithi029:> rm -fr /tmp/tmp.alF3ZunlQA 2023-10-14T18:20:47.873 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/data/mon.b.tgz 2023-10-14T18:20:47.873 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2023-10-14T18:20:47.935 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.NgBZnBJE8n 2023-10-14T18:20:47.936 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.NgBZnBJE8n 2023-10-14T18:20:48.115 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:48.125 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:48.840 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.NgBZnBJE8n 2023-10-14T18:20:48.993 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.NgBZnBJE8n is 4MB 2023-10-14T18:20:49.160 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.NgBZnBJE8n 2023-10-14T18:20:49.220 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/data/mon.c.tgz 2023-10-14T18:20:49.220 DEBUG:teuthology.orchestra.run.smithi186:> mktemp 2023-10-14T18:20:49.271 INFO:teuthology.orchestra.run.smithi186.stdout:/tmp/tmp.j1PIzhUrin 2023-10-14T18:20:49.271 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.j1PIzhUrin 2023-10-14T18:20:50.001 DEBUG:teuthology.orchestra.run.smithi186:> sudo chmod 0666 /tmp/tmp.j1PIzhUrin 2023-10-14T18:20:50.138 DEBUG:teuthology.orchestra.remote:smithi186:/tmp/tmp.j1PIzhUrin is 4MB 2023-10-14T18:20:50.302 DEBUG:teuthology.orchestra.run.smithi186:> rm -fr /tmp/tmp.j1PIzhUrin 2023-10-14T18:20:50.362 INFO:tasks.ceph:Cleaning ceph cluster... 2023-10-14T18:20:50.362 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T18:20:50.372 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-10-14T18:20:50.473 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-10-14T18:20:50.474 DEBUG:teuthology.orchestra.run.smithi029:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T18:20:50.483 DEBUG:teuthology.orchestra.run.smithi186:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-10-14T18:20:50.580 INFO:teuthology.orchestra.run.smithi029.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T18:20:50.589 INFO:teuthology.orchestra.run.smithi186.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-10-14T18:20:50.601 INFO:tasks.ceph:Archiving crash dumps... 2023-10-14T18:20:50.603 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi029/crash 2023-10-14T18:20:50.605 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T18:20:50.689 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/lib/ceph/crash to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi186/crash 2023-10-14T18:20:50.690 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-10-14T18:20:50.786 INFO:tasks.ceph:Shutting down logrotate 2023-10-14T18:20:50.787 DEBUG:teuthology.orchestra.run.smithi029:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:50.796 DEBUG:teuthology.orchestra.run.smithi186:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:51.017 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:51.118 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm /etc/logrotate.d/ceph-test.conf 2023-10-14T18:20:51.221 INFO:tasks.ceph:Compressing logs... 2023-10-14T18:20:51.222 DEBUG:teuthology.orchestra.run.smithi029:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T18:20:51.231 DEBUG:teuthology.orchestra.run.smithi186:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-14T18:20:51.323 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2023-10-14T18:20:51.325 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.1.92770.log 2023-10-14T18:20:51.326 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87615.log 2023-10-14T18:20:51.327 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.1.92770.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.92770.log.gz 2023-10-14T18:20:51.327 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T18:20:51.329 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87615.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2023-10-14T18:20:51.329 INFO:teuthology.orchestra.run.smithi186.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.87615.log.gz 2023-10-14T18:20:51.329 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152640.log 2023-10-14T18:20:51.330 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87535.log 2023-10-14T18:20:51.331 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87495.log 2023-10-14T18:20:51.331 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141083.log 2023-10-14T18:20:51.332 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141911.log 2023-10-14T18:20:51.333 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152640.log.gz 2023-10-14T18:20:51.333 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87535.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.87535.log.gz 2023-10-14T18:20:51.333 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2023-10-14T18:20:51.334 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141083.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91645.log 2023-10-14T18:20:51.335 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.141083.log.gz 2023-10-14T18:20:51.335 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154601.log 2023-10-14T18:20:51.335 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.141911.log.gz 2023-10-14T18:20:51.336 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91645.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148409.log 2023-10-14T18:20:51.337 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91645.log.gz 2023-10-14T18:20:51.337 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87375.log 2023-10-14T18:20:51.337 INFO:teuthology.orchestra.run.smithi186.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.87495.log.gz 2023-10-14T18:20:51.338 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.93161.log 2023-10-14T18:20:51.339 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87375.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87575.log 2023-10-14T18:20:51.339 INFO:teuthology.orchestra.run.smithi186.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.87375.log.gz 2023-10-14T18:20:51.340 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.3.93161.log: 0.0% -- replaced with /var/log/ceph/ceph-client.3.93161.log.gz 2023-10-14T18:20:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154601.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154601.log.gz 2023-10-14T18:20:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118246.log 2023-10-14T18:20:51.341 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148409.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.148409.log.gz 2023-10-14T18:20:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138812.log 2023-10-14T18:20:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118246.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107519.log 2023-10-14T18:20:51.342 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118246.log.gz 2023-10-14T18:20:51.342 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138812.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126870.log 2023-10-14T18:20:51.342 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.138812.log.gz 2023-10-14T18:20:51.345 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2023-10-14T18:20:51.345 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87575.log: 93.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T18:20:51.345 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2023-10-14T18:20:51.346 INFO:teuthology.orchestra.run.smithi186.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.87575.log.gz 2023-10-14T18:20:51.346 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2023-10-14T18:20:51.346 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87455.log 2023-10-14T18:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.107519.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107519.log.gz 2023-10-14T18:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97948.log 2023-10-14T18:20:51.347 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126870.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.126870.log.gz 2023-10-14T18:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158338.log 2023-10-14T18:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97948.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136999.log 2023-10-14T18:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97948.log.gz 2023-10-14T18:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158338.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92387.log 2023-10-14T18:20:51.348 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.158338.log.gz 2023-10-14T18:20:51.349 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136999.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134082.log 2023-10-14T18:20:51.349 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.136999.log.gz 2023-10-14T18:20:51.351 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87415.log 2023-10-14T18:20:51.351 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87455.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.87455.log.gz 2023-10-14T18:20:51.351 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87415.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.87415.log.gz 2023-10-14T18:20:51.351 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T18:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92387.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105336.log 2023-10-14T18:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.92387.log.gz 2023-10-14T18:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134082.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134082.log.gz 2023-10-14T18:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.94373.log 2023-10-14T18:20:51.353 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113514.log 2023-10-14T18:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105336.log.gz 2023-10-14T18:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.94373.log: gzip 0.0% -- replaced with /var/log/ceph/ceph-client.0.94373.log.gz -5 2023-10-14T18:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.113034.log 2023-10-14T18:20:51.354 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113514.log: 87.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.113514.log.gz 2023-10-14T18:20:51.355 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.106101.log 2023-10-14T18:20:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2023-10-14T18:20:51.357 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.92761.log 2023-10-14T18:20:51.358 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2023-10-14T18:20:51.358 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118665.log 2023-10-14T18:20:51.358 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113034.log.gz 2023-10-14T18:20:51.358 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99400.log 2023-10-14T18:20:51.359 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.106101.log.gz 2023-10-14T18:20:51.359 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118665.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2023-10-14T18:20:51.359 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118665.log.gz 2023-10-14T18:20:51.359 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99400.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99400.log.gz 2023-10-14T18:20:51.361 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143671.log 2023-10-14T18:20:51.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114473.log 2023-10-14T18:20:51.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121155.log 2023-10-14T18:20:51.361 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.143671.log.gz 2023-10-14T18:20:51.362 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114473.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114473.log.gz 2023-10-14T18:20:51.362 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92716.log 2023-10-14T18:20:51.366 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-client.1.92761.log: 0.0% -- replaced with /var/log/ceph/ceph-client.1.92761.log.gz 2023-10-14T18:20:51.366 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87335.log 2023-10-14T18:20:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135034.log 2023-10-14T18:20:51.367 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121155.log.gz 2023-10-14T18:20:51.367 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92716.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137936.log 2023-10-14T18:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92716.log.gz 2023-10-14T18:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120203.log 2023-10-14T18:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.135034.log.gz 2023-10-14T18:20:51.368 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137936.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.137936.log.gz 2023-10-14T18:20:51.369 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121111.log 2023-10-14T18:20:51.370 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.3.93157.log 2023-10-14T18:20:51.370 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.admin.87335.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.87335.log.gz 2023-10-14T18:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160826.log 2023-10-14T18:20:51.371 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120203.log.gz 2023-10-14T18:20:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111072.log 2023-10-14T18:20:51.372 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.121111.log.gz 2023-10-14T18:20:51.372 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160826.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123535.log 2023-10-14T18:20:51.372 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160826.log.gz 2023-10-14T18:20:51.374 INFO:teuthology.orchestra.run.smithi186.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2023-10-14T18:20:51.374 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-client.3.93157.log: 0.0% 94.1% -- replaced with /var/log/ceph/ceph-client.3.93157.log.gz 2023-10-14T18:20:51.374 INFO:teuthology.orchestra.run.smithi186.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T18:20:51.375 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111072.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111072.log.gz 2023-10-14T18:20:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106269.log 2023-10-14T18:20:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123535.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156210.log 2023-10-14T18:20:51.376 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123535.log.gz 2023-10-14T18:20:51.376 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138274.log 2023-10-14T18:20:51.376 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106269.log.gz 2023-10-14T18:20:51.377 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156210.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.156210.log.gz 2023-10-14T18:20:51.377 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120558.log 2023-10-14T18:20:51.378 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142290.log 2023-10-14T18:20:51.378 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138274.log.gz 2023-10-14T18:20:51.378 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118204.log 2023-10-14T18:20:51.379 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120558.log.gz 2023-10-14T18:20:51.379 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142290.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.142290.log.gz 2023-10-14T18:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135991.log 2023-10-14T18:20:51.380 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141303.log 2023-10-14T18:20:51.381 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118204.log.gz 2023-10-14T18:20:51.381 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135991.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123577.log 2023-10-14T18:20:51.382 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.135991.log.gz 2023-10-14T18:20:51.382 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141303.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.141303.log.gz 2023-10-14T18:20:51.383 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92372.log 2023-10-14T18:20:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123577.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.123577.log.gz 2023-10-14T18:20:51.384 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107094.log 2023-10-14T18:20:51.385 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.135759.log 2023-10-14T18:20:51.386 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.92372.log.gz 2023-10-14T18:20:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107094.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.107094.log.gz 2023-10-14T18:20:51.387 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105105.log 2023-10-14T18:20:51.388 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.135759.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.135759.log.gz 2023-10-14T18:20:51.388 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153531.log 2023-10-14T18:20:51.389 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93053.log 2023-10-14T18:20:51.389 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105105.log.gz 2023-10-14T18:20:51.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153531.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152766.log 2023-10-14T18:20:51.390 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153531.log.gz 2023-10-14T18:20:51.391 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93053.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143821.log 2023-10-14T18:20:51.391 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93053.log.gz 2023-10-14T18:20:51.392 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152766.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101269.log 2023-10-14T18:20:51.393 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152766.log.gz 2023-10-14T18:20:51.393 INFO:teuthology.orchestra.run.smithi186.stderr:/var/log/ceph/ceph-mon.c.log: 92.6% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2023-10-14T18:20:51.394 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154853.log 2023-10-14T18:20:51.394 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.143821.log.gz 2023-10-14T18:20:51.395 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133213.log 2023-10-14T18:20:51.395 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101269.log.gz 2023-10-14T18:20:51.396 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154853.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129638.log 2023-10-14T18:20:51.396 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.154853.log.gz 2023-10-14T18:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133213.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133213.log.gz 2023-10-14T18:20:51.397 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124554.log 2023-10-14T18:20:51.398 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147508.log 2023-10-14T18:20:51.399 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129638.log.gz 2023-10-14T18:20:51.400 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136621.log 2023-10-14T18:20:51.400 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124554.log.gz 2023-10-14T18:20:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147508.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.147508.log.gz 2023-10-14T18:20:51.401 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136265.log 2023-10-14T18:20:51.402 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136621.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136715.log 2023-10-14T18:20:51.402 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.136621.log.gz 2023-10-14T18:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133255.log 2023-10-14T18:20:51.403 INFO:teuthology.orchestra.run.smithi029.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.136265.log.gz 2023-10-14T18:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136715.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136715.log.gz 2023-10-14T18:20:51.405 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134862.log 2023-10-14T18:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146872.log 2023-10-14T18:20:51.406 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.133255.log.gz 2023-10-14T18:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134862.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154961.log 2023-10-14T18:20:51.407 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.134862.log.gz 2023-10-14T18:20:51.408 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146872.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.153994.log 2023-10-14T18:20:51.408 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.146872.log.gz 2023-10-14T18:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154961.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.154961.log.gz 2023-10-14T18:20:51.409 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117890.log 2023-10-14T18:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.153994.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.153994.log.gz 2023-10-14T18:20:51.410 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117502.log 2023-10-14T18:20:51.411 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117890.log: gzip 89.8% -- replaced with /var/log/ceph/ceph-client.admin.117890.log.gz -5 2023-10-14T18:20:51.411 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.134904.log 2023-10-14T18:20:51.412 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94986.log 2023-10-14T18:20:51.413 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117502.log.gz 2023-10-14T18:20:51.414 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137280.log 2023-10-14T18:20:51.414 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.134904.log.gz 2023-10-14T18:20:51.415 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98032.log 2023-10-14T18:20:51.415 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.94986.log.gz 2023-10-14T18:20:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137280.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137280.log.gz 2023-10-14T18:20:51.416 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98706.log 2023-10-14T18:20:51.418 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98032.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96109.log 2023-10-14T18:20:51.418 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98032.log.gz 2023-10-14T18:20:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98706.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.98706.log.gz 2023-10-14T18:20:51.419 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102153.log 2023-10-14T18:20:51.420 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96109.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.96109.log.gz 2023-10-14T18:20:51.421 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126044.log 2023-10-14T18:20:51.422 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95322.log 2023-10-14T18:20:51.422 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.102153.log.gz 2023-10-14T18:20:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126044.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126044.log.gz 2023-10-14T18:20:51.423 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117117.log 2023-10-14T18:20:51.424 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121460.log 2023-10-14T18:20:51.424 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.95322.log.gz 2023-10-14T18:20:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117117.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117117.log.gz 2023-10-14T18:20:51.425 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111585.log 2023-10-14T18:20:51.426 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120899.log 2023-10-14T18:20:51.426 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.121460.log.gz 2023-10-14T18:20:51.427 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111585.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.111585.log.gz 2023-10-14T18:20:51.427 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119131.log 2023-10-14T18:20:51.428 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120899.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151973.log 2023-10-14T18:20:51.429 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120899.log.gz 2023-10-14T18:20:51.429 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119131.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.119131.log.gz 2023-10-14T18:20:51.430 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118815.log 2023-10-14T18:20:51.431 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151973.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.112314.log 2023-10-14T18:20:51.431 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.151973.log.gz 2023-10-14T18:20:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118815.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.118815.log.gz 2023-10-14T18:20:51.432 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161078.log 2023-10-14T18:20:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.112314.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112314.log.gz 2023-10-14T18:20:51.433 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120143.log 2023-10-14T18:20:51.434 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161078.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.161078.log.gz 2023-10-14T18:20:51.434 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.125497.log 2023-10-14T18:20:51.435 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126302.log 2023-10-14T18:20:51.435 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120143.log.gz 2023-10-14T18:20:51.436 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.125497.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.125497.log.gz 2023-10-14T18:20:51.436 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122735.log 2023-10-14T18:20:51.437 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126302.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105651.log 2023-10-14T18:20:51.437 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.126302.log.gz 2023-10-14T18:20:51.438 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98622.log 2023-10-14T18:20:51.439 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.122735.log.gz 2023-10-14T18:20:51.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105651.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142914.log 2023-10-14T18:20:51.440 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105651.log.gz 2023-10-14T18:20:51.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98622.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98622.log.gz 2023-10-14T18:20:51.441 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160509.log 2023-10-14T18:20:51.442 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161318.log 2023-10-14T18:20:51.442 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.142914.log.gz 2023-10-14T18:20:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160509.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.160509.log.gz 2023-10-14T18:20:51.443 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118396.log 2023-10-14T18:20:51.444 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103527.log 2023-10-14T18:20:51.444 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.161318.log.gz 2023-10-14T18:20:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118396.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.118396.log.gz 2023-10-14T18:20:51.445 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149121.log 2023-10-14T18:20:51.446 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147322.log 2023-10-14T18:20:51.446 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.103527.log.gz 2023-10-14T18:20:51.447 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149121.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149121.log.gz 2023-10-14T18:20:51.447 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157320.log 2023-10-14T18:20:51.448 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152430.log 2023-10-14T18:20:51.449 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147322.log.gz 2023-10-14T18:20:51.449 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157320.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157320.log.gz 2023-10-14T18:20:51.450 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134377.log 2023-10-14T18:20:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152430.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.152430.log.gz 2023-10-14T18:20:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151931.log 2023-10-14T18:20:51.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134377.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.134377.log.gz 2023-10-14T18:20:51.452 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141767.log 2023-10-14T18:20:51.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153149.log 2023-10-14T18:20:51.453 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.151931.log.gz 2023-10-14T18:20:51.454 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141767.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.141767.log.gz 2023-10-14T18:20:51.454 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103001.log 2023-10-14T18:20:51.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153149.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119376.log 2023-10-14T18:20:51.455 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153149.log.gz 2023-10-14T18:20:51.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103001.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.103001.log.gz 2023-10-14T18:20:51.457 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.91476.log 2023-10-14T18:20:51.457 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119376.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.119376.log.gz 2023-10-14T18:20:51.458 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155629.log 2023-10-14T18:20:51.459 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91476.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.91476.log.gz 2023-10-14T18:20:51.459 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123943.log 2023-10-14T18:20:51.460 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155629.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120600.log 2023-10-14T18:20:51.460 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155629.log.gz 2023-10-14T18:20:51.461 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123943.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.123943.log.gz 2023-10-14T18:20:51.461 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148325.log 2023-10-14T18:20:51.462 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96465.log 2023-10-14T18:20:51.462 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120600.log.gz 2023-10-14T18:20:51.463 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148325.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148325.log.gz 2023-10-14T18:20:51.463 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149849.log 2023-10-14T18:20:51.464 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96465.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160593.log 2023-10-14T18:20:51.464 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.96465.log.gz 2023-10-14T18:20:51.465 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149849.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149849.log.gz 2023-10-14T18:20:51.466 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142801.log 2023-10-14T18:20:51.466 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160593.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.160593.log.gz 2023-10-14T18:20:51.467 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140922.log 2023-10-14T18:20:51.468 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95658.log 2023-10-14T18:20:51.468 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142801.log.gz 2023-10-14T18:20:51.469 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140922.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.140922.log.gz 2023-10-14T18:20:51.470 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160036.log 2023-10-14T18:20:51.471 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95658.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95658.log.gz 2023-10-14T18:20:51.471 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143713.log 2023-10-14T18:20:51.472 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160036.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101858.log 2023-10-14T18:20:51.472 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.160036.log.gz 2023-10-14T18:20:51.473 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143713.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.143713.log.gz 2023-10-14T18:20:51.473 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159294.log 2023-10-14T18:20:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101858.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101858.log.gz 2023-10-14T18:20:51.475 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96063.log 2023-10-14T18:20:51.476 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153447.log 2023-10-14T18:20:51.476 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.159294.log.gz 2023-10-14T18:20:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96063.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96063.log.gz 2023-10-14T18:20:51.477 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106968.log 2023-10-14T18:20:51.478 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153447.log: gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153447.log.gz -5 2023-10-14T18:20:51.478 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.150481.log 2023-10-14T18:20:51.479 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106968.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106968.log.gz 2023-10-14T18:20:51.480 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160120.log 2023-10-14T18:20:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150481.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.150481.log.gz 2023-10-14T18:20:51.481 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133094.log 2023-10-14T18:20:51.482 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160120.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.160120.log.gz 2023-10-14T18:20:51.482 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126104.log 2023-10-14T18:20:51.483 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133094.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133094.log.gz 2023-10-14T18:20:51.483 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139230.log 2023-10-14T18:20:51.484 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126104.log: gzip -5 --verbose 87.1% -- /var/log/ceph/ceph-client.admin.105147.log -- replaced with /var/log/ceph/ceph-client.admin.126104.log.gz 2023-10-14T18:20:51.485 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139230.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.139230.log.gz 2023-10-14T18:20:51.486 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137635.log 2023-10-14T18:20:51.487 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98916.log 2023-10-14T18:20:51.487 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105147.log.gz 2023-10-14T18:20:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137635.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137635.log.gz 2023-10-14T18:20:51.488 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128782.log 2023-10-14T18:20:51.490 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98916.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141356.log 2023-10-14T18:20:51.490 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.98916.log.gz 2023-10-14T18:20:51.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128782.log: gzip -5 --verbose -- 87.9% /var/log/ceph/ceph-client.admin.159573.log 2023-10-14T18:20:51.491 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128782.log.gz 2023-10-14T18:20:51.492 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141356.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141356.log.gz 2023-10-14T18:20:51.492 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119596.log 2023-10-14T18:20:51.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159573.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.159573.log.gz 2023-10-14T18:20:51.494 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91953.log 2023-10-14T18:20:51.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152682.log 2023-10-14T18:20:51.495 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119596.log.gz 2023-10-14T18:20:51.496 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91953.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.91953.log.gz 2023-10-14T18:20:51.496 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160868.log 2023-10-14T18:20:51.497 INFO:teuthology.orchestra.run.smithi186.stderr: 93.1% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2023-10-14T18:20:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152682.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152682.log.gz 2023-10-14T18:20:51.498 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143779.log 2023-10-14T18:20:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160868.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160868.log.gz 2023-10-14T18:20:51.499 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157194.log 2023-10-14T18:20:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143779.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.143779.log.gz 2023-10-14T18:20:51.500 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143252.log 2023-10-14T18:20:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157194.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.157194.log.gz 2023-10-14T18:20:51.501 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110508.log 2023-10-14T18:20:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143252.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.143252.log.gz 2023-10-14T18:20:51.502 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124430.log 2023-10-14T18:20:51.503 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110508.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110508.log.gz 2023-10-14T18:20:51.504 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152159.log 2023-10-14T18:20:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124430.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124430.log.gz 2023-10-14T18:20:51.505 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111247.log 2023-10-14T18:20:51.506 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152159.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.152159.log.gz -5 2023-10-14T18:20:51.506 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.101185.log 2023-10-14T18:20:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111247.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111247.log.gz 2023-10-14T18:20:51.507 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160994.log 2023-10-14T18:20:51.508 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101185.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.101185.log.gz 2023-10-14T18:20:51.509 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92386.log 2023-10-14T18:20:51.510 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160994.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.160994.log.gz 2023-10-14T18:20:51.510 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99930.log 2023-10-14T18:20:51.511 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92386.log: gzip 87.5% -- replaced with /var/log/ceph/ceph-client.admin.92386.log.gz 2023-10-14T18:20:51.511 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.112310.log 2023-10-14T18:20:51.512 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99930.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99930.log.gz 2023-10-14T18:20:51.512 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151762.log 2023-10-14T18:20:51.513 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.112310.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.112310.log.gz 2023-10-14T18:20:51.514 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100299.log 2023-10-14T18:20:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151762.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151762.log.gz 2023-10-14T18:20:51.515 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151450.log 2023-10-14T18:20:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100299.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100299.log.gz 2023-10-14T18:20:51.516 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135572.log 2023-10-14T18:20:51.517 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151450.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151450.log.gz 2023-10-14T18:20:51.518 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99476.log 2023-10-14T18:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135572.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135572.log.gz 2023-10-14T18:20:51.519 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116991.log 2023-10-14T18:20:51.520 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99476.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.99476.log.gz 2023-10-14T18:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128698.log 2023-10-14T18:20:51.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116991.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116991.log.gz 2023-10-14T18:20:51.522 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134419.log 2023-10-14T18:20:51.523 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128698.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.128698.log.gz 2023-10-14T18:20:51.524 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137856.log 2023-10-14T18:20:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134419.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.134419.log.gz 2023-10-14T18:20:51.525 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115916.log 2023-10-14T18:20:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137856.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.137856.log.gz 2023-10-14T18:20:51.526 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95448.log 2023-10-14T18:20:51.527 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115916.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.115916.log.gz 2023-10-14T18:20:51.528 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110550.log 2023-10-14T18:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95448.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.95448.log.gz 2023-10-14T18:20:51.529 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141209.log 2023-10-14T18:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110550.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116394.log 2023-10-14T18:20:51.530 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110550.log.gz 2023-10-14T18:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141209.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.141209.log.gz 2023-10-14T18:20:51.531 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116906.log 2023-10-14T18:20:51.532 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116394.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.116394.log.gz 2023-10-14T18:20:51.533 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112601.log 2023-10-14T18:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116906.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.116906.log.gz 2023-10-14T18:20:51.534 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142845.log 2023-10-14T18:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112601.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.112601.log.gz 2023-10-14T18:20:51.535 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129052.log 2023-10-14T18:20:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142845.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142845.log.gz 2023-10-14T18:20:51.537 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96639.log 2023-10-14T18:20:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129052.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129052.log.gz 2023-10-14T18:20:51.538 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150582.log 2023-10-14T18:20:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96639.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.96639.log.gz 2023-10-14T18:20:51.539 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110676.log 2023-10-14T18:20:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150582.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.150582.log.gz 2023-10-14T18:20:51.541 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.2.94815.log 2023-10-14T18:20:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110676.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110676.log.gz 2023-10-14T18:20:51.542 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157746.log 2023-10-14T18:20:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.94815.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.94815.log.gz 2023-10-14T18:20:51.543 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117583.log 2023-10-14T18:20:51.545 INFO:teuthology.orchestra.run.smithi186.stderr: 92.4% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2023-10-14T18:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157746.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.157746.log.gz 2023-10-14T18:20:51.545 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97305.log 2023-10-14T18:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149291.log 2023-10-14T18:20:51.546 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117583.log.gz 2023-10-14T18:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97305.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97305.log.gz 2023-10-14T18:20:51.547 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128049.log 2023-10-14T18:20:51.548 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149291.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142589.log 2023-10-14T18:20:51.549 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149291.log.gz 2023-10-14T18:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128049.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128049.log.gz 2023-10-14T18:20:51.550 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156984.log 2023-10-14T18:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142589.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.142589.log.gz 2023-10-14T18:20:51.551 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115267.log 2023-10-14T18:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156984.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156984.log.gz 2023-10-14T18:20:51.552 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112992.log 2023-10-14T18:20:51.554 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115267.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.115267.log.gzgzip 2023-10-14T18:20:51.554 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.135464.log 2023-10-14T18:20:51.555 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98246.log 2023-10-14T18:20:51.555 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112992.log.gz 2023-10-14T18:20:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135464.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.135464.log.gz 2023-10-14T18:20:51.556 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95784.log 2023-10-14T18:20:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98246.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98246.log.gz 2023-10-14T18:20:51.558 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93675.log 2023-10-14T18:20:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95784.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.95784.log.gz 2023-10-14T18:20:51.559 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.80980.log 2023-10-14T18:20:51.560 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124087.log 2023-10-14T18:20:51.561 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.93675.log.gz 2023-10-14T18:20:51.561 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.tmp-client.admin.80980.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.80980.log.gz 2023-10-14T18:20:51.562 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96211.log 2023-10-14T18:20:51.563 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98158.log 2023-10-14T18:20:51.563 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.124087.log.gz 2023-10-14T18:20:51.564 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96211.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96211.log.gz 2023-10-14T18:20:51.565 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100565.log 2023-10-14T18:20:51.565 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98158.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98158.log.gz 2023-10-14T18:20:51.566 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117285.log 2023-10-14T18:20:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100565.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155503.log 2023-10-14T18:20:51.570 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100565.log.gz 2023-10-14T18:20:51.570 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117285.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117285.log.gz 2023-10-14T18:20:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156031.log 2023-10-14T18:20:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155503.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155503.log.gz 2023-10-14T18:20:51.571 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129537.log 2023-10-14T18:20:51.572 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138060.log 2023-10-14T18:20:51.572 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156031.log.gz 2023-10-14T18:20:51.573 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129537.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.129537.log.gz 2023-10-14T18:20:51.573 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138144.log 2023-10-14T18:20:51.574 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138060.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138060.log.gz 2023-10-14T18:20:51.574 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117710.log 2023-10-14T18:20:51.575 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138144.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138144.log.gz 2023-10-14T18:20:51.575 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144096.log 2023-10-14T18:20:51.576 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117710.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117710.log.gz 2023-10-14T18:20:51.577 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100618.log 2023-10-14T18:20:51.577 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144096.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.144096.log.gz 2023-10-14T18:20:51.578 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111497.log 2023-10-14T18:20:51.579 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100618.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.100618.log.gz 2023-10-14T18:20:51.579 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153615.log 2023-10-14T18:20:51.580 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111497.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111497.log.gz 2023-10-14T18:20:51.580 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160910.log 2023-10-14T18:20:51.581 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153615.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153615.log.gz 2023-10-14T18:20:51.582 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106610.log 2023-10-14T18:20:51.583 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160910.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160910.log.gz 2023-10-14T18:20:51.583 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113472.log 2023-10-14T18:20:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106610.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.106610.log.gz 2023-10-14T18:20:51.584 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159847.log 2023-10-14T18:20:51.585 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91825.log 2023-10-14T18:20:51.586 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113472.log.gz 2023-10-14T18:20:51.587 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159847.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.159847.log.gz 2023-10-14T18:20:51.587 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154389.log 2023-10-14T18:20:51.588 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91825.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.91825.log.gz 2023-10-14T18:20:51.588 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129429.log 2023-10-14T18:20:51.589 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154389.log: gzip 89.2% -- replaced with /var/log/ceph/ceph-client.admin.154389.log.gz -5 2023-10-14T18:20:51.589 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.119173.log 2023-10-14T18:20:51.590 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129429.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.129429.log.gzgzip 2023-10-14T18:20:51.591 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.123493.log 2023-10-14T18:20:51.592 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119173.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.119173.log.gz 2023-10-14T18:20:51.592 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121586.log 2023-10-14T18:20:51.593 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123493.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123493.log.gz 2023-10-14T18:20:51.593 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150743.log 2023-10-14T18:20:51.594 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121586.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.121586.log.gz 2023-10-14T18:20:51.595 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141851.log 2023-10-14T18:20:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150743.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.150743.log.gz 2023-10-14T18:20:51.596 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93633.log 2023-10-14T18:20:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141851.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.141851.log.gz 2023-10-14T18:20:51.597 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133475.log 2023-10-14T18:20:51.598 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93633.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93633.log.gz 2023-10-14T18:20:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114090.log 2023-10-14T18:20:51.599 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133475.log: gzip -5 --verbose 89.2% -- -- replaced with /var/log/ceph/ceph-client.admin.133475.log.gz /var/log/ceph/ceph-client.admin.136453.log 2023-10-14T18:20:51.600 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.601 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114090.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114090.log.gz -5 2023-10-14T18:20:51.601 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.113651.log 2023-10-14T18:20:51.602 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136453.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.136453.log.gz 2023-10-14T18:20:51.602 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110140.log 2023-10-14T18:20:51.603 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113651.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113651.log.gz 2023-10-14T18:20:51.604 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160341.log 2023-10-14T18:20:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110140.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110140.log.gz 2023-10-14T18:20:51.605 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133801.log 2023-10-14T18:20:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160341.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160341.log.gz 2023-10-14T18:20:51.606 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157026.log 2023-10-14T18:20:51.607 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133801.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.133801.log.gz 2023-10-14T18:20:51.608 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146248.log 2023-10-14T18:20:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157026.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.157026.log.gz 2023-10-14T18:20:51.609 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151702.log 2023-10-14T18:20:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146248.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146248.log.gz 2023-10-14T18:20:51.610 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149037.log 2023-10-14T18:20:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151702.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151702.log.gz 2023-10-14T18:20:51.611 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124350.log 2023-10-14T18:20:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149037.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149037.log.gz 2023-10-14T18:20:51.613 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99354.log 2023-10-14T18:20:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124350.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124350.log.gz 2023-10-14T18:20:51.614 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114779.log 2023-10-14T18:20:51.615 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99354.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95070.log 2023-10-14T18:20:51.616 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99354.log.gz 2023-10-14T18:20:51.617 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114779.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.114779.log.gz 2023-10-14T18:20:51.617 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151847.log 2023-10-14T18:20:51.618 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149597.log 2023-10-14T18:20:51.618 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95070.log.gz 2023-10-14T18:20:51.619 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151847.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151847.log.gz 2023-10-14T18:20:51.619 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133339.log 2023-10-14T18:20:51.621 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149597.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.149597.log.gz -- 2023-10-14T18:20:51.621 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.107328.log 2023-10-14T18:20:51.622 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133339.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.133339.log.gz 2023-10-14T18:20:51.622 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116035.log 2023-10-14T18:20:51.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107328.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107328.log.gz 2023-10-14T18:20:51.623 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113598.log 2023-10-14T18:20:51.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116035.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.116035.log.gz 2023-10-14T18:20:51.625 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126384.log 2023-10-14T18:20:51.626 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113598.log: 89.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.113598.log.gz 2023-10-14T18:20:51.626 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.2.94811.log 2023-10-14T18:20:51.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126384.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.126384.log.gz 2023-10-14T18:20:51.627 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149469.log 2023-10-14T18:20:51.628 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.2.94811.log: 0.0% -- replaced with /var/log/ceph/ceph-client.2.94811.log.gz 2023-10-14T18:20:51.629 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120516.log 2023-10-14T18:20:51.630 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150785.log 2023-10-14T18:20:51.630 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149469.log.gz 2023-10-14T18:20:51.631 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120516.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120516.log.gz 2023-10-14T18:20:51.632 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123816.log 2023-10-14T18:20:51.632 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150785.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126425.log 2023-10-14T18:20:51.633 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.150785.log.gz 2023-10-14T18:20:51.634 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123816.log: gzip -5 --verbose -- 89.2% /var/log/ceph/ceph-client.admin.106842.log -- replaced with /var/log/ceph/ceph-client.admin.123816.log.gz 2023-10-14T18:20:51.634 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.635 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126425.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.126425.log.gz 2023-10-14T18:20:51.635 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107806.log 2023-10-14T18:20:51.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106842.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106842.log.gz 2023-10-14T18:20:51.637 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94511.log 2023-10-14T18:20:51.638 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107806.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.107178.log -- replaced with /var/log/ceph/ceph-client.admin.107806.log.gz 2023-10-14T18:20:51.638 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.639 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94511.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94511.log.gz 2023-10-14T18:20:51.639 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152345.log 2023-10-14T18:20:51.640 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107178.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107178.log.gz 2023-10-14T18:20:51.640 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128654.log 2023-10-14T18:20:51.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152345.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152345.log.gz 2023-10-14T18:20:51.642 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119691.log 2023-10-14T18:20:51.643 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127595.log 2023-10-14T18:20:51.643 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.128654.log.gz 2023-10-14T18:20:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119691.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.119691.log.gz 2023-10-14T18:20:51.644 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137677.log 2023-10-14T18:20:51.645 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127595.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127595.log.gz 2023-10-14T18:20:51.646 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121694.log 2023-10-14T18:20:51.647 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137677.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.137677.log.gz 2023-10-14T18:20:51.647 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119502.log 2023-10-14T18:20:51.648 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121694.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.121694.log.gz 2023-10-14T18:20:51.649 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118288.log 2023-10-14T18:20:51.650 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119502.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.119502.log.gz 2023-10-14T18:20:51.650 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161612.log 2023-10-14T18:20:51.651 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118288.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118288.log.gz 2023-10-14T18:20:51.651 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154517.log 2023-10-14T18:20:51.652 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161612.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.161612.log.gz 2023-10-14T18:20:51.653 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156690.log 2023-10-14T18:20:51.654 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154517.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154517.log.gz 2023-10-14T18:20:51.654 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111677.log 2023-10-14T18:20:51.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156690.log: gzip -5 --verbose 88.4% -- -- replaced with /var/log/ceph/ceph-client.admin.156690.log.gz /var/log/ceph/ceph-client.admin.99810.log 2023-10-14T18:20:51.655 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.656 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111677.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111677.log.gz 2023-10-14T18:20:51.656 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150523.log 2023-10-14T18:20:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99810.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.99810.log.gz 2023-10-14T18:20:51.658 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92721.log 2023-10-14T18:20:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150523.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150523.log.gz 2023-10-14T18:20:51.659 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135422.log 2023-10-14T18:20:51.660 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92721.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.92721.log.gz 2023-10-14T18:20:51.661 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106652.log 2023-10-14T18:20:51.662 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129854.log 2023-10-14T18:20:51.662 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.135422.log.gz 2023-10-14T18:20:51.663 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106652.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139080.log 2023-10-14T18:20:51.663 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106652.log.gz 2023-10-14T18:20:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129854.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.129854.log.gz 2023-10-14T18:20:51.664 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137509.log 2023-10-14T18:20:51.666 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139080.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.139080.log.gzgzip 2023-10-14T18:20:51.666 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.107220.log 2023-10-14T18:20:51.667 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137509.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.137509.log.gz 2023-10-14T18:20:51.667 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.150437.log 2023-10-14T18:20:51.668 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107220.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107220.log.gz 2023-10-14T18:20:51.668 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146409.log 2023-10-14T18:20:51.669 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150437.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.150437.log.gz 2023-10-14T18:20:51.670 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109867.log 2023-10-14T18:20:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146409.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.146409.log.gz 2023-10-14T18:20:51.671 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123985.log 2023-10-14T18:20:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109867.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109867.log.gz 2023-10-14T18:20:51.672 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117625.log 2023-10-14T18:20:51.673 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123985.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.123985.log.gz 2023-10-14T18:20:51.674 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114216.log 2023-10-14T18:20:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117625.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.117625.log.gz 2023-10-14T18:20:51.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135084.log 2023-10-14T18:20:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114216.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114216.log.gz 2023-10-14T18:20:51.676 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116299.log 2023-10-14T18:20:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135084.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135084.log.gz 2023-10-14T18:20:51.677 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87505.log 2023-10-14T18:20:51.678 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147634.log 2023-10-14T18:20:51.679 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116299.log.gz 2023-10-14T18:20:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87505.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.87505.log.gz 2023-10-14T18:20:51.680 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150953.log 2023-10-14T18:20:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147634.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147634.log.gz 2023-10-14T18:20:51.681 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147771.log 2023-10-14T18:20:51.682 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150953.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150953.log.gz 2023-10-14T18:20:51.683 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120006.log 2023-10-14T18:20:51.684 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147771.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.147771.log.gz 2023-10-14T18:20:51.684 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160383.log 2023-10-14T18:20:51.685 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120006.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.120006.log.gz 2023-10-14T18:20:51.685 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134335.log 2023-10-14T18:20:51.686 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160383.log: gzip -5 --verbose -- 88.8% /var/log/ceph/ceph-client.admin.152015.log -- replaced with /var/log/ceph/ceph-client.admin.160383.log.gz 2023-10-14T18:20:51.686 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.687 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134335.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134335.log.gz 2023-10-14T18:20:51.688 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149999.log 2023-10-14T18:20:51.689 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125973.log 2023-10-14T18:20:51.689 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152015.log.gz 2023-10-14T18:20:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149999.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.149999.log.gz 2023-10-14T18:20:51.690 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141546.log 2023-10-14T18:20:51.691 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125973.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125973.log.gz 2023-10-14T18:20:51.692 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109825.log 2023-10-14T18:20:51.692 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141546.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.141546.log.gz 2023-10-14T18:20:51.693 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117327.log 2023-10-14T18:20:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109825.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.109825.log.gz 2023-10-14T18:20:51.694 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119418.log 2023-10-14T18:20:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117327.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117327.log.gz 2023-10-14T18:20:51.695 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156942.log 2023-10-14T18:20:51.697 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119418.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123756.log 2023-10-14T18:20:51.697 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.119418.log.gz 2023-10-14T18:20:51.698 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91733.log 2023-10-14T18:20:51.698 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156942.log.gz 2023-10-14T18:20:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123756.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.123756.log.gz 2023-10-14T18:20:51.699 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122344.log 2023-10-14T18:20:51.700 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91733.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.91733.log.gz 2023-10-14T18:20:51.701 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142332.log 2023-10-14T18:20:51.702 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122344.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.122344.log.gz 2023-10-14T18:20:51.702 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98074.log 2023-10-14T18:20:51.703 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142332.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.142332.log.gz 2023-10-14T18:20:51.703 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142956.log 2023-10-14T18:20:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98074.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.98074.log.gz 2023-10-14T18:20:51.704 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113076.log 2023-10-14T18:20:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142956.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.142956.log.gz 2023-10-14T18:20:51.706 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160215.log 2023-10-14T18:20:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113076.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.113076.log.gz 2023-10-14T18:20:51.707 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143000.log 2023-10-14T18:20:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160215.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.160215.log.gz 2023-10-14T18:20:51.708 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104860.log 2023-10-14T18:20:51.709 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143000.log: gzip -5 --verbose 88.3% -- -- replaced with /var/log/ceph/ceph-client.admin.143000.log.gz /var/log/ceph/ceph-client.admin.134990.log 2023-10-14T18:20:51.710 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104860.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.104860.log.gz 2023-10-14T18:20:51.711 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110455.log 2023-10-14T18:20:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134990.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.134990.log.gz 2023-10-14T18:20:51.712 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128474.log 2023-10-14T18:20:51.714 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110455.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116204.log 2023-10-14T18:20:51.714 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110455.log.gz 2023-10-14T18:20:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128474.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.128474.log.gz 2023-10-14T18:20:51.715 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106017.log 2023-10-14T18:20:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116204.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116204.log.gz 2023-10-14T18:20:51.716 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150299.log 2023-10-14T18:20:51.717 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106017.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106017.log.gz 2023-10-14T18:20:51.718 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110592.log 2023-10-14T18:20:51.719 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114821.log 2023-10-14T18:20:51.719 INFO:teuthology.orchestra.run.smithi029.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.150299.log.gz 2023-10-14T18:20:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110592.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110592.log.gz 2023-10-14T18:20:51.720 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92704.log 2023-10-14T18:20:51.721 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114821.log: gzip 89.0% -- replaced with /var/log/ceph/ceph-client.admin.114821.log.gz 2023-10-14T18:20:51.721 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.110988.log 2023-10-14T18:20:51.723 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92704.log: gzip 88.1% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.92704.log.gz -- 2023-10-14T18:20:51.723 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.133759.log 2023-10-14T18:20:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110988.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110988.log.gz 2023-10-14T18:20:51.724 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104597.log 2023-10-14T18:20:51.725 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133759.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.133759.log.gz 2023-10-14T18:20:51.725 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114557.log 2023-10-14T18:20:51.726 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104597.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.104597.log.gz 2023-10-14T18:20:51.727 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137083.log 2023-10-14T18:20:51.728 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114557.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151534.log 2023-10-14T18:20:51.728 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.114557.log.gz 2023-10-14T18:20:51.729 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137083.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.137083.log.gz 2023-10-14T18:20:51.729 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157488.log 2023-10-14T18:20:51.730 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151534.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.151534.log.gz 2023-10-14T18:20:51.731 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149163.log 2023-10-14T18:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157488.log: gzip 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157488.log.gz -5 2023-10-14T18:20:51.732 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.152598.log 2023-10-14T18:20:51.733 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149163.log: gzip -5 88.0% --verbose -- replaced with /var/log/ceph/ceph-client.admin.149163.log.gz -- 2023-10-14T18:20:51.733 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.157788.log 2023-10-14T18:20:51.734 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152598.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.152598.log.gz 2023-10-14T18:20:51.735 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126466.log 2023-10-14T18:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157788.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157788.log.gz 2023-10-14T18:20:51.736 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154919.log 2023-10-14T18:20:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126466.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.126466.log.gz 2023-10-14T18:20:51.737 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111455.log 2023-10-14T18:20:51.738 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154919.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.154919.log.gz -5 2023-10-14T18:20:51.739 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.118078.log 2023-10-14T18:20:51.740 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111455.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111455.log.gz 2023-10-14T18:20:51.740 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119334.log 2023-10-14T18:20:51.741 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124988.log 2023-10-14T18:20:51.741 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.118078.log.gz 2023-10-14T18:20:51.742 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119334.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119334.log.gz 2023-10-14T18:20:51.743 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155713.log 2023-10-14T18:20:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124988.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.124988.log.gz 2023-10-14T18:20:51.744 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151889.log 2023-10-14T18:20:51.745 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105567.log 2023-10-14T18:20:51.745 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155713.log.gz 2023-10-14T18:20:51.746 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151889.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151889.log.gz 2023-10-14T18:20:51.747 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92377.log 2023-10-14T18:20:51.748 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105567.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.105567.log.gzgzip 2023-10-14T18:20:51.748 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.125493.log 2023-10-14T18:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92377.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109664.log 2023-10-14T18:20:51.749 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92377.log.gz 2023-10-14T18:20:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.125493.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.125493.log.gz 2023-10-14T18:20:51.750 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158170.log 2023-10-14T18:20:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109664.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.109664.log.gz 2023-10-14T18:20:51.752 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158044.log 2023-10-14T18:20:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158170.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.158170.log.gz 2023-10-14T18:20:51.753 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136810.log 2023-10-14T18:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119030.log 2023-10-14T18:20:51.754 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.158044.log.gz 2023-10-14T18:20:51.755 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136810.log: 88.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.136810.log.gz 2023-10-14T18:20:51.756 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.147135.log 2023-10-14T18:20:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119030.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.119030.log.gz 2023-10-14T18:20:51.757 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157958.log 2023-10-14T18:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153405.log 2023-10-14T18:20:51.758 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.147135.log.gz 2023-10-14T18:20:51.759 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157958.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157958.log.gz 2023-10-14T18:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155971.log 2023-10-14T18:20:51.760 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153405.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153405.log.gz 2023-10-14T18:20:51.761 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127965.log 2023-10-14T18:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111763.log 2023-10-14T18:20:51.762 INFO:teuthology.orchestra.run.smithi029.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.155971.log.gz 2023-10-14T18:20:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127965.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127965.log.gz 2023-10-14T18:20:51.763 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113556.log 2023-10-14T18:20:51.764 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111763.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111763.log.gz 2023-10-14T18:20:51.765 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106926.log 2023-10-14T18:20:51.766 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113556.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159531.log 2023-10-14T18:20:51.766 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113556.log.gz 2023-10-14T18:20:51.767 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106926.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.106926.log.gz 2023-10-14T18:20:51.767 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.108022.log 2023-10-14T18:20:51.768 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159531.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159531.log.gz -5 2023-10-14T18:20:51.768 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.160425.log 2023-10-14T18:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.108022.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.108022.log.gzgzip 2023-10-14T18:20:51.770 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.142122.log 2023-10-14T18:20:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160425.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.160425.log.gz 2023-10-14T18:20:51.771 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110371.log 2023-10-14T18:20:51.773 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142122.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.142122.log.gz 2023-10-14T18:20:51.774 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160784.log 2023-10-14T18:20:51.774 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110371.log: gzip 87.8% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.110371.log.gz -- 2023-10-14T18:20:51.774 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.115703.log 2023-10-14T18:20:51.775 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160784.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.160784.log.gz 2023-10-14T18:20:51.775 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135254.log 2023-10-14T18:20:51.776 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115703.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115703.log.gz 2023-10-14T18:20:51.776 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107262.log 2023-10-14T18:20:51.777 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135254.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.135254.log.gz 2023-10-14T18:20:51.778 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136537.log 2023-10-14T18:20:51.779 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148017.log 2023-10-14T18:20:51.779 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.107262.log.gz 2023-10-14T18:20:51.780 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136537.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.136537.log.gz 2023-10-14T18:20:51.780 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123619.log 2023-10-14T18:20:51.781 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148017.log: 88.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.148017.log.gz 2023-10-14T18:20:51.781 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98874.log 2023-10-14T18:20:51.783 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129579.log 2023-10-14T18:20:51.783 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.123619.log.gz 2023-10-14T18:20:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98874.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98874.log.gz 2023-10-14T18:20:51.784 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151660.log 2023-10-14T18:20:51.785 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129579.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.129579.log.gz 2023-10-14T18:20:51.785 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116846.log 2023-10-14T18:20:51.786 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151660.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.151660.log.gz 2023-10-14T18:20:51.787 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121292.log 2023-10-14T18:20:51.788 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116846.log: gzip -5 --verbose -- 88.1% /var/log/ceph/ceph-client.admin.120288.log 2023-10-14T18:20:51.788 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.116846.log.gz 2023-10-14T18:20:51.789 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121292.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.121292.log.gz 2023-10-14T18:20:51.789 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101227.log 2023-10-14T18:20:51.790 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120288.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.120288.log.gz 2023-10-14T18:20:51.790 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97179.log 2023-10-14T18:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101227.log: gzip 87.5% -5 -- replaced with /var/log/ceph/ceph-client.admin.101227.log.gz --verbose 2023-10-14T18:20:51.792 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.105891.log 2023-10-14T18:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97179.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97179.log.gz 2023-10-14T18:20:51.793 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146956.log 2023-10-14T18:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105891.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105891.log.gz 2023-10-14T18:20:51.794 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93423.log 2023-10-14T18:20:51.795 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146956.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146956.log.gz 2023-10-14T18:20:51.796 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113693.log 2023-10-14T18:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119215.log 2023-10-14T18:20:51.797 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.93423.log.gz 2023-10-14T18:20:51.798 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113693.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.113693.log.gz 2023-10-14T18:20:51.798 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149555.log 2023-10-14T18:20:51.799 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119215.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.119215.log.gz 2023-10-14T18:20:51.800 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156252.log 2023-10-14T18:20:51.801 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149555.log: 88.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.149555.log.gz 2023-10-14T18:20:51.801 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.141996.log 2023-10-14T18:20:51.802 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158908.log 2023-10-14T18:20:51.802 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156252.log.gz 2023-10-14T18:20:51.803 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141996.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.141996.log.gz 2023-10-14T18:20:51.804 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142757.log 2023-10-14T18:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158908.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.158908.log.gz 2023-10-14T18:20:51.805 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111931.log 2023-10-14T18:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142757.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142757.log.gz 2023-10-14T18:20:51.806 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97780.log 2023-10-14T18:20:51.807 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111931.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.111931.log.gz 2023-10-14T18:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109554.log 2023-10-14T18:20:51.808 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97780.log: gzip 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97780.log.gz -5 2023-10-14T18:20:51.809 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.110634.log 2023-10-14T18:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127755.log 2023-10-14T18:20:51.810 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109554.log.gz 2023-10-14T18:20:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110634.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110634.log.gz 2023-10-14T18:20:51.811 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140574.log 2023-10-14T18:20:51.812 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127755.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.127755.log.gz 2023-10-14T18:20:51.813 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110098.log 2023-10-14T18:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140574.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.140574.log.gz 2023-10-14T18:20:51.814 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2023-10-14T18:20:51.815 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128516.log 2023-10-14T18:20:51.816 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110098.log.gz 2023-10-14T18:20:51.817 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92163.log 2023-10-14T18:20:51.818 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128516.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128516.log.gz 2023-10-14T18:20:51.818 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115033.log 2023-10-14T18:20:51.819 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157914.log 2023-10-14T18:20:51.820 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.92163.log.gz 2023-10-14T18:20:51.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115033.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.115033.log.gz 2023-10-14T18:20:51.821 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114391.log 2023-10-14T18:20:51.822 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123346.log 2023-10-14T18:20:51.822 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.157914.log.gz 2023-10-14T18:20:51.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114391.log: 90.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2023-10-14T18:20:51.824 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.114391.log.gz 2023-10-14T18:20:51.824 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153807.log 2023-10-14T18:20:51.825 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128186.log 2023-10-14T18:20:51.825 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.123346.log.gz 2023-10-14T18:20:51.826 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153807.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.153807.log.gz 2023-10-14T18:20:51.826 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117668.log 2023-10-14T18:20:51.827 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128186.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.128186.log.gz 2023-10-14T18:20:51.828 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105746.log 2023-10-14T18:20:51.829 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117668.log: gzip 88.2% -5 -- replaced with /var/log/ceph/ceph-client.admin.117668.log.gz --verbose 2023-10-14T18:20:51.829 INFO:teuthology.orchestra.run.smithi029.stderr: -- /var/log/ceph/ceph-client.admin.92205.log 2023-10-14T18:20:51.830 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105746.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.105746.log.gz 2023-10-14T18:20:51.830 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87310.log 2023-10-14T18:20:51.831 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111413.log 2023-10-14T18:20:51.832 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92205.log.gz 2023-10-14T18:20:51.832 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87310.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87310.log.gz 2023-10-14T18:20:51.833 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95616.log 2023-10-14T18:20:51.834 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111413.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111413.log.gz 2023-10-14T18:20:51.834 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126667.log 2023-10-14T18:20:51.835 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95616.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.95616.log.gz 2023-10-14T18:20:51.835 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106059.log 2023-10-14T18:20:51.836 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126667.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126667.log.gz 2023-10-14T18:20:51.837 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100471.log 2023-10-14T18:20:51.838 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.153998.log 2023-10-14T18:20:51.838 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.106059.log.gz 2023-10-14T18:20:51.839 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100471.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.100471.log.gz 2023-10-14T18:20:51.839 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127332.log 2023-10-14T18:20:51.840 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.153998.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.153998.log.gz 2023-10-14T18:20:51.840 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143593.log 2023-10-14T18:20:51.842 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127332.log: 87.3%gzip -- replaced with /var/log/ceph/ceph-client.admin.127332.log.gz 2023-10-14T18:20:51.842 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.133938.log 2023-10-14T18:20:51.843 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.143593.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143593.log.gz 2023-10-14T18:20:51.843 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97738.log 2023-10-14T18:20:51.844 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133938.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.133938.log.gz 2023-10-14T18:20:51.844 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128560.log 2023-10-14T18:20:51.845 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97738.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.97738.log.gz 2023-10-14T18:20:51.846 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154475.log 2023-10-14T18:20:51.847 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128560.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128560.log.gz 2023-10-14T18:20:51.847 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115458.log 2023-10-14T18:20:51.848 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105285.log 89.2% 2023-10-14T18:20:51.848 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.154475.log.gz 2023-10-14T18:20:51.849 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.115458.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115458.log.gz 2023-10-14T18:20:51.849 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133297.log 2023-10-14T18:20:51.850 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105285.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.105285.log.gz 2023-10-14T18:20:51.851 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96969.log 2023-10-14T18:20:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133297.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133297.log.gz 2023-10-14T18:20:51.852 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97137.log 2023-10-14T18:20:51.853 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96969.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96969.log.gz 2023-10-14T18:20:51.853 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127655.log 2023-10-14T18:20:51.854 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97137.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97137.log.gz 2023-10-14T18:20:51.854 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92714.log 2023-10-14T18:20:51.855 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127655.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127655.log.gz 2023-10-14T18:20:51.856 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141809.log 2023-10-14T18:20:51.857 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92714.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92714.log.gz 2023-10-14T18:20:51.857 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148995.log 2023-10-14T18:20:51.858 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141809.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141809.log.gz 2023-10-14T18:20:51.858 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107136.log 2023-10-14T18:20:51.859 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148995.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148995.log.gz 2023-10-14T18:20:51.860 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161275.log 2023-10-14T18:20:51.861 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135170.log 2023-10-14T18:20:51.861 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107136.log.gz 2023-10-14T18:20:51.862 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102094.log 2023-10-14T18:20:51.862 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.161275.log.gz 2023-10-14T18:20:51.863 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135170.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.135170.log.gz 2023-10-14T18:20:51.863 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115642.log 2023-10-14T18:20:51.865 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104979.log 2023-10-14T18:20:51.865 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.102094.log.gz 2023-10-14T18:20:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115642.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.115642.log.gz 2023-10-14T18:20:51.866 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160257.log 2023-10-14T18:20:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104979.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.104979.log.gz 2023-10-14T18:20:51.867 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101976.log 2023-10-14T18:20:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160257.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.160257.log.gz 2023-10-14T18:20:51.868 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100110.log 2023-10-14T18:20:51.869 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101976.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.101976.log.gz 2023-10-14T18:20:51.870 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97431.log 2023-10-14T18:20:51.871 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100110.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100110.log.gz 2023-10-14T18:20:51.871 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91689.log 2023-10-14T18:20:51.872 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97431.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97431.log.gzgzip 2023-10-14T18:20:51.872 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.143168.log 2023-10-14T18:20:51.873 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91689.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.91689.log.gz 2023-10-14T18:20:51.874 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96597.log 2023-10-14T18:20:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143168.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143168.log.gz 2023-10-14T18:20:51.875 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135338.log 2023-10-14T18:20:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96597.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96597.log.gz 2023-10-14T18:20:51.876 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146683.log 2023-10-14T18:20:51.877 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135338.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135338.log.gz 2023-10-14T18:20:51.877 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141125.log 2023-10-14T18:20:51.878 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146683.log: 89.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.146683.log.gz 2023-10-14T18:20:51.879 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.149513.log 2023-10-14T18:20:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141125.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141125.log.gz 2023-10-14T18:20:51.880 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146725.log 2023-10-14T18:20:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149513.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.149513.log.gz 2023-10-14T18:20:51.881 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152117.log 2023-10-14T18:20:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146725.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146725.log.gz 2023-10-14T18:20:51.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134461.log 2023-10-14T18:20:51.883 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152117.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152117.log.gz 2023-10-14T18:20:51.884 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136579.log 2023-10-14T18:20:51.885 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158840.log 2023-10-14T18:20:51.885 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.134461.log.gz 2023-10-14T18:20:51.886 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136579.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.136579.log.gz 2023-10-14T18:20:51.886 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135883.log 2023-10-14T18:20:51.887 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158840.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.158840.log.gz 2023-10-14T18:20:51.888 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154643.log 2023-10-14T18:20:51.888 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135883.log: gzip -5 88.7% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.135883.log.gz /var/log/ceph/ceph-client.admin.134209.log 2023-10-14T18:20:51.889 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.890 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154643.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154643.log.gz 2023-10-14T18:20:51.890 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148281.log 2023-10-14T18:20:51.891 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134209.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.134209.log.gz 2023-10-14T18:20:51.891 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146914.log 2023-10-14T18:20:51.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148281.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.148281.log.gz 2023-10-14T18:20:51.893 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128432.log 2023-10-14T18:20:51.894 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106526.log 2023-10-14T18:20:51.894 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146914.log.gz 2023-10-14T18:20:51.895 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128432.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128432.log.gz 2023-10-14T18:20:51.895 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117422.log 2023-10-14T18:20:51.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106526.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.106526.log.gz 2023-10-14T18:20:51.897 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93297.log 2023-10-14T18:20:51.898 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141672.log 2023-10-14T18:20:51.898 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.117422.log.gz 2023-10-14T18:20:51.899 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93297.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93297.log.gz 2023-10-14T18:20:51.899 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151313.log 2023-10-14T18:20:51.900 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141672.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141672.log.gz 2023-10-14T18:20:51.900 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123859.log 2023-10-14T18:20:51.902 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148493.log 2023-10-14T18:20:51.902 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.151313.log.gz 2023-10-14T18:20:51.903 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123859.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.123859.log.gz 2023-10-14T18:20:51.903 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136495.log 2023-10-14T18:20:51.904 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148493.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148493.log.gz 2023-10-14T18:20:51.904 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92037.log 2023-10-14T18:20:51.905 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120642.log 2023-10-14T18:20:51.906 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136495.log.gz 2023-10-14T18:20:51.907 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92037.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.92037.log.gz 2023-10-14T18:20:51.907 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138662.log 2023-10-14T18:20:51.908 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120642.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120642.log.gz 2023-10-14T18:20:51.908 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137467.log 2023-10-14T18:20:51.909 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132083.log 2023-10-14T18:20:51.909 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.138662.log.gz 2023-10-14T18:20:51.910 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137467.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.137467.log.gz 2023-10-14T18:20:51.911 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93861.log 2023-10-14T18:20:51.912 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132083.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.132083.log.gz 2023-10-14T18:20:51.912 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97906.log 2023-10-14T18:20:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93861.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93861.log.gz 2023-10-14T18:20:51.913 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147364.log 2023-10-14T18:20:51.915 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97906.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161780.log 2023-10-14T18:20:51.915 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.97906.log.gz 2023-10-14T18:20:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147364.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147364.log.gz 2023-10-14T18:20:51.916 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156504.log 2023-10-14T18:20:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161780.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.161780.log.gz 2023-10-14T18:20:51.917 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161570.log 2023-10-14T18:20:51.918 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127797.log 2023-10-14T18:20:51.918 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156504.log.gz 2023-10-14T18:20:51.919 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161570.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.161570.log.gz 2023-10-14T18:20:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157278.log 2023-10-14T18:20:51.920 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127797.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.127797.log.gz 2023-10-14T18:20:51.921 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128091.log 2023-10-14T18:20:51.922 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157278.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96723.log 2023-10-14T18:20:51.923 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.157278.log.gz 2023-10-14T18:20:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128091.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128091.log.gz 2023-10-14T18:20:51.924 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107010.log 2023-10-14T18:20:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96723.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.96723.log.gz 2023-10-14T18:20:51.925 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158966.log 2023-10-14T18:20:51.926 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107010.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.107010.log.gz 2023-10-14T18:20:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97347.log 2023-10-14T18:20:51.927 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158966.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158966.log.gz 2023-10-14T18:20:51.928 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136767.log 2023-10-14T18:20:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97347.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97347.log.gz 2023-10-14T18:20:51.929 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95028.log 2023-10-14T18:20:51.930 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136767.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91911.log 2023-10-14T18:20:51.930 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.136767.log.gz 2023-10-14T18:20:51.931 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95028.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.95028.log.gz 2023-10-14T18:20:51.932 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151164.log 2023-10-14T18:20:51.933 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91911.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151355.log 2023-10-14T18:20:51.933 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.91911.log.gz 2023-10-14T18:20:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151164.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.151164.log.gz 2023-10-14T18:20:51.934 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96166.log 2023-10-14T18:20:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151355.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.151355.log.gz 2023-10-14T18:20:51.936 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138704.log 2023-10-14T18:20:51.937 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112434.log 2023-10-14T18:20:51.937 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96166.log.gz 2023-10-14T18:20:51.938 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138704.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138704.log.gz 2023-10-14T18:20:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154243.log 2023-10-14T18:20:51.939 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112434.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112434.log.gz 2023-10-14T18:20:51.940 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147448.log 2023-10-14T18:20:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154243.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154243.log.gz 2023-10-14T18:20:51.941 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146630.log 2023-10-14T18:20:51.942 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147448.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98202.log 2023-10-14T18:20:51.942 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.147448.log.gz 2023-10-14T18:20:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146630.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.146630.log.gz 2023-10-14T18:20:51.943 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137238.log 2023-10-14T18:20:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98202.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98202.log.gz 2023-10-14T18:20:51.945 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161360.log 2023-10-14T18:20:51.946 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154285.log 2023-10-14T18:20:51.946 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137238.log.gz 2023-10-14T18:20:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161360.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.161360.log.gz 2023-10-14T18:20:51.947 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110849.log 2023-10-14T18:20:51.948 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154285.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154285.log.gz 2023-10-14T18:20:51.949 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153741.log 2023-10-14T18:20:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110849.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.110849.log.gz 2023-10-14T18:20:51.950 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161528.log 2023-10-14T18:20:51.951 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106694.log 2023-10-14T18:20:51.952 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.153741.log.gz 2023-10-14T18:20:51.952 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161528.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.161528.log.gz 2023-10-14T18:20:51.953 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141588.log 2023-10-14T18:20:51.954 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106694.log: gzip -5 --verbose -- 88.5% /var/log/ceph/ceph-client.admin.125240.log -- replaced with /var/log/ceph/ceph-client.admin.106694.log.gz 2023-10-14T18:20:51.954 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141588.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.141588.log.gz 2023-10-14T18:20:51.955 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117932.log 2023-10-14T18:20:51.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100514.log 2023-10-14T18:20:51.957 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.125240.log.gz 2023-10-14T18:20:51.957 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117932.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.117932.log.gz 2023-10-14T18:20:51.958 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114907.log 2023-10-14T18:20:51.959 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137125.log 2023-10-14T18:20:51.959 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.100514.log.gz 2023-10-14T18:20:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114907.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.114907.log.gz 2023-10-14T18:20:51.960 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138746.log 2023-10-14T18:20:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137125.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.137125.log.gz 2023-10-14T18:20:51.961 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159370.log 2023-10-14T18:20:51.963 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158212.log 2023-10-14T18:20:51.963 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.138746.log.gz 2023-10-14T18:20:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159370.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.159370.log.gz 2023-10-14T18:20:51.964 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155129.log 2023-10-14T18:20:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158212.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.158212.log.gz 2023-10-14T18:20:51.965 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125072.log 2023-10-14T18:20:51.966 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156420.log 2023-10-14T18:20:51.967 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155129.log.gz 2023-10-14T18:20:51.968 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125072.log: gzip -5 --verbose 88.8% -- -- replaced with /var/log/ceph/ceph-client.admin.125072.log.gz /var/log/ceph/ceph-client.admin.93058.log 2023-10-14T18:20:51.968 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156420.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.156420.log.gz 2023-10-14T18:20:51.969 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95574.log 2023-10-14T18:20:51.970 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93058.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.93058.log.gz 2023-10-14T18:20:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114174.log 2023-10-14T18:20:51.971 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95574.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95574.log.gz 2023-10-14T18:20:51.972 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93035.log 2023-10-14T18:20:51.973 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114174.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157872.log 2023-10-14T18:20:51.973 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.114174.log.gz 2023-10-14T18:20:51.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93035.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93035.log.gz 2023-10-14T18:20:51.975 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95868.log 2023-10-14T18:20:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157872.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.157872.log.gz 2023-10-14T18:20:51.976 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2023-10-14T18:20:51.977 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98442.log 2023-10-14T18:20:51.978 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95868.log.gz 2023-10-14T18:20:51.978 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116161.log 2023-10-14T18:20:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98442.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98442.log.gz 2023-10-14T18:20:51.980 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124724.log 2023-10-14T18:20:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116161.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.116161.log.gz 2023-10-14T18:20:51.981 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93549.log 2023-10-14T18:20:51.982 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114599.log 2023-10-14T18:20:51.983 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.124724.log.gz 2023-10-14T18:20:51.984 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125682.log 2023-10-14T18:20:51.984 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.93549.log.gz 2023-10-14T18:20:51.985 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114599.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.114599.log.gz 2023-10-14T18:20:51.985 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96681.log 2023-10-14T18:20:51.986 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125682.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.125682.log.gz 2023-10-14T18:20:51.987 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134251.log 2023-10-14T18:20:51.988 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96681.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96681.log.gz 2023-10-14T18:20:51.988 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157236.log 2023-10-14T18:20:51.989 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134251.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.134251.log.gz 2023-10-14T18:20:51.989 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160162.log 2023-10-14T18:20:51.991 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157236.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.157236.log.gz 2023-10-14T18:20:51.991 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149957.log 2023-10-14T18:20:51.992 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160162.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.160162.log.gz 2023-10-14T18:20:51.992 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148451.log 2023-10-14T18:20:51.994 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149957.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.149957.log.gz 2023-10-14T18:20:51.994 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154201.log 2023-10-14T18:20:51.995 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148451.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148451.log.gz 2023-10-14T18:20:51.995 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119964.log 2023-10-14T18:20:51.996 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154201.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.154201.log.gz 2023-10-14T18:20:51.996 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152941.log 2023-10-14T18:20:51.997 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119964.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119964.log.gz 2023-10-14T18:20:51.998 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156816.log 2023-10-14T18:20:51.999 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152941.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.152941.log.gz 2023-10-14T18:20:51.999 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150911.log 2023-10-14T18:20:52.000 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156816.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156816.log.gz 2023-10-14T18:20:52.000 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152075.log 2023-10-14T18:20:52.001 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150911.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.150911.log.gz 2023-10-14T18:20:52.002 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138620.log 2023-10-14T18:20:52.002 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152075.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.152075.log.gz 2023-10-14T18:20:52.003 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.148914.log 2023-10-14T18:20:52.005 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138620.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138620.log.gz 2023-10-14T18:20:52.005 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133980.log 2023-10-14T18:20:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.148914.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.148914.log.gz 2023-10-14T18:20:52.006 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91995.log 2023-10-14T18:20:52.007 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147851.log 2023-10-14T18:20:52.007 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.133980.log.gz 2023-10-14T18:20:52.008 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91995.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91995.log.gz 2023-10-14T18:20:52.009 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96339.log 2023-10-14T18:20:52.010 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147851.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160551.log 2023-10-14T18:20:52.010 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.147851.log.gz 2023-10-14T18:20:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96339.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.96339.log.gz 2023-10-14T18:20:52.011 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116351.log 2023-10-14T18:20:52.013 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134820.log 2023-10-14T18:20:52.013 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.160551.log.gz 2023-10-14T18:20:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116351.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.116351.log.gz 2023-10-14T18:20:52.014 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158422.log 2023-10-14T18:20:52.015 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134820.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159615.log 2023-10-14T18:20:52.016 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.134820.log.gz 2023-10-14T18:20:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158422.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.158422.log.gz 2023-10-14T18:20:52.017 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.158717.log 2023-10-14T18:20:52.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159615.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.135763.log 2023-10-14T18:20:52.018 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.159615.log.gz 2023-10-14T18:20:52.019 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.158717.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.158717.log.gz 2023-10-14T18:20:52.020 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95112.log 2023-10-14T18:20:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.135763.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.135763.log.gz 2023-10-14T18:20:52.021 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122828.log 2023-10-14T18:20:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95112.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.95112.log.gz 2023-10-14T18:20:52.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149427.log 2023-10-14T18:20:52.024 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122828.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124171.log 2023-10-14T18:20:52.024 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122828.log.gz 2023-10-14T18:20:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149427.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.149427.log.gz 2023-10-14T18:20:52.025 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93056.log 2023-10-14T18:20:52.026 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124171.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91603.log 2023-10-14T18:20:52.026 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.124171.log.gz 2023-10-14T18:20:52.027 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93056.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.93056.log.gz 2023-10-14T18:20:52.028 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116077.log 2023-10-14T18:20:52.029 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153321.log 2023-10-14T18:20:52.029 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91603.log.gz 2023-10-14T18:20:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116077.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.116077.log.gz 2023-10-14T18:20:52.030 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135212.log 2023-10-14T18:20:52.032 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154727.log 2023-10-14T18:20:52.032 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153321.log.gz 2023-10-14T18:20:52.033 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135212.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.135212.log.gz 2023-10-14T18:20:52.033 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127048.log 2023-10-14T18:20:52.034 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141167.log 2023-10-14T18:20:52.034 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.154727.log.gz 2023-10-14T18:20:52.035 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127048.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.127048.log.gz 2023-10-14T18:20:52.036 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91869.log 2023-10-14T18:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141167.log: 87.7%gzip -- replaced with /var/log/ceph/ceph-client.admin.141167.log.gz 2023-10-14T18:20:52.037 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97263.log 2023-10-14T18:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91869.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.91869.log.gz 2023-10-14T18:20:52.038 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105379.log 2023-10-14T18:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97263.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117201.log 2023-10-14T18:20:52.040 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.97263.log.gz 2023-10-14T18:20:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105379.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105379.log.gz 2023-10-14T18:20:52.041 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95532.log 2023-10-14T18:20:52.042 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114729.log 2023-10-14T18:20:52.042 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.117201.log.gz 2023-10-14T18:20:52.043 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95532.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95532.log.gz 2023-10-14T18:20:52.044 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96885.log 2023-10-14T18:20:52.045 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111167.log 2023-10-14T18:20:52.045 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.114729.log.gz 2023-10-14T18:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96885.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96885.log.gz 2023-10-14T18:20:52.046 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161036.log 2023-10-14T18:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111167.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97534.log 2023-10-14T18:20:52.048 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.111167.log.gz 2023-10-14T18:20:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161036.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.161036.log.gz 2023-10-14T18:20:52.049 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153363.log 2023-10-14T18:20:52.050 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97534.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157830.log 2023-10-14T18:20:52.050 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.97534.log.gz 2023-10-14T18:20:52.051 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153363.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.153363.log.gz 2023-10-14T18:20:52.052 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116625.log 2023-10-14T18:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157830.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114006.log 2023-10-14T18:20:52.053 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.157830.log.gz 2023-10-14T18:20:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116625.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116625.log.gz 2023-10-14T18:20:52.054 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147592.log 2023-10-14T18:20:52.055 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114006.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121025.log 2023-10-14T18:20:52.056 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114006.log.gz 2023-10-14T18:20:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147592.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.147592.log.gz 2023-10-14T18:20:52.057 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158380.log 2023-10-14T18:20:52.058 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121025.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152243.log 2023-10-14T18:20:52.058 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121025.log.gz 2023-10-14T18:20:52.059 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158380.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.158380.log.gz 2023-10-14T18:20:52.060 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152472.log 2023-10-14T18:20:52.061 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152243.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126343.log 2023-10-14T18:20:52.061 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.152243.log.gz 2023-10-14T18:20:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152472.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.152472.log.gz 2023-10-14T18:20:52.062 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157404.log 2023-10-14T18:20:52.063 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137196.log 2023-10-14T18:20:52.064 INFO:teuthology.orchestra.run.smithi029.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.126343.log.gz 2023-10-14T18:20:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157404.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.157404.log.gz 2023-10-14T18:20:52.065 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139497.log 2023-10-14T18:20:52.066 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107639.log 2023-10-14T18:20:52.066 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.137196.log.gz 2023-10-14T18:20:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139497.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.139497.log.gz 2023-10-14T18:20:52.067 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129471.log 2023-10-14T18:20:52.069 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93735.log 2023-10-14T18:20:52.069 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.107639.log.gz 2023-10-14T18:20:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129471.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.129471.log.gz 2023-10-14T18:20:52.070 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93777.log 2023-10-14T18:20:52.071 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91434.log 2023-10-14T18:20:52.072 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.93735.log.gz 2023-10-14T18:20:52.072 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93777.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.93777.log.gz 2023-10-14T18:20:52.073 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105241.log 2023-10-14T18:20:52.074 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91434.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96927.log 2023-10-14T18:20:52.074 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91434.log.gz 2023-10-14T18:20:52.075 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105241.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105241.log.gz 2023-10-14T18:20:52.076 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117752.log 2023-10-14T18:20:52.077 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96927.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97491.log 2023-10-14T18:20:52.077 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.96927.log.gz 2023-10-14T18:20:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117752.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.117752.log.gz 2023-10-14T18:20:52.078 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97656.log 2023-10-14T18:20:52.079 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152201.log 2023-10-14T18:20:52.080 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.97491.log.gz 2023-10-14T18:20:52.081 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97656.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97656.log.gz 2023-10-14T18:20:52.081 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122070.log 2023-10-14T18:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157110.log 2023-10-14T18:20:52.082 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.152201.log.gz 2023-10-14T18:20:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122070.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.122070.log.gz 2023-10-14T18:20:52.083 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96843.log 2023-10-14T18:20:52.084 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157110.log: gzip -5 --verbose -- 87.4% /var/log/ceph/ceph-client.admin.120101.log 2023-10-14T18:20:52.085 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.157110.log.gz 2023-10-14T18:20:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96843.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.96843.log.gz 2023-10-14T18:20:52.086 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153107.log 2023-10-14T18:20:52.087 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143880.log 2023-10-14T18:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120101.log.gz 2023-10-14T18:20:52.088 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153107.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.153107.log.gz 2023-10-14T18:20:52.089 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156157.log 2023-10-14T18:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143880.log: gzip -5 88.6% --verbose -- replaced with /var/log/ceph/ceph-client.admin.143880.log.gz -- 2023-10-14T18:20:52.090 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.137761.log 2023-10-14T18:20:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156157.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.156157.log.gz 2023-10-14T18:20:52.091 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149741.log 2023-10-14T18:20:52.092 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125156.log 2023-10-14T18:20:52.093 INFO:teuthology.orchestra.run.smithi029.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.137761.log.gz 2023-10-14T18:20:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149741.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.149741.log.gz 2023-10-14T18:20:52.094 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121962.log 2023-10-14T18:20:52.095 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125156.log: gzip -5 --verbose 87.9% -- -- replaced with /var/log/ceph/ceph-client.admin.125156.log.gz /var/log/ceph/ceph-client.admin.114311.log 2023-10-14T18:20:52.095 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:52.096 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137041.log 2023-10-14T18:20:52.097 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.121962.log.gz 2023-10-14T18:20:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114311.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.114311.log.gz 2023-10-14T18:20:52.098 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155671.log 2023-10-14T18:20:52.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159198.log 2023-10-14T18:20:52.100 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.137041.log.gz 2023-10-14T18:20:52.101 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155671.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.155671.log.gz 2023-10-14T18:20:52.101 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120737.log 2023-10-14T18:20:52.102 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159198.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135296.log 2023-10-14T18:20:52.102 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.159198.log.gz 2023-10-14T18:20:52.103 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120737.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.120737.log.gz 2023-10-14T18:20:52.104 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142673.log 2023-10-14T18:20:52.105 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101437.log 2023-10-14T18:20:52.105 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.135296.log.gz 2023-10-14T18:20:52.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142673.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.142673.log.gz 2023-10-14T18:20:52.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136223.log 2023-10-14T18:20:52.108 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127416.log 2023-10-14T18:20:52.108 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101437.log.gz 2023-10-14T18:20:52.109 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136223.log: gzip -5 --verbose -- 88.6% /var/log/ceph/ceph-client.admin.121334.log -- replaced with /var/log/ceph/ceph-client.admin.136223.log.gz 2023-10-14T18:20:52.109 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:52.111 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155070.log 2023-10-14T18:20:52.111 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.127416.log.gz 2023-10-14T18:20:52.112 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106738.log 2023-10-14T18:20:52.112 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.121334.log.gz 2023-10-14T18:20:52.114 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150395.log 2023-10-14T18:20:52.114 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.155070.log.gz 2023-10-14T18:20:52.115 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128740.log 2023-10-14T18:20:52.115 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.106738.log.gz 2023-10-14T18:20:52.116 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150395.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.150395.log.gz 2023-10-14T18:20:52.117 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92121.log 2023-10-14T18:20:52.118 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128740.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.99173.log 2023-10-14T18:20:52.118 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.128740.log.gz 2023-10-14T18:20:52.119 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138102.log 2023-10-14T18:20:52.119 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.92121.log.gz 2023-10-14T18:20:52.120 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.99173.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99173.log.gz 2023-10-14T18:20:52.121 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137719.log 2023-10-14T18:20:52.122 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.115454.log 2023-10-14T18:20:52.122 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138102.log.gz 2023-10-14T18:20:52.123 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137719.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.137719.log.gz 2023-10-14T18:20:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119460.log 2023-10-14T18:20:52.124 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.115454.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.115454.log.gz 2023-10-14T18:20:52.125 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121206.log 2023-10-14T18:20:52.126 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124768.log 2023-10-14T18:20:52.126 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.119460.log.gz 2023-10-14T18:20:52.127 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157362.log 2023-10-14T18:20:52.128 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.121206.log.gz 2023-10-14T18:20:52.129 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124768.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124768.log.gz 2023-10-14T18:20:52.129 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93903.log 2023-10-14T18:20:52.130 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157362.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143402.log 2023-10-14T18:20:52.131 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.157362.log.gz 2023-10-14T18:20:52.132 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141398.log 2023-10-14T18:20:52.132 INFO:teuthology.orchestra.run.smithi029.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.93903.log.gz 2023-10-14T18:20:52.133 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128968.log 2023-10-14T18:20:52.133 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143402.log.gz 2023-10-14T18:20:52.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141398.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141398.log.gzgzip 2023-10-14T18:20:52.135 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.93465.log 2023-10-14T18:20:52.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128968.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147550.log 2023-10-14T18:20:52.136 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.128968.log.gz 2023-10-14T18:20:52.137 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93465.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93465.log.gz 2023-10-14T18:20:52.138 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161738.log 2023-10-14T18:20:52.139 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147550.log: 87.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.147550.log.gz 2023-10-14T18:20:52.139 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.0.94382.log 2023-10-14T18:20:52.140 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141714.log 2023-10-14T18:20:52.140 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.161738.log.gz 2023-10-14T18:20:52.141 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.94382.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.94382.log.gz 2023-10-14T18:20:52.142 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104334.log 2023-10-14T18:20:52.143 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128390.log 2023-10-14T18:20:52.143 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.141714.log.gz 2023-10-14T18:20:52.144 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147676.log 2023-10-14T18:20:52.144 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.104334.log.gz 2023-10-14T18:20:52.145 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128390.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.128390.log.gz 2023-10-14T18:20:52.146 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121885.log 2023-10-14T18:20:52.147 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147676.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.147676.log.gz 2023-10-14T18:20:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113355.log 2023-10-14T18:20:52.148 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.121885.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121885.log.gz 2023-10-14T18:20:52.149 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142038.log 2023-10-14T18:20:52.150 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113355.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92956.log 2023-10-14T18:20:52.150 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.113355.log.gz 2023-10-14T18:20:52.151 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142038.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.142038.log.gz 2023-10-14T18:20:52.152 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149385.log 2023-10-14T18:20:52.153 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161121.log 2023-10-14T18:20:52.153 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.92956.log.gz 2023-10-14T18:20:52.154 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149385.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.149385.log.gz 2023-10-14T18:20:52.155 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146536.log 2023-10-14T18:20:52.156 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161121.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.161121.log.gz 2023-10-14T18:20:52.156 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124512.log 2023-10-14T18:20:52.157 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97221.log 2023-10-14T18:20:52.157 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.146536.log.gz 2023-10-14T18:20:52.158 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124512.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124512.log.gz 2023-10-14T18:20:52.159 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161654.log 2023-10-14T18:20:52.160 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97221.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111847.log 2023-10-14T18:20:52.160 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.97221.log.gz 2023-10-14T18:20:52.161 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161654.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.161654.log.gz 2023-10-14T18:20:52.161 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2023-10-14T18:20:52.162 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111847.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96423.log 2023-10-14T18:20:52.162 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111847.log.gz 2023-10-14T18:20:52.163 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146998.log 2023-10-14T18:20:52.165 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96423.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.96423.log.gz 2023-10-14T18:20:52.165 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150827.log 2023-10-14T18:20:52.166 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125030.log 87.8% 2023-10-14T18:20:52.167 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.146998.log.gz 2023-10-14T18:20:52.168 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111371.log 2023-10-14T18:20:52.168 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150827.log.gz 2023-10-14T18:20:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125030.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.125030.log.gz 2023-10-14T18:20:52.170 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122004.log 2023-10-14T18:20:52.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159657.log 2023-10-14T18:20:52.171 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.111371.log.gz 2023-10-14T18:20:52.172 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101681.log 2023-10-14T18:20:52.173 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.122004.log.gz 2023-10-14T18:20:52.174 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142248.log 2023-10-14T18:20:52.174 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.159657.log.gz 2023-10-14T18:20:52.175 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101681.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111635.log 2023-10-14T18:20:52.176 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.101681.log.gz 2023-10-14T18:20:52.177 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105704.log 2023-10-14T18:20:52.177 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.142248.log.gz 2023-10-14T18:20:52.178 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111635.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.111635.log.gzgzip 2023-10-14T18:20:52.179 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98116.log 2023-10-14T18:20:52.180 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154075.log 2023-10-14T18:20:52.180 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.105704.log.gz 2023-10-14T18:20:52.181 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129094.log 2023-10-14T18:20:52.182 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.98116.log.gz 2023-10-14T18:20:52.185 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98982.log 2023-10-14T18:20:52.186 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154075.log.gz 2023-10-14T18:20:52.187 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129094.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.129351.log 2023-10-14T18:20:52.187 INFO:teuthology.orchestra.run.smithi029.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.129094.log.gz 2023-10-14T18:20:52.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98982.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124596.log 2023-10-14T18:20:52.188 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.98982.log.gz 2023-10-14T18:20:52.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.129351.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.129351.log.gz 2023-10-14T18:20:52.189 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124860.log 2023-10-14T18:20:52.190 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127839.log 2023-10-14T18:20:52.190 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.124596.log.gz 2023-10-14T18:20:52.191 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105021.log 2023-10-14T18:20:52.191 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.124860.log.gz 2023-10-14T18:20:52.192 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118583.log 2023-10-14T18:20:52.193 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.127839.log.gz 2023-10-14T18:20:52.194 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113838.log 2023-10-14T18:20:52.194 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.105021.log.gz 2023-10-14T18:20:52.195 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.118583.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118583.log.gz 2023-10-14T18:20:52.196 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121502.log 2023-10-14T18:20:52.197 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113838.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115159.log 2023-10-14T18:20:52.197 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.113838.log.gz 2023-10-14T18:20:52.199 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105525.log 2023-10-14T18:20:52.199 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.121502.log.gz 2023-10-14T18:20:52.200 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114991.log 2023-10-14T18:20:52.200 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.115159.log.gz 2023-10-14T18:20:52.201 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140532.log 2023-10-14T18:20:52.202 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.105525.log.gz 2023-10-14T18:20:52.203 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114991.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114991.log.gz 2023-10-14T18:20:52.203 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147280.log 2023-10-14T18:20:52.205 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140532.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97990.log 2023-10-14T18:20:52.205 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140532.log.gz 2023-10-14T18:20:52.207 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147280.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152861.log 2023-10-14T18:20:52.207 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147280.log.gz 2023-10-14T18:20:52.208 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116804.log 2023-10-14T18:20:52.208 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97990.log.gz 2023-10-14T18:20:52.210 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152861.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134778.log 2023-10-14T18:20:52.210 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.152861.log.gz 2023-10-14T18:20:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116804.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116804.log.gz 2023-10-14T18:20:52.211 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141954.log 2023-10-14T18:20:52.213 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134778.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101395.log 2023-10-14T18:20:52.213 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.134778.log.gz 2023-10-14T18:20:52.214 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141954.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.141954.log.gz 2023-10-14T18:20:52.215 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105063.log 2023-10-14T18:20:52.216 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135506.log 2023-10-14T18:20:52.216 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.101395.log.gz 2023-10-14T18:20:52.217 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138324.log 2023-10-14T18:20:52.217 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.105063.log.gz 2023-10-14T18:20:52.219 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146155.log 2023-10-14T18:20:52.219 INFO:teuthology.orchestra.run.smithi029.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.135506.log.gz 2023-10-14T18:20:52.220 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161486.log 2023-10-14T18:20:52.221 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138324.log.gz 2023-10-14T18:20:52.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146155.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139122.log 2023-10-14T18:20:52.222 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.146155.log.gz 2023-10-14T18:20:52.223 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161486.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.161486.log.gz 2023-10-14T18:20:52.224 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87041.log 2023-10-14T18:20:52.225 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124680.log 2023-10-14T18:20:52.225 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.139122.log.gz 2023-10-14T18:20:52.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.87041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157152.log 2023-10-14T18:20:52.227 INFO:teuthology.orchestra.run.smithi029.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.87041.log.gz 2023-10-14T18:20:52.228 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100342.log 2023-10-14T18:20:52.228 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.124680.log.gz 2023-10-14T18:20:52.229 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157152.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92379.log 2023-10-14T18:20:52.230 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.157152.log.gz 2023-10-14T18:20:52.231 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109783.log 2023-10-14T18:20:52.231 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.100342.log.gz 2023-10-14T18:20:52.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92379.log: 87.9%gzip -- replaced with /var/log/ceph/ceph-client.admin.92379.log.gz 2023-10-14T18:20:52.233 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.147040.log 2023-10-14T18:20:52.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157446.log 2023-10-14T18:20:52.234 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.109783.log.gz 2023-10-14T18:20:52.235 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147040.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93255.log 2023-10-14T18:20:52.236 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147040.log.gz 2023-10-14T18:20:52.237 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100385.log 2023-10-14T18:20:52.237 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.157446.log.gz 2023-10-14T18:20:52.239 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96296.log 2023-10-14T18:20:52.239 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93255.log.gz 2023-10-14T18:20:52.240 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109909.log 2023-10-14T18:20:52.241 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100385.log.gz 2023-10-14T18:20:52.242 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95238.log 2023-10-14T18:20:52.242 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.96296.log.gz 2023-10-14T18:20:52.243 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109909.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.109909.log.gz 2023-10-14T18:20:52.244 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156900.log 2023-10-14T18:20:52.245 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95238.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113236.log 2023-10-14T18:20:52.246 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.95238.log.gz 2023-10-14T18:20:52.247 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109951.log 87.6% 2023-10-14T18:20:52.247 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.156900.log.gz 2023-10-14T18:20:52.248 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113236.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112950.log 2023-10-14T18:20:52.248 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.113236.log.gz 2023-10-14T18:20:52.250 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.109951.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.109951.log.gz 2023-10-14T18:20:52.250 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127923.log 2023-10-14T18:20:52.251 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112950.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112950.log.gz 2023-10-14T18:20:52.252 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93028.log 2023-10-14T18:20:52.253 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127923.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160078.log 2023-10-14T18:20:52.253 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.127923.log.gz 2023-10-14T18:20:52.255 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93028.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123252.log 2023-10-14T18:20:52.255 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.93028.log.gz 2023-10-14T18:20:52.256 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160078.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91519.log 2023-10-14T18:20:52.256 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.160078.log.gz 2023-10-14T18:20:52.257 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123252.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.107515.log 2023-10-14T18:20:52.258 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.123252.log.gz 2023-10-14T18:20:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91519.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.91519.log.gz 2023-10-14T18:20:52.259 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110287.log 2023-10-14T18:20:52.261 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.107515.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.107515.log.gz 2023-10-14T18:20:52.261 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102035.log 2023-10-14T18:20:52.263 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110287.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101622.log 2023-10-14T18:20:52.263 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.110287.log.gz 2023-10-14T18:20:52.264 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119838.log 2023-10-14T18:20:52.264 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.102035.log.gz 2023-10-14T18:20:52.265 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127374.log 2023-10-14T18:20:52.265 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.101622.log.gz 2023-10-14T18:20:52.267 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119838.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.119838.log.gz 2023-10-14T18:20:52.267 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97011.log 2023-10-14T18:20:52.269 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127553.log 2023-10-14T18:20:52.269 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127374.log.gz 2023-10-14T18:20:52.270 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97011.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142164.log 2023-10-14T18:20:52.270 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.97011.log.gz 2023-10-14T18:20:52.271 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127553.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.127553.log.gz 2023-10-14T18:20:52.272 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121628.log 2023-10-14T18:20:52.273 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142164.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.142164.log.gz 2023-10-14T18:20:52.273 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142507.log 2023-10-14T18:20:52.274 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121628.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159241.log 2023-10-14T18:20:52.275 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142507.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.121628.log.gz 2023-10-14T18:20:52.276 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.142507.log.gz 2023-10-14T18:20:52.276 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142631.log 2023-10-14T18:20:52.277 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99666.log 2023-10-14T18:20:52.278 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.159241.log.gz 2023-10-14T18:20:52.279 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101563.log 2023-10-14T18:20:52.279 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.142631.log.gz 2023-10-14T18:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99666.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99666.log.gz 2023-10-14T18:20:52.281 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100256.log 2023-10-14T18:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155342.log 2023-10-14T18:20:52.282 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.101563.log.gz 2023-10-14T18:20:52.283 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92381.log 2023-10-14T18:20:52.284 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100256.log.gz 2023-10-14T18:20:52.285 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155342.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96801.log 2023-10-14T18:20:52.285 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.155342.log.gz 2023-10-14T18:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92381.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.128926.log -- replaced with /var/log/ceph/ceph-client.admin.92381.log.gz 2023-10-14T18:20:52.287 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158464.log 2023-10-14T18:20:52.288 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96801.log.gz 2023-10-14T18:20:52.289 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128926.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.128926.log.gz 2023-10-14T18:20:52.290 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122112.log 2023-10-14T18:20:52.291 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91783.log 2023-10-14T18:20:52.291 INFO:teuthology.orchestra.run.smithi029.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.158464.log.gz 2023-10-14T18:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95742.log 2023-10-14T18:20:52.292 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.122112.log.gz 2023-10-14T18:20:52.294 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91783.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.91783.log.gz 2023-10-14T18:20:52.294 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116119.log 2023-10-14T18:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95742.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140044.log 2023-10-14T18:20:52.296 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95742.log.gz 2023-10-14T18:20:52.297 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149699.log 2023-10-14T18:20:52.298 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.116119.log.gz 2023-10-14T18:20:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140044.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.140044.log.gz 2023-10-14T18:20:52.299 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113735.log 2023-10-14T18:20:52.300 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158086.log 2023-10-14T18:20:52.301 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.149699.log.gz 2023-10-14T18:20:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113735.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.113735.log.gz 2023-10-14T18:20:52.302 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156115.log 2023-10-14T18:20:52.303 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158086.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.158086.log.gz 2023-10-14T18:20:52.304 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98340.log 2023-10-14T18:20:52.305 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161444.log 2023-10-14T18:20:52.305 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.156115.log.gz 2023-10-14T18:20:52.306 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98340.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.98340.log.gz 2023-10-14T18:20:52.307 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154433.log 2023-10-14T18:20:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161444.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.161444.log.gz 2023-10-14T18:20:52.308 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160467.log 2023-10-14T18:20:52.309 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154433.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125574.log 2023-10-14T18:20:52.309 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154433.log.gz 2023-10-14T18:20:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160467.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.160467.log.gz 2023-10-14T18:20:52.311 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161698.log 2023-10-14T18:20:52.312 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125574.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.125574.log.gz 2023-10-14T18:20:52.313 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148661.log 2023-10-14T18:20:52.314 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140002.log 2023-10-14T18:20:52.314 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161698.log.gz 2023-10-14T18:20:52.317 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116709.log 2023-10-14T18:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.148661.log.gz 2023-10-14T18:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126828.log 2023-10-14T18:20:52.318 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.140002.log.gz 2023-10-14T18:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116709.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95196.log 2023-10-14T18:20:52.319 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.116709.log.gz 2023-10-14T18:20:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126828.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.126828.log.gz 2023-10-14T18:20:52.320 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153573.log 2023-10-14T18:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95196.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.95196.log.gz 2023-10-14T18:20:52.321 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106444.log 2023-10-14T18:20:52.322 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143042.log 2023-10-14T18:20:52.323 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.153573.log.gz 2023-10-14T18:20:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106444.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.106444.log.gz 2023-10-14T18:20:52.324 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150701.log 2023-10-14T18:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143042.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98790.log 2023-10-14T18:20:52.326 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143042.log.gz 2023-10-14T18:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150701.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.150701.log.gz 2023-10-14T18:20:52.327 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141630.log 2023-10-14T18:20:52.329 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133843.log 2023-10-14T18:20:52.329 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.98790.log.gz 2023-10-14T18:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141630.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.141630.log.gz 2023-10-14T18:20:52.330 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.140473.log 2023-10-14T18:20:52.331 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133843.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.133843.log.gz 2023-10-14T18:20:52.332 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149205.log 2023-10-14T18:20:52.333 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.140473.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112542.log 2023-10-14T18:20:52.334 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.140473.log.gz 2023-10-14T18:20:52.335 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149205.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149205.log.gz 2023-10-14T18:20:52.335 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.138999.log 2023-10-14T18:20:52.336 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137383.log 2023-10-14T18:20:52.337 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.112542.log.gz 2023-10-14T18:20:52.338 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.138999.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.138999.log.gz 2023-10-14T18:20:52.338 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123031.log 2023-10-14T18:20:52.339 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137383.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.137383.log.gz 2023-10-14T18:20:52.340 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99870.log 2023-10-14T18:20:52.341 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95490.log 2023-10-14T18:20:52.341 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.123031.log.gz 2023-10-14T18:20:52.343 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99870.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.99870.log.gz 2023-10-14T18:20:52.343 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102738.log 2023-10-14T18:20:52.344 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95490.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95490.log.gz 2023-10-14T18:20:52.344 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110904.log 2023-10-14T18:20:52.345 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102738.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124904.log 2023-10-14T18:20:52.346 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.102738.log.gz 2023-10-14T18:20:52.347 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110904.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.110904.log.gz 2023-10-14T18:20:52.347 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143336.log 2023-10-14T18:20:52.348 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124904.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.124904.log.gz 2023-10-14T18:20:52.349 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155011.log 2023-10-14T18:20:52.350 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96021.log 2023-10-14T18:20:52.351 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155011.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.143336.log.gz 2023-10-14T18:20:52.351 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.155011.log.gz 2023-10-14T18:20:52.352 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129160.log 2023-10-14T18:20:52.353 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96021.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96021.log.gz 2023-10-14T18:20:52.353 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91353.log 2023-10-14T18:20:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129160.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.129160.log.gz 2023-10-14T18:20:52.355 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153023.log 2023-10-14T18:20:52.356 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93051.log 2023-10-14T18:20:52.356 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.91353.log.gz 2023-10-14T18:20:52.358 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153023.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.153023.log.gz 2023-10-14T18:20:52.358 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135380.log 2023-10-14T18:20:52.359 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93051.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159918.log 2023-10-14T18:20:52.360 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.93051.log.gz 2023-10-14T18:20:52.361 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118857.log 2023-10-14T18:20:52.361 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.135380.log.gz 2023-10-14T18:20:52.362 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111541.log 2023-10-14T18:20:52.362 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.159918.log.gz 2023-10-14T18:20:52.364 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118857.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135841.log 2023-10-14T18:20:52.364 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.118857.log.gz 2023-10-14T18:20:52.366 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103264.log 2023-10-14T18:20:52.366 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.111541.log.gz 2023-10-14T18:20:52.367 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.118587.log 2023-10-14T18:20:52.368 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.135841.log.gz 2023-10-14T18:20:52.369 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103264.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.103264.log.gz 2023-10-14T18:20:52.369 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2023-10-14T18:20:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.118587.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.118587.log.gz 2023-10-14T18:20:52.370 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147237.log 2023-10-14T18:20:52.372 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128007.log 2023-10-14T18:20:52.373 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147237.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137425.log 2023-10-14T18:20:52.374 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.147237.log.gz 2023-10-14T18:20:52.375 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128007.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105933.log 2023-10-14T18:20:52.375 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.128007.log.gz 2023-10-14T18:20:52.376 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137425.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137425.log.gz 2023-10-14T18:20:52.377 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138452.log 2023-10-14T18:20:52.378 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150353.log 2023-10-14T18:20:52.378 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.105933.log.gz 2023-10-14T18:20:52.379 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106227.log 2023-10-14T18:20:52.380 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138452.log.gz 2023-10-14T18:20:52.381 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150353.log: gzip 88.2% -- replaced with /var/log/ceph/ceph-client.admin.150353.log.gz -5 2023-10-14T18:20:52.381 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.138410.log 2023-10-14T18:20:52.382 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123672.log 2023-10-14T18:20:52.383 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106227.log.gz 2023-10-14T18:20:52.384 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138410.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124129.log 2023-10-14T18:20:52.384 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.138410.log.gz 2023-10-14T18:20:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105848.log 2023-10-14T18:20:52.386 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.123672.log.gz 2023-10-14T18:20:52.386 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124129.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146451.log 2023-10-14T18:20:52.387 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124129.log.gz 2023-10-14T18:20:52.388 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156774.log 2023-10-14T18:20:52.388 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.105848.log.gz 2023-10-14T18:20:52.390 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111329.log 2023-10-14T18:20:52.390 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.146451.log.gz 2023-10-14T18:20:52.391 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.158721.log 2023-10-14T18:20:52.391 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.156774.log.gz 2023-10-14T18:20:52.393 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.143589.log 2023-10-14T18:20:52.393 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.111329.log.gz 2023-10-14T18:20:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.158721.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.158721.log.gz 2023-10-14T18:20:52.394 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124818.log 2023-10-14T18:20:52.395 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.143589.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.143589.log.gz 2023-10-14T18:20:52.396 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107597.log 2023-10-14T18:20:52.397 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124818.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98832.log 2023-10-14T18:20:52.397 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.124818.log.gz 2023-10-14T18:20:52.399 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135126.log 2023-10-14T18:20:52.399 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.107597.log.gz 2023-10-14T18:20:52.400 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133518.log 2023-10-14T18:20:52.400 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.98832.log.gz 2023-10-14T18:20:52.402 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137340.log 2023-10-14T18:20:52.402 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.135126.log.gz 2023-10-14T18:20:52.403 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133518.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148239.log 2023-10-14T18:20:52.403 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.133518.log.gz 2023-10-14T18:20:52.405 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119649.log 2023-10-14T18:20:52.405 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.137340.log.gz 2023-10-14T18:20:52.406 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148239.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.148239.log.gz 2023-10-14T18:20:52.406 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.98382.log 2023-10-14T18:20:52.407 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119649.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110045.log 2023-10-14T18:20:52.408 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.119649.log.gz 2023-10-14T18:20:52.409 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105609.log 2023-10-14T18:20:52.409 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.98382.log.gz 2023-10-14T18:20:52.411 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110045.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126786.log 2023-10-14T18:20:52.411 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110045.log.gz 2023-10-14T18:20:52.412 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155461.log 2023-10-14T18:20:52.412 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.105609.log.gz 2023-10-14T18:20:52.413 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126786.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150058.log 2023-10-14T18:20:52.414 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.126786.log.gz 2023-10-14T18:20:52.415 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101917.log 2023-10-14T18:20:52.415 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.155461.log.gz 2023-10-14T18:20:52.417 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150058.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115745.log 2023-10-14T18:20:52.417 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.150058.log.gz 2023-10-14T18:20:52.418 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116583.log 2023-10-14T18:20:52.418 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101917.log.gz 2023-10-14T18:20:52.420 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110802.log 2023-10-14T18:20:52.420 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115745.log.gz 2023-10-14T18:20:52.421 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116583.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147975.log 2023-10-14T18:20:52.421 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.116583.log.gz 2023-10-14T18:20:52.422 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125616.log 2023-10-14T18:20:52.422 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.110802.log.gz 2023-10-14T18:20:52.424 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152724.log 2023-10-14T18:20:52.424 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.147975.log.gz 2023-10-14T18:20:52.425 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120414.log 2023-10-14T18:20:52.425 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.125616.log.gz 2023-10-14T18:20:52.427 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92717.log 2023-10-14T18:20:52.427 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.152724.log.gz 2023-10-14T18:20:52.428 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120414.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128884.log 2023-10-14T18:20:52.428 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120414.log.gz 2023-10-14T18:20:52.430 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92717.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159489.log 2023-10-14T18:20:52.430 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.92717.log.gz 2023-10-14T18:20:52.431 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157068.log 2023-10-14T18:20:52.431 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.128884.log.gz 2023-10-14T18:20:52.433 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159489.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.159489.log.gz 2023-10-14T18:20:52.433 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134598.log 2023-10-14T18:20:52.434 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157068.log: gzip -5 --verbose -- 87.5% /var/log/ceph/ceph-client.admin.138018.log 2023-10-14T18:20:52.434 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.157068.log.gz 2023-10-14T18:20:52.436 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134598.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113170.log 2023-10-14T18:20:52.436 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.134598.log.gz 2023-10-14T18:20:52.438 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138018.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100428.log 2023-10-14T18:20:52.438 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.138018.log.gz 2023-10-14T18:20:52.439 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151271.log 2023-10-14T18:20:52.439 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.113170.log.gz 2023-10-14T18:20:52.440 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100428.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148145.log 2023-10-14T18:20:52.440 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.100428.log.gz 2023-10-14T18:20:52.442 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128348.log 2023-10-14T18:20:52.443 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.151271.log.gz 2023-10-14T18:20:52.443 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152514.log 2023-10-14T18:20:52.443 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148145.log.gz 2023-10-14T18:20:52.445 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128348.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149335.log 2023-10-14T18:20:52.445 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128348.log.gz 2023-10-14T18:20:52.446 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152514.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107052.log 2023-10-14T18:20:52.447 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.152514.log.gz 2023-10-14T18:20:52.447 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149335.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112123.log 2023-10-14T18:20:52.448 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.149335.log.gz 2023-10-14T18:20:52.449 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107052.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136050.log 2023-10-14T18:20:52.450 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.107052.log.gz 2023-10-14T18:20:52.451 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112123.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117159.log 2023-10-14T18:20:52.451 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112123.log.gz 2023-10-14T18:20:52.452 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112908.log 2023-10-14T18:20:52.452 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.136050.log.gz 2023-10-14T18:20:52.453 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153235.log 2023-10-14T18:20:52.454 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.117159.log.gz 2023-10-14T18:20:52.455 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115075.log 2023-10-14T18:20:52.455 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.112908.log.gz 2023-10-14T18:20:52.456 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127290.log 2023-10-14T18:20:52.456 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153235.log.gz 2023-10-14T18:20:52.458 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115075.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143126.log 2023-10-14T18:20:52.458 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.115075.log.gz 2023-10-14T18:20:52.459 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116762.log 2023-10-14T18:20:52.459 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.127290.log.gz 2023-10-14T18:20:52.460 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143126.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112789.log 2023-10-14T18:20:52.461 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.143126.log.gz 2023-10-14T18:20:52.462 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158254.log 2023-10-14T18:20:52.462 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.116762.log.gz 2023-10-14T18:20:52.464 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93819.log 2023-10-14T18:20:52.464 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.112789.log.gz 2023-10-14T18:20:52.465 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158254.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133885.log 2023-10-14T18:20:52.465 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.158254.log.gz 2023-10-14T18:20:52.466 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102195.log 2023-10-14T18:20:52.467 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.93819.log.gz 2023-10-14T18:20:52.468 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133885.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153699.log 2023-10-14T18:20:52.468 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.133885.log.gz 2023-10-14T18:20:52.469 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102195.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128842.log 2023-10-14T18:20:52.470 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.102195.log.gz 2023-10-14T18:20:52.471 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153489.log 2023-10-14T18:20:52.471 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.153699.log.gz 2023-10-14T18:20:52.472 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143210.log 2023-10-14T18:20:52.472 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128842.log.gz 2023-10-14T18:20:52.474 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124027.log 2023-10-14T18:20:52.474 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.153489.log.gz 2023-10-14T18:20:52.475 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143210.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.132041.log 2023-10-14T18:20:52.476 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.143210.log.gz 2023-10-14T18:20:52.476 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124027.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113880.log 2023-10-14T18:20:52.477 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.124027.log.gz 2023-10-14T18:20:52.478 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.132041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159798.log 2023-10-14T18:20:52.478 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.132041.log.gz 2023-10-14T18:20:52.480 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121067.log 2023-10-14T18:20:52.480 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.113880.log.gz 2023-10-14T18:20:52.481 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99990.log 2023-10-14T18:20:52.481 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.159798.log.gz 2023-10-14T18:20:52.482 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121067.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148101.log 2023-10-14T18:20:52.483 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.121067.log.gz 2023-10-14T18:20:52.484 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100213.log 2023-10-14T18:20:52.484 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.99990.log.gz 2023-10-14T18:20:52.486 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148101.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111030.log 2023-10-14T18:20:52.486 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.148101.log.gz 2023-10-14T18:20:52.487 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100213.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.104071.log 2023-10-14T18:20:52.487 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.100213.log.gz 2023-10-14T18:20:52.488 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114641.log 2023-10-14T18:20:52.489 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111030.log.gz 2023-10-14T18:20:52.490 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.104071.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138230.log 2023-10-14T18:20:52.490 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.104071.log.gz 2023-10-14T18:20:52.491 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152285.log 2023-10-14T18:20:52.491 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114641.log.gz 2023-10-14T18:20:52.493 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98580.log 2023-10-14T18:20:52.493 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.138230.log.gz 2023-10-14T18:20:52.494 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127698.log 2023-10-14T18:20:52.494 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.152285.log.gz 2023-10-14T18:20:52.495 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122989.log 2023-10-14T18:20:52.496 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.98580.log.gz 2023-10-14T18:20:52.497 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127698.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155818.log 2023-10-14T18:20:52.498 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.127698.log.gz 2023-10-14T18:20:52.499 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122989.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96381.log 2023-10-14T18:20:52.499 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.122989.log.gz 2023-10-14T18:20:52.500 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155818.log: gzip 88.1% -- replaced with /var/log/ceph/ceph-client.admin.155818.log.gz -5 2023-10-14T18:20:52.500 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.149079.log 2023-10-14T18:20:52.501 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.96381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125724.log 2023-10-14T18:20:52.502 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.96381.log.gz 2023-10-14T18:20:52.503 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124255.log 2023-10-14T18:20:52.503 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.149079.log.gz 2023-10-14T18:20:52.505 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125724.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102475.log 2023-10-14T18:20:52.505 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.125724.log.gz 2023-10-14T18:20:52.506 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.124255.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.124255.log.gz 2023-10-14T18:20:52.506 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.120246.log 2023-10-14T18:20:52.507 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.102475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146494.log 2023-10-14T18:20:52.508 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.102475.log.gz 2023-10-14T18:20:52.509 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120246.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2023-10-14T18:20:52.509 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.120246.log.gz 2023-10-14T18:20:52.510 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.146494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147933.log 2023-10-14T18:20:52.511 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.146494.log.gz 2023-10-14T18:20:52.512 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134125.log 2023-10-14T18:20:52.514 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147933.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151618.log 2023-10-14T18:20:52.514 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147933.log.gz 2023-10-14T18:20:52.515 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148727.log 2023-10-14T18:20:52.515 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.134125.log.gz 2023-10-14T18:20:52.516 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.141041.log 2023-10-14T18:20:52.517 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151618.log.gz 2023-10-14T18:20:52.518 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148727.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95364.log 2023-10-14T18:20:52.518 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.148727.log.gz 2023-10-14T18:20:52.519 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.141041.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154159.log 2023-10-14T18:20:52.520 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.141041.log.gz 2023-10-14T18:20:52.521 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95364.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127100.log 2023-10-14T18:20:52.521 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.95364.log.gz 2023-10-14T18:20:52.522 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.135945.log 2023-10-14T18:20:52.522 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.154159.log.gz 2023-10-14T18:20:52.524 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127100.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149783.log 2023-10-14T18:20:52.524 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.127100.log.gz 2023-10-14T18:20:52.525 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.135945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107979.log 2023-10-14T18:20:52.526 INFO:teuthology.orchestra.run.smithi029.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.135945.log.gz 2023-10-14T18:20:52.527 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149783.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.149783.log.gz 2023-10-14T18:20:52.528 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122947.log 2023-10-14T18:20:52.528 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107979.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.107979.log.gz 2023-10-14T18:20:52.529 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117994.log 2023-10-14T18:20:52.531 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134022.log 2023-10-14T18:20:52.531 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.122947.log.gz 2023-10-14T18:20:52.532 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117994.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123115.log 2023-10-14T18:20:52.533 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.117994.log.gz 2023-10-14T18:20:52.533 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98748.log 2023-10-14T18:20:52.534 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.134022.log.gz 2023-10-14T18:20:52.535 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128604.log 2023-10-14T18:20:52.535 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.123115.log.gz 2023-10-14T18:20:52.536 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110946.log 2023-10-14T18:20:52.536 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.98748.log.gz 2023-10-14T18:20:52.538 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114685.log 2023-10-14T18:20:52.538 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.128604.log.gz 2023-10-14T18:20:52.539 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110946.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119880.log 2023-10-14T18:20:52.539 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110946.log.gz 2023-10-14T18:20:52.541 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120817.log 2023-10-14T18:20:52.541 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114685.log.gz 2023-10-14T18:20:52.542 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.119880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133570.log 2023-10-14T18:20:52.542 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.119880.log.gz 2023-10-14T18:20:52.544 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153657.log 2023-10-14T18:20:52.544 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120817.log.gz 2023-10-14T18:20:52.545 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133570.log: 87.5%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125783.log 2023-10-14T18:20:52.545 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.133570.log.gz 2023-10-14T18:20:52.547 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153657.log: gzip -5 --verbose -- 89.5% -- replaced with /var/log/ceph/ceph-client.admin.153657.log.gz 2023-10-14T18:20:52.547 INFO:teuthology.orchestra.run.smithi029.stderr: /var/log/ceph/ceph-client.admin.136334.log 2023-10-14T18:20:52.548 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125783.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.125783.log.gz 2023-10-14T18:20:52.549 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137593.log 2023-10-14T18:20:52.550 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136334.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92374.log 2023-10-14T18:20:52.550 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.136334.log.gz 2023-10-14T18:20:52.552 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97822.log 2023-10-14T18:20:52.552 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.137593.log.gz 2023-10-14T18:20:52.553 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92374.log: gzip 87.6% -- replaced with /var/log/ceph/ceph-client.admin.92374.log.gz 2023-10-14T18:20:52.553 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.153279.log 2023-10-14T18:20:52.554 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97822.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.97822.log.gz 2023-10-14T18:20:52.555 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154685.log 2023-10-14T18:20:52.556 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153279.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.153279.log.gz 2023-10-14T18:20:52.557 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93059.log 2023-10-14T18:20:52.558 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154685.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154769.log 2023-10-14T18:20:52.558 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.154685.log.gz 2023-10-14T18:20:52.559 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97576.log 2023-10-14T18:20:52.560 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.93059.log.gz 2023-10-14T18:20:52.560 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154769.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.154769.log.gz 2023-10-14T18:20:52.561 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.137551.log 2023-10-14T18:20:52.562 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97576.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.97576.log.gz 2023-10-14T18:20:52.563 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156336.log 2023-10-14T18:20:52.564 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.137551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120983.log 2023-10-14T18:20:52.565 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.137551.log.gz 2023-10-14T18:20:52.566 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156336.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.103790.log 2023-10-14T18:20:52.566 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.156336.log.gz 2023-10-14T18:20:52.567 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120983.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157664.log 2023-10-14T18:20:52.567 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.120983.log.gz 2023-10-14T18:20:52.568 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.103790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149891.log 2023-10-14T18:20:52.569 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.103790.log.gz 2023-10-14T18:20:52.570 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156294.log 2023-10-14T18:20:52.570 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.157664.log.gz 2023-10-14T18:20:52.571 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149891.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156073.log 2023-10-14T18:20:52.572 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.149891.log.gz 2023-10-14T18:20:52.573 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156294.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120372.log 2023-10-14T18:20:52.573 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156294.log.gz 2023-10-14T18:20:52.574 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133381.log 2023-10-14T18:20:52.575 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.156073.log.gz 2023-10-14T18:20:52.576 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120372.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143084.log 2023-10-14T18:20:52.576 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.120372.log.gz 2023-10-14T18:20:52.577 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99250.log 2023-10-14T18:20:52.578 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.133381.log.gz 2023-10-14T18:20:52.579 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154327.log 2023-10-14T18:20:52.579 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.143084.log.gz 2023-10-14T18:20:52.580 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99250.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97864.log 2023-10-14T18:20:52.581 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.99250.log.gz 2023-10-14T18:20:52.582 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154327.log: 88.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.154327.log.gz 2023-10-14T18:20:52.582 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.97389.log 2023-10-14T18:20:52.584 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112057.log 2023-10-14T18:20:52.584 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97864.log.gz 2023-10-14T18:20:52.585 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.97389.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117033.log 2023-10-14T18:20:52.585 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97389.log.gz 2023-10-14T18:20:52.587 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112057.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111973.log 2023-10-14T18:20:52.587 INFO:teuthology.orchestra.run.smithi029.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.112057.log.gz 2023-10-14T18:20:52.588 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117033.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.117033.log.gz 2023-10-14T18:20:52.588 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2023-10-14T18:20:52.589 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111973.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156546.log 2023-10-14T18:20:52.590 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111973.log.gz 2023-10-14T18:20:52.591 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110760.log 2023-10-14T18:20:52.593 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156546.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.156546.log.gz 2023-10-14T18:20:52.594 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.110760.log: /var/log/ceph/ceph-client.admin.101311.log 2023-10-14T18:20:52.594 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110760.log.gz 2023-10-14T18:20:52.595 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98493.log 2023-10-14T18:20:52.597 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101311.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.101311.log.gz 2023-10-14T18:20:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121248.log 2023-10-14T18:20:52.598 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98493.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.98493.log.gz 2023-10-14T18:20:52.599 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113922.log 2023-10-14T18:20:52.601 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121248.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.121248.log.gz 2023-10-14T18:20:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156588.log 2023-10-14T18:20:52.602 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113922.log: 89.1% -- replaced with /var/log/ceph/ceph.log.gz 2023-10-14T18:20:52.603 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.113922.log.gz 2023-10-14T18:20:52.603 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127143.log 2023-10-14T18:20:52.605 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123714.log 88.9% -- replaced with /var/log/ceph/ceph-client.admin.156588.log.gz 2023-10-14T18:20:52.605 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127143.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.127143.log.gz 2023-10-14T18:20:52.606 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2023-10-14T18:20:52.608 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100050.log 2023-10-14T18:20:52.608 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123714.log.gz 2023-10-14T18:20:52.610 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91257.log 2023-10-14T18:20:52.611 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100050.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.100050.log.gz 2023-10-14T18:20:52.612 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134503.log 2023-10-14T18:20:52.613 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91257.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.91257.log.gz 2023-10-14T18:20:52.614 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125114.log 2023-10-14T18:20:52.615 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134503.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.134503.log.gz 2023-10-14T18:20:52.616 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.136957.log 2023-10-14T18:20:52.617 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125114.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.125114.log.gz 2023-10-14T18:20:52.618 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107747.log 2023-10-14T18:20:52.619 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.136957.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.136957.log.gz 2023-10-14T18:20:52.620 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127458.log 2023-10-14T18:20:52.621 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.107747.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107747.log.gz 2023-10-14T18:20:52.622 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156462.log 2023-10-14T18:20:52.623 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127458.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127458.log.gz 2023-10-14T18:20:52.624 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94944.log 2023-10-14T18:20:52.625 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156462.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.156462.log.gz 2023-10-14T18:20:52.626 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126164.log 2023-10-14T18:20:52.627 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94944.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.94944.log.gz 2023-10-14T18:20:52.628 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2023-10-14T18:20:52.629 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126164.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.126164.log.gz 2023-10-14T18:20:52.630 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151805.log 2023-10-14T18:20:52.633 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-mds.c.log: /var/log/ceph/ceph-client.admin.151805.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.151805.log.gz 2023-10-14T18:20:52.634 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127511.log 2023-10-14T18:20:52.636 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100170.log 2023-10-14T18:20:52.637 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.127511.log.gz 2023-10-14T18:20:52.639 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106884.log 2023-10-14T18:20:52.641 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.100170.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.100170.log.gz 2023-10-14T18:20:52.644 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.107689.log 2023-10-14T18:20:52.645 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106884.log.gz 2023-10-14T18:20:52.648 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.107689.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.93507.log 2023-10-14T18:20:52.649 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.107689.log.gz 2023-10-14T18:20:52.650 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124946.log 2023-10-14T18:20:52.655 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93507.log: /var/log/ceph/ceph-client.admin.124946.log: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142080.log 2023-10-14T18:20:52.655 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93507.log.gz 2023-10-14T18:20:52.657 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- 89.1% /var/log/ceph/ceph-client.admin.117796.log 2023-10-14T18:20:52.657 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.142080.log: -- replaced with /var/log/ceph/ceph-client.admin.124946.log.gz 2023-10-14T18:20:52.660 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8%/var/log/ceph/ceph-client.admin.117796.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.117796.log.gz 2023-10-14T18:20:52.661 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116667.log 2023-10-14T18:20:52.661 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.142080.log.gz 2023-10-14T18:20:52.664 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116667.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126507.log 2023-10-14T18:20:52.665 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.116667.log.gz 2023-10-14T18:20:52.667 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138494.log 2023-10-14T18:20:52.671 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126507.log: /var/log/ceph/ceph-client.admin.138494.log: 86.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126912.log 2023-10-14T18:20:52.671 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.126507.log.gz 2023-10-14T18:20:52.673 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.138494.log.gz 2023-10-14T18:20:52.674 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126912.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.126912.log.gz 2023-10-14T18:20:52.675 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160952.log 2023-10-14T18:20:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160952.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.160952.log.gz 2023-10-14T18:20:52.679 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91157.log 2023-10-14T18:20:52.682 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91157.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121376.log 2023-10-14T18:20:52.683 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.91157.log.gz 2023-10-14T18:20:52.685 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154559.log 2023-10-14T18:20:52.690 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121376.log: /var/log/ceph/ceph-client.admin.154559.log: 88.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110718.log 2023-10-14T18:20:52.690 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.121376.log.gz 2023-10-14T18:20:52.691 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.154559.log.gz 2023-10-14T18:20:52.694 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.110718.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.101479.log 2023-10-14T18:20:52.698 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101479.log: 88.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.110718.log.gz 2023-10-14T18:20:52.698 INFO:teuthology.orchestra.run.smithi029.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.92723.log 2023-10-14T18:20:52.700 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.101479.log.gz 2023-10-14T18:20:52.701 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110329.log 2023-10-14T18:20:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110329.log: /var/log/ceph/ceph-client.admin.92723.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.92723.log.gz 2023-10-14T18:20:52.706 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142206.log 2023-10-14T18:20:52.708 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.110329.log.gz 2023-10-14T18:20:52.708 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.144053.log 2023-10-14T18:20:52.712 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.144053.log: /var/log/ceph/ceph-client.admin.142206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98664.log 2023-10-14T18:20:52.714 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.142206.log.gz 2023-10-14T18:20:52.715 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.144053.log.gz 2023-10-14T18:20:52.715 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151058.log 2023-10-14T18:20:52.719 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151058.log: /var/log/ceph/ceph-client.admin.98664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92079.log 2023-10-14T18:20:52.721 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.98664.log.gz 2023-10-14T18:20:52.722 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.151058.log.gz 2023-10-14T18:20:52.723 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159751.log 2023-10-14T18:20:52.727 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159751.log: /var/log/ceph/ceph-client.admin.92079.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106568.log 2023-10-14T18:20:52.729 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.92079.log.gz 2023-10-14T18:20:52.729 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.159751.log.gz 2023-10-14T18:20:52.730 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114865.log 2023-10-14T18:20:52.733 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114865.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.114865.log.gz 2023-10-14T18:20:52.735 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106568.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.106568.log.gz 2023-10-14T18:20:52.736 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158798.log 2023-10-14T18:20:52.739 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154117.log 2023-10-14T18:20:52.740 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158798.log.gz 2023-10-14T18:20:52.741 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99292.log 2023-10-14T18:20:52.743 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.133612.log 2023-10-14T18:20:52.745 INFO:teuthology.orchestra.run.smithi029.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.154117.log.gz 2023-10-14T18:20:52.746 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.159025.log 2023-10-14T18:20:52.749 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.133612.log: /var/log/ceph/ceph-client.admin.99292.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.99292.log.gz 2023-10-14T18:20:52.750 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120059.log 2023-10-14T18:20:52.751 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.159025.log: -- replaced with /var/log/ceph/ceph-client.admin.133612.log.gz 2023-10-14T18:20:52.752 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.159025.log.gz 2023-10-14T18:20:52.754 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120059.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.94041.log 2023-10-14T18:20:52.757 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.94041.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.120059.log.gz 2023-10-14T18:20:52.757 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148189.log 2023-10-14T18:20:52.759 INFO:teuthology.orchestra.run.smithi029.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.94041.log.gz 2023-10-14T18:20:52.759 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118036.log 2023-10-14T18:20:52.762 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118036.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.118036.log.gz 2023-10-14T18:20:52.762 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148577.log 2023-10-14T18:20:52.765 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148189.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.148189.log.gz 2023-10-14T18:20:52.767 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92703.log 2023-10-14T18:20:52.771 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92703.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.148577.log.gz 2023-10-14T18:20:52.771 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124638.log 2023-10-14T18:20:52.773 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.92703.log.gz 2023-10-14T18:20:52.774 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106364.log 2023-10-14T18:20:52.778 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106364.log: /var/log/ceph/ceph-client.admin.124638.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111889.log 2023-10-14T18:20:52.779 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.124638.log.gz 2023-10-14T18:20:52.780 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106364.log.gz 2023-10-14T18:20:52.782 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120456.log 2023-10-14T18:20:52.787 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120456.log: /var/log/ceph/ceph-client.admin.111889.log: gzip 87.1% -5 --verbose -- /var/log/ceph/ceph-client.admin.142427.log 2023-10-14T18:20:52.787 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111889.log.gz 2023-10-14T18:20:52.788 INFO:teuthology.orchestra.run.smithi029.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.120456.log.gz 2023-10-14T18:20:52.790 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115201.log 2023-10-14T18:20:52.793 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115201.log: /var/log/ceph/ceph-client.admin.142427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155545.log 2023-10-14T18:20:52.795 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% 91.0% -- replaced with /var/log/ceph/ceph-client.admin.142427.log.gz 2023-10-14T18:20:52.796 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.115201.log.gz 2023-10-14T18:20:52.797 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158296.log 2023-10-14T18:20:52.801 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158296.log: /var/log/ceph/ceph-client.admin.155545.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.154811.log 2023-10-14T18:20:52.803 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% 88.4% -- replaced with /var/log/ceph/ceph-client.admin.155545.log.gz 2023-10-14T18:20:52.804 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.158296.log.gz 2023-10-14T18:20:52.804 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139306.log 2023-10-14T18:20:52.807 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93381.log 2023-10-14T18:20:52.808 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.154811.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.154811.log.gz 2023-10-14T18:20:52.809 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.139306.log.gz 2023-10-14T18:20:52.810 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111721.log 2023-10-14T18:20:52.814 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111721.log: /var/log/ceph/ceph-client.admin.93381.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115577.log 2023-10-14T18:20:52.816 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.93381.log.gz 2023-10-14T18:20:52.817 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.111721.log.gz 2023-10-14T18:20:52.818 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95406.log 2023-10-14T18:20:52.821 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95406.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.95406.log.gz 2023-10-14T18:20:52.823 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115577.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.115577.log.gz 2023-10-14T18:20:52.824 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.157584.log 2023-10-14T18:20:52.827 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.157584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134946.log 2023-10-14T18:20:52.828 INFO:teuthology.orchestra.run.smithi029.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.157584.log.gz 2023-10-14T18:20:52.830 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158530.log 2023-10-14T18:20:52.835 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134946.log: /var/log/ceph/ceph-client.admin.158530.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.158530.log.gz 2023-10-14T18:20:52.835 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97053.log 2023-10-14T18:20:52.838 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6%/var/log/ceph/ceph-client.admin.97053.log: -- replaced with /var/log/ceph/ceph-client.admin.134946.log.gz 2023-10-14T18:20:52.839 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129010.log 2023-10-14T18:20:52.840 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97053.log.gz 2023-10-14T18:20:52.841 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95910.log 2023-10-14T18:20:52.846 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129010.log: /var/log/ceph/ceph-client.admin.95910.log: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96555.log 2023-10-14T18:20:52.847 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.129010.log.gz 2023-10-14T18:20:52.848 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.95910.log.gz 2023-10-14T18:20:52.851 INFO:teuthology.orchestra.run.smithi029.stderr:gzip/var/log/ceph/ceph-client.admin.96555.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.123209.log 2023-10-14T18:20:52.854 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123209.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.96555.log.gz 2023-10-14T18:20:52.855 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134167.log 2023-10-14T18:20:52.857 INFO:teuthology.orchestra.run.smithi029.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118162.log 2023-10-14T18:20:52.858 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.123209.log.gz 2023-10-14T18:20:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134167.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.134167.log.gz 2023-10-14T18:20:52.859 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.139181.log 2023-10-14T18:20:52.861 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118162.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.118162.log.gz 2023-10-14T18:20:52.863 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.139181.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153065.log 2023-10-14T18:20:52.865 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126954.log 2023-10-14T18:20:52.865 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.139181.log.gz 2023-10-14T18:20:52.867 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.153065.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.153065.log.gz 2023-10-14T18:20:52.871 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126954.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.126954.log.gz 2023-10-14T18:20:52.872 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161402.log 2023-10-14T18:20:52.874 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.161402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.129347.log 2023-10-14T18:20:52.876 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.161402.log.gz 2023-10-14T18:20:52.876 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114515.log 2023-10-14T18:20:52.879 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.129347.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.129347.log.gz 2023-10-14T18:20:52.881 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114515.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.114515.log.gz 2023-10-14T18:20:52.882 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147177.log 2023-10-14T18:20:52.884 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.155587.log 2023-10-14T18:20:52.886 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.147177.log.gz 2023-10-14T18:20:52.888 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150256.log 2023-10-14T18:20:52.891 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.155587.log: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120941.log 2023-10-14T18:20:52.892 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150256.log: -- replaced with /var/log/ceph/ceph-client.admin.155587.log.gz 2023-10-14T18:20:52.894 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.150256.log.gz 2023-10-14T18:20:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120941.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120941.log.gz 2023-10-14T18:20:52.896 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.126224.log 2023-10-14T18:20:52.899 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.126224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148367.log 2023-10-14T18:20:52.900 INFO:teuthology.orchestra.run.smithi029.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.126224.log.gz 2023-10-14T18:20:52.902 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148535.log 2023-10-14T18:20:52.906 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148367.log: /var/log/ceph/ceph-client.admin.148535.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138578.log 2023-10-14T18:20:52.907 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.148367.log.gz 2023-10-14T18:20:52.908 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148535.log.gz 2023-10-14T18:20:52.910 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.120330.log 2023-10-14T18:20:52.913 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.120330.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138578.log.gz 2023-10-14T18:20:52.914 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147406.log 2023-10-14T18:20:52.915 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.120330.log.gz 2023-10-14T18:20:52.917 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114132.log 2023-10-14T18:20:52.920 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114132.log: /var/log/ceph/ceph-client.admin.147406.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95154.log 2023-10-14T18:20:52.922 INFO:teuthology.orchestra.run.smithi029.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.147406.log.gz 2023-10-14T18:20:52.923 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.114132.log.gz 2023-10-14T18:20:52.924 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114048.log 2023-10-14T18:20:52.928 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.114048.log: /var/log/ceph/ceph-client.admin.95154.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.119922.log 2023-10-14T18:20:52.930 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.95154.log.gz 87.6% 2023-10-14T18:20:52.930 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.114048.log.gz 2023-10-14T18:20:52.932 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117243.log 2023-10-14T18:20:52.936 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117243.log: /var/log/ceph/ceph-client.admin.119922.log: 89.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101353.log 2023-10-14T18:20:52.936 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.119922.log.gz 2023-10-14T18:20:52.937 INFO:teuthology.orchestra.run.smithi029.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.117243.log.gz 2023-10-14T18:20:52.939 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93591.log 2023-10-14T18:20:52.943 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93591.log: /var/log/ceph/ceph-client.admin.101353.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92384.log 2023-10-14T18:20:52.945 INFO:teuthology.orchestra.run.smithi029.stderr: 88.4% 89.0% -- replaced with /var/log/ceph/ceph-client.admin.101353.log.gz 2023-10-14T18:20:52.945 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93591.log.gz 2023-10-14T18:20:52.946 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.122171.log 2023-10-14T18:20:52.947 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92384.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.92384.log.gz 2023-10-14T18:20:52.949 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151492.log 2023-10-14T18:20:52.951 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.122171.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.122171.log.gz 2023-10-14T18:20:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151492.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.151492.log.gz 2023-10-14T18:20:52.953 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158002.log 2023-10-14T18:20:52.956 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158002.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.128266.log 2023-10-14T18:20:52.957 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.158002.log.gz 2023-10-14T18:20:52.959 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161233.log 2023-10-14T18:20:52.963 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.128266.log: /var/log/ceph/ceph-client.admin.161233.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.161233.log.gz 2023-10-14T18:20:52.964 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111805.log 2023-10-14T18:20:52.965 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.128266.log.gz 2023-10-14T18:20:52.967 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.111805.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138186.log 2023-10-14T18:20:52.968 INFO:teuthology.orchestra.run.smithi029.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.111805.log.gz 2023-10-14T18:20:52.970 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.142715.log 2023-10-14T18:20:52.974 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138186.log: /var/log/ceph/ceph-client.admin.142715.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.138186.log.gz 2023-10-14T18:20:52.975 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.161166.log 2023-10-14T18:20:52.976 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.142715.log.gz 2023-10-14T18:20:52.979 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5/var/log/ceph/ceph-client.admin.161166.log: --verbose -- /var/log/ceph/ceph-client.admin.105788.log 2023-10-14T18:20:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105788.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.161166.log.gz 2023-10-14T18:20:52.983 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149247.log 2023-10-14T18:20:52.984 INFO:teuthology.orchestra.run.smithi029.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105788.log.gz 2023-10-14T18:20:52.986 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156732.log 2023-10-14T18:20:52.990 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156732.log: /var/log/ceph/ceph-client.admin.149247.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.114949.log 2023-10-14T18:20:52.991 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.149247.log.gz 2023-10-14T18:20:52.992 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.156732.log.gz 2023-10-14T18:20:52.993 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92722.log 2023-10-14T18:20:52.997 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.92722.log: /var/log/ceph/ceph-client.admin.114949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106143.log 2023-10-14T18:20:52.999 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% 88.5% -- replaced with /var/log/ceph/ceph-client.admin.114949.log.gz 2023-10-14T18:20:52.999 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.92722.log.gz 2023-10-14T18:20:53.000 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148619.log 2023-10-14T18:20:53.004 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.148619.log: /var/log/ceph/ceph-client.admin.106143.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113795.log 2023-10-14T18:20:53.007 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106143.log.gz 2023-10-14T18:20:53.007 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113795.log: -- replaced with /var/log/ceph/ceph-client.admin.148619.log.gz 2023-10-14T18:20:53.008 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.129811.log 2023-10-14T18:20:53.010 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116541.log 2023-10-14T18:20:53.011 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.129811.log: -- replaced with /var/log/ceph/ceph-client.admin.113795.log.gz 2023-10-14T18:20:53.013 INFO:teuthology.orchestra.run.smithi029.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.129811.log.gz 2023-10-14T18:20:53.014 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112392.log 2023-10-14T18:20:53.018 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112392.log: /var/log/ceph/ceph-client.admin.116541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.146367.log 2023-10-14T18:20:53.020 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0% 87.4% -- replaced with /var/log/ceph/ceph-client.admin.116541.log.gz 2023-10-14T18:20:53.020 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.112392.log.gz 2023-10-14T18:20:53.022 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.93339.log 2023-10-14T18:20:53.025 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.93339.log: /var/log/ceph/ceph-client.admin.146367.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.121881.log 2023-10-14T18:20:53.028 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% 87.9% -- replaced with /var/log/ceph/ceph-client.admin.146367.log.gz 2023-10-14T18:20:53.028 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.93339.log.gz 2023-10-14T18:20:53.029 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110413.log 2023-10-14T18:20:53.033 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.110413.log: /var/log/ceph/ceph-client.0.121881.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138536.log 2023-10-14T18:20:53.034 INFO:teuthology.orchestra.run.smithi029.stderr: 0.0% -- replaced with /var/log/ceph/ceph-client.0.121881.log.gz 2023-10-14T18:20:53.035 INFO:teuthology.orchestra.run.smithi029.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.110413.log.gz 2023-10-14T18:20:53.037 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.160299.log 2023-10-14T18:20:53.040 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.160299.log: /var/log/ceph/ceph-client.admin.138536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.148059.log 2023-10-14T18:20:53.042 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.138536.log.gz 2023-10-14T18:20:53.043 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.160299.log.gz 2023-10-14T18:20:53.044 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113283.log 2023-10-14T18:20:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113283.log: /var/log/ceph/ceph-client.admin.148059.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.148059.log.gz 2023-10-14T18:20:53.049 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118707.log 2023-10-14T18:20:53.051 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105975.log 2023-10-14T18:20:53.052 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118707.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.113283.log.gz 2023-10-14T18:20:53.052 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.118707.log.gz 2023-10-14T18:20:53.054 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.105975.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.105975.log.gz 2023-10-14T18:20:53.055 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99737.log 2023-10-14T18:20:53.057 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.99737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.150869.log 2023-10-14T18:20:53.059 INFO:teuthology.orchestra.run.smithi029.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.99737.log.gz 2023-10-14T18:20:53.060 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.147093.log 2023-10-14T18:20:53.062 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.150869.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.150869.log.gz 2023-10-14T18:20:53.065 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.147093.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.147093.log.gz 2023-10-14T18:20:53.066 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112015.log 2023-10-14T18:20:53.068 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112015.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101521.log 2023-10-14T18:20:53.070 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.112015.log.gz 2023-10-14T18:20:53.071 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95280.log 2023-10-14T18:20:53.073 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101521.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.101521.log.gz 2023-10-14T18:20:53.076 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95280.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95280.log.gz 2023-10-14T18:20:53.077 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117840.log 2023-10-14T18:20:53.080 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.138366.log 2023-10-14T18:20:53.081 INFO:teuthology.orchestra.run.smithi029.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.117840.log.gz 2023-10-14T18:20:53.082 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101799.log 2023-10-14T18:20:53.087 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.138366.log: 89.1%/var/log/ceph/ceph-client.admin.101799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156858.log 2023-10-14T18:20:53.088 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.138366.log.gz 2023-10-14T18:20:53.089 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.101799.log.gz 2023-10-14T18:20:53.091 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152556.log 2023-10-14T18:20:53.095 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156858.log: /var/log/ceph/ceph-client.admin.152556.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.156858.log.gz 2023-10-14T18:20:53.096 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134678.log 2023-10-14T18:20:53.097 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.124213.log 2023-10-14T18:20:53.099 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134678.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.152556.log.gz 2023-10-14T18:20:53.100 INFO:teuthology.orchestra.run.smithi029.stderr: 88.0%/var/log/ceph/ceph-client.admin.124213.log: -- replaced with /var/log/ceph/ceph-client.admin.134678.log.gz 2023-10-14T18:20:53.100 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123304.log 2023-10-14T18:20:53.101 INFO:teuthology.orchestra.run.smithi029.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.124213.log.gz 2023-10-14T18:20:53.103 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98537.log 2023-10-14T18:20:53.104 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123304.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123304.log.gz 2023-10-14T18:20:53.106 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98537.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.98537.log.gz 2023-10-14T18:20:53.107 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123901.log 2023-10-14T18:20:53.110 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123901.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115117.log 2023-10-14T18:20:53.113 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115117.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.123901.log.gz 2023-10-14T18:20:53.114 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106782.log 2023-10-14T18:20:53.117 INFO:teuthology.orchestra.run.smithi029.stderr:gzip 88.6% -5 --verbose -- /var/log/ceph/ceph-client.0.99169.log 2023-10-14T18:20:53.118 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106782.log: -- replaced with /var/log/ceph/ceph-client.admin.115117.log.gz 2023-10-14T18:20:53.120 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.106782.log.gz 2023-10-14T18:20:53.121 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.153191.log 2023-10-14T18:20:53.125 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.99169.log: /var/log/ceph/ceph-client.admin.153191.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.99169.log.gz 2023-10-14T18:20:53.126 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125198.log 2023-10-14T18:20:53.129 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125198.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.153191.log.gz 2023-10-14T18:20:53.130 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118760.log 2023-10-14T18:20:53.134 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118760.log: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.134293.log 2023-10-14T18:20:53.136 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.134293.log: -- replaced with /var/log/ceph/ceph-client.admin.125198.log.gz 2023-10-14T18:20:53.137 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 88.0% -- replaced with /var/log/ceph/ceph-client.admin.118760.log.gz 2023-10-14T18:20:53.137 INFO:teuthology.orchestra.run.smithi029.stderr: --verbose -- /var/log/ceph/ceph-client.admin.156378.log 2023-10-14T18:20:53.139 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.134293.log.gz 2023-10-14T18:20:53.140 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121544.log 2023-10-14T18:20:53.144 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.121544.log: /var/log/ceph/ceph-client.admin.156378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.158128.log 2023-10-14T18:20:53.146 INFO:teuthology.orchestra.run.smithi029.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91561.log 2023-10-14T18:20:53.147 INFO:teuthology.orchestra.run.smithi029.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.156378.log.gz 2023-10-14T18:20:53.149 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.91561.log: -- replaced with /var/log/ceph/ceph-client.admin.121544.log.gz 2023-10-14T18:20:53.150 INFO:teuthology.orchestra.run.smithi029.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.91561.log.gz 2023-10-14T18:20:53.152 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.158128.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.158128.log.gz 2023-10-14T18:20:53.153 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118330.log 2023-10-14T18:20:53.154 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2023-10-14T18:20:53.158 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118330.log: /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101740.log 2023-10-14T18:20:53.159 INFO:teuthology.orchestra.run.smithi029.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.118330.log.gz 2023-10-14T18:20:53.162 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.139003.log 2023-10-14T18:20:53.165 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.101740.log: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.0.148918.log 2023-10-14T18:20:53.166 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.139003.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.139003.log.gz 2023-10-14T18:20:53.168 INFO:teuthology.orchestra.run.smithi029.stderr: -- replaced with /var/log/ceph/ceph-client.admin.101740.log.gz 2023-10-14T18:20:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.0.148918.log: 0.0% -- replaced with /var/log/ceph/ceph-client.0.148918.log.gz 2023-10-14T18:20:53.171 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151397.log 2023-10-14T18:20:53.175 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95700.log 2023-10-14T18:20:53.177 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.121418.log 2023-10-14T18:20:53.180 INFO:teuthology.orchestra.run.smithi029.stderr: 88.7%/var/log/ceph/ceph-client.admin.95700.log: /var/log/ceph/ceph-client.admin.121418.log: -- replaced with /var/log/ceph/ceph-client.admin.151397.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.115535.log 2023-10-14T18:20:53.180 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:53.182 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.95700.log.gz 2023-10-14T18:20:53.183 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97095.log 2023-10-14T18:20:53.184 INFO:teuthology.orchestra.run.smithi029.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.121418.log.gz 2023-10-14T18:20:53.188 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.115535.log: 88.1%/var/log/ceph/ceph-client.admin.97095.log: -- replaced with /var/log/ceph/ceph-client.admin.115535.log.gz 2023-10-14T18:20:53.189 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.98296.log 2023-10-14T18:20:53.191 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106185.log 2023-10-14T18:20:53.193 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.98296.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.97095.log.gz 2023-10-14T18:20:53.196 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.106185.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.106185.log.gz 2023-10-14T18:20:53.196 INFO:teuthology.orchestra.run.smithi029.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.98296.log.gz 2023-10-14T18:20:53.197 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.151576.log 2023-10-14T18:20:53.199 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.127881.log 2023-10-14T18:20:53.201 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.151576.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.151576.log.gz 2023-10-14T18:20:53.203 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.125306.log 2023-10-14T18:20:53.205 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.127881.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.127881.log.gz 2023-10-14T18:20:53.205 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.156648.log 2023-10-14T18:20:53.207 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.125306.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.125306.log.gz 2023-10-14T18:20:53.209 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95826.log 2023-10-14T18:20:53.211 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.156648.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.156648.log.gz 2023-10-14T18:20:53.213 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.95826.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.95826.log.gz 2023-10-14T18:20:53.214 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.149639.log 2023-10-14T18:20:53.219 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.118120.log 2023-10-14T18:20:53.221 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.149639.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.123073.log 2023-10-14T18:20:53.221 INFO:teuthology.orchestra.run.smithi029.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.149639.log.gz 2023-10-14T18:20:53.222 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.118120.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.118120.log.gz 2023-10-14T18:20:53.225 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.152388.log 2023-10-14T18:20:53.226 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.123073.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.123073.log.gz 2023-10-14T18:20:53.231 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.112476.log 2023-10-14T18:20:53.232 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.152388.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.152388.log.gz 2023-10-14T18:20:53.234 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.112476.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.117075.log 2023-10-14T18:20:53.235 INFO:teuthology.orchestra.run.smithi029.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.112476.log.gz 2023-10-14T18:20:53.236 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.113964.log 2023-10-14T18:20:53.238 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.117075.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.117075.log.gz 2023-10-14T18:20:53.241 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.113964.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.113964.log.gz 2023-10-14T18:20:53.242 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.116949.log 2023-10-14T18:20:53.246 INFO:teuthology.orchestra.run.smithi029.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.143294.log 2023-10-14T18:20:53.248 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.116949.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.116949.log.gz 2023-10-14T18:20:53.253 INFO:teuthology.orchestra.run.smithi029.stderr:/var/log/ceph/ceph-client.admin.143294.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.143294.log.gz 2023-10-14T18:20:53.262 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2023-10-14T18:20:53.281 INFO:teuthology.orchestra.run.smithi186.stderr: 92.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2023-10-14T18:20:53.335 INFO:teuthology.orchestra.run.smithi029.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2023-10-14T18:20:54.078 INFO:teuthology.orchestra.run.smithi186.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2023-10-14T18:20:54.772 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2023-10-14T18:20:54.773 INFO:teuthology.orchestra.run.smithi029.stderr: 92.1% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2023-10-14T18:20:54.924 INFO:teuthology.orchestra.run.smithi186.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2023-10-14T18:20:54.932 INFO:teuthology.orchestra.run.smithi029.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2023-10-14T18:20:55.377 INFO:teuthology.orchestra.run.smithi186.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2023-10-14T18:20:55.380 INFO:teuthology.orchestra.run.smithi186.stderr: 2023-10-14T18:20:55.380 INFO:teuthology.orchestra.run.smithi186.stderr:real 0m4.099s 2023-10-14T18:20:55.381 INFO:teuthology.orchestra.run.smithi186.stderr:user 0m16.696s 2023-10-14T18:20:55.381 INFO:teuthology.orchestra.run.smithi186.stderr:sys 0m1.458s 2023-10-14T18:20:55.544 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2023-10-14T18:20:55.843 INFO:teuthology.orchestra.run.smithi029.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2023-10-14T18:20:55.888 INFO:teuthology.orchestra.run.smithi029.stderr: 93.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2023-10-14T18:20:56.260 INFO:teuthology.orchestra.run.smithi029.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2023-10-14T18:20:57.410 INFO:teuthology.orchestra.run.smithi029.stderr: 94.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2023-10-14T18:20:57.415 INFO:teuthology.orchestra.run.smithi029.stderr: 2023-10-14T18:20:57.416 INFO:teuthology.orchestra.run.smithi029.stderr:real 0m6.131s 2023-10-14T18:20:57.416 INFO:teuthology.orchestra.run.smithi029.stderr:user 0m23.751s 2023-10-14T18:20:57.416 INFO:teuthology.orchestra.run.smithi029.stderr:sys 0m4.878s 2023-10-14T18:20:57.416 INFO:tasks.ceph:Archiving logs... 2023-10-14T18:20:57.417 DEBUG:teuthology.misc:Transferring archived files from smithi029:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi029/log 2023-10-14T18:20:57.418 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T18:21:01.091 DEBUG:teuthology.misc:Transferring archived files from smithi186:/var/log/ceph to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi186/log 2023-10-14T18:21:01.092 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-14T18:21:03.313 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-10-14T18:21:03.325 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-10-14T18:21:03.325 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T18:21:03.335 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-10-14T18:21:03.496 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-14T18:21:03.507 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-14T18:21:03.508 DEBUG:teuthology.orchestra.run.smithi029:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T18:21:03.517 DEBUG:teuthology.orchestra.run.smithi186:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-14T18:21:03.569 INFO:teuthology.orchestra.run.smithi029.stderr:bash: line 1: ntpq: command not found 2023-10-14T18:21:03.574 INFO:teuthology.orchestra.run.smithi186.stderr:bash: line 1: ntpq: command not found 2023-10-14T18:21:03.579 INFO:teuthology.orchestra.run.smithi029.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T18:21:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:=============================================================================== 2023-10-14T18:21:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv01.front.sepia.ceph.com 2 7 377 85 +599us[ +550us] +/- 88ms 2023-10-14T18:21:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:^* hv02.front.sepia.ceph.com 3 8 377 82 -371us[ -420us] +/- 57ms 2023-10-14T18:21:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -110us[ -110us] +/- 63ms 2023-10-14T18:21:03.580 INFO:teuthology.orchestra.run.smithi029.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T18:21:03.584 INFO:teuthology.orchestra.run.smithi186.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-14T18:21:03.585 INFO:teuthology.orchestra.run.smithi186.stdout:=============================================================================== 2023-10-14T18:21:03.585 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 23 +597us[ +580us] +/- 89ms 2023-10-14T18:21:03.585 INFO:teuthology.orchestra.run.smithi186.stdout:^* hv02.front.sepia.ceph.com 3 7 377 23 -358us[ -375us] +/- 49ms 2023-10-14T18:21:03.585 INFO:teuthology.orchestra.run.smithi186.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 84 +51us[ +41us] +/- 92ms 2023-10-14T18:21:03.585 INFO:teuthology.orchestra.run.smithi186.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2023-10-14T18:21:03.586 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-14T18:21:03.596 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-14T18:21:03.597 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-14T18:21:03.607 DEBUG:teuthology.orchestra.run.smithi029:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-14T18:21:03.819 DEBUG:teuthology.orchestra.run.smithi186:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-14T18:21:04.112 DEBUG:teuthology.orchestra.run.smithi029:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T18:21:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:21:04.227 DEBUG:teuthology.orchestra.run.smithi186:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-14T18:21:04.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:21:04.336 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-14T18:21:04.348 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-14T18:21:04.369 INFO:teuthology.task.internal:Duration was 6860.169482 seconds 2023-10-14T18:21:04.370 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-14T18:21:04.381 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-14T18:21:04.381 DEBUG:teuthology.orchestra.run.smithi029:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T18:21:04.391 DEBUG:teuthology.orchestra.run.smithi186:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-14T18:21:04.542 INFO:teuthology.orchestra.run.smithi029.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T18:21:04.560 INFO:teuthology.orchestra.run.smithi186.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-14T18:21:04.894 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-14T18:21:04.894 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi029.front.sepia.ceph.com 2023-10-14T18:21:04.895 DEBUG:teuthology.orchestra.run.smithi029:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T18:21:04.960 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi186.front.sepia.ceph.com 2023-10-14T18:21:04.961 DEBUG:teuthology.orchestra.run.smithi186:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-14T18:21:05.042 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-14T18:21:05.043 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T18:21:05.052 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-14T18:21:05.155 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-14T18:21:05.155 DEBUG:teuthology.orchestra.run.smithi029:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T18:21:05.161 DEBUG:teuthology.orchestra.run.smithi186:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-14T18:21:05.432 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-14T18:21:05.443 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-14T18:21:05.443 DEBUG:teuthology.orchestra.run.smithi029:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T18:21:05.546 DEBUG:teuthology.orchestra.run.smithi186:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-14T18:21:05.652 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-14T18:21:05.664 DEBUG:teuthology.orchestra.run.smithi029:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T18:21:05.674 DEBUG:teuthology.orchestra.run.smithi186:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-14T18:21:05.787 INFO:teuthology.orchestra.run.smithi029.stdout:kernel.core_pattern = core 2023-10-14T18:21:05.787 INFO:teuthology.orchestra.run.smithi186.stdout:kernel.core_pattern = core 2023-10-14T18:21:05.843 DEBUG:teuthology.orchestra.run.smithi029:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T18:21:05.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:21:05.896 DEBUG:teuthology.orchestra.run.smithi186:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-14T18:21:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-14T18:21:05.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-14T18:21:05.966 INFO:teuthology.task.internal:Transferring archived files... 2023-10-14T18:21:05.967 DEBUG:teuthology.misc:Transferring archived files from smithi029:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi029 2023-10-14T18:21:05.967 DEBUG:teuthology.orchestra.run.smithi029:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T18:21:06.085 DEBUG:teuthology.misc:Transferring archived files from smithi186:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857/remote/smithi186 2023-10-14T18:21:06.086 DEBUG:teuthology.orchestra.run.smithi186:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-14T18:21:06.195 INFO:teuthology.task.internal:Removing archive directory... 2023-10-14T18:21:06.196 DEBUG:teuthology.orchestra.run.smithi029:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T18:21:06.204 DEBUG:teuthology.orchestra.run.smithi186:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-14T18:21:06.262 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-14T18:21:06.274 INFO:teuthology.task.internal:Not uploading archives. 2023-10-14T18:21:06.274 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-14T18:21:06.309 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-14T18:21:06.310 DEBUG:teuthology.orchestra.run.smithi029:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T18:21:06.318 DEBUG:teuthology.orchestra.run.smithi186:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-14T18:21:06.373 INFO:teuthology.orchestra.run.smithi029.stdout: 524301 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 14 18:21 /home/ubuntu/cephtest 2023-10-14T18:21:06.379 INFO:teuthology.orchestra.run.smithi186.stdout: 393229 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 14 18:21 /home/ubuntu/cephtest 2023-10-14T18:21:06.382 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-14T18:21:06.393 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-14T18:21:06.419 INFO:teuthology.nuke:Checking targets against current locks 2023-10-14T18:21:06.445 DEBUG:teuthology.nuke:shortname: smithi029 2023-10-14T18:21:06.446 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T18:21:06.471 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi029.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 16:16:54.151268', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIenUNMjS8jLbCKDUoA7LlCVAfZYMnpf8GelG1yGs0Dgy+1xDbuTILZXW1iFdqKmiHAbnH8lIKg+zsbMUw3XwYQ='} 2023-10-14T18:21:06.475 DEBUG:teuthology.nuke:shortname: smithi186 2023-10-14T18:21:06.475 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-14T18:21:06.499 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi186.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2023-10-14_16:12:46-fs-wip-mchangir-mgr-snap_schedule-support-subvol-and-group-arguments-distro-default-smithi/7427857', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-14 16:16:54.149741', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPU6xheaByxOWomGjN7eaKx5atbspCF5BwcdL4eoU8q6q3asLPy0dBw6fYz2uLTcrEYRjiqWnzO8he55rz4DVHM='} 2023-10-14T18:21:06.531 INFO:teuthology.orchestra.console.smithi029:Power off 2023-10-14T18:21:06.532 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T18:21:06.546 INFO:teuthology.orchestra.console.smithi186:Power off 2023-10-14T18:21:06.547 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-14T18:21:06.555 DEBUG:teuthology.orchestra.console.smithi029:power off output: Chassis Power Control: Down/Off 2023-10-14T18:21:06.556 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:06.568 DEBUG:teuthology.orchestra.console.smithi186:power off output: Chassis Power Control: Down/Off 2023-10-14T18:21:06.569 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:06.581 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-14T18:21:06.591 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2023-10-14T18:21:10.582 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:10.592 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:10.693 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is on 2023-10-14T18:21:10.701 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is on 2023-10-14T18:21:14.698 DEBUG:teuthology.orchestra.console.smithi029:pexpect command: ipmitool -H smithi029.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:14.709 DEBUG:teuthology.orchestra.console.smithi186:pexpect command: ipmitool -H smithi186.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-14T18:21:14.811 DEBUG:teuthology.orchestra.console.smithi029:check power output: Chassis Power is off 2023-10-14T18:21:14.819 DEBUG:teuthology.orchestra.console.smithi186:check power output: Chassis Power is off 2023-10-14T18:21:14.913 INFO:teuthology.orchestra.console.smithi029:Power off completed 2023-10-14T18:21:14.920 INFO:teuthology.orchestra.console.smithi186:Power off completed 2023-10-14T18:21:15.025 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 client.1-kernel-sha1: 47672119149b103ec7cec2cc65c063ccbb001ac7 description: fs/functional/{begin/{0-install 1-ceph 2-logrotate} clusters/1a3s-mds-4c-client conf/{client mds mon osd} distro/{centos_latest} mount/kclient/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile} tasks/snap-schedule} duration: 6860.169481515884 failure_reason: '"2023-10-14T16:55:33.247598+0000 mon.a (mon.0) 576 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY)" in cluster log' flavor: default owner: mchangir success: false 2023-10-14T18:21:15.026 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-14T18:21:15.102 INFO:teuthology.run:FAIL